diff --git a/.github/workflows/cultmesh-portability.yml b/.github/workflows/cultmesh-portability.yml index bafe70fc..0b8aca1d 100644 --- a/.github/workflows/cultmesh-portability.yml +++ b/.github/workflows/cultmesh-portability.yml @@ -158,7 +158,7 @@ jobs: # (only the interop example built, inside the C# store tests). The target directory is under RUNNER_TEMP, away from the checkout. - name: Verify the CultCache document owner with the Rust cache shell: bash - run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml + run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --manifest-path packages/cultcache-rs/Cargo.toml # tests/GameCult.Caching.Tests, the C# reference cache's own suite (about 1,260 tests, under a minute), ran in no # forge workflow; only the Mesh tests above exercised the reference. diff --git a/.github/workflows/cultnet-interop.yml b/.github/workflows/cultnet-interop.yml index 0d193aef..e053e123 100644 --- a/.github/workflows/cultnet-interop.yml +++ b/.github/workflows/cultnet-interop.yml @@ -135,10 +135,10 @@ jobs: python -m unittest discover -s packages/cultmesh-py/tests - name: Verify Rust RUDP reliability - run: cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml rudp + run: cargo test --manifest-path packages/cultnet-rs/Cargo.toml rudp - name: Verify Rust CultMesh runtime-local surface - run: cargo test --locked --manifest-path packages/cultmesh-rs/Cargo.toml + run: cargo test --manifest-path packages/cultmesh-rs/Cargo.toml - name: Verify C# RUDP reliability run: dotnet test tests/GameCult.Networking.Tests/GameCult.Networking.Tests.csproj --filter "FullyQualifiedName~Rudp" diff --git a/README.md b/README.md index a555aa92..b916cabe 100644 --- a/README.md +++ b/README.md @@ -212,8 +212,8 @@ identity-owned C# session after provider route replacement. Rust package tests: ```powershell -cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml -cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml +cargo test --manifest-path packages/cultcache-rs/Cargo.toml +cargo test --manifest-path packages/cultnet-rs/Cargo.toml ``` Kotlin package build: diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index 9136b667..8db67be5 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -3,7 +3,7 @@ CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local tool in `dotnet-tools.json`. Coverage is added project by project; a project with no `stryker-config.json` next to its test project has not been scoped -yet (see "Scope and follow-ups" below for what's outstanding). +yet (see "Follow-ups" below for what's outstanding). ## Running it @@ -126,17 +126,14 @@ project as precedent for the `vstest` caveat itself - that caveat is `mtp`-runner-specific and documented above for `static readonly` field initializers, and GameCult.Geometry has never actually hit it. -## Scope and follow-ups +## Follow-ups -CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the -projects currently scoped. GameCult.Geometry's `stryker-config.json` has no -`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and +CultMath.Tests and GameCult.Geometry.Tests are the only projects currently +scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner` +entry: the project is NUnit 4 with `NUnit3TestAdapter` and `Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3 -suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`) -and its `stryker-config.json` is driven the same way, by the repo-local -`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run -dotnet-stryker` from `tests/GameCult.Caching.Tests`. +suite). GameCult.Geometry has only ever been run `--since:`-scoped to the cut that added surface nets; a full-project baseline (running mutation against all of @@ -154,83 +151,7 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that run; a future cut whose diff is C#-only, against a base commit that already has `stryker-config.json`, should scope correctly. -### GameCult.Caching.Tests - -`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of -`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard -mutation level, `break` 0, the same thresholds as Geometry). Caching was -scoped before Networking because its suite is deterministic and quick (1415 -tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking -is about 20k lines and its tests lean on socket timing, so a mutant timing out -there says little about the test. A full Caching run is about 2,900 candidate -mutants (3,602 created, minus 730 that fail to compile) at 1.25 to 1.6 s -each, so about 1 to 1.3 hours; run it detached and read the JSON report, -never as one attached session. - -The first measurement was a narrow pass over the three files that carry the -store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`: - -```bash -dotnet tool run dotnet-stryker \ - -m '**/src/GameCult.Caching/StoreReplace.cs' \ - -m '**/src/GameCult.Caching/CultStoreLinks.cs' \ - -m '**/src/GameCult.Caching/CultElementIds.cs' -``` - -A clean single run of that command tests 243 to 265 mutants (the count moves with the revision) and scored 85.28% on -568a60fa, before any test of this scope was added; 86.79% on d38b5728, with -the first batch of tests; and 87.17% on 76d9e7d1, with the throwing-enumerator -case, in about 10 minutes. The first measurement of it overlapped two Stryker -runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`, -where tests that kill a mutant by hand reported `Survived` or `Timeout`) are -not measurements and are not quoted here. Never run two Stryker runs against -one checkout at once, and check `ps` before starting one. A clean single run -on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208 -`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor -was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now -kills; the other 11 are equivalent mutants. With that case a clean single run -on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%. - -Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each -failing when its mutant is applied by hand): the order of the id path (the -codec refuses at the first type MessagePack cannot allow, so order is what an -error names first; `Append` to `Prepend` at 163, 167 and 180); the -union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so -the red is a crashed test host: a depth guard would need production code in -`ConcreteTypes`, since the recursion is private and a stack overflow cannot be -caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an -id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the -`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter, -`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the -`[DataContract]` without `[MessagePackObject]` and `||` arms). - -Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full -suite; none can be told apart by any test: - -- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to - `true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument - changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument - (`DataContractAttribute` is `Inherited = false`). -- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which - are never empty. -- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial - capacity. -- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`; - `Classify` unwraps nullables itself, so the unwrap is dead. A cut - candidate, not a test gap. -- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and - `MapTypes`; `type` matches only when it is itself `IEnumerable<>`, - `IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second - matching interface, so the position of `type` never decides the answer. - -The Windows-only P/Invoke group is exercised only on Windows: its mutants are -`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a -Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its -negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the -negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the -same result. - -`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), +`GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), `GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes (`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own `stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for diff --git a/docs/start/README.md b/docs/start/README.md index d877469c..6a7d6d74 100644 --- a/docs/start/README.md +++ b/docs/start/README.md @@ -16,8 +16,8 @@ this page. | | CultCache | CultMath | Geometry | CultNet | CultMesh | | --- | --- | --- | --- | --- | --- | -| **C#** | [first call, tested](#c-first-call); [update, tested](#c-update-with-retry) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` | -| **TypeScript** | [first call, tested](#typescript-first-call); [update, tested](#typescript-update-with-retry) | not yet | not yet | [first call, tested](#typescript-cultnet-first-call); [package](../../packages/cultnet-ts/README.md) | [package](../../packages/cultmesh-ts/README.md); snippet not yet; [recipe: two-client .io game](io-game.md), tested with two live clients, browser page not yet run | +| **C#** | [first call, tested](#c-first-call) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` | +| **TypeScript** | [first call, tested](#typescript-first-call) | not yet | not yet | [first call, tested](#typescript-cultnet-first-call); [package](../../packages/cultnet-ts/README.md) | [package](../../packages/cultmesh-ts/README.md); snippet not yet; [recipe: two-client .io game](io-game.md), tested with two live clients, browser page not yet run | | **Rust** | [first call, tested](#rust-first-call) | [native core](../../packages/cultmath/native/cultmath-core); snippet not yet | not yet | [first call, tested](#rust-cultnet-first-call); [package](../../packages/cultnet-rs/README.md), [examples](../../packages/cultnet-rs/examples) | [package](../../packages/cultmesh-rs); snippet not yet | | **Python** | [first call, tested](#python-first-call) | not yet | not yet | [first call, tested](#python-cultnet-first-call); [package](../../packages/cultnet-py/README.md) | [package](../../packages/cultmesh-py/README.md); snippet not yet | @@ -70,37 +70,6 @@ public static class FirstCall `FirstCall.Run` is the call; a second test in that file fails if the block above stops matching it. More: `src/GameCult.Caching/README.md`. -## C# update with retry - -Change a record you may not be the only one changing. `Update` reads the record as its store holds it, hands your function a fresh -copy of it, and commits the result only if the record is still the one it read. If another writer got there first, it -reads again and runs your function again, up to five times. The source is -[`tests/GameCult.Caching.Tests/UpdateCallTests.cs`](../../tests/GameCult.Caching.Tests/UpdateCallTests.cs); the -snippet reuses `Item` from the first call. - -```csharp -using GameCult.Caching; -using GameCult.Caching.MessagePack; - -public static class UpdateCall -{ - public static CultUpdateResult Restock(string path, CultRecordKey key) - { - var cache = CultCacheMessagePack.Create(path); - return cache.Update(key, item => - { - item ??= new Item { Name = "Potion", Category = "Consumable" }; - item.Value += 10; - return item; - }); - } -} -``` - -The write is durable when `Update` returns, with no `PushAll`. A lost race re-runs the function, so keep it free of -side effects; when the attempts run out the result says `LostRace` instead of throwing. `UpsertAsync` plus -`PushAll` is the blind opt-out: last writer wins. - ## TypeScript first call Same shape in TypeScript: the document is a name, a schema id and a `parse`-able schema (zod here), @@ -150,36 +119,12 @@ async function firstCall(path: string) { The test imports from `../src/index` where the block reads `@gamecult/cultcache-ts`; a second test in that file fails if the block stops matching the rest. More: `packages/cultcache-ts/README.md`. -## TypeScript update with retry - -`update` changes a record from what the store holds now and commits only if it is still that record. The source is the -second region of -[`packages/cultcache-ts/test/first-call.test.ts`](../../packages/cultcache-ts/test/first-call.test.ts), which uses the -definitions of the first call above; a test in that file fails if the block stops matching the file. - -```ts -async function restock(path: string) { - const cache = CultCache.builder() - .withRegistry(registry) - .withGenericStore(new SingleFileMessagePackBackingStore(path)) - .build(); - await cache.pullAllBackingStores(); - return cache.update(itemDocument, "item:potion", (item) => { - const current = item ?? { name: "Potion", category: "Consumable", value: 0 }; - return { ...current, value: current.value + 10 }; - }); -} -``` - -The write is durable when the promise resolves. A lost race re-runs the function, so keep it free of side effects; when the -attempts run out the result says `lostRace` instead of throwing. `put` is the blind opt-out: last writer wins. - ## Rust first call Same shape with `derive(DatabaseEntry)`: slots are `#[cultcache(key = N)]`, so the `.cc` payload is the slot-indexed array the C# runtime writes. The source is [`packages/cultcache-rs/tests/first_call.rs`](../../packages/cultcache-rs/tests/first_call.rs), run by -`cargo test --locked --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block +`cargo test --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block below stops matching the file. The Rust runtime has no name lookup, so the read-back is by key. More: `packages/cultcache-rs/README.md`. @@ -396,7 +341,7 @@ async function firstCall() { ### Rust CultNet first call Source: [`packages/cultnet-rs/tests/first_call.rs`](../../packages/cultnet-rs/tests/first_call.rs), run by -`cargo test --locked --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`. +`cargo test --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`. ```rust use std::net::{TcpListener, TcpStream}; diff --git a/packages/cultcache-py/CHANGELOG.md b/packages/cultcache-py/CHANGELOG.md index 77324c91..ea3f72fd 100644 --- a/packages/cultcache-py/CHANGELOG.md +++ b/packages/cultcache-py/CHANGELOG.md @@ -8,17 +8,11 @@ ### Changed -- `store_write_lock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. `JsonLinesBackingStore` takes no cross-process lock (only its thread lock, under which it already refuses a link), so it has no lock wait to re-check across. Pinned by a test that puts the link while a foreign process holds the lock. -- `CultCache.is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store. -- `JsonLinesBackingStore` writes LF line endings on Windows. It wrote CRLF there, through `write_text` newline translation, before the commit_store_file change (#154, adbbb780). - `msgpack` is a hard dependency, as `pyproject.toml` declared it. The optional-import fallbacks and the `cultcache-py[msgpack]` extra they named (which never existed) are gone: a missing msgpack is an `ImportError` at import. -- `JsonLinesBackingStore` writes through the same `commit_store_file` as the single-file store: a unique staged file, fsynced before the replace and removed when the replace fails, so a refused replace no longer leaves `.tmp` behind. It staged a fixed `.tmp` without an fsync. - The single-file store's `push`, `push_all` and `delete` take the cross-process lock on `.lock` (flock; LockFileEx over the whole file on Windows) across the read and the replace, waiting up to `lock_timeout` seconds (30 by default, `TimeoutError` after), and exclude the C#, Rust and TypeScript runtimes' writers. The staged file is `.tmp--`, fsynced before the replace, with a directory fsync after it off Windows. A fixed `.tmp` is no longer used. ### Breaking -- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`. -- A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work. - A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`. - A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` ` payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#. - A cache holds one record per key whatever its type (`variants:ruling:cache-refuses-retype`). A `put`, `put_envelope`, `put_envelopes` or `put_global` of a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store, and changes nothing; retype a key by deleting it, then putting the new type. A write may replace another registered type's record only when the written type owns its schema id (declares it or lists it compatible) and both types share one home store. A load that reads one key from two stores, whatever the types, is refused with `CultCacheError` naming the key and both types, and admits nothing. A store that holds a record no registered type claims now loads and is carried: the cache does not serve it, the store keeps its bytes through every write, `delete` or `put` at its key is refused with `SchemaConflictError`, and registering a type that declares its schema id claims it at the next load. It raised `CultCacheError` "Unknown persisted document type". Two global types share the key `__global__`, so a second global type in one cache is now refused, and a store that already holds two global types at `__global__` (a JSONL store written by caches that never pulled could produce one) no longer loads: the pull raises `CultCacheError` and admits nothing. C# refuses a second persisted global on pull the same way (`cultcache-store-composition.md`, "Globals never invented"); repair such a file by deleting one of the two records with the runtime that wrote it. diff --git a/packages/cultcache-py/src/cultcache_py/__init__.py b/packages/cultcache-py/src/cultcache_py/__init__.py index a2752511..69828bf3 100644 --- a/packages/cultcache-py/src/cultcache_py/__init__.py +++ b/packages/cultcache-py/src/cultcache_py/__init__.py @@ -19,7 +19,7 @@ from .documents import ( read_payload_tree, ) from .leaves import SlotCodec -from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreLinkedError, StoreUnreadableError +from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreUnreadableError __all__ = [ "BackingStore", @@ -38,7 +38,6 @@ __all__ = [ "SchemaConflictError", "SingleFileMessagePackBackingStore", "SlotCodec", - "StoreLinkedError", "StoreUnreadableError", "UnionArm", "UnreadSlots", diff --git a/packages/cultcache-py/src/cultcache_py/backing_store.py b/packages/cultcache-py/src/cultcache_py/backing_store.py index bded4b11..5a21e496 100644 --- a/packages/cultcache-py/src/cultcache_py/backing_store.py +++ b/packages/cultcache-py/src/cultcache_py/backing_store.py @@ -67,20 +67,10 @@ class BackingStore(Protocol): def delete(self, type: str, key: str) -> None: ... - def pull_or_absent(self) -> list[CultCacheEnvelope] | None: - """The envelopes the reader reads (empty included), None when nothing is at the path; raises when the path cannot be read. - - The default is pull_all, which cannot say absent: a store that inherits it is always a store the reader read. A store kind - that can tell nothing-at-the-path overrides this, and the default push_all then writes where nothing is on a soft push.""" - return self.pull_all() - - def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None: - # Soft: write only where pull_or_absent says nothing is; skip over a store it read, empty included; its raise refuses. - pulled = self.pull_or_absent() - if soft and pulled is not None: - return - pulled = pulled or [] - existing = {(envelope.type, envelope.key): envelope for envelope in pulled} + def push_all(self, envelopes: list[CultCacheEnvelope]) -> None: + existing = { + (envelope.type, envelope.key): envelope for envelope in self.pull_all() + } for envelope in envelopes: existing[(envelope.type, envelope.key)] = envelope self._replace_all(list(existing.values())) diff --git a/packages/cultcache-py/src/cultcache_py/cache.py b/packages/cultcache-py/src/cultcache_py/cache.py index 80a12706..85b01d89 100644 --- a/packages/cultcache-py/src/cultcache_py/cache.py +++ b/packages/cultcache-py/src/cultcache_py/cache.py @@ -636,8 +636,7 @@ class CultCache: a record held read-only (its payload carried slots past the declared ones, at any depth, and nothing consents to the loss, so writing it would drop data a newer writer stored), for a key a store carries foreign, and for a key held as another type's record this document does not own. Advisory (the write still decides): it answers at call time, and a key nothing holds is - writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it with - `StoreLinkedError`.""" + writable.""" self._assert_registered(document) try: self._refuse_at(document, key) diff --git a/packages/cultcache-py/src/cultcache_py/stores.py b/packages/cultcache-py/src/cultcache_py/stores.py index 894252c8..c15ac32e 100644 --- a/packages/cultcache-py/src/cultcache_py/stores.py +++ b/packages/cultcache-py/src/cultcache_py/stores.py @@ -76,17 +76,10 @@ class JsonLinesBackingStore: self._lock = threading.RLock() def pull_all(self) -> list[CultCacheEnvelope]: - return self._pull() or [] - - def pull_or_absent(self) -> list[CultCacheEnvelope] | None: - return self._pull() - - def _pull(self) -> list[CultCacheEnvelope] | None: - """The envelopes the reader reads, or None when nothing is at the path.""" with self._lock: text = _read_store(self.path, lambda path: path.read_text(encoding="utf-8")) if text is None: - return None + return [] envelopes: list[CultCacheEnvelope] = [] for line_number, line in enumerate(text.splitlines(), start=1): if not line.strip(): @@ -108,31 +101,26 @@ class JsonLinesBackingStore: return envelopes def push(self, envelope: CultCacheEnvelope) -> None: - self.push_all([envelope]) + with self._lock: + existing = {(item.type, item.key): item for item in self.pull_all()} + existing[(envelope.type, envelope.key)] = envelope + self._replace_all(list(existing.values())) - def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None: - def merge(existing: dict[tuple[str, str], CultCacheEnvelope]) -> None: + def push_all(self, envelopes: list[CultCacheEnvelope]) -> None: + with self._lock: + existing = {(item.type, item.key): item for item in self.pull_all()} for envelope in envelopes: existing[(envelope.type, envelope.key)] = envelope - self._rewrite(merge, soft) + self._replace_all(list(existing.values())) def delete(self, type: str, key: str) -> None: - self._rewrite(lambda existing: existing.pop((type, key), None)) - - def _rewrite(self, change: Callable[[dict[tuple[str, str], CultCacheEnvelope]], object], soft: bool = False) -> None: - """The one entry of every write: refuses a link at the path, then reads the file (a soft write stops here when the - reader read it), applies the change and replaces it.""" with self._lock: - _refuse_linked(self.path) - arrived = self._pull() - if soft and arrived is not None: - return - existing = {(item.type, item.key): item for item in arrived or []} - change(existing) - self._replace_all(list(existing.values())) + existing = [item for item in self.pull_all() if not (item.type == type and item.key == key)] + self._replace_all(existing) def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: self.path.parent.mkdir(parents=True, exist_ok=True) + temp = self.path.with_suffix(self.path.suffix + ".tmp") lines = [] for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)): lines.append(json.dumps({ @@ -141,7 +129,8 @@ class JsonLinesBackingStore: "payload": base64.b64encode(envelope.payload).decode("ascii"), "stored_at": envelope.stored_at, }, ensure_ascii=False, sort_keys=True)) - commit_store_file(self.path, ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8")) + temp.write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8") + temp.replace(self.path) class SingleFileMessagePackBackingStore: @@ -153,20 +142,13 @@ class SingleFileMessagePackBackingStore: self._format = STORE_FORMAT_VERSION def pull_all(self) -> list[CultCacheEnvelope]: - return self._pull() or [] - - def pull_or_absent(self) -> list[CultCacheEnvelope] | None: - return self._pull() - - def _pull(self) -> list[CultCacheEnvelope] | None: - """The envelopes the reader reads, or None when nothing is at the path.""" with self._lock: # An empty file is not a store (no writer leaves one), so it is decoded and refused. # The disk decides the header: a file that is gone is not marked. self._format = STORE_FORMAT_VERSION data = _read_store(self.path, Path.read_bytes) if data is None: - return None + return [] try: decoded = msgpack.unpackb(data, raw=False) snapshot = _decode_snapshot(decoded) @@ -181,14 +163,9 @@ class SingleFileMessagePackBackingStore: with self._write_transaction(): self._replace_all([envelope], self.pull_all()) - def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None: - """A soft push writes only where nothing is at the path: a store the reader reads is left byte for byte, and a file it - cannot read is refused by the same read, as in TS and Rust.""" + def push_all(self, envelopes: list[CultCacheEnvelope]) -> None: with self._write_transaction(): - arrived = self._pull() - if soft and arrived is not None: - return - self._replace_all(list(envelopes), arrived or []) + self._replace_all(list(envelopes), self.pull_all()) def delete(self, type: str, key: str) -> None: with self._write_transaction(): @@ -346,7 +323,7 @@ def commit_store_file(path: Path, data: bytes) -> None: os.fsync(fd) finally: os.close(fd) - _replace_store(staged, path) + os.replace(staged, path) except BaseException: try: os.unlink(staged) @@ -361,136 +338,11 @@ def commit_store_file(path: Path, data: bytes) -> None: os.close(directory) -REPLACE_ATTEMPTS = 25 -REPLACE_RETRY_SECONDS = 0.02 -_ERROR_ACCESS_DENIED = 5 -_ERROR_SHARING_VIOLATION = 32 -_ERROR_LOCK_VIOLATION = 33 -_POSIX_UNSUPPORTED = (1, 50, 87) # incorrect function, not supported, invalid parameter: FAT, exFAT, before Windows 10 1607 -MOVEFILE_REPLACE_WRITE_THROUGH = 0x9 # MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH - - -class ReplaceKernel: - """The kernel32 calls a Windows store replace needs, each answering 0 or a Win32 error code.""" - - def rename_posix(self, staged: str, destination: str) -> int: - """CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle.""" - raise NotImplementedError - - def move_file_ex(self, staged: str, destination: str, flags: int) -> int: - raise NotImplementedError - - def sleep(self, seconds: float) -> None: - time.sleep(seconds) - - -def _replace_store(staged: str | os.PathLike[str], destination: str | os.PathLike[str]) -> None: - """Replaces `destination` with `staged`: POSIX-semantics rename first on Windows (so a reader holding the store open - keeps it and rarely sees it absent; a stalled rename can still leave it absent for 5-47 ms on NTFS, - variants:ruling:posix-rename-absent-window-document), os.replace elsewhere.""" - if sys.platform == "win32": - replace_store_windows(os.fspath(staged), os.fspath(destination), WindowsReplaceKernel()) - else: - os.replace(staged, destination) - - -def replace_store_windows(staged: str, destination: str, kernel: ReplaceKernel) -> None: - """POSIX-semantics rename, falling back for the rest of the call to write-through MoveFileExW where POSIX semantics are - unsupported; errors 5, 32 and 33 retry up to 25 times 20 ms apart, any other code raises at once. The TypeScript and C# - runtimes take the same decision table.""" - posix = True - last = 0 - for _ in range(REPLACE_ATTEMPTS): - code = kernel.rename_posix(staged, destination) if posix else kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) - if posix and code in _POSIX_UNSUPPORTED: - posix = False - code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) - if code == 0: - return - if code not in (_ERROR_ACCESS_DENIED, _ERROR_SHARING_VIOLATION, _ERROR_LOCK_VIOLATION): - raise _replace_error(f"CultCache could not replace the store: Win32 error {code}.", code) - last = code - kernel.sleep(REPLACE_RETRY_SECONDS) - raise _replace_error(f"CultCache could not replace the store after {REPLACE_ATTEMPTS} attempts: last Win32 error {last}.", last) - - -def _replace_error(message: str, code: int) -> OSError: - error = OSError(message) - error.winerror = code # type: ignore[attr-defined] - return error - - -class WindowsReplaceKernel(ReplaceKernel): - """kernel32 bound by ctypes, as _try_lock_windows binds it. Windows only.""" - - def __init__(self) -> None: - import ctypes - from ctypes import wintypes - - self._ctypes = ctypes - k = self._k = ctypes.WinDLL("kernel32", use_last_error=True) - k.CreateFileW.restype = wintypes.HANDLE - k.CreateFileW.argtypes = [wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p, wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE] - k.SetFileInformationByHandle.restype = wintypes.BOOL - k.SetFileInformationByHandle.argtypes = [wintypes.HANDLE, ctypes.c_int, ctypes.c_void_p, wintypes.DWORD] - k.CloseHandle.restype = wintypes.BOOL - k.CloseHandle.argtypes = [wintypes.HANDLE] - k.MoveFileExW.restype = wintypes.BOOL - k.MoveFileExW.argtypes = [wintypes.LPCWSTR, wintypes.LPCWSTR, wintypes.DWORD] - - def rename_posix(self, staged: str, destination: str) -> int: - ctypes, k = self._ctypes, self._k - delete_synchronize, share_all, open_existing, normal = 0x00110000, 0x7, 3, 0x80 - handle = k.CreateFileW(staged, delete_synchronize, share_all, None, open_existing, normal, None) - if handle is None or ctypes.c_void_p(handle).value in (None, ctypes.c_void_p(-1).value): - return ctypes.get_last_error() - try: - # FILE_RENAME_INFO: Flags (DWORD), RootDirectory (pointer-aligned HANDLE), FileNameLength (DWORD), FileName (WCHAR[]). - name = os.path.abspath(destination).encode("utf-16-le") - pointer = ctypes.sizeof(ctypes.c_void_p) - length_at = pointer + pointer - info = ctypes.create_string_buffer(length_at + 4 + len(name) + 2) - info[0:4] = (0x3).to_bytes(4, "little") # FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS - info[length_at:length_at + 4] = len(name).to_bytes(4, "little") - info[length_at + 4:length_at + 4 + len(name)] = name - file_rename_info_ex = 22 - if k.SetFileInformationByHandle(handle, file_rename_info_ex, info, len(info)): - return 0 - return ctypes.get_last_error() - finally: - k.CloseHandle(handle) - - def move_file_ex(self, staged: str, destination: str, flags: int) -> int: - return 0 if self._k.MoveFileExW(staged, destination, flags) else self._ctypes.get_last_error() - - -class StoreLinkedError(OSError): - """A write was asked of a store path that is a link: a replace there would swap the link for a regular file.""" - - def __init__(self, path: Path) -> None: - super().__init__(f"CultCache store {path} is a link, and a write at it would replace the link; configure the link's target as the store path.") - self.path = path - - -def _refuse_linked(path: Path) -> None: - """Refuses a link at the store path: a file or directory link, and on Windows a junction or any reparse point (islink misses - junctions before Python 3.12). Absent is not a link.""" - linked = os.path.islink(path) - if not linked and sys.platform == "win32": - try: - linked = bool(os.lstat(path).st_file_attributes & 0x400) # FILE_ATTRIBUTE_REPARSE_POINT - except FileNotFoundError: - pass - if linked: - raise StoreLinkedError(path) - - @contextmanager def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Iterator[None]: """Holds the exclusive store lock on `.lock`, the lock the C#, Rust and TypeScript runtimes take: flock on Unix, LockFileEx over the whole file on Windows, polled every 10 ms with no handle held between polls. The lock file is created if absent and never written or deleted.""" - _refuse_linked(path) lock_path = path.with_name(path.name + ".lock") deadline = time.monotonic() + timeout while True: @@ -501,8 +353,6 @@ def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Itera raise TimeoutError(f"CultCache timed out after {timeout} s waiting for the store lock {lock_path}.") time.sleep(_LOCK_POLL_SECONDS) try: - # A link may have been put at the path while the lock was awaited; the look above only saw the path before. - _refuse_linked(path) yield finally: release() diff --git a/packages/cultcache-py/tests/test_cultcache.py b/packages/cultcache-py/tests/test_cultcache.py index 6b95f638..a8c1dbe7 100644 --- a/packages/cultcache-py/tests/test_cultcache.py +++ b/packages/cultcache-py/tests/test_cultcache.py @@ -12,7 +12,6 @@ from cultcache_py import ( CultCache, JsonLinesBackingStore, SingleFileMessagePackBackingStore, - StoreLinkedError, StoreUnreadableError, define_database_entry_type, define_document_type, @@ -1432,10 +1431,10 @@ class CultCacheTests(unittest.TestCase): self.assertNotIn("SECRET", str(caught.exception)) self.assertIn("not a valid store", str(caught.exception)) - # A write at a store path that is a link is refused: a replace there would swap the link for a regular file and - # detach the store from its target. Reads through a live link keep working. + # What a write does through a symbolic link today. R3 decides whether a write resolves + # the link or refuses it; this pins the current behaviour so that change is deliberate. @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") - def test_a_write_at_a_live_link_is_refused_and_nothing_changes_while_a_read_through_it_works(self) -> None: + def test_r3_decides_resolve_or_refuse_a_push_through_a_live_link_replaces_the_link(self) -> None: with tempfile.TemporaryDirectory() as tmp: volume = Path(tmp) / "volume" volume.mkdir() @@ -1443,91 +1442,12 @@ class CultCacheTests(unittest.TestCase): envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z") SingleFileMessagePackBackingStore(target).push(envelope) before = target.read_bytes() - volume_before = sorted(p.name for p in volume.iterdir()) - link = Path(tmp) / "link.cc" + link = Path(tmp) / "store.cc" link.symlink_to(target) - store = SingleFileMessagePackBackingStore(link) - for write in ( - lambda: store.push(replace(envelope, key="b")), - lambda: store.push_all([replace(envelope, key="b")]), - lambda: store.delete("t", "a"), - ): - with self.assertRaises(StoreLinkedError) as caught: - write() - self.assertIn(str(link), str(caught.exception)) - self.assertTrue(link.is_symlink()) + SingleFileMessagePackBackingStore(link).push(replace(envelope, key="b")) + self.assertFalse(link.is_symlink()) + self.assertTrue(link.is_file()) self.assertEqual(target.read_bytes(), before) - self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.cc", "volume"]) - self.assertEqual(sorted(p.name for p in volume.iterdir()), volume_before) - self.assertEqual([item.key for item in store.pull_all()], ["a"]) - - @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") - def test_a_json_lines_write_at_a_live_link_is_refused_and_the_link_survives(self) -> None: - with tempfile.TemporaryDirectory() as tmp: - volume = Path(tmp) / "volume" - volume.mkdir() - target = volume / "store.jsonl" - envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z") - JsonLinesBackingStore(target).push(envelope) - before = target.read_bytes() - link = Path(tmp) / "link.jsonl" - link.symlink_to(target) - store = JsonLinesBackingStore(link) - for write in ( - lambda: store.push(replace(envelope, key="b")), - lambda: store.push_all([replace(envelope, key="b")]), - lambda: store.delete("t", "a"), - ): - with self.assertRaises(StoreLinkedError): - write() - self.assertTrue(link.is_symlink()) - self.assertEqual(target.read_bytes(), before) - self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.jsonl", "volume"]) - self.assertEqual(sorted(p.name for p in volume.iterdir()), ["store.jsonl"]) - - # The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse. - @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") - def test_is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused(self) -> None: - with tempfile.TemporaryDirectory() as tmp: - volume = Path(tmp) / "volume" - volume.mkdir() - target = volume / "store.cc" - document = item_doc() - seed = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(target)).build() - seed.put(document, "held", Item(name="Held", category="c", value=1)) - before = target.read_bytes() - link = Path(tmp) / "link.cc" - link.symlink_to(target) - cache = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(link)).build() - cache.pull_all_backing_stores() - names_before = (sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir())) - - self.assertTrue(cache.is_writable(document, "held")) - self.assertTrue(cache.is_writable(document, "fresh")) - with self.assertRaises(StoreLinkedError) as caught: - cache.put(document, "fresh", Item(name="Fresh", category="d", value=2)) - self.assertEqual(Path(caught.exception.path), link) - self.assertTrue(link.is_symlink()) - self.assertEqual(target.read_bytes(), before) - self.assertEqual((sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir())), names_before) - - def test_store_linked_error_is_an_os_error_carrying_the_path(self) -> None: - error = StoreLinkedError(Path("some/store.cc")) - self.assertIsInstance(error, OSError) - self.assertEqual(error.path, Path("some/store.cc")) - - @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") - def test_a_write_at_a_live_directory_link_is_refused_as_a_link(self) -> None: - with tempfile.TemporaryDirectory() as tmp: - real = Path(tmp) / "real" - real.mkdir() - link = Path(tmp) / "dirlink" - link.symlink_to(real, target_is_directory=True) - with self.assertRaises(StoreLinkedError): - SingleFileMessagePackBackingStore(link).push( - CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")) - self.assertTrue(link.is_symlink()) - self.assertEqual(list(real.iterdir()), []) @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") def test_a_dangling_symlink_at_a_store_path_is_an_os_error_and_nothing_is_written(self) -> None: @@ -1540,7 +1460,7 @@ class CultCacheTests(unittest.TestCase): store = store_type(path) with self.assertRaises(FileNotFoundError): store.pull_all() - with self.assertRaises(StoreLinkedError): + with self.assertRaises(FileNotFoundError): store.push(CultCacheEnvelope(key="k", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")) self.assertTrue(path.is_symlink()) self.assertEqual(list(volume.iterdir()), []) diff --git a/packages/cultcache-py/tests/test_soft_push_all.py b/packages/cultcache-py/tests/test_soft_push_all.py deleted file mode 100644 index d30483a3..00000000 --- a/packages/cultcache-py/tests/test_soft_push_all.py +++ /dev/null @@ -1,140 +0,0 @@ -from __future__ import annotations - -import shutil -import tempfile -import unittest -from pathlib import Path - -from cultcache_py import JsonLinesBackingStore, SingleFileMessagePackBackingStore, StoreUnreadableError -from cultcache_py.backing_store import BackingStore, CultCacheEnvelope - - -def envelope(key: str) -> CultCacheEnvelope: - return CultCacheEnvelope(key=key, type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z") - - -class SoftPushAllTests(unittest.TestCase): - """A soft push writes where nothing is, skips a store it reads, and refuses one it cannot read (TS and Rust parity).""" - - def setUp(self) -> None: - self.dir = Path(tempfile.mkdtemp()) - self.addCleanup(shutil.rmtree, self.dir, ignore_errors=True) - self.path = self.dir / "state.cc" - - def test_over_a_readable_store_it_leaves_the_file_byte_for_byte(self) -> None: - for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): - with self.subTest(make.__name__): - path = self.dir / make.__name__ - store = make(path) - store.push_all([envelope("a")]) - before = path.read_bytes() - store.push_all([envelope("b")], soft=True) - self.assertEqual(before, path.read_bytes()) - self.assertEqual(["a"], [item.key for item in make(path).pull_all()]) - store.push_all([envelope("b")]) # a plain push still lands what the caller holds - self.assertEqual(["a", "b"], sorted(item.key for item in make(path).pull_all())) - - def test_over_a_readable_store_holding_no_records_it_leaves_the_file_byte_for_byte(self) -> None: - for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): - with self.subTest(make.__name__): - path = self.dir / make.__name__ - store = make(path) - store.push_all([envelope("a")]) - store.delete("t", "a") - self.assertTrue(path.exists()) - self.assertEqual([], make(path).pull_all()) - before = path.read_bytes() - store.push_all([envelope("b")], soft=True) - self.assertEqual(before, path.read_bytes()) - self.assertEqual([], make(path).pull_all()) - - def test_over_an_absent_path_it_writes(self) -> None: - for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): - with self.subTest(make.__name__): - path = self.dir / make.__name__ - make(path).push_all([envelope("a")], soft=True) - self.assertEqual(["a"], [item.key for item in make(path).pull_all()]) - - def test_over_an_unreadable_file_it_refuses_and_leaves_it(self) -> None: - junk = b"\xc1\x01\x02\x03" - self.path.write_bytes(junk) - with self.assertRaises(StoreUnreadableError): - SingleFileMessagePackBackingStore(self.path).push_all([envelope("a")], soft=True) - self.assertEqual(junk, self.path.read_bytes()) - - def test_over_an_unreadable_jsonl_file_it_refuses_and_leaves_it(self) -> None: - junk = b"this is not json\n" - self.path.write_bytes(junk) - with self.assertRaises(ValueError): - JsonLinesBackingStore(self.path).push_all([envelope("a")], soft=True) - self.assertEqual(junk, self.path.read_bytes()) - - def test_the_protocol_default_push_all_decides_soft_by_what_pull_or_absent_says(self) -> None: - class Memory(BackingStore): - def __init__(self) -> None: - self.items: list[CultCacheEnvelope] | None = None # None: nothing at the path - - def pull_all(self) -> list[CultCacheEnvelope]: - return list(self.items or []) - - def pull_or_absent(self) -> list[CultCacheEnvelope] | None: - return None if self.items is None else list(self.items) - - def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: - self.items = list(envelopes) - - store = Memory() - store.push_all([envelope("a")], soft=True) # nothing at the path: writes - self.assertEqual(["a"], [item.key for item in store.items or []]) - store.push_all([envelope("b")], soft=True) # a store with records: skips - self.assertEqual(["a"], [item.key for item in store.items or []]) - store.items = [] - store.push_all([envelope("b")], soft=True) # a store the reader read, empty: skips - self.assertEqual([], store.items) - store.push_all([envelope("b")]) # plain: lands - self.assertEqual(["b"], [item.key for item in store.items]) - - def test_a_store_that_only_pulls_all_is_read_so_a_soft_push_skips_it(self) -> None: - class PullOnly(BackingStore): - def __init__(self) -> None: - self.items: list[CultCacheEnvelope] = [] - - def pull_all(self) -> list[CultCacheEnvelope]: - return list(self.items) - - def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: - self.items = list(envelopes) - - store = PullOnly() - store.push_all([envelope("a")], soft=True) - self.assertEqual([], store.items) - - def test_the_single_file_and_jsonl_stores_say_absent_and_read_through_pull_or_absent(self) -> None: - for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): - with self.subTest(make.__name__): - path = self.dir / make.__name__ - store = make(path) - self.assertIsNone(store.pull_or_absent()) - store.push_all([envelope("a")]) - self.assertEqual(["a"], [item.key for item in store.pull_or_absent() or []]) - - def test_the_protocol_default_push_all_refuses_where_its_pull_refuses(self) -> None: - class Unreadable(BackingStore): - replaced = False - - def pull_all(self) -> list[CultCacheEnvelope]: - raise StoreUnreadableError(Path("x"), ValueError("junk")) - - def pull_or_absent(self) -> list[CultCacheEnvelope] | None: - raise StoreUnreadableError(Path("x"), ValueError("junk")) - - def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: - Unreadable.replaced = True - - with self.assertRaises(StoreUnreadableError): - Unreadable().push_all([envelope("a")], soft=True) - self.assertFalse(Unreadable.replaced) - - -if __name__ == "__main__": - unittest.main() diff --git a/packages/cultcache-py/tests/test_store_lock.py b/packages/cultcache-py/tests/test_store_lock.py index 364106bf..5ee29aa2 100644 --- a/packages/cultcache-py/tests/test_store_lock.py +++ b/packages/cultcache-py/tests/test_store_lock.py @@ -1,25 +1,21 @@ from __future__ import annotations import os -import select import shutil import subprocess import sys import tempfile -import threading import time import unittest from pathlib import Path from unittest import mock from cultcache_py import SingleFileMessagePackBackingStore, stores -from cultcache_py.stores import StoreLinkedError, store_write_lock from cultcache_py.backing_store import CultCacheEnvelope REPO = Path(__file__).resolve().parents[3] TS_STORE_FILE = REPO / "packages" / "cultcache-ts" / "dist" / "store-file.js" HOLD_SECONDS = 1.5 -HELD_WAIT_SECONDS = 3 # A holder that takes the lock the way the Rust runtime does (an exclusive file lock), not through this package. PY_HOLDER = """ @@ -45,14 +41,7 @@ def envelope(key: str) -> CultCacheEnvelope: def start_holder(argv: list[str]) -> subprocess.Popen[str]: holder = subprocess.Popen(argv, stdout=subprocess.PIPE, text=True) - assert holder.stdout is not None - ready, _, _ = select.select([holder.stdout], [], [], HELD_WAIT_SECONDS) # a holder blocked on a leaked lock never prints - if not ready: - holder.kill() - holder.wait() - holder.stdout.close() - raise AssertionError(f"the holder never took {argv[-2]}.lock within {HELD_WAIT_SECONDS} s: an earlier writer leaked the lock") - assert holder.stdout.readline().strip() == "HELD" + assert holder.stdout is not None and holder.stdout.readline().strip() == "HELD" holder.held_at = time.monotonic() # type: ignore[attr-defined] return holder @@ -84,48 +73,6 @@ class StoreLockTests(unittest.TestCase): holder = start_holder(["node", "-e", TS_HOLDER, str(TS_STORE_FILE), str(self.path), str(HOLD_SECONDS)]) self.assert_write_waits_for(holder) - def test_a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it(self) -> None: - target = self.dir / "volume" / "store.cc" - target.parent.mkdir() - SingleFileMessagePackBackingStore(target).push(envelope("a")) - before = target.read_bytes() - writes = { - "push": lambda store: store.push(envelope("b")), - "push_all": lambda store: store.push_all([envelope("b")]), - "delete": lambda store: store.delete("t", "a"), - } - missing = self.dir / "volume" / "missing.cc" - for (name, write), link_to in [(item, to) for item in writes.items() for to in (target, missing)]: - with self.subTest(name, dangling=link_to == missing): - if os.path.lexists(self.path): - os.unlink(self.path) - holder = start_holder([sys.executable, "-c", PY_HOLDER, str(self.path), str(HOLD_SECONDS)]) - self.addCleanup(holder.wait) - self.addCleanup(holder.stdout.close) # type: ignore[union-attr] - outcome: list[BaseException | None] = [] - - def run() -> None: - try: - write(SingleFileMessagePackBackingStore(self.path, lock_timeout=10)) - outcome.append(None) - except BaseException as exc: # noqa: BLE001 - the thread hands the outcome to the test - outcome.append(exc) - - writer = threading.Thread(target=run) - writer.start() - time.sleep(0.3) - os.symlink(link_to, self.path) # after the writer's pre-lock look, while it waits - writer.join() - holder.wait() - self.assertEqual(1, len(outcome)) - self.assertIsInstance(outcome[0], StoreLinkedError) - self.assertTrue(os.path.islink(self.path), "the link was replaced") - self.assertEqual(before, target.read_bytes()) - self.assertFalse(missing.exists(), "the dangling target was created") - os.unlink(self.path) # a link is refused before the lock, so remove it to probe the writer's own lock file - with store_write_lock(self.path, timeout=0.2): # the lock is free - pass - def test_the_lock_is_released_after_each_write(self) -> None: store = SingleFileMessagePackBackingStore(self.path, lock_timeout=0.2) for key in "abc": diff --git a/packages/cultcache-py/tests/test_store_replace.py b/packages/cultcache-py/tests/test_store_replace.py deleted file mode 100644 index af8a2595..00000000 --- a/packages/cultcache-py/tests/test_store_replace.py +++ /dev/null @@ -1,137 +0,0 @@ -from __future__ import annotations - -import os -import sys -import tempfile -import unittest -from pathlib import Path -from unittest import mock - -from cultcache_py import SingleFileMessagePackBackingStore, stores -from cultcache_py.backing_store import CultCacheEnvelope -from cultcache_py.stores import MOVEFILE_REPLACE_WRITE_THROUGH, ReplaceKernel, replace_store_windows - - -class FakeKernel(ReplaceKernel): - def __init__(self, posix: list[int], move_ex: list[int]) -> None: - self.posix, self.move_ex = list(posix), list(move_ex) - self.log: list[str] = [] - self.sleeps: list[float] = [] - - def rename_posix(self, staged, destination): - self.log.append("posix") - return self.posix.pop(0) if self.posix else 0 - - def move_file_ex(self, staged, destination, flags): - self.log.append(f"moveex:{flags}") - return self.move_ex.pop(0) if self.move_ex else 0 - - def sleep(self, seconds): - self.sleeps.append(seconds) - - -def run(posix, move_ex): - kernel = FakeKernel(posix, move_ex) - return kernel, lambda: replace_store_windows("a", "b", kernel) - - -class WindowsReplaceTable(unittest.TestCase): - def test_falls_back_to_write_through_move_where_posix_rename_is_unsupported(self): - self.assertEqual(MOVEFILE_REPLACE_WRITE_THROUGH, 0x9) - for unsupported in (1, 50, 87): - kernel, go = run([unsupported], [0]) - go() - self.assertEqual(kernel.log, ["posix", "moveex:9"]) - - def test_once_on_move_the_rest_of_the_call_stays_there(self): - kernel, go = run([87], [32, 0]) - go() - self.assertEqual(kernel.log, ["posix", "moveex:9", "moveex:9"]) - - def test_transient_codes_retry_the_same_primitive(self): - for transient in (5, 32, 33): - kernel, go = run([transient, transient, 0], []) - go() - self.assertEqual(kernel.log, ["posix", "posix", "posix"]) - self.assertEqual(kernel.sleeps, [0.02, 0.02]) - - def test_a_working_posix_rename_is_the_whole_replace(self): - kernel, go = run([0], []) - go() - self.assertEqual(kernel.log, ["posix"]) - self.assertEqual(kernel.sleeps, []) - - def test_unsupported_after_the_fallback_is_an_error_not_another_fallback(self): - kernel, go = run([87], [32, 87]) - with self.assertRaisesRegex(OSError, "Win32 error 87") as caught: - go() - self.assertEqual(caught.exception.winerror, 87) - self.assertEqual(kernel.log, ["posix", "moveex:9", "moveex:9"]) - for unsupported in (1, 50, 87): - kernel, go = run([87], [unsupported]) - with self.assertRaisesRegex(OSError, f"Win32 error {unsupported}"): - go() - self.assertEqual(kernel.log, ["posix", "moveex:9"]) - - def test_gives_up_after_25_attempts_naming_the_last_code(self): - kernel, go = run([32] * 30, []) - with self.assertRaisesRegex(OSError, "after 25 attempts: last Win32 error 32\\."): - go() - self.assertEqual(len(kernel.log), 25) - - def test_other_codes_fail_at_once(self): - kernel, go = run([2], []) - with self.assertRaisesRegex(OSError, "Win32 error 2"): - go() - self.assertEqual(len(kernel.log), 1) - kernel, go = run([87], [3]) - with self.assertRaisesRegex(OSError, "Win32 error 3"): - go() - - -class StoreWritesRouteThroughTheReplace(unittest.TestCase): - def test_both_stores_replace_through_replace_store(self): - seen: list[tuple[Path, Path]] = [] - real = stores._replace_store - - def spy(staged, destination): - seen.append((Path(staged), Path(destination))) - real(staged, destination) - - with tempfile.TemporaryDirectory() as directory, mock.patch.object(stores, "_replace_store", spy): - envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z") - binary = Path(directory) / "s.cc" - SingleFileMessagePackBackingStore(binary).push_all([envelope]) - lines = Path(directory) / "s.jsonl" - stores.JsonLinesBackingStore(lines).push_all([envelope]) - self.assertEqual([destination for _, destination in seen], [binary, lines]) - self.assertTrue(binary.read_bytes() and lines.read_text()) - - def test_a_failed_jsonl_replace_leaves_no_staged_file_and_the_old_store(self): - envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z") - with tempfile.TemporaryDirectory() as directory: - lines = Path(directory) / "s.jsonl" - store = stores.JsonLinesBackingStore(lines) - store.push_all([envelope]) - before = lines.read_bytes() - with mock.patch.object(stores, "_replace_store", side_effect=OSError("replace refused")): - with self.assertRaises(OSError): - store.push_all([CultCacheEnvelope(key="k2", type="t", payload=b"\x02", stored_at="2026-11-06T00:00:00Z")]) - self.assertEqual(lines.read_bytes(), before) - self.assertEqual([p.name for p in Path(directory).iterdir() if ".tmp" in p.name], []) - - @unittest.skipUnless(sys.platform == "win32", "needs a real Windows") - def test_real_kernel_replaces_through_both_primitives(self): - kernel = stores.WindowsReplaceKernel() - with tempfile.TemporaryDirectory() as directory: - for name, replace in (("posix", kernel.rename_posix), ("moveex", lambda s, d: kernel.move_file_ex(s, d, MOVEFILE_REPLACE_WRITE_THROUGH))): - staged, store = Path(directory, name + ".tmp"), Path(directory, name) - store.write_bytes(b"old") - staged.write_bytes(b"new") - self.assertEqual(replace(str(staged), str(store)), 0) - self.assertEqual(store.read_bytes(), b"new") - self.assertEqual(kernel.rename_posix(str(Path(directory, "missing")), str(Path(directory, "x"))), 2) - - -if __name__ == "__main__": - unittest.main() diff --git a/packages/cultcache-rs/.gitignore b/packages/cultcache-rs/.gitignore index ea8c4bf7..96ef6c0b 100644 --- a/packages/cultcache-rs/.gitignore +++ b/packages/cultcache-rs/.gitignore @@ -1 +1,2 @@ /target +Cargo.lock diff --git a/packages/cultcache-rs/CHANGELOG.md b/packages/cultcache-rs/CHANGELOG.md index 327f42aa..7f3caa73 100644 --- a/packages/cultcache-rs/CHANGELOG.md +++ b/packages/cultcache-rs/CHANGELOG.md @@ -6,16 +6,10 @@ only in git history. ## Unreleased -### Changed - -- The single-file store re-checks for a link once the store lock is held (every write and the try-exchange), so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, as C# does, instead of being replaced. Pinned by a test that puts the link while a second holder owns the lock. -- `CultCache::is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store. - ### Breaking Semver: this is a breaking change in the 0.x line (0.3.x to 0.4.0). -- A write at a store path that is a symbolic link, live or dangling, is refused with the new public `StoreLinkedError` (`error.downcast_ref::()`) before the lock and the replace, and writes nothing: `push`, `push_all`, `delete`, the exchanges, and a soft `push_all`, which used to skip a link and now refuses. Before, a write at a live link replaced the link with a regular file and a dangling link failed with `io::ErrorKind::NotFound`. The redb stores (`RedbMessagePackBackingStore`, `OwnedRedbMessagePackBackingStore`) are exempt: they are ruled migrated and deleted (`paged-store-redb-migrate-and-delete`) and still open a live link's target. Configure the link's target as the store path. Reads through a live link work. - A record whose payload carries unread slots is read-only. A payload with a slot past its type's declared slots, at any depth (record, nested record, list element, union arm, unit arm), still decodes, and every write or removal of its key is now refused before any store is touched: `put`, `put_envelope`, `put_raw_envelope`, `put_prepared_batch` (the whole batch) and `delete`, so `update` too. The refusal is `SchemaConflictError` with the new `unread_slots`; its message names the type and both slot counts and never a slot value. Holders of such a record used to overwrite it and drop the extra slots silently. The payload nesting refusal still runs first. A held payload this build cannot read at all (an arm it lacks from a newer writer, or a corrupt record) is read-only too: every write or removal is refused with an error naming the type and key (a newer writer's record is read with that build; a corrupt one is restored from a backup or removed at the store itself, as no cache call removes it), and `is_writable` answers false. `load_soa` is a write of every held key it omits or changes, so it refuses to drop or replace a read-only record. `is_writable` also answers false for a key a store carries as a foreign record and for a type other than the one holding the key, as the writes do. Each write over a held key decodes the held payload once to ask (about 40% of a `put` of a 200,000-element record, measured in release); keys nothing holds pay nothing. - cultmesh-rs `pull_rudp_catalog_snapshot` applies a snapshot as one cache commit (`put_prepared_batch`), as C# does: a snapshot document for a key the node holds read-only, or any other refusal, applies nothing. It used to apply document by document, so a refusal ended the pull with the earlier documents kept. A snapshot whose types route to different backing stores is now refused whole, as a batch cannot span stores. C# `CultCache.Commit` refuses a cross-store commit the same way (`The batch spans A and B; a commit lands in one home store`), so the two agree; a test pins it. - `SchemaConflictError` gains the public field `unread_slots: Option`; code that builds it with a struct literal must set it (`None`). diff --git a/packages/cultcache-rs/Cargo.lock b/packages/cultcache-rs/Cargo.lock deleted file mode 100644 index 40d9e1e8..00000000 --- a/packages/cultcache-rs/Cargo.lock +++ /dev/null @@ -1,730 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "android_system_properties" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" -dependencies = [ - "libc", -] - -[[package]] -name = "anyhow" -version = "1.0.104" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" - -[[package]] -name = "ar_archive_writer" -version = "0.5.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "523e5e187fff36fcb8d1d19ccf0ca4ca43ec3d158316057f66dc6f72716bc5ec" -dependencies = [ - "object", -] - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "bitflags" -version = "2.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" - -[[package]] -name = "bumpalo" -version = "3.20.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" - -[[package]] -name = "cc" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ff2a8126fff0f8c839080d619675a585e61720bb12d2baceae45769fd66f155" -dependencies = [ - "find-msvc-tools", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" - -[[package]] -name = "chrono" -version = "0.4.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" -dependencies = [ - "iana-time-zone", - "js-sys", - "num-traits", - "serde", - "wasm-bindgen", - "windows-link", -] - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "cultcache-rs" -version = "0.3.1" -dependencies = [ - "anyhow", - "chrono", - "cultcache-rs-derive", - "fs2", - "pretty_assertions", - "redb", - "rmp-serde", - "serde", - "serde_bytes", - "serde_json", - "serde_stacker", - "stacker", - "tempfile", - "uuid", - "windows-sys 0.59.0", -] - -[[package]] -name = "cultcache-rs-derive" -version = "0.2.0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "diff" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8" - -[[package]] -name = "errno" -version = "0.3.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "fastrand" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" - -[[package]] -name = "find-msvc-tools" -version = "0.1.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48" - -[[package]] -name = "fs2" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" -dependencies = [ - "libc", - "winapi", -] - -[[package]] -name = "futures-core" -version = "0.3.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" - -[[package]] -name = "futures-task" -version = "0.3.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" - -[[package]] -name = "futures-util" -version = "0.3.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" -dependencies = [ - "futures-core", - "futures-task", - "pin-project-lite", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi", -] - -[[package]] -name = "iana-time-zone" -version = "0.1.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "log", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "itoa" -version = "1.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4" - -[[package]] -name = "js-sys" -version = "0.3.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a" -dependencies = [ - "cfg-if", - "futures-util", - "wasm-bindgen", -] - -[[package]] -name = "libc" -version = "0.2.191" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a" - -[[package]] -name = "linux-raw-sys" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" - -[[package]] -name = "log" -version = "0.4.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d15f86d12da44cd8d4ec3c73984537eb7f4cadcb9776a575a0a8a05ca11005c" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "object" -version = "0.39.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" -dependencies = [ - "memchr", -] - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "pretty_assertions" -version = "1.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d" -dependencies = [ - "diff", - "yansi", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "psm" -version = "0.1.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622" -dependencies = [ - "ar_archive_writer", - "cc", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "redb" -version = "4.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b317ef681e18f6091efbfab532eaf924a135c492d9d0299d0fbfae07e92a39b" -dependencies = [ - "libc", -] - -[[package]] -name = "rmp" -version = "0.8.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c" -dependencies = [ - "num-traits", -] - -[[package]] -name = "rmp-serde" -version = "1.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155" -dependencies = [ - "rmp", - "serde", -] - -[[package]] -name = "rustix" -version = "1.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa" -dependencies = [ - "bitflags", - "errno", - "libc", - "linux-raw-sys", - "windows-sys 0.61.2", -] - -[[package]] -name = "rustversion" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_bytes" -version = "0.11.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" -dependencies = [ - "serde", - "serde_core", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.10", -] - -[[package]] -name = "serde_json" -version = "1.0.154" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_stacker" -version = "0.1.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4936375d50c4be7eff22293a9344f8e46f323ed2b3c243e52f89138d9bb0f4a" -dependencies = [ - "serde", - "serde_core", - "stacker", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "stacker" -version = "0.1.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967" -dependencies = [ - "cc", - "cfg-if", - "libc", - "psm", - "windows-sys 0.61.2", -] - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ae9ccbea3a7b0b09443036490c8f34a8b1b760fd345e5c0dae0aac718faea79" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "tempfile" -version = "3.27.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" -dependencies = [ - "fastrand", - "getrandom", - "once_cell", - "rustix", - "windows-sys 0.61.2", -] - -[[package]] -name = "unicode-ident" -version = "1.0.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" - -[[package]] -name = "uuid" -version = "1.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bed99139f7dac9b5cfe5f17be4d66dc7079a34dee1d74bcc77217119240e267c" -dependencies = [ - "getrandom", - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c" -dependencies = [ - "cfg-if", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn 3.0.10", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-link", - "windows-result", - "windows-strings", -] - -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-sys" -version = "0.59.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" -dependencies = [ - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" - -[[package]] -name = "windows_i686_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" - -[[package]] -name = "windows_i686_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" - -[[package]] -name = "yansi" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/packages/cultcache-rs/Cargo.toml b/packages/cultcache-rs/Cargo.toml index db8a46ac..7954370c 100644 --- a/packages/cultcache-rs/Cargo.toml +++ b/packages/cultcache-rs/Cargo.toml @@ -31,4 +31,4 @@ pretty_assertions = "1.4" tempfile = "3" [target.'cfg(windows)'.dependencies] -windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_Security", "Win32_System_WindowsProgramming"] } +windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem"] } diff --git a/packages/cultcache-rs/README.md b/packages/cultcache-rs/README.md index 06c530b1..2d217083 100644 --- a/packages/cultcache-rs/README.md +++ b/packages/cultcache-rs/README.md @@ -384,10 +384,9 @@ an empty store. Any other failure to reach it, such as a symlink loop, a file where a parent directory should be, or a missing permission, is an I/O error. A store file that exists and holds zero bytes is `Undecodable`: no CultCache writer leaves one. A symbolic link whose target is gone is not a missing store: -reading it is an I/O error. A write at a store path that is a link, live or dangling, is -refused with `StoreLinkedError` (`error.downcast_ref::()`) before the -lock and the replace, so no write swaps a link for a regular file; configure the link's -target as the store path. Reads through a live link work. A refusal names no +reading it is an I/O error, so a read-then-write (`push`, `push_all`, `delete`, the exchanges) +fails before writing. Writes do not yet follow links: a write through a live link +replaces the link with a regular file and leaves its target alone. A refusal names no value read from the file; it names a record only by its key and schema id, and a header only when it has the shape `cultcache.store.v`. diff --git a/packages/cultcache-rs/src/lib.rs b/packages/cultcache-rs/src/lib.rs index 8fccdf97..b3d6a441 100644 --- a/packages/cultcache-rs/src/lib.rs +++ b/packages/cultcache-rs/src/lib.rs @@ -480,39 +480,6 @@ struct PersistedSchemaCatalogEntry( Vec, ); -/// One entry of a store file's schema catalog, read-only: the scalar fields every runtime publishes for a schema id. The members and -/// the canonical schema JSON the file also carries are not exposed. `schema_version` is the version string the writer published -/// (C# `CultSchemaCatalogEntry.SchemaVersion`). -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct SchemaCatalogEntry { - pub schema_id: String, - pub schema_name: String, - pub schema_version: String, - pub content_hash: String, - pub compatible_schema_ids: Vec, -} - -impl From<&PersistedSchemaCatalogEntry> for SchemaCatalogEntry { - fn from(entry: &PersistedSchemaCatalogEntry) -> Self { - Self { - schema_id: entry.0.clone(), - schema_name: entry.1.clone(), - schema_version: entry.2.clone(), - content_hash: entry.3.clone(), - compatible_schema_ids: entry.5.clone(), - } - } -} - -/// The catalog entry a record's schema id names, as the decoder resolves it: the first entry that has the id as its own, else the -/// first entry that lists it as compatible; `None` when the catalog publishes neither. -pub fn resolve_schema_entry<'a>(catalog: &'a [SchemaCatalogEntry], schema_id: &str) -> Option<&'a SchemaCatalogEntry> { - catalog - .iter() - .find(|entry| entry.schema_id == schema_id) - .or_else(|| catalog.iter().find(|entry| entry.compatible_schema_ids.iter().any(|id| id == schema_id))) -} - #[derive(Clone, Debug, serde::Serialize, serde::Deserialize)] struct PersistedSchemaCatalogMember( u32, @@ -1209,35 +1176,6 @@ impl std::fmt::Display for CultCacheStoreUnreadable { impl std::error::Error for CultCacheStoreUnreadable {} -/// A write was asked of a store path that is a link (a file or directory symbolic link, or a Windows -/// junction): a replace there would swap the link for a regular file and detach the store from its -/// target. Nothing was locked or written. Configure the link's target as the store path. Reads through -/// a live link are unaffected. Find it with `error.downcast_ref::()`. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct StoreLinkedError { - pub path: PathBuf, -} - -impl std::fmt::Display for StoreLinkedError { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!( - formatter, - "CultCache store {} is a link, and a write at it would replace the link; configure the link's target as the store path", - self.path.display() - ) - } -} - -impl std::error::Error for StoreLinkedError {} - -/// Refuses a write at a link. A path that cannot be inspected is left to the open that follows. -fn refuse_linked(path: &Path) -> Result<()> { - match fs::symlink_metadata(path) { - Ok(metadata) if metadata.file_type().is_symlink() => Err(StoreLinkedError { path: path.to_path_buf() }.into()), - _ => Ok(()), - } -} - /// A write to a single-file store that failed, and what it left on disk. It sits in the /// error chain of every such failure, above the cause, so a consumer tells the outcomes /// apart with `error.downcast_ref::()`. @@ -1730,16 +1668,6 @@ impl SingleFileMessagePackBackingStore { self.read_all_unlocked() } - /// The schema catalog the store file publishes, read under the same shared lock [`CacheBackingStore::pull_all`] takes. An absent - /// file and a legacy envelope array have no catalog and give an empty one. Resolve a record's `schema_id` against it with - /// [`resolve_schema_entry`]. - pub fn pull_schema_catalog(&self) -> Result> { - if is_absent(&self.path)? && is_absent(&self.lock_path())? { - return Ok(Vec::new()); - } - self.with_shared_lock(|| Ok(self.read_store_unlocked()?.catalog.iter().map(SchemaCatalogEntry::from).collect())) - } - fn write_all_unlocked(&self, entries: &[CultCacheEnvelope]) -> Result<()> { let failed = |kind| CultCacheStoreWriteFailed { path: self.path.clone(), @@ -1791,11 +1719,10 @@ impl SingleFileMessagePackBackingStore { } fn with_exclusive_lock(&self, action: impl FnOnce() -> Result) -> Result { - refuse_linked(&self.path)?; let lock = self.open_lock_file()?; fs2::FileExt::lock_exclusive(&lock) .with_context(|| format!("failed to lock {}", self.lock_path().display()))?; - self.run_locked(lock, action) + release_lock(lock, action()) } /// Replaces the named entries only if each still matches its expectation. @@ -1903,12 +1830,7 @@ impl SingleFileMessagePackBackingStore { } fn try_with_exclusive_lock(&self, action: impl FnOnce() -> Result) -> Result> { - refuse_linked(&self.path)?; - #[cfg(test)] - run_after_pre_lock_look(); - let Some(lock) = self.try_open_lock_file()? else { - return Ok(None); - }; + let lock = self.open_lock_file()?; match fs2::FileExt::try_lock_exclusive(&lock) { Ok(()) => {} Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => return Ok(None), @@ -1917,40 +1839,22 @@ impl SingleFileMessagePackBackingStore { .with_context(|| format!("failed to lock {}", self.lock_path().display())); } } - self.run_locked(lock, action).map(Some) + release_lock(lock, action()).map(Some) } - /// The one tail of both exclusive entries: with the lock held, look for a link again (one may have been put at the - /// path while the lock was awaited; the look before the lock saw the path earlier), run the action, release. - fn run_locked(&self, lock: File, action: impl FnOnce() -> Result) -> Result { - release_lock(lock, refuse_linked(&self.path).and_then(|()| action())) - } - - /// Opens the lock file, waiting through a Windows sharing violation the way the lock itself waits. fn open_lock_file(&self) -> Result { - let lock_path = self.prepare_lock_dir()?; - retry_while_busy( - || open_lock_once(&lock_path), - is_sharing_violation, - || std::thread::sleep(std::time::Duration::from_millis(5)), - ) - .with_context(|| format!("failed to open {}", lock_path.display())) - } - - /// The single-attempt open for the try entry: a sharing violation is `None`, the same answer as a held lock. - fn try_open_lock_file(&self) -> Result> { - let lock_path = self.prepare_lock_dir()?; - open_unless_busy(|| open_lock_once(&lock_path), is_sharing_violation) - .with_context(|| format!("failed to open {}", lock_path.display())) - } - - fn prepare_lock_dir(&self) -> Result { let lock_path = self.lock_path(); if let Some(parent) = lock_path.parent() { fs::create_dir_all(parent) .with_context(|| format!("failed to create {}", parent.display()))?; } - Ok(lock_path) + OpenOptions::new() + .create(true) + .read(true) + .truncate(false) + .write(true) + .open(&lock_path) + .with_context(|| format!("failed to open {}", lock_path.display())) } fn lock_path(&self) -> PathBuf { @@ -3374,8 +3278,7 @@ impl CultCache { /// Whether a write or removal of `(type_id, key)` would be admitted: false when the held payload carries slots past /// its registered type's declared slots, at any depth, or cannot be read by this build at all (the record is /// read-only), when the cache holds no image of its stores (see [`CultCacheLoadRefused`]), when a store carries a foreign record at the key, or when another registered type holds it. True for a - /// key nothing holds. An unregistered type is an error. It does not look at store paths: a link at a store path is not - /// predicted, and the write refuses it with [`StoreLinkedError`]. Advisory: the write entries ask the same checks themselves. + /// key nothing holds. An unregistered type is an error. Advisory: the write entries ask the same checks themselves. pub fn is_writable(&self, type_id: &str, key: &str) -> Result { if self.require_loaded().is_err() { return Ok(false); @@ -4462,324 +4365,61 @@ fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> { #[cfg(windows)] fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> { + use std::os::windows::ffi::OsStrExt; + use windows_sys::Win32::Storage::FileSystem::{ + MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW, + }; + injected_write_fault(WriteStep::Rename).context("failed to atomically replace the store")?; - replace_with(&WindowsKernel, staged, destination) -} -/// The two Windows replace primitives, each answering 0 or a Win32 error code, and the wait -/// between attempts. `replace_with` owns every decision about them. -#[cfg_attr(not(windows), allow(dead_code))] -trait ReplaceKernel { - /// CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with - /// REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle. - fn rename_posix(&self, staged: &Path, destination: &Path) -> u32; - /// MoveFileExW(staged, destination, flags). - fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32; - fn sleep(&self, milliseconds: u64); -} - -/// MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH. -#[cfg_attr(not(windows), allow(dead_code))] -const MOVEFILE_REPLACE_WRITE_THROUGH: u32 = 0x9; - -/// Replaces `destination` with `staged` on Windows: a POSIX-semantics rename first, so a reader -/// that opened the old store keeps it and the destination is rarely seen absent (a stalled rename -/// can still leave it absent for 5-47 ms on NTFS: `variants:ruling:posix-rename-absent-window-document`); -/// MoveFileExW with -/// write-through where POSIX semantics are unsupported (FAT, exFAT, before Windows 10 1607), -/// for the rest of the call; and a bounded retry on access, sharing and lock violations. -/// A concurrent reader (another process, an indexer, an antivirus scan) makes the replace fail -/// transiently, and a daemon that treats that as fatal dies on the first overlap (Muninn serve -/// did, 2026-09-11). Ruling `variants:ruling:windows-replace-posix-first`: cultcache-ts carries the -/// same table on main; cultcache-cs carries it only once its pending PR lands. -#[cfg_attr(not(windows), allow(dead_code))] -fn replace_with(kernel: &dyn ReplaceKernel, staged: &Path, destination: &Path) -> Result<()> { - const ERROR_INVALID_FUNCTION: u32 = 1; - const ERROR_ACCESS_DENIED: u32 = 5; - const ERROR_SHARING_VIOLATION: u32 = 32; - const ERROR_LOCK_VIOLATION: u32 = 33; - const ERROR_NOT_SUPPORTED: u32 = 50; - const ERROR_INVALID_PARAMETER: u32 = 87; + let staged_wide = staged + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + let destination_wide = destination + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + // A concurrent reader of the destination (another process opening the + // store, an indexer, an antivirus scan) makes the replace fail with a + // sharing or access error for as long as it holds the handle. That is + // transient by construction; a daemon that treats it as fatal dies on + // the first overlap (Muninn serve did, 2026-09-11). Retry briefly. + const ERROR_ACCESS_DENIED: i32 = 5; + const ERROR_SHARING_VIOLATION: i32 = 32; + const ERROR_LOCK_VIOLATION: i32 = 33; const ATTEMPTS: u32 = 25; - const BACKOFF_MS: u64 = 20; - let mut posix = true; - let mut last_code = 0; - for attempt in 1..=ATTEMPTS { - let mut code = if posix { - kernel.rename_posix(staged, destination) - } else { - kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) + const BACKOFF: std::time::Duration = std::time::Duration::from_millis(20); + let mut attempt = 0; + loop { + let success = unsafe { + MoveFileExW( + staged_wide.as_ptr(), + destination_wide.as_ptr(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, + ) }; - if posix && matches!(code, ERROR_INVALID_FUNCTION | ERROR_NOT_SUPPORTED | ERROR_INVALID_PARAMETER) { - posix = false; - code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH); - } - if code == 0 { + if success != 0 { return Ok(()); } - if !matches!(code, ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION) { - return Err(anyhow::anyhow!( - "failed to atomically replace {} with {}: Win32 error {code} after {attempt} attempt(s)", - destination.display(), - staged.display() - )); - } - last_code = code; - kernel.sleep(BACKOFF_MS); - } - Err(anyhow::anyhow!( - "failed to atomically replace {} with {} after {ATTEMPTS} attempt(s): last Win32 error {last_code}", - destination.display(), - staged.display() - )) -} - -#[cfg(windows)] -struct WindowsKernel; - -#[cfg(windows)] -fn wide(path: &Path) -> Vec { - use std::os::windows::ffi::OsStrExt; - path.as_os_str().encode_wide().chain(std::iter::once(0)).collect() -} - -#[cfg(windows)] -impl ReplaceKernel for WindowsKernel { - fn rename_posix(&self, staged: &Path, destination: &Path) -> u32 { - use windows_sys::Win32::Foundation::{CloseHandle, GetLastError, INVALID_HANDLE_VALUE}; - use windows_sys::Win32::Storage::FileSystem::{ - CreateFileW, FileRenameInfoEx, SetFileInformationByHandle, DELETE, FILE_FLAG_BACKUP_SEMANTICS, - FILE_RENAME_INFO, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, OPEN_EXISTING, SYNCHRONIZE, - }; - use windows_sys::Win32::System::WindowsProgramming::{ - FILE_RENAME_FLAG_POSIX_SEMANTICS, FILE_RENAME_FLAG_REPLACE_IF_EXISTS, - }; - - let staged_wide = wide(staged); - let destination_wide = wide(destination); - // FILE_RENAME_INFO ends in a variable-length name: allocate it as u64s for alignment. - let name_bytes = (destination_wide.len() - 1) * 2; - let header = std::mem::size_of::(); - let total = header + name_bytes + 2; - let mut buffer = vec![0u64; total.div_ceil(8)]; - unsafe { - let info = buffer.as_mut_ptr() as *mut FILE_RENAME_INFO; - (*info).Anonymous.Flags = FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS; - (*info).RootDirectory = std::ptr::null_mut(); - (*info).FileNameLength = name_bytes as u32; - let name = std::ptr::addr_of_mut!((*info).FileName) as *mut u16; - std::ptr::copy_nonoverlapping(destination_wide.as_ptr(), name, destination_wide.len()); - - let handle = CreateFileW( - staged_wide.as_ptr(), - DELETE | SYNCHRONIZE, - FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, - std::ptr::null(), - OPEN_EXISTING, - FILE_FLAG_BACKUP_SEMANTICS, - std::ptr::null_mut(), - ); - if handle == INVALID_HANDLE_VALUE { - return GetLastError(); - } - let ok = SetFileInformationByHandle(handle, FileRenameInfoEx, info as *const _, total as u32); - let code = if ok == 0 { GetLastError() } else { 0 }; - CloseHandle(handle); - code - } - } - - fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32 { - use windows_sys::Win32::Foundation::GetLastError; - use windows_sys::Win32::Storage::FileSystem::MoveFileExW; - let ok = unsafe { MoveFileExW(wide(staged).as_ptr(), wide(destination).as_ptr(), flags) }; - if ok != 0 { - 0 - } else { - unsafe { GetLastError() } - } - } - - fn sleep(&self, milliseconds: u64) { - std::thread::sleep(std::time::Duration::from_millis(milliseconds)); - } -} - -fn open_lock_once(lock_path: &Path) -> std::io::Result { - OpenOptions::new() - .create(true) - .read(true) - .truncate(false) - .write(true) - .open(lock_path) -} - -/// Windows ERROR_SHARING_VIOLATION (raw os error 32): another handle holds the lock file open without sharing. On Unix -/// errno 32 is EPIPE and is never a busy file. -#[cfg(windows)] -fn is_sharing_violation(error: &std::io::Error) -> bool { - error.raw_os_error() == Some(32) -} - -#[cfg(not(windows))] -fn is_sharing_violation(_error: &std::io::Error) -> bool { - false -} - -/// Repeats `open` while it fails busy, sleeping between tries and without a bound, as the lock waits; any other error surfaces. -fn retry_while_busy( - mut open: impl FnMut() -> std::io::Result, - busy: impl Fn(&std::io::Error) -> bool, - mut sleep: impl FnMut(), -) -> std::io::Result { - loop { - match open() { - Err(error) if busy(&error) => sleep(), - other => return other, - } - } -} - -/// One attempt: `None` when the open fails busy, any other error surfaces. -fn open_unless_busy( - open: impl FnOnce() -> std::io::Result, - busy: impl Fn(&std::io::Error) -> bool, -) -> std::io::Result> { - match open() { - Ok(file) => Ok(Some(file)), - Err(error) if busy(&error) => Ok(None), - Err(error) => Err(error), - } -} - -#[cfg(test)] -mod replace_with_tests { - use super::{replace_with, ReplaceKernel}; - use std::cell::RefCell; - use std::collections::VecDeque; - use std::path::Path; - - #[derive(Debug, PartialEq)] - enum Call { - Posix, - MoveFile(u32), - Sleep(u64), - } - - /// Answers each primitive from a script; an empty script answers 0. - #[derive(Default)] - struct FakeKernel { - posix: RefCell>, - move_file: RefCell>, - calls: RefCell>, - } - - impl FakeKernel { - fn scripted(posix: &[u32], move_file: &[u32]) -> Self { - Self { - posix: RefCell::new(posix.iter().copied().collect()), - move_file: RefCell::new(move_file.iter().copied().collect()), - ..Self::default() - } - } - fn calls(&self) -> Vec { - self.calls.replace(Vec::new()) - } - } - - impl ReplaceKernel for FakeKernel { - fn rename_posix(&self, _: &Path, _: &Path) -> u32 { - self.calls.borrow_mut().push(Call::Posix); - self.posix.borrow_mut().pop_front().unwrap_or(0) - } - fn move_file_ex(&self, _: &Path, _: &Path, flags: u32) -> u32 { - self.calls.borrow_mut().push(Call::MoveFile(flags)); - self.move_file.borrow_mut().pop_front().unwrap_or(0) - } - fn sleep(&self, milliseconds: u64) { - self.calls.borrow_mut().push(Call::Sleep(milliseconds)); - } - } - - fn run(kernel: &FakeKernel) -> anyhow::Result<()> { - replace_with(kernel, Path::new("staged"), Path::new("store.cc")) - } - - #[test] - fn a_posix_rename_that_lands_is_the_whole_replace() { - let kernel = FakeKernel::scripted(&[0], &[]); - run(&kernel).unwrap(); - assert_eq!(kernel.calls(), vec![Call::Posix]); - } - - #[test] - fn unsupported_posix_semantics_fall_back_to_write_through_and_stay_there() { - for code in [1, 50, 87] { - // The fallback itself is held once, so the retry must use MoveFileExW, not POSIX again. - let kernel = FakeKernel::scripted(&[code], &[32, 0]); - run(&kernel).unwrap_or_else(|e| panic!("code {code}: {e}")); - assert_eq!( - kernel.calls(), - vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)], - "code {code}" - ); - } - } - - #[test] - fn a_held_destination_is_retried_with_posix_until_it_lets_go() { - for code in [5, 32, 33] { - let kernel = FakeKernel::scripted(&[code, code, 0], &[]); - run(&kernel).unwrap(); - assert_eq!( - kernel.calls(), - vec![Call::Posix, Call::Sleep(20), Call::Posix, Call::Sleep(20), Call::Posix], - "code {code}" - ); - } - } - - #[test] - fn a_destination_held_forever_fails_after_twenty_five_attempts_naming_the_code() { - for code in [5, 32, 33] { - let kernel = FakeKernel::scripted(&[code; 40], &[]); - let error = run(&kernel).unwrap_err().to_string(); - let calls = kernel.calls(); - assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 25); - assert_eq!(calls.iter().filter(|c| **c == Call::Sleep(20)).count(), 25); - assert!(error.contains("25 attempt(s)") && error.contains(&format!("error {code}")), "{error}"); - } - } - - #[test] - fn a_held_destination_after_fallback_exhausts_through_move_file_ex() { - let kernel = FakeKernel::scripted(&[87], &[32; 40]); - assert!(run(&kernel).is_err()); - let calls = kernel.calls(); - assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 1); - assert_eq!(calls.iter().filter(|c| matches!(c, Call::MoveFile(0x9))).count(), 25); - } - - #[test] - fn any_other_code_fails_at_once_without_a_retry() { - for (posix, move_file) in [(&[2u32][..], &[][..]), (&[87][..], &[3][..])] { - let kernel = FakeKernel::scripted(posix, move_file); - let error = run(&kernel).unwrap_err().to_string(); - assert!(!kernel.calls().contains(&Call::Sleep(20)), "{error}"); - assert!(error.contains("1 attempt(s)"), "{error}"); - } - } - - #[test] - fn move_file_ex_answering_unsupported_is_an_error_not_a_cue_to_fall_back_again() { - let kernel = FakeKernel::scripted(&[87], &[32, 87]); - let error = run(&kernel).unwrap_err().to_string(); - assert_eq!( - kernel.calls(), - vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)], - "{error}" + let error = std::io::Error::last_os_error(); + attempt += 1; + let transient = matches!( + error.raw_os_error(), + Some(ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION) ); - assert!(error.contains("error 87") && error.contains("2 attempt(s)"), "{error}"); + if !transient || attempt >= ATTEMPTS { + return Err(error).with_context(|| { + format!( + "failed to atomically replace {} with {} after {attempt} attempt(s)", + destination.display(), + staged.display() + ) + }); + } + std::thread::sleep(BACKOFF); } } @@ -4788,35 +4428,6 @@ mod windows_replace_tests { use super::replace_file_atomically; use std::os::windows::fs::OpenOptionsExt; - const FILE_SHARE_ALL: u32 = 0x1 | 0x2 | 0x4; // READ | WRITE | DELETE - - /// A reader that shares delete holds the old store open. A POSIX-semantics rename - /// replaces the name while the reader keeps the old content; MoveFileExW alone fails - /// there with error 5 (Soul's Starfire probe, build 26200), so this test passes only - /// if the POSIX path is live. Compile-checked only: it has never run on Windows. - #[test] - fn a_replace_succeeds_over_a_delete_sharing_reader_which_keeps_the_old_content() { - use std::io::Read; - let dir = std::env::temp_dir().join(format!("cultcache-posix-{}", std::process::id())); - std::fs::create_dir_all(&dir).unwrap(); - let destination = dir.join("store.cc"); - let staged = dir.join("store.cc.tmp"); - std::fs::write(&destination, b"old").unwrap(); - std::fs::write(&staged, b"new").unwrap(); - let mut reader = std::fs::OpenOptions::new() - .read(true) - .share_mode(FILE_SHARE_ALL) - .open(&destination) - .unwrap(); - replace_file_atomically(&staged, &destination).expect("POSIX rename over a sharing reader"); - let mut seen = Vec::new(); - reader.read_to_end(&mut seen).unwrap(); - assert_eq!(seen, b"old", "the reader keeps the old content"); - assert_eq!(std::fs::read(&destination).unwrap(), b"new"); - drop(reader); - let _ = std::fs::remove_dir_all(&dir); - } - /// A reader holding the destination open with no sharing makes the /// replace fail for as long as it holds on; it must succeed once the /// reader lets go, not report the first refusal as the outcome. @@ -4879,20 +4490,6 @@ fn injected_write_fault(step: WriteStep) -> std::io::Result<()> { Ok(()) } -// The one gap a test cannot hold open from outside: a try-lock never waits, so what happens between its pre-lock look and -// the lock is only reachable by running the test's closure there. -#[cfg(test)] -thread_local! { - static AFTER_PRE_LOCK_LOOK: std::cell::RefCell>> = const { std::cell::RefCell::new(None) }; -} - -#[cfg(test)] -fn run_after_pre_lock_look() { - if let Some(hook) = AFTER_PRE_LOCK_LOOK.with(|hook| hook.borrow_mut().take()) { - hook(); - } -} - #[cfg(not(test))] #[inline(always)] fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> { @@ -4901,8 +4498,7 @@ fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> { #[cfg(windows)] fn sync_parent_directory(_path: &Path) -> Result<()> { - // Publication durability is the replace's: the MoveFileExW fallback writes through; the POSIX-semantics - // rename has no write-through flag and its durability is unmeasured (follow-up windows-replace-run-and-durability). + // MoveFileExW with MOVEFILE_WRITE_THROUGH owns publication durability. Ok(()) } @@ -5778,6 +5374,12 @@ mod tests { Ok(()) } + fn io_not_found(error: &anyhow::Error) -> bool { + error + .chain() + .any(|cause| cause.downcast_ref::().is_some_and(|io| io.kind() == std::io::ErrorKind::NotFound)) + } + fn test_envelope(r#type: &str, key: &str, payload: &[u8]) -> CultCacheEnvelope { CultCacheEnvelope { key: key.to_string(), @@ -7712,14 +7314,7 @@ mod tests { let temp = tempfile::tempdir()?; let path = temp.path().join("store.cc"); std::os::unix::fs::symlink(&path, &path)?; - let [pull, snapshot, exchange] = every_read(&SingleFileMessagePackBackingStore::new(&path)); - for error in [pull.unwrap_err(), snapshot.unwrap_err()] { - let io = io_error_in(&error).expect("the failure to reach the store is an io::Error"); - assert_ne!(io.kind(), std::io::ErrorKind::NotFound, "{error:#}"); - assert_eq!(unreadable_kind(&error), None, "{error:#}"); - } - // The exchange is a write: a link at the path is refused before it reads. - assert!(exchange.unwrap_err().downcast_ref::().is_some()); + assert_io_failure_on_every_read(&SingleFileMessagePackBackingStore::new(&path)); assert!(fs::symlink_metadata(&path)?.file_type().is_symlink()); Ok(()) } @@ -8158,13 +7753,12 @@ mod tests { let path = temp.path().join("store.cc"); std::os::unix::fs::symlink(&target, &path)?; let store = SingleFileMessagePackBackingStore::new(&path); - let [pull, snapshot, exchange] = every_read(&store); - for error in [pull.unwrap_err(), snapshot.unwrap_err()] { + for result in every_read(&store) { + let error = result.expect_err("a dangling link must not read as an empty store"); assert!(io_error_in(&error).is_some(), "{error:#}"); assert_eq!(unreadable_kind(&error), None, "{error:#}"); } - assert!(exchange.unwrap_err().downcast_ref::().is_some()); - // Every writer refuses the link before its lock, so none can replace it. + // Every merging writer refuses on its own read, so none can replace the link. let mut writer = SingleFileMessagePackBackingStore::new(&path); let delta = snapshot_envelope("delta", b"4"); let writes = [ @@ -8176,7 +7770,7 @@ mod tests { ]; for (index, result) in writes.into_iter().enumerate() { let error = result.expect_err("a merging write through a dangling link must refuse"); - assert!(error.downcast_ref::().is_some(), "write {index}: {error:#}"); + assert!(io_error_in(&error).is_some(), "write {index}: {error:#}"); assert_eq!(write_failure_kind(&error), None, "write {index}: {error:#}"); } assert!(fs::symlink_metadata(&path)?.file_type().is_symlink()); @@ -8302,21 +7896,11 @@ mod tests { with_write_fault(WriteStep::Unlock, || store.push(&pushed))?; assert!(with_write_fault(WriteStep::Unlock, || store.pull_all())?.contains(&pushed)); - // The lock was released: another writer takes it. The shared lock of the faulted `pull_all` just above was left to the - // close of its descriptor (see `release_lock`), which frees it only once no other descriptor shares its open file - // description, and a test that spawns a child (a fork-exec duplicates every descriptor of the process until the exec) - // can hold such a duplicate on another test thread for a moment. A faulted exclusive lock cannot contend the try-lock: - // the blocking `pull_all` in its argument waits it out. So a contended try-lock is retried for up to two seconds; a - // lock that was never released stays contended and still fails. - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2); - let outcome = loop { - let outcome = store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?; - if !matches!(outcome, TryCompareExchangeSnapshotOutcome::LockContended) || std::time::Instant::now() >= deadline { - break outcome; - } - std::thread::sleep(std::time::Duration::from_millis(5)); - }; - assert!(matches!(outcome, TryCompareExchangeSnapshotOutcome::Exchanged), "{outcome:?}"); + // The lock was released: another writer takes it at once. + assert!(matches!( + store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?, + TryCompareExchangeSnapshotOutcome::Exchanged + )); // The keyed store releases its lock the same way. let mut keyed = RedbMessagePackBackingStore::new(temp.path().join("keyed.redb"))?; @@ -8325,265 +7909,27 @@ mod tests { Ok(()) } - // A link put at the store path while a writer waits on the lock is refused once the lock is taken, as C# does, and the - // lock is free afterwards. + // What a write does through a symbolic link today. R3 decides whether a write + // resolves the link or refuses it; until then these pin the current behaviour so the + // change is deliberate. Reads refuse a dangling link (see the test above). #[cfg(unix)] #[test] - fn a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it() -> Result<()> { + fn r3_decides_resolve_or_refuse_a_push_through_a_live_link_replaces_the_link() -> Result<()> { let temp = tempfile::tempdir()?; let volume = temp.path().join("volume"); fs::create_dir(&volume)?; let target = volume.join("store.cc"); SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; let before = fs::read(&target)?; - let dangling = volume.join("missing.cc"); - let path = temp.path().join("late.cc"); - let writers: [(&str, fn(&mut SingleFileMessagePackBackingStore) -> Result<()>); 4] = [ - ("push", |s| s.push(&snapshot_envelope("b", b"2"))), - ("push_all", |s| s.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default())), - ("delete", |s| s.delete(&snapshot_envelope("a", b"1"))), - ("compare_exchange_snapshot", |s| s.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).map(|_| ())), - ]; - for (name, write, link_to) in writers.into_iter().flat_map(|(n, w)| [(n, w, &target), (n, w, &dangling)]) { - let _ = fs::remove_file(&path); - let store = SingleFileMessagePackBackingStore::new(&path); - let holder = store.open_lock_file()?; - fs2::FileExt::lock_exclusive(&holder)?; - let mut waiting = SingleFileMessagePackBackingStore::new(&path); - let handle = std::thread::spawn(move || write(&mut waiting)); - std::thread::sleep(std::time::Duration::from_millis(300)); - std::os::unix::fs::symlink(link_to, &path)?; - fs2::FileExt::unlock(&holder)?; - let error = handle.join().expect("writer thread").expect_err("the late link must be refused"); - assert!(error.downcast_ref::().is_some(), "{name}: {error:#}"); - assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "{name}: the link was replaced"); - assert_eq!(fs::read(&target)?, before, "{name}: the target changed"); - assert!(!dangling.exists(), "{name}: the dangling target was created"); - let probe = store.open_lock_file()?; - fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("{name}: lock still held: {e}"))?; - } - Ok(()) - } - - fn busy_error() -> std::io::Error { - std::io::Error::from_raw_os_error(32) - } - - fn is_busy(error: &std::io::Error) -> bool { - error.raw_os_error() == Some(32) - } - - #[test] - fn the_lock_open_waits_through_busy_failures_then_opens() { - let (mut tries, mut sleeps) = (0, 0); - let opened = retry_while_busy( - || { - tries += 1; - if tries <= 4 { Err(busy_error()) } else { Ok(tries) } - }, - is_busy, - || sleeps += 1, - ); - assert_eq!(opened.unwrap(), 5); - assert_eq!(sleeps, 4); - } - - #[test] - fn the_lock_open_surfaces_a_non_busy_error_at_once() { - let (mut tries, mut sleeps) = (0, 0); - let error = retry_while_busy::<()>( - || { - tries += 1; - Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied)) - }, - is_busy, - || sleeps += 1, - ) - .expect_err("a non-busy error must surface"); - assert_eq!(error.kind(), std::io::ErrorKind::PermissionDenied); - assert_eq!((tries, sleeps), (1, 0)); - } - - #[test] - fn the_try_open_maps_a_sharing_violation_to_none_without_retrying() { - let mut tries = 0; - let opened = open_unless_busy::<()>( - || { - tries += 1; - Err(busy_error()) - }, - is_busy, - ); - assert!(opened.unwrap().is_none()); - assert_eq!(tries, 1); - let other = open_unless_busy::<()>(|| Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied)), is_busy); - assert!(other.is_err()); - } - - // The try entry must answer at once while another handle holds the lock. The attempt runs on a thread and its - // answer is awaited with a timeout, so an entry that waits fails this test by timeout instead of hanging the suite. - #[cfg(unix)] - #[test] - fn the_try_entry_returns_at_once_while_the_lock_is_held() -> Result<()> { - let temp = tempfile::tempdir()?; - let path = temp.path().join("held.cc"); - let store = SingleFileMessagePackBackingStore::new(&path); - let holder = store.open_lock_file()?; - fs2::FileExt::lock_exclusive(&holder)?; - let (sender, receiver) = std::sync::mpsc::channel(); - let attempt = SingleFileMessagePackBackingStore::new(&path); - std::thread::spawn(move || { - let _ = sender.send(attempt.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")])); - }); - let outcome = receiver - .recv_timeout(std::time::Duration::from_secs(3)) - .map_err(|_| anyhow!("the try entry waited on a held lock instead of returning"))?; - assert!(matches!(outcome?, TryCompareExchangeSnapshotOutcome::LockContended)); - assert!(!path.exists(), "the contended attempt wrote"); - let again = store.open_lock_file()?; - assert!(fs2::FileExt::try_lock_exclusive(&again).is_err(), "the held lock was disturbed"); - fs2::FileExt::unlock(&holder)?; - Ok(()) - } - - // Off Windows errno 32 is EPIPE, never a busy file: the predicate is false and the wait surfaces it on the first try. - #[cfg(not(windows))] - #[test] - fn error_32_is_not_busy_off_windows_and_surfaces_at_once() { - assert!(!is_sharing_violation(&busy_error())); - let (mut tries, mut sleeps) = (0, 0); - let error = retry_while_busy::<()>( - || { - tries += 1; - Err(busy_error()) - }, - is_sharing_violation, - || sleeps += 1, - ) - .expect_err("error 32 must surface off Windows"); - assert_eq!(error.raw_os_error(), Some(32)); - assert_eq!((tries, sleeps), (1, 0)); - } - - // The try entry has no wait, so the late window is the gap between its look and its lock: a link put there is refused - // once the lock is held, nothing is written through or over it, and the lock is released. - #[cfg(unix)] - #[test] - fn a_link_put_between_the_try_entrys_look_and_its_lock_is_refused_and_the_lock_is_released() -> Result<()> { - let temp = tempfile::tempdir()?; - let target = temp.path().join("target.cc"); - SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; - let before = fs::read(&target)?; - let dangling = temp.path().join("missing.cc"); - let path = temp.path().join("late.cc"); - for link_to in [target.clone(), dangling.clone()] { - let store = SingleFileMessagePackBackingStore::new(&path); - let (link, at) = (link_to.clone(), path.clone()); - AFTER_PRE_LOCK_LOOK.with(|hook| { - *hook.borrow_mut() = Some(Box::new(move || std::os::unix::fs::symlink(&link, &at).expect("put the link"))); - }); - let error = store - .try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]) - .expect_err("the late link must be refused"); - assert!(error.downcast_ref::().is_some(), "{error:#}"); - assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "the link was replaced"); - assert_eq!(fs::read(&target)?, before, "the target changed"); - assert!(!dangling.exists(), "the dangling target was created"); - let probe = store.open_lock_file()?; - fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("lock still held: {e}"))?; - drop(probe); - fs::remove_file(&path)?; - } - Ok(()) - } - - // A write at a store path that is a link is refused: a replace there would swap the link for a regular file and - // detach the store from its target. Reads through a live link keep working. - #[cfg(unix)] - #[test] - fn a_write_at_a_live_link_is_refused_and_nothing_changes_while_a_read_through_it_works() -> Result<()> { - let temp = tempfile::tempdir()?; - let volume = temp.path().join("volume"); - fs::create_dir(&volume)?; - let target = volume.join("store.cc"); - SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; - let before = fs::read(&target)?; - let volume_before = dir_names(&volume)?; - let link = temp.path().join("link.cc"); + let link = temp.path().join("store.cc"); std::os::unix::fs::symlink(&target, &link)?; - let mut store = SingleFileMessagePackBackingStore::new(&link); - let errors = [ - store.push(&snapshot_envelope("b", b"2")).unwrap_err(), - store.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default()).unwrap_err(), - store.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions { soft: true }).unwrap_err(), - store.delete(&snapshot_envelope("a", b"1")).unwrap_err(), - store.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).unwrap_err(), - store.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).err().expect("refused"), - ]; - for error in &errors { - let linked = error.downcast_ref::().unwrap_or_else(|| panic!("{error:#}")); - assert_eq!(linked.path, link); - assert!(error.to_string().contains(&link.display().to_string()), "{error:#}"); - } - assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was"); - assert_eq!(fs::read(&target)?, before); - assert_eq!(dir_names(&volume)?, volume_before, "no lock or staging file beside the target"); - assert_eq!(dir_names(temp.path())?, ["link.cc", "volume"], "no lock or staging file beside the link"); - assert_eq!(store.pull_all()?, vec![snapshot_envelope("a", b"1")]); + SingleFileMessagePackBackingStore::new(&link).push(&snapshot_envelope("b", b"2"))?; + assert!(fs::symlink_metadata(&link)?.file_type().is_file(), "the link was replaced by a regular file"); + assert_eq!(fs::read(&target)?, before, "the store the link pointed to is unchanged"); Ok(()) } - // The advisory query is the held record's and the cache's judges; a link at the store path is the write's to refuse. - #[cfg(unix)] - #[test] - fn is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused() -> Result<()> { - let temp = tempfile::tempdir()?; - let target = temp.path().join("store.cc"); - store_with(&target, |cache| cache.put("held", &settings_one()).map(drop))?; - let before = fs::read(&target)?; - let link = temp.path().join("link.cc"); - std::os::unix::fs::symlink(&target, &link)?; - - let mut cache = CultCache::new(); - cache.register_registry(TestEntries)?; - cache.add_backing_store(SingleFileMessagePackBackingStore::new(&link), [Settings::TYPE])?; - cache.pull_all_backing_stores()?; - let names_before = dir_names(temp.path())?; - - assert!(cache.is_writable(Settings::TYPE, "held")?, "the query does not look at the store path"); - assert!(cache.is_writable(Settings::TYPE, "fresh")?, "the query does not look at the store path"); - let error = cache.put("fresh", &settings_one()).unwrap_err(); - let linked = error.downcast_ref::().unwrap_or_else(|| panic!("{error:#}")); - assert_eq!(linked.path, link); - assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was"); - assert_eq!(fs::read(&target)?, before); - assert_eq!(dir_names(temp.path())?, names_before, "the refused write left no lock or staging file"); - Ok(()) - } - - #[cfg(unix)] - #[test] - fn a_write_at_a_live_directory_link_is_refused_as_a_link() -> Result<()> { - let temp = tempfile::tempdir()?; - let real = temp.path().join("real"); - fs::create_dir(&real)?; - let link = temp.path().join("dirlink"); - std::os::unix::fs::symlink(&real, &link)?; - let error = SingleFileMessagePackBackingStore::new(&link).push(&snapshot_envelope("a", b"1")).unwrap_err(); - assert!(error.downcast_ref::().is_some(), "{error:#}"); - assert!(fs::symlink_metadata(&link)?.file_type().is_symlink()); - assert_eq!(fs::read_dir(&real)?.count(), 0); - Ok(()) - } - - #[cfg(unix)] - fn dir_names(dir: &Path) -> Result> { - let mut names = fs::read_dir(dir)?.map(|entry| Ok(entry?.file_name().to_string_lossy().into_owned())).collect::>>()?; - names.sort(); - Ok(names) - } - // The two cases the vectors cannot hold: nothing at the path is written, and a dangling link is refused and left as it is. #[test] fn a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link() -> Result<()> { @@ -8605,7 +7951,7 @@ mod tests { let error = SingleFileMessagePackBackingStore::new(&link) .push_all(&[snapshot_envelope("a", b"1")], PushAllOptions { soft: true }) .unwrap_err(); - assert!(error.downcast_ref::().is_some(), "{error:#}"); + assert!(io_not_found(&error), "{error:#}"); assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link was replaced"); assert_eq!(fs::read_dir(&volume)?.count(), 0, "nothing reached the link's volume"); } diff --git a/packages/cultcache-rs/tests/schema_catalog.rs b/packages/cultcache-rs/tests/schema_catalog.rs deleted file mode 100644 index 36ec3bed..00000000 --- a/packages/cultcache-rs/tests/schema_catalog.rs +++ /dev/null @@ -1,165 +0,0 @@ -//! A consumer of a single-file store reads the schema version a record was published under from the store's catalog, through the -//! public surface only: `pull_all`, `pull_schema_catalog` and `resolve_schema_entry` -//! (`cultlib-gaps:follow_up:rs-snapshot-schema-version`). - -use std::path::PathBuf; - -use cultcache_rs::{ - CacheBackingStore, CultCache, DatabaseEntry, SchemaCatalogEntry, SingleFileMessagePackBackingStore, resolve_schema_entry, -}; - -#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] -#[cultcache(type = "tests.sc_one", schema = "tests.sc_one")] -struct One { - #[cultcache(key = 0)] - text: String, -} - -#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] -#[cultcache(type = "tests.sc_two", schema = "tests.sc_two")] -struct Two { - #[cultcache(key = 0)] - text: String, -} - -fn vectors(folder: &str) -> PathBuf { - PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../tests/vectors").join(folder) -} - -/// The version a store publishes for each record, in key order: `(key, schema name, schema version)`. -fn versions(store: &SingleFileMessagePackBackingStore) -> Vec<(String, String, String)> { - let catalog = store.pull_schema_catalog().unwrap(); - let mut rows: Vec<_> = store - .pull_all() - .unwrap() - .iter() - .map(|envelope| { - let id = envelope.schema_id.as_deref().unwrap(); - let entry = resolve_schema_entry(&catalog, id).unwrap_or_else(|| panic!("no entry for {id}")); - (envelope.key.clone(), entry.schema_name.clone(), entry.schema_version.clone()) - }) - .collect(); - rows.sort(); - rows -} - -fn copy(from: PathBuf) -> (tempfile::TempDir, SingleFileMessagePackBackingStore) { - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("store.cc"); - std::fs::copy(from, &path).unwrap(); - (dir, SingleFileMessagePackBackingStore::new(&path)) -} - -#[test] -fn a_consumer_reads_the_version_of_each_type_a_rust_store_holds() { - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("store.cc"); - let mut cache = CultCache::new(); - cache.register_entry_type::().unwrap(); - cache.register_entry_type::().unwrap(); - cache.add_backing_store(SingleFileMessagePackBackingStore::new(&path), ["tests.sc_one", "tests.sc_two"]).unwrap(); - cache.pull_all_backing_stores().unwrap(); - cache.put("a", &One { text: "1".into() }).unwrap(); - cache.put("b", &Two { text: "2".into() }).unwrap(); - drop(cache); - - let consumer = SingleFileMessagePackBackingStore::new(&path); - assert_eq!( - versions(&consumer), - vec![ - ("a".to_string(), "tests.sc_one".to_string(), "tests.sc_one.v1".to_string()), - ("b".to_string(), "tests.sc_two".to_string(), "tests.sc_two.v1".to_string()), - ] - ); -} - -#[test] -fn an_id_only_an_entry_lists_as_compatible_resolves_to_that_entry() { - let (_dir, store) = copy(vectors("document-variants-c2a/readability/compatible-id-only-v3.bin")); - let catalog = store.pull_schema_catalog().unwrap(); - assert_eq!(catalog.len(), 1); - assert_eq!(catalog[0].schema_id, "vectors.item.next"); - assert_eq!(catalog[0].compatible_schema_ids, vec!["vectors.item".to_string()]); - let entry = resolve_schema_entry(&catalog, "vectors.item").unwrap(); - assert_eq!(entry.schema_id, "vectors.item.next"); - assert_eq!(entry.schema_version, "vectors.item.v1"); - assert_eq!(resolve_schema_entry(&catalog, "vectors.nowhere"), None); -} - -#[test] -fn an_id_an_entry_owns_resolves_to_its_owner_not_to_an_entry_listing_it() { - for vector in ["own-id-over-compatible-v3.bin", "compatible-before-owner-v3.bin"] { - let (_dir, store) = copy(vectors("document-variants-c2a/readability").join(vector)); - let catalog = store.pull_schema_catalog().unwrap(); - assert_eq!(catalog.len(), 2, "{vector}"); - let entry = resolve_schema_entry(&catalog, "vectors.item").unwrap(); - assert_eq!((entry.schema_id.as_str(), entry.schema_version.as_str()), ("vectors.item", "vectors.item.v1"), "{vector}"); - } -} - -#[test] -fn a_legacy_array_and_an_absent_file_publish_no_catalog() { - let (_dir, store) = copy(vectors("document-variants-c2a/readability/legacy-envelopes.bin")); - assert!(!store.pull_all().unwrap().is_empty()); - assert_eq!(store.pull_schema_catalog().unwrap(), Vec::::new()); - - let dir = tempfile::tempdir().unwrap(); - let absent = SingleFileMessagePackBackingStore::new(dir.path().join("none.cc")); - assert_eq!(absent.pull_schema_catalog().unwrap(), Vec::::new()); - assert!(!dir.path().join("none.cc").exists() && !dir.path().join("none.cc.lock").exists(), "reading an absent store creates nothing"); -} - -// `tests/vectors/key-identity/foreign-store.cc` is written by the C# reference: two types, each published with its version. -#[test] -fn a_store_the_csharp_reference_wrote_gives_its_versions() { - let (_dir, store) = copy(vectors("key-identity/foreign-store.cc")); - assert_eq!( - versions(&store), - vec![ - ("j".to_string(), "tests.ki_alpha".to_string(), "tests.ki_alpha.v1".to_string()), - ("k".to_string(), "tests.ki_gamma".to_string(), "tests.ki_gamma.v1".to_string()), - ] - ); - let catalog = store.pull_schema_catalog().unwrap(); - assert_eq!(catalog[0].content_hash, "sha256:02f0c8b7ea07c11705c1365fc29136ada08c8076209cacfb8d5526229461ce7a"); -} - -type Entry = (&'static str, &'static str, &'static str, &'static str, &'static str, Vec<&'static str>, Vec<()>); - -// A hand-built v3 store whose versions are not the `.v1` the writers above publish, so a version read from anywhere but the -// version slot cannot pass. `shared` is listed by two entries (the first lister wins); `id.c` is owned by one entry and listed by -// an earlier one (the owner wins). -#[test] -fn distinct_versions_come_through_with_the_decoders_resolution() { - let catalog: Vec = vec![ - ("id.a", "N.a", "9.9.9", "hash-a", "{}", vec!["id.a", "shared"], vec![]), - ("id.b", "N.b", "7.0", "hash-b", "{}", vec!["id.b", "shared", "id.c"], vec![]), - ("id.c", "N.c", "3", "hash-c", "{}", vec!["id.c"], vec![]), - ]; - let record = |key: &str, id: &str| (key.to_string(), id.to_string(), "2026-11-13T00:00:00Z".to_string(), b"x".to_vec()); - let records = vec![record("r1", "shared"), record("r2", "id.c"), record("r3", "id.b"), record("r4", "id.a")]; - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("store.cc"); - std::fs::write(&path, rmp_serde::to_vec(&("cultcache.store.v3", catalog, records)).unwrap()).unwrap(); - - let store = SingleFileMessagePackBackingStore::new(&path); - let listed = store.pull_schema_catalog().unwrap(); - assert_eq!(listed.iter().map(|e| e.schema_version.as_str()).collect::>(), vec!["9.9.9", "7.0", "3"]); - assert_eq!(listed[1].content_hash, "hash-b"); - assert_eq!(listed[1].compatible_schema_ids, vec!["id.b", "shared", "id.c"]); - let got: Vec<(String, String, String)> = { - let mut rows: Vec<_> = store - .pull_all() - .unwrap() - .iter() - .map(|e| { - let entry = resolve_schema_entry(&listed, e.schema_id.as_deref().unwrap()).unwrap(); - (e.key.clone(), entry.schema_name.clone(), entry.schema_version.clone()) - }) - .collect(); - rows.sort(); - rows - }; - let row = |k: &str, n: &str, v: &str| (k.to_string(), n.to_string(), v.to_string()); - assert_eq!(got, vec![row("r1", "N.a", "9.9.9"), row("r2", "N.c", "3"), row("r3", "N.b", "7.0"), row("r4", "N.a", "9.9.9")]); -} diff --git a/packages/cultcache-ts/CHANGELOG.md b/packages/cultcache-ts/CHANGELOG.md index b5ab90bd..fea670db 100644 --- a/packages/cultcache-ts/CHANGELOG.md +++ b/packages/cultcache-ts/CHANGELOG.md @@ -5,21 +5,9 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following ## [Unreleased] -### Added - -- `CacheBackingStore` gains the optional `read(key)` (the record at the key with its content version, from one read) and `commitIf(entry, expectedVersion)` (write only if the version still holds, under the store's write lock; returns the storedAt written). The single-file store implements both. The content version is the lowercase hex SHA-256 of the record's persisted encoding `[key, schemaId, storedAt, payload]`, equal to the C# reference's on `tests/vectors/content-version`. A store without them still works for everything except `update`. -- `CultCacheUpdateResult` and `CultCacheUpdateOptions` (`maxAttempts`, default 5), exported from the package entry and the browser entry. - -### Changed - -- A write that replaces a record at its key with different content (another schema id or other payload bytes) is stored strictly later than it: a supplied `storedAt` not later than the stored one is written one millisecond after it, so a content version that includes `storedAt` never repeats across writes. A rewrite of identical content (same schema id and payload bytes) keeps the stored stamp and so its version, which is what keeps a periodic flush of an unchanged cache from changing every record's version; C# re-mints every replacing write, so this is a departure from it, and the writer's type and catalog entry are written either way. `push`, `pushAll` and `commitIf` alike; the single-file store's `push` returns the stamp it wrote (`CacheBackingStore.push` may resolve `{ storedAt }`), and `put`, `putGlobal` and `putEnvelope` install that stamp in the cache, so the cache and the file agree after a put over a record stamped at or after now (a push result that is not a parseable stamp is ignored). A stamp that would pass 9999-12-31T23:59:59.999Z is refused with `RangeError` and nothing is written. -- `withStoreWriteLock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. Pinned by a test that puts the link while another holder owns the lock. -- `CultCache.isWritable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store. - ### Breaking -- `update` and `updateGlobal` are the conditional golden path and no longer the blind pair. `update(definition, key, apply, options?)` reads the record and its content version from the store, passes `apply` a fresh copy (never the held instance), commits only if the version still holds and runs `apply` again on a lost race, up to `maxAttempts`. `apply` may be async, may return `undefined` to write nothing, and must be free of side effects. It resolves to `{ outcome: "updated" | "declined" | "lostRace", document }` instead of the written value, so a caller reads `result.document`; a lost race that outlasts the attempts is `lostRace`, not a throw. `apply` returning `undefined` or `null` declines. `maxAttempts` is an integer of at least 1; `undefined` is the default 5 and `null` is refused with `RangeError`. A conditional write over a global loaded from a legacy key is refused wherever it enters (`updateGlobal`, or `update` at `CultCache.GLOBAL_KEY`) and leaves the legacy record untouched; `apply` may have run once before the write is refused. A store attached to a store-less cache while `apply` awaits makes that attempt a lost race. It needs a store with `read` and `commitIf` (the single-file store has them); a cache with no store compares the held entry. `put` and `putGlobal` are the blind writes and are unchanged. -- A write at a single-file store path that is a symbolic link, live or dangling, is refused with the new `StoreLinkedError` (exported from the package entry; `error.path` names the path) before the lock and the replace, and writes nothing: `push`, `pushAll`, `delete` and a soft flush alike. Before, a write at a live link replaced the link with a regular file, a dangling link failed with `ENOENT` (or, for `pushAll`, was replaced), and a soft flush at a link skipped. A caller catching `ENOENT` around a write at a dangling link now catches `StoreLinkedError`; configure the link's target as the store path. Reads through a live link work. An `lstat` failure other than `ENOENT` (`ENOTDIR`, `EACCES`, `ELOOP`) is rethrown as it is. +- `SingleFileMessagePackBackingStore.pushAll` through a dangling symbolic link at the store path is refused with `ENOENT`, as `push` and `pullAll` are, and writes nothing; it used to replace the link with a new store. The reader owns the verdict on what is at the path. - `SingleFileMessagePackBackingStore.pushAll` writes the given entries onto the store as it is under the lock, as `push` does: a record under the same key is replaced and no other is removed. It used to replace the whole file with the entries, so `CultMesh.flush()` from a cache that held less than the store (a node created with `pullOnStart: false`, or one whose image was stale) wiped the records it never read. A caller that relied on `pushAll` to reset a store removes the file first. A soft flush is unchanged. `CultCache.storedSnapshot()` is the image as the stores hold it (a global loaded from a legacy key under that key) and is what `CultMesh.flush` persists, so a flush no longer leaves such a global under two keys; `snapshot()` is unchanged. - A single-file store marked `cultcache.store.v3` (it holds an element id, written by the C# runtime) is read, and a rewrite keeps the mark and the ids; this runtime does not decide when a document carries ids. A runtime from before this change refuses a v3 store by name, so rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - `indexes` are unique, as C# `[CultIndex]`: a write or a load that leaves two records of a type holding one non-blank @@ -29,7 +17,7 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following the many-row selection aliases (never indexed for lookup, same catalog `indexAlias`). The per-key migration rule is in the README; classify every key before the CultLib your tools load is updated. The refusal verdict is the contract, its text is informational. - The single-file reader (v1 and the legacy envelope array) refuses a store that holds two records under one key as `StoreUnreadableError`; a store that opened before may now be refused. Repair it by removing the extra record with the runtime that wrote it. -- A `put`, `push` or `pushAll` that would land a second type under a held key (against the file, or within one batch) is refused with `SchemaConflictError`, and nothing is written. The written type may replace the record when its catalog entry carries the stored schema id as its own or lists it compatible (a type renamed under a stable id restamps the record, as C# does, when its payload changes; an identical payload keeps the stamp); otherwise retyping a key is a delete, then a put. The C# allowance for an undeclared older id of the same schema name that reads whole is not ported: a same-named type keeps passing, including one that drops persisted slots, which C# refuses. +- A `put`, `push` or `pushAll` that would land a second type under a held key (against the file, or within one batch) is refused with `SchemaConflictError`, and nothing is written. The written type may replace the record when its catalog entry carries the stored schema id as its own or lists it compatible (a type renamed under a stable id restamps the record, as C# does); otherwise retyping a key is a delete, then a put. The C# allowance for an undeclared older id of the same schema name that reads whole is not ported: a same-named type keeps passing, including one that drops persisted slots, which C# refuses. - Same-type writes replace by key; in one `pushAll` batch the last entry for a key wins. - A store whose canonical-schema catalog slot is not a string is refused as `StoreUnreadableError`. - The cache holds one record per key (variants:ruling:cache-refuses-retype). A `put` or `putEnvelope` that lands a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store (a memory-only cache used to hold both types at the key). A type that owns the held record's schema id (as for the store) replaces it, unless the held record is in a store other than the written type's home: that rename is refused with `SchemaConflictError` (remove the record first). Two stores holding one key, registered or foreign, refuse the load and admit nothing. A store's `delete` removes the record whose stored schema id the removal's type owns, whatever the arrived record's `type` string. diff --git a/packages/cultcache-ts/README.md b/packages/cultcache-ts/README.md index 2aa78c9c..625d28dd 100644 --- a/packages/cultcache-ts/README.md +++ b/packages/cultcache-ts/README.md @@ -168,8 +168,10 @@ const settings = cache.getRequiredGlobal(settingsDocument); - `getByIndex(...)` - `getGlobal(...)` - `getRequiredGlobal(...)` -- `update(...)` and `updateGlobal(...)`, the conditional golden path: apply a function to a fresh copy of the record as the store holds it, commit only if it is unchanged, re-run on a lost race (`updated`, `declined` or `lostRace`) -- `put(...)` and `putGlobal(...)`, the blind opt-out: last writer wins +- `put(...)` +- `putGlobal(...)` +- `update(...)` +- `updateGlobal(...)` - `delete(...)` - `deleteGlobal(...)` - `snapshot()` (the logical image) and `storedSnapshot()` (the image as the stores hold it, for a whole flush) diff --git a/packages/cultcache-ts/src/browser.ts b/packages/cultcache-ts/src/browser.ts index 05a5ecd6..98995acb 100644 --- a/packages/cultcache-ts/src/browser.ts +++ b/packages/cultcache-ts/src/browser.ts @@ -35,8 +35,6 @@ export type { InferCultCacheSchemaValue, CultCacheStoreRegistration, CultCacheUnreadSlots, - CultCacheUpdateOptions, - CultCacheUpdateResult, PushAllOptions, } from "./types"; export type { diff --git a/packages/cultcache-ts/src/cult-cache.ts b/packages/cultcache-ts/src/cult-cache.ts index c8d1cc62..a88bd66d 100644 --- a/packages/cultcache-ts/src/cult-cache.ts +++ b/packages/cultcache-ts/src/cult-cache.ts @@ -22,8 +22,6 @@ import type { CultCacheSchemaCatalogEntry, CultCacheStoreRegistration, CultCacheUnreadSlots, - CultCacheUpdateOptions, - CultCacheUpdateResult, } from "./types"; type Accessor = (value: unknown) => string | undefined; @@ -50,10 +48,6 @@ type EntryLookups = { readonly indexes: ReadonlyMap; }; -// What a conditional write requires of the record at its key: the content version its store reported (null: no record), or for a -// cache with no store the entry it held. Every write installs a new entry object, so the identity catches an A-B-A. -type WriteCondition = { readonly version: string | null } | { readonly held: HydratedEntry | undefined }; - type HydratedEntry = CultCacheEnvelope & { catalogEntry: NonNullable; value: unknown; @@ -308,88 +302,85 @@ export class CultCache { // HydratedState, running every check and every user accessor, and replaces the cache's state // in one assignment only once the build finishes. A refused load admits nothing. pullAllBackingStores(): Promise { - return this.#serial(() => this.#pullNow()); - } + return this.#serial(async () => { + const next = emptyHydratedState(); + const persistedGlobalKeys = new Map(); - // The pull itself, for a caller already inside a serial operation. - async #pullNow(): Promise { - const next = emptyHydratedState(); - const persistedGlobalKeys = new Map(); + for (const registration of this.#stores) { + const entries = await registration.store.pullAll(); - for (const registration of this.#stores) { - const entries = await registration.store.pullAll(); + for (const entry of entries) { + const registered = this.#resolveDefinitionForEnvelope(entry); + if (!registered) { + // A store may hold what this cache has no type for; it is carried, not served, and not a reason to refuse the load. + next.foreign.push({ + key: entry.key, + schemaId: schemaIdForEnvelope(entry), + schemaName: entry.catalogEntry?.schemaName ?? entry.type, + storedAt: entry.storedAt, + }); + continue; + } - for (const entry of entries) { - const registered = this.#resolveDefinitionForEnvelope(entry); - if (!registered) { - // A store may hold what this cache has no type for; it is carried, not served, and not a reason to refuse the load. - next.foreign.push({ - key: entry.key, - schemaId: schemaIdForEnvelope(entry), - schemaName: entry.catalogEntry?.schemaName ?? entry.type, - storedAt: entry.storedAt, - }); - continue; - } - - const type = registered.definition.type; - const home = this.#resolveRegistration(type); - if (home?.store !== registration.store) { - throw new Error( - `CultCache "${type}" record "${entry.key}" was loaded from ${this.#describe(registration)}, ` + - `but its home is ${this.#describe(home)}.`, - ); - } - - const payload = this.#cloneBytes(entry.payload); - const { value, unreadSlots } = registered.formatter.decode(payload); - let admitted: CultCacheEnvelope = { ...entry, type, payload }; - if (registered.global) { - const persistedKey = persistedGlobalKeys.get(type); - if (persistedKey !== undefined) { + const type = registered.definition.type; + const home = this.#resolveRegistration(type); + if (home?.store !== registration.store) { throw new Error( - `CultCache global document type "${type}" has multiple persisted entries: "${persistedKey}" and "${entry.key}".`, + `CultCache "${type}" record "${entry.key}" was loaded from ${this.#describe(registration)}, ` + + `but its home is ${this.#describe(home)}.`, ); } - persistedGlobalKeys.set(type, entry.key); - if (entry.key !== CultCache.GLOBAL_KEY) { - next.legacyGlobals.set(type, admitted); - admitted = { ...admitted, key: CultCache.GLOBAL_KEY }; + + const payload = this.#cloneBytes(entry.payload); + const { value, unreadSlots } = registered.formatter.decode(payload); + let admitted: CultCacheEnvelope = { ...entry, type, payload }; + if (registered.global) { + const persistedKey = persistedGlobalKeys.get(type); + if (persistedKey !== undefined) { + throw new Error( + `CultCache global document type "${type}" has multiple persisted entries: "${persistedKey}" and "${entry.key}".`, + ); + } + persistedGlobalKeys.set(type, entry.key); + if (entry.key !== CultCache.GLOBAL_KEY) { + next.legacyGlobals.set(type, admitted); + admitted = { ...admitted, key: CultCache.GLOBAL_KEY }; + } + } + const hydrated = this.#admitEntry(next, registered, admitted, value, unreadSlots); + const other = this.#heldUnderOtherType(next, type, hydrated.key); + if (other !== undefined) { + throw new Error(`Record ${hydrated.key} is held by two stores, as type "${other.type}" and as type "${type}".`); + } + this.#installEntry(next, registered, hydrated); + } + } + + const foreignByKey = new Map(); + for (const record of next.foreign) { + const first = foreignByKey.get(record.key); + if (first !== undefined) { + throw new Error(`Record ${record.key} is held by two stores, as schema "${first}" and as schema "${record.schemaName}".`); + } + foreignByKey.set(record.key, record.schemaName); + const other = [...this.#definitions.keys()].map((type) => next.entries.get(this.#entryId(type, record.key))).find((held) => held !== undefined); + if (other !== undefined) { + throw new Error(`Record ${record.key} is held by two stores, as type "${other.type}" and as schema "${record.schemaName}".`); + } + } + + // Judged whole, once: the refusal names the least violation of the state this load would leave, whatever the + // order of the stores and of the entries in them. + refuseLeastIndexSharing(function* () { + for (const entry of next.entries.values()) { + for (const [alias, value] of entry.lookups.indexes) { + yield { type: entry.type, schemaName: entry.catalogEntry.schemaName, alias, value, key: entry.key }; } } - const hydrated = this.#admitEntry(next, registered, admitted, value, unreadSlots); - const other = this.#heldUnderOtherType(next, type, hydrated.key); - if (other !== undefined) { - throw new Error(`Record ${hydrated.key} is held by two stores, as type "${other.type}" and as type "${type}".`); - } - this.#installEntry(next, registered, hydrated); - } - } + }()); - const foreignByKey = new Map(); - for (const record of next.foreign) { - const first = foreignByKey.get(record.key); - if (first !== undefined) { - throw new Error(`Record ${record.key} is held by two stores, as schema "${first}" and as schema "${record.schemaName}".`); - } - foreignByKey.set(record.key, record.schemaName); - const other = [...this.#definitions.keys()].map((type) => next.entries.get(this.#entryId(type, record.key))).find((held) => held !== undefined); - if (other !== undefined) { - throw new Error(`Record ${record.key} is held by two stores, as type "${other.type}" and as schema "${record.schemaName}".`); - } - } - - // Judged whole, once: the refusal names the least violation of the state this load would leave, whatever the - // order of the stores and of the entries in them. - refuseLeastIndexSharing(function* () { - for (const entry of next.entries.values()) { - for (const [alias, value] of entry.lookups.indexes) { - yield { type: entry.type, schemaName: entry.catalogEntry.schemaName, alias, value, key: entry.key }; - } - } - }()); - - this.#hydrated = next; + this.#hydrated = next; + }); } get( @@ -522,16 +513,12 @@ export class CultCache { return key ? this.get(definition, key) : undefined; } - /** - * The blind write: the record at the key becomes `value`, whatever it held, and the last writer wins. `update` is the - * conditional path, which changes a record from what the store holds now and re-runs on a lost race. - */ put( definition: TDefinition, key: string, value: CultCacheDocumentValue, ): Promise> { - return this.#serial(async () => (await this.#putNow(definition, key, value)).value); + return this.#serial(() => this.#putNow(definition, key, value)); } putEnvelope( @@ -573,68 +560,31 @@ export class CultCache { definition: TDefinition, value: CultCacheDocumentValue, ): Promise> { - return this.#serial(async () => { + return this.#serial(() => { this.#requireGlobal(definition); - return (await this.#putNow(definition, CultCache.GLOBAL_KEY, value)).value; + return this.#putNow(definition, CultCache.GLOBAL_KEY, value); }); } - /** - * Changes the record at the key from what the store holds now. `apply` may run up to `maxAttempts` times (default 5): each - * attempt gets a fresh copy of the record as the store holds it (undefined when absent), which it may change and return, and - * the result is written only if the record is still that version; a lost race runs `apply` again on the newer record. It must - * not write this cache or have side effects. Returning undefined writes nothing (`declined`). A lost race that outlasts - * `maxAttempts` returns `lostRace` and does not throw; a refusal, or a throw from `apply`, rejects and is not retried. The - * write is durable when the promise resolves. `document` is the cache's record after the outcome. `put` is the blind write. - */ - async update( + update( definition: TDefinition, key: string, - apply: ( + updater: ( current: CultCacheDocumentValue | undefined, - ) => CultCacheDocumentValue | undefined | Promise | undefined>, - options: CultCacheUpdateOptions = {}, - ): Promise>> { - if (typeof apply !== "function") { - throw new TypeError("CultCache update needs a function to apply."); - } - // undefined is the default; anything else must be an integer of at least 1, so null is refused. - const maxAttempts = options.maxAttempts === undefined ? 5 : options.maxAttempts; - if (!Number.isInteger(maxAttempts) || maxAttempts < 1) { - throw new RangeError(`CultCache update maxAttempts must be an integer of at least 1, not ${String(maxAttempts)}.`); - } - - for (let attempt = 1; ; attempt++) { - const { expected, copy } = await this.#serial(() => this.#readForUpdate(definition, key)); - // Outside the serial queue and the store's lock: apply may await, and may write this cache through other keys. - const next = await apply(copy as CultCacheDocumentValue | undefined); - if (next === undefined || next === null) { - return { outcome: "declined", document: this.get(definition, key) }; - } - const written = await this.#serial(() => this.#putNow(definition, key, next, expected)); - if (written.outcome === "committed") { - return { outcome: "updated", document: this.get(definition, key) as CultCacheDocumentValue }; - } - if (attempt === maxAttempts) { - // Display only. A cache with no store has nothing to pull, and a pull would replace what it holds with nothing. - if (this.#stores.length > 0) { - await this.#serial(() => this.#pullNow()); - } - return { outcome: "lostRace", document: this.get(definition, key) }; - } - } + ) => CultCacheDocumentValue, + ): Promise> { + return this.#serial(() => this.#putNow(definition, key, updater(this.get(definition, key)))); } - /** `update` of the type's global record. */ - async updateGlobal( + updateGlobal( definition: TDefinition, - apply: ( + updater: ( current: CultCacheDocumentValue | undefined, - ) => CultCacheDocumentValue | undefined | Promise | undefined>, - options: CultCacheUpdateOptions = {}, - ): Promise>> { - this.#requireGlobal(definition); - return this.update(definition, CultCache.GLOBAL_KEY, apply, options); + ) => CultCacheDocumentValue, + ): Promise> { + return this.#serial(() => + this.#putNow(definition, CultCache.GLOBAL_KEY, updater(this.getGlobal(definition))), + ); } delete( @@ -665,8 +615,7 @@ export class CultCache { * another key, a record carrying unread slots nothing consents to). It answers the WRITE form. A removal asks the judge's * removal form and differs in two states: an owner's `delete` of a record held in another store than its own home succeeds * where this is false (a write would push to the wrong store), and `delete` of a global type at a key other than __global__ is - * the plain false of an unheld key where this is false. It does not look at store paths: a link at a store path is not predicted, and - * the write refuses it with `StoreLinkedError`. Live and advisory: the write or removal still decides. + * the plain false of an unheld key where this is false. Live and advisory: the write or removal still decides. */ isWritable(definition: AnyCultCacheDocumentDefinition, key: string): boolean { const registered = this.#requireDefinition(definition); @@ -774,55 +723,11 @@ export class CultCache { this.#stores.push(candidate); } - // One attempt's read for update: the record at the key as its home holds it now, the condition a commit must meet, and a copy of - // the record for apply to change. A store-backed cache asks its store, which reports the record and its version from one read; - // a cache with no store has only the held entry, whose identity is the condition and whose admitted payload is the copy. - async #readForUpdate( - definition: AnyCultCacheDocumentDefinition, - key: string, - ): Promise<{ expected: WriteCondition; copy: unknown }> { - const registered = this.#requireDefinition(definition); - this.#requireGlobalKey(registered, key); - const type = definition.type; - this.#refuseLegacyConditional(type); - const home = this.#homeStore(type); - if (home === undefined) { - const held = this.#hydrated.entries.get(this.#entryId(type, key)); - if (held === undefined && this.#heldUnderOtherType(this.#hydrated, type, key) !== undefined) { - throw new Error(`Record ${key} is not a "${type}"; update cannot copy it.`); - } - return { expected: { held }, copy: held ? registered.formatter.decode(this.#cloneBytes(held.payload)).value : undefined }; - } - if (home.read === undefined || home.commitIf === undefined) { - throw new Error(`CultCache backing store for type "${type}" cannot write conditionally; update needs a store with read and commitIf.`); - } - const found = await home.read(key); - if (found === undefined) { - return { expected: { version: null }, copy: undefined }; - } - if (this.#resolveDefinitionForEnvelope(found.envelope) !== registered) { - throw new Error(`Record ${key} is not a "${type}"; update cannot copy it.`); - } - return { expected: { version: found.version }, copy: registered.formatter.decode(this.#cloneBytes(found.envelope.payload)).value }; - } - - #putNow( - definition: TDefinition, - key: string, - value: CultCacheDocumentValue, - ): Promise<{ outcome: "committed"; value: CultCacheDocumentValue }>; - #putNow( - definition: TDefinition, - key: string, - value: CultCacheDocumentValue, - expected: WriteCondition, - ): Promise<{ outcome: "committed"; value: CultCacheDocumentValue } | { outcome: "mismatch" }>; async #putNow( definition: TDefinition, key: string, value: CultCacheDocumentValue, - expected?: WriteCondition, - ): Promise<{ outcome: "committed"; value: CultCacheDocumentValue } | { outcome: "mismatch" }> { + ): Promise> { const registered = this.#requireDefinition(definition); const payload = registered.formatter.encode(value); const { value: parsed, unreadSlots } = registered.formatter.decode(payload) as { value: CultCacheDocumentValue; unreadSlots?: NotedSlots }; @@ -834,55 +739,19 @@ export class CultCache { schemaId: registered.catalogEntry.schemaId, catalogEntry: registered.catalogEntry, }; - const outcome = await this.#writeNow(registered, entry, parsed, unreadSlots, expected); - return outcome === "committed" ? { outcome, value: parsed } : { outcome }; + await this.#writeNow(registered, entry, parsed, unreadSlots); + return parsed; } // Validates the whole write (home, global singleton, name and index accessors) before the store // is touched; once the store accepts, applying to the cache has no fallible step. - // With a condition, the write lands only if the record at the key is still the one the caller read: the store's version - // (commitIf), or for a cache with no store the held entry itself. Otherwise nothing is written or installed and the answer is - // "mismatch". Without one the write is blind, as put's. - async #writeNow( - registered: RegisteredDefinition, - entry: CultCacheEnvelope, - value: unknown, - unreadSlots?: NotedSlots, - expected?: WriteCondition, - ): Promise<"committed" | "mismatch"> { + async #writeNow(registered: RegisteredDefinition, entry: CultCacheEnvelope, value: unknown, unreadSlots?: NotedSlots): Promise { const home = this.#homeStore(entry.type); const retyped = this.#judge(registered, entry.key, entry); const hydrated = this.#admitEntry(this.#hydrated, registered, entry, value, unreadSlots); this.#refuseIndexSharing(this.#hydrated, entry.type, registered.catalogEntry.schemaName, entry.key, hydrated.lookups); const legacy = this.#hydrated.legacyGlobals.get(entry.type); - // A conditional write over a legacy global is refused here too (a pull while apply awaited can have loaded one): the condition - // was read from the global's own key, the record sits under its legacy key, and the retire below would delete it unseen. - if (expected !== undefined) { - this.#refuseLegacyConditional(entry.type); - } - if (expected !== undefined && "held" in expected) { - // The condition was fixed at read for a cache with no store. A store attached since has its own record at the key that - // this read never saw, so the held identity no longer judges the write: lose the race and read again. - if (home !== undefined || this.#hydrated.entries.get(this.#entryId(entry.type, entry.key)) !== expected.held) { - return "mismatch"; - } - } - if (expected !== undefined && "version" in expected) { - if (home?.commitIf === undefined) { - throw new Error(`CultCache backing store for type "${entry.type}" cannot write conditionally; update needs a store with read and commitIf.`); - } - const committed = await home.commitIf(entry, expected.version); - if (committed.outcome === "mismatch") { - return "mismatch"; - } - hydrated.storedAt = committed.storedAt; - } else { - // A store that re-stamps a write (storedAt later than the record it replaces) reports the stamp; the cache holds that one. - const pushed = await home?.push(entry); - if (typeof pushed === "object" && pushed !== null && typeof pushed.storedAt === "string" && !Number.isNaN(Date.parse(pushed.storedAt))) { - hydrated.storedAt = pushed.storedAt; - } - } + await home?.push(entry); if (home && legacy) { await this.#retireLegacyGlobal(home, legacy, entry); } @@ -892,17 +761,6 @@ export class CultCache { this.#removeEntry(this.#hydrated, this.#definitions.get(retyped.type)!, retyped); } this.#installEntry(this.#hydrated, registered, hydrated); - return "committed"; - } - - // The one rule that a conditional write (update, whichever entry reaches it) never touches a global loaded from a legacy key: - // only putGlobal retires that key. Asked before apply runs and again at the write. - #refuseLegacyConditional(type: string): void { - if (this.#hydrated.legacyGlobals.has(type)) { - throw new Error( - `CultCache global "${type}" was loaded from a legacy key; write it once with putGlobal to retire its legacy key.`, - ); - } } // An envelope of the definition's type at the key, carrying nothing: what the judge asks about a write or a removal by that type. diff --git a/packages/cultcache-ts/src/index.ts b/packages/cultcache-ts/src/index.ts index 8d726e3d..f55dd049 100644 --- a/packages/cultcache-ts/src/index.ts +++ b/packages/cultcache-ts/src/index.ts @@ -1,3 +1,2 @@ export * from "./browser"; export { SingleFileMessagePackBackingStore, readSingleFileStore, type SingleFileStoreRead } from "./single-file-messagepack-backing-store"; -export { StoreLinkedError } from "./store-file"; diff --git a/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts b/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts index 03b93683..b5199ae3 100644 --- a/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts +++ b/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts @@ -1,4 +1,3 @@ -import { createHash } from "node:crypto"; import { lstatSync, readFileSync } from "node:fs"; import { mkdir } from "node:fs/promises"; import { dirname, resolve } from "node:path"; @@ -56,42 +55,14 @@ export class SingleFileMessagePackBackingStore implements CacheBackingStore { return disk.envelopes; } - async push(entry: CultCacheEnvelope): Promise<{ storedAt: string }> { - return this.#enqueue(async () => { + async push(entry: CultCacheEnvelope): Promise { + await this.#enqueue(async () => { const disk = this.#readDisk(); this.#format = disk.format; const existing = disk.envelopes; requireOneTypePerKey(this.filePath, [entry], existing); - const [written] = await this.#writeAll([entry], existing); - return { storedAt: written!.storedAt }; - }); - } - - // The record at the key as the file holds it now, with its content version, from one read. Like pullAll it takes no lock: a - // reader sees a whole file (the commit renames one over the other), and the version it reports is the one commitIf judges. - async read(key: string): Promise<{ envelope: CultCacheEnvelope; version: string } | undefined> { - const envelope = readSingleFileStore(this.filePath).envelopes.find((candidate) => candidate.key === key); - return envelope === undefined ? undefined : { envelope, version: recordVersion(envelope) }; - } - - // Writes `entry` the way push does, but only if the record at its key (of any type) still has `expectedVersion`; null expects - // none. The comparison and the write are one hold of the write lock, so no other writer lands between them. - async commitIf( - entry: CultCacheEnvelope, - expectedVersion: string | null, - ): Promise<{ outcome: "committed"; storedAt: string } | { outcome: "mismatch" }> { - return this.#enqueue(async () => { - const disk = this.#readDisk(); - this.#format = disk.format; - const existing = disk.envelopes; - const current = existing.find((candidate) => candidate.key === entry.key); - if ((current === undefined ? null : recordVersion(current)) !== expectedVersion) { - return { outcome: "mismatch" as const }; - } - - requireOneTypePerKey(this.filePath, [entry], existing); - const [written] = await this.#writeAll([entry], existing); - return { outcome: "committed" as const, storedAt: written!.storedAt }; + const filtered = existing.filter((candidate) => candidate.key !== entry.key); + await this.#writeAll([entry], filtered); }); } @@ -155,67 +126,13 @@ export class SingleFileMessagePackBackingStore implements CacheBackingStore { return result; } - // Writes the supplied records onto the arrived ones and returns the supplied records as written. A supplied record whose - // schema id and payload bytes differ from the arrived one it replaces is never stamped earlier than the record it replaces, so a content version that includes storedAt - // cannot repeat across writes (variants:ruling:storedat-on-change): one not later than the arrived stamp is written one - // millisecond after it. - async #writeAll(supplied: CultCacheEnvelope[], arrived: CultCacheEnvelope[]): Promise { - const arrivedByKey = new Map(arrived.map((entry) => [entry.key, entry])); - const written = supplied.map((entry) => { - const prior = arrivedByKey.get(entry.key); - if (prior === undefined) { - return entry; - } - // A rewrite of the identical record (same schema id and payload bytes) keeps the stamp and so the version it has; the - // writer's type and catalog entry still go down with it. - if (sameContent(entry, prior)) { - return { ...entry, storedAt: prior.storedAt }; - } - return { ...entry, storedAt: laterStoredAt(entry.storedAt, prior.storedAt) }; - }); + async #writeAll(supplied: CultCacheEnvelope[], arrived: CultCacheEnvelope[]): Promise { await mkdir(dirname(this.filePath), { recursive: true }); - await commitStoreFile(this.filePath, encode(encodeSnapshot(written, arrived, this.#format))); - return written; + await commitStoreFile(this.filePath, encode(encodeSnapshot(supplied, arrived, this.#format))); } } -// The content version of a stored record: the lowercase hex SHA-256 of its persisted encoding, the MessagePack array -// [key, schemaId, storedAt, payload] as the store writes it (the C# reference's StoredVersion). Private to this module so -// node:crypto stays out of the browser-safe cache. -export function recordVersion(entry: CultCacheEnvelope): string { - return createHash("sha256") - .update(encode([entry.key, schemaIdForEnvelope(entry), entry.storedAt, documentPayloadBytes(entry.payload)])) - .digest("hex"); -} - -function sameContent(a: CultCacheEnvelope, b: CultCacheEnvelope): boolean { - if (schemaIdForEnvelope(a) !== schemaIdForEnvelope(b)) { - return false; - } - const x = documentPayloadBytes(a.payload); - const y = documentPayloadBytes(b.payload); - return x.length === y.length && x.every((byte, i) => byte === y[i]); -} - -// `storedAt` if it is later than `prior`, else `prior` plus one millisecond. A stamp that does not parse is not later; a prior -// that does not parse leaves nothing to be later than. -function laterStoredAt(storedAt: string, prior: string): string { - const priorMs = Date.parse(prior); - if (Number.isNaN(priorMs)) { - return storedAt; - } - const storedMs = Date.parse(storedAt); - if (!(Number.isNaN(storedMs) || storedMs <= priorMs)) { - return storedAt; - } - // The stamp is ISO-8601 with a four-digit year; one past 9999-12-31T23:59:59.999Z would print as an extended year. - if (priorMs + 1 > Date.UTC(9999, 11, 31, 23, 59, 59, 999)) { - throw new RangeError(`CultCache cannot stamp a write later than the record it replaces (${prior}): the stamp would leave the four-digit years.`); - } - return new Date(priorMs + 1).toISOString(); -} - // A store holds a key once, and the reader refuses a file that holds it twice, so a write that would put a second type under a // key (against a record on disk, or within its own batch) is refused before anything is written. Retyping a record is a delete // of the old type followed by a put of the new one. diff --git a/packages/cultcache-ts/src/store-file.ts b/packages/cultcache-ts/src/store-file.ts index 0488960e..4c7015cc 100644 --- a/packages/cultcache-ts/src/store-file.ts +++ b/packages/cultcache-ts/src/store-file.ts @@ -5,7 +5,7 @@ // lazily inside `loadKernel`, so a consumer that never writes a store never loads it. import { closeSync, openSync, constants as fsConstants } from "node:fs"; -import { lstat, mkdir, open, rename, rm } from "node:fs/promises"; +import { mkdir, open, rename, rm } from "node:fs/promises"; import type { FileHandle } from "node:fs/promises"; import { dirname } from "node:path"; @@ -25,30 +25,6 @@ export class StoreReplacedNotDurableError extends Error { } } -/** Thrown when a write is asked of a store path that is a link: a replace there would swap the link for a regular file. */ -export class StoreLinkedError extends Error { - constructor(readonly path: string) { - super(`CultCache store ${path} is a link, and a write at it would replace the link; configure the link's target as the store path.`); - this.name = "StoreLinkedError"; - } -} - -/** Refuses a link at the store path (file or directory link; libuv reports Windows junctions as links). Absent is not a link. */ -export async function refuseLinkedStore(storePath: string): Promise { - let linked = false; - try { - linked = (await lstat(storePath)).isSymbolicLink(); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ENOENT") { - throw error; - } - } - - if (linked) { - throw new StoreLinkedError(storePath); - } -} - /** The file operations of one commit. Tests record or fail them; production uses `nodeStoreFileOps`. */ export interface StoreFileOps { createExclusive(path: string): Promise; @@ -201,7 +177,6 @@ export async function withStoreWriteLock( ): Promise { const lockPath = `${storePath}.lock`; const timeoutMs = options.timeoutMs ?? LOCK_TIMEOUT_MS; - await refuseLinkedStore(storePath); await mkdir(dirname(lockPath), { recursive: true }); const started = Date.now(); @@ -218,8 +193,6 @@ export async function withStoreWriteLock( } try { - // A link may have been put at the path while the lock was awaited; the look above only saw the path before. - await refuseLinkedStore(storePath); return await operation(); } finally { release(); diff --git a/packages/cultcache-ts/src/types.ts b/packages/cultcache-ts/src/types.ts index 60e0b878..10371053 100644 --- a/packages/cultcache-ts/src/types.ts +++ b/packages/cultcache-ts/src/types.ts @@ -125,34 +125,9 @@ export interface PushAllOptions { export interface CacheBackingStore { pullAll(): Promise; - /** A store that re-stamps the write returns the storedAt it wrote; the cache then holds that stamp, not its own. */ - push(entry: CultCacheEnvelope): Promise; + push(entry: CultCacheEnvelope): Promise; delete(entry: CultCacheEnvelope): Promise; pushAll?(entries: CultCacheEnvelope[], options?: PushAllOptions): Promise; - /** - * The record at the key as the store holds it now, with its content version, from one read; undefined when the store holds - * none. The version is the store's own: the cache carries it from `read` to `commitIf` and never computes it. - */ - read?(key: string): Promise<{ envelope: CultCacheEnvelope; version: string } | undefined>; - /** - * Writes the entry only if the record at its key still has `expectedVersion` (null: no record), judged and written under the - * store's write lock. A mismatch writes nothing. A commit returns the storedAt actually written. - */ - commitIf?( - entry: CultCacheEnvelope, - expectedVersion: string | null, - ): Promise<{ outcome: "committed"; storedAt: string } | { outcome: "mismatch" }>; -} - -/** The outcome of `CultCache.update`: what `apply` wrote, declined to write, or lost to a rival writer. */ -export type CultCacheUpdateResult = - | { outcome: "updated"; document: TValue } - | { outcome: "declined"; document: TValue | undefined } - | { outcome: "lostRace"; document: TValue | undefined }; - -export interface CultCacheUpdateOptions { - /** How many times `apply` may run before a lost race is returned as `lostRace`. An integer of at least 1; default 5. */ - maxAttempts?: number; } export interface CultCacheStoreRegistration { diff --git a/packages/cultcache-ts/test/cult-cache.test.ts b/packages/cultcache-ts/test/cult-cache.test.ts index dc1ddb33..7c559af7 100644 --- a/packages/cultcache-ts/test/cult-cache.test.ts +++ b/packages/cultcache-ts/test/cult-cache.test.ts @@ -15,7 +15,6 @@ import { CultCache } from "../src/cult-cache"; import { inspectCultCacheBytes } from "../src/cult-cache-inspector"; import { defineDocumentRegistry, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; import { SingleFileMessagePackBackingStore, readSingleFileStore, readStoreBytes } from "../src/single-file-messagepack-backing-store"; -import { StoreLinkedError, withStoreWriteLock } from "../src/store-file"; import { SchemaConflictError, StoreUnreadableError, refuseUnreadStoreBytes } from "../src/store-format"; import type { AnyCultCacheDocumentDefinition, CacheBackingStore, CultCacheEnvelope, CultCacheSchema } from "../src/types"; @@ -2106,7 +2105,7 @@ async function readTsInteropStore(file: string): Promise { async function buildInteropPeers(): Promise { if (!(await exists(rustInteropBinary))) { - await execAsync(`"${cargoCommand}" build --locked --quiet --example cultcache_interop`, { + await execAsync(`"${cargoCommand}" build --quiet --example cultcache_interop`, { cwd: cultcacheRsRoot, }); } @@ -2467,36 +2466,12 @@ test("SingleFileMessagePackBackingStore refuses a dangling symlink at its path a () => store.pullAll(), (error: NodeJS.ErrnoException) => error.code === "ENOENT" && error.message.includes("symbolic link") && error.message.includes(file), ); - await assert.rejects(() => store.push(envelope), (error: unknown) => error instanceof StoreLinkedError); - await assert.rejects(() => store.pushAll([envelope], { soft: true }), (error: unknown) => error instanceof StoreLinkedError); + await assert.rejects(() => store.push(envelope), (error: NodeJS.ErrnoException) => error.code === "ENOENT"); + await assert.rejects(() => store.pushAll([envelope], { soft: true }), (error: NodeJS.ErrnoException) => error.code === "ENOENT"); assert.ok((await lstat(file)).isSymbolicLink()); assert.deepEqual(await readdir(volume), []); }); -// The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse. -test("isWritable over a store at a live link answers true while the cache write is refused", { skip: process.platform === "win32" }, async () => { - const dir = await mkdtemp(join(tmpdir(), "cultcache-writable-link-")); - const volume = join(dir, "volume"); - await mkdir(volume); - const target = join(volume, "store.cc"); - const link = join(dir, "link.cc"); - const document = defineDocumentType({ type: "note", schema: z.object({ text: z.string() }) }); - await CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(target)).build() - .put(document, "held", { text: "held" }); - const before = await readFile(target); - await symlink(target, link); - const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(link)).build(); - await cache.pullAllBackingStores(); - const namesBefore = [(await readdir(dir)).sort(), (await readdir(volume)).sort()]; - - assert.equal(cache.isWritable(document, "held"), true); - assert.equal(cache.isWritable(document, "fresh"), true); - await assert.rejects(() => cache.put(document, "fresh", { text: "fresh" }), (error: unknown) => error instanceof StoreLinkedError && error.path === link); - assert.ok((await lstat(link)).isSymbolicLink()); - assert.deepEqual(await readFile(target), before); - assert.deepEqual([(await readdir(dir)).sort(), (await readdir(volume)).sort()], namesBefore); -}); - // The second look of the read, staged: nothing is at the path only when the read and the lstat both find nothing. A path // that changed between them (a link in a parent swapped for a file, a store that appeared) is never an empty store. test("readStoreBytes reports nothing at the path only when the lstat finds nothing too", async () => { @@ -2533,9 +2508,9 @@ test("SingleFileMessagePackBackingStore record refusal names only a string key", } }); -// A write at a store path that is a link is refused: a replace there would swap the link for a regular file and -// detach the store from its target. Reads through a live link keep working. -test("a write at a live link is refused and nothing changes, while a read through it still works", { skip: process.platform === "win32" }, async () => { +// What a write does through a symbolic link today. R3 decides whether a write resolves the link +// or refuses it; these pin the current behaviour so that change is deliberate. +test("R3 decides resolve-or-refuse: a push through a live link replaces the link", { skip: process.platform === "win32" }, async () => { const dir = await mkdtemp(join(tmpdir(), "cultcache-live-link-")); const volume = join(dir, "volume"); await mkdir(volume); @@ -2543,31 +2518,11 @@ test("a write at a live link is refused and nothing changes, while a read throug const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; await new SingleFileMessagePackBackingStore(target).push(envelope); const before = await readFile(target); - const volumeBefore = (await readdir(volume)).sort(); - const link = join(dir, "link.cc"); + const link = join(dir, "store.cc"); await symlink(target, link); - const store = new SingleFileMessagePackBackingStore(link); - const refused = (error: unknown) => error instanceof StoreLinkedError && error.path === link && error.message.includes(link); - await assert.rejects(() => store.push({ ...envelope, key: "b" }), refused); - await assert.rejects(() => store.pushAll([{ ...envelope, key: "b" }]), refused); - await assert.rejects(() => store.delete(envelope), refused); - assert.ok((await lstat(link)).isSymbolicLink()); + await new SingleFileMessagePackBackingStore(link).push({ ...envelope, key: "b" }); + assert.ok((await lstat(link)).isFile()); assert.deepEqual(await readFile(target), before); - assert.deepEqual((await readdir(dir)).sort(), ["link.cc", "volume"]); - assert.deepEqual((await readdir(volume)).sort(), volumeBefore); - assert.deepEqual((await store.pullAll()).map((e) => e.key), ["a"]); -}); - -test("a write at a live directory link is refused as a link", { skip: process.platform === "win32" }, async () => { - const dir = await mkdtemp(join(tmpdir(), "cultcache-dir-link-")); - const realDir = join(dir, "real"); - await mkdir(realDir); - const link = join(dir, "dirlink"); - await symlink(realDir, link); - const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; - await assert.rejects(() => new SingleFileMessagePackBackingStore(link).push(envelope), (error: unknown) => error instanceof StoreLinkedError); - assert.ok((await lstat(link)).isSymbolicLink()); - assert.deepEqual(await readdir(realDir), []); }); test("a pushAll through a dangling link is refused like push, and the link and its volume are left as they were", { skip: process.platform === "win32" }, async () => { @@ -2577,7 +2532,7 @@ test("a pushAll through a dangling link is refused like push, and the link and i const link = join(dir, "store.cc"); await symlink(join(volume, "store.cc"), link); const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; - await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([envelope]), (error: unknown) => error instanceof StoreLinkedError); + await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([envelope]), (error: NodeJS.ErrnoException) => error.code === "ENOENT"); assert.ok((await lstat(link)).isSymbolicLink()); assert.deepEqual(await readdir(volume), []); }); @@ -2872,6 +2827,7 @@ test("every registration refusal is a SchemaConflictError naming the id and both // Cross-runtime store lock: each runtime's writer excludes the others on `.lock`. import { spawn } from "node:child_process"; +import { withStoreWriteLock } from "../src/store-file"; type LockHolder = { exited: Promise; stderr: () => string }; diff --git a/packages/cultcache-ts/test/first-call.test.ts b/packages/cultcache-ts/test/first-call.test.ts index 9ef3d308..45f6edd2 100644 --- a/packages/cultcache-ts/test/first-call.test.ts +++ b/packages/cultcache-ts/test/first-call.test.ts @@ -46,21 +46,6 @@ async function firstCall(path: string) { } // snippet:end -// The update snippet, which docs/start/README.md shows after the first call and which uses its definitions. -// snippet:update:begin -async function restock(path: string) { - const cache = CultCache.builder() - .withRegistry(registry) - .withGenericStore(new SingleFileMessagePackBackingStore(path)) - .build(); - await cache.pullAllBackingStores(); - return cache.update(itemDocument, "item:potion", (item) => { - const current = item ?? { name: "Potion", category: "Consumable", value: 0 }; - return { ...current, value: current.value + 10 }; - }); -} -// snippet:update:end - test("define, store and read back a typed document", async () => { const dir = await mkdtemp(join(tmpdir(), "cultcache-first-call-")); try { @@ -72,35 +57,13 @@ test("define, store and read back a typed document", async () => { } }); -test("update with retry: restock twice reads value 20 through a second cache", async () => { - const dir = await mkdtemp(join(tmpdir(), "cultcache-first-call-")); - try { - const path = join(dir, "restock.cc"); - assert.equal((await restock(path)).outcome, "updated"); - assert.equal((await restock(path)).outcome, "updated"); - const reopened = CultCache.builder() - .withRegistry(registry) - .withGenericStore(new SingleFileMessagePackBackingStore(path)) - .build(); - await reopened.pullAllBackingStores(); - assert.equal(reopened.get(itemDocument, "item:potion")?.value, 20); - } finally { - await rm(dir, { recursive: true, force: true }); - } -}); - -// The docs show the region between `// snippet:begin` and `end` markers verbatim, the package import aside. -async function assertDocsShow(region: "" | "update:") { +test("docs show the tested snippet", async () => { // Compiled to dist-test/test/first-call.test.js, so the package root is two levels up. const source = await readFile(join(__dirname, "..", "..", "test", "first-call.test.ts"), "utf8"); const snippet = source - .split(`// snippet:${region}begin\n`)[1] - .split(`// snippet:${region}end\n`)[0] + .split("// snippet:begin\n")[1] + .split("// snippet:end\n")[0] .replace('"../src/index"', '"@gamecult/cultcache-ts"'); const docs = (await readFile(join(__dirname, "..", "..", "..", "..", "docs", "start", "README.md"), "utf8")).replace(/\r\n/g, "\n"); - assert.ok(docs.includes("```ts\n" + snippet + "```"), `docs/start/README.md no longer shows the ${region || "first-call "}region of test/first-call.test.ts`); -} - -test("docs show the tested snippet", () => assertDocsShow("")); - -test("docs show the tested update snippet", () => assertDocsShow("update:")); + assert.ok(docs.includes("```ts\n" + snippet + "```"), "docs/start/README.md no longer shows test/first-call.test.ts"); +}); diff --git a/packages/cultcache-ts/test/store-conditional.test.ts b/packages/cultcache-ts/test/store-conditional.test.ts deleted file mode 100644 index 5f8f4ed5..00000000 --- a/packages/cultcache-ts/test/store-conditional.test.ts +++ /dev/null @@ -1,175 +0,0 @@ -import assert from "node:assert/strict"; -import { createHash } from "node:crypto"; -import { existsSync, readFileSync } from "node:fs"; -import { copyFile, mkdtemp, readFile, rm } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import { join, resolve } from "node:path"; -import test from "node:test"; - -import { decode, encode } from "@msgpack/msgpack"; - -import { documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; -import { SingleFileMessagePackBackingStore, readSingleFileStore, recordVersion } from "../src/single-file-messagepack-backing-store"; -import { schemaIdForEnvelope } from "../src/store-format"; -import type { CultCacheEnvelope } from "../src/types"; - -// The store owns the content version and the conditional verdict (variants:ruling:write-set-content-version-declared-ownership): -// read reports a record with its version, commitIf writes only while the version holds, and a write that replaces a record -// is stamped later than it (variants:ruling:storedat-on-change). - -function findVectors(from: string): string { - for (let dir = from; ; dir = resolve(dir, "..")) { - if (existsSync(join(dir, "tests", "vectors", "content-version"))) return join(dir, "tests", "vectors", "content-version"); - if (resolve(dir, "..") === dir) throw new Error("tests/vectors/content-version not found above the test."); - } -} - -const entry = (key: string, byte: number, storedAt = "2026-01-01T00:00:00.000Z", type = "t"): CultCacheEnvelope => ({ - key, - type, - payload: documentPayloadFromBytes(Uint8Array.of(0x91, byte)), - storedAt, -}); - -async function scratch(): Promise { - return mkdtemp(join(tmpdir(), "cultcache-conditional-")); -} - -test("the content version of every record of the C#-written store equals the version the reference read", async () => { - const vectors = findVectors(__dirname); - const rows = decode(readFileSync(join(vectors, "versions.msgpack"))) as Array<[string, string]>; - assert.equal(rows.length, 2); - const directory = await scratch(); - try { - const path = join(directory, "store.cc"); - await copyFile(join(vectors, "store.cc"), path); - const store = new SingleFileMessagePackBackingStore(path); - for (const [key, version] of rows) { - const found = await store.read(key); - assert.equal(found?.version, version, `read(${key}).version`); - assert.equal(recordVersion(found!.envelope), version, `recordVersion(${key})`); - } - assert.equal(await store.read("absent"), undefined); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("the version is the SHA-256 of the record's persisted encoding and changes with every field of it", async () => { - const base = entry("k", 1); - const hex = (e: CultCacheEnvelope) => - createHash("sha256").update(encode([e.key, schemaIdForEnvelope(e), e.storedAt, documentPayloadBytes(e.payload)])).digest("hex"); - assert.equal(recordVersion(base), hex(base)); - const versions = new Set([base, entry("k", 2), entry("k", 1, "2026-01-01T00:00:00.001Z"), entry("j", 1), { ...base, schemaId: "sha256:other" }].map(recordVersion)); - assert.equal(versions.size, 5); -}); - -test("commitIf writes nothing when the version does not hold, and commits when it does", async () => { - const directory = await scratch(); - try { - const path = join(directory, "store.cc"); - const store = new SingleFileMessagePackBackingStore(path); - // null expects no record: it commits over an absent key and mismatches over a present one. - assert.deepEqual(await store.commitIf(entry("k", 1), "not-a-version"), { outcome: "mismatch" }); - assert.equal(existsSync(path), false, "a mismatch creates no file"); - assert.equal((await store.commitIf(entry("k", 1), null)).outcome, "committed"); - const first = (await store.read("k"))!; - const before = await readFile(path); - assert.deepEqual(await store.commitIf(entry("k", 2), null), { outcome: "mismatch" }); - assert.deepEqual(await store.commitIf(entry("k", 2), "0".repeat(64)), { outcome: "mismatch" }); - assert.deepEqual([...(await readFile(path))], [...before], "a mismatch leaves the bytes identical"); - assert.equal((await store.commitIf(entry("k", 2), first.version)).outcome, "committed"); - const second = (await store.read("k"))!; - assert.notEqual(second.version, first.version); - assert.deepEqual([...documentPayloadBytes(second.envelope.payload)], [0x91, 2]); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("commitIf judges the record at the key whatever its type, and a record of another key does not matter", async () => { - const directory = await scratch(); - try { - const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); - await store.push(entry("a", 1, undefined, "u")); - await store.push(entry("b", 1)); - const other = (await store.read("a"))!; - assert.deepEqual(await store.commitIf(entry("b", 2), other.version), { outcome: "mismatch" }); - assert.equal((await store.commitIf(entry("b", 2), recordVersion((await store.read("b"))!.envelope))).outcome, "committed"); - assert.deepEqual([...documentPayloadBytes((await store.read("a"))!.envelope.payload)], [0x91, 1], "the record at another key is untouched"); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("a write that replaces a record is stored strictly later than it, whatever the write's own storedAt", async () => { - const directory = await scratch(); - try { - const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); - const at = "2026-03-04T05:06:07.100Z"; - await store.push(entry("k", 1, at)); - // Not later (equal, earlier, unparseable) is written one millisecond after the record it replaces, by push, pushAll and commitIf. - await store.push(entry("k", 2, at)); - assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.101Z"); - await store.push(entry("k", 3, "2020-01-01T00:00:00.000Z")); - assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.102Z"); - await store.pushAll([entry("k", 4, "not a time")]); - assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.103Z"); - const committed = await store.commitIf(entry("k", 5, "2020-01-01T00:00:00.000Z"), (await store.read("k"))!.version); - assert.deepEqual(committed, { outcome: "committed", storedAt: "2026-03-04T05:06:07.104Z" }); - assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.104Z"); - // A later stamp is kept as it is, and a key with no record before it is stamped as it is. - await store.push(entry("k", 6, "2030-01-01T00:00:00.000Z")); - await store.push(entry("fresh", 1, "2000-01-01T00:00:00.000Z")); - assert.equal((await store.read("k"))!.envelope.storedAt, "2030-01-01T00:00:00.000Z"); - assert.equal((await store.read("fresh"))!.envelope.storedAt, "2000-01-01T00:00:00.000Z"); - assert.equal(readSingleFileStore(join(directory, "store.cc")).envelopes.length, 2); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("a replacing write that would stamp past year 9999 is refused and writes nothing", async () => { - const directory = await scratch(); - try { - const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); - const last = "9999-12-31T23:59:59.999Z"; - await store.push(entry("k", 1, last)); - await assert.rejects(() => store.push(entry("k", 2, last)), RangeError); - assert.equal((await store.read("k"))!.envelope.storedAt, last); - assert.deepEqual([...documentPayloadBytes((await store.read("k"))!.envelope.payload)], [0x91, 1]); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("a write that restores an earlier value does not restore its version", async () => { - const directory = await scratch(); - try { - const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); - const at = "2026-03-04T05:06:07.100Z"; - await store.push(entry("k", 1, at)); - const first = (await store.read("k"))!.version; - await store.push(entry("k", 5, at)); - await store.push(entry("k", 1, at)); - assert.notEqual((await store.read("k"))!.version, first, "an A-B-A within one millisecond keeps no version"); - assert.deepEqual(await store.commitIf(entry("k", 9, at), first), { outcome: "mismatch" }); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("two stores on one file never both commit against the same version", async () => { - const directory = await scratch(); - try { - const path = join(directory, "store.cc"); - const left = new SingleFileMessagePackBackingStore(path); - const right = new SingleFileMessagePackBackingStore(path); - await left.push(entry("k", 0)); - const version = (await left.read("k"))!.version; - const outcomes = await Promise.all(Array.from({ length: 6 }, (_, i) => (i % 2 === 0 ? left : right).commitIf(entry("k", i + 1), version))); - assert.equal(outcomes.filter((o) => o.outcome === "committed").length, 1); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); diff --git a/packages/cultcache-ts/test/store-file.test.ts b/packages/cultcache-ts/test/store-file.test.ts index 6cb70392..39436e30 100644 --- a/packages/cultcache-ts/test/store-file.test.ts +++ b/packages/cultcache-ts/test/store-file.test.ts @@ -10,7 +10,6 @@ import { promisify } from "node:util"; import { decode, encode } from "@msgpack/msgpack"; -import { StoreLinkedError as PackageStoreLinkedError } from "../src/index"; import { documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; import { SingleFileMessagePackBackingStore } from "../src/single-file-messagepack-backing-store"; import { SchemaConflictError, StoreUnreadableError } from "../src/store-format"; @@ -24,7 +23,6 @@ import { replaceStoreFile, replaceStoreFileWindows, windowsKernel, - refuseLinkedStore, withStoreWriteLock, } from "../src/store-file"; import type { ReplaceKernel, StoreFileOps } from "../src/store-file"; @@ -417,43 +415,6 @@ test("a second writer in this process waits for the store lock and runs after re } }); -test("a link put while a writer waits on the store lock is refused once it holds the lock", { skip: process.platform === "win32" }, async () => { - const directory = await scratch(); - try { - const target = join(directory, "target.cc"); - await new SingleFileMessagePackBackingStore(target).push(noteEnvelope("a")); - const before = await readFile(target); - const writes: Array<[string, (store: SingleFileMessagePackBackingStore) => Promise]> = [ - ["push", async (store) => void (await store.push(noteEnvelope("b")))], - ["pushAll", (store) => store.pushAll([noteEnvelope("b")])], - ["delete", (store) => store.delete(noteEnvelope("a"))], - ]; - const missing = join(directory, "missing.cc"); - for (const [name, write, linkTo] of writes.flatMap(([n, w]) => [[n, w, target], [`${n} (dangling)`, w, missing]] as const)) { - const link = join(directory, `${name.replace(/\W+/g, "-")}.cc`); - let writer: Promise | undefined; - await withStoreWriteLock(link, async () => { - writer = write(new SingleFileMessagePackBackingStore(link)); - writer.catch(() => undefined); - await sleepFor(150); - await fsp.symlink(linkTo, link); // after the writer's pre-lock look, while it waits - }); - await assert.rejects(writer!, (error: unknown) => error instanceof PackageStoreLinkedError, name); - assert.ok((await fsp.lstat(link)).isSymbolicLink(), `${name}: the link was replaced`); - assert.deepEqual(await readFile(target), before, `${name}: the target changed`); - await assert.rejects(fsp.lstat(missing), { code: "ENOENT" }, `${name}: the dangling target was created`); - await rm(link); // the link is refused before the lock, so remove it to probe the lock - assert.equal(await withStoreWriteLock(link, async () => "free", { timeoutMs: 200 }), "free", `${name}: the lock is held`); - } - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -function noteEnvelope(key: string): CultCacheEnvelope { - return { key, type: "note", payload: documentPayloadFromBytes(Uint8Array.of(0x91, 1)), storedAt: "2026-10-20T00:00:00Z" }; -} - test("the store lock gives up after its wait bound and names the lock path", async () => { const directory = await scratch(); try { @@ -857,35 +818,3 @@ test("opening writes nothing: a missing store leaves no lock file, a legacy stor await rm(directory, { recursive: true, force: true }); } }); - -test("StoreLinkedError is importable from the package entry and is what a write at a link throws", { skip: process.platform === "win32" }, async () => { - const directory = await scratch(); - try { - const target = join(directory, "target.cc"); - await writeFile(target, "x"); - const link = join(directory, "link.cc"); - await fsp.symlink(target, link); - await assert.rejects( - () => refuseLinkedStore(link), - (error: unknown) => error instanceof PackageStoreLinkedError && error.path === link && error.name === "StoreLinkedError", - ); - await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([]), (error: unknown) => error instanceof PackageStoreLinkedError); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); - -test("refuseLinkedStore lets a path that is absent pass and rethrows any other lstat failure", { skip: process.platform === "win32" }, async () => { - const directory = await scratch(); - try { - await refuseLinkedStore(join(directory, "absent.cc")); - const file = join(directory, "file"); - await writeFile(file, "x"); - await assert.rejects( - () => refuseLinkedStore(join(file, "child.cc")), - (error: unknown) => (error as NodeJS.ErrnoException).code === "ENOTDIR", - ); - } finally { - await rm(directory, { recursive: true, force: true }); - } -}); diff --git a/packages/cultcache-ts/test/union-surface-fixture.ts b/packages/cultcache-ts/test/union-surface-fixture.ts index 28089522..1b556eb1 100644 --- a/packages/cultcache-ts/test/union-surface-fixture.ts +++ b/packages/cultcache-ts/test/union-surface-fixture.ts @@ -7,7 +7,6 @@ import { z } from "zod"; import { codecs } from "../src"; import type { Codec } from "../src/codec"; import { defineDocumentType, documentPayloadFromBytes } from "../src/document"; -import { recordVersion } from "../src/single-file-messagepack-backing-store"; import type { CacheBackingStore, CultCacheEnvelope } from "../src/types"; // The union-surface fixture: the codecs, schema and value tests/vectors/union-parity was written from, shared by the @@ -236,16 +235,6 @@ export function memoryStore(entries: CultCacheEnvelope[]): CacheBackingStore & { async pullAll() { return entries.map((entry) => ({ ...entry })); }, async push(entry) { entries.push(entry); }, async delete() {}, - async read(key) { - const envelope = [...entries].reverse().find((entry) => entry.key === key); - return envelope === undefined ? undefined : { envelope, version: recordVersion(envelope) }; - }, - async commitIf(entry, expectedVersion) { - const current = [...entries].reverse().find((candidate) => candidate.key === entry.key); - if ((current === undefined ? null : recordVersion(current)) !== expectedVersion) return { outcome: "mismatch" }; - entries.push(entry); - return { outcome: "committed", storedAt: entry.storedAt }; - }, }; } diff --git a/packages/cultcache-ts/test/update.test.ts b/packages/cultcache-ts/test/update.test.ts deleted file mode 100644 index fb0d56fd..00000000 --- a/packages/cultcache-ts/test/update.test.ts +++ /dev/null @@ -1,600 +0,0 @@ -import assert from "node:assert/strict"; -import { mkdtemp, readFile, rm } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import test from "node:test"; -import { setImmediate as tick } from "node:timers/promises"; - -import { decode, encode } from "@msgpack/msgpack"; -import { z } from "zod"; - -import { CultCache } from "../src/cult-cache"; -import { defineDocumentRegistry, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; -import { SingleFileMessagePackBackingStore, readSingleFileStore } from "../src/single-file-messagepack-backing-store"; -import { SchemaConflictError, schemaIdForEnvelope } from "../src/store-format"; -import type { CacheBackingStore } from "../src/types"; -import { envelopeOf, memoryStore, read, surfaceDocument } from "./union-surface-fixture"; - -// CultCache.update: read the record and its content version from the store, apply a function to a copy, commit only if the -// version still holds, run again on a lost race (variants:cut_spec:cut-update-with-retry-ts.r1; the C# tests in -// tests/GameCult.Caching.Tests/UpdateTests.cs are the parity). - -const counterDocument = defineDocumentType({ - type: "update.counter", - schemaId: "update.counter.v1", - schemaName: "update.counter", - schemaVersion: "update.counter.v1", - schema: z.object({ value: z.number().int() }), -}); -const tallyDocument = defineDocumentType({ - type: "update.tally", - schemaId: "update.tally.v1", - schemaName: "update.tally", - schemaVersion: "update.tally.v1", - schema: z.object({ count: z.number().int() }), -}); -const registry = defineDocumentRegistry(counterDocument, tallyDocument); -const settingsFields = { schemaName: "update.settings", schemaVersion: "update.settings.v1", schemaId: "update.settings.v1", schema: z.object({ theme: z.string() }) }; -const settingsDocument = defineDocumentType({ type: "update.settings", ...settingsFields, global: true }); -const plainSettingsDocument = defineDocumentType({ type: "update.settings", ...settingsFields }); - -const KEY = "counter"; - -async function withDirectory(body: (path: string) => Promise): Promise { - const directory = await mkdtemp(join(tmpdir(), "cultcache-update-")); - try { - return await body(join(directory, "store.cc")); - } finally { - await rm(directory, { recursive: true, force: true }); - } -} - -type Registry = Parameters["withRegistry"]>[0]; - -async function open(path: string, documents: Registry = registry): Promise { - const cache = CultCache.builder().withRegistry(documents).withGenericStore(new SingleFileMessagePackBackingStore(path)).build(); - await cache.pullAllBackingStores(); - return cache; -} - -// What the file holds at the key, read without a cache. -function stored(path: string, key = KEY): number | undefined { - const envelope = readSingleFileStore(path).envelopes.find((candidate) => candidate.key === key); - return envelope === undefined ? undefined : (decode(documentPayloadBytes(envelope.payload)) as { value: number }).value; -} - -const plus = (n: number) => (current: { value: number } | undefined) => ({ value: (current?.value ?? 0) + n }); - -test("two writers updating lose no increment", async () => { - await withDirectory(async (path) => { - const [left, right] = [await open(path), await open(path)]; - let calls = 0; - const loop = async (cache: CultCache) => { - const outcomes: string[] = []; - for (let i = 0; i < 50; i++) { - const result = await cache.update(counterDocument, KEY, async (current) => { - calls++; - await tick(); - return { value: (current?.value ?? 0) + 1 }; - }, { maxAttempts: 50 }); - outcomes.push(result.outcome); - } - return outcomes; - }; - const outcomes = (await Promise.all([loop(left), loop(right)])).flat(); - assert.ok(outcomes.every((outcome) => outcome === "updated"), outcomes.join()); - assert.equal(stored(path), 100); - assert.ok(calls > 100, `the caches raced: apply ran ${calls} times`); - }); -}); - -test("a lost attempt reruns apply on the fresh value", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - const seen: Array = []; - const result = await cache.update(counterDocument, KEY, async (current) => { - seen.push(current?.value); - if (seen.length === 1) await rival.put(counterDocument, KEY, { value: 100 }); - return { value: current!.value + 1 }; - }); - assert.deepEqual(seen, [1, 100]); - assert.equal(result.outcome, "updated"); - assert.equal(result.document?.value, 101); - assert.equal(stored(path), 101); - }); -}); - -test("exhausted attempts return lostRace, and the cache then shows the store", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - let calls = 0; - const result = await cache.update( - counterDocument, - KEY, - async (current) => { - calls++; - await rival.put(counterDocument, KEY, { value: 100 * calls }); - return { value: current!.value + 1 }; - }, - { maxAttempts: 3 }, - ); - assert.equal(calls, 3); - assert.equal(result.outcome, "lostRace"); - assert.equal(result.document?.value, 300); - assert.equal(stored(path), 300); - }); -}); - -test("five attempts by default", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - let calls = 0; - const result = await cache.update(counterDocument, KEY, async () => { - calls++; - await rival.put(counterDocument, KEY, { value: calls }); - return { value: -1 }; - }); - assert.equal(result.outcome, "lostRace"); - assert.equal(calls, 5); - }); -}); - -test("maxAttempts 1 makes one call", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - let calls = 0; - const apply = async (current: { value: number } | undefined) => { - calls++; - await rival.put(counterDocument, KEY, { value: 50 + calls }); - return { value: current!.value + 1 }; - }; - assert.equal((await cache.update(counterDocument, KEY, apply, { maxAttempts: 1 })).outcome, "lostRace"); - assert.equal(calls, 1); - }); -}); - -// An apply that declines resolves at once whatever the bound, so a validation that lets a bad value through fails the assertion -// below instead of looping: no apply here ever loses a race. -test("invalid maxAttempts (undefined is the default, null is refused) and a non-function reject before any call", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - let calls = 0; - const apply = () => (calls++, undefined); - for (const maxAttempts of [0, -1, 1.5, Number.NaN, Number.POSITIVE_INFINITY, null as never]) { - await assert.rejects(() => cache.update(counterDocument, KEY, apply, { maxAttempts }), RangeError, `maxAttempts ${String(maxAttempts)}`); - } - await assert.rejects(() => cache.update(counterDocument, KEY, "no" as never), TypeError); - assert.equal(calls, 0); - assert.equal((await cache.update(counterDocument, KEY, apply, { maxAttempts: undefined })).outcome, "declined"); - assert.equal(calls, 1); - }); -}); - -test("null from apply declines as undefined does: nothing is written", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - await cache.put(counterDocument, KEY, { value: 1 }); - const before = await readFile(path); - const result = await cache.update(counterDocument, KEY, () => null as never); - assert.deepEqual(result, { outcome: "declined", document: cache.get(counterDocument, KEY) }); - assert.deepEqual([...(await readFile(path))], [...before]); - }); -}); - -test("a cache with no store refuses a key held as another type before apply runs", async () => { - const cache = CultCache.builder().withRegistry(registry).build(); - await cache.put(tallyDocument, KEY, { count: 1 }); - let calls = 0; - await assert.rejects(() => cache.update(counterDocument, KEY, () => (calls++, { value: 1 })), /is not a "update.counter"/); - assert.equal(calls, 0); -}); - -test("a store attached while apply awaits turns the commit into a lost race, never a blind push", async () => { - await withDirectory(async (path) => { - const writer = await open(path); - await writer.put(counterDocument, KEY, { value: 50 }); - const cache = CultCache.builder().withRegistry(registry).build(); - const seen: Array = []; - const result = await cache.update(counterDocument, KEY, async (current) => { - seen.push(current?.value); - if (seen.length === 1) cache.addBackingStore(new SingleFileMessagePackBackingStore(path)); - return { value: (current?.value ?? 0) + 1 }; - }); - assert.deepEqual(seen, [undefined, 50], "the second attempt reads the store's record"); - assert.equal(result.outcome, "updated"); - assert.equal(stored(path), 51); - }); -}); - -test("put installs the storedAt the store stamped, so the cache and the file agree", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - await cache.put(counterDocument, KEY, { value: 1 }); - const store = new SingleFileMessagePackBackingStore(path); - const [envelope] = readSingleFileStore(path).envelopes; - await store.push({ ...envelope!, payload: documentPayloadFromBytes(encode({ value: 99 })), storedAt: "2099-01-01T00:00:00.000Z" }); - await cache.put(counterDocument, KEY, { value: 2 }); - const onDisk = readSingleFileStore(path).envelopes[0]!.storedAt; - assert.equal(onDisk, "2099-01-01T00:00:00.001Z"); - assert.equal(cache.getEnvelope(counterDocument, KEY)!.storedAt, onDisk); - }); -}); - -test("apply gets a fresh copy: changing it changes nothing held, and a lost attempt's change is not seen by the next", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - const argument: Array<{ value: number } | undefined> = []; - const seen: number[] = []; - await cache.update(counterDocument, KEY, async (current) => { - argument.push(current); - assert.notEqual(current, cache.get(counterDocument, KEY)); - seen.push(current!.value); - current!.value += 1000; - assert.equal(cache.get(counterDocument, KEY)!.value, 1); - if (seen.length === 1) await rival.put(counterDocument, KEY, { value: 7 }); - return { value: current!.value + 1 }; - }); - assert.deepEqual(seen, [1, 7], "the second attempt does not see the first one's change"); - assert.notEqual(argument[0], argument[1]); - }); -}); - -test("an in-place change to the held value is neither applied nor persisted", async () => { - await withDirectory(async (path) => { - const [cache, fresh] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - cache.get(counterDocument, KEY)!.value = 999; - const seen: number[] = []; - const result = await cache.update(counterDocument, KEY, (current) => { - seen.push(current!.value); - return { value: current!.value + 1 }; - }); - assert.deepEqual(seen, [1]); - assert.equal(stored(path), 2); - await fresh.pullAllBackingStores(); - assert.equal(fresh.get(counterDocument, KEY)!.value, 2); - result.document!.value = 999; - await cache.update(counterDocument, KEY, plus(1)); - assert.equal(stored(path), 3); - }); -}); - -test("undefined from apply declines: nothing is written", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - await cache.put(counterDocument, KEY, { value: 1 }); - const before = await readFile(path); - const held = await cache.update(counterDocument, KEY, () => undefined); - assert.equal(held.outcome, "declined"); - assert.equal(held.document, cache.get(counterDocument, KEY)); - const absent = await cache.update(counterDocument, "absent", () => undefined); - assert.deepEqual(absent, { outcome: "declined", document: undefined }); - assert.deepEqual([...(await readFile(path))], [...before]); - }); -}); - -test("an absent key is created once under a race", async () => { - await withDirectory(async (path) => { - const [left, right] = [await open(path), await open(path)]; - const outcomes = await Promise.all( - [left, right].map((cache) => - cache.update(counterDocument, KEY, async (current) => { - await tick(); - return { value: (current?.value ?? 0) + 1 }; - }), - ), - ); - assert.deepEqual(outcomes.map((result) => result.outcome), ["updated", "updated"]); - assert.equal(stored(path), 2); - }); -}); - -test("a rival writing and restoring the value still reruns apply", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - const seen: number[] = []; - await cache.update(counterDocument, KEY, async (current) => { - seen.push(current!.value); - if (seen.length === 1) { - await rival.put(counterDocument, KEY, { value: 5 }); - await rival.put(counterDocument, KEY, { value: 1 }); - } - return { value: current!.value + 1 }; - }); - assert.deepEqual(seen, [1, 1]); - assert.equal(stored(path), 2); - }); -}); - -test("a rival removal is seen as undefined", async () => { - await withDirectory(async (path) => { - const [cache, rival] = [await open(path), await open(path)]; - await cache.put(counterDocument, KEY, { value: 1 }); - const seen: Array = []; - await cache.update(counterDocument, KEY, async (current) => { - seen.push(current?.value); - if (seen.length === 1) { - await rival.pullAllBackingStores(); - await rival.delete(counterDocument, KEY); - } - return { value: (current?.value ?? 0) + 1 }; - }); - assert.deepEqual(seen, [1, undefined]); - assert.equal(stored(path), 1); - }); -}); - -test("a stale cache's first attempt reads the store", async () => { - await withDirectory(async (path) => { - const [stale, rival] = [await open(path), await open(path)]; - await rival.put(counterDocument, KEY, { value: 1 }); - await stale.pullAllBackingStores(); - await rival.put(counterDocument, KEY, { value: 7 }); - const seen: number[] = []; - await stale.update(counterDocument, KEY, (current) => { - seen.push(current!.value); - return { value: current!.value + 1 }; - }); - assert.deepEqual(seen, [7]); - assert.equal(stored(path), 8); - }); -}); - -test("refusals are not retried and apply is not called for a record it cannot copy", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - await cache.put(tallyDocument, "tally", { count: 1 }); - const before = await readFile(path); - let calls = 0; - const apply = (current: { value: number } | undefined) => { - calls++; - return { value: (current?.value ?? 0) + 1 }; - }; - // Another type at the key. - await assert.rejects(() => cache.update(counterDocument, "tally", apply), /not a "update.counter"/); - // A foreign record: a type this cache has no definition for. - const narrow = await open(path, defineDocumentRegistry(counterDocument)); - await assert.rejects(() => narrow.update(counterDocument, "tally", apply), /not a "update.counter"/); - assert.equal(calls, 0); - assert.deepEqual([...(await readFile(path))], [...before]); - - // A store without read and commitIf cannot take a conditional write. - const bare: CacheBackingStore = { pullAll: async () => [], push: async () => {}, delete: async () => {} }; - const plain = CultCache.builder().withRegistry(registry).withGenericStore(bare).build(); - await plain.pullAllBackingStores(); - await assert.rejects(() => plain.update(counterDocument, KEY, apply), /read and commitIf/); - assert.equal(calls, 0); - - // A record held read-only is refused by the judge after apply ran once; apply that throws is not retried either. - const surface = surfaceDocument(); - const entries = [envelopeOf(await read("extra-slot-record.msgpack"))]; - const readOnly = CultCache.builder().withDocumentType(surface).withGenericStore(memoryStore(entries)).build(); - await readOnly.pullAllBackingStores(); - let surfaceCalls = 0; - await assert.rejects(() => readOnly.update(surface, "surface", (value) => (surfaceCalls++, value)), SchemaConflictError); - assert.equal(surfaceCalls, 1); - assert.equal(entries.length, 1, "nothing reached the store"); - await assert.rejects( - () => cache.update(counterDocument, KEY, () => { calls++; throw new Error("boom"); }), - /boom/, - ); - assert.equal(calls, 1); - }); -}); - -test("own writes cause no retry, and updated returns the held instance", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - await cache.put(counterDocument, KEY, { value: 1 }); - let calls = 0; - const apply = (current: { value: number } | undefined) => { - calls++; - return { value: current!.value + 1 }; - }; - await cache.update(counterDocument, KEY, apply); - const second = await cache.update(counterDocument, KEY, apply); - assert.equal(calls, 2); - assert.equal(second.outcome, "updated"); - assert.equal(second.document, cache.get(counterDocument, KEY)); - assert.equal(stored(path), 3); - // The cache holds the record as the store wrote it, the stamp the store chose included: over a record stamped in the future - // the store writes one millisecond after it, whatever the clock says. - const future = readSingleFileStore(path).envelopes[0]!; - await new SingleFileMessagePackBackingStore(path).push({ ...future, payload: documentPayloadFromBytes(encode({ value: 99 })), storedAt: "2099-01-01T00:00:00.000Z" }); - await cache.update(counterDocument, KEY, apply); - assert.equal(readSingleFileStore(path).envelopes[0]!.storedAt, "2099-01-01T00:00:00.001Z"); - assert.equal(cache.getEnvelope(counterDocument, KEY)!.storedAt, "2099-01-01T00:00:00.001Z"); - }); -}); - -test("apply runs outside the cache's serial queue", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - let calls = 0; - const result = await cache.update(counterDocument, KEY, async (current) => { - calls++; - await Promise.race([ - cache.put(tallyDocument, "other", { count: 1 }), - new Promise((_, reject) => setTimeout(() => reject(new Error("a cache call from apply did not finish")), 5000).unref()), - ]); - return { value: (current?.value ?? 0) + 1 }; - }); - assert.equal(result.outcome, "updated"); - assert.equal(calls, 1); - }); -}); - -test("a cache with no store loses no increment under interleaved updates", async () => { - const cache = CultCache.builder().withRegistry(registry).build(); - await cache.pullAllBackingStores(); - let calls = 0; - const loop = async () => { - let updated = 0; - for (let i = 0; i < 50; i++) { - const result = await cache.update(counterDocument, KEY, async (current) => { - calls++; - await tick(); - return { value: (current?.value ?? 0) + 1 }; - }); - if (result.outcome === "updated") updated++; - } - return updated; - }; - // The two loops run in lockstep, so one may lose every round (a lostRace is the answer, not a lost write): what is held is - // exactly what the updated outcomes wrote. - const updated = (await Promise.all([loop(), loop()])).reduce((sum, n) => sum + n, 0); - assert.equal(cache.get(counterDocument, KEY)!.value, updated); - assert.ok(calls > updated, `the loops raced: apply ran ${calls} times for ${updated} writes`); -}); - -test("a store-less cache copies the bytes it admitted, not the held value", async () => { - const cache = CultCache.builder().withRegistry(registry).build(); - await cache.put(counterDocument, KEY, { value: 1 }); - cache.get(counterDocument, KEY)!.value = 999; - const seen: number[] = []; - await cache.update(counterDocument, KEY, (current) => (seen.push(current!.value), { value: current!.value + 1 })); - assert.deepEqual(seen, [1]); -}); - -test("updateGlobal updates the global record and refuses one loaded from a legacy key until putGlobal retires it", async () => { - await withDirectory(async (path) => { - const settings = defineDocumentRegistry(settingsDocument); - const cache = await open(path, settings); - const first = await cache.updateGlobal(settingsDocument, (current) => ({ theme: `${current?.theme ?? ""}a` })); - assert.equal(first.outcome, "updated"); - assert.deepEqual(cache.getGlobal(settingsDocument), { theme: "a" }); - assert.equal(readSingleFileStore(path).envelopes[0]!.key, CultCache.GLOBAL_KEY); - await rm(path); - - const legacy = await open(path, defineDocumentRegistry(plainSettingsDocument)); - await legacy.put(plainSettingsDocument, "old-settings", { theme: "x" }); - const loaded = await open(path, settings); - let calls = 0; - await assert.rejects(() => loaded.updateGlobal(settingsDocument, (current) => (calls++, current)), /putGlobal to retire its legacy key/); - assert.equal(calls, 0); - // Every path to the write is refused, the global's own key included, and the legacy record survives. - const before = await readFile(path); - await assert.rejects(() => loaded.update(settingsDocument, CultCache.GLOBAL_KEY, () => ({ theme: "lost" })), /putGlobal to retire its legacy key/); - assert.deepEqual([...(await readFile(path))], [...before]); - assert.deepEqual(readSingleFileStore(path).envelopes.map((e) => e.key), ["old-settings"]); - await loaded.putGlobal(settingsDocument, { theme: "y" }); - assert.equal((await loaded.updateGlobal(settingsDocument, (current) => ({ theme: `${current!.theme}z` }))).outcome, "updated"); - assert.deepEqual(loaded.getGlobal(settingsDocument), { theme: "yz" }); - await assert.rejects(() => cache.update(settingsDocument, "not-global", () => ({ theme: "q" })), /must use key/); - }); -}); - -test("a legacy global that a pull loads while apply awaits is refused at the write, and its legacy record survives", async () => { - await withDirectory(async (path) => { - const settings = defineDocumentRegistry(settingsDocument); - const cache = await open(path, settings); - const legacy = await open(path, defineDocumentRegistry(plainSettingsDocument)); - await assert.rejects( - () => - cache.update(settingsDocument, CultCache.GLOBAL_KEY, async () => { - await legacy.put(plainSettingsDocument, "old-settings", { theme: "x" }); - await cache.pullAllBackingStores(); - return { theme: "lost" }; - }), - /putGlobal to retire its legacy key/, - ); - assert.deepEqual(readSingleFileStore(path).envelopes.map((e) => e.key), ["old-settings"]); - }); -}); - -test("the ledger holds under contention: the stored value is the number of updated outcomes", async () => { - await withDirectory(async (path) => { - const caches = await Promise.all([open(path), open(path), open(path)]); - const results = await Promise.all( - caches.map(async (cache) => { - let updated = 0; - for (let i = 0; i < 25; i++) { - const result = await cache.update(counterDocument, KEY, async (current) => { - await tick(); - return { value: (current?.value ?? 0) + 1 }; - }); - if (result.outcome === "updated") updated++; - } - return updated; - }), - ); - assert.equal(stored(path), results.reduce((sum, n) => sum + n, 0)); - }); -}); - -test("a flush of an unchanged cache rewrites the file byte for byte, and an update does not lose a race to it", async () => { - await withDirectory(async (path) => { - const cache = await open(path); - await cache.put(counterDocument, KEY, { value: 1 }); - await cache.put(counterDocument, "other", { value: 7 }); - const store = new SingleFileMessagePackBackingStore(path); - const before = await readFile(path); - // What CultMesh.flush does: push the whole stored image, twice. - await store.pushAll(await store.pullAll()); - await store.pushAll(await store.pullAll()); - assert.deepEqual(await readFile(path), before); - const result = await cache.update(counterDocument, KEY, async (current) => { - await store.pushAll(await store.pullAll()); - return { value: current!.value + 1 }; - }, { maxAttempts: 1 }); - assert.equal(result.outcome, "updated"); - assert.equal(stored(path), 2); - }); -}); - -test("a changed record is re-stamped later than the one it replaces, and a revert to earlier bytes too", async () => { - await withDirectory(async (path) => { - const store = new SingleFileMessagePackBackingStore(path); - const cache = await open(path); - await cache.put(counterDocument, KEY, { value: 1 }); - const first = readSingleFileStore(path).envelopes[0]!; - const same = await store.push({ ...first, storedAt: "2099-01-01T00:00:00.000Z" }); - assert.equal(same.storedAt, first.storedAt, "identical content keeps its stamp"); - await cache.put(counterDocument, KEY, { value: 2 }); - const second = readSingleFileStore(path).envelopes[0]!; - const reverted = await store.push({ ...first, storedAt: second.storedAt }); - assert.ok(Date.parse(reverted.storedAt) > Date.parse(second.storedAt)); - }); -}); - -test("a push result that is not a parseable stamp is ignored", async () => { - for (const result of [true, {}, "", "garbage", { storedAt: "" }, { storedAt: "garbage" }]) { - const entries: Parameters[0] = []; - const store = { ...memoryStore(entries), push: async (entry: (typeof entries)[number]) => { entries.push(entry); return result as never; } }; - const cache = CultCache.builder().withRegistry(registry).withGenericStore(store).build(); - await cache.put(counterDocument, KEY, { value: 1 }); - const held = cache.getEnvelope(counterDocument, KEY)!.storedAt; - assert.equal(Number.isNaN(Date.parse(held)), false, `held stamp ${held} after a push result of ${JSON.stringify(result)}`); - } -}); - -test("a rewrite of identical content keeps the stamp but writes the writer's type and catalog entry", async () => { - await withDirectory(async (path) => { - const old = defineDocumentType({ type: "update.rn.old", schemaId: "update.rn.v1", schemaName: "update.rn.old", schemaVersion: "update.rn.v1", schema: z.object({ value: z.number().int() }) }); - const renamed = defineDocumentType({ type: "update.rn.new", schemaId: "update.rn.v1", schemaName: "update.rn.new", schemaVersion: "update.rn.v1", schema: z.object({ value: z.number().int() }) }); - await (await open(path, defineDocumentRegistry(old))).put(old, KEY, { value: 1 }); - const before = readSingleFileStore(path).envelopes[0]!; - await (await open(path, defineDocumentRegistry(renamed))).put(renamed, KEY, { value: 1 }); - const after = readSingleFileStore(path).envelopes[0]!; - assert.equal(after.storedAt, before.storedAt, "identical content keeps its stamp"); - assert.equal(after.type, "update.rn.new", "the writer's type is what the file says"); - }); -}); - -test("a rewrite of the same bytes under another schema id is a changed record: stamped later, written under the new id", async () => { - await withDirectory(async (path) => { - const v1 = defineDocumentType({ type: "update.sid", schemaId: "update.sid.v1", schemaName: "update.sid", schemaVersion: "update.sid.v1", schema: z.object({ value: z.number().int() }) }); - const v2 = defineDocumentType({ type: "update.sid", schemaId: "update.sid.v2", schemaName: "update.sid", schemaVersion: "update.sid.v2", compatibleSchemaIds: ["update.sid.v1"], schema: z.object({ value: z.number().int() }) }); - await (await open(path, defineDocumentRegistry(v1))).put(v1, KEY, { value: 1 }); - const before = readSingleFileStore(path).envelopes[0]!; - await (await open(path, defineDocumentRegistry(v2))).put(v2, KEY, { value: 1 }); - const after = readSingleFileStore(path).envelopes[0]!; - assert.ok(Date.parse(after.storedAt) > Date.parse(before.storedAt)); - assert.equal(schemaIdForEnvelope(after), "update.sid.v2"); - }); -}); diff --git a/packages/cultmath/README.md b/packages/cultmath/README.md index 6760a1f5..b144c56d 100644 --- a/packages/cultmath/README.md +++ b/packages/cultmath/README.md @@ -147,8 +147,8 @@ CultMath names with tests instead of copied project-local helper drift. ```powershell dotnet build packages/cultmath/src/CultMath/CultMath.csproj dotnet test packages/cultmath/tests/CultMath.Tests/CultMath.Tests.csproj -cargo test --locked --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml -cargo build --locked --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml +cargo test --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml +cargo build --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml .\packages\cultmath\scripts\build-unity-package.ps1 ``` diff --git a/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts b/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts index 14d9e443..092d5ffb 100644 --- a/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts +++ b/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts @@ -122,7 +122,7 @@ const rust: Target = { name: "rust", payload: (text) => encode({ text }), async start() { - if (!prebuilt) await execFileAsync(cargo, ["build", "--locked", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir }); + if (!prebuilt) await execFileAsync(cargo, ["build", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir }); const metadata = spawnSync(cargo, ["metadata", "--format-version", "1", "--no-deps"], { cwd: rustManifestDir, encoding: "utf8" }); if (metadata.status !== 0) throw new Error(`cargo metadata failed: ${metadata.stderr}`); const exe = join(JSON.parse(metadata.stdout).target_directory, "debug", "examples", `receipt_interop_server${process.platform === "win32" ? ".exe" : ""}`); diff --git a/packages/cultnet-rs/.gitignore b/packages/cultnet-rs/.gitignore index ea8c4bf7..96ef6c0b 100644 --- a/packages/cultnet-rs/.gitignore +++ b/packages/cultnet-rs/.gitignore @@ -1 +1,2 @@ /target +Cargo.lock diff --git a/packages/cultnet-rs/Cargo.lock b/packages/cultnet-rs/Cargo.lock deleted file mode 100644 index ed0aab43..00000000 --- a/packages/cultnet-rs/Cargo.lock +++ /dev/null @@ -1,1227 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aead" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" -dependencies = [ - "crypto-common", - "generic-array", -] - -[[package]] -name = "aes" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" -dependencies = [ - "cfg-if", - "cipher", - "cpufeatures", -] - -[[package]] -name = "aes-gcm" -version = "0.10.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" -dependencies = [ - "aead", - "aes", - "cipher", - "ctr", - "ghash", - "subtle", -] - -[[package]] -name = "android_system_properties" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" -dependencies = [ - "libc", -] - -[[package]] -name = "anyhow" -version = "1.0.104" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" - -[[package]] -name = "ar_archive_writer" -version = "0.5.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "523e5e187fff36fcb8d1d19ccf0ca4ca43ec3d158316057f66dc6f72716bc5ec" -dependencies = [ - "object", -] - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "base64" -version = "0.22.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" - -[[package]] -name = "base64ct" -version = "1.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" - -[[package]] -name = "bitflags" -version = "2.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" - -[[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - -[[package]] -name = "bumpalo" -version = "3.20.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" - -[[package]] -name = "cc" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ff2a8126fff0f8c839080d619675a585e61720bb12d2baceae45769fd66f155" -dependencies = [ - "find-msvc-tools", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" - -[[package]] -name = "chrono" -version = "0.4.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" -dependencies = [ - "iana-time-zone", - "js-sys", - "num-traits", - "serde", - "wasm-bindgen", - "windows-link", -] - -[[package]] -name = "cipher" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" -dependencies = [ - "crypto-common", - "inout", -] - -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "cpufeatures" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" -dependencies = [ - "libc", -] - -[[package]] -name = "crypto-common" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" -dependencies = [ - "generic-array", - "rand_core 0.6.4", - "typenum", -] - -[[package]] -name = "ctr" -version = "0.9.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" -dependencies = [ - "cipher", -] - -[[package]] -name = "cultcache-rs" -version = "0.3.1" -dependencies = [ - "anyhow", - "chrono", - "cultcache-rs-derive", - "fs2", - "redb", - "rmp-serde", - "serde", - "serde_bytes", - "serde_json", - "serde_stacker", - "stacker", - "uuid", - "windows-sys 0.59.0", -] - -[[package]] -name = "cultcache-rs-derive" -version = "0.2.0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "cultnet-rs" -version = "0.3.0" -dependencies = [ - "aes-gcm", - "anyhow", - "base64", - "chrono", - "cultcache-rs", - "ed25519-dalek", - "fs2", - "hmac", - "libc", - "pretty_assertions", - "rand", - "rmp-serde", - "rmpv", - "serde", - "serde_bytes", - "serde_json", - "sha2", - "socket2", - "tempfile", - "uuid", - "windows-sys 0.61.2", - "zeroize", -] - -[[package]] -name = "curve25519-dalek" -version = "4.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" -dependencies = [ - "cfg-if", - "cpufeatures", - "curve25519-dalek-derive", - "digest", - "fiat-crypto", - "rustc_version", - "subtle", - "zeroize", -] - -[[package]] -name = "curve25519-dalek-derive" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "der" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" -dependencies = [ - "const-oid", - "zeroize", -] - -[[package]] -name = "diff" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8" - -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer", - "crypto-common", - "subtle", -] - -[[package]] -name = "ed25519" -version = "2.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" -dependencies = [ - "pkcs8", - "signature", -] - -[[package]] -name = "ed25519-dalek" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" -dependencies = [ - "curve25519-dalek", - "ed25519", - "serde", - "sha2", - "subtle", - "zeroize", -] - -[[package]] -name = "errno" -version = "0.3.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776" -dependencies = [ - "libc", - "windows-sys 0.61.2", -] - -[[package]] -name = "fastrand" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" - -[[package]] -name = "fiat-crypto" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" - -[[package]] -name = "find-msvc-tools" -version = "0.1.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48" - -[[package]] -name = "fs2" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" -dependencies = [ - "libc", - "winapi", -] - -[[package]] -name = "futures-core" -version = "0.3.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" - -[[package]] -name = "futures-task" -version = "0.3.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" - -[[package]] -name = "futures-util" -version = "0.3.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" -dependencies = [ - "futures-core", - "futures-task", - "pin-project-lite", - "slab", -] - -[[package]] -name = "generic-array" -version = "0.14.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" -dependencies = [ - "typenum", - "version_check", -] - -[[package]] -name = "getrandom" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" -dependencies = [ - "cfg-if", - "libc", - "wasi", -] - -[[package]] -name = "getrandom" -version = "0.3.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" -dependencies = [ - "cfg-if", - "libc", - "r-efi 5.3.0", - "wasip2", -] - -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi 6.0.0", -] - -[[package]] -name = "ghash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" -dependencies = [ - "opaque-debug", - "polyval", -] - -[[package]] -name = "hmac" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" -dependencies = [ - "digest", -] - -[[package]] -name = "iana-time-zone" -version = "0.1.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "log", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "inout" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" -dependencies = [ - "generic-array", -] - -[[package]] -name = "itoa" -version = "1.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4" - -[[package]] -name = "js-sys" -version = "0.3.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a" -dependencies = [ - "cfg-if", - "futures-util", - "wasm-bindgen", -] - -[[package]] -name = "libc" -version = "0.2.191" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a" - -[[package]] -name = "linux-raw-sys" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" - -[[package]] -name = "log" -version = "0.4.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d15f86d12da44cd8d4ec3c73984537eb7f4cadcb9776a575a0a8a05ca11005c" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "object" -version = "0.39.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" -dependencies = [ - "memchr", -] - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "opaque-debug" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "pkcs8" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "spki", -] - -[[package]] -name = "polyval" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" -dependencies = [ - "cfg-if", - "cpufeatures", - "opaque-debug", - "universal-hash", -] - -[[package]] -name = "ppv-lite86" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" -dependencies = [ - "zerocopy", -] - -[[package]] -name = "pretty_assertions" -version = "1.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d" -dependencies = [ - "diff", - "yansi", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "psm" -version = "0.1.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622" -dependencies = [ - "ar_archive_writer", - "cc", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "5.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "rand" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" -dependencies = [ - "rand_chacha", - "rand_core 0.9.5", -] - -[[package]] -name = "rand_chacha" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" -dependencies = [ - "ppv-lite86", - "rand_core 0.9.5", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom 0.2.17", -] - -[[package]] -name = "rand_core" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" -dependencies = [ - "getrandom 0.3.4", -] - -[[package]] -name = "redb" -version = "4.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b317ef681e18f6091efbfab532eaf924a135c492d9d0299d0fbfae07e92a39b" -dependencies = [ - "libc", -] - -[[package]] -name = "rmp" -version = "0.8.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c" -dependencies = [ - "num-traits", -] - -[[package]] -name = "rmp-serde" -version = "1.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155" -dependencies = [ - "rmp", - "serde", -] - -[[package]] -name = "rmpv" -version = "1.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a4e1d4b9b938a26d2996af33229f0ca0956c652c1375067f0b45291c1df8417" -dependencies = [ - "rmp", - "serde", - "serde_bytes", -] - -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - -[[package]] -name = "rustix" -version = "1.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa" -dependencies = [ - "bitflags", - "errno", - "libc", - "linux-raw-sys", - "windows-sys 0.61.2", -] - -[[package]] -name = "rustversion" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" - -[[package]] -name = "semver" -version = "1.0.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" - -[[package]] -name = "serde" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_bytes" -version = "0.11.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" -dependencies = [ - "serde", - "serde_core", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.10", -] - -[[package]] -name = "serde_json" -version = "1.0.154" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_stacker" -version = "0.1.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4936375d50c4be7eff22293a9344f8e46f323ed2b3c243e52f89138d9bb0f4a" -dependencies = [ - "serde", - "serde_core", - "stacker", -] - -[[package]] -name = "sha2" -version = "0.10.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" -dependencies = [ - "cfg-if", - "cpufeatures", - "digest", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "signature" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" -dependencies = [ - "rand_core 0.6.4", -] - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "socket2" -version = "0.5.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678" -dependencies = [ - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "spki" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" -dependencies = [ - "base64ct", - "der", -] - -[[package]] -name = "stacker" -version = "0.1.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967" -dependencies = [ - "cc", - "cfg-if", - "libc", - "psm", - "windows-sys 0.61.2", -] - -[[package]] -name = "subtle" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" - -[[package]] -name = "syn" -version = "2.0.119" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "3.0.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ae9ccbea3a7b0b09443036490c8f34a8b1b760fd345e5c0dae0aac718faea79" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "tempfile" -version = "3.27.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" -dependencies = [ - "fastrand", - "getrandom 0.4.3", - "once_cell", - "rustix", - "windows-sys 0.61.2", -] - -[[package]] -name = "typenum" -version = "1.20.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" - -[[package]] -name = "unicode-ident" -version = "1.0.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" - -[[package]] -name = "universal-hash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" -dependencies = [ - "crypto-common", - "subtle", -] - -[[package]] -name = "uuid" -version = "1.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bed99139f7dac9b5cfe5f17be4d66dc7079a34dee1d74bcc77217119240e267c" -dependencies = [ - "getrandom 0.4.3", - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - -[[package]] -name = "wasi" -version = "0.11.1+wasi-snapshot-preview1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" - -[[package]] -name = "wasip2" -version = "1.0.4+wasi-0.2.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" -dependencies = [ - "wit-bindgen", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c" -dependencies = [ - "cfg-if", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn 3.0.10", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.130" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-link", - "windows-result", - "windows-strings", -] - -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-sys" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.59.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-targets" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" -dependencies = [ - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" - -[[package]] -name = "windows_i686_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" - -[[package]] -name = "windows_i686_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" - -[[package]] -name = "wit-bindgen" -version = "0.57.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" - -[[package]] -name = "yansi" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" - -[[package]] -name = "zerocopy" -version = "0.8.63" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5fe1f8f1b06191a00962174c61aa5005e0bb391a6d80d07e24d115c01a92ed8" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.63" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "863ad3ac83293fb4d740aedbfdc9240dd8d1a50c1099acd76ce80ce7c7230c7f" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "zeroize" -version = "1.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879" - -[[package]] -name = "zmij" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/packages/cultnet-rs/README.md b/packages/cultnet-rs/README.md index 69a8e70d..c356b97b 100644 --- a/packages/cultnet-rs/README.md +++ b/packages/cultnet-rs/README.md @@ -30,7 +30,7 @@ knock on the contract before touching the furniture. ## Receipts ```powershell -cargo test --locked +cargo test ``` The initial tests prove: diff --git a/packages/cultnet-rs/examples/cultnet_impair/main.rs b/packages/cultnet-rs/examples/cultnet_impair/main.rs index 35a2168d..c2ebd874 100644 --- a/packages/cultnet-rs/examples/cultnet_impair/main.rs +++ b/packages/cultnet-rs/examples/cultnet_impair/main.rs @@ -7,7 +7,7 @@ //! traffic crosses the same schedule. The schedule is `policy.rs`. //! //! ```text -//! cargo run --locked --release --example cultnet_impair -- \ +//! cargo run --release --example cultnet_impair -- \ //! --listen 0.0.0.0:17890 --upstream 127.0.0.1:17990 \ //! --profile tests/impairment/loss-1pct.toml --seed 424242 --metrics loss-1pct.csv //! ``` diff --git a/packages/cultnet-rs/tests/media_fec_bench.rs b/packages/cultnet-rs/tests/media_fec_bench.rs index f0279495..40c9de78 100644 --- a/packages/cultnet-rs/tests/media_fec_bench.rs +++ b/packages/cultnet-rs/tests/media_fec_bench.rs @@ -3,7 +3,7 @@ //! Ignored by default; run in release, where the numbers mean something: //! //! ```text -//! cargo test --locked --release --test media_fec_bench -- --ignored --nocapture +//! cargo test --release --test media_fec_bench -- --ignored --nocapture //! ``` //! //! Each line is nanoseconds per call through the public surface, so record diff --git a/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md b/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md index 86dd821e..7856c78d 100644 --- a/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md +++ b/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md @@ -6,7 +6,7 @@ cultnet-rs only, and Idunn's `IdunnRuntimeActivationLaunch` is that same crate's It needs Linux and a machine-id equal to the `machine-id` file here, because the provider identity is bound to it. On Yggdrasil: - DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh HEAD rust 'cd packages/cultnet-rs && cp /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run --locked -q --example idunn_runtime_fixture /tmp/out' + DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh HEAD rust 'cd packages/cultnet-rs && cp /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run -q --example idunn_runtime_fixture /tmp/out' then copy `/tmp/out` over this directory. `--cap-add SYS_ADMIN` exists only so the bind mount can present the committed `machine-id`, which keeps Yggdrasil's real machine-id out of the repo. Every `presence-*.bin` was accepted by `authenticate_runtime_presence_claim` and correlated by @@ -26,7 +26,7 @@ The generator takes its specification as input, so a consumer needs no patch to specification is command-line flags; `@` splices in a response file (one `--flag value` per line, `#` comments). CultLib's own sets are `fixture.args` here: - cargo run --locked -q --example idunn_runtime_fixture /tmp/out @/fixture.args + cargo run -q --example idunn_runtime_fixture /tmp/out @/fixture.args The consumer keeps its own `.args` file in its repository and runs the same command with the same machine-id bind mount as above. Global flags: `--source-repository`, `--detail`. Each diff --git a/packages/cultnet-ts/test/interop/cultnet-interop.test.ts b/packages/cultnet-ts/test/interop/cultnet-interop.test.ts index 3b8b7bc9..d0e732ce 100644 --- a/packages/cultnet-ts/test/interop/cultnet-interop.test.ts +++ b/packages/cultnet-ts/test/interop/cultnet-interop.test.ts @@ -2795,7 +2795,7 @@ async function buildInteropPeers(): Promise { } async function buildRustInteropPeer(): Promise { - rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--locked", "--quiet", "--example", "cultnet_interop_peer"], { + rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--quiet", "--example", "cultnet_interop_peer"], { cwd: cultnetRsRoot, }).then(() => undefined); await rustInteropPeerBuild; diff --git a/packages/cultnet-ts/test/runtime-presence-health.test.ts b/packages/cultnet-ts/test/runtime-presence-health.test.ts index 1d7c4fa6..2dc4a845 100644 --- a/packages/cultnet-ts/test/runtime-presence-health.test.ts +++ b/packages/cultnet-ts/test/runtime-presence-health.test.ts @@ -2,7 +2,7 @@ // // The vectors below were produced by `cultnet-rs`, not written by hand: // -// cargo run --locked --example runtime_presence_vector -p cultnet-rs +// cargo run --example runtime_presence_vector -p cultnet-rs // // That matters. A fixture authored for both sides at once agrees with itself // and proves nothing; the contract is only real if one fork emits the bytes and diff --git a/packages/heimdall-consent/README.md b/packages/heimdall-consent/README.md index 926b7d1a..3782e720 100644 --- a/packages/heimdall-consent/README.md +++ b/packages/heimdall-consent/README.md @@ -7,7 +7,7 @@ plus the rules that need no store to state (`Answer::is_training_data`, `Answer: (comments 682 and 759). Specification only: no collector, tally, export or tombstone hashing lives here. Entry points: `src/lib.rs`; golden vectors `tests/vectors/consent_golden.csv`, run by `tests/golden.rs` -(`cargo test --locked`). Planned, not live: the collector, the tally, the training export. +(`cargo test`). Planned, not live: the collector, the tally, the training export. ## Where each ruling lands (doc lines at aaeb175) diff --git a/packages/heimdall-stake/README.md b/packages/heimdall-stake/README.md index 9b6acb19..5c8ae095 100644 --- a/packages/heimdall-stake/README.md +++ b/packages/heimdall-stake/README.md @@ -11,8 +11,8 @@ adversary, storage, and any projection. Every parameter is a placeholder until t ## First call ``` -cargo run --locked --example first_call # prints the Meta example and a splitting table -cargo test --locked +cargo run --example first_call # prints the Meta example and a splitting table +cargo test ``` ```rust diff --git a/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs b/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs index 6e0ce6cd..f916a13b 100644 --- a/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs +++ b/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs @@ -64,7 +64,48 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore var reports = new List(); var loaded = new Dictionary(StringComparer.Ordinal); var foreign = new List(); - LoadManifestPages(null, loaded, reports, foreign, Trace); + for (var attempt = 1; ; attempt++) + { + loaded.Clear(); + reports.Clear(); + foreign.Clear(); + // Only pages named by the manifest read under the lease are loaded; orphaned pages are never loaded. With no + // lock to lease (none created yet, or a store copied without it) a writer may commit mid-load, so an unleased + // load stands only if its manifest is still current; if the manifest moved it reloads, under the lease the + // writer created. A page that fails under an unmoved manifest is corruption, not a race. + using var lease = AcquireCommitLease(wait: true, create: false); + var manifestBytes = ReadManifestBytes(); + var (manifest, _) = ParseManifest(manifestBytes); + Trace($"manifest records={manifest.Records.Length}"); + try + { + if (lease == null) + UnleasedManifestRead?.Invoke(); + LoadRecordPages( + manifest.Records.OrderBy(record => record.Key, StringComparer.Ordinal).ToArray(), + manifest.SchemaCatalog, + loaded, + reports, + foreign); + if (lease != null || !ManifestMoved(manifestBytes)) + { + Trace($"indexed-pages loaded={loaded.Count}"); + break; + } + } + catch (Exception exception) when (lease == null && IsTornGeneration(exception)) + { + // The re-read stays out of the filter: an exception thrown inside a filter is swallowed as false. + if (!ManifestMoved(manifestBytes)) + throw; + if (attempt == UnleasedLoadAttempts) + throw Unsettled(exception); + continue; + } + + if (attempt == UnleasedLoadAttempts) + throw Unsettled(null); + } bool Staged(string key) => _dirtyKeys.ContainsKey(key) || _deletedKeys.ContainsKey(key); var arrived = loaded @@ -95,70 +136,6 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore Trace("publish"); } - // The one load of manifest pages, for the whole store (key null) or one record: under the lease, with the moved-manifest retry. - // It reads only; adopting what it loaded is the caller's. - private void LoadManifestPages( - string? key, - Dictionary loaded, - List reports, - List foreign, - Action trace) - { - for (var attempt = 1; ; attempt++) - { - loaded.Clear(); - reports.Clear(); - foreign.Clear(); - // Only pages named by the manifest read under the lease are loaded; orphaned pages are never loaded. With no - // lock to lease (none created yet, or a store copied without it) a writer may commit mid-load, so an unleased - // load stands only if its manifest is still current; if the manifest moved it reloads, under the lease the - // writer created. A page that fails under an unmoved manifest is corruption, not a race. - using var lease = AcquireCommitLease(wait: true, create: false); - var manifestBytes = ReadManifestBytes(); - var (manifest, _) = ParseManifest(manifestBytes); - trace($"manifest records={manifest.Records.Length}"); - try - { - if (lease == null) - UnleasedManifestRead?.Invoke(); - LoadRecordPages( - manifest.Records.Where(record => key == null || string.Equals(record.Key, key, StringComparison.Ordinal)) - .OrderBy(record => record.Key, StringComparer.Ordinal).ToArray(), - manifest.SchemaCatalog, - loaded, - reports, - foreign); - if (lease != null || !ManifestMoved(manifestBytes)) - { - trace($"indexed-pages loaded={loaded.Count}"); - break; - } - } - catch (Exception exception) when (lease == null && IsTornGeneration(exception)) - { - // The re-read stays out of the filter: an exception thrown inside a filter is swallowed as false. - if (!ManifestMoved(manifestBytes)) - throw; - if (attempt == UnleasedLoadAttempts) - throw Unsettled(exception); - continue; - } - - if (attempt == UnleasedLoadAttempts) - throw Unsettled(null); - } - } - - // The record at a key as its page holds it now, decoded fresh and stamped with VersionOf its manifest payload, the version - // CommitBatch compares. Null when none is there or it is foreign. Touches no entry, hydration, report or foreign list of the - // store, and raises nothing. - internal override CultStoredDocument? ReadDurable(string key) => Held(() => - { - var loaded = new Dictionary(StringComparer.Ordinal); - LoadManifestPages(key, loaded, new List(), new List(), _ => { }); - return loaded.TryGetValue(key, out var stored) ? stored : null; - }); - // Variants wait for the directory store's own cut; until then it refuses one rather than persist a resolved view as a plain page. private static void RefuseVariant(CultStoredDocument entry) { @@ -260,7 +237,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore }); } - public override void PushAll(bool soft) + public override void PushAll() { ThrowIfReadOnly(); Held(() => @@ -270,12 +247,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore Directory.CreateDirectory(_manifestFile.DirectoryName!); using var commitLease = AcquireCommitLease(wait: true, create: true); - var manifestBytes = ReadManifestBytes(); - var (manifest, catalogBytes) = ParseManifest(manifestBytes); - // A soft flush writes only where no manifest is: the reader's answer (null is absent), not a second look at the path. - // A manifest it read stays as it is, its staged writes staged. - if (soft && manifestBytes != null) - return; + var (manifest, catalogBytes) = ReadManifest(); RefuseUndeclaredOnDisk(manifest, StagedKeys, Array.Empty(), Array.Empty(), Array.Empty()); if (IsDirty) WriteGeneration(manifest, catalogBytes); @@ -709,7 +681,14 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore stream.Flush(true); } - StoreReplace.Replace(tempPath, path); + if (File.Exists(path)) + { + ReplaceExistingFile(tempPath, path); + } + else + { + File.Move(tempPath, path); + } } finally { @@ -789,4 +768,26 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore return held; } } + + private static void ReplaceExistingFile(string sourcePath, string destinationPath) + { + for (var attempt = 0; attempt < 3; attempt++) + { + try + { + File.Replace(sourcePath, destinationPath, null); + return; + } + catch (IOException) when (attempt < 2) + { + Thread.Sleep(10 * (attempt + 1)); + } + catch (UnauthorizedAccessException) when (attempt < 2) + { + Thread.Sleep(10 * (attempt + 1)); + } + } + + File.Replace(sourcePath, destinationPath, null); + } } diff --git a/src/GameCult.Caching/CultCache.cs b/src/GameCult.Caching/CultCache.cs index 303d3737..c461bc67 100644 --- a/src/GameCult.Caching/CultCache.cs +++ b/src/GameCult.Caching/CultCache.cs @@ -2052,29 +2052,6 @@ namespace GameCult.Caching Contended } - public enum CultUpdateOutcome - { - Updated, - Declined, - LostRace - } - - /// - /// What did. A lost race is a value, not an exception. is the instance the - /// cache holds at the key when Update returns: the written one for Updated, null when the key is absent. - /// - public readonly struct CultUpdateResult where T : class - { - public CultUpdateResult(CultUpdateOutcome outcome, T? document) - { - Outcome = outcome; - Document = document; - } - - public CultUpdateOutcome Outcome { get; } - public T? Document { get; } - } - public sealed class CultCommitRequest { internal CultCommitRequest( @@ -2560,24 +2537,19 @@ namespace GameCult.Caching return Task.CompletedTask; } - public void FlushAllBackingStores(bool soft = false) => Held(() => + public void FlushAllBackingStores() => Held(() => { foreach (var (store, _) in _stores) { if (!store.IsReadOnly && (store.IsDirty || store.HasStagedChecks)) - store.PushAll(soft); + store.PushAll(); } return true; }); - /// - /// Flushes every dirty backing store. With the flush has one meaning, decided by each store's - /// own read: a store with nothing at its path is written; a store that reads (empty included) is skipped and its - /// staged writes stay staged; a store that cannot be read refuses with . - /// - public Task FlushAsync(bool soft = false) + public Task FlushAsync() { - FlushAllBackingStores(soft); + FlushAllBackingStores(); return Task.CompletedTask; } @@ -2586,7 +2558,6 @@ namespace GameCult.Caching return UpsertAsync(document, handle); } - // A blind write, last-writer-wins per key; Update is the conditional golden path. // Refused with CultSchemaConflictException when the key holds a record stored under an id no registered type declares. public Task> UpsertAsync(T document, CultRecordHandle? handle = null) { @@ -2663,7 +2634,6 @@ namespace GameCult.Caching /// /// Commits the staged batch. Returns false when a condition failed: nothing was written, changed in memory, or published. - /// Without Expect a commit is blind; retries a conditional one. /// /// /// A commit that succeeds and then fails in a journal or observer still stands and is published, and the writer @@ -2681,93 +2651,6 @@ namespace GameCult.Caching return Land(stage, wait: false); } - /// - /// Reads the record at as its home store holds it, applies to a fresh copy of it, and - /// commits the result only if the record is still the one that was read; when another writer got there first it re-reads and runs - /// again, up to times. The write is durable when Update returns. - /// - /// - /// may run up to times. It gets the record as the store that holds it has it now, read from every home - /// of T and its routed subtypes and decoded afresh for this attempt (null when the key holds nothing), which it may change and return. A record held by two stores, or an apply that changes the record's home store, throws. A change made in place to an - /// instance the cache holds is neither seen nor written. It must not write to the cache or have other side effects. Return null to write - /// nothing. A store with staged blind writes refuses Update until it is flushed, as it refuses any conditional commit. A cache with no - /// store attached copies its held instance and needs a codec; a store kind outside the library cannot serve Update (). Running out of attempts returns ; every other failure throws. - /// - public CultUpdateResult Update(CultRecordKey key, Func apply, int maxAttempts = 5) where T : class - { - if (apply == null) throw new ArgumentNullException(nameof(apply)); - if (maxAttempts < 1) throw new ArgumentOutOfRangeException(nameof(maxAttempts), maxAttempts, "At least one attempt is required."); - for (var attempt = 1; ; attempt++) - { - var (version, copy, source) = Held(() => - { - if (_stores.Count == 0) - { - // No store: the held entry is the record. - _entries.TryGetValue(key.Value, out var held); - var codec = _codec ?? throw new InvalidOperationException("This cache has no codec; create it with CultCacheMessagePack.Create."); - if (held == null) - return ((string?)null, (T?)null, (CacheBackingStore?)null); - if (held.Variant != null || held.Document is not T) - throw new InvalidOperationException($"Record {key.Value} is a variant or not a {typeof(T).Name}; Update cannot copy it."); - return (held.StoredVersion, (T?)codec.Clone(held.Document, held.Descriptor.DocumentType), null); - } - - // The store with the key is among the homes of T and its routed subtypes; the cache's own view of the key may be stale, so it is not asked. - var homes = CandidateHomes(typeof(T)); - if (homes.Length == 0) - throw new InvalidOperationException("A conditional batch names its home store through a record it upserts or removes."); - CacheBackingStore? holder = null; - CultStoredDocument? durable = null; - foreach (var candidate in homes) - { - if (candidate.ReadDurable(key.Value) is not { } found) - continue; - if (holder != null) - throw new InvalidOperationException($"Record {key.Value} is held by two stores, {holder} and {candidate}; Update cannot choose."); - holder = candidate; - durable = found; - } - - if (durable == null) - return ((string?)null, (T?)null, (CacheBackingStore?)null); - if (durable.Variant != null || durable.Document is not T typed) - throw new InvalidOperationException($"Record {key.Value} is a variant or not a {typeof(T).Name}; Update cannot copy it."); - CultElementIds.Plan(durable.Descriptor.DocumentType, durable.Document, key.Value, deterministic: true).Apply(); - return (durable.StoredVersion, typed, holder); - }); - var next = apply(copy); - if (next == null) - return new CultUpdateResult(CultUpdateOutcome.Declined, Get(key)); - if (source != null && Held(() => Home(next.GetType())) is var landing && landing != source) - throw new InvalidOperationException( - $"Update of {key.Value} would move the record from {source} to {landing?.ToString() ?? "memory"}; remove it and write it at its new home."); - CultCommitOutcome outcome; - try - { - outcome = Land(batch => - { - batch.Expected.Add((key, version)); - batch.Upsert(next, new CultRecordHandle(key)); - }, wait: true); - } - catch (CultWriteConflictException conflict) when (conflict.InnerException == null) - { - outcome = CultCommitOutcome.Mismatch; - } - - if (outcome == CultCommitOutcome.Committed) - return new CultUpdateResult(CultUpdateOutcome.Updated, next); - if (outcome != CultCommitOutcome.Mismatch) - throw new InvalidOperationException($"Update of {key.Value} was refused: {outcome}."); - if (attempt >= maxAttempts) - { - Held(() => { Home(next.GetType())?.PullAll(); return true; }); - return new CultUpdateResult(CultUpdateOutcome.LostRace, Get(key)); - } - } - } - public CultRecordHandle? TryGetHandle(T document) { if (document == null) throw new ArgumentNullException(nameof(document)); @@ -2788,8 +2671,7 @@ namespace GameCult.Caching /// when that store holds it unowned (undeclared drift that drops persisted slots, or a payload carrying slots past this build's, /// see ). For a key the cache does not hold, the removal check every store makes of it /// (false when a store lists it foreign). One write it does not predict: a Flatten of a variant whose base chain holds an unowned record is refused, though - /// the variant itself is writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it - /// with . Live and advisory: the write still decides. + /// the variant itself is writable. Live and advisory: the write still decides. /// public bool IsWritable(CultRecordKey key) { @@ -3785,18 +3667,6 @@ namespace GameCult.Caching return changes; } - // Every store a record of this type, or of a registered subtype of it, can live in. - private CacheBackingStore[] CandidateHomes(Type type) - { - var homes = new HashSet(); - if (Home(type) is { } own) - homes.Add(own); - foreach (var descriptor in _registry.AllDescriptors) - if (type.IsAssignableFrom(descriptor.DocumentType) && Home(descriptor.DocumentType) is { } routed) - homes.Add(routed); - return _stores.Select(entry => entry.Store).Where(homes.Contains).ToArray(); - } - // The most derived routed home type assignable from the document type, else the untyped store. private CacheBackingStore? Home(Type documentType) { @@ -4073,24 +3943,9 @@ namespace GameCult.Caching internal void UseRegistry(CultDocumentRegistry? registry) => _registry = registry; public abstract void PullAll(); - - /// - /// The record at as this store holds it now, decoded as a load decodes it by this call (a new instance nobody - /// else holds) and stamped with the version this store's commit compares. Null when the store holds nothing there or holds it foreign. - /// It changes none of the store's entries, staged writes or reports and raises nothing. Internal because only this library can stamp a version. - /// - internal virtual CultStoredDocument? ReadDurable(string key) => - throw new NotSupportedException($"{this} cannot read one record, which Update needs."); - public abstract void Push(CultStoredDocument entry); public abstract void Delete(CultStoredDocument entry); - public void PushAll() => PushAll(soft: false); - - /// - /// Applies the staged writes. A soft flush writes only where nothing is at the path: over a store it reads it returns without - /// writing and leaves the staged writes staged and dirty for a later plain flush, and over a file it cannot read it refuses. - /// - public abstract void PushAll(bool soft); + public abstract void PushAll(); /// /// One durable step under the store's exclusive lock: the batch and every staged write land onto the store as it is then, @@ -4329,7 +4184,7 @@ namespace GameCult.Caching /// /// /// A writable store kind that reads its durable state under a lock overrides this to call , and - /// judges at its next or as it judges + /// judges at its next or as it judges /// : under the lock, with the keys it writes, refused through the same refusal, never /// removed. It calls when that step succeeds, which clears them. A read-only store has no /// such step and keeps the default. Both shipped store kinds do this. A batch's own checks are judged through @@ -4806,7 +4661,7 @@ namespace GameCult.Caching // only per staged key. A key whose durable record is under an id this cache does not declare refuses the write, and in a store // holding a variant it lands only onto the store as this cache last read it (both ApplyWriteSet). // A file it cannot read (truncated, trailing bytes, not a store, a format it does not read) is refused and left as it is. - public override void PushAll(bool soft) + public override void PushAll() { ThrowIfReadOnly(); Held(() => @@ -4814,8 +4669,6 @@ namespace GameCult.Caching using (AcquireLock(wait: true)) { var disk = ReadSnapshot(); - if (soft && disk != null) - return; var (upserts, removals) = StagedWrites(Array.Empty(), Array.Empty()); Wrote(ApplyWriteSet(disk, upserts, removals, Array.Empty())); } @@ -5228,9 +5081,6 @@ namespace GameCult.Caching return ToStoredDocument(record, durable.Version(key), durable.Snapshot.SchemaCatalog, DeserializeMarked, out _); } - // Update's read: the durable record at the key, decoded afresh, stamped with the version CommitBatch compares. - internal override CultStoredDocument? ReadDurable(string key) => Held(() => DurableAt(ReadSnapshot(), key)); - // The file now holds every record written, under the id and at the storedAt the write gave it, and the version of those bytes: // the store's entries and the cache's say so. A copied record is on disk as the file had it, so nothing learns anything new about it: in particular the // ids it minted at load are still in memory only. @@ -5369,7 +5219,14 @@ namespace GameCult.Caching stream.Flush(true); } - StoreReplace.Replace(tempPath, path); + if (File.Exists(path)) + { + File.Replace(tempPath, path, null); + } + else + { + File.Move(tempPath, path); + } } finally { diff --git a/src/GameCult.Caching/CultInspectorModel.cs b/src/GameCult.Caching/CultInspectorModel.cs index 60dec974..7aa6ab74 100644 --- a/src/GameCult.Caching/CultInspectorModel.cs +++ b/src/GameCult.Caching/CultInspectorModel.cs @@ -352,7 +352,7 @@ namespace GameCult.Caching } } - public object Clone(object document, Type type) => _codec.Clone(document, type); + public object Clone(object document, Type type) => _deserialize(type, _serialize(document, type, type)); public CultInspectorEdit BeginEdit(CultStoredDocument record) { diff --git a/src/GameCult.Caching/CultVariants.cs b/src/GameCult.Caching/CultVariants.cs index e5c28bbf..48909c19 100644 --- a/src/GameCult.Caching/CultVariants.cs +++ b/src/GameCult.Caching/CultVariants.cs @@ -29,11 +29,6 @@ namespace GameCult.Caching // (document payload, slot and encoded value pairs) public Func>, byte[]> Overlay { get; } - // The one copy of a document: its encoding, decoded again. Update's private argument and CultInspectorModel.Clone share it. - internal object Clone(object document, Type type) => - Deserialize(type, type, Serialize(document, type, type)) - ?? throw new InvalidOperationException($"The codec returned null for {type.FullName}."); - /// /// Decodes a document payload read from a store file, as a cache load decodes it: the ids a load mints are minted from the key /// and their path, so every reader of one store mints the same ids, and a payload a cache write would refuse is refused here too. diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index dffce80a..6c76f5cd 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -13,7 +13,6 @@ Cacheable models are plain classes annotated with cache intent. using GameCult.Caching; using MessagePack; -[MessagePackObject] [CultDocument("gamecult.item_data", "gamecult.item_data.v1")] public sealed class ItemData { @@ -132,33 +131,18 @@ using GameCult.Caching; using GameCult.Caching.MessagePack; var cache = await CultCacheMessagePack.OpenAsync("Data.msgpack"); -var key = new CultRecordKey("potion"); -// The golden path: read, change a private copy, commit only if the record is unchanged, re-run on a lost race. -// Durable when it returns; no flush. -var result = cache.Update(key, item => -{ - item ??= new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable" }; - item.Value += 10; - return item; -}); - -// The blind opt-out: last writer wins per key. Name the key, or a new key claims the same unique Sku and is refused. var handle = await cache.UpsertAsync(new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable", Value = 50 -}, new CultRecordHandle(key)); +}); await cache.FlushAsync(); ``` -`Update` may run your function more than once, so it must have no side effects. Returning null writes nothing, and a -lost race that outlasts `maxAttempts` comes back as `CultUpdateOutcome.LostRace`. `UpsertAsync` plus `FlushAsync` never -checks what it overwrites. - If you want the explicit assembly rite, the lower-level `CultCache` + `SingleFileMessagePackBackingStore` path is still there. The helper is just the blessed front door. @@ -310,10 +294,13 @@ Useful surfaces: - `cache.IsDirty` - `store.IsDirty` - `cache.Document(key)` -- `cache.Update(key, apply)` -- `cache.UpsertAsync(document)` (blind write) +- `cache.Soa()` +- `cache.UpsertAsync(document)` - `cache.FlushAllBackingStores()` - `cache.FlushAsync()` +- `cache.FlushBackingStore(store)` +- `cache.PrepareForReloadOrShutdown()` +- `cache.PrepareForReloadOrShutdownAsync()` - `cache.FlushAttachedStoresOnDispose` - `store.FlushOnDispose` diff --git a/src/GameCult.Caching/StoreReplace.cs b/src/GameCult.Caching/StoreReplace.cs deleted file mode 100644 index 5f16322c..00000000 --- a/src/GameCult.Caching/StoreReplace.cs +++ /dev/null @@ -1,182 +0,0 @@ -using System; -using System.IO; -using System.Runtime.InteropServices; -using System.Threading; - -namespace GameCult.Caching -{ - /// The kernel32 calls a Windows replace needs, each answering 0 or a Win32 error code. - internal interface IReplaceKernel - { - /// CreateFileW(staged, DELETE|SYNCHRONIZE), FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, CloseHandle. - int RenamePosix(string staged, string destination); - - /// MoveFileExW(staged, destination, flags). - int MoveFileEx(string staged, string destination, uint flags); - - void Sleep(int milliseconds); - } - - /// - /// Owns how a staged file becomes the store, for the single-file and directory stores. Windows tries a POSIX-semantics - /// rename first (which makes a moment with the path absent rare for a lock-free reader, not impossible: a stalled rename was - /// measured leaving it absent for 5-47 ms), falls back to write-through MoveFileExW where that is unsupported or does not - /// resolve, falls back to the portable File.Replace if MoveFileExW itself does not resolve, and retries transient sharing - /// errors; other OSes keep rename. Mirrors cultcache-ts store-file.ts - /// replaceStoreFile (ruling variants:ruling:windows-replace-posix-first). - /// - internal static class StoreReplace - { - private const int ErrorAccessDenied = 5; - private const int ErrorSharingViolation = 32; - private const int ErrorLockViolation = 33; - private const int ErrorNotSupported = 50; - private const int ErrorInvalidParameter = 87; - private const int ReplaceAttempts = 25; - private const int ReplaceRetryMs = 20; - - /// MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH. - internal const uint MoveFileReplaceWriteThrough = 0x9; - - internal static readonly IReplaceKernel WindowsKernel = new Kernel32Replace(); - - public static void Replace(string staged, string destination) - { - if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) - { - ReplaceWindows(staged, destination, WindowsKernel); - } - else - { - ReplacePortable(staged, destination); - } - } - - private static void ReplacePortable(string staged, string destination) - { - if (File.Exists(destination)) - { - File.Replace(staged, destination, null); - } - else - { - File.Move(staged, destination); - } - } - - private static bool Unresolved(Exception ex) => ex is DllNotFoundException || ex is EntryPointNotFoundException; - - internal static void ReplaceWindows(string staged, string destination, IReplaceKernel kernel) - { - var posix = true; - var lastCode = 0; - for (var attempt = 1; attempt <= ReplaceAttempts; attempt++) - { - int code; - try - { - code = posix ? RenamePosixOrUnsupported(kernel, staged, destination) : kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough); - if (posix && (code == 1 || code == ErrorNotSupported || code == ErrorInvalidParameter)) - { - // POSIX semantics are unsupported here (FAT, exFAT, before Windows 10 1607): the rest of this call is write-through. - posix = false; - code = kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough); - } - } - catch (Exception ex) when (Unresolved(ex)) - { - // kernel32's MoveFileExW does not resolve here (a sandboxed player): the runtime-owned replace is all that is left. - ReplacePortable(staged, destination); - return; - } - - if (code == 0) - { - return; - } - - if (code != ErrorAccessDenied && code != ErrorSharingViolation && code != ErrorLockViolation) - { - throw new IOException($"CultCache could not replace the store: Win32 error {code}."); - } - - lastCode = code; - kernel.Sleep(ReplaceRetryMs); - } - - throw new IOException($"CultCache could not replace the store after {ReplaceAttempts} attempts: last Win32 error {lastCode}."); - } - - private static int RenamePosixOrUnsupported(IReplaceKernel kernel, string staged, string destination) - { - try - { - return kernel.RenamePosix(staged, destination); - } - catch (Exception ex) when (Unresolved(ex)) - { - // The POSIX call does not resolve: the same answer as "unsupported", so the replace falls back to MoveFileExW. - return ErrorNotSupported; - } - } - - private sealed class Kernel32Replace : IReplaceKernel - { - private const uint DeleteSynchronize = 0x00110000; - private const uint ShareAll = 0x7; - private const uint OpenExisting = 3; - private const uint FileAttributeNormal = 0x80; - private const int FileRenameInfoEx = 22; - private const uint RenameReplacePosix = 0x3; - - public int RenamePosix(string staged, string destination) - { - var handle = CreateFileW(staged, DeleteSynchronize, ShareAll, IntPtr.Zero, OpenExisting, FileAttributeNormal, IntPtr.Zero); - if (handle == new IntPtr(-1)) - { - return Marshal.GetLastWin32Error(); - } - - try - { - // FILE_RENAME_INFO: Flags u32 at 0, RootDirectory handle, FileNameLength u32, FileName (UTF-16, offsets for this pointer size). - var name = System.Text.Encoding.Unicode.GetBytes(Path.GetFullPath(destination)); - var lengthOffset = 2 * IntPtr.Size; - var nameOffset = lengthOffset + 4; - var size = nameOffset + name.Length + 2; - var info = Marshal.AllocHGlobal(size); - try - { - for (var i = 0; i < size; i++) Marshal.WriteByte(info, i, 0); - Marshal.WriteInt32(info, 0, (int)RenameReplacePosix); - Marshal.WriteInt32(info, lengthOffset, name.Length); - Marshal.Copy(name, 0, info + nameOffset, name.Length); - return SetFileInformationByHandle(handle, FileRenameInfoEx, info, (uint)size) ? 0 : Marshal.GetLastWin32Error(); - } - finally { Marshal.FreeHGlobal(info); } - } - finally { CloseHandle(handle); } - } - - public int MoveFileEx(string staged, string destination, uint flags) - => MoveFileExW(staged, destination, flags) ? 0 : Marshal.GetLastWin32Error(); - - public void Sleep(int milliseconds) => Thread.Sleep(milliseconds); - - [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] - private static extern IntPtr CreateFileW(string fileName, uint access, uint share, IntPtr security, uint disposition, uint flags, IntPtr template); - - [DllImport("kernel32.dll", SetLastError = true)] - [return: MarshalAs(UnmanagedType.Bool)] - private static extern bool SetFileInformationByHandle(IntPtr handle, int informationClass, IntPtr information, uint size); - - [DllImport("kernel32.dll", SetLastError = true)] - [return: MarshalAs(UnmanagedType.Bool)] - private static extern bool CloseHandle(IntPtr handle); - - [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] - [return: MarshalAs(UnmanagedType.Bool)] - private static extern bool MoveFileExW(string existing, string target, uint flags); - } - } -} diff --git a/src/GameCult.Mesh/CultMesh.cs b/src/GameCult.Mesh/CultMesh.cs index 32b45e1f..e3f23430 100644 --- a/src/GameCult.Mesh/CultMesh.cs +++ b/src/GameCult.Mesh/CultMesh.cs @@ -428,14 +428,11 @@ namespace GameCult.Mesh } /// - /// Flushes the durable cache. With the flush has one meaning, decided by the store's own - /// read and never by what the cache holds in memory: a store with nothing at its path is written; a store that reads - /// (empty included) is skipped and its staged writes stay staged; a store that cannot be read refuses with - /// and is left untouched. + /// Flushes the durable cache. /// public Task FlushAsync(bool soft = false) { - return _host.FlushAsync(soft); + return _host.FlushAsync(); } /// diff --git a/src/GameCult.Networking/CultNetLocal.cs b/src/GameCult.Networking/CultNetLocal.cs index 60937574..87942ce8 100644 --- a/src/GameCult.Networking/CultNetLocal.cs +++ b/src/GameCult.Networking/CultNetLocal.cs @@ -96,14 +96,11 @@ namespace GameCult.Networking } /// - /// Flushes the durable cache. With the flush has one meaning, decided by the store's own - /// read and never by what the cache holds in memory: a store with nothing at its path is written; a store that reads - /// (empty included) is skipped and its staged writes stay staged; a store that cannot be read refuses with - /// and is left untouched. + /// Flushes the durable cache. /// public Task FlushAsync(bool soft = false) { - return Cache.FlushAsync(soft); + return Cache.FlushAsync(); } /// diff --git a/tests/GameCult.Caching.ConsumerBuild/Program.cs b/tests/GameCult.Caching.ConsumerBuild/Program.cs index f09edb8c..60376068 100644 --- a/tests/GameCult.Caching.ConsumerBuild/Program.cs +++ b/tests/GameCult.Caching.ConsumerBuild/Program.cs @@ -56,7 +56,7 @@ public sealed class OutsideStore : CacheBackingStore public override void Delete(CultStoredDocument entry) { } - public override void PushAll(bool soft) { } + public override void PushAll() { } public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => request.ConditionsHold(new Dictionary { ["w"] = "v1" }, Entries.Values, ForeignRecords) diff --git a/tests/GameCult.Caching.OutsideStoreTests/HeldInstanceUpdateTests.cs b/tests/GameCult.Caching.OutsideStoreTests/HeldInstanceUpdateTests.cs deleted file mode 100644 index 6b02b72f..00000000 --- a/tests/GameCult.Caching.OutsideStoreTests/HeldInstanceUpdateTests.cs +++ /dev/null @@ -1,131 +0,0 @@ -using System; -using System.Collections.Generic; -using System.IO; -using GameCult.Caching.MessagePack; -using MessagePack; -using NUnit.Framework; - -namespace GameCult.Caching.OutsideStoreTests -{ - // variants:follow_up:update-copy-from-stored-bytes: Update takes its version and its copy from one durable read of the home - // store, never from the held instance. Consumer side: public API only, so a change made in place to an instance the cache - // hands out can never be committed as if it had been read. - public class HeldInstanceUpdateTests - { - [CultDocument("tests.held_counter", "tests.held_counter.v1")] - [MessagePackObject] - public sealed class HeldCounter - { - [Key(0)] [CultName] public string Name { get; set; } = string.Empty; - [Key(1)] public int Value { get; set; } - } - - private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(HeldCounter) }); - private static readonly CultRecordKey Key = new("counter"); - - private string _directory = string.Empty; - - [SetUp] - public void CreateDirectory() - { - _directory = Path.Combine(Path.GetTempPath(), $"cultlib-held-update-{Guid.NewGuid():N}"); - Directory.CreateDirectory(_directory); - } - - [TearDown] - public void DeleteDirectory() - { - if (Directory.Exists(_directory)) - Directory.Delete(_directory, recursive: true); - } - - private static CultCache Open(string path, bool directory) - { - var cache = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)); - cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path)); - cache.PullAllBackingStoresAsync().GetAwaiter().GetResult(); - return cache; - } - - private string Seed(bool directory, int value) - { - var path = Path.Combine(_directory, "held.cc"); - using var cache = Open(path, directory); - Assert.That(cache.Commit(batch => batch.Upsert(new HeldCounter { Name = "counter", Value = value }, new CultRecordHandle(Key))), Is.True); - return path; - } - - private static int StoredValue(string path, bool directory) - { - using var reader = Open(path, directory); - return reader.Get(Key)!.Value; - } - - private static HeldCounter Plus(HeldCounter? counter, int by) - { - counter!.Value += by; - return counter; - } - - [TestCase(false)] - [TestCase(true)] - public void InPlaceChangeToHeldInstanceIsNeitherAppliedNorPersisted(bool directory) - { - var path = Seed(directory, 1); - using var cache = Open(path, directory); - cache.Get(Key)!.Value = 999; - var seen = new List(); - - var result = cache.Update(Key, counter => { seen.Add(counter!.Value); return Plus(counter, 1); }); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(seen, Is.EqualTo(new[] { 1 }), "apply saw the in-place change"); - Assert.That(result.Document!.Value, Is.EqualTo(2)); - Assert.That(StoredValue(path, directory), Is.EqualTo(2), "the in-place 999 was persisted"); - } - - [TestCase(false)] - [TestCase(true)] - public void InPlaceChangeToAPreviousResultIsNotCommitted(bool directory) - { - var path = Seed(directory, 1); - using var cache = Open(path, directory); - var first = cache.Update(Key, counter => Plus(counter, 1)); - Assert.That(first.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - first.Document!.Value = 999; - var seen = new List(); - - var second = cache.Update(Key, counter => { seen.Add(counter!.Value); return Plus(counter, 1); }); - - Assert.That(second.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(seen, Is.EqualTo(new[] { 2 }), "apply ran more than once or saw the in-place change"); - Assert.That(StoredValue(path, directory), Is.EqualTo(3)); - } - - [TestCase(false)] - [TestCase(true)] - public void DeclineAfterInPlaceChangeWritesNothing(bool directory) - { - var path = Seed(directory, 1); - using var cache = Open(path, directory); - cache.Get(Key)!.Value = 999; - var before = Snapshot(path); - - Assert.That(cache.Update(Key, _ => null).Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); - - Assert.That(Snapshot(path), Is.EqualTo(before)); - Assert.That(StoredValue(path, directory), Is.EqualTo(1)); - } - - // Every file of the store: the single file, or the manifest and its pages. - private static Dictionary Snapshot(string path) - { - var files = new Dictionary { [path] = File.ReadAllBytes(path) }; - var pages = path + ".records"; - if (Directory.Exists(pages)) - foreach (var file in Directory.GetFiles(pages, "*", SearchOption.AllDirectories)) - files[file] = File.ReadAllBytes(file); - return files; - } - } -} diff --git a/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs b/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs index 86949fa0..e47e4d5f 100644 --- a/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs +++ b/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs @@ -136,7 +136,7 @@ namespace GameCult.Caching.OutsideStoreTests IsDirty = true; }); - public override void PushAll(bool soft) => Held(() => Apply(Array.Empty(), Array.Empty(), Array.Empty())); + public override void PushAll() => Held(() => Apply(Array.Empty(), Array.Empty(), Array.Empty())); public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => Held(() => { @@ -364,18 +364,5 @@ namespace GameCult.Caching.OutsideStoreTests Assert.That(() => cache.Commit(batch => batch.Remove(K)), Throws.InstanceOf(), "removal"); Assert.That(disk.Records["k"], Is.EqualTo(before)); } - - // Only the library can stamp a version on a decoded record, so a store kind outside it cannot read one honestly: Update refuses before apply. - [Test] - public void UpdateOverAnOutsideStoreIsNotSupported() - { - var disk = new Disk(); - PutOnDisk(disk, AB, "k", new OutA { Name = "k", Value = 1 }); - using var cache = Open(disk, AB); - var calls = 0; - - Assert.That(() => cache.Update(K, a => { calls++; return a; }), Throws.TypeOf()); - Assert.That(calls, Is.EqualTo(0), "apply ran for a store that cannot serve Update"); - } } } diff --git a/tests/GameCult.Caching.Tests/BackingStoreTests.cs b/tests/GameCult.Caching.Tests/BackingStoreTests.cs index 8d5c3424..7e88c283 100644 --- a/tests/GameCult.Caching.Tests/BackingStoreTests.cs +++ b/tests/GameCult.Caching.Tests/BackingStoreTests.cs @@ -56,137 +56,6 @@ namespace GameCult.Caching.Tests } } - [Test] - public async Task SoftFlush_Over_Readable_Store_Leaves_It_Byte_For_Byte_And_Keeps_Staged_Writes_Dirty() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); - try - { - var store = new SingleFileMessagePackBackingStore(filePath); - var cache = new CultCache(); - cache.AddBackingStore(store); - await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); - await cache.FlushAsync(); - var before = File.ReadAllBytes(filePath); - - var staged = await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" }); - await cache.FlushAsync(soft: true); - - Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft flush rewrote a store it reads"); - Assert.That(store.IsDirty, Is.True, "a skipped soft flush discarded a staged write"); - - await cache.FlushAsync(); - var reader = new CultCache(); - reader.AddBackingStore(new SingleFileMessagePackBackingStore(filePath)); - await reader.PullAllBackingStoresAsync(); - Assert.That(reader.Get(staged.Key)!.Value, Is.EqualTo("two")); - } - finally - { - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task SoftFlush_Over_Readable_Store_Holding_No_Records_Leaves_It() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); - try - { - var store = new SingleFileMessagePackBackingStore(filePath); - var cache = new CultCache(); - cache.AddBackingStore(store); - var first = await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); - await cache.FlushAsync(); - cache.Remove(first.Key); - await cache.FlushAsync(); - var before = File.ReadAllBytes(filePath); - var empty = new CultCache(); - empty.AddBackingStore(new SingleFileMessagePackBackingStore(filePath)); - Assert.That(empty.AllEntries, Is.Empty, "the store is not readable-empty"); - - await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" }); - await cache.FlushAsync(soft: true); - - Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft flush wrote over a readable store that holds no records"); - Assert.That(store.IsDirty, Is.True); - } - finally - { - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task PlainPushAll_Over_Readable_Store_Writes_The_Staged_Records() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); - try - { - var store = new SingleFileMessagePackBackingStore(filePath); - var cache = new CultCache(); - cache.AddBackingStore(store); - await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); - store.PushAll(); - var before = File.ReadAllBytes(filePath); - - await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" }); - store.PushAll(); - - Assert.That(File.ReadAllBytes(filePath), Is.Not.EqualTo(before), "a plain PushAll skipped a store it reads"); - Assert.That(store.IsDirty, Is.False); - } - finally - { - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task SoftFlush_Over_Absent_Path_Writes() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); - try - { - var store = new SingleFileMessagePackBackingStore(filePath); - var cache = new CultCache(); - cache.AddBackingStore(store); - await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); - - await cache.FlushAsync(soft: true); - - Assert.That(File.Exists(filePath), Is.True); - Assert.That(store.IsDirty, Is.False); - } - finally - { - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task SoftFlush_Over_Unreadable_File_Refuses_And_Leaves_It() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); - try - { - var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; - var store = new SingleFileMessagePackBackingStore(filePath); - var cache = new CultCache(); - cache.AddBackingStore(store); - await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); - File.WriteAllBytes(filePath, junk); - - Assert.That(() => store.PushAll(soft: true), Throws.InstanceOf()); - Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(junk)); - Assert.That(store.IsDirty, Is.True); - } - finally - { - if (File.Exists(filePath)) File.Delete(filePath); - } - } - [Test] public async Task SingleFileMessagePackBackingStore_PullAll_SeesFileCreatedAfterReaderOpened() { diff --git a/tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs b/tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs deleted file mode 100644 index 7c2cd5a8..00000000 --- a/tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs +++ /dev/null @@ -1,115 +0,0 @@ -#nullable enable -using System; -using System.Buffers; -using System.IO; -using System.Linq; -using GameCult.Caching; -using GameCult.Caching.MessagePack; -using MessagePack; -using NUnit.Framework; - -namespace GameCult.Caching.Tests -{ - // tests/vectors/content-version: the content version of a stored record, the lowercase hex SHA-256 of its persisted encoding - // (variants:ruling:write-set-content-version-declared-ownership), as the reference reads it from a single-file store. The C# - // reference is the only writer: under CULTCACHE_WRITE_VECTORS=1 the store is written and the versions read from it are written - // beside it; every other run opens the committed store.cc and compares what the reference reads to versions.msgpack. Another - // runtime computes the version from the same file and must equal these rows. - public class ContentVersionVectorTests - { - private const string Vector = "content-version"; - - private static readonly Type Item = - EmittedDocumentTypes.Emit("CvItem", "tests.cv_item", "tests.cv_item.v1", new[] { new EmittedDocumentTypes.Field("Text", typeof(string), 0) }); - - private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { Item }); - - private static readonly (string Key, string Text)[] Records = { ("alpha", "first"), ("beta", "second") }; - - private static byte[] StoreBytes() - { - var directory = Path.Combine(Path.GetTempPath(), $"cultlib-cvv-{Guid.NewGuid():N}"); - Directory.CreateDirectory(directory); - try - { - var path = Path.Combine(directory, "store.cc"); - using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, StoreFlushOnDispose = true })) - { - cache.Clock = () => VectorFiles.StoredAt; - foreach (var (key, text) in Records) - cache.UpsertAsync(Item, EmittedDocumentTypes.New(Item, ("Text", text)), new CultRecordKey(key)).GetAwaiter().GetResult(); - cache.FlushAsync().GetAwaiter().GetResult(); - } - - return File.ReadAllBytes(path); - } - finally - { - Directory.Delete(directory, recursive: true); - } - } - - // What the reference reads from a store file: [key, version] for each record, sorted by key. - private static (string Key, string Version)[] ReadVersions(byte[] store) - { - var directory = Path.Combine(Path.GetTempPath(), $"cultlib-cvr-{Guid.NewGuid():N}"); - Directory.CreateDirectory(directory); - try - { - var path = Path.Combine(directory, "store.cc"); - File.WriteAllBytes(path, store); - using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); - return cache.AllStoredDocuments - .Select(stored => (Key: stored.Key.Value, Version: stored.StoredVersion)) - .OrderBy(row => row.Key, StringComparer.Ordinal) - .ToArray(); - } - finally - { - Directory.Delete(directory, recursive: true); - } - } - - private static byte[] RowsFile((string Key, string Version)[] rows) - { - var buffer = new ArrayBufferWriter(); - var writer = new MessagePackWriter(buffer); - writer.WriteArrayHeader(rows.Length); - foreach (var (key, version) in rows) - { - writer.WriteArrayHeader(2); - writer.Write(key); - writer.Write(version); - } - - writer.Flush(); - return buffer.WrittenSpan.ToArray(); - } - - [Test] - public void TheReferenceReadsTheCommittedVersions() - { - if (VectorFiles.Writing) - { - var store = StoreBytes(); - VectorFiles.Check(Vector, "store.cc", store); - VectorFiles.Check(Vector, "versions.msgpack", RowsFile(ReadVersions(store))); - return; - } - - var committed = VectorFiles.Committed(Vector, "store.cc"); - var rows = ReadVersions(committed); - Assert.That(rows.Select(row => row.Key), Is.EqualTo(Records.Select(record => record.Key)), "the store holds the two records"); - Assert.That(rows.Select(row => row.Version).Distinct().Count(), Is.EqualTo(rows.Length), "two different records, two versions"); - Assert.That(rows.All(row => row.Version.Length == 64 && row.Version == row.Version.ToLowerInvariant()), Is.True, "lowercase hex SHA-256"); - Assert.That(Convert.ToHexString(RowsFile(rows)), Is.EqualTo(Convert.ToHexString(VectorFiles.Committed(Vector, "versions.msgpack"))), "versions.msgpack"); - } - - [Test] - public void TheCommittedStoreIsWhatTheReferenceWrites() - { - if (VectorFiles.Writing) return; - Assert.That(Convert.ToHexString(StoreBytes()), Is.EqualTo(Convert.ToHexString(VectorFiles.Committed(Vector, "store.cc"))), "store.cc"); - } - } -} diff --git a/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs b/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs index f4009a6c..404a75cb 100644 --- a/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs +++ b/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs @@ -338,7 +338,7 @@ namespace GameCult.Caching.Tests public override void Push(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { } - public override void PushAll(bool soft) { } + public override void PushAll() { } public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; } diff --git a/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs b/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs index 92861214..b2c8e7f5 100644 --- a/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs +++ b/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs @@ -180,7 +180,7 @@ namespace GameCult.Caching.Tests public override void Push(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { } - public override void PushAll(bool soft) { } + public override void PushAll() { } public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; } diff --git a/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs b/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs index 942ae891..840110bc 100644 --- a/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs +++ b/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs @@ -208,72 +208,6 @@ namespace GameCult.Caching.Tests Assert.That(reader.AllEntries, Is.Empty); } - [Test] - public async Task SoftFlushOverAbsentManifestWrites() - { - using var cache = Open(out var store); - var key = await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); - - cache.FlushAllBackingStores(soft: true); - - Assert.That(File.Exists(_manifest), Is.True); - Assert.That(store.IsDirty, Is.False); - using var reader = Open(out _); - Assert.That(reader.Get(key)?.Text, Is.EqualTo("one")); - } - - [Test] - public async Task SoftFlushOverReadableManifestLeavesItAndKeepsStagedWritesDirty() - { - using var cache = Open(out var store); - await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); - cache.FlushAllBackingStores(); - var before = File.ReadAllBytes(_manifest); - var pages = Directory.GetFiles(_records, "*.msgpack").OrderBy(path => path).ToArray(); - var second = await cache.UpsertAsync(typeof(Page), new Page { Name = "b", Text = "two" }); - - cache.FlushAllBackingStores(soft: true); - - Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(before), "a soft flush rewrote a manifest it reads"); - Assert.That(Directory.GetFiles(_records, "*.msgpack").OrderBy(path => path).ToArray(), Is.EqualTo(pages)); - Assert.That(store.IsDirty, Is.True, "a skipped soft flush discarded a staged write"); - - cache.FlushAllBackingStores(); - using var reader = Open(out _); - Assert.That(reader.Get(second)?.Text, Is.EqualTo("two")); - } - - [Test] - public async Task SoftFlushOverReadableEmptyManifestLeavesIt() - { - using var cache = Open(out var store); - var key = await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); - cache.FlushAllBackingStores(); - cache.Remove(key); - cache.FlushAllBackingStores(); - var before = File.ReadAllBytes(_manifest); - using (var reader = Open(out _)) Assert.That(reader.AllEntries, Is.Empty, "the store is not readable-empty"); - await cache.UpsertAsync(typeof(Page), new Page { Name = "b", Text = "two" }); - - cache.FlushAllBackingStores(soft: true); - - Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(before), "a soft flush wrote over a readable store that holds no records"); - Assert.That(store.IsDirty, Is.True); - } - - [Test] - public async Task SoftFlushOverUnreadableManifestRefusesAndLeavesIt() - { - using var cache = Open(out var store); - await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); - var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; - File.WriteAllBytes(_manifest, junk); - - Assert.That(() => store.PushAll(soft: true), Throws.Exception); - Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(junk)); - Assert.That(store.IsDirty, Is.True); - } - private CultCache Open(out DirectoryMessagePackBackingStore store) { var cache = new CultCache(Registry); diff --git a/tests/GameCult.Caching.Tests/ElementIdTests.cs b/tests/GameCult.Caching.Tests/ElementIdTests.cs index d40ffe25..a6c68c94 100644 --- a/tests/GameCult.Caching.Tests/ElementIdTests.cs +++ b/tests/GameCult.Caching.Tests/ElementIdTests.cs @@ -224,28 +224,6 @@ namespace GameCult.Caching.Tests Assert.That(AllIds(reread), Is.EqualTo(first), "the ids minted at load are the ids written, and stay through repeated writes"); } - // Update reads the store's bytes, which hold no ids; it mints the ids the load minted, so the write keeps the ids the cache showed. - [Test] - public void UpdateOfAPreIdRecordPersistsTheIdsTheLoadMinted() - { - var path = PathOf("old.cc"); - WritePreIdStore(path, null, ("a", "a", 2)); - var key = new CultRecordKey("a"); - - string[] shown; - using (var cache = Open(path)) - { - shown = AllIds(cache.Get(key)!); - Assert.That(shown, Has.All.Not.Empty); - var result = cache.Update(key, deck => { deck!.Name = "a2"; return deck; }); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - } - - var written = MessagePackSerializer.Deserialize(DiskRecord(path, "a").Payload); - Assert.That(written.Name, Is.EqualTo("a2")); - Assert.That(AllIds(written), Is.EqualTo(shown), "the write replaced the ids the cache showed with random ones"); - } - [Test] public void MintElementIdsRewritesEveryPreIdRecordAndNothingElse() { diff --git a/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs deleted file mode 100644 index bd6cb903..00000000 --- a/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs +++ /dev/null @@ -1,151 +0,0 @@ -#nullable enable -using System; -using System.Collections.Generic; -using System.Linq; -using System.Runtime.Serialization; -using GameCult.Caching; -using MessagePack; -using NUnit.Framework; - -namespace GameCult.Caching.Tests -{ - // The id walk's own contracts, one mutant-sized fact at a time: the order of the id path, a union that names itself, a list of - // values never counting as an id, and the attribute predicates that decide whether a type is walked or written whole. - public class ElementIdWalkTests - { - // The codec asks MessagePack about the id path in order and refuses at the first type it cannot allow, so the order is what - // an error names first: the document, then the collection types from the outermost in, then the element and its arms. - [Test] - public void TheIdPathListsCollectionsFromTheOutermostIn() - { - Assert.That(CultElementIds.Inspect(typeof(MsMapMap)).IdPath, Is.EqualTo(new[] - { - typeof(MsMapMap), typeof(Dictionary>>), - typeof(Dictionary>), typeof(List), typeof(ElementIdEngineTests.EIded) - }), "maps nested in maps"); - Assert.That(CultElementIds.Inspect(typeof(MsListDeep)).IdPath, Is.EqualTo(new[] - { - typeof(MsListDeep), typeof(List>>), typeof(List>), - typeof(List), typeof(ElementIdEngineTests.EIded) - }), "lists nested in lists"); - } - - // ConcreteTypes stops at a type it has already expanded, so a union that lists its own base is finished, not a stack overflow. - [Test] - public void AUnionThatNamesItselfIsWalkedOnce() - { - var survey = CultElementIds.Inspect(typeof(MsSelfHolder)); - Assert.That(survey.Problems, Is.Empty); - Assert.That(survey.IdPath, Is.EquivalentTo(new[] { typeof(MsSelfHolder), typeof(List), typeof(MsSelfBase), typeof(MsSelfArm) })); - } - - // Value lists hold no ids however they are nested, and a root that is itself a value list holds none either. - [Test] - public void ListsOfValuesHoldNoIds() - { - var values = new MsValues { Numbers = { 1 }, Tags = { "a" }, Grid = { new List { 1 } }, Map = { ["k"] = new List { 2 } } }; - Assert.That(CultElementIds.Holds(typeof(MsValues), values), Is.False); - Assert.That(CultElementIds.Holds(typeof(List), new List { 1, 2 }), Is.False); - Assert.That(CultElementIds.Holds(typeof(List>), new List> { new() { 1 } }), Is.False); - Assert.That(CultElementIds.Holds(typeof(List), new List { "a" }), Is.False); - Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated"); - } - - // A nullable struct is walked as the struct it wraps; a nullable value is not walked at all. - [Test] - public void ANullableKeyedStructWalksAsItsOwner() - { - Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct)), Is.True); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct?)), Is.True); - Assert.That(CultElementIds.WalksAsOwner(typeof(int?)), Is.False); - Assert.That(CultElementIds.WalksAsOwner(typeof(int)), Is.False); - } - - // [Key] is read through an override, so an element whose id property overrides a keyed base member is still an id with a key. - [Test] - public void AKeyOnTheBaseOfAnOverrideCountsForTheOverride() - { - Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).Problems, Is.Empty); - Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).IdPath, Does.Contain(typeof(MsOverrideEl))); - } - - // A type is walked by key when it is [MessagePackObject] or [Union], read through its bases for the former; it is one whole - // value when it declares its own formatter, is [DataContract] without being [MessagePackObject], or is keyed by name. - [Test] - public void KeyedTypesWalkAndWholeTypesDoNot() - { - Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpBase)), Is.True, "[MessagePackObject]"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsUBase)), Is.True, "[Union]"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpDerived)), Is.True, "[MessagePackObject] inherited"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsPlain)), Is.False, "[Key] members without [MessagePackObject] or [Union]"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsUArm)), Is.True, "an arm that is itself [MessagePackObject]"); - - Assert.That(CultElementIds.WalksAsOwner(typeof(MsDc)), Is.False, "[DataContract] alone"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcMp)), Is.True, "[DataContract] beside [MessagePackObject]"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcOverMpBase)), Is.True, "[DataContract] over an inherited [MessagePackObject]"); - - Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropName)), Is.False, "KeyAsPropertyName"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropNameDerived)), Is.False, "KeyAsPropertyName inherited"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsStringKey)), Is.False, "a string [Key]"); - - Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedBase)), Is.False, "[MessagePackFormatter]"); - Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedDerived)), Is.False, "[MessagePackFormatter] inherited"); - } - - [MessagePackObject] public class MsMapMap { [Key(0)] public Dictionary>> M { get; set; } = new(); } - [MessagePackObject] public class MsListDeep { [Key(0)] public List>> G { get; set; } = new(); } - - [Union(0, typeof(MsSelfBase)), Union(1, typeof(MsSelfArm))] public abstract class MsSelfBase { } - [MessagePackObject] public class MsSelfArm : MsSelfBase { [Key(0)] [CultElementId] public string Id { get; set; } = ""; } - [MessagePackObject] public class MsSelfHolder { [Key(0)] public List Items { get; set; } = new(); } - - [MessagePackObject] - public class MsValues - { - [Key(0)] public List Numbers { get; set; } = new(); - [Key(1)] public List Tags { get; set; } = new(); - [Key(2)] public List> Grid { get; set; } = new(); - [Key(3)] public Dictionary> Map { get; set; } = new(); - } - - public sealed class MsBoom : IEnumerable - { - public IEnumerator GetEnumerator() => throw new InvalidOperationException("enumerated"); - System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator(); - } - - [MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; } - - [MessagePackObject] - public class MsOverrideBase - { - [Key(0)] [CultElementId] public virtual string Id { get; set; } = ""; - } - - [MessagePackObject] - public class MsOverrideEl : MsOverrideBase - { - public override string Id { get; set; } = ""; - } - - [MessagePackObject] public class MsOverrideHolder { [Key(0)] public List Items { get; set; } = new(); } - - [MessagePackObject] public class MsMpBase { [Key(0)] public int A { get; set; } } - public class MsMpDerived : MsMpBase { } - public class MsPlain { [Key(0)] public int A { get; set; } } - [Union(0, typeof(MsUArm))] public abstract class MsUBase { } - [MessagePackObject] public class MsUArm : MsUBase { [Key(0)] public int A { get; set; } } - - [DataContract] public class MsDc { [DataMember] public int A { get; set; } } - [DataContract] [MessagePackObject] public class MsDcMp { [Key(0)] public int A { get; set; } } - [DataContract] public class MsDcOverMpBase : MsMpBase { } - - [MessagePackObject(keyAsPropertyName: true)] public class MsPropName { public int A { get; set; } } - public class MsPropNameDerived : MsPropName { } - [MessagePackObject] public class MsStringKey { [Key("a")] public int A { get; set; } } - - public sealed class MsNoFormatter { } - [MessagePackFormatter(typeof(MsNoFormatter))] [MessagePackObject] public class MsFormattedBase { [Key(0)] public int A { get; set; } } - [MessagePackObject] public class MsFormattedDerived : MsFormattedBase { [Key(1)] public int B { get; set; } } - } -} diff --git a/tests/GameCult.Caching.Tests/FirstCallTests.cs b/tests/GameCult.Caching.Tests/FirstCallTests.cs index d8febc49..8fad1229 100644 --- a/tests/GameCult.Caching.Tests/FirstCallTests.cs +++ b/tests/GameCult.Caching.Tests/FirstCallTests.cs @@ -62,28 +62,17 @@ namespace GameCult.Caching.Tests } [Test] - public void DocsShowTheTestedSnippet() => StartDocs.AssertShowsSnippet(SourcePath()); - - static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; - } - - // docs/start/README.md shows each tested snippet verbatim: the region between snippet:begin and snippet:end of its source file. - internal static class StartDocs - { - public static void AssertShowsSnippet(string sourcePath) => - AssertShowsSnippet(sourcePath, Path.Combine("..", "..", "docs", "start", "README.md")); - - // The same check against another document, its path relative to the directory of sourcePath. The snippet there is the - // statements of a block, so it need not open the fence (the block's usings come first). - public static void AssertShowsSnippet(string sourcePath, string docsRelativePath, bool wholeBlock = true) + public void DocsShowTheTestedSnippet() { - var source = File.ReadAllText(sourcePath).Replace("\r\n", "\n"); + var source = File.ReadAllText(SourcePath()); const string begin = "// snippet:begin\n"; var start = source.IndexOf(begin, StringComparison.Ordinal) + begin.Length; var snippet = source[start..source.IndexOf("// snippet:end\n", StringComparison.Ordinal)]; - var docs = File.ReadAllText(Path.Combine(Path.GetDirectoryName(sourcePath)!, docsRelativePath)); - Assert.That(docs.Replace("\r\n", "\n"), Does.Contain(wholeBlock ? "```csharp\n" + snippet + "```" : snippet), - $"{docsRelativePath} no longer shows {Path.GetFileName(sourcePath)} verbatim"); + var docs = File.ReadAllText(Path.Combine(Path.GetDirectoryName(SourcePath())!, "..", "..", "docs", "start", "README.md")); + Assert.That(docs.Replace("\r\n", "\n"), Does.Contain("```csharp\n" + snippet + "```"), + "docs/start/README.md no longer shows tests/GameCult.Caching.Tests/FirstCallTests.cs verbatim"); } + + static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; } } diff --git a/tests/GameCult.Caching.Tests/ReadmeHappyPathTests.cs b/tests/GameCult.Caching.Tests/ReadmeHappyPathTests.cs deleted file mode 100644 index de2dabb4..00000000 --- a/tests/GameCult.Caching.Tests/ReadmeHappyPathTests.cs +++ /dev/null @@ -1,81 +0,0 @@ -#nullable enable -using System; -using System.IO; -using System.Threading.Tasks; -using GameCult.Caching; -using GameCult.Caching.MessagePack; -using MessagePack; -using NUnit.Framework; - -namespace GameCult.Caching.Tests -{ - // The Happy Path of src/GameCult.Caching/README.md, run as written: its statements are the region between snippet:begin and - // snippet:end below, and ReadmeShowsTheTestedSnippet fails when the README stops showing them verbatim. - public class ReadmeHappyPathTests - { - [CultDocument("gamecult.item_data", "gamecult.item_data.v1")] - [MessagePackObject] - public sealed class ItemData - { - [Key(0)] [CultName] public string Name = string.Empty; - [Key(1)] [CultIndex] public string Sku = string.Empty; - [Key(2)] public string Category = string.Empty; - [Key(3)] public int Value; - } - - private static async Task HappyPath() - { -// snippet:begin -var cache = await CultCacheMessagePack.OpenAsync("Data.msgpack"); -var key = new CultRecordKey("potion"); - -// The golden path: read, change a private copy, commit only if the record is unchanged, re-run on a lost race. -// Durable when it returns; no flush. -var result = cache.Update(key, item => -{ - item ??= new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable" }; - item.Value += 10; - return item; -}); - -// The blind opt-out: last writer wins per key. Name the key, or a new key claims the same unique Sku and is refused. -var handle = await cache.UpsertAsync(new ItemData -{ - Name = "Potion", - Sku = "POT-1", - Category = "Consumable", - Value = 50 -}, new CultRecordHandle(key)); - -await cache.FlushAsync(); -// snippet:end - return cache; - } - - [Test] - public async Task HappyPathRunsAsWritten() - { - var directory = Path.Combine(Path.GetTempPath(), $"readme-happy-{Guid.NewGuid():N}"); - Directory.CreateDirectory(directory); - var previous = Environment.CurrentDirectory; - try - { - Environment.CurrentDirectory = directory; // the README opens a relative path - using var cache = await HappyPath(); - Assert.That(cache.Get(new CultRecordKey("potion"))!.Value, Is.EqualTo(50)); - Assert.That(cache.GetByIndex("Sku", "POT-1"), Is.Not.Null); - } - finally - { - Environment.CurrentDirectory = previous; - Directory.Delete(directory, recursive: true); - } - } - - [Test] - public void ReadmeShowsTheTestedSnippet() => - StartDocs.AssertShowsSnippet(SourcePath(), Path.Combine("..", "..", "src", "GameCult.Caching", "README.md"), wholeBlock: false); - - static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; - } -} diff --git a/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs b/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs index 64536b77..4444b176 100644 --- a/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs +++ b/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs @@ -170,26 +170,6 @@ namespace GameCult.Caching.Tests Assert.That(IsALink(link), "the link was replaced"); } - [Test] - public void AnAdvisoryQueryOverACacheAtALiveLinkAnswersForTheRecordAndTheWriteIsStillRefused() - { - var path = PathOf("store.cc"); - var link = PathOf("link.cc"); - SeedSingle(path); - Link(link, path); - var before = File.ReadAllBytes(path); - using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); - - Assert.That(cache.IsWritable(new CultRecordKey("seed")), Is.True, "the query does not look at the store path"); - Assert.That(cache.IsWritable(new CultRecordKey("fresh")), Is.True, "the query does not look at the store path"); - var refusal = Assert.Throws(() => - cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("fresh"))))!; - - Assert.That(refusal.Path, Is.EqualTo(link)); - Assert.That(IsALink(link), "the link was replaced"); - Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); - } - // A batch that upserts into one store and removes a key the cache does not hold has the other store judge that removal // under its own lock (JudgeChecks). private CultCache OpenTwoStores(string homePath, string otherPath) diff --git a/tests/GameCult.Caching.Tests/StoreReplaceTests.cs b/tests/GameCult.Caching.Tests/StoreReplaceTests.cs deleted file mode 100644 index 5f71b4a3..00000000 --- a/tests/GameCult.Caching.Tests/StoreReplaceTests.cs +++ /dev/null @@ -1,268 +0,0 @@ -#nullable enable -using System; -using System.Collections.Generic; -using System.IO; -using System.Runtime.InteropServices; -using System.Threading; -using NUnit.Framework; - -namespace GameCult.Caching.Tests -{ - public class StoreReplaceTests - { - private sealed class FakeKernel : IReplaceKernel - { - private readonly Queue _posix; - private readonly Queue _moveEx; - public readonly List Log = new List(); - public readonly List Sleeps = new List(); - public Exception? PosixThrows; - public Exception? MoveExThrows; - - public FakeKernel(int[] posix, int[] moveEx) - { - _posix = new Queue(posix); - _moveEx = new Queue(moveEx); - } - - public int RenamePosix(string staged, string destination) - { - Log.Add("posix"); - if (PosixThrows != null) throw PosixThrows; - return _posix.Count > 0 ? _posix.Dequeue() : 0; - } - - public int MoveFileEx(string staged, string destination, uint flags) - { - Log.Add("moveex:" + flags); - if (MoveExThrows != null) throw MoveExThrows; - return _moveEx.Count > 0 ? _moveEx.Dequeue() : 0; - } - - public void Sleep(int milliseconds) => Sleeps.Add(milliseconds); - } - - private static IOException Fails(FakeKernel kernel) - => Assert.Throws(() => StoreReplace.ReplaceWindows("a", "b", kernel))!; - - [Test] - public void APosixRenameThatWorksIsTheWholeReplace() - { - var kernel = new FakeKernel(new[] { 0 }, new int[0]); - StoreReplace.ReplaceWindows("a", "b", kernel); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix" })); - Assert.That(kernel.Sleeps, Is.Empty); - } - - [TestCase(1)] - [TestCase(50)] - [TestCase(87)] - public void WhereAPosixRenameIsUnsupportedTheReplaceFallsBackToWriteThroughMoveFileEx(int unsupported) - { - var kernel = new FakeKernel(new[] { unsupported }, new[] { 0 }); - StoreReplace.ReplaceWindows("a", "b", kernel); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); - } - - [Test] - public void OnceThePosixRenameIsUnsupportedTheRestOfTheCallStaysOnMoveFileEx() - { - var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 0 }); - StoreReplace.ReplaceWindows("a", "b", kernel); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" })); - Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20 })); - } - - [TestCase(5)] - [TestCase(32)] - [TestCase(33)] - public void ATransientErrorRetriesTheSamePrimitiveAfterTwentyMilliseconds(int transient) - { - var kernel = new FakeKernel(new[] { transient, transient, 0 }, new int[0]); - StoreReplace.ReplaceWindows("a", "b", kernel); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "posix", "posix" })); - Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20, 20 })); - } - - [TestCase(1)] - [TestCase(50)] - [TestCase(87)] - public void AnUnsupportedCodeAfterTheFallbackIsAnErrorNotAnotherFallback(int unsupported) - { - var kernel = new FakeKernel(new[] { 87 }, new[] { unsupported }); - var error = Fails(kernel); - Assert.That(error.Message, Does.Contain("Win32 error " + unsupported)); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); - } - - private static IEnumerable Unresolved() - { - yield return new DllNotFoundException("kernel32.dll"); - yield return new EntryPointNotFoundException("SetFileInformationByHandle"); - } - - [TestCaseSource(nameof(Unresolved))] - public void APosixCallThatDoesNotResolveTakesTheUnsupportedFallback(Exception unresolved) - { - var kernel = new FakeKernel(new int[0], new[] { 0 }) { PosixThrows = unresolved }; - StoreReplace.ReplaceWindows("a", "b", kernel); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); - } - - [TestCaseSource(nameof(Unresolved))] - public void AMoveFileExThatDoesNotResolveDegradesToThePortableReplace(Exception unresolved) - { - var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N")); - Directory.CreateDirectory(dir); - try - { - var destination = Path.Combine(dir, "store.cc"); - var staged = Path.Combine(dir, "staged"); - File.WriteAllBytes(destination, new byte[] { 1 }); - File.WriteAllBytes(staged, new byte[] { 2 }); - var kernel = new FakeKernel(new[] { 87 }, new int[0]) { MoveExThrows = unresolved }; - StoreReplace.ReplaceWindows(staged, destination, kernel); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); - Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 })); - Assert.That(File.Exists(staged), Is.False); - - // Both calls unresolved: the same portable replace, here onto an absent destination. - File.Delete(destination); - File.WriteAllBytes(staged, new byte[] { 3 }); - var both = new FakeKernel(new int[0], new int[0]) { PosixThrows = unresolved, MoveExThrows = unresolved }; - StoreReplace.ReplaceWindows(staged, destination, both); - Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 })); - } - finally { Directory.Delete(dir, true); } - } - - [Test] - public void TheFallbackRunsInTheSameAttemptEvenOnTheLastOne() - { - var posix = new int[25].Fill(32); - posix[24] = 87; - var kernel = new FakeKernel(posix, new[] { 0 }); - StoreReplace.ReplaceWindows("a", "b", kernel); - Assert.That(kernel.Log.Count, Is.EqualTo(26)); - Assert.That(kernel.Log[24], Is.EqualTo("posix")); - Assert.That(kernel.Log[25], Is.EqualTo("moveex:9")); - } - - [Test] - public void ARetriedMoveFileExThatAnswersUnsupportedFailsAfterOneCallNotTwo() - { - var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 87 }); - Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 87")); - Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" })); - } - - [Test] - public void ARetryThatNeverSucceedsGivesUpAfterTwentyFiveAttemptsNamingTheCode() - { - var kernel = new FakeKernel(new int[30].Fill(32), new int[0]); - var error = Fails(kernel); - Assert.That(error.Message, Does.Contain("25 attempts").And.Contain("last Win32 error 32")); - Assert.That(kernel.Log.Count, Is.EqualTo(25)); - Assert.That(kernel.Sleeps.Count, Is.EqualTo(25)); - } - - [Test] - public void AnyOtherCodeThrowsAtOnceNamingTheCode() - { - var kernel = new FakeKernel(new[] { 2 }, new int[0]); - var error = Fails(kernel); - Assert.That(error.Message, Does.Contain("Win32 error 2")); - Assert.That(kernel.Log.Count, Is.EqualTo(1)); - Assert.That(kernel.Sleeps, Is.Empty); - } - - [Test] - public void AFallbackThatFailsOtherwiseThrowsItsCode() - { - var kernel = new FakeKernel(new[] { 87 }, new[] { 3 }); - Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 3")); - } - - [Test] - public void ReplaceOverAnExistingAndThenAnAbsentDestinationLeavesTheStagedBytes() - { - var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N")); - Directory.CreateDirectory(dir); - try - { - var destination = Path.Combine(dir, "store.cc"); - var staged = Path.Combine(dir, "staged"); - File.WriteAllBytes(destination, new byte[] { 1 }); - File.WriteAllBytes(staged, new byte[] { 2 }); - StoreReplace.Replace(staged, destination); - Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 })); - Assert.That(File.Exists(staged), Is.False); - - File.Delete(destination); - File.WriteAllBytes(staged, new byte[] { 3 }); - StoreReplace.Replace(staged, destination); - Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 })); - } - finally { Directory.Delete(dir, true); } - } - - [Test] - public void OnWindowsAReplaceOverADestinationAReaderHoldsOpenSucceedsOnTheFirstAttempt() - { - if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) - Assert.Ignore("The real kernel runs on Windows only."); - - var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N")); - Directory.CreateDirectory(dir); - try - { - var destination = Path.Combine(dir, "store.cc"); - var staged = Path.Combine(dir, "staged"); - File.WriteAllBytes(destination, new byte[] { 1 }); - File.WriteAllBytes(staged, new byte[] { 2 }); - using (new FileStream(destination, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete)) - { - var kernel = new CountingKernel(StoreReplace.WindowsKernel); - StoreReplace.ReplaceWindows(staged, destination, kernel); - Assert.That(kernel.PosixCalls, Is.EqualTo(1)); - Assert.That(kernel.MoveExCalls, Is.EqualTo(0)); - Assert.That(kernel.Sleeps, Is.EqualTo(0)); - Assert.That(kernel.LastCode, Is.EqualTo(0)); - } - - Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 })); - } - finally { Directory.Delete(dir, true); } - } - - private sealed class CountingKernel : IReplaceKernel - { - private readonly IReplaceKernel _inner; - public int PosixCalls, MoveExCalls, Sleeps, LastCode; - public CountingKernel(IReplaceKernel inner) { _inner = inner; } - - public int RenamePosix(string staged, string destination) - { - PosixCalls++; - return LastCode = _inner.RenamePosix(staged, destination); - } - - public int MoveFileEx(string staged, string destination, uint flags) - { - MoveExCalls++; - return LastCode = _inner.MoveFileEx(staged, destination, flags); - } - - public void Sleep(int milliseconds) { Sleeps++; _inner.Sleep(milliseconds); } - } - } - - internal static class ArrayFill - { - public static int[] Fill(this int[] array, int value) - { - for (var i = 0; i < array.Length; i++) array[i] = value; - return array; - } - } -} diff --git a/tests/GameCult.Caching.Tests/StoreRoutingTests.cs b/tests/GameCult.Caching.Tests/StoreRoutingTests.cs index 3ccd7a04..af7e7da7 100644 --- a/tests/GameCult.Caching.Tests/StoreRoutingTests.cs +++ b/tests/GameCult.Caching.Tests/StoreRoutingTests.cs @@ -699,10 +699,10 @@ namespace GameCult.Caching.Tests _nested = nested; } - public override void PushAll(bool soft) + public override void PushAll() { _nested(); - base.PushAll(soft); + base.PushAll(); } } diff --git a/tests/GameCult.Caching.Tests/UpdateCallTests.cs b/tests/GameCult.Caching.Tests/UpdateCallTests.cs deleted file mode 100644 index d9a8b31e..00000000 --- a/tests/GameCult.Caching.Tests/UpdateCallTests.cs +++ /dev/null @@ -1,54 +0,0 @@ -#nullable enable -using System; -using System.IO; -using NUnit.Framework; - -// snippet:begin -using GameCult.Caching; -using GameCult.Caching.MessagePack; - -public static class UpdateCall -{ - public static CultUpdateResult Restock(string path, CultRecordKey key) - { - var cache = CultCacheMessagePack.Create(path); - return cache.Update(key, item => - { - item ??= new Item { Name = "Potion", Category = "Consumable" }; - item.Value += 10; - return item; - }); - } -} -// snippet:end - -namespace GameCult.Caching.Tests -{ - // The C# update snippet that docs/start/README.md shows verbatim; it reuses FirstCallTests' Item. - public class UpdateCallTests - { - [Test] - public void RestockTwiceReadsTwenty() - { - var path = Path.Combine(Path.GetTempPath(), $"update-call-{Guid.NewGuid():N}.cc"); - try - { - var key = new CultRecordKey("potion"); - Assert.That(UpdateCall.Restock(path, key).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(UpdateCall.Restock(path, key).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - - using var reader = CultCacheMessagePack.Create(path); - Assert.That(reader.Get(key)!.Value, Is.EqualTo(20)); - } - finally - { - File.Delete(path); - } - } - - [Test] - public void DocsShowTheTestedSnippet() => StartDocs.AssertShowsSnippet(SourcePath()); - - static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; - } -} diff --git a/tests/GameCult.Caching.Tests/UpdateTests.cs b/tests/GameCult.Caching.Tests/UpdateTests.cs deleted file mode 100644 index 63462ba6..00000000 --- a/tests/GameCult.Caching.Tests/UpdateTests.cs +++ /dev/null @@ -1,803 +0,0 @@ -#nullable enable -using System; -using System.Collections.Generic; -using System.IO; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using GameCult.Caching; -using GameCult.Caching.MessagePack; -using MessagePack; -using NUnit.Framework; -using R3; - -namespace GameCult.Caching.Tests -{ - // CultCache.Update: read the content version, apply a pure function to a private copy, commit conditionally, re-run on a lost - // race (variants:ruling:write-set-content-version-declared-ownership; variants:question:update-apply-copy). - public class UpdateTests - { - private static readonly CultRecordKey Key = new("counter"); - private static readonly CultDocumentRegistry Registry = - CultDocumentRegistry.ForTypes(new[] { typeof(UpdateCounter), typeof(UpdateTally), typeof(UpdateGear), typeof(UpdateBase), typeof(UpdateDerived) }); - - private string _directory = string.Empty; - - [SetUp] - public void CreateDirectory() - { - _directory = Path.Combine(Path.GetTempPath(), $"cultlib-update-{Guid.NewGuid():N}"); - Directory.CreateDirectory(_directory); - } - - [TearDown] - public void DeleteDirectory() - { - if (Directory.Exists(_directory)) - Directory.Delete(_directory, recursive: true); - } - - private string PathOf(string name) => Path.Combine(_directory, name); - - private static CultCache Open(string path, bool directory, bool codec = true) - { - var cache = codec ? new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)) : new CultCache(Registry); - cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path)); - return cache; - } - - private static void Seed(string path, bool directory, int value) - { - using var cache = Open(path, directory); - Assert.That(cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = value }, new CultRecordHandle(Key))), Is.True); - } - - private static void Rival(CultCache cache, int value) => - Assert.That(cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = value }, new CultRecordHandle(Key))), Is.True); - - private static int StoredValue(string path, bool directory) - { - using var reader = Open(path, directory); - return reader.Get(Key)!.Value; - } - - private static UpdateCounter Plus(UpdateCounter? counter, int by) - { - counter ??= new UpdateCounter { Name = "counter" }; - counter.Value += by; - return counter; - } - - [TestCase(false)] - [TestCase(true)] - public void TwoWritersUpdatingLoseNoIncrement(bool directory) - { - const int n = 100; - var path = PathOf("race.cc"); - Seed(path, directory, 0); - using var a = Open(path, directory); - using var b = Open(path, directory); - var applied = 0; - // Each round starts both writers together: an attempt reads fresh from the store, so only overlapping read-commit windows lose. - var round = new Barrier(2); - - var racers = new[] { a, b }.Select(cache => Task.Run(() => - { - var results = new List(); - for (var i = 0; i < n; i++) - { - round.SignalAndWait(); - results.Add(cache.Update(Key, counter => - { - Interlocked.Increment(ref applied); - return Plus(counter, 1); - }, maxAttempts: 200).Outcome); - } - - return results; - })).ToArray(); - Assert.That(Task.WaitAll(racers, TimeSpan.FromMinutes(2)), Is.True, "the racers did not finish"); - - Assert.That(racers.SelectMany(racer => racer.Result), Is.All.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(StoredValue(path, directory), Is.EqualTo(2 * n)); - Assert.That(applied, Is.GreaterThan(2 * n), "the writers never actually raced"); - } - - [Test] - public void LostAttemptRerunsApplyOnTheFreshValue() - { - var path = PathOf("rerun.cc"); - Seed(path, directory: false, 1); - using var cache = Open(path, false); - using var rival = Open(path, false); - var seen = new List(); - - var result = cache.Update(Key, counter => - { - seen.Add(counter!.Value); - if (seen.Count == 1) - Rival(rival, 100); - return Plus(counter, 1); - }); - - Assert.That(seen, Is.EqualTo(new[] { 1, 100 })); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(result.Document!.Value, Is.EqualTo(101)); - Assert.That(StoredValue(path, false), Is.EqualTo(101)); - } - - [Test] - public void ExhaustedAttemptsReturnLostRace() - { - var path = PathOf("lost.cc"); - Seed(path, false, 1); - using var cache = Open(path, false); - using var rival = Open(path, false); - var calls = 0; - - var result = cache.Update(Key, counter => - { - Rival(rival, 100 * ++calls); - return Plus(counter, 1); - }, maxAttempts: 3); - - Assert.That(calls, Is.EqualTo(3)); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace)); - Assert.That(result.Document!.Value, Is.EqualTo(300)); - Assert.That(StoredValue(path, false), Is.EqualTo(300), "the store holds the rival's last write"); - } - - [Test] - public void ApplyGetsAPrivateCopy() - { - var path = PathOf("copy.cc"); - Seed(path, false, 1); - using var cache = Open(path, false); - using var rival = Open(path, false); - var published = new List(); - using var subscription = cache.Watch().Subscribe(change => published.Add(change.Document!.Value)); - var arguments = new List(); - var calls = 0; - - var result = cache.Update(Key, counter => - { - calls++; - arguments.Add(counter!.Value); - Assert.That(counter, Is.Not.SameAs(cache.Get(Key)), "apply was handed the held instance"); - counter.Value += 1000; - Assert.That(cache.Get(Key)!.Value, Is.Not.EqualTo(counter.Value), "the change reached the cache before any commit"); - if (calls == 1) - Rival(rival, 5); - return counter; - }); - - Assert.That(arguments, Is.EqualTo(new[] { 1, 5 }), "the second call saw the first call's change"); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(published, Does.Not.Contain(1001), "an uncommitted value was published"); - Assert.That(published.Count(value => value == 1005), Is.EqualTo(1)); - Assert.That(StoredValue(path, false), Is.EqualTo(1005)); - } - - // No publication in the middle of an Update: a rival's write that cost this cache an attempt is never published on its own, - // only the value this Update commits (a pull after every lost attempt would publish both). - [TestCase(false)] - [TestCase(true)] - public void LostAttemptPublishesNothingMidUpdate(bool directory) - { - var path = PathOf("midupdate.cc"); - Seed(path, directory, 1); - using var cache = Open(path, directory); - using var rival = Open(path, directory); - var published = new List(); - using var subscription = cache.Watch().Subscribe(change => published.Add(change.Document!.Value)); - var calls = 0; - - var result = cache.Update(Key, counter => - { - if (++calls == 1) - Rival(rival, 5); - return Plus(counter, 1); - }); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(calls, Is.EqualTo(2)); - Assert.That(published, Is.EqualTo(new[] { 6 }), "the rival's 5 was published in the middle of the Update"); - } - - // The directory store reads only the key's own page, so a damaged page of another record does not stop an Update. - [Test] - public void DirectoryUpdateReadsOnlyItsOwnPage() - { - var path = PathOf("pages.cc"); - var other = new CultRecordKey("other"); - using (var seed = Open(path, true)) - Assert.That(seed.Commit(batch => batch.Upsert(new UpdateCounter { Name = "other", Value = 9 }, new CultRecordHandle(other))), Is.True); - var otherPages = Directory.GetFiles(path + ".records", "*.msgpack"); - Assert.That(otherPages, Has.Length.EqualTo(1)); - Seed(path, true, 1); - using var cache = Open(path, true); - File.WriteAllBytes(otherPages[0], new byte[] { 1, 2, 3, 4, 5 }); - - var result = cache.Update(Key, counter => Plus(counter, 1)); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(result.Document!.Value, Is.EqualTo(2)); - } - - // The durable read is not a load: it leaves the store's foreign list, its migration reports and what the cache publishes alone. - [TestCase(false)] - [TestCase(true)] - public void ReadDurableLeavesTheStoreViewAlone(bool directory) - { - var path = PathOf("view.cc"); - Seed(path, directory, 1); - using (var writer = Open(path, directory)) - Assert.That(writer.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 3 }, new CultRecordHandle(new CultRecordKey("gear")))), Is.True); - var narrow = CultDocumentRegistry.ForTypes(new[] { typeof(UpdateCounter) }); - using var cache = new CultCache(narrow); - cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path)); - using var rival = Open(path, directory); - var store = cache.BackingStores.Single(); - var foreign = store.ForeignRecords; - var reports = store.LastSchemaMigrationReports; - Assert.That(foreign.Select(record => record.Key), Is.EqualTo(new[] { "gear" }), "the fixture holds a foreign record"); - Assert.That(reports, Is.Not.Empty, "the fixture holds a migration report"); - var foreignKeys = foreign.Select(record => (record.Key, record.SchemaId, record.StoredAt)).ToArray(); - var reportKinds = reports.Select(report => (report.PersistedSchemaName, report.Kind)).ToArray(); - var changes = 0; - using var subscription = cache.Watch().Subscribe(_ => changes++); - var calls = 0; - - var result = cache.Update(Key, counter => - { - if (++calls == 1) - Rival(rival, 5); - return Plus(counter, 1); - }); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(calls, Is.EqualTo(2), "the Update reread the key"); - Assert.That(store.ForeignRecords, Is.SameAs(foreign)); - Assert.That(store.LastSchemaMigrationReports, Is.SameAs(reports)); - Assert.That(store.ForeignRecords.Select(record => (record.Key, record.SchemaId, record.StoredAt)), Is.EqualTo(foreignKeys)); - Assert.That(store.LastSchemaMigrationReports.Select(report => (report.PersistedSchemaName, report.Kind)), Is.EqualTo(reportKinds)); - Assert.That(changes, Is.EqualTo(1), "only the committed write was published"); - } - - // Routing: store A is home to UpdateBase, store B to UpdateDerived : UpdateBase. Land judges a derived record at B. - private CultCache OpenRouted() - { - var cache = new CultCache(Registry); - cache.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf("a.cc")), typeof(UpdateBase)); - cache.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf("b.cc")), typeof(UpdateDerived)); - return cache; - } - - private CultCache OpenOneRouted(string file, Type home) - { - var cache = new CultCache(Registry); - cache.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf(file)), home); - return cache; - } - - private byte[]? BytesOf(string file) => File.Exists(PathOf(file)) ? File.ReadAllBytes(PathOf(file)) : null; - - private static readonly CultRecordHandle BaseHandle = new(Key); - - private static void RivalDerived(CultCache cache, int value) => - Assert.That(cache.Commit(batch => batch.Upsert(new UpdateDerived { Name = "counter", Value = value }, BaseHandle)), Is.True); - - [Test] - public void RoutedSubtypeStaleCacheConverges() - { - using var cache = OpenRouted(); - using var rival = OpenRouted(); - RivalDerived(rival, 1); - var seen = new List(); - - var result = cache.Update(Key, document => - { - seen.Add((document as UpdateDerived)?.Value); - var derived = (UpdateDerived)document!; - derived.Value += 1; - return derived; - }); - - Assert.That(seen, Is.EqualTo(new int?[] { 1 }), "apply did not see the rival's record in its home store, once"); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - using var reader = OpenRouted(); - Assert.That(((UpdateDerived)reader.Get(Key)!).Value, Is.EqualTo(2)); - } - - [Test] - public void RoutedSubtypeCreateLandsInItsHome() - { - using (var cache = OpenRouted()) - { - var calls = 0; - var result = cache.Update(Key, document => - { - calls++; - Assert.That(document, Is.Null); - return new UpdateDerived { Name = "counter", Value = 1 }; - }); - Assert.That(calls, Is.EqualTo(1)); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - } - - using (var inB = OpenOneRouted("b.cc", typeof(UpdateDerived))) - Assert.That(inB.Get(Key), Is.InstanceOf()); - using (var inA = OpenOneRouted("a.cc", typeof(UpdateBase))) - Assert.That(inA.Get(Key), Is.Null, "the record landed in the wrong store"); - } - - [Test] - public void MovingTheRecordToAnotherHomeRefuses() - { - using (var rival = OpenRouted()) - RivalDerived(rival, 1); - var before = (A: BytesOf("a.cc"), B: BytesOf("b.cc")); - using var cache = OpenRouted(); - var calls = 0; - - var refusal = Assert.Throws(() => - cache.Update(Key, _ => { calls++; return new UpdateBase { Name = "counter" }; }))!; - - Assert.That(calls, Is.EqualTo(1), "a refusal was retried"); - Assert.That(refusal.Message, Does.Contain("a.cc").And.Contain("b.cc").And.Contain("move")); - Assert.That(BytesOf("a.cc"), Is.EqualTo(before.A)); - Assert.That(BytesOf("b.cc"), Is.EqualTo(before.B)); - } - - [Test] - public void ARecordHeldByTwoStoresRefusesBeforeApply() - { - using var cache = OpenRouted(); - using (var inA = OpenOneRouted("a.cc", typeof(UpdateBase))) - Assert.That(inA.Commit(batch => batch.Upsert(new UpdateBase { Name = "counter" }, BaseHandle)), Is.True); - using (var inB = OpenOneRouted("b.cc", typeof(UpdateDerived))) - Assert.That(inB.Commit(batch => batch.Upsert(new UpdateDerived { Name = "counter", Value = 1 }, BaseHandle)), Is.True); - var calls = 0; - - var refusal = Assert.Throws(() => cache.Update(Key, document => { calls++; return document; }))!; - - Assert.That(calls, Is.EqualTo(0)); - Assert.That(refusal.Message, Does.Contain("held by two stores")); - } - - [Test] - public void NullFromApplyDeclines() - { - var path = PathOf("decline.cc"); - Seed(path, false, 7); - var before = File.ReadAllBytes(path); - using var cache = Open(path, false); - var changes = 0; - using var subscription = cache.Watch().Subscribe(_ => changes++); - - var result = cache.Update(Key, _ => null); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); - Assert.That(result.Document!.Value, Is.EqualTo(7)); - Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); - Assert.That(changes, Is.EqualTo(0)); - } - - [TestCase(false)] - [TestCase(true)] - public void AbsentKeyCreatesOnceUnderRace(bool directory) - { - var path = PathOf("create.cc"); - using var a = Open(path, directory); - using var b = Open(path, directory); - var started = new Barrier(2); - - var racers = new[] { a, b }.Select(cache => Task.Run(() => - { - started.SignalAndWait(); - return cache.Update(Key, counter => Plus(counter, 1)).Outcome; - })).ToArray(); - Assert.That(Task.WaitAll(racers, TimeSpan.FromMinutes(1)), Is.True); - - Assert.That(racers.Select(racer => racer.Result), Is.All.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(StoredValue(path, directory), Is.EqualTo(2)); - } - - [Test] - public void RefusalsAreNotRetried() - { - var path = PathOf("refuse.cc"); - Seed(path, false, 1); - var before = File.ReadAllBytes(path); - var calls = 0; - Func apply = counter => { calls++; return Plus(counter, 1); }; - - // A cache with no store copies its held instance, so it needs a codec. - using (var storeLess = new CultCache(Registry)) - { - storeLess.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = 1 }, new CultRecordHandle(Key))); - Assert.That(() => storeLess.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("no codec")); - } - - // Stores are attached, but none is home to the type: refused before apply. - using (var noHome = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry))) - { - noHome.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf("nohome.cc")), typeof(UpdateTally)); - Assert.That(() => noHome.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("names its home store")); - } - using (var cache = Open(path, false)) - { - Assert.That(() => cache.Update(Key, apply, maxAttempts: 0), Throws.TypeOf()); - Assert.That(() => cache.Update(Key, null!), Throws.TypeOf()); - Assert.That(() => cache.Update(Key, _ => new UpdateTally { Name = "counter" }), Throws.InvalidOperationException); - } - - using (var cache = Open(path, false)) - { - cache.Commit(batch => - { - batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle(new CultRecordKey("gear"))); - }); - cache.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"), - new[] { cache.Override(nameof(UpdateGear.Name), "gear big"), cache.Override(nameof(UpdateGear.Power), 9) })); - } - - before = File.ReadAllBytes(path); - var gearCalls = 0; - using (var cache = Open(path, false)) - Assert.That(() => cache.Update(new CultRecordKey("gear-big"), gear => { gearCalls++; return gear; }), Throws.InvalidOperationException.With.Message.Contains("Update cannot copy it")); - Assert.That(gearCalls, Is.EqualTo(0), "apply ran for a variant"); - Assert.That(calls, Is.EqualTo(0), "apply ran for a refused call"); - Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); - - using (var cache = Open(path, false)) - { - cache.UpsertAsync(new UpdateTally { Name = "staged" }).GetAwaiter().GetResult(); - Assert.That(() => cache.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("flush before a conditional commit")); - Assert.That(calls, Is.LessThanOrEqualTo(1)); - } - - Assert.That(File.ReadAllBytes(path), Is.EqualTo(before), "a refusal wrote"); - } - - [Test] - public void VariantStoreConflictIsALostAttempt() - { - var path = PathOf("variant.cc"); - Seed(path, false, 1); - using (var seed = Open(path, false)) - { - seed.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle(new CultRecordKey("gear")))); - seed.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"), - new[] { seed.Override(nameof(UpdateGear.Name), "gear big"), seed.Override(nameof(UpdateGear.Power), 9) })); - } - - using var cache = Open(path, false); - using (var rival = Open(path, false)) - Assert.That(rival.Commit(batch => batch.Upsert(new UpdateTally { Name = "other" }, new CultRecordHandle(new CultRecordKey("other")))), Is.True); - var calls = 0; - - var result = cache.Update(Key, counter => { calls++; return Plus(counter, 1); }); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(calls, Is.EqualTo(2), "the conflict was not a lost attempt that re-ran apply"); - Assert.That(StoredValue(path, false), Is.EqualTo(2)); - } - - private static CultPersistedStoreSnapshot ReadSnapshot(string path) => - CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path)); - - // A reload that fails inside a conflict is a read failure, not a lost race: Update lets it propagate and never re-runs apply. - // Here the gear a variant is based on has gone foreign in the file, so the reload the conflict owes cannot drop it. - [Test] - public void ConflictWhoseReloadFailedPropagates() - { - var path = PathOf("reload-failed.cc"); - Seed(path, false, 1); - using var cache = Open(path, false); - cache.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle(new CultRecordKey("gear")))); - cache.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"), - new[] { cache.Override(nameof(UpdateGear.Name), "gear big"), cache.Override(nameof(UpdateGear.Power), 9) })); - const string foreignId = "sha256:00000000000000000000000000000000000000000000000000000000000050de"; - var snapshot = ReadSnapshot(path); - snapshot.Records.Single(record => record.Key == "gear").SchemaId = foreignId; - snapshot.SchemaCatalog = snapshot.SchemaCatalog.Append(new CultSchemaCatalogEntry - { - SchemaId = foreignId, SchemaName = "tests.update_foreign", SchemaVersion = "tests.update_foreign.v1", - ContentHash = "foreign", CanonicalSchemaJson = "{}" - }).ToArray(); - File.WriteAllBytes(path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot)); - var calls = 0; - - var conflict = Assert.Throws(() => cache.Update(Key, counter => { calls++; return Plus(counter, 1); }))!; - - Assert.That(conflict.InnerException, Is.Not.Null, "the reload that failed is the cause"); - Assert.That(calls, Is.EqualTo(1), "a failed reload was retried"); - } - - [TestCase(false)] - [TestCase(true)] - public void RivalWritingAndRestoringTheValueStillRerunsApply(bool directory) - { - var path = PathOf("aba.cc"); - Seed(path, directory, 1); - using var cache = Open(path, directory); - using var rival = Open(path, directory); - var seen = new List(); - - var result = cache.Update(Key, counter => - { - seen.Add(counter!.Value); - if (seen.Count == 1) - { - Rival(rival, 5); - Rival(rival, 1); - } - return Plus(counter, 1); - }); - - Assert.That(seen, Is.EqualTo(new[] { 1, 1 }), "the content version missed A-B-A"); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(StoredValue(path, directory), Is.EqualTo(2)); - } - - [TestCase(false)] - [TestCase(true)] - public void RivalRemovalIsSeenAsNull(bool directory) - { - var path = PathOf("removal.cc"); - Seed(path, directory, 1); - using var cache = Open(path, directory); - using var rival = Open(path, directory); - var seen = new List(); - - var result = cache.Update(Key, counter => - { - seen.Add(counter?.Value); - if (seen.Count == 1) - rival.Commit(batch => batch.Remove(Key)); - return Plus(counter, 1); - }); - - Assert.That(seen, Is.EqualTo(new int?[] { 1, null })); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(StoredValue(path, directory), Is.EqualTo(1)); - } - - [Test] - public void ExhaustedAttemptsReturnLostRaceInADirectoryStore() - { - var path = PathOf("lost-dir.cc"); - Seed(path, true, 1); - using var cache = Open(path, true); - using var rival = Open(path, true); - var calls = 0; - - var result = cache.Update(Key, counter => - { - Rival(rival, 100 * ++calls); - return Plus(counter, 1); - }, maxAttempts: 2); - - Assert.That(calls, Is.EqualTo(2)); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace)); - Assert.That(result.Document!.Value, Is.EqualTo(200)); - Assert.That(StoredValue(path, true), Is.EqualTo(200)); - } - - [Test] - public void OneAttemptMeansOneApplyAndNonPositiveThrows() - { - var path = PathOf("one.cc"); - Seed(path, false, 1); - using var cache = Open(path, false); - using var rival = Open(path, false); - var calls = 0; - - var result = cache.Update(Key, counter => { calls++; Rival(rival, 50); return Plus(counter, 1); }, maxAttempts: 1); - - Assert.That(calls, Is.EqualTo(1)); - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace)); - Assert.That(() => cache.Update(Key, counter => counter, maxAttempts: -1), Throws.TypeOf()); - Assert.That(calls, Is.EqualTo(1)); - } - - [TestCase(false)] - [TestCase(true)] - public void UpdatedDocumentIsTheHeldInstanceAndOwnWritesCauseNoRetry(bool directory) - { - var path = PathOf("own.cc"); - Seed(path, directory, 1); - using var cache = Open(path, directory); - cache.UpsertAsync(new UpdateCounter { Name = "counter", Value = 10 }, new CultRecordHandle(Key)).GetAwaiter().GetResult(); - cache.FlushAsync().GetAwaiter().GetResult(); - var calls = 0; - - var first = cache.Update(Key, counter => { calls++; return Plus(counter, 1); }); - var second = cache.Update(Key, counter => { calls++; return Plus(counter, 1); }); - - Assert.That(calls, Is.EqualTo(2), "the cache's own writes made a later Update retry"); - Assert.That(first.Document, Is.Not.Null); - Assert.That(second.Document, Is.SameAs(cache.Get(Key)), "Updated returned a copy, not the held instance"); - Assert.That(StoredValue(path, directory), Is.EqualTo(12)); - } - - [Test] - public void ApplyRunsOutsideTheCacheGate() - { - var path = PathOf("gate.cc"); - Seed(path, false, 1); - using var cache = Open(path, false); - var reached = false; - - cache.Update(Key, counter => - { - reached = Task.Run(() => cache.Get(Key)).Wait(TimeSpan.FromSeconds(10)); - return Plus(counter, 1); - }); - - Assert.That(reached, Is.True, "another thread could not read the cache while apply ran"); - } - - [TestCase(false)] - [TestCase(true)] - public void ContendedLedgerStoredValueEqualsUpdatedCount(bool directory) - { - var path = PathOf("ledger.cc"); - Seed(path, directory, 0); - var caches = Enumerable.Range(0, 3).Select(_ => Open(path, directory)).ToArray(); - try - { - var tasks = caches.Select(cache => Task.Run(() => - { - var updated = 0; - for (var i = 0; i < 25; i++) - if (cache.Update(Key, counter => Plus(counter, 1)).Outcome == CultUpdateOutcome.Updated) - updated++; - return updated; - })).ToArray(); - Assert.That(Task.WaitAll(tasks, TimeSpan.FromMinutes(2)), Is.True); - - Assert.That(StoredValue(path, directory), Is.EqualTo(tasks.Sum(task => task.Result)), "an Updated outcome was lost or a LostRace wrote"); - } - finally - { - foreach (var cache in caches) - cache.Dispose(); - } - } - - [Test] - public void InMemoryCacheUpdatesUnderThreadRace() - { - using var cache = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)); - cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter" }, new CultRecordHandle(Key))); - - var threads = Enumerable.Range(0, 2).Select(_ => Task.Run(() => - { - for (var i = 0; i < 100; i++) - Assert.That(cache.Update(Key, counter => Plus(counter, 1), maxAttempts: 200).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - })).ToArray(); - Assert.That(Task.WaitAll(threads, TimeSpan.FromMinutes(1)), Is.True); - - Assert.That(cache.Get(Key)!.Value, Is.EqualTo(200)); - } - - [TestCase(false)] - [TestCase(true)] - public void StaleCacheFirstAttemptSeesTheStore(bool directory) - { - var path = PathOf("stale.cc"); - Seed(path, directory, 1); - using var a = Open(path, directory); - using var b = Open(path, directory); - Assert.That(b.Get(Key)!.Value, Is.EqualTo(1)); - Rival(a, 7); - var seen = new List(); - - var result = b.Update(Key, counter => { seen.Add(counter!.Value); return Plus(counter, 1); }); - - Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - Assert.That(seen, Is.EqualTo(new[] { 7 }), "the first attempt read the cache's stale copy"); - Assert.That(StoredValue(path, directory), Is.EqualTo(8)); - } - - // The durable read is not a load: it leaves the store's view of what the cache holds alone, so a later pull still delivers the rival's write. - [TestCase(false)] - [TestCase(true)] - public void ADeclinedUpdateDoesNotHideTheRivalsWriteFromALaterPull(bool directory) - { - var path = PathOf("hidden.cc"); - Seed(path, directory, 1); - using var a = Open(path, directory); - using var b = Open(path, directory); - Assert.That(b.Get(Key)!.Value, Is.EqualTo(1)); - Rival(a, 7); - - Assert.That(b.Update(Key, _ => null).Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); - Assert.That(b.Get(Key)!.Value, Is.EqualTo(1), "a declined Update changed what the cache holds"); - b.PullAllBackingStoresAsync().GetAwaiter().GetResult(); - - Assert.That(b.Get(Key)!.Value, Is.EqualTo(7)); - } - - [Test] - public void CodecLessStoreBackedCacheUpdates() - { - var path = PathOf("codecless.cc"); - Seed(path, false, 1); - var calls = 0; - using (var cache = Open(path, false, codec: false)) - Assert.That(cache.Update(Key, counter => { calls++; return Plus(counter, 1); }).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); - - Assert.That(calls, Is.EqualTo(1)); - Assert.That(StoredValue(path, false), Is.EqualTo(2)); - } - - [TestCase(false)] - [TestCase(true)] - public void AStoreHoldingNothingAtTheKeyIsAbsentEvenIfTheCacheHoldsIt(bool directory) - { - var path = PathOf("gone.cc"); - Seed(path, directory, 5); - using var a = Open(path, directory); - using var b = Open(path, directory); - Assert.That(b.Get(Key), Is.Not.Null); - Assert.That(a.Commit(batch => batch.Remove(Key)), Is.True); - var seen = new List(); - - Assert.That(b.Update(Key, counter => { seen.Add(counter); return null; }).Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); - Assert.That(seen, Is.EqualTo(new UpdateCounter?[] { null })); - } - - [CultDocument("tests.update_counter", "tests.update_counter.v1")] - [MessagePackObject(AllowPrivate = true)] - internal sealed class UpdateCounter - { - [Key(0)] - [CultName] - public string Name = string.Empty; - - [Key(1)] - public int Value; - } - - [CultDocument("tests.update_tally", "tests.update_tally.v1")] - [MessagePackObject(AllowPrivate = true)] - internal sealed class UpdateTally - { - [Key(0)] - [CultName] - public string Name = string.Empty; - } - - [CultDocument("tests.update_gear", "tests.update_gear.v1")] - [MessagePackObject(AllowPrivate = true)] - internal sealed class UpdateGear - { - [Key(0)] - [CultName] - public string Name = string.Empty; - - [Key(1)] - public int Power; - } - - [CultDocument("tests.update_base", "tests.update_base.v1")] - [MessagePackObject(AllowPrivate = true)] - internal class UpdateBase - { - [Key(0)] - [CultName] - public string Name = string.Empty; - } - - [CultDocument("tests.update_derived", "tests.update_derived.v1")] - [MessagePackObject(AllowPrivate = true)] - internal sealed class UpdateDerived : UpdateBase - { - [Key(1)] - public int Value; - } - } -} diff --git a/tests/GameCult.Caching.Tests/stryker-config.json b/tests/GameCult.Caching.Tests/stryker-config.json deleted file mode 100644 index ac366203..00000000 --- a/tests/GameCult.Caching.Tests/stryker-config.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "stryker-config": { - "project": "GameCult.Caching.csproj", - "mutate": [ - "**/src/GameCult.Caching/**/*.cs" - ], - "mutation-level": "Standard", - "reporters": [ "progress", "html", "json" ], - "thresholds": { "high": 80, "low": 60, "break": 0 } - } -} diff --git a/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs b/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs index 19c08a4b..7e210081 100644 --- a/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs +++ b/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs @@ -4,7 +4,6 @@ using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; -using System.Runtime.InteropServices; using System.Threading; using System.Threading.Tasks; using FluentAssertions; @@ -365,11 +364,11 @@ public sealed class CultMeshSingleFileStoreTests [Test] public void ACacheOpeningDuringMeshWritesNeverSeesAnEmptyStore() { - // Windows replaces with a POSIX-semantics rename first (StoreReplace), which makes a lock-free reader finding the path - // absent rare, not impossible: a stalled rename was measured leaving it absent for 5-47 ms on Windows. So on Windows a - // brief not-found (a read throwing, or a cache opening onto an absent path) is tolerated and reported; elsewhere rename - // is atomic and none is allowed. - var windows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows); + // Windows replaces with ReplaceFile, which leaves a moment with nothing at the path for a lock-free reader: 1,722 empty + // stores in 17,313 opens at b1ff2dcb. That is the store's replace, not Mesh's, and it is the same for a cache flush beside a + // cache open; the fork is in the cut report. + if (OperatingSystem.IsWindows()) + Assert.Ignore("Windows File.Replace is not atomic to a lock-free reader; see cut report fork mesh-reader-windows-replace-window."); var path = Path.Combine(_root, "race.cc"); CultMesh.WriteSingleFileDocument(path, Key, Publication()); @@ -387,7 +386,7 @@ public sealed class CultMeshSingleFileStoreTests Volatile.Write(ref stop, 1); }); - int opens = 0, empty = 0, reads = 0, absent = 0; + int opens = 0, empty = 0, reads = 0; var clock = Stopwatch.StartNew(); while (Volatile.Read(ref stop) == 0 && clock.Elapsed < TimeSpan.FromSeconds(60)) { @@ -398,10 +397,6 @@ public sealed class CultMeshSingleFileStoreTests CultMesh.ReadSingleFileDocumentPayload(path, Key, SchemaId()); reads++; } - catch (Exception ex) when (windows && (ex is FileNotFoundException || ex is DirectoryNotFoundException)) - { - absent++; - } catch (Exception ex) { errors.Enqueue("read: " + ex.GetType().Name + ": " + ex.Message); @@ -409,10 +404,9 @@ public sealed class CultMeshSingleFileStoreTests } writer.Wait(TimeSpan.FromSeconds(60)).Should().BeTrue(); - TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} not-found reads={absent} mesh reads ok={reads} errors={errors.Count}"); + TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} mesh reads ok={reads} errors={errors.Count}"); errors.Take(3).Should().BeEmpty(); - if (!windows) - empty.Should().Be(0, "a cache opened in the window where nothing was at the path"); + empty.Should().Be(0, "a cache opened in the window where nothing was at the path"); opens.Should().BeGreaterThan(0); } diff --git a/tests/GameCult.Networking.Tests/NetworkingTests.cs b/tests/GameCult.Networking.Tests/NetworkingTests.cs index 1aee870d..606d5625 100644 --- a/tests/GameCult.Networking.Tests/NetworkingTests.cs +++ b/tests/GameCult.Networking.Tests/NetworkingTests.cs @@ -7377,183 +7377,6 @@ namespace GameCult.Networking.Tests Assert.That(developmentServer.LogSensitivePayloads, Is.True); } - // Records the flag each PushAll took, and holds nothing: a flush that decided soft itself and never asked the store records nothing. - private sealed class SoftFlagSpyStore : CacheBackingStore - { - public sealed class SpyHome { } - - public readonly List Flags = new(); - public void Stage() => IsDirty = true; - public override void PullAll() { } - public override void Push(CultStoredDocument entry) { } - public override void Delete(CultStoredDocument entry) { } - public override void PushAll(bool soft) => Flags.Add(soft); - public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; - } - - [Test] - public async Task CultNetLocal_Host_FlushAsync_Hands_The_Store_The_Flag_It_Was_Given() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-spy-{Guid.NewGuid():N}.msgpack"); - CultNetHost? host = null; - try - { - host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false }); - var spy = new SoftFlagSpyStore(); - host.Cache.AddBackingStore(spy, typeof(SoftFlagSpyStore.SpyHome)); - spy.Stage(); - - await host.FlushAsync(soft: true); - await host.FlushAsync(); - - Assert.That(spy.Flags, Is.EqualTo(new[] { true, false })); - } - finally - { - host?.Dispose(); - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task CultMeshNode_FlushAsync_Hands_The_Store_The_Flag_It_Was_Given() - { - var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-spy", Guid.NewGuid().ToString("N")); - Directory.CreateDirectory(rootPath); - var documents = new CultNetDocumentRegistry() - .Register(CultNetDocumentBinding.ForDocument()); - using var node = await CultMesh.CreateNodeAsync(Path.Combine(rootPath, "world.ccmp"), new CultMeshNodeOptions - { - StartServer = false, - DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-spy-local", DocumentRegistry = documents } - }); - var spy = new SoftFlagSpyStore(); - node.Cache.AddBackingStore(spy, typeof(SoftFlagSpyStore.SpyHome)); - spy.Stage(); - - await node.FlushAsync(soft: true); - await node.FlushAsync(); - - Assert.That(spy.Flags, Is.EqualTo(new[] { true, false })); - } - - [Test] - public async Task CultNetLocal_Host_SoftFlush_Passes_The_Flag_To_The_Store() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-soft-{Guid.NewGuid():N}.msgpack"); - CultNetHost? host = null; - - try - { - host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false }); - await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "a@example.test", PasswordHash = "hash", Username = "First" }); - await host.FlushAsync(); - var before = File.ReadAllBytes(filePath); - - await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "b@example.test", PasswordHash = "hash", Username = "Second" }); - await host.FlushAsync(soft: true); - Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft host flush rewrote a store it reads"); - - await host.FlushAsync(); - Assert.That(File.ReadAllBytes(filePath), Is.Not.EqualTo(before), "a plain host flush after a skipped soft flush did not land the staged write"); - } - finally - { - host?.Dispose(); - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task CultMeshNode_SoftFlush_Passes_The_Flag_To_The_Store() - { - var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-flush", Guid.NewGuid().ToString("N")); - Directory.CreateDirectory(rootPath); - var cachePath = Path.Combine(rootPath, "world.ccmp"); - var documents = new CultNetDocumentRegistry() - .Register(CultNetDocumentBinding.ForDocument()); - - using var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions - { - StartServer = false, - DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-flush-local", DocumentRegistry = documents } - }); - await node.Database.PutAsync(new CultRecordKey("note:one"), new MeshQuickstartNote { NoteId = "note:one", Body = "one" }); - await node.FlushAsync(); - var before = File.ReadAllBytes(cachePath); - - await node.Database.PutAsync(new CultRecordKey("note:two"), new MeshQuickstartNote { NoteId = "note:two", Body = "two" }); - await node.FlushAsync(soft: true); - Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(before), "a soft node flush rewrote a store it reads"); - - await node.FlushAsync(); - Assert.That(File.ReadAllBytes(cachePath), Is.Not.EqualTo(before), "a plain node flush after a skipped soft flush did not land the staged write"); - } - - // The cache HOLDS a record in every step: a passthrough that decides soft from the cache's memory instead of - // handing the store the flag fails the absent step (it skips and writes nothing) or the unreadable step (it - // never reaches the reader that refuses). - [Test] - public async Task CultNetLocal_Host_SoftFlush_With_A_Held_Record_Writes_Absent_Skips_Readable_Refuses_Unreadable() - { - var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-soft-held-{Guid.NewGuid():N}.msgpack"); - CultNetHost? host = null; - try - { - host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false }); - await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "a@example.test", PasswordHash = "hash", Username = "First" }); - Assert.That(File.Exists(filePath), Is.False); - - await host.FlushAsync(soft: true); - Assert.That(File.Exists(filePath), Is.True, "a soft host flush over an absent store, with a record held, wrote nothing"); - var written = File.ReadAllBytes(filePath); - - await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "b@example.test", PasswordHash = "hash", Username = "Second" }); - await host.FlushAsync(soft: true); - Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(written), "a soft host flush rewrote a readable store"); - - var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; - File.WriteAllBytes(filePath, junk); - Assert.That(async () => await host.FlushAsync(soft: true), Throws.InstanceOf()); - Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(junk)); - } - finally - { - host?.Dispose(); - if (File.Exists(filePath)) File.Delete(filePath); - } - } - - [Test] - public async Task CultMeshNode_SoftFlush_With_A_Held_Record_Writes_Absent_Skips_Readable_Refuses_Unreadable() - { - var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-held", Guid.NewGuid().ToString("N")); - Directory.CreateDirectory(rootPath); - var cachePath = Path.Combine(rootPath, "world.ccmp"); - var documents = new CultNetDocumentRegistry() - .Register(CultNetDocumentBinding.ForDocument()); - using var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions - { - StartServer = false, - DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-held-local", DocumentRegistry = documents } - }); - await node.Database.PutAsync(new CultRecordKey("note:one"), new MeshQuickstartNote { NoteId = "note:one", Body = "one" }); - Assert.That(File.Exists(cachePath), Is.False); - - await node.FlushAsync(soft: true); - Assert.That(File.Exists(cachePath), Is.True, "a soft node flush over an absent store, with a record held, wrote nothing"); - var written = File.ReadAllBytes(cachePath); - - await node.Database.PutAsync(new CultRecordKey("note:two"), new MeshQuickstartNote { NoteId = "note:two", Body = "two" }); - await node.FlushAsync(soft: true); - Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(written), "a soft node flush rewrote a readable store"); - - var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; - File.WriteAllBytes(cachePath, junk); - Assert.That(async () => await node.FlushAsync(soft: true), Throws.InstanceOf()); - Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(junk)); - } - [Test] public async Task CultNetLocal_CreateHostAsync_Wires_Durable_Cache_Without_Starting_Server() { diff --git a/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs b/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs index ad7dcaca..3a076d16 100644 --- a/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs +++ b/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs @@ -341,7 +341,7 @@ namespace GameCult.Networking.Tests { var path = Path.Combine(RepoRoot(), "contracts", "cultnet", "interop", "selection-vectors.rs-written.json"); if (!File.Exists(path)) - Assert.Inconclusive($"{path} is missing - run cargo test --locked -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first."); + Assert.Inconclusive($"{path} is missing - run cargo test -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first."); var (registry, rows, _) = BuildFixture(); using var document = JsonDocument.Parse(File.ReadAllText(path)); diff --git a/tests/vectors/content-version/README.md b/tests/vectors/content-version/README.md deleted file mode 100644 index cddd7bff..00000000 --- a/tests/vectors/content-version/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# content-version vectors - -Written only by the C# reference, `tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs`, with -`CULTCACHE_WRITE_VECTORS=1` (never set in CI). Every other run opens the committed `store.cc`, compares what the reference reads -to `versions.msgpack` and compares the store bytes the reference writes to the file; a changed vector is a reviewed commit. -Other runtimes compute the version of each record of `store.cc` and must equal these rows -(`variants:ruling:write-set-content-version-declared-ownership`). - -RULE: the content version of a stored record is the lowercase hex SHA-256 of its persisted encoding, the MessagePack array -`[key, schemaId, storedAt, payload bytes]` exactly as the store writes the record. - -## Files - -- `store.cc`: a single-file store of one type, `tests.cv_item` (one string slot, `[0: Text]`), holding `alpha` = "first" and - `beta` = "second", stored-at pinned to `2026-01-01T00:00:00Z`. -- `versions.msgpack`: an array of `[key, version]`, sorted by key, the version the reference reads for the record - (`GetStored(key).StoredVersion`). diff --git a/tests/vectors/content-version/store.cc b/tests/vectors/content-version/store.cc deleted file mode 100644 index 3962430a..00000000 Binary files a/tests/vectors/content-version/store.cc and /dev/null differ diff --git a/tests/vectors/content-version/versions.msgpack b/tests/vectors/content-version/versions.msgpack deleted file mode 100644 index 6584d69a..00000000 --- a/tests/vectors/content-version/versions.msgpack +++ /dev/null @@ -1 +0,0 @@ -’’¥alphaÙ@d330976df4f3946d0ad590d2091666aed4d5ebeac5d473792b4ef401b25c92fd’¤betaÙ@878026249c0df9a969ac1cac67bcdaab61f1b085ce4916c3c63f0fbef8f6da9f \ No newline at end of file diff --git a/tests/vectors/document-variants-c2a/old-reader-refusal.sh b/tests/vectors/document-variants-c2a/old-reader-refusal.sh index 5b9165f0..bfafa01d 100755 --- a/tests/vectors/document-variants-c2a/old-reader-refusal.sh +++ b/tests/vectors/document-variants-c2a/old-reader-refusal.sh @@ -111,7 +111,6 @@ probe() { node ../../node_modules/typescript/bin/tsc -p tsconfig.test.json node --test dist-test/test/old-reader-v3.test.js ;; python) PYTHONPATH=packages/cultcache-py/src python3 packages/cultcache-py/tests/test_old_reader_v3.py -v ;; - # The staged tree is the frozen cultcache-rs-v0.3.1 revision, which has no committed Cargo.lock: --locked cannot work here. rust) cd packages/cultcache-rs && cargo test --test old_reader_v3 -- --nocapture ;; *) usage ;; esac diff --git a/unity/org.gamecult.cultlib/CHANGELOG.md b/unity/org.gamecult.cultlib/CHANGELOG.md index 612aee04..a4a2bc11 100644 --- a/unity/org.gamecult.cultlib/CHANGELOG.md +++ b/unity/org.gamecult.cultlib/CHANGELOG.md @@ -6,7 +6,6 @@ All notable changes to this package are documented in this file. ### Breaking -- `CacheBackingStore.PushAll()` is no longer abstract: it is a non-virtual convenience over the new abstract `PushAll(bool soft)`. A store kind that overrides `PushAll()` no longer compiles; override `PushAll(bool soft)` instead. A soft flush writes only where nothing is at the path: over a store the reader reads, empty included, it returns without writing and leaves the staged writes staged and dirty for a later plain flush; over a file the reader cannot read it refuses. `CultCache.FlushAsync(bool soft = false)`, `CultCache.FlushAllBackingStores(bool soft = false)`, `CultMeshNode.FlushAsync(soft)` and `CultNetHost.FlushAsync(soft)` pass the flag to the stores; `soft: true` was accepted and ignored before, so a caller that passed it now stops landing writes on an existing store. Semver: breaking for C# consumers with their own store kind. - A document payload with bytes after its first MessagePack value is refused on read and on write (`MessagePackSerializationException` naming the type and the count of bytes after the value, never the bytes); the tail is not walked, so a tail nested past the limit is refused as bytes after the value, not as depth. Before, the tail was ignored on read and dropped on the next write. A payload that ends early is still the deserializer's to refuse. A CDN manifest is a record payload and follows the same rule: the acceptance vectors `trailing_nil`, `trailing_reserved_code` and `trailing_truncated_array` in `contracts/cultmesh/content-acceptance-vectors.json` are now refusals. `CultMeshProviderSessionWire.DecodeCommandDocument` refuses a command record whose payload has bytes after its first value (`MessagePackSerializationException` with the count); before, the tail was ignored. Semver: breaking for C# consumers, a major bump. - A single-file store holding two records under one key is unreadable: the reader refuses it with `CultStoreUnreadableException` naming the key (keys compare by ordinal code units, no case folding or normalisation), so a cache, `ReadStoreSnapshot` and a flush refuse the file and leave it as it is. Before, the later record silently replaced the earlier. A directory store reads its manifest through the same reader and refuses a duplicate key the same way. Repair the duplicate with the runtime that wrote it before reading the store here. - A record whose payload carries a slot past the slots its reading type declares, at any depth (the record, a nested object, a list element, a union arm or a unit arm), is loaded read-only: it is served, and every staged write or removal of its key (`Upsert`, `UpsertVariant`, `Remove`, a batch, and `Flatten` of a variant) is refused with `CultSchemaConflictException` before anything is staged, and the store keeps the record byte for byte. Before, the extra slots were skipped on read and dropped on the next write. Declaring an id compatible consents only to depth-0 slots the schema comparison reports ignored under that id; any other extra slot keeps the record read-only whatever is declared, including under the reader's own id. A variant overriding a slot its type does not have is read-only too (it used to be shed on flush). Write such a record from the build that declares those slots. Missing trailing slots still default and stay writable.