diff --git a/.github/workflows/cultmesh-portability.yml b/.github/workflows/cultmesh-portability.yml index 0b8aca1d..bafe70fc 100644 --- a/.github/workflows/cultmesh-portability.yml +++ b/.github/workflows/cultmesh-portability.yml @@ -158,7 +158,7 @@ jobs: # (only the interop example built, inside the C# store tests). The target directory is under RUNNER_TEMP, away from the checkout. - name: Verify the CultCache document owner with the Rust cache shell: bash - run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --manifest-path packages/cultcache-rs/Cargo.toml + run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml # tests/GameCult.Caching.Tests, the C# reference cache's own suite (about 1,260 tests, under a minute), ran in no # forge workflow; only the Mesh tests above exercised the reference. diff --git a/.github/workflows/cultnet-interop.yml b/.github/workflows/cultnet-interop.yml index 8f85506e..0d193aef 100644 --- a/.github/workflows/cultnet-interop.yml +++ b/.github/workflows/cultnet-interop.yml @@ -29,6 +29,12 @@ jobs: steps: - name: Check out CultLib uses: actions/checkout@v4 + with: + # packages/cultnet-py/tests/test_declared_dependency_ranges.py reads the cultcache-py + # release tags and their file contents with `git show`; a shallow checkout has no tags + # and the test skips itself. + fetch-depth: 0 + fetch-tags: true - name: Set up .NET uses: actions/setup-dotnet@v4 @@ -129,10 +135,10 @@ jobs: python -m unittest discover -s packages/cultmesh-py/tests - name: Verify Rust RUDP reliability - run: cargo test --manifest-path packages/cultnet-rs/Cargo.toml rudp + run: cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml rudp - name: Verify Rust CultMesh runtime-local surface - run: cargo test --manifest-path packages/cultmesh-rs/Cargo.toml + run: cargo test --locked --manifest-path packages/cultmesh-rs/Cargo.toml - name: Verify C# RUDP reliability run: dotnet test tests/GameCult.Networking.Tests/GameCult.Networking.Tests.csproj --filter "FullyQualifiedName~Rudp" diff --git a/README.md b/README.md index b916cabe..a555aa92 100644 --- a/README.md +++ b/README.md @@ -212,8 +212,8 @@ identity-owned C# session after provider route replacement. Rust package tests: ```powershell -cargo test --manifest-path packages/cultcache-rs/Cargo.toml -cargo test --manifest-path packages/cultnet-rs/Cargo.toml +cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml +cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml ``` Kotlin package build: diff --git a/contracts/cultnet/interop/write-receipt-vectors.json b/contracts/cultnet/interop/write-receipt-vectors.json index e95c3180..04f50309 100644 --- a/contracts/cultnet/interop/write-receipt-vectors.json +++ b/contracts/cultnet/interop/write-receipt-vectors.json @@ -115,7 +115,7 @@ "validPutPayloadHex": "81A16101", "whiteSpaceCodePoints": ["0009","000A","000B","000C","000D","0020","0085","00A0","1680","2000","2001","2002","2003","2004","2005","2006","2007","2008","2009","200A","2028","2029","202F","205F","3000"], "notWhiteSpaceCodePoints": ["200B","200C","200D","2060","FEFF","180E","001C","001D","001E","001F"], - "codePointCases": "For every code point in whiteSpaceCodePoints, each of schemaId, recordKey, storedAt, sourceRuntimeId, sourceAgentId, sourceRole and one tags entry, set alone to a string of just that code point (and to that code point repeated three times), is refused; for every code point in notWhiteSpaceCodePoints the same string is not blank and the put is admitted.", + "codePointCases": "For every code point in whiteSpaceCodePoints, each of schemaId, recordKey, storedAt, sourceRuntimeId, sourceAgentId, sourceRole and one tags entry, set alone to a string of just that code point (and to that code point repeated three times), is refused; for every code point in notWhiteSpaceCodePoints the same string in each of those fields except storedAt is not blank and the put is admitted (a storedAt must be a stamp: a non-blank storedAt that is no stamp is refused message_invalid, field document, after the write authority, and is covered by tests/vectors/stored-at-stamp).", "cases": [ { "name": "valid_document_is_admitted", "set": {}, "outcome": "admitted" }, { "name": "empty_schema_id", "set": { "schemaId": "" }, "outcome": "refused" }, @@ -140,7 +140,7 @@ { "name": "zero_width_space_schema_id_is_a_name", "set": { "schemaId": "\u200b" }, "outcome": "admitted" }, { "name": "zero_width_space_record_key_is_a_name", "set": { "recordKey": "\u200b" }, "outcome": "admitted" }, { "name": "zero_width_space_tag_is_a_name", "set": { "tags": ["\u200b"] }, "outcome": "admitted" }, - { "name": "bom_stored_at_is_a_name", "set": { "storedAt": "\ufeff" }, "outcome": "admitted" }, + { "name": "bom_record_key_is_a_name", "set": { "recordKey": "\ufeff" }, "outcome": "admitted" }, { "name": "mongolian_vowel_separator_source_role_is_a_name", "set": { "sourceRole": "\u180e" }, "outcome": "admitted" }, { "name": "padded_name_is_a_name", "set": { "schemaId": "\u00a0a\u2003", "tags": [" a"] }, "outcome": "admitted" }, { "name": "file_separator_schema_id_is_a_name", "set": { "schemaId": "\u001c" }, "outcome": "admitted" }, diff --git a/contracts/cultnet/interop/write-receipt.md b/contracts/cultnet/interop/write-receipt.md index bfcbbb86..121f77f1 100644 --- a/contracts/cultnet/interop/write-receipt.md +++ b/contracts/cultnet/interop/write-receipt.md @@ -40,7 +40,7 @@ A connection is receipt-capable when its `cultnet.hello.v0` (sent before its fir - **What is remembered.** Per entry a server keeps only fixed-size digests (SHA-256) of the `messageId` and of the content hash, plus the answer (`admitted`, or the refusal's code and details). Memory per entry is bounded independent of id length and payload size. Lookup compares digests. - **Remembered-write limits.** The per-session table holds at most 1024 entries. A server MAY also enforce a server-wide memory budget across all sessions (reference: 32 MiB of remembered entries). A NEW write (a key not already in the table) that would exceed either bound is refused `limit_reached` for that write alone, `messageId` the write's id, `details` all null; the write is not applied and the session continues. A resend of a remembered key is still answered from the table. - **Limit refusals are remembered.** A `limit_reached` refusal is remembered like any other: a resend of the same id and content hash replays `limit_reached`. No slot is freed mid-session, so an id refused at a bound never becomes admitted later in that session (a server at the cap keeps the refused digest pair in a bounded refusal set of at most 1024 per session, outside the entry budget; past it, a further new write is refused `limit_reached` without being remembered, and the session refuses every id it has not seen for the rest of its life, even if a server-wide budget has since freed, so a refusal that was not remembered is never followed by an admission of the same id). The client mints a new id on a new session. - - **Required fields.** A put or delete whose document or `schemaId` is null or missing is refused `message_invalid` with `details.field` `"document"` or `"schemaId"` (the first missing in that order) and fixed text; the write is not applied. A put or delete that is present but names nothing is not "missing": the write is refused `message_invalid` exactly as a frame that does not read, with `details.field` `"message"`, the fixed unreadable text and the write's id (vector `error_v1_empty_schema_id_is_unreadable_and_names_the_write`), and is never applied; a server must not hand it to its store. Names nothing means, on a put or delete, a `schemaId` or `recordKey` that is blank; on a put's document, also a `storedAt` that is blank, a `payload` of zero bytes (a MessagePack value is at least one byte, so an empty payload is no document), a `sourceRuntimeId`, `sourceAgentId` or `sourceRole` that is present and blank, and a `tags` entry that is blank. **Blank** means empty or made only of Unicode White_Space code points (the PropList property) and nothing else, in every runtime, whatever its platform's own whitespace test says. The set is exactly U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. U+200B, U+200C, U+200D, U+2060, U+FEFF, U+180E and U+001C to U+001F are not White_Space (some platforms' whitespace tests accept U+001C to U+001F or U+180E, for example Java's `Character.isWhitespace` and older .NET, so a runtime must check the list, not its platform test): a string of them names something. The vectors list both sets (`whiteSpaceCodePoints`, `notWhiteSpaceCodePoints`) and a runtime checks every code point in every covered field. Absent or nil source ids and tags, and an empty `tags` list, are valid. The rule holds on every connection, receipt-capable or not (a legacy connection is refused with the `error.v0` of the same code, details and text); the vectors' `blankFields` section lists each case. + - **Required fields.** A put or delete whose document or `schemaId` is null or missing is refused `message_invalid` with `details.field` `"document"` or `"schemaId"` (the first missing in that order) and fixed text; the write is not applied. A put or delete that is present but names nothing is not "missing": the write is refused `message_invalid` exactly as a frame that does not read, with `details.field` `"message"`, the fixed unreadable text and the write's id (vector `error_v1_empty_schema_id_is_unreadable_and_names_the_write`), and is never applied; a server must not hand it to its store. Names nothing means, on a put or delete, a `schemaId` or `recordKey` that is blank; on a put's document, also a `storedAt` that is blank, a `payload` of zero bytes (a MessagePack value is at least one byte, so an empty payload is no document), a `sourceRuntimeId`, `sourceAgentId` or `sourceRole` that is present and blank, and a `tags` entry that is blank. **Blank** means empty or made only of Unicode White_Space code points (the PropList property) and nothing else, in every runtime, whatever its platform's own whitespace test says. The set is exactly U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. U+200B, U+200C, U+200D, U+2060, U+FEFF, U+180E and U+001C to U+001F are not White_Space (some platforms' whitespace tests accept U+001C to U+001F or U+180E, for example Java's `Character.isWhitespace` and older .NET, so a runtime must check the list, not its platform test): a string of them names something. The vectors list both sets (`whiteSpaceCodePoints`, `notWhiteSpaceCodePoints`) and a runtime checks every code point in every covered field. A put's `storedAt` is the exception to "present and not blank is a name": it must be a stamp, so a non-blank `storedAt` that is no stamp (`tests/vectors/stored-at-stamp`) is refused `message_invalid` with `details.field` `"document"`, after the write authority has decided (an ungranted peer is told `write_not_authorized` first), and nothing is stored or minted; only a blank `storedAt` is refused as unreadable with field `"message"`, before the authority. The non-White_Space code point rows therefore name something in every covered field except `storedAt`. Absent or nil source ids and tags, and an empty `tags` list, are valid. The rule holds on every connection, receipt-capable or not (a legacy connection is refused with the `error.v0` of the same code, details and text); the vectors' `blankFields` section lists each case. - **Fixed texts.** The `error` string of these refusals is fixed, so vectors can compare it: `message_invalid: messageId exceeds 256 bytes`, `message_invalid: document is required`, `message_invalid: schemaId is required`, `message_invalid: the message could not be read` (a frame that does not decode, or a write that names nothing; the text is the same for both and never carries a decoder's own wording), `message_invalid: receipt sessions require the reliable channel`, `limit_reached: the session's remembered-write limit is full`. - **Reliable channel only.** A write is accepted on a receipt session only on a channel that is reliable and not sequenced (reliable ordered and reliable unordered both qualify; a sequenced channel drops superseded packets, so a write on it can vanish without trace). A write arriving on any other channel is refused `message_invalid` with `details.field: "channel"` (the transport profile's word for it) and fixed text. Receipts and refusals travel the reliable channel; a write that could be lost without trace cannot be answered once. The server refuses a sequenced write whenever it can read the write's id: from the delivered frame, and from a superseded packet that carries its whole frame. A write sent as several fragments whose frame is superseded is dropped whole and may go unanswered; the client's unconfirmed outcome covers it. A repeat of a packet the server already took, or a packet too far ahead to be taken, draws no further answer (rule 1). - Past the session (or a server's longer retention, if it keeps one) a resend is a new write. diff --git a/docs/cultcache-union-parity-map.md b/docs/cultcache-union-parity-map.md index 368be16a..1442abdc 100644 --- a/docs/cultcache-union-parity-map.md +++ b/docs/cultcache-union-parity-map.md @@ -259,7 +259,7 @@ landed in `ead5260c`, Python in `5618d8dd`, Rust in `c028c511`. TypeScript is no | Runtime | Owner of the decision (constant and walker) | Doors routed through it | |---|---|---| | C# | `CultPayloadDepth.Check` (`src/GameCult.Caching.MessagePack/CultPayloadDepth.cs`), limit `CultMessagePackSecurity.MaximumPayloadDepth`. MessagePack's own graph counter is set to the same value as a backstop only. | Write: `CultDocumentMessagePackSerialization.SerializeUntyped`. Read: `DeserializeUntyped`. The cache store (`SerializePayload`/`DeserializePayload`), the directory store, `CultMesh` and single-file documents all call those two; C# has no separate raw-envelope or prepared-batch door. | -| Rust | `cultcache_rs::check_payload_depth` and `MAX_PAYLOAD_DEPTH` (`packages/cultcache-rs/src/lib.rs`). | Read: `decode_payload` (get, list, `validate_envelope`/`put_envelope`, `finish_prepared_entry`). Write: `put`/`prepare_entry`/`prepare_entry_named` reach it through `finish_prepared_entry`, which decodes what it just encoded. Raw envelope: `put_raw_envelope`, the door CultMesh network puts use (`cultmesh-rs` `lib.rs`, two call sites). Prepared batch: `put_prepared_batch` via `require_payload_depth`. cultnet-rs: every free-form or skippable wire payload decodes through `cultcache_rs::decode_payload` (message decode in `contracts.rs`, `provider_session`, `media_stream_wire`, `replication`, `service_identity`). | +| Rust | `check_record_payload` (private; depth, then no bytes after the first value) over `first_value_end`, and `MAX_PAYLOAD_DEPTH`; `cultcache_rs::check_payload_depth` is the depth-only probe for outside callers (`packages/cultcache-rs/src/lib.rs`). | Read: `decode_payload` (get, list, `validate_envelope`/`put_envelope`, `finish_prepared_entry`). Write: `put`/`prepare_entry`/`prepare_entry_named` reach it through `finish_prepared_entry`, which decodes what it just encoded. Raw envelope: `put_raw_envelope`, the door CultMesh network puts use (`cultmesh-rs` `lib.rs`, two call sites). Prepared batch: `put_prepared_batch` via `require_record_payload`. Load: `pull_all_backing_stores` and `load_soa` run `require_record_payload` on every record, so a store holding a payload past the bound (or with bytes after it) is refused at load, and the cache then refuses every write and removal (`CultCacheLoadRefused`) until a load succeeds. cultnet-rs: every free-form or skippable wire payload decodes through `cultcache_rs::decode_payload` (message decode in `contracts.rs`, `provider_session`, `media_stream_wire`, `replication`, `service_identity`). | | Python | `_refuse_past_depth_limit` and `MAX_PAYLOAD_DEPTH` (`packages/cultcache-py/src/cultcache_py/documents.py`); a msgpack `StackError` on read maps to the same error. | `DocumentDefinition.encode_payload` (write) and `decode_payload` (read). `CultCache.put`, `put_envelope`, `put_envelopes` and load-time hydration go through them; there is no raw-envelope door. cultmesh-py round trips call the same pair. | - **Doors deliberately not routed.** cultnet-rs `idunn_lifecycle_brake`, `provider_health_contracts` and diff --git a/docs/document-variants-cut.md b/docs/document-variants-cut.md index e9dc51ed..f363521f 100644 --- a/docs/document-variants-cut.md +++ b/docs/document-variants-cut.md @@ -1620,7 +1620,7 @@ The one door per runtime, and who walks through it: - **Shared paths.** Open, flush, commit, `ReadDurable`, `ReplaceDurable`. Rust `pull_all` RS:1243 also checks `is_absent` on the store and on the lock path before it takes the shared lock; that skips lock creation only, and the verdict stays with the reader. - **Cut line.** `merge-r1` h2 (C#) and h4/h6 (TS `readStoreBytes`); `mesh-single-file-reader` h1. - **Pinned by.** C#: `StoreReadabilityTests` `ADanglingFileLinkIsAnIoErrorOnOpenNotAnEmptyStore` :208, `ADanglingDirectoryLinkIsAnIoErrorOnOpenNotAnEmptyStore` :185, and the commit variants :196, :219. Mesh: `ADanglingLinkIsAnIoErrorOnTypedWriteRawWriteAndReadNeverAMissingFile` :107 and `OnlyNothingAtThePathIsAMissingFile` :134 (`CultMeshSingleFileStoreTests.cs`). Rust: `a_dangling_symlink_at_the_store_path_is_an_error_and_nothing_is_written` RS:5832. TS: "readStoreBytes reports nothing at the path only when the lstat finds nothing too". -- **Known gaps.** The writer side still tolerates a dangling link: Rust `write_all_unlocked` RS:992-996 and TS `pushAll` TS:94-100 replace it on a whole-store write (`variants:follow_up:writer-dangling-link-tolerance`, owner `c2a-write-set`). What a replace does to a live link is `variants:follow_up:store-replace-cut` item 2. Windows junction behavior was proven at `42f05f66` and `b1ff2dcb`, not at the final head. +- **Known gaps.** The writer no longer tolerates a dangling link: the reader's refusal stands for every write, whole-store `pushAll` included (Rust `a_push_all_through_a_dangling_link_is_refused_like_push_and_writes_nothing`, TS `a pushAll through a dangling link is refused like push`; `variants:follow_up:writer-dangling-link-tolerance`). What a replace does to a live link is `variants:follow_up:store-replace-cut` item 2. Windows junction behavior was proven at `42f05f66` and `b1ff2dcb`, not at the final head. ### C. Zero-byte file @@ -1672,7 +1672,7 @@ The one door per runtime, and who walks through it: - **Shared paths.** TS `pushAll` and Rust `push_all` only. - **Cut line.** `merge-r1` h4 (TS) and h6 (Rust). - **Pinned by.** The manifest walks run a soft flush over every vector in TS (the `softPushAll` operation, TS test :2336) and Rust (`soft_push_all`, RS:5149), each with an absent and a dangling-link case; `a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link` RS:6019. -- **Known gaps.** C# and Python have no soft decision, and the C# Mesh flag is dead; no follow-up is registered (h6 `undone` names it). `writer-dangling-link-tolerance` applies to the non-soft branch. +- **Known gaps.** C# and Python have no soft decision, and the C# Mesh flag is dead; no follow-up is registered (h6 `undone` names it). `writer-dangling-link-tolerance` is settled: the non-soft branch refuses a dangling link too. ### What no cut owns yet diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index 8db67be5..9136b667 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -3,7 +3,7 @@ CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local tool in `dotnet-tools.json`. Coverage is added project by project; a project with no `stryker-config.json` next to its test project has not been scoped -yet (see "Follow-ups" below for what's outstanding). +yet (see "Scope and follow-ups" below for what's outstanding). ## Running it @@ -126,14 +126,17 @@ project as precedent for the `vstest` caveat itself - that caveat is `mtp`-runner-specific and documented above for `static readonly` field initializers, and GameCult.Geometry has never actually hit it. -## Follow-ups +## Scope and follow-ups -CultMath.Tests and GameCult.Geometry.Tests are the only projects currently -scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner` -entry: the project is NUnit 4 with `NUnit3TestAdapter` and +CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the +projects currently scoped. GameCult.Geometry's `stryker-config.json` has no +`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and `Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3 -suite). +suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`) +and its `stryker-config.json` is driven the same way, by the repo-local +`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run +dotnet-stryker` from `tests/GameCult.Caching.Tests`. GameCult.Geometry has only ever been run `--since:`-scoped to the cut that added surface nets; a full-project baseline (running mutation against all of @@ -151,7 +154,83 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that run; a future cut whose diff is C#-only, against a base commit that already has `stryker-config.json`, should scope correctly. -`GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), +### GameCult.Caching.Tests + +`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of +`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard +mutation level, `break` 0, the same thresholds as Geometry). Caching was +scoped before Networking because its suite is deterministic and quick (1415 +tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking +is about 20k lines and its tests lean on socket timing, so a mutant timing out +there says little about the test. A full Caching run is about 2,900 candidate +mutants (3,602 created, minus 730 that fail to compile) at 1.25 to 1.6 s +each, so about 1 to 1.3 hours; run it detached and read the JSON report, +never as one attached session. + +The first measurement was a narrow pass over the three files that carry the +store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`: + +```bash +dotnet tool run dotnet-stryker \ + -m '**/src/GameCult.Caching/StoreReplace.cs' \ + -m '**/src/GameCult.Caching/CultStoreLinks.cs' \ + -m '**/src/GameCult.Caching/CultElementIds.cs' +``` + +A clean single run of that command tests 243 to 265 mutants (the count moves with the revision) and scored 85.28% on +568a60fa, before any test of this scope was added; 86.79% on d38b5728, with +the first batch of tests; and 87.17% on 76d9e7d1, with the throwing-enumerator +case, in about 10 minutes. The first measurement of it overlapped two Stryker +runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`, +where tests that kill a mutant by hand reported `Survived` or `Timeout`) are +not measurements and are not quoted here. Never run two Stryker runs against +one checkout at once, and check `ps` before starting one. A clean single run +on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208 +`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor +was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now +kills; the other 11 are equivalent mutants. With that case a clean single run +on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%. + +Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each +failing when its mutant is applied by hand): the order of the id path (the +codec refuses at the first type MessagePack cannot allow, so order is what an +error names first; `Append` to `Prepend` at 163, 167 and 180); the +union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so +the red is a crashed test host: a depth guard would need production code in +`ConcreteTypes`, since the recursion is private and a stack overflow cannot be +caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an +id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the +`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter, +`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the +`[DataContract]` without `[MessagePackObject]` and `||` arms). + +Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full +suite; none can be told apart by any test: + +- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to + `true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument + changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument + (`DataContractAttribute` is `Inherited = false`). +- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which + are never empty. +- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial + capacity. +- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`; + `Classify` unwraps nullables itself, so the unwrap is dead. A cut + candidate, not a test gap. +- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and + `MapTypes`; `type` matches only when it is itself `IEnumerable<>`, + `IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second + matching interface, so the position of `type` never decides the answer. + +The Windows-only P/Invoke group is exercised only on Windows: its mutants are +`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a +Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its +negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the +negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the +same result. + +`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), `GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes (`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own `stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for diff --git a/docs/start/README.md b/docs/start/README.md index 6a7d6d74..c478467c 100644 --- a/docs/start/README.md +++ b/docs/start/README.md @@ -16,10 +16,10 @@ this page. | | CultCache | CultMath | Geometry | CultNet | CultMesh | | --- | --- | --- | --- | --- | --- | -| **C#** | [first call, tested](#c-first-call) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` | -| **TypeScript** | [first call, tested](#typescript-first-call) | not yet | not yet | [first call, tested](#typescript-cultnet-first-call); [package](../../packages/cultnet-ts/README.md) | [package](../../packages/cultmesh-ts/README.md); snippet not yet; [recipe: two-client .io game](io-game.md), tested with two live clients, browser page not yet run | -| **Rust** | [first call, tested](#rust-first-call) | [native core](../../packages/cultmath/native/cultmath-core); snippet not yet | not yet | [first call, tested](#rust-cultnet-first-call); [package](../../packages/cultnet-rs/README.md), [examples](../../packages/cultnet-rs/examples) | [package](../../packages/cultmesh-rs); snippet not yet | -| **Python** | [first call, tested](#python-first-call) | not yet | not yet | [first call, tested](#python-cultnet-first-call); [package](../../packages/cultnet-py/README.md) | [package](../../packages/cultmesh-py/README.md); snippet not yet | +| **C#** | [first call, tested](#c-first-call); [update, tested](#c-update-with-retry) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` | +| **TypeScript** | [first call, tested](#typescript-first-call); [update, tested](#typescript-update-with-retry) | not yet | not yet | [first call, tested](#typescript-cultnet-first-call); [package](../../packages/cultnet-ts/README.md) | [package](../../packages/cultmesh-ts/README.md); snippet not yet; [recipe: two-client .io game](io-game.md), tested with two live clients, browser page not yet run | +| **Rust** | [first call, tested](#rust-first-call); [update, tested](#rust-update-with-retry) | [native core](../../packages/cultmath/native/cultmath-core); snippet not yet | not yet | [first call, tested](#rust-cultnet-first-call); [package](../../packages/cultnet-rs/README.md), [examples](../../packages/cultnet-rs/examples) | [package](../../packages/cultmesh-rs); snippet not yet | +| **Python** | [first call, tested](#python-first-call); [update, tested](#python-update-with-retry) | not yet | not yet | [first call, tested](#python-cultnet-first-call); [package](../../packages/cultnet-py/README.md) | [package](../../packages/cultmesh-py/README.md); snippet not yet | For the whole picture of what each piece is for, see the @@ -70,6 +70,37 @@ public static class FirstCall `FirstCall.Run` is the call; a second test in that file fails if the block above stops matching it. More: `src/GameCult.Caching/README.md`. +## C# update with retry + +Change a record you may not be the only one changing. `Update` reads the record as its store holds it, hands your function a fresh +copy of it, and commits the result only if the record is still the one it read. If another writer got there first, it +reads again and runs your function again, up to five times. The source is +[`tests/GameCult.Caching.Tests/UpdateCallTests.cs`](../../tests/GameCult.Caching.Tests/UpdateCallTests.cs); the +snippet reuses `Item` from the first call. + +```csharp +using GameCult.Caching; +using GameCult.Caching.MessagePack; + +public static class UpdateCall +{ + public static CultUpdateResult Restock(string path, CultRecordKey key) + { + var cache = CultCacheMessagePack.Create(path); + return cache.Update(key, item => + { + item ??= new Item { Name = "Potion", Category = "Consumable" }; + item.Value += 10; + return item; + }); + } +} +``` + +The write is durable when `Update` returns, with no `PushAll`. A lost race re-runs the function, so keep it free of +side effects; when the attempts run out the result says `LostRace` instead of throwing. `UpsertAsync` plus +`PushAll` is the blind opt-out: last writer wins. + ## TypeScript first call Same shape in TypeScript: the document is a name, a schema id and a `parse`-able schema (zod here), @@ -119,12 +150,36 @@ async function firstCall(path: string) { The test imports from `../src/index` where the block reads `@gamecult/cultcache-ts`; a second test in that file fails if the block stops matching the rest. More: `packages/cultcache-ts/README.md`. +## TypeScript update with retry + +`update` changes a record from what the store holds now and commits only if it is still that record. The source is the +second region of +[`packages/cultcache-ts/test/first-call.test.ts`](../../packages/cultcache-ts/test/first-call.test.ts), which uses the +definitions of the first call above; a test in that file fails if the block stops matching the file. + +```ts +async function restock(path: string) { + const cache = CultCache.builder() + .withRegistry(registry) + .withGenericStore(new SingleFileMessagePackBackingStore(path)) + .build(); + await cache.pullAllBackingStores(); + return cache.update(itemDocument, "item:potion", (item) => { + const current = item ?? { name: "Potion", category: "Consumable", value: 0 }; + return { ...current, value: current.value + 10 }; + }); +} +``` + +The write is durable when the promise resolves. A lost race re-runs the function, so keep it free of side effects; when the +attempts run out the result says `lostRace` instead of throwing. `put` is the blind opt-out: last writer wins. + ## Rust first call Same shape with `derive(DatabaseEntry)`: slots are `#[cultcache(key = N)]`, so the `.cc` payload is the slot-indexed array the C# runtime writes. The source is [`packages/cultcache-rs/tests/first_call.rs`](../../packages/cultcache-rs/tests/first_call.rs), run by -`cargo test --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block +`cargo test --locked --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block below stops matching the file. The Rust runtime has no name lookup, so the read-back is by key. More: `packages/cultcache-rs/README.md`. @@ -165,6 +220,36 @@ fn first_call() -> anyhow::Result<()> { } ``` +## Rust update with retry + +`update` changes a record from what the store holds now and commits only if it is still that record. The source is the +second region of +[`packages/cultcache-rs/tests/first_call.rs`](../../packages/cultcache-rs/tests/first_call.rs), which uses the `Item` of +the first call above; a test in that file fails if the block stops matching the file. + +```rust +fn restock(path: &Path) -> anyhow::Result> { + let mut cache = CultCache::new(); + cache.register_entry_type::()?; + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(path))?; + cache.pull_all_backing_stores()?; + cache.update::("item:potion", |item| { + // The store's copy of the record, or nothing if it has none; this may run again on a lost race. + let mut item = item.unwrap_or(Item { + name: "Potion".to_string(), + category: "Consumable".to_string(), + value: 0, + }); + item.value += 10; + Some(item) + }) +} +``` + +The write is durable when `update` returns. A lost race re-runs the closure, so keep it free of side effects; when the +attempts run out (five, or the bound `update_with_attempts` takes) the outcome says `LostRace` instead of failing, and +returning `None` writes nothing and says `Declined`. `put` is the blind opt-out: last writer wins. + ## Python first call A document is a definition with an encoder and decoder; `name=` registers the name lookup, `indexes=` unique lookups and @@ -220,6 +305,29 @@ def first_call() -> Item: return reopened.get_required(item_doc, "item:potion") ``` +## Python update with retry + +`update` changes a record from what the store holds now and commits only if it is still that record. The source is the +`update` region of +[`packages/cultcache-py/tests/test_first_call.py`](../../packages/cultcache-py/tests/test_first_call.py), which uses the +definitions of the first call above; a test in that file fails if the block stops matching the file. + +```python +def restock(path: Path): + cache = ( + CultCache.builder() + .register_document_type(item_doc) + .add_generic_store(SingleFileMessagePackBackingStore(path)) + .build() + ) + cache.pull_all_backing_stores() + return cache.update(item_doc, "item:potion", lambda item: Item("Potion", "Consumable", (item.value if item else 0) + 10)) +``` + +The write is durable when `update` returns. A lost race re-runs the function, so keep it free of side effects; when the +attempts run out the result's `outcome` is `UpdateOutcome.LOST_RACE` instead of a raise. `put` is the blind opt-out: last +writer wins. + ## CultNet first calls CultNet is schema-v0 messages in length-prefixed frames. The first call in each runtime is the same @@ -341,7 +449,7 @@ async function firstCall() { ### Rust CultNet first call Source: [`packages/cultnet-rs/tests/first_call.rs`](../../packages/cultnet-rs/tests/first_call.rs), run by -`cargo test --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`. +`cargo test --locked --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`. ```rust use std::net::{TcpListener, TcpStream}; diff --git a/docs/write-ownership-map.md b/docs/write-ownership-map.md index 44452dc6..79140c2b 100644 --- a/docs/write-ownership-map.md +++ b/docs/write-ownership-map.md @@ -107,8 +107,8 @@ Authority map (also in the spec): - `variants:follow_up:public-store-writes-behind-cache`: **leave**. It asks who may stage behind an attached cache (API shape); this cut changes what any staging is judged by. No collision: direct `Push`/`CommitBatch` reach the same on-disk judge, so they get the per-key rule for free. Hands must not give direct staging a different predicate. -- `variants:follow_up:writer-dangling-link-tolerance`: **leave**. Rust/TS whole-store `push_all` reading a dangling link is - a reader/writer tolerance question, not ownership; this cut touches no leaf. +- `variants:follow_up:writer-dangling-link-tolerance`: **settled** (the reader owns the verdict; every runtime's whole-store + `push_all` refuses a dangling link, as `push` does). ## Deferred doc findings this cut fixes diff --git a/package-lock.json b/package-lock.json index 0a1558da..e5fbf13b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1205,7 +1205,7 @@ "license": "MIT", "dependencies": { "@gamecult/cultcache-ts": "^0.15.0", - "@gamecult/cultnet-ts": "^0.3.0" + "@gamecult/cultnet-ts": "^0.4.0" }, "devDependencies": { "@gamecult/cultmesh-ts": "^0.2.0", @@ -1222,7 +1222,7 @@ "license": "MIT", "dependencies": { "@gamecult/cultcache-ts": "^0.15.0", - "@gamecult/cultnet-ts": "^0.3.0", + "@gamecult/cultnet-ts": "^0.4.0", "koffi": "3.3.0" }, "devDependencies": { @@ -1236,7 +1236,7 @@ }, "packages/cultnet-ts": { "name": "@gamecult/cultnet-ts", - "version": "0.3.0", + "version": "0.4.0", "license": "MIT", "dependencies": { "@gamecult/cultcache-ts": "^0.15.0", diff --git a/package.json b/package.json index a79661f2..1f883fa1 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,7 @@ "scripts": { "build:ts": "node scripts/run-typescript-workspaces.mjs build", "test:ts": "node scripts/run-typescript-workspaces.mjs test", - "test:pack:ts": "node scripts/test-typescript-package-closure.mjs && node scripts/test-declared-dependency-ranges.mjs", + "test:pack:ts": "node --test scripts/test-declared-dependency-ranges.test.mjs && node scripts/test-typescript-package-closure.mjs && node scripts/test-declared-dependency-ranges.mjs", "test:eve-browser-network": "node scripts/verify-eve-browser-network.mjs" }, "devDependencies": { diff --git a/packages/cultcache-py/CHANGELOG.md b/packages/cultcache-py/CHANGELOG.md index 95f82042..30b60850 100644 --- a/packages/cultcache-py/CHANGELOG.md +++ b/packages/cultcache-py/CHANGELOG.md @@ -4,11 +4,31 @@ ### Changed +- `parse_stored_at` (was `_parse_stored_at`) is public: cultmesh-py judges a peer's `storedAt` by it. A peer's `document_put_raw` whose `storedAt` is missing, empty or not a stamp is refused `message_invalid` (field `document`) and nothing is minted or stored (variants:ruling:raw-put-without-stamp-refused). A subscribed server's stampless put now raises `ValueError` from `CultMeshClient.read_next_change` instead of being stored under a minted stamp. + +### Added + +- `CultCache.update` and `update_global` return `UpdateResult(outcome, document)`; `UpdateOutcome` is `UPDATED`, `DECLINED` or `LOST_RACE`. The single-file store gains `read(key)` (the record and its content version, the SHA-256 C# computes) and `commit_if(envelope, expected)`, the version is the SHA-256 of the record re-encoded as `[key, schemaId, storedAt, payload]` with minimal MessagePack encodings, which is the SHA-256 of the record's stored bytes exactly when the writer encoded it minimally (every writer in this library does; a record another writer encoded with a wider-than-needed length or integer has the version of its minimal form, not of its bytes). Both looked up by the cache and not `BackingStore` protocol members (variants:cut_spec:cut-update-with-retry-py). +- `read_payload_tree(type_name, payload, decoder=None)`: the exported owner of the one-value, depth-first payload read that `decode_payload` uses, for payloads read outside a definition (`cultnet-py`'s witness bundle decode needs it, so `cultnet-py` now requires `cultcache-py>=0.4.0`). The manifest version is 0.4.0, the minor these Unreleased breaking changes imply, set in this change so the workspace install resolves the new bound; the `## Unreleased` heading becomes `## [0.4.0]` when it is tagged. + +### Changed + +- A write that replaces a record is stored at a later `stored_at` than the record it replaces, in the single-file store, the JSONL store and a store-less cache, with no exception for identical content (`variants:ruling:py-rs-update-follow-csharp-restamp`); the cache holds the stamp the store wrote. A stored_at is read only in the shape the runtimes write (an RFC 3339 date-time with T, a fraction of 1 to 9 digits, Z or a numeric offset), the same on every Python; any other text, or an instant outside `datetime`'s range, is not a stamp and a write over it is stamped at now. A write over a record stamped at the last instant `datetime` can hold (9999-12-31T23:59:59.999999Z) is refused with `ValueError` and writes nothing. +- `store_write_lock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. `JsonLinesBackingStore` takes no cross-process lock (only its thread lock, under which it already refuses a link), so it has no lock wait to re-check across. Pinned by a test that puts the link while a foreign process holds the lock. +- `CultCache.is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store. +- `JsonLinesBackingStore` writes LF line endings on Windows. It wrote CRLF there, through `write_text` newline translation, before the commit_store_file change (#154, adbbb780). - `msgpack` is a hard dependency, as `pyproject.toml` declared it. The optional-import fallbacks and the `cultcache-py[msgpack]` extra they named (which never existed) are gone: a missing msgpack is an `ImportError` at import. +- `JsonLinesBackingStore` writes through the same `commit_store_file` as the single-file store: a unique staged file, fsynced before the replace and removed when the replace fails, so a refused replace no longer leaves `.tmp` behind. It staged a fixed `.tmp` without an fsync. - The single-file store's `push`, `push_all` and `delete` take the cross-process lock on `.lock` (flock; LockFileEx over the whole file on Windows) across the read and the replace, waiting up to `lock_timeout` seconds (30 by default, `TimeoutError` after), and exclude the C#, Rust and TypeScript runtimes' writers. The staged file is `.tmp--`, fsynced before the replace, with a directory fsync after it off Windows. A fixed `.tmp` is no longer used. ### Breaking +- `update(document, key, apply, *, max_attempts=5)` and `update_global(document, apply, *, max_attempts=5)` replace the blind pair. Each attempt reads the record from the home store, runs `apply` on a copy (None when absent) outside every lock, and commits only if the record is unchanged, running `apply` again on a lost race; it returns `UpdateResult`, not the value. `apply` returning None writes nothing; a lost race that outlasts `max_attempts` returns `LOST_RACE`; a store without `read` and `commit_if` (JSON lines among them) refuses before `apply`. `put` is the blind write. +- `push` returns the `stored_at` it wrote and `push_all` a dict of them by key (`{}` when a soft push skipped); a store returning None keeps the cache's own stamp. +- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`. +- A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work. +- A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`. +- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` ` payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#. - A cache holds one record per key whatever its type (`variants:ruling:cache-refuses-retype`). A `put`, `put_envelope`, `put_envelopes` or `put_global` of a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store, and changes nothing; retype a key by deleting it, then putting the new type. A write may replace another registered type's record only when the written type owns its schema id (declares it or lists it compatible) and both types share one home store. A load that reads one key from two stores, whatever the types, is refused with `CultCacheError` naming the key and both types, and admits nothing. A store that holds a record no registered type claims now loads and is carried: the cache does not serve it, the store keeps its bytes through every write, `delete` or `put` at its key is refused with `SchemaConflictError`, and registering a type that declares its schema id claims it at the next load. It raised `CultCacheError` "Unknown persisted document type". Two global types share the key `__global__`, so a second global type in one cache is now refused, and a store that already holds two global types at `__global__` (a JSONL store written by caches that never pulled could produce one) no longer loads: the pull raises `CultCacheError` and admits nothing. C# refuses a second persisted global on pull the same way (`cultcache-store-composition.md`, "Globals never invented"); repair such a file by deleting one of the two records with the runtime that wrote it. - Added `foreign_records()` and the frozen `ForeignRecord(key, schema_id, schema_name, stored_at)`, sorted by key in UTF-16 code-unit order. `stores.retyped` and `stores.written_owns` lose their underscore: the cache judge and the single-file store's commit guard ask one predicate. diff --git a/packages/cultcache-py/README.md b/packages/cultcache-py/README.md index 43e6c90e..0a03a671 100644 --- a/packages/cultcache-py/README.md +++ b/packages/cultcache-py/README.md @@ -19,6 +19,7 @@ discovery, sessions, local server) build on it, mirroring `cultcache-rs` / `[format, schemaCatalog, records]` - records store raw MessagePack payload bytes under schema-catalog identity - payloads decode only through the registered document definition for that type +- a payload is exactly one MessagePack value: bytes after it are refused on read, by count - a cache holds one record per key whatever its type: a write of a second type at a held key is refused with `SchemaConflictError` (in memory too), and two stores holding one key refuse the load - a store record no registered type claims loads and is carried as foreign: `cache.foreign_records()` lists it, the store @@ -93,10 +94,10 @@ store. - `put(...)` - `put_envelopes(...)` - `put_global(...)` -- `update(...)` -- `update_global(...)` +- `update(document, key, apply, *, max_attempts=5)` and `update_global(document, apply, *, max_attempts=5)`: the conditional golden path; reads the record as its home store holds it now, runs `apply` on a fresh copy, writes only if the record is unchanged and runs `apply` again on a lost race; returns `UpdateResult(outcome, document)` with `UpdateOutcome.UPDATED`, `DECLINED` (`apply` returned `None`) or `LOST_RACE` (see below) - `delete(...)` - `delete_global(...)` +- `is_writable(document, key)` (advisory: would a write or removal of the key be refused; see below) - `snapshot()` ## Indexes And Roles @@ -141,6 +142,8 @@ used by the TypeScript, Rust, and C# runtimes: canonicalSchemaJson, compatibleSchemaIds, members]` - records: `[key, schemaId, storedAt, payload]` +`update` needs a home store with the optional `read(key)` (the record at the key, of any type, and its content version, from one read) and `commit_if(envelope, expected)` (write only if the record at the key is still that version, under the store's write lock). The content version is the lowercase hex SHA-256 of the record re-encoded as the store writes it, `[key, schemaId, storedAt, payload]` with minimal MessagePack encodings, the same value C# computes (`tests/vectors/content-version`). It equals the SHA-256 of the record's bytes in the file exactly when the writer encoded the record minimally, as every writer in this library does; a record encoded with a wider-than-needed length or integer has the version of its minimal form. The single-file store has both; `JsonLinesBackingStore` and a store without them refuse `update` before `apply` runs. A cache with no store copies its held instance. Every write that replaces a record is stored at a later `stored_at` than the one it replaces, in each store kind and in a store-less cache; `push` and `push_all` return the stamp written, and the cache holds that stamp. + Large corpora should use a sharded store or a real database. A single snapshot file is a scalpel, not a forklift. ## Store format and rollback @@ -180,6 +183,16 @@ The payload is a MessagePack array. Field keys are durable slot indexes: - newly added fields should use new keys - fields with defaults tolerate missing or nil slots when older payloads are read +A payload carrying a slot past the slots a reader declares, at any depth (the document, a nested record, a list element, a +union arm, a unit arm), was written by a newer build. It is read, and the record is held read-only: `put`, `update`, +`put_envelope(s)`, `delete` and their global forms refuse its key with `SchemaConflictError` before any store is touched, and +`SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` naming the +first type read with extra slots (never a slot value). `CultCache.is_writable(document, key)` answers the same question before +the write: False for such a record, for a key a store carries foreign, and for a key held as another type's record; the write +still decides. `DocumentDefinition.decode_payload_marked(payload)` returns the value and its `UnreadSlots` (or None). Declaring the +stored id in `compatible_schema_ids` consents to the loss of depth-0 slots that id's catalog entry reports; a nested mark never is. +A store keeps a read-only record's bytes through every other write. A document with a custom `decode` never marks. + That matches the Rust `#[derive(DatabaseEntry)]` formatter shape and the C# `[Key(n)]` intent: schema evolution comes from stable field slots, not source member order. diff --git a/packages/cultcache-py/pyproject.toml b/packages/cultcache-py/pyproject.toml index 6c46d102..15b724ba 100644 --- a/packages/cultcache-py/pyproject.toml +++ b/packages/cultcache-py/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "cultcache-py" -version = "0.3.0" +version = "0.4.0" description = "Python CultCache runtime for GameCult typed state." readme = "README.md" requires-python = ">=3.10" diff --git a/packages/cultcache-py/src/cultcache_py/__init__.py b/packages/cultcache-py/src/cultcache_py/__init__.py index dc73ab1b..0c26cb25 100644 --- a/packages/cultcache-py/src/cultcache_py/__init__.py +++ b/packages/cultcache-py/src/cultcache_py/__init__.py @@ -1,7 +1,7 @@ from .backing_store import BackingStore, CultCacheEnvelope from .backing_store import CultCacheSchemaCatalogEntry, CultCacheSchemaCatalogMember from . import leaves -from .cache import CultCache, CultCacheBuilder, ForeignRecord +from .cache import CultCache, CultCacheBuilder, ForeignRecord, UpdateOutcome, UpdateResult from .documents import ( MAX_PAYLOAD_DEPTH, MAX_SLOT_KEY, @@ -9,15 +9,17 @@ from .documents import ( DatabaseEntryField, DocumentDefinition, UnionArm, + UnreadSlots, database_entry_field, define_database_entry_type, define_document_registry, define_document_type, define_record_codec, define_union, + read_payload_tree, ) from .leaves import SlotCodec -from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreUnreadableError +from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreLinkedError, StoreUnreadableError __all__ = [ "BackingStore", @@ -36,8 +38,13 @@ __all__ = [ "SchemaConflictError", "SingleFileMessagePackBackingStore", "SlotCodec", + "StoreLinkedError", "StoreUnreadableError", "UnionArm", + "UnreadSlots", + "UpdateOutcome", + "UpdateResult", + "read_payload_tree", "leaves", "database_entry_field", "define_database_entry_type", diff --git a/packages/cultcache-py/src/cultcache_py/backing_store.py b/packages/cultcache-py/src/cultcache_py/backing_store.py index 5a21e496..c20ac661 100644 --- a/packages/cultcache-py/src/cultcache_py/backing_store.py +++ b/packages/cultcache-py/src/cultcache_py/backing_store.py @@ -58,19 +58,36 @@ class CultCacheEnvelope: class BackingStore(Protocol): + """The members every store has. A store may also offer `read(key) -> tuple[CultCacheEnvelope, str] | None` (the record at the + key, of any type, and its content version, from one read) and `commit_if(envelope, expected: str | None) -> str | None` (write + only if the record at the key is still the version `expected`, None meaning no record; returns the stored_at written, or None + with nothing written). They are looked up with getattr, not Protocol members, and `CultCache.update` needs both. `push` and + `push_all` may report the stored_at they wrote (`str` and `dict[str, str]` by key); None keeps the cache's own stamp. A store + stamps a record that replaces another strictly later than the one it replaces.""" + def pull_all(self) -> list[CultCacheEnvelope]: ... - def push(self, envelope: CultCacheEnvelope) -> None: + def push(self, envelope: CultCacheEnvelope) -> str | None: ... def delete(self, type: str, key: str) -> None: ... - def push_all(self, envelopes: list[CultCacheEnvelope]) -> None: - existing = { - (envelope.type, envelope.key): envelope for envelope in self.pull_all() - } + def pull_or_absent(self) -> list[CultCacheEnvelope] | None: + """The envelopes the reader reads (empty included), None when nothing is at the path; raises when the path cannot be read. + + The default is pull_all, which cannot say absent: a store that inherits it is always a store the reader read. A store kind + that can tell nothing-at-the-path overrides this, and the default push_all then writes where nothing is on a soft push.""" + return self.pull_all() + + def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> dict[str, str] | None: + # Soft: write only where pull_or_absent says nothing is; skip over a store it read, empty included; its raise refuses. + pulled = self.pull_or_absent() + if soft and pulled is not None: + return + pulled = pulled or [] + existing = {(envelope.type, envelope.key): envelope for envelope in pulled} for envelope in envelopes: existing[(envelope.type, envelope.key)] = envelope self._replace_all(list(existing.values())) diff --git a/packages/cultcache-py/src/cultcache_py/cache.py b/packages/cultcache-py/src/cultcache_py/cache.py index 315e64ac..f82bd7d5 100644 --- a/packages/cultcache-py/src/cultcache_py/cache.py +++ b/packages/cultcache-py/src/cultcache_py/cache.py @@ -1,5 +1,6 @@ from __future__ import annotations +import enum import functools import math import threading @@ -8,14 +9,40 @@ from dataclasses import dataclass, field, replace from typing import Any, Callable, Generic, Iterable, TypeVar from .backing_store import BackingStore, CultCacheEnvelope -from .documents import DocumentDefinition, extract_value -from .stores import SchemaConflictError, retyped, written_owns +from .documents import DocumentDefinition, UnreadSlots, extract_value +from .stores import SchemaConflictError, parse_stored_at, retyped, stored_at_over, written_owns T = TypeVar("T") F = TypeVar("F", bound=Callable[..., Any]) GLOBAL_KEY = "__global__" +class UpdateOutcome(enum.Enum): + UPDATED = "updated" + DECLINED = "declined" + LOST_RACE = "lost_race" + + +@dataclass(frozen=True) +class UpdateResult(Generic[T]): + """What `CultCache.update` did and the cache's record at the key after it (None when it holds nothing). The document decides + nothing: it is a read after the outcome.""" + + outcome: UpdateOutcome + document: T | None + + +@dataclass(frozen=True) +class _Expected: + """What an update's commit requires to still hold. A cache with no store requires the held envelope object at the key (None: + nothing held); a store-backed one requires the type's home to be `store` and the record there to have content version + `version` (None: no record).""" + + held: CultCacheEnvelope | None = None + store: BackingStore | None = None + version: str | None = None + + def _locked(method: F) -> F: """Runs a read holding the cache's lock. Reads are reentrant and may run inside a mutation.""" @@ -123,6 +150,10 @@ class _State: # The records the last pull read that no registered type claims, sorted by key. Derived from the pull and swapped with the # rest of the state, written nowhere else; it decides only refusals at its keys. A store keeps their bytes. foreign: list["ForeignRecord"] = field(default_factory=list) + # The records held read-only, by key (a key holds one record) with the type that holds it: the payload carried slots past the + # declared ones and nothing consents to the loss. Derived from the load or the write that admitted the record and swapped with + # the rest of the state; it decides only refusals at its keys, and any other key is absent. + unread: dict[str, tuple[str, UnreadSlots]] = field(default_factory=dict) @dataclass(frozen=True) @@ -337,6 +368,7 @@ class CultCache: routes, generic = self._state.stores_by_type, self._state.generic_store seen_globals: set[str] = set() foreign: list[ForeignRecord] = [] + loaded_unread: dict[str, tuple[str, UnreadSlots]] = {} type_at_key: dict[str, tuple[str, int]] = {} # one record per key across every store: its type (or schema name) and store stores: list[BackingStore] = [] for candidate in [*self._state.stores_by_type.values(), self._state.generic_store]: @@ -376,8 +408,10 @@ class CultCache: loaded_legacy_global_keys[envelope.type] = envelope.key envelope = replace(envelope, key=GLOBAL_KEY) self._claim_key(type_at_key, envelope.key, envelope.type, position) - envelope = self._stamped(document, envelope) - value = document.decode_payload(envelope.payload) + arrived, envelope = envelope, self._stamped(document, envelope) + value, unread = self._decode_marked(document, arrived) + if unread is not None: + loaded_unread[envelope.key] = (envelope.type, unread) loaded_values.setdefault(envelope.type, {})[envelope.key] = value read_values[(envelope.type, envelope.key)] = value loaded_envelopes.setdefault(envelope.type, {})[envelope.key] = envelope @@ -393,6 +427,7 @@ class CultCache: self._state.lookup_keys, self._state.legacy_global_keys, self._state.foreign, + self._state.unread, ) = ( loaded_values, loaded_envelopes, @@ -401,6 +436,7 @@ class CultCache: loaded_lookup_keys, loaded_legacy_global_keys, sorted(foreign, key=lambda record: _utf16_order(record.key)), + loaded_unread, ) @staticmethod @@ -473,7 +509,7 @@ class CultCache: def put(self, document: DocumentDefinition[T], key: str, value: T) -> None: self._put(document, key, value) - def _put(self, document: DocumentDefinition[T], key: str, value: T) -> None: + def _put(self, document: DocumentDefinition[T], key: str, value: T, expected: _Expected | None = None) -> str: self._assert_registered(document) if document.global_document and key != GLOBAL_KEY: raise CultCacheError(f"Global document {document.type} must use key {GLOBAL_KEY}") @@ -485,15 +521,15 @@ class CultCache: schema_id=catalog_entry.schema_id, catalog_entry=catalog_entry, ) - self._write(document, [(envelope, value)], batch=False) + return self._write(document, [(envelope, value)], batch=False, expected=expected) @_mutating def put_envelope(self, document: DocumentDefinition[T], envelope: CultCacheEnvelope) -> T: self._assert_registered(document) self._check_envelope(document, envelope) - envelope = self._stamped(document, envelope) - value = document.decode_payload(envelope.payload) - self._write(document, [(envelope, value)], batch=False) + arrived, envelope = envelope, self._stamped(document, envelope) + value, unread = self._decode_marked(document, envelope) + self._write(document, [(envelope, value)], batch=False, unread={envelope.key: unread}) return value @_mutating @@ -501,13 +537,33 @@ class CultCache: self._assert_registered(document) values: list[T] = [] stamped: list[CultCacheEnvelope] = [] + unread: dict[str, UnreadSlots | None] = {} for envelope in envelopes: self._check_envelope(document, envelope) stamped.append(self._stamped(document, envelope)) - values.append(document.decode_payload(envelope.payload)) - self._write(document, list(zip(stamped, values)), batch=True) + value, unread[envelope.key] = self._decode_marked(document, stamped[-1]) + values.append(value) + self._write(document, list(zip(stamped, values)), batch=True, unread=unread) return values + @staticmethod + def _decode_marked(document: DocumentDefinition[Any], held_as: CultCacheEnvelope) -> tuple[Any, UnreadSlots | None]: + """The payload decoded under the document, with the mark of any slot it carried past the declared ones, or no mark when the + loss is consented. `held_as` is the envelope under the id the record is stored under: a load's arrival, a write's stamped + envelope, so a key is writable in memory exactly when it is after a reload. A depth-0 mark is consented only when that id is + one the document lists compatible and its catalog entry has slots past the document's that cover the mark's slot count (the + loss a compatible-id declaration consents to, cultlib-gaps:ruling:slot-mark-consent). Any other mark keeps the record read-only.""" + value, slots = document.decode_payload_marked(held_as.payload) + if slots is None: + return value, None + entry, stored = document.catalog_entry(), held_as.catalog_entry + if slots.depth0_only and stored is not None and held_as.schema_id in entry.compatible_schema_ids and held_as.schema_id != entry.schema_id: + declared = {member.slot for member in entry.members} + reported = [member.slot for member in stored.members if member.slot not in declared] + if reported and slots.slot_count <= max(reported) + 1: + return value, None + return value, slots + @staticmethod def _stamped(document: DocumentDefinition[Any], envelope: CultCacheEnvelope) -> CultCacheEnvelope: """The record under its registered schema's id, whatever id the envelope arrived under, so the catalog entry a store @@ -527,14 +583,23 @@ class CultCache: raise CultCacheError(f"Global document {document.type} must use key {GLOBAL_KEY}") def _write( - self, document: DocumentDefinition[Any], records: list[tuple[CultCacheEnvelope, Any]], *, batch: bool - ) -> None: + self, + document: DocumentDefinition[Any], + records: list[tuple[CultCacheEnvelope, Any]], + *, + batch: bool, + unread: dict[str, UnreadSlots | None] | None = None, + expected: _Expected | None = None, + ) -> str: """Validates every record (home and name and index extractors included) before the store is - touched; once the store accepts, applying to the cache has no fallible step.""" + touched; once the store accepts, applying to the cache has no fallible step. With `expected` (an update's commit) the write + also requires the record to be unchanged since it was read and returns 'mismatch', having written and installed nothing, + when it is not; otherwise 'committed'. A record is held under the stored_at its store reports writing.""" type = document.type # A batch naming a key twice lands the last record for it, as C# CultCacheBatch stages one operation per key. The # judge, the store and the apply all read this one list. records = list({envelope.key: (envelope, value) for envelope, value in records}.values()) + self._refuse_unread([envelope.key for envelope, _ in records]) name_extractor = self._state.name_extractors.get(type) index_extractors = self._state.index_extractors.get(type, {}) admitted = [ @@ -549,13 +614,34 @@ class CultCache: ) store = self._store_for_type(type) legacy_key = self._state.legacy_global_keys.get(type) - if store is not None: - if batch: - store.push_all([envelope for envelope, _ in records]) + stamps: dict[str, Any] = {} + if expected is not None: + if legacy_key is not None: + raise self._legacy_global_refusal(document, legacy_key) + if expected.store is None: + held = self._state.envelopes.get(type, {}).get(records[0][0].key) + if store is not None or held is not expected.held: + return "mismatch" + elif store is not expected.store: + return "mismatch" else: - store.push(records[0][0]) + stamp = store.commit_if(records[0][0], expected.version) # type: ignore[attr-defined] + if stamp is None: + return "mismatch" + stamps[records[0][0].key] = stamp + elif store is not None: + if batch: + reported = store.push_all([envelope for envelope, _ in records]) + stamps = reported if isinstance(reported, dict) else {} + else: + stamps[records[0][0].key] = store.push(records[0][0]) if legacy_key is not None: self._retire_legacy_global(store, type, legacy_key) + else: + for envelope, _ in records: + held_at = next((held for held in (by_type.get(envelope.key) for by_type in self._state.envelopes.values()) if held is not None), None) + if held_at is not None: + stamps[envelope.key] = stored_at_over(envelope.stored_at, held_at.stored_at) self._state.legacy_global_keys.pop(type, None) values = self._state.values.setdefault(type, {}) envelopes = self._state.envelopes.setdefault(type, {}) @@ -563,14 +649,21 @@ class CultCache: for other, key in displaced: self._drop(other, key) for envelope, value, keys in admitted: + stamp = stamps.get(envelope.key) + if isinstance(stamp, str) and parse_stored_at(stamp) is not None: + envelope = replace(envelope, stored_at=stamp) self._remove_lookups(type, envelope.key) values[envelope.key] = value envelopes[envelope.key] = envelope + # A held mark refused this write above, so a key reaching here carries none to clear. + if (mark := (unread or {}).get(envelope.key)) is not None: + self._state.unread[envelope.key] = (type, mark) lookup_keys[envelope.key] = keys if keys.name is not None: self._state.names[(type, keys.name)] = envelope.key for index, index_value in keys.indexes.items(): self._state.indexes[(type, index, index_value)] = envelope.key + return "committed" def _refuse_retype(self, document: DocumentDefinition[Any], written: list[CultCacheEnvelope]) -> list[tuple[str, str]]: """The one key-identity judge (variants:ruling:cache-refuses-retype): a key holds one record whatever its type, so a write @@ -592,6 +685,68 @@ class CultCache: replaced.append((other, envelope.key)) return replaced + @_locked + def is_writable(self, document: DocumentDefinition[Any], key: str) -> bool: + """Whether a write or removal of the key by this document would pass the judges that run before any store is touched: False for + a record held read-only (its payload carried slots past the declared ones, at any depth, and nothing consents to the loss, so + writing it would drop data a newer writer stored), for a key a store carries foreign, and for a key held as another type's + record this document does not own. Advisory (the write still decides): it answers at call time, and a key nothing holds is + writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it with + `StoreLinkedError`.""" + self._assert_registered(document) + try: + self._refuse_at(document, key) + except (SchemaConflictError, CultCacheError): + return False + return True + + def _refuse_at(self, document: DocumentDefinition[Any], key: str) -> list[tuple[str, str]]: + """The judges a write or removal of the key by this document passes before any store is touched, asked by `is_writable` and + by `delete`: a global document writes only its global key, a record held read-only is never replaced, and a key held as + another type's record or carried foreign is refused (`_refuse_retype`).""" + if document.global_document and key != GLOBAL_KEY: + raise CultCacheError(f"Global document {document.type} must use key {GLOBAL_KEY}") + self._refuse_unread([key]) + entry = document.catalog_entry() + probe = CultCacheEnvelope.create(key=key, type=document.type, payload=b"", schema_id=entry.schema_id, catalog_entry=entry) + return self._refuse_retype(document, [probe]) + + def _refuse_unread(self, keys: list[str]) -> None: + """The one check every write and removal asks before any store is touched.""" + for key in keys: + held = self._state.unread.get(key) + if held is None: + continue + type, slots = held + raise self._read_only_refusal(self._state.documents[type], key, self._state.envelopes[type][key].schema_id, slots) + + @staticmethod + def _read_only_refusal( + document: DocumentDefinition[Any], key: str, schema_id: str | None, slots: UnreadSlots + ) -> SchemaConflictError: + """The one refusal of a write to a record that carries slots this build would drop: a record the cache holds read-only, + or one an update reads from the store.""" + name = document.catalog_entry().schema_name + return SchemaConflictError( + schema_id or document.type, + [name], + key, + message=( + f"{name} record {key} is read-only: {slots.type_name} was read with " + f"{slots.slot_count} slots where this build declares {slots.declared_count} ({slots.occurrences} object(s) in the " + "record). Writing it would drop data a newer writer stored; read it with types that declare those slots, or " + "write it from that build." + ), + unread_slots=slots, + ) + + @staticmethod + def _legacy_global_refusal(document: DocumentDefinition[Any], legacy_key: str) -> CultCacheError: + return CultCacheError( + f"Global {document.type} is held under the legacy key {legacy_key!r}; update cannot copy it. " + "put_global writes the global and retires the legacy key." + ) + @staticmethod def _foreign_refusal(record: ForeignRecord) -> SchemaConflictError: return SchemaConflictError( @@ -623,20 +778,81 @@ class CultCache: self._assert_global(document) self._put(document, GLOBAL_KEY, value) - @_mutating - def update(self, document: DocumentDefinition[T], key: str, updater: Any) -> T: - return self._update(document, key, updater) + def update( + self, document: DocumentDefinition[T], key: str, apply: Callable[[T | None], T | None], *, max_attempts: int = 5 + ) -> UpdateResult[T]: + """Reads the record at the key as its home store holds it, applies `apply` to a fresh copy of it, and commits the result + only if the record is still the one that was read; when another writer got there first it reads again and runs `apply` + again, up to `max_attempts` times. - def _update(self, document: DocumentDefinition[T], key: str, updater: Any) -> T: - current = self.get_required(document, key) - updated = updater(current) - self._put(document, key, updated) - return updated + `apply` may run up to `max_attempts` times, on a fresh copy of the record as the store holds it now (None when the key + holds nothing), which it may change and return. It must not write the same key through this cache or have side effects. + Returning None writes nothing (DECLINED). A lost race that outlasts `max_attempts` returns LOST_RACE and does not raise; + a refusal, or a raise from `apply`, propagates and is not retried. The write is durable when update returns. The + result's document is the cache's record after the outcome. `put` is the blind write. A cache with no store copies its + held instance; a store without `read` and `commit_if` (the JSON lines store among them) refuses before `apply` runs.""" + if not callable(apply): + raise TypeError("CultCache update needs a function to apply.") + if not isinstance(max_attempts, int) or isinstance(max_attempts, bool): + raise TypeError(f"CultCache update max_attempts must be an int, not {type(max_attempts).__name__}.") + if max_attempts < 1: + raise ValueError(f"CultCache update max_attempts must be at least 1, not {max_attempts}.") + for attempt in range(1, max_attempts + 1): + copy, expected = self._read_for_update(document, key) + # Outside the cache's lock and the store's: apply may take its time and may write other keys through this cache. + next_value = apply(copy) + if next_value is None: + return UpdateResult(UpdateOutcome.DECLINED, self.get(document, key)) + if self._commit_update(document, key, next_value, expected) == "committed": + return UpdateResult(UpdateOutcome.UPDATED, self.get(document, key)) + # Display only: the cache shows what the winner left. + if self._state.stores_by_type or self._state.generic_store is not None: + self.pull_all_backing_stores() + return UpdateResult(UpdateOutcome.LOST_RACE, self.get(document, key)) @_mutating - def update_global(self, document: DocumentDefinition[T], updater: Any) -> T: + def _read_for_update(self, document: DocumentDefinition[T], key: str) -> tuple[T | None, _Expected]: + """One attempt's read: the copy to hand `apply` and the condition its commit holds. Every refusal that does not depend on + what `apply` returns is raised here, so `apply` never runs for a write that cannot land. Installs nothing.""" + self._assert_registered(document) + self._refuse_at(document, key) + legacy_key = self._state.legacy_global_keys.get(document.type) + if legacy_key is not None: + raise self._legacy_global_refusal(document, legacy_key) + store = self._store_for_type(document.type) + if store is None: + held = self._state.envelopes.get(document.type, {}).get(key) + return (None if held is None else document.decode_payload(held.payload)), _Expected(held=held) + read, commit_if = getattr(store, "read", None), getattr(store, "commit_if", None) + if read is None or commit_if is None: + raise CultCacheError( + f"Backing store {type(store).__name__} for {document.type} cannot write conditionally; " + "update needs a store with read and commit_if." + ) + if document.global_document: + for envelope in store.pull_all(): + if envelope.key != GLOBAL_KEY and self._resolve_document_for_envelope(envelope) is document: + raise self._legacy_global_refusal(document, envelope.key) + found = read(key) + if found is None: + return None, _Expected(store=store) + envelope, version = found + if self._resolve_document_for_envelope(envelope) is not document: + raise CultCacheError(f"Record {key} is not a {document.type}; update cannot copy it.") + value, unread = self._decode_marked(document, envelope) + if unread is not None: + raise self._read_only_refusal(document, key, envelope.schema_id, unread) + return value, _Expected(store=store, version=version) + + @_mutating + def _commit_update(self, document: DocumentDefinition[T], key: str, value: T, expected: _Expected) -> str: + return self._put(document, key, value, expected) + + def update_global( + self, document: DocumentDefinition[T], apply: Callable[[T | None], T | None], *, max_attempts: int = 5 + ) -> UpdateResult[T]: self._assert_global(document) - return self._update(document, GLOBAL_KEY, updater) + return self.update(document, GLOBAL_KEY, apply, max_attempts=max_attempts) @_mutating def delete(self, document: DocumentDefinition[Any], key: str) -> None: @@ -644,16 +860,18 @@ class CultCache: def _delete(self, document: DocumentDefinition[Any], key: str) -> None: self._assert_registered(document) - if key not in self._state.envelopes.get(document.type, {}): - carried = next((record for record in self._state.foreign if record.key == key), None) - if carried is not None: - raise self._foreign_refusal(carried) + # A key held as another type's record this document owns is removed, as a write of the key would replace it (C# Remove is by key). + owned = self._refuse_at(document, key) store = self._store_for_type(document.type) legacy_key = self._state.legacy_global_keys.get(document.type) if key == GLOBAL_KEY else None if store is not None: - store.delete(document.type, key if legacy_key is None else legacy_key) + # One record per key: the owned record when this document does not hold the key, else its own, in one store call. + holder = owned[0][0] if owned else document.type + store.delete(holder, key if legacy_key is None else legacy_key) if legacy_key is not None: del self._state.legacy_global_keys[document.type] + for other, _ in owned: + self._drop(other, key) self._drop(document.type, key) def _drop(self, type: str, key: str) -> None: diff --git a/packages/cultcache-py/src/cultcache_py/documents.py b/packages/cultcache-py/src/cultcache_py/documents.py index 6101c136..ff9039dc 100644 --- a/packages/cultcache-py/src/cultcache_py/documents.py +++ b/packages/cultcache-py/src/cultcache_py/documents.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +import threading from dataclasses import MISSING, dataclass, field from collections.abc import Mapping, Sequence, Set from itertools import chain @@ -30,6 +31,49 @@ MAX_SLOT_KEY = 65535 _STACK_FAULTS: tuple[type[BaseException], ...] = (_MsgpackStackError,) +@dataclass(frozen=True) +class UnreadSlots: + """What a payload decode found past the slots this runtime declares, at any depth: the first type read with extra slots, how + many slots it carried against how many it declares, and how many objects in the payload did. It holds no slot value, so a + refusal can name it. `depth0_only` is True when every mark sat in the document's own record (the half of consent the payload + decides). A record carrying it is read-only unless the schema comparison explains every mark (see `CultCache.is_writable`).""" + + type_name: str + slot_count: int + declared_count: int + occurrences: int + depth0_only: bool + + +class _SlotScope: + """Opened by `DocumentDefinition.decode_payload_marked`; the record codec and the union codec note into it. It is thread-local, + so no codec signature carries it, and a codec decoded outside a payload decode notes nothing.""" + + def __init__(self) -> None: + self.depth = 0 # records entered: 0 while the document's own record is read + self.first: tuple[str, int, int] | None = None + self.occurrences = 0 + self.depth0_only = True + + def note(self, type_name: str, slot_count: int, declared_count: int, depth0: bool) -> None: + if self.first is None: + self.first = (type_name, slot_count, declared_count) + self.occurrences += 1 + self.depth0_only = self.depth0_only and depth0 + + def result(self) -> UnreadSlots | None: + if self.first is None: + return None + return UnreadSlots(*self.first, self.occurrences, self.depth0_only) + + +_scope = threading.local() + + +def _current_scope() -> _SlotScope | None: + return getattr(_scope, "open", None) + + _CONTAINERS = (Mapping, Sequence, Set) _SCALAR_SEQUENCES = (str, bytes, bytearray, memoryview) @@ -114,6 +158,26 @@ def _messagepack_payload_decoder(payload: bytes) -> Any: return msgpack.unpackb(payload, raw=False) +def read_payload_tree(type_name: str, payload: bytes, decoder: Callable[[bytes], Any] | None = None) -> Any: + """The one read of a payload's tree: exactly one MessagePack value, within the nesting limit. Depth is judged first, + on the value read so far, so a tail never pre-empts it (C# throws at container 257 before it reaches the tail); then a + tail is refused by count and never echoed. Raised outside the handler so no `__context__` keeps the tail.""" + after = -1 + unpacked: Any = None + try: + tree = (decoder or _messagepack_payload_decoder)(payload) + except _STACK_FAULTS: + raise CultCachePayloadDepthError(type_name, "read") from None + except msgpack.ExtraData as error: + after = len(error.extra) + unpacked = error.unpacked + if after >= 0: + _refuse_past_depth_limit(unpacked, type_name, "read") + raise ValueError(f"{type_name} payload has {after} bytes after its first MessagePack value (refused on read).") + _refuse_past_depth_limit(tree, type_name, "read") + return tree + + @dataclass(frozen=True) class DatabaseEntryField: name: str @@ -185,15 +249,21 @@ class DocumentDefinition(Generic[T]): return self.payload_encoder(tree) def decode_payload(self, payload: bytes) -> T: + return self.decode_payload_marked(payload)[0] + + def decode_payload_marked(self, payload: bytes) -> tuple[T, UnreadSlots | None]: + """The value and what the decode found past the declared slots, at any depth. The payload depth check runs first: a + payload past the limit is refused, never marked.""" + tree = read_payload_tree(self.type, payload, self.payload_decoder) + outer, opened = _current_scope(), _SlotScope() + _scope.open = opened try: - tree = self.payload_decoder(payload) - except _STACK_FAULTS: - raise CultCachePayloadDepthError(self.type, "read") from None - _refuse_past_depth_limit(tree, self.type, "read") - value = self.decode(tree) + value = self.decode(tree) + finally: + _scope.open = outer if self.validate: self.validate(value) - return value + return value, opened.result() def __post_init__(self) -> None: # Judged here so every constructor passes it: the factories, the dataclass constructor and dataclasses.replace. @@ -418,6 +488,9 @@ def define_union( raise ValueError(f"Union {name!r} value is malformed: expected [armKey, [armSlots]]") arm_name, codec = by_key[key] if codec is None: + scope = _current_scope() + if scope is not None and raw[1]: + scope.note(arm_name, len(raw[1]), 0, False) return UnionArm(arm_name, None) try: return UnionArm(arm_name, codec.decode(raw[1])) @@ -479,17 +552,26 @@ def _record_codec( def decode(raw: Any) -> Any: if not isinstance(raw, list): raise ValueError(f"DatabaseEntry document type {type!r} expected a MessagePack array") + scope = _current_scope() + if scope is not None and len(raw) > max_key + 1: + scope.note(type, len(raw), max_key + 1, scope.depth == 0) values: dict[str, Any] = {} - for field in normalized: - if field.key < len(raw): - try: - values[field.name] = field.codec.decode(raw[field.key]) - except ValueError as error: - raise ValueError(f"{type} slot {field.key} ({field.name}): {error}") from None - elif field.default is MISSING: - values[field.name] = None - else: - values[field.name] = field.default() if callable(field.default) else field.default + if scope is not None: + scope.depth += 1 + try: + for field in normalized: + if field.key < len(raw): + try: + values[field.name] = field.codec.decode(raw[field.key]) + except ValueError as error: + raise ValueError(f"{type} slot {field.key} ({field.name}): {error}") from None + elif field.default is MISSING: + values[field.name] = None + else: + values[field.name] = field.default() if callable(field.default) else field.default + finally: + if scope is not None: + scope.depth -= 1 if cls is None: return values return cls(**values) diff --git a/packages/cultcache-py/src/cultcache_py/stores.py b/packages/cultcache-py/src/cultcache_py/stores.py index 61841956..99e38d0d 100644 --- a/packages/cultcache-py/src/cultcache_py/stores.py +++ b/packages/cultcache-py/src/cultcache_py/stores.py @@ -1,18 +1,23 @@ from __future__ import annotations import base64 +import hashlib import json import os +import re import sys import threading import time import uuid from contextlib import contextmanager +from dataclasses import replace +from datetime import datetime, timedelta, timezone from pathlib import Path from typing import Any, Callable, Iterator, TypeVar import msgpack +from .documents import UnreadSlots from .backing_store import ( CultCacheEnvelope, CultCacheSchemaCatalogEntry, @@ -46,9 +51,19 @@ class SchemaConflictError(ValueError): """A write the catalog cannot describe: records of different types under one schema id, two arrived schemas that share an id and disagree on the schema name, or a record no chosen entry publishes. Nothing is written. It names the id, the names involved (`schema_names`) and a record key (`record_key`). A cache raises it too, with an empty record key, when a - document registers a schema id another registered document already carries.""" + document registers a schema id another registered document already carries. `unread_slots` is None except for the refusal to + write or remove a record held read-only because its payload carries slots past the declared ones; its message names the type + and both slot counts and never holds a slot value.""" - def __init__(self, schema_id: str, schema_names: list[str], record_key: str, *, message: str | None = None) -> None: + def __init__( + self, + schema_id: str, + schema_names: list[str], + record_key: str, + *, + message: str | None = None, + unread_slots: UnreadSlots | None = None, + ) -> None: super().__init__( message or f"Schema id {schema_id!r} cannot describe record {record_key!r}: {', '.join(repr(name) for name in schema_names)}" @@ -56,6 +71,7 @@ class SchemaConflictError(ValueError): self.schema_id = schema_id self.schema_names = schema_names self.record_key = record_key + self.unread_slots = unread_slots class JsonLinesBackingStore: @@ -64,10 +80,17 @@ class JsonLinesBackingStore: self._lock = threading.RLock() def pull_all(self) -> list[CultCacheEnvelope]: + return self._pull() or [] + + def pull_or_absent(self) -> list[CultCacheEnvelope] | None: + return self._pull() + + def _pull(self) -> list[CultCacheEnvelope] | None: + """The envelopes the reader reads, or None when nothing is at the path.""" with self._lock: text = _read_store(self.path, lambda path: path.read_text(encoding="utf-8")) if text is None: - return [] + return None envelopes: list[CultCacheEnvelope] = [] for line_number, line in enumerate(text.splitlines(), start=1): if not line.strip(): @@ -88,27 +111,41 @@ class JsonLinesBackingStore: raise ValueError(f"Malformed CultCache JSONL envelope at {self.path}:{line_number}: missing {exc}") from exc return envelopes - def push(self, envelope: CultCacheEnvelope) -> None: - with self._lock: - existing = {(item.type, item.key): item for item in self.pull_all()} - existing[(envelope.type, envelope.key)] = envelope - self._replace_all(list(existing.values())) + def push(self, envelope: CultCacheEnvelope) -> str | None: + return self.push_all([envelope])[envelope.key] - def push_all(self, envelopes: list[CultCacheEnvelope]) -> None: - with self._lock: - existing = {(item.type, item.key): item for item in self.pull_all()} + def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> dict[str, str]: + """Merges the envelopes, each stored later than any record it replaces, and returns the stamps written by key + ({} when a soft push skipped).""" + stamps: dict[str, str] = {} + + def merge(existing: dict[tuple[str, str], CultCacheEnvelope]) -> None: for envelope in envelopes: + replaced = existing.get((envelope.type, envelope.key)) + if replaced is not None: + envelope = replace(envelope, stored_at=stored_at_over(envelope.stored_at, replaced.stored_at)) existing[(envelope.type, envelope.key)] = envelope - self._replace_all(list(existing.values())) + stamps[envelope.key] = envelope.stored_at + self._rewrite(merge, soft) + return stamps def delete(self, type: str, key: str) -> None: + self._rewrite(lambda existing: existing.pop((type, key), None)) + + def _rewrite(self, change: Callable[[dict[tuple[str, str], CultCacheEnvelope]], object], soft: bool = False) -> None: + """The one entry of every write: refuses a link at the path, then reads the file (a soft write stops here when the + reader read it), applies the change and replaces it.""" with self._lock: - existing = [item for item in self.pull_all() if not (item.type == type and item.key == key)] - self._replace_all(existing) + _refuse_linked(self.path) + arrived = self._pull() + if soft and arrived is not None: + return + existing = {(item.type, item.key): item for item in arrived or []} + change(existing) + self._replace_all(list(existing.values())) def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: self.path.parent.mkdir(parents=True, exist_ok=True) - temp = self.path.with_suffix(self.path.suffix + ".tmp") lines = [] for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)): lines.append(json.dumps({ @@ -117,8 +154,7 @@ class JsonLinesBackingStore: "payload": base64.b64encode(envelope.payload).decode("ascii"), "stored_at": envelope.stored_at, }, ensure_ascii=False, sort_keys=True)) - temp.write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8") - temp.replace(self.path) + commit_store_file(self.path, ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8")) class SingleFileMessagePackBackingStore: @@ -130,13 +166,20 @@ class SingleFileMessagePackBackingStore: self._format = STORE_FORMAT_VERSION def pull_all(self) -> list[CultCacheEnvelope]: + return self._pull() or [] + + def pull_or_absent(self) -> list[CultCacheEnvelope] | None: + return self._pull() + + def _pull(self) -> list[CultCacheEnvelope] | None: + """The envelopes the reader reads, or None when nothing is at the path.""" with self._lock: # An empty file is not a store (no writer leaves one), so it is decoded and refused. # The disk decides the header: a file that is gone is not marked. self._format = STORE_FORMAT_VERSION data = _read_store(self.path, Path.read_bytes) if data is None: - return [] + return None try: decoded = msgpack.unpackb(data, raw=False) snapshot = _decode_snapshot(decoded) @@ -147,18 +190,40 @@ class SingleFileMessagePackBackingStore: except Exception as exc: raise StoreUnreadableError(self.path, exc) from exc - def push(self, envelope: CultCacheEnvelope) -> None: + def push(self, envelope: CultCacheEnvelope) -> str | None: with self._write_transaction(): - self._replace_all([envelope], self.pull_all()) + return self._replace_all([envelope], self.pull_all())[envelope.key] - def push_all(self, envelopes: list[CultCacheEnvelope]) -> None: + def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> dict[str, str]: + """A soft push writes only where nothing is at the path: a store the reader reads is left byte for byte, and a file it + cannot read is refused by the same read, as in TS and Rust. Returns the stamps written by key ({} when skipped).""" with self._write_transaction(): - self._replace_all(list(envelopes), self.pull_all()) + arrived = self._pull() + if soft and arrived is not None: + return {} + return self._replace_all(list(envelopes), arrived or []) def delete(self, type: str, key: str) -> None: with self._write_transaction(): self._replace_all([], [item for item in self.pull_all() if not (item.type == type and item.key == key)]) + def read(self, key: str) -> tuple[CultCacheEnvelope, str] | None: + """The record at the key, of any type, and its content version, from one read; None when the key holds nothing.""" + for envelope in self._pull() or []: + if envelope.key == key: + return envelope, record_version(envelope) + return None + + def commit_if(self, envelope: CultCacheEnvelope, expected: str | None) -> str | None: + """Writes the envelope only if the record at its key (any type) is still the one whose version is `expected` + (None: no record), judged under the write lock; returns the stored_at written, or None with nothing written.""" + with self._write_transaction(): + arrived = self._pull() or [] + current = next((item for item in arrived if item.key == envelope.key), None) + if (None if current is None else record_version(current)) != expected: + return None + return self._replace_all([envelope], arrived)[envelope.key] + @contextmanager def _write_transaction(self) -> Iterator[None]: """The thread lock, then the cross-process lock on `.lock`, held across the read and the replace so two @@ -168,8 +233,10 @@ class SingleFileMessagePackBackingStore: with store_write_lock(self.path, self.lock_timeout): yield - def _replace_all(self, supplied: list[CultCacheEnvelope], arrived: list[CultCacheEnvelope]) -> None: - commit_store_file(self.path, msgpack.packb(_encode_snapshot(supplied, arrived, self._format), use_bin_type=True)) + def _replace_all(self, supplied: list[CultCacheEnvelope], arrived: list[CultCacheEnvelope]) -> dict[str, str]: + snapshot, stamps = _encode_snapshot(supplied, arrived, self._format) + commit_store_file(self.path, msgpack.packb(snapshot, use_bin_type=True)) + return stamps @@ -270,7 +337,7 @@ def written_owns(held: CultCacheEnvelope, written: CultCacheEnvelope) -> bool: def _encode_snapshot( supplied: list[CultCacheEnvelope], arrived: list[CultCacheEnvelope], format_version: str -) -> list[Any]: +) -> tuple[list[Any], dict[str, str]]: # A key names one record whatever its type, so a supplied record replaces the arrived one by key alone, and only of the # type that owns it: retyping a key is a removal and an add, so a record another type owns under the key refuses the write. held_by_key = {envelope.key: envelope for envelope in arrived} @@ -279,8 +346,12 @@ def _encode_snapshot( for held in (landing.get(envelope.key), held_by_key.get(envelope.key)): if held is not None and not written_owns(held, envelope): raise retyped(held, envelope) + held = held_by_key.get(envelope.key) + if held is not None: + envelope = replace(envelope, stored_at=stored_at_over(envelope.stored_at, held.stored_at)) landing[envelope.key] = envelope supplied = list(landing.values()) + stamps = {envelope.key: envelope.stored_at for envelope in supplied} replaced = set(landing) kept = [envelope for envelope in arrived if envelope.key not in replaced] envelopes = [*kept, *supplied] @@ -295,7 +366,7 @@ def _encode_snapshot( [envelope.key, _schema_id_for(envelope), envelope.stored_at, envelope.payload] for envelope in sorted(envelopes, key=lambda item: item.key) ] - return [format_version, catalog, records] + return [format_version, catalog, records], stamps def commit_store_file(path: Path, data: bytes) -> None: """Writes `data` to a unique staged file beside the store, fsyncs it, replaces the store with it and, off Windows, @@ -311,7 +382,7 @@ def commit_store_file(path: Path, data: bytes) -> None: os.fsync(fd) finally: os.close(fd) - os.replace(staged, path) + _replace_store(staged, path) except BaseException: try: os.unlink(staged) @@ -326,11 +397,136 @@ def commit_store_file(path: Path, data: bytes) -> None: os.close(directory) +REPLACE_ATTEMPTS = 25 +REPLACE_RETRY_SECONDS = 0.02 +_ERROR_ACCESS_DENIED = 5 +_ERROR_SHARING_VIOLATION = 32 +_ERROR_LOCK_VIOLATION = 33 +_POSIX_UNSUPPORTED = (1, 50, 87) # incorrect function, not supported, invalid parameter: FAT, exFAT, before Windows 10 1607 +MOVEFILE_REPLACE_WRITE_THROUGH = 0x9 # MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH + + +class ReplaceKernel: + """The kernel32 calls a Windows store replace needs, each answering 0 or a Win32 error code.""" + + def rename_posix(self, staged: str, destination: str) -> int: + """CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle.""" + raise NotImplementedError + + def move_file_ex(self, staged: str, destination: str, flags: int) -> int: + raise NotImplementedError + + def sleep(self, seconds: float) -> None: + time.sleep(seconds) + + +def _replace_store(staged: str | os.PathLike[str], destination: str | os.PathLike[str]) -> None: + """Replaces `destination` with `staged`: POSIX-semantics rename first on Windows (so a reader holding the store open + keeps it and rarely sees it absent; a stalled rename can still leave it absent for 5-47 ms on NTFS, + variants:ruling:posix-rename-absent-window-document), os.replace elsewhere.""" + if sys.platform == "win32": + replace_store_windows(os.fspath(staged), os.fspath(destination), WindowsReplaceKernel()) + else: + os.replace(staged, destination) + + +def replace_store_windows(staged: str, destination: str, kernel: ReplaceKernel) -> None: + """POSIX-semantics rename, falling back for the rest of the call to write-through MoveFileExW where POSIX semantics are + unsupported; errors 5, 32 and 33 retry up to 25 times 20 ms apart, any other code raises at once. The TypeScript and C# + runtimes take the same decision table.""" + posix = True + last = 0 + for _ in range(REPLACE_ATTEMPTS): + code = kernel.rename_posix(staged, destination) if posix else kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) + if posix and code in _POSIX_UNSUPPORTED: + posix = False + code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) + if code == 0: + return + if code not in (_ERROR_ACCESS_DENIED, _ERROR_SHARING_VIOLATION, _ERROR_LOCK_VIOLATION): + raise _replace_error(f"CultCache could not replace the store: Win32 error {code}.", code) + last = code + kernel.sleep(REPLACE_RETRY_SECONDS) + raise _replace_error(f"CultCache could not replace the store after {REPLACE_ATTEMPTS} attempts: last Win32 error {last}.", last) + + +def _replace_error(message: str, code: int) -> OSError: + error = OSError(message) + error.winerror = code # type: ignore[attr-defined] + return error + + +class WindowsReplaceKernel(ReplaceKernel): + """kernel32 bound by ctypes, as _try_lock_windows binds it. Windows only.""" + + def __init__(self) -> None: + import ctypes + from ctypes import wintypes + + self._ctypes = ctypes + k = self._k = ctypes.WinDLL("kernel32", use_last_error=True) + k.CreateFileW.restype = wintypes.HANDLE + k.CreateFileW.argtypes = [wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p, wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE] + k.SetFileInformationByHandle.restype = wintypes.BOOL + k.SetFileInformationByHandle.argtypes = [wintypes.HANDLE, ctypes.c_int, ctypes.c_void_p, wintypes.DWORD] + k.CloseHandle.restype = wintypes.BOOL + k.CloseHandle.argtypes = [wintypes.HANDLE] + k.MoveFileExW.restype = wintypes.BOOL + k.MoveFileExW.argtypes = [wintypes.LPCWSTR, wintypes.LPCWSTR, wintypes.DWORD] + + def rename_posix(self, staged: str, destination: str) -> int: + ctypes, k = self._ctypes, self._k + delete_synchronize, share_all, open_existing, normal = 0x00110000, 0x7, 3, 0x80 + handle = k.CreateFileW(staged, delete_synchronize, share_all, None, open_existing, normal, None) + if handle is None or ctypes.c_void_p(handle).value in (None, ctypes.c_void_p(-1).value): + return ctypes.get_last_error() + try: + # FILE_RENAME_INFO: Flags (DWORD), RootDirectory (pointer-aligned HANDLE), FileNameLength (DWORD), FileName (WCHAR[]). + name = os.path.abspath(destination).encode("utf-16-le") + pointer = ctypes.sizeof(ctypes.c_void_p) + length_at = pointer + pointer + info = ctypes.create_string_buffer(length_at + 4 + len(name) + 2) + info[0:4] = (0x3).to_bytes(4, "little") # FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS + info[length_at:length_at + 4] = len(name).to_bytes(4, "little") + info[length_at + 4:length_at + 4 + len(name)] = name + file_rename_info_ex = 22 + if k.SetFileInformationByHandle(handle, file_rename_info_ex, info, len(info)): + return 0 + return ctypes.get_last_error() + finally: + k.CloseHandle(handle) + + def move_file_ex(self, staged: str, destination: str, flags: int) -> int: + return 0 if self._k.MoveFileExW(staged, destination, flags) else self._ctypes.get_last_error() + + +class StoreLinkedError(OSError): + """A write was asked of a store path that is a link: a replace there would swap the link for a regular file.""" + + def __init__(self, path: Path) -> None: + super().__init__(f"CultCache store {path} is a link, and a write at it would replace the link; configure the link's target as the store path.") + self.path = path + + +def _refuse_linked(path: Path) -> None: + """Refuses a link at the store path: a file or directory link, and on Windows a junction or any reparse point (islink misses + junctions before Python 3.12). Absent is not a link.""" + linked = os.path.islink(path) + if not linked and sys.platform == "win32": + try: + linked = bool(os.lstat(path).st_file_attributes & 0x400) # FILE_ATTRIBUTE_REPARSE_POINT + except FileNotFoundError: + pass + if linked: + raise StoreLinkedError(path) + + @contextmanager def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Iterator[None]: """Holds the exclusive store lock on `.lock`, the lock the C#, Rust and TypeScript runtimes take: flock on Unix, LockFileEx over the whole file on Windows, polled every 10 ms with no handle held between polls. The lock file is created if absent and never written or deleted.""" + _refuse_linked(path) lock_path = path.with_name(path.name + ".lock") deadline = time.monotonic() + timeout while True: @@ -341,6 +537,8 @@ def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Itera raise TimeoutError(f"CultCache timed out after {timeout} s waiting for the store lock {lock_path}.") time.sleep(_LOCK_POLL_SECONDS) try: + # A link may have been put at the path while the lock was awaited; the look above only saw the path before. + _refuse_linked(path) yield finally: release() @@ -577,6 +775,65 @@ def _schema_id_for(envelope: CultCacheEnvelope) -> str: return envelope.schema_id or envelope.type +_STORED_AT_SHAPE = re.compile( + r"([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2}):([0-9]{2}):([0-9]{2})(?:\.([0-9]{1,9}))?(?:(Z)|([+-])([01][0-9]|2[0-3]):([0-5][0-9]))" +) + + +def parse_stored_at(text: Any) -> datetime | None: + """The UTC instant a stored_at text names, or None. Only the shape the runtimes write is a stamp: an RFC 3339 date-time with T, + an optional fraction of 1 to 9 digits (cut to the microsecond) and Z or a numeric offset of hours 00-23 and minutes 00-59. Checked here, not left to + fromisoformat, whose accepted set differs by interpreter (3.10 and 3.13) and from Rust and TypeScript. An instant outside + datetime's range is not a stamp.""" + if not isinstance(text, str): + return None + shape = _STORED_AT_SHAPE.fullmatch(text) + if shape is None: + return None + year, month, day, hour, minute, second, fraction, zulu, sign, offset_hours, offset_minutes = shape.groups() + try: + offset = timezone.utc if zulu else timezone( + (-1 if sign == "-" else 1) * timedelta(hours=int(offset_hours), minutes=int(offset_minutes)) + ) + local = datetime( + int(year), int(month), int(day), int(hour), int(minute), int(second), int(((fraction or "") + "000000")[:6]), offset + ) + return local.astimezone(timezone.utc) + except (ValueError, OverflowError): + return None + + +def mint_stored_at(previous: str | None, now: datetime) -> str: + """A stored_at later than `previous` (C# MintStoredAt): `now`, or the microsecond after `previous` when `now` is not later. + A previous that does not parse leaves `now`. Raises ValueError, nothing minted, when `previous` is the last instant a stamp + can name, so a write over it is refused and writes nothing.""" + now = now.astimezone(timezone.utc) + last = parse_stored_at(previous) + if last is not None and now <= last: + try: + now = last + timedelta(microseconds=1) + except OverflowError: + raise ValueError(f"stored_at {previous!r} is the last instant a stamp can name; a write over it cannot be stored later") from None + return now.isoformat(timespec="microseconds") + + +def stored_at_over(staged: str, replaced: str) -> str: + """The stored_at a record replacing `replaced` is written with (C# StoredAtOver): `staged` when both parse and it is strictly + later, else one minted over `replaced`. No content is compared: every replacing write is stamped later.""" + later, earlier = parse_stored_at(staged), parse_stored_at(replaced) + if later is not None and earlier is not None and later > earlier: + return staged + return mint_stored_at(replaced, datetime.now(timezone.utc)) + + +def record_version(envelope: CultCacheEnvelope) -> str: + """The content version of a stored record: the lowercase hex SHA-256 of its persisted encoding, the array + [key, schema id, stored_at, payload] exactly as the single-file store writes it + (variants:ruling:write-set-content-version-declared-ownership).""" + record = [envelope.key, _schema_id_for(envelope), envelope.stored_at, envelope.payload] + return hashlib.sha256(msgpack.packb(record, use_bin_type=True)).hexdigest() + + def _default_catalog_entry(envelope: CultCacheEnvelope) -> CultCacheSchemaCatalogEntry: schema_id = _schema_id_for(envelope) return CultCacheSchemaCatalogEntry( diff --git a/packages/cultcache-py/tests/test_content_version_vector.py b/packages/cultcache-py/tests/test_content_version_vector.py new file mode 100644 index 00000000..393784e0 --- /dev/null +++ b/packages/cultcache-py/tests/test_content_version_vector.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import shutil +import tempfile +import unittest +from pathlib import Path + +import msgpack + +from cultcache_py import SingleFileMessagePackBackingStore +from cultcache_py.stores import record_version + +# variants:ruling:write-set-content-version-declared-ownership. The vector in tests/vectors/content-version is written only by +# the C# reference; every row of it is computed here from the store file, by the store's read and by record_version. +VECTORS = Path(__file__).resolve().parents[3] / "tests" / "vectors" / "content-version" + + +class ContentVersionVectorTests(unittest.TestCase): + def test_every_row_of_the_vector_equals_this_runtimes_version(self) -> None: + rows = msgpack.unpackb((VECTORS / "versions.msgpack").read_bytes(), raw=False) + self.assertEqual([key for key, _ in rows], ["alpha", "beta"]) + directory = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, directory, ignore_errors=True) + copy = directory / "store.cc" + shutil.copyfile(VECTORS / "store.cc", copy) + store = SingleFileMessagePackBackingStore(copy) + by_key = {envelope.key: envelope for envelope in store.pull_all()} + for key, version in rows: + with self.subTest(key=key): + found = store.read(key) + self.assertIsNotNone(found) + self.assertEqual(found[1], version) + self.assertEqual(record_version(by_key[key]), version) + self.assertIsNone(store.read("gamma")) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultcache-py/tests/test_cultcache.py b/packages/cultcache-py/tests/test_cultcache.py index a8c1dbe7..7ff4e297 100644 --- a/packages/cultcache-py/tests/test_cultcache.py +++ b/packages/cultcache-py/tests/test_cultcache.py @@ -12,6 +12,7 @@ from cultcache_py import ( CultCache, JsonLinesBackingStore, SingleFileMessagePackBackingStore, + StoreLinkedError, StoreUnreadableError, define_database_entry_type, define_document_type, @@ -85,7 +86,7 @@ class CultCacheTests(unittest.TestCase): ) cache.pull_all_backing_stores() cache.put(document, "item:potion", Item(name="Potion", category="Consumable", value=50)) - cache.update(document, "item:potion", lambda item: Item(item.name, item.category, item.value + 5)) + cache.put(document, "item:potion", Item(name="Potion", category="Consumable", value=55)) self.assertEqual(cache.get_required(document, "item:potion").value, 55) cache.delete(document, "item:potion") self.assertIsNone(cache.get(document, "item:potion")) @@ -1431,10 +1432,10 @@ class CultCacheTests(unittest.TestCase): self.assertNotIn("SECRET", str(caught.exception)) self.assertIn("not a valid store", str(caught.exception)) - # What a write does through a symbolic link today. R3 decides whether a write resolves - # the link or refuses it; this pins the current behaviour so that change is deliberate. + # A write at a store path that is a link is refused: a replace there would swap the link for a regular file and + # detach the store from its target. Reads through a live link keep working. @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") - def test_r3_decides_resolve_or_refuse_a_push_through_a_live_link_replaces_the_link(self) -> None: + def test_a_write_at_a_live_link_is_refused_and_nothing_changes_while_a_read_through_it_works(self) -> None: with tempfile.TemporaryDirectory() as tmp: volume = Path(tmp) / "volume" volume.mkdir() @@ -1442,12 +1443,91 @@ class CultCacheTests(unittest.TestCase): envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z") SingleFileMessagePackBackingStore(target).push(envelope) before = target.read_bytes() - link = Path(tmp) / "store.cc" + volume_before = sorted(p.name for p in volume.iterdir()) + link = Path(tmp) / "link.cc" link.symlink_to(target) - SingleFileMessagePackBackingStore(link).push(replace(envelope, key="b")) - self.assertFalse(link.is_symlink()) - self.assertTrue(link.is_file()) + store = SingleFileMessagePackBackingStore(link) + for write in ( + lambda: store.push(replace(envelope, key="b")), + lambda: store.push_all([replace(envelope, key="b")]), + lambda: store.delete("t", "a"), + ): + with self.assertRaises(StoreLinkedError) as caught: + write() + self.assertIn(str(link), str(caught.exception)) + self.assertTrue(link.is_symlink()) self.assertEqual(target.read_bytes(), before) + self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.cc", "volume"]) + self.assertEqual(sorted(p.name for p in volume.iterdir()), volume_before) + self.assertEqual([item.key for item in store.pull_all()], ["a"]) + + @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") + def test_a_json_lines_write_at_a_live_link_is_refused_and_the_link_survives(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + volume = Path(tmp) / "volume" + volume.mkdir() + target = volume / "store.jsonl" + envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z") + JsonLinesBackingStore(target).push(envelope) + before = target.read_bytes() + link = Path(tmp) / "link.jsonl" + link.symlink_to(target) + store = JsonLinesBackingStore(link) + for write in ( + lambda: store.push(replace(envelope, key="b")), + lambda: store.push_all([replace(envelope, key="b")]), + lambda: store.delete("t", "a"), + ): + with self.assertRaises(StoreLinkedError): + write() + self.assertTrue(link.is_symlink()) + self.assertEqual(target.read_bytes(), before) + self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.jsonl", "volume"]) + self.assertEqual(sorted(p.name for p in volume.iterdir()), ["store.jsonl"]) + + # The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse. + @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") + def test_is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + volume = Path(tmp) / "volume" + volume.mkdir() + target = volume / "store.cc" + document = item_doc() + seed = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(target)).build() + seed.put(document, "held", Item(name="Held", category="c", value=1)) + before = target.read_bytes() + link = Path(tmp) / "link.cc" + link.symlink_to(target) + cache = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(link)).build() + cache.pull_all_backing_stores() + names_before = (sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir())) + + self.assertTrue(cache.is_writable(document, "held")) + self.assertTrue(cache.is_writable(document, "fresh")) + with self.assertRaises(StoreLinkedError) as caught: + cache.put(document, "fresh", Item(name="Fresh", category="d", value=2)) + self.assertEqual(Path(caught.exception.path), link) + self.assertTrue(link.is_symlink()) + self.assertEqual(target.read_bytes(), before) + self.assertEqual((sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir())), names_before) + + def test_store_linked_error_is_an_os_error_carrying_the_path(self) -> None: + error = StoreLinkedError(Path("some/store.cc")) + self.assertIsInstance(error, OSError) + self.assertEqual(error.path, Path("some/store.cc")) + + @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") + def test_a_write_at_a_live_directory_link_is_refused_as_a_link(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + real = Path(tmp) / "real" + real.mkdir() + link = Path(tmp) / "dirlink" + link.symlink_to(real, target_is_directory=True) + with self.assertRaises(StoreLinkedError): + SingleFileMessagePackBackingStore(link).push( + CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")) + self.assertTrue(link.is_symlink()) + self.assertEqual(list(real.iterdir()), []) @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") def test_a_dangling_symlink_at_a_store_path_is_an_os_error_and_nothing_is_written(self) -> None: @@ -1460,7 +1540,7 @@ class CultCacheTests(unittest.TestCase): store = store_type(path) with self.assertRaises(FileNotFoundError): store.pull_all() - with self.assertRaises(FileNotFoundError): + with self.assertRaises(StoreLinkedError): store.push(CultCacheEnvelope(key="k", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")) self.assertTrue(path.is_symlink()) self.assertEqual(list(volume.iterdir()), []) diff --git a/packages/cultcache-py/tests/test_first_call.py b/packages/cultcache-py/tests/test_first_call.py index 10d6eb27..232f6c7d 100644 --- a/packages/cultcache-py/tests/test_first_call.py +++ b/packages/cultcache-py/tests/test_first_call.py @@ -4,6 +4,8 @@ import unittest from pathlib import Path +from cultcache_py import UpdateOutcome + # snippet:begin import tempfile from dataclasses import asdict, dataclass @@ -51,15 +53,44 @@ def first_call() -> Item: # snippet:end +# snippet:update:begin +def restock(path: Path): + cache = ( + CultCache.builder() + .register_document_type(item_doc) + .add_generic_store(SingleFileMessagePackBackingStore(path)) + .build() + ) + cache.pull_all_backing_stores() + return cache.update(item_doc, "item:potion", lambda item: Item("Potion", "Consumable", (item.value if item else 0) + 10)) +# snippet:update:end + + class FirstCallTests(unittest.TestCase): def test_first_call_round_trips_through_a_second_cache(self) -> None: self.assertEqual(first_call(), Item("Potion", "Consumable", 50)) - def test_docs_show_the_tested_snippet(self) -> None: + def test_update_with_retry_through_a_second_cache(self) -> None: + path = Path(tempfile.mkdtemp()) / "restock.cc" + restock(path) + result = restock(path) + self.assertIs(result.outcome, UpdateOutcome.UPDATED) + reopened = ( + CultCache.builder() + .register_document_type(item_doc) + .add_generic_store(SingleFileMessagePackBackingStore(path)) + .build() + ) + reopened.pull_all_backing_stores() + self.assertEqual(reopened.get_required(item_doc, "item:potion"), Item("Potion", "Consumable", 20)) + + def test_docs_show_the_tested_snippets(self) -> None: source = Path(__file__).read_text(encoding="utf-8") - snippet = source.split("# snippet:begin\n", 1)[1].split("# snippet:end\n", 1)[0] docs = (Path(__file__).parents[3] / "docs" / "start" / "README.md").read_text(encoding="utf-8") - self.assertIn("```python\n" + snippet + "```", docs) + for region in ("snippet", "snippet:update"): + with self.subTest(region=region): + snippet = source.split(f"# {region}:begin\n", 1)[1].split(f"# {region}:end\n", 1)[0] + self.assertIn("```python\n" + snippet + "```", docs) if __name__ == "__main__": diff --git a/packages/cultcache-py/tests/test_slot_readonly.py b/packages/cultcache-py/tests/test_slot_readonly.py new file mode 100644 index 00000000..68e389f3 --- /dev/null +++ b/packages/cultcache-py/tests/test_slot_readonly.py @@ -0,0 +1,509 @@ +from __future__ import annotations + +import dataclasses +import shutil +import threading +import tempfile +import unittest +from pathlib import Path + +import msgpack +from cultcache_py.cache import CultCacheError + +from cultcache_py import ( + CultCache, + CultCachePayloadDepthError, + SlotCodec, + SchemaConflictError, + SingleFileMessagePackBackingStore, + StoreUnreadableError, + UnionArm, + UnreadSlots, + database_entry_field, + define_database_entry_type, + define_record_codec, + leaves as t, +) +import test_union_parity as _parity +from test_union_parity import ( + VECTORS, + _surface_value, + _vector, + surface_document, + unit_outcome, +) + +SLOT = "extra-slot" # the canary: the value of every extra slot in the vectors; no refusal may carry it + +# The vectors name the C# types; this runtime names a type by its codec's name, a document by its document type. +_NAMES = {"UnionSurface": surface_document.type, "SurfaceDeclaring": surface_document.type, "Denied2": "Denied"} + +slot_unit_document = define_database_entry_type( + "tests.slot_unit", + [database_entry_field("name", 0, codec=t.string), database_entry_field("outcome", 1, codec=unit_outcome)], + name="name", +) +# The reader of extra-slot-consent.cc: the surface document's slots under a new schema id that lists the old one compatible. +declaring_document = define_database_entry_type( + "tests.union_surface", + [ + database_entry_field("id", 0, codec=t.string), + database_entry_field("version", 1, codec=t.int32), + database_entry_field("root", 2, codec=_parity._node), + database_entry_field("outcomes", 3, codec=t.list(_parity.outcome)), + database_entry_field("part", 4, codec=t.nullable(_parity.item)), + database_entry_field("crafted", 5, codec=t.nullable(_parity.crafted)), + database_entry_field("missing", 6, codec=_parity._node), + ], + name="id", + schema_id="tests.union_surface.v2", + schema_name="tests.union_surface", + compatible_schema_ids=("tests.union_surface.wide",), +) + +# file -> (type, slots carried, slots declared, every mark at depth 0): the README table. +_PAYLOADS = { + "extra-slot-record.msgpack": ("UnionSurface", 8, 7, True), + "extra-slot-nested-record.msgpack": ("Composition", 3, 2, False), + "extra-slot-arm.msgpack": ("Box", 3, 2, False), + "extra-slot-list-element.msgpack": ("Text", 3, 2, False), + "depth-skipped-at-limit.msgpack": ("UnionSurface", 8, 7, True), +} + + +def _copy_store_without_variants(source: Path, target: Path) -> None: + """This runtime holds no variant records (a five-slot record makes a store unreadable here), so the store vector is + walked with its variant row removed; every other byte is the C# writer's.""" + [snapshot] = list(msgpack.Unpacker(open(source, "rb"), raw=False)) + snapshot[2] = [record for record in snapshot[2] if len(record) <= 4] + target.write_bytes(msgpack.packb(snapshot, use_bin_type=True)) + + +def _open(path: Path, *documents) -> CultCache: + builder = CultCache.builder() + for document in documents: + builder.register_document_type(document) + cache = builder.add_generic_store(SingleFileMessagePackBackingStore(path)).build() + cache.pull_all_backing_stores() + return cache + + +def _rows(): + return msgpack.unpackb(_vector("slot-outcomes.msgpack"), raw=False) + + +class PayloadMarkTests(unittest.TestCase): + def test_every_read_only_payload_decodes_and_names_its_mark(self) -> None: + for file, (type_name, carried, declared, depth0) in _PAYLOADS.items(): + with self.subTest(file=file): + value, mark = surface_document.decode_payload_marked(_vector(file)) + self.assertEqual(value, _surface_value()) + self.assertEqual(mark, UnreadSlots(_NAMES.get(type_name, type_name), carried, declared, 1, depth0)) + self.assertEqual(surface_document.decode_payload(_vector(file)), _surface_value()) + + def test_the_outcomes_manifest_still_counts_nine_rows_and_four_read_only(self) -> None: + rows = msgpack.unpackb(_vector("outcomes.msgpack"), raw=False) + self.assertEqual(len(rows), 9) + self.assertEqual([row[0] for row in rows if row[1] == "read-only"], [f for f in _PAYLOADS if f != "depth-skipped-at-limit.msgpack"]) + + def test_a_clean_payload_and_a_short_one_carry_no_mark(self) -> None: + self.assertEqual(surface_document.decode_payload_marked(_vector("union-surface.msgpack")), (_surface_value(), None)) + slots = msgpack.unpackb(_vector("union-surface.msgpack"), raw=False) + slots[2] = [1, ["only-id"]] # a Text arm with a missing trailing slot: defaults, no mark + _, mark = surface_document.decode_payload_marked(msgpack.packb(slots, use_bin_type=True)) + self.assertIsNone(mark) + + def test_a_retired_slot_below_the_highest_key_does_not_mark(self) -> None: + slots = msgpack.unpackb(_vector("union-surface.msgpack"), raw=False) + # Image's declared slots are 0, 1, 3 (2 retired): four carried, four declared. + image = slots[2][1][1][2] + self.assertEqual(image[0], 5) + image[1][2] = "retired-slot-value" + _, mark = surface_document.decode_payload_marked(msgpack.packb(slots, use_bin_type=True)) + self.assertIsNone(mark) + + def test_the_depth_check_runs_before_the_mark(self) -> None: + with self.assertRaises(CultCachePayloadDepthError): + surface_document.decode_payload_marked(_vector("depth-skipped-past-limit.msgpack")) + + def test_the_unit_arm_with_extra_slots_is_read_past_and_marked(self) -> None: + payload = msgpack.packb(["unit-arm", [2, [SLOT]]], use_bin_type=True) + value, mark = slot_unit_document.decode_payload_marked(payload) + self.assertEqual(value, {"name": "unit-arm", "outcome": UnionArm("Denied")}) + self.assertEqual(mark, UnreadSlots("Denied", 1, 0, 1, False)) + self.assertEqual(slot_unit_document.decode_payload_marked(msgpack.packb(["ok", [2, []]]))[1], None) + + def test_a_codec_decoded_outside_a_payload_decode_notes_nothing(self) -> None: + self.assertEqual(unit_outcome.decode([2, [SLOT]]), UnionArm("Denied")) + + def test_the_scope_belongs_to_its_thread(self) -> None: + def decode_elsewhere(raw): + thread = threading.Thread(target=lambda: unit_outcome.decode([2, [SLOT]])) + thread.start() + thread.join() + return raw + + outer = define_database_entry_type("tests.threaded", [database_entry_field("a", 0, codec=SlotCodec(lambda v: v, decode_elsewhere))]) + self.assertIsNone(outer.decode_payload_marked(msgpack.packb(["x"]))[1]) + + def test_a_nested_payload_decode_restores_the_outer_scope(self) -> None: + def decode_clean_inside(raw): + surface_document.decode_payload_marked(_vector("union-surface.msgpack")) + return raw + + inner = define_record_codec("Inner", [database_entry_field("x", 0, codec=t.string)]) + outer = define_database_entry_type( + "tests.nested", + [ + database_entry_field("a", 0, codec=SlotCodec(lambda v: v, decode_clean_inside)), + database_entry_field("b", 1, codec=inner), + ], + ) + _, mark = outer.decode_payload_marked(msgpack.packb(["x", ["v", SLOT]])) + self.assertEqual(mark, UnreadSlots("Inner", 2, 1, 1, False)) + + +class StoreRowTests(unittest.TestCase): + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, self.tmp, True) + + def _store(self, name: str) -> tuple[Path, CultCache]: + path = self.tmp / name + _copy_store_without_variants(VECTORS / name, path) + documents = (declaring_document,) if name == "extra-slot-consent.cc" else (surface_document, slot_unit_document) + return path, _open(path, *documents) + + def _document(self, store: str, key: str): + if key == "unit-arm": + return slot_unit_document + return declaring_document if store == "extra-slot-consent.cc" else surface_document + + def test_the_variant_row_is_refused_as_a_store_not_walked_and_the_rest_of_the_store_reads_without_it(self) -> None: + # This runtime holds no variant records (ruling: refuse, not carry; the reference's marked-variant row, the one + # row with a flattenRefusedBy, is the C# reader's). A five-slot persisted record makes the whole store + # unreadable, the file is left as it was, and the store without exactly that row reads. + variants = [row for row in _rows() if row[8] is not None] + self.assertEqual([(row[0], row[1], row[8]) for row in variants], [("extra-slot-store.cc", "marked-variant", "marked")]) + path = self.tmp / "extra-slot-store.cc" + shutil.copyfile(VECTORS / "extra-slot-store.cc", path) + before = path.read_bytes() + with self.assertRaises(StoreUnreadableError): + _open(path, surface_document, slot_unit_document) + self.assertEqual(path.read_bytes(), before) + _, cache = self._store("extra-slot-store.cc") + self.assertIsNone(cache.get(surface_document, "marked-variant")) + self.assertIsNotNone(cache.get(surface_document, "marked")) + + def test_every_row_is_writable_exactly_where_the_vector_says_and_refuses_with_its_mark(self) -> None: + walked = 0 + for store, key, writable, explained, type_name, carried, declared, occurrences, flatten in _rows(): + if flatten is not None: + continue # a variant: this runtime holds none (deviation) + walked += 1 + with self.subTest(store=store, key=key): + path, cache = self._store(store) + document = self._document(store, key) + before = path.read_bytes() + self.assertEqual(cache.is_writable(document, key), writable) + held = cache.get(document, key) + self.assertIsNotNone(held) + if writable: + continue + self.assertEqual(explained, False) + for attempt in ( + lambda: cache.put(document, key, held), + lambda: cache.delete(document, key), + lambda: cache.put_envelope(document, cache.get_required_envelope(document, key)), + lambda: cache.put_envelopes(document, [cache.get_required_envelope(document, key)]), + lambda: cache.update(document, key, lambda value: value), + ): + with self.assertRaises(SchemaConflictError) as caught: + attempt() + refusal = caught.exception + self.assertEqual( + refusal.unread_slots, + UnreadSlots(_NAMES.get(type_name, type_name), carried, declared, occurrences, refusal.unread_slots.depth0_only), + ) + self.assertEqual(refusal.record_key, key) + message = str(refusal) + self.assertNotIn(SLOT, message) + self.assertIn(refusal.unread_slots.type_name, message) + self.assertIn(f"{carried} slots", message) + self.assertIn(f"declares {declared}", message) + self.assertEqual(path.read_bytes(), before) + self.assertTrue(cache.get(document, key) is held) + self.assertEqual(walked, 7) # eight rows, the variant not walked + + def test_a_key_no_store_holds_and_another_key_are_writable_while_a_marked_one_is_held(self) -> None: + path, cache = self._store("extra-slot-store.cc") + self.assertFalse(cache.is_writable(surface_document, "marked")) + self.assertTrue(cache.is_writable(surface_document, "nothing-here")) + cache.put(surface_document, "other", cache.get_required(surface_document, "clean")) + self.assertEqual(cache.get(surface_document, "other"), _surface_value()) + + def test_consent_lands_an_explained_mark_and_refuses_the_others(self) -> None: + path, cache = self._store("extra-slot-consent.cc") + self.assertTrue(cache.is_writable(declaring_document, "consented")) + cache.put(declaring_document, "consented", cache.get_required(declaring_document, "consented")) + [stored] = [e for e in SingleFileMessagePackBackingStore(path).pull_all() if e.key == "consented"] + self.assertEqual(stored.schema_id, "tests.union_surface.v2") + self.assertEqual(len(msgpack.unpackb(stored.payload, raw=False)), 7) # the consented slot is gone + self.assertTrue(cache.is_writable(declaring_document, "consented")) + for key in ("over-consent", "nested"): + self.assertFalse(cache.is_writable(declaring_document, key), key) + with self.assertRaises(SchemaConflictError): + cache.put(declaring_document, key, cache.get_required(declaring_document, key)) + + def test_a_reload_marks_again_and_a_write_of_a_clean_envelope_over_a_marked_key_is_refused(self) -> None: + path, cache = self._store("extra-slot-store.cc") + envelope = cache.get_required_envelope(surface_document, "clean") + with self.assertRaises(SchemaConflictError): + cache.put_envelope(surface_document, type(envelope)(**{**envelope.__dict__, "key": "marked"})) + cache.pull_all_backing_stores() + self.assertFalse(cache.is_writable(surface_document, "marked")) + + def test_a_written_envelope_that_carries_extra_slots_is_held_read_only(self) -> None: + path, cache = self._store("extra-slot-store.cc") + envelope = cache.get_required_envelope(surface_document, "clean") + carrying = type(envelope)(**{**envelope.__dict__, "key": "arrived", "payload": _vector("extra-slot-arm.msgpack")}) + cache.put_envelope(surface_document, carrying) + self.assertFalse(cache.is_writable(surface_document, "arrived")) + with self.assertRaises(SchemaConflictError): + cache.delete(surface_document, "arrived") + cache.pull_all_backing_stores() + self.assertFalse(cache.is_writable(surface_document, "arrived")) + + def test_a_batch_of_envelopes_that_carry_extra_slots_is_held_read_only(self) -> None: + path, cache = self._store("extra-slot-store.cc") + envelope = cache.get_required_envelope(surface_document, "clean") + batch = [ + type(envelope)(**{**envelope.__dict__, "key": key, "payload": _vector(file)}) + for key, file in (("b1", "extra-slot-arm.msgpack"), ("b2", "union-surface.msgpack")) + ] + cache.put_envelopes(surface_document, batch) + self.assertEqual([cache.is_writable(surface_document, key) for key in ("b1", "b2")], [False, True]) + + def test_a_mark_under_an_id_the_reader_does_not_list_compatible_is_not_consented(self) -> None: + unlisted = define_database_entry_type( + "tests.union_surface", + [database_entry_field("id", 0, codec=t.string)] + [database_entry_field(n, i, codec=c) for n, i, c in ( + ("version", 1, t.int32), ("root", 2, _parity._node), ("outcomes", 3, t.list(_parity.outcome)), + ("part", 4, t.nullable(_parity.item)), ("crafted", 5, t.nullable(_parity.crafted)), ("missing", 6, _parity._node))], + name="id", + schema_id="tests.union_surface.v2", + schema_name="tests.union_surface", + ) + path = self.tmp / "unlisted.cc" + _copy_store_without_variants(VECTORS / "extra-slot-consent.cc", path) + cache = _open(path, unlisted) + self.assertFalse(cache.is_writable(unlisted, "consented")) + + def test_a_written_envelope_is_judged_on_the_id_it_is_stored_under_so_a_reload_agrees(self) -> None: + path, cache = self._store("extra-slot-consent.cc") + arrived = next(e for e in SingleFileMessagePackBackingStore(path).pull_all() if e.key == "consented") + self.assertEqual(arrived.schema_id, "tests.union_surface.wide") + cache.put_envelope(declaring_document, dataclasses.replace(arrived, key="via-envelope")) + cache.put_envelopes(declaring_document, [dataclasses.replace(arrived, key="via-batch")]) + in_memory = [cache.is_writable(declaring_document, k) for k in ("via-envelope", "via-batch")] + cache.pull_all_backing_stores() + self.assertEqual([cache.is_writable(declaring_document, k) for k in ("via-envelope", "via-batch")], in_memory) + self.assertEqual(in_memory, [False, False]) + + def test_is_writable_agrees_with_the_write_for_a_key_a_store_carries_foreign(self) -> None: + path = self.tmp / "foreign.cc" + _copy_store_without_variants(VECTORS / "extra-slot-store.cc", path) + cache = _open(path, surface_document) # tests.slot_unit is not registered: its record is carried foreign + self.assertEqual([r.key for r in cache.foreign_records()], ["unit-arm"]) + self.assertFalse(cache.is_writable(surface_document, "unit-arm")) + with self.assertRaises(SchemaConflictError): + cache.put(surface_document, "unit-arm", _surface_value()) + with self.assertRaises(SchemaConflictError): + cache.delete(surface_document, "unit-arm") + + def test_every_key_of_a_batch_is_checked_wherever_the_marked_one_stands(self) -> None: + path, cache = self._store("extra-slot-store.cc") + before = path.read_bytes() + envelope = cache.get_required_envelope(surface_document, "clean") + fresh = dataclasses.replace(envelope, key="fresh") + marked = dataclasses.replace(envelope, key="marked") + for batch in ([fresh, marked], [marked, fresh], [fresh, dataclasses.replace(envelope, key="f2"), marked]): + with self.assertRaises(SchemaConflictError) as caught: + cache.put_envelopes(surface_document, batch) + self.assertEqual(caught.exception.record_key, "marked") + self.assertEqual(path.read_bytes(), before) + self.assertIsNone(cache.get(surface_document, "fresh")) + + def test_only_the_marks_that_hold_a_record_read_only_are_kept_by_key(self) -> None: + _, store = self._store("extra-slot-store.cc") + self.assertEqual({k: v[0] for k, v in store._state.unread.items()}, {"marked": surface_document.type, "unit-arm": "tests.slot_unit"}) + _, consent = self._store("extra-slot-consent.cc") + self.assertEqual(sorted(consent._state.unread), ["nested", "over-consent"]) # the consented mark is no state + + def test_an_arrival_with_no_catalog_entry_under_a_listed_id_is_not_consented(self) -> None: + wide = next(e for e in SingleFileMessagePackBackingStore(VECTORS / "extra-slot-consent.cc").pull_all() if e.key == "consented") + bare = dataclasses.replace(wide, catalog_entry=None) + + class Held: + def pull_all(self): + return [bare] + + def push(self, envelope): ... + + def push_all(self, envelopes): ... + + def delete(self, type, key): ... + + cache = CultCache.builder().register_document_type(declaring_document).add_generic_store(Held()).build() + cache.pull_all_backing_stores() + self.assertFalse(cache.is_writable(declaring_document, "consented")) + + def test_a_removal_is_refused_where_is_writable_says_no(self) -> None: + path, cache = self._store("extra-slot-store.cc") + before = path.read_bytes() + other = define_database_entry_type("tests.other", [database_entry_field("a", 0, codec=t.string)], name="a") + cache2 = _open(self.tmp / "extra-slot-store.cc", surface_document, slot_unit_document, other) + self.assertFalse(cache2.is_writable(other, "clean")) + with self.assertRaises(SchemaConflictError): + cache2.delete(other, "clean") + self.assertEqual(path.read_bytes(), before) + self.assertIsNotNone(cache2.get(surface_document, "clean")) + self.assertTrue(cache2.is_writable(other, "nothing-here")) + + def test_a_global_document_is_not_writable_at_a_key_other_than_the_global_one(self) -> None: + solo = define_database_entry_type("tests.solo", [database_entry_field("a", 0, codec=t.string)], global_document=True) + cache = CultCache.builder().register_document_type(solo).build() + self.assertFalse(cache.is_writable(solo, "elsewhere")) + with self.assertRaises(CultCacheError): + cache.put(solo, "elsewhere", {"a": "x"}) + with self.assertRaises(CultCacheError): + cache.delete(solo, "elsewhere") + self.assertTrue(cache.is_writable(solo, CultCache.GLOBAL_KEY)) + + def test_a_reload_replaces_the_marks_wholesale(self) -> None: + path, cache = self._store("extra-slot-store.cc") + self.assertFalse(cache.is_writable(surface_document, "marked")) + # The store is repaired by its owner: the marked record is rewritten clean. The reload must forget the old mark. + [snapshot] = list(msgpack.Unpacker(open(path, "rb"), raw=False)) + clean = next(r for r in snapshot[2] if r[0] == "clean") + for record in snapshot[2]: + if record[0] in ("marked", "unit-arm"): + record[3] = clean[3] if record[0] == "marked" else record[3] + snapshot[2] = [r for r in snapshot[2] if r[0] != "unit-arm"] + path.write_bytes(msgpack.packb(snapshot, use_bin_type=True)) + cache.pull_all_backing_stores() + self.assertTrue(cache.is_writable(surface_document, "marked")) + self.assertEqual(cache._state.unread, {}) + + def test_a_listed_id_whose_catalog_reports_no_extra_slot_does_not_consent(self) -> None: + wide = next(e for e in SingleFileMessagePackBackingStore(VECTORS / "extra-slot-consent.cc").pull_all() if e.key == "consented") + same = dataclasses.replace(wide, catalog_entry=declaring_document.catalog_entry()) # the stored entry has no slot past the reader's + + class Held: + def pull_all(self): + return [same] + + def push(self, envelope): ... + + def push_all(self, envelopes): ... + + def delete(self, type, key): ... + + cache = CultCache.builder().register_document_type(declaring_document).add_generic_store(Held()).build() + cache.pull_all_backing_stores() + self.assertFalse(cache.is_writable(declaring_document, "consented")) + + def test_a_record_stored_under_the_readers_own_id_is_never_consented(self) -> None: + own = dataclasses.replace( + next(e for e in SingleFileMessagePackBackingStore(VECTORS / "extra-slot-consent.cc").pull_all() if e.key == "consented"), + schema_id="tests.union_surface.v2", + ) + wider = dataclasses.replace(own.catalog_entry, schema_id="tests.union_surface.v2", compatible_schema_ids=("tests.union_surface.v2",)) + listing = dataclasses.replace( + declaring_document, schema_id="tests.union_surface.v2", compatible_schema_ids=("tests.union_surface.wide", "tests.union_surface.v2") + ) + + class Held: + def pull_all(self): + return [dataclasses.replace(own, catalog_entry=wider)] + + def push(self, envelope): ... + + def push_all(self, envelopes): ... + + def delete(self, type, key): ... + + cache = CultCache.builder().register_document_type(listing).add_generic_store(Held()).build() + cache.pull_all_backing_stores() + self.assertFalse(cache.is_writable(listing, "consented")) + + def test_a_removal_by_the_owner_of_a_key_held_as_another_type_removes_it_as_a_write_would_replace_it(self) -> None: + a = define_database_entry_type( + "tests.owned_a", [database_entry_field("a", 0, codec=t.string)], name="a", indexes={"by_a": "a"} + ) + b = define_database_entry_type( + "tests.owned_b", + [database_entry_field("a", 0, codec=t.string)], + compatible_schema_ids=(a.catalog_entry().schema_id,), + ) + path = self.tmp / "owned.cc" + writer = _open(path, a) + writer.put(a, "k", {"a": "x"}) + writer.put(a, "other", {"a": "y"}) + cache = _open(path, a, b) + self.assertTrue(cache.is_writable(b, "k")) + cache.delete(b, "k") + self.assertIsNone(cache.get(a, "k")) + self.assertIsNone(cache.get_by_index(a, "by_a", "x")) + self.assertEqual([e.key for e in SingleFileMessagePackBackingStore(path).pull_all()], ["other"]) + self.assertTrue(cache.is_writable(b, "k")) + + def test_an_owners_removal_is_one_store_call_so_the_cache_and_the_store_agree_when_it_fails(self) -> None: + a = define_database_entry_type("tests.one_a", [database_entry_field("a", 0, codec=t.string)], name="a") + b = define_database_entry_type( + "tests.one_b", [database_entry_field("a", 0, codec=t.string)], compatible_schema_ids=(a.catalog_entry().schema_id,) + ) + path = self.tmp / "one.cc" + _open(path, a).put(a, "k", {"a": "x"}) + + class SecondDeleteFails(SingleFileMessagePackBackingStore): + calls = 0 + + def delete(self, type, key): + self.calls += 1 + if self.calls > 1: + raise OSError("second delete") + super().delete(type, key) + + store = SecondDeleteFails(path) + cache = CultCache.builder().register_document_type(a).register_document_type(b).add_generic_store(store).build() + cache.pull_all_backing_stores() + cache.delete(b, "k") + self.assertEqual(store.calls, 1) + self.assertIsNone(cache.get(a, "k")) + self.assertEqual(SingleFileMessagePackBackingStore(path).pull_all(), []) + + def test_writing_another_record_and_flushing_carries_the_marked_one_unchanged(self) -> None: + path, cache = self._store("extra-slot-store.cc") + original = {e.key: e for e in SingleFileMessagePackBackingStore(path).pull_all()} + cache.put(surface_document, "clean", {**_surface_value(), "version": 9}) + after = {e.key: e for e in SingleFileMessagePackBackingStore(path).pull_all()} + for key in ("marked", "unit-arm"): + self.assertEqual(after[key].payload, original[key].payload, key) + self.assertEqual(after[key].schema_id, original[key].schema_id, key) + self.assertEqual(after[key].stored_at, original[key].stored_at, key) + self.assertNotEqual(after["clean"].payload, original["clean"].payload) + self.assertEqual(after["marked"].catalog_entry, original["marked"].catalog_entry) + + def test_the_vector_store_loads_under_the_reference_documents_and_marks_nothing_else(self) -> None: + path, cache = self._store("extra-slot-store.cc") + self.assertTrue(cache.is_writable(surface_document, "clean")) + self.assertFalse(cache.is_writable(surface_document, "marked")) + self.assertFalse(cache.is_writable(slot_unit_document, "unit-arm")) + self.assertEqual( + cache.get(slot_unit_document, "unit-arm"), {"name": "unit-arm", "outcome": UnionArm("Denied")} + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultcache-py/tests/test_soft_push_all.py b/packages/cultcache-py/tests/test_soft_push_all.py new file mode 100644 index 00000000..d30483a3 --- /dev/null +++ b/packages/cultcache-py/tests/test_soft_push_all.py @@ -0,0 +1,140 @@ +from __future__ import annotations + +import shutil +import tempfile +import unittest +from pathlib import Path + +from cultcache_py import JsonLinesBackingStore, SingleFileMessagePackBackingStore, StoreUnreadableError +from cultcache_py.backing_store import BackingStore, CultCacheEnvelope + + +def envelope(key: str) -> CultCacheEnvelope: + return CultCacheEnvelope(key=key, type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z") + + +class SoftPushAllTests(unittest.TestCase): + """A soft push writes where nothing is, skips a store it reads, and refuses one it cannot read (TS and Rust parity).""" + + def setUp(self) -> None: + self.dir = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, self.dir, ignore_errors=True) + self.path = self.dir / "state.cc" + + def test_over_a_readable_store_it_leaves_the_file_byte_for_byte(self) -> None: + for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): + with self.subTest(make.__name__): + path = self.dir / make.__name__ + store = make(path) + store.push_all([envelope("a")]) + before = path.read_bytes() + store.push_all([envelope("b")], soft=True) + self.assertEqual(before, path.read_bytes()) + self.assertEqual(["a"], [item.key for item in make(path).pull_all()]) + store.push_all([envelope("b")]) # a plain push still lands what the caller holds + self.assertEqual(["a", "b"], sorted(item.key for item in make(path).pull_all())) + + def test_over_a_readable_store_holding_no_records_it_leaves_the_file_byte_for_byte(self) -> None: + for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): + with self.subTest(make.__name__): + path = self.dir / make.__name__ + store = make(path) + store.push_all([envelope("a")]) + store.delete("t", "a") + self.assertTrue(path.exists()) + self.assertEqual([], make(path).pull_all()) + before = path.read_bytes() + store.push_all([envelope("b")], soft=True) + self.assertEqual(before, path.read_bytes()) + self.assertEqual([], make(path).pull_all()) + + def test_over_an_absent_path_it_writes(self) -> None: + for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): + with self.subTest(make.__name__): + path = self.dir / make.__name__ + make(path).push_all([envelope("a")], soft=True) + self.assertEqual(["a"], [item.key for item in make(path).pull_all()]) + + def test_over_an_unreadable_file_it_refuses_and_leaves_it(self) -> None: + junk = b"\xc1\x01\x02\x03" + self.path.write_bytes(junk) + with self.assertRaises(StoreUnreadableError): + SingleFileMessagePackBackingStore(self.path).push_all([envelope("a")], soft=True) + self.assertEqual(junk, self.path.read_bytes()) + + def test_over_an_unreadable_jsonl_file_it_refuses_and_leaves_it(self) -> None: + junk = b"this is not json\n" + self.path.write_bytes(junk) + with self.assertRaises(ValueError): + JsonLinesBackingStore(self.path).push_all([envelope("a")], soft=True) + self.assertEqual(junk, self.path.read_bytes()) + + def test_the_protocol_default_push_all_decides_soft_by_what_pull_or_absent_says(self) -> None: + class Memory(BackingStore): + def __init__(self) -> None: + self.items: list[CultCacheEnvelope] | None = None # None: nothing at the path + + def pull_all(self) -> list[CultCacheEnvelope]: + return list(self.items or []) + + def pull_or_absent(self) -> list[CultCacheEnvelope] | None: + return None if self.items is None else list(self.items) + + def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: + self.items = list(envelopes) + + store = Memory() + store.push_all([envelope("a")], soft=True) # nothing at the path: writes + self.assertEqual(["a"], [item.key for item in store.items or []]) + store.push_all([envelope("b")], soft=True) # a store with records: skips + self.assertEqual(["a"], [item.key for item in store.items or []]) + store.items = [] + store.push_all([envelope("b")], soft=True) # a store the reader read, empty: skips + self.assertEqual([], store.items) + store.push_all([envelope("b")]) # plain: lands + self.assertEqual(["b"], [item.key for item in store.items]) + + def test_a_store_that_only_pulls_all_is_read_so_a_soft_push_skips_it(self) -> None: + class PullOnly(BackingStore): + def __init__(self) -> None: + self.items: list[CultCacheEnvelope] = [] + + def pull_all(self) -> list[CultCacheEnvelope]: + return list(self.items) + + def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: + self.items = list(envelopes) + + store = PullOnly() + store.push_all([envelope("a")], soft=True) + self.assertEqual([], store.items) + + def test_the_single_file_and_jsonl_stores_say_absent_and_read_through_pull_or_absent(self) -> None: + for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): + with self.subTest(make.__name__): + path = self.dir / make.__name__ + store = make(path) + self.assertIsNone(store.pull_or_absent()) + store.push_all([envelope("a")]) + self.assertEqual(["a"], [item.key for item in store.pull_or_absent() or []]) + + def test_the_protocol_default_push_all_refuses_where_its_pull_refuses(self) -> None: + class Unreadable(BackingStore): + replaced = False + + def pull_all(self) -> list[CultCacheEnvelope]: + raise StoreUnreadableError(Path("x"), ValueError("junk")) + + def pull_or_absent(self) -> list[CultCacheEnvelope] | None: + raise StoreUnreadableError(Path("x"), ValueError("junk")) + + def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: + Unreadable.replaced = True + + with self.assertRaises(StoreUnreadableError): + Unreadable().push_all([envelope("a")], soft=True) + self.assertFalse(Unreadable.replaced) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultcache-py/tests/test_store_lock.py b/packages/cultcache-py/tests/test_store_lock.py index 5ee29aa2..364106bf 100644 --- a/packages/cultcache-py/tests/test_store_lock.py +++ b/packages/cultcache-py/tests/test_store_lock.py @@ -1,21 +1,25 @@ from __future__ import annotations import os +import select import shutil import subprocess import sys import tempfile +import threading import time import unittest from pathlib import Path from unittest import mock from cultcache_py import SingleFileMessagePackBackingStore, stores +from cultcache_py.stores import StoreLinkedError, store_write_lock from cultcache_py.backing_store import CultCacheEnvelope REPO = Path(__file__).resolve().parents[3] TS_STORE_FILE = REPO / "packages" / "cultcache-ts" / "dist" / "store-file.js" HOLD_SECONDS = 1.5 +HELD_WAIT_SECONDS = 3 # A holder that takes the lock the way the Rust runtime does (an exclusive file lock), not through this package. PY_HOLDER = """ @@ -41,7 +45,14 @@ def envelope(key: str) -> CultCacheEnvelope: def start_holder(argv: list[str]) -> subprocess.Popen[str]: holder = subprocess.Popen(argv, stdout=subprocess.PIPE, text=True) - assert holder.stdout is not None and holder.stdout.readline().strip() == "HELD" + assert holder.stdout is not None + ready, _, _ = select.select([holder.stdout], [], [], HELD_WAIT_SECONDS) # a holder blocked on a leaked lock never prints + if not ready: + holder.kill() + holder.wait() + holder.stdout.close() + raise AssertionError(f"the holder never took {argv[-2]}.lock within {HELD_WAIT_SECONDS} s: an earlier writer leaked the lock") + assert holder.stdout.readline().strip() == "HELD" holder.held_at = time.monotonic() # type: ignore[attr-defined] return holder @@ -73,6 +84,48 @@ class StoreLockTests(unittest.TestCase): holder = start_holder(["node", "-e", TS_HOLDER, str(TS_STORE_FILE), str(self.path), str(HOLD_SECONDS)]) self.assert_write_waits_for(holder) + def test_a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it(self) -> None: + target = self.dir / "volume" / "store.cc" + target.parent.mkdir() + SingleFileMessagePackBackingStore(target).push(envelope("a")) + before = target.read_bytes() + writes = { + "push": lambda store: store.push(envelope("b")), + "push_all": lambda store: store.push_all([envelope("b")]), + "delete": lambda store: store.delete("t", "a"), + } + missing = self.dir / "volume" / "missing.cc" + for (name, write), link_to in [(item, to) for item in writes.items() for to in (target, missing)]: + with self.subTest(name, dangling=link_to == missing): + if os.path.lexists(self.path): + os.unlink(self.path) + holder = start_holder([sys.executable, "-c", PY_HOLDER, str(self.path), str(HOLD_SECONDS)]) + self.addCleanup(holder.wait) + self.addCleanup(holder.stdout.close) # type: ignore[union-attr] + outcome: list[BaseException | None] = [] + + def run() -> None: + try: + write(SingleFileMessagePackBackingStore(self.path, lock_timeout=10)) + outcome.append(None) + except BaseException as exc: # noqa: BLE001 - the thread hands the outcome to the test + outcome.append(exc) + + writer = threading.Thread(target=run) + writer.start() + time.sleep(0.3) + os.symlink(link_to, self.path) # after the writer's pre-lock look, while it waits + writer.join() + holder.wait() + self.assertEqual(1, len(outcome)) + self.assertIsInstance(outcome[0], StoreLinkedError) + self.assertTrue(os.path.islink(self.path), "the link was replaced") + self.assertEqual(before, target.read_bytes()) + self.assertFalse(missing.exists(), "the dangling target was created") + os.unlink(self.path) # a link is refused before the lock, so remove it to probe the writer's own lock file + with store_write_lock(self.path, timeout=0.2): # the lock is free + pass + def test_the_lock_is_released_after_each_write(self) -> None: store = SingleFileMessagePackBackingStore(self.path, lock_timeout=0.2) for key in "abc": diff --git a/packages/cultcache-py/tests/test_store_replace.py b/packages/cultcache-py/tests/test_store_replace.py new file mode 100644 index 00000000..af8a2595 --- /dev/null +++ b/packages/cultcache-py/tests/test_store_replace.py @@ -0,0 +1,137 @@ +from __future__ import annotations + +import os +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +from cultcache_py import SingleFileMessagePackBackingStore, stores +from cultcache_py.backing_store import CultCacheEnvelope +from cultcache_py.stores import MOVEFILE_REPLACE_WRITE_THROUGH, ReplaceKernel, replace_store_windows + + +class FakeKernel(ReplaceKernel): + def __init__(self, posix: list[int], move_ex: list[int]) -> None: + self.posix, self.move_ex = list(posix), list(move_ex) + self.log: list[str] = [] + self.sleeps: list[float] = [] + + def rename_posix(self, staged, destination): + self.log.append("posix") + return self.posix.pop(0) if self.posix else 0 + + def move_file_ex(self, staged, destination, flags): + self.log.append(f"moveex:{flags}") + return self.move_ex.pop(0) if self.move_ex else 0 + + def sleep(self, seconds): + self.sleeps.append(seconds) + + +def run(posix, move_ex): + kernel = FakeKernel(posix, move_ex) + return kernel, lambda: replace_store_windows("a", "b", kernel) + + +class WindowsReplaceTable(unittest.TestCase): + def test_falls_back_to_write_through_move_where_posix_rename_is_unsupported(self): + self.assertEqual(MOVEFILE_REPLACE_WRITE_THROUGH, 0x9) + for unsupported in (1, 50, 87): + kernel, go = run([unsupported], [0]) + go() + self.assertEqual(kernel.log, ["posix", "moveex:9"]) + + def test_once_on_move_the_rest_of_the_call_stays_there(self): + kernel, go = run([87], [32, 0]) + go() + self.assertEqual(kernel.log, ["posix", "moveex:9", "moveex:9"]) + + def test_transient_codes_retry_the_same_primitive(self): + for transient in (5, 32, 33): + kernel, go = run([transient, transient, 0], []) + go() + self.assertEqual(kernel.log, ["posix", "posix", "posix"]) + self.assertEqual(kernel.sleeps, [0.02, 0.02]) + + def test_a_working_posix_rename_is_the_whole_replace(self): + kernel, go = run([0], []) + go() + self.assertEqual(kernel.log, ["posix"]) + self.assertEqual(kernel.sleeps, []) + + def test_unsupported_after_the_fallback_is_an_error_not_another_fallback(self): + kernel, go = run([87], [32, 87]) + with self.assertRaisesRegex(OSError, "Win32 error 87") as caught: + go() + self.assertEqual(caught.exception.winerror, 87) + self.assertEqual(kernel.log, ["posix", "moveex:9", "moveex:9"]) + for unsupported in (1, 50, 87): + kernel, go = run([87], [unsupported]) + with self.assertRaisesRegex(OSError, f"Win32 error {unsupported}"): + go() + self.assertEqual(kernel.log, ["posix", "moveex:9"]) + + def test_gives_up_after_25_attempts_naming_the_last_code(self): + kernel, go = run([32] * 30, []) + with self.assertRaisesRegex(OSError, "after 25 attempts: last Win32 error 32\\."): + go() + self.assertEqual(len(kernel.log), 25) + + def test_other_codes_fail_at_once(self): + kernel, go = run([2], []) + with self.assertRaisesRegex(OSError, "Win32 error 2"): + go() + self.assertEqual(len(kernel.log), 1) + kernel, go = run([87], [3]) + with self.assertRaisesRegex(OSError, "Win32 error 3"): + go() + + +class StoreWritesRouteThroughTheReplace(unittest.TestCase): + def test_both_stores_replace_through_replace_store(self): + seen: list[tuple[Path, Path]] = [] + real = stores._replace_store + + def spy(staged, destination): + seen.append((Path(staged), Path(destination))) + real(staged, destination) + + with tempfile.TemporaryDirectory() as directory, mock.patch.object(stores, "_replace_store", spy): + envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z") + binary = Path(directory) / "s.cc" + SingleFileMessagePackBackingStore(binary).push_all([envelope]) + lines = Path(directory) / "s.jsonl" + stores.JsonLinesBackingStore(lines).push_all([envelope]) + self.assertEqual([destination for _, destination in seen], [binary, lines]) + self.assertTrue(binary.read_bytes() and lines.read_text()) + + def test_a_failed_jsonl_replace_leaves_no_staged_file_and_the_old_store(self): + envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z") + with tempfile.TemporaryDirectory() as directory: + lines = Path(directory) / "s.jsonl" + store = stores.JsonLinesBackingStore(lines) + store.push_all([envelope]) + before = lines.read_bytes() + with mock.patch.object(stores, "_replace_store", side_effect=OSError("replace refused")): + with self.assertRaises(OSError): + store.push_all([CultCacheEnvelope(key="k2", type="t", payload=b"\x02", stored_at="2026-11-06T00:00:00Z")]) + self.assertEqual(lines.read_bytes(), before) + self.assertEqual([p.name for p in Path(directory).iterdir() if ".tmp" in p.name], []) + + @unittest.skipUnless(sys.platform == "win32", "needs a real Windows") + def test_real_kernel_replaces_through_both_primitives(self): + kernel = stores.WindowsReplaceKernel() + with tempfile.TemporaryDirectory() as directory: + for name, replace in (("posix", kernel.rename_posix), ("moveex", lambda s, d: kernel.move_file_ex(s, d, MOVEFILE_REPLACE_WRITE_THROUGH))): + staged, store = Path(directory, name + ".tmp"), Path(directory, name) + store.write_bytes(b"old") + staged.write_bytes(b"new") + self.assertEqual(replace(str(staged), str(store)), 0) + self.assertEqual(store.read_bytes(), b"new") + self.assertEqual(kernel.rename_posix(str(Path(directory, "missing")), str(Path(directory, "x"))), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultcache-py/tests/test_union_parity.py b/packages/cultcache-py/tests/test_union_parity.py index 451cd030..793933ff 100644 --- a/packages/cultcache-py/tests/test_union_parity.py +++ b/packages/cultcache-py/tests/test_union_parity.py @@ -261,7 +261,9 @@ class UnionSurfaceVectorTests(unittest.TestCase): else: read_only += 1 self.assertEqual(outcome_kind, "read-only", file) - self.assertEqual(surface_document.decode_payload(payload), _surface_value(), file) + value, mark = surface_document.decode_payload_marked(payload) + self.assertEqual(value, _surface_value(), file) + self.assertIsNotNone(mark, file) # read past and marked: tests/test_slot_readonly.py pins what the mark holds self.assertEqual((refused, read_only), (5, 4)) @@ -526,7 +528,9 @@ class BoundsVectorTests(unittest.TestCase): surface_document.encode_payload(surface_document.decode_payload(payload)), _vector(detail), file ) elif outcome_kind == "read-only": - self.assertEqual(surface_document.decode_payload(payload), _surface_value(), file) + value, mark = surface_document.decode_payload_marked(payload) + self.assertEqual(value, _surface_value(), file) + self.assertIsNotNone(mark, file) else: with self.assertRaises(ValueError, msg=file) as caught: surface_document.decode_payload(payload) @@ -657,6 +661,78 @@ def _as_tuple(tree: object) -> object: return tuple(_as_tuple(child) for child in tree) if isinstance(tree, list) else tree + +class BytesAfterVectorTests(unittest.TestCase): + CLAUSE = "bytes after its first MessagePack value (refused on read)" + + def _assert_chain_clean(self, error: BaseException, file: str) -> None: + link: BaseException | None = error + while link is not None: + self.assertNotIn(CANARY, str(link), file) + self.assertNotIsInstance(link, msgpack.ExtraData, file) + link = link.__cause__ or link.__context__ + + def test_every_bytes_after_vector_is_refused(self) -> None: + base = _vector("union-surface.msgpack") + entries = msgpack.unpackb(_vector("bytes-after-outcomes.msgpack"), raw=False) + self.assertEqual(len(entries), 4) + for file, outcome_kind, union, key, detail in entries: + self.assertEqual((outcome_kind, union, key, detail), ("refused", "UnionSurface", None, "bytes-after"), file) + payload = _vector(file) + self.assertTrue(payload.startswith(base), file) + count = len(payload) - len(base) + with self.assertRaises(ValueError, msg=file) as caught: + surface_document.decode_payload(payload) + error = caught.exception + self.assertIs(type(error), ValueError, file) + self.assertNotIsInstance(error, CultCachePayloadDepthError, file) + self.assertEqual( + str(error), f"{surface_document.type} payload has {count} {self.CLAUSE}.", file + ) + self.assertNotIn("the limit is", str(error), file) + self._assert_chain_clean(error, file) + + def test_a_single_file_store_holding_a_tail_as_a_record_payload_refuses_on_load(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "tail.cc" + shutil.copyfile(VECTORS / "union-surface.cc", path) + store = SingleFileMessagePackBackingStore(path) + [envelope] = store.pull_all() + store.push(dataclasses.replace(envelope, payload=_vector("bytes-after-nil.msgpack"))) + cache = ( + CultCache.builder() + .register_document_type(surface_document) + .add_generic_store(SingleFileMessagePackBackingStore(path)) + .build() + ) + with self.assertRaises(ValueError) as caught: + cache.pull_all_backing_stores() + self.assertIn(f"1 {self.CLAUSE}", str(caught.exception)) + self.assertIn(surface_document.type, str(caught.exception)) + self.assertIsNone(cache.get(surface_document, "surface")) + + def test_depth_wins_over_a_tail_whatever_the_unpackers_own_stack_limit(self) -> None: + # Vector-independent: a value nested past 256 is the depth refusal even with a tail, as C# throws at container 257 + # before it reaches the tail; 2000 is past msgpack's own stack, so the winner is not a library constant. + for depth in (257, 300, 900, 2000, 100000): + for tail in (b"\xc0", b"\xc1\xc1"): + with self.subTest(depth=depth, tail=len(tail)): + with self.assertRaises(CultCachePayloadDepthError) as caught: + surface_document.decode_payload(b"\x91" * depth + b"\xc0" + tail) + self.assertIn("refused on read", str(caught.exception)) + for depth in (1, 255, 256): + with self.subTest(at=depth): + with self.assertRaises(ValueError) as caught: + surface_document.decode_payload(b"\x91" * depth + b"\xc0\xc0") + self.assertNotIsInstance(caught.exception, CultCachePayloadDepthError) + self.assertIn(f"1 {self.CLAUSE}", str(caught.exception)) + + def test_a_payload_that_ends_early_is_the_decoders_not_bytes_after(self) -> None: + truncated = _vector("union-surface.msgpack")[:-1] + with self.assertRaises(Exception) as caught: + surface_document.decode_payload(truncated) + self.assertNotIn("bytes after", str(caught.exception)) + class PayloadDepthRuleTests(unittest.TestCase): """The nesting rule at its seam, on definitions that are not unions: the same routine serves any payload codec.""" diff --git a/packages/cultcache-py/tests/test_unique_index.py b/packages/cultcache-py/tests/test_unique_index.py index accc27dd..6e831b9a 100644 --- a/packages/cultcache-py/tests/test_unique_index.py +++ b/packages/cultcache-py/tests/test_unique_index.py @@ -148,7 +148,6 @@ class UniqueIndexTests(unittest.TestCase): "put": lambda cache, d, k, i: cache.put(d, k, i), "put_envelope": lambda cache, d, k, i: cache.put_envelope(d, envelope(d, k, i)), "put_envelopes": lambda cache, d, k, i: cache.put_envelopes(d, [envelope(d, k, i)]), - "update": lambda cache, d, k, i: cache.update(d, k, lambda _held: i), } for name, write in paths.items(): with self.subTest(alias=alias, path=name): diff --git a/packages/cultcache-py/tests/test_update.py b/packages/cultcache-py/tests/test_update.py new file mode 100644 index 00000000..107adf3b --- /dev/null +++ b/packages/cultcache-py/tests/test_update.py @@ -0,0 +1,731 @@ +"""CultCache.update (variants:cut_spec:cut-update-with-retry-py): read the record as its home store holds it, apply a function, +commit only if the record is unchanged, run the function again on a lost race. Mirrors C# UpdateTests and the TS update tests.""" + +from __future__ import annotations + +import shutil +import tempfile +import threading +import time +import unittest +from dataclasses import asdict, dataclass +from datetime import datetime, timezone +from pathlib import Path + +from cultcache_py import ( + CultCache, + CultCacheEnvelope, + JsonLinesBackingStore, + SchemaConflictError, + SingleFileMessagePackBackingStore, + UpdateOutcome, + UpdateResult, + database_entry_field, + define_database_entry_type, + define_document_type, + leaves as t, +) +from cultcache_py.cache import GLOBAL_KEY, CultCacheError +from cultcache_py.stores import parse_stored_at, mint_stored_at, record_version, stored_at_over + + +@dataclass +class Item: + name: str = "n" + value: int = 0 + sku: str | None = None + + +def _doc(type: str, **extra): + return define_document_type(type, encode=asdict, decode=lambda raw: Item(**raw), **extra) + + +item_doc = _doc("item") +other_doc = _doc("other") +sku_doc = _doc("skuitem", indexes={"sku": "sku"}) +global_doc = _doc("gitem", global_document=True) + +@dataclass +class Wide: + name: str = "" + extra: str = "" + + +@dataclass +class Narrow: + name: str = "" + + +ALPHA = "2099-01-01T00:00:00.0000000Z" + + +class Fixture(unittest.TestCase): + def setUp(self) -> None: + self.dir = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, self.dir, ignore_errors=True) + self.path = self.dir / "state.cc" + + def cache(self, *documents, path: Path | None = None, store=True) -> CultCache: + builder = CultCache.builder() + for document in documents or (item_doc,): + builder.register_document_type(document) + if store: + builder.add_generic_store(SingleFileMessagePackBackingStore(path or self.path)) + cache = builder.build() + if store: + cache.pull_all_backing_stores() + return cache + + def stored(self, key: str = "k", document=item_doc): + """What a fresh cache reads at the key.""" + return self.cache(document).get(document, key) + + def bytes(self) -> bytes | None: + return self.path.read_bytes() if self.path.exists() else None + + +class Counting: + """An apply that records what it was given.""" + + def __init__(self, fn=None) -> None: + self.seen: list = [] + self.fn = fn or (lambda item, calls: Item("n", (item.value if item else 0) + 1)) + + def __call__(self, item): + self.seen.append(None if item is None else item.value) + # An always-losing apply must fail by assertion when the attempt bound is gone, not hang. + assert len(self.seen) <= 10, "apply ran past the attempt bound" + return self.fn(item, len(self.seen)) + + @property + def calls(self) -> int: + return len(self.seen) + + +class UpdateTests(Fixture): + def test_two_writers_updating_lose_no_increment(self) -> None: + results: list[UpdateResult] = [] + calls = [0] + lock = threading.Lock() + a, b = self.cache(), self.cache() + a.put(item_doc, "k", Item("n", 0)) + + def bump(item): + with lock: + calls[0] += 1 + time.sleep(0) + return Item("n", item.value + 1) + + def run(cache) -> None: + for _ in range(50): + results.append(cache.update(item_doc, "k", bump, max_attempts=200)) + + threads = [threading.Thread(target=run, args=(cache,)) for cache in (a, b)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + self.assertTrue(all(result.outcome is UpdateOutcome.UPDATED for result in results)) + self.assertEqual(self.stored().value, 100) + self.assertGreater(calls[0], 100) + + def test_a_lost_attempt_reruns_apply_on_the_fresh_value(self) -> None: + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "k", Item("n", 1)) + rival.pull_all_backing_stores() + apply = Counting(lambda item, calls: (rival.put(item_doc, "k", Item("n", 100)) if calls == 1 else None) or Item("n", item.value + 1)) + result = cache.update(item_doc, "k", apply) + self.assertEqual(apply.seen, [1, 100]) + self.assertIs(result.outcome, UpdateOutcome.UPDATED) + self.assertEqual(result.document.value, 101) + self.assertEqual(self.stored().value, 101) + + def test_exhausted_attempts_return_lost_race(self) -> None: + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "k", Item("n", 1)) + rival.pull_all_backing_stores() + apply = Counting(lambda item, calls: rival.put(item_doc, "k", Item("n", 100 * calls)) or Item("n", -1)) + result = cache.update(item_doc, "k", apply, max_attempts=3) + self.assertEqual(apply.calls, 3) + self.assertIs(result.outcome, UpdateOutcome.LOST_RACE) + self.assertEqual(result.document.value, 300) # the pull before LOST_RACE shows the winner + self.assertEqual(self.stored().value, 300) + + def test_max_attempts_bounds(self) -> None: + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "k", Item("n", 1)) + rival.pull_all_backing_stores() + + def always_losing(item, calls): + rival.put(item_doc, "k", Item("n", 100 + calls)) + return Item("n", -1) + + apply = Counting(always_losing) + result = cache.update(item_doc, "k", apply, max_attempts=1) + self.assertEqual((apply.calls, result.outcome), (1, UpdateOutcome.LOST_RACE)) + for bad, error in ((0, ValueError), (-1, ValueError), (True, TypeError), (1.5, TypeError), (None, TypeError)): + with self.subTest(max_attempts=bad): + untouched = Counting() + with self.assertRaises(error): + cache.update(item_doc, "k", untouched, max_attempts=bad) + self.assertEqual(untouched.calls, 0) + with self.assertRaises(TypeError): + cache.update(item_doc, "k", "not callable") + with self.assertRaises(TypeError): + cache.update(item_doc, "k", None) + + def test_apply_gets_a_fresh_copy(self) -> None: + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "k", Item("n", 1)) + rival.pull_all_backing_stores() + arguments: list[Item] = [] + + def apply(item): + arguments.append(item) + self.assertIsNot(item, cache.get(item_doc, "k")) + if len(arguments) == 1: + item.value = 777 # changing the copy in place changes nothing the cache or the next attempt sees + rival.put(item_doc, "k", Item("n", 5)) + self.assertEqual(cache.get(item_doc, "k").value, 1) + return Item("n", item.value + 1) + + result = cache.update(item_doc, "k", apply) + self.assertEqual([item.value for item in arguments], [777, 5]) + self.assertIsNot(arguments[0], arguments[1]) + self.assertEqual(result.document.value, 6) + + def test_an_in_place_change_to_the_held_value_is_neither_applied_nor_persisted(self) -> None: + cache = self.cache() + cache.put(item_doc, "k", Item("n", 1)) + cache.get(item_doc, "k").value = 999 + apply = Counting(lambda item, calls: Item("n", item.value + 1)) + result = cache.update(item_doc, "k", apply) + self.assertEqual(apply.seen, [1]) + self.assertEqual(self.stored().value, 2) + result.document.value = 999 + cache.update(item_doc, "k", apply) + self.assertEqual(apply.seen, [1, 2]) + self.assertEqual(self.stored().value, 3) + + bare = self.cache(store=False) # a cache with no store copies its held instance + bare.put(item_doc, "k", Item("n", 1)) + bare.get(item_doc, "k").value = 999 + seen = Counting(lambda item, calls: Item("n", item.value + 1)) + self.assertEqual(bare.update(item_doc, "k", seen).document.value, 2) + self.assertEqual(seen.seen, [1]) + + def test_none_declines(self) -> None: + cache = self.cache() + cache.put(item_doc, "k", Item("n", 1)) + before = self.bytes() + result = cache.update(item_doc, "k", lambda item: None) + self.assertIs(result.outcome, UpdateOutcome.DECLINED) + self.assertEqual(result.document, cache.get(item_doc, "k")) + self.assertEqual(self.bytes(), before) + absent = cache.update(item_doc, "missing", lambda item: None) + self.assertEqual((absent.outcome, absent.document), (UpdateOutcome.DECLINED, None)) + self.assertEqual(self.bytes(), before) + empty = Fixture.cache(self, path=self.dir / "empty.cc") + self.assertEqual(empty.update(item_doc, "k", lambda item: None).outcome, UpdateOutcome.DECLINED) + self.assertFalse((self.dir / "empty.cc").exists()) + + def test_a_declined_or_lost_attempt_installs_nothing(self) -> None: + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "k", Item("n", 1)) + rival.pull_all_backing_stores() + rival.put(item_doc, "k", Item("n", 5)) + self.assertIs(cache.update(item_doc, "k", lambda item: None).outcome, UpdateOutcome.DECLINED) + self.assertEqual(cache.get(item_doc, "k").value, 1) + cache.pull_all_backing_stores() + self.assertEqual(cache.get(item_doc, "k").value, 5) + + during: list[int] = [] + + def apply(item, calls): + during.append(cache.get(item_doc, "k").value) + if calls == 1: + rival.put(item_doc, "k", Item("n", 9)) + return Item("n", item.value + 1) + + cache.update(item_doc, "k", Counting(apply)) + self.assertEqual(during, [5, 5]) # the lost attempt published nothing mid-update + + def test_absent_key_rival_removal_and_a_store_that_holds_nothing(self) -> None: + a, b = self.cache(), self.cache() + bump = lambda item: Item("n", (item.value if item else 0) + 1) # noqa: E731 + self.assertIs(a.update(item_doc, "k", bump).outcome, UpdateOutcome.UPDATED) + self.assertIs(b.update(item_doc, "k", bump).outcome, UpdateOutcome.UPDATED) + self.assertEqual(self.stored().value, 2) + + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "d", Item("n", 1)) + rival.pull_all_backing_stores() + apply = Counting(lambda item, calls: (rival.delete(item_doc, "d") if calls == 1 else None) or Item("n", (item.value if item else 0) + 1)) + cache.update(item_doc, "d", apply) + self.assertEqual(apply.seen, [1, None]) + self.assertEqual(self.stored("d").value, 1) + + holder = self.cache() + holder.put(item_doc, "h", Item("n", 3)) + rival.pull_all_backing_stores() + rival.delete(item_doc, "h") + nothing = Counting(lambda item, calls: Item("n", 1)) + holder.update(item_doc, "h", nothing) + self.assertEqual(nothing.seen, [None]) + + def test_a_rival_writing_and_restoring_the_value_still_reruns_apply(self) -> None: + cache, rival = self.cache(), self.cache() + cache.put(item_doc, "k", Item("n", 1)) + rival.pull_all_backing_stores() + + def apply(item, calls): + if calls == 1: + rival.put(item_doc, "k", Item("n", 5)) + rival.put(item_doc, "k", Item("n", 1)) + return Item("n", item.value + 1) + + counting = Counting(apply) + cache.update(item_doc, "k", counting) + self.assertEqual(counting.seen, [1, 1]) + self.assertEqual(self.stored().value, 2) + + def test_a_stale_caches_first_attempt_reads_the_store(self) -> None: + a, b = self.cache(), self.cache() + a.put(item_doc, "k", Item("n", 1)) + b.pull_all_backing_stores() + a.put(item_doc, "k", Item("n", 7)) + apply = Counting(lambda item, calls: Item("n", item.value + 1)) + b.update(item_doc, "k", apply) + self.assertEqual(apply.seen, [7]) + self.assertEqual(self.stored().value, 8) + + +class UpdateRefusalTests(Fixture): + def refused(self, cache: CultCache, key: str = "k", document=item_doc, *errors) -> None: + before = self.bytes() + apply = Counting() + with self.assertRaises(errors or (CultCacheError, SchemaConflictError)): + cache.update(document, key, apply) + self.assertEqual(apply.calls, 0) + self.assertEqual(self.bytes(), before) + + def test_refusals_before_apply(self) -> None: + writer = self.cache(item_doc, other_doc) + writer.put(other_doc, "k", Item("n", 1)) + self.refused(writer) # the record at the key is another document's, held + self.refused(self.fresh(item_doc, other_doc)) # unpulled: the store's record is another document's + self.refused(self.cache(item_doc)) # foreign: the store holds a type this cache never registered + self.refused(self.fresh(item_doc)) + storeless = self.cache(item_doc, other_doc, store=False) + storeless.put(other_doc, "k", Item("n", 1)) + apply = Counting() + with self.assertRaises((CultCacheError, SchemaConflictError)): + storeless.update(item_doc, "k", apply) + self.assertEqual(apply.calls, 0) + + def fresh(self, *documents) -> CultCache: + """A cache that has not pulled: whatever it reads, it reads from the store at update.""" + builder = CultCache.builder() + for document in documents: + builder.register_document_type(document) + return builder.add_generic_store(SingleFileMessagePackBackingStore(self.path)).build() + + def test_a_home_without_read_and_commit_if_refuses(self) -> None: + jsonl = JsonLinesBackingStore(self.dir / "state.jsonl") + cache = CultCache.builder().register_document_type(item_doc).add_generic_store(jsonl).build() + cache.pull_all_backing_stores() + cache.put(item_doc, "k", Item("n", 1)) + before = (self.dir / "state.jsonl").read_bytes() + apply = Counting() + with self.assertRaisesRegex(CultCacheError, "cannot write conditionally"): + cache.update(item_doc, "k", apply) + self.assertEqual(apply.calls, 0) + self.assertEqual((self.dir / "state.jsonl").read_bytes(), before) + self.assertFalse(hasattr(JsonLinesBackingStore, "read") or hasattr(JsonLinesBackingStore, "commit_if")) + + class Bare: + def pull_all(self): + return [] + + def push(self, envelope): + raise AssertionError("update wrote through a store that cannot write conditionally") + + def push_all(self, envelopes, soft=False): + raise AssertionError("push_all") + + def delete(self, type, key): + raise AssertionError("delete") + + bare = CultCache.builder().register_document_type(item_doc).add_generic_store(Bare()).build() + with self.assertRaisesRegex(CultCacheError, "Bare"): + bare.update(item_doc, "k", apply) + self.assertEqual(apply.calls, 0) + + def test_read_only_records_refuse_before_apply(self) -> None: + wide = define_database_entry_type( + "tests.rw", + [database_entry_field("name", 0, codec=t.string), database_entry_field("extra", 1, codec=t.string)], + cls=Wide, + schema_id="tests.rw.v1", + ) + narrow = define_database_entry_type("tests.rw", [database_entry_field("name", 0, codec=t.string)], cls=Narrow, schema_id="tests.rw.v1") + self.cache(wide).put(wide, "k", Wide("n", "x")) + before = self.bytes() + for pulled in (True, False): # held read-only, then a store record the cache never loaded + with self.subTest(pulled=pulled): + cache = self.cache(narrow) if pulled else self.fresh(narrow) + apply = Counting() + with self.assertRaises(SchemaConflictError) as caught: + cache.update(narrow, "k", apply) + self.assertEqual(caught.exception.record_key, "k") + self.assertEqual(apply.calls, 0) + self.assertEqual(self.bytes(), before) + + def test_a_non_global_key_for_a_global_document_refuses(self) -> None: + cache = self.cache(global_doc) + apply = Counting() + with self.assertRaises(CultCacheError): + cache.update(global_doc, "not-global", apply) + self.assertEqual(apply.calls, 0) + + def test_refusals_after_apply_are_not_retried(self) -> None: + cache = self.cache() + cache.put(item_doc, "k", Item("n", 1)) + before = self.bytes() + boom = Counting(lambda item, calls: (_ for _ in ()).throw(RuntimeError("boom"))) + with self.assertRaisesRegex(RuntimeError, "boom"): + cache.update(item_doc, "k", boom) + self.assertEqual(boom.calls, 1) + self.assertEqual(self.bytes(), before) + + indexed = self.cache(sku_doc, path=self.dir / "sku.cc") + indexed.put(sku_doc, "a", Item("n", 1, "X")) + indexed.put(sku_doc, "b", Item("n", 1, "Y")) + before = (self.dir / "sku.cc").read_bytes() + clash = Counting(lambda item, calls: Item("n", 2, "X")) + with self.assertRaisesRegex(CultCacheError, "unique index"): + indexed.update(sku_doc, "b", clash) + self.assertEqual(clash.calls, 1) + self.assertEqual((self.dir / "sku.cc").read_bytes(), before) + + +class UpdateOwnWritesTests(Fixture): + def test_own_writes_and_the_held_instance(self) -> None: + cache = self.cache() + cache.put(item_doc, "k", Item("n", 1)) + apply = Counting(lambda item, calls: Item("n", item.value + 1)) + cache.update(item_doc, "k", apply) + second = cache.update(item_doc, "k", apply) + self.assertEqual(apply.calls, 2) + self.assertIs(second.document, cache.get(item_doc, "k")) + self.assertEqual(second.document.value, 3) + + def sideways(item, calls): + cache.put(item_doc, "side", Item("n", 5)) # apply runs outside the cache's hold + return Item("n", item.value + 1) + + sideways_apply = Counting(sideways) + result = cache.update(item_doc, "k", sideways_apply) + self.assertEqual((result.outcome, sideways_apply.calls), (UpdateOutcome.UPDATED, 1)) + self.assertEqual(self.stored("side").value, 5) + + def test_store_less_cache(self) -> None: + cache = self.cache(store=False) + cache.put(item_doc, "k", Item("n", 0)) + results: list[UpdateResult] = [] + + def bump(item): + time.sleep(0) + return Item("n", item.value + 1) + + def run() -> None: + for _ in range(50): + results.append(cache.update(item_doc, "k", bump, max_attempts=500)) + + threads = [threading.Thread(target=run) for _ in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + self.assertTrue(all(result.outcome is UpdateOutcome.UPDATED for result in results)) + self.assertEqual(cache.get(item_doc, "k").value, 100) + + # A store that becomes the type's home while apply runs holds a record the cache never read: the commit loses. + cache = self.cache(store=False) # holds no record of the type, so a store may become its home + other = self.cache(path=self.dir / "late.cc") + other.put(item_doc, "fresh", Item("n", 40)) + late = SingleFileMessagePackBackingStore(self.dir / "late.cc") + + def attach(item, calls): + if calls == 1: + cache.add_generic_store(late) + return Item("n", (item.value if item else 0) + 1) + + apply = Counting(attach) + result = cache.update(item_doc, "fresh", apply) + self.assertEqual(apply.seen, [None, 40]) + self.assertIs(result.outcome, UpdateOutcome.UPDATED) + self.assertEqual(self.cache(path=self.dir / "late.cc").get(item_doc, "fresh").value, 41) + + def test_a_home_attached_while_apply_runs_is_reread_or_refused(self) -> None: + """The cache's home for the type changes between the read and the commit (attaching is allowed while it holds no record + of the type): the version read from the old home is no condition on the new one.""" + cache = self.cache(path=self.dir / "generic.cc") + late = SingleFileMessagePackBackingStore(self.dir / "late.cc") + + def attach(item, calls): + if calls == 1: + cache.add_backing_store(late, ["item"]) + return Item("n", 1) + + apply = Counting(attach) + self.assertIs(cache.update(item_doc, "k", apply).outcome, UpdateOutcome.UPDATED) + self.assertEqual(apply.seen, [None, None]) # reread from the new home, not committed with the old home's version + self.assertEqual(self.cache(path=self.dir / "late.cc").get(item_doc, "k").value, 1) + + cache = self.cache(path=self.dir / "generic2.cc") + jsonl = JsonLinesBackingStore(self.dir / "late.jsonl") + + def attach_jsonl(item, calls): + cache.add_backing_store(jsonl, ["item"]) + return Item("n", 1) + + apply = Counting(attach_jsonl) + with self.assertRaisesRegex(CultCacheError, "cannot write conditionally"): + cache.update(item_doc, "k", apply) + self.assertEqual(apply.calls, 1) + self.assertFalse((self.dir / "late.jsonl").exists()) + + def test_ledger_under_contention(self) -> None: + caches = [self.cache() for _ in range(3)] + outcomes: list[UpdateOutcome] = [] + + def run(cache) -> None: + for _ in range(25): + outcomes.append(cache.update(item_doc, "k", lambda item: Item("n", (item.value if item else 0) + 1)).outcome) + + threads = [threading.Thread(target=run, args=(cache,)) for cache in caches] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + self.assertEqual(self.stored().value, outcomes.count(UpdateOutcome.UPDATED)) + self.assertEqual(len(outcomes), 75) + + +class UpdateLegacyGlobalTests(Fixture): + def plant(self, key: str = "legacy") -> None: + entry = global_doc.catalog_entry() + SingleFileMessagePackBackingStore(self.path).push( + CultCacheEnvelope.create(key=key, type=global_doc.type, payload=global_doc.encode_payload(Item("n", 1)), schema_id=entry.schema_id, catalog_entry=entry) + ) + + def test_update_global_updates_the_global_record(self) -> None: + cache = self.cache(global_doc) + result = cache.update_global(global_doc, lambda item: Item("n", (item.value if item else 0) + 1)) + self.assertIs(result.outcome, UpdateOutcome.UPDATED) + self.assertEqual(self.cache(global_doc).get(global_doc, GLOBAL_KEY).value, 1) + + def test_a_legacy_global_is_refused_until_put_global_retires_it(self) -> None: + self.plant() + cache = self.cache(global_doc) # loaded from the legacy key + before = self.bytes() + for call in (lambda a: cache.update_global(global_doc, a), lambda a: cache.update(global_doc, GLOBAL_KEY, a)): + apply = Counting() + with self.assertRaisesRegex(CultCacheError, "put_global"): + call(apply) + self.assertEqual(apply.calls, 0) + self.assertEqual(self.bytes(), before) + # Judged from the cache's load, not only from the store: the legacy record is gone from the store, the cache still holds it. + SingleFileMessagePackBackingStore(self.path).delete(global_doc.type, "legacy") + apply = Counting() + with self.assertRaisesRegex(CultCacheError, "put_global"): + cache.update_global(global_doc, apply) + self.assertEqual(apply.calls, 0) + cache.put_global(global_doc, Item("n", 5)) + self.assertEqual(cache.update_global(global_doc, lambda item: Item("n", item.value + 1)).document.value, 6) + + def test_a_pull_inside_apply_that_loads_a_legacy_global_fails_the_commit(self) -> None: + cache = self.cache(global_doc) + + def apply(item): + self.plant() + cache.pull_all_backing_stores() + return Item("n", 9) + + with self.assertRaisesRegex(CultCacheError, "put_global"): + cache.update_global(global_doc, apply) + self.assertEqual([envelope.key for envelope in SingleFileMessagePackBackingStore(self.path).pull_all()], ["legacy"]) + + def test_an_unpulled_cache_judges_the_legacy_global_from_the_store(self) -> None: + self.plant() + before = self.bytes() + cache = SingleFileMessagePackBackingStore(self.path) + unpulled = CultCache.builder().register_document_type(global_doc).add_generic_store(cache).build() + apply = Counting() + with self.assertRaisesRegex(CultCacheError, "put_global"): + unpulled.update_global(global_doc, apply) + self.assertEqual(apply.calls, 0) + self.assertEqual(self.bytes(), before) + + +class StoreRestampTests(Fixture): + def envelope(self, value: int, stored_at: str = ALPHA, key: str = "k") -> CultCacheEnvelope: + entry = item_doc.catalog_entry() + return CultCacheEnvelope(key, "item", item_doc.encode_payload(Item("n", value)), stored_at, entry.schema_id, entry) + + def stamp_on_disk(self, store, key: str = "k") -> str: + return next(envelope.stored_at for envelope in store.pull_all() if envelope.key == key) + + def later_than_alpha(self, text: str) -> bool: + return parse_stored_at(text) > parse_stored_at(ALPHA) + + def test_commit_if_and_the_restamp_at_the_store(self) -> None: + store = SingleFileMessagePackBackingStore(self.path) + self.assertIsNone(store.read("k")) + stamp = store.commit_if(self.envelope(1), None) # None against an absent key commits + self.assertEqual(stamp, ALPHA) + envelope, version = store.read("k") + self.assertEqual(version, record_version(envelope)) + before = self.bytes() + self.assertIsNone(store.commit_if(self.envelope(2, "2100-01-01T00:00:00Z"), None)) # a record is there: not absent + self.assertIsNone(store.commit_if(self.envelope(2, "2100-01-01T00:00:00Z"), "0" * 64)) + self.assertEqual(self.bytes(), before) + written = store.commit_if(self.envelope(2), version) # a staged stamp that is no later is minted over + self.assertTrue(self.later_than_alpha(written)) + self.assertEqual(self.stamp_on_disk(store), written) + self.assertEqual(stamp_of(store, "k"), written) + + def test_every_replacing_write_is_stored_strictly_later(self) -> None: + for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore): + for name in ("push", "push_all", "commit_if"): + if name == "commit_if" and make is JsonLinesBackingStore: + continue + with self.subTest(store=make.__name__, write=name): + path = self.dir / f"{make.__name__}-{name}" + store = make(path) + store.push(self.envelope(1)) + if name == "push": + reported = store.push(self.envelope(2)) + elif name == "push_all": + reported = store.push_all([self.envelope(2)])["k"] + else: + reported = store.commit_if(self.envelope(2), store.read("k")[1]) + self.assertTrue(self.later_than_alpha(reported)) + self.assertEqual(self.stamp_on_disk(store), reported) + + def test_the_cache_holds_the_stamp_the_store_wrote(self) -> None: + for name in ("put", "put_envelope", "put_envelopes", "update"): + with self.subTest(write=name): + path = self.dir / name + SingleFileMessagePackBackingStore(path).push(self.envelope(1)) + cache = self.cache(path=path) + self.assertEqual(cache.get_envelope(item_doc, "k").stored_at, ALPHA) + if name == "put": + cache.put(item_doc, "k", Item("n", 2)) + elif name == "put_envelope": + cache.put_envelope(item_doc, self.envelope(2)) + elif name == "put_envelopes": + cache.put_envelopes(item_doc, [self.envelope(2)]) + else: + cache.update(item_doc, "k", lambda item: Item("n", 2)) + held = cache.get_envelope(item_doc, "k").stored_at + self.assertTrue(self.later_than_alpha(held)) + self.assertEqual(held, self.stamp_on_disk(SingleFileMessagePackBackingStore(path))) + + def test_a_store_less_cache_stamps_a_replacing_write_later(self) -> None: + cache = self.cache(store=False) + cache.put_envelope(item_doc, self.envelope(1)) + cache.put_envelope(item_doc, self.envelope(2, "2000-01-01T00:00:00Z")) + self.assertTrue(self.later_than_alpha(cache.get_envelope(item_doc, "k").stored_at)) + + def test_only_the_stamp_shape_the_runtimes_write_parses(self) -> None: + """An RFC 3339 date-time with T, a fraction of 1 to 9 digits and Z or a numeric offset, on every interpreter (the shape is + checked here, not by fromisoformat), so a stamp reads the same on 3.10 and 3.13. The edges follow RFC 3339: offset hours 00-23, + minutes 00-59.""" + utc = timezone.utc + accepted = { + "2026-01-01T00:00:00Z": datetime(2026, 1, 1, tzinfo=utc), + "2026-01-01T00:00:00.1Z": datetime(2026, 1, 1, 0, 0, 0, 100000, utc), + "2026-01-01T00:00:00.123456789Z": datetime(2026, 1, 1, 0, 0, 0, 123456, utc), + "2026-01-01T00:00:00.1234567+00:00": datetime(2026, 1, 1, 0, 0, 0, 123456, utc), + "2026-01-01T05:30:00+05:30": datetime(2026, 1, 1, tzinfo=utc), + "2025-12-31T19:00:00-05:00": datetime(2026, 1, 1, tzinfo=utc), + "2026-01-01T23:59:00+23:59": datetime(2026, 1, 1, 0, 0, tzinfo=utc), + } + for text, instant in accepted.items(): + with self.subTest(text=text): + self.assertEqual(parse_stored_at(text), instant) + for text in ( + "2026-01-01", "20260101T000000Z", "2026-W01-1T00:00:00Z", "2026-01-01 00:00:00Z", "2026-01-01T00:00:00,5Z", + "2026-01-01T00Z", "2026-01-01T00:00:00", "2026-01-01T00:00:00.Z", "2026-01-01T00:00:00.1234567890Z", + "2026-01-01T00:00:00z", " 2026-01-01T00:00:00Z", "2026-01-01T00:00:00Z ", "2026-01-01T24:00:00Z", + "2026-13-01T00:00:00Z", "2026-01-01T00:00:00+24:00", "2026-01-01T00:00:00+05:60", "2026-01-01T00:00:00+05:75", "2026-01-01T00:00Z", "2026-01-01T00:00+00:00", + "026-01-01T00:00:00Z", "12026-01-01T00:00:00Z", "2026-01-01T00:00:00+5:00", "2026-01-01T00:00:00+0000", "\u0662026-01-01T00:00:00Z", + "", "garbage", + ): + with self.subTest(text=text): + self.assertIsNone(parse_stored_at(text)) + + def test_a_stamp_outside_the_range_is_unparseable_and_one_at_the_edge_refuses_a_later_stamp(self) -> None: + for text in ("9999-12-31T23:30:00-05:00", "0001-01-01T00:00:00+05:00"): + with self.subTest(text=text): + self.assertIsNone(parse_stored_at(text)) + now = datetime(2030, 1, 1, tzinfo=timezone.utc) + self.assertEqual(mint_stored_at("9999-12-31T23:30:00-05:00", now), "2030-01-01T00:00:00.000000+00:00") + edge = "9999-12-31T23:59:59.9999999Z" + with self.assertRaisesRegex(ValueError, "last instant"): + mint_stored_at(edge, now) + with self.assertRaisesRegex(ValueError, "last instant"): + stored_at_over("2000-01-01T00:00:00Z", edge) + # A write over a record stamped at the edge is refused by the store and writes nothing, from push, commit_if and update. + store = SingleFileMessagePackBackingStore(self.path) + entry = item_doc.catalog_entry() + record = lambda value, at: CultCacheEnvelope("k", "item", item_doc.encode_payload(Item("n", value)), at, entry.schema_id, entry) # noqa: E731 + store.push(record(1, edge)) + before = self.bytes() + version = store.read("k")[1] + for write in (lambda: store.push(record(2, ALPHA)), lambda: store.push_all([record(2, ALPHA)]), lambda: store.commit_if(record(2, ALPHA), version)): + with self.assertRaises(ValueError): + write() + cache = self.cache() + apply = Counting() + with self.assertRaises(ValueError): + cache.update(item_doc, "k", apply) + self.assertEqual(apply.calls, 1) + self.assertEqual(self.bytes(), before) + self.assertEqual(cache.get_envelope(item_doc, "k").stored_at, edge) + + def test_stamp_parsing_and_install(self) -> None: + for text in ("2026-01-01T00:00:00.1234567+00:00", "2026-01-01T00:00:00Z", "2026-01-01T00:00:00.123456+00:00"): + with self.subTest(text=text): + self.assertIsNotNone(parse_stored_at(text)) + self.assertEqual(parse_stored_at(text).tzinfo, timezone.utc) + self.assertIsNone(parse_stored_at("garbage")) + self.assertIsNone(parse_stored_at(None)) + now = datetime(2030, 1, 1, tzinfo=timezone.utc) + self.assertEqual(mint_stored_at("garbage", now), "2030-01-01T00:00:00.000000+00:00") + self.assertEqual(stored_at_over("2000-01-01T00:00:00Z", "2000-01-01T00:00:00Z")[:4], str(datetime.now(timezone.utc).year)) # no later: minted at now + self.assertEqual(mint_stored_at("2030-01-01T00:00:00Z", now), "2030-01-01T00:00:00.000001+00:00") # equal is not later + self.assertEqual(mint_stored_at("2099-01-01T00:00:00.1234567Z", now), "2099-01-01T00:00:00.123457+00:00") + self.assertEqual(stored_at_over("2000-01-01T00:00:00Z", "2099-01-01T00:00:00.1234567Z"), "2099-01-01T00:00:00.123457+00:00") + + for reported in (True, 1, "garbage"): + with self.subTest(reported=reported): + class Lying(SingleFileMessagePackBackingStore): + def push(self, envelope): + super().push(envelope) + return reported + + cache = CultCache.builder().register_document_type(item_doc).add_generic_store(Lying(self.dir / f"lying-{reported}")).build() + cache.pull_all_backing_stores() + cache.put_envelope(item_doc, self.envelope(1, "2026-02-02T00:00:00.000000+00:00")) + self.assertEqual(cache.get_envelope(item_doc, "k").stored_at, "2026-02-02T00:00:00.000000+00:00") + + +def stamp_of(store, key: str) -> str: + return store.read(key)[0].stored_at + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultcache-rs/.gitignore b/packages/cultcache-rs/.gitignore index 96ef6c0b..ea8c4bf7 100644 --- a/packages/cultcache-rs/.gitignore +++ b/packages/cultcache-rs/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/packages/cultcache-rs/CHANGELOG.md b/packages/cultcache-rs/CHANGELOG.md index ed7cd60a..23e217e8 100644 --- a/packages/cultcache-rs/CHANGELOG.md +++ b/packages/cultcache-rs/CHANGELOG.md @@ -6,8 +6,34 @@ only in git history. ## Unreleased +### Added + +- `CultCache::update_with_attempts` and the conditional `update`: each attempt reads the record from its home store with its content version in one read, applies a function to a copy and commits only over that version, rerunning on a lost race (default five attempts). The result is `CultUpdateResult`, outcome `Updated`, `Declined` (the function returned `None`) or `LostRace`, with the cache's document. Public types `CultStoredRecord`, `CultCommitIf`, `CultUpdateOutcome`, `CultUpdateResult`. +- `CacheBackingStore::read(key)` and `commit_if(entry, expected)`, provided methods whose defaults refuse with an error naming the store kind; `SingleFileMessagePackBackingStore` implements both. The content version is the lowercase hex SHA-256 of the record re-encoded as the store file writes it (key, schema id, `stored_at`, payload). It equals the SHA-256 of the record's bytes in the file, and so the C# reference's `tests/vectors/content-version`, whenever the file's MessagePack encodings are minimal, which every runtime's writer produces; a hand-edited file with a non-minimal encoding (for example a `str8` key that fits a fixstr) hashes differently from its slice here. New dependency `sha2` 0.10 and eight new lockfile crates: `sha2`, `digest`, `block-buffer`, `crypto-common`, `generic-array`, `typenum`, `cpufeatures`, `version_check`. + +### Changed + +- `is_stored_at_stamp(&str)` (new, public) is the one shape a peer's `storedAt` must parse as, pinned by `tests/vectors/stored-at-stamp` (RFC 3339 `YYYY-MM-DDThh:mm:ss`, a 1 to 9 digit fraction, `Z` or `+hh:mm`/`-hh:mm`; uppercase `T`/`Z`, a real date, seconds 00-59, years 0001-9999), hand-checked because `chrono`'s RFC 3339 reader takes other shapes. `put_raw_envelope` and `put_prepared_batch` refuse a `stored_at` that is not a stamp (before: only a blank one) and mint nothing (variants:ruling:raw-put-without-stamp-refused). +- The single-file store's `push`, `push_all` and `commit_if` stamp a replacing write strictly later than the record it replaces at the same key (a staged stamp stands only when it parses and is strictly later; otherwise the stamp is minted as the replaced stamp plus one microsecond, or now if later), with no identical-content exception, as C# does. A stamp with no successor (the replaced stamp is at year 9999's last microsecond) refuses the write and writes nothing, as C# `MintStoredAt` does. `commit_if` replaces only the record of the entry's own type under the key, so a retype is refused as `push` refuses it. The cache holds the stamp the store wrote. The redb stores and the store-level exact-envelope writes (`compare_and_swap_entry` and its family) still write the caller's envelope unchanged. +- The single-file store re-checks for a link once the store lock is held (every write and the try-exchange), so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, as C# does, instead of being replaced. Pinned by a test that puts the link while a second holder owns the lock. +- `CultCache::is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store. + ### Breaking +Semver: this is a breaking change in the 0.x line (0.3.x to 0.4.0). + +- `CultCache::update` is replaced, not extended: `update::(key, apply)` takes `FnMut(Option) -> Option` and returns `Result>`. The blind `FnOnce(Option) -> T` form that returned `T` is gone; use `put` for a blind write. It reads the home store, not the cache's image, so a cache whose store has no `read` and `commit_if` (the redb stores, a custom store) refuses it before `apply` runs. +- `CacheBackingStore::push` returns `Result`, the `stored_at` written, and `push_all` returns `Result>`, one stamp per entry given in order (a soft write that wrote nothing returns an empty vec). A custom store must return what it wrote; the redb stores return the entries' own stamps. Huginn implements `CacheBackingStore` in five test stores under a pinned git rev: they change at its next bump. +- A write at a store path that is a symbolic link, live or dangling, is refused with the new public `StoreLinkedError` (`error.downcast_ref::()`) before the lock and the replace, and writes nothing: `push`, `push_all`, `delete`, the exchanges, and a soft `push_all`, which used to skip a link and now refuses. Before, a write at a live link replaced the link with a regular file and a dangling link failed with `io::ErrorKind::NotFound`. The redb stores (`RedbMessagePackBackingStore`, `OwnedRedbMessagePackBackingStore`) are exempt: they are ruled migrated and deleted (`paged-store-redb-migrate-and-delete`) and still open a live link's target. Configure the link's target as the store path. Reads through a live link work. +- A record whose payload carries unread slots is read-only. A payload with a slot past its type's declared slots, at any depth (record, nested record, list element, union arm, unit arm), still decodes, and every write or removal of its key is now refused before any store is touched: `put`, `put_envelope`, `put_raw_envelope`, `put_prepared_batch` (the whole batch) and `delete`, so `update` too. The refusal is `SchemaConflictError` with the new `unread_slots`; its message names the type and both slot counts and never a slot value. Holders of such a record used to overwrite it and drop the extra slots silently. The payload nesting refusal still runs first. A held payload this build cannot read at all (an arm it lacks from a newer writer, or a corrupt record) is read-only too: every write or removal is refused with an error naming the type and key (a newer writer's record is read with that build; a corrupt one is restored from a backup or removed at the store itself, as no cache call removes it), and `is_writable` answers false. `load_soa` is a write of every held key it omits or changes, so it refuses to drop or replace a read-only record. `is_writable` also answers false for a key a store carries as a foreign record and for a type other than the one holding the key, as the writes do. Each write over a held key decodes the held payload once to ask (about 40% of a `put` of a 200,000-element record, measured in release); keys nothing holds pay nothing. +- cultmesh-rs `pull_rudp_catalog_snapshot` applies a snapshot as one cache commit (`put_prepared_batch`), as C# does: a snapshot document for a key the node holds read-only, or any other refusal, applies nothing. It used to apply document by document, so a refusal ended the pull with the earlier documents kept. A snapshot whose types route to different backing stores is now refused whole, as a batch cannot span stores. C# `CultCache.Commit` refuses a cross-store commit the same way (`The batch spans A and B; a commit lands in one home store`), so the two agree; a test pins it. +- `SchemaConflictError` gains the public field `unread_slots: Option`; code that builds it with a struct literal must set it (`None`). +- A unit arm with slots (`[N, [extra...]]`) decodes as the unit arm and marks the record, where the derive refused it. +- The registration map's value changes from the schema name to a private struct holding the schema name and the unread-slots check. +- A cache whose last `pull_all_backing_stores` was refused (a tail, a payload past the nesting bound, a duplicate unique-index value, a retype conflict, a record from the wrong store) refuses every `put`, `put_envelope`, `put_raw_envelope`, `put_prepared_batch`, `update` and `delete` with `CultCacheLoadRefused` until a pull succeeds. Before, it stayed writable over an empty image, and the next batch write replaced the store with only the new records. +- `CacheBackingStore::push_all` writes the given entries onto the store as it is when the write lands, under the store's lock or transaction: a record under the same identity is replaced and no other record is removed. It used to replace everything the store held, so a cache whose image was stale (another writer had added a record since its pull) wiped that record with `put_prepared_batch`. `put_prepared_batch` now hands the store its staged records alone, as C# `CommitBatch` and Python `push_all` do; the lay-back of foreign records is gone because they are never rewritten. A custom `CacheBackingStore` must give `push_all` merge semantics. `push_all` reads the store, so through a dangling link it is refused like `push` and no longer replaces the link; a soft flush is unchanged. +- A cache with a backing store also refuses every write and removal, with `CultCacheLoadRefused`, until it has pulled: a cache that never pulled, or that gained a store since its last pull, holds no image of its stores, and a batch from it replaced a populated store with only the new records. `is_writable` answers false meanwhile. A cache with no store needs no pull. Callers that attach a store and write (cultmesh nodes created with `pull_on_start: false`) must call `pull_all_backing_stores` first. +- `decode_payload`, `put_raw_envelope` and prepared batches refuse a payload with bytes after its first MessagePack value (`payload has {n} bytes after its first MessagePack value`), by count; the tail is never walked or quoted. They used to read the first value and keep or drop the tail, so a record with a stray tail was served and rewritten without it. A store holding such a record (single-file, redb, `load_soa`) is refused at load, naming the type, the key and the count, and stays unreadable until the record is repaired; the same check now runs on every record at load, as the C# and Python runtimes do at open. The nesting bound is part of that check, so a stored payload nested past it is refused at load too, where it used to load and refuse on read. - A single-file store written as `cultcache.store.v3` is read and written: a store the C# runtime marked v3 (it holds a record with an element id: an owned list element with a `[CultElementId]` member) keeps its header here, and is carried through every rewrite with its ids; this runtime never drops the mark. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - A write over a file the reader refuses (a zero-byte file, a truncated store, a stamp this runtime does not read) refuses with `CultCacheStoreUnreadable` and leaves the file as it is. A whole-store write used to replace such a file. This reverses 0.3.1, where `push_all` replaced a file with no readable stamp (healing it); the stamp rule now has one owner, the reader every write asks under the lock, and it accepts v1 and v3 (0.3.1 refused v3). - `#[cultcache(index)]` means unique, as in C# `RefuseIndexSharing`: within one type and one alias no two records hold one non-blank value. A put, batch, push or load that would make two records hold a value is refused, naming both keys and the alias, and a store that already holds a duplicate does not open. A grouping index is no longer expressible; a consumer that relied on shared values must drop the `index` and look up by another way. @@ -22,6 +48,7 @@ only in git history. ### Added +- `UnreadSlots { type_name, slot_count, declared_count, occurrences }`, `decode_payload_marked` (the decode that returns the mark) and `CultCache::is_writable(type, key)`. The derives' slot visitors note into a thread-local scope the decode entry opens (`__enter_slots`, `__note_unread_slots`, `__UnitArmSlots`, doc-hidden); a decode outside that entry notes nothing. - `CultCache::foreign_records()` and `ForeignRecord { key, schema_id, schema_name, stored_at }`: the records the last pull found that no registered type claims, sorted by key. - Lookups, matching the C# `GetByName`, `GetByIndex` and `GetGlobal`. `#[cultcache(name)]` and `#[cultcache(index)]` / `#[cultcache(index = "alias")]` on `DatabaseEntry` fields and `#[cultcache(global)]` on the struct give `CultCache::get_by_name`, `get_by_index` and `get_global`. An empty name and a blank index value are not found; records of one type sharing a name find the one with the smallest key (an index value is unique, below). A second record of a global type under another key is refused on every write path and on load with `CultCacheSecondGlobal`. Two `name` members, or two members with one index alias, are refused at compile time. A member's text comes from `CultLookupValue` (`String`, `Option` of one, integers, `bool`, `char`). - `SchemaConflictError`, the typed refusal of a write the catalog cannot describe: records of different types under one schema id, or an arrived schema and a registered one that share an id and disagree on the schema name. A whole-store write (`push_all`) refuses with it, names the schema id, the names involved and a record key, writes nothing and leaves the file as it is. A caller tells it apart with `error.downcast_ref::()`. diff --git a/packages/cultcache-rs/Cargo.lock b/packages/cultcache-rs/Cargo.lock new file mode 100644 index 00000000..6fad548a --- /dev/null +++ b/packages/cultcache-rs/Cargo.lock @@ -0,0 +1,802 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "android_system_properties" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" +dependencies = [ + "libc", +] + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "ar_archive_writer" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "523e5e187fff36fcb8d1d19ccf0ca4ca43ec3d158316057f66dc6f72716bc5ec" +dependencies = [ + "object", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bitflags" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "cc" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ff2a8126fff0f8c839080d619675a585e61720bb12d2baceae45769fd66f155" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "cultcache-rs" +version = "0.3.1" +dependencies = [ + "anyhow", + "chrono", + "cultcache-rs-derive", + "fs2", + "pretty_assertions", + "redb", + "rmp-serde", + "serde", + "serde_bytes", + "serde_json", + "serde_stacker", + "sha2", + "stacker", + "tempfile", + "uuid", + "windows-sys 0.59.0", +] + +[[package]] +name = "cultcache-rs-derive" +version = "0.2.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "diff" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8" + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", +] + +[[package]] +name = "errno" +version = "0.3.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "find-msvc-tools" +version = "0.1.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48" + +[[package]] +name = "fs2" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "futures-core" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" + +[[package]] +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "itoa" +version = "1.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4" + +[[package]] +name = "js-sys" +version = "0.3.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.191" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "log" +version = "0.4.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d15f86d12da44cd8d4ec3c73984537eb7f4cadcb9776a575a0a8a05ca11005c" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "object" +version = "0.39.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" +dependencies = [ + "memchr", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pretty_assertions" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d" +dependencies = [ + "diff", + "yansi", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "psm" +version = "0.1.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622" +dependencies = [ + "ar_archive_writer", + "cc", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "redb" +version = "4.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b317ef681e18f6091efbfab532eaf924a135c492d9d0299d0fbfae07e92a39b" +dependencies = [ + "libc", +] + +[[package]] +name = "rmp" +version = "0.8.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "rmp-serde" +version = "1.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155" +dependencies = [ + "rmp", + "serde", +] + +[[package]] +name = "rustix" +version = "1.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_bytes" +version = "0.11.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.10", +] + +[[package]] +name = "serde_json" +version = "1.0.154" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_stacker" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4936375d50c4be7eff22293a9344f8e46f323ed2b3c243e52f89138d9bb0f4a" +dependencies = [ + "serde", + "serde_core", + "stacker", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "stacker" +version = "0.1.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967" +dependencies = [ + "cc", + "cfg-if", + "libc", + "psm", + "windows-sys 0.61.2", +] + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ae9ccbea3a7b0b09443036490c8f34a8b1b760fd345e5c0dae0aac718faea79" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" + +[[package]] +name = "uuid" +version = "1.29.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bed99139f7dac9b5cfe5f17be4d66dc7079a34dee1d74bcc77217119240e267c" +dependencies = [ + "getrandom", + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasm-bindgen" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c" +dependencies = [ + "cfg-if", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 3.0.10", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.59.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" +dependencies = [ + "windows-targets", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "yansi" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/packages/cultcache-rs/Cargo.toml b/packages/cultcache-rs/Cargo.toml index 7954370c..1ed1ba3f 100644 --- a/packages/cultcache-rs/Cargo.toml +++ b/packages/cultcache-rs/Cargo.toml @@ -23,6 +23,7 @@ serde = { version = "1", features = ["derive"] } serde_bytes = "0.11" serde_json = "1" serde_stacker = "0.1.14" +sha2 = "0.10" stacker = "0.1.25" uuid = { version = "1", features = ["v4"] } @@ -31,4 +32,4 @@ pretty_assertions = "1.4" tempfile = "3" [target.'cfg(windows)'.dependencies] -windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem"] } +windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_Security", "Win32_System_WindowsProgramming"] } diff --git a/packages/cultcache-rs/README.md b/packages/cultcache-rs/README.md index 8a8b8f5e..d52fdea2 100644 --- a/packages/cultcache-rs/README.md +++ b/packages/cultcache-rs/README.md @@ -39,8 +39,8 @@ CultCache is a domain cache with persistence adapters. file access with a sidecar lock file. This is not an ORM, not a database, and not distributed consensus in a novelty -hat. If multiple processes write the same backing file, use an external lock or -a coordinator. +hat. Several processes may write the same single-file store, one at a time under +its sidecar lock; write ordering across keys or stores needs a coordinator. ## Current API @@ -186,7 +186,7 @@ without pretending Rust has runtime reflection hiding under the floorboards. - `soa::` - `put::` - `put_envelope::` -- `update::` +- `update::` and `update_with_attempts::`: the conditional golden-path write (read the stored record, apply, commit only over what was read, rerun on a lost race) - `delete::` - `snapshot` - `snapshot_soa` @@ -236,6 +236,17 @@ assert_eq!(rmp_serde::from_slice::(&bytes)?, tree); # Ok::<(), Box>(()) ``` +## Records written by a newer build + +A payload that carries a slot past the slots its type declares, in a record, a nested record, a list element or a union +arm (a unit arm declares none), still decodes: the extra slots are skipped. The cache then holds that record read-only, +because writing it would drop what the newer writer stored. `put`, `put_envelope`, `put_raw_envelope`, +`put_prepared_batch` and `delete` of its key are refused with `SchemaConflictError`, whose `unread_slots` names the type +and the slot counts and never a value; `CultCache::is_writable(type, key)` answers the same check. An untouched +read-only record keeps its bytes through every other write. A short payload (missing trailing slots) is not marked, and +a payload nested past `MAX_PAYLOAD_DEPTH` is refused before anything is read. `decode_payload_marked` is the decode that +returns the `UnreadSlots` mark. This runtime has no compatible schema ids, so no mark is consented. + ## Structure Of Arrays The C# CultCache hot path exposes CPU-local SoA scans as: @@ -293,6 +304,14 @@ The contract, shared with every runtime, is one home store per document type ## Persistence Semantics +`update` is the golden path for changing a record another writer may also change: each attempt reads the record and its content +version from the home store, hands `apply` a copy, and commits only if the store still holds that version. `put` is the blind +write. The single-file store stamps a replacing write (`push`, `push_all` and `update`'s commit) strictly later than the record +it replaces, and the cache holds the stamp the store reports; the redb stores and the exact-envelope family +(`compare_and_swap_entry` and the other store-level conditional writes) write the caller's envelope byte for byte. A store +needs `read` and `commit_if` to serve `update`; the redb stores do not have them. +The version is the SHA-256 of the record re-encoded as the store file writes it; that equals the hash of the record's bytes in the file (and the C# reference's vector) whenever the file's MessagePack encodings are minimal, as every runtime's writer produces. + `put` persists before mutating the in-memory cache. If persistence fails, the cache does not pretend the write succeeded. @@ -373,10 +392,10 @@ an empty store. Any other failure to reach it, such as a symlink loop, a file where a parent directory should be, or a missing permission, is an I/O error. A store file that exists and holds zero bytes is `Undecodable`: no CultCache writer leaves one. A symbolic link whose target is gone is not a missing store: -reading it is an I/O error, so a read-then-write (`push`, `delete`, the exchanges) -fails before writing. Writes do not yet follow links: a write through a live link, -and a `push_all` through a dangling one, replace the link with a regular file and -leave its target alone. A refusal names no +reading it is an I/O error. A write at a store path that is a link, live or dangling, is +refused with `StoreLinkedError` (`error.downcast_ref::()`) before the +lock and the replace, so no write swaps a link for a regular file; configure the link's +target as the store path. Reads through a live link work. A refusal names no value read from the file; it names a record only by its key and schema id, and a header only when it has the shape `cultcache.store.v`. diff --git a/packages/cultcache-rs/cultcache-rs-derive/src/lib.rs b/packages/cultcache-rs/cultcache-rs-derive/src/lib.rs index c7f9a3a2..b426ae01 100644 --- a/packages/cultcache-rs/cultcache-rs-derive/src/lib.rs +++ b/packages/cultcache-rs/cultcache-rs-derive/src/lib.rs @@ -381,6 +381,7 @@ fn slot_codec(ident: &syn::Ident, slots: &[CultCacheSlot]) -> proc_macro2::Token where __A: ::serde::de::SeqAccess<'de>, { + let __ticket = ::cultcache_rs::__enter_slots(); #(#option_declarations)* let mut index = 0usize; while index <= #max_slot { @@ -390,7 +391,15 @@ fn slot_codec(ident: &syn::Ident, slots: &[CultCacheSlot]) -> proc_macro2::Token } index += 1; } - while let ::std::option::Option::Some(_) = seq.next_element::<::serde::de::IgnoredAny>()? {} + // The slots past the declared ones are skipped and counted: the decode entry marks the record + // that carried them read-only. + let mut __extra = 0usize; + while seq.next_element::<::serde::de::IgnoredAny>()?.is_some() { + __extra += 1; + } + if __extra > 0 { + ::cultcache_rs::__note_unread_slots(__ticket, stringify!(#ident), #max_slot + 1 + __extra, #max_slot + 1); + } Ok(#ident { #(#build_fields),* }) @@ -567,10 +576,13 @@ fn union_codec(input: DeriveInput) -> syn::Result { quote! { #key => { match seq - .next_element::<::std::option::Option<[(); 0]>>() + .next_element::<::std::option::Option<::cultcache_rs::__UnitArmSlots>>() .map_err(|_| ::serde::de::Error::custom(#arity_message))? { - ::std::option::Option::Some(::std::option::Option::Some(_)) => #ident::#variant, + ::std::option::Option::Some(::std::option::Option::Some(slots)) => { + slots.note(stringify!(#variant)); + #ident::#variant + } ::std::option::Option::Some(::std::option::Option::None) => { return ::std::result::Result::Err(::serde::de::Error::custom( format!(#nil_slots_message, #key), diff --git a/packages/cultcache-rs/examples/cultcache_interop.rs b/packages/cultcache-rs/examples/cultcache_interop.rs index 8efb9395..dcafee5f 100644 --- a/packages/cultcache-rs/examples/cultcache_interop.rs +++ b/packages/cultcache-rs/examples/cultcache_interop.rs @@ -1,6 +1,8 @@ use anyhow::Result; use anyhow::anyhow; use cultcache_rs::CultCache; +use cultcache_rs::CultSlots; +use cultcache_rs::CultUnion; use cultcache_rs::DatabaseEntry; use cultcache_rs::SingleFileMessagePackBackingStore; use serde_json::json; @@ -39,6 +41,100 @@ struct CultCacheInteropDeck { marks: Vec, } +// A union field and nested bytes: the shapes a Bifrost-style store holds beyond plain strings. +#[derive(Clone, Debug, PartialEq, Eq, CultUnion)] +enum InteropArm { + #[cultcache(key = 0)] + Text(InteropTextArm), + #[cultcache(key = 1)] + Pack(InteropPackArm), +} + +#[derive(Clone, Debug, PartialEq, Eq, CultSlots)] +struct InteropTextArm { + #[cultcache(key = 0)] + text: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, CultSlots)] +struct InteropPackArm { + #[cultcache(key = 0)] + label: String, + #[cultcache(key = 1)] + chunks: Vec, +} + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "cultcache.interop-shape", schema = "CultCacheInteropShape")] +struct CultCacheInteropShape { + #[cultcache(key = 0)] + document_id: String, + #[cultcache(key = 1)] + arm: InteropArm, + #[cultcache(key = 2)] + raw: serde_bytes::ByteBuf, +} + +fn shapes() -> Vec { + vec![ + CultCacheInteropShape { + document_id: "shape:text".into(), + arm: InteropArm::Text(InteropTextArm { text: "union arm 0".into() }), + raw: (0..=255u8).collect::>().into(), + }, + CultCacheInteropShape { + document_id: "shape:pack".into(), + arm: InteropArm::Pack(InteropPackArm { + label: "union arm 1".into(), + chunks: vec![vec![0xc0, 0xc1, 0x00].into(), vec![].into(), vec![0x92, 0x01, 0x02].into()], + }), + raw: vec![0xde, 0xad, 0xbe, 0xef].into(), + }, + ] +} + +fn write_shapes(file: &str) -> Result<()> { + let mut cache = CultCache::new(); + cache.register_entry_type::()?; + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(file))?; + cache.pull_all_backing_stores()?; + for shape in shapes() { + cache.put(&shape.document_id, &shape)?; + } + print_shapes(shapes()) +} + +/// Opens the store through the read-only snapshot door: no lock file, no staging file, no rewrite. +fn read_shapes(file: &str) -> Result<()> { + let envelopes = SingleFileMessagePackBackingStore::new(file).pull_all_read_only_snapshot()?; + let mut found = Vec::new(); + for envelope in envelopes.iter().filter(|envelope| envelope.r#type == CultCacheInteropShape::TYPE) { + found.push(cultcache_rs::decode_payload::(&envelope.payload)?); + } + print_shapes(found) +} + +fn print_shapes(mut found: Vec) -> Result<()> { + let hex = |bytes: &[u8]| bytes.iter().map(|byte| format!("{byte:02X}")).collect::(); + found.sort_by(|a, b| a.document_id.cmp(&b.document_id)); + let rows: Vec<_> = found + .iter() + .map(|shape| { + let arm = match &shape.arm { + InteropArm::Text(text) => json!({ "kind": "text", "text": text.text }), + InteropArm::Pack(pack) => json!({ + "kind": "pack", + "label": pack.label, + "chunks": pack.chunks.iter().map(|chunk| hex(chunk)).collect::>(), + }), + }; + json!({ "documentId": shape.document_id, "arm": arm, "raw": hex(&shape.raw) }) + }) + .collect(); + println!("{}", serde_json::to_string(&rows)?); + Ok(()) +} + fn read_deck(file: &str) -> Result<()> { let mut cache = CultCache::new(); cache.register_entry_type::()?; @@ -70,6 +166,8 @@ fn main() -> Result<()> { "write" => write_note(file, require_arg(&options, "runtime-id")?)?, "read" => read_note(file)?, "read-deck" => read_deck(file)?, + "write-shapes" => write_shapes(file)?, + "read-shapes" => read_shapes(file)?, "hold-lock" => hold_lock(file, require_arg(&options, "ms")?.parse()?)?, _ => return Err(anyhow!("unknown mode {mode}")), } diff --git a/packages/cultcache-rs/src/lib.rs b/packages/cultcache-rs/src/lib.rs index 686547f4..0e60a0c2 100644 --- a/packages/cultcache-rs/src/lib.rs +++ b/packages/cultcache-rs/src/lib.rs @@ -480,6 +480,39 @@ struct PersistedSchemaCatalogEntry( Vec, ); +/// One entry of a store file's schema catalog, read-only: the scalar fields every runtime publishes for a schema id. The members and +/// the canonical schema JSON the file also carries are not exposed. `schema_version` is the version string the writer published +/// (C# `CultSchemaCatalogEntry.SchemaVersion`). +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SchemaCatalogEntry { + pub schema_id: String, + pub schema_name: String, + pub schema_version: String, + pub content_hash: String, + pub compatible_schema_ids: Vec, +} + +impl From<&PersistedSchemaCatalogEntry> for SchemaCatalogEntry { + fn from(entry: &PersistedSchemaCatalogEntry) -> Self { + Self { + schema_id: entry.0.clone(), + schema_name: entry.1.clone(), + schema_version: entry.2.clone(), + content_hash: entry.3.clone(), + compatible_schema_ids: entry.5.clone(), + } + } +} + +/// The catalog entry a record's schema id names, as the decoder resolves it: the first entry that has the id as its own, else the +/// first entry that lists it as compatible; `None` when the catalog publishes neither. +pub fn resolve_schema_entry<'a>(catalog: &'a [SchemaCatalogEntry], schema_id: &str) -> Option<&'a SchemaCatalogEntry> { + catalog + .iter() + .find(|entry| entry.schema_id == schema_id) + .or_else(|| catalog.iter().find(|entry| entry.compatible_schema_ids.iter().any(|id| id == schema_id))) +} + #[derive(Clone, Debug, serde::Serialize, serde::Deserialize)] struct PersistedSchemaCatalogMember( u32, @@ -600,14 +633,22 @@ impl std::error::Error for PayloadTooDeep {} /// Walks the MessagePack headers of one payload without recursion, counting every array and map and /// refusing the first container past [`MAX_PAYLOAD_DEPTH`]. Scalars are skipped by length and nothing -/// is allocated per element. A payload that ends early is the decoder's to refuse. +/// is allocated per element. Only the first value is walked; a payload that ends early is the +/// decoder's to refuse. A depth-only probe: [`decode_payload`] also refuses bytes after the value. pub fn check_payload_depth(bytes: &[u8]) -> std::result::Result<(), PayloadTooDeep> { + first_value_end(bytes).map(|_| ()) +} + +/// Where the first MessagePack value of `bytes` ends, walking only that value (its tail is never +/// read). `None` when the payload ends inside a header; a final scalar cut short ends past the +/// payload's length, which is not bytes after the value either. Both are the decoder's to refuse. +fn first_value_end(bytes: &[u8]) -> std::result::Result, PayloadTooDeep> { let mut remaining = [0usize; MAX_PAYLOAD_DEPTH]; let mut depth = 0usize; let mut position = 0usize; while position < bytes.len() { let Some((advance, children)) = element_header(bytes, position) else { - return Ok(()); + return Ok(None); }; position = position.saturating_add(advance); if let Some(children) = children { @@ -629,7 +670,18 @@ pub fn check_payload_depth(bytes: &[u8]) -> std::result::Result<(), PayloadTooDe depth -= 1; } if depth == 0 { - return Ok(()); + return Ok(Some(position)); + } + } + Ok(None) +} + +/// The one rule for "is this payload one value": the nesting bound, then no bytes after the first +/// value. The count is all it reports; the tail is never walked or quoted. +fn check_record_payload(bytes: &[u8]) -> Result<()> { + if let Some(end) = first_value_end(bytes)? { + if end < bytes.len() { + bail!("payload has {} bytes after its first MessagePack value", bytes.len() - end); } } Ok(()) @@ -784,7 +836,7 @@ fn decode_grown(bytes: &[u8]) -> std::result::Re grown(&mut reader, |grown| T::deserialize(grown)) } -/// The one decode of a document payload: the nesting bound first, then [`decode_grown`], whose +/// The one decode of a document payload: the payload check first (nesting bound, one value), then [`decode_grown`], whose /// rmp-serde depth counter (1024) is only a backstop. A failure keeps its kind and never quotes the /// bytes. Readers of stored envelope payloads outside this crate decode through it too. /// @@ -796,14 +848,163 @@ fn decode_grown(bytes: &[u8]) -> std::result::Re /// verified on x86_64-pc-windows-msvc by hand (`decode_stack`, revision 10a39dac, 13 tests, debug and release); a /// `stacker` bump or a change to the stack-size tests repeats it. pub fn decode_payload(payload: &[u8]) -> Result { - check_payload_depth(payload)?; + check_record_payload(payload)?; decode_grown(payload).map_err(opaque_decode_error) } -/// The nesting bound for an envelope admitted without a typed decode, from the same walk that -/// [`decode_payload`] runs first. Names the entry's type and key, never the payload. -fn require_payload_depth(entry: &CultCacheEnvelope) -> Result<()> { - check_payload_depth(&entry.payload).with_context(|| { +/// What a payload decode found past the slots this build declares, at any depth: the first type read with extra slots, +/// how many slots it carried against how many it declares, and how many objects in the payload did. It holds no slot +/// value, so a refusal can name it. A record whose payload carries it is read-only in the cache +/// ([`CultCache::is_writable`]); this runtime has no compatible schema ids, so every mark is unconsented. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct UnreadSlots { + /// The first type read with extra slots (a unit arm is named by its variant). + pub type_name: String, + /// The slots that object carried. + pub slot_count: usize, + /// The slots this build declares for it. + pub declared_count: usize, + /// The objects in the payload that carried extra slots. + pub occurrences: usize, +} + +/// The open mark scope of a decode: how many slot objects it has entered, and the mark of the first entered one that +/// carried extra slots (the reference names the first type read, which is the outermost: it is entered before its members). +#[derive(Default)] +struct MarkScope { + entered: usize, + mark: Option<(usize, UnreadSlots)>, +} + +thread_local! { + /// The open mark scope of a decode on this thread: `None` when no decode entry has one open, so a decode through + /// serde alone notes nothing. The slot visitors the derives write note into it. + static UNREAD_SLOTS: std::cell::RefCell> = const { std::cell::RefCell::new(None) }; +} + +/// The derives' sensor, first half: a slot object is entered. Its ticket orders it among the objects of the decode. +#[doc(hidden)] +pub fn __enter_slots() -> usize { + UNREAD_SLOTS.with(|scope| { + scope.borrow_mut().as_mut().map_or(0, |open| { + open.entered += 1; + open.entered + }) + }) +} + +/// The derives' sensor, second half: the object entered as `ticket`, of `type_name`, carried `carried` slots where this +/// build declares `declared`. +#[doc(hidden)] +pub fn __note_unread_slots(ticket: usize, type_name: &str, carried: usize, declared: usize) { + UNREAD_SLOTS.with(|scope| { + let mut scope = scope.borrow_mut(); + let Some(open) = scope.as_mut() else { + return; + }; + let occurrences = open.mark.as_ref().map_or(0, |(_, found)| found.occurrences) + 1; + if open.mark.as_ref().is_none_or(|(first, _)| ticket < *first) { + let named = UnreadSlots { type_name: type_name.to_string(), slot_count: carried, declared_count: declared, occurrences }; + open.mark = Some((ticket, named)); + } else if let Some((_, found)) = open.mark.as_mut() { + found.occurrences = occurrences; + } + }); +} + +/// A unit arm is a zero-slot record: its slots array is read and counted, not refused. +#[doc(hidden)] +pub struct __UnitArmSlots { + ticket: usize, + extra: usize, +} + +impl __UnitArmSlots { + /// Notes the arm if its array held any element. + pub fn note(self, variant: &str) { + if self.extra > 0 { + __note_unread_slots(self.ticket, variant, self.extra, 0); + } + } +} + +impl<'de> serde::Deserialize<'de> for __UnitArmSlots { + fn deserialize>(deserializer: D) -> std::result::Result { + struct Count; + impl<'de> serde::de::Visitor<'de> for Count { + type Value = __UnitArmSlots; + fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { + formatter.write_str("a slots array") + } + fn visit_seq>(self, mut seq: A) -> std::result::Result { + let ticket = __enter_slots(); + let mut extra = 0usize; + while seq.next_element::()?.is_some() { + extra += 1; + } + Ok(__UnitArmSlots { ticket, extra }) + } + } + deserializer.deserialize_seq(Count) + } +} + +/// The scope one marked decode opens on this thread; closed when it ends, whatever the decode did. A marked decode is +/// never re-entered (only the cache's own checks open one), so scopes do not nest. +struct UnreadScope; + +impl UnreadScope { + fn open() -> Self { + UNREAD_SLOTS.with(|scope| *scope.borrow_mut() = Some(MarkScope::default())); + Self + } + + fn mark(&self) -> Option { + UNREAD_SLOTS.with(|scope| scope.borrow_mut().as_mut().and_then(|open| open.mark.take()).map(|(_, mark)| mark)) + } +} + +impl Drop for UnreadScope { + fn drop(&mut self) { + UNREAD_SLOTS.with(|scope| *scope.borrow_mut() = None); + } +} + +/// [`decode_payload`] with the mark: the same payload check and grown decode, inside a scope the derives' slot +/// visitors note into. A payload past the nesting limit or with bytes after its value is refused before the +/// decode, never marked. +pub fn decode_payload_marked(payload: &[u8]) -> Result<(T, Option)> { + check_record_payload(payload)?; + let scope = UnreadScope::open(); + let value = decode_grown(payload).map_err(opaque_decode_error)?; + Ok((value, scope.mark())) +} + +/// The registered check of a held payload: monomorphized by [`CultCache::register_entry_type`] so the untyped write +/// entries can decode a payload they cannot name. +/// A payload the type cannot decode at all (an arm this build lacks, a corrupt record) is an error here, and the held +/// record is read-only: the cache cannot tell what a writer of another build stored in it. +fn unread_slots_of(payload: &[u8]) -> Result> { + decode_payload_marked::(payload).map(|(_, mark)| mark) +} + +/// What the cache finds in a held payload when asked to write or remove its key. +enum HeldFinding { + Free, + Marked(UnreadSlots), + Unreadable(anyhow::Error), +} + +/// What a registered type is to the cache: its schema name and how to read the slots a held payload carries. +struct Definition { + schema_name: &'static str, + unread_slots: fn(&[u8]) -> Result>, +} + +/// The payload check for an envelope admitted without a typed decode, the one [`decode_payload`] +/// runs first. Names the entry's type and key, never the payload. +fn require_record_payload(entry: &CultCacheEnvelope) -> Result<()> { + check_record_payload(&entry.payload).with_context(|| { format!( "refused CultCache envelope {:?} at key {:?}", entry.r#type, entry.key @@ -938,10 +1139,26 @@ pub struct SchemaConflictError { pub schema_id: String, pub schema_names: Vec, pub record_key: String, + /// Set when the refusal is a write or removal of a record held read-only for the slots its payload carries past + /// this build's; `None` for every other refusal. + pub unread_slots: Option, } impl std::fmt::Display for SchemaConflictError { fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + if let Some(unread) = &self.unread_slots { + return write!( + formatter, + "{} record {} is read-only: {} was read with {} slots where this build declares {} ({} object(s) in the record). \ + Writing it would drop data a newer writer stored; read it with types that declare those slots, or write it from that build.", + self.schema_names.first().map_or("", String::as_str), + self.record_key, + unread.type_name, + unread.slot_count, + unread.declared_count, + unread.occurrences + ); + } write!( formatter, "schema id {:?} cannot describe record {:?}: {:?}", @@ -992,6 +1209,35 @@ impl std::fmt::Display for CultCacheStoreUnreadable { impl std::error::Error for CultCacheStoreUnreadable {} +/// A write was asked of a store path that is a link (a file or directory symbolic link, or a Windows +/// junction): a replace there would swap the link for a regular file and detach the store from its +/// target. Nothing was locked or written. Configure the link's target as the store path. Reads through +/// a live link are unaffected. Find it with `error.downcast_ref::()`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct StoreLinkedError { + pub path: PathBuf, +} + +impl std::fmt::Display for StoreLinkedError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + formatter, + "CultCache store {} is a link, and a write at it would replace the link; configure the link's target as the store path", + self.path.display() + ) + } +} + +impl std::error::Error for StoreLinkedError {} + +/// Refuses a write at a link. A path that cannot be inspected is left to the open that follows. +fn refuse_linked(path: &Path) -> Result<()> { + match fs::symlink_metadata(path) { + Ok(metadata) if metadata.file_type().is_symlink() => Err(StoreLinkedError { path: path.to_path_buf() }.into()), + _ => Ok(()), + } +} + /// A write to a single-file store that failed, and what it left on disk. It sits in the /// error chain of every such failure, above the cause, so a consumer tells the outcomes /// apart with `error.downcast_ref::()`. @@ -1044,6 +1290,39 @@ pub struct PushAllOptions { pub soft: bool, } +/// A record read from a store with its content version, both from one read. The version is the lowercase hex SHA-256 of the +/// record as the store file encodes it (key, schema id, `stored_at`, payload), so a rewrite of the same bytes under a later +/// stamp is a different version. +#[derive(Clone, Debug, PartialEq)] +pub struct CultStoredRecord { + pub envelope: CultCacheEnvelope, + pub version: String, +} + +/// The verdict of [`CacheBackingStore::commit_if`]: the write landed and carries this stamp, or the record at the key was not at +/// the expected version and nothing was written. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum CultCommitIf { + Committed { stored_at: String }, + Mismatch, +} + +/// How [`CultCache::update`] ended. `Declined` wrote nothing because `apply` returned `None`; `LostRace` wrote nothing because +/// every attempt found the record changed under it. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CultUpdateOutcome { + Updated, + Declined, + LostRace, +} + +/// The outcome of an update and the cache's document at the key after it (a read that decides nothing). +#[derive(Clone, Debug, PartialEq)] +pub struct CultUpdateResult { + pub outcome: CultUpdateOutcome, + pub document: Option, +} + /// One per-entry expectation for [`SingleFileMessagePackBackingStore::compare_exchange`]. /// `current` is the envelope the caller believes is stored, or `None` when it /// believes the identity is absent. @@ -1066,15 +1345,21 @@ pub enum TryCompareExchangeSnapshotOutcome { pub trait CacheBackingStore: Send { fn pull_all(&self) -> Result>; - fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()>; + /// Writes `entry` and returns the `stored_at` the store wrote: a store that restamps a replacing write reports the stamp it + /// minted, and the cache holds that one. + fn push(&mut self, entry: &CultCacheEnvelope) -> Result; fn delete(&mut self, entry: &CultCacheEnvelope) -> Result<()>; - /// Replaces everything the store holds with `entries` in one atomic step: - /// on `Err` the store still holds exactly what it held before. A cache - /// batch (`put_prepared_batch`) is all-or-nothing only through this. There - /// is no default, because a replace assembled from `delete` and `push` - /// calls leaves the store emptied or partial when one of them fails; a - /// store without an atomic replace does not compile: + /// Writes `entries` onto the store as it is when the write lands, in one + /// atomic step: a record under the same identity is replaced and no other + /// record is removed or changed, so a writer whose image is stale cannot + /// drop a record it never saw. On `Err` the store still holds exactly what + /// it held before. A cache batch (`put_prepared_batch`) is all-or-nothing + /// only through this. There is no default, because a write assembled from + /// `push` calls leaves the store partial when one of them fails; a store + /// without an atomic write does not compile. It returns one stamp per entry given, in order, the `stored_at` written for it + /// (an entry deduplicated away gets the stamp of the one that landed for its schema id and key); a soft write that wrote + /// nothing returns none: /// /// ```compile_fail,E0046 /// use cultcache_rs::{CacheBackingStore, CultCacheEnvelope}; @@ -1085,15 +1370,33 @@ pub trait CacheBackingStore: Send { /// fn pull_all(&self) -> anyhow::Result> { /// Ok(Vec::new()) /// } - /// fn push(&mut self, _entry: &CultCacheEnvelope) -> anyhow::Result<()> { - /// Ok(()) + /// fn push(&mut self, _entry: &CultCacheEnvelope) -> anyhow::Result { + /// Ok(String::new()) /// } /// fn delete(&mut self, _entry: &CultCacheEnvelope) -> anyhow::Result<()> { /// Ok(()) /// } /// } /// ``` - fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> Result<()>; + fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> Result>; + + /// The record at `key`, of any type, with its content version, from one read; `None` when the store holds nothing there. + /// [`CultCache::update`] needs this and [`CacheBackingStore::commit_if`]; a store without them refuses it. + fn read(&self, _key: &str) -> Result> { + Err(anyhow!( + "{} has no read and commit_if: CultCache::update needs read and commit_if", + std::any::type_name::() + )) + } + + /// Writes `entry` over the record at its key only if that record's version is `expected` (`None`: nothing is there), in + /// one step under the store's own exclusion. A mismatch writes nothing. + fn commit_if(&mut self, _entry: &CultCacheEnvelope, _expected: Option<&str>) -> Result { + Err(anyhow!( + "{} has no read and commit_if: CultCache::update needs read and commit_if", + std::any::type_name::() + )) + } } #[derive(Clone, Debug, PartialEq, Eq)] @@ -1482,6 +1785,16 @@ impl SingleFileMessagePackBackingStore { self.read_all_unlocked() } + /// The schema catalog the store file publishes, read under the same shared lock [`CacheBackingStore::pull_all`] takes. An absent + /// file and a legacy envelope array have no catalog and give an empty one. Resolve a record's `schema_id` against it with + /// [`resolve_schema_entry`]. + pub fn pull_schema_catalog(&self) -> Result> { + if is_absent(&self.path)? && is_absent(&self.lock_path())? { + return Ok(Vec::new()); + } + self.with_shared_lock(|| Ok(self.read_store_unlocked()?.catalog.iter().map(SchemaCatalogEntry::from).collect())) + } + fn write_all_unlocked(&self, entries: &[CultCacheEnvelope]) -> Result<()> { let failed = |kind| CultCacheStoreWriteFailed { path: self.path.clone(), @@ -1491,13 +1804,7 @@ impl SingleFileMessagePackBackingStore { // payloads this store passes through may hold ids), one that is gone or legacy is written v1. The file is read by // the same reader as an open, so a store this runtime cannot read completely, a variant included, is refused // and never overwritten. - let disk = match self.read_store_unlocked() { - // A dangling link holds no store, so a whole-file replace has no header to keep and replaces the link (R3 pins - // this). The only NotFound the reader returns is that one: a merging writer has already refused it when it - // read the current entries. - Err(error) if io_not_found(&error) => DiskStore::absent(), - other => other?, - }; + let disk = self.read_store_unlocked()?; let entries = settle_one_record_per_key(entries, &disk)?; let bytes = encode_store_snapshot(&entries, &disk) .and_then(|snapshot| rmp_serde::to_vec(&snapshot).context("failed to encode MessagePack")) @@ -1539,10 +1846,11 @@ impl SingleFileMessagePackBackingStore { } fn with_exclusive_lock(&self, action: impl FnOnce() -> Result) -> Result { + refuse_linked(&self.path)?; let lock = self.open_lock_file()?; fs2::FileExt::lock_exclusive(&lock) .with_context(|| format!("failed to lock {}", self.lock_path().display()))?; - release_lock(lock, action()) + self.run_locked(lock, action) } /// Replaces the named entries only if each still matches its expectation. @@ -1650,7 +1958,12 @@ impl SingleFileMessagePackBackingStore { } fn try_with_exclusive_lock(&self, action: impl FnOnce() -> Result) -> Result> { - let lock = self.open_lock_file()?; + refuse_linked(&self.path)?; + #[cfg(test)] + run_after_pre_lock_look(); + let Some(lock) = self.try_open_lock_file()? else { + return Ok(None); + }; match fs2::FileExt::try_lock_exclusive(&lock) { Ok(()) => {} Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => return Ok(None), @@ -1659,22 +1972,40 @@ impl SingleFileMessagePackBackingStore { .with_context(|| format!("failed to lock {}", self.lock_path().display())); } } - release_lock(lock, action()).map(Some) + self.run_locked(lock, action).map(Some) } + /// The one tail of both exclusive entries: with the lock held, look for a link again (one may have been put at the + /// path while the lock was awaited; the look before the lock saw the path earlier), run the action, release. + fn run_locked(&self, lock: File, action: impl FnOnce() -> Result) -> Result { + release_lock(lock, refuse_linked(&self.path).and_then(|()| action())) + } + + /// Opens the lock file, waiting through a Windows sharing violation the way the lock itself waits. fn open_lock_file(&self) -> Result { + let lock_path = self.prepare_lock_dir()?; + retry_while_busy( + || open_lock_once(&lock_path), + is_sharing_violation, + || std::thread::sleep(std::time::Duration::from_millis(5)), + ) + .with_context(|| format!("failed to open {}", lock_path.display())) + } + + /// The single-attempt open for the try entry: a sharing violation is `None`, the same answer as a held lock. + fn try_open_lock_file(&self) -> Result> { + let lock_path = self.prepare_lock_dir()?; + open_unless_busy(|| open_lock_once(&lock_path), is_sharing_violation) + .with_context(|| format!("failed to open {}", lock_path.display())) + } + + fn prepare_lock_dir(&self) -> Result { let lock_path = self.lock_path(); if let Some(parent) = lock_path.parent() { fs::create_dir_all(parent) .with_context(|| format!("failed to create {}", parent.display()))?; } - OpenOptions::new() - .create(true) - .read(true) - .truncate(false) - .write(true) - .open(&lock_path) - .with_context(|| format!("failed to open {}", lock_path.display())) + Ok(lock_path) } fn lock_path(&self) -> PathBuf { @@ -1750,14 +2081,46 @@ impl CacheBackingStore for SingleFileMessagePackBackingStore { self.with_shared_lock(|| self.read_all_unlocked()) } - fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()> { + fn push(&mut self, entry: &CultCacheEnvelope) -> Result { self.with_exclusive_lock(|| { let mut entries = self.read_all_unlocked()?; + let entry = restamped_over(entry, &entries)?; // A write replaces the record of its own type under its key; a second type under the key reaches the commit guard. - entries.retain(|candidate| entry_id(candidate) != entry_id(entry)); + entries.retain(|candidate| entry_id(candidate) != entry_id(&entry)); entries.push(entry.clone()); entries.sort_by_key(entry_id); - self.write_all_unlocked(&entries) + self.write_all_unlocked(&entries)?; + Ok(entry.stored_at) + }) + } + + fn read(&self, key: &str) -> Result> { + if is_absent(&self.path)? && is_absent(&self.lock_path())? { + return Ok(None); + } + self.with_shared_lock(|| { + Ok(self + .read_all_unlocked()? + .into_iter() + .find(|entry| entry.key == key) + .map(|envelope| CultStoredRecord { version: record_version(&envelope), envelope })) + }) + } + + fn commit_if(&mut self, entry: &CultCacheEnvelope, expected: Option<&str>) -> Result { + self.with_exclusive_lock(|| { + let mut entries = self.read_all_unlocked()?; + let actual = entries.iter().find(|held| held.key == entry.key).map(record_version); + if actual.as_deref() != expected { + return Ok(CultCommitIf::Mismatch); + } + let entry = restamped_over(entry, &entries)?; + // Only the record of the entry's own identity is replaced; a second type under the key reaches the commit guard, as push does. + entries.retain(|held| entry_id(held) != entry_id(&entry)); + entries.push(entry.clone()); + entries.sort_by_key(entry_id); + self.write_all_unlocked(&entries)?; + Ok(CultCommitIf::Committed { stored_at: entry.stored_at }) }) } @@ -1769,23 +2132,32 @@ impl CacheBackingStore for SingleFileMessagePackBackingStore { }) } - fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> Result<()> { + fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> Result> { self.with_exclusive_lock(|| { // A soft flush asks the one reader: nothing at the path is written, a store it reads is left alone, and anything // the reader refuses (an unreadable file, a dangling link) is refused here as it is for open. if options.soft && !self.read_store_unlocked()?.absent { - return Ok(()); + return Ok(Vec::new()); } // Of entries sharing a key and a schema id the last one given is the record, whatever their type names (a type - // renamed under a stable id); a second schema id under a key reaches the commit guard. - let mut by_id = BTreeMap::<(String, String), &CultCacheEnvelope>::new(); + // renamed under a stable id); it replaces the held record under that key and schema id. A second schema id + // under a key reaches the commit guard, which judges the merged store. + let mut merged = self.read_all_unlocked()?; + let mut by_id = BTreeMap::<(String, String), CultCacheEnvelope>::new(); for entry in entries { - by_id.insert((schema_identity(entry).to_string(), entry.key.clone()), entry); + by_id.insert((schema_identity(entry).to_string(), entry.key.clone()), restamped_over(entry, &merged)?); } - let mut entries: Vec = by_id.into_values().cloned().collect(); - entries.sort_by_key(entry_id); - self.write_all_unlocked(&entries) + let replaced: BTreeSet<_> = by_id.values().map(entry_id).collect(); + merged.retain(|held| !replaced.contains(&entry_id(held))); + let stamps = entries + .iter() + .map(|entry| by_id[&(schema_identity(entry).to_string(), entry.key.clone())].stored_at.clone()) + .collect(); + merged.extend(by_id.into_values()); + merged.sort_by_key(entry_id); + self.write_all_unlocked(&merged)?; + Ok(stamps) }) } } @@ -2547,14 +2919,14 @@ impl CacheBackingStore for OwnedRedbMessagePackBackingStore { read_all_redb(&table) } - fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()> { + fn push(&mut self, entry: &CultCacheEnvelope) -> Result { let write = self.inner.database.begin_write()?; { let mut table = write.open_table(REDB_ENVELOPES)?; insert_redb_entry(&mut table, entry)?; } write.commit()?; - Ok(()) + Ok(entry.stored_at.clone()) } fn delete(&mut self, entry: &CultCacheEnvelope) -> Result<()> { @@ -2568,24 +2940,17 @@ impl CacheBackingStore for OwnedRedbMessagePackBackingStore { Ok(()) } - fn push_all(&mut self, entries: &[CultCacheEnvelope], _options: PushAllOptions) -> Result<()> { + fn push_all(&mut self, entries: &[CultCacheEnvelope], _options: PushAllOptions) -> Result> { unique_batch_ids(entries, "push all")?; let write = self.inner.database.begin_write()?; { let mut table = write.open_table(REDB_ENVELOPES)?; - let keys = table - .iter()? - .map(|row| Ok(row?.0.value().to_vec())) - .collect::>>()?; - for key in keys { - table.remove(key.as_slice())?; - } for entry in entries { insert_redb_entry(&mut table, entry)?; } } write.commit()?; - Ok(()) + Ok(entries.iter().map(|entry| entry.stored_at.clone()).collect()) } } @@ -2684,7 +3049,7 @@ impl CacheBackingStore for RedbMessagePackBackingStore { }) } - fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()> { + fn push(&mut self, entry: &CultCacheEnvelope) -> Result { self.with_database(|database| { let write = database.begin_write()?; { @@ -2692,7 +3057,7 @@ impl CacheBackingStore for RedbMessagePackBackingStore { insert_redb_entry(&mut table, entry)?; } write.commit()?; - Ok(()) + Ok(entry.stored_at.clone()) }) } @@ -2709,25 +3074,18 @@ impl CacheBackingStore for RedbMessagePackBackingStore { }) } - fn push_all(&mut self, entries: &[CultCacheEnvelope], _options: PushAllOptions) -> Result<()> { + fn push_all(&mut self, entries: &[CultCacheEnvelope], _options: PushAllOptions) -> Result> { unique_batch_ids(entries, "push all")?; self.with_database(|database| { let write = database.begin_write()?; { let mut table = write.open_table(REDB_ENVELOPES)?; - let keys = table - .iter()? - .map(|row| Ok(row?.0.value().to_vec())) - .collect::>>()?; - for key in keys { - table.remove(key.as_slice())?; - } for entry in entries { insert_redb_entry(&mut table, entry)?; } } write.commit()?; - Ok(()) + Ok(entries.iter().map(|entry| entry.stored_at.clone()).collect()) }) } } @@ -2738,7 +3096,7 @@ struct CultCacheStoreRegistration { } pub struct CultCache { - definitions: BTreeMap, + definitions: BTreeMap, /// The registered types declared `#[cultcache(global)]`. globals: BTreeSet, schema_name_definitions: BTreeMap, @@ -2753,8 +3111,41 @@ pub struct CultCache { /// Records a store holds that no registered type claims: (store index, the envelope as the store read it). Derived from the last /// pull and replaced with it; written nowhere else. It decides only refusals at its keys. foreign: Vec<(usize, CultCacheEnvelope)>, + /// Whether the cache holds an image of its stores. While it is not `Pulled`, every write and removal is refused. + load: LoadState, } +/// What the cache knows of its stores. `Unpulled` is the state of a cache that has never pulled, or that gained a store +/// since: its image lacks whatever the stores hold, and a batch would replace them with it. +#[derive(Clone, Debug, PartialEq, Eq)] +enum LoadState { + Unpulled, + Pulled, + /// Why the last pull was refused. + Refused(String), +} + +/// A write or removal refused because the cache holds no image of its backing stores: its last load was refused, or it has +/// not pulled since its stores were attached. Any write would persist an image that lacks the records it never read. +/// Pull; for a refused load, first repair the store named by `reason` (which names a type, a key and a clause, never a +/// payload). +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CultCacheLoadRefused { + pub reason: String, +} + +impl std::fmt::Display for CultCacheLoadRefused { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + formatter, + "CultCache refuses writes and removals: it holds no image of its backing stores ({}); repair the store if it was refused and pull again", + self.reason + ) + } +} + +impl std::error::Error for CultCacheLoadRefused {} + /// A record a store holds that no registered type claims: carried, never served. #[derive(Clone, Debug, PartialEq, Eq)] pub struct ForeignRecord { @@ -2816,6 +3207,7 @@ impl CultCache { projectors: BTreeMap::new(), stores: Vec::new(), foreign: Vec::new(), + load: LoadState::Unpulled, } } @@ -2825,8 +3217,8 @@ impl CultCache { "CultCache entry types must declare a non-empty type" )); } - if let Some(existing_schema) = self.definitions.get(T::TYPE) - && *existing_schema != T::SCHEMA_NAME + if let Some(existing) = self.definitions.get(T::TYPE) + && existing.schema_name != T::SCHEMA_NAME { return Err(anyhow!( "CultCache already has a different definition registered for type {:?}", @@ -2875,7 +3267,7 @@ impl CultCache { } } } - self.definitions.insert(T::TYPE.to_string(), T::SCHEMA_NAME); + self.definitions.insert(T::TYPE.to_string(), Definition { schema_name: T::SCHEMA_NAME, unread_slots: unread_slots_of:: }); self.schema_name_definitions .insert(T::SCHEMA_NAME.to_string(), T::TYPE.to_string()); Ok(()) @@ -2921,6 +3313,10 @@ impl CultCache { store: Box::new(store), types, }); + let loaded_before = self.load.clone(); + if self.load == LoadState::Pulled { + self.load = LoadState::Unpulled; + } // A record's home cannot change after it is admitted. let candidate = self.stores.len() - 1; let held: BTreeSet = self.entries.keys().map(|(type_id, _)| type_id.clone()).collect(); @@ -2939,6 +3335,7 @@ impl CultCache { self.describe_store(after) ); self.stores.pop(); + self.load = loaded_before; return Err(error); } } @@ -2952,12 +3349,45 @@ impl CultCache { /// Reads every store into a staged image and replaces the cache's view /// only when all of them loaded, so a refused load admits nothing. pub fn pull_all_backing_stores(&mut self) -> Result<()> { + match self.stage_pull() { + Ok((loaded, foreign, holders)) => { + self.entries = loaded; + self.foreign = foreign; + self.index_holders = holders; + self.load = LoadState::Pulled; + Ok(()) + } + Err(error) => { + self.load = LoadState::Refused(format!("{error:#}")); + Err(error) + } + } + } + + /// The one gate of every write and removal: a cache whose load was refused holds no image of its stores. + /// A cache without stores has nothing to lose and needs no pull. + fn require_loaded(&self) -> Result<()> { + match &self.load { + _ if self.stores.is_empty() => Ok(()), + LoadState::Pulled => Ok(()), + LoadState::Refused(reason) => Err(CultCacheLoadRefused { reason: reason.clone() }.into()), + LoadState::Unpulled => Err(CultCacheLoadRefused { + reason: "pull_all_backing_stores has not run since the stores were attached".to_string(), + } + .into()), + } + } + + /// The image every store holds, judged whole, installed nowhere. + #[allow(clippy::type_complexity)] + fn stage_pull(&self) -> Result<(BTreeMap<(String, String), CultCacheEnvelope>, Vec<(usize, CultCacheEnvelope)>, IndexHolders)> { let mut loaded = BTreeMap::new(); let mut foreign = Vec::new(); let known_types: BTreeSet = self.definitions.keys().cloned().collect(); let schema_name_definitions = self.schema_name_definitions.clone(); for (index, registration) in self.stores.iter().enumerate() { for mut entry in registration.store.pull_all()? { + require_record_payload(&entry)?; let Some(canonical_type) = resolve_registered_type( &known_types, &schema_name_definitions, @@ -2990,10 +3420,7 @@ impl CultCache { } self.refuse_second_globals(&loaded)?; let holders = self.judge_image(&loaded)?; - self.entries = loaded; - self.foreign = foreign; - self.index_holders = holders; - Ok(()) + Ok((loaded, foreign, holders)) } /// The records the last pull found in a store that no registered type claims, sorted by key (UTF-16 order). The cache does not @@ -3036,6 +3463,85 @@ impl CultCache { Ok(Some(payload)) } + /// Whether a write or removal of `(type_id, key)` would be admitted: false when the held payload carries slots past + /// its registered type's declared slots, at any depth, or cannot be read by this build at all (the record is + /// read-only), when the cache holds no image of its stores (see [`CultCacheLoadRefused`]), when a store carries a foreign record at the key, or when another registered type holds it. True for a + /// key nothing holds. An unregistered type is an error. It does not look at store paths: a link at a store path is not + /// predicted, and the write refuses it with [`StoreLinkedError`]. Advisory: the write entries ask the same checks themselves. + pub fn is_writable(&self, type_id: &str, key: &str) -> Result { + if self.require_loaded().is_err() { + return Ok(false); + } + if !matches!(self.held_finding(type_id, key)?, HeldFinding::Free) { + return Ok(false); + } + let probe = CultCacheEnvelope { + key: key.to_string(), + r#type: type_id.to_string(), + payload: Vec::new(), + stored_at: String::new(), + schema_id: None, + }; + Ok(self.home_index(type_id).is_ok() && self.refuse_retype(&self.entries, &self.foreign, &probe).is_ok()) + } + + /// The one decode behind every write entry and [`CultCache::is_writable`]: the held envelope at `(type_id, key)` + /// read under the registered type's mark. + fn held_finding(&self, type_id: &str, key: &str) -> Result { + let Some(definition) = self.definitions.get(type_id) else { + bail!("No entry type registered for CultCache envelope type {type_id:?}"); + }; + let Some(held) = self.entries.get(&entry_id_parts(type_id, key)) else { + return Ok(HeldFinding::Free); + }; + Ok(match (definition.unread_slots)(&held.payload) { + Ok(None) => HeldFinding::Free, + Ok(Some(unread)) => HeldFinding::Marked(unread), + Err(error) => HeldFinding::Unreadable(error), + }) + } + + /// Refuses a write or removal of a record held read-only, before any store is touched: for the slots its payload + /// carries past this build's, or because this build cannot read the payload at all. + fn refuse_unread_slots(&self, type_id: &str, key: &str) -> Result<()> { + match self.held_finding(type_id, key)? { + HeldFinding::Free => Ok(()), + finding => { + let held = &self.entries[&entry_id_parts(type_id, key)]; + self.refuse_finding(type_id, key, schema_identity(held), finding) + } + } + } + + /// The same refusal for a record read from a store: [`CultCache::update`] copies it, so it must not carry slots this build + /// would drop or bytes it cannot read. + fn refuse_record_slots(&self, type_id: &str, key: &str, schema_id: &str, payload: &[u8]) -> Result<()> { + let Some(definition) = self.definitions.get(type_id) else { + bail!("No entry type registered for CultCache envelope type {type_id:?}"); + }; + let finding = match (definition.unread_slots)(payload) { + Ok(None) => return Ok(()), + Ok(Some(unread)) => HeldFinding::Marked(unread), + Err(error) => HeldFinding::Unreadable(error), + }; + self.refuse_finding(type_id, key, schema_id, finding) + } + + fn refuse_finding(&self, type_id: &str, key: &str, schema_id: &str, finding: HeldFinding) -> Result<()> { + match finding { + HeldFinding::Free => Ok(()), + HeldFinding::Unreadable(error) => Err(error.context(format!( + "CultCache entry {type_id:?} at key {key:?} is held read-only: this build cannot read its payload. A newer writer's record is read with that build; a corrupt one is restored from a backup or removed at the store itself, not through the cache" + ))), + HeldFinding::Marked(unread) => Err(anyhow::Error::new(SchemaConflictError { + schema_id: schema_id.to_string(), + schema_names: vec![self.definitions[type_id].schema_name.to_string()], + record_key: key.to_string(), + unread_slots: Some(unread), + })), + } + } + pub fn get_required(&self, key: &str) -> Result { self.get::(key)? .ok_or_else(|| anyhow!("CultCache has no {:?} entry at key {:?}", T::TYPE, key)) @@ -3165,15 +3671,10 @@ impl CultCache { } pub fn put(&mut self, key: impl Into, value: &T) -> Result { + self.require_loaded()?; let (entry, parsed) = self.prepare_entry(key, value)?; - self.refuse_second_global(&self.entries, &entry.r#type, &entry.key)?; - self.refuse_retype(&self.entries, &self.foreign, &entry)?; - let change = self.judge_unique(&[&entry])?; - if let Some(home) = self.home_index(T::TYPE)? { - self.stores[home].store.push(&entry)?; - } - self.entries.insert(entry_id(&entry), entry); - change.apply(&mut self.index_holders); + self.refuse_unread_slots(&entry.r#type, &entry.key)?; + self.write_entry(entry, None)?; Ok(parsed) } @@ -3241,6 +3742,7 @@ impl CultCache { } pub fn put_prepared_batch(&mut self, staged: Vec) -> Result<()> { + self.require_loaded()?; if staged.is_empty() { return Err(anyhow!("CultCache batch must contain at least one entry")); } @@ -3259,13 +3761,9 @@ impl CultCache { entry.r#type )); } - if entry.stored_at.trim().is_empty() { - return Err(anyhow!( - "CultCache envelope stored_at for type {:?} must be non-empty", - entry.r#type - )); - } - require_payload_depth(entry)?; + require_stored_at_stamp(entry)?; + require_record_payload(entry)?; + self.refuse_unread_slots(&entry.r#type, &entry.key)?; self.refuse_claimed_id(entry)?; let Some(home) = self.home_index(&entry.r#type)? else { continue; @@ -3288,18 +3786,24 @@ impl CultCache { } let change = self.judge_unique(&staged.iter().collect::>())?; if let Some(store_index) = store_index { - // The home store receives its own records only: those it already holds in this cache plus the staged ones, and - // the records it carries that no registered type claims: push_all replaces the store's records, so a foreign - // record is laid back as the store read it. - let snapshot: Vec<_> = next - .values() - .filter(|entry| self.resolve_route_index(&entry.r#type) == Some(store_index)) - .chain(self.foreign.iter().filter(|(home, _)| *home == store_index).map(|(_, entry)| entry)) - .cloned() + // The home store is handed the staged records alone and applies them onto what it holds now. + // A key staged twice lands as its last record, whatever the store. + let landing: Vec = staged + .iter() + .map(|entry| (entry_id(entry), entry.clone())) + .collect::>() + .into_values() .collect(); - self.stores[store_index] - .store - .push_all(&snapshot, PushAllOptions::default())?; + let stamps = self.stores[store_index].store.push_all(&landing, PushAllOptions::default())?; + if stamps.len() == landing.len() { + for (entry, stamp) in landing.iter().zip(stamps) { + if chrono::DateTime::parse_from_rfc3339(&stamp).is_ok() + && let Some(held) = next.get_mut(&entry_id(entry)) + { + held.stored_at = stamp; + } + } + } } self.entries = next; change.apply(&mut self.index_holders); @@ -3307,16 +3811,11 @@ impl CultCache { } pub fn put_envelope(&mut self, entry: CultCacheEnvelope) -> Result { + self.require_loaded()?; let parsed = self.validate_envelope::(&entry)?; + self.refuse_unread_slots(&entry.r#type, &entry.key)?; self.refuse_claimed_id(&entry)?; - self.refuse_second_global(&self.entries, &entry.r#type, &entry.key)?; - self.refuse_retype(&self.entries, &self.foreign, &entry)?; - let change = self.judge_unique(&[&entry])?; - if let Some(home) = self.home_index(T::TYPE)? { - self.stores[home].store.push(&entry)?; - } - self.entries.insert(entry_id(&entry), entry); - change.apply(&mut self.index_holders); + self.write_entry(entry, None)?; Ok(parsed) } @@ -3326,6 +3825,7 @@ impl CultCache { /// admission; CultCache still enforces registered type, identity, timestamp, /// the payload nesting bound ([`MAX_PAYLOAD_DEPTH`]), routing, and persistence. pub fn put_raw_envelope(&mut self, entry: CultCacheEnvelope) -> Result<()> { + self.require_loaded()?; if !self.definitions.contains_key(&entry.r#type) { return Err(anyhow!( "No entry type registered for CultCache envelope type {:?}", @@ -3338,23 +3838,37 @@ impl CultCache { entry.r#type )); } - if entry.stored_at.trim().is_empty() { - return Err(anyhow!( - "CultCache envelope stored_at for type {:?} must be non-empty", - entry.r#type - )); - } - require_payload_depth(&entry)?; + require_stored_at_stamp(&entry)?; + require_record_payload(&entry)?; + self.refuse_unread_slots(&entry.r#type, &entry.key)?; self.refuse_claimed_id(&entry)?; + self.write_entry(entry, None)?; + Ok(()) + } + + /// The one landing of a single record, for `put`, `put_envelope`, `put_raw_envelope` and `update`: the global, retype and + /// unique judges, then the home store's write, then the install of the stamp the store reports (kept only when it parses + /// as RFC 3339, else the entry's own). `expected` is `None` for a blind write; `Some(version)` writes only over the record at + /// that version (`Some(None)`: over nothing) and returns `false`, installing nothing, when the store found another. + fn write_entry(&mut self, mut entry: CultCacheEnvelope, expected: Option>) -> Result { self.refuse_second_global(&self.entries, &entry.r#type, &entry.key)?; self.refuse_retype(&self.entries, &self.foreign, &entry)?; let change = self.judge_unique(&[&entry])?; if let Some(home) = self.home_index(&entry.r#type)? { - self.stores[home].store.push(&entry)?; + let stored_at = match expected { + None => self.stores[home].store.push(&entry)?, + Some(version) => match self.stores[home].store.commit_if(&entry, version.as_deref())? { + CultCommitIf::Committed { stored_at } => stored_at, + CultCommitIf::Mismatch => return Ok(false), + }, + }; + if chrono::DateTime::parse_from_rfc3339(&stored_at).is_ok() { + entry.stored_at = stored_at; + } } self.entries.insert(entry_id(&entry), entry); change.apply(&mut self.index_holders); - Ok(()) + Ok(true) } fn validate_envelope(&self, entry: &CultCacheEnvelope) -> Result { @@ -3389,17 +3903,88 @@ impl CultCache { }) } - pub fn update(&mut self, key: &str, updater: F) -> Result + /// Read the record at `key` as its home store holds it now, apply `apply` to a copy, and write the result only if the store + /// still holds the record that was read; on a lost race run `apply` again on a fresh read, up to 5 attempts + /// ([`CultCache::update_with_attempts`] takes the bound). `apply` may run several times and must have no side effects. + /// `None` from `apply` writes nothing ([`CultUpdateOutcome::Declined`]); losing every attempt is a value + /// ([`CultUpdateOutcome::LostRace`]), not an error. On `Updated` the write is durable. [`CultCache::put`] is the blind write. + pub fn update(&mut self, key: &str, apply: F) -> Result> where T: DatabaseEntry, - F: FnOnce(Option) -> T, + F: FnMut(Option) -> Option, { - let current = self.get::(key)?; - self.put::(key.to_string(), &updater(current)) + self.update_with_attempts(key, 5, apply) + } + + /// [`CultCache::update`] with its attempt bound: `max_attempts` of 0 is refused before `apply` runs. A refusal, a store + /// error or a failed judge is returned at once and never retried; after the last lost attempt the cache pulls its stores + /// (display only) and reports `LostRace`. + pub fn update_with_attempts(&mut self, key: &str, max_attempts: usize, mut apply: F) -> Result> + where + T: DatabaseEntry, + F: FnMut(Option) -> Option, + { + if max_attempts == 0 { + bail!("update max_attempts must be at least 1"); + } + for attempt in 1..=max_attempts { + let (copy, expected) = self.read_for_update::(key)?; + let Some(next) = apply(copy) else { + return Ok(CultUpdateResult { outcome: CultUpdateOutcome::Declined, document: self.get::(key)? }); + }; + let (entry, _) = self.prepare_entry(key, &next)?; + if self.write_entry(entry, expected)? { + return Ok(CultUpdateResult { outcome: CultUpdateOutcome::Updated, document: self.get::(key)? }); + } + if attempt == max_attempts { + self.pull_all_backing_stores()?; + return Ok(CultUpdateResult { outcome: CultUpdateOutcome::LostRace, document: self.get::(key)? }); + } + } + unreachable!("the last attempt returns") + } + + /// One attempt's read: the refusals that precede `apply`, then the record at `key` as the home store holds it with the version + /// to commit against (`Some(None)` for nothing there; `None` for a cache with no store, which has no version). Installs + /// nothing in the cache. + #[allow(clippy::type_complexity)] + fn read_for_update(&self, key: &str) -> Result<(Option, Option>)> { + self.require_loaded()?; + self.require_entry_type::()?; + self.refuse_unread_slots(T::TYPE, key)?; + let probe = CultCacheEnvelope { + key: key.to_string(), + r#type: T::TYPE.to_string(), + payload: Vec::new(), + stored_at: String::new(), + schema_id: None, + }; + self.refuse_retype(&self.entries, &self.foreign, &probe)?; + let Some(home) = self.home_index(T::TYPE)? else { + return Ok((self.get::(key)?, None)); + }; + let Some(found) = self.stores[home].store.read(key)? else { + return Ok((None, Some(None))); + }; + let known_types: BTreeSet = self.definitions.keys().cloned().collect(); + let resolved = resolve_registered_type( + &known_types, + &self.schema_name_definitions, + found.envelope.schema_id.as_deref(), + &found.envelope.r#type, + ); + if resolved.as_deref() != Some(T::TYPE) { + bail!("Record {key} is not a {}; update cannot copy it.", T::TYPE); + } + self.refuse_record_slots(T::TYPE, key, schema_identity(&found.envelope), &found.envelope.payload)?; + let copy = decode_payload::(&found.envelope.payload)?; + Ok((Some(copy), Some(Some(found.version)))) } pub fn delete(&mut self, key: &str) -> Result { + self.require_loaded()?; self.require_entry_type::()?; + self.refuse_unread_slots(T::TYPE, key)?; let id = entry_id_parts(T::TYPE, key); let Some(entry) = self.entries.get(&id).cloned() else { if let Some((_, foreign)) = self.foreign.iter().find(|(_, foreign)| foreign.key == key) { @@ -3427,7 +4012,9 @@ impl CultCache { /// Replaces the cache's view with the table's records. Every record is /// checked (registered type, and a home store once any store is attached) /// into a local image first, so a refused load admits nothing and leaves - /// the prior view intact. Loading writes no store. + /// the prior view intact. Loading writes no store, but a held record that is read-only (unread slots, or a payload + /// this build cannot read) may not be omitted or replaced by the table: the next batch would lay the image over the + /// store. pub fn load_soa(&mut self, table: CultCacheSoaTable) -> Result<()> { let mut loaded = BTreeMap::new(); for entry in table.into_envelopes()? { @@ -3437,11 +4024,18 @@ impl CultCache { entry.r#type )); } + require_record_payload(&entry)?; self.home_index(&entry.r#type)?; self.refuse_claimed_id(&entry)?; self.refuse_retype(&loaded, &self.foreign, &entry)?; loaded.insert(entry_id(&entry), entry); } + // A load is a write of every held key it omits or changes: the one check refuses those held read-only. + for ((type_id, key), held) in &self.entries { + if loaded.get(&(type_id.clone(), key.clone())).is_none_or(|incoming| incoming.payload != held.payload) { + self.refuse_unread_slots(type_id, key)?; + } + } self.refuse_second_globals(&loaded)?; let holders = self.judge_image(&loaded)?; self.entries = loaded; @@ -3518,7 +4112,7 @@ impl CultCache { } for (slot, keys) in &ordered { if let [first, second, ..] = keys.as_slice() { - let schema = self.definitions.get(&slot.0).copied().unwrap_or(&slot.0); + let schema = self.definitions.get(&slot.0).map_or(slot.0.as_str(), |definition| definition.schema_name); let violation = [slot.1.as_str(), first.as_str(), second.as_str(), schema]; if least.is_none_or(|best| violation_cmp(&violation, &best).is_lt()) { least = Some(violation); @@ -3553,7 +4147,7 @@ impl CultCache { .find(|held_key| held_key.as_str() != key); match held_other { Some(existing_key) => Err(CultCacheSecondGlobal { - schema_name: self.definitions.get(type_id).copied().unwrap_or(type_id).to_string(), + schema_name: self.definitions.get(type_id).map_or(type_id, |definition| definition.schema_name).to_string(), existing_key: existing_key.clone(), key: key.to_string(), } @@ -3597,6 +4191,7 @@ impl CultCache { schema_id: schema_identity(entry).to_string(), schema_names: vec![resolved, entry.r#type.clone()], record_key: entry.key.clone(), + unread_slots: None, })), _ => Ok(()), } @@ -3611,7 +4206,7 @@ impl CultCache { fn require_entry_type(&self) -> Result<()> { match self.definitions.get(T::TYPE) { - Some(schema_name) if *schema_name == T::SCHEMA_NAME => Ok(()), + Some(definition) if definition.schema_name == T::SCHEMA_NAME => Ok(()), _ => Err(anyhow!( "CultCache entry type {:?} is not registered on this cache instance", T::TYPE @@ -3669,6 +4264,127 @@ fn entry_id_parts(r#type: &str, key: &str) -> (String, String) { (r#type.to_string(), key.to_string()) } +/// The content version of a stored record: lowercase hex SHA-256 of the record exactly as the store file encodes it, so it +/// covers the key, schema id, `stored_at` and payload. The only hash in this crate. +fn record_version(entry: &CultCacheEnvelope) -> String { + use sha2::{Digest, Sha256}; + let record = PersistedRecord( + entry.key.clone(), + schema_identity(entry).to_string(), + entry.stored_at.clone(), + entry.payload.clone(), + ); + let bytes = rmp_serde::to_vec(&record).expect("a record of strings and bytes encodes"); + Sha256::digest(bytes).iter().map(|byte| format!("{byte:02x}")).collect() +} + +/// A stamp strictly later than `previous` (to the microsecond): `now` truncated to the microsecond when that is later, +/// otherwise `previous` truncated plus one microsecond. An absent or unparseable `previous` gives `now`. A successor past +/// year 9999 does not exist as a stamp any runtime reads, so it is refused (C# `MintStoredAt` throws there). +fn mint_stored_at(previous: Option<&str>, now: chrono::DateTime) -> Result { + use chrono::SubsecRound; + let now = now.trunc_subsecs(6); + let minted = match previous.and_then(|text| chrono::DateTime::parse_from_rfc3339(text).ok()) { + Some(previous) => { + let previous = previous.with_timezone(&chrono::Utc).trunc_subsecs(6); + if now <= previous { + previous + .checked_add_signed(chrono::Duration::microseconds(1)) + .filter(|successor| chrono::Datelike::year(successor) <= 9999) + .ok_or_else(|| anyhow!("stored_at {previous} has no later stamp: the successor is past year 9999"))? + } else { + now + } + } + None => now, + }; + Ok(minted.to_rfc3339_opts(chrono::SecondsFormat::Micros, true)) +} + +/// The stamp a replacing write carries: `staged` stands only when both stamps parse and it is strictly later than `replaced`. +fn stored_at_over(staged: &str, replaced: &str) -> Result { + match (chrono::DateTime::parse_from_rfc3339(staged), chrono::DateTime::parse_from_rfc3339(replaced)) { + (Ok(staged_at), Ok(replaced_at)) if staged_at > replaced_at => Ok(staged.to_string()), + _ => mint_stored_at(Some(replaced), chrono::Utc::now()), + } +} + +/// `entry` stamped later than the record the store holds at its key, of any type; unchanged where nothing is held. +fn restamped_over(entry: &CultCacheEnvelope, held: &[CultCacheEnvelope]) -> Result { + let mut entry = entry.clone(); + if let Some(replaced) = held.iter().find(|candidate| candidate.key == entry.key) { + entry.stored_at = stored_at_over(&entry.stored_at, &replaced.stored_at)?; + } + Ok(entry) +} + +/// Whether `text` is a `stored_at` stamp: the one shape pinned in `tests/vectors/stored-at-stamp/`. An RFC 3339 date-time, +/// `YYYY-MM-DDThh:mm:ss`, an optional fraction of 1 to 9 digits, then `Z` or `+hh:mm` / `-hh:mm` (hours 00-23, minutes 00-59); +/// uppercase `T` and `Z`, ASCII digits, no surrounding whitespace, a real calendar date, seconds 00-59, a local year of 0001-9999, and an instant (the +/// offset applied) within years 0001-9999. Checked by hand, not by `chrono`'s RFC 3339 reader, which takes other shapes +/// (lowercase `t`, a space, a leap second). A peer's `storedAt` is stored as it arrives, so one that is not a stamp is refused. +pub fn is_stored_at_stamp(text: &str) -> bool { + let b = text.as_bytes(); + let digits = |from: usize, count: usize| -> Option { + let part = b.get(from..from + count)?; + part.iter().all(u8::is_ascii_digit).then(|| part.iter().fold(0, |n, d| n * 10 + u32::from(d - b'0'))) + }; + let at = |index: usize, byte: u8| b.get(index) == Some(&byte); + let (Some(year), Some(month), Some(day), Some(hour), Some(minute), Some(second)) = + (digits(0, 4), digits(5, 2), digits(8, 2), digits(11, 2), digits(14, 2), digits(17, 2)) + else { + return false; + }; + if !(at(4, b'-') && at(7, b'-') && at(10, b'T') && at(13, b':') && at(16, b':')) { + return false; + } + let mut next = 19; + if at(next, b'.') { + let fraction = b[next + 1..].iter().take_while(|byte| byte.is_ascii_digit()).count(); + if !(1..=9).contains(&fraction) { + return false; + } + next += 1 + fraction; + } + let offset_seconds = if b.len() == next + 1 && at(next, b'Z') { + 0 + } else if b.len() == next + 6 && (at(next, b'+') || at(next, b'-')) && at(next + 3, b':') { + let (Some(hours), Some(minutes)) = (digits(next + 1, 2), digits(next + 4, 2)) else { + return false; + }; + if hours > 23 || minutes > 59 { + return false; + } + let magnitude = i64::from(hours * 3600 + minutes * 60); + if at(next, b'-') { -magnitude } else { magnitude } + } else { + return false; + }; + if second > 59 || year < 1 { + return false; + } + let Some(local) = chrono::NaiveDate::from_ymd_opt(year as i32, month, day) + .and_then(|date| date.and_hms_opt(hour, minute, second)) + else { + return false; + }; + use chrono::Datelike; + local + .checked_sub_signed(chrono::TimeDelta::seconds(offset_seconds)) + .is_some_and(|utc| (1..=9999).contains(&utc.year())) +} + +/// A raw envelope is stored under the stamp it carries, so one that is no stamp is refused and nothing is minted for it. +fn require_stored_at_stamp(entry: &CultCacheEnvelope) -> Result<()> { + if is_stored_at_stamp(&entry.stored_at) { + return Ok(()); + } + Err(anyhow!( + "CultCache envelope stored_at for type {:?} must be an RFC 3339 stamp", + entry.r#type + )) +} + fn now_utc_second() -> String { chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Secs, true) } @@ -3693,6 +4409,7 @@ fn encode_store_snapshot(entries: &[CultCacheEnvelope], disk: &DiskStore) -> Res schema_id, schema_names: vec![existing_type, entry.r#type.clone()], record_key: entry.key.clone(), + unread_slots: None, })); } } @@ -3710,6 +4427,7 @@ fn encode_store_snapshot(entries: &[CultCacheEnvelope], disk: &DiskStore) -> Res schema_id, schema_names: vec![arrived_type, document_type], record_key: key, + unread_slots: None, })); } } @@ -3808,7 +4526,7 @@ struct DiskStore { } impl DiskStore { - /// A file that is gone, or a dangling link a whole-file replace is about to replace: an empty v1 store. + /// Nothing at the path: an empty v1 store. A dangling link is something at the path and is refused, not read as this. fn absent() -> Self { Self { absent: true, format: STORE_FORMAT_V1, envelopes: Vec::new(), catalog: Vec::new() } } @@ -3846,6 +4564,7 @@ fn foreign_conflict(carried: &CultCacheEnvelope, written_type: &str) -> anyhow:: schema_id: schema_identity(carried).to_string(), schema_names: vec![carried.r#type.clone(), written_type.to_string()], record_key: carried.key.clone(), + unread_slots: None, }) } @@ -3856,6 +4575,7 @@ fn retype_conflict(held: &CultCacheEnvelope, written: &CultCacheEnvelope) -> any schema_id: schema_identity(written).to_string(), schema_names: vec![held.r#type.clone(), written.r#type.clone()], record_key: written.key.clone(), + unread_slots: None, }) } @@ -3990,12 +4710,6 @@ fn release_lock(lock: File, result: Result) -> Result { result } -fn io_not_found(error: &anyhow::Error) -> bool { - error - .chain() - .any(|cause| cause.downcast_ref::().is_some_and(|io| io.kind() == std::io::ErrorKind::NotFound)) -} - /// True only when nothing is at `path`; a symbolic link, dangling or not, is something. /// A path that cannot be inspected (a file where a directory should be, no permission) /// is an error, never absent. @@ -4057,61 +4771,324 @@ fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> { #[cfg(windows)] fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> { - use std::os::windows::ffi::OsStrExt; - use windows_sys::Win32::Storage::FileSystem::{ - MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW, - }; - injected_write_fault(WriteStep::Rename).context("failed to atomically replace the store")?; + replace_with(&WindowsKernel, staged, destination) +} - let staged_wide = staged - .as_os_str() - .encode_wide() - .chain(std::iter::once(0)) - .collect::>(); - let destination_wide = destination - .as_os_str() - .encode_wide() - .chain(std::iter::once(0)) - .collect::>(); - // A concurrent reader of the destination (another process opening the - // store, an indexer, an antivirus scan) makes the replace fail with a - // sharing or access error for as long as it holds the handle. That is - // transient by construction; a daemon that treats it as fatal dies on - // the first overlap (Muninn serve did, 2026-09-11). Retry briefly. - const ERROR_ACCESS_DENIED: i32 = 5; - const ERROR_SHARING_VIOLATION: i32 = 32; - const ERROR_LOCK_VIOLATION: i32 = 33; +/// The two Windows replace primitives, each answering 0 or a Win32 error code, and the wait +/// between attempts. `replace_with` owns every decision about them. +#[cfg_attr(not(windows), allow(dead_code))] +trait ReplaceKernel { + /// CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with + /// REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle. + fn rename_posix(&self, staged: &Path, destination: &Path) -> u32; + /// MoveFileExW(staged, destination, flags). + fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32; + fn sleep(&self, milliseconds: u64); +} + +/// MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH. +#[cfg_attr(not(windows), allow(dead_code))] +const MOVEFILE_REPLACE_WRITE_THROUGH: u32 = 0x9; + +/// Replaces `destination` with `staged` on Windows: a POSIX-semantics rename first, so a reader +/// that opened the old store keeps it and the destination is rarely seen absent (a stalled rename +/// can still leave it absent for 5-47 ms on NTFS: `variants:ruling:posix-rename-absent-window-document`); +/// MoveFileExW with +/// write-through where POSIX semantics are unsupported (FAT, exFAT, before Windows 10 1607), +/// for the rest of the call; and a bounded retry on access, sharing and lock violations. +/// A concurrent reader (another process, an indexer, an antivirus scan) makes the replace fail +/// transiently, and a daemon that treats that as fatal dies on the first overlap (Muninn serve +/// did, 2026-09-11). Ruling `variants:ruling:windows-replace-posix-first`: cultcache-ts carries the +/// same table on main; cultcache-cs carries it only once its pending PR lands. +#[cfg_attr(not(windows), allow(dead_code))] +fn replace_with(kernel: &dyn ReplaceKernel, staged: &Path, destination: &Path) -> Result<()> { + const ERROR_INVALID_FUNCTION: u32 = 1; + const ERROR_ACCESS_DENIED: u32 = 5; + const ERROR_SHARING_VIOLATION: u32 = 32; + const ERROR_LOCK_VIOLATION: u32 = 33; + const ERROR_NOT_SUPPORTED: u32 = 50; + const ERROR_INVALID_PARAMETER: u32 = 87; const ATTEMPTS: u32 = 25; - const BACKOFF: std::time::Duration = std::time::Duration::from_millis(20); - let mut attempt = 0; - loop { - let success = unsafe { - MoveFileExW( - staged_wide.as_ptr(), - destination_wide.as_ptr(), - MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, - ) + const BACKOFF_MS: u64 = 20; + let mut posix = true; + let mut last_code = 0; + for attempt in 1..=ATTEMPTS { + let mut code = if posix { + kernel.rename_posix(staged, destination) + } else { + kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) }; - if success != 0 { + if posix && matches!(code, ERROR_INVALID_FUNCTION | ERROR_NOT_SUPPORTED | ERROR_INVALID_PARAMETER) { + posix = false; + code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH); + } + if code == 0 { return Ok(()); } - let error = std::io::Error::last_os_error(); - attempt += 1; - let transient = matches!( - error.raw_os_error(), - Some(ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION) - ); - if !transient || attempt >= ATTEMPTS { - return Err(error).with_context(|| { - format!( - "failed to atomically replace {} with {} after {attempt} attempt(s)", - destination.display(), - staged.display() - ) - }); + if !matches!(code, ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION) { + return Err(anyhow::anyhow!( + "failed to atomically replace {} with {}: Win32 error {code} after {attempt} attempt(s)", + destination.display(), + staged.display() + )); } - std::thread::sleep(BACKOFF); + last_code = code; + kernel.sleep(BACKOFF_MS); + } + Err(anyhow::anyhow!( + "failed to atomically replace {} with {} after {ATTEMPTS} attempt(s): last Win32 error {last_code}", + destination.display(), + staged.display() + )) +} + +#[cfg(windows)] +struct WindowsKernel; + +#[cfg(windows)] +fn wide(path: &Path) -> Vec { + use std::os::windows::ffi::OsStrExt; + path.as_os_str().encode_wide().chain(std::iter::once(0)).collect() +} + +#[cfg(windows)] +impl ReplaceKernel for WindowsKernel { + fn rename_posix(&self, staged: &Path, destination: &Path) -> u32 { + use windows_sys::Win32::Foundation::{CloseHandle, GetLastError, INVALID_HANDLE_VALUE}; + use windows_sys::Win32::Storage::FileSystem::{ + CreateFileW, FileRenameInfoEx, SetFileInformationByHandle, DELETE, FILE_FLAG_BACKUP_SEMANTICS, + FILE_RENAME_INFO, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, OPEN_EXISTING, SYNCHRONIZE, + }; + use windows_sys::Win32::System::WindowsProgramming::{ + FILE_RENAME_FLAG_POSIX_SEMANTICS, FILE_RENAME_FLAG_REPLACE_IF_EXISTS, + }; + + let staged_wide = wide(staged); + let destination_wide = wide(destination); + // FILE_RENAME_INFO ends in a variable-length name: allocate it as u64s for alignment. + let name_bytes = (destination_wide.len() - 1) * 2; + let header = std::mem::size_of::(); + let total = header + name_bytes + 2; + let mut buffer = vec![0u64; total.div_ceil(8)]; + unsafe { + let info = buffer.as_mut_ptr() as *mut FILE_RENAME_INFO; + (*info).Anonymous.Flags = FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS; + (*info).RootDirectory = std::ptr::null_mut(); + (*info).FileNameLength = name_bytes as u32; + let name = std::ptr::addr_of_mut!((*info).FileName) as *mut u16; + std::ptr::copy_nonoverlapping(destination_wide.as_ptr(), name, destination_wide.len()); + + let handle = CreateFileW( + staged_wide.as_ptr(), + DELETE | SYNCHRONIZE, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, + std::ptr::null(), + OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS, + std::ptr::null_mut(), + ); + if handle == INVALID_HANDLE_VALUE { + return GetLastError(); + } + let ok = SetFileInformationByHandle(handle, FileRenameInfoEx, info as *const _, total as u32); + let code = if ok == 0 { GetLastError() } else { 0 }; + CloseHandle(handle); + code + } + } + + fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32 { + use windows_sys::Win32::Foundation::GetLastError; + use windows_sys::Win32::Storage::FileSystem::MoveFileExW; + let ok = unsafe { MoveFileExW(wide(staged).as_ptr(), wide(destination).as_ptr(), flags) }; + if ok != 0 { + 0 + } else { + unsafe { GetLastError() } + } + } + + fn sleep(&self, milliseconds: u64) { + std::thread::sleep(std::time::Duration::from_millis(milliseconds)); + } +} + +fn open_lock_once(lock_path: &Path) -> std::io::Result { + OpenOptions::new() + .create(true) + .read(true) + .truncate(false) + .write(true) + .open(lock_path) +} + +/// Windows ERROR_SHARING_VIOLATION (raw os error 32): another handle holds the lock file open without sharing. On Unix +/// errno 32 is EPIPE and is never a busy file. +#[cfg(windows)] +fn is_sharing_violation(error: &std::io::Error) -> bool { + error.raw_os_error() == Some(32) +} + +#[cfg(not(windows))] +fn is_sharing_violation(_error: &std::io::Error) -> bool { + false +} + +/// Repeats `open` while it fails busy, sleeping between tries and without a bound, as the lock waits; any other error surfaces. +fn retry_while_busy( + mut open: impl FnMut() -> std::io::Result, + busy: impl Fn(&std::io::Error) -> bool, + mut sleep: impl FnMut(), +) -> std::io::Result { + loop { + match open() { + Err(error) if busy(&error) => sleep(), + other => return other, + } + } +} + +/// One attempt: `None` when the open fails busy, any other error surfaces. +fn open_unless_busy( + open: impl FnOnce() -> std::io::Result, + busy: impl Fn(&std::io::Error) -> bool, +) -> std::io::Result> { + match open() { + Ok(file) => Ok(Some(file)), + Err(error) if busy(&error) => Ok(None), + Err(error) => Err(error), + } +} + +#[cfg(test)] +mod replace_with_tests { + use super::{replace_with, ReplaceKernel}; + use std::cell::RefCell; + use std::collections::VecDeque; + use std::path::Path; + + #[derive(Debug, PartialEq)] + enum Call { + Posix, + MoveFile(u32), + Sleep(u64), + } + + /// Answers each primitive from a script; an empty script answers 0. + #[derive(Default)] + struct FakeKernel { + posix: RefCell>, + move_file: RefCell>, + calls: RefCell>, + } + + impl FakeKernel { + fn scripted(posix: &[u32], move_file: &[u32]) -> Self { + Self { + posix: RefCell::new(posix.iter().copied().collect()), + move_file: RefCell::new(move_file.iter().copied().collect()), + ..Self::default() + } + } + fn calls(&self) -> Vec { + self.calls.replace(Vec::new()) + } + } + + impl ReplaceKernel for FakeKernel { + fn rename_posix(&self, _: &Path, _: &Path) -> u32 { + self.calls.borrow_mut().push(Call::Posix); + self.posix.borrow_mut().pop_front().unwrap_or(0) + } + fn move_file_ex(&self, _: &Path, _: &Path, flags: u32) -> u32 { + self.calls.borrow_mut().push(Call::MoveFile(flags)); + self.move_file.borrow_mut().pop_front().unwrap_or(0) + } + fn sleep(&self, milliseconds: u64) { + self.calls.borrow_mut().push(Call::Sleep(milliseconds)); + } + } + + fn run(kernel: &FakeKernel) -> anyhow::Result<()> { + replace_with(kernel, Path::new("staged"), Path::new("store.cc")) + } + + #[test] + fn a_posix_rename_that_lands_is_the_whole_replace() { + let kernel = FakeKernel::scripted(&[0], &[]); + run(&kernel).unwrap(); + assert_eq!(kernel.calls(), vec![Call::Posix]); + } + + #[test] + fn unsupported_posix_semantics_fall_back_to_write_through_and_stay_there() { + for code in [1, 50, 87] { + // The fallback itself is held once, so the retry must use MoveFileExW, not POSIX again. + let kernel = FakeKernel::scripted(&[code], &[32, 0]); + run(&kernel).unwrap_or_else(|e| panic!("code {code}: {e}")); + assert_eq!( + kernel.calls(), + vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)], + "code {code}" + ); + } + } + + #[test] + fn a_held_destination_is_retried_with_posix_until_it_lets_go() { + for code in [5, 32, 33] { + let kernel = FakeKernel::scripted(&[code, code, 0], &[]); + run(&kernel).unwrap(); + assert_eq!( + kernel.calls(), + vec![Call::Posix, Call::Sleep(20), Call::Posix, Call::Sleep(20), Call::Posix], + "code {code}" + ); + } + } + + #[test] + fn a_destination_held_forever_fails_after_twenty_five_attempts_naming_the_code() { + for code in [5, 32, 33] { + let kernel = FakeKernel::scripted(&[code; 40], &[]); + let error = run(&kernel).unwrap_err().to_string(); + let calls = kernel.calls(); + assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 25); + assert_eq!(calls.iter().filter(|c| **c == Call::Sleep(20)).count(), 25); + assert!(error.contains("25 attempt(s)") && error.contains(&format!("error {code}")), "{error}"); + } + } + + #[test] + fn a_held_destination_after_fallback_exhausts_through_move_file_ex() { + let kernel = FakeKernel::scripted(&[87], &[32; 40]); + assert!(run(&kernel).is_err()); + let calls = kernel.calls(); + assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 1); + assert_eq!(calls.iter().filter(|c| matches!(c, Call::MoveFile(0x9))).count(), 25); + } + + #[test] + fn any_other_code_fails_at_once_without_a_retry() { + for (posix, move_file) in [(&[2u32][..], &[][..]), (&[87][..], &[3][..])] { + let kernel = FakeKernel::scripted(posix, move_file); + let error = run(&kernel).unwrap_err().to_string(); + assert!(!kernel.calls().contains(&Call::Sleep(20)), "{error}"); + assert!(error.contains("1 attempt(s)"), "{error}"); + } + } + + #[test] + fn move_file_ex_answering_unsupported_is_an_error_not_a_cue_to_fall_back_again() { + let kernel = FakeKernel::scripted(&[87], &[32, 87]); + let error = run(&kernel).unwrap_err().to_string(); + assert_eq!( + kernel.calls(), + vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)], + "{error}" + ); + assert!(error.contains("error 87") && error.contains("2 attempt(s)"), "{error}"); } } @@ -4120,6 +5097,35 @@ mod windows_replace_tests { use super::replace_file_atomically; use std::os::windows::fs::OpenOptionsExt; + const FILE_SHARE_ALL: u32 = 0x1 | 0x2 | 0x4; // READ | WRITE | DELETE + + /// A reader that shares delete holds the old store open. A POSIX-semantics rename + /// replaces the name while the reader keeps the old content; MoveFileExW alone fails + /// there with error 5 (Soul's Starfire probe, build 26200), so this test passes only + /// if the POSIX path is live. Compile-checked only: it has never run on Windows. + #[test] + fn a_replace_succeeds_over_a_delete_sharing_reader_which_keeps_the_old_content() { + use std::io::Read; + let dir = std::env::temp_dir().join(format!("cultcache-posix-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let destination = dir.join("store.cc"); + let staged = dir.join("store.cc.tmp"); + std::fs::write(&destination, b"old").unwrap(); + std::fs::write(&staged, b"new").unwrap(); + let mut reader = std::fs::OpenOptions::new() + .read(true) + .share_mode(FILE_SHARE_ALL) + .open(&destination) + .unwrap(); + replace_file_atomically(&staged, &destination).expect("POSIX rename over a sharing reader"); + let mut seen = Vec::new(); + reader.read_to_end(&mut seen).unwrap(); + assert_eq!(seen, b"old", "the reader keeps the old content"); + assert_eq!(std::fs::read(&destination).unwrap(), b"new"); + drop(reader); + let _ = std::fs::remove_dir_all(&dir); + } + /// A reader holding the destination open with no sharing makes the /// replace fail for as long as it holds on; it must succeed once the /// reader lets go, not report the first refusal as the outcome. @@ -4182,6 +5188,20 @@ fn injected_write_fault(step: WriteStep) -> std::io::Result<()> { Ok(()) } +// The one gap a test cannot hold open from outside: a try-lock never waits, so what happens between its pre-lock look and +// the lock is only reachable by running the test's closure there. +#[cfg(test)] +thread_local! { + static AFTER_PRE_LOCK_LOOK: std::cell::RefCell>> = const { std::cell::RefCell::new(None) }; +} + +#[cfg(test)] +fn run_after_pre_lock_look() { + if let Some(hook) = AFTER_PRE_LOCK_LOOK.with(|hook| hook.borrow_mut().take()) { + hook(); + } +} + #[cfg(not(test))] #[inline(always)] fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> { @@ -4190,7 +5210,8 @@ fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> { #[cfg(windows)] fn sync_parent_directory(_path: &Path) -> Result<()> { - // MoveFileExW with MOVEFILE_WRITE_THROUGH owns publication durability. + // Publication durability is the replace's: the MoveFileExW fallback writes through; the POSIX-semantics + // rename has no write-through flag and its durability is unmeasured (follow-up windows-replace-run-and-durability). Ok(()) } @@ -4212,7 +5233,7 @@ mod tests { fn pull_all(&self) -> Result> { self.inner.pull_all() } - fn push(&mut self, entry: &CultCacheEnvelope) -> Result<()> { + fn push(&mut self, entry: &CultCacheEnvelope) -> Result { self.inner.push(entry) } fn delete(&mut self, entry: &CultCacheEnvelope) -> Result<()> { @@ -4222,7 +5243,7 @@ mod tests { &mut self, entries: &[CultCacheEnvelope], options: PushAllOptions, - ) -> Result<()> { + ) -> Result> { if self.refuse_batch.load(Ordering::SeqCst) { return Err(anyhow!("injected batch refusal")); } @@ -4415,6 +5436,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(path))?; + cache.pull_all_backing_stores()?; records(&mut cache) } @@ -4656,6 +5678,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_backing_store(SingleFileMessagePackBackingStore::new(&routed), [Settings::TYPE])?; + cache.pull_all_backing_stores()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic))?; cache.pull_all_backing_stores()?; let (staged, _) = cache.prepare_entry("other", &settings_one())?; @@ -4757,6 +5780,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put("shared", &settings_one())?; let before = std::fs::read(&store_path)?; @@ -4789,6 +5813,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put("kept", &settings_one())?; let before = std::fs::read(&store_path)?; @@ -4838,8 +5863,8 @@ mod tests { let absent_note = CultCacheExpectedEnvelope { key: "k".to_string(), r#type: Note::TYPE.to_string(), current: None }; type Writer = fn(&mut SingleFileMessagePackBackingStore, &[CultCacheEnvelope; 3], &CultCacheExpectedEnvelope) -> Result; let writers: Vec<(&str, Writer)> = vec![ - ("push", |s, e, _| s.push(&e[2]).map(|()| true)), - ("push_all", |s, e, _| s.push_all(&[e[0].clone(), e[2].clone()], PushAllOptions::default()).map(|()| true)), + ("push", |s, e, _| s.push(&e[2]).map(|_| true)), + ("push_all", |s, e, _| s.push_all(&[e[0].clone(), e[2].clone()], PushAllOptions::default()).map(|_| true)), ("insert_entry_if_absent", |s, e, _| s.insert_entry_if_absent(e[2].clone())), ("compare_and_swap_entry", |s, e, _| s.compare_and_swap_entry(&e[0], e[2].clone())), ("compare_exchange", |s, e, x| s.compare_exchange(std::slice::from_ref(x), &[e[2].clone()])), @@ -4897,6 +5922,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; let (settings, _) = cache.prepare_entry( "app", @@ -4998,6 +6024,52 @@ mod tests { Ok(()) } + fn is_minted_text(text: &str) -> bool { + let bytes = text.as_bytes(); + bytes.len() == 27 + && bytes.ends_with(b"Z") + && bytes[19] == b'.' + && bytes[20..26].iter().all(u8::is_ascii_digit) + && chrono::DateTime::parse_from_rfc3339(text).is_ok() + } + + fn at(text: &str) -> chrono::DateTime { + chrono::DateTime::parse_from_rfc3339(text).unwrap() + } + + #[test] + fn stamps_stand_only_when_strictly_later_and_otherwise_mint_over_the_replaced() { + // A strictly later staged stamp stands as written. + assert_eq!(stored_at_over("2099-01-01T00:00:00Z", "2026-01-01T00:00:00Z").unwrap(), "2099-01-01T00:00:00Z"); + // An equal or earlier one is replaced by a minted stamp later than the replaced, over C#'s seven digits too. + for replaced in ["2026-01-01T00:00:00.1234567+00:00", "2026-01-01T00:00:00Z", "2099-01-01T00:00:00Z"] { + for staged in [replaced, "2020-01-01T00:00:00Z"] { + let minted = stored_at_over(staged, replaced).unwrap(); + assert!(is_minted_text(&minted), "{minted}"); + assert!(at(&minted) > at(replaced), "{minted} over {replaced}"); + } + } + assert_eq!(stored_at_over("2020-01-01T00:00:00Z", "2099-01-01T00:00:00Z").unwrap(), "2099-01-01T00:00:00.000001Z"); + // An unparseable stamp on either side is never kept: the minted stamp is the replaced's successor or now. + let minted = stored_at_over("not a stamp", "2099-01-01T00:00:00Z").unwrap(); + assert_eq!(minted, "2099-01-01T00:00:00.000001Z"); + let minted = stored_at_over("2099-01-01T00:00:00Z", "not a stamp").unwrap(); + assert!(is_minted_text(&minted)); + assert!((chrono::Utc::now() - at(&minted).with_timezone(&chrono::Utc)).num_seconds().abs() < 60, "{minted}"); + assert!(is_minted_text(&mint_stored_at(None, chrono::Utc::now()).unwrap())); + } + + #[test] + fn a_stamp_with_no_successor_in_year_9999_is_refused() { + for replaced in ["9999-12-31T23:59:59.9999999+00:00", "9999-12-31T23:59:59.999999Z"] { + for staged in [replaced, "2020-01-01T00:00:00Z"] { + assert!(stored_at_over(staged, replaced).is_err(), "{staged} over {replaced}"); + } + } + // One microsecond earlier still has a successor. + assert_eq!(stored_at_over("2020-01-01T00:00:00Z", "9999-12-31T23:59:59.999998Z").unwrap(), "9999-12-31T23:59:59.999999Z"); + } + #[test] fn failed_snapshot_publication_preserves_the_committed_destination() -> Result<()> { let temp = tempfile::tempdir()?; @@ -5603,6 +6675,7 @@ mod tests { inner: SingleFileMessagePackBackingStore::new(&store_path), refuse_batch: refusal.clone(), })?; + cache.pull_all_backing_stores()?; cache.put( "app", &Settings { @@ -5643,6 +6716,7 @@ mod tests { ["settings"], )?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; + cache.pull_all_backing_stores()?; cache.put( "app", @@ -5671,6 +6745,7 @@ mod tests { let temp = tempfile::tempdir()?; let mut cache = CultCache::new(); cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("a.cc")))?; + cache.pull_all_backing_stores()?; let error = cache .add_generic_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("b.cc"))) .unwrap_err(); @@ -5684,6 +6759,7 @@ mod tests { let temp = tempfile::tempdir()?; let mut cache = CultCache::new(); cache.add_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("a.cc")), ["settings"])?; + cache.pull_all_backing_stores()?; let error = cache .add_backing_store( SingleFileMessagePackBackingStore::new(temp.path().join("b.cc")), @@ -5693,6 +6769,7 @@ mod tests { assert!(error.to_string().contains("already routed"), "{error}"); // A refused registration claims nothing, so "note" is still free. cache.add_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("c.cc")), ["note"])?; + cache.pull_all_backing_stores()?; assert_eq!(cache.stores.len(), 2); Ok(()) } @@ -5705,6 +6782,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; + cache.pull_all_backing_stores()?; cache.put("app", &Settings { theme: "v1-generic".to_string(), retries: 1 })?; let error = cache @@ -5734,15 +6812,17 @@ mod tests { let mut writer = CultCache::new(); writer.register_entry_type::()?; writer.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; + writer.pull_all_backing_stores()?; writer.put("app", &Settings { theme: "stray".to_string(), retries: 0 })?; let generic_bytes = fs::read(&generic_path)?; let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_backing_store(SingleFileMessagePackBackingStore::new(&settings_path), ["settings"])?; - cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; + cache.pull_all_backing_stores()?; // The cache already holds a record, in its home store, before the refused load. cache.put("held", &Settings { theme: "home".to_string(), retries: 0 })?; + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; let before = cache.snapshot(); let settings_bytes = fs::read(&settings_path)?; let error = cache.pull_all_backing_stores().unwrap_err(); @@ -5766,7 +6846,9 @@ mod tests { let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_backing_store(SingleFileMessagePackBackingStore::new(&settings_path), [Settings::TYPE])?; + cache.pull_all_backing_stores()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; + cache.pull_all_backing_stores()?; cache.put("app", &Settings { theme: "before".to_string(), retries: 1 })?; let before = cache.snapshot(); @@ -5808,6 +6890,7 @@ mod tests { let mut homeless = CultCache::new(); homeless.register_registry(TestEntries)?; homeless.add_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("settings.cc")), ["settings"])?; + homeless.pull_all_backing_stores()?; homeless.put("held", &Settings { theme: "held".to_string(), retries: 0 })?; let before = homeless.snapshot(); let error = homeless.load_soa(table).unwrap_err(); @@ -5824,7 +6907,9 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_backing_store(SingleFileMessagePackBackingStore::new(&settings_path), ["settings"])?; + cache.pull_all_backing_stores()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&generic_path))?; + cache.pull_all_backing_stores()?; cache.put("memo", &Note { title: "generic".to_string(), body: "home".to_string() })?; cache.put("old", &Settings { theme: "kept".to_string(), retries: 0 })?; @@ -5855,6 +6940,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put("app", &Settings { theme: "before".to_string(), retries: 1 })?; let before = cache.snapshot(); let store_bytes = fs::read(&store_path)?; @@ -5866,7 +6952,7 @@ mod tests { let (mut empty_stored_at, _) = cache.prepare_entry("n", &Note { title: "t".to_string(), body: "b".to_string() })?; empty_stored_at.stored_at = " ".to_string(); let error = cache.put_prepared_batch(vec![empty_stored_at]).unwrap_err(); - assert!(error.to_string().contains("stored_at for type \"note\" must be non-empty"), "{error}"); + assert!(error.to_string().contains("stored_at for type \"note\" must be an RFC 3339 stamp"), "{error}"); assert_eq!(cache.snapshot(), before); assert_eq!(fs::read(&store_path)?, store_bytes); @@ -5894,6 +6980,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; let updated = cache.update::("app", |current| { let mut current = current.unwrap_or(Settings { @@ -5901,9 +6988,10 @@ mod tests { retries: 0, }); current.retries += 1; - current + Some(current) })?; - assert_eq!(updated.retries, 1); + assert_eq!(updated.outcome, CultUpdateOutcome::Updated); + assert_eq!(updated.document.map(|settings| settings.retries), Some(1)); assert!(cache.delete::("app")?); assert!(cache.get::("app")?.is_none()); Ok(()) @@ -5916,6 +7004,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put( "app", &Settings { @@ -5979,6 +7068,7 @@ mod tests { let mut origin = CultCache::new(); origin.register_entry_type::()?; origin.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&origin_store))?; + origin.pull_all_backing_stores()?; origin.put( "app", &Settings { @@ -5992,6 +7082,7 @@ mod tests { let mut target = CultCache::new(); target.register_entry_type::()?; target.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&target_store))?; + target.pull_all_backing_stores()?; let applied = target.put_envelope::(envelope.clone())?; assert_eq!( @@ -6025,6 +7116,7 @@ mod tests { cache.register_entry_type::()?; cache.register_entry_type::()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put_raw_envelope(envelope.clone())?; assert_eq!(cache.snapshot(), vec![envelope.clone()]); @@ -6060,6 +7152,35 @@ mod tests { Ok(()) } + #[test] + fn put_raw_envelope_refuses_bytes_after_the_value() -> Result<()> { + let temp = tempfile::tempdir()?; + let store_path = temp.path().join("tail.cc"); + let mut cache = CultCache::new(); + cache.register_entry_type::()?; + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; + let tailed = CultCacheEnvelope { + key: "app".to_string(), + r#type: Settings::TYPE.to_string(), + payload: vec![0x91, 0x01, 0xc0], + stored_at: now_utc_second(), + schema_id: None, + }; + let error = format!("{:#}", cache.put_raw_envelope(tailed.clone()).unwrap_err()); + assert!(error.contains("payload has 1 bytes after its first MessagePack value"), "{error}"); + assert!(error.contains("app") && error.contains(Settings::TYPE), "{error}"); + assert!(cache.snapshot().is_empty()); + assert!(!store_path.exists() || SingleFileMessagePackBackingStore::new(&store_path).pull_all()?.is_empty()); + let error = format!("{:#}", cache.put_prepared_batch(vec![tailed.clone()]).unwrap_err()); + assert!(error.contains("payload has 1 bytes after its first MessagePack value"), "{error}"); + assert!(cache.snapshot().is_empty()); + // The same bytes without the tail are admitted: the refusal is the count, not the shape. + cache.put_raw_envelope(CultCacheEnvelope { payload: vec![0x91, 0x01], ..tailed })?; + assert_eq!(cache.snapshot().len(), 1); + Ok(()) + } + #[test] fn soa_snapshot_round_trips_registered_entries() -> Result<()> { let temp = tempfile::tempdir()?; @@ -6067,6 +7188,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put( "app", &Settings { @@ -6104,6 +7226,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_registry(TestEntries)?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put( "settings:a", &Settings { @@ -6186,6 +7309,7 @@ mod tests { let mut cache = CultCache::new(); cache.register_entry_type::()?; cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store_path))?; + cache.pull_all_backing_stores()?; cache.put( "primary", &Settings { @@ -6583,10 +7707,10 @@ mod tests { let mut store = SingleFileMessagePackBackingStore::new(&path); let outcome = match operation { "open" => store.pull_all().map(|_| ()), - "soft_push_all" => store.push_all(&[snapshot_envelope("x", b"one")], PushAllOptions { soft: true }), - "push_all" => store.push_all(&[snapshot_envelope("x", b"one")], PushAllOptions::default()), + "soft_push_all" => store.push_all(&[snapshot_envelope("x", b"one")], PushAllOptions { soft: true }).map(|_| ()), + "push_all" => store.push_all(&[snapshot_envelope("x", b"one")], PushAllOptions::default()).map(|_| ()), "delete" => store.delete(&snapshot_envelope("x", b"one")), - _ => store.push(&snapshot_envelope("x", b"one")), + _ => store.push(&snapshot_envelope("x", b"one")).map(|_| ()), }; let what = format!("{vector} {operation}"); assert_eq!(outcome.is_ok(), reads, "{what}: {outcome:?}"); @@ -6944,7 +8068,14 @@ mod tests { let temp = tempfile::tempdir()?; let path = temp.path().join("store.cc"); std::os::unix::fs::symlink(&path, &path)?; - assert_io_failure_on_every_read(&SingleFileMessagePackBackingStore::new(&path)); + let [pull, snapshot, exchange] = every_read(&SingleFileMessagePackBackingStore::new(&path)); + for error in [pull.unwrap_err(), snapshot.unwrap_err()] { + let io = io_error_in(&error).expect("the failure to reach the store is an io::Error"); + assert_ne!(io.kind(), std::io::ErrorKind::NotFound, "{error:#}"); + assert_eq!(unreadable_kind(&error), None, "{error:#}"); + } + // The exchange is a write: a link at the path is refused before it reads. + assert!(exchange.unwrap_err().downcast_ref::().is_some()); assert!(fs::symlink_metadata(&path)?.file_type().is_symlink()); Ok(()) } @@ -7241,7 +8372,7 @@ mod tests { let mut store = SingleFileMessagePackBackingStore::new(&path); let outcome = match operation { "open" => store.pull_all().map(|_| ()), - _ => store.push_all(&[snapshot_envelope("x", b"one")], PushAllOptions::default()), + _ => store.push_all(&[snapshot_envelope("x", b"one")], PushAllOptions::default()).map(|_| ()), }; let error = outcome.expect_err(&format!("{vector} {operation}")); assert!(unreadable_kind(&error).is_some(), "{vector} {operation}: {error:#}"); @@ -7383,16 +8514,17 @@ mod tests { let path = temp.path().join("store.cc"); std::os::unix::fs::symlink(&target, &path)?; let store = SingleFileMessagePackBackingStore::new(&path); - for result in every_read(&store) { - let error = result.expect_err("a dangling link must not read as an empty store"); + let [pull, snapshot, exchange] = every_read(&store); + for error in [pull.unwrap_err(), snapshot.unwrap_err()] { assert!(io_error_in(&error).is_some(), "{error:#}"); assert_eq!(unreadable_kind(&error), None, "{error:#}"); } - // Every merging writer refuses on its own read, so none can replace the link. + assert!(exchange.unwrap_err().downcast_ref::().is_some()); + // Every writer refuses the link before its lock, so none can replace it. let mut writer = SingleFileMessagePackBackingStore::new(&path); let delta = snapshot_envelope("delta", b"4"); let writes = [ - writer.push(&delta), + writer.push(&delta).map(|_| ()), writer.delete(&delta), writer.insert_entry_if_absent(delta.clone()).map(|_| ()), writer.compare_and_swap_batch(&[], vec![delta.clone()]).map(|_| ()), @@ -7400,7 +8532,7 @@ mod tests { ]; for (index, result) in writes.into_iter().enumerate() { let error = result.expect_err("a merging write through a dangling link must refuse"); - assert!(io_error_in(&error).is_some(), "write {index}: {error:#}"); + assert!(error.downcast_ref::().is_some(), "write {index}: {error:#}"); assert_eq!(write_failure_kind(&error), None, "write {index}: {error:#}"); } assert!(fs::symlink_metadata(&path)?.file_type().is_symlink()); @@ -7526,11 +8658,21 @@ mod tests { with_write_fault(WriteStep::Unlock, || store.push(&pushed))?; assert!(with_write_fault(WriteStep::Unlock, || store.pull_all())?.contains(&pushed)); - // The lock was released: another writer takes it at once. - assert!(matches!( - store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?, - TryCompareExchangeSnapshotOutcome::Exchanged - )); + // The lock was released: another writer takes it. The shared lock of the faulted `pull_all` just above was left to the + // close of its descriptor (see `release_lock`), which frees it only once no other descriptor shares its open file + // description, and a test that spawns a child (a fork-exec duplicates every descriptor of the process until the exec) + // can hold such a duplicate on another test thread for a moment. A faulted exclusive lock cannot contend the try-lock: + // the blocking `pull_all` in its argument waits it out. So a contended try-lock is retried for up to two seconds; a + // lock that was never released stays contended and still fails. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2); + let outcome = loop { + let outcome = store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?; + if !matches!(outcome, TryCompareExchangeSnapshotOutcome::LockContended) || std::time::Instant::now() >= deadline { + break outcome; + } + std::thread::sleep(std::time::Duration::from_millis(5)); + }; + assert!(matches!(outcome, TryCompareExchangeSnapshotOutcome::Exchanged), "{outcome:?}"); // The keyed store releases its lock the same way. let mut keyed = RedbMessagePackBackingStore::new(temp.path().join("keyed.redb"))?; @@ -7539,27 +8681,265 @@ mod tests { Ok(()) } - // What a write does through a symbolic link today. R3 decides whether a write - // resolves the link or refuses it; until then these pin the current behaviour so the - // change is deliberate. Reads refuse a dangling link (see the test above). + // A link put at the store path while a writer waits on the lock is refused once the lock is taken, as C# does, and the + // lock is free afterwards. #[cfg(unix)] #[test] - fn r3_decides_resolve_or_refuse_a_push_through_a_live_link_replaces_the_link() -> Result<()> { + fn a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it() -> Result<()> { let temp = tempfile::tempdir()?; let volume = temp.path().join("volume"); fs::create_dir(&volume)?; let target = volume.join("store.cc"); SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; let before = fs::read(&target)?; - let link = temp.path().join("store.cc"); + let dangling = volume.join("missing.cc"); + let path = temp.path().join("late.cc"); + let writers: [(&str, fn(&mut SingleFileMessagePackBackingStore) -> Result<()>); 4] = [ + ("push", |s| s.push(&snapshot_envelope("b", b"2")).map(|_| ())), + ("push_all", |s| s.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default()).map(|_| ())), + ("delete", |s| s.delete(&snapshot_envelope("a", b"1"))), + ("compare_exchange_snapshot", |s| s.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).map(|_| ())), + ]; + for (name, write, link_to) in writers.into_iter().flat_map(|(n, w)| [(n, w, &target), (n, w, &dangling)]) { + let _ = fs::remove_file(&path); + let store = SingleFileMessagePackBackingStore::new(&path); + let holder = store.open_lock_file()?; + fs2::FileExt::lock_exclusive(&holder)?; + let mut waiting = SingleFileMessagePackBackingStore::new(&path); + let handle = std::thread::spawn(move || write(&mut waiting)); + std::thread::sleep(std::time::Duration::from_millis(300)); + std::os::unix::fs::symlink(link_to, &path)?; + fs2::FileExt::unlock(&holder)?; + let error = handle.join().expect("writer thread").expect_err("the late link must be refused"); + assert!(error.downcast_ref::().is_some(), "{name}: {error:#}"); + assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "{name}: the link was replaced"); + assert_eq!(fs::read(&target)?, before, "{name}: the target changed"); + assert!(!dangling.exists(), "{name}: the dangling target was created"); + let probe = store.open_lock_file()?; + fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("{name}: lock still held: {e}"))?; + } + Ok(()) + } + + fn busy_error() -> std::io::Error { + std::io::Error::from_raw_os_error(32) + } + + fn is_busy(error: &std::io::Error) -> bool { + error.raw_os_error() == Some(32) + } + + #[test] + fn the_lock_open_waits_through_busy_failures_then_opens() { + let (mut tries, mut sleeps) = (0, 0); + let opened = retry_while_busy( + || { + tries += 1; + if tries <= 4 { Err(busy_error()) } else { Ok(tries) } + }, + is_busy, + || sleeps += 1, + ); + assert_eq!(opened.unwrap(), 5); + assert_eq!(sleeps, 4); + } + + #[test] + fn the_lock_open_surfaces_a_non_busy_error_at_once() { + let (mut tries, mut sleeps) = (0, 0); + let error = retry_while_busy::<()>( + || { + tries += 1; + Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied)) + }, + is_busy, + || sleeps += 1, + ) + .expect_err("a non-busy error must surface"); + assert_eq!(error.kind(), std::io::ErrorKind::PermissionDenied); + assert_eq!((tries, sleeps), (1, 0)); + } + + #[test] + fn the_try_open_maps_a_sharing_violation_to_none_without_retrying() { + let mut tries = 0; + let opened = open_unless_busy::<()>( + || { + tries += 1; + Err(busy_error()) + }, + is_busy, + ); + assert!(opened.unwrap().is_none()); + assert_eq!(tries, 1); + let other = open_unless_busy::<()>(|| Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied)), is_busy); + assert!(other.is_err()); + } + + // The try entry must answer at once while another handle holds the lock. The attempt runs on a thread and its + // answer is awaited with a timeout, so an entry that waits fails this test by timeout instead of hanging the suite. + #[cfg(unix)] + #[test] + fn the_try_entry_returns_at_once_while_the_lock_is_held() -> Result<()> { + let temp = tempfile::tempdir()?; + let path = temp.path().join("held.cc"); + let store = SingleFileMessagePackBackingStore::new(&path); + let holder = store.open_lock_file()?; + fs2::FileExt::lock_exclusive(&holder)?; + let (sender, receiver) = std::sync::mpsc::channel(); + let attempt = SingleFileMessagePackBackingStore::new(&path); + std::thread::spawn(move || { + let _ = sender.send(attempt.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")])); + }); + let outcome = receiver + .recv_timeout(std::time::Duration::from_secs(3)) + .map_err(|_| anyhow!("the try entry waited on a held lock instead of returning"))?; + assert!(matches!(outcome?, TryCompareExchangeSnapshotOutcome::LockContended)); + assert!(!path.exists(), "the contended attempt wrote"); + let again = store.open_lock_file()?; + assert!(fs2::FileExt::try_lock_exclusive(&again).is_err(), "the held lock was disturbed"); + fs2::FileExt::unlock(&holder)?; + Ok(()) + } + + // Off Windows errno 32 is EPIPE, never a busy file: the predicate is false and the wait surfaces it on the first try. + #[cfg(not(windows))] + #[test] + fn error_32_is_not_busy_off_windows_and_surfaces_at_once() { + assert!(!is_sharing_violation(&busy_error())); + let (mut tries, mut sleeps) = (0, 0); + let error = retry_while_busy::<()>( + || { + tries += 1; + Err(busy_error()) + }, + is_sharing_violation, + || sleeps += 1, + ) + .expect_err("error 32 must surface off Windows"); + assert_eq!(error.raw_os_error(), Some(32)); + assert_eq!((tries, sleeps), (1, 0)); + } + + // The try entry has no wait, so the late window is the gap between its look and its lock: a link put there is refused + // once the lock is held, nothing is written through or over it, and the lock is released. + #[cfg(unix)] + #[test] + fn a_link_put_between_the_try_entrys_look_and_its_lock_is_refused_and_the_lock_is_released() -> Result<()> { + let temp = tempfile::tempdir()?; + let target = temp.path().join("target.cc"); + SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; + let before = fs::read(&target)?; + let dangling = temp.path().join("missing.cc"); + let path = temp.path().join("late.cc"); + for link_to in [target.clone(), dangling.clone()] { + let store = SingleFileMessagePackBackingStore::new(&path); + let (link, at) = (link_to.clone(), path.clone()); + AFTER_PRE_LOCK_LOOK.with(|hook| { + *hook.borrow_mut() = Some(Box::new(move || std::os::unix::fs::symlink(&link, &at).expect("put the link"))); + }); + let error = store + .try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]) + .expect_err("the late link must be refused"); + assert!(error.downcast_ref::().is_some(), "{error:#}"); + assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "the link was replaced"); + assert_eq!(fs::read(&target)?, before, "the target changed"); + assert!(!dangling.exists(), "the dangling target was created"); + let probe = store.open_lock_file()?; + fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("lock still held: {e}"))?; + drop(probe); + fs::remove_file(&path)?; + } + Ok(()) + } + + // A write at a store path that is a link is refused: a replace there would swap the link for a regular file and + // detach the store from its target. Reads through a live link keep working. + #[cfg(unix)] + #[test] + fn a_write_at_a_live_link_is_refused_and_nothing_changes_while_a_read_through_it_works() -> Result<()> { + let temp = tempfile::tempdir()?; + let volume = temp.path().join("volume"); + fs::create_dir(&volume)?; + let target = volume.join("store.cc"); + SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; + let before = fs::read(&target)?; + let volume_before = dir_names(&volume)?; + let link = temp.path().join("link.cc"); std::os::unix::fs::symlink(&target, &link)?; - SingleFileMessagePackBackingStore::new(&link).push(&snapshot_envelope("b", b"2"))?; - assert!(fs::symlink_metadata(&link)?.file_type().is_file(), "the link was replaced by a regular file"); - assert_eq!(fs::read(&target)?, before, "the store the link pointed to is unchanged"); + let mut store = SingleFileMessagePackBackingStore::new(&link); + let errors = [ + store.push(&snapshot_envelope("b", b"2")).unwrap_err(), + store.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default()).unwrap_err(), + store.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions { soft: true }).unwrap_err(), + store.delete(&snapshot_envelope("a", b"1")).unwrap_err(), + store.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).unwrap_err(), + store.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).err().expect("refused"), + ]; + for error in &errors { + let linked = error.downcast_ref::().unwrap_or_else(|| panic!("{error:#}")); + assert_eq!(linked.path, link); + assert!(error.to_string().contains(&link.display().to_string()), "{error:#}"); + } + assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was"); + assert_eq!(fs::read(&target)?, before); + assert_eq!(dir_names(&volume)?, volume_before, "no lock or staging file beside the target"); + assert_eq!(dir_names(temp.path())?, ["link.cc", "volume"], "no lock or staging file beside the link"); + assert_eq!(store.pull_all()?, vec![snapshot_envelope("a", b"1")]); Ok(()) } + // The advisory query is the held record's and the cache's judges; a link at the store path is the write's to refuse. + #[cfg(unix)] + #[test] + fn is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused() -> Result<()> { + let temp = tempfile::tempdir()?; + let target = temp.path().join("store.cc"); + store_with(&target, |cache| cache.put("held", &settings_one()).map(drop))?; + let before = fs::read(&target)?; + let link = temp.path().join("link.cc"); + std::os::unix::fs::symlink(&target, &link)?; + + let mut cache = CultCache::new(); + cache.register_registry(TestEntries)?; + cache.add_backing_store(SingleFileMessagePackBackingStore::new(&link), [Settings::TYPE])?; + cache.pull_all_backing_stores()?; + let names_before = dir_names(temp.path())?; + + assert!(cache.is_writable(Settings::TYPE, "held")?, "the query does not look at the store path"); + assert!(cache.is_writable(Settings::TYPE, "fresh")?, "the query does not look at the store path"); + let error = cache.put("fresh", &settings_one()).unwrap_err(); + let linked = error.downcast_ref::().unwrap_or_else(|| panic!("{error:#}")); + assert_eq!(linked.path, link); + assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was"); + assert_eq!(fs::read(&target)?, before); + assert_eq!(dir_names(temp.path())?, names_before, "the refused write left no lock or staging file"); + Ok(()) + } + + #[cfg(unix)] + #[test] + fn a_write_at_a_live_directory_link_is_refused_as_a_link() -> Result<()> { + let temp = tempfile::tempdir()?; + let real = temp.path().join("real"); + fs::create_dir(&real)?; + let link = temp.path().join("dirlink"); + std::os::unix::fs::symlink(&real, &link)?; + let error = SingleFileMessagePackBackingStore::new(&link).push(&snapshot_envelope("a", b"1")).unwrap_err(); + assert!(error.downcast_ref::().is_some(), "{error:#}"); + assert!(fs::symlink_metadata(&link)?.file_type().is_symlink()); + assert_eq!(fs::read_dir(&real)?.count(), 0); + Ok(()) + } + + #[cfg(unix)] + fn dir_names(dir: &Path) -> Result> { + let mut names = fs::read_dir(dir)?.map(|entry| Ok(entry?.file_name().to_string_lossy().into_owned())).collect::>>()?; + names.sort(); + Ok(names) + } + // The two cases the vectors cannot hold: nothing at the path is written, and a dangling link is refused and left as it is. #[test] fn a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link() -> Result<()> { @@ -7581,7 +8961,7 @@ mod tests { let error = SingleFileMessagePackBackingStore::new(&link) .push_all(&[snapshot_envelope("a", b"1")], PushAllOptions { soft: true }) .unwrap_err(); - assert!(io_not_found(&error), "{error:#}"); + assert!(error.downcast_ref::().is_some(), "{error:#}"); assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link was replaced"); assert_eq!(fs::read_dir(&volume)?.count(), 0, "nothing reached the link's volume"); } @@ -7590,15 +8970,17 @@ mod tests { #[cfg(unix)] #[test] - fn r3_decides_resolve_or_refuse_a_push_all_through_a_dangling_link_replaces_the_link() -> Result<()> { + fn a_push_all_through_a_dangling_link_is_refused_like_push_and_writes_nothing() -> Result<()> { let temp = tempfile::tempdir()?; let volume = temp.path().join("volume"); fs::create_dir(&volume)?; let link = temp.path().join("store.cc"); std::os::unix::fs::symlink(volume.join("store.cc"), &link)?; - SingleFileMessagePackBackingStore::new(&link).push_all(&[snapshot_envelope("a", b"1")], PushAllOptions::default())?; - assert!(fs::symlink_metadata(&link)?.file_type().is_file(), "the link was replaced by a regular file"); + let mut store = SingleFileMessagePackBackingStore::new(&link); + assert!(store.push_all(&[snapshot_envelope("a", b"1")], PushAllOptions::default()).is_err()); + assert!(store.push(&snapshot_envelope("a", b"1")).is_err()); + assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was"); assert_eq!(fs::read_dir(&volume)?.count(), 0, "nothing reached the link's volume"); Ok(()) } @@ -7714,4 +9096,176 @@ mod tests { assert_eq!(fs::read(&path)?, before); Ok(()) } -} \ No newline at end of file + + /// A store already holding `Settings:"held"`, then a fresh cache attached to it that has not pulled. + fn unpulled_cache_over_populated_stores() -> Result<(tempfile::TempDir, Vec, Vec)> { + let temp = tempfile::tempdir()?; + let single = temp.path().join("single.cc"); + let redb = temp.path().join("redb.db"); + let held = Settings { theme: "held".to_string(), retries: 1 }; + let mut seeder = CultCache::new(); + seeder.register_registry(TestEntries)?; + seeder.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&single))?; + seeder.pull_all_backing_stores()?; + seeder.put("held", &held)?; + let mut seeder = CultCache::new(); + seeder.register_registry(TestEntries)?; + seeder.add_generic_backing_store(RedbMessagePackBackingStore::new(&redb)?)?; + seeder.pull_all_backing_stores()?; + seeder.put("held", &held)?; + + let mut over_single = CultCache::new(); + over_single.register_registry(TestEntries)?; + over_single.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&single))?; + let mut over_redb = CultCache::new(); + over_redb.register_registry(TestEntries)?; + over_redb.add_generic_backing_store(RedbMessagePackBackingStore::new(&redb)?)?; + Ok((temp, vec![over_single, over_redb], vec![single, redb])) + } + + #[test] + fn a_cache_that_never_pulled_refuses_every_write_and_removal() -> Result<()> { + let (_temp, caches, _paths) = unpulled_cache_over_populated_stores()?; + for mut cache in caches { + let fresh = Note { title: "t".to_string(), body: "b".to_string() }; + let (staged, _) = cache.prepare_entry("fresh", &fresh)?; + for error in [ + cache.put_prepared_batch(vec![staged.clone()]).unwrap_err(), + cache.put("fresh", &fresh).unwrap_err(), + cache.put_raw_envelope(staged.clone()).unwrap_err(), + cache.delete::("held").unwrap_err(), + ] { + assert!(error.downcast_ref::().is_some(), "{error:#}"); + } + assert!(cache.snapshot().is_empty()); + } + Ok(()) + } + + #[test] + fn a_batch_from_a_cache_that_never_pulled_cannot_wipe_the_store() -> Result<()> { + let (_temp, caches, paths) = unpulled_cache_over_populated_stores()?; + for (mut cache, path) in caches.into_iter().zip(paths) { + let (staged, _) = cache.prepare_entry("fresh", &Note { title: "t".to_string(), body: "b".to_string() })?; + let _ = cache.put_prepared_batch(vec![staged]); + let mut reader = CultCache::new(); + reader.register_registry(TestEntries)?; + if path.extension().is_some_and(|extension| extension == "db") { + reader.add_generic_backing_store(RedbMessagePackBackingStore::new(&path)?)?; + reader.pull_all_backing_stores()?; + } else { + reader.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path))?; + reader.pull_all_backing_stores()?; + } + reader.pull_all_backing_stores()?; + assert_eq!(reader.get_required::("held")?.theme, "held", "{path:?}"); + } + Ok(()) + } + + #[test] + fn pulling_lifts_the_gate_and_a_store_attached_later_closes_it_again() -> Result<()> { + let (temp, mut caches, _paths) = unpulled_cache_over_populated_stores()?; + let mut cache = caches.remove(0); + cache.pull_all_backing_stores()?; + cache.put("fresh", &Note { title: "t".to_string(), body: "b".to_string() })?; + assert_eq!(cache.get_required::("held")?.theme, "held"); + + let mut routed = CultCache::new(); + routed.register_registry(TestEntries)?; + let settings = Settings { theme: "t".to_string(), retries: 0 }; + routed.add_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("routed.cc")), [Settings::TYPE])?; + routed.pull_all_backing_stores()?; + routed.put("fresh", &settings)?; + routed.add_generic_backing_store(SingleFileMessagePackBackingStore::new(temp.path().join("other.cc")))?; + assert!(routed.put("again", &settings).unwrap_err().downcast_ref::().is_some()); + routed.pull_all_backing_stores()?; + routed.put("again", &settings)?; + Ok(()) + } + + #[test] + fn a_stale_cache_batch_keeps_the_records_another_writer_added() -> Result<()> { + let temp = tempfile::tempdir()?; + let single = temp.path().join("single.cc"); + let redb = temp.path().join("redb.db"); + let open = |path: &Path| -> Result { + let mut cache = CultCache::new(); + cache.register_registry(TestEntries)?; + if path.extension().is_some_and(|extension| extension == "db") { + cache.add_generic_backing_store(RedbMessagePackBackingStore::new(path)?)?; + } else { + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(path))?; + } + cache.pull_all_backing_stores()?; + Ok(cache) + }; + for path in [&single, &redb] { + let mut seed = open(path)?; + seed.put("held", &Settings { theme: "held".to_string(), retries: 1 })?; + let mut stale = open(path)?; + let mut other = open(path)?; + other.put("held", &Settings { theme: "other-newer".to_string(), retries: 3 })?; + other.put("added", &Settings { theme: "other".to_string(), retries: 2 })?; + let (staged, _) = stale.prepare_entry("mine", &Note { title: "t".to_string(), body: "b".to_string() })?; + stale.put_prepared_batch(vec![staged])?; + let reader = open(path)?; + assert_eq!(reader.get_required::("added")?.theme, "other", "{path:?}"); + assert_eq!(reader.get_required::("held")?.theme, "other-newer", "a record another writer updated is not reverted"); + assert_eq!(reader.get_required::("mine")?.title, "t", "{path:?}"); + } + Ok(()) + } + + #[test] + fn every_store_merges_push_all_and_lands_the_last_of_a_repeated_key() -> Result<()> { + let temp = tempfile::tempdir()?; + let stores: Vec> = vec![ + Box::new(SingleFileMessagePackBackingStore::new(temp.path().join("single.cc"))), + Box::new(RedbMessagePackBackingStore::new(temp.path().join("redb.db"))?), + Box::new(OwnedRedbMessagePackBackingStore::new(temp.path().join("owned.db"))?), + ]; + for mut store in stores { + store.push(&test_envelope("pushall.merge", "kept", b"kept"))?; + store.push_all(&[test_envelope("pushall.merge", "new", b"new")], PushAllOptions::default())?; + let keys: Vec<_> = store.pull_all()?.into_iter().map(|entry| entry.key).collect(); + assert_eq!(keys, ["kept", "new"]); + } + // A batch naming a key twice lands the last, in the single-file and the redb stores alike. + for path in [temp.path().join("dup.cc"), temp.path().join("dup.db")] { + let mut cache = CultCache::new(); + cache.register_registry(TestEntries)?; + if path.extension().is_some_and(|extension| extension == "db") { + cache.add_generic_backing_store(RedbMessagePackBackingStore::new(&path)?)?; + } else { + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path))?; + } + cache.pull_all_backing_stores()?; + let (first, _) = cache.prepare_entry("k", &Settings { theme: "first".to_string(), retries: 1 })?; + let (last, _) = cache.prepare_entry("k", &Settings { theme: "last".to_string(), retries: 2 })?; + cache.put_prepared_batch(vec![first, last])?; + assert_eq!(cache.get_required::("k")?.theme, "last"); + } + Ok(()) + } + + #[test] + fn push_all_replaces_a_record_held_under_another_schema_id_as_push_does() -> Result<()> { + let temp = tempfile::tempdir()?; + let mut store = SingleFileMessagePackBackingStore::new(temp.path().join("store.cc")); + let mut held = snapshot_envelope("k", b"old"); + held.schema_id = Some("legacy.id".to_string()); + store.push(&held)?; + let mut replacement = snapshot_envelope("k", b"new"); + replacement.schema_id = None; + let mut by_push = store.clone(); + by_push.push(&replacement)?; + assert_eq!(store.pull_all()?.len(), 1); + store.push(&held)?; + store.push_all(&[replacement.clone()], PushAllOptions::default())?; + let held_now = store.pull_all()?; + assert_eq!(held_now.len(), 1, "one record under the key"); + assert_eq!(held_now[0].payload, replacement.payload); + Ok(()) + } +} diff --git a/packages/cultcache-rs/tests/common/mod.rs b/packages/cultcache-rs/tests/common/mod.rs new file mode 100644 index 00000000..3f893c0f --- /dev/null +++ b/packages/cultcache-rs/tests/common/mod.rs @@ -0,0 +1,235 @@ +//! The union-parity vectors' types and constructed value, shared by the test files that walk them. +#![allow(dead_code)] + +use std::path::PathBuf; + +use cultcache_rs::CultSlots; +use cultcache_rs::CultUnion; +use cultcache_rs::DatabaseEntry; + + +#[derive(Clone, Debug, PartialEq, CultUnion)] +pub enum Node { + #[cultcache(key = 0)] + Box(BoxArm), + #[cultcache(key = 1)] + Text(TextArm), + #[cultcache(key = 2)] + Use(UseArm), + #[cultcache(key = 5)] + Image(ImageArm), + #[cultcache(key = 6)] + Wrap(WrapArm), +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct WrapArm { + #[cultcache(key = 0)] + pub inner: Option>, + #[cultcache(key = 1)] + pub id: String, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct BoxArm { + #[cultcache(key = 0)] + pub id: String, + #[cultcache(key = 1)] + pub children: Vec, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct TextArm { + #[cultcache(key = 0)] + pub id: String, + #[cultcache(key = 1)] + pub content: String, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct UseArm { + #[cultcache(key = 0)] + pub id: String, + #[cultcache(key = 1)] + pub composition: Composition, + #[cultcache(key = 2)] + pub slots: Vec>, +} + +// Slot 2 is retired: a gap written nil. +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct ImageArm { + #[cultcache(key = 0)] + pub id: String, + #[cultcache(key = 1)] + pub uri: String, + #[cultcache(key = 3)] + pub alt: String, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct Composition { + #[cultcache(key = 0)] + pub name: String, + #[cultcache(key = 1)] + pub body: Option>, +} + +#[derive(Clone, Debug, PartialEq, CultUnion)] +pub enum Outcome { + #[cultcache(key = 0)] + Pending(PendingArm), + #[cultcache(key = 1)] + Completed(CompletedArm), + #[cultcache(key = 2)] + Denied(IdArm), + #[cultcache(key = -2147483648)] + Floor(IdArm), + #[cultcache(key = 2147483647)] + Ceiling(IdArm), +} + +// The codec's unit arm (no members, written [key, []]), kept apart from the golden: a document list's arms carry ids. +#[derive(Clone, Debug, PartialEq, CultUnion)] +pub enum UnitOutcome { + #[cultcache(key = 2)] + Denied, + #[cultcache(key = -2147483648)] + Floor, + #[cultcache(key = 2147483647)] + Ceiling, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct PendingArm { + #[cultcache(key = 0)] + pub message: String, + #[cultcache(key = 1)] + pub id: String, +} + +// An arm with no members of its own: just its element id. +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct IdArm { + #[cultcache(key = 0)] + pub id: String, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct CompletedArm { + #[cultcache(key = 0)] + pub source_version: i32, + #[cultcache(key = 1)] + pub id: String, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct NamedArm { + #[cultcache(key = 0)] + pub name: String, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct UsesArm { + #[cultcache(key = 0)] + pub name: String, + #[cultcache(key = 1)] + pub uses: i32, +} + +#[derive(Clone, Debug, PartialEq, CultSlots)] +pub struct CountArm { + #[cultcache(key = 0)] + pub name: String, + #[cultcache(key = 1)] + pub count: i32, +} + +#[derive(Clone, Debug, PartialEq, CultUnion)] +pub enum Item { + #[cultcache(key = 0)] + Gem(NamedArm), + #[cultcache(key = 1)] + Tool(UsesArm), + #[cultcache(key = 2)] + Part(CountArm), +} + +// The same arms as Item under its own keys: key 2 is unknown to it. +#[derive(Clone, Debug, PartialEq, CultUnion)] +pub enum CraftedItem { + #[cultcache(key = 0)] + Tool(UsesArm), + #[cultcache(key = 1)] + Part(CountArm), +} + +#[derive(Clone, Debug, PartialEq, DatabaseEntry)] +#[cultcache(type = "tests.union_surface", schema = "tests.union_surface.v1")] +pub struct UnionSurface { + #[cultcache(key = 0)] + pub id: String, + #[cultcache(key = 1)] + pub version: i32, + #[cultcache(key = 2)] + pub root: Option, + #[cultcache(key = 3)] + pub outcomes: Vec, + #[cultcache(key = 4)] + pub part: Option, + #[cultcache(key = 5)] + pub crafted: Option, + #[cultcache(key = 6)] + pub missing: Option, +} + +pub fn text(id: &str, content: &str) -> Node { + Node::Text(TextArm { id: id.into(), content: content.into() }) +} + +pub fn image(id: &str, uri: &str, alt: &str) -> Node { + Node::Image(ImageArm { id: id.into(), uri: uri.into(), alt: alt.into() }) +} + +pub fn boxed(id: &str, children: Vec) -> Node { + Node::Box(BoxArm { id: id.into(), children }) +} + +pub fn constructed() -> UnionSurface { + let inner = boxed("0000000000d0", vec![text("0000000000d1", "x"), image("0000000000d2", "uri://a", "alt")]); + UnionSurface { + id: "surface".into(), + version: 3, + root: Some(boxed( + "0000000000c0", + vec![ + text("0000000000c1", "hello"), + Node::Use(UseArm { + id: "0000000000c2".into(), + composition: Composition { name: "comp".into(), body: Some(std::boxed::Box::new(inner)) }, + slots: vec![Some(text("0000000000e1", "slot")), None, Some(boxed("0000000000e2", vec![]))], + }), + image("0000000000c3", "uri://b", ""), + ], + )), + outcomes: vec![ + Outcome::Pending(PendingArm { message: "wait".into(), id: "0000000000a1".into() }), + Outcome::Completed(CompletedArm { source_version: 7, id: "0000000000a2".into() }), + Outcome::Denied(id_arm("0000000000a3")), + ], + part: Some(Item::Tool(UsesArm { name: "hammer".into(), uses: 12 })), + crafted: Some(CraftedItem::Part(CountArm { name: "gear".into(), count: 4 })), + missing: None, + } +} + +pub fn id_arm(id: &str) -> IdArm { + IdArm { id: id.into() } +} + +pub fn vector(name: &str) -> Vec { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../tests/vectors/union-parity") + .join(name); + std::fs::read(&path).unwrap_or_else(|error| panic!("{}: {error}", path.display())) +} diff --git a/packages/cultcache-rs/tests/content_version_vector.rs b/packages/cultcache-rs/tests/content_version_vector.rs new file mode 100644 index 00000000..56304b6f --- /dev/null +++ b/packages/cultcache-rs/tests/content_version_vector.rs @@ -0,0 +1,26 @@ +//! `tests/vectors/content-version`, written only by the C# reference: the content version of a stored record is the lowercase +//! hex SHA-256 of its persisted encoding (`variants:ruling:write-set-content-version-declared-ownership`). The store reports +//! the version of every record of the committed `store.cc` as `versions.msgpack` says. + +use std::path::PathBuf; + +use cultcache_rs::{CacheBackingStore, SingleFileMessagePackBackingStore}; + +fn vectors() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../tests/vectors/content-version") +} + +#[test] +fn read_reports_every_version_the_reference_reads() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("store.cc"); + std::fs::copy(vectors().join("store.cc"), &path).unwrap(); + let rows: Vec<(String, String)> = rmp_serde::from_slice(&std::fs::read(vectors().join("versions.msgpack")).unwrap()).unwrap(); + assert!(rows.len() >= 2); + let store = SingleFileMessagePackBackingStore::new(&path); + for (key, version) in rows { + let record = store.read(&key).unwrap().unwrap_or_else(|| panic!("no record {key}")); + assert_eq!(record.version, version, "{key}"); + } + assert_eq!(store.read("absent").unwrap(), None); +} diff --git a/packages/cultcache-rs/tests/first_call.rs b/packages/cultcache-rs/tests/first_call.rs index 6859ae07..c6d8d7f5 100644 --- a/packages/cultcache-rs/tests/first_call.rs +++ b/packages/cultcache-rs/tests/first_call.rs @@ -1,7 +1,9 @@ -//! The Rust CultCache first call from docs/start/README.md. The docs show the region between the -//! `snippet` markers verbatim, and `docs_show_the_tested_snippet` fails when they differ. +//! The Rust CultCache first call and update from docs/start/README.md. The docs show each region between its +//! `snippet` markers verbatim, and `docs_show_the_tested_snippets` fails when they differ. -use cultcache_rs::{CultCache, DatabaseEntry, SingleFileMessagePackBackingStore}; +use std::path::Path; + +use cultcache_rs::{CultCache, CultUpdateResult, DatabaseEntry, SingleFileMessagePackBackingStore}; // snippet:begin #[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] @@ -40,17 +42,59 @@ fn first_call() -> anyhow::Result<()> { } // snippet:end +// snippet:update:begin +fn restock(path: &Path) -> anyhow::Result> { + let mut cache = CultCache::new(); + cache.register_entry_type::()?; + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(path))?; + cache.pull_all_backing_stores()?; + cache.update::("item:potion", |item| { + // The store's copy of the record, or nothing if it has none; this may run again on a lost race. + let mut item = item.unwrap_or(Item { + name: "Potion".to_string(), + category: "Consumable".to_string(), + value: 0, + }); + item.value += 10; + Some(item) + }) +} +// snippet:update:end + #[test] fn first_call_round_trips_through_a_second_cache() { first_call().unwrap(); } #[test] -fn docs_show_the_tested_snippet() { +fn restock_twice_reads_twenty_through_a_second_cache() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("restock.cc"); + assert_eq!(restock(&path).unwrap().outcome, cultcache_rs::CultUpdateOutcome::Updated); + let second = restock(&path).unwrap(); + assert_eq!(second.outcome, cultcache_rs::CultUpdateOutcome::Updated); + assert_eq!(second.document.unwrap().value, 20); + + let mut reopened = CultCache::new(); + reopened.register_entry_type::().unwrap(); + reopened.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + reopened.pull_all_backing_stores().unwrap(); + assert_eq!(reopened.get_required::("item:potion").unwrap().value, 20); +} + +/// The docs show the region between `// snippet:begin` and `end` of this file as one fenced block. +fn docs_show_region(region: &str) { let source = std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/first_call.rs")).unwrap(); - let begin = source.find("// snippet:begin\n").unwrap() + "// snippet:begin\n".len(); - let end = source.find("// snippet:end\n").unwrap(); + let (begin_marker, end_marker) = (format!("// snippet:{region}begin\n"), format!("// snippet:{region}end\n")); + let begin = source.find(&begin_marker).unwrap() + begin_marker.len(); + let end = source.find(&end_marker).unwrap(); let docs = std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/../../docs/start/README.md")).unwrap(); let block = format!("```rust\n{}```", &source[begin..end]); - assert!(docs.contains(&block), "docs/start/README.md no longer shows tests/first_call.rs verbatim"); + assert!(docs.contains(&block), "docs/start/README.md no longer shows the {region:?} region of tests/first_call.rs verbatim"); +} + +#[test] +fn docs_show_the_tested_snippets() { + docs_show_region(""); + docs_show_region("update:"); } diff --git a/packages/cultcache-rs/tests/schema_catalog.rs b/packages/cultcache-rs/tests/schema_catalog.rs new file mode 100644 index 00000000..36ec3bed --- /dev/null +++ b/packages/cultcache-rs/tests/schema_catalog.rs @@ -0,0 +1,165 @@ +//! A consumer of a single-file store reads the schema version a record was published under from the store's catalog, through the +//! public surface only: `pull_all`, `pull_schema_catalog` and `resolve_schema_entry` +//! (`cultlib-gaps:follow_up:rs-snapshot-schema-version`). + +use std::path::PathBuf; + +use cultcache_rs::{ + CacheBackingStore, CultCache, DatabaseEntry, SchemaCatalogEntry, SingleFileMessagePackBackingStore, resolve_schema_entry, +}; + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "tests.sc_one", schema = "tests.sc_one")] +struct One { + #[cultcache(key = 0)] + text: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "tests.sc_two", schema = "tests.sc_two")] +struct Two { + #[cultcache(key = 0)] + text: String, +} + +fn vectors(folder: &str) -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../tests/vectors").join(folder) +} + +/// The version a store publishes for each record, in key order: `(key, schema name, schema version)`. +fn versions(store: &SingleFileMessagePackBackingStore) -> Vec<(String, String, String)> { + let catalog = store.pull_schema_catalog().unwrap(); + let mut rows: Vec<_> = store + .pull_all() + .unwrap() + .iter() + .map(|envelope| { + let id = envelope.schema_id.as_deref().unwrap(); + let entry = resolve_schema_entry(&catalog, id).unwrap_or_else(|| panic!("no entry for {id}")); + (envelope.key.clone(), entry.schema_name.clone(), entry.schema_version.clone()) + }) + .collect(); + rows.sort(); + rows +} + +fn copy(from: PathBuf) -> (tempfile::TempDir, SingleFileMessagePackBackingStore) { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("store.cc"); + std::fs::copy(from, &path).unwrap(); + (dir, SingleFileMessagePackBackingStore::new(&path)) +} + +#[test] +fn a_consumer_reads_the_version_of_each_type_a_rust_store_holds() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("store.cc"); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.register_entry_type::().unwrap(); + cache.add_backing_store(SingleFileMessagePackBackingStore::new(&path), ["tests.sc_one", "tests.sc_two"]).unwrap(); + cache.pull_all_backing_stores().unwrap(); + cache.put("a", &One { text: "1".into() }).unwrap(); + cache.put("b", &Two { text: "2".into() }).unwrap(); + drop(cache); + + let consumer = SingleFileMessagePackBackingStore::new(&path); + assert_eq!( + versions(&consumer), + vec![ + ("a".to_string(), "tests.sc_one".to_string(), "tests.sc_one.v1".to_string()), + ("b".to_string(), "tests.sc_two".to_string(), "tests.sc_two.v1".to_string()), + ] + ); +} + +#[test] +fn an_id_only_an_entry_lists_as_compatible_resolves_to_that_entry() { + let (_dir, store) = copy(vectors("document-variants-c2a/readability/compatible-id-only-v3.bin")); + let catalog = store.pull_schema_catalog().unwrap(); + assert_eq!(catalog.len(), 1); + assert_eq!(catalog[0].schema_id, "vectors.item.next"); + assert_eq!(catalog[0].compatible_schema_ids, vec!["vectors.item".to_string()]); + let entry = resolve_schema_entry(&catalog, "vectors.item").unwrap(); + assert_eq!(entry.schema_id, "vectors.item.next"); + assert_eq!(entry.schema_version, "vectors.item.v1"); + assert_eq!(resolve_schema_entry(&catalog, "vectors.nowhere"), None); +} + +#[test] +fn an_id_an_entry_owns_resolves_to_its_owner_not_to_an_entry_listing_it() { + for vector in ["own-id-over-compatible-v3.bin", "compatible-before-owner-v3.bin"] { + let (_dir, store) = copy(vectors("document-variants-c2a/readability").join(vector)); + let catalog = store.pull_schema_catalog().unwrap(); + assert_eq!(catalog.len(), 2, "{vector}"); + let entry = resolve_schema_entry(&catalog, "vectors.item").unwrap(); + assert_eq!((entry.schema_id.as_str(), entry.schema_version.as_str()), ("vectors.item", "vectors.item.v1"), "{vector}"); + } +} + +#[test] +fn a_legacy_array_and_an_absent_file_publish_no_catalog() { + let (_dir, store) = copy(vectors("document-variants-c2a/readability/legacy-envelopes.bin")); + assert!(!store.pull_all().unwrap().is_empty()); + assert_eq!(store.pull_schema_catalog().unwrap(), Vec::::new()); + + let dir = tempfile::tempdir().unwrap(); + let absent = SingleFileMessagePackBackingStore::new(dir.path().join("none.cc")); + assert_eq!(absent.pull_schema_catalog().unwrap(), Vec::::new()); + assert!(!dir.path().join("none.cc").exists() && !dir.path().join("none.cc.lock").exists(), "reading an absent store creates nothing"); +} + +// `tests/vectors/key-identity/foreign-store.cc` is written by the C# reference: two types, each published with its version. +#[test] +fn a_store_the_csharp_reference_wrote_gives_its_versions() { + let (_dir, store) = copy(vectors("key-identity/foreign-store.cc")); + assert_eq!( + versions(&store), + vec![ + ("j".to_string(), "tests.ki_alpha".to_string(), "tests.ki_alpha.v1".to_string()), + ("k".to_string(), "tests.ki_gamma".to_string(), "tests.ki_gamma.v1".to_string()), + ] + ); + let catalog = store.pull_schema_catalog().unwrap(); + assert_eq!(catalog[0].content_hash, "sha256:02f0c8b7ea07c11705c1365fc29136ada08c8076209cacfb8d5526229461ce7a"); +} + +type Entry = (&'static str, &'static str, &'static str, &'static str, &'static str, Vec<&'static str>, Vec<()>); + +// A hand-built v3 store whose versions are not the `.v1` the writers above publish, so a version read from anywhere but the +// version slot cannot pass. `shared` is listed by two entries (the first lister wins); `id.c` is owned by one entry and listed by +// an earlier one (the owner wins). +#[test] +fn distinct_versions_come_through_with_the_decoders_resolution() { + let catalog: Vec = vec![ + ("id.a", "N.a", "9.9.9", "hash-a", "{}", vec!["id.a", "shared"], vec![]), + ("id.b", "N.b", "7.0", "hash-b", "{}", vec!["id.b", "shared", "id.c"], vec![]), + ("id.c", "N.c", "3", "hash-c", "{}", vec!["id.c"], vec![]), + ]; + let record = |key: &str, id: &str| (key.to_string(), id.to_string(), "2026-11-13T00:00:00Z".to_string(), b"x".to_vec()); + let records = vec![record("r1", "shared"), record("r2", "id.c"), record("r3", "id.b"), record("r4", "id.a")]; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("store.cc"); + std::fs::write(&path, rmp_serde::to_vec(&("cultcache.store.v3", catalog, records)).unwrap()).unwrap(); + + let store = SingleFileMessagePackBackingStore::new(&path); + let listed = store.pull_schema_catalog().unwrap(); + assert_eq!(listed.iter().map(|e| e.schema_version.as_str()).collect::>(), vec!["9.9.9", "7.0", "3"]); + assert_eq!(listed[1].content_hash, "hash-b"); + assert_eq!(listed[1].compatible_schema_ids, vec!["id.b", "shared", "id.c"]); + let got: Vec<(String, String, String)> = { + let mut rows: Vec<_> = store + .pull_all() + .unwrap() + .iter() + .map(|e| { + let entry = resolve_schema_entry(&listed, e.schema_id.as_deref().unwrap()).unwrap(); + (e.key.clone(), entry.schema_name.clone(), entry.schema_version.clone()) + }) + .collect(); + rows.sort(); + rows + }; + let row = |k: &str, n: &str, v: &str| (k.to_string(), n.to_string(), v.to_string()); + assert_eq!(got, vec![row("r1", "N.a", "9.9.9"), row("r2", "N.c", "3"), row("r3", "N.b", "7.0"), row("r4", "N.a", "9.9.9")]); +} diff --git a/packages/cultcache-rs/tests/slot_readonly.rs b/packages/cultcache-rs/tests/slot_readonly.rs new file mode 100644 index 00000000..7851c818 --- /dev/null +++ b/packages/cultcache-rs/tests/slot_readonly.rs @@ -0,0 +1,541 @@ +//! A record whose payload carries unread slots at any depth is read-only: the C# reference's union-parity vectors +//! (README "Extra trailing slot" and "Slot stores") walked through the cache. This runtime has no compatible schema +//! ids, so every mark is unconsented and no `explained` outcome can arise. + +mod common; + +use std::path::Path; + +use common::*; +use cultcache_rs::CultCache; +use cultcache_rs::CultCacheEnvelope; +use cultcache_rs::DatabaseEntry; +use cultcache_rs::SchemaConflictError; +use cultcache_rs::SingleFileMessagePackBackingStore; +use cultcache_rs::UnreadSlots; +use cultcache_rs::decode_payload_marked; + +/// A row of `slot-outcomes.msgpack`: store, key, writable, explained, type, slots, declared, occurrences, flattenRefusedBy. +type SlotRow = (String, String, bool, Option, Option, Option, Option, Option, Option); + +const SLOT_VALUE: &str = "extra-slot"; + +/// `tests.slot_unit`, a document `[Name, Outcome: UnitOutcome]`. +#[derive(Clone, Debug, PartialEq, DatabaseEntry)] +#[cultcache(type = "tests.slot_unit", schema = "tests.slot_unit.v1")] +struct SlotUnit { + #[cultcache(key = 0)] + name: String, + #[cultcache(key = 1)] + outcome: UnitOutcome, +} + +/// Two declared slots, the second defaulted: a payload of one slot is short, not marked. +#[derive(Clone, Debug, PartialEq, DatabaseEntry)] +#[cultcache(type = "tests.shortened", schema = "tests.shortened.v1")] +struct Shortened { + #[cultcache(key = 0)] + id: String, + #[cultcache(key = 1, default)] + note: String, +} + +fn registered() -> CultCache { + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.register_entry_type::().unwrap(); + cache +} + +/// Any MessagePack value a store holds, enough to drop one record from a store and write the rest back. +#[derive(serde::Serialize, serde::Deserialize)] +#[serde(untagged)] +enum Packed { + Nil(()), + Bool(bool), + Int(i64), + Text(String), + Bin(#[serde(with = "serde_bytes")] Vec), + Array(Vec), +} + +/// The committed store without its `marked-variant` record. A variant is a five-slot record, which this runtime's +/// store reader refuses whole (`the_variant_row_...`), so the other rows are walked from a copy without it. +fn without_variant_row(store: &[u8]) -> Vec { + let Packed::Array(mut parts) = rmp_serde::from_slice(store).unwrap() else { panic!("a store is an array") }; + let Packed::Array(records) = parts.last_mut().unwrap() else { panic!("a store ends in its records") }; + let before = records.len(); + records.retain(|record| !matches!(record, Packed::Array(fields) if matches!(fields.first(), Some(Packed::Text(key)) if key == "marked-variant"))); + assert_eq!(records.len(), before - 1); + rmp_serde::to_vec(&parts).unwrap() +} + +/// A copy of a committed store, opened with both types registered; the vector itself is never written. The slot store +/// is read without its variant row. +fn open_copy(file: &str, dir: &Path) -> (CultCache, std::path::PathBuf) { + let path = dir.join(file); + let bytes = if file == "extra-slot-store.cc" { without_variant_row(&vector(file)) } else { vector(file) }; + std::fs::write(&path, bytes).unwrap(); + let mut cache = registered(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + cache.pull_all_backing_stores().unwrap(); + (cache, path) +} + +fn envelope(key: &str, payload: Vec) -> CultCacheEnvelope { + CultCacheEnvelope { + key: key.into(), + r#type: UnionSurface::TYPE.into(), + payload, + stored_at: "2026-01-01T00:00:00Z".into(), + schema_id: None, + } +} + +fn conflict(error: &anyhow::Error) -> &SchemaConflictError { + error.downcast_ref::().unwrap_or_else(|| panic!("not a schema conflict: {error:#}")) +} + +fn mark(type_name: &str, slot_count: usize, declared_count: usize, occurrences: usize) -> UnreadSlots { + UnreadSlots { type_name: type_name.into(), slot_count, declared_count, occurrences } +} + +/// The marks the README table pins, by `detail`. The reference names an arm by its class; here an arm is the +/// `...Arm` struct the union wraps. +fn expected_mark(detail: &str) -> UnreadSlots { + match detail { + "record" | "depth" => mark("UnionSurface", 8, 7, 1), + "nested-record" => mark("Composition", 3, 2, 1), + "arm" => mark("BoxArm", 3, 2, 1), + "list-element" => mark("TextArm", 3, 2, 1), + other => panic!("unknown detail {other}"), + } +} + +fn assert_refused_with(error: &anyhow::Error, key: &str, expected: &UnreadSlots) { + let refusal = conflict(error); + assert_eq!(refusal.unread_slots.as_ref(), Some(expected)); + assert_eq!(refusal.record_key, key); + assert_eq!(refusal.schema_names.len(), 1); + let message = error.to_string(); + assert!(message.contains(&expected.type_name), "{message}"); + assert!(message.contains(&expected.slot_count.to_string()) && message.contains(&expected.declared_count.to_string()), "{message}"); + assert!(!message.contains(SLOT_VALUE), "a refusal echoed the slot value: {message}"); +} + +#[test] +fn every_read_only_vector_decodes_marked_and_is_not_writable() { + let rows: Vec<(String, String, String, Option, String)> = rmp_serde::from_slice(&vector("outcomes.msgpack")).unwrap(); + assert_eq!(rows.len(), 9); + let bounds: Vec = rmp_serde::from_slice(&vector("bounds-outcomes.msgpack")).unwrap(); + let mut cases: Vec<(String, String)> = rows + .into_iter() + .filter(|(_, outcome, ..)| outcome == "read-only") + .map(|(file, _, _, _, detail)| (file, detail)) + .collect(); + for entry in &bounds[1..] { + let fields: Vec<&str> = entry.as_array().unwrap().iter().filter_map(|field| field.as_str()).collect(); + if fields[1] == "read-only" { + cases.push((fields[0].to_string(), fields[3].to_string())); + } + } + assert_eq!(cases.len(), 5); + for (file, detail) in cases { + let payload = vector(&file); + let (value, found) = decode_payload_marked::(&payload).unwrap_or_else(|error| panic!("{file}: {error:#}")); + assert_eq!(value, constructed(), "{file}"); + assert_eq!(found, Some(expected_mark(&detail)), "{file}"); + + let mut cache = registered(); + let dir = tempfile::tempdir().unwrap(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(dir.path().join("s.cc"))).unwrap(); + cache.pull_all_backing_stores().unwrap(); + cache.put_raw_envelope(envelope("k", payload.clone())).unwrap(); + assert_eq!(cache.get::("k").unwrap(), Some(constructed()), "{file}"); + assert!(!cache.is_writable(UnionSurface::TYPE, "k").unwrap(), "{file}"); + assert!(cache.is_writable(UnionSurface::TYPE, "unheld").unwrap(), "{file}"); + let error = cache.put("k", &constructed()).unwrap_err(); + assert_refused_with(&error, "k", &expected_mark(&detail)); + } +} + +#[test] +fn clean_and_short_payloads_are_writable_and_unmarked() { + let (value, found) = decode_payload_marked::(&vector("union-surface.msgpack")).unwrap(); + assert_eq!((value, found), (constructed(), None)); + // A payload short of its declared slots keeps its defaults and does not mark. + let (value, found) = decode_payload_marked::(&rmp_serde::to_vec(&("only",)).unwrap()).unwrap(); + assert_eq!((value.id.as_str(), value.note.as_str(), found), ("only", "", None)); +} + +#[test] +fn a_payload_past_the_nesting_limit_is_refused_never_marked() { + for file in ["depth-skipped-past-limit.msgpack", "depth-past-limit.msgpack"] { + let error = decode_payload_marked::(&vector(file)).unwrap_err(); + assert!(error.to_string().contains("the limit is 256"), "{file}: {error:#}"); + } +} + +#[test] +fn a_plain_serde_decode_marks_nothing_and_does_not_leak_a_scope() { + // No scope is open: the visitors note into nothing. + let value: UnionSurface = rmp_serde::from_slice(&vector("extra-slot-record.msgpack")).unwrap(); + assert_eq!(value, constructed()); + // A marked decode followed by a clean one: the first mark does not survive into the second. + assert!(decode_payload_marked::(&vector("extra-slot-record.msgpack")).unwrap().1.is_some()); + assert_eq!(decode_payload_marked::(&vector("union-surface.msgpack")).unwrap().1, None); +} + +#[test] +fn the_slot_store_rows_walk_through_the_cache() { + let rows: Vec = + rmp_serde::from_slice(&vector("slot-outcomes.msgpack")).unwrap(); + assert_eq!(rows.len(), 8); + let dir = tempfile::tempdir().unwrap(); + let mut walked = 0; + for (store, key, writable, _explained, type_name, slots, declared, occurrences, flatten_refused_by) in rows { + // The variant row is a record this runtime cannot hold: see the variant test. The consent store's `writable` and + // `explained` answer the compatible-id declaration, which this runtime has none of; its marks are walked below. + if flatten_refused_by.is_some() || store == "extra-slot-consent.cc" { + continue; + } + let case = dir.path().join(&key); + std::fs::create_dir_all(&case).unwrap(); + let (mut cache, _) = open_copy(&store, &case); + let type_id = if key == "unit-arm" { SlotUnit::TYPE } else { UnionSurface::TYPE }; + assert_eq!(cache.is_writable(type_id, &key).unwrap(), writable, "{store} {key}"); + let removal = if key == "unit-arm" { cache.delete::(&key) } else { cache.delete::(&key) }; + match writable { + true => assert!(removal.unwrap()), + false => { + let found = conflict(&removal.unwrap_err()).unread_slots.clone().unwrap(); + // The reference names the unit arm's class, Denied2; here the variant. + let expected_type = match type_name.as_deref().unwrap() { + "Denied2" => "Denied", + other => other, + }; + assert_eq!((found.type_name.as_str(), Some(found.slot_count), Some(found.declared_count), Some(found.occurrences)), (expected_type, slots, declared, occurrences)); + assert!(cache.snapshot_envelopes().iter().any(|held| held.key == key), "a refused removal removed {key}"); + } + } + walked += 1; + } + assert_eq!(walked, 3); +} + +#[test] +fn the_consent_store_marks_are_unconsented_here() { + let rows: Vec = + rmp_serde::from_slice(&vector("slot-outcomes.msgpack")).unwrap(); + let dir = tempfile::tempdir().unwrap(); + let mut found = 0; + for (store, key, _writable, _explained, _type_name, slots, declared, occurrences, _) in rows { + if store != "extra-slot-consent.cc" { + continue; + } + let (cache, _) = open_copy(&store, dir.path()); + let held = cache.snapshot_envelopes().into_iter().find(|held| held.key == key).unwrap(); + let (_, mark) = decode_payload_marked::(&held.payload).unwrap(); + match slots { + // The reference's `consented` row is writable there (a listed compatible id covers it); here every mark refuses. + None => assert_eq!(mark, None, "{key}"), + Some(slots) => { + let mark = mark.unwrap_or_else(|| panic!("{key} carried no mark")); + // The first type read is the document, whose reference name is SurfaceDeclaring. + assert_eq!(mark.type_name, "UnionSurface", "{key}"); + assert_eq!((Some(mark.slot_count), Some(mark.declared_count), Some(mark.occurrences)), (Some(slots), declared, occurrences), "{key}"); + assert!(!cache.is_writable(UnionSurface::TYPE, &key).unwrap(), "{key}"); + } + } + found += 1; + } + assert_eq!(found, 4); +} + +#[test] +fn every_write_entry_refuses_a_marked_key_before_any_store_is_touched() { + let dir = tempfile::tempdir().unwrap(); + let (mut cache, path) = open_copy("extra-slot-store.cc", dir.path()); + let expected = mark("Composition", 3, 2, 1); + let before = std::fs::read(&path).unwrap(); + let held = cache.snapshot_envelopes().into_iter().find(|held| held.key == "marked").unwrap(); + let clean = cache.snapshot_envelopes().into_iter().find(|held| held.key == "clean").unwrap(); + assert!(!cache.is_writable(UnionSurface::TYPE, "marked").unwrap()); + assert!(cache.is_writable(UnionSurface::TYPE, "clean").unwrap()); + + let replacing = envelope("marked", rmp_serde::to_vec(&constructed()).unwrap()); + assert_refused_with(&cache.put("marked", &constructed()).unwrap_err(), "marked", &expected); + assert_refused_with(&cache.put_envelope::(replacing.clone()).unwrap_err(), "marked", &expected); + assert_refused_with(&cache.put_raw_envelope(replacing.clone()).unwrap_err(), "marked", &expected); + assert_refused_with(&cache.put_prepared_batch(vec![replacing]).unwrap_err(), "marked", &expected); + assert_refused_with(&cache.delete::("marked").unwrap_err(), "marked", &expected); + assert_refused_with( + &cache.update::("marked", |current| current).unwrap_err(), + "marked", + &expected, + ); + + // A batch with a marked key refuses whole: its clean member is not written either. + let mut rewritten = clean.clone(); + rewritten.payload = rmp_serde::to_vec(&UnionSurface { version: 9, ..constructed() }).unwrap(); + let batch = vec![rewritten, envelope("marked", rmp_serde::to_vec(&constructed()).unwrap())]; + assert_refused_with(&cache.put_prepared_batch(batch).unwrap_err(), "marked", &expected); + + assert_eq!(std::fs::read(&path).unwrap(), before, "a refused write touched the store"); + assert_eq!(cache.snapshot_envelopes().into_iter().find(|entry| entry.key == "marked").unwrap(), held); + assert_eq!(cache.snapshot_envelopes().into_iter().find(|entry| entry.key == "clean").unwrap(), clean); + + // A write to another key succeeds, and the marked record's bytes, schema id and stored-at are carried unchanged. + cache.put("clean", &UnionSurface { version: 9, ..constructed() }).unwrap(); + cache.put("other", &constructed()).unwrap(); + assert!(cache.delete::("other").unwrap()); + let mut reopened = registered(); + reopened.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + reopened.pull_all_backing_stores().unwrap(); + let carried = reopened.snapshot_envelopes().into_iter().find(|entry| entry.key == "marked").unwrap(); + assert_eq!(carried, held); + assert_eq!(carried.payload, vector("extra-slot-nested-record.msgpack")); + assert_eq!(reopened.get::("clean").unwrap().unwrap().version, 9); +} + +#[test] +fn a_unit_arm_with_slots_decodes_as_the_unit_arm_and_marks() { + let (document, found) = decode_payload_marked::(&unit_arm_payload()).unwrap(); + assert_eq!(document, SlotUnit { name: "unit-arm".into(), outcome: UnitOutcome::Denied }); + assert_eq!(found, Some(mark("Denied", 1, 0, 1))); + + let dir = tempfile::tempdir().unwrap(); + let (mut cache, _) = open_copy("extra-slot-store.cc", dir.path()); + assert_eq!(cache.get::("unit-arm").unwrap(), Some(document.clone())); + assert!(!cache.is_writable(SlotUnit::TYPE, "unit-arm").unwrap()); + let error = cache.put("unit-arm", &document).unwrap_err(); + assert_refused_with(&error, "unit-arm", &mark("Denied", 1, 0, 1)); + assert_refused_with(&cache.delete::("unit-arm").unwrap_err(), "unit-arm", &mark("Denied", 1, 0, 1)); + + // Several elements are still one marked object, and a clean unit arm is unmarked. + let (_, found) = decode_payload_marked::(&rmp_serde::to_vec(&(2u64, ("a", "b", "c"))).unwrap()).unwrap(); + assert_eq!(found, Some(mark("Denied", 3, 0, 1))); + let (_, found) = decode_payload_marked::(&vector("unit-arm-denied.msgpack")).unwrap(); + assert_eq!(found, None); +} + +fn unit_arm_payload() -> Vec { + let dir = tempfile::tempdir().unwrap(); + let (cache, _) = open_copy("extra-slot-store.cc", dir.path()); + cache.snapshot_envelopes().into_iter().find(|held| held.key == "unit-arm").unwrap().payload +} + +#[test] +fn the_variant_row_of_the_slot_store_is_refused_whole_by_the_store_reader() { + // `marked-variant` is a five-slot record (a variant): this runtime holds no variants, and its v1 snapshot reader + // refuses such a record, so the store as written cannot load here and the other rows are walked without it. + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("extra-slot-store.cc"); + std::fs::write(&path, vector("extra-slot-store.cc")).unwrap(); + let mut cache = registered(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + let error = cache.pull_all_backing_stores().unwrap_err(); + assert!(format!("{error:#}").contains("is not decodable"), "{error:#}"); +} + +fn slot_unit_envelope(key: &str, payload: Vec) -> CultCacheEnvelope { + CultCacheEnvelope { r#type: SlotUnit::TYPE.into(), ..envelope(key, payload) } +} + +#[test] +fn a_held_payload_the_type_cannot_read_is_read_only_and_refused_by_name() { + let mut cache = registered(); + // A newer writer's record: an arm this build lacks and an extra root slot. get cannot read it, and it is the data + // read-only exists to protect. + let newer = rmp_serde::to_vec(&("CANARY-name", (99u64, ("x",)), "CANARY-extra")).unwrap(); + cache.put_raw_envelope(slot_unit_envelope("n", newer)).unwrap(); + assert!(cache.get::("n").is_err()); + assert!(!cache.is_writable(SlotUnit::TYPE, "n").unwrap()); + let replacement = SlotUnit { name: "n".into(), outcome: UnitOutcome::Denied }; + let held = cache.snapshot_envelopes(); + for error in [ + cache.put("n", &replacement).unwrap_err(), + cache.put_raw_envelope(slot_unit_envelope("n", rmp_serde::to_vec(&replacement).unwrap())).unwrap_err(), + cache.delete::("n").unwrap_err(), + ] { + let message = format!("{error:#}"); + assert!(message.contains("\"n\"") && message.contains("tests.slot_unit") && message.contains("read-only"), "{message}"); + assert!(!message.contains("CANARY"), "{message}"); + } + assert_eq!(cache.snapshot_envelopes(), held); +} + +#[test] +fn a_corrupt_held_payload_is_refused_naming_the_key_never_its_bytes() { + let mut cache = registered(); + cache.put_raw_envelope(envelope("broken", rmp_serde::to_vec(&("CANARY-id", "CANARY-version")).unwrap())).unwrap(); + assert!(!cache.is_writable(UnionSurface::TYPE, "broken").unwrap()); + let error = cache.delete::("broken").unwrap_err(); + let message = format!("{error:#}"); + assert!(message.contains("\"broken\"") && message.contains("backup") && message.contains("store"), "{message}"); + assert!(!message.contains("build that wrote it"), "{message}"); + assert!(!message.contains("CANARY"), "{message}"); + assert!(cache.is_writable("tests.unregistered", "broken").is_err()); +} + +fn held_marked_and_clean() -> (CultCache, tempfile::TempDir, std::path::PathBuf) { + let dir = tempfile::tempdir().unwrap(); + let (cache, path) = open_copy("extra-slot-store.cc", dir.path()); + (cache, dir, path) +} + +#[test] +fn load_soa_does_not_replace_or_drop_a_marked_record() { + use cultcache_rs::CultCacheSoaTable; + let (mut cache, _dir, path) = held_marked_and_clean(); + let all = cache.snapshot_envelopes(); + let held = all.iter().find(|held| held.key == "marked").unwrap().clone(); + let without: Vec<_> = all.iter().filter(|held| held.key != "marked").cloned().collect(); + let mut replaced = all.clone(); + replaced.iter_mut().find(|held| held.key == "marked").unwrap().payload = rmp_serde::to_vec(&constructed()).unwrap(); + for table in [without, replaced] { + let error = cache.load_soa(CultCacheSoaTable::from_envelopes(table)).unwrap_err(); + assert_refused_with(&error, "marked", &mark("Composition", 3, 2, 1)); + assert_eq!(cache.snapshot_envelopes(), all, "a refused load changed the image"); + } + // The same record carried unchanged, with another key added, loads. + let mut kept = all.clone(); + kept.push(envelope("added", rmp_serde::to_vec(&constructed()).unwrap())); + cache.load_soa(CultCacheSoaTable::from_envelopes(kept)).unwrap(); + assert_eq!(cache.snapshot_envelopes().iter().find(|held| held.key == "marked"), Some(&held)); + // A clean record may be dropped by a load. + let no_clean: Vec<_> = cache.snapshot_envelopes().into_iter().filter(|held| held.key != "clean").collect(); + cache.load_soa(CultCacheSoaTable::from_envelopes(no_clean)).unwrap(); + // The marked record is still on disk after the next batch. + let rewrite = envelope("added", rmp_serde::to_vec(&UnionSurface { version: 5, ..constructed() }).unwrap()); + cache.put_prepared_batch(vec![rewrite]).unwrap(); + let mut reopened = registered(); + reopened.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + reopened.pull_all_backing_stores().unwrap(); + assert_eq!(reopened.snapshot_envelopes().iter().find(|held| held.key == "marked"), Some(&held)); +} + +#[test] +fn is_writable_agrees_with_the_writes_for_foreign_keys_and_other_types() { + let dir = tempfile::tempdir().unwrap(); + // Only UnionSurface registered: the store's unit-arm record is foreign here. + let path = dir.path().join("extra-slot-store.cc"); + std::fs::write(&path, without_variant_row(&vector("extra-slot-store.cc"))).unwrap(); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + cache.pull_all_backing_stores().unwrap(); + assert!(cache.put("unit-arm", &constructed()).is_err()); + assert!(!cache.is_writable(UnionSurface::TYPE, "unit-arm").unwrap()); + assert!(cache.is_writable(UnionSurface::TYPE, "unheld").unwrap()); + + // Another registered type at a key a record of this one holds. + let (mut cache, _dir, _) = held_marked_and_clean(); + let other = SlotUnit { name: "clean".into(), outcome: UnitOutcome::Denied }; + assert!(cache.put("clean", &other).is_err()); + assert!(!cache.is_writable(SlotUnit::TYPE, "clean").unwrap()); + assert!(!cache.is_writable(SlotUnit::TYPE, "marked").unwrap()); + assert!(cache.is_writable(UnionSurface::TYPE, "clean").unwrap()); +} + +#[test] +fn a_marked_key_refuses_a_batch_wherever_it_sits() { + let (mut cache, _dir, path) = held_marked_and_clean(); + let before = std::fs::read(&path).unwrap(); + let marked = envelope("marked", rmp_serde::to_vec(&constructed()).unwrap()); + let clean = |key: &str| envelope(key, rmp_serde::to_vec(&constructed()).unwrap()); + for batch in [ + vec![marked.clone(), clean("a"), clean("b")], + vec![clean("a"), marked.clone(), clean("b")], + vec![clean("a"), clean("b"), marked.clone()], + ] { + let error = cache.put_prepared_batch(batch).unwrap_err(); + assert_refused_with(&error, "marked", &mark("Composition", 3, 2, 1)); + assert_eq!(std::fs::read(&path).unwrap(), before); + assert!(cache.snapshot_envelopes().iter().all(|held| held.key != "a" && held.key != "b")); + } +} + +#[test] +fn a_marked_decode_leaves_no_scope_open() { + assert_eq!(cultcache_rs::__enter_slots(), 0); + decode_payload_marked::(&vector("extra-slot-record.msgpack")).unwrap(); + assert_eq!(cultcache_rs::__enter_slots(), 0, "a marked decode left its scope open on this thread"); + let _ = decode_payload_marked::(&vector("depth-skipped-past-limit.msgpack")); + assert_eq!(cultcache_rs::__enter_slots(), 0); +} + +#[test] +fn is_writable_is_false_for_a_type_with_no_home_store() { + let dir = tempfile::tempdir().unwrap(); + let mut cache = registered(); + // One store, routed to SlotUnit only: UnionSurface has no home. + cache.add_backing_store(SingleFileMessagePackBackingStore::new(dir.path().join("s.cc")), [SlotUnit::TYPE]).unwrap(); + cache.pull_all_backing_stores().unwrap(); + assert!(cache.put("k", &constructed()).is_err()); + assert!(!cache.is_writable(UnionSurface::TYPE, "k").unwrap()); + assert!(cache.is_writable(SlotUnit::TYPE, "k").unwrap()); +} + +#[test] +fn load_soa_refuses_a_same_length_replacement_of_a_marked_record() { + use cultcache_rs::CultCacheSoaTable; + let (mut cache, _dir, _) = held_marked_and_clean(); + let all = cache.snapshot_envelopes(); + let marked = all.iter().find(|held| held.key == "marked").unwrap(); + // One value (a bin16) of the marked record's length: same length, different bytes. A value followed by padding + // would be refused at admission as bytes after the value, before the read-only check is reached. + let length = marked.payload.len(); + let mut same_length = vec![0u8; length]; + same_length[..3].copy_from_slice(&[0xc5, ((length - 3) >> 8) as u8, (length - 3) as u8]); + assert_eq!(same_length.len(), marked.payload.len()); + let mut replaced = all.clone(); + replaced.iter_mut().find(|held| held.key == "marked").unwrap().payload = same_length; + let error = cache.load_soa(CultCacheSoaTable::from_envelopes(replaced)).unwrap_err(); + assert_refused_with(&error, "marked", &mark("Composition", 3, 2, 1)); + assert_eq!(cache.snapshot_envelopes(), all); +} + +// The payload check runs before the mark scope opens: a tail on a slot-bearing payload is refused by count, never marked. +#[test] +fn a_tail_on_a_slot_payload_is_refused_before_any_mark() { + let mut tailed = vector("extra-slot-record.msgpack"); + tailed.push(0xc0); + let error = decode_payload_marked::(&tailed).unwrap_err(); + assert!(format!("{error:#}").contains("payload has 1 bytes after its first MessagePack value"), "{error:#}"); + // Nothing leaked: the next marked decode finds exactly its own mark. + let found = decode_payload_marked::(&vector("extra-slot-record.msgpack")).unwrap().1.unwrap(); + assert_eq!(found.occurrences, 1); +} + +// A store holding marked records and one with a tail is refused at load; the cache then refuses every write with the load +// refusal (not a per-key mark conflict), and the store file is untouched. +#[test] +fn a_store_with_marked_records_and_a_tail_is_refused_and_its_cache_writes_nothing() { + use cultcache_rs::CacheBackingStore; + use cultcache_rs::CultCacheLoadRefused; + let (mut held, _dir, path) = held_marked_and_clean(); + let mut tailed = vector("union-surface.msgpack"); + tailed.push(0xc0); + SingleFileMessagePackBackingStore::new(&path).push(&envelope("tailed", tailed)).unwrap(); + let before = std::fs::read(&path).unwrap(); + // The cache that held the store re-reads it and is refused. + assert!(format!("{:#}", held.pull_all_backing_stores().unwrap_err()).contains("bytes after")); + let fresh_value = constructed(); + let mut fresh = registered(); + fresh.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + assert!(fresh.pull_all_backing_stores().is_err()); + for cache in [&mut held, &mut fresh] { + let staged = cache.prepare_entry("new", &fresh_value).unwrap().0; + let errors = [ + cache.put("new", &fresh_value).unwrap_err(), + cache.put_prepared_batch(vec![staged.clone()]).unwrap_err(), + cache.put_raw_envelope(staged).unwrap_err(), + cache.delete::("new").unwrap_err(), + ]; + for error in errors { + assert!(error.downcast_ref::().is_some(), "{error:#}"); + } + } + assert_eq!(std::fs::read(&path).unwrap(), before); +} diff --git a/packages/cultcache-rs/tests/stored_at_stamp_vector.rs b/packages/cultcache-rs/tests/stored_at_stamp_vector.rs new file mode 100644 index 00000000..d1ab7f9a --- /dev/null +++ b/packages/cultcache-rs/tests/stored_at_stamp_vector.rs @@ -0,0 +1,58 @@ +//! `tests/vectors/stored-at-stamp`: the one shape a peer's `storedAt` must parse as. Every row is walked against +//! `is_stored_at_stamp`, and each row that is no stamp against the cache's two raw doors, `put_raw_envelope` and +//! `put_prepared_batch`: refused, nothing stored or minted, the store still writable after +//! (`variants:ruling:raw-put-without-stamp-refused`). + +use std::path::PathBuf; + +use cultcache_rs::{CacheBackingStore, CultCache, DatabaseEntry, SingleFileMessagePackBackingStore, is_stored_at_stamp}; + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "tests.stamp_note", schema = "tests.stamp_note")] +struct Note { + #[cultcache(key = 0)] + text: String, +} + +fn rows() -> Vec<(String, bool)> { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../tests/vectors/stored-at-stamp/stamps.msgpack"); + rmp_serde::from_slice(&std::fs::read(path).unwrap()).unwrap() +} + +#[test] +fn the_stamp_check_agrees_with_every_vector_row() { + let rows = rows(); + assert!(rows.len() > 30, "the vector was not read"); + for (text, parses) in rows { + assert_eq!(is_stored_at_stamp(&text), parses, "{text:?}"); + } +} + +#[test] +fn a_raw_envelope_that_is_no_stamp_is_refused_and_nothing_is_stored() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("stamps.cc"); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + cache.pull_all_backing_stores().unwrap(); + for (text, parses) in rows() { + if parses { + continue; + } + let (mut entry, _) = cache.prepare_entry("a", &Note { text: "peer".into() }).unwrap(); + entry.stored_at = text.clone(); + let error = cache.put_raw_envelope(entry.clone()).unwrap_err(); + assert!(format!("{error:#}").contains("stored_at"), "{text:?}: {error:#}"); + let error = cache.put_prepared_batch(vec![entry]).unwrap_err(); + assert!(format!("{error:#}").contains("stored_at"), "{text:?}: {error:#}"); + assert!(cache.get::("a").unwrap().is_none(), "{text:?} was stored"); + assert!(!path.exists() || SingleFileMessagePackBackingStore::new(&path).pull_all().unwrap().is_empty(), "{text:?} reached the file"); + } + // Refused writes left the store writable: a stamped one lands under the stamp it carries. + let (mut entry, _) = cache.prepare_entry("b", &Note { text: "stamped".into() }).unwrap(); + entry.stored_at = "2026-11-07T00:00:00Z".into(); + cache.put_raw_envelope(entry).unwrap(); + assert_eq!(cache.get::("b").unwrap().unwrap().text, "stamped"); + assert!(cache.get::("a").unwrap().is_none()); +} diff --git a/packages/cultcache-rs/tests/union_parity.rs b/packages/cultcache-rs/tests/union_parity.rs index 24fe999a..81a0da7a 100644 --- a/packages/cultcache-rs/tests/union_parity.rs +++ b/packages/cultcache-rs/tests/union_parity.rs @@ -1,244 +1,21 @@ //! The Rust runtime held to tests/vectors/union-parity, written by the C# reference. //! Bytes and outcomes are read from the committed files; only the constructed value is spelled here. -use std::path::PathBuf; - use cultcache_rs::CultCache; use cultcache_rs::CacheBackingStore; use cultcache_rs::CultCacheEnvelope; +use cultcache_rs::CultCacheLoadRefused; +use cultcache_rs::CultCacheSoaTable; use cultcache_rs::MAX_PAYLOAD_DEPTH; use cultcache_rs::PayloadTooDeep; use cultcache_rs::check_payload_depth; -use cultcache_rs::CultSlots; -use cultcache_rs::CultUnion; +use cultcache_rs::decode_payload; +use cultcache_rs::RedbMessagePackBackingStore; use cultcache_rs::DatabaseEntry; use cultcache_rs::SingleFileMessagePackBackingStore; -#[derive(Clone, Debug, PartialEq, CultUnion)] -enum Node { - #[cultcache(key = 0)] - Box(BoxArm), - #[cultcache(key = 1)] - Text(TextArm), - #[cultcache(key = 2)] - Use(UseArm), - #[cultcache(key = 5)] - Image(ImageArm), - #[cultcache(key = 6)] - Wrap(WrapArm), -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct WrapArm { - #[cultcache(key = 0)] - inner: Option>, - #[cultcache(key = 1)] - id: String, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct BoxArm { - #[cultcache(key = 0)] - id: String, - #[cultcache(key = 1)] - children: Vec, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct TextArm { - #[cultcache(key = 0)] - id: String, - #[cultcache(key = 1)] - content: String, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct UseArm { - #[cultcache(key = 0)] - id: String, - #[cultcache(key = 1)] - composition: Composition, - #[cultcache(key = 2)] - slots: Vec>, -} - -// Slot 2 is retired: a gap written nil. -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct ImageArm { - #[cultcache(key = 0)] - id: String, - #[cultcache(key = 1)] - uri: String, - #[cultcache(key = 3)] - alt: String, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct Composition { - #[cultcache(key = 0)] - name: String, - #[cultcache(key = 1)] - body: Option>, -} - -#[derive(Clone, Debug, PartialEq, CultUnion)] -enum Outcome { - #[cultcache(key = 0)] - Pending(PendingArm), - #[cultcache(key = 1)] - Completed(CompletedArm), - #[cultcache(key = 2)] - Denied(IdArm), - #[cultcache(key = -2147483648)] - Floor(IdArm), - #[cultcache(key = 2147483647)] - Ceiling(IdArm), -} - -// The codec's unit arm (no members, written [key, []]), kept apart from the golden: a document list's arms carry ids. -#[derive(Clone, Debug, PartialEq, CultUnion)] -enum UnitOutcome { - #[cultcache(key = 2)] - Denied, - #[cultcache(key = -2147483648)] - Floor, - #[cultcache(key = 2147483647)] - Ceiling, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct PendingArm { - #[cultcache(key = 0)] - message: String, - #[cultcache(key = 1)] - id: String, -} - -// An arm with no members of its own: just its element id. -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct IdArm { - #[cultcache(key = 0)] - id: String, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct CompletedArm { - #[cultcache(key = 0)] - source_version: i32, - #[cultcache(key = 1)] - id: String, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct NamedArm { - #[cultcache(key = 0)] - name: String, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct UsesArm { - #[cultcache(key = 0)] - name: String, - #[cultcache(key = 1)] - uses: i32, -} - -#[derive(Clone, Debug, PartialEq, CultSlots)] -struct CountArm { - #[cultcache(key = 0)] - name: String, - #[cultcache(key = 1)] - count: i32, -} - -#[derive(Clone, Debug, PartialEq, CultUnion)] -enum Item { - #[cultcache(key = 0)] - Gem(NamedArm), - #[cultcache(key = 1)] - Tool(UsesArm), - #[cultcache(key = 2)] - Part(CountArm), -} - -// The same arms as Item under its own keys: key 2 is unknown to it. -#[derive(Clone, Debug, PartialEq, CultUnion)] -enum CraftedItem { - #[cultcache(key = 0)] - Tool(UsesArm), - #[cultcache(key = 1)] - Part(CountArm), -} - -#[derive(Clone, Debug, PartialEq, DatabaseEntry)] -#[cultcache(type = "tests.union_surface", schema = "tests.union_surface.v1")] -struct UnionSurface { - #[cultcache(key = 0)] - id: String, - #[cultcache(key = 1)] - version: i32, - #[cultcache(key = 2)] - root: Option, - #[cultcache(key = 3)] - outcomes: Vec, - #[cultcache(key = 4)] - part: Option, - #[cultcache(key = 5)] - crafted: Option, - #[cultcache(key = 6)] - missing: Option, -} - -fn text(id: &str, content: &str) -> Node { - Node::Text(TextArm { id: id.into(), content: content.into() }) -} - -fn image(id: &str, uri: &str, alt: &str) -> Node { - Node::Image(ImageArm { id: id.into(), uri: uri.into(), alt: alt.into() }) -} - -fn boxed(id: &str, children: Vec) -> Node { - Node::Box(BoxArm { id: id.into(), children }) -} - -fn constructed() -> UnionSurface { - let inner = boxed("0000000000d0", vec![text("0000000000d1", "x"), image("0000000000d2", "uri://a", "alt")]); - UnionSurface { - id: "surface".into(), - version: 3, - root: Some(boxed( - "0000000000c0", - vec![ - text("0000000000c1", "hello"), - Node::Use(UseArm { - id: "0000000000c2".into(), - composition: Composition { name: "comp".into(), body: Some(std::boxed::Box::new(inner)) }, - slots: vec![Some(text("0000000000e1", "slot")), None, Some(boxed("0000000000e2", vec![]))], - }), - image("0000000000c3", "uri://b", ""), - ], - )), - outcomes: vec![ - Outcome::Pending(PendingArm { message: "wait".into(), id: "0000000000a1".into() }), - Outcome::Completed(CompletedArm { source_version: 7, id: "0000000000a2".into() }), - Outcome::Denied(id_arm("0000000000a3")), - ], - part: Some(Item::Tool(UsesArm { name: "hammer".into(), uses: 12 })), - crafted: Some(CraftedItem::Part(CountArm { name: "gear".into(), count: 4 })), - missing: None, - } -} - -fn id_arm(id: &str) -> IdArm { - IdArm { id: id.into() } -} - -fn vector(name: &str) -> Vec { - let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("../../tests/vectors/union-parity") - .join(name); - std::fs::read(&path).unwrap_or_else(|error| panic!("{}: {error}", path.display())) -} +mod common; +use common::*; fn decode(bytes: &[u8]) -> Result { rmp_serde::from_slice(bytes).map_err(|error| error.to_string()) @@ -325,11 +102,10 @@ fn an_unknown_arm_lists_the_declared_keys_and_never_echoes_the_payload() { #[test] fn a_union_that_is_not_a_key_and_slots_pair_is_refused_naming_the_enum() { - let cases: [(&str, Vec); 4] = [ + let cases: [(&str, Vec); 3] = [ ("an empty array", vec![0x90]), ("a trailing element", rmp_serde::to_vec(&(1u64, (7u8,), 0u8)).unwrap()), ("a string key", rmp_serde::to_vec(&("CANARY-key", (7u8,))).unwrap()), - ("a unit arm with slots", rmp_serde::to_vec(&(2u64, ("CANARY-slot",))).unwrap()), ]; for (label, bytes) in cases { let message = rmp_serde::from_slice::(&bytes).unwrap_err().to_string(); @@ -692,7 +468,7 @@ fn envelope(key: &str, payload: &[u8]) -> CultCacheEnvelope { } // A payload past the bound cannot be admitted through the cache, so a store file written behind its -// back stands in for one a peer or an older runtime left. +// back stands in for one a peer or an older runtime left; load or read refuses it. fn through_cache(payload: &[u8]) -> Result { let temp = tempfile::tempdir().unwrap(); let path = temp.path().join("foreign.cc"); @@ -700,7 +476,10 @@ fn through_cache(payload: &[u8]) -> Result { let mut cache = CultCache::new(); cache.register_entry_type::().unwrap(); cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); - cache.pull_all_backing_stores().unwrap(); + // The payload check runs at load, so a payload past the bound is refused there, before any read. + if let Err(error) = cache.pull_all_backing_stores() { + return Err(format!("{error:#}")); + } match cache.get::("k") { Ok(value) => Ok(value.expect("the envelope was stored")), Err(error) => Err(format!("{error:#}")), @@ -757,6 +536,174 @@ fn bounds_vectors() { assert_eq!((accepted, refused, read_only), (4, 7, 1)); } +// A payload is one MessagePack value (bytes-after-outcomes.msgpack): every door refuses a tail by +// its count, names the clause, never the depth limit (the tail is not walked) and never a value. +#[test] +fn bytes_after_vectors() { + let rows: Vec = rmp_serde::from_slice(&vector("bytes-after-outcomes.msgpack")).unwrap(); + assert_eq!(rows.len(), 4); + let surface = vector("union-surface.msgpack"); + for row in &rows { + let fields = row.as_array().unwrap(); + let (file, outcome, union, detail) = (fields[0].as_str().unwrap(), fields[1].as_str().unwrap(), fields[2].as_str().unwrap(), fields[4].as_str().unwrap()); + assert_eq!((outcome, union, detail), ("refused", "UnionSurface", "bytes-after"), "{file}"); + let bytes = vector(file); + assert_eq!(&bytes[..surface.len()], &surface[..], "{file}"); + let clause = format!("payload has {} bytes after its first MessagePack value", bytes.len() - surface.len()); + let check = |door: &str, message: &str| { + assert!(message.contains(&clause), "{file} {door}: {message}"); + assert!(!message.contains("the limit is"), "{file} {door} walked the tail: {message}"); + assert!(!message.contains("CANARY-bytes-after-7d1e"), "{file} {door} echoed the tail: {message}"); + assert!(!message.contains(CANARY), "{file} {door} echoed a value: {message}"); + }; + let error = decode_payload::(&bytes).unwrap_err(); + assert!(error.downcast_ref::().is_none(), "{file}"); + check("decode_payload", &format!("{error:#}")); + // A store written behind the cache's back is refused at load, naming the type, the key and the count. + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("foreign.cc"); + SingleFileMessagePackBackingStore::new(&path).push(&envelope("k", &bytes)).unwrap(); + let mut single = CultCache::new(); + single.register_entry_type::().unwrap(); + single.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + let message = format!("{:#}", single.pull_all_backing_stores().unwrap_err()); + check("single-file load", &message); + assert!(message.contains("\"k\"") && message.contains(UnionSurface::TYPE), "{file}: {message}"); + let redb_path = temp.path().join("foreign.redb"); + RedbMessagePackBackingStore::new(&redb_path).unwrap().push(&envelope("k", &bytes)).unwrap(); + let mut redb = CultCache::new(); + redb.register_entry_type::().unwrap(); + redb.add_generic_backing_store(RedbMessagePackBackingStore::new(&redb_path).unwrap()).unwrap(); + let message = format!("{:#}", redb.pull_all_backing_stores().unwrap_err()); + check("redb load", &message); + assert!(message.contains("\"k\"") && message.contains(UnionSurface::TYPE), "{file}: {message}"); + let mut soa = CultCache::new(); + soa.register_entry_type::().unwrap(); + let message = format!("{:#}", soa.load_soa(CultCacheSoaTable::from_envelopes(vec![envelope("k", &bytes)])).unwrap_err()); + check("load_soa", &message); + assert!(message.contains("\"k\"") && message.contains(UnionSurface::TYPE), "{file}: {message}"); + assert!(soa.snapshot().is_empty(), "{file}"); + // Raw admission refuses and stages nothing. + let mut raw = CultCache::new(); + raw.register_entry_type::().unwrap(); + let message = format!("{:#}", raw.put_raw_envelope(envelope("k", &bytes)).unwrap_err()); + check("put_raw_envelope", &message); + assert!(raw.snapshot().is_empty(), "{file}"); + } +} + +// Depth is found inside the first value, so a tail never hides it: a payload nested past the bound +// with a tail after it is the depth refusal, not bytes after. +#[test] +fn depth_beats_a_tail() { + let mut deep = vec![0x91u8; MAX_PAYLOAD_DEPTH + 1]; + deep.push(0xc0); + deep.push(0xc0); + let error = decode_payload::(&deep).unwrap_err(); + assert_eq!(error.downcast_ref::(), Some(&TOO_DEEP), "{error:#}"); +} + +// Depth is refused where the payload is loaded: a store holding one nested past the bound is not read. +#[test] +fn depth_is_refused_at_load() { + let mut deep = vec![0x91u8; MAX_PAYLOAD_DEPTH + 1]; + deep.push(0xc0); + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("deep.cc"); + SingleFileMessagePackBackingStore::new(&path).push(&envelope("k", &deep)).unwrap(); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + let error = cache.pull_all_backing_stores().unwrap_err(); + assert_eq!(error.downcast_ref::(), Some(&TOO_DEEP), "{error:#}"); + assert!(cache.snapshot().is_empty()); +} + +// load_soa is a load too: a table holding a payload nested past the bound is refused with the typed depth refusal, the +// prior view is kept, and a tail behind the depth is still the depth refusal. +#[test] +fn load_soa_refuses_a_payload_nested_past_the_bound() { + let mut deep = vec![0x91u8; MAX_PAYLOAD_DEPTH + 1]; + deep.push(0xc0); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.load_soa(CultCacheSoaTable::from_envelopes(vec![envelope("kept", &vector("union-surface.msgpack"))])).unwrap(); + let before = cache.snapshot(); + for payload in [deep.clone(), [deep.clone(), vec![0xc0]].concat()] { + let error = cache.load_soa(CultCacheSoaTable::from_envelopes(vec![envelope("k", &payload)])).unwrap_err(); + assert_eq!(error.downcast_ref::(), Some(&TOO_DEEP), "{error:#}"); + assert_eq!(cache.snapshot(), before, "a refused load keeps the prior view"); + } +} + +// A cache whose load was refused refuses every write and removal and leaves the store as it was: a +// later write would otherwise persist the empty image over the records it never read. +#[test] +fn a_refused_load_refuses_every_write() { + let tail = { + let mut bytes = vector("union-surface.msgpack"); + bytes.push(0xc0); + bytes + }; + let good = envelope("good", &vector("union-surface.msgpack")); + let bad = envelope("bad", &tail); + for redb in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join(if redb { "seed.redb" } else { "seed.cc" }); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + if redb { + let mut store = RedbMessagePackBackingStore::new(&path).unwrap(); + store.push(&good).unwrap(); + store.push(&bad).unwrap(); + drop(store); + cache.add_generic_backing_store(RedbMessagePackBackingStore::new(&path).unwrap()).unwrap(); + } else { + let mut store = SingleFileMessagePackBackingStore::new(&path); + store.push(&good).unwrap(); + store.push(&bad).unwrap(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + } + let load = cache.pull_all_backing_stores().unwrap_err(); + assert!(format!("{load:#}").contains("bytes after"), "{load:#}"); + let before = std::fs::read(&path).unwrap(); + let value = constructed(); + let staged = cache.prepare_entry("new", &value).unwrap().0; + let results: Vec<(&str, anyhow::Error)> = vec![ + ("put", cache.put("new", &value).unwrap_err()), + ("put_prepared_batch", cache.put_prepared_batch(vec![staged.clone()]).unwrap_err()), + ("put_envelope", cache.put_envelope::(staged.clone()).unwrap_err()), + ("put_raw_envelope", cache.put_raw_envelope(staged.clone()).unwrap_err()), + ("update", cache.update::("good", |current| Some(current.unwrap_or_else(constructed))).unwrap_err()), + ("delete", cache.delete::("good").unwrap_err()), + ("delete a key the image lacks", cache.delete::("absent").unwrap_err()), + ]; + for (door, error) in results { + let refusal = error.downcast_ref::().unwrap_or_else(|| panic!("{door} redb={redb}: {error:#}")); + assert!(refusal.to_string().contains("bytes after"), "{door}: {refusal}"); + } + assert_eq!(std::fs::read(&path).unwrap(), before, "redb={redb}: the store changed"); + assert!(cache.snapshot().is_empty()); + // Repairing the store and loading again lifts it. + if redb { + RedbMessagePackBackingStore::new(&path).unwrap().delete(&bad).unwrap(); + } else { + SingleFileMessagePackBackingStore::new(&path).delete(&bad).unwrap(); + } + cache.pull_all_backing_stores().unwrap(); + cache.put("new", &value).unwrap(); + } +} + +// A payload that ends early is not bytes after the value: the decoder refuses it, not the check. +#[test] +fn a_truncated_payload_is_the_decoders_not_bytes_after() { + let mut truncated = vec![0x91, 0xd9, 0x40]; + truncated.extend_from_slice(b"short"); + let message = format!("{:#}", decode_payload::(&truncated).unwrap_err()); + assert!(!message.contains("bytes after"), "{message}"); +} + fn wrap_chain(wraps: usize, leaf: Node) -> UnionSurface { let mut node = leaf; for _ in 0..wraps { @@ -788,6 +735,7 @@ fn write_refused_past_limit() { let mut cache = CultCache::new(); cache.register_entry_type::().unwrap(); cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + cache.pull_all_backing_stores().unwrap(); let at_limit = wrap_chain(126, boxed("0000000000f0", vec![])); assert_eq!(cache.put("at", &at_limit).unwrap(), at_limit); @@ -871,6 +819,7 @@ fn the_raw_doors_hold_the_bound_for_arms_maps_and_arrays() { cache.register_entry_type::().unwrap(); cache.register_entry_type::().unwrap(); cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + cache.pull_all_backing_stores().unwrap(); cache.put_raw_envelope(raw(type_name, "at-raw", at.clone())).unwrap(); cache.put_prepared_batch(vec![raw(type_name, "at-batch", at.clone())]).unwrap(); let stored = cache.snapshot_envelopes(); diff --git a/packages/cultcache-rs/tests/unique_index.rs b/packages/cultcache-rs/tests/unique_index.rs index 193ba6fb..1e3ea6a2 100644 --- a/packages/cultcache-rs/tests/unique_index.rs +++ b/packages/cultcache-rs/tests/unique_index.rs @@ -6,7 +6,7 @@ use std::path::{Path, PathBuf}; use cultcache_rs::{ - CultCache, CultCacheEnvelope, CultCacheSoaTable, DatabaseEntry, SingleFileMessagePackBackingStore, + CultCache, CultCacheEnvelope, CultCacheLoadRefused, CultCacheSoaTable, DatabaseEntry, SingleFileMessagePackBackingStore, }; use serde::Deserialize; @@ -221,7 +221,7 @@ fn a_record_rewriting_its_own_key_keeps_its_value_and_a_change_releases_the_old_ assert_eq!(cache.get_by_index::("Sku", "S").unwrap().unwrap().sku, "S"); assert_eq!(cache.get_by_index::("Sku", "T").unwrap().unwrap().sku, "T"); // Updating through `update` is the same rule. - assert!(cache.update::("k", |_| unique("S")).is_err()); + assert!(cache.update::("k", |_| Some(unique("S"))).is_err()); assert_eq!(cache.get::("k").unwrap().unwrap().sku, "T"); } @@ -326,7 +326,7 @@ fn every_write_path_refuses_and_changes_nothing() { assert_eq!(text, refusal(first, second, "Multi", alias)); }; pick(&error_text(cache.put("m:new", clash))); - pick(&error_text(cache.update::("m:new", |_| clash.clone()))); + pick(&error_text(cache.update::("m:new", |_| Some(clash.clone())))); let envelope = staged(&cache, &[("m:new", clash.clone())]).remove(0); pick(&error_text(cache.put_envelope::(envelope.clone()))); pick(&error_text(cache.put_raw_envelope(envelope.clone()))); @@ -419,6 +419,54 @@ fn a_load_of_a_duplicate_image_is_refused_and_keeps_the_prior_view() { soa.put("item:y", &unique("K")).unwrap(); } +// A pull refused for a duplicate unique-index value, or for one key held under two types in two stores, closes the same +// write gate a payload refusal does: the cache holds no image of its stores, so a write would lay the empty image over them. +#[test] +fn a_pull_refused_for_a_duplicate_index_or_a_retype_closes_every_write() { + fn assert_gated(cache: &mut CultCache, path: &Path, door_label: &str) { + let before = std::fs::read(path).unwrap(); + let errors = [ + ("put", cache.put("item:new", &unique("N")).unwrap_err()), + ("put_raw_envelope", cache.put_raw_envelope(cache.prepare_entry("item:new", &unique("N")).unwrap().0).unwrap_err()), + ("delete", cache.delete::("item:keep").unwrap_err()), + ]; + for (door, error) in errors { + assert!(error.downcast_ref::().is_some(), "{door_label} {door}: {error:#}"); + } + assert_eq!(std::fs::read(path).unwrap(), before, "{door_label}: the store changed"); + } + + // Duplicate index value found in the stored image. + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("dup.cc"); + let mut cache = cache_at(&path); + cache.put("item:keep", &unique("K")).unwrap(); + let mut twin = twin_cache_at(&path); + twin.put("item:b", &Twin { sku: "S".into(), category: "c".into() }).unwrap(); + twin.put("item:a", &Twin { sku: "S".into(), category: "c".into() }).unwrap(); + assert!(error_text(cache.pull_all_backing_stores()).contains("both hold unique index 'Sku'")); + assert_gated(&mut cache, &path, "duplicate index"); + + // One key held under two types by two stores. The cache pulled cleanly first, so only the refused pull can close the gate. + let (first, second) = (dir.path().join("first.cc"), dir.path().join("second.cc")); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.register_entry_type::().unwrap(); + cache.register_entry_type::().unwrap(); + cache.add_backing_store(SingleFileMessagePackBackingStore::new(&first), [Plain::TYPE]).unwrap(); + cache.add_backing_store(SingleFileMessagePackBackingStore::new(&second), [Other::TYPE]).unwrap(); + cache.pull_all_backing_stores().unwrap(); + cache.put("k", &Plain { value: "p".into() }).unwrap(); + let mut seed = CultCache::new(); + seed.register_entry_type::().unwrap(); + seed.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&second)).unwrap(); + seed.pull_all_backing_stores().unwrap(); + seed.put("k", &Other { sku: "o".into() }).unwrap(); + assert!(error_text(cache.pull_all_backing_stores()).contains("\"k\"")); + assert_gated(&mut cache, &first, "retype"); + assert_eq!(cache.get::("k").unwrap().unwrap().value, "p", "a refused pull keeps the prior view"); +} + #[test] fn a_multi_index_store_with_a_duplicate_on_the_second_alias_refuses_to_open() { let dir = tempfile::tempdir().unwrap(); diff --git a/packages/cultcache-rs/tests/update.rs b/packages/cultcache-rs/tests/update.rs new file mode 100644 index 00000000..0cc19d95 --- /dev/null +++ b/packages/cultcache-rs/tests/update.rs @@ -0,0 +1,770 @@ +//! `CultCache::update`: read the record as its home store holds it, apply, commit only over what was read, rerun on a lost race +//! (C# `CultCache.Update`, as amended by PR #170). A rival is a second `CultCache` on the same file. + +use std::cell::Cell; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Barrier}; + +use cultcache_rs::{ + CacheBackingStore, CultCache, CultCacheEnvelope, CultCommitIf, CultStoredRecord, CultUpdateOutcome, DatabaseEntry, + PushAllOptions, RedbMessagePackBackingStore, SingleFileMessagePackBackingStore, +}; + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "counter")] +struct Counter { + #[cultcache(key = 0)] + n: u32, +} + +/// The same type with a second slot: a record it writes carries a slot `Counter` does not read. +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "counter")] +struct WideCounter { + #[cultcache(key = 0)] + n: u32, + #[cultcache(key = 1)] + extra: u32, +} + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "note")] +struct Note { + #[cultcache(key = 0)] + text: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)] +#[cultcache(type = "unique_sku")] +struct Sku { + #[cultcache(key = 0, index = "Sku")] + sku: String, +} + +fn counter(n: u32) -> Counter { + Counter { n } +} + +fn open(path: &Path) -> CultCache { + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache.register_entry_type::().unwrap(); + cache.register_entry_type::().unwrap(); + cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(path)).unwrap(); + cache.pull_all_backing_stores().unwrap(); + cache +} + +fn path_in(dir: &tempfile::TempDir) -> PathBuf { + dir.path().join("store.cc") +} + +/// What a fresh cache reads at `key`. +fn stored(path: &Path, key: &str) -> Option { + open(path).get::(key).unwrap().map(|counter| counter.n) +} + +fn file(path: &Path) -> Option> { + std::fs::read(path).ok() +} + +#[test] +fn two_writers_updating_lose_no_increment() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let calls = Arc::new(AtomicUsize::new(0)); + let start = Arc::new(Barrier::new(2)); + let threads: Vec<_> = (0..2) + .map(|_| { + let (path, calls, start) = (path.clone(), calls.clone(), start.clone()); + std::thread::spawn(move || { + let mut cache = open(&path); + start.wait(); + for _ in 0..50 { + let result = cache + .update_with_attempts::("c", 100, |current| { + calls.fetch_add(1, Ordering::SeqCst); + // The window a rival can land in. + std::thread::sleep(std::time::Duration::from_millis(1)); + Some(counter(current.map_or(0, |held| held.n) + 1)) + }) + .unwrap(); + assert_eq!(result.outcome, CultUpdateOutcome::Updated); + } + }) + }) + .collect(); + for thread in threads { + thread.join().unwrap(); + } + assert_eq!(stored(&path, "c"), Some(100)); + assert!(calls.load(Ordering::SeqCst) > 100, "no apply was rerun, so no race was exercised"); +} + +#[test] +fn a_lost_attempt_reruns_apply_on_the_fresh_value() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let mut rival = open(&path); + let mut seen = Vec::new(); + let result = cache + .update::("c", |current| { + let current = current.unwrap().n; + if seen.is_empty() { + rival.put("c", &counter(100)).unwrap(); + } + seen.push(current); + Some(counter(current + 1)) + }) + .unwrap(); + assert_eq!(seen, [1, 100]); + assert_eq!(result.outcome, CultUpdateOutcome::Updated); + assert_eq!(result.document, Some(counter(101))); + assert_eq!(stored(&path, "c"), Some(101)); +} + +#[test] +fn exhausted_attempts_return_lost_race() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let mut rival = open(&path); + let mut calls = 0; + let result = cache + .update_with_attempts::("c", 3, |current| { + calls += 1; + rival.put("c", &counter(100 * calls)).unwrap(); + current + }) + .unwrap(); + assert_eq!(calls, 3); + assert_eq!(result.outcome, CultUpdateOutcome::LostRace); + // The pull before LostRace shows the value that won. + assert_eq!(result.document, Some(counter(300))); + assert_eq!(stored(&path, "c"), Some(300)); +} + +#[test] +fn max_attempts_bounds() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let mut rival = open(&path); + // Always loses; panics past 10 calls so a weakened bound fails here instead of hanging. + let mut losing = |calls: &Cell| { + let n = calls.get() + 1; + calls.set(n); + assert!(n <= 10, "apply ran past the bound"); + rival.put("c", &counter(1000 + n)).unwrap(); + Some(counter(n)) + }; + + let calls = Cell::new(0); + let one = cache.update_with_attempts::("c", 1, |_| losing(&calls)).unwrap(); + assert_eq!((calls.get(), one.outcome), (1, CultUpdateOutcome::LostRace)); + + let calls = Cell::new(0); + assert!(cache.update_with_attempts::("c", 0, |_| losing(&calls)).is_err()); + assert_eq!(calls.get(), 0); + + // The default bound is 5. + let calls = Cell::new(0); + let default = cache.update::("c", |_| losing(&calls)).unwrap(); + assert_eq!((calls.get(), default.outcome), (5, CultUpdateOutcome::LostRace)); +} + +#[test] +fn none_declines_and_writes_nothing() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + let absent = cache.update::("c", |current| { + assert_eq!(current, None); + None + }); + let absent = absent.unwrap(); + assert_eq!((absent.outcome, absent.document), (CultUpdateOutcome::Declined, None)); + assert_eq!(file(&path), None, "a declined update created the store"); + + cache.put("c", &counter(1)).unwrap(); + let before = file(&path); + let held = cache.update::("c", |_| None).unwrap(); + assert_eq!((held.outcome, held.document), (CultUpdateOutcome::Declined, Some(counter(1)))); + assert_eq!(file(&path), before); +} + +#[test] +fn a_declined_update_installs_nothing() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let mut rival = open(&path); + rival.put("c", &counter(7)).unwrap(); + let mut seen = None; + let result = cache + .update::("c", |current| { + seen = current; + None + }) + .unwrap(); + assert_eq!(seen, Some(counter(7)), "apply saw the held copy, not the store's"); + assert_eq!(result.document, Some(counter(1))); + assert_eq!(cache.get::("c").unwrap(), Some(counter(1))); + cache.pull_all_backing_stores().unwrap(); + assert_eq!(cache.get::("c").unwrap(), Some(counter(7))); +} + +#[test] +fn absent_key_rival_removal_and_a_store_holding_nothing() { + // Two writers creating one absent key. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let start = Arc::new(Barrier::new(2)); + let threads: Vec<_> = (0..2) + .map(|_| { + let (path, start) = (path.clone(), start.clone()); + std::thread::spawn(move || { + let mut cache = open(&path); + start.wait(); + let result = cache + .update_with_attempts::("new", 20, |current| { + std::thread::sleep(std::time::Duration::from_millis(1)); + Some(counter(current.map_or(0, |held| held.n) + 1)) + }) + .unwrap(); + assert_eq!(result.outcome, CultUpdateOutcome::Updated); + }) + }) + .collect(); + for thread in threads { + thread.join().unwrap(); + } + assert_eq!(stored(&path, "new"), Some(2)); + + // A rival deletes the record inside apply: the retry sees nothing. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let mut rival = open(&path); + let mut seen = Vec::new(); + let result = cache + .update::("c", |current| { + if seen.is_empty() { + assert!(rival.delete::("c").unwrap()); + } + seen.push(current.clone()); + Some(counter(current.map_or(0, |held| held.n) + 1)) + }) + .unwrap(); + assert_eq!(seen, [Some(counter(1)), None]); + assert_eq!(result.outcome, CultUpdateOutcome::Updated); + assert_eq!(stored(&path, "c"), Some(1)); + + // The cache holds a record the store no longer has: apply is handed None. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(5)).unwrap(); + assert!(open(&path).delete::("c").unwrap()); + let mut seen = Vec::new(); + cache + .update::("c", |current| { + seen.push(current); + Some(counter(1)) + }) + .unwrap(); + assert_eq!(seen, [None]); + assert_eq!(stored(&path, "c"), Some(1)); +} + +#[test] +fn a_rival_writing_and_restoring_the_value_still_reruns_apply() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let mut rival = open(&path); + let mut seen = Vec::new(); + cache + .update::("c", |current| { + let current = current.unwrap().n; + if seen.is_empty() { + rival.put("c", &counter(5)).unwrap(); + rival.put("c", &counter(1)).unwrap(); + } + seen.push(current); + Some(counter(current + 1)) + }) + .unwrap(); + assert_eq!(seen, [1, 1]); + assert_eq!(stored(&path, "c"), Some(2)); +} + +#[test] +fn a_stale_caches_first_attempt_reads_the_store() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut a = open(&path); + a.put("c", &counter(1)).unwrap(); + let mut b = open(&path); + a.put("c", &counter(7)).unwrap(); + let mut seen = Vec::new(); + b.update::("c", |current| { + seen.push(current.clone()); + current.map(|held| counter(held.n + 1)) + }) + .unwrap(); + assert_eq!(seen, [Some(counter(7))]); + assert_eq!(stored(&path, "c"), Some(8)); +} + +/// A custom store over the single-file store that has no `read` or `commit_if`. +struct ReadlessStore(SingleFileMessagePackBackingStore); + +impl CacheBackingStore for ReadlessStore { + fn pull_all(&self) -> anyhow::Result> { + self.0.pull_all() + } + fn push(&mut self, entry: &CultCacheEnvelope) -> anyhow::Result { + self.0.push(entry) + } + fn delete(&mut self, entry: &CultCacheEnvelope) -> anyhow::Result<()> { + self.0.delete(entry) + } + fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> anyhow::Result> { + self.0.push_all(entries, options) + } +} + +/// Runs `update` with an apply that counts its calls; the refusal must come before the first. +fn refused(cache: &mut CultCache, path: &Path, key: &str) -> String { + let before = file(path); + let calls = Cell::new(0); + let error = cache + .update::(key, |current| { + calls.set(calls.get() + 1); + current + }) + .expect_err("the update must be refused"); + assert_eq!(calls.get(), 0, "apply ran before the refusal: {error:#}"); + assert_eq!(file(path), before, "a refused update touched the store"); + format!("{error:#}") +} + +#[test] +fn refusals_come_before_apply() { + // Another type's record written at the key since the pull: the read resolves it. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + open(&path).put("k", &Note { text: "n".into() }).unwrap(); + let text = refused(&mut cache, &path, "k"); + assert!(text.contains("update cannot copy it"), "{text}"); + + // Another type held at the key. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("k", &Note { text: "n".into() }).unwrap(); + refused(&mut cache, &path, "k"); + + // A cache with no store holds another type at the key. + let mut memory = CultCache::new(); + memory.register_entry_type::().unwrap(); + memory.register_entry_type::().unwrap(); + memory.put("k", &Note { text: "n".into() }).unwrap(); + let calls = Cell::new(0); + assert!(memory + .update::("k", |current| { + calls.set(calls.get() + 1); + current + }) + .is_err()); + assert_eq!(calls.get(), 0); + + // A foreign record (no registered type claims it) held at the key, and one that appears after the pull. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + open(&path).put("k", &Note { text: "n".into() }).unwrap(); + let mut only_counter = CultCache::new(); + only_counter.register_entry_type::().unwrap(); + only_counter.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + only_counter.pull_all_backing_stores().unwrap(); + refused(&mut only_counter, &path, "k"); + let mut later = CultCache::new(); + later.register_entry_type::().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + later.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + later.pull_all_backing_stores().unwrap(); + open(&path).put("z", &Note { text: "n".into() }).unwrap(); + let text = refused(&mut later, &path, "z"); + assert!(text.contains("update cannot copy it"), "{text}"); + + // A redb home and a custom store without read. + let dir = tempfile::tempdir().unwrap(); + let redb_path = dir.path().join("store.redb"); + let mut redb = CultCache::new(); + redb.register_entry_type::().unwrap(); + redb.add_generic_backing_store(RedbMessagePackBackingStore::new(&redb_path).unwrap()).unwrap(); + redb.pull_all_backing_stores().unwrap(); + redb.put("c", &counter(1)).unwrap(); + let text = refused(&mut redb, &redb_path, "c"); + assert!(text.contains("read and commit_if"), "{text}"); + let path = path_in(&dir); + let mut custom = CultCache::new(); + custom.register_entry_type::().unwrap(); + custom.add_generic_backing_store(ReadlessStore(SingleFileMessagePackBackingStore::new(&path))).unwrap(); + custom.pull_all_backing_stores().unwrap(); + custom.put("c", &counter(1)).unwrap(); + let text = refused(&mut custom, &path, "c"); + assert!(text.contains("read and commit_if"), "{text}"); + + // A held read-only record, and a store record with slots the cache never loaded. + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut wide = CultCache::new(); + wide.register_entry_type::().unwrap(); + wide.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + wide.pull_all_backing_stores().unwrap(); + wide.put("held", &WideCounter { n: 1, extra: 2 }).unwrap(); + let mut cache = open(&path); + let text = refused(&mut cache, &path, "held"); + assert!(text.contains("held"), "{text}"); + wide.put("unloaded", &WideCounter { n: 1, extra: 2 }).unwrap(); + refused(&mut cache, &path, "unloaded"); + + // An unpulled cache. + let mut unpulled = CultCache::new(); + unpulled.register_entry_type::().unwrap(); + unpulled.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap(); + refused(&mut unpulled, &path, "held"); +} + +#[test] +fn a_refusal_after_apply_is_not_retried() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("a", &Sku { sku: "S".into() }).unwrap(); + cache.put("b", &Sku { sku: "T".into() }).unwrap(); + let before = file(&path); + let calls = Cell::new(0); + let error = cache + .update::("b", |_| { + calls.set(calls.get() + 1); + Some(Sku { sku: "S".into() }) + }) + .expect_err("a unique-index clash"); + assert_eq!(calls.get(), 1, "{error:#}"); + assert_eq!(file(&path), before); +} + +#[test] +fn own_writes_and_the_store_less_cache() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = open(&path); + cache.put("c", &counter(1)).unwrap(); + let calls = Cell::new(0); + for expected in [2, 3] { + let result = cache + .update::("c", |current| { + calls.set(calls.get() + 1); + current.map(|held| counter(held.n + 1)) + }) + .unwrap(); + assert_eq!(result.outcome, CultUpdateOutcome::Updated); + assert_eq!(result.document, Some(counter(expected))); + assert_eq!(result.document, cache.get::("c").unwrap()); + } + assert_eq!(calls.get(), 2); + + let mut memory = CultCache::new(); + memory.register_entry_type::().unwrap(); + let calls = Cell::new(0); + let result = memory + .update::("c", |current| { + calls.set(calls.get() + 1); + Some(counter(current.map_or(0, |held| held.n) + 1)) + }) + .unwrap(); + assert_eq!((calls.get(), result.outcome, result.document), (1, CultUpdateOutcome::Updated, Some(counter(1)))); +} + +#[test] +fn the_ledger_holds_under_contention() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let start = Arc::new(Barrier::new(3)); + let threads: Vec<_> = (0..3) + .map(|_| { + let (path, start) = (path.clone(), start.clone()); + std::thread::spawn(move || { + let mut cache = open(&path); + start.wait(); + (0..25) + .filter(|_| { + let result = cache + .update::("c", |current| Some(counter(current.map_or(0, |held| held.n) + 1))) + .unwrap(); + result.outcome == CultUpdateOutcome::Updated + }) + .count() as u32 + }) + }) + .collect(); + let updated: u32 = threads.into_iter().map(|thread| thread.join().unwrap()).sum(); + assert_eq!(stored(&path, "c"), Some(updated)); +} + +fn envelope_of(cache: &CultCache, key: &str, value: &T, stamp: &str) -> CultCacheEnvelope { + let (mut entry, _) = cache.prepare_entry(key, value).unwrap(); + entry.stored_at = stamp.to_string(); + entry +} + +const FUTURE: &str = "2099-01-01T00:00:00Z"; + +fn later_than_future(stamp: &str) -> bool { + chrono::DateTime::parse_from_rfc3339(stamp).unwrap() > chrono::DateTime::parse_from_rfc3339(FUTURE).unwrap() +} + +#[test] +fn commit_if_writes_only_over_the_version_it_was_given() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let cache = open(&path); + let mut store = SingleFileMessagePackBackingStore::new(&path); + assert_eq!(store.read("c").unwrap(), None); + let one = envelope_of(&cache, "c", &counter(1), FUTURE); + // Against an absent key, expecting a version mismatches; expecting nothing commits. + assert_eq!(store.commit_if(&one, Some("0000")).unwrap(), CultCommitIf::Mismatch); + assert_eq!(file(&path), None); + assert_eq!(store.commit_if(&one, None).unwrap(), CultCommitIf::Committed { stored_at: FUTURE.to_string() }); + let CultStoredRecord { envelope, version } = store.read("c").unwrap().unwrap(); + assert_eq!(envelope.stored_at, FUTURE); + + let before = file(&path); + let two = envelope_of(&cache, "c", &counter(2), FUTURE); + assert_eq!(store.commit_if(&two, None).unwrap(), CultCommitIf::Mismatch); + assert_eq!(store.commit_if(&two, Some("not-the-version")).unwrap(), CultCommitIf::Mismatch); + assert_eq!(file(&path), before, "a mismatch wrote"); + let CultCommitIf::Committed { stored_at } = store.commit_if(&two, Some(&version)).unwrap() else { panic!("the version matches") }; + assert!(later_than_future(&stored_at), "{stored_at}"); + assert_eq!(stored(&path, "c"), Some(2)); +} + +#[test] +fn the_cache_facing_writes_stamp_over_a_replaced_record() { + let seeded = |dir: &tempfile::TempDir| { + let path = path_in(dir); + let seed = open(&path); + let mut store = SingleFileMessagePackBackingStore::new(&path); + store.push(&envelope_of(&seed, "c", &counter(1), FUTURE)).unwrap(); + (path, seed) + }; + let on_disk = |path: &Path| { + let mut store = SingleFileMessagePackBackingStore::new(path); + let stamp = store.read("c").unwrap().unwrap().envelope.stored_at; + let _ = &mut store; + stamp + }; + + // The store: push, push_all and commit_if each return the stamp they wrote, strictly later than what they replaced. + let dir = tempfile::tempdir().unwrap(); + let (path, seed) = seeded(&dir); + let mut store = SingleFileMessagePackBackingStore::new(&path); + let stamp = store.push(&envelope_of(&seed, "c", &counter(2), FUTURE)).unwrap(); + assert!(later_than_future(&stamp) && stamp == on_disk(&path), "push: {stamp}"); + let stamps = store.push_all(&[envelope_of(&seed, "c", &counter(3), FUTURE)], PushAllOptions::default()).unwrap(); + assert!(later_than_future(&stamps[0]) && stamps[0] == on_disk(&path), "push_all: {stamps:?}"); + let version = store.read("c").unwrap().unwrap().version; + let CultCommitIf::Committed { stored_at } = store.commit_if(&envelope_of(&seed, "c", &counter(4), FUTURE), Some(&version)).unwrap() + else { + panic!("matching version") + }; + assert!(later_than_future(&stored_at) && stored_at == on_disk(&path), "commit_if: {stored_at}"); + // One stamp per input, in order, a duplicate getting the stamp of the entry that landed; a soft write that wrote nothing none. + let stamps = store + .push_all( + &[envelope_of(&seed, "c", &counter(5), FUTURE), envelope_of(&seed, "d", &counter(6), FUTURE), envelope_of(&seed, "c", &counter(7), FUTURE)], + PushAllOptions::default(), + ) + .unwrap(); + assert_eq!(stamps.len(), 3); + assert_eq!(stamps[0], stamps[2]); + assert_eq!(stamps[1], FUTURE, "a record replacing nothing keeps its stamp"); + assert_eq!(stamps[0], on_disk(&path)); + assert!(store.push_all(&[envelope_of(&seed, "e", &counter(1), FUTURE)], PushAllOptions { soft: true }).unwrap().is_empty()); + + // The cache holds the stamp the store wrote after every write entry. + type Write = fn(&mut CultCache) -> (); + let writes: [(&str, Write); 5] = [ + ("put", |cache| drop(cache.put("c", &counter(9)).unwrap())), + ("put_envelope", |cache| { + let entry = envelope_of(cache, "c", &counter(9), FUTURE); + drop(cache.put_envelope::(entry).unwrap()) + }), + ("put_raw_envelope", |cache| { + let entry = envelope_of(cache, "c", &counter(9), FUTURE); + cache.put_raw_envelope(entry).unwrap() + }), + ("put_prepared_batch", |cache| { + let entry = envelope_of(cache, "c", &counter(9), FUTURE); + cache.put_prepared_batch(vec![entry]).unwrap() + }), + ("update", |cache| drop(cache.update::("c", |_| Some(counter(9))).unwrap())), + ]; + for (name, write) in writes { + let dir = tempfile::tempdir().unwrap(); + let (path, _) = seeded(&dir); + let mut cache = open(&path); + write(&mut cache); + let held = cache.get_envelope::("c").unwrap().unwrap().stored_at; + assert!(later_than_future(&held), "{name}: {held}"); + assert_eq!(held, on_disk(&path), "{name}: the cache holds a stamp the store did not write"); + } +} + +#[test] +fn the_byte_exact_writers_keep_the_callers_stamp() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let cache = open(&path); + let mut store = SingleFileMessagePackBackingStore::new(&path); + let held = envelope_of(&cache, "c", &counter(1), FUTURE); + store.push(&held).unwrap(); + let swapped = envelope_of(&cache, "c", &counter(2), "2026-01-01T00:00:00Z"); + assert!(store.compare_and_swap_entry(&held, swapped.clone()).unwrap()); + assert_eq!(store.read("c").unwrap().unwrap().envelope, swapped, "compare_and_swap_entry restamped"); + + let redb_path = dir.path().join("store.redb"); + let mut redb = RedbMessagePackBackingStore::new(&redb_path).unwrap(); + redb.push(&held).unwrap(); + let older = envelope_of(&cache, "c", &counter(3), "2026-01-01T00:00:00Z"); + assert_eq!(redb.push(&older).unwrap(), older.stored_at); + assert_eq!(redb.pull_all().unwrap(), vec![older.clone()]); + assert_eq!(redb.push_all(std::slice::from_ref(&held), PushAllOptions::default()).unwrap(), vec![held.stored_at.clone()]); + assert_eq!(redb.pull_all().unwrap(), vec![held]); +} + +/// A single-file store that reports whatever stamp it is told to, as a custom store might. +struct ReportingStore { + inner: SingleFileMessagePackBackingStore, + report: String, +} + +impl CacheBackingStore for ReportingStore { + fn pull_all(&self) -> anyhow::Result> { + self.inner.pull_all() + } + fn push(&mut self, entry: &CultCacheEnvelope) -> anyhow::Result { + self.inner.push(entry)?; + Ok(self.report.clone()) + } + fn delete(&mut self, entry: &CultCacheEnvelope) -> anyhow::Result<()> { + self.inner.delete(entry) + } + fn push_all(&mut self, entries: &[CultCacheEnvelope], options: PushAllOptions) -> anyhow::Result> { + self.inner.push_all(entries, options)?; + Ok(entries.iter().map(|_| self.report.clone()).collect()) + } + fn read(&self, key: &str) -> anyhow::Result> { + self.inner.read(key) + } + fn commit_if(&mut self, entry: &CultCacheEnvelope, expected: Option<&str>) -> anyhow::Result { + Ok(match self.inner.commit_if(entry, expected)? { + CultCommitIf::Committed { .. } => CultCommitIf::Committed { stored_at: self.report.clone() }, + CultCommitIf::Mismatch => CultCommitIf::Mismatch, + }) + } +} + +#[test] +fn the_cache_holds_a_reported_stamp_only_when_it_parses_as_rfc3339() { + type Write = fn(&mut CultCache); + let writes: [(&str, Write); 4] = [ + ("put", |cache| drop(cache.put("c", &counter(1)).unwrap())), + ("put_raw_envelope", |cache| { + let entry = envelope_of(cache, "c", &counter(1), "2020-01-01T00:00:00Z"); + cache.put_raw_envelope(entry).unwrap() + }), + ("put_prepared_batch", |cache| { + let entry = envelope_of(cache, "c", &counter(1), "2020-01-01T00:00:00Z"); + cache.put_prepared_batch(vec![entry]).unwrap() + }), + ("update", |cache| drop(cache.update::("c", |_| Some(counter(1))).unwrap())), + ]; + for (report, parses) in [ + ("2030-01-01T00:00:00Z", true), + ("2030-01-01T00:00:00.123456+00:00", true), + ("not a stamp", false), + ("", false), + ("2030-01-01", false), + ("2030-01-01 00:00:00", false), + ("2030-01-01T00:00:00", false), + ("1893456000", false), + ] { + for (name, write) in writes { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let mut cache = CultCache::new(); + cache.register_entry_type::().unwrap(); + cache + .add_generic_backing_store(ReportingStore { inner: SingleFileMessagePackBackingStore::new(&path), report: report.to_string() }) + .unwrap(); + cache.pull_all_backing_stores().unwrap(); + write(&mut cache); + let held = cache.get_envelope::("c").unwrap().unwrap().stored_at; + if parses { + assert_eq!(held, report, "{name}"); + } else { + assert_ne!(held, report, "{name} held a stamp that is not RFC 3339"); + assert!(chrono::DateTime::parse_from_rfc3339(&held).is_ok(), "{name}: {held}"); + } + } + } +} + +#[test] +fn a_stamp_with_no_successor_refuses_every_single_file_write_and_writes_nothing() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let cache = open(&path); + let mut store = SingleFileMessagePackBackingStore::new(&path); + let last = "9999-12-31T23:59:59.999999Z"; + store.push(&envelope_of(&cache, "c", &counter(1), last)).unwrap(); + let before = file(&path); + let version = store.read("c").unwrap().unwrap().version; + let next = envelope_of(&cache, "c", &counter(2), "2020-01-01T00:00:00Z"); + assert!(store.push(&next).is_err()); + assert!(store.push_all(std::slice::from_ref(&next), PushAllOptions::default()).is_err()); + assert!(store.commit_if(&next, Some(&version)).is_err()); + assert_eq!(file(&path), before, "a refused mint wrote"); +} + +#[test] +fn commit_if_refuses_a_retype_as_push_does() { + let dir = tempfile::tempdir().unwrap(); + let path = path_in(&dir); + let cache = open(&path); + let mut store = SingleFileMessagePackBackingStore::new(&path); + let note = envelope_of(&cache, "k", &Note { text: "n".into() }, "2020-01-01T00:00:00Z"); + store.push(¬e).unwrap(); + let version = store.read("k").unwrap().unwrap().version; + let before = file(&path); + let counter_at_k = envelope_of(&cache, "k", &counter(1), "2020-01-01T00:00:00Z"); + let error = store.commit_if(&counter_at_k, Some(&version)).expect_err("a retype"); + assert!(error.downcast_ref::().is_some(), "{error:#}"); + assert!(store.push(&counter_at_k).is_err()); + assert_eq!(file(&path), before, "a refused retype wrote"); + assert_eq!(store.read("k").unwrap().unwrap().envelope.r#type, "note"); +} diff --git a/packages/cultcache-ts/CHANGELOG.md b/packages/cultcache-ts/CHANGELOG.md index e71821c8..b41455a3 100644 --- a/packages/cultcache-ts/CHANGELOG.md +++ b/packages/cultcache-ts/CHANGELOG.md @@ -5,8 +5,25 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following ## [Unreleased] +## [0.16.0] - 2026-11-14 + +### Added + +- `isStoredAtStamp(text)` (package and browser entries): whether a `storedAt` is a stamp, the one shape pinned in `tests/vectors/stored-at-stamp/` (an RFC 3339 date-time with `T`, an optional 1 to 9 digit fraction, `Z` or `+hh:mm`/`-hh:mm`, a real calendar date, seconds 00-59, years 0001-9999). cultnet-ts and cultmesh-ts judge a peer's put by it (variants:ruling:raw-put-without-stamp-refused). +- `CacheBackingStore` gains the optional `read(key)` (the record at the key with its content version, from one read) and `commitIf(entry, expectedVersion)` (write only if the version still holds, under the store's write lock; returns the storedAt written). The single-file store implements both. The content version is the lowercase hex SHA-256 of the record's persisted encoding `[key, schemaId, storedAt, payload]`, equal to the C# reference's on `tests/vectors/content-version`. A store without them still works for everything except `update`. +- `CultCacheUpdateResult` and `CultCacheUpdateOptions` (`maxAttempts`, default 5), exported from the package entry and the browser entry. + +### Changed + +- A write that replaces a record at its key with different content (another schema id or other payload bytes) is stored strictly later than it: a supplied `storedAt` not later than the stored one is written one millisecond after it, so a content version that includes `storedAt` never repeats across writes. A rewrite of identical content (same schema id and payload bytes) keeps the stored stamp and so its version, which is what keeps a periodic flush of an unchanged cache from changing every record's version; C# re-mints every replacing write, so this is a departure from it, and the writer's type and catalog entry are written either way. `push`, `pushAll` and `commitIf` alike; the single-file store's `push` returns the stamp it wrote (`CacheBackingStore.push` may resolve `{ storedAt }`), and `put`, `putGlobal` and `putEnvelope` install that stamp in the cache, so the cache and the file agree after a put over a record stamped at or after now (a push result that is not a parseable stamp is ignored). A stamp that would pass 9999-12-31T23:59:59.999Z is refused with `RangeError` and nothing is written. +- `withStoreWriteLock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. Pinned by a test that puts the link while another holder owns the lock. +- `CultCache.isWritable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store. + ### Breaking +- `update` and `updateGlobal` are the conditional golden path and no longer the blind pair. `update(definition, key, apply, options?)` reads the record and its content version from the store, passes `apply` a fresh copy (never the held instance), commits only if the version still holds and runs `apply` again on a lost race, up to `maxAttempts`. `apply` may be async, may return `undefined` to write nothing, and must be free of side effects. It resolves to `{ outcome: "updated" | "declined" | "lostRace", document }` instead of the written value, so a caller reads `result.document`; a lost race that outlasts the attempts is `lostRace`, not a throw. `apply` returning `undefined` or `null` declines. `maxAttempts` is an integer of at least 1; `undefined` is the default 5 and `null` is refused with `RangeError`. A conditional write over a global loaded from a legacy key is refused wherever it enters (`updateGlobal`, or `update` at `CultCache.GLOBAL_KEY`) and leaves the legacy record untouched; `apply` may have run once before the write is refused. A store attached to a store-less cache while `apply` awaits makes that attempt a lost race. It needs a store with `read` and `commitIf` (the single-file store has them); a cache with no store compares the held entry. `put` and `putGlobal` are the blind writes and are unchanged. +- A write at a single-file store path that is a symbolic link, live or dangling, is refused with the new `StoreLinkedError` (exported from the package entry; `error.path` names the path) before the lock and the replace, and writes nothing: `push`, `pushAll`, `delete` and a soft flush alike. Before, a write at a live link replaced the link with a regular file, a dangling link failed with `ENOENT` (or, for `pushAll`, was replaced), and a soft flush at a link skipped. A caller catching `ENOENT` around a write at a dangling link now catches `StoreLinkedError`; configure the link's target as the store path. Reads through a live link work. An `lstat` failure other than `ENOENT` (`ENOTDIR`, `EACCES`, `ELOOP`) is rethrown as it is. +- `SingleFileMessagePackBackingStore.pushAll` writes the given entries onto the store as it is under the lock, as `push` does: a record under the same key is replaced and no other is removed. It used to replace the whole file with the entries, so `CultMesh.flush()` from a cache that held less than the store (a node created with `pullOnStart: false`, or one whose image was stale) wiped the records it never read. A caller that relied on `pushAll` to reset a store removes the file first. A soft flush is unchanged. `CultCache.storedSnapshot()` is the image as the stores hold it (a global loaded from a legacy key under that key) and is what `CultMesh.flush` persists, so a flush no longer leaves such a global under two keys; `snapshot()` is unchanged. - A single-file store marked `cultcache.store.v3` (it holds an element id, written by the C# runtime) is read, and a rewrite keeps the mark and the ids; this runtime does not decide when a document carries ids. A runtime from before this change refuses a v3 store by name, so rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - `indexes` are unique, as C# `[CultIndex]`: a write or a load that leaves two records of a type holding one non-blank value under one alias is refused naming both keys and the alias, and writes nothing. Index text is the C# canonical @@ -15,18 +32,21 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following the many-row selection aliases (never indexed for lookup, same catalog `indexAlias`). The per-key migration rule is in the README; classify every key before the CultLib your tools load is updated. The refusal verdict is the contract, its text is informational. - The single-file reader (v1 and the legacy envelope array) refuses a store that holds two records under one key as `StoreUnreadableError`; a store that opened before may now be refused. Repair it by removing the extra record with the runtime that wrote it. -- A `put`, `push` or `pushAll` that would land a second type under a held key (against the file, or within one batch) is refused with `SchemaConflictError`, and nothing is written. The written type may replace the record when its catalog entry carries the stored schema id as its own or lists it compatible (a type renamed under a stable id restamps the record, as C# does); otherwise retyping a key is a delete, then a put. The C# allowance for an undeclared older id of the same schema name that reads whole is not ported: a same-named type keeps passing, including one that drops persisted slots, which C# refuses. +- A `put`, `push` or `pushAll` that would land a second type under a held key (against the file, or within one batch) is refused with `SchemaConflictError`, and nothing is written. The written type may replace the record when its catalog entry carries the stored schema id as its own or lists it compatible (a type renamed under a stable id restamps the record, as C# does, when its payload changes; an identical payload keeps the stamp); otherwise retyping a key is a delete, then a put. The C# allowance for an undeclared older id of the same schema name that reads whole is not ported: a same-named type keeps passing, including one that drops persisted slots, which C# refuses. - Same-type writes replace by key; in one `pushAll` batch the last entry for a key wins. - A store whose canonical-schema catalog slot is not a string is refused as `StoreUnreadableError`. - The cache holds one record per key (variants:ruling:cache-refuses-retype). A `put` or `putEnvelope` that lands a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store (a memory-only cache used to hold both types at the key). A type that owns the held record's schema id (as for the store) replaces it, unless the held record is in a store other than the written type's home: that rename is refused with `SchemaConflictError` (remove the record first). Two stores holding one key, registered or foreign, refuse the load and admit nothing. A store's `delete` removes the record whose stored schema id the removal's type owns, whatever the arrived record's `type` string. - A store holding a record no registered type claims now loads (it used to throw "No schema is registered..."): the cache does not serve the record, lists it in `foreignRecords()`, keeps its bytes through every write, and refuses a `put` or a `delete` at its key with `SchemaConflictError` (variants:ruling:unheld-removal-refused-at-commit). Registering a type that claims its schema id serves it at the next `pullAllBackingStores`. +- A record whose payload carries a slot past its reader's declared slots, at any depth (the record, a nested record, a list element, a union arm, a unit arm), is now held read-only: it loads and reads, and every `put`, `putEnvelope`, `update` or `delete` of its key is refused with `SchemaConflictError` before a store is touched (cultlib-gaps:ruling:extra-trailing-slots, as the C# reference). A load used to skip the extra slots silently and a later write shed them. The refusal names the schema, the key, the type that was read with extra slots and both slot counts, never a slot value, and carries them as `SchemaConflictError.unreadSlots`. An untouched record keeps its bytes through writes of other keys. Missing trailing slots keep their declared defaults and mark nothing. A `put` never marks (the value is encoded by this runtime); a `putEnvelope` of a payload carrying extra slots lands its bytes and holds the record read-only. A mark at depth 0 is consented to only when the record arrived under a schema id other than the reader's own that the reader lists compatible, and that id's catalog entry lists those slots (a writer that adds a slot and keeps the reader's id is never consented to) (`unreadSlots.explained`); every other mark refuses whatever is declared. New: `CultCache.isWritable(definition, key)` (live, advisory), `CultCache.unreadSlots(definition, key)` and the exported `CultCacheUnreadSlots`. Documents with no slot codec (a plain map, a user formatter) never mark. A `delete` is by key: a type that owns the schema id of the record held at the key as another type removes that one record (one store call, the cache dropped once the store has removed it), and a type that does not own it is refused with `SchemaConflictError`, as a `put` is; `isWritable` answers both. Implied semver: a minor bump (0.16.0) while the package is 0.x; a major one at 1.0 and after. ### Added +- `checkPayloadDepth` returns the offset where the first value ends, or `undefined` when the payload ends early (a header, a container or a str/bin/scalar body cut short; it was `void`); it still refuses only depth. The bytes-after refusal is internal (`checkPayloadIsOneValue`, not exported); no new public name. - `CultCache.foreignRecords()`: the `{key, schemaId, schemaName, storedAt}` of every record the last pull carried and no registered type claims, sorted by key ordinal. ### Changed +- A document payload is exactly one MessagePack value (cultlib-gaps:ruling:trailing-bytes-refuse). `decodeDocumentPayload`, the one reader (store loads, `put`/`putEnvelope`, CultNet replication, the provider command), and the inspector refuse bytes after the first value with `document type "T" payload has N bytes after its first MessagePack value.`, counted from the walk's end and never walked or echoed; a payload that ends early stays the decoder's to refuse. The refused set is unchanged; only the text changed (it was `@msgpack`'s `RangeError` "Extra N of M byte(s)" for a plain document and the codec reader's "bytes after its value" for a codec document, and the inspector's preview no longer shows a tail), so code matching either text must match this one. `readWire` reads one value and no longer checks what follows. Stated limit: a definition with its own `formatter` is not walked, so whether its payload is one value is the formatter's. Implied bump: 0.16.0 (pre-1.0 minor, with the Breaking entries above). - Opening a legacy (pre-v1) store is a pure read: payloads that were not bytes are normalized in memory and the file is left as it was, so a codec that refuses the content cannot lose the old bytes. The next push, delete or flush rewrites the file as v1, merging what the open read. diff --git a/packages/cultcache-ts/README.md b/packages/cultcache-ts/README.md index 2096d258..2aa78c9c 100644 --- a/packages/cultcache-ts/README.md +++ b/packages/cultcache-ts/README.md @@ -49,7 +49,9 @@ through `decodeDocumentPayload`, and when you need the bytes themselves (hashing write) take them with `documentPayloadBytes`, the sanctioned escape. - `decodeDocumentPayload(definition, payload)` reads a payload through the definition's formatter, - codec or plain map and then its schema. The definition is required. + codec or plain map and then its schema. The definition is required. A payload is exactly one + MessagePack value: bytes after the first are refused by count, naming the type, and never walked or + echoed (a payload that ends early is the decoder's to refuse). - `encodeDocumentPayload(definition, value)` is the writer: the schema first, then the formatter, codec or plain map. - `documentPayloadFromBytes(bytes)` is where bytes enter (a record you validated, a store read, a @@ -166,13 +168,11 @@ const settings = cache.getRequiredGlobal(settingsDocument); - `getByIndex(...)` - `getGlobal(...)` - `getRequiredGlobal(...)` -- `put(...)` -- `putGlobal(...)` -- `update(...)` -- `updateGlobal(...)` +- `update(...)` and `updateGlobal(...)`, the conditional golden path: apply a function to a fresh copy of the record as the store holds it, commit only if it is unchanged, re-run on a lost race (`updated`, `declined` or `lostRace`) +- `put(...)` and `putGlobal(...)`, the blind opt-out: last writer wins - `delete(...)` - `deleteGlobal(...)` -- `snapshot()` +- `snapshot()` (the logical image) and `storedSnapshot()` (the image as the stores hold it, for a whole flush) - `getEnvelope(...)` - `getRequiredEnvelope(...)` - `getGlobalEnvelope(...)` @@ -327,7 +327,7 @@ If you do not supply one, `CultCacheTS` uses MessagePack plus Zod validation by The 256-container nesting bound (`MAX_PAYLOAD_DEPTH`, refused with `CultCachePayloadDepthError`) covers the MessagePack the library itself writes: the default format and a codec. A payload your formatter writes is opaque bytes in any format, so the -library does not walk it as MessagePack; its format and its bounds are your formatter's. +library does not walk it as MessagePack; its format, its bounds and whether it is one value are your formatter's. Zod is no longer mandatory. Any schema object with a `parse(...)` function will do, which means generated Ajv-backed JSON-schema contracts can feed the cache @@ -371,8 +371,11 @@ const sceneDocument = defineDocumentType({ }); ``` -- A slot is `[name, codec, default?]`. A gap is written nil and skipped on read; an extra trailing slot is skipped; a - slot missing from the payload (or nil, when a default is declared) takes its default, and is refused without one. +- A slot is `[name, codec, default?]`. A gap is written nil and skipped on read; an extra trailing slot is read past and + marks the record read-only (every write or removal of its key is refused with `SchemaConflictError`, `cache.isWritable` + asks first). The one writable case is a depth-0 extra slot of a record that arrived under a schema id other than the reader's own, which the + reader lists in `compatibleSchemaIds`, whose catalog entry lists those slots: declaring that id consents to the loss, and a write restamps + the record. A nested slot, or a writer that keeps the reader's own id, is never consented to. A slot missing from the payload (or nil, when a default is declared) takes its default, and is refused without one. - A union refuses an undeclared arm key on read, naming the union, the key and the declared keys. Nothing falls back to a default arm. An arm with no slots is `codecs.unit` and is written `[key, []]`. - A refusal names the union, the field and the slot; it never echoes the value it refused. diff --git a/packages/cultcache-ts/package.json b/packages/cultcache-ts/package.json index b6803979..f444f51a 100644 --- a/packages/cultcache-ts/package.json +++ b/packages/cultcache-ts/package.json @@ -1,6 +1,6 @@ { "name": "@gamecult/cultcache-ts", - "version": "0.15.0", + "version": "0.16.0", "description": "CultCache-style typed persistence for TypeScript with schema-backed polymorphic MessagePack storage.", "main": "dist/index.js", "types": "dist/index.d.ts", diff --git a/packages/cultcache-ts/src/browser.ts b/packages/cultcache-ts/src/browser.ts index 668fc0b8..529bc53b 100644 --- a/packages/cultcache-ts/src/browser.ts +++ b/packages/cultcache-ts/src/browser.ts @@ -16,6 +16,7 @@ export { } from "./document"; export { CultCachePayloadDepthError, MAX_PAYLOAD_DEPTH } from "./payload-depth"; export type { DocumentPayload } from "./document"; +export { isStoredAtStamp } from "./stored-at"; export { SchemaConflictError, StoreUnreadableError } from "./store-format"; export type { AnyCultCacheDocumentDefinition, @@ -34,6 +35,9 @@ export type { CultCacheIndexScalar, InferCultCacheSchemaValue, CultCacheStoreRegistration, + CultCacheUnreadSlots, + CultCacheUpdateOptions, + CultCacheUpdateResult, PushAllOptions, } from "./types"; export type { diff --git a/packages/cultcache-ts/src/codec.ts b/packages/cultcache-ts/src/codec.ts index 9b885159..a2dd7b15 100644 --- a/packages/cultcache-ts/src/codec.ts +++ b/packages/cultcache-ts/src/codec.ts @@ -198,6 +198,66 @@ type SlotSpec = type SlotsSpec = Readonly }[keyof TValue & string]>>; +// The slots a payload decode read past, noted by `slots` and the unit arm while the one decode entry (decodeNoting) has a scope +// open, and read out by that entry. Module-scoped as the reference's is thread-bound: a decode is synchronous, and a decode +// outside the entry (a codec called directly, a CultMesh read) notes nothing. No codec signature carries it. +export interface NotedSlots { + /** The first type read with extra slots, outermost first. */ + readonly typeName: string; + readonly slotCount: number; + readonly declaredCount: number; + readonly occurrences: number; + /** True when every noted object was the document's own: a union adds no depth, each record or arm inside a field adds one. */ + readonly depth0Only: boolean; +} + +interface SlotScope { + depth: number; + found?: { typeName: string; slotCount: number; declaredCount: number; occurrences: number; depth0Only: boolean }; +} + +let scope: SlotScope | undefined; + +/** The one decode entry: runs the decode with a scope open and returns its value with what it noted. */ +export function decodeNoting(decode: () => T): { value: T; unreadSlots?: NotedSlots } { + const outer = scope; + const mine: SlotScope = { depth: 0 }; + scope = mine; + try { + const value = decode(); + return mine.found ? { value, unreadSlots: mine.found } : { value }; + } finally { + scope = outer; + } +} + +function noteExtraSlots(typeName: string, slotCount: number, declaredCount: number): void { + if (!scope) return; + const found = scope.found; + if (found) { + found.occurrences += 1; + found.depth0Only &&= scope.depth === 0; + } else { + scope.found = { typeName, slotCount, declaredCount, occurrences: 1, depth0Only: scope.depth === 0 }; + } +} + +// What a record inside a field reads at: one deeper than the record holding the field. +function nested(read: () => T): T { + if (!scope) return read(); + scope.depth += 1; + try { + return read(); + } finally { + scope.depth -= 1; + } +} + +// The reference's type name for a mark: the type, not its namespace or the type it is nested in. +function simpleTypeName(typeName: string | undefined): string { + return typeName === undefined ? "record" : typeName.slice(Math.max(typeName.lastIndexOf("."), typeName.lastIndexOf("+")) + 1); +} + const INT32_MIN = -(2 ** 31); const INT32_MAX = 2 ** 31 - 1; @@ -232,7 +292,7 @@ function pascalCase(name: string): string { } // A positional record: slot `key` holds field `name`. Gaps are written nil and skipped on read, extra trailing slots -// are skipped, and a slot missing from the payload takes its declared default or is refused. A default fills only a +// are read past and noted (decodeNoting): the record that carried them is read-only, never rewritten without them, and a slot missing from the payload takes its declared default or is refused. A default fills only a // missing trailing slot, never a nil: as in the reference, a nil in a reference slot (string, bytes, record, union, // list) reads as null and writes back as nil, so a record the reference wrote re-encodes byte-identical, and a nil in // a value slot (int, float, bool) is refused unless the slot is nullable. @@ -281,6 +341,9 @@ export function slots(spec: SlotsSpec, typeName?: decode(raw) { expect(raw, "an array of slots", Array.isArray(raw)); const input = raw as unknown[]; + if (input.length > length) { + noteExtraSlots(simpleTypeName(typeName), input.length, length); + } const out: Record = {}; for (const field of fields) { const present = field.key < input.length; @@ -291,7 +354,7 @@ export function slots(spec: SlotsSpec, typeName?: ? defaultOf(field) : input[field.key] === null && !field.codec.valueType ? null - : at(`${field.name} (slot ${field.key})`, () => field.codec.decode(input[field.key])); + : at(`${field.name} (slot ${field.key})`, () => nested(() => field.codec.decode(input[field.key]))); } return out as TValue; }, @@ -363,6 +426,9 @@ export function union( if (arm === undefined) { throw new Error(`Union ${name}: unknown arm key ${key} (declared keys: ${declared}).`); } + if (arm.body === unit && raw[1].length > 0) { + noteExtraSlots(arm.armName, raw[1].length, 0); + } const fields = arm.body === unit ? {} : at(`arm ${arm.armName}`, () => (arm.body as SlotsCodec).decode(raw[1])); return { kind: arm.armName, ...fields } as unknown as TValue; }, diff --git a/packages/cultcache-ts/src/cult-cache-inspector.ts b/packages/cultcache-ts/src/cult-cache-inspector.ts index 1c359157..71643913 100644 --- a/packages/cultcache-ts/src/cult-cache-inspector.ts +++ b/packages/cultcache-ts/src/cult-cache-inspector.ts @@ -1,6 +1,6 @@ import { decode } from "@msgpack/msgpack"; -import { checkPayloadDepth } from "./payload-depth"; +import { checkPayloadDepth, checkPayloadIsOneValue } from "./payload-depth"; import { type StoreFormat, isStoreSnapshot, refuseUnreadStoreBytes, requireStoreSlots, requireV1RecordSlots } from "./store-format"; export interface CultCacheInspection { @@ -235,12 +235,22 @@ function decodeLegacyEnvelope(value: unknown): InspectedRecord { }; } -// The inspector's one payload read: the depth walk first, so a payload past the limit is refused before any decoder sees it. +// The inspector's one payload read: the depth walk first, so a payload past the limit is refused before any decoder sees it, +// and bytes after the first value are refused with the shared clause (the tail is never decoded or shown). function decodeBoundedPayload(payload: Uint8Array): unknown { - checkPayloadDepth(payload); + checkPayloadIsOneValue(payload, "document payload"); return decode(payload); } +// Where the first value ends (the whole payload when it ends early or cannot be walked): a preview never shows a tail. +function valueEnd(payload: Uint8Array): number { + try { + return checkPayloadDepth(payload) ?? payload.length; + } catch { + return payload.length; + } +} + function previewPayload(payload: Uint8Array): Pick { try { return { payloadPreview: toJsonSafe(decodeBoundedPayload(payload)) }; @@ -248,7 +258,7 @@ function previewPayload(payload: Uint8Array): Pick value.toString(16).padStart(2, "0")) .join(" "), }, diff --git a/packages/cultcache-ts/src/cult-cache.ts b/packages/cultcache-ts/src/cult-cache.ts index 443c0c18..c8d1cc62 100644 --- a/packages/cultcache-ts/src/cult-cache.ts +++ b/packages/cultcache-ts/src/cult-cache.ts @@ -1,6 +1,6 @@ import { declaresRole, - decodeDocumentPayload, + decodeDocumentPayloadMarked, documentPayloadBytes, documentPayloadFromBytes, encodeDocumentPayload, @@ -8,6 +8,7 @@ import { schemaIdentityOf, } from "./document"; import { resolveDocumentFormatter } from "./document-formatter"; +import type { NotedSlots } from "./codec"; import type { DocumentPayload } from "./document"; import { SchemaConflictError, schemaIdForEnvelope, writtenOwns } from "./store-format"; import type { @@ -20,6 +21,9 @@ import type { CultCacheEnvelope, CultCacheSchemaCatalogEntry, CultCacheStoreRegistration, + CultCacheUnreadSlots, + CultCacheUpdateOptions, + CultCacheUpdateResult, } from "./types"; type Accessor = (value: unknown) => string | undefined; @@ -27,7 +31,7 @@ type Accessor = (value: unknown) => string | undefined; // A registered definition reads and writes its documents through the owner, on payloads the cache has cloned. type RegisteredFormatter = { encode(value: unknown): DocumentPayload; - decode(payload: DocumentPayload): unknown; + decode(payload: DocumentPayload): { value: unknown; unreadSlots?: NotedSlots }; }; type RegisteredDefinition = { @@ -46,10 +50,18 @@ type EntryLookups = { readonly indexes: ReadonlyMap; }; +// What a conditional write requires of the record at its key: the content version its store reported (null: no record), or for a +// cache with no store the entry it held. Every write installs a new entry object, so the identity catches an A-B-A. +type WriteCondition = { readonly version: string | null } | { readonly held: HydratedEntry | undefined }; + type HydratedEntry = CultCacheEnvelope & { catalogEntry: NonNullable; value: unknown; lookups: EntryLookups; + // What the payload decode read past: set when the record was admitted, and it makes the record read-only unless explained. + unreadSlots?: CultCacheUnreadSlots; + // The schema id the record arrived under, before it was admitted under its reader's: what an unread-slots refusal names. + storedSchemaId?: string; }; // Everything a pull replaces. Built whole, then swapped in with one assignment. @@ -84,6 +96,20 @@ type DerivedTypeLookups = { readonly indexMaps: Map>; }; +// The consent a compatible-id declaration gives (cultlib-gaps:ruling:slot-mark-consent): a mark is explained only when the record +// arrived under an id other than the reader's own that the reader lists compatible, it is at +// depth 0 and the schema comparison of the catalog entry the record arrived with against the reader's own reported the slots it +// carries as ignored, so its slot count is at most the highest ignored slot plus one. A record written or put through the cache +// arrives under the reader's own entry, which ignores nothing, so no mark of it is explained. +function explainedBy(noted: NotedSlots, reader: CultCacheSchemaCatalogEntry, arrivedId: string, arrived: CultCacheSchemaCatalogEntry | undefined): boolean { + if (arrivedId === reader.schemaId || !(reader.compatibleSchemaIds ?? []).includes(arrivedId)) { + return false; + } + const declared = new Set((reader.members ?? []).map((member) => member.slot)); + const ignored = (arrived?.members ?? []).map((member) => member.slot).filter((slot) => !declared.has(slot)); + return noted.depth0Only && noted.slotCount <= Math.max(-1, ...ignored) + 1; +} + function emptyHydratedState(): HydratedState { return { entries: new Map(), @@ -282,85 +308,88 @@ export class CultCache { // HydratedState, running every check and every user accessor, and replaces the cache's state // in one assignment only once the build finishes. A refused load admits nothing. pullAllBackingStores(): Promise { - return this.#serial(async () => { - const next = emptyHydratedState(); - const persistedGlobalKeys = new Map(); + return this.#serial(() => this.#pullNow()); + } - for (const registration of this.#stores) { - const entries = await registration.store.pullAll(); + // The pull itself, for a caller already inside a serial operation. + async #pullNow(): Promise { + const next = emptyHydratedState(); + const persistedGlobalKeys = new Map(); - for (const entry of entries) { - const registered = this.#resolveDefinitionForEnvelope(entry); - if (!registered) { - // A store may hold what this cache has no type for; it is carried, not served, and not a reason to refuse the load. - next.foreign.push({ - key: entry.key, - schemaId: schemaIdForEnvelope(entry), - schemaName: entry.catalogEntry?.schemaName ?? entry.type, - storedAt: entry.storedAt, - }); - continue; - } + for (const registration of this.#stores) { + const entries = await registration.store.pullAll(); - const type = registered.definition.type; - const home = this.#resolveRegistration(type); - if (home?.store !== registration.store) { + for (const entry of entries) { + const registered = this.#resolveDefinitionForEnvelope(entry); + if (!registered) { + // A store may hold what this cache has no type for; it is carried, not served, and not a reason to refuse the load. + next.foreign.push({ + key: entry.key, + schemaId: schemaIdForEnvelope(entry), + schemaName: entry.catalogEntry?.schemaName ?? entry.type, + storedAt: entry.storedAt, + }); + continue; + } + + const type = registered.definition.type; + const home = this.#resolveRegistration(type); + if (home?.store !== registration.store) { + throw new Error( + `CultCache "${type}" record "${entry.key}" was loaded from ${this.#describe(registration)}, ` + + `but its home is ${this.#describe(home)}.`, + ); + } + + const payload = this.#cloneBytes(entry.payload); + const { value, unreadSlots } = registered.formatter.decode(payload); + let admitted: CultCacheEnvelope = { ...entry, type, payload }; + if (registered.global) { + const persistedKey = persistedGlobalKeys.get(type); + if (persistedKey !== undefined) { throw new Error( - `CultCache "${type}" record "${entry.key}" was loaded from ${this.#describe(registration)}, ` + - `but its home is ${this.#describe(home)}.`, + `CultCache global document type "${type}" has multiple persisted entries: "${persistedKey}" and "${entry.key}".`, ); } - - const payload = this.#cloneBytes(entry.payload); - const value = registered.formatter.decode(payload); - let admitted: CultCacheEnvelope = { ...entry, type, payload }; - if (registered.global) { - const persistedKey = persistedGlobalKeys.get(type); - if (persistedKey !== undefined) { - throw new Error( - `CultCache global document type "${type}" has multiple persisted entries: "${persistedKey}" and "${entry.key}".`, - ); - } - persistedGlobalKeys.set(type, entry.key); - if (entry.key !== CultCache.GLOBAL_KEY) { - next.legacyGlobals.set(type, admitted); - admitted = { ...admitted, key: CultCache.GLOBAL_KEY }; - } + persistedGlobalKeys.set(type, entry.key); + if (entry.key !== CultCache.GLOBAL_KEY) { + next.legacyGlobals.set(type, admitted); + admitted = { ...admitted, key: CultCache.GLOBAL_KEY }; } - const hydrated = this.#admitEntry(next, registered, admitted, value); - const other = this.#heldUnderOtherType(next, type, hydrated.key); - if (other !== undefined) { - throw new Error(`Record ${hydrated.key} is held by two stores, as type "${other.type}" and as type "${type}".`); - } - this.#installEntry(next, registered, hydrated); } - } - - const foreignByKey = new Map(); - for (const record of next.foreign) { - const first = foreignByKey.get(record.key); - if (first !== undefined) { - throw new Error(`Record ${record.key} is held by two stores, as schema "${first}" and as schema "${record.schemaName}".`); - } - foreignByKey.set(record.key, record.schemaName); - const other = [...this.#definitions.keys()].map((type) => next.entries.get(this.#entryId(type, record.key))).find((held) => held !== undefined); + const hydrated = this.#admitEntry(next, registered, admitted, value, unreadSlots); + const other = this.#heldUnderOtherType(next, type, hydrated.key); if (other !== undefined) { - throw new Error(`Record ${record.key} is held by two stores, as type "${other.type}" and as schema "${record.schemaName}".`); + throw new Error(`Record ${hydrated.key} is held by two stores, as type "${other.type}" and as type "${type}".`); + } + this.#installEntry(next, registered, hydrated); + } + } + + const foreignByKey = new Map(); + for (const record of next.foreign) { + const first = foreignByKey.get(record.key); + if (first !== undefined) { + throw new Error(`Record ${record.key} is held by two stores, as schema "${first}" and as schema "${record.schemaName}".`); + } + foreignByKey.set(record.key, record.schemaName); + const other = [...this.#definitions.keys()].map((type) => next.entries.get(this.#entryId(type, record.key))).find((held) => held !== undefined); + if (other !== undefined) { + throw new Error(`Record ${record.key} is held by two stores, as type "${other.type}" and as schema "${record.schemaName}".`); + } + } + + // Judged whole, once: the refusal names the least violation of the state this load would leave, whatever the + // order of the stores and of the entries in them. + refuseLeastIndexSharing(function* () { + for (const entry of next.entries.values()) { + for (const [alias, value] of entry.lookups.indexes) { + yield { type: entry.type, schemaName: entry.catalogEntry.schemaName, alias, value, key: entry.key }; } } + }()); - // Judged whole, once: the refusal names the least violation of the state this load would leave, whatever the - // order of the stores and of the entries in them. - refuseLeastIndexSharing(function* () { - for (const entry of next.entries.values()) { - for (const [alias, value] of entry.lookups.indexes) { - yield { type: entry.type, schemaName: entry.catalogEntry.schemaName, alias, value, key: entry.key }; - } - } - }()); - - this.#hydrated = next; - }); + this.#hydrated = next; } get( @@ -493,12 +522,16 @@ export class CultCache { return key ? this.get(definition, key) : undefined; } + /** + * The blind write: the record at the key becomes `value`, whatever it held, and the last writer wins. `update` is the + * conditional path, which changes a record from what the store holds now and re-runs on a lost race. + */ put( definition: TDefinition, key: string, value: CultCacheDocumentValue, ): Promise> { - return this.#serial(() => this.#putNow(definition, key, value)); + return this.#serial(async () => (await this.#putNow(definition, key, value)).value); } putEnvelope( @@ -518,10 +551,9 @@ export class CultCache { if (!envelope.storedAt || envelope.storedAt.trim().length === 0) { throw new Error(`CultCache envelope storedAt for type "${definition.type}" must be non-empty.`); } - this.#requireGlobalKey(registered, envelope.key); const payload = this.#cloneBytes(envelope.payload); - const parsed = registered.formatter.decode(payload) as CultCacheDocumentValue; + const { value: parsed, unreadSlots } = registered.formatter.decode(payload) as { value: CultCacheDocumentValue; unreadSlots?: NotedSlots }; const entry: CultCacheEnvelope = { key: envelope.key, type: envelope.type, @@ -532,7 +564,7 @@ export class CultCache { schemaId: registered.catalogEntry.schemaId, catalogEntry: registered.catalogEntry, }; - await this.#writeNow(registered, entry, parsed); + await this.#writeNow(registered, entry, parsed, unreadSlots); return parsed; }); } @@ -541,31 +573,68 @@ export class CultCache { definition: TDefinition, value: CultCacheDocumentValue, ): Promise> { - return this.#serial(() => { + return this.#serial(async () => { this.#requireGlobal(definition); - return this.#putNow(definition, CultCache.GLOBAL_KEY, value); + return (await this.#putNow(definition, CultCache.GLOBAL_KEY, value)).value; }); } - update( + /** + * Changes the record at the key from what the store holds now. `apply` may run up to `maxAttempts` times (default 5): each + * attempt gets a fresh copy of the record as the store holds it (undefined when absent), which it may change and return, and + * the result is written only if the record is still that version; a lost race runs `apply` again on the newer record. It must + * not write this cache or have side effects. Returning undefined writes nothing (`declined`). A lost race that outlasts + * `maxAttempts` returns `lostRace` and does not throw; a refusal, or a throw from `apply`, rejects and is not retried. The + * write is durable when the promise resolves. `document` is the cache's record after the outcome. `put` is the blind write. + */ + async update( definition: TDefinition, key: string, - updater: ( + apply: ( current: CultCacheDocumentValue | undefined, - ) => CultCacheDocumentValue, - ): Promise> { - return this.#serial(() => this.#putNow(definition, key, updater(this.get(definition, key)))); + ) => CultCacheDocumentValue | undefined | Promise | undefined>, + options: CultCacheUpdateOptions = {}, + ): Promise>> { + if (typeof apply !== "function") { + throw new TypeError("CultCache update needs a function to apply."); + } + // undefined is the default; anything else must be an integer of at least 1, so null is refused. + const maxAttempts = options.maxAttempts === undefined ? 5 : options.maxAttempts; + if (!Number.isInteger(maxAttempts) || maxAttempts < 1) { + throw new RangeError(`CultCache update maxAttempts must be an integer of at least 1, not ${String(maxAttempts)}.`); + } + + for (let attempt = 1; ; attempt++) { + const { expected, copy } = await this.#serial(() => this.#readForUpdate(definition, key)); + // Outside the serial queue and the store's lock: apply may await, and may write this cache through other keys. + const next = await apply(copy as CultCacheDocumentValue | undefined); + if (next === undefined || next === null) { + return { outcome: "declined", document: this.get(definition, key) }; + } + const written = await this.#serial(() => this.#putNow(definition, key, next, expected)); + if (written.outcome === "committed") { + return { outcome: "updated", document: this.get(definition, key) as CultCacheDocumentValue }; + } + if (attempt === maxAttempts) { + // Display only. A cache with no store has nothing to pull, and a pull would replace what it holds with nothing. + if (this.#stores.length > 0) { + await this.#serial(() => this.#pullNow()); + } + return { outcome: "lostRace", document: this.get(definition, key) }; + } + } } - updateGlobal( + /** `update` of the type's global record. */ + async updateGlobal( definition: TDefinition, - updater: ( + apply: ( current: CultCacheDocumentValue | undefined, - ) => CultCacheDocumentValue, - ): Promise> { - return this.#serial(() => - this.#putNow(definition, CultCache.GLOBAL_KEY, updater(this.getGlobal(definition))), - ); + ) => CultCacheDocumentValue | undefined | Promise | undefined>, + options: CultCacheUpdateOptions = {}, + ): Promise>> { + this.#requireGlobal(definition); + return this.update(definition, CultCache.GLOBAL_KEY, apply, options); } delete( @@ -585,6 +654,30 @@ export class CultCache { }); } + /** What the record's payload carried past this build's declared slots, or undefined: the held record is read-only unless `explained`. */ + unreadSlots(definition: AnyCultCacheDocumentDefinition, key: string): CultCacheUnreadSlots | undefined { + return this.#hydrated.entries.get(this.#entryId(this.#requireDefinition(definition).definition.type, key))?.unreadSlots; + } + + /** + * Whether a write of the record's own type at the key would be let through: it asks the judge #writeNow asks, of the record the + * cache holds (a key a store carries foreign, a record of another type the write does not own or cannot replace, a global at + * another key, a record carrying unread slots nothing consents to). It answers the WRITE form. A removal asks the judge's + * removal form and differs in two states: an owner's `delete` of a record held in another store than its own home succeeds + * where this is false (a write would push to the wrong store), and `delete` of a global type at a key other than __global__ is + * the plain false of an unheld key where this is false. It does not look at store paths: a link at a store path is not predicted, and + * the write refuses it with `StoreLinkedError`. Live and advisory: the write or removal still decides. + */ + isWritable(definition: AnyCultCacheDocumentDefinition, key: string): boolean { + const registered = this.#requireDefinition(definition); + try { + this.#judge(registered, key, this.#probe(registered, key)); + return true; + } catch { + return false; + } + } + // What the last pull found that no registered type claims, by key ordinal. A later registration claims a record at the next pull. foreignRecords(): readonly CultCacheForeignRecord[] { return [...this.#hydrated.foreign].sort((left, right) => compareOrdinal(left.key, right.key)); @@ -594,6 +687,13 @@ export class CultCache { return [...this.#hydrated.entries.values()].map((entry) => this.#toEnvelope(entry)); } + // The image as the stores hold it, for persisting it whole (a flush): a global loaded from a legacy key is handed out under + // that key until its first write retires it, so a flush cannot leave the store holding it under two keys. `snapshot` is the + // logical image and always names a global by CultCache.GLOBAL_KEY. + storedSnapshot(): CultCacheEnvelope[] { + return [...this.#hydrated.entries.values()].map((entry) => this.#hydrated.legacyGlobals.get(entry.type) ?? this.#toEnvelope(entry)); + } + #registerNow(definition: AnyCultCacheDocumentDefinition): void { // Every registration refusal is a SchemaConflictError naming the new definition's schema id and both schema names. const refuse = (holder: RegisteredDefinition, claimed: CultCacheSchemaCatalogEntry, what: string) => @@ -674,15 +774,58 @@ export class CultCache { this.#stores.push(candidate); } + // One attempt's read for update: the record at the key as its home holds it now, the condition a commit must meet, and a copy of + // the record for apply to change. A store-backed cache asks its store, which reports the record and its version from one read; + // a cache with no store has only the held entry, whose identity is the condition and whose admitted payload is the copy. + async #readForUpdate( + definition: AnyCultCacheDocumentDefinition, + key: string, + ): Promise<{ expected: WriteCondition; copy: unknown }> { + const registered = this.#requireDefinition(definition); + this.#requireGlobalKey(registered, key); + const type = definition.type; + this.#refuseLegacyConditional(type); + const home = this.#homeStore(type); + if (home === undefined) { + const held = this.#hydrated.entries.get(this.#entryId(type, key)); + if (held === undefined && this.#heldUnderOtherType(this.#hydrated, type, key) !== undefined) { + throw new Error(`Record ${key} is not a "${type}"; update cannot copy it.`); + } + return { expected: { held }, copy: held ? registered.formatter.decode(this.#cloneBytes(held.payload)).value : undefined }; + } + if (home.read === undefined || home.commitIf === undefined) { + throw new Error(`CultCache backing store for type "${type}" cannot write conditionally; update needs a store with read and commitIf.`); + } + const found = await home.read(key); + if (found === undefined) { + return { expected: { version: null }, copy: undefined }; + } + if (this.#resolveDefinitionForEnvelope(found.envelope) !== registered) { + throw new Error(`Record ${key} is not a "${type}"; update cannot copy it.`); + } + return { expected: { version: found.version }, copy: registered.formatter.decode(this.#cloneBytes(found.envelope.payload)).value }; + } + + #putNow( + definition: TDefinition, + key: string, + value: CultCacheDocumentValue, + ): Promise<{ outcome: "committed"; value: CultCacheDocumentValue }>; + #putNow( + definition: TDefinition, + key: string, + value: CultCacheDocumentValue, + expected: WriteCondition, + ): Promise<{ outcome: "committed"; value: CultCacheDocumentValue } | { outcome: "mismatch" }>; async #putNow( definition: TDefinition, key: string, value: CultCacheDocumentValue, - ): Promise> { + expected?: WriteCondition, + ): Promise<{ outcome: "committed"; value: CultCacheDocumentValue } | { outcome: "mismatch" }> { const registered = this.#requireDefinition(definition); - this.#requireGlobalKey(registered, key); const payload = registered.formatter.encode(value); - const parsed = registered.formatter.decode(payload) as CultCacheDocumentValue; + const { value: parsed, unreadSlots } = registered.formatter.decode(payload) as { value: CultCacheDocumentValue; unreadSlots?: NotedSlots }; const entry: CultCacheEnvelope = { key, type: definition.type, @@ -691,29 +834,55 @@ export class CultCache { schemaId: registered.catalogEntry.schemaId, catalogEntry: registered.catalogEntry, }; - await this.#writeNow(registered, entry, parsed); - return parsed; + const outcome = await this.#writeNow(registered, entry, parsed, unreadSlots, expected); + return outcome === "committed" ? { outcome, value: parsed } : { outcome }; } // Validates the whole write (home, global singleton, name and index accessors) before the store // is touched; once the store accepts, applying to the cache has no fallible step. - async #writeNow(registered: RegisteredDefinition, entry: CultCacheEnvelope, value: unknown): Promise { + // With a condition, the write lands only if the record at the key is still the one the caller read: the store's version + // (commitIf), or for a cache with no store the held entry itself. Otherwise nothing is written or installed and the answer is + // "mismatch". Without one the write is blind, as put's. + async #writeNow( + registered: RegisteredDefinition, + entry: CultCacheEnvelope, + value: unknown, + unreadSlots?: NotedSlots, + expected?: WriteCondition, + ): Promise<"committed" | "mismatch"> { const home = this.#homeStore(entry.type); - const retyped = this.#refuseRetype(entry); - if (retyped !== undefined && this.#homeStore(retyped.type) !== home) { - // The write owns the held record's id and would replace it, but the held record is in another store; the push - // reaches only this type's home, so the old record would stay in its own file and load as a second holder. - throw new SchemaConflictError( - schemaIdForEnvelope(entry), - [retyped.type, entry.type], - entry.key, - `CultCache holds record "${entry.key}" as type "${retyped.type}" in another store than the home of type "${entry.type}", so the rename cannot replace it; remove the record first.`, - ); - } - const hydrated = this.#admitEntry(this.#hydrated, registered, entry, value); + const retyped = this.#judge(registered, entry.key, entry); + const hydrated = this.#admitEntry(this.#hydrated, registered, entry, value, unreadSlots); this.#refuseIndexSharing(this.#hydrated, entry.type, registered.catalogEntry.schemaName, entry.key, hydrated.lookups); const legacy = this.#hydrated.legacyGlobals.get(entry.type); - await home?.push(entry); + // A conditional write over a legacy global is refused here too (a pull while apply awaited can have loaded one): the condition + // was read from the global's own key, the record sits under its legacy key, and the retire below would delete it unseen. + if (expected !== undefined) { + this.#refuseLegacyConditional(entry.type); + } + if (expected !== undefined && "held" in expected) { + // The condition was fixed at read for a cache with no store. A store attached since has its own record at the key that + // this read never saw, so the held identity no longer judges the write: lose the race and read again. + if (home !== undefined || this.#hydrated.entries.get(this.#entryId(entry.type, entry.key)) !== expected.held) { + return "mismatch"; + } + } + if (expected !== undefined && "version" in expected) { + if (home?.commitIf === undefined) { + throw new Error(`CultCache backing store for type "${entry.type}" cannot write conditionally; update needs a store with read and commitIf.`); + } + const committed = await home.commitIf(entry, expected.version); + if (committed.outcome === "mismatch") { + return "mismatch"; + } + hydrated.storedAt = committed.storedAt; + } else { + // A store that re-stamps a write (storedAt later than the record it replaces) reports the stamp; the cache holds that one. + const pushed = await home?.push(entry); + if (typeof pushed === "object" && pushed !== null && typeof pushed.storedAt === "string" && !Number.isNaN(Date.parse(pushed.storedAt))) { + hydrated.storedAt = pushed.storedAt; + } + } if (home && legacy) { await this.#retireLegacyGlobal(home, legacy, entry); } @@ -723,6 +892,96 @@ export class CultCache { this.#removeEntry(this.#hydrated, this.#definitions.get(retyped.type)!, retyped); } this.#installEntry(this.#hydrated, registered, hydrated); + return "committed"; + } + + // The one rule that a conditional write (update, whichever entry reaches it) never touches a global loaded from a legacy key: + // only putGlobal retires that key. Asked before apply runs and again at the write. + #refuseLegacyConditional(type: string): void { + if (this.#hydrated.legacyGlobals.has(type)) { + throw new Error( + `CultCache global "${type}" was loaded from a legacy key; write it once with putGlobal to retire its legacy key.`, + ); + } + } + + // An envelope of the definition's type at the key, carrying nothing: what the judge asks about a write or a removal by that type. + #probe(registered: RegisteredDefinition, key: string): CultCacheEnvelope { + return { + key, type: registered.definition.type, payload: documentPayloadFromBytes(new Uint8Array(0)), storedAt: "", + schemaId: registered.catalogEntry.schemaId, catalogEntry: registered.catalogEntry, + }; + } + + // The one judge of whether a key may be written (a written envelope) or removed (none), before any store is touched: it throws the + // refusal, and #writeNow, #deleteNow and isWritable all ask it, so the query cannot disagree with the write. For a write it + // returns the held record of another type that the write owns and so restamps, if any. A refusal names keys and schemas, + // never a payload or a slot value. + #judge(registered: RegisteredDefinition, key: string, written?: CultCacheEnvelope): HydratedEntry | undefined { + const type = registered.definition.type; + const held = this.#hydrated.entries.get(this.#entryId(type, key)); + if (written === undefined) { + // A removal is by key: the record it removes is the one held at the key, of this type or of another type whose schema id + // this type owns (the record a write of this type would replace). A held record of a type it does not own is refused. + this.#refuseForeignKey(key, type, "removed"); + this.#refuseUnread(held); + if (held !== undefined) { + return undefined; + } + const other = this.#heldUnderOtherType(this.#hydrated, type, key); + if (other === undefined) { + return undefined; + } + if (!writtenOwns(this.#probe(registered, key), schemaIdForEnvelope(other))) { + throw new SchemaConflictError( + registered.catalogEntry.schemaId, + [other.type, type], + key, + `CultCache holds record "${key}" as type "${other.type}", so type "${type}" cannot remove it.`, + ); + } + this.#refuseUnread(other); + return other; + } + this.#requireGlobalKey(registered, key); + const retyped = this.#refuseRetype(written); + this.#refuseUnread(held); + this.#refuseUnread(retyped); + if (retyped !== undefined && this.#homeStore(retyped.type) !== this.#homeStore(type)) { + // The write owns the held record's id and would replace it, but the held record is in another store; the push + // reaches only this type's home, so the old record would stay in its own file and load as a second holder. + throw new SchemaConflictError( + schemaIdForEnvelope(written), + [retyped.type, type], + key, + `CultCache holds record "${key}" as type "${retyped.type}" in another store than the home of type "${type}", so the rename cannot replace it; remove the record first.`, + ); + } + return retyped; + } + + // The one refusal of a record held read-only for the slots its payload carries past this build's: a write or a removal of + // its key. It names the type and the counts, never a slot value. A mark that is explained (see #admitEntry) is consented to. + #unreadRefusal(held: HydratedEntry | undefined): SchemaConflictError | undefined { + const unread = held?.unreadSlots; + if (held === undefined || unread === undefined || unread.explained) { + return undefined; + } + return new SchemaConflictError( + held.storedSchemaId ?? held.schemaId ?? held.type, + [held.catalogEntry.schemaName], + held.key, + `${held.catalogEntry.schemaName} record ${held.key} is read-only: ${unread.typeName} was read with ${unread.slotCount} slots where this build declares ${unread.declaredCount} ` + + `(${unread.occurrences} object(s) in the record). Writing it would drop data a newer writer stored; read it with types that declare those slots, or write it from that build.`, + unread, + ); + } + + #refuseUnread(held: HydratedEntry | undefined): void { + const refusal = this.#unreadRefusal(held); + if (refusal !== undefined) { + throw refusal; + } } // The one key-identity judge of the cache: may this write land at its key, given what the cache holds there under @@ -782,18 +1041,20 @@ export class CultCache { key: string, ): Promise { const registered = this.#requireDefinition(definition); - const entry = this.#hydrated.entries.get(this.#entryId(definition.type, key)); + // variants:ruling:unheld-removal-refused-at-commit: a key a store carries foreign is refused, a key held by nobody is the + // existing false; a key held as another type is removed by an owner of its id and refused to any other type. + // Removal is by key, in one store call: the record held at the key of this type, else the one of another type this type owns. + const owned = this.#judge(registered, key); + const entry = this.#hydrated.entries.get(this.#entryId(definition.type, key)) ?? owned; if (!entry) { - // variants:ruling:unheld-removal-refused-at-commit: a key a store carries foreign is refused, a key held under - // another type or by nobody is the existing false. - this.#refuseForeignKey(key, definition.type, "removed"); return false; } - const home = this.#homeStore(definition.type); - await home?.delete(this.#hydrated.legacyGlobals.get(definition.type) ?? this.#toEnvelope(entry)); - this.#hydrated.legacyGlobals.delete(definition.type); - this.#removeEntry(this.#hydrated, registered, entry); + // The cache drops the record only once the store has removed it. + const home = this.#homeStore(entry.type); + await home?.delete(this.#hydrated.legacyGlobals.get(entry.type) ?? this.#toEnvelope(entry)); + this.#hydrated.legacyGlobals.delete(entry.type); + this.#removeEntry(this.#hydrated, this.#definitions.get(entry.type)!, entry); return true; } @@ -820,7 +1081,7 @@ export class CultCache { return { encode: (value: unknown) => this.#cloneBytes(encodeDocumentPayload(definition, value)), - decode: (payload: DocumentPayload) => decodeDocumentPayload(definition, this.#cloneBytes(payload)), + decode: (payload: DocumentPayload) => decodeDocumentPayloadMarked(definition, this.#cloneBytes(payload)), }; } @@ -1000,6 +1261,7 @@ export class CultCache { registered: RegisteredDefinition, entry: CultCacheEnvelope, value: unknown, + noted?: NotedSlots, ): HydratedEntry { if (registered.global) { const currentGlobalKey = state.globalKeys.get(entry.type); @@ -1020,6 +1282,8 @@ export class CultCache { catalogEntry: registered.catalogEntry, value, lookups, + storedSchemaId: entry.schemaId ?? entry.type, + ...(noted ? { unreadSlots: { ...noted, explained: explainedBy(noted, registered.catalogEntry, entry.schemaId ?? entry.type, entry.catalogEntry) } } : {}), }; } diff --git a/packages/cultcache-ts/src/document-formatter.ts b/packages/cultcache-ts/src/document-formatter.ts index 799b2558..92d45df1 100644 --- a/packages/cultcache-ts/src/document-formatter.ts +++ b/packages/cultcache-ts/src/document-formatter.ts @@ -1,7 +1,7 @@ import { decode, encode } from "@msgpack/msgpack"; import { MAX_PAYLOAD_DEPTH } from "./payload-depth"; -import type { SlotsCodec } from "./codec"; +import { decodeNoting, type NotedSlots, type SlotsCodec } from "./codec"; import type { AnyCultCacheDocumentDefinition, CultCacheDocumentFormatter } from "./types"; import { readWire, writeWire } from "./wire"; @@ -16,6 +16,13 @@ export function resolveDocumentFormatter(definition: AnyCultCacheDocumentDefinit return definition.formatter ?? (definition.codec ? codecFormatter(definition.codec) : plainDocumentFormatter); } +// The one decode of a document's bytes, with the slots it read past. A plain formatter or a user formatter has no slot codec +// and carries everything, so it notes nothing. +export function decodeMarked(definition: AnyCultCacheDocumentDefinition, bytes: Uint8Array): { value: unknown; unreadSlots?: NotedSlots } { + const formatter = resolveDocumentFormatter(definition); + return decodeNoting(() => formatter.decode(bytes)); +} + const plainDocumentFormatter: CultCacheDocumentFormatter = { // The encoder's own depth refusal (100 by default) is set above the limit, so the payload walk decides. encode: (value: unknown) => encode(value, { maxDepth: MAX_PAYLOAD_DEPTH * 2 }), diff --git a/packages/cultcache-ts/src/document.ts b/packages/cultcache-ts/src/document.ts index 10af96a6..cb1ef589 100644 --- a/packages/cultcache-ts/src/document.ts +++ b/packages/cultcache-ts/src/document.ts @@ -1,5 +1,6 @@ -import { resolveDocumentFormatter } from "./document-formatter"; -import { checkPayloadDepth } from "./payload-depth"; +import type { NotedSlots } from "./codec"; +import { decodeMarked, resolveDocumentFormatter } from "./document-formatter"; +import { checkPayloadDepth, checkPayloadIsOneValue } from "./payload-depth"; import type { AnyCultCacheDocumentDefinition, CultCacheSchemaCatalogMember, @@ -118,10 +119,24 @@ export function decodeDocumentPayload { + return decodeDocumentPayloadMarked(definition, payload).value; +} + +// The same read with the slots it read past (cult-cache-ts slot-readonly). The depth check runs first: a payload past the +// limit is refused, never marked. +export function decodeDocumentPayloadMarked( + definition: TDefinition, + payload: DocumentPayload, +): { value: CultCacheDocumentValue; unreadSlots?: NotedSlots } { const bytes = documentPayloadBytes(payload); - if (writesLibraryMessagePack(definition)) checkPayloadDepth(bytes, `document type "${definition.type}" payload`); - const decoded = resolveDocumentFormatter(definition).decode(bytes); - return definition.schema.parse(decoded) as CultCacheDocumentValue; + if (writesLibraryMessagePack(definition)) { + // A payload is exactly one MessagePack value: bytes after the first are refused by count before any formatter reads, + // so a tail is refused before any slot is marked. + checkPayloadIsOneValue(bytes, `document type "${definition.type}" payload`); + } + const { value: decoded, unreadSlots } = decodeMarked(definition, bytes); + const value = definition.schema.parse(decoded) as CultCacheDocumentValue; + return unreadSlots ? { value, unreadSlots } : { value }; } // The one writer: the definition's schema first, then its formatter, codec or plain map. diff --git a/packages/cultcache-ts/src/index.ts b/packages/cultcache-ts/src/index.ts index f55dd049..8d726e3d 100644 --- a/packages/cultcache-ts/src/index.ts +++ b/packages/cultcache-ts/src/index.ts @@ -1,2 +1,3 @@ export * from "./browser"; export { SingleFileMessagePackBackingStore, readSingleFileStore, type SingleFileStoreRead } from "./single-file-messagepack-backing-store"; +export { StoreLinkedError } from "./store-file"; diff --git a/packages/cultcache-ts/src/payload-depth.ts b/packages/cultcache-ts/src/payload-depth.ts index 40ca7afe..45b8d5ca 100644 --- a/packages/cultcache-ts/src/payload-depth.ts +++ b/packages/cultcache-ts/src/payload-depth.ts @@ -12,8 +12,9 @@ export class CultCachePayloadDepthError extends Error { // Walks the headers of one MessagePack value iteratively (a count stack bounded by the limit, scalars skipped by length, // nothing read) and throws at the first container past the limit. The message names the subject, the depth and the limit, -// never payload bytes. A payload that ends early is the reader's to refuse. -export function checkPayloadDepth(bytes: Uint8Array, subject = "document payload"): void { +// never payload bytes. Returns the offset where the first value ends, or undefined when the payload ends early (the +// reader's to refuse); what follows that offset is never walked, and whether it may exist is the caller's rule. +export function checkPayloadDepth(bytes: Uint8Array, subject = "document payload"): number | undefined { const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); const remaining = new Float64Array(MAX_PAYLOAD_DEPTH); let depth = 0; @@ -33,12 +34,12 @@ export function checkPayloadDepth(bytes: Uint8Array, subject = "document payload else if (code <= 0x8f) { container = true; header = 1; children = 2 * (code & 0x0f); } else if (code <= 0x9f) { container = true; header = 1; children = code & 0x0f; } else if (code <= 0xbf) skip = 1 + (code & 0x1f); - else if (code === 0xc4 || code === 0xd9) { const n = width(1); if (n === undefined) return; skip = 2 + n; } - else if (code === 0xc5 || code === 0xda) { const n = width(2); if (n === undefined) return; skip = 3 + n; } - else if (code === 0xc6 || code === 0xdb) { const n = width(4); if (n === undefined) return; skip = 5 + n; } - else if (code === 0xc7) { const n = width(1); if (n === undefined) return; skip = 3 + n; } - else if (code === 0xc8) { const n = width(2); if (n === undefined) return; skip = 4 + n; } - else if (code === 0xc9) { const n = width(4); if (n === undefined) return; skip = 6 + n; } + else if (code === 0xc4 || code === 0xd9) { const n = width(1); if (n === undefined) return undefined; skip = 2 + n; } + else if (code === 0xc5 || code === 0xda) { const n = width(2); if (n === undefined) return undefined; skip = 3 + n; } + else if (code === 0xc6 || code === 0xdb) { const n = width(4); if (n === undefined) return undefined; skip = 5 + n; } + else if (code === 0xc7) { const n = width(1); if (n === undefined) return undefined; skip = 3 + n; } + else if (code === 0xc8) { const n = width(2); if (n === undefined) return undefined; skip = 4 + n; } + else if (code === 0xc9) { const n = width(4); if (n === undefined) return undefined; skip = 6 + n; } else if (code === 0xca || code === 0xce || code === 0xd2) skip = 5; else if (code === 0xcb || code === 0xcf || code === 0xd3) skip = 9; else if (code === 0xcc || code === 0xd0) skip = 2; @@ -47,10 +48,10 @@ export function checkPayloadDepth(bytes: Uint8Array, subject = "document payload else if (code === 0xd6) skip = 6; else if (code === 0xd7) skip = 10; else if (code === 0xd8) skip = 18; - else if (code === 0xdc) { const n = width(2); if (n === undefined) return; container = true; header = 3; children = n; } - else if (code === 0xdd) { const n = width(4); if (n === undefined) return; container = true; header = 5; children = n; } - else if (code === 0xde) { const n = width(2); if (n === undefined) return; container = true; header = 3; children = 2 * n; } - else { const n = width(4); if (n === undefined) return; container = true; header = 5; children = 2 * n; } + else if (code === 0xdc) { const n = width(2); if (n === undefined) return undefined; container = true; header = 3; children = n; } + else if (code === 0xdd) { const n = width(4); if (n === undefined) return undefined; container = true; header = 5; children = n; } + else if (code === 0xde) { const n = width(2); if (n === undefined) return undefined; container = true; header = 3; children = 2 * n; } + else { const n = width(4); if (n === undefined) return undefined; container = true; header = 5; children = 2 * n; } if (container) { if (depth === MAX_PAYLOAD_DEPTH) throw new CultCachePayloadDepthError(MAX_PAYLOAD_DEPTH + 1, MAX_PAYLOAD_DEPTH, subject); @@ -63,8 +64,20 @@ export function checkPayloadDepth(bytes: Uint8Array, subject = "document payload position += skip; } + // A scalar whose declared length runs past the payload is an early end, not a value. + if (position > bytes.length) return undefined; // One element is complete; close every container it was the last element of. while (depth > 0 && --remaining[depth - 1] === 0) depth--; - if (depth === 0) return; + if (depth === 0) return position; } + return undefined; +} + +// The one rule that a payload is exactly one MessagePack value: the depth walk, then a refusal of bytes after the first +// value, by count, naming the subject, from the walk's end. The tail is never walked or echoed; a payload that ends early +// stays the decoder's to refuse. decodeDocumentPayload and the inspector speak this clause and no other. +export function checkPayloadIsOneValue(bytes: Uint8Array, subject: string): number | undefined { + const end = checkPayloadDepth(bytes, subject); + if (end !== undefined && end < bytes.length) throw new Error(`${subject} has ${bytes.length - end} bytes after its first MessagePack value.`); + return end; } diff --git a/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts b/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts index 5dfcfc74..03b93683 100644 --- a/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts +++ b/packages/cultcache-ts/src/single-file-messagepack-backing-store.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { lstatSync, readFileSync } from "node:fs"; import { mkdir } from "node:fs/promises"; import { dirname, resolve } from "node:path"; @@ -55,14 +56,42 @@ export class SingleFileMessagePackBackingStore implements CacheBackingStore { return disk.envelopes; } - async push(entry: CultCacheEnvelope): Promise { - await this.#enqueue(async () => { + async push(entry: CultCacheEnvelope): Promise<{ storedAt: string }> { + return this.#enqueue(async () => { const disk = this.#readDisk(); this.#format = disk.format; const existing = disk.envelopes; requireOneTypePerKey(this.filePath, [entry], existing); - const filtered = existing.filter((candidate) => candidate.key !== entry.key); - await this.#writeAll([entry], filtered); + const [written] = await this.#writeAll([entry], existing); + return { storedAt: written!.storedAt }; + }); + } + + // The record at the key as the file holds it now, with its content version, from one read. Like pullAll it takes no lock: a + // reader sees a whole file (the commit renames one over the other), and the version it reports is the one commitIf judges. + async read(key: string): Promise<{ envelope: CultCacheEnvelope; version: string } | undefined> { + const envelope = readSingleFileStore(this.filePath).envelopes.find((candidate) => candidate.key === key); + return envelope === undefined ? undefined : { envelope, version: recordVersion(envelope) }; + } + + // Writes `entry` the way push does, but only if the record at its key (of any type) still has `expectedVersion`; null expects + // none. The comparison and the write are one hold of the write lock, so no other writer lands between them. + async commitIf( + entry: CultCacheEnvelope, + expectedVersion: string | null, + ): Promise<{ outcome: "committed"; storedAt: string } | { outcome: "mismatch" }> { + return this.#enqueue(async () => { + const disk = this.#readDisk(); + this.#format = disk.format; + const existing = disk.envelopes; + const current = existing.find((candidate) => candidate.key === entry.key); + if ((current === undefined ? null : recordVersion(current)) !== expectedVersion) { + return { outcome: "mismatch" as const }; + } + + requireOneTypePerKey(this.filePath, [entry], existing); + const [written] = await this.#writeAll([entry], existing); + return { outcome: "committed" as const, storedAt: written!.storedAt }; }); } @@ -82,31 +111,24 @@ export class SingleFileMessagePackBackingStore implements CacheBackingStore { async pushAll(entries: CultCacheEnvelope[], options: PushAllOptions = {}): Promise { await this.#enqueue(async () => { - // A flush of the whole store writes the header the file on disk carries, read now: a file marked for element ids - // stays marked, and one that is not (or is gone or legacy) is written unmarked. The file is read by the same - // reader as `pullAll`, so one it would refuse (not exactly one store, a variant, a body it cannot decode) is refused - // and left as it is. A soft flush asks that reader too: nothing at the path is written, a store is left alone. - try { - const disk = this.#readDisk(); - if (options.soft && !disk.absent) { - return; - } - - this.#format = disk.format; - } catch (error) { - // Only a dangling link reaches here with ENOENT: it holds no store, so a whole-store flush has no header to keep and - // replaces the link (R3 pins this). A soft flush does not replace what it cannot read, and every merging writer - // has already refused the link when it read the current entries. - if (options.soft || (error as NodeJS.ErrnoException).code !== "ENOENT") { - throw error; - } - - this.#format = STORE_FORMAT_VERSION; + // A flush writes `entries` onto the store as it is under the lock, the way a push does: a record under the same key + // is replaced and no other is removed, so a cache whose image is stale (or empty, never pulled) cannot drop what + // another writer added. The store is read by the same reader as `pullAll`, so one it would refuse (not exactly one + // store, a variant, a body it cannot decode) is refused and left as it is. A soft flush asks that reader too: nothing + // at the path is written, a store is left alone. + // The reader owns the verdict on what is at the path: a dangling link is refused here as it is for open and push. + const disk = this.#readDisk(); + if (options.soft && !disk.absent) { + return; } - requireOneTypePerKey(this.filePath, entries, []); + this.#format = disk.format; + const held = disk.envelopes; + // A key is held once: of same-type entries sharing a key in one batch, the last is kept. - await this.#writeAll([...new Map(entries.map((entry) => [entry.key, entry])).values()], []); + const supplied = [...new Map(entries.map((entry) => [entry.key, entry])).values()]; + requireOneTypePerKey(this.filePath, entries, held); + await this.#writeAll(supplied, held); }); } @@ -133,13 +155,67 @@ export class SingleFileMessagePackBackingStore implements CacheBackingStore { return result; } - async #writeAll(supplied: CultCacheEnvelope[], arrived: CultCacheEnvelope[]): Promise { + // Writes the supplied records onto the arrived ones and returns the supplied records as written. A supplied record whose + // schema id and payload bytes differ from the arrived one it replaces is never stamped earlier than the record it replaces, so a content version that includes storedAt + // cannot repeat across writes (variants:ruling:storedat-on-change): one not later than the arrived stamp is written one + // millisecond after it. + async #writeAll(supplied: CultCacheEnvelope[], arrived: CultCacheEnvelope[]): Promise { + const arrivedByKey = new Map(arrived.map((entry) => [entry.key, entry])); + const written = supplied.map((entry) => { + const prior = arrivedByKey.get(entry.key); + if (prior === undefined) { + return entry; + } + // A rewrite of the identical record (same schema id and payload bytes) keeps the stamp and so the version it has; the + // writer's type and catalog entry still go down with it. + if (sameContent(entry, prior)) { + return { ...entry, storedAt: prior.storedAt }; + } + return { ...entry, storedAt: laterStoredAt(entry.storedAt, prior.storedAt) }; + }); await mkdir(dirname(this.filePath), { recursive: true }); - await commitStoreFile(this.filePath, encode(encodeSnapshot(supplied, arrived, this.#format))); + await commitStoreFile(this.filePath, encode(encodeSnapshot(written, arrived, this.#format))); + return written; } } +// The content version of a stored record: the lowercase hex SHA-256 of its persisted encoding, the MessagePack array +// [key, schemaId, storedAt, payload] as the store writes it (the C# reference's StoredVersion). Private to this module so +// node:crypto stays out of the browser-safe cache. +export function recordVersion(entry: CultCacheEnvelope): string { + return createHash("sha256") + .update(encode([entry.key, schemaIdForEnvelope(entry), entry.storedAt, documentPayloadBytes(entry.payload)])) + .digest("hex"); +} + +function sameContent(a: CultCacheEnvelope, b: CultCacheEnvelope): boolean { + if (schemaIdForEnvelope(a) !== schemaIdForEnvelope(b)) { + return false; + } + const x = documentPayloadBytes(a.payload); + const y = documentPayloadBytes(b.payload); + return x.length === y.length && x.every((byte, i) => byte === y[i]); +} + +// `storedAt` if it is later than `prior`, else `prior` plus one millisecond. A stamp that does not parse is not later; a prior +// that does not parse leaves nothing to be later than. +function laterStoredAt(storedAt: string, prior: string): string { + const priorMs = Date.parse(prior); + if (Number.isNaN(priorMs)) { + return storedAt; + } + const storedMs = Date.parse(storedAt); + if (!(Number.isNaN(storedMs) || storedMs <= priorMs)) { + return storedAt; + } + // The stamp is ISO-8601 with a four-digit year; one past 9999-12-31T23:59:59.999Z would print as an extended year. + if (priorMs + 1 > Date.UTC(9999, 11, 31, 23, 59, 59, 999)) { + throw new RangeError(`CultCache cannot stamp a write later than the record it replaces (${prior}): the stamp would leave the four-digit years.`); + } + return new Date(priorMs + 1).toISOString(); +} + // A store holds a key once, and the reader refuses a file that holds it twice, so a write that would put a second type under a // key (against a record on disk, or within its own batch) is refused before anything is written. Retyping a record is a delete // of the old type followed by a put of the new one. diff --git a/packages/cultcache-ts/src/store-file.ts b/packages/cultcache-ts/src/store-file.ts index 4c7015cc..0488960e 100644 --- a/packages/cultcache-ts/src/store-file.ts +++ b/packages/cultcache-ts/src/store-file.ts @@ -5,7 +5,7 @@ // lazily inside `loadKernel`, so a consumer that never writes a store never loads it. import { closeSync, openSync, constants as fsConstants } from "node:fs"; -import { mkdir, open, rename, rm } from "node:fs/promises"; +import { lstat, mkdir, open, rename, rm } from "node:fs/promises"; import type { FileHandle } from "node:fs/promises"; import { dirname } from "node:path"; @@ -25,6 +25,30 @@ export class StoreReplacedNotDurableError extends Error { } } +/** Thrown when a write is asked of a store path that is a link: a replace there would swap the link for a regular file. */ +export class StoreLinkedError extends Error { + constructor(readonly path: string) { + super(`CultCache store ${path} is a link, and a write at it would replace the link; configure the link's target as the store path.`); + this.name = "StoreLinkedError"; + } +} + +/** Refuses a link at the store path (file or directory link; libuv reports Windows junctions as links). Absent is not a link. */ +export async function refuseLinkedStore(storePath: string): Promise { + let linked = false; + try { + linked = (await lstat(storePath)).isSymbolicLink(); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") { + throw error; + } + } + + if (linked) { + throw new StoreLinkedError(storePath); + } +} + /** The file operations of one commit. Tests record or fail them; production uses `nodeStoreFileOps`. */ export interface StoreFileOps { createExclusive(path: string): Promise; @@ -177,6 +201,7 @@ export async function withStoreWriteLock( ): Promise { const lockPath = `${storePath}.lock`; const timeoutMs = options.timeoutMs ?? LOCK_TIMEOUT_MS; + await refuseLinkedStore(storePath); await mkdir(dirname(lockPath), { recursive: true }); const started = Date.now(); @@ -193,6 +218,8 @@ export async function withStoreWriteLock( } try { + // A link may have been put at the path while the lock was awaited; the look above only saw the path before. + await refuseLinkedStore(storePath); return await operation(); } finally { release(); diff --git a/packages/cultcache-ts/src/store-format.ts b/packages/cultcache-ts/src/store-format.ts index acd10207..885b1c8e 100644 --- a/packages/cultcache-ts/src/store-format.ts +++ b/packages/cultcache-ts/src/store-format.ts @@ -1,6 +1,6 @@ import { decode } from "@msgpack/msgpack"; -import type { CultCacheEnvelope } from "./types"; +import type { CultCacheEnvelope, CultCacheUnreadSlots } from "./types"; export const STORE_FORMAT_VERSION = "cultcache.store.v1"; @@ -83,13 +83,16 @@ export class SchemaConflictError extends Error { readonly schemaId: string; readonly schemaNames: string[]; readonly recordKey: string; + /** Set when the refusal is for unread slots; undefined for every other refusal. */ + readonly unreadSlots?: CultCacheUnreadSlots; - constructor(schemaId: string, schemaNames: string[], recordKey: string, message?: string) { + constructor(schemaId: string, schemaNames: string[], recordKey: string, message?: string, unreadSlots?: CultCacheUnreadSlots) { super(message ?? `Schema id "${schemaId}" cannot describe record "${recordKey}": ${schemaNames.map((name) => `"${name}"`).join(", ")}.`); this.name = "SchemaConflictError"; this.schemaId = schemaId; this.schemaNames = schemaNames; this.recordKey = recordKey; + this.unreadSlots = unreadSlots; } } diff --git a/packages/cultcache-ts/src/stored-at.ts b/packages/cultcache-ts/src/stored-at.ts new file mode 100644 index 00000000..d630a411 --- /dev/null +++ b/packages/cultcache-ts/src/stored-at.ts @@ -0,0 +1,33 @@ +// The one shape a `storedAt` stamp has (tests/vectors/stored-at-stamp/README.md; reference `parse_stored_at` in cultcache-py): an +// RFC 3339 date-time `YYYY-MM-DDThh:mm:ss`, an optional fraction of 1 to 9 digits, and `Z` or `+hh:mm` / `-hh:mm` (hours 00-23, +// minutes 00-59). A real calendar date, seconds 00-59, and an instant within years 0001-9999. Checked here, not left to +// `Date.parse`, which accepts shapes this one does not and whose accepted set differs by engine. +const STAMP = /^([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2}):([0-9]{2}):([0-9]{2})(?:\.[0-9]{1,9})?(?:Z|([+-])([01][0-9]|2[0-3]):([0-5][0-9]))$/; + +const MS_PER_MINUTE = 60_000; + +/** True when `text` is a string that is a stamp. Anything else (missing, empty, wrong type, other shapes) is not. */ +export function isStoredAtStamp(text: unknown): text is string { + if (typeof text !== "string") return false; + const shape = STAMP.exec(text); + if (shape === null) return false; + const [year, month, day, hour, minute, second] = shape.slice(1, 7).map(Number) as [number, number, number, number, number, number]; + const leap = (year % 4 === 0 && year % 100 !== 0) || year % 400 === 0; + const daysInMonth = [31, leap ? 29 : 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31][month - 1]; + if (year < 1 || month < 1 || month > 12 || day < 1 || day > (daysInMonth as number) || hour > 23 || minute > 59 || second > 59) { + return false; + } + // The UTC instant must stay within years 0001-9999, so an offset can push a boundary date out of range. + const local = new Date(0); + local.setUTCFullYear(year, month - 1, day); + local.setUTCHours(hour, minute, second, 0); + const offsetMinutes = shape[7] === undefined ? 0 : (shape[7] === "-" ? -1 : 1) * (Number(shape[8]) * 60 + Number(shape[9])); + const utcMs = local.getTime() - offsetMinutes * MS_PER_MINUTE; + const first = new Date(0); + first.setUTCFullYear(1, 0, 1); + first.setUTCHours(0, 0, 0, 0); + const last = new Date(0); + last.setUTCFullYear(9999, 11, 31); + last.setUTCHours(23, 59, 59, 999); + return utcMs >= first.getTime() && utcMs <= last.getTime(); +} diff --git a/packages/cultcache-ts/src/types.ts b/packages/cultcache-ts/src/types.ts index 6f6c0923..60e0b878 100644 --- a/packages/cultcache-ts/src/types.ts +++ b/packages/cultcache-ts/src/types.ts @@ -99,15 +99,60 @@ export interface CultCacheEnvelope { catalogEntry?: CultCacheSchemaCatalogEntry; } +/** + * What a payload decode found past the slots this build declares, at any depth: the first type read with extra slots, how many + * slots it carried against how many it declares, and how many objects in the record did. It holds no slot value, so a refusal + * can name it. A record carrying it is read-only unless `explained`. + */ +export interface CultCacheUnreadSlots { + /** The first type read with extra slots (the unit arm's arm name for a unit arm). */ + readonly typeName: string; + readonly slotCount: number; + readonly declaredCount: number; + readonly occurrences: number; + /** True when every mark sat in the document's own object. */ + readonly depth0Only: boolean; + /** + * True only when every mark is at depth 0 and the schema comparison of the id the record was stored under reported every slot + * the mark carries (slotCount is at most the highest reported slot plus one): the loss a compatible-id declaration consents to. + */ + readonly explained: boolean; +} + export interface PushAllOptions { soft?: boolean; } export interface CacheBackingStore { pullAll(): Promise; - push(entry: CultCacheEnvelope): Promise; + /** A store that re-stamps the write returns the storedAt it wrote; the cache then holds that stamp, not its own. */ + push(entry: CultCacheEnvelope): Promise; delete(entry: CultCacheEnvelope): Promise; pushAll?(entries: CultCacheEnvelope[], options?: PushAllOptions): Promise; + /** + * The record at the key as the store holds it now, with its content version, from one read; undefined when the store holds + * none. The version is the store's own: the cache carries it from `read` to `commitIf` and never computes it. + */ + read?(key: string): Promise<{ envelope: CultCacheEnvelope; version: string } | undefined>; + /** + * Writes the entry only if the record at its key still has `expectedVersion` (null: no record), judged and written under the + * store's write lock. A mismatch writes nothing. A commit returns the storedAt actually written. + */ + commitIf?( + entry: CultCacheEnvelope, + expectedVersion: string | null, + ): Promise<{ outcome: "committed"; storedAt: string } | { outcome: "mismatch" }>; +} + +/** The outcome of `CultCache.update`: what `apply` wrote, declined to write, or lost to a rival writer. */ +export type CultCacheUpdateResult = + | { outcome: "updated"; document: TValue } + | { outcome: "declined"; document: TValue | undefined } + | { outcome: "lostRace"; document: TValue | undefined }; + +export interface CultCacheUpdateOptions { + /** How many times `apply` may run before a lost race is returned as `lostRace`. An integer of at least 1; default 5. */ + maxAttempts?: number; } export interface CultCacheStoreRegistration { diff --git a/packages/cultcache-ts/src/wire.ts b/packages/cultcache-ts/src/wire.ts index 12bf572a..d351c446 100644 --- a/packages/cultcache-ts/src/wire.ts +++ b/packages/cultcache-ts/src/wire.ts @@ -128,7 +128,6 @@ export function readWire(payload: Uint8Array): unknown { } } - const tree = next(); - if (offset !== payload.length) throw new Error("the MessagePack payload has bytes after its value."); - return tree; + // One value is read; the caller owns what follows it (decodeDocumentPayload refuses a payload with a tail). + return next(); } diff --git a/packages/cultcache-ts/test/cult-cache.test.ts b/packages/cultcache-ts/test/cult-cache.test.ts index b575170f..dc1ddb33 100644 --- a/packages/cultcache-ts/test/cult-cache.test.ts +++ b/packages/cultcache-ts/test/cult-cache.test.ts @@ -15,6 +15,7 @@ import { CultCache } from "../src/cult-cache"; import { inspectCultCacheBytes } from "../src/cult-cache-inspector"; import { defineDocumentRegistry, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; import { SingleFileMessagePackBackingStore, readSingleFileStore, readStoreBytes } from "../src/single-file-messagepack-backing-store"; +import { StoreLinkedError, withStoreWriteLock } from "../src/store-file"; import { SchemaConflictError, StoreUnreadableError, refuseUnreadStoreBytes } from "../src/store-format"; import type { AnyCultCacheDocumentDefinition, CacheBackingStore, CultCacheEnvelope, CultCacheSchema } from "../src/types"; @@ -33,8 +34,7 @@ const cultcacheRsRoot = existsSync(resolve(cultLibRoot, "packages", "cultcache-r ? resolve(cultLibRoot, "packages", "cultcache-rs") : resolve(cultCacheTsRoot, "..", "cultcache-rs"); const rustInteropBinary = resolve( - cultcacheRsRoot, - "target", + process.env.CARGO_TARGET_DIR ?? resolve(cultcacheRsRoot, "target"), "debug", "examples", process.platform === "win32" ? "cultcache_interop.exe" : "cultcache_interop", @@ -1019,6 +1019,7 @@ test("legacyKeyGlobalLoadsWithoutWritingAndFirstWriteLeavesOneGlobal", async () assert.deepEqual(reloaded.getRequiredGlobal(settingsDocument), { t: "new" }); // Deleting an adopted global removes the legacy record. + await rm(storePath); await new SingleFileMessagePackBackingStore(storePath).pushAll([legacy]); const deleting = build(); await deleting.pullAllBackingStores(); @@ -1026,10 +1027,30 @@ test("legacyKeyGlobalLoadsWithoutWritingAndFirstWriteLeavesOneGlobal", async () assert.deepEqual(await new SingleFileMessagePackBackingStore(storePath).pullAll(), []); // Two globals of one type on disk, under any keys, are refused. + await rm(storePath); await new SingleFileMessagePackBackingStore(storePath).pushAll([legacy, { ...legacy, key: CultCache.GLOBAL_KEY }]); await assert.rejects(build().pullAllBackingStores(), /has multiple persisted entries/u); }); +test("a flush of a cache that loaded a legacy-key global leaves the store reopenable", async () => { + const settingsDocument = defineDocumentType({ type: "settings", schema: z.object({ t: z.string() }), global: true }); + const storePath = join(await mkdtemp(join(tmpdir(), "cultcache-legacy-flush-")), "generic.cc"); + const store = new SingleFileMessagePackBackingStore(storePath); + const build = () => CultCache.builder().withRegistry(defineDocumentRegistry(settingsDocument)).withGenericStore(store).build(); + const seed = CultCache.builder().withRegistry(defineDocumentRegistry(settingsDocument)).build(); + await seed.putGlobal(settingsDocument, { t: "old" }); + await store.push({ ...seed.getRequiredEnvelope(settingsDocument, CultCache.GLOBAL_KEY), key: "legacy" }); + + const cache = build(); + await cache.pullAllBackingStores(); + assert.equal(cache.snapshot()[0]!.key, CultCache.GLOBAL_KEY, "the logical image names the global by the global key"); + await store.pushAll(cache.storedSnapshot()); + assert.deepEqual((await store.pullAll()).map((entry) => entry.key), ["legacy"], "the flush wrote the record under the key the store holds it"); + const reopened = build(); + await reopened.pullAllBackingStores(); + assert.deepEqual(reopened.getRequiredGlobal(settingsDocument), { t: "old" }); +}); + test("concurrentPutAndAttachCannotLandRecordInNonHomeStore", async () => { const settingsDocument = defineDocumentType({ type: "settings", schema: z.object({ t: z.string() }) }); const tempDir = await mkdtemp(join(tmpdir(), "cultcache-put-attach-race-")); @@ -1252,6 +1273,48 @@ test("CultCache v1 MessagePack stores are readable across TS, Rust, C#, and Pyth } }); +// A Bifrost-shaped store (a union field, nested bytes) written by one of C# and Rust is opened by the other: C# reads through +// its store, Rust through the read-only snapshot door, which takes no lock and writes nothing. Leftover staging files from an +// interrupted write of either runtime sit beside the store; a read neither deletes them nor lets them change what it reads, and +// the store file's bytes are the same after every read. +test("A store written by C# or Rust with a union and nested bytes is read back equal by the other, read-only, past leftover staging files", async () => { + await buildInteropPeers(); + const tempDir = await mkdtemp(join(tmpdir(), "cultcache-shapes-")); + const rustRead = (file: string) => runJsonCommand("rust-read-shapes", rustInteropBinary, ["read-shapes", "--file", file], cultcacheRsRoot); + const csharpRead = (file: string) => runJsonCommand("csharp-read-shapes", dotnetCommand, [csharpInteropDll, "read-shapes", "--file", file], cultLibRoot); + const peers = [ + { + writer: "csharp", + write: (file: string) => runJsonCommand("csharp-write-shapes", dotnetCommand, [csharpInteropDll, "write-shapes", "--file", file], cultLibRoot), + read: rustRead, + }, + { + writer: "rust", + write: (file: string) => runJsonCommand("rust-write-shapes", rustInteropBinary, ["write-shapes", "--file", file], cultcacheRsRoot), + read: csharpRead, + }, + ]; + for (const { writer, write, read } of peers) { + const file = join(tempDir, `${writer}.cc`); + const written = await write(file); + assert.equal(written.length, 2); + // The leftovers an interrupted write leaves: C# stages as `...tmp`, Rust as `..tmp`; both hold partial bytes. + const leftovers = [ + join(tempDir, `.${writer}.cc.0123456789abcdef0123456789abcdef.tmp`), + join(tempDir, `${writer}.cc.00000000-0000-4000-8000-000000000000.tmp`), + ]; + for (const leftover of leftovers) { + await writeFile(leftover, Buffer.from([0x93, 0xb2])); + } + const before = await readFile(file); + assert.deepEqual(await read(file), written, `${writer}'s store did not read back equal`); + assert.deepEqual(await readFile(file), before, `reading ${writer}'s store changed its bytes`); + for (const leftover of leftovers) { + assert.ok(await exists(leftover), `reading ${writer}'s store removed ${leftover}`); + } + } +}); + // Records and catalog entries a runtime does not own survive its writes. C# writes a store and each other runtime writes its own // note into it: C# reads its note back, and its record and catalog entry are as C# wrote them. C# then writes again, and the other // runtime's record and catalog entry are as that runtime wrote them. Identity is compared on the decoded MessagePack values, every @@ -2043,7 +2106,7 @@ async function readTsInteropStore(file: string): Promise { async function buildInteropPeers(): Promise { if (!(await exists(rustInteropBinary))) { - await execAsync(`"${cargoCommand}" build --quiet --example cultcache_interop`, { + await execAsync(`"${cargoCommand}" build --locked --quiet --example cultcache_interop`, { cwd: cultcacheRsRoot, }); } @@ -2261,7 +2324,7 @@ test("a stamp this runtime reads, followed by bytes that are not UTF-8, gets the } }); -test("pushAll leaves a store it reads alone when soft, and replaces one it reads otherwise", async () => { +test("pushAll leaves a store it reads alone when soft, and writes onto one it reads otherwise", async () => { const vector = await readFile(resolve(cultLibRoot, "tests", "vectors", "document-variants-c2a", "v3-base.msgpack")); const envelope = { key: "probe", type: "probe", payload: documentPayloadFromBytes(new Uint8Array([0x80])), storedAt: "2026-10-22T00:00:00Z" }; const existing = join(await mkdtemp(join(tmpdir(), "cultcache-pushall-soft-")), "store.msgpack"); @@ -2273,7 +2336,7 @@ test("pushAll leaves a store it reads alone when soft, and replaces one it reads const store = new SingleFileMessagePackBackingStore(own); await store.pushAll([envelope]); await store.pushAll([{ ...envelope, key: "second" }]); - assert.deepEqual((await store.pullAll()).map((entry) => entry.key), ["second"]); + assert.deepEqual((await store.pullAll()).map((entry) => entry.key), ["probe", "second"]); }); test("SingleFileMessagePackBackingStore reads only a missing store as empty", async () => { @@ -2404,12 +2467,36 @@ test("SingleFileMessagePackBackingStore refuses a dangling symlink at its path a () => store.pullAll(), (error: NodeJS.ErrnoException) => error.code === "ENOENT" && error.message.includes("symbolic link") && error.message.includes(file), ); - await assert.rejects(() => store.push(envelope), (error: NodeJS.ErrnoException) => error.code === "ENOENT"); - await assert.rejects(() => store.pushAll([envelope], { soft: true }), (error: NodeJS.ErrnoException) => error.code === "ENOENT"); + await assert.rejects(() => store.push(envelope), (error: unknown) => error instanceof StoreLinkedError); + await assert.rejects(() => store.pushAll([envelope], { soft: true }), (error: unknown) => error instanceof StoreLinkedError); assert.ok((await lstat(file)).isSymbolicLink()); assert.deepEqual(await readdir(volume), []); }); +// The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse. +test("isWritable over a store at a live link answers true while the cache write is refused", { skip: process.platform === "win32" }, async () => { + const dir = await mkdtemp(join(tmpdir(), "cultcache-writable-link-")); + const volume = join(dir, "volume"); + await mkdir(volume); + const target = join(volume, "store.cc"); + const link = join(dir, "link.cc"); + const document = defineDocumentType({ type: "note", schema: z.object({ text: z.string() }) }); + await CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(target)).build() + .put(document, "held", { text: "held" }); + const before = await readFile(target); + await symlink(target, link); + const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(link)).build(); + await cache.pullAllBackingStores(); + const namesBefore = [(await readdir(dir)).sort(), (await readdir(volume)).sort()]; + + assert.equal(cache.isWritable(document, "held"), true); + assert.equal(cache.isWritable(document, "fresh"), true); + await assert.rejects(() => cache.put(document, "fresh", { text: "fresh" }), (error: unknown) => error instanceof StoreLinkedError && error.path === link); + assert.ok((await lstat(link)).isSymbolicLink()); + assert.deepEqual(await readFile(target), before); + assert.deepEqual([(await readdir(dir)).sort(), (await readdir(volume)).sort()], namesBefore); +}); + // The second look of the read, staged: nothing is at the path only when the read and the lstat both find nothing. A path // that changed between them (a link in a parent swapped for a file, a store that appeared) is never an empty store. test("readStoreBytes reports nothing at the path only when the lstat finds nothing too", async () => { @@ -2446,9 +2533,9 @@ test("SingleFileMessagePackBackingStore record refusal names only a string key", } }); -// What a write does through a symbolic link today. R3 decides whether a write resolves the link -// or refuses it; these pin the current behaviour so that change is deliberate. -test("R3 decides resolve-or-refuse: a push through a live link replaces the link", { skip: process.platform === "win32" }, async () => { +// A write at a store path that is a link is refused: a replace there would swap the link for a regular file and +// detach the store from its target. Reads through a live link keep working. +test("a write at a live link is refused and nothing changes, while a read through it still works", { skip: process.platform === "win32" }, async () => { const dir = await mkdtemp(join(tmpdir(), "cultcache-live-link-")); const volume = join(dir, "volume"); await mkdir(volume); @@ -2456,21 +2543,42 @@ test("R3 decides resolve-or-refuse: a push through a live link replaces the link const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; await new SingleFileMessagePackBackingStore(target).push(envelope); const before = await readFile(target); - const link = join(dir, "store.cc"); + const volumeBefore = (await readdir(volume)).sort(); + const link = join(dir, "link.cc"); await symlink(target, link); - await new SingleFileMessagePackBackingStore(link).push({ ...envelope, key: "b" }); - assert.ok((await lstat(link)).isFile()); + const store = new SingleFileMessagePackBackingStore(link); + const refused = (error: unknown) => error instanceof StoreLinkedError && error.path === link && error.message.includes(link); + await assert.rejects(() => store.push({ ...envelope, key: "b" }), refused); + await assert.rejects(() => store.pushAll([{ ...envelope, key: "b" }]), refused); + await assert.rejects(() => store.delete(envelope), refused); + assert.ok((await lstat(link)).isSymbolicLink()); assert.deepEqual(await readFile(target), before); + assert.deepEqual((await readdir(dir)).sort(), ["link.cc", "volume"]); + assert.deepEqual((await readdir(volume)).sort(), volumeBefore); + assert.deepEqual((await store.pullAll()).map((e) => e.key), ["a"]); }); -test("R3 decides resolve-or-refuse: a pushAll through a dangling link replaces the link", { skip: process.platform === "win32" }, async () => { +test("a write at a live directory link is refused as a link", { skip: process.platform === "win32" }, async () => { + const dir = await mkdtemp(join(tmpdir(), "cultcache-dir-link-")); + const realDir = join(dir, "real"); + await mkdir(realDir); + const link = join(dir, "dirlink"); + await symlink(realDir, link); + const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; + await assert.rejects(() => new SingleFileMessagePackBackingStore(link).push(envelope), (error: unknown) => error instanceof StoreLinkedError); + assert.ok((await lstat(link)).isSymbolicLink()); + assert.deepEqual(await readdir(realDir), []); +}); + +test("a pushAll through a dangling link is refused like push, and the link and its volume are left as they were", { skip: process.platform === "win32" }, async () => { const dir = await mkdtemp(join(tmpdir(), "cultcache-dangling-push-all-")); const volume = join(dir, "volume"); await mkdir(volume); const link = join(dir, "store.cc"); await symlink(join(volume, "store.cc"), link); - await new SingleFileMessagePackBackingStore(link).pushAll([{ key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }]); - assert.ok((await lstat(link)).isFile()); + const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; + await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([envelope]), (error: unknown) => error instanceof StoreLinkedError); + assert.ok((await lstat(link)).isSymbolicLink()); assert.deepEqual(await readdir(volume), []); }); @@ -2575,7 +2683,7 @@ test("SingleFileMessagePackBackingStore pushAll refuses a store whose header it // A file is replaced by a rewrite exactly when this runtime's own reader opens it: one verdict per file, asked by pullAll, // pushAll, soft pushAll, push and delete alike; a soft flush leaves the files that read as they are. The bytes and every runtime's verdict: tests/vectors/document-variants-c2a/readability. -test("SingleFileMessagePackBackingStore replaces a file exactly when it reads", async () => { +test("SingleFileMessagePackBackingStore rewrites a file exactly when it reads", async () => { const root = join(c2aVectors, "readability"); const rows = (await readFile(join(root, "manifest.txt"), "utf8")) .split(/\r?\n/u) @@ -2612,6 +2720,7 @@ test("SingleFileMessagePackBackingStore replaces a file exactly when it reads", ? store.push(envelopes[0]!) : store.delete(envelopes[0]!); if (reads) { + const heldBefore = operation === "pushAll" ? (await new SingleFileMessagePackBackingStore(file).pullAll()).map((entry) => entry.key) : []; await run(); if (operation === "pullAll") { continue; @@ -2628,7 +2737,8 @@ test("SingleFileMessagePackBackingStore replaces a file exactly when it reads", assert.equal((decode(await readFile(file)) as unknown[])[0], vector.includes("v3") ? "cultcache.store.v3" : "cultcache.store.v1", `${vector} ${operation}`); const keys = (await new SingleFileMessagePackBackingStore(file).pullAll()).map((entry) => entry.key); if (operation === "pushAll") { - assert.deepEqual(keys, envelopes.map((entry) => entry.key).sort(), `${vector} ${operation}`); + assert.deepEqual(envelopes.map((entry) => entry.key).filter((key) => !keys.includes(key)), [], `${vector} ${operation}: every written key is held`); + assert.deepEqual(heldBefore.filter((key) => !keys.includes(key)), [], `${vector} ${operation}: nothing else is removed`); } else if (operation === "push") { assert.ok(keys.includes(envelopes[0]!.key), `${vector} ${operation}`); } else { @@ -2762,7 +2872,6 @@ test("every registration refusal is a SchemaConflictError naming the id and both // Cross-runtime store lock: each runtime's writer excludes the others on `.lock`. import { spawn } from "node:child_process"; -import { withStoreWriteLock } from "../src/store-file"; type LockHolder = { exited: Promise; stderr: () => string }; diff --git a/packages/cultcache-ts/test/first-call.test.ts b/packages/cultcache-ts/test/first-call.test.ts index 45f6edd2..9ef3d308 100644 --- a/packages/cultcache-ts/test/first-call.test.ts +++ b/packages/cultcache-ts/test/first-call.test.ts @@ -46,6 +46,21 @@ async function firstCall(path: string) { } // snippet:end +// The update snippet, which docs/start/README.md shows after the first call and which uses its definitions. +// snippet:update:begin +async function restock(path: string) { + const cache = CultCache.builder() + .withRegistry(registry) + .withGenericStore(new SingleFileMessagePackBackingStore(path)) + .build(); + await cache.pullAllBackingStores(); + return cache.update(itemDocument, "item:potion", (item) => { + const current = item ?? { name: "Potion", category: "Consumable", value: 0 }; + return { ...current, value: current.value + 10 }; + }); +} +// snippet:update:end + test("define, store and read back a typed document", async () => { const dir = await mkdtemp(join(tmpdir(), "cultcache-first-call-")); try { @@ -57,13 +72,35 @@ test("define, store and read back a typed document", async () => { } }); -test("docs show the tested snippet", async () => { +test("update with retry: restock twice reads value 20 through a second cache", async () => { + const dir = await mkdtemp(join(tmpdir(), "cultcache-first-call-")); + try { + const path = join(dir, "restock.cc"); + assert.equal((await restock(path)).outcome, "updated"); + assert.equal((await restock(path)).outcome, "updated"); + const reopened = CultCache.builder() + .withRegistry(registry) + .withGenericStore(new SingleFileMessagePackBackingStore(path)) + .build(); + await reopened.pullAllBackingStores(); + assert.equal(reopened.get(itemDocument, "item:potion")?.value, 20); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +// The docs show the region between `// snippet:begin` and `end` markers verbatim, the package import aside. +async function assertDocsShow(region: "" | "update:") { // Compiled to dist-test/test/first-call.test.js, so the package root is two levels up. const source = await readFile(join(__dirname, "..", "..", "test", "first-call.test.ts"), "utf8"); const snippet = source - .split("// snippet:begin\n")[1] - .split("// snippet:end\n")[0] + .split(`// snippet:${region}begin\n`)[1] + .split(`// snippet:${region}end\n`)[0] .replace('"../src/index"', '"@gamecult/cultcache-ts"'); const docs = (await readFile(join(__dirname, "..", "..", "..", "..", "docs", "start", "README.md"), "utf8")).replace(/\r\n/g, "\n"); - assert.ok(docs.includes("```ts\n" + snippet + "```"), "docs/start/README.md no longer shows test/first-call.test.ts"); -}); + assert.ok(docs.includes("```ts\n" + snippet + "```"), `docs/start/README.md no longer shows the ${region || "first-call "}region of test/first-call.test.ts`); +} + +test("docs show the tested snippet", () => assertDocsShow("")); + +test("docs show the tested update snippet", () => assertDocsShow("update:")); diff --git a/packages/cultcache-ts/test/payload-depth.test.ts b/packages/cultcache-ts/test/payload-depth.test.ts index 382f0672..6fb81265 100644 --- a/packages/cultcache-ts/test/payload-depth.test.ts +++ b/packages/cultcache-ts/test/payload-depth.test.ts @@ -119,6 +119,19 @@ test("a payload that ends early, is empty or carries trailing bytes is the reade assert.doesNotThrow(() => checkPayloadDepth(bytes(0xc0))); }); +test("the walk returns where the first value ends, and undefined when the payload ends early", () => { + assert.equal(checkPayloadDepth(bytes(0xc0)), 1); + assert.equal(checkPayloadDepth(bytes(0xc0, 0xc1, 0xc1)), 1, "a top-level scalar ends at its own length; the tail is not judged"); + assert.equal(checkPayloadDepth(bytes(0x92, 0x01, 0xa1, 0x61, 0xc0)), 5 - 1, "a container ends when its last child completes"); + assert.equal(checkPayloadDepth(cat(bytes(0x90), arrays(L + 50))), 1, "the tail is never walked"); + assert.equal(checkPayloadDepth(bytes()), undefined); + assert.equal(checkPayloadDepth(bytes(0x92, 0x01)), undefined); + assert.equal(checkPayloadDepth(bytes(0xd9)), undefined); + for (const cut of [[0xcc], [0xcb, 0x3f, 0xf8], [0xa5, 0x61, 0x62, 0x63], [0xc4, 0x02, 0x01], [0xd7, 0x01]]) { + assert.equal(checkPayloadDepth(bytes(...cut)), undefined, `a scalar cut short: ${cut.map((value) => value.toString(16)).join(" ")}`); + } +}); + test("the walk is iterative: 100000 nested arrays are refused with the depth error, not a RangeError", () => { refused(arrays(100_000), "100000 nested arrays"); refused(chain(bytes(0x81, 0x00), 100_000), "100000 nested maps"); @@ -198,6 +211,13 @@ test("the inspector previews a payload at the limit and reports one past it, or } }); +test("the inspector refuses bytes after a payload's first value with the shared clause and previews no tail", async () => { + const file = await storeWith(bytes(0xc0, 0xc1, 0xc1)); + const record = inspectCultCacheBytes(file, await readFile(file)).records[0]!; + assert.equal(record.payloadDecodeError, "document payload has 2 bytes after its first MessagePack value."); + assert.deepEqual(record.payloadPreview, { bytes: 3, hexPrefix: "c0" }); +}); + // A header whose length or count needs all four bytes: the low sixteen bits of the length are small, the whole is not. const be32 = (n: number) => bytes((n >>> 24) & 0xff, (n >>> 16) & 0xff, (n >>> 8) & 0xff, n & 0xff); diff --git a/packages/cultcache-ts/test/slot-readonly.test.ts b/packages/cultcache-ts/test/slot-readonly.test.ts new file mode 100644 index 00000000..e3285f07 --- /dev/null +++ b/packages/cultcache-ts/test/slot-readonly.test.ts @@ -0,0 +1,461 @@ +import assert from "node:assert/strict"; +import { copyFile, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; + +import { decode, encode } from "@msgpack/msgpack"; +import { z } from "zod"; + +import { codecs } from "../src"; +import { CultCache } from "../src/cult-cache"; +import { defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; +import { SingleFileMessagePackBackingStore, readSingleFileStore } from "../src/single-file-messagepack-backing-store"; +import { SchemaConflictError, StoreUnreadableError } from "../src/store-format"; +import type { CacheBackingStore, CultCacheEnvelope, CultCacheUnreadSlots } from "../src/types"; +import { + type Surface, envelopeOf, golden, hex, memoryStore, read, slotUnitDocument, surfaceCodec, surfaceDeclaringDocument, surfaceDocument, surfaceSchema, vectors, +} from "./union-surface-fixture"; + +// cultlib-gaps:cut:slot-readonly-ts. A payload carrying a slot past its reader's declared slots, at any depth, is read, held +// read-only, and every write or removal of its key is refused before a store is touched. The vectors are written by the C# +// reference: tests/vectors/union-parity/README.md, "Extra trailing slot" and "Slot stores". + +const { slots, string } = codecs; +const SLOT_VALUE = "extra-slot"; + +function mark(typeName: string, slotCount: number, declaredCount: number, depth0Only: boolean, explained = false, occurrences = 1): CultCacheUnreadSlots { + return { typeName, slotCount, declaredCount, occurrences, depth0Only, explained }; +} + +// File, then the mark the README table gives for it. +const PAYLOAD_ROWS: [string, CultCacheUnreadSlots][] = [ + ["extra-slot-record.msgpack", mark("UnionSurface", 8, 7, true)], + ["extra-slot-nested-record.msgpack", mark("Composition", 3, 2, false)], + ["extra-slot-arm.msgpack", mark("Box", 3, 2, false)], + ["extra-slot-list-element.msgpack", mark("Text", 3, 2, false)], + ["depth-skipped-at-limit.msgpack", mark("UnionSurface", 8, 7, true)], +]; + +async function readOnlyFiles(name: string): Promise { + const rows = decode(await read(name)) as unknown[][]; + return rows.filter((row) => row[1] === "read-only").map((row) => row[0] as string); +} + +test("the read-only payload vectors are exactly the ones this walker names: four in outcomes, one in bounds", async () => { + const listed = [...(await readOnlyFiles("outcomes.msgpack")), ...(await readOnlyFiles("bounds-outcomes.msgpack"))]; + assert.deepEqual(listed.sort(), PAYLOAD_ROWS.map(([file]) => file).sort()); +}); + +for (const [file, expected] of PAYLOAD_ROWS) { + test(`${file}: read past the extra slot, held read-only with the mark, every write and removal of the key refused`, async () => { + const document = surfaceDocument(); + const entries = [envelopeOf(await read(file))]; + const before = hex(documentPayloadBytes(entries[0]!.payload)); + const cache = CultCache.builder().withDocumentType(document).withGenericStore(memoryStore(entries)).build(); + await cache.pullAllBackingStores(); + + assert.deepEqual(cache.get(document, "surface"), golden, "the extra slot is read past"); + assert.deepEqual(cache.unreadSlots(document, "surface"), expected); + assert.equal(cache.isWritable(document, "surface"), false); + assert.equal(cache.isWritable(document, "another"), true); + + const refused = (error: unknown) => { + assert.ok(error instanceof SchemaConflictError, "a SchemaConflictError"); + assert.deepEqual(error.unreadSlots, expected); + assert.equal(error.recordKey, "surface"); + assert.ok(error.message.includes(expected.typeName), error.message); + assert.ok(error.message.includes(`read with ${expected.slotCount} slots where this build declares ${expected.declaredCount} (${expected.occurrences} object(s)`), error.message); + assert.ok(!error.message.includes(SLOT_VALUE), "the refusal never holds a slot value"); + return true; + }; + await assert.rejects(() => cache.put(document, "surface", golden), refused); + await assert.rejects(() => cache.update(document, "surface", (value) => ({ ...value!, version: 4 })), refused); + const replacement = envelopeOf(await read("union-surface.msgpack")); + await assert.rejects(() => cache.putEnvelope(document, replacement), refused); + await assert.rejects(() => cache.delete(document, "surface"), refused); + assert.equal(entries.length, 1, "nothing reached the store"); + assert.equal(hex(documentPayloadBytes(entries[0]!.payload)), before, "the held bytes are untouched"); + assert.deepEqual(cache.get(document, "surface"), golden, "the record is still held"); + + await cache.put(document, "another", golden); + assert.equal(entries.length, 2, "another key of the same store writes"); + }); +} + +test("a clean payload and a short one are writable: missing trailing slots keep the defaults and mark nothing", async () => { + const document = surfaceDocument(); + const short = encode([...(decode(await read("union-surface.msgpack")) as unknown[]).slice(0, 6)]); + const entries = [envelopeOf(await read("union-surface.msgpack")), { ...envelopeOf(short), key: "short" }]; + const cache = CultCache.builder().withDocumentType(document).withGenericStore(memoryStore(entries)).build(); + await cache.pullAllBackingStores(); + for (const key of ["surface", "short"]) { + assert.equal(cache.unreadSlots(document, key), undefined); + assert.equal(cache.isWritable(document, key), true); + } + await cache.put(document, "short", golden); +}); + +test("a payload past the nesting limit is refused, never marked", async () => { + const entries = [envelopeOf(await read("depth-skipped-past-limit.msgpack"))]; + const cache = CultCache.builder().withDocumentType(surfaceDocument()).withGenericStore(memoryStore(entries)).build(); + await assert.rejects(() => cache.pullAllBackingStores(), /nesting|depth/iu); +}); + +test("putEnvelope of a payload carrying extra slots lands the bytes and holds the record read-only", async () => { + const document = surfaceDocument(); + const entries: ReturnType[] = []; + const cache = CultCache.builder().withDocumentType(document).withGenericStore(memoryStore(entries)).build(); + const marked = envelopeOf(await read("extra-slot-nested-record.msgpack")); + await cache.putEnvelope(document, marked); + assert.equal(hex(documentPayloadBytes(entries[0]!.payload)), hex(await read("extra-slot-nested-record.msgpack")), "the bytes are carried"); + assert.deepEqual(cache.unreadSlots(document, "surface"), mark("Composition", 3, 2, false)); + assert.equal(cache.isWritable(document, "surface"), false); + await assert.rejects(() => cache.put(document, "surface", golden), SchemaConflictError); +}); + +test("a key a store carries foreign is not writable", async () => { + const document = surfaceDocument(); + const foreign = { ...envelopeOf(await read("union-surface.msgpack")), key: "stranger", type: "tests.unknown", schemaId: "tests.unknown.v1" }; + const cache = CultCache.builder().withDocumentType(document).withGenericStore(memoryStore([foreign])).build(); + await cache.pullAllBackingStores(); + assert.equal(cache.isWritable(document, "stranger"), false); + assert.equal(cache.isWritable(document, "other"), true); +}); + +test("a write by a type that owns the held record's id restamps it, and is refused when that record carries unread slots", async () => { + const typed = (type: string, schemaId: string, compatibleSchemaIds: string[] = []) => + defineDocumentType({ type, schemaId, schemaName: type, schemaVersion: `${type}.v1`, compatibleSchemaIds, schema: surfaceSchema, codec: surfaceCodec, name: "id" }); + const older = typed("tests.surface_older", "tests.surface_older.v1"); + const newer = typed("tests.surface_newer", "tests.surface_newer.v1", ["tests.surface_older.v1"]); + for (const [file, lands] of [["union-surface.msgpack", true], ["extra-slot-record.msgpack", false]] as const) { + const held = { ...envelopeOf(await read(file)), type: older.type, schemaId: "tests.surface_older.v1" }; + const cache = CultCache.builder().withDocumentType(older).withDocumentType(newer).withGenericStore(memoryStore([held])).build(); + await cache.pullAllBackingStores(); + assert.equal(cache.isWritable(newer, "surface"), lands, `${file}: isWritable agrees with the write by the type that owns the id`); + const write = cache.put(newer, "surface", golden); + if (lands) { + await write; + } else { + await assert.rejects(() => write, (error: unknown) => error instanceof SchemaConflictError && error.unreadSlots?.typeName === "UnionSurface"); + } + } +}); + +// The store vectors. TS has no variant records: extra-slot-store.cc carries one (marked-variant), so the whole file is refused +// by name, and the walk runs over the file without that record. +async function copyOf(name: string): Promise { + const path = join(await mkdtemp(join(tmpdir(), "slot-readonly-")), name); + await copyFile(join(vectors, name), path); + return path; +} + +async function withoutVariant(path: string): Promise { + const [format, catalog, records] = decode(await readFile(path)) as [unknown, unknown[], unknown[][]]; + await writeFile(path, encode([format, catalog, records.filter((record) => record[0] !== "marked-variant")])); +} + +type SlotRow = [string, string, boolean, boolean | null, string | null, number | null, number | null, number | null, string | null]; + +async function slotRows(): Promise { + return decode(await read("slot-outcomes.msgpack")) as SlotRow[]; +} + +// The catalog entry the file publishes for a schema: its id and the reference's content hash, which a store rewrites with the +// definition's own. +async function catalogOf(file: string, schemaName: string): Promise<[string, string]> { + const [, catalog] = decode(await read(file)) as [unknown, string[][]]; + const entry = catalog.find((candidate) => candidate[1] === schemaName)!; + return [entry[0]!, entry[3]!]; +} + +async function openStore(file: string, consent: boolean) { + const path = await copyOf(file); + if (file === "extra-slot-store.cc") await withoutVariant(path); + const surface = consent ? surfaceDeclaringDocument() : surfaceDocument(...(await catalogOf(file, "tests.union_surface"))); + const slotUnit = consent ? undefined : slotUnitDocument(...(await catalogOf(file, "tests.slot_unit"))); + const builder = CultCache.builder().withDocumentType(surface).withGenericStore(new SingleFileMessagePackBackingStore(path)); + const cache = (slotUnit ? builder.withDocumentType(slotUnit) : builder).build(); + await cache.pullAllBackingStores(); + return { path, cache, surface, slotUnit }; +} + +test("extra-slot-store.cc carries a variant, which this runtime refuses by name; the rest of the file is walked", async () => { + const path = await copyOf("extra-slot-store.cc"); + assert.throws(() => readSingleFileStore(path), StoreUnreadableError); +}); + +test("every slot-outcomes row holds: the mark, IsWritable, and the removal refused exactly where the record is not writable", async () => { + const rows = await slotRows(); + assert.equal(rows.length, 8); + let walked = 0; + for (const [file, key, writable, explained, typeName, slots, declared, occurrences, flattenRefusedBy] of rows) { + if (flattenRefusedBy !== null) continue; // a variant: this runtime holds none + walked += 1; + const { path, cache, surface, slotUnit } = await openStore(file!, file === "extra-slot-consent.cc"); + const document = key === "unit-arm" ? slotUnit! : surface; + const label = `${file} ${key}`; + const found = cache.unreadSlots(document, key!); + if (typeName === null) { + assert.equal(found, undefined, label); + } else { + assert.ok(found, label); + assert.deepEqual([found.explained, found.typeName, found.slotCount, found.declaredCount, found.occurrences], [explained, typeName, slots, declared, occurrences], label); + } + assert.equal(cache.isWritable(document, key!), writable, label); + + const original = await readFile(path); + if (writable) { + assert.equal(await cache.delete(document, key!), true, label); + } else { + await assert.rejects( + () => cache.delete(document, key!), + (error: unknown) => error instanceof SchemaConflictError && error.unreadSlots?.typeName === typeName && !error.message.includes(SLOT_VALUE), + label, + ); + assert.equal(Buffer.compare(original, await readFile(path)), 0, `${label}: the file is unchanged`); + } + } + assert.equal(walked, 7); +}); + +test("consent: a write of the explained record lands; the over-consent write is refused; the nested mark is refused", async () => { + const consented = await openStore("extra-slot-consent.cc", true); + const value = consented.cache.get(consented.surface, "consented") as Surface; + assert.deepEqual(value, golden); + await consented.cache.put(consented.surface, "consented", value); + assert.equal(consented.cache.unreadSlots(consented.surface, "consented"), undefined, "the write dropped the explained slot and the mark with it"); + assert.equal(readSingleFileStore(consented.path).envelopes.find((e) => e.key === "consented")!.schemaId, "tests.union_surface.v2"); + + for (const key of ["over-consent", "nested"]) { + const refused = await openStore("extra-slot-consent.cc", true); + await assert.rejects( + () => refused.cache.put(refused.surface, key, golden), + (error: unknown) => error instanceof SchemaConflictError && error.schemaId === "tests.union_surface.wide", + `${key}: the refusal names the id the record was stored under`, + ); + } +}); + +test("carried: after writing another record, the marked record's bytes, schema id, stored-at and every catalog entry are the originals", async () => { + const { path, cache, surface } = await openStore("extra-slot-store.cc", false); + const originals = readSingleFileStore(path); + await cache.put(surface, "clean", golden); + const after = readSingleFileStore(path); + for (const key of ["marked", "unit-arm"]) { + const was = originals.envelopes.find((e) => e.key === key)!; + const now = after.envelopes.find((e) => e.key === key)!; + assert.equal(hex(documentPayloadBytes(now.payload)), hex(documentPayloadBytes(was.payload)), key); + assert.equal(now.schemaId, was.schemaId, key); + assert.equal(now.storedAt, was.storedAt, key); + assert.deepEqual(now.catalogEntry, was.catalogEntry, key); + } +}); + +// A writer that adds a slot and keeps its schema id (TS ids are hand-written; C# ids hash the members): the reader of that +// same id has declared nothing, so it consents to nothing. +test("a wider writer under the reader's own schema id is read-only, and a real round trip does not shed the slot", async () => { + const doc = (columns: 1 | 2, compatibleSchemaIds: string[] = []) => defineDocumentType({ + type: "tests.own_id", + schemaId: "tests.own_id.v1", + schemaName: "tests.own_id", + compatibleSchemaIds, + schema: z.object({ n: z.string(), extra: z.string().optional() }), + codec: columns === 2 + ? slots<{ n: string; extra: string }>({ 0: ["n", string], 1: ["extra", string] }, "Own") + : slots<{ n: string }>({ 0: ["n", string] }, "Own"), + name: "n", + }); + const path = join(await mkdtemp(join(tmpdir(), "slot-readonly-")), "own.cc"); + const wide = doc(2); + const writer = CultCache.builder().withDocumentType(wide).withGenericStore(new SingleFileMessagePackBackingStore(path)).build(); + await writer.put(wide, "k", { n: "n1", extra: "kept" }); + const bytes = await readFile(path); + + for (const compatible of [[], ["tests.own_id.v1"]]) { + const reader = doc(1, compatible); + const cache = CultCache.builder().withDocumentType(reader).withGenericStore(new SingleFileMessagePackBackingStore(path)).build(); + await cache.pullAllBackingStores(); + assert.deepEqual(cache.unreadSlots(reader, "k"), mark("Own", 2, 1, true), "unexplained: the arrived id is the reader's own"); + assert.equal(cache.isWritable(reader, "k"), false); + await assert.rejects(() => cache.put(reader, "k", { n: "n2" }), SchemaConflictError); + await assert.rejects(() => cache.delete(reader, "k"), SchemaConflictError); + assert.equal(Buffer.compare(bytes, await readFile(path)), 0, "the file still holds the slot"); + } +}); + +test("a record that arrived under an id the reader does not list compatible is never consented to", async () => { + const path = await copyOf("extra-slot-consent.cc"); + const reader = surfaceDocument("tests.union_surface.v2"); // resolves by type name; lists nothing + const cache = CultCache.builder().withDocumentType(reader).withGenericStore(new SingleFileMessagePackBackingStore(path)).build(); + await cache.pullAllBackingStores(); + assert.equal(cache.unreadSlots(reader, "consented")?.explained, false); + assert.equal(cache.isWritable(reader, "consented"), false); +}); + +// isWritable is the write's own judge, not a second one: for every key state it answers what put does. +test("isWritable equals the outcome of put for every key state the write judges", async () => { + const typed = (type: string, schemaId: string, extra: Record = {}) => + defineDocumentType({ type, schemaId, schemaName: type, schemaVersion: `${type}.v1`, schema: surfaceSchema, codec: surfaceCodec, name: "id", ...extra }); + const older = typed("tests.surface_older", "tests.surface_older.v1"); + const newer = typed("tests.surface_newer", "tests.surface_newer.v1", { compatibleSchemaIds: ["tests.surface_older.v1"] }); + const stranger = typed("tests.surface_stranger", "tests.surface_stranger.v1"); + const global = typed("tests.surface_global", "tests.surface_global.v1", { global: true }); + const clean = (key: string, type: string, schemaId: string) => ({ ...envelopeOf(new Uint8Array()), key, type, schemaId }); + const held = async (key: string, file: string, type: string, schemaId: string) => + ({ ...envelopeOf(await read(file)), key, type, schemaId }); + + type Row = { name: string; build: () => Promise<{ cache: CultCache; target: typeof older; key: string }> }; + const rows: Row[] = [ + { name: "owned clean record in the writer's own store", build: async () => { + const cache = CultCache.builder().withDocumentType(older).withDocumentType(newer) + .withGenericStore(memoryStore([await held("k", "union-surface.msgpack", older.type, older.schemaId!)])).build(); + return { cache, target: newer, key: "k" }; + } }, + { name: "owned record held in a store other than the writer's home", build: async () => { + const cache = CultCache.builder().withDocumentType(older).withDocumentType(newer) + .withBackingStore(memoryStore([await held("k", "union-surface.msgpack", older.type, older.schemaId!)]), older) + .withBackingStore(memoryStore([]), newer).build(); + return { cache, target: newer, key: "k" }; + } }, + { name: "a record of a type that does not own the writer's id", build: async () => { + const cache = CultCache.builder().withDocumentType(older).withDocumentType(stranger) + .withGenericStore(memoryStore([await held("k", "union-surface.msgpack", older.type, older.schemaId!)])).build(); + return { cache, target: stranger, key: "k" }; + } }, + { name: "a global type at a key other than the global key", build: async () => { + const cache = CultCache.builder().withDocumentType(global).withGenericStore(memoryStore([])).build(); + return { cache, target: global, key: "x" }; + } }, + { name: "a key a store carries foreign", build: async () => { + const cache = CultCache.builder().withDocumentType(older) + .withGenericStore(memoryStore([{ ...clean("k", "tests.unknown", "tests.unknown.v1"), payload: documentPayloadFromBytes(await read("union-surface.msgpack")) }])).build(); + return { cache, target: older, key: "k" }; + } }, + { name: "a marked record of the writer's own type", build: async () => { + const cache = CultCache.builder().withDocumentType(older) + .withGenericStore(memoryStore([await held("k", "extra-slot-record.msgpack", older.type, older.schemaId!)])).build(); + return { cache, target: older, key: "k" }; + } }, + { name: "an unheld key", build: async () => ({ cache: CultCache.builder().withDocumentType(older).withGenericStore(memoryStore([])).build(), target: older, key: "k" }) }, + ]; + const outcomes: [string, boolean, boolean][] = []; + for (const row of rows) { + const { cache, target, key } = await row.build(); + await cache.pullAllBackingStores(); + const predicted = cache.isWritable(target, key); + const landed = await cache.put(target, key === "x" ? "x" : key, golden).then(() => true, () => false); + assert.equal(predicted, landed, row.name); + outcomes.push([row.name, predicted, landed]); + } + assert.deepEqual(outcomes.map(([, writable]) => writable), [true, false, false, false, false, false, true]); + // The same judge refuses a removal of a key held read-only or carried foreign. + const removals = CultCache.builder().withDocumentType(older) + .withGenericStore(memoryStore([await held("k", "extra-slot-record.msgpack", older.type, older.schemaId!)])).build(); + await removals.pullAllBackingStores(); + await assert.rejects(() => removals.delete(older, "k"), SchemaConflictError); + const stranger2 = CultCache.builder().withDocumentType(older) + .withGenericStore(memoryStore([{ ...envelopeOf(await read("union-surface.msgpack")), key: "f", type: "tests.unknown", schemaId: "tests.unknown.v1" }])).build(); + await stranger2.pullAllBackingStores(); + await assert.rejects(() => stranger2.delete(older, "f"), SchemaConflictError); +}); + +// A removal is by key (C# Remove): the owner of a held record's id removes the one record that exists, in one store call and +// with the cache dropped only once the store has; a type that does not own it is refused like put, and isWritable agrees. +test("delete of a key held as another type: the owner removes it once, a stranger is refused, a failing store leaves cache and store agreeing", async () => { + const typed = (type: string, schemaId: string, extra: Record = {}) => + defineDocumentType({ type, schemaId, schemaName: type, schemaVersion: `${type}.v1`, schema: surfaceSchema, codec: surfaceCodec, name: "id", ...extra }); + const older = typed("tests.surface_older", "tests.surface_older.v1"); + const newer = typed("tests.surface_newer", "tests.surface_newer.v1", { compatibleSchemaIds: ["tests.surface_older.v1"] }); + const stranger = typed("tests.surface_stranger", "tests.surface_stranger.v1"); + const heldRecord = async (): Promise => ({ ...envelopeOf(await read("union-surface.msgpack")), key: "k", type: older.type, schemaId: older.schemaId }); + const open = async (failing = false) => { + const rows = [await heldRecord()]; + const calls: string[] = []; + const store: CacheBackingStore = { + async pullAll() { return rows.map((row) => ({ ...row })); }, + async push(entry) { rows.push(entry); }, + async delete(entry) { + calls.push(entry.key); + if (failing) throw new Error("store refused the removal"); + rows.splice(rows.findIndex((row) => row.key === entry.key), 1); + }, + }; + const cache = CultCache.builder().withDocumentType(older).withDocumentType(newer).withDocumentType(stranger).withGenericStore(store).build(); + await cache.pullAllBackingStores(); + return { cache, rows, calls }; + }; + + const owner = await open(); + assert.equal(owner.cache.isWritable(newer, "k"), true); + assert.equal(await owner.cache.delete(newer, "k"), true, "the owner's removal removes the one record"); + assert.deepEqual([owner.rows.length, owner.calls.length, owner.cache.get(older, "k")], [0, 1, undefined], "one store call, record gone from store and cache"); + + const refused = await open(); + assert.equal(refused.cache.isWritable(stranger, "k"), false); + await assert.rejects(() => refused.cache.delete(stranger, "k"), SchemaConflictError); + assert.deepEqual([refused.rows.length, refused.calls.length, refused.cache.get(older, "k") !== undefined], [1, 0, true]); + + const failing = await open(true); + await assert.rejects(() => failing.cache.delete(newer, "k"), /store refused/u); + assert.deepEqual([failing.rows.length, failing.calls.length, failing.cache.get(older, "k") !== undefined], [1, 1, true], "the store failed: neither side dropped it"); + + const same = await open(); + assert.equal(await same.cache.delete(older, "k"), true); + assert.equal(await same.cache.delete(older, "k"), false, "an unheld key is still the plain false"); +}); + +// The removal form's three finer behaviours (cultlib-gaps:finding:cut-slot-readonly-ts.s4.removal-form-unpinned). +test("an owner's delete: refused for a marked record, taken from the held record's store, and clears the held type's lookups", async () => { + const typed = (type: string, schemaId: string, extra: Record = {}) => + defineDocumentType({ type, schemaId, schemaName: type, schemaVersion: `${type}.v1`, schema: surfaceSchema, codec: surfaceCodec, name: "id", ...extra }); + const older = typed("tests.surface_older", "tests.surface_older.v1", { indexes: { byVersion: (value: Surface) => String(value.version) } }); + const newer = typed("tests.surface_newer", "tests.surface_newer.v1", { compatibleSchemaIds: ["tests.surface_older.v1"] }); + const heldIn = async (file: string): Promise => + ({ ...envelopeOf(await read(file)), key: "k", type: older.type, schemaId: older.schemaId }); + const recording = (rows: CultCacheEnvelope[]) => { + const removed: string[] = []; + const store: CacheBackingStore = { + async pullAll() { return rows.map((row) => ({ ...row })); }, + async push(entry) { rows.push(entry); }, + async delete(entry) { removed.push(entry.key); rows.splice(rows.findIndex((row) => row.key === entry.key), 1); }, + }; + return { store, removed, rows }; + }; + + // 1. The owner may not remove a record carrying unread slots nothing consents to. + const marked = recording([await heldIn("extra-slot-record.msgpack")]); + const markedCache = CultCache.builder().withDocumentType(older).withDocumentType(newer).withGenericStore(marked.store).build(); + await markedCache.pullAllBackingStores(); + await assert.rejects(() => markedCache.delete(newer, "k"), (error: unknown) => error instanceof SchemaConflictError && error.unreadSlots?.typeName === "UnionSurface"); + assert.deepEqual([marked.rows.length, marked.removed.length, markedCache.get(older, "k") !== undefined], [1, 0, true]); + + // 2. The removal goes to the store of the record that exists, not the deleting type's; there the write form is false (it would + // push to the wrong store) while the removal succeeds. + const olderStore = recording([await heldIn("union-surface.msgpack")]); + const newerStore = recording([]); + const split = CultCache.builder().withDocumentType(older).withDocumentType(newer) + .withBackingStore(olderStore.store, older).withBackingStore(newerStore.store, newer).build(); + await split.pullAllBackingStores(); + assert.equal(split.isWritable(newer, "k"), false, "isWritable answers the write form: the rename cannot replace a record in another store"); + assert.equal(await split.delete(newer, "k"), true); + assert.deepEqual([olderStore.removed, newerStore.removed, split.get(older, "k")], [["k"], [], undefined]); + + // 3. The held type's name and index lookups are cleared with the record. + const single = recording([await heldIn("union-surface.msgpack")]); + const cache = CultCache.builder().withDocumentType(older).withDocumentType(newer).withGenericStore(single.store).build(); + await cache.pullAllBackingStores(); + assert.equal(cache.getKeyByName(older, "surface"), "k"); + assert.equal(cache.getKeyByIndex(older, "byVersion", "3"), "k"); + await cache.delete(newer, "k"); + assert.equal(cache.getKeyByName(older, "surface"), undefined); + assert.equal(cache.getKeyByIndex(older, "byVersion", "3"), undefined); + assert.equal(cache.getAll(older).length, 0); +}); + +test("a global type at a key other than the global key: isWritable is false and delete is the plain false of an unheld key", async () => { + const global = defineDocumentType({ type: "tests.surface_global", schemaId: "tests.surface_global.v1", schemaName: "tests.surface_global", schemaVersion: "tests.surface_global.v1", schema: surfaceSchema, codec: surfaceCodec, name: "id", global: true }); + const cache = CultCache.builder().withDocumentType(global).withGenericStore(memoryStore([])).build(); + assert.equal(cache.isWritable(global, "x"), false); + assert.equal(await cache.delete(global, "x"), false); +}); diff --git a/packages/cultcache-ts/test/store-conditional.test.ts b/packages/cultcache-ts/test/store-conditional.test.ts new file mode 100644 index 00000000..5f8f4ed5 --- /dev/null +++ b/packages/cultcache-ts/test/store-conditional.test.ts @@ -0,0 +1,175 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { existsSync, readFileSync } from "node:fs"; +import { copyFile, mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import test from "node:test"; + +import { decode, encode } from "@msgpack/msgpack"; + +import { documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; +import { SingleFileMessagePackBackingStore, readSingleFileStore, recordVersion } from "../src/single-file-messagepack-backing-store"; +import { schemaIdForEnvelope } from "../src/store-format"; +import type { CultCacheEnvelope } from "../src/types"; + +// The store owns the content version and the conditional verdict (variants:ruling:write-set-content-version-declared-ownership): +// read reports a record with its version, commitIf writes only while the version holds, and a write that replaces a record +// is stamped later than it (variants:ruling:storedat-on-change). + +function findVectors(from: string): string { + for (let dir = from; ; dir = resolve(dir, "..")) { + if (existsSync(join(dir, "tests", "vectors", "content-version"))) return join(dir, "tests", "vectors", "content-version"); + if (resolve(dir, "..") === dir) throw new Error("tests/vectors/content-version not found above the test."); + } +} + +const entry = (key: string, byte: number, storedAt = "2026-01-01T00:00:00.000Z", type = "t"): CultCacheEnvelope => ({ + key, + type, + payload: documentPayloadFromBytes(Uint8Array.of(0x91, byte)), + storedAt, +}); + +async function scratch(): Promise { + return mkdtemp(join(tmpdir(), "cultcache-conditional-")); +} + +test("the content version of every record of the C#-written store equals the version the reference read", async () => { + const vectors = findVectors(__dirname); + const rows = decode(readFileSync(join(vectors, "versions.msgpack"))) as Array<[string, string]>; + assert.equal(rows.length, 2); + const directory = await scratch(); + try { + const path = join(directory, "store.cc"); + await copyFile(join(vectors, "store.cc"), path); + const store = new SingleFileMessagePackBackingStore(path); + for (const [key, version] of rows) { + const found = await store.read(key); + assert.equal(found?.version, version, `read(${key}).version`); + assert.equal(recordVersion(found!.envelope), version, `recordVersion(${key})`); + } + assert.equal(await store.read("absent"), undefined); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("the version is the SHA-256 of the record's persisted encoding and changes with every field of it", async () => { + const base = entry("k", 1); + const hex = (e: CultCacheEnvelope) => + createHash("sha256").update(encode([e.key, schemaIdForEnvelope(e), e.storedAt, documentPayloadBytes(e.payload)])).digest("hex"); + assert.equal(recordVersion(base), hex(base)); + const versions = new Set([base, entry("k", 2), entry("k", 1, "2026-01-01T00:00:00.001Z"), entry("j", 1), { ...base, schemaId: "sha256:other" }].map(recordVersion)); + assert.equal(versions.size, 5); +}); + +test("commitIf writes nothing when the version does not hold, and commits when it does", async () => { + const directory = await scratch(); + try { + const path = join(directory, "store.cc"); + const store = new SingleFileMessagePackBackingStore(path); + // null expects no record: it commits over an absent key and mismatches over a present one. + assert.deepEqual(await store.commitIf(entry("k", 1), "not-a-version"), { outcome: "mismatch" }); + assert.equal(existsSync(path), false, "a mismatch creates no file"); + assert.equal((await store.commitIf(entry("k", 1), null)).outcome, "committed"); + const first = (await store.read("k"))!; + const before = await readFile(path); + assert.deepEqual(await store.commitIf(entry("k", 2), null), { outcome: "mismatch" }); + assert.deepEqual(await store.commitIf(entry("k", 2), "0".repeat(64)), { outcome: "mismatch" }); + assert.deepEqual([...(await readFile(path))], [...before], "a mismatch leaves the bytes identical"); + assert.equal((await store.commitIf(entry("k", 2), first.version)).outcome, "committed"); + const second = (await store.read("k"))!; + assert.notEqual(second.version, first.version); + assert.deepEqual([...documentPayloadBytes(second.envelope.payload)], [0x91, 2]); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("commitIf judges the record at the key whatever its type, and a record of another key does not matter", async () => { + const directory = await scratch(); + try { + const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); + await store.push(entry("a", 1, undefined, "u")); + await store.push(entry("b", 1)); + const other = (await store.read("a"))!; + assert.deepEqual(await store.commitIf(entry("b", 2), other.version), { outcome: "mismatch" }); + assert.equal((await store.commitIf(entry("b", 2), recordVersion((await store.read("b"))!.envelope))).outcome, "committed"); + assert.deepEqual([...documentPayloadBytes((await store.read("a"))!.envelope.payload)], [0x91, 1], "the record at another key is untouched"); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("a write that replaces a record is stored strictly later than it, whatever the write's own storedAt", async () => { + const directory = await scratch(); + try { + const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); + const at = "2026-03-04T05:06:07.100Z"; + await store.push(entry("k", 1, at)); + // Not later (equal, earlier, unparseable) is written one millisecond after the record it replaces, by push, pushAll and commitIf. + await store.push(entry("k", 2, at)); + assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.101Z"); + await store.push(entry("k", 3, "2020-01-01T00:00:00.000Z")); + assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.102Z"); + await store.pushAll([entry("k", 4, "not a time")]); + assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.103Z"); + const committed = await store.commitIf(entry("k", 5, "2020-01-01T00:00:00.000Z"), (await store.read("k"))!.version); + assert.deepEqual(committed, { outcome: "committed", storedAt: "2026-03-04T05:06:07.104Z" }); + assert.equal((await store.read("k"))!.envelope.storedAt, "2026-03-04T05:06:07.104Z"); + // A later stamp is kept as it is, and a key with no record before it is stamped as it is. + await store.push(entry("k", 6, "2030-01-01T00:00:00.000Z")); + await store.push(entry("fresh", 1, "2000-01-01T00:00:00.000Z")); + assert.equal((await store.read("k"))!.envelope.storedAt, "2030-01-01T00:00:00.000Z"); + assert.equal((await store.read("fresh"))!.envelope.storedAt, "2000-01-01T00:00:00.000Z"); + assert.equal(readSingleFileStore(join(directory, "store.cc")).envelopes.length, 2); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("a replacing write that would stamp past year 9999 is refused and writes nothing", async () => { + const directory = await scratch(); + try { + const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); + const last = "9999-12-31T23:59:59.999Z"; + await store.push(entry("k", 1, last)); + await assert.rejects(() => store.push(entry("k", 2, last)), RangeError); + assert.equal((await store.read("k"))!.envelope.storedAt, last); + assert.deepEqual([...documentPayloadBytes((await store.read("k"))!.envelope.payload)], [0x91, 1]); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("a write that restores an earlier value does not restore its version", async () => { + const directory = await scratch(); + try { + const store = new SingleFileMessagePackBackingStore(join(directory, "store.cc")); + const at = "2026-03-04T05:06:07.100Z"; + await store.push(entry("k", 1, at)); + const first = (await store.read("k"))!.version; + await store.push(entry("k", 5, at)); + await store.push(entry("k", 1, at)); + assert.notEqual((await store.read("k"))!.version, first, "an A-B-A within one millisecond keeps no version"); + assert.deepEqual(await store.commitIf(entry("k", 9, at), first), { outcome: "mismatch" }); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("two stores on one file never both commit against the same version", async () => { + const directory = await scratch(); + try { + const path = join(directory, "store.cc"); + const left = new SingleFileMessagePackBackingStore(path); + const right = new SingleFileMessagePackBackingStore(path); + await left.push(entry("k", 0)); + const version = (await left.read("k"))!.version; + const outcomes = await Promise.all(Array.from({ length: 6 }, (_, i) => (i % 2 === 0 ? left : right).commitIf(entry("k", i + 1), version))); + assert.equal(outcomes.filter((o) => o.outcome === "committed").length, 1); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); diff --git a/packages/cultcache-ts/test/store-file.test.ts b/packages/cultcache-ts/test/store-file.test.ts index eb41b77b..6cb70392 100644 --- a/packages/cultcache-ts/test/store-file.test.ts +++ b/packages/cultcache-ts/test/store-file.test.ts @@ -10,6 +10,7 @@ import { promisify } from "node:util"; import { decode, encode } from "@msgpack/msgpack"; +import { StoreLinkedError as PackageStoreLinkedError } from "../src/index"; import { documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; import { SingleFileMessagePackBackingStore } from "../src/single-file-messagepack-backing-store"; import { SchemaConflictError, StoreUnreadableError } from "../src/store-format"; @@ -23,6 +24,7 @@ import { replaceStoreFile, replaceStoreFileWindows, windowsKernel, + refuseLinkedStore, withStoreWriteLock, } from "../src/store-file"; import type { ReplaceKernel, StoreFileOps } from "../src/store-file"; @@ -415,6 +417,43 @@ test("a second writer in this process waits for the store lock and runs after re } }); +test("a link put while a writer waits on the store lock is refused once it holds the lock", { skip: process.platform === "win32" }, async () => { + const directory = await scratch(); + try { + const target = join(directory, "target.cc"); + await new SingleFileMessagePackBackingStore(target).push(noteEnvelope("a")); + const before = await readFile(target); + const writes: Array<[string, (store: SingleFileMessagePackBackingStore) => Promise]> = [ + ["push", async (store) => void (await store.push(noteEnvelope("b")))], + ["pushAll", (store) => store.pushAll([noteEnvelope("b")])], + ["delete", (store) => store.delete(noteEnvelope("a"))], + ]; + const missing = join(directory, "missing.cc"); + for (const [name, write, linkTo] of writes.flatMap(([n, w]) => [[n, w, target], [`${n} (dangling)`, w, missing]] as const)) { + const link = join(directory, `${name.replace(/\W+/g, "-")}.cc`); + let writer: Promise | undefined; + await withStoreWriteLock(link, async () => { + writer = write(new SingleFileMessagePackBackingStore(link)); + writer.catch(() => undefined); + await sleepFor(150); + await fsp.symlink(linkTo, link); // after the writer's pre-lock look, while it waits + }); + await assert.rejects(writer!, (error: unknown) => error instanceof PackageStoreLinkedError, name); + assert.ok((await fsp.lstat(link)).isSymbolicLink(), `${name}: the link was replaced`); + assert.deepEqual(await readFile(target), before, `${name}: the target changed`); + await assert.rejects(fsp.lstat(missing), { code: "ENOENT" }, `${name}: the dangling target was created`); + await rm(link); // the link is refused before the lock, so remove it to probe the lock + assert.equal(await withStoreWriteLock(link, async () => "free", { timeoutMs: 200 }), "free", `${name}: the lock is held`); + } + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +function noteEnvelope(key: string): CultCacheEnvelope { + return { key, type: "note", payload: documentPayloadFromBytes(Uint8Array.of(0x91, 1)), storedAt: "2026-10-20T00:00:00Z" }; +} + test("the store lock gives up after its wait bound and names the lock path", async () => { const directory = await scratch(); try { @@ -670,7 +709,7 @@ test("delete removes only the record at that type and key", async () => { } }); -test("pushAll replaces the whole store, and a soft pushAll leaves an existing store alone", async () => { +test("pushAll writes onto the store as it is: it replaces by key and removes nothing, and a soft pushAll leaves an existing store alone", async () => { const directory = await scratch(); try { const path = join(directory, "store.cc"); @@ -681,7 +720,14 @@ test("pushAll replaces the whole store, and a soft pushAll leaves an existing st await store.pushAll([entryOf("b", "t", 2)], { soft: true }); assert.deepEqual([...(await readFile(path))], [...before], "soft leaves an existing store byte for byte"); await store.pushAll([entryOf("b", "t", 2)]); - assert.deepEqual(await snapshotOf(store), ["t/b:2"]); + assert.deepEqual(await snapshotOf(store), ["t/a:1", "t/b:2"], "a record the batch does not name stays"); + await store.pushAll([entryOf("a", "t", 7)]); + assert.deepEqual(await snapshotOf(store), ["t/a:7", "t/b:2"], "a record the batch names is replaced"); + await store.pushAll([]); + assert.deepEqual(await snapshotOf(store), ["t/a:7", "t/b:2"], "an empty flush from a cache that holds nothing wipes nothing"); + const before2 = [...(await readFile(path))]; + await assert.rejects(() => store.pushAll([entryOf("a", "u", 1)]), SchemaConflictError); + assert.deepEqual([...(await readFile(path))], before2, "a batch that retypes a held key is refused and the file is unchanged"); } finally { await rm(directory, { recursive: true, force: true }); } @@ -811,3 +857,35 @@ test("opening writes nothing: a missing store leaves no lock file, a legacy stor await rm(directory, { recursive: true, force: true }); } }); + +test("StoreLinkedError is importable from the package entry and is what a write at a link throws", { skip: process.platform === "win32" }, async () => { + const directory = await scratch(); + try { + const target = join(directory, "target.cc"); + await writeFile(target, "x"); + const link = join(directory, "link.cc"); + await fsp.symlink(target, link); + await assert.rejects( + () => refuseLinkedStore(link), + (error: unknown) => error instanceof PackageStoreLinkedError && error.path === link && error.name === "StoreLinkedError", + ); + await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([]), (error: unknown) => error instanceof PackageStoreLinkedError); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); + +test("refuseLinkedStore lets a path that is absent pass and rethrows any other lstat failure", { skip: process.platform === "win32" }, async () => { + const directory = await scratch(); + try { + await refuseLinkedStore(join(directory, "absent.cc")); + const file = join(directory, "file"); + await writeFile(file, "x"); + await assert.rejects( + () => refuseLinkedStore(join(file, "child.cc")), + (error: unknown) => (error as NodeJS.ErrnoException).code === "ENOTDIR", + ); + } finally { + await rm(directory, { recursive: true, force: true }); + } +}); diff --git a/packages/cultcache-ts/test/stored-at-stamp.test.ts b/packages/cultcache-ts/test/stored-at-stamp.test.ts new file mode 100644 index 00000000..d1ed62d2 --- /dev/null +++ b/packages/cultcache-ts/test/stored-at-stamp.test.ts @@ -0,0 +1,20 @@ +// isStoredAtStamp agrees with every row of tests/vectors/stored-at-stamp/stamps.msgpack, the one shape a peer's storedAt must parse as. +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import test from "node:test"; +import { decode } from "@msgpack/msgpack"; +import { isStoredAtStamp } from "../src"; + +// dist-test/test -> the repository root. +export const STAMP_VECTOR = join(__dirname, "..", "..", "..", "..", "tests", "vectors", "stored-at-stamp", "stamps.msgpack"); + +test("isStoredAtStamp answers every row of the stored-at-stamp vector", () => { + const rows = decode(readFileSync(STAMP_VECTOR)) as [string, boolean][]; + assert.ok(rows.length > 20, "the vector was read"); + for (const [text, parses] of rows) assert.equal(isStoredAtStamp(text), parses, JSON.stringify(text)); +}); + +test("a stamp is a string: anything else is not one", () => { + for (const value of [undefined, null, 0, 1_700_000_000_000, true, {}, [], new Date()]) assert.equal(isStoredAtStamp(value), false); +}); diff --git a/packages/cultcache-ts/test/union-parity.test.ts b/packages/cultcache-ts/test/union-parity.test.ts index 46dcb42f..d917ea7a 100644 --- a/packages/cultcache-ts/test/union-parity.test.ts +++ b/packages/cultcache-ts/test/union-parity.test.ts @@ -1,8 +1,7 @@ import assert from "node:assert/strict"; -import { existsSync } from "node:fs"; import { mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; -import { join, resolve } from "node:path"; +import { join } from "node:path"; import { test } from "node:test"; import { decode, encode } from "@msgpack/msgpack"; @@ -22,190 +21,14 @@ import { encodeDocumentPayload, } from "../src/document"; import { SingleFileMessagePackBackingStore } from "../src/single-file-messagepack-backing-store"; -import type { AnyCultCacheDocumentDefinition, CacheBackingStore, CultCacheEnvelope } from "../src/types"; +import type { AnyCultCacheDocumentDefinition, CultCacheEnvelope } from "../src/types"; + +import { + CANARY, type Node, type Surface, golden, hex, node, outcome, read, surfaceCodec, surfaceDocument, surfaceSchema, vectors, envelopeOf, memoryStore, +} from "./union-surface-fixture"; const { array, boolean, bytes, float32, float64, int32, int64, lazy, list, nullable, slots, string, union, unit } = codecs; -// The names the C# reference gives the fixture's types (CultSchemaTypeNames.FromType), as union-surface.cc carries them. -const CS = "GameCult.Caching.Tests.UnionParityVectorTests+"; - -// The golden is tests/vectors/union-parity, written by the C# reference. Everything below that is bytes or an expected -// outcome is read from those committed files; only the value the payload must decode to is spelled here. -function findVectors(from: string): string { - for (let dir = from; ; dir = resolve(dir, "..")) { - if (existsSync(join(dir, "tests", "vectors", "union-parity"))) return join(dir, "tests", "vectors", "union-parity"); - if (resolve(dir, "..") === dir) throw new Error("tests/vectors/union-parity not found above the test."); - } -} -const vectors = findVectors(__dirname); -const read = async (name: string) => new Uint8Array(await readFile(join(vectors, name))); - -// The string the C# writer put inside every refusal payload, to prove a refusal never echoes the payload. -const CANARY = "CANARY-union-golden-4c2a"; - -type Node = - | { kind: "Box"; id: string; children: (Node | null)[] } - | { kind: "Text"; id: string; content: string } - | { kind: "Use"; id: string; composition: Composition; slots: (Node | null)[] } - | { kind: "Image"; id: string; uri: string; alt: string } - | { kind: "Wrap"; inner: Node | null; id: string }; -interface Composition { name: string; body: Node | null } -type Outcome = { kind: "Pending"; message: string } | { kind: "Completed"; sourceVersion: number } | { kind: "Denied" } | { kind: "Floor" } | { kind: "Ceiling" }; -// The golden's Outcome: the arms of a list carry an element id, so Denied, Floor and Ceiling are one-slot records. -type GoldenOutcome = - | { kind: "Pending"; message: string; id: string } - | { kind: "Completed"; sourceVersion: number; id: string } - | { kind: "Denied"; id: string } - | { kind: "Floor"; id: string } - | { kind: "Ceiling"; id: string }; -type Item = { kind: "Gem"; name: string } | { kind: "Tool"; name: string; uses: number } | { kind: "Part"; name: string; count: number }; -type CraftedItem = { kind: "Tool"; name: string; uses: number } | { kind: "Part"; name: string; count: number }; -interface Surface { - id: string; - version: number; - root: Node | null; - outcomes: (GoldenOutcome | null)[]; - part: Item | null; - crafted: CraftedItem | null; - missing: Node | null; -} - -const composition = slots({ 0: ["name", string], 1: ["body", lazy(() => node)] }, `${CS}Composition`); -const node: Codec = union("Node", { - 0: ["Box", slots<{ id: string; children: (Node | null)[] }>({ 0: ["id", string], 1: ["children", list(lazy(() => node))] })], - 1: ["Text", slots<{ id: string; content: string }>({ 0: ["id", string], 1: ["content", string] })], - 2: ["Use", slots<{ id: string; composition: Composition; slots: (Node | null)[] }>({ - 0: ["id", string], 1: ["composition", composition], 2: ["slots", list(lazy(() => node))], - })], - 5: ["Image", slots<{ id: string; uri: string; alt: string }>({ 0: ["id", string], 1: ["uri", string], 3: ["alt", string] })], - 6: ["Wrap", slots<{ inner: Node | null; id: string }>({ 0: ["inner", lazy(() => node)], 1: ["id", string] })], -}, `${CS}Node`); -const outcome = union("Outcome", { - 0: ["Pending", slots<{ message: string }>({ 0: ["message", string] })], - 1: ["Completed", slots<{ sourceVersion: number }>({ 0: ["sourceVersion", int32] })], - 2: ["Denied", unit], - [-2147483648]: ["Floor", unit], - 2147483647: ["Ceiling", unit], -}, `${CS}Outcome`); -const idOnly = slots<{ id: string }>({ 0: ["id", string] }); -const goldenOutcome = union("Outcome", { - 0: ["Pending", slots<{ message: string; id: string }>({ 0: ["message", string], 1: ["id", string] })], - 1: ["Completed", slots<{ sourceVersion: number; id: string }>({ 0: ["sourceVersion", int32], 1: ["id", string] })], - 2: ["Denied", idOnly], - [-2147483648]: ["Floor", idOnly], - 2147483647: ["Ceiling", idOnly], -}, `${CS}Outcome`); -const toolArm = slots<{ name: string; uses: number }>({ 0: ["name", string], 1: ["uses", int32] }); -const partArm = slots<{ name: string; count: number }>({ 0: ["name", string], 1: ["count", int32] }); -const item = union("Item", { - 0: ["Gem", slots<{ name: string }>({ 0: ["name", string] })], - 1: ["Tool", toolArm], - 2: ["Part", partArm], -}, `${CS}Item`); -const craftedItem = union("CraftedItem", { 0: ["Tool", toolArm], 1: ["Part", partArm] }, `${CS}CraftedItem`); -const surfaceCodec = slots({ - 0: ["id", string], - 1: ["version", int32], - 2: ["root", node], - 3: ["outcomes", list(goldenOutcome)], - 4: ["part", item], - 5: ["crafted", craftedItem], - 6: ["missing", node, null], -}, `${CS}UnionSurface`); - -const nodeSchema: z.ZodType = z.lazy(() => - z.union([ - z.null(), - z.discriminatedUnion("kind", [ - z.object({ kind: z.literal("Box"), id: z.string(), children: z.array(nodeSchema) }), - z.object({ kind: z.literal("Text"), id: z.string(), content: z.string() }), - z.object({ - kind: z.literal("Use"), - id: z.string(), - composition: z.object({ name: z.string(), body: nodeSchema }), - slots: z.array(nodeSchema), - }), - z.object({ kind: z.literal("Image"), id: z.string(), uri: z.string(), alt: z.string() }), - z.object({ kind: z.literal("Wrap"), inner: nodeSchema, id: z.string() }), - ]), - ])); -const surfaceSchema = z.object({ - id: z.string(), - version: z.number(), - root: nodeSchema, - outcomes: z.array(z.union([ - z.null(), - z.discriminatedUnion("kind", [ - z.object({ kind: z.literal("Pending"), message: z.string(), id: z.string() }), - z.object({ kind: z.literal("Completed"), sourceVersion: z.number(), id: z.string() }), - z.object({ kind: z.literal("Denied"), id: z.string() }), - z.object({ kind: z.literal("Floor"), id: z.string() }), - z.object({ kind: z.literal("Ceiling"), id: z.string() }), - ]), - ])), - part: z.union([z.null(), z.discriminatedUnion("kind", [ - z.object({ kind: z.literal("Gem"), name: z.string() }), - z.object({ kind: z.literal("Tool"), name: z.string(), uses: z.number() }), - z.object({ kind: z.literal("Part"), name: z.string(), count: z.number() }), - ])]), - crafted: z.union([z.null(), z.discriminatedUnion("kind", [ - z.object({ kind: z.literal("Tool"), name: z.string(), uses: z.number() }), - z.object({ kind: z.literal("Part"), name: z.string(), count: z.number() }), - ])]), - missing: nodeSchema, -}) as unknown as z.ZodType; - -function surfaceDocument(schemaId?: string) { - return defineDocumentType({ - type: "tests.union_surface", - schemaId: schemaId ?? "tests.union_surface.v1", - schemaName: "tests.union_surface", - schemaVersion: "tests.union_surface.v1", - schema: surfaceSchema, - codec: surfaceCodec, - name: "id", - }); -} - -const golden: Surface = { - id: "surface", - version: 3, - root: { - kind: "Box", - id: "0000000000c0", - children: [ - { kind: "Text", id: "0000000000c1", content: "hello" }, - { - kind: "Use", - id: "0000000000c2", - composition: { - name: "comp", - body: { - kind: "Box", - id: "0000000000d0", - children: [ - { kind: "Text", id: "0000000000d1", content: "x" }, - { kind: "Image", id: "0000000000d2", uri: "uri://a", alt: "alt" }, - ], - }, - }, - slots: [{ kind: "Text", id: "0000000000e1", content: "slot" }, null, { kind: "Box", id: "0000000000e2", children: [] }], - }, - { kind: "Image", id: "0000000000c3", uri: "uri://b", alt: "" }, - ], - }, - outcomes: [ - { kind: "Pending", message: "wait", id: "0000000000a1" }, - { kind: "Completed", sourceVersion: 7, id: "0000000000a2" }, - { kind: "Denied", id: "0000000000a3" }, - ], - part: { kind: "Tool", name: "hammer", uses: 12 }, - crafted: { kind: "Part", name: "gear", count: 4 }, - missing: null, -}; - -const hex = (data: Uint8Array) => Buffer.from(data).toString("hex"); - async function outcomeEntries() { const entries = decode(await read("outcomes.msgpack")) as [string, string, string, number | null, string][]; return entries.map(([file, outcome, unionName, key, detail]) => ({ file, outcome, unionName, key, detail })); @@ -515,7 +338,7 @@ test("the wire reader keeps 64-bit integers whole, tells a float from an integer assert.throws(() => readWire(fromHex("dd")), /truncated/u); assert.throws(() => readWire(fromHex("ddffffffff")), /truncated/u, "a huge array header reserves nothing"); assert.throws(() => readWire(fromHex("a5616263")), /truncated/u); - assert.throws(() => readWire(fromHex("0102")), /bytes after its value/u); + assert.equal(readWire(fromHex("0102")), 1, "the reader reads one value; the document owner refuses what follows"); assert.throws(() => readWire(fromHex("81a161a162")), /map, extension or reserved/u); assert.throws(() => readWire(fromHex("c1")), /map, extension or reserved/u); assert.throws(() => readWire(fromHex("d4010a")), /map, extension or reserved/u); @@ -729,19 +552,6 @@ test("the Int32 edges of an arm key: the largest is an arm, one past either end assert.throws(() => edge.decode(readWire(fromHex("92d3ffffffff8000000090"))), { message: "Union U: unknown arm key -2147483648 (declared keys: 2147483647)." }); }); -function envelopeOf(payload: Uint8Array): CultCacheEnvelope { - return { key: "surface", type: "tests.union_surface", payload: documentPayloadFromBytes(payload), storedAt: new Date().toISOString() }; -} - -function memoryStore(entries: CultCacheEnvelope[]): CacheBackingStore & { entries: CultCacheEnvelope[] } { - return { - entries, - async pullAll() { return entries.map((entry) => ({ ...entry })); }, - async push(entry) { entries.push(entry); }, - async delete() {}, - }; -} - test("arm keys span Int32 and are listed ascending; slot keys are non-negative Int32", () => { const keys = union<{ kind: "High" | "Zero" | "Neg" | "Low" }>("U", { 2147483647: ["High", unit], @@ -915,3 +725,114 @@ test("a cache load refuses a store whose record is past the limit, naming the ty const refusing = CultCache.builder().withDocumentType(document).withGenericStore(memoryStore([envelopeOf(await read("depth-past-limit.msgpack"))])).build(); await assert.rejects(() => refusing.pullAllBackingStores(), (error: Error) => error instanceof CultCachePayloadDepthError && !error.message.includes(CANARY)); }); + +// ---- bytes-after vectors: a payload is exactly one MessagePack value (tests/vectors/union-parity/README.md) ---- + +// The C# reference writes the vectors: union-surface.msgpack followed by a tail the first value does not own. The one +// owner refuses by count, naming the type, from the walk's end; it never walks or echoes the tail. +async function bytesAfterEntries() { + const raw = decode(await read("bytes-after-outcomes.msgpack")) as unknown[][]; + return raw.map(([file, outcome, unionName, key, detail]) => ({ file: file as string, outcome: outcome as string, unionName: unionName as string, key: key as number | null, detail: detail as string })); +} + +// The C# vector writer's canary (BytesAfterVectorTests): it sits in the deep tail, so an echo of the tail would carry it. +const TAIL_CANARY = "CANARY-bytes-after-7d1e"; +const valueLength = async () => (await read("union-surface.msgpack")).length; +const plainDocument = () => defineDocumentType({ type: "tests.plain_surface", schema: z.unknown() }); +const codecDocument = () => defineDocumentType({ type: "tests.codec_surface", schema: surfaceSchema, codec: surfaceCodec }); + +async function assertBytesAfterRefused(file: string, tail: number, run: () => unknown | Promise, type: string) { + const clause = `${type}`; + await assert.rejects( + async () => run(), + (error: Error) => { + assert.ok(!(error instanceof CultCachePayloadDepthError), `${file}: the tail was walked`); + assert.ok(!error.message.includes(CANARY) && !error.message.includes(TAIL_CANARY), `${file}: the refusal echoed the tail`); + assert.ok(!error.message.includes("the limit is"), `${file}: the depth refusal`); + assert.ok(error.message.includes(clause), `${file}: names the type: ${error.message}`); + assert.ok(error.message.includes(`${tail} bytes after its first MessagePack value`), `${file}: names the count: ${error.message}`); + return true; + }, + file, + ); +} + +test("every bytes-after vector is refused by the one owner, by count, on the codec, plain and store paths", async () => { + const entries = await bytesAfterEntries(); + assert.equal(entries.length, 4); + const length = await valueLength(); + const surface = surfaceDocument(); + const plain = plainDocument(); + const codecOnly = codecDocument(); + const dir = await mkdtemp(join(tmpdir(), "bytes-after-")); + for (const { file, outcome, unionName, detail } of entries) { + assert.equal(outcome, "refused"); + assert.equal(unionName, "UnionSurface"); + assert.equal(detail, "bytes-after"); + const bytes = await read(file); + const tail = bytes.length - length; + assert.ok(tail > 0, file); + if (file === "bytes-after-deep-tail.msgpack") assert.ok(Buffer.from(bytes).includes(TAIL_CANARY), "the deep tail carries the canary the assertions look for"); + const payload = documentPayloadFromBytes(bytes); + await assertBytesAfterRefused(file, tail, () => decodeDocumentPayload(surface, payload), surface.type); + await assertBytesAfterRefused(file, tail, () => decodeDocumentPayload(plain, payload), plain.type); + await assertBytesAfterRefused(file, tail, () => decodeDocumentPayload(codecOnly, payload), codecOnly.type); + + // The store path: a single-file store holding the vector as a record payload refuses on load. + const path = join(dir, `${file}.cc`); + await new SingleFileMessagePackBackingStore(path).push(envelopeOf(bytes)); + const cache = CultCache.builder().withDocumentType(surface).withGenericStore(new SingleFileMessagePackBackingStore(path)).build(); + await assertBytesAfterRefused(file, tail, () => cache.pullAllBackingStores(), surface.type); + + // The put path: an envelope arriving with the tail is refused and held nowhere. + const written: CultCacheEnvelope[] = []; + const held = CultCache.builder().withDocumentType(surface).withGenericStore(memoryStore(written)).build(); + await assertBytesAfterRefused(file, tail, () => held.putEnvelope(surface, envelopeOf(bytes)), surface.type); + assert.equal(written.length, 0, `${file}: nothing reached the store`); + assert.equal(held.get(surface, "surface"), undefined, `${file}: nothing is held in memory`); + } +}); + +test("a payload with an unread slot and a tail is refused by count before any slot is marked or the record held", async () => { + const withSlot = await read("extra-slot-record.msgpack"); + const surface = surfaceDocument(); + const tailed = Uint8Array.from([...withSlot, 0xc0]); + const whole = CultCache.builder().withDocumentType(surface).withGenericStore(memoryStore([envelopeOf(withSlot)])).build(); + await whole.pullAllBackingStores(); + assert.ok(whole.unreadSlots(surface, "surface"), "without a tail the record is held with its unread slots"); + + const loading = CultCache.builder().withDocumentType(surface).withGenericStore(memoryStore([envelopeOf(tailed)])).build(); + await assertBytesAfterRefused("extra-slot-record+tail", 1, () => loading.pullAllBackingStores(), surface.type); + assert.equal(loading.get(surface, "surface"), undefined); + assert.equal(loading.unreadSlots(surface, "surface"), undefined); + + const written: CultCacheEnvelope[] = []; + const putting = CultCache.builder().withDocumentType(surface).withGenericStore(memoryStore(written)).build(); + await assertBytesAfterRefused("extra-slot-record+tail", 1, () => putting.putEnvelope(surface, envelopeOf(tailed)), surface.type); + assert.equal(written.length, 0); + assert.equal(putting.get(surface, "surface"), undefined); + assert.equal(putting.unreadSlots(surface, "surface"), undefined); +}); + +test("a payload that ends early is not bytes after its value: the decoder refuses it, and a user formatter's bytes are not walked", async () => { + const whole = await read("union-surface.msgpack"); + const surface = surfaceDocument(); + for (const cut of [0, 1, 10, whole.length - 1]) { + assert.throws( + () => decodeDocumentPayload(surface, documentPayloadFromBytes(whole.slice(0, cut))), + (error: Error) => !error.message.includes("bytes after") && !(error instanceof CultCachePayloadDepthError), + `truncated at ${cut}`, + ); + } + // A header cut short of its length is an early end too, whatever width it declares. + for (const early of ["dc00", "dd000000", "de00", "c4", "d9", "a5616263", "9291"]) { + assert.throws( + () => decodeDocumentPayload(surface, documentPayloadFromBytes(fromHex(early))), + (error: Error) => !error.message.includes("bytes after") && !(error instanceof CultCachePayloadDepthError), + `early end ${early}`, + ); + } + const opaque = defineDocumentType({ type: "tests.opaque", schema: z.unknown(), formatter: { encode: (value: Uint8Array) => value, decode: (bytes: Uint8Array) => bytes } }); + const tailed = await read("bytes-after-nil.msgpack"); + assert.equal((decodeDocumentPayload(opaque, documentPayloadFromBytes(tailed)) as Uint8Array).length, tailed.length); +}); diff --git a/packages/cultcache-ts/test/union-surface-fixture.ts b/packages/cultcache-ts/test/union-surface-fixture.ts new file mode 100644 index 00000000..28089522 --- /dev/null +++ b/packages/cultcache-ts/test/union-surface-fixture.ts @@ -0,0 +1,251 @@ +import { existsSync } from "node:fs"; +import { readFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; + +import { z } from "zod"; + +import { codecs } from "../src"; +import type { Codec } from "../src/codec"; +import { defineDocumentType, documentPayloadFromBytes } from "../src/document"; +import { recordVersion } from "../src/single-file-messagepack-backing-store"; +import type { CacheBackingStore, CultCacheEnvelope } from "../src/types"; + +// The union-surface fixture: the codecs, schema and value tests/vectors/union-parity was written from, shared by the +// union-parity walker and the slot-readonly walker so the shape is spelled once. +const { list, int32, lazy, slots, string, union, unit } = codecs; + + +// The names the C# reference gives the fixture's types (CultSchemaTypeNames.FromType), as union-surface.cc carries them. +export const CS = "GameCult.Caching.Tests.UnionParityVectorTests+"; + +// The golden is tests/vectors/union-parity, written by the C# reference. Everything below that is bytes or an expected +// outcome is read from those committed files; only the value the payload must decode to is spelled here. +export function findVectors(from: string): string { + for (let dir = from; ; dir = resolve(dir, "..")) { + if (existsSync(join(dir, "tests", "vectors", "union-parity"))) return join(dir, "tests", "vectors", "union-parity"); + if (resolve(dir, "..") === dir) throw new Error("tests/vectors/union-parity not found above the test."); + } +} +export const vectors = findVectors(__dirname); +export const read = async (name: string) => new Uint8Array(await readFile(join(vectors, name))); + +// The string the C# writer put inside every refusal payload, to prove a refusal never echoes the payload. +export const CANARY = "CANARY-union-golden-4c2a"; + +export type Node = + | { kind: "Box"; id: string; children: (Node | null)[] } + | { kind: "Text"; id: string; content: string } + | { kind: "Use"; id: string; composition: Composition; slots: (Node | null)[] } + | { kind: "Image"; id: string; uri: string; alt: string } + | { kind: "Wrap"; inner: Node | null; id: string }; +export interface Composition { name: string; body: Node | null } +export type Outcome = { kind: "Pending"; message: string } | { kind: "Completed"; sourceVersion: number } | { kind: "Denied" } | { kind: "Floor" } | { kind: "Ceiling" }; +// The golden's Outcome: the arms of a list carry an element id, so Denied, Floor and Ceiling are one-slot records. +export type GoldenOutcome = + | { kind: "Pending"; message: string; id: string } + | { kind: "Completed"; sourceVersion: number; id: string } + | { kind: "Denied"; id: string } + | { kind: "Floor"; id: string } + | { kind: "Ceiling"; id: string }; +export type Item = { kind: "Gem"; name: string } | { kind: "Tool"; name: string; uses: number } | { kind: "Part"; name: string; count: number }; +export type CraftedItem = { kind: "Tool"; name: string; uses: number } | { kind: "Part"; name: string; count: number }; +export interface Surface { + id: string; + version: number; + root: Node | null; + outcomes: (GoldenOutcome | null)[]; + part: Item | null; + crafted: CraftedItem | null; + missing: Node | null; +} + +export const composition = slots({ 0: ["name", string], 1: ["body", lazy(() => node)] }, `${CS}Composition`); +export const node: Codec = union("Node", { + 0: ["Box", slots<{ id: string; children: (Node | null)[] }>({ 0: ["id", string], 1: ["children", list(lazy(() => node))] }, `${CS}Box`)], + 1: ["Text", slots<{ id: string; content: string }>({ 0: ["id", string], 1: ["content", string] }, `${CS}Text`)], + 2: ["Use", slots<{ id: string; composition: Composition; slots: (Node | null)[] }>({ + 0: ["id", string], 1: ["composition", composition], 2: ["slots", list(lazy(() => node))], + })], + 5: ["Image", slots<{ id: string; uri: string; alt: string }>({ 0: ["id", string], 1: ["uri", string], 3: ["alt", string] })], + 6: ["Wrap", slots<{ inner: Node | null; id: string }>({ 0: ["inner", lazy(() => node)], 1: ["id", string] })], +}, `${CS}Node`); +export const outcome = union("Outcome", { + 0: ["Pending", slots<{ message: string }>({ 0: ["message", string] })], + 1: ["Completed", slots<{ sourceVersion: number }>({ 0: ["sourceVersion", int32] })], + 2: ["Denied", unit], + [-2147483648]: ["Floor", unit], + 2147483647: ["Ceiling", unit], +}, `${CS}Outcome`); +export const idOnly = slots<{ id: string }>({ 0: ["id", string] }); +export const goldenOutcome = union("Outcome", { + 0: ["Pending", slots<{ message: string; id: string }>({ 0: ["message", string], 1: ["id", string] })], + 1: ["Completed", slots<{ sourceVersion: number; id: string }>({ 0: ["sourceVersion", int32], 1: ["id", string] })], + 2: ["Denied", idOnly], + [-2147483648]: ["Floor", idOnly], + 2147483647: ["Ceiling", idOnly], +}, `${CS}Outcome`); +export const toolArm = slots<{ name: string; uses: number }>({ 0: ["name", string], 1: ["uses", int32] }); +export const partArm = slots<{ name: string; count: number }>({ 0: ["name", string], 1: ["count", int32] }); +export const item = union("Item", { + 0: ["Gem", slots<{ name: string }>({ 0: ["name", string] })], + 1: ["Tool", toolArm], + 2: ["Part", partArm], +}, `${CS}Item`); +export const craftedItem = union("CraftedItem", { 0: ["Tool", toolArm], 1: ["Part", partArm] }, `${CS}CraftedItem`); +export const surfaceCodecOf = (typeName: string) => slots({ + 0: ["id", string], + 1: ["version", int32], + 2: ["root", node], + 3: ["outcomes", list(goldenOutcome)], + 4: ["part", item], + 5: ["crafted", craftedItem], + 6: ["missing", node, null], +}, typeName); +export const surfaceCodec = surfaceCodecOf(`${CS}UnionSurface`); + +export const nodeSchema: z.ZodType = z.lazy(() => + z.union([ + z.null(), + z.discriminatedUnion("kind", [ + z.object({ kind: z.literal("Box"), id: z.string(), children: z.array(nodeSchema) }), + z.object({ kind: z.literal("Text"), id: z.string(), content: z.string() }), + z.object({ + kind: z.literal("Use"), + id: z.string(), + composition: z.object({ name: z.string(), body: nodeSchema }), + slots: z.array(nodeSchema), + }), + z.object({ kind: z.literal("Image"), id: z.string(), uri: z.string(), alt: z.string() }), + z.object({ kind: z.literal("Wrap"), inner: nodeSchema, id: z.string() }), + ]), + ])); +export const surfaceSchema = z.object({ + id: z.string(), + version: z.number(), + root: nodeSchema, + outcomes: z.array(z.union([ + z.null(), + z.discriminatedUnion("kind", [ + z.object({ kind: z.literal("Pending"), message: z.string(), id: z.string() }), + z.object({ kind: z.literal("Completed"), sourceVersion: z.number(), id: z.string() }), + z.object({ kind: z.literal("Denied"), id: z.string() }), + z.object({ kind: z.literal("Floor"), id: z.string() }), + z.object({ kind: z.literal("Ceiling"), id: z.string() }), + ]), + ])), + part: z.union([z.null(), z.discriminatedUnion("kind", [ + z.object({ kind: z.literal("Gem"), name: z.string() }), + z.object({ kind: z.literal("Tool"), name: z.string(), uses: z.number() }), + z.object({ kind: z.literal("Part"), name: z.string(), count: z.number() }), + ])]), + crafted: z.union([z.null(), z.discriminatedUnion("kind", [ + z.object({ kind: z.literal("Tool"), name: z.string(), uses: z.number() }), + z.object({ kind: z.literal("Part"), name: z.string(), count: z.number() }), + ])]), + missing: nodeSchema, +}) as unknown as z.ZodType; + +// The reference's SurfaceDeclaring: the same layout, schema tests.union_surface.v2, listing the older wide id compatible. +export function surfaceDeclaringDocument() { + return defineDocumentType({ + type: "tests.union_surface", + schemaId: "tests.union_surface.v2", + schemaName: "tests.union_surface", + schemaVersion: "tests.union_surface.v2", + compatibleSchemaIds: ["tests.union_surface.wide"], + schema: surfaceSchema, + codec: surfaceCodecOf(`${CS}SurfaceDeclaring`), + name: "id", + }); +} + +export function surfaceDocument(schemaId?: string, contentHash?: string) { + return defineDocumentType({ + ...(contentHash === undefined ? {} : { contentHash }), + type: "tests.union_surface", + schemaId: schemaId ?? "tests.union_surface.v1", + schemaName: "tests.union_surface", + schemaVersion: "tests.union_surface.v1", + schema: surfaceSchema, + codec: surfaceCodec, + name: "id", + }); +} + +export const golden: Surface = { + id: "surface", + version: 3, + root: { + kind: "Box", + id: "0000000000c0", + children: [ + { kind: "Text", id: "0000000000c1", content: "hello" }, + { + kind: "Use", + id: "0000000000c2", + composition: { + name: "comp", + body: { + kind: "Box", + id: "0000000000d0", + children: [ + { kind: "Text", id: "0000000000d1", content: "x" }, + { kind: "Image", id: "0000000000d2", uri: "uri://a", alt: "alt" }, + ], + }, + }, + slots: [{ kind: "Text", id: "0000000000e1", content: "slot" }, null, { kind: "Box", id: "0000000000e2", children: [] }], + }, + { kind: "Image", id: "0000000000c3", uri: "uri://b", alt: "" }, + ], + }, + outcomes: [ + { kind: "Pending", message: "wait", id: "0000000000a1" }, + { kind: "Completed", sourceVersion: 7, id: "0000000000a2" }, + { kind: "Denied", id: "0000000000a3" }, + ], + part: { kind: "Tool", name: "hammer", uses: 12 }, + crafted: { kind: "Part", name: "gear", count: 4 }, + missing: null, +}; +export const hex = (data: Uint8Array) => Buffer.from(data).toString("hex"); + +// tests.slot_unit: a document [Name, Outcome: UnitOutcome], whose one arm Denied2 is a unit arm (extra-slot-store.cc, record +// unit-arm). The reader's names are the reference's: a mark names Denied2. +export const unitOutcome = union<{ kind: "Denied2" }>("UnitOutcome", { 2: ["Denied2", unit] }, `${CS}UnitOutcome`); +export function slotUnitDocument(schemaId: string, contentHash?: string) { + return defineDocumentType({ + ...(contentHash === undefined ? {} : { contentHash }), + type: "tests.slot_unit", + schemaId, + schemaName: "tests.slot_unit", + schemaVersion: "tests.slot_unit.v1", + schema: z.object({ name: z.string(), outcome: z.union([z.null(), z.object({ kind: z.literal("Denied2") })]) }), + codec: slots<{ name: string; outcome: { kind: "Denied2" } | null }>({ 0: ["name", string], 1: ["outcome", unitOutcome] }, `${CS}SlotUnit`), + name: "name", + }); +} + +export function envelopeOf(payload: Uint8Array): CultCacheEnvelope { + return { key: "surface", type: "tests.union_surface", payload: documentPayloadFromBytes(payload), storedAt: new Date().toISOString() }; +} + +export function memoryStore(entries: CultCacheEnvelope[]): CacheBackingStore & { entries: CultCacheEnvelope[] } { + return { + entries, + async pullAll() { return entries.map((entry) => ({ ...entry })); }, + async push(entry) { entries.push(entry); }, + async delete() {}, + async read(key) { + const envelope = [...entries].reverse().find((entry) => entry.key === key); + return envelope === undefined ? undefined : { envelope, version: recordVersion(envelope) }; + }, + async commitIf(entry, expectedVersion) { + const current = [...entries].reverse().find((candidate) => candidate.key === entry.key); + if ((current === undefined ? null : recordVersion(current)) !== expectedVersion) return { outcome: "mismatch" }; + entries.push(entry); + return { outcome: "committed", storedAt: entry.storedAt }; + }, + }; +} + diff --git a/packages/cultcache-ts/test/update.test.ts b/packages/cultcache-ts/test/update.test.ts new file mode 100644 index 00000000..fb0d56fd --- /dev/null +++ b/packages/cultcache-ts/test/update.test.ts @@ -0,0 +1,600 @@ +import assert from "node:assert/strict"; +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { setImmediate as tick } from "node:timers/promises"; + +import { decode, encode } from "@msgpack/msgpack"; +import { z } from "zod"; + +import { CultCache } from "../src/cult-cache"; +import { defineDocumentRegistry, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; +import { SingleFileMessagePackBackingStore, readSingleFileStore } from "../src/single-file-messagepack-backing-store"; +import { SchemaConflictError, schemaIdForEnvelope } from "../src/store-format"; +import type { CacheBackingStore } from "../src/types"; +import { envelopeOf, memoryStore, read, surfaceDocument } from "./union-surface-fixture"; + +// CultCache.update: read the record and its content version from the store, apply a function to a copy, commit only if the +// version still holds, run again on a lost race (variants:cut_spec:cut-update-with-retry-ts.r1; the C# tests in +// tests/GameCult.Caching.Tests/UpdateTests.cs are the parity). + +const counterDocument = defineDocumentType({ + type: "update.counter", + schemaId: "update.counter.v1", + schemaName: "update.counter", + schemaVersion: "update.counter.v1", + schema: z.object({ value: z.number().int() }), +}); +const tallyDocument = defineDocumentType({ + type: "update.tally", + schemaId: "update.tally.v1", + schemaName: "update.tally", + schemaVersion: "update.tally.v1", + schema: z.object({ count: z.number().int() }), +}); +const registry = defineDocumentRegistry(counterDocument, tallyDocument); +const settingsFields = { schemaName: "update.settings", schemaVersion: "update.settings.v1", schemaId: "update.settings.v1", schema: z.object({ theme: z.string() }) }; +const settingsDocument = defineDocumentType({ type: "update.settings", ...settingsFields, global: true }); +const plainSettingsDocument = defineDocumentType({ type: "update.settings", ...settingsFields }); + +const KEY = "counter"; + +async function withDirectory(body: (path: string) => Promise): Promise { + const directory = await mkdtemp(join(tmpdir(), "cultcache-update-")); + try { + return await body(join(directory, "store.cc")); + } finally { + await rm(directory, { recursive: true, force: true }); + } +} + +type Registry = Parameters["withRegistry"]>[0]; + +async function open(path: string, documents: Registry = registry): Promise { + const cache = CultCache.builder().withRegistry(documents).withGenericStore(new SingleFileMessagePackBackingStore(path)).build(); + await cache.pullAllBackingStores(); + return cache; +} + +// What the file holds at the key, read without a cache. +function stored(path: string, key = KEY): number | undefined { + const envelope = readSingleFileStore(path).envelopes.find((candidate) => candidate.key === key); + return envelope === undefined ? undefined : (decode(documentPayloadBytes(envelope.payload)) as { value: number }).value; +} + +const plus = (n: number) => (current: { value: number } | undefined) => ({ value: (current?.value ?? 0) + n }); + +test("two writers updating lose no increment", async () => { + await withDirectory(async (path) => { + const [left, right] = [await open(path), await open(path)]; + let calls = 0; + const loop = async (cache: CultCache) => { + const outcomes: string[] = []; + for (let i = 0; i < 50; i++) { + const result = await cache.update(counterDocument, KEY, async (current) => { + calls++; + await tick(); + return { value: (current?.value ?? 0) + 1 }; + }, { maxAttempts: 50 }); + outcomes.push(result.outcome); + } + return outcomes; + }; + const outcomes = (await Promise.all([loop(left), loop(right)])).flat(); + assert.ok(outcomes.every((outcome) => outcome === "updated"), outcomes.join()); + assert.equal(stored(path), 100); + assert.ok(calls > 100, `the caches raced: apply ran ${calls} times`); + }); +}); + +test("a lost attempt reruns apply on the fresh value", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + const seen: Array = []; + const result = await cache.update(counterDocument, KEY, async (current) => { + seen.push(current?.value); + if (seen.length === 1) await rival.put(counterDocument, KEY, { value: 100 }); + return { value: current!.value + 1 }; + }); + assert.deepEqual(seen, [1, 100]); + assert.equal(result.outcome, "updated"); + assert.equal(result.document?.value, 101); + assert.equal(stored(path), 101); + }); +}); + +test("exhausted attempts return lostRace, and the cache then shows the store", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + let calls = 0; + const result = await cache.update( + counterDocument, + KEY, + async (current) => { + calls++; + await rival.put(counterDocument, KEY, { value: 100 * calls }); + return { value: current!.value + 1 }; + }, + { maxAttempts: 3 }, + ); + assert.equal(calls, 3); + assert.equal(result.outcome, "lostRace"); + assert.equal(result.document?.value, 300); + assert.equal(stored(path), 300); + }); +}); + +test("five attempts by default", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + let calls = 0; + const result = await cache.update(counterDocument, KEY, async () => { + calls++; + await rival.put(counterDocument, KEY, { value: calls }); + return { value: -1 }; + }); + assert.equal(result.outcome, "lostRace"); + assert.equal(calls, 5); + }); +}); + +test("maxAttempts 1 makes one call", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + let calls = 0; + const apply = async (current: { value: number } | undefined) => { + calls++; + await rival.put(counterDocument, KEY, { value: 50 + calls }); + return { value: current!.value + 1 }; + }; + assert.equal((await cache.update(counterDocument, KEY, apply, { maxAttempts: 1 })).outcome, "lostRace"); + assert.equal(calls, 1); + }); +}); + +// An apply that declines resolves at once whatever the bound, so a validation that lets a bad value through fails the assertion +// below instead of looping: no apply here ever loses a race. +test("invalid maxAttempts (undefined is the default, null is refused) and a non-function reject before any call", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + let calls = 0; + const apply = () => (calls++, undefined); + for (const maxAttempts of [0, -1, 1.5, Number.NaN, Number.POSITIVE_INFINITY, null as never]) { + await assert.rejects(() => cache.update(counterDocument, KEY, apply, { maxAttempts }), RangeError, `maxAttempts ${String(maxAttempts)}`); + } + await assert.rejects(() => cache.update(counterDocument, KEY, "no" as never), TypeError); + assert.equal(calls, 0); + assert.equal((await cache.update(counterDocument, KEY, apply, { maxAttempts: undefined })).outcome, "declined"); + assert.equal(calls, 1); + }); +}); + +test("null from apply declines as undefined does: nothing is written", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + await cache.put(counterDocument, KEY, { value: 1 }); + const before = await readFile(path); + const result = await cache.update(counterDocument, KEY, () => null as never); + assert.deepEqual(result, { outcome: "declined", document: cache.get(counterDocument, KEY) }); + assert.deepEqual([...(await readFile(path))], [...before]); + }); +}); + +test("a cache with no store refuses a key held as another type before apply runs", async () => { + const cache = CultCache.builder().withRegistry(registry).build(); + await cache.put(tallyDocument, KEY, { count: 1 }); + let calls = 0; + await assert.rejects(() => cache.update(counterDocument, KEY, () => (calls++, { value: 1 })), /is not a "update.counter"/); + assert.equal(calls, 0); +}); + +test("a store attached while apply awaits turns the commit into a lost race, never a blind push", async () => { + await withDirectory(async (path) => { + const writer = await open(path); + await writer.put(counterDocument, KEY, { value: 50 }); + const cache = CultCache.builder().withRegistry(registry).build(); + const seen: Array = []; + const result = await cache.update(counterDocument, KEY, async (current) => { + seen.push(current?.value); + if (seen.length === 1) cache.addBackingStore(new SingleFileMessagePackBackingStore(path)); + return { value: (current?.value ?? 0) + 1 }; + }); + assert.deepEqual(seen, [undefined, 50], "the second attempt reads the store's record"); + assert.equal(result.outcome, "updated"); + assert.equal(stored(path), 51); + }); +}); + +test("put installs the storedAt the store stamped, so the cache and the file agree", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + await cache.put(counterDocument, KEY, { value: 1 }); + const store = new SingleFileMessagePackBackingStore(path); + const [envelope] = readSingleFileStore(path).envelopes; + await store.push({ ...envelope!, payload: documentPayloadFromBytes(encode({ value: 99 })), storedAt: "2099-01-01T00:00:00.000Z" }); + await cache.put(counterDocument, KEY, { value: 2 }); + const onDisk = readSingleFileStore(path).envelopes[0]!.storedAt; + assert.equal(onDisk, "2099-01-01T00:00:00.001Z"); + assert.equal(cache.getEnvelope(counterDocument, KEY)!.storedAt, onDisk); + }); +}); + +test("apply gets a fresh copy: changing it changes nothing held, and a lost attempt's change is not seen by the next", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + const argument: Array<{ value: number } | undefined> = []; + const seen: number[] = []; + await cache.update(counterDocument, KEY, async (current) => { + argument.push(current); + assert.notEqual(current, cache.get(counterDocument, KEY)); + seen.push(current!.value); + current!.value += 1000; + assert.equal(cache.get(counterDocument, KEY)!.value, 1); + if (seen.length === 1) await rival.put(counterDocument, KEY, { value: 7 }); + return { value: current!.value + 1 }; + }); + assert.deepEqual(seen, [1, 7], "the second attempt does not see the first one's change"); + assert.notEqual(argument[0], argument[1]); + }); +}); + +test("an in-place change to the held value is neither applied nor persisted", async () => { + await withDirectory(async (path) => { + const [cache, fresh] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + cache.get(counterDocument, KEY)!.value = 999; + const seen: number[] = []; + const result = await cache.update(counterDocument, KEY, (current) => { + seen.push(current!.value); + return { value: current!.value + 1 }; + }); + assert.deepEqual(seen, [1]); + assert.equal(stored(path), 2); + await fresh.pullAllBackingStores(); + assert.equal(fresh.get(counterDocument, KEY)!.value, 2); + result.document!.value = 999; + await cache.update(counterDocument, KEY, plus(1)); + assert.equal(stored(path), 3); + }); +}); + +test("undefined from apply declines: nothing is written", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + await cache.put(counterDocument, KEY, { value: 1 }); + const before = await readFile(path); + const held = await cache.update(counterDocument, KEY, () => undefined); + assert.equal(held.outcome, "declined"); + assert.equal(held.document, cache.get(counterDocument, KEY)); + const absent = await cache.update(counterDocument, "absent", () => undefined); + assert.deepEqual(absent, { outcome: "declined", document: undefined }); + assert.deepEqual([...(await readFile(path))], [...before]); + }); +}); + +test("an absent key is created once under a race", async () => { + await withDirectory(async (path) => { + const [left, right] = [await open(path), await open(path)]; + const outcomes = await Promise.all( + [left, right].map((cache) => + cache.update(counterDocument, KEY, async (current) => { + await tick(); + return { value: (current?.value ?? 0) + 1 }; + }), + ), + ); + assert.deepEqual(outcomes.map((result) => result.outcome), ["updated", "updated"]); + assert.equal(stored(path), 2); + }); +}); + +test("a rival writing and restoring the value still reruns apply", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + const seen: number[] = []; + await cache.update(counterDocument, KEY, async (current) => { + seen.push(current!.value); + if (seen.length === 1) { + await rival.put(counterDocument, KEY, { value: 5 }); + await rival.put(counterDocument, KEY, { value: 1 }); + } + return { value: current!.value + 1 }; + }); + assert.deepEqual(seen, [1, 1]); + assert.equal(stored(path), 2); + }); +}); + +test("a rival removal is seen as undefined", async () => { + await withDirectory(async (path) => { + const [cache, rival] = [await open(path), await open(path)]; + await cache.put(counterDocument, KEY, { value: 1 }); + const seen: Array = []; + await cache.update(counterDocument, KEY, async (current) => { + seen.push(current?.value); + if (seen.length === 1) { + await rival.pullAllBackingStores(); + await rival.delete(counterDocument, KEY); + } + return { value: (current?.value ?? 0) + 1 }; + }); + assert.deepEqual(seen, [1, undefined]); + assert.equal(stored(path), 1); + }); +}); + +test("a stale cache's first attempt reads the store", async () => { + await withDirectory(async (path) => { + const [stale, rival] = [await open(path), await open(path)]; + await rival.put(counterDocument, KEY, { value: 1 }); + await stale.pullAllBackingStores(); + await rival.put(counterDocument, KEY, { value: 7 }); + const seen: number[] = []; + await stale.update(counterDocument, KEY, (current) => { + seen.push(current!.value); + return { value: current!.value + 1 }; + }); + assert.deepEqual(seen, [7]); + assert.equal(stored(path), 8); + }); +}); + +test("refusals are not retried and apply is not called for a record it cannot copy", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + await cache.put(tallyDocument, "tally", { count: 1 }); + const before = await readFile(path); + let calls = 0; + const apply = (current: { value: number } | undefined) => { + calls++; + return { value: (current?.value ?? 0) + 1 }; + }; + // Another type at the key. + await assert.rejects(() => cache.update(counterDocument, "tally", apply), /not a "update.counter"/); + // A foreign record: a type this cache has no definition for. + const narrow = await open(path, defineDocumentRegistry(counterDocument)); + await assert.rejects(() => narrow.update(counterDocument, "tally", apply), /not a "update.counter"/); + assert.equal(calls, 0); + assert.deepEqual([...(await readFile(path))], [...before]); + + // A store without read and commitIf cannot take a conditional write. + const bare: CacheBackingStore = { pullAll: async () => [], push: async () => {}, delete: async () => {} }; + const plain = CultCache.builder().withRegistry(registry).withGenericStore(bare).build(); + await plain.pullAllBackingStores(); + await assert.rejects(() => plain.update(counterDocument, KEY, apply), /read and commitIf/); + assert.equal(calls, 0); + + // A record held read-only is refused by the judge after apply ran once; apply that throws is not retried either. + const surface = surfaceDocument(); + const entries = [envelopeOf(await read("extra-slot-record.msgpack"))]; + const readOnly = CultCache.builder().withDocumentType(surface).withGenericStore(memoryStore(entries)).build(); + await readOnly.pullAllBackingStores(); + let surfaceCalls = 0; + await assert.rejects(() => readOnly.update(surface, "surface", (value) => (surfaceCalls++, value)), SchemaConflictError); + assert.equal(surfaceCalls, 1); + assert.equal(entries.length, 1, "nothing reached the store"); + await assert.rejects( + () => cache.update(counterDocument, KEY, () => { calls++; throw new Error("boom"); }), + /boom/, + ); + assert.equal(calls, 1); + }); +}); + +test("own writes cause no retry, and updated returns the held instance", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + await cache.put(counterDocument, KEY, { value: 1 }); + let calls = 0; + const apply = (current: { value: number } | undefined) => { + calls++; + return { value: current!.value + 1 }; + }; + await cache.update(counterDocument, KEY, apply); + const second = await cache.update(counterDocument, KEY, apply); + assert.equal(calls, 2); + assert.equal(second.outcome, "updated"); + assert.equal(second.document, cache.get(counterDocument, KEY)); + assert.equal(stored(path), 3); + // The cache holds the record as the store wrote it, the stamp the store chose included: over a record stamped in the future + // the store writes one millisecond after it, whatever the clock says. + const future = readSingleFileStore(path).envelopes[0]!; + await new SingleFileMessagePackBackingStore(path).push({ ...future, payload: documentPayloadFromBytes(encode({ value: 99 })), storedAt: "2099-01-01T00:00:00.000Z" }); + await cache.update(counterDocument, KEY, apply); + assert.equal(readSingleFileStore(path).envelopes[0]!.storedAt, "2099-01-01T00:00:00.001Z"); + assert.equal(cache.getEnvelope(counterDocument, KEY)!.storedAt, "2099-01-01T00:00:00.001Z"); + }); +}); + +test("apply runs outside the cache's serial queue", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + let calls = 0; + const result = await cache.update(counterDocument, KEY, async (current) => { + calls++; + await Promise.race([ + cache.put(tallyDocument, "other", { count: 1 }), + new Promise((_, reject) => setTimeout(() => reject(new Error("a cache call from apply did not finish")), 5000).unref()), + ]); + return { value: (current?.value ?? 0) + 1 }; + }); + assert.equal(result.outcome, "updated"); + assert.equal(calls, 1); + }); +}); + +test("a cache with no store loses no increment under interleaved updates", async () => { + const cache = CultCache.builder().withRegistry(registry).build(); + await cache.pullAllBackingStores(); + let calls = 0; + const loop = async () => { + let updated = 0; + for (let i = 0; i < 50; i++) { + const result = await cache.update(counterDocument, KEY, async (current) => { + calls++; + await tick(); + return { value: (current?.value ?? 0) + 1 }; + }); + if (result.outcome === "updated") updated++; + } + return updated; + }; + // The two loops run in lockstep, so one may lose every round (a lostRace is the answer, not a lost write): what is held is + // exactly what the updated outcomes wrote. + const updated = (await Promise.all([loop(), loop()])).reduce((sum, n) => sum + n, 0); + assert.equal(cache.get(counterDocument, KEY)!.value, updated); + assert.ok(calls > updated, `the loops raced: apply ran ${calls} times for ${updated} writes`); +}); + +test("a store-less cache copies the bytes it admitted, not the held value", async () => { + const cache = CultCache.builder().withRegistry(registry).build(); + await cache.put(counterDocument, KEY, { value: 1 }); + cache.get(counterDocument, KEY)!.value = 999; + const seen: number[] = []; + await cache.update(counterDocument, KEY, (current) => (seen.push(current!.value), { value: current!.value + 1 })); + assert.deepEqual(seen, [1]); +}); + +test("updateGlobal updates the global record and refuses one loaded from a legacy key until putGlobal retires it", async () => { + await withDirectory(async (path) => { + const settings = defineDocumentRegistry(settingsDocument); + const cache = await open(path, settings); + const first = await cache.updateGlobal(settingsDocument, (current) => ({ theme: `${current?.theme ?? ""}a` })); + assert.equal(first.outcome, "updated"); + assert.deepEqual(cache.getGlobal(settingsDocument), { theme: "a" }); + assert.equal(readSingleFileStore(path).envelopes[0]!.key, CultCache.GLOBAL_KEY); + await rm(path); + + const legacy = await open(path, defineDocumentRegistry(plainSettingsDocument)); + await legacy.put(plainSettingsDocument, "old-settings", { theme: "x" }); + const loaded = await open(path, settings); + let calls = 0; + await assert.rejects(() => loaded.updateGlobal(settingsDocument, (current) => (calls++, current)), /putGlobal to retire its legacy key/); + assert.equal(calls, 0); + // Every path to the write is refused, the global's own key included, and the legacy record survives. + const before = await readFile(path); + await assert.rejects(() => loaded.update(settingsDocument, CultCache.GLOBAL_KEY, () => ({ theme: "lost" })), /putGlobal to retire its legacy key/); + assert.deepEqual([...(await readFile(path))], [...before]); + assert.deepEqual(readSingleFileStore(path).envelopes.map((e) => e.key), ["old-settings"]); + await loaded.putGlobal(settingsDocument, { theme: "y" }); + assert.equal((await loaded.updateGlobal(settingsDocument, (current) => ({ theme: `${current!.theme}z` }))).outcome, "updated"); + assert.deepEqual(loaded.getGlobal(settingsDocument), { theme: "yz" }); + await assert.rejects(() => cache.update(settingsDocument, "not-global", () => ({ theme: "q" })), /must use key/); + }); +}); + +test("a legacy global that a pull loads while apply awaits is refused at the write, and its legacy record survives", async () => { + await withDirectory(async (path) => { + const settings = defineDocumentRegistry(settingsDocument); + const cache = await open(path, settings); + const legacy = await open(path, defineDocumentRegistry(plainSettingsDocument)); + await assert.rejects( + () => + cache.update(settingsDocument, CultCache.GLOBAL_KEY, async () => { + await legacy.put(plainSettingsDocument, "old-settings", { theme: "x" }); + await cache.pullAllBackingStores(); + return { theme: "lost" }; + }), + /putGlobal to retire its legacy key/, + ); + assert.deepEqual(readSingleFileStore(path).envelopes.map((e) => e.key), ["old-settings"]); + }); +}); + +test("the ledger holds under contention: the stored value is the number of updated outcomes", async () => { + await withDirectory(async (path) => { + const caches = await Promise.all([open(path), open(path), open(path)]); + const results = await Promise.all( + caches.map(async (cache) => { + let updated = 0; + for (let i = 0; i < 25; i++) { + const result = await cache.update(counterDocument, KEY, async (current) => { + await tick(); + return { value: (current?.value ?? 0) + 1 }; + }); + if (result.outcome === "updated") updated++; + } + return updated; + }), + ); + assert.equal(stored(path), results.reduce((sum, n) => sum + n, 0)); + }); +}); + +test("a flush of an unchanged cache rewrites the file byte for byte, and an update does not lose a race to it", async () => { + await withDirectory(async (path) => { + const cache = await open(path); + await cache.put(counterDocument, KEY, { value: 1 }); + await cache.put(counterDocument, "other", { value: 7 }); + const store = new SingleFileMessagePackBackingStore(path); + const before = await readFile(path); + // What CultMesh.flush does: push the whole stored image, twice. + await store.pushAll(await store.pullAll()); + await store.pushAll(await store.pullAll()); + assert.deepEqual(await readFile(path), before); + const result = await cache.update(counterDocument, KEY, async (current) => { + await store.pushAll(await store.pullAll()); + return { value: current!.value + 1 }; + }, { maxAttempts: 1 }); + assert.equal(result.outcome, "updated"); + assert.equal(stored(path), 2); + }); +}); + +test("a changed record is re-stamped later than the one it replaces, and a revert to earlier bytes too", async () => { + await withDirectory(async (path) => { + const store = new SingleFileMessagePackBackingStore(path); + const cache = await open(path); + await cache.put(counterDocument, KEY, { value: 1 }); + const first = readSingleFileStore(path).envelopes[0]!; + const same = await store.push({ ...first, storedAt: "2099-01-01T00:00:00.000Z" }); + assert.equal(same.storedAt, first.storedAt, "identical content keeps its stamp"); + await cache.put(counterDocument, KEY, { value: 2 }); + const second = readSingleFileStore(path).envelopes[0]!; + const reverted = await store.push({ ...first, storedAt: second.storedAt }); + assert.ok(Date.parse(reverted.storedAt) > Date.parse(second.storedAt)); + }); +}); + +test("a push result that is not a parseable stamp is ignored", async () => { + for (const result of [true, {}, "", "garbage", { storedAt: "" }, { storedAt: "garbage" }]) { + const entries: Parameters[0] = []; + const store = { ...memoryStore(entries), push: async (entry: (typeof entries)[number]) => { entries.push(entry); return result as never; } }; + const cache = CultCache.builder().withRegistry(registry).withGenericStore(store).build(); + await cache.put(counterDocument, KEY, { value: 1 }); + const held = cache.getEnvelope(counterDocument, KEY)!.storedAt; + assert.equal(Number.isNaN(Date.parse(held)), false, `held stamp ${held} after a push result of ${JSON.stringify(result)}`); + } +}); + +test("a rewrite of identical content keeps the stamp but writes the writer's type and catalog entry", async () => { + await withDirectory(async (path) => { + const old = defineDocumentType({ type: "update.rn.old", schemaId: "update.rn.v1", schemaName: "update.rn.old", schemaVersion: "update.rn.v1", schema: z.object({ value: z.number().int() }) }); + const renamed = defineDocumentType({ type: "update.rn.new", schemaId: "update.rn.v1", schemaName: "update.rn.new", schemaVersion: "update.rn.v1", schema: z.object({ value: z.number().int() }) }); + await (await open(path, defineDocumentRegistry(old))).put(old, KEY, { value: 1 }); + const before = readSingleFileStore(path).envelopes[0]!; + await (await open(path, defineDocumentRegistry(renamed))).put(renamed, KEY, { value: 1 }); + const after = readSingleFileStore(path).envelopes[0]!; + assert.equal(after.storedAt, before.storedAt, "identical content keeps its stamp"); + assert.equal(after.type, "update.rn.new", "the writer's type is what the file says"); + }); +}); + +test("a rewrite of the same bytes under another schema id is a changed record: stamped later, written under the new id", async () => { + await withDirectory(async (path) => { + const v1 = defineDocumentType({ type: "update.sid", schemaId: "update.sid.v1", schemaName: "update.sid", schemaVersion: "update.sid.v1", schema: z.object({ value: z.number().int() }) }); + const v2 = defineDocumentType({ type: "update.sid", schemaId: "update.sid.v2", schemaName: "update.sid", schemaVersion: "update.sid.v2", compatibleSchemaIds: ["update.sid.v1"], schema: z.object({ value: z.number().int() }) }); + await (await open(path, defineDocumentRegistry(v1))).put(v1, KEY, { value: 1 }); + const before = readSingleFileStore(path).envelopes[0]!; + await (await open(path, defineDocumentRegistry(v2))).put(v2, KEY, { value: 1 }); + const after = readSingleFileStore(path).envelopes[0]!; + assert.ok(Date.parse(after.storedAt) > Date.parse(before.storedAt)); + assert.equal(schemaIdForEnvelope(after), "update.sid.v2"); + }); +}); diff --git a/packages/cultmath/README.md b/packages/cultmath/README.md index b144c56d..6760a1f5 100644 --- a/packages/cultmath/README.md +++ b/packages/cultmath/README.md @@ -147,8 +147,8 @@ CultMath names with tests instead of copied project-local helper drift. ```powershell dotnet build packages/cultmath/src/CultMath/CultMath.csproj dotnet test packages/cultmath/tests/CultMath.Tests/CultMath.Tests.csproj -cargo test --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml -cargo build --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml +cargo test --locked --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml +cargo build --locked --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml .\packages\cultmath\scripts\build-unity-package.ps1 ``` diff --git a/packages/cultmesh-browser/CHANGELOG.md b/packages/cultmesh-browser/CHANGELOG.md index ccda16a0..e55344ba 100644 --- a/packages/cultmesh-browser/CHANGELOG.md +++ b/packages/cultmesh-browser/CHANGELOG.md @@ -9,6 +9,10 @@ All notable changes to `@gamecult/cultmesh-browser` are recorded here, following - The package is renamed `cultmesh-browser` to `@gamecult/cultmesh-browser` (the repo already scopes `@gamecult/cultcache-ts`), so the unscoped name cannot be claimed by someone else. Import from `@gamecult/cultmesh-browser` and its subpaths; the API is unchanged. Depends on `@gamecult/cultnet-ts`; the interop test uses `@gamecult/cultmesh-ts`. +### Dependencies + +- Needs `@gamecult/cultnet-ts` `^0.4.0` (was `^0.3.0`), moving with the `cultnet-ts` 0.4.0 release. + ## [0.4.0] ### Added diff --git a/packages/cultmesh-browser/package.json b/packages/cultmesh-browser/package.json index 0984ae6d..5e72bfbc 100644 --- a/packages/cultmesh-browser/package.json +++ b/packages/cultmesh-browser/package.json @@ -41,8 +41,8 @@ "node": ">=20" }, "dependencies": { - "@gamecult/cultcache-ts": "^0.15.0", - "@gamecult/cultnet-ts": "^0.3.0" + "@gamecult/cultcache-ts": "^0.16.0", + "@gamecult/cultnet-ts": "^0.4.0" }, "devDependencies": { "@types/node": "^24.0.0", diff --git a/packages/cultmesh-py/pyproject.toml b/packages/cultmesh-py/pyproject.toml index 5f9bda4d..18f23060 100644 --- a/packages/cultmesh-py/pyproject.toml +++ b/packages/cultmesh-py/pyproject.toml @@ -16,7 +16,7 @@ classifiers = [ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ] -dependencies = ["cultcache-py>=0.2.0", "cultnet-py>=0.1.0", "cryptography>=41"] +dependencies = ["cultcache-py>=0.4.0", "cultnet-py>=0.1.0", "cryptography>=41"] [project.scripts] cultmesh-py-daemon = "cultmesh_py.daemon:main" cultmesh-py-verify = "cultmesh_py.verify:main" diff --git a/packages/cultmesh-py/src/cultmesh_py/node.py b/packages/cultmesh-py/src/cultmesh_py/node.py index 619aed36..eb439683 100644 --- a/packages/cultmesh-py/src/cultmesh_py/node.py +++ b/packages/cultmesh-py/src/cultmesh_py/node.py @@ -12,6 +12,7 @@ from typing import Any, Callable from cultcache_py import CultCache, CultCacheEnvelope, SingleFileMessagePackBackingStore from cultcache_py.cache import CultCacheError, index_text from cultcache_py.documents import DocumentDefinition, extract_value +from cultnet_py.replication import raw_record_to_envelope from cultnet_py import ( CultNetAppliedRecord, CultNetFileShardMutationLogStore, @@ -570,12 +571,14 @@ class CultMeshDatabase: catalog_entry=catalog_entry, ) value = self.cache.put_envelope(registered, envelope) + # The stamp the write was stored at, which a replacing write mints later than the record it replaces: the cache holds it. + stored_at = self.cache.get_required_envelope(registered, key).stored_at self._publish_local_change(registered, key, "added" if previous is None else "updated", value, previous) message = document_put_raw( message_id=message_id, key=key, schema_id=envelope.schema_id or catalog_entry.schema_id, - stored_at=envelope.stored_at, + stored_at=stored_at, payload=envelope.payload, source_runtime_id=self.runtime_id, shard_id=shard_id, @@ -662,10 +665,10 @@ class CultMeshDatabase: ): return None try: - prepared = self._raw_put_envelope(message) + document = self._raw_put_document(message) except Exception: # noqa: BLE001 - any other malformed shape return None - return None if prepared is None else (prepared[0].catalog_entry().schema_id, prepared[1].key) + return None if document is None else (document.catalog_entry().schema_id, document_record["recordKey"]) def raw_delete_target(self, message: dict[str, Any]) -> tuple[str, str] | None: """The canonical schema id and record key a raw delete names, or None when its schema is not registered.""" @@ -683,10 +686,11 @@ class CultMeshDatabase: value = self.cache.put_envelope(document, envelope) change_kind = "added" if previous is None else "updated" change = self._publish_local_change(document, record_key, change_kind, value, previous) - self._append_shard_log_put(CultNetMessage( - "cultnet.document_put_raw.v0", - {key: value for key, value in message.items() if key != "schemaVersion"}, - ), change_kind) + # The log carries the stamp the cache holds, which a replacing put mints later than the record it replaces, not the peer's. + stored_at = self.cache.get_required_envelope(document, record_key).stored_at + logged = {key: value for key, value in message.items() if key != "schemaVersion"} + logged["document"] = {**message["document"], "storedAt": stored_at} + self._append_shard_log_put(CultNetMessage("cultnet.document_put_raw.v0", logged), change_kind) return change def raw_put_served_alone(self, message: dict[str, Any], *, message_id: str) -> dict[str, Any] | None: @@ -703,27 +707,25 @@ class CultMeshDatabase: documents=(self._served_record(prepared[1]),), ).to_wire() - def _raw_put_envelope( - self, message: dict[str, Any] - ) -> tuple[DocumentDefinition[Any], CultCacheEnvelope] | None: + def _raw_put_document(self, message: dict[str, Any]) -> DocumentDefinition[Any] | None: + """The registered document a raw put names, or None when its record is not a map or its schema is not registered.""" document_record = message.get("document") if not isinstance(document_record, dict): return None schema_id = str(document_record.get("schemaId") or "") - documents_by_schema_id = schema_document_map(self.documents) try: - document = resolve_document_for_raw_record(documents_by_schema_id, schema_id, document_record) + return resolve_document_for_raw_record(schema_document_map(self.documents), schema_id, document_record) except KeyError: return None - # One id: the local schema id, under which the cache stores the record. - return document, CultCacheEnvelope( - key=str(document_record["recordKey"]), - type=document.type, - schema_id=document.catalog_entry().schema_id, - payload=bytes(document_record["payload"]), - stored_at=str(document_record.get("storedAt") or ""), - catalog_entry=document.catalog_entry(), - ) + + def _raw_put_envelope( + self, message: dict[str, Any] + ) -> tuple[DocumentDefinition[Any], CultCacheEnvelope] | None: + document = self._raw_put_document(message) + if document is None: + return None + # One id: the local schema id, under which the cache stores the record. The stamp is judged where a record becomes an envelope. + return document, raw_record_to_envelope(document, document.catalog_entry().schema_id, message["document"]) @staticmethod def _served_record(envelope: CultCacheEnvelope) -> CultNetRawDocumentRecord: diff --git a/packages/cultmesh-py/src/cultmesh_py/server.py b/packages/cultmesh-py/src/cultmesh_py/server.py index ab5267a7..363cddbf 100644 --- a/packages/cultmesh-py/src/cultmesh_py/server.py +++ b/packages/cultmesh-py/src/cultmesh_py/server.py @@ -10,6 +10,7 @@ from typing import Any import msgpack +from cultcache_py.stores import parse_stored_at from cultnet_py import ( error_message, message_invalid_error, @@ -686,9 +687,11 @@ class CultMeshLocalServer: message_id=str(message.get("messageId") or ""), code="unregistered_document_put", )] + # Subscribers are sent the stamp the store holds, which a replacing put mints later than the record it replaces, not the peer's. + stored = self.node.database.cache.get_required_envelope(change.document, change.record_key).stored_at return self._database_change_notifications( message, - document_record, + {**document_record, "storedAt": stored}, change.change_kind, subscriptions, resolved_schema_id=change.schema_id, @@ -967,7 +970,7 @@ class CultMeshLocalServer: as a snapshot would serve it under the shortest message id, must fit max_snapshot_bytes; a put arriving over RUDP must also fit, as fragments of rudp_max_fragment_bytes, in one message and the peer's reliable queue. A put whose document has no string recordKey or - binary payload could never be stored or served, and is refused as malformed.""" + binary payload, or a storedAt that is no stamp, could never be stored or served, and is refused as malformed.""" document_record = message.get("document") if not ( isinstance(document_record, dict) @@ -975,6 +978,8 @@ class CultMeshLocalServer: and isinstance(document_record.get("payload"), (bytes, bytearray)) ): return self._message_invalid(message, "document", "Raw put documents must carry a string recordKey and a binary payload.") + if parse_stored_at(document_record.get("storedAt")) is None: + return self._message_invalid(message, "document", "Raw put documents must carry a storedAt stamp.") if self.max_snapshot_bytes is None and not over_rudp: return None alone = self.node.database.raw_put_served_alone(message, message_id=_SHORTEST_SERVED_MESSAGE_ID) diff --git a/packages/cultmesh-py/tests/test_cultmesh.py b/packages/cultmesh-py/tests/test_cultmesh.py index d96c4ab2..4595f29c 100644 --- a/packages/cultmesh-py/tests/test_cultmesh.py +++ b/packages/cultmesh-py/tests/test_cultmesh.py @@ -19,6 +19,7 @@ from typing import Any, Callable import msgpack from cultcache_py import ( CultCache, + CultCacheEnvelope, SingleFileMessagePackBackingStore, define_database_entry_type, define_document_type, @@ -796,6 +797,51 @@ class CultMeshTests(unittest.TestCase): self.assertEqual(node.get_required(document, "note:second").body, "second catalog publication") self.assertIsInstance(node.get_required(alias, "note:second"), UiNote) + def test_raw_put_publishes_and_logs_the_stamp_the_store_wrote(self) -> None: + document = define_database_entry_type("mesh.stamp_item", [("name", 0), ("category", 1), ("value", 2, 0)], cls=Item) + node = create_node(runtime_id="python-node") + node.register_document(document) + cache = node.database.cache + entry = document.catalog_entry() + future = "2099-01-01T00:00:00.000000+00:00" + cache.put_envelope( + document, + CultCacheEnvelope( + key="item:stamp", + type=document.type, + payload=document.encode_payload(Item("old", "gear", 1)), + stored_at=future, + schema_id=entry.schema_id, + catalog_entry=entry, + ), + ) + # A replacing write is stored later than the record it replaces, so the stamp is not the one the write was created with. + wire = node.put_raw_message(document, "item:stamp", Item("new", "gear", 2), shard_id="mesh", shard_epoch=1).to_wire() + held = cache.get_required_envelope(document, "item:stamp").stored_at + self.assertGreater(held, future) + self.assertEqual(wire["document"]["storedAt"], held) + self.assertEqual(node.database._shard_logs["mesh"][-1]["put"]["document"]["storedAt"], held) + + def test_applied_raw_put_logs_the_stamp_the_cache_holds(self) -> None: + document = define_database_entry_type("mesh.applied_stamp", [("name", 0), ("category", 1), ("value", 2, 0)], cls=Item) + node = create_node(runtime_id="python-node") + node.register_document(document) + cache = node.database.cache + entry = document.catalog_entry() + + def envelope(value: int, stored_at: str) -> CultCacheEnvelope: + return CultCacheEnvelope("item:applied", document.type, document.encode_payload(Item("n", "gear", value)), stored_at, entry.schema_id, entry) + + cache.put_envelope(document, envelope(1, "2099-01-01T00:00:00.000000+00:00")) + peer = node.put_raw_message(document, "item:peer", Item("peer", "gear", 2)).to_wire() # a well-formed put to replay + peer["document"] = {**peer["document"], "recordKey": "item:applied", "storedAt": "2000-01-01T00:00:00Z", + "payload": envelope(2, "x").payload} + peer["shardId"], peer["shardEpoch"] = "mesh", 1 + node.database.apply_raw_put_message(peer) + held = cache.get_required_envelope(document, "item:applied").stored_at + self.assertGreater(held, "2099-01-01T00:00:00.000000+00:00") + self.assertEqual(node.database._shard_logs["mesh"][-1]["put"]["document"]["storedAt"], held) + def test_cultmesh_node_emits_raw_put_and_delete_messages_for_local_writes(self) -> None: document = define_database_entry_type( "mesh.emit_item", diff --git a/packages/cultmesh-py/tests/test_remote_write_authority.py b/packages/cultmesh-py/tests/test_remote_write_authority.py index cafb163f..7397577d 100644 --- a/packages/cultmesh-py/tests/test_remote_write_authority.py +++ b/packages/cultmesh-py/tests/test_remote_write_authority.py @@ -8,6 +8,7 @@ the peer the connection loop would pass, and one test drives the same path over from __future__ import annotations import json +import tempfile import unittest from pathlib import Path from typing import Any @@ -23,7 +24,7 @@ from cultmesh_py import ( CultMeshServerPeer, create_node, ) -from cultmesh_py.server import CultMeshLocalServer +from cultmesh_py.server import CultMeshLocalServer, _DatabaseSubscription from cultnet_py import CultNetPeerError, create_rudp_schema_transport, document_delete, document_put_raw, error_message SCHEMA_ID = "sha256:authority-note" @@ -55,12 +56,12 @@ class Recording: return self._decide(request) -def put_of(key: str, text: str, *, schema_id: str = SCHEMA_ID) -> dict[str, Any]: +def put_of(key: str, text: str, *, schema_id: str = SCHEMA_ID, stored_at: str = "2026-11-07T00:00:00Z") -> dict[str, Any]: return document_put_raw( message_id="wire", key=key, schema_id=schema_id, - stored_at="2026-11-07T00:00:00Z", + stored_at=stored_at, payload=NOTE.encode_payload({"schema_version": "tests.authority_note.v1", "text": text}), shard_id=SHARD_ID, shard_epoch=1, @@ -73,6 +74,64 @@ def delete_of(key: str, *, schema_id: str = SCHEMA_ID) -> dict[str, Any]: ).to_wire() +class NotificationStampTests(unittest.TestCase): + """A subscriber is sent the stamp the store holds after a peer's put, as the shard log is, not the peer's stamp.""" + + def test_the_notification_carries_the_stamp_the_cache_holds(self) -> None: + local, peer_stamp = "2099-01-01T00:00:00.000000+00:00", "2000-01-01T00:00:00Z" + with tempfile.TemporaryDirectory() as directory: + for backing in (None, Path(directory) / "node.cc"): # store-less, then store-backed + with self.subTest(store=backing is not None): + node = create_node(backing, runtime_id="stamp-server") + node.database.register_document(NOTE) + server = CultMeshLocalServer(node=node, port=0, write_authority=CultMeshGrantAllWrites()) + node.database.apply_raw_put_message(put_of(KEY_A, "server", stored_at=local)) + subscriptions = {"s": _DatabaseSubscription("s", set(), set())} + replies = server._handle_connection_message(put_of(KEY_A, "peer", stored_at=peer_stamp), subscriptions, peer=PEER) + held = node.cache.get_required_envelope(NOTE, KEY_A).stored_at + self.assertGreater(held, local) + self.assertEqual([reply["document"]["storedAt"] for reply in replies], [held]) + self.assertEqual(node.database._shard_logs[SHARD_ID][-1]["put"]["document"]["storedAt"], held) + + +class StamplessPutTests(unittest.TestCase): + """A peer's put with no storedAt, or one that is no stamp, is refused message_invalid; nothing is minted or stored.""" + + def test_a_put_without_a_stamp_is_refused_and_the_store_stays_writable(self) -> None: + with tempfile.TemporaryDirectory() as directory: + node = create_node(Path(directory) / "node.cc", runtime_id="stampless-server") + node.database.register_document(NOTE) + server = CultMeshLocalServer(node=node, port=0, write_authority=CultMeshGrantAllWrites()) + sized = CultMeshLocalServer(node=node, port=0, write_authority=CultMeshGrantAllWrites(), max_snapshot_bytes=10**6) + rudp = CultMeshLocalServer(node=node, port=0, write_authority=CultMeshGrantAllWrites()) + for label, stamp in (("missing", None), ("empty", ""), ("garbage", "yesterday"), ("no zone", "2026-11-07T00:00:00")): + with self.subTest(label): + message = put_of(KEY_A, "peer") + del message["document"]["storedAt"] + if stamp is not None: + message["document"]["storedAt"] = stamp + # The size-limited and RUDP paths measure the document as a snapshot would serve it, so they must refuse first. + for limited in (server, sized, rudp): + [refusal] = limited._handle_connection_message(message, {}, over_rudp=limited is rudp, peer=PEER) + self.assertEqual(refusal["code"], "message_invalid") + self.assertEqual(refusal["details"]["field"], "document") + with self.assertRaises(ValueError): + node.database.apply_raw_put_message(message) + self.assertIsNone(node.cache.get(NOTE, KEY_A)) + self.assertNotIn(SHARD_ID, node.database._shard_logs) + # An aliased put with a bad stamp still asks the authority about the canonical id, and is refused before the stamp is judged. + authority = Recording(lambda request: CultMeshRemoteWriteDecision.refuse("no")) + guarded = CultMeshLocalServer(node=node, port=0, write_authority=authority) + aliased = put_of(KEY_A, "peer", schema_id=ALIAS, stored_at="yesterday") + [refusal] = guarded._handle_connection_message(aliased, {}, peer=PEER) + self.assertEqual(refusal["code"], "write_not_authorized") + self.assertEqual(authority.requests[0].schema_id, SCHEMA_ID) + # The refused puts left the file readable: the next put of any key is stored. + replies = server._handle_connection_message(put_of(KEY_B, "stamped"), {}, peer=PEER) + self.assertEqual(replies, []) + self.assertEqual(node.cache.get(NOTE, KEY_B)["text"], "stamped") + + class RemoteWriteAuthorityTests(unittest.TestCase): def build(self, authority: Any = None) -> CultMeshLocalServer: node = create_node(runtime_id="authority-server") @@ -180,6 +239,21 @@ class RemoteWriteAuthorityTests(unittest.TestCase): reply = self.send(granting, unowned) self.assertEqual(reply[0]["code"], "unregistered_document_put", "a granted put still meets the store's refusal") + def test_a_put_with_a_payload_tail_and_an_unknown_schema_id_is_refused_whole_under_the_canonical_id(self) -> None: + # The payload head names the registered schema, so the put routes by stamp; the tail makes the payload more + # than one MessagePack value. It is answered message_invalid (not unregistered_document_put), nothing is + # stored, and the authority is asked under the canonical id, not the id as sent. + authority = Recording(lambda _: CultMeshRemoteWriteDecision.grant()) + server = self.build(authority) + tailed = put_of(KEY_A, "x", schema_id="sha256:nobody-owns-this") + tailed["document"]["payload"] = bytes(tailed["document"]["payload"]) + b"\x01" + + replies = self.send(server, tailed) + + self.assertEqual([(reply["code"], reply["details"]["field"]) for reply in replies], [("message_invalid", "document")]) + self.assertIsNone(self.text_of(server, KEY_A)) + self.assertEqual([request.schema_id for request in authority.requests], [SCHEMA_ID]) + def test_an_authority_that_raises_refuses_and_is_logged_as_an_error(self) -> None: def explode(_: Any) -> Any: raise RuntimeError("authority exploded") diff --git a/packages/cultmesh-py/tests/test_stampless_replication.py b/packages/cultmesh-py/tests/test_stampless_replication.py new file mode 100644 index 00000000..6acb647a --- /dev/null +++ b/packages/cultmesh-py/tests/test_stampless_replication.py @@ -0,0 +1,212 @@ +"""A record a peer replicates to this node is stored under the stamp it carries, so one with no stamp is refused where it becomes an envelope. + +The stamp shape is tests/vectors/stored-at-stamp/stamps.msgpack, held to the same answers as the raw put of the server +(variants:ruling:raw-put-without-stamp-refused). A snapshot response is refused whole with ValueError before anything is stored (as a record of an unregistered schema is refused whole). A shard log is +refused whole too: every put it would apply is judged for its stamp before the first entry is stored (a put of an unregistered schema is skipped, as ever, and is not judged). The refusal of a put is +message_invalid (field `document`), tested in test_remote_write_authority.py. +""" + +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path +from typing import Any + +import msgpack +from cultcache_py import define_database_entry_type +from cultcache_py.stores import parse_stored_at +from cultmesh_py import CultMeshGrantAllWrites, CultMeshServerPeer, create_node +from cultmesh_py.client import CultMeshDocumentSubscription +from cultmesh_py.server import CultMeshLocalServer +from cultnet_py import CultNetDatabaseChange, document_put_raw + +SCHEMA_ID = "sha256:stampless-note" +SHARD_ID = "stampless-shard" +PEER = CultMeshServerPeer("127.0.0.1:50000", "tcp") +VECTOR = Path(__file__).resolve().parents[3] / "tests/vectors/stored-at-stamp/stamps.msgpack" +ROWS: list[tuple[str, bool]] = [(text, parses) for text, parses in msgpack.unpackb(VECTOR.read_bytes(), raw=False)] +# The vector's texts, plus the shapes a record can lack a stamp in: the field missing, and not a string at all. +REFUSED: list[tuple[str, Any]] = [(f"vector {text!r}", text) for text, parses in ROWS if not parses] + [ + ("missing", None), + ("not a string", 12345), +] + +NOTE = define_database_entry_type( + "tests.stampless_note", + [("schema_version", 0), ("text", 1)], + schema_id=SCHEMA_ID, + schema_name="tests.stampless_note", + schema_version="tests.stampless_note.v1", +) + + +def record_of(key: str, text: str, stamp: Any = "2026-11-07T00:00:00Z") -> dict[str, Any]: + record = { + "schemaId": SCHEMA_ID, + "recordKey": key, + "payloadEncoding": "messagepack", + "payload": NOTE.encode_payload({"schema_version": "tests.stampless_note.v1", "text": text}), + } + if stamp is not None: + record["storedAt"] = stamp + return record + + +def snapshot_of(*records: dict[str, Any]) -> dict[str, Any]: + return {"schemaVersion": "cultnet.snapshot_response_raw.v0", "messageId": "snapshot", "documents": list(records)} + + +def shard_log_of(*records: dict[str, Any]) -> dict[str, Any]: + entries = [ + { + "sequence": index + 1, + "committedAt": "2026-11-07T00:00:00Z", + "changeKind": "added", + "put": document_put_raw( + message_id=f"put-{index}", key=record["recordKey"], schema_id=SCHEMA_ID, stored_at="2026-11-07T00:00:00Z", + payload=record["payload"], shard_id=SHARD_ID, shard_epoch=1, + ).to_wire() | {"document": record}, + } + for index, record in enumerate(records) + ] + return { + "schemaVersion": "cultnet.shard_log_response.v0", "messageId": "log", "shardId": SHARD_ID, "shardEpoch": 1, + "entries": entries, "resyncRequired": False, + } + + +class StampVectorTests(unittest.TestCase): + def test_python_parses_every_row_of_the_vector_as_the_vector_says(self) -> None: + self.assertGreater(len(ROWS), 30) + for text, parses in ROWS: + with self.subTest(text=text): + self.assertEqual(parse_stored_at(text) is not None, parses) + + def test_every_vector_row_through_the_server_put_handler_is_stored_or_refused_as_the_vector_says(self) -> None: + with tempfile.TemporaryDirectory() as directory: + node = create_node(Path(directory) / "node.cc", runtime_id="vector-server") + node.database.register_document(NOTE) + server = CultMeshLocalServer(node=node, port=0, write_authority=CultMeshGrantAllWrites()) + sized = CultMeshLocalServer(node=node, port=0, write_authority=CultMeshGrantAllWrites(), max_snapshot_bytes=10**6) + for index, (text, parses) in enumerate(ROWS): + for label, handler, over_rudp in (("plain", server, False), ("sized", sized, False), ("rudp", server, True)): + with self.subTest(text=text, path=label): + key = f"vector:{index}:{label}" + message = document_put_raw( + message_id="wire", key=key, schema_id=SCHEMA_ID, stored_at=text, + payload=record_of(key, "peer")["payload"], shard_id=SHARD_ID, shard_epoch=1, + ).to_wire() + replies = handler._handle_connection_message(message, {}, over_rudp=over_rudp, peer=PEER) + if parses: + self.assertEqual(replies, []) + self.assertEqual(node.cache.get(NOTE, key)["text"], "peer") + else: + self.assertEqual([(r["code"], r["details"]["field"]) for r in replies], [("message_invalid", "document")]) + self.assertIsNone(node.cache.get(NOTE, key)) + + +class StamplessSnapshotTests(unittest.TestCase): + def build(self, directory: str) -> Any: + node = create_node(Path(directory) / "node.cc", runtime_id="stampless-client") + node.database.register_document(NOTE) + node.database.pull() + return node + + def test_a_snapshot_with_a_stampless_record_is_refused_whole_and_the_store_survives_a_reopen(self) -> None: + for label, stamp in REFUSED: + with self.subTest(label), tempfile.TemporaryDirectory() as directory: + node = self.build(directory) + good, bad = record_of("note:good", "good"), record_of("note:bad", "bad", stamp) + for order in ((good, bad), (bad, good)): + with self.assertRaises(ValueError): + node.database.apply_snapshot_response(snapshot_of(*order)) + self.assertIsNone(node.cache.get(NOTE, "note:good"), "a refused snapshot stores none of its records") + self.assertIsNone(node.cache.get(NOTE, "note:bad")) + # The store file reads back, and is still writable by a snapshot of stamped records. + reopened = self.build(directory) + self.assertIsNone(reopened.cache.get(NOTE, "note:bad")) + applied = reopened.database.apply_snapshot_response(snapshot_of(good)) + self.assertEqual([a.record_key for a in applied], ["note:good"]) + self.assertEqual(self.build(directory).cache.get(NOTE, "note:good")["text"], "good") + + def test_a_snapshot_of_every_stamped_vector_row_is_stored(self) -> None: + with tempfile.TemporaryDirectory() as directory: + node = self.build(directory) + rows = [(index, text) for index, (text, parses) in enumerate(ROWS) if parses] + node.database.apply_snapshot_response(snapshot_of(*[record_of(f"stamped:{i}", "ok", text) for i, text in rows])) + self.assertEqual(len(self.build(directory).cache.get_all(NOTE)), len(rows)) + for index, _ in rows: + self.assertEqual(node.cache.get(NOTE, f"stamped:{index}")["text"], "ok") + + +class StamplessShardLogTests(unittest.TestCase): + def test_a_shard_log_put_with_no_stamp_is_refused_and_the_store_stays_writable(self) -> None: + for label, stamp in REFUSED: + with self.subTest(label), tempfile.TemporaryDirectory() as directory: + node = create_node(Path(directory) / "node.cc", runtime_id="stampless-log") + node.database.register_document(NOTE) + with self.assertRaises(ValueError): + node.database.apply_shard_log_response(shard_log_of(record_of("note:bad", "bad", stamp))) + self.assertIsNone(node.cache.get(NOTE, "note:bad")) + reopened = create_node(Path(directory) / "node.cc", runtime_id="stampless-log") + reopened.database.register_document(NOTE) + reopened.database.pull() + reopened.database.apply_shard_log_response(shard_log_of(record_of("note:good", "good"))) + self.assertEqual(reopened.cache.get(NOTE, "note:good")["text"], "good") + again = create_node(Path(directory) / "node.cc", runtime_id="stampless-log") + again.database.register_document(NOTE) + again.database.pull() + self.assertEqual(again.cache.get(NOTE, "note:good")["text"], "good") + + + def test_a_shard_log_with_a_bad_entry_in_the_middle_stores_nothing_and_tells_no_watcher(self) -> None: + for label, stamp in REFUSED: + with self.subTest(label), tempfile.TemporaryDirectory() as directory: + node = create_node(Path(directory) / "node.cc", runtime_id="stampless-multi") + node.database.register_document(NOTE) + seen: list[Any] = [] + node.database.watch(seen.append) + log = shard_log_of(record_of("note:a", "a"), record_of("note:b", "b", stamp), record_of("note:c", "c")) + with self.assertRaises(ValueError): + node.database.apply_shard_log_response(log) + for key in ("note:a", "note:b", "note:c"): + self.assertIsNone(node.cache.get(NOTE, key), f"{key} was not stored") + self.assertEqual(seen, [], "no watcher heard of a refused log") + node.database.apply_shard_log_response(shard_log_of(record_of("note:a", "a"), record_of("note:c", "c"))) + self.assertEqual([node.cache.get(NOTE, k)["text"] for k in ("note:a", "note:c")], ["a", "c"]) + self.assertEqual(len(seen), 2, "a later good log applies and is heard") + + +class _StubSubscription: + def __init__(self, change: CultNetDatabaseChange) -> None: + self._change = change + + def read_next_change(self) -> CultNetDatabaseChange: + return self._change + + +class StamplessClientReplayTests(unittest.TestCase): + def test_a_subscribed_put_with_no_stamp_is_refused_by_read_next_change_and_nothing_is_stored(self) -> None: + for label, stamp in REFUSED: + with self.subTest(label), tempfile.TemporaryDirectory() as directory: + node = create_node(Path(directory) / "node.cc", runtime_id="stampless-subscriber") + node.database.register_document(NOTE) + subscription = CultMeshDocumentSubscription(client=None, database=node.database, document=NOTE, key="note:bad") # type: ignore[arg-type] + + def deliver(record: dict[str, Any]) -> Any: + message = { + "schemaVersion": "cultnet.database_change_raw.v0", "messageId": "change", "changeKind": "added", + "schemaId": SCHEMA_ID, "recordKey": record["recordKey"], "document": record, + } + subscription._subscription = _StubSubscription(CultNetDatabaseChange.from_wire(message)) # type: ignore[assignment] + return subscription.read_next_change() + + with self.assertRaises(ValueError): + deliver(record_of("note:bad", "bad", stamp)) + self.assertIsNone(node.cache.get(NOTE, "note:bad")) + self.assertEqual(deliver(record_of("note:bad", "stamped")).value["text"], "stamped") + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultmesh-rs/Cargo.lock b/packages/cultmesh-rs/Cargo.lock index de7c506e..c45da112 100644 --- a/packages/cultmesh-rs/Cargo.lock +++ b/packages/cultmesh-rs/Cargo.lock @@ -213,6 +213,7 @@ dependencies = [ "serde_bytes", "serde_json", "serde_stacker", + "sha2", "stacker", "uuid", "windows-sys 0.59.0", diff --git a/packages/cultmesh-rs/src/lib.rs b/packages/cultmesh-rs/src/lib.rs index 33cb4858..e3132ece 100644 --- a/packages/cultmesh-rs/src/lib.rs +++ b/packages/cultmesh-rs/src/lib.rs @@ -732,21 +732,37 @@ impl CultMeshNode { let cultnet_rs::CultNetMessage::SnapshotResponseRaw { documents, .. } = response else { anyhow::bail!("expected cultnet.snapshot_response_raw.v0"); }; - let mut applied = 0usize; + self.apply_raw_snapshot_documents(documents) + } + + /// Applies the snapshot documents as one cache commit, as C# does: a refusal (a key this node holds read-only because + /// of unread slots from a newer peer or a payload this build cannot read, or any other the cache judges) applies + /// nothing. A schema id the registry does not know is skipped, except one with a `.vN` suffix, which is taken as + /// registered and refuses the commit when its type is not (C# throws on an unknown schema too). The commit lands in + /// one backing store, so a snapshot spanning stores is refused whole; C# `CultCache.Commit` refuses a cross-store batch too + /// ("The batch spans A and B; a commit lands in one home store"), so the two agree. + fn apply_raw_snapshot_documents( + &mut self, + documents: Vec, + ) -> Result { + let mut staged = Vec::new(); for document in documents { let Some(document_type) = registered_document_type_for_schema(&self.documents, &document.schema_id) else { continue; }; - self.cache.put_raw_envelope(CultCacheEnvelope { + staged.push(CultCacheEnvelope { key: document.record_key, r#type: document_type, payload: document.payload, stored_at: document.stored_at, schema_id: Some(document.schema_id), - })?; - applied += 1; + }); + } + let applied = staged.len(); + if applied > 0 { + self.cache.put_prepared_batch(staged)?; } Ok(applied) } @@ -1195,10 +1211,135 @@ mod tests { body: String, } + #[derive(Clone, Debug, PartialEq, Eq, cultcache_rs::DatabaseEntry)] + #[cultcache(type = "cultmesh.test.memo", schema = "CultMeshTestMemo")] + struct Memo { + #[cultcache(key = 0)] + body: String, + } + cultmesh_documents!(TestDocuments { Note => "cultmesh.test.note.v0", }); + cultmesh_documents!(TwoTypeDocuments { + Note => "cultmesh.test.note.v0", + Memo => "cultmesh.test.memo.v0", + }); + + fn raw_note(key: &str, payload: Vec) -> cultnet_rs::CultNetRawDocumentRecord { + cultnet_rs::CultNetRawDocumentRecord { + schema_id: "cultmesh.test.note.v0".to_string(), + record_key: key.to_string(), + stored_at: "2026-01-01T00:00:00Z".to_string(), + payload_encoding: cultnet_rs::CultNetRawPayloadEncoding::Messagepack, + payload, + source_runtime_id: None, + source_agent_id: None, + source_role: None, + tags: None, + } + } + + #[test] + fn a_snapshot_document_for_a_read_only_key_refuses_the_whole_pull_as_csharp_does() -> Result<()> { + let temp = tempfile::tempdir()?; + let mut node = CultMesh::create_node(temp.path().join("m.cc"), TestDocuments, Default::default())?; + // A newer peer's note: one slot past the declared one. + let newer = rmp_serde::to_vec(&("newer body", "extra slot"))?; + let plain = |body: &str| rmp_serde::to_vec(&(body,)); + assert_eq!(node.apply_raw_snapshot_documents(vec![raw_note("held", newer)])?, 1); + assert_eq!(node.get_required::("held")?.body, "newer body"); + + let error = node + .apply_raw_snapshot_documents(vec![ + raw_note("before", plain("b")?), + raw_note("held", plain("overwrite")?), + raw_note("after", plain("a")?), + ]) + .unwrap_err(); + let conflict = error.downcast_ref::().expect("typed refusal"); + assert_eq!(conflict.record_key, "held"); + assert!(conflict.unread_slots.is_some()); + assert!(!error.to_string().contains("extra slot")); + assert!(node.get::("before")?.is_none()); + assert_eq!(node.get_required::("held")?.body, "newer body"); + assert!(node.get::("after")?.is_none()); + Ok(()) + } + + #[test] + fn a_snapshot_spanning_two_backing_stores_is_refused_whole_and_applies_nothing() -> Result<()> { + let temp = tempfile::tempdir()?; + let mut node = CultMesh::create_node(temp.path().join("m.cc"), TwoTypeDocuments, Default::default())?; + // A node has one store through the public API; route Memo to a second one by hand (C# refuses the same commit). + node.cache.add_backing_store( + SingleFileMessagePackBackingStore::new(temp.path().join("memo.cc")), + [Memo::TYPE], + )?; + node.cache.pull_all_backing_stores()?; + let plain = |body: &str| rmp_serde::to_vec(&(body,)); + let memo = |key: &str, payload| { + let mut record = raw_note(key, payload); + record.schema_id = "cultmesh.test.memo.v0".to_string(); + record + }; + let error = node + .apply_raw_snapshot_documents(vec![ + raw_note("n", plain("note")?), + memo("m", plain("memo")?), + ]) + .unwrap_err(); + assert!(error.to_string().contains("cannot span backing stores"), "{error}"); + assert!(node.get::("n")?.is_none()); + assert!(node.get::("m")?.is_none()); + assert!(!temp.path().join("m.cc").exists() && !temp.path().join("memo.cc").exists()); + + // Each store alone still takes its own snapshot. + assert_eq!(node.apply_raw_snapshot_documents(vec![memo("m", plain("memo")?)])?, 1); + assert_eq!(node.get_required::("m")?.body, "memo"); + Ok(()) + } + + /// The node's store is the second owner of the stamp rule: a snapshot document whose `storedAt` is no stamp refuses + /// the whole pull, nothing is minted or stored, and the node stays writable. + #[test] + fn a_snapshot_document_whose_stored_at_is_no_stamp_refuses_the_pull_and_stores_nothing() -> Result<()> { + let temp = tempfile::tempdir()?; + let path = temp.path().join("m.cc"); + let mut node = CultMesh::create_node(&path, TestDocuments, Default::default())?; + let plain = |body: &str| rmp_serde::to_vec(&(body,)); + for bad in ["", "yesterday", "2026-01-01T00:00:00", "2026-01-01t00:00:00z", "2026-01-01T00:00:60Z"] { + let mut stampless = raw_note("bad", plain("bad")?); + stampless.stored_at = bad.to_string(); + let error = node + .apply_raw_snapshot_documents(vec![raw_note("good", plain("good")?), stampless]) + .unwrap_err(); + assert!(format!("{error:#}").contains("stored_at"), "{bad:?}: {error:#}"); + assert!(node.get::("good")?.is_none() && node.get::("bad")?.is_none(), "{bad:?}"); + } + assert_eq!(node.apply_raw_snapshot_documents(vec![raw_note("after", plain("after")?)])?, 1); + let reopened = CultMesh::create_node(&path, TestDocuments, Default::default())?; + assert_eq!(reopened.get_required::("after")?.body, "after"); + assert!(reopened.get::("good")?.is_none()); + Ok(()) + } + + #[test] + fn a_snapshot_applied_to_a_stale_node_keeps_the_records_another_writer_added() -> Result<()> { + let temp = tempfile::tempdir()?; + let path = temp.path().join("m.cc"); + let mut node = CultMesh::create_node(&path, TestDocuments, Default::default())?; + let mut other = CultMesh::create_node(&path, TestDocuments, Default::default())?; + other.put("added", &Note { body: "other writer".to_string() })?; + let plain = |body: &str| rmp_serde::to_vec(&(body,)); + assert_eq!(node.apply_raw_snapshot_documents(vec![raw_note("snap", plain("s")?)])?, 1); + let reopened = CultMesh::create_node(&path, TestDocuments, Default::default())?; + assert_eq!(reopened.get_required::("added")?.body, "other writer"); + assert_eq!(reopened.get_required::("snap")?.body, "s"); + Ok(()) + } + #[test] fn node_round_trips_registered_documents() -> Result<()> { let temp = tempfile::tempdir()?; diff --git a/packages/cultmesh-rs/src/rudp_document_server.rs b/packages/cultmesh-rs/src/rudp_document_server.rs index 9aae5be9..1c440a14 100644 --- a/packages/cultmesh-rs/src/rudp_document_server.rs +++ b/packages/cultmesh-rs/src/rudp_document_server.rs @@ -69,7 +69,8 @@ pub struct CultMeshRudpSnapshotQuery { pub trait CultMeshRudpRawDocumentSink { /// Whether this put may be written, decided without writing it. A receipt-capable session asks it only /// the first time it sees a message id; a resend is answered from what the session remembers - /// (write-receipt.md, rule 5). The default admits everything and leaves the decision to + /// (write-receipt.md, rule 5). A legacy session asks it only for a put whose `storedAt` is no stamp, so the + /// authority's refusal comes before the stamp's. The default admits everything and leaves the decision to /// `accept_raw_document`. fn authorize_raw_document(&mut self, _receipt: &CultMeshRudpRawDocumentReceipt) -> Result<()> { Ok(()) @@ -293,6 +294,9 @@ pub enum CultMeshRudpRejectionReason { DocumentMissing, /// A receipt-capable session's put or delete has a null or missing `schemaId`. SchemaIdMissing, + /// A put's `storedAt` is not a stamp (`cultcache_rs::is_stored_at_stamp`). It is judged after the sink's + /// authorization and before the document is measured or the sink commits it: nothing is minted or stored. + StoredAtNotStamp, } impl std::fmt::Display for CultMeshRudpRejectionReason { @@ -307,6 +311,7 @@ impl std::fmt::Display for CultMeshRudpRejectionReason { Self::ChannelNotReliable => f.write_str("a receipt session's write arrived on an unreliable channel"), Self::DocumentMissing => f.write_str("the write has no document"), Self::SchemaIdMissing => f.write_str("the write has no schemaId"), + Self::StoredAtNotStamp => f.write_str("the document's storedAt is not a stamp"), Self::DocumentUnservable { response_bytes, max_snapshot_response_bytes, @@ -347,7 +352,7 @@ impl std::fmt::Display for CultMeshRudpRejectionReason { /// The texts a refused peer is sent, one per rejection variant. They are fixed: /// a sink's or source's error text can quote what a peer sent, so it stays with /// the caller that receives the rejection and never reaches the wire. -pub(crate) const REFUSAL_TEXTS: [&str; 17] = [ +pub(crate) const REFUSAL_TEXTS: [&str; 18] = [ "the catalog refused the document", "the document can never be served", "the snapshot source failed", @@ -365,6 +370,7 @@ pub(crate) const REFUSAL_TEXTS: [&str; 17] = [ "receipt sessions require the reliable channel", "document is required", "schemaId is required", + "document must carry a storedAt stamp", ]; impl CultMeshRudpRejectionReason { @@ -393,7 +399,8 @@ impl CultMeshRudpRejectionReason { 13 => Self::MessageIdTooLong, 14 => Self::ChannelNotReliable, 15 => Self::DocumentMissing, - _ => Self::SchemaIdMissing, + 16 => Self::SchemaIdMissing, + _ => Self::StoredAtNotStamp, } } @@ -416,6 +423,7 @@ impl CultMeshRudpRejectionReason { Self::ChannelNotReliable => 14, Self::DocumentMissing => 15, Self::SchemaIdMissing => 16, + Self::StoredAtNotStamp => 17, } } } @@ -498,6 +506,7 @@ impl CultMeshRudpApplicationRejection { (CultMeshRudpRejectionReason::MessageIdTooLong, _) => message_invalid_message("messageId", text), (CultMeshRudpRejectionReason::DocumentMissing, _) => message_invalid_message("document", text), (CultMeshRudpRejectionReason::SchemaIdMissing, _) => message_invalid_message("schemaId", text), + (CultMeshRudpRejectionReason::StoredAtNotStamp, _) => message_invalid_message("document", text), (CultMeshRudpRejectionReason::ChannelNotReliable, _) => message_invalid_message("channel", text), (CultMeshRudpRejectionReason::WriteLimitReached, _) => CultNetMessage::Error { error: format!("limit_reached: {text}"), @@ -1601,11 +1610,19 @@ where receipt: CultMeshRudpRawDocumentReceipt, authorize: bool, ) -> std::result::Result<(), CultMeshRudpRejectionReason> { - if authorize { + let stamped = cultcache_rs::is_stored_at_stamp(&receipt.document.stored_at); + // The authority speaks first. A legacy session's sink decides at `accept_raw_document`, so it is asked here + // only when the stamp is bad: an ungranted peer is refused `write_not_authorized`, not told its stamp is wrong. + if authorize || !stamped { if let Err(error) = self.sink.authorize_raw_document(&receipt) { return Err(CultMeshRudpRejectionReason::SinkRefused(format!("{error:#}"))); } } + // A peer's put is stored under the stamp it carries, so one that is no stamp is refused before anything is + // measured, minted or handed to the sink. + if !stamped { + return Err(CultMeshRudpRejectionReason::StoredAtNotStamp); + } let document = &receipt.document; // Admit only what some snapshot request can return: the smallest // response that could carry this document is the record the snapshot diff --git a/packages/cultmesh-rs/tests/rudp_document_server.rs b/packages/cultmesh-rs/tests/rudp_document_server.rs index 60596ee5..3f4d391a 100644 --- a/packages/cultmesh-rs/tests/rudp_document_server.rs +++ b/packages/cultmesh-rs/tests/rudp_document_server.rs @@ -3546,6 +3546,10 @@ fn blank_means_exactly_the_unicode_white_space_code_points_in_every_field() -> R for (index, (hex, blank)) in space.iter().map(|h| (h, true)).chain(not_space.iter().map(|h| (h, false))).enumerate() { let hex = hex.as_str().unwrap(); for (field_index, field) in fields.iter().enumerate() { + // A storedAt must be a stamp: a non-blank one that is no stamp is not a name (write-receipt.md, Required fields). + if !blank && *field == "storedAt" { + continue; + } // A session that is sent refusals is allowed to end, so each code point and field gets its own. let sink = Sink::default(); let mut server = server(Default::default(), Clock::new(70_000), sink.clone(), Source::default())?; @@ -3626,3 +3630,177 @@ fn an_unreadable_frame_on_a_receipt_connection_is_sent_exactly_the_fixed_text() ); Ok(()) } + +// ---- A peer's `storedAt` must be a stamp (variants:ruling:raw-put-without-stamp-refused) ---- +// +// `tests/vectors/stored-at-stamp/stamps.msgpack`: `[text, parses]` rows. Every row that is no stamp is refused and +// nothing reaches the sink. A blank text names nothing, so it is refused as a frame that does not read (field +// "message", write-receipt.md, Required fields); any other text that is no stamp is refused field "document". + +fn stamp_rows() -> Vec<(String, bool)> { + let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../tests/vectors/stored-at-stamp/stamps.msgpack"); + rmp_serde::from_slice(&std::fs::read(path).unwrap()).unwrap() +} + +fn is_blank(text: &str) -> bool { + text.chars().all(char::is_whitespace) +} + +/// A put of `key` whose `storedAt` is `stamp` as it would arrive: a blank one is swapped into the bytes, since the +/// encoder refuses it. +fn put_stamped_bytes(id: &str, key: &str, payload: Vec, stamp: &str) -> Result> { + let mut record = document(key, payload); + record.stored_at = if is_blank(stamp) { "zz-stamp-zz".into() } else { stamp.into() }; + let bytes = encode_cultnet_message_to_vec( + &CultNetMessage::DocumentPutRaw { message_id: id.into(), document: record }, + CultNetWireContract::CultNetSchemaV0, + )?; + Ok(if is_blank(stamp) { swap_fixstr(&bytes, "zz-stamp-zz", stamp) } else { bytes }) +} + +#[test] +fn a_receipt_session_put_that_is_no_stamp_is_refused_for_every_vector_row_and_the_store_stays_writable() -> Result<()> { + let rows = stamp_rows(); + assert!(rows.len() > 30); + let sink = Sink::default(); + let mut server = server(Default::default(), Clock::new(70_000), sink.clone(), Source::default())?; + let mut peer = client(server.local_addr()?, 700)?; + connect(&mut server, &mut [&mut peer])?; + send_reliable(&mut peer, &receipt_hello())?; + assert_eq!(frames_after(&mut server, &mut peer, 1)?, vec![server_hello()]); + let mut expected = Vec::new(); + let mut answers = Vec::new(); + for (index, (text, parses)) in rows.iter().enumerate() { + let id = format!("r{index}"); + peer.send_reliable("schema", put_stamped_bytes(&id, &format!("note:{index}"), vec![1, 2, 3], text)?)?; + expected.push((id, text.clone(), *parses)); + // One write in flight at a time: the answer arrives before the next row is sent. + answers.extend(frames_after(&mut server, &mut peer, 1)?); + } + peer.send_reliable("schema", valid_put("after"))?; + answers.extend(frames_after(&mut server, &mut peer, 1)?); + for (answer, (id, text, parses)) in answers.iter().zip(&expected) { + if *parses { + assert_eq!(*answer, admitted(id, &id.replace('r', "note:")), "{text:?}"); + continue; + } + assert_eq!(refusal_of(answer), (Some(CultNetErrorCode::MessageInvalid), Some(id.clone())), "{text:?}"); + let field = if is_blank(text) { "message" } else { "document" }; + assert_eq!(error_text_and_field(answer).1, Some(field.into()), "{text:?}"); + } + // Every stamp row was stored exactly as sent; no refused row was stored; the put after them all was. + let state = sink.0.lock().unwrap(); + let stored: Vec<&str> = state.receipts.iter().map(|receipt| receipt.document.stored_at.as_str()).collect(); + let stamps: Vec<&str> = rows.iter().filter(|(_, parses)| *parses).map(|(text, _)| text.as_str()).collect(); + assert_eq!(&stored[..stamps.len()], &stamps[..]); + assert_eq!(stored.len(), stamps.len() + 1, "only the stamped puts and the final put were stored"); + assert_eq!(answers.last(), Some(&admitted("put-after", "after"))); + Ok(()) +} + +#[test] +fn a_put_with_no_storedat_key_is_refused_and_a_resend_replays_the_refusal() -> Result<()> { + let sink = Sink::default(); + let mut server = server(Default::default(), Clock::new(70_000), sink.clone(), Source::default())?; + let mut peer = client(server.local_addr()?, 701)?; + connect(&mut server, &mut [&mut peer])?; + send_reliable(&mut peer, &receipt_hello())?; + let missing = swap_fixstr(&valid_put("a"), "storedAt", "storedAx"); + peer.send_reliable("schema", missing)?; + // The same garbage stamp twice under one id: the refusal is remembered (by index) and replays as it was. + peer.send_reliable("schema", put_stamped_bytes("g1", "note:g", vec![1], "yesterday")?)?; + peer.send_reliable("schema", put_stamped_bytes("g1", "note:g", vec![1], "yesterday")?)?; + let answers = answers_after(&mut server, &mut peer, 3)?; + assert_eq!(error_text_and_field(&answers[0]).1, Some("message".into())); + assert_eq!(refusal_of(&answers[1]), (Some(CultNetErrorCode::MessageInvalid), Some("g1".into()))); + assert_eq!(error_text_and_field(&answers[1]).1, Some("document".into())); + assert_eq!(answers[2], answers[1]); + assert!(sink.0.lock().unwrap().receipts.is_empty()); + Ok(()) +} + +#[test] +fn a_legacy_session_put_that_is_no_stamp_is_refused_for_every_vector_row_and_nothing_is_stored() -> Result<()> { + let sink = Sink::default(); + let mut server = server(Default::default(), Clock::new(70_000), sink.clone(), Source::default())?; + let mut connection_id = 800; + for (text, parses) in stamp_rows() { + if parses { + continue; + } + connection_id += 1; + let mut peer = client(server.local_addr()?, connection_id)?; + connect(&mut server, &mut [&mut peer])?; + let receipt = peer.send_reliable("schema", put_stamped_bytes("bad", "note:bad", vec![1, 2, 3], &text)?)?; + let reason = poll_until_rejected_or_stored(&mut server, &mut peer, &sink, 1)?; + if is_blank(&text) { + assert!(matches!(reason, Some(CultMeshRudpRejectionReason::MessageUnreadable(_))), "{text:?}: {reason:?}"); + } else { + assert_eq!(reason, Some(CultMeshRudpRejectionReason::StoredAtNotStamp), "{text:?}"); + let (error, code, details) = typed_refusal_seen_by(&mut peer, &receipt, Some(&mut server))?; + assert_eq!(code, Some(CultNetErrorCode::MessageInvalid), "{text:?}: {error}"); + assert_eq!(details.and_then(|d| d.field), Some("document".into()), "{text:?}"); + } + assert!(sink.0.lock().unwrap().receipts.is_empty(), "{text:?} reached the sink"); + } + // The server and its sink are still writable. + connection_id += 1; + let mut peer = client(server.local_addr()?, connection_id)?; + connect(&mut server, &mut [&mut peer])?; + stored_after_poll(&mut server, &mut peer, &sink, 1)?; + Ok(()) +} + +/// The size gate measures the document as a snapshot would serve it: a put over the bound that is also no stamp is +/// refused for its stamp first, and nothing is measured for one that is. +#[test] +fn the_stamp_is_judged_before_the_size_limit() -> Result<()> { + let over = document("big", vec![7; 101]); + let options = CultMeshRudpDocumentServerOptions { + max_snapshot_response_bytes: served_alone_bytes(&over)? - 1, + ..Default::default() + }; + let sink = Sink::default(); + let mut server = server(options, Clock::new(70_000), sink.clone(), Source::default())?; + let mut peer = client(server.local_addr()?, 900)?; + connect(&mut server, &mut [&mut peer])?; + peer.send_reliable("schema", put_stamped_bytes("o1", "big", vec![7; 101], "yesterday")?)?; + assert_eq!( + poll_until_rejected_or_stored(&mut server, &mut peer, &sink, 1)?, + Some(CultMeshRudpRejectionReason::StoredAtNotStamp) + ); + let mut stamped = client(server.local_addr()?, 901)?; + connect(&mut server, &mut [&mut stamped])?; + stamped.send_reliable("schema", put_stamped_bytes("o2", "big", vec![7; 101], "2026-06-25T00:00:00Z")?)?; + assert!(matches!( + poll_until_rejected_or_stored(&mut server, &mut stamped, &sink, 1)?, + Some(CultMeshRudpRejectionReason::DocumentUnservable { .. }) + )); + assert!(sink.0.lock().unwrap().receipts.is_empty()); + Ok(()) +} + +/// An ungranted peer is refused `write_not_authorized` before its stamp is judged, on a receipt session and on a +/// legacy one, and the authority is asked about the write. +#[test] +fn an_ungranted_peer_is_refused_for_authority_before_its_stamp_is_judged() -> Result<()> { + let sink = Sink::denying(&["secret"]); + let mut server = server(Default::default(), Clock::new(70_000), sink.clone(), Source::default())?; + let mut receipt_peer = client(server.local_addr()?, 950)?; + connect(&mut server, &mut [&mut receipt_peer])?; + send_reliable(&mut receipt_peer, &receipt_hello())?; + receipt_peer.send_reliable("schema", put_stamped_bytes("a1", "secret", vec![1], "yesterday")?)?; + let answers = answers_after(&mut server, &mut receipt_peer, 1)?; + assert_eq!(refusal_of(&answers[0]), (Some(CultNetErrorCode::WriteNotAuthorized), Some("a1".into()))); + assert_eq!(sink.0.lock().unwrap().authorized, vec!["a1".to_string()]); + + let mut legacy_peer = client(server.local_addr()?, 951)?; + connect(&mut server, &mut [&mut legacy_peer])?; + legacy_peer.send_reliable("schema", put_stamped_bytes("a2", "secret", vec![1], "yesterday")?)?; + assert!(matches!( + poll_until_rejected_or_stored(&mut server, &mut legacy_peer, &sink, 1)?, + Some(CultMeshRudpRejectionReason::SinkRefused(_)) + )); + assert!(sink.0.lock().unwrap().receipts.is_empty()); + Ok(()) +} diff --git a/packages/cultmesh-ts/CHANGELOG.md b/packages/cultmesh-ts/CHANGELOG.md index 5df85ad3..574a06a6 100644 --- a/packages/cultmesh-ts/CHANGELOG.md +++ b/packages/cultmesh-ts/CHANGELOG.md @@ -11,6 +11,7 @@ All notable changes to `@gamecult/cultmesh-ts` are recorded here, following ### Breaking +- `createRudpDocumentServer` refuses a peer put whose `storedAt` is missing, empty or not a stamp (`isStoredAtStamp`, the shape pinned in `tests/vectors/stored-at-stamp/`) with `message_invalid` naming `document`, after the write authority has been asked and before the put is sized; `onDocumentPutRaw` is not called, so nothing is stored and no stamp is minted (variants:ruling:raw-put-without-stamp-refused). `CultMeshRudpDocumentPut.storedAt` is the stamp the peer sent, never a server clock; a document synced from a peer with a stampless record is refused by the registry. - `publishRudpDocumentOnce` no longer treats a window with no refusal as published: a put nothing settled in its result window (500 ms, or 5000 ms against a receipt-capable server) rejects `CultMeshRudpWriteUnconfirmedError`, and a legacy server's echo of the put's id is not admission. It now resolves `"admitted"` (only on a `write_admitted`) or `"sent"`. **Migration:** a caller whose server cannot answer a write yet (for example Odin, until it is receipt-capable) passes `sendOnly: true` per call and treats `"sent"` as "sent", never as admitted. Receipts stay opt-in (`writeReceipts: true`); neither mode is the default. - The package is renamed `cultmesh-ts` to `@gamecult/cultmesh-ts` (the repo already scopes `@gamecult/cultcache-ts`), so the unscoped name cannot be claimed by someone else. Import from `@gamecult/cultmesh-ts` and its subpaths; the API is unchanged. Depends on `@gamecult/cultnet-ts`. - `asSchemaAlias` is removed from `CultMeshDocumentHandle`, `CultMeshCollectionHandle` and their bound forms. Read the handle by the schema you mean. @@ -29,6 +30,8 @@ All notable changes to `@gamecult/cultmesh-ts` are recorded here, following - Needs the cultcache-ts release that ships with this change (its manifest range moves with that release): it imports `schemaIdentityOf` and `SchemaConflictError`, which 0.14.x does not export. +- Needs `@gamecult/cultnet-ts` `^0.4.0` (was `^0.3.0`): this package imports names that `cultnet-ts` 0.3.0 does not export (`cultNetMessageInvalidError`, `cultNetTextError`, `cultNetWriteNotAuthorizedError`, `WriteTracker`, `CULTNET_WRITE_RECEIPT_MESSAGE_VERSIONS`, `helloAdvertisesWriteReceipts`, `resultWindowMs`). The declared-range check (`npm run test:pack:ts`) found the old range admitting a release that lacks them. + ## [0.2.0] ### Added diff --git a/packages/cultmesh-ts/package.json b/packages/cultmesh-ts/package.json index 39e0abba..af8e3584 100644 --- a/packages/cultmesh-ts/package.json +++ b/packages/cultmesh-ts/package.json @@ -49,8 +49,8 @@ "node": ">=20" }, "dependencies": { - "@gamecult/cultcache-ts": "^0.15.0", - "@gamecult/cultnet-ts": "^0.3.0", + "@gamecult/cultcache-ts": "^0.16.0", + "@gamecult/cultnet-ts": "^0.4.0", "koffi": "3.3.0" }, "devDependencies": { diff --git a/packages/cultmesh-ts/src/index.ts b/packages/cultmesh-ts/src/index.ts index 2f734cfd..50583888 100644 --- a/packages/cultmesh-ts/src/index.ts +++ b/packages/cultmesh-ts/src/index.ts @@ -5,6 +5,7 @@ import { CultCache, SchemaConflictError, SingleFileMessagePackBackingStore, + isStoredAtStamp, schemaIdentityOf, type AnyCultCacheDocumentDefinition, type CacheBackingStore, @@ -3933,7 +3934,7 @@ export class CultMeshNode { } public async flush(soft = false): Promise { - await this.store.pushAll?.(this.cache.snapshot(), { soft }); + await this.store.pushAll?.(this.cache.storedSnapshot(), { soft }); } } @@ -5813,6 +5814,12 @@ export class CultMesh { )); return; } + // A peer's put is stored under the stamp it carries: one that is no stamp is refused, after the authority + // has spoken and before the put is sized, so the size-limited path never measures it. Nothing is minted. + if (!isStoredAtStamp(message.document.storedAt)) { + sendSchemaMessage(record, cultNetMessageInvalidError("document", "Raw put documents must carry a storedAt stamp.")); + return; + } } if (options.onDocumentPutRaw) { const unservable = unservablePut(message); @@ -6635,7 +6642,7 @@ function normalizeRudpDocumentPut( return { schemaId: document.schemaId, recordKey: document.recordKey, - storedAt: document.storedAt ?? new Date().toISOString(), + storedAt: document.storedAt, sourceRuntimeId: document.sourceRuntimeId ?? null, sourceAgentId: document.sourceAgentId ?? null, sourceRole: document.sourceRole ?? null, diff --git a/packages/cultmesh-ts/test/cultmesh.test.ts b/packages/cultmesh-ts/test/cultmesh.test.ts index 10de21b9..41230aa9 100644 --- a/packages/cultmesh-ts/test/cultmesh.test.ts +++ b/packages/cultmesh-ts/test/cultmesh.test.ts @@ -6,7 +6,7 @@ import test from "node:test"; import assert from "node:assert/strict"; import { decodeWireValue, encodeWireValue } from "@gamecult/cultnet-ts/wire"; import { z } from "zod"; -import { SchemaConflictError, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "@gamecult/cultcache-ts"; +import { SchemaConflictError, SingleFileMessagePackBackingStore, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "@gamecult/cultcache-ts"; import { CultNetDocumentRegistry, CultNetPeer, @@ -109,6 +109,34 @@ test("CultMesh TS opens a durable local node and registers document bindings", a assert.ok(reopened.documents.getBySchemaId("cultmesh.note.v0")); }); +test("CultMesh TS flush from a node that never pulled leaves the records the store holds", async () => { + const filePath = join(await mkdtemp(join(tmpdir(), "cultmesh-ts-flush-")), "node.ccmp"); + const first = await CultMesh.startNode(filePath, { documents: [noteDocument] }); + await first.put(noteDocument, "note:held", { noteId: "note:held", body: "already in the store" }); + await first.flush(); + + const unpulled = await CultMesh.startNode(filePath, { documents: [noteDocument], pullOnStart: false }); + await unpulled.flush(); + + const reopened = await CultMesh.startNode(filePath, { documents: [noteDocument] }); + assert.equal(reopened.getRequired(noteDocument, "note:held").body, "already in the store"); +}); + +test("CultMesh TS flush of a node that loaded a legacy-key global leaves the store reopenable", async () => { + const settings = defineDocumentType({ type: "cultmesh.settings", schema: z.object({ theme: z.string() }), global: true }); + const filePath = join(await mkdtemp(join(tmpdir(), "cultmesh-ts-legacy-flush-")), "node.ccmp"); + const seed = await CultMesh.startNode(join(await mkdtemp(join(tmpdir(), "cultmesh-ts-seed-")), "seed.ccmp"), { documents: [settings] }); + await seed.cache.putGlobal(settings, { theme: "iron" }); + const legacy = { ...seed.cache.getRequiredEnvelope(settings, "__global__"), key: "legacy" }; + await new SingleFileMessagePackBackingStore(filePath).push(legacy); + + const node = await CultMesh.startNode(filePath, { documents: [settings] }); + await node.flush(); + + const reopened = await CultMesh.startNode(filePath, { documents: [settings] }); + assert.deepEqual(reopened.cache.getRequiredGlobal(settings), { theme: "iron" }); +}); + test("CultMesh TS document handles hide local cache plumbing behind typed reactive reads", async () => { const filePath = join(await mkdtemp(join(tmpdir(), "cultmesh-ts-doc-")), "node.ccmp"); const node = await CultMesh.startNode(filePath, { @@ -3395,11 +3423,12 @@ const servedBoundRegistry = () => /** * A put's own storedAt, kept by the served record and replaced in the receipt by the server's - * 24-character clock. `SERVED_STORED_AT` is long, so an echoed receipt is smaller than the served - * reply; `RECEIPT_STORED_AT` is short, so an echoed receipt is the larger reply, which shows that + * 24-character clock. `SERVED_STORED_AT` is as long as a stamp can be (a stamp is judged before a put is + * sized, so padding one is refused), so an echoed receipt is smaller than the served reply; + * `RECEIPT_STORED_AT` is short, so an echoed receipt is the larger reply, which shows that * admission sizes only the served reply. */ -const SERVED_STORED_AT = `2026-09-30T00:00:00.000Z${"~".repeat(200)}`; +const SERVED_STORED_AT = "2026-09-30T00:00:00.123456789+00:00"; // the longest a stamp can be, 35 characters const RECEIPT_STORED_AT = "2026-09-30T00:00:00.000Z"; const SERVED_BOUND_SERVER = "cultmesh-ts-served-bound-server"; diff --git a/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts b/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts index 092d5ffb..14d9e443 100644 --- a/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts +++ b/packages/cultmesh-ts/test/interop/write-receipt-interop.test.ts @@ -122,7 +122,7 @@ const rust: Target = { name: "rust", payload: (text) => encode({ text }), async start() { - if (!prebuilt) await execFileAsync(cargo, ["build", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir }); + if (!prebuilt) await execFileAsync(cargo, ["build", "--locked", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir }); const metadata = spawnSync(cargo, ["metadata", "--format-version", "1", "--no-deps"], { cwd: rustManifestDir, encoding: "utf8" }); if (metadata.status !== 0) throw new Error(`cargo metadata failed: ${metadata.stderr}`); const exe = join(JSON.parse(metadata.stdout).target_directory, "debug", "examples", `receipt_interop_server${process.platform === "win32" ? ".exe" : ""}`); diff --git a/packages/cultmesh-ts/test/provider-session-wire.test.ts b/packages/cultmesh-ts/test/provider-session-wire.test.ts index 4905f161..69314fbe 100644 --- a/packages/cultmesh-ts/test/provider-session-wire.test.ts +++ b/packages/cultmesh-ts/test/provider-session-wire.test.ts @@ -83,3 +83,16 @@ test("a provider command that is not a map, or lacks a required field, is refuse (error: Error) => error.message === "CultMesh provider command must be a MessagePack map.", ); }); + +test("a provider command payload with bytes after its first value is refused by count, without echoing the tail", () => { + const command = { commandId: "c1", commandKind: "k", providerId: "p", serviceInstanceId: "s", payload: { n: 1 } }; + const whole = documentPayloadBytes(encodeDocumentPayload(providerCommandDocument, command)); + const tailed = documentPayloadFromBytes(Uint8Array.from([...whole, ...Buffer.from("canary-tail-8c1f")])); + assert.throws( + () => decodeProviderCommandDocument({ + schemaId: cultMeshProviderSessionSchemas.command, recordKey: "p/s/c1", storedAt: "2026-10-11T00:00:00Z", + payloadEncoding: "messagepack" as const, payload: tailed, + }), + (error: Error) => error.message.includes("16 bytes after its first MessagePack value") && !error.message.includes("canary-tail-8c1f"), + ); +}); diff --git a/packages/cultmesh-ts/test/remote-write-authority.test.ts b/packages/cultmesh-ts/test/remote-write-authority.test.ts index 3c35171e..d019e71b 100644 --- a/packages/cultmesh-ts/test/remote-write-authority.test.ts +++ b/packages/cultmesh-ts/test/remote-write-authority.test.ts @@ -7,6 +7,7 @@ import { join } from "node:path"; import test from "node:test"; import { z } from "zod"; import { documentPayloadFromBytes, defineDocumentType } from "@gamecult/cultcache-ts"; +import { decode } from "@msgpack/msgpack"; import { decodeRudpPacket } from "@gamecult/cultnet-ts"; import { encodeWireValue } from "@gamecult/cultnet-ts/wire"; import { @@ -310,6 +311,64 @@ test("the refusal on the wire is the pinned error-vectors bytes", async () => { }); }); +// A peer's put is stored under the stamp it carries: every row of the shared vector goes through the real handler. A row that +// is no stamp is refused message_invalid naming `document`, on the plain, the size-limited and (RUDP is this server's only +// transport) the capped path; the handler never runs, so nothing is stored or minted, and a stamped put after it still lands. +const STAMP_VECTOR = join(__dirname, "..", "..", "..", "..", "tests", "vectors", "stored-at-stamp", "stamps.msgpack"); + +test("a put whose storedAt is no stamp is refused message_invalid (document), for every row of the stamp vector", async () => { + const rows = decode(readFileSync(STAMP_VECTOR)) as [string, boolean][]; + const cases: [string, unknown, boolean][] = [...rows.map(([text, parses]): [string, unknown, boolean] => [JSON.stringify(text), text, parses]), + ["missing", undefined, false], ["number", 1_700_000_000_000, false]]; + for (const extra of [{}, { maxPayloadBytes: 4096 }]) { + await using(await rig(new CultMeshGrantAllWrites(), extra), async (r) => { + let accepted = 0; + for (const [label, storedAt, parses] of cases) { + const message = put(KEY_A) as unknown as { document: Record }; + if (storedAt === undefined) delete message.document.storedAt; else message.document.storedAt = storedAt; + const seen = r.replies.length; + const handled = r.puts.length; + r.peer.send(message as unknown as CultNetMessage); + await sleep(40); + if (parses) { + accepted += 1; + assert.equal(r.puts.length, handled + 1, `${label}: a stamp reaches the handler`); + assert.equal(r.replies.length, seen, label); + } else { + assert.equal(r.puts.length, handled, `${label}: nothing handled, so nothing minted or stored`); + const reply = r.replies[seen] as { schemaVersion?: string; code?: string; details?: { field?: string } } | undefined; + assert.equal(reply?.schemaVersion, "cultnet.error.v0", label); + assert.equal(reply?.code, "message_invalid", label); + assert.equal(reply?.details?.field, "document", label); + } + } + assert.ok(accepted > 0); + assert.deepEqual(r.errors, []); + // The large put under a garbage stamp is refused for its stamp, before it is measured. + const large = put(KEY_A, {}, SCHEMA_ID, new Uint8Array(20_000)) as unknown as { document: Record }; + large.document.storedAt = "yesterday"; + r.peer.send(large as unknown as CultNetMessage); + await sleep(60); + const last = r.replies[r.replies.length - 1] as { code?: string; details?: { field?: string } }; + assert.equal(last.code, "message_invalid"); + assert.equal(last.details?.field, "document"); + }); + } + // The stamp is judged after the authority: an ungranted peer hears write_not_authorized, aliased or not. + const refusing = new Recording(() => ({ granted: false, reason: "no" })); + await using(await rig(refusing), async (r) => { + const aliased = put(KEY_A, {}, ALIAS) as unknown as { document: Record }; + aliased.document.storedAt = "yesterday"; + r.peer.send(aliased as unknown as CultNetMessage); + r.peer.send(put(KEY_B)); + await r.settle(); + assertRefused(r.replies[0], KEY_A); + assertRefused(r.replies[1], KEY_B); + assert.equal(refusing.requests[0]?.schemaId, SCHEMA_ID, "an aliased put asks under the canonical id"); + assert.equal(r.puts.length, 0); + }); +}); + test("wrong-typed shard fields on a put are judged with null shard fields, then refused as malformed once granted", async () => { const authority = new Recording(grant); await using(await rig(authority), async (r) => { diff --git a/packages/cultnet-py/CHANGELOG.md b/packages/cultnet-py/CHANGELOG.md index b8a3773e..0652ad0a 100644 --- a/packages/cultnet-py/CHANGELOG.md +++ b/packages/cultnet-py/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased (breaking; version 0.x, so the next minor) +- `raw_record_to_envelope`, where a replicated record becomes an envelope, refuses a record whose `storedAt` is missing, empty or not a stamp (`cultcache_py.stores.parse_stored_at`, the shape pinned by `tests/vectors/stored-at-stamp`) with `ValueError`, nothing minted or stored (`variants:ruling:raw-put-without-stamp-refused`). `apply_raw_document_record`, `apply_raw_snapshot` and `apply_shard_log_response` come through it: a snapshot response with such a record is refused whole before any of its records is stored, as one naming an unregistered schema is; a shard log refuses at that entry, the entries before it already applied. It stored the record under `""`, and the next open of the store raised `CultCacheError` `Missing envelope`. `cultmesh-py`'s raw put builds its envelope through the same function, so there is one owner of the check there; its server gate still answers a peer's put `message_invalid`. A client's `read_next_change` lets the `ValueError` out. +- `decode_witness_artifact_bundle_payload` (and `CultNetWitnessArtifactBundle.from_payload`) read the payload through `cultcache_py.read_payload_tree`: bytes after its first MessagePack value raise `ValueError` `cultnet.witness_artifact_bundle payload has N bytes after its first MessagePack value (refused on read).` instead of `msgpack.ExtraData`, and a payload nested past 256 raises `CultCachePayloadDepthError`. Needs `cultcache-py` 0.4.0, the release that exports `read_payload_tree`: the dependency is now `cultcache-py>=0.4.0` (it was `>=0.2.0`, which admits releases whose `import cultnet_py` fails), and `tests/test_declared_dependency_ranges.py` fails when a declared bound admits a tagged release lacking a name the package imports. The same check found `cultmesh-py` importing `cache.index_text`, which `cultcache-py` 0.2.0 lacks; its bound is `cultcache-py>=0.4.0` too (it has no changelog of its own). `resolve_document_for_raw_record` reads the schema hint from a raw record's first value whether or not bytes follow it, so a record routed by its stamp is refused by count when applied (`ValueError`, in a snapshot, a shard log and a direct apply) like one routed by a registered schema id; `apply_shard_log_response` used to skip it silently as an unknown schema. A tail on a payload whose first value names no registered schema is still `KeyError`, which a shard log skips. - `apply_raw_document_record` and `apply_raw_snapshot` raise `SchemaConflictError` when a remote record is of another type than the one the cache holds at its key (`cultcache-py` holds one record per key whatever its type); the cache is unchanged, and the caller decides whether to drop the record or the peer. `cultmesh-py`'s server answers such a put with `message_invalid`; its client's `read_next_change` lets the error out. - `CultNetDatabaseChange` carries `commit_id` and `commit_count` (`commitId`/`commitCount`; None outside a multi-record commit, as C# writes nil): `from_wire` refuses an empty or non-string id and a count that is not an integer of at least 2, `to_wire` writes them when set, and the published `database_change_raw` schema lists both. Python applies no batch. The `simulation_fact` document's `shard`, `frame`, `subject` and `claim_kind` aliases are `roles` (many facts share each); the catalog bytes do not move. - Decodes the write receipt frames of `contracts/cultnet/interop/write-receipt.md`: `parse_message` validates `cultnet.write_admitted.v0` and `cultnet.error.v1` against their closed schemas (unknown keys, empty or non-string ids, an unknown operation or code are refused; `error.v1` requires `messageId`, string or null; `limit_reached` is `error.v1` only). Added `CULTNET_ERROR_V1_CODES`, `CULTNET_WRITE_RECEIPT_MESSAGE_VERSIONS`, `write_content_hash` and `hello_advertises_write_receipts`; `hello` takes `supported_message_versions`. `tests/test_write_receipt.py` replays the vectors' `contentHash`, `wire` and `schemaInvalid` sections. Not implemented here: the client write tracker (`scenarios`) and the server rules (`dedupe`, `blankFields`); this package has no receipt server and no write client. diff --git a/packages/cultnet-py/pyproject.toml b/packages/cultnet-py/pyproject.toml index 41c4e8f6..09db2733 100644 --- a/packages/cultnet-py/pyproject.toml +++ b/packages/cultnet-py/pyproject.toml @@ -16,7 +16,7 @@ classifiers = [ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ] -dependencies = ["cultcache-py>=0.2.0", "msgpack>=1.0"] +dependencies = ["cultcache-py>=0.4.0", "msgpack>=1.0"] [project.optional-dependencies] crypto = ["cryptography>=42"] diff --git a/packages/cultnet-py/src/cultnet_py/messages.py b/packages/cultnet-py/src/cultnet_py/messages.py index d5e26b7a..c9b7d4a6 100644 --- a/packages/cultnet-py/src/cultnet_py/messages.py +++ b/packages/cultnet-py/src/cultnet_py/messages.py @@ -5,6 +5,7 @@ from dataclasses import dataclass from typing import Any import msgpack +from cultcache_py import read_payload_tree from .unicode_messageid_tables import DEFAULT_IGNORABLE, LETTER_OR_NUMBER @@ -544,7 +545,7 @@ def encode_witness_artifact_bundle_payload(bundle: dict[str, Any] | CultNetWitne def decode_witness_artifact_bundle_payload(payload: bytes) -> dict[str, Any]: - slots = msgpack.unpackb(payload, raw=False) + slots = read_payload_tree("cultnet.witness_artifact_bundle", payload) if not isinstance(slots, list) or len(slots) < 8: raise ValueError("CultNet witness artifact bundle payload must be an 8-slot MessagePack array") return { diff --git a/packages/cultnet-py/src/cultnet_py/replication.py b/packages/cultnet-py/src/cultnet_py/replication.py index 531c0012..41d63295 100644 --- a/packages/cultnet-py/src/cultnet_py/replication.py +++ b/packages/cultnet-py/src/cultnet_py/replication.py @@ -5,6 +5,7 @@ from typing import Any, Iterable from cultcache_py import CultCache, CultCacheEnvelope from cultcache_py.documents import DocumentDefinition +from cultcache_py.stores import parse_stored_at from .shard_log import CultNetShardLogResponse from .snapshot import CultNetRawDocumentRecord, CultNetRawSnapshotResponse @@ -50,17 +51,29 @@ def apply_raw_document_record( ) +def require_stored_at_stamp(record: dict[str, Any]) -> str: + """The storedAt a replicated raw record carries; ValueError when it is missing or no stamp. The one place the rule is judged.""" + stored_at = record.get("storedAt") + if parse_stored_at(stored_at) is None: + raise ValueError("A replicated raw record must carry a storedAt stamp.") + return stored_at + + def raw_record_to_envelope( document: DocumentDefinition[Any], schema_id: str, record: dict[str, Any], ) -> CultCacheEnvelope: + """The envelope a replicated raw record is stored as; the one place a record becomes an envelope (a direct apply, a snapshot and a + shard log all come through here). A peer's record is stored under the stamp it carries, so one whose storedAt is missing, empty or + no stamp is refused with ValueError and nothing is minted or stored (variants:ruling:raw-put-without-stamp-refused).""" + stored_at = require_stored_at_stamp(record) return CultCacheEnvelope( key=str(record["recordKey"]), type=document.type, schema_id=schema_id, payload=bytes(record["payload"]), - stored_at=str(record.get("storedAt") or ""), + stored_at=stored_at, catalog_entry=document.catalog_entry(), ) @@ -117,6 +130,12 @@ def apply_shard_log_response( reason = response.get("reason") or "unspecified" raise ValueError(f"Shard log response requires resync: {reason}") documents_by_schema_id = schema_document_map(documents) + # The whole log is refused before anything is stored: every put that would be applied is judged for its stamp first. + for entry in response.get("entries", []): + if isinstance(entry, dict) and entry.get("changeKind") in ("added", "updated") and isinstance(entry.get("put"), dict): + record = entry["put"]["document"] + if can_resolve_document_for_raw_record(documents_by_schema_id, str(record["schemaId"]), record): + require_stored_at_stamp(record) applied: list[CultNetAppliedRecord] = [] for entry in response.get("entries", []): if not isinstance(entry, dict): @@ -171,9 +190,14 @@ def resolve_document_for_raw_record( def _infer_schema_version_from_payload(payload: bytes) -> str | None: + import msgpack # type: ignore try: - import msgpack # type: ignore decoded = msgpack.unpackb(payload, raw=False) + except msgpack.ExtraData as extra: + # The hint is read from the first value; the tail is not this function's to judge. The routed definition refuses + # it by count when the record is applied, as it does for a record routed by a registered schema id. Treating a + # tail as "no hint" made a stamp-routed record look like an unknown schema, which a shard log skips silently. + decoded = extra.unpacked except Exception: return None if isinstance(decoded, list) and decoded and isinstance(decoded[0], str): diff --git a/packages/cultnet-py/tests/test_cultnet.py b/packages/cultnet-py/tests/test_cultnet.py index 48972221..73b41a79 100644 --- a/packages/cultnet-py/tests/test_cultnet.py +++ b/packages/cultnet-py/tests/test_cultnet.py @@ -1971,6 +1971,49 @@ class CultNetTests(unittest.TestCase): apply_raw_document_record(cache, schema_document_map([first, second]), raw(first)) # the same type still replaces self.assertEqual(cache.get_required(first, "shared").name, "remote") + def test_a_raw_record_with_a_tail_is_routed_by_its_stamp_and_refused_by_count_on_every_apply_path(self) -> None: + import msgpack # type: ignore + from cultcache_py import define_document_type + from cultnet_py import resolve_document_for_raw_record + + document = define_document_type("tests.tail_note", schema_id="tests.tail_note", schema_version="tests.tail_note.v1", + payload_encoder=msgpack.packb, payload_decoder=lambda payload: msgpack.unpackb(payload, raw=False)) + known = document.catalog_entry().schema_id + head = msgpack.packb(["tests.tail_note.v1", "x"], use_bin_type=True) + + def raw(schema_id: str, payload: bytes) -> dict: + return {"schemaId": schema_id, "recordKey": "k", "storedAt": "2026-06-13T00:00:00Z", + "payloadEncoding": "messagepack", "payload": payload} + + documents = schema_document_map([document]) + # The stamp is read from the first value whether or not a tail follows, so an unknown schema id whose payload names + # a registered schema routes to it; the tail is then refused by count where the record is applied. + self.assertIs(resolve_document_for_raw_record(documents, "unknown", raw("unknown", head + b"\xc0")), document) + for schema_id in (known, "unknown"): + cache = CultCache() + cache.register_document_type(document) + with self.assertRaises(ValueError, msg=schema_id) as caught: + apply_raw_document_record(cache, documents, raw(schema_id, head + b"\xc0")) + self.assertIn("1 bytes after its first MessagePack value", str(caught.exception)) + self.assertIsNone(cache.get(document, "k")) + snapshot = {"schemaVersion": "cultnet.snapshot_response_raw.v0", "messageId": "s", + "documents": [raw(schema_id, head + b"\xc0")]} + with self.assertRaises(ValueError, msg=schema_id): + apply_raw_snapshot(cache, [document], snapshot) + log = {"schemaVersion": "cultnet.shard_log_response.v0", "messageId": "l", "shardId": "s", "shardEpoch": 1, + "resyncRequired": False, + "entries": [{"sequence": 0, "changeKind": "updated", + "put": {"schemaVersion": "cultnet.document_put_raw.v0", "messageId": "p", + "document": raw(schema_id, head + b"\xc0"), "shardId": "s", "shardEpoch": 1}}]} + with self.assertRaises(ValueError, msg=schema_id) as caught: + apply_shard_log_response(cache, [document], log) + self.assertIn("1 bytes after its first MessagePack value", str(caught.exception)) + self.assertIsNone(cache.get(document, "k")) + # A tail on a payload that names no registered schema is still an unknown schema, which a shard log skips. + stranger = msgpack.packb(["tests.stranger.v1", "x"], use_bin_type=True) + b"\xc0" + with self.assertRaises(KeyError): + resolve_document_for_raw_record(documents, "unknown", raw("unknown", stranger)) + def test_cultnet_replication_helpers_apply_raw_snapshot_and_shard_log(self) -> None: document = define_database_entry_type( "replica.item", @@ -2661,6 +2704,23 @@ class CultNetTests(unittest.TestCase): self.assertEqual(responses[0]["details"]["value"], schema_version) self.assertNotIn("messageId", responses[0]) + def test_witness_artifact_bundle_payload_is_one_value_and_depth_wins_over_a_tail(self) -> None: + import msgpack # type: ignore + from cultcache_py import CultCachePayloadDepthError + + payload = msgpack.packb(["b", "k", "t", {}, [], [], [], {}], use_bin_type=True) + decode_witness_artifact_bundle_payload(payload) + for tail, count in ((b"\xc0", 1), (b"\xc1\xc1", 2)): + with self.assertRaises(ValueError) as caught: + decode_witness_artifact_bundle_payload(payload + tail) + self.assertIs(type(caught.exception), ValueError) + self.assertEqual( + str(caught.exception), + f"cultnet.witness_artifact_bundle payload has {count} bytes after its first MessagePack value (refused on read).", + ) + with self.assertRaises(CultCachePayloadDepthError): + decode_witness_artifact_bundle_payload(b"\x91" * 300 + b"\xc0\xc0") + def test_cultnet_simulation_consensus_dedupes_witnesses_and_requires_quorum(self) -> None: from cultnet_py import CultNetSimulationConsensus diff --git a/packages/cultnet-py/tests/test_declared_dependency_ranges.py b/packages/cultnet-py/tests/test_declared_dependency_ranges.py new file mode 100644 index 00000000..9baf15fc --- /dev/null +++ b/packages/cultnet-py/tests/test_declared_dependency_ranges.py @@ -0,0 +1,94 @@ +"""A sibling package resolves through the workspace's editable install, so a package that starts importing a new name from +cultcache-py passes whatever range it declares. A consumer outside the workspace installs the newest release the range +admits. This is the Python form of scripts/test-declared-dependency-ranges.mjs: for every tagged cultcache-py release +(`cultcache-py-vX.Y.Z`) that the `cultcache-py>=` bound of cultnet-py or cultmesh-py admits, every name those packages +import from cultcache_py must be defined by that release's source. It needs git with the release tags and skips without +them. Only `>=` bounds are read; star imports and names reached by attribute access are not checked.""" + +from __future__ import annotations + +import ast +import re +import subprocess +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[3] +PACKAGES = ROOT / "packages" +CONSUMERS = ("cultnet-py", "cultmesh-py") + + +def _version(text: str) -> tuple[int, ...]: + return tuple(int(part) for part in text.split(".")) + + +def _git(*args: str) -> str | None: + try: + done = subprocess.run(["git", *args], cwd=ROOT, capture_output=True, text=True, check=True) + except (OSError, subprocess.CalledProcessError): + return None + return done.stdout + + +def _lower_bound(package: str) -> tuple[int, ...]: + manifest = (PACKAGES / package / "pyproject.toml").read_text(encoding="utf-8") + match = re.search(r'"cultcache-py>=(\d+\.\d+\.\d+)"', manifest) + assert match, f"{package} declares no cultcache-py>= bound" + return _version(match.group(1)) + + +def _imports(package: str) -> dict[str, set[str]]: + """module -> names imported from it, for every `from cultcache_py[.x] import names` under the package's src.""" + found: dict[str, set[str]] = {} + for source in (PACKAGES / package / "src").rglob("*.py"): + for node in ast.walk(ast.parse(source.read_text(encoding="utf-8-sig"))): + if isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + if node.module == "cultcache_py" or node.module.startswith("cultcache_py."): + found.setdefault(node.module, set()).update(alias.name for alias in node.names if alias.name != "*") + return found + + +def _defined(tag: str, module: str) -> set[str] | None: + base = "packages/cultcache-py/src/" + module.replace(".", "/") + for path in (base + ".py", base + "/__init__.py"): + text = _git("show", f"{tag}:{path}") + if text is None: + continue + names: set[str] = set() + for node in ast.parse(text.lstrip("\ufeff")).body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + names.add(node.name) + elif isinstance(node, (ast.Import, ast.ImportFrom)): + names.update((alias.asname or alias.name).split(".")[0] for alias in node.names) + elif isinstance(node, (ast.Assign, ast.AnnAssign)): + for target in node.targets if isinstance(node, ast.Assign) else [node.target]: + if isinstance(target, ast.Name): + names.add(target.id) + return names + return None + + +class DeclaredDependencyRangeTests(unittest.TestCase): + def test_every_admitted_cultcache_release_defines_what_the_consumers_import(self) -> None: + listed = _git("tag", "--list", "cultcache-py-v*") + tags = sorted((t for t in (listed or "").split() if re.fullmatch(r"cultcache-py-v\d+\.\d+\.\d+", t)), + key=lambda t: _version(t.rsplit("v", 1)[1])) + if not tags: + self.skipTest("git with the cultcache-py release tags is not available") + current = re.search(r'^version = "(\d+\.\d+\.\d+)"', (PACKAGES / "cultcache-py" / "pyproject.toml").read_text(encoding="utf-8"), re.M) + assert current + for package in CONSUMERS: + bound = _lower_bound(package) + self.assertLessEqual(bound, _version(current.group(1)), f"{package}'s bound is above the workspace cultcache-py") + imports = _imports(package) + for tag in tags: + if _version(tag.rsplit("v", 1)[1]) < bound: + continue + for module, names in imports.items(): + defined = _defined(tag, module) + self.assertIsNotNone(defined, f"{package} imports {module}, which {tag} does not have") + self.assertEqual(sorted(names - defined), [], f"{package} imports names {tag} lacks from {module}") + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultnet-py/tests/test_raw_record_stamp.py b/packages/cultnet-py/tests/test_raw_record_stamp.py new file mode 100644 index 00000000..d6e6cba6 --- /dev/null +++ b/packages/cultnet-py/tests/test_raw_record_stamp.py @@ -0,0 +1,57 @@ +"""raw_record_to_envelope is the one place a replicated raw record becomes an envelope; it refuses a record whose storedAt is no stamp. + +The stamp shape is tests/vectors/stored-at-stamp/stamps.msgpack (variants:ruling:raw-put-without-stamp-refused); cultmesh-py tests the +whole replication paths, this tests the function itself. +""" + +from __future__ import annotations + +import unittest +from pathlib import Path +from typing import Any + +import msgpack +from cultcache_py import define_database_entry_type +from cultnet_py.replication import raw_record_to_envelope + +VECTOR = Path(__file__).resolve().parents[3] / "tests/vectors/stored-at-stamp/stamps.msgpack" +ROWS: list[tuple[str, bool]] = [(text, parses) for text, parses in msgpack.unpackb(VECTOR.read_bytes(), raw=False)] +NOTE = define_database_entry_type( + "tests.envelope_note", + [("schema_version", 0), ("text", 1)], + schema_id="sha256:envelope-note", + schema_name="tests.envelope_note", + schema_version="tests.envelope_note.v1", +) + + +def record_with(stamp: Any) -> dict[str, Any]: + record = { + "schemaId": "sha256:envelope-note", + "recordKey": "note:1", + "payload": NOTE.encode_payload({"schema_version": "tests.envelope_note.v1", "text": "t"}), + } + if stamp is not None: + record["storedAt"] = stamp + return record + + +class RawRecordToEnvelopeStampTests(unittest.TestCase): + def test_every_vector_row_is_an_envelope_or_a_refusal_as_the_vector_says(self) -> None: + self.assertGreater(len(ROWS), 30) + for text, parses in ROWS: + with self.subTest(text=text): + if parses: + self.assertEqual(raw_record_to_envelope(NOTE, "sha256:envelope-note", record_with(text)).stored_at, text) + else: + with self.assertRaises(ValueError): + raw_record_to_envelope(NOTE, "sha256:envelope-note", record_with(text)) + + def test_a_missing_or_non_string_stamp_is_refused(self) -> None: + for stamp in (None, 12345, b"2026-11-07T00:00:00Z"): + with self.subTest(stamp=stamp), self.assertRaises(ValueError): + raw_record_to_envelope(NOTE, "sha256:envelope-note", record_with(stamp)) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/cultnet-rs/.gitignore b/packages/cultnet-rs/.gitignore index 96ef6c0b..ea8c4bf7 100644 --- a/packages/cultnet-rs/.gitignore +++ b/packages/cultnet-rs/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/packages/cultnet-rs/CHANGELOG.md b/packages/cultnet-rs/CHANGELOG.md index 384395ec..1a01dbe1 100644 --- a/packages/cultnet-rs/CHANGELOG.md +++ b/packages/cultnet-rs/CHANGELOG.md @@ -15,5 +15,6 @@ only in git history. ### Breaking +- A CultNet frame with bytes after its message is refused, by count (through `cultcache_rs::decode_payload`); the decoder used to read the message and drop the tail. No producer emits one. Behaviour change: the next release is a minor bump (0.x breaking). - `CultNetDocumentRegistry::register` returns `Result<&mut Self>` where it returned `&mut Self`. A binding that claims a document type or a schema id another binding already holds is refused with `cultcache_rs::SchemaConflictError` (naming the new binding's schema id and both document types) and nothing is bound; binding a type to the schema id it already holds still replaces that binding. Callers chain `register(...)?`. - `CultMeshCdnChunkRef` gains a public `id` field, the decimal text of `offset` (the reference's element id, slot 4). A manifest written before ids reads `""`; encoding writes the slot. diff --git a/packages/cultnet-rs/Cargo.lock b/packages/cultnet-rs/Cargo.lock new file mode 100644 index 00000000..e1af9bd6 --- /dev/null +++ b/packages/cultnet-rs/Cargo.lock @@ -0,0 +1,1228 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "aead" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" +dependencies = [ + "crypto-common", + "generic-array", +] + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures", +] + +[[package]] +name = "aes-gcm" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" +dependencies = [ + "aead", + "aes", + "cipher", + "ctr", + "ghash", + "subtle", +] + +[[package]] +name = "android_system_properties" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" +dependencies = [ + "libc", +] + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "ar_archive_writer" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "523e5e187fff36fcb8d1d19ccf0ca4ca43ec3d158316057f66dc6f72716bc5ec" +dependencies = [ + "object", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + +[[package]] +name = "bitflags" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "cc" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ff2a8126fff0f8c839080d619675a585e61720bb12d2baceae45769fd66f155" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "rand_core 0.6.4", + "typenum", +] + +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher", +] + +[[package]] +name = "cultcache-rs" +version = "0.3.1" +dependencies = [ + "anyhow", + "chrono", + "cultcache-rs-derive", + "fs2", + "redb", + "rmp-serde", + "serde", + "serde_bytes", + "serde_json", + "serde_stacker", + "sha2", + "stacker", + "uuid", + "windows-sys 0.59.0", +] + +[[package]] +name = "cultcache-rs-derive" +version = "0.2.0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "cultnet-rs" +version = "0.3.0" +dependencies = [ + "aes-gcm", + "anyhow", + "base64", + "chrono", + "cultcache-rs", + "ed25519-dalek", + "fs2", + "hmac", + "libc", + "pretty_assertions", + "rand", + "rmp-serde", + "rmpv", + "serde", + "serde_bytes", + "serde_json", + "sha2", + "socket2", + "tempfile", + "uuid", + "windows-sys 0.61.2", + "zeroize", +] + +[[package]] +name = "curve25519-dalek" +version = "4.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" +dependencies = [ + "cfg-if", + "cpufeatures", + "curve25519-dalek-derive", + "digest", + "fiat-crypto", + "rustc_version", + "subtle", + "zeroize", +] + +[[package]] +name = "curve25519-dalek-derive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid", + "zeroize", +] + +[[package]] +name = "diff" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8" + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "ed25519" +version = "2.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" +dependencies = [ + "pkcs8", + "signature", +] + +[[package]] +name = "ed25519-dalek" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" +dependencies = [ + "curve25519-dalek", + "ed25519", + "serde", + "sha2", + "subtle", + "zeroize", +] + +[[package]] +name = "errno" +version = "0.3.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "fiat-crypto" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" + +[[package]] +name = "find-msvc-tools" +version = "0.1.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48" + +[[package]] +name = "fs2" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "futures-core" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" + +[[package]] +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", +] + +[[package]] +name = "ghash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" +dependencies = [ + "opaque-debug", + "polyval", +] + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "generic-array", +] + +[[package]] +name = "itoa" +version = "1.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4" + +[[package]] +name = "js-sys" +version = "0.3.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.191" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "log" +version = "0.4.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d15f86d12da44cd8d4ec3c73984537eb7f4cadcb9776a575a0a8a05ca11005c" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "object" +version = "0.39.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" +dependencies = [ + "memchr", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opaque-debug" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkcs8" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" +dependencies = [ + "der", + "spki", +] + +[[package]] +name = "polyval" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" +dependencies = [ + "cfg-if", + "cpufeatures", + "opaque-debug", + "universal-hash", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "pretty_assertions" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d" +dependencies = [ + "diff", + "yansi", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "psm" +version = "0.1.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622" +dependencies = [ + "ar_archive_writer", + "cc", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "redb" +version = "4.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b317ef681e18f6091efbfab532eaf924a135c492d9d0299d0fbfae07e92a39b" +dependencies = [ + "libc", +] + +[[package]] +name = "rmp" +version = "0.8.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "rmp-serde" +version = "1.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155" +dependencies = [ + "rmp", + "serde", +] + +[[package]] +name = "rmpv" +version = "1.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a4e1d4b9b938a26d2996af33229f0ca0956c652c1375067f0b45291c1df8417" +dependencies = [ + "rmp", + "serde", + "serde_bytes", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rustix" +version = "1.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_bytes" +version = "0.11.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.10", +] + +[[package]] +name = "serde_json" +version = "1.0.154" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_stacker" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4936375d50c4be7eff22293a9344f8e46f323ed2b3c243e52f89138d9bb0f4a" +dependencies = [ + "serde", + "serde_core", + "stacker", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "signature" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" +dependencies = [ + "rand_core 0.6.4", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "socket2" +version = "0.5.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + +[[package]] +name = "stacker" +version = "0.1.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967" +dependencies = [ + "cc", + "cfg-if", + "libc", + "psm", + "windows-sys 0.61.2", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ae9ccbea3a7b0b09443036490c8f34a8b1b760fd345e5c0dae0aac718faea79" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.3", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" + +[[package]] +name = "universal-hash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" +dependencies = [ + "crypto-common", + "subtle", +] + +[[package]] +name = "uuid" +version = "1.29.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bed99139f7dac9b5cfe5f17be4d66dc7079a34dee1d74bcc77217119240e267c" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c" +dependencies = [ + "cfg-if", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 3.0.10", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.130" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets", +] + +[[package]] +name = "windows-sys" +version = "0.59.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" +dependencies = [ + "windows-targets", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "yansi" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" + +[[package]] +name = "zerocopy" +version = "0.8.63" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5fe1f8f1b06191a00962174c61aa5005e0bb391a6d80d07e24d115c01a92ed8" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.63" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "863ad3ac83293fb4d740aedbfdc9240dd8d1a50c1099acd76ce80ce7c7230c7f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zeroize" +version = "1.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/packages/cultnet-rs/README.md b/packages/cultnet-rs/README.md index c356b97b..69a8e70d 100644 --- a/packages/cultnet-rs/README.md +++ b/packages/cultnet-rs/README.md @@ -30,7 +30,7 @@ knock on the contract before touching the furniture. ## Receipts ```powershell -cargo test +cargo test --locked ``` The initial tests prove: diff --git a/packages/cultnet-rs/examples/cultnet_impair/main.rs b/packages/cultnet-rs/examples/cultnet_impair/main.rs index c2ebd874..35a2168d 100644 --- a/packages/cultnet-rs/examples/cultnet_impair/main.rs +++ b/packages/cultnet-rs/examples/cultnet_impair/main.rs @@ -7,7 +7,7 @@ //! traffic crosses the same schedule. The schedule is `policy.rs`. //! //! ```text -//! cargo run --release --example cultnet_impair -- \ +//! cargo run --locked --release --example cultnet_impair -- \ //! --listen 0.0.0.0:17890 --upstream 127.0.0.1:17990 \ //! --profile tests/impairment/loss-1pct.toml --seed 424242 --metrics loss-1pct.csv //! ``` diff --git a/packages/cultnet-rs/src/idunn_runtime_provider/tests.rs b/packages/cultnet-rs/src/idunn_runtime_provider/tests.rs index 9f2d53ec..31760f2c 100644 --- a/packages/cultnet-rs/src/idunn_runtime_provider/tests.rs +++ b/packages/cultnet-rs/src/idunn_runtime_provider/tests.rs @@ -51,6 +51,7 @@ fn write_record(path: &Path, key: &str, ty: &str, schema: &str, payload: Vec stored_at: "2026-10-29T00:00:00Z".into(), schema_id: Some(schema.into()), }) + .map(|_| ()) } // ------------------------------------------------------------------ fake host diff --git a/packages/cultnet-rs/src/replication.rs b/packages/cultnet-rs/src/replication.rs index 743a28fd..7816e7dd 100644 --- a/packages/cultnet-rs/src/replication.rs +++ b/packages/cultnet-rs/src/replication.rs @@ -146,6 +146,7 @@ impl CultNetDocumentRegistry { schema_id: binding.schema_id.clone(), schema_names: vec![holder.document_type.clone(), binding.document_type.clone()], record_key: String::new(), + unread_slots: None, })); } self.bindings_by_schema_id diff --git a/packages/cultnet-rs/src/service_identity.rs b/packages/cultnet-rs/src/service_identity.rs index 63ec29f6..8ea978c8 100644 --- a/packages/cultnet-rs/src/service_identity.rs +++ b/packages/cultnet-rs/src/service_identity.rs @@ -347,7 +347,9 @@ pub fn export_service_identity_trust_anchor( }; let mut backing = SingleFileMessagePackBackingStore::new(path); match backing.pull_all()?.as_slice() { - [] => backing.push(&envelope)?, + [] => { + backing.push(&envelope)?; + } [current] if current == &envelope => {} _ => bail!("service identity trust anchor output already contains different state"), } diff --git a/packages/cultnet-rs/tests/cultnet.rs b/packages/cultnet-rs/tests/cultnet.rs index 64c57aaf..7bc84957 100644 --- a/packages/cultnet-rs/tests/cultnet.rs +++ b/packages/cultnet-rs/tests/cultnet.rs @@ -1810,12 +1810,10 @@ fn a_replicated_payload_nested_past_the_bound_is_refused_typed_on_apply() -> Res }; assert_eq!(documents.len(), 1); - backing.push(&envelope("past", limit + 1))?; - let mut cache = CultCache::new(); - cache.register_entry_type::()?; - cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&store))?; - cache.pull_all_backing_stores()?; - let past = registry.create_raw_snapshot_response(&cache, "past", None, Some(&["past".to_string()]))?; + // The sender cannot load a payload past the bound, so the frame is the at-limit one with its bytes swapped. + let mut past = at_limit.clone(); + let CultNetMessage::SnapshotResponseRaw { documents, .. } = &mut past else { unreachable!() }; + documents[0].payload = envelope("past", limit + 1).payload; let mut target = CultCache::new(); target.register_entry_type::()?; let error = registry diff --git a/packages/cultnet-rs/tests/decode_door.rs b/packages/cultnet-rs/tests/decode_door.rs index fd5bab6a..c55a6404 100644 --- a/packages/cultnet-rs/tests/decode_door.rs +++ b/packages/cultnet-rs/tests/decode_door.rs @@ -190,6 +190,16 @@ fn message_past_the_bound_is_refused_typed() { }); } +// A frame is one message: cultcache's payload rule refuses bytes after it, by count. +#[test] +fn frame_with_bytes_after_is_refused() { + let mut bytes = hello_with(&nested(0)); + assert!(decode_message(bytes.clone(), BIG).is_ok()); + bytes.push(0xc0); + let error = format!("{:#}", decode_message(bytes, BIG).unwrap_err()); + assert!(error.contains("1 bytes after its first MessagePack value"), "{error}"); +} + fn raw_v1_bytes(header_unknown: Option<&[u8]>) -> Vec { let mut header: Vec<(rmpv::Value, rmpv::Value)> = vec![ ("schemaId".into(), "s".into()), diff --git a/packages/cultnet-rs/tests/media_fec_bench.rs b/packages/cultnet-rs/tests/media_fec_bench.rs index 40c9de78..f0279495 100644 --- a/packages/cultnet-rs/tests/media_fec_bench.rs +++ b/packages/cultnet-rs/tests/media_fec_bench.rs @@ -3,7 +3,7 @@ //! Ignored by default; run in release, where the numbers mean something: //! //! ```text -//! cargo test --release --test media_fec_bench -- --ignored --nocapture +//! cargo test --locked --release --test media_fec_bench -- --ignored --nocapture //! ``` //! //! Each line is nanoseconds per call through the public surface, so record diff --git a/packages/cultnet-ts/CHANGELOG.md b/packages/cultnet-ts/CHANGELOG.md index cd5a32f2..b2580aba 100644 --- a/packages/cultnet-ts/CHANGELOG.md +++ b/packages/cultnet-ts/CHANGELOG.md @@ -3,10 +3,11 @@ All notable changes to `@gamecult/cultnet-ts` are recorded here, following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). -## Unreleased +## [0.4.0] ### Breaking +- `CultNetDocumentRegistry.applyRawDocumentPutMessage` and `createRawSnapshotResponseForPut` throw `A raw put must carry a storedAt stamp.` when the put's `storedAt` is missing, empty or not a stamp (`isStoredAtStamp` of cultcache-ts), and nothing is stored. The parser no longer refuses a `cultnet.document_put_raw.v0` for its `storedAt`: the server and the node judge it, so a refusal can name `document` (snapshot records and database changes are still read strictly as non-empty). - `decodeManifest` refuses a manifest with bytes after its first MessagePack value (`CDN artifact manifest has N bytes after its first MessagePack value.`), as the C# reference does; before, the tail was ignored. `contracts/cultmesh/content-acceptance-vectors.json` now holds `trailing_nil`, `trailing_reserved_code` and `trailing_truncated_array` as refused. ### Added diff --git a/packages/cultnet-ts/package.json b/packages/cultnet-ts/package.json index 60c9e13f..1ebba407 100644 --- a/packages/cultnet-ts/package.json +++ b/packages/cultnet-ts/package.json @@ -1,6 +1,6 @@ { "name": "@gamecult/cultnet-ts", - "version": "0.3.0", + "version": "0.4.0", "description": "CultNet-style typed direct-pipe messaging and state replication for TypeScript runtimes.", "main": "dist/index.js", "types": "dist/index.d.ts", @@ -73,7 +73,7 @@ "dependencies": { "@msgpack/msgpack": "^3.1.3", "ajv": "^8.17.1", - "@gamecult/cultcache-ts": "^0.15.0" + "@gamecult/cultcache-ts": "^0.16.0" }, "devDependencies": { "@types/node": "^24.0.0", diff --git a/packages/cultnet-ts/src/contracts.ts b/packages/cultnet-ts/src/contracts.ts index b733c959..5b06d5bc 100644 --- a/packages/cultnet-ts/src/contracts.ts +++ b/packages/cultnet-ts/src/contracts.ts @@ -1289,7 +1289,9 @@ function validateDocumentPutRawMessage(input: unknown): asserts input is CultNet throw new Error("Validation failed for cultnet.document_put_raw.v0: /document: is required"); } - validateRawDocumentRecord(candidate.document, "/document"); + // A peer's put is judged for its stamp by the server and the node (isStoredAtStamp), which answer message_invalid naming + // `document` after the write authority has spoken; the parser must let such a put through to them. + validateRawDocumentRecord(candidate.document, "/document", { judgesStoredAt: false }); } function validateSnapshotResponseRawMessage( @@ -1450,6 +1452,7 @@ function validateBinaryMessageRoot( function validateRawDocumentRecord( input: unknown, instancePath: string, + { judgesStoredAt = true }: { judgesStoredAt?: boolean } = {}, ): asserts input is CultNetRawDocumentRecord { if (!input || typeof input !== "object") { throw new Error(`Validation failed for raw CultNet document record: ${instancePath}: must be an object`); @@ -1461,7 +1464,9 @@ function validateRawDocumentRecord( requireOptionalNonEmptyString(candidate.schemaVersion, `${instancePath}/schemaVersion`); requireOptionalNonEmptyString(candidate.schemaContentHash, `${instancePath}/schemaContentHash`); requireNonEmptyString(candidate.recordKey, `${instancePath}/recordKey`); - requireNonEmptyString(candidate.storedAt, `${instancePath}/storedAt`); + if (judgesStoredAt) { + requireNonEmptyString(candidate.storedAt, `${instancePath}/storedAt`); + } if (candidate.payloadEncoding !== "messagepack") { throw new Error( `Validation failed for raw CultNet document record: ${instancePath}/payloadEncoding: must equal messagepack`, diff --git a/packages/cultnet-ts/src/replication.ts b/packages/cultnet-ts/src/replication.ts index 32ee4163..96b07cc3 100644 --- a/packages/cultnet-ts/src/replication.ts +++ b/packages/cultnet-ts/src/replication.ts @@ -6,6 +6,7 @@ import type { } from "@gamecult/cultcache-ts"; import { SchemaConflictError, + isStoredAtStamp, decodeDocumentPayload, documentPayloadBytes, documentPayloadFromBytes, @@ -323,6 +324,10 @@ export class CultNetDocumentRegistry { envelope: CultCacheEnvelope; } { const binding = this.#requireSchemaBinding(message.document.schemaId); + // A put is stored under the stamp it carries; one that is no stamp is refused, and nothing is minted for it. + if (!isStoredAtStamp(message.document.storedAt)) { + throw new Error("A raw put must carry a storedAt stamp."); + } return { binding, envelope: { diff --git a/packages/cultnet-ts/test/content.test.ts b/packages/cultnet-ts/test/content.test.ts index ec23b64e..e2ae5ed9 100644 --- a/packages/cultnet-ts/test/content.test.ts +++ b/packages/cultnet-ts/test/content.test.ts @@ -392,6 +392,13 @@ test("decodeManifest accepts and refuses exactly what the reference does, and re } }); +test("a manifest with bytes after its first value is refused by count, whatever the tail is", () => { + const whole = encodeManifest(manifestOf()); + for (const tail of [[0xc0], [0xc1, 0xc1], [0x90]]) { + assert.throws(() => decodeManifest(Uint8Array.from([...whole, ...tail])), new RegExp(`${tail.length} bytes after its first MessagePack value`)); + } +}); + test("a manifest written as a map names the manifest, not a field inside it", () => { assert.throws(() => decodeManifest(encode({ 0: "a", 9: { k: "v" } })), /field 'manifest' has the wrong type/); }); diff --git a/packages/cultnet-ts/test/cultnet.test.ts b/packages/cultnet-ts/test/cultnet.test.ts index ac4f539d..581fd66a 100644 --- a/packages/cultnet-ts/test/cultnet.test.ts +++ b/packages/cultnet-ts/test/cultnet.test.ts @@ -14,6 +14,8 @@ import { SchemaConflictError, SingleFileMessagePackBackingStore, defineDocumentType, + documentPayloadBytes, + documentPayloadFromBytes, schemaIdentityOf, } from "@gamecult/cultcache-ts"; @@ -3140,3 +3142,24 @@ test("CultNet registry names the canonical schema id of a compatible alias, and assert.equal(registry.canonicalSchemaId("p3.nobody"), undefined); assert.equal(registry.canonicalSchemaId("p3.shared"), undefined, "an id several bindings claim resolves to none"); }); + +test("a replicated record with bytes after its first MessagePack value is refused by count and held nowhere", async () => { + const definition = defineDocumentType({ type: "tails.note", schemaId: "tails.note.v1", schema: z.object({ body: z.string() }) }); + const registry = new CultNetDocumentRegistry([defineCultNetDocumentBinding({ definition })]); + const origin = CultCache.builder().withDocumentType(definition).build(); + await origin.put(definition, "k", { body: "x" }); + const document = registry.createRawSnapshotResponse(origin, "tails").documents[0]!; + const whole = documentPayloadBytes(document.payload); + const target = CultCache.builder().withDocumentType(definition).build(); + await assert.rejects( + registry.applyRawDocumentPutMessage(target, { + schemaVersion: "cultnet.document_put_raw.v0", + messageId: "tails", + document: { ...document, payload: documentPayloadFromBytes(Uint8Array.from([...whole, 0xc0, 0xc1])) }, + }), + (error: Error) => error.message.includes('document type "tails.note" payload has 2 bytes after its first MessagePack value'), + ); + assert.deepEqual(target.snapshot(), []); + await registry.applyRawDocumentPutMessage(target, { schemaVersion: "cultnet.document_put_raw.v0", messageId: "whole", document }); + assert.deepEqual(target.getRequired(definition, "k"), { body: "x" }); +}); diff --git a/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md b/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md index 7856c78d..86dd821e 100644 --- a/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md +++ b/packages/cultnet-ts/test/fixtures/idunn-runtime/README.md @@ -6,7 +6,7 @@ cultnet-rs only, and Idunn's `IdunnRuntimeActivationLaunch` is that same crate's It needs Linux and a machine-id equal to the `machine-id` file here, because the provider identity is bound to it. On Yggdrasil: - DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh HEAD rust 'cd packages/cultnet-rs && cp /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run -q --example idunn_runtime_fixture /tmp/out' + DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh HEAD rust 'cd packages/cultnet-rs && cp /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run --locked -q --example idunn_runtime_fixture /tmp/out' then copy `/tmp/out` over this directory. `--cap-add SYS_ADMIN` exists only so the bind mount can present the committed `machine-id`, which keeps Yggdrasil's real machine-id out of the repo. Every `presence-*.bin` was accepted by `authenticate_runtime_presence_claim` and correlated by @@ -26,7 +26,7 @@ The generator takes its specification as input, so a consumer needs no patch to specification is command-line flags; `@` splices in a response file (one `--flag value` per line, `#` comments). CultLib's own sets are `fixture.args` here: - cargo run -q --example idunn_runtime_fixture /tmp/out @/fixture.args + cargo run --locked -q --example idunn_runtime_fixture /tmp/out @/fixture.args The consumer keeps its own `.args` file in its repository and runs the same command with the same machine-id bind mount as above. Global flags: `--source-repository`, `--detail`. Each diff --git a/packages/cultnet-ts/test/interop/cultnet-interop.test.ts b/packages/cultnet-ts/test/interop/cultnet-interop.test.ts index d0e732ce..3b8b7bc9 100644 --- a/packages/cultnet-ts/test/interop/cultnet-interop.test.ts +++ b/packages/cultnet-ts/test/interop/cultnet-interop.test.ts @@ -2795,7 +2795,7 @@ async function buildInteropPeers(): Promise { } async function buildRustInteropPeer(): Promise { - rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--quiet", "--example", "cultnet_interop_peer"], { + rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--locked", "--quiet", "--example", "cultnet_interop_peer"], { cwd: cultnetRsRoot, }).then(() => undefined); await rustInteropPeerBuild; diff --git a/packages/cultnet-ts/test/raw-put-stamp.test.ts b/packages/cultnet-ts/test/raw-put-stamp.test.ts new file mode 100644 index 00000000..e358c080 --- /dev/null +++ b/packages/cultnet-ts/test/raw-put-stamp.test.ts @@ -0,0 +1,53 @@ +// A peer's raw put is stored under the stamp it carries; one that is no stamp is refused and nothing is minted or stored +// (variants:ruling:raw-put-without-stamp-refused). The registry is the node-level owner: every row of the shared vector. +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import test from "node:test"; +import { decode } from "@msgpack/msgpack"; +import { z } from "zod"; +import { CultCache, defineDocumentType, encodeDocumentPayload } from "@gamecult/cultcache-ts"; +import { CultNetDocumentRegistry, defineCultNetDocumentBinding, parseCultNetMessage, type CultNetMessage } from "../src"; + +const VECTOR = join(__dirname, "..", "..", "..", "..", "tests", "vectors", "stored-at-stamp", "stamps.msgpack"); +const note = defineDocumentType({ + type: "tests.stamped_note", + schemaId: "tests.stamped_note.v0", + schema: z.object({ noteId: z.string(), text: z.string() }), + name: "noteId", +}); +const registry = new CultNetDocumentRegistry([defineCultNetDocumentBinding({ definition: note })]); + +function putOf(key: string, storedAt: unknown): CultNetMessage { + const payload = encodeDocumentPayload(note, { noteId: key, text: "peer" }); + const document: Record = { schemaId: note.schemaId, recordKey: key, payloadEncoding: "messagepack", payload }; + if (storedAt !== undefined) document.storedAt = storedAt; + return { schemaVersion: "cultnet.document_put_raw.v0", messageId: `put-${key}`, document } as unknown as CultNetMessage; +} + +test("the registry stores a raw put only under a stamp, for every row of the vector and for a missing or mistyped one", async () => { + const rows = decode(readFileSync(VECTOR)) as [string, boolean][]; + const cases: [string, unknown, boolean][] = [...rows.map(([text, parses]): [string, unknown, boolean] => [JSON.stringify(text), text, parses]), + ["missing", undefined, false], ["number", 1_700_000_000_000, false], ["null", null, false]]; + const cache = CultCache.builder().withDocumentType(note).build(); + for (const [label, storedAt, parses] of cases) { + const message = putOf("k", storedAt); + // The parser lets a put through to its judges: the server and the registry refuse it, naming the document. + parseCultNetMessage(message, "cultnet.schema.v0"); + const before = cache.getEnvelope(note, "k"); + if (parses) { + await registry.applyRawDocumentPutMessage(cache, message as never); + assert.equal(cache.getRequiredEnvelope(note, "k").storedAt.length > 0, true, label); + } else { + await assert.rejects(registry.applyRawDocumentPutMessage(cache, message as never), /storedAt stamp/, label); + assert.deepEqual(cache.getEnvelope(note, "k"), before, `${label}: nothing minted or stored`); + } + } + // The refused puts left the store writable. + await registry.applyRawDocumentPutMessage(cache, putOf("after", "2026-11-07T00:00:00Z") as never); + assert.equal(cache.getRequired(note, "after").text, "peer"); +}); + +test("a refused put does not serve either: createRawSnapshotResponseForPut refuses it too", () => { + assert.throws(() => registry.createRawSnapshotResponseForPut(putOf("k", "yesterday") as never, "m"), /storedAt stamp/); +}); diff --git a/packages/cultnet-ts/test/runtime-presence-health.test.ts b/packages/cultnet-ts/test/runtime-presence-health.test.ts index 2dc4a845..1d7c4fa6 100644 --- a/packages/cultnet-ts/test/runtime-presence-health.test.ts +++ b/packages/cultnet-ts/test/runtime-presence-health.test.ts @@ -2,7 +2,7 @@ // // The vectors below were produced by `cultnet-rs`, not written by hand: // -// cargo run --example runtime_presence_vector -p cultnet-rs +// cargo run --locked --example runtime_presence_vector -p cultnet-rs // // That matters. A fixture authored for both sides at once agrees with itself // and proves nothing; the contract is only real if one fork emits the bytes and diff --git a/packages/heimdall-consent/README.md b/packages/heimdall-consent/README.md index 3782e720..926b7d1a 100644 --- a/packages/heimdall-consent/README.md +++ b/packages/heimdall-consent/README.md @@ -7,7 +7,7 @@ plus the rules that need no store to state (`Answer::is_training_data`, `Answer: (comments 682 and 759). Specification only: no collector, tally, export or tombstone hashing lives here. Entry points: `src/lib.rs`; golden vectors `tests/vectors/consent_golden.csv`, run by `tests/golden.rs` -(`cargo test`). Planned, not live: the collector, the tally, the training export. +(`cargo test --locked`). Planned, not live: the collector, the tally, the training export. ## Where each ruling lands (doc lines at aaeb175) diff --git a/packages/heimdall-stake/README.md b/packages/heimdall-stake/README.md index 5c8ae095..9b6acb19 100644 --- a/packages/heimdall-stake/README.md +++ b/packages/heimdall-stake/README.md @@ -11,8 +11,8 @@ adversary, storage, and any projection. Every parameter is a placeholder until t ## First call ``` -cargo run --example first_call # prints the Meta example and a splitting table -cargo test +cargo run --locked --example first_call # prints the Meta example and a splitting table +cargo test --locked ``` ```rust diff --git a/scripts/check-changelog-semver.test.mjs b/scripts/check-changelog-semver.test.mjs index 39c07799..3773136d 100644 --- a/scripts/check-changelog-semver.test.mjs +++ b/scripts/check-changelog-semver.test.mjs @@ -857,7 +857,7 @@ test("workflow: each job releases only through release-after-suite.mjs", () => { }); test("release-after-suite checks the tag, then runs the suite, the semver check and the package's tests, and releases only when all passed", async () => { - const { releaseAfterSuite, suite, checker } = await import("./release-after-suite.mjs"); + const { releaseAfterSuite, suite, checker, rangeCheck } = await import("./release-after-suite.mjs"); const npmVersion = JSON.parse(readFileSync(join(repoRoot, "packages", "cultcache-ts", "package.json"), "utf8")).version; const npmDir = join(repoRoot, "packages", "cultcache-ts"); const pyDir = join(repoRoot, "packages", "cultnet-py"); @@ -881,13 +881,15 @@ test("release-after-suite checks the tag, then runs the suite, the semver check const npmCalls = [ suiteCall, checkerCall("@gamecult/cultcache-ts", npmVersion), + ["npm", ["run", "build:ts"], repoRoot, checked], + [process.execPath, [rangeCheck], repoRoot, checked], ["npm", ["test"], npmDir, checked], ["npm", ["publish", "--access", "public"], npmDir, env], ]; assert.equal(go(["npm-publish", npmDir, npmTag]), 0); assert.deepEqual(calls, npmCalls); // any check that fails stops the release at that check - for (const fail of [0, 1, 2]) { + for (const fail of [0, 1, 2, 3, 4]) { assert.equal(go(["npm-publish", npmDir, npmTag], fail), 1, `check ${fail}`); assert.deepEqual(calls, npmCalls.slice(0, fail + 1), `check ${fail}`); } diff --git a/scripts/release-after-suite.mjs b/scripts/release-after-suite.mjs index 7832b23c..a7f372ca 100644 --- a/scripts/release-after-suite.mjs +++ b/scripts/release-after-suite.mjs @@ -1,7 +1,9 @@ // How publish-packages.yml releases a package. From this one process, in order: the release tag must name // the package's declared prefix and its manifest version, the release-checks suite must pass, the semver -// policy check must pass for that version, the package's own tests must pass, and only then the release -// action runs. No workflow condition can then release what any of those checks did not pass. +// policy check must pass for that version, an npm release's declared dependency ranges must hold (the +// workspace is built, then every tagged release a sibling's range admits is checked against what the +// sibling imports), the package's own tests must pass, and only then the release action runs. No workflow +// condition can then release what any of those checks did not pass. // docs/semver-policy.md, "The suite gates every release". // // node scripts/release-after-suite.mjs npm-publish npm publish --access public in @@ -18,6 +20,7 @@ import releasePackages from "./release-packages.mjs"; export const suite = fileURLToPath(new URL("./check-changelog-semver.test.mjs", import.meta.url)); export const checker = fileURLToPath(new URL("./check-changelog-semver.mjs", import.meta.url)); +export const rangeCheck = fileURLToPath(new URL("./test-declared-dependency-ranges.mjs", import.meta.url)); const repoRoot = fileURLToPath(new URL("..", import.meta.url)); // The checks run without the variables that can make `node --test` run nothing and exit 0 @@ -34,6 +37,8 @@ export const actions = { "npm-publish": { manifest: (dir) => JSON.parse(readFileSync(join(dir, "package.json"), "utf8")), test: (dir) => ["npm", ["test"], dir], + // The range check builds each tagged release against the workspace's built dists, so it follows a workspace build. + ranges: [["npm", ["run", "build:ts"], repoRoot], [process.execPath, [rangeCheck], repoRoot]], release: (dir) => ["npm", ["publish", "--access", "public"], dir], tagOnly: true, }, @@ -83,6 +88,9 @@ export function releaseAfterSuite(argv, run = spawn, env = process.env) { if (isTag && run(process.execPath, [checker, "--package", manifest.name, "--version", manifest.version, "--cwd", repoRoot], undefined, clean) !== 0) { return refuse(`${manifest.name} ${manifest.version} fails the semver policy (docs/semver-policy.md)`); } + for (const [command, args, cwd] of isTag ? action.ranges ?? [] : []) { + if (run(command, args, cwd, clean) !== 0) return refuse(`${manifest.name}'s declared dependency ranges do not hold (scripts/test-declared-dependency-ranges.mjs)`); + } if (run(...action.test(dir), clean) !== 0) return refuse(`${manifest.name}'s own tests failed`); return run(...action.release(dir), env); } diff --git a/scripts/test-declared-dependency-ranges.mjs b/scripts/test-declared-dependency-ranges.mjs index b528fb75..40935e39 100644 --- a/scripts/test-declared-dependency-ranges.mjs +++ b/scripts/test-declared-dependency-ranges.mjs @@ -22,6 +22,7 @@ import { fileURLToPath } from "node:url"; const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const rootRequire = createRequire(join(repoRoot, "package.json")); const ts = rootRequire("typescript"); +const npmCli = join(npmRoot(), "bin", "npm-cli.js"); const semver = createRequire(join(npmRoot(), "package.json"))("semver"); // npm ships semver, so the check needs no dependency of its own. npm's root is beside the node binary on Windows and under lib/ on Unix. @@ -84,16 +85,17 @@ function sourceFiles(directory) { }); } -// Builds the package as the tag released it, in a scratch directory that borrows the workspace's node_modules. +// Builds the package as the tag released it (with the repo-level contracts its generator reads), in a scratch directory that borrows the workspace's node_modules. function buildRelease(directory, tag, scratch) { const archive = join(scratch, `${tag}.tar`); - git("archive", "--format=tar", "-o", archive, tag, `packages/${directory}`); + git("archive", "--format=tar", "-o", archive, tag, `packages/${directory}`, "contracts"); const extracted = join(scratch, tag); mkdirSync(extracted); execFileSync("tar", ["-xf", archive, "-C", extracted]); symlinkSync(join(repoRoot, "node_modules"), join(extracted, "node_modules"), "junction"); const packageRoot = join(extracted, "packages", directory); - execFileSync(process.execPath, [rootRequire.resolve("typescript/bin/tsc"), "-p", "tsconfig.json"], { cwd: packageRoot, stdio: "inherit" }); + // The package's own build, which is how a release was made: cultnet-ts generates its contracts module first. + execFileSync(process.execPath, [npmCli, "run", "build", "--silent"], { cwd: packageRoot, stdio: "inherit" }); return rootRequire(join(packageRoot, "dist", "index.js")); } @@ -127,7 +129,7 @@ try { const prefix = `${dependency.directory}-v`; const admitted = tags.filter((tag) => { const version = tag.startsWith(prefix) ? tag.slice(prefix.length) : undefined; - return version !== undefined && semver.valid(version) !== null && semver.satisfies(version, range) && version !== dependency.manifest.version; + return version !== undefined && semver.valid(version) !== null && semver.satisfies(version, range); }); for (const tag of admitted) { if (!built.has(tag)) { diff --git a/scripts/test-declared-dependency-ranges.test.mjs b/scripts/test-declared-dependency-ranges.test.mjs new file mode 100644 index 00000000..644bfc1a --- /dev/null +++ b/scripts/test-declared-dependency-ranges.test.mjs @@ -0,0 +1,71 @@ +// test-declared-dependency-ranges.mjs against a fixture repo: a dependency that gained an export a dependent imports, +// while its manifest still carries the version of its latest release tag, must fail the check, because a consumer's +// range admits that tagged release and it lacks the export. Bumping the version and the range makes it pass. +// The fixture package a builds like cultnet-ts: its build script first generates a module from the repo-level contracts/ +// directory, so a tag built by a bare tsc, or archived without contracts/, fails here as it does on the real tree. +import assert from "node:assert/strict"; +import { execFileSync, spawnSync } from "node:child_process"; +import { copyFileSync, mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; + +const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const script = join(repoRoot, "scripts", "test-declared-dependency-ranges.mjs"); + +const write = (root, path, text) => { + mkdirSync(dirname(join(root, path)), { recursive: true }); + writeFileSync(join(root, path), text); +}; +const git = (root, ...args) => execFileSync("git", ["-c", "user.name=t", "-c", "user.email=t@t", ...args], { cwd: root, stdio: "pipe" }); +const tsc = "node ../../node_modules/typescript/bin/tsc -p tsconfig.json"; +const manifest = (name, version, extra = {}, build = tsc) => + JSON.stringify({ name: `@t/${name}`, version, scripts: { build }, ...extra }); +const generated = "node tools/generate.mjs && " + tsc; +const tsconfig = JSON.stringify({ compilerOptions: { outDir: "dist", module: "commonjs", target: "es2020", declaration: false }, include: ["src"] }); + +function fixture() { + const root = mkdtempSync(join(tmpdir(), "declared-ranges-fixture-")); + write(root, "package.json", JSON.stringify({ name: "fixture", private: true, workspaces: ["packages/a", "packages/b"] })); + write(root, "contracts/value.json", "2"); + write(root, "packages/a/tools/generate.mjs", + 'import { readFileSync, writeFileSync } from "node:fs";\n' + + 'writeFileSync("src/data.ts", `export const contractValue = ${readFileSync("../../contracts/value.json", "utf8")};\\n`);\n'); + write(root, "packages/a/package.json", manifest("a", "1.0.0", {}, generated)); + write(root, "packages/a/tsconfig.json", tsconfig); + write(root, "packages/a/src/index.ts", 'export { contractValue } from "./data";\nexport const foo = 1;\n'); + write(root, "packages/b/package.json", manifest("b", "1.0.0", { dependencies: { "@t/a": "^1.0.0" } })); + write(root, "packages/b/tsconfig.json", tsconfig); + write(root, "packages/b/src/index.ts", 'import { foo } from "@t/a";\nexport const b = foo;\n'); + mkdirSync(join(root, "scripts")); + copyFileSync(script, join(root, "scripts", "test-declared-dependency-ranges.mjs")); + symlinkSync(join(repoRoot, "node_modules"), join(root, "node_modules"), "junction"); + git(root, "init", "-q", "-b", "main"); + git(root, "add", "package.json", "contracts", "packages", "scripts"); + git(root, "commit", "-q", "-m", "release a 1.0.0"); + git(root, "tag", "a-v1.0.0"); + return root; +} + +const check = (root) => spawnSync(process.execPath, [join(root, "scripts", "test-declared-dependency-ranges.mjs")], { cwd: root, encoding: "utf8" }); + +test("a dependency that gained an export a dependent imports, still at its latest tag's version, fails the check", () => { + const root = fixture(); + try { + assert.equal(check(root).status, 0, "the fixture passes before the export is added"); + + write(root, "packages/a/src/index.ts", 'export { contractValue } from "./data";\nexport const foo = 1;\nexport const bar = 2;\n'); + write(root, "packages/b/src/index.ts", 'import { foo, bar } from "@t/a";\nexport const b = foo + bar;\n'); + const stale = check(root); + assert.notEqual(stale.status, 0, stale.stdout); + assert.match(stale.stderr, /a-v1\.0\.0 does not export bar/u); + + write(root, "packages/a/package.json", manifest("a", "1.1.0", {}, generated)); + write(root, "packages/b/package.json", manifest("b", "1.0.0", { dependencies: { "@t/a": "^1.1.0" } })); + const bumped = check(root); + assert.equal(bumped.status, 0, bumped.stderr); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); diff --git a/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs b/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs index 8f76cbae..6e0ce6cd 100644 --- a/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs +++ b/src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs @@ -64,48 +64,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore var reports = new List(); var loaded = new Dictionary(StringComparer.Ordinal); var foreign = new List(); - for (var attempt = 1; ; attempt++) - { - loaded.Clear(); - reports.Clear(); - foreign.Clear(); - // Only pages named by the manifest read under the lease are loaded; orphaned pages are never loaded. With no - // lock to lease (none created yet, or a store copied without it) a writer may commit mid-load, so an unleased - // load stands only if its manifest is still current; if the manifest moved it reloads, under the lease the - // writer created. A page that fails under an unmoved manifest is corruption, not a race. - using var lease = AcquireCommitLease(wait: true, create: false); - var manifestBytes = ReadManifestBytes(); - var (manifest, _) = ParseManifest(manifestBytes); - Trace($"manifest records={manifest.Records.Length}"); - try - { - if (lease == null) - UnleasedManifestRead?.Invoke(); - LoadRecordPages( - manifest.Records.OrderBy(record => record.Key, StringComparer.Ordinal).ToArray(), - manifest.SchemaCatalog, - loaded, - reports, - foreign); - if (lease != null || !ManifestMoved(manifestBytes)) - { - Trace($"indexed-pages loaded={loaded.Count}"); - break; - } - } - catch (Exception exception) when (lease == null && IsTornGeneration(exception)) - { - // The re-read stays out of the filter: an exception thrown inside a filter is swallowed as false. - if (!ManifestMoved(manifestBytes)) - throw; - if (attempt == UnleasedLoadAttempts) - throw Unsettled(exception); - continue; - } - - if (attempt == UnleasedLoadAttempts) - throw Unsettled(null); - } + LoadManifestPages(null, loaded, reports, foreign, Trace); bool Staged(string key) => _dirtyKeys.ContainsKey(key) || _deletedKeys.ContainsKey(key); var arrived = loaded @@ -136,6 +95,70 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore Trace("publish"); } + // The one load of manifest pages, for the whole store (key null) or one record: under the lease, with the moved-manifest retry. + // It reads only; adopting what it loaded is the caller's. + private void LoadManifestPages( + string? key, + Dictionary loaded, + List reports, + List foreign, + Action trace) + { + for (var attempt = 1; ; attempt++) + { + loaded.Clear(); + reports.Clear(); + foreign.Clear(); + // Only pages named by the manifest read under the lease are loaded; orphaned pages are never loaded. With no + // lock to lease (none created yet, or a store copied without it) a writer may commit mid-load, so an unleased + // load stands only if its manifest is still current; if the manifest moved it reloads, under the lease the + // writer created. A page that fails under an unmoved manifest is corruption, not a race. + using var lease = AcquireCommitLease(wait: true, create: false); + var manifestBytes = ReadManifestBytes(); + var (manifest, _) = ParseManifest(manifestBytes); + trace($"manifest records={manifest.Records.Length}"); + try + { + if (lease == null) + UnleasedManifestRead?.Invoke(); + LoadRecordPages( + manifest.Records.Where(record => key == null || string.Equals(record.Key, key, StringComparison.Ordinal)) + .OrderBy(record => record.Key, StringComparer.Ordinal).ToArray(), + manifest.SchemaCatalog, + loaded, + reports, + foreign); + if (lease != null || !ManifestMoved(manifestBytes)) + { + trace($"indexed-pages loaded={loaded.Count}"); + break; + } + } + catch (Exception exception) when (lease == null && IsTornGeneration(exception)) + { + // The re-read stays out of the filter: an exception thrown inside a filter is swallowed as false. + if (!ManifestMoved(manifestBytes)) + throw; + if (attempt == UnleasedLoadAttempts) + throw Unsettled(exception); + continue; + } + + if (attempt == UnleasedLoadAttempts) + throw Unsettled(null); + } + } + + // The record at a key as its page holds it now, decoded fresh and stamped with VersionOf its manifest payload, the version + // CommitBatch compares. Null when none is there or it is foreign. Touches no entry, hydration, report or foreign list of the + // store, and raises nothing. + internal override CultStoredDocument? ReadDurable(string key) => Held(() => + { + var loaded = new Dictionary(StringComparer.Ordinal); + LoadManifestPages(key, loaded, new List(), new List(), _ => { }); + return loaded.TryGetValue(key, out var stored) ? stored : null; + }); + // Variants wait for the directory store's own cut; until then it refuses one rather than persist a resolved view as a plain page. private static void RefuseVariant(CultStoredDocument entry) { @@ -237,7 +260,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore }); } - public override void PushAll() + public override void PushAll(bool soft) { ThrowIfReadOnly(); Held(() => @@ -247,7 +270,12 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore Directory.CreateDirectory(_manifestFile.DirectoryName!); using var commitLease = AcquireCommitLease(wait: true, create: true); - var (manifest, catalogBytes) = ReadManifest(); + var manifestBytes = ReadManifestBytes(); + var (manifest, catalogBytes) = ParseManifest(manifestBytes); + // A soft flush writes only where no manifest is: the reader's answer (null is absent), not a second look at the path. + // A manifest it read stays as it is, its staged writes staged. + if (soft && manifestBytes != null) + return; RefuseUndeclaredOnDisk(manifest, StagedKeys, Array.Empty(), Array.Empty(), Array.Empty()); if (IsDirty) WriteGeneration(manifest, catalogBytes); @@ -681,14 +709,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore stream.Flush(true); } - if (File.Exists(path)) - { - ReplaceExistingFile(tempPath, path); - } - else - { - File.Move(tempPath, path); - } + StoreReplace.Replace(tempPath, path); } finally { @@ -710,17 +731,23 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore // Only a writer creates the records folder and the lock, so opening the store creates nothing. A reader opens an // existing lock and returns null when there is none; that does not mean no writer committed (a store may be copied // without its lock, and a first writer may create it mid-load), so an unleased load checks that it settled. + // A writer's lease is also where a link at the manifest path is refused (a commit would replace the link with a file): before + // the records folder or the lock is made, and again once the lease is held, because a link can be put there while it waited. private FileStream? AcquireCommitLease(bool wait, bool create) { if (create) + { + CultStoreLinks.Refuse(_manifestFile.FullName); Directory.CreateDirectory(_recordDirectory.FullName); + } var lockPath = Path.Combine(_recordDirectory.FullName, ".commit.lock"); var started = Stopwatch.StartNew(); while (true) { + FileStream held; try { - return new FileStream( + held = new FileStream( lockPath, create ? FileMode.OpenOrCreate : FileMode.Open, FileAccess.ReadWrite, @@ -743,29 +770,23 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30)) { Thread.Sleep(10); + continue; } - } - } - private static void ReplaceExistingFile(string sourcePath, string destinationPath) - { - for (var attempt = 0; attempt < 3; attempt++) - { - try + if (create) { - File.Replace(sourcePath, destinationPath, null); - return; + try + { + CultStoreLinks.Refuse(_manifestFile.FullName); + } + catch + { + held.Dispose(); + throw; + } } - catch (IOException) when (attempt < 2) - { - Thread.Sleep(10 * (attempt + 1)); - } - catch (UnauthorizedAccessException) when (attempt < 2) - { - Thread.Sleep(10 * (attempt + 1)); - } - } - File.Replace(sourcePath, destinationPath, null); + return held; + } } } diff --git a/src/GameCult.Caching.MessagePack/GameCult.Caching.MessagePack.csproj b/src/GameCult.Caching.MessagePack/GameCult.Caching.MessagePack.csproj index 63cbf614..0afdd0d3 100644 --- a/src/GameCult.Caching.MessagePack/GameCult.Caching.MessagePack.csproj +++ b/src/GameCult.Caching.MessagePack/GameCult.Caching.MessagePack.csproj @@ -8,7 +8,7 @@ - + diff --git a/src/GameCult.Caching/Contracts/cultcache-store-composition.md b/src/GameCult.Caching/Contracts/cultcache-store-composition.md index 01f7a5c9..3e7bd425 100644 --- a/src/GameCult.Caching/Contracts/cultcache-store-composition.md +++ b/src/GameCult.Caching/Contracts/cultcache-store-composition.md @@ -315,11 +315,138 @@ refuses to replace a file whose header it cannot read, and leaves it as it was. in that store, and leaves the store clean. - A conditional commit lands the batch onto the file as it is under the lock. -C#, Rust and TypeScript writers exclude each other on the shared lock file; -Python takes no cross-process lock yet (`variants:follow_up:py-shared-write-lock`), -and Rust's open fails rather than waits while a C# writer holds the lock -(`variants:follow_up:rust-lock-open-waits`). Another runtime need not obey the -`storedAt` rule. +C#, Rust, TypeScript and Python single-file writers exclude each other on the +shared lock file. Another runtime need not obey the `storedAt` rule. + +## Operating a single-file store + +What the lock guarantees, per runtime. Every claim cites the code. + +- **Scope.** The lock is the sidecar `.lock`, taken by every write of a + single-file store, never by a read except in Rust. It is advisory: it excludes + only openers that also take it, in this process and others. C# opens it + share-none (`CultCache.cs:5166-5183`; .NET emulates that with `flock(LOCK_EX)` + on Unix, switched off by `DOTNET_SYSTEM_IO_DISABLEFILELOCKING`, see + `docs/research/cross-process-file-locking-prior-art.md`); Rust takes + `fs2` exclusive (`cultcache-rs/src/lib.rs:1721-1726`); TypeScript `flock` or + `LockFileEx` (`cultcache-ts/src/store-file.ts:340-382`); Python the same + (`cultcache-py/src/cultcache_py/stores.py:342-389`). A tool that opens and + writes the `.cc` file without the lock is not excluded. `flock` is not + reliable on every network filesystem; keep stores on a local one. +- **What a second writer sees.** It waits. C# polls every 10 ms for 30 seconds, + then the `IOException` propagates (`CultCache.cs:5166-5183`); with `wait: false` + (`TryCommit`) it returns `Contended` (`CultCache.cs:2052`, `2897`, `4691-4693`). + TypeScript times out after 30 s (`store-file.ts:173-195`), Python raises + `TimeoutError` after `lock_timeout`, 30 s by default (`stores.py:342-354`). + Rust waits without a timeout on its write paths (`lib.rs:1721-1726`); only its + `try_with_exclusive_lock` returns without waiting (`lib.rs:1832-1843`). Each + write re-reads the file after taking the lock and applies its staged keys onto + it (`CultCache.cs:4659-4676`), so waiting writers merge rather than overwrite. + The in-memory cache of a writer is not refreshed by another's write until its + next `PullAll`, which does nothing while that cache is dirty + (`CultCache.cs:4582-4588`). +- **What a reader sees.** C#, TypeScript and Python read without the lock + (`CultCache.cs:5139`, `5227`; `single-file-messagepack-backing-store.ts:200`; + `stores.py:144-152`). The store file is only ever replaced by rename of a fully + written, fsynced temp file (`CultCache.cs:5191-5213`, `lib.rs:1693-1712`, + `store-file.ts:70-95`, `stores.py:316-326`), so a read gets one complete + generation, old or new, never a mix. Rust `pull_all` takes the lock shared and + so waits behind a writer (`lib.rs:1926-1930`); it creates the `.lock` file if + absent, so on a read-only directory use `pull_all_read_only_snapshot` + (`lib.rs:1667-1669`), which takes no lock and relies on the producer's atomic + replace. +- **Not covered.** A unique-index check or condition spanning two stores, or any + invariant between two files: each store has its own lock and no store carries a + generation counter (the file is `[formatVersion, schemaCatalog, records]`); + a record's identity is its `version`, the SHA-256 of its stored bytes. No + library writer holds two store locks at once (`CultCache.cs:2917-2926` judges + one store, releases, then commits the home). Python's `JsonLinesBackingStore` + takes only an in-process lock (`stores.py:73-80`) and is not a single-file store + in this sense. +- **The `.lock` file is not a stale-lock marker.** It is never written or deleted + by a library writer (`store-file.ts:167-172`, `stores.py:342-345`), and the + lock is released when the holding process exits. A `.lock` file left on disk is + normal. Do not delete it while any process may open the store: a new opener + would create a second file and take a second, independent lock on it. + +### A leftover temp file + +A write stages its bytes in a temp file beside the store and renames it over the +store. The name is consumed by the rename, so a temp file that still exists is a +write that was never applied. The store file itself is never partial because of +it. + +| Runtime | Temp name | Source | +|---|---|---| +| C# | `..<32 hex>.tmp` | `CultCache.cs:5191` | +| Rust | `..tmp` | `lib.rs:4268-4275` | +| TypeScript | `.tmp---` | `store-file.ts:70` | +| Python | `.tmp--` | `stores.py:316` | + +- **Which temp files are safe to delete.** One that no writer is writing: + delete only while you hold `.lock` yourself + (`flock .lock rm `, Linux, util-linux) or with every writer of + the store stopped. Merely testing that the lock is free is not enough: a writer + can take it and stage a new temp file the moment after the test. While you hold + the lock, every temp file of that store is abandoned. Deleting one a writer is + still writing destroys that write. +- **How to know the write was incomplete.** The temp file existing is the proof. + Its content may be partial (killed during the write) or whole (killed between + the fsync and the rename); either way it was not applied, and the writer that + made it never saw its flush succeed. The store holds the last completed write. + Do not rename a temp file over the store by hand. Rust additionally reports a + rename that landed but whose directory fsync failed as `ReplacedNotDurable` + (`lib.rs:1685-1688`; TypeScript `StoreReplacedNotDurableError`, + `store-file.ts:88-92`): after a power loss the store may still be the earlier + generation. +- **What the store does about it.** Opening and reading ignore temp files and + never delete them. A writer removes only its own temp file when its write + throws (`CultCache.cs:5216-5222`, `store-file.ts:82-84`, `stores.py:327-331`, + `lib.rs:1707-1710`); a killed process skips that. Rust alone sweeps: its next + write, under the exclusive lock, deletes every regular file named + `..tmp` (`lib.rs:1698`, `4318-4352`). C#, TypeScript and + Python never sweep, and their unique names mean leftovers accumulate until an + operator deletes them. A store file that cannot be read (truncated, wrong + header) is refused at open and left byte-identical (the open reads through + `ReadSnapshot`, `CultCache.cs:4590`, `5104-5118`, whose `ReadStore` must throw + `CultStoreUnreadableException` for anything that is not exactly one store, + `4510-4516`); that is the store, not a temp file, and is + restored from a copy, never deleted and rewritten. + +### Backing up a pair of stores + +Each store is whole on its own: copying one `.cc` file is safe at any moment, +because the file is only ever replaced whole, and a copy of the path is one +complete generation. Two stores have two independent locks and no shared +generation, so copying them one after the other can capture store A at one +moment and store B at a later one. If an application invariant spans the pair, +that copy may violate it. Back up the pair at one point in time: + +1. **Preferred, quiesce.** Stop (or have flush and close) every process that + writes either store. Flush staged writes first; `PullAll` and the dispose + flush do not capture unflushed state of a still-running cache. Copy both `.cc` + files. Do not copy `.lock` files or temp files. Start the writers again. +2. **Without stopping (Unix): per-store consistency only.** Take both locks in + one fixed order (sorted by full path), copy while holding both, release: + `flock A.cc.lock flock B.cc.lock cp A.cc B.cc /backup/dir/`. + This does not make the pair consistent. A writer holds one store lock at a + time, so it can land A and then B with a gap between; holding both locks + cannot pause a writer that has landed A and not yet B, so the copy can hold A + after and B before that pair of writes. It guarantees each copy is one + complete generation of its store and nothing more. Any invariant spanning the + pair requires step 1. Cooperating writers wait (30 s in C#, TypeScript and + Python; Rust waits indefinitely), so keep the hold short. Unflushed state of + live caches is not in it. On Windows, no shipped tool holds the C# share-none + lock for you: quiesce. +3. **Verify.** Open each copy read-only with the runtime that will use it (for + Rust, `pull_all_read_only_snapshot`); an unreadable copy is refused, not + repaired (`CultCache.cs:4510-4516`, `5104-5118`). +4. **Restore both files together** from the same backup, with every writer + stopped, then restart so each cache loads fresh. Copy each file to a temp + name in the store's own directory and `mv` it over the store: a plain `cp` + over the store is not atomic, and C#, TypeScript and Python readers take no + lock, so they can read a torn file. Restoring one half of a pair from a + different backup recreates the skew. Leave the `.lock` files as they are. ## Serialization options diff --git a/src/GameCult.Caching/CultCache.cs b/src/GameCult.Caching/CultCache.cs index 48bad3a3..303d3737 100644 --- a/src/GameCult.Caching/CultCache.cs +++ b/src/GameCult.Caching/CultCache.cs @@ -2052,6 +2052,29 @@ namespace GameCult.Caching Contended } + public enum CultUpdateOutcome + { + Updated, + Declined, + LostRace + } + + /// + /// What did. A lost race is a value, not an exception. is the instance the + /// cache holds at the key when Update returns: the written one for Updated, null when the key is absent. + /// + public readonly struct CultUpdateResult where T : class + { + public CultUpdateResult(CultUpdateOutcome outcome, T? document) + { + Outcome = outcome; + Document = document; + } + + public CultUpdateOutcome Outcome { get; } + public T? Document { get; } + } + public sealed class CultCommitRequest { internal CultCommitRequest( @@ -2537,19 +2560,24 @@ namespace GameCult.Caching return Task.CompletedTask; } - public void FlushAllBackingStores() => Held(() => + public void FlushAllBackingStores(bool soft = false) => Held(() => { foreach (var (store, _) in _stores) { if (!store.IsReadOnly && (store.IsDirty || store.HasStagedChecks)) - store.PushAll(); + store.PushAll(soft); } return true; }); - public Task FlushAsync() + /// + /// Flushes every dirty backing store. With the flush has one meaning, decided by each store's + /// own read: a store with nothing at its path is written; a store that reads (empty included) is skipped and its + /// staged writes stay staged; a store that cannot be read refuses with . + /// + public Task FlushAsync(bool soft = false) { - FlushAllBackingStores(); + FlushAllBackingStores(soft); return Task.CompletedTask; } @@ -2558,6 +2586,7 @@ namespace GameCult.Caching return UpsertAsync(document, handle); } + // A blind write, last-writer-wins per key; Update is the conditional golden path. // Refused with CultSchemaConflictException when the key holds a record stored under an id no registered type declares. public Task> UpsertAsync(T document, CultRecordHandle? handle = null) { @@ -2634,6 +2663,7 @@ namespace GameCult.Caching /// /// Commits the staged batch. Returns false when a condition failed: nothing was written, changed in memory, or published. + /// Without Expect a commit is blind; retries a conditional one. /// /// /// A commit that succeeds and then fails in a journal or observer still stands and is published, and the writer @@ -2651,6 +2681,93 @@ namespace GameCult.Caching return Land(stage, wait: false); } + /// + /// Reads the record at as its home store holds it, applies to a fresh copy of it, and + /// commits the result only if the record is still the one that was read; when another writer got there first it re-reads and runs + /// again, up to times. The write is durable when Update returns. + /// + /// + /// may run up to times. It gets the record as the store that holds it has it now, read from every home + /// of T and its routed subtypes and decoded afresh for this attempt (null when the key holds nothing), which it may change and return. A record held by two stores, or an apply that changes the record's home store, throws. A change made in place to an + /// instance the cache holds is neither seen nor written. It must not write to the cache or have other side effects. Return null to write + /// nothing. A store with staged blind writes refuses Update until it is flushed, as it refuses any conditional commit. A cache with no + /// store attached copies its held instance and needs a codec; a store kind outside the library cannot serve Update (). Running out of attempts returns ; every other failure throws. + /// + public CultUpdateResult Update(CultRecordKey key, Func apply, int maxAttempts = 5) where T : class + { + if (apply == null) throw new ArgumentNullException(nameof(apply)); + if (maxAttempts < 1) throw new ArgumentOutOfRangeException(nameof(maxAttempts), maxAttempts, "At least one attempt is required."); + for (var attempt = 1; ; attempt++) + { + var (version, copy, source) = Held(() => + { + if (_stores.Count == 0) + { + // No store: the held entry is the record. + _entries.TryGetValue(key.Value, out var held); + var codec = _codec ?? throw new InvalidOperationException("This cache has no codec; create it with CultCacheMessagePack.Create."); + if (held == null) + return ((string?)null, (T?)null, (CacheBackingStore?)null); + if (held.Variant != null || held.Document is not T) + throw new InvalidOperationException($"Record {key.Value} is a variant or not a {typeof(T).Name}; Update cannot copy it."); + return (held.StoredVersion, (T?)codec.Clone(held.Document, held.Descriptor.DocumentType), null); + } + + // The store with the key is among the homes of T and its routed subtypes; the cache's own view of the key may be stale, so it is not asked. + var homes = CandidateHomes(typeof(T)); + if (homes.Length == 0) + throw new InvalidOperationException("A conditional batch names its home store through a record it upserts or removes."); + CacheBackingStore? holder = null; + CultStoredDocument? durable = null; + foreach (var candidate in homes) + { + if (candidate.ReadDurable(key.Value) is not { } found) + continue; + if (holder != null) + throw new InvalidOperationException($"Record {key.Value} is held by two stores, {holder} and {candidate}; Update cannot choose."); + holder = candidate; + durable = found; + } + + if (durable == null) + return ((string?)null, (T?)null, (CacheBackingStore?)null); + if (durable.Variant != null || durable.Document is not T typed) + throw new InvalidOperationException($"Record {key.Value} is a variant or not a {typeof(T).Name}; Update cannot copy it."); + CultElementIds.Plan(durable.Descriptor.DocumentType, durable.Document, key.Value, deterministic: true).Apply(); + return (durable.StoredVersion, typed, holder); + }); + var next = apply(copy); + if (next == null) + return new CultUpdateResult(CultUpdateOutcome.Declined, Get(key)); + if (source != null && Held(() => Home(next.GetType())) is var landing && landing != source) + throw new InvalidOperationException( + $"Update of {key.Value} would move the record from {source} to {landing?.ToString() ?? "memory"}; remove it and write it at its new home."); + CultCommitOutcome outcome; + try + { + outcome = Land(batch => + { + batch.Expected.Add((key, version)); + batch.Upsert(next, new CultRecordHandle(key)); + }, wait: true); + } + catch (CultWriteConflictException conflict) when (conflict.InnerException == null) + { + outcome = CultCommitOutcome.Mismatch; + } + + if (outcome == CultCommitOutcome.Committed) + return new CultUpdateResult(CultUpdateOutcome.Updated, next); + if (outcome != CultCommitOutcome.Mismatch) + throw new InvalidOperationException($"Update of {key.Value} was refused: {outcome}."); + if (attempt >= maxAttempts) + { + Held(() => { Home(next.GetType())?.PullAll(); return true; }); + return new CultUpdateResult(CultUpdateOutcome.LostRace, Get(key)); + } + } + } + public CultRecordHandle? TryGetHandle(T document) { if (document == null) throw new ArgumentNullException(nameof(document)); @@ -2671,7 +2788,8 @@ namespace GameCult.Caching /// when that store holds it unowned (undeclared drift that drops persisted slots, or a payload carrying slots past this build's, /// see ). For a key the cache does not hold, the removal check every store makes of it /// (false when a store lists it foreign). One write it does not predict: a Flatten of a variant whose base chain holds an unowned record is refused, though - /// the variant itself is writable. Live and advisory: the write still decides. + /// the variant itself is writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it + /// with . Live and advisory: the write still decides. /// public bool IsWritable(CultRecordKey key) { @@ -3667,6 +3785,18 @@ namespace GameCult.Caching return changes; } + // Every store a record of this type, or of a registered subtype of it, can live in. + private CacheBackingStore[] CandidateHomes(Type type) + { + var homes = new HashSet(); + if (Home(type) is { } own) + homes.Add(own); + foreach (var descriptor in _registry.AllDescriptors) + if (type.IsAssignableFrom(descriptor.DocumentType) && Home(descriptor.DocumentType) is { } routed) + homes.Add(routed); + return _stores.Select(entry => entry.Store).Where(homes.Contains).ToArray(); + } + // The most derived routed home type assignable from the document type, else the untyped store. private CacheBackingStore? Home(Type documentType) { @@ -3943,9 +4073,24 @@ namespace GameCult.Caching internal void UseRegistry(CultDocumentRegistry? registry) => _registry = registry; public abstract void PullAll(); + + /// + /// The record at as this store holds it now, decoded as a load decodes it by this call (a new instance nobody + /// else holds) and stamped with the version this store's commit compares. Null when the store holds nothing there or holds it foreign. + /// It changes none of the store's entries, staged writes or reports and raises nothing. Internal because only this library can stamp a version. + /// + internal virtual CultStoredDocument? ReadDurable(string key) => + throw new NotSupportedException($"{this} cannot read one record, which Update needs."); + public abstract void Push(CultStoredDocument entry); public abstract void Delete(CultStoredDocument entry); - public abstract void PushAll(); + public void PushAll() => PushAll(soft: false); + + /// + /// Applies the staged writes. A soft flush writes only where nothing is at the path: over a store it reads it returns without + /// writing and leaves the staged writes staged and dirty for a later plain flush, and over a file it cannot read it refuses. + /// + public abstract void PushAll(bool soft); /// /// One durable step under the store's exclusive lock: the batch and every staged write land onto the store as it is then, @@ -4184,7 +4329,7 @@ namespace GameCult.Caching /// /// /// A writable store kind that reads its durable state under a lock overrides this to call , and - /// judges at its next or as it judges + /// judges at its next or as it judges /// : under the lock, with the keys it writes, refused through the same refusal, never /// removed. It calls when that step succeeds, which clears them. A read-only store has no /// such step and keeps the default. Both shipped store kinds do this. A batch's own checks are judged through @@ -4661,7 +4806,7 @@ namespace GameCult.Caching // only per staged key. A key whose durable record is under an id this cache does not declare refuses the write, and in a store // holding a variant it lands only onto the store as this cache last read it (both ApplyWriteSet). // A file it cannot read (truncated, trailing bytes, not a store, a format it does not read) is refused and left as it is. - public override void PushAll() + public override void PushAll(bool soft) { ThrowIfReadOnly(); Held(() => @@ -4669,6 +4814,8 @@ namespace GameCult.Caching using (AcquireLock(wait: true)) { var disk = ReadSnapshot(); + if (soft && disk != null) + return; var (upserts, removals) = StagedWrites(Array.Empty(), Array.Empty()); Wrote(ApplyWriteSet(disk, upserts, removals, Array.Empty())); } @@ -5081,6 +5228,9 @@ namespace GameCult.Caching return ToStoredDocument(record, durable.Version(key), durable.Snapshot.SchemaCatalog, DeserializeMarked, out _); } + // Update's read: the durable record at the key, decoded afresh, stamped with the version CommitBatch compares. + internal override CultStoredDocument? ReadDurable(string key) => Held(() => DurableAt(ReadSnapshot(), key)); + // The file now holds every record written, under the id and at the storedAt the write gave it, and the version of those bytes: // the store's entries and the cache's say so. A copied record is on disk as the file had it, so nothing learns anything new about it: in particular the // ids it minted at load are still in memory only. @@ -5132,7 +5282,7 @@ namespace GameCult.Caching var attributes = FileInfo.Attributes; if ((int)attributes == -1) return null; - if ((attributes & FileAttributes.Directory) != 0 && (attributes & FileAttributes.ReparsePoint) == 0) + if ((attributes & FileAttributes.Directory) != 0 && Directory.Exists(FileInfo.FullName)) throw new IOException("The store path is a directory, not a store file."); try { @@ -5154,25 +5304,27 @@ namespace GameCult.Caching } // Something is at the path even when the path cannot be opened: a link keeps the reparse-point attribute whether or not its - // target is there. A path with nothing at it reports no attributes at all (-1). - private bool PathIsALink() - { - FileInfo.Refresh(); - var attributes = FileInfo.Attributes; - return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0; - } + // target is there. + private bool PathIsALink() => CultStoreLinks.IsLink(FileInfo.FullName); + + // A write never goes through a link at the store path: replacing it would silently detach the store from its link. + private void RefuseLinkedStore() => CultStoreLinks.Refuse(FileInfo.FullName); // The lock is a sidecar opened exclusively, which excludes other handles in this process and in others alike. + // Every write takes the lock first, so the link is refused here twice: before the lock file or anything else is made, and + // again once the lock is held, because a link can be put at the path while this writer waited on another's lock. private FileStream? AcquireLock(bool wait) { + RefuseLinkedStore(); Directory.CreateDirectory(FileInfo.DirectoryName!); var lockPath = FileInfo.FullName + ".lock"; var started = Stopwatch.StartNew(); while (true) { + FileStream held; try { - return new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, bufferSize: 1); + held = new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, bufferSize: 1); } catch (IOException) when (!wait) { @@ -5181,7 +5333,20 @@ namespace GameCult.Caching catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30)) { Thread.Sleep(10); + continue; } + + try + { + RefuseLinkedStore(); + } + catch + { + held.Dispose(); + throw; + } + + return held; } } @@ -5204,14 +5369,7 @@ namespace GameCult.Caching stream.Flush(true); } - if (File.Exists(path)) - { - File.Replace(tempPath, path, null); - } - else - { - File.Move(tempPath, path); - } + StoreReplace.Replace(tempPath, path); } finally { diff --git a/src/GameCult.Caching/CultInspectorModel.cs b/src/GameCult.Caching/CultInspectorModel.cs index 7aa6ab74..60dec974 100644 --- a/src/GameCult.Caching/CultInspectorModel.cs +++ b/src/GameCult.Caching/CultInspectorModel.cs @@ -352,7 +352,7 @@ namespace GameCult.Caching } } - public object Clone(object document, Type type) => _deserialize(type, _serialize(document, type, type)); + public object Clone(object document, Type type) => _codec.Clone(document, type); public CultInspectorEdit BeginEdit(CultStoredDocument record) { diff --git a/src/GameCult.Caching/CultStoreLinkedException.cs b/src/GameCult.Caching/CultStoreLinkedException.cs new file mode 100644 index 00000000..c7272fe0 --- /dev/null +++ b/src/GameCult.Caching/CultStoreLinkedException.cs @@ -0,0 +1,24 @@ +using System.IO; + +namespace GameCult.Caching +{ + /// + /// A write was attempted at a store path that is a link (a symbolic link to a file or a directory, a Windows junction or any + /// other reparse-point link). The write is refused before the lock, the staging file or the replace, and again once the lock + /// is held, so a link that exists when a writer takes the lock leaves the link and its target untouched. A link created by + /// something that ignores the lock after that check, while the commit builds and writes the new image, is still replaced; + /// that window cannot be closed without native no-follow opens. Configure the link's target as the store path to write to + /// it; reads through a live link are unchanged. + /// + public sealed class CultStoreLinkedException : IOException + { + public CultStoreLinkedException(string path) + : base($"CultCache store {path} is a link, and a write at it would replace the link; configure the link's target as the store path.") + { + Path = path; + } + + /// The store path that is a link. + public string Path { get; } + } +} diff --git a/src/GameCult.Caching/CultStoreLinks.cs b/src/GameCult.Caching/CultStoreLinks.cs new file mode 100644 index 00000000..cc9ec2b0 --- /dev/null +++ b/src/GameCult.Caching/CultStoreLinks.cs @@ -0,0 +1,22 @@ +using System.IO; + +namespace GameCult.Caching +{ + // The one decision on whether a store-owned file is a link, shared by every store kind that replaces files. + internal static class CultStoreLinks + { + // A path with nothing at it reports no attributes at all (-1). Any reparse point counts as a link, whether or not its + // target is there. + internal static bool IsLink(string path) + { + var attributes = new FileInfo(path).Attributes; + return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0; + } + + internal static void Refuse(string path) + { + if (IsLink(path)) + throw new CultStoreLinkedException(path); + } + } +} diff --git a/src/GameCult.Caching/CultVariants.cs b/src/GameCult.Caching/CultVariants.cs index 48909c19..e5c28bbf 100644 --- a/src/GameCult.Caching/CultVariants.cs +++ b/src/GameCult.Caching/CultVariants.cs @@ -29,6 +29,11 @@ namespace GameCult.Caching // (document payload, slot and encoded value pairs) public Func>, byte[]> Overlay { get; } + // The one copy of a document: its encoding, decoded again. Update's private argument and CultInspectorModel.Clone share it. + internal object Clone(object document, Type type) => + Deserialize(type, type, Serialize(document, type, type)) + ?? throw new InvalidOperationException($"The codec returned null for {type.FullName}."); + /// /// Decodes a document payload read from a store file, as a cache load decodes it: the ids a load mints are minted from the key /// and their path, so every reader of one store mints the same ids, and a payload a cache write would refuse is refused here too. diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index 6defed61..dffce80a 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -13,6 +13,7 @@ Cacheable models are plain classes annotated with cache intent. using GameCult.Caching; using MessagePack; +[MessagePackObject] [CultDocument("gamecult.item_data", "gamecult.item_data.v1")] public sealed class ItemData { @@ -131,18 +132,33 @@ using GameCult.Caching; using GameCult.Caching.MessagePack; var cache = await CultCacheMessagePack.OpenAsync("Data.msgpack"); +var key = new CultRecordKey("potion"); +// The golden path: read, change a private copy, commit only if the record is unchanged, re-run on a lost race. +// Durable when it returns; no flush. +var result = cache.Update(key, item => +{ + item ??= new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable" }; + item.Value += 10; + return item; +}); + +// The blind opt-out: last writer wins per key. Name the key, or a new key claims the same unique Sku and is refused. var handle = await cache.UpsertAsync(new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable", Value = 50 -}); +}, new CultRecordHandle(key)); await cache.FlushAsync(); ``` +`Update` may run your function more than once, so it must have no side effects. Returning null writes nothing, and a +lost race that outlasts `maxAttempts` comes back as `CultUpdateOutcome.LostRace`. `UpsertAsync` plus `FlushAsync` never +checks what it overwrites. + If you want the explicit assembly rite, the lower-level `CultCache` + `SingleFileMessagePackBackingStore` path is still there. The helper is just the blessed front door. @@ -294,21 +310,29 @@ Useful surfaces: - `cache.IsDirty` - `store.IsDirty` - `cache.Document(key)` -- `cache.Soa()` -- `cache.UpsertAsync(document)` +- `cache.Update(key, apply)` +- `cache.UpsertAsync(document)` (blind write) - `cache.FlushAllBackingStores()` - `cache.FlushAsync()` -- `cache.FlushBackingStore(store)` -- `cache.PrepareForReloadOrShutdown()` -- `cache.PrepareForReloadOrShutdownAsync()` - `cache.FlushAttachedStoresOnDispose` - `store.FlushOnDispose` This is built for callers like Aquarium that want durable runtime state while still choosing *when* to pay the write cost. -Single-file MessagePack remains a single-writer format. The atomic replace path -protects against partial writes, not competing writers trampling each other. +A single-file store admits any number of writers, in one process or many, one at +a time. Every write (flush, `PushAll`, `CommitBatch`, cacheless `WriteRecord`) +takes an exclusive advisory lock on the sidecar `.lock`, reads the file as +it is then, applies its own staged keys onto it and replaces the file by a +write-to-temp-then-rename. Two writers never interleave bytes, and a record +another writer put there survives; a key both staged holds the last write +(`Contracts/cultcache-store-composition.md`, "Locking"). What the lock does not +give is cross-key or cross-store atomicity, a lock that binds a program that +ignores it, or protection on a filesystem without working `flock`. A second +writer waits (30 seconds, then `IOException`; `TryCommit` reports `Contended` +instead of waiting); readers take no lock and always see one complete file. The +operator procedures for a leftover temp file and for backing up two stores are +in the same contract document, "Operating a single-file store". ## Schema migration diagnostics diff --git a/src/GameCult.Caching/StoreReplace.cs b/src/GameCult.Caching/StoreReplace.cs new file mode 100644 index 00000000..5f16322c --- /dev/null +++ b/src/GameCult.Caching/StoreReplace.cs @@ -0,0 +1,182 @@ +using System; +using System.IO; +using System.Runtime.InteropServices; +using System.Threading; + +namespace GameCult.Caching +{ + /// The kernel32 calls a Windows replace needs, each answering 0 or a Win32 error code. + internal interface IReplaceKernel + { + /// CreateFileW(staged, DELETE|SYNCHRONIZE), FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, CloseHandle. + int RenamePosix(string staged, string destination); + + /// MoveFileExW(staged, destination, flags). + int MoveFileEx(string staged, string destination, uint flags); + + void Sleep(int milliseconds); + } + + /// + /// Owns how a staged file becomes the store, for the single-file and directory stores. Windows tries a POSIX-semantics + /// rename first (which makes a moment with the path absent rare for a lock-free reader, not impossible: a stalled rename was + /// measured leaving it absent for 5-47 ms), falls back to write-through MoveFileExW where that is unsupported or does not + /// resolve, falls back to the portable File.Replace if MoveFileExW itself does not resolve, and retries transient sharing + /// errors; other OSes keep rename. Mirrors cultcache-ts store-file.ts + /// replaceStoreFile (ruling variants:ruling:windows-replace-posix-first). + /// + internal static class StoreReplace + { + private const int ErrorAccessDenied = 5; + private const int ErrorSharingViolation = 32; + private const int ErrorLockViolation = 33; + private const int ErrorNotSupported = 50; + private const int ErrorInvalidParameter = 87; + private const int ReplaceAttempts = 25; + private const int ReplaceRetryMs = 20; + + /// MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH. + internal const uint MoveFileReplaceWriteThrough = 0x9; + + internal static readonly IReplaceKernel WindowsKernel = new Kernel32Replace(); + + public static void Replace(string staged, string destination) + { + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + ReplaceWindows(staged, destination, WindowsKernel); + } + else + { + ReplacePortable(staged, destination); + } + } + + private static void ReplacePortable(string staged, string destination) + { + if (File.Exists(destination)) + { + File.Replace(staged, destination, null); + } + else + { + File.Move(staged, destination); + } + } + + private static bool Unresolved(Exception ex) => ex is DllNotFoundException || ex is EntryPointNotFoundException; + + internal static void ReplaceWindows(string staged, string destination, IReplaceKernel kernel) + { + var posix = true; + var lastCode = 0; + for (var attempt = 1; attempt <= ReplaceAttempts; attempt++) + { + int code; + try + { + code = posix ? RenamePosixOrUnsupported(kernel, staged, destination) : kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough); + if (posix && (code == 1 || code == ErrorNotSupported || code == ErrorInvalidParameter)) + { + // POSIX semantics are unsupported here (FAT, exFAT, before Windows 10 1607): the rest of this call is write-through. + posix = false; + code = kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough); + } + } + catch (Exception ex) when (Unresolved(ex)) + { + // kernel32's MoveFileExW does not resolve here (a sandboxed player): the runtime-owned replace is all that is left. + ReplacePortable(staged, destination); + return; + } + + if (code == 0) + { + return; + } + + if (code != ErrorAccessDenied && code != ErrorSharingViolation && code != ErrorLockViolation) + { + throw new IOException($"CultCache could not replace the store: Win32 error {code}."); + } + + lastCode = code; + kernel.Sleep(ReplaceRetryMs); + } + + throw new IOException($"CultCache could not replace the store after {ReplaceAttempts} attempts: last Win32 error {lastCode}."); + } + + private static int RenamePosixOrUnsupported(IReplaceKernel kernel, string staged, string destination) + { + try + { + return kernel.RenamePosix(staged, destination); + } + catch (Exception ex) when (Unresolved(ex)) + { + // The POSIX call does not resolve: the same answer as "unsupported", so the replace falls back to MoveFileExW. + return ErrorNotSupported; + } + } + + private sealed class Kernel32Replace : IReplaceKernel + { + private const uint DeleteSynchronize = 0x00110000; + private const uint ShareAll = 0x7; + private const uint OpenExisting = 3; + private const uint FileAttributeNormal = 0x80; + private const int FileRenameInfoEx = 22; + private const uint RenameReplacePosix = 0x3; + + public int RenamePosix(string staged, string destination) + { + var handle = CreateFileW(staged, DeleteSynchronize, ShareAll, IntPtr.Zero, OpenExisting, FileAttributeNormal, IntPtr.Zero); + if (handle == new IntPtr(-1)) + { + return Marshal.GetLastWin32Error(); + } + + try + { + // FILE_RENAME_INFO: Flags u32 at 0, RootDirectory handle, FileNameLength u32, FileName (UTF-16, offsets for this pointer size). + var name = System.Text.Encoding.Unicode.GetBytes(Path.GetFullPath(destination)); + var lengthOffset = 2 * IntPtr.Size; + var nameOffset = lengthOffset + 4; + var size = nameOffset + name.Length + 2; + var info = Marshal.AllocHGlobal(size); + try + { + for (var i = 0; i < size; i++) Marshal.WriteByte(info, i, 0); + Marshal.WriteInt32(info, 0, (int)RenameReplacePosix); + Marshal.WriteInt32(info, lengthOffset, name.Length); + Marshal.Copy(name, 0, info + nameOffset, name.Length); + return SetFileInformationByHandle(handle, FileRenameInfoEx, info, (uint)size) ? 0 : Marshal.GetLastWin32Error(); + } + finally { Marshal.FreeHGlobal(info); } + } + finally { CloseHandle(handle); } + } + + public int MoveFileEx(string staged, string destination, uint flags) + => MoveFileExW(staged, destination, flags) ? 0 : Marshal.GetLastWin32Error(); + + public void Sleep(int milliseconds) => Thread.Sleep(milliseconds); + + [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern IntPtr CreateFileW(string fileName, uint access, uint share, IntPtr security, uint disposition, uint flags, IntPtr template); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool SetFileInformationByHandle(IntPtr handle, int informationClass, IntPtr information, uint size); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool CloseHandle(IntPtr handle); + + [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool MoveFileExW(string existing, string target, uint flags); + } + } +} diff --git a/src/GameCult.Geometry/GameCult.Geometry.csproj b/src/GameCult.Geometry/GameCult.Geometry.csproj index 9736e1b7..744bd109 100644 --- a/src/GameCult.Geometry/GameCult.Geometry.csproj +++ b/src/GameCult.Geometry/GameCult.Geometry.csproj @@ -12,7 +12,7 @@ - + diff --git a/src/GameCult.Mesh/CultMesh.cs b/src/GameCult.Mesh/CultMesh.cs index e3f23430..32b45e1f 100644 --- a/src/GameCult.Mesh/CultMesh.cs +++ b/src/GameCult.Mesh/CultMesh.cs @@ -428,11 +428,14 @@ namespace GameCult.Mesh } /// - /// Flushes the durable cache. + /// Flushes the durable cache. With the flush has one meaning, decided by the store's own + /// read and never by what the cache holds in memory: a store with nothing at its path is written; a store that reads + /// (empty included) is skipped and its staged writes stay staged; a store that cannot be read refuses with + /// and is left untouched. /// public Task FlushAsync(bool soft = false) { - return _host.FlushAsync(); + return _host.FlushAsync(soft); } /// diff --git a/src/GameCult.Networking/CultNetDatabase.cs b/src/GameCult.Networking/CultNetDatabase.cs index 9db3cd1b..abd04d01 100644 --- a/src/GameCult.Networking/CultNetDatabase.cs +++ b/src/GameCult.Networking/CultNetDatabase.cs @@ -1055,6 +1055,13 @@ namespace GameCult.Networking var key = new CultRecordKey(message.Document.RecordKey); var descriptor = _documents.ResolveDescriptorForRawDocument(message.Document); authorize?.Invoke(descriptor.SchemaId, key); + // A peer's put is judged for its stamp after its authority and before any shard is asked (so it is never forwarded): + // the cache would mint a stamp for one that is no stamp, and a refused put stores and mints nothing. + if (authorize != null && !CultNetStoredAtStamp.IsStamp(message.Document.StoredAt)) + { + throw new CultNetStoredAtStampException(); + } + var shard = ResolveShardInternal(descriptor, key); EnsurePrimary(shard, descriptor.SchemaId, key, message.ShardEpoch); var document = _documents.DeserializeRawDocument(message.Document); diff --git a/src/GameCult.Networking/CultNetDatabaseServer.cs b/src/GameCult.Networking/CultNetDatabaseServer.cs index 32f302fc..573c8f0f 100644 --- a/src/GameCult.Networking/CultNetDatabaseServer.cs +++ b/src/GameCult.Networking/CultNetDatabaseServer.cs @@ -475,6 +475,11 @@ namespace GameCult.Networking _server.Logger.LogWarning($"CultNet raw put refused: {ex.Message}"); Refuse(peer, write, CultNetErrorMessage.ForRemoteWriteRefused(ex.Request, ex.Reason)); } + catch (CultNetStoredAtStampException ex) + { + _server.Logger.LogWarning($"CultNet raw put refused: {ex.Message}"); + Refuse(peer, write, CultNetErrorMessage.ForMessageInvalid("document", "document.storedAt is not a stamp")); + } catch (CultNetShardAuthorityException ex) { bool forwarded; diff --git a/src/GameCult.Networking/CultNetLocal.cs b/src/GameCult.Networking/CultNetLocal.cs index 87942ce8..60937574 100644 --- a/src/GameCult.Networking/CultNetLocal.cs +++ b/src/GameCult.Networking/CultNetLocal.cs @@ -96,11 +96,14 @@ namespace GameCult.Networking } /// - /// Flushes the durable cache. + /// Flushes the durable cache. With the flush has one meaning, decided by the store's own + /// read and never by what the cache holds in memory: a store with nothing at its path is written; a store that reads + /// (empty included) is skipped and its staged writes stay staged; a store that cannot be read refuses with + /// and is left untouched. /// public Task FlushAsync(bool soft = false) { - return Cache.FlushAsync(); + return Cache.FlushAsync(soft); } /// diff --git a/src/GameCult.Networking/CultNetStoredAtStamp.cs b/src/GameCult.Networking/CultNetStoredAtStamp.cs new file mode 100644 index 00000000..a4ead645 --- /dev/null +++ b/src/GameCult.Networking/CultNetStoredAtStamp.cs @@ -0,0 +1,63 @@ +#nullable enable +using System; + +namespace GameCult.Networking +{ + // The one shape a peer's storedAt must have: RFC 3339 date-time, YYYY-MM-DDThh:mm:ss, an optional fraction of 1 to 9 digits, + // then Z or +hh:mm / -hh:mm. Uppercase T and Z, ASCII digits, no whitespace, a real calendar date, seconds 00-59, years + // 0001-9999 for both the local date and the UTC instant, offset hours 00-23 and minutes 00-59. Checked by shape and range, never by DateTimeOffset.TryParse, which is + // culture-aware and takes far more than this (tests/vectors/stored-at-stamp, variants:ruling:raw-put-without-stamp-refused). + internal static class CultNetStoredAtStamp + { + public static bool IsStamp(string? text) + { + if (text == null || text.Length < 20) return false; + if (!Digits(text, 0, 4, out var year) || text[4] != '-' || !Digits(text, 5, 2, out var month) || text[7] != '-' + || !Digits(text, 8, 2, out var day) || text[10] != 'T' || !Digits(text, 11, 2, out var hour) || text[13] != ':' + || !Digits(text, 14, 2, out var minute) || text[16] != ':' || !Digits(text, 17, 2, out var second)) + return false; + if (year < 1 || month is < 1 or > 12 || day < 1 || day > DateTime.DaysInMonth(year, month) + || hour > 23 || minute > 59 || second > 59) + return false; + + var i = 19; + if (text[i] == '.') + { + var start = ++i; + while (i < text.Length && IsDigit(text[i])) i++; + if (i - start is < 1 or > 9) return false; + } + + if (i >= text.Length) return false; + if (text[i] == 'Z') return i == text.Length - 1; + if (!(text.Length - i == 6 && (text[i] == '+' || text[i] == '-') + && Digits(text, i + 1, 2, out var offsetHour) && text[i + 3] == ':' && Digits(text, i + 4, 2, out var offsetMinute) + && offsetHour <= 23 && offsetMinute <= 59)) + return false; + // The UTC instant must lie in years 0001-9999 too: local minus offset may not leave DateTime's range. + var offsetTicks = (offsetHour * 60L + offsetMinute) * TimeSpan.TicksPerMinute * (text[i] == '-' ? -1 : 1); + var utcTicks = new DateTime(year, month, day, hour, minute, second).Ticks - offsetTicks; + return utcTicks >= DateTime.MinValue.Ticks && utcTicks <= DateTime.MaxValue.Ticks; + } + + private static bool IsDigit(char c) => c is >= '0' and <= '9'; + + private static bool Digits(string text, int start, int count, out int value) + { + value = 0; + for (var i = start; i < start + count; i++) + { + if (i >= text.Length || !IsDigit(text[i])) return false; + value = value * 10 + (text[i] - '0'); + } + + return true; + } + } + + // A peer's raw put whose storedAt is no stamp: refused message_invalid, field document, nothing stored or minted. + internal sealed class CultNetStoredAtStampException : Exception + { + public CultNetStoredAtStampException() : base("a raw put document must carry a storedAt stamp") { } + } +} diff --git a/src/GameCult.Networking/GameCult.Networking.csproj b/src/GameCult.Networking/GameCult.Networking.csproj index 08ef2ed6..68e13794 100644 --- a/src/GameCult.Networking/GameCult.Networking.csproj +++ b/src/GameCult.Networking/GameCult.Networking.csproj @@ -9,7 +9,7 @@ - + diff --git a/tests/GameCult.Caching.ConsumerBuild/Program.cs b/tests/GameCult.Caching.ConsumerBuild/Program.cs index 60376068..f09edb8c 100644 --- a/tests/GameCult.Caching.ConsumerBuild/Program.cs +++ b/tests/GameCult.Caching.ConsumerBuild/Program.cs @@ -56,7 +56,7 @@ public sealed class OutsideStore : CacheBackingStore public override void Delete(CultStoredDocument entry) { } - public override void PushAll() { } + public override void PushAll(bool soft) { } public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => request.ConditionsHold(new Dictionary { ["w"] = "v1" }, Entries.Values, ForeignRecords) diff --git a/tests/GameCult.Caching.InteropPeer/Program.cs b/tests/GameCult.Caching.InteropPeer/Program.cs index 3c8eb92b..f0f0479e 100644 --- a/tests/GameCult.Caching.InteropPeer/Program.cs +++ b/tests/GameCult.Caching.InteropPeer/Program.cs @@ -51,6 +51,12 @@ static async Task ProgramMainAsync(string[] args) case "edit-in-place": await EditInPlaceAsync(file, RequireArg(options, "kind")); return 0; + case "write-shapes": + await WriteShapesAsync(file); + return 0; + case "read-shapes": + await ReadShapesAsync(file); + return 0; case "hold-lock": HoldLock(file, int.Parse(RequireArg(options, "ms"), CultureInfo.InvariantCulture)); return 0; @@ -96,6 +102,49 @@ static async Task WriteAsync(string file, string runtimeId) WriteJsonLine(note); } +// Records a reader of a Bifrost-shaped store must open: a union field (two arms) and nested bytes (a bin member, a bin list). +static async Task WriteShapesAsync(string file) +{ + var cache = BuildCache(file); + await cache.PullAllBackingStoresAsync(); + foreach (var shape in InteropShapes()) + await cache.AddAsync(shape, new CultRecordHandle(new CultRecordKey(shape.DocumentId))); + cache.FlushAllBackingStores(); + WriteShapesJsonLine(InteropShapes()); +} + +static async Task ReadShapesAsync(string file) +{ + var cache = BuildCache(file); + await cache.PullAllBackingStoresAsync(); + WriteShapesJsonLine(cache.AllEntries.OfType()); +} + +static CultCacheInteropShape[] InteropShapes() => +[ + new() { DocumentId = "shape:text", Arm = new InteropTextArm { Text = "union arm 0" }, Raw = Enumerable.Range(0, 256).Select(i => (byte)i).ToArray() }, + new() + { + DocumentId = "shape:pack", + Arm = new InteropPackArm { Label = "union arm 1", Chunks = [new byte[] { 0xc0, 0xc1, 0x00 }, [], [0x92, 0x01, 0x02]] }, + Raw = [0xde, 0xad, 0xbe, 0xef] + } +]; + +// One line, ordered by document id, in the form the Rust peer prints. +static void WriteShapesJsonLine(IEnumerable shapes) => + Console.Out.WriteLine(JsonSerializer.Serialize(shapes.OrderBy(shape => shape.DocumentId, StringComparer.Ordinal).Select(shape => new + { + documentId = shape.DocumentId, + arm = shape.Arm switch + { + InteropTextArm text => (object)new { kind = "text", text = text.Text }, + InteropPackArm pack => new { kind = "pack", label = pack.Label, chunks = pack.Chunks.Select(Convert.ToHexString).ToArray() }, + _ => throw new InvalidOperationException("Unknown arm.") + }, + raw = Convert.ToHexString(shape.Raw) + }))); + static async Task ReadAsync(string file) { var cache = BuildCache(file); @@ -328,3 +377,28 @@ public sealed class CultCacheInteropMark [Key(0)] public string Label { get; set; } = string.Empty; [Key(1)] [CultElementId] public string Id { get; set; } = string.Empty; } + +[CultDocument("cultcache.interop-shape", "cultcache.interop_shape.v1")] +[MessagePackObject] +public sealed class CultCacheInteropShape +{ + [Key(0)] [CultName] public string DocumentId { get; set; } = string.Empty; + [Key(1)] public InteropArm Arm { get; set; } = new InteropTextArm(); + [Key(2)] public byte[] Raw { get; set; } = Array.Empty(); +} + +[Union(0, typeof(InteropTextArm)), Union(1, typeof(InteropPackArm))] +public abstract class InteropArm { } + +[MessagePackObject] +public sealed class InteropTextArm : InteropArm +{ + [Key(0)] public string Text { get; set; } = string.Empty; +} + +[MessagePackObject] +public sealed class InteropPackArm : InteropArm +{ + [Key(0)] public string Label { get; set; } = string.Empty; + [Key(1)] public List Chunks { get; set; } = new(); +} diff --git a/tests/GameCult.Caching.OutsideStoreTests/HeldInstanceUpdateTests.cs b/tests/GameCult.Caching.OutsideStoreTests/HeldInstanceUpdateTests.cs new file mode 100644 index 00000000..6b02b72f --- /dev/null +++ b/tests/GameCult.Caching.OutsideStoreTests/HeldInstanceUpdateTests.cs @@ -0,0 +1,131 @@ +using System; +using System.Collections.Generic; +using System.IO; +using GameCult.Caching.MessagePack; +using MessagePack; +using NUnit.Framework; + +namespace GameCult.Caching.OutsideStoreTests +{ + // variants:follow_up:update-copy-from-stored-bytes: Update takes its version and its copy from one durable read of the home + // store, never from the held instance. Consumer side: public API only, so a change made in place to an instance the cache + // hands out can never be committed as if it had been read. + public class HeldInstanceUpdateTests + { + [CultDocument("tests.held_counter", "tests.held_counter.v1")] + [MessagePackObject] + public sealed class HeldCounter + { + [Key(0)] [CultName] public string Name { get; set; } = string.Empty; + [Key(1)] public int Value { get; set; } + } + + private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(HeldCounter) }); + private static readonly CultRecordKey Key = new("counter"); + + private string _directory = string.Empty; + + [SetUp] + public void CreateDirectory() + { + _directory = Path.Combine(Path.GetTempPath(), $"cultlib-held-update-{Guid.NewGuid():N}"); + Directory.CreateDirectory(_directory); + } + + [TearDown] + public void DeleteDirectory() + { + if (Directory.Exists(_directory)) + Directory.Delete(_directory, recursive: true); + } + + private static CultCache Open(string path, bool directory) + { + var cache = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)); + cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path)); + cache.PullAllBackingStoresAsync().GetAwaiter().GetResult(); + return cache; + } + + private string Seed(bool directory, int value) + { + var path = Path.Combine(_directory, "held.cc"); + using var cache = Open(path, directory); + Assert.That(cache.Commit(batch => batch.Upsert(new HeldCounter { Name = "counter", Value = value }, new CultRecordHandle(Key))), Is.True); + return path; + } + + private static int StoredValue(string path, bool directory) + { + using var reader = Open(path, directory); + return reader.Get(Key)!.Value; + } + + private static HeldCounter Plus(HeldCounter? counter, int by) + { + counter!.Value += by; + return counter; + } + + [TestCase(false)] + [TestCase(true)] + public void InPlaceChangeToHeldInstanceIsNeitherAppliedNorPersisted(bool directory) + { + var path = Seed(directory, 1); + using var cache = Open(path, directory); + cache.Get(Key)!.Value = 999; + var seen = new List(); + + var result = cache.Update(Key, counter => { seen.Add(counter!.Value); return Plus(counter, 1); }); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(seen, Is.EqualTo(new[] { 1 }), "apply saw the in-place change"); + Assert.That(result.Document!.Value, Is.EqualTo(2)); + Assert.That(StoredValue(path, directory), Is.EqualTo(2), "the in-place 999 was persisted"); + } + + [TestCase(false)] + [TestCase(true)] + public void InPlaceChangeToAPreviousResultIsNotCommitted(bool directory) + { + var path = Seed(directory, 1); + using var cache = Open(path, directory); + var first = cache.Update(Key, counter => Plus(counter, 1)); + Assert.That(first.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + first.Document!.Value = 999; + var seen = new List(); + + var second = cache.Update(Key, counter => { seen.Add(counter!.Value); return Plus(counter, 1); }); + + Assert.That(second.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(seen, Is.EqualTo(new[] { 2 }), "apply ran more than once or saw the in-place change"); + Assert.That(StoredValue(path, directory), Is.EqualTo(3)); + } + + [TestCase(false)] + [TestCase(true)] + public void DeclineAfterInPlaceChangeWritesNothing(bool directory) + { + var path = Seed(directory, 1); + using var cache = Open(path, directory); + cache.Get(Key)!.Value = 999; + var before = Snapshot(path); + + Assert.That(cache.Update(Key, _ => null).Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); + + Assert.That(Snapshot(path), Is.EqualTo(before)); + Assert.That(StoredValue(path, directory), Is.EqualTo(1)); + } + + // Every file of the store: the single file, or the manifest and its pages. + private static Dictionary Snapshot(string path) + { + var files = new Dictionary { [path] = File.ReadAllBytes(path) }; + var pages = path + ".records"; + if (Directory.Exists(pages)) + foreach (var file in Directory.GetFiles(pages, "*", SearchOption.AllDirectories)) + files[file] = File.ReadAllBytes(file); + return files; + } + } +} diff --git a/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs b/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs index e47e4d5f..86949fa0 100644 --- a/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs +++ b/tests/GameCult.Caching.OutsideStoreTests/OutsideStoreJudgeTests.cs @@ -136,7 +136,7 @@ namespace GameCult.Caching.OutsideStoreTests IsDirty = true; }); - public override void PushAll() => Held(() => Apply(Array.Empty(), Array.Empty(), Array.Empty())); + public override void PushAll(bool soft) => Held(() => Apply(Array.Empty(), Array.Empty(), Array.Empty())); public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => Held(() => { @@ -364,5 +364,18 @@ namespace GameCult.Caching.OutsideStoreTests Assert.That(() => cache.Commit(batch => batch.Remove(K)), Throws.InstanceOf(), "removal"); Assert.That(disk.Records["k"], Is.EqualTo(before)); } + + // Only the library can stamp a version on a decoded record, so a store kind outside it cannot read one honestly: Update refuses before apply. + [Test] + public void UpdateOverAnOutsideStoreIsNotSupported() + { + var disk = new Disk(); + PutOnDisk(disk, AB, "k", new OutA { Name = "k", Value = 1 }); + using var cache = Open(disk, AB); + var calls = 0; + + Assert.That(() => cache.Update(K, a => { calls++; return a; }), Throws.TypeOf()); + Assert.That(calls, Is.EqualTo(0), "apply ran for a store that cannot serve Update"); + } } } diff --git a/tests/GameCult.Caching.Tests/BackingStoreTests.cs b/tests/GameCult.Caching.Tests/BackingStoreTests.cs index 7e88c283..8d5c3424 100644 --- a/tests/GameCult.Caching.Tests/BackingStoreTests.cs +++ b/tests/GameCult.Caching.Tests/BackingStoreTests.cs @@ -56,6 +56,137 @@ namespace GameCult.Caching.Tests } } + [Test] + public async Task SoftFlush_Over_Readable_Store_Leaves_It_Byte_For_Byte_And_Keeps_Staged_Writes_Dirty() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); + try + { + var store = new SingleFileMessagePackBackingStore(filePath); + var cache = new CultCache(); + cache.AddBackingStore(store); + await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); + await cache.FlushAsync(); + var before = File.ReadAllBytes(filePath); + + var staged = await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" }); + await cache.FlushAsync(soft: true); + + Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft flush rewrote a store it reads"); + Assert.That(store.IsDirty, Is.True, "a skipped soft flush discarded a staged write"); + + await cache.FlushAsync(); + var reader = new CultCache(); + reader.AddBackingStore(new SingleFileMessagePackBackingStore(filePath)); + await reader.PullAllBackingStoresAsync(); + Assert.That(reader.Get(staged.Key)!.Value, Is.EqualTo("two")); + } + finally + { + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task SoftFlush_Over_Readable_Store_Holding_No_Records_Leaves_It() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); + try + { + var store = new SingleFileMessagePackBackingStore(filePath); + var cache = new CultCache(); + cache.AddBackingStore(store); + var first = await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); + await cache.FlushAsync(); + cache.Remove(first.Key); + await cache.FlushAsync(); + var before = File.ReadAllBytes(filePath); + var empty = new CultCache(); + empty.AddBackingStore(new SingleFileMessagePackBackingStore(filePath)); + Assert.That(empty.AllEntries, Is.Empty, "the store is not readable-empty"); + + await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" }); + await cache.FlushAsync(soft: true); + + Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft flush wrote over a readable store that holds no records"); + Assert.That(store.IsDirty, Is.True); + } + finally + { + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task PlainPushAll_Over_Readable_Store_Writes_The_Staged_Records() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); + try + { + var store = new SingleFileMessagePackBackingStore(filePath); + var cache = new CultCache(); + cache.AddBackingStore(store); + await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); + store.PushAll(); + var before = File.ReadAllBytes(filePath); + + await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" }); + store.PushAll(); + + Assert.That(File.ReadAllBytes(filePath), Is.Not.EqualTo(before), "a plain PushAll skipped a store it reads"); + Assert.That(store.IsDirty, Is.False); + } + finally + { + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task SoftFlush_Over_Absent_Path_Writes() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); + try + { + var store = new SingleFileMessagePackBackingStore(filePath); + var cache = new CultCache(); + cache.AddBackingStore(store); + await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); + + await cache.FlushAsync(soft: true); + + Assert.That(File.Exists(filePath), Is.True); + Assert.That(store.IsDirty, Is.False); + } + finally + { + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task SoftFlush_Over_Unreadable_File_Refuses_And_Leaves_It() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack"); + try + { + var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; + var store = new SingleFileMessagePackBackingStore(filePath); + var cache = new CultCache(); + cache.AddBackingStore(store); + await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" }); + File.WriteAllBytes(filePath, junk); + + Assert.That(() => store.PushAll(soft: true), Throws.InstanceOf()); + Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(junk)); + Assert.That(store.IsDirty, Is.True); + } + finally + { + if (File.Exists(filePath)) File.Delete(filePath); + } + } + [Test] public async Task SingleFileMessagePackBackingStore_PullAll_SeesFileCreatedAfterReaderOpened() { diff --git a/tests/GameCult.Caching.Tests/BytesAfterVectorTests.cs b/tests/GameCult.Caching.Tests/BytesAfterVectorTests.cs index 86d677e6..65b1d6da 100644 --- a/tests/GameCult.Caching.Tests/BytesAfterVectorTests.cs +++ b/tests/GameCult.Caching.Tests/BytesAfterVectorTests.cs @@ -110,6 +110,15 @@ namespace GameCult.Caching.Tests Assert.That(message, Does.Contain(typeof(UnionSurface).FullName).And.Contain($"1 {Clause}").And.Contain("refused on read")); } + // The inspector reads documents through CultCodec.Deserialize (type == document), not DecodeDocument: its branch is its own path. + [Test] + public void CodecDeserializeOfADocumentRefusesATail() + { + var codec = CultCacheMessagePack.CreateCodec(Registry); + var message = Message(() => codec.Deserialize(typeof(UnionSurface), typeof(UnionSurface), Base.Concat(new byte[] { 0xc0 }).ToArray())); + Assert.That(message, Does.Contain(typeof(UnionSurface).FullName).And.Contain($"1 {Clause}").And.Contain("refused on read")); + } + [Test] public void StoreLoadRefuses() { diff --git a/tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs b/tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs new file mode 100644 index 00000000..7c2cd5a8 --- /dev/null +++ b/tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs @@ -0,0 +1,115 @@ +#nullable enable +using System; +using System.Buffers; +using System.IO; +using System.Linq; +using GameCult.Caching; +using GameCult.Caching.MessagePack; +using MessagePack; +using NUnit.Framework; + +namespace GameCult.Caching.Tests +{ + // tests/vectors/content-version: the content version of a stored record, the lowercase hex SHA-256 of its persisted encoding + // (variants:ruling:write-set-content-version-declared-ownership), as the reference reads it from a single-file store. The C# + // reference is the only writer: under CULTCACHE_WRITE_VECTORS=1 the store is written and the versions read from it are written + // beside it; every other run opens the committed store.cc and compares what the reference reads to versions.msgpack. Another + // runtime computes the version from the same file and must equal these rows. + public class ContentVersionVectorTests + { + private const string Vector = "content-version"; + + private static readonly Type Item = + EmittedDocumentTypes.Emit("CvItem", "tests.cv_item", "tests.cv_item.v1", new[] { new EmittedDocumentTypes.Field("Text", typeof(string), 0) }); + + private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { Item }); + + private static readonly (string Key, string Text)[] Records = { ("alpha", "first"), ("beta", "second") }; + + private static byte[] StoreBytes() + { + var directory = Path.Combine(Path.GetTempPath(), $"cultlib-cvv-{Guid.NewGuid():N}"); + Directory.CreateDirectory(directory); + try + { + var path = Path.Combine(directory, "store.cc"); + using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, StoreFlushOnDispose = true })) + { + cache.Clock = () => VectorFiles.StoredAt; + foreach (var (key, text) in Records) + cache.UpsertAsync(Item, EmittedDocumentTypes.New(Item, ("Text", text)), new CultRecordKey(key)).GetAwaiter().GetResult(); + cache.FlushAsync().GetAwaiter().GetResult(); + } + + return File.ReadAllBytes(path); + } + finally + { + Directory.Delete(directory, recursive: true); + } + } + + // What the reference reads from a store file: [key, version] for each record, sorted by key. + private static (string Key, string Version)[] ReadVersions(byte[] store) + { + var directory = Path.Combine(Path.GetTempPath(), $"cultlib-cvr-{Guid.NewGuid():N}"); + Directory.CreateDirectory(directory); + try + { + var path = Path.Combine(directory, "store.cc"); + File.WriteAllBytes(path, store); + using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); + return cache.AllStoredDocuments + .Select(stored => (Key: stored.Key.Value, Version: stored.StoredVersion)) + .OrderBy(row => row.Key, StringComparer.Ordinal) + .ToArray(); + } + finally + { + Directory.Delete(directory, recursive: true); + } + } + + private static byte[] RowsFile((string Key, string Version)[] rows) + { + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + writer.WriteArrayHeader(rows.Length); + foreach (var (key, version) in rows) + { + writer.WriteArrayHeader(2); + writer.Write(key); + writer.Write(version); + } + + writer.Flush(); + return buffer.WrittenSpan.ToArray(); + } + + [Test] + public void TheReferenceReadsTheCommittedVersions() + { + if (VectorFiles.Writing) + { + var store = StoreBytes(); + VectorFiles.Check(Vector, "store.cc", store); + VectorFiles.Check(Vector, "versions.msgpack", RowsFile(ReadVersions(store))); + return; + } + + var committed = VectorFiles.Committed(Vector, "store.cc"); + var rows = ReadVersions(committed); + Assert.That(rows.Select(row => row.Key), Is.EqualTo(Records.Select(record => record.Key)), "the store holds the two records"); + Assert.That(rows.Select(row => row.Version).Distinct().Count(), Is.EqualTo(rows.Length), "two different records, two versions"); + Assert.That(rows.All(row => row.Version.Length == 64 && row.Version == row.Version.ToLowerInvariant()), Is.True, "lowercase hex SHA-256"); + Assert.That(Convert.ToHexString(RowsFile(rows)), Is.EqualTo(Convert.ToHexString(VectorFiles.Committed(Vector, "versions.msgpack"))), "versions.msgpack"); + } + + [Test] + public void TheCommittedStoreIsWhatTheReferenceWrites() + { + if (VectorFiles.Writing) return; + Assert.That(Convert.ToHexString(StoreBytes()), Is.EqualTo(Convert.ToHexString(VectorFiles.Committed(Vector, "store.cc"))), "store.cc"); + } + } +} diff --git a/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs b/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs index 404a75cb..f4009a6c 100644 --- a/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs +++ b/tests/GameCult.Caching.Tests/CultCacheJournalTests.cs @@ -338,7 +338,7 @@ namespace GameCult.Caching.Tests public override void Push(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { } - public override void PushAll() { } + public override void PushAll(bool soft) { } public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; } diff --git a/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs b/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs index b2c8e7f5..92861214 100644 --- a/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs +++ b/tests/GameCult.Caching.Tests/CultCacheObserverTests.cs @@ -180,7 +180,7 @@ namespace GameCult.Caching.Tests public override void Push(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { } - public override void PushAll() { } + public override void PushAll(bool soft) { } public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; } diff --git a/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs b/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs index 840110bc..942ae891 100644 --- a/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs +++ b/tests/GameCult.Caching.Tests/DirectoryStoreDurabilityTests.cs @@ -208,6 +208,72 @@ namespace GameCult.Caching.Tests Assert.That(reader.AllEntries, Is.Empty); } + [Test] + public async Task SoftFlushOverAbsentManifestWrites() + { + using var cache = Open(out var store); + var key = await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); + + cache.FlushAllBackingStores(soft: true); + + Assert.That(File.Exists(_manifest), Is.True); + Assert.That(store.IsDirty, Is.False); + using var reader = Open(out _); + Assert.That(reader.Get(key)?.Text, Is.EqualTo("one")); + } + + [Test] + public async Task SoftFlushOverReadableManifestLeavesItAndKeepsStagedWritesDirty() + { + using var cache = Open(out var store); + await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); + cache.FlushAllBackingStores(); + var before = File.ReadAllBytes(_manifest); + var pages = Directory.GetFiles(_records, "*.msgpack").OrderBy(path => path).ToArray(); + var second = await cache.UpsertAsync(typeof(Page), new Page { Name = "b", Text = "two" }); + + cache.FlushAllBackingStores(soft: true); + + Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(before), "a soft flush rewrote a manifest it reads"); + Assert.That(Directory.GetFiles(_records, "*.msgpack").OrderBy(path => path).ToArray(), Is.EqualTo(pages)); + Assert.That(store.IsDirty, Is.True, "a skipped soft flush discarded a staged write"); + + cache.FlushAllBackingStores(); + using var reader = Open(out _); + Assert.That(reader.Get(second)?.Text, Is.EqualTo("two")); + } + + [Test] + public async Task SoftFlushOverReadableEmptyManifestLeavesIt() + { + using var cache = Open(out var store); + var key = await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); + cache.FlushAllBackingStores(); + cache.Remove(key); + cache.FlushAllBackingStores(); + var before = File.ReadAllBytes(_manifest); + using (var reader = Open(out _)) Assert.That(reader.AllEntries, Is.Empty, "the store is not readable-empty"); + await cache.UpsertAsync(typeof(Page), new Page { Name = "b", Text = "two" }); + + cache.FlushAllBackingStores(soft: true); + + Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(before), "a soft flush wrote over a readable store that holds no records"); + Assert.That(store.IsDirty, Is.True); + } + + [Test] + public async Task SoftFlushOverUnreadableManifestRefusesAndLeavesIt() + { + using var cache = Open(out var store); + await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" }); + var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; + File.WriteAllBytes(_manifest, junk); + + Assert.That(() => store.PushAll(soft: true), Throws.Exception); + Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(junk)); + Assert.That(store.IsDirty, Is.True); + } + private CultCache Open(out DirectoryMessagePackBackingStore store) { var cache = new CultCache(Registry); diff --git a/tests/GameCult.Caching.Tests/ElementIdTests.cs b/tests/GameCult.Caching.Tests/ElementIdTests.cs index a6c68c94..d40ffe25 100644 --- a/tests/GameCult.Caching.Tests/ElementIdTests.cs +++ b/tests/GameCult.Caching.Tests/ElementIdTests.cs @@ -224,6 +224,28 @@ namespace GameCult.Caching.Tests Assert.That(AllIds(reread), Is.EqualTo(first), "the ids minted at load are the ids written, and stay through repeated writes"); } + // Update reads the store's bytes, which hold no ids; it mints the ids the load minted, so the write keeps the ids the cache showed. + [Test] + public void UpdateOfAPreIdRecordPersistsTheIdsTheLoadMinted() + { + var path = PathOf("old.cc"); + WritePreIdStore(path, null, ("a", "a", 2)); + var key = new CultRecordKey("a"); + + string[] shown; + using (var cache = Open(path)) + { + shown = AllIds(cache.Get(key)!); + Assert.That(shown, Has.All.Not.Empty); + var result = cache.Update(key, deck => { deck!.Name = "a2"; return deck; }); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + } + + var written = MessagePackSerializer.Deserialize(DiskRecord(path, "a").Payload); + Assert.That(written.Name, Is.EqualTo("a2")); + Assert.That(AllIds(written), Is.EqualTo(shown), "the write replaced the ids the cache showed with random ones"); + } + [Test] public void MintElementIdsRewritesEveryPreIdRecordAndNothingElse() { diff --git a/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs new file mode 100644 index 00000000..bd6cb903 --- /dev/null +++ b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs @@ -0,0 +1,151 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.Serialization; +using GameCult.Caching; +using MessagePack; +using NUnit.Framework; + +namespace GameCult.Caching.Tests +{ + // The id walk's own contracts, one mutant-sized fact at a time: the order of the id path, a union that names itself, a list of + // values never counting as an id, and the attribute predicates that decide whether a type is walked or written whole. + public class ElementIdWalkTests + { + // The codec asks MessagePack about the id path in order and refuses at the first type it cannot allow, so the order is what + // an error names first: the document, then the collection types from the outermost in, then the element and its arms. + [Test] + public void TheIdPathListsCollectionsFromTheOutermostIn() + { + Assert.That(CultElementIds.Inspect(typeof(MsMapMap)).IdPath, Is.EqualTo(new[] + { + typeof(MsMapMap), typeof(Dictionary>>), + typeof(Dictionary>), typeof(List), typeof(ElementIdEngineTests.EIded) + }), "maps nested in maps"); + Assert.That(CultElementIds.Inspect(typeof(MsListDeep)).IdPath, Is.EqualTo(new[] + { + typeof(MsListDeep), typeof(List>>), typeof(List>), + typeof(List), typeof(ElementIdEngineTests.EIded) + }), "lists nested in lists"); + } + + // ConcreteTypes stops at a type it has already expanded, so a union that lists its own base is finished, not a stack overflow. + [Test] + public void AUnionThatNamesItselfIsWalkedOnce() + { + var survey = CultElementIds.Inspect(typeof(MsSelfHolder)); + Assert.That(survey.Problems, Is.Empty); + Assert.That(survey.IdPath, Is.EquivalentTo(new[] { typeof(MsSelfHolder), typeof(List), typeof(MsSelfBase), typeof(MsSelfArm) })); + } + + // Value lists hold no ids however they are nested, and a root that is itself a value list holds none either. + [Test] + public void ListsOfValuesHoldNoIds() + { + var values = new MsValues { Numbers = { 1 }, Tags = { "a" }, Grid = { new List { 1 } }, Map = { ["k"] = new List { 2 } } }; + Assert.That(CultElementIds.Holds(typeof(MsValues), values), Is.False); + Assert.That(CultElementIds.Holds(typeof(List), new List { 1, 2 }), Is.False); + Assert.That(CultElementIds.Holds(typeof(List>), new List> { new() { 1 } }), Is.False); + Assert.That(CultElementIds.Holds(typeof(List), new List { "a" }), Is.False); + Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated"); + } + + // A nullable struct is walked as the struct it wraps; a nullable value is not walked at all. + [Test] + public void ANullableKeyedStructWalksAsItsOwner() + { + Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct)), Is.True); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct?)), Is.True); + Assert.That(CultElementIds.WalksAsOwner(typeof(int?)), Is.False); + Assert.That(CultElementIds.WalksAsOwner(typeof(int)), Is.False); + } + + // [Key] is read through an override, so an element whose id property overrides a keyed base member is still an id with a key. + [Test] + public void AKeyOnTheBaseOfAnOverrideCountsForTheOverride() + { + Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).Problems, Is.Empty); + Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).IdPath, Does.Contain(typeof(MsOverrideEl))); + } + + // A type is walked by key when it is [MessagePackObject] or [Union], read through its bases for the former; it is one whole + // value when it declares its own formatter, is [DataContract] without being [MessagePackObject], or is keyed by name. + [Test] + public void KeyedTypesWalkAndWholeTypesDoNot() + { + Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpBase)), Is.True, "[MessagePackObject]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsUBase)), Is.True, "[Union]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpDerived)), Is.True, "[MessagePackObject] inherited"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsPlain)), Is.False, "[Key] members without [MessagePackObject] or [Union]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsUArm)), Is.True, "an arm that is itself [MessagePackObject]"); + + Assert.That(CultElementIds.WalksAsOwner(typeof(MsDc)), Is.False, "[DataContract] alone"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcMp)), Is.True, "[DataContract] beside [MessagePackObject]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcOverMpBase)), Is.True, "[DataContract] over an inherited [MessagePackObject]"); + + Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropName)), Is.False, "KeyAsPropertyName"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropNameDerived)), Is.False, "KeyAsPropertyName inherited"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsStringKey)), Is.False, "a string [Key]"); + + Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedBase)), Is.False, "[MessagePackFormatter]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedDerived)), Is.False, "[MessagePackFormatter] inherited"); + } + + [MessagePackObject] public class MsMapMap { [Key(0)] public Dictionary>> M { get; set; } = new(); } + [MessagePackObject] public class MsListDeep { [Key(0)] public List>> G { get; set; } = new(); } + + [Union(0, typeof(MsSelfBase)), Union(1, typeof(MsSelfArm))] public abstract class MsSelfBase { } + [MessagePackObject] public class MsSelfArm : MsSelfBase { [Key(0)] [CultElementId] public string Id { get; set; } = ""; } + [MessagePackObject] public class MsSelfHolder { [Key(0)] public List Items { get; set; } = new(); } + + [MessagePackObject] + public class MsValues + { + [Key(0)] public List Numbers { get; set; } = new(); + [Key(1)] public List Tags { get; set; } = new(); + [Key(2)] public List> Grid { get; set; } = new(); + [Key(3)] public Dictionary> Map { get; set; } = new(); + } + + public sealed class MsBoom : IEnumerable + { + public IEnumerator GetEnumerator() => throw new InvalidOperationException("enumerated"); + System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator(); + } + + [MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; } + + [MessagePackObject] + public class MsOverrideBase + { + [Key(0)] [CultElementId] public virtual string Id { get; set; } = ""; + } + + [MessagePackObject] + public class MsOverrideEl : MsOverrideBase + { + public override string Id { get; set; } = ""; + } + + [MessagePackObject] public class MsOverrideHolder { [Key(0)] public List Items { get; set; } = new(); } + + [MessagePackObject] public class MsMpBase { [Key(0)] public int A { get; set; } } + public class MsMpDerived : MsMpBase { } + public class MsPlain { [Key(0)] public int A { get; set; } } + [Union(0, typeof(MsUArm))] public abstract class MsUBase { } + [MessagePackObject] public class MsUArm : MsUBase { [Key(0)] public int A { get; set; } } + + [DataContract] public class MsDc { [DataMember] public int A { get; set; } } + [DataContract] [MessagePackObject] public class MsDcMp { [Key(0)] public int A { get; set; } } + [DataContract] public class MsDcOverMpBase : MsMpBase { } + + [MessagePackObject(keyAsPropertyName: true)] public class MsPropName { public int A { get; set; } } + public class MsPropNameDerived : MsPropName { } + [MessagePackObject] public class MsStringKey { [Key("a")] public int A { get; set; } } + + public sealed class MsNoFormatter { } + [MessagePackFormatter(typeof(MsNoFormatter))] [MessagePackObject] public class MsFormattedBase { [Key(0)] public int A { get; set; } } + [MessagePackObject] public class MsFormattedDerived : MsFormattedBase { [Key(1)] public int B { get; set; } } + } +} diff --git a/tests/GameCult.Caching.Tests/FirstCallTests.cs b/tests/GameCult.Caching.Tests/FirstCallTests.cs index 8fad1229..d8febc49 100644 --- a/tests/GameCult.Caching.Tests/FirstCallTests.cs +++ b/tests/GameCult.Caching.Tests/FirstCallTests.cs @@ -62,17 +62,28 @@ namespace GameCult.Caching.Tests } [Test] - public void DocsShowTheTestedSnippet() - { - var source = File.ReadAllText(SourcePath()); - const string begin = "// snippet:begin\n"; - var start = source.IndexOf(begin, StringComparison.Ordinal) + begin.Length; - var snippet = source[start..source.IndexOf("// snippet:end\n", StringComparison.Ordinal)]; - var docs = File.ReadAllText(Path.Combine(Path.GetDirectoryName(SourcePath())!, "..", "..", "docs", "start", "README.md")); - Assert.That(docs.Replace("\r\n", "\n"), Does.Contain("```csharp\n" + snippet + "```"), - "docs/start/README.md no longer shows tests/GameCult.Caching.Tests/FirstCallTests.cs verbatim"); - } + public void DocsShowTheTestedSnippet() => StartDocs.AssertShowsSnippet(SourcePath()); static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; } + + // docs/start/README.md shows each tested snippet verbatim: the region between snippet:begin and snippet:end of its source file. + internal static class StartDocs + { + public static void AssertShowsSnippet(string sourcePath) => + AssertShowsSnippet(sourcePath, Path.Combine("..", "..", "docs", "start", "README.md")); + + // The same check against another document, its path relative to the directory of sourcePath. The snippet there is the + // statements of a block, so it need not open the fence (the block's usings come first). + public static void AssertShowsSnippet(string sourcePath, string docsRelativePath, bool wholeBlock = true) + { + var source = File.ReadAllText(sourcePath).Replace("\r\n", "\n"); + const string begin = "// snippet:begin\n"; + var start = source.IndexOf(begin, StringComparison.Ordinal) + begin.Length; + var snippet = source[start..source.IndexOf("// snippet:end\n", StringComparison.Ordinal)]; + var docs = File.ReadAllText(Path.Combine(Path.GetDirectoryName(sourcePath)!, docsRelativePath)); + Assert.That(docs.Replace("\r\n", "\n"), Does.Contain(wholeBlock ? "```csharp\n" + snippet + "```" : snippet), + $"{docsRelativePath} no longer shows {Path.GetFileName(sourcePath)} verbatim"); + } + } } diff --git a/tests/GameCult.Caching.Tests/ReadmeHappyPathTests.cs b/tests/GameCult.Caching.Tests/ReadmeHappyPathTests.cs new file mode 100644 index 00000000..de2dabb4 --- /dev/null +++ b/tests/GameCult.Caching.Tests/ReadmeHappyPathTests.cs @@ -0,0 +1,81 @@ +#nullable enable +using System; +using System.IO; +using System.Threading.Tasks; +using GameCult.Caching; +using GameCult.Caching.MessagePack; +using MessagePack; +using NUnit.Framework; + +namespace GameCult.Caching.Tests +{ + // The Happy Path of src/GameCult.Caching/README.md, run as written: its statements are the region between snippet:begin and + // snippet:end below, and ReadmeShowsTheTestedSnippet fails when the README stops showing them verbatim. + public class ReadmeHappyPathTests + { + [CultDocument("gamecult.item_data", "gamecult.item_data.v1")] + [MessagePackObject] + public sealed class ItemData + { + [Key(0)] [CultName] public string Name = string.Empty; + [Key(1)] [CultIndex] public string Sku = string.Empty; + [Key(2)] public string Category = string.Empty; + [Key(3)] public int Value; + } + + private static async Task HappyPath() + { +// snippet:begin +var cache = await CultCacheMessagePack.OpenAsync("Data.msgpack"); +var key = new CultRecordKey("potion"); + +// The golden path: read, change a private copy, commit only if the record is unchanged, re-run on a lost race. +// Durable when it returns; no flush. +var result = cache.Update(key, item => +{ + item ??= new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable" }; + item.Value += 10; + return item; +}); + +// The blind opt-out: last writer wins per key. Name the key, or a new key claims the same unique Sku and is refused. +var handle = await cache.UpsertAsync(new ItemData +{ + Name = "Potion", + Sku = "POT-1", + Category = "Consumable", + Value = 50 +}, new CultRecordHandle(key)); + +await cache.FlushAsync(); +// snippet:end + return cache; + } + + [Test] + public async Task HappyPathRunsAsWritten() + { + var directory = Path.Combine(Path.GetTempPath(), $"readme-happy-{Guid.NewGuid():N}"); + Directory.CreateDirectory(directory); + var previous = Environment.CurrentDirectory; + try + { + Environment.CurrentDirectory = directory; // the README opens a relative path + using var cache = await HappyPath(); + Assert.That(cache.Get(new CultRecordKey("potion"))!.Value, Is.EqualTo(50)); + Assert.That(cache.GetByIndex("Sku", "POT-1"), Is.Not.Null); + } + finally + { + Environment.CurrentDirectory = previous; + Directory.Delete(directory, recursive: true); + } + } + + [Test] + public void ReadmeShowsTheTestedSnippet() => + StartDocs.AssertShowsSnippet(SourcePath(), Path.Combine("..", "..", "src", "GameCult.Caching", "README.md"), wholeBlock: false); + + static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; + } +} diff --git a/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs b/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs new file mode 100644 index 00000000..64536b77 --- /dev/null +++ b/tests/GameCult.Caching.Tests/StoreLinkRefusalTests.cs @@ -0,0 +1,350 @@ +#nullable enable +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using GameCult.Caching.MessagePack; +using NUnit.Framework; +using static GameCult.Caching.Tests.ElementIdTests; +using static GameCult.Caching.Tests.StoreReadabilityTests; + +namespace GameCult.Caching.Tests +{ + // A write never replaces a link at a store-owned path. The refusal holds for every way a write reaches a store (waiting or + // not, as a judge of another store's commit), for a link put there while the write waited on the store's lock, and for the + // directory store's manifest. + public class StoreLinkRefusalTests + { + private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(IdDeck), typeof(VectorItem) }); + private string _directory = ""; + + [SetUp] + public void SetUp() + { + _directory = Path.Combine(Path.GetTempPath(), "cultlib-link-refusal-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(_directory); + } + + [TearDown] + public void TearDown() + { + if (!Directory.Exists(_directory)) + return; + foreach (var entry in Directory.EnumerateFileSystemEntries(_directory).Where(IsALink)) + File.Delete(entry); + Directory.Delete(_directory, true); + } + + private string PathOf(string name) => Path.Combine(_directory, name); + + private static bool IsALink(string path) + { + var attributes = new FileInfo(path).Attributes; + return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0; + } + + private static void Link(string path, string target) + { + try + { + File.CreateSymbolicLink(path, target); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Assert.Ignore("A file symlink needs a privilege this account lacks: " + ex.Message); + } + } + + private static void SeedSingle(string path) + { + using var seed = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); + seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed"))); + } + + private static void SeedItem(string path) + { + using var seed = new CultCache(Registry); + seed.AddBackingStore(new SingleFileMessagePackBackingStore(path), typeof(VectorItem)); + seed.Commit(batch => batch.Upsert(typeof(VectorItem), new VectorItem(), new CultRecordKey("item"))); + } + + // Runs `write` while another holder has `lockPath`; `swap` runs while it waits, then the lock is released. + private static Exception? WriteWhileWaiting(string lockPath, Action write, Action swap) + { + using var held = new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None); + var task = Task.Run(write); + Assert.That(task.Wait(TimeSpan.FromMilliseconds(600)), Is.False, "the lock is held, so the write must wait"); + swap(); + held.Dispose(); + try + { + Assert.That(task.Wait(TimeSpan.FromSeconds(10)), Is.True); + return null; + } + catch (AggregateException ex) + { + return ex.GetBaseException(); + } + } + + private static void ReplaceWithLink(string path, string target) + { + File.Delete(path); + Link(path, target); + } + + // ---- a link put at the store path while the write waited on the lock ---- + + [Test] + public void ACommitRefusesALinkPutAtTheStoreWhileItWaitedOnTheLock() + { + var path = PathOf("store.cc"); + var target = PathOf("shared.cc"); + SeedSingle(path); + SeedSingle(target); + var before = File.ReadAllBytes(target); + using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + + var error = WriteWhileWaiting(path + ".lock", + () => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late"))), + () => ReplaceWithLink(path, target)); + + Assert.That(error, Is.InstanceOf(), error?.ToString()); + Assert.That(((CultStoreLinkedException)error!).Path, Is.EqualTo(path)); + Assert.That(IsALink(path), "the link was replaced"); + Assert.That(File.ReadAllBytes(target), Is.EqualTo(before)); + } + + [Test] + public void AFlushRefusesALinkPutAtTheStoreWhileItWaitedOnTheLock() + { + var path = PathOf("store.cc"); + var target = PathOf("shared.cc"); + SeedSingle(path); + SeedSingle(target); + var before = File.ReadAllBytes(target); + using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late")).GetAwaiter().GetResult(); + + var error = WriteWhileWaiting(path + ".lock", () => cache.FlushAllBackingStores(), () => ReplaceWithLink(path, target)); + + Assert.That(error, Is.InstanceOf(), error?.ToString()); + Assert.That(IsALink(path), "the link was replaced"); + Assert.That(File.ReadAllBytes(target), Is.EqualTo(before)); + } + + // ---- the non-waiting and judging write paths ---- + + [Test] + public void ATryCommitAtALiveLinkIsRefusedNotContendedOrReplaced() + { + var path = PathOf("store.cc"); + var link = PathOf("link.cc"); + SeedSingle(path); + Link(link, path); + var before = File.ReadAllBytes(path); + using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + + var refusal = Assert.Throws(() => + cache.TryCommit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("t"))))!; + + Assert.That(refusal.Path, Is.EqualTo(link)); + Assert.That(refusal.Message, Does.Contain(link)); + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); + Assert.That(File.Exists(link + ".lock"), Is.False, "the refused write took the lock"); + } + + [Test] + public void ATryCommitAtALiveLinkWhoseLockIsHeldIsStillRefused() + { + var path = PathOf("store.cc"); + var link = PathOf("link.cc"); + SeedSingle(path); + Link(link, path); + using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + using var held = new FileStream(link + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None); + + Assert.Throws(() => + cache.TryCommit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("t")))); + Assert.That(IsALink(link), "the link was replaced"); + } + + [Test] + public void AnAdvisoryQueryOverACacheAtALiveLinkAnswersForTheRecordAndTheWriteIsStillRefused() + { + var path = PathOf("store.cc"); + var link = PathOf("link.cc"); + SeedSingle(path); + Link(link, path); + var before = File.ReadAllBytes(path); + using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + + Assert.That(cache.IsWritable(new CultRecordKey("seed")), Is.True, "the query does not look at the store path"); + Assert.That(cache.IsWritable(new CultRecordKey("fresh")), Is.True, "the query does not look at the store path"); + var refusal = Assert.Throws(() => + cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("fresh"))))!; + + Assert.That(refusal.Path, Is.EqualTo(link)); + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); + } + + // A batch that upserts into one store and removes a key the cache does not hold has the other store judge that removal + // under its own lock (JudgeChecks). + private CultCache OpenTwoStores(string homePath, string otherPath) + { + var cache = new CultCache(Registry); + cache.AddBackingStore(new SingleFileMessagePackBackingStore(homePath), typeof(IdDeck)); + cache.AddBackingStore(new SingleFileMessagePackBackingStore(otherPath), typeof(VectorItem)); + return cache; + } + + [Test] + public void AnotherStoreAtALiveLinkRefusesTheChecksItIsAskedToJudge() + { + var other = PathOf("other.cc"); + var link = PathOf("other-link.cc"); + SeedItem(other); + Link(link, other); + var before = File.ReadAllBytes(other); + using var cache = OpenTwoStores(PathOf("home.cc"), link); + + var refusal = Assert.Throws(() => cache.Commit(batch => + { + batch.Upsert(typeof(IdDeck), new IdDeck { Name = "home" }, new CultRecordKey("home")); + batch.Remove(new CultRecordKey("ghost")); + }))!; + + Assert.That(refusal.Path, Is.EqualTo(link)); + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(other), Is.EqualTo(before)); + Assert.That(File.Exists(PathOf("home.cc")), Is.False, "the home store was written though a judge refused"); + } + + [Test] + public void AJudgeReadsTheFileItHoldsUnderTheLockNotTheFileBeforeItWaited() + { + var other = PathOf("other.cc"); + SeedItem(other); + using var cache = OpenTwoStores(PathOf("home.cc"), other); + + var error = WriteWhileWaiting(other + ".lock", + () => cache.Commit(batch => + { + batch.Upsert(typeof(IdDeck), new IdDeck { Name = "home" }, new CultRecordKey("home")); + batch.Remove(new CultRecordKey("ghost")); + }), + () => File.WriteAllBytes(other, Array.Empty())); + + Assert.That(error, Is.InstanceOf(), error?.ToString()); + } + + // ---- the directory store's manifest ---- + + private DirectoryMessagePackBackingStore SeedDirectory(string manifest) + { + using (var seed = new CultCache(Registry)) + { + seed.AddBackingStore(new DirectoryMessagePackBackingStore(manifest), typeof(IdDeck)); + seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed"))); + } + + return new DirectoryMessagePackBackingStore(manifest); + } + + private CultCache OpenDirectory(DirectoryMessagePackBackingStore store) + { + var cache = new CultCache(Registry); + cache.AddBackingStore(store, typeof(IdDeck)); + return cache; + } + + [Test] + public void ADirectoryStoreCommitAtALiveManifestLinkIsRefusedAndNothingChanges() + { + var manifest = PathOf("real.cc"); + var link = PathOf("link.cc"); + SeedDirectory(manifest); + Link(link, manifest); + Directory.Move(manifest + ".records", link + ".records"); + var before = File.ReadAllBytes(manifest); + var pages = Directory.GetFileSystemEntries(link + ".records").OrderBy(entry => entry).ToArray(); + using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link)); + + var refusal = Assert.Throws(() => + cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))))!; + + Assert.That(refusal.Path, Is.EqualTo(link)); + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(manifest), Is.EqualTo(before)); + Assert.That(Directory.GetFileSystemEntries(link + ".records").OrderBy(entry => entry), Is.EqualTo(pages), "the refused write left files beside the manifest"); + } + + // The refusal comes before the records folder or the lease file is made: a store that was never written leaves nothing behind. + [Test] + public void ADirectoryStoreCommitAtAManifestLinkPutThereAfterOpenCreatesNothing() + { + var link = PathOf("link.cc"); + using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link)); + Link(link, PathOf("unmounted.cc")); + + Assert.Throws(() => + cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b")))); + + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(Directory.Exists(link + ".records"), Is.False, "the refused write made the records folder or its lease"); + } + + [Test] + public void ADirectoryStoreFlushAtALiveManifestLinkIsRefused() + { + var manifest = PathOf("real.cc"); + var link = PathOf("link.cc"); + SeedDirectory(manifest); + Link(link, manifest); + Directory.Move(manifest + ".records", link + ".records"); + var before = File.ReadAllBytes(manifest); + using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link)); + cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "f" }, new CultRecordKey("f")).GetAwaiter().GetResult(); + + Assert.Throws(() => cache.FlushAllBackingStores()); + + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(manifest), Is.EqualTo(before)); + } + + [Test] + public void ADirectoryStoreCommitRefusesAManifestLinkPutThereWhileItWaitedOnTheLease() + { + var manifest = PathOf("store.cc"); + var target = PathOf("shared.cc"); + SeedDirectory(manifest); + SeedDirectory(target); + var before = File.ReadAllBytes(target); + using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(manifest)); + + var error = WriteWhileWaiting(Path.Combine(manifest + ".records", ".commit.lock"), + () => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late"))), + () => ReplaceWithLink(manifest, target)); + + Assert.That(error, Is.InstanceOf(), error?.ToString()); + Assert.That(IsALink(manifest), "the link was replaced"); + Assert.That(File.ReadAllBytes(target), Is.EqualTo(before)); + } + + [Test] + public void ADirectoryStoreReadsThroughALiveManifestLink() + { + var manifest = PathOf("real.cc"); + var link = PathOf("link.cc"); + SeedDirectory(manifest); + Link(link, manifest); + Directory.Move(manifest + ".records", link + ".records"); + + using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link)); + + Assert.That(cache.Get(new CultRecordKey("seed"))?.Name, Is.EqualTo("seed")); + } + } +} diff --git a/tests/GameCult.Caching.Tests/StoreReadabilityTests.cs b/tests/GameCult.Caching.Tests/StoreReadabilityTests.cs index f923dbfb..b244dc5f 100644 --- a/tests/GameCult.Caching.Tests/StoreReadabilityTests.cs +++ b/tests/GameCult.Caching.Tests/StoreReadabilityTests.cs @@ -266,7 +266,7 @@ namespace GameCult.Caching.Tests using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); CreateDanglingDirectoryLink(path, Path.Combine(_directory, "unmounted")); - Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf()); + Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf()); Assert.That(IsALink(path), "the link was replaced"); Assert.That(Directory.Exists(Path.Combine(_directory, "unmounted")), Is.False, "a write went through the link"); } @@ -292,11 +292,66 @@ namespace GameCult.Caching.Tests using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); File.Delete(target); - Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf()); + Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf()); Assert.That(new FileInfo(path).LinkTarget, Is.Not.Null, "the link was replaced"); Assert.That(File.Exists(target), Is.False, "a write went through the link"); } + // A write at a store path that is a link is refused with a typed error before the lock or the replace: the link, its target + // and the directory beside them are as they were. + [Test] + public void ACacheCommitAtALiveLinkIsRefusedAndNothingChanges() + { + var target = Path.Combine(_directory, "target.cc"); + var link = Path.Combine(_directory, "link.cc"); + using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry })) + seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a"))); + CreateFileLinkOrIgnore(link, target); + var before = File.ReadAllBytes(target); + using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + + var refusal = Assert.Throws(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))))!; + + Assert.That(refusal.Message, Does.Contain(link)); + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(target), Is.EqualTo(before)); + Assert.That(File.Exists(link + ".lock"), Is.False, "the refused write took the lock"); + Assert.That(Directory.GetFileSystemEntries(_directory).Where(entry => entry.EndsWith(".tmp")), Is.Empty); + } + + [Test] + public void AFlushAtALiveLinkIsRefusedAndNothingChanges() + { + var target = Path.Combine(_directory, "target.cc"); + var link = Path.Combine(_directory, "link.cc"); + using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry })) + seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a"))); + CreateFileLinkOrIgnore(link, target); + var before = File.ReadAllBytes(target); + using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b")).GetAwaiter().GetResult(); + + Assert.Throws(() => cache.FlushAllBackingStores()); + + Assert.That(IsALink(link), "the link was replaced"); + Assert.That(File.ReadAllBytes(target), Is.EqualTo(before)); + Assert.That(File.Exists(link + ".lock"), Is.False); + } + + [Test] + public void AnOpenAtALiveLinkReadsTheTargetsRecords() + { + var target = Path.Combine(_directory, "target.cc"); + var link = Path.Combine(_directory, "link.cc"); + using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry })) + seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a"))); + CreateFileLinkOrIgnore(link, target); + + using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + + Assert.That(cache.Get(new CultRecordKey("a"))?.Name, Is.EqualTo("a")); + } + // A directory at the path is something at the path: it is an I/O error on open and on commit, never an empty store, and // it is left as it was. [Test] diff --git a/tests/GameCult.Caching.Tests/StoreReplaceTests.cs b/tests/GameCult.Caching.Tests/StoreReplaceTests.cs new file mode 100644 index 00000000..5f71b4a3 --- /dev/null +++ b/tests/GameCult.Caching.Tests/StoreReplaceTests.cs @@ -0,0 +1,268 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.IO; +using System.Runtime.InteropServices; +using System.Threading; +using NUnit.Framework; + +namespace GameCult.Caching.Tests +{ + public class StoreReplaceTests + { + private sealed class FakeKernel : IReplaceKernel + { + private readonly Queue _posix; + private readonly Queue _moveEx; + public readonly List Log = new List(); + public readonly List Sleeps = new List(); + public Exception? PosixThrows; + public Exception? MoveExThrows; + + public FakeKernel(int[] posix, int[] moveEx) + { + _posix = new Queue(posix); + _moveEx = new Queue(moveEx); + } + + public int RenamePosix(string staged, string destination) + { + Log.Add("posix"); + if (PosixThrows != null) throw PosixThrows; + return _posix.Count > 0 ? _posix.Dequeue() : 0; + } + + public int MoveFileEx(string staged, string destination, uint flags) + { + Log.Add("moveex:" + flags); + if (MoveExThrows != null) throw MoveExThrows; + return _moveEx.Count > 0 ? _moveEx.Dequeue() : 0; + } + + public void Sleep(int milliseconds) => Sleeps.Add(milliseconds); + } + + private static IOException Fails(FakeKernel kernel) + => Assert.Throws(() => StoreReplace.ReplaceWindows("a", "b", kernel))!; + + [Test] + public void APosixRenameThatWorksIsTheWholeReplace() + { + var kernel = new FakeKernel(new[] { 0 }, new int[0]); + StoreReplace.ReplaceWindows("a", "b", kernel); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix" })); + Assert.That(kernel.Sleeps, Is.Empty); + } + + [TestCase(1)] + [TestCase(50)] + [TestCase(87)] + public void WhereAPosixRenameIsUnsupportedTheReplaceFallsBackToWriteThroughMoveFileEx(int unsupported) + { + var kernel = new FakeKernel(new[] { unsupported }, new[] { 0 }); + StoreReplace.ReplaceWindows("a", "b", kernel); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); + } + + [Test] + public void OnceThePosixRenameIsUnsupportedTheRestOfTheCallStaysOnMoveFileEx() + { + var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 0 }); + StoreReplace.ReplaceWindows("a", "b", kernel); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" })); + Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20 })); + } + + [TestCase(5)] + [TestCase(32)] + [TestCase(33)] + public void ATransientErrorRetriesTheSamePrimitiveAfterTwentyMilliseconds(int transient) + { + var kernel = new FakeKernel(new[] { transient, transient, 0 }, new int[0]); + StoreReplace.ReplaceWindows("a", "b", kernel); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "posix", "posix" })); + Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20, 20 })); + } + + [TestCase(1)] + [TestCase(50)] + [TestCase(87)] + public void AnUnsupportedCodeAfterTheFallbackIsAnErrorNotAnotherFallback(int unsupported) + { + var kernel = new FakeKernel(new[] { 87 }, new[] { unsupported }); + var error = Fails(kernel); + Assert.That(error.Message, Does.Contain("Win32 error " + unsupported)); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); + } + + private static IEnumerable Unresolved() + { + yield return new DllNotFoundException("kernel32.dll"); + yield return new EntryPointNotFoundException("SetFileInformationByHandle"); + } + + [TestCaseSource(nameof(Unresolved))] + public void APosixCallThatDoesNotResolveTakesTheUnsupportedFallback(Exception unresolved) + { + var kernel = new FakeKernel(new int[0], new[] { 0 }) { PosixThrows = unresolved }; + StoreReplace.ReplaceWindows("a", "b", kernel); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); + } + + [TestCaseSource(nameof(Unresolved))] + public void AMoveFileExThatDoesNotResolveDegradesToThePortableReplace(Exception unresolved) + { + var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + var destination = Path.Combine(dir, "store.cc"); + var staged = Path.Combine(dir, "staged"); + File.WriteAllBytes(destination, new byte[] { 1 }); + File.WriteAllBytes(staged, new byte[] { 2 }); + var kernel = new FakeKernel(new[] { 87 }, new int[0]) { MoveExThrows = unresolved }; + StoreReplace.ReplaceWindows(staged, destination, kernel); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" })); + Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 })); + Assert.That(File.Exists(staged), Is.False); + + // Both calls unresolved: the same portable replace, here onto an absent destination. + File.Delete(destination); + File.WriteAllBytes(staged, new byte[] { 3 }); + var both = new FakeKernel(new int[0], new int[0]) { PosixThrows = unresolved, MoveExThrows = unresolved }; + StoreReplace.ReplaceWindows(staged, destination, both); + Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 })); + } + finally { Directory.Delete(dir, true); } + } + + [Test] + public void TheFallbackRunsInTheSameAttemptEvenOnTheLastOne() + { + var posix = new int[25].Fill(32); + posix[24] = 87; + var kernel = new FakeKernel(posix, new[] { 0 }); + StoreReplace.ReplaceWindows("a", "b", kernel); + Assert.That(kernel.Log.Count, Is.EqualTo(26)); + Assert.That(kernel.Log[24], Is.EqualTo("posix")); + Assert.That(kernel.Log[25], Is.EqualTo("moveex:9")); + } + + [Test] + public void ARetriedMoveFileExThatAnswersUnsupportedFailsAfterOneCallNotTwo() + { + var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 87 }); + Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 87")); + Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" })); + } + + [Test] + public void ARetryThatNeverSucceedsGivesUpAfterTwentyFiveAttemptsNamingTheCode() + { + var kernel = new FakeKernel(new int[30].Fill(32), new int[0]); + var error = Fails(kernel); + Assert.That(error.Message, Does.Contain("25 attempts").And.Contain("last Win32 error 32")); + Assert.That(kernel.Log.Count, Is.EqualTo(25)); + Assert.That(kernel.Sleeps.Count, Is.EqualTo(25)); + } + + [Test] + public void AnyOtherCodeThrowsAtOnceNamingTheCode() + { + var kernel = new FakeKernel(new[] { 2 }, new int[0]); + var error = Fails(kernel); + Assert.That(error.Message, Does.Contain("Win32 error 2")); + Assert.That(kernel.Log.Count, Is.EqualTo(1)); + Assert.That(kernel.Sleeps, Is.Empty); + } + + [Test] + public void AFallbackThatFailsOtherwiseThrowsItsCode() + { + var kernel = new FakeKernel(new[] { 87 }, new[] { 3 }); + Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 3")); + } + + [Test] + public void ReplaceOverAnExistingAndThenAnAbsentDestinationLeavesTheStagedBytes() + { + var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + var destination = Path.Combine(dir, "store.cc"); + var staged = Path.Combine(dir, "staged"); + File.WriteAllBytes(destination, new byte[] { 1 }); + File.WriteAllBytes(staged, new byte[] { 2 }); + StoreReplace.Replace(staged, destination); + Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 })); + Assert.That(File.Exists(staged), Is.False); + + File.Delete(destination); + File.WriteAllBytes(staged, new byte[] { 3 }); + StoreReplace.Replace(staged, destination); + Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 })); + } + finally { Directory.Delete(dir, true); } + } + + [Test] + public void OnWindowsAReplaceOverADestinationAReaderHoldsOpenSucceedsOnTheFirstAttempt() + { + if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + Assert.Ignore("The real kernel runs on Windows only."); + + var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + var destination = Path.Combine(dir, "store.cc"); + var staged = Path.Combine(dir, "staged"); + File.WriteAllBytes(destination, new byte[] { 1 }); + File.WriteAllBytes(staged, new byte[] { 2 }); + using (new FileStream(destination, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete)) + { + var kernel = new CountingKernel(StoreReplace.WindowsKernel); + StoreReplace.ReplaceWindows(staged, destination, kernel); + Assert.That(kernel.PosixCalls, Is.EqualTo(1)); + Assert.That(kernel.MoveExCalls, Is.EqualTo(0)); + Assert.That(kernel.Sleeps, Is.EqualTo(0)); + Assert.That(kernel.LastCode, Is.EqualTo(0)); + } + + Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 })); + } + finally { Directory.Delete(dir, true); } + } + + private sealed class CountingKernel : IReplaceKernel + { + private readonly IReplaceKernel _inner; + public int PosixCalls, MoveExCalls, Sleeps, LastCode; + public CountingKernel(IReplaceKernel inner) { _inner = inner; } + + public int RenamePosix(string staged, string destination) + { + PosixCalls++; + return LastCode = _inner.RenamePosix(staged, destination); + } + + public int MoveFileEx(string staged, string destination, uint flags) + { + MoveExCalls++; + return LastCode = _inner.MoveFileEx(staged, destination, flags); + } + + public void Sleep(int milliseconds) { Sleeps++; _inner.Sleep(milliseconds); } + } + } + + internal static class ArrayFill + { + public static int[] Fill(this int[] array, int value) + { + for (var i = 0; i < array.Length; i++) array[i] = value; + return array; + } + } +} diff --git a/tests/GameCult.Caching.Tests/StoreRoutingTests.cs b/tests/GameCult.Caching.Tests/StoreRoutingTests.cs index af7e7da7..3ccd7a04 100644 --- a/tests/GameCult.Caching.Tests/StoreRoutingTests.cs +++ b/tests/GameCult.Caching.Tests/StoreRoutingTests.cs @@ -699,10 +699,10 @@ namespace GameCult.Caching.Tests _nested = nested; } - public override void PushAll() + public override void PushAll(bool soft) { _nested(); - base.PushAll(); + base.PushAll(soft); } } diff --git a/tests/GameCult.Caching.Tests/StoreWriteReadsUnderTheLockTests.cs b/tests/GameCult.Caching.Tests/StoreWriteReadsUnderTheLockTests.cs new file mode 100644 index 00000000..971f3478 --- /dev/null +++ b/tests/GameCult.Caching.Tests/StoreWriteReadsUnderTheLockTests.cs @@ -0,0 +1,132 @@ +#nullable enable +using System; +using System.IO; +using System.Threading.Tasks; +using GameCult.Caching.MessagePack; +using NUnit.Framework; +using static GameCult.Caching.Tests.ElementIdTests; + +namespace GameCult.Caching.Tests +{ + // A write decides from the file it holds the lock over, not from the file as it was before it waited: whatever another + // writer left while this one waited for .lock is what the write applies onto, or refuses. + public class StoreWriteReadsUnderTheLockTests + { + private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(IdDeck) }); + private static readonly CultSchemaCatalogEntry One = new() { SchemaId = "tests.under-lock.one", SchemaName = "tests.under-lock.one" }; + private string _directory = ""; + private string _path = ""; + + [SetUp] + public void SetUp() + { + _directory = Path.Combine(Path.GetTempPath(), "cultlib-under-lock-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(_directory); + _path = Path.Combine(_directory, "store.cc"); + } + + [TearDown] + public void TearDown() + { + if (Directory.Exists(_directory)) + Directory.Delete(_directory, true); + } + + private void Seed() + { + using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry }); + cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed"))); + } + + // The write starts and waits on the held lock; while it waits, the file is replaced by what `swap` makes of it. + private Exception? WriteWhileTheFileChangesUnderIt(Action write, Action swap) + { + using var held = new FileStream(_path + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None); + var task = Task.Run(write); + Assert.That(task.Wait(TimeSpan.FromMilliseconds(600)), Is.False, "the lock is held, so the write must wait"); + swap(); + held.Dispose(); + try + { + Assert.That(task.Wait(TimeSpan.FromSeconds(10)), Is.True); + return null; + } + catch (AggregateException ex) + { + return ex.GetBaseException(); + } + } + + private static void ZeroBytes(string path) => File.WriteAllBytes(path, Array.Empty()); + + private static CultPersistedStoreSnapshot Snapshot(string path) => + CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path)); + + private void MarkV3() + { + var snapshot = Snapshot(_path); + snapshot.FormatVersion = CultPersistedStoreSnapshot.FormatV3; + File.WriteAllBytes(_path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot)); + } + + private void CommitWrite(Action swap, out Exception? error) + { + using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + error = WriteWhileTheFileChangesUnderIt( + () => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "committed" }, new CultRecordKey("committed"))), swap); + } + + private void FlushWriteWhile(Action swap, out Exception? error) + { + using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false }); + cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "flushed" }, new CultRecordKey("flushed")).GetAwaiter().GetResult(); + error = WriteWhileTheFileChangesUnderIt(() => cache.FlushAllBackingStores(), swap); + } + + private void CachelessWriteWhile(Action swap, out Exception? error) => + error = WriteWhileTheFileChangesUnderIt( + () => new SingleFileMessagePackBackingStore(_path).WriteRecord( + new CultPersistedRecord { Key = "written", SchemaId = One.SchemaId, StoredAt = "2026-10-22T00:00:00.0000000+00:00", Payload = new byte[] { 1 } }, One), + swap); + + private delegate void WriteWhile(Action swap, out Exception? error); + + [Test] + public void AFlushDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(FlushWriteWhile); + + [Test] + public void ACommitDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(CommitWrite); + + [Test] + public void ACachelessWriteDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(CachelessWriteWhile); + + [Test] + public void AFlushKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(FlushWriteWhile); + + [Test] + public void ACommitKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(CommitWrite); + + [Test] + public void ACachelessWriteKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(CachelessWriteWhile); + + private void RefusesWhatItFindsUnderTheLock(WriteWhile write) + { + Seed(); + + write(() => ZeroBytes(_path), out var error); + + Assert.That(error, Is.InstanceOf()); + Assert.That(new FileInfo(_path).Length, Is.Zero, "the refused file was written over"); + } + + private void KeepsTheV3HeaderItFindsUnderTheLock(WriteWhile write) + { + Seed(); + + write(MarkV3, out var error); + + Assert.That(error, Is.Null, error?.ToString()); + Assert.That(Snapshot(_path).FormatVersion, Is.EqualTo(CultPersistedStoreSnapshot.FormatV3), "the write read the file as it was before the lock"); + } + } +} diff --git a/tests/GameCult.Caching.Tests/UpdateCallTests.cs b/tests/GameCult.Caching.Tests/UpdateCallTests.cs new file mode 100644 index 00000000..d9a8b31e --- /dev/null +++ b/tests/GameCult.Caching.Tests/UpdateCallTests.cs @@ -0,0 +1,54 @@ +#nullable enable +using System; +using System.IO; +using NUnit.Framework; + +// snippet:begin +using GameCult.Caching; +using GameCult.Caching.MessagePack; + +public static class UpdateCall +{ + public static CultUpdateResult Restock(string path, CultRecordKey key) + { + var cache = CultCacheMessagePack.Create(path); + return cache.Update(key, item => + { + item ??= new Item { Name = "Potion", Category = "Consumable" }; + item.Value += 10; + return item; + }); + } +} +// snippet:end + +namespace GameCult.Caching.Tests +{ + // The C# update snippet that docs/start/README.md shows verbatim; it reuses FirstCallTests' Item. + public class UpdateCallTests + { + [Test] + public void RestockTwiceReadsTwenty() + { + var path = Path.Combine(Path.GetTempPath(), $"update-call-{Guid.NewGuid():N}.cc"); + try + { + var key = new CultRecordKey("potion"); + Assert.That(UpdateCall.Restock(path, key).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(UpdateCall.Restock(path, key).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + + using var reader = CultCacheMessagePack.Create(path); + Assert.That(reader.Get(key)!.Value, Is.EqualTo(20)); + } + finally + { + File.Delete(path); + } + } + + [Test] + public void DocsShowTheTestedSnippet() => StartDocs.AssertShowsSnippet(SourcePath()); + + static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path; + } +} diff --git a/tests/GameCult.Caching.Tests/UpdateTests.cs b/tests/GameCult.Caching.Tests/UpdateTests.cs new file mode 100644 index 00000000..63462ba6 --- /dev/null +++ b/tests/GameCult.Caching.Tests/UpdateTests.cs @@ -0,0 +1,803 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using GameCult.Caching; +using GameCult.Caching.MessagePack; +using MessagePack; +using NUnit.Framework; +using R3; + +namespace GameCult.Caching.Tests +{ + // CultCache.Update: read the content version, apply a pure function to a private copy, commit conditionally, re-run on a lost + // race (variants:ruling:write-set-content-version-declared-ownership; variants:question:update-apply-copy). + public class UpdateTests + { + private static readonly CultRecordKey Key = new("counter"); + private static readonly CultDocumentRegistry Registry = + CultDocumentRegistry.ForTypes(new[] { typeof(UpdateCounter), typeof(UpdateTally), typeof(UpdateGear), typeof(UpdateBase), typeof(UpdateDerived) }); + + private string _directory = string.Empty; + + [SetUp] + public void CreateDirectory() + { + _directory = Path.Combine(Path.GetTempPath(), $"cultlib-update-{Guid.NewGuid():N}"); + Directory.CreateDirectory(_directory); + } + + [TearDown] + public void DeleteDirectory() + { + if (Directory.Exists(_directory)) + Directory.Delete(_directory, recursive: true); + } + + private string PathOf(string name) => Path.Combine(_directory, name); + + private static CultCache Open(string path, bool directory, bool codec = true) + { + var cache = codec ? new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)) : new CultCache(Registry); + cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path)); + return cache; + } + + private static void Seed(string path, bool directory, int value) + { + using var cache = Open(path, directory); + Assert.That(cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = value }, new CultRecordHandle(Key))), Is.True); + } + + private static void Rival(CultCache cache, int value) => + Assert.That(cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = value }, new CultRecordHandle(Key))), Is.True); + + private static int StoredValue(string path, bool directory) + { + using var reader = Open(path, directory); + return reader.Get(Key)!.Value; + } + + private static UpdateCounter Plus(UpdateCounter? counter, int by) + { + counter ??= new UpdateCounter { Name = "counter" }; + counter.Value += by; + return counter; + } + + [TestCase(false)] + [TestCase(true)] + public void TwoWritersUpdatingLoseNoIncrement(bool directory) + { + const int n = 100; + var path = PathOf("race.cc"); + Seed(path, directory, 0); + using var a = Open(path, directory); + using var b = Open(path, directory); + var applied = 0; + // Each round starts both writers together: an attempt reads fresh from the store, so only overlapping read-commit windows lose. + var round = new Barrier(2); + + var racers = new[] { a, b }.Select(cache => Task.Run(() => + { + var results = new List(); + for (var i = 0; i < n; i++) + { + round.SignalAndWait(); + results.Add(cache.Update(Key, counter => + { + Interlocked.Increment(ref applied); + return Plus(counter, 1); + }, maxAttempts: 200).Outcome); + } + + return results; + })).ToArray(); + Assert.That(Task.WaitAll(racers, TimeSpan.FromMinutes(2)), Is.True, "the racers did not finish"); + + Assert.That(racers.SelectMany(racer => racer.Result), Is.All.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(StoredValue(path, directory), Is.EqualTo(2 * n)); + Assert.That(applied, Is.GreaterThan(2 * n), "the writers never actually raced"); + } + + [Test] + public void LostAttemptRerunsApplyOnTheFreshValue() + { + var path = PathOf("rerun.cc"); + Seed(path, directory: false, 1); + using var cache = Open(path, false); + using var rival = Open(path, false); + var seen = new List(); + + var result = cache.Update(Key, counter => + { + seen.Add(counter!.Value); + if (seen.Count == 1) + Rival(rival, 100); + return Plus(counter, 1); + }); + + Assert.That(seen, Is.EqualTo(new[] { 1, 100 })); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(result.Document!.Value, Is.EqualTo(101)); + Assert.That(StoredValue(path, false), Is.EqualTo(101)); + } + + [Test] + public void ExhaustedAttemptsReturnLostRace() + { + var path = PathOf("lost.cc"); + Seed(path, false, 1); + using var cache = Open(path, false); + using var rival = Open(path, false); + var calls = 0; + + var result = cache.Update(Key, counter => + { + Rival(rival, 100 * ++calls); + return Plus(counter, 1); + }, maxAttempts: 3); + + Assert.That(calls, Is.EqualTo(3)); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace)); + Assert.That(result.Document!.Value, Is.EqualTo(300)); + Assert.That(StoredValue(path, false), Is.EqualTo(300), "the store holds the rival's last write"); + } + + [Test] + public void ApplyGetsAPrivateCopy() + { + var path = PathOf("copy.cc"); + Seed(path, false, 1); + using var cache = Open(path, false); + using var rival = Open(path, false); + var published = new List(); + using var subscription = cache.Watch().Subscribe(change => published.Add(change.Document!.Value)); + var arguments = new List(); + var calls = 0; + + var result = cache.Update(Key, counter => + { + calls++; + arguments.Add(counter!.Value); + Assert.That(counter, Is.Not.SameAs(cache.Get(Key)), "apply was handed the held instance"); + counter.Value += 1000; + Assert.That(cache.Get(Key)!.Value, Is.Not.EqualTo(counter.Value), "the change reached the cache before any commit"); + if (calls == 1) + Rival(rival, 5); + return counter; + }); + + Assert.That(arguments, Is.EqualTo(new[] { 1, 5 }), "the second call saw the first call's change"); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(published, Does.Not.Contain(1001), "an uncommitted value was published"); + Assert.That(published.Count(value => value == 1005), Is.EqualTo(1)); + Assert.That(StoredValue(path, false), Is.EqualTo(1005)); + } + + // No publication in the middle of an Update: a rival's write that cost this cache an attempt is never published on its own, + // only the value this Update commits (a pull after every lost attempt would publish both). + [TestCase(false)] + [TestCase(true)] + public void LostAttemptPublishesNothingMidUpdate(bool directory) + { + var path = PathOf("midupdate.cc"); + Seed(path, directory, 1); + using var cache = Open(path, directory); + using var rival = Open(path, directory); + var published = new List(); + using var subscription = cache.Watch().Subscribe(change => published.Add(change.Document!.Value)); + var calls = 0; + + var result = cache.Update(Key, counter => + { + if (++calls == 1) + Rival(rival, 5); + return Plus(counter, 1); + }); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(calls, Is.EqualTo(2)); + Assert.That(published, Is.EqualTo(new[] { 6 }), "the rival's 5 was published in the middle of the Update"); + } + + // The directory store reads only the key's own page, so a damaged page of another record does not stop an Update. + [Test] + public void DirectoryUpdateReadsOnlyItsOwnPage() + { + var path = PathOf("pages.cc"); + var other = new CultRecordKey("other"); + using (var seed = Open(path, true)) + Assert.That(seed.Commit(batch => batch.Upsert(new UpdateCounter { Name = "other", Value = 9 }, new CultRecordHandle(other))), Is.True); + var otherPages = Directory.GetFiles(path + ".records", "*.msgpack"); + Assert.That(otherPages, Has.Length.EqualTo(1)); + Seed(path, true, 1); + using var cache = Open(path, true); + File.WriteAllBytes(otherPages[0], new byte[] { 1, 2, 3, 4, 5 }); + + var result = cache.Update(Key, counter => Plus(counter, 1)); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(result.Document!.Value, Is.EqualTo(2)); + } + + // The durable read is not a load: it leaves the store's foreign list, its migration reports and what the cache publishes alone. + [TestCase(false)] + [TestCase(true)] + public void ReadDurableLeavesTheStoreViewAlone(bool directory) + { + var path = PathOf("view.cc"); + Seed(path, directory, 1); + using (var writer = Open(path, directory)) + Assert.That(writer.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 3 }, new CultRecordHandle(new CultRecordKey("gear")))), Is.True); + var narrow = CultDocumentRegistry.ForTypes(new[] { typeof(UpdateCounter) }); + using var cache = new CultCache(narrow); + cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path)); + using var rival = Open(path, directory); + var store = cache.BackingStores.Single(); + var foreign = store.ForeignRecords; + var reports = store.LastSchemaMigrationReports; + Assert.That(foreign.Select(record => record.Key), Is.EqualTo(new[] { "gear" }), "the fixture holds a foreign record"); + Assert.That(reports, Is.Not.Empty, "the fixture holds a migration report"); + var foreignKeys = foreign.Select(record => (record.Key, record.SchemaId, record.StoredAt)).ToArray(); + var reportKinds = reports.Select(report => (report.PersistedSchemaName, report.Kind)).ToArray(); + var changes = 0; + using var subscription = cache.Watch().Subscribe(_ => changes++); + var calls = 0; + + var result = cache.Update(Key, counter => + { + if (++calls == 1) + Rival(rival, 5); + return Plus(counter, 1); + }); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(calls, Is.EqualTo(2), "the Update reread the key"); + Assert.That(store.ForeignRecords, Is.SameAs(foreign)); + Assert.That(store.LastSchemaMigrationReports, Is.SameAs(reports)); + Assert.That(store.ForeignRecords.Select(record => (record.Key, record.SchemaId, record.StoredAt)), Is.EqualTo(foreignKeys)); + Assert.That(store.LastSchemaMigrationReports.Select(report => (report.PersistedSchemaName, report.Kind)), Is.EqualTo(reportKinds)); + Assert.That(changes, Is.EqualTo(1), "only the committed write was published"); + } + + // Routing: store A is home to UpdateBase, store B to UpdateDerived : UpdateBase. Land judges a derived record at B. + private CultCache OpenRouted() + { + var cache = new CultCache(Registry); + cache.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf("a.cc")), typeof(UpdateBase)); + cache.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf("b.cc")), typeof(UpdateDerived)); + return cache; + } + + private CultCache OpenOneRouted(string file, Type home) + { + var cache = new CultCache(Registry); + cache.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf(file)), home); + return cache; + } + + private byte[]? BytesOf(string file) => File.Exists(PathOf(file)) ? File.ReadAllBytes(PathOf(file)) : null; + + private static readonly CultRecordHandle BaseHandle = new(Key); + + private static void RivalDerived(CultCache cache, int value) => + Assert.That(cache.Commit(batch => batch.Upsert(new UpdateDerived { Name = "counter", Value = value }, BaseHandle)), Is.True); + + [Test] + public void RoutedSubtypeStaleCacheConverges() + { + using var cache = OpenRouted(); + using var rival = OpenRouted(); + RivalDerived(rival, 1); + var seen = new List(); + + var result = cache.Update(Key, document => + { + seen.Add((document as UpdateDerived)?.Value); + var derived = (UpdateDerived)document!; + derived.Value += 1; + return derived; + }); + + Assert.That(seen, Is.EqualTo(new int?[] { 1 }), "apply did not see the rival's record in its home store, once"); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + using var reader = OpenRouted(); + Assert.That(((UpdateDerived)reader.Get(Key)!).Value, Is.EqualTo(2)); + } + + [Test] + public void RoutedSubtypeCreateLandsInItsHome() + { + using (var cache = OpenRouted()) + { + var calls = 0; + var result = cache.Update(Key, document => + { + calls++; + Assert.That(document, Is.Null); + return new UpdateDerived { Name = "counter", Value = 1 }; + }); + Assert.That(calls, Is.EqualTo(1)); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + } + + using (var inB = OpenOneRouted("b.cc", typeof(UpdateDerived))) + Assert.That(inB.Get(Key), Is.InstanceOf()); + using (var inA = OpenOneRouted("a.cc", typeof(UpdateBase))) + Assert.That(inA.Get(Key), Is.Null, "the record landed in the wrong store"); + } + + [Test] + public void MovingTheRecordToAnotherHomeRefuses() + { + using (var rival = OpenRouted()) + RivalDerived(rival, 1); + var before = (A: BytesOf("a.cc"), B: BytesOf("b.cc")); + using var cache = OpenRouted(); + var calls = 0; + + var refusal = Assert.Throws(() => + cache.Update(Key, _ => { calls++; return new UpdateBase { Name = "counter" }; }))!; + + Assert.That(calls, Is.EqualTo(1), "a refusal was retried"); + Assert.That(refusal.Message, Does.Contain("a.cc").And.Contain("b.cc").And.Contain("move")); + Assert.That(BytesOf("a.cc"), Is.EqualTo(before.A)); + Assert.That(BytesOf("b.cc"), Is.EqualTo(before.B)); + } + + [Test] + public void ARecordHeldByTwoStoresRefusesBeforeApply() + { + using var cache = OpenRouted(); + using (var inA = OpenOneRouted("a.cc", typeof(UpdateBase))) + Assert.That(inA.Commit(batch => batch.Upsert(new UpdateBase { Name = "counter" }, BaseHandle)), Is.True); + using (var inB = OpenOneRouted("b.cc", typeof(UpdateDerived))) + Assert.That(inB.Commit(batch => batch.Upsert(new UpdateDerived { Name = "counter", Value = 1 }, BaseHandle)), Is.True); + var calls = 0; + + var refusal = Assert.Throws(() => cache.Update(Key, document => { calls++; return document; }))!; + + Assert.That(calls, Is.EqualTo(0)); + Assert.That(refusal.Message, Does.Contain("held by two stores")); + } + + [Test] + public void NullFromApplyDeclines() + { + var path = PathOf("decline.cc"); + Seed(path, false, 7); + var before = File.ReadAllBytes(path); + using var cache = Open(path, false); + var changes = 0; + using var subscription = cache.Watch().Subscribe(_ => changes++); + + var result = cache.Update(Key, _ => null); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); + Assert.That(result.Document!.Value, Is.EqualTo(7)); + Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); + Assert.That(changes, Is.EqualTo(0)); + } + + [TestCase(false)] + [TestCase(true)] + public void AbsentKeyCreatesOnceUnderRace(bool directory) + { + var path = PathOf("create.cc"); + using var a = Open(path, directory); + using var b = Open(path, directory); + var started = new Barrier(2); + + var racers = new[] { a, b }.Select(cache => Task.Run(() => + { + started.SignalAndWait(); + return cache.Update(Key, counter => Plus(counter, 1)).Outcome; + })).ToArray(); + Assert.That(Task.WaitAll(racers, TimeSpan.FromMinutes(1)), Is.True); + + Assert.That(racers.Select(racer => racer.Result), Is.All.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(StoredValue(path, directory), Is.EqualTo(2)); + } + + [Test] + public void RefusalsAreNotRetried() + { + var path = PathOf("refuse.cc"); + Seed(path, false, 1); + var before = File.ReadAllBytes(path); + var calls = 0; + Func apply = counter => { calls++; return Plus(counter, 1); }; + + // A cache with no store copies its held instance, so it needs a codec. + using (var storeLess = new CultCache(Registry)) + { + storeLess.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = 1 }, new CultRecordHandle(Key))); + Assert.That(() => storeLess.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("no codec")); + } + + // Stores are attached, but none is home to the type: refused before apply. + using (var noHome = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry))) + { + noHome.AddBackingStore(new SingleFileMessagePackBackingStore(PathOf("nohome.cc")), typeof(UpdateTally)); + Assert.That(() => noHome.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("names its home store")); + } + using (var cache = Open(path, false)) + { + Assert.That(() => cache.Update(Key, apply, maxAttempts: 0), Throws.TypeOf()); + Assert.That(() => cache.Update(Key, null!), Throws.TypeOf()); + Assert.That(() => cache.Update(Key, _ => new UpdateTally { Name = "counter" }), Throws.InvalidOperationException); + } + + using (var cache = Open(path, false)) + { + cache.Commit(batch => + { + batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle(new CultRecordKey("gear"))); + }); + cache.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"), + new[] { cache.Override(nameof(UpdateGear.Name), "gear big"), cache.Override(nameof(UpdateGear.Power), 9) })); + } + + before = File.ReadAllBytes(path); + var gearCalls = 0; + using (var cache = Open(path, false)) + Assert.That(() => cache.Update(new CultRecordKey("gear-big"), gear => { gearCalls++; return gear; }), Throws.InvalidOperationException.With.Message.Contains("Update cannot copy it")); + Assert.That(gearCalls, Is.EqualTo(0), "apply ran for a variant"); + Assert.That(calls, Is.EqualTo(0), "apply ran for a refused call"); + Assert.That(File.ReadAllBytes(path), Is.EqualTo(before)); + + using (var cache = Open(path, false)) + { + cache.UpsertAsync(new UpdateTally { Name = "staged" }).GetAwaiter().GetResult(); + Assert.That(() => cache.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("flush before a conditional commit")); + Assert.That(calls, Is.LessThanOrEqualTo(1)); + } + + Assert.That(File.ReadAllBytes(path), Is.EqualTo(before), "a refusal wrote"); + } + + [Test] + public void VariantStoreConflictIsALostAttempt() + { + var path = PathOf("variant.cc"); + Seed(path, false, 1); + using (var seed = Open(path, false)) + { + seed.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle(new CultRecordKey("gear")))); + seed.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"), + new[] { seed.Override(nameof(UpdateGear.Name), "gear big"), seed.Override(nameof(UpdateGear.Power), 9) })); + } + + using var cache = Open(path, false); + using (var rival = Open(path, false)) + Assert.That(rival.Commit(batch => batch.Upsert(new UpdateTally { Name = "other" }, new CultRecordHandle(new CultRecordKey("other")))), Is.True); + var calls = 0; + + var result = cache.Update(Key, counter => { calls++; return Plus(counter, 1); }); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(calls, Is.EqualTo(2), "the conflict was not a lost attempt that re-ran apply"); + Assert.That(StoredValue(path, false), Is.EqualTo(2)); + } + + private static CultPersistedStoreSnapshot ReadSnapshot(string path) => + CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path)); + + // A reload that fails inside a conflict is a read failure, not a lost race: Update lets it propagate and never re-runs apply. + // Here the gear a variant is based on has gone foreign in the file, so the reload the conflict owes cannot drop it. + [Test] + public void ConflictWhoseReloadFailedPropagates() + { + var path = PathOf("reload-failed.cc"); + Seed(path, false, 1); + using var cache = Open(path, false); + cache.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle(new CultRecordKey("gear")))); + cache.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"), + new[] { cache.Override(nameof(UpdateGear.Name), "gear big"), cache.Override(nameof(UpdateGear.Power), 9) })); + const string foreignId = "sha256:00000000000000000000000000000000000000000000000000000000000050de"; + var snapshot = ReadSnapshot(path); + snapshot.Records.Single(record => record.Key == "gear").SchemaId = foreignId; + snapshot.SchemaCatalog = snapshot.SchemaCatalog.Append(new CultSchemaCatalogEntry + { + SchemaId = foreignId, SchemaName = "tests.update_foreign", SchemaVersion = "tests.update_foreign.v1", + ContentHash = "foreign", CanonicalSchemaJson = "{}" + }).ToArray(); + File.WriteAllBytes(path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot)); + var calls = 0; + + var conflict = Assert.Throws(() => cache.Update(Key, counter => { calls++; return Plus(counter, 1); }))!; + + Assert.That(conflict.InnerException, Is.Not.Null, "the reload that failed is the cause"); + Assert.That(calls, Is.EqualTo(1), "a failed reload was retried"); + } + + [TestCase(false)] + [TestCase(true)] + public void RivalWritingAndRestoringTheValueStillRerunsApply(bool directory) + { + var path = PathOf("aba.cc"); + Seed(path, directory, 1); + using var cache = Open(path, directory); + using var rival = Open(path, directory); + var seen = new List(); + + var result = cache.Update(Key, counter => + { + seen.Add(counter!.Value); + if (seen.Count == 1) + { + Rival(rival, 5); + Rival(rival, 1); + } + return Plus(counter, 1); + }); + + Assert.That(seen, Is.EqualTo(new[] { 1, 1 }), "the content version missed A-B-A"); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(StoredValue(path, directory), Is.EqualTo(2)); + } + + [TestCase(false)] + [TestCase(true)] + public void RivalRemovalIsSeenAsNull(bool directory) + { + var path = PathOf("removal.cc"); + Seed(path, directory, 1); + using var cache = Open(path, directory); + using var rival = Open(path, directory); + var seen = new List(); + + var result = cache.Update(Key, counter => + { + seen.Add(counter?.Value); + if (seen.Count == 1) + rival.Commit(batch => batch.Remove(Key)); + return Plus(counter, 1); + }); + + Assert.That(seen, Is.EqualTo(new int?[] { 1, null })); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(StoredValue(path, directory), Is.EqualTo(1)); + } + + [Test] + public void ExhaustedAttemptsReturnLostRaceInADirectoryStore() + { + var path = PathOf("lost-dir.cc"); + Seed(path, true, 1); + using var cache = Open(path, true); + using var rival = Open(path, true); + var calls = 0; + + var result = cache.Update(Key, counter => + { + Rival(rival, 100 * ++calls); + return Plus(counter, 1); + }, maxAttempts: 2); + + Assert.That(calls, Is.EqualTo(2)); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace)); + Assert.That(result.Document!.Value, Is.EqualTo(200)); + Assert.That(StoredValue(path, true), Is.EqualTo(200)); + } + + [Test] + public void OneAttemptMeansOneApplyAndNonPositiveThrows() + { + var path = PathOf("one.cc"); + Seed(path, false, 1); + using var cache = Open(path, false); + using var rival = Open(path, false); + var calls = 0; + + var result = cache.Update(Key, counter => { calls++; Rival(rival, 50); return Plus(counter, 1); }, maxAttempts: 1); + + Assert.That(calls, Is.EqualTo(1)); + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace)); + Assert.That(() => cache.Update(Key, counter => counter, maxAttempts: -1), Throws.TypeOf()); + Assert.That(calls, Is.EqualTo(1)); + } + + [TestCase(false)] + [TestCase(true)] + public void UpdatedDocumentIsTheHeldInstanceAndOwnWritesCauseNoRetry(bool directory) + { + var path = PathOf("own.cc"); + Seed(path, directory, 1); + using var cache = Open(path, directory); + cache.UpsertAsync(new UpdateCounter { Name = "counter", Value = 10 }, new CultRecordHandle(Key)).GetAwaiter().GetResult(); + cache.FlushAsync().GetAwaiter().GetResult(); + var calls = 0; + + var first = cache.Update(Key, counter => { calls++; return Plus(counter, 1); }); + var second = cache.Update(Key, counter => { calls++; return Plus(counter, 1); }); + + Assert.That(calls, Is.EqualTo(2), "the cache's own writes made a later Update retry"); + Assert.That(first.Document, Is.Not.Null); + Assert.That(second.Document, Is.SameAs(cache.Get(Key)), "Updated returned a copy, not the held instance"); + Assert.That(StoredValue(path, directory), Is.EqualTo(12)); + } + + [Test] + public void ApplyRunsOutsideTheCacheGate() + { + var path = PathOf("gate.cc"); + Seed(path, false, 1); + using var cache = Open(path, false); + var reached = false; + + cache.Update(Key, counter => + { + reached = Task.Run(() => cache.Get(Key)).Wait(TimeSpan.FromSeconds(10)); + return Plus(counter, 1); + }); + + Assert.That(reached, Is.True, "another thread could not read the cache while apply ran"); + } + + [TestCase(false)] + [TestCase(true)] + public void ContendedLedgerStoredValueEqualsUpdatedCount(bool directory) + { + var path = PathOf("ledger.cc"); + Seed(path, directory, 0); + var caches = Enumerable.Range(0, 3).Select(_ => Open(path, directory)).ToArray(); + try + { + var tasks = caches.Select(cache => Task.Run(() => + { + var updated = 0; + for (var i = 0; i < 25; i++) + if (cache.Update(Key, counter => Plus(counter, 1)).Outcome == CultUpdateOutcome.Updated) + updated++; + return updated; + })).ToArray(); + Assert.That(Task.WaitAll(tasks, TimeSpan.FromMinutes(2)), Is.True); + + Assert.That(StoredValue(path, directory), Is.EqualTo(tasks.Sum(task => task.Result)), "an Updated outcome was lost or a LostRace wrote"); + } + finally + { + foreach (var cache in caches) + cache.Dispose(); + } + } + + [Test] + public void InMemoryCacheUpdatesUnderThreadRace() + { + using var cache = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)); + cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter" }, new CultRecordHandle(Key))); + + var threads = Enumerable.Range(0, 2).Select(_ => Task.Run(() => + { + for (var i = 0; i < 100; i++) + Assert.That(cache.Update(Key, counter => Plus(counter, 1), maxAttempts: 200).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + })).ToArray(); + Assert.That(Task.WaitAll(threads, TimeSpan.FromMinutes(1)), Is.True); + + Assert.That(cache.Get(Key)!.Value, Is.EqualTo(200)); + } + + [TestCase(false)] + [TestCase(true)] + public void StaleCacheFirstAttemptSeesTheStore(bool directory) + { + var path = PathOf("stale.cc"); + Seed(path, directory, 1); + using var a = Open(path, directory); + using var b = Open(path, directory); + Assert.That(b.Get(Key)!.Value, Is.EqualTo(1)); + Rival(a, 7); + var seen = new List(); + + var result = b.Update(Key, counter => { seen.Add(counter!.Value); return Plus(counter, 1); }); + + Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + Assert.That(seen, Is.EqualTo(new[] { 7 }), "the first attempt read the cache's stale copy"); + Assert.That(StoredValue(path, directory), Is.EqualTo(8)); + } + + // The durable read is not a load: it leaves the store's view of what the cache holds alone, so a later pull still delivers the rival's write. + [TestCase(false)] + [TestCase(true)] + public void ADeclinedUpdateDoesNotHideTheRivalsWriteFromALaterPull(bool directory) + { + var path = PathOf("hidden.cc"); + Seed(path, directory, 1); + using var a = Open(path, directory); + using var b = Open(path, directory); + Assert.That(b.Get(Key)!.Value, Is.EqualTo(1)); + Rival(a, 7); + + Assert.That(b.Update(Key, _ => null).Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); + Assert.That(b.Get(Key)!.Value, Is.EqualTo(1), "a declined Update changed what the cache holds"); + b.PullAllBackingStoresAsync().GetAwaiter().GetResult(); + + Assert.That(b.Get(Key)!.Value, Is.EqualTo(7)); + } + + [Test] + public void CodecLessStoreBackedCacheUpdates() + { + var path = PathOf("codecless.cc"); + Seed(path, false, 1); + var calls = 0; + using (var cache = Open(path, false, codec: false)) + Assert.That(cache.Update(Key, counter => { calls++; return Plus(counter, 1); }).Outcome, Is.EqualTo(CultUpdateOutcome.Updated)); + + Assert.That(calls, Is.EqualTo(1)); + Assert.That(StoredValue(path, false), Is.EqualTo(2)); + } + + [TestCase(false)] + [TestCase(true)] + public void AStoreHoldingNothingAtTheKeyIsAbsentEvenIfTheCacheHoldsIt(bool directory) + { + var path = PathOf("gone.cc"); + Seed(path, directory, 5); + using var a = Open(path, directory); + using var b = Open(path, directory); + Assert.That(b.Get(Key), Is.Not.Null); + Assert.That(a.Commit(batch => batch.Remove(Key)), Is.True); + var seen = new List(); + + Assert.That(b.Update(Key, counter => { seen.Add(counter); return null; }).Outcome, Is.EqualTo(CultUpdateOutcome.Declined)); + Assert.That(seen, Is.EqualTo(new UpdateCounter?[] { null })); + } + + [CultDocument("tests.update_counter", "tests.update_counter.v1")] + [MessagePackObject(AllowPrivate = true)] + internal sealed class UpdateCounter + { + [Key(0)] + [CultName] + public string Name = string.Empty; + + [Key(1)] + public int Value; + } + + [CultDocument("tests.update_tally", "tests.update_tally.v1")] + [MessagePackObject(AllowPrivate = true)] + internal sealed class UpdateTally + { + [Key(0)] + [CultName] + public string Name = string.Empty; + } + + [CultDocument("tests.update_gear", "tests.update_gear.v1")] + [MessagePackObject(AllowPrivate = true)] + internal sealed class UpdateGear + { + [Key(0)] + [CultName] + public string Name = string.Empty; + + [Key(1)] + public int Power; + } + + [CultDocument("tests.update_base", "tests.update_base.v1")] + [MessagePackObject(AllowPrivate = true)] + internal class UpdateBase + { + [Key(0)] + [CultName] + public string Name = string.Empty; + } + + [CultDocument("tests.update_derived", "tests.update_derived.v1")] + [MessagePackObject(AllowPrivate = true)] + internal sealed class UpdateDerived : UpdateBase + { + [Key(1)] + public int Value; + } + } +} diff --git a/tests/GameCult.Caching.Tests/stryker-config.json b/tests/GameCult.Caching.Tests/stryker-config.json new file mode 100644 index 00000000..ac366203 --- /dev/null +++ b/tests/GameCult.Caching.Tests/stryker-config.json @@ -0,0 +1,11 @@ +{ + "stryker-config": { + "project": "GameCult.Caching.csproj", + "mutate": [ + "**/src/GameCult.Caching/**/*.cs" + ], + "mutation-level": "Standard", + "reporters": [ "progress", "html", "json" ], + "thresholds": { "high": 80, "low": 60, "break": 0 } + } +} diff --git a/tests/GameCult.Mesh.Tests/BytesAfterPathTests.cs b/tests/GameCult.Mesh.Tests/BytesAfterPathTests.cs index ed9b904b..28c6d3de 100644 --- a/tests/GameCult.Mesh.Tests/BytesAfterPathTests.cs +++ b/tests/GameCult.Mesh.Tests/BytesAfterPathTests.cs @@ -58,4 +58,22 @@ public class BytesAfterPathTests .MakeGenericMethod(typeof(CultMeshCdnArtifactManifest)); Refusal(() => read.Invoke(null, new object[] { snapshot, record.SchemaId, record.RecordKey })).Should().Contain(Clause); } + + [Test] + public void ASingleFileDocumentReadRefusesATail() + { + var path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), $"cultlib-bytesafter-mesh-{Guid.NewGuid():N}.cc"); + try + { + var descriptor = CultDocumentRegistry.Shared.GetRequired(); + var schema = new CultMeshSingleFileDocumentSchema(descriptor.SchemaId, descriptor.SchemaName, descriptor.SchemaVersion); + var payload = WithTail(Manifest()).Payload; + CultMesh.WriteSingleFileDocumentPayload(path, new CultRecordKey("k"), schema, null, payload); + Refusal(() => CultMesh.ReadSingleFileDocument(path, new CultRecordKey("k"))).Should().Contain(Clause); + } + finally + { + System.IO.File.Delete(path); + } + } } diff --git a/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs b/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs index 7ce94ed2..19c08a4b 100644 --- a/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs +++ b/tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs @@ -4,6 +4,7 @@ using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; +using System.Runtime.InteropServices; using System.Threading; using System.Threading.Tasks; using FluentAssertions; @@ -364,11 +365,11 @@ public sealed class CultMeshSingleFileStoreTests [Test] public void ACacheOpeningDuringMeshWritesNeverSeesAnEmptyStore() { - // Windows replaces with ReplaceFile, which leaves a moment with nothing at the path for a lock-free reader: 1,722 empty - // stores in 17,313 opens at b1ff2dcb. That is the store's replace, not Mesh's, and it is the same for a cache flush beside a - // cache open; the fork is in the cut report. - if (OperatingSystem.IsWindows()) - Assert.Ignore("Windows File.Replace is not atomic to a lock-free reader; see cut report fork mesh-reader-windows-replace-window."); + // Windows replaces with a POSIX-semantics rename first (StoreReplace), which makes a lock-free reader finding the path + // absent rare, not impossible: a stalled rename was measured leaving it absent for 5-47 ms on Windows. So on Windows a + // brief not-found (a read throwing, or a cache opening onto an absent path) is tolerated and reported; elsewhere rename + // is atomic and none is allowed. + var windows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows); var path = Path.Combine(_root, "race.cc"); CultMesh.WriteSingleFileDocument(path, Key, Publication()); @@ -386,7 +387,7 @@ public sealed class CultMeshSingleFileStoreTests Volatile.Write(ref stop, 1); }); - int opens = 0, empty = 0, reads = 0; + int opens = 0, empty = 0, reads = 0, absent = 0; var clock = Stopwatch.StartNew(); while (Volatile.Read(ref stop) == 0 && clock.Elapsed < TimeSpan.FromSeconds(60)) { @@ -397,6 +398,10 @@ public sealed class CultMeshSingleFileStoreTests CultMesh.ReadSingleFileDocumentPayload(path, Key, SchemaId()); reads++; } + catch (Exception ex) when (windows && (ex is FileNotFoundException || ex is DirectoryNotFoundException)) + { + absent++; + } catch (Exception ex) { errors.Enqueue("read: " + ex.GetType().Name + ": " + ex.Message); @@ -404,9 +409,10 @@ public sealed class CultMeshSingleFileStoreTests } writer.Wait(TimeSpan.FromSeconds(60)).Should().BeTrue(); - TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} mesh reads ok={reads} errors={errors.Count}"); + TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} not-found reads={absent} mesh reads ok={reads} errors={errors.Count}"); errors.Take(3).Should().BeEmpty(); - empty.Should().Be(0, "a cache opened in the window where nothing was at the path"); + if (!windows) + empty.Should().Be(0, "a cache opened in the window where nothing was at the path"); opens.Should().BeGreaterThan(0); } @@ -434,32 +440,74 @@ public sealed class CultMeshSingleFileStoreTests TempFiles().Should().BeEmpty(); } - // The store's replace decides what a write leaves where a live link was, and a Mesh write leaves what a cache commit leaves: - // one replace, whatever the platform does with a link. - [Test] - public void AMeshWriteOverALiveLinkLeavesWhatACacheCommitLeaves() + // A write at a store path that is a link is refused before the lock, the staging file and the replace: the link and its + // target stay as they were, and nothing is left beside them. + private void AssertRefusedAndUntouched(string link, string target, byte[] targetBefore, Action write) { - string Outcome(string name, Action write) + var refusal = Assert.Throws(() => write())!; + refusal.Message.Should().Contain(link); + IsALink(link).Should().BeTrue("the link was replaced"); + File.ReadAllBytes(target).Should().Equal(targetBefore); + File.Exists(link + ".lock").Should().BeFalse("the refused write took the lock"); + TempFiles().Should().BeEmpty(); + } + + [Test] + public void AMeshWriteAtALiveLinkIsRefusedAndNothingChanges() + { + var target = Path.Combine(_root, "target.cc"); + var link = Path.Combine(_root, "link.cc"); + CultMesh.WriteSingleFileDocument(target, Key, Publication("first")); + CreateLiveLinkOrIgnore(link, target); + var before = File.ReadAllBytes(target); + + AssertRefusedAndUntouched(link, target, before, () => CultMesh.WriteSingleFileDocument(link, Key, Publication("second"))); + AssertRefusedAndUntouched(link, target, before, () => CultMesh.WriteSingleFileDocumentPayload(link, Key, Raw, null, EmptyArray)); + } + + [Test] + public void AReadThroughALiveLinkStillWorks() + { + var target = Path.Combine(_root, "target.cc"); + var link = Path.Combine(_root, "link.cc"); + CultMesh.WriteSingleFileDocument(target, Key, Publication("first")); + CreateLiveLinkOrIgnore(link, target); + + CultMesh.ReadSingleFileDocument(link, Key).ProducerId.Should().Be("first"); + RecordsSeenByACache(link).Should().Be(1); + } + + // A link to a directory that is there is a directory, not an unreachable store; a write at it is a link refusal. + [Test] + public void ALiveDirectoryLinkIsADirectoryErrorOnReadAndALinkRefusalOnWrite() + { + var directory = Path.Combine(_root, "real-directory"); + var link = Path.Combine(_root, "directory-link.cc"); + Directory.CreateDirectory(directory); + CreateDirectoryLink(link, directory); + + Assert.Catch(() => CultMesh.ReadSingleFileDocumentPayload(link, Key, "raw:schema"))!.Message.Should().Contain("directory"); + Assert.Catch(() => RecordsSeenByACache(link))!.Message.Should().Contain("directory"); + Assert.Throws(() => CultMesh.WriteSingleFileDocument(link, Key, Publication())); + Directory.GetFileSystemEntries(directory).Should().BeEmpty(); + IsALink(link).Should().BeTrue(); + } + + private static void CreateDirectoryLink(string path, string target) + { + if (!OperatingSystem.IsWindows()) { - var target = Path.Combine(_root, name + "-target.cc"); - var link = Path.Combine(_root, name + "-link.cc"); - CultMesh.WriteSingleFileDocument(target, Key, Publication("first")); - CreateLiveLinkOrIgnore(link, target); - var before = File.ReadAllBytes(target); - - write(link); - - CultMesh.ReadSingleFileDocument(link, Key).ProducerId.Should().Be("second", name); - return $"linkKept={IsALink(link)} targetChanged={!File.ReadAllBytes(target).SequenceEqual(before)}"; + Directory.CreateSymbolicLink(path, target); + return; } - var viaMesh = Outcome("mesh", link => CultMesh.WriteSingleFileDocument(link, Key, Publication("second"))); - var viaCache = Outcome("cache", link => + var mklink = Process.Start(new ProcessStartInfo("cmd.exe", $"/c mklink /J \"{path}\" \"{target}\"") { - using var cache = CultCacheMessagePack.Create(link, ReadOnlyOpen); - cache.Commit(batch => batch.Upsert(typeof(CultMeshBodyPublicationDocument), Publication("second"), Key)); - }); - - viaMesh.Should().Be(viaCache); + CreateNoWindow = true, + RedirectStandardOutput = true, + RedirectStandardError = true, + })!; + mklink.WaitForExit(); + Assert.That(mklink.ExitCode, Is.Zero, "mklink /J failed"); } } diff --git a/tests/GameCult.Networking.Tests/BytesAfterPathTests.cs b/tests/GameCult.Networking.Tests/BytesAfterPathTests.cs index 6a56ceb1..0b125075 100644 --- a/tests/GameCult.Networking.Tests/BytesAfterPathTests.cs +++ b/tests/GameCult.Networking.Tests/BytesAfterPathTests.cs @@ -21,7 +21,15 @@ namespace GameCult.Networking.Tests [Key(1)] public int N = 3; } - private static readonly CultDocumentRegistry Documents = CultDocumentRegistry.ForTypes(new[] { typeof(TailDoc) }); + [CultDocument("tests.bytes_after_deep", "tests.bytes_after_deep.v1")] + [MessagePackObject] + public sealed class DeepDoc + { + [Key(0)] [CultName] public string Id = "a"; + [Key(1)] public DeepDoc Inner; + } + + private static readonly CultDocumentRegistry Documents = CultDocumentRegistry.ForTypes(new[] { typeof(TailDoc), typeof(DeepDoc) }); private static byte[] WithTail() => CultDocumentMessagePackSerialization.SerializeUntyped(new TailDoc(), typeof(TailDoc), Documents).Concat(new byte[] { 0xc0 }).ToArray(); @@ -58,5 +66,28 @@ namespace GameCult.Networking.Tests var registry = new CultNetDocumentRegistry(Documents); Refuses(() => registry.DeserializeRawDocument(Raw(registry, WithTail()))); } + + // The write side of the same rule: a record with no binding is serialized by SerializeUntyped, and a payload nested past + // the limit is refused there, before it can reach the wire. + [Test] + public void ToRawRecordWithNoBindingRefusesAPayloadNestedPastTheLimit() + { + var registry = new CultNetDocumentRegistry(Documents); + var descriptor = Documents.GetRequired(typeof(DeepDoc)); + DeepDoc Chain(int links) + { + var head = new DeepDoc(); + var tip = head; + for (var i = 0; i < links; i++) tip = tip.Inner = new DeepDoc(); + return head; + } + + // Each link is one array; the root is depth 1, so 255 links is depth 256 (at the limit) and 256 links is 257. + Assert.That(registry.ToRawRecord(descriptor, new CultRecordKey("a"), Chain(255), "1970-01-01T00:00:00.000Z").Payload, Is.Not.Empty); + var refused = Assert.Throws( + () => registry.ToRawRecord(descriptor, new CultRecordKey("a"), Chain(256), "1970-01-01T00:00:00.000Z"))!; + Assert.That(refused.Message, Does.Contain("refused on write")); + Assert.That(refused.Message, Does.Contain(typeof(DeepDoc).FullName)); + } } } diff --git a/tests/GameCult.Networking.Tests/CultNetWriteReceiptServerTests.cs b/tests/GameCult.Networking.Tests/CultNetWriteReceiptServerTests.cs index e42961a2..bbaf2f68 100644 --- a/tests/GameCult.Networking.Tests/CultNetWriteReceiptServerTests.cs +++ b/tests/GameCult.Networking.Tests/CultNetWriteReceiptServerTests.cs @@ -1119,6 +1119,8 @@ namespace GameCult.Networking.Tests foreach (var field in BlankRuleFields) foreach (var times in new[] { 1, 3 }) { + // A storedAt must be a stamp: a non-blank one that is no stamp is not "a name" (write-receipt.md, Required fields). + if (!blank && field == "storedAt") continue; var text = string.Concat(Enumerable.Repeat(char.ConvertFromUtf32(Convert.ToInt32(hex, 16)), times)); var rig = Build(); var wire = new Wire(); diff --git a/tests/GameCult.Networking.Tests/NetworkingTests.cs b/tests/GameCult.Networking.Tests/NetworkingTests.cs index 606d5625..1aee870d 100644 --- a/tests/GameCult.Networking.Tests/NetworkingTests.cs +++ b/tests/GameCult.Networking.Tests/NetworkingTests.cs @@ -7377,6 +7377,183 @@ namespace GameCult.Networking.Tests Assert.That(developmentServer.LogSensitivePayloads, Is.True); } + // Records the flag each PushAll took, and holds nothing: a flush that decided soft itself and never asked the store records nothing. + private sealed class SoftFlagSpyStore : CacheBackingStore + { + public sealed class SpyHome { } + + public readonly List Flags = new(); + public void Stage() => IsDirty = true; + public override void PullAll() { } + public override void Push(CultStoredDocument entry) { } + public override void Delete(CultStoredDocument entry) { } + public override void PushAll(bool soft) => Flags.Add(soft); + public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; + } + + [Test] + public async Task CultNetLocal_Host_FlushAsync_Hands_The_Store_The_Flag_It_Was_Given() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-spy-{Guid.NewGuid():N}.msgpack"); + CultNetHost? host = null; + try + { + host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false }); + var spy = new SoftFlagSpyStore(); + host.Cache.AddBackingStore(spy, typeof(SoftFlagSpyStore.SpyHome)); + spy.Stage(); + + await host.FlushAsync(soft: true); + await host.FlushAsync(); + + Assert.That(spy.Flags, Is.EqualTo(new[] { true, false })); + } + finally + { + host?.Dispose(); + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task CultMeshNode_FlushAsync_Hands_The_Store_The_Flag_It_Was_Given() + { + var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-spy", Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(rootPath); + var documents = new CultNetDocumentRegistry() + .Register(CultNetDocumentBinding.ForDocument()); + using var node = await CultMesh.CreateNodeAsync(Path.Combine(rootPath, "world.ccmp"), new CultMeshNodeOptions + { + StartServer = false, + DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-spy-local", DocumentRegistry = documents } + }); + var spy = new SoftFlagSpyStore(); + node.Cache.AddBackingStore(spy, typeof(SoftFlagSpyStore.SpyHome)); + spy.Stage(); + + await node.FlushAsync(soft: true); + await node.FlushAsync(); + + Assert.That(spy.Flags, Is.EqualTo(new[] { true, false })); + } + + [Test] + public async Task CultNetLocal_Host_SoftFlush_Passes_The_Flag_To_The_Store() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-soft-{Guid.NewGuid():N}.msgpack"); + CultNetHost? host = null; + + try + { + host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false }); + await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "a@example.test", PasswordHash = "hash", Username = "First" }); + await host.FlushAsync(); + var before = File.ReadAllBytes(filePath); + + await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "b@example.test", PasswordHash = "hash", Username = "Second" }); + await host.FlushAsync(soft: true); + Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft host flush rewrote a store it reads"); + + await host.FlushAsync(); + Assert.That(File.ReadAllBytes(filePath), Is.Not.EqualTo(before), "a plain host flush after a skipped soft flush did not land the staged write"); + } + finally + { + host?.Dispose(); + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task CultMeshNode_SoftFlush_Passes_The_Flag_To_The_Store() + { + var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-flush", Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(rootPath); + var cachePath = Path.Combine(rootPath, "world.ccmp"); + var documents = new CultNetDocumentRegistry() + .Register(CultNetDocumentBinding.ForDocument()); + + using var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions + { + StartServer = false, + DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-flush-local", DocumentRegistry = documents } + }); + await node.Database.PutAsync(new CultRecordKey("note:one"), new MeshQuickstartNote { NoteId = "note:one", Body = "one" }); + await node.FlushAsync(); + var before = File.ReadAllBytes(cachePath); + + await node.Database.PutAsync(new CultRecordKey("note:two"), new MeshQuickstartNote { NoteId = "note:two", Body = "two" }); + await node.FlushAsync(soft: true); + Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(before), "a soft node flush rewrote a store it reads"); + + await node.FlushAsync(); + Assert.That(File.ReadAllBytes(cachePath), Is.Not.EqualTo(before), "a plain node flush after a skipped soft flush did not land the staged write"); + } + + // The cache HOLDS a record in every step: a passthrough that decides soft from the cache's memory instead of + // handing the store the flag fails the absent step (it skips and writes nothing) or the unreadable step (it + // never reaches the reader that refuses). + [Test] + public async Task CultNetLocal_Host_SoftFlush_With_A_Held_Record_Writes_Absent_Skips_Readable_Refuses_Unreadable() + { + var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-soft-held-{Guid.NewGuid():N}.msgpack"); + CultNetHost? host = null; + try + { + host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false }); + await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "a@example.test", PasswordHash = "hash", Username = "First" }); + Assert.That(File.Exists(filePath), Is.False); + + await host.FlushAsync(soft: true); + Assert.That(File.Exists(filePath), Is.True, "a soft host flush over an absent store, with a record held, wrote nothing"); + var written = File.ReadAllBytes(filePath); + + await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "b@example.test", PasswordHash = "hash", Username = "Second" }); + await host.FlushAsync(soft: true); + Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(written), "a soft host flush rewrote a readable store"); + + var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; + File.WriteAllBytes(filePath, junk); + Assert.That(async () => await host.FlushAsync(soft: true), Throws.InstanceOf()); + Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(junk)); + } + finally + { + host?.Dispose(); + if (File.Exists(filePath)) File.Delete(filePath); + } + } + + [Test] + public async Task CultMeshNode_SoftFlush_With_A_Held_Record_Writes_Absent_Skips_Readable_Refuses_Unreadable() + { + var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-held", Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(rootPath); + var cachePath = Path.Combine(rootPath, "world.ccmp"); + var documents = new CultNetDocumentRegistry() + .Register(CultNetDocumentBinding.ForDocument()); + using var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions + { + StartServer = false, + DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-held-local", DocumentRegistry = documents } + }); + await node.Database.PutAsync(new CultRecordKey("note:one"), new MeshQuickstartNote { NoteId = "note:one", Body = "one" }); + Assert.That(File.Exists(cachePath), Is.False); + + await node.FlushAsync(soft: true); + Assert.That(File.Exists(cachePath), Is.True, "a soft node flush over an absent store, with a record held, wrote nothing"); + var written = File.ReadAllBytes(cachePath); + + await node.Database.PutAsync(new CultRecordKey("note:two"), new MeshQuickstartNote { NoteId = "note:two", Body = "two" }); + await node.FlushAsync(soft: true); + Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(written), "a soft node flush rewrote a readable store"); + + var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 }; + File.WriteAllBytes(cachePath, junk); + Assert.That(async () => await node.FlushAsync(soft: true), Throws.InstanceOf()); + Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(junk)); + } + [Test] public async Task CultNetLocal_CreateHostAsync_Wires_Durable_Cache_Without_Starting_Server() { diff --git a/tests/GameCult.Networking.Tests/RemoteWriteAuthorityTests.cs b/tests/GameCult.Networking.Tests/RemoteWriteAuthorityTests.cs index 897674b1..7cdaecbc 100644 --- a/tests/GameCult.Networking.Tests/RemoteWriteAuthorityTests.cs +++ b/tests/GameCult.Networking.Tests/RemoteWriteAuthorityTests.cs @@ -405,6 +405,114 @@ namespace GameCult.Networking.Tests Assert.That(rig.Logger.Errors, Is.Empty, "a null decision is a refusal, not a fault"); } + // ---- a peer's storedAt must be a stamp (tests/vectors/stored-at-stamp, variants:ruling:raw-put-without-stamp-refused) ---- + + private static (string Text, bool Parses)[] StampVectors() + { + var dir = new System.IO.DirectoryInfo(TestContext.CurrentContext.TestDirectory); + while (dir != null && !System.IO.Directory.Exists(System.IO.Path.Combine(dir.FullName, "tests", "vectors"))) dir = dir.Parent; + var bytes = System.IO.File.ReadAllBytes(System.IO.Path.Combine(dir!.FullName, "tests", "vectors", "stored-at-stamp", "stamps.msgpack")); + return MessagePack.MessagePackSerializer.Deserialize(bytes) + .Select(row => (Row: (object[])row, 0)).Select(r => ((string)r.Row[0], (bool)r.Row[1])).ToArray(); + } + + // A blank stamp is a frame that does not read (write-receipt.md, vector blankFields: field "message"); any other text + // that is no stamp is message_invalid on field "document". Both are refused before the store is asked. + private static bool IsBlankText(string text) => text.All(c => char.IsWhiteSpace(c)); + + [Test] + public async Task EveryStampVectorRowIsJudgedThroughTheRealServerHandler() + { + using var wire = new WirePair(); + var rig = Build(new RecordingAuthority(_ => CultNetRemoteWriteDecision.Grant()), peer: wire.Peer); + var rows = StampVectors(); + Assert.That(rows, Has.Length.GreaterThan(30)); + var n = 0; + foreach (var (text, parses) in rows) + { + var key = new CultRecordKey($"tests:stamp:{n++}"); + var put = rig.PutOf(key, "peer"); + put.Document.StoredAt = text; + var publishedBefore = rig.Published.Count; + var logBefore = rig.LogLength; + + await rig.Put(put); + + if (parses) + { + Assert.That(rig.TextOf(key), Is.EqualTo("peer"), $"stamp '{text}' is admitted"); + Assert.That(wire.Errors, Is.Empty, $"stamp '{text}' is not answered with an error"); + continue; + } + + var reply = await wire.NextErrorAsync(); + Assert.That(reply.Code, Is.EqualTo("message_invalid"), $"'{text}'"); + Assert.That(reply.Details!.Field, Is.EqualTo(IsBlankText(text) ? "message" : "document"), $"'{text}'"); + Assert.That(rig.TextOf(key), Is.Null, $"'{text}' stored nothing"); + Assert.That(rig.Cache.Get(key), Is.Null, $"'{text}' minted nothing"); + Assert.That(rig.Published, Has.Count.EqualTo(publishedBefore), $"'{text}' published nothing"); + Assert.That(rig.LogLength, Is.EqualTo(logBefore), $"'{text}' logged nothing"); + Assert.That(wire.Errors, Is.Empty, "one reply per refused write"); + } + + var fresh = new CultRecordKey("tests:stamp:after"); + await rig.Put(rig.PutOf(fresh, "after")); + Assert.That(rig.TextOf(fresh), Is.EqualTo("after"), "the store stays writable after the refusals"); + Assert.That(wire.Errors, Is.Empty); + } + + // The host's own door (the public ApplyPutAsync overload, no authorize) is exempt from the stamp judgment: the cache mints + // a stamp for it. Applying the check to every call would refuse this put. + [Test] + public async Task TheHostsOwnDoorIsExemptFromTheStampJudgment() + { + var rig = Build(new RecordingAuthority(_ => CultNetRemoteWriteDecision.Refuse("no"))); + var put = rig.PutOf(KeyA, "host"); + put.Document.StoredAt = "yesterday"; + + await rig.Database.ApplyPutAsync(put); + + Assert.That(rig.TextOf(KeyA), Is.EqualTo("host"), "the host's put with no stamp is applied, not refused"); + } + + [Test] + public async Task AnUngrantedPeerIsToldItIsNotAuthorizedBeforeItsStampIsJudgedAndAnAliasAsksTheCanonicalId() + { + using var wire = new WirePair(); + var authority = new RecordingAuthority(_ => CultNetRemoteWriteDecision.Refuse("no")); + var rig = Build(authority, peer: wire.Peer); + var put = rig.PutOf(KeyA, "peer"); + put.Document.StoredAt = "yesterday"; + put.Document.SchemaId = "runtime.generated.network-note.ui.42"; + + await rig.Put(put); + var reply = await wire.NextErrorAsync(); + + Assert.That(reply.Code, Is.EqualTo("write_not_authorized"), "authority first, the stamp is not judged"); + Assert.That(authority.Requests.Single().SchemaId, Is.EqualTo(rig.SchemaId)); + Assert.That(rig.TextOf(KeyA), Is.Null); + } + + [Test] + public async Task AGarbageStampIsNeverForwardedToThePrimary() + { + using var wire = new WirePair(); + var forwarder = new CountingForwarder(); + var rig = Build(new RecordingAuthority(_ => CultNetRemoteWriteDecision.Grant()), primary: false, forwarder, wire.Peer); + var put = rig.PutOf(KeyA, "peer"); + put.Document.StoredAt = "2026-06-25T00:00:00"; + + await rig.Put(put); + var reply = await wire.NextErrorAsync(); + + Assert.That(reply.Code, Is.EqualTo("message_invalid")); + Assert.That(reply.Details!.Field, Is.EqualTo("document")); + Assert.That(forwarder.Puts, Is.EqualTo(0)); + + await rig.Put(rig.PutOf(KeyA, "peer")); + Assert.That(forwarder.Puts, Is.EqualTo(1), "a stamped put still forwards"); + } + private sealed class RecordingAuthority : ICultNetRemoteWriteAuthority { private readonly Func _decide; diff --git a/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs b/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs index 3a076d16..ad7dcaca 100644 --- a/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs +++ b/tests/GameCult.Networking.Tests/SelectionParityVectorTests.cs @@ -341,7 +341,7 @@ namespace GameCult.Networking.Tests { var path = Path.Combine(RepoRoot(), "contracts", "cultnet", "interop", "selection-vectors.rs-written.json"); if (!File.Exists(path)) - Assert.Inconclusive($"{path} is missing - run cargo test -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first."); + Assert.Inconclusive($"{path} is missing - run cargo test --locked -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first."); var (registry, rows, _) = BuildFixture(); using var document = JsonDocument.Parse(File.ReadAllText(path)); diff --git a/tests/vectors/content-version/README.md b/tests/vectors/content-version/README.md new file mode 100644 index 00000000..cddd7bff --- /dev/null +++ b/tests/vectors/content-version/README.md @@ -0,0 +1,17 @@ +# content-version vectors + +Written only by the C# reference, `tests/GameCult.Caching.Tests/ContentVersionVectorTests.cs`, with +`CULTCACHE_WRITE_VECTORS=1` (never set in CI). Every other run opens the committed `store.cc`, compares what the reference reads +to `versions.msgpack` and compares the store bytes the reference writes to the file; a changed vector is a reviewed commit. +Other runtimes compute the version of each record of `store.cc` and must equal these rows +(`variants:ruling:write-set-content-version-declared-ownership`). + +RULE: the content version of a stored record is the lowercase hex SHA-256 of its persisted encoding, the MessagePack array +`[key, schemaId, storedAt, payload bytes]` exactly as the store writes the record. + +## Files + +- `store.cc`: a single-file store of one type, `tests.cv_item` (one string slot, `[0: Text]`), holding `alpha` = "first" and + `beta` = "second", stored-at pinned to `2026-01-01T00:00:00Z`. +- `versions.msgpack`: an array of `[key, version]`, sorted by key, the version the reference reads for the record + (`GetStored(key).StoredVersion`). diff --git a/tests/vectors/content-version/store.cc b/tests/vectors/content-version/store.cc new file mode 100644 index 00000000..3962430a Binary files /dev/null and b/tests/vectors/content-version/store.cc differ diff --git a/tests/vectors/content-version/versions.msgpack b/tests/vectors/content-version/versions.msgpack new file mode 100644 index 00000000..6584d69a --- /dev/null +++ b/tests/vectors/content-version/versions.msgpack @@ -0,0 +1 @@ +’’¥alphaÙ@d330976df4f3946d0ad590d2091666aed4d5ebeac5d473792b4ef401b25c92fd’¤betaÙ@878026249c0df9a969ac1cac67bcdaab61f1b085ce4916c3c63f0fbef8f6da9f \ No newline at end of file diff --git a/tests/vectors/document-variants-c2a/old-reader-refusal.sh b/tests/vectors/document-variants-c2a/old-reader-refusal.sh index bfafa01d..5b9165f0 100755 --- a/tests/vectors/document-variants-c2a/old-reader-refusal.sh +++ b/tests/vectors/document-variants-c2a/old-reader-refusal.sh @@ -111,6 +111,7 @@ probe() { node ../../node_modules/typescript/bin/tsc -p tsconfig.test.json node --test dist-test/test/old-reader-v3.test.js ;; python) PYTHONPATH=packages/cultcache-py/src python3 packages/cultcache-py/tests/test_old_reader_v3.py -v ;; + # The staged tree is the frozen cultcache-rs-v0.3.1 revision, which has no committed Cargo.lock: --locked cannot work here. rust) cd packages/cultcache-rs && cargo test --test old_reader_v3 -- --nocapture ;; *) usage ;; esac diff --git a/tests/vectors/stored-at-stamp/README.md b/tests/vectors/stored-at-stamp/README.md new file mode 100644 index 00000000..76c0f216 --- /dev/null +++ b/tests/vectors/stored-at-stamp/README.md @@ -0,0 +1,11 @@ +# stored-at-stamp vectors + +`stamps.msgpack`: an array of `[text, parses]`. `text` (str) is a `storedAt` a peer might send; `parses` is whether it is a stamp. +Every runtime that judges a peer's `storedAt` (the raw put of cultmesh-py, and the C#, TypeScript and Rust servers) is held to +these answers, and a text that does not parse is refused `message_invalid` (field `document`) with nothing minted or stored +(`variants:ruling:raw-put-without-stamp-refused`). + +A stamp is an RFC 3339 date-time: `YYYY-MM-DDThh:mm:ss`, an optional fraction of 1 to 9 digits, and `Z` or a numeric offset +`+hh:mm` / `-hh:mm` (hours 00-23, minutes 00-59). Uppercase `T` and `Z`, ASCII digits, no surrounding whitespace, a real +calendar date, seconds 00-59 (no leap second). The local year must be 1-9999 AND the UTC instant (after the offset is applied) must lie in years 0001-9999. The reference is `parse_stored_at` in +`packages/cultcache-py/src/cultcache_py/stores.py` (#172); this file was written from it and is a reviewed commit when it changes. diff --git a/tests/vectors/stored-at-stamp/stamps.msgpack b/tests/vectors/stored-at-stamp/stamps.msgpack new file mode 100644 index 00000000..56427db2 Binary files /dev/null and b/tests/vectors/stored-at-stamp/stamps.msgpack differ diff --git a/unity/org.gamecult.cultlib/CHANGELOG.md b/unity/org.gamecult.cultlib/CHANGELOG.md index a4a2bc11..612aee04 100644 --- a/unity/org.gamecult.cultlib/CHANGELOG.md +++ b/unity/org.gamecult.cultlib/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to this package are documented in this file. ### Breaking +- `CacheBackingStore.PushAll()` is no longer abstract: it is a non-virtual convenience over the new abstract `PushAll(bool soft)`. A store kind that overrides `PushAll()` no longer compiles; override `PushAll(bool soft)` instead. A soft flush writes only where nothing is at the path: over a store the reader reads, empty included, it returns without writing and leaves the staged writes staged and dirty for a later plain flush; over a file the reader cannot read it refuses. `CultCache.FlushAsync(bool soft = false)`, `CultCache.FlushAllBackingStores(bool soft = false)`, `CultMeshNode.FlushAsync(soft)` and `CultNetHost.FlushAsync(soft)` pass the flag to the stores; `soft: true` was accepted and ignored before, so a caller that passed it now stops landing writes on an existing store. Semver: breaking for C# consumers with their own store kind. - A document payload with bytes after its first MessagePack value is refused on read and on write (`MessagePackSerializationException` naming the type and the count of bytes after the value, never the bytes); the tail is not walked, so a tail nested past the limit is refused as bytes after the value, not as depth. Before, the tail was ignored on read and dropped on the next write. A payload that ends early is still the deserializer's to refuse. A CDN manifest is a record payload and follows the same rule: the acceptance vectors `trailing_nil`, `trailing_reserved_code` and `trailing_truncated_array` in `contracts/cultmesh/content-acceptance-vectors.json` are now refusals. `CultMeshProviderSessionWire.DecodeCommandDocument` refuses a command record whose payload has bytes after its first value (`MessagePackSerializationException` with the count); before, the tail was ignored. Semver: breaking for C# consumers, a major bump. - A single-file store holding two records under one key is unreadable: the reader refuses it with `CultStoreUnreadableException` naming the key (keys compare by ordinal code units, no case folding or normalisation), so a cache, `ReadStoreSnapshot` and a flush refuse the file and leave it as it is. Before, the later record silently replaced the earlier. A directory store reads its manifest through the same reader and refuses a duplicate key the same way. Repair the duplicate with the runtime that wrote it before reading the store here. - A record whose payload carries a slot past the slots its reading type declares, at any depth (the record, a nested object, a list element, a union arm or a unit arm), is loaded read-only: it is served, and every staged write or removal of its key (`Upsert`, `UpsertVariant`, `Remove`, a batch, and `Flatten` of a variant) is refused with `CultSchemaConflictException` before anything is staged, and the store keeps the record byte for byte. Before, the extra slots were skipped on read and dropped on the next write. Declaring an id compatible consents only to depth-0 slots the schema comparison reports ignored under that id; any other extra slot keeps the record read-only whatever is declared, including under the reader's own id. A variant overriding a slot its type does not have is read-only too (it used to be shed on flush). Write such a record from the build that declares those slots. Missing trailing slots still default and stay writable.