Compare commits

..
Author SHA1 Message Date
Libby
16dc9d8aaa Retrigger CI (runs 785-789 cancelled at 30:00; one attempt on idle runner)
All checks were successful
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Successful in 22m56s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 17:47:07 +00:00
Libby
7c1ccbc18b Retrigger CI (run 786 was cancelled)
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 14:20:11 +00:00
Libby
c72bb435d6 Docs: pair backup is per-store consistent only; atomic restore; hold the lock to delete temps; cite the read path
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 11:05:21 +00:00
Libby
eda3920135 Docs: state the single-file lock contract, leftover-temp and pair-backup procedures
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Replace the 'single-writer' wording with what the lock guarantees; drop two stale
follow-up claims (Python takes the lock; Rust reads wait); fix the Rust README's
'external lock' advice.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 10:58:51 +00:00
93 changed files with 327 additions and 5821 deletions

View file

@ -158,7 +158,7 @@ jobs:
# (only the interop example built, inside the C# store tests). The target directory is under RUNNER_TEMP, away from the checkout. # (only the interop example built, inside the C# store tests). The target directory is under RUNNER_TEMP, away from the checkout.
- name: Verify the CultCache document owner with the Rust cache - name: Verify the CultCache document owner with the Rust cache
shell: bash shell: bash
run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --manifest-path packages/cultcache-rs/Cargo.toml
# tests/GameCult.Caching.Tests, the C# reference cache's own suite (about 1,260 tests, under a minute), ran in no # tests/GameCult.Caching.Tests, the C# reference cache's own suite (about 1,260 tests, under a minute), ran in no
# forge workflow; only the Mesh tests above exercised the reference. # forge workflow; only the Mesh tests above exercised the reference.

View file

@ -29,12 +29,6 @@ jobs:
steps: steps:
- name: Check out CultLib - name: Check out CultLib
uses: actions/checkout@v4 uses: actions/checkout@v4
with:
# packages/cultnet-py/tests/test_declared_dependency_ranges.py reads the cultcache-py
# release tags and their file contents with `git show`; a shallow checkout has no tags
# and the test skips itself.
fetch-depth: 0
fetch-tags: true
- name: Set up .NET - name: Set up .NET
uses: actions/setup-dotnet@v4 uses: actions/setup-dotnet@v4
@ -135,10 +129,10 @@ jobs:
python -m unittest discover -s packages/cultmesh-py/tests python -m unittest discover -s packages/cultmesh-py/tests
- name: Verify Rust RUDP reliability - name: Verify Rust RUDP reliability
run: cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml rudp run: cargo test --manifest-path packages/cultnet-rs/Cargo.toml rudp
- name: Verify Rust CultMesh runtime-local surface - name: Verify Rust CultMesh runtime-local surface
run: cargo test --locked --manifest-path packages/cultmesh-rs/Cargo.toml run: cargo test --manifest-path packages/cultmesh-rs/Cargo.toml
- name: Verify C# RUDP reliability - name: Verify C# RUDP reliability
run: dotnet test tests/GameCult.Networking.Tests/GameCult.Networking.Tests.csproj --filter "FullyQualifiedName~Rudp" run: dotnet test tests/GameCult.Networking.Tests/GameCult.Networking.Tests.csproj --filter "FullyQualifiedName~Rudp"

View file

@ -212,8 +212,8 @@ identity-owned C# session after provider route replacement.
Rust package tests: Rust package tests:
```powershell ```powershell
cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml cargo test --manifest-path packages/cultcache-rs/Cargo.toml
cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml cargo test --manifest-path packages/cultnet-rs/Cargo.toml
``` ```
Kotlin package build: Kotlin package build:

View file

@ -1620,7 +1620,7 @@ The one door per runtime, and who walks through it:
- **Shared paths.** Open, flush, commit, `ReadDurable`, `ReplaceDurable`. Rust `pull_all` RS:1243 also checks `is_absent` on the store and on the lock path before it takes the shared lock; that skips lock creation only, and the verdict stays with the reader. - **Shared paths.** Open, flush, commit, `ReadDurable`, `ReplaceDurable`. Rust `pull_all` RS:1243 also checks `is_absent` on the store and on the lock path before it takes the shared lock; that skips lock creation only, and the verdict stays with the reader.
- **Cut line.** `merge-r1` h2 (C#) and h4/h6 (TS `readStoreBytes`); `mesh-single-file-reader` h1. - **Cut line.** `merge-r1` h2 (C#) and h4/h6 (TS `readStoreBytes`); `mesh-single-file-reader` h1.
- **Pinned by.** C#: `StoreReadabilityTests` `ADanglingFileLinkIsAnIoErrorOnOpenNotAnEmptyStore` :208, `ADanglingDirectoryLinkIsAnIoErrorOnOpenNotAnEmptyStore` :185, and the commit variants :196, :219. Mesh: `ADanglingLinkIsAnIoErrorOnTypedWriteRawWriteAndReadNeverAMissingFile` :107 and `OnlyNothingAtThePathIsAMissingFile` :134 (`CultMeshSingleFileStoreTests.cs`). Rust: `a_dangling_symlink_at_the_store_path_is_an_error_and_nothing_is_written` RS:5832. TS: "readStoreBytes reports nothing at the path only when the lstat finds nothing too". - **Pinned by.** C#: `StoreReadabilityTests` `ADanglingFileLinkIsAnIoErrorOnOpenNotAnEmptyStore` :208, `ADanglingDirectoryLinkIsAnIoErrorOnOpenNotAnEmptyStore` :185, and the commit variants :196, :219. Mesh: `ADanglingLinkIsAnIoErrorOnTypedWriteRawWriteAndReadNeverAMissingFile` :107 and `OnlyNothingAtThePathIsAMissingFile` :134 (`CultMeshSingleFileStoreTests.cs`). Rust: `a_dangling_symlink_at_the_store_path_is_an_error_and_nothing_is_written` RS:5832. TS: "readStoreBytes reports nothing at the path only when the lstat finds nothing too".
- **Known gaps.** The writer no longer tolerates a dangling link: the reader's refusal stands for every write, whole-store `pushAll` included (Rust `a_push_all_through_a_dangling_link_is_refused_like_push_and_writes_nothing`, TS `a pushAll through a dangling link is refused like push`; `variants:follow_up:writer-dangling-link-tolerance`). What a replace does to a live link is `variants:follow_up:store-replace-cut` item 2. Windows junction behavior was proven at `42f05f66` and `b1ff2dcb`, not at the final head. - **Known gaps.** The writer side still tolerates a dangling link: Rust `write_all_unlocked` RS:992-996 and TS `pushAll` TS:94-100 replace it on a whole-store write (`variants:follow_up:writer-dangling-link-tolerance`, owner `c2a-write-set`). What a replace does to a live link is `variants:follow_up:store-replace-cut` item 2. Windows junction behavior was proven at `42f05f66` and `b1ff2dcb`, not at the final head.
### C. Zero-byte file ### C. Zero-byte file
@ -1672,7 +1672,7 @@ The one door per runtime, and who walks through it:
- **Shared paths.** TS `pushAll` and Rust `push_all` only. - **Shared paths.** TS `pushAll` and Rust `push_all` only.
- **Cut line.** `merge-r1` h4 (TS) and h6 (Rust). - **Cut line.** `merge-r1` h4 (TS) and h6 (Rust).
- **Pinned by.** The manifest walks run a soft flush over every vector in TS (the `softPushAll` operation, TS test :2336) and Rust (`soft_push_all`, RS:5149), each with an absent and a dangling-link case; `a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link` RS:6019. - **Pinned by.** The manifest walks run a soft flush over every vector in TS (the `softPushAll` operation, TS test :2336) and Rust (`soft_push_all`, RS:5149), each with an absent and a dangling-link case; `a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link` RS:6019.
- **Known gaps.** C# and Python have no soft decision, and the C# Mesh flag is dead; no follow-up is registered (h6 `undone` names it). `writer-dangling-link-tolerance` is settled: the non-soft branch refuses a dangling link too. - **Known gaps.** C# and Python have no soft decision, and the C# Mesh flag is dead; no follow-up is registered (h6 `undone` names it). `writer-dangling-link-tolerance` applies to the non-soft branch.
### What no cut owns yet ### What no cut owns yet

View file

@ -3,7 +3,7 @@
CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local
tool in `dotnet-tools.json`. Coverage is added project by project; a project tool in `dotnet-tools.json`. Coverage is added project by project; a project
with no `stryker-config.json` next to its test project has not been scoped with no `stryker-config.json` next to its test project has not been scoped
yet (see "Scope and follow-ups" below for what's outstanding). yet (see "Follow-ups" below for what's outstanding).
## Running it ## Running it
@ -126,17 +126,14 @@ project as precedent for the `vstest` caveat itself - that caveat is
`mtp`-runner-specific and documented above for `static readonly` field `mtp`-runner-specific and documented above for `static readonly` field
initializers, and GameCult.Geometry has never actually hit it. initializers, and GameCult.Geometry has never actually hit it.
## Scope and follow-ups ## Follow-ups
CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the CultMath.Tests and GameCult.Geometry.Tests are the only projects currently
projects currently scoped. GameCult.Geometry's `stryker-config.json` has no scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner`
`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and entry: the project is NUnit 4 with `NUnit3TestAdapter` and
`Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies `Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies
directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3 directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3
suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`) suite).
and its `stryker-config.json` is driven the same way, by the repo-local
`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run
dotnet-stryker` from `tests/GameCult.Caching.Tests`.
GameCult.Geometry has only ever been run `--since:`-scoped to the cut that GameCult.Geometry has only ever been run `--since:`-scoped to the cut that
added surface nets; a full-project baseline (running mutation against all of added surface nets; a full-project baseline (running mutation against all of
@ -154,83 +151,7 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that
run; a future cut whose diff is C#-only, against a base commit that already run; a future cut whose diff is C#-only, against a base commit that already
has `stryker-config.json`, should scope correctly. has `stryker-config.json`, should scope correctly.
### GameCult.Caching.Tests `GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of
`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard
mutation level, `break` 0, the same thresholds as Geometry). Caching was
scoped before Networking because its suite is deterministic and quick (1415
tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking
is about 20k lines and its tests lean on socket timing, so a mutant timing out
there says little about the test. A full Caching run is about 2,900 candidate
mutants (3,602 created, minus 730 that fail to compile) at 1.25 to 1.6 s
each, so about 1 to 1.3 hours; run it detached and read the JSON report,
never as one attached session.
The first measurement was a narrow pass over the three files that carry the
store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`:
```bash
dotnet tool run dotnet-stryker \
-m '**/src/GameCult.Caching/StoreReplace.cs' \
-m '**/src/GameCult.Caching/CultStoreLinks.cs' \
-m '**/src/GameCult.Caching/CultElementIds.cs'
```
A clean single run of that command tests 243 to 265 mutants (the count moves with the revision) and scored 85.28% on
568a60fa, before any test of this scope was added; 86.79% on d38b5728, with
the first batch of tests; and 87.17% on 76d9e7d1, with the throwing-enumerator
case, in about 10 minutes. The first measurement of it overlapped two Stryker
runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`,
where tests that kill a mutant by hand reported `Survived` or `Timeout`) are
not measurements and are not quoted here. Never run two Stryker runs against
one checkout at once, and check `ps` before starting one. A clean single run
on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208
`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor
was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now
kills; the other 11 are equivalent mutants. With that case a clean single run
on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%.
Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each
failing when its mutant is applied by hand): the order of the id path (the
codec refuses at the first type MessagePack cannot allow, so order is what an
error names first; `Append` to `Prepend` at 163, 167 and 180); the
union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so
the red is a crashed test host: a depth guard would need production code in
`ConcreteTypes`, since the recursion is private and a stack overflow cannot be
caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an
id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the
`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter,
`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the
`[DataContract]` without `[MessagePackObject]` and `||` arms).
Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full
suite; none can be told apart by any test:
- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to
`true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument
changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument
(`DataContractAttribute` is `Inherited = false`).
- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which
are never empty.
- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial
capacity.
- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`;
`Classify` unwraps nullables itself, so the unwrap is dead. A cut
candidate, not a test gap.
- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and
`MapTypes`; `type` matches only when it is itself `IEnumerable<>`,
`IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second
matching interface, so the position of `type` never decides the answer.
The Windows-only P/Invoke group is exercised only on Windows: its mutants are
`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a
Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its
negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the
negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the
same result.
`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
`GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes `GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes
(`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own (`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own
`stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for `stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for

View file

@ -124,7 +124,7 @@ The test imports from `../src/index` where the block reads `@gamecult/cultcache-
Same shape with `derive(DatabaseEntry)`: slots are `#[cultcache(key = N)]`, so the `.cc` payload is the Same shape with `derive(DatabaseEntry)`: slots are `#[cultcache(key = N)]`, so the `.cc` payload is the
slot-indexed array the C# runtime writes. The source is slot-indexed array the C# runtime writes. The source is
[`packages/cultcache-rs/tests/first_call.rs`](../../packages/cultcache-rs/tests/first_call.rs), run by [`packages/cultcache-rs/tests/first_call.rs`](../../packages/cultcache-rs/tests/first_call.rs), run by
`cargo test --locked --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block `cargo test --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block
below stops matching the file. The Rust runtime has no name lookup, so the read-back is by key. More: below stops matching the file. The Rust runtime has no name lookup, so the read-back is by key. More:
`packages/cultcache-rs/README.md`. `packages/cultcache-rs/README.md`.
@ -341,7 +341,7 @@ async function firstCall() {
### Rust CultNet first call ### Rust CultNet first call
Source: [`packages/cultnet-rs/tests/first_call.rs`](../../packages/cultnet-rs/tests/first_call.rs), run by Source: [`packages/cultnet-rs/tests/first_call.rs`](../../packages/cultnet-rs/tests/first_call.rs), run by
`cargo test --locked --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`. `cargo test --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`.
```rust ```rust
use std::net::{TcpListener, TcpStream}; use std::net::{TcpListener, TcpStream};

View file

@ -107,8 +107,8 @@ Authority map (also in the spec):
- `variants:follow_up:public-store-writes-behind-cache`: **leave**. It asks who may stage behind an attached cache (API - `variants:follow_up:public-store-writes-behind-cache`: **leave**. It asks who may stage behind an attached cache (API
shape); this cut changes what any staging is judged by. No collision: direct `Push`/`CommitBatch` reach the same on-disk shape); this cut changes what any staging is judged by. No collision: direct `Push`/`CommitBatch` reach the same on-disk
judge, so they get the per-key rule for free. Hands must not give direct staging a different predicate. judge, so they get the per-key rule for free. Hands must not give direct staging a different predicate.
- `variants:follow_up:writer-dangling-link-tolerance`: **settled** (the reader owns the verdict; every runtime's whole-store - `variants:follow_up:writer-dangling-link-tolerance`: **leave**. Rust/TS whole-store `push_all` reading a dangling link is
`push_all` refuses a dangling link, as `push` does). a reader/writer tolerance question, not ownership; this cut touches no leaf.
## Deferred doc findings this cut fixes ## Deferred doc findings this cut fixes

6
package-lock.json generated
View file

@ -1205,7 +1205,7 @@
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@gamecult/cultcache-ts": "^0.15.0", "@gamecult/cultcache-ts": "^0.15.0",
"@gamecult/cultnet-ts": "^0.4.0" "@gamecult/cultnet-ts": "^0.3.0"
}, },
"devDependencies": { "devDependencies": {
"@gamecult/cultmesh-ts": "^0.2.0", "@gamecult/cultmesh-ts": "^0.2.0",
@ -1222,7 +1222,7 @@
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@gamecult/cultcache-ts": "^0.15.0", "@gamecult/cultcache-ts": "^0.15.0",
"@gamecult/cultnet-ts": "^0.4.0", "@gamecult/cultnet-ts": "^0.3.0",
"koffi": "3.3.0" "koffi": "3.3.0"
}, },
"devDependencies": { "devDependencies": {
@ -1236,7 +1236,7 @@
}, },
"packages/cultnet-ts": { "packages/cultnet-ts": {
"name": "@gamecult/cultnet-ts", "name": "@gamecult/cultnet-ts",
"version": "0.4.0", "version": "0.3.0",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@gamecult/cultcache-ts": "^0.15.0", "@gamecult/cultcache-ts": "^0.15.0",

View file

@ -10,7 +10,7 @@
"scripts": { "scripts": {
"build:ts": "node scripts/run-typescript-workspaces.mjs build", "build:ts": "node scripts/run-typescript-workspaces.mjs build",
"test:ts": "node scripts/run-typescript-workspaces.mjs test", "test:ts": "node scripts/run-typescript-workspaces.mjs test",
"test:pack:ts": "node --test scripts/test-declared-dependency-ranges.test.mjs && node scripts/test-typescript-package-closure.mjs && node scripts/test-declared-dependency-ranges.mjs", "test:pack:ts": "node scripts/test-typescript-package-closure.mjs && node scripts/test-declared-dependency-ranges.mjs",
"test:eve-browser-network": "node scripts/verify-eve-browser-network.mjs" "test:eve-browser-network": "node scripts/verify-eve-browser-network.mjs"
}, },
"devDependencies": { "devDependencies": {

View file

@ -8,17 +8,11 @@
### Changed ### Changed
- `store_write_lock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. `JsonLinesBackingStore` takes no cross-process lock (only its thread lock, under which it already refuses a link), so it has no lock wait to re-check across. Pinned by a test that puts the link while a foreign process holds the lock.
- `CultCache.is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
- `JsonLinesBackingStore` writes LF line endings on Windows. It wrote CRLF there, through `write_text` newline translation, before the commit_store_file change (#154, adbbb780).
- `msgpack` is a hard dependency, as `pyproject.toml` declared it. The optional-import fallbacks and the `cultcache-py[msgpack]` extra they named (which never existed) are gone: a missing msgpack is an `ImportError` at import. - `msgpack` is a hard dependency, as `pyproject.toml` declared it. The optional-import fallbacks and the `cultcache-py[msgpack]` extra they named (which never existed) are gone: a missing msgpack is an `ImportError` at import.
- `JsonLinesBackingStore` writes through the same `commit_store_file` as the single-file store: a unique staged file, fsynced before the replace and removed when the replace fails, so a refused replace no longer leaves `<path>.tmp` behind. It staged a fixed `<path>.tmp` without an fsync.
- The single-file store's `push`, `push_all` and `delete` take the cross-process lock on `<path>.lock` (flock; LockFileEx over the whole file on Windows) across the read and the replace, waiting up to `lock_timeout` seconds (30 by default, `TimeoutError` after), and exclude the C#, Rust and TypeScript runtimes' writers. The staged file is `<path>.tmp-<pid>-<uuid>`, fsynced before the replace, with a directory fsync after it off Windows. A fixed `<path>.tmp` is no longer used. - The single-file store's `push`, `push_all` and `delete` take the cross-process lock on `<path>.lock` (flock; LockFileEx over the whole file on Windows) across the read and the replace, waiting up to `lock_timeout` seconds (30 by default, `TimeoutError` after), and exclude the C#, Rust and TypeScript runtimes' writers. The staged file is `<path>.tmp-<pid>-<uuid>`, fsynced before the replace, with a directory fsync after it off Windows. A fixed `<path>.tmp` is no longer used.
### Breaking ### Breaking
- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
- A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work.
- A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`. - A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`.
- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#. - A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#.
- A cache holds one record per key whatever its type (`variants:ruling:cache-refuses-retype`). A `put`, `put_envelope`, `put_envelopes` or `put_global` of a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store, and changes nothing; retype a key by deleting it, then putting the new type. A write may replace another registered type's record only when the written type owns its schema id (declares it or lists it compatible) and both types share one home store. A load that reads one key from two stores, whatever the types, is refused with `CultCacheError` naming the key and both types, and admits nothing. A store that holds a record no registered type claims now loads and is carried: the cache does not serve it, the store keeps its bytes through every write, `delete` or `put` at its key is refused with `SchemaConflictError`, and registering a type that declares its schema id claims it at the next load. It raised `CultCacheError` "Unknown persisted document type". Two global types share the key `__global__`, so a second global type in one cache is now refused, and a store that already holds two global types at `__global__` (a JSONL store written by caches that never pulled could produce one) no longer loads: the pull raises `CultCacheError` and admits nothing. C# refuses a second persisted global on pull the same way (`cultcache-store-composition.md`, "Globals never invented"); repair such a file by deleting one of the two records with the runtime that wrote it. - A cache holds one record per key whatever its type (`variants:ruling:cache-refuses-retype`). A `put`, `put_envelope`, `put_envelopes` or `put_global` of a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store, and changes nothing; retype a key by deleting it, then putting the new type. A write may replace another registered type's record only when the written type owns its schema id (declares it or lists it compatible) and both types share one home store. A load that reads one key from two stores, whatever the types, is refused with `CultCacheError` naming the key and both types, and admits nothing. A store that holds a record no registered type claims now loads and is carried: the cache does not serve it, the store keeps its bytes through every write, `delete` or `put` at its key is refused with `SchemaConflictError`, and registering a type that declares its schema id claims it at the next load. It raised `CultCacheError` "Unknown persisted document type". Two global types share the key `__global__`, so a second global type in one cache is now refused, and a store that already holds two global types at `__global__` (a JSONL store written by caches that never pulled could produce one) no longer loads: the pull raises `CultCacheError` and admits nothing. C# refuses a second persisted global on pull the same way (`cultcache-store-composition.md`, "Globals never invented"); repair such a file by deleting one of the two records with the runtime that wrote it.

View file

@ -19,7 +19,7 @@ from .documents import (
read_payload_tree, read_payload_tree,
) )
from .leaves import SlotCodec from .leaves import SlotCodec
from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreLinkedError, StoreUnreadableError from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreUnreadableError
__all__ = [ __all__ = [
"BackingStore", "BackingStore",
@ -38,7 +38,6 @@ __all__ = [
"SchemaConflictError", "SchemaConflictError",
"SingleFileMessagePackBackingStore", "SingleFileMessagePackBackingStore",
"SlotCodec", "SlotCodec",
"StoreLinkedError",
"StoreUnreadableError", "StoreUnreadableError",
"UnionArm", "UnionArm",
"UnreadSlots", "UnreadSlots",

View file

@ -67,20 +67,10 @@ class BackingStore(Protocol):
def delete(self, type: str, key: str) -> None: def delete(self, type: str, key: str) -> None:
... ...
def pull_or_absent(self) -> list[CultCacheEnvelope] | None: def push_all(self, envelopes: list[CultCacheEnvelope]) -> None:
"""The envelopes the reader reads (empty included), None when nothing is at the path; raises when the path cannot be read. existing = {
(envelope.type, envelope.key): envelope for envelope in self.pull_all()
The default is pull_all, which cannot say absent: a store that inherits it is always a store the reader read. A store kind }
that can tell nothing-at-the-path overrides this, and the default push_all then writes where nothing is on a soft push."""
return self.pull_all()
def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None:
# Soft: write only where pull_or_absent says nothing is; skip over a store it read, empty included; its raise refuses.
pulled = self.pull_or_absent()
if soft and pulled is not None:
return
pulled = pulled or []
existing = {(envelope.type, envelope.key): envelope for envelope in pulled}
for envelope in envelopes: for envelope in envelopes:
existing[(envelope.type, envelope.key)] = envelope existing[(envelope.type, envelope.key)] = envelope
self._replace_all(list(existing.values())) self._replace_all(list(existing.values()))

View file

@ -636,8 +636,7 @@ class CultCache:
a record held read-only (its payload carried slots past the declared ones, at any depth, and nothing consents to the loss, so a record held read-only (its payload carried slots past the declared ones, at any depth, and nothing consents to the loss, so
writing it would drop data a newer writer stored), for a key a store carries foreign, and for a key held as another type's writing it would drop data a newer writer stored), for a key a store carries foreign, and for a key held as another type's
record this document does not own. Advisory (the write still decides): it answers at call time, and a key nothing holds is record this document does not own. Advisory (the write still decides): it answers at call time, and a key nothing holds is
writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it with writable."""
`StoreLinkedError`."""
self._assert_registered(document) self._assert_registered(document)
try: try:
self._refuse_at(document, key) self._refuse_at(document, key)

View file

@ -76,17 +76,10 @@ class JsonLinesBackingStore:
self._lock = threading.RLock() self._lock = threading.RLock()
def pull_all(self) -> list[CultCacheEnvelope]: def pull_all(self) -> list[CultCacheEnvelope]:
return self._pull() or []
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
return self._pull()
def _pull(self) -> list[CultCacheEnvelope] | None:
"""The envelopes the reader reads, or None when nothing is at the path."""
with self._lock: with self._lock:
text = _read_store(self.path, lambda path: path.read_text(encoding="utf-8")) text = _read_store(self.path, lambda path: path.read_text(encoding="utf-8"))
if text is None: if text is None:
return None return []
envelopes: list[CultCacheEnvelope] = [] envelopes: list[CultCacheEnvelope] = []
for line_number, line in enumerate(text.splitlines(), start=1): for line_number, line in enumerate(text.splitlines(), start=1):
if not line.strip(): if not line.strip():
@ -108,31 +101,26 @@ class JsonLinesBackingStore:
return envelopes return envelopes
def push(self, envelope: CultCacheEnvelope) -> None: def push(self, envelope: CultCacheEnvelope) -> None:
self.push_all([envelope]) with self._lock:
existing = {(item.type, item.key): item for item in self.pull_all()}
existing[(envelope.type, envelope.key)] = envelope
self._replace_all(list(existing.values()))
def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None: def push_all(self, envelopes: list[CultCacheEnvelope]) -> None:
def merge(existing: dict[tuple[str, str], CultCacheEnvelope]) -> None: with self._lock:
existing = {(item.type, item.key): item for item in self.pull_all()}
for envelope in envelopes: for envelope in envelopes:
existing[(envelope.type, envelope.key)] = envelope existing[(envelope.type, envelope.key)] = envelope
self._rewrite(merge, soft) self._replace_all(list(existing.values()))
def delete(self, type: str, key: str) -> None: def delete(self, type: str, key: str) -> None:
self._rewrite(lambda existing: existing.pop((type, key), None))
def _rewrite(self, change: Callable[[dict[tuple[str, str], CultCacheEnvelope]], object], soft: bool = False) -> None:
"""The one entry of every write: refuses a link at the path, then reads the file (a soft write stops here when the
reader read it), applies the change and replaces it."""
with self._lock: with self._lock:
_refuse_linked(self.path) existing = [item for item in self.pull_all() if not (item.type == type and item.key == key)]
arrived = self._pull() self._replace_all(existing)
if soft and arrived is not None:
return
existing = {(item.type, item.key): item for item in arrived or []}
change(existing)
self._replace_all(list(existing.values()))
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None: def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True) self.path.parent.mkdir(parents=True, exist_ok=True)
temp = self.path.with_suffix(self.path.suffix + ".tmp")
lines = [] lines = []
for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)): for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)):
lines.append(json.dumps({ lines.append(json.dumps({
@ -141,7 +129,8 @@ class JsonLinesBackingStore:
"payload": base64.b64encode(envelope.payload).decode("ascii"), "payload": base64.b64encode(envelope.payload).decode("ascii"),
"stored_at": envelope.stored_at, "stored_at": envelope.stored_at,
}, ensure_ascii=False, sort_keys=True)) }, ensure_ascii=False, sort_keys=True))
commit_store_file(self.path, ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8")) temp.write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8")
temp.replace(self.path)
class SingleFileMessagePackBackingStore: class SingleFileMessagePackBackingStore:
@ -153,20 +142,13 @@ class SingleFileMessagePackBackingStore:
self._format = STORE_FORMAT_VERSION self._format = STORE_FORMAT_VERSION
def pull_all(self) -> list[CultCacheEnvelope]: def pull_all(self) -> list[CultCacheEnvelope]:
return self._pull() or []
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
return self._pull()
def _pull(self) -> list[CultCacheEnvelope] | None:
"""The envelopes the reader reads, or None when nothing is at the path."""
with self._lock: with self._lock:
# An empty file is not a store (no writer leaves one), so it is decoded and refused. # An empty file is not a store (no writer leaves one), so it is decoded and refused.
# The disk decides the header: a file that is gone is not marked. # The disk decides the header: a file that is gone is not marked.
self._format = STORE_FORMAT_VERSION self._format = STORE_FORMAT_VERSION
data = _read_store(self.path, Path.read_bytes) data = _read_store(self.path, Path.read_bytes)
if data is None: if data is None:
return None return []
try: try:
decoded = msgpack.unpackb(data, raw=False) decoded = msgpack.unpackb(data, raw=False)
snapshot = _decode_snapshot(decoded) snapshot = _decode_snapshot(decoded)
@ -181,14 +163,9 @@ class SingleFileMessagePackBackingStore:
with self._write_transaction(): with self._write_transaction():
self._replace_all([envelope], self.pull_all()) self._replace_all([envelope], self.pull_all())
def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None: def push_all(self, envelopes: list[CultCacheEnvelope]) -> None:
"""A soft push writes only where nothing is at the path: a store the reader reads is left byte for byte, and a file it
cannot read is refused by the same read, as in TS and Rust."""
with self._write_transaction(): with self._write_transaction():
arrived = self._pull() self._replace_all(list(envelopes), self.pull_all())
if soft and arrived is not None:
return
self._replace_all(list(envelopes), arrived or [])
def delete(self, type: str, key: str) -> None: def delete(self, type: str, key: str) -> None:
with self._write_transaction(): with self._write_transaction():
@ -346,7 +323,7 @@ def commit_store_file(path: Path, data: bytes) -> None:
os.fsync(fd) os.fsync(fd)
finally: finally:
os.close(fd) os.close(fd)
_replace_store(staged, path) os.replace(staged, path)
except BaseException: except BaseException:
try: try:
os.unlink(staged) os.unlink(staged)
@ -361,136 +338,11 @@ def commit_store_file(path: Path, data: bytes) -> None:
os.close(directory) os.close(directory)
REPLACE_ATTEMPTS = 25
REPLACE_RETRY_SECONDS = 0.02
_ERROR_ACCESS_DENIED = 5
_ERROR_SHARING_VIOLATION = 32
_ERROR_LOCK_VIOLATION = 33
_POSIX_UNSUPPORTED = (1, 50, 87) # incorrect function, not supported, invalid parameter: FAT, exFAT, before Windows 10 1607
MOVEFILE_REPLACE_WRITE_THROUGH = 0x9 # MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH
class ReplaceKernel:
"""The kernel32 calls a Windows store replace needs, each answering 0 or a Win32 error code."""
def rename_posix(self, staged: str, destination: str) -> int:
"""CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle."""
raise NotImplementedError
def move_file_ex(self, staged: str, destination: str, flags: int) -> int:
raise NotImplementedError
def sleep(self, seconds: float) -> None:
time.sleep(seconds)
def _replace_store(staged: str | os.PathLike[str], destination: str | os.PathLike[str]) -> None:
"""Replaces `destination` with `staged`: POSIX-semantics rename first on Windows (so a reader holding the store open
keeps it and rarely sees it absent; a stalled rename can still leave it absent for 5-47 ms on NTFS,
variants:ruling:posix-rename-absent-window-document), os.replace elsewhere."""
if sys.platform == "win32":
replace_store_windows(os.fspath(staged), os.fspath(destination), WindowsReplaceKernel())
else:
os.replace(staged, destination)
def replace_store_windows(staged: str, destination: str, kernel: ReplaceKernel) -> None:
"""POSIX-semantics rename, falling back for the rest of the call to write-through MoveFileExW where POSIX semantics are
unsupported; errors 5, 32 and 33 retry up to 25 times 20 ms apart, any other code raises at once. The TypeScript and C#
runtimes take the same decision table."""
posix = True
last = 0
for _ in range(REPLACE_ATTEMPTS):
code = kernel.rename_posix(staged, destination) if posix else kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH)
if posix and code in _POSIX_UNSUPPORTED:
posix = False
code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH)
if code == 0:
return
if code not in (_ERROR_ACCESS_DENIED, _ERROR_SHARING_VIOLATION, _ERROR_LOCK_VIOLATION):
raise _replace_error(f"CultCache could not replace the store: Win32 error {code}.", code)
last = code
kernel.sleep(REPLACE_RETRY_SECONDS)
raise _replace_error(f"CultCache could not replace the store after {REPLACE_ATTEMPTS} attempts: last Win32 error {last}.", last)
def _replace_error(message: str, code: int) -> OSError:
error = OSError(message)
error.winerror = code # type: ignore[attr-defined]
return error
class WindowsReplaceKernel(ReplaceKernel):
"""kernel32 bound by ctypes, as _try_lock_windows binds it. Windows only."""
def __init__(self) -> None:
import ctypes
from ctypes import wintypes
self._ctypes = ctypes
k = self._k = ctypes.WinDLL("kernel32", use_last_error=True)
k.CreateFileW.restype = wintypes.HANDLE
k.CreateFileW.argtypes = [wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p, wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE]
k.SetFileInformationByHandle.restype = wintypes.BOOL
k.SetFileInformationByHandle.argtypes = [wintypes.HANDLE, ctypes.c_int, ctypes.c_void_p, wintypes.DWORD]
k.CloseHandle.restype = wintypes.BOOL
k.CloseHandle.argtypes = [wintypes.HANDLE]
k.MoveFileExW.restype = wintypes.BOOL
k.MoveFileExW.argtypes = [wintypes.LPCWSTR, wintypes.LPCWSTR, wintypes.DWORD]
def rename_posix(self, staged: str, destination: str) -> int:
ctypes, k = self._ctypes, self._k
delete_synchronize, share_all, open_existing, normal = 0x00110000, 0x7, 3, 0x80
handle = k.CreateFileW(staged, delete_synchronize, share_all, None, open_existing, normal, None)
if handle is None or ctypes.c_void_p(handle).value in (None, ctypes.c_void_p(-1).value):
return ctypes.get_last_error()
try:
# FILE_RENAME_INFO: Flags (DWORD), RootDirectory (pointer-aligned HANDLE), FileNameLength (DWORD), FileName (WCHAR[]).
name = os.path.abspath(destination).encode("utf-16-le")
pointer = ctypes.sizeof(ctypes.c_void_p)
length_at = pointer + pointer
info = ctypes.create_string_buffer(length_at + 4 + len(name) + 2)
info[0:4] = (0x3).to_bytes(4, "little") # FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS
info[length_at:length_at + 4] = len(name).to_bytes(4, "little")
info[length_at + 4:length_at + 4 + len(name)] = name
file_rename_info_ex = 22
if k.SetFileInformationByHandle(handle, file_rename_info_ex, info, len(info)):
return 0
return ctypes.get_last_error()
finally:
k.CloseHandle(handle)
def move_file_ex(self, staged: str, destination: str, flags: int) -> int:
return 0 if self._k.MoveFileExW(staged, destination, flags) else self._ctypes.get_last_error()
class StoreLinkedError(OSError):
"""A write was asked of a store path that is a link: a replace there would swap the link for a regular file."""
def __init__(self, path: Path) -> None:
super().__init__(f"CultCache store {path} is a link, and a write at it would replace the link; configure the link's target as the store path.")
self.path = path
def _refuse_linked(path: Path) -> None:
"""Refuses a link at the store path: a file or directory link, and on Windows a junction or any reparse point (islink misses
junctions before Python 3.12). Absent is not a link."""
linked = os.path.islink(path)
if not linked and sys.platform == "win32":
try:
linked = bool(os.lstat(path).st_file_attributes & 0x400) # FILE_ATTRIBUTE_REPARSE_POINT
except FileNotFoundError:
pass
if linked:
raise StoreLinkedError(path)
@contextmanager @contextmanager
def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Iterator[None]: def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Iterator[None]:
"""Holds the exclusive store lock on `<path>.lock`, the lock the C#, Rust and TypeScript runtimes take: flock on Unix, """Holds the exclusive store lock on `<path>.lock`, the lock the C#, Rust and TypeScript runtimes take: flock on Unix,
LockFileEx over the whole file on Windows, polled every 10 ms with no handle held between polls. The lock file is LockFileEx over the whole file on Windows, polled every 10 ms with no handle held between polls. The lock file is
created if absent and never written or deleted.""" created if absent and never written or deleted."""
_refuse_linked(path)
lock_path = path.with_name(path.name + ".lock") lock_path = path.with_name(path.name + ".lock")
deadline = time.monotonic() + timeout deadline = time.monotonic() + timeout
while True: while True:
@ -501,8 +353,6 @@ def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Itera
raise TimeoutError(f"CultCache timed out after {timeout} s waiting for the store lock {lock_path}.") raise TimeoutError(f"CultCache timed out after {timeout} s waiting for the store lock {lock_path}.")
time.sleep(_LOCK_POLL_SECONDS) time.sleep(_LOCK_POLL_SECONDS)
try: try:
# A link may have been put at the path while the lock was awaited; the look above only saw the path before.
_refuse_linked(path)
yield yield
finally: finally:
release() release()

View file

@ -12,7 +12,6 @@ from cultcache_py import (
CultCache, CultCache,
JsonLinesBackingStore, JsonLinesBackingStore,
SingleFileMessagePackBackingStore, SingleFileMessagePackBackingStore,
StoreLinkedError,
StoreUnreadableError, StoreUnreadableError,
define_database_entry_type, define_database_entry_type,
define_document_type, define_document_type,
@ -1432,10 +1431,10 @@ class CultCacheTests(unittest.TestCase):
self.assertNotIn("SECRET", str(caught.exception)) self.assertNotIn("SECRET", str(caught.exception))
self.assertIn("not a valid store", str(caught.exception)) self.assertIn("not a valid store", str(caught.exception))
# A write at a store path that is a link is refused: a replace there would swap the link for a regular file and # What a write does through a symbolic link today. R3 decides whether a write resolves
# detach the store from its target. Reads through a live link keep working. # the link or refuses it; this pins the current behaviour so that change is deliberate.
@unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here")
def test_a_write_at_a_live_link_is_refused_and_nothing_changes_while_a_read_through_it_works(self) -> None: def test_r3_decides_resolve_or_refuse_a_push_through_a_live_link_replaces_the_link(self) -> None:
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
volume = Path(tmp) / "volume" volume = Path(tmp) / "volume"
volume.mkdir() volume.mkdir()
@ -1443,91 +1442,12 @@ class CultCacheTests(unittest.TestCase):
envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z") envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")
SingleFileMessagePackBackingStore(target).push(envelope) SingleFileMessagePackBackingStore(target).push(envelope)
before = target.read_bytes() before = target.read_bytes()
volume_before = sorted(p.name for p in volume.iterdir()) link = Path(tmp) / "store.cc"
link = Path(tmp) / "link.cc"
link.symlink_to(target) link.symlink_to(target)
store = SingleFileMessagePackBackingStore(link) SingleFileMessagePackBackingStore(link).push(replace(envelope, key="b"))
for write in ( self.assertFalse(link.is_symlink())
lambda: store.push(replace(envelope, key="b")), self.assertTrue(link.is_file())
lambda: store.push_all([replace(envelope, key="b")]),
lambda: store.delete("t", "a"),
):
with self.assertRaises(StoreLinkedError) as caught:
write()
self.assertIn(str(link), str(caught.exception))
self.assertTrue(link.is_symlink())
self.assertEqual(target.read_bytes(), before) self.assertEqual(target.read_bytes(), before)
self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.cc", "volume"])
self.assertEqual(sorted(p.name for p in volume.iterdir()), volume_before)
self.assertEqual([item.key for item in store.pull_all()], ["a"])
@unittest.skipIf(os.name != "posix", "symbolic links need POSIX here")
def test_a_json_lines_write_at_a_live_link_is_refused_and_the_link_survives(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
volume = Path(tmp) / "volume"
volume.mkdir()
target = volume / "store.jsonl"
envelope = CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")
JsonLinesBackingStore(target).push(envelope)
before = target.read_bytes()
link = Path(tmp) / "link.jsonl"
link.symlink_to(target)
store = JsonLinesBackingStore(link)
for write in (
lambda: store.push(replace(envelope, key="b")),
lambda: store.push_all([replace(envelope, key="b")]),
lambda: store.delete("t", "a"),
):
with self.assertRaises(StoreLinkedError):
write()
self.assertTrue(link.is_symlink())
self.assertEqual(target.read_bytes(), before)
self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.jsonl", "volume"])
self.assertEqual(sorted(p.name for p in volume.iterdir()), ["store.jsonl"])
# The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse.
@unittest.skipIf(os.name != "posix", "symbolic links need POSIX here")
def test_is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
volume = Path(tmp) / "volume"
volume.mkdir()
target = volume / "store.cc"
document = item_doc()
seed = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(target)).build()
seed.put(document, "held", Item(name="Held", category="c", value=1))
before = target.read_bytes()
link = Path(tmp) / "link.cc"
link.symlink_to(target)
cache = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(link)).build()
cache.pull_all_backing_stores()
names_before = (sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir()))
self.assertTrue(cache.is_writable(document, "held"))
self.assertTrue(cache.is_writable(document, "fresh"))
with self.assertRaises(StoreLinkedError) as caught:
cache.put(document, "fresh", Item(name="Fresh", category="d", value=2))
self.assertEqual(Path(caught.exception.path), link)
self.assertTrue(link.is_symlink())
self.assertEqual(target.read_bytes(), before)
self.assertEqual((sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir())), names_before)
def test_store_linked_error_is_an_os_error_carrying_the_path(self) -> None:
error = StoreLinkedError(Path("some/store.cc"))
self.assertIsInstance(error, OSError)
self.assertEqual(error.path, Path("some/store.cc"))
@unittest.skipIf(os.name != "posix", "symbolic links need POSIX here")
def test_a_write_at_a_live_directory_link_is_refused_as_a_link(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
real = Path(tmp) / "real"
real.mkdir()
link = Path(tmp) / "dirlink"
link.symlink_to(real, target_is_directory=True)
with self.assertRaises(StoreLinkedError):
SingleFileMessagePackBackingStore(link).push(
CultCacheEnvelope(key="a", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z"))
self.assertTrue(link.is_symlink())
self.assertEqual(list(real.iterdir()), [])
@unittest.skipIf(os.name != "posix", "symbolic links need POSIX here") @unittest.skipIf(os.name != "posix", "symbolic links need POSIX here")
def test_a_dangling_symlink_at_a_store_path_is_an_os_error_and_nothing_is_written(self) -> None: def test_a_dangling_symlink_at_a_store_path_is_an_os_error_and_nothing_is_written(self) -> None:
@ -1540,7 +1460,7 @@ class CultCacheTests(unittest.TestCase):
store = store_type(path) store = store_type(path)
with self.assertRaises(FileNotFoundError): with self.assertRaises(FileNotFoundError):
store.pull_all() store.pull_all()
with self.assertRaises(StoreLinkedError): with self.assertRaises(FileNotFoundError):
store.push(CultCacheEnvelope(key="k", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z")) store.push(CultCacheEnvelope(key="k", type="t", payload=b"\x90", stored_at="2026-09-30T00:00:00Z"))
self.assertTrue(path.is_symlink()) self.assertTrue(path.is_symlink())
self.assertEqual(list(volume.iterdir()), []) self.assertEqual(list(volume.iterdir()), [])

View file

@ -16,7 +16,6 @@ from cultcache_py import (
SlotCodec, SlotCodec,
SchemaConflictError, SchemaConflictError,
SingleFileMessagePackBackingStore, SingleFileMessagePackBackingStore,
StoreUnreadableError,
UnionArm, UnionArm,
UnreadSlots, UnreadSlots,
database_entry_field, database_entry_field,
@ -179,22 +178,6 @@ class StoreRowTests(unittest.TestCase):
return slot_unit_document return slot_unit_document
return declaring_document if store == "extra-slot-consent.cc" else surface_document return declaring_document if store == "extra-slot-consent.cc" else surface_document
def test_the_variant_row_is_refused_as_a_store_not_walked_and_the_rest_of_the_store_reads_without_it(self) -> None:
# This runtime holds no variant records (ruling: refuse, not carry; the reference's marked-variant row, the one
# row with a flattenRefusedBy, is the C# reader's). A five-slot persisted record makes the whole store
# unreadable, the file is left as it was, and the store without exactly that row reads.
variants = [row for row in _rows() if row[8] is not None]
self.assertEqual([(row[0], row[1], row[8]) for row in variants], [("extra-slot-store.cc", "marked-variant", "marked")])
path = self.tmp / "extra-slot-store.cc"
shutil.copyfile(VECTORS / "extra-slot-store.cc", path)
before = path.read_bytes()
with self.assertRaises(StoreUnreadableError):
_open(path, surface_document, slot_unit_document)
self.assertEqual(path.read_bytes(), before)
_, cache = self._store("extra-slot-store.cc")
self.assertIsNone(cache.get(surface_document, "marked-variant"))
self.assertIsNotNone(cache.get(surface_document, "marked"))
def test_every_row_is_writable_exactly_where_the_vector_says_and_refuses_with_its_mark(self) -> None: def test_every_row_is_writable_exactly_where_the_vector_says_and_refuses_with_its_mark(self) -> None:
walked = 0 walked = 0
for store, key, writable, explained, type_name, carried, declared, occurrences, flatten in _rows(): for store, key, writable, explained, type_name, carried, declared, occurrences, flatten in _rows():

View file

@ -1,140 +0,0 @@
from __future__ import annotations
import shutil
import tempfile
import unittest
from pathlib import Path
from cultcache_py import JsonLinesBackingStore, SingleFileMessagePackBackingStore, StoreUnreadableError
from cultcache_py.backing_store import BackingStore, CultCacheEnvelope
def envelope(key: str) -> CultCacheEnvelope:
return CultCacheEnvelope(key=key, type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z")
class SoftPushAllTests(unittest.TestCase):
"""A soft push writes where nothing is, skips a store it reads, and refuses one it cannot read (TS and Rust parity)."""
def setUp(self) -> None:
self.dir = Path(tempfile.mkdtemp())
self.addCleanup(shutil.rmtree, self.dir, ignore_errors=True)
self.path = self.dir / "state.cc"
def test_over_a_readable_store_it_leaves_the_file_byte_for_byte(self) -> None:
for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
with self.subTest(make.__name__):
path = self.dir / make.__name__
store = make(path)
store.push_all([envelope("a")])
before = path.read_bytes()
store.push_all([envelope("b")], soft=True)
self.assertEqual(before, path.read_bytes())
self.assertEqual(["a"], [item.key for item in make(path).pull_all()])
store.push_all([envelope("b")]) # a plain push still lands what the caller holds
self.assertEqual(["a", "b"], sorted(item.key for item in make(path).pull_all()))
def test_over_a_readable_store_holding_no_records_it_leaves_the_file_byte_for_byte(self) -> None:
for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
with self.subTest(make.__name__):
path = self.dir / make.__name__
store = make(path)
store.push_all([envelope("a")])
store.delete("t", "a")
self.assertTrue(path.exists())
self.assertEqual([], make(path).pull_all())
before = path.read_bytes()
store.push_all([envelope("b")], soft=True)
self.assertEqual(before, path.read_bytes())
self.assertEqual([], make(path).pull_all())
def test_over_an_absent_path_it_writes(self) -> None:
for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
with self.subTest(make.__name__):
path = self.dir / make.__name__
make(path).push_all([envelope("a")], soft=True)
self.assertEqual(["a"], [item.key for item in make(path).pull_all()])
def test_over_an_unreadable_file_it_refuses_and_leaves_it(self) -> None:
junk = b"\xc1\x01\x02\x03"
self.path.write_bytes(junk)
with self.assertRaises(StoreUnreadableError):
SingleFileMessagePackBackingStore(self.path).push_all([envelope("a")], soft=True)
self.assertEqual(junk, self.path.read_bytes())
def test_over_an_unreadable_jsonl_file_it_refuses_and_leaves_it(self) -> None:
junk = b"this is not json\n"
self.path.write_bytes(junk)
with self.assertRaises(ValueError):
JsonLinesBackingStore(self.path).push_all([envelope("a")], soft=True)
self.assertEqual(junk, self.path.read_bytes())
def test_the_protocol_default_push_all_decides_soft_by_what_pull_or_absent_says(self) -> None:
class Memory(BackingStore):
def __init__(self) -> None:
self.items: list[CultCacheEnvelope] | None = None # None: nothing at the path
def pull_all(self) -> list[CultCacheEnvelope]:
return list(self.items or [])
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
return None if self.items is None else list(self.items)
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.items = list(envelopes)
store = Memory()
store.push_all([envelope("a")], soft=True) # nothing at the path: writes
self.assertEqual(["a"], [item.key for item in store.items or []])
store.push_all([envelope("b")], soft=True) # a store with records: skips
self.assertEqual(["a"], [item.key for item in store.items or []])
store.items = []
store.push_all([envelope("b")], soft=True) # a store the reader read, empty: skips
self.assertEqual([], store.items)
store.push_all([envelope("b")]) # plain: lands
self.assertEqual(["b"], [item.key for item in store.items])
def test_a_store_that_only_pulls_all_is_read_so_a_soft_push_skips_it(self) -> None:
class PullOnly(BackingStore):
def __init__(self) -> None:
self.items: list[CultCacheEnvelope] = []
def pull_all(self) -> list[CultCacheEnvelope]:
return list(self.items)
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.items = list(envelopes)
store = PullOnly()
store.push_all([envelope("a")], soft=True)
self.assertEqual([], store.items)
def test_the_single_file_and_jsonl_stores_say_absent_and_read_through_pull_or_absent(self) -> None:
for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
with self.subTest(make.__name__):
path = self.dir / make.__name__
store = make(path)
self.assertIsNone(store.pull_or_absent())
store.push_all([envelope("a")])
self.assertEqual(["a"], [item.key for item in store.pull_or_absent() or []])
def test_the_protocol_default_push_all_refuses_where_its_pull_refuses(self) -> None:
class Unreadable(BackingStore):
replaced = False
def pull_all(self) -> list[CultCacheEnvelope]:
raise StoreUnreadableError(Path("x"), ValueError("junk"))
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
raise StoreUnreadableError(Path("x"), ValueError("junk"))
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
Unreadable.replaced = True
with self.assertRaises(StoreUnreadableError):
Unreadable().push_all([envelope("a")], soft=True)
self.assertFalse(Unreadable.replaced)
if __name__ == "__main__":
unittest.main()

View file

@ -1,25 +1,21 @@
from __future__ import annotations from __future__ import annotations
import os import os
import select
import shutil import shutil
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
import threading
import time import time
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
from cultcache_py import SingleFileMessagePackBackingStore, stores from cultcache_py import SingleFileMessagePackBackingStore, stores
from cultcache_py.stores import StoreLinkedError, store_write_lock
from cultcache_py.backing_store import CultCacheEnvelope from cultcache_py.backing_store import CultCacheEnvelope
REPO = Path(__file__).resolve().parents[3] REPO = Path(__file__).resolve().parents[3]
TS_STORE_FILE = REPO / "packages" / "cultcache-ts" / "dist" / "store-file.js" TS_STORE_FILE = REPO / "packages" / "cultcache-ts" / "dist" / "store-file.js"
HOLD_SECONDS = 1.5 HOLD_SECONDS = 1.5
HELD_WAIT_SECONDS = 3
# A holder that takes the lock the way the Rust runtime does (an exclusive file lock), not through this package. # A holder that takes the lock the way the Rust runtime does (an exclusive file lock), not through this package.
PY_HOLDER = """ PY_HOLDER = """
@ -45,14 +41,7 @@ def envelope(key: str) -> CultCacheEnvelope:
def start_holder(argv: list[str]) -> subprocess.Popen[str]: def start_holder(argv: list[str]) -> subprocess.Popen[str]:
holder = subprocess.Popen(argv, stdout=subprocess.PIPE, text=True) holder = subprocess.Popen(argv, stdout=subprocess.PIPE, text=True)
assert holder.stdout is not None assert holder.stdout is not None and holder.stdout.readline().strip() == "HELD"
ready, _, _ = select.select([holder.stdout], [], [], HELD_WAIT_SECONDS) # a holder blocked on a leaked lock never prints
if not ready:
holder.kill()
holder.wait()
holder.stdout.close()
raise AssertionError(f"the holder never took {argv[-2]}.lock within {HELD_WAIT_SECONDS} s: an earlier writer leaked the lock")
assert holder.stdout.readline().strip() == "HELD"
holder.held_at = time.monotonic() # type: ignore[attr-defined] holder.held_at = time.monotonic() # type: ignore[attr-defined]
return holder return holder
@ -84,48 +73,6 @@ class StoreLockTests(unittest.TestCase):
holder = start_holder(["node", "-e", TS_HOLDER, str(TS_STORE_FILE), str(self.path), str(HOLD_SECONDS)]) holder = start_holder(["node", "-e", TS_HOLDER, str(TS_STORE_FILE), str(self.path), str(HOLD_SECONDS)])
self.assert_write_waits_for(holder) self.assert_write_waits_for(holder)
def test_a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it(self) -> None:
target = self.dir / "volume" / "store.cc"
target.parent.mkdir()
SingleFileMessagePackBackingStore(target).push(envelope("a"))
before = target.read_bytes()
writes = {
"push": lambda store: store.push(envelope("b")),
"push_all": lambda store: store.push_all([envelope("b")]),
"delete": lambda store: store.delete("t", "a"),
}
missing = self.dir / "volume" / "missing.cc"
for (name, write), link_to in [(item, to) for item in writes.items() for to in (target, missing)]:
with self.subTest(name, dangling=link_to == missing):
if os.path.lexists(self.path):
os.unlink(self.path)
holder = start_holder([sys.executable, "-c", PY_HOLDER, str(self.path), str(HOLD_SECONDS)])
self.addCleanup(holder.wait)
self.addCleanup(holder.stdout.close) # type: ignore[union-attr]
outcome: list[BaseException | None] = []
def run() -> None:
try:
write(SingleFileMessagePackBackingStore(self.path, lock_timeout=10))
outcome.append(None)
except BaseException as exc: # noqa: BLE001 - the thread hands the outcome to the test
outcome.append(exc)
writer = threading.Thread(target=run)
writer.start()
time.sleep(0.3)
os.symlink(link_to, self.path) # after the writer's pre-lock look, while it waits
writer.join()
holder.wait()
self.assertEqual(1, len(outcome))
self.assertIsInstance(outcome[0], StoreLinkedError)
self.assertTrue(os.path.islink(self.path), "the link was replaced")
self.assertEqual(before, target.read_bytes())
self.assertFalse(missing.exists(), "the dangling target was created")
os.unlink(self.path) # a link is refused before the lock, so remove it to probe the writer's own lock file
with store_write_lock(self.path, timeout=0.2): # the lock is free
pass
def test_the_lock_is_released_after_each_write(self) -> None: def test_the_lock_is_released_after_each_write(self) -> None:
store = SingleFileMessagePackBackingStore(self.path, lock_timeout=0.2) store = SingleFileMessagePackBackingStore(self.path, lock_timeout=0.2)
for key in "abc": for key in "abc":

View file

@ -1,137 +0,0 @@
from __future__ import annotations
import os
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from cultcache_py import SingleFileMessagePackBackingStore, stores
from cultcache_py.backing_store import CultCacheEnvelope
from cultcache_py.stores import MOVEFILE_REPLACE_WRITE_THROUGH, ReplaceKernel, replace_store_windows
class FakeKernel(ReplaceKernel):
def __init__(self, posix: list[int], move_ex: list[int]) -> None:
self.posix, self.move_ex = list(posix), list(move_ex)
self.log: list[str] = []
self.sleeps: list[float] = []
def rename_posix(self, staged, destination):
self.log.append("posix")
return self.posix.pop(0) if self.posix else 0
def move_file_ex(self, staged, destination, flags):
self.log.append(f"moveex:{flags}")
return self.move_ex.pop(0) if self.move_ex else 0
def sleep(self, seconds):
self.sleeps.append(seconds)
def run(posix, move_ex):
kernel = FakeKernel(posix, move_ex)
return kernel, lambda: replace_store_windows("a", "b", kernel)
class WindowsReplaceTable(unittest.TestCase):
def test_falls_back_to_write_through_move_where_posix_rename_is_unsupported(self):
self.assertEqual(MOVEFILE_REPLACE_WRITE_THROUGH, 0x9)
for unsupported in (1, 50, 87):
kernel, go = run([unsupported], [0])
go()
self.assertEqual(kernel.log, ["posix", "moveex:9"])
def test_once_on_move_the_rest_of_the_call_stays_there(self):
kernel, go = run([87], [32, 0])
go()
self.assertEqual(kernel.log, ["posix", "moveex:9", "moveex:9"])
def test_transient_codes_retry_the_same_primitive(self):
for transient in (5, 32, 33):
kernel, go = run([transient, transient, 0], [])
go()
self.assertEqual(kernel.log, ["posix", "posix", "posix"])
self.assertEqual(kernel.sleeps, [0.02, 0.02])
def test_a_working_posix_rename_is_the_whole_replace(self):
kernel, go = run([0], [])
go()
self.assertEqual(kernel.log, ["posix"])
self.assertEqual(kernel.sleeps, [])
def test_unsupported_after_the_fallback_is_an_error_not_another_fallback(self):
kernel, go = run([87], [32, 87])
with self.assertRaisesRegex(OSError, "Win32 error 87") as caught:
go()
self.assertEqual(caught.exception.winerror, 87)
self.assertEqual(kernel.log, ["posix", "moveex:9", "moveex:9"])
for unsupported in (1, 50, 87):
kernel, go = run([87], [unsupported])
with self.assertRaisesRegex(OSError, f"Win32 error {unsupported}"):
go()
self.assertEqual(kernel.log, ["posix", "moveex:9"])
def test_gives_up_after_25_attempts_naming_the_last_code(self):
kernel, go = run([32] * 30, [])
with self.assertRaisesRegex(OSError, "after 25 attempts: last Win32 error 32\\."):
go()
self.assertEqual(len(kernel.log), 25)
def test_other_codes_fail_at_once(self):
kernel, go = run([2], [])
with self.assertRaisesRegex(OSError, "Win32 error 2"):
go()
self.assertEqual(len(kernel.log), 1)
kernel, go = run([87], [3])
with self.assertRaisesRegex(OSError, "Win32 error 3"):
go()
class StoreWritesRouteThroughTheReplace(unittest.TestCase):
def test_both_stores_replace_through_replace_store(self):
seen: list[tuple[Path, Path]] = []
real = stores._replace_store
def spy(staged, destination):
seen.append((Path(staged), Path(destination)))
real(staged, destination)
with tempfile.TemporaryDirectory() as directory, mock.patch.object(stores, "_replace_store", spy):
envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z")
binary = Path(directory) / "s.cc"
SingleFileMessagePackBackingStore(binary).push_all([envelope])
lines = Path(directory) / "s.jsonl"
stores.JsonLinesBackingStore(lines).push_all([envelope])
self.assertEqual([destination for _, destination in seen], [binary, lines])
self.assertTrue(binary.read_bytes() and lines.read_text())
def test_a_failed_jsonl_replace_leaves_no_staged_file_and_the_old_store(self):
envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z")
with tempfile.TemporaryDirectory() as directory:
lines = Path(directory) / "s.jsonl"
store = stores.JsonLinesBackingStore(lines)
store.push_all([envelope])
before = lines.read_bytes()
with mock.patch.object(stores, "_replace_store", side_effect=OSError("replace refused")):
with self.assertRaises(OSError):
store.push_all([CultCacheEnvelope(key="k2", type="t", payload=b"\x02", stored_at="2026-11-06T00:00:00Z")])
self.assertEqual(lines.read_bytes(), before)
self.assertEqual([p.name for p in Path(directory).iterdir() if ".tmp" in p.name], [])
@unittest.skipUnless(sys.platform == "win32", "needs a real Windows")
def test_real_kernel_replaces_through_both_primitives(self):
kernel = stores.WindowsReplaceKernel()
with tempfile.TemporaryDirectory() as directory:
for name, replace in (("posix", kernel.rename_posix), ("moveex", lambda s, d: kernel.move_file_ex(s, d, MOVEFILE_REPLACE_WRITE_THROUGH))):
staged, store = Path(directory, name + ".tmp"), Path(directory, name)
store.write_bytes(b"old")
staged.write_bytes(b"new")
self.assertEqual(replace(str(staged), str(store)), 0)
self.assertEqual(store.read_bytes(), b"new")
self.assertEqual(kernel.rename_posix(str(Path(directory, "missing")), str(Path(directory, "x"))), 2)
if __name__ == "__main__":
unittest.main()

View file

@ -1 +1,2 @@
/target /target
Cargo.lock

View file

@ -6,16 +6,10 @@ only in git history.
## Unreleased ## Unreleased
### Changed
- The single-file store re-checks for a link once the store lock is held (every write and the try-exchange), so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, as C# does, instead of being replaced. Pinned by a test that puts the link while a second holder owns the lock.
- `CultCache::is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
### Breaking ### Breaking
Semver: this is a breaking change in the 0.x line (0.3.x to 0.4.0). Semver: this is a breaking change in the 0.x line (0.3.x to 0.4.0).
- A write at a store path that is a symbolic link, live or dangling, is refused with the new public `StoreLinkedError` (`error.downcast_ref::<StoreLinkedError>()`) before the lock and the replace, and writes nothing: `push`, `push_all`, `delete`, the exchanges, and a soft `push_all`, which used to skip a link and now refuses. Before, a write at a live link replaced the link with a regular file and a dangling link failed with `io::ErrorKind::NotFound`. The redb stores (`RedbMessagePackBackingStore`, `OwnedRedbMessagePackBackingStore`) are exempt: they are ruled migrated and deleted (`paged-store-redb-migrate-and-delete`) and still open a live link's target. Configure the link's target as the store path. Reads through a live link work.
- A record whose payload carries unread slots is read-only. A payload with a slot past its type's declared slots, at any depth (record, nested record, list element, union arm, unit arm), still decodes, and every write or removal of its key is now refused before any store is touched: `put`, `put_envelope`, `put_raw_envelope`, `put_prepared_batch` (the whole batch) and `delete`, so `update` too. The refusal is `SchemaConflictError` with the new `unread_slots`; its message names the type and both slot counts and never a slot value. Holders of such a record used to overwrite it and drop the extra slots silently. The payload nesting refusal still runs first. A held payload this build cannot read at all (an arm it lacks from a newer writer, or a corrupt record) is read-only too: every write or removal is refused with an error naming the type and key (a newer writer's record is read with that build; a corrupt one is restored from a backup or removed at the store itself, as no cache call removes it), and `is_writable` answers false. `load_soa` is a write of every held key it omits or changes, so it refuses to drop or replace a read-only record. `is_writable` also answers false for a key a store carries as a foreign record and for a type other than the one holding the key, as the writes do. Each write over a held key decodes the held payload once to ask (about 40% of a `put` of a 200,000-element record, measured in release); keys nothing holds pay nothing. - A record whose payload carries unread slots is read-only. A payload with a slot past its type's declared slots, at any depth (record, nested record, list element, union arm, unit arm), still decodes, and every write or removal of its key is now refused before any store is touched: `put`, `put_envelope`, `put_raw_envelope`, `put_prepared_batch` (the whole batch) and `delete`, so `update` too. The refusal is `SchemaConflictError` with the new `unread_slots`; its message names the type and both slot counts and never a slot value. Holders of such a record used to overwrite it and drop the extra slots silently. The payload nesting refusal still runs first. A held payload this build cannot read at all (an arm it lacks from a newer writer, or a corrupt record) is read-only too: every write or removal is refused with an error naming the type and key (a newer writer's record is read with that build; a corrupt one is restored from a backup or removed at the store itself, as no cache call removes it), and `is_writable` answers false. `load_soa` is a write of every held key it omits or changes, so it refuses to drop or replace a read-only record. `is_writable` also answers false for a key a store carries as a foreign record and for a type other than the one holding the key, as the writes do. Each write over a held key decodes the held payload once to ask (about 40% of a `put` of a 200,000-element record, measured in release); keys nothing holds pay nothing.
- cultmesh-rs `pull_rudp_catalog_snapshot` applies a snapshot as one cache commit (`put_prepared_batch`), as C# does: a snapshot document for a key the node holds read-only, or any other refusal, applies nothing. It used to apply document by document, so a refusal ended the pull with the earlier documents kept. A snapshot whose types route to different backing stores is now refused whole, as a batch cannot span stores. C# `CultCache.Commit` refuses a cross-store commit the same way (`The batch spans A and B; a commit lands in one home store`), so the two agree; a test pins it. - cultmesh-rs `pull_rudp_catalog_snapshot` applies a snapshot as one cache commit (`put_prepared_batch`), as C# does: a snapshot document for a key the node holds read-only, or any other refusal, applies nothing. It used to apply document by document, so a refusal ended the pull with the earlier documents kept. A snapshot whose types route to different backing stores is now refused whole, as a batch cannot span stores. C# `CultCache.Commit` refuses a cross-store commit the same way (`The batch spans A and B; a commit lands in one home store`), so the two agree; a test pins it.
- `SchemaConflictError` gains the public field `unread_slots: Option<UnreadSlots>`; code that builds it with a struct literal must set it (`None`). - `SchemaConflictError` gains the public field `unread_slots: Option<UnreadSlots>`; code that builds it with a struct literal must set it (`None`).

View file

@ -1,730 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "android_system_properties"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc"
dependencies = [
"libc",
]
[[package]]
name = "anyhow"
version = "1.0.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]]
name = "ar_archive_writer"
version = "0.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "523e5e187fff36fcb8d1d19ccf0ca4ca43ec3d158316057f66dc6f72716bc5ec"
dependencies = [
"object",
]
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "bitflags"
version = "2.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
[[package]]
name = "bumpalo"
version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "cc"
version = "1.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ff2a8126fff0f8c839080d619675a585e61720bb12d2baceae45769fd66f155"
dependencies = [
"find-msvc-tools",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "chrono"
version = "0.4.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327"
dependencies = [
"iana-time-zone",
"js-sys",
"num-traits",
"serde",
"wasm-bindgen",
"windows-link",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "cultcache-rs"
version = "0.3.1"
dependencies = [
"anyhow",
"chrono",
"cultcache-rs-derive",
"fs2",
"pretty_assertions",
"redb",
"rmp-serde",
"serde",
"serde_bytes",
"serde_json",
"serde_stacker",
"stacker",
"tempfile",
"uuid",
"windows-sys 0.59.0",
]
[[package]]
name = "cultcache-rs-derive"
version = "0.2.0"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "diff"
version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8"
[[package]]
name = "errno"
version = "0.3.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776"
dependencies = [
"libc",
"windows-sys 0.61.2",
]
[[package]]
name = "fastrand"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
[[package]]
name = "find-msvc-tools"
version = "0.1.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48"
[[package]]
name = "fs2"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213"
dependencies = [
"libc",
"winapi",
]
[[package]]
name = "futures-core"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
[[package]]
name = "futures-task"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd"
[[package]]
name = "futures-util"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc"
dependencies = [
"futures-core",
"futures-task",
"pin-project-lite",
"slab",
]
[[package]]
name = "getrandom"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
"libc",
"r-efi",
]
[[package]]
name = "iana-time-zone"
version = "0.1.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470"
dependencies = [
"android_system_properties",
"core-foundation-sys",
"iana-time-zone-haiku",
"js-sys",
"log",
"wasm-bindgen",
"windows-core",
]
[[package]]
name = "iana-time-zone-haiku"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
dependencies = [
"cc",
]
[[package]]
name = "itoa"
version = "1.0.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "js-sys"
version = "0.3.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a"
dependencies = [
"cfg-if",
"futures-util",
"wasm-bindgen",
]
[[package]]
name = "libc"
version = "0.2.191"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a"
[[package]]
name = "linux-raw-sys"
version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "log"
version = "0.4.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d15f86d12da44cd8d4ec3c73984537eb7f4cadcb9776a575a0a8a05ca11005c"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "object"
version = "0.39.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b"
dependencies = [
"memchr",
]
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "pin-project-lite"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
[[package]]
name = "pretty_assertions"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d"
dependencies = [
"diff",
"yansi",
]
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "psm"
version = "0.1.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622"
dependencies = [
"ar_archive_writer",
"cc",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "redb"
version = "4.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b317ef681e18f6091efbfab532eaf924a135c492d9d0299d0fbfae07e92a39b"
dependencies = [
"libc",
]
[[package]]
name = "rmp"
version = "0.8.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c"
dependencies = [
"num-traits",
]
[[package]]
name = "rmp-serde"
version = "1.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155"
dependencies = [
"rmp",
"serde",
]
[[package]]
name = "rustix"
version = "1.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa"
dependencies = [
"bitflags",
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
]
[[package]]
name = "rustversion"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_bytes"
version = "0.11.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8"
dependencies = [
"serde",
"serde_core",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.10",
]
[[package]]
name = "serde_json"
version = "1.0.154"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "serde_stacker"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4936375d50c4be7eff22293a9344f8e46f323ed2b3c243e52f89138d9bb0f4a"
dependencies = [
"serde",
"serde_core",
"stacker",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "slab"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "stacker"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967"
dependencies = [
"cc",
"cfg-if",
"libc",
"psm",
"windows-sys 0.61.2",
]
[[package]]
name = "syn"
version = "2.0.119"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ae9ccbea3a7b0b09443036490c8f34a8b1b760fd345e5c0dae0aac718faea79"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "tempfile"
version = "3.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom",
"once_cell",
"rustix",
"windows-sys 0.61.2",
]
[[package]]
name = "unicode-ident"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563"
[[package]]
name = "uuid"
version = "1.29.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bed99139f7dac9b5cfe5f17be4d66dc7079a34dee1d74bcc77217119240e267c"
dependencies = [
"getrandom",
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "wasm-bindgen"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c"
dependencies = [
"cfg-if",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
]
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a"
dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn 3.0.10",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e"
dependencies = [
"unicode-ident",
]
[[package]]
name = "winapi"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
dependencies = [
"winapi-i686-pc-windows-gnu",
"winapi-x86_64-pc-windows-gnu",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windows-core"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [
"windows-implement",
"windows-interface",
"windows-link",
"windows-result",
"windows-strings",
]
[[package]]
name = "windows-implement"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "windows-interface"
version = "0.59.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-strings"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "yansi"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"

View file

@ -31,4 +31,4 @@ pretty_assertions = "1.4"
tempfile = "3" tempfile = "3"
[target.'cfg(windows)'.dependencies] [target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_Security", "Win32_System_WindowsProgramming"] } windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }

View file

@ -384,10 +384,9 @@ an empty store. Any other failure to reach it, such as a symlink loop, a file
where a parent directory should be, or a missing permission, is an I/O error. where a parent directory should be, or a missing permission, is an I/O error.
A store file that exists and holds zero bytes is `Undecodable`: no CultCache A store file that exists and holds zero bytes is `Undecodable`: no CultCache
writer leaves one. A symbolic link whose target is gone is not a missing store: writer leaves one. A symbolic link whose target is gone is not a missing store:
reading it is an I/O error. A write at a store path that is a link, live or dangling, is reading it is an I/O error, so a read-then-write (`push`, `push_all`, `delete`, the exchanges)
refused with `StoreLinkedError` (`error.downcast_ref::<StoreLinkedError>()`) before the fails before writing. Writes do not yet follow links: a write through a live link
lock and the replace, so no write swaps a link for a regular file; configure the link's replaces the link with a regular file and leaves its target alone. A refusal names no
target as the store path. Reads through a live link work. A refusal names no
value read from the file; it names a record only by its key and schema id, and value read from the file; it names a record only by its key and schema id, and
a header only when it has the shape `cultcache.store.v<digits>`. a header only when it has the shape `cultcache.store.v<digits>`.

View file

@ -1,8 +1,6 @@
use anyhow::Result; use anyhow::Result;
use anyhow::anyhow; use anyhow::anyhow;
use cultcache_rs::CultCache; use cultcache_rs::CultCache;
use cultcache_rs::CultSlots;
use cultcache_rs::CultUnion;
use cultcache_rs::DatabaseEntry; use cultcache_rs::DatabaseEntry;
use cultcache_rs::SingleFileMessagePackBackingStore; use cultcache_rs::SingleFileMessagePackBackingStore;
use serde_json::json; use serde_json::json;
@ -41,100 +39,6 @@ struct CultCacheInteropDeck {
marks: Vec<CultCacheInteropMark>, marks: Vec<CultCacheInteropMark>,
} }
// A union field and nested bytes: the shapes a Bifrost-style store holds beyond plain strings.
#[derive(Clone, Debug, PartialEq, Eq, CultUnion)]
enum InteropArm {
#[cultcache(key = 0)]
Text(InteropTextArm),
#[cultcache(key = 1)]
Pack(InteropPackArm),
}
#[derive(Clone, Debug, PartialEq, Eq, CultSlots)]
struct InteropTextArm {
#[cultcache(key = 0)]
text: String,
}
#[derive(Clone, Debug, PartialEq, Eq, CultSlots)]
struct InteropPackArm {
#[cultcache(key = 0)]
label: String,
#[cultcache(key = 1)]
chunks: Vec<serde_bytes::ByteBuf>,
}
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(type = "cultcache.interop-shape", schema = "CultCacheInteropShape")]
struct CultCacheInteropShape {
#[cultcache(key = 0)]
document_id: String,
#[cultcache(key = 1)]
arm: InteropArm,
#[cultcache(key = 2)]
raw: serde_bytes::ByteBuf,
}
fn shapes() -> Vec<CultCacheInteropShape> {
vec![
CultCacheInteropShape {
document_id: "shape:text".into(),
arm: InteropArm::Text(InteropTextArm { text: "union arm 0".into() }),
raw: (0..=255u8).collect::<Vec<_>>().into(),
},
CultCacheInteropShape {
document_id: "shape:pack".into(),
arm: InteropArm::Pack(InteropPackArm {
label: "union arm 1".into(),
chunks: vec![vec![0xc0, 0xc1, 0x00].into(), vec![].into(), vec![0x92, 0x01, 0x02].into()],
}),
raw: vec![0xde, 0xad, 0xbe, 0xef].into(),
},
]
}
fn write_shapes(file: &str) -> Result<()> {
let mut cache = CultCache::new();
cache.register_entry_type::<CultCacheInteropShape>()?;
cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(file))?;
cache.pull_all_backing_stores()?;
for shape in shapes() {
cache.put(&shape.document_id, &shape)?;
}
print_shapes(shapes())
}
/// Opens the store through the read-only snapshot door: no lock file, no staging file, no rewrite.
fn read_shapes(file: &str) -> Result<()> {
let envelopes = SingleFileMessagePackBackingStore::new(file).pull_all_read_only_snapshot()?;
let mut found = Vec::new();
for envelope in envelopes.iter().filter(|envelope| envelope.r#type == CultCacheInteropShape::TYPE) {
found.push(cultcache_rs::decode_payload::<CultCacheInteropShape>(&envelope.payload)?);
}
print_shapes(found)
}
fn print_shapes(mut found: Vec<CultCacheInteropShape>) -> Result<()> {
let hex = |bytes: &[u8]| bytes.iter().map(|byte| format!("{byte:02X}")).collect::<String>();
found.sort_by(|a, b| a.document_id.cmp(&b.document_id));
let rows: Vec<_> = found
.iter()
.map(|shape| {
let arm = match &shape.arm {
InteropArm::Text(text) => json!({ "kind": "text", "text": text.text }),
InteropArm::Pack(pack) => json!({
"kind": "pack",
"label": pack.label,
"chunks": pack.chunks.iter().map(|chunk| hex(chunk)).collect::<Vec<_>>(),
}),
};
json!({ "documentId": shape.document_id, "arm": arm, "raw": hex(&shape.raw) })
})
.collect();
println!("{}", serde_json::to_string(&rows)?);
Ok(())
}
fn read_deck(file: &str) -> Result<()> { fn read_deck(file: &str) -> Result<()> {
let mut cache = CultCache::new(); let mut cache = CultCache::new();
cache.register_entry_type::<CultCacheInteropDeck>()?; cache.register_entry_type::<CultCacheInteropDeck>()?;
@ -166,8 +70,6 @@ fn main() -> Result<()> {
"write" => write_note(file, require_arg(&options, "runtime-id")?)?, "write" => write_note(file, require_arg(&options, "runtime-id")?)?,
"read" => read_note(file)?, "read" => read_note(file)?,
"read-deck" => read_deck(file)?, "read-deck" => read_deck(file)?,
"write-shapes" => write_shapes(file)?,
"read-shapes" => read_shapes(file)?,
"hold-lock" => hold_lock(file, require_arg(&options, "ms")?.parse()?)?, "hold-lock" => hold_lock(file, require_arg(&options, "ms")?.parse()?)?,
_ => return Err(anyhow!("unknown mode {mode}")), _ => return Err(anyhow!("unknown mode {mode}")),
} }

View file

@ -1176,35 +1176,6 @@ impl std::fmt::Display for CultCacheStoreUnreadable {
impl std::error::Error for CultCacheStoreUnreadable {} impl std::error::Error for CultCacheStoreUnreadable {}
/// A write was asked of a store path that is a link (a file or directory symbolic link, or a Windows
/// junction): a replace there would swap the link for a regular file and detach the store from its
/// target. Nothing was locked or written. Configure the link's target as the store path. Reads through
/// a live link are unaffected. Find it with `error.downcast_ref::<StoreLinkedError>()`.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct StoreLinkedError {
pub path: PathBuf,
}
impl std::fmt::Display for StoreLinkedError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
formatter,
"CultCache store {} is a link, and a write at it would replace the link; configure the link's target as the store path",
self.path.display()
)
}
}
impl std::error::Error for StoreLinkedError {}
/// Refuses a write at a link. A path that cannot be inspected is left to the open that follows.
fn refuse_linked(path: &Path) -> Result<()> {
match fs::symlink_metadata(path) {
Ok(metadata) if metadata.file_type().is_symlink() => Err(StoreLinkedError { path: path.to_path_buf() }.into()),
_ => Ok(()),
}
}
/// A write to a single-file store that failed, and what it left on disk. It sits in the /// A write to a single-file store that failed, and what it left on disk. It sits in the
/// error chain of every such failure, above the cause, so a consumer tells the outcomes /// error chain of every such failure, above the cause, so a consumer tells the outcomes
/// apart with `error.downcast_ref::<CultCacheStoreWriteFailed>()`. /// apart with `error.downcast_ref::<CultCacheStoreWriteFailed>()`.
@ -1748,11 +1719,10 @@ impl SingleFileMessagePackBackingStore {
} }
fn with_exclusive_lock<T>(&self, action: impl FnOnce() -> Result<T>) -> Result<T> { fn with_exclusive_lock<T>(&self, action: impl FnOnce() -> Result<T>) -> Result<T> {
refuse_linked(&self.path)?;
let lock = self.open_lock_file()?; let lock = self.open_lock_file()?;
fs2::FileExt::lock_exclusive(&lock) fs2::FileExt::lock_exclusive(&lock)
.with_context(|| format!("failed to lock {}", self.lock_path().display()))?; .with_context(|| format!("failed to lock {}", self.lock_path().display()))?;
self.run_locked(lock, action) release_lock(lock, action())
} }
/// Replaces the named entries only if each still matches its expectation. /// Replaces the named entries only if each still matches its expectation.
@ -1860,12 +1830,7 @@ impl SingleFileMessagePackBackingStore {
} }
fn try_with_exclusive_lock<T>(&self, action: impl FnOnce() -> Result<T>) -> Result<Option<T>> { fn try_with_exclusive_lock<T>(&self, action: impl FnOnce() -> Result<T>) -> Result<Option<T>> {
refuse_linked(&self.path)?; let lock = self.open_lock_file()?;
#[cfg(test)]
run_after_pre_lock_look();
let Some(lock) = self.try_open_lock_file()? else {
return Ok(None);
};
match fs2::FileExt::try_lock_exclusive(&lock) { match fs2::FileExt::try_lock_exclusive(&lock) {
Ok(()) => {} Ok(()) => {}
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => return Ok(None), Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => return Ok(None),
@ -1874,40 +1839,22 @@ impl SingleFileMessagePackBackingStore {
.with_context(|| format!("failed to lock {}", self.lock_path().display())); .with_context(|| format!("failed to lock {}", self.lock_path().display()));
} }
} }
self.run_locked(lock, action).map(Some) release_lock(lock, action()).map(Some)
} }
/// The one tail of both exclusive entries: with the lock held, look for a link again (one may have been put at the
/// path while the lock was awaited; the look before the lock saw the path earlier), run the action, release.
fn run_locked<T>(&self, lock: File, action: impl FnOnce() -> Result<T>) -> Result<T> {
release_lock(lock, refuse_linked(&self.path).and_then(|()| action()))
}
/// Opens the lock file, waiting through a Windows sharing violation the way the lock itself waits.
fn open_lock_file(&self) -> Result<File> { fn open_lock_file(&self) -> Result<File> {
let lock_path = self.prepare_lock_dir()?;
retry_while_busy(
|| open_lock_once(&lock_path),
is_sharing_violation,
|| std::thread::sleep(std::time::Duration::from_millis(5)),
)
.with_context(|| format!("failed to open {}", lock_path.display()))
}
/// The single-attempt open for the try entry: a sharing violation is `None`, the same answer as a held lock.
fn try_open_lock_file(&self) -> Result<Option<File>> {
let lock_path = self.prepare_lock_dir()?;
open_unless_busy(|| open_lock_once(&lock_path), is_sharing_violation)
.with_context(|| format!("failed to open {}", lock_path.display()))
}
fn prepare_lock_dir(&self) -> Result<PathBuf> {
let lock_path = self.lock_path(); let lock_path = self.lock_path();
if let Some(parent) = lock_path.parent() { if let Some(parent) = lock_path.parent() {
fs::create_dir_all(parent) fs::create_dir_all(parent)
.with_context(|| format!("failed to create {}", parent.display()))?; .with_context(|| format!("failed to create {}", parent.display()))?;
} }
Ok(lock_path) OpenOptions::new()
.create(true)
.read(true)
.truncate(false)
.write(true)
.open(&lock_path)
.with_context(|| format!("failed to open {}", lock_path.display()))
} }
fn lock_path(&self) -> PathBuf { fn lock_path(&self) -> PathBuf {
@ -3331,8 +3278,7 @@ impl CultCache {
/// Whether a write or removal of `(type_id, key)` would be admitted: false when the held payload carries slots past /// Whether a write or removal of `(type_id, key)` would be admitted: false when the held payload carries slots past
/// its registered type's declared slots, at any depth, or cannot be read by this build at all (the record is /// its registered type's declared slots, at any depth, or cannot be read by this build at all (the record is
/// read-only), when the cache holds no image of its stores (see [`CultCacheLoadRefused`]), when a store carries a foreign record at the key, or when another registered type holds it. True for a /// read-only), when the cache holds no image of its stores (see [`CultCacheLoadRefused`]), when a store carries a foreign record at the key, or when another registered type holds it. True for a
/// key nothing holds. An unregistered type is an error. It does not look at store paths: a link at a store path is not /// key nothing holds. An unregistered type is an error. Advisory: the write entries ask the same checks themselves.
/// predicted, and the write refuses it with [`StoreLinkedError`]. Advisory: the write entries ask the same checks themselves.
pub fn is_writable(&self, type_id: &str, key: &str) -> Result<bool> { pub fn is_writable(&self, type_id: &str, key: &str) -> Result<bool> {
if self.require_loaded().is_err() { if self.require_loaded().is_err() {
return Ok(false); return Ok(false);
@ -4174,7 +4120,7 @@ struct DiskStore {
} }
impl DiskStore { impl DiskStore {
/// Nothing at the path: an empty v1 store. A dangling link is something at the path and is refused, not read as this. /// A file that is gone, or a dangling link a whole-file replace is about to replace: an empty v1 store.
fn absent() -> Self { fn absent() -> Self {
Self { absent: true, format: STORE_FORMAT_V1, envelopes: Vec::new(), catalog: Vec::new() } Self { absent: true, format: STORE_FORMAT_V1, envelopes: Vec::new(), catalog: Vec::new() }
} }
@ -4419,324 +4365,61 @@ fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> {
#[cfg(windows)] #[cfg(windows)]
fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> { fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::{
MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW,
};
injected_write_fault(WriteStep::Rename).context("failed to atomically replace the store")?; injected_write_fault(WriteStep::Rename).context("failed to atomically replace the store")?;
replace_with(&WindowsKernel, staged, destination)
}
/// The two Windows replace primitives, each answering 0 or a Win32 error code, and the wait let staged_wide = staged
/// between attempts. `replace_with` owns every decision about them. .as_os_str()
#[cfg_attr(not(windows), allow(dead_code))] .encode_wide()
trait ReplaceKernel { .chain(std::iter::once(0))
/// CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with .collect::<Vec<_>>();
/// REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle. let destination_wide = destination
fn rename_posix(&self, staged: &Path, destination: &Path) -> u32; .as_os_str()
/// MoveFileExW(staged, destination, flags). .encode_wide()
fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32; .chain(std::iter::once(0))
fn sleep(&self, milliseconds: u64); .collect::<Vec<_>>();
} // A concurrent reader of the destination (another process opening the
// store, an indexer, an antivirus scan) makes the replace fail with a
/// MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH. // sharing or access error for as long as it holds the handle. That is
#[cfg_attr(not(windows), allow(dead_code))] // transient by construction; a daemon that treats it as fatal dies on
const MOVEFILE_REPLACE_WRITE_THROUGH: u32 = 0x9; // the first overlap (Muninn serve did, 2026-09-11). Retry briefly.
const ERROR_ACCESS_DENIED: i32 = 5;
/// Replaces `destination` with `staged` on Windows: a POSIX-semantics rename first, so a reader const ERROR_SHARING_VIOLATION: i32 = 32;
/// that opened the old store keeps it and the destination is rarely seen absent (a stalled rename const ERROR_LOCK_VIOLATION: i32 = 33;
/// can still leave it absent for 5-47 ms on NTFS: `variants:ruling:posix-rename-absent-window-document`);
/// MoveFileExW with
/// write-through where POSIX semantics are unsupported (FAT, exFAT, before Windows 10 1607),
/// for the rest of the call; and a bounded retry on access, sharing and lock violations.
/// A concurrent reader (another process, an indexer, an antivirus scan) makes the replace fail
/// transiently, and a daemon that treats that as fatal dies on the first overlap (Muninn serve
/// did, 2026-09-11). Ruling `variants:ruling:windows-replace-posix-first`: cultcache-ts carries the
/// same table on main; cultcache-cs carries it only once its pending PR lands.
#[cfg_attr(not(windows), allow(dead_code))]
fn replace_with(kernel: &dyn ReplaceKernel, staged: &Path, destination: &Path) -> Result<()> {
const ERROR_INVALID_FUNCTION: u32 = 1;
const ERROR_ACCESS_DENIED: u32 = 5;
const ERROR_SHARING_VIOLATION: u32 = 32;
const ERROR_LOCK_VIOLATION: u32 = 33;
const ERROR_NOT_SUPPORTED: u32 = 50;
const ERROR_INVALID_PARAMETER: u32 = 87;
const ATTEMPTS: u32 = 25; const ATTEMPTS: u32 = 25;
const BACKOFF_MS: u64 = 20; const BACKOFF: std::time::Duration = std::time::Duration::from_millis(20);
let mut posix = true; let mut attempt = 0;
let mut last_code = 0; loop {
for attempt in 1..=ATTEMPTS { let success = unsafe {
let mut code = if posix { MoveFileExW(
kernel.rename_posix(staged, destination) staged_wide.as_ptr(),
} else { destination_wide.as_ptr(),
kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH) MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
}; };
if posix && matches!(code, ERROR_INVALID_FUNCTION | ERROR_NOT_SUPPORTED | ERROR_INVALID_PARAMETER) { if success != 0 {
posix = false;
code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH);
}
if code == 0 {
return Ok(()); return Ok(());
} }
if !matches!(code, ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION) { let error = std::io::Error::last_os_error();
return Err(anyhow::anyhow!( attempt += 1;
"failed to atomically replace {} with {}: Win32 error {code} after {attempt} attempt(s)", let transient = matches!(
destination.display(), error.raw_os_error(),
staged.display() Some(ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION)
));
}
last_code = code;
kernel.sleep(BACKOFF_MS);
}
Err(anyhow::anyhow!(
"failed to atomically replace {} with {} after {ATTEMPTS} attempt(s): last Win32 error {last_code}",
destination.display(),
staged.display()
))
}
#[cfg(windows)]
struct WindowsKernel;
#[cfg(windows)]
fn wide(path: &Path) -> Vec<u16> {
use std::os::windows::ffi::OsStrExt;
path.as_os_str().encode_wide().chain(std::iter::once(0)).collect()
}
#[cfg(windows)]
impl ReplaceKernel for WindowsKernel {
fn rename_posix(&self, staged: &Path, destination: &Path) -> u32 {
use windows_sys::Win32::Foundation::{CloseHandle, GetLastError, INVALID_HANDLE_VALUE};
use windows_sys::Win32::Storage::FileSystem::{
CreateFileW, FileRenameInfoEx, SetFileInformationByHandle, DELETE, FILE_FLAG_BACKUP_SEMANTICS,
FILE_RENAME_INFO, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, OPEN_EXISTING, SYNCHRONIZE,
};
use windows_sys::Win32::System::WindowsProgramming::{
FILE_RENAME_FLAG_POSIX_SEMANTICS, FILE_RENAME_FLAG_REPLACE_IF_EXISTS,
};
let staged_wide = wide(staged);
let destination_wide = wide(destination);
// FILE_RENAME_INFO ends in a variable-length name: allocate it as u64s for alignment.
let name_bytes = (destination_wide.len() - 1) * 2;
let header = std::mem::size_of::<FILE_RENAME_INFO>();
let total = header + name_bytes + 2;
let mut buffer = vec![0u64; total.div_ceil(8)];
unsafe {
let info = buffer.as_mut_ptr() as *mut FILE_RENAME_INFO;
(*info).Anonymous.Flags = FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS;
(*info).RootDirectory = std::ptr::null_mut();
(*info).FileNameLength = name_bytes as u32;
let name = std::ptr::addr_of_mut!((*info).FileName) as *mut u16;
std::ptr::copy_nonoverlapping(destination_wide.as_ptr(), name, destination_wide.len());
let handle = CreateFileW(
staged_wide.as_ptr(),
DELETE | SYNCHRONIZE,
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
std::ptr::null(),
OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS,
std::ptr::null_mut(),
);
if handle == INVALID_HANDLE_VALUE {
return GetLastError();
}
let ok = SetFileInformationByHandle(handle, FileRenameInfoEx, info as *const _, total as u32);
let code = if ok == 0 { GetLastError() } else { 0 };
CloseHandle(handle);
code
}
}
fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32 {
use windows_sys::Win32::Foundation::GetLastError;
use windows_sys::Win32::Storage::FileSystem::MoveFileExW;
let ok = unsafe { MoveFileExW(wide(staged).as_ptr(), wide(destination).as_ptr(), flags) };
if ok != 0 {
0
} else {
unsafe { GetLastError() }
}
}
fn sleep(&self, milliseconds: u64) {
std::thread::sleep(std::time::Duration::from_millis(milliseconds));
}
}
fn open_lock_once(lock_path: &Path) -> std::io::Result<File> {
OpenOptions::new()
.create(true)
.read(true)
.truncate(false)
.write(true)
.open(lock_path)
}
/// Windows ERROR_SHARING_VIOLATION (raw os error 32): another handle holds the lock file open without sharing. On Unix
/// errno 32 is EPIPE and is never a busy file.
#[cfg(windows)]
fn is_sharing_violation(error: &std::io::Error) -> bool {
error.raw_os_error() == Some(32)
}
#[cfg(not(windows))]
fn is_sharing_violation(_error: &std::io::Error) -> bool {
false
}
/// Repeats `open` while it fails busy, sleeping between tries and without a bound, as the lock waits; any other error surfaces.
fn retry_while_busy<T>(
mut open: impl FnMut() -> std::io::Result<T>,
busy: impl Fn(&std::io::Error) -> bool,
mut sleep: impl FnMut(),
) -> std::io::Result<T> {
loop {
match open() {
Err(error) if busy(&error) => sleep(),
other => return other,
}
}
}
/// One attempt: `None` when the open fails busy, any other error surfaces.
fn open_unless_busy<T>(
open: impl FnOnce() -> std::io::Result<T>,
busy: impl Fn(&std::io::Error) -> bool,
) -> std::io::Result<Option<T>> {
match open() {
Ok(file) => Ok(Some(file)),
Err(error) if busy(&error) => Ok(None),
Err(error) => Err(error),
}
}
#[cfg(test)]
mod replace_with_tests {
use super::{replace_with, ReplaceKernel};
use std::cell::RefCell;
use std::collections::VecDeque;
use std::path::Path;
#[derive(Debug, PartialEq)]
enum Call {
Posix,
MoveFile(u32),
Sleep(u64),
}
/// Answers each primitive from a script; an empty script answers 0.
#[derive(Default)]
struct FakeKernel {
posix: RefCell<VecDeque<u32>>,
move_file: RefCell<VecDeque<u32>>,
calls: RefCell<Vec<Call>>,
}
impl FakeKernel {
fn scripted(posix: &[u32], move_file: &[u32]) -> Self {
Self {
posix: RefCell::new(posix.iter().copied().collect()),
move_file: RefCell::new(move_file.iter().copied().collect()),
..Self::default()
}
}
fn calls(&self) -> Vec<Call> {
self.calls.replace(Vec::new())
}
}
impl ReplaceKernel for FakeKernel {
fn rename_posix(&self, _: &Path, _: &Path) -> u32 {
self.calls.borrow_mut().push(Call::Posix);
self.posix.borrow_mut().pop_front().unwrap_or(0)
}
fn move_file_ex(&self, _: &Path, _: &Path, flags: u32) -> u32 {
self.calls.borrow_mut().push(Call::MoveFile(flags));
self.move_file.borrow_mut().pop_front().unwrap_or(0)
}
fn sleep(&self, milliseconds: u64) {
self.calls.borrow_mut().push(Call::Sleep(milliseconds));
}
}
fn run(kernel: &FakeKernel) -> anyhow::Result<()> {
replace_with(kernel, Path::new("staged"), Path::new("store.cc"))
}
#[test]
fn a_posix_rename_that_lands_is_the_whole_replace() {
let kernel = FakeKernel::scripted(&[0], &[]);
run(&kernel).unwrap();
assert_eq!(kernel.calls(), vec![Call::Posix]);
}
#[test]
fn unsupported_posix_semantics_fall_back_to_write_through_and_stay_there() {
for code in [1, 50, 87] {
// The fallback itself is held once, so the retry must use MoveFileExW, not POSIX again.
let kernel = FakeKernel::scripted(&[code], &[32, 0]);
run(&kernel).unwrap_or_else(|e| panic!("code {code}: {e}"));
assert_eq!(
kernel.calls(),
vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)],
"code {code}"
);
}
}
#[test]
fn a_held_destination_is_retried_with_posix_until_it_lets_go() {
for code in [5, 32, 33] {
let kernel = FakeKernel::scripted(&[code, code, 0], &[]);
run(&kernel).unwrap();
assert_eq!(
kernel.calls(),
vec![Call::Posix, Call::Sleep(20), Call::Posix, Call::Sleep(20), Call::Posix],
"code {code}"
);
}
}
#[test]
fn a_destination_held_forever_fails_after_twenty_five_attempts_naming_the_code() {
for code in [5, 32, 33] {
let kernel = FakeKernel::scripted(&[code; 40], &[]);
let error = run(&kernel).unwrap_err().to_string();
let calls = kernel.calls();
assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 25);
assert_eq!(calls.iter().filter(|c| **c == Call::Sleep(20)).count(), 25);
assert!(error.contains("25 attempt(s)") && error.contains(&format!("error {code}")), "{error}");
}
}
#[test]
fn a_held_destination_after_fallback_exhausts_through_move_file_ex() {
let kernel = FakeKernel::scripted(&[87], &[32; 40]);
assert!(run(&kernel).is_err());
let calls = kernel.calls();
assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 1);
assert_eq!(calls.iter().filter(|c| matches!(c, Call::MoveFile(0x9))).count(), 25);
}
#[test]
fn any_other_code_fails_at_once_without_a_retry() {
for (posix, move_file) in [(&[2u32][..], &[][..]), (&[87][..], &[3][..])] {
let kernel = FakeKernel::scripted(posix, move_file);
let error = run(&kernel).unwrap_err().to_string();
assert!(!kernel.calls().contains(&Call::Sleep(20)), "{error}");
assert!(error.contains("1 attempt(s)"), "{error}");
}
}
#[test]
fn move_file_ex_answering_unsupported_is_an_error_not_a_cue_to_fall_back_again() {
let kernel = FakeKernel::scripted(&[87], &[32, 87]);
let error = run(&kernel).unwrap_err().to_string();
assert_eq!(
kernel.calls(),
vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)],
"{error}"
); );
assert!(error.contains("error 87") && error.contains("2 attempt(s)"), "{error}"); if !transient || attempt >= ATTEMPTS {
return Err(error).with_context(|| {
format!(
"failed to atomically replace {} with {} after {attempt} attempt(s)",
destination.display(),
staged.display()
)
});
}
std::thread::sleep(BACKOFF);
} }
} }
@ -4745,35 +4428,6 @@ mod windows_replace_tests {
use super::replace_file_atomically; use super::replace_file_atomically;
use std::os::windows::fs::OpenOptionsExt; use std::os::windows::fs::OpenOptionsExt;
const FILE_SHARE_ALL: u32 = 0x1 | 0x2 | 0x4; // READ | WRITE | DELETE
/// A reader that shares delete holds the old store open. A POSIX-semantics rename
/// replaces the name while the reader keeps the old content; MoveFileExW alone fails
/// there with error 5 (Soul's Starfire probe, build 26200), so this test passes only
/// if the POSIX path is live. Compile-checked only: it has never run on Windows.
#[test]
fn a_replace_succeeds_over_a_delete_sharing_reader_which_keeps_the_old_content() {
use std::io::Read;
let dir = std::env::temp_dir().join(format!("cultcache-posix-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let destination = dir.join("store.cc");
let staged = dir.join("store.cc.tmp");
std::fs::write(&destination, b"old").unwrap();
std::fs::write(&staged, b"new").unwrap();
let mut reader = std::fs::OpenOptions::new()
.read(true)
.share_mode(FILE_SHARE_ALL)
.open(&destination)
.unwrap();
replace_file_atomically(&staged, &destination).expect("POSIX rename over a sharing reader");
let mut seen = Vec::new();
reader.read_to_end(&mut seen).unwrap();
assert_eq!(seen, b"old", "the reader keeps the old content");
assert_eq!(std::fs::read(&destination).unwrap(), b"new");
drop(reader);
let _ = std::fs::remove_dir_all(&dir);
}
/// A reader holding the destination open with no sharing makes the /// A reader holding the destination open with no sharing makes the
/// replace fail for as long as it holds on; it must succeed once the /// replace fail for as long as it holds on; it must succeed once the
/// reader lets go, not report the first refusal as the outcome. /// reader lets go, not report the first refusal as the outcome.
@ -4836,20 +4490,6 @@ fn injected_write_fault(step: WriteStep) -> std::io::Result<()> {
Ok(()) Ok(())
} }
// The one gap a test cannot hold open from outside: a try-lock never waits, so what happens between its pre-lock look and
// the lock is only reachable by running the test's closure there.
#[cfg(test)]
thread_local! {
static AFTER_PRE_LOCK_LOOK: std::cell::RefCell<Option<Box<dyn FnOnce()>>> = const { std::cell::RefCell::new(None) };
}
#[cfg(test)]
fn run_after_pre_lock_look() {
if let Some(hook) = AFTER_PRE_LOCK_LOOK.with(|hook| hook.borrow_mut().take()) {
hook();
}
}
#[cfg(not(test))] #[cfg(not(test))]
#[inline(always)] #[inline(always)]
fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> { fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> {
@ -4858,8 +4498,7 @@ fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> {
#[cfg(windows)] #[cfg(windows)]
fn sync_parent_directory(_path: &Path) -> Result<()> { fn sync_parent_directory(_path: &Path) -> Result<()> {
// Publication durability is the replace's: the MoveFileExW fallback writes through; the POSIX-semantics // MoveFileExW with MOVEFILE_WRITE_THROUGH owns publication durability.
// rename has no write-through flag and its durability is unmeasured (follow-up windows-replace-run-and-durability).
Ok(()) Ok(())
} }
@ -5735,6 +5374,12 @@ mod tests {
Ok(()) Ok(())
} }
fn io_not_found(error: &anyhow::Error) -> bool {
error
.chain()
.any(|cause| cause.downcast_ref::<std::io::Error>().is_some_and(|io| io.kind() == std::io::ErrorKind::NotFound))
}
fn test_envelope(r#type: &str, key: &str, payload: &[u8]) -> CultCacheEnvelope { fn test_envelope(r#type: &str, key: &str, payload: &[u8]) -> CultCacheEnvelope {
CultCacheEnvelope { CultCacheEnvelope {
key: key.to_string(), key: key.to_string(),
@ -7669,14 +7314,7 @@ mod tests {
let temp = tempfile::tempdir()?; let temp = tempfile::tempdir()?;
let path = temp.path().join("store.cc"); let path = temp.path().join("store.cc");
std::os::unix::fs::symlink(&path, &path)?; std::os::unix::fs::symlink(&path, &path)?;
let [pull, snapshot, exchange] = every_read(&SingleFileMessagePackBackingStore::new(&path)); assert_io_failure_on_every_read(&SingleFileMessagePackBackingStore::new(&path));
for error in [pull.unwrap_err(), snapshot.unwrap_err()] {
let io = io_error_in(&error).expect("the failure to reach the store is an io::Error");
assert_ne!(io.kind(), std::io::ErrorKind::NotFound, "{error:#}");
assert_eq!(unreadable_kind(&error), None, "{error:#}");
}
// The exchange is a write: a link at the path is refused before it reads.
assert!(exchange.unwrap_err().downcast_ref::<StoreLinkedError>().is_some());
assert!(fs::symlink_metadata(&path)?.file_type().is_symlink()); assert!(fs::symlink_metadata(&path)?.file_type().is_symlink());
Ok(()) Ok(())
} }
@ -8115,13 +7753,12 @@ mod tests {
let path = temp.path().join("store.cc"); let path = temp.path().join("store.cc");
std::os::unix::fs::symlink(&target, &path)?; std::os::unix::fs::symlink(&target, &path)?;
let store = SingleFileMessagePackBackingStore::new(&path); let store = SingleFileMessagePackBackingStore::new(&path);
let [pull, snapshot, exchange] = every_read(&store); for result in every_read(&store) {
for error in [pull.unwrap_err(), snapshot.unwrap_err()] { let error = result.expect_err("a dangling link must not read as an empty store");
assert!(io_error_in(&error).is_some(), "{error:#}"); assert!(io_error_in(&error).is_some(), "{error:#}");
assert_eq!(unreadable_kind(&error), None, "{error:#}"); assert_eq!(unreadable_kind(&error), None, "{error:#}");
} }
assert!(exchange.unwrap_err().downcast_ref::<StoreLinkedError>().is_some()); // Every merging writer refuses on its own read, so none can replace the link.
// Every writer refuses the link before its lock, so none can replace it.
let mut writer = SingleFileMessagePackBackingStore::new(&path); let mut writer = SingleFileMessagePackBackingStore::new(&path);
let delta = snapshot_envelope("delta", b"4"); let delta = snapshot_envelope("delta", b"4");
let writes = [ let writes = [
@ -8133,7 +7770,7 @@ mod tests {
]; ];
for (index, result) in writes.into_iter().enumerate() { for (index, result) in writes.into_iter().enumerate() {
let error = result.expect_err("a merging write through a dangling link must refuse"); let error = result.expect_err("a merging write through a dangling link must refuse");
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "write {index}: {error:#}"); assert!(io_error_in(&error).is_some(), "write {index}: {error:#}");
assert_eq!(write_failure_kind(&error), None, "write {index}: {error:#}"); assert_eq!(write_failure_kind(&error), None, "write {index}: {error:#}");
} }
assert!(fs::symlink_metadata(&path)?.file_type().is_symlink()); assert!(fs::symlink_metadata(&path)?.file_type().is_symlink());
@ -8259,21 +7896,11 @@ mod tests {
with_write_fault(WriteStep::Unlock, || store.push(&pushed))?; with_write_fault(WriteStep::Unlock, || store.push(&pushed))?;
assert!(with_write_fault(WriteStep::Unlock, || store.pull_all())?.contains(&pushed)); assert!(with_write_fault(WriteStep::Unlock, || store.pull_all())?.contains(&pushed));
// The lock was released: another writer takes it. The shared lock of the faulted `pull_all` just above was left to the // The lock was released: another writer takes it at once.
// close of its descriptor (see `release_lock`), which frees it only once no other descriptor shares its open file assert!(matches!(
// description, and a test that spawns a child (a fork-exec duplicates every descriptor of the process until the exec) store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?,
// can hold such a duplicate on another test thread for a moment. A faulted exclusive lock cannot contend the try-lock: TryCompareExchangeSnapshotOutcome::Exchanged
// the blocking `pull_all` in its argument waits it out. So a contended try-lock is retried for up to two seconds; a ));
// lock that was never released stays contended and still fails.
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2);
let outcome = loop {
let outcome = store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?;
if !matches!(outcome, TryCompareExchangeSnapshotOutcome::LockContended) || std::time::Instant::now() >= deadline {
break outcome;
}
std::thread::sleep(std::time::Duration::from_millis(5));
};
assert!(matches!(outcome, TryCompareExchangeSnapshotOutcome::Exchanged), "{outcome:?}");
// The keyed store releases its lock the same way. // The keyed store releases its lock the same way.
let mut keyed = RedbMessagePackBackingStore::new(temp.path().join("keyed.redb"))?; let mut keyed = RedbMessagePackBackingStore::new(temp.path().join("keyed.redb"))?;
@ -8282,265 +7909,27 @@ mod tests {
Ok(()) Ok(())
} }
// A link put at the store path while a writer waits on the lock is refused once the lock is taken, as C# does, and the // What a write does through a symbolic link today. R3 decides whether a write
// lock is free afterwards. // resolves the link or refuses it; until then these pin the current behaviour so the
// change is deliberate. Reads refuse a dangling link (see the test above).
#[cfg(unix)] #[cfg(unix)]
#[test] #[test]
fn a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it() -> Result<()> { fn r3_decides_resolve_or_refuse_a_push_through_a_live_link_replaces_the_link() -> Result<()> {
let temp = tempfile::tempdir()?; let temp = tempfile::tempdir()?;
let volume = temp.path().join("volume"); let volume = temp.path().join("volume");
fs::create_dir(&volume)?; fs::create_dir(&volume)?;
let target = volume.join("store.cc"); let target = volume.join("store.cc");
SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?; SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?;
let before = fs::read(&target)?; let before = fs::read(&target)?;
let dangling = volume.join("missing.cc"); let link = temp.path().join("store.cc");
let path = temp.path().join("late.cc");
let writers: [(&str, fn(&mut SingleFileMessagePackBackingStore) -> Result<()>); 4] = [
("push", |s| s.push(&snapshot_envelope("b", b"2"))),
("push_all", |s| s.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default())),
("delete", |s| s.delete(&snapshot_envelope("a", b"1"))),
("compare_exchange_snapshot", |s| s.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).map(|_| ())),
];
for (name, write, link_to) in writers.into_iter().flat_map(|(n, w)| [(n, w, &target), (n, w, &dangling)]) {
let _ = fs::remove_file(&path);
let store = SingleFileMessagePackBackingStore::new(&path);
let holder = store.open_lock_file()?;
fs2::FileExt::lock_exclusive(&holder)?;
let mut waiting = SingleFileMessagePackBackingStore::new(&path);
let handle = std::thread::spawn(move || write(&mut waiting));
std::thread::sleep(std::time::Duration::from_millis(300));
std::os::unix::fs::symlink(link_to, &path)?;
fs2::FileExt::unlock(&holder)?;
let error = handle.join().expect("writer thread").expect_err("the late link must be refused");
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "{name}: {error:#}");
assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "{name}: the link was replaced");
assert_eq!(fs::read(&target)?, before, "{name}: the target changed");
assert!(!dangling.exists(), "{name}: the dangling target was created");
let probe = store.open_lock_file()?;
fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("{name}: lock still held: {e}"))?;
}
Ok(())
}
fn busy_error() -> std::io::Error {
std::io::Error::from_raw_os_error(32)
}
fn is_busy(error: &std::io::Error) -> bool {
error.raw_os_error() == Some(32)
}
#[test]
fn the_lock_open_waits_through_busy_failures_then_opens() {
let (mut tries, mut sleeps) = (0, 0);
let opened = retry_while_busy(
|| {
tries += 1;
if tries <= 4 { Err(busy_error()) } else { Ok(tries) }
},
is_busy,
|| sleeps += 1,
);
assert_eq!(opened.unwrap(), 5);
assert_eq!(sleeps, 4);
}
#[test]
fn the_lock_open_surfaces_a_non_busy_error_at_once() {
let (mut tries, mut sleeps) = (0, 0);
let error = retry_while_busy::<()>(
|| {
tries += 1;
Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied))
},
is_busy,
|| sleeps += 1,
)
.expect_err("a non-busy error must surface");
assert_eq!(error.kind(), std::io::ErrorKind::PermissionDenied);
assert_eq!((tries, sleeps), (1, 0));
}
#[test]
fn the_try_open_maps_a_sharing_violation_to_none_without_retrying() {
let mut tries = 0;
let opened = open_unless_busy::<()>(
|| {
tries += 1;
Err(busy_error())
},
is_busy,
);
assert!(opened.unwrap().is_none());
assert_eq!(tries, 1);
let other = open_unless_busy::<()>(|| Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied)), is_busy);
assert!(other.is_err());
}
// The try entry must answer at once while another handle holds the lock. The attempt runs on a thread and its
// answer is awaited with a timeout, so an entry that waits fails this test by timeout instead of hanging the suite.
#[cfg(unix)]
#[test]
fn the_try_entry_returns_at_once_while_the_lock_is_held() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = temp.path().join("held.cc");
let store = SingleFileMessagePackBackingStore::new(&path);
let holder = store.open_lock_file()?;
fs2::FileExt::lock_exclusive(&holder)?;
let (sender, receiver) = std::sync::mpsc::channel();
let attempt = SingleFileMessagePackBackingStore::new(&path);
std::thread::spawn(move || {
let _ = sender.send(attempt.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]));
});
let outcome = receiver
.recv_timeout(std::time::Duration::from_secs(3))
.map_err(|_| anyhow!("the try entry waited on a held lock instead of returning"))?;
assert!(matches!(outcome?, TryCompareExchangeSnapshotOutcome::LockContended));
assert!(!path.exists(), "the contended attempt wrote");
let again = store.open_lock_file()?;
assert!(fs2::FileExt::try_lock_exclusive(&again).is_err(), "the held lock was disturbed");
fs2::FileExt::unlock(&holder)?;
Ok(())
}
// Off Windows errno 32 is EPIPE, never a busy file: the predicate is false and the wait surfaces it on the first try.
#[cfg(not(windows))]
#[test]
fn error_32_is_not_busy_off_windows_and_surfaces_at_once() {
assert!(!is_sharing_violation(&busy_error()));
let (mut tries, mut sleeps) = (0, 0);
let error = retry_while_busy::<()>(
|| {
tries += 1;
Err(busy_error())
},
is_sharing_violation,
|| sleeps += 1,
)
.expect_err("error 32 must surface off Windows");
assert_eq!(error.raw_os_error(), Some(32));
assert_eq!((tries, sleeps), (1, 0));
}
// The try entry has no wait, so the late window is the gap between its look and its lock: a link put there is refused
// once the lock is held, nothing is written through or over it, and the lock is released.
#[cfg(unix)]
#[test]
fn a_link_put_between_the_try_entrys_look_and_its_lock_is_refused_and_the_lock_is_released() -> Result<()> {
let temp = tempfile::tempdir()?;
let target = temp.path().join("target.cc");
SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?;
let before = fs::read(&target)?;
let dangling = temp.path().join("missing.cc");
let path = temp.path().join("late.cc");
for link_to in [target.clone(), dangling.clone()] {
let store = SingleFileMessagePackBackingStore::new(&path);
let (link, at) = (link_to.clone(), path.clone());
AFTER_PRE_LOCK_LOOK.with(|hook| {
*hook.borrow_mut() = Some(Box::new(move || std::os::unix::fs::symlink(&link, &at).expect("put the link")));
});
let error = store
.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")])
.expect_err("the late link must be refused");
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "{error:#}");
assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "the link was replaced");
assert_eq!(fs::read(&target)?, before, "the target changed");
assert!(!dangling.exists(), "the dangling target was created");
let probe = store.open_lock_file()?;
fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("lock still held: {e}"))?;
drop(probe);
fs::remove_file(&path)?;
}
Ok(())
}
// A write at a store path that is a link is refused: a replace there would swap the link for a regular file and
// detach the store from its target. Reads through a live link keep working.
#[cfg(unix)]
#[test]
fn a_write_at_a_live_link_is_refused_and_nothing_changes_while_a_read_through_it_works() -> Result<()> {
let temp = tempfile::tempdir()?;
let volume = temp.path().join("volume");
fs::create_dir(&volume)?;
let target = volume.join("store.cc");
SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?;
let before = fs::read(&target)?;
let volume_before = dir_names(&volume)?;
let link = temp.path().join("link.cc");
std::os::unix::fs::symlink(&target, &link)?; std::os::unix::fs::symlink(&target, &link)?;
let mut store = SingleFileMessagePackBackingStore::new(&link); SingleFileMessagePackBackingStore::new(&link).push(&snapshot_envelope("b", b"2"))?;
let errors = [ assert!(fs::symlink_metadata(&link)?.file_type().is_file(), "the link was replaced by a regular file");
store.push(&snapshot_envelope("b", b"2")).unwrap_err(), assert_eq!(fs::read(&target)?, before, "the store the link pointed to is unchanged");
store.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default()).unwrap_err(),
store.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions { soft: true }).unwrap_err(),
store.delete(&snapshot_envelope("a", b"1")).unwrap_err(),
store.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).unwrap_err(),
store.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).err().expect("refused"),
];
for error in &errors {
let linked = error.downcast_ref::<StoreLinkedError>().unwrap_or_else(|| panic!("{error:#}"));
assert_eq!(linked.path, link);
assert!(error.to_string().contains(&link.display().to_string()), "{error:#}");
}
assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was");
assert_eq!(fs::read(&target)?, before);
assert_eq!(dir_names(&volume)?, volume_before, "no lock or staging file beside the target");
assert_eq!(dir_names(temp.path())?, ["link.cc", "volume"], "no lock or staging file beside the link");
assert_eq!(store.pull_all()?, vec![snapshot_envelope("a", b"1")]);
Ok(()) Ok(())
} }
// The advisory query is the held record's and the cache's judges; a link at the store path is the write's to refuse.
#[cfg(unix)]
#[test]
fn is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused() -> Result<()> {
let temp = tempfile::tempdir()?;
let target = temp.path().join("store.cc");
store_with(&target, |cache| cache.put("held", &settings_one()).map(drop))?;
let before = fs::read(&target)?;
let link = temp.path().join("link.cc");
std::os::unix::fs::symlink(&target, &link)?;
let mut cache = CultCache::new();
cache.register_registry(TestEntries)?;
cache.add_backing_store(SingleFileMessagePackBackingStore::new(&link), [Settings::TYPE])?;
cache.pull_all_backing_stores()?;
let names_before = dir_names(temp.path())?;
assert!(cache.is_writable(Settings::TYPE, "held")?, "the query does not look at the store path");
assert!(cache.is_writable(Settings::TYPE, "fresh")?, "the query does not look at the store path");
let error = cache.put("fresh", &settings_one()).unwrap_err();
let linked = error.downcast_ref::<StoreLinkedError>().unwrap_or_else(|| panic!("{error:#}"));
assert_eq!(linked.path, link);
assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was");
assert_eq!(fs::read(&target)?, before);
assert_eq!(dir_names(temp.path())?, names_before, "the refused write left no lock or staging file");
Ok(())
}
#[cfg(unix)]
#[test]
fn a_write_at_a_live_directory_link_is_refused_as_a_link() -> Result<()> {
let temp = tempfile::tempdir()?;
let real = temp.path().join("real");
fs::create_dir(&real)?;
let link = temp.path().join("dirlink");
std::os::unix::fs::symlink(&real, &link)?;
let error = SingleFileMessagePackBackingStore::new(&link).push(&snapshot_envelope("a", b"1")).unwrap_err();
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "{error:#}");
assert!(fs::symlink_metadata(&link)?.file_type().is_symlink());
assert_eq!(fs::read_dir(&real)?.count(), 0);
Ok(())
}
#[cfg(unix)]
fn dir_names(dir: &Path) -> Result<Vec<String>> {
let mut names = fs::read_dir(dir)?.map(|entry| Ok(entry?.file_name().to_string_lossy().into_owned())).collect::<Result<Vec<_>>>()?;
names.sort();
Ok(names)
}
// The two cases the vectors cannot hold: nothing at the path is written, and a dangling link is refused and left as it is. // The two cases the vectors cannot hold: nothing at the path is written, and a dangling link is refused and left as it is.
#[test] #[test]
fn a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link() -> Result<()> { fn a_soft_push_all_writes_where_nothing_is_and_refuses_a_dangling_link() -> Result<()> {
@ -8562,7 +7951,7 @@ mod tests {
let error = SingleFileMessagePackBackingStore::new(&link) let error = SingleFileMessagePackBackingStore::new(&link)
.push_all(&[snapshot_envelope("a", b"1")], PushAllOptions { soft: true }) .push_all(&[snapshot_envelope("a", b"1")], PushAllOptions { soft: true })
.unwrap_err(); .unwrap_err();
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "{error:#}"); assert!(io_not_found(&error), "{error:#}");
assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link was replaced"); assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link was replaced");
assert_eq!(fs::read_dir(&volume)?.count(), 0, "nothing reached the link's volume"); assert_eq!(fs::read_dir(&volume)?.count(), 0, "nothing reached the link's volume");
} }

View file

@ -619,23 +619,6 @@ fn depth_is_refused_at_load() {
assert!(cache.snapshot().is_empty()); assert!(cache.snapshot().is_empty());
} }
// load_soa is a load too: a table holding a payload nested past the bound is refused with the typed depth refusal, the
// prior view is kept, and a tail behind the depth is still the depth refusal.
#[test]
fn load_soa_refuses_a_payload_nested_past_the_bound() {
let mut deep = vec![0x91u8; MAX_PAYLOAD_DEPTH + 1];
deep.push(0xc0);
let mut cache = CultCache::new();
cache.register_entry_type::<UnionSurface>().unwrap();
cache.load_soa(CultCacheSoaTable::from_envelopes(vec![envelope("kept", &vector("union-surface.msgpack"))])).unwrap();
let before = cache.snapshot();
for payload in [deep.clone(), [deep.clone(), vec![0xc0]].concat()] {
let error = cache.load_soa(CultCacheSoaTable::from_envelopes(vec![envelope("k", &payload)])).unwrap_err();
assert_eq!(error.downcast_ref::<PayloadTooDeep>(), Some(&TOO_DEEP), "{error:#}");
assert_eq!(cache.snapshot(), before, "a refused load keeps the prior view");
}
}
// A cache whose load was refused refuses every write and removal and leaves the store as it was: a // A cache whose load was refused refuses every write and removal and leaves the store as it was: a
// later write would otherwise persist the empty image over the records it never read. // later write would otherwise persist the empty image over the records it never read.
#[test] #[test]

View file

@ -6,7 +6,7 @@
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use cultcache_rs::{ use cultcache_rs::{
CultCache, CultCacheEnvelope, CultCacheLoadRefused, CultCacheSoaTable, DatabaseEntry, SingleFileMessagePackBackingStore, CultCache, CultCacheEnvelope, CultCacheSoaTable, DatabaseEntry, SingleFileMessagePackBackingStore,
}; };
use serde::Deserialize; use serde::Deserialize;
@ -419,54 +419,6 @@ fn a_load_of_a_duplicate_image_is_refused_and_keeps_the_prior_view() {
soa.put("item:y", &unique("K")).unwrap(); soa.put("item:y", &unique("K")).unwrap();
} }
// A pull refused for a duplicate unique-index value, or for one key held under two types in two stores, closes the same
// write gate a payload refusal does: the cache holds no image of its stores, so a write would lay the empty image over them.
#[test]
fn a_pull_refused_for_a_duplicate_index_or_a_retype_closes_every_write() {
fn assert_gated(cache: &mut CultCache, path: &Path, door_label: &str) {
let before = std::fs::read(path).unwrap();
let errors = [
("put", cache.put("item:new", &unique("N")).unwrap_err()),
("put_raw_envelope", cache.put_raw_envelope(cache.prepare_entry("item:new", &unique("N")).unwrap().0).unwrap_err()),
("delete", cache.delete::<Unique>("item:keep").unwrap_err()),
];
for (door, error) in errors {
assert!(error.downcast_ref::<CultCacheLoadRefused>().is_some(), "{door_label} {door}: {error:#}");
}
assert_eq!(std::fs::read(path).unwrap(), before, "{door_label}: the store changed");
}
// Duplicate index value found in the stored image.
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("dup.cc");
let mut cache = cache_at(&path);
cache.put("item:keep", &unique("K")).unwrap();
let mut twin = twin_cache_at(&path);
twin.put("item:b", &Twin { sku: "S".into(), category: "c".into() }).unwrap();
twin.put("item:a", &Twin { sku: "S".into(), category: "c".into() }).unwrap();
assert!(error_text(cache.pull_all_backing_stores()).contains("both hold unique index 'Sku'"));
assert_gated(&mut cache, &path, "duplicate index");
// One key held under two types by two stores. The cache pulled cleanly first, so only the refused pull can close the gate.
let (first, second) = (dir.path().join("first.cc"), dir.path().join("second.cc"));
let mut cache = CultCache::new();
cache.register_entry_type::<Plain>().unwrap();
cache.register_entry_type::<Other>().unwrap();
cache.register_entry_type::<Unique>().unwrap();
cache.add_backing_store(SingleFileMessagePackBackingStore::new(&first), [Plain::TYPE]).unwrap();
cache.add_backing_store(SingleFileMessagePackBackingStore::new(&second), [Other::TYPE]).unwrap();
cache.pull_all_backing_stores().unwrap();
cache.put("k", &Plain { value: "p".into() }).unwrap();
let mut seed = CultCache::new();
seed.register_entry_type::<Other>().unwrap();
seed.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&second)).unwrap();
seed.pull_all_backing_stores().unwrap();
seed.put("k", &Other { sku: "o".into() }).unwrap();
assert!(error_text(cache.pull_all_backing_stores()).contains("\"k\""));
assert_gated(&mut cache, &first, "retype");
assert_eq!(cache.get::<Plain>("k").unwrap().unwrap().value, "p", "a refused pull keeps the prior view");
}
#[test] #[test]
fn a_multi_index_store_with_a_duplicate_on_the_second_alias_refuses_to_open() { fn a_multi_index_store_with_a_duplicate_on_the_second_alias_refuses_to_open() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();

View file

@ -5,15 +5,9 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following
## [Unreleased] ## [Unreleased]
### Changed
- `withStoreWriteLock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. Pinned by a test that puts the link while another holder owns the lock.
- `CultCache.isWritable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
### Breaking ### Breaking
- A write at a single-file store path that is a symbolic link, live or dangling, is refused with the new `StoreLinkedError` (exported from the package entry; `error.path` names the path) before the lock and the replace, and writes nothing: `push`, `pushAll`, `delete` and a soft flush alike. Before, a write at a live link replaced the link with a regular file, a dangling link failed with `ENOENT` (or, for `pushAll`, was replaced), and a soft flush at a link skipped. A caller catching `ENOENT` around a write at a dangling link now catches `StoreLinkedError`; configure the link's target as the store path. Reads through a live link work. An `lstat` failure other than `ENOENT` (`ENOTDIR`, `EACCES`, `ELOOP`) is rethrown as it is. - `SingleFileMessagePackBackingStore.pushAll` writes the given entries onto the store as it is under the lock, as `push` does: a record under the same key is replaced and no other is removed. It used to replace the whole file with the entries, so `CultMesh.flush()` from a cache that held less than the store (a node created with `pullOnStart: false`, or one whose image was stale) wiped the records it never read. A caller that relied on `pushAll` to reset a store removes the file first. A soft flush and the dangling-link case are unchanged. `CultCache.storedSnapshot()` is the image as the stores hold it (a global loaded from a legacy key under that key) and is what `CultMesh.flush` persists, so a flush no longer leaves such a global under two keys; `snapshot()` is unchanged.
- `SingleFileMessagePackBackingStore.pushAll` writes the given entries onto the store as it is under the lock, as `push` does: a record under the same key is replaced and no other is removed. It used to replace the whole file with the entries, so `CultMesh.flush()` from a cache that held less than the store (a node created with `pullOnStart: false`, or one whose image was stale) wiped the records it never read. A caller that relied on `pushAll` to reset a store removes the file first. A soft flush is unchanged. `CultCache.storedSnapshot()` is the image as the stores hold it (a global loaded from a legacy key under that key) and is what `CultMesh.flush` persists, so a flush no longer leaves such a global under two keys; `snapshot()` is unchanged.
- A single-file store marked `cultcache.store.v3` (it holds an element id, written by the C# runtime) is read, and a rewrite keeps the mark and the ids; this runtime does not decide when a document carries ids. A runtime from before this change refuses a v3 store by name, so rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - A single-file store marked `cultcache.store.v3` (it holds an element id, written by the C# runtime) is read, and a rewrite keeps the mark and the ids; this runtime does not decide when a document carries ids. A runtime from before this change refuses a v3 store by name, so rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading.
- `indexes` are unique, as C# `[CultIndex]`: a write or a load that leaves two records of a type holding one non-blank - `indexes` are unique, as C# `[CultIndex]`: a write or a load that leaves two records of a type holding one non-blank
value under one alias is refused naming both keys and the alias, and writes nothing. Index text is the C# canonical value under one alias is refused naming both keys and the alias, and writes nothing. Index text is the C# canonical

View file

@ -615,8 +615,7 @@ export class CultCache {
* another key, a record carrying unread slots nothing consents to). It answers the WRITE form. A removal asks the judge's * another key, a record carrying unread slots nothing consents to). It answers the WRITE form. A removal asks the judge's
* removal form and differs in two states: an owner's `delete` of a record held in another store than its own home succeeds * removal form and differs in two states: an owner's `delete` of a record held in another store than its own home succeeds
* where this is false (a write would push to the wrong store), and `delete` of a global type at a key other than __global__ is * where this is false (a write would push to the wrong store), and `delete` of a global type at a key other than __global__ is
* the plain false of an unheld key where this is false. It does not look at store paths: a link at a store path is not predicted, and * the plain false of an unheld key where this is false. Live and advisory: the write or removal still decides.
* the write refuses it with `StoreLinkedError`. Live and advisory: the write or removal still decides.
*/ */
isWritable(definition: AnyCultCacheDocumentDefinition, key: string): boolean { isWritable(definition: AnyCultCacheDocumentDefinition, key: string): boolean {
const registered = this.#requireDefinition(definition); const registered = this.#requireDefinition(definition);

View file

@ -1,3 +1,2 @@
export * from "./browser"; export * from "./browser";
export { SingleFileMessagePackBackingStore, readSingleFileStore, type SingleFileStoreRead } from "./single-file-messagepack-backing-store"; export { SingleFileMessagePackBackingStore, readSingleFileStore, type SingleFileStoreRead } from "./single-file-messagepack-backing-store";
export { StoreLinkedError } from "./store-file";

View file

@ -87,14 +87,25 @@ export class SingleFileMessagePackBackingStore implements CacheBackingStore {
// another writer added. The store is read by the same reader as `pullAll`, so one it would refuse (not exactly one // another writer added. The store is read by the same reader as `pullAll`, so one it would refuse (not exactly one
// store, a variant, a body it cannot decode) is refused and left as it is. A soft flush asks that reader too: nothing // store, a variant, a body it cannot decode) is refused and left as it is. A soft flush asks that reader too: nothing
// at the path is written, a store is left alone. // at the path is written, a store is left alone.
// The reader owns the verdict on what is at the path: a dangling link is refused here as it is for open and push. let held: CultCacheEnvelope[] = [];
const disk = this.#readDisk(); try {
if (options.soft && !disk.absent) { const disk = this.#readDisk();
return; if (options.soft && !disk.absent) {
} return;
}
this.#format = disk.format; this.#format = disk.format;
const held = disk.envelopes; held = disk.envelopes;
} catch (error) {
// Only a dangling link reaches here with ENOENT: it holds no store, so a flush has nothing to merge with and no
// header to keep, and replaces the link (R3 pins this). A soft flush does not replace what it cannot read, and every
// merging writer has already refused the link when it read the current entries.
if (options.soft || (error as NodeJS.ErrnoException).code !== "ENOENT") {
throw error;
}
this.#format = STORE_FORMAT_VERSION;
}
// A key is held once: of same-type entries sharing a key in one batch, the last is kept. // A key is held once: of same-type entries sharing a key in one batch, the last is kept.
const supplied = [...new Map(entries.map((entry) => [entry.key, entry])).values()]; const supplied = [...new Map(entries.map((entry) => [entry.key, entry])).values()];

View file

@ -5,7 +5,7 @@
// lazily inside `loadKernel`, so a consumer that never writes a store never loads it. // lazily inside `loadKernel`, so a consumer that never writes a store never loads it.
import { closeSync, openSync, constants as fsConstants } from "node:fs"; import { closeSync, openSync, constants as fsConstants } from "node:fs";
import { lstat, mkdir, open, rename, rm } from "node:fs/promises"; import { mkdir, open, rename, rm } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises"; import type { FileHandle } from "node:fs/promises";
import { dirname } from "node:path"; import { dirname } from "node:path";
@ -25,30 +25,6 @@ export class StoreReplacedNotDurableError extends Error {
} }
} }
/** Thrown when a write is asked of a store path that is a link: a replace there would swap the link for a regular file. */
export class StoreLinkedError extends Error {
constructor(readonly path: string) {
super(`CultCache store ${path} is a link, and a write at it would replace the link; configure the link's target as the store path.`);
this.name = "StoreLinkedError";
}
}
/** Refuses a link at the store path (file or directory link; libuv reports Windows junctions as links). Absent is not a link. */
export async function refuseLinkedStore(storePath: string): Promise<void> {
let linked = false;
try {
linked = (await lstat(storePath)).isSymbolicLink();
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") {
throw error;
}
}
if (linked) {
throw new StoreLinkedError(storePath);
}
}
/** The file operations of one commit. Tests record or fail them; production uses `nodeStoreFileOps`. */ /** The file operations of one commit. Tests record or fail them; production uses `nodeStoreFileOps`. */
export interface StoreFileOps { export interface StoreFileOps {
createExclusive(path: string): Promise<FileHandle>; createExclusive(path: string): Promise<FileHandle>;
@ -201,7 +177,6 @@ export async function withStoreWriteLock<T>(
): Promise<T> { ): Promise<T> {
const lockPath = `${storePath}.lock`; const lockPath = `${storePath}.lock`;
const timeoutMs = options.timeoutMs ?? LOCK_TIMEOUT_MS; const timeoutMs = options.timeoutMs ?? LOCK_TIMEOUT_MS;
await refuseLinkedStore(storePath);
await mkdir(dirname(lockPath), { recursive: true }); await mkdir(dirname(lockPath), { recursive: true });
const started = Date.now(); const started = Date.now();
@ -218,8 +193,6 @@ export async function withStoreWriteLock<T>(
} }
try { try {
// A link may have been put at the path while the lock was awaited; the look above only saw the path before.
await refuseLinkedStore(storePath);
return await operation(); return await operation();
} finally { } finally {
release(); release();

View file

@ -15,7 +15,6 @@ import { CultCache } from "../src/cult-cache";
import { inspectCultCacheBytes } from "../src/cult-cache-inspector"; import { inspectCultCacheBytes } from "../src/cult-cache-inspector";
import { defineDocumentRegistry, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; import { defineDocumentRegistry, defineDocumentType, documentPayloadBytes, documentPayloadFromBytes } from "../src/document";
import { SingleFileMessagePackBackingStore, readSingleFileStore, readStoreBytes } from "../src/single-file-messagepack-backing-store"; import { SingleFileMessagePackBackingStore, readSingleFileStore, readStoreBytes } from "../src/single-file-messagepack-backing-store";
import { StoreLinkedError, withStoreWriteLock } from "../src/store-file";
import { SchemaConflictError, StoreUnreadableError, refuseUnreadStoreBytes } from "../src/store-format"; import { SchemaConflictError, StoreUnreadableError, refuseUnreadStoreBytes } from "../src/store-format";
import type { AnyCultCacheDocumentDefinition, CacheBackingStore, CultCacheEnvelope, CultCacheSchema } from "../src/types"; import type { AnyCultCacheDocumentDefinition, CacheBackingStore, CultCacheEnvelope, CultCacheSchema } from "../src/types";
@ -34,7 +33,8 @@ const cultcacheRsRoot = existsSync(resolve(cultLibRoot, "packages", "cultcache-r
? resolve(cultLibRoot, "packages", "cultcache-rs") ? resolve(cultLibRoot, "packages", "cultcache-rs")
: resolve(cultCacheTsRoot, "..", "cultcache-rs"); : resolve(cultCacheTsRoot, "..", "cultcache-rs");
const rustInteropBinary = resolve( const rustInteropBinary = resolve(
process.env.CARGO_TARGET_DIR ?? resolve(cultcacheRsRoot, "target"), cultcacheRsRoot,
"target",
"debug", "debug",
"examples", "examples",
process.platform === "win32" ? "cultcache_interop.exe" : "cultcache_interop", process.platform === "win32" ? "cultcache_interop.exe" : "cultcache_interop",
@ -1273,48 +1273,6 @@ test("CultCache v1 MessagePack stores are readable across TS, Rust, C#, and Pyth
} }
}); });
// A Bifrost-shaped store (a union field, nested bytes) written by one of C# and Rust is opened by the other: C# reads through
// its store, Rust through the read-only snapshot door, which takes no lock and writes nothing. Leftover staging files from an
// interrupted write of either runtime sit beside the store; a read neither deletes them nor lets them change what it reads, and
// the store file's bytes are the same after every read.
test("A store written by C# or Rust with a union and nested bytes is read back equal by the other, read-only, past leftover staging files", async () => {
await buildInteropPeers();
const tempDir = await mkdtemp(join(tmpdir(), "cultcache-shapes-"));
const rustRead = (file: string) => runJsonCommand("rust-read-shapes", rustInteropBinary, ["read-shapes", "--file", file], cultcacheRsRoot);
const csharpRead = (file: string) => runJsonCommand("csharp-read-shapes", dotnetCommand, [csharpInteropDll, "read-shapes", "--file", file], cultLibRoot);
const peers = [
{
writer: "csharp",
write: (file: string) => runJsonCommand("csharp-write-shapes", dotnetCommand, [csharpInteropDll, "write-shapes", "--file", file], cultLibRoot),
read: rustRead,
},
{
writer: "rust",
write: (file: string) => runJsonCommand("rust-write-shapes", rustInteropBinary, ["write-shapes", "--file", file], cultcacheRsRoot),
read: csharpRead,
},
];
for (const { writer, write, read } of peers) {
const file = join(tempDir, `${writer}.cc`);
const written = await write(file);
assert.equal(written.length, 2);
// The leftovers an interrupted write leaves: C# stages as `.<name>.<guid N>.tmp`, Rust as `<name>.<guid>.tmp`; both hold partial bytes.
const leftovers = [
join(tempDir, `.${writer}.cc.0123456789abcdef0123456789abcdef.tmp`),
join(tempDir, `${writer}.cc.00000000-0000-4000-8000-000000000000.tmp`),
];
for (const leftover of leftovers) {
await writeFile(leftover, Buffer.from([0x93, 0xb2]));
}
const before = await readFile(file);
assert.deepEqual(await read(file), written, `${writer}'s store did not read back equal`);
assert.deepEqual(await readFile(file), before, `reading ${writer}'s store changed its bytes`);
for (const leftover of leftovers) {
assert.ok(await exists(leftover), `reading ${writer}'s store removed ${leftover}`);
}
}
});
// Records and catalog entries a runtime does not own survive its writes. C# writes a store and each other runtime writes its own // Records and catalog entries a runtime does not own survive its writes. C# writes a store and each other runtime writes its own
// note into it: C# reads its note back, and its record and catalog entry are as C# wrote them. C# then writes again, and the other // note into it: C# reads its note back, and its record and catalog entry are as C# wrote them. C# then writes again, and the other
// runtime's record and catalog entry are as that runtime wrote them. Identity is compared on the decoded MessagePack values, every // runtime's record and catalog entry are as that runtime wrote them. Identity is compared on the decoded MessagePack values, every
@ -2106,7 +2064,7 @@ async function readTsInteropStore(file: string): Promise<InteropNote> {
async function buildInteropPeers(): Promise<void> { async function buildInteropPeers(): Promise<void> {
if (!(await exists(rustInteropBinary))) { if (!(await exists(rustInteropBinary))) {
await execAsync(`"${cargoCommand}" build --locked --quiet --example cultcache_interop`, { await execAsync(`"${cargoCommand}" build --quiet --example cultcache_interop`, {
cwd: cultcacheRsRoot, cwd: cultcacheRsRoot,
}); });
} }
@ -2467,36 +2425,12 @@ test("SingleFileMessagePackBackingStore refuses a dangling symlink at its path a
() => store.pullAll(), () => store.pullAll(),
(error: NodeJS.ErrnoException) => error.code === "ENOENT" && error.message.includes("symbolic link") && error.message.includes(file), (error: NodeJS.ErrnoException) => error.code === "ENOENT" && error.message.includes("symbolic link") && error.message.includes(file),
); );
await assert.rejects(() => store.push(envelope), (error: unknown) => error instanceof StoreLinkedError); await assert.rejects(() => store.push(envelope), (error: NodeJS.ErrnoException) => error.code === "ENOENT");
await assert.rejects(() => store.pushAll([envelope], { soft: true }), (error: unknown) => error instanceof StoreLinkedError); await assert.rejects(() => store.pushAll([envelope], { soft: true }), (error: NodeJS.ErrnoException) => error.code === "ENOENT");
assert.ok((await lstat(file)).isSymbolicLink()); assert.ok((await lstat(file)).isSymbolicLink());
assert.deepEqual(await readdir(volume), []); assert.deepEqual(await readdir(volume), []);
}); });
// The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse.
test("isWritable over a store at a live link answers true while the cache write is refused", { skip: process.platform === "win32" }, async () => {
const dir = await mkdtemp(join(tmpdir(), "cultcache-writable-link-"));
const volume = join(dir, "volume");
await mkdir(volume);
const target = join(volume, "store.cc");
const link = join(dir, "link.cc");
const document = defineDocumentType({ type: "note", schema: z.object({ text: z.string() }) });
await CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(target)).build()
.put(document, "held", { text: "held" });
const before = await readFile(target);
await symlink(target, link);
const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(link)).build();
await cache.pullAllBackingStores();
const namesBefore = [(await readdir(dir)).sort(), (await readdir(volume)).sort()];
assert.equal(cache.isWritable(document, "held"), true);
assert.equal(cache.isWritable(document, "fresh"), true);
await assert.rejects(() => cache.put(document, "fresh", { text: "fresh" }), (error: unknown) => error instanceof StoreLinkedError && error.path === link);
assert.ok((await lstat(link)).isSymbolicLink());
assert.deepEqual(await readFile(target), before);
assert.deepEqual([(await readdir(dir)).sort(), (await readdir(volume)).sort()], namesBefore);
});
// The second look of the read, staged: nothing is at the path only when the read and the lstat both find nothing. A path // The second look of the read, staged: nothing is at the path only when the read and the lstat both find nothing. A path
// that changed between them (a link in a parent swapped for a file, a store that appeared) is never an empty store. // that changed between them (a link in a parent swapped for a file, a store that appeared) is never an empty store.
test("readStoreBytes reports nothing at the path only when the lstat finds nothing too", async () => { test("readStoreBytes reports nothing at the path only when the lstat finds nothing too", async () => {
@ -2533,9 +2467,9 @@ test("SingleFileMessagePackBackingStore record refusal names only a string key",
} }
}); });
// A write at a store path that is a link is refused: a replace there would swap the link for a regular file and // What a write does through a symbolic link today. R3 decides whether a write resolves the link
// detach the store from its target. Reads through a live link keep working. // or refuses it; these pin the current behaviour so that change is deliberate.
test("a write at a live link is refused and nothing changes, while a read through it still works", { skip: process.platform === "win32" }, async () => { test("R3 decides resolve-or-refuse: a push through a live link replaces the link", { skip: process.platform === "win32" }, async () => {
const dir = await mkdtemp(join(tmpdir(), "cultcache-live-link-")); const dir = await mkdtemp(join(tmpdir(), "cultcache-live-link-"));
const volume = join(dir, "volume"); const volume = join(dir, "volume");
await mkdir(volume); await mkdir(volume);
@ -2543,42 +2477,21 @@ test("a write at a live link is refused and nothing changes, while a read throug
const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" };
await new SingleFileMessagePackBackingStore(target).push(envelope); await new SingleFileMessagePackBackingStore(target).push(envelope);
const before = await readFile(target); const before = await readFile(target);
const volumeBefore = (await readdir(volume)).sort(); const link = join(dir, "store.cc");
const link = join(dir, "link.cc");
await symlink(target, link); await symlink(target, link);
const store = new SingleFileMessagePackBackingStore(link); await new SingleFileMessagePackBackingStore(link).push({ ...envelope, key: "b" });
const refused = (error: unknown) => error instanceof StoreLinkedError && error.path === link && error.message.includes(link); assert.ok((await lstat(link)).isFile());
await assert.rejects(() => store.push({ ...envelope, key: "b" }), refused);
await assert.rejects(() => store.pushAll([{ ...envelope, key: "b" }]), refused);
await assert.rejects(() => store.delete(envelope), refused);
assert.ok((await lstat(link)).isSymbolicLink());
assert.deepEqual(await readFile(target), before); assert.deepEqual(await readFile(target), before);
assert.deepEqual((await readdir(dir)).sort(), ["link.cc", "volume"]);
assert.deepEqual((await readdir(volume)).sort(), volumeBefore);
assert.deepEqual((await store.pullAll()).map((e) => e.key), ["a"]);
}); });
test("a write at a live directory link is refused as a link", { skip: process.platform === "win32" }, async () => { test("R3 decides resolve-or-refuse: a pushAll through a dangling link replaces the link", { skip: process.platform === "win32" }, async () => {
const dir = await mkdtemp(join(tmpdir(), "cultcache-dir-link-"));
const realDir = join(dir, "real");
await mkdir(realDir);
const link = join(dir, "dirlink");
await symlink(realDir, link);
const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" };
await assert.rejects(() => new SingleFileMessagePackBackingStore(link).push(envelope), (error: unknown) => error instanceof StoreLinkedError);
assert.ok((await lstat(link)).isSymbolicLink());
assert.deepEqual(await readdir(realDir), []);
});
test("a pushAll through a dangling link is refused like push, and the link and its volume are left as they were", { skip: process.platform === "win32" }, async () => {
const dir = await mkdtemp(join(tmpdir(), "cultcache-dangling-push-all-")); const dir = await mkdtemp(join(tmpdir(), "cultcache-dangling-push-all-"));
const volume = join(dir, "volume"); const volume = join(dir, "volume");
await mkdir(volume); await mkdir(volume);
const link = join(dir, "store.cc"); const link = join(dir, "store.cc");
await symlink(join(volume, "store.cc"), link); await symlink(join(volume, "store.cc"), link);
const envelope = { key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }; await new SingleFileMessagePackBackingStore(link).pushAll([{ key: "a", type: "t", payload: documentPayloadFromBytes(Uint8Array.of(0x90)), storedAt: "2026-09-30T00:00:00Z" }]);
await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([envelope]), (error: unknown) => error instanceof StoreLinkedError); assert.ok((await lstat(link)).isFile());
assert.ok((await lstat(link)).isSymbolicLink());
assert.deepEqual(await readdir(volume), []); assert.deepEqual(await readdir(volume), []);
}); });
@ -2872,6 +2785,7 @@ test("every registration refusal is a SchemaConflictError naming the id and both
// Cross-runtime store lock: each runtime's writer excludes the others on `<file>.lock`. // Cross-runtime store lock: each runtime's writer excludes the others on `<file>.lock`.
import { spawn } from "node:child_process"; import { spawn } from "node:child_process";
import { withStoreWriteLock } from "../src/store-file";
type LockHolder = { exited: Promise<number | null>; stderr: () => string }; type LockHolder = { exited: Promise<number | null>; stderr: () => string };

View file

@ -10,7 +10,6 @@ import { promisify } from "node:util";
import { decode, encode } from "@msgpack/msgpack"; import { decode, encode } from "@msgpack/msgpack";
import { StoreLinkedError as PackageStoreLinkedError } from "../src/index";
import { documentPayloadBytes, documentPayloadFromBytes } from "../src/document"; import { documentPayloadBytes, documentPayloadFromBytes } from "../src/document";
import { SingleFileMessagePackBackingStore } from "../src/single-file-messagepack-backing-store"; import { SingleFileMessagePackBackingStore } from "../src/single-file-messagepack-backing-store";
import { SchemaConflictError, StoreUnreadableError } from "../src/store-format"; import { SchemaConflictError, StoreUnreadableError } from "../src/store-format";
@ -24,7 +23,6 @@ import {
replaceStoreFile, replaceStoreFile,
replaceStoreFileWindows, replaceStoreFileWindows,
windowsKernel, windowsKernel,
refuseLinkedStore,
withStoreWriteLock, withStoreWriteLock,
} from "../src/store-file"; } from "../src/store-file";
import type { ReplaceKernel, StoreFileOps } from "../src/store-file"; import type { ReplaceKernel, StoreFileOps } from "../src/store-file";
@ -417,43 +415,6 @@ test("a second writer in this process waits for the store lock and runs after re
} }
}); });
test("a link put while a writer waits on the store lock is refused once it holds the lock", { skip: process.platform === "win32" }, async () => {
const directory = await scratch();
try {
const target = join(directory, "target.cc");
await new SingleFileMessagePackBackingStore(target).push(noteEnvelope("a"));
const before = await readFile(target);
const writes: Array<[string, (store: SingleFileMessagePackBackingStore) => Promise<void>]> = [
["push", (store) => store.push(noteEnvelope("b"))],
["pushAll", (store) => store.pushAll([noteEnvelope("b")])],
["delete", (store) => store.delete(noteEnvelope("a"))],
];
const missing = join(directory, "missing.cc");
for (const [name, write, linkTo] of writes.flatMap(([n, w]) => [[n, w, target], [`${n} (dangling)`, w, missing]] as const)) {
const link = join(directory, `${name.replace(/\W+/g, "-")}.cc`);
let writer: Promise<void> | undefined;
await withStoreWriteLock(link, async () => {
writer = write(new SingleFileMessagePackBackingStore(link));
writer.catch(() => undefined);
await sleepFor(150);
await fsp.symlink(linkTo, link); // after the writer's pre-lock look, while it waits
});
await assert.rejects(writer!, (error: unknown) => error instanceof PackageStoreLinkedError, name);
assert.ok((await fsp.lstat(link)).isSymbolicLink(), `${name}: the link was replaced`);
assert.deepEqual(await readFile(target), before, `${name}: the target changed`);
await assert.rejects(fsp.lstat(missing), { code: "ENOENT" }, `${name}: the dangling target was created`);
await rm(link); // the link is refused before the lock, so remove it to probe the lock
assert.equal(await withStoreWriteLock(link, async () => "free", { timeoutMs: 200 }), "free", `${name}: the lock is held`);
}
} finally {
await rm(directory, { recursive: true, force: true });
}
});
function noteEnvelope(key: string): CultCacheEnvelope {
return { key, type: "note", payload: documentPayloadFromBytes(Uint8Array.of(0x91, 1)), storedAt: "2026-10-20T00:00:00Z" };
}
test("the store lock gives up after its wait bound and names the lock path", async () => { test("the store lock gives up after its wait bound and names the lock path", async () => {
const directory = await scratch(); const directory = await scratch();
try { try {
@ -857,35 +818,3 @@ test("opening writes nothing: a missing store leaves no lock file, a legacy stor
await rm(directory, { recursive: true, force: true }); await rm(directory, { recursive: true, force: true });
} }
}); });
test("StoreLinkedError is importable from the package entry and is what a write at a link throws", { skip: process.platform === "win32" }, async () => {
const directory = await scratch();
try {
const target = join(directory, "target.cc");
await writeFile(target, "x");
const link = join(directory, "link.cc");
await fsp.symlink(target, link);
await assert.rejects(
() => refuseLinkedStore(link),
(error: unknown) => error instanceof PackageStoreLinkedError && error.path === link && error.name === "StoreLinkedError",
);
await assert.rejects(() => new SingleFileMessagePackBackingStore(link).pushAll([]), (error: unknown) => error instanceof PackageStoreLinkedError);
} finally {
await rm(directory, { recursive: true, force: true });
}
});
test("refuseLinkedStore lets a path that is absent pass and rethrows any other lstat failure", { skip: process.platform === "win32" }, async () => {
const directory = await scratch();
try {
await refuseLinkedStore(join(directory, "absent.cc"));
const file = join(directory, "file");
await writeFile(file, "x");
await assert.rejects(
() => refuseLinkedStore(join(file, "child.cc")),
(error: unknown) => (error as NodeJS.ErrnoException).code === "ENOTDIR",
);
} finally {
await rm(directory, { recursive: true, force: true });
}
});

View file

@ -147,8 +147,8 @@ CultMath names with tests instead of copied project-local helper drift.
```powershell ```powershell
dotnet build packages/cultmath/src/CultMath/CultMath.csproj dotnet build packages/cultmath/src/CultMath/CultMath.csproj
dotnet test packages/cultmath/tests/CultMath.Tests/CultMath.Tests.csproj dotnet test packages/cultmath/tests/CultMath.Tests/CultMath.Tests.csproj
cargo test --locked --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml cargo test --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml
cargo build --locked --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml cargo build --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml
.\packages\cultmath\scripts\build-unity-package.ps1 .\packages\cultmath\scripts\build-unity-package.ps1
``` ```

View file

@ -9,10 +9,6 @@ All notable changes to `@gamecult/cultmesh-browser` are recorded here, following
- The package is renamed `cultmesh-browser` to `@gamecult/cultmesh-browser` (the repo already scopes `@gamecult/cultcache-ts`), so the unscoped name cannot be claimed by someone else. Import from `@gamecult/cultmesh-browser` and its subpaths; the API is unchanged. Depends on `@gamecult/cultnet-ts`; the interop test uses `@gamecult/cultmesh-ts`. - The package is renamed `cultmesh-browser` to `@gamecult/cultmesh-browser` (the repo already scopes `@gamecult/cultcache-ts`), so the unscoped name cannot be claimed by someone else. Import from `@gamecult/cultmesh-browser` and its subpaths; the API is unchanged. Depends on `@gamecult/cultnet-ts`; the interop test uses `@gamecult/cultmesh-ts`.
### Dependencies
- Needs `@gamecult/cultnet-ts` `^0.4.0` (was `^0.3.0`), moving with the `cultnet-ts` 0.4.0 release.
## [0.4.0] ## [0.4.0]
### Added ### Added

View file

@ -42,7 +42,7 @@
}, },
"dependencies": { "dependencies": {
"@gamecult/cultcache-ts": "^0.15.0", "@gamecult/cultcache-ts": "^0.15.0",
"@gamecult/cultnet-ts": "^0.4.0" "@gamecult/cultnet-ts": "^0.3.0"
}, },
"devDependencies": { "devDependencies": {
"@types/node": "^24.0.0", "@types/node": "^24.0.0",

View file

@ -180,21 +180,6 @@ class RemoteWriteAuthorityTests(unittest.TestCase):
reply = self.send(granting, unowned) reply = self.send(granting, unowned)
self.assertEqual(reply[0]["code"], "unregistered_document_put", "a granted put still meets the store's refusal") self.assertEqual(reply[0]["code"], "unregistered_document_put", "a granted put still meets the store's refusal")
def test_a_put_with_a_payload_tail_and_an_unknown_schema_id_is_refused_whole_under_the_canonical_id(self) -> None:
# The payload head names the registered schema, so the put routes by stamp; the tail makes the payload more
# than one MessagePack value. It is answered message_invalid (not unregistered_document_put), nothing is
# stored, and the authority is asked under the canonical id, not the id as sent.
authority = Recording(lambda _: CultMeshRemoteWriteDecision.grant())
server = self.build(authority)
tailed = put_of(KEY_A, "x", schema_id="sha256:nobody-owns-this")
tailed["document"]["payload"] = bytes(tailed["document"]["payload"]) + b"\x01"
replies = self.send(server, tailed)
self.assertEqual([(reply["code"], reply["details"]["field"]) for reply in replies], [("message_invalid", "document")])
self.assertIsNone(self.text_of(server, KEY_A))
self.assertEqual([request.schema_id for request in authority.requests], [SCHEMA_ID])
def test_an_authority_that_raises_refuses_and_is_logged_as_an_error(self) -> None: def test_an_authority_that_raises_refuses_and_is_logged_as_an_error(self) -> None:
def explode(_: Any) -> Any: def explode(_: Any) -> Any:
raise RuntimeError("authority exploded") raise RuntimeError("authority exploded")

View file

@ -29,8 +29,6 @@ All notable changes to `@gamecult/cultmesh-ts` are recorded here, following
- Needs the cultcache-ts release that ships with this change (its manifest range moves with that release): it imports `schemaIdentityOf` and `SchemaConflictError`, which 0.14.x does not export. - Needs the cultcache-ts release that ships with this change (its manifest range moves with that release): it imports `schemaIdentityOf` and `SchemaConflictError`, which 0.14.x does not export.
- Needs `@gamecult/cultnet-ts` `^0.4.0` (was `^0.3.0`): this package imports names that `cultnet-ts` 0.3.0 does not export (`cultNetMessageInvalidError`, `cultNetTextError`, `cultNetWriteNotAuthorizedError`, `WriteTracker`, `CULTNET_WRITE_RECEIPT_MESSAGE_VERSIONS`, `helloAdvertisesWriteReceipts`, `resultWindowMs`). The declared-range check (`npm run test:pack:ts`) found the old range admitting a release that lacks them.
## [0.2.0] ## [0.2.0]
### Added ### Added

View file

@ -50,7 +50,7 @@
}, },
"dependencies": { "dependencies": {
"@gamecult/cultcache-ts": "^0.15.0", "@gamecult/cultcache-ts": "^0.15.0",
"@gamecult/cultnet-ts": "^0.4.0", "@gamecult/cultnet-ts": "^0.3.0",
"koffi": "3.3.0" "koffi": "3.3.0"
}, },
"devDependencies": { "devDependencies": {

View file

@ -122,7 +122,7 @@ const rust: Target = {
name: "rust", name: "rust",
payload: (text) => encode({ text }), payload: (text) => encode({ text }),
async start() { async start() {
if (!prebuilt) await execFileAsync(cargo, ["build", "--locked", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir }); if (!prebuilt) await execFileAsync(cargo, ["build", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir });
const metadata = spawnSync(cargo, ["metadata", "--format-version", "1", "--no-deps"], { cwd: rustManifestDir, encoding: "utf8" }); const metadata = spawnSync(cargo, ["metadata", "--format-version", "1", "--no-deps"], { cwd: rustManifestDir, encoding: "utf8" });
if (metadata.status !== 0) throw new Error(`cargo metadata failed: ${metadata.stderr}`); if (metadata.status !== 0) throw new Error(`cargo metadata failed: ${metadata.stderr}`);
const exe = join(JSON.parse(metadata.stdout).target_directory, "debug", "examples", `receipt_interop_server${process.platform === "win32" ? ".exe" : ""}`); const exe = join(JSON.parse(metadata.stdout).target_directory, "debug", "examples", `receipt_interop_server${process.platform === "win32" ? ".exe" : ""}`);

View file

@ -2,7 +2,7 @@
## Unreleased (breaking; version 0.x, so the next minor) ## Unreleased (breaking; version 0.x, so the next minor)
- `decode_witness_artifact_bundle_payload` (and `CultNetWitnessArtifactBundle.from_payload`) read the payload through `cultcache_py.read_payload_tree`: bytes after its first MessagePack value raise `ValueError` `cultnet.witness_artifact_bundle payload has N bytes after its first MessagePack value (refused on read).` instead of `msgpack.ExtraData`, and a payload nested past 256 raises `CultCachePayloadDepthError`. Needs `cultcache-py` 0.4.0, the release that exports `read_payload_tree`: the dependency is now `cultcache-py>=0.4.0` (it was `>=0.2.0`, which admits releases whose `import cultnet_py` fails), and `tests/test_declared_dependency_ranges.py` fails when a declared bound admits a tagged release lacking a name the package imports. The same check found `cultmesh-py` importing `cache.index_text`, which `cultcache-py` 0.2.0 lacks; its bound is `cultcache-py>=0.4.0` too (it has no changelog of its own). `resolve_document_for_raw_record` reads the schema hint from a raw record's first value whether or not bytes follow it, so a record routed by its stamp is refused by count when applied (`ValueError`, in a snapshot, a shard log and a direct apply) like one routed by a registered schema id; `apply_shard_log_response` used to skip it silently as an unknown schema. A tail on a payload whose first value names no registered schema is still `KeyError`, which a shard log skips. - `decode_witness_artifact_bundle_payload` (and `CultNetWitnessArtifactBundle.from_payload`) read the payload through `cultcache_py.read_payload_tree`: bytes after its first MessagePack value raise `ValueError` `cultnet.witness_artifact_bundle payload has N bytes after its first MessagePack value (refused on read).` instead of `msgpack.ExtraData`, and a payload nested past 256 raises `CultCachePayloadDepthError`. Needs `cultcache-py` 0.4.0, the release that exports `read_payload_tree`: the dependency is now `cultcache-py>=0.4.0` (it was `>=0.2.0`, which admits releases whose `import cultnet_py` fails), and `tests/test_declared_dependency_ranges.py` fails when a declared bound admits a tagged release lacking a name the package imports. The same check found `cultmesh-py` importing `cache.index_text`, which `cultcache-py` 0.2.0 lacks; its bound is `cultcache-py>=0.4.0` too (it has no changelog of its own). A raw record with bytes after its first value gives no schema hint to `resolve_document_for_raw_record`, so an unknown schema id stays `KeyError` rather than being routed; a registered one is refused by count when applied.
- `apply_raw_document_record` and `apply_raw_snapshot` raise `SchemaConflictError` when a remote record is of another type than the one the cache holds at its key (`cultcache-py` holds one record per key whatever its type); the cache is unchanged, and the caller decides whether to drop the record or the peer. `cultmesh-py`'s server answers such a put with `message_invalid`; its client's `read_next_change` lets the error out. - `apply_raw_document_record` and `apply_raw_snapshot` raise `SchemaConflictError` when a remote record is of another type than the one the cache holds at its key (`cultcache-py` holds one record per key whatever its type); the cache is unchanged, and the caller decides whether to drop the record or the peer. `cultmesh-py`'s server answers such a put with `message_invalid`; its client's `read_next_change` lets the error out.
- `CultNetDatabaseChange` carries `commit_id` and `commit_count` (`commitId`/`commitCount`; None outside a multi-record commit, as C# writes nil): `from_wire` refuses an empty or non-string id and a count that is not an integer of at least 2, `to_wire` writes them when set, and the published `database_change_raw` schema lists both. Python applies no batch. The `simulation_fact` document's `shard`, `frame`, `subject` and `claim_kind` aliases are `roles` (many facts share each); the catalog bytes do not move. - `CultNetDatabaseChange` carries `commit_id` and `commit_count` (`commitId`/`commitCount`; None outside a multi-record commit, as C# writes nil): `from_wire` refuses an empty or non-string id and a count that is not an integer of at least 2, `to_wire` writes them when set, and the published `database_change_raw` schema lists both. Python applies no batch. The `simulation_fact` document's `shard`, `frame`, `subject` and `claim_kind` aliases are `roles` (many facts share each); the catalog bytes do not move.
- Decodes the write receipt frames of `contracts/cultnet/interop/write-receipt.md`: `parse_message` validates `cultnet.write_admitted.v0` and `cultnet.error.v1` against their closed schemas (unknown keys, empty or non-string ids, an unknown operation or code are refused; `error.v1` requires `messageId`, string or null; `limit_reached` is `error.v1` only). Added `CULTNET_ERROR_V1_CODES`, `CULTNET_WRITE_RECEIPT_MESSAGE_VERSIONS`, `write_content_hash` and `hello_advertises_write_receipts`; `hello` takes `supported_message_versions`. `tests/test_write_receipt.py` replays the vectors' `contentHash`, `wire` and `schemaInvalid` sections. Not implemented here: the client write tracker (`scenarios`) and the server rules (`dedupe`, `blankFields`); this package has no receipt server and no write client. - Decodes the write receipt frames of `contracts/cultnet/interop/write-receipt.md`: `parse_message` validates `cultnet.write_admitted.v0` and `cultnet.error.v1` against their closed schemas (unknown keys, empty or non-string ids, an unknown operation or code are refused; `error.v1` requires `messageId`, string or null; `limit_reached` is `error.v1` only). Added `CULTNET_ERROR_V1_CODES`, `CULTNET_WRITE_RECEIPT_MESSAGE_VERSIONS`, `write_content_hash` and `hello_advertises_write_receipts`; `hello` takes `supported_message_versions`. `tests/test_write_receipt.py` replays the vectors' `contentHash`, `wire` and `schemaInvalid` sections. Not implemented here: the client write tracker (`scenarios`) and the server rules (`dedupe`, `blankFields`); this package has no receipt server and no write client.

View file

@ -174,11 +174,10 @@ def _infer_schema_version_from_payload(payload: bytes) -> str | None:
import msgpack # type: ignore import msgpack # type: ignore
try: try:
decoded = msgpack.unpackb(payload, raw=False) decoded = msgpack.unpackb(payload, raw=False)
except msgpack.ExtraData as extra: except msgpack.ExtraData:
# The hint is read from the first value; the tail is not this function's to judge. The routed definition refuses # A payload with bytes after its value gives no routing hint: the record is unroutable by inference, and a
# it by count when the record is applied, as it does for a record routed by a registered schema id. Treating a # registered schema id routes it to a definition whose decode refuses the tail by count.
# tail as "no hint" made a stamp-routed record look like an unknown schema, which a shard log skips silently. return None
decoded = extra.unpacked
except Exception: except Exception:
return None return None
if isinstance(decoded, list) and decoded and isinstance(decoded[0], str): if isinstance(decoded, list) and decoded and isinstance(decoded[0], str):

View file

@ -1971,7 +1971,7 @@ class CultNetTests(unittest.TestCase):
apply_raw_document_record(cache, schema_document_map([first, second]), raw(first)) # the same type still replaces apply_raw_document_record(cache, schema_document_map([first, second]), raw(first)) # the same type still replaces
self.assertEqual(cache.get_required(first, "shared").name, "remote") self.assertEqual(cache.get_required(first, "shared").name, "remote")
def test_a_raw_record_with_a_tail_is_routed_by_its_stamp_and_refused_by_count_on_every_apply_path(self) -> None: def test_a_raw_record_with_a_tail_is_unroutable_by_inference_and_refused_by_count_when_routed(self) -> None:
import msgpack # type: ignore import msgpack # type: ignore
from cultcache_py import define_document_type from cultcache_py import define_document_type
from cultnet_py import resolve_document_for_raw_record from cultnet_py import resolve_document_for_raw_record
@ -1986,33 +1986,18 @@ class CultNetTests(unittest.TestCase):
"payloadEncoding": "messagepack", "payload": payload} "payloadEncoding": "messagepack", "payload": payload}
documents = schema_document_map([document]) documents = schema_document_map([document])
# The stamp is read from the first value whether or not a tail follows, so an unknown schema id whose payload names # Routing by inference reads a hint, not a payload: a payload with a tail gives no hint, so an unknown schema id
# a registered schema routes to it; the tail is then refused by count where the record is applied. # stays the unknown-schema error (KeyError) even when the first value names a registered schema.
self.assertIs(resolve_document_for_raw_record(documents, "unknown", raw("unknown", head + b"\xc0")), document) self.assertIs(resolve_document_for_raw_record(documents, "unknown", raw("unknown", head)), document)
for schema_id in (known, "unknown"):
cache = CultCache()
cache.register_document_type(document)
with self.assertRaises(ValueError, msg=schema_id) as caught:
apply_raw_document_record(cache, documents, raw(schema_id, head + b"\xc0"))
self.assertIn("1 bytes after its first MessagePack value", str(caught.exception))
self.assertIsNone(cache.get(document, "k"))
snapshot = {"schemaVersion": "cultnet.snapshot_response_raw.v0", "messageId": "s",
"documents": [raw(schema_id, head + b"\xc0")]}
with self.assertRaises(ValueError, msg=schema_id):
apply_raw_snapshot(cache, [document], snapshot)
log = {"schemaVersion": "cultnet.shard_log_response.v0", "messageId": "l", "shardId": "s", "shardEpoch": 1,
"resyncRequired": False,
"entries": [{"sequence": 0, "changeKind": "updated",
"put": {"schemaVersion": "cultnet.document_put_raw.v0", "messageId": "p",
"document": raw(schema_id, head + b"\xc0"), "shardId": "s", "shardEpoch": 1}}]}
with self.assertRaises(ValueError, msg=schema_id) as caught:
apply_shard_log_response(cache, [document], log)
self.assertIn("1 bytes after its first MessagePack value", str(caught.exception))
self.assertIsNone(cache.get(document, "k"))
# A tail on a payload that names no registered schema is still an unknown schema, which a shard log skips.
stranger = msgpack.packb(["tests.stranger.v1", "x"], use_bin_type=True) + b"\xc0"
with self.assertRaises(KeyError): with self.assertRaises(KeyError):
resolve_document_for_raw_record(documents, "unknown", raw("unknown", stranger)) resolve_document_for_raw_record(documents, "unknown", raw("unknown", head + b"\xc0"))
# A registered schema id routes, and the tail is refused by count when the record is applied.
cache = CultCache()
cache.register_document_type(document)
with self.assertRaises(ValueError) as caught:
apply_raw_document_record(cache, documents, raw(known, head + b"\xc0"))
self.assertIn("1 bytes after its first MessagePack value", str(caught.exception))
self.assertIsNone(cache.get(document, "k"))
def test_cultnet_replication_helpers_apply_raw_snapshot_and_shard_log(self) -> None: def test_cultnet_replication_helpers_apply_raw_snapshot_and_shard_log(self) -> None:
document = define_database_entry_type( document = define_database_entry_type(

View file

@ -1 +1,2 @@
/target /target
Cargo.lock

File diff suppressed because it is too large Load diff

View file

@ -30,7 +30,7 @@ knock on the contract before touching the furniture.
## Receipts ## Receipts
```powershell ```powershell
cargo test --locked cargo test
``` ```
The initial tests prove: The initial tests prove:

View file

@ -7,7 +7,7 @@
//! traffic crosses the same schedule. The schedule is `policy.rs`. //! traffic crosses the same schedule. The schedule is `policy.rs`.
//! //!
//! ```text //! ```text
//! cargo run --locked --release --example cultnet_impair -- \ //! cargo run --release --example cultnet_impair -- \
//! --listen 0.0.0.0:17890 --upstream 127.0.0.1:17990 \ //! --listen 0.0.0.0:17890 --upstream 127.0.0.1:17990 \
//! --profile tests/impairment/loss-1pct.toml --seed 424242 --metrics loss-1pct.csv //! --profile tests/impairment/loss-1pct.toml --seed 424242 --metrics loss-1pct.csv
//! ``` //! ```

View file

@ -3,7 +3,7 @@
//! Ignored by default; run in release, where the numbers mean something: //! Ignored by default; run in release, where the numbers mean something:
//! //!
//! ```text //! ```text
//! cargo test --locked --release --test media_fec_bench -- --ignored --nocapture //! cargo test --release --test media_fec_bench -- --ignored --nocapture
//! ``` //! ```
//! //!
//! Each line is nanoseconds per call through the public surface, so record //! Each line is nanoseconds per call through the public surface, so record

View file

@ -3,7 +3,7 @@
All notable changes to `@gamecult/cultnet-ts` are recorded here, following All notable changes to `@gamecult/cultnet-ts` are recorded here, following
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [0.4.0] ## Unreleased
### Breaking ### Breaking

View file

@ -1,6 +1,6 @@
{ {
"name": "@gamecult/cultnet-ts", "name": "@gamecult/cultnet-ts",
"version": "0.4.0", "version": "0.3.0",
"description": "CultNet-style typed direct-pipe messaging and state replication for TypeScript runtimes.", "description": "CultNet-style typed direct-pipe messaging and state replication for TypeScript runtimes.",
"main": "dist/index.js", "main": "dist/index.js",
"types": "dist/index.d.ts", "types": "dist/index.d.ts",

View file

@ -6,7 +6,7 @@ cultnet-rs only, and Idunn's `IdunnRuntimeActivationLaunch` is that same crate's
It needs Linux and a machine-id equal to the `machine-id` file here, because the It needs Linux and a machine-id equal to the `machine-id` file here, because the
provider identity is bound to it. On Yggdrasil: provider identity is bound to it. On Yggdrasil:
DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh <CultLib> HEAD rust 'cd packages/cultnet-rs && cp <fixed-machine-id> /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run --locked -q --example idunn_runtime_fixture /tmp/out' DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh <CultLib> HEAD rust 'cd packages/cultnet-rs && cp <fixed-machine-id> /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run -q --example idunn_runtime_fixture /tmp/out'
then copy `/tmp/out` over this directory. `--cap-add SYS_ADMIN` exists only so the bind mount can present the committed `machine-id`, which keeps Yggdrasil's real machine-id out of the repo. Every `presence-*.bin` was accepted by then copy `/tmp/out` over this directory. `--cap-add SYS_ADMIN` exists only so the bind mount can present the committed `machine-id`, which keeps Yggdrasil's real machine-id out of the repo. Every `presence-*.bin` was accepted by
`authenticate_runtime_presence_claim` and correlated by `authenticate_runtime_presence_claim` and correlated by
@ -26,7 +26,7 @@ The generator takes its specification as input, so a consumer needs no patch to
specification is command-line flags; `@<file>` splices in a response file (one `--flag value` specification is command-line flags; `@<file>` splices in a response file (one `--flag value`
per line, `#` comments). CultLib's own sets are `fixture.args` here: per line, `#` comments). CultLib's own sets are `fixture.args` here:
cargo run --locked -q --example idunn_runtime_fixture /tmp/out @<path>/fixture.args cargo run -q --example idunn_runtime_fixture /tmp/out @<path>/fixture.args
The consumer keeps its own `.args` file in its repository and runs the same command with the The consumer keeps its own `.args` file in its repository and runs the same command with the
same machine-id bind mount as above. Global flags: `--source-repository`, `--detail`. Each same machine-id bind mount as above. Global flags: `--source-repository`, `--detail`. Each

View file

@ -2795,7 +2795,7 @@ async function buildInteropPeers(): Promise<void> {
} }
async function buildRustInteropPeer(): Promise<void> { async function buildRustInteropPeer(): Promise<void> {
rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--locked", "--quiet", "--example", "cultnet_interop_peer"], { rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--quiet", "--example", "cultnet_interop_peer"], {
cwd: cultnetRsRoot, cwd: cultnetRsRoot,
}).then(() => undefined); }).then(() => undefined);
await rustInteropPeerBuild; await rustInteropPeerBuild;

View file

@ -2,7 +2,7 @@
// //
// The vectors below were produced by `cultnet-rs`, not written by hand: // The vectors below were produced by `cultnet-rs`, not written by hand:
// //
// cargo run --locked --example runtime_presence_vector -p cultnet-rs // cargo run --example runtime_presence_vector -p cultnet-rs
// //
// That matters. A fixture authored for both sides at once agrees with itself // That matters. A fixture authored for both sides at once agrees with itself
// and proves nothing; the contract is only real if one fork emits the bytes and // and proves nothing; the contract is only real if one fork emits the bytes and

View file

@ -7,7 +7,7 @@ plus the rules that need no store to state (`Answer::is_training_data`, `Answer:
(comments 682 and 759). Specification only: no collector, tally, export or tombstone hashing lives here. (comments 682 and 759). Specification only: no collector, tally, export or tombstone hashing lives here.
Entry points: `src/lib.rs`; golden vectors `tests/vectors/consent_golden.csv`, run by `tests/golden.rs` Entry points: `src/lib.rs`; golden vectors `tests/vectors/consent_golden.csv`, run by `tests/golden.rs`
(`cargo test --locked`). Planned, not live: the collector, the tally, the training export. (`cargo test`). Planned, not live: the collector, the tally, the training export.
## Where each ruling lands (doc lines at aaeb175) ## Where each ruling lands (doc lines at aaeb175)

View file

@ -11,8 +11,8 @@ adversary, storage, and any projection. Every parameter is a placeholder until t
## First call ## First call
``` ```
cargo run --locked --example first_call # prints the Meta example and a splitting table cargo run --example first_call # prints the Meta example and a splitting table
cargo test --locked cargo test
``` ```
```rust ```rust

View file

@ -857,7 +857,7 @@ test("workflow: each job releases only through release-after-suite.mjs", () => {
}); });
test("release-after-suite checks the tag, then runs the suite, the semver check and the package's tests, and releases only when all passed", async () => { test("release-after-suite checks the tag, then runs the suite, the semver check and the package's tests, and releases only when all passed", async () => {
const { releaseAfterSuite, suite, checker, rangeCheck } = await import("./release-after-suite.mjs"); const { releaseAfterSuite, suite, checker } = await import("./release-after-suite.mjs");
const npmVersion = JSON.parse(readFileSync(join(repoRoot, "packages", "cultcache-ts", "package.json"), "utf8")).version; const npmVersion = JSON.parse(readFileSync(join(repoRoot, "packages", "cultcache-ts", "package.json"), "utf8")).version;
const npmDir = join(repoRoot, "packages", "cultcache-ts"); const npmDir = join(repoRoot, "packages", "cultcache-ts");
const pyDir = join(repoRoot, "packages", "cultnet-py"); const pyDir = join(repoRoot, "packages", "cultnet-py");
@ -881,15 +881,13 @@ test("release-after-suite checks the tag, then runs the suite, the semver check
const npmCalls = [ const npmCalls = [
suiteCall, suiteCall,
checkerCall("@gamecult/cultcache-ts", npmVersion), checkerCall("@gamecult/cultcache-ts", npmVersion),
["npm", ["run", "build:ts"], repoRoot, checked],
[process.execPath, [rangeCheck], repoRoot, checked],
["npm", ["test"], npmDir, checked], ["npm", ["test"], npmDir, checked],
["npm", ["publish", "--access", "public"], npmDir, env], ["npm", ["publish", "--access", "public"], npmDir, env],
]; ];
assert.equal(go(["npm-publish", npmDir, npmTag]), 0); assert.equal(go(["npm-publish", npmDir, npmTag]), 0);
assert.deepEqual(calls, npmCalls); assert.deepEqual(calls, npmCalls);
// any check that fails stops the release at that check // any check that fails stops the release at that check
for (const fail of [0, 1, 2, 3, 4]) { for (const fail of [0, 1, 2]) {
assert.equal(go(["npm-publish", npmDir, npmTag], fail), 1, `check ${fail}`); assert.equal(go(["npm-publish", npmDir, npmTag], fail), 1, `check ${fail}`);
assert.deepEqual(calls, npmCalls.slice(0, fail + 1), `check ${fail}`); assert.deepEqual(calls, npmCalls.slice(0, fail + 1), `check ${fail}`);
} }

View file

@ -1,9 +1,7 @@
// How publish-packages.yml releases a package. From this one process, in order: the release tag must name // How publish-packages.yml releases a package. From this one process, in order: the release tag must name
// the package's declared prefix and its manifest version, the release-checks suite must pass, the semver // the package's declared prefix and its manifest version, the release-checks suite must pass, the semver
// policy check must pass for that version, an npm release's declared dependency ranges must hold (the // policy check must pass for that version, the package's own tests must pass, and only then the release
// workspace is built, then every tagged release a sibling's range admits is checked against what the // action runs. No workflow condition can then release what any of those checks did not pass.
// sibling imports), the package's own tests must pass, and only then the release action runs. No workflow
// condition can then release what any of those checks did not pass.
// docs/semver-policy.md, "The suite gates every release". // docs/semver-policy.md, "The suite gates every release".
// //
// node scripts/release-after-suite.mjs npm-publish <package dir> <git ref> npm publish --access public in <dir> // node scripts/release-after-suite.mjs npm-publish <package dir> <git ref> npm publish --access public in <dir>
@ -20,7 +18,6 @@ import releasePackages from "./release-packages.mjs";
export const suite = fileURLToPath(new URL("./check-changelog-semver.test.mjs", import.meta.url)); export const suite = fileURLToPath(new URL("./check-changelog-semver.test.mjs", import.meta.url));
export const checker = fileURLToPath(new URL("./check-changelog-semver.mjs", import.meta.url)); export const checker = fileURLToPath(new URL("./check-changelog-semver.mjs", import.meta.url));
export const rangeCheck = fileURLToPath(new URL("./test-declared-dependency-ranges.mjs", import.meta.url));
const repoRoot = fileURLToPath(new URL("..", import.meta.url)); const repoRoot = fileURLToPath(new URL("..", import.meta.url));
// The checks run without the variables that can make `node --test` run nothing and exit 0 // The checks run without the variables that can make `node --test` run nothing and exit 0
@ -37,8 +34,6 @@ export const actions = {
"npm-publish": { "npm-publish": {
manifest: (dir) => JSON.parse(readFileSync(join(dir, "package.json"), "utf8")), manifest: (dir) => JSON.parse(readFileSync(join(dir, "package.json"), "utf8")),
test: (dir) => ["npm", ["test"], dir], test: (dir) => ["npm", ["test"], dir],
// The range check builds each tagged release against the workspace's built dists, so it follows a workspace build.
ranges: [["npm", ["run", "build:ts"], repoRoot], [process.execPath, [rangeCheck], repoRoot]],
release: (dir) => ["npm", ["publish", "--access", "public"], dir], release: (dir) => ["npm", ["publish", "--access", "public"], dir],
tagOnly: true, tagOnly: true,
}, },
@ -88,9 +83,6 @@ export function releaseAfterSuite(argv, run = spawn, env = process.env) {
if (isTag && run(process.execPath, [checker, "--package", manifest.name, "--version", manifest.version, "--cwd", repoRoot], undefined, clean) !== 0) { if (isTag && run(process.execPath, [checker, "--package", manifest.name, "--version", manifest.version, "--cwd", repoRoot], undefined, clean) !== 0) {
return refuse(`${manifest.name} ${manifest.version} fails the semver policy (docs/semver-policy.md)`); return refuse(`${manifest.name} ${manifest.version} fails the semver policy (docs/semver-policy.md)`);
} }
for (const [command, args, cwd] of isTag ? action.ranges ?? [] : []) {
if (run(command, args, cwd, clean) !== 0) return refuse(`${manifest.name}'s declared dependency ranges do not hold (scripts/test-declared-dependency-ranges.mjs)`);
}
if (run(...action.test(dir), clean) !== 0) return refuse(`${manifest.name}'s own tests failed`); if (run(...action.test(dir), clean) !== 0) return refuse(`${manifest.name}'s own tests failed`);
return run(...action.release(dir), env); return run(...action.release(dir), env);
} }

View file

@ -22,7 +22,6 @@ import { fileURLToPath } from "node:url";
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const rootRequire = createRequire(join(repoRoot, "package.json")); const rootRequire = createRequire(join(repoRoot, "package.json"));
const ts = rootRequire("typescript"); const ts = rootRequire("typescript");
const npmCli = join(npmRoot(), "bin", "npm-cli.js");
const semver = createRequire(join(npmRoot(), "package.json"))("semver"); const semver = createRequire(join(npmRoot(), "package.json"))("semver");
// npm ships semver, so the check needs no dependency of its own. npm's root is beside the node binary on Windows and under lib/ on Unix. // npm ships semver, so the check needs no dependency of its own. npm's root is beside the node binary on Windows and under lib/ on Unix.
@ -85,17 +84,16 @@ function sourceFiles(directory) {
}); });
} }
// Builds the package as the tag released it (with the repo-level contracts its generator reads), in a scratch directory that borrows the workspace's node_modules. // Builds the package as the tag released it, in a scratch directory that borrows the workspace's node_modules.
function buildRelease(directory, tag, scratch) { function buildRelease(directory, tag, scratch) {
const archive = join(scratch, `${tag}.tar`); const archive = join(scratch, `${tag}.tar`);
git("archive", "--format=tar", "-o", archive, tag, `packages/${directory}`, "contracts"); git("archive", "--format=tar", "-o", archive, tag, `packages/${directory}`);
const extracted = join(scratch, tag); const extracted = join(scratch, tag);
mkdirSync(extracted); mkdirSync(extracted);
execFileSync("tar", ["-xf", archive, "-C", extracted]); execFileSync("tar", ["-xf", archive, "-C", extracted]);
symlinkSync(join(repoRoot, "node_modules"), join(extracted, "node_modules"), "junction"); symlinkSync(join(repoRoot, "node_modules"), join(extracted, "node_modules"), "junction");
const packageRoot = join(extracted, "packages", directory); const packageRoot = join(extracted, "packages", directory);
// The package's own build, which is how a release was made: cultnet-ts generates its contracts module first. execFileSync(process.execPath, [rootRequire.resolve("typescript/bin/tsc"), "-p", "tsconfig.json"], { cwd: packageRoot, stdio: "inherit" });
execFileSync(process.execPath, [npmCli, "run", "build", "--silent"], { cwd: packageRoot, stdio: "inherit" });
return rootRequire(join(packageRoot, "dist", "index.js")); return rootRequire(join(packageRoot, "dist", "index.js"));
} }
@ -129,7 +127,7 @@ try {
const prefix = `${dependency.directory}-v`; const prefix = `${dependency.directory}-v`;
const admitted = tags.filter((tag) => { const admitted = tags.filter((tag) => {
const version = tag.startsWith(prefix) ? tag.slice(prefix.length) : undefined; const version = tag.startsWith(prefix) ? tag.slice(prefix.length) : undefined;
return version !== undefined && semver.valid(version) !== null && semver.satisfies(version, range); return version !== undefined && semver.valid(version) !== null && semver.satisfies(version, range) && version !== dependency.manifest.version;
}); });
for (const tag of admitted) { for (const tag of admitted) {
if (!built.has(tag)) { if (!built.has(tag)) {

View file

@ -1,71 +0,0 @@
// test-declared-dependency-ranges.mjs against a fixture repo: a dependency that gained an export a dependent imports,
// while its manifest still carries the version of its latest release tag, must fail the check, because a consumer's
// range admits that tagged release and it lacks the export. Bumping the version and the range makes it pass.
// The fixture package a builds like cultnet-ts: its build script first generates a module from the repo-level contracts/
// directory, so a tag built by a bare tsc, or archived without contracts/, fails here as it does on the real tree.
import assert from "node:assert/strict";
import { execFileSync, spawnSync } from "node:child_process";
import { copyFileSync, mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import test from "node:test";
import { fileURLToPath } from "node:url";
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const script = join(repoRoot, "scripts", "test-declared-dependency-ranges.mjs");
const write = (root, path, text) => {
mkdirSync(dirname(join(root, path)), { recursive: true });
writeFileSync(join(root, path), text);
};
const git = (root, ...args) => execFileSync("git", ["-c", "user.name=t", "-c", "user.email=t@t", ...args], { cwd: root, stdio: "pipe" });
const tsc = "node ../../node_modules/typescript/bin/tsc -p tsconfig.json";
const manifest = (name, version, extra = {}, build = tsc) =>
JSON.stringify({ name: `@t/${name}`, version, scripts: { build }, ...extra });
const generated = "node tools/generate.mjs && " + tsc;
const tsconfig = JSON.stringify({ compilerOptions: { outDir: "dist", module: "commonjs", target: "es2020", declaration: false }, include: ["src"] });
function fixture() {
const root = mkdtempSync(join(tmpdir(), "declared-ranges-fixture-"));
write(root, "package.json", JSON.stringify({ name: "fixture", private: true, workspaces: ["packages/a", "packages/b"] }));
write(root, "contracts/value.json", "2");
write(root, "packages/a/tools/generate.mjs",
'import { readFileSync, writeFileSync } from "node:fs";\n' +
'writeFileSync("src/data.ts", `export const contractValue = ${readFileSync("../../contracts/value.json", "utf8")};\\n`);\n');
write(root, "packages/a/package.json", manifest("a", "1.0.0", {}, generated));
write(root, "packages/a/tsconfig.json", tsconfig);
write(root, "packages/a/src/index.ts", 'export { contractValue } from "./data";\nexport const foo = 1;\n');
write(root, "packages/b/package.json", manifest("b", "1.0.0", { dependencies: { "@t/a": "^1.0.0" } }));
write(root, "packages/b/tsconfig.json", tsconfig);
write(root, "packages/b/src/index.ts", 'import { foo } from "@t/a";\nexport const b = foo;\n');
mkdirSync(join(root, "scripts"));
copyFileSync(script, join(root, "scripts", "test-declared-dependency-ranges.mjs"));
symlinkSync(join(repoRoot, "node_modules"), join(root, "node_modules"), "junction");
git(root, "init", "-q", "-b", "main");
git(root, "add", "package.json", "contracts", "packages", "scripts");
git(root, "commit", "-q", "-m", "release a 1.0.0");
git(root, "tag", "a-v1.0.0");
return root;
}
const check = (root) => spawnSync(process.execPath, [join(root, "scripts", "test-declared-dependency-ranges.mjs")], { cwd: root, encoding: "utf8" });
test("a dependency that gained an export a dependent imports, still at its latest tag's version, fails the check", () => {
const root = fixture();
try {
assert.equal(check(root).status, 0, "the fixture passes before the export is added");
write(root, "packages/a/src/index.ts", 'export { contractValue } from "./data";\nexport const foo = 1;\nexport const bar = 2;\n');
write(root, "packages/b/src/index.ts", 'import { foo, bar } from "@t/a";\nexport const b = foo + bar;\n');
const stale = check(root);
assert.notEqual(stale.status, 0, stale.stdout);
assert.match(stale.stderr, /a-v1\.0\.0 does not export bar/u);
write(root, "packages/a/package.json", manifest("a", "1.1.0", {}, generated));
write(root, "packages/b/package.json", manifest("b", "1.0.0", { dependencies: { "@t/a": "^1.1.0" } }));
const bumped = check(root);
assert.equal(bumped.status, 0, bumped.stderr);
} finally {
rmSync(root, { recursive: true, force: true });
}
});

View file

@ -237,7 +237,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
}); });
} }
public override void PushAll(bool soft) public override void PushAll()
{ {
ThrowIfReadOnly(); ThrowIfReadOnly();
Held(() => Held(() =>
@ -247,12 +247,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
Directory.CreateDirectory(_manifestFile.DirectoryName!); Directory.CreateDirectory(_manifestFile.DirectoryName!);
using var commitLease = AcquireCommitLease(wait: true, create: true); using var commitLease = AcquireCommitLease(wait: true, create: true);
var manifestBytes = ReadManifestBytes(); var (manifest, catalogBytes) = ReadManifest();
var (manifest, catalogBytes) = ParseManifest(manifestBytes);
// A soft flush writes only where no manifest is: the reader's answer (null is absent), not a second look at the path.
// A manifest it read stays as it is, its staged writes staged.
if (soft && manifestBytes != null)
return;
RefuseUndeclaredOnDisk(manifest, StagedKeys, Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>(), Array.Empty<string>()); RefuseUndeclaredOnDisk(manifest, StagedKeys, Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>(), Array.Empty<string>());
if (IsDirty) if (IsDirty)
WriteGeneration(manifest, catalogBytes); WriteGeneration(manifest, catalogBytes);
@ -686,7 +681,14 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
stream.Flush(true); stream.Flush(true);
} }
StoreReplace.Replace(tempPath, path); if (File.Exists(path))
{
ReplaceExistingFile(tempPath, path);
}
else
{
File.Move(tempPath, path);
}
} }
finally finally
{ {
@ -708,23 +710,17 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
// Only a writer creates the records folder and the lock, so opening the store creates nothing. A reader opens an // Only a writer creates the records folder and the lock, so opening the store creates nothing. A reader opens an
// existing lock and returns null when there is none; that does not mean no writer committed (a store may be copied // existing lock and returns null when there is none; that does not mean no writer committed (a store may be copied
// without its lock, and a first writer may create it mid-load), so an unleased load checks that it settled. // without its lock, and a first writer may create it mid-load), so an unleased load checks that it settled.
// A writer's lease is also where a link at the manifest path is refused (a commit would replace the link with a file): before
// the records folder or the lock is made, and again once the lease is held, because a link can be put there while it waited.
private FileStream? AcquireCommitLease(bool wait, bool create) private FileStream? AcquireCommitLease(bool wait, bool create)
{ {
if (create) if (create)
{
CultStoreLinks.Refuse(_manifestFile.FullName);
Directory.CreateDirectory(_recordDirectory.FullName); Directory.CreateDirectory(_recordDirectory.FullName);
}
var lockPath = Path.Combine(_recordDirectory.FullName, ".commit.lock"); var lockPath = Path.Combine(_recordDirectory.FullName, ".commit.lock");
var started = Stopwatch.StartNew(); var started = Stopwatch.StartNew();
while (true) while (true)
{ {
FileStream held;
try try
{ {
held = new FileStream( return new FileStream(
lockPath, lockPath,
create ? FileMode.OpenOrCreate : FileMode.Open, create ? FileMode.OpenOrCreate : FileMode.Open,
FileAccess.ReadWrite, FileAccess.ReadWrite,
@ -747,23 +743,29 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30)) catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30))
{ {
Thread.Sleep(10); Thread.Sleep(10);
continue;
} }
if (create)
{
try
{
CultStoreLinks.Refuse(_manifestFile.FullName);
}
catch
{
held.Dispose();
throw;
}
}
return held;
} }
} }
private static void ReplaceExistingFile(string sourcePath, string destinationPath)
{
for (var attempt = 0; attempt < 3; attempt++)
{
try
{
File.Replace(sourcePath, destinationPath, null);
return;
}
catch (IOException) when (attempt < 2)
{
Thread.Sleep(10 * (attempt + 1));
}
catch (UnauthorizedAccessException) when (attempt < 2)
{
Thread.Sleep(10 * (attempt + 1));
}
}
File.Replace(sourcePath, destinationPath, null);
}
} }

View file

@ -8,7 +8,7 @@
<ProjectReference Include="..\GameCult.Caching\GameCult.Caching.csproj" /> <ProjectReference Include="..\GameCult.Caching\GameCult.Caching.csproj" />
<ProjectReference Include="..\..\packages\cultmath\src\CultMath\CultMath.csproj" /> <ProjectReference Include="..\..\packages\cultmath\src\CultMath\CultMath.csproj" />
<PackageReference Include="MessagePack" Version="3.1.7" /> <PackageReference Include="MessagePack" Version="3.1.7" />
<PackageReference Include="System.Text.Json" Version="8.0.5" /> <PackageReference Include="System.Text.Json" Version="9.0.10" />
<PackageReference Include="MessagePackAnalyzer" Version="3.1.7" PrivateAssets="all" /> <PackageReference Include="MessagePackAnalyzer" Version="3.1.7" PrivateAssets="all" />
<InternalsVisibleTo Include="GameCult.Caching.Tests" /> <InternalsVisibleTo Include="GameCult.Caching.Tests" />
<InternalsVisibleTo Include="GameCult.Caching.GeneratorConsumer" /> <InternalsVisibleTo Include="GameCult.Caching.GeneratorConsumer" />

View file

@ -2537,24 +2537,19 @@ namespace GameCult.Caching
return Task.CompletedTask; return Task.CompletedTask;
} }
public void FlushAllBackingStores(bool soft = false) => Held(() => public void FlushAllBackingStores() => Held(() =>
{ {
foreach (var (store, _) in _stores) foreach (var (store, _) in _stores)
{ {
if (!store.IsReadOnly && (store.IsDirty || store.HasStagedChecks)) if (!store.IsReadOnly && (store.IsDirty || store.HasStagedChecks))
store.PushAll(soft); store.PushAll();
} }
return true; return true;
}); });
/// <summary> public Task FlushAsync()
/// Flushes every dirty backing store. With <paramref name="soft"/> the flush has one meaning, decided by each store's
/// own read: a store with nothing at its path is written; a store that reads (empty included) is skipped and its
/// staged writes stay staged; a store that cannot be read refuses with <see cref="CultStoreUnreadableException"/>.
/// </summary>
public Task FlushAsync(bool soft = false)
{ {
FlushAllBackingStores(soft); FlushAllBackingStores();
return Task.CompletedTask; return Task.CompletedTask;
} }
@ -2676,8 +2671,7 @@ namespace GameCult.Caching
/// when that store holds it unowned (undeclared drift that drops persisted slots, or a payload carrying slots past this build's, /// when that store holds it unowned (undeclared drift that drops persisted slots, or a payload carrying slots past this build's,
/// see <see cref="CultStoredDocument.UnreadSlots"/>). For a key the cache does not hold, the removal check every store makes of it /// see <see cref="CultStoredDocument.UnreadSlots"/>). For a key the cache does not hold, the removal check every store makes of it
/// (false when a store lists it foreign). One write it does not predict: a Flatten of a variant whose base chain holds an unowned record is refused, though /// (false when a store lists it foreign). One write it does not predict: a Flatten of a variant whose base chain holds an unowned record is refused, though
/// the variant itself is writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it /// the variant itself is writable. Live and advisory: the write still decides.
/// with <see cref="CultStoreLinkedException"/>. Live and advisory: the write still decides.
/// </summary> /// </summary>
public bool IsWritable(CultRecordKey key) public bool IsWritable(CultRecordKey key)
{ {
@ -3951,13 +3945,7 @@ namespace GameCult.Caching
public abstract void PullAll(); public abstract void PullAll();
public abstract void Push(CultStoredDocument entry); public abstract void Push(CultStoredDocument entry);
public abstract void Delete(CultStoredDocument entry); public abstract void Delete(CultStoredDocument entry);
public void PushAll() => PushAll(soft: false); public abstract void PushAll();
/// <summary>
/// Applies the staged writes. A soft flush writes only where nothing is at the path: over a store it reads it returns without
/// writing and leaves the staged writes staged and dirty for a later plain flush, and over a file it cannot read it refuses.
/// </summary>
public abstract void PushAll(bool soft);
/// <summary> /// <summary>
/// One durable step under the store's exclusive lock: the batch and every staged write land onto the store as it is then, /// One durable step under the store's exclusive lock: the batch and every staged write land onto the store as it is then,
@ -4196,7 +4184,7 @@ namespace GameCult.Caching
/// </para> /// </para>
/// <para> /// <para>
/// A writable store kind that reads its durable state under a lock overrides this to call <see cref="StageChecks"/>, and /// A writable store kind that reads its durable state under a lock overrides this to call <see cref="StageChecks"/>, and
/// judges <see cref="StagedChecks"/> at its next <see cref="PushAll(bool)"/> or <see cref="CommitBatch"/> as it judges /// judges <see cref="StagedChecks"/> at its next <see cref="PushAll"/> or <see cref="CommitBatch"/> as it judges
/// <see cref="CultCommitRequest.Checks"/>: under the lock, with the keys it writes, refused through the same refusal, never /// <see cref="CultCommitRequest.Checks"/>: under the lock, with the keys it writes, refused through the same refusal, never
/// removed. It calls <see cref="MarkFlushSucceeded"/> when that step succeeds, which clears them. A read-only store has no /// removed. It calls <see cref="MarkFlushSucceeded"/> when that step succeeds, which clears them. A read-only store has no
/// such step and keeps the default. Both shipped store kinds do this. A batch's own checks are judged through /// such step and keeps the default. Both shipped store kinds do this. A batch's own checks are judged through
@ -4673,7 +4661,7 @@ namespace GameCult.Caching
// only per staged key. A key whose durable record is under an id this cache does not declare refuses the write, and in a store // only per staged key. A key whose durable record is under an id this cache does not declare refuses the write, and in a store
// holding a variant it lands only onto the store as this cache last read it (both ApplyWriteSet). // holding a variant it lands only onto the store as this cache last read it (both ApplyWriteSet).
// A file it cannot read (truncated, trailing bytes, not a store, a format it does not read) is refused and left as it is. // A file it cannot read (truncated, trailing bytes, not a store, a format it does not read) is refused and left as it is.
public override void PushAll(bool soft) public override void PushAll()
{ {
ThrowIfReadOnly(); ThrowIfReadOnly();
Held(() => Held(() =>
@ -4681,8 +4669,6 @@ namespace GameCult.Caching
using (AcquireLock(wait: true)) using (AcquireLock(wait: true))
{ {
var disk = ReadSnapshot(); var disk = ReadSnapshot();
if (soft && disk != null)
return;
var (upserts, removals) = StagedWrites(Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>()); var (upserts, removals) = StagedWrites(Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>());
Wrote(ApplyWriteSet(disk, upserts, removals, Array.Empty<string>())); Wrote(ApplyWriteSet(disk, upserts, removals, Array.Empty<string>()));
} }
@ -5146,7 +5132,7 @@ namespace GameCult.Caching
var attributes = FileInfo.Attributes; var attributes = FileInfo.Attributes;
if ((int)attributes == -1) if ((int)attributes == -1)
return null; return null;
if ((attributes & FileAttributes.Directory) != 0 && Directory.Exists(FileInfo.FullName)) if ((attributes & FileAttributes.Directory) != 0 && (attributes & FileAttributes.ReparsePoint) == 0)
throw new IOException("The store path is a directory, not a store file."); throw new IOException("The store path is a directory, not a store file.");
try try
{ {
@ -5168,27 +5154,25 @@ namespace GameCult.Caching
} }
// Something is at the path even when the path cannot be opened: a link keeps the reparse-point attribute whether or not its // Something is at the path even when the path cannot be opened: a link keeps the reparse-point attribute whether or not its
// target is there. // target is there. A path with nothing at it reports no attributes at all (-1).
private bool PathIsALink() => CultStoreLinks.IsLink(FileInfo.FullName); private bool PathIsALink()
{
// A write never goes through a link at the store path: replacing it would silently detach the store from its link. FileInfo.Refresh();
private void RefuseLinkedStore() => CultStoreLinks.Refuse(FileInfo.FullName); var attributes = FileInfo.Attributes;
return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
}
// The lock is a sidecar opened exclusively, which excludes other handles in this process and in others alike. // The lock is a sidecar opened exclusively, which excludes other handles in this process and in others alike.
// Every write takes the lock first, so the link is refused here twice: before the lock file or anything else is made, and
// again once the lock is held, because a link can be put at the path while this writer waited on another's lock.
private FileStream? AcquireLock(bool wait) private FileStream? AcquireLock(bool wait)
{ {
RefuseLinkedStore();
Directory.CreateDirectory(FileInfo.DirectoryName!); Directory.CreateDirectory(FileInfo.DirectoryName!);
var lockPath = FileInfo.FullName + ".lock"; var lockPath = FileInfo.FullName + ".lock";
var started = Stopwatch.StartNew(); var started = Stopwatch.StartNew();
while (true) while (true)
{ {
FileStream held;
try try
{ {
held = new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, bufferSize: 1); return new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, bufferSize: 1);
} }
catch (IOException) when (!wait) catch (IOException) when (!wait)
{ {
@ -5197,20 +5181,7 @@ namespace GameCult.Caching
catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30)) catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30))
{ {
Thread.Sleep(10); Thread.Sleep(10);
continue;
} }
try
{
RefuseLinkedStore();
}
catch
{
held.Dispose();
throw;
}
return held;
} }
} }
@ -5233,7 +5204,14 @@ namespace GameCult.Caching
stream.Flush(true); stream.Flush(true);
} }
StoreReplace.Replace(tempPath, path); if (File.Exists(path))
{
File.Replace(tempPath, path, null);
}
else
{
File.Move(tempPath, path);
}
} }
finally finally
{ {

View file

@ -1,24 +0,0 @@
using System.IO;
namespace GameCult.Caching
{
/// <summary>
/// A write was attempted at a store path that is a link (a symbolic link to a file or a directory, a Windows junction or any
/// other reparse-point link). The write is refused before the lock, the staging file or the replace, and again once the lock
/// is held, so a link that exists when a writer takes the lock leaves the link and its target untouched. A link created by
/// something that ignores the lock after that check, while the commit builds and writes the new image, is still replaced;
/// that window cannot be closed without native no-follow opens. Configure the link's target as the store path to write to
/// it; reads through a live link are unchanged.
/// </summary>
public sealed class CultStoreLinkedException : IOException
{
public CultStoreLinkedException(string path)
: base($"CultCache store {path} is a link, and a write at it would replace the link; configure the link's target as the store path.")
{
Path = path;
}
/// <summary>The store path that is a link.</summary>
public string Path { get; }
}
}

View file

@ -1,22 +0,0 @@
using System.IO;
namespace GameCult.Caching
{
// The one decision on whether a store-owned file is a link, shared by every store kind that replaces files.
internal static class CultStoreLinks
{
// A path with nothing at it reports no attributes at all (-1). Any reparse point counts as a link, whether or not its
// target is there.
internal static bool IsLink(string path)
{
var attributes = new FileInfo(path).Attributes;
return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
}
internal static void Refuse(string path)
{
if (IsLink(path))
throw new CultStoreLinkedException(path);
}
}
}

View file

@ -1,182 +0,0 @@
using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
namespace GameCult.Caching
{
/// <summary>The kernel32 calls a Windows replace needs, each answering 0 or a Win32 error code.</summary>
internal interface IReplaceKernel
{
/// <summary>CreateFileW(staged, DELETE|SYNCHRONIZE), FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, CloseHandle.</summary>
int RenamePosix(string staged, string destination);
/// <summary>MoveFileExW(staged, destination, flags).</summary>
int MoveFileEx(string staged, string destination, uint flags);
void Sleep(int milliseconds);
}
/// <summary>
/// Owns how a staged file becomes the store, for the single-file and directory stores. Windows tries a POSIX-semantics
/// rename first (which makes a moment with the path absent rare for a lock-free reader, not impossible: a stalled rename was
/// measured leaving it absent for 5-47 ms), falls back to write-through MoveFileExW where that is unsupported or does not
/// resolve, falls back to the portable File.Replace if MoveFileExW itself does not resolve, and retries transient sharing
/// errors; other OSes keep rename. Mirrors cultcache-ts store-file.ts
/// replaceStoreFile (ruling variants:ruling:windows-replace-posix-first).
/// </summary>
internal static class StoreReplace
{
private const int ErrorAccessDenied = 5;
private const int ErrorSharingViolation = 32;
private const int ErrorLockViolation = 33;
private const int ErrorNotSupported = 50;
private const int ErrorInvalidParameter = 87;
private const int ReplaceAttempts = 25;
private const int ReplaceRetryMs = 20;
/// <summary>MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH.</summary>
internal const uint MoveFileReplaceWriteThrough = 0x9;
internal static readonly IReplaceKernel WindowsKernel = new Kernel32Replace();
public static void Replace(string staged, string destination)
{
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
ReplaceWindows(staged, destination, WindowsKernel);
}
else
{
ReplacePortable(staged, destination);
}
}
private static void ReplacePortable(string staged, string destination)
{
if (File.Exists(destination))
{
File.Replace(staged, destination, null);
}
else
{
File.Move(staged, destination);
}
}
private static bool Unresolved(Exception ex) => ex is DllNotFoundException || ex is EntryPointNotFoundException;
internal static void ReplaceWindows(string staged, string destination, IReplaceKernel kernel)
{
var posix = true;
var lastCode = 0;
for (var attempt = 1; attempt <= ReplaceAttempts; attempt++)
{
int code;
try
{
code = posix ? RenamePosixOrUnsupported(kernel, staged, destination) : kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough);
if (posix && (code == 1 || code == ErrorNotSupported || code == ErrorInvalidParameter))
{
// POSIX semantics are unsupported here (FAT, exFAT, before Windows 10 1607): the rest of this call is write-through.
posix = false;
code = kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough);
}
}
catch (Exception ex) when (Unresolved(ex))
{
// kernel32's MoveFileExW does not resolve here (a sandboxed player): the runtime-owned replace is all that is left.
ReplacePortable(staged, destination);
return;
}
if (code == 0)
{
return;
}
if (code != ErrorAccessDenied && code != ErrorSharingViolation && code != ErrorLockViolation)
{
throw new IOException($"CultCache could not replace the store: Win32 error {code}.");
}
lastCode = code;
kernel.Sleep(ReplaceRetryMs);
}
throw new IOException($"CultCache could not replace the store after {ReplaceAttempts} attempts: last Win32 error {lastCode}.");
}
private static int RenamePosixOrUnsupported(IReplaceKernel kernel, string staged, string destination)
{
try
{
return kernel.RenamePosix(staged, destination);
}
catch (Exception ex) when (Unresolved(ex))
{
// The POSIX call does not resolve: the same answer as "unsupported", so the replace falls back to MoveFileExW.
return ErrorNotSupported;
}
}
private sealed class Kernel32Replace : IReplaceKernel
{
private const uint DeleteSynchronize = 0x00110000;
private const uint ShareAll = 0x7;
private const uint OpenExisting = 3;
private const uint FileAttributeNormal = 0x80;
private const int FileRenameInfoEx = 22;
private const uint RenameReplacePosix = 0x3;
public int RenamePosix(string staged, string destination)
{
var handle = CreateFileW(staged, DeleteSynchronize, ShareAll, IntPtr.Zero, OpenExisting, FileAttributeNormal, IntPtr.Zero);
if (handle == new IntPtr(-1))
{
return Marshal.GetLastWin32Error();
}
try
{
// FILE_RENAME_INFO: Flags u32 at 0, RootDirectory handle, FileNameLength u32, FileName (UTF-16, offsets for this pointer size).
var name = System.Text.Encoding.Unicode.GetBytes(Path.GetFullPath(destination));
var lengthOffset = 2 * IntPtr.Size;
var nameOffset = lengthOffset + 4;
var size = nameOffset + name.Length + 2;
var info = Marshal.AllocHGlobal(size);
try
{
for (var i = 0; i < size; i++) Marshal.WriteByte(info, i, 0);
Marshal.WriteInt32(info, 0, (int)RenameReplacePosix);
Marshal.WriteInt32(info, lengthOffset, name.Length);
Marshal.Copy(name, 0, info + nameOffset, name.Length);
return SetFileInformationByHandle(handle, FileRenameInfoEx, info, (uint)size) ? 0 : Marshal.GetLastWin32Error();
}
finally { Marshal.FreeHGlobal(info); }
}
finally { CloseHandle(handle); }
}
public int MoveFileEx(string staged, string destination, uint flags)
=> MoveFileExW(staged, destination, flags) ? 0 : Marshal.GetLastWin32Error();
public void Sleep(int milliseconds) => Thread.Sleep(milliseconds);
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern IntPtr CreateFileW(string fileName, uint access, uint share, IntPtr security, uint disposition, uint flags, IntPtr template);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SetFileInformationByHandle(IntPtr handle, int informationClass, IntPtr information, uint size);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool CloseHandle(IntPtr handle);
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool MoveFileExW(string existing, string target, uint flags);
}
}
}

View file

@ -12,7 +12,7 @@
<ProjectReference Include="..\GameCult.Caching\GameCult.Caching.csproj" /> <ProjectReference Include="..\GameCult.Caching\GameCult.Caching.csproj" />
<ProjectReference Include="..\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" /> <ProjectReference Include="..\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<PackageReference Include="MessagePack" Version="3.1.7" /> <PackageReference Include="MessagePack" Version="3.1.7" />
<PackageReference Include="System.Text.Json" Version="8.0.5" /> <PackageReference Include="System.Text.Json" Version="9.0.10" />
</ItemGroup> </ItemGroup>
<Import Project="..\GameCult.Caching.MessagePack\buildTransitive\GameCult.Caching.MessagePack.targets" /> <Import Project="..\GameCult.Caching.MessagePack\buildTransitive\GameCult.Caching.MessagePack.targets" />

View file

@ -428,14 +428,11 @@ namespace GameCult.Mesh
} }
/// <summary> /// <summary>
/// Flushes the durable cache. With <paramref name="soft"/> the flush has one meaning, decided by the store's own /// Flushes the durable cache.
/// read and never by what the cache holds in memory: a store with nothing at its path is written; a store that reads
/// (empty included) is skipped and its staged writes stay staged; a store that cannot be read refuses with
/// <see cref="CultStoreUnreadableException"/> and is left untouched.
/// </summary> /// </summary>
public Task FlushAsync(bool soft = false) public Task FlushAsync(bool soft = false)
{ {
return _host.FlushAsync(soft); return _host.FlushAsync();
} }
/// <summary> /// <summary>

View file

@ -96,14 +96,11 @@ namespace GameCult.Networking
} }
/// <summary> /// <summary>
/// Flushes the durable cache. With <paramref name="soft"/> the flush has one meaning, decided by the store's own /// Flushes the durable cache.
/// read and never by what the cache holds in memory: a store with nothing at its path is written; a store that reads
/// (empty included) is skipped and its staged writes stay staged; a store that cannot be read refuses with
/// <see cref="CultStoreUnreadableException"/> and is left untouched.
/// </summary> /// </summary>
public Task FlushAsync(bool soft = false) public Task FlushAsync(bool soft = false)
{ {
return Cache.FlushAsync(soft); return Cache.FlushAsync();
} }
/// <summary> /// <summary>

View file

@ -9,7 +9,7 @@
<ProjectReference Include="..\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" /> <ProjectReference Include="..\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<PackageReference Include="MessagePack" Version="3.1.7" /> <PackageReference Include="MessagePack" Version="3.1.7" />
<PackageReference Include="MessagePackAnalyzer" Version="3.1.7" PrivateAssets="all" /> <PackageReference Include="MessagePackAnalyzer" Version="3.1.7" PrivateAssets="all" />
<PackageReference Include="System.Text.Json" Version="8.0.5" /> <PackageReference Include="System.Text.Json" Version="9.0.10" />
<PackageReference Include="R3" Version="1.3.0" /> <PackageReference Include="R3" Version="1.3.0" />
<PackageReference Include="LiteNetLib" Version="2.1.2" /> <PackageReference Include="LiteNetLib" Version="2.1.2" />
<PackageReference Include="Isopoh.Cryptography.Argon2" Version="2.0.0" /> <PackageReference Include="Isopoh.Cryptography.Argon2" Version="2.0.0" />

View file

@ -56,7 +56,7 @@ public sealed class OutsideStore : CacheBackingStore
public override void Delete(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { }
public override void PushAll(bool soft) { } public override void PushAll() { }
public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) =>
request.ConditionsHold(new Dictionary<string, string> { ["w"] = "v1" }, Entries.Values, ForeignRecords) request.ConditionsHold(new Dictionary<string, string> { ["w"] = "v1" }, Entries.Values, ForeignRecords)

View file

@ -51,12 +51,6 @@ static async Task<int> ProgramMainAsync(string[] args)
case "edit-in-place": case "edit-in-place":
await EditInPlaceAsync(file, RequireArg(options, "kind")); await EditInPlaceAsync(file, RequireArg(options, "kind"));
return 0; return 0;
case "write-shapes":
await WriteShapesAsync(file);
return 0;
case "read-shapes":
await ReadShapesAsync(file);
return 0;
case "hold-lock": case "hold-lock":
HoldLock(file, int.Parse(RequireArg(options, "ms"), CultureInfo.InvariantCulture)); HoldLock(file, int.Parse(RequireArg(options, "ms"), CultureInfo.InvariantCulture));
return 0; return 0;
@ -102,49 +96,6 @@ static async Task WriteAsync(string file, string runtimeId)
WriteJsonLine(note); WriteJsonLine(note);
} }
// Records a reader of a Bifrost-shaped store must open: a union field (two arms) and nested bytes (a bin member, a bin list).
static async Task WriteShapesAsync(string file)
{
var cache = BuildCache(file);
await cache.PullAllBackingStoresAsync();
foreach (var shape in InteropShapes())
await cache.AddAsync(shape, new CultRecordHandle<CultCacheInteropShape>(new CultRecordKey(shape.DocumentId)));
cache.FlushAllBackingStores();
WriteShapesJsonLine(InteropShapes());
}
static async Task ReadShapesAsync(string file)
{
var cache = BuildCache(file);
await cache.PullAllBackingStoresAsync();
WriteShapesJsonLine(cache.AllEntries.OfType<CultCacheInteropShape>());
}
static CultCacheInteropShape[] InteropShapes() =>
[
new() { DocumentId = "shape:text", Arm = new InteropTextArm { Text = "union arm 0" }, Raw = Enumerable.Range(0, 256).Select(i => (byte)i).ToArray() },
new()
{
DocumentId = "shape:pack",
Arm = new InteropPackArm { Label = "union arm 1", Chunks = [new byte[] { 0xc0, 0xc1, 0x00 }, [], [0x92, 0x01, 0x02]] },
Raw = [0xde, 0xad, 0xbe, 0xef]
}
];
// One line, ordered by document id, in the form the Rust peer prints.
static void WriteShapesJsonLine(IEnumerable<CultCacheInteropShape> shapes) =>
Console.Out.WriteLine(JsonSerializer.Serialize(shapes.OrderBy(shape => shape.DocumentId, StringComparer.Ordinal).Select(shape => new
{
documentId = shape.DocumentId,
arm = shape.Arm switch
{
InteropTextArm text => (object)new { kind = "text", text = text.Text },
InteropPackArm pack => new { kind = "pack", label = pack.Label, chunks = pack.Chunks.Select(Convert.ToHexString).ToArray() },
_ => throw new InvalidOperationException("Unknown arm.")
},
raw = Convert.ToHexString(shape.Raw)
})));
static async Task ReadAsync(string file) static async Task ReadAsync(string file)
{ {
var cache = BuildCache(file); var cache = BuildCache(file);
@ -377,28 +328,3 @@ public sealed class CultCacheInteropMark
[Key(0)] public string Label { get; set; } = string.Empty; [Key(0)] public string Label { get; set; } = string.Empty;
[Key(1)] [CultElementId] public string Id { get; set; } = string.Empty; [Key(1)] [CultElementId] public string Id { get; set; } = string.Empty;
} }
[CultDocument("cultcache.interop-shape", "cultcache.interop_shape.v1")]
[MessagePackObject]
public sealed class CultCacheInteropShape
{
[Key(0)] [CultName] public string DocumentId { get; set; } = string.Empty;
[Key(1)] public InteropArm Arm { get; set; } = new InteropTextArm();
[Key(2)] public byte[] Raw { get; set; } = Array.Empty<byte>();
}
[Union(0, typeof(InteropTextArm)), Union(1, typeof(InteropPackArm))]
public abstract class InteropArm { }
[MessagePackObject]
public sealed class InteropTextArm : InteropArm
{
[Key(0)] public string Text { get; set; } = string.Empty;
}
[MessagePackObject]
public sealed class InteropPackArm : InteropArm
{
[Key(0)] public string Label { get; set; } = string.Empty;
[Key(1)] public List<byte[]> Chunks { get; set; } = new();
}

View file

@ -136,7 +136,7 @@ namespace GameCult.Caching.OutsideStoreTests
IsDirty = true; IsDirty = true;
}); });
public override void PushAll(bool soft) => Held(() => Apply(Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>(), Array.Empty<string>())); public override void PushAll() => Held(() => Apply(Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>(), Array.Empty<string>()));
public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => Held(() => public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => Held(() =>
{ {

View file

@ -56,137 +56,6 @@ namespace GameCult.Caching.Tests
} }
} }
[Test]
public async Task SoftFlush_Over_Readable_Store_Leaves_It_Byte_For_Byte_And_Keeps_Staged_Writes_Dirty()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack");
try
{
var store = new SingleFileMessagePackBackingStore(filePath);
var cache = new CultCache();
cache.AddBackingStore(store);
await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" });
await cache.FlushAsync();
var before = File.ReadAllBytes(filePath);
var staged = await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" });
await cache.FlushAsync(soft: true);
Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft flush rewrote a store it reads");
Assert.That(store.IsDirty, Is.True, "a skipped soft flush discarded a staged write");
await cache.FlushAsync();
var reader = new CultCache();
reader.AddBackingStore(new SingleFileMessagePackBackingStore(filePath));
await reader.PullAllBackingStoresAsync();
Assert.That(reader.Get<NamedTestEntry>(staged.Key)!.Value, Is.EqualTo("two"));
}
finally
{
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task SoftFlush_Over_Readable_Store_Holding_No_Records_Leaves_It()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack");
try
{
var store = new SingleFileMessagePackBackingStore(filePath);
var cache = new CultCache();
cache.AddBackingStore(store);
var first = await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" });
await cache.FlushAsync();
cache.Remove(first.Key);
await cache.FlushAsync();
var before = File.ReadAllBytes(filePath);
var empty = new CultCache();
empty.AddBackingStore(new SingleFileMessagePackBackingStore(filePath));
Assert.That(empty.AllEntries, Is.Empty, "the store is not readable-empty");
await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" });
await cache.FlushAsync(soft: true);
Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft flush wrote over a readable store that holds no records");
Assert.That(store.IsDirty, Is.True);
}
finally
{
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task PlainPushAll_Over_Readable_Store_Writes_The_Staged_Records()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack");
try
{
var store = new SingleFileMessagePackBackingStore(filePath);
var cache = new CultCache();
cache.AddBackingStore(store);
await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" });
store.PushAll();
var before = File.ReadAllBytes(filePath);
await cache.AddAsync(new NamedTestEntry { Name = "second", Value = "two" });
store.PushAll();
Assert.That(File.ReadAllBytes(filePath), Is.Not.EqualTo(before), "a plain PushAll skipped a store it reads");
Assert.That(store.IsDirty, Is.False);
}
finally
{
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task SoftFlush_Over_Absent_Path_Writes()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack");
try
{
var store = new SingleFileMessagePackBackingStore(filePath);
var cache = new CultCache();
cache.AddBackingStore(store);
await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" });
await cache.FlushAsync(soft: true);
Assert.That(File.Exists(filePath), Is.True);
Assert.That(store.IsDirty, Is.False);
}
finally
{
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task SoftFlush_Over_Unreadable_File_Refuses_And_Leaves_It()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultlib-tests-{Guid.NewGuid():N}.msgpack");
try
{
var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 };
var store = new SingleFileMessagePackBackingStore(filePath);
var cache = new CultCache();
cache.AddBackingStore(store);
await cache.AddAsync(new NamedTestEntry { Name = "first", Value = "one" });
File.WriteAllBytes(filePath, junk);
Assert.That(() => store.PushAll(soft: true), Throws.InstanceOf<CultStoreUnreadableException>());
Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(junk));
Assert.That(store.IsDirty, Is.True);
}
finally
{
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test] [Test]
public async Task SingleFileMessagePackBackingStore_PullAll_SeesFileCreatedAfterReaderOpened() public async Task SingleFileMessagePackBackingStore_PullAll_SeesFileCreatedAfterReaderOpened()
{ {

View file

@ -110,15 +110,6 @@ namespace GameCult.Caching.Tests
Assert.That(message, Does.Contain(typeof(UnionSurface).FullName).And.Contain($"1 {Clause}").And.Contain("refused on read")); Assert.That(message, Does.Contain(typeof(UnionSurface).FullName).And.Contain($"1 {Clause}").And.Contain("refused on read"));
} }
// The inspector reads documents through CultCodec.Deserialize (type == document), not DecodeDocument: its branch is its own path.
[Test]
public void CodecDeserializeOfADocumentRefusesATail()
{
var codec = CultCacheMessagePack.CreateCodec(Registry);
var message = Message(() => codec.Deserialize(typeof(UnionSurface), typeof(UnionSurface), Base.Concat(new byte[] { 0xc0 }).ToArray()));
Assert.That(message, Does.Contain(typeof(UnionSurface).FullName).And.Contain($"1 {Clause}").And.Contain("refused on read"));
}
[Test] [Test]
public void StoreLoadRefuses() public void StoreLoadRefuses()
{ {

View file

@ -338,7 +338,7 @@ namespace GameCult.Caching.Tests
public override void Push(CultStoredDocument entry) { } public override void Push(CultStoredDocument entry) { }
public override void Delete(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { }
public override void PushAll(bool soft) { } public override void PushAll() { }
public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed;
} }

View file

@ -180,7 +180,7 @@ namespace GameCult.Caching.Tests
public override void Push(CultStoredDocument entry) { } public override void Push(CultStoredDocument entry) { }
public override void Delete(CultStoredDocument entry) { } public override void Delete(CultStoredDocument entry) { }
public override void PushAll(bool soft) { } public override void PushAll() { }
public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed; public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed;
} }

View file

@ -208,72 +208,6 @@ namespace GameCult.Caching.Tests
Assert.That(reader.AllEntries, Is.Empty); Assert.That(reader.AllEntries, Is.Empty);
} }
[Test]
public async Task SoftFlushOverAbsentManifestWrites()
{
using var cache = Open(out var store);
var key = await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" });
cache.FlushAllBackingStores(soft: true);
Assert.That(File.Exists(_manifest), Is.True);
Assert.That(store.IsDirty, Is.False);
using var reader = Open(out _);
Assert.That(reader.Get<Page>(key)?.Text, Is.EqualTo("one"));
}
[Test]
public async Task SoftFlushOverReadableManifestLeavesItAndKeepsStagedWritesDirty()
{
using var cache = Open(out var store);
await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" });
cache.FlushAllBackingStores();
var before = File.ReadAllBytes(_manifest);
var pages = Directory.GetFiles(_records, "*.msgpack").OrderBy(path => path).ToArray();
var second = await cache.UpsertAsync(typeof(Page), new Page { Name = "b", Text = "two" });
cache.FlushAllBackingStores(soft: true);
Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(before), "a soft flush rewrote a manifest it reads");
Assert.That(Directory.GetFiles(_records, "*.msgpack").OrderBy(path => path).ToArray(), Is.EqualTo(pages));
Assert.That(store.IsDirty, Is.True, "a skipped soft flush discarded a staged write");
cache.FlushAllBackingStores();
using var reader = Open(out _);
Assert.That(reader.Get<Page>(second)?.Text, Is.EqualTo("two"));
}
[Test]
public async Task SoftFlushOverReadableEmptyManifestLeavesIt()
{
using var cache = Open(out var store);
var key = await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" });
cache.FlushAllBackingStores();
cache.Remove(key);
cache.FlushAllBackingStores();
var before = File.ReadAllBytes(_manifest);
using (var reader = Open(out _)) Assert.That(reader.AllEntries, Is.Empty, "the store is not readable-empty");
await cache.UpsertAsync(typeof(Page), new Page { Name = "b", Text = "two" });
cache.FlushAllBackingStores(soft: true);
Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(before), "a soft flush wrote over a readable store that holds no records");
Assert.That(store.IsDirty, Is.True);
}
[Test]
public async Task SoftFlushOverUnreadableManifestRefusesAndLeavesIt()
{
using var cache = Open(out var store);
await cache.UpsertAsync(typeof(Page), new Page { Name = "a", Text = "one" });
var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 };
File.WriteAllBytes(_manifest, junk);
Assert.That(() => store.PushAll(soft: true), Throws.Exception);
Assert.That(File.ReadAllBytes(_manifest), Is.EqualTo(junk));
Assert.That(store.IsDirty, Is.True);
}
private CultCache Open(out DirectoryMessagePackBackingStore store) private CultCache Open(out DirectoryMessagePackBackingStore store)
{ {
var cache = new CultCache(Registry); var cache = new CultCache(Registry);

View file

@ -1,151 +0,0 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using GameCult.Caching;
using MessagePack;
using NUnit.Framework;
namespace GameCult.Caching.Tests
{
// The id walk's own contracts, one mutant-sized fact at a time: the order of the id path, a union that names itself, a list of
// values never counting as an id, and the attribute predicates that decide whether a type is walked or written whole.
public class ElementIdWalkTests
{
// The codec asks MessagePack about the id path in order and refuses at the first type it cannot allow, so the order is what
// an error names first: the document, then the collection types from the outermost in, then the element and its arms.
[Test]
public void TheIdPathListsCollectionsFromTheOutermostIn()
{
Assert.That(CultElementIds.Inspect(typeof(MsMapMap)).IdPath, Is.EqualTo(new[]
{
typeof(MsMapMap), typeof(Dictionary<string, Dictionary<string, List<ElementIdEngineTests.EIded>>>),
typeof(Dictionary<string, List<ElementIdEngineTests.EIded>>), typeof(List<ElementIdEngineTests.EIded>), typeof(ElementIdEngineTests.EIded)
}), "maps nested in maps");
Assert.That(CultElementIds.Inspect(typeof(MsListDeep)).IdPath, Is.EqualTo(new[]
{
typeof(MsListDeep), typeof(List<List<List<ElementIdEngineTests.EIded>>>), typeof(List<List<ElementIdEngineTests.EIded>>),
typeof(List<ElementIdEngineTests.EIded>), typeof(ElementIdEngineTests.EIded)
}), "lists nested in lists");
}
// ConcreteTypes stops at a type it has already expanded, so a union that lists its own base is finished, not a stack overflow.
[Test]
public void AUnionThatNamesItselfIsWalkedOnce()
{
var survey = CultElementIds.Inspect(typeof(MsSelfHolder));
Assert.That(survey.Problems, Is.Empty);
Assert.That(survey.IdPath, Is.EquivalentTo(new[] { typeof(MsSelfHolder), typeof(List<MsSelfBase>), typeof(MsSelfBase), typeof(MsSelfArm) }));
}
// Value lists hold no ids however they are nested, and a root that is itself a value list holds none either.
[Test]
public void ListsOfValuesHoldNoIds()
{
var values = new MsValues { Numbers = { 1 }, Tags = { "a" }, Grid = { new List<int> { 1 } }, Map = { ["k"] = new List<int> { 2 } } };
Assert.That(CultElementIds.Holds(typeof(MsValues), values), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<int>), new List<int> { 1, 2 }), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<List<int>>), new List<List<int>> { new() { 1 } }), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<string>), new List<string> { "a" }), Is.False);
Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated");
}
// A nullable struct is walked as the struct it wraps; a nullable value is not walked at all.
[Test]
public void ANullableKeyedStructWalksAsItsOwner()
{
Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct)), Is.True);
Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct?)), Is.True);
Assert.That(CultElementIds.WalksAsOwner(typeof(int?)), Is.False);
Assert.That(CultElementIds.WalksAsOwner(typeof(int)), Is.False);
}
// [Key] is read through an override, so an element whose id property overrides a keyed base member is still an id with a key.
[Test]
public void AKeyOnTheBaseOfAnOverrideCountsForTheOverride()
{
Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).Problems, Is.Empty);
Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).IdPath, Does.Contain(typeof(MsOverrideEl)));
}
// A type is walked by key when it is [MessagePackObject] or [Union], read through its bases for the former; it is one whole
// value when it declares its own formatter, is [DataContract] without being [MessagePackObject], or is keyed by name.
[Test]
public void KeyedTypesWalkAndWholeTypesDoNot()
{
Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpBase)), Is.True, "[MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsUBase)), Is.True, "[Union]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpDerived)), Is.True, "[MessagePackObject] inherited");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsPlain)), Is.False, "[Key] members without [MessagePackObject] or [Union]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsUArm)), Is.True, "an arm that is itself [MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsDc)), Is.False, "[DataContract] alone");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcMp)), Is.True, "[DataContract] beside [MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcOverMpBase)), Is.True, "[DataContract] over an inherited [MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropName)), Is.False, "KeyAsPropertyName");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropNameDerived)), Is.False, "KeyAsPropertyName inherited");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsStringKey)), Is.False, "a string [Key]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedBase)), Is.False, "[MessagePackFormatter]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedDerived)), Is.False, "[MessagePackFormatter] inherited");
}
[MessagePackObject] public class MsMapMap { [Key(0)] public Dictionary<string, Dictionary<string, List<ElementIdEngineTests.EIded>>> M { get; set; } = new(); }
[MessagePackObject] public class MsListDeep { [Key(0)] public List<List<List<ElementIdEngineTests.EIded>>> G { get; set; } = new(); }
[Union(0, typeof(MsSelfBase)), Union(1, typeof(MsSelfArm))] public abstract class MsSelfBase { }
[MessagePackObject] public class MsSelfArm : MsSelfBase { [Key(0)] [CultElementId] public string Id { get; set; } = ""; }
[MessagePackObject] public class MsSelfHolder { [Key(0)] public List<MsSelfBase> Items { get; set; } = new(); }
[MessagePackObject]
public class MsValues
{
[Key(0)] public List<int> Numbers { get; set; } = new();
[Key(1)] public List<string> Tags { get; set; } = new();
[Key(2)] public List<List<int>> Grid { get; set; } = new();
[Key(3)] public Dictionary<string, List<int>> Map { get; set; } = new();
}
public sealed class MsBoom : IEnumerable<int>
{
public IEnumerator<int> GetEnumerator() => throw new InvalidOperationException("enumerated");
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator();
}
[MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; }
[MessagePackObject]
public class MsOverrideBase
{
[Key(0)] [CultElementId] public virtual string Id { get; set; } = "";
}
[MessagePackObject]
public class MsOverrideEl : MsOverrideBase
{
public override string Id { get; set; } = "";
}
[MessagePackObject] public class MsOverrideHolder { [Key(0)] public List<MsOverrideEl> Items { get; set; } = new(); }
[MessagePackObject] public class MsMpBase { [Key(0)] public int A { get; set; } }
public class MsMpDerived : MsMpBase { }
public class MsPlain { [Key(0)] public int A { get; set; } }
[Union(0, typeof(MsUArm))] public abstract class MsUBase { }
[MessagePackObject] public class MsUArm : MsUBase { [Key(0)] public int A { get; set; } }
[DataContract] public class MsDc { [DataMember] public int A { get; set; } }
[DataContract] [MessagePackObject] public class MsDcMp { [Key(0)] public int A { get; set; } }
[DataContract] public class MsDcOverMpBase : MsMpBase { }
[MessagePackObject(keyAsPropertyName: true)] public class MsPropName { public int A { get; set; } }
public class MsPropNameDerived : MsPropName { }
[MessagePackObject] public class MsStringKey { [Key("a")] public int A { get; set; } }
public sealed class MsNoFormatter { }
[MessagePackFormatter(typeof(MsNoFormatter))] [MessagePackObject] public class MsFormattedBase { [Key(0)] public int A { get; set; } }
[MessagePackObject] public class MsFormattedDerived : MsFormattedBase { [Key(1)] public int B { get; set; } }
}
}

View file

@ -1,350 +0,0 @@
#nullable enable
using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using GameCult.Caching.MessagePack;
using NUnit.Framework;
using static GameCult.Caching.Tests.ElementIdTests;
using static GameCult.Caching.Tests.StoreReadabilityTests;
namespace GameCult.Caching.Tests
{
// A write never replaces a link at a store-owned path. The refusal holds for every way a write reaches a store (waiting or
// not, as a judge of another store's commit), for a link put there while the write waited on the store's lock, and for the
// directory store's manifest.
public class StoreLinkRefusalTests
{
private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(IdDeck), typeof(VectorItem) });
private string _directory = "";
[SetUp]
public void SetUp()
{
_directory = Path.Combine(Path.GetTempPath(), "cultlib-link-refusal-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(_directory);
}
[TearDown]
public void TearDown()
{
if (!Directory.Exists(_directory))
return;
foreach (var entry in Directory.EnumerateFileSystemEntries(_directory).Where(IsALink))
File.Delete(entry);
Directory.Delete(_directory, true);
}
private string PathOf(string name) => Path.Combine(_directory, name);
private static bool IsALink(string path)
{
var attributes = new FileInfo(path).Attributes;
return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
}
private static void Link(string path, string target)
{
try
{
File.CreateSymbolicLink(path, target);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Assert.Ignore("A file symlink needs a privilege this account lacks: " + ex.Message);
}
}
private static void SeedSingle(string path)
{
using var seed = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry });
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed")));
}
private static void SeedItem(string path)
{
using var seed = new CultCache(Registry);
seed.AddBackingStore(new SingleFileMessagePackBackingStore(path), typeof(VectorItem));
seed.Commit(batch => batch.Upsert(typeof(VectorItem), new VectorItem(), new CultRecordKey("item")));
}
// Runs `write` while another holder has `lockPath`; `swap` runs while it waits, then the lock is released.
private static Exception? WriteWhileWaiting(string lockPath, Action write, Action swap)
{
using var held = new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
var task = Task.Run(write);
Assert.That(task.Wait(TimeSpan.FromMilliseconds(600)), Is.False, "the lock is held, so the write must wait");
swap();
held.Dispose();
try
{
Assert.That(task.Wait(TimeSpan.FromSeconds(10)), Is.True);
return null;
}
catch (AggregateException ex)
{
return ex.GetBaseException();
}
}
private static void ReplaceWithLink(string path, string target)
{
File.Delete(path);
Link(path, target);
}
// ---- a link put at the store path while the write waited on the lock ----
[Test]
public void ACommitRefusesALinkPutAtTheStoreWhileItWaitedOnTheLock()
{
var path = PathOf("store.cc");
var target = PathOf("shared.cc");
SeedSingle(path);
SeedSingle(target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
var error = WriteWhileWaiting(path + ".lock",
() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late"))),
() => ReplaceWithLink(path, target));
Assert.That(error, Is.InstanceOf<CultStoreLinkedException>(), error?.ToString());
Assert.That(((CultStoreLinkedException)error!).Path, Is.EqualTo(path));
Assert.That(IsALink(path), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
}
[Test]
public void AFlushRefusesALinkPutAtTheStoreWhileItWaitedOnTheLock()
{
var path = PathOf("store.cc");
var target = PathOf("shared.cc");
SeedSingle(path);
SeedSingle(target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late")).GetAwaiter().GetResult();
var error = WriteWhileWaiting(path + ".lock", () => cache.FlushAllBackingStores(), () => ReplaceWithLink(path, target));
Assert.That(error, Is.InstanceOf<CultStoreLinkedException>(), error?.ToString());
Assert.That(IsALink(path), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
}
// ---- the non-waiting and judging write paths ----
[Test]
public void ATryCommitAtALiveLinkIsRefusedNotContendedOrReplaced()
{
var path = PathOf("store.cc");
var link = PathOf("link.cc");
SeedSingle(path);
Link(link, path);
var before = File.ReadAllBytes(path);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
var refusal = Assert.Throws<CultStoreLinkedException>(() =>
cache.TryCommit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("t"))))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(refusal.Message, Does.Contain(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
Assert.That(File.Exists(link + ".lock"), Is.False, "the refused write took the lock");
}
[Test]
public void ATryCommitAtALiveLinkWhoseLockIsHeldIsStillRefused()
{
var path = PathOf("store.cc");
var link = PathOf("link.cc");
SeedSingle(path);
Link(link, path);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
using var held = new FileStream(link + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
Assert.Throws<CultStoreLinkedException>(() =>
cache.TryCommit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("t"))));
Assert.That(IsALink(link), "the link was replaced");
}
[Test]
public void AnAdvisoryQueryOverACacheAtALiveLinkAnswersForTheRecordAndTheWriteIsStillRefused()
{
var path = PathOf("store.cc");
var link = PathOf("link.cc");
SeedSingle(path);
Link(link, path);
var before = File.ReadAllBytes(path);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
Assert.That(cache.IsWritable(new CultRecordKey("seed")), Is.True, "the query does not look at the store path");
Assert.That(cache.IsWritable(new CultRecordKey("fresh")), Is.True, "the query does not look at the store path");
var refusal = Assert.Throws<CultStoreLinkedException>(() =>
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("fresh"))))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
}
// A batch that upserts into one store and removes a key the cache does not hold has the other store judge that removal
// under its own lock (JudgeChecks).
private CultCache OpenTwoStores(string homePath, string otherPath)
{
var cache = new CultCache(Registry);
cache.AddBackingStore(new SingleFileMessagePackBackingStore(homePath), typeof(IdDeck));
cache.AddBackingStore(new SingleFileMessagePackBackingStore(otherPath), typeof(VectorItem));
return cache;
}
[Test]
public void AnotherStoreAtALiveLinkRefusesTheChecksItIsAskedToJudge()
{
var other = PathOf("other.cc");
var link = PathOf("other-link.cc");
SeedItem(other);
Link(link, other);
var before = File.ReadAllBytes(other);
using var cache = OpenTwoStores(PathOf("home.cc"), link);
var refusal = Assert.Throws<CultStoreLinkedException>(() => cache.Commit(batch =>
{
batch.Upsert(typeof(IdDeck), new IdDeck { Name = "home" }, new CultRecordKey("home"));
batch.Remove(new CultRecordKey("ghost"));
}))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(other), Is.EqualTo(before));
Assert.That(File.Exists(PathOf("home.cc")), Is.False, "the home store was written though a judge refused");
}
[Test]
public void AJudgeReadsTheFileItHoldsUnderTheLockNotTheFileBeforeItWaited()
{
var other = PathOf("other.cc");
SeedItem(other);
using var cache = OpenTwoStores(PathOf("home.cc"), other);
var error = WriteWhileWaiting(other + ".lock",
() => cache.Commit(batch =>
{
batch.Upsert(typeof(IdDeck), new IdDeck { Name = "home" }, new CultRecordKey("home"));
batch.Remove(new CultRecordKey("ghost"));
}),
() => File.WriteAllBytes(other, Array.Empty<byte>()));
Assert.That(error, Is.InstanceOf<CultStoreUnreadableException>(), error?.ToString());
}
// ---- the directory store's manifest ----
private DirectoryMessagePackBackingStore SeedDirectory(string manifest)
{
using (var seed = new CultCache(Registry))
{
seed.AddBackingStore(new DirectoryMessagePackBackingStore(manifest), typeof(IdDeck));
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed")));
}
return new DirectoryMessagePackBackingStore(manifest);
}
private CultCache OpenDirectory(DirectoryMessagePackBackingStore store)
{
var cache = new CultCache(Registry);
cache.AddBackingStore(store, typeof(IdDeck));
return cache;
}
[Test]
public void ADirectoryStoreCommitAtALiveManifestLinkIsRefusedAndNothingChanges()
{
var manifest = PathOf("real.cc");
var link = PathOf("link.cc");
SeedDirectory(manifest);
Link(link, manifest);
Directory.Move(manifest + ".records", link + ".records");
var before = File.ReadAllBytes(manifest);
var pages = Directory.GetFileSystemEntries(link + ".records").OrderBy(entry => entry).ToArray();
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
var refusal = Assert.Throws<CultStoreLinkedException>(() =>
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(manifest), Is.EqualTo(before));
Assert.That(Directory.GetFileSystemEntries(link + ".records").OrderBy(entry => entry), Is.EqualTo(pages), "the refused write left files beside the manifest");
}
// The refusal comes before the records folder or the lease file is made: a store that was never written leaves nothing behind.
[Test]
public void ADirectoryStoreCommitAtAManifestLinkPutThereAfterOpenCreatesNothing()
{
var link = PathOf("link.cc");
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
Link(link, PathOf("unmounted.cc"));
Assert.Throws<CultStoreLinkedException>(() =>
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(Directory.Exists(link + ".records"), Is.False, "the refused write made the records folder or its lease");
}
[Test]
public void ADirectoryStoreFlushAtALiveManifestLinkIsRefused()
{
var manifest = PathOf("real.cc");
var link = PathOf("link.cc");
SeedDirectory(manifest);
Link(link, manifest);
Directory.Move(manifest + ".records", link + ".records");
var before = File.ReadAllBytes(manifest);
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "f" }, new CultRecordKey("f")).GetAwaiter().GetResult();
Assert.Throws<CultStoreLinkedException>(() => cache.FlushAllBackingStores());
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(manifest), Is.EqualTo(before));
}
[Test]
public void ADirectoryStoreCommitRefusesAManifestLinkPutThereWhileItWaitedOnTheLease()
{
var manifest = PathOf("store.cc");
var target = PathOf("shared.cc");
SeedDirectory(manifest);
SeedDirectory(target);
var before = File.ReadAllBytes(target);
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(manifest));
var error = WriteWhileWaiting(Path.Combine(manifest + ".records", ".commit.lock"),
() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late"))),
() => ReplaceWithLink(manifest, target));
Assert.That(error, Is.InstanceOf<CultStoreLinkedException>(), error?.ToString());
Assert.That(IsALink(manifest), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
}
[Test]
public void ADirectoryStoreReadsThroughALiveManifestLink()
{
var manifest = PathOf("real.cc");
var link = PathOf("link.cc");
SeedDirectory(manifest);
Link(link, manifest);
Directory.Move(manifest + ".records", link + ".records");
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
Assert.That(cache.Get<IdDeck>(new CultRecordKey("seed"))?.Name, Is.EqualTo("seed"));
}
}
}

View file

@ -266,7 +266,7 @@ namespace GameCult.Caching.Tests
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry });
CreateDanglingDirectoryLink(path, Path.Combine(_directory, "unmounted")); CreateDanglingDirectoryLink(path, Path.Combine(_directory, "unmounted"));
Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<CultStoreLinkedException>()); Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<IOException>());
Assert.That(IsALink(path), "the link was replaced"); Assert.That(IsALink(path), "the link was replaced");
Assert.That(Directory.Exists(Path.Combine(_directory, "unmounted")), Is.False, "a write went through the link"); Assert.That(Directory.Exists(Path.Combine(_directory, "unmounted")), Is.False, "a write went through the link");
} }
@ -292,66 +292,11 @@ namespace GameCult.Caching.Tests
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }); using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry });
File.Delete(target); File.Delete(target);
Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<CultStoreLinkedException>()); Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<IOException>());
Assert.That(new FileInfo(path).LinkTarget, Is.Not.Null, "the link was replaced"); Assert.That(new FileInfo(path).LinkTarget, Is.Not.Null, "the link was replaced");
Assert.That(File.Exists(target), Is.False, "a write went through the link"); Assert.That(File.Exists(target), Is.False, "a write went through the link");
} }
// A write at a store path that is a link is refused with a typed error before the lock or the replace: the link, its target
// and the directory beside them are as they were.
[Test]
public void ACacheCommitAtALiveLinkIsRefusedAndNothingChanges()
{
var target = Path.Combine(_directory, "target.cc");
var link = Path.Combine(_directory, "link.cc");
using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry }))
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a")));
CreateFileLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
var refusal = Assert.Throws<CultStoreLinkedException>(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))))!;
Assert.That(refusal.Message, Does.Contain(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
Assert.That(File.Exists(link + ".lock"), Is.False, "the refused write took the lock");
Assert.That(Directory.GetFileSystemEntries(_directory).Where(entry => entry.EndsWith(".tmp")), Is.Empty);
}
[Test]
public void AFlushAtALiveLinkIsRefusedAndNothingChanges()
{
var target = Path.Combine(_directory, "target.cc");
var link = Path.Combine(_directory, "link.cc");
using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry }))
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a")));
CreateFileLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b")).GetAwaiter().GetResult();
Assert.Throws<CultStoreLinkedException>(() => cache.FlushAllBackingStores());
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
Assert.That(File.Exists(link + ".lock"), Is.False);
}
[Test]
public void AnOpenAtALiveLinkReadsTheTargetsRecords()
{
var target = Path.Combine(_directory, "target.cc");
var link = Path.Combine(_directory, "link.cc");
using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry }))
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a")));
CreateFileLinkOrIgnore(link, target);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
Assert.That(cache.Get<IdDeck>(new CultRecordKey("a"))?.Name, Is.EqualTo("a"));
}
// A directory at the path is something at the path: it is an I/O error on open and on commit, never an empty store, and // A directory at the path is something at the path: it is an I/O error on open and on commit, never an empty store, and
// it is left as it was. // it is left as it was.
[Test] [Test]

View file

@ -1,268 +0,0 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
using NUnit.Framework;
namespace GameCult.Caching.Tests
{
public class StoreReplaceTests
{
private sealed class FakeKernel : IReplaceKernel
{
private readonly Queue<int> _posix;
private readonly Queue<int> _moveEx;
public readonly List<string> Log = new List<string>();
public readonly List<int> Sleeps = new List<int>();
public Exception? PosixThrows;
public Exception? MoveExThrows;
public FakeKernel(int[] posix, int[] moveEx)
{
_posix = new Queue<int>(posix);
_moveEx = new Queue<int>(moveEx);
}
public int RenamePosix(string staged, string destination)
{
Log.Add("posix");
if (PosixThrows != null) throw PosixThrows;
return _posix.Count > 0 ? _posix.Dequeue() : 0;
}
public int MoveFileEx(string staged, string destination, uint flags)
{
Log.Add("moveex:" + flags);
if (MoveExThrows != null) throw MoveExThrows;
return _moveEx.Count > 0 ? _moveEx.Dequeue() : 0;
}
public void Sleep(int milliseconds) => Sleeps.Add(milliseconds);
}
private static IOException Fails(FakeKernel kernel)
=> Assert.Throws<IOException>(() => StoreReplace.ReplaceWindows("a", "b", kernel))!;
[Test]
public void APosixRenameThatWorksIsTheWholeReplace()
{
var kernel = new FakeKernel(new[] { 0 }, new int[0]);
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix" }));
Assert.That(kernel.Sleeps, Is.Empty);
}
[TestCase(1)]
[TestCase(50)]
[TestCase(87)]
public void WhereAPosixRenameIsUnsupportedTheReplaceFallsBackToWriteThroughMoveFileEx(int unsupported)
{
var kernel = new FakeKernel(new[] { unsupported }, new[] { 0 });
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
}
[Test]
public void OnceThePosixRenameIsUnsupportedTheRestOfTheCallStaysOnMoveFileEx()
{
var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 0 });
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" }));
Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20 }));
}
[TestCase(5)]
[TestCase(32)]
[TestCase(33)]
public void ATransientErrorRetriesTheSamePrimitiveAfterTwentyMilliseconds(int transient)
{
var kernel = new FakeKernel(new[] { transient, transient, 0 }, new int[0]);
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "posix", "posix" }));
Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20, 20 }));
}
[TestCase(1)]
[TestCase(50)]
[TestCase(87)]
public void AnUnsupportedCodeAfterTheFallbackIsAnErrorNotAnotherFallback(int unsupported)
{
var kernel = new FakeKernel(new[] { 87 }, new[] { unsupported });
var error = Fails(kernel);
Assert.That(error.Message, Does.Contain("Win32 error " + unsupported));
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
}
private static IEnumerable<Exception> Unresolved()
{
yield return new DllNotFoundException("kernel32.dll");
yield return new EntryPointNotFoundException("SetFileInformationByHandle");
}
[TestCaseSource(nameof(Unresolved))]
public void APosixCallThatDoesNotResolveTakesTheUnsupportedFallback(Exception unresolved)
{
var kernel = new FakeKernel(new int[0], new[] { 0 }) { PosixThrows = unresolved };
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
}
[TestCaseSource(nameof(Unresolved))]
public void AMoveFileExThatDoesNotResolveDegradesToThePortableReplace(Exception unresolved)
{
var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
var destination = Path.Combine(dir, "store.cc");
var staged = Path.Combine(dir, "staged");
File.WriteAllBytes(destination, new byte[] { 1 });
File.WriteAllBytes(staged, new byte[] { 2 });
var kernel = new FakeKernel(new[] { 87 }, new int[0]) { MoveExThrows = unresolved };
StoreReplace.ReplaceWindows(staged, destination, kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 }));
Assert.That(File.Exists(staged), Is.False);
// Both calls unresolved: the same portable replace, here onto an absent destination.
File.Delete(destination);
File.WriteAllBytes(staged, new byte[] { 3 });
var both = new FakeKernel(new int[0], new int[0]) { PosixThrows = unresolved, MoveExThrows = unresolved };
StoreReplace.ReplaceWindows(staged, destination, both);
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 }));
}
finally { Directory.Delete(dir, true); }
}
[Test]
public void TheFallbackRunsInTheSameAttemptEvenOnTheLastOne()
{
var posix = new int[25].Fill(32);
posix[24] = 87;
var kernel = new FakeKernel(posix, new[] { 0 });
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log.Count, Is.EqualTo(26));
Assert.That(kernel.Log[24], Is.EqualTo("posix"));
Assert.That(kernel.Log[25], Is.EqualTo("moveex:9"));
}
[Test]
public void ARetriedMoveFileExThatAnswersUnsupportedFailsAfterOneCallNotTwo()
{
var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 87 });
Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 87"));
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" }));
}
[Test]
public void ARetryThatNeverSucceedsGivesUpAfterTwentyFiveAttemptsNamingTheCode()
{
var kernel = new FakeKernel(new int[30].Fill(32), new int[0]);
var error = Fails(kernel);
Assert.That(error.Message, Does.Contain("25 attempts").And.Contain("last Win32 error 32"));
Assert.That(kernel.Log.Count, Is.EqualTo(25));
Assert.That(kernel.Sleeps.Count, Is.EqualTo(25));
}
[Test]
public void AnyOtherCodeThrowsAtOnceNamingTheCode()
{
var kernel = new FakeKernel(new[] { 2 }, new int[0]);
var error = Fails(kernel);
Assert.That(error.Message, Does.Contain("Win32 error 2"));
Assert.That(kernel.Log.Count, Is.EqualTo(1));
Assert.That(kernel.Sleeps, Is.Empty);
}
[Test]
public void AFallbackThatFailsOtherwiseThrowsItsCode()
{
var kernel = new FakeKernel(new[] { 87 }, new[] { 3 });
Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 3"));
}
[Test]
public void ReplaceOverAnExistingAndThenAnAbsentDestinationLeavesTheStagedBytes()
{
var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
var destination = Path.Combine(dir, "store.cc");
var staged = Path.Combine(dir, "staged");
File.WriteAllBytes(destination, new byte[] { 1 });
File.WriteAllBytes(staged, new byte[] { 2 });
StoreReplace.Replace(staged, destination);
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 }));
Assert.That(File.Exists(staged), Is.False);
File.Delete(destination);
File.WriteAllBytes(staged, new byte[] { 3 });
StoreReplace.Replace(staged, destination);
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 }));
}
finally { Directory.Delete(dir, true); }
}
[Test]
public void OnWindowsAReplaceOverADestinationAReaderHoldsOpenSucceedsOnTheFirstAttempt()
{
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
Assert.Ignore("The real kernel runs on Windows only.");
var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
var destination = Path.Combine(dir, "store.cc");
var staged = Path.Combine(dir, "staged");
File.WriteAllBytes(destination, new byte[] { 1 });
File.WriteAllBytes(staged, new byte[] { 2 });
using (new FileStream(destination, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete))
{
var kernel = new CountingKernel(StoreReplace.WindowsKernel);
StoreReplace.ReplaceWindows(staged, destination, kernel);
Assert.That(kernel.PosixCalls, Is.EqualTo(1));
Assert.That(kernel.MoveExCalls, Is.EqualTo(0));
Assert.That(kernel.Sleeps, Is.EqualTo(0));
Assert.That(kernel.LastCode, Is.EqualTo(0));
}
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 }));
}
finally { Directory.Delete(dir, true); }
}
private sealed class CountingKernel : IReplaceKernel
{
private readonly IReplaceKernel _inner;
public int PosixCalls, MoveExCalls, Sleeps, LastCode;
public CountingKernel(IReplaceKernel inner) { _inner = inner; }
public int RenamePosix(string staged, string destination)
{
PosixCalls++;
return LastCode = _inner.RenamePosix(staged, destination);
}
public int MoveFileEx(string staged, string destination, uint flags)
{
MoveExCalls++;
return LastCode = _inner.MoveFileEx(staged, destination, flags);
}
public void Sleep(int milliseconds) { Sleeps++; _inner.Sleep(milliseconds); }
}
}
internal static class ArrayFill
{
public static int[] Fill(this int[] array, int value)
{
for (var i = 0; i < array.Length; i++) array[i] = value;
return array;
}
}
}

View file

@ -699,10 +699,10 @@ namespace GameCult.Caching.Tests
_nested = nested; _nested = nested;
} }
public override void PushAll(bool soft) public override void PushAll()
{ {
_nested(); _nested();
base.PushAll(soft); base.PushAll();
} }
} }

View file

@ -1,132 +0,0 @@
#nullable enable
using System;
using System.IO;
using System.Threading.Tasks;
using GameCult.Caching.MessagePack;
using NUnit.Framework;
using static GameCult.Caching.Tests.ElementIdTests;
namespace GameCult.Caching.Tests
{
// A write decides from the file it holds the lock over, not from the file as it was before it waited: whatever another
// writer left while this one waited for <path>.lock is what the write applies onto, or refuses.
public class StoreWriteReadsUnderTheLockTests
{
private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(IdDeck) });
private static readonly CultSchemaCatalogEntry One = new() { SchemaId = "tests.under-lock.one", SchemaName = "tests.under-lock.one" };
private string _directory = "";
private string _path = "";
[SetUp]
public void SetUp()
{
_directory = Path.Combine(Path.GetTempPath(), "cultlib-under-lock-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(_directory);
_path = Path.Combine(_directory, "store.cc");
}
[TearDown]
public void TearDown()
{
if (Directory.Exists(_directory))
Directory.Delete(_directory, true);
}
private void Seed()
{
using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry });
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed")));
}
// The write starts and waits on the held lock; while it waits, the file is replaced by what `swap` makes of it.
private Exception? WriteWhileTheFileChangesUnderIt(Action write, Action swap)
{
using var held = new FileStream(_path + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
var task = Task.Run(write);
Assert.That(task.Wait(TimeSpan.FromMilliseconds(600)), Is.False, "the lock is held, so the write must wait");
swap();
held.Dispose();
try
{
Assert.That(task.Wait(TimeSpan.FromSeconds(10)), Is.True);
return null;
}
catch (AggregateException ex)
{
return ex.GetBaseException();
}
}
private static void ZeroBytes(string path) => File.WriteAllBytes(path, Array.Empty<byte>());
private static CultPersistedStoreSnapshot Snapshot(string path) =>
CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
private void MarkV3()
{
var snapshot = Snapshot(_path);
snapshot.FormatVersion = CultPersistedStoreSnapshot.FormatV3;
File.WriteAllBytes(_path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot));
}
private void CommitWrite(Action swap, out Exception? error)
{
using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
error = WriteWhileTheFileChangesUnderIt(
() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "committed" }, new CultRecordKey("committed"))), swap);
}
private void FlushWriteWhile(Action swap, out Exception? error)
{
using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "flushed" }, new CultRecordKey("flushed")).GetAwaiter().GetResult();
error = WriteWhileTheFileChangesUnderIt(() => cache.FlushAllBackingStores(), swap);
}
private void CachelessWriteWhile(Action swap, out Exception? error) =>
error = WriteWhileTheFileChangesUnderIt(
() => new SingleFileMessagePackBackingStore(_path).WriteRecord(
new CultPersistedRecord { Key = "written", SchemaId = One.SchemaId, StoredAt = "2026-10-22T00:00:00.0000000+00:00", Payload = new byte[] { 1 } }, One),
swap);
private delegate void WriteWhile(Action swap, out Exception? error);
[Test]
public void AFlushDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(FlushWriteWhile);
[Test]
public void ACommitDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(CommitWrite);
[Test]
public void ACachelessWriteDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(CachelessWriteWhile);
[Test]
public void AFlushKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(FlushWriteWhile);
[Test]
public void ACommitKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(CommitWrite);
[Test]
public void ACachelessWriteKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(CachelessWriteWhile);
private void RefusesWhatItFindsUnderTheLock(WriteWhile write)
{
Seed();
write(() => ZeroBytes(_path), out var error);
Assert.That(error, Is.InstanceOf<CultStoreUnreadableException>());
Assert.That(new FileInfo(_path).Length, Is.Zero, "the refused file was written over");
}
private void KeepsTheV3HeaderItFindsUnderTheLock(WriteWhile write)
{
Seed();
write(MarkV3, out var error);
Assert.That(error, Is.Null, error?.ToString());
Assert.That(Snapshot(_path).FormatVersion, Is.EqualTo(CultPersistedStoreSnapshot.FormatV3), "the write read the file as it was before the lock");
}
}
}

View file

@ -1,11 +0,0 @@
{
"stryker-config": {
"project": "GameCult.Caching.csproj",
"mutate": [
"**/src/GameCult.Caching/**/*.cs"
],
"mutation-level": "Standard",
"reporters": [ "progress", "html", "json" ],
"thresholds": { "high": 80, "low": 60, "break": 0 }
}
}

View file

@ -58,22 +58,4 @@ public class BytesAfterPathTests
.MakeGenericMethod(typeof(CultMeshCdnArtifactManifest)); .MakeGenericMethod(typeof(CultMeshCdnArtifactManifest));
Refusal(() => read.Invoke(null, new object[] { snapshot, record.SchemaId, record.RecordKey })).Should().Contain(Clause); Refusal(() => read.Invoke(null, new object[] { snapshot, record.SchemaId, record.RecordKey })).Should().Contain(Clause);
} }
[Test]
public void ASingleFileDocumentReadRefusesATail()
{
var path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), $"cultlib-bytesafter-mesh-{Guid.NewGuid():N}.cc");
try
{
var descriptor = CultDocumentRegistry.Shared.GetRequired<CultMeshCdnArtifactManifest>();
var schema = new CultMeshSingleFileDocumentSchema(descriptor.SchemaId, descriptor.SchemaName, descriptor.SchemaVersion);
var payload = WithTail(Manifest()).Payload;
CultMesh.WriteSingleFileDocumentPayload(path, new CultRecordKey("k"), schema, null, payload);
Refusal(() => CultMesh.ReadSingleFileDocument<CultMeshCdnArtifactManifest>(path, new CultRecordKey("k"))).Should().Contain(Clause);
}
finally
{
System.IO.File.Delete(path);
}
}
} }

View file

@ -4,7 +4,6 @@ using System.Collections.Generic;
using System.Diagnostics; using System.Diagnostics;
using System.IO; using System.IO;
using System.Linq; using System.Linq;
using System.Runtime.InteropServices;
using System.Threading; using System.Threading;
using System.Threading.Tasks; using System.Threading.Tasks;
using FluentAssertions; using FluentAssertions;
@ -365,11 +364,11 @@ public sealed class CultMeshSingleFileStoreTests
[Test] [Test]
public void ACacheOpeningDuringMeshWritesNeverSeesAnEmptyStore() public void ACacheOpeningDuringMeshWritesNeverSeesAnEmptyStore()
{ {
// Windows replaces with a POSIX-semantics rename first (StoreReplace), which makes a lock-free reader finding the path // Windows replaces with ReplaceFile, which leaves a moment with nothing at the path for a lock-free reader: 1,722 empty
// absent rare, not impossible: a stalled rename was measured leaving it absent for 5-47 ms on Windows. So on Windows a // stores in 17,313 opens at b1ff2dcb. That is the store's replace, not Mesh's, and it is the same for a cache flush beside a
// brief not-found (a read throwing, or a cache opening onto an absent path) is tolerated and reported; elsewhere rename // cache open; the fork is in the cut report.
// is atomic and none is allowed. if (OperatingSystem.IsWindows())
var windows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows); Assert.Ignore("Windows File.Replace is not atomic to a lock-free reader; see cut report fork mesh-reader-windows-replace-window.");
var path = Path.Combine(_root, "race.cc"); var path = Path.Combine(_root, "race.cc");
CultMesh.WriteSingleFileDocument(path, Key, Publication()); CultMesh.WriteSingleFileDocument(path, Key, Publication());
@ -387,7 +386,7 @@ public sealed class CultMeshSingleFileStoreTests
Volatile.Write(ref stop, 1); Volatile.Write(ref stop, 1);
}); });
int opens = 0, empty = 0, reads = 0, absent = 0; int opens = 0, empty = 0, reads = 0;
var clock = Stopwatch.StartNew(); var clock = Stopwatch.StartNew();
while (Volatile.Read(ref stop) == 0 && clock.Elapsed < TimeSpan.FromSeconds(60)) while (Volatile.Read(ref stop) == 0 && clock.Elapsed < TimeSpan.FromSeconds(60))
{ {
@ -398,10 +397,6 @@ public sealed class CultMeshSingleFileStoreTests
CultMesh.ReadSingleFileDocumentPayload(path, Key, SchemaId()); CultMesh.ReadSingleFileDocumentPayload(path, Key, SchemaId());
reads++; reads++;
} }
catch (Exception ex) when (windows && (ex is FileNotFoundException || ex is DirectoryNotFoundException))
{
absent++;
}
catch (Exception ex) catch (Exception ex)
{ {
errors.Enqueue("read: " + ex.GetType().Name + ": " + ex.Message); errors.Enqueue("read: " + ex.GetType().Name + ": " + ex.Message);
@ -409,10 +404,9 @@ public sealed class CultMeshSingleFileStoreTests
} }
writer.Wait(TimeSpan.FromSeconds(60)).Should().BeTrue(); writer.Wait(TimeSpan.FromSeconds(60)).Should().BeTrue();
TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} not-found reads={absent} mesh reads ok={reads} errors={errors.Count}"); TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} mesh reads ok={reads} errors={errors.Count}");
errors.Take(3).Should().BeEmpty(); errors.Take(3).Should().BeEmpty();
if (!windows) empty.Should().Be(0, "a cache opened in the window where nothing was at the path");
empty.Should().Be(0, "a cache opened in the window where nothing was at the path");
opens.Should().BeGreaterThan(0); opens.Should().BeGreaterThan(0);
} }
@ -440,74 +434,32 @@ public sealed class CultMeshSingleFileStoreTests
TempFiles().Should().BeEmpty(); TempFiles().Should().BeEmpty();
} }
// A write at a store path that is a link is refused before the lock, the staging file and the replace: the link and its // The store's replace decides what a write leaves where a live link was, and a Mesh write leaves what a cache commit leaves:
// target stay as they were, and nothing is left beside them. // one replace, whatever the platform does with a link.
private void AssertRefusedAndUntouched(string link, string target, byte[] targetBefore, Action write)
{
var refusal = Assert.Throws<CultStoreLinkedException>(() => write())!;
refusal.Message.Should().Contain(link);
IsALink(link).Should().BeTrue("the link was replaced");
File.ReadAllBytes(target).Should().Equal(targetBefore);
File.Exists(link + ".lock").Should().BeFalse("the refused write took the lock");
TempFiles().Should().BeEmpty();
}
[Test] [Test]
public void AMeshWriteAtALiveLinkIsRefusedAndNothingChanges() public void AMeshWriteOverALiveLinkLeavesWhatACacheCommitLeaves()
{ {
var target = Path.Combine(_root, "target.cc"); string Outcome(string name, Action<string> write)
var link = Path.Combine(_root, "link.cc");
CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
CreateLiveLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
AssertRefusedAndUntouched(link, target, before, () => CultMesh.WriteSingleFileDocument(link, Key, Publication("second")));
AssertRefusedAndUntouched(link, target, before, () => CultMesh.WriteSingleFileDocumentPayload(link, Key, Raw, null, EmptyArray));
}
[Test]
public void AReadThroughALiveLinkStillWorks()
{
var target = Path.Combine(_root, "target.cc");
var link = Path.Combine(_root, "link.cc");
CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
CreateLiveLinkOrIgnore(link, target);
CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(link, Key).ProducerId.Should().Be("first");
RecordsSeenByACache(link).Should().Be(1);
}
// A link to a directory that is there is a directory, not an unreachable store; a write at it is a link refusal.
[Test]
public void ALiveDirectoryLinkIsADirectoryErrorOnReadAndALinkRefusalOnWrite()
{
var directory = Path.Combine(_root, "real-directory");
var link = Path.Combine(_root, "directory-link.cc");
Directory.CreateDirectory(directory);
CreateDirectoryLink(link, directory);
Assert.Catch<IOException>(() => CultMesh.ReadSingleFileDocumentPayload(link, Key, "raw:schema"))!.Message.Should().Contain("directory");
Assert.Catch<IOException>(() => RecordsSeenByACache(link))!.Message.Should().Contain("directory");
Assert.Throws<CultStoreLinkedException>(() => CultMesh.WriteSingleFileDocument(link, Key, Publication()));
Directory.GetFileSystemEntries(directory).Should().BeEmpty();
IsALink(link).Should().BeTrue();
}
private static void CreateDirectoryLink(string path, string target)
{
if (!OperatingSystem.IsWindows())
{ {
Directory.CreateSymbolicLink(path, target); var target = Path.Combine(_root, name + "-target.cc");
return; var link = Path.Combine(_root, name + "-link.cc");
CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
CreateLiveLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
write(link);
CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(link, Key).ProducerId.Should().Be("second", name);
return $"linkKept={IsALink(link)} targetChanged={!File.ReadAllBytes(target).SequenceEqual(before)}";
} }
var mklink = Process.Start(new ProcessStartInfo("cmd.exe", $"/c mklink /J \"{path}\" \"{target}\"") var viaMesh = Outcome("mesh", link => CultMesh.WriteSingleFileDocument(link, Key, Publication("second")));
var viaCache = Outcome("cache", link =>
{ {
CreateNoWindow = true, using var cache = CultCacheMessagePack.Create(link, ReadOnlyOpen);
RedirectStandardOutput = true, cache.Commit(batch => batch.Upsert(typeof(CultMeshBodyPublicationDocument), Publication("second"), Key));
RedirectStandardError = true, });
})!;
mklink.WaitForExit(); viaMesh.Should().Be(viaCache);
Assert.That(mklink.ExitCode, Is.Zero, "mklink /J failed");
} }
} }

View file

@ -21,15 +21,7 @@ namespace GameCult.Networking.Tests
[Key(1)] public int N = 3; [Key(1)] public int N = 3;
} }
[CultDocument("tests.bytes_after_deep", "tests.bytes_after_deep.v1")] private static readonly CultDocumentRegistry Documents = CultDocumentRegistry.ForTypes(new[] { typeof(TailDoc) });
[MessagePackObject]
public sealed class DeepDoc
{
[Key(0)] [CultName] public string Id = "a";
[Key(1)] public DeepDoc Inner;
}
private static readonly CultDocumentRegistry Documents = CultDocumentRegistry.ForTypes(new[] { typeof(TailDoc), typeof(DeepDoc) });
private static byte[] WithTail() => private static byte[] WithTail() =>
CultDocumentMessagePackSerialization.SerializeUntyped(new TailDoc(), typeof(TailDoc), Documents).Concat(new byte[] { 0xc0 }).ToArray(); CultDocumentMessagePackSerialization.SerializeUntyped(new TailDoc(), typeof(TailDoc), Documents).Concat(new byte[] { 0xc0 }).ToArray();
@ -66,28 +58,5 @@ namespace GameCult.Networking.Tests
var registry = new CultNetDocumentRegistry(Documents); var registry = new CultNetDocumentRegistry(Documents);
Refuses(() => registry.DeserializeRawDocument(Raw(registry, WithTail()))); Refuses(() => registry.DeserializeRawDocument(Raw(registry, WithTail())));
} }
// The write side of the same rule: a record with no binding is serialized by SerializeUntyped, and a payload nested past
// the limit is refused there, before it can reach the wire.
[Test]
public void ToRawRecordWithNoBindingRefusesAPayloadNestedPastTheLimit()
{
var registry = new CultNetDocumentRegistry(Documents);
var descriptor = Documents.GetRequired(typeof(DeepDoc));
DeepDoc Chain(int links)
{
var head = new DeepDoc();
var tip = head;
for (var i = 0; i < links; i++) tip = tip.Inner = new DeepDoc();
return head;
}
// Each link is one array; the root is depth 1, so 255 links is depth 256 (at the limit) and 256 links is 257.
Assert.That(registry.ToRawRecord(descriptor, new CultRecordKey("a"), Chain(255), "1970-01-01T00:00:00.000Z").Payload, Is.Not.Empty);
var refused = Assert.Throws<MessagePackSerializationException>(
() => registry.ToRawRecord(descriptor, new CultRecordKey("a"), Chain(256), "1970-01-01T00:00:00.000Z"))!;
Assert.That(refused.Message, Does.Contain("refused on write"));
Assert.That(refused.Message, Does.Contain(typeof(DeepDoc).FullName));
}
} }
} }

View file

@ -7377,183 +7377,6 @@ namespace GameCult.Networking.Tests
Assert.That(developmentServer.LogSensitivePayloads, Is.True); Assert.That(developmentServer.LogSensitivePayloads, Is.True);
} }
// Records the flag each PushAll took, and holds nothing: a flush that decided soft itself and never asked the store records nothing.
private sealed class SoftFlagSpyStore : CacheBackingStore
{
public sealed class SpyHome { }
public readonly List<bool> Flags = new();
public void Stage() => IsDirty = true;
public override void PullAll() { }
public override void Push(CultStoredDocument entry) { }
public override void Delete(CultStoredDocument entry) { }
public override void PushAll(bool soft) => Flags.Add(soft);
public override CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait) => CultCommitOutcome.Committed;
}
[Test]
public async Task CultNetLocal_Host_FlushAsync_Hands_The_Store_The_Flag_It_Was_Given()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-spy-{Guid.NewGuid():N}.msgpack");
CultNetHost? host = null;
try
{
host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false });
var spy = new SoftFlagSpyStore();
host.Cache.AddBackingStore(spy, typeof(SoftFlagSpyStore.SpyHome));
spy.Stage();
await host.FlushAsync(soft: true);
await host.FlushAsync();
Assert.That(spy.Flags, Is.EqualTo(new[] { true, false }));
}
finally
{
host?.Dispose();
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task CultMeshNode_FlushAsync_Hands_The_Store_The_Flag_It_Was_Given()
{
var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-spy", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(rootPath);
var documents = new CultNetDocumentRegistry()
.Register(CultNetDocumentBinding.ForDocument<MeshQuickstartNote>());
using var node = await CultMesh.CreateNodeAsync(Path.Combine(rootPath, "world.ccmp"), new CultMeshNodeOptions
{
StartServer = false,
DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-spy-local", DocumentRegistry = documents }
});
var spy = new SoftFlagSpyStore();
node.Cache.AddBackingStore(spy, typeof(SoftFlagSpyStore.SpyHome));
spy.Stage();
await node.FlushAsync(soft: true);
await node.FlushAsync();
Assert.That(spy.Flags, Is.EqualTo(new[] { true, false }));
}
[Test]
public async Task CultNetLocal_Host_SoftFlush_Passes_The_Flag_To_The_Store()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-soft-{Guid.NewGuid():N}.msgpack");
CultNetHost? host = null;
try
{
host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false });
await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "a@example.test", PasswordHash = "hash", Username = "First" });
await host.FlushAsync();
var before = File.ReadAllBytes(filePath);
await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "b@example.test", PasswordHash = "hash", Username = "Second" });
await host.FlushAsync(soft: true);
Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(before), "a soft host flush rewrote a store it reads");
await host.FlushAsync();
Assert.That(File.ReadAllBytes(filePath), Is.Not.EqualTo(before), "a plain host flush after a skipped soft flush did not land the staged write");
}
finally
{
host?.Dispose();
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task CultMeshNode_SoftFlush_Passes_The_Flag_To_The_Store()
{
var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-flush", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(rootPath);
var cachePath = Path.Combine(rootPath, "world.ccmp");
var documents = new CultNetDocumentRegistry()
.Register(CultNetDocumentBinding.ForDocument<MeshQuickstartNote>());
using var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions
{
StartServer = false,
DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-flush-local", DocumentRegistry = documents }
});
await node.Database.PutAsync(new CultRecordKey("note:one"), new MeshQuickstartNote { NoteId = "note:one", Body = "one" });
await node.FlushAsync();
var before = File.ReadAllBytes(cachePath);
await node.Database.PutAsync(new CultRecordKey("note:two"), new MeshQuickstartNote { NoteId = "note:two", Body = "two" });
await node.FlushAsync(soft: true);
Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(before), "a soft node flush rewrote a store it reads");
await node.FlushAsync();
Assert.That(File.ReadAllBytes(cachePath), Is.Not.EqualTo(before), "a plain node flush after a skipped soft flush did not land the staged write");
}
// The cache HOLDS a record in every step: a passthrough that decides soft from the cache's memory instead of
// handing the store the flag fails the absent step (it skips and writes nothing) or the unreadable step (it
// never reaches the reader that refuses).
[Test]
public async Task CultNetLocal_Host_SoftFlush_With_A_Held_Record_Writes_Absent_Skips_Readable_Refuses_Unreadable()
{
var filePath = Path.Combine(Path.GetTempPath(), $"cultnet-host-soft-held-{Guid.NewGuid():N}.msgpack");
CultNetHost? host = null;
try
{
host = await CultNetLocal.CreateHostAsync(filePath, new CultNetHostOptions { StartServer = false });
await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "a@example.test", PasswordHash = "hash", Username = "First" });
Assert.That(File.Exists(filePath), Is.False);
await host.FlushAsync(soft: true);
Assert.That(File.Exists(filePath), Is.True, "a soft host flush over an absent store, with a record held, wrote nothing");
var written = File.ReadAllBytes(filePath);
await host.Cache.UpsertAsync(new PlayerData { PlayerId = Guid.NewGuid(), Email = "b@example.test", PasswordHash = "hash", Username = "Second" });
await host.FlushAsync(soft: true);
Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(written), "a soft host flush rewrote a readable store");
var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 };
File.WriteAllBytes(filePath, junk);
Assert.That(async () => await host.FlushAsync(soft: true), Throws.InstanceOf<CultStoreUnreadableException>());
Assert.That(File.ReadAllBytes(filePath), Is.EqualTo(junk));
}
finally
{
host?.Dispose();
if (File.Exists(filePath)) File.Delete(filePath);
}
}
[Test]
public async Task CultMeshNode_SoftFlush_With_A_Held_Record_Writes_Absent_Skips_Readable_Refuses_Unreadable()
{
var rootPath = Path.Combine(TestContext.CurrentContext.WorkDirectory, "cultmesh-soft-held", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(rootPath);
var cachePath = Path.Combine(rootPath, "world.ccmp");
var documents = new CultNetDocumentRegistry()
.Register(CultNetDocumentBinding.ForDocument<MeshQuickstartNote>());
using var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions
{
StartServer = false,
DatabaseOptions = new CultNetDatabaseOptions { RuntimeId = "soft-held-local", DocumentRegistry = documents }
});
await node.Database.PutAsync(new CultRecordKey("note:one"), new MeshQuickstartNote { NoteId = "note:one", Body = "one" });
Assert.That(File.Exists(cachePath), Is.False);
await node.FlushAsync(soft: true);
Assert.That(File.Exists(cachePath), Is.True, "a soft node flush over an absent store, with a record held, wrote nothing");
var written = File.ReadAllBytes(cachePath);
await node.Database.PutAsync(new CultRecordKey("note:two"), new MeshQuickstartNote { NoteId = "note:two", Body = "two" });
await node.FlushAsync(soft: true);
Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(written), "a soft node flush rewrote a readable store");
var junk = new byte[] { 0xC1, 0x01, 0x02, 0x03 };
File.WriteAllBytes(cachePath, junk);
Assert.That(async () => await node.FlushAsync(soft: true), Throws.InstanceOf<CultStoreUnreadableException>());
Assert.That(File.ReadAllBytes(cachePath), Is.EqualTo(junk));
}
[Test] [Test]
public async Task CultNetLocal_CreateHostAsync_Wires_Durable_Cache_Without_Starting_Server() public async Task CultNetLocal_CreateHostAsync_Wires_Durable_Cache_Without_Starting_Server()
{ {

View file

@ -341,7 +341,7 @@ namespace GameCult.Networking.Tests
{ {
var path = Path.Combine(RepoRoot(), "contracts", "cultnet", "interop", "selection-vectors.rs-written.json"); var path = Path.Combine(RepoRoot(), "contracts", "cultnet", "interop", "selection-vectors.rs-written.json");
if (!File.Exists(path)) if (!File.Exists(path))
Assert.Inconclusive($"{path} is missing - run cargo test --locked -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first."); Assert.Inconclusive($"{path} is missing - run cargo test -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first.");
var (registry, rows, _) = BuildFixture(); var (registry, rows, _) = BuildFixture();
using var document = JsonDocument.Parse(File.ReadAllText(path)); using var document = JsonDocument.Parse(File.ReadAllText(path));

View file

@ -111,7 +111,6 @@ probe() {
node ../../node_modules/typescript/bin/tsc -p tsconfig.test.json node ../../node_modules/typescript/bin/tsc -p tsconfig.test.json
node --test dist-test/test/old-reader-v3.test.js ;; node --test dist-test/test/old-reader-v3.test.js ;;
python) PYTHONPATH=packages/cultcache-py/src python3 packages/cultcache-py/tests/test_old_reader_v3.py -v ;; python) PYTHONPATH=packages/cultcache-py/src python3 packages/cultcache-py/tests/test_old_reader_v3.py -v ;;
# The staged tree is the frozen cultcache-rs-v0.3.1 revision, which has no committed Cargo.lock: --locked cannot work here.
rust) cd packages/cultcache-rs && cargo test --test old_reader_v3 -- --nocapture ;; rust) cd packages/cultcache-rs && cargo test --test old_reader_v3 -- --nocapture ;;
*) usage ;; *) usage ;;
esac esac

View file

@ -6,7 +6,6 @@ All notable changes to this package are documented in this file.
### Breaking ### Breaking
- `CacheBackingStore.PushAll()` is no longer abstract: it is a non-virtual convenience over the new abstract `PushAll(bool soft)`. A store kind that overrides `PushAll()` no longer compiles; override `PushAll(bool soft)` instead. A soft flush writes only where nothing is at the path: over a store the reader reads, empty included, it returns without writing and leaves the staged writes staged and dirty for a later plain flush; over a file the reader cannot read it refuses. `CultCache.FlushAsync(bool soft = false)`, `CultCache.FlushAllBackingStores(bool soft = false)`, `CultMeshNode.FlushAsync(soft)` and `CultNetHost.FlushAsync(soft)` pass the flag to the stores; `soft: true` was accepted and ignored before, so a caller that passed it now stops landing writes on an existing store. Semver: breaking for C# consumers with their own store kind.
- A document payload with bytes after its first MessagePack value is refused on read and on write (`MessagePackSerializationException` naming the type and the count of bytes after the value, never the bytes); the tail is not walked, so a tail nested past the limit is refused as bytes after the value, not as depth. Before, the tail was ignored on read and dropped on the next write. A payload that ends early is still the deserializer's to refuse. A CDN manifest is a record payload and follows the same rule: the acceptance vectors `trailing_nil`, `trailing_reserved_code` and `trailing_truncated_array` in `contracts/cultmesh/content-acceptance-vectors.json` are now refusals. `CultMeshProviderSessionWire.DecodeCommandDocument` refuses a command record whose payload has bytes after its first value (`MessagePackSerializationException` with the count); before, the tail was ignored. Semver: breaking for C# consumers, a major bump. - A document payload with bytes after its first MessagePack value is refused on read and on write (`MessagePackSerializationException` naming the type and the count of bytes after the value, never the bytes); the tail is not walked, so a tail nested past the limit is refused as bytes after the value, not as depth. Before, the tail was ignored on read and dropped on the next write. A payload that ends early is still the deserializer's to refuse. A CDN manifest is a record payload and follows the same rule: the acceptance vectors `trailing_nil`, `trailing_reserved_code` and `trailing_truncated_array` in `contracts/cultmesh/content-acceptance-vectors.json` are now refusals. `CultMeshProviderSessionWire.DecodeCommandDocument` refuses a command record whose payload has bytes after its first value (`MessagePackSerializationException` with the count); before, the tail was ignored. Semver: breaking for C# consumers, a major bump.
- A single-file store holding two records under one key is unreadable: the reader refuses it with `CultStoreUnreadableException` naming the key (keys compare by ordinal code units, no case folding or normalisation), so a cache, `ReadStoreSnapshot` and a flush refuse the file and leave it as it is. Before, the later record silently replaced the earlier. A directory store reads its manifest through the same reader and refuses a duplicate key the same way. Repair the duplicate with the runtime that wrote it before reading the store here. - A single-file store holding two records under one key is unreadable: the reader refuses it with `CultStoreUnreadableException` naming the key (keys compare by ordinal code units, no case folding or normalisation), so a cache, `ReadStoreSnapshot` and a flush refuse the file and leave it as it is. Before, the later record silently replaced the earlier. A directory store reads its manifest through the same reader and refuses a duplicate key the same way. Repair the duplicate with the runtime that wrote it before reading the store here.
- A record whose payload carries a slot past the slots its reading type declares, at any depth (the record, a nested object, a list element, a union arm or a unit arm), is loaded read-only: it is served, and every staged write or removal of its key (`Upsert`, `UpsertVariant`, `Remove`, a batch, and `Flatten` of a variant) is refused with `CultSchemaConflictException` before anything is staged, and the store keeps the record byte for byte. Before, the extra slots were skipped on read and dropped on the next write. Declaring an id compatible consents only to depth-0 slots the schema comparison reports ignored under that id; any other extra slot keeps the record read-only whatever is declared, including under the reader's own id. A variant overriding a slot its type does not have is read-only too (it used to be shed on flush). Write such a record from the build that declares those slots. Missing trailing slots still default and stay writable. - A record whose payload carries a slot past the slots its reading type declares, at any depth (the record, a nested object, a list element, a union arm or a unit arm), is loaded read-only: it is served, and every staged write or removal of its key (`Upsert`, `UpsertVariant`, `Remove`, a batch, and `Flatten` of a variant) is refused with `CultSchemaConflictException` before anything is staged, and the store keeps the record byte for byte. Before, the extra slots were skipped on read and dropped on the next write. Declaring an id compatible consents only to depth-0 slots the schema comparison reports ignored under that id; any other extra slot keeps the record read-only whatever is declared, including under the reader's own id. A variant overriding a slot its type does not have is read-only too (it used to be shed on flush). Write such a record from the build that declares those slots. Missing trailing slots still default and stay writable.