Compare commits

..
Author SHA1 Message Date
Libby
1f9fffee83 Merge origin/main into soft-flush: JSONL push_all keeps the one _rewrite entry (refuse the link, read, decide soft)
All checks were successful
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Successful in 24m23s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-13 00:02:50 +00:00
Libby
8df0c4b25a Soft flush: pin the host and node passthroughs with a held record (absent writes, readable skips, unreadable refuses); XML-doc the one meaning on the three FlushAsync(bool soft) overloads
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 23:23:31 +00:00
Libby
80eed16c0d Soft flush: the reader's own answer decides (no second stat, protocol default included); pin the readable-empty store, spy the flag through Mesh and host, changelog the breaking PushAll
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 23:13:57 +00:00
Libby
fb80e32749 Soft flush: tests for the survivors of the hand mutations (directory store, JSONL and protocol default, host and node passthrough, plain PushAll); revert the benchmark.py touch
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
The InMemoryBackingStore in cultnet-py benchmark.py has no soft caller (the Python cache never passes soft), so the keyword there was unowned.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 22:44:01 +00:00
Libby
0a1486eccd C#: soft flush skips a readable store and keeps staged writes dirty; Mesh and CultNetLocal pass the flag
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
CacheBackingStore.PushAll(bool soft) is the one abstract owner; PushAll() stays as the plain call.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 22:33:01 +00:00
Libby
4e1ba0eb82 cultcache-py: push_all(soft) skips a readable store, writes an absent one, refuses an unreadable one
benchmark.py's InMemoryBackingStore implements the BackingStore protocol and takes the new keyword.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 22:30:37 +00:00
57 changed files with 208 additions and 5240 deletions

View file

@ -158,7 +158,7 @@ jobs:
# (only the interop example built, inside the C# store tests). The target directory is under RUNNER_TEMP, away from the checkout.
- name: Verify the CultCache document owner with the Rust cache
shell: bash
run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml
run: CARGO_TARGET_DIR="$RUNNER_TEMP/cultcache-rs-target" cargo test --manifest-path packages/cultcache-rs/Cargo.toml
# tests/GameCult.Caching.Tests, the C# reference cache's own suite (about 1,260 tests, under a minute), ran in no
# forge workflow; only the Mesh tests above exercised the reference.

View file

@ -135,10 +135,10 @@ jobs:
python -m unittest discover -s packages/cultmesh-py/tests
- name: Verify Rust RUDP reliability
run: cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml rudp
run: cargo test --manifest-path packages/cultnet-rs/Cargo.toml rudp
- name: Verify Rust CultMesh runtime-local surface
run: cargo test --locked --manifest-path packages/cultmesh-rs/Cargo.toml
run: cargo test --manifest-path packages/cultmesh-rs/Cargo.toml
- name: Verify C# RUDP reliability
run: dotnet test tests/GameCult.Networking.Tests/GameCult.Networking.Tests.csproj --filter "FullyQualifiedName~Rudp"

View file

@ -212,8 +212,8 @@ identity-owned C# session after provider route replacement.
Rust package tests:
```powershell
cargo test --locked --manifest-path packages/cultcache-rs/Cargo.toml
cargo test --locked --manifest-path packages/cultnet-rs/Cargo.toml
cargo test --manifest-path packages/cultcache-rs/Cargo.toml
cargo test --manifest-path packages/cultnet-rs/Cargo.toml
```
Kotlin package build:

View file

@ -3,7 +3,7 @@
CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local
tool in `dotnet-tools.json`. Coverage is added project by project; a project
with no `stryker-config.json` next to its test project has not been scoped
yet (see "Scope and follow-ups" below for what's outstanding).
yet (see "Follow-ups" below for what's outstanding).
## Running it
@ -126,17 +126,14 @@ project as precedent for the `vstest` caveat itself - that caveat is
`mtp`-runner-specific and documented above for `static readonly` field
initializers, and GameCult.Geometry has never actually hit it.
## Scope and follow-ups
## Follow-ups
CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the
projects currently scoped. GameCult.Geometry's `stryker-config.json` has no
`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and
CultMath.Tests and GameCult.Geometry.Tests are the only projects currently
scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner`
entry: the project is NUnit 4 with `NUnit3TestAdapter` and
`Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies
directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3
suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`)
and its `stryker-config.json` is driven the same way, by the repo-local
`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run
dotnet-stryker` from `tests/GameCult.Caching.Tests`.
suite).
GameCult.Geometry has only ever been run `--since:`-scoped to the cut that
added surface nets; a full-project baseline (running mutation against all of
@ -154,83 +151,7 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that
run; a future cut whose diff is C#-only, against a base commit that already
has `stryker-config.json`, should scope correctly.
### GameCult.Caching.Tests
`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of
`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard
mutation level, `break` 0, the same thresholds as Geometry). Caching was
scoped before Networking because its suite is deterministic and quick (1415
tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking
is about 20k lines and its tests lean on socket timing, so a mutant timing out
there says little about the test. A full Caching run is about 2,900 candidate
mutants (3,602 created, minus 730 that fail to compile) at 1.25 to 1.6 s
each, so about 1 to 1.3 hours; run it detached and read the JSON report,
never as one attached session.
The first measurement was a narrow pass over the three files that carry the
store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`:
```bash
dotnet tool run dotnet-stryker \
-m '**/src/GameCult.Caching/StoreReplace.cs' \
-m '**/src/GameCult.Caching/CultStoreLinks.cs' \
-m '**/src/GameCult.Caching/CultElementIds.cs'
```
A clean single run of that command tests 243 to 265 mutants (the count moves with the revision) and scored 85.28% on
568a60fa, before any test of this scope was added; 86.79% on d38b5728, with
the first batch of tests; and 87.17% on 76d9e7d1, with the throwing-enumerator
case, in about 10 minutes. The first measurement of it overlapped two Stryker
runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`,
where tests that kill a mutant by hand reported `Survived` or `Timeout`) are
not measurements and are not quoted here. Never run two Stryker runs against
one checkout at once, and check `ps` before starting one. A clean single run
on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208
`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor
was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now
kills; the other 11 are equivalent mutants. With that case a clean single run
on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%.
Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each
failing when its mutant is applied by hand): the order of the id path (the
codec refuses at the first type MessagePack cannot allow, so order is what an
error names first; `Append` to `Prepend` at 163, 167 and 180); the
union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so
the red is a crashed test host: a depth guard would need production code in
`ConcreteTypes`, since the recursion is private and a stack overflow cannot be
caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an
id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the
`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter,
`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the
`[DataContract]` without `[MessagePackObject]` and `||` arms).
Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full
suite; none can be told apart by any test:
- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to
`true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument
changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument
(`DataContractAttribute` is `Inherited = false`).
- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which
are never empty.
- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial
capacity.
- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`;
`Classify` unwraps nullables itself, so the unwrap is dead. A cut
candidate, not a test gap.
- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and
`MapTypes`; `type` matches only when it is itself `IEnumerable<>`,
`IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second
matching interface, so the position of `type` never decides the answer.
The Windows-only P/Invoke group is exercised only on Windows: its mutants are
`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a
Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its
negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the
negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the
same result.
`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
`GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
`GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes
(`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own
`stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for

View file

@ -16,7 +16,7 @@ this page.
| | CultCache | CultMath | Geometry | CultNet | CultMesh |
| --- | --- | --- | --- | --- | --- |
| **C#** | [first call, tested](#c-first-call); [update, tested](#c-update-with-retry) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` |
| **C#** | [first call, tested](#c-first-call) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` |
| **TypeScript** | [first call, tested](#typescript-first-call) | not yet | not yet | [first call, tested](#typescript-cultnet-first-call); [package](../../packages/cultnet-ts/README.md) | [package](../../packages/cultmesh-ts/README.md); snippet not yet; [recipe: two-client .io game](io-game.md), tested with two live clients, browser page not yet run |
| **Rust** | [first call, tested](#rust-first-call) | [native core](../../packages/cultmath/native/cultmath-core); snippet not yet | not yet | [first call, tested](#rust-cultnet-first-call); [package](../../packages/cultnet-rs/README.md), [examples](../../packages/cultnet-rs/examples) | [package](../../packages/cultmesh-rs); snippet not yet |
| **Python** | [first call, tested](#python-first-call) | not yet | not yet | [first call, tested](#python-cultnet-first-call); [package](../../packages/cultnet-py/README.md) | [package](../../packages/cultmesh-py/README.md); snippet not yet |
@ -70,37 +70,6 @@ public static class FirstCall
`FirstCall.Run` is the call; a second test in that file fails if the block above stops matching it. More: `src/GameCult.Caching/README.md`.
## C# update with retry
Change a record you may not be the only one changing. `Update` reads the record, hands your function a private
copy, and commits the result only if the record is still the one it read. If another writer got there first, it
reads again and runs your function again, up to five times. The source is
[`tests/GameCult.Caching.Tests/UpdateCallTests.cs`](../../tests/GameCult.Caching.Tests/UpdateCallTests.cs); the
snippet reuses `Item` from the first call.
```csharp
using GameCult.Caching;
using GameCult.Caching.MessagePack;
public static class UpdateCall
{
public static CultUpdateResult<Item> Restock(string path, CultRecordKey key)
{
var cache = CultCacheMessagePack.Create(path);
return cache.Update<Item>(key, item =>
{
item ??= new Item { Name = "Potion", Category = "Consumable" };
item.Value += 10;
return item;
});
}
}
```
The write is durable when `Update` returns, with no `PushAll`. A lost race re-runs the function, so keep it free of
side effects; when the attempts run out the result says `LostRace` instead of throwing. `UpsertAsync` plus
`PushAll` is the blind opt-out: last writer wins.
## TypeScript first call
Same shape in TypeScript: the document is a name, a schema id and a `parse`-able schema (zod here),
@ -155,7 +124,7 @@ The test imports from `../src/index` where the block reads `@gamecult/cultcache-
Same shape with `derive(DatabaseEntry)`: slots are `#[cultcache(key = N)]`, so the `.cc` payload is the
slot-indexed array the C# runtime writes. The source is
[`packages/cultcache-rs/tests/first_call.rs`](../../packages/cultcache-rs/tests/first_call.rs), run by
`cargo test --locked --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block
`cargo test --test first_call` in `packages/cultcache-rs`; a second test in that file fails if the block
below stops matching the file. The Rust runtime has no name lookup, so the read-back is by key. More:
`packages/cultcache-rs/README.md`.
@ -372,7 +341,7 @@ async function firstCall() {
### Rust CultNet first call
Source: [`packages/cultnet-rs/tests/first_call.rs`](../../packages/cultnet-rs/tests/first_call.rs), run by
`cargo test --locked --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`.
`cargo test --test first_call` in `packages/cultnet-rs`. More: `packages/cultnet-rs/README.md`.
```rust
use std::net::{TcpListener, TcpStream};

View file

@ -8,16 +8,12 @@
### Changed
- `store_write_lock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. `JsonLinesBackingStore` takes no cross-process lock (only its thread lock, under which it already refuses a link), so it has no lock wait to re-check across. Pinned by a test that puts the link while a foreign process holds the lock.
- `CultCache.is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
- `JsonLinesBackingStore` writes LF line endings on Windows. It wrote CRLF there, through `write_text` newline translation, before the commit_store_file change (#154, adbbb780).
- `msgpack` is a hard dependency, as `pyproject.toml` declared it. The optional-import fallbacks and the `cultcache-py[msgpack]` extra they named (which never existed) are gone: a missing msgpack is an `ImportError` at import.
- `JsonLinesBackingStore` writes through the same `commit_store_file` as the single-file store: a unique staged file, fsynced before the replace and removed when the replace fails, so a refused replace no longer leaves `<path>.tmp` behind. It staged a fixed `<path>.tmp` without an fsync.
- The single-file store's `push`, `push_all` and `delete` take the cross-process lock on `<path>.lock` (flock; LockFileEx over the whole file on Windows) across the read and the replace, waiting up to `lock_timeout` seconds (30 by default, `TimeoutError` after), and exclude the C#, Rust and TypeScript runtimes' writers. The staged file is `<path>.tmp-<pid>-<uuid>`, fsynced before the replace, with a directory fsync after it off Windows. A fixed `<path>.tmp` is no longer used.
### Breaking
- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol's default `push_all` cannot tell an absent store from an empty one (a pull returns a list), so a soft push through it always skips a store whose pull returns; a store kind that can tell nothing-at-the-path overrides `push_all`. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
- A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work.
- A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`.
- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#.

View file

@ -67,19 +67,12 @@ class BackingStore(Protocol):
def delete(self, type: str, key: str) -> None:
...
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
"""The envelopes the reader reads (empty included), None when nothing is at the path; raises when the path cannot be read.
The default is pull_all, which cannot say absent: a store that inherits it is always a store the reader read. A store kind
that can tell nothing-at-the-path overrides this, and the default push_all then writes where nothing is on a soft push."""
return self.pull_all()
def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None:
# Soft: write only where pull_or_absent says nothing is; skip over a store it read, empty included; its raise refuses.
pulled = self.pull_or_absent()
if soft and pulled is not None:
# A pull that returns is a store the reader read, empty included: the protocol cannot say "absent", so a soft push
# through this default always skips. A store that can tell nothing-at-the-path overrides push_all.
pulled = self.pull_all()
if soft:
return
pulled = pulled or []
existing = {(envelope.type, envelope.key): envelope for envelope in pulled}
for envelope in envelopes:
existing[(envelope.type, envelope.key)] = envelope

View file

@ -636,8 +636,7 @@ class CultCache:
a record held read-only (its payload carried slots past the declared ones, at any depth, and nothing consents to the loss, so
writing it would drop data a newer writer stored), for a key a store carries foreign, and for a key held as another type's
record this document does not own. Advisory (the write still decides): it answers at call time, and a key nothing holds is
writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it with
`StoreLinkedError`."""
writable."""
self._assert_registered(document)
try:
self._refuse_at(document, key)

View file

@ -78,9 +78,6 @@ class JsonLinesBackingStore:
def pull_all(self) -> list[CultCacheEnvelope]:
return self._pull() or []
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
return self._pull()
def _pull(self) -> list[CultCacheEnvelope] | None:
"""The envelopes the reader reads, or None when nothing is at the path."""
with self._lock:
@ -133,6 +130,7 @@ class JsonLinesBackingStore:
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
temp = self.path.with_suffix(self.path.suffix + ".tmp")
lines = []
for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)):
lines.append(json.dumps({
@ -141,7 +139,8 @@ class JsonLinesBackingStore:
"payload": base64.b64encode(envelope.payload).decode("ascii"),
"stored_at": envelope.stored_at,
}, ensure_ascii=False, sort_keys=True))
commit_store_file(self.path, ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8"))
temp.write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8")
_replace_store(temp, self.path)
class SingleFileMessagePackBackingStore:
@ -155,9 +154,6 @@ class SingleFileMessagePackBackingStore:
def pull_all(self) -> list[CultCacheEnvelope]:
return self._pull() or []
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
return self._pull()
def _pull(self) -> list[CultCacheEnvelope] | None:
"""The envelopes the reader reads, or None when nothing is at the path."""
with self._lock:
@ -501,8 +497,6 @@ def store_write_lock(path: Path, timeout: float = LOCK_TIMEOUT_SECONDS) -> Itera
raise TimeoutError(f"CultCache timed out after {timeout} s waiting for the store lock {lock_path}.")
time.sleep(_LOCK_POLL_SECONDS)
try:
# A link may have been put at the path while the lock was awaited; the look above only saw the path before.
_refuse_linked(path)
yield
finally:
release()

View file

@ -1485,32 +1485,6 @@ class CultCacheTests(unittest.TestCase):
self.assertEqual(sorted(p.name for p in Path(tmp).iterdir()), ["link.jsonl", "volume"])
self.assertEqual(sorted(p.name for p in volume.iterdir()), ["store.jsonl"])
# The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse.
@unittest.skipIf(os.name != "posix", "symbolic links need POSIX here")
def test_is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
volume = Path(tmp) / "volume"
volume.mkdir()
target = volume / "store.cc"
document = item_doc()
seed = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(target)).build()
seed.put(document, "held", Item(name="Held", category="c", value=1))
before = target.read_bytes()
link = Path(tmp) / "link.cc"
link.symlink_to(target)
cache = CultCache.builder().register_document_type(document).add_generic_store(SingleFileMessagePackBackingStore(link)).build()
cache.pull_all_backing_stores()
names_before = (sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir()))
self.assertTrue(cache.is_writable(document, "held"))
self.assertTrue(cache.is_writable(document, "fresh"))
with self.assertRaises(StoreLinkedError) as caught:
cache.put(document, "fresh", Item(name="Fresh", category="d", value=2))
self.assertEqual(Path(caught.exception.path), link)
self.assertTrue(link.is_symlink())
self.assertEqual(target.read_bytes(), before)
self.assertEqual((sorted(p.name for p in Path(tmp).iterdir()), sorted(p.name for p in volume.iterdir())), names_before)
def test_store_linked_error_is_an_os_error_carrying_the_path(self) -> None:
error = StoreLinkedError(Path("some/store.cc"))
self.assertIsInstance(error, OSError)

View file

@ -69,33 +69,8 @@ class SoftPushAllTests(unittest.TestCase):
JsonLinesBackingStore(self.path).push_all([envelope("a")], soft=True)
self.assertEqual(junk, self.path.read_bytes())
def test_the_protocol_default_push_all_decides_soft_by_what_pull_or_absent_says(self) -> None:
def test_the_protocol_default_push_all_skips_what_its_pull_read_empty_included(self) -> None:
class Memory(BackingStore):
def __init__(self) -> None:
self.items: list[CultCacheEnvelope] | None = None # None: nothing at the path
def pull_all(self) -> list[CultCacheEnvelope]:
return list(self.items or [])
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
return None if self.items is None else list(self.items)
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.items = list(envelopes)
store = Memory()
store.push_all([envelope("a")], soft=True) # nothing at the path: writes
self.assertEqual(["a"], [item.key for item in store.items or []])
store.push_all([envelope("b")], soft=True) # a store with records: skips
self.assertEqual(["a"], [item.key for item in store.items or []])
store.items = []
store.push_all([envelope("b")], soft=True) # a store the reader read, empty: skips
self.assertEqual([], store.items)
store.push_all([envelope("b")]) # plain: lands
self.assertEqual(["b"], [item.key for item in store.items])
def test_a_store_that_only_pulls_all_is_read_so_a_soft_push_skips_it(self) -> None:
class PullOnly(BackingStore):
def __init__(self) -> None:
self.items: list[CultCacheEnvelope] = []
@ -105,18 +80,14 @@ class SoftPushAllTests(unittest.TestCase):
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.items = list(envelopes)
store = PullOnly()
store.push_all([envelope("a")], soft=True)
store = Memory()
store.push_all([envelope("a")], soft=True) # the pull read an empty store: skips
self.assertEqual([], store.items)
def test_the_single_file_and_jsonl_stores_say_absent_and_read_through_pull_or_absent(self) -> None:
for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
with self.subTest(make.__name__):
path = self.dir / make.__name__
store = make(path)
self.assertIsNone(store.pull_or_absent())
store.push_all([envelope("a")])
self.assertEqual(["a"], [item.key for item in store.pull_or_absent() or []])
store.push_all([envelope("a")]) # plain: lands
store.push_all([envelope("b")], soft=True) # a store with records: skips
self.assertEqual(["a"], [item.key for item in store.items])
store.push_all([envelope("b")])
self.assertEqual(["a", "b"], sorted(item.key for item in store.items))
def test_the_protocol_default_push_all_refuses_where_its_pull_refuses(self) -> None:
class Unreadable(BackingStore):
@ -125,9 +96,6 @@ class SoftPushAllTests(unittest.TestCase):
def pull_all(self) -> list[CultCacheEnvelope]:
raise StoreUnreadableError(Path("x"), ValueError("junk"))
def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
raise StoreUnreadableError(Path("x"), ValueError("junk"))
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
Unreadable.replaced = True

View file

@ -1,25 +1,21 @@
from __future__ import annotations
import os
import select
import shutil
import subprocess
import sys
import tempfile
import threading
import time
import unittest
from pathlib import Path
from unittest import mock
from cultcache_py import SingleFileMessagePackBackingStore, stores
from cultcache_py.stores import StoreLinkedError, store_write_lock
from cultcache_py.backing_store import CultCacheEnvelope
REPO = Path(__file__).resolve().parents[3]
TS_STORE_FILE = REPO / "packages" / "cultcache-ts" / "dist" / "store-file.js"
HOLD_SECONDS = 1.5
HELD_WAIT_SECONDS = 3
# A holder that takes the lock the way the Rust runtime does (an exclusive file lock), not through this package.
PY_HOLDER = """
@ -45,14 +41,7 @@ def envelope(key: str) -> CultCacheEnvelope:
def start_holder(argv: list[str]) -> subprocess.Popen[str]:
holder = subprocess.Popen(argv, stdout=subprocess.PIPE, text=True)
assert holder.stdout is not None
ready, _, _ = select.select([holder.stdout], [], [], HELD_WAIT_SECONDS) # a holder blocked on a leaked lock never prints
if not ready:
holder.kill()
holder.wait()
holder.stdout.close()
raise AssertionError(f"the holder never took {argv[-2]}.lock within {HELD_WAIT_SECONDS} s: an earlier writer leaked the lock")
assert holder.stdout.readline().strip() == "HELD"
assert holder.stdout is not None and holder.stdout.readline().strip() == "HELD"
holder.held_at = time.monotonic() # type: ignore[attr-defined]
return holder
@ -84,48 +73,6 @@ class StoreLockTests(unittest.TestCase):
holder = start_holder(["node", "-e", TS_HOLDER, str(TS_STORE_FILE), str(self.path), str(HOLD_SECONDS)])
self.assert_write_waits_for(holder)
def test_a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it(self) -> None:
target = self.dir / "volume" / "store.cc"
target.parent.mkdir()
SingleFileMessagePackBackingStore(target).push(envelope("a"))
before = target.read_bytes()
writes = {
"push": lambda store: store.push(envelope("b")),
"push_all": lambda store: store.push_all([envelope("b")]),
"delete": lambda store: store.delete("t", "a"),
}
missing = self.dir / "volume" / "missing.cc"
for (name, write), link_to in [(item, to) for item in writes.items() for to in (target, missing)]:
with self.subTest(name, dangling=link_to == missing):
if os.path.lexists(self.path):
os.unlink(self.path)
holder = start_holder([sys.executable, "-c", PY_HOLDER, str(self.path), str(HOLD_SECONDS)])
self.addCleanup(holder.wait)
self.addCleanup(holder.stdout.close) # type: ignore[union-attr]
outcome: list[BaseException | None] = []
def run() -> None:
try:
write(SingleFileMessagePackBackingStore(self.path, lock_timeout=10))
outcome.append(None)
except BaseException as exc: # noqa: BLE001 - the thread hands the outcome to the test
outcome.append(exc)
writer = threading.Thread(target=run)
writer.start()
time.sleep(0.3)
os.symlink(link_to, self.path) # after the writer's pre-lock look, while it waits
writer.join()
holder.wait()
self.assertEqual(1, len(outcome))
self.assertIsInstance(outcome[0], StoreLinkedError)
self.assertTrue(os.path.islink(self.path), "the link was replaced")
self.assertEqual(before, target.read_bytes())
self.assertFalse(missing.exists(), "the dangling target was created")
os.unlink(self.path) # a link is refused before the lock, so remove it to probe the writer's own lock file
with store_write_lock(self.path, timeout=0.2): # the lock is free
pass
def test_the_lock_is_released_after_each_write(self) -> None:
store = SingleFileMessagePackBackingStore(self.path, lock_timeout=0.2)
for key in "abc":

View file

@ -107,19 +107,6 @@ class StoreWritesRouteThroughTheReplace(unittest.TestCase):
self.assertEqual([destination for _, destination in seen], [binary, lines])
self.assertTrue(binary.read_bytes() and lines.read_text())
def test_a_failed_jsonl_replace_leaves_no_staged_file_and_the_old_store(self):
envelope = CultCacheEnvelope(key="k", type="t", payload=b"\x01", stored_at="2026-11-06T00:00:00Z")
with tempfile.TemporaryDirectory() as directory:
lines = Path(directory) / "s.jsonl"
store = stores.JsonLinesBackingStore(lines)
store.push_all([envelope])
before = lines.read_bytes()
with mock.patch.object(stores, "_replace_store", side_effect=OSError("replace refused")):
with self.assertRaises(OSError):
store.push_all([CultCacheEnvelope(key="k2", type="t", payload=b"\x02", stored_at="2026-11-06T00:00:00Z")])
self.assertEqual(lines.read_bytes(), before)
self.assertEqual([p.name for p in Path(directory).iterdir() if ".tmp" in p.name], [])
@unittest.skipUnless(sys.platform == "win32", "needs a real Windows")
def test_real_kernel_replaces_through_both_primitives(self):
kernel = stores.WindowsReplaceKernel()

View file

@ -1 +1,2 @@
/target
Cargo.lock

View file

@ -6,11 +6,6 @@ only in git history.
## Unreleased
### Changed
- The single-file store re-checks for a link once the store lock is held (every write and the try-exchange), so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, as C# does, instead of being replaced. Pinned by a test that puts the link while a second holder owns the lock.
- `CultCache::is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
### Breaking
Semver: this is a breaking change in the 0.x line (0.3.x to 0.4.0).

View file

@ -1,730 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "android_system_properties"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc"
dependencies = [
"libc",
]
[[package]]
name = "anyhow"
version = "1.0.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]]
name = "ar_archive_writer"
version = "0.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "523e5e187fff36fcb8d1d19ccf0ca4ca43ec3d158316057f66dc6f72716bc5ec"
dependencies = [
"object",
]
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "bitflags"
version = "2.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
[[package]]
name = "bumpalo"
version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "cc"
version = "1.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ff2a8126fff0f8c839080d619675a585e61720bb12d2baceae45769fd66f155"
dependencies = [
"find-msvc-tools",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "chrono"
version = "0.4.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327"
dependencies = [
"iana-time-zone",
"js-sys",
"num-traits",
"serde",
"wasm-bindgen",
"windows-link",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "cultcache-rs"
version = "0.3.1"
dependencies = [
"anyhow",
"chrono",
"cultcache-rs-derive",
"fs2",
"pretty_assertions",
"redb",
"rmp-serde",
"serde",
"serde_bytes",
"serde_json",
"serde_stacker",
"stacker",
"tempfile",
"uuid",
"windows-sys 0.59.0",
]
[[package]]
name = "cultcache-rs-derive"
version = "0.2.0"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "diff"
version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8"
[[package]]
name = "errno"
version = "0.3.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776"
dependencies = [
"libc",
"windows-sys 0.61.2",
]
[[package]]
name = "fastrand"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
[[package]]
name = "find-msvc-tools"
version = "0.1.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48"
[[package]]
name = "fs2"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213"
dependencies = [
"libc",
"winapi",
]
[[package]]
name = "futures-core"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
[[package]]
name = "futures-task"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd"
[[package]]
name = "futures-util"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc"
dependencies = [
"futures-core",
"futures-task",
"pin-project-lite",
"slab",
]
[[package]]
name = "getrandom"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
"libc",
"r-efi",
]
[[package]]
name = "iana-time-zone"
version = "0.1.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470"
dependencies = [
"android_system_properties",
"core-foundation-sys",
"iana-time-zone-haiku",
"js-sys",
"log",
"wasm-bindgen",
"windows-core",
]
[[package]]
name = "iana-time-zone-haiku"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
dependencies = [
"cc",
]
[[package]]
name = "itoa"
version = "1.0.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "js-sys"
version = "0.3.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a"
dependencies = [
"cfg-if",
"futures-util",
"wasm-bindgen",
]
[[package]]
name = "libc"
version = "0.2.191"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a"
[[package]]
name = "linux-raw-sys"
version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "log"
version = "0.4.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d15f86d12da44cd8d4ec3c73984537eb7f4cadcb9776a575a0a8a05ca11005c"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "object"
version = "0.39.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b"
dependencies = [
"memchr",
]
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "pin-project-lite"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
[[package]]
name = "pretty_assertions"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d"
dependencies = [
"diff",
"yansi",
]
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "psm"
version = "0.1.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622"
dependencies = [
"ar_archive_writer",
"cc",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "redb"
version = "4.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b317ef681e18f6091efbfab532eaf924a135c492d9d0299d0fbfae07e92a39b"
dependencies = [
"libc",
]
[[package]]
name = "rmp"
version = "0.8.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c"
dependencies = [
"num-traits",
]
[[package]]
name = "rmp-serde"
version = "1.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155"
dependencies = [
"rmp",
"serde",
]
[[package]]
name = "rustix"
version = "1.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa"
dependencies = [
"bitflags",
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
]
[[package]]
name = "rustversion"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_bytes"
version = "0.11.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8"
dependencies = [
"serde",
"serde_core",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.10",
]
[[package]]
name = "serde_json"
version = "1.0.154"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "serde_stacker"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4936375d50c4be7eff22293a9344f8e46f323ed2b3c243e52f89138d9bb0f4a"
dependencies = [
"serde",
"serde_core",
"stacker",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "slab"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "stacker"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967"
dependencies = [
"cc",
"cfg-if",
"libc",
"psm",
"windows-sys 0.61.2",
]
[[package]]
name = "syn"
version = "2.0.119"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ae9ccbea3a7b0b09443036490c8f34a8b1b760fd345e5c0dae0aac718faea79"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "tempfile"
version = "3.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom",
"once_cell",
"rustix",
"windows-sys 0.61.2",
]
[[package]]
name = "unicode-ident"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563"
[[package]]
name = "uuid"
version = "1.29.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bed99139f7dac9b5cfe5f17be4d66dc7079a34dee1d74bcc77217119240e267c"
dependencies = [
"getrandom",
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "wasm-bindgen"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c"
dependencies = [
"cfg-if",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
]
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a"
dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn 3.0.10",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e"
dependencies = [
"unicode-ident",
]
[[package]]
name = "winapi"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
dependencies = [
"winapi-i686-pc-windows-gnu",
"winapi-x86_64-pc-windows-gnu",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windows-core"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [
"windows-implement",
"windows-interface",
"windows-link",
"windows-result",
"windows-strings",
]
[[package]]
name = "windows-implement"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "windows-interface"
version = "0.59.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-strings"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "yansi"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"

View file

@ -31,4 +31,4 @@ pretty_assertions = "1.4"
tempfile = "3"
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_Security", "Win32_System_WindowsProgramming"] }
windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }

View file

@ -480,39 +480,6 @@ struct PersistedSchemaCatalogEntry(
Vec<PersistedSchemaCatalogMember>,
);
/// One entry of a store file's schema catalog, read-only: the scalar fields every runtime publishes for a schema id. The members and
/// the canonical schema JSON the file also carries are not exposed. `schema_version` is the version string the writer published
/// (C# `CultSchemaCatalogEntry.SchemaVersion`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SchemaCatalogEntry {
pub schema_id: String,
pub schema_name: String,
pub schema_version: String,
pub content_hash: String,
pub compatible_schema_ids: Vec<String>,
}
impl From<&PersistedSchemaCatalogEntry> for SchemaCatalogEntry {
fn from(entry: &PersistedSchemaCatalogEntry) -> Self {
Self {
schema_id: entry.0.clone(),
schema_name: entry.1.clone(),
schema_version: entry.2.clone(),
content_hash: entry.3.clone(),
compatible_schema_ids: entry.5.clone(),
}
}
}
/// The catalog entry a record's schema id names, as the decoder resolves it: the first entry that has the id as its own, else the
/// first entry that lists it as compatible; `None` when the catalog publishes neither.
pub fn resolve_schema_entry<'a>(catalog: &'a [SchemaCatalogEntry], schema_id: &str) -> Option<&'a SchemaCatalogEntry> {
catalog
.iter()
.find(|entry| entry.schema_id == schema_id)
.or_else(|| catalog.iter().find(|entry| entry.compatible_schema_ids.iter().any(|id| id == schema_id)))
}
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
struct PersistedSchemaCatalogMember(
u32,
@ -1730,16 +1697,6 @@ impl SingleFileMessagePackBackingStore {
self.read_all_unlocked()
}
/// The schema catalog the store file publishes, read under the same shared lock [`CacheBackingStore::pull_all`] takes. An absent
/// file and a legacy envelope array have no catalog and give an empty one. Resolve a record's `schema_id` against it with
/// [`resolve_schema_entry`].
pub fn pull_schema_catalog(&self) -> Result<Vec<SchemaCatalogEntry>> {
if is_absent(&self.path)? && is_absent(&self.lock_path())? {
return Ok(Vec::new());
}
self.with_shared_lock(|| Ok(self.read_store_unlocked()?.catalog.iter().map(SchemaCatalogEntry::from).collect()))
}
fn write_all_unlocked(&self, entries: &[CultCacheEnvelope]) -> Result<()> {
let failed = |kind| CultCacheStoreWriteFailed {
path: self.path.clone(),
@ -1795,7 +1752,7 @@ impl SingleFileMessagePackBackingStore {
let lock = self.open_lock_file()?;
fs2::FileExt::lock_exclusive(&lock)
.with_context(|| format!("failed to lock {}", self.lock_path().display()))?;
self.run_locked(lock, action)
release_lock(lock, action())
}
/// Replaces the named entries only if each still matches its expectation.
@ -1904,11 +1861,7 @@ impl SingleFileMessagePackBackingStore {
fn try_with_exclusive_lock<T>(&self, action: impl FnOnce() -> Result<T>) -> Result<Option<T>> {
refuse_linked(&self.path)?;
#[cfg(test)]
run_after_pre_lock_look();
let Some(lock) = self.try_open_lock_file()? else {
return Ok(None);
};
let lock = self.open_lock_file()?;
match fs2::FileExt::try_lock_exclusive(&lock) {
Ok(()) => {}
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => return Ok(None),
@ -1917,40 +1870,22 @@ impl SingleFileMessagePackBackingStore {
.with_context(|| format!("failed to lock {}", self.lock_path().display()));
}
}
self.run_locked(lock, action).map(Some)
release_lock(lock, action()).map(Some)
}
/// The one tail of both exclusive entries: with the lock held, look for a link again (one may have been put at the
/// path while the lock was awaited; the look before the lock saw the path earlier), run the action, release.
fn run_locked<T>(&self, lock: File, action: impl FnOnce() -> Result<T>) -> Result<T> {
release_lock(lock, refuse_linked(&self.path).and_then(|()| action()))
}
/// Opens the lock file, waiting through a Windows sharing violation the way the lock itself waits.
fn open_lock_file(&self) -> Result<File> {
let lock_path = self.prepare_lock_dir()?;
retry_while_busy(
|| open_lock_once(&lock_path),
is_sharing_violation,
|| std::thread::sleep(std::time::Duration::from_millis(5)),
)
.with_context(|| format!("failed to open {}", lock_path.display()))
}
/// The single-attempt open for the try entry: a sharing violation is `None`, the same answer as a held lock.
fn try_open_lock_file(&self) -> Result<Option<File>> {
let lock_path = self.prepare_lock_dir()?;
open_unless_busy(|| open_lock_once(&lock_path), is_sharing_violation)
.with_context(|| format!("failed to open {}", lock_path.display()))
}
fn prepare_lock_dir(&self) -> Result<PathBuf> {
let lock_path = self.lock_path();
if let Some(parent) = lock_path.parent() {
fs::create_dir_all(parent)
.with_context(|| format!("failed to create {}", parent.display()))?;
}
Ok(lock_path)
OpenOptions::new()
.create(true)
.read(true)
.truncate(false)
.write(true)
.open(&lock_path)
.with_context(|| format!("failed to open {}", lock_path.display()))
}
fn lock_path(&self) -> PathBuf {
@ -3374,8 +3309,7 @@ impl CultCache {
/// Whether a write or removal of `(type_id, key)` would be admitted: false when the held payload carries slots past
/// its registered type's declared slots, at any depth, or cannot be read by this build at all (the record is
/// read-only), when the cache holds no image of its stores (see [`CultCacheLoadRefused`]), when a store carries a foreign record at the key, or when another registered type holds it. True for a
/// key nothing holds. An unregistered type is an error. It does not look at store paths: a link at a store path is not
/// predicted, and the write refuses it with [`StoreLinkedError`]. Advisory: the write entries ask the same checks themselves.
/// key nothing holds. An unregistered type is an error. Advisory: the write entries ask the same checks themselves.
pub fn is_writable(&self, type_id: &str, key: &str) -> Result<bool> {
if self.require_loaded().is_err() {
return Ok(false);
@ -4462,324 +4396,61 @@ fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> {
#[cfg(windows)]
fn replace_file_atomically(staged: &Path, destination: &Path) -> Result<()> {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::{
MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW,
};
injected_write_fault(WriteStep::Rename).context("failed to atomically replace the store")?;
replace_with(&WindowsKernel, staged, destination)
}
/// The two Windows replace primitives, each answering 0 or a Win32 error code, and the wait
/// between attempts. `replace_with` owns every decision about them.
#[cfg_attr(not(windows), allow(dead_code))]
trait ReplaceKernel {
/// CreateFileW(staged, DELETE|SYNCHRONIZE), then FileRenameInfoEx with
/// REPLACE_IF_EXISTS|POSIX_SEMANTICS, then CloseHandle.
fn rename_posix(&self, staged: &Path, destination: &Path) -> u32;
/// MoveFileExW(staged, destination, flags).
fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32;
fn sleep(&self, milliseconds: u64);
}
/// MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH.
#[cfg_attr(not(windows), allow(dead_code))]
const MOVEFILE_REPLACE_WRITE_THROUGH: u32 = 0x9;
/// Replaces `destination` with `staged` on Windows: a POSIX-semantics rename first, so a reader
/// that opened the old store keeps it and the destination is rarely seen absent (a stalled rename
/// can still leave it absent for 5-47 ms on NTFS: `variants:ruling:posix-rename-absent-window-document`);
/// MoveFileExW with
/// write-through where POSIX semantics are unsupported (FAT, exFAT, before Windows 10 1607),
/// for the rest of the call; and a bounded retry on access, sharing and lock violations.
/// A concurrent reader (another process, an indexer, an antivirus scan) makes the replace fail
/// transiently, and a daemon that treats that as fatal dies on the first overlap (Muninn serve
/// did, 2026-09-11). Ruling `variants:ruling:windows-replace-posix-first`: cultcache-ts carries the
/// same table on main; cultcache-cs carries it only once its pending PR lands.
#[cfg_attr(not(windows), allow(dead_code))]
fn replace_with(kernel: &dyn ReplaceKernel, staged: &Path, destination: &Path) -> Result<()> {
const ERROR_INVALID_FUNCTION: u32 = 1;
const ERROR_ACCESS_DENIED: u32 = 5;
const ERROR_SHARING_VIOLATION: u32 = 32;
const ERROR_LOCK_VIOLATION: u32 = 33;
const ERROR_NOT_SUPPORTED: u32 = 50;
const ERROR_INVALID_PARAMETER: u32 = 87;
let staged_wide = staged
.as_os_str()
.encode_wide()
.chain(std::iter::once(0))
.collect::<Vec<_>>();
let destination_wide = destination
.as_os_str()
.encode_wide()
.chain(std::iter::once(0))
.collect::<Vec<_>>();
// A concurrent reader of the destination (another process opening the
// store, an indexer, an antivirus scan) makes the replace fail with a
// sharing or access error for as long as it holds the handle. That is
// transient by construction; a daemon that treats it as fatal dies on
// the first overlap (Muninn serve did, 2026-09-11). Retry briefly.
const ERROR_ACCESS_DENIED: i32 = 5;
const ERROR_SHARING_VIOLATION: i32 = 32;
const ERROR_LOCK_VIOLATION: i32 = 33;
const ATTEMPTS: u32 = 25;
const BACKOFF_MS: u64 = 20;
let mut posix = true;
let mut last_code = 0;
for attempt in 1..=ATTEMPTS {
let mut code = if posix {
kernel.rename_posix(staged, destination)
} else {
kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH)
const BACKOFF: std::time::Duration = std::time::Duration::from_millis(20);
let mut attempt = 0;
loop {
let success = unsafe {
MoveFileExW(
staged_wide.as_ptr(),
destination_wide.as_ptr(),
MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
};
if posix && matches!(code, ERROR_INVALID_FUNCTION | ERROR_NOT_SUPPORTED | ERROR_INVALID_PARAMETER) {
posix = false;
code = kernel.move_file_ex(staged, destination, MOVEFILE_REPLACE_WRITE_THROUGH);
}
if code == 0 {
if success != 0 {
return Ok(());
}
if !matches!(code, ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION) {
return Err(anyhow::anyhow!(
"failed to atomically replace {} with {}: Win32 error {code} after {attempt} attempt(s)",
destination.display(),
staged.display()
));
}
last_code = code;
kernel.sleep(BACKOFF_MS);
}
Err(anyhow::anyhow!(
"failed to atomically replace {} with {} after {ATTEMPTS} attempt(s): last Win32 error {last_code}",
destination.display(),
staged.display()
))
}
#[cfg(windows)]
struct WindowsKernel;
#[cfg(windows)]
fn wide(path: &Path) -> Vec<u16> {
use std::os::windows::ffi::OsStrExt;
path.as_os_str().encode_wide().chain(std::iter::once(0)).collect()
}
#[cfg(windows)]
impl ReplaceKernel for WindowsKernel {
fn rename_posix(&self, staged: &Path, destination: &Path) -> u32 {
use windows_sys::Win32::Foundation::{CloseHandle, GetLastError, INVALID_HANDLE_VALUE};
use windows_sys::Win32::Storage::FileSystem::{
CreateFileW, FileRenameInfoEx, SetFileInformationByHandle, DELETE, FILE_FLAG_BACKUP_SEMANTICS,
FILE_RENAME_INFO, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, OPEN_EXISTING, SYNCHRONIZE,
};
use windows_sys::Win32::System::WindowsProgramming::{
FILE_RENAME_FLAG_POSIX_SEMANTICS, FILE_RENAME_FLAG_REPLACE_IF_EXISTS,
};
let staged_wide = wide(staged);
let destination_wide = wide(destination);
// FILE_RENAME_INFO ends in a variable-length name: allocate it as u64s for alignment.
let name_bytes = (destination_wide.len() - 1) * 2;
let header = std::mem::size_of::<FILE_RENAME_INFO>();
let total = header + name_bytes + 2;
let mut buffer = vec![0u64; total.div_ceil(8)];
unsafe {
let info = buffer.as_mut_ptr() as *mut FILE_RENAME_INFO;
(*info).Anonymous.Flags = FILE_RENAME_FLAG_REPLACE_IF_EXISTS | FILE_RENAME_FLAG_POSIX_SEMANTICS;
(*info).RootDirectory = std::ptr::null_mut();
(*info).FileNameLength = name_bytes as u32;
let name = std::ptr::addr_of_mut!((*info).FileName) as *mut u16;
std::ptr::copy_nonoverlapping(destination_wide.as_ptr(), name, destination_wide.len());
let handle = CreateFileW(
staged_wide.as_ptr(),
DELETE | SYNCHRONIZE,
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
std::ptr::null(),
OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS,
std::ptr::null_mut(),
);
if handle == INVALID_HANDLE_VALUE {
return GetLastError();
}
let ok = SetFileInformationByHandle(handle, FileRenameInfoEx, info as *const _, total as u32);
let code = if ok == 0 { GetLastError() } else { 0 };
CloseHandle(handle);
code
}
}
fn move_file_ex(&self, staged: &Path, destination: &Path, flags: u32) -> u32 {
use windows_sys::Win32::Foundation::GetLastError;
use windows_sys::Win32::Storage::FileSystem::MoveFileExW;
let ok = unsafe { MoveFileExW(wide(staged).as_ptr(), wide(destination).as_ptr(), flags) };
if ok != 0 {
0
} else {
unsafe { GetLastError() }
}
}
fn sleep(&self, milliseconds: u64) {
std::thread::sleep(std::time::Duration::from_millis(milliseconds));
}
}
fn open_lock_once(lock_path: &Path) -> std::io::Result<File> {
OpenOptions::new()
.create(true)
.read(true)
.truncate(false)
.write(true)
.open(lock_path)
}
/// Windows ERROR_SHARING_VIOLATION (raw os error 32): another handle holds the lock file open without sharing. On Unix
/// errno 32 is EPIPE and is never a busy file.
#[cfg(windows)]
fn is_sharing_violation(error: &std::io::Error) -> bool {
error.raw_os_error() == Some(32)
}
#[cfg(not(windows))]
fn is_sharing_violation(_error: &std::io::Error) -> bool {
false
}
/// Repeats `open` while it fails busy, sleeping between tries and without a bound, as the lock waits; any other error surfaces.
fn retry_while_busy<T>(
mut open: impl FnMut() -> std::io::Result<T>,
busy: impl Fn(&std::io::Error) -> bool,
mut sleep: impl FnMut(),
) -> std::io::Result<T> {
loop {
match open() {
Err(error) if busy(&error) => sleep(),
other => return other,
}
}
}
/// One attempt: `None` when the open fails busy, any other error surfaces.
fn open_unless_busy<T>(
open: impl FnOnce() -> std::io::Result<T>,
busy: impl Fn(&std::io::Error) -> bool,
) -> std::io::Result<Option<T>> {
match open() {
Ok(file) => Ok(Some(file)),
Err(error) if busy(&error) => Ok(None),
Err(error) => Err(error),
}
}
#[cfg(test)]
mod replace_with_tests {
use super::{replace_with, ReplaceKernel};
use std::cell::RefCell;
use std::collections::VecDeque;
use std::path::Path;
#[derive(Debug, PartialEq)]
enum Call {
Posix,
MoveFile(u32),
Sleep(u64),
}
/// Answers each primitive from a script; an empty script answers 0.
#[derive(Default)]
struct FakeKernel {
posix: RefCell<VecDeque<u32>>,
move_file: RefCell<VecDeque<u32>>,
calls: RefCell<Vec<Call>>,
}
impl FakeKernel {
fn scripted(posix: &[u32], move_file: &[u32]) -> Self {
Self {
posix: RefCell::new(posix.iter().copied().collect()),
move_file: RefCell::new(move_file.iter().copied().collect()),
..Self::default()
}
}
fn calls(&self) -> Vec<Call> {
self.calls.replace(Vec::new())
}
}
impl ReplaceKernel for FakeKernel {
fn rename_posix(&self, _: &Path, _: &Path) -> u32 {
self.calls.borrow_mut().push(Call::Posix);
self.posix.borrow_mut().pop_front().unwrap_or(0)
}
fn move_file_ex(&self, _: &Path, _: &Path, flags: u32) -> u32 {
self.calls.borrow_mut().push(Call::MoveFile(flags));
self.move_file.borrow_mut().pop_front().unwrap_or(0)
}
fn sleep(&self, milliseconds: u64) {
self.calls.borrow_mut().push(Call::Sleep(milliseconds));
}
}
fn run(kernel: &FakeKernel) -> anyhow::Result<()> {
replace_with(kernel, Path::new("staged"), Path::new("store.cc"))
}
#[test]
fn a_posix_rename_that_lands_is_the_whole_replace() {
let kernel = FakeKernel::scripted(&[0], &[]);
run(&kernel).unwrap();
assert_eq!(kernel.calls(), vec![Call::Posix]);
}
#[test]
fn unsupported_posix_semantics_fall_back_to_write_through_and_stay_there() {
for code in [1, 50, 87] {
// The fallback itself is held once, so the retry must use MoveFileExW, not POSIX again.
let kernel = FakeKernel::scripted(&[code], &[32, 0]);
run(&kernel).unwrap_or_else(|e| panic!("code {code}: {e}"));
assert_eq!(
kernel.calls(),
vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)],
"code {code}"
);
}
}
#[test]
fn a_held_destination_is_retried_with_posix_until_it_lets_go() {
for code in [5, 32, 33] {
let kernel = FakeKernel::scripted(&[code, code, 0], &[]);
run(&kernel).unwrap();
assert_eq!(
kernel.calls(),
vec![Call::Posix, Call::Sleep(20), Call::Posix, Call::Sleep(20), Call::Posix],
"code {code}"
);
}
}
#[test]
fn a_destination_held_forever_fails_after_twenty_five_attempts_naming_the_code() {
for code in [5, 32, 33] {
let kernel = FakeKernel::scripted(&[code; 40], &[]);
let error = run(&kernel).unwrap_err().to_string();
let calls = kernel.calls();
assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 25);
assert_eq!(calls.iter().filter(|c| **c == Call::Sleep(20)).count(), 25);
assert!(error.contains("25 attempt(s)") && error.contains(&format!("error {code}")), "{error}");
}
}
#[test]
fn a_held_destination_after_fallback_exhausts_through_move_file_ex() {
let kernel = FakeKernel::scripted(&[87], &[32; 40]);
assert!(run(&kernel).is_err());
let calls = kernel.calls();
assert_eq!(calls.iter().filter(|c| **c == Call::Posix).count(), 1);
assert_eq!(calls.iter().filter(|c| matches!(c, Call::MoveFile(0x9))).count(), 25);
}
#[test]
fn any_other_code_fails_at_once_without_a_retry() {
for (posix, move_file) in [(&[2u32][..], &[][..]), (&[87][..], &[3][..])] {
let kernel = FakeKernel::scripted(posix, move_file);
let error = run(&kernel).unwrap_err().to_string();
assert!(!kernel.calls().contains(&Call::Sleep(20)), "{error}");
assert!(error.contains("1 attempt(s)"), "{error}");
}
}
#[test]
fn move_file_ex_answering_unsupported_is_an_error_not_a_cue_to_fall_back_again() {
let kernel = FakeKernel::scripted(&[87], &[32, 87]);
let error = run(&kernel).unwrap_err().to_string();
assert_eq!(
kernel.calls(),
vec![Call::Posix, Call::MoveFile(0x9), Call::Sleep(20), Call::MoveFile(0x9)],
"{error}"
let error = std::io::Error::last_os_error();
attempt += 1;
let transient = matches!(
error.raw_os_error(),
Some(ERROR_ACCESS_DENIED | ERROR_SHARING_VIOLATION | ERROR_LOCK_VIOLATION)
);
assert!(error.contains("error 87") && error.contains("2 attempt(s)"), "{error}");
if !transient || attempt >= ATTEMPTS {
return Err(error).with_context(|| {
format!(
"failed to atomically replace {} with {} after {attempt} attempt(s)",
destination.display(),
staged.display()
)
});
}
std::thread::sleep(BACKOFF);
}
}
@ -4788,35 +4459,6 @@ mod windows_replace_tests {
use super::replace_file_atomically;
use std::os::windows::fs::OpenOptionsExt;
const FILE_SHARE_ALL: u32 = 0x1 | 0x2 | 0x4; // READ | WRITE | DELETE
/// A reader that shares delete holds the old store open. A POSIX-semantics rename
/// replaces the name while the reader keeps the old content; MoveFileExW alone fails
/// there with error 5 (Soul's Starfire probe, build 26200), so this test passes only
/// if the POSIX path is live. Compile-checked only: it has never run on Windows.
#[test]
fn a_replace_succeeds_over_a_delete_sharing_reader_which_keeps_the_old_content() {
use std::io::Read;
let dir = std::env::temp_dir().join(format!("cultcache-posix-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let destination = dir.join("store.cc");
let staged = dir.join("store.cc.tmp");
std::fs::write(&destination, b"old").unwrap();
std::fs::write(&staged, b"new").unwrap();
let mut reader = std::fs::OpenOptions::new()
.read(true)
.share_mode(FILE_SHARE_ALL)
.open(&destination)
.unwrap();
replace_file_atomically(&staged, &destination).expect("POSIX rename over a sharing reader");
let mut seen = Vec::new();
reader.read_to_end(&mut seen).unwrap();
assert_eq!(seen, b"old", "the reader keeps the old content");
assert_eq!(std::fs::read(&destination).unwrap(), b"new");
drop(reader);
let _ = std::fs::remove_dir_all(&dir);
}
/// A reader holding the destination open with no sharing makes the
/// replace fail for as long as it holds on; it must succeed once the
/// reader lets go, not report the first refusal as the outcome.
@ -4879,20 +4521,6 @@ fn injected_write_fault(step: WriteStep) -> std::io::Result<()> {
Ok(())
}
// The one gap a test cannot hold open from outside: a try-lock never waits, so what happens between its pre-lock look and
// the lock is only reachable by running the test's closure there.
#[cfg(test)]
thread_local! {
static AFTER_PRE_LOCK_LOOK: std::cell::RefCell<Option<Box<dyn FnOnce()>>> = const { std::cell::RefCell::new(None) };
}
#[cfg(test)]
fn run_after_pre_lock_look() {
if let Some(hook) = AFTER_PRE_LOCK_LOOK.with(|hook| hook.borrow_mut().take()) {
hook();
}
}
#[cfg(not(test))]
#[inline(always)]
fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> {
@ -4901,8 +4529,7 @@ fn injected_write_fault(_step: WriteStep) -> std::io::Result<()> {
#[cfg(windows)]
fn sync_parent_directory(_path: &Path) -> Result<()> {
// Publication durability is the replace's: the MoveFileExW fallback writes through; the POSIX-semantics
// rename has no write-through flag and its durability is unmeasured (follow-up windows-replace-run-and-durability).
// MoveFileExW with MOVEFILE_WRITE_THROUGH owns publication durability.
Ok(())
}
@ -8302,21 +7929,11 @@ mod tests {
with_write_fault(WriteStep::Unlock, || store.push(&pushed))?;
assert!(with_write_fault(WriteStep::Unlock, || store.pull_all())?.contains(&pushed));
// The lock was released: another writer takes it. The shared lock of the faulted `pull_all` just above was left to the
// close of its descriptor (see `release_lock`), which frees it only once no other descriptor shares its open file
// description, and a test that spawns a child (a fork-exec duplicates every descriptor of the process until the exec)
// can hold such a duplicate on another test thread for a moment. A faulted exclusive lock cannot contend the try-lock:
// the blocking `pull_all` in its argument waits it out. So a contended try-lock is retried for up to two seconds; a
// lock that was never released stays contended and still fails.
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2);
let outcome = loop {
let outcome = store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?;
if !matches!(outcome, TryCompareExchangeSnapshotOutcome::LockContended) || std::time::Instant::now() >= deadline {
break outcome;
}
std::thread::sleep(std::time::Duration::from_millis(5));
};
assert!(matches!(outcome, TryCompareExchangeSnapshotOutcome::Exchanged), "{outcome:?}");
// The lock was released: another writer takes it at once.
assert!(matches!(
store.try_compare_exchange_snapshot(&store.pull_all()?, &replacement)?,
TryCompareExchangeSnapshotOutcome::Exchanged
));
// The keyed store releases its lock the same way.
let mut keyed = RedbMessagePackBackingStore::new(temp.path().join("keyed.redb"))?;
@ -8325,178 +7942,6 @@ mod tests {
Ok(())
}
// A link put at the store path while a writer waits on the lock is refused once the lock is taken, as C# does, and the
// lock is free afterwards.
#[cfg(unix)]
#[test]
fn a_link_put_while_a_writer_waits_on_the_lock_is_refused_once_it_holds_it() -> Result<()> {
let temp = tempfile::tempdir()?;
let volume = temp.path().join("volume");
fs::create_dir(&volume)?;
let target = volume.join("store.cc");
SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?;
let before = fs::read(&target)?;
let dangling = volume.join("missing.cc");
let path = temp.path().join("late.cc");
let writers: [(&str, fn(&mut SingleFileMessagePackBackingStore) -> Result<()>); 4] = [
("push", |s| s.push(&snapshot_envelope("b", b"2"))),
("push_all", |s| s.push_all(&[snapshot_envelope("b", b"2")], PushAllOptions::default())),
("delete", |s| s.delete(&snapshot_envelope("a", b"1"))),
("compare_exchange_snapshot", |s| s.compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]).map(|_| ())),
];
for (name, write, link_to) in writers.into_iter().flat_map(|(n, w)| [(n, w, &target), (n, w, &dangling)]) {
let _ = fs::remove_file(&path);
let store = SingleFileMessagePackBackingStore::new(&path);
let holder = store.open_lock_file()?;
fs2::FileExt::lock_exclusive(&holder)?;
let mut waiting = SingleFileMessagePackBackingStore::new(&path);
let handle = std::thread::spawn(move || write(&mut waiting));
std::thread::sleep(std::time::Duration::from_millis(300));
std::os::unix::fs::symlink(link_to, &path)?;
fs2::FileExt::unlock(&holder)?;
let error = handle.join().expect("writer thread").expect_err("the late link must be refused");
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "{name}: {error:#}");
assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "{name}: the link was replaced");
assert_eq!(fs::read(&target)?, before, "{name}: the target changed");
assert!(!dangling.exists(), "{name}: the dangling target was created");
let probe = store.open_lock_file()?;
fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("{name}: lock still held: {e}"))?;
}
Ok(())
}
fn busy_error() -> std::io::Error {
std::io::Error::from_raw_os_error(32)
}
fn is_busy(error: &std::io::Error) -> bool {
error.raw_os_error() == Some(32)
}
#[test]
fn the_lock_open_waits_through_busy_failures_then_opens() {
let (mut tries, mut sleeps) = (0, 0);
let opened = retry_while_busy(
|| {
tries += 1;
if tries <= 4 { Err(busy_error()) } else { Ok(tries) }
},
is_busy,
|| sleeps += 1,
);
assert_eq!(opened.unwrap(), 5);
assert_eq!(sleeps, 4);
}
#[test]
fn the_lock_open_surfaces_a_non_busy_error_at_once() {
let (mut tries, mut sleeps) = (0, 0);
let error = retry_while_busy::<()>(
|| {
tries += 1;
Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied))
},
is_busy,
|| sleeps += 1,
)
.expect_err("a non-busy error must surface");
assert_eq!(error.kind(), std::io::ErrorKind::PermissionDenied);
assert_eq!((tries, sleeps), (1, 0));
}
#[test]
fn the_try_open_maps_a_sharing_violation_to_none_without_retrying() {
let mut tries = 0;
let opened = open_unless_busy::<()>(
|| {
tries += 1;
Err(busy_error())
},
is_busy,
);
assert!(opened.unwrap().is_none());
assert_eq!(tries, 1);
let other = open_unless_busy::<()>(|| Err(std::io::Error::from(std::io::ErrorKind::PermissionDenied)), is_busy);
assert!(other.is_err());
}
// The try entry must answer at once while another handle holds the lock. The attempt runs on a thread and its
// answer is awaited with a timeout, so an entry that waits fails this test by timeout instead of hanging the suite.
#[cfg(unix)]
#[test]
fn the_try_entry_returns_at_once_while_the_lock_is_held() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = temp.path().join("held.cc");
let store = SingleFileMessagePackBackingStore::new(&path);
let holder = store.open_lock_file()?;
fs2::FileExt::lock_exclusive(&holder)?;
let (sender, receiver) = std::sync::mpsc::channel();
let attempt = SingleFileMessagePackBackingStore::new(&path);
std::thread::spawn(move || {
let _ = sender.send(attempt.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")]));
});
let outcome = receiver
.recv_timeout(std::time::Duration::from_secs(3))
.map_err(|_| anyhow!("the try entry waited on a held lock instead of returning"))?;
assert!(matches!(outcome?, TryCompareExchangeSnapshotOutcome::LockContended));
assert!(!path.exists(), "the contended attempt wrote");
let again = store.open_lock_file()?;
assert!(fs2::FileExt::try_lock_exclusive(&again).is_err(), "the held lock was disturbed");
fs2::FileExt::unlock(&holder)?;
Ok(())
}
// Off Windows errno 32 is EPIPE, never a busy file: the predicate is false and the wait surfaces it on the first try.
#[cfg(not(windows))]
#[test]
fn error_32_is_not_busy_off_windows_and_surfaces_at_once() {
assert!(!is_sharing_violation(&busy_error()));
let (mut tries, mut sleeps) = (0, 0);
let error = retry_while_busy::<()>(
|| {
tries += 1;
Err(busy_error())
},
is_sharing_violation,
|| sleeps += 1,
)
.expect_err("error 32 must surface off Windows");
assert_eq!(error.raw_os_error(), Some(32));
assert_eq!((tries, sleeps), (1, 0));
}
// The try entry has no wait, so the late window is the gap between its look and its lock: a link put there is refused
// once the lock is held, nothing is written through or over it, and the lock is released.
#[cfg(unix)]
#[test]
fn a_link_put_between_the_try_entrys_look_and_its_lock_is_refused_and_the_lock_is_released() -> Result<()> {
let temp = tempfile::tempdir()?;
let target = temp.path().join("target.cc");
SingleFileMessagePackBackingStore::new(&target).push(&snapshot_envelope("a", b"1"))?;
let before = fs::read(&target)?;
let dangling = temp.path().join("missing.cc");
let path = temp.path().join("late.cc");
for link_to in [target.clone(), dangling.clone()] {
let store = SingleFileMessagePackBackingStore::new(&path);
let (link, at) = (link_to.clone(), path.clone());
AFTER_PRE_LOCK_LOOK.with(|hook| {
*hook.borrow_mut() = Some(Box::new(move || std::os::unix::fs::symlink(&link, &at).expect("put the link")));
});
let error = store
.try_compare_exchange_snapshot(&[], &[snapshot_envelope("b", b"2")])
.expect_err("the late link must be refused");
assert!(error.downcast_ref::<StoreLinkedError>().is_some(), "{error:#}");
assert!(fs::symlink_metadata(&path)?.file_type().is_symlink(), "the link was replaced");
assert_eq!(fs::read(&target)?, before, "the target changed");
assert!(!dangling.exists(), "the dangling target was created");
let probe = store.open_lock_file()?;
fs2::FileExt::try_lock_exclusive(&probe).map_err(|e| anyhow!("lock still held: {e}"))?;
drop(probe);
fs::remove_file(&path)?;
}
Ok(())
}
// A write at a store path that is a link is refused: a replace there would swap the link for a regular file and
// detach the store from its target. Reads through a live link keep working.
#[cfg(unix)]
@ -8534,34 +7979,6 @@ mod tests {
Ok(())
}
// The advisory query is the held record's and the cache's judges; a link at the store path is the write's to refuse.
#[cfg(unix)]
#[test]
fn is_writable_over_a_store_at_a_live_link_answers_true_while_the_cache_write_is_refused() -> Result<()> {
let temp = tempfile::tempdir()?;
let target = temp.path().join("store.cc");
store_with(&target, |cache| cache.put("held", &settings_one()).map(drop))?;
let before = fs::read(&target)?;
let link = temp.path().join("link.cc");
std::os::unix::fs::symlink(&target, &link)?;
let mut cache = CultCache::new();
cache.register_registry(TestEntries)?;
cache.add_backing_store(SingleFileMessagePackBackingStore::new(&link), [Settings::TYPE])?;
cache.pull_all_backing_stores()?;
let names_before = dir_names(temp.path())?;
assert!(cache.is_writable(Settings::TYPE, "held")?, "the query does not look at the store path");
assert!(cache.is_writable(Settings::TYPE, "fresh")?, "the query does not look at the store path");
let error = cache.put("fresh", &settings_one()).unwrap_err();
let linked = error.downcast_ref::<StoreLinkedError>().unwrap_or_else(|| panic!("{error:#}"));
assert_eq!(linked.path, link);
assert!(fs::symlink_metadata(&link)?.file_type().is_symlink(), "the link is left as it was");
assert_eq!(fs::read(&target)?, before);
assert_eq!(dir_names(temp.path())?, names_before, "the refused write left no lock or staging file");
Ok(())
}
#[cfg(unix)]
#[test]
fn a_write_at_a_live_directory_link_is_refused_as_a_link() -> Result<()> {

View file

@ -1,165 +0,0 @@
//! A consumer of a single-file store reads the schema version a record was published under from the store's catalog, through the
//! public surface only: `pull_all`, `pull_schema_catalog` and `resolve_schema_entry`
//! (`cultlib-gaps:follow_up:rs-snapshot-schema-version`).
use std::path::PathBuf;
use cultcache_rs::{
CacheBackingStore, CultCache, DatabaseEntry, SchemaCatalogEntry, SingleFileMessagePackBackingStore, resolve_schema_entry,
};
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(type = "tests.sc_one", schema = "tests.sc_one")]
struct One {
#[cultcache(key = 0)]
text: String,
}
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(type = "tests.sc_two", schema = "tests.sc_two")]
struct Two {
#[cultcache(key = 0)]
text: String,
}
fn vectors(folder: &str) -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../tests/vectors").join(folder)
}
/// The version a store publishes for each record, in key order: `(key, schema name, schema version)`.
fn versions(store: &SingleFileMessagePackBackingStore) -> Vec<(String, String, String)> {
let catalog = store.pull_schema_catalog().unwrap();
let mut rows: Vec<_> = store
.pull_all()
.unwrap()
.iter()
.map(|envelope| {
let id = envelope.schema_id.as_deref().unwrap();
let entry = resolve_schema_entry(&catalog, id).unwrap_or_else(|| panic!("no entry for {id}"));
(envelope.key.clone(), entry.schema_name.clone(), entry.schema_version.clone())
})
.collect();
rows.sort();
rows
}
fn copy(from: PathBuf) -> (tempfile::TempDir, SingleFileMessagePackBackingStore) {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("store.cc");
std::fs::copy(from, &path).unwrap();
(dir, SingleFileMessagePackBackingStore::new(&path))
}
#[test]
fn a_consumer_reads_the_version_of_each_type_a_rust_store_holds() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("store.cc");
let mut cache = CultCache::new();
cache.register_entry_type::<One>().unwrap();
cache.register_entry_type::<Two>().unwrap();
cache.add_backing_store(SingleFileMessagePackBackingStore::new(&path), ["tests.sc_one", "tests.sc_two"]).unwrap();
cache.pull_all_backing_stores().unwrap();
cache.put("a", &One { text: "1".into() }).unwrap();
cache.put("b", &Two { text: "2".into() }).unwrap();
drop(cache);
let consumer = SingleFileMessagePackBackingStore::new(&path);
assert_eq!(
versions(&consumer),
vec![
("a".to_string(), "tests.sc_one".to_string(), "tests.sc_one.v1".to_string()),
("b".to_string(), "tests.sc_two".to_string(), "tests.sc_two.v1".to_string()),
]
);
}
#[test]
fn an_id_only_an_entry_lists_as_compatible_resolves_to_that_entry() {
let (_dir, store) = copy(vectors("document-variants-c2a/readability/compatible-id-only-v3.bin"));
let catalog = store.pull_schema_catalog().unwrap();
assert_eq!(catalog.len(), 1);
assert_eq!(catalog[0].schema_id, "vectors.item.next");
assert_eq!(catalog[0].compatible_schema_ids, vec!["vectors.item".to_string()]);
let entry = resolve_schema_entry(&catalog, "vectors.item").unwrap();
assert_eq!(entry.schema_id, "vectors.item.next");
assert_eq!(entry.schema_version, "vectors.item.v1");
assert_eq!(resolve_schema_entry(&catalog, "vectors.nowhere"), None);
}
#[test]
fn an_id_an_entry_owns_resolves_to_its_owner_not_to_an_entry_listing_it() {
for vector in ["own-id-over-compatible-v3.bin", "compatible-before-owner-v3.bin"] {
let (_dir, store) = copy(vectors("document-variants-c2a/readability").join(vector));
let catalog = store.pull_schema_catalog().unwrap();
assert_eq!(catalog.len(), 2, "{vector}");
let entry = resolve_schema_entry(&catalog, "vectors.item").unwrap();
assert_eq!((entry.schema_id.as_str(), entry.schema_version.as_str()), ("vectors.item", "vectors.item.v1"), "{vector}");
}
}
#[test]
fn a_legacy_array_and_an_absent_file_publish_no_catalog() {
let (_dir, store) = copy(vectors("document-variants-c2a/readability/legacy-envelopes.bin"));
assert!(!store.pull_all().unwrap().is_empty());
assert_eq!(store.pull_schema_catalog().unwrap(), Vec::<SchemaCatalogEntry>::new());
let dir = tempfile::tempdir().unwrap();
let absent = SingleFileMessagePackBackingStore::new(dir.path().join("none.cc"));
assert_eq!(absent.pull_schema_catalog().unwrap(), Vec::<SchemaCatalogEntry>::new());
assert!(!dir.path().join("none.cc").exists() && !dir.path().join("none.cc.lock").exists(), "reading an absent store creates nothing");
}
// `tests/vectors/key-identity/foreign-store.cc` is written by the C# reference: two types, each published with its version.
#[test]
fn a_store_the_csharp_reference_wrote_gives_its_versions() {
let (_dir, store) = copy(vectors("key-identity/foreign-store.cc"));
assert_eq!(
versions(&store),
vec![
("j".to_string(), "tests.ki_alpha".to_string(), "tests.ki_alpha.v1".to_string()),
("k".to_string(), "tests.ki_gamma".to_string(), "tests.ki_gamma.v1".to_string()),
]
);
let catalog = store.pull_schema_catalog().unwrap();
assert_eq!(catalog[0].content_hash, "sha256:02f0c8b7ea07c11705c1365fc29136ada08c8076209cacfb8d5526229461ce7a");
}
type Entry = (&'static str, &'static str, &'static str, &'static str, &'static str, Vec<&'static str>, Vec<()>);
// A hand-built v3 store whose versions are not the `<name>.v1` the writers above publish, so a version read from anywhere but the
// version slot cannot pass. `shared` is listed by two entries (the first lister wins); `id.c` is owned by one entry and listed by
// an earlier one (the owner wins).
#[test]
fn distinct_versions_come_through_with_the_decoders_resolution() {
let catalog: Vec<Entry> = vec![
("id.a", "N.a", "9.9.9", "hash-a", "{}", vec!["id.a", "shared"], vec![]),
("id.b", "N.b", "7.0", "hash-b", "{}", vec!["id.b", "shared", "id.c"], vec![]),
("id.c", "N.c", "3", "hash-c", "{}", vec!["id.c"], vec![]),
];
let record = |key: &str, id: &str| (key.to_string(), id.to_string(), "2026-11-13T00:00:00Z".to_string(), b"x".to_vec());
let records = vec![record("r1", "shared"), record("r2", "id.c"), record("r3", "id.b"), record("r4", "id.a")];
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("store.cc");
std::fs::write(&path, rmp_serde::to_vec(&("cultcache.store.v3", catalog, records)).unwrap()).unwrap();
let store = SingleFileMessagePackBackingStore::new(&path);
let listed = store.pull_schema_catalog().unwrap();
assert_eq!(listed.iter().map(|e| e.schema_version.as_str()).collect::<Vec<_>>(), vec!["9.9.9", "7.0", "3"]);
assert_eq!(listed[1].content_hash, "hash-b");
assert_eq!(listed[1].compatible_schema_ids, vec!["id.b", "shared", "id.c"]);
let got: Vec<(String, String, String)> = {
let mut rows: Vec<_> = store
.pull_all()
.unwrap()
.iter()
.map(|e| {
let entry = resolve_schema_entry(&listed, e.schema_id.as_deref().unwrap()).unwrap();
(e.key.clone(), entry.schema_name.clone(), entry.schema_version.clone())
})
.collect();
rows.sort();
rows
};
let row = |k: &str, n: &str, v: &str| (k.to_string(), n.to_string(), v.to_string());
assert_eq!(got, vec![row("r1", "N.a", "9.9.9"), row("r2", "N.c", "3"), row("r3", "N.b", "7.0"), row("r4", "N.a", "9.9.9")]);
}

View file

@ -5,11 +5,6 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following
## [Unreleased]
### Changed
- `withStoreWriteLock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. Pinned by a test that puts the link while another holder owns the lock.
- `CultCache.isWritable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
### Breaking
- A write at a single-file store path that is a symbolic link, live or dangling, is refused with the new `StoreLinkedError` (exported from the package entry; `error.path` names the path) before the lock and the replace, and writes nothing: `push`, `pushAll`, `delete` and a soft flush alike. Before, a write at a live link replaced the link with a regular file, a dangling link failed with `ENOENT` (or, for `pushAll`, was replaced), and a soft flush at a link skipped. A caller catching `ENOENT` around a write at a dangling link now catches `StoreLinkedError`; configure the link's target as the store path. Reads through a live link work. An `lstat` failure other than `ENOENT` (`ENOTDIR`, `EACCES`, `ELOOP`) is rethrown as it is.

View file

@ -615,8 +615,7 @@ export class CultCache {
* another key, a record carrying unread slots nothing consents to). It answers the WRITE form. A removal asks the judge's
* removal form and differs in two states: an owner's `delete` of a record held in another store than its own home succeeds
* where this is false (a write would push to the wrong store), and `delete` of a global type at a key other than __global__ is
* the plain false of an unheld key where this is false. It does not look at store paths: a link at a store path is not predicted, and
* the write refuses it with `StoreLinkedError`. Live and advisory: the write or removal still decides.
* the plain false of an unheld key where this is false. Live and advisory: the write or removal still decides.
*/
isWritable(definition: AnyCultCacheDocumentDefinition, key: string): boolean {
const registered = this.#requireDefinition(definition);

View file

@ -218,8 +218,6 @@ export async function withStoreWriteLock<T>(
}
try {
// A link may have been put at the path while the lock was awaited; the look above only saw the path before.
await refuseLinkedStore(storePath);
return await operation();
} finally {
release();

View file

@ -2106,7 +2106,7 @@ async function readTsInteropStore(file: string): Promise<InteropNote> {
async function buildInteropPeers(): Promise<void> {
if (!(await exists(rustInteropBinary))) {
await execAsync(`"${cargoCommand}" build --locked --quiet --example cultcache_interop`, {
await execAsync(`"${cargoCommand}" build --quiet --example cultcache_interop`, {
cwd: cultcacheRsRoot,
});
}
@ -2473,30 +2473,6 @@ test("SingleFileMessagePackBackingStore refuses a dangling symlink at its path a
assert.deepEqual(await readdir(volume), []);
});
// The advisory query answers for the held record and the cache's judges; a link at the store path is the write's to refuse.
test("isWritable over a store at a live link answers true while the cache write is refused", { skip: process.platform === "win32" }, async () => {
const dir = await mkdtemp(join(tmpdir(), "cultcache-writable-link-"));
const volume = join(dir, "volume");
await mkdir(volume);
const target = join(volume, "store.cc");
const link = join(dir, "link.cc");
const document = defineDocumentType({ type: "note", schema: z.object({ text: z.string() }) });
await CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(target)).build()
.put(document, "held", { text: "held" });
const before = await readFile(target);
await symlink(target, link);
const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(link)).build();
await cache.pullAllBackingStores();
const namesBefore = [(await readdir(dir)).sort(), (await readdir(volume)).sort()];
assert.equal(cache.isWritable(document, "held"), true);
assert.equal(cache.isWritable(document, "fresh"), true);
await assert.rejects(() => cache.put(document, "fresh", { text: "fresh" }), (error: unknown) => error instanceof StoreLinkedError && error.path === link);
assert.ok((await lstat(link)).isSymbolicLink());
assert.deepEqual(await readFile(target), before);
assert.deepEqual([(await readdir(dir)).sort(), (await readdir(volume)).sort()], namesBefore);
});
// The second look of the read, staged: nothing is at the path only when the read and the lstat both find nothing. A path
// that changed between them (a link in a parent swapped for a file, a store that appeared) is never an empty store.
test("readStoreBytes reports nothing at the path only when the lstat finds nothing too", async () => {

View file

@ -417,43 +417,6 @@ test("a second writer in this process waits for the store lock and runs after re
}
});
test("a link put while a writer waits on the store lock is refused once it holds the lock", { skip: process.platform === "win32" }, async () => {
const directory = await scratch();
try {
const target = join(directory, "target.cc");
await new SingleFileMessagePackBackingStore(target).push(noteEnvelope("a"));
const before = await readFile(target);
const writes: Array<[string, (store: SingleFileMessagePackBackingStore) => Promise<void>]> = [
["push", (store) => store.push(noteEnvelope("b"))],
["pushAll", (store) => store.pushAll([noteEnvelope("b")])],
["delete", (store) => store.delete(noteEnvelope("a"))],
];
const missing = join(directory, "missing.cc");
for (const [name, write, linkTo] of writes.flatMap(([n, w]) => [[n, w, target], [`${n} (dangling)`, w, missing]] as const)) {
const link = join(directory, `${name.replace(/\W+/g, "-")}.cc`);
let writer: Promise<void> | undefined;
await withStoreWriteLock(link, async () => {
writer = write(new SingleFileMessagePackBackingStore(link));
writer.catch(() => undefined);
await sleepFor(150);
await fsp.symlink(linkTo, link); // after the writer's pre-lock look, while it waits
});
await assert.rejects(writer!, (error: unknown) => error instanceof PackageStoreLinkedError, name);
assert.ok((await fsp.lstat(link)).isSymbolicLink(), `${name}: the link was replaced`);
assert.deepEqual(await readFile(target), before, `${name}: the target changed`);
await assert.rejects(fsp.lstat(missing), { code: "ENOENT" }, `${name}: the dangling target was created`);
await rm(link); // the link is refused before the lock, so remove it to probe the lock
assert.equal(await withStoreWriteLock(link, async () => "free", { timeoutMs: 200 }), "free", `${name}: the lock is held`);
}
} finally {
await rm(directory, { recursive: true, force: true });
}
});
function noteEnvelope(key: string): CultCacheEnvelope {
return { key, type: "note", payload: documentPayloadFromBytes(Uint8Array.of(0x91, 1)), storedAt: "2026-10-20T00:00:00Z" };
}
test("the store lock gives up after its wait bound and names the lock path", async () => {
const directory = await scratch();
try {

View file

@ -147,8 +147,8 @@ CultMath names with tests instead of copied project-local helper drift.
```powershell
dotnet build packages/cultmath/src/CultMath/CultMath.csproj
dotnet test packages/cultmath/tests/CultMath.Tests/CultMath.Tests.csproj
cargo test --locked --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml
cargo build --locked --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml
cargo test --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml
cargo build --release --manifest-path packages/cultmath/native/cultmath-core/Cargo.toml
.\packages\cultmath\scripts\build-unity-package.ps1
```

View file

@ -122,7 +122,7 @@ const rust: Target = {
name: "rust",
payload: (text) => encode({ text }),
async start() {
if (!prebuilt) await execFileAsync(cargo, ["build", "--locked", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir });
if (!prebuilt) await execFileAsync(cargo, ["build", "--example", "receipt_interop_server", "--message-format", "short"], { cwd: rustManifestDir });
const metadata = spawnSync(cargo, ["metadata", "--format-version", "1", "--no-deps"], { cwd: rustManifestDir, encoding: "utf8" });
if (metadata.status !== 0) throw new Error(`cargo metadata failed: ${metadata.stderr}`);
const exe = join(JSON.parse(metadata.stdout).target_directory, "debug", "examples", `receipt_interop_server${process.platform === "win32" ? ".exe" : ""}`);

View file

@ -1 +1,2 @@
/target
Cargo.lock

File diff suppressed because it is too large Load diff

View file

@ -30,7 +30,7 @@ knock on the contract before touching the furniture.
## Receipts
```powershell
cargo test --locked
cargo test
```
The initial tests prove:

View file

@ -7,7 +7,7 @@
//! traffic crosses the same schedule. The schedule is `policy.rs`.
//!
//! ```text
//! cargo run --locked --release --example cultnet_impair -- \
//! cargo run --release --example cultnet_impair -- \
//! --listen 0.0.0.0:17890 --upstream 127.0.0.1:17990 \
//! --profile tests/impairment/loss-1pct.toml --seed 424242 --metrics loss-1pct.csv
//! ```

View file

@ -3,7 +3,7 @@
//! Ignored by default; run in release, where the numbers mean something:
//!
//! ```text
//! cargo test --locked --release --test media_fec_bench -- --ignored --nocapture
//! cargo test --release --test media_fec_bench -- --ignored --nocapture
//! ```
//!
//! Each line is nanoseconds per call through the public surface, so record

View file

@ -6,7 +6,7 @@ cultnet-rs only, and Idunn's `IdunnRuntimeActivationLaunch` is that same crate's
It needs Linux and a machine-id equal to the `machine-id` file here, because the
provider identity is bound to it. On Yggdrasil:
DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh <CultLib> HEAD rust 'cd packages/cultnet-rs && cp <fixed-machine-id> /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run --locked -q --example idunn_runtime_fixture /tmp/out'
DOCKER_ARGS='--cap-add SYS_ADMIN --security-opt apparmor=unconfined' bash ygg-verify.sh <CultLib> HEAD rust 'cd packages/cultnet-rs && cp <fixed-machine-id> /tmp/mid && mount --bind /tmp/mid /etc/machine-id && cargo run -q --example idunn_runtime_fixture /tmp/out'
then copy `/tmp/out` over this directory. `--cap-add SYS_ADMIN` exists only so the bind mount can present the committed `machine-id`, which keeps Yggdrasil's real machine-id out of the repo. Every `presence-*.bin` was accepted by
`authenticate_runtime_presence_claim` and correlated by
@ -26,7 +26,7 @@ The generator takes its specification as input, so a consumer needs no patch to
specification is command-line flags; `@<file>` splices in a response file (one `--flag value`
per line, `#` comments). CultLib's own sets are `fixture.args` here:
cargo run --locked -q --example idunn_runtime_fixture /tmp/out @<path>/fixture.args
cargo run -q --example idunn_runtime_fixture /tmp/out @<path>/fixture.args
The consumer keeps its own `.args` file in its repository and runs the same command with the
same machine-id bind mount as above. Global flags: `--source-repository`, `--detail`. Each

View file

@ -2795,7 +2795,7 @@ async function buildInteropPeers(): Promise<void> {
}
async function buildRustInteropPeer(): Promise<void> {
rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--locked", "--quiet", "--example", "cultnet_interop_peer"], {
rustInteropPeerBuild ??= execFileAsync(cargoCommand, ["build", "--quiet", "--example", "cultnet_interop_peer"], {
cwd: cultnetRsRoot,
}).then(() => undefined);
await rustInteropPeerBuild;

View file

@ -2,7 +2,7 @@
//
// The vectors below were produced by `cultnet-rs`, not written by hand:
//
// cargo run --locked --example runtime_presence_vector -p cultnet-rs
// cargo run --example runtime_presence_vector -p cultnet-rs
//
// That matters. A fixture authored for both sides at once agrees with itself
// and proves nothing; the contract is only real if one fork emits the bytes and

View file

@ -7,7 +7,7 @@ plus the rules that need no store to state (`Answer::is_training_data`, `Answer:
(comments 682 and 759). Specification only: no collector, tally, export or tombstone hashing lives here.
Entry points: `src/lib.rs`; golden vectors `tests/vectors/consent_golden.csv`, run by `tests/golden.rs`
(`cargo test --locked`). Planned, not live: the collector, the tally, the training export.
(`cargo test`). Planned, not live: the collector, the tally, the training export.
## Where each ruling lands (doc lines at aaeb175)

View file

@ -11,8 +11,8 @@ adversary, storage, and any projection. Every parameter is a placeholder until t
## First call
```
cargo run --locked --example first_call # prints the Meta example and a splitting table
cargo test --locked
cargo run --example first_call # prints the Meta example and a splitting table
cargo test
```
```rust

View file

@ -686,7 +686,14 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
stream.Flush(true);
}
StoreReplace.Replace(tempPath, path);
if (File.Exists(path))
{
ReplaceExistingFile(tempPath, path);
}
else
{
File.Move(tempPath, path);
}
}
finally
{
@ -708,23 +715,17 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
// Only a writer creates the records folder and the lock, so opening the store creates nothing. A reader opens an
// existing lock and returns null when there is none; that does not mean no writer committed (a store may be copied
// without its lock, and a first writer may create it mid-load), so an unleased load checks that it settled.
// A writer's lease is also where a link at the manifest path is refused (a commit would replace the link with a file): before
// the records folder or the lock is made, and again once the lease is held, because a link can be put there while it waited.
private FileStream? AcquireCommitLease(bool wait, bool create)
{
if (create)
{
CultStoreLinks.Refuse(_manifestFile.FullName);
Directory.CreateDirectory(_recordDirectory.FullName);
}
var lockPath = Path.Combine(_recordDirectory.FullName, ".commit.lock");
var started = Stopwatch.StartNew();
while (true)
{
FileStream held;
try
{
held = new FileStream(
return new FileStream(
lockPath,
create ? FileMode.OpenOrCreate : FileMode.Open,
FileAccess.ReadWrite,
@ -747,23 +748,29 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30))
{
Thread.Sleep(10);
continue;
}
if (create)
{
try
{
CultStoreLinks.Refuse(_manifestFile.FullName);
}
catch
{
held.Dispose();
throw;
}
}
return held;
}
}
private static void ReplaceExistingFile(string sourcePath, string destinationPath)
{
for (var attempt = 0; attempt < 3; attempt++)
{
try
{
File.Replace(sourcePath, destinationPath, null);
return;
}
catch (IOException) when (attempt < 2)
{
Thread.Sleep(10 * (attempt + 1));
}
catch (UnauthorizedAccessException) when (attempt < 2)
{
Thread.Sleep(10 * (attempt + 1));
}
}
File.Replace(sourcePath, destinationPath, null);
}
}

View file

@ -2052,29 +2052,6 @@ namespace GameCult.Caching
Contended
}
public enum CultUpdateOutcome
{
Updated,
Declined,
LostRace
}
/// <summary>
/// What <see cref="CultCache.Update{T}"/> did. A lost race is a value, not an exception. <see cref="Document"/> is the instance the
/// cache holds at the key when Update returns: the written one for Updated, null when the key is absent.
/// </summary>
public readonly struct CultUpdateResult<T> where T : class
{
public CultUpdateResult(CultUpdateOutcome outcome, T? document)
{
Outcome = outcome;
Document = document;
}
public CultUpdateOutcome Outcome { get; }
public T? Document { get; }
}
public sealed class CultCommitRequest
{
internal CultCommitRequest(
@ -2586,7 +2563,6 @@ namespace GameCult.Caching
return UpsertAsync(document, handle);
}
// A blind write, last-writer-wins per key; Update is the conditional golden path.
// Refused with CultSchemaConflictException when the key holds a record stored under an id no registered type declares.
public Task<CultRecordHandle<T>> UpsertAsync<T>(T document, CultRecordHandle<T>? handle = null)
{
@ -2663,7 +2639,6 @@ namespace GameCult.Caching
/// <summary>
/// Commits the staged batch. Returns false when a condition failed: nothing was written, changed in memory, or published.
/// Without <c>Expect</c> a commit is blind; <see cref="Update{T}"/> retries a conditional one.
/// </summary>
/// <remarks>
/// A commit that succeeds and then fails in a journal or observer still stands and is published, and the writer
@ -2681,59 +2656,6 @@ namespace GameCult.Caching
return Land(stage, wait: false);
}
/// <summary>
/// Reads the record at <paramref name="key"/>, applies <paramref name="apply"/> to a private copy of it, and commits the result only
/// if the record is still the one that was read; when another writer got there first it re-reads and runs <paramref name="apply"/>
/// again, up to <paramref name="maxAttempts"/> times. The write is durable when Update returns.
/// </summary>
/// <remarks>
/// <paramref name="apply"/> may run up to <paramref name="maxAttempts"/> times. It gets a private copy (null when the key holds
/// nothing), which it may change and return. It must not write to the cache or have other side effects. Return null to write
/// nothing. A store with staged blind writes refuses Update until it is flushed, as it refuses any conditional commit. The cache
/// needs a codec to copy with. Running out of attempts returns <see cref="CultUpdateOutcome.LostRace"/>; every other failure throws.
/// </remarks>
public CultUpdateResult<T> Update<T>(CultRecordKey key, Func<T?, T?> apply, int maxAttempts = 5) where T : class
{
if (apply == null) throw new ArgumentNullException(nameof(apply));
if (maxAttempts < 1) throw new ArgumentOutOfRangeException(nameof(maxAttempts), maxAttempts, "At least one attempt is required.");
var codec = _codec ?? throw new InvalidOperationException("This cache has no codec; create it with CultCacheMessagePack.Create.");
for (var attempt = 1; ; attempt++)
{
var (version, copy) = Held(() =>
{
if (!_entries.TryGetValue(key.Value, out var held))
return ((string?)null, (T?)null);
if (held.Variant != null || held.Document is not T)
throw new InvalidOperationException($"Record {key.Value} is a variant or not a {typeof(T).Name}; Update cannot copy it.");
return (held.StoredVersion, (T?)codec.Clone(held.Document, held.Descriptor.DocumentType));
});
var next = apply(copy);
if (next == null)
return new CultUpdateResult<T>(CultUpdateOutcome.Declined, Get<T>(key));
CultCommitOutcome outcome;
try
{
outcome = Land(batch =>
{
batch.Expected.Add((key, version));
batch.Upsert(next, new CultRecordHandle<T>(key));
}, wait: true);
}
catch (CultWriteConflictException conflict) when (conflict.InnerException == null)
{
outcome = CultCommitOutcome.Mismatch;
}
if (outcome == CultCommitOutcome.Committed)
return new CultUpdateResult<T>(CultUpdateOutcome.Updated, next);
if (outcome != CultCommitOutcome.Mismatch)
throw new InvalidOperationException($"Update of {key.Value} was refused: {outcome}.");
Held(() => { Home(next.GetType())?.PullAll(); return true; });
if (attempt >= maxAttempts)
return new CultUpdateResult<T>(CultUpdateOutcome.LostRace, Get<T>(key));
}
}
public CultRecordHandle<T>? TryGetHandle<T>(T document)
{
if (document == null) throw new ArgumentNullException(nameof(document));
@ -2754,8 +2676,7 @@ namespace GameCult.Caching
/// when that store holds it unowned (undeclared drift that drops persisted slots, or a payload carrying slots past this build's,
/// see <see cref="CultStoredDocument.UnreadSlots"/>). For a key the cache does not hold, the removal check every store makes of it
/// (false when a store lists it foreign). One write it does not predict: a Flatten of a variant whose base chain holds an unowned record is refused, though
/// the variant itself is writable. It does not look at store paths: a link at a store path is not predicted, and the write refuses it
/// with <see cref="CultStoreLinkedException"/>. Live and advisory: the write still decides.
/// the variant itself is writable. Live and advisory: the write still decides.
/// </summary>
public bool IsWritable(CultRecordKey key)
{
@ -5224,7 +5145,7 @@ namespace GameCult.Caching
var attributes = FileInfo.Attributes;
if ((int)attributes == -1)
return null;
if ((attributes & FileAttributes.Directory) != 0 && Directory.Exists(FileInfo.FullName))
if ((attributes & FileAttributes.Directory) != 0 && (attributes & FileAttributes.ReparsePoint) == 0)
throw new IOException("The store path is a directory, not a store file.");
try
{
@ -5246,27 +5167,25 @@ namespace GameCult.Caching
}
// Something is at the path even when the path cannot be opened: a link keeps the reparse-point attribute whether or not its
// target is there.
private bool PathIsALink() => CultStoreLinks.IsLink(FileInfo.FullName);
// A write never goes through a link at the store path: replacing it would silently detach the store from its link.
private void RefuseLinkedStore() => CultStoreLinks.Refuse(FileInfo.FullName);
// target is there. A path with nothing at it reports no attributes at all (-1).
private bool PathIsALink()
{
FileInfo.Refresh();
var attributes = FileInfo.Attributes;
return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
}
// The lock is a sidecar opened exclusively, which excludes other handles in this process and in others alike.
// Every write takes the lock first, so the link is refused here twice: before the lock file or anything else is made, and
// again once the lock is held, because a link can be put at the path while this writer waited on another's lock.
private FileStream? AcquireLock(bool wait)
{
RefuseLinkedStore();
Directory.CreateDirectory(FileInfo.DirectoryName!);
var lockPath = FileInfo.FullName + ".lock";
var started = Stopwatch.StartNew();
while (true)
{
FileStream held;
try
{
held = new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, bufferSize: 1);
return new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None, bufferSize: 1);
}
catch (IOException) when (!wait)
{
@ -5275,20 +5194,7 @@ namespace GameCult.Caching
catch (IOException) when (started.Elapsed < TimeSpan.FromSeconds(30))
{
Thread.Sleep(10);
continue;
}
try
{
RefuseLinkedStore();
}
catch
{
held.Dispose();
throw;
}
return held;
}
}
@ -5311,7 +5217,14 @@ namespace GameCult.Caching
stream.Flush(true);
}
StoreReplace.Replace(tempPath, path);
if (File.Exists(path))
{
File.Replace(tempPath, path, null);
}
else
{
File.Move(tempPath, path);
}
}
finally
{

View file

@ -352,7 +352,7 @@ namespace GameCult.Caching
}
}
public object Clone(object document, Type type) => _codec.Clone(document, type);
public object Clone(object document, Type type) => _deserialize(type, _serialize(document, type, type));
public CultInspectorEdit BeginEdit(CultStoredDocument record)
{

View file

@ -1,24 +0,0 @@
using System.IO;
namespace GameCult.Caching
{
/// <summary>
/// A write was attempted at a store path that is a link (a symbolic link to a file or a directory, a Windows junction or any
/// other reparse-point link). The write is refused before the lock, the staging file or the replace, and again once the lock
/// is held, so a link that exists when a writer takes the lock leaves the link and its target untouched. A link created by
/// something that ignores the lock after that check, while the commit builds and writes the new image, is still replaced;
/// that window cannot be closed without native no-follow opens. Configure the link's target as the store path to write to
/// it; reads through a live link are unchanged.
/// </summary>
public sealed class CultStoreLinkedException : IOException
{
public CultStoreLinkedException(string path)
: base($"CultCache store {path} is a link, and a write at it would replace the link; configure the link's target as the store path.")
{
Path = path;
}
/// <summary>The store path that is a link.</summary>
public string Path { get; }
}
}

View file

@ -1,22 +0,0 @@
using System.IO;
namespace GameCult.Caching
{
// The one decision on whether a store-owned file is a link, shared by every store kind that replaces files.
internal static class CultStoreLinks
{
// A path with nothing at it reports no attributes at all (-1). Any reparse point counts as a link, whether or not its
// target is there.
internal static bool IsLink(string path)
{
var attributes = new FileInfo(path).Attributes;
return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
}
internal static void Refuse(string path)
{
if (IsLink(path))
throw new CultStoreLinkedException(path);
}
}
}

View file

@ -29,11 +29,6 @@ namespace GameCult.Caching
// (document payload, slot and encoded value pairs)
public Func<byte[], IReadOnlyList<KeyValuePair<int, byte[]>>, byte[]> Overlay { get; }
// The one copy of a document: its encoding, decoded again. Update's private argument and CultInspectorModel.Clone share it.
internal object Clone(object document, Type type) =>
Deserialize(type, type, Serialize(document, type, type))
?? throw new InvalidOperationException($"The codec returned null for {type.FullName}.");
/// <summary>
/// Decodes a document payload read from a store file, as a cache load decodes it: the ids a load mints are minted from the key
/// and their path, so every reader of one store mints the same ids, and a payload a cache write would refuse is refused here too.

View file

@ -13,7 +13,6 @@ Cacheable models are plain classes annotated with cache intent.
using GameCult.Caching;
using MessagePack;
[MessagePackObject]
[CultDocument("gamecult.item_data", "gamecult.item_data.v1")]
public sealed class ItemData
{
@ -132,33 +131,18 @@ using GameCult.Caching;
using GameCult.Caching.MessagePack;
var cache = await CultCacheMessagePack.OpenAsync("Data.msgpack");
var key = new CultRecordKey("potion");
// The golden path: read, change a private copy, commit only if the record is unchanged, re-run on a lost race.
// Durable when it returns; no flush.
var result = cache.Update<ItemData>(key, item =>
{
item ??= new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable" };
item.Value += 10;
return item;
});
// The blind opt-out: last writer wins per key. Name the key, or a new key claims the same unique Sku and is refused.
var handle = await cache.UpsertAsync(new ItemData
{
Name = "Potion",
Sku = "POT-1",
Category = "Consumable",
Value = 50
}, new CultRecordHandle<ItemData>(key));
});
await cache.FlushAsync();
```
`Update` may run your function more than once, so it must have no side effects. Returning null writes nothing, and a
lost race that outlasts `maxAttempts` comes back as `CultUpdateOutcome.LostRace`. `UpsertAsync` plus `FlushAsync` never
checks what it overwrites.
If you want the explicit assembly rite, the lower-level `CultCache` +
`SingleFileMessagePackBackingStore` path is still there. The helper is just the
blessed front door.
@ -310,10 +294,13 @@ Useful surfaces:
- `cache.IsDirty`
- `store.IsDirty`
- `cache.Document<T>(key)`
- `cache.Update<T>(key, apply)`
- `cache.UpsertAsync(document)` (blind write)
- `cache.Soa<T>()`
- `cache.UpsertAsync(document)`
- `cache.FlushAllBackingStores()`
- `cache.FlushAsync()`
- `cache.FlushBackingStore(store)`
- `cache.PrepareForReloadOrShutdown()`
- `cache.PrepareForReloadOrShutdownAsync()`
- `cache.FlushAttachedStoresOnDispose`
- `store.FlushOnDispose`

View file

@ -1,182 +0,0 @@
using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
namespace GameCult.Caching
{
/// <summary>The kernel32 calls a Windows replace needs, each answering 0 or a Win32 error code.</summary>
internal interface IReplaceKernel
{
/// <summary>CreateFileW(staged, DELETE|SYNCHRONIZE), FileRenameInfoEx with REPLACE_IF_EXISTS|POSIX_SEMANTICS, CloseHandle.</summary>
int RenamePosix(string staged, string destination);
/// <summary>MoveFileExW(staged, destination, flags).</summary>
int MoveFileEx(string staged, string destination, uint flags);
void Sleep(int milliseconds);
}
/// <summary>
/// Owns how a staged file becomes the store, for the single-file and directory stores. Windows tries a POSIX-semantics
/// rename first (which makes a moment with the path absent rare for a lock-free reader, not impossible: a stalled rename was
/// measured leaving it absent for 5-47 ms), falls back to write-through MoveFileExW where that is unsupported or does not
/// resolve, falls back to the portable File.Replace if MoveFileExW itself does not resolve, and retries transient sharing
/// errors; other OSes keep rename. Mirrors cultcache-ts store-file.ts
/// replaceStoreFile (ruling variants:ruling:windows-replace-posix-first).
/// </summary>
internal static class StoreReplace
{
private const int ErrorAccessDenied = 5;
private const int ErrorSharingViolation = 32;
private const int ErrorLockViolation = 33;
private const int ErrorNotSupported = 50;
private const int ErrorInvalidParameter = 87;
private const int ReplaceAttempts = 25;
private const int ReplaceRetryMs = 20;
/// <summary>MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH.</summary>
internal const uint MoveFileReplaceWriteThrough = 0x9;
internal static readonly IReplaceKernel WindowsKernel = new Kernel32Replace();
public static void Replace(string staged, string destination)
{
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
ReplaceWindows(staged, destination, WindowsKernel);
}
else
{
ReplacePortable(staged, destination);
}
}
private static void ReplacePortable(string staged, string destination)
{
if (File.Exists(destination))
{
File.Replace(staged, destination, null);
}
else
{
File.Move(staged, destination);
}
}
private static bool Unresolved(Exception ex) => ex is DllNotFoundException || ex is EntryPointNotFoundException;
internal static void ReplaceWindows(string staged, string destination, IReplaceKernel kernel)
{
var posix = true;
var lastCode = 0;
for (var attempt = 1; attempt <= ReplaceAttempts; attempt++)
{
int code;
try
{
code = posix ? RenamePosixOrUnsupported(kernel, staged, destination) : kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough);
if (posix && (code == 1 || code == ErrorNotSupported || code == ErrorInvalidParameter))
{
// POSIX semantics are unsupported here (FAT, exFAT, before Windows 10 1607): the rest of this call is write-through.
posix = false;
code = kernel.MoveFileEx(staged, destination, MoveFileReplaceWriteThrough);
}
}
catch (Exception ex) when (Unresolved(ex))
{
// kernel32's MoveFileExW does not resolve here (a sandboxed player): the runtime-owned replace is all that is left.
ReplacePortable(staged, destination);
return;
}
if (code == 0)
{
return;
}
if (code != ErrorAccessDenied && code != ErrorSharingViolation && code != ErrorLockViolation)
{
throw new IOException($"CultCache could not replace the store: Win32 error {code}.");
}
lastCode = code;
kernel.Sleep(ReplaceRetryMs);
}
throw new IOException($"CultCache could not replace the store after {ReplaceAttempts} attempts: last Win32 error {lastCode}.");
}
private static int RenamePosixOrUnsupported(IReplaceKernel kernel, string staged, string destination)
{
try
{
return kernel.RenamePosix(staged, destination);
}
catch (Exception ex) when (Unresolved(ex))
{
// The POSIX call does not resolve: the same answer as "unsupported", so the replace falls back to MoveFileExW.
return ErrorNotSupported;
}
}
private sealed class Kernel32Replace : IReplaceKernel
{
private const uint DeleteSynchronize = 0x00110000;
private const uint ShareAll = 0x7;
private const uint OpenExisting = 3;
private const uint FileAttributeNormal = 0x80;
private const int FileRenameInfoEx = 22;
private const uint RenameReplacePosix = 0x3;
public int RenamePosix(string staged, string destination)
{
var handle = CreateFileW(staged, DeleteSynchronize, ShareAll, IntPtr.Zero, OpenExisting, FileAttributeNormal, IntPtr.Zero);
if (handle == new IntPtr(-1))
{
return Marshal.GetLastWin32Error();
}
try
{
// FILE_RENAME_INFO: Flags u32 at 0, RootDirectory handle, FileNameLength u32, FileName (UTF-16, offsets for this pointer size).
var name = System.Text.Encoding.Unicode.GetBytes(Path.GetFullPath(destination));
var lengthOffset = 2 * IntPtr.Size;
var nameOffset = lengthOffset + 4;
var size = nameOffset + name.Length + 2;
var info = Marshal.AllocHGlobal(size);
try
{
for (var i = 0; i < size; i++) Marshal.WriteByte(info, i, 0);
Marshal.WriteInt32(info, 0, (int)RenameReplacePosix);
Marshal.WriteInt32(info, lengthOffset, name.Length);
Marshal.Copy(name, 0, info + nameOffset, name.Length);
return SetFileInformationByHandle(handle, FileRenameInfoEx, info, (uint)size) ? 0 : Marshal.GetLastWin32Error();
}
finally { Marshal.FreeHGlobal(info); }
}
finally { CloseHandle(handle); }
}
public int MoveFileEx(string staged, string destination, uint flags)
=> MoveFileExW(staged, destination, flags) ? 0 : Marshal.GetLastWin32Error();
public void Sleep(int milliseconds) => Thread.Sleep(milliseconds);
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern IntPtr CreateFileW(string fileName, uint access, uint share, IntPtr security, uint disposition, uint flags, IntPtr template);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SetFileInformationByHandle(IntPtr handle, int informationClass, IntPtr information, uint size);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool CloseHandle(IntPtr handle);
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool MoveFileExW(string existing, string target, uint flags);
}
}
}

View file

@ -1,151 +0,0 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using GameCult.Caching;
using MessagePack;
using NUnit.Framework;
namespace GameCult.Caching.Tests
{
// The id walk's own contracts, one mutant-sized fact at a time: the order of the id path, a union that names itself, a list of
// values never counting as an id, and the attribute predicates that decide whether a type is walked or written whole.
public class ElementIdWalkTests
{
// The codec asks MessagePack about the id path in order and refuses at the first type it cannot allow, so the order is what
// an error names first: the document, then the collection types from the outermost in, then the element and its arms.
[Test]
public void TheIdPathListsCollectionsFromTheOutermostIn()
{
Assert.That(CultElementIds.Inspect(typeof(MsMapMap)).IdPath, Is.EqualTo(new[]
{
typeof(MsMapMap), typeof(Dictionary<string, Dictionary<string, List<ElementIdEngineTests.EIded>>>),
typeof(Dictionary<string, List<ElementIdEngineTests.EIded>>), typeof(List<ElementIdEngineTests.EIded>), typeof(ElementIdEngineTests.EIded)
}), "maps nested in maps");
Assert.That(CultElementIds.Inspect(typeof(MsListDeep)).IdPath, Is.EqualTo(new[]
{
typeof(MsListDeep), typeof(List<List<List<ElementIdEngineTests.EIded>>>), typeof(List<List<ElementIdEngineTests.EIded>>),
typeof(List<ElementIdEngineTests.EIded>), typeof(ElementIdEngineTests.EIded)
}), "lists nested in lists");
}
// ConcreteTypes stops at a type it has already expanded, so a union that lists its own base is finished, not a stack overflow.
[Test]
public void AUnionThatNamesItselfIsWalkedOnce()
{
var survey = CultElementIds.Inspect(typeof(MsSelfHolder));
Assert.That(survey.Problems, Is.Empty);
Assert.That(survey.IdPath, Is.EquivalentTo(new[] { typeof(MsSelfHolder), typeof(List<MsSelfBase>), typeof(MsSelfBase), typeof(MsSelfArm) }));
}
// Value lists hold no ids however they are nested, and a root that is itself a value list holds none either.
[Test]
public void ListsOfValuesHoldNoIds()
{
var values = new MsValues { Numbers = { 1 }, Tags = { "a" }, Grid = { new List<int> { 1 } }, Map = { ["k"] = new List<int> { 2 } } };
Assert.That(CultElementIds.Holds(typeof(MsValues), values), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<int>), new List<int> { 1, 2 }), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<List<int>>), new List<List<int>> { new() { 1 } }), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<string>), new List<string> { "a" }), Is.False);
Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated");
}
// A nullable struct is walked as the struct it wraps; a nullable value is not walked at all.
[Test]
public void ANullableKeyedStructWalksAsItsOwner()
{
Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct)), Is.True);
Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct?)), Is.True);
Assert.That(CultElementIds.WalksAsOwner(typeof(int?)), Is.False);
Assert.That(CultElementIds.WalksAsOwner(typeof(int)), Is.False);
}
// [Key] is read through an override, so an element whose id property overrides a keyed base member is still an id with a key.
[Test]
public void AKeyOnTheBaseOfAnOverrideCountsForTheOverride()
{
Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).Problems, Is.Empty);
Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).IdPath, Does.Contain(typeof(MsOverrideEl)));
}
// A type is walked by key when it is [MessagePackObject] or [Union], read through its bases for the former; it is one whole
// value when it declares its own formatter, is [DataContract] without being [MessagePackObject], or is keyed by name.
[Test]
public void KeyedTypesWalkAndWholeTypesDoNot()
{
Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpBase)), Is.True, "[MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsUBase)), Is.True, "[Union]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpDerived)), Is.True, "[MessagePackObject] inherited");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsPlain)), Is.False, "[Key] members without [MessagePackObject] or [Union]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsUArm)), Is.True, "an arm that is itself [MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsDc)), Is.False, "[DataContract] alone");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcMp)), Is.True, "[DataContract] beside [MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcOverMpBase)), Is.True, "[DataContract] over an inherited [MessagePackObject]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropName)), Is.False, "KeyAsPropertyName");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropNameDerived)), Is.False, "KeyAsPropertyName inherited");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsStringKey)), Is.False, "a string [Key]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedBase)), Is.False, "[MessagePackFormatter]");
Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedDerived)), Is.False, "[MessagePackFormatter] inherited");
}
[MessagePackObject] public class MsMapMap { [Key(0)] public Dictionary<string, Dictionary<string, List<ElementIdEngineTests.EIded>>> M { get; set; } = new(); }
[MessagePackObject] public class MsListDeep { [Key(0)] public List<List<List<ElementIdEngineTests.EIded>>> G { get; set; } = new(); }
[Union(0, typeof(MsSelfBase)), Union(1, typeof(MsSelfArm))] public abstract class MsSelfBase { }
[MessagePackObject] public class MsSelfArm : MsSelfBase { [Key(0)] [CultElementId] public string Id { get; set; } = ""; }
[MessagePackObject] public class MsSelfHolder { [Key(0)] public List<MsSelfBase> Items { get; set; } = new(); }
[MessagePackObject]
public class MsValues
{
[Key(0)] public List<int> Numbers { get; set; } = new();
[Key(1)] public List<string> Tags { get; set; } = new();
[Key(2)] public List<List<int>> Grid { get; set; } = new();
[Key(3)] public Dictionary<string, List<int>> Map { get; set; } = new();
}
public sealed class MsBoom : IEnumerable<int>
{
public IEnumerator<int> GetEnumerator() => throw new InvalidOperationException("enumerated");
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator();
}
[MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; }
[MessagePackObject]
public class MsOverrideBase
{
[Key(0)] [CultElementId] public virtual string Id { get; set; } = "";
}
[MessagePackObject]
public class MsOverrideEl : MsOverrideBase
{
public override string Id { get; set; } = "";
}
[MessagePackObject] public class MsOverrideHolder { [Key(0)] public List<MsOverrideEl> Items { get; set; } = new(); }
[MessagePackObject] public class MsMpBase { [Key(0)] public int A { get; set; } }
public class MsMpDerived : MsMpBase { }
public class MsPlain { [Key(0)] public int A { get; set; } }
[Union(0, typeof(MsUArm))] public abstract class MsUBase { }
[MessagePackObject] public class MsUArm : MsUBase { [Key(0)] public int A { get; set; } }
[DataContract] public class MsDc { [DataMember] public int A { get; set; } }
[DataContract] [MessagePackObject] public class MsDcMp { [Key(0)] public int A { get; set; } }
[DataContract] public class MsDcOverMpBase : MsMpBase { }
[MessagePackObject(keyAsPropertyName: true)] public class MsPropName { public int A { get; set; } }
public class MsPropNameDerived : MsPropName { }
[MessagePackObject] public class MsStringKey { [Key("a")] public int A { get; set; } }
public sealed class MsNoFormatter { }
[MessagePackFormatter(typeof(MsNoFormatter))] [MessagePackObject] public class MsFormattedBase { [Key(0)] public int A { get; set; } }
[MessagePackObject] public class MsFormattedDerived : MsFormattedBase { [Key(1)] public int B { get; set; } }
}
}

View file

@ -62,28 +62,17 @@ namespace GameCult.Caching.Tests
}
[Test]
public void DocsShowTheTestedSnippet() => StartDocs.AssertShowsSnippet(SourcePath());
static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path;
}
// docs/start/README.md shows each tested snippet verbatim: the region between snippet:begin and snippet:end of its source file.
internal static class StartDocs
{
public static void AssertShowsSnippet(string sourcePath) =>
AssertShowsSnippet(sourcePath, Path.Combine("..", "..", "docs", "start", "README.md"));
// The same check against another document, its path relative to the directory of sourcePath. The snippet there is the
// statements of a block, so it need not open the fence (the block's usings come first).
public static void AssertShowsSnippet(string sourcePath, string docsRelativePath, bool wholeBlock = true)
public void DocsShowTheTestedSnippet()
{
var source = File.ReadAllText(sourcePath).Replace("\r\n", "\n");
var source = File.ReadAllText(SourcePath());
const string begin = "// snippet:begin\n";
var start = source.IndexOf(begin, StringComparison.Ordinal) + begin.Length;
var snippet = source[start..source.IndexOf("// snippet:end\n", StringComparison.Ordinal)];
var docs = File.ReadAllText(Path.Combine(Path.GetDirectoryName(sourcePath)!, docsRelativePath));
Assert.That(docs.Replace("\r\n", "\n"), Does.Contain(wholeBlock ? "```csharp\n" + snippet + "```" : snippet),
$"{docsRelativePath} no longer shows {Path.GetFileName(sourcePath)} verbatim");
var docs = File.ReadAllText(Path.Combine(Path.GetDirectoryName(SourcePath())!, "..", "..", "docs", "start", "README.md"));
Assert.That(docs.Replace("\r\n", "\n"), Does.Contain("```csharp\n" + snippet + "```"),
"docs/start/README.md no longer shows tests/GameCult.Caching.Tests/FirstCallTests.cs verbatim");
}
static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path;
}
}

View file

@ -1,81 +0,0 @@
#nullable enable
using System;
using System.IO;
using System.Threading.Tasks;
using GameCult.Caching;
using GameCult.Caching.MessagePack;
using MessagePack;
using NUnit.Framework;
namespace GameCult.Caching.Tests
{
// The Happy Path of src/GameCult.Caching/README.md, run as written: its statements are the region between snippet:begin and
// snippet:end below, and ReadmeShowsTheTestedSnippet fails when the README stops showing them verbatim.
public class ReadmeHappyPathTests
{
[CultDocument("gamecult.item_data", "gamecult.item_data.v1")]
[MessagePackObject]
public sealed class ItemData
{
[Key(0)] [CultName] public string Name = string.Empty;
[Key(1)] [CultIndex] public string Sku = string.Empty;
[Key(2)] public string Category = string.Empty;
[Key(3)] public int Value;
}
private static async Task<CultCache> HappyPath()
{
// snippet:begin
var cache = await CultCacheMessagePack.OpenAsync("Data.msgpack");
var key = new CultRecordKey("potion");
// The golden path: read, change a private copy, commit only if the record is unchanged, re-run on a lost race.
// Durable when it returns; no flush.
var result = cache.Update<ItemData>(key, item =>
{
item ??= new ItemData { Name = "Potion", Sku = "POT-1", Category = "Consumable" };
item.Value += 10;
return item;
});
// The blind opt-out: last writer wins per key. Name the key, or a new key claims the same unique Sku and is refused.
var handle = await cache.UpsertAsync(new ItemData
{
Name = "Potion",
Sku = "POT-1",
Category = "Consumable",
Value = 50
}, new CultRecordHandle<ItemData>(key));
await cache.FlushAsync();
// snippet:end
return cache;
}
[Test]
public async Task HappyPathRunsAsWritten()
{
var directory = Path.Combine(Path.GetTempPath(), $"readme-happy-{Guid.NewGuid():N}");
Directory.CreateDirectory(directory);
var previous = Environment.CurrentDirectory;
try
{
Environment.CurrentDirectory = directory; // the README opens a relative path
using var cache = await HappyPath();
Assert.That(cache.Get<ItemData>(new CultRecordKey("potion"))!.Value, Is.EqualTo(50));
Assert.That(cache.GetByIndex<ItemData>("Sku", "POT-1"), Is.Not.Null);
}
finally
{
Environment.CurrentDirectory = previous;
Directory.Delete(directory, recursive: true);
}
}
[Test]
public void ReadmeShowsTheTestedSnippet() =>
StartDocs.AssertShowsSnippet(SourcePath(), Path.Combine("..", "..", "src", "GameCult.Caching", "README.md"), wholeBlock: false);
static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path;
}
}

View file

@ -1,350 +0,0 @@
#nullable enable
using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using GameCult.Caching.MessagePack;
using NUnit.Framework;
using static GameCult.Caching.Tests.ElementIdTests;
using static GameCult.Caching.Tests.StoreReadabilityTests;
namespace GameCult.Caching.Tests
{
// A write never replaces a link at a store-owned path. The refusal holds for every way a write reaches a store (waiting or
// not, as a judge of another store's commit), for a link put there while the write waited on the store's lock, and for the
// directory store's manifest.
public class StoreLinkRefusalTests
{
private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(IdDeck), typeof(VectorItem) });
private string _directory = "";
[SetUp]
public void SetUp()
{
_directory = Path.Combine(Path.GetTempPath(), "cultlib-link-refusal-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(_directory);
}
[TearDown]
public void TearDown()
{
if (!Directory.Exists(_directory))
return;
foreach (var entry in Directory.EnumerateFileSystemEntries(_directory).Where(IsALink))
File.Delete(entry);
Directory.Delete(_directory, true);
}
private string PathOf(string name) => Path.Combine(_directory, name);
private static bool IsALink(string path)
{
var attributes = new FileInfo(path).Attributes;
return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
}
private static void Link(string path, string target)
{
try
{
File.CreateSymbolicLink(path, target);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Assert.Ignore("A file symlink needs a privilege this account lacks: " + ex.Message);
}
}
private static void SeedSingle(string path)
{
using var seed = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry });
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed")));
}
private static void SeedItem(string path)
{
using var seed = new CultCache(Registry);
seed.AddBackingStore(new SingleFileMessagePackBackingStore(path), typeof(VectorItem));
seed.Commit(batch => batch.Upsert(typeof(VectorItem), new VectorItem(), new CultRecordKey("item")));
}
// Runs `write` while another holder has `lockPath`; `swap` runs while it waits, then the lock is released.
private static Exception? WriteWhileWaiting(string lockPath, Action write, Action swap)
{
using var held = new FileStream(lockPath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
var task = Task.Run(write);
Assert.That(task.Wait(TimeSpan.FromMilliseconds(600)), Is.False, "the lock is held, so the write must wait");
swap();
held.Dispose();
try
{
Assert.That(task.Wait(TimeSpan.FromSeconds(10)), Is.True);
return null;
}
catch (AggregateException ex)
{
return ex.GetBaseException();
}
}
private static void ReplaceWithLink(string path, string target)
{
File.Delete(path);
Link(path, target);
}
// ---- a link put at the store path while the write waited on the lock ----
[Test]
public void ACommitRefusesALinkPutAtTheStoreWhileItWaitedOnTheLock()
{
var path = PathOf("store.cc");
var target = PathOf("shared.cc");
SeedSingle(path);
SeedSingle(target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
var error = WriteWhileWaiting(path + ".lock",
() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late"))),
() => ReplaceWithLink(path, target));
Assert.That(error, Is.InstanceOf<CultStoreLinkedException>(), error?.ToString());
Assert.That(((CultStoreLinkedException)error!).Path, Is.EqualTo(path));
Assert.That(IsALink(path), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
}
[Test]
public void AFlushRefusesALinkPutAtTheStoreWhileItWaitedOnTheLock()
{
var path = PathOf("store.cc");
var target = PathOf("shared.cc");
SeedSingle(path);
SeedSingle(target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late")).GetAwaiter().GetResult();
var error = WriteWhileWaiting(path + ".lock", () => cache.FlushAllBackingStores(), () => ReplaceWithLink(path, target));
Assert.That(error, Is.InstanceOf<CultStoreLinkedException>(), error?.ToString());
Assert.That(IsALink(path), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
}
// ---- the non-waiting and judging write paths ----
[Test]
public void ATryCommitAtALiveLinkIsRefusedNotContendedOrReplaced()
{
var path = PathOf("store.cc");
var link = PathOf("link.cc");
SeedSingle(path);
Link(link, path);
var before = File.ReadAllBytes(path);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
var refusal = Assert.Throws<CultStoreLinkedException>(() =>
cache.TryCommit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("t"))))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(refusal.Message, Does.Contain(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
Assert.That(File.Exists(link + ".lock"), Is.False, "the refused write took the lock");
}
[Test]
public void ATryCommitAtALiveLinkWhoseLockIsHeldIsStillRefused()
{
var path = PathOf("store.cc");
var link = PathOf("link.cc");
SeedSingle(path);
Link(link, path);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
using var held = new FileStream(link + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
Assert.Throws<CultStoreLinkedException>(() =>
cache.TryCommit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("t"))));
Assert.That(IsALink(link), "the link was replaced");
}
[Test]
public void AnAdvisoryQueryOverACacheAtALiveLinkAnswersForTheRecordAndTheWriteIsStillRefused()
{
var path = PathOf("store.cc");
var link = PathOf("link.cc");
SeedSingle(path);
Link(link, path);
var before = File.ReadAllBytes(path);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
Assert.That(cache.IsWritable(new CultRecordKey("seed")), Is.True, "the query does not look at the store path");
Assert.That(cache.IsWritable(new CultRecordKey("fresh")), Is.True, "the query does not look at the store path");
var refusal = Assert.Throws<CultStoreLinkedException>(() =>
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "t" }, new CultRecordKey("fresh"))))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
}
// A batch that upserts into one store and removes a key the cache does not hold has the other store judge that removal
// under its own lock (JudgeChecks).
private CultCache OpenTwoStores(string homePath, string otherPath)
{
var cache = new CultCache(Registry);
cache.AddBackingStore(new SingleFileMessagePackBackingStore(homePath), typeof(IdDeck));
cache.AddBackingStore(new SingleFileMessagePackBackingStore(otherPath), typeof(VectorItem));
return cache;
}
[Test]
public void AnotherStoreAtALiveLinkRefusesTheChecksItIsAskedToJudge()
{
var other = PathOf("other.cc");
var link = PathOf("other-link.cc");
SeedItem(other);
Link(link, other);
var before = File.ReadAllBytes(other);
using var cache = OpenTwoStores(PathOf("home.cc"), link);
var refusal = Assert.Throws<CultStoreLinkedException>(() => cache.Commit(batch =>
{
batch.Upsert(typeof(IdDeck), new IdDeck { Name = "home" }, new CultRecordKey("home"));
batch.Remove(new CultRecordKey("ghost"));
}))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(other), Is.EqualTo(before));
Assert.That(File.Exists(PathOf("home.cc")), Is.False, "the home store was written though a judge refused");
}
[Test]
public void AJudgeReadsTheFileItHoldsUnderTheLockNotTheFileBeforeItWaited()
{
var other = PathOf("other.cc");
SeedItem(other);
using var cache = OpenTwoStores(PathOf("home.cc"), other);
var error = WriteWhileWaiting(other + ".lock",
() => cache.Commit(batch =>
{
batch.Upsert(typeof(IdDeck), new IdDeck { Name = "home" }, new CultRecordKey("home"));
batch.Remove(new CultRecordKey("ghost"));
}),
() => File.WriteAllBytes(other, Array.Empty<byte>()));
Assert.That(error, Is.InstanceOf<CultStoreUnreadableException>(), error?.ToString());
}
// ---- the directory store's manifest ----
private DirectoryMessagePackBackingStore SeedDirectory(string manifest)
{
using (var seed = new CultCache(Registry))
{
seed.AddBackingStore(new DirectoryMessagePackBackingStore(manifest), typeof(IdDeck));
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed")));
}
return new DirectoryMessagePackBackingStore(manifest);
}
private CultCache OpenDirectory(DirectoryMessagePackBackingStore store)
{
var cache = new CultCache(Registry);
cache.AddBackingStore(store, typeof(IdDeck));
return cache;
}
[Test]
public void ADirectoryStoreCommitAtALiveManifestLinkIsRefusedAndNothingChanges()
{
var manifest = PathOf("real.cc");
var link = PathOf("link.cc");
SeedDirectory(manifest);
Link(link, manifest);
Directory.Move(manifest + ".records", link + ".records");
var before = File.ReadAllBytes(manifest);
var pages = Directory.GetFileSystemEntries(link + ".records").OrderBy(entry => entry).ToArray();
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
var refusal = Assert.Throws<CultStoreLinkedException>(() =>
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))))!;
Assert.That(refusal.Path, Is.EqualTo(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(manifest), Is.EqualTo(before));
Assert.That(Directory.GetFileSystemEntries(link + ".records").OrderBy(entry => entry), Is.EqualTo(pages), "the refused write left files beside the manifest");
}
// The refusal comes before the records folder or the lease file is made: a store that was never written leaves nothing behind.
[Test]
public void ADirectoryStoreCommitAtAManifestLinkPutThereAfterOpenCreatesNothing()
{
var link = PathOf("link.cc");
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
Link(link, PathOf("unmounted.cc"));
Assert.Throws<CultStoreLinkedException>(() =>
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(Directory.Exists(link + ".records"), Is.False, "the refused write made the records folder or its lease");
}
[Test]
public void ADirectoryStoreFlushAtALiveManifestLinkIsRefused()
{
var manifest = PathOf("real.cc");
var link = PathOf("link.cc");
SeedDirectory(manifest);
Link(link, manifest);
Directory.Move(manifest + ".records", link + ".records");
var before = File.ReadAllBytes(manifest);
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "f" }, new CultRecordKey("f")).GetAwaiter().GetResult();
Assert.Throws<CultStoreLinkedException>(() => cache.FlushAllBackingStores());
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(manifest), Is.EqualTo(before));
}
[Test]
public void ADirectoryStoreCommitRefusesAManifestLinkPutThereWhileItWaitedOnTheLease()
{
var manifest = PathOf("store.cc");
var target = PathOf("shared.cc");
SeedDirectory(manifest);
SeedDirectory(target);
var before = File.ReadAllBytes(target);
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(manifest));
var error = WriteWhileWaiting(Path.Combine(manifest + ".records", ".commit.lock"),
() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "late" }, new CultRecordKey("late"))),
() => ReplaceWithLink(manifest, target));
Assert.That(error, Is.InstanceOf<CultStoreLinkedException>(), error?.ToString());
Assert.That(IsALink(manifest), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
}
[Test]
public void ADirectoryStoreReadsThroughALiveManifestLink()
{
var manifest = PathOf("real.cc");
var link = PathOf("link.cc");
SeedDirectory(manifest);
Link(link, manifest);
Directory.Move(manifest + ".records", link + ".records");
using var cache = OpenDirectory(new DirectoryMessagePackBackingStore(link));
Assert.That(cache.Get<IdDeck>(new CultRecordKey("seed"))?.Name, Is.EqualTo("seed"));
}
}
}

View file

@ -266,7 +266,7 @@ namespace GameCult.Caching.Tests
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry });
CreateDanglingDirectoryLink(path, Path.Combine(_directory, "unmounted"));
Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<CultStoreLinkedException>());
Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<IOException>());
Assert.That(IsALink(path), "the link was replaced");
Assert.That(Directory.Exists(Path.Combine(_directory, "unmounted")), Is.False, "a write went through the link");
}
@ -292,66 +292,11 @@ namespace GameCult.Caching.Tests
using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry });
File.Delete(target);
Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<CultStoreLinkedException>());
Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))), Throws.TypeOf<IOException>());
Assert.That(new FileInfo(path).LinkTarget, Is.Not.Null, "the link was replaced");
Assert.That(File.Exists(target), Is.False, "a write went through the link");
}
// A write at a store path that is a link is refused with a typed error before the lock or the replace: the link, its target
// and the directory beside them are as they were.
[Test]
public void ACacheCommitAtALiveLinkIsRefusedAndNothingChanges()
{
var target = Path.Combine(_directory, "target.cc");
var link = Path.Combine(_directory, "link.cc");
using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry }))
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a")));
CreateFileLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
var refusal = Assert.Throws<CultStoreLinkedException>(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b"))))!;
Assert.That(refusal.Message, Does.Contain(link));
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
Assert.That(File.Exists(link + ".lock"), Is.False, "the refused write took the lock");
Assert.That(Directory.GetFileSystemEntries(_directory).Where(entry => entry.EndsWith(".tmp")), Is.Empty);
}
[Test]
public void AFlushAtALiveLinkIsRefusedAndNothingChanges()
{
var target = Path.Combine(_directory, "target.cc");
var link = Path.Combine(_directory, "link.cc");
using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry }))
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a")));
CreateFileLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "b" }, new CultRecordKey("b")).GetAwaiter().GetResult();
Assert.Throws<CultStoreLinkedException>(() => cache.FlushAllBackingStores());
Assert.That(IsALink(link), "the link was replaced");
Assert.That(File.ReadAllBytes(target), Is.EqualTo(before));
Assert.That(File.Exists(link + ".lock"), Is.False);
}
[Test]
public void AnOpenAtALiveLinkReadsTheTargetsRecords()
{
var target = Path.Combine(_directory, "target.cc");
var link = Path.Combine(_directory, "link.cc");
using (var seed = CultCacheMessagePack.Create(target, new CultCacheOpenOptions { Registry = Registry }))
seed.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "a" }, new CultRecordKey("a")));
CreateFileLinkOrIgnore(link, target);
using var cache = CultCacheMessagePack.Create(link, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
Assert.That(cache.Get<IdDeck>(new CultRecordKey("a"))?.Name, Is.EqualTo("a"));
}
// A directory at the path is something at the path: it is an I/O error on open and on commit, never an empty store, and
// it is left as it was.
[Test]

View file

@ -1,268 +0,0 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
using NUnit.Framework;
namespace GameCult.Caching.Tests
{
public class StoreReplaceTests
{
private sealed class FakeKernel : IReplaceKernel
{
private readonly Queue<int> _posix;
private readonly Queue<int> _moveEx;
public readonly List<string> Log = new List<string>();
public readonly List<int> Sleeps = new List<int>();
public Exception? PosixThrows;
public Exception? MoveExThrows;
public FakeKernel(int[] posix, int[] moveEx)
{
_posix = new Queue<int>(posix);
_moveEx = new Queue<int>(moveEx);
}
public int RenamePosix(string staged, string destination)
{
Log.Add("posix");
if (PosixThrows != null) throw PosixThrows;
return _posix.Count > 0 ? _posix.Dequeue() : 0;
}
public int MoveFileEx(string staged, string destination, uint flags)
{
Log.Add("moveex:" + flags);
if (MoveExThrows != null) throw MoveExThrows;
return _moveEx.Count > 0 ? _moveEx.Dequeue() : 0;
}
public void Sleep(int milliseconds) => Sleeps.Add(milliseconds);
}
private static IOException Fails(FakeKernel kernel)
=> Assert.Throws<IOException>(() => StoreReplace.ReplaceWindows("a", "b", kernel))!;
[Test]
public void APosixRenameThatWorksIsTheWholeReplace()
{
var kernel = new FakeKernel(new[] { 0 }, new int[0]);
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix" }));
Assert.That(kernel.Sleeps, Is.Empty);
}
[TestCase(1)]
[TestCase(50)]
[TestCase(87)]
public void WhereAPosixRenameIsUnsupportedTheReplaceFallsBackToWriteThroughMoveFileEx(int unsupported)
{
var kernel = new FakeKernel(new[] { unsupported }, new[] { 0 });
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
}
[Test]
public void OnceThePosixRenameIsUnsupportedTheRestOfTheCallStaysOnMoveFileEx()
{
var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 0 });
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" }));
Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20 }));
}
[TestCase(5)]
[TestCase(32)]
[TestCase(33)]
public void ATransientErrorRetriesTheSamePrimitiveAfterTwentyMilliseconds(int transient)
{
var kernel = new FakeKernel(new[] { transient, transient, 0 }, new int[0]);
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "posix", "posix" }));
Assert.That(kernel.Sleeps, Is.EqualTo(new[] { 20, 20 }));
}
[TestCase(1)]
[TestCase(50)]
[TestCase(87)]
public void AnUnsupportedCodeAfterTheFallbackIsAnErrorNotAnotherFallback(int unsupported)
{
var kernel = new FakeKernel(new[] { 87 }, new[] { unsupported });
var error = Fails(kernel);
Assert.That(error.Message, Does.Contain("Win32 error " + unsupported));
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
}
private static IEnumerable<Exception> Unresolved()
{
yield return new DllNotFoundException("kernel32.dll");
yield return new EntryPointNotFoundException("SetFileInformationByHandle");
}
[TestCaseSource(nameof(Unresolved))]
public void APosixCallThatDoesNotResolveTakesTheUnsupportedFallback(Exception unresolved)
{
var kernel = new FakeKernel(new int[0], new[] { 0 }) { PosixThrows = unresolved };
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
}
[TestCaseSource(nameof(Unresolved))]
public void AMoveFileExThatDoesNotResolveDegradesToThePortableReplace(Exception unresolved)
{
var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
var destination = Path.Combine(dir, "store.cc");
var staged = Path.Combine(dir, "staged");
File.WriteAllBytes(destination, new byte[] { 1 });
File.WriteAllBytes(staged, new byte[] { 2 });
var kernel = new FakeKernel(new[] { 87 }, new int[0]) { MoveExThrows = unresolved };
StoreReplace.ReplaceWindows(staged, destination, kernel);
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9" }));
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 }));
Assert.That(File.Exists(staged), Is.False);
// Both calls unresolved: the same portable replace, here onto an absent destination.
File.Delete(destination);
File.WriteAllBytes(staged, new byte[] { 3 });
var both = new FakeKernel(new int[0], new int[0]) { PosixThrows = unresolved, MoveExThrows = unresolved };
StoreReplace.ReplaceWindows(staged, destination, both);
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 }));
}
finally { Directory.Delete(dir, true); }
}
[Test]
public void TheFallbackRunsInTheSameAttemptEvenOnTheLastOne()
{
var posix = new int[25].Fill(32);
posix[24] = 87;
var kernel = new FakeKernel(posix, new[] { 0 });
StoreReplace.ReplaceWindows("a", "b", kernel);
Assert.That(kernel.Log.Count, Is.EqualTo(26));
Assert.That(kernel.Log[24], Is.EqualTo("posix"));
Assert.That(kernel.Log[25], Is.EqualTo("moveex:9"));
}
[Test]
public void ARetriedMoveFileExThatAnswersUnsupportedFailsAfterOneCallNotTwo()
{
var kernel = new FakeKernel(new[] { 87 }, new[] { 32, 87 });
Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 87"));
Assert.That(kernel.Log, Is.EqualTo(new[] { "posix", "moveex:9", "moveex:9" }));
}
[Test]
public void ARetryThatNeverSucceedsGivesUpAfterTwentyFiveAttemptsNamingTheCode()
{
var kernel = new FakeKernel(new int[30].Fill(32), new int[0]);
var error = Fails(kernel);
Assert.That(error.Message, Does.Contain("25 attempts").And.Contain("last Win32 error 32"));
Assert.That(kernel.Log.Count, Is.EqualTo(25));
Assert.That(kernel.Sleeps.Count, Is.EqualTo(25));
}
[Test]
public void AnyOtherCodeThrowsAtOnceNamingTheCode()
{
var kernel = new FakeKernel(new[] { 2 }, new int[0]);
var error = Fails(kernel);
Assert.That(error.Message, Does.Contain("Win32 error 2"));
Assert.That(kernel.Log.Count, Is.EqualTo(1));
Assert.That(kernel.Sleeps, Is.Empty);
}
[Test]
public void AFallbackThatFailsOtherwiseThrowsItsCode()
{
var kernel = new FakeKernel(new[] { 87 }, new[] { 3 });
Assert.That(Fails(kernel).Message, Does.Contain("Win32 error 3"));
}
[Test]
public void ReplaceOverAnExistingAndThenAnAbsentDestinationLeavesTheStagedBytes()
{
var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
var destination = Path.Combine(dir, "store.cc");
var staged = Path.Combine(dir, "staged");
File.WriteAllBytes(destination, new byte[] { 1 });
File.WriteAllBytes(staged, new byte[] { 2 });
StoreReplace.Replace(staged, destination);
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 }));
Assert.That(File.Exists(staged), Is.False);
File.Delete(destination);
File.WriteAllBytes(staged, new byte[] { 3 });
StoreReplace.Replace(staged, destination);
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 3 }));
}
finally { Directory.Delete(dir, true); }
}
[Test]
public void OnWindowsAReplaceOverADestinationAReaderHoldsOpenSucceedsOnTheFirstAttempt()
{
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
Assert.Ignore("The real kernel runs on Windows only.");
var dir = Path.Combine(TestContext.CurrentContext.WorkDirectory, "storereplace-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(dir);
try
{
var destination = Path.Combine(dir, "store.cc");
var staged = Path.Combine(dir, "staged");
File.WriteAllBytes(destination, new byte[] { 1 });
File.WriteAllBytes(staged, new byte[] { 2 });
using (new FileStream(destination, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete))
{
var kernel = new CountingKernel(StoreReplace.WindowsKernel);
StoreReplace.ReplaceWindows(staged, destination, kernel);
Assert.That(kernel.PosixCalls, Is.EqualTo(1));
Assert.That(kernel.MoveExCalls, Is.EqualTo(0));
Assert.That(kernel.Sleeps, Is.EqualTo(0));
Assert.That(kernel.LastCode, Is.EqualTo(0));
}
Assert.That(File.ReadAllBytes(destination), Is.EqualTo(new byte[] { 2 }));
}
finally { Directory.Delete(dir, true); }
}
private sealed class CountingKernel : IReplaceKernel
{
private readonly IReplaceKernel _inner;
public int PosixCalls, MoveExCalls, Sleeps, LastCode;
public CountingKernel(IReplaceKernel inner) { _inner = inner; }
public int RenamePosix(string staged, string destination)
{
PosixCalls++;
return LastCode = _inner.RenamePosix(staged, destination);
}
public int MoveFileEx(string staged, string destination, uint flags)
{
MoveExCalls++;
return LastCode = _inner.MoveFileEx(staged, destination, flags);
}
public void Sleep(int milliseconds) { Sleeps++; _inner.Sleep(milliseconds); }
}
}
internal static class ArrayFill
{
public static int[] Fill(this int[] array, int value)
{
for (var i = 0; i < array.Length; i++) array[i] = value;
return array;
}
}
}

View file

@ -1,132 +0,0 @@
#nullable enable
using System;
using System.IO;
using System.Threading.Tasks;
using GameCult.Caching.MessagePack;
using NUnit.Framework;
using static GameCult.Caching.Tests.ElementIdTests;
namespace GameCult.Caching.Tests
{
// A write decides from the file it holds the lock over, not from the file as it was before it waited: whatever another
// writer left while this one waited for <path>.lock is what the write applies onto, or refuses.
public class StoreWriteReadsUnderTheLockTests
{
private static readonly CultDocumentRegistry Registry = CultDocumentRegistry.ForTypes(new[] { typeof(IdDeck) });
private static readonly CultSchemaCatalogEntry One = new() { SchemaId = "tests.under-lock.one", SchemaName = "tests.under-lock.one" };
private string _directory = "";
private string _path = "";
[SetUp]
public void SetUp()
{
_directory = Path.Combine(Path.GetTempPath(), "cultlib-under-lock-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(_directory);
_path = Path.Combine(_directory, "store.cc");
}
[TearDown]
public void TearDown()
{
if (Directory.Exists(_directory))
Directory.Delete(_directory, true);
}
private void Seed()
{
using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry });
cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "seed" }, new CultRecordKey("seed")));
}
// The write starts and waits on the held lock; while it waits, the file is replaced by what `swap` makes of it.
private Exception? WriteWhileTheFileChangesUnderIt(Action write, Action swap)
{
using var held = new FileStream(_path + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
var task = Task.Run(write);
Assert.That(task.Wait(TimeSpan.FromMilliseconds(600)), Is.False, "the lock is held, so the write must wait");
swap();
held.Dispose();
try
{
Assert.That(task.Wait(TimeSpan.FromSeconds(10)), Is.True);
return null;
}
catch (AggregateException ex)
{
return ex.GetBaseException();
}
}
private static void ZeroBytes(string path) => File.WriteAllBytes(path, Array.Empty<byte>());
private static CultPersistedStoreSnapshot Snapshot(string path) =>
CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
private void MarkV3()
{
var snapshot = Snapshot(_path);
snapshot.FormatVersion = CultPersistedStoreSnapshot.FormatV3;
File.WriteAllBytes(_path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot));
}
private void CommitWrite(Action swap, out Exception? error)
{
using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
error = WriteWhileTheFileChangesUnderIt(
() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "committed" }, new CultRecordKey("committed"))), swap);
}
private void FlushWriteWhile(Action swap, out Exception? error)
{
using var cache = CultCacheMessagePack.Create(_path, new CultCacheOpenOptions { Registry = Registry, FlushOnDispose = false, StoreFlushOnDispose = false });
cache.UpsertAsync(typeof(IdDeck), new IdDeck { Name = "flushed" }, new CultRecordKey("flushed")).GetAwaiter().GetResult();
error = WriteWhileTheFileChangesUnderIt(() => cache.FlushAllBackingStores(), swap);
}
private void CachelessWriteWhile(Action swap, out Exception? error) =>
error = WriteWhileTheFileChangesUnderIt(
() => new SingleFileMessagePackBackingStore(_path).WriteRecord(
new CultPersistedRecord { Key = "written", SchemaId = One.SchemaId, StoredAt = "2026-10-22T00:00:00.0000000+00:00", Payload = new byte[] { 1 } }, One),
swap);
private delegate void WriteWhile(Action swap, out Exception? error);
[Test]
public void AFlushDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(FlushWriteWhile);
[Test]
public void ACommitDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(CommitWrite);
[Test]
public void ACachelessWriteDecidesFromTheFileHeldUnderTheLock() => RefusesWhatItFindsUnderTheLock(CachelessWriteWhile);
[Test]
public void AFlushKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(FlushWriteWhile);
[Test]
public void ACommitKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(CommitWrite);
[Test]
public void ACachelessWriteKeepsTheIdHeaderOfTheFileHeldUnderTheLock() => KeepsTheV3HeaderItFindsUnderTheLock(CachelessWriteWhile);
private void RefusesWhatItFindsUnderTheLock(WriteWhile write)
{
Seed();
write(() => ZeroBytes(_path), out var error);
Assert.That(error, Is.InstanceOf<CultStoreUnreadableException>());
Assert.That(new FileInfo(_path).Length, Is.Zero, "the refused file was written over");
}
private void KeepsTheV3HeaderItFindsUnderTheLock(WriteWhile write)
{
Seed();
write(MarkV3, out var error);
Assert.That(error, Is.Null, error?.ToString());
Assert.That(Snapshot(_path).FormatVersion, Is.EqualTo(CultPersistedStoreSnapshot.FormatV3), "the write read the file as it was before the lock");
}
}
}

View file

@ -1,54 +0,0 @@
#nullable enable
using System;
using System.IO;
using NUnit.Framework;
// snippet:begin
using GameCult.Caching;
using GameCult.Caching.MessagePack;
public static class UpdateCall
{
public static CultUpdateResult<Item> Restock(string path, CultRecordKey key)
{
var cache = CultCacheMessagePack.Create(path);
return cache.Update<Item>(key, item =>
{
item ??= new Item { Name = "Potion", Category = "Consumable" };
item.Value += 10;
return item;
});
}
}
// snippet:end
namespace GameCult.Caching.Tests
{
// The C# update snippet that docs/start/README.md shows verbatim; it reuses FirstCallTests' Item.
public class UpdateCallTests
{
[Test]
public void RestockTwiceReadsTwenty()
{
var path = Path.Combine(Path.GetTempPath(), $"update-call-{Guid.NewGuid():N}.cc");
try
{
var key = new CultRecordKey("potion");
Assert.That(UpdateCall.Restock(path, key).Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
Assert.That(UpdateCall.Restock(path, key).Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
using var reader = CultCacheMessagePack.Create(path);
Assert.That(reader.Get<Item>(key)!.Value, Is.EqualTo(20));
}
finally
{
File.Delete(path);
}
}
[Test]
public void DocsShowTheTestedSnippet() => StartDocs.AssertShowsSnippet(SourcePath());
static string SourcePath([System.Runtime.CompilerServices.CallerFilePath] string path = "") => path;
}
}

View file

@ -1,515 +0,0 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using GameCult.Caching;
using GameCult.Caching.MessagePack;
using MessagePack;
using NUnit.Framework;
using R3;
namespace GameCult.Caching.Tests
{
// CultCache.Update: read the content version, apply a pure function to a private copy, commit conditionally, re-run on a lost
// race (variants:ruling:write-set-content-version-declared-ownership; variants:question:update-apply-copy).
public class UpdateTests
{
private static readonly CultRecordKey Key = new("counter");
private static readonly CultDocumentRegistry Registry =
CultDocumentRegistry.ForTypes(new[] { typeof(UpdateCounter), typeof(UpdateTally), typeof(UpdateGear) });
private string _directory = string.Empty;
[SetUp]
public void CreateDirectory()
{
_directory = Path.Combine(Path.GetTempPath(), $"cultlib-update-{Guid.NewGuid():N}");
Directory.CreateDirectory(_directory);
}
[TearDown]
public void DeleteDirectory()
{
if (Directory.Exists(_directory))
Directory.Delete(_directory, recursive: true);
}
private string PathOf(string name) => Path.Combine(_directory, name);
private static CultCache Open(string path, bool directory, bool codec = true)
{
var cache = codec ? new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry)) : new CultCache(Registry);
cache.AddBackingStore(directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path));
return cache;
}
private static void Seed(string path, bool directory, int value)
{
using var cache = Open(path, directory);
Assert.That(cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = value }, new CultRecordHandle<UpdateCounter>(Key))), Is.True);
}
private static void Rival(CultCache cache, int value) =>
Assert.That(cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter", Value = value }, new CultRecordHandle<UpdateCounter>(Key))), Is.True);
private static int StoredValue(string path, bool directory)
{
using var reader = Open(path, directory);
return reader.Get<UpdateCounter>(Key)!.Value;
}
private static UpdateCounter Plus(UpdateCounter? counter, int by)
{
counter ??= new UpdateCounter { Name = "counter" };
counter.Value += by;
return counter;
}
[TestCase(false)]
[TestCase(true)]
public void TwoWritersUpdatingLoseNoIncrement(bool directory)
{
const int n = 100;
var path = PathOf("race.cc");
Seed(path, directory, 0);
using var a = Open(path, directory);
using var b = Open(path, directory);
var applied = 0;
var racers = new[] { a, b }.Select(cache => Task.Run(() =>
{
var results = new List<CultUpdateOutcome>();
for (var i = 0; i < n; i++)
results.Add(cache.Update<UpdateCounter>(Key, counter =>
{
Interlocked.Increment(ref applied);
return Plus(counter, 1);
}, maxAttempts: 200).Outcome);
return results;
})).ToArray();
Assert.That(Task.WaitAll(racers, TimeSpan.FromMinutes(2)), Is.True, "the racers did not finish");
Assert.That(racers.SelectMany(racer => racer.Result), Is.All.EqualTo(CultUpdateOutcome.Updated));
Assert.That(StoredValue(path, directory), Is.EqualTo(2 * n));
Assert.That(applied, Is.GreaterThan(2 * n), "the writers never actually raced");
}
[Test]
public void LostAttemptRerunsApplyOnTheFreshValue()
{
var path = PathOf("rerun.cc");
Seed(path, directory: false, 1);
using var cache = Open(path, false);
using var rival = Open(path, false);
var seen = new List<int>();
var result = cache.Update<UpdateCounter>(Key, counter =>
{
seen.Add(counter!.Value);
if (seen.Count == 1)
Rival(rival, 100);
return Plus(counter, 1);
});
Assert.That(seen, Is.EqualTo(new[] { 1, 100 }));
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
Assert.That(result.Document!.Value, Is.EqualTo(101));
Assert.That(StoredValue(path, false), Is.EqualTo(101));
}
[Test]
public void ExhaustedAttemptsReturnLostRace()
{
var path = PathOf("lost.cc");
Seed(path, false, 1);
using var cache = Open(path, false);
using var rival = Open(path, false);
var calls = 0;
var result = cache.Update<UpdateCounter>(Key, counter =>
{
Rival(rival, 100 * ++calls);
return Plus(counter, 1);
}, maxAttempts: 3);
Assert.That(calls, Is.EqualTo(3));
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace));
Assert.That(result.Document!.Value, Is.EqualTo(300));
Assert.That(StoredValue(path, false), Is.EqualTo(300), "the store holds the rival's last write");
}
[Test]
public void ApplyGetsAPrivateCopy()
{
var path = PathOf("copy.cc");
Seed(path, false, 1);
using var cache = Open(path, false);
using var rival = Open(path, false);
var published = new List<int>();
using var subscription = cache.Watch<UpdateCounter>().Subscribe(change => published.Add(change.Document!.Value));
var arguments = new List<int>();
var calls = 0;
var result = cache.Update<UpdateCounter>(Key, counter =>
{
calls++;
arguments.Add(counter!.Value);
Assert.That(counter, Is.Not.SameAs(cache.Get<UpdateCounter>(Key)), "apply was handed the held instance");
counter.Value += 1000;
Assert.That(cache.Get<UpdateCounter>(Key)!.Value, Is.Not.EqualTo(counter.Value), "the change reached the cache before any commit");
if (calls == 1)
Rival(rival, 5);
return counter;
});
Assert.That(arguments, Is.EqualTo(new[] { 1, 5 }), "the second call saw the first call's change");
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
Assert.That(published, Does.Not.Contain(1001), "an uncommitted value was published");
Assert.That(published.Count(value => value == 1005), Is.EqualTo(1));
Assert.That(StoredValue(path, false), Is.EqualTo(1005));
}
[Test]
public void NullFromApplyDeclines()
{
var path = PathOf("decline.cc");
Seed(path, false, 7);
var before = File.ReadAllBytes(path);
using var cache = Open(path, false);
var changes = 0;
using var subscription = cache.Watch<UpdateCounter>().Subscribe(_ => changes++);
var result = cache.Update<UpdateCounter>(Key, _ => null);
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Declined));
Assert.That(result.Document!.Value, Is.EqualTo(7));
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
Assert.That(changes, Is.EqualTo(0));
}
[TestCase(false)]
[TestCase(true)]
public void AbsentKeyCreatesOnceUnderRace(bool directory)
{
var path = PathOf("create.cc");
using var a = Open(path, directory);
using var b = Open(path, directory);
var started = new Barrier(2);
var racers = new[] { a, b }.Select(cache => Task.Run(() =>
{
started.SignalAndWait();
return cache.Update<UpdateCounter>(Key, counter => Plus(counter, 1)).Outcome;
})).ToArray();
Assert.That(Task.WaitAll(racers, TimeSpan.FromMinutes(1)), Is.True);
Assert.That(racers.Select(racer => racer.Result), Is.All.EqualTo(CultUpdateOutcome.Updated));
Assert.That(StoredValue(path, directory), Is.EqualTo(2));
}
[Test]
public void RefusalsAreNotRetried()
{
var path = PathOf("refuse.cc");
Seed(path, false, 1);
var before = File.ReadAllBytes(path);
var calls = 0;
Func<UpdateCounter?, UpdateCounter?> apply = counter => { calls++; return Plus(counter, 1); };
using (var codecLess = Open(path, false, codec: false))
Assert.That(() => codecLess.Update(Key, apply), Throws.InvalidOperationException);
using (var cache = Open(path, false))
{
Assert.That(() => cache.Update(Key, apply, maxAttempts: 0), Throws.TypeOf<ArgumentOutOfRangeException>());
Assert.That(() => cache.Update<UpdateCounter>(Key, null!), Throws.TypeOf<ArgumentNullException>());
Assert.That(() => cache.Update<UpdateTally>(Key, _ => new UpdateTally { Name = "counter" }), Throws.InvalidOperationException);
}
using (var cache = Open(path, false))
{
cache.Commit(batch =>
{
batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle<UpdateGear>(new CultRecordKey("gear")));
});
cache.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"),
new[] { cache.Override<UpdateGear>(nameof(UpdateGear.Name), "gear big"), cache.Override<UpdateGear>(nameof(UpdateGear.Power), 9) }));
}
before = File.ReadAllBytes(path);
var gearCalls = 0;
using (var cache = Open(path, false))
Assert.That(() => cache.Update<UpdateGear>(new CultRecordKey("gear-big"), gear => { gearCalls++; return gear; }), Throws.InvalidOperationException);
Assert.That(gearCalls, Is.EqualTo(0), "apply ran for a variant");
Assert.That(calls, Is.EqualTo(0), "apply ran for a refused call");
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
using (var cache = Open(path, false))
{
cache.UpsertAsync(new UpdateTally { Name = "staged" }).GetAwaiter().GetResult();
Assert.That(() => cache.Update(Key, apply), Throws.InvalidOperationException.With.Message.Contains("flush before a conditional commit"));
Assert.That(calls, Is.LessThanOrEqualTo(1));
}
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before), "a refusal wrote");
}
[Test]
public void VariantStoreConflictIsALostAttempt()
{
var path = PathOf("variant.cc");
Seed(path, false, 1);
using (var seed = Open(path, false))
{
seed.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle<UpdateGear>(new CultRecordKey("gear"))));
seed.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"),
new[] { seed.Override<UpdateGear>(nameof(UpdateGear.Name), "gear big"), seed.Override<UpdateGear>(nameof(UpdateGear.Power), 9) }));
}
using var cache = Open(path, false);
using (var rival = Open(path, false))
Assert.That(rival.Commit(batch => batch.Upsert(new UpdateTally { Name = "other" }, new CultRecordHandle<UpdateTally>(new CultRecordKey("other")))), Is.True);
var calls = 0;
var result = cache.Update<UpdateCounter>(Key, counter => { calls++; return Plus(counter, 1); });
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
Assert.That(calls, Is.EqualTo(2), "the conflict was not a lost attempt that re-ran apply");
Assert.That(StoredValue(path, false), Is.EqualTo(2));
}
private static CultPersistedStoreSnapshot ReadSnapshot(string path) =>
CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
// A reload that fails inside a conflict is a read failure, not a lost race: Update lets it propagate and never re-runs apply.
// Here the gear a variant is based on has gone foreign in the file, so the reload the conflict owes cannot drop it.
[Test]
public void ConflictWhoseReloadFailedPropagates()
{
var path = PathOf("reload-failed.cc");
Seed(path, false, 1);
using var cache = Open(path, false);
cache.Commit(batch => batch.Upsert(new UpdateGear { Name = "gear", Power = 1 }, new CultRecordHandle<UpdateGear>(new CultRecordKey("gear"))));
cache.Commit(batch => batch.UpsertVariant(new CultRecordKey("gear-big"), new CultRecordKey("gear"),
new[] { cache.Override<UpdateGear>(nameof(UpdateGear.Name), "gear big"), cache.Override<UpdateGear>(nameof(UpdateGear.Power), 9) }));
const string foreignId = "sha256:00000000000000000000000000000000000000000000000000000000000050de";
var snapshot = ReadSnapshot(path);
snapshot.Records.Single(record => record.Key == "gear").SchemaId = foreignId;
snapshot.SchemaCatalog = snapshot.SchemaCatalog.Append(new CultSchemaCatalogEntry
{
SchemaId = foreignId, SchemaName = "tests.update_foreign", SchemaVersion = "tests.update_foreign.v1",
ContentHash = "foreign", CanonicalSchemaJson = "{}"
}).ToArray();
File.WriteAllBytes(path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot));
var calls = 0;
var conflict = Assert.Throws<CultWriteConflictException>(() => cache.Update<UpdateCounter>(Key, counter => { calls++; return Plus(counter, 1); }))!;
Assert.That(conflict.InnerException, Is.Not.Null, "the reload that failed is the cause");
Assert.That(calls, Is.EqualTo(1), "a failed reload was retried");
}
[TestCase(false)]
[TestCase(true)]
public void RivalWritingAndRestoringTheValueStillRerunsApply(bool directory)
{
var path = PathOf("aba.cc");
Seed(path, directory, 1);
using var cache = Open(path, directory);
using var rival = Open(path, directory);
var seen = new List<int>();
var result = cache.Update<UpdateCounter>(Key, counter =>
{
seen.Add(counter!.Value);
if (seen.Count == 1)
{
Rival(rival, 5);
Rival(rival, 1);
}
return Plus(counter, 1);
});
Assert.That(seen, Is.EqualTo(new[] { 1, 1 }), "the content version missed A-B-A");
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
Assert.That(StoredValue(path, directory), Is.EqualTo(2));
}
[TestCase(false)]
[TestCase(true)]
public void RivalRemovalIsSeenAsNull(bool directory)
{
var path = PathOf("removal.cc");
Seed(path, directory, 1);
using var cache = Open(path, directory);
using var rival = Open(path, directory);
var seen = new List<int?>();
var result = cache.Update<UpdateCounter>(Key, counter =>
{
seen.Add(counter?.Value);
if (seen.Count == 1)
rival.Commit(batch => batch.Remove(Key));
return Plus(counter, 1);
});
Assert.That(seen, Is.EqualTo(new int?[] { 1, null }));
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
Assert.That(StoredValue(path, directory), Is.EqualTo(1));
}
[Test]
public void ExhaustedAttemptsReturnLostRaceInADirectoryStore()
{
var path = PathOf("lost-dir.cc");
Seed(path, true, 1);
using var cache = Open(path, true);
using var rival = Open(path, true);
var calls = 0;
var result = cache.Update<UpdateCounter>(Key, counter =>
{
Rival(rival, 100 * ++calls);
return Plus(counter, 1);
}, maxAttempts: 2);
Assert.That(calls, Is.EqualTo(2));
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace));
Assert.That(result.Document!.Value, Is.EqualTo(200));
Assert.That(StoredValue(path, true), Is.EqualTo(200));
}
[Test]
public void OneAttemptMeansOneApplyAndNonPositiveThrows()
{
var path = PathOf("one.cc");
Seed(path, false, 1);
using var cache = Open(path, false);
using var rival = Open(path, false);
var calls = 0;
var result = cache.Update<UpdateCounter>(Key, counter => { calls++; Rival(rival, 50); return Plus(counter, 1); }, maxAttempts: 1);
Assert.That(calls, Is.EqualTo(1));
Assert.That(result.Outcome, Is.EqualTo(CultUpdateOutcome.LostRace));
Assert.That(() => cache.Update<UpdateCounter>(Key, counter => counter, maxAttempts: -1), Throws.TypeOf<ArgumentOutOfRangeException>());
Assert.That(calls, Is.EqualTo(1));
}
[TestCase(false)]
[TestCase(true)]
public void UpdatedDocumentIsTheHeldInstanceAndOwnWritesCauseNoRetry(bool directory)
{
var path = PathOf("own.cc");
Seed(path, directory, 1);
using var cache = Open(path, directory);
cache.UpsertAsync(new UpdateCounter { Name = "counter", Value = 10 }, new CultRecordHandle<UpdateCounter>(Key)).GetAwaiter().GetResult();
cache.FlushAsync().GetAwaiter().GetResult();
var calls = 0;
var first = cache.Update<UpdateCounter>(Key, counter => { calls++; return Plus(counter, 1); });
var second = cache.Update<UpdateCounter>(Key, counter => { calls++; return Plus(counter, 1); });
Assert.That(calls, Is.EqualTo(2), "the cache's own writes made a later Update retry");
Assert.That(first.Document, Is.Not.Null);
Assert.That(second.Document, Is.SameAs(cache.Get<UpdateCounter>(Key)), "Updated returned a copy, not the held instance");
Assert.That(StoredValue(path, directory), Is.EqualTo(12));
}
[Test]
public void ApplyRunsOutsideTheCacheGate()
{
var path = PathOf("gate.cc");
Seed(path, false, 1);
using var cache = Open(path, false);
var reached = false;
cache.Update<UpdateCounter>(Key, counter =>
{
reached = Task.Run(() => cache.Get<UpdateCounter>(Key)).Wait(TimeSpan.FromSeconds(10));
return Plus(counter, 1);
});
Assert.That(reached, Is.True, "another thread could not read the cache while apply ran");
}
[TestCase(false)]
[TestCase(true)]
public void ContendedLedgerStoredValueEqualsUpdatedCount(bool directory)
{
var path = PathOf("ledger.cc");
Seed(path, directory, 0);
var caches = Enumerable.Range(0, 3).Select(_ => Open(path, directory)).ToArray();
try
{
var tasks = caches.Select(cache => Task.Run(() =>
{
var updated = 0;
for (var i = 0; i < 25; i++)
if (cache.Update<UpdateCounter>(Key, counter => Plus(counter, 1)).Outcome == CultUpdateOutcome.Updated)
updated++;
return updated;
})).ToArray();
Assert.That(Task.WaitAll(tasks, TimeSpan.FromMinutes(2)), Is.True);
Assert.That(StoredValue(path, directory), Is.EqualTo(tasks.Sum(task => task.Result)), "an Updated outcome was lost or a LostRace wrote");
}
finally
{
foreach (var cache in caches)
cache.Dispose();
}
}
[Test]
public void InMemoryCacheUpdatesUnderThreadRace()
{
using var cache = new CultCache(Registry, CultCacheMessagePack.CreateCodec(Registry));
cache.Commit(batch => batch.Upsert(new UpdateCounter { Name = "counter" }, new CultRecordHandle<UpdateCounter>(Key)));
var threads = Enumerable.Range(0, 2).Select(_ => Task.Run(() =>
{
for (var i = 0; i < 100; i++)
Assert.That(cache.Update<UpdateCounter>(Key, counter => Plus(counter, 1), maxAttempts: 200).Outcome, Is.EqualTo(CultUpdateOutcome.Updated));
})).ToArray();
Assert.That(Task.WaitAll(threads, TimeSpan.FromMinutes(1)), Is.True);
Assert.That(cache.Get<UpdateCounter>(Key)!.Value, Is.EqualTo(200));
}
[CultDocument("tests.update_counter", "tests.update_counter.v1")]
[MessagePackObject(AllowPrivate = true)]
internal sealed class UpdateCounter
{
[Key(0)]
[CultName]
public string Name = string.Empty;
[Key(1)]
public int Value;
}
[CultDocument("tests.update_tally", "tests.update_tally.v1")]
[MessagePackObject(AllowPrivate = true)]
internal sealed class UpdateTally
{
[Key(0)]
[CultName]
public string Name = string.Empty;
}
[CultDocument("tests.update_gear", "tests.update_gear.v1")]
[MessagePackObject(AllowPrivate = true)]
internal sealed class UpdateGear
{
[Key(0)]
[CultName]
public string Name = string.Empty;
[Key(1)]
public int Power;
}
}
}

View file

@ -1,11 +0,0 @@
{
"stryker-config": {
"project": "GameCult.Caching.csproj",
"mutate": [
"**/src/GameCult.Caching/**/*.cs"
],
"mutation-level": "Standard",
"reporters": [ "progress", "html", "json" ],
"thresholds": { "high": 80, "low": 60, "break": 0 }
}
}

View file

@ -4,7 +4,6 @@ using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
@ -365,11 +364,11 @@ public sealed class CultMeshSingleFileStoreTests
[Test]
public void ACacheOpeningDuringMeshWritesNeverSeesAnEmptyStore()
{
// Windows replaces with a POSIX-semantics rename first (StoreReplace), which makes a lock-free reader finding the path
// absent rare, not impossible: a stalled rename was measured leaving it absent for 5-47 ms on Windows. So on Windows a
// brief not-found (a read throwing, or a cache opening onto an absent path) is tolerated and reported; elsewhere rename
// is atomic and none is allowed.
var windows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows);
// Windows replaces with ReplaceFile, which leaves a moment with nothing at the path for a lock-free reader: 1,722 empty
// stores in 17,313 opens at b1ff2dcb. That is the store's replace, not Mesh's, and it is the same for a cache flush beside a
// cache open; the fork is in the cut report.
if (OperatingSystem.IsWindows())
Assert.Ignore("Windows File.Replace is not atomic to a lock-free reader; see cut report fork mesh-reader-windows-replace-window.");
var path = Path.Combine(_root, "race.cc");
CultMesh.WriteSingleFileDocument(path, Key, Publication());
@ -387,7 +386,7 @@ public sealed class CultMeshSingleFileStoreTests
Volatile.Write(ref stop, 1);
});
int opens = 0, empty = 0, reads = 0, absent = 0;
int opens = 0, empty = 0, reads = 0;
var clock = Stopwatch.StartNew();
while (Volatile.Read(ref stop) == 0 && clock.Elapsed < TimeSpan.FromSeconds(60))
{
@ -398,10 +397,6 @@ public sealed class CultMeshSingleFileStoreTests
CultMesh.ReadSingleFileDocumentPayload(path, Key, SchemaId());
reads++;
}
catch (Exception ex) when (windows && (ex is FileNotFoundException || ex is DirectoryNotFoundException))
{
absent++;
}
catch (Exception ex)
{
errors.Enqueue("read: " + ex.GetType().Name + ": " + ex.Message);
@ -409,10 +404,9 @@ public sealed class CultMeshSingleFileStoreTests
}
writer.Wait(TimeSpan.FromSeconds(60)).Should().BeTrue();
TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} not-found reads={absent} mesh reads ok={reads} errors={errors.Count}");
TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} mesh reads ok={reads} errors={errors.Count}");
errors.Take(3).Should().BeEmpty();
if (!windows)
empty.Should().Be(0, "a cache opened in the window where nothing was at the path");
empty.Should().Be(0, "a cache opened in the window where nothing was at the path");
opens.Should().BeGreaterThan(0);
}
@ -440,74 +434,32 @@ public sealed class CultMeshSingleFileStoreTests
TempFiles().Should().BeEmpty();
}
// A write at a store path that is a link is refused before the lock, the staging file and the replace: the link and its
// target stay as they were, and nothing is left beside them.
private void AssertRefusedAndUntouched(string link, string target, byte[] targetBefore, Action write)
{
var refusal = Assert.Throws<CultStoreLinkedException>(() => write())!;
refusal.Message.Should().Contain(link);
IsALink(link).Should().BeTrue("the link was replaced");
File.ReadAllBytes(target).Should().Equal(targetBefore);
File.Exists(link + ".lock").Should().BeFalse("the refused write took the lock");
TempFiles().Should().BeEmpty();
}
// The store's replace decides what a write leaves where a live link was, and a Mesh write leaves what a cache commit leaves:
// one replace, whatever the platform does with a link.
[Test]
public void AMeshWriteAtALiveLinkIsRefusedAndNothingChanges()
public void AMeshWriteOverALiveLinkLeavesWhatACacheCommitLeaves()
{
var target = Path.Combine(_root, "target.cc");
var link = Path.Combine(_root, "link.cc");
CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
CreateLiveLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
AssertRefusedAndUntouched(link, target, before, () => CultMesh.WriteSingleFileDocument(link, Key, Publication("second")));
AssertRefusedAndUntouched(link, target, before, () => CultMesh.WriteSingleFileDocumentPayload(link, Key, Raw, null, EmptyArray));
}
[Test]
public void AReadThroughALiveLinkStillWorks()
{
var target = Path.Combine(_root, "target.cc");
var link = Path.Combine(_root, "link.cc");
CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
CreateLiveLinkOrIgnore(link, target);
CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(link, Key).ProducerId.Should().Be("first");
RecordsSeenByACache(link).Should().Be(1);
}
// A link to a directory that is there is a directory, not an unreachable store; a write at it is a link refusal.
[Test]
public void ALiveDirectoryLinkIsADirectoryErrorOnReadAndALinkRefusalOnWrite()
{
var directory = Path.Combine(_root, "real-directory");
var link = Path.Combine(_root, "directory-link.cc");
Directory.CreateDirectory(directory);
CreateDirectoryLink(link, directory);
Assert.Catch<IOException>(() => CultMesh.ReadSingleFileDocumentPayload(link, Key, "raw:schema"))!.Message.Should().Contain("directory");
Assert.Catch<IOException>(() => RecordsSeenByACache(link))!.Message.Should().Contain("directory");
Assert.Throws<CultStoreLinkedException>(() => CultMesh.WriteSingleFileDocument(link, Key, Publication()));
Directory.GetFileSystemEntries(directory).Should().BeEmpty();
IsALink(link).Should().BeTrue();
}
private static void CreateDirectoryLink(string path, string target)
{
if (!OperatingSystem.IsWindows())
string Outcome(string name, Action<string> write)
{
Directory.CreateSymbolicLink(path, target);
return;
var target = Path.Combine(_root, name + "-target.cc");
var link = Path.Combine(_root, name + "-link.cc");
CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
CreateLiveLinkOrIgnore(link, target);
var before = File.ReadAllBytes(target);
write(link);
CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(link, Key).ProducerId.Should().Be("second", name);
return $"linkKept={IsALink(link)} targetChanged={!File.ReadAllBytes(target).SequenceEqual(before)}";
}
var mklink = Process.Start(new ProcessStartInfo("cmd.exe", $"/c mklink /J \"{path}\" \"{target}\"")
var viaMesh = Outcome("mesh", link => CultMesh.WriteSingleFileDocument(link, Key, Publication("second")));
var viaCache = Outcome("cache", link =>
{
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
})!;
mklink.WaitForExit();
Assert.That(mklink.ExitCode, Is.Zero, "mklink /J failed");
using var cache = CultCacheMessagePack.Create(link, ReadOnlyOpen);
cache.Commit(batch => batch.Upsert(typeof(CultMeshBodyPublicationDocument), Publication("second"), Key));
});
viaMesh.Should().Be(viaCache);
}
}

View file

@ -341,7 +341,7 @@ namespace GameCult.Networking.Tests
{
var path = Path.Combine(RepoRoot(), "contracts", "cultnet", "interop", "selection-vectors.rs-written.json");
if (!File.Exists(path))
Assert.Inconclusive($"{path} is missing - run cargo test --locked -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first.");
Assert.Inconclusive($"{path} is missing - run cargo test -p cultnet-rs --test selection with CULTNET_WRITE_VECTORS=1 first.");
var (registry, rows, _) = BuildFixture();
using var document = JsonDocument.Parse(File.ReadAllText(path));

View file

@ -111,7 +111,6 @@ probe() {
node ../../node_modules/typescript/bin/tsc -p tsconfig.test.json
node --test dist-test/test/old-reader-v3.test.js ;;
python) PYTHONPATH=packages/cultcache-py/src python3 packages/cultcache-py/tests/test_old_reader_v3.py -v ;;
# The staged tree is the frozen cultcache-rs-v0.3.1 revision, which has no committed Cargo.lock: --locked cannot work here.
rust) cd packages/cultcache-rs && cargo test --test old_reader_v3 -- --nocapture ;;
*) usage ;;
esac