From eda3920135957de444bfa9bcef3ea7615d6f04d0 Mon Sep 17 00:00:00 2001 From: Libby Date: Thu, 12 Nov 2026 10:58:51 +0000 Subject: [PATCH 1/4] Docs: state the single-file lock contract, leftover-temp and pair-backup procedures Replace the 'single-writer' wording with what the lock guarantees; drop two stale follow-up claims (Python takes the lock; Rust reads wait); fix the Rust README's 'external lock' advice. Co-Authored-By: Claude Sonnet 5.5 --- packages/cultcache-rs/README.md | 4 +- .../Contracts/cultcache-store-composition.md | 126 +++++++++++++++++- src/GameCult.Caching/README.md | 15 ++- 3 files changed, 136 insertions(+), 9 deletions(-) diff --git a/packages/cultcache-rs/README.md b/packages/cultcache-rs/README.md index da4185e2..2d217083 100644 --- a/packages/cultcache-rs/README.md +++ b/packages/cultcache-rs/README.md @@ -39,8 +39,8 @@ CultCache is a domain cache with persistence adapters. file access with a sidecar lock file. This is not an ORM, not a database, and not distributed consensus in a novelty -hat. If multiple processes write the same backing file, use an external lock or -a coordinator. +hat. Several processes may write the same single-file store, one at a time under +its sidecar lock; write ordering across keys or stores needs a coordinator. ## Current API diff --git a/src/GameCult.Caching/Contracts/cultcache-store-composition.md b/src/GameCult.Caching/Contracts/cultcache-store-composition.md index 01f7a5c9..88f86ab8 100644 --- a/src/GameCult.Caching/Contracts/cultcache-store-composition.md +++ b/src/GameCult.Caching/Contracts/cultcache-store-composition.md @@ -315,11 +315,127 @@ refuses to replace a file whose header it cannot read, and leaves it as it was. in that store, and leaves the store clean. - A conditional commit lands the batch onto the file as it is under the lock. -C#, Rust and TypeScript writers exclude each other on the shared lock file; -Python takes no cross-process lock yet (`variants:follow_up:py-shared-write-lock`), -and Rust's open fails rather than waits while a C# writer holds the lock -(`variants:follow_up:rust-lock-open-waits`). Another runtime need not obey the -`storedAt` rule. +C#, Rust, TypeScript and Python single-file writers exclude each other on the +shared lock file. Another runtime need not obey the `storedAt` rule. + +## Operating a single-file store + +What the lock guarantees, per runtime. Every claim cites the code. + +- **Scope.** The lock is the sidecar `.lock`, taken by every write of a + single-file store, never by a read except in Rust. It is advisory: it excludes + only openers that also take it, in this process and others. C# opens it + share-none (`CultCache.cs:5166-5183`; .NET emulates that with `flock(LOCK_EX)` + on Unix, switched off by `DOTNET_SYSTEM_IO_DISABLEFILELOCKING`, see + `docs/research/cross-process-file-locking-prior-art.md`); Rust takes + `fs2` exclusive (`cultcache-rs/src/lib.rs:1721-1726`); TypeScript `flock` or + `LockFileEx` (`cultcache-ts/src/store-file.ts:340-382`); Python the same + (`cultcache-py/src/cultcache_py/stores.py:342-389`). A tool that opens and + writes the `.cc` file without the lock is not excluded. `flock` is not + reliable on every network filesystem; keep stores on a local one. +- **What a second writer sees.** It waits. C# polls every 10 ms for 30 seconds, + then the `IOException` propagates (`CultCache.cs:5166-5183`); with `wait: false` + (`TryCommit`) it returns `Contended` (`CultCache.cs:2052`, `2897`, `4691-4693`). + TypeScript times out after 30 s (`store-file.ts:173-195`), Python raises + `TimeoutError` after `lock_timeout`, 30 s by default (`stores.py:342-354`). + Rust waits without a timeout on its write paths (`lib.rs:1721-1726`); only its + `try_with_exclusive_lock` returns without waiting (`lib.rs:1832-1843`). Each + write re-reads the file after taking the lock and applies its staged keys onto + it (`CultCache.cs:4659-4676`), so waiting writers merge rather than overwrite. + The in-memory cache of a writer is not refreshed by another's write until its + next `PullAll`, which does nothing while that cache is dirty + (`CultCache.cs:4582-4588`). +- **What a reader sees.** C#, TypeScript and Python read without the lock + (`CultCache.cs:5139`, `5227`; `single-file-messagepack-backing-store.ts:200`; + `stores.py:144-152`). The store file is only ever replaced by rename of a fully + written, fsynced temp file (`CultCache.cs:5191-5213`, `lib.rs:1693-1712`, + `store-file.ts:70-95`, `stores.py:316-326`), so a read gets one complete + generation, old or new, never a mix. Rust `pull_all` takes the lock shared and + so waits behind a writer (`lib.rs:1926-1930`); it creates the `.lock` file if + absent, so on a read-only directory use `pull_all_read_only_snapshot` + (`lib.rs:1667-1669`), which takes no lock and relies on the producer's atomic + replace. +- **Not covered.** A unique-index check or condition spanning two stores, or any + invariant between two files: each store has its own lock and no store carries a + generation counter (the file is `[formatVersion, schemaCatalog, records]`); + a record's identity is its `version`, the SHA-256 of its stored bytes. No + library writer holds two store locks at once (`CultCache.cs:2917-2926` judges + one store, releases, then commits the home). Python's `JsonLinesBackingStore` + takes only an in-process lock (`stores.py:73-80`) and is not a single-file store + in this sense. +- **The `.lock` file is not a stale-lock marker.** It is never written or deleted + by a library writer (`store-file.ts:167-172`, `stores.py:342-345`), and the + lock is released when the holding process exits. A `.lock` file left on disk is + normal. Do not delete it while any process may open the store: a new opener + would create a second file and take a second, independent lock on it. + +### A leftover temp file + +A write stages its bytes in a temp file beside the store and renames it over the +store. The name is consumed by the rename, so a temp file that still exists is a +write that was never applied. The store file itself is never partial because of +it. + +| Runtime | Temp name | Source | +|---|---|---| +| C# | `..<32 hex>.tmp` | `CultCache.cs:5191` | +| Rust | `..tmp` | `lib.rs:4268-4275` | +| TypeScript | `.tmp---` | `store-file.ts:70` | +| Python | `.tmp--` | `stores.py:316` | + +- **Which temp files are safe to delete.** One that is not being written now: + no process holds `.lock`. Test with `flock -n .lock true` (Linux, + util-linux): success means no writer is mid-write, so every temp file of that + store is abandoned. Deleting one while a writer holds the lock destroys that + write. +- **How to know the write was incomplete.** The temp file existing is the proof. + Its content may be partial (killed during the write) or whole (killed between + the fsync and the rename); either way it was not applied, and the writer that + made it never saw its flush succeed. The store holds the last completed write. + Do not rename a temp file over the store by hand. Rust additionally reports a + rename that landed but whose directory fsync failed as `ReplacedNotDurable` + (`lib.rs:1685-1688`; TypeScript `StoreReplacedNotDurableError`, + `store-file.ts:88-92`): after a power loss the store may still be the earlier + generation. +- **What the store does about it.** Opening and reading ignore temp files and + never delete them. A writer removes only its own temp file when its write + throws (`CultCache.cs:5216-5222`, `store-file.ts:82-84`, `stores.py:327-331`, + `lib.rs:1707-1710`); a killed process skips that. Rust alone sweeps: its next + write, under the exclusive lock, deletes every regular file named + `..tmp` (`lib.rs:1698`, `4318-4352`). C#, TypeScript and + Python never sweep, and their unique names mean leftovers accumulate until an + operator deletes them. A store file that cannot be read (truncated, wrong + header) is refused at open and left byte-identical + (`CultCache.cs:4659-4663`); that is the store, not a temp file, and is + restored from a copy, never deleted and rewritten. + +### Backing up a pair of stores + +Each store is whole on its own: copying one `.cc` file is safe at any moment, +because the file is only ever replaced whole, and a copy of the path is one +complete generation. Two stores have two independent locks and no shared +generation, so copying them one after the other can capture store A at one +moment and store B at a later one. If an application invariant spans the pair, +that copy may violate it. Back up the pair at one point in time: + +1. **Preferred, quiesce.** Stop (or have flush and close) every process that + writes either store. Flush staged writes first; `PullAll` and the dispose + flush do not capture unflushed state of a still-running cache. Copy both `.cc` + files. Do not copy `.lock` files or temp files. Start the writers again. +2. **Without stopping (Unix).** Take both locks in one fixed order (sorted by + full path), copy while holding both, release: + `flock A.cc.lock flock B.cc.lock cp A.cc B.cc /backup/dir/`. + Cooperating writers wait (30 s in C#, TypeScript and Python; Rust waits + indefinitely), so keep the hold short and copy locally. This captures what + was durable at that instant; unflushed state of live caches is not in it. On + Windows, no shipped tool holds the C# share-none lock for you: quiesce. +3. **Verify.** Open each copy read-only with the runtime that will use it (for + Rust, `pull_all_read_only_snapshot`); an unreadable copy is refused, not + repaired (`CultCache.cs:4659-4663`). +4. **Restore both files together** from the same backup, with every writer + stopped, then restart so each cache loads fresh. Restoring one half of a pair + from a different backup recreates the skew. Leave the `.lock` files as they + are. ## Serialization options diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index 6defed61..6c76f5cd 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -307,8 +307,19 @@ Useful surfaces: This is built for callers like Aquarium that want durable runtime state while still choosing *when* to pay the write cost. -Single-file MessagePack remains a single-writer format. The atomic replace path -protects against partial writes, not competing writers trampling each other. +A single-file store admits any number of writers, in one process or many, one at +a time. Every write (flush, `PushAll`, `CommitBatch`, cacheless `WriteRecord`) +takes an exclusive advisory lock on the sidecar `.lock`, reads the file as +it is then, applies its own staged keys onto it and replaces the file by a +write-to-temp-then-rename. Two writers never interleave bytes, and a record +another writer put there survives; a key both staged holds the last write +(`Contracts/cultcache-store-composition.md`, "Locking"). What the lock does not +give is cross-key or cross-store atomicity, a lock that binds a program that +ignores it, or protection on a filesystem without working `flock`. A second +writer waits (30 seconds, then `IOException`; `TryCommit` reports `Contended` +instead of waiting); readers take no lock and always see one complete file. The +operator procedures for a leftover temp file and for backing up two stores are +in the same contract document, "Operating a single-file store". ## Schema migration diagnostics -- 2.47.3 From c72bb435d622646767159b680f561f88f7ee757e Mon Sep 17 00:00:00 2001 From: Libby Date: Thu, 12 Nov 2026 11:05:21 +0000 Subject: [PATCH 2/4] Docs: pair backup is per-store consistent only; atomic restore; hold the lock to delete temps; cite the read path Co-Authored-By: Claude Sonnet 5.5 --- .../Contracts/cultcache-store-composition.md | 45 ++++++++++++------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/src/GameCult.Caching/Contracts/cultcache-store-composition.md b/src/GameCult.Caching/Contracts/cultcache-store-composition.md index 88f86ab8..3e7bd425 100644 --- a/src/GameCult.Caching/Contracts/cultcache-store-composition.md +++ b/src/GameCult.Caching/Contracts/cultcache-store-composition.md @@ -383,11 +383,13 @@ it. | TypeScript | `.tmp---` | `store-file.ts:70` | | Python | `.tmp--` | `stores.py:316` | -- **Which temp files are safe to delete.** One that is not being written now: - no process holds `.lock`. Test with `flock -n .lock true` (Linux, - util-linux): success means no writer is mid-write, so every temp file of that - store is abandoned. Deleting one while a writer holds the lock destroys that - write. +- **Which temp files are safe to delete.** One that no writer is writing: + delete only while you hold `.lock` yourself + (`flock .lock rm `, Linux, util-linux) or with every writer of + the store stopped. Merely testing that the lock is free is not enough: a writer + can take it and stage a new temp file the moment after the test. While you hold + the lock, every temp file of that store is abandoned. Deleting one a writer is + still writing destroys that write. - **How to know the write was incomplete.** The temp file existing is the proof. Its content may be partial (killed during the write) or whole (killed between the fsync and the rename); either way it was not applied, and the writer that @@ -405,8 +407,10 @@ it. `..tmp` (`lib.rs:1698`, `4318-4352`). C#, TypeScript and Python never sweep, and their unique names mean leftovers accumulate until an operator deletes them. A store file that cannot be read (truncated, wrong - header) is refused at open and left byte-identical - (`CultCache.cs:4659-4663`); that is the store, not a temp file, and is + header) is refused at open and left byte-identical (the open reads through + `ReadSnapshot`, `CultCache.cs:4590`, `5104-5118`, whose `ReadStore` must throw + `CultStoreUnreadableException` for anything that is not exactly one store, + `4510-4516`); that is the store, not a temp file, and is restored from a copy, never deleted and rewritten. ### Backing up a pair of stores @@ -422,20 +426,27 @@ that copy may violate it. Back up the pair at one point in time: writes either store. Flush staged writes first; `PullAll` and the dispose flush do not capture unflushed state of a still-running cache. Copy both `.cc` files. Do not copy `.lock` files or temp files. Start the writers again. -2. **Without stopping (Unix).** Take both locks in one fixed order (sorted by - full path), copy while holding both, release: +2. **Without stopping (Unix): per-store consistency only.** Take both locks in + one fixed order (sorted by full path), copy while holding both, release: `flock A.cc.lock flock B.cc.lock cp A.cc B.cc /backup/dir/`. - Cooperating writers wait (30 s in C#, TypeScript and Python; Rust waits - indefinitely), so keep the hold short and copy locally. This captures what - was durable at that instant; unflushed state of live caches is not in it. On - Windows, no shipped tool holds the C# share-none lock for you: quiesce. + This does not make the pair consistent. A writer holds one store lock at a + time, so it can land A and then B with a gap between; holding both locks + cannot pause a writer that has landed A and not yet B, so the copy can hold A + after and B before that pair of writes. It guarantees each copy is one + complete generation of its store and nothing more. Any invariant spanning the + pair requires step 1. Cooperating writers wait (30 s in C#, TypeScript and + Python; Rust waits indefinitely), so keep the hold short. Unflushed state of + live caches is not in it. On Windows, no shipped tool holds the C# share-none + lock for you: quiesce. 3. **Verify.** Open each copy read-only with the runtime that will use it (for Rust, `pull_all_read_only_snapshot`); an unreadable copy is refused, not - repaired (`CultCache.cs:4659-4663`). + repaired (`CultCache.cs:4510-4516`, `5104-5118`). 4. **Restore both files together** from the same backup, with every writer - stopped, then restart so each cache loads fresh. Restoring one half of a pair - from a different backup recreates the skew. Leave the `.lock` files as they - are. + stopped, then restart so each cache loads fresh. Copy each file to a temp + name in the store's own directory and `mv` it over the store: a plain `cp` + over the store is not atomic, and C#, TypeScript and Python readers take no + lock, so they can read a torn file. Restoring one half of a pair from a + different backup recreates the skew. Leave the `.lock` files as they are. ## Serialization options -- 2.47.3 From 7c1ccbc18b1cd80d0becdf7bd3631028646c53b5 Mon Sep 17 00:00:00 2001 From: Libby Date: Thu, 12 Nov 2026 14:20:11 +0000 Subject: [PATCH 3/4] Retrigger CI (run 786 was cancelled) Co-Authored-By: Claude Sonnet 5.5 -- 2.47.3 From 16dc9d8aaa7cdec88a64c0a2b21d40b95df17d9c Mon Sep 17 00:00:00 2001 From: Libby Date: Thu, 12 Nov 2026 17:47:07 +0000 Subject: [PATCH 4/4] Retrigger CI (runs 785-789 cancelled at 30:00; one attempt on idle runner) Co-Authored-By: Claude Sonnet 5.5 -- 2.47.3