diff --git a/research/store-writes-behind-cache/prior-art-2026-10-27.md b/research/store-writes-behind-cache/prior-art-2026-10-27.md new file mode 100644 index 00000000..b0ea2598 --- /dev/null +++ b/research/store-writes-behind-cache/prior-art-2026-10-27.md @@ -0,0 +1,72 @@ +# Prior art: direct writes to a store that a cache sits in front of + +Serves fork `variants:follow_up:public-store-writes-behind-cache` (campaign `variants`, session `self-2026-10-27-eureka`). +Facts with evidence pointers; no recommendation. Status tags: VERIFIED (read in the fetched source this session), SECONDARY (search-result summary or third-party), UNVERIFIED (not read). + +## A. Code today (worktree `C:\cl-wt\c2a-land`, branch hands/variants-c2a, HEAD 193cf1c3). File: `src\GameCult.Caching\CultCache.cs` unless noted + +1. Public abstract surface of `CacheBackingStore`: `Push` :3717, `Delete` :3718, `PushAll` :3719, `CommitBatch` :3732 (all `public abstract`). `PullAll` :3716. `IsDirty` public getter, protected setter :3576. +2. Cache-routed writes reach the store through the same methods: upsert/variant `home?.Push(docs[0])` :2387, :2401, :2666; `Remove` `home?.Delete(existing)` :2425; batch `home.CommitBatch(request, wait)` :2726; flush `store.PushAll()` :2347 (`FlushAllBackingStores`). All admissions go through `Admit(...)` (~:2750), which validates, then calls the `land` lambda. +3. A direct store call is not refused when attached. `Gate` :3656 / `Held` :3687 only chooses which lock to take: attached, the call runs under the cache's gate (comment :3647-3655); detached, under the store's gate. Only the cacheless writes refuse: `WriteRecord` :4354 and `WriteDocument` :4378 throw `AttachedRefusal` (const :3600, text "The store is attached to a cache, write through the cache.") via `Gate(..., cacheless:true)`. `Push/Delete/PushAll/CommitBatch` pass `cacheless:false`. +4. What a direct `Push` does (:4224-4232): under the gate, `Entries[key] = entry` (the store's own dictionary, :3571, separate from the cache's `_entries`), `_staged.Add(key)`, `IsDirty = true`. `Delete` :4235 mirrors it. Nothing is handed to the cache: the cache's `_entries` and change feed are untouched. (c2a-land-map.md:284-290: Soul s4 probe `other-thread-push`, file `a,b`, cache `a`.) +5. Effects on later operations: + - `PushAll` :4251 / `CommitBatch` :4276 write the store's `_staged` keys read from the store's `Entries` (`StagedWrites`, ~:4300), so a directly staged record is written to disk with the cache's next flush or batch; then store `Wrote` :4661 updates store entries and calls `Cache?.Wrote` (:2081), which only updates versions of keys the cache already holds (skips others). + - `PullAllCore` :4170-4175 returns early `if (IsDirty)` (comment: pulling with staged mutations would let an older snapshot erase them). While a direct stage is pending, a cache pull of that store is a no-op. + - Conditional batch: :2724 throws if `home.IsDirty` ("has staged writes; flush before a conditional commit"). + - Attach: :2273 refuses a dirty store ("attach it clean"). +6. A direct `PushAll`/`CommitBatch` writes the file under the file lock with the store's staged set plus the batch (:4276-4296); `CommitBatch` sets store `Entries` from `request.Upserts/Deletes` (:4290-4293) without telling the cache. +7. Lock order comment: a cache's gate, then a store's (:3706). `InStoreHold` :2830 refuses a second store's gate inside a hold. + +## B. Overlap with kept research (cite, do not repeat) + +- `F:\Projects\CultLib\docs\research\2026-10-23-store-lock-user-code-prior-art.md` (commit 7b1cd00c): user code under a lock (JCIP ch.10, Sutter, lockdep, ConcurrentHashMap re-check, LMDB one txn per thread, SQLite btmutex). Bears on why the store gate refuses re-entry; says nothing about cache/store write routing. +- `F:\Projects\CultLib\docs\research\2026-10-23-schema-drift-write-back-prior-art.md` (commit 217a9480): old-schema write-back (Avro, protobuf, Schema Registry, Mongo/Realm/Core Data/EF Core stored versions). Not about identity-map coherence. +- `F:\Projects\CultLib\docs\research\windows-store-replace-prior-art.md`, `cross-process-file-locking-prior-art.md`: file-level lock/replace; not read in full. A grep for "identity map|write-through|ExecuteUpdate|stale cache" over docs\research and research\ found no hits. +- `F:\Projects\CultLib\research\cultnet-distributed-database\summary.md` (+ dynamo, firebase offline, rethinkdb changefeeds) and `research\witness-authoritative-networking\prior-art-2026-10-26.md`: multi-writer/replication; only listed, not read this session. + +## C. Established systems + +### C1. ORMs / identity maps + +- **Hibernate / JPA bulk HQL and native SQL.** Bulk UPDATE/DELETE runs in the database and does not touch loaded entities; per the JPA spec text relayed by in.relation.to, bulk operations "may result in inconsistencies between the database and the entities in the active persistence context"; advice: run them first or in a separate transaction. Hibernate invalidates the affected second-level cache regions after a bulk op (in.relation.to). Manual escape hatches exist (refresh, evict/clear: user guide TOC sections 6.11, 6.14, 14.7.1; section bodies not read, page truncated). Known issues: HHH-15247 (2nd-level collection cache not invalidated after HQL delete); Red Hat KB 5589111 (native update invalidated unrelated query-cache results); empty query space means whole-cache invalidation (blog, SECONDARY). Policy: hazard documented, manual refresh/clear provided, L2 cache invalidated automatically. + Sources: https://in.relation.to/2005/07/20/bulk-operations/ (VERIFIED, short), https://hibernate.atlassian.net/browse/HHH-15247 (SECONDARY), https://access.redhat.com/solutions/5589111 (SECONDARY). +- **EF Core `ExecuteUpdate`/`ExecuteDelete`.** Documented as "completely unaware of EF's change tracker, and have no interaction with it whatsoever"; example: a tracked entity with a stale value is later saved by `SaveChanges` and "the change performed by ExecuteUpdate is overwritten". Guidance: "usually a good idea to avoid mixing" them; no implicit transaction; no automatic concurrency check, rows-affected returned for manual checks. Policy: separate API surface, hazard documented, no coherence. VERIFIED: https://learn.microsoft.com/en-us/ef/core/saving/execute-insert-update-delete +- **SQLAlchemy 2.0 ORM bulk UPDATE/DELETE with WHERE.** `synchronize_session`: default `'auto'` (`'fetch'` where RETURNING works, else `'evaluate'`); `'fetch'` finds affected rows and refreshes or expunges matching in-session objects; `'evaluate'` evaluates the WHERE in Python (can fail on complex criteria); `False` leaves session state unchanged. Legacy session bulk methods lack session synchronization. Policy: the write path itself takes a sync mode, default synchronizes. VERIFIED: https://docs.sqlalchemy.org/en/20/orm/queryguide/dml.html (page did not state how newer bulk insert/update-by-PK treat the identity map). + +### C2. Embedded stores + +- **SQLite.** Cache is per connection; coherence by a file change counter at header offset 24 that other processes monitor ("a process that sees the counter change should flush its page cache"); in WAL mode detection is via the wal-index instead. Automatic, not caller-managed. Shared-cache mode (one cache across a process's connections) is called "an obsolete feature" on the project's page; WAL recommended. VERIFIED: https://sqlite.org/fileformat2.html, https://sqlite.org/sharedcache.html +- **redb.** Default `ConcurrencyMode::ExclusiveWriter`: a writer "excludes every other process" by whole-file lock; a second open for write fails with `DatabaseError::DatabaseAlreadyOpen`; `ReadOnlyDatabase` may not coexist with a `Database` in that mode; one write transaction at a time (`begin_write` blocks); page cache internal (`set_cache_size`, `cache_stats`). Policy: forbid by refusing the second handle. VERIFIED (first 100k of 190k chars of src/db.rs): https://raw.githubusercontent.com/cberner/redb/master/src/db.rs +- **LMDB.** No user-space cache (mmap). Header warns against opening one environment twice in a process ("close()ing it breaks fcntl() advisory locking"); do not mix MDB_WRITEMAP and non-WRITEMAP processes; with MDB_NOLOCK "the caller must manage all concurrency itself"; another process growing the map surfaces as MDB_MAP_RESIZED at txn begin. Policy: document the hazard, forbid some mixes. VERIFIED: https://raw.githubusercontent.com/LMDB/lmdb/mdb.master/libraries/liblmdb/lmdb.h +- **RocksDB.** A secondary instance sees primary writes only by manual `TryCatchUpWithPrimary()` (applies MANIFEST changes, rebuilds memtables from WAL); "lags until the next manual catch-up call"; writes with WAL disabled are invisible; read-only instances never catch up; page does not address caches or the LOCK file. Policy: second reader is explicit, read-only, stale by design. VERIFIED: https://github.com/facebook/rocksdb/wiki/Read-only-and-Secondary-instances + +### C3. Write-through / write-behind caches (the cache is the front door) + +- **Caffeine.** `CacheWriter` deprecated in 2.9, removed in 3.0 in favor of `Map` compute methods; eviction handled by `evictionListener`. Wiki lists limits of the writer: loads/refreshes/computes not reported to it; writer failure leaves the mapping unchanged; write-back can leave invalid cache state on failure. No stated reason for removal. Policy: store writes happen inside the cache's per-key atomic compute. VERIFIED: https://github.com/ben-manes/caffeine/wiki/Writer; deprecation notice SECONDARY: javadoc.io Caffeine 2.9.2 deprecated list. +- **Ehcache 3 `CacheLoaderWriter`.** Write-through is synchronous (put calls `write`); write-behind queues, batches, coalesces (only the last mutation per key reaches the writer); a read-through-only cache "risks making it inconsistent with the underlying system of record" if mutated, so no-op or throw in `write*/delete*`; page does not address other writers to the SoR. VERIFIED: https://www.ehcache.org/documentation/3.1/writers.html +- **Apache Ignite.** Write-through persists updates to the external store; write-behind: "some updates could be lost due to node failures", only latest value per key written; page does not address external changes by others. VERIFIED: https://ignite.apache.org/docs/latest/persistence/external-storage +- **Hazelcast MapStore.** Write-through is `write-delay-seconds` 0; MapLoader fires on a miss only, so loaded entries persist until changed or evicted (search-result summary of 5.x docs; no doc page found addressing database-side edits; the stale consequence is inferred). SECONDARY/UNVERIFIED: https://docs.hazelcast.com/hazelcast/5.5/mapstore/implement-a-mapstore +- **Memcache at Facebook (Nishtala et al., NSDI 2013).** Writes go to the database, then the cache entry is deleted (invalidation, via mcsqueal); leases stop a stale `set` after a newer write; MIT 6.824 FAQ: a window of stale reads exists by design. SECONDARY (paper not read): https://pdos.csail.mit.edu/6.824/papers/memcache-fb.pdf, https://pdos.csail.mit.edu/6.824/papers/memcache-faq.txt + +### C4. Paper/design note on single-writer embedded-store cache coherence + +- None found this session (searches covered Memcache only; none targeted embedded stores directly). Not exhaustive. + +## D. Pattern table (facts only) + +| System | Direct write path next to the cache | What the API does | +|---|---|---| +| Hibernate | bulk HQL/native SQL | exists; L2 invalidated; L1 not; hazard documented; refresh/clear provided | +| EF Core | ExecuteUpdate/Delete | exists; tracker unaware; hazard documented | +| SQLAlchemy | bulk UPDATE/DELETE | exists; `synchronize_session` default `auto` refreshes/expunges | +| SQLite | second connection | allowed; change counter invalidates the other cache | +| redb | second handle | refused (`DatabaseAlreadyOpen`) | +| LMDB | second env in process | documented as prohibited | +| RocksDB | secondary instance | allowed read-only, manual catch-up | +| Caffeine / Ehcache / Ignite | writer inside the cache | store written only via cache operations; direct SoR writes not addressed in pages read | +| Hazelcast | direct DB edits | not addressed in docs found | +| CultCache today | public Push/Delete/PushAll/CommitBatch | allowed attached; cache not told; refused only for WriteRecord/WriteDocument | + +## E. Not done + +Hibernate user-guide sections 13.3 / 14.7 bodies not read; Hazelcast official page empty on fetch; Nishtala paper and the Dynamo/Firebase/RethinkDB files not read; no GitHub-issues search for redb/RocksDB/SQLite incident reports; Soul s4 probe not run.