An exception thrown inside a `when` filter is swallowed as false, so a manifest
re-read failing mid-File.Replace turned a race that should retry into a thrown
torn-page error. The re-read now runs in the handler, and a failed re-read
counts as moved. Parallel.For's AggregateException is unwrapped where the pages
are read, so the page's own failure propagates and the torn check no longer
walks wrappers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A missing or hash-failing page under an unchanged manifest is corruption,
not a writer mid-commit, but every InvalidDataException was retried five
times and reported as "did not settle" with the real cause buried. The
retry now also requires the manifest bytes to have changed; otherwise the
corruption error is thrown on the first attempt. A vanished page directory
is a race under the same condition.
The reload test's writer now commits under every unleased read, so it only
settles if the retry takes the lease that commit created.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A pull takes the commit lease only when .commit.lock exists. Without it
(no writer has created one yet, or the store was copied without it) the
reader loaded with no lease, and a writer committing between its manifest
read and its page reads left it with a vanished page or a stale manifest.
An unleased pull now keeps its load only if the manifest bytes it read are
still current afterwards; a changed manifest, or a named page that is
missing or fails its content hash, reloads it, at most five times, and a
pull that has not settled by then throws. A retry takes the lease once a
writer has created it. Opening still creates nothing. The lease comment no
longer claims a missing lock means no writer ever committed, and the store
composition contract states the rule.
The test deletes the lock from a seeded store and commits from a second
cache through an internal seam between the manifest read and the page
reads, once replacing the page the reader named and once only adding a key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Opening a directory store, read-only or writable, created the .records
folder and its .commit.lock before reading a manifest that might not
exist. Only a write creates them now: PullAll takes the commit lease with
create: false, which opens an existing lock and returns no lease when
there is none (no writer has ever committed there, so there is nothing to
exclude). CommitBatch and PushAll create the folder and lock as before.
Test DirectoryMessagePackBackingStore_Open_CreatesNothing_UntilAWrite:
opening a missing store and a manifest-less .records folder, read-only
and writable, leaves the filesystem unchanged; the first flush creates the
manifest, the lock and one page. With the previous lease code restored
the test fails on the first read-only open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cache-wide unpublished list let any thread drain and deliver another
thread's changes: a throwing observer dropped the rest of the queue and threw
into an unrelated writer, writes returned before their change was published,
and a third-party load could wait indefinitely.
Every admission now runs in a hold (Held). The outermost hold on a thread
collects exactly the changes it admits and publishes them after it leaves the
gate, before it returns, on its caller's thread; nested holds append to it.
Every change is delivered even when an observer throws; the first exception
(AggregateException for several) is then rethrown, and a failing body's own
exception wins over observer exceptions.
Stores take the hold instead of locking a shared Gate object, so a direct
PullAll on an attached store publishes its own load and the empty follow-up
Loaded hand-over is gone. Gate is replaced by protected Held.
Tests: OwnChangeIsPublishedBeforeWriteReturns and
ThrowingObserverDoesNotLoseOtherChanges fail on 4eacd87 and pass here. The
repull tests now record a write that reaches the gate while the pull is paused
instead of inferring readiness from thread state; both fail when the pull does
not hold the gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Observers and OnUpdate never run under the cache gate. Admit queues changes
while the gate is held; the outermost entry point (attach, pull, commit,
upsert, remove) publishes them after releasing it. A store's direct PullAll
hands over an empty load outside its lock so the cache publishes then.
A throwing observer can no longer split store from cache: publication starts
only after the store has adopted its loaded view. The directory store adopts
its durable catalog only after the cache admits the load; the write-only
_durableIndex is deleted.
Single-file Push, Delete, PushAll and CommitBatch take the gate like the
directory store; Dispose flushes under it. Write mints storedAt and admits
under one gate hold, and Land re-stamps staged upserts under the gate that
admits them, so a concurrent pull cannot land a later storedAt in between.
Tests: ObserverDoesNotRunUnderTheGate,
ThrowingOnUpdateHandlerDoesNotResurrectDeletedRecord,
RepullNeverErasesAStagedWriteOnDirectoryStore,
DirectCallsOnAnAttachedStoreShareTheCacheGate; the directory last-writer
result is pinned to {a,c,x}; the single-file re-pull test waits for the write
to reach the store or block instead of sleeping. Pausing tests always release
the pause so a failure cannot deadlock Dispose.
Docs: observer rule, direct store calls share the gate, directory TryCommit
contention and gate wait, per-key last-writer-wins, directory flush merge.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An attached store takes its cache's gate as its own lock (CacheBackingStore.Gate),
so a pull's load callback into the cache can no longer take the gate after the
store lock. The directory store's private mutation gate is deleted; the
single-file pull now runs under the gate; PullAllBackingStoresAsync takes the
gate around each store's pull. Fixes the pull/upsert deadlock on directory
stores and the re-pull that erased a staged single-file write.
An unconditional single-file Commit writes the store's whole view plus the batch,
exactly as a flush would, instead of merging onto the file; it persists earlier
staged writes and leaves the store clean. Conditional commits still land onto
the file under the lock. Directory-store commit and flush already share one
generation writer (staged keys onto the current manifest) and are unchanged.
Contracts state both rules, the lock order, that OnUpdate fires for loads only,
that a missing single file keeps staged writes, and that a store failing to
hydrate on open throws and is left byte-identical.
Tests: PullAndUpsertOnDirectoryStoreDoNotDeadlock, RepullNeverErasesAStagedWrite,
UnconditionalCommitIsLastWriterWinsLikeFlush, UnconditionalCommitPersistsStagedWrites,
OpeningACorruptStoreThrowsAndLeavesItUntouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cut 3 of the CultCache migration (Aetheria docs/cultcache-migration-cut.md, 2.1
and 2.2). One commit, because routing, the batch that replaces the ambient
transaction, and conditional commit share one admission path.
Deleted first:
- globals: InitializeGlobals, MaterializeMissingGlobals, the two-argument
constructor, the initializeGlobals plumbing in CultCacheMessagePack,
ContainsDurableRecord (cache and directory store)
- replication: the push-to-every-store loops and the attach-time push of
existing records
- typeof(T) as the schema owner in AddAsync/UpsertAsync and in
CultNetDatabase.PutAsync/PutPredictedAsync
- the ambient transaction: ExecuteTransactionAsync x2, CultCacheTransaction,
CommitTransaction, VisibleStoredDocuments, the AsyncLocal, the SemaphoreSlim
gate, and every ambient check
- CultCacheOpenOptions.PullOnOpen; OpenAsync is Task.FromResult(Create(...))
- DirectoryMessagePackBackingStore.ResolveLegacyUncataloguedRecordCatalog: the
catalog precommit (fe55bff) is an ancestor of the v4 page format (eea8248),
so no v4 manifest can reach it
- CultCache.DeleteAsync<T>; its one caller, CultNetDatabaseSubscriptionClient,
called it by reflection and now calls Remove(key)
Added:
- AddBackingStore(store, params Type[] homes): attach validates (clean, not
already attached, one untyped store, no claimed home type, no admitted
record changes home) and then pulls under the gate
- Home(type): most derived routed home, else the untyped store
- Admit: every add and remove, from a write, a batch or a load, validates
(home, read-only, loaded-from-home, global singleton), lands durably, then
applies and publishes; a pull hands its whole delta over in one call, so a
refused record leaves the store and the cache unchanged
- read-only stores (CacheBackingStore(readOnly), both store constructors,
CultCacheOpenOptions.ReadOnly); flush skips them and writes only dirty stores
- assignable GetGlobal<T>/GetByName<T>/GetByIndex<T>/Watch<T>; more than one
candidate throws naming them
- Commit(Action<CultCacheBatch>) returning bool, TryCommit returning
CultCommitOutcome, CultCacheBatch (Upsert, Remove, Expect, ExpectUnchanged,
sealed after commit), CultCommitRequest; CacheBackingStore.CommitBatch
evaluates conditions under the store's exclusive lock
- single-file store: <path>.lock sidecar for CommitBatch and PushAll; a clean
store commits onto the file as it is, so records it never loaded survive
- storedAt minted in one place and bumped a tick when not later than the record
it replaces
Consumer adjustments in this repo: Mesh streaming tests drop PullOnOpen; the
Studio bridge stops reflecting on it; BackingStoreTests' global test creates
its global explicitly and its transaction tests are rewritten against Commit
(the two that proved ambient hydration rules are deleted); the pre-Cut 2
refusal test expects the attach to throw.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parked/cultcache-soa tag message says the managed document was deleted
separately. parked-features.md now supersedes that; the tag is unchanged.
The directory store gains its design statement and a comment that only
manifest-named pages are loaded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every /// line in GameCult.Caching and GameCult.Caching.MessagePack goes; they
restated member names. Three blocks that state a rule stay as // comments:
MaterializeMissingGlobals runs after hydration so persisted globals win,
ExecuteTransactionAsync commits durably before the cache and observers
advance, and CommitBatch implementations restore their staged view on
failure. Both projects add NoWarn CS1591, since GenerateDocumentationFile is
on in src/Directory.Build.props.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DirectoryMessagePackBackingStore loses HydrationFilter, FlushStageProbe,
ReadStageProbe and their stage constants, ReadPersistedGeneration,
PullSelected, and every legacy path: the v2 directory-indexed and v3
immutable-pages formats, v1 inline records and split pages, the index
upgrade on flush, and the legacy page paths. CacheBackingStore.PullSelected
and CultPersistedRecordMetadata go with them.
A missing manifest is an empty store. An existing manifest whose
FormatVersion is not cultcache.store.v4.directory-content-addressed-pages
throws "Directory store {path} is {format}; only {v4} is readable." A clean
flush writes nothing, since there is no legacy index left to upgrade.
Tests deleted with the surface they injected through: the v1 missing-catalog
load, pages-before-manifest, fault-before-manifest (4 cases), same-timestamp
page, mutation-during-flush and reader-lease tests. The orphan-page test now
checks through a fresh store pull instead of ReadPersistedGeneration.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Removes FlushBackingStore, both PrepareForReloadOrShutdown overloads,
PullBackingStoreRecordsAsync, TryGet<T>, TryGetByName, TryGetByIndex,
Resolve<T>, the cache and store Logger, and LastSuccessfulFlushAtUtc on the
cache and stores. GetStoredDocuments<T> folds into GetAll<T>.
The soft flag goes from FlushAllBackingStores, ExecuteTransactionAsync,
CommitBatch and PushAll. It stays on FlushAsync(bool) because
CultCacheStudioWindow reflects on that signature until Cut 6.
CultMesh.FlushAsync and CultNetLocal.FlushAsync keep their parameter and
stop forwarding it.
The store's EntryAdded/EntryUpdated/EntryDeleted subjects become
protected internal Loaded/Unloaded callbacks set by the cache at attach.
AddStoredDocumentInternal is synchronous, so the GetAwaiter().GetResult()
bridges on the store subscriptions and the global materializer are gone.
Tests: drop the LastSuccessfulFlushAtUtc asserts; OpenAsync and GetAll tests
read through Get/GetAll; the persisted-global test proves the durable global
loads without staging a default instead of going through selective hydration;
the lease-before-observers test observes through CultCache.Watch; the
PullSelected selective-hydration test is deleted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
