# CultCache publication: the cache owns "published once", and order stays data Status, 2026-09-29: Cuts 1-3 landed on `main`. Cut 3 merged at `ae8576d`. Imagination pass 4 re-took Cuts 4-6 against `3bf1c0c`. Q-P5 is ruled (A). Cut 4 is ready for Hands. - F4, pinning `Held`'s publish-on-throw guard, is done: Cut 3 pinned it. - F3 (observer removal on unsubscribe) and F1 (the false "getters run once" claim for conditional commits on variant stores) were Cut 3 follow-ups; confirm against the Cut 3 merge. Recorded: - F2, the Q-P2 hazard. A rethrown observer exception makes a committed write look cancelled or refused. The only live typed catch around a cache write is AetheriaEve `Aetheria.State.Daemon/Program.cs:3722`. It needs a fail-fast handler, and no production code installs one. - F5: subscribe and unsubscribe cost O(n). - F6: writes after `Dispose` are dropped silently. Map written by Imagination pass 1, 2026-09-29. The ends are the operator ruling of 2026-09-30: every committed cache change reaches subscribers exactly once. This document owns the means. - **Body.** CultLib `main` `65311c9ad770828adafe40c2c14b2d3da517628e`. Every `file:line` is against it unless it names the branch. - **Superseded attempt.** `origin/hands/cultnet-publish-commits`, tip `28248eb`, base `f9f107a`. Soul pass 3 said do not merge it, and that stands. - Main has not touched `CultCache.cs`, `CultNetDatabase.cs`, the composition doc or the networking tests since `f9f107a`: `git diff --stat` is empty. - `git merge-tree 65311c9 28248eb` is clean. - **Scope.** The C# cache's publication and CultNetDatabase's use of it. There is no wire-format change and no other runtime changes (§6). ## Rulings (operator, 2026-09-30) - **Q-P1 A: order as data.** Each change is published exactly once, on its writer's thread, before the write returns, and carries its `Sequence`. The shard log is written in cache order by the gate-held journal. Wire peers drop changes that are stale by `Sequence`. The turnstile (`bcae483`/`4562340`) and the drainer are both rejected. - **Q-P2 A:** a throwing observer does not stop the others. Every observer runs, the exception is rethrown to the writer afterwards, and the observer stays subscribed. The commit stands. - **Q-P3 A:** a public `CultCache.AddJournal(...)`, with reentry guarded. - **Q-P4 A:** Cut 6, which drops stale changes per key on the wire, lands in this campaign. - **Q-P5 A (2026-09-29):** a primary that cannot log a committed change throws a typed `CultNetShardLogException` naming the shard and the burned sequence, after publication, not derived from `InvalidOperationException`. ## The finding that shapes this map The direction Imagination was handed was: "ordered delivery belongs in the cache's own publication, on the writer's thread, so a writer delivers its own changes before returning". That design **has been built before, and it was deleted**. - `bcae483` (2026-09-13 20:09) made the cache a delivery scheduler: tickets, a delivery gate, turn waits and per-thread queues for reentrant writes. - `4562340`, 34 minutes later, deleted it: "It deadlocked inside CultLib: a load's delivery reaching a CultNetDatabaseSubscriptionServer observer that needs _lifecycleGate, while a subscribe's demand handler holding that gate writes the cache and waits for the delivery's turn." - It replaced it with **order as data**: a per-cache `Sequence`, and no delivery scheduler. - The regression tests it left, `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries` (`ConditionalCommitTests.cs:763`) and `SubscriptionDemandHandlerWritingCacheDuringDeliveryDoesNotDeadlock` (`NetworkingTests.cs:2244`), are live on main. This pass rebuilt that scheduler as a prototype. It is the per-cache turnstile with same-thread trampolining at `79f4639`, and it reproduces the deletion: - Caching tests: 278 of 279 pass, and `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries` fails. - The networking suite hangs, and Yggdrasil's idle watchdog killed it at 900 s. The blame-hang run that names the hung test is recorded below. The reason is structural, not an implementation slip. Once a writer waits for another thread's delivery, any lock that an observer takes and a writer holds becomes a deadlock: - total cross-thread order; - a writer delivering before it returns; - no writer waiting on another thread. Pick two. The branch's drainer chose order plus no-wait, and lost before-return (F3 and F4). The turnstile chose order plus before-return, and lost no-wait (the `bcae483` deadlock). **So this map keeps order as data (the `4562340` ruling) and moves the one order-sensitive consumer, the CultNet mutation log, to the place order is decided: under the cache gate.** - Every writer keeps delivering its own changes, on its own thread, before it returns, exactly as main does. F3 is satisfied by not regressing it. - The log becomes a **journal** written in the admitting hold, in Sequence order. - The reorder buffer and drainer die, and no scheduler replaces them. This departs from Soul's stated direction on purpose, and **Q-P1** puts it to the operator. ## Probe evidence All of this ran on Yggdrasil via `ygg-verify.sh`, from scratch commits in a throwaway worktree that no branch holds. - Probe file: `tests/GameCult.Caching.Tests/PublicationProbeTests.cs` in scratch commit `79f4639`. The worktree has been removed. The commit is unreferenced in the local object store, where gc can collect it; it also survives on the Yggdrasil mirror as `refs/verify/79f4639f8f5b95db7ce16f244a081c1ec101a468`. - `9c5abf5` is the probes on main. `79f4639` is the same probes plus the turnstile prototype in `CultCache.cs`. | # | Probe | main `65311c9` | Turnstile prototype `79f4639` | |---|---|---|---| | Q1 | A batch `{ok, boom}` where boom's `[CultName]` getter throws | `TargetInvocationException`. `_sequence` goes from 0 to 1, because ok was minted. **Both ok and boom are in memory**, and `GetByName("ok")` finds ok. **Nothing is published**, and the next write gets Sequence 2. F1 is worse than stated: memory is corrupt and unpublished. In a store-backed cache, `land` (`:2325`) already ran, so the store holds it too. | No stall, but memory is still corrupt. | | Q2 | Two `Watch` observers, the first throws | With the default R3 handler, the second observer sees 2 of 2 and the writer sees nothing. **With a rethrowing handler, the second observer sees 0 of 2 and every writer gets the exception.** The comment at `CultCache.cs:2374`, "OnNext does not throw", is false under fail-fast. | With its own observer array and a per-observer catch, the second observer sees 2 of 2 and the writer gets the exception afterwards. | | Q3 | Writer A's observer blocks on `a`, then B writes `b` | The recorder sees `b(2)` before `a(1)`. There is no cross-thread order, by design since `4562340`. | Sees `a` then `b`, because B waited for A. | | Q4 | An observer on `x` writes `y` | `y(2)` before `x(1)`: nested, depth-first, and the nested write returns after `y` is published. | `x` then `y`, and the nested write returns before `y` is published. | | Q5 | `OnUpdate` on Upsert and on Commit | 0 fires, so loads only. `Watch` sees 2. | Unchanged. | | Q6 | 8 threads × 1000 writes, plus reentrant writes | 67 ms, 2645 of 8800 cross-thread inversions, and 0 writes returned undelivered. | 516 ms, 0 inversions, 0 undelivered. | | Q7 | Soul's P2 shape: a handler waits 1 s for another thread's write | Observed in time. | Not observed in time. | | Suites | Caching, Networking and Mesh | (green on main) | Caching fails `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries`, and Networking **hangs** until the 900 s idle kill. Rerun with `--blame-hang-timeout 90s`: 286 passed, then the test host hung and was dumped. The blame sequence file was not extracted, so **the hung test is not named**. It is consistent with, but not proven to be, `SubscriptionDemandHandlerWritingCacheDuringDeliveryDoesNotDeadlock`, whose own comment says "disposal would hang too". | ## Imagination pass 4: Cut 4 re-taken on the landed journal Cut 3 landed at `ae8576d`. Its Soul left two questions that Cut 4 had to settle before relying on the journal: (a) what a throwing journal means for the log, and (b) whether a journal needs the change's origin. Both have one coherent answer. The one fork, Q-P5, was about who hears of a log failure; the operator ruled it A (2026-09-29). ### Probe evidence (pass 4) - Probe file: `Pub4ProbeTests.cs` (Networking tests). It was run on Yggdrasil from scratch commits: - `0d7397d`: main plus the probes; - `6ec2f73`: the same, merged with the branch `28248eb`; - `376cd74`: main plus probe C. - The commits survive on the Yggdrasil mirror. The scratch worktree is gone. - The probes report facts and assert nothing. `FlakyLogStore` is an in-memory `ICultNetShardMutationLogStore` that can refuse `Append` or run a hook inside `GetCompactedThrough`. | # | Probe | main `3bf1c0c` | branch `28248eb` (queue) | |---|---|---|---| | A1 | Primary with a durable log. Put 1, Put 2 (its `Append` throws), Put 3. A replica pulls, then retries. | Writer gets `IOException`. The cache holds 1, 2 and 3, but **2 is never published** (2 of 3). Memory log `[1,2,3]`, durable `[1,3]`. The replica pull throws "log has a gap. Expected 2, received 3", and so does **every retry**: the replica is stuck for good. | **Writer sees nothing**: R3 swallowed the drainer's exception. 2 is not published. The replica is stuck in the same way. | | A2 | The same failure at the tail, then the primary restarts over the same cache and log store. | The replica applied 1. After restart, note 3 is **minted sequence 2**. The replica takes it, reports applied 2, and holds `1,3` against the primary's `1,2,3`: **silent divergence**, a reused sequence. | Same. | | A3 | A replica whose log `Append` throws once. | The pull throws and the cursor stays at 0. The retry converges (`1,2`), but the memory log is `[1,1,2]`: a duplicate, because memory is written before disk and kept as a `List`. | The pull "succeeds": the exception was swallowed and the cursor went to 2. The durable log is `[2]`, so **entry 1 is lost**, and a chained replica sees a gap. | | B1 | Primary with a file store. Another writer puts `pulled` into the file, the primary runs `PullAll`, and a replica pulls the log. | `pulled` is published but **not logged**. The replica gets nothing. | Logged. The replica converges on `pulled`. | | B2 | A store attached after the database exists. | Nothing is logged. | All three hydrated records are logged. | | S | `CreateShardSnapshotResponse` with a write landing between its document read and its sequence read (hooked in `GetCompactedThrough`). | The snapshot holds `[1]` at log sequence 2. The replica applies it and pulls, and still lacks 2: **divergence**. | Same. | | C | Compact through 2, restart, then Put 3. | Note 3 is **minted sequence 1**, and `GetLatestMutationLogSequence` returns 1, below the floor of 2. | (not run; the code is unchanged) | What the probes establish: - A1, A2, S and C are pre-existing defects on main. Cut 4 inherits them and must not build on them: the resync path it relies on for (a) is itself broken by S and C. - On a replica, the pull loop already retries. The cursor advances only after an entry applies (`CultNetDatabase.cs:1038-1083`), so a replica needs no new mechanism. It needs only to keep the exception reaching that loop, which the branch's queue broke (A3). - Loads on a primary must be logged, or replicas diverge (B1). ### (a) A journal that cannot log a committed change: one answer The facts: - The log store is a second durable store written after the cache's store. Without an atomic commit across the two, the log can lag the store. It must never lead: a replica would then apply a change the primary refused. - So lag is unavoidable, and it has to be detectable and repaired. - The protocol already has a repair, and nothing else needs to be invented. - "History the primary cannot serve" is `compacted_history`: `CultNetDatabaseServer.cs:189-201` answers `ResyncRequired`. - `CultNetShardReplication.cs:199-214` then snapshot-resyncs. **Rule: a primary that cannot log a committed change burns that change's sequence and compacts through it.** - In the database journal, on a primary, for each change that is not `NoLog`: 1. Mint `S`. 2. Build the entry: the wire message from the door, or `ToLogEntryMessage`. 3. `Append` durably, **then** record it in memory. Disk goes first, so memory never serves what disk lacks. - If step 2 or 3 throws: - record nothing; - raise the database's in-memory floor for the shard to `S`; - try `store.CompactThrough(shard, S)`; - still set `_lastWriteSequence` to `S`, because the row was written at `S`; - carry on with the rest of the admission; - throw `CultNetShardLogException` once at the end of the journal call (Q-P5 A). - Every replica behind `S` is then told `compacted_history` and resyncs by snapshot. The sequence gap at `S` is never served, because the floor covers it. - This **replaces** the old §4 step 3 claim, "a serialization failure then mints nothing, so the log can never hold a gap". Minting nothing hides a replication gap behind a dense sequence. That is exactly A2's silent divergence. - **Replica:** there is no floor. The journal's exception reaches `ApplyShardLogResponseAsync` and the cursor does not advance. The next pull re-applies the entry: a fresh instance, so the door matches again, and the entry is re-recorded under the primary's sequence. The branch's absent-key direct log follows the same disk-first order. - **Required fixes** that the rule depends on, all in Cut 4: - `GetCompactedMutationLogSequence` (`CultNetDatabase.cs:597-602`) returns `max(store floor, in-memory floor)`. - `InitializeLogSequencesFromStore` (`:1363-1395`) resumes at `max(highest retained, store.GetCompactedThrough(shard)) + 1`. This is probe C. - `GetLatestMutationLogSequence` (`:607-616`) never reports below the floor. - `CreateShardSnapshotResponse` (`:624-670`) reads `GetLatestMutationLogSequence` **before** `SelectAll`, on both returns, `:645` and `:668`. The documents then reflect at least that sequence, and replaying later entries converges: they are whole-document puts and deletes, applied in order. This is probe S. - `RecordMutationLogEntry` (`:1331-1352`) writes disk first. The in-memory log becomes `SortedList` per shard, replacing by sequence as the durable store does. This fixes A3's duplicate. - **Residual, recorded rather than forked.** Suppose the store refuses both `Append` and `CompactThrough`, and then the process restarts. - The durable log does not know about the hole, so A2's tail reuse can recur. - Fixing that needs a durable write per mint, a sequence reservation, which doubles log I/O under the gate. - The thrown exception names both failures, so the outage is loud. - Refusing later writes ("fail closed") is not coherent. The database can refuse only its own doors, while bare cache writes are journaled regardless: that would be split authority. ### (b) A pulled or loaded change on a primary: no origin needed - On a primary the log is the shard's **state history**, whoever wrote the state. B1 shows that leaving loads out of the log leaves replicas permanently diverged. - The only origins that change what is logged are already carried by the door table, keyed by instance identity and matched by shape. They are: - `Replica` (log the primary's entry); - `NoLog` (prediction, snapshot resync); - `Put`/`Delete` (the wire message); - `Predicts`/`Authoritative`. - A load or pull has no door, so on a primary it is logged like any unattributed commit. On a replica it logs nothing, like any local write. That is what the branch does, and B1 converges under it. - Cost: - B2: a store attached **after** the database logs its hydration. - `CultNetHost` opens the cache before building the database (`CultNetLocal.cs:146-152`), so production hydration is not journaled. - A re-pull admits only the records that changed. In B1 the seed was not re-logged. - `AddJournal` keeps its signature. If a future consumer (audit, say) needs "load or commit", the fact goes on `CultCacheDocumentChange` as an additive property, as `Sequence` did. The cache already knows it: `Admit`'s `source != null` at `CultCache.cs:2420`. It never needs a new `AddJournal` shape. - This is recorded, not forked: no consumer needs it today. ### The two recorded hazards, settled for Cut 4 - **Q-P2 typed-exception hazard.** Cut 4 makes it frequent. Every cache write on a primary with a durable log now does log I/O inside its admission. - AetheriaEve opts into durable logs (`Aetheria.State/AetheriaStateNode.cs:144`). - Its `AetheriaHangarCommandJournal.ValidateAsync` writes an envelope (`Program.cs:6509`) inside the `catch (InvalidOperationException)` at `Program.cs:3722`. - A log failure surfacing as `InvalidOperationException` would therefore produce a "denied" receipt for a committed envelope. - Q-P5 A (ruled): the writer gets a typed `CultNetShardLogException` that **does not derive from `InvalidOperationException`**. - **Journal deadlock (the Cut 3 note).** The database journal waits only on `_logGate` and on the log store's own lock. - Lock order: cache gate, then `_logGate`, then the log store's lock. - Invariant: **no code holding `_logGate` calls the cache, and no log store calls the cache or waits on a thread that does.** - Readers that take `_logGate` read only log state: `GetMutationLog`, `LastWriteSequence`, `CurrentAsOf`. Readers already inside a cache read, such as the selection evaluator's `ordinalOf`, nest `_logGate` under the cache gate, which is the same order. - The store rule goes into the `ICultNetShardMutationLogStore` XML doc (`CultNetShardMutationLogStorage.cs:13-34`), because a user store runs under the cache's gate. - A log store that writes the cache already throws, through Cut 3's guard. Cut 4 pins it with a test. ## Authority map - **Owner.** `CultCache`. The hold that mints a Sequence range also: 1. journals the range under the gate, in Sequence order (new); 2. publishes the range to every observer, isolated, on the minting writer's thread, after it releases the gate and before it returns (main's model, now isolated). Nothing waits on another call's delivery. - **Inputs.** - The changes returned by `Apply` (`CultCache.cs:2707`), the single mutation site. - The pre-mint projection of names and indexes (Cut 1). - The journal list, the observer array and `OnUpdate`. - **Outputs.** - Journal calls: one per admission, under the gate, in Sequence order. - `Watch` / `WatchRecord` notifications that carry `Sequence`. - `OnUpdate` for loads. - Observer and journal exceptions, rethrown to the writer. - **Derived state.** - `Sequence` is order as data. - CultNet's primary log sequence is minted in the journal, so log order is cache order. - A replica's log sequence is copied from the primary entry. - The database change kind (`Predicted`, `Reconciled`, `Added`, `Updated`, `Removed`) is decided in the journal. - `CultNetDatabaseChange.Sequence` (new) is copied from the cache change. - Wire projections drop stale by it (Cut 6). - **Forbidden writers.** - Nothing re-sequences or schedules cache changes. The branch's `_parked`, `_ready`, `_nextSequence`, `_draining`, `PendingChange` and the drain body of `OnCacheChange` die, and so do the turnstile, `bcae483` and the prototype. - No R3 `Subject` fans out cache changes (Q2). - `Apply` and `Index` run no user code. - Doors never read or write prediction state: the branch's `HasPrediction` and `_predictionGate` die. - The database's Watch observer never logs; only the journal logs. - A journal never enters the cache. - **Shared paths.** Every path goes `Held` → `Admit` → `Resolve` → **Project** → `land` → `Apply` → **journal** → leave gate → **publish**: - loads: `AddBackingStore` `:1909`, `Loaded` `:1940`, `PullAll*` `:1959` and `:3052`; - `Commit` / `TryCommit` (`Land`, `:2272`); - `UpsertAsync`, `UpsertVariantAsync`, `FlattenAsync` and `Remove` (`:2004-2056`, `Write` `:2252`); - a variant dependent's re-resolution (`VariantPlan.Dependents`, in the same `Apply`); - direct calls on an attached store (`:2924`); - `Dispose` (`:2142`); - the database doors, replica apply and snapshot resync, which only write the cache and hand context to the journal. The single exception is the replica's absent-key delete, which has no cache change. It logs directly (§4). - **Deletion line.** - In the cache: `Subject _changes` (`:1809`), the `_changes.OnNext` call and its false comment (`:2374-2375`), `_changes.Dispose()` (`:2154`), and the accessor calls in `Index` (`:2798-2804`). - In the database, on the salvaged branch: the reorder buffer and drainer, `_predictionGate` and `HasPrediction`, the logging in `Deliver`, and the `Subject _changes` fan-out. ## The design, question by question ### 1. Validation before minting: a refused change mints nothing - **Where.** Add **Project** to `Admit` (`:2313`), between `Resolve` (`:2324`) and `land` (`:2325`). - **What.** Over `plan.Admitted ∪ plan.Dependents`, it evaluates once per record: `NameAccessor(document)` and every `IndexAccessors[alias](document)`. It keeps `(Name, (Alias, Value)[])` per key on the plan. - **A throwing getter refuses the whole admission.** It raises an `InvalidOperationException` that names the key, the member and the unwrapped message; today the caller sees a bare `TargetInvocationException`. The refusal comes before `land`, so the admission writes no store, touches no memory and mints nothing. - **Readers take the projection, not the document.** - `Index` (`:2793-2806`). - `RefuseVariantIndexSharing` (`:2659-2660`). - The variant-name rule in `ResolveOne` (`:2616`). Its ancestor read at `:2621` reads held documents, pre-land, and may stay. `Resolve` currently runs the refusal rules before any projection exists. Hands makes Project the first pass over the resolved plan and moves both rules after it, so rules read the projection and only the projection reads documents. - **Apply becomes user-code-free.** It then does only dictionary and `ConditionalWeakTable` work on precomputed data. - **Loads refuse too.** A store whose record's getter throws fails to open, loudly. That is the rule "hydration failure on open is loud" (composition doc `:88-92`). Today it half-indexes silently. ### 2. Observer isolation - **Replace the `Subject` with the cache's own observer array**, copy-on-write under a small lock. - `Watch` (`:1887`) becomes `Observable.Create(register)` followed by the existing `Where` and `Select`. - `Dispose` calls `OnCompleted` on each observer, which is what `Subject.Dispose` did. - **Publish loops change → observer, and wraps each `OnNext` in its own try/catch.** - R3's `Observer` already catches handler exceptions (`OnErrorResume`, then the unhandled handler). So an exception reaches the cache only when fail-fast was asked for: a rethrowing `ObservableSystem` handler, or a custom observer. - Escaped exceptions join the `OnUpdate` failures. They are rethrown to the writer after every observer has received every change of the hold. That is the existing `OnUpdate` policy (composition doc `:77-82`). - The observer stays subscribed (**Q-P2**). The commit stands either way. - **Prototype.** At `79f4639` (Q2) this took about 15 lines. Hands keeps that part and drops the turnstile around it. ### 3. Delivery thread, order and reentrancy - **Unchanged from main, on purpose.** Each outermost hold publishes exactly its own changes, on its thread, after it leaves the gate and before it returns (composition doc `:59-65`, `Held` `:2339`). - A write from inside a handler is its own outermost hold. It delivers depth-first before returning, as Q4 shows on main. - There is no cross-thread order. Every change carries `Sequence`, and latest-state consumers drop stale changes by it (doc `:68-73`, and the Mesh mirrors at `CultMesh.cs:1534-1540`). - **Why this is not a regression of the ruling.** The ruling asks for exactly-once, not for a total delivery order. - Exactly-once in the cache comes from the single mint site plus isolation plus Cut 1 (§1-2). - Order is needed by exactly one consumer, the shard log. It gets that order from the journal (§4), not from delivery. - **The journal is the new mechanism.** - Signature: `IDisposable CultCache.AddJournal(Action>>)`. Its name and shape are **Q-P3**. - It is called in `Admit` right after `Apply` (`:2328-2329`), under the gate, once per admission, with that admission's changes in Sequence order. Journals run in registration order. - A journal that enters the cache, meaning any hold on this cache while a journal runs, throws `InvalidOperationException`. It is guarded by a flag checked in `Held`'s `Monitor.IsEntered` branch (`:2341`). This is the store rule ("a store call must not wait for a write to its own cache"), enforced. - Journal exceptions follow the observer policy: the admission stands, it is still published, and the exception is rethrown to the writer after publication. - Adding or removing a journal takes the gate, so registration is atomic with respect to admissions. - **Lock order.** Cache gate, then journal-internal leaf locks (the database's `_logGate`). Observers still run outside the gate, so the `bcae483` cycle (an observer needing `_lifecycleGate` while a gate holder writes the cache) cannot form. No journal takes `_lifecycleGate`, and no writer waits on a delivery. - **Soul's P2 passes.** A handler waiting on another thread's write sees it, because nothing orders deliveries (Q7 on main). ### 4. What CultNetDatabase keeps The database is rebuilt on the journal. It has no queue. - **Journal (under the cache gate, per admitted change, in Sequence order):** 1. `TakeDoor(change)`, with the shape rule salvaged from the branch (`:1219-1234`). A put door matches a change whose new document is its instance and whose previous document is not. A removal door matches a change whose previous document is its instance and which has no new one. - The consumption now happens inside the door's own hold, before any observer runs. So an observer's later write to the door's key or instance can never wear the context (the `f3343c8` defect), and unconditional `Dispose`-removal stays correct. 2. Predictions. A `Predicts` door adds the prediction. On an `Authoritative` door (`ApplyPutAsync`, replica put), if a prediction exists for the key, the kind is `Reconciled` and the prediction is removed. - The gate serializes this, so `_predictionGate` and `HasPrediction` are deleted (F5). - Soul's P6 cannot happen: the prediction is recorded before `PutPredictedAsync` returns. 3. Primacy. - On a shard primary, where the door is not `NoLog`: mint, build, append disk-first and record, with a failed append burning the sequence and compacting through it. See "(a) A journal that cannot log a committed change" in Imagination pass 4; the earlier claim that a serialization failure mints nothing is withdrawn. - On a replica, record the primary's entry from the `Replica` door, under its own sequence. - For a local write on a replica, log nothing. 4. Stash `(cache Sequence → CultNetDatabaseChange)` under `_logGate`, for the publisher. - **Watch observer (after the gate, on the writer's thread, before it returns):** it removes the stash entry for `change.Sequence` and publishes it. - A change with no stash was admitted before the database existed and is ignored. - Register the observer first and the journal second, in the constructor, so a stash can never be orphaned. - **Log-before-publish** is structural: the journal runs before the gate opens. - **The replica's absent-key delete (F6)** logs directly under `_logGate`. It is in order because every earlier entry of the same pull loop was journaled inside its own hold, before that `UpsertAsync`/`Remove` returned. This holds even when a pull runs inside a handler, since nested holds journal synchronously. - **The in-memory log is keyed and ordered by sequence.** It becomes a `SortedList` per shard, to match the durable store (`CultNetShardMutationLogStorage.cs:55-64, 75-89`: that store replaces by sequence and orders by sequence). - **Kept from the branch:** - the door table keyed by instance identity, and the shape rule; - "only a primary mints; replicas publish"; - log-before-publish; - the snapshot resync "logs nothing" rule. Its removal door (branch `:733`) must set `Removal = true`, or the shape rule never matches it. - **Database fan-out isolation.** The database's `Subject _changes` (main `:487`) has the Q2 defect one layer up: the subscription server's per-peer handlers sit on it. Replace it with the same isolated observer-array pattern as §2 (about 20 lines). Its exceptions reach the writer through the cache's own per-observer catch. - **`CultNetDatabaseChange` gains `long Sequence`**, the cache change's Sequence, as an optional constructor argument, exactly as `CultCacheDocumentChange` did (composition doc `:57-58`). Without it, in-process latest-state consumers cannot drop stale changes. - **Gone:** F3 (no deferred delivery), F4 (no drainer), F7 (no queue to strand) and F8 (no guard). ### 5. Backpressure and memory bounds - **No buffer exists.** A writer delivers its own changes, and they are held only in its `mine` list for the length of its own call. - **The journal stash** holds at most the changes of holds between their journal and their publication. That is one hold per writing thread, and each entry is removed by the same thread that added it. - **A slow observer slows only the writer whose change it is handling.** That is main's behaviour. It does not slow other writers, and it does not grow memory. This replaces the branch's unbounded drainer backlog (F4, Soul P5). - **The journal runs under the gate, so a slow journal slows every writer.** - The database journal does in-memory work plus one durable `mutationLogStore.Append` per logged change. `Append` rewrites the shard file (`CultNetShardMutationLogStorage.cs:83-89`). - That is I/O under the gate, which the locking doctrine already allows for stores ("A store doing I/O on behalf of its cache ... holds the gate", composition doc `:98-101`). It is still a new cost, so Cut 4's verification measures commit latency with a file log store. - The whole-file rewrite is a pre-existing monolith. It is recorded as a follow-up, not fixed here. ### 6. Cross-runtime impact The Eyes survey was run at `65311c9`. - **`cultcache-rs`, `cultcache-ts` and `cultcache-py` have no publication surface:** no watch, no listener and no Sequence. - Rust: `lib.rs:1970-1975, 2213-2420`. - TypeScript: `cult-cache.ts:164-181`, "Observer delivery is not scheduled here". - Python: `cache.py:30-45`, which refuses reentrant mutation. - **`cultnet-rs`, `cultnet-ts` and `cultnet-py` have no CultNetDatabase equivalent and no cache watcher.** The only server-side shard log is the Python interop test peer (`interop_peer.py:565, 583`), which mints `len(log)+1` under its own lock, inline with the write. That is already the journal's shape. - **Wire parity is unaffected.** - Cache `Sequence` never goes on the wire (composition doc `:47-49`). - `CultNetShardLogEntryMessage` does not change. - `CultNetDatabaseChangeRawMessage` (`CultNetSchemaMessages.cs:1016`) carries no sequence, and this map adds none: stale protection for wire subscriptions is server-side (Cut 6). - No vector in `tests/vectors/` encodes publication order. - **No other runtime must change.** - **Follow-up.** `cultmesh-py`'s `CultMeshDatabase` watch (`node.py:421-505, 1059-1065`) has F2 (a throwing callback skips the rest and reaches the writer), and it has no Sequence. When Python grows cache publication, it takes this contract. ## Cuts Every cut is on CultLib, branch `hands/cultcache-publication` from `main` `65311c9`, in a worktree Self creates. Cuts 1-3 touch only the cache and land first. Cuts 4-6 depend on Cut 3. Heavy verification runs on Yggdrasil. **The negative checks that stay green in every cut** are the `4562340` tests: - `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries`; - `SubscriptionDemandHandlerWritingCacheDuringDeliveryDoesNotDeadlock`; - `SequenceIncreasesInAdmissionOrder`; - `WatchRecordIgnoresStaleSequence`. They are the proof that no scheduler has come back. ### Cut 1. Project before mint (cache, behaviour) - **First.** Commit Q1 as a failing test named for the rule. - **Deletes first.** - Accessor calls in `Index` (`:2798-2804`). - Direct accessor reads in `RefuseVariantIndexSharing` (`:2659-2660`) and `ResolveOne` (`:2616`). - **Adds.** - `Project` in `Admit` (`:2324-2325`). - A per-key projection on `VariantPlan` (`:2456`). - The early returns in `Resolve` (`:2477-2478`, `:2530-2534`) still pass through `Project`, because it is called from `Admit`. - **Authority.** Project owns "this admission's name and index values", and the rules and `Index` read them. `Apply` runs no user code. - **Verification.** - Tests: - **A refused getter mints nothing and changes nothing.** `GetWithSequence` is unchanged, both keys are absent, the name lookup misses, nothing is published, and the next write is previous+1. - **Store-backed batch:** the store file is byte-identical, and a reopen holds neither key. - **A load whose getter throws** refuses the open and leaves the file byte-identical. - **Once per landed record:** a counting getter runs exactly once per landed record, including a variant dependent re-resolved by a base edit. - **The refusal names** the key and the member. - Soul P4. - Negative grep: `rg -n "NameAccessor\?\.Invoke|IndexAccessors" src/GameCult.Caching/CultCache.cs` has no hit in `Apply`, `Index` or `Unindex`. `:315` is store-side and legitimate. - Stryker.NET on the diff. - **Ledger.** About −15 and +45. ### Cut 2. Observer isolation (cache, behaviour) - **Deletes first.** - `Subject _changes` (`:1809`). - `_changes.OnNext` and the false comment (`:2374-2375`). - `_changes.Dispose()` (`:2154`). - **Adds.** - The observer array and `Observable.Create` in `Watch` (`:1887-1897`). - The per-observer catch in `Publish` (`:2369`), with escapes joining `failures`. - `OnCompleted` on `Dispose`. - **Docs.** Composition doc `:77-82`: escaped observer exceptions follow the `OnUpdate` policy. - **Verification.** - Tests: - **Q2 with a rethrowing handler:** the second observer receives every change. The writer gets the escaped exception after delivery, and the thrower is still subscribed (per Q-P2). - **Subscription order:** subscribing and unsubscribing during a delivery leaves every other observer's stream complete. - **Dispose** completes observers. - Soul P3. - The four `4562340` tests stay green. - Stryker on the diff, especially removing the catch and breaking out of the loop on the first failure. - **Ledger.** About −5 and +30. ### Cut 3. The journal (cache, behaviour) - **Adds.** - `AddJournal` (Q-P3), with registration under the gate. - The invocation in `Admit` after `Apply` (`:2328`). - The no-reentry guard in `Held` (`:2341`). - Exceptions routed like observer failures. - **Docs.** A composition-doc paragraph in "Store composition" and in "Locking": journals run under the gate, in Sequence order, and must not enter the cache or wait for anything that needs it. - **Verification.** - Tests: - **Order:** 8 threads × 1000 writes. The journal sees contiguous, strictly increasing Sequences, and each call's changes arrive together. - **Timing:** the journal runs before any observer of the same change, and after `Apply` (`Get` inside the journal sees the new document). - **Reentry:** a journal that writes the cache throws, and the admission stands. - **Refusal:** a refused admission journals nothing. - **Dependents:** a variant dependent is journaled in the same call as its base edit. - **Failure:** a throwing journal still lets the change publish, and the writer gets the exception. - The `4562340` tests stay green. - Stryker on the diff. - **Ledger.** About +50. ### Cut 4. Salvage the branch and rebuild CultNetDatabase on the journal (networking) - **4a (history).** `git merge origin/hands/cultnet-publish-commits` into the cut branch from main `3bf1c0c`. - It is clean: the probe merge `6ec2f73` auto-merged `NetworkingTests.cs` and the composition doc, then built and ran. - The merge brings the tests. The queue dies in 4b. - Self may squash-salvage instead, with the same content. - **4b (subtraction first).** Delete at branch `28248eb` `CultNetDatabase.cs`: - `_parked`, `_ready`, `_nextSequence` and `_draining` (`:502-505`); - `_predictionGate` (`:506-507`); - the Watch subscription and handshake (`:529-531`); - `OnCacheChange` (`:1080-1135`); - `PendingChange` (`:1265-1275`); - `HasPrediction` (`:1209-1213`) and its calls (`:916`, `:1574`); - the logging and prediction bookkeeping in `Deliver` (`:1137-1201`). Main's `PublishCacheUpdate` / `OnUpdate +=` (`:505`, `:1157`, `:1161-1191`) is already gone on the branch. Hands confirms it is absent after the merge. - **4c (the journal).** Register the Watch observer first and the journal second (`AddJournal`), in the constructor. Per admission, under the cache gate, per change in Sequence order: 1. `TakeDoor(change)`, keeping the branch's shape rule (`:1219-1234`). 2. Resolve the shard and kind. Predictions: a `Predicts` door adds its key. An `Authoritative` door (`ApplyPutAsync`, replica put) holding a prediction for the key makes the kind `Reconciled` and removes the key. 3. Log, per §(a): - a replica door records the primary's entry, disk first; - a primary change that is not `NoLog` mints, builds, appends and records, with failure going to the floor; - anything else logs nothing. 4. Stash `(cache Sequence → CultNetDatabaseChange)` under `_logGate`. 5. If step 3 failed, throw once at the end of the call, after all changes are handled (Q-P5). - **4c, publication.** The Watch observer removes the stash entry for `change.Sequence` and publishes it. A missing stash means the change was admitted before this database existed, so it is ignored. `Dispose` removes the journal and the observer and clears the stash. - **4d (the §(a) fixes).** - The floor: `GetCompactedMutationLogSequence`, `GetLatestMutationLogSequence`, and resume in `InitializeLogSequencesFromStore`. - Disk-first `RecordMutationLogEntry`. - A `SortedList` per shard. - The snapshot reads its sequence first (`:660` moves below `:668`'s read). - `_logGate` around `_lastWriteSequence`/`_nextLogSequences` readers (`:1302`, `:1314`, `:1325`). - **Door changes.** - Flags: `Predicts` and `Authoritative`. - The snapshot removal door (branch `:733`) sets `Removal = true`. - The replica absent-key delete logs directly and publishes nothing (branch `:1612-1621`). - Doors neither log nor publish. They register context, write the cache, and dispose the registration. - **Verification.** - Every salvaged test passes: `CultNetPublishesCommitsTests` (13) and `CultNetPublicationOrderTests` (S1, S1b, S2, S3, S4, S6, S7, log-before-publish, in-place re-upsert). - Soul pass 3's P1-P7 are committed and passing. - The pass 4 probes become tests named for the rule: - **A1:** a refused primary `Append` publishes the change. The writer gets `CultNetShardLogException`, which names the shard and `S`. A replica pull answers `compacted_history`, and a snapshot resync converges. - **A2:** the same at the tail with a restart: the replica converges after resync, and no sequence is reused. The store's `CompactThrough` succeeded. - **A3:** a refused replica `Append` leaves the cursor unchanged. The retry converges, and the memory log holds `[1,2]`. - **B1:** a pull on a primary is logged, and the replica converges. - **S:** a snapshot racing a write converges. - **C:** after compacting through 2 and restarting, the next mint is 3. - New tests: - a reentrant `database.PutAsync` from inside a database handler is logged with its context and published once; - a reentrant `ApplyShardLogResponseAsync` from inside a handler keeps the replica log in primary order; - a log store whose `Append` writes the cache fails with Cut 3's reentry guard, wrapped, and the admission stands. - Commit latency with `CultNetFileShardMutationLogStore`, main against Cut 4. Also measure a `CultMesh` node with `EnableDurableShardLogs`, where every bookkeeping write is now logged. Report it; do not gate on it. - Negative grep: `rg -n "_parked|_ready|_draining|_nextSequence|_predictionGate|HasPrediction|PublishCacheUpdate" src/GameCult.Networking` returns nothing. - Review check: no `_cache.` call inside a `lock (_logGate)` block. - The `4562340` tests stay green. - Test runs use `--blame-hang-timeout 2m`. - Stryker on the diff: dropping the floor raise, memory-before-disk, and the snapshot read order must each kill a test. - **Docs.** - Composition doc `:50-53`: replace the "streams derive from `OnUpdate`" sentence. CultNet derives from `Watch` plus one journal, and the log is written in cache order. - A new paragraph: a primary that cannot log a change compacts through it. - The log-store rule under Locking. - **Blockers.** None. Q-P5 is ruled A. - **Ledger.** Against the branch, about −130 and +95. The extra over the old estimate is the floor, the disk-first order, the `SortedList`, the snapshot order, and the exception type. ### Cut 5. Database fan-out isolation and Sequence (networking, behaviour) - **Deletes first.** The database's `Subject _changes` (main `:485`, branch `:487`) and its `Dispose` (`:1158`). - **Adds.** - The isolated observer array, the same pattern as the cache's `Register`/`Publish` (`CultCache.cs:1929-1944`, `:2470-2503`), behind `Watch`, `WatchAllChanges` and the rest (`:1089-1145`). - `CultNetDatabaseChange.Sequence` (`:319-371`), as an optional constructor argument. The journal's stash sets it from the cache change. A change with no cache change behind it carries 0. - **Docs.** The binary-signature note, as the composition doc gave for `CultCacheDocumentChange` (`:57-58`). - **Verification.** - With a rethrowing handler and one throwing database subscriber, the other subscriber and a live subscription-server peer receive every change. The writer gets the exception after delivery (Q-P2). - The disposed-database test. - `Sequence` equals the cache change's. - **Blockers.** Cut 4. - **Ledger.** About −10 and +35. ### Cut 6. Stale protection on wire projections (networking, behaviour; Q-P4 A) - **The rule.** A projection applies a change to a key only if its `Sequence` is above what the projection last applied for that key. A snapshot or reconcile applies everything at or below the cache sequence it read **before** reading documents. - That read is `Cache.GetWithSequence().Sequence`, the branch's idiom (branch `:531`). No new API is needed. - **Subscription server** (`CultNetDatabaseSubscriptionServer.cs`): - `SubscriptionProjectionState` (`:630`) gains `long Floor` and `Dictionary Applied`. - The single-row fast path in `Watch` (`:296-340`) reads the change's `Sequence` (reflection, like `Key`). Under `_lifecycleGate` it drops the change if `Sequence <= max(Floor, Applied[key])`. Otherwise it applies the change and records it. Removals are included. - `Reconcile` (`:347`) and the subscribe snapshot (`:168-181`) read `seq0` before `CreateProjectedSnapshot` (`:402`), set `Floor = seq0`, and clear `Applied`. - This is sound: every change applied earlier was applied under the same gate, after its admission, so its `Sequence` is at most `seq0`. - Hop and unauthorized paths already reconcile from current state, and they set the floor too. - **`CultNetDatabaseServer.CreateSubscription`** (`:501-513`): - A per-subscription `(Floor, Applied)` under a per-subscription lock, held across check, record and send. - `HandleSubscribeAsync`'s snapshot (`:362-369`) takes `seq0` first and sets the floor. - **Verification.** - Two threads write one key, with a blocking cache observer that forces the inverted delivery (Q3 shape). On both servers, the peer's last state for the key equals the cache's. - A removal overtaken by an older upsert leaves the key absent at the peer. - The negative check: without the drop, both tests fail. - **Blockers.** Cut 5, which provides `Sequence` on database changes. - **Ledger.** About +45. ### Salvage from `hands/cultnet-publish-commits` (revised) - **Keep:** - both test files; - the `NetworkingTests.cs` edits; - the door table and the shape rule; - "replicas never mint"; - log-before-publish; - "the snapshot logs nothing"; - "loads and pulls on a primary are logged" (B1); - the composition-doc paragraph, rewritten to name the journal. - **Drop:** - the reorder buffer and the drainer; - `_predictionGate`; - logging from the observer; - the swallowed exception path (A1 and A3 on the branch). ## Subtraction ledger (estimate) | Cut | Removed | Added | Other | |---|---|---|---| | 1 | ~15 | ~45 | none | | 2 | ~5 | ~30 | the cache no longer uses R3 `Subject` | | 3 | 0 | ~50 | one public method (Q-P3) | | 4 | ~130 vs branch | ~95 | the merge brings +1109/-243, of which about 780 are tests; the floor, disk-first order, `SortedList`, snapshot order and `CultNetShardLogException` are the extra over the first estimate | | 5 | ~10 | ~35 | the database `Subject` goes, and one public property | | 6 | 0 | ~45 | none | - **Source, net against main:** about +245. - **Source against the branch:** no scheduler anywhere. - The budget is pressure, not a metric. ## Operator questions - Q-P1..Q-P4 are ruled (see Rulings). Their option analysis is in the map's git history. - **Q-P5 A (ruled 2026-09-29): a primary that cannot log a committed change throws a typed `CultNetShardLogException` naming the shard and the burned sequence, after publication, not derived from `InvalidOperationException`.** - The commit stands, the change is published, and replicas behind it snapshot-resync (Imagination pass 4 (a)). - The writer hears of it. AetheriaEve `Program.cs:3722` cannot mistake it for a denial. Recorded, not forked: - (b): no origin on the journal. Loads and pulls are logged on primaries. - (a)'s restart residual when the store refuses both writes. - The replica keeps retry-by-cursor. - A throwing getter refuses loads too. - Reentrant writes keep main's depth-first delivery. Order is data. - Journal I/O under the gate follows the store precedent. ## Follow-ups outside this campaign - **`cultmesh-py` watch.** It has no isolation and no Sequence (`node.py:1059-1065`). - **Durable log whole-file rewrite.** `CultNetShardMutationLogStorage.cs:83-89` rewrites the whole file per `Append`. It is a monolith, and under the gate its cost is now every writer's. - **Mesh database-backed handles.** `CultMesh.cs:2362` and `:3396` can drop stale changes once `CultNetDatabaseChange.Sequence` exists. - **Skill scar.** The Eureka SKILL.md already says "a ticket system that serialized delivery was built, deadlocked, and was deleted". The Soul pass 3 direction re-proposed it anyway. The Soul brief template should tell Soul to check `git log -S` for a proposed mechanism before recommending it. - **A chained replica after a snapshot.** A replica that snapshot-resyncs has no log entries below the snapshot sequence. It should raise its own floor to that sequence, so a replica chained on it resyncs instead of seeing a gap (`ApplyShardSnapshotResponseAsync` `:675-736`). It predates this campaign. - **AetheriaEve `Program.cs:3722`.** It reads a committed write's `InvalidOperationException` as a denial. Under Q-P5 A, Cut 4's log failures cannot reach it. Observer and `OnUpdate` exceptions still can, so the Cuts 1-2 recording stands.- **Cut 4 Soul, recorded 2026-09-30 (Self), not fixed in Cut 4:** - **File log store is not crash-safe (hard prerequisite).** `CultNetFileShardMutationLogStore.Append` and `CompactThrough` use `File.WriteAllBytes` (truncate, rewrite, no temp+rename, no fsync). A failure mid-write truncates the log; every later append burns a sequence and a restart cannot construct the database. It also rewrites the whole file per append, which Cut 4 now does under the cache gate (2.2 ms per write at 100 entries, 33 ms at 950, a 2.49 s stall). An atomic, append-only store must land before any live daemon enables durable shard logs. Today only AetheriaEve's offline tools do; the live Aetheria daemon passes `enableDurableShardLogs: false`. - **A slow log append blocks cache readers** (`CultCache.Get` takes the gate). Accepted with a note in the store's XML doc. - **Crash between the store write and the log append** leaves the log lagging the store, and nothing detects it; a restarted primary reuses that sequence (A2's divergence reached by a plain kill). The spec called this lag detectable; for a crash it is not. - **Cut 4 final Soul (4d), recorded 2026-09-30 (Self), not fixed in Cut 4:** - **F2, read cost.** `CultNetDatabase.Serves` (`:1980-1989`) runs per row in `GetAll`/`GetByName`/`GetByIndex` for every layout except a single owns-everything shard: a reflection `GetTrackedKey`, a gate lock and a shard re-scan per row. `database.GetAll` measured 24.8x `cache.GetAll` on 20,000 rows under a key-prefixed shard. Fix: filter by key inside the cache enumeration (the `rowFilter` path already has keys) and hoist the owns-everything check. - **F4, dispose drain edges.** An observer that removed its stash entry before the drain can publish after `Dispose` returns (dropped); a throwing `_cacheJournal.Dispose()` skips the drain. Neither double-publishes or deadlocks. - **F5, refusal naming.** A write whose schema is owned but whose key is outside the shard's prefix also answers `unowned_schema` with details `{field:"schemaId"}`; the peer cannot tell a key refusal from a schema refusal. - **Pre-existing, separate task.** Python emits error code `unsupported_schema_version` (`cultnet-py/.../interop_peer.py:415`, `cultmesh-py/.../server.py:161`), which is not in the error-code enum; TS and Rust refuse the whole message.