840 lines
50 KiB
Markdown
840 lines
50 KiB
Markdown
# CultCache publication: the cache owns "published once", and order stays data
|
||
|
||
Status, 2026-09-29: Cuts 1-3 landed on `main`. Cut 3 merged at `ae8576d`. Imagination pass 4 re-took Cuts 4-6
|
||
against `3bf1c0c`. Q-P5 is ruled (A). Cut 4 is ready for Hands.
|
||
- F4, pinning `Held`'s publish-on-throw guard, is done: Cut 3 pinned it.
|
||
- F3 (observer removal on unsubscribe) and F1 (the false "getters run once" claim for conditional commits on variant
|
||
stores) were Cut 3 follow-ups; confirm against the Cut 3 merge.
|
||
|
||
Recorded:
|
||
- F2, the Q-P2 hazard. A rethrown observer exception makes a committed write look cancelled or refused. The
|
||
only live typed catch around a cache write is AetheriaEve `Aetheria.State.Daemon/Program.cs:3722`. It needs a
|
||
fail-fast handler, and no production code installs one.
|
||
- F5: subscribe and unsubscribe cost O(n).
|
||
- F6: writes after `Dispose` are dropped silently.
|
||
|
||
Map written by Imagination pass 1, 2026-09-29. The ends are the
|
||
operator ruling of 2026-09-30: every committed cache change reaches subscribers
|
||
exactly once. This document owns the means.
|
||
|
||
- **Body.** CultLib `main` `65311c9ad770828adafe40c2c14b2d3da517628e`. Every
|
||
`file:line` is against it unless it names the branch.
|
||
- **Superseded attempt.** `origin/hands/cultnet-publish-commits`, tip
|
||
`28248eb`, base `f9f107a`. Soul pass 3 said do not merge it, and that stands.
|
||
- Main has not touched `CultCache.cs`, `CultNetDatabase.cs`, the composition
|
||
doc or the networking tests since `f9f107a`: `git diff --stat` is empty.
|
||
- `git merge-tree 65311c9 28248eb` is clean.
|
||
- **Scope.** The C# cache's publication and CultNetDatabase's use of it. There
|
||
is no wire-format change and no other runtime changes (§6).
|
||
|
||
|
||
## Rulings (operator, 2026-09-30)
|
||
|
||
- **Q-P1 A: order as data.** Each change is published exactly once, on its writer's thread, before the write
|
||
returns, and carries its `Sequence`. The shard log is written in cache order by the gate-held journal. Wire
|
||
peers drop changes that are stale by `Sequence`. The turnstile (`bcae483`/`4562340`) and the drainer are
|
||
both rejected.
|
||
- **Q-P2 A:** a throwing observer does not stop the others. Every observer runs, the exception is rethrown to
|
||
the writer afterwards, and the observer stays subscribed. The commit stands.
|
||
- **Q-P3 A:** a public `CultCache.AddJournal(...)`, with reentry guarded.
|
||
- **Q-P4 A:** Cut 6, which drops stale changes per key on the wire, lands in this campaign.
|
||
- **Q-P5 A (2026-09-29):** a primary that cannot log a committed change throws a typed `CultNetShardLogException`
|
||
naming the shard and the burned sequence, after publication, not derived from `InvalidOperationException`.
|
||
|
||
## The finding that shapes this map
|
||
|
||
The direction Imagination was handed was: "ordered delivery belongs in the
|
||
cache's own publication, on the writer's thread, so a writer delivers its own
|
||
changes before returning". That design **has been built before, and it was
|
||
deleted**.
|
||
|
||
- `bcae483` (2026-09-13 20:09) made the cache a delivery scheduler: tickets, a
|
||
delivery gate, turn waits and per-thread queues for reentrant writes.
|
||
- `4562340`, 34 minutes later, deleted it: "It deadlocked inside CultLib: a
|
||
load's delivery reaching a CultNetDatabaseSubscriptionServer observer that
|
||
needs _lifecycleGate, while a subscribe's demand handler holding that gate
|
||
writes the cache and waits for the delivery's turn."
|
||
- It replaced it with **order as data**: a per-cache `Sequence`, and no
|
||
delivery scheduler.
|
||
- The regression tests it left, `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries`
|
||
(`ConditionalCommitTests.cs:763`) and
|
||
`SubscriptionDemandHandlerWritingCacheDuringDeliveryDoesNotDeadlock`
|
||
(`NetworkingTests.cs:2244`), are live on main.
|
||
|
||
This pass rebuilt that scheduler as a prototype. It is the per-cache turnstile
|
||
with same-thread trampolining at `79f4639`, and it reproduces the deletion:
|
||
|
||
- Caching tests: 278 of 279 pass, and `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries`
|
||
fails.
|
||
- The networking suite hangs, and Yggdrasil's idle watchdog killed it at 900 s.
|
||
The blame-hang run that names the hung test is recorded below.
|
||
|
||
The reason is structural, not an implementation slip. Once a writer waits for
|
||
another thread's delivery, any lock that an observer takes and a writer holds
|
||
becomes a deadlock:
|
||
|
||
- total cross-thread order;
|
||
- a writer delivering before it returns;
|
||
- no writer waiting on another thread.
|
||
|
||
Pick two. The branch's drainer chose order plus no-wait, and lost
|
||
before-return (F3 and F4). The turnstile chose order plus before-return, and
|
||
lost no-wait (the `bcae483` deadlock).
|
||
|
||
**So this map keeps order as data (the `4562340` ruling) and moves the one
|
||
order-sensitive consumer, the CultNet mutation log, to the place order is
|
||
decided: under the cache gate.**
|
||
|
||
- Every writer keeps delivering its own changes, on its own thread, before it
|
||
returns, exactly as main does. F3 is satisfied by not regressing it.
|
||
- The log becomes a **journal** written in the admitting hold, in Sequence
|
||
order.
|
||
- The reorder buffer and drainer die, and no scheduler replaces them. This
|
||
departs from Soul's stated direction on purpose, and **Q-P1** puts it to the
|
||
operator.
|
||
|
||
## Probe evidence
|
||
|
||
All of this ran on Yggdrasil via `ygg-verify.sh`, from scratch commits in a
|
||
throwaway worktree that no branch holds.
|
||
|
||
- Probe file: `tests/GameCult.Caching.Tests/PublicationProbeTests.cs` in scratch commit `79f4639`. The worktree has been removed. The commit is unreferenced in the local object store, where gc can collect it; it also survives on the Yggdrasil mirror as `refs/verify/79f4639f8f5b95db7ce16f244a081c1ec101a468`.
|
||
- `9c5abf5` is the probes on main. `79f4639` is the same probes plus the
|
||
turnstile prototype in `CultCache.cs`.
|
||
|
||
| # | Probe | main `65311c9` | Turnstile prototype `79f4639` |
|
||
|---|---|---|---|
|
||
| Q1 | A batch `{ok, boom}` where boom's `[CultName]` getter throws | `TargetInvocationException`. `_sequence` goes from 0 to 1, because ok was minted. **Both ok and boom are in memory**, and `GetByName("ok")` finds ok. **Nothing is published**, and the next write gets Sequence 2. F1 is worse than stated: memory is corrupt and unpublished. In a store-backed cache, `land` (`:2325`) already ran, so the store holds it too. | No stall, but memory is still corrupt. |
|
||
| Q2 | Two `Watch` observers, the first throws | With the default R3 handler, the second observer sees 2 of 2 and the writer sees nothing. **With a rethrowing handler, the second observer sees 0 of 2 and every writer gets the exception.** The comment at `CultCache.cs:2374`, "OnNext does not throw", is false under fail-fast. | With its own observer array and a per-observer catch, the second observer sees 2 of 2 and the writer gets the exception afterwards. |
|
||
| Q3 | Writer A's observer blocks on `a`, then B writes `b` | The recorder sees `b(2)` before `a(1)`. There is no cross-thread order, by design since `4562340`. | Sees `a` then `b`, because B waited for A. |
|
||
| Q4 | An observer on `x` writes `y` | `y(2)` before `x(1)`: nested, depth-first, and the nested write returns after `y` is published. | `x` then `y`, and the nested write returns before `y` is published. |
|
||
| Q5 | `OnUpdate` on Upsert and on Commit | 0 fires, so loads only. `Watch` sees 2. | Unchanged. |
|
||
| Q6 | 8 threads × 1000 writes, plus reentrant writes | 67 ms, 2645 of 8800 cross-thread inversions, and 0 writes returned undelivered. | 516 ms, 0 inversions, 0 undelivered. |
|
||
| Q7 | Soul's P2 shape: a handler waits 1 s for another thread's write | Observed in time. | Not observed in time. |
|
||
| Suites | Caching, Networking and Mesh | (green on main) | Caching fails `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries`, and Networking **hangs** until the 900 s idle kill. Rerun with `--blame-hang-timeout 90s`: 286 passed, then the test host hung and was dumped. The blame sequence file was not extracted, so **the hung test is not named**. It is consistent with, but not proven to be, `SubscriptionDemandHandlerWritingCacheDuringDeliveryDoesNotDeadlock`, whose own comment says "disposal would hang too". |
|
||
|
||
## Imagination pass 4: Cut 4 re-taken on the landed journal
|
||
|
||
Cut 3 landed at `ae8576d`. Its Soul left two questions that Cut 4 had to settle before relying on the
|
||
journal: (a) what a throwing journal means for the log, and (b) whether a journal needs the change's origin.
|
||
Both have one coherent answer. The one fork, Q-P5, was about who hears of a log failure; the operator ruled it A (2026-09-29).
|
||
|
||
### Probe evidence (pass 4)
|
||
|
||
- Probe file: `Pub4ProbeTests.cs` (Networking tests). It was run on Yggdrasil from scratch commits:
|
||
- `0d7397d`: main plus the probes;
|
||
- `6ec2f73`: the same, merged with the branch `28248eb`;
|
||
- `376cd74`: main plus probe C.
|
||
- The commits survive on the Yggdrasil mirror. The scratch worktree is gone.
|
||
- The probes report facts and assert nothing. `FlakyLogStore` is an in-memory `ICultNetShardMutationLogStore`
|
||
that can refuse `Append` or run a hook inside `GetCompactedThrough`.
|
||
|
||
| # | Probe | main `3bf1c0c` | branch `28248eb` (queue) |
|
||
|---|---|---|---|
|
||
| A1 | Primary with a durable log. Put 1, Put 2 (its `Append` throws), Put 3. A replica pulls, then retries. | Writer gets `IOException`. The cache holds 1, 2 and 3, but **2 is never published** (2 of 3). Memory log `[1,2,3]`, durable `[1,3]`. The replica pull throws "log has a gap. Expected 2, received 3", and so does **every retry**: the replica is stuck for good. | **Writer sees nothing**: R3 swallowed the drainer's exception. 2 is not published. The replica is stuck in the same way. |
|
||
| A2 | The same failure at the tail, then the primary restarts over the same cache and log store. | The replica applied 1. After restart, note 3 is **minted sequence 2**. The replica takes it, reports applied 2, and holds `1,3` against the primary's `1,2,3`: **silent divergence**, a reused sequence. | Same. |
|
||
| A3 | A replica whose log `Append` throws once. | The pull throws and the cursor stays at 0. The retry converges (`1,2`), but the memory log is `[1,1,2]`: a duplicate, because memory is written before disk and kept as a `List`. | The pull "succeeds": the exception was swallowed and the cursor went to 2. The durable log is `[2]`, so **entry 1 is lost**, and a chained replica sees a gap. |
|
||
| B1 | Primary with a file store. Another writer puts `pulled` into the file, the primary runs `PullAll`, and a replica pulls the log. | `pulled` is published but **not logged**. The replica gets nothing. | Logged. The replica converges on `pulled`. |
|
||
| B2 | A store attached after the database exists. | Nothing is logged. | All three hydrated records are logged. |
|
||
| S | `CreateShardSnapshotResponse` with a write landing between its document read and its sequence read (hooked in `GetCompactedThrough`). | The snapshot holds `[1]` at log sequence 2. The replica applies it and pulls, and still lacks 2: **divergence**. | Same. |
|
||
| C | Compact through 2, restart, then Put 3. | Note 3 is **minted sequence 1**, and `GetLatestMutationLogSequence` returns 1, below the floor of 2. | (not run; the code is unchanged) |
|
||
|
||
What the probes establish:
|
||
- A1, A2, S and C are pre-existing defects on main. Cut 4 inherits them and must not build on them: the
|
||
resync path it relies on for (a) is itself broken by S and C.
|
||
- On a replica, the pull loop already retries. The cursor advances only after an entry applies
|
||
(`CultNetDatabase.cs:1038-1083`), so a replica needs no new mechanism. It needs only to keep the exception
|
||
reaching that loop, which the branch's queue broke (A3).
|
||
- Loads on a primary must be logged, or replicas diverge (B1).
|
||
|
||
### (a) A journal that cannot log a committed change: one answer
|
||
|
||
The facts:
|
||
- The log store is a second durable store written after the cache's store. Without an atomic commit across
|
||
the two, the log can lag the store. It must never lead: a replica would then apply a change the primary
|
||
refused.
|
||
- So lag is unavoidable, and it has to be detectable and repaired.
|
||
- The protocol already has a repair, and nothing else needs to be invented.
|
||
- "History the primary cannot serve" is `compacted_history`: `CultNetDatabaseServer.cs:189-201` answers
|
||
`ResyncRequired`.
|
||
- `CultNetShardReplication.cs:199-214` then snapshot-resyncs.
|
||
|
||
**Rule: a primary that cannot log a committed change burns that change's sequence and compacts through it.**
|
||
|
||
- In the database journal, on a primary, for each change that is not `NoLog`:
|
||
1. Mint `S`.
|
||
2. Build the entry: the wire message from the door, or `ToLogEntryMessage`.
|
||
3. `Append` durably, **then** record it in memory. Disk goes first, so memory never serves what disk lacks.
|
||
- If step 2 or 3 throws:
|
||
- record nothing;
|
||
- raise the database's in-memory floor for the shard to `S`;
|
||
- try `store.CompactThrough(shard, S)`;
|
||
- still set `_lastWriteSequence` to `S`, because the row was written at `S`;
|
||
- carry on with the rest of the admission;
|
||
- throw `CultNetShardLogException` once at the end of the journal call (Q-P5 A).
|
||
- Every replica behind `S` is then told `compacted_history` and resyncs by snapshot. The sequence gap at `S`
|
||
is never served, because the floor covers it.
|
||
- This **replaces** the old §4 step 3 claim, "a serialization failure then mints nothing, so the log can never
|
||
hold a gap". Minting nothing hides a replication gap behind a dense sequence. That is exactly A2's silent
|
||
divergence.
|
||
- **Replica:** there is no floor. The journal's exception reaches `ApplyShardLogResponseAsync` and the cursor
|
||
does not advance. The next pull re-applies the entry: a fresh instance, so the door matches again, and the
|
||
entry is re-recorded under the primary's sequence. The branch's absent-key direct log follows the same
|
||
disk-first order.
|
||
- **Required fixes** that the rule depends on, all in Cut 4:
|
||
- `GetCompactedMutationLogSequence` (`CultNetDatabase.cs:597-602`) returns
|
||
`max(store floor, in-memory floor)`.
|
||
- `InitializeLogSequencesFromStore` (`:1363-1395`) resumes at
|
||
`max(highest retained, store.GetCompactedThrough(shard)) + 1`. This is probe C.
|
||
- `GetLatestMutationLogSequence` (`:607-616`) never reports below the floor.
|
||
- `CreateShardSnapshotResponse` (`:624-670`) reads `GetLatestMutationLogSequence` **before** `SelectAll`, on
|
||
both returns, `:645` and `:668`. The documents then reflect at least that sequence, and replaying later
|
||
entries converges: they are whole-document puts and deletes, applied in order. This is probe S.
|
||
- `RecordMutationLogEntry` (`:1331-1352`) writes disk first. The in-memory log becomes
|
||
`SortedList<long, entry>` per shard, replacing by sequence as the durable store does. This fixes A3's
|
||
duplicate.
|
||
- **Residual, recorded rather than forked.** Suppose the store refuses both `Append` and `CompactThrough`, and
|
||
then the process restarts.
|
||
- The durable log does not know about the hole, so A2's tail reuse can recur.
|
||
- Fixing that needs a durable write per mint, a sequence reservation, which doubles log I/O under the gate.
|
||
- The thrown exception names both failures, so the outage is loud.
|
||
- Refusing later writes ("fail closed") is not coherent. The database can refuse only its own doors, while
|
||
bare cache writes are journaled regardless: that would be split authority.
|
||
|
||
### (b) A pulled or loaded change on a primary: no origin needed
|
||
|
||
- On a primary the log is the shard's **state history**, whoever wrote the state. B1 shows that leaving loads
|
||
out of the log leaves replicas permanently diverged.
|
||
- The only origins that change what is logged are already carried by the door table, keyed by instance
|
||
identity and matched by shape. They are:
|
||
- `Replica` (log the primary's entry);
|
||
- `NoLog` (prediction, snapshot resync);
|
||
- `Put`/`Delete` (the wire message);
|
||
- `Predicts`/`Authoritative`.
|
||
- A load or pull has no door, so on a primary it is logged like any unattributed commit. On a replica it logs
|
||
nothing, like any local write. That is what the branch does, and B1 converges under it.
|
||
- Cost:
|
||
- B2: a store attached **after** the database logs its hydration.
|
||
- `CultNetHost` opens the cache before building the database (`CultNetLocal.cs:146-152`), so production
|
||
hydration is not journaled.
|
||
- A re-pull admits only the records that changed. In B1 the seed was not re-logged.
|
||
- `AddJournal` keeps its signature. If a future consumer (audit, say) needs "load or commit", the fact goes on
|
||
`CultCacheDocumentChange` as an additive property, as `Sequence` did. The cache already knows it: `Admit`'s
|
||
`source != null` at `CultCache.cs:2420`. It never needs a new `AddJournal` shape.
|
||
- This is recorded, not forked: no consumer needs it today.
|
||
|
||
### The two recorded hazards, settled for Cut 4
|
||
|
||
- **Q-P2 typed-exception hazard.** Cut 4 makes it frequent. Every cache write on a primary with a durable log
|
||
now does log I/O inside its admission.
|
||
- AetheriaEve opts into durable logs (`Aetheria.State/AetheriaStateNode.cs:144`).
|
||
- Its `AetheriaHangarCommandJournal.ValidateAsync` writes an envelope (`Program.cs:6509`) inside the `catch
|
||
(InvalidOperationException)` at `Program.cs:3722`.
|
||
- A log failure surfacing as `InvalidOperationException` would therefore produce a "denied" receipt for a
|
||
committed envelope.
|
||
- Q-P5 A (ruled): the writer gets a typed `CultNetShardLogException` that **does not derive from
|
||
`InvalidOperationException`**.
|
||
- **Journal deadlock (the Cut 3 note).** The database journal waits only on `_logGate` and on the log store's
|
||
own lock.
|
||
- Lock order: cache gate, then `_logGate`, then the log store's lock.
|
||
- Invariant: **no code holding `_logGate` calls the cache, and no log store calls the cache or waits on a
|
||
thread that does.**
|
||
- Readers that take `_logGate` read only log state: `GetMutationLog`, `LastWriteSequence`, `CurrentAsOf`.
|
||
Readers already inside a cache read, such as the selection evaluator's `ordinalOf`, nest `_logGate` under
|
||
the cache gate, which is the same order.
|
||
- The store rule goes into the `ICultNetShardMutationLogStore` XML doc (`CultNetShardMutationLogStorage.cs:13-34`),
|
||
because a user store runs under the cache's gate.
|
||
- A log store that writes the cache already throws, through Cut 3's guard. Cut 4 pins it with a test.
|
||
|
||
## Authority map
|
||
|
||
- **Owner.** `CultCache`. The hold that mints a Sequence range also:
|
||
1. journals the range under the gate, in Sequence order (new);
|
||
2. publishes the range to every observer, isolated, on the minting writer's
|
||
thread, after it releases the gate and before it returns (main's model,
|
||
now isolated).
|
||
|
||
Nothing waits on another call's delivery.
|
||
- **Inputs.**
|
||
- The changes returned by `Apply` (`CultCache.cs:2707`), the single mutation
|
||
site.
|
||
- The pre-mint projection of names and indexes (Cut 1).
|
||
- The journal list, the observer array and `OnUpdate`.
|
||
- **Outputs.**
|
||
- Journal calls: one per admission, under the gate, in Sequence order.
|
||
- `Watch` / `WatchRecord` notifications that carry `Sequence`.
|
||
- `OnUpdate` for loads.
|
||
- Observer and journal exceptions, rethrown to the writer.
|
||
- **Derived state.**
|
||
- `Sequence` is order as data.
|
||
- CultNet's primary log sequence is minted in the journal, so log order is
|
||
cache order.
|
||
- A replica's log sequence is copied from the primary entry.
|
||
- The database change kind (`Predicted`, `Reconciled`, `Added`, `Updated`,
|
||
`Removed`) is decided in the journal.
|
||
- `CultNetDatabaseChange.Sequence` (new) is copied from the cache change.
|
||
- Wire projections drop stale by it (Cut 6).
|
||
- **Forbidden writers.**
|
||
- Nothing re-sequences or schedules cache changes. The branch's `_parked`,
|
||
`_ready`, `_nextSequence`, `_draining`, `PendingChange` and the drain body
|
||
of `OnCacheChange` die, and so do the turnstile, `bcae483` and the
|
||
prototype.
|
||
- No R3 `Subject` fans out cache changes (Q2).
|
||
- `Apply` and `Index` run no user code.
|
||
- Doors never read or write prediction state: the branch's `HasPrediction`
|
||
and `_predictionGate` die.
|
||
- The database's Watch observer never logs; only the journal logs.
|
||
- A journal never enters the cache.
|
||
- **Shared paths.** Every path goes `Held` → `Admit` → `Resolve` → **Project**
|
||
→ `land` → `Apply` → **journal** → leave gate → **publish**:
|
||
- loads: `AddBackingStore` `:1909`, `Loaded` `:1940`, `PullAll*` `:1959` and
|
||
`:3052`;
|
||
- `Commit` / `TryCommit` (`Land`, `:2272`);
|
||
- `UpsertAsync`, `UpsertVariantAsync`, `FlattenAsync` and `Remove`
|
||
(`:2004-2056`, `Write` `:2252`);
|
||
- a variant dependent's re-resolution (`VariantPlan.Dependents`, in the same
|
||
`Apply`);
|
||
- direct calls on an attached store (`:2924`);
|
||
- `Dispose` (`:2142`);
|
||
- the database doors, replica apply and snapshot resync, which only write
|
||
the cache and hand context to the journal.
|
||
|
||
The single exception is the replica's absent-key delete, which has no cache
|
||
change. It logs directly (§4).
|
||
- **Deletion line.**
|
||
- In the cache: `Subject<Change> _changes` (`:1809`), the `_changes.OnNext`
|
||
call and its false comment (`:2374-2375`), `_changes.Dispose()`
|
||
(`:2154`), and the accessor calls in `Index` (`:2798-2804`).
|
||
- In the database, on the salvaged branch: the reorder buffer and drainer,
|
||
`_predictionGate` and `HasPrediction`, the logging in `Deliver`, and the
|
||
`Subject<object> _changes` fan-out.
|
||
|
||
## The design, question by question
|
||
|
||
### 1. Validation before minting: a refused change mints nothing
|
||
|
||
- **Where.** Add **Project** to `Admit` (`:2313`), between `Resolve` (`:2324`)
|
||
and `land` (`:2325`).
|
||
- **What.** Over `plan.Admitted ∪ plan.Dependents`, it evaluates once per
|
||
record: `NameAccessor(document)` and every `IndexAccessors[alias](document)`.
|
||
It keeps `(Name, (Alias, Value)[])` per key on the plan.
|
||
- **A throwing getter refuses the whole admission.** It raises an
|
||
`InvalidOperationException` that names the key, the member and the unwrapped
|
||
message; today the caller sees a bare `TargetInvocationException`. The
|
||
refusal comes before `land`, so the admission writes no store, touches no
|
||
memory and mints nothing.
|
||
- **Readers take the projection, not the document.**
|
||
- `Index` (`:2793-2806`).
|
||
- `RefuseVariantIndexSharing` (`:2659-2660`).
|
||
- The variant-name rule in `ResolveOne` (`:2616`). Its ancestor read at
|
||
`:2621` reads held documents, pre-land, and may stay.
|
||
|
||
`Resolve` currently runs the refusal rules before any projection exists.
|
||
Hands makes Project the first pass over the resolved plan and moves both
|
||
rules after it, so rules read the projection and only the projection reads
|
||
documents.
|
||
- **Apply becomes user-code-free.** It then does only dictionary and
|
||
`ConditionalWeakTable` work on precomputed data.
|
||
- **Loads refuse too.** A store whose record's getter throws fails to open,
|
||
loudly. That is the rule "hydration failure on open is loud" (composition doc
|
||
`:88-92`). Today it half-indexes silently.
|
||
|
||
### 2. Observer isolation
|
||
|
||
- **Replace the `Subject<Change>` with the cache's own observer array**,
|
||
copy-on-write under a small lock.
|
||
- `Watch<T>` (`:1887`) becomes `Observable.Create<Change>(register)` followed
|
||
by the existing `Where` and `Select`.
|
||
- `Dispose` calls `OnCompleted` on each observer, which is what
|
||
`Subject.Dispose` did.
|
||
- **Publish loops change → observer, and wraps each `OnNext` in its own
|
||
try/catch.**
|
||
- R3's `Observer<T>` already catches handler exceptions (`OnErrorResume`, then
|
||
the unhandled handler). So an exception reaches the cache only when fail-fast
|
||
was asked for: a rethrowing `ObservableSystem` handler, or a custom observer.
|
||
- Escaped exceptions join the `OnUpdate` failures. They are rethrown to the
|
||
writer after every observer has received every change of the hold. That is
|
||
the existing `OnUpdate` policy (composition doc `:77-82`).
|
||
- The observer stays subscribed (**Q-P2**). The commit stands either way.
|
||
- **Prototype.** At `79f4639` (Q2) this took about 15 lines. Hands keeps that
|
||
part and drops the turnstile around it.
|
||
|
||
### 3. Delivery thread, order and reentrancy
|
||
|
||
- **Unchanged from main, on purpose.** Each outermost hold publishes exactly its
|
||
own changes, on its thread, after it leaves the gate and before it returns
|
||
(composition doc `:59-65`, `Held` `:2339`).
|
||
- A write from inside a handler is its own outermost hold. It delivers
|
||
depth-first before returning, as Q4 shows on main.
|
||
- There is no cross-thread order. Every change carries `Sequence`, and
|
||
latest-state consumers drop stale changes by it (doc `:68-73`, and the Mesh
|
||
mirrors at `CultMesh.cs:1534-1540`).
|
||
- **Why this is not a regression of the ruling.** The ruling asks for
|
||
exactly-once, not for a total delivery order.
|
||
- Exactly-once in the cache comes from the single mint site plus isolation
|
||
plus Cut 1 (§1-2).
|
||
- Order is needed by exactly one consumer, the shard log. It gets that order
|
||
from the journal (§4), not from delivery.
|
||
- **The journal is the new mechanism.**
|
||
- Signature: `IDisposable CultCache.AddJournal(Action<IReadOnlyList<CultCacheDocumentChange<object>>>)`.
|
||
Its name and shape are **Q-P3**.
|
||
- It is called in `Admit` right after `Apply` (`:2328-2329`), under the gate,
|
||
once per admission, with that admission's changes in Sequence order.
|
||
Journals run in registration order.
|
||
- A journal that enters the cache, meaning any hold on this cache while a
|
||
journal runs, throws `InvalidOperationException`. It is guarded by a flag
|
||
checked in `Held`'s `Monitor.IsEntered` branch (`:2341`). This is the
|
||
store rule ("a store call must not wait for a write to its own cache"),
|
||
enforced.
|
||
- Journal exceptions follow the observer policy: the admission stands, it is
|
||
still published, and the exception is rethrown to the writer after
|
||
publication.
|
||
- Adding or removing a journal takes the gate, so registration is atomic with
|
||
respect to admissions.
|
||
- **Lock order.** Cache gate, then journal-internal leaf locks (the database's
|
||
`_logGate`). Observers still run outside the gate, so the `bcae483` cycle
|
||
(an observer needing `_lifecycleGate` while a gate holder writes the cache)
|
||
cannot form. No journal takes `_lifecycleGate`, and no writer waits on a
|
||
delivery.
|
||
- **Soul's P2 passes.** A handler waiting on another thread's write sees it,
|
||
because nothing orders deliveries (Q7 on main).
|
||
|
||
### 4. What CultNetDatabase keeps
|
||
|
||
The database is rebuilt on the journal. It has no queue.
|
||
|
||
- **Journal (under the cache gate, per admitted change, in Sequence order):**
|
||
1. `TakeDoor(change)`, with the shape rule salvaged from the branch
|
||
(`:1219-1234`). A put door matches a change whose new document is its
|
||
instance and whose previous document is not. A removal door matches a
|
||
change whose previous document is its instance and which has no new one.
|
||
- The consumption now happens inside the door's own hold, before any
|
||
observer runs. So an observer's later write to the door's key or
|
||
instance can never wear the context (the `f3343c8` defect), and
|
||
unconditional `Dispose`-removal stays correct.
|
||
2. Predictions. A `Predicts` door adds the prediction. On an `Authoritative`
|
||
door (`ApplyPutAsync`, replica put), if a prediction exists for the key,
|
||
the kind is `Reconciled` and the prediction is removed.
|
||
- The gate serializes this, so `_predictionGate` and `HasPrediction` are
|
||
deleted (F5).
|
||
- Soul's P6 cannot happen: the prediction is recorded before
|
||
`PutPredictedAsync` returns.
|
||
3. Primacy.
|
||
- On a shard primary, where the door is not `NoLog`: mint, build, append disk-first and record, with a failed
|
||
append burning the sequence and compacting through it. See "(a) A journal that cannot log a committed
|
||
change" in Imagination pass 4; the earlier claim that a serialization failure mints nothing is withdrawn.
|
||
- On a replica, record the primary's entry from the `Replica` door,
|
||
under its own sequence.
|
||
- For a local write on a replica, log nothing.
|
||
4. Stash `(cache Sequence → CultNetDatabaseChange)` under `_logGate`, for the
|
||
publisher.
|
||
- **Watch observer (after the gate, on the writer's thread, before it
|
||
returns):** it removes the stash entry for `change.Sequence` and publishes it.
|
||
- A change with no stash was admitted before the database existed and is
|
||
ignored.
|
||
- Register the observer first and the journal second, in the constructor, so
|
||
a stash can never be orphaned.
|
||
- **Log-before-publish** is structural: the journal runs before the gate opens.
|
||
- **The replica's absent-key delete (F6)** logs directly under `_logGate`. It
|
||
is in order because every earlier entry of the same pull loop was journaled
|
||
inside its own hold, before that `UpsertAsync`/`Remove` returned. This holds
|
||
even when a pull runs inside a handler, since nested holds journal
|
||
synchronously.
|
||
- **The in-memory log is keyed and ordered by sequence.** It becomes a
|
||
`SortedList<long, entry>` per shard, to match the durable store
|
||
(`CultNetShardMutationLogStorage.cs:55-64, 75-89`: that store replaces by
|
||
sequence and orders by sequence).
|
||
- **Kept from the branch:**
|
||
- the door table keyed by instance identity, and the shape rule;
|
||
- "only a primary mints; replicas publish";
|
||
- log-before-publish;
|
||
- the snapshot resync "logs nothing" rule. Its removal door (branch `:733`)
|
||
must set `Removal = true`, or the shape rule never matches it.
|
||
- **Database fan-out isolation.** The database's `Subject<object> _changes`
|
||
(main `:487`) has the Q2 defect one layer up: the subscription server's
|
||
per-peer handlers sit on it. Replace it with the same isolated
|
||
observer-array pattern as §2 (about 20 lines). Its exceptions reach the
|
||
writer through the cache's own per-observer catch.
|
||
- **`CultNetDatabaseChange<T>` gains `long Sequence`**, the cache change's
|
||
Sequence, as an optional constructor argument, exactly as
|
||
`CultCacheDocumentChange` did (composition doc `:57-58`). Without it,
|
||
in-process latest-state consumers cannot drop stale changes.
|
||
- **Gone:** F3 (no deferred delivery), F4 (no drainer), F7 (no queue to
|
||
strand) and F8 (no guard).
|
||
|
||
### 5. Backpressure and memory bounds
|
||
|
||
- **No buffer exists.** A writer delivers its own changes, and they are held
|
||
only in its `mine` list for the length of its own call.
|
||
- **The journal stash** holds at most the changes of holds between their
|
||
journal and their publication. That is one hold per writing thread, and each
|
||
entry is removed by the same thread that added it.
|
||
- **A slow observer slows only the writer whose change it is handling.** That
|
||
is main's behaviour. It does not slow other writers, and it does not grow
|
||
memory. This replaces the branch's unbounded drainer backlog (F4, Soul P5).
|
||
- **The journal runs under the gate, so a slow journal slows every writer.**
|
||
- The database journal does in-memory work plus one durable
|
||
`mutationLogStore.Append` per logged change. `Append` rewrites the shard
|
||
file (`CultNetShardMutationLogStorage.cs:83-89`).
|
||
- That is I/O under the gate, which the locking doctrine already allows for
|
||
stores ("A store doing I/O on behalf of its cache ... holds the gate",
|
||
composition doc `:98-101`). It is still a new cost, so Cut 4's
|
||
verification measures commit latency with a file log store.
|
||
- The whole-file rewrite is a pre-existing monolith. It is recorded as a
|
||
follow-up, not fixed here.
|
||
|
||
### 6. Cross-runtime impact
|
||
|
||
The Eyes survey was run at `65311c9`.
|
||
|
||
- **`cultcache-rs`, `cultcache-ts` and `cultcache-py` have no publication
|
||
surface:** no watch, no listener and no Sequence.
|
||
- Rust: `lib.rs:1970-1975, 2213-2420`.
|
||
- TypeScript: `cult-cache.ts:164-181`, "Observer delivery is not scheduled
|
||
here".
|
||
- Python: `cache.py:30-45`, which refuses reentrant mutation.
|
||
- **`cultnet-rs`, `cultnet-ts` and `cultnet-py` have no CultNetDatabase
|
||
equivalent and no cache watcher.** The only server-side shard log is the
|
||
Python interop test peer (`interop_peer.py:565, 583`), which mints
|
||
`len(log)+1` under its own lock, inline with the write. That is already the
|
||
journal's shape.
|
||
- **Wire parity is unaffected.**
|
||
- Cache `Sequence` never goes on the wire (composition doc `:47-49`).
|
||
- `CultNetShardLogEntryMessage` does not change.
|
||
- `CultNetDatabaseChangeRawMessage` (`CultNetSchemaMessages.cs:1016`) carries
|
||
no sequence, and this map adds none: stale protection for wire
|
||
subscriptions is server-side (Cut 6).
|
||
- No vector in `tests/vectors/` encodes publication order.
|
||
- **No other runtime must change.**
|
||
- **Follow-up.** `cultmesh-py`'s `CultMeshDatabase` watch (`node.py:421-505,
|
||
1059-1065`) has F2 (a throwing callback skips the rest and reaches the
|
||
writer), and it has no Sequence. When Python grows cache publication, it
|
||
takes this contract.
|
||
|
||
## Cuts
|
||
|
||
Every cut is on CultLib, branch `hands/cultcache-publication` from `main`
|
||
`65311c9`, in a worktree Self creates. Cuts 1-3 touch only the cache and land
|
||
first. Cuts 4-6 depend on Cut 3. Heavy verification runs on Yggdrasil.
|
||
|
||
**The negative checks that stay green in every cut** are the `4562340` tests:
|
||
|
||
- `ObserverWritingTheSameCacheDoesNotWaitOnOtherDeliveries`;
|
||
- `SubscriptionDemandHandlerWritingCacheDuringDeliveryDoesNotDeadlock`;
|
||
- `SequenceIncreasesInAdmissionOrder`;
|
||
- `WatchRecordIgnoresStaleSequence`.
|
||
|
||
They are the proof that no scheduler has come back.
|
||
|
||
### Cut 1. Project before mint (cache, behaviour)
|
||
|
||
- **First.** Commit Q1 as a failing test named for the rule.
|
||
- **Deletes first.**
|
||
- Accessor calls in `Index` (`:2798-2804`).
|
||
- Direct accessor reads in `RefuseVariantIndexSharing` (`:2659-2660`) and
|
||
`ResolveOne` (`:2616`).
|
||
- **Adds.**
|
||
- `Project` in `Admit` (`:2324-2325`).
|
||
- A per-key projection on `VariantPlan` (`:2456`).
|
||
- The early returns in `Resolve` (`:2477-2478`, `:2530-2534`) still pass
|
||
through `Project`, because it is called from `Admit`.
|
||
- **Authority.** Project owns "this admission's name and index values", and
|
||
the rules and `Index` read them. `Apply` runs no user code.
|
||
- **Verification.**
|
||
- Tests:
|
||
- **A refused getter mints nothing and changes nothing.** `GetWithSequence`
|
||
is unchanged, both keys are absent, the name lookup misses, nothing is
|
||
published, and the next write is previous+1.
|
||
- **Store-backed batch:** the store file is byte-identical, and a reopen
|
||
holds neither key.
|
||
- **A load whose getter throws** refuses the open and leaves the file
|
||
byte-identical.
|
||
- **Once per landed record:** a counting getter runs exactly once per
|
||
landed record, including a variant dependent re-resolved by a base edit.
|
||
- **The refusal names** the key and the member.
|
||
- Soul P4.
|
||
- Negative grep: `rg -n "NameAccessor\?\.Invoke|IndexAccessors" src/GameCult.Caching/CultCache.cs`
|
||
has no hit in `Apply`, `Index` or `Unindex`. `:315` is store-side and
|
||
legitimate.
|
||
- Stryker.NET on the diff.
|
||
- **Ledger.** About −15 and +45.
|
||
|
||
### Cut 2. Observer isolation (cache, behaviour)
|
||
|
||
- **Deletes first.**
|
||
- `Subject<Change> _changes` (`:1809`).
|
||
- `_changes.OnNext` and the false comment (`:2374-2375`).
|
||
- `_changes.Dispose()` (`:2154`).
|
||
- **Adds.**
|
||
- The observer array and `Observable.Create` in `Watch<T>` (`:1887-1897`).
|
||
- The per-observer catch in `Publish` (`:2369`), with escapes joining
|
||
`failures`.
|
||
- `OnCompleted` on `Dispose`.
|
||
- **Docs.** Composition doc `:77-82`: escaped observer exceptions follow the
|
||
`OnUpdate` policy.
|
||
- **Verification.**
|
||
- Tests:
|
||
- **Q2 with a rethrowing handler:** the second observer receives every
|
||
change. The writer gets the escaped exception after delivery, and the
|
||
thrower is still subscribed (per Q-P2).
|
||
- **Subscription order:** subscribing and unsubscribing during a delivery
|
||
leaves every other observer's stream complete.
|
||
- **Dispose** completes observers.
|
||
- Soul P3.
|
||
- The four `4562340` tests stay green.
|
||
- Stryker on the diff, especially removing the catch and breaking out of
|
||
the loop on the first failure.
|
||
- **Ledger.** About −5 and +30.
|
||
|
||
### Cut 3. The journal (cache, behaviour)
|
||
|
||
- **Adds.**
|
||
- `AddJournal` (Q-P3), with registration under the gate.
|
||
- The invocation in `Admit` after `Apply` (`:2328`).
|
||
- The no-reentry guard in `Held` (`:2341`).
|
||
- Exceptions routed like observer failures.
|
||
- **Docs.** A composition-doc paragraph in "Store composition" and in
|
||
"Locking": journals run under the gate, in Sequence order, and must not
|
||
enter the cache or wait for anything that needs it.
|
||
- **Verification.**
|
||
- Tests:
|
||
- **Order:** 8 threads × 1000 writes. The journal sees contiguous,
|
||
strictly increasing Sequences, and each call's changes arrive together.
|
||
- **Timing:** the journal runs before any observer of the same change, and
|
||
after `Apply` (`Get` inside the journal sees the new document).
|
||
- **Reentry:** a journal that writes the cache throws, and the admission
|
||
stands.
|
||
- **Refusal:** a refused admission journals nothing.
|
||
- **Dependents:** a variant dependent is journaled in the same call as its
|
||
base edit.
|
||
- **Failure:** a throwing journal still lets the change publish, and the
|
||
writer gets the exception.
|
||
- The `4562340` tests stay green.
|
||
- Stryker on the diff.
|
||
- **Ledger.** About +50.
|
||
|
||
### Cut 4. Salvage the branch and rebuild CultNetDatabase on the journal (networking)
|
||
|
||
- **4a (history).** `git merge origin/hands/cultnet-publish-commits` into the cut branch from main
|
||
`3bf1c0c`.
|
||
- It is clean: the probe merge `6ec2f73` auto-merged `NetworkingTests.cs` and the composition doc, then
|
||
built and ran.
|
||
- The merge brings the tests. The queue dies in 4b.
|
||
- Self may squash-salvage instead, with the same content.
|
||
- **4b (subtraction first).** Delete at branch `28248eb` `CultNetDatabase.cs`:
|
||
- `_parked`, `_ready`, `_nextSequence` and `_draining` (`:502-505`);
|
||
- `_predictionGate` (`:506-507`);
|
||
- the Watch subscription and handshake (`:529-531`);
|
||
- `OnCacheChange` (`:1080-1135`);
|
||
- `PendingChange` (`:1265-1275`);
|
||
- `HasPrediction` (`:1209-1213`) and its calls (`:916`, `:1574`);
|
||
- the logging and prediction bookkeeping in `Deliver` (`:1137-1201`).
|
||
|
||
Main's `PublishCacheUpdate` / `OnUpdate +=` (`:505`, `:1157`, `:1161-1191`) is already gone on the branch.
|
||
Hands confirms it is absent after the merge.
|
||
- **4c (the journal).** Register the Watch observer first and the journal second (`AddJournal`), in the
|
||
constructor. Per admission, under the cache gate, per change in Sequence order:
|
||
1. `TakeDoor(change)`, keeping the branch's shape rule (`:1219-1234`).
|
||
2. Resolve the shard and kind. Predictions: a `Predicts` door adds its key. An `Authoritative` door
|
||
(`ApplyPutAsync`, replica put) holding a prediction for the key makes the kind `Reconciled` and removes
|
||
the key.
|
||
3. Log, per §(a):
|
||
- a replica door records the primary's entry, disk first;
|
||
- a primary change that is not `NoLog` mints, builds, appends and records, with failure going to the
|
||
floor;
|
||
- anything else logs nothing.
|
||
4. Stash `(cache Sequence → CultNetDatabaseChange)` under `_logGate`.
|
||
5. If step 3 failed, throw once at the end of the call, after all changes are handled (Q-P5).
|
||
- **4c, publication.** The Watch observer removes the stash entry for `change.Sequence` and publishes it. A
|
||
missing stash means the change was admitted before this database existed, so it is ignored. `Dispose`
|
||
removes the journal and the observer and clears the stash.
|
||
- **4d (the §(a) fixes).**
|
||
- The floor: `GetCompactedMutationLogSequence`, `GetLatestMutationLogSequence`, and resume in
|
||
`InitializeLogSequencesFromStore`.
|
||
- Disk-first `RecordMutationLogEntry`.
|
||
- A `SortedList` per shard.
|
||
- The snapshot reads its sequence first (`:660` moves below `:668`'s read).
|
||
- `_logGate` around `_lastWriteSequence`/`_nextLogSequences` readers (`:1302`, `:1314`, `:1325`).
|
||
- **Door changes.**
|
||
- Flags: `Predicts` and `Authoritative`.
|
||
- The snapshot removal door (branch `:733`) sets `Removal = true`.
|
||
- The replica absent-key delete logs directly and publishes nothing (branch `:1612-1621`).
|
||
- Doors neither log nor publish. They register context, write the cache, and dispose the registration.
|
||
- **Verification.**
|
||
- Every salvaged test passes: `CultNetPublishesCommitsTests` (13) and `CultNetPublicationOrderTests` (S1,
|
||
S1b, S2, S3, S4, S6, S7, log-before-publish, in-place re-upsert).
|
||
- Soul pass 3's P1-P7 are committed and passing.
|
||
- The pass 4 probes become tests named for the rule:
|
||
- **A1:** a refused primary `Append` publishes the change. The writer gets `CultNetShardLogException`,
|
||
which names the shard and `S`. A replica pull answers `compacted_history`, and a snapshot resync
|
||
converges.
|
||
- **A2:** the same at the tail with a restart: the replica converges after resync, and no sequence is
|
||
reused. The store's `CompactThrough` succeeded.
|
||
- **A3:** a refused replica `Append` leaves the cursor unchanged. The retry converges, and the memory log
|
||
holds `[1,2]`.
|
||
- **B1:** a pull on a primary is logged, and the replica converges.
|
||
- **S:** a snapshot racing a write converges.
|
||
- **C:** after compacting through 2 and restarting, the next mint is 3.
|
||
- New tests:
|
||
- a reentrant `database.PutAsync` from inside a database handler is logged with its context and published
|
||
once;
|
||
- a reentrant `ApplyShardLogResponseAsync` from inside a handler keeps the replica log in primary order;
|
||
- a log store whose `Append` writes the cache fails with Cut 3's reentry guard, wrapped, and the admission
|
||
stands.
|
||
- Commit latency with `CultNetFileShardMutationLogStore`, main against Cut 4. Also measure a `CultMesh` node
|
||
with `EnableDurableShardLogs`, where every bookkeeping write is now logged. Report it; do not gate on it.
|
||
- Negative grep: `rg -n "_parked|_ready|_draining|_nextSequence|_predictionGate|HasPrediction|PublishCacheUpdate" src/GameCult.Networking`
|
||
returns nothing.
|
||
- Review check: no `_cache.` call inside a `lock (_logGate)` block.
|
||
- The `4562340` tests stay green.
|
||
- Test runs use `--blame-hang-timeout 2m`.
|
||
- Stryker on the diff: dropping the floor raise, memory-before-disk, and the snapshot read order must each
|
||
kill a test.
|
||
- **Docs.**
|
||
- Composition doc `:50-53`: replace the "streams derive from `OnUpdate`" sentence. CultNet derives from
|
||
`Watch` plus one journal, and the log is written in cache order.
|
||
- A new paragraph: a primary that cannot log a change compacts through it.
|
||
- The log-store rule under Locking.
|
||
- **Blockers.** None. Q-P5 is ruled A.
|
||
- **Ledger.** Against the branch, about −130 and +95. The extra over the old estimate is the floor, the
|
||
disk-first order, the `SortedList`, the snapshot order, and the exception type.
|
||
|
||
### Cut 5. Database fan-out isolation and Sequence (networking, behaviour)
|
||
|
||
- **Deletes first.** The database's `Subject<object> _changes` (main `:485`, branch `:487`) and its
|
||
`Dispose` (`:1158`).
|
||
- **Adds.**
|
||
- The isolated observer array, the same pattern as the cache's `Register`/`Publish`
|
||
(`CultCache.cs:1929-1944`, `:2470-2503`), behind `Watch<T>`, `WatchAllChanges` and the rest (`:1089-1145`).
|
||
- `CultNetDatabaseChange<T>.Sequence` (`:319-371`), as an optional constructor argument. The journal's stash
|
||
sets it from the cache change. A change with no cache change behind it carries 0.
|
||
- **Docs.** The binary-signature note, as the composition doc gave for `CultCacheDocumentChange` (`:57-58`).
|
||
- **Verification.**
|
||
- With a rethrowing handler and one throwing database subscriber, the other subscriber and a live
|
||
subscription-server peer receive every change. The writer gets the exception after delivery (Q-P2).
|
||
- The disposed-database test.
|
||
- `Sequence` equals the cache change's.
|
||
- **Blockers.** Cut 4.
|
||
- **Ledger.** About −10 and +35.
|
||
|
||
### Cut 6. Stale protection on wire projections (networking, behaviour; Q-P4 A)
|
||
|
||
- **The rule.** A projection applies a change to a key only if its `Sequence` is above what the projection
|
||
last applied for that key. A snapshot or reconcile applies everything at or below the cache sequence it read
|
||
**before** reading documents.
|
||
- That read is `Cache.GetWithSequence(<any key>).Sequence`, the branch's idiom (branch `:531`). No new API
|
||
is needed.
|
||
- **Subscription server** (`CultNetDatabaseSubscriptionServer.cs`):
|
||
- `SubscriptionProjectionState` (`:630`) gains `long Floor` and `Dictionary<string, long> Applied`.
|
||
- The single-row fast path in `Watch` (`:296-340`) reads the change's `Sequence` (reflection, like `Key`).
|
||
Under `_lifecycleGate` it drops the change if `Sequence <= max(Floor, Applied[key])`. Otherwise it applies
|
||
the change and records it. Removals are included.
|
||
- `Reconcile` (`:347`) and the subscribe snapshot (`:168-181`) read `seq0` before `CreateProjectedSnapshot`
|
||
(`:402`), set `Floor = seq0`, and clear `Applied`.
|
||
- This is sound: every change applied earlier was applied under the same gate, after its admission, so
|
||
its `Sequence` is at most `seq0`.
|
||
- Hop and unauthorized paths already reconcile from current state, and they set the floor too.
|
||
- **`CultNetDatabaseServer.CreateSubscription`** (`:501-513`):
|
||
- A per-subscription `(Floor, Applied)` under a per-subscription lock, held across check, record and send.
|
||
- `HandleSubscribeAsync`'s snapshot (`:362-369`) takes `seq0` first and sets the floor.
|
||
- **Verification.**
|
||
- Two threads write one key, with a blocking cache observer that forces the inverted delivery (Q3 shape).
|
||
On both servers, the peer's last state for the key equals the cache's.
|
||
- A removal overtaken by an older upsert leaves the key absent at the peer.
|
||
- The negative check: without the drop, both tests fail.
|
||
- **Blockers.** Cut 5, which provides `Sequence` on database changes.
|
||
- **Ledger.** About +45.
|
||
|
||
### Salvage from `hands/cultnet-publish-commits` (revised)
|
||
|
||
- **Keep:**
|
||
- both test files;
|
||
- the `NetworkingTests.cs` edits;
|
||
- the door table and the shape rule;
|
||
- "replicas never mint";
|
||
- log-before-publish;
|
||
- "the snapshot logs nothing";
|
||
- "loads and pulls on a primary are logged" (B1);
|
||
- the composition-doc paragraph, rewritten to name the journal.
|
||
- **Drop:**
|
||
- the reorder buffer and the drainer;
|
||
- `_predictionGate`;
|
||
- logging from the observer;
|
||
- the swallowed exception path (A1 and A3 on the branch).
|
||
|
||
## Subtraction ledger (estimate)
|
||
|
||
| Cut | Removed | Added | Other |
|
||
|---|---|---|---|
|
||
| 1 | ~15 | ~45 | none |
|
||
| 2 | ~5 | ~30 | the cache no longer uses R3 `Subject` |
|
||
| 3 | 0 | ~50 | one public method (Q-P3) |
|
||
| 4 | ~130 vs branch | ~95 | the merge brings +1109/-243, of which about 780 are tests; the floor, disk-first order, `SortedList`, snapshot order and `CultNetShardLogException` are the extra over the first estimate |
|
||
| 5 | ~10 | ~35 | the database `Subject` goes, and one public property |
|
||
| 6 | 0 | ~45 | none |
|
||
|
||
- **Source, net against main:** about +245.
|
||
- **Source against the branch:** no scheduler anywhere.
|
||
- The budget is pressure, not a metric.
|
||
|
||
## Operator questions
|
||
|
||
- Q-P1..Q-P4 are ruled (see Rulings). Their option analysis is in the map's git history.
|
||
|
||
- **Q-P5 A (ruled 2026-09-29): a primary that cannot log a committed change throws a typed
|
||
`CultNetShardLogException` naming the shard and the burned sequence, after publication, not derived from
|
||
`InvalidOperationException`.**
|
||
- The commit stands, the change is published, and replicas behind it snapshot-resync (Imagination pass 4 (a)).
|
||
- The writer hears of it. AetheriaEve `Program.cs:3722` cannot mistake it for a denial.
|
||
|
||
Recorded, not forked:
|
||
- (b): no origin on the journal. Loads and pulls are logged on primaries.
|
||
- (a)'s restart residual when the store refuses both writes.
|
||
- The replica keeps retry-by-cursor.
|
||
- A throwing getter refuses loads too.
|
||
- Reentrant writes keep main's depth-first delivery. Order is data.
|
||
- Journal I/O under the gate follows the store precedent.
|
||
|
||
## Follow-ups outside this campaign
|
||
|
||
- **`cultmesh-py` watch.** It has no isolation and no Sequence
|
||
(`node.py:1059-1065`).
|
||
- **Durable log whole-file rewrite.** `CultNetShardMutationLogStorage.cs:83-89`
|
||
rewrites the whole file per `Append`. It is a monolith, and under the gate
|
||
its cost is now every writer's.
|
||
- **Mesh database-backed handles.** `CultMesh.cs:2362` and `:3396` can drop
|
||
stale changes once `CultNetDatabaseChange.Sequence` exists.
|
||
- **Skill scar.** The Eureka SKILL.md already says "a ticket system that
|
||
serialized delivery was built, deadlocked, and was deleted". The Soul pass 3
|
||
direction re-proposed it anyway. The Soul brief template should tell Soul to
|
||
check `git log -S` for a proposed mechanism before recommending it.
|
||
- **A chained replica after a snapshot.** A replica that snapshot-resyncs has no log entries below the snapshot
|
||
sequence. It should raise its own floor to that sequence, so a replica chained on it resyncs instead of seeing
|
||
a gap (`ApplyShardSnapshotResponseAsync` `:675-736`). It predates this campaign.
|
||
- **AetheriaEve `Program.cs:3722`.** It reads a committed write's `InvalidOperationException` as a denial. Under
|
||
Q-P5 A, Cut 4's log failures cannot reach it. Observer and `OnUpdate` exceptions still can, so the Cuts 1-2
|
||
recording stands.- **Cut 4 Soul, recorded 2026-09-30 (Self), not fixed in Cut 4:**
|
||
- **File log store is not crash-safe (hard prerequisite).** `CultNetFileShardMutationLogStore.Append` and
|
||
`CompactThrough` use `File.WriteAllBytes` (truncate, rewrite, no temp+rename, no fsync). A failure mid-write
|
||
truncates the log; every later append burns a sequence and a restart cannot construct the database. It also
|
||
rewrites the whole file per append, which Cut 4 now does under the cache gate (2.2 ms per write at 100
|
||
entries, 33 ms at 950, a 2.49 s stall). An atomic, append-only store must land before any live daemon
|
||
enables durable shard logs. Today only AetheriaEve's offline tools do; the live Aetheria daemon passes
|
||
`enableDurableShardLogs: false`.
|
||
- **A slow log append blocks cache readers** (`CultCache.Get` takes the gate). Accepted with a note in the
|
||
store's XML doc.
|
||
- **Crash between the store write and the log append** leaves the log lagging the store, and nothing detects
|
||
it; a restarted primary reuses that sequence (A2's divergence reached by a plain kill). The spec called this
|
||
lag detectable; for a crash it is not.
|
||
- **Cut 4 final Soul (4d), recorded 2026-09-30 (Self), not fixed in Cut 4:**
|
||
- **F2, read cost.** `CultNetDatabase.Serves` (`:1980-1989`) runs per row in `GetAll`/`GetByName`/`GetByIndex` for every
|
||
layout except a single owns-everything shard: a reflection `GetTrackedKey`, a gate lock and a shard re-scan per row.
|
||
`database.GetAll` measured 24.8x `cache.GetAll` on 20,000 rows under a key-prefixed shard. Fix: filter by key inside
|
||
the cache enumeration (the `rowFilter` path already has keys) and hoist the owns-everything check.
|
||
- **F4, dispose drain edges.** An observer that removed its stash entry before the drain can publish after `Dispose`
|
||
returns (dropped); a throwing `_cacheJournal.Dispose()` skips the drain. Neither double-publishes or deadlocks.
|
||
- **F5, refusal naming.** A write whose schema is owned but whose key is outside the shard's prefix also answers
|
||
`unowned_schema` with details `{field:"schemaId"}`; the peer cannot tell a key refusal from a schema refusal.
|
||
- **Pre-existing, separate task.** Python emits error code `unsupported_schema_version`
|
||
(`cultnet-py/.../interop_peer.py:415`, `cultmesh-py/.../server.py:161`), which is not in the error-code enum; TS and
|
||
Rust refuse the whole message.
|