Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8e1de662c7 | ||
|
|
f901b38fb7 | ||
|
|
665dc0856c | ||
|
|
2fa1bffc38 | ||
|
|
1262fcdcb5 | ||
|
|
3854bb5eaf | ||
|
|
07501c7c6e | ||
|
|
95522dcd0e | ||
|
|
47850dfcd6 | ||
|
|
59ef4da9dd | ||
|
|
57cf8df181 | ||
|
|
80af7ef041 | ||
|
|
e7fbbbcade | ||
|
|
cdc57bda2d | ||
|
|
1d9bc0246d | ||
|
|
77373cb563 | ||
|
|
8c84857670 | ||
|
|
ae8fa5571f |
2 changed files with 1196 additions and 1 deletions
570
docs/store-recovery-cut.md
Normal file
570
docs/store-recovery-cut.md
Normal file
|
|
@ -0,0 +1,570 @@
|
|||
# Odin store repair and recovery: cut map
|
||||
|
||||
**Operator rulings, 2026-09-30:**
|
||||
- **Q-R1:** "You got it". A foreign file under a running Odin is set aside and Odin continues.
|
||||
- **Q-R2:** "Yep". Keep the previous generation on disk.
|
||||
- **Q-R3:** "Start empty". Nothing decodable at activation means Odin starts empty.
|
||||
- **Q-R4:** "Yep, and Idunn needs better reporting capabilities so it can serve as watchdog. I ought to get a message
|
||||
about this on Discord if it ever happens."
|
||||
- Stop writing after the third set-aside in 10 minutes, and keep serving.
|
||||
- **New requirement:** Idunn watchdog reporting that reaches the operator on Discord. It is mapped as its own
|
||||
campaign and is not an Odin cut.
|
||||
- **Q-R5:** "That's fine, Odin still runs after all, right?" Yes. Odin keeps serving, and the store condition is
|
||||
published as its own typed record, not as a degraded presence.
|
||||
- **Q-R6:** "Approved". Both drills.
|
||||
|
||||
**R1 LANDED: CultLib main `daedfdd6` (2026-09-30).** The merged tree `0e6891b3` was verified on Yggdrasil in every
|
||||
runtime, including the four-runtime store interop test. That test runs close to its 30 s per-command limit under
|
||||
load (`cult-cache.test.ts:1407`), so it is fragile: follow-up. Consumer pins still gate their own bumps: Bifrost only
|
||||
with R3b's TS fsync; Muninn and Idunn with their error-text matches.
|
||||
|
||||
**R1 status (Self, 2026-09-30).** Built on CultLib `hands/cultcache-read-failures` (`c0b56e06`..`b7908668`), then
|
||||
fix batch 2 (`..2d7a3c78`). Soul's verdict on batch 2 was to hold the merge. Batch 3 is in Hands, fixing:
|
||||
- **Blocker, F1 (zero-byte stores on Yggdrasil).**
|
||||
- Two zero-byte stores are live: `/srv/bifrost/persona-feedback/{inbox/observations,outbox/deliveries}.cc`.
|
||||
- They are pre-created by `touch` in gamecult-ops `install-bifrost-persona-feedback-native-yggdrasil.sh:42,46`
|
||||
and `bootstrap-bifrost-persona-feedback-yggdrasil.sh:22,29`.
|
||||
- Bifrost `persona-feedback.mjs serve` and Epiphany `epiphany-swarm` crash-loop on them once their CultLib pins
|
||||
move past R1.
|
||||
- **Neither pin may move until the scripts stop pre-creating stores and Self has moved the live files aside.**
|
||||
- **F4.** Rust errors echo record values through serde's text (`lib.rs:753-758`). The header is echoed without bound.
|
||||
- **F6.** A dangling symlink reads as absent, and the writer then replaces it.
|
||||
- **F7.** The same NotFound-only rule in Python's `JsonLinesBackingStore` and in TS `pullAll`'s repair write.
|
||||
- **M7.** A header-prefix fixture.
|
||||
Recorded for later cuts:
|
||||
- **F3 → R2/R3b.** TS, Python and C# decode the whole buffer before reading the header. A truncated newer-format
|
||||
store (20..n-1 bytes) is "unsupported format" in Rust and a raw decode error elsewhere.
|
||||
- **F5 → R3.** C# `ReadSnapshot` reads ENOTDIR/EISDIR as an empty store (`FileInfo.Exists`, `CultCache.cs:3439`).
|
||||
C# `DirectoryMessagePackBackingStore` `File.Exists` (`:409`, `:497`) is the same. This is a live disagreement between
|
||||
runtimes until R3.
|
||||
- **F8 → R3b.** Python leaves its staging file behind when a write fails (`stores.py:124-128`).
|
||||
- **Tooling.**
|
||||
- `packages/cultcache-rs` has no committed `Cargo.lock`, so the map's `cargo test --locked` cannot run.
|
||||
- The cultcache-ts interop test needs `CARGO_TARGET_DIR` unset under ygg-verify.
|
||||
- **Soul on batch 3 (`2d7a3c78..3c9e195b`, gamecult-ops `bca7978`), 2026-09-30: merge both.** Pre-merge batch 4 is in
|
||||
Hands: fix the README symlink claim (S1), close value echoes in the redb and typed-decode paths (S2) and in the
|
||||
Python/TS non-string key echo (S3), make the unlock policy coherent (S9), classify encode refusals separately from
|
||||
`NotReplaced` (S10), and fix a test timeout (S11). The ops branch deletes the guard test (S6) and the dead compose
|
||||
layout (S5).
|
||||
- **Open for R3:** writes through a symlinked store path replace the link, so a store behind a link to a mounted
|
||||
volume moves into the container layer. R3 decides resolve or refuse.
|
||||
- **Open for R3b:** Python and TS never fsync. After F1 their first create-by-rename is not covered by ext4
|
||||
`auto_da_alloc`, so a crash can leave zero bytes, which every post-R1 reader refuses forever. **Bifrost's CultLib
|
||||
pin moves past R1 only together with R3b's TS fsync.**
|
||||
- **For W0 (Odin):** classify a write failure through `CultCacheStoreWriteFailed.kind` together with its `io::Error`,
|
||||
never retry on `NotReplaced` alone (S10).
|
||||
- Deployed Epiphany is `d2ca6630`, using a vendored cultcache-rs (`ba6a4874`) that reads a missing store as empty.
|
||||
All Epiphany units have been inactive since 2026-08-22.
|
||||
- **Done 2026-09-30 20:48 CEST (Self).** gamecult-ops merged at `3fa22aa`. The inbox is setgid 2750. All three
|
||||
zero-byte stores were renamed in place and moved to `/root/zero-byte-stores-20260930/` on Yggdrasil, with
|
||||
`manifest.txt` recording owner, mode and mtime. None had gained bytes. All three Bifrost units stayed active, and
|
||||
persona-feedback logged nothing and did not restart.
|
||||
- The compose/container lane (the compose file, two deploy scripts, the `.ps1` uploader, two runbooks) is dead:
|
||||
the live Bifrost units run native `dotnet` and `node`. **Deleted 2026-09-30, operator: "delete the Bifrost lane".**
|
||||
gamecult-ops `fdb4356`. Its live trigger, the Idunn legacy actuator route `deploy bifrost-persona-feedback`, was
|
||||
retired in Idunn `7793b5b` and installed on Yggdrasil at 21:50 CEST (sha256 `d95fa08c...`; the previous binary is
|
||||
at `/root/idunn-yggdrasil.pre-bifrost-retire-20260930`). The manifest and `/srv/bifrost/compose` were moved to
|
||||
`/srv/retired/retired-2026-09-30/`. All three Bifrost units stayed active. **Follow-up:** persona-feedback has no
|
||||
Idunn deploy route now. Give it a real Idunn binding.
|
||||
- Live zero-byte stores to move aside after the ops merge, in Soul's order (setgid on inbox, then rename in place,
|
||||
then relocate): `observations.cc`, `deliveries.cc`, and a third, `/var/lib/gamecult/epiphany/idunn-provider-health-admission.cc`.
|
||||
- **Text matches.** gamecult-ops `scripts/idunn/start-idunn-local.ps1:143` matches the same removed text as Muninn
|
||||
(below). It must change with Idunn's next CultLib bump.
|
||||
- **Consumer follow-up that must travel with the next Muninn CultLib bump:**
|
||||
`Muninn scripts/restart-starfire-muninn.ps1:144` matches `*failed to decode MessagePack*` to trigger
|
||||
`Reset-CorruptMuninnStore`. R1 removes that text, so the reset would stop firing silently. The script must match
|
||||
the typed `CultCacheStoreUnreadable` kind, or the new wording, in the same change.
|
||||
- C#'s truncated-store exception (`EndOfStreamException`) is an `IOException`, the same I/O-versus-undecodable
|
||||
overlap. It goes with R3's C# work.
|
||||
|
||||
Status: cut map, Imagination pass 1 (Opus), 2026-09-30. Nothing has landed. This map owns the means for repairing
|
||||
and recovering `/var/lib/gamecult/odin/topology.cc`. It follows `docs/write-pattern-cut.md` (Cuts 1-3) and starts
|
||||
from that campaign's landed head.
|
||||
|
||||
Anchors:
|
||||
- Odin `hands/odin-write` at `cda2087` (Cuts 1-2 plus two test commits). Q3 B (peer puts fsynced before ACK) is
|
||||
being finished on that branch now. **Re-anchor every Odin `file:line` below on the head that lands; the line
|
||||
numbers here are `cda2087`.**
|
||||
- CultLib `6d0cc963`. `packages/cultcache-rs` is byte-identical to Odin's pin `3bf1c0ce`, so the Rust line numbers
|
||||
hold at both.
|
||||
- Idunn `a96ad9d`, Heimdall `e8e2832`, gamecult-ops `runbooks/gamecult-state-backup.md` as of 2026-09-30.
|
||||
|
||||
Operator, 2026-09-30, on the three options Self offered for an undecodable foreign write (degraded but loud, end
|
||||
Odin, unchanged): **"None of these options involve repair and recovery, which is what ultimately needs to happen."**
|
||||
|
||||
Open: Q-R1 to Q-R6 (section 8). D1 (section 6, O1) is a Self default the operator may overrule.
|
||||
|
||||
## Short answer
|
||||
|
||||
Odin already holds the one thing a repair needs: its in-memory working set, which is authoritative for everything
|
||||
it owns while it holds the write lease. Today it never uses it for repair. It ends on a decodable foreign write, and
|
||||
it silently stops persisting on an undecodable one. At activation it has no working set and no second copy, so it
|
||||
crash-loops.
|
||||
|
||||
The repair has one rule: **a store file that is not what Odin last wrote is set aside, never deleted and never
|
||||
merged, and Odin writes its working set over it under the lease.** At activation Odin has no working set, so it
|
||||
falls back to a previous generation kept on disk, and then to an empty store that providers refill. The
|
||||
correlation sequence gets a time floor, so no rollback can recreate the watermark deadlock. Recovery is published as
|
||||
a typed record in Odin's own catalog. The mechanism (typed read failures, a set-aside exchange, a retained previous
|
||||
generation, no staging file left behind by a failed write) belongs to CultLib, because every consumer of the
|
||||
single-file store has the same gap.
|
||||
|
||||
## 1. Body facts (probed)
|
||||
|
||||
### 1.1 Probe: cultcache-rs single-file store under fault
|
||||
|
||||
The probe was a scratch crate at CultLib `3bf1c0ce`, `scratchpad/odin-recovery-probe` (Imagination session), commit
|
||||
`b19eba9`, run through the stopgap on Yggdrasil in slot 2. It ran with `--tmpfs /small:size=64k --tmpfs /ro:ro`,
|
||||
using `trap '' XFSZ; ulimit -f 40` for the file-size case. The log is `scratchpad/odin-recovery-probe.log`.
|
||||
|
||||
| Fault | Result |
|
||||
|---|---|
|
||||
| P1: file truncated to half | `compare_exchange_snapshot` and `pull_all` return `Err`, **not** `Ok(false)`. There is no `io::Error` in the chain; the decode error is flattened to text (`lib.rs:720-725`). The message is wrong: "invalid type: string \"cultcache.store.v1\", expected struct CultCacheEnvelope". `store_header` (`lib.rs:317-341`) deserializes the *whole* array to find the header, so a truncated v1 file has no header and falls through to the legacy decoder. The file is unchanged by the failed exchange |
|
||||
| P2: 12 bytes of garbage appended | `pull_all` returns `Ok`, all 3 records. `compare_exchange_snapshot(expected)` returns **`Ok(true)` and silently drops the tail**. `rmp_serde::from_slice` ignores trailing bytes. C#'s `DeserializeSnapshot` (`CultDocumentMessagePackSerialization.cs:179-215`) never checks for the end either |
|
||||
| P2b: one byte flipped inside a payload | `pull_all` returns `Ok` (payloads are opaque bytes). `compare_exchange_snapshot` returns **`Ok(false)`**. To Odin, bit rot is a *decodable foreign write* |
|
||||
| P3: header `cultcache.store.v4...` | `Err` with a distinct message, "not readable; this runtime reads v1 only". It is distinguishable only by its text |
|
||||
| P4: another handle wrote a valid snapshot | `Ok(false)` |
|
||||
| P5: `hard_link(topology.cc, topology.cc.displaced.<ts>.x)`, then an exchange through the store | The link keeps the old bytes exactly, and the store writes the new file. The staging sweep (`remove_abandoned_staging_files`, `lib.rs:2730-2766`) removes a `<name>.<uuid>.tmp` beside it and leaves the displaced name alone |
|
||||
| read-only filesystem, no store | `pull_all` returns `Ok(0)`. The exchange fails opening the lock: `io::Error` `ReadOnlyFilesystem` (os 30) |
|
||||
| `RLIMIT_FSIZE` 40 KiB, 200 KB write | `Err` `io::Error` `FileTooLarge` (os 27). The old file is intact and still decodes. **A 40,960 B partial staging file is left behind.** The next attempt sweeps it and leaves a new one |
|
||||
| 64 KiB tmpfs, 200 KB write (ENOSPC) | `Err` `StorageFull` (os 28). The old file is intact. **A 61,440 B partial staging file is left behind: all of the filesystem's remaining space.** The same holds on every retry |
|
||||
|
||||
What follows from the probe:
|
||||
- **Decodable is not the same as legitimate.** A flipped payload byte is "decodable and foreign", and appended
|
||||
garbage is "decodable and ours". The decodable/undecodable split cannot be what decides recovery. One rule has to
|
||||
cover both: *not what Odin last wrote*.
|
||||
- **A write that fails for lack of space holds the space it failed on.** `/var/lib/gamecult` is shared by every
|
||||
service on Yggdrasil. Odin retries once per second (and, under Q3 B, on every put), so it re-takes whatever
|
||||
another service frees. C# deletes its temp file in a `finally` (`CultCache.cs:3496-3531`); Rust does not
|
||||
(`write_all_unlocked`, `lib.rs:741-768`). This is a Rust parity defect.
|
||||
- **The atomic write keeps no previous generation.** `write_all_unlocked` stages, fsyncs, renames over the file and
|
||||
fsyncs the directory. After the rename, the old inode is gone. The brief's "last good checkpoint kept by the
|
||||
atomic write" does not exist on disk. The only last-good copy is Odin's in-memory `flushed` snapshot
|
||||
(`lib.rs:506-510`), and it dies with the process.
|
||||
|
||||
### 1.2 Odin today (`cda2087`)
|
||||
|
||||
| Case | Path | Outcome |
|
||||
|---|---|---|
|
||||
| Decodable foreign write while serving | `MemoryOdinTopologyStore::flush` `lib.rs:613-626`: `compare_exchange_snapshot(flushed, current)` returns `false`, so `ForeignStoreWrite`. `survive` `main.rs:879-895` returns it, and Odin ends | Idunn restarts Odin (`control_plane.rs:68-71`: 6 restarts per hour, the wait doubling from 5 s). The new Odin **loads the foreign file as truth**. Odin's own unflushed changes, and every record of Odin's that the foreign file lacks, are lost without a trace. A flipped payload byte (P2b) takes this path |
|
||||
| Undecodable foreign write while serving | the same exchange returns `Err` (decode). `survive` logs it and retries every second, forever. Nothing is persisted again | **Idunn's crossing breaks too.** `receive` (`Idunn drivers.rs:5005-5030`) reads the same file and propagates the error to `admit_latest_topology` (`control_plane.rs:7903-7907`) and `refresh_admitted_topology` (`:6040`). Every Odin-correlated deployment and refresh fails until the file is readable, which is never |
|
||||
| Corrupt file at activation | `try_activate` `main.rs:246-297` calls `MemoryOdinTopologyStore::load` `lib.rs:519-538`, which calls `pull_all()?`. `main.rs:762` `while !try_activate()?` exits the process | Crash loop until Idunn's budget is spent. Then Idunn reports once, to its log only (`control_plane.rs:5709-5723`, `report_once` `:4599-4610` is `eprintln`), and stops. The Verse rendezvous is down |
|
||||
| Disk full or read-only | the exchange returns `Err` (io). `survive` logs and retries every second | Odin serves from memory. Each attempt leaves a partial staging file (1.1). A stop or deploy in this state loses everything unflushed. Nothing typed says so |
|
||||
|
||||
Detection latency for a foreign write: `flush` writes only when dirty (`main.rs:307-314`). Odin's own heartbeat
|
||||
(`HEARTBEAT_INTERVAL` 5 s, `main.rs:69`) dirties the set, so a foreign write is met at the latest one heartbeat plus
|
||||
one flush interval (about 6 s) after it happens. No polling read is needed, and Cut 1's rule (no reads while
|
||||
serving) stands.
|
||||
|
||||
### 1.3 Other sources of truth, and what they can recover
|
||||
|
||||
| Source | Holds | Freshness | Recovers |
|
||||
|---|---|---|---|
|
||||
| Odin's working set (W) | everything, including unflushed changes | live | everything, but only while this process lives and holds the lease |
|
||||
| `flushed` (in memory) | the last snapshot Odin wrote | at most one interval old | nothing more than W; it is the comparison base, not a store |
|
||||
| previous generation on disk (new, O3 and R3) | the snapshot before the last one this store's owner verified and replaced | one write old | everything except the last write's delta, including publish-once documents, presence history and the watermark |
|
||||
| nightly backup (`gamecult-state-backup.md`) | `/var/lib/gamecult/odin` whole, shared-lock copy | up to 24 h old, 3 days on Yggdrasil, 14 days plus 8 weekly on Raven | publish-once documents. **Never automatically**: "It never becomes live authority, and nothing restores it automatically"; "restore a store only after stopping its owner" |
|
||||
| providers republishing | presences (2.7-12 s), Muninn documents (2 s), ads (about 30 s); Heimdall HEAD republishes `heimdall.command_boundary` and its three siblings every 60 s (`Heimdall src/index.ts:19,35-39`, gated on its write lease) | seconds to a minute | every document whose publisher republishes |
|
||||
| Idunn's projection plus presences | correlations | one refresh pass (250 ms) after the presences return | correlations. They are a projection and never need restoring |
|
||||
| Idunn's `odin_publisher_sequence_cursor` (`control_plane.rs:1456,1537,6081`) | the highest correlation sequence Idunn admitted, per target | live | nothing directly (Idunn owns it), but it is why the watermark must never roll back |
|
||||
|
||||
Premise correction: `heimdall.command_boundary` is not publish-once at Heimdall HEAD. A `stored_at` of 09-23
|
||||
proves only that the publisher's own `stored_at` has not changed, because Odin stores the publisher's
|
||||
`stored_at` (`persist_generic_document`, `main.rs:1330-1347`). Whether the deployed Heimdall re-puts is
|
||||
unverified; O3's outstanding-republish list measures it.
|
||||
|
||||
**The watermark is the one record whose loss is not self-healing.** Idunn refuses any correlation sequence at or
|
||||
below its cursor (`control_plane.rs:1763,1785`). An Odin that starts from an empty store, from the previous
|
||||
generation, or from a backup restarts a target's count below that cursor. That is the deadlock documented at
|
||||
`lib.rs:33-49`: Idunn refuses every correlation until the count climbs past the old mark, and the refusal
|
||||
withdraws the correlation before it can climb. Every recovery source except W has this defect. O1 removes it.
|
||||
|
||||
Other facts the design leans on:
|
||||
- **Idunn fences before it grants.** `CultCacheWriteLeaseDriver` (`Idunn drivers.rs:4050-4205`): "revocation
|
||||
follows an exact incumbent stop", and grant is a nonblocking exchange from empty. A second *Odin* cannot hold
|
||||
the lease while the first is still writing, so a foreign writer seen by a lease-holding Odin is not an Odin.
|
||||
- **Idunn has a lifecycle brake** (`idunn-provision lifecycle-brake-engage|release|status`,
|
||||
`provisioning.rs:95-97`; `odin-lifecycle-brake.cc` per `runbooks/odin-yggdrasil.md:29-30`). An engaged brake
|
||||
denies continuity restarts (`control_plane.rs:5771-5774`, `:6219`). This is the operator's hold for a manual
|
||||
restore.
|
||||
- **A non-`active` Odin presence fans out.** `reconcile` sets `ready` only when the presence state is `active`
|
||||
(`lib.rs:946`), and `dependency_evidence` (`lib.rs:1184ff`) makes every managed dependent of Odin not Ready.
|
||||
The presence schema allows `degraded` (`cultnet-rs runtime_authority_contracts.rs:193`), but using it would
|
||||
be a readiness statement about every dependent, not a store report.
|
||||
- **The catalog serves only peer types and presences** (`public_document`, `main.rs:590-640`). No `odin.*` record
|
||||
is served today, and neither Odin nor Idunn publishes an Eve surface. F3 in the write map (the unowned
|
||||
`surface:gamecult.network.status`) is where an Eve lowering belongs.
|
||||
|
||||
## 2. Ends
|
||||
|
||||
1. No single fault, whether a foreign write of any kind, a corrupt file, or a failed write, ends Odin or leaves it
|
||||
silently unpersisted. Only losing the write lease ends it.
|
||||
2. A store file that is not what Odin last wrote is preserved byte for byte beside the store and never deleted,
|
||||
merged or adopted by any automatic path. The owner (Odin under the lease) writes its own state over it.
|
||||
3. At activation a corrupt store is recovered from the best source on the host (previous generation, then empty),
|
||||
and never from the backup automatically.
|
||||
4. No recovery can make a correlation sequence repeat or fall below one Idunn has admitted.
|
||||
5. Every repair, and every stretch in which Odin is serving without persisting, is typed state that can be
|
||||
queried through Odin's catalog while it is happening, including when the disk refuses writes.
|
||||
|
||||
Invariants carried over: one writer (the lease, checked immediately before every write); Idunn's lockless
|
||||
crossing reads a whole `.cc` snapshot at `topology.cc`; the path never becomes absent (Idunn reads an absent file
|
||||
as an empty snapshot, which withdraws every correlation); no reads of the store while serving; one bad record
|
||||
never stops Odin.
|
||||
|
||||
## 3. Identity, lifecycle, authority
|
||||
|
||||
| Persistent kind | Named by | Over time | Decided by |
|
||||
|---|---|---|---|
|
||||
| store file `topology.cc` | the state root plus `TOPOLOGY_SLOT` | replaced whole by the owner's exchange; never absent once created | Odin under the lease |
|
||||
| displaced file (new) | `topology.cc.displaced.<UTC yyyymmddThhmmssZ>.<uuid>`, same directory (it must not end in `.tmp`) | created by a set-aside exchange as a hard link to the replaced inode; never modified; never deleted by any program. Travels to Raven with the nightly backup | created by Odin through CultLib; **disposed of only by the operator** |
|
||||
| previous generation (new, Q-R2) | `topology.cc.previous` | replaced on every *verified* exchange (current == expected) by a hard link to the outgoing inode; **not** rotated by a set-aside exchange, so it always holds a generation that some writer verified before replacing | CultLib mechanism; Odin opts in |
|
||||
| store condition record (new) | type `odin.store_condition`, schema `odin.store_condition.v1`, key `topology` | one record held in W; rewritten when the condition changes; flushed with the store; served by the catalog | Odin |
|
||||
| correlation sequence floor (new rule, not a record) | per target, in the existing watermark | next = max(mark + 1, prior + 1, now in unix ms) | Odin |
|
||||
| backup snapshot | `/var/backups/gamecult-state/<date>/` and Raven | nightly; never live authority | the operator restores it, with the lifecycle brake engaged (runbook) |
|
||||
|
||||
## 4. Authority map (the whole campaign)
|
||||
|
||||
- **Owner:** Odin, while it holds the current process-write lease, decides what `topology.cc` holds, when it is
|
||||
set aside, and what activation recovers from. CultLib owns the mechanism (read classification, set-aside
|
||||
exchange, previous generation, staging cleanup) and no policy. Idunn owns whether Odin runs (continuity budget,
|
||||
lifecycle brake, deploys). The operator owns restores from backup and the disposal of displaced files.
|
||||
- **Inputs:** W; `flushed`; the lease (read immediately before every write); at activation, `topology.cc` and then
|
||||
`topology.cc.previous`; the clock (the sequence floor).
|
||||
- **Outputs:** one atomic `topology.cc`; displaced files; `topology.cc.previous`; the `odin.store_condition` record
|
||||
in the catalog.
|
||||
- **Derived state:** `topology.cc` is a checkpoint and Idunn's crossing, never an input while serving. The
|
||||
displaced files are evidence only: nothing reads them automatically. The previous generation is read only by
|
||||
activation recovery. The condition record is a report and decides nothing.
|
||||
- `ForeignStoreWrite` **is no longer an owner of Odin's life**. A foreign write becomes a set-aside event.
|
||||
- **Forbidden writers and deciders:**
|
||||
- Any path that ends Odin on a store condition. `survive`'s `ForeignStoreWrite` arm (`main.rs:884`) is
|
||||
deleted; `WriteLeaseLost` stays, and so does the temporary `PresenceAuthorityRefused` rule.
|
||||
- Any automatic delete of a displaced or previous file.
|
||||
- Any merge of records from a foreign or displaced file into W.
|
||||
- Any automatic restore from the backup.
|
||||
- Idunn writing or repairing `topology.cc`.
|
||||
- Any second writer of the file inside Odin: activation never writes. It prepares W and `flushed`, and the one
|
||||
`flush` makes the first write through the same set-aside exchange.
|
||||
- **Shared paths:**
|
||||
- The interval flush, the Q3 B per-put flush, the stop flush and the activation-recovery first flush all go
|
||||
through one primitive, `replace_snapshot_setting_aside`.
|
||||
- Every path that reserves a correlation sequence goes through the one floored reservation.
|
||||
- **Deletion line:** `ForeignStoreWrite` (`lib.rs:630-641`) and its `survive` arm are deleted before the set-aside
|
||||
exchange is wired in. The `Err` escape in `load` (`lib.rs:521`, `pull_all()?`) is replaced by the recovery
|
||||
ladder, not wrapped.
|
||||
|
||||
## 5. Recovery by case
|
||||
|
||||
| Case | Detected by | Action | Recovered from | Loss bound | Visible as |
|
||||
|---|---|---|---|---|---|
|
||||
| **A. Decodable foreign write while serving** (a hand edit, a tool, a restore made against the runbook, or bit rot in a payload per P2b) | the next flush: current != `flushed` | the lease is checked; the file is set aside (hard link); W is written; Odin keeps serving | W | none of Odin's own records. The foreign file's content is preserved, not adopted | condition event `displaced`, reason `mismatch`, with path, sha256, bytes and time |
|
||||
| **B. Undecodable foreign write while serving** (truncation, garbage, a foreign format, and after R2 trailing bytes too) | the next flush: current does not decode | the same | W | none. Idunn's crossing is unreadable for at most one heartbeat plus one flush (about 6 s), instead of forever | the same, reason `undecodable` |
|
||||
| **A/B repeated** (a live foreign writer) | a third set-aside inside 10 minutes in one activation | Odin stops writing and keeps serving; puts are not ACKed (Q3 B: ACK means durable); no new displaced files (Q-R4) | W stays in memory | nothing while the process lives | condition `contested`, since when, with the displaced files |
|
||||
| **C1. Undecodable store at activation** | `load` gets `Undecodable` | W is built from `topology.cc.previous` if it decodes; `flushed` is set to "unreadable", so the first flush sets the corrupt file aside and writes W | the previous generation | the last write's delta: one interval of Odin bookkeeping, or one fsynced put. **A put ACKed as durable can be lost here; say so in Q3 B's record** | event `recovered-previous`, plus an outstanding-republish list: peer records in the previous generation not yet re-put by their publisher, cleared as they arrive |
|
||||
| **C2. Nothing decodable at activation** | `.previous` absent or undecodable too | start empty (Q-R3); the first flush sets the corrupt file aside | providers republish; correlations re-derive in one pass; the sequence floor holds | every publish-once document; presence history (F4 makes this harmless) | event `started-empty`; outstanding list is `unknown`. The operator's path is the backup, with the lifecycle brake engaged |
|
||||
| **C3. Unsupported format at activation** (a newer runtime wrote v4, or a downgrade) | `load` gets `UnsupportedFormat` | refuse to activate. The file is untouched and nothing is set aside | the deploy or rollback that caused it; Idunn owns that | none | Idunn's continuity budget and the Odin journal. This is a deployment fault, not a store fault |
|
||||
| **C4. I/O error reading the store at activation** (EIO, EACCES after a restore with the wrong owner, EISDIR) | `load` gets an `io::Error` | refuse to activate. Nothing is set aside: **only bytes that were read are ever judged** | the operator fixes the host or the mode | none | as C3 |
|
||||
| **D. Write fails while serving** (ENOSPC, EDQUOT, EFBIG, EROFS, EIO) | the exchange returns an `io::Error` while staging or renaming | nothing is set aside; the staging file is removed (R1); W is kept; retry at the flush interval; puts are not ACKed; Odin keeps serving | W, when the fault clears | nothing, unless Odin is stopped or redeployed before the fault clears (follow-up I-F1) | condition `unpersisted` since T, with the last error kind (`no-space`, `read-only`, `io`) and the dirty-record count, served from memory |
|
||||
| **D'. Stop while unpersisted** | the stop flush fails | log it and stop, as today (`main.rs:794-799`) | the next activation loads the last good file | the unflushed delta | the final journal line only. I-F1 is the owner fix |
|
||||
|
||||
Rollback safety, for C1, C2 and an operator restore: every source except W carries a watermark below Idunn's
|
||||
cursor. The floor (O1) makes the next sequence exceed every sequence published before, whatever the store holds.
|
||||
|
||||
## 6. Cuts
|
||||
|
||||
Order: the Rust substrate gaps first (R1), then the watermark floor (O1, independent and small), the set-aside
|
||||
primitive (R3), Odin's serving repair (O2), activation recovery (O3), parity hardening (R2, R3b), and finally the
|
||||
runbook and drill (X1). Each cut is its own branch and its own Soul pass. Odin verification:
|
||||
`cargo test --locked -p odin-daemon` from the Odin root. CultLib Rust: `cd packages/cultcache-rs && cargo test
|
||||
--locked`. All of it runs on Yggdrasil through the stopgap. **A permission-based test is unproven there (root).**
|
||||
The injections below avoid permissions: truncation, a directory at the store path (EISDIR), tmpfs size and
|
||||
read-only options, and `RLIMIT_FSIZE`.
|
||||
|
||||
### R1. cultcache-rs: typed read failures, no staging file left behind, a header that survives truncation (CultLib)
|
||||
|
||||
- **Repo/branch:** CultLib `hands/cultcache-rs-store-faults` from `main`. Depends on nothing.
|
||||
- **Deletes first:** the flattening `map_err(|error| anyhow!(...))` in `read_all_unlocked` (`lib.rs:720-725`);
|
||||
`store_header`'s whole-array visitor (`lib.rs:317-341`).
|
||||
- **Adds:**
|
||||
- `pub struct CultCacheStoreUnreadable { path, kind: Undecodable | UnsupportedFormat }`, carried in the error
|
||||
chain of every read of a single-file store whose bytes were read and not accepted, so consumers can
|
||||
`downcast_ref`. An `io::Error` stays an `io::Error`, and the two never overlap.
|
||||
- `store_header` reads only the array header and the first string, reusing `read_array_header` and
|
||||
`read_string` (`lib.rs:2661-2700`). A truncated v1 file then reports as an undecodable v1 file, not as legacy.
|
||||
- `write_all_unlocked` removes its own staging file on every failure after creating it. This is C# parity with
|
||||
`CultCache.cs:3525-3530`.
|
||||
- **Rules that must die under their own mutation:**
|
||||
- a truncated store is `Undecodable`, and a v4 header is `UnsupportedFormat`;
|
||||
- an `io::Error` read failure is never classified `Unreadable`;
|
||||
- after a failed write, the directory holds no `<name>.*.tmp`.
|
||||
- **Verification:**
|
||||
- Tests describe behaviour. The staging-cleanup test forces the failure without permissions. One option is a
|
||||
child process under `RLIMIT_FSIZE` with SIGXFSZ ignored (the probe's method); another is a path whose parent
|
||||
fills a size-capped tmpfs, if the test runner can mount one. Hands picks one and names it.
|
||||
- `cargo mutants --in-diff`.
|
||||
- **Estimate:** about -30 / +70.
|
||||
|
||||
### O1. The correlation sequence has a time floor (Odin) — default D1
|
||||
|
||||
- **Repo/branch:** Odin `hands/odin-sequence-floor` from the landed `hands/odin-write` head. Depends on nothing
|
||||
else.
|
||||
- **Change:** `reserve_publisher_sequence` (`lib.rs:702-723`) returns max(mark + 1, prior + 1, now in unix ms),
|
||||
and records it as the mark. The doc comment at `lib.rs:33-49` gains the rule: "A mark read from any older copy
|
||||
of the store is a floor, never the next value; the clock makes every rollback safe."
|
||||
- **Why a floor and not a read of Idunn's cursor:** it needs no new crossing. It holds for every recovery source,
|
||||
including the backup, and the host clock is already trusted for presence freshness.
|
||||
- **Assumption:** the clock does not step back across a rollback by more than the rollback's age. Soul checks that
|
||||
Idunn accepts a jump: its rule is "strictly above the cursor" (`control_plane.rs:1763,1785`). No test may pin a
|
||||
gap limit.
|
||||
- **Verification:**
|
||||
- Existing tests pinning counts (`withdrawing_a_correlation_does_not_reset_its_publisher_sequence`
|
||||
`lib.rs:1996-2014`) are restated as order properties.
|
||||
- New test: a store rolled back to an earlier snapshot, then one refresh, yields a correlation sequence greater
|
||||
than every sequence published before the rollback.
|
||||
- New test: two reservations inside one millisecond are strictly increasing.
|
||||
- The clock must be injectable at the reservation. At least one mutant must be a function of the time (an offset
|
||||
or a scale), with probe values where it would show.
|
||||
- **Estimate:** about -5 / +25.
|
||||
|
||||
### R3. The set-aside exchange and the previous generation (CultLib, Rust and C#; TS and Python in R3b)
|
||||
|
||||
- **Repo/branch:** CultLib `hands/cultcache-set-aside` from R1. Q-R2 decides the previous-generation half.
|
||||
- **Adds, on `SingleFileMessagePackBackingStore`:**
|
||||
- `replace_snapshot_setting_aside(expected, replacements) -> Result<SnapshotReplacement>`, with
|
||||
`SnapshotReplacement::{Exchanged, SetAside { displaced: PathBuf, sha256, bytes, reason: Mismatch |
|
||||
Undecodable }}`. It makes one exclusive lock hold, in this order:
|
||||
1. stage the replacement (write and fsync);
|
||||
2. read current;
|
||||
3. if current equals expected, rename the staged file over (rotating `.previous` first when enabled);
|
||||
4. if the read was an `io::Error`, remove the staging file, return the error and change nothing;
|
||||
5. otherwise hard-link current to the displaced name, fsync the directory, rename the staged file over, and
|
||||
fsync the directory again.
|
||||
|
||||
Staging comes first, so a full disk sets nothing aside. The path is never absent: a link, never a rename. A
|
||||
set-aside never rotates `.previous`.
|
||||
- Opt-in previous generation: a constructor option (`retaining_previous_generation`, or Hands' spelling of it)
|
||||
plus `pull_previous_generation() -> Result<Vec<CultCacheEnvelope>>`, read under the shared lock and classified
|
||||
like any read. Off by default, because existing consumers must not grow files they did not ask for.
|
||||
- Filesystems without hard links: return an explicit error naming the filesystem. Do not fall back to a
|
||||
copy-then-rename, which would make the path briefly hold new content while the evidence is still being
|
||||
written. Windows NTFS links work; the Windows path is **not yet reached** by the Yggdrasil suite and is
|
||||
recorded as such.
|
||||
- C# reference: the same operation on `SingleFileBackingStore` (`CultCache.cs:3268ff`), with the same names
|
||||
across runtimes.
|
||||
- **Verification (Rust and C#):**
|
||||
- a mismatch preserves the foreign bytes exactly and writes the replacement;
|
||||
- an undecodable current is preserved and replaced;
|
||||
- an I/O read failure (a directory at the path) changes nothing and leaves no staging file;
|
||||
- a staging failure (size cap) changes nothing and sets nothing aside;
|
||||
- the path is readable at every instant: a reader thread loops `pull_all_read_only_snapshot` through 1,000
|
||||
exchanges and never sees an absent file or an empty snapshot;
|
||||
- `.previous` rotates only on a verified exchange;
|
||||
- two set-asides in the same second get distinct names;
|
||||
- mutation tools: `cargo mutants --in-diff`, and Stryker.NET on the C# diff.
|
||||
- **R3b (parity, not blocking Odin):** cultcache-ts `single-file-messagepack-backing-store.ts:34` and cultcache-py
|
||||
`stores.py:83` gain the same operation and option, with a shared fixture test.
|
||||
- **Estimate:** Rust about +140, C# about +120, R3b about +200.
|
||||
|
||||
### O2. A foreign store file while serving is set aside, not fatal (Odin)
|
||||
|
||||
- **Repo/branch:** Odin `hands/odin-store-repair` from O1, with a CultLib pin bump to R3. Needs Q-R1 and Q-R4.
|
||||
- **Deletes first:** `ForeignStoreWrite` (`lib.rs:630-641`) and its arm in `survive` (`main.rs:884`). The test
|
||||
`a_flush_writes_only_changes_and_never_over_a_foreign_file` (`lib.rs:2055-2090`) and the `survive` tests at
|
||||
`main.rs:2842-2844` and `:3110` pin the old rule: restate them, do not delete them.
|
||||
- **Changes:**
|
||||
- `MemoryOdinTopologyStore::flush` (`lib.rs:613-626`) calls `replace_snapshot_setting_aside(flushed, current)`.
|
||||
On `SetAside` it records a `displaced` event and counts it toward the contested bound. On an I/O failure it
|
||||
records `unpersisted` and returns the error, which `survive` logs and retries as today.
|
||||
- `RuntimeState::flush` (`main.rs:307-314`) keeps its lease check immediately before the write.
|
||||
- The Q3 B per-put path uses this same flush. Hands re-anchors on the Q3 B head.
|
||||
- **Adds:**
|
||||
- The `odin.store_condition` record (schema in `odin-core` `documents.rs`), held in W. It carries:
|
||||
- `generation` and `written_by`, updated inside the flush before the snapshot is taken;
|
||||
- `unpersisted_since` and `last_write_error`;
|
||||
- `contested_since`;
|
||||
- a bounded list of the last 16 recovery events: kind, time, displaced path, sha256 and bytes, source, records
|
||||
loaded, and the outstanding list (bounded).
|
||||
- One arm in `public_document` (`main.rs:590-608`) that serves exactly this type. Providers still cannot write
|
||||
it: `is_peer_document_type` is unchanged.
|
||||
- **Rules that must die under their own mutation:**
|
||||
- a flush never leaves the store dirty (the condition record is written inside the flush, not after it);
|
||||
- a set-aside is preceded by a lease check;
|
||||
- the third set-aside inside the window stops writing, and the second does not;
|
||||
- after `contested`, no file is written or created;
|
||||
- on an I/O failure, nothing is set aside;
|
||||
- the condition is served while writes fail.
|
||||
- **Authority map:** as in section 4. The deletion line is the `ForeignStoreWrite` type and arm.
|
||||
- **Verification (fault injection, each at the file layer, not a log line):**
|
||||
- T-A: activate, serve, write a different valid snapshot behind Odin. After the next pass the file decodes to
|
||||
W, the displaced file's bytes equal the foreign bytes, `serving_pass` returned `Ok`, the catalog is unchanged,
|
||||
and the served condition names the displaced file and its sha256.
|
||||
- T-A2 (P2b): flip one payload byte behind Odin; the same outcome, reason `mismatch`.
|
||||
- T-B: truncate the file behind Odin; the same outcome, reason `undecodable`, and
|
||||
`pull_all_read_only_snapshot` (Idunn's crossing) decodes the file after one pass.
|
||||
- T-B-latency: with no puts, the set-aside happens within one heartbeat plus one flush interval.
|
||||
- T-contested: three foreign writes inside the window make exactly two set-asides and a third `contested`
|
||||
report. After that the file and directory are untouched by further passes, and Odin still serves.
|
||||
- T-D: replace the store path with a directory (EISDIR, which works as root). Nothing is set aside, no staging
|
||||
file remains, the condition is `unpersisted` and is served, and a put is not ACKed. Remove the directory, and
|
||||
the next pass writes W, clears `unpersisted` and ACKs.
|
||||
- T-lease: the lease withdrawn before a flush that would set aside means no link, no write, and Odin ends
|
||||
(`WriteLeaseLost`).
|
||||
- Negative: `rg -n "ForeignStoreWrite" crates` matches nothing; `rg -n "remove_file|remove_dir" crates/odin-daemon/src`
|
||||
matches nothing new outside tests.
|
||||
- `cargo mutants --in-diff`.
|
||||
- **Estimate:** about -40 / +160.
|
||||
|
||||
### O3. Activation recovers a corrupt store (Odin)
|
||||
|
||||
- **Repo/branch:** Odin `hands/odin-store-activation` from O2. Needs Q-R2 and Q-R3.
|
||||
- **Change:** `MemoryOdinTopologyStore::load` (`lib.rs:519-538`) becomes the recovery ladder:
|
||||
|
||||
| `load` result | Outcome |
|
||||
|---|---|
|
||||
| `Ok` | as today |
|
||||
| absent | empty, as today |
|
||||
| `Unreadable::Undecodable` | try `pull_previous_generation()`, then empty (Q-R3); `flushed` becomes a sentinel meaning "the file is not a snapshot Odin can compare", so the first flush's exchange sets it aside |
|
||||
| `Unreadable::UnsupportedFormat` | return the error: refuse |
|
||||
| `io::Error` | return the error: refuse |
|
||||
|
||||
Activation itself never writes. The first `serving_pass` flush (`main.rs:833-842`, due at once) writes W. That
|
||||
set-aside is recorded under the activation event and is not counted toward the contested bound.
|
||||
- **Adds:** the outstanding-republish list. It holds the peer-type records present in the recovered previous
|
||||
generation, each cleared when its publisher re-puts it. After 10 minutes the remainder is marked "not
|
||||
republished", which is the live answer to "who is publish-once". For `started-empty` the list is `unknown`.
|
||||
- **Rules that must die under their own mutation:**
|
||||
- Unsupported-format and I/O failures never set aside, never fall back, and never start.
|
||||
- A recovered activation writes before it serves its first catalog snapshot, or it serves the recovered set and
|
||||
marks the condition before the first write. Hands picks one and states it.
|
||||
- **Verification:**
|
||||
- T-C1: a truncated `topology.cc` and a valid `.previous` activate with the previous content. The first pass
|
||||
writes it and sets the corrupt file aside byte-exact, and the condition is `recovered-previous` with its
|
||||
outstanding list. Re-putting one listed document clears it.
|
||||
- T-C2: both undecodable (Q-R3 A) start empty. Both files are preserved, the condition is `started-empty`, and
|
||||
the next correlation sequence is above the floor.
|
||||
- T-C3: a v4 header refuses to activate. The file is byte-identical afterwards, and there is no displaced file
|
||||
and no `.previous` change.
|
||||
- T-C4: a directory at the store path refuses, as C3.
|
||||
- T-restart: a recovered Odin stopped and restarted loads its own first write normally, with no second
|
||||
set-aside.
|
||||
- `cargo mutants --in-diff`.
|
||||
- **Estimate:** about -10 / +110.
|
||||
|
||||
### R2. Single-file decoders reject trailing bytes, in every runtime (CultLib)
|
||||
|
||||
- **Repo/branch:** CultLib `hands/cultcache-trailing-bytes`. Independent of Odin; land it before O3 ships so that
|
||||
an appended file is caught as undecodable, not overwritten silently (P2).
|
||||
- **Change:** C# `DeserializeSnapshot` (`CultDocumentMessagePackSerialization.cs:179-215`) checks `reader.End`.
|
||||
Rust `decode_store_snapshot` (`lib.rs:2583ff`) and the legacy path check that the slice is consumed. TS and
|
||||
Python do the same. **One release**: a runtime that accepts what another rejects is a parity split.
|
||||
- **Verification:** one shared fixture (a valid store plus one byte) is rejected as undecodable by all four
|
||||
runtimes, and the valid store is accepted by all four.
|
||||
- **Estimate:** about +40 plus tests across four runtimes.
|
||||
|
||||
### X1. Runbook and recovery drill (gamecult-ops, then live)
|
||||
|
||||
- **Repo:** gamecult-ops `runbooks/odin-yggdrasil.md` gains a "Store recovery" section:
|
||||
- how to read `odin.store_condition` from the catalog;
|
||||
- what displaced and previous files mean, and that disposing of them is the operator's decision;
|
||||
- the manual restore: engage Odin's lifecycle brake, stop the `idunn-odin-*` unit, set the live file aside with
|
||||
`ln` (never `rm`), extract the snapshot's `topology.cc` with its owner, mode and `.lock`, release the brake,
|
||||
and read the condition. The sequence floor makes an old watermark safe.
|
||||
- **Drill D-1 (off-live, after O3; needs the Q2-style approval, Q-R6):** extract the newest Yggdrasil snapshot's
|
||||
`topology.cc` into a container on Yggdrasil. Activate a test-harness Odin on it through the existing fixture
|
||||
authority. Then truncate the copy and activate again with the extracted file as `.previous`. Report the records
|
||||
per type, the outstanding list and the sequence floor. Delete the copy afterwards.
|
||||
- **Drill D-2 (live, operator-scheduled, Q-R6), after O3 deploys:**
|
||||
1. Record the baseline: catalog types, Idunn correlated targets, route timeouts.
|
||||
2. Engage the lifecycle brake and stop Odin.
|
||||
3. `ln topology.cc /var/tmp/odin-drill-keep.cc` (the evidence of the evidence), then truncate `topology.cc` in
|
||||
place to half.
|
||||
4. Release the brake and let Idunn restart Odin.
|
||||
5. Pass when all of these hold:
|
||||
- Odin is serving within one restart;
|
||||
- the condition is `recovered-previous`, with a displaced file whose sha256 equals the truncated file's;
|
||||
- every Odin-correlated target is Ready again within 60 s;
|
||||
- there are 0 route-challenge timeouts in the next hour;
|
||||
- the catalog's type set equals the baseline after 10 minutes, or the outstanding list names every
|
||||
difference.
|
||||
6. Then run D-2b, a foreign write while serving: `ln` the live file aside, then truncate it. It passes when the
|
||||
condition shows `displaced` within about 6 s and Idunn's correlation reads never fail for longer than that
|
||||
(the journal).
|
||||
|
||||
## 7. Subtraction ledger (estimate)
|
||||
|
||||
| Cut | Removed | Added | Formats and targets |
|
||||
|---|---|---|---|
|
||||
| R1 | about 30 | about 70 | none |
|
||||
| O1 | about 5 | about 25 | none |
|
||||
| R3 | 0 | about 260 (Rust 140, C# 120); R3b about 200 | two sibling-file conventions (`.displaced.*`, `.previous`); no format change |
|
||||
| O2 | about 40 | about 160 | one Odin schema, `odin.store_condition.v1` |
|
||||
| O3 | about 10 | about 110 | none |
|
||||
| R2 | 0 | about 40 plus tests | none (it tightens the decoders) |
|
||||
|
||||
The campaign is net additive, at about +570 without R3b. It buys ends 1-5, which the operator asked for, and
|
||||
removes one fatal path and one silent path. Every addition in CultLib is a capability a general consumer of the
|
||||
single-file store would expect, and two of them (staging cleanup, and the header that survives truncation) are
|
||||
defects.
|
||||
|
||||
## 8. Operator questions
|
||||
|
||||
- **Q-R1. A store file changed underneath a running Odin (decodable or not).**
|
||||
- Context: Idunn fences the old Odin before granting the lease, so the other writer is never an Odin. It is a
|
||||
tool, a hand edit, a restore made against the runbook, or disk corruption; P2b shows a flipped byte looks
|
||||
exactly like a foreign write. Today a decodable change ends Odin, and the restart adopts the foreign file
|
||||
silently. An undecodable change stops persistence forever and breaks Idunn's crossing.
|
||||
- A: set the file aside (kept byte for byte, never deleted), write Odin's working set over it, keep serving,
|
||||
and report it as typed state.
|
||||
- B: set it aside, write the working set, then end Odin so a fresh process starts from the rewritten file.
|
||||
- **Recommended: A.** B's restart protects nothing that the lease check before the write does not already
|
||||
protect, and costs a route outage.
|
||||
- **Q-R2. Keep one previous generation on disk (`topology.cc.previous`)?**
|
||||
- Context: without it, a store that is corrupt at activation has no on-host source except "empty", and the
|
||||
backup is up to 24 h old and needs the operator. It costs one hard link per write (a directory update, no data
|
||||
written) and one more file of the store's size. It is a CultLib option, off by default for other consumers.
|
||||
- A: yes. B: no; activation recovery goes straight to empty.
|
||||
- **Recommended: A.** It is the only source that recovers publish-once documents, presence history and
|
||||
correlations without the operator. Its loss bound is one write.
|
||||
- **Q-R3. Activation finds nothing decodable: no usable `topology.cc`, no usable previous generation.**
|
||||
- Context: presences and most catalog documents return within seconds to a minute, and correlations re-derive
|
||||
in one refresh pass. Publish-once documents (if any exist, see 1.3) stay missing until their publisher
|
||||
re-puts them, or until the operator restores from the backup.
|
||||
- A: start empty automatically, set the corrupt file aside, and report `started-empty`.
|
||||
- B: refuse to start, so Odin crash-loops until the operator restores. Every service's rendezvous is down
|
||||
meanwhile.
|
||||
- **Recommended: A.** A down Odin loses everything the Verse needs; an empty one loses only what nobody
|
||||
republishes, and the restore path stays open.
|
||||
- **Q-R4. A foreign writer keeps writing (a third set-aside within 10 minutes).**
|
||||
- Context: repeated repair against a live writer makes a displaced file every second.
|
||||
- A: stop writing, keep serving from memory, withhold put ACKs, and report `contested` until the operator
|
||||
finds the writer. The working set survives.
|
||||
- B: end Odin. Idunn restarts it, and the restart loads whatever the foreign writer last wrote.
|
||||
- **Recommended: A.** B hands the store to the unknown writer. A is loud and loses nothing that is in memory.
|
||||
- **Q-R5. Should a store condition change Odin's presence to `degraded`?**
|
||||
- Context: a non-`active` Odin presence makes every managed dependent of Odin not Ready (`lib.rs:946`,
|
||||
`dependency_evidence`). The store condition does not affect serving.
|
||||
- A: no. The presence stays `active`, and the condition record is the signal.
|
||||
- B: `degraded` while `unpersisted` or `contested`.
|
||||
- **Recommended: A.** Otherwise a full disk becomes an outage of every dependent, and a restart that Idunn might
|
||||
take in response would destroy the only complete copy.
|
||||
- **Q-R6. The drills.**
|
||||
- D-1: decode a copy of last night's backup `topology.cc` in a container on Yggdrasil, the same terms as Q2 for
|
||||
the write map, with the copy deleted afterwards.
|
||||
- D-2: after O3 deploys, a scheduled live drill that truncates the real `topology.cc` behind the lifecycle brake,
|
||||
keeping a hard link of the original.
|
||||
- **Recommended: yes to both**, D-1 before O3 ships and D-2 in a window the operator names.
|
||||
|
||||
## 9. Follow-ups outside this map
|
||||
|
||||
- **I-F1 (Idunn):** a deploy or a stop of Odin while `odin.store_condition` says `unpersisted` or `contested`
|
||||
destroys the only complete copy. Idunn could read the condition from the catalog and hold the deployment
|
||||
(deployment brake, never continuity). This is Idunn's decision, raised with the Idunn map.
|
||||
- **I-F2 (Idunn):** "Idunn stopped restarting admitted odin" is a log line (`report_once`, `control_plane.rs:4599`),
|
||||
not typed state. It belongs to Idunn's own operator surface.
|
||||
- **F3 (from the write map):** the Eve lowering of `odin.store_condition` belongs to Odin's operator surface.
|
||||
This map publishes the typed record only.
|
||||
- **H-F1 (Heimdall):** confirm whether the deployed Heimdall re-puts its Odin documents every 60 s. O3's
|
||||
outstanding list measures it after any recovery.
|
||||
- **Q3 B's record in the write map:** "an ACK means durable" holds against crashes, not against a corruption at
|
||||
activation, which loses at most the last write (C1).
|
||||
|
|
@ -3,7 +3,86 @@
|
|||
**2026-09-30, Self.** Operator: "Let's fix Odin up, that's some furious bookkeeping right there. Look at Odin's
|
||||
write pattern and see if append only or per-document is the way to go." Answer: neither is the first fix. Cuts 1-2
|
||||
go to Hands now. Q3 (a 1 s interval) and Q4 (the Rust directory store as a CultLib follow-up) are Self defaults,
|
||||
and the operator may overrule them. Q1 and Q2 are put to the operator, and Cut 3 waits on Q1.
|
||||
and the operator may overrule them. **Operator, 2026-09-30: "Q1: yes", so CultMesh refuses puts it could never
|
||||
serve and Cut 3 goes ahead; "Q2: yeah, of course", so a read-only decode of a copy of live `topology.cc` is
|
||||
approved. The copy is taken with `sudo cp` and decoded off the live path, and the copy is deleted afterwards.**
|
||||
|
||||
**Soul on Cuts 1-2, and two operator rulings (2026-09-30).**
|
||||
- **Q3 restated.** The premise "puts carry no application acknowledgement" was false. The cultmesh-rs server
|
||||
ACKs a put only after `accept_raw_document` returns Ok, so after Cut 1 an ACK meant "in memory", and a SIGKILL
|
||||
lost acknowledged puts in 6 of 6 probe runs.
|
||||
- **Operator ruling: "Q3: B".** Peer puts are fsynced before they are acknowledged. Only Odin's own
|
||||
bookkeeping (heartbeats, correlations) waits for the 1 s interval. An ACK means durable.
|
||||
- **An undecodable foreign write.** Today Odin stays up, serves from memory and never persists again. Self offered
|
||||
three options: degraded but loud, end Odin, or unchanged.
|
||||
- **Operator: "None of these options involve repair and recovery, which is what ultimately needs to happen."**
|
||||
- Recovery is mapped as its own pass. It covers an undecodable foreign write, a decodable one, and a failed
|
||||
load at activation.
|
||||
|
||||
**Soul on write batch 2 (2026-09-30).** Q3 B changes the expected effect. Cut 2's row in section 6 ("at most
|
||||
3.5 MB/s, at most 1.05 writes/s") assumed Q3 A and is **superseded**. Under Q3 B, each peer put (presence
|
||||
heartbeats included) writes the store once.
|
||||
- With the live-shaped 3 MB store that is about 2.5-3 writes/s, about 7.5-9 MB/s, and 70-170 ms of the loop
|
||||
blocked each second.
|
||||
- Measured: route-challenge p99 461 ms and 11 kernel drops.
|
||||
- After Cut 3's Odin half the store is about 80 KB and a put write is p50 about 4 ms.
|
||||
- **Cut 2 therefore deploys only together with Cut 3's Odin half**, which is now being built on `hands/odin-write`.
|
||||
|
||||
**Q2 decode result (Eyes, 2026-09-30; full table in the CultLib session scratchpad `odin-topology-decode.md`).**
|
||||
- 77 records; 3,222,550 B of payload in a 3,240,078 B file.
|
||||
- The two "~1.55 MB records" are per-type totals. Each type is dominated by one Ghostlight operator surface:
|
||||
`eve:operator:ghostlight.campaign.27d444b3-...` (1,456,210 B) and its `operator-state` twin (1,456,341 B).
|
||||
Together they are 89.9% of the file.
|
||||
- Publisher: provider `gamecult.ghostlight.dungeon`. 99.5% of each record is one Eve `code` node holding a
|
||||
pretty-printed JSON dump of the whole campaign state.
|
||||
- These records are dead residue. They were stored 2026-09-23 and have not been rewritten since. Ghostlight `HEAD`
|
||||
(`eeb3a68`) no longer contains the emitter (last touched at `6bb6869`, 2026-09-02).
|
||||
- **So Q1's refusal breaks no live publisher.** Cut 3's Odin half drops exactly these two at activation. Every other
|
||||
surface is 70 KB or less.
|
||||
- Correction to section 3: the 1 MiB snapshot refusals in the journal are per-type responses, not per-record.
|
||||
Cut 3's per-document bound is still right: each of these records is over 1 MiB alone. A type whose many small
|
||||
records add up past 1 MiB would still be unservable as a whole-type snapshot. That is a follow-up for CultMesh
|
||||
(paged snapshot responses), not part of this campaign.
|
||||
- Presence history: 41 records, 42 KB, 37 of them withdrawn. F4 is a slow leak, not a size problem.
|
||||
- The current `streampixels-web` incarnation has a correlation but no presence record; its last presence was
|
||||
09-29 19:53 UTC. This matches Idunn's "not Odin-correlated" state.
|
||||
|
||||
**Soul on write batch 3 + Cut 3 Odin half (`f4a1f89..726e0f3`, 2026-09-30): not fit to deploy.**
|
||||
- **S-1 (high; operator question Q5, below).** Every write, the fsync pair included, runs on the serving loop
|
||||
(`main.rs:358` → `flush` → `compare_exchange_snapshot`; the interval flush at `:883` is on the same thread).
|
||||
- A disk stall passes into route-challenge latency one for one. A probe that held the store lock measured: a 0 ms
|
||||
stall gave 86 ms, 300 ms gave 288 ms, 1,500 ms gave 1,487 ms.
|
||||
- Interleaved runs (ABBA) under IO pressure of 5-38% gave a challenge p99 no better with Cut 3 than without it.
|
||||
- Cut 3 halves the bytes written, not the stalls. **The premise that the store's size caused the stalls was
|
||||
wrong; they come from the disk.**
|
||||
- **S-2 (high).** Q1's put refusal has not landed. Odin's pin `3bf1c0ce` has no put bound, so a 1.2 MB put is
|
||||
ACKed, fsynced, and dropped at the next activation. It waits on the CultLib put-serve merge, then a pin bump with
|
||||
a `served_record` override.
|
||||
- **S-3 (medium).** `attempted` grows about 98 KB per failed write, roughly 0.9 GB/h on a full disk, which ends in
|
||||
OOM.
|
||||
- Fix in the owner: cultcache-rs returns a typed `NotReplaced`/`ReplacedNotDurable` write outcome (added to CultLib
|
||||
R1 batch 3), and Odin keeps at most one candidate.
|
||||
- **S-4 (medium).** Two admitted presences for one target, which a provider in a deploy window produces, refuse the
|
||||
whole catalog (`main.rs:~628`).
|
||||
- **S-5 (medium).** A RUDP transport ACK is recorded at receipt, so with pipelined puts a later put is ACKed while
|
||||
an earlier one is still held. With the first datagram lost, the second is never delivered. That is data loss
|
||||
under the QUIC-pivot rule; it is tracked in the CultLib ack map. `dcd36fc`'s doc line overstates.
|
||||
- **S-6, S-8.** Test gaps (the limit taken through `main`; `remove_not_dirty`).
|
||||
- **S-7.** The drop log echoes the key. Self ruled keys and schema ids are identities and may be logged, as in
|
||||
CultLib R1.
|
||||
- **Held:** F1's logic, F4's projection fallback, F5, F3's refusal, and Cut 3's activation bound, drop and removal.
|
||||
Sizing uses `peer_document`, so it matches what is served.
|
||||
- **Batch 4 in Hands:** S-4, S-6, S-8, the S-5 doc line, the attempted-list fixture gap, and committing Soul's
|
||||
probes. S-1 waits on Q5; S-2 and S-3 wait on CultLib.
|
||||
|
||||
**CultLib side landed (2026-09-30).** Ack Cut D was merged at `2f06ef84`, R1 at `daedfdd6` and the put-serve bound at
|
||||
`9dcb3aa1`. That unblocks the Q5 sequence:
|
||||
1. R1 batch 3's typed write outcome is on main. It carries `Rejected` as well as `NotReplaced`/`ReplacedNotDurable`.
|
||||
W0 must classify by kind together with the `io::Error`, never by `NotReplaced` alone.
|
||||
2. **C5 next:** cultmesh-rs deferred put reply, on main.
|
||||
3. **W0:** one Odin pin bump. It closes S-2 (put refusal with a `served_record` override) and S-3 (delete
|
||||
`attempted`), and it brings in Cut D's and the put-serve error texts.
|
||||
4. **W1:** the writer thread.
|
||||
|
||||
Status: cut map, Imagination pass 1 (Opus), 2026-09-30. Nothing has landed. Anchors are Odin `main` at
|
||||
`44951a1` (the deployed build, live on Yggdrasil since 2026-09-30 04:20:48 UTC) and CultLib `3bf1c0ce`
|
||||
|
|
@ -352,6 +431,532 @@ records; update the check's type list in the same pass.
|
|||
- **Q4. CultLib parity: port `DirectoryMessagePackBackingStore` (v4) to cultcache-rs?** It is not needed by
|
||||
Odin after Cuts 1-3. A reasonable consumer of the Rust runtime would expect the reference's store.
|
||||
**Recommended: record it as CultLib follow-up F1 with no deadline.** It is independent of this map.
|
||||
- **Q5 (2026-09-30, from Soul S-1). Where does the durable write run?** Today the serving loop runs the fsync, so
|
||||
any disk stall stalls route challenges, catalog reads and every other put.
|
||||
- A: a writer thread with group commit.
|
||||
- The loop hands each accepted put to one writer thread. The writer writes and fsyncs everything queued since
|
||||
its last write in one go, then the loop sends those puts' replies.
|
||||
- Challenges and reads are never behind the disk. A stall delays only put replies, and Q3 B still holds (a
|
||||
reply means durable).
|
||||
- It needs cultmesh-rs to let a put handler reply later (a CultLib API change) plus one Odin thread.
|
||||
- B: answer route challenges on their own path. Everything else still stalls, and the challenge would report
|
||||
Odin healthy while its catalog is stuck.
|
||||
- C: reopen Q3 and go back to the 1 s interval. That brings back lost acknowledged puts on a crash.
|
||||
|
||||
**Recommended: A.** **RULED A by the operator, 2026-09-30: "agree on Odin writer thread".** Mapping is next
|
||||
(Imagination): the cultmesh-rs deferred put reply, then the Odin writer thread with group commit. Mapped below in
|
||||
"Q5 cut: the writer thread" (cuts C5, W0, W1; forks Q5-A and Q5-B).
|
||||
|
||||
## Q5 cut: the writer thread
|
||||
|
||||
**Imagination pass (Opus), 2026-09-30.** Maps the operator's ruling on Q5, "agree on Odin writer thread" (option A,
|
||||
section 7). Anchors:
|
||||
- Odin `hands/odin-write` at `434756e` (`crates/odin-daemon/src/main.rs`, `lib.rs`), CultLib pin `3bf1c0ce`.
|
||||
- CultLib `hands/cultmesh-put-serve-bound` tip `e4e2a0b9` (`packages/cultmesh-rs/src/rudp_document_server.rs`).
|
||||
Its merge base is `42ba0f08`, which already holds ack Cuts 1, 1b and 1c.
|
||||
- CultLib `hands/ack-cutD` tip `e9d5ae2c` (lands on main first), and R1 batch 3 `hands/cultcache-read-failures` tip
|
||||
`93ac5945` (`CultCacheStoreWriteFailed`).
|
||||
|
||||
The mapping raises two forks, **Q5-A** and **Q5-B** (at the end of this section). Cuts C5 and W1 wait on Q5-A.
|
||||
|
||||
**RULED, operator 2026-09-30: "Q5-A: A, but make a note of this for when we move on QUIC, Q5-B: A".**
|
||||
- **Q5-A = A.** The server withholds the session's transport ACK until the put is answered. The wire is unchanged.
|
||||
The note for QUIC (a put-result message) is in the QUIC campaign handoff.
|
||||
- **Q5-B = A.** Puts are held during a writer stall, bounded by the existing session and byte limits.
|
||||
|
||||
### Q5.1 Body facts (probed and read)
|
||||
|
||||
**Probe O1, Odin `434756e` + scratch test, Yggdrasil slot 3 (`imag-q5-odin@31953c93`).** A test process holds the
|
||||
store's sibling lock for 1,500 ms, the way Soul modelled a disk stall. Full serving passes run meanwhile. A route
|
||||
challenge is sent at 200 ms, then a catalog read.
|
||||
|
||||
| Case | Route challenge | Catalog read |
|
||||
|---|---|---|
|
||||
| nothing waiting to be written | served in **39 ms** | served in 29 ms |
|
||||
| only Odin's own heartbeat waiting, and the interval write due | served in **1,342 ms** | 33 ms (sent after the release) |
|
||||
|
||||
- No read path takes the store lock. At `434756e` the lock is taken only by `MemoryOdinTopologyStore::load`
|
||||
(`lib.rs:509`, at activation) and by `flush` (`lib.rs:612` → `compare_exchange_snapshot`).
|
||||
- The stall reaches the loop through **both** writes on it: a peer put (`main.rs:363`, Soul S-1) and the interval
|
||||
write of Odin's own bookkeeping (`main.rs:933-941`). The interval write stalls the loop with no peer put at all.
|
||||
So the bookkeeping write must move to the writer too, not only the put write.
|
||||
|
||||
**The loop still reads two other files.** They are not the store and take no lock, but they are disk reads on the
|
||||
loop:
|
||||
- The lease file, on every route challenge and catalog read: `raw_snapshot` `main.rs:389-409` →
|
||||
`require_current_write_lease` `:546-567` → `read_process_write_lease` `:1323` (`pull_all_read_only_snapshot`).
|
||||
- Idunn's projection, on every catalog read (`:608`), refresh (`:572`) and heartbeat (`:524-544`).
|
||||
- Both files are small and rewritten rarely by Idunn, so they are normally served from the page cache. Probe O1
|
||||
cannot show whether a real disk stall evicts them. Only the Yggdrasil rerun (Q5.6) can. If it does, the answer
|
||||
is to hold the lease record in memory and re-check it where it is decided. That is its own follow-up, not this
|
||||
cut.
|
||||
|
||||
**What a put's reply is on the wire today.** CultNet has no put-acknowledgement message: the list in
|
||||
`cultnet-rs/src/contracts.rs` has `document_put_raw` and `error.v0`, and no put result.
|
||||
- A Rust publisher (`publish_cultnet_message_to_rudp_catalog`, `cultmesh-rs/src/lib.rs:871-924` at `e4e2a0b9`)
|
||||
succeeds when its reliable packet is acknowledged by the RUDP transport, and fails on a `cultnet.error.v0`.
|
||||
- Its default `flush_timeout` is 300 ms, with a resend every 50 ms (`:144-157`).
|
||||
- The document server sends that acknowledgement at the end of `poll_once`, after the sink returns
|
||||
(`rudp_document_server.rs:468-476`). A refusal ends the session with an Error frame that acknowledges nothing
|
||||
(`end_rejected_session` `:760-793`, put-serve branch).
|
||||
- **So today the put's reply is the transport acknowledgement.** That is how Q3 B's "an ACK means durable" holds:
|
||||
the sink writes before it returns.
|
||||
|
||||
**Every packet a session sends carries acknowledgement fields** (`ack`, `ack_mask`, from `ack_state`, cultnet-rs
|
||||
`rudp.rs` at main).
|
||||
- The fields report every *received* reliable sequence, not only delivered ones.
|
||||
- `create_packet` stamps them on each new packet: data, Pong, the end-of-poll Ack.
|
||||
- A reliable packet is stored with the fields it was created with, and resent unchanged (`track_reliable`,
|
||||
`due_resends`).
|
||||
|
||||
Probes P1-P4 (cultnet-rs sessions at `e4e2a0b9`, scratch test `imag_q5_probe.rs`, Yggdrasil, `imag-q5-cultlib@8dcfe36a`):
|
||||
|
||||
| # | Case | Result |
|
||||
|---|---|---|
|
||||
| P1 | two pipelined ordered puts, the first lost | the second is delivered to nobody, yet `create_ack` acknowledges it. **S-5 is still open after ack Cuts 1-1c.** `ack_state` at main is the same code, so the ack map's to-do resolves to "not closed by the pin bump" |
|
||||
| P2 | a put delivered, then a Ping | the Pong acknowledges the put |
|
||||
| P3 | the resend of a reply created **before** the put | does not acknowledge it |
|
||||
| P3 | a reliable reply created **after** the put | acknowledges it, and so does its resend |
|
||||
| P4 | a retransmit of a delivered put | not delivered again (the server's end-of-poll ack still answers it, `:468`) |
|
||||
|
||||
Consequence for a reply sent later: holding back only the end-of-poll ack is not enough.
|
||||
- A Pong, or a snapshot response created on the same session after the put arrived, acknowledges the put too.
|
||||
- The snapshot response keeps doing so on every resend, even if its first copy were patched.
|
||||
- So a session with a put awaiting its answer must be **sent nothing created after that put arrived**. Resends of
|
||||
earlier packets are safe: they carry the fields they were created with.
|
||||
|
||||
**The other runtimes (parity).**
|
||||
- **TS** (`cultmesh-ts/src/index.ts:5891-5914` at `e4e2a0b9`):
|
||||
- It acknowledges each packet at receipt (`createAckForReceived`), before the handler runs.
|
||||
- It runs each session's frames through one promise chain (`record.work`), so handling is serial per session.
|
||||
- `onDocumentPutRaw` may return a Promise and an application-level receipt document, which is sent when it
|
||||
resolves (`:5951-5972`).
|
||||
- So TS already "replies later", but at the application level, and its transport ack does not mean handled.
|
||||
- **Python** (`cultmesh-py/src/cultmesh_py/server.py:407`, `_handle_raw_put` `:415`): the handler is synchronous
|
||||
and returns its responses.
|
||||
- Neither has a durability meaning for its ack. Parity is follow-up **F-Q5a** (Q5.8).
|
||||
|
||||
**Consumers of the sink trait** (`CultMeshRudpRawDocumentSink`):
|
||||
- Odin's `SinkHandle` (`main.rs:168-172`).
|
||||
- Ratatoskr's catalog test (`Ratatoskr/crates/ratatoskr-core/tests/catalog.rs:29`).
|
||||
- CultLib's own tests, which use closures through the blanket impl (`rudp_document_server.rs:60-67`).
|
||||
- Vendored copies (StreamPixels, Heimdall) do not implement it.
|
||||
|
||||
**cultcache-rs write outcome (R1 batch 3, `93ac5945`).**
|
||||
- Every failed single-file write carries `CultCacheStoreWriteFailed { kind: NotReplaced | ReplacedNotDurable }`.
|
||||
- Any other error from `compare_exchange_snapshot` (lock, read, decode, validation) comes before the write step,
|
||||
so the file is unchanged.
|
||||
- One hole: `with_exclusive_lock` (`cultcache-rs/src/lib.rs:924-932` on that branch) returns the **unlock** error
|
||||
after a write that succeeded, with no marker. Odin would read that as "not replaced". Its next write would then
|
||||
find its own snapshot and report `ForeignStoreWrite`, and Odin would end.
|
||||
- That is rare (unlock failing), and it belongs to R1: follow-up **F-Q5b**, not this cut.
|
||||
|
||||
### Q5.2 Target shape
|
||||
|
||||
1. **CultLib (cut C5).** A put's sink may answer later.
|
||||
- While any put of a session awaits its answer, the server sends that session nothing new.
|
||||
- The one exception is resends of packets created before the put arrived.
|
||||
- A snapshot request behind the pending put waits in the session's order.
|
||||
- An accepted answer releases one ordinary acknowledgement.
|
||||
- A refusal ends the session with the Error frame the put-serve branch sends.
|
||||
- The wire does not change, and existing publishers keep "acknowledged means the sink accepted", which for Odin
|
||||
is durable (Q5-A).
|
||||
2. **Odin (cut W1).** One writer thread owns the store file after activation.
|
||||
- The loop owns the working set, decides when to write, and answers put replies from the writer's outcomes.
|
||||
- Every put accepted while a write is in flight is covered by the next single write (group commit).
|
||||
- Route challenges and catalog reads never wait on the writer.
|
||||
3. **Odin (cut W0), before W1.**
|
||||
- A pin bump.
|
||||
- S-2: a `served_record` override so Odin's admission is exact.
|
||||
- S-3: the `attempted` list dies, because the typed write outcome says what the file holds.
|
||||
|
||||
### Q5.3 Identity, lifecycle, authority
|
||||
|
||||
| Kind | Named by | Over time | Decided by |
|
||||
|---|---|---|---|
|
||||
| pending put (CultLib) | a server-minted `u64` put id, unique for the server's life, never reused across session generations | created when an admitted put is offered to the sink; ends when it is answered, or when its session ends (the answer is then discarded) | the sink answers; the server alone sends |
|
||||
| put reply handle (CultLib) | the put id, inside a `CultMeshRudpPutReply` | owned by whoever the sink hands it to; consumed by `accept` or `refuse`; dropping it unanswered refuses | the sink's owner |
|
||||
| working-set version (Odin) | a `u64` counter in `MemoryOdinTopologyStore` | +1 on every mutation that changes a record; never reset within an activation | the loop |
|
||||
| written-through version (Odin) | a `u64` in `RuntimeState` | raised only by a writer outcome `Written { version }` | the writer's outcome, applied by the loop |
|
||||
| write request (Odin) | its version | at most one in flight; replaced by nothing (the next request is built when the outcome arrives) | the loop submits; the writer executes |
|
||||
| what the file holds (Odin, `flushed`) | none: one snapshot | set at activation from the load; replaced on `Ok(true)` and on `ReplacedNotDurable` | the writer alone |
|
||||
|
||||
"Dirty" stops being a flag (`lib.rs:502`, `dirty: Cell<bool>`): it is derived as `version > written_through`.
|
||||
|
||||
### Q5.4 Cuts
|
||||
|
||||
#### Cut C5. A put's sink may answer later (CultLib, cultmesh-rs, behaviour)
|
||||
|
||||
- **Repo/branch:** CultLib `hands/cultmesh-put-reply-later` from `main` after the put-serve branch merges (Q5.7).
|
||||
Anchors below are at put-serve `e4e2a0b9`; they move by Cut D's diff (Q5.7 names the conflicts).
|
||||
- **Deletes first:** the synchronous sink call and its error mapping in the put arm, `rudp_document_server.rs:621-633`
|
||||
(they are replaced, not wrapped). Nothing else is deleted.
|
||||
- **Public API** (default D-1 below):
|
||||
|
||||
```rust
|
||||
pub trait CultMeshRudpRawDocumentSink {
|
||||
/// Answer through `reply` now or later, from any thread.
|
||||
fn accept_raw_document(&mut self, receipt: CultMeshRudpRawDocumentReceipt, reply: CultMeshRudpPutReply);
|
||||
}
|
||||
// Closures keep working and answer at once:
|
||||
impl<F: FnMut(CultMeshRudpRawDocumentReceipt) -> Result<()>> CultMeshRudpRawDocumentSink for F { .. }
|
||||
|
||||
#[must_use] pub struct CultMeshRudpPutReply { /* put id, Sender to the server */ } // Send, !Clone
|
||||
impl CultMeshRudpPutReply {
|
||||
pub fn accept(self);
|
||||
pub fn refuse(self, reason: impl std::fmt::Display);
|
||||
}
|
||||
// Drop without an answer refuses: "the put was not answered".
|
||||
```
|
||||
|
||||
- `CultMeshRudpRawDocumentReceipt` (`:37-43`) is unchanged; the reply is a separate argument, so the receipt stays
|
||||
`Clone + PartialEq`.
|
||||
- `CultMeshRudpPollOutcome` (`:286-290`) is unchanged. A later refusal is returned as `ApplicationRejected` with
|
||||
`SinkRefused(reason)` from the `poll_once` that sends it.
|
||||
- **Server changes, `rudp_document_server.rs`:**
|
||||
- `CultMeshRudpDocumentServer` (`:303-311`) gains:
|
||||
- The answer channel (`std::sync::mpsc`: the server keeps the receiver, each reply holds a sender).
|
||||
- `answered: VecDeque`, the answers taken off the channel and not yet handled.
|
||||
- A next put id.
|
||||
- `pending: BTreeMap<u64, CultMeshRudpSessionKey>`.
|
||||
- `SessionEntry` (`:292-297`) gains `pending_puts: BTreeSet<u64>` and `waiting: VecDeque<(u32, CultNetMessage)>`,
|
||||
the delivered frames that may not be answered yet.
|
||||
- **One predicate owns "withheld":** `fn acknowledgement_withheld(&self, key) -> bool`, true while
|
||||
`pending_puts` is non-empty. The S-5 fix, if the ack map takes it here, adds its clause to this function and
|
||||
nowhere else.
|
||||
- `poll_once`:
|
||||
- Drain answers first (before `maintain`, `:366`). Each answer is looked up in `pending`; an unknown id (its
|
||||
session ended) is discarded.
|
||||
- Refuse → `end_rejected_session(key, reason)`, and return one `ApplicationRejected`. Further answers stay in the
|
||||
channel for the next poll.
|
||||
- Accept → remove the id. When the session has none left, it is released: send its ack (the rule for which
|
||||
poll sends it is under the put arm below), then handle `waiting` in order.
|
||||
- `:436-438`: a `result.reply` (Pong) is not sent while withheld.
|
||||
- `:468-476`: the end-of-poll ack is not sent while withheld.
|
||||
- `deliver_application_message` (`:565`):
|
||||
- Put arm (`:572`): admission is unchanged (put-serve `:587-619`). Then mint the id, record it, and call the sink
|
||||
with a reply.
|
||||
- `poll_once` looks at answers again after the delivered-frame loop (`:445-466`), before the end-of-poll ack,
|
||||
but handles only those for this poll's session. The rest stay queued for the next poll's first step. (Answers
|
||||
move from the channel into one server-side queue, so they can be taken selectively.) A sink that answered at
|
||||
once is therefore acknowledged, or refused, in the same poll with the same datagrams as today, and another
|
||||
session's refusal can never cut this session's ack short.
|
||||
- A release sends `create_ack()` from the drain, except for the session whose packet this poll is handling:
|
||||
that session gets its ordinary end-of-poll ack, so a synchronous sink's datagrams stay exactly as they were.
|
||||
- Snapshot arm (`:634`): if the session is withheld, push the request onto `waiting` and return; otherwise
|
||||
answer as today.
|
||||
- A put that arrives while `waiting` is non-empty also queues, so a session's frames are handled in order. A
|
||||
put behind another put that is only pending is offered at once, so pipelined puts can share one write.
|
||||
- Session end, wherever a session is removed (`:441`, `:486-489`, `:523`, `end_refused_session`,
|
||||
`end_rejected_session`, Cut D's `end_unsendable_session`): remove its ids from `pending`, so their answers are
|
||||
discarded.
|
||||
- **Put this in one `remove_session(key)` helper, so no removal path can forget it.** It is the one new helper
|
||||
the cut earns: there are six removal sites today.
|
||||
- Doc comments: the struct doc (`:300-302`, "no background thread") stays true. The sink trait doc says what
|
||||
"acknowledged" means: the sink accepted.
|
||||
- **Authority map:**
|
||||
- Owner of what the peer is told about a put: the server, and only through `poll_once`. The sink decides accept or
|
||||
refuse; it never sends.
|
||||
- Inputs: sink answers (channel); transport receipts.
|
||||
- Outputs: one ack per release; one refusal per refused put's session.
|
||||
- Derived: "withheld" is derived from `pending_puts`; it is not a flag.
|
||||
- Forbidden writers:
|
||||
- No send path to a withheld session except `maintain`'s resends.
|
||||
- The sink must not be able to acknowledge by returning. The return type carries no answer, so this holds by
|
||||
construction.
|
||||
- Shared paths: a synchronous closure sink and a deferred sink answer through the same reply and the same drain.
|
||||
There is no second code path for "answered at once".
|
||||
- **Verification (CultLib, Yggdrasil):**
|
||||
- Wire-level test, the observer layer: a raw client socket records every datagram the server sends to its session.
|
||||
- With the reply held across at least five resend intervals, including a Ping and a snapshot request on that
|
||||
session, no datagram acknowledges the put's sequence.
|
||||
- After `accept()`, the next poll sends an ack that does, and then the snapshot response.
|
||||
- Mutations that must fail it: sending the end-of-poll ack regardless; sending the Pong; answering the snapshot
|
||||
request at once.
|
||||
- Another session's snapshot request is answered while the first session's put is held (the loop is not blocked).
|
||||
- `refuse()` and dropping the reply: the peer receives `cultnet.error.v0` and a goodbye, and nothing acknowledges
|
||||
the put (reuse the put-serve refusal assertions).
|
||||
- Two pipelined puts on one session, each with its own held reply:
|
||||
- Answering the second first releases nothing.
|
||||
- Answering the first then releases one ack that covers both.
|
||||
- This pins that a later put's acknowledgement never leaves before an earlier one's.
|
||||
- Session ends while held (peer Disconnect; a new Connect from the same key): a later `accept()` sends nothing to
|
||||
the new generation, and `pending` is empty.
|
||||
- Closure sink: the datagram sequence is byte-identical to the pre-cut server for accept and refuse. This pins that
|
||||
the synchronous path did not change.
|
||||
- An unservable put is refused before the sink sees it: the sink is never called and no reply is minted.
|
||||
- `cargo mutants --in-diff` on the cut's diff; every survivor is triaged.
|
||||
- **Estimate:** about +170 source, -15 source, +350 tests. No new target, dependency or wire field.
|
||||
|
||||
#### Cut W0. Pin bump, exact admission and the one-candidate store (Odin, subtraction first)
|
||||
|
||||
- **Repo/branch:** Odin `hands/odin-write-w0` from `hands/odin-write` (`434756e`, or batch 4's tip). It bumps the
|
||||
CultLib pin to the main commit holding Cut D, put-serve, R1 batch 3 and C5 (Q5.7).
|
||||
- **Deletes first:**
|
||||
- `attempted` (`lib.rs:494-499`, `:525`), the retry loop over it in `flush` (`:619-625`) and its push (`:628-633`).
|
||||
- The tests that describe a file holding some older failed attempt:
|
||||
- `a_write_that_failed_before_replacing_keeps_every_earlier_attempt_odins` (`lib.rs:2172`).
|
||||
- `the_file_may_hold_a_later_failed_attempt_not_only_the_first` (`:2205`).
|
||||
- Each models a `NotReplaced` failure that nonetheless changed the file, which the typed outcome now rules out.
|
||||
- **Changes:**
|
||||
- `flush` (`lib.rs:612-640`) matches the error's `CultCacheStoreWriteFailed` kind (`downcast_ref`):
|
||||
- `ReplacedNotDurable` → `flushed = current`, and the store stays dirty so the next write re-syncs.
|
||||
- Anything else → no change.
|
||||
- `a_write_that_failed_after_replacing_the_file_is_not_another_writer` (`:2141`) is re-targeted to inject
|
||||
`ReplacedNotDurable` as the real outcome, not a hand-made replace.
|
||||
- `a_failed_write_is_made_by_the_next_without_another_change` (`:2118`) stays.
|
||||
- **S-2:** Odin's `SnapshotHandle` (`main.rs:174-181`) overrides `served_record`. It builds the envelope as
|
||||
`persist_generic_document` (`:1462`) would store it, then serves it through `peer_document` (`:1400`), the
|
||||
function `drop_unservable_documents` (`:1428`) sizes with. Admission and activation then use one sizing.
|
||||
- A runtime-presence put keeps the default (as received). Presences are served in `public_document`'s shape and
|
||||
are never near the bound, and `drop_unservable_documents` does not check them either.
|
||||
- A schema that `persist_generic_document` would refuse returns an error, which the server turns into a
|
||||
refusal before the sink.
|
||||
- `accept_raw_document`'s doc (`main.rs:332-351`) drops its S-5 paragraph; it moves to W1's writer doc.
|
||||
- **Authority map:** what the file holds is decided by the one write outcome, not by a list of guesses.
|
||||
`ForeignStoreWrite` means exactly "a snapshot Odin did not write".
|
||||
- **Verification:**
|
||||
- Test: a `ReplacedNotDurable` failure, then a change, then a write → made over the failed snapshot.
|
||||
- Test: a `NotReplaced` failure, after which another writer puts the failed snapshot into the file → `ForeignStoreWrite`.
|
||||
Mutation: keeping a candidate fails it.
|
||||
- Test: 1,000 failed writes leave nothing retained. This pins S-3; observe the store's retained records, or the
|
||||
absence of the field.
|
||||
- S-2 test: a 1.2 MB put is refused as `DocumentUnservable` and is not in the working set.
|
||||
- Mutation: sizing with the received record instead of `peer_document` survives nothing.
|
||||
- Negative: `rg -n attempted crates/odin-daemon/src` matches nothing.
|
||||
- The pin bump carries ack Cuts 1-3 and D into Odin. The full Odin suite runs on Yggdrasil, and Soul reads this cut
|
||||
as a transport change, not only a subtraction.
|
||||
- **Estimate:** about -150 (including the two tests) / +60.
|
||||
|
||||
#### Cut W1. The writer thread (Odin, behaviour)
|
||||
|
||||
- **Repo/branch:** Odin `hands/odin-writer` from W0.
|
||||
- **Deletes first:**
|
||||
- `MemoryOdinTopologyStore::flush` (`lib.rs:602-640`), its `path` (`:490`) and `flushed` (`:493`) fields, and the
|
||||
`dirty` flag (`:502`, `:600-602`).
|
||||
- `RuntimeState::flush` (`main.rs:317-330`).
|
||||
- The `self.flush()?` in `accept_raw_document` (`main.rs:363`).
|
||||
- `stop`'s direct write (`main.rs:891-898`).
|
||||
- The flush step of `serving_pass` (`main.rs:933-941`).
|
||||
- No path is left on the loop that can write the file. That holds before the writer is added.
|
||||
- **Keeps:**
|
||||
- `MemoryOdinTopologyStore::load` (`lib.rs:509`): the one locked read, at activation, on the loop, before any writer
|
||||
exists. `load` returns the loaded snapshot beside the store, to become the writer's `flushed`.
|
||||
- `ForeignStoreWrite`, `WriteLeaseLost`, `survive` (`main.rs:979-994`), unchanged in meaning.
|
||||
- **Adds, `crates/odin-daemon/src/writer.rs`** (a module, not a crate):
|
||||
- `StoreWriter`: `spawn(file, flushed, lease) -> StoreWriter`, `submit(WriteRequest)`,
|
||||
`try_outcome() -> Option<WriteOutcome>` and `finish(self)`.
|
||||
- The channel is `sync_channel(1)`. The loop submits only when nothing is in flight, so `submit` never blocks.
|
||||
- `WriteRequest { version, records }`.
|
||||
- `WriteOutcome { version, result }`, where `result` is one of `Written`, `Failed(text)`, `Foreign` or
|
||||
`LeaseLost(text)`.
|
||||
- The thread:
|
||||
- Checks the lease (the check extracted from `require_current_write_lease` `main.rs:546-567` into a free
|
||||
function that the loop and the writer share), then `compare_exchange_snapshot(flushed, records)`.
|
||||
- Applies W0's outcome rule to `flushed`, and sends the outcome.
|
||||
- It never touches reply handles, the working set or the socket.
|
||||
- Mock point: the thread writes through a one-method trait `StoreFile { compare_exchange_snapshot }`, implemented
|
||||
for `SingleFileMessagePackBackingStore`. Tests gate it and count it. The lock-hold (Soul's method) is used where
|
||||
the real file must be on the path.
|
||||
- **Changes, `main.rs`:**
|
||||
- `RuntimeState` (`:141-161`) gains `writer: Option<StoreWriter>` (spawned in `try_activate` after the load,
|
||||
`:284-305`), `written_through: u64`, `in_flight: Option<u64>` and `replies: VecDeque<(u64, CultMeshRudpPutReply)>`.
|
||||
- `SinkHandle::accept_raw_document` (`:168-172`) takes the reply.
|
||||
- `RuntimeState::accept_raw_document` (`:352-367`):
|
||||
- Validates and mutates as today.
|
||||
- Pushes `(store.version(), reply)`, or accepts at once when `version <= written_through` (nothing new to make
|
||||
durable).
|
||||
- Refuses through the reply on a validation error.
|
||||
- Submits if nothing is in flight.
|
||||
- A new `serving_pass` step, in place of `:933-941`, run every pass:
|
||||
- `try_outcome`. On `Written { v }`: `written_through = v`, and accept every reply with version `<= v`, in queue
|
||||
order.
|
||||
- On `Failed`: refuse the replies with version `<= v`; they remain in the working set (refusal runs one way, as
|
||||
documented at `:344-350`).
|
||||
- On `Foreign` / `LeaseLost`: return the error to `survive`, which ends Odin.
|
||||
- Then, if nothing is in flight and `version > written_through`, submit when a reply is waiting or when
|
||||
`FLUSH_INTERVAL` is due.
|
||||
- So puts never wait for the interval; bookkeeping-only changes do.
|
||||
- `stop` (`:894`):
|
||||
- If the lease is still current and anything is unwritten or in flight, wait for the in-flight outcome, submit the
|
||||
rest, and wait for that outcome too. This is a blocking wait; it is the only one, and it happens only on the
|
||||
way out.
|
||||
- Answer the replies, run one `poll_server` so the acks leave, then `finish` (drop the sender and join the
|
||||
thread).
|
||||
- After `WriteLeaseLost` or `ForeignStoreWrite`, write nothing. Dropping the replies refuses them.
|
||||
- `FLUSH_INTERVAL`'s comment (`:73-77`) and the store's doc (`lib.rs:484-488`) say: the writer thread owns the
|
||||
file; puts are covered by the next write; bookkeeping waits for the interval.
|
||||
- **Authority map:**
|
||||
- Owner of the store file after activation: the writer thread, the only holder of a
|
||||
`SingleFileMessagePackBackingStore` for the store path. It also owns what the file holds (`flushed`).
|
||||
- Owner of the working set, of when to write, and of put replies: the loop (`RuntimeState`).
|
||||
- Inputs:
|
||||
- Writer: one request at a time, and the lease file.
|
||||
- Loop: provider puts, Idunn's projection, the lease file, and writer outcomes.
|
||||
- Outputs: one atomic `.cc` snapshot per request; one outcome per request; replies answered in version order.
|
||||
- Derived state:
|
||||
- `dirty` is derived (`version > written_through`).
|
||||
- `flushed` is no longer the loop's; it lives only in the writer.
|
||||
- The interval decides only bookkeeping-only writes.
|
||||
- Forbidden writers: no loop code path calls `compare_exchange_snapshot`, takes the store's lock, or answers a
|
||||
put reply before the outcome covering its version.
|
||||
- Shared paths: peer puts, self-presence, refresh, activation-time drops and the stop write all mutate the working
|
||||
set, and reach the file only through `submit`.
|
||||
- Deletion line: every loop-side write (above) is deleted before `writer.rs` is added.
|
||||
- **Batching rule, stated once.** The loop keeps at most one write in flight. Everything that changed while it was in
|
||||
flight, puts and bookkeeping alike, goes into the next request as one snapshot of the working set. N puts accepted
|
||||
during one write cost one more write and one fsync pair, not N.
|
||||
- **Backpressure** (Q5-B, recommended: hold):
|
||||
- A stalled writer holds put replies. The loop keeps serving, and the working set keeps admitting (the catalog
|
||||
serves admitted-not-durable puts, as it does today after a failed write).
|
||||
- The queue is bounded by the server's existing limits: 64 sessions and 32 MiB of admitted payload. It adds no new
|
||||
bound.
|
||||
- A publisher that stops waiting (the Rust default is 300 ms) disconnects. Its session ends, its answer is
|
||||
discarded, and its put is still written by the next write. That is what a publisher sees today during a stall,
|
||||
minus the stalled challenges.
|
||||
- **Ordering.**
|
||||
- Replies are answered in version order from one queue, and the server sends acks in answer order.
|
||||
- Within a session, C5 releases one ack only when every pending put of the session is answered.
|
||||
- So no later put's reply leaves before an earlier one's.
|
||||
- **Crash mid-batch.**
|
||||
- The write is one atomic replace, so the file holds the old snapshot or the new one.
|
||||
- No reply of a batch is answered before its `Written` outcome, so a SIGKILL anywhere loses no acknowledged put.
|
||||
- A crash after the rename but before the directory sync is `ReplacedNotDurable` if the process lives; if it dies,
|
||||
nothing of that batch was acknowledged.
|
||||
- **Activation.** The store is read once, by `load`, before the writer is spawned. While the writer lives, the loop
|
||||
never reads the file (Cut 1's rule, unchanged).
|
||||
- **Verification (Odin, Yggdrasil):**
|
||||
- **Loop latency under a stall.** This pins Q5's ruling; it is Soul's S-3 probe and O1, inverted into assertions.
|
||||
- Hold the store's sibling lock for 1,500 ms, with a provider put in flight and Odin's heartbeat dirty with the
|
||||
interval due.
|
||||
- A route challenge and a catalog read are each served within 200 ms. The put's publisher sees no ack until the
|
||||
lock is released, and then does, with the put on disk.
|
||||
- Mutation that must fail it: running the write synchronously on the loop, by calling the `StoreFile` directly
|
||||
from `serving_pass`.
|
||||
- **Group commit.** Gated `StoreFile`, held on the first write, with 10 puts from 10 publishers accepted meanwhile.
|
||||
- After release, exactly 2 writes are made and all 10 publishers are acknowledged.
|
||||
- Mutation: one request per put gives 11 writes.
|
||||
- **Ordering.** Gated `StoreFile`; put A's write is held and put B arrives.
|
||||
- B's publisher is not acknowledged before A's; the answer order is observed at the server's answer channel.
|
||||
- Mutation: answering the queue from the back.
|
||||
- **Durability.** `daemon_process.rs`, extending `an_acknowledged_put_survives_sigkill` (`:326`).
|
||||
- 20 rounds, each a stream of puts from 4 publishers while the test holds and releases the store's lock at random
|
||||
offsets. SIGKILL at a random offset.
|
||||
- Every put its publisher saw acknowledged is in the file after the kill.
|
||||
- Mutation: accepting replies at submit instead of at `Written`.
|
||||
- **One writer.** With the writer's `StoreFile` gated shut, N serving passes, a heartbeat and a stop request make no
|
||||
write from the loop: the file's identity (`file_identity`, `main.rs:3343`) is unchanged until the gate opens.
|
||||
- Negative grep: `rg -n "compare_exchange_snapshot|pull_all\(" crates/odin-daemon/src` matches only `writer.rs`,
|
||||
`load` and the tests.
|
||||
- **Lease lost with a write queued.** No write is made, Odin ends, and the queued replies are refused. The file's
|
||||
identity is unchanged.
|
||||
- **Foreign write.** Odin ends without writing over it (existing `:3696`, re-targeted through the writer).
|
||||
- **Stop.** SIGTERM with a put held behind the lock: after the release Odin writes, the publisher is acknowledged,
|
||||
and it exits 0. With the lease lost, it writes nothing.
|
||||
- Re-targeted, not deleted: `changes_inside_one_interval_are_one_write` (`:3356`, now bookkeeping-only),
|
||||
`a_put_is_written_before_it_is_accepted` (`:3389`, observed at the publisher's ack),
|
||||
`a_put_whose_write_is_not_made_is_refused` (`:3516`), and the `ending_*` tests (`:3728-3800`).
|
||||
- `cargo mutants --in-diff` on the cut's diff.
|
||||
- **Estimate:** about -120 / +230 source, +300 tests. One new module, one thread, no new crate, dependency or target.
|
||||
|
||||
### Q5.5 What stays on the loop, and what does not
|
||||
|
||||
| Path | Before (`434756e`) | After W1 |
|
||||
|---|---|---|
|
||||
| route challenge (`raw_snapshot`, exact self query) | lease-file read, sign; blocked behind any write on the loop | the same reads; never behind a write |
|
||||
| catalog read (`stored_snapshot`) | working set, projection read | unchanged; never behind a write |
|
||||
| peer put | mutate, then write and fsync on the loop | mutate, queue the reply, submit; the reply is sent after `Written` |
|
||||
| heartbeat, refresh | mutate; the interval write on the loop | mutate; the interval write on the writer |
|
||||
| store lock | taken by `flush` on the loop | taken only by `load` (activation) and the writer |
|
||||
|
||||
### Q5.6 What only Yggdrasil can show
|
||||
|
||||
- **Rerun Soul's ABBA** (`soul-c3/latency_cmd.txt`: `soul_loop_stall_under_live_put_rate`, base against cut, in
|
||||
interleaved rounds, with `/proc/pressure/io` before and after each).
|
||||
- The base is `434756e`; the cut is W1's tip. Expect route-challenge p99 to stop tracking IO pressure, while put
|
||||
round trip p99 still does.
|
||||
- Soul's figures at `434756e` and its base over 8 rounds, under 5-38% pressure: challenge p99 310-2,129 ms, put
|
||||
round-trip p99 1,121-2,593 ms (`soul-c3/latency.log`).
|
||||
- Whether the loop's remaining disk reads (the lease and projection files) ever stall under real pressure. The ABBA
|
||||
run shows it as challenge p99 outliers that coincide with pressure; if they appear, open the lease-in-memory
|
||||
follow-up.
|
||||
- After deploy, section 6's script, plus the count of "admitted odin route continuity: timed out" per hour.
|
||||
|
||||
### Q5.7 Sequencing (recommended order)
|
||||
|
||||
1. **Cut D merges to CultLib main** (already slated first).
|
||||
2. **The put-serve branch rebases onto it and merges.** The conflicts:
|
||||
- Cut D changes `send_packet` to return `Option<io::Error>` and adds `end_unsendable_session`.
|
||||
- Its snapshot-send failure builds `reason: format!(..)`, where the put-serve branch made `reason` a typed
|
||||
`CultMeshRudpRejectionReason`. It needs a variant (for example `ResponseSendFailed(String)`), and the put-serve
|
||||
Hands owns it.
|
||||
- Both branches edit `poll_once`'s ack site (`:468-476`) and `end_rejected_session`.
|
||||
3. **R1 batch 3 merges** (cultcache-rs only; no conflict with 1-2).
|
||||
4. **C5 is cut from that main** (it rewrites the lines 1-2 touched, so it goes after them), then Soul, then merge.
|
||||
5. **One Odin pin bump to that commit, as W0**, carrying S-2, S-3 and the transport cuts. Then **W1** on top.
|
||||
|
||||
W0 may start on the commit after step 3 if C5 slips, at the cost of a second bump before W1. One bump is recommended:
|
||||
each bump moves Odin across ack Cuts 1, 1b, 1c, 3 and D, and a single Soul pass over that transport delta is cheaper
|
||||
than two.
|
||||
|
||||
### Q5.8 Follow-ups this map does not own
|
||||
|
||||
- **F-Q5a (parity).** TS and Python acknowledge a put at receipt, and TS's reply-later is an application receipt.
|
||||
- Either they adopt C5's rule ("acknowledged means the sink accepted"), or the QUIC campaign defines an
|
||||
application-level put reply for every runtime.
|
||||
- QUIC has no transport ack an application can read, so it needs one anyway. Recommended owner: the QUIC campaign.
|
||||
Rust C5 is then the RUDP-era contract, and TS's receipt is the seed of the QUIC shape.
|
||||
- **F-Q5b (R1).** `with_exclusive_lock` returns an unlock error after a successful write, with no write-outcome
|
||||
marker (`cultcache-rs/src/lib.rs:924-932` at `93ac5945`). Odin would call its own next write foreign. Fix it in
|
||||
cultcache-rs: tag it, or ignore the unlock error, since dropping the file releases the lock.
|
||||
- **S-5 interface.** C5 adds `acknowledgement_withheld` as the one place that decides "send no ack to this session".
|
||||
The ack map's S-5 decision (probe P1) plugs in there as a second clause if it is fixed in the document server. C5
|
||||
does not fix S-5.
|
||||
|
||||
### Q5.9 Subtraction ledger (estimate)
|
||||
|
||||
| Cut | Removed | Added | Targets, dependencies, wire |
|
||||
|---|---|---|---|
|
||||
| C5 (CultLib) | ~15 | ~170 source, ~350 tests | none; sink trait signature changes (Odin, Ratatoskr test) |
|
||||
| W0 (Odin) | ~150 (the `attempted` list and 2 tests) | ~60 | pin bump |
|
||||
| W1 (Odin) | ~120 (every loop-side write) | ~230 source, ~300 tests | one module, one thread |
|
||||
|
||||
Net source is positive: about +175. It buys the ruled capability (challenges and reads independent of the disk, with
|
||||
Q3 B kept), and it removes S-3's unbounded list.
|
||||
|
||||
### Q5.10 Operator questions and defaults
|
||||
|
||||
- **Q5-A. What does a put's reply mean on the RUDP wire once it can come later?**
|
||||
- A: the document server withholds the transport acknowledgement until the sink answers. Nothing created after the
|
||||
put is sent to that session until then. The wire is unchanged, and every existing publisher (Rust
|
||||
`publish_cultnet_message_to_rudp_catalog`, TS and Python publishers) keeps "acknowledged means accepted", which
|
||||
for Odin is durable.
|
||||
- B: a new CultNet message, a put result, sent after durability; the transport ack means received. This is a wire
|
||||
change in the C# reference and four runtimes. Every Odin publisher that waits for the transport ack today must
|
||||
be changed to wait for it; until then Q3 B silently weakens to "received".
|
||||
- **Recommended: A.** It keeps Q3 B for every publisher with a server-only change, and RUDP is frozen except for
|
||||
crash and data-loss fixes. B is the right shape under QUIC (F-Q5a), not a reason to change the RUDP wire now.
|
||||
- What depends on it: C5's whole design.
|
||||
- A also narrows the ack map's Q-A2 meaning for the document server: acknowledged means delivered *and answered*.
|
||||
That was already true of the synchronous sink.
|
||||
- **Q5-B. A writer stalled for seconds: hold put replies, or refuse with a typed error?**
|
||||
- A: hold. Replies wait, bounded by the server's session and payload limits. Publishers time out on their own
|
||||
clock, and the put is written when the disk returns.
|
||||
- B: refuse once more than N replies wait or the oldest waits longer than T. This needs a new `cultnet.error.v0`
|
||||
code (the error-contract map owns the registry) and two tunables.
|
||||
- **Recommended: A.** A refusal ends the publisher's session exactly as its own timeout does, but adds a code and
|
||||
two numbers, and it tells the publisher less than the truth: the put may still become durable. B earns its place
|
||||
only if publishers ever wait without a timeout.
|
||||
- **Default D-1 (Self, overridable). The API is a completion handle** (`CultMeshRudpPutReply`: `accept`, `refuse`,
|
||||
and Drop refuses).
|
||||
- The alternatives were a token with `server.answer_put(token, result)`, and a reply method keyed by session and
|
||||
message id.
|
||||
- Keying by session and message id is ambiguous: message ids are publisher-chosen, and a reconnect reuses the
|
||||
session key.
|
||||
- A token cannot be answered from another thread without routing it back to the server, and a forgotten token
|
||||
leaves a session withheld until it ends.
|
||||
- The handle is the Rust expression of TS's Promise-returning `onDocumentPutRaw`, and it keeps closure sinks
|
||||
source-compatible.
|
||||
|
||||
## 8. Related findings
|
||||
|
||||
|
|
@ -381,3 +986,23 @@ records; update the check's type list in the same pass.
|
|||
- **F7** `accept_raw_document` (`main.rs:269-280`) checks `activated()` but not the *current* lease before
|
||||
mutating. Cut 2 moves the lease check to the only write, which closes this for the file. Soul should confirm
|
||||
that nothing else acts on a lost lease.
|
||||
|
||||
## C5 status (2026-10-01)
|
||||
|
||||
**Hands done, Soul not yet run.** Drain mode: Soul waits for the operator's go.
|
||||
|
||||
- **Branch:** CultLib `hands/cultmesh-deferred-put-reply`, commits `08e0c229..d708247e`, based on `9dcb3aa1`.
|
||||
- **Mechanism:** the sink answers through `CultMeshRudpPutReply` with `accept()` or `refuse(reason)`. Dropping the reply unanswered refuses with `UNANSWERED_PUT_REASON`. `acknowledgement_withheld` is the single owner of the hold: while a put is unanswered, the session gets no end-of-poll ack, Pong, snapshot response or reply to a repeated Connect.
|
||||
- **Verification (Hands):**
|
||||
- cultmesh-rs: 21 unit and 32 integration tests pass. A byte-exact golden test also passes before the cut.
|
||||
- cultnet-rs: 486 pass.
|
||||
- Interop: 27 pass, 0 fail.
|
||||
- cargo-mutants: 57 mutants; 0 missed, 5 timeouts, all in the `take_answers` loop.
|
||||
- **Spec deviations Soul should attack:**
|
||||
- A repeated Connect's ack is now withheld.
|
||||
- Puts queued behind a waiting snapshot count as withheld. Snapshot responses are held in a new `responses_owed` queue: P1 pending, then S, then P2 must not acknowledge P2 early.
|
||||
- Answers are checked after each message.
|
||||
- A release that arrives during an unreliable packet sends its own ack.
|
||||
- **Structural delta:** +406/−96 in `rudp_document_server.rs`, against a +170/−15 estimate.
|
||||
- **Consumers:** Odin `SinkHandle` and Ratatoskr `tests/catalog.rs:29` break until the W0 pin bump.
|
||||
- **Still open:** S-5. Its fix belongs in `acknowledgement_withheld`.
|
||||
|
|
|
|||
Loading…
Reference in a new issue