| full read and decode | 3.24 MB, 0.78 ms | 131 KB, 0.14 ms |
| one presence heartbeat as coded today (read, reserve watermark CAS, correlation CAS) | 16.2 MB read, **6.49 MB written, 54.8 ms** | 663 KB, 265 KB, 14.4 ms |
| the same heartbeat folded into one CAS | 6.49 MB read, 3.24 MB written, 36.5 ms | 268 KB, **134 KB, 7.1 ms** |
| redb (cultcache-rs `RedbMessagePackBackingStore`), one small-row CAS | 33 KB, 42.6 KB, 5.4 ms | 33 KB, 42.6 KB, 8.7 ms |
Odin's serving loop is single-threaded (`main.rs:739-747`), so every millisecond in the table is a
millisecond in which no datagram is read. Five writes per second at 36-55 ms each block the loop for 180-275
ms of every second on a quiet disk, and for longer under pressure. That is the socket drops and the route
challenge timeouts.
### 1.3 What gets written, and why (source, `44951a1`)
Every mutation reads the whole file, then CAS-rewrites the whole file, through CultLib's
`SingleFileMessagePackBackingStore` (`cultcache-rs/src/lib.rs` at `3bf1c0ce`: `compare_exchange` reads all
entries under an exclusive lock, rewrites all of them, then `fsync`s the file and its directory).
| Trigger | Rate (measured from catalog `stored_at` churn over 88 s, and from the publisher's source) | Store ops per event |
|---|---|---|
| Muninn republishes `muninn.obs_stream_catalog`, `muninn.move_hue_program` and `gamecult.media_stream_advertisement` every 2 s (Muninn `main.rs:101`, `:2241`). The payload changes each time: `updated_at` (`:3263`) | about 1.5/s | `persist_generic_document``main.rs:1266-1309`: 1 read, then 1 CAS (read and write) |
| Provider presence heartbeats: raven-muninn about every 2.7 s, Odin 5 s, streampixels-service about 10 s, ghostlight about 12 s | about 0.8/s | `admit_presence`, then `reconcile``lib.rs:771-977`: `store.read` (1 read), `reserve_publisher_sequence``lib.rs:613-647` (read and **write**), `compare_exchange``lib.rs:525-584` (read and **write**) |
| Odin's own heartbeat (`HEARTBEAT_INTERVAL` 5 s, `main.rs:71`) | 0.2/s | the same path, through `publish_self_presence``main.rs:422-427` |
| Provider ads and other documents restamped (voidbot.swarm about every 30 s, and others) | about 0.1/s | `persist_generic_document` |
| Projection refresh every 250 ms (`main.rs:72`, `:770-774`) | 4/s | `incarnation_keys(store)``main.rs:486-489` (1 read), then per incarnation `refresh` (projection read, then `store.read`). A write only when facts change |
| Catalog queries (Hermodr, Idunn, probes) | on request | `stored_snapshot``main.rs:509-543` (1 full read, then a projection read per presence) |
- A correlation embeds the observed presence sequence and hash (`lib.rs:887-895`), so
`same_correlation_facts` (`lib.rs:1273-1282`) never dedupes a heartbeat. **Every heartbeat costs two
whole-store writes**: first the watermark, then the correlation.
- About 2.1 of the 5.0 writes per second are accounted for by Muninn and the other catalog documents, and about
2.0 by heartbeats. The remaining 0.9 per second are refresh-driven correlation changes (dependency freshness
windows, O2 in the Idunn audit) plus writes that were not attributed. The two 1.5 MB records are not among the
catalog churn seen through the probe; their publisher and rate are unknown (Q2).
- Reads: about 30 whole-store decodes per second. Most are the 250 ms refresh (1 + N incarnations per pass),
and the rest are admission reads.
### 1.4 What is in the store: truth, cache or projection?
| Kind | Who owns the truth | Odin's role | Who reads Odin's copy | Must it be durable? |
|---|---|---|---|---|
| Provider catalog documents (ads, boundaries, profiles, media ads, schema catalog, surface_state) | the provider | relay cache | the Verse, via the catalog | yes, but not per write. Some publishers put once and never again (for example `heimdall.command_boundary`, whose `stored_at` is from 09-23), so losing the catalog on a restart would lose them. The CultMesh put has no application acknowledgement (`cultmesh-rs rudp_document_server.rs:464-482`), so a put promises the publisher no durability |
| `gamecult.eve.surface` and `surface_state`, 1.55 and 1.56 MB | unknown provider (Q2) | a cache **that can never be served**: a single-record snapshot is over the 1 MiB limit (Odin journal: "snapshot response is 1547739 bytes; limit is 1048576", and 1563575) | nobody can | no. They are 93% of the store's bytes and ride along in every rewrite |
| Runtime presences | the provider (signed) | admitted evidence with anti-replay (O5, O6) | the Verse; Odin's own sequence continuation (`main.rs:1184-1211`) | the latest per incarnation. The 30 s observation window (O6) bounds any replay after a lost write |
| Correlations and the publisher watermark | Odin | a **projection** of presence plus Idunn's projection | Idunn, lockless, from this same file (`--odin-correlation-store /var/lib/gamecult/odin/topology.cc`; Idunn `drivers.rs:5005-5030`) | the watermark must never let a *published* sequence repeat. "Published" means written to this file, because since batch 2 nothing else serves correlations. Both die at Idunn A3 and Odin O-R (`Idunn/docs/route-continuity-cut.md` section 4) |
| Presence history of withdrawn incarnations | nobody | retained on purpose (`withdrawn_expected_removes_current_correlation_but_preserves_presence_history`, `lib.rs:2814`) | only Odin's own sequence continuation reads it, and only Odin's own key | grows without bound with every deploy (follow-up F4) |
## 2. Ends
1. Odin reads its store once, at activation, and never again while it holds the write lease. Its working
state is in memory.
2. Odin writes its store at most once per flush interval, only when the content changed, as one atomic write
under the lock. A foreign write to the file is detected and ends Odin.
3. The store holds nothing Odin cannot serve.
Invariants that must survive:
- One writer: the process-write lease (`main.rs:457-474`) is the only authority to write. Nothing writes after
the lease is lost.
- Idunn's crossing: until A3, Idunn reads correlations from `topology.cc` with
`pull_all_read_only_snapshot`. The path, the single-file `.cc` format and the atomic replace do not change.
- Correlation sequences never repeat, as published.
- One bad record never stops Odin. This now holds at load: records are skipped and logged, never fatal
(`dc46be2`).
- Presence anti-replay within an activation (O5).
## 3. Identity, lifecycle, authority
| Persistent kind | Named by | Over time | Decided by |
|---|---|---|---|
| catalog document | `(type, key)`, type from the schema id (`main.rs:1311-1326`) | replaced by each provider put; never deleted | the provider decides the content; Odin decides admission (allow-list `main.rs:1237-1264`, plus the size rule in Cut 3) |
| presence | `presence_store_key(incarnation, signer)``lib.rs:1528` | replaced each heartbeat; kept after withdrawal (F4) | provider bytes, Odin's admission |
| correlation | incarnation key | replaced when facts change; deleted on withdrawal (`lib.rs:586-611`) | Odin; dies at O-R |
| watermark | target | only ever increases | Odin; dies at O-R |
| in-memory working set (new) | the same keys | loaded at activation, mutated per event, never reloaded while serving | Odin under the lease |
| flush (new) | none: one atomic snapshot | at most one per interval when dirty; one final flush on stop while the lease is held | Odin's serving loop |
## 4. Cuts
Subtraction cuts come before behaviour cuts. Each cut is its own branch and Soul pass. All verification runs on
Yggdrasil through the stopgap (`cargo test --locked -p odin-daemon` from the Odin root).
### Cut 1. The file stops being working memory (Odin, subtraction)
- **Repo/branch:** Odin `hands/odin-memory-store` from `44951a1`. Depends on nothing.
- **First:** a 60 s `/proc/<pid>/io` sample of live Odin, plus the `ss -m` drop counter (section 6 script).
The numbers in 1.1 serve as the baseline if they are taken within a day.
- **Deletes first:**
-`lib.rs:432-648``CultCacheOdinTopologyStore` (about 215 lines): every `pull_all_read_only_snapshot` and
CAS against the store file, and its four `CAS_ATTEMPTS` retry loops (`:453`, `:588`, `:614`, and the one
in `reconcile` at `:799`). With one writer under the lease, an in-memory compare-exchange cannot lose, so
the retry loops protect nothing.
-`main.rs:1266-1309``persist_generic_document`'s file CAS loop.
-`main.rs:486-489`: the store half of `incarnation_keys` (the projection half stays).
- **So today the put's reply is the transport acknowledgement.** That is how Q3 B's "an ACK means durable" holds:
the sink writes before it returns.
**Every packet a session sends carries acknowledgement fields** (`ack`, `ack_mask`, from `ack_state`, cultnet-rs
`rudp.rs` at main).
- The fields report every *received* reliable sequence, not only delivered ones.
-`create_packet` stamps them on each new packet: data, Pong, the end-of-poll Ack.
- A reliable packet is stored with the fields it was created with, and resent unchanged (`track_reliable`,
`due_resends`).
Probes P1-P4 (cultnet-rs sessions at `e4e2a0b9`, scratch test `imag_q5_probe.rs`, Yggdrasil, `imag-q5-cultlib@8dcfe36a`):
| # | Case | Result |
|---|---|---|
| P1 | two pipelined ordered puts, the first lost | the second is delivered to nobody, yet `create_ack` acknowledges it. **S-5 is still open after ack Cuts 1-1c.**`ack_state` at main is the same code, so the ack map's to-do resolves to "not closed by the pin bump" |
| P2 | a put delivered, then a Ping | the Pong acknowledges the put |
| P3 | the resend of a reply created **before** the put | does not acknowledge it |
| P3 | a reliable reply created **after** the put | acknowledges it, and so does its resend |
| P4 | a retransmit of a delivered put | not delivered again (the server's end-of-poll ack still answers it, `:468`) |
Consequence for a reply sent later: holding back only the end-of-poll ack is not enough.
- A Pong, or a snapshot response created on the same session after the put arrived, acknowledges the put too.
- The snapshot response keeps doing so on every resend, even if its first copy were patched.
- So a session with a put awaiting its answer must be **sent nothing created after that put arrived**. Resends of
earlier packets are safe: they carry the fields they were created with.
**The other runtimes (parity).**
- **TS** (`cultmesh-ts/src/index.ts:5891-5914` at `e4e2a0b9`):
- It acknowledges each packet at receipt (`createAckForReceived`), before the handler runs.
- It runs each session's frames through one promise chain (`record.work`), so handling is serial per session.
-`onDocumentPutRaw` may return a Promise and an application-level receipt document, which is sent when it
resolves (`:5951-5972`).
- So TS already "replies later", but at the application level, and its transport ack does not mean handled.
- **Python** (`cultmesh-py/src/cultmesh_py/server.py:407`, `_handle_raw_put``:415`): the handler is synchronous
and returns its responses.
- Neither has a durability meaning for its ack. Parity is follow-up **F-Q5a** (Q5.8).
**Consumers of the sink trait** (`CultMeshRudpRawDocumentSink`):
- Odin's `SinkHandle` (`main.rs:168-172`).
- Ratatoskr's catalog test (`Ratatoskr/crates/ratatoskr-core/tests/catalog.rs:29`).
- CultLib's own tests, which use closures through the blanket impl (`rudp_document_server.rs:60-67`).
- Vendored copies (StreamPixels, Heimdall) do not implement it.
- Any other error from `compare_exchange_snapshot` (lock, read, decode, validation) comes before the write step,
so the file is unchanged.
- One hole: `with_exclusive_lock` (`cultcache-rs/src/lib.rs:924-932` on that branch) returns the **unlock** error
after a write that succeeded, with no marker. Odin would read that as "not replaced". Its next write would then
find its own snapshot and report `ForeignStoreWrite`, and Odin would end.
- That is rare (unlock failing), and it belongs to R1: follow-up **F-Q5b**, not this cut.
### Q5.2 Target shape
1.**CultLib (cut C5).** A put's sink may answer later.
- While any put of a session awaits its answer, the server sends that session nothing new.
- The one exception is resends of packets created before the put arrived.
- A snapshot request behind the pending put waits in the session's order.
- An accepted answer releases one ordinary acknowledgement.
- A refusal ends the session with the Error frame the put-serve branch sends.
- The wire does not change, and existing publishers keep "acknowledged means the sink accepted", which for Odin
is durable (Q5-A).
2.**Odin (cut W1).** One writer thread owns the store file after activation.
- The loop owns the working set, decides when to write, and answers put replies from the writer's outcomes.
- Every put accepted while a write is in flight is covered by the next single write (group commit).
- Route challenges and catalog reads never wait on the writer.
3.**Odin (cut W0), before W1.**
- A pin bump.
- S-2: a `served_record` override so Odin's admission is exact.
- S-3: the `attempted` list dies, because the typed write outcome says what the file holds.
### Q5.3 Identity, lifecycle, authority
| Kind | Named by | Over time | Decided by |
|---|---|---|---|
| pending put (CultLib) | a server-minted `u64` put id, unique for the server's life, never reused across session generations | created when an admitted put is offered to the sink; ends when it is answered, or when its session ends (the answer is then discarded) | the sink answers; the server alone sends |
| put reply handle (CultLib) | the put id, inside a `CultMeshRudpPutReply` | owned by whoever the sink hands it to; consumed by `accept` or `refuse`; dropping it unanswered refuses | the sink's owner |
| working-set version (Odin) | a `u64` counter in `MemoryOdinTopologyStore` | +1 on every mutation that changes a record; never reset within an activation | the loop |
| written-through version (Odin) | a `u64` in `RuntimeState` | raised only by a writer outcome `Written { version }` | the writer's outcome, applied by the loop |
| write request (Odin) | its version | at most one in flight; replaced by nothing (the next request is built when the outcome arrives) | the loop submits; the writer executes |
| what the file holds (Odin, `flushed`) | none: one snapshot | set at activation from the load; replaced on `Ok(true)` and on `ReplacedNotDurable` | the writer alone |
"Dirty" stops being a flag (`lib.rs:502`, `dirty: Cell<bool>`): it is derived as `version > written_through`.
### Q5.4 Cuts
#### Cut C5. A put's sink may answer later (CultLib, cultmesh-rs, behaviour)
- **Repo/branch:** CultLib `hands/cultmesh-put-reply-later` from `main` after the put-serve branch merges (Q5.7).
Anchors below are at put-serve `e4e2a0b9`; they move by Cut D's diff (Q5.7 names the conflicts).
- **Deletes first:** the synchronous sink call and its error mapping in the put arm, `rudp_document_server.rs:621-633`
(they are replaced, not wrapped). Nothing else is deleted.
- **Public API** (default D-1 below):
```rust
pub trait CultMeshRudpRawDocumentSink {
/// Answer through `reply` now or later, from any thread.
- 20 rounds, each a stream of puts from 4 publishers while the test holds and releases the store's lock at random
offsets. SIGKILL at a random offset.
- Every put its publisher saw acknowledged is in the file after the kill.
- Mutation: accepting replies at submit instead of at `Written`.
- **One writer.** With the writer's `StoreFile` gated shut, N serving passes, a heartbeat and a stop request make no
write from the loop: the file's identity (`file_identity`, `main.rs:3343`) is unchanged until the gate opens.
- Negative grep: `rg -n "compare_exchange_snapshot|pull_all\(" crates/odin-daemon/src` matches only `writer.rs`,
`load` and the tests.
- **Lease lost with a write queued.** No write is made, Odin ends, and the queued replies are refused. The file's
identity is unchanged.
- **Foreign write.** Odin ends without writing over it (existing `:3696`, re-targeted through the writer).
- **Stop.** SIGTERM with a put held behind the lock: after the release Odin writes, the publisher is acknowledged,
and it exits 0. With the lease lost, it writes nothing.
- Re-targeted, not deleted: `changes_inside_one_interval_are_one_write` (`:3356`, now bookkeeping-only),
`a_put_is_written_before_it_is_accepted` (`:3389`, observed at the publisher's ack),
`a_put_whose_write_is_not_made_is_refused` (`:3516`), and the `ending_*` tests (`:3728-3800`).
-`cargo mutants --in-diff` on the cut's diff.
- **Estimate:** about -120 / +230 source, +300 tests. One new module, one thread, no new crate, dependency or target.
### Q5.5 What stays on the loop, and what does not
| Path | Before (`434756e`) | After W1 |
|---|---|---|
| route challenge (`raw_snapshot`, exact self query) | lease-file read, sign; blocked behind any write on the loop | the same reads; never behind a write |
| catalog read (`stored_snapshot`) | working set, projection read | unchanged; never behind a write |
| peer put | mutate, then write and fsync on the loop | mutate, queue the reply, submit; the reply is sent after `Written` |
| heartbeat, refresh | mutate; the interval write on the loop | mutate; the interval write on the writer |
| store lock | taken by `flush` on the loop | taken only by `load` (activation) and the writer |
### Q5.6 What only Yggdrasil can show
- **Rerun Soul's ABBA** (`soul-c3/latency_cmd.txt`: `soul_loop_stall_under_live_put_rate`, base against cut, in
interleaved rounds, with `/proc/pressure/io` before and after each).
- The base is `434756e`; the cut is W1's tip. Expect route-challenge p99 to stop tracking IO pressure, while put
round trip p99 still does.
- Soul's figures at `434756e` and its base over 8 rounds, under 5-38% pressure: challenge p99 310-2,129 ms, put
round-trip p99 1,121-2,593 ms (`soul-c3/latency.log`).
- Whether the loop's remaining disk reads (the lease and projection files) ever stall under real pressure. The ABBA
run shows it as challenge p99 outliers that coincide with pressure; if they appear, open the lease-in-memory
follow-up.
- After deploy, section 6's script, plus the count of "admitted odin route continuity: timed out" per hour.
### Q5.7 Sequencing (recommended order)
1.**Cut D merges to CultLib main** (already slated first).
2.**The put-serve branch rebases onto it and merges.** The conflicts:
- Cut D changes `send_packet` to return `Option<io::Error>` and adds `end_unsendable_session`.
- Its snapshot-send failure builds `reason: format!(..)`, where the put-serve branch made `reason` a typed
`CultMeshRudpRejectionReason`. It needs a variant (for example `ResponseSendFailed(String)`), and the put-serve
Hands owns it.
- Both branches edit `poll_once`'s ack site (`:468-476`) and `end_rejected_session`.
3.**R1 batch 3 merges** (cultcache-rs only; no conflict with 1-2).
4.**C5 is cut from that main** (it rewrites the lines 1-2 touched, so it goes after them), then Soul, then merge.
5.**One Odin pin bump to that commit, as W0**, carrying S-2, S-3 and the transport cuts. Then **W1** on top.
W0 may start on the commit after step 3 if C5 slips, at the cost of a second bump before W1. One bump is recommended:
each bump moves Odin across ack Cuts 1, 1b, 1c, 3 and D, and a single Soul pass over that transport delta is cheaper
than two.
### Q5.8 Follow-ups this map does not own
- **F-Q5a (parity).** TS and Python acknowledge a put at receipt, and TS's reply-later is an application receipt.
- Either they adopt C5's rule ("acknowledged means the sink accepted"), or the QUIC campaign defines an
application-level put reply for every runtime.
- QUIC has no transport ack an application can read, so it needs one anyway. Recommended owner: the QUIC campaign.
Rust C5 is then the RUDP-era contract, and TS's receipt is the seed of the QUIC shape.
- **F-Q5b (R1).** `with_exclusive_lock` returns an unlock error after a successful write, with no write-outcome
marker (`cultcache-rs/src/lib.rs:924-932` at `93ac5945`). Odin would call its own next write foreign. Fix it in
cultcache-rs: tag it, or ignore the unlock error, since dropping the file releases the lock.
- **S-5 interface.** C5 adds `acknowledgement_withheld` as the one place that decides "send no ack to this session".
The ack map's S-5 decision (probe P1) plugs in there as a second clause if it is fixed in the document server. C5
**Hands done, Soul not yet run.** Drain mode: Soul waits for the operator's go.
- **Branch:** CultLib `hands/cultmesh-deferred-put-reply`, commits `08e0c229..d708247e`, based on `9dcb3aa1`.
- **Mechanism:** the sink answers through `CultMeshRudpPutReply` with `accept()` or `refuse(reason)`. Dropping the reply unanswered refuses with `UNANSWERED_PUT_REASON`. `acknowledgement_withheld` is the single owner of the hold: while a put is unanswered, the session gets no end-of-poll ack, Pong, snapshot response or reply to a repeated Connect.
- **Verification (Hands):**
- cultmesh-rs: 21 unit and 32 integration tests pass. A byte-exact golden test also passes before the cut.
- cultnet-rs: 486 pass.
- Interop: 27 pass, 0 fail.
- cargo-mutants: 57 mutants; 0 missed, 5 timeouts, all in the `take_answers` loop.
- **Spec deviations Soul should attack:**
- A repeated Connect's ack is now withheld.
- Puts queued behind a waiting snapshot count as withheld. Snapshot responses are held in a new `responses_owed` queue: P1 pending, then S, then P2 must not acknowledge P2 early.
- Answers are checked after each message.
- A release that arrives during an unreliable packet sends its own ack.
- **Structural delta:** +406/−96 in `rudp_document_server.rs`, against a +170/−15 estimate.
- **Consumers:** Odin `SinkHandle` and Ratatoskr `tests/catalog.rs:29` break until the W0 pin bump.
- **Still open:** S-5. Its fix belongs in `acknowledgement_withheld`.
