| full read and decode | 3.24 MB, 0.78 ms | 131 KB, 0.14 ms |
| one presence heartbeat as coded today (read, reserve watermark CAS, correlation CAS) | 16.2 MB read, **6.49 MB written, 54.8 ms** | 663 KB, 265 KB, 14.4 ms |
| the same heartbeat folded into one CAS | 6.49 MB read, 3.24 MB written, 36.5 ms | 268 KB, **134 KB, 7.1 ms** |
| redb (cultcache-rs `RedbMessagePackBackingStore`), one small-row CAS | 33 KB, 42.6 KB, 5.4 ms | 33 KB, 42.6 KB, 8.7 ms |
Odin's serving loop is single-threaded (`main.rs:739-747`), so every millisecond in the table is a
millisecond in which no datagram is read. Five writes per second at 36-55 ms each block the loop for 180-275
ms of every second on a quiet disk, and for longer under pressure. That is the socket drops and the route
challenge timeouts.
### 1.3 What gets written, and why (source, `44951a1`)
Every mutation reads the whole file, then CAS-rewrites the whole file, through CultLib's
`SingleFileMessagePackBackingStore` (`cultcache-rs/src/lib.rs` at `3bf1c0ce`: `compare_exchange` reads all
entries under an exclusive lock, rewrites all of them, then `fsync`s the file and its directory).
| Trigger | Rate (measured from catalog `stored_at` churn over 88 s, and from the publisher's source) | Store ops per event |
|---|---|---|
| Muninn republishes `muninn.obs_stream_catalog`, `muninn.move_hue_program` and `gamecult.media_stream_advertisement` every 2 s (Muninn `main.rs:101`, `:2241`). The payload changes each time: `updated_at` (`:3263`) | about 1.5/s | `persist_generic_document``main.rs:1266-1309`: 1 read, then 1 CAS (read and write) |
| Provider presence heartbeats: raven-muninn about every 2.7 s, Odin 5 s, streampixels-service about 10 s, ghostlight about 12 s | about 0.8/s | `admit_presence`, then `reconcile``lib.rs:771-977`: `store.read` (1 read), `reserve_publisher_sequence``lib.rs:613-647` (read and **write**), `compare_exchange``lib.rs:525-584` (read and **write**) |
| Odin's own heartbeat (`HEARTBEAT_INTERVAL` 5 s, `main.rs:71`) | 0.2/s | the same path, through `publish_self_presence``main.rs:422-427` |
| Provider ads and other documents restamped (voidbot.swarm about every 30 s, and others) | about 0.1/s | `persist_generic_document` |
| Projection refresh every 250 ms (`main.rs:72`, `:770-774`) | 4/s | `incarnation_keys(store)``main.rs:486-489` (1 read), then per incarnation `refresh` (projection read, then `store.read`). A write only when facts change |
| Catalog queries (Hermodr, Idunn, probes) | on request | `stored_snapshot``main.rs:509-543` (1 full read, then a projection read per presence) |
- A correlation embeds the observed presence sequence and hash (`lib.rs:887-895`), so
`same_correlation_facts` (`lib.rs:1273-1282`) never dedupes a heartbeat. **Every heartbeat costs two
whole-store writes**: first the watermark, then the correlation.
- About 2.1 of the 5.0 writes per second are accounted for by Muninn and the other catalog documents, and about
2.0 by heartbeats. The remaining 0.9 per second are refresh-driven correlation changes (dependency freshness
windows, O2 in the Idunn audit) plus writes that were not attributed. The two 1.5 MB records are not among the
catalog churn seen through the probe; their publisher and rate are unknown (Q2).
- Reads: about 30 whole-store decodes per second. Most are the 250 ms refresh (1 + N incarnations per pass),
and the rest are admission reads.
### 1.4 What is in the store: truth, cache or projection?
| Kind | Who owns the truth | Odin's role | Who reads Odin's copy | Must it be durable? |
|---|---|---|---|---|
| Provider catalog documents (ads, boundaries, profiles, media ads, schema catalog, surface_state) | the provider | relay cache | the Verse, via the catalog | yes, but not per write. Some publishers put once and never again (for example `heimdall.command_boundary`, whose `stored_at` is from 09-23), so losing the catalog on a restart would lose them. The CultMesh put has no application acknowledgement (`cultmesh-rs rudp_document_server.rs:464-482`), so a put promises the publisher no durability |
| `gamecult.eve.surface` and `surface_state`, 1.55 and 1.56 MB | unknown provider (Q2) | a cache **that can never be served**: a single-record snapshot is over the 1 MiB limit (Odin journal: "snapshot response is 1547739 bytes; limit is 1048576", and 1563575) | nobody can | no. They are 93% of the store's bytes and ride along in every rewrite |
| Runtime presences | the provider (signed) | admitted evidence with anti-replay (O5, O6) | the Verse; Odin's own sequence continuation (`main.rs:1184-1211`) | the latest per incarnation. The 30 s observation window (O6) bounds any replay after a lost write |
| Correlations and the publisher watermark | Odin | a **projection** of presence plus Idunn's projection | Idunn, lockless, from this same file (`--odin-correlation-store /var/lib/gamecult/odin/topology.cc`; Idunn `drivers.rs:5005-5030`) | the watermark must never let a *published* sequence repeat. "Published" means written to this file, because since batch 2 nothing else serves correlations. Both die at Idunn A3 and Odin O-R (`Idunn/docs/route-continuity-cut.md` section 4) |
| Presence history of withdrawn incarnations | nobody | retained on purpose (`withdrawn_expected_removes_current_correlation_but_preserves_presence_history`, `lib.rs:2814`) | only Odin's own sequence continuation reads it, and only Odin's own key | grows without bound with every deploy (follow-up F4) |
## 2. Ends
1. Odin reads its store once, at activation, and never again while it holds the write lease. Its working
state is in memory.
2. Odin writes its store at most once per flush interval, only when the content changed, as one atomic write
under the lock. A foreign write to the file is detected and ends Odin.
3. The store holds nothing Odin cannot serve.
Invariants that must survive:
- One writer: the process-write lease (`main.rs:457-474`) is the only authority to write. Nothing writes after
the lease is lost.
- Idunn's crossing: until A3, Idunn reads correlations from `topology.cc` with
`pull_all_read_only_snapshot`. The path, the single-file `.cc` format and the atomic replace do not change.
- Correlation sequences never repeat, as published.
- One bad record never stops Odin. This now holds at load: records are skipped and logged, never fatal
(`dc46be2`).
- Presence anti-replay within an activation (O5).
## 3. Identity, lifecycle, authority
| Persistent kind | Named by | Over time | Decided by |
|---|---|---|---|
| catalog document | `(type, key)`, type from the schema id (`main.rs:1311-1326`) | replaced by each provider put; never deleted | the provider decides the content; Odin decides admission (allow-list `main.rs:1237-1264`, plus the size rule in Cut 3) |
| presence | `presence_store_key(incarnation, signer)``lib.rs:1528` | replaced each heartbeat; kept after withdrawal (F4) | provider bytes, Odin's admission |
| correlation | incarnation key | replaced when facts change; deleted on withdrawal (`lib.rs:586-611`) | Odin; dies at O-R |
| watermark | target | only ever increases | Odin; dies at O-R |
| in-memory working set (new) | the same keys | loaded at activation, mutated per event, never reloaded while serving | Odin under the lease |
| flush (new) | none: one atomic snapshot | at most one per interval when dirty; one final flush on stop while the lease is held | Odin's serving loop |
## 4. Cuts
Subtraction cuts come before behaviour cuts. Each cut is its own branch and Soul pass. All verification runs on
Yggdrasil through the stopgap (`cargo test --locked -p odin-daemon` from the Odin root).
### Cut 1. The file stops being working memory (Odin, subtraction)
- **Repo/branch:** Odin `hands/odin-memory-store` from `44951a1`. Depends on nothing.
- **First:** a 60 s `/proc/<pid>/io` sample of live Odin, plus the `ss -m` drop counter (section 6 script).
The numbers in 1.1 serve as the baseline if they are taken within a day.
- **Deletes first:**
-`lib.rs:432-648``CultCacheOdinTopologyStore` (about 215 lines): every `pull_all_read_only_snapshot` and
CAS against the store file, and its four `CAS_ATTEMPTS` retry loops (`:453`, `:588`, `:614`, and the one
in `reconcile` at `:799`). With one writer under the lease, an in-memory compare-exchange cannot lose, so
the retry loops protect nothing.
-`main.rs:1266-1309``persist_generic_document`'s file CAS loop.
-`main.rs:486-489`: the store half of `incarnation_keys` (the projection half stays).