Compare commits

..
Author SHA1 Message Date
Meta Vens
434756e4f2 Odin test: pipelined puts with a lost datagram are both written
Adapted from Soul's S-6 probe. Two puts go out on one session through a
relay that drops the first reliable datagram. The transport resends it and
delivers both in order; Odin accepts both, so both are on disk and the file
matches the working set. The probe's other finding, that the transport
acknowledges the second before the first is resent, is the transport's
defect (tracked by Self) and is not pinned here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:39:32 +02:00
Meta Vens
30156f21d2 Odin: Odin's answer to a put is the promise, not the transport's ack
dcd36fc said an acknowledged put is always on disk. At the pinned
cultmesh-rs that is false: RUDP acknowledges a reliable datagram when it is
received, before ordered delivery hands the put to Odin, so a later put can
be acknowledged while an earlier one is still being resent (Soul probed it
with two pipelined puts and the first datagram dropped). What holds is
Odin's own answer: a put accepted by accept_raw_document is on disk, and one
it refuses is rejected by name. The comments now say that.

Soul S-5. The transport defect is tracked by Self.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:39:00 +02:00
Meta Vens
e2c0c8f626 Odin tests: a later failed attempt, and a removal on its own
Two rules had no test of their own (Soul's mutants survived):
- only_oldest_attempt_tried: every fixture's file held the oldest failed
  attempt. The new test leaves the file holding a later one.
- remove_not_dirty: the heartbeat always dirtied the store, so a removal
  that did not mark it survived. The new test removes a record with nothing
  else changed and checks the write drops it (and that removing what is not
  held writes nothing).

Soul S-8 and the attempted-list gap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:38:30 +02:00
Meta Vens
f9b0926830 Odin: a target being replaced has one presence in the catalog, not a refusal
The catalog refused itself whole ("duplicate public document identities")
whenever two projected incarnations of one target both held an admitted
presence, which every provider has during a deploy window: presences are
served under (schema, target), and the incumbent and the candidate share it.

One is now served and the other skipped, logged once per interval: the
incarnation whose Odin correlation is Ready, and between two that are both
or neither Ready, the later observation. A duplicate identity between
anything but two presences still refuses the query.

Tests: the incumbent (Ready) is served over a newer Warming candidate; with
no correlations to decide, the later presence is served, from either side.

Soul S-4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:38:13 +02:00
Meta Vens
a129da90fe Odin: the activation limit is set where the server is built, from its options
Soul's mutant main_limit_halved survived: main passed the server's
max_snapshot_response_bytes into RuntimeState::open by hand, and the tests
built RuntimeState and the server separately, so nothing tied the two. One
function, serve_with, now builds the server from its options and gives the
state that same limit; main and the test fixture both go through it. The
state has no limit until then, and Odin does not activate without one (the
fixture checks it refuses before serve_with).

Soul S-6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:37:09 +02:00
Meta Vens
726e0f31a1 Odin test: size the small document inside the calibrated range
The small document asked for a 200-byte response, less than the response's
own fixed overhead, so sizing it underflowed. It now asks for 2,000 bytes,
inside the range where the response grows one byte per payload byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:07:25 +02:00
Meta Vens
bdbecc3308 Odin Cut 3: size a stored document with a message id CultNet admits
CultNet refuses an empty message id, so sizing a document's snapshot
response with one failed for every record, and the failure was read as
"keep": nothing was ever dropped. The response is now sized with a one-byte
message id, the shortest CultNet admits, and a record that cannot be sized
is logged and kept rather than silently kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:58:28 +02:00
Meta Vens
723e74d5d1 Odin test: the unreadable-projection case is a projection that does not decode
The second fault in the test (removing the projection's lock file) did not
make the read fail, so it asserted a fallback that never ran. The test keeps
the fault that does: a projection file that does not decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:58:28 +02:00
Meta Vens
6786faa430 Odin Cut 3: a stored document the server could never serve is dropped at activation
Odin's store carried two ~1.456 MB Eve records that no snapshot response
could ever return (the server sends at most max_snapshot_response_bytes,
1 MiB), and every write rewrote them. Under Q3 B every put writes the
store, so they cost a ~3 MB write per put.

At activation, each stored peer document is sized as the server would
answer a snapshot of it alone: the same encoder, the shortest message id.
One larger than the server's limit is dropped from the working set and
logged by type, key and size; the first write removes it from the file.
The limit is the server's own option, handed to RuntimeState where main
builds the server, so the two cannot disagree. Presences and records the
catalog does not serve are not checked.

Q1 A (refuse the unservable), Q2 (the two records are dead residue). The
CultLib half, refusing such a put at the server, is its own cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:50:09 +02:00
Meta Vens
dcd36fccb9 Odin: say what a refused put promises
An acknowledged put is always on disk. A put refused because its write
failed may still become durable and be served: it stays in the working set,
and the next write lands it, or the failed write had already replaced the
file. Puts are latest-value per type and key, so that is sound; the
promise is written down at accept_raw_document.

Soul finding 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:48:12 +02:00
Meta Vens
749c6b9bbe Odin test: with nothing to write, the next change is written on the next pass
Pins the rule that only an attempted write starts the flush interval: a due
write with nothing to write leaves the next change to land on the first pass
after it. Kills the mutant that starts the interval on every due pass.

Soul finding 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:48:12 +02:00
Meta Vens
03f01ddba0 Odin: an unreadable projection file skips presences, not the catalog
Reading Idunn's projection once per catalog query (620efb8) made a
projection file that cannot be read refuse the whole catalog, where before
it failed each presence lookup and left the peer documents served. The
query now answers as if Idunn projected nothing, logged once per interval:
peer documents are served and presences skipped. Odin's own projection
check (require_own_projection) is unchanged and still retries or ends Odin
as before.

The new test restores file-level coverage: a projection that does not
decode, and one whose lock cannot be opened.

Soul finding 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:47:33 +02:00
Meta Vens
08ae183444 Odin: every failed write since the last success is Odin's own
A write that fails does not say whether it replaced the file, so the file
may hold what any failed write since the last success tried to write. The
store kept only the last attempt: a failure after the replace followed by a
failure before it left the file holding an attempt Odin no longer
recognised, and the next write called Odin's own file foreign and ended
it. Every attempt since the last successful write is now an accepted
baseline, cleared when a write lands.

Soul finding 1, with Soul's probe as the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:47:00 +02:00
Meta Vens
f4a1f8900a Odin: drop doc lines that restate the code they sit on
One was also wrong: `records` is in key order, not the file's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:32:41 +02:00
Meta Vens
c94b5c4f85 Odin tests: the real binary; an acknowledged put survives SIGKILL, SIGTERM stops it cleanly
From Soul's SIGTERM probe (scratch 5c379dd), made a test. It runs the built
odin-daemon under an Idunn-shaped world with the warming probe and lease
grant made over RUDP. With puts written before they are acknowledged, the
SIGKILL round is the rule's test; the SIGTERM case pins `main`'s signal
wiring, which no in-process test reaches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:30:32 +02:00
Meta Vens
0a09df9b7c Odin: a peer's put is written before it is acknowledged (Q3 B)
Operator ruling Q3 B: `accept_raw_document` returns Ok, and so the server
acknowledges the put, only once the store holding it is on disk. The write
is the one `flush`, under the same lease check, so it also carries every
other unwritten change. A write that is not made refuses the put. Odin's own
bookkeeping stays on the one-second interval.

The interval tests now change Odin's own bookkeeping (its heartbeat) instead
of putting, since a put is no longer an interval change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:30:28 +02:00
Meta Vens
3afa3fb778 Odin: every way out of the serving loop makes the last write
The loop is `serve`, and it ends in one place: a stop request, a condition
`survive` returns, or a dead socket all reach `stop`, whose write is made
only while the lease is current and the file is still Odin's. Tested for
each exit kind: a stop request, a refused own authority and a dead socket
write what changed; a lost lease and a foreign write write nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:27:53 +02:00
Meta Vens
a4aa5fcf97 Odin: a failed write is retried as it stands, and its own replaced file is not a foreign write
The records stay dirty until a write lands, so the next write makes the
failed change without another mutation (tested with no heartbeat to
re-dirty them). A write that fails after it replaced the file (the
directory sync) leaves Odin's own records there; they are remembered as
the write's attempt and accepted as a second baseline until a write lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:26:25 +02:00
Meta Vens
620efb85d8 Odin: a catalog query reads Idunn's projection once
The file-backed projection source no longer answers lookups itself, so no
caller can read the file once per lookup: every lookup is made against one
snapshot. The catalog query takes one per query, as the refresh pass does;
startup, activation and the own-projection check take one each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:24:32 +02:00
Meta Vens
2ad60183a1 Odin: the self presence counts from its own launch; the stored-sequence continuation is gone
Every launch is issued a fresh activation (Idunn control_plane.rs, the
continuity demotion), presences are ordered only within one activation
(select_presence), and Idunn compares only the warming presence's digest.
Continuing from a stored sequence protected nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:14:25 +02:00
soul-probe
cda20870fd Odin tests: the write starts the next interval; unprojected correlations are withdrawn
cargo-mutants survivors, both real gaps:
- serving_pass with the timer condition inverted wrote every change on the
  next pass after a write; the interval test set the timer by hand on every
  pass and never saw the loop set it. A change right after a write now waits.
- correlated_incarnations returning nothing left a correlation whose
  incarnation Idunn stopped projecting in place forever; the lib test
  refreshed that incarnation by name. The refresh pass now has to find it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:07:44 +02:00
soul-probe
35ea0d45fc Odin test: with nothing to write, the write step reads nothing, not even the lease
The flush step is due on every pass until something changes. Hand mutant M5b
(the daemon skipping its own dirty check) left every write rule intact but read
the lease file on every 2 ms pass; no test saw it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:39:47 +02:00
soul-probe
8189623411 Odin: the correlation record type is imported only where it is used
Cut 1 left it imported at the top of main.rs, where only the tests use it;
the release build warned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:32:11 +02:00
soul-probe
b5f13308e9 Odin Cut 2: one write of the store file per interval, only when it changed
Every admission, put and refresh that changed something rewrote and fsynced
the whole store; a heartbeat did it twice (watermark, then correlation).
Now the serving loop writes the file at most once per FLUSH_INTERVAL (1 s,
operator default Q3), and only when the working set changed.

Deleted first: the per-mutation writes Cut 1 left in accept_raw_document,
publish_self_presence and refresh_all_correlations. The watermark's own write
went with the file store in Cut 1; its mark now travels in the same atomic
write as the correlations that use it. "Published" means written.

Added: FLUSH_INTERVAL, the flush step in serving_pass, and a final write on
SIGTERM/SIGINT. The dirty flag lives in MemoryOdinTopologyStore. RuntimeState::
flush is the only write of the file: it checks the lease immediately before
writing, and CultLib's snapshot compare-exchange against what Odin last loaded
or wrote refuses a file another process wrote (ForeignStoreWrite). survive
treats ForeignStoreWrite like WriteLeaseLost: both end Odin. An attempt, failed
or not, starts the interval; an interval with nothing to write does not.

Tests: 50 changes inside one interval are one write, not made at 0.9 s and
made at 1.1 s after the last; nothing changed is no write; a lease lost before
the write writes nothing and ends Odin; a foreign write ends Odin and is not
written over; stopping writes only under the lease; a crash and reload never
republishes a written sequence, withdrawal included; a flush writes only
changes and never over a foreign file. The failed-publication test becomes a
failed-write test: the store is no longer read by publication.

Map: docs/write-pattern-cut.md, Cut 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:00:53 +02:00
soul-probe
409d92f279 Odin Cut 1: the store file stops being working memory
Odin read and decoded its whole 3.24 MB store file about 30 times a second:
every refresh pass, every admission, every catalog query and every CAS retry
read it again. The working set now lives in memory under the write lease.

Deleted: CultCacheOdinTopologyStore (every file read and CAS against the
store, and its four CAS_ATTEMPTS retry loops, including the one in
reconcile), persist_generic_document's file CAS loop, the store half of
incarnation_keys, stored_snapshot's file read, and
prior_self_publisher_sequence's file read in RuntimeState::open. With one
writer under the lease an in-memory exchange cannot lose, so nothing retries.

Added: MemoryOdinTopologyStore, keyed (type, key), implementing the
OdinTopologyStore trait (still the mock point). It is loaded once, with the
locked pull_all, in try_activate after the lease is verified; legacy
retirement and the self publisher sequence are operations on the loaded set.
The trait's compare_exchange collapses into commit. Refresh reads Idunn's
projection once per pass (IdunnProjectionSnapshot) instead of once per
incarnation; admission reads it once per admission.

Not in this cut: the write policy. Each mutating operation still writes the
whole file immediately, now from memory (flush after an admission, a put or a
refresh pass that changed something).

Tests: the two tests that corrupted the store while Odin served now put the
bad record in the file at activation. New: rewriting the file behind Odin
changes nothing it serves; refresh passes and catalog reads succeed with the
file replaced by bytes that do not decode; a loaded store never reads its
file again.

Map: docs/write-pattern-cut.md, Cut 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 12:58:52 +02:00
5 changed files with 2399 additions and 1957 deletions

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,350 @@
//! The real odin-daemon binary, run under an Idunn-shaped world: its bundle,
//! signer descriptors, projection and write lease on disk, and Idunn's warming
//! probe and lease grant made over RUDP as Idunn makes them. It reaches what
//! the in-process tests cannot: `main`'s signal wiring and a process that dies.
//! Unix only, like the daemon.
use std::fs::File;
use std::net::{SocketAddr, UdpSocket};
use std::path::{Path, PathBuf};
use std::process::{Child, Command, ExitStatus, Stdio};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use anyhow::{Result, bail, ensure};
use cultcache_rs::{CacheBackingStore, CultCacheEnvelope, DatabaseEntry, SingleFileMessagePackBackingStore};
use cultmesh_rs::{
CultMeshRudpDocumentPublishOptions, CultMeshRudpSnapshotOptions,
publish_cultnet_message_to_rudp_catalog, request_raw_snapshot_from_rudp_catalog,
};
use cultnet_rs::*;
use odin_daemon::IncarnationRef;
fn digest(byte: char) -> String {
format!("sha256-{}", byte.to_string().repeat(64))
}
fn now() -> u64 {
SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_millis() as u64
}
fn rfc3339(millis: u64) -> String {
chrono::DateTime::<chrono::Utc>::from_timestamp_millis(millis as i64)
.unwrap()
.to_rfc3339()
}
/// Replace the whole store at `path` with `entries`.
fn write_store(path: &Path, entries: &[CultCacheEnvelope]) -> Result<()> {
let store = SingleFileMessagePackBackingStore::new(path);
let current = if path.is_file() {
store.pull_all_read_only_snapshot()?
} else {
Vec::new()
};
ensure!(store.compare_exchange_snapshot(&current, entries)?, "store write lost a race");
Ok(())
}
fn credential_bytes(store: &Path) -> Result<Vec<u8>> {
Ok(rmp_serde::to_vec(&SingleFileMessagePackBackingStore::new(store).pull_all()?)?)
}
struct Odin {
_temp: tempfile::TempDir,
root: PathBuf,
candidate: SocketAddr,
child: Child,
store: PathBuf,
}
impl Odin {
fn log(&self) -> String {
std::fs::read_to_string(self.root.join("odin.log")).unwrap_or_default()
}
/// Put one provider document and return once the server acknowledged it.
fn put(&self, key: &str) -> Result<()> {
let message = CultNetMessage::DocumentPutRaw {
message_id: format!("put-{key}"),
document: CultNetRawDocumentRecord {
schema_id: "ghostlight.doc.v1".into(),
record_key: key.into(),
stored_at: rfc3339(now()),
payload_encoding: CultNetRawPayloadEncoding::Messagepack,
payload: vec![1],
source_runtime_id: None,
source_agent_id: None,
source_role: None,
tags: None,
},
};
let mut options = CultMeshRudpDocumentPublishOptions::odin(self.candidate, "test-publisher");
options.flush_timeout = Duration::from_secs(3);
publish_cultnet_message_to_rudp_catalog(&message, options)
}
fn holds(&self, key: &str) -> Result<bool> {
Ok(SingleFileMessagePackBackingStore::new(&self.store)
.pull_all_read_only_snapshot()?
.iter()
.any(|entry| entry.r#type == "ghostlight.doc" && entry.key == key))
}
fn signal(&mut self, signal: &str) -> Result<ExitStatus> {
let pid = self.child.id().to_string();
ensure!(Command::new("kill").args([signal, &pid]).status()?.success());
let deadline = Instant::now() + Duration::from_secs(20);
loop {
if let Some(status) = self.child.try_wait()? {
return Ok(status);
}
ensure!(Instant::now() < deadline, "Odin did not exit on {signal}");
std::thread::sleep(Duration::from_millis(10));
}
}
}
impl Drop for Odin {
fn drop(&mut self) {
let _ = self.child.kill();
let _ = self.child.wait();
}
}
/// Launch Odin as Idunn does and grant it its lease; returns once Odin has
/// written its store, which it does on its first pass after activating.
fn start_odin() -> Result<Odin> {
let temp = tempfile::tempdir()?;
let root = temp.path().to_path_buf();
let candidate = UdpSocket::bind("127.0.0.1:0")?.local_addr()?;
let t0 = now();
let idunn = enroll_service_identity_at::<IdunnServiceIdentity>(&root.join("idunn.cc"))?;
enroll_service_identity_at::<OdinTopologyIdentity>(&root.join("odin-topology.cc"))?;
let provider =
enroll_service_identity_at::<GameCultProviderHealthIdentity>(&root.join("odin-provider.cc"))?;
std::fs::write(root.join("topology-cred"), credential_bytes(&root.join("odin-topology.cc"))?)?;
std::fs::write(root.join("provider-cred"), credential_bytes(&root.join("odin-provider.cc"))?)?;
export_service_identity_trust_anchor::<IdunnServiceIdentity>(&idunn, &root.join("idunn-anchor.cc"))?;
let expected = IdunnExpectedIncarnationRecord {
schema_version: IDUNN_EXPECTED_INCARNATION_SCHEMA.into(),
target: "odin".into(),
plan_id: digest('1'),
incarnation_id: "odin/generation-1".into(),
sealed_release_id: digest('2'),
source_repository: "github.com/GameCult/Odin".into(),
source_revision: "3".repeat(40),
recipe_sha256: digest('4'),
runtime_id: "odin-runtime".into(),
expected_signer_identity_id: provider.entry().identity_id.clone(),
health_contract: "odin.runtime-health.v1".into(),
artifact_sha256: digest('5'),
state_schema_generation: Some("odin-v2".into()),
state_contract_sha256: Some(
"sha256-4f2f2dcd931d16f6b02bf295f41227b867aa982765661d55fa9f29fb2db7e449".into(),
),
write_lease_required: true,
route: Some(IdunnExpectedRoute {
route_id: "odin-route".into(),
transport: "rudp".into(),
stable_endpoint: "rudp://odin.internal:1000".into(),
candidate_endpoint: format!("rudp://{candidate}"),
}),
capabilities: vec![IdunnExpectedCapability {
capability: "odin.verse-rendezvous".into(),
schema: "odin.verse-topology.v1".into(),
compatibility: "v1".into(),
minimum_capacity: 1,
}],
dependencies: Vec::new(),
};
expected.validate()?;
let launch = IdunnRuntimeActivationLaunch::issue(&expected, digest('7'), t0 - 20, &idunn)?;
let activation = launch.activation().clone();
let mut activation_credential = Vec::new();
launch.write_credential(&mut activation_credential)?;
std::fs::write(root.join("activation-cred"), &activation_credential)?;
let expected_envelope = |key: &str| -> Result<CultCacheEnvelope> {
Ok(CultCacheEnvelope {
key: key.into(),
r#type: IdunnExpectedIncarnationRecord::TYPE.into(),
payload: expected.canonical_bytes()?,
stored_at: rfc3339(t0 - 30),
schema_id: Some(IDUNN_EXPECTED_INCARNATION_SCHEMA.into()),
})
};
let activation_envelope = |key: &str| -> Result<CultCacheEnvelope> {
Ok(CultCacheEnvelope {
key: key.into(),
r#type: IdunnRuntimeActivationRecord::TYPE.into(),
payload: activation.canonical_bytes()?,
stored_at: rfc3339(activation.issued_at_unix_millis),
schema_id: Some(IDUNN_RUNTIME_ACTIVATION_SCHEMA.into()),
})
};
let bundle = root.join("bundle");
std::fs::create_dir_all(&bundle)?;
write_store(&bundle.join("expected.cc"), &[expected_envelope("odin")?])?;
write_store(&bundle.join("activation.cc"), &[activation_envelope("odin")?])?;
let anchor = GameCultServiceTrustAnchorRecord {
schema_version: GAMECULT_SERVICE_TRUST_ANCHOR_SCHEMA.into(),
trust_anchor_id: "root/odin/runtime-presence".into(),
service_id: "odin".into(),
runtime_id: expected.runtime_id.clone(),
signer_identity_id: provider.entry().identity_id.clone(),
signer_public_key: provider.entry().public_key.clone(),
signature_algorithm: "ed25519".into(),
signing_purpose: GAMECULT_RUNTIME_PRESENCE_HEALTH_SIGNING_PURPOSE.into(),
signed_schema: GAMECULT_RUNTIME_PRESENCE_HEALTH_SCHEMA.into(),
binding_authority: "root".into(),
bound_at_unix_millis: t0 - 100,
expires_at_unix_millis: None,
private_state_exposed: false,
};
let incarnation_key = IncarnationRef::of(&expected)?.key();
let projection = root.join("projection.cc");
let mut projected = vec![
expected_envelope(&incarnation_key)?,
CultCacheEnvelope {
key: anchor.trust_anchor_id.clone(),
r#type: GameCultServiceTrustAnchorRecord::TYPE.into(),
payload: rmp_serde::to_vec(&anchor)?,
stored_at: rfc3339(anchor.bound_at_unix_millis),
schema_id: Some(GAMECULT_SERVICE_TRUST_ANCHOR_SCHEMA.into()),
},
activation_envelope(&incarnation_key)?,
];
write_store(&projection, &projected)?;
let lease_path = root.join("lease.cc");
File::create(root.join("lease.cc.lock"))?;
let state_root = root.join("state");
std::fs::create_dir_all(&state_root)?;
// Descriptors 3 and 4 carry the signer credentials, as systemd passes them.
let child = Command::new("bash")
.arg("-c")
.arg(format!(
"exec 3<{activation} 4<{provider}; exec {bin} --state-root {state} --idunn-projection {projection} --idunn-anchor {anchor}",
activation = root.join("activation-cred").display(),
provider = root.join("provider-cred").display(),
bin = env!("CARGO_BIN_EXE_odin-daemon"),
state = state_root.display(),
projection = projection.display(),
anchor = root.join("idunn-anchor.cc").display(),
))
.env("GAMECULT_IDUNN_RUNTIME_BUNDLE", &bundle)
.env("GAMECULT_IDUNN_CANDIDATE_BIND", candidate.to_string())
.env("GAMECULT_IDUNN_PROCESS_WRITE_LEASE", &lease_path)
.env("ODIN_TOPOLOGY_IDENTITY", root.join("topology-cred"))
.env("LISTEN_PID", "1")
.env("LISTEN_FDS", "2")
.env(
"LISTEN_FDNAMES",
format!("{IDUNN_RUNTIME_ACTIVATION_CREDENTIAL_NAME}:gamecult-runtime-presence-identity"),
)
.stdout(Stdio::null())
.stderr(Stdio::from(File::create(root.join("odin.log"))?))
.spawn()?;
let mut odin = Odin {
_temp: temp,
root,
candidate,
child,
store: state_root.join("topology.cc"),
};
// Idunn's warming probe, then the lease naming that exact warming presence.
let deadline = Instant::now() + Duration::from_secs(20);
let response = loop {
match request_raw_snapshot_from_rudp_catalog(CultMeshRudpSnapshotOptions {
target: candidate,
runtime_id: "test-idunn".into(),
message_id: format!("warming-{}", now()),
schema_ids: Some(vec![GAMECULT_RUNTIME_PRESENCE_HEALTH_SCHEMA.into()]),
record_keys: Some(vec!["odin".into()]),
..Default::default()
}) {
Ok(response) => break response,
Err(error) => {
if let Some(status) = odin.child.try_wait()? {
bail!("Odin exited while warming ({status}): {}", odin.log());
}
ensure!(Instant::now() < deadline, "Odin never answered warming: {error:#}");
std::thread::sleep(Duration::from_millis(100));
}
}
};
let CultNetMessage::SnapshotResponseRaw { documents, .. } = response else {
bail!("the warming probe was not answered with raw documents")
};
let presence: GameCultRuntimePresenceHealthRecord = rmp_serde::from_slice(&documents[0].payload)?;
ensure!(presence.state == "warming", "{}", presence.state);
let lease = IdunnProcessWriteLeaseRecord {
schema_version: IDUNN_PROCESS_WRITE_LEASE_SCHEMA.into(),
target: "odin".into(),
expected_projection_sha256: expected.canonical_sha256()?,
plan_id: expected.plan_id.clone(),
incarnation_id: expected.incarnation_id.clone(),
sealed_release_id: expected.sealed_release_id.clone(),
activation_witness_sha256: activation.canonical_sha256()?,
state_schema_generation: "odin-v2".into(),
state_contract_sha256: expected.state_contract_sha256.clone().unwrap(),
runtime_id: expected.runtime_id.clone(),
runtime_instance_id: activation.runtime_instance_id.clone(),
warming_presence_sha256: presence.canonical_sha256()?,
lease_epoch: 1,
issued_at_unix_millis: now() - 5,
};
let lease_envelope = |key: &str| -> Result<CultCacheEnvelope> {
Ok(CultCacheEnvelope {
key: key.into(),
r#type: IdunnProcessWriteLeaseRecord::TYPE.into(),
payload: lease.canonical_bytes()?,
stored_at: rfc3339(lease.issued_at_unix_millis),
schema_id: Some(IDUNN_PROCESS_WRITE_LEASE_SCHEMA.into()),
})
};
write_store(&lease_path, &[lease_envelope("odin")?])?;
projected.push(lease_envelope(&incarnation_key)?);
write_store(&projection, &projected)?;
while !odin.store.is_file() {
if let Some(status) = odin.child.try_wait()? {
bail!("Odin exited while activating ({status}): {}", odin.log());
}
ensure!(Instant::now() < deadline, "Odin never activated: {}", odin.log());
std::thread::sleep(Duration::from_millis(20));
}
Ok(odin)
}
/// A put Odin acknowledged is on disk: killed outright right after the
/// acknowledgement, inside the interval its own changes wait for, Odin loses
/// none of them.
#[test]
fn an_acknowledged_put_survives_sigkill() -> Result<()> {
for round in 0..3 {
let mut odin = start_odin()?;
// Inside the interval that Odin's activation write started.
std::thread::sleep(Duration::from_millis(300));
let key = format!("doc-{round}");
odin.put(&key)?;
odin.signal("-KILL")?;
ensure!(odin.holds(&key)?, "round {round}: the acknowledged put was lost: {}", odin.log());
}
Ok(())
}
/// SIGTERM reaches the serving loop as a stop request: Odin makes its last
/// write and exits cleanly, rather than waiting to be killed.
#[test]
fn a_stop_signal_ends_odin_cleanly() -> Result<()> {
let mut odin = start_odin()?;
let status = odin.signal("-TERM")?;
let log = odin.log();
ensure!(status.success(), "Odin exited {status}: {log}");
ensure!(log.contains("Odin stopping on request"), "{log}");
ensure!(!log.contains("final store write was not made"), "{log}");
Ok(())
}

View file

@ -1,570 +0,0 @@
# Odin store repair and recovery: cut map
**Operator rulings, 2026-09-30:**
- **Q-R1:** "You got it". A foreign file under a running Odin is set aside and Odin continues.
- **Q-R2:** "Yep". Keep the previous generation on disk.
- **Q-R3:** "Start empty". Nothing decodable at activation means Odin starts empty.
- **Q-R4:** "Yep, and Idunn needs better reporting capabilities so it can serve as watchdog. I ought to get a message
about this on Discord if it ever happens."
- Stop writing after the third set-aside in 10 minutes, and keep serving.
- **New requirement:** Idunn watchdog reporting that reaches the operator on Discord. It is mapped as its own
campaign and is not an Odin cut.
- **Q-R5:** "That's fine, Odin still runs after all, right?" Yes. Odin keeps serving, and the store condition is
published as its own typed record, not as a degraded presence.
- **Q-R6:** "Approved". Both drills.
**R1 LANDED: CultLib main `daedfdd6` (2026-09-30).** The merged tree `0e6891b3` was verified on Yggdrasil in every
runtime, including the four-runtime store interop test. That test runs close to its 30 s per-command limit under
load (`cult-cache.test.ts:1407`), so it is fragile: follow-up. Consumer pins still gate their own bumps: Bifrost only
with R3b's TS fsync; Muninn and Idunn with their error-text matches.
**R1 status (Self, 2026-09-30).** Built on CultLib `hands/cultcache-read-failures` (`c0b56e06`..`b7908668`), then
fix batch 2 (`..2d7a3c78`). Soul's verdict on batch 2 was to hold the merge. Batch 3 is in Hands, fixing:
- **Blocker, F1 (zero-byte stores on Yggdrasil).**
- Two zero-byte stores are live: `/srv/bifrost/persona-feedback/{inbox/observations,outbox/deliveries}.cc`.
- They are pre-created by `touch` in gamecult-ops `install-bifrost-persona-feedback-native-yggdrasil.sh:42,46`
and `bootstrap-bifrost-persona-feedback-yggdrasil.sh:22,29`.
- Bifrost `persona-feedback.mjs serve` and Epiphany `epiphany-swarm` crash-loop on them once their CultLib pins
move past R1.
- **Neither pin may move until the scripts stop pre-creating stores and Self has moved the live files aside.**
- **F4.** Rust errors echo record values through serde's text (`lib.rs:753-758`). The header is echoed without bound.
- **F6.** A dangling symlink reads as absent, and the writer then replaces it.
- **F7.** The same NotFound-only rule in Python's `JsonLinesBackingStore` and in TS `pullAll`'s repair write.
- **M7.** A header-prefix fixture.
Recorded for later cuts:
- **F3 → R2/R3b.** TS, Python and C# decode the whole buffer before reading the header. A truncated newer-format
store (20..n-1 bytes) is "unsupported format" in Rust and a raw decode error elsewhere.
- **F5 → R3.** C# `ReadSnapshot` reads ENOTDIR/EISDIR as an empty store (`FileInfo.Exists`, `CultCache.cs:3439`).
C# `DirectoryMessagePackBackingStore` `File.Exists` (`:409`, `:497`) is the same. This is a live disagreement between
runtimes until R3.
- **F8 → R3b.** Python leaves its staging file behind when a write fails (`stores.py:124-128`).
- **Tooling.**
- `packages/cultcache-rs` has no committed `Cargo.lock`, so the map's `cargo test --locked` cannot run.
- The cultcache-ts interop test needs `CARGO_TARGET_DIR` unset under ygg-verify.
- **Soul on batch 3 (`2d7a3c78..3c9e195b`, gamecult-ops `bca7978`), 2026-09-30: merge both.** Pre-merge batch 4 is in
Hands: fix the README symlink claim (S1), close value echoes in the redb and typed-decode paths (S2) and in the
Python/TS non-string key echo (S3), make the unlock policy coherent (S9), classify encode refusals separately from
`NotReplaced` (S10), and fix a test timeout (S11). The ops branch deletes the guard test (S6) and the dead compose
layout (S5).
- **Open for R3:** writes through a symlinked store path replace the link, so a store behind a link to a mounted
volume moves into the container layer. R3 decides resolve or refuse.
- **Open for R3b:** Python and TS never fsync. After F1 their first create-by-rename is not covered by ext4
`auto_da_alloc`, so a crash can leave zero bytes, which every post-R1 reader refuses forever. **Bifrost's CultLib
pin moves past R1 only together with R3b's TS fsync.**
- **For W0 (Odin):** classify a write failure through `CultCacheStoreWriteFailed.kind` together with its `io::Error`,
never retry on `NotReplaced` alone (S10).
- Deployed Epiphany is `d2ca6630`, using a vendored cultcache-rs (`ba6a4874`) that reads a missing store as empty.
All Epiphany units have been inactive since 2026-08-22.
- **Done 2026-09-30 20:48 CEST (Self).** gamecult-ops merged at `3fa22aa`. The inbox is setgid 2750. All three
zero-byte stores were renamed in place and moved to `/root/zero-byte-stores-20260930/` on Yggdrasil, with
`manifest.txt` recording owner, mode and mtime. None had gained bytes. All three Bifrost units stayed active, and
persona-feedback logged nothing and did not restart.
- The compose/container lane (the compose file, two deploy scripts, the `.ps1` uploader, two runbooks) is dead:
the live Bifrost units run native `dotnet` and `node`. **Deleted 2026-09-30, operator: "delete the Bifrost lane".**
gamecult-ops `fdb4356`. Its live trigger, the Idunn legacy actuator route `deploy bifrost-persona-feedback`, was
retired in Idunn `7793b5b` and installed on Yggdrasil at 21:50 CEST (sha256 `d95fa08c...`; the previous binary is
at `/root/idunn-yggdrasil.pre-bifrost-retire-20260930`). The manifest and `/srv/bifrost/compose` were moved to
`/srv/retired/retired-2026-09-30/`. All three Bifrost units stayed active. **Follow-up:** persona-feedback has no
Idunn deploy route now. Give it a real Idunn binding.
- Live zero-byte stores to move aside after the ops merge, in Soul's order (setgid on inbox, then rename in place,
then relocate): `observations.cc`, `deliveries.cc`, and a third, `/var/lib/gamecult/epiphany/idunn-provider-health-admission.cc`.
- **Text matches.** gamecult-ops `scripts/idunn/start-idunn-local.ps1:143` matches the same removed text as Muninn
(below). It must change with Idunn's next CultLib bump.
- **Consumer follow-up that must travel with the next Muninn CultLib bump:**
`Muninn scripts/restart-starfire-muninn.ps1:144` matches `*failed to decode MessagePack*` to trigger
`Reset-CorruptMuninnStore`. R1 removes that text, so the reset would stop firing silently. The script must match
the typed `CultCacheStoreUnreadable` kind, or the new wording, in the same change.
- C#'s truncated-store exception (`EndOfStreamException`) is an `IOException`, the same I/O-versus-undecodable
overlap. It goes with R3's C# work.
Status: cut map, Imagination pass 1 (Opus), 2026-09-30. Nothing has landed. This map owns the means for repairing
and recovering `/var/lib/gamecult/odin/topology.cc`. It follows `docs/write-pattern-cut.md` (Cuts 1-3) and starts
from that campaign's landed head.
Anchors:
- Odin `hands/odin-write` at `cda2087` (Cuts 1-2 plus two test commits). Q3 B (peer puts fsynced before ACK) is
being finished on that branch now. **Re-anchor every Odin `file:line` below on the head that lands; the line
numbers here are `cda2087`.**
- CultLib `6d0cc963`. `packages/cultcache-rs` is byte-identical to Odin's pin `3bf1c0ce`, so the Rust line numbers
hold at both.
- Idunn `a96ad9d`, Heimdall `e8e2832`, gamecult-ops `runbooks/gamecult-state-backup.md` as of 2026-09-30.
Operator, 2026-09-30, on the three options Self offered for an undecodable foreign write (degraded but loud, end
Odin, unchanged): **"None of these options involve repair and recovery, which is what ultimately needs to happen."**
Open: Q-R1 to Q-R6 (section 8). D1 (section 6, O1) is a Self default the operator may overrule.
## Short answer
Odin already holds the one thing a repair needs: its in-memory working set, which is authoritative for everything
it owns while it holds the write lease. Today it never uses it for repair. It ends on a decodable foreign write, and
it silently stops persisting on an undecodable one. At activation it has no working set and no second copy, so it
crash-loops.
The repair has one rule: **a store file that is not what Odin last wrote is set aside, never deleted and never
merged, and Odin writes its working set over it under the lease.** At activation Odin has no working set, so it
falls back to a previous generation kept on disk, and then to an empty store that providers refill. The
correlation sequence gets a time floor, so no rollback can recreate the watermark deadlock. Recovery is published as
a typed record in Odin's own catalog. The mechanism (typed read failures, a set-aside exchange, a retained previous
generation, no staging file left behind by a failed write) belongs to CultLib, because every consumer of the
single-file store has the same gap.
## 1. Body facts (probed)
### 1.1 Probe: cultcache-rs single-file store under fault
The probe was a scratch crate at CultLib `3bf1c0ce`, `scratchpad/odin-recovery-probe` (Imagination session), commit
`b19eba9`, run through the stopgap on Yggdrasil in slot 2. It ran with `--tmpfs /small:size=64k --tmpfs /ro:ro`,
using `trap '' XFSZ; ulimit -f 40` for the file-size case. The log is `scratchpad/odin-recovery-probe.log`.
| Fault | Result |
|---|---|
| P1: file truncated to half | `compare_exchange_snapshot` and `pull_all` return `Err`, **not** `Ok(false)`. There is no `io::Error` in the chain; the decode error is flattened to text (`lib.rs:720-725`). The message is wrong: "invalid type: string \"cultcache.store.v1\", expected struct CultCacheEnvelope". `store_header` (`lib.rs:317-341`) deserializes the *whole* array to find the header, so a truncated v1 file has no header and falls through to the legacy decoder. The file is unchanged by the failed exchange |
| P2: 12 bytes of garbage appended | `pull_all` returns `Ok`, all 3 records. `compare_exchange_snapshot(expected)` returns **`Ok(true)` and silently drops the tail**. `rmp_serde::from_slice` ignores trailing bytes. C#'s `DeserializeSnapshot` (`CultDocumentMessagePackSerialization.cs:179-215`) never checks for the end either |
| P2b: one byte flipped inside a payload | `pull_all` returns `Ok` (payloads are opaque bytes). `compare_exchange_snapshot` returns **`Ok(false)`**. To Odin, bit rot is a *decodable foreign write* |
| P3: header `cultcache.store.v4...` | `Err` with a distinct message, "not readable; this runtime reads v1 only". It is distinguishable only by its text |
| P4: another handle wrote a valid snapshot | `Ok(false)` |
| P5: `hard_link(topology.cc, topology.cc.displaced.<ts>.x)`, then an exchange through the store | The link keeps the old bytes exactly, and the store writes the new file. The staging sweep (`remove_abandoned_staging_files`, `lib.rs:2730-2766`) removes a `<name>.<uuid>.tmp` beside it and leaves the displaced name alone |
| read-only filesystem, no store | `pull_all` returns `Ok(0)`. The exchange fails opening the lock: `io::Error` `ReadOnlyFilesystem` (os 30) |
| `RLIMIT_FSIZE` 40 KiB, 200 KB write | `Err` `io::Error` `FileTooLarge` (os 27). The old file is intact and still decodes. **A 40,960 B partial staging file is left behind.** The next attempt sweeps it and leaves a new one |
| 64 KiB tmpfs, 200 KB write (ENOSPC) | `Err` `StorageFull` (os 28). The old file is intact. **A 61,440 B partial staging file is left behind: all of the filesystem's remaining space.** The same holds on every retry |
What follows from the probe:
- **Decodable is not the same as legitimate.** A flipped payload byte is "decodable and foreign", and appended
garbage is "decodable and ours". The decodable/undecodable split cannot be what decides recovery. One rule has to
cover both: *not what Odin last wrote*.
- **A write that fails for lack of space holds the space it failed on.** `/var/lib/gamecult` is shared by every
service on Yggdrasil. Odin retries once per second (and, under Q3 B, on every put), so it re-takes whatever
another service frees. C# deletes its temp file in a `finally` (`CultCache.cs:3496-3531`); Rust does not
(`write_all_unlocked`, `lib.rs:741-768`). This is a Rust parity defect.
- **The atomic write keeps no previous generation.** `write_all_unlocked` stages, fsyncs, renames over the file and
fsyncs the directory. After the rename, the old inode is gone. The brief's "last good checkpoint kept by the
atomic write" does not exist on disk. The only last-good copy is Odin's in-memory `flushed` snapshot
(`lib.rs:506-510`), and it dies with the process.
### 1.2 Odin today (`cda2087`)
| Case | Path | Outcome |
|---|---|---|
| Decodable foreign write while serving | `MemoryOdinTopologyStore::flush` `lib.rs:613-626`: `compare_exchange_snapshot(flushed, current)` returns `false`, so `ForeignStoreWrite`. `survive` `main.rs:879-895` returns it, and Odin ends | Idunn restarts Odin (`control_plane.rs:68-71`: 6 restarts per hour, the wait doubling from 5 s). The new Odin **loads the foreign file as truth**. Odin's own unflushed changes, and every record of Odin's that the foreign file lacks, are lost without a trace. A flipped payload byte (P2b) takes this path |
| Undecodable foreign write while serving | the same exchange returns `Err` (decode). `survive` logs it and retries every second, forever. Nothing is persisted again | **Idunn's crossing breaks too.** `receive` (`Idunn drivers.rs:5005-5030`) reads the same file and propagates the error to `admit_latest_topology` (`control_plane.rs:7903-7907`) and `refresh_admitted_topology` (`:6040`). Every Odin-correlated deployment and refresh fails until the file is readable, which is never |
| Corrupt file at activation | `try_activate` `main.rs:246-297` calls `MemoryOdinTopologyStore::load` `lib.rs:519-538`, which calls `pull_all()?`. `main.rs:762` `while !try_activate()?` exits the process | Crash loop until Idunn's budget is spent. Then Idunn reports once, to its log only (`control_plane.rs:5709-5723`, `report_once` `:4599-4610` is `eprintln`), and stops. The Verse rendezvous is down |
| Disk full or read-only | the exchange returns `Err` (io). `survive` logs and retries every second | Odin serves from memory. Each attempt leaves a partial staging file (1.1). A stop or deploy in this state loses everything unflushed. Nothing typed says so |
Detection latency for a foreign write: `flush` writes only when dirty (`main.rs:307-314`). Odin's own heartbeat
(`HEARTBEAT_INTERVAL` 5 s, `main.rs:69`) dirties the set, so a foreign write is met at the latest one heartbeat plus
one flush interval (about 6 s) after it happens. No polling read is needed, and Cut 1's rule (no reads while
serving) stands.
### 1.3 Other sources of truth, and what they can recover
| Source | Holds | Freshness | Recovers |
|---|---|---|---|
| Odin's working set (W) | everything, including unflushed changes | live | everything, but only while this process lives and holds the lease |
| `flushed` (in memory) | the last snapshot Odin wrote | at most one interval old | nothing more than W; it is the comparison base, not a store |
| previous generation on disk (new, O3 and R3) | the snapshot before the last one this store's owner verified and replaced | one write old | everything except the last write's delta, including publish-once documents, presence history and the watermark |
| nightly backup (`gamecult-state-backup.md`) | `/var/lib/gamecult/odin` whole, shared-lock copy | up to 24 h old, 3 days on Yggdrasil, 14 days plus 8 weekly on Raven | publish-once documents. **Never automatically**: "It never becomes live authority, and nothing restores it automatically"; "restore a store only after stopping its owner" |
| providers republishing | presences (2.7-12 s), Muninn documents (2 s), ads (about 30 s); Heimdall HEAD republishes `heimdall.command_boundary` and its three siblings every 60 s (`Heimdall src/index.ts:19,35-39`, gated on its write lease) | seconds to a minute | every document whose publisher republishes |
| Idunn's projection plus presences | correlations | one refresh pass (250 ms) after the presences return | correlations. They are a projection and never need restoring |
| Idunn's `odin_publisher_sequence_cursor` (`control_plane.rs:1456,1537,6081`) | the highest correlation sequence Idunn admitted, per target | live | nothing directly (Idunn owns it), but it is why the watermark must never roll back |
Premise correction: `heimdall.command_boundary` is not publish-once at Heimdall HEAD. A `stored_at` of 09-23
proves only that the publisher's own `stored_at` has not changed, because Odin stores the publisher's
`stored_at` (`persist_generic_document`, `main.rs:1330-1347`). Whether the deployed Heimdall re-puts is
unverified; O3's outstanding-republish list measures it.
**The watermark is the one record whose loss is not self-healing.** Idunn refuses any correlation sequence at or
below its cursor (`control_plane.rs:1763,1785`). An Odin that starts from an empty store, from the previous
generation, or from a backup restarts a target's count below that cursor. That is the deadlock documented at
`lib.rs:33-49`: Idunn refuses every correlation until the count climbs past the old mark, and the refusal
withdraws the correlation before it can climb. Every recovery source except W has this defect. O1 removes it.
Other facts the design leans on:
- **Idunn fences before it grants.** `CultCacheWriteLeaseDriver` (`Idunn drivers.rs:4050-4205`): "revocation
follows an exact incumbent stop", and grant is a nonblocking exchange from empty. A second *Odin* cannot hold
the lease while the first is still writing, so a foreign writer seen by a lease-holding Odin is not an Odin.
- **Idunn has a lifecycle brake** (`idunn-provision lifecycle-brake-engage|release|status`,
`provisioning.rs:95-97`; `odin-lifecycle-brake.cc` per `runbooks/odin-yggdrasil.md:29-30`). An engaged brake
denies continuity restarts (`control_plane.rs:5771-5774`, `:6219`). This is the operator's hold for a manual
restore.
- **A non-`active` Odin presence fans out.** `reconcile` sets `ready` only when the presence state is `active`
(`lib.rs:946`), and `dependency_evidence` (`lib.rs:1184ff`) makes every managed dependent of Odin not Ready.
The presence schema allows `degraded` (`cultnet-rs runtime_authority_contracts.rs:193`), but using it would
be a readiness statement about every dependent, not a store report.
- **The catalog serves only peer types and presences** (`public_document`, `main.rs:590-640`). No `odin.*` record
is served today, and neither Odin nor Idunn publishes an Eve surface. F3 in the write map (the unowned
`surface:gamecult.network.status`) is where an Eve lowering belongs.
## 2. Ends
1. No single fault, whether a foreign write of any kind, a corrupt file, or a failed write, ends Odin or leaves it
silently unpersisted. Only losing the write lease ends it.
2. A store file that is not what Odin last wrote is preserved byte for byte beside the store and never deleted,
merged or adopted by any automatic path. The owner (Odin under the lease) writes its own state over it.
3. At activation a corrupt store is recovered from the best source on the host (previous generation, then empty),
and never from the backup automatically.
4. No recovery can make a correlation sequence repeat or fall below one Idunn has admitted.
5. Every repair, and every stretch in which Odin is serving without persisting, is typed state that can be
queried through Odin's catalog while it is happening, including when the disk refuses writes.
Invariants carried over: one writer (the lease, checked immediately before every write); Idunn's lockless
crossing reads a whole `.cc` snapshot at `topology.cc`; the path never becomes absent (Idunn reads an absent file
as an empty snapshot, which withdraws every correlation); no reads of the store while serving; one bad record
never stops Odin.
## 3. Identity, lifecycle, authority
| Persistent kind | Named by | Over time | Decided by |
|---|---|---|---|
| store file `topology.cc` | the state root plus `TOPOLOGY_SLOT` | replaced whole by the owner's exchange; never absent once created | Odin under the lease |
| displaced file (new) | `topology.cc.displaced.<UTC yyyymmddThhmmssZ>.<uuid>`, same directory (it must not end in `.tmp`) | created by a set-aside exchange as a hard link to the replaced inode; never modified; never deleted by any program. Travels to Raven with the nightly backup | created by Odin through CultLib; **disposed of only by the operator** |
| previous generation (new, Q-R2) | `topology.cc.previous` | replaced on every *verified* exchange (current == expected) by a hard link to the outgoing inode; **not** rotated by a set-aside exchange, so it always holds a generation that some writer verified before replacing | CultLib mechanism; Odin opts in |
| store condition record (new) | type `odin.store_condition`, schema `odin.store_condition.v1`, key `topology` | one record held in W; rewritten when the condition changes; flushed with the store; served by the catalog | Odin |
| correlation sequence floor (new rule, not a record) | per target, in the existing watermark | next = max(mark + 1, prior + 1, now in unix ms) | Odin |
| backup snapshot | `/var/backups/gamecult-state/<date>/` and Raven | nightly; never live authority | the operator restores it, with the lifecycle brake engaged (runbook) |
## 4. Authority map (the whole campaign)
- **Owner:** Odin, while it holds the current process-write lease, decides what `topology.cc` holds, when it is
set aside, and what activation recovers from. CultLib owns the mechanism (read classification, set-aside
exchange, previous generation, staging cleanup) and no policy. Idunn owns whether Odin runs (continuity budget,
lifecycle brake, deploys). The operator owns restores from backup and the disposal of displaced files.
- **Inputs:** W; `flushed`; the lease (read immediately before every write); at activation, `topology.cc` and then
`topology.cc.previous`; the clock (the sequence floor).
- **Outputs:** one atomic `topology.cc`; displaced files; `topology.cc.previous`; the `odin.store_condition` record
in the catalog.
- **Derived state:** `topology.cc` is a checkpoint and Idunn's crossing, never an input while serving. The
displaced files are evidence only: nothing reads them automatically. The previous generation is read only by
activation recovery. The condition record is a report and decides nothing.
- `ForeignStoreWrite` **is no longer an owner of Odin's life**. A foreign write becomes a set-aside event.
- **Forbidden writers and deciders:**
- Any path that ends Odin on a store condition. `survive`'s `ForeignStoreWrite` arm (`main.rs:884`) is
deleted; `WriteLeaseLost` stays, and so does the temporary `PresenceAuthorityRefused` rule.
- Any automatic delete of a displaced or previous file.
- Any merge of records from a foreign or displaced file into W.
- Any automatic restore from the backup.
- Idunn writing or repairing `topology.cc`.
- Any second writer of the file inside Odin: activation never writes. It prepares W and `flushed`, and the one
`flush` makes the first write through the same set-aside exchange.
- **Shared paths:**
- The interval flush, the Q3 B per-put flush, the stop flush and the activation-recovery first flush all go
through one primitive, `replace_snapshot_setting_aside`.
- Every path that reserves a correlation sequence goes through the one floored reservation.
- **Deletion line:** `ForeignStoreWrite` (`lib.rs:630-641`) and its `survive` arm are deleted before the set-aside
exchange is wired in. The `Err` escape in `load` (`lib.rs:521`, `pull_all()?`) is replaced by the recovery
ladder, not wrapped.
## 5. Recovery by case
| Case | Detected by | Action | Recovered from | Loss bound | Visible as |
|---|---|---|---|---|---|
| **A. Decodable foreign write while serving** (a hand edit, a tool, a restore made against the runbook, or bit rot in a payload per P2b) | the next flush: current != `flushed` | the lease is checked; the file is set aside (hard link); W is written; Odin keeps serving | W | none of Odin's own records. The foreign file's content is preserved, not adopted | condition event `displaced`, reason `mismatch`, with path, sha256, bytes and time |
| **B. Undecodable foreign write while serving** (truncation, garbage, a foreign format, and after R2 trailing bytes too) | the next flush: current does not decode | the same | W | none. Idunn's crossing is unreadable for at most one heartbeat plus one flush (about 6 s), instead of forever | the same, reason `undecodable` |
| **A/B repeated** (a live foreign writer) | a third set-aside inside 10 minutes in one activation | Odin stops writing and keeps serving; puts are not ACKed (Q3 B: ACK means durable); no new displaced files (Q-R4) | W stays in memory | nothing while the process lives | condition `contested`, since when, with the displaced files |
| **C1. Undecodable store at activation** | `load` gets `Undecodable` | W is built from `topology.cc.previous` if it decodes; `flushed` is set to "unreadable", so the first flush sets the corrupt file aside and writes W | the previous generation | the last write's delta: one interval of Odin bookkeeping, or one fsynced put. **A put ACKed as durable can be lost here; say so in Q3 B's record** | event `recovered-previous`, plus an outstanding-republish list: peer records in the previous generation not yet re-put by their publisher, cleared as they arrive |
| **C2. Nothing decodable at activation** | `.previous` absent or undecodable too | start empty (Q-R3); the first flush sets the corrupt file aside | providers republish; correlations re-derive in one pass; the sequence floor holds | every publish-once document; presence history (F4 makes this harmless) | event `started-empty`; outstanding list is `unknown`. The operator's path is the backup, with the lifecycle brake engaged |
| **C3. Unsupported format at activation** (a newer runtime wrote v4, or a downgrade) | `load` gets `UnsupportedFormat` | refuse to activate. The file is untouched and nothing is set aside | the deploy or rollback that caused it; Idunn owns that | none | Idunn's continuity budget and the Odin journal. This is a deployment fault, not a store fault |
| **C4. I/O error reading the store at activation** (EIO, EACCES after a restore with the wrong owner, EISDIR) | `load` gets an `io::Error` | refuse to activate. Nothing is set aside: **only bytes that were read are ever judged** | the operator fixes the host or the mode | none | as C3 |
| **D. Write fails while serving** (ENOSPC, EDQUOT, EFBIG, EROFS, EIO) | the exchange returns an `io::Error` while staging or renaming | nothing is set aside; the staging file is removed (R1); W is kept; retry at the flush interval; puts are not ACKed; Odin keeps serving | W, when the fault clears | nothing, unless Odin is stopped or redeployed before the fault clears (follow-up I-F1) | condition `unpersisted` since T, with the last error kind (`no-space`, `read-only`, `io`) and the dirty-record count, served from memory |
| **D'. Stop while unpersisted** | the stop flush fails | log it and stop, as today (`main.rs:794-799`) | the next activation loads the last good file | the unflushed delta | the final journal line only. I-F1 is the owner fix |
Rollback safety, for C1, C2 and an operator restore: every source except W carries a watermark below Idunn's
cursor. The floor (O1) makes the next sequence exceed every sequence published before, whatever the store holds.
## 6. Cuts
Order: the Rust substrate gaps first (R1), then the watermark floor (O1, independent and small), the set-aside
primitive (R3), Odin's serving repair (O2), activation recovery (O3), parity hardening (R2, R3b), and finally the
runbook and drill (X1). Each cut is its own branch and its own Soul pass. Odin verification:
`cargo test --locked -p odin-daemon` from the Odin root. CultLib Rust: `cd packages/cultcache-rs && cargo test
--locked`. All of it runs on Yggdrasil through the stopgap. **A permission-based test is unproven there (root).**
The injections below avoid permissions: truncation, a directory at the store path (EISDIR), tmpfs size and
read-only options, and `RLIMIT_FSIZE`.
### R1. cultcache-rs: typed read failures, no staging file left behind, a header that survives truncation (CultLib)
- **Repo/branch:** CultLib `hands/cultcache-rs-store-faults` from `main`. Depends on nothing.
- **Deletes first:** the flattening `map_err(|error| anyhow!(...))` in `read_all_unlocked` (`lib.rs:720-725`);
`store_header`'s whole-array visitor (`lib.rs:317-341`).
- **Adds:**
- `pub struct CultCacheStoreUnreadable { path, kind: Undecodable | UnsupportedFormat }`, carried in the error
chain of every read of a single-file store whose bytes were read and not accepted, so consumers can
`downcast_ref`. An `io::Error` stays an `io::Error`, and the two never overlap.
- `store_header` reads only the array header and the first string, reusing `read_array_header` and
`read_string` (`lib.rs:2661-2700`). A truncated v1 file then reports as an undecodable v1 file, not as legacy.
- `write_all_unlocked` removes its own staging file on every failure after creating it. This is C# parity with
`CultCache.cs:3525-3530`.
- **Rules that must die under their own mutation:**
- a truncated store is `Undecodable`, and a v4 header is `UnsupportedFormat`;
- an `io::Error` read failure is never classified `Unreadable`;
- after a failed write, the directory holds no `<name>.*.tmp`.
- **Verification:**
- Tests describe behaviour. The staging-cleanup test forces the failure without permissions. One option is a
child process under `RLIMIT_FSIZE` with SIGXFSZ ignored (the probe's method); another is a path whose parent
fills a size-capped tmpfs, if the test runner can mount one. Hands picks one and names it.
- `cargo mutants --in-diff`.
- **Estimate:** about -30 / +70.
### O1. The correlation sequence has a time floor (Odin) — default D1
- **Repo/branch:** Odin `hands/odin-sequence-floor` from the landed `hands/odin-write` head. Depends on nothing
else.
- **Change:** `reserve_publisher_sequence` (`lib.rs:702-723`) returns max(mark + 1, prior + 1, now in unix ms),
and records it as the mark. The doc comment at `lib.rs:33-49` gains the rule: "A mark read from any older copy
of the store is a floor, never the next value; the clock makes every rollback safe."
- **Why a floor and not a read of Idunn's cursor:** it needs no new crossing. It holds for every recovery source,
including the backup, and the host clock is already trusted for presence freshness.
- **Assumption:** the clock does not step back across a rollback by more than the rollback's age. Soul checks that
Idunn accepts a jump: its rule is "strictly above the cursor" (`control_plane.rs:1763,1785`). No test may pin a
gap limit.
- **Verification:**
- Existing tests pinning counts (`withdrawing_a_correlation_does_not_reset_its_publisher_sequence`
`lib.rs:1996-2014`) are restated as order properties.
- New test: a store rolled back to an earlier snapshot, then one refresh, yields a correlation sequence greater
than every sequence published before the rollback.
- New test: two reservations inside one millisecond are strictly increasing.
- The clock must be injectable at the reservation. At least one mutant must be a function of the time (an offset
or a scale), with probe values where it would show.
- **Estimate:** about -5 / +25.
### R3. The set-aside exchange and the previous generation (CultLib, Rust and C#; TS and Python in R3b)
- **Repo/branch:** CultLib `hands/cultcache-set-aside` from R1. Q-R2 decides the previous-generation half.
- **Adds, on `SingleFileMessagePackBackingStore`:**
- `replace_snapshot_setting_aside(expected, replacements) -> Result<SnapshotReplacement>`, with
`SnapshotReplacement::{Exchanged, SetAside { displaced: PathBuf, sha256, bytes, reason: Mismatch |
Undecodable }}`. It makes one exclusive lock hold, in this order:
1. stage the replacement (write and fsync);
2. read current;
3. if current equals expected, rename the staged file over (rotating `.previous` first when enabled);
4. if the read was an `io::Error`, remove the staging file, return the error and change nothing;
5. otherwise hard-link current to the displaced name, fsync the directory, rename the staged file over, and
fsync the directory again.
Staging comes first, so a full disk sets nothing aside. The path is never absent: a link, never a rename. A
set-aside never rotates `.previous`.
- Opt-in previous generation: a constructor option (`retaining_previous_generation`, or Hands' spelling of it)
plus `pull_previous_generation() -> Result<Vec<CultCacheEnvelope>>`, read under the shared lock and classified
like any read. Off by default, because existing consumers must not grow files they did not ask for.
- Filesystems without hard links: return an explicit error naming the filesystem. Do not fall back to a
copy-then-rename, which would make the path briefly hold new content while the evidence is still being
written. Windows NTFS links work; the Windows path is **not yet reached** by the Yggdrasil suite and is
recorded as such.
- C# reference: the same operation on `SingleFileBackingStore` (`CultCache.cs:3268ff`), with the same names
across runtimes.
- **Verification (Rust and C#):**
- a mismatch preserves the foreign bytes exactly and writes the replacement;
- an undecodable current is preserved and replaced;
- an I/O read failure (a directory at the path) changes nothing and leaves no staging file;
- a staging failure (size cap) changes nothing and sets nothing aside;
- the path is readable at every instant: a reader thread loops `pull_all_read_only_snapshot` through 1,000
exchanges and never sees an absent file or an empty snapshot;
- `.previous` rotates only on a verified exchange;
- two set-asides in the same second get distinct names;
- mutation tools: `cargo mutants --in-diff`, and Stryker.NET on the C# diff.
- **R3b (parity, not blocking Odin):** cultcache-ts `single-file-messagepack-backing-store.ts:34` and cultcache-py
`stores.py:83` gain the same operation and option, with a shared fixture test.
- **Estimate:** Rust about +140, C# about +120, R3b about +200.
### O2. A foreign store file while serving is set aside, not fatal (Odin)
- **Repo/branch:** Odin `hands/odin-store-repair` from O1, with a CultLib pin bump to R3. Needs Q-R1 and Q-R4.
- **Deletes first:** `ForeignStoreWrite` (`lib.rs:630-641`) and its arm in `survive` (`main.rs:884`). The test
`a_flush_writes_only_changes_and_never_over_a_foreign_file` (`lib.rs:2055-2090`) and the `survive` tests at
`main.rs:2842-2844` and `:3110` pin the old rule: restate them, do not delete them.
- **Changes:**
- `MemoryOdinTopologyStore::flush` (`lib.rs:613-626`) calls `replace_snapshot_setting_aside(flushed, current)`.
On `SetAside` it records a `displaced` event and counts it toward the contested bound. On an I/O failure it
records `unpersisted` and returns the error, which `survive` logs and retries as today.
- `RuntimeState::flush` (`main.rs:307-314`) keeps its lease check immediately before the write.
- The Q3 B per-put path uses this same flush. Hands re-anchors on the Q3 B head.
- **Adds:**
- The `odin.store_condition` record (schema in `odin-core` `documents.rs`), held in W. It carries:
- `generation` and `written_by`, updated inside the flush before the snapshot is taken;
- `unpersisted_since` and `last_write_error`;
- `contested_since`;
- a bounded list of the last 16 recovery events: kind, time, displaced path, sha256 and bytes, source, records
loaded, and the outstanding list (bounded).
- One arm in `public_document` (`main.rs:590-608`) that serves exactly this type. Providers still cannot write
it: `is_peer_document_type` is unchanged.
- **Rules that must die under their own mutation:**
- a flush never leaves the store dirty (the condition record is written inside the flush, not after it);
- a set-aside is preceded by a lease check;
- the third set-aside inside the window stops writing, and the second does not;
- after `contested`, no file is written or created;
- on an I/O failure, nothing is set aside;
- the condition is served while writes fail.
- **Authority map:** as in section 4. The deletion line is the `ForeignStoreWrite` type and arm.
- **Verification (fault injection, each at the file layer, not a log line):**
- T-A: activate, serve, write a different valid snapshot behind Odin. After the next pass the file decodes to
W, the displaced file's bytes equal the foreign bytes, `serving_pass` returned `Ok`, the catalog is unchanged,
and the served condition names the displaced file and its sha256.
- T-A2 (P2b): flip one payload byte behind Odin; the same outcome, reason `mismatch`.
- T-B: truncate the file behind Odin; the same outcome, reason `undecodable`, and
`pull_all_read_only_snapshot` (Idunn's crossing) decodes the file after one pass.
- T-B-latency: with no puts, the set-aside happens within one heartbeat plus one flush interval.
- T-contested: three foreign writes inside the window make exactly two set-asides and a third `contested`
report. After that the file and directory are untouched by further passes, and Odin still serves.
- T-D: replace the store path with a directory (EISDIR, which works as root). Nothing is set aside, no staging
file remains, the condition is `unpersisted` and is served, and a put is not ACKed. Remove the directory, and
the next pass writes W, clears `unpersisted` and ACKs.
- T-lease: the lease withdrawn before a flush that would set aside means no link, no write, and Odin ends
(`WriteLeaseLost`).
- Negative: `rg -n "ForeignStoreWrite" crates` matches nothing; `rg -n "remove_file|remove_dir" crates/odin-daemon/src`
matches nothing new outside tests.
- `cargo mutants --in-diff`.
- **Estimate:** about -40 / +160.
### O3. Activation recovers a corrupt store (Odin)
- **Repo/branch:** Odin `hands/odin-store-activation` from O2. Needs Q-R2 and Q-R3.
- **Change:** `MemoryOdinTopologyStore::load` (`lib.rs:519-538`) becomes the recovery ladder:
| `load` result | Outcome |
|---|---|
| `Ok` | as today |
| absent | empty, as today |
| `Unreadable::Undecodable` | try `pull_previous_generation()`, then empty (Q-R3); `flushed` becomes a sentinel meaning "the file is not a snapshot Odin can compare", so the first flush's exchange sets it aside |
| `Unreadable::UnsupportedFormat` | return the error: refuse |
| `io::Error` | return the error: refuse |
Activation itself never writes. The first `serving_pass` flush (`main.rs:833-842`, due at once) writes W. That
set-aside is recorded under the activation event and is not counted toward the contested bound.
- **Adds:** the outstanding-republish list. It holds the peer-type records present in the recovered previous
generation, each cleared when its publisher re-puts it. After 10 minutes the remainder is marked "not
republished", which is the live answer to "who is publish-once". For `started-empty` the list is `unknown`.
- **Rules that must die under their own mutation:**
- Unsupported-format and I/O failures never set aside, never fall back, and never start.
- A recovered activation writes before it serves its first catalog snapshot, or it serves the recovered set and
marks the condition before the first write. Hands picks one and states it.
- **Verification:**
- T-C1: a truncated `topology.cc` and a valid `.previous` activate with the previous content. The first pass
writes it and sets the corrupt file aside byte-exact, and the condition is `recovered-previous` with its
outstanding list. Re-putting one listed document clears it.
- T-C2: both undecodable (Q-R3 A) start empty. Both files are preserved, the condition is `started-empty`, and
the next correlation sequence is above the floor.
- T-C3: a v4 header refuses to activate. The file is byte-identical afterwards, and there is no displaced file
and no `.previous` change.
- T-C4: a directory at the store path refuses, as C3.
- T-restart: a recovered Odin stopped and restarted loads its own first write normally, with no second
set-aside.
- `cargo mutants --in-diff`.
- **Estimate:** about -10 / +110.
### R2. Single-file decoders reject trailing bytes, in every runtime (CultLib)
- **Repo/branch:** CultLib `hands/cultcache-trailing-bytes`. Independent of Odin; land it before O3 ships so that
an appended file is caught as undecodable, not overwritten silently (P2).
- **Change:** C# `DeserializeSnapshot` (`CultDocumentMessagePackSerialization.cs:179-215`) checks `reader.End`.
Rust `decode_store_snapshot` (`lib.rs:2583ff`) and the legacy path check that the slice is consumed. TS and
Python do the same. **One release**: a runtime that accepts what another rejects is a parity split.
- **Verification:** one shared fixture (a valid store plus one byte) is rejected as undecodable by all four
runtimes, and the valid store is accepted by all four.
- **Estimate:** about +40 plus tests across four runtimes.
### X1. Runbook and recovery drill (gamecult-ops, then live)
- **Repo:** gamecult-ops `runbooks/odin-yggdrasil.md` gains a "Store recovery" section:
- how to read `odin.store_condition` from the catalog;
- what displaced and previous files mean, and that disposing of them is the operator's decision;
- the manual restore: engage Odin's lifecycle brake, stop the `idunn-odin-*` unit, set the live file aside with
`ln` (never `rm`), extract the snapshot's `topology.cc` with its owner, mode and `.lock`, release the brake,
and read the condition. The sequence floor makes an old watermark safe.
- **Drill D-1 (off-live, after O3; needs the Q2-style approval, Q-R6):** extract the newest Yggdrasil snapshot's
`topology.cc` into a container on Yggdrasil. Activate a test-harness Odin on it through the existing fixture
authority. Then truncate the copy and activate again with the extracted file as `.previous`. Report the records
per type, the outstanding list and the sequence floor. Delete the copy afterwards.
- **Drill D-2 (live, operator-scheduled, Q-R6), after O3 deploys:**
1. Record the baseline: catalog types, Idunn correlated targets, route timeouts.
2. Engage the lifecycle brake and stop Odin.
3. `ln topology.cc /var/tmp/odin-drill-keep.cc` (the evidence of the evidence), then truncate `topology.cc` in
place to half.
4. Release the brake and let Idunn restart Odin.
5. Pass when all of these hold:
- Odin is serving within one restart;
- the condition is `recovered-previous`, with a displaced file whose sha256 equals the truncated file's;
- every Odin-correlated target is Ready again within 60 s;
- there are 0 route-challenge timeouts in the next hour;
- the catalog's type set equals the baseline after 10 minutes, or the outstanding list names every
difference.
6. Then run D-2b, a foreign write while serving: `ln` the live file aside, then truncate it. It passes when the
condition shows `displaced` within about 6 s and Idunn's correlation reads never fail for longer than that
(the journal).
## 7. Subtraction ledger (estimate)
| Cut | Removed | Added | Formats and targets |
|---|---|---|---|
| R1 | about 30 | about 70 | none |
| O1 | about 5 | about 25 | none |
| R3 | 0 | about 260 (Rust 140, C# 120); R3b about 200 | two sibling-file conventions (`.displaced.*`, `.previous`); no format change |
| O2 | about 40 | about 160 | one Odin schema, `odin.store_condition.v1` |
| O3 | about 10 | about 110 | none |
| R2 | 0 | about 40 plus tests | none (it tightens the decoders) |
The campaign is net additive, at about +570 without R3b. It buys ends 1-5, which the operator asked for, and
removes one fatal path and one silent path. Every addition in CultLib is a capability a general consumer of the
single-file store would expect, and two of them (staging cleanup, and the header that survives truncation) are
defects.
## 8. Operator questions
- **Q-R1. A store file changed underneath a running Odin (decodable or not).**
- Context: Idunn fences the old Odin before granting the lease, so the other writer is never an Odin. It is a
tool, a hand edit, a restore made against the runbook, or disk corruption; P2b shows a flipped byte looks
exactly like a foreign write. Today a decodable change ends Odin, and the restart adopts the foreign file
silently. An undecodable change stops persistence forever and breaks Idunn's crossing.
- A: set the file aside (kept byte for byte, never deleted), write Odin's working set over it, keep serving,
and report it as typed state.
- B: set it aside, write the working set, then end Odin so a fresh process starts from the rewritten file.
- **Recommended: A.** B's restart protects nothing that the lease check before the write does not already
protect, and costs a route outage.
- **Q-R2. Keep one previous generation on disk (`topology.cc.previous`)?**
- Context: without it, a store that is corrupt at activation has no on-host source except "empty", and the
backup is up to 24 h old and needs the operator. It costs one hard link per write (a directory update, no data
written) and one more file of the store's size. It is a CultLib option, off by default for other consumers.
- A: yes. B: no; activation recovery goes straight to empty.
- **Recommended: A.** It is the only source that recovers publish-once documents, presence history and
correlations without the operator. Its loss bound is one write.
- **Q-R3. Activation finds nothing decodable: no usable `topology.cc`, no usable previous generation.**
- Context: presences and most catalog documents return within seconds to a minute, and correlations re-derive
in one refresh pass. Publish-once documents (if any exist, see 1.3) stay missing until their publisher
re-puts them, or until the operator restores from the backup.
- A: start empty automatically, set the corrupt file aside, and report `started-empty`.
- B: refuse to start, so Odin crash-loops until the operator restores. Every service's rendezvous is down
meanwhile.
- **Recommended: A.** A down Odin loses everything the Verse needs; an empty one loses only what nobody
republishes, and the restore path stays open.
- **Q-R4. A foreign writer keeps writing (a third set-aside within 10 minutes).**
- Context: repeated repair against a live writer makes a displaced file every second.
- A: stop writing, keep serving from memory, withhold put ACKs, and report `contested` until the operator
finds the writer. The working set survives.
- B: end Odin. Idunn restarts it, and the restart loads whatever the foreign writer last wrote.
- **Recommended: A.** B hands the store to the unknown writer. A is loud and loses nothing that is in memory.
- **Q-R5. Should a store condition change Odin's presence to `degraded`?**
- Context: a non-`active` Odin presence makes every managed dependent of Odin not Ready (`lib.rs:946`,
`dependency_evidence`). The store condition does not affect serving.
- A: no. The presence stays `active`, and the condition record is the signal.
- B: `degraded` while `unpersisted` or `contested`.
- **Recommended: A.** Otherwise a full disk becomes an outage of every dependent, and a restart that Idunn might
take in response would destroy the only complete copy.
- **Q-R6. The drills.**
- D-1: decode a copy of last night's backup `topology.cc` in a container on Yggdrasil, the same terms as Q2 for
the write map, with the copy deleted afterwards.
- D-2: after O3 deploys, a scheduled live drill that truncates the real `topology.cc` behind the lifecycle brake,
keeping a hard link of the original.
- **Recommended: yes to both**, D-1 before O3 ships and D-2 in a window the operator names.
## 9. Follow-ups outside this map
- **I-F1 (Idunn):** a deploy or a stop of Odin while `odin.store_condition` says `unpersisted` or `contested`
destroys the only complete copy. Idunn could read the condition from the catalog and hold the deployment
(deployment brake, never continuity). This is Idunn's decision, raised with the Idunn map.
- **I-F2 (Idunn):** "Idunn stopped restarting admitted odin" is a log line (`report_once`, `control_plane.rs:4599`),
not typed state. It belongs to Idunn's own operator surface.
- **F3 (from the write map):** the Eve lowering of `odin.store_condition` belongs to Odin's operator surface.
This map publishes the typed record only.
- **H-F1 (Heimdall):** confirm whether the deployed Heimdall re-puts its Odin documents every 60 s. O3's
outstanding list measures it after any recovery.
- **Q3 B's record in the write map:** "an ACK means durable" holds against crashes, not against a corruption at
activation, which loses at most the last write (C1).

View file

@ -3,86 +3,7 @@
**2026-09-30, Self.** Operator: "Let's fix Odin up, that's some furious bookkeeping right there. Look at Odin's
write pattern and see if append only or per-document is the way to go." Answer: neither is the first fix. Cuts 1-2
go to Hands now. Q3 (a 1 s interval) and Q4 (the Rust directory store as a CultLib follow-up) are Self defaults,
and the operator may overrule them. **Operator, 2026-09-30: "Q1: yes", so CultMesh refuses puts it could never
serve and Cut 3 goes ahead; "Q2: yeah, of course", so a read-only decode of a copy of live `topology.cc` is
approved. The copy is taken with `sudo cp` and decoded off the live path, and the copy is deleted afterwards.**
**Soul on Cuts 1-2, and two operator rulings (2026-09-30).**
- **Q3 restated.** The premise "puts carry no application acknowledgement" was false. The cultmesh-rs server
ACKs a put only after `accept_raw_document` returns Ok, so after Cut 1 an ACK meant "in memory", and a SIGKILL
lost acknowledged puts in 6 of 6 probe runs.
- **Operator ruling: "Q3: B".** Peer puts are fsynced before they are acknowledged. Only Odin's own
bookkeeping (heartbeats, correlations) waits for the 1 s interval. An ACK means durable.
- **An undecodable foreign write.** Today Odin stays up, serves from memory and never persists again. Self offered
three options: degraded but loud, end Odin, or unchanged.
- **Operator: "None of these options involve repair and recovery, which is what ultimately needs to happen."**
- Recovery is mapped as its own pass. It covers an undecodable foreign write, a decodable one, and a failed
load at activation.
**Soul on write batch 2 (2026-09-30).** Q3 B changes the expected effect. Cut 2's row in section 6 ("at most
3.5 MB/s, at most 1.05 writes/s") assumed Q3 A and is **superseded**. Under Q3 B, each peer put (presence
heartbeats included) writes the store once.
- With the live-shaped 3 MB store that is about 2.5-3 writes/s, about 7.5-9 MB/s, and 70-170 ms of the loop
blocked each second.
- Measured: route-challenge p99 461 ms and 11 kernel drops.
- After Cut 3's Odin half the store is about 80 KB and a put write is p50 about 4 ms.
- **Cut 2 therefore deploys only together with Cut 3's Odin half**, which is now being built on `hands/odin-write`.
**Q2 decode result (Eyes, 2026-09-30; full table in the CultLib session scratchpad `odin-topology-decode.md`).**
- 77 records; 3,222,550 B of payload in a 3,240,078 B file.
- The two "~1.55 MB records" are per-type totals. Each type is dominated by one Ghostlight operator surface:
`eve:operator:ghostlight.campaign.27d444b3-...` (1,456,210 B) and its `operator-state` twin (1,456,341 B).
Together they are 89.9% of the file.
- Publisher: provider `gamecult.ghostlight.dungeon`. 99.5% of each record is one Eve `code` node holding a
pretty-printed JSON dump of the whole campaign state.
- These records are dead residue. They were stored 2026-09-23 and have not been rewritten since. Ghostlight `HEAD`
(`eeb3a68`) no longer contains the emitter (last touched at `6bb6869`, 2026-09-02).
- **So Q1's refusal breaks no live publisher.** Cut 3's Odin half drops exactly these two at activation. Every other
surface is 70 KB or less.
- Correction to section 3: the 1 MiB snapshot refusals in the journal are per-type responses, not per-record.
Cut 3's per-document bound is still right: each of these records is over 1 MiB alone. A type whose many small
records add up past 1 MiB would still be unservable as a whole-type snapshot. That is a follow-up for CultMesh
(paged snapshot responses), not part of this campaign.
- Presence history: 41 records, 42 KB, 37 of them withdrawn. F4 is a slow leak, not a size problem.
- The current `streampixels-web` incarnation has a correlation but no presence record; its last presence was
09-29 19:53 UTC. This matches Idunn's "not Odin-correlated" state.
**Soul on write batch 3 + Cut 3 Odin half (`f4a1f89..726e0f3`, 2026-09-30): not fit to deploy.**
- **S-1 (high; operator question Q5, below).** Every write, the fsync pair included, runs on the serving loop
(`main.rs:358` → `flush` → `compare_exchange_snapshot`; the interval flush at `:883` is on the same thread).
- A disk stall passes into route-challenge latency one for one. A probe that held the store lock measured: a 0 ms
stall gave 86 ms, 300 ms gave 288 ms, 1,500 ms gave 1,487 ms.
- Interleaved runs (ABBA) under IO pressure of 5-38% gave a challenge p99 no better with Cut 3 than without it.
- Cut 3 halves the bytes written, not the stalls. **The premise that the store's size caused the stalls was
wrong; they come from the disk.**
- **S-2 (high).** Q1's put refusal has not landed. Odin's pin `3bf1c0ce` has no put bound, so a 1.2 MB put is
ACKed, fsynced, and dropped at the next activation. It waits on the CultLib put-serve merge, then a pin bump with
a `served_record` override.
- **S-3 (medium).** `attempted` grows about 98 KB per failed write, roughly 0.9 GB/h on a full disk, which ends in
OOM.
- Fix in the owner: cultcache-rs returns a typed `NotReplaced`/`ReplacedNotDurable` write outcome (added to CultLib
R1 batch 3), and Odin keeps at most one candidate.
- **S-4 (medium).** Two admitted presences for one target, which a provider in a deploy window produces, refuse the
whole catalog (`main.rs:~628`).
- **S-5 (medium).** A RUDP transport ACK is recorded at receipt, so with pipelined puts a later put is ACKed while
an earlier one is still held. With the first datagram lost, the second is never delivered. That is data loss
under the QUIC-pivot rule; it is tracked in the CultLib ack map. `dcd36fc`'s doc line overstates.
- **S-6, S-8.** Test gaps (the limit taken through `main`; `remove_not_dirty`).
- **S-7.** The drop log echoes the key. Self ruled keys and schema ids are identities and may be logged, as in
CultLib R1.
- **Held:** F1's logic, F4's projection fallback, F5, F3's refusal, and Cut 3's activation bound, drop and removal.
Sizing uses `peer_document`, so it matches what is served.
- **Batch 4 in Hands:** S-4, S-6, S-8, the S-5 doc line, the attempted-list fixture gap, and committing Soul's
probes. S-1 waits on Q5; S-2 and S-3 wait on CultLib.
**CultLib side landed (2026-09-30).** Ack Cut D was merged at `2f06ef84`, R1 at `daedfdd6` and the put-serve bound at
`9dcb3aa1`. That unblocks the Q5 sequence:
1. R1 batch 3's typed write outcome is on main. It carries `Rejected` as well as `NotReplaced`/`ReplacedNotDurable`.
W0 must classify by kind together with the `io::Error`, never by `NotReplaced` alone.
2. **C5 next:** cultmesh-rs deferred put reply, on main.
3. **W0:** one Odin pin bump. It closes S-2 (put refusal with a `served_record` override) and S-3 (delete
`attempted`), and it brings in Cut D's and the put-serve error texts.
4. **W1:** the writer thread.
and the operator may overrule them. Q1 and Q2 are put to the operator, and Cut 3 waits on Q1.
Status: cut map, Imagination pass 1 (Opus), 2026-09-30. Nothing has landed. Anchors are Odin `main` at
`44951a1` (the deployed build, live on Yggdrasil since 2026-09-30 04:20:48 UTC) and CultLib `3bf1c0ce`
@ -431,532 +352,6 @@ records; update the check's type list in the same pass.
- **Q4. CultLib parity: port `DirectoryMessagePackBackingStore` (v4) to cultcache-rs?** It is not needed by
Odin after Cuts 1-3. A reasonable consumer of the Rust runtime would expect the reference's store.
**Recommended: record it as CultLib follow-up F1 with no deadline.** It is independent of this map.
- **Q5 (2026-09-30, from Soul S-1). Where does the durable write run?** Today the serving loop runs the fsync, so
any disk stall stalls route challenges, catalog reads and every other put.
- A: a writer thread with group commit.
- The loop hands each accepted put to one writer thread. The writer writes and fsyncs everything queued since
its last write in one go, then the loop sends those puts' replies.
- Challenges and reads are never behind the disk. A stall delays only put replies, and Q3 B still holds (a
reply means durable).
- It needs cultmesh-rs to let a put handler reply later (a CultLib API change) plus one Odin thread.
- B: answer route challenges on their own path. Everything else still stalls, and the challenge would report
Odin healthy while its catalog is stuck.
- C: reopen Q3 and go back to the 1 s interval. That brings back lost acknowledged puts on a crash.
**Recommended: A.** **RULED A by the operator, 2026-09-30: "agree on Odin writer thread".** Mapping is next
(Imagination): the cultmesh-rs deferred put reply, then the Odin writer thread with group commit. Mapped below in
"Q5 cut: the writer thread" (cuts C5, W0, W1; forks Q5-A and Q5-B).
## Q5 cut: the writer thread
**Imagination pass (Opus), 2026-09-30.** Maps the operator's ruling on Q5, "agree on Odin writer thread" (option A,
section 7). Anchors:
- Odin `hands/odin-write` at `434756e` (`crates/odin-daemon/src/main.rs`, `lib.rs`), CultLib pin `3bf1c0ce`.
- CultLib `hands/cultmesh-put-serve-bound` tip `e4e2a0b9` (`packages/cultmesh-rs/src/rudp_document_server.rs`).
Its merge base is `42ba0f08`, which already holds ack Cuts 1, 1b and 1c.
- CultLib `hands/ack-cutD` tip `e9d5ae2c` (lands on main first), and R1 batch 3 `hands/cultcache-read-failures` tip
`93ac5945` (`CultCacheStoreWriteFailed`).
The mapping raises two forks, **Q5-A** and **Q5-B** (at the end of this section). Cuts C5 and W1 wait on Q5-A.
**RULED, operator 2026-09-30: "Q5-A: A, but make a note of this for when we move on QUIC, Q5-B: A".**
- **Q5-A = A.** The server withholds the session's transport ACK until the put is answered. The wire is unchanged.
The note for QUIC (a put-result message) is in the QUIC campaign handoff.
- **Q5-B = A.** Puts are held during a writer stall, bounded by the existing session and byte limits.
### Q5.1 Body facts (probed and read)
**Probe O1, Odin `434756e` + scratch test, Yggdrasil slot 3 (`imag-q5-odin@31953c93`).** A test process holds the
store's sibling lock for 1,500 ms, the way Soul modelled a disk stall. Full serving passes run meanwhile. A route
challenge is sent at 200 ms, then a catalog read.
| Case | Route challenge | Catalog read |
|---|---|---|
| nothing waiting to be written | served in **39 ms** | served in 29 ms |
| only Odin's own heartbeat waiting, and the interval write due | served in **1,342 ms** | 33 ms (sent after the release) |
- No read path takes the store lock. At `434756e` the lock is taken only by `MemoryOdinTopologyStore::load`
(`lib.rs:509`, at activation) and by `flush` (`lib.rs:612` → `compare_exchange_snapshot`).
- The stall reaches the loop through **both** writes on it: a peer put (`main.rs:363`, Soul S-1) and the interval
write of Odin's own bookkeeping (`main.rs:933-941`). The interval write stalls the loop with no peer put at all.
So the bookkeeping write must move to the writer too, not only the put write.
**The loop still reads two other files.** They are not the store and take no lock, but they are disk reads on the
loop:
- The lease file, on every route challenge and catalog read: `raw_snapshot` `main.rs:389-409` →
`require_current_write_lease` `:546-567` → `read_process_write_lease` `:1323` (`pull_all_read_only_snapshot`).
- Idunn's projection, on every catalog read (`:608`), refresh (`:572`) and heartbeat (`:524-544`).
- Both files are small and rewritten rarely by Idunn, so they are normally served from the page cache. Probe O1
cannot show whether a real disk stall evicts them. Only the Yggdrasil rerun (Q5.6) can. If it does, the answer
is to hold the lease record in memory and re-check it where it is decided. That is its own follow-up, not this
cut.
**What a put's reply is on the wire today.** CultNet has no put-acknowledgement message: the list in
`cultnet-rs/src/contracts.rs` has `document_put_raw` and `error.v0`, and no put result.
- A Rust publisher (`publish_cultnet_message_to_rudp_catalog`, `cultmesh-rs/src/lib.rs:871-924` at `e4e2a0b9`)
succeeds when its reliable packet is acknowledged by the RUDP transport, and fails on a `cultnet.error.v0`.
- Its default `flush_timeout` is 300 ms, with a resend every 50 ms (`:144-157`).
- The document server sends that acknowledgement at the end of `poll_once`, after the sink returns
(`rudp_document_server.rs:468-476`). A refusal ends the session with an Error frame that acknowledges nothing
(`end_rejected_session` `:760-793`, put-serve branch).
- **So today the put's reply is the transport acknowledgement.** That is how Q3 B's "an ACK means durable" holds:
the sink writes before it returns.
**Every packet a session sends carries acknowledgement fields** (`ack`, `ack_mask`, from `ack_state`, cultnet-rs
`rudp.rs` at main).
- The fields report every *received* reliable sequence, not only delivered ones.
- `create_packet` stamps them on each new packet: data, Pong, the end-of-poll Ack.
- A reliable packet is stored with the fields it was created with, and resent unchanged (`track_reliable`,
`due_resends`).
Probes P1-P4 (cultnet-rs sessions at `e4e2a0b9`, scratch test `imag_q5_probe.rs`, Yggdrasil, `imag-q5-cultlib@8dcfe36a`):
| # | Case | Result |
|---|---|---|
| P1 | two pipelined ordered puts, the first lost | the second is delivered to nobody, yet `create_ack` acknowledges it. **S-5 is still open after ack Cuts 1-1c.** `ack_state` at main is the same code, so the ack map's to-do resolves to "not closed by the pin bump" |
| P2 | a put delivered, then a Ping | the Pong acknowledges the put |
| P3 | the resend of a reply created **before** the put | does not acknowledge it |
| P3 | a reliable reply created **after** the put | acknowledges it, and so does its resend |
| P4 | a retransmit of a delivered put | not delivered again (the server's end-of-poll ack still answers it, `:468`) |
Consequence for a reply sent later: holding back only the end-of-poll ack is not enough.
- A Pong, or a snapshot response created on the same session after the put arrived, acknowledges the put too.
- The snapshot response keeps doing so on every resend, even if its first copy were patched.
- So a session with a put awaiting its answer must be **sent nothing created after that put arrived**. Resends of
earlier packets are safe: they carry the fields they were created with.
**The other runtimes (parity).**
- **TS** (`cultmesh-ts/src/index.ts:5891-5914` at `e4e2a0b9`):
- It acknowledges each packet at receipt (`createAckForReceived`), before the handler runs.
- It runs each session's frames through one promise chain (`record.work`), so handling is serial per session.
- `onDocumentPutRaw` may return a Promise and an application-level receipt document, which is sent when it
resolves (`:5951-5972`).
- So TS already "replies later", but at the application level, and its transport ack does not mean handled.
- **Python** (`cultmesh-py/src/cultmesh_py/server.py:407`, `_handle_raw_put` `:415`): the handler is synchronous
and returns its responses.
- Neither has a durability meaning for its ack. Parity is follow-up **F-Q5a** (Q5.8).
**Consumers of the sink trait** (`CultMeshRudpRawDocumentSink`):
- Odin's `SinkHandle` (`main.rs:168-172`).
- Ratatoskr's catalog test (`Ratatoskr/crates/ratatoskr-core/tests/catalog.rs:29`).
- CultLib's own tests, which use closures through the blanket impl (`rudp_document_server.rs:60-67`).
- Vendored copies (StreamPixels, Heimdall) do not implement it.
**cultcache-rs write outcome (R1 batch 3, `93ac5945`).**
- Every failed single-file write carries `CultCacheStoreWriteFailed { kind: NotReplaced | ReplacedNotDurable }`.
- Any other error from `compare_exchange_snapshot` (lock, read, decode, validation) comes before the write step,
so the file is unchanged.
- One hole: `with_exclusive_lock` (`cultcache-rs/src/lib.rs:924-932` on that branch) returns the **unlock** error
after a write that succeeded, with no marker. Odin would read that as "not replaced". Its next write would then
find its own snapshot and report `ForeignStoreWrite`, and Odin would end.
- That is rare (unlock failing), and it belongs to R1: follow-up **F-Q5b**, not this cut.
### Q5.2 Target shape
1. **CultLib (cut C5).** A put's sink may answer later.
- While any put of a session awaits its answer, the server sends that session nothing new.
- The one exception is resends of packets created before the put arrived.
- A snapshot request behind the pending put waits in the session's order.
- An accepted answer releases one ordinary acknowledgement.
- A refusal ends the session with the Error frame the put-serve branch sends.
- The wire does not change, and existing publishers keep "acknowledged means the sink accepted", which for Odin
is durable (Q5-A).
2. **Odin (cut W1).** One writer thread owns the store file after activation.
- The loop owns the working set, decides when to write, and answers put replies from the writer's outcomes.
- Every put accepted while a write is in flight is covered by the next single write (group commit).
- Route challenges and catalog reads never wait on the writer.
3. **Odin (cut W0), before W1.**
- A pin bump.
- S-2: a `served_record` override so Odin's admission is exact.
- S-3: the `attempted` list dies, because the typed write outcome says what the file holds.
### Q5.3 Identity, lifecycle, authority
| Kind | Named by | Over time | Decided by |
|---|---|---|---|
| pending put (CultLib) | a server-minted `u64` put id, unique for the server's life, never reused across session generations | created when an admitted put is offered to the sink; ends when it is answered, or when its session ends (the answer is then discarded) | the sink answers; the server alone sends |
| put reply handle (CultLib) | the put id, inside a `CultMeshRudpPutReply` | owned by whoever the sink hands it to; consumed by `accept` or `refuse`; dropping it unanswered refuses | the sink's owner |
| working-set version (Odin) | a `u64` counter in `MemoryOdinTopologyStore` | +1 on every mutation that changes a record; never reset within an activation | the loop |
| written-through version (Odin) | a `u64` in `RuntimeState` | raised only by a writer outcome `Written { version }` | the writer's outcome, applied by the loop |
| write request (Odin) | its version | at most one in flight; replaced by nothing (the next request is built when the outcome arrives) | the loop submits; the writer executes |
| what the file holds (Odin, `flushed`) | none: one snapshot | set at activation from the load; replaced on `Ok(true)` and on `ReplacedNotDurable` | the writer alone |
"Dirty" stops being a flag (`lib.rs:502`, `dirty: Cell<bool>`): it is derived as `version > written_through`.
### Q5.4 Cuts
#### Cut C5. A put's sink may answer later (CultLib, cultmesh-rs, behaviour)
- **Repo/branch:** CultLib `hands/cultmesh-put-reply-later` from `main` after the put-serve branch merges (Q5.7).
Anchors below are at put-serve `e4e2a0b9`; they move by Cut D's diff (Q5.7 names the conflicts).
- **Deletes first:** the synchronous sink call and its error mapping in the put arm, `rudp_document_server.rs:621-633`
(they are replaced, not wrapped). Nothing else is deleted.
- **Public API** (default D-1 below):
```rust
pub trait CultMeshRudpRawDocumentSink {
/// Answer through `reply` now or later, from any thread.
fn accept_raw_document(&mut self, receipt: CultMeshRudpRawDocumentReceipt, reply: CultMeshRudpPutReply);
}
// Closures keep working and answer at once:
impl<F: FnMut(CultMeshRudpRawDocumentReceipt) -> Result<()>> CultMeshRudpRawDocumentSink for F { .. }
#[must_use] pub struct CultMeshRudpPutReply { /* put id, Sender to the server */ } // Send, !Clone
impl CultMeshRudpPutReply {
pub fn accept(self);
pub fn refuse(self, reason: impl std::fmt::Display);
}
// Drop without an answer refuses: "the put was not answered".
```
- `CultMeshRudpRawDocumentReceipt` (`:37-43`) is unchanged; the reply is a separate argument, so the receipt stays
`Clone + PartialEq`.
- `CultMeshRudpPollOutcome` (`:286-290`) is unchanged. A later refusal is returned as `ApplicationRejected` with
`SinkRefused(reason)` from the `poll_once` that sends it.
- **Server changes, `rudp_document_server.rs`:**
- `CultMeshRudpDocumentServer` (`:303-311`) gains:
- The answer channel (`std::sync::mpsc`: the server keeps the receiver, each reply holds a sender).
- `answered: VecDeque`, the answers taken off the channel and not yet handled.
- A next put id.
- `pending: BTreeMap<u64, CultMeshRudpSessionKey>`.
- `SessionEntry` (`:292-297`) gains `pending_puts: BTreeSet<u64>` and `waiting: VecDeque<(u32, CultNetMessage)>`,
the delivered frames that may not be answered yet.
- **One predicate owns "withheld":** `fn acknowledgement_withheld(&self, key) -> bool`, true while
`pending_puts` is non-empty. The S-5 fix, if the ack map takes it here, adds its clause to this function and
nowhere else.
- `poll_once`:
- Drain answers first (before `maintain`, `:366`). Each answer is looked up in `pending`; an unknown id (its
session ended) is discarded.
- Refuse → `end_rejected_session(key, reason)`, and return one `ApplicationRejected`. Further answers stay in the
channel for the next poll.
- Accept → remove the id. When the session has none left, it is released: send its ack (the rule for which
poll sends it is under the put arm below), then handle `waiting` in order.
- `:436-438`: a `result.reply` (Pong) is not sent while withheld.
- `:468-476`: the end-of-poll ack is not sent while withheld.
- `deliver_application_message` (`:565`):
- Put arm (`:572`): admission is unchanged (put-serve `:587-619`). Then mint the id, record it, and call the sink
with a reply.
- `poll_once` looks at answers again after the delivered-frame loop (`:445-466`), before the end-of-poll ack,
but handles only those for this poll's session. The rest stay queued for the next poll's first step. (Answers
move from the channel into one server-side queue, so they can be taken selectively.) A sink that answered at
once is therefore acknowledged, or refused, in the same poll with the same datagrams as today, and another
session's refusal can never cut this session's ack short.
- A release sends `create_ack()` from the drain, except for the session whose packet this poll is handling:
that session gets its ordinary end-of-poll ack, so a synchronous sink's datagrams stay exactly as they were.
- Snapshot arm (`:634`): if the session is withheld, push the request onto `waiting` and return; otherwise
answer as today.
- A put that arrives while `waiting` is non-empty also queues, so a session's frames are handled in order. A
put behind another put that is only pending is offered at once, so pipelined puts can share one write.
- Session end, wherever a session is removed (`:441`, `:486-489`, `:523`, `end_refused_session`,
`end_rejected_session`, Cut D's `end_unsendable_session`): remove its ids from `pending`, so their answers are
discarded.
- **Put this in one `remove_session(key)` helper, so no removal path can forget it.** It is the one new helper
the cut earns: there are six removal sites today.
- Doc comments: the struct doc (`:300-302`, "no background thread") stays true. The sink trait doc says what
"acknowledged" means: the sink accepted.
- **Authority map:**
- Owner of what the peer is told about a put: the server, and only through `poll_once`. The sink decides accept or
refuse; it never sends.
- Inputs: sink answers (channel); transport receipts.
- Outputs: one ack per release; one refusal per refused put's session.
- Derived: "withheld" is derived from `pending_puts`; it is not a flag.
- Forbidden writers:
- No send path to a withheld session except `maintain`'s resends.
- The sink must not be able to acknowledge by returning. The return type carries no answer, so this holds by
construction.
- Shared paths: a synchronous closure sink and a deferred sink answer through the same reply and the same drain.
There is no second code path for "answered at once".
- **Verification (CultLib, Yggdrasil):**
- Wire-level test, the observer layer: a raw client socket records every datagram the server sends to its session.
- With the reply held across at least five resend intervals, including a Ping and a snapshot request on that
session, no datagram acknowledges the put's sequence.
- After `accept()`, the next poll sends an ack that does, and then the snapshot response.
- Mutations that must fail it: sending the end-of-poll ack regardless; sending the Pong; answering the snapshot
request at once.
- Another session's snapshot request is answered while the first session's put is held (the loop is not blocked).
- `refuse()` and dropping the reply: the peer receives `cultnet.error.v0` and a goodbye, and nothing acknowledges
the put (reuse the put-serve refusal assertions).
- Two pipelined puts on one session, each with its own held reply:
- Answering the second first releases nothing.
- Answering the first then releases one ack that covers both.
- This pins that a later put's acknowledgement never leaves before an earlier one's.
- Session ends while held (peer Disconnect; a new Connect from the same key): a later `accept()` sends nothing to
the new generation, and `pending` is empty.
- Closure sink: the datagram sequence is byte-identical to the pre-cut server for accept and refuse. This pins that
the synchronous path did not change.
- An unservable put is refused before the sink sees it: the sink is never called and no reply is minted.
- `cargo mutants --in-diff` on the cut's diff; every survivor is triaged.
- **Estimate:** about +170 source, -15 source, +350 tests. No new target, dependency or wire field.
#### Cut W0. Pin bump, exact admission and the one-candidate store (Odin, subtraction first)
- **Repo/branch:** Odin `hands/odin-write-w0` from `hands/odin-write` (`434756e`, or batch 4's tip). It bumps the
CultLib pin to the main commit holding Cut D, put-serve, R1 batch 3 and C5 (Q5.7).
- **Deletes first:**
- `attempted` (`lib.rs:494-499`, `:525`), the retry loop over it in `flush` (`:619-625`) and its push (`:628-633`).
- The tests that describe a file holding some older failed attempt:
- `a_write_that_failed_before_replacing_keeps_every_earlier_attempt_odins` (`lib.rs:2172`).
- `the_file_may_hold_a_later_failed_attempt_not_only_the_first` (`:2205`).
- Each models a `NotReplaced` failure that nonetheless changed the file, which the typed outcome now rules out.
- **Changes:**
- `flush` (`lib.rs:612-640`) matches the error's `CultCacheStoreWriteFailed` kind (`downcast_ref`):
- `ReplacedNotDurable` → `flushed = current`, and the store stays dirty so the next write re-syncs.
- Anything else → no change.
- `a_write_that_failed_after_replacing_the_file_is_not_another_writer` (`:2141`) is re-targeted to inject
`ReplacedNotDurable` as the real outcome, not a hand-made replace.
- `a_failed_write_is_made_by_the_next_without_another_change` (`:2118`) stays.
- **S-2:** Odin's `SnapshotHandle` (`main.rs:174-181`) overrides `served_record`. It builds the envelope as
`persist_generic_document` (`:1462`) would store it, then serves it through `peer_document` (`:1400`), the
function `drop_unservable_documents` (`:1428`) sizes with. Admission and activation then use one sizing.
- A runtime-presence put keeps the default (as received). Presences are served in `public_document`'s shape and
are never near the bound, and `drop_unservable_documents` does not check them either.
- A schema that `persist_generic_document` would refuse returns an error, which the server turns into a
refusal before the sink.
- `accept_raw_document`'s doc (`main.rs:332-351`) drops its S-5 paragraph; it moves to W1's writer doc.
- **Authority map:** what the file holds is decided by the one write outcome, not by a list of guesses.
`ForeignStoreWrite` means exactly "a snapshot Odin did not write".
- **Verification:**
- Test: a `ReplacedNotDurable` failure, then a change, then a write → made over the failed snapshot.
- Test: a `NotReplaced` failure, after which another writer puts the failed snapshot into the file → `ForeignStoreWrite`.
Mutation: keeping a candidate fails it.
- Test: 1,000 failed writes leave nothing retained. This pins S-3; observe the store's retained records, or the
absence of the field.
- S-2 test: a 1.2 MB put is refused as `DocumentUnservable` and is not in the working set.
- Mutation: sizing with the received record instead of `peer_document` survives nothing.
- Negative: `rg -n attempted crates/odin-daemon/src` matches nothing.
- The pin bump carries ack Cuts 1-3 and D into Odin. The full Odin suite runs on Yggdrasil, and Soul reads this cut
as a transport change, not only a subtraction.
- **Estimate:** about -150 (including the two tests) / +60.
#### Cut W1. The writer thread (Odin, behaviour)
- **Repo/branch:** Odin `hands/odin-writer` from W0.
- **Deletes first:**
- `MemoryOdinTopologyStore::flush` (`lib.rs:602-640`), its `path` (`:490`) and `flushed` (`:493`) fields, and the
`dirty` flag (`:502`, `:600-602`).
- `RuntimeState::flush` (`main.rs:317-330`).
- The `self.flush()?` in `accept_raw_document` (`main.rs:363`).
- `stop`'s direct write (`main.rs:891-898`).
- The flush step of `serving_pass` (`main.rs:933-941`).
- No path is left on the loop that can write the file. That holds before the writer is added.
- **Keeps:**
- `MemoryOdinTopologyStore::load` (`lib.rs:509`): the one locked read, at activation, on the loop, before any writer
exists. `load` returns the loaded snapshot beside the store, to become the writer's `flushed`.
- `ForeignStoreWrite`, `WriteLeaseLost`, `survive` (`main.rs:979-994`), unchanged in meaning.
- **Adds, `crates/odin-daemon/src/writer.rs`** (a module, not a crate):
- `StoreWriter`: `spawn(file, flushed, lease) -> StoreWriter`, `submit(WriteRequest)`,
`try_outcome() -> Option<WriteOutcome>` and `finish(self)`.
- The channel is `sync_channel(1)`. The loop submits only when nothing is in flight, so `submit` never blocks.
- `WriteRequest { version, records }`.
- `WriteOutcome { version, result }`, where `result` is one of `Written`, `Failed(text)`, `Foreign` or
`LeaseLost(text)`.
- The thread:
- Checks the lease (the check extracted from `require_current_write_lease` `main.rs:546-567` into a free
function that the loop and the writer share), then `compare_exchange_snapshot(flushed, records)`.
- Applies W0's outcome rule to `flushed`, and sends the outcome.
- It never touches reply handles, the working set or the socket.
- Mock point: the thread writes through a one-method trait `StoreFile { compare_exchange_snapshot }`, implemented
for `SingleFileMessagePackBackingStore`. Tests gate it and count it. The lock-hold (Soul's method) is used where
the real file must be on the path.
- **Changes, `main.rs`:**
- `RuntimeState` (`:141-161`) gains `writer: Option<StoreWriter>` (spawned in `try_activate` after the load,
`:284-305`), `written_through: u64`, `in_flight: Option<u64>` and `replies: VecDeque<(u64, CultMeshRudpPutReply)>`.
- `SinkHandle::accept_raw_document` (`:168-172`) takes the reply.
- `RuntimeState::accept_raw_document` (`:352-367`):
- Validates and mutates as today.
- Pushes `(store.version(), reply)`, or accepts at once when `version <= written_through` (nothing new to make
durable).
- Refuses through the reply on a validation error.
- Submits if nothing is in flight.
- A new `serving_pass` step, in place of `:933-941`, run every pass:
- `try_outcome`. On `Written { v }`: `written_through = v`, and accept every reply with version `<= v`, in queue
order.
- On `Failed`: refuse the replies with version `<= v`; they remain in the working set (refusal runs one way, as
documented at `:344-350`).
- On `Foreign` / `LeaseLost`: return the error to `survive`, which ends Odin.
- Then, if nothing is in flight and `version > written_through`, submit when a reply is waiting or when
`FLUSH_INTERVAL` is due.
- So puts never wait for the interval; bookkeeping-only changes do.
- `stop` (`:894`):
- If the lease is still current and anything is unwritten or in flight, wait for the in-flight outcome, submit the
rest, and wait for that outcome too. This is a blocking wait; it is the only one, and it happens only on the
way out.
- Answer the replies, run one `poll_server` so the acks leave, then `finish` (drop the sender and join the
thread).
- After `WriteLeaseLost` or `ForeignStoreWrite`, write nothing. Dropping the replies refuses them.
- `FLUSH_INTERVAL`'s comment (`:73-77`) and the store's doc (`lib.rs:484-488`) say: the writer thread owns the
file; puts are covered by the next write; bookkeeping waits for the interval.
- **Authority map:**
- Owner of the store file after activation: the writer thread, the only holder of a
`SingleFileMessagePackBackingStore` for the store path. It also owns what the file holds (`flushed`).
- Owner of the working set, of when to write, and of put replies: the loop (`RuntimeState`).
- Inputs:
- Writer: one request at a time, and the lease file.
- Loop: provider puts, Idunn's projection, the lease file, and writer outcomes.
- Outputs: one atomic `.cc` snapshot per request; one outcome per request; replies answered in version order.
- Derived state:
- `dirty` is derived (`version > written_through`).
- `flushed` is no longer the loop's; it lives only in the writer.
- The interval decides only bookkeeping-only writes.
- Forbidden writers: no loop code path calls `compare_exchange_snapshot`, takes the store's lock, or answers a
put reply before the outcome covering its version.
- Shared paths: peer puts, self-presence, refresh, activation-time drops and the stop write all mutate the working
set, and reach the file only through `submit`.
- Deletion line: every loop-side write (above) is deleted before `writer.rs` is added.
- **Batching rule, stated once.** The loop keeps at most one write in flight. Everything that changed while it was in
flight, puts and bookkeeping alike, goes into the next request as one snapshot of the working set. N puts accepted
during one write cost one more write and one fsync pair, not N.
- **Backpressure** (Q5-B, recommended: hold):
- A stalled writer holds put replies. The loop keeps serving, and the working set keeps admitting (the catalog
serves admitted-not-durable puts, as it does today after a failed write).
- The queue is bounded by the server's existing limits: 64 sessions and 32 MiB of admitted payload. It adds no new
bound.
- A publisher that stops waiting (the Rust default is 300 ms) disconnects. Its session ends, its answer is
discarded, and its put is still written by the next write. That is what a publisher sees today during a stall,
minus the stalled challenges.
- **Ordering.**
- Replies are answered in version order from one queue, and the server sends acks in answer order.
- Within a session, C5 releases one ack only when every pending put of the session is answered.
- So no later put's reply leaves before an earlier one's.
- **Crash mid-batch.**
- The write is one atomic replace, so the file holds the old snapshot or the new one.
- No reply of a batch is answered before its `Written` outcome, so a SIGKILL anywhere loses no acknowledged put.
- A crash after the rename but before the directory sync is `ReplacedNotDurable` if the process lives; if it dies,
nothing of that batch was acknowledged.
- **Activation.** The store is read once, by `load`, before the writer is spawned. While the writer lives, the loop
never reads the file (Cut 1's rule, unchanged).
- **Verification (Odin, Yggdrasil):**
- **Loop latency under a stall.** This pins Q5's ruling; it is Soul's S-3 probe and O1, inverted into assertions.
- Hold the store's sibling lock for 1,500 ms, with a provider put in flight and Odin's heartbeat dirty with the
interval due.
- A route challenge and a catalog read are each served within 200 ms. The put's publisher sees no ack until the
lock is released, and then does, with the put on disk.
- Mutation that must fail it: running the write synchronously on the loop, by calling the `StoreFile` directly
from `serving_pass`.
- **Group commit.** Gated `StoreFile`, held on the first write, with 10 puts from 10 publishers accepted meanwhile.
- After release, exactly 2 writes are made and all 10 publishers are acknowledged.
- Mutation: one request per put gives 11 writes.
- **Ordering.** Gated `StoreFile`; put A's write is held and put B arrives.
- B's publisher is not acknowledged before A's; the answer order is observed at the server's answer channel.
- Mutation: answering the queue from the back.
- **Durability.** `daemon_process.rs`, extending `an_acknowledged_put_survives_sigkill` (`:326`).
- 20 rounds, each a stream of puts from 4 publishers while the test holds and releases the store's lock at random
offsets. SIGKILL at a random offset.
- Every put its publisher saw acknowledged is in the file after the kill.
- Mutation: accepting replies at submit instead of at `Written`.
- **One writer.** With the writer's `StoreFile` gated shut, N serving passes, a heartbeat and a stop request make no
write from the loop: the file's identity (`file_identity`, `main.rs:3343`) is unchanged until the gate opens.
- Negative grep: `rg -n "compare_exchange_snapshot|pull_all\(" crates/odin-daemon/src` matches only `writer.rs`,
`load` and the tests.
- **Lease lost with a write queued.** No write is made, Odin ends, and the queued replies are refused. The file's
identity is unchanged.
- **Foreign write.** Odin ends without writing over it (existing `:3696`, re-targeted through the writer).
- **Stop.** SIGTERM with a put held behind the lock: after the release Odin writes, the publisher is acknowledged,
and it exits 0. With the lease lost, it writes nothing.
- Re-targeted, not deleted: `changes_inside_one_interval_are_one_write` (`:3356`, now bookkeeping-only),
`a_put_is_written_before_it_is_accepted` (`:3389`, observed at the publisher's ack),
`a_put_whose_write_is_not_made_is_refused` (`:3516`), and the `ending_*` tests (`:3728-3800`).
- `cargo mutants --in-diff` on the cut's diff.
- **Estimate:** about -120 / +230 source, +300 tests. One new module, one thread, no new crate, dependency or target.
### Q5.5 What stays on the loop, and what does not
| Path | Before (`434756e`) | After W1 |
|---|---|---|
| route challenge (`raw_snapshot`, exact self query) | lease-file read, sign; blocked behind any write on the loop | the same reads; never behind a write |
| catalog read (`stored_snapshot`) | working set, projection read | unchanged; never behind a write |
| peer put | mutate, then write and fsync on the loop | mutate, queue the reply, submit; the reply is sent after `Written` |
| heartbeat, refresh | mutate; the interval write on the loop | mutate; the interval write on the writer |
| store lock | taken by `flush` on the loop | taken only by `load` (activation) and the writer |
### Q5.6 What only Yggdrasil can show
- **Rerun Soul's ABBA** (`soul-c3/latency_cmd.txt`: `soul_loop_stall_under_live_put_rate`, base against cut, in
interleaved rounds, with `/proc/pressure/io` before and after each).
- The base is `434756e`; the cut is W1's tip. Expect route-challenge p99 to stop tracking IO pressure, while put
round trip p99 still does.
- Soul's figures at `434756e` and its base over 8 rounds, under 5-38% pressure: challenge p99 310-2,129 ms, put
round-trip p99 1,121-2,593 ms (`soul-c3/latency.log`).
- Whether the loop's remaining disk reads (the lease and projection files) ever stall under real pressure. The ABBA
run shows it as challenge p99 outliers that coincide with pressure; if they appear, open the lease-in-memory
follow-up.
- After deploy, section 6's script, plus the count of "admitted odin route continuity: timed out" per hour.
### Q5.7 Sequencing (recommended order)
1. **Cut D merges to CultLib main** (already slated first).
2. **The put-serve branch rebases onto it and merges.** The conflicts:
- Cut D changes `send_packet` to return `Option<io::Error>` and adds `end_unsendable_session`.
- Its snapshot-send failure builds `reason: format!(..)`, where the put-serve branch made `reason` a typed
`CultMeshRudpRejectionReason`. It needs a variant (for example `ResponseSendFailed(String)`), and the put-serve
Hands owns it.
- Both branches edit `poll_once`'s ack site (`:468-476`) and `end_rejected_session`.
3. **R1 batch 3 merges** (cultcache-rs only; no conflict with 1-2).
4. **C5 is cut from that main** (it rewrites the lines 1-2 touched, so it goes after them), then Soul, then merge.
5. **One Odin pin bump to that commit, as W0**, carrying S-2, S-3 and the transport cuts. Then **W1** on top.
W0 may start on the commit after step 3 if C5 slips, at the cost of a second bump before W1. One bump is recommended:
each bump moves Odin across ack Cuts 1, 1b, 1c, 3 and D, and a single Soul pass over that transport delta is cheaper
than two.
### Q5.8 Follow-ups this map does not own
- **F-Q5a (parity).** TS and Python acknowledge a put at receipt, and TS's reply-later is an application receipt.
- Either they adopt C5's rule ("acknowledged means the sink accepted"), or the QUIC campaign defines an
application-level put reply for every runtime.
- QUIC has no transport ack an application can read, so it needs one anyway. Recommended owner: the QUIC campaign.
Rust C5 is then the RUDP-era contract, and TS's receipt is the seed of the QUIC shape.
- **F-Q5b (R1).** `with_exclusive_lock` returns an unlock error after a successful write, with no write-outcome
marker (`cultcache-rs/src/lib.rs:924-932` at `93ac5945`). Odin would call its own next write foreign. Fix it in
cultcache-rs: tag it, or ignore the unlock error, since dropping the file releases the lock.
- **S-5 interface.** C5 adds `acknowledgement_withheld` as the one place that decides "send no ack to this session".
The ack map's S-5 decision (probe P1) plugs in there as a second clause if it is fixed in the document server. C5
does not fix S-5.
### Q5.9 Subtraction ledger (estimate)
| Cut | Removed | Added | Targets, dependencies, wire |
|---|---|---|---|
| C5 (CultLib) | ~15 | ~170 source, ~350 tests | none; sink trait signature changes (Odin, Ratatoskr test) |
| W0 (Odin) | ~150 (the `attempted` list and 2 tests) | ~60 | pin bump |
| W1 (Odin) | ~120 (every loop-side write) | ~230 source, ~300 tests | one module, one thread |
Net source is positive: about +175. It buys the ruled capability (challenges and reads independent of the disk, with
Q3 B kept), and it removes S-3's unbounded list.
### Q5.10 Operator questions and defaults
- **Q5-A. What does a put's reply mean on the RUDP wire once it can come later?**
- A: the document server withholds the transport acknowledgement until the sink answers. Nothing created after the
put is sent to that session until then. The wire is unchanged, and every existing publisher (Rust
`publish_cultnet_message_to_rudp_catalog`, TS and Python publishers) keeps "acknowledged means accepted", which
for Odin is durable.
- B: a new CultNet message, a put result, sent after durability; the transport ack means received. This is a wire
change in the C# reference and four runtimes. Every Odin publisher that waits for the transport ack today must
be changed to wait for it; until then Q3 B silently weakens to "received".
- **Recommended: A.** It keeps Q3 B for every publisher with a server-only change, and RUDP is frozen except for
crash and data-loss fixes. B is the right shape under QUIC (F-Q5a), not a reason to change the RUDP wire now.
- What depends on it: C5's whole design.
- A also narrows the ack map's Q-A2 meaning for the document server: acknowledged means delivered *and answered*.
That was already true of the synchronous sink.
- **Q5-B. A writer stalled for seconds: hold put replies, or refuse with a typed error?**
- A: hold. Replies wait, bounded by the server's session and payload limits. Publishers time out on their own
clock, and the put is written when the disk returns.
- B: refuse once more than N replies wait or the oldest waits longer than T. This needs a new `cultnet.error.v0`
code (the error-contract map owns the registry) and two tunables.
- **Recommended: A.** A refusal ends the publisher's session exactly as its own timeout does, but adds a code and
two numbers, and it tells the publisher less than the truth: the put may still become durable. B earns its place
only if publishers ever wait without a timeout.
- **Default D-1 (Self, overridable). The API is a completion handle** (`CultMeshRudpPutReply`: `accept`, `refuse`,
and Drop refuses).
- The alternatives were a token with `server.answer_put(token, result)`, and a reply method keyed by session and
message id.
- Keying by session and message id is ambiguous: message ids are publisher-chosen, and a reconnect reuses the
session key.
- A token cannot be answered from another thread without routing it back to the server, and a forgotten token
leaves a session withheld until it ends.
- The handle is the Rust expression of TS's Promise-returning `onDocumentPutRaw`, and it keeps closure sinks
source-compatible.
## 8. Related findings
@ -986,23 +381,3 @@ Q3 B kept), and it removes S-3's unbounded list.
- **F7** `accept_raw_document` (`main.rs:269-280`) checks `activated()` but not the *current* lease before
mutating. Cut 2 moves the lease check to the only write, which closes this for the file. Soul should confirm
that nothing else acts on a lost lease.
## C5 status (2026-10-01)
**Hands done, Soul not yet run.** Drain mode: Soul waits for the operator's go.
- **Branch:** CultLib `hands/cultmesh-deferred-put-reply`, commits `08e0c229..d708247e`, based on `9dcb3aa1`.
- **Mechanism:** the sink answers through `CultMeshRudpPutReply` with `accept()` or `refuse(reason)`. Dropping the reply unanswered refuses with `UNANSWERED_PUT_REASON`. `acknowledgement_withheld` is the single owner of the hold: while a put is unanswered, the session gets no end-of-poll ack, Pong, snapshot response or reply to a repeated Connect.
- **Verification (Hands):**
- cultmesh-rs: 21 unit and 32 integration tests pass. A byte-exact golden test also passes before the cut.
- cultnet-rs: 486 pass.
- Interop: 27 pass, 0 fail.
- cargo-mutants: 57 mutants; 0 missed, 5 timeouts, all in the `take_answers` loop.
- **Spec deviations Soul should attack:**
- A repeated Connect's ack is now withheld.
- Puts queued behind a waiting snapshot count as withheld. Snapshot responses are held in a new `responses_owed` queue: P1 pending, then S, then P2 must not acknowledge P2 early.
- Answers are checked after each message.
- A release that arrives during an unreliable packet sends its own ack.
- **Structural delta:** +406/−96 in `rudp_document_server.rs`, against a +170/−15 estimate.
- **Consumers:** Odin `SinkHandle` and Ratatoskr `tests/catalog.rs:29` break until the W0 pin bump.
- **Still open:** S-5. Its fix belongs in `acknowledgement_withheld`.