provider-health-identity-id and host-actuator-identity-id are read-only.
The id is what a binding names as expected_signer_identity_id and what
a host is keyed by; deriving it here closes the gap where an operator
had to decode an anchor by hand to fill a binding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A managed host runs idunn-host in the user session. It dials Idunn's hub
over CultNet RUDP on the mesh, proves itself with a host actuator service
identity, and executes the runner and workload ports for its host: fetch
the exact frozen revision, build with the host's cargo under the binding's
allowed programs, install the sealed release, start the workload as a
child, report pid, creation time and executable digest, stop on request.
It keeps no transaction state and decides nothing.
On the Idunn side the engine chooses the runner and workload driver per
plan from the binding's driver tag; the yggdrasil daemon is still the
single deployment and continuity authority, and a binding of the old kind
takes the same path it always did. Every request is signed by Idunn's
identity and every report by the host's; both ends refuse the other's
message unless signature, session nonce and sequence hold. The hub admits
its own firewall allow the way the route driver does.
idunn-provision learns to enroll and export the host actuator identity.
The unit gains --host-actuator-bind on the mesh address.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
RunnerBinding and WorkloadBinding become enums tagged by driver, so a
host-native runner and a host-actuator workload can be declared without
counterfeiting docker images or systemd units. The single-variant
RunnerDriver and WorkloadDriver enums are gone; the driver is the tag.
WorkloadObservation is now Systemd or Host, untagged on the wire so every
transaction the previous Idunn persisted decodes unchanged (proved under
both MessagePack encodings CultCache uses). IsolationEvidence follows:
Linux proves uid, pid and mount namespaces; a host proves two live
processes distinct by pid and creation time. Mixed kinds are refused.
The build machine-id seal is unix-only so the library compiles on the
Windows host that will run the actuator.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Odin re-stamps a correlation only when its facts change, and a live presence
changes them every heartbeat, so a correlation older than the trusted window
means nothing about that incarnation has arrived lately. Admission and
supervise now read that one refusal as nothing-new. It had failed every
Heimdall deployment and the first observed Odin deployment at Warming.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 118 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Admitting the latest topology persists the sequence cursor; asking twice
with the same record failed the second time as "transaction changed before
topology admission". Only reachable now that an admitted Odin can observe a
candidate Odin under incarnation keys; it failed the first fold-back
deployment on yggdrasil at Warming.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 118 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A route that is admitted must be reachable without an operator opening a
port by hand. NginxRouteDriver admits one ufw allow for the binding's stable
address, port and transport, tagged with the route id, before it reloads a
fragment, and withdraws that one rule when the last membership is removed.
A rule already gone reads as the state asked for. Odin on yggdrasil is
admitted on 10.77.0.1:17971 with ufw still admitting only the previous
generation's 17871; this closes that class.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 118 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A replaced incumbent, an aborted candidate whose reconciliation was skipped,
and a record left by the projection migration all leave an Expected-only
incarnation in the projection that nothing owns. Supervise now withdraws one
per tick for each admitted target: any projected incarnation that is neither
the admitted generation nor a live transaction's. One owner for all three
sources, instead of a per-commit field.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 117 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Transactions that finished before finished transactions travelled to history
are still resident with their commands, gating nothing but growing the store
every decision reads. They leave the same way a transaction finishing today
does, one per tick.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 116 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Commands whose transactions were retired to history before commands travelled
with them were still resident and read as queued; freeze took the oldest
first, so one stale odin command was re-frozen hundreds of times today and
the newest command never got a turn. freeze_one_queued_command now reads
history, retires such commands on sight, and skips a busy target instead of
returning. status reports a live transaction as running rather than an older
attempt's failure, and prints completion and cleanup evidence for --command.
README carries the Linux build recipe; Windows does not compile.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 114 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Failed restarts retire to history.cc as soon as they finish, so the give-up
counter that read only the live set found zero every time and restarted a
release that could not start forever. Seen on yggdrasil during the
projection-key migration.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 114 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A candidate that died while its correlation was pending was never observed
again: each phase returned on no-evidence-yet every tick, the error that
triggers the post-fencing abort never happened, and the transaction owned the
target forever behind a process that was gone. Seen on yggdrasil with a
continuity transaction parked in Committing across the projection-key
migration.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 114 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The projection kept one Expected/activation/lease slot per target. A
deployment's Sealing overwrote that slot with the candidate's Expected, the
incumbent failed activation-expected-projection on its next heartbeat and
exited, the deployment was aborted for having killed its incumbent, and the
command -- left resident after its transaction was retired to history --
read as queued and was frozen again. Odin cycled on that every few minutes.
- Topology driver records are keyed {target}@{expected sha}; the incumbent and
the sealed candidate are projected side by side. One CAS mutation replaces
the six bespoke loops. withdraw_incarnation removes only one incarnation;
demote_to_expected_only replaces restore_admitted_expected_only and is used
only for an incarnation whose process is gone. Legacy target-keyed records
are retired on the first publish for that target.
- receive() takes the incarnation; the read-as-absence compensator for a
correlation about another incarnation is gone.
- Candidate Expected is published as Starting's first step, after the brake
admitted the transaction, not in Sealing before it.
- A pre-fencing abort withdraws only the candidate; a post-fencing abort also
demotes the stopped incumbent.
- A consumed command is archived with its last transaction; status still
answers for it from history.
- An engaged lifecycle brake stops continuity being minted instead of parking
a transaction on the target.
- idunn cancel <command-id> retires a queued command as an operator refusal.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 114 passed,
2 ignored.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The control store held every transaction Idunn had ever run -- 260 on
yggdrasil, all terminal, 2.6 MB re-read and byte-exact checked on every
scheduler tick. A finished transaction gates nothing; it only makes the
live set slower to read and gives history more ways to refuse one.
replace_transaction now moves a transaction out of control.cc the moment
it becomes terminal. History first, then the live copy: a crash between
the two leaves the record in both, which reads as still-resident and is
archived again next time. The other order can lose it, and a completion
nobody can observe is what R11 refused to trade away for a smaller live
set. insert_entry_if_absent makes the repeat a no-op and the delete is a
compare-and-swap on the exact envelope archived, so a transaction that
changed underneath is left alone rather than dropped.
history.cc is read leniently and for display only; control.cc keeps the
byte-exact canonical check because its records still gate decisions. That
asymmetry is the whole reason the two files are separate. status merges
both, so a completed command still reports what it did rather than
appearing never to have run.
No sweep. The earlier attempt archived resident terminal records from
inside the scheduler tick's propagating prefix, so an unwritable history
starved every live transaction on every other target. One primitive, one
caller, at the moment of completion.
Verified against a copy of the live store: status output is byte-identical
between the installed binary and this one, so the 260 existing records are
unaffected. They stay resident -- moving them is a separate explicit
operation, not something to bolt onto a scheduler.
cargo test --locked -> test result: ok. 113 passed; 0 failed; 2 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
validate_durable_authority read the deployment-brake operator anchor
unconditionally and then re-proved every transaction in the store. Two
consequences, both live until now:
An absent brake artifact refused startup against an empty store. That is
a deployment brake gating Idunn itself, which F:\Projects\CLAUDE.md
forbids in as many words -- a target brake "may never gate Idunn itself
or unrelated service lifecycle". The anchor is now read on demand, when
a record actually carries an authorization to verify.
Terminal transactions were re-proved on every boot. They describe what
already happened and authorize nothing further, so the only thing that
re-proving them can do is invent new ways for history to refuse a boot --
a rotated signer on a record from last week making the daemon
unstartable. They are skipped. On yggdrasil that is 259 of 260 records,
and this is what R4 meant by terminal history not being live authority.
Live records keep every check unchanged.
Known gap, deliberately not closed here: a live record whose signed
evidence no longer verifies still refuses startup. The right answer is to
abort that transaction rather than refuse the boot, which needs the
post-fencing abort path and is its own cut.
cargo test --locked -> test result: ok. 110 passed; 0 failed; 2 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
serve's loop propagated every scheduler-body error with `?`. Idunn runs
under Restart=always, so a single unreadable record turned into a
crashloop that took every unrelated target's continuity down with it --
the daemon-survival organ killed by exactly the thing it exists to
survive. This is the second half of the store-history work; the first
half landed as the admission gates.
run_scheduler_tick extracts one pass with the loop's ordering preserved
exactly, including that Ok(true) means "go round again immediately",
which is what the caller's `continue` did. The loop logs a faulting tick
and keeps polling. The fault recurs every poll until someone fixes it,
which is louder than a restart loop and cheaper than an outage.
Adds EngineFixture, the first Engine-level test harness this crate has
had -- enrolled Idunn and Odin identities, an exported anchor, tempdir
roots. Its absence is why "does one bad record kill the daemon" was
arguable rather than measurable.
The test proves the tick is a clean unit of failure: a 0xc1 payload (the
one byte msgpack never emits) makes three consecutive ticks report an
error and leaves the control store byte-identical, so a tick that cannot
read does not rewrite, truncate or silently repair. The loop's own match
arm is not covered -- serve never returns, so no unit test reaches it --
and that part rests on inspection.
cargo test --locked -> test result: ok. 109 passed; 0 failed; 2 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
is_permanently_stopped is called on the incumbent as well as the
candidate since 8448f83, so a message naming only the candidate would
misdirect the first person who hits it. Unreachable today -- observe_unit
guarantees every recorded workload is Restart=no -- which is exactly why
the wording would be read cold.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Present correlation with no activation is refused by the schema before
authentication, and a foreign Odin signature only names a different
identity when the record carries that identity's id.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul pass on the five gate changes. Negative checks only:
- another incarnation's perfect correlation, another activation of the same
Expected, and a foreign target beside our projection are all refused by
authentication alone (the deleted identity gate was a subset of it)
- the signing fixture refuses at every link: activation/Expected binding,
Idunn anchor, provider key, foreign Odin signer, forged signature
- a durable receipt replays at its own admitted_at and not at now
- is_permanently_stopped is true only for failed or not-found units; every
transitional ActiveState reads as running, and a systemctl failure errors
- an ignored live-store inventory probe
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prove_isolation compared the candidate's live process identity against
the incumbent's identity as recorded at admission. When the incumbent has
stopped, systemd has already released its DynamicUser UID and is free to
hand that same UID to the candidate -- so identical values mean ordinary
UID reuse, not a namespace collision, and fencing refused forever with
"candidate and incumbent are not distinct by UID, PID namespace, and
mount namespace".
Isolation is a property of two processes running at once. When the
admitted incumbent is permanently stopped there is nothing to be isolated
from, so the comparison is skipped. is_permanently_stopped already exists
on WorkloadPort for exactly this distinction.
Found by deploying: odin's incumbent died mid-deployment, and every
subsequent attempt to replace it was refused by this check.
cargo test --locked -> test result: ok. 102 passed; 0 failed; 1 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying the invariant fix cleared the equality gate and immediately
surfaced the next one wearing the same clothes: "Odin topology
correlation is outside the trusted observation window".
advance_committing called rehydrate_ready_token with require_current,
which authenticates the stored Ready receipt using `now` as the trusted
receive time. The receipt is durable evidence -- once it is older than
DEFAULT_TOPOLOGY_MAXIMUM_AGE_MILLIS (30s) it can never satisfy that,
which is the same unsatisfiable demand as the equality: a record that
does not change being asked to keep up with one that does.
Route admission at :3906 refreshes the receipt to the latest observation
before requiring currency, so `true` is right there and stays. By
Committing the receipt is history. Mirroring the refresh here would
livelock instead: refresh-then-return cannot converge against a publisher
emitting several correlations a second. Currency at commit is answered by
the latest observation, which is authenticated and checked for semantic
readiness immediately above.
Settled tx-65574579 on yggdrasil. Before: phase Committing, waiting on
the observation window. After: phase Complete, command succeeded, no
transactions in flight. Odin was never touched -- same PID 3911519, unit
ActiveEnterTimestamp still 2026-09-07 23:36:41. Idunn 0 restarts.
cargo test --locked -> test result: ok. 102 passed; 0 failed; 1 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Idunn's tests could not construct a signed Odin correlation. Every
topology fixture in the file was vec![byte], so no test could reach the
authentication layer, and gates were written above it on the assumption
that it did less than it does.
TopologyFixture builds the real chain, modelled on odin-daemon's
TestWorld: enrolled Idunn, Odin-topology and provider identities, an
Idunn-signed activation via IdunnRuntimeActivationLaunch::issue, and a
correlation signed with OdinRuntimeTopologyCorrelationPurpose. It leads
with the_fixture_actually_authenticates, which proves the chain REFUSES
-- tampered content under a valid signature, a zero signature, and a
record outside the observation window. A fixture that accepts everything
would make every gate below look satisfied.
The first thing it proved is that observation_describes_declared_incarnation
was redundant. authenticate_odin_runtime_topology_correlation already
calls validate_against_expected and then requires the record's
current_activation_sha256 AND runtime_instance_id to match the authority
-- which is built from this transaction's own expected and activation.
Target, projection and instance were all bound one layer down, more
strictly than the gate above restated them. Deleted at all three sites.
So the answer to "does Idunn want this feature" is no, twice over: not
the receipt-vs-cursor equality, and not either replacement I wrote for
it. Admission needs is_semantic_ready on a record that authenticated.
Also deletes 180 lines of vec![byte] topology fixtures the real one
replaces.
cargo test --locked -> test result: ok. 102 passed; 0 failed; 1 ignored
Live store, tx-65574579 at ready_seq=3841 latest_seq=678207:
old gate (receipt == cursor): REFUSE
binds declared incarnation (what authentication enforces): true
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
c09993c replaced the gate without asking what the gate was for, and kept
comparing a receipt to a cursor -- just loosely enough to survive. That
is the compensator shape, so this deletes it instead.
latest_odin_observation is not a receipt. It is a replay cursor: it is
written in lockstep with odin_publisher_sequence_cursor in both places it
is set (:3212, :4815), and AdmittedGeneration::validate requires the two
to agree. The field's own validator says what it is for. The commit gate
used it as proof of readiness, which conflates a high-water mark with a
proof -- and because TopologyEvidence derives PartialEq over five fields,
the comparison also demanded the two records agree on
admitted_at_unix_millis, which is Idunn's own wall clock.
Admission does not need a second observation to compare against. It needs
one question answered: does Odin, right now, say the incarnation this
transaction declared it was deploying is ready? The declaration is the
transaction's own Expected projection and prepared activation.
observation_describes_declared_incarnation checks target, Expected
projection digest and runtime instance against those; is_semantic_ready
supplies the "is ready" half. latest_still_confirms_ready,
latest_is_not_older and same_incarnation are deleted -- anti-replay is
already sequence_requires_admission plus the cursor, which is the
cursor's actual job.
The same conflation was copy-pasted into rehydrate_admitted_ready, so any
managed provider dropped out of the dependency-provider set the moment it
published anything. Unnoticed only because nothing in production has a
managed dependency yet.
Verified against the live store:
old gate (receipt == cursor): REFUSE
cursor says: projection=sha256-1f855526... instance=sha256-25f32a8d... ready=true
declared: projection=sha256-1f855526... instance=sha256-25f32a8d...
new gate: PASS
cargo test --locked -> test result: ok. 100 passed; 0 failed; 1 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
advance_committing required the latest Odin observation to equal the
Ready receipt byte for byte. Odin republishes whenever any fact it
observes changes -- a dependency going ready, a presence sequence
advancing, a capability set widening -- so the gate required the
observer to have stopped observing. No target with a working observer
can satisfy it.
This is why Idunn-managed Odin was never admitted. Every deployment
defect fixed this week was real; fixing them moved the transaction to
Committing, where this gate held it. On yggdrasil tx-65574579 has sat
there with ready_seq=3841 against latest_seq=297765 -- and rising -- while
the candidate it deployed serves traffic under
idunn-odin-25f32a8d.service.
The gate was standing in for something true and narrower: the newest
thing Odin says about this target still describes the incarnation we
proved ready. That is identity of the subject, not equality of the
statement. latest_still_confirms_ready requires the observation to be no
older, carry the same publisher, and name the same target, Expected
projection and runtime instance; callers pair it with is_semantic_ready,
which is what establishes the newer statement still says ready.
Same replacement at the final admission challenge and in
rehydrate_admitted_ready, which now authenticates the latest observation
rather than inferring its content from equality with the receipt. The
pre-filter in current_ready_provider_tokens is deleted: rehydrate owns
that judgment and reports why when it refuses.
Split into latest_is_not_older and same_incarnation because Idunn's tests
cannot construct a signed Odin correlation, which is much of why this
gate was never covered. Verified against the live store:
old gate (byte equality): REFUSE
new gate, ordering: PASS
new gate, incarnation: PASS
ready/latest both projection=sha256-1f855526... instance=sha256-25f32a8d...
cargo test --locked -> test result: ok. 101 passed; 0 failed
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cd662cd claimed no target is Idunn-managed and that every idunn-odin-*
unit was failed. Both wrong. idunn-odin-25f32a8d is active and running
release 928da3ef, and it is the live Odin -- the odin.service unit still
in /etc/systemd/system is active with MainPID 0 and owns no process.
That claim came from reading the first 20 lines of an alphabetically
sorted unit list. 25f32a8d sorts below the cut. Truncated enumeration,
stated as a whole-system fact, in the same commit that corrected someone
else for a whole-system fact taken from the wrong artifact.
Heimdall genuinely is not migrated; that half stands.
Also records why odin's transaction is stuck in Committing: the gate
requires the latest Odin observation to equal the Ready receipt, which
no target with a working observer can satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The header claimed "no target has been migrated" and that the installed
binary was the previous generation. Half of that was stale: yggdrasil
runs the declarative binary and idunn status answers.
The other half was right, and a proposed "correction" to say odin and
heimdall are now Idunn-managed would have been false. Verified on the
host: heimdall.service and odin.service are hand-written units under
/etc/systemd/system, both active, and every idunn-heimdall-* and
idunn-odin-* unit is failed.
The "Correction: Odin is migrated first, not last" section reads like a
completion record and is not one -- it changed the ordering, it did not
migrate a target. Said so in the header, because that misreading has now
happened twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Odin's stored correlation outlives the incarnation that wrote it, so a target
being replaced -- Odin above all, which writes its own -- routinely finds one
describing the Expected it is superseding. Idunn treated that as a
contradiction and failed the deployment, and only the superseded process could
have refreshed it, so the deployment that would have fixed the target was the
one thing the stale record prevented.
It is now read as absence. That says nothing about trust: the record is
unauthenticated at that point and is used only to decide it is about something
else, which is the same conclusion an attacker gets for free by publishing
nothing. Everything after it is fully authenticated.
Two more copies of the same "first bootstrap" assumption are aligned with the
rule that replaced it. Whether Odin may be observed directly is decided once,
where the question is whether any Odin can report; the presence authentication
and the warming refresh were re-deciding it with the narrower "has an Odin ever
been admitted" test and contradicting that caller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Continuity restarts a release; it cannot repair one. When the admitted release
will not start, rescheduling it forever keeps the target permanently occupied
-- and a target with a live transaction accepts no deployment, so the one
action that could fix it is exactly the one that is locked out. Odin sat in
that state through 196 restarts.
It now stops after three attempts and says so, leaving the target free. The
count is kept against the admitted generation id, so a restart that succeeds
admits a new generation and starts the count over rather than carrying old
failures forward.
The yield to continuity is conditioned on the same count. Handing a deployment
back to a continuity that has already given up is not caution, it is closing
the last door: there is nothing left to yield to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Warming presence reaches Idunn as Odin's signed topology correlation, with one
exception for Odin's own first bootstrap. That exception was scoped too
narrowly: it asked whether an Odin had ever been admitted, when the question
that matters is whether an Odin can currently report.
A continuity restart of Odin cannot be observed through Odin -- the incumbent
that would do the observing is the process being restarted. Neither can a
deployment made while the admitted Odin has no usable correlation. Both sat
waiting for evidence that could not arrive, and continuity restarted the
target into that wait every few seconds.
The rule is now the one that justified the bootstrap case in the first place:
direct observation exactly when "observed through Odin" is unavailable, and
only ever for this one target. Every other target still waits for Odin, which
is what makes Odin the root of the chain.
Also adds the projection-presence check continuity uses to tell a projection
that still describes a dead incarnation from one already demoted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Continuity's projection repair required a fresh workload observation, so it
could only run while the target was healthy. But a demoted projection is one
of the reasons a target stops being healthy: aborting a deployment restores
the incumbent's Expected without its activation, and a target that cannot see
its own activation cannot publish presence, so it exits -- and no observation
is ever available to authorise the repair. Odin crashlooped exactly this way
after a failed redeploy, with continuity restarting it into the same wall
every few seconds and no path out.
Nothing in the repair is inferred from the live process. Expected, activation
and lease all come from the admitted generation, and the workload observation
used is the one Idunn recorded when it admitted them, so this republishes what
Idunn already decided rather than ratifying whatever happens to be running.
It now runs before anything is asked of the workload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Idunn refuses an Odin publisher sequence at or below the highest it has seen
for a target, which is what stops an old signed correlation being replayed.
The watermark counted every transaction, including ones that failed.
A failed transaction was never built upon: the deployment was abandoned and
its projection withdrawn. Carrying its readings forward only raises the bar
for the next attempt, and with Odin restarting a withdrawn correlation's count
that made a target which had failed once into a target that could never deploy
again. Odin does not reuse a sequence, so forgetting the readings of an
attempt that came to nothing weakens nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five more preprovisioning requirements, each of which cost a candidate that
started, ran, and then died on a file it could not open: bundle traversal,
the setgid runtime root the write-lease hardening assumes, the lease record's
group, a default ACL on the published projection, and the route directory's
ownership left over from unprivileged Idunn.
Also states the two shapes that made every one of those faults arrive in
disguise: a store's .lock is opened whenever the store is, and CultLib reports
an unreadable file as an empty store rather than an error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultCache opens the `.lock` sibling whenever it opens a store, so a 0640 lock
denies a workload's read exactly as a 0640 store does -- and the lock is
created fresh under Idunn's UMask=027 on every publish, so setting the store's
mode alone was not enough. Odin's candidate got past the write lease and then
died on the projection for this reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding `post_fencing_abort` was a schema change, not a new optional field.
The control store refuses noncanonical bytes -- a record must re-encode to
exactly what is stored, which is what makes tampering visible -- so every
transaction written before the field existed re-encodes with one extra key and
reads as tampered. Deploying the previous commit took the daemon down in a
crashloop on its own control store; it was rolled back within minutes.
A v2 record is now decoded by key, which tolerates the absent one, and lifted
with no post-fencing abort, because a transaction written before the field
existed cannot have had one. Byte-exactness cannot apply across a version
boundary -- the bytes are a different schema by definition -- so the lift
leans on the full semantic validate() instead, and everything downstream sees
only the current shape. A startup pass then rewrites stale records as v3 by
compare-exchange, so the store converges rather than keeping two shapes for as
long as retention holds the oldest terminal transaction.
Verified against the live control store rather than a fixture: 41 real v2
records migrated on start, and the post-fence abort then released a
transaction that had been wedged in Leasing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A transaction past Fencing had no terminal path. begin_pre_fencing_abort is
gated on phase < Fencing, so every later error went to record_resumable_error
and retried forever. That is right while a candidate can still recover, and a
trap once it cannot: Idunn launches candidates as transient units with
Restart=no, so a failed one never comes back, and the transaction holds its
target with every later command queued behind it. There was no operator verb
to clear one and no server-side expiry.
So a candidate that is permanently stopped -- unit failed, or gone from
systemd entirely -- now aborts. The withdrawal runs in the one order that
never leaves two writers: route, then write lease, then the process, then the
projection. Reconciling the projection restores the incumbent's admitted
Expected where there is one, which is what lets continuity bring the incumbent
back after fencing stopped it; where there is none, the failed Expected is
withdrawn.
The transaction is released only once every one of those is durably complete,
which is what the cleanup evidence is for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reachability check only accepted o+x, but the other correct shape for a
runtime root is 0750 group-owned by the state group -- and that is the shape
the write-lease hardening already assumes, since it derives the record's group
from this directory. Checking only o+x would have refused the configuration
that makes the lease readable.
The workload's uid is allocated by DynamicUser and owns nothing, so traversal
can only come from the state group or from other. Both are now accepted, and
a directory group-owned by some *other* group still is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first deployment of any routed target could never succeed, and wedged
itself permanently on the way down.
The route preflight bind-mounts its candidate over config_path to run
`nginx -t` in a private mount namespace. When that path does not exist yet,
systemd materializes the mount point and leaves an empty file on the host.
Read as content, that made the baseline "change" during validation, failing
the transaction -- and every retry then saw a zero-length fragment where the
transaction expected none and reported an unadmitted incumbent, which no
amount of retrying could clear.
A fragment with no bytes grants no route, so it is now read as absence. That
makes the preflight idempotent whether or not systemd materialized the mount
point, and lets a first deployment retry cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The topology store is Idunn's published surface -- every managed target reads
it to verify its own Expected incarnation against the Idunn anchor -- but
Idunn runs with UMask=027, so each publish left it 0640 root:root and a
DynamicUser workload got EACCES. Odin's candidate died on exactly that.
Fixing the mode by hand does not hold, because every publish writes a new
file. Integrity here comes from the signatures over the records, not from the
mode, so the published copy is now explicitly readable after each write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultLib's backing store reports an unreadable file as an empty store, so a
workload that cannot traverse to its runtime bundle sees zero records rather
than a permission error. Odin reported "runtime authority store must contain
exactly one record" and Heimdall simply concluded it held no write lease and
warmed forever. Neither names the actual fault, and both cost real time.
The cause is that `ReadOnlyPaths=` binds the leaf into the unit's namespace
but grants no traversal above it, so a runtime root under a 0700 parent is
unreachable to a DynamicUser. Idunn now checks the whole chain when it writes
the bundle and says exactly which directory is closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five things that had never been exercised failed on the way through Odin, and
each is now a test: the unprojectable RUDP route, step environment not being
ambient, build containers having no machine identity, the published projection
sitting inside Idunn's private root, and a signing identity living in the
state root the workload can write.
Also documents that `idunn validate` proves nothing about deployability
without a binding -- admit is where every cross-document rule lives -- and
repairs apostrophes an earlier pass mangled into ".s".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A target whose tests enrol a service identity could not be built at all.
CultLib's Linux protector binds the seed to /etc/machine-id, the build image
carries none, and the runner is --read-only, so Odin's suite failed every
step with "Linux machine-id is unavailable".
Mounting the host's machine-id would let a build container protect a seed
that unwraps on the host, which is exactly the property the binding exists to
deny. So each frozen workspace now gets its own identity, derived from the
workspace path and mounted read-only: stable for the life of a build, and
useless anywhere else including this host.
Also moves Idunn's topology store to /var/lib/gamecult/idunn-projection. It
is Idunn's published surface -- every managed target must read it to verify
its own Expected incarnation -- and it was sitting inside Idunn's private
0750 root where no workload could traverse to it. Odin's own daemon already
named that directory in its tests.
Odin's three roots become writable for the same reason Heimdall's are: a
target Idunn cannot write is a target it cannot deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A routed RUDP target was admissible but unprojectable. `admit` has an
explicit arm for (route_required, Rudp, nginx-stream-udp) against an
`rudp://` stable endpoint, and the nginx driver already writes
`listen ... udp reuseport` for it -- but `endpoint_scheme` lumped Rudp in
with Private and refused to name a scheme, so `expected_projection` bailed
before any routed RUDP candidate could be described.
Odin is the only routed RUDP target and it is the bootstrap target: nothing
else in the estate can warm until Odin is admitted, and Odin could not be.
Private keeps bailing, which is correct -- `route_required` is false for it,
so a private target never reaches this function.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying Heimdall proved the migration order backwards. A candidate's
warming presence -- the evidence Idunn requires before fencing an incumbent
-- is observed through Odin's authenticated runtime topology correlation.
advance_warming has exactly one exception: target == "odin" while no Odin
is admitted may warm on directly observed presence. Every other target
waits for Odin to publish.
So with the previous generation's Odin still running, a Heimdall candidate
starts, serves on its assigned bind, publishes health, and warms forever.
It never fails and never promotes. That is not a defect; it is the
observation chain being rooted somewhere I had not looked. It also explains
why Odin has the only recipe in the estate that was ever kept current.
The document said "migrate the authority's own dependencies last". Odin is
not a dependency to defer, it is the root, and the entry now says so.
The wait also recorded no reason, so a candidate could sit in Warming
indefinitely with nothing to read -- the fourth coarse-signal case today.
It now records what it is waiting for and why the wait cannot resolve
without Odin.
Noted alongside: health published over RUDP to the previous generation's
--rudp-health-bind goes nowhere, since the current serve has no such flag.
That produces a steady stream of timeouts in the candidate's journal and is
not the reason promotion stalls.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
