A candidate that died while its correlation was pending was never observed
again: each phase returned on no-evidence-yet every tick, the error that
triggers the post-fencing abort never happened, and the transaction owned the
target forever behind a process that was gone. Seen on yggdrasil with a
continuity transaction parked in Committing across the projection-key
migration.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 114 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The projection kept one Expected/activation/lease slot per target. A
deployment's Sealing overwrote that slot with the candidate's Expected, the
incumbent failed activation-expected-projection on its next heartbeat and
exited, the deployment was aborted for having killed its incumbent, and the
command -- left resident after its transaction was retired to history --
read as queued and was frozen again. Odin cycled on that every few minutes.
- Topology driver records are keyed {target}@{expected sha}; the incumbent and
the sealed candidate are projected side by side. One CAS mutation replaces
the six bespoke loops. withdraw_incarnation removes only one incarnation;
demote_to_expected_only replaces restore_admitted_expected_only and is used
only for an incarnation whose process is gone. Legacy target-keyed records
are retired on the first publish for that target.
- receive() takes the incarnation; the read-as-absence compensator for a
correlation about another incarnation is gone.
- Candidate Expected is published as Starting's first step, after the brake
admitted the transaction, not in Sealing before it.
- A pre-fencing abort withdraws only the candidate; a post-fencing abort also
demotes the stopped incumbent.
- A consumed command is archived with its last transaction; status still
answers for it from history.
- An engaged lifecycle brake stops continuity being minted instead of parking
a transaction on the target.
- idunn cancel <command-id> retires a queued command as an operator refusal.
Verified with cargo test --lib in rust:1.95-bookworm on yggdrasil: 114 passed,
2 ignored.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The control store held every transaction Idunn had ever run -- 260 on
yggdrasil, all terminal, 2.6 MB re-read and byte-exact checked on every
scheduler tick. A finished transaction gates nothing; it only makes the
live set slower to read and gives history more ways to refuse one.
replace_transaction now moves a transaction out of control.cc the moment
it becomes terminal. History first, then the live copy: a crash between
the two leaves the record in both, which reads as still-resident and is
archived again next time. The other order can lose it, and a completion
nobody can observe is what R11 refused to trade away for a smaller live
set. insert_entry_if_absent makes the repeat a no-op and the delete is a
compare-and-swap on the exact envelope archived, so a transaction that
changed underneath is left alone rather than dropped.
history.cc is read leniently and for display only; control.cc keeps the
byte-exact canonical check because its records still gate decisions. That
asymmetry is the whole reason the two files are separate. status merges
both, so a completed command still reports what it did rather than
appearing never to have run.
No sweep. The earlier attempt archived resident terminal records from
inside the scheduler tick's propagating prefix, so an unwritable history
starved every live transaction on every other target. One primitive, one
caller, at the moment of completion.
Verified against a copy of the live store: status output is byte-identical
between the installed binary and this one, so the 260 existing records are
unaffected. They stay resident -- moving them is a separate explicit
operation, not something to bolt onto a scheduler.
cargo test --locked -> test result: ok. 113 passed; 0 failed; 2 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
validate_durable_authority read the deployment-brake operator anchor
unconditionally and then re-proved every transaction in the store. Two
consequences, both live until now:
An absent brake artifact refused startup against an empty store. That is
a deployment brake gating Idunn itself, which F:\Projects\CLAUDE.md
forbids in as many words -- a target brake "may never gate Idunn itself
or unrelated service lifecycle". The anchor is now read on demand, when
a record actually carries an authorization to verify.
Terminal transactions were re-proved on every boot. They describe what
already happened and authorize nothing further, so the only thing that
re-proving them can do is invent new ways for history to refuse a boot --
a rotated signer on a record from last week making the daemon
unstartable. They are skipped. On yggdrasil that is 259 of 260 records,
and this is what R4 meant by terminal history not being live authority.
Live records keep every check unchanged.
Known gap, deliberately not closed here: a live record whose signed
evidence no longer verifies still refuses startup. The right answer is to
abort that transaction rather than refuse the boot, which needs the
post-fencing abort path and is its own cut.
cargo test --locked -> test result: ok. 110 passed; 0 failed; 2 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
serve's loop propagated every scheduler-body error with `?`. Idunn runs
under Restart=always, so a single unreadable record turned into a
crashloop that took every unrelated target's continuity down with it --
the daemon-survival organ killed by exactly the thing it exists to
survive. This is the second half of the store-history work; the first
half landed as the admission gates.
run_scheduler_tick extracts one pass with the loop's ordering preserved
exactly, including that Ok(true) means "go round again immediately",
which is what the caller's `continue` did. The loop logs a faulting tick
and keeps polling. The fault recurs every poll until someone fixes it,
which is louder than a restart loop and cheaper than an outage.
Adds EngineFixture, the first Engine-level test harness this crate has
had -- enrolled Idunn and Odin identities, an exported anchor, tempdir
roots. Its absence is why "does one bad record kill the daemon" was
arguable rather than measurable.
The test proves the tick is a clean unit of failure: a 0xc1 payload (the
one byte msgpack never emits) makes three consecutive ticks report an
error and leaves the control store byte-identical, so a tick that cannot
read does not rewrite, truncate or silently repair. The loop's own match
arm is not covered -- serve never returns, so no unit test reaches it --
and that part rests on inspection.
cargo test --locked -> test result: ok. 109 passed; 0 failed; 2 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
is_permanently_stopped is called on the incumbent as well as the
candidate since 8448f83, so a message naming only the candidate would
misdirect the first person who hits it. Unreachable today -- observe_unit
guarantees every recorded workload is Restart=no -- which is exactly why
the wording would be read cold.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Present correlation with no activation is refused by the schema before
authentication, and a foreign Odin signature only names a different
identity when the record carries that identity's id.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul pass on the five gate changes. Negative checks only:
- another incarnation's perfect correlation, another activation of the same
Expected, and a foreign target beside our projection are all refused by
authentication alone (the deleted identity gate was a subset of it)
- the signing fixture refuses at every link: activation/Expected binding,
Idunn anchor, provider key, foreign Odin signer, forged signature
- a durable receipt replays at its own admitted_at and not at now
- is_permanently_stopped is true only for failed or not-found units; every
transitional ActiveState reads as running, and a systemctl failure errors
- an ignored live-store inventory probe
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prove_isolation compared the candidate's live process identity against
the incumbent's identity as recorded at admission. When the incumbent has
stopped, systemd has already released its DynamicUser UID and is free to
hand that same UID to the candidate -- so identical values mean ordinary
UID reuse, not a namespace collision, and fencing refused forever with
"candidate and incumbent are not distinct by UID, PID namespace, and
mount namespace".
Isolation is a property of two processes running at once. When the
admitted incumbent is permanently stopped there is nothing to be isolated
from, so the comparison is skipped. is_permanently_stopped already exists
on WorkloadPort for exactly this distinction.
Found by deploying: odin's incumbent died mid-deployment, and every
subsequent attempt to replace it was refused by this check.
cargo test --locked -> test result: ok. 102 passed; 0 failed; 1 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying the invariant fix cleared the equality gate and immediately
surfaced the next one wearing the same clothes: "Odin topology
correlation is outside the trusted observation window".
advance_committing called rehydrate_ready_token with require_current,
which authenticates the stored Ready receipt using `now` as the trusted
receive time. The receipt is durable evidence -- once it is older than
DEFAULT_TOPOLOGY_MAXIMUM_AGE_MILLIS (30s) it can never satisfy that,
which is the same unsatisfiable demand as the equality: a record that
does not change being asked to keep up with one that does.
Route admission at :3906 refreshes the receipt to the latest observation
before requiring currency, so `true` is right there and stays. By
Committing the receipt is history. Mirroring the refresh here would
livelock instead: refresh-then-return cannot converge against a publisher
emitting several correlations a second. Currency at commit is answered by
the latest observation, which is authenticated and checked for semantic
readiness immediately above.
Settled tx-65574579 on yggdrasil. Before: phase Committing, waiting on
the observation window. After: phase Complete, command succeeded, no
transactions in flight. Odin was never touched -- same PID 3911519, unit
ActiveEnterTimestamp still 2026-09-07 23:36:41. Idunn 0 restarts.
cargo test --locked -> test result: ok. 102 passed; 0 failed; 1 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Idunn's tests could not construct a signed Odin correlation. Every
topology fixture in the file was vec![byte], so no test could reach the
authentication layer, and gates were written above it on the assumption
that it did less than it does.
TopologyFixture builds the real chain, modelled on odin-daemon's
TestWorld: enrolled Idunn, Odin-topology and provider identities, an
Idunn-signed activation via IdunnRuntimeActivationLaunch::issue, and a
correlation signed with OdinRuntimeTopologyCorrelationPurpose. It leads
with the_fixture_actually_authenticates, which proves the chain REFUSES
-- tampered content under a valid signature, a zero signature, and a
record outside the observation window. A fixture that accepts everything
would make every gate below look satisfied.
The first thing it proved is that observation_describes_declared_incarnation
was redundant. authenticate_odin_runtime_topology_correlation already
calls validate_against_expected and then requires the record's
current_activation_sha256 AND runtime_instance_id to match the authority
-- which is built from this transaction's own expected and activation.
Target, projection and instance were all bound one layer down, more
strictly than the gate above restated them. Deleted at all three sites.
So the answer to "does Idunn want this feature" is no, twice over: not
the receipt-vs-cursor equality, and not either replacement I wrote for
it. Admission needs is_semantic_ready on a record that authenticated.
Also deletes 180 lines of vec![byte] topology fixtures the real one
replaces.
cargo test --locked -> test result: ok. 102 passed; 0 failed; 1 ignored
Live store, tx-65574579 at ready_seq=3841 latest_seq=678207:
old gate (receipt == cursor): REFUSE
binds declared incarnation (what authentication enforces): true
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
c09993c replaced the gate without asking what the gate was for, and kept
comparing a receipt to a cursor -- just loosely enough to survive. That
is the compensator shape, so this deletes it instead.
latest_odin_observation is not a receipt. It is a replay cursor: it is
written in lockstep with odin_publisher_sequence_cursor in both places it
is set (:3212, :4815), and AdmittedGeneration::validate requires the two
to agree. The field's own validator says what it is for. The commit gate
used it as proof of readiness, which conflates a high-water mark with a
proof -- and because TopologyEvidence derives PartialEq over five fields,
the comparison also demanded the two records agree on
admitted_at_unix_millis, which is Idunn's own wall clock.
Admission does not need a second observation to compare against. It needs
one question answered: does Odin, right now, say the incarnation this
transaction declared it was deploying is ready? The declaration is the
transaction's own Expected projection and prepared activation.
observation_describes_declared_incarnation checks target, Expected
projection digest and runtime instance against those; is_semantic_ready
supplies the "is ready" half. latest_still_confirms_ready,
latest_is_not_older and same_incarnation are deleted -- anti-replay is
already sequence_requires_admission plus the cursor, which is the
cursor's actual job.
The same conflation was copy-pasted into rehydrate_admitted_ready, so any
managed provider dropped out of the dependency-provider set the moment it
published anything. Unnoticed only because nothing in production has a
managed dependency yet.
Verified against the live store:
old gate (receipt == cursor): REFUSE
cursor says: projection=sha256-1f855526... instance=sha256-25f32a8d... ready=true
declared: projection=sha256-1f855526... instance=sha256-25f32a8d...
new gate: PASS
cargo test --locked -> test result: ok. 100 passed; 0 failed; 1 ignored
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
advance_committing required the latest Odin observation to equal the
Ready receipt byte for byte. Odin republishes whenever any fact it
observes changes -- a dependency going ready, a presence sequence
advancing, a capability set widening -- so the gate required the
observer to have stopped observing. No target with a working observer
can satisfy it.
This is why Idunn-managed Odin was never admitted. Every deployment
defect fixed this week was real; fixing them moved the transaction to
Committing, where this gate held it. On yggdrasil tx-65574579 has sat
there with ready_seq=3841 against latest_seq=297765 -- and rising -- while
the candidate it deployed serves traffic under
idunn-odin-25f32a8d.service.
The gate was standing in for something true and narrower: the newest
thing Odin says about this target still describes the incarnation we
proved ready. That is identity of the subject, not equality of the
statement. latest_still_confirms_ready requires the observation to be no
older, carry the same publisher, and name the same target, Expected
projection and runtime instance; callers pair it with is_semantic_ready,
which is what establishes the newer statement still says ready.
Same replacement at the final admission challenge and in
rehydrate_admitted_ready, which now authenticates the latest observation
rather than inferring its content from equality with the receipt. The
pre-filter in current_ready_provider_tokens is deleted: rehydrate owns
that judgment and reports why when it refuses.
Split into latest_is_not_older and same_incarnation because Idunn's tests
cannot construct a signed Odin correlation, which is much of why this
gate was never covered. Verified against the live store:
old gate (byte equality): REFUSE
new gate, ordering: PASS
new gate, incarnation: PASS
ready/latest both projection=sha256-1f855526... instance=sha256-25f32a8d...
cargo test --locked -> test result: ok. 101 passed; 0 failed
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cd662cd claimed no target is Idunn-managed and that every idunn-odin-*
unit was failed. Both wrong. idunn-odin-25f32a8d is active and running
release 928da3ef, and it is the live Odin -- the odin.service unit still
in /etc/systemd/system is active with MainPID 0 and owns no process.
That claim came from reading the first 20 lines of an alphabetically
sorted unit list. 25f32a8d sorts below the cut. Truncated enumeration,
stated as a whole-system fact, in the same commit that corrected someone
else for a whole-system fact taken from the wrong artifact.
Heimdall genuinely is not migrated; that half stands.
Also records why odin's transaction is stuck in Committing: the gate
requires the latest Odin observation to equal the Ready receipt, which
no target with a working observer can satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The header claimed "no target has been migrated" and that the installed
binary was the previous generation. Half of that was stale: yggdrasil
runs the declarative binary and idunn status answers.
The other half was right, and a proposed "correction" to say odin and
heimdall are now Idunn-managed would have been false. Verified on the
host: heimdall.service and odin.service are hand-written units under
/etc/systemd/system, both active, and every idunn-heimdall-* and
idunn-odin-* unit is failed.
The "Correction: Odin is migrated first, not last" section reads like a
completion record and is not one -- it changed the ordering, it did not
migrate a target. Said so in the header, because that misreading has now
happened twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Odin's stored correlation outlives the incarnation that wrote it, so a target
being replaced -- Odin above all, which writes its own -- routinely finds one
describing the Expected it is superseding. Idunn treated that as a
contradiction and failed the deployment, and only the superseded process could
have refreshed it, so the deployment that would have fixed the target was the
one thing the stale record prevented.
It is now read as absence. That says nothing about trust: the record is
unauthenticated at that point and is used only to decide it is about something
else, which is the same conclusion an attacker gets for free by publishing
nothing. Everything after it is fully authenticated.
Two more copies of the same "first bootstrap" assumption are aligned with the
rule that replaced it. Whether Odin may be observed directly is decided once,
where the question is whether any Odin can report; the presence authentication
and the warming refresh were re-deciding it with the narrower "has an Odin ever
been admitted" test and contradicting that caller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Continuity restarts a release; it cannot repair one. When the admitted release
will not start, rescheduling it forever keeps the target permanently occupied
-- and a target with a live transaction accepts no deployment, so the one
action that could fix it is exactly the one that is locked out. Odin sat in
that state through 196 restarts.
It now stops after three attempts and says so, leaving the target free. The
count is kept against the admitted generation id, so a restart that succeeds
admits a new generation and starts the count over rather than carrying old
failures forward.
The yield to continuity is conditioned on the same count. Handing a deployment
back to a continuity that has already given up is not caution, it is closing
the last door: there is nothing left to yield to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Warming presence reaches Idunn as Odin's signed topology correlation, with one
exception for Odin's own first bootstrap. That exception was scoped too
narrowly: it asked whether an Odin had ever been admitted, when the question
that matters is whether an Odin can currently report.
A continuity restart of Odin cannot be observed through Odin -- the incumbent
that would do the observing is the process being restarted. Neither can a
deployment made while the admitted Odin has no usable correlation. Both sat
waiting for evidence that could not arrive, and continuity restarted the
target into that wait every few seconds.
The rule is now the one that justified the bootstrap case in the first place:
direct observation exactly when "observed through Odin" is unavailable, and
only ever for this one target. Every other target still waits for Odin, which
is what makes Odin the root of the chain.
Also adds the projection-presence check continuity uses to tell a projection
that still describes a dead incarnation from one already demoted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Continuity's projection repair required a fresh workload observation, so it
could only run while the target was healthy. But a demoted projection is one
of the reasons a target stops being healthy: aborting a deployment restores
the incumbent's Expected without its activation, and a target that cannot see
its own activation cannot publish presence, so it exits -- and no observation
is ever available to authorise the repair. Odin crashlooped exactly this way
after a failed redeploy, with continuity restarting it into the same wall
every few seconds and no path out.
Nothing in the repair is inferred from the live process. Expected, activation
and lease all come from the admitted generation, and the workload observation
used is the one Idunn recorded when it admitted them, so this republishes what
Idunn already decided rather than ratifying whatever happens to be running.
It now runs before anything is asked of the workload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Idunn refuses an Odin publisher sequence at or below the highest it has seen
for a target, which is what stops an old signed correlation being replayed.
The watermark counted every transaction, including ones that failed.
A failed transaction was never built upon: the deployment was abandoned and
its projection withdrawn. Carrying its readings forward only raises the bar
for the next attempt, and with Odin restarting a withdrawn correlation's count
that made a target which had failed once into a target that could never deploy
again. Odin does not reuse a sequence, so forgetting the readings of an
attempt that came to nothing weakens nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five more preprovisioning requirements, each of which cost a candidate that
started, ran, and then died on a file it could not open: bundle traversal,
the setgid runtime root the write-lease hardening assumes, the lease record's
group, a default ACL on the published projection, and the route directory's
ownership left over from unprivileged Idunn.
Also states the two shapes that made every one of those faults arrive in
disguise: a store's .lock is opened whenever the store is, and CultLib reports
an unreadable file as an empty store rather than an error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultCache opens the `.lock` sibling whenever it opens a store, so a 0640 lock
denies a workload's read exactly as a 0640 store does -- and the lock is
created fresh under Idunn's UMask=027 on every publish, so setting the store's
mode alone was not enough. Odin's candidate got past the write lease and then
died on the projection for this reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding `post_fencing_abort` was a schema change, not a new optional field.
The control store refuses noncanonical bytes -- a record must re-encode to
exactly what is stored, which is what makes tampering visible -- so every
transaction written before the field existed re-encodes with one extra key and
reads as tampered. Deploying the previous commit took the daemon down in a
crashloop on its own control store; it was rolled back within minutes.
A v2 record is now decoded by key, which tolerates the absent one, and lifted
with no post-fencing abort, because a transaction written before the field
existed cannot have had one. Byte-exactness cannot apply across a version
boundary -- the bytes are a different schema by definition -- so the lift
leans on the full semantic validate() instead, and everything downstream sees
only the current shape. A startup pass then rewrites stale records as v3 by
compare-exchange, so the store converges rather than keeping two shapes for as
long as retention holds the oldest terminal transaction.
Verified against the live control store rather than a fixture: 41 real v2
records migrated on start, and the post-fence abort then released a
transaction that had been wedged in Leasing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A transaction past Fencing had no terminal path. begin_pre_fencing_abort is
gated on phase < Fencing, so every later error went to record_resumable_error
and retried forever. That is right while a candidate can still recover, and a
trap once it cannot: Idunn launches candidates as transient units with
Restart=no, so a failed one never comes back, and the transaction holds its
target with every later command queued behind it. There was no operator verb
to clear one and no server-side expiry.
So a candidate that is permanently stopped -- unit failed, or gone from
systemd entirely -- now aborts. The withdrawal runs in the one order that
never leaves two writers: route, then write lease, then the process, then the
projection. Reconciling the projection restores the incumbent's admitted
Expected where there is one, which is what lets continuity bring the incumbent
back after fencing stopped it; where there is none, the failed Expected is
withdrawn.
The transaction is released only once every one of those is durably complete,
which is what the cleanup evidence is for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reachability check only accepted o+x, but the other correct shape for a
runtime root is 0750 group-owned by the state group -- and that is the shape
the write-lease hardening already assumes, since it derives the record's group
from this directory. Checking only o+x would have refused the configuration
that makes the lease readable.
The workload's uid is allocated by DynamicUser and owns nothing, so traversal
can only come from the state group or from other. Both are now accepted, and
a directory group-owned by some *other* group still is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first deployment of any routed target could never succeed, and wedged
itself permanently on the way down.
The route preflight bind-mounts its candidate over config_path to run
`nginx -t` in a private mount namespace. When that path does not exist yet,
systemd materializes the mount point and leaves an empty file on the host.
Read as content, that made the baseline "change" during validation, failing
the transaction -- and every retry then saw a zero-length fragment where the
transaction expected none and reported an unadmitted incumbent, which no
amount of retrying could clear.
A fragment with no bytes grants no route, so it is now read as absence. That
makes the preflight idempotent whether or not systemd materialized the mount
point, and lets a first deployment retry cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The topology store is Idunn's published surface -- every managed target reads
it to verify its own Expected incarnation against the Idunn anchor -- but
Idunn runs with UMask=027, so each publish left it 0640 root:root and a
DynamicUser workload got EACCES. Odin's candidate died on exactly that.
Fixing the mode by hand does not hold, because every publish writes a new
file. Integrity here comes from the signatures over the records, not from the
mode, so the published copy is now explicitly readable after each write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultLib's backing store reports an unreadable file as an empty store, so a
workload that cannot traverse to its runtime bundle sees zero records rather
than a permission error. Odin reported "runtime authority store must contain
exactly one record" and Heimdall simply concluded it held no write lease and
warmed forever. Neither names the actual fault, and both cost real time.
The cause is that `ReadOnlyPaths=` binds the leaf into the unit's namespace
but grants no traversal above it, so a runtime root under a 0700 parent is
unreachable to a DynamicUser. Idunn now checks the whole chain when it writes
the bundle and says exactly which directory is closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five things that had never been exercised failed on the way through Odin, and
each is now a test: the unprojectable RUDP route, step environment not being
ambient, build containers having no machine identity, the published projection
sitting inside Idunn's private root, and a signing identity living in the
state root the workload can write.
Also documents that `idunn validate` proves nothing about deployability
without a binding -- admit is where every cross-document rule lives -- and
repairs apostrophes an earlier pass mangled into ".s".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A target whose tests enrol a service identity could not be built at all.
CultLib's Linux protector binds the seed to /etc/machine-id, the build image
carries none, and the runner is --read-only, so Odin's suite failed every
step with "Linux machine-id is unavailable".
Mounting the host's machine-id would let a build container protect a seed
that unwraps on the host, which is exactly the property the binding exists to
deny. So each frozen workspace now gets its own identity, derived from the
workspace path and mounted read-only: stable for the life of a build, and
useless anywhere else including this host.
Also moves Idunn's topology store to /var/lib/gamecult/idunn-projection. It
is Idunn's published surface -- every managed target must read it to verify
its own Expected incarnation -- and it was sitting inside Idunn's private
0750 root where no workload could traverse to it. Odin's own daemon already
named that directory in its tests.
Odin's three roots become writable for the same reason Heimdall's are: a
target Idunn cannot write is a target it cannot deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A routed RUDP target was admissible but unprojectable. `admit` has an
explicit arm for (route_required, Rudp, nginx-stream-udp) against an
`rudp://` stable endpoint, and the nginx driver already writes
`listen ... udp reuseport` for it -- but `endpoint_scheme` lumped Rudp in
with Private and refused to name a scheme, so `expected_projection` bailed
before any routed RUDP candidate could be described.
Odin is the only routed RUDP target and it is the bootstrap target: nothing
else in the estate can warm until Odin is admitted, and Odin could not be.
Private keeps bailing, which is correct -- `route_required` is false for it,
so a private target never reaches this function.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying Heimdall proved the migration order backwards. A candidate's
warming presence -- the evidence Idunn requires before fencing an incumbent
-- is observed through Odin's authenticated runtime topology correlation.
advance_warming has exactly one exception: target == "odin" while no Odin
is admitted may warm on directly observed presence. Every other target
waits for Odin to publish.
So with the previous generation's Odin still running, a Heimdall candidate
starts, serves on its assigned bind, publishes health, and warms forever.
It never fails and never promotes. That is not a defect; it is the
observation chain being rooted somewhere I had not looked. It also explains
why Odin has the only recipe in the estate that was ever kept current.
The document said "migrate the authority's own dependencies last". Odin is
not a dependency to defer, it is the root, and the entry now says so.
The wait also recorded no reason, so a candidate could sit in Warming
indefinitely with nothing to read -- the fourth coarse-signal case today.
It now records what it is waiting for and why the wait cannot resolve
without Odin.
Noted alongside: health published over RUDP to the previous generation's
--rudp-health-bind goes nowhere, since the current serve has no such flag.
That produces a steady stream of timeouts in the candidate's journal and is
not the reason promotion stalls.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The launch-contract assertion carried the same empty-User clause I removed
from observe_unit an hour ago -- a second copy in a second checker, failing
for the same reason: systemd reports the identity it allocated for a
DynamicUser unit, so User is never empty for a running workload.
It also bundled thirty conditions behind one message. "The launch contract
differs from the admitted launch" leaves an operator bisecting by hand
against a workload that has already exited, and that is exactly how the
last two rounds went. Each clause now carries a name and the failure says
which one.
That is the third time today a check knew precisely what was wrong and
reported something too coarse to act on: the sealing gate recorded a brake
reason status never printed, the isolation contract hid one property among
twenty-one, and this hid one among thirty. A control plane that cannot say
which invariant it is enforcing costs more than the invariant is worth.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The credential check required the delivered file to be owned by the
workload's uid and gid with mode exactly 0400. systemd does not deliver
credentials that way and never has: it writes them root:root, closed to
group and world, and grants the workload's dynamic uid read access through
a POSIX ACL. Observed directly on systemd 257 rather than inferred:
-r--r-----+ 1 0 0
user::r-- user:64459:r-- group::--- mask::r-- other::---
64459 was the running workload's uid, so the invariant the check wanted --
readable only by the workload identity -- holds exactly. It just holds
through an ACL rather than through ownership, so the assertion failed for
every credential Idunn has ever delivered.
Now asserts what the file mode can actually carry: root-owned, unaliased,
no group or world access. The workload-only grant is the ACL's, which is
systemd's to enforce rather than ours to re-derive; saying so plainly in
the comment beats an assertion that looks strict and cannot pass.
Second unsatisfiable contract in this observer today, after the empty-User
assertion. Both asserted a shape systemd does not produce, and both failed
after every other property matched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workload isolation contract required `User` to be empty. systemd
populates that property with the identity it allocates for a DynamicUser
unit -- the unit name, or a generated `_du…` when the unit name is too long
-- so it is never empty for a running workload. Every workload Idunn starts
failed the check it had just satisfied in every other respect, reported as
"systemd workload isolation properties differ from the Idunn contract"
with no indication of which property differed.
Confirmed against systemd directly rather than reasoned about: a transient
unit with DynamicUser=yes and nothing else reports
User=idunn-user-probe-2869674. All twenty other asserted properties matched
exactly.
The clause could not have distinguished an allocated identity from a pinned
one anyway, since both surface the same way. The invariant it reached for --
the workload does not run as a persistent named account -- is carried by
`dynamic_user` and by Idunn never emitting a User property at all. The
observed value is still recorded, as evidence of which identity ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first launch with a full environment failed EACCES on
/srv/heimdall/keys/signing-key.pem: the workload runs under DynamicUser and
cannot open a heimdall:heimdall 0600 file. The wrong fix is a supplementary
group -- widening access to resolve a permissions error.
Idunn already has the right one. A [workload.secret_files] entry is loaded
as a systemd credential, and the driver sets an env var named after the
entry whose value is the credential path. Naming the entry after the
variable the service already reads wires it up with no code change, and
setting that name in [workload.environment] too is refused as a collision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime presence identity is handed to the workload as an inherited
file descriptor, so Idunn requires the source file to be root-owned, mode
0400, and singly linked. Enrolment leaves it 0600, which fails.
The nlink check is the interesting one: a second hard link would be a
second path to the signing key, and a descriptor that cannot be reopened
is the whole point of passing it this way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProtectSystem=full makes /etc read-only, so Idunn could not create Heimdall's
runtime bundle under /etc/gamecult/heimdall/runtime. A target whose roots are
not in ReadWritePaths is a target Idunn cannot deploy. Every migration adds
exactly three paths -- runtime root, state root, release root -- which keeps
the write set enumerated and is the honest blast radius of running as root.
Also records the per-target preprovisioning Idunn requires rather than
creates, each discovered by a failed deployment: the state root must be
root:<state_group> mode 2770 so the workload's dynamic user inherits group
access through setgid; the runner cache must be 0700 owned by the runner's
container uid, not the source identity, with a root-owned parent (/srv/build
is idunn:idunn and fails); the frozen source stage must be root-owned,
because source is frozen as idunn and then copied into a root-owned
actuation stage; and the binding's docker network must exist.
None of these are Idunn being fussy. Each is a check that a binding cannot
widen access by naming a path someone else can write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The RUDP connection id 0x1d0d0001 appeared nowhere in this repository. It lived
only in Odin's publisher implementation, which meant the constant defining how
to reach Idunn was owned by a consumer.
The three constants are now stated here: connection id, signed statement schema,
and the unsigned diagnostic schema. The publishing transport moves to CultLib as
a generic module that takes them as a contract, so consumers declare them
against this document rather than each carrying their own copy of the transport.
Recorded a discrepancy while writing it down: the wire value for the unsigned
schema is idunn.daemon_health, with no version suffix, while prose in guide.md
and earlier in this file says idunn.daemon_health.v1. The table gives the wire
value. Reconciling the prose is a separate question about whether the wire
should carry the suffix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CultLib forks do not disagree. cultcache-ts reads Idunn.s live control
store -- 36 records -- and ordinary CultCache stores share one framing
across runtimes. Only service identity private stores use the bare
single-envelope container, deliberately, because they hold a key rather
than a record set.
The section now says which shape is which and names the failure it nearly
caused: a consumer that hand-decodes the private-store framing and applies
it to expected.cc and the write lease fails on first deployment while its
tests pass, because hand-built fixtures agree with the hand-built reader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enrolling Heimdall's signer surfaced that cultcache-ts cannot read a
single-file store written by cultcache-rs at all: pullAll() throws
invalid_type, expected object received array. Verified against the real
idunn-provision enrolment on yggdrasil.
Idunn writes every store a target must read -- presence identity, process
write lease, expected incarnation -- with the Rust client. Rust targets
consume them natively; TypeScript targets cannot, and must decode the
layout by hand. That makes this a per-runtime blocker rather than a
Heimdall quirk, so it belongs in the migration route.
Worth separating what broke from what held: the record schemas are
identical across runtimes, including the identity's protector binding. The
contracts are fine. The store framing drifted.
Which framing is canonical is a CultLib decision and is not taken here; the
C# implementation is the stated reference and is the tiebreaker.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shared prerequisite for every routed target, done on yggdrasil. Debian
ships ngx_stream_module as a separate dynamic package and it was absent:
nginx -V showed --with-stream=dynamic with no .so. Installed
libnginx-mod-stream at 1.26.3-3+deb13u7, matching the running nginx
exactly so nothing else moved, added a top-level stream block including
/etc/nginx/idunn-stream-routes/*.conf, and created that directory
root:idunn 0775. nginx -t passed before the reload; gamecult.org and
heimdall.gamecult.org/healthz both returned 200 after it. Backup at
/root/nginx.conf.bak-20260906-stream.
Worth recording how the module check nearly went wrong: the first
`nginx -V` ran without sudo, hit "command not found" because the binary is
not on the unprivileged PATH, and reported NO STREAM MODULE. That is the
estate's own trap -- a failed read presented as a negative result -- caught
only because the answer looked too convenient.
The larger correction: the unit I wrote yesterday ran User=idunn, and this
generation cannot work that way. There is no sudo anywhere in Idunn's
source; it calls systemctl, systemd-run, nginx and docker directly, so a
confined user actuates nothing. deployment-authority.md states the intended
model plainly -- freeze source as idunn, copy into a root-owned actuation
stage, the privileged driver never opens Git -- and --source-uid/--source-gid
are validated as non-zero, which only makes sense for a privileged daemon
dropping down for source work.
So the unit now runs as root with --source-uid 986 --source-gid 978, keeps
ProtectSystem=full and re-opens each actuated path individually, and adds
RuntimeDirectory=idunn for the route driver's preflight staging. The
comment block explains the tradeoff rather than leaving the next reader to
infer it: privilege separation is internal, and what makes it tolerable is
that Idunn accepts no imperative input.
That is a genuine increase in blast radius over a confined process, and
migration.md says so instead of selling it. The counter-argument is that
the old shape was confined in name only -- a sandboxed process whose sudo
grant pointed at an unaudited root shell with eleven hardcoded targets.
systemd-analyze verify passes on the rewritten unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
StreamPixels is changing direction: 2D animation could not express what
the spec required, so it is moving to a 3D-rendered overlay, and whether
that runs in WebGL or needs dedicated rendering infrastructure is open.
The client funding it cannot absorb rendering overhead, which is what makes
the WebGL question load-bearing rather than a preference.
Its deployment shape is downstream of that answer -- a WebGL overlay keeps
roughly the current two-unit shape, rendering infrastructure is a new
target class with hardware requirements this host has never expressed. A
binding written now would pin the shape being replaced.
The survey of its current defects stays as a record but is marked
superseded, with an explicit instruction not to act on it. The tsx-at-
runtime path is exactly the kind of thing a passing agent would tidy, and
tidying it would be work spent on something being replaced.
heimdall leads the remaining three instead. Its prerequisite is deeper
than repixelizer's but far better understood: a vendor/CultLib gitlink,
following Ghostlight's existing vendor/eve pattern, replacing a
cultlib_commit hardcoded in a shell manifest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I called these three "the simple ones" and "compose-shaped". Both were
wrong. None of them is compose -- the legacy actuator restarts all three
with `docker compose -f /srv/compose/yggdrasil-apps.yaml`, and that file
does not exist; /srv/compose holds only odin, voidbot and voidbot-retrieval.
The path is guarded, so every restart of these three has been failing at
the guard. They are running only because nothing has asked them to restart,
which is the difference between a supervised service and one that happens
to be up.
Each needs a change in its own repository before a recipe means anything,
and the changes are different shapes, so this is not one piece of work:
streampixels has real build scripts (tsc, next build) and no external
dependencies, but its service unit runs TypeScript through tsx from a
checkout, and it is two units under one target name. A recipe declares one
[service]; the two have different environments, ports, and an ordering
dependency. It was one target only because a shell line restarted both.
repixelizer has the cleanest dependencies of the three -- no CultLib, no
submodules -- but runs a venv interpreter against a checkout, so there is
no artifact to seal and no revision observable from the process.
heimdall is the most release-shaped and needs the deepest change: it takes
CultLib as file:../CultLib, a sibling checkout outside the repository. The
current manifest works around that by cloning CultLib separately with
cultlib_commit hardcoded in shell -- exactly the untyped authority recipes
exist to replace. It needs vendor/CultLib as a gitlink, like Ghostlight's
vendor/eve.
Also corrects the advice to copy CodexConnector's recipe. That recipe is
stale: it declares required_gitlinks for vendor/cultcache-rs and
vendor/cultnet-rs, and the repo has no .gitmodules, no vendor/, and takes
CultLib as a pinned cargo git dep instead. Recipes drift from their repos
silently because nothing re-checks them; run `idunn validate` in each
target's CI so drift fails at the repo rather than on the host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The migration procedure told operators to dry-run a target, and no dry-run
existed. `up` needs a running daemon and a state store, so the only way to
find out whether a binding was well-formed was to install it and see
whether `serve` refused to start -- on the host, with the authority down.
`idunn validate --recipe PATH [--binding PATH]` runs the same
TargetDeclaration::parse and OperatorBinding::admit the daemon runs at
load. It opens no store, contacts no host, and actuates nothing.
Owner: the existing control-plane CLI, no new module or target. Consumer:
whoever authors a binding, starting with this migration. Invariant: a
binding installed on the host admits against its target's recipe. It
replaces the install-and-see loop rather than adding a layer.
Verified against the real estate: Odin, CodexConnector and Ghostlight
recipes all parse. Negative paths checked too, because a validator that
only ever passes is worse than none -- an unknown recipe field (the
privilege-escalation shape), a missing --recipe, and an unreadable file
all fail with the reason.
A test asserts validate stays declarative and refuses --deploy-command,
matching the guarantee cli_exposes_only_declarative_commands makes for the
rest of the surface. 64 passed / 18 failed, against a 63/18 baseline: one
added pass, no new failures, and the 18 remain the known Windows
absolute-path cases. cargo check for x86_64-unknown-linux-gnu is clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gjallar had a brief useful life on Nightwing's framebuffer, broke, and was
never fixed. Its Yggdrasil incarnation ran --headless --refresh-hz 2 for
eleven days: a compositor with no display recomposing twice a second, 55
minutes of CPU and 125M resident producing frames nothing rendered. The
only reference to gjallar.overview in the estate is one Odin test file.
Stopped and disabled; 420M under /srv/gjallar is reclaimable.
Removed the codex-connector and gjallar sudo grants from the host and
synced deploy/legacy/idunn-yggdrasil.sudoers to match. Four grants gone,
verified by diffing the grant set against a backup, nothing else changed.
That diff caught a defect I introduced: removing the last entry of a
Cmnd_Alias leaves a dangling continuation, and my repair silently dropped
the wildcard from `deploy streampixels`, which would have failed every
streampixels deployment. The procedure now says validate with visudo -c -f
before installing and diff the grant set afterwards.
The larger finding: the installed generation CANNOT express that a target
is retired. Its target list is compiled in, it offers restart and redeploy
and no way to drop a daemon from supervision, and its only brake is the
deployment brake -- which doctrine forbids using for this, since a
deployment brake must not suspend continuity. Stopping codex-connector
therefore produced eighty "decision: restart" lines in ten minutes against
a service we had just retired. Suspending continuity for one target is
precisely the current generation's lifecycle brake. The estate found the
argument for its own migration by trying to throw something away.
Gjallar was handled correctly by contrast: its failure classified as
dependency-unavailable, so Idunn raised an alarm instead of resurrecting
it. The old generation can tell failures it owns from failures it does
not; it cannot tell a broken target from a finished one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stopped and disabled on yggdrasil 2026-09-05; port 4103 closed, the
three-minute 401 loop ended. It had burned 1min 30s CPU and peaked at 1.2G
retrying expired credentials against a provider the estate has left.
Two things the stop surfaced, both worth having written down:
ghostlight-dungeon.service still carries GHOSTLIGHT_MODEL_CONNECTOR
pointed at 127.0.0.1:4103, which is now nothing. It had logged nothing for
24 hours beforehand so no working path broke, but that environment variable
is where the provider replacement lands first.
epiphany-model-connector.service binds the same port with --model gpt-5.4
and its own --codex-home. It was already inactive and disabled. Two Codex
connectors were bound to one endpoint, one of which has no repository in
the swarm -- duplicate authority of exactly the shape the census went
looking for, found by stopping one of them rather than by reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator is not returning to Codex, which removes the consumer for a
credential-isolated Codex subscription transport. The deployed daemon is
already dead in practice: codex-connector.service on yggdrasil is active but
has been failing 401 token_expired every three minutes against
chatgpt.com/backend-api/codex/models, with nothing connected to port 4103.
It was this document's first migration target; writing a binding for it
would have been work spent on a corpse. gjallar leads now.
Adds a Retiring Codex section with the blast radius in removal order --
the running unit, the sudoers grants, GameCult/CodexConnector,
Epiphany's openai-codex-spine and codex-bridge plus vendored app-server,
VoidBot's Codex model config, and the AGENTS.md doctrine twin maintained
for a second agent runtime.
It also says what NOT to do: Epiphany and Ghostlight still need an
inference transport, and CodexConnector was one implementation of that
seam rather than the seam itself. Deleting the boundary and letting each
mind grow its own HTTP client would re-fork exactly the wire law that
README exists to prevent. Replace the backend, keep the seam, and settle
that before writing epiphany bindings.
Also corrects a false claim I made in the first draft. Odin's recipe is
not the only worked example -- there are eight, including Ghostlight's and
four Ghostlight-worlds variants. Recipe authoring is not the bottleneck
for those targets; bindings and the daemon cutover are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
