The first deployment of any routed target could never succeed, and wedged
itself permanently on the way down.
The route preflight bind-mounts its candidate over config_path to run
`nginx -t` in a private mount namespace. When that path does not exist yet,
systemd materializes the mount point and leaves an empty file on the host.
Read as content, that made the baseline "change" during validation, failing
the transaction -- and every retry then saw a zero-length fragment where the
transaction expected none and reported an unadmitted incumbent, which no
amount of retrying could clear.
A fragment with no bytes grants no route, so it is now read as absence. That
makes the preflight idempotent whether or not systemd materialized the mount
point, and lets a first deployment retry cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The topology store is Idunn's published surface -- every managed target reads
it to verify its own Expected incarnation against the Idunn anchor -- but
Idunn runs with UMask=027, so each publish left it 0640 root:root and a
DynamicUser workload got EACCES. Odin's candidate died on exactly that.
Fixing the mode by hand does not hold, because every publish writes a new
file. Integrity here comes from the signatures over the records, not from the
mode, so the published copy is now explicitly readable after each write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultLib's backing store reports an unreadable file as an empty store, so a
workload that cannot traverse to its runtime bundle sees zero records rather
than a permission error. Odin reported "runtime authority store must contain
exactly one record" and Heimdall simply concluded it held no write lease and
warmed forever. Neither names the actual fault, and both cost real time.
The cause is that `ReadOnlyPaths=` binds the leaf into the unit's namespace
but grants no traversal above it, so a runtime root under a 0700 parent is
unreachable to a DynamicUser. Idunn now checks the whole chain when it writes
the bundle and says exactly which directory is closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A target whose tests enrol a service identity could not be built at all.
CultLib's Linux protector binds the seed to /etc/machine-id, the build image
carries none, and the runner is --read-only, so Odin's suite failed every
step with "Linux machine-id is unavailable".
Mounting the host's machine-id would let a build container protect a seed
that unwraps on the host, which is exactly the property the binding exists to
deny. So each frozen workspace now gets its own identity, derived from the
workspace path and mounted read-only: stable for the life of a build, and
useless anywhere else including this host.
Also moves Idunn's topology store to /var/lib/gamecult/idunn-projection. It
is Idunn's published surface -- every managed target must read it to verify
its own Expected incarnation -- and it was sitting inside Idunn's private
0750 root where no workload could traverse to it. Odin's own daemon already
named that directory in its tests.
Odin's three roots become writable for the same reason Heimdall's are: a
target Idunn cannot write is a target it cannot deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A routed RUDP target was admissible but unprojectable. `admit` has an
explicit arm for (route_required, Rudp, nginx-stream-udp) against an
`rudp://` stable endpoint, and the nginx driver already writes
`listen ... udp reuseport` for it -- but `endpoint_scheme` lumped Rudp in
with Private and refused to name a scheme, so `expected_projection` bailed
before any routed RUDP candidate could be described.
Odin is the only routed RUDP target and it is the bootstrap target: nothing
else in the estate can warm until Odin is admitted, and Odin could not be.
Private keeps bailing, which is correct -- `route_required` is false for it,
so a private target never reaches this function.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying Heimdall proved the migration order backwards. A candidate's
warming presence -- the evidence Idunn requires before fencing an incumbent
-- is observed through Odin's authenticated runtime topology correlation.
advance_warming has exactly one exception: target == "odin" while no Odin
is admitted may warm on directly observed presence. Every other target
waits for Odin to publish.
So with the previous generation's Odin still running, a Heimdall candidate
starts, serves on its assigned bind, publishes health, and warms forever.
It never fails and never promotes. That is not a defect; it is the
observation chain being rooted somewhere I had not looked. It also explains
why Odin has the only recipe in the estate that was ever kept current.
The document said "migrate the authority's own dependencies last". Odin is
not a dependency to defer, it is the root, and the entry now says so.
The wait also recorded no reason, so a candidate could sit in Warming
indefinitely with nothing to read -- the fourth coarse-signal case today.
It now records what it is waiting for and why the wait cannot resolve
without Odin.
Noted alongside: health published over RUDP to the previous generation's
--rudp-health-bind goes nowhere, since the current serve has no such flag.
That produces a steady stream of timeouts in the candidate's journal and is
not the reason promotion stalls.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The launch-contract assertion carried the same empty-User clause I removed
from observe_unit an hour ago -- a second copy in a second checker, failing
for the same reason: systemd reports the identity it allocated for a
DynamicUser unit, so User is never empty for a running workload.
It also bundled thirty conditions behind one message. "The launch contract
differs from the admitted launch" leaves an operator bisecting by hand
against a workload that has already exited, and that is exactly how the
last two rounds went. Each clause now carries a name and the failure says
which one.
That is the third time today a check knew precisely what was wrong and
reported something too coarse to act on: the sealing gate recorded a brake
reason status never printed, the isolation contract hid one property among
twenty-one, and this hid one among thirty. A control plane that cannot say
which invariant it is enforcing costs more than the invariant is worth.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The credential check required the delivered file to be owned by the
workload's uid and gid with mode exactly 0400. systemd does not deliver
credentials that way and never has: it writes them root:root, closed to
group and world, and grants the workload's dynamic uid read access through
a POSIX ACL. Observed directly on systemd 257 rather than inferred:
-r--r-----+ 1 0 0
user::r-- user:64459:r-- group::--- mask::r-- other::---
64459 was the running workload's uid, so the invariant the check wanted --
readable only by the workload identity -- holds exactly. It just holds
through an ACL rather than through ownership, so the assertion failed for
every credential Idunn has ever delivered.
Now asserts what the file mode can actually carry: root-owned, unaliased,
no group or world access. The workload-only grant is the ACL's, which is
systemd's to enforce rather than ours to re-derive; saying so plainly in
the comment beats an assertion that looks strict and cannot pass.
Second unsatisfiable contract in this observer today, after the empty-User
assertion. Both asserted a shape systemd does not produce, and both failed
after every other property matched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workload isolation contract required `User` to be empty. systemd
populates that property with the identity it allocates for a DynamicUser
unit -- the unit name, or a generated `_du…` when the unit name is too long
-- so it is never empty for a running workload. Every workload Idunn starts
failed the check it had just satisfied in every other respect, reported as
"systemd workload isolation properties differ from the Idunn contract"
with no indication of which property differed.
Confirmed against systemd directly rather than reasoned about: a transient
unit with DynamicUser=yes and nothing else reports
User=idunn-user-probe-2869674. All twenty other asserted properties matched
exactly.
The clause could not have distinguished an allocated identity from a pinned
one anyway, since both surface the same way. The invariant it reached for --
the workload does not run as a persistent named account -- is carried by
`dynamic_user` and by Idunn never emitting a User property at all. The
observed value is still recorded, as evidence of which identity ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The migration procedure told operators to dry-run a target, and no dry-run
existed. `up` needs a running daemon and a state store, so the only way to
find out whether a binding was well-formed was to install it and see
whether `serve` refused to start -- on the host, with the authority down.
`idunn validate --recipe PATH [--binding PATH]` runs the same
TargetDeclaration::parse and OperatorBinding::admit the daemon runs at
load. It opens no store, contacts no host, and actuates nothing.
Owner: the existing control-plane CLI, no new module or target. Consumer:
whoever authors a binding, starting with this migration. Invariant: a
binding installed on the host admits against its target's recipe. It
replaces the install-and-see loop rather than adding a layer.
Verified against the real estate: Odin, CodexConnector and Ghostlight
recipes all parse. Negative paths checked too, because a validator that
only ever passes is worse than none -- an unknown recipe field (the
privilege-escalation shape), a missing --recipe, and an unreadable file
all fail with the reason.
A test asserts validate stays declarative and refuses --deploy-command,
matching the guarantee cli_exposes_only_declarative_commands makes for the
rest of the surface. 64 passed / 18 failed, against a 63/18 baseline: one
added pass, no new failures, and the 18 remain the known Windows
absolute-path cases. cargo check for x86_64-unknown-linux-gnu is clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
IdunnDaemonHealthTrustBindingRecord and its schema constant moved to
cultnet-rs (GameCult/CultLib@c13b6ba). odin-core now imports them from
cultnet_rs instead of defining them, and idunn-daemon/provisioning.rs
pulls all six provider-health symbols from cultnet_rs directly, merged
into its existing cultnet_rs use block. Dropped the odin-core path
dependency from idunn-daemon/Cargo.toml — nothing in the crate reaches
odin_core:: anymore. Bumped the CultLib git rev across all six
consuming manifests to the new commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This had been sitting uncommitted in the worktree. It is real work rather than
stray edits: RouteBinding gains stable_socket and an NginxStreamUdp driver so a
route can be actuated over UDP, control_plane and drivers grow the Idunn-owned
projection and bounded RUDP transport, and crates/odin-daemon arrives as a new
member of the workspace.
The deployment recipe comes with it, including the required_gitlinks fix the
core-stack repoint made necessary: it named vendor/cultnet-rs, which no longer
exists now that CultNet comes from CultLib, so Idunn would have refused to admit
the target. The list is empty because Odin has no submodules left; the core
stack is pinned by revision in Cargo.lock inside the frozen source tree.
Everything except odin-daemon builds here. odin-daemon reads systemd listen
file descriptors through std::os::fd and cannot compile on Windows by design, so
Linux CI is its first real witness. muninn and sleipnir pass 162 tests; the one
failure expects an Unreliable media channel that no CultNet lineage has ever
declared, and predates this.
Committed rather than left in flight. rustfmt reports 94 files across the repo,
which predates this work, so formatting is left for its own pass instead of
being buried in a 4,000-line diff.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
