Two CI tests and one ignored live trace against HEAD ed1dc48.
soul4_terminal_record_left_by_a_crashed_admission_commit_is_never_moved_and_never_blocks
proves the R23.2 crash window: advance_committing writes its born-terminal
record and AdmittedGeneration in one CAS and only then archives, and with
the sweep deleted a crash between the two leaves that record resident in
control.cc forever -- unarchived, invisible to read_history_for_command,
and not blocking its target. A leak into the byte-exact live set, not a
wedge.
soul4_complete_record_with_pending_cleanup_cannot_take_the_abort_r23_3_routes_it_to
proves that for the Complete-with-pending-cleanup wedge R23.3's commit
message names, begin_post_fencing_abort cannot persist: validate refuses
"post-fencing abort collides with another terminal path", so the tick
returns that error instead, last_error is never written, and the cleanup
is retried next tick exactly as before the change.
soul4_live_odin_committing_record_fate_on_first_boot (ignored, SOUL4_ROOT)
seeds a sandbox with yggdrasil's live odin command, transaction and
AdmittedGeneration verbatim plus the live public anchors and Odin's live
correlation store, and runs the real resume_one_transaction: tick 1 makes
a post-fencing abort durable with every cleanup Pending. The candidate it
would stop is the Odin unit currently active on the host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The flipped version still broke history.cc (as a directory) to prove
the old sweep's failure mode. That was wrong once the sweep was gone:
the live transaction's OWN finalization also archives to history.cc,
so breaking it made the live transaction fail to finalize too, for a
reason that has nothing to do with the sweep. Leave history.cc healthy
so the test isolates its actual claim -- no sweep runs ahead of the
candidate loop -- from a separate, already-covered claim about
archival failure. Assert the live transaction's own archival
(disappears from control.cc, appears in history.cc) instead of
expecting it to still be resident and terminal, since successful
archival is exactly what removes it from control.cc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four pre-existing tests from earlier passes (short_record_is_refused_not_tolerated,
soul_live_record_in_foreign_encoding_is_refused_by_control_read,
soul2_shortened_record_escapes_the_byte_exact_check,
soul2_one_noncanonical_record_starves_every_scheduler_tick) still
asserted R20's quarantine behaviour (bad record silently absent from
the snapshot) that R23.1 deleted; they were not named soul3_* so the
earlier pass missed them. Flip them to assert refusal.
soul2_stateful_continuity_admission_mints_a_terminal_record_into_control
was a direct test of the resident-terminal sweep R23.2 deleted; rewrite
it to prove there is no sweep left to find a pre-existing resident
terminal record; the production path it originally stood in for is now
covered by advance_committing's own archive call.
The R23.3 post-fencing-abort fix also had a real bug: dropping the
candidate_is_permanently_stopped gate without also requiring
phase >= Fencing meant an advance error on a transaction whose
pre-fencing abort was ALREADY in progress (one of its own cleanup
steps failing, phase still < Fencing) fell into the post-fencing
branch and hit begin_post_fencing_abort's own phase guard, ensure!,
turning a resumable retry into a hard error. Restore the phase >=
Fencing condition on that branch; a pre-fencing abort already underway
still retries via record_resumable_error, as before.
Found by running the full suite on the build host, not by inspection.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Its only production caller was the arity exemption R20 already
deleted; it survived as #[cfg(test)]-only, used by exactly one
assertion that checked a hand-built payload's slot count as an
intermediate step before the test's real assertion (that the record is
refused). Drop the assertion along with the function it called.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
verify_resident_deployment_authorization's own doc comment already
promised that a failure aborts the transaction through the pre-/post-
fencing abort path for any advance error. The pre-fencing side already
did that unconditionally. The post-fencing side did not: it only
called begin_post_fencing_abort when candidate_is_permanently_stopped,
and otherwise fell through to record_resumable_error forever. A
rotated brake operator identity, or a Complete record whose post-
commit cleanup can never re-authorize, wedged the target permanently
with its incumbent fenced (or its cleanup pending) and no path off
that state -- blocking new target mutation forever, since
blocks_new_target_mutation is true for exactly this state.
Drop the permanently-stopped gate so a post-fencing advance error
aborts on its first occurrence, same as a pre-fencing one, and delete
candidate_is_permanently_stopped now that its only caller is gone. The
code now matches what its own doc comment already claimed (R3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
archive_rejected_transaction and the resident-terminal sweep were two
answers to one problem. The sweep was the wrong survivor: it ran in
resume_one_transaction's propagating prefix, so an unwritable
history.cc starved every live transaction on every other target before
a single one of them got touched -- the same whole-tick starvation the
R23.1 quarantine deletion was fixing, one function later. On the live
store it would have swept 252 resident terminal records serially,
re-reading a 2.5MB file each time, before ever reaching a live
transaction.
Factor the archive write into the single primitive
archive_terminal_transaction already provides and have
advance_committing call it directly: a stateful Continuity admission
with no pending incumbent cleanup mints its transaction terminal at
birth, and now archives that record to history.cc in the same call
that writes it and its AdmittedGeneration, instead of leaving it
resident for a sweep to find later. replace_transaction keeps calling
the same primitive for a transaction that turns terminal while already
resident. One primitive, two callers, no sweep.
Flip the starvation test to the surviving shape, and delete the sweep-
idempotency test: the primitive-level idempotency claim it was making
already has a home in
archive_terminal_transaction_tolerates_a_pre_existing_history_entry,
which drives the primitive directly rather than through the deleted
sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ControlSnapshot::read/ingest logged and skipped a record that failed
ingest instead of failing the read. run_scheduler_tick already keeps
the daemon alive across a failed read (R4's second clause), so the
skip bought nothing except letting Idunn go blind to exactly the
records worth being suspicious of: Soul proved a one-byte-noncanonical
live transaction lets freeze_one_queued_command mint a second owner
for its target, and a garbled AdmittedGeneration reads as never-
deployed and rolls Odin authority back to the bootstrap anchor.
Flip the two soul3 tests that asserted the quarantine's silent-drop
behaviour to assert refusal instead, and rewrite the two schema tests
the same way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four tests demonstrating that ControlSnapshot::read's quarantine fails
open for every decision a skipped record gated, that the restored
resident-terminal sweep is a new whole-tick starvation point, that the
sweep is idempotent against a pre-existing history entry, and one
ignored probe that drives the real read over a copy of yggdrasil's
control.cc (507 records, 0 quarantined, read Ok).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
authorization off startup, quarantine bad records at read, and restore
the resident-terminal sweep
Per rulings-wave-1.md R20 (third pass on this branch, Soul's second
falsification):
1. decode_record_canonical: deleted the stored_len < canonical_len
exemption outright instead of repairing it further. Measured the
live control.cc through Idunn's own decoder first (throwaway
#[ignore] test, removed before this commit): 507 records -- 253
commands, 253 transactions, 1 admitted generation -- all already at
full current arity. Zero short records exist, so there is nothing
to be lenient toward; a future field addition is a migration to run
at that moment, not a standing hole in this check.
2. validate_durable_authority is gone from serve. Warming and ready
evidence were already re-authenticated at their own consumption
points on every resume (rehydrate_warming_token,
rehydrate_ready_token, rehydrate_admitted_ready) -- that part of
the old startup gate was pure duplication. The one real gap was the
deployment brake authorization, which nothing re-verified after
Sealing consumed it once. Added
verify_resident_deployment_authorization, called from the top of
advance_transaction for every resident transaction that carries
one: a rotated operator identity now fails only that transaction's
advance (caught by resume_one_transaction's existing pre-/post-
fencing abort handling), never serve. An empty store, or one with
no Deploy transaction past Sealing, never touches the brake anchor
at all.
3. ControlSnapshot::read no longer fails whole on one bad record.
Split the per-envelope decode into ControlSnapshot::ingest; a
record that fails it is logged and skipped, and the read returns
everything else. A quarantined record is refused from acting as
live authority by omission rather than by taking every other
record in the store down with it -- which also means the byte-
exact/foreign-encoding/short-record refusals from cut 1 now surface
as the tampered record being absent from the snapshot, not as
ControlSnapshot::read itself erroring.
4. Restored the resident-terminal sweep in resume_one_transaction
(fef226a's shape), rather than routing the admission commit through
replace_transaction: that commit writes the Complete transaction
and the new AdmittedGeneration atomically in one compare_exchange,
which replace_transaction cannot do since it only ever moves a
single already-resident record. The sweep finds any transaction
already terminal in control.cc -- born terminal by that commit, or
left by a crash mid-archive -- and archives it through
replace_transaction before scheduling anything else.
All five soul2_* tests flipped to assert the closed behaviour instead
of the hole (short_record_is_refused_not_tolerated,
soul2_shortened_record_escapes_the_byte_exact_check,
soul2_startup_authority_check_needs_the_brake_anchor_with_no_records,
soul2_one_live_record_with_a_rotated_brake_signer_refuses_startup,
soul2_one_noncanonical_record_starves_every_scheduler_tick,
soul2_stateful_continuity_admission_mints_a_terminal_record_into_control),
none deleted. Two more pre-existing tests
(legacy_mutable_command_schema_is_quarantined,
loadcredential_era_control_schemas_are_quarantined_not_rehydrated) and
one Soul claim-1 test (soul_live_record_in_foreign_encoding_is_refused_by_control_read)
updated for the same quarantine-not-refuse semantics.
Net -17 lines against the second pass (312 insertions, 329 deletions)
and net -71 against origin/main on source with tests excluded (was
+859 stated in R20; this cut's source-only delta is roughly flat to
slightly negative -- see the CutReport for the exact numbers). All 111
tests pass under `cargo test --locked` in the Debian/rustc 1.95
container on ygg with /etc/machine-id mounted.
Deviation from "commit each cut separately": landed as one commit.
The four cuts touch overlapping regions of the same file and, more to
the point, several tests had to be re-edited a second time once cut 3
(quarantine) was in place -- cut 1's and Soul's claim-1 tests originally
asserted a hard ControlSnapshot::read refusal, and quarantine changed
that refusal into an absence-from-snapshot instead. A four-way split
along the original cut boundaries would not have been independently
buildable/testable at each step without redoing that same test
rewrite twice. Splitting was not attempted rather than faked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five tests that document what the d2dbfe3..2d65785 surface actually does:
the shortened-array exemption in decode_record_canonical is attacker-
selectable, the restored startup authority gate refuses boot on one live
record (or on a missing brake anchor with no records), a stateful
continuity admission mints a terminal record straight into control.cc that
nothing archives, and one noncanonical record starves every scheduler tick.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Result::expect_err requires the Ok variant to implement Debug for its
panic message. ControlSnapshot intentionally does not derive Debug;
match on the Result instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per R19 item 1: control.cc reads keep the byte-exact canonical check;
history.cc stays lenient. The literal e8d8747-era check (decoded value
re-encodes to exactly the stored bytes) can't be restored unmodified,
though: cultcache-rs-derive's DatabaseEntry::serialize always emits
every slot up to the type's current maximum key, so a record written
before a field existed (the additive-slot tolerance this same cut
added, see additive_slot_does_not_break_stored_records) can never
re-encode to its own stored length, let alone match it byte-for-byte.
That is schema growth, not tampering.
decode_record_canonical reads the stored msgpack array's slot count
via msgpack_array_len and only requires a byte-exact match when it
equals the current schema's slot count. A shorter (genuinely older)
record is exempted from the byte match (decode-time tolerance already
governs whether it is accepted at all); a full-arity record must be
exactly what Idunn's own serializer would have written, which is what
makes Soul's claim 1 attack -- re-encoding an existing field with a
wider msgpack integer width, same value, more bytes, same slot count
-- visible again.
read_transaction_record takes a `canonical` flag so the one function
serving both stores can enforce it for control.cc and skip it for
history.cc. Soul's test is flipped to
soul_live_record_in_foreign_encoding_is_refused_by_control_read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
plan_id is a content digest, not a signature: a stored plan's
binding_blob can be edited after freeze (here, widening
allowed_programs from ["cargo"] to ["sh"]) and recomputing plan_id
produces a self-consistent, validate()-passing record. admit() was the
only check that ever refused that pair, and removing it let an edited
binding reach the phase engine with no admission at all.
Per R19 item 3: this is the recipe/binding admission boundary, not the
terminal-history coupling R4/R11/R12 cut -- re-admitting a resident
plan on every read (restart included) is what lets Idunn run as root at
all. Soul's test is flipped to assert refusal on both validate() and
parsed_inputs(): soul_stored_plan_with_inadmissible_binding_is_refused_on_every_read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
replace_transaction archives a transaction to history.cc the instant it
turns terminal instead of writing the terminal value back to control.cc.
resume_one_transaction's post-advance progress check treated the
resulting absence from control.cc as "transaction disappeared", a fatal
error that serve's `?` propagated -- so every ordinary completion ended
the daemon process once under Restart=always, strictly worse than the
incident this cut exists to fix.
Soul's test is flipped to assert success: soul_terminal_completion_inside_resume_is_ordinary_progress.
Per Soul falsification claim 3 / R19 item 4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four tests, no source changes:
- soul_live_record_in_foreign_encoding_is_accepted_by_control_read
- soul_stored_plan_with_inadmissible_binding_passes_every_read_check
- soul_terminal_completion_inside_resume_returns_an_error_serve_propagates
- soul_real_crash_window_leaves_a_live_record_that_the_archive_filter_cannot_see
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
additive_slot_does_not_break_stored_records: encodes a DeploymentTransaction
the way a binary predating post_fencing_abort (key 34) would have -- a
34-element tuple instead of 35 -- and asserts ControlSnapshot::read accepts
it. This is the exact shape that made e8d8747 necessary; the read path now
tolerates it without a schema lift.
terminal_write_archives_in_the_same_primitive_and_moves_its_command: drives
replace_transaction to a terminal write and asserts the transaction (and,
being the command's only transaction, the command too) leaves control.cc
and lands in history.cc with identical semantic content.
archive_resume_completes_a_crash_between_append_and_delete: fixtures the
half-finished state (history already has the record, control.cc still
does too) and asserts one more call finishes the delete without
duplicating the history entry.
read_history_for_command_is_lenient_about_unreadable_records: a corrupted
transaction payload and a foreign-schema transaction in history.cc are
both reported as unreadable without blocking the read of the command
actually asked about.
98 pre-existing tests plus these 4 pass: 102 passed; 0 failed.
Once replace_transaction archives a terminal transaction, submit's
--wait loop and status can no longer find its command in control.cc.
Both now fall back to a lenient scan of history.cc when the command is
absent from the live snapshot: read_history_for_command decodes what it
can and reports anything it cannot as an UnreadableHistoryRecord rather
than failing the whole read, because a corrupted or foreign archived
record is display-only and must never block an operator from seeing
the outcome of everything that did archive cleanly.
decode_record, read_transaction_record and ControlSnapshot's strict
path now name type, key and schema_id in their error context, which is
what a crashloop's journald line needs to be diagnosable.
Unfiltered `status` (no command id) still lists live commands only;
history is consulted only for a named command that is no longer live.
Widening the unfiltered listing to walk all of history.cc as well is
left for a follow-up if an operator asks for it.
replace_transaction is already the one commit primitive every phase
transition, same-phase persist, and abort completion funnels through.
When next.is_terminal() it now appends the record (and, if it was the
last resident transaction of its command, the command too) to a new
sibling store history.cc and deletes it from control.cc, instead of
writing it back in place. The append and the delete are each their own
CAS (insert_entry_if_absent / delete_batch_if_unchanged), so a crash
between them leaves the record resident-but-archived, which is not a
fault: resume_one_transaction now routes any transaction already
terminal in control.cc through the same primitive before scheduling
anything else, and the append is a no-op the second time.
No new store, flag, or unit-file change: history_store_path derives
history.cc from the existing state-store path (sibling file, same
directory).
Per R11 (rulings-wave-1.md): archive rather than delete-with-no-record,
so idunn status/up --wait keep an observable outcome after a
transaction leaves the live set. The lenient reader for those two CLI
paths, and the tests, are the next commits in this pass.
Remove validate_durable_authority, the v2->v3 transaction schema lift
(migrate_transactions_to_current_schema, the v2 branch of
read_transaction_record, DEPLOYMENT_TRANSACTION_SCHEMA_V2), the
byte-exact re-encode check in decode_record, and the read-time
binding.admit() re-admission in DeploymentPlan::parsed_inputs.
Verified on the live host: zero v2 envelopes exist anywhere, 511 v3,
and the only store any live unit names is
/var/lib/gamecult/idunn/control.cc, so the v2 lift added in e8d8747 is
now dead weight rather than a compatibility path. Byte-exactness only
ever caught e8d8747 itself and protects nothing else: control.cc is
unsigned and written only by Idunn under flock. Read-time admission was
a second admission authority racing freeze-time compile_deployment_plan.
Per rulings R4/R11/R12 (rulings-wave-1.md): this is deletion-only, no
replacement added yet. decode_record now names the offending record in
its error context instead of asserting canonical re-encode.
Odin's stored correlation outlives the incarnation that wrote it, so a target
being replaced -- Odin above all, which writes its own -- routinely finds one
describing the Expected it is superseding. Idunn treated that as a
contradiction and failed the deployment, and only the superseded process could
have refreshed it, so the deployment that would have fixed the target was the
one thing the stale record prevented.
It is now read as absence. That says nothing about trust: the record is
unauthenticated at that point and is used only to decide it is about something
else, which is the same conclusion an attacker gets for free by publishing
nothing. Everything after it is fully authenticated.
Two more copies of the same "first bootstrap" assumption are aligned with the
rule that replaced it. Whether Odin may be observed directly is decided once,
where the question is whether any Odin can report; the presence authentication
and the warming refresh were re-deciding it with the narrower "has an Odin ever
been admitted" test and contradicting that caller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Continuity restarts a release; it cannot repair one. When the admitted release
will not start, rescheduling it forever keeps the target permanently occupied
-- and a target with a live transaction accepts no deployment, so the one
action that could fix it is exactly the one that is locked out. Odin sat in
that state through 196 restarts.
It now stops after three attempts and says so, leaving the target free. The
count is kept against the admitted generation id, so a restart that succeeds
admits a new generation and starts the count over rather than carrying old
failures forward.
The yield to continuity is conditioned on the same count. Handing a deployment
back to a continuity that has already given up is not caution, it is closing
the last door: there is nothing left to yield to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Warming presence reaches Idunn as Odin's signed topology correlation, with one
exception for Odin's own first bootstrap. That exception was scoped too
narrowly: it asked whether an Odin had ever been admitted, when the question
that matters is whether an Odin can currently report.
A continuity restart of Odin cannot be observed through Odin -- the incumbent
that would do the observing is the process being restarted. Neither can a
deployment made while the admitted Odin has no usable correlation. Both sat
waiting for evidence that could not arrive, and continuity restarted the
target into that wait every few seconds.
The rule is now the one that justified the bootstrap case in the first place:
direct observation exactly when "observed through Odin" is unavailable, and
only ever for this one target. Every other target still waits for Odin, which
is what makes Odin the root of the chain.
Also adds the projection-presence check continuity uses to tell a projection
that still describes a dead incarnation from one already demoted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Continuity's projection repair required a fresh workload observation, so it
could only run while the target was healthy. But a demoted projection is one
of the reasons a target stops being healthy: aborting a deployment restores
the incumbent's Expected without its activation, and a target that cannot see
its own activation cannot publish presence, so it exits -- and no observation
is ever available to authorise the repair. Odin crashlooped exactly this way
after a failed redeploy, with continuity restarting it into the same wall
every few seconds and no path out.
Nothing in the repair is inferred from the live process. Expected, activation
and lease all come from the admitted generation, and the workload observation
used is the one Idunn recorded when it admitted them, so this republishes what
Idunn already decided rather than ratifying whatever happens to be running.
It now runs before anything is asked of the workload.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Idunn refuses an Odin publisher sequence at or below the highest it has seen
for a target, which is what stops an old signed correlation being replayed.
The watermark counted every transaction, including ones that failed.
A failed transaction was never built upon: the deployment was abandoned and
its projection withdrawn. Carrying its readings forward only raises the bar
for the next attempt, and with Odin restarting a withdrawn correlation's count
that made a target which had failed once into a target that could never deploy
again. Odin does not reuse a sequence, so forgetting the readings of an
attempt that came to nothing weakens nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five more preprovisioning requirements, each of which cost a candidate that
started, ran, and then died on a file it could not open: bundle traversal,
the setgid runtime root the write-lease hardening assumes, the lease record's
group, a default ACL on the published projection, and the route directory's
ownership left over from unprivileged Idunn.
Also states the two shapes that made every one of those faults arrive in
disguise: a store's .lock is opened whenever the store is, and CultLib reports
an unreadable file as an empty store rather than an error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultCache opens the `.lock` sibling whenever it opens a store, so a 0640 lock
denies a workload's read exactly as a 0640 store does -- and the lock is
created fresh under Idunn's UMask=027 on every publish, so setting the store's
mode alone was not enough. Odin's candidate got past the write lease and then
died on the projection for this reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding `post_fencing_abort` was a schema change, not a new optional field.
The control store refuses noncanonical bytes -- a record must re-encode to
exactly what is stored, which is what makes tampering visible -- so every
transaction written before the field existed re-encodes with one extra key and
reads as tampered. Deploying the previous commit took the daemon down in a
crashloop on its own control store; it was rolled back within minutes.
A v2 record is now decoded by key, which tolerates the absent one, and lifted
with no post-fencing abort, because a transaction written before the field
existed cannot have had one. Byte-exactness cannot apply across a version
boundary -- the bytes are a different schema by definition -- so the lift
leans on the full semantic validate() instead, and everything downstream sees
only the current shape. A startup pass then rewrites stale records as v3 by
compare-exchange, so the store converges rather than keeping two shapes for as
long as retention holds the oldest terminal transaction.
Verified against the live control store rather than a fixture: 41 real v2
records migrated on start, and the post-fence abort then released a
transaction that had been wedged in Leasing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A transaction past Fencing had no terminal path. begin_pre_fencing_abort is
gated on phase < Fencing, so every later error went to record_resumable_error
and retried forever. That is right while a candidate can still recover, and a
trap once it cannot: Idunn launches candidates as transient units with
Restart=no, so a failed one never comes back, and the transaction holds its
target with every later command queued behind it. There was no operator verb
to clear one and no server-side expiry.
So a candidate that is permanently stopped -- unit failed, or gone from
systemd entirely -- now aborts. The withdrawal runs in the one order that
never leaves two writers: route, then write lease, then the process, then the
projection. Reconciling the projection restores the incumbent's admitted
Expected where there is one, which is what lets continuity bring the incumbent
back after fencing stopped it; where there is none, the failed Expected is
withdrawn.
The transaction is released only once every one of those is durably complete,
which is what the cleanup evidence is for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reachability check only accepted o+x, but the other correct shape for a
runtime root is 0750 group-owned by the state group -- and that is the shape
the write-lease hardening already assumes, since it derives the record's group
from this directory. Checking only o+x would have refused the configuration
that makes the lease readable.
The workload's uid is allocated by DynamicUser and owns nothing, so traversal
can only come from the state group or from other. Both are now accepted, and
a directory group-owned by some *other* group still is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first deployment of any routed target could never succeed, and wedged
itself permanently on the way down.
The route preflight bind-mounts its candidate over config_path to run
`nginx -t` in a private mount namespace. When that path does not exist yet,
systemd materializes the mount point and leaves an empty file on the host.
Read as content, that made the baseline "change" during validation, failing
the transaction -- and every retry then saw a zero-length fragment where the
transaction expected none and reported an unadmitted incumbent, which no
amount of retrying could clear.
A fragment with no bytes grants no route, so it is now read as absence. That
makes the preflight idempotent whether or not systemd materialized the mount
point, and lets a first deployment retry cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The topology store is Idunn's published surface -- every managed target reads
it to verify its own Expected incarnation against the Idunn anchor -- but
Idunn runs with UMask=027, so each publish left it 0640 root:root and a
DynamicUser workload got EACCES. Odin's candidate died on exactly that.
Fixing the mode by hand does not hold, because every publish writes a new
file. Integrity here comes from the signatures over the records, not from the
mode, so the published copy is now explicitly readable after each write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CultLib's backing store reports an unreadable file as an empty store, so a
workload that cannot traverse to its runtime bundle sees zero records rather
than a permission error. Odin reported "runtime authority store must contain
exactly one record" and Heimdall simply concluded it held no write lease and
warmed forever. Neither names the actual fault, and both cost real time.
The cause is that `ReadOnlyPaths=` binds the leaf into the unit's namespace
but grants no traversal above it, so a runtime root under a 0700 parent is
unreachable to a DynamicUser. Idunn now checks the whole chain when it writes
the bundle and says exactly which directory is closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five things that had never been exercised failed on the way through Odin, and
each is now a test: the unprojectable RUDP route, step environment not being
ambient, build containers having no machine identity, the published projection
sitting inside Idunn's private root, and a signing identity living in the
state root the workload can write.
Also documents that `idunn validate` proves nothing about deployability
without a binding -- admit is where every cross-document rule lives -- and
repairs apostrophes an earlier pass mangled into ".s".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A target whose tests enrol a service identity could not be built at all.
CultLib's Linux protector binds the seed to /etc/machine-id, the build image
carries none, and the runner is --read-only, so Odin's suite failed every
step with "Linux machine-id is unavailable".
Mounting the host's machine-id would let a build container protect a seed
that unwraps on the host, which is exactly the property the binding exists to
deny. So each frozen workspace now gets its own identity, derived from the
workspace path and mounted read-only: stable for the life of a build, and
useless anywhere else including this host.
Also moves Idunn's topology store to /var/lib/gamecult/idunn-projection. It
is Idunn's published surface -- every managed target must read it to verify
its own Expected incarnation -- and it was sitting inside Idunn's private
0750 root where no workload could traverse to it. Odin's own daemon already
named that directory in its tests.
Odin's three roots become writable for the same reason Heimdall's are: a
target Idunn cannot write is a target it cannot deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A routed RUDP target was admissible but unprojectable. `admit` has an
explicit arm for (route_required, Rudp, nginx-stream-udp) against an
`rudp://` stable endpoint, and the nginx driver already writes
`listen ... udp reuseport` for it -- but `endpoint_scheme` lumped Rudp in
with Private and refused to name a scheme, so `expected_projection` bailed
before any routed RUDP candidate could be described.
Odin is the only routed RUDP target and it is the bootstrap target: nothing
else in the estate can warm until Odin is admitted, and Odin could not be.
Private keeps bailing, which is correct -- `route_required` is false for it,
so a private target never reaches this function.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploying Heimdall proved the migration order backwards. A candidate's
warming presence -- the evidence Idunn requires before fencing an incumbent
-- is observed through Odin's authenticated runtime topology correlation.
advance_warming has exactly one exception: target == "odin" while no Odin
is admitted may warm on directly observed presence. Every other target
waits for Odin to publish.
So with the previous generation's Odin still running, a Heimdall candidate
starts, serves on its assigned bind, publishes health, and warms forever.
It never fails and never promotes. That is not a defect; it is the
observation chain being rooted somewhere I had not looked. It also explains
why Odin has the only recipe in the estate that was ever kept current.
The document said "migrate the authority's own dependencies last". Odin is
not a dependency to defer, it is the root, and the entry now says so.
The wait also recorded no reason, so a candidate could sit in Warming
indefinitely with nothing to read -- the fourth coarse-signal case today.
It now records what it is waiting for and why the wait cannot resolve
without Odin.
Noted alongside: health published over RUDP to the previous generation's
--rudp-health-bind goes nowhere, since the current serve has no such flag.
That produces a steady stream of timeouts in the candidate's journal and is
not the reason promotion stalls.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The launch-contract assertion carried the same empty-User clause I removed
from observe_unit an hour ago -- a second copy in a second checker, failing
for the same reason: systemd reports the identity it allocated for a
DynamicUser unit, so User is never empty for a running workload.
It also bundled thirty conditions behind one message. "The launch contract
differs from the admitted launch" leaves an operator bisecting by hand
against a workload that has already exited, and that is exactly how the
last two rounds went. Each clause now carries a name and the failure says
which one.
That is the third time today a check knew precisely what was wrong and
reported something too coarse to act on: the sealing gate recorded a brake
reason status never printed, the isolation contract hid one property among
twenty-one, and this hid one among thirty. A control plane that cannot say
which invariant it is enforcing costs more than the invariant is worth.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The credential check required the delivered file to be owned by the
workload's uid and gid with mode exactly 0400. systemd does not deliver
credentials that way and never has: it writes them root:root, closed to
group and world, and grants the workload's dynamic uid read access through
a POSIX ACL. Observed directly on systemd 257 rather than inferred:
-r--r-----+ 1 0 0
user::r-- user:64459:r-- group::--- mask::r-- other::---
64459 was the running workload's uid, so the invariant the check wanted --
readable only by the workload identity -- holds exactly. It just holds
through an ACL rather than through ownership, so the assertion failed for
every credential Idunn has ever delivered.
Now asserts what the file mode can actually carry: root-owned, unaliased,
no group or world access. The workload-only grant is the ACL's, which is
systemd's to enforce rather than ours to re-derive; saying so plainly in
the comment beats an assertion that looks strict and cannot pass.
Second unsatisfiable contract in this observer today, after the empty-User
assertion. Both asserted a shape systemd does not produce, and both failed
after every other property matched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workload isolation contract required `User` to be empty. systemd
populates that property with the identity it allocates for a DynamicUser
unit -- the unit name, or a generated `_du…` when the unit name is too long
-- so it is never empty for a running workload. Every workload Idunn starts
failed the check it had just satisfied in every other respect, reported as
"systemd workload isolation properties differ from the Idunn contract"
with no indication of which property differed.
Confirmed against systemd directly rather than reasoned about: a transient
unit with DynamicUser=yes and nothing else reports
User=idunn-user-probe-2869674. All twenty other asserted properties matched
exactly.
The clause could not have distinguished an allocated identity from a pinned
one anyway, since both surface the same way. The invariant it reached for --
the workload does not run as a persistent named account -- is carried by
`dynamic_user` and by Idunn never emitting a User property at all. The
observed value is still recorded, as evidence of which identity ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first launch with a full environment failed EACCES on
/srv/heimdall/keys/signing-key.pem: the workload runs under DynamicUser and
cannot open a heimdall:heimdall 0600 file. The wrong fix is a supplementary
group -- widening access to resolve a permissions error.
Idunn already has the right one. A [workload.secret_files] entry is loaded
as a systemd credential, and the driver sets an env var named after the
entry whose value is the credential path. Naming the entry after the
variable the service already reads wires it up with no code change, and
setting that name in [workload.environment] too is refused as a collision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime presence identity is handed to the workload as an inherited
file descriptor, so Idunn requires the source file to be root-owned, mode
0400, and singly linked. Enrolment leaves it 0600, which fails.
The nlink check is the interesting one: a second hard link would be a
second path to the signing key, and a descriptor that cannot be reopened
is the whole point of passing it this way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProtectSystem=full makes /etc read-only, so Idunn could not create Heimdall's
runtime bundle under /etc/gamecult/heimdall/runtime. A target whose roots are
not in ReadWritePaths is a target Idunn cannot deploy. Every migration adds
exactly three paths -- runtime root, state root, release root -- which keeps
the write set enumerated and is the honest blast radius of running as root.
Also records the per-target preprovisioning Idunn requires rather than
creates, each discovered by a failed deployment: the state root must be
root:<state_group> mode 2770 so the workload's dynamic user inherits group
access through setgid; the runner cache must be 0700 owned by the runner's
container uid, not the source identity, with a root-owned parent (/srv/build
is idunn:idunn and fails); the frozen source stage must be root-owned,
because source is frozen as idunn and then copied into a root-owned
actuation stage; and the binding's docker network must exist.
None of these are Idunn being fussy. Each is a check that a binding cannot
widen access by naming a path someone else can write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The RUDP connection id 0x1d0d0001 appeared nowhere in this repository. It lived
only in Odin's publisher implementation, which meant the constant defining how
to reach Idunn was owned by a consumer.
The three constants are now stated here: connection id, signed statement schema,
and the unsigned diagnostic schema. The publishing transport moves to CultLib as
a generic module that takes them as a contract, so consumers declare them
against this document rather than each carrying their own copy of the transport.
Recorded a discrepancy while writing it down: the wire value for the unsigned
schema is idunn.daemon_health, with no version suffix, while prose in guide.md
and earlier in this file says idunn.daemon_health.v1. The table gives the wire
value. Reconciling the prose is a separate question about whether the wire
should carry the suffix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CultLib forks do not disagree. cultcache-ts reads Idunn.s live control
store -- 36 records -- and ordinary CultCache stores share one framing
across runtimes. Only service identity private stores use the bare
single-envelope container, deliberately, because they hold a key rather
than a record set.
The section now says which shape is which and names the failure it nearly
caused: a consumer that hand-decodes the private-store framing and applies
it to expected.cc and the write lease fails on first deployment while its
tests pass, because hand-built fixtures agree with the hand-built reader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enrolling Heimdall's signer surfaced that cultcache-ts cannot read a
single-file store written by cultcache-rs at all: pullAll() throws
invalid_type, expected object received array. Verified against the real
idunn-provision enrolment on yggdrasil.
Idunn writes every store a target must read -- presence identity, process
write lease, expected incarnation -- with the Rust client. Rust targets
consume them natively; TypeScript targets cannot, and must decode the
layout by hand. That makes this a per-runtime blocker rather than a
Heimdall quirk, so it belongs in the migration route.
Worth separating what broke from what held: the record schemas are
identical across runtimes, including the identity's protector binding. The
contracts are fine. The store framing drifted.
Which framing is canonical is a CultLib decision and is not taken here; the
C# implementation is the stated reference and is the tiebreaker.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shared prerequisite for every routed target, done on yggdrasil. Debian
ships ngx_stream_module as a separate dynamic package and it was absent:
nginx -V showed --with-stream=dynamic with no .so. Installed
libnginx-mod-stream at 1.26.3-3+deb13u7, matching the running nginx
exactly so nothing else moved, added a top-level stream block including
/etc/nginx/idunn-stream-routes/*.conf, and created that directory
root:idunn 0775. nginx -t passed before the reload; gamecult.org and
heimdall.gamecult.org/healthz both returned 200 after it. Backup at
/root/nginx.conf.bak-20260906-stream.
Worth recording how the module check nearly went wrong: the first
`nginx -V` ran without sudo, hit "command not found" because the binary is
not on the unprivileged PATH, and reported NO STREAM MODULE. That is the
estate's own trap -- a failed read presented as a negative result -- caught
only because the answer looked too convenient.
The larger correction: the unit I wrote yesterday ran User=idunn, and this
generation cannot work that way. There is no sudo anywhere in Idunn's
source; it calls systemctl, systemd-run, nginx and docker directly, so a
confined user actuates nothing. deployment-authority.md states the intended
model plainly -- freeze source as idunn, copy into a root-owned actuation
stage, the privileged driver never opens Git -- and --source-uid/--source-gid
are validated as non-zero, which only makes sense for a privileged daemon
dropping down for source work.
So the unit now runs as root with --source-uid 986 --source-gid 978, keeps
ProtectSystem=full and re-opens each actuated path individually, and adds
RuntimeDirectory=idunn for the route driver's preflight staging. The
comment block explains the tradeoff rather than leaving the next reader to
infer it: privilege separation is internal, and what makes it tolerable is
that Idunn accepts no imperative input.
That is a genuine increase in blast radius over a confined process, and
migration.md says so instead of selling it. The counter-argument is that
the old shape was confined in name only -- a sandboxed process whose sudo
grant pointed at an unaudited root shell with eleven hardcoded targets.
systemd-analyze verify passes on the rewritten unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
StreamPixels is changing direction: 2D animation could not express what
the spec required, so it is moving to a 3D-rendered overlay, and whether
that runs in WebGL or needs dedicated rendering infrastructure is open.
The client funding it cannot absorb rendering overhead, which is what makes
the WebGL question load-bearing rather than a preference.
Its deployment shape is downstream of that answer -- a WebGL overlay keeps
roughly the current two-unit shape, rendering infrastructure is a new
target class with hardware requirements this host has never expressed. A
binding written now would pin the shape being replaced.
The survey of its current defects stays as a record but is marked
superseded, with an explicit instruction not to act on it. The tsx-at-
runtime path is exactly the kind of thing a passing agent would tidy, and
tidying it would be work spent on something being replaced.
heimdall leads the remaining three instead. Its prerequisite is deeper
than repixelizer's but far better understood: a vendor/CultLib gitlink,
following Ghostlight's existing vendor/eve pattern, replacing a
cultlib_commit hardcoded in a shell manifest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I called these three "the simple ones" and "compose-shaped". Both were
wrong. None of them is compose -- the legacy actuator restarts all three
with `docker compose -f /srv/compose/yggdrasil-apps.yaml`, and that file
does not exist; /srv/compose holds only odin, voidbot and voidbot-retrieval.
The path is guarded, so every restart of these three has been failing at
the guard. They are running only because nothing has asked them to restart,
which is the difference between a supervised service and one that happens
to be up.
Each needs a change in its own repository before a recipe means anything,
and the changes are different shapes, so this is not one piece of work:
streampixels has real build scripts (tsc, next build) and no external
dependencies, but its service unit runs TypeScript through tsx from a
checkout, and it is two units under one target name. A recipe declares one
[service]; the two have different environments, ports, and an ordering
dependency. It was one target only because a shell line restarted both.
repixelizer has the cleanest dependencies of the three -- no CultLib, no
submodules -- but runs a venv interpreter against a checkout, so there is
no artifact to seal and no revision observable from the process.
heimdall is the most release-shaped and needs the deepest change: it takes
CultLib as file:../CultLib, a sibling checkout outside the repository. The
current manifest works around that by cloning CultLib separately with
cultlib_commit hardcoded in shell -- exactly the untyped authority recipes
exist to replace. It needs vendor/CultLib as a gitlink, like Ghostlight's
vendor/eve.
Also corrects the advice to copy CodexConnector's recipe. That recipe is
stale: it declares required_gitlinks for vendor/cultcache-rs and
vendor/cultnet-rs, and the repo has no .gitmodules, no vendor/, and takes
CultLib as a pinned cargo git dep instead. Recipes drift from their repos
silently because nothing re-checks them; run `idunn validate` in each
target's CI so drift fails at the repo rather than on the host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
