Compare commits

..
Author SHA1 Message Date
Meta Vens
64d63cb417 Merge steward orders 1 (r3): an order is one systemd job; refusal reads the named unit only
Soul cut-steward-orders-1.s5 holds at 77e2295. Not installed: steward-orders-2 installs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 23:57:39 +01:00
Meta Vens
77e2295e97 Steward orders: refuse Upholds and ConsistsOf units, pin the rule's gaps, README names the residuals
A unit with Upholds= is refused for start and restart (systemd would start the
upheld unit in a full transaction). A wrapper with ConsistsOf= is refused for
stop, restart and reload with a typed Refused::Wrapper naming the first member,
so it cannot report ran while touching nothing. Tests pin empty BoundBy, a
single show per refusal, no show for a refused name and each posted reason.
README: the one-job rule's residual full transactions. Removes the unused
BTreeMap import and the never_loop in the host actuator test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 23:32:30 +01:00
Meta Vens
fb1e7b5dd9 Steward orders: README states the one-job rule; pin the end-to-end argv
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 22:45:08 +01:00
Meta Vens
8ebe0208a6 Steward orders: pin the one-job rule on the real unit set and a real user manager
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 22:41:56 +01:00
Meta Vens
a859914b0b Steward orders: run every unit verb as one job and check only the named unit
Delete the transaction model (Edge, EDGES, action_target, Transaction, the
early-boot rule). Every unit verb runs systemctl <verb>
--job-mode=ignore-requirements -- <unit>; refusal_for reads one systemctl show
of the ordered unit: its names, the host-ending names, what fires when it
ends, and who is bound to it. Targets are no longer orderable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 22:40:58 +01:00
Meta Vens
4894ad20df README: an order's transaction follows emergency actions and OnFailure/OnSuccess; stopped boot units are refused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 21:52:41 +01:00
Meta Vens
879b76a842 Steward orders: pin every emergency action spelling to its target
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 21:50:23 +01:00
Meta Vens
631fb7836b Steward orders: pin the new show properties; the unknown-action probe counts runs instead of expecting none
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 21:29:13 +01:00
Meta Vens
d29bf93be6 Steward orders: refuse a start or stop whose unit ends by emergency action or isolates; boot and recovery units are never started from stopped
refusal_for now reads SuccessAction, FailureAction, StartLimitAction and
JobTimeoutAction (each mapped to the target it runs, so reboot, poweroff,
halt, kexec, soft-reboot and exit, with -force and -immediate) and OnFailure
and OnSuccess as units a touched unit fires, followed through what they pull
in and what those fire. An action it does not know fails the order closed.
A stopped DefaultDependencies=no unit in the transaction is refused (Via::
EarlyBoot): rescue.service ends by isolating default.target, which no edge
describes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 21:26:29 +01:00
Meta Vens
d9710c988a Steward orders: the ledger reads an order only under its own schema too
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 06:35:43 +01:00
Meta Vens
caacbb4bba Steward orders: the end-to-end test waits 10 s for its posts
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 06:35:31 +01:00
Meta Vens
9653d270e5 Steward orders: pin the ledger schema guards, invocation ids and post backoff; tests fail fast instead of waiting
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 06:16:04 +01:00
Meta Vens
8a1beb5c87 README: an order is gated by the steward bound at execution; the socket is the steward's; root reads
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 05:35:30 +01:00
Meta Vens
7e15bf1a7b Steward orders: the Synapse deadline test fails instead of hanging when the deadline is gone
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 05:33:23 +01:00
Meta Vens
572f9e6aa4 Steward orders: the end-to-end test keeps serve's umask until it has read control.cc
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 05:31:34 +01:00
Meta Vens
56edad95c1 Steward orders: one steward gate for fresh and recovered orders; the orders unit leaks no group or umask; root reads; kernel-attested issuer; end-to-end serve test
execute refuses, with a record and a post, an order whose principal or issuing
uid is not the steward bound now, before grammar, scope and brake; the brake
is asked for the bound principal only. idunn-orders.service drops Group= and
UMask=: bind_socket gives the socket to the steward uid at 0600 and serve sets
Idunn's own umask, so a deploy order writes control.cc 0640. Root may show and
list, never order. The issuer unit is the first unit under system.slice, never
a name the peer chose below it. Synapse waits for a reply within one deadline.
due_posts takes the deploy-state reader so a finished deploy's post is tested.
Tests: the gate both ways, root, per-order posts, deploy result, intake and
Synapse timeouts, parse_serve, user_uid, socket and store modes, and the whole
service as root with a fake systemctl, systemd-run and homeserver.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 05:03:17 +01:00
Meta Vens
ca439a29ff Pin the systemctl show property list and unquoted backslashes
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 04:17:12 +01:00
Meta Vens
27ca875194 Steward orders tests: clear four clippy hits in the module
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 03:53:41 +01:00
Meta Vens
be3b95bc21 Import BTreeMap for the transaction facts cache
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 03:50:20 +01:00
Meta Vens
cea17c06a7 README: a unit order is refused by the systemd transaction; a package order installs only what it names
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 03:49:36 +01:00
Meta Vens
ec6222e2a1 Steward unit orders: refuse by the systemd transaction, install exactly the named packages
An order is refused when the closure systemd would build for it reaches an
always-refused unit: a start pulls Requires/Requisite/BindsTo/Wants/Upholds and
stops Conflicts/ConflictedBy, a stop takes down RequiredBy/BoundBy/RequisiteOf/
ConsistsOf/PropagatesStopTo/StopPropagatedFrom, a restart does both. A refused
unit already running is left alone by a start. Names are unquoted from
systemctl show; a unit systemd cannot show fails the order.

Package installs run apt-get --no-remove with ?exact-name(<name>) per package
and a final dpkg -s step that fails unless every named package is installed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 03:49:26 +01:00
Meta Vens
7d5cd85728 Reword a test comment so the no-brake-release tripwire reads clean
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 02:33:12 +01:00
Meta Vens
1a3634e046 Document idunn order and add the idunn-orders unit
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 02:33:02 +01:00
Meta Vens
e50a721dbc Steward orders: typed package, unit and deploy orders from one steward, in a sibling root unit
idunn orders serve takes orders over a root-only socket from the steward uid, records
order/start/outcome/post in its own orders.cc, and posts acceptance and result to
#operator as a retried projection. Unit orders may act on any unit except the compiled
always-refused set (idunn-*, mind-self@*, mind-persona@*), matched on a unit's Id and every
name, and for stop and restart on the units systemd stops with it. A refusal names a field
and a code, never the value. A started order is never run twice. Deploy orders commit
through the same primitive as idunn up.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 02:33:01 +01:00
Meta Vens
024f2d7a09 Workload: incarnation units are collected when they end, so a failed candidate no longer stays loaded and marks the host degraded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:54:26 +01:00
Meta Vens
8400d0712f Merge idunn/runner-failure-evidence: a failed runner keeps its evidence
Cut idunn-deploy-provenance:cut_spec:cut-runner-failure-evidence.r1,
reports h1-h3, Soul s1-s3. The docker runner kept only stderr, so a step
reporting on stdout failed with an empty message (Vili's node --test). A
failed step's name and bounded stdout/stderr tails go to a root-only typed
store read by idunn status; errors and the journal carry no output; one
allowlist escape at the status sink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 08:32:18 +01:00
Meta Vens
1518cfb13c drivers test: read the kept evidence through show_status, the one status sink
Fixes the test build broken by 549f4be (render is private now).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 07:18:51 +01:00
Meta Vens
549f4be093 Runner evidence: one allowlist sink for status, quarantine only on a decode failure, a displaced set-aside is journaled
Status writes everything it prints from the evidence store, including the
unreadable-store line, through one function (emit): printable ASCII and tab
pass, every other code point is a \u{..} escape and every invalid UTF-8 byte
a \x.. escape. The bidi and C1 class lists are deleted.

The writer reads the store through a seam. An I/O error leaves the store in
place, keeps no evidence and names its kind; the store is set aside only when
bytes that were read still read the same on a second read and do not decode.
A set-aside that displaces an earlier one says so on the journal (last-written
time, no content). Tests pin a failed set-aside, its no-echo, the I/O seam and
the replacement.

Not yet built or run; verified in the next job.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 07:16:09 +01:00
Meta Vens
4a68df520f Runner evidence: set an undecodable store aside; escape bidi and pin C1 in status
The writer moves a store it cannot decode (corrupt, or a newer schema) to
runner-evidence.unreadable.cc, root-only, one slot replaced by the next, and
opens a fresh store; the journal names the file and none of its content. A
store that cannot be read at all is left in place and the write fails as before.
Status escapes the bidi controls (U+061C, U+200E/F, U+202A-E, U+2066-9) in
tails, subject and runner, and a test pins U+0080-U+009F.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 03:15:45 +01:00
Meta Vens
134c1fb5a2 drain_tail tests: a scripted reader pins the Interrupted guard
An interrupted read is retried with no byte lost, a persistent read error ends
the drain instead of spinning, and a non-interrupted error after data is
returned. Kills the three Interrupted-guard mutants (false, !=, true).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 03:15:45 +01:00
Meta Vens
54a75ef9b0 Runner failure regression: a stdout-only report is kept with an empty stderr
The vili failure: node --test wrote only to stdout, stderr was empty, the step
exited 1. Pin the cap at the literal 32 and the Debug impls that keep tails out
of text.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 02:36:34 +01:00
Meta Vens
7ef31ee699 Runner failure engine tests: declare readiness, read both control stores
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 02:18:06 +01:00
Meta Vens
22bd904ed9 Runner failure tests: named step, bounded tail, never echoed, private store
Drive the real DockerRunnerDriver over a fake docker program, through the
engine's Sealing path: the second step is named with its exit status, both
tails are kept in a 0600 store capped at 32, no error, control.cc or journal
line carries the output, status escapes control bytes, an unwritable store
does not stop the abort, and control.cc's record bytes equal those captured at
base 804ed75.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 02:16:03 +01:00
Meta Vens
ef82355bab Runner failure: a typed, bounded step failure kept in runner-evidence.cc
DockerRunnerDriver::docker no longer buffers whole streams or interpolates
stderr into the error. It pipes both streams through an 8 KiB tail and returns
RunnerStepFailure (subject, runner, status, tails); the engine's pre-fence
path is the one writer of runner-evidence.cc (0600, newest 32) and journals a
line that names the store and no output. idunn status --command renders the
record with control bytes escaped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-25 02:11:14 +01:00
Meta Vens
78086705a0 deploy-provenance map: runner failure evidence (body facts R1-R8)
Imagination for cut runner-failure-evidence: the docker runner discards
stdout, so a step that reports on stdout fails with an empty message; the
tail moves to a root-only typed evidence store read by idunn status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 02:04:37 +01:00
Meta Vens
804ed759c9 Merge idunn/isolation-credential-evidence: delivered credentials, not LoadCredential
Cut idunn-deploy-provenance:cut_spec:cut-cut-isolation-credential-evidence.r1,
report h1, Soul s1 (all claims hold). systemd 257 prints LoadCredential as
[unprintable]; the workload's credential directory, read from its MainPID's
root, must hold exactly the delivered names. Unblocks Vili's promotion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 16:05:26 +02:00
Meta Vens
dfdc226c25 Idunn credential rule: an unlistable credential directory is a refusal
cargo-mutants showed that treating every read_dir error as the empty set
survived. Test the non-NotFound case (the unit path is a file).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 15:14:15 +02:00
Meta Vens
ae6a1bea55 Idunn: judge workload credentials from the delivered directory, not LoadCredential
systemd 257 displays LoadCredential as [unprintable] for every unit, so the
conjunct was always false for a unit with only the presence identity and
vacuous for one with credentials. Delete the show argument, the conjunct and
its twin clause; the isolation conjunction becomes
require_systemd_isolation_contract. The credential rule is now
require_delivered_credentials: entry names of /run/credentials/<unit> in the
MainPID's root equal Idunn's launch names. load_credential stays a retired
wire slot (written empty, out of the drift list) so persisted records decode.

Not yet built or verified.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 14:54:15 +02:00
Meta Vens
def0b78378 Fixtures: systemd 257 show captures and pre-cut observation records
Real systemctl show output (Vili presence-only, Odin with one service
credential) and the two MessagePack encodings of a pre-cut observation, as
inputs for the credential-evidence cut. Nothing reads them yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 14:53:15 +02:00
Meta Vens
69eb3e6d6f deploy-provenance map: isolation credential evidence (body facts I1-I8)
Imagination for cut isolation-credential-evidence: systemd 257 prints
LoadCredential as [unprintable] for every unit, so observe_unit's comparison
never passes a target with no extra credential and proves nothing otherwise;
the rule moves to the delivered credential directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 14:51:10 +02:00
Meta Vens
112a7a5ebb Idunn unit: Vili's roots are optional so a missing target cannot stop Idunn
systemd refuses to start a unit whose ReadWritePaths names a missing path,
and none of Vili's three roots exist on Yggdrasil yet. Idunn's startup must
not depend on a managed target, so the Vili line takes the "-" prefix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:15:16 +02:00
Meta Vens
a3c6386aa4 Grant Idunn Vili's runtime, state and release roots
motion-pipeline:follow_up:idunn-unit-vili-roots: Vili's admission
(motion-pipeline:cut_spec:cut-vili-admission.r1) needs its three roots
writable under ProtectSystem=full, beside StreamPixels'. Lands before the
d3222e5 install, which restarts idunn-yggdrasil anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:14:39 +02:00
Meta Vens
d3222e5724 Merge idunn/tree-executable-members: tree artifacts declare executable members
Cut idunn-deploy-provenance:cut_spec:cut-tree-executable-members.r2, reports
h1-h3, Soul s1-s3. Members sealed 0555, the rest 0444; link containment
decided once in the digest walk (leading dotdot within depth, no absolute);
one seal step for collect and host. Real PBS CPython+torch and Blender trees
seal with all 1128 links kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 23:52:27 +02:00
Meta Vens
e8c4c69728 Seal tests: refused links whose targets resolve (absolute, escaping, after-a-name; file and directory)
Closes the live-target bypass: a judge guarded by exists() or is_dir() passed every dangling fixture.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 23:26:50 +02:00
Meta Vens
68d6f96808 Seal: one seal_artifact for both targets; link containment decided at seal by a shape rule
collect_artifact and host.rs materialize both call seal_artifact. digest_tree
judges every link with check_release_link (relative, `..` only first, climbs no
higher than the link's depth); validate_release_symlink and harden's link
judgement are gone. hash_file_exact owns the streamed-length check. A file
directly under a tree's top is pinned 0444.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 21:57:07 +02:00
Meta Vens
07616697af deploy-provenance map: link containment rule, host targets, shared seal path (r2)
Imagination r2 for cut tree-executable-members after Soul s1: link census of
the real PBS and Blender trees, why lexical normalisation does not prove
containment, the leading-dotdot-within-depth rule and its prior art, members
on host targets carried not applied, one seal step for collect and host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:54:00 +02:00
Meta Vens
fd040f09c1 Tests: an io error other than NotFound is not reported as a missing member
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 20:28:15 +02:00
Meta Vens
76cec0b540 Tests: the sealed-mode helper no longer redefines the module's existing mode_of
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 19:54:35 +02:00
Meta Vens
04ff2c8ce3 README: a directory artifact is sealed 0444 except its executable_members
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 19:53:16 +02:00
Meta Vens
022dbece5b Directory artifacts declare executable_members; the seal checks them, the receipt carries them, harden seals exactly those 0555
ArtifactOutput and ArtifactReceipt gain executable_members (the receipt field is
skipped when empty, so existing sealed release ids are unchanged).
check_artifact_modes is the one seal-time owner of mode declarations and runs
from both collect_artifact and the host materializer. harden_installed_release
drains the declared set and fails when a declared path was never sealed. Digests
stream through one hash_file_streamed helper; the encoding is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 19:53:16 +02:00
Meta Vens
332dbd0149 deploy-provenance map: tree executable members (body facts, rationale)
Imagination pass for cut tree-executable-members (session
self-2026-10-23-eureka-b): seal, digest and receipt facts, symlink rule,
live binary and install path, disk sizing for the acceptance job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 19:42:34 +02:00
Meta Vens
d74ac8da5c Launcher: presence freshness treats a stamp after the reading as age zero; pin the bound literally
Reading the clock after the cached result still left the answer to ordering; the
one place freshness is decided is the age subtraction, and monotonic time cannot
run backwards, so a result stamped after the reading is fresher than it and ages
zero. The bound test names 86400 and 86401 so raising the constant is caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 15:01:56 +02:00
Meta Vens
d111986093 Launcher: pin deadline clock sources, read presence freshness in a race-free order, bound declared durations
The test clock's wall now starts a Unix-epoch away from the monotonic clock, so a
deadline or schedule that reads the wrong source fails. serving_state reads the
cached probe before the clock (a stamp is taken before it is stored, so the later
reading is never behind it) and ages with saturating_sub; the test that pinned the
old wrong answer is replaced by a deterministic descheduled-reader test. The
lease-wait, startup and heartbeat seconds are bounded to one day at parse with a
refusal that names the flag and not the value, and every deadline is computed by
one saturating helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:23:28 +02:00
Meta Vens
06782d38f2 idunn-launch tests: the presence tripwire trips on the monotonic read, before any lock is held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:27:38 +02:00
Meta Vens
5d8f7e18c5 idunn-launch: freshness, deadlines and the probe schedule run on a monotonic clock
Clock gains monotonic_millis; the cached health result's age, the freshness
decision, the probe schedule, heartbeat and lease/startup deadlines all read it.
Wall time stays on the signed record only. A result stamped after now is not
evidence. Tests step the wall clock both ways, pin the warming-proof cap, and
bound the lease-timeout join.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:26:22 +02:00
Meta Vens
b5d13652e1 idunn-launch tests: the first Active probe is also held to the refresh cadence
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:50:09 +02:00
Meta Vens
6b1d14d159 idunn-launch tests: an Active child is probed at the refresh cadence
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:43:04 +02:00
Meta Vens
2ea0572a4c Docs: member connection limit and the cached presence answer
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:34:28 +02:00
Meta Vens
1207f21948 Real-nginx tests report how many challenges were answered
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:34:14 +02:00
Meta Vens
60b64e283d Real-nginx tests stop nginx through nginx so its workers do not hold the test's pipe
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:33:23 +02:00
Meta Vens
d6ebd328e1 Fix route limit bound (nginx allows below 65536); the dead-acceptor test kills the presence thread through the clock
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:16:44 +02:00
Meta Vens
c38bb36560 Fix route limit bound (nginx allows below 65536); the dead-acceptor test kills the presence thread through the clock
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:15:50 +02:00
Meta Vens
e5294db40c idunn-launch: a challenge answers from the lifecycle's last probe, stale or failed reads degraded; tests bound their waits and pin startup timeout and heartbeat cadence
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:12:57 +02:00
Meta Vens
36114b8b7b Idunn routes tests: the rendered member block pins its limit zone; real-nginx flood tests prove presence stays answerable
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:12:56 +02:00
Meta Vens
2218ffb8a8 Idunn routes: the member server block renders a limit_conn on a per-route zone; [route.member].max_connections defaults to 128
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 04:10:12 +02:00
Meta Vens
b379fbd700 Tests kill mutation survivors: each owned-name site is pinned alone, the member range below the route's is accepted, a challenge near the frame bound is answered; accept errors share one backoff
Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 01:45:19 +02:00
Meta Vens
9173a0e8e6 idunn-launch answers presence only: the splice, the unix child endpoint and --listener are deleted; the child binds the member bind and the probe reaches it over TCP
PresenceAcceptor serves one connection at a time within a 1 s deadline and never ends on an accept error; a dead presence thread fails the unit. Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 01:28:30 +02:00
Meta Vens
7805c0aa9a Idunn routes: an optional [route.member] renders a second nginx server block and sets GAMECULT_IDUNN_MEMBER_BIND; Idunn-owned launch names live in one list
The member candidate port is derived from the candidate port; both stable sockets get firewall rules; native targets render the same bytes as before. Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 01:18:26 +02:00
Meta Vens
e2dee7c16a idunn-launch tests: the inherited lease lock must stay held for a second, so a concurrent fork cannot mask a missing descriptor
Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-23 00:15:33 +02:00
Meta Vens
8246de7c37 idunn-launch tests: lock-release wait tolerates a concurrent fork; health probe test reads the request line, not a 64-byte read
Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 23:26:35 +02:00
Meta Vens
d36786f76c idunn-launch: credentials only under CREDENTIALS_DIRECTORY, bounded route acceptor, lease lock inherited by the child, startup probe rules pinned
Soul s1 fixes. Credential placeholders resolve files under
$CREDENTIALS_DIRECTORY only, never an environment variable. The route
acceptor has a first-byte deadline, a total challenge deadline and a
connection cap with unspawned refusal; a dead acceptor degrades presence.
The shared lease lock is a plain descriptor the child inherits, so the lock
lives as long as any writer. Tests pin a late-healthy child and a hung probe.
One commit: the spawner and lease trait signatures change together with
their fakes. Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 23:16:42 +02:00
Meta Vens
9c69d00641 Tests kill survivors: grant binding and holder custody, parse fields, credential names, env credentials, Odin endpoint, half-close splice, heartbeat pacing, SIGTERM escalation
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 20:48:29 +02:00
Meta Vens
0e09488c19 idunn-launch tests: wait_until takes FnMut
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 20:09:26 +02:00
Meta Vens
d94150d396 idunn-launch: release root is the launcher's own release; docs for foreign workloads; probe, spawner and authority tests
Not built at this commit; verified at the head by the final job.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 19:58:21 +02:00
Meta Vens
6866d75536 idunn-launch tests: lease gate, foreign lease, lease loss, stateless, route challenge, verbatim splice, closed placeholders, child exit (unbuilt)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 19:10:58 +02:00
Meta Vens
b9f27ce1e7 idunn-launch: presence launcher, lease hold, closed-set config renderer, route acceptor (unbuilt)
Does not build-verified yet; tests follow. drivers.rs: holder-side lease read, one lease-binding rule shared with grant.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 18:53:29 +02:00
Meta Vens
491459e6a0 State kind external-directory: process-owned directory, lease-gated like external-file
Not built or tested yet; verified at the end of the cut.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 18:47:20 +02:00
Meta Vens
6e2a31cf31 Body map after reincarnation cut 2: two slots, take_target, forward retry, operator-required
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 02:40:55 +02:00
Meta Vens
2f8671ac35 Reincarnation map: cut 2 installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 02:36:25 +02:00
Meta Vens
dbbdbf12fe Merge idunn/reincarnation-c2: phase-split handover, owner deadlines, forward retry, continuity yields
Cuts c2-phase-split, c2b-owner-deadlines, c2c-post-fence-forward, c2d-continuity-yields with their Soul passes; installed on Yggdrasil 2026-10-20 00:34 UTC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 02:36:24 +02:00
Meta Vens
8760a4fab3 A boot lift that errs fails closed
If the boot lift returned Err, boot logged and served with nothing held, so
every pending continuity abort ran and stopped a running candidate. Boot now
sets lift_unsettled; retry_held_lift runs the lift for each continuity abort
until one succeeds, and no abort step runs before. Boot's call moves into
Engine::boot_lift so the failure is testable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:51:57 +02:00
Meta Vens
c4edc66c48 The boot lift yields to a Deploy by the continuity's own yield test
The lift kept an abort running for any awaits_handover Deploy, so a braked
Deploy released a held record to an abort that stopped the only copy of the
body. One predicate, deploy_the_continuity_yields_to (candidate unproven, a
waiting Deploy that deploy_can_move), is now what both the deadline's yield
and the lift ask (rulings c2d-yield-only-unproven-to-movable,
c2c-lift-defers-on-observe-error).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:50:46 +02:00
Meta Vens
b993615c69 Negative tests: the lift must not yield to a braked waiting Deploy
Two tests, from Soul's h2 attacks: a running candidate at boot beside a braked
stop-then-start Deploy at its Sealing tail, and a held (unobservable)
candidate when such a Deploy appears. Both must leave the candidate unstopped.
They fail here: the lift treats any awaits_handover Deploy as a reason to keep
aborting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:47:48 +02:00
Meta Vens
b119e500ad c2d fix 2/2: a continuity yields only while unproven, and only to a Deploy that can move
serving_proven (Committing, or routing evidence recorded) ends the yield:
yields_to_a_deploy replaces retries_forward, and wait_for_owner reads it, so
a Deploy no longer waits on an owner that will keep serving. deploy_can_move
asks the deployment brake live for a Deploy at its Sealing tail, so a braked
Deploy draws no yield and the yield happens once the brake releases. The
tests of 04389e6 are repaired to reach their worlds (the failures they
showed at 1beca61 were the right ones). Rulings
c2d-yield-only-unproven-to-movable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:43:34 +02:00
Meta Vens
04389e6afc c2d fix 1/2: negative tests, no yield to a serving continuity or a braked Deploy
Fail at 1beca61 (the c2d logic is 81bfb03's): a Committing continuity past
its deadline yields to a waiting Deploy and stops serving; a stop-then-start
Deploy at its Sealing tail with the deployment brake unreleased draws the
yield. The second test releases the brake and pins that the yield then
happens and closes the continuity's operator-required incident Recovered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:38:10 +02:00
Meta Vens
1beca61ac4 c2c fix 3/3: the refusal table's recorded-withdrawal case ends at the lift
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:31:36 +02:00
Meta Vens
35d9c7fcf8 c2c fix 2/3: the lift defers on an observe error; the retry exit is worded from the driver
An abort over a candidate the boot lift could not observe is held in
process memory: resume_candidate runs no step of it and retries the lift
each pass (retry_held_lift) until observation answers Ok (lifted) or the
driver proves the candidate dead (the abort proceeds). The predicate and
write move into consider_lifting_continuity_abort, which boot and the
retry share. Ruling c2c-lift-defers-on-observe-error.

The page and the expire refusal name the manual exit from the plan's
workload binding: a unit to stop for systemd, a process to kill for a
host actuator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:30:18 +02:00
Meta Vens
11f7fdbf09 c2c fix 1/3: negative test, a lift that cannot observe must not let the abort stop the candidate
Fails at 81bfb03: an observe error leaves the abort running and its first
step stops the candidate. Also pins the lease_withdrawal==Complete conjunct
(case LeaseWithdrawalRecordedComplete) and makes the refusal table's
dead-candidate case provably dead (SwitchWorkload::bury).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:28:06 +02:00
Meta Vens
81bfb03712 route-continuity-target invariant 5: the continuity yields to a deployment awaiting the handover
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:11:10 +02:00
Meta Vens
4e66a65758 c2d stateful yield test: drop the Fencing half the seeded Deploy cannot honestly assert
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:10:18 +02:00
Meta Vens
09bd127643 c2d: stateful yield test, an Odin-shaped continuity yields its lease to a waiting Deploy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:09:28 +02:00
Meta Vens
f79deecdb7 c2d tests: the helper stops short of the deadline; each test passes it 2026-10-19 20:07:30 +02:00
Meta Vens
fd6e3e65af c2d yield table: show how row (i) ends 2026-10-19 20:06:48 +02:00
Meta Vens
a868321259 c2d yield table: name the row an error comes from
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:06:00 +02:00
Meta Vens
196e0dc884 c2d: a resolver write spends the resume backoff; the boot lift leaves a yield
resume_candidate removes the transaction's backoff when the resolver
moved its record, so a yield abort is not held behind the forward retry's
backoff (ruling c2d-yield-mechanics). return_wedged_continuity_aborts_forward
skips an abort whose target holds a Deploy awaiting the handover. Tests:
the yield table, the unit rows, the c2b waiter-still-ends row, the
backoff spend, and lift case (vii).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 20:03:48 +02:00
Meta Vens
4f68809f0b Continuity yields to a Deploy awaiting the handover (ruling continuity-yields-to-deploy)
A continuity past its deadline at or past Fencing begins its own
post-fencing abort when a Deploy on its target awaits the handover; the
waiter keeps waiting on an owner that retries forward. Adds
retries_forward and awaits_handover, both derived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 19:59:37 +02:00
Meta Vens
a7ad3de559 c2d negative: a waiting Deploy must land over a forward-retrying continuity past its deadline
Red at e529858: the waiter ends FailedBeforeFencing owner-left-to-operator.
The yield (ruling continuity-yields-to-deploy) lands in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 19:58:09 +02:00
Meta Vens
e5298584a4 Merge origin/main (bounded_detail hotfix) into idunn/reincarnation-c2
Conflict in begin_post_fencing_abort: kept c2's now_millis binding and
the hotfix's bounded_detail in place of truncate. The owner-wait note
written by wait_for_owner now goes through bounded_detail too; every
other c2/c2b/c2c detail writer already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 19:34:20 +02:00
Meta Vens
ec45906ead bounded_detail: use std char-boundary floor/ceil instead of hand loops
The hand loops hung under mutation (infinite loops); std does it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 19:07:23 +02:00
Meta Vens
8f00df079f Sealing-failure test: clear the resume backoff between ticks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:40:52 +02:00
Meta Vens
d9846e8847 Cut c2c: route-continuity invariant 5 states the continuity forward retry
A continuity past its post-fencing deadline is left to the operator and retries
forward on a capped backoff; a stateless deployment aborts to its incumbent; a
stateful deployment is left to the operator (ruling post-fence-forward-retry).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:39:33 +02:00
Meta Vens
bff308aea7 Cut c2c: expire's refusal names its reason without reading the deadline predicate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:39:32 +02:00
Meta Vens
18c5b643a7 Every persisted detail goes through bounded_detail; delete truncate
require_detail judges bytes, truncate cut chars, so a long multibyte build
error made the abort write invalid and wedged the record in Sealing.
bounded_detail keeps head and tail within 2048 bytes, drops NUL, and
never returns empty. Also bounds gate-wait notes, cancel notes, and the
continuity deferral reason at their writers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:37:38 +02:00
Meta Vens
cc4a72ad43 Cut c2c: a record left to the operator backs off to the continuity cap
ResumeBackoff stores the ceiling it was armed with and the skip check reads it.
resume_ceiling is the continuity cap (300 s) for a record that leaves_to_operator
and the short cap otherwise, for every such record: a forward-retrying
continuity, a stateful deploy, a wedged abort, a drain. A step that returns Ok
without moving a record left to the operator is backed off like a failed
attempt, so a silent candidate is challenged once per backoff, not every poll.
Nothing in TargetSupervision is touched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:37:03 +02:00
Meta Vens
81ab69046b Negative test: long multibyte sealing failure must abort and free the target
Fails at 5b421c3 (abort write refused by require_detail).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:36:55 +02:00
Meta Vens
063409489e Cut c2c: a continuity past Fencing retries forward and pages, never aborts
PhaseEnd::RetryForward: phase_end answers it for every continuity at or past
Fencing, because aborting restores the body its candidate already runs (ruling
c2c-retry-decided-by-record). The resolver leaves it to the operator
(operator-required incident, report_once) and the phase step keeps running.
leaves_to_operator covers it. DeploymentTransaction::expirable is the one
spelling the expire CLI, the spent-request retirement and the resolver's
requested flag read: a continuity past Fencing cannot be expired. Deploy keeps
abort-to-incumbent and the stateful operator-required arm.

Tests: the PhaseEnd table, the Routing deadline forward retry and its incident
episode, the stateful deadline, expiry refusal and spent-request retirement,
and a deployment still aborting to its incumbent. Abort-mechanics tests begin
their aborts directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:33:29 +02:00
Meta Vens
58f213fe15 Cut c2c: boot lift returns a wedged continuity abort to forward retry
A pre-c2c Idunn aborted a continuity at its phase deadline, which stops the
only running copy of the admitted body. At boot, a continuity abort that has
not stopped its running candidate (candidate_cleanup Pending, lease_withdrawal
not Complete, workload observed, issued lease exact on disk) is withdrawn and
the record retimed, the exact inverse of begin_post_fencing_abort. Negative
tests carry the live shapes of tx-4eaed669 (Routing, adopted lease) and
tx-039b91e9 (AwaitingReady, granted lease); the lift's refusals are pinned per
predicate input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:24:22 +02:00
Meta Vens
8230c544ac Cut c2b fix: a pre-fencing abort keeps its deadline through its last cleanup write (a Sealing waiter's abort could not finish) 2026-10-19 17:58:13 +02:00
Meta Vens
9844806eb1 Cut c2b fix: the waiter's own entry attempt drives the owner harness 2026-10-19 17:56:28 +02:00
Meta Vens
95e8076618 Cut c2b fix: the stop-first owner harness uses the host-actuator world with its workload port 2026-10-19 17:55:04 +02:00
Meta Vens
2ff57533b1 Cut c2b fix: a test world runs a host-actuator target's workload through its own port 2026-10-19 17:53:58 +02:00
Meta Vens
c6805d48a7 Cut c2b fix: restore stop_first_binding's runner split 2026-10-19 17:51:38 +02:00
Meta Vens
8b6c539a6a Cut c2b fix: the stop-first harness owner runs on a systemd unit 2026-10-19 17:51:02 +02:00
Meta Vens
a254a3e6a5 Cut c2b fix: pin the forever-waits at the waiter, the pre-fence abort deadline, the stateful incident lifecycle and the archive vanish
Negative tests (a)(b)(c) are one marked harness that ages a deadline only when
one exists and ends on the waiter's own record, so at the base they fail at the
waiter assertion (still waiting), not at setup. (c) now runs the Odin gate-wait
it names: a stop-then-start continuity on a systemd unit observes and waits,
where the host-actuator step of a world with no actuator errs at once.

New: a stop-first owner's wedged pre-fence abort pages and ends its waiter; a
waiter's own wedged pre-fence abort pages beside a standing owner; a stateful
deploy's incident is silent under an early expiry, one per episode, and closes
while the record goes on; an unparsable drain plan is bounded; a transaction
gone from both live set and history stays a fault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:49:14 +02:00
Meta Vens
21849eb097 Cut c2b fix: a drain whose stored plan no longer parses is bounded, not unbounded
deadline_for_current_activity's drain arm returned None when parsed_inputs
failed (the running binary re-parses and re-admits the frozen binding), which
left the drain with no deadline, unpaged, and disarmed every waiter behind it.
The retire time is now the promotion plus the drain when the binding parses,
and otherwise now; the frozen committing duration from it bounds the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:48:03 +02:00
Meta Vens
0bfadcfc71 Cut c2b test: pin the stateful-deploy arm of leaves_to_operator and the incident close predicate
Survivors of cargo mutants --in-diff on 9ea6c71..126d710: the OperatorRequired
clause of leaves_to_operator (now a table beside phase_end) and the close
predicate (a vanished subject closes while another transaction is still left).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:10:11 +02:00
Meta Vens
126d710de3 Cut c2b: operator-required incident, opened at the decision site and closed when spent
IncidentCondition::OperatorRequired (subject: transaction id) opens in
leave_to_operator, only for a record that leaves_to_operator, and closes as
recovered in close_spent_operator_incidents once no transaction in the
snapshot is left to the operator. Incident writes are never progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:39:42 +02:00
Meta Vens
a14055115c Cut c2b: owner deadlines; a waiter never waits on an owner Idunn will not finish
wait_for_owner disarms a waiter Warming deadline (the owner bounds the wait)
and ends the waiter through its own pre-fencing abort when the owner leaves to
the operator (owner-left-to-operator); it never aborts the owner. The resolver
times Warming for every transaction, lifts a deadline onto a record that needs
one, skips an owned waiter, and leaves an abort or drain past its deadline to
the operator (report only; the incident follows). Rulings c2b-waiter-ends-on-
operator-owner, c2b-warming-deadline, c2b-abort-and-drain-deadlines.

Negative tests recorded failing at 9ea6c71.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:37:21 +02:00
Meta Vens
86926c4e42 Cut c2b: one derivation of every deadline; Warming, aborts and drains are timed
WARMING_DEADLINE_SECONDS and PRE_FENCING_ABORT_DEADLINE_SECONDS are Idunn
constants (no PhaseDeadlines field: plan ids stay). DEADLINE_PHASES is the one
spelling of the deadline range. deadline_for_current_activity derives the
duration of a phase, an abort from its beginning, or a drain to its retire;
retime is the only non-test writer. leaves_to_operator is derived from it.
Validation asks for an activity, not a post-fencing phase. Records written by
this cut are forward-only (ruling c2b-forward-only-records).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:23:59 +02:00
Meta Vens
ffb22e368b Cut c2b: the post-fencing abort stops its candidate before revoking the lifetime-held lease
ruling c2b-abort-stops-before-revoke; fixes follow-up
ghostlight-abort-lease-before-stop. Mirrors advance_fencing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:15:47 +02:00
Meta Vens
3265ccb575 Cut c2b test: an abort stops a candidate that holds its lease before revoking it
Negative test for ghostlight-abort-lease-before-stop: the candidate holds the
lease lock until stopped. Recorded failing at this commit; the fix follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:15:07 +02:00
Meta Vens
9ea6c71921 Cut 2 test: the stop-first fixture builds without a network input
A host-native runner has no network profile, so the host-actuator recipe
drops the external input the systemd recipe fetches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:01:59 +02:00
Meta Vens
94d3b53b55 Cut 2 fixes: stop-then-start handover fixture, supervision owner skip, rebase preflight, archive-as-progress
- Add an engine-level stop-then-start fixture (route-less host-actuator plan)
  and a test that two such transactions never both enter Starting, in either
  order; it kills enter taking the CAS only at Fencing and takes_target_at
  ignoring the rollout. Discharges follow-up stop-first-handover-fixture.
- Pin supervision's skip of an owned target: a fenced generation is not
  refreshed from Odin, and a stop-first owner's stopped incumbent is not
  restarted.
- The rebase test now carries a route preflight, so keeping it dies.
- The step that ends a transaction archives it inside its write; the progress
  read now treats a terminal archived record as progress, not a fault.
- enter's doc no longer claims every phase move goes through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:00:49 +02:00
Meta Vens
5b421c3cc7 Deploy provenance target: no per-deploy operator gate, typed provenance on the transaction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:04:27 +02:00
Meta Vens
6ea0d127d8 Cut 2 test: a finished transaction retired mid-handover is not a lost race
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 12:15:19 +02:00
Meta Vens
dd8707b5a2 Cut 2: pin the candidate-port reservation and the rebase guards directly
candidate_ports_in_use is the one reading of which ports admitted, live and
draining transactions hold. Tests pin it, select_candidate_port's routed and
unrouted answers, and the three conditions under which a transaction is not
rebased.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 12:14:46 +02:00
Meta Vens
1953bf233b Cut 2: one per-target lookup, and tests that kill the survivors it exposed
ControlSnapshot::on_target is the one place a transaction is filed under a
target; target_owner, the supervision gates and the handover read it.
Tests: ownership is per target and names only a holder; a draining target is
not restarted until its cleanup ends; the admitted Expected is repaired
beside a building deployment; a live continuity alone holds back a second
mint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:57:31 +02:00
Meta Vens
a7bfbc2296 Cut 2 tests: a launching switch workload, so a real continuity restart runs beside a deployment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:22:58 +02:00
Meta Vens
278cc5c408 Cut 2, commit 4: open the gates; fold in the sustained-reset test reach
Freeze refuses only a second live Deploy of a target, so a profile command
freezes over crash-looping targets. Supervision defers to an owner, a drain
or its own live continuity, never to a deployment that has not taken the
target, and the admitted Expected repair runs beside one. Cut 1's interim
test (no continuity beside a pre-fencing deployment) goes with the state it
asserted.

Fold-in c1-sustained-reset-test-reach: the lower edge to the millisecond,
the newest attempt (not the oldest) deciding the reset, and the mid-wait
crossing recorded in continuity_backoff's doc and pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:20:30 +02:00
Meta Vens
9139d7edb8 Cut 2, commit 3: rebase before the handover, wait in Warming, supersede a continuity
settle_before_handover runs after the two abort dispatches and before any
pre-handover step. A Deploy rebases onto the admitted generation when it
differs from the one its evidence was proven against: incumbent_generation_id
is rewritten, warming, route_preflight and isolation are dropped and the
Odin cursor is raised, nothing else. In Warming it waits (a gate note, no
error, so no abort) while another transaction owns or drains the target.
A Continuity is never rebased: when its target is owned or its generation
replaced it ends itself through the ordinary pre-fencing abort.

Tests drive a real continuity restart to Complete beside a deployment and
check the deployment rebases onto it and fences it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:16:05 +02:00
Meta Vens
755306d654 Cut 2, commit 2: one phase-entry seam and one handover CAS
Engine::enter is the only writer of a move into the next phase;
transition and the Sealing tail both call it. Entering the phase where a
transaction takes its target (takes_target_at: Fencing, or Starting for
stop-then-start) goes through Engine::take_target, one CAS that expects the
transaction's own envelope, the target's admitted envelope as read (or its
absence) and every other non-terminal transaction of the target, and
writes only the transaction. Of two entrants from one view exactly one
wins; a losing deployment waits with the owner named, a losing continuity
is superseded.

exact_incumbent now says what a mismatch is: a fault, not a race.

Test helper: the brake release is per transaction (a world can hold two
deployments).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:13:38 +02:00
Meta Vens
b4bb158707 Cut 2, commit 1: split the ownership predicate and the one-transaction-per-target validation
owns_target_authority and blocks_new_target_mutation answered four
questions with one phase-blind rule. They become is_live, owns_target
(derived from the phase and takes_target_at) and drains_incumbent, each
read by the consumer that asks that question: sibling order, validation,
status, ports, freeze and the supervision gates.

Validation now holds, per target, at most one owner, one live Deploy and
one live Continuity, each refused with its own message, so a continuity
beside a pre-handover deployment is readable. Freeze and the mint gate are
still conservative (any live or draining transaction refuses), so no
intermediate commit lets two transactions coexist unguarded.

seeded_transaction mints a fresh command id when its default is taken, so a
world can hold two of one kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:11:36 +02:00
Meta Vens
d91ecba759 Cut 2 negative tests, failing at 6138329: restart beside a building deployment, profile freeze over a crash loop
a_dead_incumbent_is_restarted_alongside_a_deployment_that_is_building and
a_profile_deploy_freezes_over_a_crash_looping_target use only base API and
pin the states cut 2 (handover-phase-split) ends. They fail until the
predicate split and the open gates land.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:07:13 +02:00
Meta Vens
61383297b2 Reincarnation map: cut 2 phase-split mapped (two slots, one handover CAS)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 10:59:51 +02:00
Meta Vens
9a0a6a2ce9 Body map after reincarnation cut 1; cut 1 installed on Yggdrasil
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 10:36:21 +02:00
Meta Vens
1da150a309 Merge idunn/reincarnation cut 1: continuity never stops, nothing yields before Fencing
Soul passes s1-s3 on cut-c1-never-stop; s3 holds with test-reach follow-ups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 03:05:36 +02:00
Meta Vens
ffb42936ca Continuity tests: admitted_at that does not make the fixture a sustained run
The clock-step test anchored a target at admitted_at 0, which the sustained-first
rule rightly reads as a sustained run; the unobserved-run test logged restarts
after its generation's admission. Both now describe the cases they name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:21:22 +02:00
Meta Vens
da9d246956 Continuity: the sustained-run reset is decided before the anchor; pin the streak guards
continuity_backoff decides the sustained reset first, so a standing anchor
(a run no pass observed) cannot hide it. The running-pass end_backoff stays:
it makes the next death wait from its own discovery, not from a stale anchor.
Behavioural tests: a brief run does not reset the streak, streaks 13..u32::MAX
still mint and store saturating, the unobserved sustained run resets, a running
pass keeps the re-materialization bound, owed_steps admission bound, the v1
lift's inner schema check. The startup message names the supervision schema;
ControlSnapshot::read drops its repeat of the shared reader's key check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:19:33 +02:00
Meta Vens
f6d2943fd3 Restore LF line endings in control_plane.rs
The previous commit rewrote the file with CRLF; this one returns it to LF so
the range diff is the real change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 01:28:05 +02:00
Meta Vens
a75e31330a Continuity: the attempt after a re-materialization waits the base step
The re-materialization charges no streak, so a streak of 0 gave Due{0} and
a failed re-materialization was followed at once by a charged attempt. It
is an attempt that failed on the generation: the next one waits the base
step, so a queued deployment gets its gap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 01:24:09 +02:00
Meta Vens
07a1d41692 Continuity: reset counts the re-materialization, a running pass ends the wait; supervision v2
Fixes Soul pass 1 of cut c1-never-stop:
- the sustained-run reset reads the newest of the paging log and the
  re-materialization, so a failed re-materialization is followed by a gap
- an observed-running pass clears the backoff anchor, so a standing anchor
  cannot suppress the reset on a later death
- TargetSupervision with keys 7-9 is schema v2; the dual-shape decoder is
  deleted and v1 records are lifted and migrated once at store open
  (deletion line named at TARGET_SUPERVISION_SCHEMA_V1)
- tests: continuity mints at every streak 1..=12 with a full or short log,
  the mint stores the reset streak, the schema ids name one shape each
- ROUTE_ACTUATION_CEILING doc no longer says continuity spends restarts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 01:22:34 +02:00
Meta Vens
32ff879790 Test: a managed-host workload is never claimed lost to a reboot (kills the surviving Ok(true) mutant)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 00:24:48 +02:00
Meta Vens
b1dfe829a1 Remove an orphaned doc line left by the deleted yield control test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:55:47 +02:00
Meta Vens
2fae3af133 Test: a queued deployment may fail on its own account, never to continuity
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:55:26 +02:00
Meta Vens
69464b11ef Test: name the abort when the queued deployment does not survive the gap
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:54:47 +02:00
Meta Vens
61fc89bbd7 Tests: count only continuity minted over an incumbent; status shows streak and next attempt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:53:40 +02:00
Meta Vens
9a6b4a128c Continuity backoff: capped streak anchored when the target is found free; one uncharged re-materialization
TargetSupervision gains the attempt streak, the backoff anchor and the
re-materialization bound (keys 7-9, same schema; pre-key records decode with
them defaulted). Every charged attempt waits backoff_wait(base, streak - 1,
300 s) from the first dead-and-free pass; a sustained run resets the streak.
WorkloadPort::lost_to_host_reboot lets a reboot-lost unit be re-materialized
once per generation with no step. The streak backoff, reboot and negative
tests land with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:52:13 +02:00
Meta Vens
6eb7931a8c Continuity never stops: delete the yield, the exhaustion stop and the window-derived wait
The paging window now only pages: restarts_page opens the incident and
reports; it ends no pass and aborts no transaction. Deletes next_restart_at,
continuity_is_waiting and the five tests that pinned the stop, the yield and
the mint-anchored spacing. The streak backoff follows in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:47:04 +02:00
Meta Vens
e36cad32f0 Negative test: a pre-fencing deployment over a dead paged incumbent is never yielded
Written first against base API; expected to fail at this commit
(deployment aborted 'yielded to continuity' once the window slides).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:46:01 +02:00
Meta Vens
f079389be0 Reincarnation: continuity never stops (target r2), cut 1 never-stop mapped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:44:14 +02:00
Meta Vens
74a700817f Research: restart budgets and rollout/restart handover prior art (idunn-reincarnation forks)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:22:33 +02:00
Meta Vens
f54c776417 Reincarnation map: body facts from the 2026-10-18 reboot, model page, cut sequence
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:17:41 +02:00
Meta Vens
c2477d43d4 Reincarnation target: a reboot is not a fault, one owner per target at every instant
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 22:51:31 +02:00
Meta Vens
6667f0f9a6 Watchdog reporting postmortem
First typed Eureka campaign (idunn-watchdog): 8 cuts, 17 Soul verdicts,
55 findings; landed about 2.5x its estimate. Idunn pages the operator on
Discord through Bifrost; closure DM still owed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:25:36 +02:00
Meta Vens
cb6096512b Merge watchdog idunn-topology-lock: no tick waits on another uid's lock
Eureka idunn-watchdog cut idunn-topology-lock, attempts 1-2. Soul verdict
idunn-watchdog:verdict:cut-idunn-topology-lock.s2 holds. Topology and
incident writes share one private-lock exchange that replaces a non-private
lock by rename; a contended publish stays in phase and retries. Measured
tick with a held lock: 0.09 s (was 5.07 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 03:59:15 +02:00
Meta Vens
424300d672 A publish that loses to a held lock is retried before the fence, not aborted
resume_candidate turned any failed pre-Fencing step into a pre-fencing
abort, so a topology publish that met a held lock (the nightly backup's
shared lock, an operator) ended the deploy or continuity restart on its
first attempt. LockContended is now a typed error (drivers::LockContended,
raised by mutate and IncidentStore::exchange with the same texts as before)
and resume_candidate's one classification keeps such a transaction in its
phase with last_error set; the backoff retries it and the phase deadline
bounds the wait. Other publish errors still abort.

Tests at the resume path: a contended Starting publish stays in Starting
across ticks and lands after release with one continuity restart; a
non-contention publish error still aborts. Both builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 03:31:48 +02:00
Meta Vens
827cfad0bb Topology lock: test the lock-creation error classes and the create race
Mutation of the first cut left make_lock_private's error classification and
lost-create-race branch unkilled. Two behavioural tests at the primitive:
an uninspectable lock (name one byte past NAME_MAX) reports an inspection
error, an uncreatable one (procfs) a creation error, and neither is retried
as a lost race; eight publishers released together to create the same lock
all find it private, so none fails for having lost.

Tests only; builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 02:05:05 +02:00
Meta Vens
102c129891 Topology publishes never wait: one private-lock exchange primitive
exchange_behind_private_lock (drivers.rs) is the single write primitive for
the projection directory: one nonblocking compare-exchange behind a lock only
Idunn can hold. An absent lock is created O_EXCL 0600; a lock that is not a
regular file, ours, mode 0600 is replaced by a fresh 0600 file renamed over it
(a chmod does not revoke a descriptor another uid already holds); a private
lock is opened as is. The topology driver's mutate and the incident store both
go through it; the blocking compare_exchange_snapshot call, publish_projection_mode
(which published the lock 0644 on a false premise that readers open it) and
incident.rs secure_lock are deleted.

mutate now retries only a lost race; a held lock is Err "the topology
projection is locked by another holder" and a no-op, so no caller records a
digest, evidence or phase for a publish that did not land.

Test helpers: the lock holder is a separate flock process, and run_while_held
counts opens of the lock (inotify), so ANY wait on a held lock fails, not only
one past the 30 s ceiling. Tests reproduce the live default ACL and use
setpriv for the other-uid half.

This commit has not been built; the verify run follows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 01:28:05 +02:00
Meta Vens
a909f931ea Merge watchdog idunn-incident: continuity exhaustion is a durable incident
Eureka idunn-watchdog cut idunn-incident, attempts 1-3. Soul verdict
idunn-watchdog:verdict:cut-idunn-incident.s3 holds. Incidents live in
their own incidents.cc (0644, lock private 0600), written with non-blocking
snapshot exchanges; the tick never waits on them. Close rule: running and
no longer exhausted (operator ruling exhaustion-close-rule-op).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 01:15:08 +02:00
Meta Vens
7e4163645d Watchdog target: tick-lock-private is uid-scoped
Operator ruled tick-lock-scope = uid-scoped (target r4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 23:54:55 +02:00
Meta Vens
8d0786dcaf Watchdog map: topology lock probes C29-C40
Only the daemon writes topology.cc; Odin and Idunn read it lock-free; the
root backup's shared lock survives a private 0600 lock. A chmod does not
revoke an existing holder, so the shared primitive renames a fresh 0600
lock over a non-private one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 23:30:44 +02:00
Meta Vens
9549d00fb9 Incident lock created 0600 before CultCache opens it; fault keys per decision site; no wall-clock bounds in lock tests
lock-acl-world-openable: CultCache creates a lock with mode 0666, so under
Yggdrasil's default ACL (u::rw g::r o::r) the incident lock landed 0644 and
any uid could hold it and keep every incident from being recorded. Every
incident write (incidents.cc and incident-history.cc) now goes through
exchange(), which first creates the lock with O_CREAT mode 0600 (the create
mode is intersected with the ACL, so 0600 holds and no wider file ever
exists) and fchmods an earlier, wider lock to 0600 on the descriptor it just
opened. The test sets the live default ACL on its directory, asserts a
control file made the way CultCache does reads 0644 there, asserts both
locks are 0600 after open, close and retire, and, as root with setpriv,
that uid 65534 cannot open them.

close-fault-repeats: record_incident keys its fault by operation and subject
(open/close per target, * for the reconcile and retire passes), so one
site's Ok no longer clears another site's standing fault.

mutation-count-poisoned: a_held_incident_lock_changes_nothing_continuity_decides
failed under parallel load because run_while_held asserted the work took
under 2 s of wall clock, and the work is a whole continuity sequence. The
holder now lets go when the work returns, or after a 30 s stuck ceiling;
the test fails only if the work was freed by the ceiling. eventually() and
the control-plane retry use the same ceiling.

Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 23:13:52 +02:00
Meta Vens
4ac96e959d Route connect test: the listener never accepts, so the test cannot hang
route_connect_retries_connection_refused_until_listener_is_ready blocked in
accept() with no deadline. If the client connected to another listener on the
reused port, accept() waited forever and the whole verify job held its slot
for 39 minutes. The kernel completes a handshake from a listener's backlog,
so the connect succeeds without an accept: the listener now only outlives the
connect, waiting on a channel with a 30 s ceiling. A lost port race fails
the test instead of hanging it.

Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 23:13:52 +02:00
Meta Vens
bca3d85394 Watchdog target: no tick waits on a lock someone else can hold
Operator ruled the topology-lock stall into this campaign (target r3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 23:09:34 +02:00
Meta Vens
b69a8e81ba Secrets map: Heimdall secrets merged at ca2366c after Soul pass 7
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:49:57 +02:00
Meta Vens
ddc031a363 Merge B5 test pins: no firewall withdrawal over an incumbent, exact expiry filter, backoff growth
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:37:51 +02:00
Meta Vens
f1b45666c6 Route map: B5 merged (fcb8ba1) and installed on Yggdrasil
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:36:02 +02:00
Meta Vens
1cfbeff083 B5 pins: firewall kept for a live incumbent, whole-id request match, backoff growth
Tests only, from Soul's last B5 check:
- F2: a rollback or withdrawal back to an incumbent route deletes no
  firewall rule (S-R2 survived).
- F3: expiry requests whose ids are a strict prefix of the transaction's
  command id, or extend it, leave it untouched (S-ME1, S-ME2 survived).
- F4: the second and third failures wait two and four polls, as back_off
  writes them (S-MJ1, S-MJ2 and a clamp survived).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:30:18 +02:00
Meta Vens
fcb8ba1c29 Merge B5: every post-fencing phase ends; reload skip confined to supervision
Phase deadlines are resolved before every post-fence step; `idunn expire` requests an
expiry through the same resolver; lease adoption is recorded from Odin's Ready; an abort
withdraws any issued lease (Prepared or Granted); install/rollback/withdrawal inside a
transaction always write and reload; a first-deploy withdrawal closes the firewall rule
before reloading. The operator-required arm for an adopted-lease deploy stays until the
Q3-ii snapshot cuts (Q-S5: B5 first).

Soul passes 1-3. Merged tree verified on Yggdrasil: 360 passed, 2 ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:28:52 +02:00
Meta Vens
7793b5bb89 Merge: retire the legacy bifrost-persona-feedback deploy route
The route ran the gamecult-ops compose lane, which would start a container beside the native
unit. Operator: "delete the Bifrost lane". Grant, dispatch and the Bifrost-authority check that
served only it are removed; bash -n and visudo -cf pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:50:03 +02:00
Meta Vens
889e23629c Retire the legacy bifrost-persona-feedback deploy route
The operator ruled "delete the Bifrost lane". This route was that lane's live
trigger: /srv/odin/deploy-manifests/bifrost-persona-feedback runs
`docker compose up persona-feedback`, and the provider actually running on
Yggdrasil is the native bifrost-persona-feedback.service. Firing the route
would start a container next to it.

Following deploy/legacy's own order, the grant goes first, then the dispatch:
- sudoers: drop `deploy bifrost-persona-feedback *`.
- idunn-yggdrasil: drop the target's policy case and its dispatch token, plus
  the Bifrost release-authority check that only it used (requires_bifrost_authority
  and the `idunn validate-release-authority` call). A deploy request that claims
  a Bifrost authority is still refused. The manifest environment keeps
  IDUNN_REQUIRES_BIFROST_AUTHORITY=false, because the installed Epiphany manifest
  checks for it.
- README: record the retirement and the counts. Also record that on 2026-09-30
  the installed actuator matched this file, but the sudoers grant was not
  installed: the idunn user cannot sudo.

Rebinding persona-feedback to a current Idunn binding is a separate follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:40:21 +02:00
Meta Vens
85964d0f00 Secrets map: Heimdall batch 6 and Soul pass 6 (hold; invalid index, collation takeover, predicate normaliser)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 20:35:30 +02:00
Meta Vens
f85e74377f B5 batch 3: firewall first on a withdrawal to nothing; four test pins
`restore` with no prior membership now withdraws the stable endpoint's
firewall allow before it writes the fragment and reloads. A failing `ufw
delete` retries without touching nginx, which bounds the withdrawal reload
loop at zero until the Q3-ii cut gives the abort a deadline. A crash between
the two steps is safe: a rule already gone reads as withdrawn, and the retry
removes the fragment and reloads.

Pins from Soul pass 2:
- MB2: a redeploy over an existing incumbent route installs, rolls back and
  withdraws with a reload each (`changes_route` as `is_none()` survived).
- ME2/ME3: the resolver's command filter is tested with an id of equal
  length and the same leading bytes.
- MJ2: the backoff test asserts the wait `back_off` writes, one poll after
  the failure.
- Soul's F2 probe returns as a test: a withdrawal whose `ufw delete` keeps
  failing reloads nginx not at all over 20 retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 20:08:58 +02:00
Meta Vens
1f7c8e8f42 Route map: B5 Soul pass 2 (merge and install after batch 3 pins); P1 reload-owed window recorded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 20:07:49 +02:00
Meta Vens
3ed5f262ba Incident lock tests retry after release; a forked child can hold a released flock
A test that held incidents.cc.lock and then released it failed once in a full
parallel run: another test's forked child briefly shared the open file
description. The follow-up calls are exactly the "retried on a later pass" the
store promises, so the tests now retry for up to two seconds instead of
asserting the first call after release lands.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 19:59:41 +02:00
Meta Vens
9ad7e4209b Incident retirement test: the archive-first crash state is the designed one
A refused delete after a successful archive leaves the record in both files by
design; the test asserted the history file was absent. It now asserts the repeat
archives nothing twice.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 19:01:30 +02:00
Meta Vens
32cf1f7bf4 Incident writes never wait; exhaustion closes on health and the window; one retention predicate
Every incident write is one nonblocking try_compare_exchange_snapshot of the
whole file against the snapshot it was decided on (as drivers.rs does for the
write lease). A held lock or a lost race is an error and a no-op for the tick,
retried on a later pass; it is also what makes one open incident per
(condition, subject) real. The incident .lock is no longer published: readers
of incidents.cc take no lock, so only Idunn can hold it. History is written the
same way.

An exhaustion incident closes as recovered only when the workload is observed
running and restarts_exhausted is false (ruling exhaustion-close-rule), so a
release that survives one pass after each restart keeps one incident.

Reconcile closes only incidents whose condition ends with admission and whose
subject is not admitted; its dead accumulator is gone. IncidentRecord owns the
retention predicate that retirement and status both use. status writes to a
writer and serve's loop is serve_while, so tests reach both.

Not verified yet: this commit has not been built.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 18:56:33 +02:00
Meta Vens
aef0da34af B5: the Odin adoption test steps through sequence admission
AwaitingReady admits a new Odin sequence in one step and records Ready from
it in the next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:51:17 +02:00
Meta Vens
dcd0aa3f2b Secrets map: Heimdall batch 5 status and Soul pass 5 (hold; batch 6 in Hands)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:50:45 +02:00
Meta Vens
8b9e8dfe11 Route map: Q-S5 ruled (a), B5 first
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:42:51 +02:00
Meta Vens
224f006983 Route map: Q-S1..Q-S4 ruled (a); Q-S5 open
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:42:42 +02:00
Meta Vens
64da79402a Deterministic backoff test (B5 Soul J)
`a_failing_step_is_retried_after_a_backoff_not_every_tick` raced the
wall-clock wait against ten ticks and lost on a loaded host (9 of 15
mutant runs). The test now owns the wait: held open, ten ticks retry
nothing; once due, the next tick retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:35:48 +02:00
Meta Vens
c04f24f518 B5: pin lease adoption from Odin's Ready word (B5 Soul F)
A stateful candidate Odin reports on is warming until it holds the lease;
the world's fixture now says so, and a post-fence warming word lets the
grant happen. Odin's Ready carrying the granted lease records the adoption,
sourced `OdinTopology` and naming the presence Odin observed. Kills Soul's
M6 (the source labelled `Direct`) and M7 (no adoption from Odin).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:35:48 +02:00
Meta Vens
e0b31eee76 B5: pin the resolver's command filter (B5 Soul E)
With two live transactions, an expiry request for one leaves the other
running. Soul's mutant "any request expires every live transaction"
survived the suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:34:48 +02:00
Meta Vens
47067cce08 B5: an abort withdraws a lease issued as Prepared (B5 Soul A)
The physical grant lands before `Granted` is durable. A continuity that
died in that window and passed its deadline was aborted from `Prepared`,
its abort withdrew only a `Granted` lease, and the lease left on disk made
every later fencing of the target refuse "unexpected process write lease".
The abort now withdraws any issued lease; revoking an absent one is a no-op.

Test: a granted lease on disk behind a `Prepared` record, deadline passed:
the abort removes it and the next fencing's revoke succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:34:30 +02:00
Meta Vens
4ee70f90ba Route driver: transaction paths always write and reload (B5 Soul B)
`6ff7d5a` skipped the write and reload in `install` and `restore` when the
fragment on disk already equalled what they would write. On a transaction
path that hides a crash between a write and its reload: an abort that
retries after the crash reloads nothing and then stops the candidate, so
nginx keeps proxying to a dead port, and a fenced install skips every retry
(and a first deploy's `ufw allow`) until its deadline. The skip on equal
bytes goes back to supervision only.

What stays: a transaction whose candidate membership is the preflight's
baseline (a continuity over its admitted route) has no route change to make
or undo, so its install, rollback and withdrawal run nothing. No write of
its own can have been cut short. `RoutePreflightReceipt::changes_route`
names it.

`restore_admitted_membership` now puts back the bytes it replaced when its
reload fails, instead of deleting the fragment, so the disk still shows
what nginx serves.

Tests: a crash between a write and its reload is reloaded on retry (install
past the fence and withdrawal, driver level; the abort, engine level); a
transaction that does not change the route runs nothing. Soul's F2 probe
test is gone with the rule it pinned: a withdrawal whose `ufw delete` keeps
failing reloads on every retry again, until the Q3-ii cut gives the abort a
deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:33:57 +02:00
Meta Vens
6854b0f825 Route map: B5 Soul pass 1 (merge after batch 2: lease wedge, reload-skip crash gap)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:29:56 +02:00
Meta Vens
5f232a6120 Route map: Q3-ii cut map (snapshot before Ready, roll forward after); Q-S1..S5 open
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 18:16:06 +02:00
Meta Vens
3fad441917 Route map: Q3-ii ruled: snapshot/restore before Ready, roll forward after
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:56:54 +02:00
Meta Vens
e00ff5ec40 Route map: Q3-i ruled (issued lease counts as possibly written); Q3-ii open
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:54:06 +02:00
Meta Vens
448f554af5 Watchdog map: Bifrost suite probes C21-C28
Only cultcache-ts was renamed in CultLib; Bifrost's tests and three tools
still use the bare name. Production Persona delivery is unaffected: its
release bundles CultLib f67f5122.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:15:38 +02:00
Meta Vens
c06ff2b84c B5: the expire test asks beside another target's live deployment
A terminal record leaves the live set, so it could not tell the verb's own
command filter from a looser one; another target's deployment in flight can.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:56:02 +02:00
Meta Vens
855ccf8247 Pin the incident retention and the status usage; keep transport names out of src
The retention constant and the usage line survived mutation untested. The
incident module doc named the delivery service, which the no-transport grep
over src/ must not find. Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 16:53:01 +02:00
Meta Vens
6394100a93 B5: pin the idunn expire verb as the CLI parses and runs it
Kills three cargo-mutants survivors: the --requested-by arm, run's
dispatch, and expire's own-live-transaction filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:51:06 +02:00
Meta Vens
7f7528b33e Add the idunn.operator_incident.v1 wire fixture
One open and one closed record, written by IncidentStore on Yggdrasil under
IDUNN_WRITE_FIXTURES=1 and read back by the_incident_fixture_is_what_idunn_reads.
Bifrost's reader tests copy this file. Builds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 16:23:57 +02:00
Meta Vens
202e5a2e10 Idunn records continuity exhaustion as a durable incident in incidents.cc
Idunn's daemon, at the continuity decision site, writes one
idunn.operator_incident.v1 record per exhaustion into its own published
incidents.cc (--incident-store), never control.cc. The record opens once per
(target, condition), closes on an observed running workload (recovered) or on
the target leaving the admitted set, and retires to incident-history.cc seven
days after closing. A store failure is traced once and never returned into the
tick. idunn status prints each target's incidents. ReportOnce is demoted to a
stderr trace.

Does not build the fixture test until tests/fixtures/idunn.operator_incident.v1.cc
lands in the next commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 16:21:02 +02:00
Meta Vens
436d6445fc Watchdog map: cut probes C1-C20; incidents live in their own incidents.cc
Imagination's cut-mapping probes and rationale. Self collapses the model
page's record and projection rows: control.cc refuses foreign types, so
incidents.cc is both the record and what Bifrost reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:12:18 +02:00
Meta Vens
487f4f14b8 Watchdog target: a closure notice follows an opening notice
Operator ruled closure-notice = opening-and-closure (target r2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:58:14 +02:00
Meta Vens
153af6ac40 Watchdog reporting map: body facts, model page, crossing rationale
Imagination step 0b for campaign idunn-watchdog. Forks are admitted in the
mind as questions delivery-crossing, paging-conditions, closure-notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:57:34 +02:00
Meta Vens
4830af8eb1 B5: a continuity over its admitted route reloads nothing while its proof fails
The live shape of Soul's F1: a continuity reuses the admitted Expected, so
its install and rollback find the admitted fragment already on disk and run
no host program, before or after its Routing deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:50:55 +02:00
Meta Vens
495112565b Watchdog reporting target: Idunn decides incidents, Bifrost delivers
First typed Eureka campaign (idunn-watchdog). The typed target and rulings
live in Eureka's mind; this doc holds the rationale and design truths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:48:18 +02:00
Meta Vens
0c80a444f6 B5: literal control snapshots in tests carry the expiry requests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:46:21 +02:00
Meta Vens
3f4d7561d9 B5: every post-fencing phase ends
A deadline resolver runs before a live post-fence transaction's step. Once
the phase deadline F0 stamps on entry has passed, or an operator has asked
for it with `idunn expire`, the phase is resolved as Q3 rules:

- A candidate never issued a write lease (every stateless target, and a
  stateful one still in Fencing or early Leasing) aborts to the incumbent
  through the existing post-fencing path.
- A continuity always aborts: its candidate is the admitted release, and
  supervision restarts it under the continuity backoff. A granted lease
  with no adoption evidence names the failure `lease-not-adopted`.
- A deployment whose candidate was issued a lease is reported and left to
  the operator. That is the open fork: what holds the incumbent after an
  adopted-lease deployment, and whether a warming candidate's silence is
  evidence of non-adoption, are not ruled.

Resume paths cannot extend a deadline: only `enter_phase` writes it. A
record lifted from before deadlines existed is timed once, from the first
time the resolver sees it.

Lease adoption is recorded with Ready (the candidate's presence, or Odin's
Ready correlation, held exactly the granted lease), and the terminal
record names `RestartAdmitted` for a continuity whose candidate adopted it.

F16: Commit goes on from the topology it admitted instead of returning, so
Odin re-stamping every tick no longer sends it round forever.

`idunn expire <command-id>` writes an `idunn.expiry_request.v1` record and
nothing else; the resolver reads it as a passed deadline, and a spent
request is deleted. A new control-store document type: an Idunn from before
this commit refuses a store holding one.

Tests commit Soul's F1 and F2 probes: a continuity whose route proof keeps
failing ends at its Routing deadline, and a withdrawal whose `ufw delete`
keeps failing reloads once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:40:05 +02:00
Meta Vens
fa830c5f6f Route map: B2 + 3d installed on Yggdrasil (a96ad9d)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:33:26 +02:00
Meta Vens
6ff7d5a9c7 Route driver: a fragment already correct is neither written nor reloaded
`install` and `restore` skip the write, the firewall step and the reload
when the fragment on disk already equals what they would write. `restore`
still withdraws the endpoint when the prior membership is none, so a
withdrawal whose `ufw delete` failed repeats only that step.

That skip is sound only while bytes on disk are what nginx last loaded, so
every write whose reload fails is now undone: `restore` puts the previous
fragment back, and `install` puts the baseline back in both the rollback
and the fenced case (it used to retain the candidate for retry, which the
retry would now take for live). An install that never loaded needs no
reload to roll back.

Tests: a retry with an already-correct fragment performs zero reloads (install
and a withdrawal with a failing `ufw delete`); a failed reload puts the
fragment back so the retry reloads. Two engine tests that counted a reload
for withdrawing a never-installed route now count the endpoint withdrawal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:32:38 +02:00
Meta Vens
a96ad9d4e6 Merge B2-3d: the route actuation ceiling counts only Forward
Operator ruling 3d A. Survival actuations are recorded, never counted or
refused. Soul: verify.sh clean at 057bdd5 (341); own mutants killed. Known,
not new: Survival retries inside one transaction reload nginx unbounded
(~120/h per target) until B5, which is the next Idunn cut. Operator
approved installing B2 ahead of B5 (2026-09-30).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:11:18 +02:00
Meta Vens
5939abd87e Route map: B2-3d Soul result; the Survival-bound premise is false
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:09:24 +02:00
Meta Vens
057bdd5c6c B2-3d: status shows a survival-only record on a generation that no longer routes
cargo-mutants on the 3d diff missed `|| -> &&` in render_supervision: no
test held meters with only survival entries under an unrouted generation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 12:39:32 +02:00
Meta Vens
e9c718e073 B2-3d: the route actuation ceiling counts only Forward
Ruling 3d A (2026-09-30): the 12/h per-target route actuation ceiling
counts only Forward actuations (deploys). Survival actuations (continuity
restarts, rollbacks, withdrawals, admitted-route repairs) are recorded for
visibility in a separate TargetSupervision.route_survivals log (key 6,
newest 12 kept) and are never counted against the ceiling or refused by it.
A target whose continuity spent its six restarts can therefore always be
rescued by a deploy.

Each Survival path keeps its own bound: a continuity's actuations the
restart log and its doubling wait, an admitted-route repair the challenge
backoff, a deployment's rollback or withdrawal the Forward change it
undoes. A candidate whose route proof keeps failing now reloads at most
twice the ceiling per window (12 installs, each with its rollback) instead
of 12.

status prints the survival record beside the ceiling line.

TargetSupervision stays at v1 with the added slot: no installed Idunn has
written the record (B2 is merged, not installed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 12:21:58 +02:00
42 changed files with 26678 additions and 3869 deletions

1
Cargo.lock generated
View file

@ -482,6 +482,7 @@ dependencies = [
"libc",
"rmp-serde",
"serde",
"serde_json",
"sha2",
"tempfile",
"toml",

View file

@ -18,6 +18,10 @@ path = "src/bin/idunn-provision.rs"
name = "idunn-host"
path = "src/bin/idunn-host.rs"
[[bin]]
name = "idunn-launch"
path = "src/bin/idunn-launch.rs"
[lib]
name = "idunn_daemon"
path = "src/lib.rs"
@ -34,6 +38,7 @@ serde = { version = "1", features = ["derive"] }
ed25519-dalek = "2.2"
toml = "0.8"
uuid = { version = "1", features = ["v4"] }
serde_json = "1"
[dev-dependencies]
tempfile = "3"

131
README.md
View file

@ -58,6 +58,64 @@ idunn status --command <id>
the bindings, so `idunn up profile:full-gamecult` brings up everything bound
into that profile.
### Steward orders
A second root unit from the same binary, `idunn-orders` (`idunn orders serve`),
takes typed orders from one steward Mind over `/run/idunn-orders/steward.sock`.
It has its own store, `orders.cc`, and its own binding, `steward.toml`; the
survival daemon takes no order. There is still no command string:
```bash
idunn order package-install <pkg>[=version]... --reason TEXT # apt-get update, install named packages, check them
idunn order unit <start|stop|restart|reload> <unit> --reason TEXT
idunn order deploy <service|profile:name> --reason TEXT # the same commit as `idunn up`
idunn order show <id> # status and output tails
idunn order list
```
A unit order submits exactly one job, for the unit it names: every verb runs
`systemctl <verb> --job-mode=ignore-requirements -- <unit>`, so systemd builds a
transaction of one job. It does not start what the unit requires or restart what
is part of it: order the dependency first, and order `postgresql@16-main.service`
rather than the `postgresql.service` wrapper. Targets are not orderable. The job is one; what the unit does afterwards is
not: systemd still runs full transactions of its own for `Restart=` after a
failure, for a `StopWhenUnneeded=` dependency left without a user, and for a
`Conflicts=` partner of a unit that fails, so an ordered stop of a unit
that something restarts does not last, and a start of a `Restart=always`
unit can report failure and be active seconds later. A unit
order may act on any unit except Idunn's own (`idunn-*`) and the minds' own
(`mind-self@*`, `mind-persona@*`), and the units that end or take the host
(`systemd-suspend`, `systemd-hibernate`, `systemd-hybrid-sleep`,
`systemd-suspend-then-hibernate`, `rescue`, `emergency`). That set is compiled
into the binary and read from no configuration. An order is refused, recorded
and posted when the ordered unit, or an alias it resolves from, is one of them;
when it has an emergency action (`SuccessAction=`, `FailureAction=`,
`StartLimitAction=`, `JobTimeoutAction=`) other than `none` or an `OnFailure=`
or `OnSuccess=` unit (not for a reload); or, for a stop or restart, when another
unit is `BoundBy=` it, because systemd stops that unit with it whatever the job
mode; for a start or restart, when it has an `Upholds=` unit, which systemd
starts in a full transaction whatever the job mode; or, for a stop, restart or
reload, when it has a `ConsistsOf=` member, a wrapper such as
`postgresql.service` whose verbs would exit 0 and touch nothing. A unit
`systemctl show` cannot resolve fails the order. Slices, scopes,
targets and globs are not units an order can name.
A package order installs the packages it names and nothing else: apt runs with
`--no-remove`, each name is passed as an exact-name pattern so `.` and `+`
match themselves, and a last step fails the order unless every named package
is installed.
An order runs once, after the steward's uid is checked and the instance's
brake check (`mind-control --instance <principal> check`) exits 0. The check
that the order's principal and uid are the bound steward's runs again at every
execution, so an order accepted before `steward.toml` changed is refused and
recorded, not run. The socket belongs to the steward user alone (mode 0600);
root may `show` and `list` orders but never issue one. An order that had
started when the service died is recorded as lost and never run again. No order releases a brake. Acceptance and result are posted to
`#operator` as a retried projection of `orders.cc`: a chat outage delays the
posts and not the orders. A refusal names the field and a code, never the value
it refused.
## Authority
**Owner.** Deployment admission, and the continuity of installed bodies.
@ -150,6 +208,79 @@ omits one. Unknown fields are rejected on both sides. A repository that gets
compromised can therefore change what is built, but not what it is allowed to
touch on the host.
An upstream daemon that does not speak Idunn's runtime contract is deployed
with `idunn-launch` as its executable artifact and the upstream tree as a
required adjacent artifact; the launcher finds the child inside the release that
holds it. The launcher is the unit's process: it binds the
candidate endpoint and answers Idunn's challenge there, renders the child's
config, starts a stateful child only once it holds the write lease, and signs
presence. Members never pass through it: the binding's `[route.member]` makes
Idunn route a second listener straight to the child, which binds the address
Idunn gives it in `GAMECULT_IDUNN_MEMBER_BIND`.
```toml
# recipe fragment — the upstream tree is built or fetched as an artifact
[service]
executable_artifact = "idunn-launch"
arguments = [
{ kind = "literal", value = "--child" },
{ kind = "literal", value = "upstream/bin/daemon" },
{ kind = "literal", value = "--template" },
{ kind = "literal", value = "config.tmpl" },
{ kind = "literal", value = "--health-path" },
{ kind = "literal", value = "/health" },
{ kind = "literal", value = "--health-contract" },
{ kind = "literal", value = "example.presence-launcher.v1" },
]
required_environment = [
"GAMECULT_IDUNN_RUNTIME_BUNDLE",
"GAMECULT_IDUNN_CANDIDATE_BIND",
"GAMECULT_IDUNN_MEMBER_BIND",
]
```
A directory artifact is sealed read-only: its files are 0444 and its
directories 0555, whatever mode the build or git gave them. A file the tree
must run, such as the launcher's `upstream/bin/daemon` child, is declared in
`executable_members` and sealed 0555. The members are part of the sealed release
address, so changing the list makes a new release. Idunn refuses the release if a
member is absolute, climbs with `..`, is missing, is a symlink or sits behind
one, or is not a regular file; `executable = true` on a directory output and
`executable_members` on a file output are refused too.
A tree may hold relative symlinks whose `..` steps all come first and climb no
higher than the artifact's top (`share/terminfo/4/4025ex -> ../t/tek4025ex` is
kept). An absolute link, a `..` after a name, or a climb out of the artifact
refuses the release when it is sealed. Host targets (Windows) record
`executable` and `executable_members` in the receipt and do not apply them.
```toml
# recipe fragment — the upstream artifact declares the child it launches
[[artifacts]]
id = "upstream"
source_kind = "runner-output"
runner = "build"
source = "dist/upstream"
destination = "upstream"
executable_members = ["bin/daemon"]
```
```toml
# binding fragment — members connect to the member endpoint, which Idunn
# forwards to the private port derived from the candidate: member
# private_port_start + (candidate port - route private_port_start)
[route]
stable_endpoint = "tcp://10.77.0.1:8008"
private_port_start = 18831
private_port_end = 18839
# ...
[route.member]
stable_endpoint = "tcp://10.77.0.1:8448"
private_port_start = 18841
# max_connections = 128 # nginx limit_conn on the member listener (1-65535)
```
## Contracts
Idunn's typed contracts are published by

View file

@ -0,0 +1,47 @@
[Unit]
Description=Idunn steward orders: typed package, unit and deploy orders from the steward
# Deliberately no After= or Requires= on any managed target, Synapse or
# idunn-yggdrasil: orders are how an operator repairs those, so none of them
# may be a condition of this unit starting. Posts are a retried projection and
# wait for Synapse on their own.
[Service]
Type=simple
# Root, from the same binary as idunn-yggdrasil, with its own store and socket.
# idunn-yggdrasil keeps its no-imperative-input invariant and its write set:
# no order code runs in `idunn serve`.
#
# The unit name matches idunn-*, which is in the always-refused set, so no
# order can stop or restart this service.
User=root
# No Group= and no UMask=: the service gives its socket to the steward user
# (mode 0600) itself and sets Idunn's own umask for the files it writes, so
# nothing here reaches control.cc. The steward user is the only peer the
# service answers: it checks SO_PEERCRED on every connection.
ExecStart=/usr/local/bin/idunn orders serve \
--store /var/lib/gamecult/idunn-orders \
--state-store /var/lib/gamecult/idunn/control.cc \
--steward-binding /etc/gamecult/idunn/steward.toml \
--socket /run/idunn-orders/steward.sock
Restart=always
RestartSec=5
TimeoutStopSec=20
RuntimeDirectory=idunn-orders
RuntimeDirectoryMode=0755
# The Synapse access token of @idunn, copied from a root-only file, and read
# through $CREDENTIALS_DIRECTORY; the service names it in steward.toml.
LoadCredential=idunn-matrix-token:/etc/gamecult/idunn/matrix-token
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
# Its own order store, and the control store a deploy order commits a command
# to. Transient units started with systemd-run run in the manager's namespace,
# so packages and units are not limited by this list.
ReadWritePaths=/var/lib/gamecult/idunn-orders /var/lib/gamecult/idunn
[Install]
WantedBy=multi-user.target

View file

@ -41,6 +41,7 @@ ExecStart=/usr/local/bin/idunn serve \
--source-root /var/lib/gamecult/idunn/sources \
--staging-root /var/lib/gamecult/idunn/staging \
--topology-store /var/lib/gamecult/idunn-projection/topology.cc \
--incident-store /var/lib/gamecult/idunn-projection/incidents.cc \
--idunn-identity-store /var/lib/gamecult/idunn/idunn-service-identity.cc \
--odin-correlation-store /var/lib/gamecult/odin/topology.cc \
--odin-trust-anchor /etc/gamecult/idunn/odin-topology-anchor.cc \
@ -76,7 +77,10 @@ ReadWritePaths=/etc/ufw
# idunn-projection is deliberately outside Idunn's own 0750 directory: the
# topology store is Idunn's *published* surface, and every managed target must
# read it to verify its own Expected incarnation against the Idunn anchor. It
# was inside the private root, where no workload could traverse to it.
# was inside the private root, where no workload could traverse to it. The
# published topology store is read lock-free (Odin) and incidents.cc is read by
# Bifrost's reader; managed targets get Expected from their runtime bundle. Only
# the stores are readable: each .lock beside them is Idunn's alone, 0600.
ReadWritePaths=/var/lib/gamecult/idunn /var/lib/gamecult/idunn-authority /var/lib/gamecult/idunn-projection
ReadWritePaths=/srv/build
# Each managed target gets only its bound roots. A target Idunn cannot write is
@ -100,6 +104,11 @@ ReadWritePaths=/etc/gamecult/ghostlight/runtime /var/lib/gamecult/ghostlight-wor
ReadWritePaths=/etc/gamecult/streampixels-service/runtime /var/lib/gamecult/streampixels-service /srv/streampixels-idunn/releases/service
# StreamPixels web is stateless; it needs only its runtime bundle and releases.
ReadWritePaths=/etc/gamecult/streampixels-web/runtime /srv/streampixels-idunn/releases/web
# Vili (motion worker): runtime bundle, state, and releases. Prefixed "-" so a
# target whose roots are not created yet cannot stop Idunn itself from starting
# (Idunn startup never depends on a managed target). A root created after Idunn
# starts is writable only from Idunn's next restart.
ReadWritePaths=-/etc/gamecult/vili/runtime -/var/lib/gamecult/vili -/srv/vili-idunn/releases
[Install]
WantedBy=multi-user.target

View file

@ -4,11 +4,16 @@
kept because they are what is *running*, and a migration needs both sides.
`idunn-yggdrasil` is a root shell actuator reached through the narrow sudo grant
in `idunn-yggdrasil.sudoers`. It hardcodes eleven targets, their repositories
in `idunn-yggdrasil.sudoers`. It hardcodes ten targets, their repositories
and their admitted refs, then delegates to executable manifests under
`/srv/odin/deploy-manifests/<target>`. It calls `idunn validate-release-authority`
and `idunn validate-minimum-source-revision` — subcommands the current binary
does not have.
`/srv/odin/deploy-manifests/<target>`. It calls `idunn validate-minimum-source-revision`,
a subcommand the current binary does not have.
Checked on yggdrasil 2026-09-30: `/usr/local/libexec/idunn-yggdrasil` is installed
and matches this file byte for byte (before the retirement below), but
`idunn-yggdrasil.sudoers` is not installed. `sudo -l -U idunn` says the `idunn`
user may not run sudo, so the daemon cannot reach this actuator; only an
operator running it as root can.
## What replaced it
@ -16,10 +21,10 @@ Everything this script does by convention, the current Idunn does by contract:
| Legacy mechanism | Current owner |
|---|---|
| `deployment_target_policy()` — a `case` over eleven target names | one operator binding per target, in `--bindings-dir` |
| `deployment_target_policy()` — a `case` over ten target names | one operator binding per target, in `--bindings-dir` |
| `upstream_ref=refs/heads/...` in shell | `[repository] admitted_ref` |
| `minimum_source_revision_required` | `[repository] minimum_revision` |
| `requires_bifrost_authority` + `--release-authority-store` | `[repository] selection = "signed-release"`, per target |
| `requires_bifrost_authority` + `--release-authority-store` (retired with its only target) | `[repository] selection = "signed-release"`, per target |
| `/srv/odin/deploy-manifests/<target>` shell scripts | recipe `[[steps]]` in the target's own repository |
| `run_manifest_while_holding_authority_locks` | the deployment transaction and process write-lease |
| `flock` on a brake store | `[brakes] deployment_store` and `lifecycle_store` |
@ -45,15 +50,26 @@ This is the real migration scope, read out of `deployment_target_policy()`:
| `heimdall` | GameCult/Heimdall | `main` | restart is `docker compose restart` |
| `epiphany` | GameCult/Epiphany | `codex/epiphany-shakedown-live` | |
| `epiphany-capstone-17` | GameCult/Epiphany | `codex/epiphany-shakedown-live` | second target, same repo |
| `bifrost-persona-feedback` | GameCult/Bifrost | `main` | **only** target requiring Bifrost release authority |
| ~~`bifrost-persona-feedback`~~ | GameCult/Bifrost | `main` | **retired 2026-09-30**, see below |
| `repixelizer` | GameCult/repixelizer | `main` | |
| `streampixels` | GameCult/StreamPixels | `main` | two units, one target |
| `gjallar` | GameCult/Gjallar | `codex/yggdrasil-aggregate-daemon` | |
Seven of eleven are pinned to a non-`main` branch. That is its own problem — the
Seven of the ten live targets are pinned to a non-`main` branch. That is its own problem — the
swarm goal is everyone on `main` and current with each other — and migrating a
target is the moment to fix its ref, not to copy it forward.
## Retired targets
- `bifrost-persona-feedback`, 2026-09-30. Its manifest ran the Bifrost compose lane
(`docker compose up persona-feedback`), while the live provider is the native
`bifrost-persona-feedback.service`, so firing it would have started a container
beside the native unit. The operator ruled "delete the Bifrost lane". Its grant,
dispatch case and the Bifrost release-authority check it alone used are gone; a
deploy request that claims a Bifrost authority is refused. It was the only
caller of `idunn validate-release-authority`. Rebinding persona-feedback to a
current Idunn binding is a separate follow-up.
## When these files die
When every target above has a binding and a recipe, and `idunn serve` owns

View file

@ -96,14 +96,12 @@ deployment_target_policy() {
daemon_id=yggdrasil-odin
repository_full_name=GameCult/Odin
upstream_ref=refs/heads/codex/ygg-idunn-independent-bootstrap
requires_bifrost_authority=false
manifest_takes_action=true
;;
codex-connector)
daemon_id=yggdrasil-codex-connector
repository_full_name=GameCult/CodexConnector
upstream_ref=refs/heads/codex/ghostlight-release-binding
requires_bifrost_authority=false
minimum_source_revision_required=true
manifest_takes_action=true
;;
@ -111,7 +109,6 @@ deployment_target_policy() {
daemon_id=yggdrasil-ghostlight
repository_full_name=GameCult/Ghostlight
upstream_ref=refs/heads/codex/ghostlight-dungeon-mvp
requires_bifrost_authority=false
minimum_source_revision_required=true
manifest_takes_action=true
;;
@ -119,52 +116,39 @@ deployment_target_policy() {
daemon_id=yggdrasil-voidbot
repository_full_name=GameCult/VoidBot
upstream_ref=refs/heads/main
requires_bifrost_authority=false
manifest_takes_action=true
;;
heimdall)
daemon_id=yggdrasil-heimdall
repository_full_name=GameCult/Heimdall
upstream_ref=refs/heads/main
requires_bifrost_authority=false
;;
epiphany-capstone-17)
daemon_id=yggdrasil-epiphany-capstone-17
repository_full_name=GameCult/Epiphany
upstream_ref=refs/heads/codex/epiphany-shakedown-live
requires_bifrost_authority=false
manifest_takes_action=true
;;
epiphany)
daemon_id=yggdrasil-epiphany
repository_full_name=GameCult/Epiphany
upstream_ref=refs/heads/codex/epiphany-shakedown-live
requires_bifrost_authority=false
manifest_takes_action=true
;;
bifrost-persona-feedback)
daemon_id=yggdrasil-bifrost-persona-feedback
repository_full_name=GameCult/Bifrost
upstream_ref=refs/heads/main
requires_bifrost_authority=true
;;
repixelizer)
daemon_id=yggdrasil-repixelizer
repository_full_name=GameCult/repixelizer
upstream_ref=refs/heads/main
requires_bifrost_authority=false
;;
streampixels)
daemon_id=yggdrasil-streampixels
repository_full_name=GameCult/StreamPixels
upstream_ref=refs/heads/main
requires_bifrost_authority=false
;;
gjallar)
daemon_id=yggdrasil-gjallar
repository_full_name=GameCult/Gjallar
upstream_ref=refs/heads/codex/yggdrasil-aggregate-daemon
requires_bifrost_authority=false
;;
*) echo "deployment target has no root actuator policy: $target" >&2; exit 1 ;;
esac
@ -190,19 +174,10 @@ require_signed_deployment_grant() {
*) echo "signed deployment request is not for root target $target" >&2; exit 1 ;;
esac
case "$requires_bifrost_authority" in
true)
[ -n "$release_authority_id" ] || { echo "missing Bifrost release authority id" >&2; exit 1; }
case "$release_authority_sha256" in
*[!0-9a-f]*|'') echo "invalid or missing Bifrost release authority digest" >&2; exit 1 ;;
esac
[ "${#release_authority_sha256}" -eq 64 ] || { echo "Bifrost release authority digest must be 64 lowercase hex characters" >&2; exit 1; }
;;
false)
[ -z "$release_authority_id" ] || { echo "source-observed release cannot claim Bifrost authority id" >&2; exit 1; }
[ -z "$release_authority_sha256" ] || { echo "source-observed release cannot claim Bifrost authority digest" >&2; exit 1; }
;;
esac
# No target carries Bifrost release authority: its only one, bifrost-persona-feedback,
# was retired on 2026-09-30. A request that claims one is refused.
[ -z "$release_authority_id" ] || { echo "source-observed release cannot claim Bifrost authority id" >&2; exit 1; }
[ -z "$release_authority_sha256" ] || { echo "source-observed release cannot claim Bifrost authority digest" >&2; exit 1; }
require_root_owned_resolved_executable "$idunn_provision"
require_root_owned_regular_path "$brake_store"
@ -234,18 +209,6 @@ require_signed_deployment_grant() {
--daemon "$daemon_id" \
--source-revision "$source_commit"
fi
if [ "$requires_bifrost_authority" = true ]; then
require_root_owned_resolved_executable /usr/local/bin/idunn
/usr/local/bin/idunn validate-release-authority \
--store /var/lib/gamecult/idunn/yggdrasil.keepalive.cc \
--release-authority-store /srv/bifrost/state/repository-release-authority.cc \
--repository-full-name "$repository_full_name" \
--upstream-ref "$upstream_ref" \
--source-revision "$source_commit" \
--release-authority-id "$release_authority_id" \
--release-authority-sha256 "$release_authority_sha256"
fi
}
manifest_group_exists() {
@ -355,7 +318,7 @@ case "$action:$target" in
run_manifest_while_holding_authority_locks "$manifest" restart
exit $?
;;
deploy:voidbot|deploy:epiphany|deploy:epiphany-capstone-17|deploy:odin|deploy:codex-connector|deploy:ghostlight|deploy:heimdall|deploy:repixelizer|deploy:streampixels|deploy:bifrost-persona-feedback|deploy:gjallar)
deploy:voidbot|deploy:epiphany|deploy:epiphany-capstone-17|deploy:odin|deploy:codex-connector|deploy:ghostlight|deploy:heimdall|deploy:repixelizer|deploy:streampixels|deploy:gjallar)
manifest="/srv/odin/deploy-manifests/$target"
require_root_owned_regular_path "$manifest"
[ -x "$manifest" ] || { echo "root actuator manifest is not executable: $manifest" >&2; exit 1; }
@ -368,7 +331,8 @@ case "$action:$target" in
export BIFROST_RELEASE_AUTHORITY_ID="$release_authority_id"
export BIFROST_RELEASE_AUTHORITY_SHA256="$release_authority_sha256"
export IDUNN_DEPLOYMENT_REQUEST_ID="$deployment_request_id"
export IDUNN_REQUIRES_BIFROST_AUTHORITY="$requires_bifrost_authority"
# Installed manifests (Epiphany's) still require this to read false.
export IDUNN_REQUIRES_BIFROST_AUTHORITY=false
if [ "$manifest_takes_action" = true ]; then
run_manifest_while_holding_authority_locks "$manifest" "$action"
exit $?

View file

@ -1,7 +1,7 @@
# Narrow root actuator grant for the Idunn Yggdrasil deployment/continuity path.
# One line per live target. codex-connector and gjallar were retired 2026-09-06;
# their grants are removed here first, because the privileged half of a retired
# target must not outlive its consumer.
# One line per live target. codex-connector and gjallar were retired 2026-09-06,
# and bifrost-persona-feedback on 2026-09-30; their grants are removed here first,
# because the privileged half of a retired target must not outlive its consumer.
Cmnd_Alias IDUNN_YGGDRASIL_DEPLOY = \
/usr/local/libexec/idunn-yggdrasil deploy odin *, \
/usr/local/libexec/idunn-yggdrasil deploy ghostlight *, \
@ -9,7 +9,6 @@ Cmnd_Alias IDUNN_YGGDRASIL_DEPLOY = \
/usr/local/libexec/idunn-yggdrasil deploy heimdall *, \
/usr/local/libexec/idunn-yggdrasil deploy epiphany-capstone-17 *, \
/usr/local/libexec/idunn-yggdrasil deploy epiphany *, \
/usr/local/libexec/idunn-yggdrasil deploy bifrost-persona-feedback *, \
/usr/local/libexec/idunn-yggdrasil deploy repixelizer *, \
/usr/local/libexec/idunn-yggdrasil deploy streampixels *
Cmnd_Alias IDUNN_YGGDRASIL_RESTART = \

View file

@ -55,7 +55,10 @@ A deployable repository publishes one strict declaration, normally
- immutable artifact outputs, optionally pinned to an expected SHA-256;
- a constrained launch contract;
- one health-contract identity;
- an optional typed state contract with state schema generation and slots;
- an optional typed state contract with state schema generation and slots
(kinds `cultcache-file`, `external-file` and `external-directory`; the
external kinds are process-owned state an external runtime imposes, which
Idunn preserves and fences but never interprets);
- provided capabilities, dependency kinds, startup requirements, capacity, and
conflicts.
@ -67,6 +70,52 @@ recipe fields.
Stateless services declare no state block. Their operator binding therefore has
no state root, state-transition policy, or process write-lease record.
## Foreign workloads: idunn-launch
An upstream daemon that cannot speak Idunn's runtime contract (Synapse, MAS)
runs under `idunn-launch`, the unit's own process. Idunn's two-witness rule is
unchanged: Idunn observes the unit, and the launcher, inside the unit and
supervising the child, signs presence with the provider identity and the
activation key.
- **Owner:** the launcher owns presence, config rendering and the child's
lifetime inside the unit, and the route answers it signs for that child.
Idunn owns admission, lease, route and continuity, as before.
- **Inputs:** the runtime bundle, the activation credential and provider
identity descriptors, the process write lease, the systemd credentials the
binding names, and the child's health over TCP at the member bind.
- **Outputs:** signed `gamecult.runtime_presence_health.v2` (Warming, Active
and failed to Odin; route observations on the candidate endpoint), one child
process, and a rendered config in the unit's private tmp that is never read
back.
- **Route:** Idunn owns both listeners of the route: the Expected route's
stable endpoint (presence) and, when the binding has `[route.member]`, the
member endpoint, rendered as a second server block in the one nginx fragment
with a `limit_conn` on one per-route zone (`max_connections`, default 128), so
members cannot exhaust the connections the presence block shares with them.
The launcher binds the candidate endpoint and answers Idunn's challenge there,
one connection at a time within a one-second deadline; any other first byte
closes the connection. The answer never waits on the child: it reports the
lifecycle's last probe, and a failed, missing or stale probe reads `degraded`.
The child binds the member bind Idunn sets in
`GAMECULT_IDUNN_MEMBER_BIND`, derived from the candidate port. Member bytes
never touch the launcher.
- **Lease:** with `write_lease_required` the launcher starts the child only
after it holds the lease (bound to this Expected, this activation and one of
its own Warming proofs) and keeps the lease store share-locked while the
child lives. Losing the lease stops the child.
- **Config:** one template from the release is rendered with the closed
placeholder set `{{child.listen}}`, `{{child.listen_host}}`,
`{{child.listen_port}}` (all three the member bind), `{{credential:NAME}}`,
`{{credential_path:NAME}}` and `{{state_root}}`. Any other placeholder
refuses to start.
- **Forbidden:** member traffic through the launcher; the child binding the
candidate endpoint; a child of a lease-requiring target running without a held lease; any presence not signed
by both keys.
- **State kind `external-directory`:** a process-owned directory imposed by an
external runtime contract. Same writer, recovery and startup rules as
`external-file`: it is lease-gated, preserved and fenced, never interpreted.
## Operator binding
The root-admitted operator binding supplies the physical policy a repository
@ -173,6 +222,41 @@ health plus Ready correlation. It changes and observes route membership only
afterward. Only after the proxy has adopted the candidate may Idunn clear the
promotion fence and drain the incumbent.
A target has two slots: the incumbent (the admitted generation) and a
transaction's candidate. Continuity owns the incumbent slot and keeps the
admitted generation alive; it never stops. A live `DeploymentTransaction` owns
its private candidate before Fencing and takes the target itself at Fencing (a
stop-then-start rollout, at Starting) through one `take_target` CAS, so of two
transactions entering from one view exactly one succeeds. A continuity
restart is itself a transaction and takes the target the same way.
Phases from Warming through Committing carry deadlines, as do aborts and
drains; an expired deadline never waits silently. A stateless deployment
aborts to its incumbent. A deployment whose candidate was issued a write lease
is left to the operator (`operator-required` incident, the second incident
condition beside `continuity-exhausted`; its subject is the transaction id and
it closes when the record ends or its step goes on). A continuity after
Fencing retries its phase forward on a capped backoff and never stops its live
candidate, since that is the only running copy of the admitted body; the
deadline opens the same incident. The one exception is yielding: a continuity
whose route proof has not succeeded gives way, through its own post-fencing
abort, to a deployment the deployment brake has released and that awaits the
handover; a continuity that has proven its route never yields. The
`ContinuityExhausted` incident (six charged attempts in an hour,
`restarts_page`) pages and gates nothing. When continuity next acts is derived
only from its own record in `TargetSupervision` (schema v2): the attempt
streak, the backoff anchor (written by the dead-and-free pass, the only
writer), and the once-per-admission re-materialization bound. The workload
driver alone answers whether a lost unit was a host reboot
(`WorkloadPort::lost_to_host_reboot`). Operator-facing incidents and
`idunn status` are projections that decide nothing. A boot lift returns a
continuity abort left by an older Idunn, one that had not stopped its running
candidate, to forward retry; if it cannot decide, every continuity abort is
held until a lift succeeds. It is a migration with a deletion line (follow-up
`c2c-delete-boot-lift`). Cut line: cuts 3 and 4 of `reincarnation-map.md`
still owe dependency failures not growing the streak and the dead-generation
route refresh.
The deployment brake gates changes to source, release, configuration, schema,
unit, or authority binding. The lifecycle brake separately gates restart of an
already-admitted incarnation. Neither target brake may gate Idunn itself or an

View file

@ -0,0 +1,394 @@
# Deployment provenance: map
Status: map, Imagination, 2026-10-23 (campaign `idunn-deploy-provenance`,
session `self-2026-10-23-eureka-b`). The ends are in
[`deploy-provenance-target.md`](deploy-provenance-target.md). Questions,
rulings, cut specs and follow-ups live in Eureka's mind; this page holds what
has no kind: body facts, rationale and sequencing. Line numbers in T1-T12 are
at `d74ac8d` (origin/main on 2026-10-23); T4, T9 and T13-T16 are at `fd040f0`,
the head of Hands attempt 1, which revision 2 of the spec builds on.
## Cut `tree-executable-members`
Spec `idunn-deploy-provenance:cut_spec:cut-tree-executable-members.r2`
(revision 2, after Soul pass 1 found the real interpreter tree refused). It
serves `motion-pipeline:follow_up:idunn-tree-executable-members` (Vili's
admission waits on it) and will serve gamecult-chat C3 too: the presence
launcher runs a child such as `upstream/bin/daemon` from inside a tree artifact
(README, launcher fragment), and Synapse needs a Python interpreter there.
### Body facts
- T1. `ArtifactOutput` (`src/deployment.rs:106-117`) carries one `executable`
bool. `ArtifactReceipt` (`src/deployment_plan.rs:731-737`) copies it and
`validate` (`:740-760`) requires it to equal the declaration.
- T2. `harden_installed_release` (`src/drivers.rs:6848-6855`) builds its
executable set from `root.join(receipt.destination)` only. `harden_root_tree`
(`:6858-6892`) seals every directory 0555 and every file 0444 unless that
exact path is in the set. A directory artifact with `executable = true` is
therefore silently meaningless today: its files are all 0444. Nothing refuses
it. No live recipe declares one (CodexConnector, Ghostlight, Heimdall, Odin,
StreamPixels service and web surveyed): `executable = true` is on files only.
- T3. A tree output is copied by `copy_artifact`/`copy_tree`
(`src/drivers.rs:8053-8126`), which recreate symlinks verbatim. The digest
(`digest_tree`, `:8142-8175`) covers names, file bytes and link targets, not
mode bits. So runner or git exec bits never reach the release: harden is the
only mode authority.
- T4. `validate_release_symlink` (`src/drivers.rs:6897-6916`) keeps a symlink
only if its target is relative with no `..`. It runs only inside
`harden_root_tree`, at install, so a tree with a `..` or absolute link copies,
digests and seals, then fails at `install_release` (Soul s1: copying and
digesting the real tree twice took 160 s before harden refused it). The rule
is sound but too narrow: it refuses the real interpreter tree (T13). The
frozen-source path has a physical resolver (`resolve_frozen_source_symlink`,
`:6617`) that walks the filesystem with a 40-link budget and a memo. Its
first version was exponential (Soul S4-1, recorded in `d562a12`), and it needs
the tree on disk to answer. Revision 2 does not reuse it (T14).
- T5. The sealed release id hashes `(schema, plan_id, artifacts,
external_inputs)` (`src/deployment_plan.rs:1000-1011`), and `plan_id` hashes
the whole plan including `recipe_blob`. A changed member list is therefore
already a new plan and a new release. The receipt must still carry the list,
because `harden_installed_release` reads receipts only; its two callers
(`install_release` `:2788`, `validate_installed_release` `:2822`) re-harden
every installed release on each observation.
- T6. Encoding is `rmp_serde::to_vec`, positional. A trailing field with
`#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]` keeps every
existing receipt's bytes and release id: the plan's `deadlines` field
(`:574-577`) is the precedent.
- T7. `digest_artifact` and `digest_tree` (`:8128-8175`) `fs::read` each file
whole. A torch CPU tree's largest file (`libtorch_cpu.so`, hundreds of MB)
is held in Idunn's memory three times per install (seal, install, every
validate). `hash_frozen_source_tree` (`:6788-6840`) already streams in 64 KiB
chunks (S9). The cut collapses both onto one streaming helper.
- T8. `require_leaf_path` (`src/deployment.rs:1951-1958`) only keeps the
artifact destination a single filename. Members are paths inside that
destination, so it does not change. `require_relative_path` (`:1936-1949`)
with `allow_current = false` already refuses absolute, `..`, root and prefix
components.
- T9. The host actuator re-spells the seal (`src/host.rs:1050-1086`): copy,
mode check, digest, pinned-digest check and receipt, the same steps as
`collect_artifact` (`src/drivers.rs:2305-2326`). Soul s1: deleting its mode
check survives every test. Revision 2 collapses both onto one `seal_artifact`
in `drivers.rs`. Each caller keeps only how it finds the source path.
- T10. The transient unit runs with `ProtectSystem=strict` and the release in
`ReadOnlyPaths` (`src/drivers.rs:3675-3698`), `/tmp` mounted `noexec`, and no
`MemoryDenyWriteExecute`. Executing a 0555 file from the release works.
Shared libraries need no exec bit.
- T11. Live body on Yggdrasil: `/usr/local/bin/idunn` sha256 `44cf8a14...`,
installed 2026-10-20 02:34 CEST, before `6e2a31c` (02:40 CEST). `src/` is
identical between `8760a4f` and `6e2a31c`, so the live code is the
reincarnation c2 merge. `origin/main` is 26 commits ahead (+4194/-112 in
`src/`: the presence launcher and member route). Installing this cut ships all
of them. Idunn's own body changes only through gamecult-ops
`scripts/install-idunn-yggdrasil-release.sh` (staged binaries, unit,
bindings). The stage at `/home/gamecultadmin/idunn-stage` is stale (2026-09-11).
- T13. Link census of the real interpreter tree (Yggdrasil probe of
`cpython-3.12.15+20261021-x86_64-unknown-linux-gnu-install_only.tar.gz`,
sha256 `5ff9f060...77191124`; torch adds no links, as Soul s1 counted 1,049
after pip): 1,049 links. 0 are absolute, 749 have no `..`, and 300 have all
their `..` components before the first name (every one is
`share/terminfo/<c>/<name> -> ../<c2>/<name2>`, one `..` each). 0 have a `..`
after a name, 0 climb above the tree, 0 dangle and 0 resolve outside it.
Blender 5.2.2 has 79 links, none absolute or with `..` (Soul s1).
- T14. Lexically normalising `<link's directory>/<target>` does not check
containment. The kernel resolves a target one component at a time, so a `..`
after a symlinked name climbs from where that link points, not from where its
name sits. Probe on Yggdrasil: `root/a/s -> ..` and `root/a/L -> s/../x`
normalise to `root` and `root/a/x`, both inside, yet `L` reads the file `x`
beside `root`. `root/a/d -> .` with `root/a/M -> d/../../x` escapes the same
way. Idunn has met this before: verify-transaction-cut F5 (`D -> .`,
`L -> D/D/../../../etc/passwd` passed a lexical check).
The sound lexical rule constrains the target's shape: all its `..` components
come before its first name, and they number no more than the link's depth
below the artifact's top. The climb then starts from the link's own
directory, which the walk reached through real directories only (copy and
digest never descend through a link), so it lands on a real ancestor inside
the artifact. After that the target only descends, and any link it passes
through obeys the same rule. By induction on the kernel's hops (capped at 40,
then ELOOP), every hop stays inside. The check is one pass over the target's
components with no filesystem reads, so its cost is linear in the tree and no
resolver is needed. The shapes it refuses (`s/../x`) occur in neither real
tree (T13).
Prior art, from documentation as known to Imagination (high confidence for Go
and securejoin, medium for the tar details):
- Go `path/filepath.IsLocal` (1.20) is lexical and says so: it does not account
for symlinks. It would admit `s/../x`. Used alone, it is the T14 bug.
- Go `os.Root` (1.24) and `cyphar/filepath-securejoin` (used by runc,
containerd and umoci) resolve each component inside the root when a path is
opened, refusing or clamping an escape. They guard one later access, need
the tree on disk, and check again on every open. Idunn instead certifies a
sealed tree once, so that the kernel's own resolution stays inside on every
later access.
- Zip-slip defences and archive extractors check entry names lexically and
protect against link targets by refusing to write through links
(libarchive's `ARCHIVE_EXTRACT_SECURE_SYMLINKS`, and `SECURE_NODOTDOT` for
names; GNU tar creates such links last). They protect the extraction, not
what a kept link points at afterwards.
- OCI layer extraction keeps link targets verbatim, absolute ones included,
because the runtime resolves them inside the container's root. Idunn runs a
release in the host's mount namespace, so absolute targets stay refused.
- Nix keeps link targets verbatim and tracks dependencies by scanning for
references. The whole store is its root, so it has no per-artifact
containment to compare with.
None of these is a static per-link rule for a tree that runs unconfined. The
T14 shape rule fits that case.
- T15. Host targets: `idunn-host` observes, starts and stops processes on
Windows only (the non-Windows `platform` module in `src/host.rs` bails), and
`host.rs` sets no file modes. On Windows a file runs by its type, not by a
mode bit, so `executable` and `executable_members` have nothing to set there.
The file `executable` flag has always been carried and never applied on host
targets. A recipe does not know which workload kind its binding will choose.
- T16. The release root holds only artifact destinations. Runner workspaces,
where external inputs land, are removed before `SealedRelease::new`
(`src/drivers.rs:2421-2423`). `install_release` (`:2781-2788`) and
`validate_installed_release` (`:2815-2822`) digest every receipt's
destination before calling harden. So a rule in the digest walk runs at seal,
at install and at every validate, over every link harden would see.
- T12. Disk: `/` on Yggdrasil has 1.3 TB free. A tree release costs about three
copies (runner workspace, staging, installed). Nothing collects old releases or
staging (`ghostlight-verse:follow_up:idunn-generation-retention`), so every
Vili deploy leaves about 7 GiB behind.
### Contract, corrected against the body
- Any directory artifact may declare `executable_members` (a set of relative
paths). The follow-up limited it to runner-output; that limit protects no
invariant, and the launcher's worktree or fetched trees need the same feature.
- `executable = true` names a file artifact. On a directory output the release
is refused, and `executable_members` on a file output is refused (T2).
- A member is refused at recipe parse if it is empty, absolute, or holds `.`,
`..`, a root or a prefix. At seal it is refused if it is missing, is a
symlink, passes through a symlink or non-directory, or is not a regular file.
Refusal happens before `SealedRelease::new`, so no sealed release can name a
member it cannot seal.
- Harden seals each member 0555 and fails if any declared executable path was
not visited as a regular file. "Every declared executable is sealed 0555, or
the install fails" holds structurally, not by a lookup that can miss.
- Symlinks: a link in a tree artifact is kept when its target is relative, its
`..` components all come before its first name, and they number no more than
the link's depth below the artifact's top (T14). One function,
`check_release_link`, decides this. Only the digest walk calls it, so a bad
link refuses the release at seal, before `SealedRelease::new`, and again at
install and validate (T16). Harden no longer judges links. The climb is
bounded by the artifact's top, not the release root: no live release has a
`..` link (the old rule refused them all), so nothing installed is newly
refused, and each artifact's digest describes a closed tree.
- Host targets carry `executable` and `executable_members` in the receipt,
shape-checked at seal by the shared `seal_artifact`, and do not apply them
(T15). Refusing members there would make a recipe's validity depend on its
binding.
- Vili's trees must
not come from `python -m venv`: its `bin/python` is an absolute link and its
`pyvenv.cfg` names the build workspace. Install packages into the relocatable
interpreter tree instead (`motion-pipeline:follow_up:vili-relocatable-python-tree`).
### Authority
Owner: the seal (`seal_artifact`, called by `collect_artifact` for systemd
targets and by `materialize` in `host.rs` for host targets) decides
membership from the recipe and writes it into the receipt. Harden enforces the
receipt and decides nothing. Runner exec bits and git `100755` modes are dead
inputs for release modes. Link containment belongs to `check_release_link`,
which only `digest_tree` calls. `validate_release_symlink` is deleted, and
harden's symlink arm leaves links as they are.
### Acceptance sizing
The big-tree job (spec `verification.operator`) needs about 12 GiB of scratch
on Yggdrasil: a 3.5 GiB tree, its staging copy, downloads and pip cache. It
removes it at the end. Revision 2 runs it on the trees as shipped, with the 300
terminfo links (T13). If a tree holds a link the T14 rule refuses, the job
prints it and stops, and the link is reported to Self, not deleted.
## Cut `isolation-credential-evidence`
Spec `idunn-deploy-provenance:cut_spec:cut-cut-isolation-credential-evidence.r1`,
at `112a7a5` (origin/main, live on Yggdrasil since 2026-10-24 11:29 UTC). It
serves Vili's admission (`motion-pipeline:cut_report:cut-vili-admission.h4`,
first deviation): every `idunn up vili` is refused before fencing.
### Body facts (probed read-only on Yggdrasil, 2026-10-24)
- I1. `idunn up vili` runs 6 and 7 failed with
`systemd workload isolation properties differ from the Idunn contract`, the
message of `observe_unit` (`src/drivers.rs:2975-3005`). Of its conjuncts,
only one differs on the failed Vili unit: `LoadCredential`. Idunn expects
`""` because Vili has no service credential besides the presence identity
(`service_credential_names` empty); systemd shows `[unprintable]`. Every
other conjunct matches, InaccessiblePaths included (one value across all
five idunn-vili/odin units).
- I2. systemd 257.13 `systemctl show -p LoadCredential` prints
`LoadCredential=[unprintable]` for every unit, whatever the credentials:
`ssh.service` and `systemd-journald.service` (not DynamicUser, zero
LoadCredential), the four failed `idunn-vili-*` units (zero), and the Odin
unit (one). The display form carries no information, so the conjunct was
vacuous whenever credentials existed and always false when none did. Odin
and StreamPixels passed only because each carries a regular credential.
- I3. The exact value is readable: D-Bus
`org.freedesktop.systemd1.Service.LoadCredential` (`busctl get-property`)
answers `a(ss) 0` for the Vili units and ssh, `a(ss) 1
"ODIN_TOPOLOGY_IDENTITY" "<path>"` for Odin.
- I4. What the workload can actually read is the delivered directory. On the
host, `/run/credentials/` holds a directory only for units with credentials
(getty, journald, Odin, StreamPixels service and web). Inside Odin's MainPID
root, `/run/credentials/<unit>/` holds exactly `ODIN_TOPOLOGY_IDENTITY`
(`-r--r-----+ root root`). The presence identity is a parent-only `OpenFile`
descriptor, never a credential.
- I5. The rule arrived whole in `1fc0dad` (2026-09-03, "Rebuild Idunn
deployment transaction authority"), with its twin clause in
`validate_launch_observation` (`:3468-3472`, `:3513-3516`); no later commit
touched it. No ruling or finding in any campaign concerns workload isolation
verification; gamecult-chat's foreign-workloads cut touched
`CREDENTIALS_DIRECTORY` reservation, not this check.
- I6. `SystemdWorkloadObservation` is persisted by previous Idunns in both
MessagePack encodings, compact (positional) included
(`persisted_systemd_observations_decode_as_the_systemd_variant`,
`:9915-9930`), so deleting the `load_credential` field shifts every later
slot of a live incumbent's record. The field is also in the drift list
(`:2462`) that `stop` (`:4064-4070`) uses to refuse stopping a changed
workload: changing what the field holds without leaving that list would make
the next Idunn refuse to stop the live Odin and StreamPixels incumbents.
- I7. Fixtures: the verbatim 257.13 `systemctl show` output of base's 27
properties for a Vili unit (presence identity only) and the Odin unit
(presence identity plus one service credential), captured 2026-10-24 into
this Imagination's scratchpad `fixtures/`.
- I8. The new rule holds for every live systemd incumbent: inside each
MainPID root the delivered entries equal its binding's `secret_files` keys
minus `GAMECULT_RUNTIME_PRESENCE_IDENTITY` (Odin 1, StreamPixels service 11,
StreamPixels web 1; key names compared, no values read). An Idunn restart onto
this cut re-observes them (`:3955-3970`) and passes.
### Decision
A property systemd cannot display does not belong in the comparison. The
invariant it reached for, that the workload holds exactly the credentials Idunn
delivered, is observed where it is decided: the entries of
`/run/credentials/<unit>` in the MainPID's root equal Idunn's service
credential names, absent directory equal to none. That covers
`LoadCredential`, `SetCredential`, `ImportCredential` and drop-ins alike, and
needs no new program. The D-Bus read (I3) is exact but sees only one of those
properties and adds a second systemd client and parser; rejected.
### Authority
Owner: `observe_unit` decides the credential rule, from the delivered
directory. `LoadCredential`'s display form is no longer read or compared; the
`load_credential` field is a retired wire slot, written empty, out of the drift
list, kept only so persisted records decode. Forbidden writers: the
`observe_unit` conjunct and the `validate_launch_observation` clause. Dropping
the slot waits on the next schema revision of the observation.
### Deploy
Idunn's body changes only through gamecult-ops
`scripts/install-idunn-yggdrasil-release.sh` (stage `idunn` only; unit and
bindings unchanged). After it, Vili's Hands reruns from step 4 of
`motion-pipeline:cut_report:cut-vili-admission.h4`'s undone list.
## Cut `runner-failure-evidence`
Spec `idunn-deploy-provenance:cut_spec:cut-runner-failure-evidence.r1`, at
`804ed75` (origin/main, live on Yggdrasil). It serves the StreamPixels-area
Self: `idunn up vili` (command `up-4866467c-ee98-4a2b-b4a9-a008d7c44acd`,
transaction `tx-a04b68c7-c742-4d7d-8220-8aa08b9febae`, 2026-10-25 00:50-00:55
UTC) failed in Sealing with `Docker runner exited with exit status: 1: ` and
nobody could tell which recipe step failed.
### Body facts (source read at `804ed75`; host read-only, 2026-10-25)
- R1. The runner is `docker run --rm` (`src/drivers.rs:2088-2090`), launched by
`DockerRunnerDriver::docker` (`:2173-2195`) with stdin null and
`Command::output()`: stdout and stderr are each buffered whole in the
daemon's memory, unbounded. On a non-zero exit the error is
`"Docker runner exited with {status}: {stderr}"`; stdout is dropped. The
container is gone the moment it exits, so `docker ps -a` and `docker logs`
hold nothing.
- R2. The empty message: the failing process wrote nothing to stderr, and
stdout, the only stream that carried its report, is discarded at
`:2182-2193`. The same interpolation echoes whatever a step does write to
stderr into `last_error`, `pre_fencing_abort.error` and
`completion.error` (`bounded_detail`, 2048 bytes), and from there into
`idunn up`'s `deployment failed:` line and `idunn status`.
- R3. No step identity reaches the error. `materialize` (`:2368-2391`) loops
over `declaration.steps` and calls `run_in_workspace` with argv only;
`step.id` is used only in the environment precondition. External inputs
(`curl` in the runner, `:2247-2258`) fail through the same helper with the
same anonymous message.
- R4. Which step failed on 2026-10-25, inferred from the failed transaction's
staging root, which survives (926 MiB,
`/var/lib/gamecult/idunn/staging/tx-a04b68c7-.../`): in `.runner-node-build`,
`node_modules/.package-lock.json` was written at 00:51:24 (so
`install-vili-dependencies`, `npm ci`, finished), nothing outside
`node_modules` is newer, and `release/vili` (the next step's output) is
absent; the python workspace holds only fetched inputs. The failing step was
`test-daemon` (`node --test ... scripts/test-vili-daemon.mjs`), whose test
reporter writes to stdout: exactly the stream R1 discards. Inference from
residue, not from output.
- R5. The failure path: `resume_one_transaction` (`src/control_plane.rs:6291-6320`)
sends a pre-fence error to `begin_pre_fencing_abort` (`:9916-9935`), which
persists `bounded_detail(format!("{error:#}"))`. Nothing reaches the journal
on this path: the idunn-yggdrasil journal for 00:45-01:10 holds only
provider-exclusion lines.
- R6. Stores. `control.cc` (root 0640) holds commands, transactions
(`idunn.deployment_transaction.v4`, keys 0-36), generations, supervision and
expiry requests; `ControlSnapshot::read` refuses any other type
(`:3420`). Records must re-encode byte-exactly (`decode_record`,
`:3717-3727`), so a new transaction field is a schema bump and a v4 lift
(`:4100-4110`). Terminal transactions move verbatim to `history.cc`
(`:4366-4396`; 39.8 MB on the host), which status reads for `--command`.
`incidents.cc` lives in the published projection directory and by its own
doc (`src/incident.rs:1-11`) carries no free text and is never `control.cc`,
because a type added there stops continuity for every target on a rollback.
- R7. The daemon runs as root (`User=root`). Runner secrets reach the
container as read-only file mounts named by env (`:2041-2047`), so any text
a step derives from them can land in its output.
- R8. Staging residue: 475 transaction roots, 55 GiB under
`/var/lib/gamecult/idunn/staging`; a failed Sealing leaves its workspaces
because `materialize` returns before `remove_tree_inside`.
### Decision
The failure becomes a typed value at the runner, `RunnerStepFailure`: the
subject (recipe step or external input, by id), the runner id, the exit code or
signal, and the last 8 KiB of stdout and of stderr with each stream's total
byte count. Its `Display` carries the subject and status, never output.
Capture streams both pipes through bounded tails, so a verbose build no longer
buffers whole in the daemon.
The tail lives in a new private store, `runner-evidence.cc`, beside
`control.cc` (mode 0600), one record per transaction
(`idunn.runner_failure.v1`), newest 32 kept by the writer. Not in the
transaction record: that is a v5 bump whose rollback makes every older Idunn
refuse its control store (R6), the scar `incident.rs` already names. Not in
`incidents.cc`: published, and free text is barred there. Not in the journal:
journal readers are a wider set than root, so the journal line names the
transaction, the subject, the status and the store, and no output.
No redaction. Matching secret bytes would make the daemon read every secret,
and still misses any encoding, prefix or transform of one; a claimed redaction
that leaks is worse than a stated rule. The rule instead is the reader set: the
store's only readers (root) can already read every secret the runner was
given.
Keeping the failed container (dropping `--rm`) is worse: its output then lives
in Docker's json-file log, unbounded, untyped, outside Idunn's state and
readable by the docker group; containers accumulate with no owner to reap them;
`docker system prune` erases the evidence; and nothing ties a container to the
transaction in `idunn status`.
### Authority
Owner: the runner decides what failed and keeps the bounded tail; the engine's
pre-fence failure path is the one writer of `runner-evidence.cc`.
`last_error`, `pre_fencing_abort.error` and `completion.error` keep only the
failure's `Display`. Forbidden writers: the `stderr` interpolation in
`DockerRunnerDriver::docker`. Retention is by count inside the writer, so the
store needs no external owner; staging roots (R8) stay with
`ghostlight-verse:follow_up:idunn-generation-retention`.
### Deploy
Idunn's body changes only through gamecult-ops
`scripts/install-idunn-yggdrasil-release.sh` (stage `idunn` only; unit and
bindings unchanged). The next failed Sealing then shows its step and tail on
`sudo idunn status --command <id>`.

View file

@ -0,0 +1,61 @@
# Deployment provenance instead of an approval gate: target
Status: target, Self, 2026-10-19 (campaign `idunn-deploy-provenance` in
Eureka's mind). It states the ends. The map will live in
`deploy-provenance-map.md`.
## Why
Operator, 2026-10-19:
> The rule about the deployment brake was instituted by a cautious agent, but
> it doesn't actually suit how GameCult operates. The whole point of Eureka is
> to give the Mind, Persona and Self agency over its own Body. It's requiring
> my approval for every deployment that violates our principles. Yes, we
> certainly want to know which agent approved the deployment and why (this is
> the authority provenance which we might want to add to the transaction), but
> that's not the same as gating every deploy on my approval.
On the same day a StreamPixels deploy (tx-b40c7240) waited on the brake
because only the root-held operator identity could release it. The agent
running the deploy could neither proceed nor record why it should.
`F:\Projects\CLAUDE.md` says a deployment brake *may* gate artifact, revision,
configuration, schema, unit or authority changes. It does not say that
operator approval is the gate.
## Invariants
1. **No deployment waits on a per-transaction operator approval.** An agent
acting for the Mind, Persona or Self may admit a deployment of a target it
is authorized for.
2. **Every transaction carries typed authority provenance.** That is: who
approved it (agent or instance, and faculty), under which session or
ruling, and why. It is written before Fencing, it is immutable, and
`idunn status` and history show it. It is part of the transaction
document, not a log line.
3. **A brake that survives is a stop, not a gate.** The operator or an
authorized agent can halt deployment of a target, and the halt is itself
provenance-bearing. The deployment brake and the continuity lifecycle brake
stay separate (`F:\Projects\CLAUDE.md` Infrastructure). Neither gates Idunn
itself, and the deployment brake never gates continuity.
4. **Existing invariants hold.** These are `idunn-reincarnation` target r2,
`route-continuity-target.md` 1-7 and `authority-map.md`.
## Scope
In:
- The deployment-brake release path in Idunn and `idunn-provision`.
- The provenance record on the transaction.
- `idunn status` and history projection of the provenance.
- The gamecult-ops runbooks that describe a per-transaction operator release
(`heimdall-discord-launch.md`, `idunn-host-raven.md`, the StreamPixels
runbook).
Out:
- `idunn-reincarnation`. This campaign is mapped after that campaign's cut 2
merges, and coordinates with it rather than bundling.
- Harness permission rules on the workstation, which are the operator's.
- Heimdall's onboarding failures.

View file

@ -305,6 +305,45 @@ continuity restart does not fetch a branch, rebuild, rerun a migration, alter
the launch contract, or consume the deployment brake. An explicit lifecycle
brake may suspend this actuation without changing deployment authority.
Continuity never stops. Six charged attempts in an hour only open the
ContinuityExhausted incident (a page; the wire name stays, nothing is
exhausted) and gate no mint, wait or yield. When the next attempt is due is a
function of continuity's own record: a stored attempt streak, a backoff anchor
written when the pass first finds the admitted generation dead and the target
free, and a capped doubling wait from that anchor (5 s base, 300 s cap). A run
of 600 s or more by the admitted body resets the streak. A generation lost to a
host reboot (the workload driver's evidence, `lost_to_host_reboot`) is
re-materialized once per admission without charge or wait.
Each target has two slots, the incumbent and a transaction's candidate. The
incumbent is continuity's until a transaction takes the target, at Fencing (at
Starting for a stop-then-start rollout), through one `take_target` CAS; of two
transactions entering from one view exactly one succeeds. Before that, a
deployment owns only its private candidate and the target is free for it
between continuity attempts.
Phases Warming through Committing, post-fencing aborts and drains all carry
deadlines (the plan's frozen `PhaseDeadlines`; `WARMING_DEADLINE_SECONDS`, and
`PRE_FENCING_ABORT_DEADLINE_SECONDS` for a pre-fencing abort). What an expired
post-fencing deadline does depends on the owner. A deployment whose candidate
was never issued a write lease aborts to the incumbent. One whose candidate was
issued a lease may have written under a new state contract and is left to the
operator. A continuity retries its phase forward on a capped backoff and never
stops its live candidate, because that candidate is the only running copy of
the admitted body. Both operator cases open an `operator-required` incident
(subject: the transaction id) that closes when the record ends or its step goes
on.
The one yield: a continuity whose route proof has not succeeded gives way to a
deployment the deployment brake has released that awaits the handover of the
target. It begins its own post-fencing abort and the deployment fences what is
admitted. A continuity that has proven its route is serving never yields.
At boot, a lift returns to forward retry any continuity abort an older Idunn
left over a running candidate. If the lift errs or cannot observe the
candidate, the abort is held, not run. The lift is a migration with a deletion
line (follow-up `c2c-delete-boot-lift`).
An admitted generation keeps its Ready and latest Odin receipts as durable
proof of the exact activation at their recorded admission times. Dependency
selection reauthenticates those receipts at those times; continuity and route

View file

@ -44,7 +44,8 @@ Idunn centralizes that lifecycle work:
- record every deployment/restart request and result as typed state;
- record release targets, deployment artifacts, state migrations, and rollout
results as typed state;
- escalate to an operator through Bifrost when human action is needed.
- record operator incidents in its published `incidents.cc`, which Bifrost reads
and delivers. Idunn never waits on delivery.
The desired shape is simple:

View file

@ -82,8 +82,9 @@ running a script, and a dead `muninn serve` is seen the tick it dies.
2. No path on yggdrasil can start, stop or observe a host workload except
through the actuator session bound to that host's anchor.
3. A host with no attached actuator is a failed observation, which
continuity counts and gives up on after six restarts inside one hour,
exactly like a unit that will not start.
continuity counts as a charged attempt and retries on its capped backoff
(it never gives up; six in an hour only pages), exactly like a unit that
will not start.
4. The actuator cannot be told to run a program the recipe did not declare;
`allowed_programs` on the host-native runner is enforced on the host.
@ -96,7 +97,8 @@ Each of these was found by running the pipe against Raven, not by reading.
bind time. `rollout.strategy = "stop-then-start"` exists for exactly a
route-less host-actuator workload: the admitted incumbent is stopped at
Starting, before the candidate starts, and supervision leaves the target
alone from Starting on. A systemd-transient workload may not declare it.
alone from Starting on: the transaction takes the target there (one
`take_target` CAS) rather than at Fencing. A systemd-transient workload may not declare it.
- **An unobservable host is silence.** The hub returns a typed
`HostUnobservable` for not-attached, detached, reconnected and unanswered.
Supervision skips that tick and logs it. Before this, an Idunn restart

View file

@ -520,7 +520,7 @@ started, ran, and then died on a file it could not open:
| `runtime_root` and every directory above it | traversable by the workload: `o+x`, or group-owned by the state group with `g+x` | the workload is a `DynamicUser` and owns nothing, and `ReadOnlyPaths=` binds the bundle's leaf into the namespace without granting traversal above it. Idunn now checks the whole chain and names the closed directory |
| `runtime_root` | setgid, group-owned by the state group (`2750 root:<state_group>`) | `harden_root_authority_file` requires the write-lease record's group to equal its parent's, and only setgid makes a newly created record inherit it. Without it the record lands `root:root` and the workload cannot read its own lease |
| the write-lease record | `root:<state_group>`, mode `0640` | follows from the setgid parent; the workload reads it through the state group. Idunn writes it, so fixing an existing wrong-group record means deleting it, and that is safe only when no transaction is live |
| `/var/lib/gamecult/idunn-projection` | default ACL `u::rw-,g::r--,o::r--` | Idunn runs with `UMask=027`, so each publish creates the store and its `.lock` at `0640`. The code re-opens them after every write, but that leaves a window; a default ACL sets the mode at creation and closes it |
| `/var/lib/gamecult/idunn-projection` | default ACL `u::rw-,g::r--,o::r--` | Idunn runs with `UMask=027`, so a store is created `0640` and each publish re-opens it `0644`; the default ACL sets the mode at creation and closes that window for stores. The `.lock` is not published and no reader opens it: Idunn creates it `0600` (replacing one found wider, so a descriptor another uid already holds is orphaned), and the ACL never decides its mode |
| `/etc/nginx/idunn-stream-routes` | `root:root`, not group-writable | "route authority parent is not canonical root-owned and nonwritable". It was `root:idunn 0775` from the generation where Idunn ran unprivileged and needed to write it as a group member |
Two shapes are worth stating plainly because they are easy to get backwards.

600
docs/reincarnation-map.md Normal file
View file

@ -0,0 +1,600 @@
# Reincarnation after reboot: map
Status: map, Imagination, 2026-10-18 (campaign `idunn-reincarnation`, session
`idunn-reincarnation-2026-10-18`). The ends are in
[`reincarnation-target.md`](reincarnation-target.md). Questions, rulings, the
cut spec and follow-ups live in Eureka's mind; this page holds what has
no kind: body facts, the model page, the authority map's rationale and the
cut sequence. Line numbers are at `74a7008`; `src/` is byte-identical to
`c2477d4`, so the earlier anchors still hold.
Cut 2 pass, Imagination, 2026-10-19 (same campaign and session): body facts
B13-B18, the cut-2 rows of the model page and the cut-2 parts of sections 3-5
are anchored at `9a0a6a2`, whose `src/` is byte-identical to `ffb4293` (cut 1's
head). The B1-B12 anchors above stay at `74a7008`.
Read the in-force set with the rulings-in-force and open-follow-ups recipes
for root `idunn-reincarnation`; the cut in flight is the specs-with-no-report
recipe. The first spec (`cut-c1-durable-stop.r1`) was withdrawn when the
operator ruled that continuity never stops; its replacement is cut
`c1-never-stop`. Cited from `idunn-watchdog`: `exhaustion-close-rule-op` and
`paging-conditions`.
## 1. Body facts
Evidence: the `idunn-yggdrasil` journal from 11:40 UTC (94,670 lines), the
per-unit journals of the streampixels-service candidates and of Odin
(`idunn-odin-7cfe5cd1…`), `idunn status` and `idunn status --command <id>`
on Yggdrasil, `journalctl -u nginx`, and the source at `c2477d4`. All times
UTC, 2026-10-18.
**B1. The reboot did not stop continuity from re-materializing anything.**
Every streampixels-service continuity attempt minted a fresh transient unit
from the installed release `sha256-df663f0a…`: 54 continuity commands from
`continuity-85887ef2` (11:46) to `continuity-52bbbd66` (19:57), most listed by `idunn status` as
`continuity-… failed systemd unit idunn-streampixels-service-<hash>.service
is not running`. A continuity transaction is a full `DeploymentTransaction`
built by `from_continuity` (`control_plane.rs:1678`) from the admitted plan,
sealed release, installed release and Expected. It runs every phase:
Starting, Warming, Fencing, Leasing, AwaitingReady, Routing, Committing. The
absent admitted unit `461c6443…` was never the problem. Each new candidate
was.
**B2. Every candidate died of a dependency, not of itself.** The
streampixels-service process exits about 30 s after start when Odin does not
admit its initial presence. Unit `2bc02905…` at 11:47:21 logged "StreamPixels
service failed to enter its Idunn incarnation: Error: Odin did not admit the
initial Idunn runtime presence ... Timed out waiting for CultNet RUDP response
after 5000 ms." The same is logged by `42ecf363…` at 13:47:13, `03acf44d…`
at 19:47:14, `45276008…` at 19:49:54 and `599b705b…` at 19:51:18. Odin's
RUDP endpoint is `rudp://10.77.0.1:17871`, which nginx proxies. nginx failed
at 11:46:26 ("bind() to 10.77.0.1:17871 failed (99: Cannot assign requested
address)") and started at 19:46:24. Between 11:46 and 19:44 Odin's journal
shows no presence message from streampixels-service at all: none arrived.
After nginx returned, the first one arrived at 19:49:25 and was rejected:
"runtime presence has no exact current activation and provider anchor". The
attempts minted at about 19:53 and 19:55 got past Fencing and then failed
"the Leasing phase passed its deadline" (FailedAfterFencing, RestoreIncumbent).
Other early failures: "observed write lease is not the current Idunn grant"
(11:46:47, the first attempt after boot), and three "reading process stat
/proc/N/stat: No such file or directory" races.
**B3. Every attempt was charged at mint.** `record_restart(now)` runs inside
the mint CAS (`control_plane.rs:6156-6157`). Nothing reads a continuity's
outcome back into the log. The only exemption anywhere is `HostUnobservable`
(`:5944`), and it applies to an observation, not to a candidate. So the 6-per-
hour window (`CONTINUITY_RESTART_ATTEMPTS` = 6, window 3,600,000 ms, `:71-72`)
filled at 11:50:52 and refilled every hour as entries slid out. The bursts are
in the journal at :47, :48, :49, :51, :52 and :53 of every hour, 39 times
"Idunn stopped restarting admitted streampixels-service". This answers Q2:
the nginx outage counted against the body's budget, and the place that decides
it is the unconditional charge at `:6156`.
**B4. The deployment was killed by a window slide, not by a live
continuity.** `up-3ad6b05b` (release `sha256-67e227b2…`, a new release) was
recorded at 20:24:51. It was still before Fencing at 20:46:41, when it was
completed `FailedBeforeFencing` with "admitted incumbent failed before
candidate fencing; deployment yielded to continuity: systemd unit
idunn-streampixels-service-461c6443….service is absent". Two seconds later
continuity minted `tx-0804185a` with the OLD release `df663f0a…`, and it was
admitted. The restart that left the window was minted at about 19:46:40 (its
unit `03acf44d…` started at 19:46:42), exactly one window before the abort.
The mechanism, from the source:
- `:5805-5821`: a live transaction for the target is the `blocker`. From
Fencing on (or Starting for stop-then-start) supervision skips the target
entirely.
- `:6033-6061`: with a blocker present, continuity never restarts and never
reports. It may abort the blocker, and the predicate for that is
`Deploy && !supervision.restarts_exhausted(now) && phase < Fencing && no
abort && not held` (`:6045-6050`).
- `:6062-6084`: only with no blocker does an exhausted continuity open the
incident and say "The target is free for a deployment to replace it".
So from 20:24:51 the deployment owned the target (`owns_target_authority`,
phase != Complete, `:1772`). Continuity deferred to it through the blocker
`continue` at `:6060`. The deployment deferred to continuity whenever the
window had a free slot, through `:6046`. "Stopped" existed only as a durable
incident (incidents.cc, a projection) and a report. The ownership decision
read the window. When the window gained a slot, continuity reclaimed the
target with no record that it had resumed. This answers Q3: the two owners are
the live `DeploymentTransaction` (it owns while not Complete) and
continuity-as-window (it owns whenever the window is not full). Continuity
defers to the deployment while any transaction is live. The deployment defers
to continuity while the window has room. The "free for a deployment" message
names the first ownership, and the code acts on the second.
**B5. Fencing an absent incumbent already works.** `advance_fencing`
(`:6869-6952`) stops the exact incumbent and revokes its lease. The systemd
driver's `stop` returns `Ok(())` for an absent unit (`drivers.rs:4003-4007`).
A deployment that is not aborted therefore replaces an absent incumbent with
nothing to fence, which is target invariant 3's last clause, already true.
`exact_incumbent` (`:7556-7569`) fails "target incumbent changed after
transaction creation" if a continuity commit replaced the generation during
the deployment. That is why a phase-split handover needs a rebase
(`handover-pre-fencing`).
**B6. The two rejection streams are consequences of a dead admitted
generation, not separate faults.** "preserved admitted streampixels-service
after rejecting topology observation: topology correlation does not bind the
current activation" was logged 46,271 times, from 11:46:49 to exactly
20:46:48, about 5,300 an hour. Ghostlight's 42,482 identical lines ran to
19:56:29. Since 20:47, raven-muninn, newly exhausted, shows the same line
621 times. `refresh_admitted_topology` (`:6395-6457`) authenticates Odin's
correlation against the admitted generation's activation. When that
incarnation is dead and Odin's latest record names another one (a
continuity candidate's, or a stale one), authentication fails with an `Err`
rather than `Ok(false)`, and `:5834-5840` prints it on every pass. The pass
order is what does it: route supervision (`:5822`) and topology refresh
(`:5834`) run before the workload is observed (`:5891`). So a dead
generation is "supervised" every tick. "rejected admitted
streampixels-service route continuity: admitted process write lease is no
longer exact" (3,615, from 20:02:23 to 20:46:48) followed the two
post-fencing aborts at 19:54 and 19:57. Fencing revoked the dead incumbent's
lease, "restore incumbent" cannot re-grant a lease to a process that does not
exist, and `supervise_admitted_route` (`:6250-6255`) then refuses the
admitted generation on every challenge. Both streams stopped the moment
`tx-0804185a` was admitted. This answers Q4: both streams follow from one
model gap, that a dead admitted generation is treated as a live incumbent.
That gap is cut 4. They are not the deadlock.
**B7. Ghostlight is a separate fault.** Its continuity `tx-039b91e9` is
wedged in a post-fencing abort. It was "lease-not-adopted" at its AwaitingReady
deadline, and the abort withdraws the candidate's lease (`:8895-8928`) before
it stops the candidate (`:8929-8946`). The candidate unit `6c51e35e…` has
been running since 19:55:15 and holds the lease's shared lock, so every
attempt fails "process write lease is held by a shared-lock consumer"
(`drivers.rs:4179/4224`). The transaction owns the target, so continuity is
locked out and the route degraded (53 consecutive failures). This is follow-up
`ghostlight-abort-lease-before-stop` (route-continuity invariant 5), not the
stopped-owner deadlock.
**B8. Unexplained, recorded.** Idunn's record of Odin's stable route reads
"degraded-since 11:46:31, consecutive-failures 1, next-challenge-at 11:47:01"
about nine hours later. It has not been re-challenged since. This is follow-up
`odin-route-record-stale`. It blocks cut 3. Under `budget-never-stop`, a
failure attributed to a declared dependency must not grow the backoff, and the
only dependency-health record Idunn holds for Odin's stable route is this
one.
**B9. Luna's weekend commits on this path (Q5).** None of them caused the
deadlock. The yield predicate dates from the route-continuity and watchdog
campaigns, not from the weekend. Where they touch:
| Commit | On the path? | Status |
|---|---|---|
| `9f00e7a` replay provider receipts at admission time | Yes: the deployment's Sealing compile (`advance_sealing` → `current_ready_provider_tokens`) logged "excluded non-current provider ghostlight / raven-muninn: admitted provider's latest receipt is not Ready at its admission time" at 20:24:53 (`:8588`, `:8609`) | Still live: route-continuity B4 has not landed. It did not block this deployment |
| `29946bb` wait for the stable route listener | Yes: the route proofs during the outage were "Connection refused" on 127.0.0.1:8831-8833 | Kept by route-continuity (honest scope) |
| `8ae00a1` cancel live stateless candidates | No: no cancel ran | Still present (`:2888`); route-continuity S2 owns its deletion |
| `ae8ac45`, `8651906`, `e976d32`, `5e94eb8` identity and launch evidence | Only in `observe` and `stop`; nothing here drifted | Kept |
| `a148802` | No | Reverted (`87fa57d`) |
The target forbids building on them. Cut 1 touches none of their lines.
**B10. The stop was the only door a deployment had into a crash loop.**
Source read at `74a7008`. A queued `idunn up` becomes a transaction only in
`freeze_command`, which refuses while any transaction for any of its targets
`blocks_new_target_mutation` (`:5741-5747`); the command just stays queued.
The scheduler tick runs `retire_one_terminal_transaction` (returns early when
it retires), then `resume_one_transaction`, then
`supervise_one_admitted_generation`, then `freeze_one_queued_command`
(`:5414-5423`). Today's doubling wait is anchored at the previous attempt's
**mint** (`next_restart_at`, `:1163-1170`). So once an attempt lasts longer than
its wait, the tick after its transaction is retired mints the next attempt
before the freeze runs, and the queued deployment never finds the target
free. At `c2477d4` that ended when the window filled and continuity stopped.
Delete the stop without opening another door and a crash-looping release can
never be replaced. The door the cut uses is kubelet's own anchor
(`docs/research/restart-budget-prior-art.md` §1.1, the exited container's
`FinishedAt`): the wait runs from the first supervision pass that finds the
target dead **and free**. That leaves at least one wait (5 s or more) with no
live transaction, and the freeze runs every tick. This is reasoning over the
source, not a probe. The cut pins it with a tick-level test.
**B11. The yield cannot outlive the stop.** With the window term gone from
`:6046`, the yield aborts every pre-fencing Deploy whenever the incumbent is
dead (`:6045-6059`). For a crash-looping release the incumbent is always
dead, so the deployment that would fix it is aborted at the first
supervision pass after it is frozen. That is the 2026-10-18 failure: on that
day the operator's `idunn up` existed to replace a release that would not
start. Ruling `handover-phase-split` already says nothing yields before
Fencing. So cut 1 deletes the whole yield. Continuity keeps deferring to any
live transaction for its target (`:6033`, the `continue`). Cut 2 keeps
the incumbent alive alongside the build. Between them, a dead incumbent stays
down while a pre-fencing deployment over it builds. That is bounded by the
deployment's own deadlines, and the deployment then fences it as nothing
(B5).
**B12. The doubling function already exists once.** `backoff_wait`
(`:4000-4007`) doubles from a base up to `RESUME_BACKOFF_CEILING_MILLIS`, and
has one caller (`back_off`, `:5621`). Continuity's wait is a second spelling
of the same rule (`5_000 << (live.len() - 1)`, `:1169`). The cut gives
`backoff_wait` a ceiling parameter and deletes the second spelling.
`#[cultcache(key = N, default)]` is how an absent key decodes as its default
(precedent `DeploymentTransaction` key 34, `:1552-1557`). New
`TargetSupervision` keys use it, because every record on Yggdrasil predates
them.
**B13. Two live transactions for one target make the control store
unreadable.** Source read at `9a0a6a2`. `ControlSnapshot::validate` refuses
"multiple transactions claim current-incarnation authority for one target"
over `owns_target_authority` (`:3245-3254`), and that predicate is just
`phase != Complete` (`:1896-1898`). Every reader of the store goes through it:
the scheduler tick, `idunn status`, cancel, expire. So if supervision minted a
continuity beside a pre-fencing deployment today, the next read would fail
the whole engine. Coexistence needs the identity rule changed first, and that
is the cut's first commit.
**B14. One predicate answers four questions, and the phase rule lives in
one caller.** `blocks_new_target_mutation` (`:1903-1909`, live or post-commit
cleanup pending) is read by the freeze busy check (`:5934-5942`), the
supervision blocker (`:5998-6001`, a single `.find`), candidate-port
reservation (`:7997`) and status (`:4613`). `owns_target_authority` is read by
validation (`:3249`) and by sibling ordering inside one command (`:3384`).
Neither looks at the phase. The only phase-aware ownership rule is inline in
supervision (`:6002-6012`, owns from Fencing, or from Starting for
stop-then-start). The freeze refusal is why a `profile:` command needs every
target free in one tick (follow-up `profile-deploy-joint-gap`).
**B15. What a pre-fencing deployment reads about its incumbent.** Warming's
semantic warming check uses the incumbent's lease digest
(`:6904-6905`, `is_semantic_warming` `:9530-9560`: Odin reports the candidate's
one disagreement as the lease the incumbent holds). Route preflight uses the
incumbent's route observation (`:7026-7047`). Isolation is proven against the
incumbent's workload (`:7050-7076`). Each goes through `exact_incumbent`
(`:7768-7781`), which bails "target incumbent changed after transaction
creation" once a continuity commit replaced the generation. Sealing, and
Starting for candidate-then-promote, read no incumbent. Stop-then-start's
Starting stops it (`:6857-6863`), which is already the owner's act. The
Sealing-to-Starting write is a `replace_transaction` at `:6806-6808`, not
`transition` (`:8914-8927`), so the two edges at which a transaction can take
a target are written by two different functions today.
**B16. The commit already guards the incumbent; nothing guards the
handover.** `advance_committing` expects the admitted envelope it read
(`:7701-7702`, CAS `:7743-7764`), so a deployment can never commit over a
generation that moved under it. Entering Fencing is an unguarded
`replace_transaction` of the transaction alone. Continuity's mint CAS
(`:6371-6400`) guards its command, transaction and supervision record, and no
other transaction. The store accepts read-only expectations: an expected
entry with no replacement only guards (CultLib
`packages/cultcache-rs/src/lib.rs:1327-1368`). One engine holds the store
(`ProcessLock::acquire`, `:5514`), but the CLI writes transactions too:
`cancel` sets a pre-fencing abort on a live deployment (`:3014-3083`). So a
handover decided by one CAS over the target's records is exact, and a tick
order is not.
**B17. What coexisting candidates share, and what they do not.** Continuity
reuses the admitted plan, so its candidate port is the admitted generation's,
already reserved (`:7988-7992`); a deployment picks another. Topology is keyed
per incarnation (`:8495-8498`): a continuity publishes under the admitted key,
a deployment under its own. No candidate holds a write lease before its own
Leasing, which follows Fencing, so two pre-fencing candidates cannot both
write. Route actuation by a continuity is always `Survival`, never refused
(`:4870-4876`); a deployment's `Forward` cannot starve it. The Odin replay
floor `max_odin_sequence` (`:3399-3435`) spans every transaction of the target
and the admitted cursor; incumbent refresh and candidate warming already share
it during every deployment today, so the replay check (`:9498-9501`) gains no
new exposure. Stop-then-start is declared only by route-less host-actuator
bindings (`deployment.rs:756-764`, `:1245-1249`), and no control-plane test
builds one (grep `StopThenStart` in `src/control_plane.rs`: the definition
site only). It is live on Yggdrasil: raven-muninn declares it
(`git grep stop-then-start` in `gamecult-ops`:
`idunn/yggdrasil/bindings/raven-muninn.toml.in`). So the Starting edge of the
handover carries production traffic, and its engine-level proof is owed
before the closing install (follow-up `stop-first-handover-fixture`).
**B18. A pre-fencing error is an abort.** `resume_candidate` turns any `Err`
from a pre-fencing step into `begin_pre_fencing_abort` unless it is
`LockContended` (`:5768-5783`). So anything the cut makes a deployment wait on
must return `Ok` (a gate wait, `record_gate_wait` `:8170-8182`), or the wait
becomes the yield that ruling `handover-phase-split` forbids. The same path is
how a superseded continuity ends itself, which is wanted.
## 2. Model page
One row per persistent kind the campaign touches.
| Kind | What names it | Lifecycle | Who decides |
|---|---|---|---|
| Paging log: `TargetSupervision.continuity_restarts` (control.cc, `idunn.target_supervision.v1`, key 3) | key = target | The newest 6 charged attempt times, ascending, read through a sliding 1 h window. Outlives every generation. After cut 1 it is the paging threshold only (`restarts_page`): it opens and holds the ContinuityExhausted incident and is shown in status. It gates no mint, wait or yield | Continuity supervision, pushed in the mint CAS |
| Attempt streak (new, cut 1: key 7, `default`) | target | Charged attempts since the admitted body last ran a sustained run. +1 in each charged mint CAS. Reset to 0 when the dead generation was admitted after the last charged attempt and ran ≥ 600 s before it was found dead. Saturating, and the wait is capped, so it can be unbounded | Continuity supervision only |
| Backoff anchor (new, cut 1: key 8, `default`) | target | Written on the first pass that finds the admitted generation dead and the target free after an attempt. The next attempt is due at anchor + `backoff_wait(5 s, streak - 1, 300 s)`. Cleared by every mint. Stale, and ignored, once older than the admitted generation's `admitted_at` | Continuity supervision only |
| Re-materialization bound (new, cut 1: key 9, `default`) | target | Set by the one attempt that is not a backoff step, when the driver reports a reboot loss. Spent while it is not older than the admitted generation's `admitted_at` | Continuity supervision, from the workload driver's evidence |
| Projection deferral (keys 4-5) | target | Unchanged: a failed demotion defers the next mint 30 s. It spaces a re-materialization too | Continuity supervision |
| Admitted generation (`idunn.admitted_generation`) | key = target, `generation_id` = `generation-<tx>` | Replaced atomically by every committed deployment or continuity. A dead incarnation keeps its record until replaced | The committing transaction |
| Incident `ContinuityExhausted` (incidents.cc) | condition + subject (target) | Opened at 6 charged attempts in the hour while the target is dead and free. Closed when the target runs and the window no longer holds 6 (`exhaustion-close-rule-op`). Retired after retention. The name is the wire name and stays, though nothing is exhausted any more | Idunn's incident writer, a notification projection: never read by ownership |
| Transaction and pre-fencing abort (`idunn.deployment_transaction.v4`, history.cc once terminal) | `tx-<uuid>` and its `command_id` (`up-…` or `continuity-…`). Keyed by transaction, so a continuity and a deployment for one target are two records with no shared key | Sealing … Complete. A pre-fencing abort completes `FailedBeforeFencing`, a post-fencing abort `FailedAfterFencing` with a named recovery. Retired to history. After cut 2, per target at any instant: at most one live Deploy, at most one live Continuity, at most one owner (validated by `ControlSnapshot::validate`, B13) | The scheduler advances it. After cut 1, continuity supervision never aborts a transaction; the transaction's own steps do. After cut 2, a continuity whose target was taken, or whose source generation was replaced, aborts itself (superseded) |
| Target ownership (cut 2: derived, no record) | target | A transaction owns its target from `takes_target_at()` (Fencing, or Starting for stop-then-start) until Complete; a Complete transaction with pending post-commit cleanup drains it. Entered only through the handover CAS (`Engine::take_target`) | Derived from the phase. The handover CAS is the one writer of a phase entry into `takes_target_at()` |
| `incumbent_generation_id` on a transaction (key 9, unchanged format) | the generation's id | Deploy: set at freeze; before the handover it names the generation its incumbent-relative evidence was proven against and is rebased when the admitted generation moves (clearing that evidence); pinned by the handover CAS, exact from then on. Continuity: its source generation, never rebased | Deploy: its own pre-handover rebase step and the handover CAS. Continuity: the mint |
| Unit materialization (systemd transient unit `idunn-<target>-<hash>`) | the activation's hash | Created by Starting. Lost on a host reboot (`/run/systemd/transient`). A failed unit stays loaded; a cleanly stopped one is collected | The workload driver actuates. Whether a loss was a reboot is the driver's evidence (`WorkloadPort::lost_to_host_reboot`, cut 1) |
No cell is open. Both forks are ruled (`budget-never-stop`,
`handover-phase-split`). The numbers that `budget-never-stop` left open are
the defaulted ruling `continuity-backoff-defaults`. Cut 2 opens one question,
`fencing-over-owning-continuity` (does a deployment ready to fence wait for a
continuity that already owns the target, or preempt it); the spec is written
for the recommended answer, wait.
## 3. Authority map: continuity and deployment
The authority map for cut 1 is in its spec. This section is the
campaign-level map, and cut 1 is one step of it.
- **Owner.** Exactly one authority actuates a target at each instant. A live
`DeploymentTransaction` owns the target from Fencing on, and a stop-first
rollout owns it from Starting. Before Fencing (`handover-phase-split`), a
deployment owns only its private candidate, and continuity owns the
incumbent slot. Continuity owns the admitted generation's survival and
never stops. When it next acts is a function of its own record: the
streak, the backoff anchor and the re-materialization bound.
- **Inputs.** For continuity: the admitted generation (id, `admitted_at`,
workload, plan), the workload observation and the driver's reboot
evidence, its own record, and whether a live transaction blocks the
target. After cut 3 it also reads Idunn's dependency-health observation.
For a deployment: its own plan, its brake, and at Fencing whatever
generation is admitted at that moment (cut 2).
- **Outputs.** Continuity: mints, either charged or the one re-materialization;
backoff anchors; deferrals; and, as notifications only, the
ContinuityExhausted incident and its once-report. Deployment: a new admitted
generation, or an abort that names its recovery.
- **Derived state.** `restarts_page(now)` (6 charged attempts in the hour) is
derived from the paging log and is notification-only: incident open and
close, and status. The next attempt's due time is derived from the
anchor and the streak. The incident and `idunn status` are projections.
- **Forbidden writers.** The paging window gates no mint, no wait and no
yield. Continuity supervision never aborts a deployment:
`begin_pre_fencing_abort` has no caller in it. No deployment path writes
continuity's fields. The incident store decides nothing. No stopped state
exists: no field, and no branch that ends a pass because a count is
reached.
- **Shared paths.** A crash restart, a restart after a sustained run, and a
reboot re-materialization all mint through the one CAS (`:6158-6183`). The
dead-and-free pass is the only writer of the backoff anchor. Every
deployment, operator or scheduled, enters through `freeze_command` and
advances through `advance_transaction`.
- **Deletion line.** Cut 1 deletes the yield (`:6034-6059`), the
exhaustion `continue` and its "free for a deployment" text (`:6062-6085`),
and the mint-anchored wait (`:1160-1184`). Cut 2 (anchors at `9a0a6a2`)
deletes `owns_target_authority` and `blocks_new_target_mutation`
(`:1894-1909`), the inline phase rule and the single-blocker `.find` in
supervision (`:5998-6012`), the blocker `continue` (`:6229-6233`), the
freeze busy check (`:5934-5942`), the one-live-transaction-per-target
validation (`:3245-3254`), and the "target incumbent changed after
transaction creation" text (`:7779`).
### Cut 2: the two slots and the handover
Each target has one **incumbent slot**: its `AdmittedGeneration`, the body
running it, its write lease and its stable-route membership. Each live
transaction has its own **candidate slot**: its incarnation key, activation,
candidate port and unit.
- **Owner.** Before any transaction takes the target, continuity supervision
owns the incumbent slot: it alone decides to mint a restart. From the
handover CAS until Complete, the one transaction that took the target owns
it, fencing, leasing, routing and committing. Its commit hands the slot
back to supervision as a new generation. A transaction owns its own
candidate slot from mint or freeze until it is terminal. A pre-handover
deployment owns nothing else.
- **The handover.** `Engine::take_target` is the only writer of a phase entry
into `takes_target_at()` (Fencing, or Starting for stop-then-start), for
both kinds. It is one CAS. It expects the transaction's own envelope, the
target's admitted envelope (the one `incumbent_generation_id` names, or
absent), and every other non-terminal transaction of the target, each
unchanged, and it writes only the transaction. Of two transactions
entering from one view, exactly one CAS succeeds. The loser's next step
re-reads: a deployment gate-waits for the owner, and a continuity aborts
itself as superseded.
- **Inputs.** Continuity: as cut 1, plus whether any transaction owns or
drains the target, or a continuity is live for it (the mint gate). A
pre-handover deployment: its plan, its brake, and the admitted generation as
it is now, which it rebases onto. The handover: the target's records above.
- **Outputs.** The handover writes the new phase. A rebase rewrites
`incumbent_generation_id` and clears the evidence proven against the old
incumbent: `warming`, `route_preflight`, `isolation`. A superseded continuity
writes its own pre-fencing abort.
- **Derived state.** `owns_target()` is derived from the phase and
`takes_target_at()`. `drains_incumbent()` is derived from the post-commit
cleanup. The freeze refusal is derived from "a live Deploy for this
target". `incumbent_generation_id` on a pre-handover deployment is the
generation its evidence was proven against, not a pin.
- **Forbidden writers.** No path enters `takes_target_at()` except
`take_target`. Continuity supervision aborts nothing, and mints nothing
while another transaction owns or drains the target or a continuity is
live. A deployment never aborts because another transaction owns the
target: it gate-waits (B18). A continuity is never rebased. Freeze reads no
continuity state. No new persistent record decides ownership: it is the
phase.
- **Shared paths.** Candidate-then-promote's Warming-to-Fencing and
stop-then-start's Sealing-to-Starting go through one phase-entry seam,
which calls `take_target` exactly when the entered phase is
`takes_target_at()`. Deployments and continuities use the same CAS. A
single-target and a `profile:` command use the same freeze rule.
Demotions, named: `incumbent_generation_id` is no longer a pin before the
handover; it is derived from the admitted generation by the rebase.
`blocks_new_target_mutation` is no longer an owner; it splits into
`is_live`, `owns_target` and `drains_incumbent`, each read by the consumer
that asks that question. The freeze busy check is no longer a
target-ownership gate; it guards only one live deployment per target.
Demotions, named: `restarts_exhausted` is no longer an owner of anything. It
is renamed `restarts_page`, and it opens and holds the incident. The
ContinuityExhausted incident was never read as "stopped", and there is now no
stopped state for it to stand for. The "free for a deployment" report is
deleted: every target is free for a deployment between attempts (B10).
## 4. Rationale
The fault is a split brain, not a missing rule. Continuity's "I have stopped"
was written in two places, a durable incident and a log message, and neither
was what the ownership decision read: the yield re-derived "stopped" from a
clock. The withdrawn first spec moved "stopped" into control state. The
operator's ruling is better, and it is also smaller: there is no stopped
state. The window it was derived from becomes a pager. Every case the stop
existed for is then handled by an owner that already exists. A broken release
is replaced by a deployment, which gets in through the gap after each
attempt (B10) and is never yielded (B11). A host-dependency outage costs one
attempt every five minutes or so, and pages.
Why the wait is anchored at "found dead and free" rather than at the mint:
the anchor is the door. A mint-anchored wait can be swallowed by a long
attempt, and then no tick ever sees the target free (B10). Kubelet anchors
CrashLoopBackOff on `FinishedAt` for the same reason, so this is the
established spelling, not a new one. A streak is stored, not recomputed from
the paging log. The log holds only six entries and slides by the hour, while
the streak must outlive both. Reading the backoff from the window count was
today's spelling, and it is exactly the mutant the cut's tests must kill.
The reset, "admitted after the last charged attempt and ran at least 600 s
before it was found dead", needs no new writer. Supervision observes the
admitted generation on every pass, so the first pass that finds it dead is
within one poll of the death. A death hidden behind a live transaction or an
unobservable host overstates the run, and so resets early. That is the safe
side, because it shortens one wait.
The reboot invariant needs evidence of what happened, not a guess. "Unit
absent" alone is not a reboot, because a cleanly stopped transient unit is
collected too. "Host booted after admission" alone is not a loss, because
the unit might have been started this boot. The systemd driver owns both
facts (a `systemctl show` that returns nothing, and the `/proc/stat` btime).
So it answers one question, `lost_to_host_reboot`, and supervision keeps the
budget. The once-per-admission bound is still needed under never-stop. A
failing re-materialization leaves the same admitted generation, unit absent,
booted after admission. Without the bound it would repeat with no step, no
wait and no page, every tick. The clock comparison is safe: admission
happens at least one full transaction, tens of seconds, after boot.
What cut 1 deliberately does not fix. A dependency failure still grows the
streak (cut 3). A dead incumbent stays down while a pre-fencing deployment
builds over it (cut 2). A dead generation is still route-supervised and
topology-refreshed every tick (cut 4). Paging residual: the cap gives 3600 /
(300 + D) attempts an hour, where D is the attempt's duration, so a crash
loop whose attempts last longer than 300 s settles under 6 an hour. Its
incident opens in the first hour's burst and stays open while the release
never runs, but a release that runs for one pass after each restart can see
it close and not reopen. That is the operator's threshold, applied as ruled.
It is recorded here, not changed.
Replayed against 2026-10-18 with only cut 1. The reboot loss at 11:46 is
re-materialized at once, with no step. It fails on the nginx outage. The
next attempt resets the streak, because the generation had run for hours,
and then waits 5, 10, 20, 40, 80, 160 s, and 300 s after that. The incident
opens within the first hour and stays open (the release never runs), so the
operator is paged during the outage, not at 20:24. The `idunn up` at 20:24
freezes in the next gap. Continuity defers to it, nothing aborts it, and it
fences the absent incumbent as nothing and admits the new release.
Why cut 2 is shaped this way. The handover is the phase, guarded by one CAS
over the target's records, rather than a new per-target "owner" record. A
record would be a second spelling of what the phase already says, and it
would need its own lifecycle, crash repair and migration. A CAS that reads
every other live transaction of the target is enough because a newly minted
or frozen transaction starts at Sealing and cannot own anything (B16).
The rebase clears exactly the evidence that names an incumbent (B15): warming
names its lease, preflight its route and isolation its workload. Keeping any
of them would let the deployment fence one generation with proofs about
another. A deployment waits in Warming while another transaction owns the
target, because Warming's steps read the incumbent that owner is fencing, and
a failed pre-fencing step is an abort (B18). Sealing and Starting keep
running, so the long part, the build, still overlaps. A continuity that loses
the target is aborted, not parked: it restarts a generation that is about to
be replaced, and while it lives it blocks continuity of the generation that
replaces it.
Prior art on question `fencing-over-owning-continuity` (wait for a
continuity that already owns the target, or preempt it), from
`docs/research/restart-budget-prior-art.md` at `74a7008`. It leans the other
way from the recommendation, and the recommendation says why. Kubernetes'
rolling update scales old ReplicaSets down unhealthy pods first, so that
"unhealthy replicas will block deployment" does not happen (§1.4,
`rolling.go`). That is preemption of a failing old replica. Nomad separates
the two by layer: the client restarts old allocations under their own policy,
and the server reconciler places the new ones (§2.3). Neither waits on an
old replica's restart. But deleting a pod is the kubelet's ordinary teardown.
In Idunn, preempting a continuity past its own Fencing means a deployment
driving another transaction's post-fencing abort. That abort revokes a lease
and restores a projection, and it is the path that wedged Ghostlight (B7).
Waiting is bounded by the owner's own phase deadlines (`PhaseDeadline`,
`:938-960`) and adds no new abort path. Recommendation: wait, at medium
confidence. Preempt is the better answer once post-fencing aborts are known
not to wedge, which is follow-up `ghostlight-abort-lease-before-stop`.
Residuals, recorded and not fixed. A wedged post-fencing abort of a
continuity (B7, follow-up `ghostlight-abort-lease-before-stop`) holds a
waiting deployment as long as it holds the target. A continuity superseded
after its mint stays charged; that lengthens one wait for a generation that
is being replaced. Stop-then-start has no control-plane fixture (B17), so in
cut 2 its edge is pinned structurally (one seam, one call) and by a unit test
of the phase mapping, not by an engine run. raven-muninn uses it, so the
engine test is owed before cut 5.
## 5. Cut sequence
1. **c1-never-stop**: spec in the mind (specs-with-no-report recipe). Depends
on nothing.
2. **c2-phase-split** (`handover-phase-split`): spec
`cut-c2-phase-split.r1` in the mind, branch `idunn/reincarnation-c2` from
`9a0a6a2`. Continuity restarts the incumbent while a pre-fencing
deployment builds. The ownership predicates split (B13, B14). The handover
is one CAS at `takes_target_at()`. A pre-handover deployment rebases onto
whatever is admitted, and a superseded continuity aborts itself. Freeze
refuses only a second live deployment for a target, which dissolves
follow-up `profile-deploy-joint-gap`. It folds in the test-only follow-up
`c1-sustained-reset-test-reach`. Open question:
`fencing-over-owning-continuity`. Depends on cut 1 (merged at `1da150a`).
3. **Dependency failures do not grow the backoff**
(`budget-never-stop`): an attempt that failed because of a declared
dependency (Odin's stable route, the target's own stable route
listener), as Idunn itself observed it, leaves the streak unchanged.
Waits on follow-up `odin-route-record-stale`, because the record it
would read is not trustworthy, and on cut 1.
4. **A dead generation is not supervised as alive** (target invariant 5):
observe the workload before route supervision and topology refresh, so
that a dead admitted generation gets no topology authentication and no
lease-exactness challenge. Its non-binding correlation is reported once,
not every tick. Waits on cut 1, because it rewrites the same function.
5. **Closing proof on Yggdrasil**: an operator-approved install of the merged
cuts. streampixels-service survives the install, and `idunn status` is
checked before and after. Waits on 1-4 and on follow-up
`stop-first-handover-fixture` (raven-muninn is stop-then-start, B17).
Ops, out of scope: nginx boot order (`gamecult-ops`), and retention of the
roughly 130 dead transient units.
## Installed
Cut 1 (`1da150a`, merge of `idunn/reincarnation` at `ffb4293`) was installed on
Yggdrasil on 2026-10-19 at 08:34:45 UTC with the operator's OK. The binary is
`/usr/local/bin/idunn` sha256 `f0e58308…`, built in `/srv/build/idunn-1da150a`
(`cargo test --lib`: 414 passed). The pre-install store backup is
`/root/idunn-store-backup-*-pre-reincarnation-c1`. Restoring it means restoring
its stores and `idunn.running` together, because the cb60965 binary cannot read
`idunn.target_supervision.v2`. At first open Idunn logged
"migrated 5 control record(s) … idunn.target_supervision.v2". streampixels-service
kept running through the restart (unit 73ca40ba, PID 968752, NRestarts 0). The
catalog returned 200, and 4443/udp was listening.
## Installed: cut 2
Cuts c2-phase-split, c2b-owner-deadlines, c2c-post-fence-forward and
c2d-continuity-yields (branch tip `8760a4f`) were installed on Yggdrasil on
2026-10-20 at 00:34:06 UTC, with the operator's OK.
- **The hotfix before it.** `ec45906` (`bounded_detail`) was installed on its
own on 2026-10-19 at 17:32 UTC. It unwedged tx-f978ab86.
- **Build.** `/srv/build/idunn-8760a4f`; `cargo test --lib` passed 471.
- **Backup.** `/root/idunn-store-backup-*-pre-reincarnation-c2` holds the
stores, `incidents.cc` and the binaries. A rollback must restore them
together: older binaries refuse two live transactions on one target, and
they refuse the `operator-required` incident condition.
- **First boot.** The boot lift returned exactly tx-039b91e9 (ghostlight) and
tx-4eaed669 (odin) to forward retry, as the Soul install gate predicted.
Odin's continuity, wedged in Routing since the 2026-10-18 reboot, committed
4 s later. Generation tx-4eaed669 is admitted, its route is healthy, and the
same PID 2571 still runs. Ghostlight's continuity is retrying forward in
AwaitingReady. Its daemon's readiness is a ghostlight-side fault
(`Permission denied` reading its session-wrapping key).
- **Live processes.** Every live MainPID was the same before and after the
install. The streampixels catalog returned 200, and 4443/udp was listening.

View file

@ -0,0 +1,95 @@
# Reincarnation after reboot: target
Status: target, Self, 2026-10-18 (campaign `idunn-reincarnation` in Eureka's
mind). It states the ends. The body facts, model page and cut sequence will
live in `reincarnation-map.md`.
## Why
Yggdrasil rebooted at 11:46 UTC on 2026-10-18. Admitted bodies run as
transient systemd units under `/run/systemd/transient`, and the reboot wiped
them all. nginx failed at boot, because it bound `10.77.0.1:17871` before
`wg-gamecult` carried that address. Every stable route, Idunn's own
`127.0.0.1:8831-8833` included, stayed down until a manual
`systemctl restart nginx` at about 19:46.
What Idunn did:
- From 11:50 onward continuity spent streampixels-service's whole restart
budget (6 per hour) and logged "The target is free for a deployment to
replace it: systemd unit … is absent". The log repeated in hourly bursts as
the window slid.
- When nginx returned, ghostlight and streampixels-web were re-incarnated, and
streampixels-service was not.
- An operator-approved `idunn up streampixels-service` (tx-e69f5402) sat in
Sealing for about 15 minutes. It then failed with "admitted incumbent failed
before candidate fencing; deployment yielded to continuity". Continuity had
freed the target for a deployment. The deployment handed it back to
continuity. Neither actuated.
- At 20:46:43 continuity re-incarnated the service under generation
tx-0804185a, once the window had slid to 4/6. So the outage was not ended by
the deployment the operator approved. The restart window bounded it, a full
hour after its cause had cleared.
- Throughout, several times a second: "preserved admitted … after rejecting
topology observation: topology correlation does not bind the current
activation", and "rejected admitted streampixels-service route continuity:
admitted process write lease is no longer exact".
## Invariants
1. **A reboot is not a fault of the body.** If an admitted body's unit is
absent because the host lost its transient units, continuity
re-materializes it from the installed body. It needs no deployment, no
operator, and no part of the restart budget for that absence.
2. **The restart budget measures the body.** A failure caused by a shared host
dependency, such as nginx, the stable route's listener or Odin, never
charges a target's restart budget. A dependency outage can never exhaust
the budget of every target behind it.
3. **Exactly one authority owns each slot at every instant, and nothing
waits on an owner that has stopped.** Continuity never stops (operator,
2026-10-18: capped backoff, and the 6 per hour count only pages). It never
aborts a deployment. Before Fencing a deployment owns only its candidate,
and continuity owns the incumbent slot. At Fencing the deployment takes the
target and fences whatever is admitted then; an absent incumbent fences as
nothing. Every continuity attempt leaves a free gap in which a queued
deployment can be admitted. There is no reachable state in which
continuity waits for a deployment while that deployment waits for
continuity. A negative test proves it.
4. **Continuity is daemon survival; deployment is change** (`F:\Projects\CLAUDE.md`).
Restarting the admitted installed body never needs a deployment and is
never gated by a deployment brake. A deployment brake gates only artifact,
revision, configuration, schema, unit or authority changes.
5. **Rejections of stale observations are not a log stream.** An observation
that cannot bind the current activation is either resolved by the owner
that can refresh it, or reported once as an incident. It is never
re-rejected several times a second.
6. **Existing invariants hold.** `route-continuity-target.md` 1 to 7 (in
particular no actuation from observation, bounded backoff, every
post-fencing phase ends) and `authority-map.md` (Idunn's own recovery
depends on no managed target).
## Scope
In:
- Idunn's continuity of admitted bodies across a reboot: absent units, the
restart budget and what charges it.
- The handover between continuity and deployment, in both directions.
- Classifying the topology-correlation and write-lease rejections: the same
fault, or a separate one with its own cut.
- Re-admitting streampixels-service on Yggdrasil. It is running again under
tx-0804185a, and the closing proof checks that it survives the fix's
install.
Out, and not consumers of this change:
- **nginx's boot order on Yggdrasil.** It is host configuration, owned by
`gamecult-ops`. It gets its own operator-approved change and record, not an
Idunn cut.
- **Ghostlight's degraded route** (53 consecutive failures, incident open) is
in scope only if the map shows the same fault. Otherwise it is a follow-up.
- **Heimdall's onboarding failures and its environment.**
- **Retention of the ~130 dead transient units** on Yggdrasil. It is an ops
follow-up already recorded under route continuity.
- **Luna's 2026-09-27 weekend patches** are not foundation to build on. Where
this map meets them, it cuts them.

View file

@ -0,0 +1,124 @@
# Restart-budget and handover prior art (Eyes facts pass)
Serves two operator questions in Eureka campaign `idunn-reincarnation` (session idunn-reincarnation-2026-10-18):
- `idunn-reincarnation:question:budget-charge` (what charges a target's continuity restart budget)
- `idunn-reincarnation:question:handover-pre-fencing` (who acts when the incumbent dies while a deployment is before Fencing)
Facts and evidence pointers only. No recommendation. Gathered 2026-10-18 from official docs and upstream source (main branches at fetch time). Quotes are under 15 words. "Not found" means the primary source read does not state it, not that it is false.
## 1. Kubernetes
### 1.1 CrashLoopBackOff schedule, cap, reset (kubelet)
- Docs, pod lifecycle, "Container restarts": backoff is "exponential backoff delay (10s, 20s, 40s, ...)", "capped at 300 seconds (5 minutes)". Reset: after the container has run "for 10 minutes without any problems". https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
- Source pkg/kubelet/kubelet.go (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go): `initialCrashLoopBackOff = time.Second * 10`; max is `MaxCrashLoopBackOff = v1beta1.MaxContainerBackOff` (value lives in the kubelet config API package, not re-read; docs say 300 s). Under alpha gate `ReduceDefaultCrashLoopBackOffDecay`: initial 1 s, max 60 s. Gate `KubeletCrashLoopBackOffMax` lets the config set the max (`CrashLoopBackOff.MaxContainerRestartPeriod`).
- Reset rule in code: `klet.crashLoopBackOff.HasExpiredFunc` returns `eventTime.Sub(lastUpdate) > 600*time.Second` (kubelet.go ~line 1102). I.e. the entry resets when more than 600 s have passed since the entry was last updated (last restart event). The client-go default is 2*maxDuration; the kubelet overrides it.
- Growth: client-go `Backoff.Next` doubles (`delay := entry.backoff * 2`) and caps with `min(delay, p.maxDuration)` (https://github.com/kubernetes/client-go/blob/master/util/flowcontrol/backoff.go).
- Window anchor: the exited container's `FinishedAt`; keyed per pod+container; event text "Back-off restarting failed container" (pkg/kubelet/kuberuntime/kuberuntime_manager.go, `doBackOff`).
- The backoff table is an in-memory object created in `NewMainKubelet` (`flowcontrol.NewBackOff(base, boMax)`); no persistence found in the code read. Not found: any doc statement that a node reboot does or does not reset or count toward it. Docs only say restart policy "only applies to restarts that make replacement containers inside the same Pod and running on the same node." Docs on kubelet restarts: pods "continue to run even during the restart" (kubelet restart, not node reboot).
### 1.2 Probes
Docs (pod lifecycle "Container probes"; https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/):
- Liveness: past the failure threshold "the kubelet restarts that container" (per pod restartPolicy).
- Readiness: the EndpointSlice controller "removes the Pod's IP address from the EndpointSlices"; the container is not restarted.
- Startup: until it succeeds "Kubernetes does not execute liveness or readiness probes"; on failure "the kubelet kills the container", subject to restart policy.
- Backoff keys on a container having exited (`doBackOff` reads the last exited container), so crash, liveness kill and startup-probe kill all reach it; readiness failure leaves the container running and does not. The docs do not describe backoff per probe type.
### 1.3 Dependency down
Docs, same page, "Dependencies and init containers": init containers "run only once (if successful), during Pod startup"; failed init containers are restarted by the kubelet when pod restartPolicy is OnFailure or Always. Readiness gates: pod is ready only when all `readinessGates` conditions are True. No general documented rule for a main container whose external dependency is down: an app that exits is restarted under backoff; an app that stays up and fails readiness is only removed from endpoints.
### 1.4 RollingUpdate and the old ReplicaSet
- Deployment docs (https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): "Once new Pods are ready, old ReplicaSet can be scaled down further"; at the end "all old ReplicaSets is scaled to 0". maxUnavailable and maxSurge default 25%. progressDeadlineSeconds default 600; when exceeded, "Kubernetes takes no action on a stalled Deployment" beyond the Progressing=False condition.
- ReplicaSet docs: it fulfills its purpose by "creating and deleting Pods as needed to reach the desired number." https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
- Deployment controller source, pkg/controller/deployment/rolling.go (https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/deployment/rolling.go): `reconcileOldReplicaSets` computes `minAvailable = replicas - maxUnavailable`, runs `cleanupUnhealthyReplicas` first (comment: "Clean up unhealthy replicas first, otherwise unhealthy replicas will block deployment"), then `scaleDownOldReplicaSetsForRollingUpdate`. The Deployment controller decides old-RS replica counts during rollout, scaling unavailable ones down first.
- Layering: restarts of a dying container are done by the kubelet (restartPolicy with backoff), not the ReplicaSet. The ReplicaSet recreates a pod only when the pod object is gone or terminal. Not found: a statement in those words that the old ReplicaSet keeps recreating its dying pods until scaled down.
## 2. Nomad
### 2.1 restart block (https://developer.hashicorp.com/nomad/docs/job-specification/restart)
- `attempts` restarts allowed within `interval`; interval "starts when the first task starts". Defaults: service/system attempts 2, interval 30m, delay 15s, mode fail; batch attempts 3, interval 24h, delay 15s, mode fail. Delay has up to 25% jitter.
- `mode = "fail"` (default): after attempts are used the allocation is marked failed and goes to `reschedule`. `mode = "delay"`: "The task waits another full interval before restarting"; the allocation is never failed by restarts.
- Source client/allocrunner/taskrunner/restarts/restarts.go (https://github.com/hashicorp/nomad/blob/main/client/allocrunner/taskrunner/restarts/restarts.go): in-memory `RestartTracker` with `count` and `startTime` set at construction (`time.Now()`); a new interval resets count; `count > Attempts` returns TaskNotRestarting in fail mode, or TaskRestarting with delay to interval end in delay mode. Constructed in `NewTaskRunner` (task_runner.go ~line 468) from the alloc's job type and the task group's policy. `SetKilled` returns TaskKilled without counting. A restart triggered with failure=false (operator/API) returns TaskRestarting without consulting the policy. Unrecoverable start errors return TaskNotRestarting.
- Client restart: restarts.go keeps no persisted counter. `TaskRunner.Restore` (task_runner.go ~line 1244) restores local state and the driver handle (`RecoverTask`); if the handle cannot be restored for a non-system job the task "will not run until server is contacted" (`waitOnServers`). The tracker is rebuilt at task-runner construction. Persisted `TaskState` carries restart counts separately; `GetState` does not read them.
### 2.2 reschedule block (https://developer.hashicorp.com/nomad/docs/job-specification/reschedule)
- Applies when an allocation fails to set up or its tasks fail more than `restart` allows. Drains trigger migrations and lost nodes trigger replacements, so reschedule is ignored in both cases. Not applied to system/sysbatch.
- Service defaults: `delay_function = exponential`, delay starting 30s, `max_delay` 1h, `unlimited = true`. Batch defaults: 1 attempt per 24h, constant 5s, unlimited false. `delay_function`: constant, exponential (doubling), fibonacci. `max_delay` ignored for constant. Minimum delay 1 s; docs warn very low delay with `unlimited = true` can cause runaway rescheduling.
- With `unlimited = true`: "The `progress_deadline` in the `update` block still applies".
### 2.3 Deployments and restarts of old allocations
- Update block (https://developer.hashicorp.com/nomad/docs/job-specification/update): `max_parallel` 1, `min_healthy_time` 10s, `healthy_deadline` 5m, `progress_deadline` 10m ("the deployment is marked as failed"), `auto_revert` false, `canary` 0. The page says nothing on restarts of old allocations. Not found: docs on deployment/restart interaction.
- Restart is client-local ("Restarts happen on the client that is running the task"); deployment is server-side. Reconciler source (scheduler/reconciler/reconcile_cluster.go, `placeAllocs`): when the deployment is not place-ready (paused/failed/canarying), rescheduling of failed allocs still proceeds, with comment about avoiding "odd user experiences", except where the failed alloc is part of the failed deployment (`partOfFailedDeployment` skips it).
## 3. systemd
### 3.1 Start limit (https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html)
- A unit started more than `StartLimitBurst=` times within `StartLimitIntervalSec=` may not start again until the interval passes. Applies to all starts, including manual, not only `Restart=`; it is checked after condition checks, so failing conditions do not count. Units with `Restart=` that hit it are no longer restarted automatically; `systemctl reset-failed` flushes the counter. `StartLimitAction=` default none. Counters are cleared when the unit is unloaded.
- Source: `unit_test_start_limit` in src/core/unit.c (`ratelimit_below(&u->start_ratelimit)`; on hit logs "Start request repeated too quickly", returns -ECANCELED); `unit_reset_failed` calls `ratelimit_reset`.
### 3.2 Job result "dependency" and the start limit
- src/core/unit.c `unit_start` order: conditions (-ECOMM), asserts (-EPROTO), `unit_verify_deps` (-ENOLINK, "The necessary dependencies are not fulfilled"), then `UNIT_VTABLE(u)->test_startable`. For services, `service_test_startable` (src/core/service.c ~6381) calls `unit_test_start_limit`. A start failing on unfulfilled dependencies returns before the start-limit counter is touched.
- src/core/job.c: `-ENOLINK` maps to `JOB_DEPENDENCY` (`job_run_and_invalidate`); `job_fail_dependencies` also finishes dependent START/VERIFY_ACTIVE jobs with `JOB_DEPENDENCY` ("Dependency failed for %s."). `n_failed_jobs` increments only for JOB_FAILED, JOB_INVALID, JOB_FROZEN, JOB_CONCURRENCY. JOB_DEPENDENCY triggers `OnFailure=` handling.
- Requires=: if the required unit fails to activate and the dependent has After= on it, the dependent is not started; without After= it may still start. Wants=: failure "has no impact on the validity of the transaction as a whole." Dependent is stopped when the required unit is explicitly stopped or restarted; not stopped if the required unit stops on its own.
### 3.3 Restart= and dependency failure
- systemd.service(5) (https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html): the restart table is by exit cause of the service's own process (exit code, signal, timeout, watchdog, OOM); no row for dependency failure. Page: "service restart is subject to unit start rate limiting." Not found: a doc statement on whether Restart= acts on a dependency failure. From the code above, a dependency-failed start job finishes with JOB_DEPENDENCY and never passes through the service's own failure state (inference, not an explicit doc statement).
- Backoff: `RestartSec=` default 100ms; `RestartSteps=` with `RestartMaxDelaySec=` give exponential growth to a cap (doc example 10s, 20s, 40s, 80s, 160s, 160s...). `RestartMaxDelaySec` default infinity; both must be set. `RestartRandomizedDelaySec=` (v262) adds jitter.
### 3.4 Transient units across reboot
- systemd.unit(5) path table (man/systemd.unit.xml ~lines 403, 465): "Dynamic configuration for transient units" lives in `/run/systemd/transient` (system) and `$XDG_RUNTIME_DIR/systemd/transient` (user). That `/run` is volatile is not stated in this file.
- systemd-run docs: failed transient units are "kept in memory until the user explicitly resets their failure state"; `--collect` is `CollectMode=inactive-or-failed`. systemd-run and https://systemd.io/TRANSIENT-SETTINGS/ do not mention reboot. Not found: an explicit sentence on survival across reboot.
## 4. Other supervisors
### 4.1 Erlang/OTP supervisor (https://www.erlang.org/doc/apps/stdlib/supervisor.html, https://www.erlang.org/doc/system/sup_princ.html)
- `intensity` (MaxR) and `period` (MaxT), defaults 1 and 5. If more than MaxR restarts occur within the last MaxT seconds, the supervisor "terminates all the child processes and then itself" (exit reason `shutdown`). The parent supervisor then restarts it or itself terminates; failure escalates up the tree. Stated purpose: prevent a process from repeatedly crashing for the same reason and being restarted indefinitely.
- Restart types: `permanent` (always), `transient` (only abnormal exit), `temporary` (never). Every restart the strategy performs counts toward intensity; the docs describe no per-cause charging and no backoff delay. Multi-level: total restarts before the top gives up is roughly the product of intensities above the failing child. Strategy `rest_for_one` restarts the failed child and those started after it.
### 4.2 runit and s6
- runit `runsv` (https://smarden.org/runit/runsv.8.html): if ./run exits immediately runsv "waits a second" before restarting; no rate limit or retry cap described; keeps restarting unless told otherwise (`o`, `d`, `x`).
- s6-supervise (https://skarnet.org/software/s6/s6-supervise.html): "There is a minimum 1-second delay between two ./run spawns." A ./finish script exiting 125 makes supervise stop restarting; otherwise it restarts indefinitely. The give-up decision is delegated to a user script, not a built-in counter.
### 4.3 Borg (Verma et al., EuroSys 2015; https://research.google/pubs/large-scale-cluster-management-at-google-with-borg/, PDF text read)
- Borglet "restarts them if they fail"; Borg "monitors the health-check URL and restarts tasks that do not respond promptly" (sections 2.6, 3.3).
- Section 4 (Availability): Borg "automatically reschedules evicted tasks"; "limits the allowed rate of task disruptions and the number of tasks from a job that can be simultaneously down"; rate-limits finding new places for tasks from unreachable machines because it cannot tell machine failure from network partition; "avoids repeating task::machine pairings that cause task or machine crashes".
- Section 2.3 (updates): "Updates are generally done in a rolling fashion", with a limit on task disruptions an update causes; changes that would exceed it "are skipped".
- Not found in the Borg paper: a crash-loop backoff schedule, a restart budget, or any distinction between charging task faults and dependency faults. Omega paper not fetched.
## 5. Facts grouped by the brief's sub-topics
- (a) Charging only body faults: Kubernetes backoff keys on container exit regardless of cause (readiness failure does not exit and does not advance it). systemd runs dependency verification (-ENOLINK, JOB_DEPENDENCY) before the start-limit test, and JOB_DEPENDENCY is not in `n_failed_jobs`. Nomad's tracker counts any recoverable task failure or start error, with no dependency classification. OTP counts every restart. Borg avoids repeating task::machine pairings that cause crashes (placement, not budget).
- (b) Give up vs capped forever: OTP gives up and escalates. Kubernetes caps at 300 s with no give-up. Nomad restart fail-mode gives up locally, then reschedule continues (service default unlimited, exponential to 1 h); delay-mode never fails. systemd start limit stops auto-restart until the interval passes or reset-failed; `RestartMaxDelaySec` default infinity. runit restarts forever at 1 s; s6 restarts forever unless finish exits 125.
- (c) Rollout vs restart supervisor: Kubernetes splits by layer (kubelet restarts containers; ReplicaSet maintains pod count; Deployment controller scales old RS during rollout). Nomad splits by node vs server (client restart tracker vs server reconciler; reconciler skips rescheduling allocs of a failed deployment). Borg bounds update-induced disruptions with a rate limit.
## 6. Memory claims vs sources
budget-charge:
1. "Kubernetes separates readiness from liveness; only liveness feeds CrashLoopBackOff": CORRECTED. Readiness never restarts and never advances backoff (confirmed). Backoff keys on any container exit (crash, liveness kill, startup-probe kill), so it is not fed only by liveness.
2. "CrashLoopBackOff caps at 5 min": CONFIRMED (docs 300 s; initial 10 s, doubling). Reset: docs say 10 min healthy; code says 600 s since the entry's last update. 60 s cap only under alpha gate.
3. "systemd does not run a unit whose required dependency failed (job result 'dependency')": CONFIRMED with qualifier: Requires= plus After= blocks the start; Requires= without After= may still start; Wants= does not block.
4. "Nomad reschedules with capped exponential delay": CONFIRMED for service jobs (exponential from 30 s, max_delay 1 h, unlimited by default); batch default is constant 5 s, 1 attempt per 24 h.
5. Dependency failure vs start limit (not in the question's claims, material): systemd checks dependencies before counting a start (confirmed by code order). Whether Restart= acts on dependency failure: UNSUPPORTED by docs.
6. Whether restarts after a node reboot count toward CrashLoopBackOff: UNSUPPORTED by docs; code read shows the kubelet backoff map in memory. Nomad's restart tracker is likewise in memory and rebuilt on client restart.
handover-pre-fencing:
1. "In a Kubernetes rolling update the old ReplicaSet keeps recreating its dying pods until the rollout scales it down": PARTLY CONFIRMED, PARTLY CORRECTED. ReplicaSets do create pods to reach their desired count, and the Deployment controller scales old RS down, unhealthy first. Container restarts are the kubelet's, not the ReplicaSet's; the RS recreates only pods that are gone.
2. "Nomad restarts old allocations by their own restart policy, independent of the deployment": PARTLY SUPPORTED. Restart is client-local using the alloc's task-group policy (code), so independent of the server-side deployment; no doc statement on the interaction. Server-side, failed allocs are rescheduled even when the deployment is not place-ready, except allocs of a failed deployment.
3. Any source saying an aborted or stalled rollout automatically hands the slot back to the old generation: UNSUPPORTED (Kubernetes takes no action on a stalled Deployment; Nomad `auto_revert` defaults false).

View file

@ -211,6 +211,85 @@ The rework is a fresh branch from `583b2a7`, not a rebase of `1786ddf`. Its map
Option D (reset the restart log when a new release is admitted) was not chosen: a new release still inherits
its predecessor's restart history. B2 is merged (`46d9c2f`) but not installed; this follow-up (B2-3d) lands
before the install.
- **B2-3d status (Self, 2026-09-30).** Built on `hands/b2-3d` (`e9c718e`, `057bdd5`). Soul re-ran `verify.sh`
clean (341) and killed every one of its own mutants on the Forward/Survival split. **But the premise the operator
ruled on is false:** "every Survival path has its own bound" does not hold.
- **F1:** a continuity transaction whose route proof keeps failing retries inside one transaction under the
60 s resume cap, reloading nginx twice per retry. That is about 120 reloads/h per target, with no end until B5.
- **F2:** a failing post-fence withdrawal reloads about 60/h. `restore` reloads even when the fragment is
already correct.
- **F3:** nothing bounds reloads across the host.
- None of these is new in 3d: Survival was never refused before either. The live Idunn, which predates B2, has
no reload bound at all. What 3d adds is the false claim, at `control_plane.rs:76-81`, plus the admitted
Deploy doubling to 24/h.
- The fix belongs to B5 ("every post-fencing phase ends"), plus a restore/install that skips the reload when
the fragment is already correct. The operator was told on 2026-09-30.
- **B2 + 3d installed (2026-09-30).** The operator accepted recommendation A: install now, with B5 next.
- Installed at 15:22 CEST: `a96ad9d`, `idunn` sha256 `848c08e5...`; the replaced binary was `01bde676`.
- The migration moved 5 control records to `deployment_transaction.v4` / `admitted_generation.v4`.
- Pre-install backup, taken with Idunn stopped and byte-compared: `/root/idunn-store-backup-20260930T132204Z-pre-route-b2/`
(control.cc `5f879f55...`, history.cc `bafa76ab...`, the other stores, and the old binaries).
- Preflight: 690 transactions, all terminal. They were identical after the install.
- No managed target's PID changed. Raven reattached after 15 s.
- After 10 min: no odin route-continuity rejections (about 95 in the previous 3 h), 0 nginx reloads, and the
Odin post-deploy check `fails=0`.
- B5 and skip-unneeded-reloads are in Hands (`hands/b5`).
- **B5 Soul pass 1 (2026-09-30, on `a96ad9d..c06ff2b`): merge after the batch-2 fixes; no install before them.**
Soul reran verify: 352 passed. Batch 2, now in Hands, fixes:
- **A (high).** A continuity aborted from `Prepared` leaves a lease on disk that `driver.grant` wrote before
`Granted` was durable (`control_plane.rs:6916-6930`, `:9081`). The next fencing refuses, and the target is
wedged.
- **B (medium, a regression from `6ff7d5a`).** The reload skip hides a crash between the write and the reload on
the `restore`/`install` transaction paths, leaving nginx proxying to a dead port. The skip goes back to
supervision only.
- **E.** The resolver's command filter is unpinned.
- **F.** Odin-correlated adoption has no committed test; mutants M6 and M7 survive.
- **J.** A flaky backoff test (`:14200`).
Recorded:
- **C** (the abort has no deadline) is closed by the Q3-ii cut.
- **D.** There is no host-wide reload bound. 170/h is about a third of the F6 storm, and 4830af8 lowers the real
worst case.
- **G.** A newer Ready can rewrite `lease_adoption`. It names the same lease, so it is harmless.
- **H.** AwaitingReady and Routing keep the F16 shape. It is unreachable at today's heartbeats.
- **I.** An `idunn expire` request written while Idunn is stopped lingers, and there is no verb to withdraw it,
which is a downgrade hazard.
- **B5 MERGED and INSTALLED (2026-09-30).** Merged to main at `fcb8ba1` (Soul passes 1-3). Built on Yggdrasil:
`/srv/build/idunn-fcb8ba1`, 360 passed. `idunn` sha256 `e96bcefa...`, `idunn-provision` `127902ef...`, unit unchanged.
Installed at 22:35 CEST.
- **Backup**, taken with Idunn stopped and byte-compared:
`/root/idunn-store-backup-20260930T203515Z-pre-route-b5/`. It holds `control.cc` `fab5b66b...`, `history.cc`
`a2627ea4...` and `idunn.running` `848c08e5...` (a96ad9d).
- **Preflight** found one live transaction: the raven-muninn continuity `990dfc25` in AwaitingReady for about 40
min. Raven had rebooted, and its actuator timed out at 21:36 and 21:49 CEST. The operator ruled "Feel free to
install".
- **After install:**
- No managed target's PID changed. Raven reattached after 15 s, and nginx made 0 reloads.
- B5 aborted the stale continuity at its passed deadline, and supervision opened `continuity-26737b91`.
- A single log line, "could not resume transaction tx-bcfbb693...: transaction disappeared while checking
scheduler progress", appeared at 20:35:38 UTC. It looks like a same-tick ordering quirk between the
resolver and resume. It goes to the next Soul pass.
- **Rollback rule:** stop Idunn and check `control.cc` for any `idunn.expiry_request` envelope. If one exists,
restore the backup instead of swapping the binary.
- **Named tradeoff (Soul batch-3 F1):** a persistent `ufw delete` failure while aborting a first routed deploy now
leaves the aborted candidate routed until ufw recovers. Before B5 the cost was reload churn. Q3-ii's abort
deadline bounds it.
- Test pins F2-F4 are in Hands (`hands/b5-pins`).
- **B5 batch 2 (`c06ff2b..aef0da3`) and Soul pass 2 (2026-09-30): merge and install after batch 3.**
- A, B, E, F and J hold. Verify: 356 passed. cargo-mutants caught 14 of 14 viable mutants, and Soul's hand
mutants killed 16 of 20.
- Batch 3, in Hands: test pins for MB2 (`changes_route` swapped for "no incumbent"), ME2/ME3 (the resolver filter
by id length or first byte) and MJ2 (the computed backoff unpinned). `restore` withdraws the firewall rule
before reloading when there is no prior route, so a failing `ufw delete` no longer reloads nginx each retry.
- Hands corrected Self's brief: the continuity path's zero reloads came from `6ff7d5a`'s skip, not `4830af8`. The
narrow no-change case (`changes_route() == false`) keeps it without a write, so there is no crash window.
- Recorded, not in B5:
- **P1:** a crash between `atomic_replace` and the reload in supervision's `restore_admitted_membership`
(`drivers.rs:5543-5545`). Idunn has no SIGTERM handler, so a stop in that window leaves nginx on the drift
until something else reloads. This also exists in live `a96ad9d`. Fix with Q3-ii or a durable
"reload owed" marker.
- **The abort revokes before it stops the candidate** (`control_plane.rs:8758` before `:8783`). Q3-ii owns the
abort order.
- **Q-B2-1 (a):** the continuity restart log moves into `TargetSupervision`; the restart ceiling reads the
target's own log; the carry compensator is deleted.
- **Q-B2-2: fail at once.** A deploy refused by the route/actuation ceiling before the fence fails immediately,
@ -567,6 +646,37 @@ have written. (c) trusts a declaration over evidence.
Depends on it: B5, the recovery variants of the terminal record, and the
replacement for `idunn cancel`.
**Q3 follow-ups from B5 Hands (`hands/b5` `c06ff2b`), 2026-09-30:**
- **Q3-i: does an issued lease with no adoption evidence count as possibly
written?** **RULED yes** by the operator, 2026-09-30. A candidate cannot prove
it holds the lease while warming, and streampixels-service migrates in exactly
that window. So any issued lease on a deploy is treated as possibly adopted.
- **Q3-ii: after an adopted-lease deploy expires, what stops continuity from
restarting the incumbent on state the candidate may have migrated?**
**RULED 2026-09-30: D before Ready, E after.** Operator, verbatim: "D before
Ready, E after; this actually closes the hole, and is a nice feature for Idunn
to have".
- **D, before the candidate reports Ready:** after the fence and before the
write lease is issued, Idunn snapshots the target's state through a
per-target snapshot/restore declared in its binding. If the deadline expires
before Ready, Idunn restores the snapshot and brings back the incumbent. A
failed restore is the only terminal `OperatorRequired`.
- **E, after Ready:** the candidate is known-good and may have served traffic,
so an expiry rolls forward. The candidate stays the release supervision keeps
alive, and Idunn completes or retries routing and commit.
- Assumption Soul must falsify: the incumbent cannot write between the fence
and the lease, so the snapshot is exact.
- Mapped: section 4, "Q3-ii cut: snapshot before Ready, roll forward after" (Imagination, 2026-09-30);
it establishes the assumption for Idunn-managed writers and names what breaks it (Q3-ii.2).
History, the question as first put:
The operator on the offered hold-until-next-`idunn up` option, verbatim: "I
don't like any of these options, unless a stateful deploy running out of time
is an extreme edge case, in which case A". It is not extreme: the default
`awaiting_ready` deadline is 300 s (`deployment_plan.rs:474`), so any stateful
release that migrates and then fails to reach Ready lands here. Alternatives
are being put to the operator.
**Q4. The operator's recovery verb after `8ae00a1` is deleted.**
**RULED (b) by the operator, 2026-09-29.**
@ -1586,6 +1696,546 @@ admitted provider … Unrouted providers fall back to their Q1 class evidence".
- An unadopted lease resolves without an operator.
- Commit completes while Odin re-stamps every tick.
### Q3-ii cut: snapshot before Ready, roll forward after
Imagination, 2026-09-30. Nothing has landed. It maps the Q3-ii ruling, verbatim: "D before Ready, E after; this
actually closes the hole, and is a nice feature for Idunn to have". Q3-i is also ruled: any issued lease counts as
possibly written.
Anchors:
- Idunn `hands/b5` = `c06ff2b` (`src/` unless named; main's `src/` = `a96ad9d`, so B5 is the only code delta);
- gamecult-ops `0961976`, StreamPixels `a49f7d8`, Heimdall `e8e2832`, Ghostlight and Odin working trees (recipes read only).
"Probe" means one of:
- a read-only command on Yggdrasil, 2026-09-30 about 16:00 UTC (18:00 CEST); nothing was written to the host;
- a throwaway `postgres:16` container on Starfire, destroyed after the run.
"Read" means source at those heads.
#### Q3-ii.1 Body facts
- **S1. Fence order (read).** The candidate starts in Starting and runs alongside the incumbent through Warming
(`candidate-then-promote`).
- Fencing (`advance_fencing` `:6712-6797`) stops the incumbent and revokes its lease. It does this **only when the
incumbent holds a lease** (`:6719`, `:6733-6750`). The phase then moves to Leasing (`:6796`).
- Leasing's first write is `LeasingEvidence::Prepared` (`:6953-6963`); under Q3-i, Prepared already counts as issued.
- Ready is written in AwaitingReady together with `lease_adoption`: route-proof at `:7018`, Odin-correlated at
`:7069`. Routing only refreshes an existing Odin receipt (`:7118`). So `ready.is_some()` is monotone inside a
transaction, and it is the Ready boundary.
- **S2. An unleased incumbent is not fenced (read).** A stateful candidate replacing an incumbent without a lease
(for example the first deploy after a recipe declares state) leaves the incumbent running until post-commit
cleanup. `incumbent_was_stopped_during_fencing` (`:9058-9066`) derives "stopped" from the incumbent's lease
digest.
- **S3. Candidate writers before the lease (read).**
- **StreamPixels** loads the application, and with it `runPostgresMigrations`, only after `waitForWriteLease`
(`deployment/idunn/runtime-presence.mjs:58-59`).
- **Heimdall** builds its app, which runs `createStore`, and then `ensureSchema`, before any lease check
(`src/index.ts:18`, `src/store/index.ts:16-18`). It also writes `runtime-presence-sequence` into its state root on
every health pulse, lease or not (`src/index.ts:115`).
- Recipes declare CultCache-file slots and validate that writable slots open after the lease
(`deployment.rs:960-969`). **Postgres is declared nowhere.** `StateRecovery` and `MigrationDeclaration` are
parsed and never consumed by the control plane (grep).
- **S4. Out-of-band writers (probe).**
- `pg_stat_activity` on `streampixels` holds one session: pid 799973, which is unit
`idunn-streampixels-service-8c469c…`, the admitted Idunn workload.
- The `heimdall` database is held by the **legacy `heimdall.service`**, which Idunn does not manage (secret-files
map P8).
- No timer or cron touches either database.
- Operator scripts `apps/service/scripts/create-creator-invite.ts` and `grant-operator-role.ts` write
`streampixels` when an operator runs them.
- **S5. Sizes and times (probe).**
| Store | Size | Snapshot |
|---|---|---|
| `streampixels` database | 9.0 MB | `pg_dump -Fc` 137 ms, 48 KB archive |
| `heimdall` database | 38 MB | `pg_dump -Fc` 1,042 ms, 13.7 MB archive |
| streampixels-service state root | 12 KB | |
| odin state root | 21 MB | |
| ghostlight-world-v2 state root | 204 KB | |
| heimdall state root | 12 KB | |
Free disk is 1.6 TB. Postgres is 16.15, on local peer auth.
- **S6. State roots do not match their recipes (probe).**
- `ghostlight-world-v2/service/play-turn-v1.cc` (plus its `.lock`) is live and **undeclared** in the Ghostlight
recipe.
- Heimdall's state root holds `provider-health-identity.cc` (a private identity, mode 0600) and
`runtime-presence-sequence`. It does not hold its declared `heimdall.service.cc`; Heimdall is not admitted.
- The installed odin binding's `state_root` is `/var/lib/gamecult/odin`. The template
`idunn/yggdrasil/bindings/odin.toml.in` says `odin-v2`, which does not exist on the host. That is template
drift (ops follow-up).
- **S7. Postgres restore mechanics (probe, container).**
- In-place `pg_restore --clean --if-exists --single-transaction` **leaves tables the candidate created**: the
probe's `sessions` table survived. An in-place restore is not exact.
- A side database restored with `--no-owner --role=<owner>` and swapped with two `ALTER DATABASE … RENAME` in one
transaction block swaps atomically and leaves no candidate residue.
- `RENAME` refuses while another session is connected: it waits 5 s, then fails "being accessed by other users".
So the swap terminates the database's sessions first.
- **Fingerprint.** Take the sha256 of the archive's text (`pg_restore -f -`), with `--` comments, blank lines and
the per-dump random `\restrict`/`\unrestrict` lines removed. It equals the same fingerprint of `pg_dump` of the
restored database and of the source. Changing one row changes it.
- Neither live database has database-level ACLs, role settings or extensions (probe). pg_hba is `local all all
peer`, so root reaches Postgres through `runuser -u postgres` with no secret, as the backup producer already does.
- **S8. Supervision is blocked for a live transaction (read).**
- While any transaction of the target is at or past Fencing and not yet Complete, the target's supervision pass
skips it entirely (`:5684-5701`). That includes a running post-fencing abort, whose phase stays below Complete.
- After the terminal write, supervision repairs the Expected projection (`:5745`) and mints continuity (`:5987`).
- So a restore **inside** the abort cannot race continuity. A restore that fails and then goes terminal can: the
next mint restarts the incumbent on half-restored state.
- **S9. The abort path has no deadline (read).**
- `resolves_by_deadline` excludes `post_fencing_abort` (`:1568-1573`), and each abort step retries every tick.
- `route_restoration` retries `ufw delete` forever. B5's own test `:18394` pins 20 attempts and no end.
- The abort order is route, lease, candidate, topology, source (`:8717-8858`). A state restore placed after the
route step would inherit that livelock.
- **S10. Commit and route supervision need a promoted route (read).**
- `AdmittedGeneration::from_transaction` requires Committing, Ready and a routing disposition (`:2370-2418`).
- Route supervision bails without `RoutingEvidence::Promoted` (`:6049-6055`).
- Its repair `restore_admitted_membership` (`:6130`) installs from a membership digest and reloads only when the
fragment differs (`6ff7d5a`).
- **S11. Continuity re-admits stored plans under current rules (read).** `parsed_inputs` runs `binding.admit`
(`deployment_plan.rs:634-643`). A new admission rule must therefore be compile-time only, as secret-files §2
already records.
- **S12. Sandbox (read, and the installed unit probed identical).** Idunn runs as root with `ProtectSystem=full`.
`/var/lib/gamecult/idunn` and every target's state root are `ReadWritePaths`, so a snapshot directory under
Idunn's root needs no unit change.
#### Q3-ii.2 The exactness assumption (question 1)
The assumption is that the incumbent cannot write between the fence and the lease. It holds for Idunn-managed
writers under three conditions. The body breaks two of them today.
1. **Every live incumbent is stopped at the fence when the candidate is stateful.** Broken by S2. Fix: when
`expected.write_lease_required`, Fencing stops any exact incumbent, leased or not. `FencingEvidence::Revoked`
records `incumbent_stopped: bool`, and `incumbent_was_stopped_during_fencing` reads that field instead of
deriving it from the lease.
2. **The candidate writes nothing before its lease.**
- This is declared and validated for CultCache slots.
- It is not enforced for Postgres. StreamPixels complies. Heimdall does not: its schema DDL runs at start, and its
sequence file is written every pulse (S3).
- The fence cannot cover this. It is a target-contract fault, and the fixes belong in Heimdall (follow-ups H-S1,
H-S2 below).
3. **No out-of-band writer.** Today that means the legacy `heimdall.service` and hand-run StreamPixels operator
scripts (S4).
**What breaks when it is not exact:**
- A restore silently discards whatever an out-of-band writer committed after the snapshot.
- If the candidate migrated before the fence, the "incumbent's state" captured is already the candidate's.
**What closes it.** Condition 1 is closed in code. For conditions 2 and 3, the snapshot drivers check the no-writer
precondition at the store. They check at snapshot start, again after the capture, and again immediately before
Leasing writes Prepared:
- a Postgres store requires zero sessions on its database;
- a file store takes every captured file's `.lock` exclusively and non-blocking; a held lock means a live owner.
A failed check is a typed refusal inside Snapshotting. No lease is issued, so the deadline ends it as a plain abort,
with nothing to restore.
With the current body, this refuses every Heimdall deploy until H-S1 lands. That is the correct outcome.
One residual stays open: a writer that connects and disconnects between two checks goes unseen. Only Q-S3 (b) closes
that.
**Order in the phase machine after this cut:** Fencing (stop every incumbent, revoke) → **Snapshotting** (new) →
Leasing (Prepared = issued) → AwaitingReady (Ready durable = the D/E boundary) → Routing → Committing.
#### Q3-ii.3 Model page additions
| Kind | What names it | Life over time | Who decides |
|---|---|---|---|
| **Snapshot declaration** (`[state_snapshot]` in the operator binding) | the binding's target; frozen into the plan through `binding_blob` | Changes only by an ops binding change and a new Deploy plan. An old plan without it keeps running continuity (S11) | gamecult-ops template; Idunn validates shape at parse and coverage at compile |
| **Snapshot evidence** (transaction key 37, `state_snapshot`) | the transaction; one per Deploy past Fencing | Written once, when every store is captured and digested. Read by the restore. Terminal with its transaction | Engine (Snapshotting). Forbidden: CLI, drivers writing the record |
| **Snapshot bytes** (`/var/lib/gamecult/idunn/snapshots/<target>/<transaction_id>/`, 0700 root) | path derived from the transaction id | Rewritten whole on a Snapshotting replay. Deleted by the transaction's own terminal cleanup step, except under a state hold, where they are deleted when the hold is released. No janitor | the transaction that owns the path |
| **State restoration** (`PostFencingAbort.state_restoration`) | the abort | `Skipped` (no lease issued) → `Pending{deadline_at}` → `Complete{proof}` or `Failed{store, kind}` | abort path. Forbidden: supervision, CLI |
| **State hold** (`TargetSupervision.state_hold`, supervision v2) | target | Set in the same CAS as a terminal `OperatorRequired{RestoreFailed}`. Released per Q-S4. Never set by anything else | Engine at the terminal write; release per Q-S4. Forbidden: continuity mint, which only reads it |
| **Roll-forward record** (transaction key 38 `roll_forward` intent; completion `Admitted{recovery: RollForward}`) | the transaction | Set by the resolver after Ready. Terminal at the forced commit | deadline resolver |
#### Q3-ii.4 Snapshot/restore contract (question 2)
**Binding shape.** It is typed and closed; there are no command strings. Idunn accepts no imperative input: see the
unit's own comment and `cli_exposes_only_declarative_commands`.
```toml
[state_snapshot]
snapshot_seconds = 120 # optional; Idunn default 120. The Snapshotting deadline
restore_seconds = 300 # optional; Idunn default 300. The abort's restore deadline
[[state_snapshot.stores]]
kind = "state-slots" # the recipe's writable slots under workload.state_root (scope: Q-S2)
[[state_snapshot.stores]]
kind = "postgres"
database = "streampixels" # local socket, peer auth as OS user postgres; no URL, no credential file
```
- **Parse** (`deployment.rs`, `OperatorBinding` `:266-287`, new `StateSnapshotBinding`):
- duration bounds as in `DeadlineBinding` (`:734-750`);
- database names are identifiers;
- no duplicate stores;
- `state-slots` requires a `workload.state_root`.
- **Compile-time coverage** goes in `compile_deployment_plan` (`deployment_plan.rs:674-722`), never in `admit()`
(S11):
- a stateful recipe with a `state-slots` store must declare writable file slots;
- the no-declaration case is Q-S1.
- The durations live in the binding table and are frozen by `binding_blob`. `PhaseDeadlines` is untouched, so no
v3 plan changes its `plan_id`.
**Where the bytes live, and who owns and prunes them.** `snapshots/<target>/<transaction_id>/` under Idunn's own
root:
- slot copies keep their relative path, owner and mode;
- `postgres/<db>.dump` is 0600.
The **manifest** is not a file. It is the typed snapshot evidence on the transaction:
- per slot: relative path, size, mode, uid, gid, sha256, and whether it existed;
- per database: archive sha256, fingerprint (S7), owner role, and the zero-session check times.
Pruning:
- the transaction deletes its own directory in a new last cleanup step, on both `PostCommitCleanup` and
`PostFencingAbort`;
- under a state hold it is kept, and deleted on release.
The path is derived from the transaction id, so a crash leaves nothing a later step cannot find. gamecult-ops adds
`/var/lib/gamecult/idunn/snapshots` to the backup `EXCLUDES`. It duplicates live stores already captured, and it
can hold state-root private identities (S6).
**Budget.** The write pause (fence stop → lease) grows by the capture time. Measured today: about 0.2 s for
streampixels-service and about 1 s for heimdall (S5). The defaults leave two orders of magnitude of headroom.
**Secrets.**
- Idunn never opens a service credential file. Postgres is reached by peer auth, so the database URL files stay
unread; this keeps secret-files §1 "Idunn never opens a service credential for reading".
- `pg_dump`/`pg_restore` stderr can quote row data, and `idunn status` prints `last_error` verbatim (secret-files P7).
So no tool stderr enters `last_error`, `report_once` or the journal. The error carries a closed `SnapshotFault`
kind, the exit status, and the stderr's sha256. The full stderr goes to `snapshots/…/stderr-<step>` (0600), which
is pruned with the directory.
**Idempotence and crash safety.**
- **Snapshotting is repeatable while the fence holds.** A crash before the evidence write means the next tick
deletes the directory and captures again.
- **Restore is decided from observable state, never from a sub-step log.**
- Postgres: if the live fingerprint already equals the recorded one, the store is done. This covers a crash after
the swap. Else, if `<db>__idunn_restore` exists and its fingerprint equals the recorded one, terminate the
database's sessions, swap both names in one transaction, and drop the displaced database. Else drop and recreate
the side database from the archive.
- Slots: write each one by temp file and rename in its own directory; remove a slot file that did not exist at
snapshot time; leave `.lock` files alone.
- Replaying any prefix of these steps converges.
**How restore proves itself.**
- Postgres: fingerprint equality on the side database **before** the swap, so a bad restore never swaps. After the
swap, the live name resolves to the side database's oid.
- Slots: a re-walk equals the manifest byte for byte, including absences.
- The proof digests go into `state_restoration: Complete{proof}`.
**No declared snapshot:** Q-S1.
#### Q3-ii.5 Phase machine (question 3)
- **Snapshotting** is a new `DeploymentPhase` variant between Fencing and Leasing (`:206-216`, `Ord`-derived).
- `PhaseDeadline::entering` (`:913-934`) gains its arm from `snapshot_seconds`.
- `resolves_by_deadline` (`:1568-1573`) covers it through the existing range.
- It is entered only by a Deploy whose plan declares a snapshot. Continuity and stateless targets go Fencing →
Leasing as today.
- Expiry in Snapshotting → `Abort{restore: false}`. The partial directory is removed by the abort's cleanup.
- **Restore is a step, not a phase.** `PostFencingAbort` (`:1382-1400`) gains `state_restoration` and
`deadline_at_unix_millis`.
- `begin_post_fencing_abort` (`:8694-8712`) stamps the deadline. For a restoring abort that is `restore_seconds`;
for any other abort it is an Idunn default of 300 s (constant).
- **Order becomes lease → candidate → state → route → topology → source.** Restore needs only the candidate
stopped and its lease revoked, and S9's livelock must not hold state hostage. "Never two writers" is about
lease-then-process, and that order is preserved.
- **Every abort ends.**
- On the abort deadline, if `state_restoration` or any step before it is still Pending, the transaction completes
`FailedAfterFencing{OperatorRequired{RestoreFailed}}`. A restore that cannot even stop the candidate is a failed
restore.
- Steps after it (route, topology, source) are marked a new `CleanupEvidence::Abandoned`, reported once. The
transaction completes with its normal recovery.
- Abandoning those steps is safe because supervision owns their repair: Expected repair and demotion (`:5745`,
`:5943-5985`), and the admitted-route restore (`:6130`).
- This ends B5's `ufw delete` loop for every abort, not only D's.
- **The Ready boundary is `transaction.ready.is_some()`** (S1). `PhaseEnd` is rebuilt as follows. The B5 table test
is rewritten to it.
| Command | Ready durable | Lease issued (Prepared or Granted) | End |
|---|---|---|---|
| Deploy | yes | any | `RollForward` |
| Deploy | no | yes | `Abort{restore: true}` (snapshot evidence required by validation) |
| Deploy | no | no | `Abort{restore: false}` |
| Continuity | any | any | `Abort{restore: false}`, as B5 |
Continuity keeps B5's arm. Its candidate is the admitted release, it restarts under the restart log, and it never
changes the state contract. That is a default, not a question.
- **E replaces the abort after Ready. It is a forced commit, and it only writes the store.**
- The resolver persists `roll_forward = Some{cause, at}` and, if needed, `enter_phase(Committing)`.
`resolves_by_deadline` excludes a record with `roll_forward`, so it fires once.
- `advance_committing` (`:7206-7392`) with `roll_forward` set skips:
- the final route proof;
- Odin re-admission;
- the workload observation;
- lease currency.
- It then commits the generation from the durable evidence:
- Routing with no route receipt: a routed target gets a new `RoutingEvidence::Unproven{membership_sha256}`, the
digest of the rendered fragment. An unrouted target gets `SkippedUnrouted`.
- Committing with `Promoted`: kept.
- `Unproven` starts `route_supervision` degraded. Supervision (`:6037-6140`) accepts `Unproven`: it repairs with
`restore_admitted_membership`, proves, and on success rewrites the receipt to `Promoted`.
- A candidate that died after Ready is admitted anyway, and continuity restarts it under the restart log. That is
"the release supervision keeps alive", literally.
- **Boundedness.** The forced commit does no actuation. Every later actuation is supervision's: route repair spaced
by the challenge backoff up to `ROUTE_CHALLENGE_BACKOFF_CAP_MILLIS` (`:92`) and reloading only on a fragment
difference; process restarts bounded by the restart log (`:69-72`). No post-fence path retries forever.
- **D and `TargetSupervision`.**
- The restore runs inside the live transaction, so supervision cannot mint over it (S8).
- A `RestoreFailed` terminal sets `state_hold` in the **same CAS** as the completion (a three-envelope CAS like the
mint at `:5987-6031`).
- The mint path checks the hold right after `restarts_exhausted` (`:5916`) and mints nothing, with one report.
- Expected repair may still run; it starts no process. Deploy commands stay admissible (Q-S4).
- **Terminal records.**
- `FailedAfterFencing{recovery}`, where `recovery` is one of:
- `RestoreIncumbent`: D restored, or no lease was issued;
- `RestartAdmitted`: continuity, as B5;
- `OperatorRequired{reason: OperatorRequiredReason::RestoreFailed{store, kind}}`: typed. B5's free string
`reason` has no production writer (watchdog map I5).
- `Admitted{generation_id, recovery: Option<TerminalRecovery>}`, with `Some(RollForward{cause})` only there.
- Validation (`:2040-2075`, `:2180-2205`) pins each variant to its completion.
- **Schemas and lifts. One store-version install.**
- `deployment_transaction.v5` carries:
- the Snapshotting variant;
- keys 37 `state_snapshot` and 38 `roll_forward`;
- `FencingEvidence::Revoked.incumbent_stopped`;
- `PostFencingAbort.{state_restoration, deadline_at_unix_millis}`;
- `CleanupEvidence::Abandoned`;
- the completion shapes above.
- The v4 lift (the `read_transaction_record` pattern, `:3659-3704`):
- `incumbent_stopped = incumbent_lease_sha256.is_some()`;
- `state_restoration = Skipped`;
- an in-flight abort's deadline is stamped once on first sight (B5's rule for lifted phase deadlines,
`:8588-8596`);
- `Admitted.recovery = None`.
- History is lifted unchanged.
- `admitted_generation.v5` carries `RoutingEvidence::Unproven`. `target_supervision.v2` carries `state_hold`, lifted
as `None`.
- Every lifted fixture must re-encode canonically: the B2 F1 scar.
- An older Idunn refuses the v5 store, so rollback is the pre-install backup, as for B2.
#### Q3-ii.6 What B5 keeps, changes and deletes (question 4)
- **Keeps:**
- the resolver's shape (`resolve_phase_deadline` `:8568-8623`);
- `PhaseDeadline` and `enter_phase` as the only deadline writer;
- `idunn expire` and `idunn.expiry_request.v1`. Before Ready it now restores; after Ready it rolls forward, through
the same resolver;
- lease-adoption evidence, `lease-not-adopted`, and the continuity arms;
- the F16 commit fix and the reload skip.
- **Changes:**
- `phase_end` (`:1583-1596`), `terminal_recovery` (`:1598-1606`) and `resolves_by_deadline` (`:1568-1573`), to the
table above;
- the resolver's match (`:8600-8620`);
- `incumbent_was_stopped_during_fencing` (`:9058-9066`);
- `post_fencing_abort_intent` (`:9081-9110`);
- the abort order (`:8717-8858`).
- **Deletes (the "left running" arm):**
- `PhaseEnd::OperatorRequired` and its doc (`:943-945`);
- the `CommandKind::Deploy if issued` arm (`:1588`);
- the resolver's `OperatorRequired` branch (`:8610-8619`, the `deadline:<tx>` report);
- the "left to the operator" comment (`deployment_plan.rs:470-473`);
- the three `OperatorRequired` rows of `an_ended_phase_aborts_unless_a_deployment_candidate_was_issued_a_lease`
(`:12887-12889`); the test is rewritten to the Ready table;
- the string-reason test `an_operator_required_recovery_must_say_why` (`:12911-12935`), rewritten for the typed
reason.
- **Recommendation: merge B5 first, after its own Soul pass. This map is a follow-on cut** (Q-S5).
- B5 bounds the live continuity reload loop (B2-3d F1).
- Its left-running arm is no worse than today's resumable-forever post-fence.
- D and E carry two schema bumps and a new driver, and need their own falsification.
- Cost of the gap: a stateful deploy that expires after a lease stays live and reported, as today. `idunn expire`
re-enters that arm (watchdog I6). The StreamPixels-service redeploy in the ship sequence runs under that
behaviour.
#### Q3-ii.7 Cuts
Order: SR-1 → SR-2 → SR-3 → SR-4. The build and tests run on Yggdrasil (`ygg-verify.sh`) against the single `idunn`
crate. The build activates no new target.
- **SR-1. Snapshot drivers (Idunn `drivers.rs`, `deployment.rs`).** No phase-machine change.
- One port: `StateSnapshotPort { capture, check_no_writer, restore, prove }`.
- Two implementations:
- `SlotSnapshotDriver`: files, with `.lock` exclusivity;
- `PostgresSnapshotDriver`: `runuser -u postgres` for `pg_dump -Fc`, `pg_restore`, `psql`; side database,
fingerprint, swap. The program paths are injectable, as the route stubs are.
- The binding table parse and validation.
- The closed `SnapshotFault` kind and the stderr quarantine.
- **SR-2. D (Idunn `control_plane.rs`, `deployment_plan.rs`).**
- Deletes first: B5's arm (Q3-ii.6).
- Then:
- the Fencing change (Q3-ii.2 item 1);
- the Snapshotting phase;
- the restore step, abort deadline, reorder and `Abandoned`;
- the state hold;
- the compile-time coverage rule;
- all three schema bumps and lifts, including E's shapes, so the store moves once.
- **SR-3. E (Idunn `control_plane.rs`).**
- the `RollForward` end and the forced commit;
- `Unproven` in Routing and supervision;
- `Admitted.recovery`.
- **SR-4. Ops (gamecult-ops).**
- `[state_snapshot]` for `streampixels-service` (`state-slots` + `postgres streampixels`), `ghostlight`
(`state-slots`), `odin` (`state-slots`), and the Heimdall template (`state-slots` + `postgres heimdall`);
- the backup exclude;
- fixing the odin template's `state_root` drift (S6).
- **Owning-repo follow-ups (not in this campaign's cuts):**
- **H-S1** Heimdall opens Postgres and runs `ensureSchema` only after its lease.
- **H-S2** Heimdall moves `runtime-presence-sequence` out of its state root; it is runtime identity, not state.
- **G-S1** Ghostlight declares `service/play-turn-v1.cc` as a slot (under Q-S2 (a)).
- The legacy `heimdall.service` retires before Heimdall's first Idunn deploy (secret-files Cut 5).
#### Q3-ii.8 Authority map
- **Owner.**
- The deadline resolver owns how an ended phase resolves (restore, roll forward, or plain abort).
- The abort path owns the restore.
- The Engine's Snapshotting step owns the snapshot.
- **Inputs:**
- the plan's frozen binding (`[state_snapshot]`);
- the recipe's slots;
- `ready`, `leasing` and `fencing` evidence;
- the phase and abort deadlines;
- expiry requests;
- the stores themselves, through the port.
- **Outputs:**
- snapshot evidence;
- `state_restoration`;
- the terminal completion with its recovery;
- `state_hold`;
- an admitted generation with `Unproven` routing.
- **Derived state:**
- Q3-i's "possibly written" is derived from `leasing`;
- "stopped at fence" is derived from `incumbent_stopped` and no longer from the lease digest;
- the snapshot directory is cache-only, and its truth is the evidence digests;
- `report_once` text is notification-only.
- **Forbidden writers:**
- supervision writing restore or snapshot state, or minting over a held target;
- the CLI writing any of it (`expire` writes only its request);
- drivers writing records;
- the backup producer restoring anything;
- a route or topology step deciding the recovery.
- **Shared paths:** deadline expiry and `idunn expire` go through one resolver. Direct and Odin-correlated Ready use
one boundary field. D aborts and plain aborts use one abort path with one deadline rule. The forced commit and the
normal commit use one `from_transaction`.
- **Deletion line:**
- B5's `OperatorRequired` left-running arm;
- lease-derived fence-stop inference;
- the unbounded abort step.
#### Q3-ii.9 Verification
Each test pins a rule. Each must be killed by the named mutation.
| Test (behaviour) | Rule pinned | Mutation that must kill it |
|---|---|---|
| a deploy that expires before Ready restores the snapshot, and the incumbent restarts on the pre-lease bytes | D | restore marks Complete without writing; or `PhaseEnd` returns `Abort{restore: false}` |
| continuity mints nothing while the restore is Pending | S8 | widen the supervision skip range's lower bound past the abort |
| a failed restore is terminal `OperatorRequired{RestoreFailed}`, and supervision mints no restart | only terminal operator case; hold | drop the hold check, or set the hold outside the completion CAS |
| a deploy that expires in Routing after Ready is admitted `Unproven`, restores nothing, and supervision later promotes it | E | Ready-side arm returns Abort |
| an expiry in Committing rolls forward from the promoted receipt | E | forced commit rewrites the routing receipt |
| the Ready table: (Deploy, Continuity) × ready × (no lease, Prepared, Granted) | boundary and Q3-i | treat Prepared as unissued |
| `idunn expire` before Ready restores; after Ready it rolls forward | shared path | a separate expire branch |
| the fence stops an unleased incumbent when the candidate is stateful | exactness 1 | revert the `:6719` condition |
| a held slot lock, or a live Postgres session (fake runner), refuses Snapshotting with no lease issued | exactness 2/3 | skip either check, or check only at start |
| crash replay at every snapshot and restore step converges (persist the prefix, rerun) | idempotence | the restore trusts a sub-step flag over observation |
| the restore removes a slot file the candidate created | exact restore | copy-over only |
| an abort whose `ufw delete` fails forever ends at its deadline: state restored first, route `Abandoned`, recovery `RestoreIncumbent` | every abort ends | route before state; or Abandoned allowed on the state step |
| a v4 store (every fixture) lifts to v5, and supervision v1 to v2; each re-encodes canonically | schema | drop a field default |
| a stateful Deploy with no `[state_snapshot]` is refused at compile, and continuity of an old admitted plan still runs | Q-S1 (a); S11 | put the rule in `admit()` |
| a canary in fake `pg_restore` stderr appears in no `last_error`, report or `status` line | secrets | copy stderr into the error |
| Postgres integration, `#[ignore]`, run on Yggdrasil against a scratch cluster or container: side-database restore drops candidate tables; fingerprint mismatch refuses the swap; the swap terminates a live session; a crash between swap and drop replays | S7 | in-place `--clean` restore |
Then run `cargo mutants --in-diff`.
**Only a live Yggdrasil rehearsal, with the operator present, can show:**
- `runuser`/`pg_dump` reach the socket under the real unit sandbox;
- the snapshot directory is written under the real `ReadWritePaths`;
- a streampixels-service Deploy from a scratch branch whose candidate migrates and never reports Ready restores
`streampixels`, after which the incumbent serves again. Use a shortened `awaiting_ready_seconds` in a scratch
binding;
- the measured write pause;
- E: a Ready candidate whose stable-route proof is blocked is admitted `Unproven`, and is promoted by supervision once
unblocked;
- the backup run skips `snapshots/`.
#### Q3-ii.10 Subtraction ledger (estimate)
| | Production | Tests |
|---|---|---|
| Deleted | about -40 (B5 arm, comment, lease-derived stop inference) | about -30 (rows, string-reason test) |
| SR-1 drivers and binding | about +450 | about +300 (fake runners), +120 (ignored Postgres integration) |
| SR-2 D, lifts, hold, abort deadline | about +450 | about +450 |
| SR-3 E | about +180 | about +250 |
| SR-4 ops | about +20 of binding TOML, 1 backup exclude line | none |
- **Net:** about +1,050 production and +1,100 test lines in Idunn. It adds two schema bumps plus one supervision
bump, installed as one store move, and no new target, service or dependency (it uses the host's `pg_dump`,
`pg_restore` and `runuser`).
- **What it buys:** the ruled capability, which closes Q3-ii's hole, and an end for every post-fence path, including
B5's abort livelock.
- **What it retires:**
- the operator-only exit for stateful deploys;
- the `b5-operator-required-no-exit` follow-up (watchdog map).
#### Q3-ii.11 Watchdog interface (question 5)
The interface is a single point. The `operator-required` incident opens at the terminal CAS that writes
`FailedAfterFencing{OperatorRequired{RestoreFailed}}` and sets `state_hold`; its subject is the transaction id, as
the watchdog map models it. It closes when that hold is released (Q-S4). Nothing else in this cut opens or closes
it.
#### Q3-ii.12 Operator questions
**Rulings, operator, 2026-09-30:** "S1: Refuse / S2: Declared state / S3: Detect and refuse / S4: deploy or release". That is **Q-S1 (a), Q-S2 (a), Q-S3 (a), Q-S4 (a)**. Q-S5: "B5 first", which is **(a)**: B5 merges after its Soul pass, and SR-1..SR-4 follow as their own cuts.
- **Q-S1. A stateful target whose binding declares no snapshot.**
- Options:
- (a) Refuse a new stateful Deploy plan at compile, with a typed error. Continuity of already-admitted plans is
unaffected.
- (b) Roll forward even before Ready: no D for that target.
- (c) Keep B5's left-running report.
- **Recommend (a).** D is only a guarantee if it cannot be skipped by omission. (b) quietly reopens the hole this
ruling closed.
- Depends on it: SR-2's compile rule, SR-4's binding list, and whether Heimdall can deploy before H-S1.
- **Q-S2. What `state-slots` captures.**
- Options:
- (a) The recipe's declared writable slots only. The recipe is the authority on what is state. Undeclared files
in the state root are reported once and not restored.
- (b) The whole state-root tree.
- **Recommend (a).** It matches "the recipe declares; Idunn infers nothing".
- Under (b), Heimdall's `runtime-presence-sequence` would be restored backwards, and its presence sequence would
regress (S6).
- Cost of (a): Ghostlight must declare `play-turn-v1.cc` (G-S1). Until then, a failed Ghostlight deploy restores
everything but that file.
- **Q-S3. How hard the fence holds against out-of-band writers.**
- Options:
- (a) Detect and refuse: zero sessions and free locks, checked at snapshot start, after the capture, and
immediately before Prepared.
- (b) Also block: `ALTER DATABASE … ALLOW_CONNECTIONS false` from snapshot to grant. Idunn owns the undo on
grant, abort and crash replay.
- **Recommend (a).** Today's only out-of-band writers are the legacy Heimdall unit, which retires, and hand-run
scripts. (b) adds a database-configuration mutation that Idunn must always undo, and a missed undo locks the
service out of its own database.
- Depends on it: SR-1's scope and the residual in Q3-ii.2.
- **Q-S4. What releases a state hold after a failed restore.**
- Options:
- (a) A successful Deploy commit of that target, **or** `idunn release <target>`. The verb writes a typed request
the daemon consumes, as `idunn expire` does, after the operator has repaired the state by hand from the
retained snapshot.
- (b) A Deploy commit only.
- (c) The verb only.
- **Recommend (a).** A deploy is already an operator act that captures its own snapshot. The verb covers "I fixed
it by hand; bring the incumbent back" without forcing a redeploy.
- Depends on it: SR-2's hold, the retained snapshot's pruning, and the watchdog incident's closing event.
- **Q-S5. Merge order.**
- Options:
- (a) Merge B5 after its Soul pass; SR-1..SR-4 follow as their own cuts.
- (b) Hold B5 until D and E land.
- **Recommend (a)**, for the reasons in Q3-ii.6. Under (a), the StreamPixels-service redeploy in the ship sequence
runs with B5's report-and-wait on a stateful expiry.
### Odin-authority cuts (Imagination audit, 2026-09-30)
Source: the Odin-lifecycle authority audit (anchors Idunn `94c7691`/`583b2a7`, Odin `5c37860`), folded in as

View file

@ -52,12 +52,18 @@ web app and service through Idunn v2. The live evidence is on Yggdrasil:
the admitted incarnation. The proof answers an Idunn challenge and is signed
with the credential Idunn issued to that launch. Invariant 3 changes where the
proof travels, not what it proves.
5. **Every post-fencing phase ends.** Each phase after fencing has an admitted
deadline. When a deadline expires, Idunn resolves it: stateless targets abort
to the incumbent. Stateful targets get an explicit terminal record with a
named recovery. A granted lease that the process does not pick up within its
window is a named failure with an Idunn-owned recovery. It is never a silent
wait. Resumable-forever is gone.
5. **Every post-fencing phase has a deadline and is never a silent wait.** A
deployment's stateless phase that passes its deadline aborts to its
incumbent. A continuity past its deadline is left to the operator
(operator-required incident) and retries its phase forward on a capped
backoff, because aborting it would stop the only running copy of the admitted
body (ruling post-fence-forward-retry), until a deployment that the
deployment brake has released awaits the handover of its target and the
continuity's route proof has not succeeded: then it yields through its own
post-fencing abort, and the deployment fences what is admitted (rulings
continuity-yields-to-deploy, c2d-yield-only-unproven-to-movable). A
continuity that has proven its route is serving and never yields. A
stateful deployment is left to the operator.
6. **The admitted receipt describes the running incarnation.** Continuity
commits update the admitted generation atomically, so the rollback path never
meets drift between the receipt and the projection. Rollback demotes the exact

View file

@ -13,6 +13,56 @@ secrets into a transcript. The operator's words: "I'll rotate before prod, not t
concerned about Anthropic having my oauth; do map the cut that'll ensure it doesn't
happen again".
**Heimdall `hands/secret-files` status (Self, 2026-09-30).**
- **Batch 5** (`efd1137..b5d7b6b`) fixed Soul pass 4's four findings:
- F1: command-plane diagnostics are fixed text plus the error's class and code, and every provider body goes
through `readProviderJson`.
- F2: patron support keeps only the Bifrost status.
- F3: the recipe test records the secrets `loadConfig` actually reads.
- F4: `checkSchema` runs on every start.
- **Soul pass 5: hold.** F1-F3 held. Batch 6, in Hands, fixes:
- **S5-1.** `checkSchema` passes a DEFERRABLE key, a `UNIQUE ... INCLUDE`, a same-named non-unique replacement for
the partial unique index, and retyped columns. Each of these breaks the store at runtime.
- **S5-2.** The privilege check demands UPDATE on `audit_events`, which the store only INSERTs into.
- **S5-3 (older than this batch).** `app.ts:1310` writes a live, 300 s redeemable completion code into the audit
event `auth_completion_created`.
- **S5-4 to S5-7.** Test gaps: canary position, the `errorIdentity` filters, pool close, spread-env reads.
- A flaky live-Postgres test.
- Noted, not in scope: **S5-8**, forced RLS with no policy passes the check.
- **Batch 6** (`b5d7b6b..6c0c876`): S5-1 through S5-7 fixed, and the flaky live test fixed (template DB clone, `fixturePool`).
Result: 241 passed, 24/24 live, 40 of 41 mutants killed. The survivor, X13, is equivalent: pg-pool destroys a
failed client.
- **Soul pass 6: hold.** Batch 7 is in Hands.
- **F1.** An invalid or not-ready unique index passes the check. That breaks sign-in (42P10) and R21.1, because a
consume takes 2 rows.
- **F2.** A key index under a nondeterministic collation passes, and the upsert then moves one account's identity
row to another account.
- **F3.** The predicate normaliser is unsound: it strips parentheses and lowercases literals.
- **F4.** Some databases the store could serve are refused: indexes are required by name, and column-level grants
are not accepted.
- **F5.** The recipe exploration's all-at-once run stops early; mutants Z4, Z5 and Z6 survive.
- **F6-F8.** An inspected `cause` is not covered, a narrowed predicate has no fixture, and an added NOT NULL on a
nullable column is not caught. CHECK constraints and triggers are out of scope, as S5-8 is.
- **Batch 7** (`6c0c876..ca2366c`): F1-F8 fixed.
- Keys must be `indisvalid` with deterministic collations.
- Predicates are printed by Postgres, using a TEMP table in a rolled-back transaction, and compared as sets of
AND conditions.
- Index names are not required. Column privileges are checked with `has_column_privilege`. A nullable column must
stay nullable.
- Results: 241 passed, 39/39 live, 60/61 mutants killed.
- **Soul pass 7: merge.** Nothing blocks.
- 1400 predicate pairs were checked, and no two different predicates compared equal.
- The `indisvalid` argument held under interrupted concurrent DDL.
- TEMP is never refused for the real deployment, because the service role owns the database.
- **MERGED: Heimdall main fast-forwarded to `ca2366c` (2026-09-30).**
- Follow-up branch in Hands:
- S7-1: the recipe exploration never runs the production start shape.
- S7-2: extra unique indexes and similar documented as out of scope.
- S7-3: the column collation must be deterministic.
- S7-4: one more fixture each for Y3, Y4, Y5 and Y8.
- **Recorded:** a least-privilege column-subset grant is refused (F4(b)). That is theoretically servable, but it is
not how Heimdall is deployed.
**Rulings and defaults, 2026-09-30 (Self).**
- **Q3 A is the operator's ruling:** "I'll rotate before prod". Cuts 1-4 land with the current values; rotation
overwrites the credential files later. The token-encryption and signing-key sub-question waits for rotation time.

View file

@ -0,0 +1,352 @@
# Watchdog reporting: map
Campaign `idunn-watchdog`. Questions, rulings, cut specs, reports and verdicts
are typed documents in Eureka's mind; this file keeps only body facts, the
model page and rationale. Target rationale: `watchdog-reporting-target.md`
(`4951125`).
## Body facts
Source reads are at Idunn `4951125` (main), Idunn `hands/b5` (`0c80a44`,
unmerged, read with `git show`/`git diff main...hands/b5` only), Bifrost
`42728a6`, Odin `cdc57bd`, VoidBot `46d891b`, gamecult-ops `14df7ce`. Host
probes ran on Yggdrasil over `ssh ygg`, read-only, 2026-09-30 13:51-13:55 UTC
(15:51-15:55 CEST).
### Idunn
| # | What | Where | Result |
|---|---|---|---|
| I1 | Alert path | `src/control_plane.rs:3798-3812` (`ReportOnce`), `:4599-4610` (`report_once`), `:5114-5138` (`note_fault`, `clear_fault`) | The only operator signal is `eprintln!`. De-dup is `fault_reports: Mutex<BTreeMap<String, ReportOnce>>` (`:4392`), in memory: an Idunn restart forgets it and reprints. `note_fault` also writes the transaction's `last_error`. |
| I2 | Continuity exhaustion (since `idunn-reincarnation` cut 1: `restarts_page`, a paging threshold only; nothing stops, and the line numbers below are pre-cut) | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:<target>` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. |
| I3 | Route degradation | `RouteSupervisionState` `:890-942`; `record_failed_challenge` sets `degraded_since_unix_millis` on the **first** failed challenge, `record_proved_challenge` clears it; reported at `:5975` under `route:<target>` | Durable per incarnation (inside `admitted_generation.v4`); a new generation starts from `default()`. Backoff doubles from the topology max age to a cap. |
| I4 | Other operator-facing faults | `:5645-5651` (`generation-down:<target>`, "holds a dead admitted generation down"), `:6016` (`route-record:<target>`), `note_fault` per transaction | Also stderr plus `last_error`. Not named by the target; listed so the paging-set question sees them. |
| I5 | `OperatorRequired` on main (pre-cut-2 reading; since `idunn-reincarnation` cut 2 `resolve` reaches it in production through `PhaseEnd::OperatorRequired` and `PhaseEnd::RetryForward`, both calling `leave_to_operator`, which opens the `operator-required` incident `IncidentCondition::OperatorRequired`, subject the transaction id; `close_spent_operator_incidents` closes it when the record no longer `leaves_to_operator`) | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. |
| I6 | `OperatorRequired` on B5 (superseded by cut 2: B5's deadlines merged as `PhaseDeadline`, and a continuity past its deadline now also lands here) | `hands/b5` `resolve_phase_deadline` (diff hunk at `control_plane.rs` ~8560-8625 on the branch), `PhaseEnd::OperatorRequired` | Deadline passed on a stateful deploy with an issued lease: `report_once("deadline:<transaction_id>")`, returns `Ok(None)`, the transaction stays live. No terminal transition and no operator action that ends it were found on the branch (`idunn expire` writes `idunn.expiry_request.v1`, which re-enters the same arm). |
| I7 | Store kinds | `:55-61`, `:936-966`; B5 adds `idunn.expiry_request.v1` | `control.cc`: `idunn.deployment_command`, `idunn.deployment_transaction`, `idunn.admitted_generation.v4`, `idunn.target_supervision.v1`. `history.cc` beside it holds terminal transactions (`:3653`), retired one per tick (`:5068-5091`). |
| I8 | Single-writer precedent inside `control.cc` | B5 `ExpiryRequest` doc comment | "The CLI only writes this record. The daemon's deadline resolver reads it ... No CLI writes a transaction field." One writer per kind. |
| I9 | Serve loop | `:5041-5058`, `run_scheduler_tick` `:5098-5110` | 500 ms poll. A tick error is logged, never propagated. |
| I10 | `idunn status` | `render_supervision` `:4215` | Renders restarts used/limit and `degraded-since`; no incident notion. |
| I11 | Idunn already talks CultNet to targets | `src/drivers.rs:5567-5600` `request_runtime_presence_at` | `CultNetMessage::SnapshotRequest { schema_ids, record_keys }` over the route transport (http/tcp/rudp) to a target endpoint. Precedent for reading a typed record out of Odin's catalog. |
| I12 | Idunn reads Odin's file read-only | `src/drivers.rs:5005-5020` (`receive`), unit `--odin-correlation-store /var/lib/gamecult/odin/topology.cc` | Absent file yields `None`; continuity never consults it. |
| I13 | Idunn's published surface | unit `deploy/idunn-yggdrasil.service` comment on `ReadWritePaths` | "the topology store is Idunn's *published* surface, and every managed target must read it". Lives in `/var/lib/gamecult/idunn-projection`, outside Idunn's 0750 root. |
| I14 | Rejected predecessor | `git show 821a8d0^:src/main.rs:2020-2060, 8206-8250` | Before the 2026-09-02 rebuild, Idunn wrote an `IdunnOperatorAlarmRecord` (`alarm_id = alarm:<daemon>:<now>`, one per decision) and ran an operator-supplied shell `--operator-alarm-command` synchronously with a timeout, passing `IDUNN_ALARM_*` env. The rebuild deleted it. |
| I15 | Stale guide lines | `docs/guide.md:47` "escalate to an operator through Bifrost"; `:328` "Idunn is not Bifrost ... Discord delivery, owner DMs" | `:47` describes a capability that does not exist; `:328` stays true. |
### Yggdrasil
| # | Command | Result |
|---|---|---|
| Y1 | `systemctl is-active idunn-yggdrasil bifrost bifrost-persona-mouth bifrost-persona-feedback` | all `active` |
| Y2 | `systemctl show idunn-yggdrasil -p User -p ProtectSystem -p ReadWritePaths` | `User=root`, `ProtectSystem=full`. `ReadWritePaths` names no Bifrost path, but `ProtectSystem=full` only makes `/usr`, `/boot`, `/etc` read-only, so root Idunn can physically write `/srv/bifrost` and `/var/lib/gamecult/bifrost`. The `/srv` and `/var/lib` entries are documentation, not a boundary. |
| Y3 | `ls -la /srv/bifrost/env` | `persona-delivery.env` **exists**: `root:root 0600`, 135 bytes, 2026-09-04 02:20. gamecult-ops `inventory.md:741` ("the file is absent") is stale. |
| Y4 | `sudo awk -F= '{print $1, length($2)}' persona-delivery.env` (values not printed) | `BIFROST_DISCORD_BOT_TOKEN` 72 chars, `DISCORD_OWNER_ID` 18 chars. The token and the recipient binding are both already in Bifrost's env. |
| Y5 | `systemctl cat bifrost-persona-mouth` | `User=bifrost-feedback`, `ProtectSystem=strict`, `EnvironmentFile=-/srv/bifrost/env/persona-delivery.env`, `ExecStart=node .../tools/persona-feedback.mjs serve-persona-delivery ...` with signed-request RUDP listener `10.77.0.1:17876` and Starfire permit requester `rudp://10.77.0.2:17877`. |
| Y6 | `systemctl show bifrost-persona-mouth -p ActiveEnterTimestamp -p NRestarts` | Started 2026-08-21 10:51:22 UTC, 0 restarts: **before** the env file was written, so the running process has no token loaded. |
| Y7 | `readlink -f .../persona-feedback/runtime/current`; `grep -c discord-dm .../cultmesh-bridge-commands.mjs` | Current release `cb3239a` (2026-08-08). It ships `cultmesh-bridge-commands.mjs`, `operator-notification.mjs`, `bifrost-bridge.mjs`, but `c063f27` (discord-dm verb, 2026-09-04) is **not** an ancestor: the deployed pump cannot send a DM. `discord-dm` count 0. |
| Y8 | `find /srv/bifrost /var/lib/gamecult/bifrost -name provider-store.cc` | None. The pump's store has never existed on the host. |
| Y9 | `getfacl /var/lib/gamecult/idunn-projection`; `ls -ld /var/lib/gamecult/idunn` | Projection dir `root`, `other::r-x`, default `other::r--`: world-readable. `/var/lib/gamecult/idunn` is `idunn:idunn 0750`; `control.cc` is unreadable to other users. |
| Y10 | `id idunn`; `getent passwd bifrost bifrost-feedback` | `idunn` 986/978; `bifrost` 111/113; `bifrost-feedback` 994/983. No shared group. |
### Bifrost
| # | What | Where | Result |
|---|---|---|---|
| BF1 | Alarm publisher | `tools/operator-notification.mjs:41-148` | Opens Bifrost's `.bifrost/provider-store.cc` (repo-relative, `:20`) and `put`s a `bifrost.bridge.discord_post_command.v1` `discord-dm`. The **caller** supplies `recipientId` (`:51-54`). `commandId = idunn-alarm-sha1(alarm JSON)[0..16]`, and the JSON includes `raisedAt`, which defaults to now (`:48`): a re-publish after a restart mints a new id. |
| BF2 | Pump | `tools/cultmesh-bridge-commands.mjs:50-71, 133-221` | One-shot `process`: selects `pending` and `running`, marks `running`, spawns `bifrost-bridge.mjs`, writes `discord_post_receipt.v1` and `completed`/`failed`. `failed` is terminal (no retry). A crash after the Discord POST and before the receipt leaves `running`, which the next run resends. |
| BF3 | Discord send | `tools/bifrost-bridge.mjs:735-770` | Opens the DM channel, posts `content`. No `nonce` is sent (`grep nonce` hits only the Persona permit tools). |
| BF4 | No caller authentication on the command path | BF1, BF2 | Whoever can write the store file can make Bifrost post anything to anyone. |
| BF5 | Authenticated precedent in Bifrost | `tools/persona-discord-delivery.mjs:27-30`, `tools/persona-discord-permit.mjs` | Persona crossing: signed requests, Epiphany-signed permits, "permit request intent persisted before network" in an execution journal. |
| BF6 | History | `git log -S publish-idunn-alarm`: `755649f` (2026-06-07), `c063f27` (2026-09-04) | `c063f27`: "an alarm is a command addressed to the gate, not a document dropped into somebody else's state." |
### Odin and VoidBot
| # | What | Where | Result |
|---|---|---|---|
| O1 | `odin.store_condition.v1` | Odin `docs/store-recovery-cut.md:158, 199, 318-325` | Not built. One record, key `topology`, held in Odin's in-memory W, rewritten on change, **flushed with the store** and served by Odin's catalog. While `contested` Odin writes no file, and while `unpersisted` it cannot: in exactly the conditions that matter, `topology.cc` on disk does not carry them. Only the catalog does. |
| O2 | I-F1, I-F2 | Odin `docs/store-recovery-cut.md:496-500` | I-F2 is this campaign's continuity-exhaustion row. I-F1 (hold deploys on a bad condition) is a deployment-brake decision, not reporting. |
| V1 | Peer precedent | VoidBot `packages/core/src/bifrost-discord-command.ts:39-68` | The caller opens Bifrost's store, derives `commandId` from its `idempotencyKey`, optionally spawns the pump itself, and **waits** for the receipt. An unconditional `put` of an existing id overwrites a `completed` command back to `pending`. |
| V2 | VoidBot `notify_owner` | Self's mapper, not re-probed | Not live on Yggdrasil; no Discord-capable VoidBot process runs there. |
### Cut-mapping probes (2026-09-30, 14:00-14:10 UTC)
Pinned heads: Idunn `487f4f14` (origin/main), Bifrost `42728a67`, gamecult-ops
`14df7ce4`, all read with `git ls-remote origin refs/heads/main`. Line numbers
in Idunn are unchanged from `4951125`, because `153af6a` and `487f4f1` touched
docs only.
| # | What | Where / command | Result |
|---|---|---|---|
| C1 | `control.cc` refuses unknown types | Idunn `src/control_plane.rs:2845` | `_ => bail!("Idunn control store contains a foreign document")`. A new type in `control.cc` makes every older binary unable to read the store at all, so a rollback of the incident cut would take continuity down for every target. |
| C2 | Idunn's store writes take a blocking exclusive lock; reads can skip it | CultLib `packages/cultcache-rs/src/lib.rs:737-785` | `pull_all_read_only_snapshot` reads unlocked. Writes are `fs2::lock_exclusive`, which blocks. A reader that held the lock could stall an Idunn write. |
| C3 | The TS store takes no lock | CultLib `packages/cultcache-ts/src/single-file-messagepack-backing-store.ts` (no `lock` anywhere); present at Bifrost's pinned CultLib `f67f5122` (`pullAll` at `:44`) | A Bifrost reader using `pullAll` cannot block Idunn. |
| C4 | A JS reader decodes an Idunn-written store | `scp ygg:/var/lib/gamecult/idunn-projection/topology.cc`; scratch `probe-read.mjs` with cultcache-ts `pullAll` | 18 envelopes decoded (`gamecult.service_trust_anchor.v1`, ...). No `.lock` file was created beside the copy. |
| C5 | Rust `DatabaseEntry` payload shape in JS | scratch `probe-decode.mjs`, `@msgpack/msgpack` `decode(envelope.payload)` | A **positional array**: index `n` is `#[cultcache(key = n)]`. The Bifrost reader decodes a tuple, so the seam must be pinned by bytes Idunn wrote, not by a hand-built object. |
| C6 | Projection publish mode | Idunn `src/drivers.rs:4385-4404`, `:7670-7685` | Publication is compare-exchange, then `chmod 0644` on the store and its `.lock` sibling. |
| C7 | Continuity's healthy site | Idunn `src/control_plane.rs:5631-5633` | `if observation.is_some() { continue; }`: the admitted workload was observed running this pass. The exhaustion decision is `:5709`, and the fault clears at `:5723`. |
| C8 | Test harness for exhaustion | Idunn `src/control_plane.rs:17659-17689` | Existing tests use `routed_world(Odin::Unreachable, 1, ...)`, `set_meters`, `routed.workload.kill()` and `routed.tick()`. |
| C9 | Idunn has no one-shot or timer workload | `grep -i "oneshot\|\.timer\|OnCalendar" src/` returns nothing; recipe schema `src/deployment.rs:1935-2000` | A target is a resident service with `[service]`, health contract and state slots. Idunn cannot deploy a systemd timer. |
| C10 | No Bifrost binding in current Idunn | `sudo ls /etc/gamecult/idunn/bindings/` on ygg | ghostlight, heimdall, odin, raven-muninn, streampixels-service, streampixels-web. No Bifrost. |
| C11 | Bifrost's old Idunn manifest is residue | `/srv/odin/deploy-manifests/bifrost-persona-feedback` on ygg (2026-09-04); `grep deploy-manifests\|IDUNN_ACTUATOR src/` in Idunn returns nothing | It requires `IDUNN_ACTUATOR=1` / `IDUNN_COMMAND_AUTHORITY=idunn-daemon` env from the pre-rebuild Idunn, which no longer exists. gamecult-ops `runbooks/bifrost-persona-feedback-yggdrasil.md` still describes that path. The persona-feedback `current` has pointed at `cb3239a` since 2026-08-08. |
| C12 | How Idunn itself is installed | ygg `/usr/local/sbin/idunn-a96ad9d-build.sh`; `ls -la /usr/local/bin/idunn` (2026-09-30 13:22) | One hand-written script per commit. It runs `cargo test --lib` and `cargo build --release` in a pinned rust image under `/srv/build/idunn-<sha>` and copies the unit. The install is manual. No such script is in any repo. |
| C13 | Legacy alarm consumers in gamecult-ops | `scripts/idunn/notify-idunn-operator-alarm.{ps1,cmd}` (25 + 2 lines); `scripts/idunn/start-idunn-local.ps1:10,134-136` | The only callers of `publish-idunn-alarm`. They pass `--operator-alarm-command` to the deleted pre-rebuild Idunn (I14). No Idunn scheduled task or process runs on Starfire (`Get-ScheduledTask`, `Get-Process`). |
| C14 | `discord-dm` consumers in Bifrost | `grep discord-dm` | `cultmesh-bridge-commands.mjs:152,227` exist only for `operator-notification.mjs` (BF6). `bifrost-bridge.mjs` `discord-dm` (`:48`, `:473-535`) is also used by `docs/bridge.md:156` and `tests/Bifrost.Web.Tests/BridgeCliTests.cs:481`, so it stays. |
| C15 | Bridge CLI contract | Bifrost `tools/bifrost-bridge.mjs:1266-1289`; `tools/persona-discord-delivery.mjs:45` | The bridge refuses to act without `--cultmesh-command-id`. The Persona precedent spawns it with `--receipt-store` under its private state directory and parses stdout JSON. |
| C16 | Crash-recovery precedent | Bifrost `tests/persona-discord-delivery.test.mjs`, "running journal recovers as terminal unknown without a second Discord post" | A `running` execution found at start becomes terminal `unknown` and is never re-posted. |
| C17 | Discord nonce | docs.discord.com/developers/resources/message (fetched 2026-09-30) | `nonce`: integer or string, "up to 25 characters". `enforce_nonce`: "checked for uniqueness in the past few minutes"; on a duplicate from the same author, "that message will be returned and no new message will be created". |
| C18 | Bifrost tests and mutation tooling | Bifrost: no root `package.json`, `node:test` files under `tests/`, `grep stryker` returns nothing | Tests run with `node --test`. StrykerJS is not installed. How the tests find CultLib: C23 (the tests ignore `VOIDBOT_CULTLIB_ROOT`). |
| C19 | Idunn verification | `scripts/verify.sh`; eureka `tools/stopgap/ygg-verify.sh` (rust image carries cargo-mutants) | `cargo check` for the Windows GNU target twice, then `cargo test --locked --lib`. |
| C20 | Idunn source mentions no transport | `grep -rin "discord\|bifrost" src/` at `487f4f1` | No hits. This is the negative-grep baseline. |
### Bifrost suite probes (2026-09-30, 15:00-15:30 UTC)
Pinned heads: Bifrost `42728a67` (origin/main) and `hands/watchdog-bifrost-retire`
`4619939e`; CultLib pin `36ea08d3` (origin/main `016df462`); Epiphany origin/main
`4d1113ef`. Probes ran in a scratch clone, never in `F:\Projects\Bifrost`. Suite
runs went through `ygg-verify.sh` in `node:24.14.1-bookworm`, with CultLib cloned
to `/CultLib`, the sibling of `/src`, then `npm ci` and `npm run build:ts`. Each
file ran alone with the TAP reporter, then the whole glob ran once.
| # | What | Where / command | Result |
|---|---|---|---|
| C21 | Bare `cultcache-ts` require sites | Bifrost `git grep -nF '("cultcache-ts")'` at `42728a67` | `tests/persona-feedback-cli.test.mjs:15,55,62`, `tests/persona-discord-delivery.test.mjs:33`, `tests/persona-discord-crossing-rust-smoke.test.mjs:13`, `tools/persona-feedback.mjs:114`, `tools/agent-transport.mjs:40`, `tools/governance-threads.mjs:36`, and `tools/operator-notification.mjs:186`, which cut `bifrost-retire-alarm` deletes. `persona-feedback.mjs` calls `loadRuntime()` at module top (`:23`), so every verb fails at import. The bare `cultnet-ts` and `cultmesh-ts` sites resolve. Tools that load `dist/index.js` by path do not depend on the name: `bifrost-crossing-documents.mjs:271`, `bifrost-repository-release-authority.mjs:161`, `cultmesh-bridge-commands.mjs:313-314` and `provider-advertisement.mjs:47-119`. |
| C22 | What CultLib publishes | `git show <rev>:packages/{cultcache,cultnet,cultmesh}-ts/package.json` at `36ea08d3` and `016df462`; `git log -S'"@gamecult/cultcache-ts"'` | Only `cultcache-ts` is scoped: `@gamecult/cultcache-ts`, renamed in `8cb3b728` (2026-09-04). `cultnet-ts` and `cultmesh-ts` are still bare and depend on `@gamecult/cultcache-ts ^0.14.0`. The commit message says they "move with it", but only their dependency moved. The directory `packages/cultcache-ts` keeps its name, and every package declares `exports`. A local resolve at `016df462`, anchored at `packages/cultcache-ts/package.json`, gives `@gamecult/cultcache-ts` -> `dist/index.js` and `cultcache-ts` -> `MODULE_NOT_FOUND`. |
| C23 | How Bifrost finds CultLib | C21 sites; `ls .github/workflows` | Tools use `VOIDBOT_CULTLIB_ROOT`, falling back to the sibling `../CultLib`. **Tests hard-code the sibling `../CultLib` and ignore the variable.** Bifrost has no `package.json` and no CultLib dependency. No CI runs the node tests: the only workflow is `publish-container.yml`. |
| C24 | **Production Persona crossing is not broken by the rename** | `ssh ygg`, read-only: `systemctl cat bifrost-persona-mouth bifrost-persona-feedback`; `cat .../runtime/current/release-manifest.txt`; `createRequire(<release>/CultLib/packages/cultcache-ts/package.json).resolve("cultcache-ts")` | Both units run release `cb3239a…-f67f5122…`. The manifest says `cultlib_commit=f67f5122`, node `v24.14.1`. `VOIDBOT_CULTLIB_ROOT` points at the release's own CultLib. There the package is named `cultcache-ts`, `node_modules` holds real `cultcache-ts`, `cultmesh-ts` and `cultnet-ts` directories, and the bare name resolves to the release's `dist/index.js`. `f67f5122` is an ancestor of `8cb3b728`, so it predates the rename. A restart reloads the same pinned bundle. Since the C21 fix, Bifrost needs CultLib `8cb3b728` or later. A release that pairs it with an older CultLib fails at import (follow_up `stale-bifrost-persona-manifest` owns that path). |
| C25 | The Rust fixture tests have no producer | `tests/persona-discord-crossing-rust-smoke.test.mjs:17`, `tests/persona-discord-rudp-cross-language.test.mjs:21`; Epiphany `git log -S persona-discord-crossing-fixture`, `git grep` at `4d1113ef` | Both tests `cargo run --bin epiphany-persona-discord-{crossing,rudp-client}-fixture` in the sibling `../Epiphany`, with `CARGO_TARGET_DIR` hard-coded to `C:\\Users\\Meta\\.cargo-target-codex`. Epiphany deleted both bins in `387afe49` (2026-08-23), and neither exists at `4d1113ef`. So since 2026-08-23 these tests fail on every host, with cargo or without. `persona_discord_crossing.rs` survives as a library module. |
| C26 | Suite at base | ygg-verify Bifrost `42728a67`, CultLib `36ea08d3` at `/CultLib` | Per file (tests/pass/fail/skip): rust-smoke 1/0/1/0 (`spawn cargo ENOENT`); delivery 9/3/6/0 (`MODULE_NOT_FOUND 'cultcache-ts'`); permit 2/2/0/0; rudp 1/0/1/0 (`spawn cargo ENOENT`); feedback-cli 1/0/1/0 (the file fails to load, so its 8 tests are counted as 1); idunn-health 3/3/0/0. Glob total 17/8/9, exit 1, the same as Hands h1. |
| C27 | Suite with the fix | scratch commit `a6770c5` on `42728a67`: the scoped name at the seven live C21 sites, a skip guard on both Rust tests, and the `CARGO_TARGET_DIR` override removed. Same command. | rust-smoke 1/0/0/1 (`# SKIP`); delivery 9/9/0/0; permit 2/2/0/0; rudp 1/0/0/1 (`# SKIP`); feedback-cli 8/8/0/0; idunn-health 3/3/0/0. Glob total 24 tests, 22 pass, 0 fail, 2 skipped, exit 0. `agent-transport.mjs` and `governance-threads.mjs` were checked only by name resolution (C22), because no test loads them. |
| C28 | C3 at the reader's install pin | `git show 36ea08d3:packages/cultcache-ts/src/single-file-messagepack-backing-store.ts` | `pullAll` is at `:43`, and the file has no lock. C3 holds at `36ea08d3`, which is the CultLib that cut `ops-notice-deploy` installs. |
### Topology-lock cut probes (2026-09-30, 21:00-21:25 UTC)
Pinned heads: Idunn origin/main `bca3d85`; `hands/watchdog-idunn-incident`
local `9549d00` (h3, not yet pushed at probe time; origin at `3ed5f26`);
`hands/idunn-projection-mode` `15f7e18` (unmerged); Odin `665dc08`;
CodexConnector `6519289`; Ghostlight `eeb3a68`; gamecult-ops `fdb4356`;
cultcache-rs at Idunn's pin, CultLib `61549aa`. Line numbers marked *trial*
are from a scratch merge of `9549d00` into `bca3d85` (clean, no conflicts),
never in `F:\Projects\Idunn`. Yggdrasil probes were read-only over `ssh ygg`;
the semantics probe (C36) ran in a container through `ygg-verify.sh`.
| # | What | Where / command | Result |
|---|---|---|---|
| C29 | Every writer of `topology.cc` | trial `git grep` for `CultCacheTopologyDriver {`, `Engine::open(`, `mutate(` | One writer: the `idunn serve` daemon. `CultCacheTopologyDriver::mutate` (trial `src/drivers.rs:4393-4411`) is the only write path; its seven callers are `demote_to_expected_only` `:4455`, `withdraw_stale_incarnation` `:4567`, and the `TopologyPort` methods `:4778, 4846, 4897, 4940, 4992`. `Engine::open` has one production caller, `boot` (`src/control_plane.rs:5330`); the CLI verbs `up`, `status`, `cancel`, `expire` and `validate` write only `control.cc`. `idunn-host` and `idunn-provision` never touch the projection. |
| C30 | Every reader of `topology.cc`, across repos | `git grep -e 'topology\.cc' -e idunn-projection HEAD` in every repo under `F:\Projects` | Readers: Odin (`crates/odin-daemon/src/lib.rs` `CultCacheIdunnProjectionSource::entries`, `pull_all_read_only_snapshot`, lock-free since its first commit `eedce4f`, 2026-09-04), Idunn itself (`snapshot`, trial `drivers.rs:4383-4388`, lock-free; every `control_plane.rs` read is `pull_all_read_only_snapshot`), and the nightly backup (C33). No managed target reads the file: CodexConnector, Ghostlight, Muninn and StreamPixels name only `odin.verse-topology.v1`, and they receive their Expected record in the runtime bundle (`GAMECULT_IDUNN_RUNTIME_BUNDLE`). The unit comment (`deploy/idunn-yggdrasil.service:77-80`) and `docs/migration.md:505-509` say that "every managed target must read it", which is stale. |
| C31 | Shared-lock readers that rely on blocking a writer | cultcache-rs `61549aa` `lib.rs:337-395` (`with_read_only_shared_snapshot`), `:905-910` (`pull_all`, which takes `lock_shared`); cross-repo grep for `with_read_only_shared_snapshot` | That API holds the sibling lock shared so that no writer can replace the snapshot during `action`. Its production users are CodexConnector (`src/idunn_health.rs:441, 562`, `src/daemon.rs:205`) and Ghostlight (`crates/ghostlight-dungeon/src/idunn_health.rs:901`). All of them read the runtime authority record, the process-write-lease record, or their own configuration. None reads `topology.cc`. The write-lease lock is a different file under `idunn-authority` (`CultCacheWriteLeaseDriver`, trial `drivers.rs:4173, 4218`, already `try_compare_exchange_snapshot`). No reader of `topology.cc` takes a lock to get a consistent view. |
| C32 | How a reader stays consistent without the lock | cultcache-rs `61549aa` `lib.rs:666-688` (`write_all_unlocked`) | Every write stages `<store>.tmp`, fsyncs it, `replace_file_atomically` renames it over the store, and fsyncs the directory. A reader that opens the path sees one whole inode. Consistency comes from the rename, not the lock. |
| C33 | The nightly backup's lock use | live `/usr/local/sbin/gamecult-state-backup` sha256 `9d8ebd84...`, identical to gamecult-ops `scripts/gamecult-state-backup-yggdrasil.sh`; lines 98-126; `systemctl cat gamecult-state-backup.service` | Every file with a `<name>.lock` sibling is re-copied under `flock -s -w 20 -E 75 <lock> cp -a`. Past the wait it falls back to a double-read. The unit has no `User=`, so it runs as root: a `0600 root` lock still opens (`flock(1)` opens the path), and the backup keeps its shared lock. It skips a store whose lock is absent (`[[ -f lock ]] || continue`) and never creates one. The copy is milliseconds of `cp` once a night. Under a try-lock, a publish in that window is contended and retried on a later pass (C35). The backup does not need the lock for a whole copy (C32). |
| C34 | Live state of the projection dir | `ls -la`, `getfacl -p /var/lib/gamecult/idunn-projection`; `/proc/locks` against the lock's inode; `find /proc/*/fd -lname` | `drwxr-xr-x+ root`, `default:user::rw- default:group::r-- default:other::r--`. `topology.cc` `0644`, 13436 bytes, written 21:07. `topology.cc.lock` is `0644 root`, inode 8227809, created 2026-09-06. No flock is held on it and no process has either file open. `incidents.cc` does not exist yet, because the incident cut is not installed. The default ACL **is** documented, in Idunn `docs/migration.md:523` (it closes the chmod window), though gamecult-ops does not record it. `idunn-yggdrasil.service`: `User=root`, `UMask=0027`, `WatchdogUSec=0`. Odin runs as DynamicUser uid 62025. |
| C35 | What a failed topology publish does inside the tick | trial `src/control_plane.rs` at each call site | Every tick-path write returns `Err` through `?` **before** any evidence of it is persisted, and the step is retried. `advance_starting` records `expected_publication_sha256` or `activation_publication_sha256` only after publish returns `Ok` (`:6605`, `:6664`). `advance_leasing` does the same for the write-lease projection digest (`:7004`, `:7073`). `advance_fencing` withdraws the lease projection before recording (`:6894`, `:6926`). The post-fencing abort and pre-fencing cleanup write `CleanupEvidence::Complete` only after `reconcile_failed_candidate_projection` (`:8956`, `:9046`, `:9180-9204`). A failed continuity demotion (`:6099`) mints no restart and defers continuity. The admitted Expected repair (`:5870`) is logged and re-checked every pass. `retire_stale_incarnations` (`:7914`) returns `Err`. **The one path that is not retried is boot** (`reconcile_failed_continuity_projections`, `:5228-5275`, called once from `boot` `:5332`). It logs a failure and does not retry, and it is outside the tick. |
| C36 | Lock semantics under the live ACL | `ygg-verify.sh Idunn bca3d85 python:3.12-slim` running scratch `probe_lock.py`, as root | (1) A file created with mode 0666 under this default ACL and umask 027 lands `0644`, CultCache's shape. (2) A separate process holding `LOCK_EX` on it makes a nonblocking try from the parent contend. (3) **`fchmod 0600` on the same inode does not revoke that holder**: the try still contends. (4) A 0600 `O_CREAT|O_EXCL` staged file renamed over the lock lands `0600`, and the try on the new inode **succeeds** while the old holder still holds the orphaned inode. (5) uid 65534 cannot open the replaced lock (`Permission denied`). |
| C37 | The incident cut's private-lock mechanism | `git show 9549d00:src/incident.rs` `:229-245` (`exchange`), `:336-375` (`secure_lock`) | `exchange` calls `secure_lock(path)`, then `try_compare_exchange_snapshot`, and turns `Mismatch` and `LockContended` into errors. `secure_lock` opens `<store>.lock` with `create` and mode `0600`, then runs `set_permissions(0600)` on that descriptor. Its doc says that a descriptor another uid already holds is not revoked. So a lock left `0644` (C34's live `topology.cc.lock`) stays holdable by whoever opened it before (C36 step 3). It builds the lock name by hand, duplicating `authority_lock_path` (trial `drivers.rs:8194`). |
| C38 | `hands/b5` | `git merge-base --is-ancestor` | `hands/b5` (`f85e743`) and `hands/b5-pins` (`1cfbeff`) are both ancestors of origin/main (merged at `fcb8ba1` and `ddc031a`). The topology cut has no B5 conflict. |
| C39 | Unmerged branches that touch the topology driver | for each branch not in origin/main: `git diff origin/main...<b> -- src/drivers.rs` | Two. `hands/watchdog-idunn-incident` (the dependency). `hands/idunn-projection-mode` (`15f7e18`, 4 commits, 2026-09-30 01:15-02:40 UTC, no campaign doc) deletes `publish_projection_mode`, has `mutate` set a default ACL on the directory by `setxattr` (`publish_directory_default_mode`), and replaces the mode test with `the_published_projection_is_never_visible_in_another_mode`. That test asserts the **lock** is always `0644`, which contradicts `tick-lock-private`. It conflicts textually with this cut at `mutate` and at `publish_projection_mode`. |
| C40 | The private root's locks | `ls -ld /var/lib/gamecult/idunn`; `sudo ls -la` inside it; `getent passwd idunn`; `ps` | `/var/lib/gamecult/idunn` is `drwxr-x--- idunn:idunn` (uid 986, group 978, which has no members; no process runs as `idunn`). `control.cc.lock` is `0644 root`. The tick's control-store writes use blocking `compare_exchange` (e.g. trial `control_plane.rs:5471`), so root (the backup's `flock -s`) or uid `idunn` could hold one. Nothing runs as `idunn` today. |
## Model page
A row per persistent kind this campaign creates or depends on. "New" means the
kind does not exist yet. Names in `code` for new kinds are working names; the
cut spec fixes them.
| Kind | Identity | Lifecycle | Authority |
|---|---|---|---|
| **Incident record** (new, `idunn.operator_incident.v1`, its own file `/var/lib/gamecult/idunn-projection/incidents.cc`, published 0644 beside `topology.cc`) | Key `<condition>:<subject>:<opened_at_unix_ms>`. Namespace: `incidents.cc`, one type; the file refuses any other. It is never written to `control.cc`, which bails on unknown types, so a rollback would stop continuity (C1). `condition` is a closed enum (`continuity-exhausted` first; `odin-store-contested`, `odin-store-unpersisted` and `operator-required` join when their sources exist). `subject` is the target for target conditions and the `transaction_id` for `operator-required`. `opened_at` is written once by Idunn's clock. Injective because of one rule: **at most one open incident per `(condition, subject)`**, checked by a compare-exchange create. | **Created** at the site that decides the condition, after that decision and before its trace line. **Closed** by writing `closed_at` and `close_reason` on the same record, never by deleting it. Continuity-exhausted closes when the pass sees the workload running (`recovered`) or the target is no longer admitted, **not** when the window slides (I2). Odin conditions close when the catalog stops reporting them. `operator-required` has **no closing path yet** (I6; follow_up `b5-operator-required-no-exit`). **Reopened**: never; a recurrence is a new record with a new `opened_at`. **Replayed after an Idunn restart**: the durable open record makes the site write nothing. **Pruned**: closed more than 7 days ago, it retires to `history.cc`. **Store failure**: reported once as a fault and never returned into the tick (`survival-independent`). | Idunn's daemon, at the decision site, is the only writer. The same file is the record and what Bifrost reads, so no separate projection exists to drift. Forbidden: the CLI (`status` reads only), Bifrost and every other reader, Odin, `ReportOnce`, `control.cc`, a repair loop that opens incidents from log text. |
| **Projection lock files** (exist: `topology.cc.lock`; new with the incident cut: `incidents.cc.lock`, `incident-history.cc.lock`) | `<store>.lock` beside its store, named by `authority_lock_path`. One per store file. | **Created** by Idunn at mode `0600` before its first exchange on that store. **Replaced** by a staged `0600` file renamed over it whenever Idunn finds it wider than `0600` or not owned by Idunn's euid; a chmod cannot revoke an existing holder (C36). **Held** only for one nonblocking exchange. **Never published**: the store is `0644`, and its lock is not. | Idunn's daemon alone opens it for writing. The root backup may take it shared for its copy (C33), and a contended exchange is retried. Forbidden: any reader opening it (none does, C30-C31), `publish_file_mode` on it, a default ACL deciding its mode. |
| **Stderr trace** (exists) | `report_once` key, in memory | Printed once per process lifetime per fault; lost on restart (I1). | **Demoted**: `ReportOnce` is no longer an owner of "has the operator been told". It de-dups stderr only. |
| **Delivery request / journal entry** (Bifrost store) | Keyed by the Idunn incident key plus the notice kind (`opened`, and `closed` if Q `closure-notice` says so). Injective because the incident key is. Not derived from payload text or time (BF1's `raisedAt` hash is the counter-example). | **Created** by Bifrost when its reader first sees a notice-worthy incident state in the projection. **Revised**: `pending` -> `sent` or `failed`, with an attempt count and last error. **Retried** until sent, with backoff; a create for an existing key writes nothing (V1's unconditional `put` is the counter-example). **Replayed after a Bifrost restart**: journal is durable; a `pending` entry is retried. **Pruned** only after the incident key has left Idunn's projection. | Bifrost's reader only. Forbidden: Idunn (it never writes Bifrost's store, even though Y2 shows it physically could), VoidBot, the operator by hand. Which Bifrost process: see Rationale. |
| **Delivery receipt** (Bifrost, `discord_post_receipt.v1` or the journal's `sent` state) | Same key as the request. | Written once on success with the Discord message id. | Bifrost. **Idunn does nothing with it**: it neither reads it in the daemon nor waits on it (`survival-independent`). `idunn status` does not show delivery state; Bifrost's journal is where a failed delivery is visible. |
| **Recipient binding** | `DISCORD_OWNER_ID` in `/srv/bifrost/env/persona-delivery.env` (Y4). One recipient, the operator. | Operator-supplied; changes by editing that file. | Bifrost (with the operator as supplier). Forbidden: Idunn and every request payload. BF1's caller-supplied `recipientId` is the counter-example and does not survive. |
| **Bot token** | `BIFROST_DISCORD_BOT_TOKEN` in the same file (Y4), the one GameCult bot. | Operator-supplied, root `0600`, loaded by systemd `EnvironmentFile`. A process started before the file changed has the old value (Y6). | Bifrost holds it; the operator supplies it. Forbidden: Idunn, any notice content (`no-sensitive-egress`). |
| **Odin store condition** (input, not built) | `odin.store_condition.v1`, key `topology` (O1). | Owned by Odin's campaign. Rewritten on change; `contested` and `unpersisted` exist only in Odin's memory and catalog, not on disk. | Odin. Idunn reads it through Odin's catalog with a `SnapshotRequest` (I11 precedent), never from `topology.cc`, and never writes it. An unreachable Odin yields no store incident; continuity already covers a dead Odin. |
| **De-dup key** | Not a separate kind: it **is** the incident key. At Idunn: one open incident per `(condition, subject)`. At Bifrost: one journal entry per `(incident key, notice kind)`. At Discord: the journal key hashed into the message `nonce` with `enforce_nonce`, which covers a crash between the POST and the journal write. The Discord de-dup window is short and unverified here; outside it, a crash in that gap can still send twice. | Lives as long as the incident and its journal entry. | Idunn owns the incident half, Bifrost the delivery half. |
| **Delivery-failure signal** | **Empty.** Nothing tells the operator that the watchdog's own delivery is failing. | | Bifrost's journal holds the state; nobody surfaces it. Out of this table's reach; left for the delivery-mechanism cut (see Rationale). |
| **Operator acknowledgement** | **Not created.** No kind lets the operator mark an incident as seen. Nothing in the target asks for one. | | |
Empty cells, stated plainly:
- `operator-required` closure: the incident closes when the transaction ends or
its step goes on (`close_spent_operator_incidents`). A stateful deployment
left to the operator still has no Idunn-owned terminal action; the operator's
`idunn expire` re-enters the same arm.
- Delivery-failure visibility: no owner.
- Discord `enforce_nonce` window: not verified against Discord's current docs
in this pass.
## Rationale
**Idunn publishes and Bifrost reads, not the reverse.** Three crossings were
weighed against the target:
1. *Idunn writes a command into Bifrost's store* (the `c063f27` shape). Idunn
would write another organ's state; it would carry the recipient (BF1); an
unconditional `put` re-arms a completed command (V1); and the deployed pump
cannot send a DM anyway (Y7). Nothing stops root Idunn from doing it (Y2),
so the boundary would be doctrine alone.
2. *Idunn sends a CultNet command to a Bifrost listener.* Idunn becomes a
delivery client with timeouts and retry state in its tick, which is where
`survival-independent` breaks. The rebuild already deleted the synchronous
version of this (I14).
3. *Idunn writes its incident into its own store and its world-readable
projection; a Bifrost reader derives its own delivery journal.* No write
crosses an owner boundary. Idunn holds no recipient and no token. Bifrost
down means a late message, not a slow tick. This is the shape of the
existing topology projection (I13), which every managed target already
reads. **Recommended.** It reverses `c063f27`'s direction, so it is a
question for the operator, not a default.
**Which Bifrost process delivers is doctrine, not a fork.** A service earns a
daemon only when isolation protects a named invariant. A notice can wait a
minute, so a systemd timer running a one-shot Bifrost reader under a Bifrost
user, with `EnvironmentFile=/srv/bifrost/env/persona-delivery.env`, is the
smallest body. Folding it into `bifrost-persona-mouth` was rejected: that
process runs the Epiphany-permit-gated Persona crossing, and a watchdog page
that needs Starfire's permit would stop whenever the workstation is off.
Idunn-invoked delivery was deleted once already (I14).
**DM is the default, not a question.** The operator's words are "I ought to
get a message", and the recipient binding that exists is `DISCORD_OWNER_ID`.
**Retiring `publish-idunn-alarm` follows from the crossing.** If the crossing
is (3), nothing calls the publisher. The unauthenticated command path (BF4)
then carries no watchdog traffic. Closing it is Bifrost's own business, and
VoidBot's caller is not live on Yggdrasil (V2). If the operator picks (1), that
path must be authenticated before Idunn uses it.
**Exhaustion closes on health, not on the window.** `restarts_exhausted` turns
false the moment one timestamp leaves the hour (I2). With a still-dead target
that happens several times an hour. Closing on it would mean one incident, and
one notice, per slide.
**Why `ReportOnce` is demoted rather than extended.** Persisting its map would
make an in-memory de-dup cache the owner of "was the operator told", keyed by
free-form strings, with no closure and no subject. The incident record puts
that decision at the condition's own site, as `record-before-delivery` asks.
**The incident store is its own file, and that file is the projection.** The
model page placed the record in `control.cc` with a separate projection
derived from it. C1 overturns that: an unknown type in `control.cc` makes every
older Idunn binary refuse the whole store, so rolling back the incident cut
would stop continuity for every target. The record therefore lives in its own
CultCache file, `/var/lib/gamecult/idunn-projection/incidents.cc`, written by
Idunn with compare-exchange and published `0644` like `topology.cc` (C6). An
incident carries nothing sensitive (`no-sensitive-egress`), so there is no
private half to keep apart, and the derived copy disappears. What is lost: the
open write is not atomic with any `control.cc` write. That costs nothing
today, because the exhaustion decision writes nothing to `control.cc` (I2),
and later sources are read-only observations. The model page's
"Incident record" and "Incident projection" rows collapse into this one file.
Self owns the edit to the model page.
**A broken incident store never reaches the tick.** Every incident operation
returns its error to one call site, which traces it once with `report_once`
and moves on. An unreadable store means no incident opens (the stderr trace
still prints); it never changes what continuity decides. The Idunn cut's
survival test pins this with an unwritable store path.
**Bifrost reads without the lock.** Idunn's writes once blocked on an
exclusive lock (C2). The incident and topology cuts make every write in the
projection directory a nonblocking exchange behind a private lock. A reader
that took the lock is therefore unnecessary. It would also be refused, because
it cannot open a `0600` lock. cultcache-ts `pullAll` takes no lock (C3, C4),
and Idunn's rename is atomic (C32), so the reader only ever sees a whole
snapshot. The reader never opens Idunn's file through a CultMesh node,
which could flush.
**Delivery at most once, with the nonce as the second guard.** This follows
the Persona precedent (C16): a journal entry found `running` at start becomes
`unknown` and is never re-posted. A bridge exit that failed is retried on a
later run, with the same nonce and `enforce_nonce` (C17). A retry inside
Discord's "past few minutes" returns the original message instead of creating
another. The remaining duplicate needs the bridge to have posted, then exited
non-zero, and the retry to fall outside that window. The attempt cap and
timer period bound it, and it is recorded as a residual risk rather than
engineered away.
**Who sees a delivery failure: Bifrost, through systemd.** There is one
channel to the operator, and it is the one that failed, so a failure cannot
page over it. The reader exits non-zero while any journal entry is `unknown`
or has exhausted its attempts. The unit then shows in `systemctl --failed` and
in its journal, and the reader's `status` verb lists the entries. Idunn reads
none of this (`survival-independent`). This is a default: the alternatives
either couple Idunn to delivery state or need a second channel that does not
exist.
**Deployment of the reader has no Idunn path.** Idunn deploys resident
services, and a timer does not fit its recipe model (C9). No Bifrost binding
exists in current Idunn (C10), and the old Bifrost manifest belongs to the
Idunn that was rebuilt away (C11). The ruled one-shot reader can be
hand-installed by a gamecult-ops script and runbook, the same interim footing
as `huginn.service` and `bifrost-persona-mouth`. Otherwise Idunn must first
learn a new workload kind. That choice is an operator question raised in the
ops cut.
**The reader gets its own release root.** It installs under
`/srv/bifrost/watchdog-notice/releases/<bifrost>-<cultlib>` and does not
repoint `/srv/bifrost/persona-feedback/runtime/current`. Repointing that link
would stage new code under `bifrost-persona-mouth`. With the token file now
present (Y3, Y6), the mouth's next restart would also make the Persona crossing
able to post. That crossing is out of scope and must not change as a side
effect.
**Cut order.** The Idunn incident cut comes first. It writes the golden
fixture that Bifrost's reader is tested against (C5). The Bifrost subtraction
cut is independent and lands before the reader, so the reader is written
against a tree with one Discord command vocabulary. The gamecult-ops
subtraction is independent too. The ops deploy cut comes last and needs both
behaviour cuts merged.
**The Bifrost suite follows the owner's name, with no fallback.** CultLib
renamed `cultcache-ts` to `@gamecult/cultcache-ts` so that the org owns the
name on a public registry (C22). The consumer changes the string at each live
require site (C21) and keeps the directory anchor, which did not move. A
try-both-names loader was rejected. It would be a local shim that keeps the
old name alive in Bifrost after its owner retired it. It would also let a
release silently pair new Bifrost with a pre-rename CultLib. After the change
that pairing fails loudly at import (C24). Production does not move: its
bundle pins CultLib `f67f5122` (C24). `cultnet-ts` and `cultmesh-ts` stay bare
because that is what CultLib publishes. Whether they should follow the scope
is CultLib's decision, not Bifrost's.
**The Rust fixture tests skip with a reason; they are not deleted.** Their
producers were deleted from Epiphany on 2026-08-23 (C25), so installing cargo
would not make them pass. Each one skips unless cargo and its named fixture
source are present, and the skip reason names the Epiphany commit that removed
the fixture. Deleting them would throw away the only Rust-authored byte seam
test the Persona crossing has. Restoring a producer belongs to Epiphany and the
Persona crossing, both outside this campaign (follow_up
`bifrost-rust-fixture-producer`). The hard-coded Windows `CARGO_TARGET_DIR`
goes, because it is wrong on every host except one workstation.
**`bifrost-suite` does not depend on `bifrost-retire-alarm`.** It leaves
`operator-notification.mjs` alone, because that cut deletes the file, so the
two branches merge in either order without conflict. The reader depends on
both. It is written against a tree with one Discord command vocabulary, and it
is verified by a suite that loads.
**The topology write stops waiting, and the lock stops being published.** The
lock was published `0644` because a comment assumed that readers open it
(`publish_projection_mode`, from `fbb9626`). None does (C30, C31), and
consistency comes from the rename (C32). So the lock is Idunn's alone, and a
contended publish is an error, not a wait. That is safe because every tick path
treats a failed publish as not done: no publication digest, phase transition,
cleanup evidence or continuity restart is persisted before the write returns
`Ok` (C35). A contended publish is retried on a later pass or after backoff,
and Idunn never believes it published what readers do not see. Boot
reconciliation is the exception. It runs once, outside the tick, and it already
logs failures and moves on. Only root processes can contend once the lock is
private: the backup's copy, a few milliseconds a night. No retry machinery is
added for it.
**One lock rule for the whole projection directory.** The topology cut moves
the incident cut's `secure_lock` into one crate-level exchange primitive that
both stores call, rather than a second copy. It also strengthens the rule:
a lock wider than `0600`, or owned by someone else, is replaced by a rename,
not chmodded. A chmod does not revoke a holder (C36), and the live
`topology.cc.lock` has been `0644` since 2026-09-06 (C34), so tightening it in
place would leave any existing holder able to block every topology publish
forever. The replacement runs only while the lock is not yet private. Once it
is, no other uid can open it, so a later replacement never splits exclusion
between two inodes. A root backup holding the old inode during the first
replacement copies one whole inode anyway (C32).

View file

@ -0,0 +1,245 @@
# Watchdog reporting postmortem
Campaign `idunn-watchdog`, 2026-09-30 to 2026-10-01. This was the first Eureka
campaign whose state was typed in Huginn's mind (instance `eureka`) rather than
kept in a prose map. Document ids below are `idunn-watchdog:<kind>:<local>`.
Read them with `view`; this page does not restate them.
## Summary
Idunn is now a watchdog that reaches the operator on Discord.
- **Idunn** opens a durable incident at the site where a continuity budget is
exhausted (since `idunn-reincarnation` cut 1 this is a paging threshold
only; continuity keeps restarting). The incident is written to `incidents.cc`, a world-readable file
in its projection directory with a private 0600 lock, and closes when the
target runs again and its restart window is no longer exhausted.
- **Bifrost's watchdog-notice reader** runs from a one-minute timer. It sends
one opening DM and at most one closure DM per incident, keeps its own
journal, and retries a failed send forever with capped backoff.
- **Scale:** 8 cuts across Idunn, Bifrost and gamecult-ops: 13 spec revisions,
17 Hands reports, 17 Soul verdicts and 55 findings. Measured against the
in-force estimates:
| Repo | Estimate (lines) | Landed (lines) |
|---|---|---|
| Idunn | +930/−101 | +2618/−354 |
| Bifrost | +246/−335 | +1182/−429 |
| gamecult-ops | +280/−35 | +505/−104 |
- **Installed on Yggdrasil:**
- Idunn `a909f93` at 2026-09-30 23:21 UTC, then `cb60965` at 2026-10-01
02:02 UTC.
- The Bifrost reader at `160fac20` with CultLib `36ea08d3`, at 2026-10-01
08:04 UTC.
- **First live notice:** the incident's first real input was raven-muninn,
which exhausted its budget minutes after the incident cut was installed. Its
opening DM was delivered at 08:04:41 UTC, and the operator confirmed it.
- **Still owed:** the closure DM for that same incident
(`follow_up:closure-dm-observed`). It depends on raven-muninn recovering, and
raven-muninn's own fault lies outside this campaign.
## Scope and invariants
Target `r4`. Its invariants are:
- `incident-owner`
- `record-before-delivery`
- `delivery-owner`
- `survival-independent`
- `one-per-incident`
- `no-sensitive-egress`
- `tick-lock-private`
The operator directions are `ruling:operator-watchdog-direction` and
`ruling:operator-bifrost-bridge`. The second made Bifrost the bridge to
Discord, so Idunn holds no credential and speaks no Discord API.
Excluded:
- **Odin's store condition and B5 `OperatorRequired`.** Neither can page yet.
Odin's O2 cut is unbuilt, and B5 has no operator exit
(`follow_up:b5-operator-required-no-exit`).
- **VoidBot's `notify_owner`.**
- **The Persona Discord crossing.**
- **Eve lowering.**
The earlier attempt was Bifrost `c063f27` (2026-09-04): an alarm was a command
written into Bifrost's store. It never ran, because no pump was deployed and
the deployed pump lacked the DM verb. It was also reversed by ruling:
`ruling:delivery-crossing` has Bifrost read Idunn's projection, so no write
crosses an owner boundary.
## Timeline
| Cut | Reports | Soul passes | Notable finding |
|---|---|---|---|
| ops-retire-alarm | h1 | 1 | none |
| bifrost-retire-alarm | h1–h2 | 2 | `silent-skip`: a stray `discord-dm` command sat pending forever |
| bifrost-suite | h1 | 1 | `suite-cultcache-scope` (pre-existing): 9 of 17 tests failed at base on a CultLib rename |
| bifrost-notice-reader | h1–h3 | 3 | `prune-keyed-on-presence`, `bridge-exit-is-failed`: duplicate DMs and blocked closures |
| bifrost-notice-retry | h1–h2 | 2 | `future-stamp-stalls-retry` |
| ops-notice-deploy | h1–h3 | 3 | `retry-window-five-minutes`: a 5-minute outage dropped a notice for good |
| idunn-incident | h1–h3 | 3 | `lock-stalls-tick` (Blocker); `lock-acl-world-openable` (High) |
| idunn-topology-lock | h1–h2 | 2 | `contended-publish-aborts`; the live stall it fixed was pre-existing |
## Structural delta
The landed code was about 2.5× its estimate, and the misses were not explained
cut by cut as the skill asks.
- **idunn-incident:** +1813 against +700. Its fix batches added the lock
primitive, load-independent lock tests, and per-operation fault keys.
- **bifrost-notice-reader:** +891, against r3's estimate of +140. That r3
figure described only the revision's own delta, not the cut, so r3 carried
the wrong baseline.
Deleted:
- Bifrost's `publish-idunn-alarm` and the pump's `discord-dm` verb.
- gamecult-ops' dead `--operator-alarm-command` scripts.
- Idunn's `publish_projection_mode` and its 0644-lock test.
- The reader's prune-on-absence logic and its `WATCHDOG_NOTICE_NOW_MS`
production clock seam.
Added:
- `idunn.operator_incident.v1` (`incidents.cc`) and
`bifrost.watchdog_notice_execution.v1` (the journal).
- `bifrost-watchdog-notice.{service,timer}`.
Nothing was parked.
## What Soul caught
Green suites passed every defect below.
- **Survival and locking.**
- Incident writes took a *blocking* flock inside Idunn's scheduler tick, on
a world-readable lock. Any local uid could freeze continuity for every
target, and `WatchdogUSec=0` meant nothing would restart Idunn
(`s1.lock-stalls-tick`).
- After the fix, the live directory's default ACL still made the lock 0644,
so any uid could stop incidents opening. The test ran in a temp directory
with no ACL (`s2.lock-acl-world-openable`).
- The same probe exposed a pre-existing live hazard: `topology.cc.lock` 0644
with a blocking write in the tick, measured at 5.07 s at base
(`s2.topology-lock-stalls-tick`). The operator ruled it into the campaign;
after the topology cut the tick takes 0.09 s.
- **Split authority.**
- The reader pruned its "already sent" journal whenever a key was absent
from Idunn's store. Restoring a backup would have re-sent old DMs.
- The reader decided subject validity with its own grammar, narrower than
Idunn's.
- Retention was decided by two predicates in Idunn.
- **Lifecycle.**
- One running observation closed an exhaustion incident. A release that
crashes after each restart would have sent about 12 DMs an hour
(`s1.flap-reopens`).
- A bridge that died after Discord accepted a message was retried up to 5
times, and the closure was then blocked forever.
- Five attempts on a one-minute timer dropped a notice after a 5-minute
Discord outage.
- **Decorative checks.**
- A compare-exchange expected `None` on a key that embeds `now`, so it
guarded nothing (`s1.cas-decorative`).
- A 2-second wall-clock bound on a lock test flaked under load and counted
equivalent mutants as caught.
- **The verifier itself.** Soul's s3 found that the stopgap forced one
`CARGO_TARGET_DIR` on every job, so cargo-mutants' parallel workers tested
each other's binaries. Two runs of the same two mutants disagreed
(`s3.mutants-share-target-dir`). Every parallel mutation count reported
through the stopgap before 2026-10-01 is unproven, in both sessions' work.
## Operator corrections
- **"Bifrost is indeed intended to be the bridge to external services like
Discord."** The agents had found two delivery paths: VoidBot's
`notify_owner` and Bifrost's command store. The ownership boundary was
missing from both repos' summaries.
- **"Stewardship is currently universal…"** Self had filed a gap treating
follow-up repos as needing stewardship. It was withdrawn; the rule was not
written in the skill.
- **"…we seem to be saving some tokens… but we're very very slow."** Self
blamed cold builds, measured, and was wrong. Cold builds took about 51 s and
warm builds 29 s. The cost was slot queueing, repeated verify jobs, a test
that hung a slot for 39 minutes, and a 4-hour job from the other session
holding a fifth of capacity.
- **"Yep, fixing Mind issues is highest priority, immediately once this
campaign closes"** (`ruling:mind-fixes-next`).
## Incidents
- **Network outages (ENOTFOUND).** Two outages killed three agents
mid-pass. Each was resumed by `SendMessage` with context intact, and nothing
was lost, because each agent's state was in commits and the mind.
- **A shared stopgap target directory** made mutation evidence noise. It is
fixed in Eureka `a755a60`.
- **Self's own brief inflated runtime.** It told Hands to mutate serially to
dodge a flake, and was corrected to skipping the flaky test by name.
- **A hung test held a verify slot for 39 minutes**
(`s2.route-connect-test-hangs`). It is fixed.
- **A Hands test helper overwrote the container's `/usr/bin/node`** through a
symlink. It was caught by the agent's own baseline and then confined
(`s2.test-clobbers-usr-bin-node`).
- **A worktree directory stays locked** by a finished process
(`F:\Projects\gamecult-ops-watchdog-deploy`, empty). Removing it is left to
the operator.
## What worked
- **Soul attacks the layer where the rule fails, not the test.** Every
High/Blocker finding came from a probe the shipped tests could not see: a
separate-process lock holder, the live directory's ACL, or a restored store.
- **Point Soul at the load-bearing invariant in the brief.** Naming "can the
lock hang the tick" produced the Blocker on the first pass.
- **Read before touching live state.** Self installed only after checking the
installed revision, the in-flight transactions and the unit diff. The repo
unit carried a BOM the live one did not.
- **One verify job per pass.** Once briefs required it, a reader fix batch
took 11 minutes, where the first reader Hands took 38.
## What to change in the pipeline
Already changed in the skill:
- The substrate gaps moved from a table to the mind.
- Imagination owns the model page.
- Soul pays for each rerun once.
- The rust image uses sccache.
- The stopgap no longer forces `CARGO_TARGET_DIR`.
Still to change:
- **Explain every ledger miss in the next report's deviations.** This campaign
ran at about 2.5× its estimate and nobody explained it per cut.
- **Make a revision's `estimate` the whole cut.** r3's estimate covered only
its own delta.
- **Verify the merged tree of a long-lived branch before pushing.** Self did
this once, for the incident cut over B5, and it should be the rule.
- **Briefs name the standard sibling build command.** The stopgap cannot
provision siblings (`follow_up:gap-verify-no-sibling-repos`), and one
hand-rolled build produced four false failures.
## Open follow-ups
Fifty follow-ups are in force. Twenty are substrate gaps (`gap-*`): the work
the operator ruled next. Query them with the substrate-gaps recipe in the
skill. Seven findings remain unresolved:
- **`cut-idunn-incident.s3.mutants-share-target-dir`.** It is fixed in the
stopgap but awaits a Soul mutation run under the fixed script.
- **Six findings whose resolution keys exceed 64 bytes**
(`follow_up:gap-unresolvable-finding-key`). Their substance is recorded in
`follow_up:notice-retry-hardening`, `follow_up:ops-refusal-test-hardening`,
`follow_up:bifrost-pump-input-hygiene` and
`follow_up:contended-publish-bounds`.
Operational follow-ups that matter most:
- `closure-dm-observed`
- `notice-unknown-settle`
- `contended-publish-bounds`
- `idunn-load-flaky-tests`
- `cultcache-rs-lock-mode`
- `cultcache-ts-readonly-pull`

View file

@ -0,0 +1,95 @@
# Watchdog reporting: target
Status: target, Self, 2026-09-30. This is the first campaign whose state is
typed in Eureka's mind (campaign `idunn-watchdog`). Questions, rulings, cut
specs and their verification live there. This document keeps the rationale
and the design truths the campaign produces. The cut map's body facts and
model page will live in `watchdog-reporting-map.md`.
## Why
The operator ruled Odin's store-recovery Q-R4 on 2026-09-30 (Odin
`docs/store-recovery-cut.md`, `cdc57bd`). A third set-aside inside ten minutes
stops Odin writing while it keeps serving. The operator added:
> Yep, and Idunn needs better reporting capabilities so it can serve as
> watchdog. I ought to get a message about this on Discord if it ever happens.
and, on where delivery lives:
> Bifrost is indeed intended to be the bridge to external services like Discord
Today nobody would hear about it. Idunn already detects several conditions an
operator must act on, and for each one it writes a single `eprintln!` line to
journald (`report_once`, `control_plane.rs:4599`):
- A continuity paging threshold is reached (6 charged attempts per hour;
`restarts_page`, formerly `restarts_exhausted`). Continuity never stops:
it keeps restarting on a capped backoff, and this threshold only pages. When
this target was written the only trace was one log line.
- An admitted route is degraded. It fails its stable-route challenge
(`supervise_admitted_route`, ~5973). The degraded mark is durable, but no
one is told.
- A post-fencing phase deadline leaves a transaction to the operator: a
stateful deployment whose candidate was issued a lease, or a continuity
retrying forward past its deadline. Since `idunn-reincarnation` cut 2 it is
the second incident condition, `operator-required` (subject: the transaction
id), beside `continuity-exhausted`; it closes when the record ends or its
step goes on.
Odin's own store condition (`odin.store_condition.v1`, `contested`) is mapped
but not built. Idunn reads nothing about Odin's store health today.
The delivery path half exists in Bifrost:
- `tools/operator-notification.mjs publish-idunn-alarm` builds a
`bifrost.bridge.discord_post_command.v1` `discord-dm` command.
- `tools/cultmesh-bridge-commands.mjs process` turns such commands into DMs,
writing a `discord_post_receipt.v1` for each.
But no pump runs anywhere, and the bot token file on Yggdrasil is absent
(gamecult-ops `inventory.md` ~740). Idunn has never called the publisher. So
the machine has a detector with no record, and a deliverer that nothing feeds.
## Design truths this campaign produces
- **Idunn decides what is an incident.** Idunn owns daemon survival, so the
judgement "an operator must look at this" is its judgement, made where the
condition is decided. An incident is typed state in Idunn's store before any
delivery is attempted. A stderr line is a trace of the record, not the
record itself.
- **Bifrost delivers.** Discord transport, the bot token, the recipient
binding and the delivery receipt belong to Bifrost. Idunn holds no Discord
credential and speaks no Discord API.
- **Survival never waits on delivery.** Bifrost is a managed target like any
other. Its absence, failure or backlog may delay a message. It never delays
or gates an Idunn decision, tick, restart, or deployment. An incident nobody
has delivered stays visible in Idunn's own state and in `idunn status`.
- **Two messages per incident at most, never one per tick.** A fault that
lasts hours produces one opening notice, not a stream of them. When it
clears, one closure notice follows, and only if the opening notice was sent.
Repeated delivery attempts for either notice are idempotent at the
recipient's end.
- **No tick waits on a lock another uid can hold.** Root holders, such as
the nightly backup's millisecond shared lock, are trusted and bounded. The
operator ruled this scope on 2026-09-30, choosing uid-scoped. Soul found that any
local uid could freeze Idunn's scheduler tick by holding a published
`.lock` file. This was true for incidents during the campaign, and for
`topology.cc` before it. Idunn's writes in its projection directory are
non-blocking exchanges. The locks behind them are private to Idunn,
whatever the directory's default ACL says. The operator ruled on
2026-09-30 that the topology fix belongs to this campaign.
- **Nothing sensitive leaves.** A notice names the target, the condition and
the time. It never carries a binding value, a path to a secret, a token, or
a line of configuration.
## Not this campaign
- Odin's store-recovery cuts, O2 included. They produce `odin.store_condition`
and are Odin's campaign, still in prose. This campaign consumes the record
once it exists.
- Idunn's phase deadlines. They landed in `idunn-reincarnation` cut 2, which
also owns the `operator-required` incident condition.
- VoidBot's `notify_owner` MCP tool and its separate DM code.
- Bifrost's Persona delivery crossing.
- Eve/TUI lowering of incidents.

96
src/bin/idunn-launch.rs Normal file
View file

@ -0,0 +1,96 @@
//! The unit's process for an upstream daemon: parse the recipe's argv, read
//! what Idunn set, run the launcher, and map the outcome to an exit code.
#[cfg(unix)]
fn main() -> std::process::ExitCode {
use std::sync::atomic::{AtomicBool, Ordering};
use anyhow::Context;
use idunn_daemon::deployment::{
IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT, IDUNN_RUNTIME_BUNDLE_ENVIRONMENT,
IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT, IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT,
};
use idunn_daemon::launcher::{self, *};
static STOP: AtomicBool = AtomicBool::new(false);
extern "C" fn on_signal(_: libc::c_int) {
STOP.store(true, Ordering::SeqCst);
}
fn launch() -> anyhow::Result<Outcome> {
let release_root = std::env::current_exe()?
.parent()
.context("the launcher is not inside a release")?
.to_owned();
let contract = LaunchContract::parse(release_root, std::env::args().skip(1))?;
let bundle = std::env::var_os(IDUNN_RUNTIME_BUNDLE_ENVIRONMENT)
.context("the Idunn runtime bundle is not set")?;
let bind = std::env::var(IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT)
.context("the Idunn candidate bind is not set")?;
let member = launcher::parse_member_bind(
&std::env::var(IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT)
.context("the Idunn member bind is not set")?,
&bind,
)?;
let (activation, provider) = launcher::inherited_authority_files()?;
let authority = RuntimeAuthority::open(bundle.as_ref(), activation, provider)?;
authority.require_candidate_bind(&bind)?;
let lease_path = std::env::var_os(IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT);
anyhow::ensure!(
lease_path.is_some() == authority.expected().write_lease_required,
"the process write lease binding differs from Expected"
);
let route = std::net::TcpListener::bind(&bind).context("binding the candidate endpoint")?;
let sink = OdinRudpSink::from_expected(authority.expected())?;
let lease = DriverLeaseReader::new(
&authority.expected().target,
lease_path.as_deref().unwrap_or_default().as_ref(),
);
let health_path = contract.health_path.clone();
let ports = Ports {
credentials: std::sync::Arc::new(CredentialsDirectory::from_environment()),
clock: std::sync::Arc::new(SystemClock),
sink: std::sync::Arc::new(sink),
lease: std::sync::Arc::new(lease),
spawner: Box::new(ProcessSpawner),
health: Box::new(move |member| {
std::sync::Arc::new(HttpHealthProbe {
member,
path: health_path.clone(),
})
}),
};
// SAFETY: the handler only stores to an atomic.
unsafe {
libc::signal(libc::SIGTERM, on_signal as *const () as libc::sighandler_t);
libc::signal(libc::SIGINT, on_signal as *const () as libc::sighandler_t);
}
Ok(launcher::run(
&contract,
authority,
route,
member,
&std::env::temp_dir(),
ports,
&STOP,
))
}
match launch() {
Ok(outcome) => {
if let Outcome::Failed(code) = &outcome {
eprintln!("idunn-launch ended: {code}");
}
std::process::ExitCode::from(outcome.exit_code() as u8)
}
Err(error) => {
eprintln!("idunn-launch refused: {error}");
std::process::ExitCode::from(2)
}
}
}
#[cfg(not(unix))]
fn main() {
eprintln!("idunn-launch runs only on Unix");
std::process::exit(2);
}

File diff suppressed because it is too large Load diff

View file

@ -9,12 +9,27 @@ pub const TARGET_DECLARATION_SCHEMA: &str = "gamecult.idunn.target_declaration.v
pub const OPERATOR_BINDING_SCHEMA: &str = "gamecult.idunn.operator_binding.v2";
pub const IDUNN_RUNTIME_BUNDLE_ENVIRONMENT: &str = "GAMECULT_IDUNN_RUNTIME_BUNDLE";
pub const IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT: &str = "GAMECULT_IDUNN_CANDIDATE_BIND";
pub const IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT: &str = "GAMECULT_IDUNN_MEMBER_BIND";
pub const IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT: &str = "GAMECULT_IDUNN_PROCESS_WRITE_LEASE";
pub const RUNTIME_PRESENCE_IDENTITY_BINDING: &str = "GAMECULT_RUNTIME_PRESENCE_IDENTITY";
/// A host actuator hands the workload its activation credential as a file
/// named by this variable; on the Idunn host it arrives as a parent-only
/// descriptor instead and this name is never set.
pub const IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT: &str = "GAMECULT_IDUNN_ACTIVATION_CREDENTIAL";
/// systemd hands a unit its credential directory under this name; Idunn owns
/// the directory, so no binding may set it.
pub const SYSTEMD_CREDENTIALS_DIRECTORY_ENVIRONMENT: &str = "CREDENTIALS_DIRECTORY";
/// Every launch-environment name Idunn sets or reserves. A binding may name
/// none of them; this is the one list both the binding check and the
/// launch-input check read.
pub(crate) const IDUNN_OWNED_LAUNCH_ENVIRONMENT: [&str; 6] = [
IDUNN_RUNTIME_BUNDLE_ENVIRONMENT,
IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT,
IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT,
IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT,
IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
SYSTEMD_CREDENTIALS_DIRECTORY_ENVIRONMENT,
];
pub const RUNTIME_PRESENCE_IDENTITY_FD_NAME: &str = "gamecult-runtime-presence-identity";
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
@ -98,6 +113,10 @@ pub struct ArtifactOutput {
pub expected_sha256: Option<String>,
#[serde(default)]
pub executable: bool,
/// Regular files inside a directory artifact that are sealed executable
/// (0555); every other file in the tree is sealed 0444.
#[serde(default)]
pub executable_members: BTreeSet<PathBuf>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
@ -184,6 +203,11 @@ pub enum StateKind {
/// Process-owned state imposed by an external runtime contract. Idunn
/// preserves and fences the file but never interprets its payload.
ExternalFile,
/// A process-owned directory imposed by an external runtime contract (for
/// example Synapse's media store). Idunn preserves and fences it and never
/// interprets its contents. Same writer, recovery and startup rules as
/// `ExternalFile`.
ExternalDirectory,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
@ -504,8 +528,7 @@ impl WorkloadBinding {
{
require_environment_name(name)?;
ensure!(
name != IDUNN_RUNTIME_BUNDLE_ENVIRONMENT
&& name != IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT,
!IDUNN_OWNED_LAUNCH_ENVIRONMENT.contains(&name.as_str()),
"operator binding attempts to replace Idunn-owned workload environment"
);
}
@ -638,9 +661,7 @@ pub struct ProcessWriteLeaseBinding {
impl ProcessWriteLeaseBinding {
pub fn lock_path(&self) -> PathBuf {
let mut lock_path = self.record_path.as_os_str().to_os_string();
lock_path.push(".lock");
PathBuf::from(lock_path)
crate::drivers::authority_lock_path(&self.record_path)
}
}
@ -662,40 +683,99 @@ pub struct RouteBinding {
pub private_port_end: u16,
pub config_path: PathBuf,
pub reload_unit: String,
/// A second listener for the route's members. When set, members reach the
/// workload straight through a second nginx server block, and the
/// workload's launcher answers presence only.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub member: Option<MemberListenerBinding>,
}
/// The member listener of a route: where members connect, and the first of the
/// private ports the workload binds for them. The private range is as wide as
/// the route's own.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct MemberListenerBinding {
pub stable_endpoint: String,
pub private_port_start: u16,
/// Most member connections nginx holds open at once for this route. Each
/// costs nginx two connections (client and upstream) from the pool the
/// route's presence block shares, so this limit is what keeps members from
/// exhausting it.
#[serde(default = "default_member_max_connections")]
pub max_connections: u16,
}
const DEFAULT_MEMBER_MAX_CONNECTIONS: u16 = 128;
fn default_member_max_connections() -> u16 {
DEFAULT_MEMBER_MAX_CONNECTIONS
}
impl RouteBinding {
pub(crate) fn stable_socket(&self) -> Result<(IpAddr, u16)> {
let authority = match self.driver {
RouteDriver::NginxStreamTcp => self
.stable_endpoint
.strip_prefix("http://")
.or_else(|| self.stable_endpoint.strip_prefix("tcp://")),
RouteDriver::NginxStreamUdp => self.stable_endpoint.strip_prefix("rudp://"),
}
.context("stable endpoint transport does not match its route driver")?;
ensure!(
!authority
.chars()
.any(|character| matches!(character, '/' | '?' | '#' | '@' | '[' | ']')),
"stable endpoint is not a plain IP address and port"
);
let (host, port) = authority
.rsplit_once(':')
.context("stable endpoint has no port")?;
let host = host
.parse::<IpAddr>()
.context("stable endpoint host is not an IP address")?;
ensure!(
matches!(host, IpAddr::V4(_)),
"route binding v2 stable endpoints use explicit IPv4 addresses"
);
let port = port
.parse::<u16>()
.context("stable endpoint port is not a u16")?;
ensure!(port > 0, "stable endpoint port is zero");
Ok((host, port))
parse_stable_socket(self.driver, &self.stable_endpoint, "stable endpoint")
}
/// The member listener's stable socket, when the route has one.
pub(crate) fn member_stable_socket(&self) -> Result<Option<(IpAddr, u16)>> {
self.member
.as_ref()
.map(|member| {
parse_stable_socket(self.driver, &member.stable_endpoint, "member endpoint")
})
.transpose()
}
/// The private port the workload binds for members when its presence
/// candidate is `candidate_port`. Derived, never stored: the member range
/// is the private range shifted to the member start.
pub(crate) fn member_candidate_port(&self, candidate_port: u16) -> Result<Option<u16>> {
let Some(member) = self.member.as_ref() else {
return Ok(None);
};
ensure!(
(self.private_port_start..=self.private_port_end).contains(&candidate_port),
"candidate port is outside the route binding"
);
let offset = candidate_port - self.private_port_start;
let port = member
.private_port_start
.checked_add(offset)
.context("member private port overflows")?;
Ok(Some(port))
}
}
fn parse_stable_socket(driver: RouteDriver, endpoint: &str, what: &str) -> Result<(IpAddr, u16)> {
let authority = match driver {
RouteDriver::NginxStreamTcp => endpoint
.strip_prefix("http://")
.or_else(|| endpoint.strip_prefix("tcp://")),
RouteDriver::NginxStreamUdp => endpoint.strip_prefix("rudp://"),
}
.with_context(|| format!("{what} transport does not match its route driver"))?;
ensure!(
!authority
.chars()
.any(|character| matches!(character, '/' | '?' | '#' | '@' | '[' | ']')),
"{what} is not a plain IP address and port"
);
let (host, port) = authority
.rsplit_once(':')
.with_context(|| format!("{what} has no port"))?;
let host = host
.parse::<IpAddr>()
.with_context(|| format!("{what} host is not an IP address"))?;
ensure!(
matches!(host, IpAddr::V4(_)),
"route binding v2 endpoints use explicit IPv4 addresses"
);
let port = port
.parse::<u16>()
.with_context(|| format!("{what} port is not a u16"))?;
ensure!(port > 0, "{what} port is zero");
Ok((host, port))
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
@ -836,6 +916,14 @@ impl TargetDeclaration {
if let Some(expected_sha256) = &artifact.expected_sha256 {
require_sha256(expected_sha256, "expected artifact sha256")?;
}
ensure!(
!(artifact.executable && !artifact.executable_members.is_empty()),
"artifact {} declares both executable and executable_members",
artifact.id
);
for member in &artifact.executable_members {
require_relative_path(member, "executable member", false)?;
}
ensure!(
destinations.insert(artifact.destination.as_path()),
"artifact destination {} is declared twice",
@ -919,6 +1007,14 @@ impl TargetDeclaration {
== self.service.route_required,
"routed services must require exactly one Idunn-owned candidate bind"
);
ensure!(
!self
.service
.required_environment
.contains(IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT)
|| self.service.route_required,
"only routed services may require the Idunn-owned member bind"
);
ensure!(
self.service
.required_environment
@ -1376,7 +1472,39 @@ impl OperatorBinding {
route.private_port_start < route.private_port_end,
"candidate rollout requires at least two private ports"
);
route.stable_socket()?;
let stable = route.stable_socket()?;
if let Some(member) = route.member.as_ref() {
ensure!(
route.driver == RouteDriver::NginxStreamTcp,
"only a TCP route may carry a member listener"
);
require_value(&member.stable_endpoint, "member endpoint")?;
let member_stable = route
.member_stable_socket()?
.context("route has no member listener")?;
ensure!(
member_stable != stable,
"member endpoint equals the route's stable endpoint"
);
ensure!(
member.private_port_start > 0,
"member private port range starts at zero"
);
ensure!(
member.max_connections > 0,
"member connection limit is zero"
);
let width = route.private_port_end - route.private_port_start;
let member_end = member
.private_port_start
.checked_add(width)
.context("member private port range overflows")?;
ensure!(
member_end < route.private_port_start
|| member.private_port_start > route.private_port_end,
"member private port range overlaps the route's private range"
);
}
require_absolute_path(&route.config_path, "route config")?;
require_unit(&route.reload_unit, "route reload unit")?;
}
@ -1554,12 +1682,7 @@ impl OperatorBinding {
.keys()
.chain(self.workload.secret_environment_names())
.collect();
for owned in [
IDUNN_RUNTIME_BUNDLE_ENVIRONMENT,
IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT,
IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT,
IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
] {
for owned in IDUNN_OWNED_LAUNCH_ENVIRONMENT {
ensure!(
!workload_names.contains(&owned.to_string()),
"operator binding cannot replace Idunn-owned launch environment {owned}"
@ -1581,6 +1704,9 @@ impl OperatorBinding {
}
if declaration.service.route_required {
available_environment.insert(IDUNN_RUNTIME_CANDIDATE_BIND_ENVIRONMENT.into());
if self.route.as_ref().is_some_and(|route| route.member.is_some()) {
available_environment.insert(IDUNN_RUNTIME_MEMBER_BIND_ENVIRONMENT.into());
}
}
if declaration.write_lease_required() {
available_environment.insert(IDUNN_PROCESS_WRITE_LEASE_ENVIRONMENT.into());
@ -1931,7 +2057,6 @@ fn require_host(value: &str) -> Result<()> {
pub(crate) mod tests {
use super::*;
// `idunn validate`'s tests in control_plane reuse these two fixtures.
pub(crate) const RECIPE: &str = r#"
schema = "gamecult.idunn.target_declaration.v1"
target = "test-service"
@ -2098,6 +2223,76 @@ nodes = ["yggdrasil"]
assert!(TargetDeclaration::parse(&input).is_err());
}
const MEMBER_CANARY: &str = "zzcanary";
fn recipe_with_members(members: &str) -> String {
let input = RECIPE.replace("executable = true", &format!("executable_members = [{members}]"));
assert_ne!(input, RECIPE, "the fixture lost its executable line");
input
}
/// A refused member list is refused by name of the field, never by echoing
/// the member.
fn assert_members_refused(members: &[&str]) {
let control = TargetDeclaration::parse(&recipe_with_members("\"bin/daemon\"")).unwrap();
assert!(control.artifacts[0].executable_members.contains(Path::new("bin/daemon")));
for member in members {
let error = TargetDeclaration::parse(&recipe_with_members(&format!("\"{member}\"")))
.expect_err(member);
assert!(
!format!("{error:#}").contains(MEMBER_CANARY),
"the refusal echoed a member path"
);
}
}
#[test]
fn recipe_executable_member_refuses_parent_traversal() {
assert_members_refused(&[
"zzcanary/../b",
"../zzcanary",
"bin/zzcanary/..",
"bin/../../zzcanary",
]);
}
#[test]
fn recipe_executable_member_refuses_absolute_path() {
assert_members_refused(&["/zzcanary/sh", "/zzcanary"]);
}
#[test]
fn recipe_executable_member_refuses_empty_and_current_dir() {
assert_members_refused(&["", ".", "./", "zzcanary/.."]);
}
#[test]
fn recipe_refuses_executable_flag_with_members() {
let input = RECIPE.replace(
"executable = true",
"executable = true\nexecutable_members = [\"bin/daemon\"]",
);
let error = TargetDeclaration::parse(&input).unwrap_err();
assert!(format!("{error:#}").contains("executable_members"));
assert!(!format!("{error:#}").contains("bin/daemon"));
// executable = true alone, and members alone, both stay admitted.
assert!(TargetDeclaration::parse(RECIPE).is_ok());
assert!(TargetDeclaration::parse(&recipe_with_members("\"bin/daemon\"")).is_ok());
}
#[test]
fn recipe_admits_members_on_both_source_kinds() {
let runner_output = TargetDeclaration::parse(&recipe_with_members("\"a/b\", \"c\"")).unwrap();
assert_eq!(runner_output.artifacts[0].source_kind, ArtifactSource::RunnerOutput);
assert_eq!(runner_output.artifacts[0].executable_members.len(), 2);
let worktree = recipe_with_members("\"a/b\", \"c\"")
.replace("source_kind = \"runner-output\"\nrunner = \"rust\"\n", "source_kind = \"worktree-tree\"\n")
.replace("expected_sha256 = \"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc\"\n", "");
let worktree = TargetDeclaration::parse(&worktree).unwrap();
assert_eq!(worktree.artifacts[0].source_kind, ArtifactSource::WorktreeTree);
assert_eq!(worktree.artifacts[0].executable_members.len(), 2);
}
#[test]
fn operator_cannot_omit_an_affordance_derived_from_the_recipe() {
let recipe = TargetDeclaration::parse(RECIPE).unwrap();
@ -2176,81 +2371,15 @@ nodes = ["yggdrasil"]
assert!(TargetDeclaration::parse(&input).is_err());
}
/// The refusal of a binding that sets a name its recipe does not declare.
/// A recipe declares a secret only as a `*_FILE` name, so this one rule is
/// what leaves a plaintext secret no place in a binding.
pub(crate) const UNDECLARED_INPUT_REFUSAL: &str =
"operator service environment contains an undeclared launch input";
/// RECIPE plus one declared plain input and one declared secret-file input.
pub(crate) fn recipe_declaring_inputs() -> String {
RECIPE.replace(
"required_environment = [",
"optional_environment = [\"DECLARED_INPUT\", \"DECLARED_SECRET_FILE\"]\nrequired_environment = [",
)
}
/// BINDING plus one `[workload.environment]` entry.
pub(crate) fn binding_with_environment(name: &str, value: &str) -> String {
BINDING.replace(
#[test]
fn extra_operator_environment_is_rejected() {
let recipe = TargetDeclaration::parse(RECIPE).unwrap();
let input = BINDING.replace(
"[workload.argument_bindings]",
&format!("[workload.environment]\n{name} = \"{value}\"\n\n[workload.argument_bindings]"),
)
}
/// BINDING plus one `[workload.secret_files]` entry.
fn binding_with_secret_file(name: &str) -> String {
BINDING.replace(
"[workload.secret_files]\n",
&format!(
"[workload.secret_files]\n{name} = \"/etc/gamecult/test-service/credentials/{name}\"\n"
),
)
}
fn admit_refusal(recipe: &TargetDeclaration, binding: &str) -> String {
let binding = OperatorBinding::parse(binding).unwrap();
format!("{:#}", binding.admit(recipe).unwrap_err())
}
#[test]
fn binding_environment_sets_only_declared_names() {
let recipe = TargetDeclaration::parse(&recipe_declaring_inputs()).unwrap();
OperatorBinding::parse(&binding_with_environment("DECLARED_INPUT", "plain"))
.unwrap()
.admit(&recipe)
.unwrap();
assert_eq!(
admit_refusal(&recipe, &binding_with_environment("UNDECLARED_INPUT", "plain")),
UNDECLARED_INPUT_REFUSAL
);
}
#[test]
fn binding_secret_files_set_only_declared_names() {
let recipe = TargetDeclaration::parse(&recipe_declaring_inputs()).unwrap();
OperatorBinding::parse(&binding_with_secret_file("DECLARED_SECRET_FILE"))
.unwrap()
.admit(&recipe)
.unwrap();
assert_eq!(
admit_refusal(&recipe, &binding_with_secret_file("UNDECLARED_SECRET_FILE")),
UNDECLARED_INPUT_REFUSAL
);
}
#[test]
fn declaring_a_secret_file_does_not_admit_its_plaintext_twin() {
// The shape of the Heimdall leak: the recipe declares only
// DECLARED_SECRET_FILE, and a binding puts the value itself under
// DECLARED_SECRET in plain environment.
let recipe = TargetDeclaration::parse(&recipe_declaring_inputs()).unwrap();
assert_eq!(
admit_refusal(&recipe, &binding_with_environment("DECLARED_SECRET", "plain")),
UNDECLARED_INPUT_REFUSAL
"[workload.environment]\nUNDECLARED = \"no\"\n\n[workload.argument_bindings]",
);
let binding = OperatorBinding::parse(&input).unwrap();
assert!(binding.admit(&recipe).is_err());
}
#[test]
@ -2297,6 +2426,203 @@ nodes = ["yggdrasil"]
}
}
/// The recipe and binding of a routed service whose members connect to a
/// listener of their own.
pub(crate) fn member_listener_pair() -> (String, String) {
let recipe = RECIPE.replace(
"required_environment = [",
"required_environment = [\"GAMECULT_IDUNN_MEMBER_BIND\", ",
);
let binding = BINDING.replace(
"reload_unit = \"nginx.service\"\n",
"reload_unit = \"nginx.service\"\n\n[route.member]\nstable_endpoint = \"http://127.0.0.1:18840\"\nprivate_port_start = 18841\n",
);
(recipe, binding)
}
fn member_binding_with(replacement: (&str, &str)) -> Result<OperatorBinding> {
let (recipe, binding) = member_listener_pair();
let recipe = TargetDeclaration::parse(&recipe)?;
let binding = OperatorBinding::parse(&binding.replace(replacement.0, replacement.1))?;
binding.admit(&recipe)?;
Ok(binding)
}
#[test]
fn a_member_listener_admits_with_a_recipe_that_requires_the_member_bind() {
let binding = member_binding_with(("never-matches", "")).unwrap();
let route = binding.route.unwrap();
assert_eq!(route.member_stable_socket().unwrap().unwrap().1, 18840);
assert_eq!(route.member_candidate_port(18831).unwrap(), Some(18841));
assert_eq!(route.member_candidate_port(18839).unwrap(), Some(18849));
assert!(route.member_candidate_port(18840).is_err());
}
#[test]
fn a_binding_without_a_member_listener_has_no_member_ports() {
let binding = OperatorBinding::parse(BINDING).unwrap();
let route = binding.route.unwrap();
assert_eq!(route.member_stable_socket().unwrap(), None);
assert_eq!(route.member_candidate_port(18831).unwrap(), None);
}
#[test]
fn a_member_listener_needs_a_recipe_that_requires_the_member_bind() {
let (recipe, binding) = member_listener_pair();
// Bound but not required: the recipe never asked for the variable.
let plain_recipe = TargetDeclaration::parse(RECIPE).unwrap();
assert!(
OperatorBinding::parse(&binding)
.unwrap()
.admit(&plain_recipe)
.is_err()
);
// Required but not bound: the variable would never be set.
let recipe = TargetDeclaration::parse(&recipe).unwrap();
assert!(
OperatorBinding::parse(BINDING)
.unwrap()
.admit(&recipe)
.is_err()
);
}
#[test]
fn only_a_routed_recipe_may_require_the_member_bind() {
let recipe = RECIPE
.replace("route_required = true", "route_required = false")
.replace(
"required_environment = [\"GAMECULT_IDUNN_CANDIDATE_BIND\", ",
"required_environment = [\"GAMECULT_IDUNN_MEMBER_BIND\", ",
);
assert!(recipe.contains("GAMECULT_IDUNN_MEMBER_BIND"));
assert!(TargetDeclaration::parse(&recipe).is_err());
}
#[test]
fn member_listener_bindings_are_validated() {
let start = |value: &str| {
member_binding_with((
"private_port_start = 18841",
&format!("private_port_start = {value}"),
))
};
// Overlapping the route's own private range, in either direction.
assert!(start("18835").is_err());
assert!(start("18825").is_err());
// Adjacent but disjoint is fine, above the route's range or below it.
assert!(start("18840").is_ok());
assert!(start("18822").is_ok());
assert!(start("18823").is_err());
// The connection limit defaults, and zero is refused.
let limit = |value: &str| {
member_binding_with((
"private_port_start = 18841",
&format!("private_port_start = 18841
max_connections = {value}"),
))
};
assert_eq!(
member_binding_with(("never-matches", "")).unwrap().route.unwrap().member.unwrap().max_connections,
128
);
assert_eq!(limit("7").unwrap().route.unwrap().member.unwrap().max_connections, 7);
assert!(limit("0").is_err());
assert!(limit("65535").is_ok());
assert!(limit("65536").is_err());
// Zero, and a width that overflows u16.
assert!(start("0").is_err());
assert!(start("65530").is_err());
// The member endpoint may not be the route's stable endpoint.
assert!(member_binding_with(("http://127.0.0.1:18840", "http://127.0.0.1:18830")).is_err());
// The same transport and address rules as the stable endpoint.
assert!(member_binding_with(("http://127.0.0.1:18840", "rudp://127.0.0.1:18840")).is_err());
assert!(member_binding_with(("http://127.0.0.1:18840", "http://example.test:18840")).is_err());
}
#[test]
fn a_udp_route_cannot_carry_a_member_listener() {
let (recipe, binding) = member_listener_pair();
let binding = binding
.replace("nginx-stream-tcp", "nginx-stream-udp")
.replace("http://127.0.0.1:18830", "rudp://127.0.0.1:18830")
.replace("http://127.0.0.1:18840", "rudp://127.0.0.1:18840");
let recipe = TargetDeclaration::parse(&recipe).unwrap();
assert!(
OperatorBinding::parse(&binding)
.and_then(|binding| binding.admit(&recipe))
.is_err()
);
}
#[test]
fn idunn_owned_names_are_reserved_in_one_list() {
let recipe = TargetDeclaration::parse(RECIPE).unwrap();
let owned = [
"GAMECULT_IDUNN_RUNTIME_BUNDLE",
"GAMECULT_IDUNN_CANDIDATE_BIND",
"GAMECULT_IDUNN_MEMBER_BIND",
"GAMECULT_IDUNN_PROCESS_WRITE_LEASE",
"GAMECULT_IDUNN_ACTIVATION_CREDENTIAL",
"CREDENTIALS_DIRECTORY",
];
let mut listed = IDUNN_OWNED_LAUNCH_ENVIRONMENT.to_vec();
listed.sort_unstable();
let mut expected = owned.to_vec();
expected.sort_unstable();
assert_eq!(listed, expected);
let as_environment = |name: &str| {
BINDING.replace(
"[workload.argument_bindings]",
&format!("[workload.environment]\n{name} = \"value\"\n\n[workload.argument_bindings]"),
)
};
let as_secret = |name: &str| {
BINDING.replace(
"[workload.secret_files]",
&format!("[workload.secret_files]\n{name} = \"/etc/gamecult/test-service/other.cc\""),
)
};
// The fixtures are well formed: an unowned name parses in both
// shapes, so the refusals below are about the name.
assert!(OperatorBinding::parse(&as_environment("UNOWNED_NAME")).is_ok());
assert!(OperatorBinding::parse(&as_secret("UNOWNED_NAME")).is_ok());
for name in owned {
// Refused when the binding is parsed...
assert!(
OperatorBinding::parse(&as_environment(name)).is_err(),
"{name} accepted as environment at parse"
);
assert!(
OperatorBinding::parse(&as_secret(name)).is_err(),
"{name} accepted as a secret file at parse"
);
// ...and again at admission, for a binding that did not come
// through parse. Each site reads the one list.
for secret in [false, true] {
let mut binding = OperatorBinding::parse(BINDING).unwrap();
let WorkloadBinding::SystemdTransient(workload) = &mut binding.workload else {
panic!("the fixture is a systemd workload");
};
if secret {
workload.secret_files.insert(name.into(), "/etc/gamecult/test-service/other.cc".into());
} else {
workload.environment.insert(name.into(), "forged".into());
}
assert!(
binding.workload.validate_launch_environment().is_err(),
"{name} (secret: {secret}) accepted by the binding check"
);
let error = binding.admit(&recipe).unwrap_err().to_string();
assert!(
error.contains("Idunn-owned launch environment"),
"{name} (secret: {secret}) accepted by the launch-input check: {error}"
);
}
}
}
#[test]
fn state_slot_path_cannot_escape_the_operator_owned_root() {
let input = RECIPE.replace(
@ -2328,6 +2654,17 @@ nodes = ["yggdrasil"]
assert!(declaration.write_lease_required());
}
#[test]
fn external_directory_state_is_lease_gated_like_external_file() {
let input = RECIPE.replace("kind = \"cultcache-file\"", "kind = \"external-directory\"");
let declaration = TargetDeclaration::parse(&input).unwrap();
let slot = &declaration.state.as_ref().unwrap().slots[0];
assert_eq!(slot.kind, StateKind::ExternalDirectory);
assert_eq!(slot.writer, StateWriter::ProcessBoundSingleWriter);
assert_eq!(slot.startup, StateStartup::CreateOrOpenAfterWriteLease);
assert!(declaration.write_lease_required());
}
#[test]
fn state_slots_cannot_claim_overlapping_paths() {
let input = RECIPE.replace(

View file

@ -467,15 +467,24 @@ pub fn select_dependencies(
/// overrides any of them; the resolved values are frozen into the plan.
pub const DEFAULT_FENCING_DEADLINE_SECONDS: u32 = 120;
pub const DEFAULT_LEASING_DEADLINE_SECONDS: u32 = 120;
// Unsafe for streampixels-service (startup = "create-or-open-after-write-lease"):
// its Postgres migrations run inside AwaitingReady. B5 requires an explicit
// binding value for stateful targets instead of this default.
// A stateful candidate may run migrations inside AwaitingReady (streampixels-
// service runs Postgres's there). When this passes, a deployment whose
// candidate was issued a write lease is left to the operator, not aborted; a
// continuity is aborted and its release restarted.
pub const DEFAULT_AWAITING_READY_DEADLINE_SECONDS: u32 = 300;
pub const DEFAULT_ROUTING_DEADLINE_SECONDS: u32 = 120;
pub const DEFAULT_COMMITTING_DEADLINE_SECONDS: u32 = 120;
/// The resolved per-phase deadline durations a plan carries. Nothing reads
/// them at the cut that introduced them.
/// Idunn's own deadlines, not binding-declared: a new `PhaseDeadlines` field
/// would change every v3 plan id (`recomputed_plan_id` re-encodes the plan).
/// Warming is timed for every transaction, owner or not.
pub const WARMING_DEADLINE_SECONDS: u32 = 600;
/// A pre-fencing abort is timed from its beginning.
pub const PRE_FENCING_ABORT_DEADLINE_SECONDS: u32 = 300;
/// The resolved per-phase deadline durations a plan carries. Entering a
/// post-fencing phase stamps its deadline from them, and the deadline
/// resolver ends the phase once it passes.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct PhaseDeadlines {
@ -725,6 +734,11 @@ pub struct ArtifactReceipt {
pub sha256: String,
pub size_bytes: u64,
pub executable: bool,
/// Files sealed executable inside a directory artifact. Skipped when empty
/// so a receipt with no members keeps its original encoding, and so its
/// sealed release id.
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
pub executable_members: BTreeSet<PathBuf>,
}
impl ArtifactReceipt {
@ -738,7 +752,9 @@ impl ArtifactReceipt {
.find(|artifact| artifact.id == self.artifact_id)
.with_context(|| format!("plan declares no artifact {}", self.artifact_id))?;
ensure!(
self.destination == declared.destination && self.executable == declared.executable,
self.destination == declared.destination
&& self.executable == declared.executable
&& self.executable_members == declared.executable_members,
"artifact receipt differs from declared output"
);
if let Some(expected_sha256) = &declared.expected_sha256 {
@ -1416,6 +1432,7 @@ nodes = ["yggdrasil"]
sha256: digest(b'c'),
size_bytes: 42,
executable: true,
executable_members: BTreeSet::new(),
}
}
@ -1782,6 +1799,113 @@ nodes = ["yggdrasil"]
assert!(changed.validate_against(&plan).is_err());
}
fn hex(bytes: &[u8]) -> String {
bytes.iter().map(|byte| format!("{byte:02x}")).collect()
}
#[test]
fn receipt_without_members_encodes_as_before() {
// Both values were computed outside this code, from the five-field
// msgpack array the receipt encoded to at base d74ac8d.
const BASE_RECEIPT: &str = "95a66461656d6f6ea773657276696365d9477368613235362d636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363636363632ac3";
const BASE_RELEASE_ID: &str =
"sha256-b9143d8f546ad71c493c70037813547aaec10fd87994a6f3bdc77ac3a8b3495e";
let receipt = artifact_receipt();
let encoded = rmp_serde::to_vec(&receipt).unwrap();
assert_eq!(hex(&encoded), BASE_RECEIPT);
let bytes = (0..BASE_RECEIPT.len())
.step_by(2)
.map(|at| u8::from_str_radix(&BASE_RECEIPT[at..at + 2], 16).unwrap())
.collect::<Vec<_>>();
assert_eq!(rmp_serde::from_slice::<ArtifactReceipt>(&bytes).unwrap(), receipt);
let release = SealedRelease {
schema: SEALED_RELEASE_SCHEMA.into(),
sealed_release_id: String::new(),
plan_id: digest(b'a'),
artifacts: vec![receipt],
external_inputs: vec![],
sealed_at_unix_millis: 0,
};
assert_eq!(release.recomputed_release_id().unwrap(), BASE_RELEASE_ID);
}
#[test]
fn sealed_release_address_covers_executable_members() {
let release_with = |members: &[&str]| SealedRelease {
schema: SEALED_RELEASE_SCHEMA.into(),
sealed_release_id: String::new(),
plan_id: digest(b'a'),
artifacts: vec![ArtifactReceipt {
executable: false,
executable_members: members.iter().map(PathBuf::from).collect(),
..artifact_receipt()
}],
external_inputs: vec![],
sealed_at_unix_millis: 0,
};
let none = release_with(&[]).recomputed_release_id().unwrap();
let one = release_with(&["bin/daemon"]).recomputed_release_id().unwrap();
let other = release_with(&["bin/other"]).recomputed_release_id().unwrap();
let two = release_with(&["bin/daemon", "bin/other"])
.recomputed_release_id()
.unwrap();
let all = [&none, &one, &other, &two];
for (index, left) in all.iter().enumerate() {
for right in &all[index + 1..] {
assert_ne!(left, right);
}
}
// The declaration is the other half: a receipt must carry exactly the
// declared list, in a plan that declares one and in one that does not.
let plain = plan();
let mut with_members = artifact_receipt();
with_members.executable_members = BTreeSet::from([PathBuf::from("bin/daemon")]);
assert!(
SealedRelease::new(&plain, vec![with_members], vec![external_input_receipt()], 120)
.is_err()
);
let recipe = RECIPE.replace("executable = true", "executable_members = [\"bin/daemon\", \"lib/tool\"]");
let declaring = compile_deployment_plan(
recipe.as_bytes(),
BINDING.as_bytes(),
source(&recipe),
"service-incarnation-1",
Some(18001),
110,
&[ready_odin_provider("odin", "odin-yggdrasil", 1)],
)
.unwrap();
let honest = ArtifactReceipt {
executable: false,
executable_members: BTreeSet::from([PathBuf::from("bin/daemon"), PathBuf::from("lib/tool")]),
..artifact_receipt()
};
let release = SealedRelease::new(
&declaring,
vec![honest.clone()],
vec![external_input_receipt()],
120,
)
.unwrap();
release.validate_against(&declaring).unwrap();
for drifted in [
BTreeSet::from([PathBuf::from("bin/daemon")]),
BTreeSet::from([PathBuf::from("bin/daemon"), PathBuf::from("lib/other")]),
BTreeSet::new(),
] {
let mut receipt = honest.clone();
receipt.executable_members = drifted;
assert!(
SealedRelease::new(&declaring, vec![receipt], vec![external_input_receipt()], 120)
.is_err()
);
}
let mut changed = release.clone();
changed.artifacts[0].executable_members.insert(PathBuf::from("lib/extra"));
assert!(changed.validate_against(&declaring).is_err());
}
fn binding_with_deadlines(table: &str) -> String {
BINDING.replace("[placement]", &format!("{table}\n[placement]"))
}

File diff suppressed because it is too large Load diff

View file

@ -30,13 +30,13 @@ use serde::Deserialize;
use crate::control_plane::read_trust_anchor;
use crate::deployment::{
ArtifactSource, HostWorkloadBinding, IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
HostWorkloadBinding, IDUNN_ACTIVATION_CREDENTIAL_ENVIRONMENT,
IDUNN_RUNTIME_BUNDLE_ENVIRONMENT, LaunchArgument, OperatorBinding, TargetDeclaration,
};
use crate::deployment_plan::{ArtifactReceipt, CompiledDeploymentPlan, SealedRelease};
use crate::deployment_plan::{CompiledDeploymentPlan, SealedRelease};
use crate::drivers::{
HostWorkloadObservation, InstalledReleaseObservation, copy_artifact, copy_tree,
digest_artifact, release_artifact, remove_tree_inside, sha256_id, write_runtime_bundle_records,
HostWorkloadObservation, InstalledReleaseObservation, copy_tree, digest_artifact,
release_artifact, remove_tree_inside, seal_artifact, sha256_id, write_runtime_bundle_records,
};
use crate::host_actuator::{
HOST_ACTUATOR_CHANNEL, HOST_ACTUATOR_CONNECTION_ID, HOST_ACTUATOR_PING_MS,
@ -1048,37 +1048,12 @@ fn materialize(
fs::create_dir_all(&staging)?;
let mut artifacts = Vec::new();
for artifact in &declaration.artifacts {
let source_path = match artifact.source_kind {
ArtifactSource::RunnerOutput | ArtifactSource::WorktreeTree => {
source.join(&artifact.source)
}
};
ensure!(
source_path.exists(),
"declared artifact output {} is absent",
artifact.id
);
let destination = staging.join(&artifact.destination);
ensure!(
destination.starts_with(&staging),
"artifact destination escaped its staging root"
);
copy_artifact(&source, &source_path, &destination)?;
let (sha256, size_bytes) = digest_artifact(&destination)?;
if let Some(expected) = &artifact.expected_sha256 {
ensure!(
&sha256 == expected,
"artifact {} differs from its recipe-pinned digest",
artifact.id
);
}
artifacts.push(ArtifactReceipt {
artifact_id: artifact.id.clone(),
destination: artifact.destination.clone(),
sha256: format!("sha256-{sha256}"),
size_bytes,
executable: artifact.executable,
});
artifacts.push(seal_artifact(
artifact,
&source,
&source.join(&artifact.source),
&staging,
)?);
}
let release = SealedRelease::new(plan, artifacts, Vec::new(), sealed_at_unix_millis)?;
Ok((release, staging))

View file

@ -963,6 +963,12 @@ impl WorkloadPort for HostActuatorWorkloadDriver<'_> {
};
Ok(!running)
}
/// A managed host's reboot is not yet observable here: follow-up
/// host-actuator-reboot. Until then continuity charges its restarts.
fn lost_to_host_reboot(&self, _: &WorkloadObservation, _: u64) -> Result<bool> {
Ok(false)
}
}
#[cfg(test)]
@ -1044,6 +1050,43 @@ mod tests {
Ok(())
}
/// A managed host's reboot is not observable yet (follow-up
/// host-actuator-reboot): the driver never claims a reboot, so continuity
/// never grants a free re-materialization it cannot justify.
#[test]
fn a_managed_host_workload_is_never_claimed_lost_to_a_reboot() -> Result<()> {
let (_temp, idunn, _host) = identities()?;
let hub = HostActuatorHub::bind("127.0.0.1:0".parse()?)?;
let driver = HostActuatorWorkloadDriver {
access: HostActuatorAccess {
hub: Arc::new(Mutex::new(hub)),
anchors: BTreeMap::new(),
signer: &idunn,
},
};
let observation = WorkloadObservation::Host(HostWorkloadObservation {
host: "raven".into(),
actuator_identity_id: "actuator".into(),
process_id: 1,
process_creation_time: 1,
session_id: 1,
user_sid: "S-1-5-21-1-2-3-1001".into(),
executable: "C:/GameCult/idunn/releases/muninn/x/muninn.exe".into(),
executable_sha256: format!("sha256-{}", "1".repeat(64)),
command_line_sha256: format!("sha256-{}", "2".repeat(64)),
environment_names: Vec::new(),
environment_contract_sha256: format!("sha256-{}", "3".repeat(64)),
runtime_bundle: "C:/GameCult/idunn/runtime/muninn/x".into(),
runtime_instance_id: format!("sha256-{}", "a".repeat(64)),
activation_signer_identity_id: "activation".into(),
activation_signer_public_key: vec![1; 32],
exit_code: Some(1),
});
assert!(!driver.lost_to_host_reboot(&observation, 1)?);
assert!(!driver.lost_to_host_reboot(&observation, u64::MAX)?);
Ok(())
}
/// A real hub and a real client socket on loopback: Hello attaches under
/// the bound anchor, a wrong host is refused, a request is answered.
#[test]
@ -1108,7 +1151,7 @@ mod tests {
let answerer = thread::spawn(move || -> Result<()> {
for _ in 0..500 {
client.poll_resends()?;
while let Some(frame) = client.receive_once()? {
if let Some(frame) = client.receive_once()? {
let envelope = HostActuatorEnvelope::decode(&frame.payload)?;
let request = envelope.open_request("raven", &idunn_anchor)?;
assert_eq!(envelope.session_nonce, nonce);

1046
src/incident.rs Normal file

File diff suppressed because it is too large Load diff

3238
src/launcher.rs Normal file

File diff suppressed because it is too large Load diff

View file

@ -4,4 +4,10 @@ pub mod deployment_plan;
pub mod drivers;
pub mod host;
pub mod host_actuator;
#[cfg(unix)]
pub mod launcher;
mod incident;
mod runner_evidence;
pub mod provisioning;
#[cfg(unix)]
mod steward_orders;

805
src/runner_evidence.rs Normal file
View file

@ -0,0 +1,805 @@
//! The output tail of a failed runner step.
//!
//! One file, `runner-evidence.cc`, beside `control.cc`, holds one type: the
//! last bytes of a failed runner container's stdout and stderr, one record per
//! transaction. The engine's pre-fence failure path is its only writer, and
//! `idunn status --command` its only reader. It is never `control.cc`, which
//! refuses any type it does not know, and never `incidents.cc`, which is
//! published and carries no free text.
//!
//! The tail is raw runner output. It is kept unredacted and readable by root
//! alone (mode 0600): root can already read every secret the runner was given.
//! It never enters an error, a transaction record or the journal. The status
//! reader is a terminal, so everything it prints from this store goes through
//! one allowlist sink (`emit`): printable ASCII passes, all else is escaped.
use std::path::{Path, PathBuf};
use anyhow::{Context, Result, bail, ensure};
use cultcache_rs::{
CultCacheEnvelope, DatabaseEntry, SingleFileMessagePackBackingStore,
TryCompareExchangeSnapshotOutcome,
};
use serde::{Deserialize, Serialize};
use crate::control_plane::{decode_record, require_id, typed_envelope};
use crate::drivers::{
LockContended, RUNNER_OUTPUT_TAIL_BYTES, RunnerStepFailure, RunnerSubject,
exchange_behind_private_lock, runner_status,
};
const RUNNER_FAILURE_SCHEMA: &str = "idunn.runner_failure.v1";
/// Records kept; the writer drops the oldest past this, so the store needs no
/// other owner.
pub(crate) const RUNNER_EVIDENCE_RECORDS: usize = 32;
/// `runner-evidence.cc`, beside the control store.
pub(crate) fn runner_evidence_store_path(state_store: &Path) -> PathBuf {
state_store.with_file_name("runner-evidence.cc")
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum SubjectKind {
Step,
ExternalInput,
}
#[derive(Clone, PartialEq, Eq, DatabaseEntry)]
#[cultcache(type = "idunn.runner_failure", schema = "idunn.runner_failure.v1")]
pub(crate) struct RunnerFailureRecord {
#[cultcache(key = 0)]
pub(crate) schema_version: String,
#[cultcache(key = 1)]
pub(crate) transaction_id: String,
#[cultcache(key = 2)]
pub(crate) command_id: String,
#[cultcache(key = 3)]
pub(crate) target: String,
#[cultcache(key = 4)]
pub(crate) subject_kind: SubjectKind,
#[cultcache(key = 5)]
pub(crate) subject_id: String,
#[cultcache(key = 6)]
pub(crate) runner: String,
#[cultcache(key = 7)]
pub(crate) exit_code: Option<i32>,
#[cultcache(key = 8)]
pub(crate) signal: Option<i32>,
#[cultcache(key = 9)]
pub(crate) stdout_tail: Vec<u8>,
#[cultcache(key = 10)]
pub(crate) stdout_total_bytes: u64,
#[cultcache(key = 11)]
pub(crate) stderr_tail: Vec<u8>,
#[cultcache(key = 12)]
pub(crate) stderr_total_bytes: u64,
#[cultcache(key = 13)]
pub(crate) recorded_at_unix_millis: u64,
}
/// Tails stay out of `Debug`, so a failed assertion or a logged record never
/// prints runner output.
impl std::fmt::Debug for RunnerFailureRecord {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("RunnerFailureRecord")
.field("transaction_id", &self.transaction_id)
.field("subject_id", &self.subject_id)
.finish_non_exhaustive()
}
}
impl RunnerFailureRecord {
pub(crate) fn from_failure(
transaction_id: &str,
command_id: &str,
target: &str,
failure: &RunnerStepFailure,
now: u64,
) -> Self {
let (subject_kind, subject_id) = match &failure.subject {
RunnerSubject::Step(id) => (SubjectKind::Step, id),
RunnerSubject::ExternalInput(id) => (SubjectKind::ExternalInput, id),
};
Self {
schema_version: RUNNER_FAILURE_SCHEMA.to_owned(),
transaction_id: transaction_id.to_owned(),
command_id: command_id.to_owned(),
target: target.to_owned(),
subject_kind,
subject_id: subject_id.clone(),
runner: failure.runner.clone(),
exit_code: failure.exit_code,
signal: failure.signal,
stdout_tail: failure.stdout_tail.clone(),
stdout_total_bytes: failure.stdout_total_bytes,
stderr_tail: failure.stderr_tail.clone(),
stderr_total_bytes: failure.stderr_total_bytes,
recorded_at_unix_millis: now,
}
}
pub(crate) fn validate(&self) -> Result<()> {
ensure!(
self.schema_version == RUNNER_FAILURE_SCHEMA,
"runner failure schema is unsupported"
);
require_id(&self.transaction_id, "runner failure transaction")?;
require_id(&self.command_id, "runner failure command")?;
require_id(&self.target, "runner failure target")?;
require_id(&self.subject_id, "runner failure subject")?;
require_id(&self.runner, "runner failure runner")?;
for (tail, total, stream) in [
(&self.stdout_tail, self.stdout_total_bytes, "stdout"),
(&self.stderr_tail, self.stderr_total_bytes, "stderr"),
] {
ensure!(
tail.len() <= RUNNER_OUTPUT_TAIL_BYTES,
"runner failure {stream} tail exceeds its bound"
);
ensure!(
tail.len() as u64 <= total,
"runner failure {stream} tail exceeds its stream"
);
}
Ok(())
}
fn envelope(&self) -> Result<CultCacheEnvelope> {
self.validate()?;
typed_envelope(
&self.transaction_id,
Self::TYPE,
RUNNER_FAILURE_SCHEMA,
self,
self.recorded_at_unix_millis,
)
}
}
/// One read of the store: the envelopes exactly as stored, and the records
/// decoded from them in the same order.
fn snapshot(path: &Path) -> Result<(Vec<CultCacheEnvelope>, Vec<RunnerFailureRecord>)> {
let envelopes = SingleFileMessagePackBackingStore::new(path)
.pull_all_read_only_snapshot()
.context("reading the runner evidence store")?;
let mut records = Vec::new();
for envelope in &envelopes {
ensure!(
envelope.r#type == RunnerFailureRecord::TYPE
&& envelope.schema_id.as_deref() == Some(RUNNER_FAILURE_SCHEMA),
"the runner evidence store holds a document that is not a runner failure"
);
let record: RunnerFailureRecord = decode_record(envelope)?;
record.validate()?;
ensure!(
envelope.key == record.transaction_id,
"runner evidence store key differs from the record's transaction"
);
records.push(record);
}
Ok((envelopes, records))
}
/// How a write left the store.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Kept {
Stored,
/// The store held bytes that do not decode; it was moved to
/// `unreadable_store_path` and the record opened a fresh one. `replaced`
/// is the last-written time (unix milliseconds) of an earlier set-aside
/// store this one displaced, which is gone: one slot, and it says so.
SetAside { replaced: Option<u64> },
}
/// Where an undecodable store is kept. One slot: the next one replaces it, so
/// the damage a bad store can cost is one extra file.
pub(crate) fn unreadable_store_path(path: &Path) -> PathBuf {
path.with_file_name("runner-evidence.unreadable.cc")
}
/// Moves the store over the previous set-aside one, root-only, and reports
/// what it displaced. Nothing of the store's content is named: a decode error
/// can quote stored values, and none is carried here.
fn set_aside_unreadable(path: &Path) -> Result<Option<u64>> {
let aside = unreadable_store_path(path);
let replaced = std::fs::symlink_metadata(&aside).ok().map(|earlier| {
earlier
.modified()
.ok()
.and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok())
.map_or(0, |since| since.as_millis() as u64)
});
std::fs::rename(path, &aside)
.context("the runner evidence store is undecodable and could not be set aside")?;
keep_private(&aside)?;
Ok(replaced)
}
/// How the writer reads the store's bytes; tests inject an I/O error here.
type ReadStore<'a> = &'a dyn Fn(&Path) -> std::io::Result<Vec<u8>>;
fn unreadable_fault(error: &std::io::Error) -> anyhow::Error {
anyhow::anyhow!("the runner evidence store could not be read: {}", error.kind())
}
/// The one writer. Upsert by transaction id; past `RUNNER_EVIDENCE_RECORDS` the
/// oldest records go in the same exchange. The file ends at mode 0600.
///
/// A store is set aside only when bytes that were read do not decode. An I/O
/// error leaves the store in place, keeps no evidence for that failure, and
/// the error names its kind for the journal.
pub(crate) fn record(path: &Path, record: &RunnerFailureRecord) -> Result<Kept> {
record_reading(path, record, &|path| std::fs::read(path))
}
fn record_reading(path: &Path, record: &RunnerFailureRecord, read: ReadStore) -> Result<Kept> {
let first = match read(path) {
Ok(bytes) => Some(bytes),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => None,
Err(error) => return Err(unreadable_fault(&error)),
};
let (envelopes, records, how) = match snapshot(path) {
Ok((envelopes, records)) => (envelopes, records, Kept::Stored),
Err(_) => {
let Some(first) = first else {
bail!("the runner evidence store could not be decoded");
};
// Undecodable only if the same bytes are still there to read: a
// second read that fails, or finds other bytes, is not a verdict.
match read(path) {
Ok(second) if second == first => {}
Ok(_) => bail!("the runner evidence store changed while it was being read"),
Err(error) => return Err(unreadable_fault(&error)),
}
let replaced = set_aside_unreadable(path)?;
(Vec::new(), Vec::new(), Kept::SetAside { replaced })
}
};
let mut kept: Vec<(u64, CultCacheEnvelope)> = envelopes
.iter()
.zip(&records)
.filter(|(_, stored)| stored.transaction_id != record.transaction_id)
.map(|(envelope, stored)| (stored.recorded_at_unix_millis, envelope.clone()))
.collect();
kept.push((record.recorded_at_unix_millis, record.envelope()?));
// Stable: records of one instant keep their stored order, so the record
// being written survives a tie.
kept.sort_by_key(|(recorded_at, _)| *recorded_at);
let excess = kept.len().saturating_sub(RUNNER_EVIDENCE_RECORDS);
let replacements: Vec<_> = kept
.into_iter()
.skip(excess)
.map(|(_, envelope)| envelope)
.collect();
match exchange_behind_private_lock(path, &envelopes, &replacements)? {
TryCompareExchangeSnapshotOutcome::Exchanged => {}
TryCompareExchangeSnapshotOutcome::Mismatch => {
bail!("the runner evidence store changed while it was being written")
}
TryCompareExchangeSnapshotOutcome::LockContended => {
return Err(LockContended {
store: "runner evidence store",
}
.into());
}
}
keep_private(path)?;
Ok(how)
}
#[cfg(unix)]
fn keep_private(path: &Path) -> Result<()> {
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))
.context("making the runner evidence store private")
}
#[cfg(not(unix))]
fn keep_private(_path: &Path) -> Result<()> {
Ok(())
}
/// The one reader. An absent store holds nothing.
pub(crate) fn for_transaction(
path: &Path,
transaction_id: &str,
) -> Result<Option<RunnerFailureRecord>> {
Ok(snapshot(path)?
.1
.into_iter()
.find(|stored| stored.transaction_id == transaction_id))
}
/// The record as `idunn status --command` shows it, as raw lines. They are
/// not text yet: `show_status` is the only caller, and `emit` the only
/// writer.
fn render(record: &RunnerFailureRecord) -> Vec<Vec<u8>> {
let noun = match record.subject_kind {
SubjectKind::Step => "step",
SubjectKind::ExternalInput => "external input",
};
let mut lines: Vec<Vec<u8>> = vec![
format!(" failed {noun} {}", record.subject_id).into_bytes(),
format!(" runner {}", record.runner).into_bytes(),
format!(" status {}", runner_status(record.exit_code, record.signal)).into_bytes(),
];
for (stream, tail, total) in [
("stdout", &record.stdout_tail, record.stdout_total_bytes),
("stderr", &record.stderr_tail, record.stderr_total_bytes),
] {
lines.push(format!(" {stream} (last {} of {total} bytes)", tail.len()).into_bytes());
let text = tail.strip_suffix(b"\n").unwrap_or(tail);
if !text.is_empty() {
lines.extend(text.split(|byte| *byte == b'\n').map(|line| {
let mut shown = b" | ".to_vec();
shown.extend_from_slice(line);
shown
}));
}
}
lines
}
/// What `idunn status --command` prints for a transaction's runner evidence:
/// its record, or one line naming why the store is unreadable. Nothing is
/// written when the transaction has no record.
pub(crate) fn show_status(
out: &mut impl std::io::Write,
store: &Path,
transaction_id: &str,
) -> std::io::Result<()> {
let lines = match for_transaction(store, transaction_id) {
Ok(Some(record)) => render(&record),
Ok(None) => return Ok(()),
Err(error) => {
vec![format!(" runner evidence unreadable: {}", error.root_cause()).into_bytes()]
}
};
emit(out, &lines)
}
/// The one sink: every byte the status read prints from the evidence store,
/// or from an error about it, is written here and nowhere else. An allowlist:
/// printable ASCII and tab pass; every other code point is a visible
/// `\u{..}` escape and every invalid UTF-8 byte a `\x..` escape. A newline
/// inside a line is escaped too: this function writes the only ones.
pub(crate) fn emit(out: &mut impl std::io::Write, lines: &[Vec<u8>]) -> std::io::Result<()> {
for line in lines {
out.write_all(escape(line).as_bytes())?;
out.write_all(b"\n")?;
}
Ok(())
}
pub(crate) fn escape(bytes: &[u8]) -> String {
let mut shown = String::with_capacity(bytes.len());
for chunk in bytes.utf8_chunks() {
for character in chunk.valid().chars() {
match character {
' '..='~' | '\t' => shown.push(character),
_ => shown.push_str(&format!("\\u{{{:x}}}", u32::from(character))),
}
}
for byte in chunk.invalid() {
shown.push_str(&format!("\\x{byte:02x}"));
}
}
shown
}
#[cfg(all(test, unix))]
mod tests {
use super::*;
use std::os::unix::fs::PermissionsExt;
fn failure(subject: RunnerSubject, stdout: &[u8], stderr: &[u8]) -> RunnerStepFailure {
RunnerStepFailure {
subject,
runner: "rust".into(),
exit_code: Some(3),
signal: None,
stdout_tail: stdout.to_vec(),
stdout_total_bytes: stdout.len() as u64,
stderr_tail: stderr.to_vec(),
stderr_total_bytes: stderr.len() as u64,
}
}
/// What status writes for a record, through the one sink.
fn shown(record: &RunnerFailureRecord) -> String {
let mut written = Vec::new();
emit(&mut written, &render(record)).expect("a Vec takes any write");
String::from_utf8(written).expect("the sink writes ASCII only")
}
fn shown_lines(record: &RunnerFailureRecord) -> Vec<String> {
shown(record).lines().map(str::to_owned).collect()
}
fn record_at(transaction: &str, now: u64) -> RunnerFailureRecord {
RunnerFailureRecord::from_failure(
transaction,
"up-service",
"service",
&failure(RunnerSubject::Step("build".into()), b"out\n", b"err\n"),
now,
)
}
#[test]
fn the_evidence_store_keeps_the_newest_32() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
// The literal, not the constant: the spec's bound is 32.
for index in 0..=32 {
record(&path, &record_at(&format!("tx-{index:02}"), 1000 + index as u64))?;
}
let kept: Vec<_> = snapshot(&path)?.1.into_iter().map(|stored| stored.transaction_id).collect();
assert_eq!(kept.len(), 32);
assert!(!kept.contains(&"tx-00".to_owned()), "the oldest survived");
assert!(kept.contains(&"tx-32".to_owned()), "the newest was evicted");
assert!(kept.contains(&"tx-01".to_owned()), "the second oldest was evicted early");
// Rewriting a resident transaction replaces it and drops nothing.
let mut again = record_at("tx-05", 5000);
again.stdout_tail = b"rewritten\n".to_vec();
again.stdout_total_bytes = 10;
record(&path, &again)?;
let kept = snapshot(&path)?.1;
assert_eq!(kept.len(), 32);
assert!(kept.iter().any(|stored| stored.transaction_id == "tx-01"));
let stored = for_transaction(&path, "tx-05")?.context("rewritten record is gone")?;
assert_eq!(stored.stdout_tail, b"rewritten\n");
assert_eq!(for_transaction(&path, "tx-00")?, None);
Ok(())
}
#[test]
fn a_record_round_trips_byte_for_byte() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
let written = RunnerFailureRecord::from_failure(
"tx-a",
"up-service",
"service",
&failure(RunnerSubject::ExternalInput("toolchain".into()), &[0, 1, 255, 10], b""),
77,
);
record(&path, &written)?;
assert_eq!(for_transaction(&path, "tx-a")?, Some(written));
assert_eq!(for_transaction(&path, "tx-b")?, None);
Ok(())
}
#[test]
fn the_evidence_store_is_private() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
// Whatever the umask, the mode is the store's own rule, on every write.
for index in 0..2 {
record(&path, &record_at(&format!("tx-{index}"), 10 + index))?;
assert_eq!(std::fs::metadata(&path)?.permissions().mode() & 0o777, 0o600);
}
Ok(())
}
#[test]
fn a_tail_over_its_bound_or_its_stream_is_refused() {
let mut oversize = record_at("tx-a", 1);
oversize.stdout_tail = vec![b'x'; RUNNER_OUTPUT_TAIL_BYTES + 1];
oversize.stdout_total_bytes = u64::MAX;
assert!(oversize.validate().is_err());
let mut impossible = record_at("tx-a", 1);
impossible.stderr_total_bytes = 1;
assert!(impossible.validate().is_err());
let mut exact = record_at("tx-a", 1);
exact.stdout_tail = vec![b'x'; RUNNER_OUTPUT_TAIL_BYTES];
exact.stdout_total_bytes = RUNNER_OUTPUT_TAIL_BYTES as u64;
assert!(exact.validate().is_ok());
}
#[test]
fn status_renders_the_tail_with_control_bytes_escaped() {
let mut stored = record_at("tx-a", 1);
stored.stdout_tail = b"first\n\x1b[2Jsecond\x00end\tTab\nlast\n".to_vec();
stored.stdout_total_bytes = 5000;
stored.stderr_tail = b"bad\r\x07".to_vec();
stored.stderr_total_bytes = 6;
let lines = shown_lines(&stored);
let text = lines.join("\n");
assert!(!text.contains('\u{1b}') && !text.contains('\0') && !text.contains('\r'));
assert!(!text.contains('\u{7}'));
assert_eq!(
lines,
vec![
" failed step build",
" runner rust",
" status exit status 3",
" stdout (last 30 of 5000 bytes)",
" | first",
" | \\u{1b}[2Jsecond\\u{0}end\tTab",
" | last",
" stderr (last 5 of 6 bytes)",
" | bad\\u{d}\\u{7}",
]
);
}
#[test]
fn debug_names_the_record_and_prints_no_tail() {
let mut stored = record_at("tx-a", 1);
stored.stdout_tail = b"SECRET-OUTPUT".to_vec();
stored.stdout_total_bytes = 13;
let debug = format!("{stored:?}");
assert!(debug.contains("tx-a") && !debug.contains("SECRET"), "{debug}");
}
#[test]
fn an_empty_tail_renders_only_its_header() {
let mut stored = record_at("tx-a", 1);
stored.stdout_tail = Vec::new();
stored.stdout_total_bytes = 0;
let lines = shown_lines(&stored);
assert!(lines.contains(&" stdout (last 0 of 0 bytes)".to_owned()));
assert!(!lines.iter().any(|line| line == " | "));
}
#[test]
fn the_reader_names_an_unreadable_store_as_an_error() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
std::fs::write(&path, b"\xc1 not a store")?;
assert!(for_transaction(&path, "tx-a").is_err());
Ok(())
}
fn files_beside(path: &Path) -> Result<Vec<String>> {
let mut names: Vec<String> = std::fs::read_dir(path.parent().context("no parent")?)?
.map(|entry| Ok(entry?.file_name().to_string_lossy().into_owned()))
.collect::<Result<_>>()?;
names.retain(|name| name.starts_with("runner-evidence") && !name.contains("lock"));
names.sort();
Ok(names)
}
/// A store a newer Idunn wrote: well-formed, but a schema this one does not know.
fn write_future_store(path: &Path) -> Result<()> {
let mut future = record_at("tx-future", 1);
future.schema_version = "idunn.runner_failure.v2".into();
let envelope = typed_envelope(
&future.transaction_id,
RunnerFailureRecord::TYPE,
"idunn.runner_failure.v2",
&future,
1,
)?;
let outcome = exchange_behind_private_lock(path, &[], &[envelope])?;
ensure!(
matches!(outcome, TryCompareExchangeSnapshotOutcome::Exchanged),
"the future store was not written"
);
Ok(())
}
#[test]
fn a_corrupt_store_is_set_aside_and_the_next_failure_keeps_its_evidence() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
let damage = b"\xc1 not a store \x00 damage";
std::fs::write(&path, damage)?;
let kept = record(&path, &record_at("tx-a", 5))?;
assert_eq!(kept, Kept::SetAside { replaced: None });
// Kept, not deleted: the damage is where the journal says, root-only.
let aside = unreadable_store_path(&path);
assert_eq!(std::fs::read(&aside)?, damage);
assert_eq!(std::fs::metadata(&aside)?.permissions().mode() & 0o777, 0o600);
// The failure's evidence is in a fresh, private store.
assert_eq!(
for_transaction(&path, "tx-a")?.map(|stored| stored.transaction_id),
Some("tx-a".to_owned())
);
assert_eq!(std::fs::metadata(&path)?.permissions().mode() & 0o777, 0o600);
// A healthy store is not set aside again.
assert_eq!(record(&path, &record_at("tx-b", 6))?, Kept::Stored);
assert_eq!(std::fs::read(&aside)?, damage);
Ok(())
}
#[test]
fn a_future_version_store_is_set_aside_and_the_next_failure_keeps_its_evidence() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
write_future_store(&path)?;
let before = std::fs::read(&path)?;
assert!(for_transaction(&path, "tx-future").is_err());
assert_eq!(
record(&path, &record_at("tx-a", 5))?,
Kept::SetAside { replaced: None }
);
assert_eq!(std::fs::read(unreadable_store_path(&path))?, before);
assert!(for_transaction(&path, "tx-a")?.is_some());
assert_eq!(for_transaction(&path, "tx-future")?, None);
Ok(())
}
#[test]
fn at_most_one_unreadable_store_is_kept() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
let damages: [&[u8]; 3] = [b"\xc1 first", b"\xc1 second", b"\xc1 third"];
for (index, damage) in damages.into_iter().enumerate() {
std::fs::write(&path, damage)?;
record(&path, &record_at(&format!("tx-{index}"), 5 + index as u64))?;
assert_eq!(
std::fs::read(unreadable_store_path(&path))?,
damage,
"the newest damage is the one kept"
);
}
assert_eq!(
files_beside(&path)?,
vec![
"runner-evidence.cc".to_owned(),
"runner-evidence.unreadable.cc".to_owned()
]
);
Ok(())
}
#[test]
fn a_store_that_cannot_be_read_is_left_in_place_and_the_write_fails() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
// A directory where the store should be: unreadable, not undecodable.
std::fs::create_dir(&path)?;
assert!(record(&path, &record_at("tx-a", 1)).is_err());
assert!(path.is_dir());
assert!(!unreadable_store_path(&path).exists());
Ok(())
}
#[test]
fn every_code_point_but_printable_ascii_is_a_visible_escape() {
// The allowlist, exhaustively: no class list decides anything.
for code in 0..=0x10ffffu32 {
let Some(character) = char::from_u32(code) else {
continue;
};
let text = character.to_string();
let shown = escape(text.as_bytes());
if matches!(character, ' '..='~' | '\t') {
assert_eq!(shown, text, "U+{code:04X} should pass");
} else {
assert_eq!(shown, format!("\\u{{{code:x}}}"), "U+{code:04X}");
}
}
// Invalid UTF-8 is escaped byte by byte; valid text round it passes.
assert_eq!(escape(b"a\xc1\xffb\xe2\x82"), "a\\xc1\\xffb\\xe2\\x82");
assert_eq!(escape(b"ok \xe2\x82\xac"), "ok \\u{20ac}");
}
#[test]
fn a_newline_inside_a_line_cannot_forge_a_line() {
let mut stored = record_at("tx-a", 1);
stored.subject_id = "build\n status exit status 0".into();
let lines = shown_lines(&stored);
assert_eq!(lines[0], " failed step build\\u{a} status exit status 0");
assert_eq!(
lines.iter().filter(|line| line.starts_with(" status ")).count(),
1
);
}
/// A store whose first record is well-formed except that its subject kind
/// holds `marker`: the decode error quotes it.
fn store_with_hostile_subject_kind(path: &Path, marker: &[u8; 4]) -> Result<()> {
record(path, &record_at("tx-a", 1))?;
let mut bytes = std::fs::read(path)?;
let at = bytes
.windows(4)
.position(|window| window == b"step")
.context("no step marker")?;
bytes[at..at + 4].copy_from_slice(marker);
std::fs::write(path, bytes)?;
Ok(())
}
#[test]
fn the_unreadable_store_line_is_escaped_with_the_rest() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
store_with_hostile_subject_kind(&path, b"\x1b[2J")?;
let mut written = Vec::new();
show_status(&mut written, &path, "tx-a")?;
assert!(!written.contains(&0x1b), "an escape byte reached the terminal");
assert!(written.iter().all(|byte| matches!(byte, b' '..=b'~' | b'\t' | b'\n')));
let text = String::from_utf8(written)?;
assert!(text.starts_with(" runner evidence unreadable: "), "{text}");
assert!(text.contains("\\u{1b}[2J"), "the quoted value was not shown escaped: {text}");
Ok(())
}
#[test]
fn a_failed_set_aside_names_itself_keeps_the_store_and_echoes_nothing() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
store_with_hostile_subject_kind(&path, b"CNRY")?;
let damage = std::fs::read(&path)?;
// Sanity: this damage decodes to an error that quotes the canary.
assert!(format!("{:#}", snapshot(&path).unwrap_err()).contains("CNRY"));
// A non-empty directory where the aside goes: rename fails even as root.
let aside = unreadable_store_path(&path);
std::fs::create_dir(&aside)?;
std::fs::write(aside.join("occupied"), b"x")?;
let error = record(&path, &record_at("tx-b", 2)).unwrap_err();
for text in [format!("{error:#}"), format!("{error:?}")] {
assert!(text.contains("could not be set aside"), "{text}");
assert!(!text.contains("CNRY"), "stored content reached the error: {text}");
}
// The failure cost nothing: the store is as it was, with no evidence
// written over it.
assert_eq!(std::fs::read(&path)?, damage);
assert!(aside.join("occupied").exists());
Ok(())
}
#[test]
fn an_io_error_leaves_the_store_in_place_and_keeps_no_evidence() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
record(&path, &record_at("tx-a", 1))?;
let before = std::fs::read(&path)?;
let denied = || std::io::Error::from(std::io::ErrorKind::PermissionDenied);
// The first read fails on a healthy store.
let error = record_reading(&path, &record_at("tx-b", 2), &|_| Err(denied())).unwrap_err();
assert!(format!("{error:#}").contains("permission denied"), "{error:#}");
assert_eq!(std::fs::read(&path)?, before, "the store changed");
assert!(!unreadable_store_path(&path).exists());
assert!(for_transaction(&path, "tx-b")?.is_none());
// The second read, the one that confirms the bytes do not decode, fails.
std::fs::write(&path, b"\xc1 not a store")?;
let calls = std::cell::Cell::new(0usize);
let read = |path: &Path| {
calls.set(calls.get() + 1);
if calls.get() == 2 {
return Err(denied());
}
std::fs::read(path)
};
let error = record_reading(&path, &record_at("tx-b", 2), &read).unwrap_err();
assert!(format!("{error:#}").contains("permission denied"), "{error:#}");
assert_eq!(std::fs::read(&path)?, b"\xc1 not a store");
assert!(!unreadable_store_path(&path).exists());
// Bytes that changed between the reads are no verdict either.
let calls = std::cell::Cell::new(0usize);
let read = |path: &Path| -> std::io::Result<Vec<u8>> {
calls.set(calls.get() + 1);
let mut bytes = std::fs::read(path)?;
bytes.push(calls.get() as u8);
Ok(bytes)
};
assert!(record_reading(&path, &record_at("tx-b", 2), &read).is_err());
assert_eq!(std::fs::read(&path)?, b"\xc1 not a store");
assert!(!unreadable_store_path(&path).exists());
Ok(())
}
#[test]
fn a_second_set_aside_says_it_replaced_the_first() -> Result<()> {
let temp = tempfile::tempdir()?;
let path = runner_evidence_store_path(&temp.path().join("control.cc"));
std::fs::write(&path, b"\xc1 first")?;
assert_eq!(
record(&path, &record_at("tx-a", 1))?,
Kept::SetAside { replaced: None }
);
std::fs::write(&path, b"\xc1 second")?;
let Kept::SetAside { replaced: Some(_) } = record(&path, &record_at("tx-b", 2))? else {
bail!("a displaced set-aside store went unreported");
};
assert_eq!(std::fs::read(unreadable_store_path(&path))?, b"\xc1 second");
Ok(())
}
}

4079
src/steward_orders.rs Normal file

File diff suppressed because it is too large Load diff

View file

@ -10,3 +10,19 @@ It is checked in because the TypeScript runtime emits a camelCase MessagePack
map while Rust `DatabaseEntry` derives tuple payloads. Idunn must prove it can
decode the producer's real wire shape rather than round-tripping a fixture it
serialized itself.
# systemd 257 `systemctl show` captures
`systemd-257/*.show` are the verbatim output of systemd 257.13 for the 27
properties `show_unit` requested at `112a7a5`, captured read-only on Yggdrasil
on 2026-10-24: `vili-presence-identity-only.show` (a failed idunn-vili unit whose
only credential is the presence identity) and
`odin-presence-and-one-service-credential.show` (the live Odin unit). systemd 257
prints `LoadCredential=[unprintable]` in both. Tests read these bytes; do not edit.
# Pre-cut systemd observation records
`pre-cut-systemd-observation.{compact,named}.hex` are the two MessagePack
encodings of a `SystemdWorkloadObservation` with `load_credential` set to
`[unprintable]`, produced by the code at `112a7a5` (the form live incumbents
persisted). They pin that later Idunns still decode such records.

Binary file not shown.

View file

@ -0,0 +1 @@
dc0033d94e6964756e6e2d616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161612e73657276696365aa4964756e6e2074657374a3696e7601a465786563a26e6fa56d69786564c3a6753631303030a6753631303030a0a0a0c3c3a9696e76697369626c65a3616c6cc3a430303037a0ad5b756e7072696e7461626c655dcd10920194cdee48cdee48cdee48cdee4894cdee48cdee48cdee48cdee4891cdee480000000000c391010708b52f6f70742f746573742f62696e2f736572766963650101d9477368613235362d31313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131d9477368613235362d61616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161a92f6f70742f74657374a92f72756e2f74657374d9477368613235362d3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323290d9477368613235362d33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333d9202f73797374656d2e736c6963652f6964756e6e2d746573742e73657276696365c090aa61637469766174696f6edc0020010101010101010101010101010101010101010101010101010101010101010190

View file

@ -0,0 +1 @@
de0033a4756e6974d94e6964756e6e2d616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161612e73657276696365b0756e69745f6465736372697074696f6eaa4964756e6e2074657374ad696e766f636174696f6e5f6964a3696e76d923657865635f6d61696e5f73746172745f74696d657374616d705f6d6f6e6f746f6e696301ac736572766963655f74797065a465786563ae726573746172745f706f6c696379a26e6fa96b696c6c5f6d6f6465a56d69786564ac64796e616d69635f75736572c3ac73797374656d645f75736572a6753631303030ad73797374656d645f67726f7570a6753631303030b4737570706c656d656e746172795f67726f757073a0b76361706162696c6974795f626f756e64696e675f736574a0b4616d6269656e745f6361706162696c6974696573a0ae707269766174655f6d6f756e7473c3ac707269766174655f70696473c3ac70726f746563745f70726f63a9696e76697369626c65ab70726f635f737562736574a3616c6cb16e6f5f6e65775f70726976696c65676573c3a5756d61736ba430303037b2696e61636365737369626c655f7061746873a0af6c6f61645f63726564656e7469616cad5b756e7072696e7461626c655da86d61696e5f706964cd1092b270726f636573735f73746172745f74696d6501ac70726f636573735f7569647394cdee48cdee48cdee48cdee48ac70726f636573735f6769647394cdee48cdee48cdee48cdee48ae70726f636573735f67726f75707391cdee48b770726f636573735f6361705f696e686572697461626c6500b570726f636573735f6361705f7065726d697474656400b570726f636573735f6361705f65666665637469766500b470726f636573735f6361705f626f756e64696e6700b370726f636573735f6361705f616d6269656e7400b970726f636573735f6e6f5f6e65775f70726976696c65676573c3b670726f636573735f6e616d6573706163655f706964739101b26d6f756e745f6e616d6573706163655f696407b07069645f6e616d6573706163655f696408aa65786563757461626c65b52f6f70742f746573742f62696e2f73657276696365b165786563757461626c655f64657669636501b065786563757461626c655f696e6f646501b165786563757461626c655f736861323536d9477368613235362d31313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131313131b372756e74696d655f696e7374616e63655f6964d9477368613235362d61616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161b1776f726b696e675f6469726563746f7279a92f6f70742f74657374ae72756e74696d655f62756e646c65a92f72756e2f74657374b3636f6d6d616e645f6c696e655f736861323536d9477368613235362d32323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232b1656e7669726f6e6d656e745f6e616d657390bb656e7669726f6e6d656e745f636f6e74726163745f736861323536d9477368613235362d33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333ad636f6e74726f6c5f67726f7570d9202f73797374656d2e736c6963652f6964756e6e2d746573742e73657276696365b563726564656e7469616c735f6469726563746f7279c0bc706172656e745f6f6e6c795f66696c655f64657363726970746f727390bd61637469766174696f6e5f7369676e65725f6964656e746974795f6964aa61637469766174696f6ebc61637469766174696f6e5f7369676e65725f7075626c69635f6b6579dc00200101010101010101010101010101010101010101010101010101010101010101b3736572766963655f63726564656e7469616c7390

View file

@ -0,0 +1,28 @@
Type=exec
Restart=no
MainPID=2571
OpenFile=/run/idunn/activation-credentials/sha256-7cfe5cd1e87839e029c7425eb1b586b6b0d94dd79d7376eed3b156386f2c7af8.credential:gamecult-idunn-runtime-activation-key:read-only
OpenFile=/etc/gamecult/odin/runtime-presence-identity.cc:gamecult-runtime-presence-identity:read-only
ExecMainStartTimestampMonotonic=20630000
ControlGroup=/system.slice/idunn-odin-7cfe5cd1e87839e029c7425eb1b586b6b0d94dd79d7376eed3b156386f2c7af8.service
UMask=0007
WorkingDirectory=/srv/odin/releases/sha256-e5c7d0b2b9c0a8ffba368aeb08abdbe37ae0d17484a785e60a66ec4a62b2f18c
CapabilityBoundingSet=
AmbientCapabilities=
User=_duc23369f1b8b167db
Group=odin
DynamicUser=yes
LoadCredential=[unprintable]
SupplementaryGroups=
InaccessiblePaths=/run/idunn/activation-credentials
PrivateMounts=yes
PrivatePIDs=yes
NoNewPrivileges=yes
ProtectProc=invisible
ProcSubset=all
KillMode=mixed
Description=Idunn odin sha256-7cfe5cd1e87839e029c7425eb1b586b6b0d94dd79d7376eed3b156386f2c7af8
LoadState=loaded
ActiveState=active
SubState=running
InvocationID=ada48921043449f39dacc87b14a1bf70

View file

@ -0,0 +1,28 @@
Type=exec
Restart=no
MainPID=0
OpenFile=/run/idunn/activation-credentials/sha256-b507e1705a70f8eee72f355e7db637acab8eb37880b5d25245fda68d80f63041.credential:gamecult-idunn-runtime-activation-key:read-only
OpenFile=/etc/gamecult/vili/runtime-presence-identity.cc:gamecult-runtime-presence-identity:read-only
ExecMainStartTimestampMonotonic=520083420000
ControlGroup=
UMask=0007
WorkingDirectory=/srv/vili-idunn/releases/sha256-aeaee4e1669742572c45a3f26bc86893c44b2a4149520975eb7d8a28aa931906
CapabilityBoundingSet=
AmbientCapabilities=
User=_du8c06c563aac6d205
Group=vili-state
DynamicUser=yes
LoadCredential=[unprintable]
SupplementaryGroups=
InaccessiblePaths=/run/idunn/activation-credentials
PrivateMounts=yes
PrivatePIDs=yes
NoNewPrivileges=yes
ProtectProc=invisible
ProcSubset=all
KillMode=mixed
Description=Idunn vili sha256-b507e1705a70f8eee72f355e7db637acab8eb37880b5d25245fda68d80f63041
LoadState=loaded
ActiveState=failed
SubState=failed
InvocationID=8f7a349f2f8f49fe87ac5d89239260b4