Idunn/docs/reincarnation-map.md
Meta Vens 2f8671ac35 Reincarnation map: cut 2 installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 02:36:25 +02:00

600 lines
41 KiB
Markdown

# Reincarnation after reboot: map
Status: map, Imagination, 2026-10-18 (campaign `idunn-reincarnation`, session
`idunn-reincarnation-2026-10-18`). The ends are in
[`reincarnation-target.md`](reincarnation-target.md). Questions, rulings, the
cut spec and follow-ups live in Eureka's mind; this page holds what has
no kind: body facts, the model page, the authority map's rationale and the
cut sequence. Line numbers are at `74a7008`; `src/` is byte-identical to
`c2477d4`, so the earlier anchors still hold.
Cut 2 pass, Imagination, 2026-10-19 (same campaign and session): body facts
B13-B18, the cut-2 rows of the model page and the cut-2 parts of sections 3-5
are anchored at `9a0a6a2`, whose `src/` is byte-identical to `ffb4293` (cut 1's
head). The B1-B12 anchors above stay at `74a7008`.
Read the in-force set with the rulings-in-force and open-follow-ups recipes
for root `idunn-reincarnation`; the cut in flight is the specs-with-no-report
recipe. The first spec (`cut-c1-durable-stop.r1`) was withdrawn when the
operator ruled that continuity never stops; its replacement is cut
`c1-never-stop`. Cited from `idunn-watchdog`: `exhaustion-close-rule-op` and
`paging-conditions`.
## 1. Body facts
Evidence: the `idunn-yggdrasil` journal from 11:40 UTC (94,670 lines), the
per-unit journals of the streampixels-service candidates and of Odin
(`idunn-odin-7cfe5cd1…`), `idunn status` and `idunn status --command <id>`
on Yggdrasil, `journalctl -u nginx`, and the source at `c2477d4`. All times
UTC, 2026-10-18.
**B1. The reboot did not stop continuity from re-materializing anything.**
Every streampixels-service continuity attempt minted a fresh transient unit
from the installed release `sha256-df663f0a…`: 54 continuity commands from
`continuity-85887ef2` (11:46) to `continuity-52bbbd66` (19:57), most listed by `idunn status` as
`continuity-… failed systemd unit idunn-streampixels-service-<hash>.service
is not running`. A continuity transaction is a full `DeploymentTransaction`
built by `from_continuity` (`control_plane.rs:1678`) from the admitted plan,
sealed release, installed release and Expected. It runs every phase:
Starting, Warming, Fencing, Leasing, AwaitingReady, Routing, Committing. The
absent admitted unit `461c6443…` was never the problem. Each new candidate
was.
**B2. Every candidate died of a dependency, not of itself.** The
streampixels-service process exits about 30 s after start when Odin does not
admit its initial presence. Unit `2bc02905…` at 11:47:21 logged "StreamPixels
service failed to enter its Idunn incarnation: Error: Odin did not admit the
initial Idunn runtime presence ... Timed out waiting for CultNet RUDP response
after 5000 ms." The same is logged by `42ecf363…` at 13:47:13, `03acf44d…`
at 19:47:14, `45276008…` at 19:49:54 and `599b705b…` at 19:51:18. Odin's
RUDP endpoint is `rudp://10.77.0.1:17871`, which nginx proxies. nginx failed
at 11:46:26 ("bind() to 10.77.0.1:17871 failed (99: Cannot assign requested
address)") and started at 19:46:24. Between 11:46 and 19:44 Odin's journal
shows no presence message from streampixels-service at all: none arrived.
After nginx returned, the first one arrived at 19:49:25 and was rejected:
"runtime presence has no exact current activation and provider anchor". The
attempts minted at about 19:53 and 19:55 got past Fencing and then failed
"the Leasing phase passed its deadline" (FailedAfterFencing, RestoreIncumbent).
Other early failures: "observed write lease is not the current Idunn grant"
(11:46:47, the first attempt after boot), and three "reading process stat
/proc/N/stat: No such file or directory" races.
**B3. Every attempt was charged at mint.** `record_restart(now)` runs inside
the mint CAS (`control_plane.rs:6156-6157`). Nothing reads a continuity's
outcome back into the log. The only exemption anywhere is `HostUnobservable`
(`:5944`), and it applies to an observation, not to a candidate. So the 6-per-
hour window (`CONTINUITY_RESTART_ATTEMPTS` = 6, window 3,600,000 ms, `:71-72`)
filled at 11:50:52 and refilled every hour as entries slid out. The bursts are
in the journal at :47, :48, :49, :51, :52 and :53 of every hour, 39 times
"Idunn stopped restarting admitted streampixels-service". This answers Q2:
the nginx outage counted against the body's budget, and the place that decides
it is the unconditional charge at `:6156`.
**B4. The deployment was killed by a window slide, not by a live
continuity.** `up-3ad6b05b` (release `sha256-67e227b2…`, a new release) was
recorded at 20:24:51. It was still before Fencing at 20:46:41, when it was
completed `FailedBeforeFencing` with "admitted incumbent failed before
candidate fencing; deployment yielded to continuity: systemd unit
idunn-streampixels-service-461c6443….service is absent". Two seconds later
continuity minted `tx-0804185a` with the OLD release `df663f0a…`, and it was
admitted. The restart that left the window was minted at about 19:46:40 (its
unit `03acf44d…` started at 19:46:42), exactly one window before the abort.
The mechanism, from the source:
- `:5805-5821`: a live transaction for the target is the `blocker`. From
Fencing on (or Starting for stop-then-start) supervision skips the target
entirely.
- `:6033-6061`: with a blocker present, continuity never restarts and never
reports. It may abort the blocker, and the predicate for that is
`Deploy && !supervision.restarts_exhausted(now) && phase < Fencing && no
abort && not held` (`:6045-6050`).
- `:6062-6084`: only with no blocker does an exhausted continuity open the
incident and say "The target is free for a deployment to replace it".
So from 20:24:51 the deployment owned the target (`owns_target_authority`,
phase != Complete, `:1772`). Continuity deferred to it through the blocker
`continue` at `:6060`. The deployment deferred to continuity whenever the
window had a free slot, through `:6046`. "Stopped" existed only as a durable
incident (incidents.cc, a projection) and a report. The ownership decision
read the window. When the window gained a slot, continuity reclaimed the
target with no record that it had resumed. This answers Q3: the two owners are
the live `DeploymentTransaction` (it owns while not Complete) and
continuity-as-window (it owns whenever the window is not full). Continuity
defers to the deployment while any transaction is live. The deployment defers
to continuity while the window has room. The "free for a deployment" message
names the first ownership, and the code acts on the second.
**B5. Fencing an absent incumbent already works.** `advance_fencing`
(`:6869-6952`) stops the exact incumbent and revokes its lease. The systemd
driver's `stop` returns `Ok(())` for an absent unit (`drivers.rs:4003-4007`).
A deployment that is not aborted therefore replaces an absent incumbent with
nothing to fence, which is target invariant 3's last clause, already true.
`exact_incumbent` (`:7556-7569`) fails "target incumbent changed after
transaction creation" if a continuity commit replaced the generation during
the deployment. That is why a phase-split handover needs a rebase
(`handover-pre-fencing`).
**B6. The two rejection streams are consequences of a dead admitted
generation, not separate faults.** "preserved admitted streampixels-service
after rejecting topology observation: topology correlation does not bind the
current activation" was logged 46,271 times, from 11:46:49 to exactly
20:46:48, about 5,300 an hour. Ghostlight's 42,482 identical lines ran to
19:56:29. Since 20:47, raven-muninn, newly exhausted, shows the same line
621 times. `refresh_admitted_topology` (`:6395-6457`) authenticates Odin's
correlation against the admitted generation's activation. When that
incarnation is dead and Odin's latest record names another one (a
continuity candidate's, or a stale one), authentication fails with an `Err`
rather than `Ok(false)`, and `:5834-5840` prints it on every pass. The pass
order is what does it: route supervision (`:5822`) and topology refresh
(`:5834`) run before the workload is observed (`:5891`). So a dead
generation is "supervised" every tick. "rejected admitted
streampixels-service route continuity: admitted process write lease is no
longer exact" (3,615, from 20:02:23 to 20:46:48) followed the two
post-fencing aborts at 19:54 and 19:57. Fencing revoked the dead incumbent's
lease, "restore incumbent" cannot re-grant a lease to a process that does not
exist, and `supervise_admitted_route` (`:6250-6255`) then refuses the
admitted generation on every challenge. Both streams stopped the moment
`tx-0804185a` was admitted. This answers Q4: both streams follow from one
model gap, that a dead admitted generation is treated as a live incumbent.
That gap is cut 4. They are not the deadlock.
**B7. Ghostlight is a separate fault.** Its continuity `tx-039b91e9` is
wedged in a post-fencing abort. It was "lease-not-adopted" at its AwaitingReady
deadline, and the abort withdraws the candidate's lease (`:8895-8928`) before
it stops the candidate (`:8929-8946`). The candidate unit `6c51e35e…` has
been running since 19:55:15 and holds the lease's shared lock, so every
attempt fails "process write lease is held by a shared-lock consumer"
(`drivers.rs:4179/4224`). The transaction owns the target, so continuity is
locked out and the route degraded (53 consecutive failures). This is follow-up
`ghostlight-abort-lease-before-stop` (route-continuity invariant 5), not the
stopped-owner deadlock.
**B8. Unexplained, recorded.** Idunn's record of Odin's stable route reads
"degraded-since 11:46:31, consecutive-failures 1, next-challenge-at 11:47:01"
about nine hours later. It has not been re-challenged since. This is follow-up
`odin-route-record-stale`. It blocks cut 3. Under `budget-never-stop`, a
failure attributed to a declared dependency must not grow the backoff, and the
only dependency-health record Idunn holds for Odin's stable route is this
one.
**B9. Luna's weekend commits on this path (Q5).** None of them caused the
deadlock. The yield predicate dates from the route-continuity and watchdog
campaigns, not from the weekend. Where they touch:
| Commit | On the path? | Status |
|---|---|---|
| `9f00e7a` replay provider receipts at admission time | Yes: the deployment's Sealing compile (`advance_sealing` → `current_ready_provider_tokens`) logged "excluded non-current provider ghostlight / raven-muninn: admitted provider's latest receipt is not Ready at its admission time" at 20:24:53 (`:8588`, `:8609`) | Still live: route-continuity B4 has not landed. It did not block this deployment |
| `29946bb` wait for the stable route listener | Yes: the route proofs during the outage were "Connection refused" on 127.0.0.1:8831-8833 | Kept by route-continuity (honest scope) |
| `8ae00a1` cancel live stateless candidates | No: no cancel ran | Still present (`:2888`); route-continuity S2 owns its deletion |
| `ae8ac45`, `8651906`, `e976d32`, `5e94eb8` identity and launch evidence | Only in `observe` and `stop`; nothing here drifted | Kept |
| `a148802` | No | Reverted (`87fa57d`) |
The target forbids building on them. Cut 1 touches none of their lines.
**B10. The stop was the only door a deployment had into a crash loop.**
Source read at `74a7008`. A queued `idunn up` becomes a transaction only in
`freeze_command`, which refuses while any transaction for any of its targets
`blocks_new_target_mutation` (`:5741-5747`); the command just stays queued.
The scheduler tick runs `retire_one_terminal_transaction` (returns early when
it retires), then `resume_one_transaction`, then
`supervise_one_admitted_generation`, then `freeze_one_queued_command`
(`:5414-5423`). Today's doubling wait is anchored at the previous attempt's
**mint** (`next_restart_at`, `:1163-1170`). So once an attempt lasts longer than
its wait, the tick after its transaction is retired mints the next attempt
before the freeze runs, and the queued deployment never finds the target
free. At `c2477d4` that ended when the window filled and continuity stopped.
Delete the stop without opening another door and a crash-looping release can
never be replaced. The door the cut uses is kubelet's own anchor
(`docs/research/restart-budget-prior-art.md` §1.1, the exited container's
`FinishedAt`): the wait runs from the first supervision pass that finds the
target dead **and free**. That leaves at least one wait (5 s or more) with no
live transaction, and the freeze runs every tick. This is reasoning over the
source, not a probe. The cut pins it with a tick-level test.
**B11. The yield cannot outlive the stop.** With the window term gone from
`:6046`, the yield aborts every pre-fencing Deploy whenever the incumbent is
dead (`:6045-6059`). For a crash-looping release the incumbent is always
dead, so the deployment that would fix it is aborted at the first
supervision pass after it is frozen. That is the 2026-10-18 failure: on that
day the operator's `idunn up` existed to replace a release that would not
start. Ruling `handover-phase-split` already says nothing yields before
Fencing. So cut 1 deletes the whole yield. Continuity keeps deferring to any
live transaction for its target (`:6033`, the `continue`). Cut 2 keeps
the incumbent alive alongside the build. Between them, a dead incumbent stays
down while a pre-fencing deployment over it builds. That is bounded by the
deployment's own deadlines, and the deployment then fences it as nothing
(B5).
**B12. The doubling function already exists once.** `backoff_wait`
(`:4000-4007`) doubles from a base up to `RESUME_BACKOFF_CEILING_MILLIS`, and
has one caller (`back_off`, `:5621`). Continuity's wait is a second spelling
of the same rule (`5_000 << (live.len() - 1)`, `:1169`). The cut gives
`backoff_wait` a ceiling parameter and deletes the second spelling.
`#[cultcache(key = N, default)]` is how an absent key decodes as its default
(precedent `DeploymentTransaction` key 34, `:1552-1557`). New
`TargetSupervision` keys use it, because every record on Yggdrasil predates
them.
**B13. Two live transactions for one target make the control store
unreadable.** Source read at `9a0a6a2`. `ControlSnapshot::validate` refuses
"multiple transactions claim current-incarnation authority for one target"
over `owns_target_authority` (`:3245-3254`), and that predicate is just
`phase != Complete` (`:1896-1898`). Every reader of the store goes through it:
the scheduler tick, `idunn status`, cancel, expire. So if supervision minted a
continuity beside a pre-fencing deployment today, the next read would fail
the whole engine. Coexistence needs the identity rule changed first, and that
is the cut's first commit.
**B14. One predicate answers four questions, and the phase rule lives in
one caller.** `blocks_new_target_mutation` (`:1903-1909`, live or post-commit
cleanup pending) is read by the freeze busy check (`:5934-5942`), the
supervision blocker (`:5998-6001`, a single `.find`), candidate-port
reservation (`:7997`) and status (`:4613`). `owns_target_authority` is read by
validation (`:3249`) and by sibling ordering inside one command (`:3384`).
Neither looks at the phase. The only phase-aware ownership rule is inline in
supervision (`:6002-6012`, owns from Fencing, or from Starting for
stop-then-start). The freeze refusal is why a `profile:` command needs every
target free in one tick (follow-up `profile-deploy-joint-gap`).
**B15. What a pre-fencing deployment reads about its incumbent.** Warming's
semantic warming check uses the incumbent's lease digest
(`:6904-6905`, `is_semantic_warming` `:9530-9560`: Odin reports the candidate's
one disagreement as the lease the incumbent holds). Route preflight uses the
incumbent's route observation (`:7026-7047`). Isolation is proven against the
incumbent's workload (`:7050-7076`). Each goes through `exact_incumbent`
(`:7768-7781`), which bails "target incumbent changed after transaction
creation" once a continuity commit replaced the generation. Sealing, and
Starting for candidate-then-promote, read no incumbent. Stop-then-start's
Starting stops it (`:6857-6863`), which is already the owner's act. The
Sealing-to-Starting write is a `replace_transaction` at `:6806-6808`, not
`transition` (`:8914-8927`), so the two edges at which a transaction can take
a target are written by two different functions today.
**B16. The commit already guards the incumbent; nothing guards the
handover.** `advance_committing` expects the admitted envelope it read
(`:7701-7702`, CAS `:7743-7764`), so a deployment can never commit over a
generation that moved under it. Entering Fencing is an unguarded
`replace_transaction` of the transaction alone. Continuity's mint CAS
(`:6371-6400`) guards its command, transaction and supervision record, and no
other transaction. The store accepts read-only expectations: an expected
entry with no replacement only guards (CultLib
`packages/cultcache-rs/src/lib.rs:1327-1368`). One engine holds the store
(`ProcessLock::acquire`, `:5514`), but the CLI writes transactions too:
`cancel` sets a pre-fencing abort on a live deployment (`:3014-3083`). So a
handover decided by one CAS over the target's records is exact, and a tick
order is not.
**B17. What coexisting candidates share, and what they do not.** Continuity
reuses the admitted plan, so its candidate port is the admitted generation's,
already reserved (`:7988-7992`); a deployment picks another. Topology is keyed
per incarnation (`:8495-8498`): a continuity publishes under the admitted key,
a deployment under its own. No candidate holds a write lease before its own
Leasing, which follows Fencing, so two pre-fencing candidates cannot both
write. Route actuation by a continuity is always `Survival`, never refused
(`:4870-4876`); a deployment's `Forward` cannot starve it. The Odin replay
floor `max_odin_sequence` (`:3399-3435`) spans every transaction of the target
and the admitted cursor; incumbent refresh and candidate warming already share
it during every deployment today, so the replay check (`:9498-9501`) gains no
new exposure. Stop-then-start is declared only by route-less host-actuator
bindings (`deployment.rs:756-764`, `:1245-1249`), and no control-plane test
builds one (grep `StopThenStart` in `src/control_plane.rs`: the definition
site only). It is live on Yggdrasil: raven-muninn declares it
(`git grep stop-then-start` in `gamecult-ops`:
`idunn/yggdrasil/bindings/raven-muninn.toml.in`). So the Starting edge of the
handover carries production traffic, and its engine-level proof is owed
before the closing install (follow-up `stop-first-handover-fixture`).
**B18. A pre-fencing error is an abort.** `resume_candidate` turns any `Err`
from a pre-fencing step into `begin_pre_fencing_abort` unless it is
`LockContended` (`:5768-5783`). So anything the cut makes a deployment wait on
must return `Ok` (a gate wait, `record_gate_wait` `:8170-8182`), or the wait
becomes the yield that ruling `handover-phase-split` forbids. The same path is
how a superseded continuity ends itself, which is wanted.
## 2. Model page
One row per persistent kind the campaign touches.
| Kind | What names it | Lifecycle | Who decides |
|---|---|---|---|
| Paging log: `TargetSupervision.continuity_restarts` (control.cc, `idunn.target_supervision.v1`, key 3) | key = target | The newest 6 charged attempt times, ascending, read through a sliding 1 h window. Outlives every generation. After cut 1 it is the paging threshold only (`restarts_page`): it opens and holds the ContinuityExhausted incident and is shown in status. It gates no mint, wait or yield | Continuity supervision, pushed in the mint CAS |
| Attempt streak (new, cut 1: key 7, `default`) | target | Charged attempts since the admitted body last ran a sustained run. +1 in each charged mint CAS. Reset to 0 when the dead generation was admitted after the last charged attempt and ran ≥ 600 s before it was found dead. Saturating, and the wait is capped, so it can be unbounded | Continuity supervision only |
| Backoff anchor (new, cut 1: key 8, `default`) | target | Written on the first pass that finds the admitted generation dead and the target free after an attempt. The next attempt is due at anchor + `backoff_wait(5 s, streak - 1, 300 s)`. Cleared by every mint. Stale, and ignored, once older than the admitted generation's `admitted_at` | Continuity supervision only |
| Re-materialization bound (new, cut 1: key 9, `default`) | target | Set by the one attempt that is not a backoff step, when the driver reports a reboot loss. Spent while it is not older than the admitted generation's `admitted_at` | Continuity supervision, from the workload driver's evidence |
| Projection deferral (keys 4-5) | target | Unchanged: a failed demotion defers the next mint 30 s. It spaces a re-materialization too | Continuity supervision |
| Admitted generation (`idunn.admitted_generation`) | key = target, `generation_id` = `generation-<tx>` | Replaced atomically by every committed deployment or continuity. A dead incarnation keeps its record until replaced | The committing transaction |
| Incident `ContinuityExhausted` (incidents.cc) | condition + subject (target) | Opened at 6 charged attempts in the hour while the target is dead and free. Closed when the target runs and the window no longer holds 6 (`exhaustion-close-rule-op`). Retired after retention. The name is the wire name and stays, though nothing is exhausted any more | Idunn's incident writer, a notification projection: never read by ownership |
| Transaction and pre-fencing abort (`idunn.deployment_transaction.v4`, history.cc once terminal) | `tx-<uuid>` and its `command_id` (`up-…` or `continuity-…`). Keyed by transaction, so a continuity and a deployment for one target are two records with no shared key | Sealing … Complete. A pre-fencing abort completes `FailedBeforeFencing`, a post-fencing abort `FailedAfterFencing` with a named recovery. Retired to history. After cut 2, per target at any instant: at most one live Deploy, at most one live Continuity, at most one owner (validated by `ControlSnapshot::validate`, B13) | The scheduler advances it. After cut 1, continuity supervision never aborts a transaction; the transaction's own steps do. After cut 2, a continuity whose target was taken, or whose source generation was replaced, aborts itself (superseded) |
| Target ownership (cut 2: derived, no record) | target | A transaction owns its target from `takes_target_at()` (Fencing, or Starting for stop-then-start) until Complete; a Complete transaction with pending post-commit cleanup drains it. Entered only through the handover CAS (`Engine::take_target`) | Derived from the phase. The handover CAS is the one writer of a phase entry into `takes_target_at()` |
| `incumbent_generation_id` on a transaction (key 9, unchanged format) | the generation's id | Deploy: set at freeze; before the handover it names the generation its incumbent-relative evidence was proven against and is rebased when the admitted generation moves (clearing that evidence); pinned by the handover CAS, exact from then on. Continuity: its source generation, never rebased | Deploy: its own pre-handover rebase step and the handover CAS. Continuity: the mint |
| Unit materialization (systemd transient unit `idunn-<target>-<hash>`) | the activation's hash | Created by Starting. Lost on a host reboot (`/run/systemd/transient`). A failed unit stays loaded; a cleanly stopped one is collected | The workload driver actuates. Whether a loss was a reboot is the driver's evidence (`WorkloadPort::lost_to_host_reboot`, cut 1) |
No cell is open. Both forks are ruled (`budget-never-stop`,
`handover-phase-split`). The numbers that `budget-never-stop` left open are
the defaulted ruling `continuity-backoff-defaults`. Cut 2 opens one question,
`fencing-over-owning-continuity` (does a deployment ready to fence wait for a
continuity that already owns the target, or preempt it); the spec is written
for the recommended answer, wait.
## 3. Authority map: continuity and deployment
The authority map for cut 1 is in its spec. This section is the
campaign-level map, and cut 1 is one step of it.
- **Owner.** Exactly one authority actuates a target at each instant. A live
`DeploymentTransaction` owns the target from Fencing on, and a stop-first
rollout owns it from Starting. Before Fencing (`handover-phase-split`), a
deployment owns only its private candidate, and continuity owns the
incumbent slot. Continuity owns the admitted generation's survival and
never stops. When it next acts is a function of its own record: the
streak, the backoff anchor and the re-materialization bound.
- **Inputs.** For continuity: the admitted generation (id, `admitted_at`,
workload, plan), the workload observation and the driver's reboot
evidence, its own record, and whether a live transaction blocks the
target. After cut 3 it also reads Idunn's dependency-health observation.
For a deployment: its own plan, its brake, and at Fencing whatever
generation is admitted at that moment (cut 2).
- **Outputs.** Continuity: mints, either charged or the one re-materialization;
backoff anchors; deferrals; and, as notifications only, the
ContinuityExhausted incident and its once-report. Deployment: a new admitted
generation, or an abort that names its recovery.
- **Derived state.** `restarts_page(now)` (6 charged attempts in the hour) is
derived from the paging log and is notification-only: incident open and
close, and status. The next attempt's due time is derived from the
anchor and the streak. The incident and `idunn status` are projections.
- **Forbidden writers.** The paging window gates no mint, no wait and no
yield. Continuity supervision never aborts a deployment:
`begin_pre_fencing_abort` has no caller in it. No deployment path writes
continuity's fields. The incident store decides nothing. No stopped state
exists: no field, and no branch that ends a pass because a count is
reached.
- **Shared paths.** A crash restart, a restart after a sustained run, and a
reboot re-materialization all mint through the one CAS (`:6158-6183`). The
dead-and-free pass is the only writer of the backoff anchor. Every
deployment, operator or scheduled, enters through `freeze_command` and
advances through `advance_transaction`.
- **Deletion line.** Cut 1 deletes the yield (`:6034-6059`), the
exhaustion `continue` and its "free for a deployment" text (`:6062-6085`),
and the mint-anchored wait (`:1160-1184`). Cut 2 (anchors at `9a0a6a2`)
deletes `owns_target_authority` and `blocks_new_target_mutation`
(`:1894-1909`), the inline phase rule and the single-blocker `.find` in
supervision (`:5998-6012`), the blocker `continue` (`:6229-6233`), the
freeze busy check (`:5934-5942`), the one-live-transaction-per-target
validation (`:3245-3254`), and the "target incumbent changed after
transaction creation" text (`:7779`).
### Cut 2: the two slots and the handover
Each target has one **incumbent slot**: its `AdmittedGeneration`, the body
running it, its write lease and its stable-route membership. Each live
transaction has its own **candidate slot**: its incarnation key, activation,
candidate port and unit.
- **Owner.** Before any transaction takes the target, continuity supervision
owns the incumbent slot: it alone decides to mint a restart. From the
handover CAS until Complete, the one transaction that took the target owns
it, fencing, leasing, routing and committing. Its commit hands the slot
back to supervision as a new generation. A transaction owns its own
candidate slot from mint or freeze until it is terminal. A pre-handover
deployment owns nothing else.
- **The handover.** `Engine::take_target` is the only writer of a phase entry
into `takes_target_at()` (Fencing, or Starting for stop-then-start), for
both kinds. It is one CAS. It expects the transaction's own envelope, the
target's admitted envelope (the one `incumbent_generation_id` names, or
absent), and every other non-terminal transaction of the target, each
unchanged, and it writes only the transaction. Of two transactions
entering from one view, exactly one CAS succeeds. The loser's next step
re-reads: a deployment gate-waits for the owner, and a continuity aborts
itself as superseded.
- **Inputs.** Continuity: as cut 1, plus whether any transaction owns or
drains the target, or a continuity is live for it (the mint gate). A
pre-handover deployment: its plan, its brake, and the admitted generation as
it is now, which it rebases onto. The handover: the target's records above.
- **Outputs.** The handover writes the new phase. A rebase rewrites
`incumbent_generation_id` and clears the evidence proven against the old
incumbent: `warming`, `route_preflight`, `isolation`. A superseded continuity
writes its own pre-fencing abort.
- **Derived state.** `owns_target()` is derived from the phase and
`takes_target_at()`. `drains_incumbent()` is derived from the post-commit
cleanup. The freeze refusal is derived from "a live Deploy for this
target". `incumbent_generation_id` on a pre-handover deployment is the
generation its evidence was proven against, not a pin.
- **Forbidden writers.** No path enters `takes_target_at()` except
`take_target`. Continuity supervision aborts nothing, and mints nothing
while another transaction owns or drains the target or a continuity is
live. A deployment never aborts because another transaction owns the
target: it gate-waits (B18). A continuity is never rebased. Freeze reads no
continuity state. No new persistent record decides ownership: it is the
phase.
- **Shared paths.** Candidate-then-promote's Warming-to-Fencing and
stop-then-start's Sealing-to-Starting go through one phase-entry seam,
which calls `take_target` exactly when the entered phase is
`takes_target_at()`. Deployments and continuities use the same CAS. A
single-target and a `profile:` command use the same freeze rule.
Demotions, named: `incumbent_generation_id` is no longer a pin before the
handover; it is derived from the admitted generation by the rebase.
`blocks_new_target_mutation` is no longer an owner; it splits into
`is_live`, `owns_target` and `drains_incumbent`, each read by the consumer
that asks that question. The freeze busy check is no longer a
target-ownership gate; it guards only one live deployment per target.
Demotions, named: `restarts_exhausted` is no longer an owner of anything. It
is renamed `restarts_page`, and it opens and holds the incident. The
ContinuityExhausted incident was never read as "stopped", and there is now no
stopped state for it to stand for. The "free for a deployment" report is
deleted: every target is free for a deployment between attempts (B10).
## 4. Rationale
The fault is a split brain, not a missing rule. Continuity's "I have stopped"
was written in two places, a durable incident and a log message, and neither
was what the ownership decision read: the yield re-derived "stopped" from a
clock. The withdrawn first spec moved "stopped" into control state. The
operator's ruling is better, and it is also smaller: there is no stopped
state. The window it was derived from becomes a pager. Every case the stop
existed for is then handled by an owner that already exists. A broken release
is replaced by a deployment, which gets in through the gap after each
attempt (B10) and is never yielded (B11). A host-dependency outage costs one
attempt every five minutes or so, and pages.
Why the wait is anchored at "found dead and free" rather than at the mint:
the anchor is the door. A mint-anchored wait can be swallowed by a long
attempt, and then no tick ever sees the target free (B10). Kubelet anchors
CrashLoopBackOff on `FinishedAt` for the same reason, so this is the
established spelling, not a new one. A streak is stored, not recomputed from
the paging log. The log holds only six entries and slides by the hour, while
the streak must outlive both. Reading the backoff from the window count was
today's spelling, and it is exactly the mutant the cut's tests must kill.
The reset, "admitted after the last charged attempt and ran at least 600 s
before it was found dead", needs no new writer. Supervision observes the
admitted generation on every pass, so the first pass that finds it dead is
within one poll of the death. A death hidden behind a live transaction or an
unobservable host overstates the run, and so resets early. That is the safe
side, because it shortens one wait.
The reboot invariant needs evidence of what happened, not a guess. "Unit
absent" alone is not a reboot, because a cleanly stopped transient unit is
collected too. "Host booted after admission" alone is not a loss, because
the unit might have been started this boot. The systemd driver owns both
facts (a `systemctl show` that returns nothing, and the `/proc/stat` btime).
So it answers one question, `lost_to_host_reboot`, and supervision keeps the
budget. The once-per-admission bound is still needed under never-stop. A
failing re-materialization leaves the same admitted generation, unit absent,
booted after admission. Without the bound it would repeat with no step, no
wait and no page, every tick. The clock comparison is safe: admission
happens at least one full transaction, tens of seconds, after boot.
What cut 1 deliberately does not fix. A dependency failure still grows the
streak (cut 3). A dead incumbent stays down while a pre-fencing deployment
builds over it (cut 2). A dead generation is still route-supervised and
topology-refreshed every tick (cut 4). Paging residual: the cap gives 3600 /
(300 + D) attempts an hour, where D is the attempt's duration, so a crash
loop whose attempts last longer than 300 s settles under 6 an hour. Its
incident opens in the first hour's burst and stays open while the release
never runs, but a release that runs for one pass after each restart can see
it close and not reopen. That is the operator's threshold, applied as ruled.
It is recorded here, not changed.
Replayed against 2026-10-18 with only cut 1. The reboot loss at 11:46 is
re-materialized at once, with no step. It fails on the nginx outage. The
next attempt resets the streak, because the generation had run for hours,
and then waits 5, 10, 20, 40, 80, 160 s, and 300 s after that. The incident
opens within the first hour and stays open (the release never runs), so the
operator is paged during the outage, not at 20:24. The `idunn up` at 20:24
freezes in the next gap. Continuity defers to it, nothing aborts it, and it
fences the absent incumbent as nothing and admits the new release.
Why cut 2 is shaped this way. The handover is the phase, guarded by one CAS
over the target's records, rather than a new per-target "owner" record. A
record would be a second spelling of what the phase already says, and it
would need its own lifecycle, crash repair and migration. A CAS that reads
every other live transaction of the target is enough because a newly minted
or frozen transaction starts at Sealing and cannot own anything (B16).
The rebase clears exactly the evidence that names an incumbent (B15): warming
names its lease, preflight its route and isolation its workload. Keeping any
of them would let the deployment fence one generation with proofs about
another. A deployment waits in Warming while another transaction owns the
target, because Warming's steps read the incumbent that owner is fencing, and
a failed pre-fencing step is an abort (B18). Sealing and Starting keep
running, so the long part, the build, still overlaps. A continuity that loses
the target is aborted, not parked: it restarts a generation that is about to
be replaced, and while it lives it blocks continuity of the generation that
replaces it.
Prior art on question `fencing-over-owning-continuity` (wait for a
continuity that already owns the target, or preempt it), from
`docs/research/restart-budget-prior-art.md` at `74a7008`. It leans the other
way from the recommendation, and the recommendation says why. Kubernetes'
rolling update scales old ReplicaSets down unhealthy pods first, so that
"unhealthy replicas will block deployment" does not happen (§1.4,
`rolling.go`). That is preemption of a failing old replica. Nomad separates
the two by layer: the client restarts old allocations under their own policy,
and the server reconciler places the new ones (§2.3). Neither waits on an
old replica's restart. But deleting a pod is the kubelet's ordinary teardown.
In Idunn, preempting a continuity past its own Fencing means a deployment
driving another transaction's post-fencing abort. That abort revokes a lease
and restores a projection, and it is the path that wedged Ghostlight (B7).
Waiting is bounded by the owner's own phase deadlines (`PhaseDeadline`,
`:938-960`) and adds no new abort path. Recommendation: wait, at medium
confidence. Preempt is the better answer once post-fencing aborts are known
not to wedge, which is follow-up `ghostlight-abort-lease-before-stop`.
Residuals, recorded and not fixed. A wedged post-fencing abort of a
continuity (B7, follow-up `ghostlight-abort-lease-before-stop`) holds a
waiting deployment as long as it holds the target. A continuity superseded
after its mint stays charged; that lengthens one wait for a generation that
is being replaced. Stop-then-start has no control-plane fixture (B17), so in
cut 2 its edge is pinned structurally (one seam, one call) and by a unit test
of the phase mapping, not by an engine run. raven-muninn uses it, so the
engine test is owed before cut 5.
## 5. Cut sequence
1. **c1-never-stop**: spec in the mind (specs-with-no-report recipe). Depends
on nothing.
2. **c2-phase-split** (`handover-phase-split`): spec
`cut-c2-phase-split.r1` in the mind, branch `idunn/reincarnation-c2` from
`9a0a6a2`. Continuity restarts the incumbent while a pre-fencing
deployment builds. The ownership predicates split (B13, B14). The handover
is one CAS at `takes_target_at()`. A pre-handover deployment rebases onto
whatever is admitted, and a superseded continuity aborts itself. Freeze
refuses only a second live deployment for a target, which dissolves
follow-up `profile-deploy-joint-gap`. It folds in the test-only follow-up
`c1-sustained-reset-test-reach`. Open question:
`fencing-over-owning-continuity`. Depends on cut 1 (merged at `1da150a`).
3. **Dependency failures do not grow the backoff**
(`budget-never-stop`): an attempt that failed because of a declared
dependency (Odin's stable route, the target's own stable route
listener), as Idunn itself observed it, leaves the streak unchanged.
Waits on follow-up `odin-route-record-stale`, because the record it
would read is not trustworthy, and on cut 1.
4. **A dead generation is not supervised as alive** (target invariant 5):
observe the workload before route supervision and topology refresh, so
that a dead admitted generation gets no topology authentication and no
lease-exactness challenge. Its non-binding correlation is reported once,
not every tick. Waits on cut 1, because it rewrites the same function.
5. **Closing proof on Yggdrasil**: an operator-approved install of the merged
cuts. streampixels-service survives the install, and `idunn status` is
checked before and after. Waits on 1-4 and on follow-up
`stop-first-handover-fixture` (raven-muninn is stop-then-start, B17).
Ops, out of scope: nginx boot order (`gamecult-ops`), and retention of the
roughly 130 dead transient units.
## Installed
Cut 1 (`1da150a`, merge of `idunn/reincarnation` at `ffb4293`) was installed on
Yggdrasil on 2026-10-19 at 08:34:45 UTC with the operator's OK. The binary is
`/usr/local/bin/idunn` sha256 `f0e58308…`, built in `/srv/build/idunn-1da150a`
(`cargo test --lib`: 414 passed). The pre-install store backup is
`/root/idunn-store-backup-*-pre-reincarnation-c1`. Restoring it means restoring
its stores and `idunn.running` together, because the cb60965 binary cannot read
`idunn.target_supervision.v2`. At first open Idunn logged
"migrated 5 control record(s) … idunn.target_supervision.v2". streampixels-service
kept running through the restart (unit 73ca40ba, PID 968752, NRestarts 0). The
catalog returned 200, and 4443/udp was listening.
## Installed: cut 2
Cuts c2-phase-split, c2b-owner-deadlines, c2c-post-fence-forward and
c2d-continuity-yields (branch tip `8760a4f`) were installed on Yggdrasil on
2026-10-20 at 00:34:06 UTC, with the operator's OK.
- **The hotfix before it.** `ec45906` (`bounded_detail`) was installed on its
own on 2026-10-19 at 17:32 UTC. It unwedged tx-f978ab86.
- **Build.** `/srv/build/idunn-8760a4f`; `cargo test --lib` passed 471.
- **Backup.** `/root/idunn-store-backup-*-pre-reincarnation-c2` holds the
stores, `incidents.cc` and the binaries. A rollback must restore them
together: older binaries refuse two live transactions on one target, and
they refuse the `operator-required` incident condition.
- **First boot.** The boot lift returned exactly tx-039b91e9 (ghostlight) and
tx-4eaed669 (odin) to forward retry, as the Soul install gate predicted.
Odin's continuity, wedged in Routing since the 2026-10-18 reboot, committed
4 s later. Generation tx-4eaed669 is admitted, its route is healthy, and the
same PID 2571 still runs. Ghostlight's continuity is retrying forward in
AwaitingReady. Its daemon's readiness is a ghostlight-side fault
(`Permission denied` reading its session-wrapping key).
- **Live processes.** Every live MainPID was the same before and after the
install. The streampixels catalog returned 200, and 4443/udp was listening.