Body map after reincarnation cut 1; cut 1 installed on Yggdrasil
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1da150a309
commit
9a0a6a2ce9
7 changed files with 55 additions and 7 deletions
|
|
@ -173,6 +173,25 @@ health plus Ready correlation. It changes and observes route membership only
|
|||
afterward. Only after the proxy has adopted the candidate may Idunn clear the
|
||||
promotion fence and drain the incumbent.
|
||||
|
||||
Ownership of a target is exclusive at each instant. A live
|
||||
`DeploymentTransaction` owns it from Fencing on (a stop-first rollout, from
|
||||
Starting); before Fencing a deployment owns only its private candidate and
|
||||
continuity owns the incumbent slot. Continuity owns the admitted generation's
|
||||
survival and never stops. There is no stopped state: six charged attempts in
|
||||
an hour (`restarts_page`) open the ContinuityExhausted incident and gate
|
||||
nothing, and no deployment path yields to continuity. When continuity next acts
|
||||
is derived only from its own record in `TargetSupervision` (schema v2, a v1
|
||||
record lifts at store open): the attempt streak, the backoff anchor (written
|
||||
by the dead-and-free pass, the only writer), and the once-per-admission
|
||||
re-materialization bound. The workload driver alone answers whether a lost unit
|
||||
was a host reboot (`WorkloadPort::lost_to_host_reboot`). Crash restart, restart
|
||||
after a sustained run and reboot re-materialization all mint through one CAS;
|
||||
continuity supervision never aborts a transaction. Operator-facing incidents
|
||||
and `idunn status` are projections that decide nothing. Cut line: cuts 2 to 4
|
||||
of `reincarnation-map.md` still owe continuity restarting alongside a
|
||||
pre-fencing build, dependency failures not growing the streak, and the
|
||||
dead-generation route refresh.
|
||||
|
||||
The deployment brake gates changes to source, release, configuration, schema,
|
||||
unit, or authority binding. The lifecycle brake separately gates restart of an
|
||||
already-admitted incarnation. Neither target brake may gate Idunn itself or an
|
||||
|
|
|
|||
|
|
@ -305,6 +305,19 @@ continuity restart does not fetch a branch, rebuild, rerun a migration, alter
|
|||
the launch contract, or consume the deployment brake. An explicit lifecycle
|
||||
brake may suspend this actuation without changing deployment authority.
|
||||
|
||||
Continuity never stops. Six charged attempts in an hour only open the
|
||||
ContinuityExhausted incident (a page; the wire name stays, nothing is
|
||||
exhausted) and gate no mint, wait or yield. When the next attempt is due is a
|
||||
function of continuity's own record: a stored attempt streak, a backoff anchor
|
||||
written when the pass first finds the admitted generation dead and the target
|
||||
free, and a capped doubling wait from that anchor (5 s base, 300 s cap). A run
|
||||
of 600 s or more by the admitted body resets the streak. A generation lost to a
|
||||
host reboot (the workload driver's evidence, `lost_to_host_reboot`) is
|
||||
re-materialized once per admission without charge or wait. The target is free
|
||||
for a deployment's candidate between attempts, and a deployment never yields to
|
||||
continuity before Fencing; from Fencing on (or from Starting for a stop-first
|
||||
rollout) the live transaction alone owns the target.
|
||||
|
||||
An admitted generation keeps its Ready and latest Odin receipts as durable
|
||||
proof of the exact activation at their recorded admission times. Dependency
|
||||
selection reauthenticates those receipts at those times; continuity and route
|
||||
|
|
|
|||
|
|
@ -82,8 +82,9 @@ running a script, and a dead `muninn serve` is seen the tick it dies.
|
|||
2. No path on yggdrasil can start, stop or observe a host workload except
|
||||
through the actuator session bound to that host's anchor.
|
||||
3. A host with no attached actuator is a failed observation, which
|
||||
continuity counts and gives up on after six restarts inside one hour,
|
||||
exactly like a unit that will not start.
|
||||
continuity counts as a charged attempt and retries on its capped backoff
|
||||
(it never gives up; six in an hour only pages), exactly like a unit that
|
||||
will not start.
|
||||
4. The actuator cannot be told to run a program the recipe did not declare;
|
||||
`allowed_programs` on the host-native runner is enforced on the host.
|
||||
|
||||
|
|
|
|||
|
|
@ -371,3 +371,16 @@ fences the absent incumbent as nothing and admits the new release.
|
|||
|
||||
Ops, out of scope: nginx boot order (`gamecult-ops`), and retention of the
|
||||
roughly 130 dead transient units.
|
||||
|
||||
## Installed
|
||||
|
||||
Cut 1 (`1da150a`, merge of `idunn/reincarnation` at `ffb4293`) was installed on
|
||||
Yggdrasil on 2026-10-19 at 08:34:45 UTC with the operator's OK. The binary is
|
||||
`/usr/local/bin/idunn` sha256 `f0e58308…`, built in `/srv/build/idunn-1da150a`
|
||||
(`cargo test --lib`: 414 passed). The pre-install store backup is
|
||||
`/root/idunn-store-backup-*-pre-reincarnation-c1`. Restoring it means restoring
|
||||
its stores and `idunn.running` together, because the cb60965 binary cannot read
|
||||
`idunn.target_supervision.v2`. At first open Idunn logged
|
||||
"migrated 5 control record(s) … idunn.target_supervision.v2". streampixels-service
|
||||
kept running through the restart (unit 73ca40ba, PID 968752, NRestarts 0). The
|
||||
catalog returned 200, and 4443/udp was listening.
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ probes ran on Yggdrasil over `ssh ygg`, read-only, 2026-09-30 13:51-13:55 UTC
|
|||
| # | What | Where | Result |
|
||||
|---|---|---|---|
|
||||
| I1 | Alert path | `src/control_plane.rs:3798-3812` (`ReportOnce`), `:4599-4610` (`report_once`), `:5114-5138` (`note_fault`, `clear_fault`) | The only operator signal is `eprintln!`. De-dup is `fault_reports: Mutex<BTreeMap<String, ReportOnce>>` (`:4392`), in memory: an Idunn restart forgets it and reprints. `note_fault` also writes the transaction's `last_error`. |
|
||||
| I2 | Continuity exhaustion | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:<target>` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. |
|
||||
| I2 | Continuity exhaustion (since `idunn-reincarnation` cut 1: `restarts_page`, a paging threshold only; nothing stops, and the line numbers below are pre-cut) | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:<target>` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. |
|
||||
| I3 | Route degradation | `RouteSupervisionState` `:890-942`; `record_failed_challenge` sets `degraded_since_unix_millis` on the **first** failed challenge, `record_proved_challenge` clears it; reported at `:5975` under `route:<target>` | Durable per incarnation (inside `admitted_generation.v4`); a new generation starts from `default()`. Backoff doubles from the topology max age to a cap. |
|
||||
| I4 | Other operator-facing faults | `:5645-5651` (`generation-down:<target>`, "holds a dead admitted generation down"), `:6016` (`route-record:<target>`), `note_fault` per transaction | Also stderr plus `last_error`. Not named by the target; listed so the paging-set question sees them. |
|
||||
| I5 | `OperatorRequired` on main | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. |
|
||||
|
|
|
|||
|
|
@ -10,7 +10,8 @@ Read them with `view`; this page does not restate them.
|
|||
Idunn is now a watchdog that reaches the operator on Discord.
|
||||
|
||||
- **Idunn** opens a durable incident at the site where a continuity budget is
|
||||
exhausted. The incident is written to `incidents.cc`, a world-readable file
|
||||
exhausted (since `idunn-reincarnation` cut 1 this is a paging threshold
|
||||
only; continuity keeps restarting). The incident is written to `incidents.cc`, a world-readable file
|
||||
in its projection directory with a private 0600 lock, and closes when the
|
||||
target runs again and its restart window is no longer exhausted.
|
||||
- **Bifrost's watchdog-notice reader** runs from a one-minute timer. It sends
|
||||
|
|
|
|||
|
|
@ -23,9 +23,10 @@ Today nobody would hear about it. Idunn already detects several conditions an
|
|||
operator must act on, and for each one it writes a single `eprintln!` line to
|
||||
journald (`report_once`, `control_plane.rs:4599`):
|
||||
|
||||
- A continuity restart budget is exhausted (6 per hour; `restarts_exhausted`,
|
||||
decided at `control_plane.rs:5709`). Idunn stops restarting the target, and
|
||||
the only trace is one log line.
|
||||
- A continuity paging threshold is reached (6 charged attempts per hour;
|
||||
`restarts_page`, formerly `restarts_exhausted`). Continuity never stops:
|
||||
it keeps restarting on a capped backoff, and this threshold only pages. When
|
||||
this target was written the only trace was one log line.
|
||||
- An admitted route is degraded. It fails its stable-route challenge
|
||||
(`supervise_admitted_route`, ~5973). The degraded mark is durable, but no
|
||||
one is told.
|
||||
|
|
|
|||
Loading…
Reference in a new issue