Body map after reincarnation cut 1; cut 1 installed on Yggdrasil
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1da150a309
commit
9a0a6a2ce9
7 changed files with 55 additions and 7 deletions
|
|
@ -173,6 +173,25 @@ health plus Ready correlation. It changes and observes route membership only
|
||||||
afterward. Only after the proxy has adopted the candidate may Idunn clear the
|
afterward. Only after the proxy has adopted the candidate may Idunn clear the
|
||||||
promotion fence and drain the incumbent.
|
promotion fence and drain the incumbent.
|
||||||
|
|
||||||
|
Ownership of a target is exclusive at each instant. A live
|
||||||
|
`DeploymentTransaction` owns it from Fencing on (a stop-first rollout, from
|
||||||
|
Starting); before Fencing a deployment owns only its private candidate and
|
||||||
|
continuity owns the incumbent slot. Continuity owns the admitted generation's
|
||||||
|
survival and never stops. There is no stopped state: six charged attempts in
|
||||||
|
an hour (`restarts_page`) open the ContinuityExhausted incident and gate
|
||||||
|
nothing, and no deployment path yields to continuity. When continuity next acts
|
||||||
|
is derived only from its own record in `TargetSupervision` (schema v2, a v1
|
||||||
|
record lifts at store open): the attempt streak, the backoff anchor (written
|
||||||
|
by the dead-and-free pass, the only writer), and the once-per-admission
|
||||||
|
re-materialization bound. The workload driver alone answers whether a lost unit
|
||||||
|
was a host reboot (`WorkloadPort::lost_to_host_reboot`). Crash restart, restart
|
||||||
|
after a sustained run and reboot re-materialization all mint through one CAS;
|
||||||
|
continuity supervision never aborts a transaction. Operator-facing incidents
|
||||||
|
and `idunn status` are projections that decide nothing. Cut line: cuts 2 to 4
|
||||||
|
of `reincarnation-map.md` still owe continuity restarting alongside a
|
||||||
|
pre-fencing build, dependency failures not growing the streak, and the
|
||||||
|
dead-generation route refresh.
|
||||||
|
|
||||||
The deployment brake gates changes to source, release, configuration, schema,
|
The deployment brake gates changes to source, release, configuration, schema,
|
||||||
unit, or authority binding. The lifecycle brake separately gates restart of an
|
unit, or authority binding. The lifecycle brake separately gates restart of an
|
||||||
already-admitted incarnation. Neither target brake may gate Idunn itself or an
|
already-admitted incarnation. Neither target brake may gate Idunn itself or an
|
||||||
|
|
|
||||||
|
|
@ -305,6 +305,19 @@ continuity restart does not fetch a branch, rebuild, rerun a migration, alter
|
||||||
the launch contract, or consume the deployment brake. An explicit lifecycle
|
the launch contract, or consume the deployment brake. An explicit lifecycle
|
||||||
brake may suspend this actuation without changing deployment authority.
|
brake may suspend this actuation without changing deployment authority.
|
||||||
|
|
||||||
|
Continuity never stops. Six charged attempts in an hour only open the
|
||||||
|
ContinuityExhausted incident (a page; the wire name stays, nothing is
|
||||||
|
exhausted) and gate no mint, wait or yield. When the next attempt is due is a
|
||||||
|
function of continuity's own record: a stored attempt streak, a backoff anchor
|
||||||
|
written when the pass first finds the admitted generation dead and the target
|
||||||
|
free, and a capped doubling wait from that anchor (5 s base, 300 s cap). A run
|
||||||
|
of 600 s or more by the admitted body resets the streak. A generation lost to a
|
||||||
|
host reboot (the workload driver's evidence, `lost_to_host_reboot`) is
|
||||||
|
re-materialized once per admission without charge or wait. The target is free
|
||||||
|
for a deployment's candidate between attempts, and a deployment never yields to
|
||||||
|
continuity before Fencing; from Fencing on (or from Starting for a stop-first
|
||||||
|
rollout) the live transaction alone owns the target.
|
||||||
|
|
||||||
An admitted generation keeps its Ready and latest Odin receipts as durable
|
An admitted generation keeps its Ready and latest Odin receipts as durable
|
||||||
proof of the exact activation at their recorded admission times. Dependency
|
proof of the exact activation at their recorded admission times. Dependency
|
||||||
selection reauthenticates those receipts at those times; continuity and route
|
selection reauthenticates those receipts at those times; continuity and route
|
||||||
|
|
|
||||||
|
|
@ -82,8 +82,9 @@ running a script, and a dead `muninn serve` is seen the tick it dies.
|
||||||
2. No path on yggdrasil can start, stop or observe a host workload except
|
2. No path on yggdrasil can start, stop or observe a host workload except
|
||||||
through the actuator session bound to that host's anchor.
|
through the actuator session bound to that host's anchor.
|
||||||
3. A host with no attached actuator is a failed observation, which
|
3. A host with no attached actuator is a failed observation, which
|
||||||
continuity counts and gives up on after six restarts inside one hour,
|
continuity counts as a charged attempt and retries on its capped backoff
|
||||||
exactly like a unit that will not start.
|
(it never gives up; six in an hour only pages), exactly like a unit that
|
||||||
|
will not start.
|
||||||
4. The actuator cannot be told to run a program the recipe did not declare;
|
4. The actuator cannot be told to run a program the recipe did not declare;
|
||||||
`allowed_programs` on the host-native runner is enforced on the host.
|
`allowed_programs` on the host-native runner is enforced on the host.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -371,3 +371,16 @@ fences the absent incumbent as nothing and admits the new release.
|
||||||
|
|
||||||
Ops, out of scope: nginx boot order (`gamecult-ops`), and retention of the
|
Ops, out of scope: nginx boot order (`gamecult-ops`), and retention of the
|
||||||
roughly 130 dead transient units.
|
roughly 130 dead transient units.
|
||||||
|
|
||||||
|
## Installed
|
||||||
|
|
||||||
|
Cut 1 (`1da150a`, merge of `idunn/reincarnation` at `ffb4293`) was installed on
|
||||||
|
Yggdrasil on 2026-10-19 at 08:34:45 UTC with the operator's OK. The binary is
|
||||||
|
`/usr/local/bin/idunn` sha256 `f0e58308…`, built in `/srv/build/idunn-1da150a`
|
||||||
|
(`cargo test --lib`: 414 passed). The pre-install store backup is
|
||||||
|
`/root/idunn-store-backup-*-pre-reincarnation-c1`. Restoring it means restoring
|
||||||
|
its stores and `idunn.running` together, because the cb60965 binary cannot read
|
||||||
|
`idunn.target_supervision.v2`. At first open Idunn logged
|
||||||
|
"migrated 5 control record(s) … idunn.target_supervision.v2". streampixels-service
|
||||||
|
kept running through the restart (unit 73ca40ba, PID 968752, NRestarts 0). The
|
||||||
|
catalog returned 200, and 4443/udp was listening.
|
||||||
|
|
|
||||||
|
|
@ -18,7 +18,7 @@ probes ran on Yggdrasil over `ssh ygg`, read-only, 2026-09-30 13:51-13:55 UTC
|
||||||
| # | What | Where | Result |
|
| # | What | Where | Result |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| I1 | Alert path | `src/control_plane.rs:3798-3812` (`ReportOnce`), `:4599-4610` (`report_once`), `:5114-5138` (`note_fault`, `clear_fault`) | The only operator signal is `eprintln!`. De-dup is `fault_reports: Mutex<BTreeMap<String, ReportOnce>>` (`:4392`), in memory: an Idunn restart forgets it and reprints. `note_fault` also writes the transaction's `last_error`. |
|
| I1 | Alert path | `src/control_plane.rs:3798-3812` (`ReportOnce`), `:4599-4610` (`report_once`), `:5114-5138` (`note_fault`, `clear_fault`) | The only operator signal is `eprintln!`. De-dup is `fault_reports: Mutex<BTreeMap<String, ReportOnce>>` (`:4392`), in memory: an Idunn restart forgets it and reprints. `note_fault` also writes the transaction's `last_error`. |
|
||||||
| I2 | Continuity exhaustion | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:<target>` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. |
|
| I2 | Continuity exhaustion (since `idunn-reincarnation` cut 1: `restarts_page`, a paging threshold only; nothing stops, and the line numbers below are pre-cut) | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:<target>` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. |
|
||||||
| I3 | Route degradation | `RouteSupervisionState` `:890-942`; `record_failed_challenge` sets `degraded_since_unix_millis` on the **first** failed challenge, `record_proved_challenge` clears it; reported at `:5975` under `route:<target>` | Durable per incarnation (inside `admitted_generation.v4`); a new generation starts from `default()`. Backoff doubles from the topology max age to a cap. |
|
| I3 | Route degradation | `RouteSupervisionState` `:890-942`; `record_failed_challenge` sets `degraded_since_unix_millis` on the **first** failed challenge, `record_proved_challenge` clears it; reported at `:5975` under `route:<target>` | Durable per incarnation (inside `admitted_generation.v4`); a new generation starts from `default()`. Backoff doubles from the topology max age to a cap. |
|
||||||
| I4 | Other operator-facing faults | `:5645-5651` (`generation-down:<target>`, "holds a dead admitted generation down"), `:6016` (`route-record:<target>`), `note_fault` per transaction | Also stderr plus `last_error`. Not named by the target; listed so the paging-set question sees them. |
|
| I4 | Other operator-facing faults | `:5645-5651` (`generation-down:<target>`, "holds a dead admitted generation down"), `:6016` (`route-record:<target>`), `note_fault` per transaction | Also stderr plus `last_error`. Not named by the target; listed so the paging-set question sees them. |
|
||||||
| I5 | `OperatorRequired` on main | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. |
|
| I5 | `OperatorRequired` on main | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. |
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,8 @@ Read them with `view`; this page does not restate them.
|
||||||
Idunn is now a watchdog that reaches the operator on Discord.
|
Idunn is now a watchdog that reaches the operator on Discord.
|
||||||
|
|
||||||
- **Idunn** opens a durable incident at the site where a continuity budget is
|
- **Idunn** opens a durable incident at the site where a continuity budget is
|
||||||
exhausted. The incident is written to `incidents.cc`, a world-readable file
|
exhausted (since `idunn-reincarnation` cut 1 this is a paging threshold
|
||||||
|
only; continuity keeps restarting). The incident is written to `incidents.cc`, a world-readable file
|
||||||
in its projection directory with a private 0600 lock, and closes when the
|
in its projection directory with a private 0600 lock, and closes when the
|
||||||
target runs again and its restart window is no longer exhausted.
|
target runs again and its restart window is no longer exhausted.
|
||||||
- **Bifrost's watchdog-notice reader** runs from a one-minute timer. It sends
|
- **Bifrost's watchdog-notice reader** runs from a one-minute timer. It sends
|
||||||
|
|
|
||||||
|
|
@ -23,9 +23,10 @@ Today nobody would hear about it. Idunn already detects several conditions an
|
||||||
operator must act on, and for each one it writes a single `eprintln!` line to
|
operator must act on, and for each one it writes a single `eprintln!` line to
|
||||||
journald (`report_once`, `control_plane.rs:4599`):
|
journald (`report_once`, `control_plane.rs:4599`):
|
||||||
|
|
||||||
- A continuity restart budget is exhausted (6 per hour; `restarts_exhausted`,
|
- A continuity paging threshold is reached (6 charged attempts per hour;
|
||||||
decided at `control_plane.rs:5709`). Idunn stops restarting the target, and
|
`restarts_page`, formerly `restarts_exhausted`). Continuity never stops:
|
||||||
the only trace is one log line.
|
it keeps restarting on a capped backoff, and this threshold only pages. When
|
||||||
|
this target was written the only trace was one log line.
|
||||||
- An admitted route is degraded. It fails its stable-route challenge
|
- An admitted route is degraded. It fails its stable-route challenge
|
||||||
(`supervise_admitted_route`, ~5973). The degraded mark is durable, but no
|
(`supervise_admitted_route`, ~5973). The degraded mark is durable, but no
|
||||||
one is told.
|
one is told.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue