diff --git a/docs/authority-map.md b/docs/authority-map.md index dc918fd..1cbf3ca 100644 --- a/docs/authority-map.md +++ b/docs/authority-map.md @@ -173,6 +173,25 @@ health plus Ready correlation. It changes and observes route membership only afterward. Only after the proxy has adopted the candidate may Idunn clear the promotion fence and drain the incumbent. +Ownership of a target is exclusive at each instant. A live +`DeploymentTransaction` owns it from Fencing on (a stop-first rollout, from +Starting); before Fencing a deployment owns only its private candidate and +continuity owns the incumbent slot. Continuity owns the admitted generation's +survival and never stops. There is no stopped state: six charged attempts in +an hour (`restarts_page`) open the ContinuityExhausted incident and gate +nothing, and no deployment path yields to continuity. When continuity next acts +is derived only from its own record in `TargetSupervision` (schema v2, a v1 +record lifts at store open): the attempt streak, the backoff anchor (written +by the dead-and-free pass, the only writer), and the once-per-admission +re-materialization bound. The workload driver alone answers whether a lost unit +was a host reboot (`WorkloadPort::lost_to_host_reboot`). Crash restart, restart +after a sustained run and reboot re-materialization all mint through one CAS; +continuity supervision never aborts a transaction. Operator-facing incidents +and `idunn status` are projections that decide nothing. Cut line: cuts 2 to 4 +of `reincarnation-map.md` still owe continuity restarting alongside a +pre-fencing build, dependency failures not growing the streak, and the +dead-generation route refresh. + The deployment brake gates changes to source, release, configuration, schema, unit, or authority binding. The lifecycle brake separately gates restart of an already-admitted incarnation. Neither target brake may gate Idunn itself or an diff --git a/docs/deployment-authority.md b/docs/deployment-authority.md index da7f028..6d464e5 100644 --- a/docs/deployment-authority.md +++ b/docs/deployment-authority.md @@ -305,6 +305,19 @@ continuity restart does not fetch a branch, rebuild, rerun a migration, alter the launch contract, or consume the deployment brake. An explicit lifecycle brake may suspend this actuation without changing deployment authority. +Continuity never stops. Six charged attempts in an hour only open the +ContinuityExhausted incident (a page; the wire name stays, nothing is +exhausted) and gate no mint, wait or yield. When the next attempt is due is a +function of continuity's own record: a stored attempt streak, a backoff anchor +written when the pass first finds the admitted generation dead and the target +free, and a capped doubling wait from that anchor (5 s base, 300 s cap). A run +of 600 s or more by the admitted body resets the streak. A generation lost to a +host reboot (the workload driver's evidence, `lost_to_host_reboot`) is +re-materialized once per admission without charge or wait. The target is free +for a deployment's candidate between attempts, and a deployment never yields to +continuity before Fencing; from Fencing on (or from Starting for a stop-first +rollout) the live transaction alone owns the target. + An admitted generation keeps its Ready and latest Odin receipts as durable proof of the exact activation at their recorded admission times. Dependency selection reauthenticates those receipts at those times; continuity and route diff --git a/docs/host-actuator.md b/docs/host-actuator.md index f2fb1c7..cdb8cde 100644 --- a/docs/host-actuator.md +++ b/docs/host-actuator.md @@ -82,8 +82,9 @@ running a script, and a dead `muninn serve` is seen the tick it dies. 2. No path on yggdrasil can start, stop or observe a host workload except through the actuator session bound to that host's anchor. 3. A host with no attached actuator is a failed observation, which - continuity counts and gives up on after six restarts inside one hour, - exactly like a unit that will not start. + continuity counts as a charged attempt and retries on its capped backoff + (it never gives up; six in an hour only pages), exactly like a unit that + will not start. 4. The actuator cannot be told to run a program the recipe did not declare; `allowed_programs` on the host-native runner is enforced on the host. diff --git a/docs/reincarnation-map.md b/docs/reincarnation-map.md index 84fdaac..7a7da85 100644 --- a/docs/reincarnation-map.md +++ b/docs/reincarnation-map.md @@ -371,3 +371,16 @@ fences the absent incumbent as nothing and admits the new release. Ops, out of scope: nginx boot order (`gamecult-ops`), and retention of the roughly 130 dead transient units. + +## Installed + +Cut 1 (`1da150a`, merge of `idunn/reincarnation` at `ffb4293`) was installed on +Yggdrasil on 2026-10-19 at 08:34:45 UTC with the operator's OK. The binary is +`/usr/local/bin/idunn` sha256 `f0e58308…`, built in `/srv/build/idunn-1da150a` +(`cargo test --lib`: 414 passed). The pre-install store backup is +`/root/idunn-store-backup-*-pre-reincarnation-c1`. Restoring it means restoring +its stores and `idunn.running` together, because the cb60965 binary cannot read +`idunn.target_supervision.v2`. At first open Idunn logged +"migrated 5 control record(s) … idunn.target_supervision.v2". streampixels-service +kept running through the restart (unit 73ca40ba, PID 968752, NRestarts 0). The +catalog returned 200, and 4443/udp was listening. diff --git a/docs/watchdog-reporting-map.md b/docs/watchdog-reporting-map.md index 5d98c4d..4c9ada7 100644 --- a/docs/watchdog-reporting-map.md +++ b/docs/watchdog-reporting-map.md @@ -18,7 +18,7 @@ probes ran on Yggdrasil over `ssh ygg`, read-only, 2026-09-30 13:51-13:55 UTC | # | What | Where | Result | |---|---|---|---| | I1 | Alert path | `src/control_plane.rs:3798-3812` (`ReportOnce`), `:4599-4610` (`report_once`), `:5114-5138` (`note_fault`, `clear_fault`) | The only operator signal is `eprintln!`. De-dup is `fault_reports: Mutex>` (`:4392`), in memory: an Idunn restart forgets it and reprints. `note_fault` also writes the transaction's `last_error`. | -| I2 | Continuity exhaustion | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. | +| I2 | Continuity exhaustion (since `idunn-reincarnation` cut 1: `restarts_page`, a paging threshold only; nothing stops, and the line numbers below are pre-cut) | `:68-71` (6 restarts / 3,600,000 ms, 5 s doubling), `:1066-1072` (`restarts_used`, `restarts_exhausted`), decided `:5709-5722` under key `continuity:` (`:5678`) | Exhaustion is **derived** from `TargetSupervision.continuity_restarts` (newest 6 times, sliding window). No write marks it. The fault clears (`:5723`) as soon as one entry leaves the window, so a permanently dead target leaves and re-enters exhaustion several times each hour. | | I3 | Route degradation | `RouteSupervisionState` `:890-942`; `record_failed_challenge` sets `degraded_since_unix_millis` on the **first** failed challenge, `record_proved_challenge` clears it; reported at `:5975` under `route:` | Durable per incarnation (inside `admitted_generation.v4`); a new generation starts from `default()`. Backoff doubles from the topology max age to a cap. | | I4 | Other operator-facing faults | `:5645-5651` (`generation-down:`, "holds a dead admitted generation down"), `:6016` (`route-record:`), `note_fault` per transaction | Also stderr plus `last_error`. Not named by the target; listed so the paging-set question sees them. | | I5 | `OperatorRequired` on main | `TerminalRecovery::OperatorRequired` `:878-885`, validated `:1954`; constructed only in tests (`:12525`, `:12535`) | No production path produces it on main. | diff --git a/docs/watchdog-reporting-postmortem.md b/docs/watchdog-reporting-postmortem.md index 3249d2d..964698d 100644 --- a/docs/watchdog-reporting-postmortem.md +++ b/docs/watchdog-reporting-postmortem.md @@ -10,7 +10,8 @@ Read them with `view`; this page does not restate them. Idunn is now a watchdog that reaches the operator on Discord. - **Idunn** opens a durable incident at the site where a continuity budget is - exhausted. The incident is written to `incidents.cc`, a world-readable file + exhausted (since `idunn-reincarnation` cut 1 this is a paging threshold + only; continuity keeps restarting). The incident is written to `incidents.cc`, a world-readable file in its projection directory with a private 0600 lock, and closes when the target runs again and its restart window is no longer exhausted. - **Bifrost's watchdog-notice reader** runs from a one-minute timer. It sends diff --git a/docs/watchdog-reporting-target.md b/docs/watchdog-reporting-target.md index 381565d..5d21c75 100644 --- a/docs/watchdog-reporting-target.md +++ b/docs/watchdog-reporting-target.md @@ -23,9 +23,10 @@ Today nobody would hear about it. Idunn already detects several conditions an operator must act on, and for each one it writes a single `eprintln!` line to journald (`report_once`, `control_plane.rs:4599`): -- A continuity restart budget is exhausted (6 per hour; `restarts_exhausted`, - decided at `control_plane.rs:5709`). Idunn stops restarting the target, and - the only trace is one log line. +- A continuity paging threshold is reached (6 charged attempts per hour; + `restarts_page`, formerly `restarts_exhausted`). Continuity never stops: + it keeps restarting on a capped backoff, and this threshold only pages. When + this target was written the only trace was one log line. - An admitted route is degraded. It fails its stable-route challenge (`supervise_admitted_route`, ~5973). The degraded mark is durable, but no one is told.