diff --git a/notes/odin-incarnation-keyed-projection-2026-09-10.md b/notes/odin-incarnation-keyed-projection-2026-09-10.md index 5fc6fb5..99da1c9 100644 --- a/notes/odin-incarnation-keyed-projection-2026-09-10.md +++ b/notes/odin-incarnation-keyed-projection-2026-09-10.md @@ -177,3 +177,33 @@ next Muninn redeploy, which is why it is not done here. zero odin-daemon errors since the release, and Idunn has logged nothing since the 21:58:17Z sweep. The old kill window was 2.5–16 minutes. Odin is up under the incarnation-keyed projection. + +## Fold-back to 17871, 2026-09-10 22:06–22:45Z + +Odin was admitted but unreachable: its stable endpoint was `10.77.0.1:17971`, +a staging port chosen while the old Node coordinator held `17871`, and every +consumer, runbook and firewall rule still named `17871`. Operator ruled: fold +back. Done end to end, with four defects found and fixed on the way, each +committed separately: + +- The coordinator container `odin-coordinator-1` is stopped with + `restart=no`; `odin.service` now starts only `hermodr`. Binding + `stable_endpoint` is `rudp://10.77.0.1:17871`; backups of the binding and + the unit are under `/tmp/projsnap/` on the host. +- Idunn `f16fb46`: the route driver owns one ufw allow per routed stable + endpoint (needs `/etc/ufw` in the unit's `ReadWritePaths`, `103c557`). +- Idunn `e81c73e`: Warming admitted the candidate's correlation twice with one + stale record and failed the second time. Once now. +- Idunn `692ba1f`: an aged-out Odin correlation is absence to wait on, not a + fault. This is what had failed every Heimdall deployment. +- Odin `b8ef6c6`: the daemon is PID 1 of its private PID namespace and ignored + SIGTERM, so every stop was a 90 s SIGKILL and the candidate's 60 s bootstrap + wait expired during fencing. Odin catches SIGTERM now; the wait is 300 s. +- Admitted at 22:44:21Z as `up-e728909f` / `tx-0e8fa81e`, release + `sha256-4d6e959a…`, listening on `10.77.0.1:17871` → `127.0.0.1:17972`. The + driver's stale `17971` allow was deleted by hand; the operator's original + `17871` rule remains beside the driver's. Continuity restored the previous + generation within 11 s after each failed attempt; Odin was never down for + more than about a minute. +- Not Odin's: `10.77.0.4` is blocked sending UDP to `10.77.0.1:17870` (Idunn + health). Handed to the Muninn session to name. diff --git a/state/map.yaml b/state/map.yaml index 0ae4af0..343d295 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -5,7 +5,7 @@ # commands that do so are in AGENTS.md. Where this disagrees with yggdrasil, # yggdrasil is right and this file is a bug. -verified_at: 2026-09-10T22:06Z +verified_at: 2026-09-10T22:45Z verified_against: yggdrasil (systemctl, idunn status) deployment_authority: @@ -15,24 +15,24 @@ deployment_authority: source_selection: ref-head on refs/heads/main minimum_revision: 08965a44e135b393a130d62b075dc5036d2df901 idunn_minimum: 1a54db1 # incarnation-keyed projection (903e1c8) plus the queue and give-up fixes - idunn_live: 1899df2 # 2026-09-10T21:58:17Z; supervise withdraws one stale projected incarnation per tick + idunn_live: 692ba1f # route driver owns the stable endpoint allow (f16fb46); warming fixes (e81c73e, 692ba1f) runtime: admitted: true - admitted_by: up-838e26d8-e997-496c-8356-6074b4aa5461 # tx-dde64dc5, 2026-09-10 21:51:02Z - release: sha256-067ab605c82828cd2b1b1feac0d431cc217e076007c28a5dbd2902ae294878e3 # Odin main 27ca3b3 - incarnation: odin@sha256-e98b0459f230345da7eb85144eac806608bb0e6606d19d5c1f53e1cb6dbaf28a + admitted_by: up-e728909f-abee-49e4-9cf3-0f88b24cdea5 # tx-0e8fa81e, 2026-09-10 22:44:21Z + release: sha256-4d6e959a655f0e7158d8a520587be10f10dbf7782abdc6092edd4c32fc18e701 # Odin main b8ef6c6 + incarnation: odin@sha256-903af5f8691a… # full digest in the projection # The live unit is idunn-odin-.service and that hash changes # on every restart. It is not this release hash. 318 failed unit residues # were loaded on 2026-09-10T19:23Z, all with distinct hashes. state_root: /var/lib/gamecult/odin projection: /var/lib/gamecult/idunn-projection/topology.cc - route: /etc/nginx/idunn-stream-routes/odin.conf # rudp://10.77.0.1:17971 + route: /etc/nginx/idunn-stream-routes/odin.conf # rudp://10.77.0.1:17871 -> 127.0.0.1:17972; the Node coordinator that held 17871 is retired # Before Odin 27ca3b3 and Idunn 1899df2, Odin exited 1 after 2.5-16 min per # unit because the queued deployment overwrote the target-keyed Expected slot. # That path no longer exists; see notes/odin-incarnation-keyed-projection-2026-09-10.md. - healthy: true # sixteen-minute hold passed 22:06Z: same PID since 21:50:48Z, zero rejections, zero daemon errors + healthy: true # admitted 22:44:21Z, zero rejections, mesh reachable on 17871 condition: admitted # A unit named odin.service exists in /etc/systemd/system and is `active` with