Idunn's daemon, at the continuity decision site, writes one idunn.operator_incident.v1 record per exhaustion into its own published incidents.cc (--incident-store), never control.cc. The record opens once per (target, condition), closes on an observed running workload (recovered) or on the target leaving the admitted set, and retires to incident-history.cc seven days after closing. A store failure is traced once and never returned into the tick. idunn status prints each target's incidents. ReportOnce is demoted to a stderr trace. Does not build the fixture test until tests/fixtures/idunn.operator_incident.v1.cc lands in the next commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
336 lines
15 KiB
Markdown
336 lines
15 KiB
Markdown
# Idunn
|
|
|
|
> **This guide describes the previous generation and is kept for the migration.**
|
|
>
|
|
> It documents the keepalive daemon driven by `--swarm-profile`, hardcoded swarm
|
|
> targets, and a root shell actuator. Commit `330e4cd` ("Rebuild Idunn around
|
|
> typed authority ports") removed that machinery, and the current binary rejects
|
|
> every command line in this document — `idunn --swarm-profile ...` returns
|
|
> `unknown Idunn command`.
|
|
>
|
|
> That generation no longer runs anywhere; the current binary and its
|
|
> `idunn serve` generation are what runs on Yggdrasil. This file is kept for
|
|
> migration history. For the current model read the [README](../README.md);
|
|
> for the route from one to the other read [migration.md](migration.md).
|
|
>
|
|
> Its durable content — what a daemon must publish, and why keepalive is not
|
|
> deployment — remains true. Its commands do not.
|
|
|
|
Idunn is the keepalive daemon for the Odin swarm.
|
|
|
|
In plain language: Idunn is the little service whose job is to know which
|
|
GameCult daemons are supposed to be alive, bring them back after a reboot or
|
|
crash, watch whether they are still healthy, and raise a clear alarm when a
|
|
human needs to intervene.
|
|
|
|
It is named for Idunn, the keeper of the apples that keep the gods young. The
|
|
job is not glamorous. That is the point. Good infrastructure should make
|
|
important things boring.
|
|
|
|
## What Idunn Is For
|
|
|
|
GameCult has many daemons: Odin, Bifrost, VoidBot, Mimir, Heimdall, Eve
|
|
providers, service workers, renderers, bridges, and local machine helpers. Each
|
|
one owns its own work, but each one should not have to carry its own private
|
|
lifecycle machinery.
|
|
|
|
Idunn centralizes that lifecycle work:
|
|
|
|
- start known daemons after machine boot;
|
|
- ensure deployed daemon artifacts match the desired target revision;
|
|
- restart daemons after crashes;
|
|
- watch health and freshness signals;
|
|
- avoid restarting services when authority is unclear;
|
|
- record every deployment/restart request and result as typed state;
|
|
- record release targets, deployment artifacts, state migrations, and rollout
|
|
results as typed state;
|
|
- record operator incidents in its published `incidents.cc`, which Bifrost reads
|
|
and delivers. Idunn never waits on delivery.
|
|
|
|
The desired shape is simple:
|
|
|
|
```text
|
|
Odin knows what exists.
|
|
Idunn keeps it alive.
|
|
Bifrost carries operator/public crossings.
|
|
Each daemon owns its own work and health publication.
|
|
```
|
|
|
|
## Who It Helps
|
|
|
|
Idunn is for anyone running a local GameCult swarm or operating a hosted
|
|
GameCult machine.
|
|
|
|
For a developer, Idunn should mean fewer mystery services to restart by hand.
|
|
For an operator, Idunn should mean health problems become visible, witnessed,
|
|
and routed to the right place. For daemon authors, Idunn should mean they can
|
|
publish health, transport profiles, and command boundaries over CultNet RUDP
|
|
instead of building one more private watchdog.
|
|
|
|
## Current State
|
|
|
|
Idunn is a Rust daemon inside Odin's Cargo workspace. Starfire and Yggdrasil
|
|
run separate host-local swarm profiles. Starfire owns its local adjunct daemons
|
|
and remote display/telemetry lanes. Yggdrasil owns its hosted release targets,
|
|
including Odin and VoidBot. Idunn starts independently of every managed target,
|
|
so an Odin outage cannot remove the organ responsible for recovering Odin.
|
|
Each target declares a daemon-owned health contract and keeps
|
|
its own interval and deploy/restart authority. Daemon truth comes from typed
|
|
CultNet/RUDP publication and daemon-owned boundary stores. Shell scripts remain
|
|
deployment, restart, and manual diagnostic lowerings only; they do not satisfy
|
|
health.
|
|
The scheduler and continuity witness now belong to one Rust process instead of
|
|
a PowerShell-herded pile of one-daemon workers.
|
|
|
|
VoidBot deployment authority is deliberately absent from `starfire-local`.
|
|
Pushing VoidBot's upstream `main` branch is the release input; Yggdrasil-local
|
|
Idunn observes `/srv/build/VoidBot` and invokes only
|
|
`sudo -n /usr/local/libexec/idunn-yggdrasil deploy voidbot`. The privileged
|
|
actuator delegates deploy and restart to the root-owned executable
|
|
`/srv/odin/deploy-manifests/voidbot`. A developer checkout is not a deployment
|
|
target and agents must not create a local keepalive or deploy path for VoidBot.
|
|
Each cycle fetches `origin/main` in `/srv/build/VoidBot` and compares it with
|
|
`DEPLOYED_REVISION` in `/srv/voidbot/deploy/deployment.env`. Drift owns the
|
|
deployment decision. An unavailable fetch or witness raises an operator alarm
|
|
instead of guessing. The privileged manifest must atomically update that
|
|
witness only after the new release is verified; Idunn rejects an otherwise
|
|
successful command when the witness has not converged.
|
|
|
|
```text
|
|
scratch/idunn/idunn.keepalive.cc
|
|
```
|
|
|
|
## Run It
|
|
|
|
From the Idunn repository root:
|
|
|
|
```powershell
|
|
cargo build --release
|
|
cargo run --release --bin idunn -- --daemon demo --rudp-health-bind <idunn-rudp-health-bind>
|
|
```
|
|
|
|
`--rudp-health-bind` is explicit. Bare one-daemon Idunn does not open a
|
|
localhost health ingress.
|
|
|
|
Install the local Starfire boot watchdog:
|
|
|
|
```powershell
|
|
.\scripts\install-idunn-startup.ps1
|
|
```
|
|
|
|
That task starts one `idunn.exe` swarm supervisor at user logon. The Rust
|
|
runtime owns the target catalog for Odin, Stonks, Muninn, the enforced
|
|
Yggdrasil source artifact lanes, and the Nightwing display services. The
|
|
current Mimir dashboard is observed but not restarted until its restart
|
|
authority is named.
|
|
|
|
It also starts `idunn-swarm-deployment-coverage`, which watches the deployment
|
|
target catalog in `scripts/idunn-deployment-targets.ps1`. A repo/service is not
|
|
allowed to vanish into shrug-space: it is either enforced by Idunn, explicitly
|
|
blocked with the missing authority named, external-owned, or not a runtime.
|
|
|
|
To record that a daemon has not published health, run a manual target without a
|
|
matching `idunn.daemon_health` RUDP publication:
|
|
|
|
```powershell
|
|
cargo run --release --bin idunn -- --daemon demo --restart-command "echo restart demo"
|
|
```
|
|
|
|
Idunn writes a non-actuating missing-publication health record and plans from
|
|
that typed absence. It does not run a local health command to decide daemon
|
|
truth.
|
|
|
|
To actually allow restart/deploy actuation after typed health planning decides
|
|
one is needed:
|
|
|
|
```powershell
|
|
cargo run --release --bin idunn -- --daemon demo --deploy-command "echo deploy demo" --restart-command "echo restart demo" --execute
|
|
```
|
|
|
|
To keep watching on a resident interval:
|
|
|
|
```powershell
|
|
cargo run --release --bin idunn -- --daemon demo --interval-seconds 30
|
|
```
|
|
|
|
To run the built-in Starfire-local swarm profile directly:
|
|
|
|
```powershell
|
|
cargo run --release --bin idunn -- --swarm-profile starfire-local --repo-root <repo-root> --execute
|
|
```
|
|
|
|
The Yggdrasil service runs the hosted profile directly:
|
|
|
|
```sh
|
|
/usr/local/bin/idunn --swarm-profile yggdrasil-local --repo-root /srv/odin/source --store /var/lib/gamecult/idunn/yggdrasil.keepalive.cc --rudp-health-bind 10.77.0.1:17870 --execute
|
|
```
|
|
|
|
Optional store override:
|
|
|
|
```powershell
|
|
cargo run --release --bin idunn -- --daemon demo --store E:\path\to\idunn.keepalive.cc
|
|
```
|
|
|
|
## What Daemons Should Publish
|
|
|
|
Idunn should not guess private service truth. A daemon should publish:
|
|
|
|
- its service id and Verse id;
|
|
- where its durable `.cc` state or witness lives;
|
|
- a health or freshness signal;
|
|
- the command boundary for deployment or artifact refresh, if one exists;
|
|
- the command boundary for restart or recovery, if one exists;
|
|
- what operator action is needed when automatic recovery is unsafe.
|
|
|
|
If that information is missing, Idunn should fail closed and create an operator
|
|
alarm instead of improvising.
|
|
|
|
For the repo swarm, deployment ownership begins with catalog coverage. Use:
|
|
|
|
```powershell
|
|
.\scripts\show-idunn-deployment-targets.ps1
|
|
.\scripts\health-idunn-swarm-deployment-coverage.cmd
|
|
```
|
|
|
|
The enforced targets are Nightwing Gjallar plus the Yggdrasil source artifact
|
|
lanes whose ops runbooks can produce and verify deployment manifests. Each
|
|
enforced target must declare upstream remote/branch, rollout strategy, state
|
|
migration authority, and zero-downtime capability. Idunn deploys the declared
|
|
declared upstream revision, not arbitrary local `HEAD`. Ghostlight's catalog
|
|
entry remains disabled until its Linux body and signed health identity are
|
|
admitted. Bifrost is blocked until
|
|
its production database migration path is coherent; mobile device installs
|
|
remain blocked at their physical approval boundaries; GitHub Pages remains
|
|
external-owned.
|
|
|
|
## Typed Records
|
|
|
|
- `idunn.desired_daemon.v1`
|
|
- `idunn.daemon_health.v1`
|
|
- `idunn.signed_health_admission.v1`
|
|
- `idunn.keepalive_decision.v1`
|
|
- `idunn.deployment_request.v2`
|
|
- `idunn.current_deployment_request.v1`
|
|
- `idunn.deployment_result.v1`
|
|
- `idunn.release_target.v1`
|
|
- `idunn.deployment_artifact.v1`
|
|
- `idunn.state_migration_plan.v1`
|
|
- `idunn.state_migration_result.v1`
|
|
- `idunn.rollout_plan.v1`
|
|
- `idunn.rollout_result.v1`
|
|
- `idunn.restart_request.v1`
|
|
- `idunn.restart_result.v1`
|
|
- `idunn.operator_alarm.v1`
|
|
- `idunn.daemon_surgery_plan.v1`
|
|
- `idunn.daemon_transport_profile.v1`
|
|
- `idunn.command_boundary.v1`
|
|
- `idunn.runtime_transport_check.v1`
|
|
- `idunn.rudp_health_ingress.v1`
|
|
|
|
Idunn publishes one `idunn.daemon_surgery_plan.v1` record per swarm target when
|
|
the swarm starts. Those records make the CultNet/RUDP migration queue explicit:
|
|
owner, objective, current mechanism, intended authority, cut line,
|
|
steps, blockers, severity, and status.
|
|
|
|
It also publishes one `idunn.daemon_transport_profile.v1` and one
|
|
`idunn.command_boundary.v1` per target. The desired daemon record links to
|
|
both. The transport profile names `cultnet.transport.rudp.v0` as the target and
|
|
marks daemon publication and daemon-owned witness stores as the health
|
|
substrate. The command boundary names restart, deploy, health, and alarm
|
|
authority separately; health authority belongs to daemon-published RUDP state.
|
|
|
|
At startup Idunn also publishes `idunn.runtime_transport_check.v1`, currently a
|
|
loopback CultNet hello over `cultnet.transport.rudp.v0`. That proves Idunn's
|
|
Rust body can use the RUDP substrate before it asks the rest of the swarm to
|
|
walk through the same door.
|
|
|
|
For deploy-enforced targets, startup also publishes `idunn.release_target.v1`,
|
|
`idunn.deployment_artifact.v1`, `idunn.state_migration_plan.v1`, and
|
|
`idunn.rollout_plan.v1`. During deployment, Idunn runs any declared
|
|
daemon-owned migration command before the deploy command; a failed migration
|
|
stops the rollout and raises an operator alarm. Zero downtime is recorded only
|
|
when the target declares a real in-place, blue/green, or rolling strategy;
|
|
otherwise Idunn says `restart-required` and verifies that path honestly.
|
|
|
|
Deploy scripts are Idunn actuators. Agents should configure the target catalog,
|
|
release target, command boundary, migration command, and daemon publication
|
|
surface, then let Idunn run the rollout. Direct script execution without
|
|
`IDUNN_ACTUATOR=1` fails on purpose; the little hand reaching for the deploy
|
|
button has been removed from the machine.
|
|
|
|
`scripts/start-idunn-local.ps1` launches the local swarm with
|
|
`--rudp-health-bind 0.0.0.0:17870` so host-local and explicitly configured peers
|
|
can publish to the configured Idunn health endpoint. It accepts raw
|
|
`idunn.daemon_health` document puts on the canonical CNR0 RUDP `schema` channel
|
|
and writes them into the keepalive store. That is daemon-owned health
|
|
publication, not restart/deploy authority. On Windows, UDP reset reports from
|
|
closed one-shot publishers are treated as nonfatal ingress noise so one accepted
|
|
health frame cannot kill the listener.
|
|
|
|
Epiphany uses a separate signed runtime-health schema. Idunn joins that document
|
|
to the exact latest Bifrost-authorized deployment request and persists
|
|
`idunn.signed_health_admission.v1`. The privileged admission verifier reads only
|
|
that fresh, current, request-bound record; generic health remains observation.
|
|
|
|
When a fresh daemon-published RUDP health record exists for a target, Idunn uses
|
|
that record for keepalive planning. The record must match the daemon id, health
|
|
contract, RUDP transport, and freshness window. If it is absent or stale, Idunn
|
|
publishes a missing-publication observation and raises/plans from that typed
|
|
absence instead of running a local health probe.
|
|
|
|
The active transport cut is no longer "make the daemons speak RUDP at all."
|
|
Nightwing Gjallar now consumes Odin's accepted
|
|
`surface:gamecult.network.status` snapshot over CultNet/RUDP, so every active
|
|
Starfire-local daemon target now publishes daemon-owned RUDP health plus typed
|
|
boundary state. The remaining debt is deleting bridge-only lowerings and
|
|
xenos-boundary ingestion shims without letting them reclaim ownership. Odin
|
|
still owes removal of hashed-store/C# witness shims once direct interoperable
|
|
witness publication is ready.
|
|
|
|
Muninn media uses a codec-owned prediction model, not a custom visual-delta
|
|
codec. The canonical LAN profile is
|
|
`muninn.rudp.low_latency_h264_lan.v1`: NVENC H.264 owns inter-frame prediction,
|
|
motion search, residuals, keyframe production, and the actual lossy visual
|
|
encoding; CultNet/RUDP owns bounded delivery of typed video access units, audio
|
|
packets, receiver feedback, retransmit pressure, expiry, and keyframe requests.
|
|
Late media is dropped when it misses the fixed latency budget. Optical flow,
|
|
foveation, ROI maps, or stochastic patch experiments may become future Muninn
|
|
profiles only after this hardware-codec RUDP path is measured and proven
|
|
insufficient.
|
|
|
|
Raven Muninn task actions are also an explicit ops invariant: Task Scheduler
|
|
must execute `wscript.exe //B //Nologo` hidden launcher actions directly for
|
|
`GameCult-Muninn`, `GameCult-Muninn-Activate`, and
|
|
`GameCult-Muninn-VideoProof`, never raw `.cmd` wrappers, and the hidden VBS
|
|
layer must call `.ps1` launchers directly instead of `cmdPath` trampolines.
|
|
Those three live Raven tasks have been repaired and verified. The repair
|
|
actuator now uploads its Raven PowerShell body with `sftp` and runs a tiny
|
|
cleanup wrapper, so the hidden-task repair does not hit Windows command-line
|
|
limits before it can run. The Muninn serve bodies no longer publish a
|
|
daemon-health document: `--idunn-rudp-health`, `--idunn-daemon` and
|
|
`--idunn-health-contract` are retired flags that the binary rejects. Raven's
|
|
liveness is Odin-correlated runtime presence; Starfire and Nightwing are
|
|
checked by their `scripts/health-*-muninn.ps1` witness scripts.
|
|
|
|
From here on out, the work is not "teach another daemon to speak." It is
|
|
"delete the witness masks and keep ownership where it belongs."
|
|
|
|
Rust no longer gets to claim the transport is imaginary:
|
|
CultLib's `cultnet-rs` carries the canonical CNR0 `cultnet.transport.rudp.v0`
|
|
session path for acknowledged CultNet messages over UDP. That is substrate
|
|
only. Idunn still marks daemon profiles as
|
|
`migration-required` until each daemon actually publishes health and command
|
|
boundary records through that path.
|
|
|
|
## Boundaries
|
|
|
|
Idunn is not Odin. It does not decide which Verses exist.
|
|
|
|
Idunn is not Bifrost. It does not own public crossings, Discord delivery, owner
|
|
DMs, or governance transport.
|
|
|
|
Idunn is not Eve. It does not own presentation.
|
|
|
|
Idunn is the keeper of continuity. If a daemon should be alive, Idunn should
|
|
know that, check it, help it recover, and leave a witness trail sharp enough
|
|
that nobody has to guess what happened.
|