Idunn/docs/watchdog-reporting-postmortem.md

# Watchdog reporting postmortem

Campaign `idunn-watchdog`, 2026-09-30 to 2026-10-01. This was the first Eureka
campaign whose state was typed in Huginn's mind (instance `eureka`) rather than
kept in a prose map. Document ids below are `idunn-watchdog:<kind>:<local>`.
Read them with `view`; this page does not restate them.

## Summary

Idunn is now a watchdog that reaches the operator on Discord.

- **Idunn** opens a durable incident at the site where a continuity budget is
  exhausted (since `idunn-reincarnation` cut 1 this is a paging threshold
  only; continuity keeps restarting). The incident is written to `incidents.cc`, a world-readable file
  in its projection directory with a private 0600 lock, and closes when the
  target runs again and its restart window is no longer exhausted.
- **Bifrost's watchdog-notice reader** runs from a one-minute timer. It sends
  one opening DM and at most one closure DM per incident, keeps its own
  journal, and retries a failed send forever with capped backoff.
- **Scale:** 8 cuts across Idunn, Bifrost and gamecult-ops: 13 spec revisions,
  17 Hands reports, 17 Soul verdicts and 55 findings. Measured against the
  in-force estimates:

| Repo | Estimate (lines) | Landed (lines) |
|---|---|---|
| Idunn | +930/−101 | +2618/−354 |
| Bifrost | +246/−335 | +1182/−429 |
| gamecult-ops | +280/−35 | +505/−104 |

- **Installed on Yggdrasil:**
  - Idunn `a909f93` at 2026-09-30 23:21 UTC, then `cb60965` at 2026-10-01
    02:02 UTC.
  - The Bifrost reader at `160fac20` with CultLib `36ea08d3`, at 2026-10-01
    08:04 UTC.
- **First live notice:** the incident's first real input was raven-muninn,
  which exhausted its budget minutes after the incident cut was installed. Its
  opening DM was delivered at 08:04:41 UTC, and the operator confirmed it.
- **Still owed:** the closure DM for that same incident
  (`follow_up:closure-dm-observed`). It depends on raven-muninn recovering, and
  raven-muninn's own fault lies outside this campaign.

## Scope and invariants

Target `r4`. Its invariants are:

- `incident-owner`
- `record-before-delivery`
- `delivery-owner`
- `survival-independent`
- `one-per-incident`
- `no-sensitive-egress`
- `tick-lock-private`

The operator directions are `ruling:operator-watchdog-direction` and
`ruling:operator-bifrost-bridge`. The second made Bifrost the bridge to
Discord, so Idunn holds no credential and speaks no Discord API.

Excluded:

- **Odin's store condition and B5 `OperatorRequired`.** Neither can page yet.
  Odin's O2 cut is unbuilt, and B5 has no operator exit
  (`follow_up:b5-operator-required-no-exit`).
- **VoidBot's `notify_owner`.**
- **The Persona Discord crossing.**
- **Eve lowering.**

The earlier attempt was Bifrost `c063f27` (2026-09-04): an alarm was a command
written into Bifrost's store. It never ran, because no pump was deployed and
the deployed pump lacked the DM verb. It was also reversed by ruling:
`ruling:delivery-crossing` has Bifrost read Idunn's projection, so no write
crosses an owner boundary.

## Timeline

| Cut | Reports | Soul passes | Notable finding |
|---|---|---|---|
| ops-retire-alarm | h1 | 1 | none |
| bifrost-retire-alarm | h1–h2 | 2 | `silent-skip`: a stray `discord-dm` command sat pending forever |
| bifrost-suite | h1 | 1 | `suite-cultcache-scope` (pre-existing): 9 of 17 tests failed at base on a CultLib rename |
| bifrost-notice-reader | h1–h3 | 3 | `prune-keyed-on-presence`, `bridge-exit-is-failed`: duplicate DMs and blocked closures |
| bifrost-notice-retry | h1–h2 | 2 | `future-stamp-stalls-retry` |
| ops-notice-deploy | h1–h3 | 3 | `retry-window-five-minutes`: a 5-minute outage dropped a notice for good |
| idunn-incident | h1–h3 | 3 | `lock-stalls-tick` (Blocker); `lock-acl-world-openable` (High) |
| idunn-topology-lock | h1–h2 | 2 | `contended-publish-aborts`; the live stall it fixed was pre-existing |

## Structural delta

The landed code was about 2.5× its estimate, and the misses were not explained
cut by cut as the skill asks.

- **idunn-incident:** +1813 against +700. Its fix batches added the lock
  primitive, load-independent lock tests, and per-operation fault keys.
- **bifrost-notice-reader:** +891, against r3's estimate of +140. That r3
  figure described only the revision's own delta, not the cut, so r3 carried
  the wrong baseline.

Deleted:

- Bifrost's `publish-idunn-alarm` and the pump's `discord-dm` verb.
- gamecult-ops' dead `--operator-alarm-command` scripts.
- Idunn's `publish_projection_mode` and its 0644-lock test.
- The reader's prune-on-absence logic and its `WATCHDOG_NOTICE_NOW_MS`
  production clock seam.

Added:

- `idunn.operator_incident.v1` (`incidents.cc`) and
  `bifrost.watchdog_notice_execution.v1` (the journal).
- `bifrost-watchdog-notice.{service,timer}`.

Nothing was parked.

## What Soul caught

Green suites passed every defect below.

- **Survival and locking.**
  - Incident writes took a *blocking* flock inside Idunn's scheduler tick, on
    a world-readable lock. Any local uid could freeze continuity for every
    target, and `WatchdogUSec=0` meant nothing would restart Idunn
    (`s1.lock-stalls-tick`).
  - After the fix, the live directory's default ACL still made the lock 0644,
    so any uid could stop incidents opening. The test ran in a temp directory
    with no ACL (`s2.lock-acl-world-openable`).
  - The same probe exposed a pre-existing live hazard: `topology.cc.lock` 0644
    with a blocking write in the tick, measured at 5.07 s at base
    (`s2.topology-lock-stalls-tick`). The operator ruled it into the campaign;
    after the topology cut the tick takes 0.09 s.
- **Split authority.**
  - The reader pruned its "already sent" journal whenever a key was absent
    from Idunn's store. Restoring a backup would have re-sent old DMs.
  - The reader decided subject validity with its own grammar, narrower than
    Idunn's.
  - Retention was decided by two predicates in Idunn.
- **Lifecycle.**
  - One running observation closed an exhaustion incident. A release that
    crashes after each restart would have sent about 12 DMs an hour
    (`s1.flap-reopens`).
  - A bridge that died after Discord accepted a message was retried up to 5
    times, and the closure was then blocked forever.
  - Five attempts on a one-minute timer dropped a notice after a 5-minute
    Discord outage.
- **Decorative checks.**
  - A compare-exchange expected `None` on a key that embeds `now`, so it
    guarded nothing (`s1.cas-decorative`).
  - A 2-second wall-clock bound on a lock test flaked under load and counted
    equivalent mutants as caught.
- **The verifier itself.** Soul's s3 found that the stopgap forced one
  `CARGO_TARGET_DIR` on every job, so cargo-mutants' parallel workers tested
  each other's binaries. Two runs of the same two mutants disagreed
  (`s3.mutants-share-target-dir`). Every parallel mutation count reported
  through the stopgap before 2026-10-01 is unproven, in both sessions' work.

## Operator corrections

- **"Bifrost is indeed intended to be the bridge to external services like
  Discord."** The agents had found two delivery paths: VoidBot's
  `notify_owner` and Bifrost's command store. The ownership boundary was
  missing from both repos' summaries.
- **"Stewardship is currently universal…"** Self had filed a gap treating
  follow-up repos as needing stewardship. It was withdrawn; the rule was not
  written in the skill.
- **"…we seem to be saving some tokens… but we're very very slow."** Self
  blamed cold builds, measured, and was wrong. Cold builds took about 51 s and
  warm builds 29 s. The cost was slot queueing, repeated verify jobs, a test
  that hung a slot for 39 minutes, and a 4-hour job from the other session
  holding a fifth of capacity.
- **"Yep, fixing Mind issues is highest priority, immediately once this
  campaign closes"** (`ruling:mind-fixes-next`).

## Incidents

- **Network outages (ENOTFOUND).** Two outages killed three agents
  mid-pass. Each was resumed by `SendMessage` with context intact, and nothing
  was lost, because each agent's state was in commits and the mind.
- **A shared stopgap target directory** made mutation evidence noise. It is
  fixed in Eureka `a755a60`.
- **Self's own brief inflated runtime.** It told Hands to mutate serially to
  dodge a flake, and was corrected to skipping the flaky test by name.
- **A hung test held a verify slot for 39 minutes**
  (`s2.route-connect-test-hangs`). It is fixed.
- **A Hands test helper overwrote the container's `/usr/bin/node`** through a
  symlink. It was caught by the agent's own baseline and then confined
  (`s2.test-clobbers-usr-bin-node`).
- **A worktree directory stays locked** by a finished process
  (`F:\Projects\gamecult-ops-watchdog-deploy`, empty). Removing it is left to
  the operator.

## What worked

- **Soul attacks the layer where the rule fails, not the test.** Every
  High/Blocker finding came from a probe the shipped tests could not see: a
  separate-process lock holder, the live directory's ACL, or a restored store.
- **Point Soul at the load-bearing invariant in the brief.** Naming "can the
  lock hang the tick" produced the Blocker on the first pass.
- **Read before touching live state.** Self installed only after checking the
  installed revision, the in-flight transactions and the unit diff. The repo
  unit carried a BOM the live one did not.
- **One verify job per pass.** Once briefs required it, a reader fix batch
  took 11 minutes, where the first reader Hands took 38.

## What to change in the pipeline

Already changed in the skill:

- The substrate gaps moved from a table to the mind.
- Imagination owns the model page.
- Soul pays for each rerun once.
- The rust image uses sccache.
- The stopgap no longer forces `CARGO_TARGET_DIR`.

Still to change:

- **Explain every ledger miss in the next report's deviations.** This campaign
  ran at about 2.5× its estimate and nobody explained it per cut.
- **Make a revision's `estimate` the whole cut.** r3's estimate covered only
  its own delta.
- **Verify the merged tree of a long-lived branch before pushing.** Self did
  this once, for the incident cut over B5, and it should be the rule.
- **Briefs name the standard sibling build command.** The stopgap cannot
  provision siblings (`follow_up:gap-verify-no-sibling-repos`), and one
  hand-rolled build produced four false failures.

## Open follow-ups

Fifty follow-ups are in force. Twenty are substrate gaps (`gap-*`): the work
the operator ruled next. Query them with the substrate-gaps recipe in the
skill. Seven findings remain unresolved:

- **`cut-idunn-incident.s3.mutants-share-target-dir`.** It is fixed in the
  stopgap but awaits a Soul mutation run under the fixed script.
- **Six findings whose resolution keys exceed 64 bytes**
  (`follow_up:gap-unresolvable-finding-key`). Their substance is recorded in
  `follow_up:notice-retry-hardening`, `follow_up:ops-refusal-test-hardening`,
  `follow_up:bifrost-pump-input-hygiene` and
  `follow_up:contended-publish-bounds`.

Operational follow-ups that matter most:

- `closure-dm-observed`
- `notice-unknown-settle`
- `contended-publish-bounds`
- `idunn-load-flaky-tests`
- `cultcache-rs-lock-mode`
- `cultcache-ts-readonly-pull`