Move verification to Yggdrasil: a stopgap runner until Idunn verify lands

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

SKILL.md

@@ -414,6 +414,14 @@ and reconciles the target doc with the Body.
   checkout, such as Soul's clone or a parallel Hands worktree, gets its own
   target subdirectory. The mutation harness is immune because it bumps the time
   on every file it writes. A plain test run is not.
+- **Heavy verification runs on Yggdrasil, not on the operator's workstation.**
+  Heavy verification means builds, test suites and mutation harnesses. The
+  owner is Idunn's verify transaction (a campaign in progress, ruled
+  2026-09-22). Until it lands, use the stopgap `tools/stopgap/ygg-verify.sh`.
+  It pushes an exact revision to a mirror on Yggdrasil and runs one command in
+  a container capped at 6 CPUs and 16 GiB, niced, with at most 2 jobs at a
+  time. Every Hands and Soul brief says so. Starfire runs only what has to run
+  on Windows, such as the QUIC win32 harness: one job at a time, never burners.
 - **The operator's workstation has a load budget, and Self owns it.** Running
   stress under load there, with CPU burners, is forbidden: use a dedicated
   host, or a container capped with `--cpus`. Run at most one heavy

references/changelog.md

@@ -2,6 +2,39 @@
 
 Record each change to the skill together with the evidence that motivated it.
 
+## 2026-09-22: verification moves off the operator's workstation
+
+Evidence: Starfire went from fine to unresponsive in seconds, and the operator
+had to force a shutdown. It was carrying four jobs at once:
+- 16 `node` CPU burners at twice the CPU count. They were started for a QUIC
+  stress run that Self's own rulings demanded. Git Bash's `$!` recorded MSYS
+  PIDs, and the burners were very likely never stopped.
+- A 30-iteration `dotnet test` loop.
+- A Huginn workspace build.
+- Two agents waiting on jobs that had already died.
+
+No single fork bomb was proven, because Windows audits no process creation.
+The load was Self's to schedule and Self did not budget it. The operator: "This
+is literally why we have Yggdrasil, so Starfire doesn't get overloaded with
+compiler jobs." The fleet inventory already said the same.
+
+- Ruled by the operator: **Idunn owns verification**, through a verify
+  transaction that seals nothing and sits outside the brake. It is mapped as
+  its own campaign.
+- The stopgap until then is `tools/stopgap/ygg-verify.sh` plus
+  `rust.Dockerfile`. It pushes an exact revision to a mirror on Yggdrasil and
+  runs the command in a container capped at 6 CPUs and 16 GiB, niced, with at
+  most two jobs at once. The target directory stays inside the scratch work
+  tree. Its header names its deletion line.
+- The smoke test found two bugs in the stopgap before any real job ran:
+  - `bash -lc` reset the image's `PATH`.
+  - `ssh` re-split the command, so `a && b` ran `b` on the host.
+
+  Arguments are now quoted with `printf %q`. After the fix, Huginn's mind
+  suite passed 67/67 on Yggdrasil.
+- SKILL.md: heavy verification goes to Yggdrasil. Starfire does only
+  Windows-bound work, one job at a time, with no burners.
+
 ## 2026-09-22: H1's kept sidecar looked like a crash, overwrites clobbered each other, case-blind no-op guard
 
 Evidence: Soul's H14 pass over the Huginn memory-organ campaign

tools/stopgap/rust.Dockerfile

@@ -0,0 +1,9 @@
+# STOPGAP image for ygg-verify.sh: Rust plus PowerShell, so the Eureka mutation
+# harness can run cargo suites. It dies with ygg-verify.sh (see that file's
+# deletion line).
+FROM rust:1.95-bookworm
+ARG PWSH_VERSION=7.5.3
+RUN curl -fsSL -o /tmp/pwsh.tar.gz \
+      "https://github.com/PowerShell/PowerShell/releases/download/v${PWSH_VERSION}/powershell-${PWSH_VERSION}-linux-x64.tar.gz" \
+ && mkdir -p /opt/pwsh && tar -xzf /tmp/pwsh.tar.gz -C /opt/pwsh && rm /tmp/pwsh.tar.gz \
+ && chmod +x /opt/pwsh/pwsh && ln -s /opt/pwsh/pwsh /usr/local/bin/pwsh

tools/stopgap/ygg-verify.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# STOPGAP: run a verification job for an exact revision on Yggdrasil, not on
+# the operator's workstation.
+#
+# Why: on 2026-09-22 Starfire froze under compiler, test and stress load that
+# Eureka agents had put on it. The operator ruled that Idunn owns verification,
+# through a verify transaction mapped as its own campaign. That transaction
+# takes a frozen revision, runs declared steps in a capped runner, returns a
+# typed verdict, seals nothing and needs no brake. This script covers the gap
+# until it lands.
+#
+# DELETION LINE: this file dies in the Idunn verify campaign's cut that makes
+# Eureka use `idunn verify`. When that lands, SKILL.md stops naming this
+# script, and `~/eureka-verify` on Yggdrasil is removed.
+#
+# What it does:
+#   1. Pushes <rev> from a local repo to a bare mirror at
+#      ~/eureka-verify/repos/<name>.git on Yggdrasil. Unpushed commits work too.
+#   2. Checks it out detached into a scratch work directory on Yggdrasil.
+#   3. Runs <command> in <image> with a CPU and memory cap, niced, with one
+#      shared registry cache per toolchain. The build output directory stays
+#      inside the work directory, never shared between checkouts.
+#   4. Removes the work directory unless KEEP=1, and exits with the job's status.
+#
+# At most $SLOTS jobs run at once on Yggdrasil, because it serves live
+# traffic. The default cap for each job is 6 of its 16 CPUs and 16 GiB.
+#
+# Usage (Git Bash on Starfire):
+#   ygg-verify.sh <local-repo> <rev> <image> '<command>'
+# Images:
+#   rust    eureka-verify-rust   (rust 1.95 plus pwsh; built from rust.Dockerfile)
+#   dotnet  mcr.microsoft.com/dotnet/sdk:10.0   (ships pwsh)
+#   any other value is used as the image name as-is.
+# The Eureka harness is copied to /harness/eureka-mutations.ps1 in the container.
+# Example:
+#   ygg-verify.sh /f/Projects/Huginn HEAD rust \
+#     'cargo test --workspace && pwsh /harness/eureka-mutations.ps1 -Repo /src -Entries tools/eureka-cut10-mutations.psd1'
+set -euo pipefail
+
+repo=${1:?local repo}; rev=${2:?revision}; image=${3:?image}; cmd=${4:?command}
+host=${YGG_HOST:-ygg}; cpus=${CPUS:-6}; mem=${MEM:-16g}; slots=${SLOTS:-2}
+here=$(cd "$(dirname "$0")" && pwd)
+harness="$here/../eureka-mutations.ps1"
+
+sha=$(git -C "$repo" rev-parse --verify "$rev^{commit}")
+name=$(basename "$(git -C "$repo" rev-parse --show-toplevel)")
+case "$image" in
+  rust)   image=eureka-verify-rust ;;
+  dotnet) image=mcr.microsoft.com/dotnet/sdk:10.0 ;;
+esac
+
+ssh -o BatchMode=yes "$host" "mkdir -p ~/eureka-verify/repos ~/eureka-verify/work ~/eureka-verify/harness && \
+  { test -d ~/eureka-verify/repos/$name.git || git init -q --bare ~/eureka-verify/repos/$name.git; }"
+git -C "$repo" push -q "$host:eureka-verify/repos/$name.git" "$sha:refs/verify/$sha" --force
+scp -q "$harness" "$host:eureka-verify/harness/eureka-mutations.ps1"
+if [ "$image" = eureka-verify-rust ]; then
+  scp -q "$here/rust.Dockerfile" "$host:eureka-verify/harness/rust.Dockerfile"
+fi
+
+# ssh joins its arguments into one string that the remote shell re-splits, so
+# every argument is quoted here. Without it, `a && b` in the command runs b
+# on the host.
+remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd")
+ssh -o BatchMode=yes "$host" "bash -s -- $remote_args" <<'REMOTE'
+set -euo pipefail
+name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8
+root=~/eureka-verify
+if [ "$image" = eureka-verify-rust ] && ! sudo docker image inspect eureka-verify-rust >/dev/null 2>&1; then
+  sudo nice -n 10 docker build -q -t eureka-verify-rust -f $root/harness/rust.Dockerfile $root/harness >/dev/null
+fi
+slot=""
+for i in $(seq 1 "$slots"); do
+  exec {fd}>"$root/slot-$i.lock"
+  if flock -n "$fd"; then slot=$i; break; fi
+  exec {fd}>&-
+done
+if [ -z "$slot" ]; then echo "ygg-verify: all $slots slots busy; waiting for slot 1" >&2; exec {fd}>"$root/slot-1.lock"; flock "$fd"; slot=1; fi
+work=$(mktemp -d "$root/work/$name-${sha:0:10}-XXXX")
+git -C "$root/repos/$name.git" worktree add -q --detach "$work" "$sha"
+trap 'if [ "$keep" != 1 ]; then sudo rm -rf "$work"; git -C "$root/repos/$name.git" worktree prune; fi' EXIT
+echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $mem, work $work" >&2
+set +e
+sudo nice -n 10 docker run --rm --cpus="$cpus" --memory="$mem" \
+  -v "$work:/src" -v "$root/harness:/harness:ro" \
+  -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages \
+  -e CARGO_TARGET_DIR=/src/target -w /src "$image" bash -c "$cmd"
+status=$?
+set -e
+echo "ygg-verify: exit $status" >&2
+exit $status
+REMOTE