Eureka
0ac55b729f
nice on the docker client never reached the job, which containerd starts. On 2026-10-31 three verify jobs held Yggdrasil's IO pressure near 75% full and the swarm's turns slowed; idle IO lets the swarm go first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
446 lines
24 KiB
Shell
446 lines
24 KiB
Shell
#!/usr/bin/env bash
|
|
# STOPGAP: run a verification job for an exact revision on Yggdrasil, not on
|
|
# the operator's workstation.
|
|
#
|
|
# Why: on 2026-09-22 Starfire froze under compiler, test and stress load that
|
|
# Eureka agents had put on it. The operator ruled that Idunn owns verification,
|
|
# through a verify transaction mapped as its own campaign. That transaction
|
|
# takes a frozen revision, runs declared steps in a capped runner, returns a
|
|
# typed verdict, seals nothing and needs no brake. This script covers the gap
|
|
# until it lands.
|
|
#
|
|
# DELETION LINE: this file dies in the Idunn verify campaign's cut that makes
|
|
# Eureka use `idunn verify`. When that lands, SKILL.md stops naming this
|
|
# script, and `~/eureka-verify` on Yggdrasil is removed.
|
|
#
|
|
# What it does:
|
|
# 1. Pushes <rev> from a local repo to a bare mirror at
|
|
# ~/eureka-verify/repos/<name>.git on Yggdrasil. Unpushed commits work too.
|
|
# 2. Clones the mirror into a scratch work directory on Yggdrasil and checks
|
|
# the commit out detached. It is a real clone, not a worktree, because a
|
|
# worktree's .git file points at a host path the container cannot see.
|
|
# 3. Runs <command> in <image> under a CPU and memory cap, niced. Each
|
|
# toolchain gets one shared registry cache. The build output directory
|
|
# stays inside the work directory and is never shared between checkouts.
|
|
# CARGO_TARGET_DIR is deliberately NOT set: cargo's default (<workspace>/
|
|
# target) is already inside /src, and forcing it made cargo-mutants' --jobs
|
|
# workers share one target dir and test each other's binaries, so parallel
|
|
# kill counts were noise (Soul, idunn-watchdog, 2026-10-01).
|
|
# 4. Removes the work directory unless KEEP=1, and exits with the job's status.
|
|
#
|
|
# At most $SLOTS jobs run at once on Yggdrasil. The operator raised this to 3 on
|
|
# 2026-09-22 ("the other cores are mostly sitting idle") and asked on 2026-09-30
|
|
# for Yggdrasil to be "absolutely pinned", so it went to 5. Measured the same
|
|
# night: 40% CPU idle and 53 of 62 GiB available with ~20 jobs queued, running
|
|
# jobs using ~1 GiB each. So it is 8, each capped at 4 CPUs and 6 GiB (MEM
|
|
# overrides per job; a big link can ask for more). The CPU caps oversubscribe
|
|
# the 16 cores on purpose: jobs run at nice 10, so the live services still win
|
|
# the scheduler. Memory is the real ceiling: 8 x 6 GiB worst case leaves the
|
|
# services (~9 GiB resident) their share of the 62.
|
|
# DISK is the ceiling that bit: at 8 slots (2026-09-30 01:50 CEST) /proc/pressure/io
|
|
# read full avg60=41%, and live Odin's fsyncs stalled long enough for Idunn's route
|
|
# challenges to time out. Back to 5 slots; CPU and memory headroom do not mean
|
|
# IO headroom. Watch /proc/pressure/io before raising this again.
|
|
#
|
|
# A job whose container sits under $5% of one core for $IDLE seconds
|
|
# (default 900) is killed and comes back red: on 2026-09-30 three hung test
|
|
# binaries held every slot for up to an hour at ~2% CPU while agents queued.
|
|
#
|
|
# There is no hand-written mutation harness: the operator retired it on
|
|
# 2026-09-22. Mutation testing uses the ecosystem's tools on a cut's diff.
|
|
#
|
|
# TWO RAKES, both paid for on 2026-09-23:
|
|
# * The command runs under `bash -o pipefail -c`, and BASH_ENV points every child
|
|
# bash at a file holding `set -o pipefail`, so a job SCRIPT run as `bash job.sh`
|
|
# inherits pipefail too. (Exporting SHELLOPTS did that but also leaked a job's
|
|
# `set -u` into bash-wrapped tools: kotlinc died on `JAVA_OPTS: unbound variable`,
|
|
# 2026-09-30.)
|
|
# (the -o flag alone does not reach child shells; Soul proved a failing piped
|
|
# test in a job script came back exit 0, 2026-09-30). Pipefail forced since
|
|
# 2026-09-30, after an ack Cut 2 job piped a failing test run into `cut` and
|
|
# came back exit 0). pipefail does not rescue `a; b`: the job's status is the
|
|
# LAST statement's, so a command ending in `git checkout`, `echo` or a filter
|
|
# that succeeds still reports zero. Keep the verdict-bearing command last, and
|
|
# a grep that matches nothing now fails the job (that is a signal, not noise).
|
|
# Under pipefail `cmd | grep -q X` reads FALSE even on a match: grep exits at the
|
|
# first hit, cmd dies of SIGPIPE, and its status wins (a media FEC mutation run
|
|
# reported killed mutants as survivors this way, 2026-09-30). Capture to a file
|
|
# and grep the file, or use `grep -c`/`grep` without -q.
|
|
# * CultLib has no Cargo.toml at its root. A rust job must `cd packages/<crate>`
|
|
# first, or cargo fails in a way that looks like the image is wrong.
|
|
#
|
|
# EDITING: bash reads a running script from disk as it goes, so editing this
|
|
# file in place breaks every job still running it (one died with "unexpected
|
|
# EOF", 2026-09-22). Write the new version to a temp file and rename it over
|
|
# this one.
|
|
#
|
|
# Usage (Git Bash on Starfire):
|
|
# ygg-verify.sh <local-repo> <rev> <image> '<command>'
|
|
# TIMEOUT=<seconds> caps the job (default 3600); a killed job exits non-zero.
|
|
# IDLE=<seconds> kills a job idle that long (default 900; 0 disables).
|
|
# RUN_ON_HOST=1 runs <command> on the Yggdrasil host instead of in one container,
|
|
# inside the same slot, clone, TIMEOUT and IDLE watchdog, with the environment
|
|
# YGG_WORK (the clone), YGG_IMAGE, YGG_CPUS, YGG_MEM, YGG_CNAME (job name),
|
|
# YGG_LABEL (every container it starts must carry `--label $YGG_LABEL`: the
|
|
# watchdog kills containers by that label), YGG_KILLED (a path that exists once
|
|
# the watchdog has killed the job: stop starting work) and DOCKER_ARGS. The
|
|
# command owns the container topology and must remove what it creates.
|
|
# tools/mutation-table.sh is the one user: it snapshots a baselined container
|
|
# and runs each mutant in a fresh container from the snapshot.
|
|
# DOCKER_ARGS='<extra docker run flags>' passes through to the container's
|
|
# own `docker run`, appended after the fixed flags and before the image name
|
|
# (e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0' or
|
|
# DOCKER_ARGS='--network host'). Word-split by the remote shell, so quote a
|
|
# flag's own value inside the string if it needs one.
|
|
#
|
|
# Summary and wait modes (agents use these; they save the polling calls):
|
|
# SUMMARY=<log-path> ygg-verify.sh <repo> <rev> <image> '<command>'
|
|
# Runs the same job but keeps the whole output in <log-path> and prints only
|
|
# a bounded summary (at most ~80 lines): the exit verdict, per-framework
|
|
# pass/fail/skip counts (cargo test, dotnet test, vitest, pytest), a ZERO
|
|
# TESTS RAN line when counts exist and are all zero, failing test names, and
|
|
# for a red job the failure sections and the log tail. When the job ends it
|
|
# writes <log-path>.summary and then <log-path>.done (holding the exit
|
|
# status). Use a fresh <log-path> per job: a reused path is unsafe for the first
|
|
# SETTLE seconds (see the run protocol below), so fresh paths are the rule.
|
|
# Launch it as ONE background shell
|
|
# (run_in_background), then:
|
|
# ygg-verify.sh wait <log-path>
|
|
# Blocks at most 270 s (WAIT_MAX can only lower that), prints the summary and
|
|
# exits with the job's status when the job is done. Otherwise prints
|
|
# `RUNNING <seconds>s` and exits 99: call it again. Exit 98 means the
|
|
# launcher is gone with no verdict (killed outright) or never started. It
|
|
# checks a local file every 2 s; no remote call, no model turn, one wake.
|
|
# The runner mints a nonce per SUMMARY run and exports it into the job as
|
|
# $YGG_NONCE. A job's own lines starting `YGG-SUMMARY/$YGG_NONCE: ` are carried
|
|
# into the summary (tools/mutation-table.sh prints its table that way); a
|
|
# `YGG-SUMMARY` line without the nonce is counted and ignored, and carried
|
|
# lines never replace a red job's failure sections. The nonce stops output a
|
|
# test merely echoes; code that reads its own environment can still forge.
|
|
# Counts are read from the log's framework lines, so a test that prints a fake
|
|
# `test result: ok. 50 passed` inflates them: not prevented, check the exit
|
|
# status and the named failures, not the counts alone. The full log
|
|
# stays at <log-path>; open it only at a named failing test, with grep -n -A 40.
|
|
# Killing the launcher does not stop the remote container: its own TIMEOUT and
|
|
# IDLE watchdog does.
|
|
# Images:
|
|
# rust eureka-verify-rust:<Dockerfile hash> (rust 1.95, cargo-mutants, sccache;
|
|
# every job shares the eureka-sccache volume at /sccache, keyed by
|
|
# input hash, so a cold /src/target still reuses compiled crates)
|
|
# kotlin eureka-verify-kotlin:<Dockerfile hash> (JDK 21, kotlinc 2.2.21, Node 24, pwsh;
|
|
# for packages/cultmesh-kotlin: `pwsh -File build.ps1`)
|
|
# dotnet mcr.microsoft.com/dotnet/sdk:10.0 (install Stryker in the command:
|
|
# dotnet tool install -g dotnet-stryker)
|
|
# any other value is used as an image name as-is.
|
|
# Example:
|
|
# ygg-verify.sh /f/Projects/Huginn HEAD rust \
|
|
# 'cargo test --workspace && git diff BASE HEAD > /tmp/d && cargo mutants --in-diff /tmp/d'
|
|
set -euo pipefail
|
|
|
|
here=$(cd "$(dirname "$0")" && pwd)
|
|
|
|
# Bounded summary of one job's log. $1 log, $2 exit status, $3 optional note.
|
|
# Carried lines are the job's own table (tools/mutation-table.sh). Only lines
|
|
# starting `YGG-SUMMARY/$YGG_NONCE: ` count: the nonce is minted per run, exported
|
|
# into the job, and absent from any fixture, data file or log a test echoes. A
|
|
# `YGG-SUMMARY` line without it is counted and ignored. Carried lines never
|
|
# replace the failure sections or the tail.
|
|
ygg_summarize() {
|
|
local log=$1 status=$2 note=${3:-} word=RED scan fw names loads p f nf own=0 untrusted=0 tag secs n
|
|
[ "$status" = 0 ] && word=GREEN
|
|
# shellcheck source=../testparse.sh
|
|
. "$here/../testparse.sh"
|
|
tag="YGG-SUMMARY/${YGG_NONCE:-}: "
|
|
untrusted=$(grep -c '^YGG-SUMMARY' "$log" || true)
|
|
if [ -n "${YGG_NONCE:-}" ]; then
|
|
own=$(awk -v t="$tag" 'index($0, t) == 1 { n++ } END { print n + 0 }' "$log")
|
|
else own=0; fi
|
|
untrusted=$(( untrusted - own ))
|
|
scan=$(tp_scan "$log")
|
|
fw=$(printf '%s\n' "$scan" | grep '^FRAMEWORK' || true)
|
|
names=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!seen[$0]++' || true)
|
|
read -r p f < <(printf '%s\n' "$fw" | awk '{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed")p+=a[2];if(a[1]=="failed")f+=a[2]}}END{print p+0, f+0}')
|
|
nf=0; [ -z "$names" ] || nf=$(printf '%s\n' "$names" | wc -l)
|
|
loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!seen[$0]++' || true)
|
|
{
|
|
echo "YGG-VERIFY exit $status $word${note:+ ($note)} log: $log"
|
|
grep -E '^ygg-verify: (TIMEOUT|IDLE|NO VERDICT)' "$log" || true
|
|
if [ "$status" = 0 ] && { [ "$nf" -gt 0 ] || [ "$f" -gt 0 ]; }; then
|
|
echo "MISMATCH: exit 0 but the log shows $nf failing test names and $f failed counts; do not trust the exit"
|
|
fi
|
|
[ -z "$fw" ] || printf '%s\n' "$fw"
|
|
if [ -z "$fw" ]; then [ "$own" -gt 0 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)"
|
|
elif [ $((p + f)) -eq 0 ]; then echo "ZERO TESTS RAN"; fi
|
|
if [ "$nf" -gt 0 ]; then
|
|
printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}'
|
|
fi
|
|
if [ "$nf" -eq 0 ] && [ -n "$loads" ]; then
|
|
printf '%s\n' "$loads" | awk 'NR<=5{print "FAILED TO LOAD " substr($0,10)} END{if(NR>5)print "... " NR-5 " more files"}'
|
|
fi
|
|
if [ "$f" -gt "$nf" ]; then
|
|
echo "NAMES MISSING: the counts show $f failed but only $nf named; the rest are unnamed here, open the log"
|
|
fi
|
|
if [ "$own" -gt 0 ]; then
|
|
awk -v t="$tag" 'index($0, t) == 1 { print substr($0, length(t) + 1) }' "$log"
|
|
fi
|
|
if [ "$untrusted" -gt 0 ]; then
|
|
echo "$untrusted YGG-SUMMARY lines without this run's nonce were ignored (a test printed them, or the table printer predates the nonce); see the log"
|
|
fi
|
|
{ grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20'
|
|
if [ "$status" != 0 ]; then
|
|
if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ] && [ -z "$loads" ]; then
|
|
echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command"
|
|
fi
|
|
secs=$(tp_sections "$log" 30 | awk 'NR<=40')
|
|
[ -z "$secs" ] || printf '%s\n' "$secs"
|
|
echo "--- log tail"
|
|
if [ -n "$secs" ]; then n=8; else n=20; fi
|
|
{ grep -v '^YGG-SUMMARY' "$log" || true; } | tail -n "$n"
|
|
elif [ -z "$fw" ] && [ "$own" -eq 0 ]; then
|
|
echo "--- log tail"
|
|
tail -n 5 "$log"
|
|
fi
|
|
} | awk 'NR<=80{print} NR==81{print "[summary truncated at 80 lines; full log is the path above]"}'
|
|
}
|
|
|
|
if [ "${1:-}" = summarize ]; then # re-read a finished log: summarize <log> <exit status> [nonce]
|
|
YGG_NONCE=${4:-${YGG_NONCE:-}}; export YGG_NONCE
|
|
ygg_summarize "${2:?log}" "${3:?exit status}"; exit 0
|
|
fi
|
|
|
|
# Run protocol for SUMMARY=<log> (every file is replaced by write-then-rename):
|
|
# <log>.pid launcher pid <log>.start launch epoch
|
|
# <log>.run this run's id, written AFTER pid/start: it is the commit point
|
|
# <log>.summary, then <log>.done holding "<run id> <exit status>"
|
|
# `wait` trusts a .done only when its run id equals the current .run. A reused
|
|
# path therefore cannot yield the previous run's result once the new launcher
|
|
# has claimed it; before that claim (the launcher is still starting) the old
|
|
# result is indistinguishable, so wait settles $SETTLE seconds (default 3)
|
|
# before accepting a result it found on entry. Fresh paths per job stay the rule.
|
|
if [ "${1:-}" = wait ]; then
|
|
wlog=${2:?usage: ygg-verify.sh wait <summary-log-path>}
|
|
cap=${WAIT_MAX:-270}; [ "$cap" -le 270 ] || cap=270
|
|
settle=${SETTLE:-3}; first=1
|
|
t0=$(date +%s)
|
|
while :; do
|
|
run=$(cat "$wlog.run" 2>/dev/null || true)
|
|
did=''; st=''
|
|
{ read -r did st <"$wlog.done"; } 2>/dev/null || true
|
|
if [ -n "$did" ] && [ -n "$st" ] && [ "$did" = "$run" ]; then
|
|
if [ "$first" = 1 ]; then first=0; sleep "$settle"; continue; fi
|
|
cat "$wlog.summary" 2>/dev/null || echo "ygg-verify: exit $st, no readable summary file; see $wlog"
|
|
exit "$st"
|
|
fi
|
|
first=0
|
|
wpid=$(cat "$wlog.pid" 2>/dev/null || true)
|
|
el=$(( $(date +%s) - t0 ))
|
|
if [ -n "$wpid" ] && ! kill -0 "$wpid" 2>/dev/null; then
|
|
did=''; { read -r did st <"$wlog.done"; } 2>/dev/null || true
|
|
if [ -z "$did" ] || [ "$did" != "$(cat "$wlog.run" 2>/dev/null || true)" ]; then
|
|
echo "ygg-verify: DEAD - launcher $wpid is gone without a verdict; the job's outcome is unknown (log $wlog)"; exit 98
|
|
fi
|
|
continue
|
|
fi
|
|
if [ -z "$wpid" ] && [ "$el" -ge 30 ]; then
|
|
echo "ygg-verify: NOT STARTED - no $wlog.pid after 30 s; was the job launched with SUMMARY=$wlog?"; exit 98
|
|
fi
|
|
if [ "$el" -ge "$cap" ]; then
|
|
started=$(cat "$wlog.start" 2>/dev/null || echo "$t0")
|
|
echo "RUNNING $(( $(date +%s) - started ))s (log $wlog; last line: $(tail -n 1 "$wlog" 2>/dev/null | cut -c1-160))"
|
|
exit 99
|
|
fi
|
|
sleep 2
|
|
done
|
|
fi
|
|
|
|
if [ -n "${SUMMARY:-}" ] && [ -z "${YGG_SUMMARY_INNER:-}" ]; then
|
|
slog=$SUMMARY; child=''
|
|
mkdir -p "$(dirname "$slog")"
|
|
runid=$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n')
|
|
[ -n "$runid" ] || runid=$(date +%s)$$$RANDOM
|
|
export YGG_NONCE=$runid
|
|
: >"$slog"; echo $$ >"$slog.pid"; date +%s >"$slog.start"
|
|
echo "$runid" >"$slog.run.tmp"; mv "$slog.run.tmp" "$slog.run"
|
|
rm -f "$slog.done" "$slog.summary" # after the claim: a wait that sees the new id ignores these anyway
|
|
ygg_finish() { # summary first, done-file last: a visible .done always has its summary
|
|
ygg_summarize "$slog" "$1" "${2:-}" >"$slog.summary.tmp" && mv "$slog.summary.tmp" "$slog.summary" || true
|
|
echo "$runid $1" >"$slog.done.tmp"; mv "$slog.done.tmp" "$slog.done"
|
|
if [ -s "$slog.summary" ]; then cat "$slog.summary"; fi
|
|
exit "$1"
|
|
}
|
|
trap '[ -z "$child" ] || kill "$child" 2>/dev/null; ygg_finish 143 "launcher killed by a signal; the remote job may still be running"' TERM INT HUP
|
|
YGG_SUMMARY_INNER=1 "$0" "$@" >"$slog" 2>&1 &
|
|
child=$!
|
|
sst=0; wait "$child" || sst=$?
|
|
ygg_finish "$sst"
|
|
fi
|
|
|
|
repo=${1:?local repo}; rev=${2:?revision}; image=${3:?image}; cmd=${4:?command}
|
|
host=${YGG_HOST:-ygg}; cpus=${CPUS:-4}; mem=${MEM:-6g}; slots=${SLOTS:-5}
|
|
timeout_s=${TIMEOUT:-3600}; idle_s=${IDLE:-900}
|
|
docker_args=${DOCKER_ARGS:-}
|
|
sshopts=(-o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=4)
|
|
|
|
sha=$(git -C "$repo" rev-parse --verify "$rev^{commit}")
|
|
name=$(basename "$(git -C "$repo" rev-parse --show-toplevel)")
|
|
case "$image" in
|
|
rust|kotlin) image=eureka-verify-$image:$(sha256sum "$here/$image.Dockerfile" | cut -c1-12) ;;
|
|
dotnet) image=mcr.microsoft.com/dotnet/sdk:10.0 ;;
|
|
esac
|
|
|
|
ssh "${sshopts[@]}" "$host" "mkdir -p ~/eureka-verify/repos ~/eureka-verify/work ~/eureka-verify/images && \
|
|
{ test -d ~/eureka-verify/repos/$name.git || git init -q --bare ~/eureka-verify/repos/$name.git; }"
|
|
# ssh:// rather than scp-style host:path: Git LFS's pre-push hook rejects the
|
|
# scp form as a remote name and stalls (Aetheria, 2026-09-30). The verify mirror
|
|
# never needs LFS objects pushed; a job that needs one smudges it itself.
|
|
GIT_LFS_SKIP_PUSH=1 git -C "$repo" push -q "ssh://$host/~/eureka-verify/repos/$name.git" "$sha:refs/verify/$sha" --force
|
|
case "$image" in
|
|
eureka-verify-*:*) df=${image#eureka-verify-}; df=${df%%:*}
|
|
scp -q "$here/$df.Dockerfile" "$host:eureka-verify/images/$df.Dockerfile" ;;
|
|
esac
|
|
|
|
# ssh joins its arguments into one string that the remote shell splits again,
|
|
# so every argument is quoted here. Without that, `a && b` in the command runs
|
|
# b on the host.
|
|
remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s" "${YGG_NONCE:-}" "${RUN_ON_HOST:-0}")
|
|
# The heredoc below travels with whatever line endings this file has on disk.
|
|
# On Windows that is CRLF, and the remote bash then fails on `set -euo pipefail
|
|
`
|
|
# BEFORE `set -e` is in force, so the script limps on and can exit 0 — a job that
|
|
# never ran, reported as success (seen 2026-09-23). `tr -d` strips the CRs on the
|
|
# far side before bash ever sees them. The sentinel below is the second guard: if
|
|
# the remote script dies early, no verdict line is printed and the local side
|
|
# refuses to report a status it did not receive.
|
|
(ssh "${sshopts[@]}" "$host" "tr -d '
|
|
' | bash -s -- $remote_args" <<'REMOTE'
|
|
set -euo pipefail
|
|
name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} nonce=${12:-} host_mode=${13:-0}
|
|
root=~/eureka-verify
|
|
case "$image" in
|
|
eureka-verify-*:*)
|
|
df=${image#eureka-verify-}; df=${df%%:*}
|
|
if ! sudo docker image inspect "$image" >/dev/null 2>&1; then
|
|
sudo nice -n 10 docker build -q -t "$image" -f "$root/images/$df.Dockerfile" "$root/images" >/dev/null
|
|
fi ;;
|
|
esac
|
|
# Take whichever slot frees first. On 2026-09-22 a job waited on one fixed slot
|
|
# while another freed, and stayed stranded for over an hour.
|
|
slot=""; waited=0
|
|
while [ -z "$slot" ]; do
|
|
for i in $(seq 1 "$slots"); do
|
|
exec {fd}>"$root/slot-$i.lock"
|
|
if flock -n "$fd"; then slot=$i; break; fi
|
|
exec {fd}>&-
|
|
done
|
|
if [ -z "$slot" ]; then
|
|
[ "$waited" -eq 0 ] && echo "ygg-verify: all $slots slots busy; waiting" >&2
|
|
waited=1; sleep 5
|
|
fi
|
|
done
|
|
work=$(mktemp -d "$root/work/$name-${sha:0:10}-XXXX")
|
|
cname="eureka-verify-$slot-$$"
|
|
label="eureka-verify-job=$cname"
|
|
killed="$root/work/$cname.killed"
|
|
hostpg="$root/work/$cname.hostpg" # RUN_ON_HOST: the command's process group id, for the watchdog
|
|
trap 'rm -f "$killed" "$hostpg"; if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT
|
|
git clone -q --no-checkout "$root/repos/$name.git" "$work"
|
|
git -C "$work" checkout -q --detach "$sha"
|
|
echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $mem, work $work" >&2
|
|
# A job that hangs must come back red, not hold its slot and its caller forever
|
|
# (2026-09-23: a test process idled at 0% CPU for 25 minutes and the agent
|
|
# waiting on it never woke). The watchdog kills the named container; killing
|
|
# the docker client alone would leave the container running.
|
|
# The watchdog's one kill: every container the job started carries $label (a
|
|
# host-side command may own a baseline container and a mutant container at once,
|
|
# so there is no single name to kill). The flag tells that command the job is over.
|
|
# A RUN_ON_HOST command is also a process group of its own (set -m below): it gets TERM,
|
|
# then KILL 20 s later, so a command that starts no container (or starts one after the
|
|
# docker kill) dies too instead of running on past TIMEOUT (Soul, 2026-10-01).
|
|
kill_job() {
|
|
touch "$killed"
|
|
local ids pg; ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true)
|
|
[ -z "$ids" ] || sudo docker kill $ids >/dev/null 2>&1 || true
|
|
pg=$(cat "$hostpg" 2>/dev/null || true)
|
|
[ -z "$pg" ] || { kill -TERM -- "-$pg" 2>/dev/null || true; sleep 20; kill -KILL -- "-$pg" 2>/dev/null || true; }
|
|
}
|
|
# The subshell must not inherit the slot lock: its sleep outlives a job that
|
|
# finishes early, and an inherited fd held slot 2 for an hour (2026-09-23).
|
|
# It also kills a job that has idled for $idle_s seconds: docker's CPU% is per
|
|
# core, and a hung test reads ~2%, a working one tens to hundreds.
|
|
( exec {fd}>&- 2>/dev/null
|
|
start=$(date +%s); idle=0
|
|
while sleep 60; do
|
|
if [ $(( $(date +%s) - start )) -ge "$timeout_s" ]; then
|
|
echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3; kill_job
|
|
exit 0
|
|
fi
|
|
[ "$idle_s" -gt 0 ] || continue
|
|
ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true)
|
|
[ -n "$ids" ] || continue
|
|
cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' $ids 2>/dev/null | tr -d '%' | awk '{ s += $1 } END { if (NR) print s }' || true)
|
|
[ -n "$cpu" ] || continue
|
|
if awk -v c="$cpu" 'BEGIN { exit !(c < 5.0) }'; then idle=$((idle + 60)); else idle=0; fi
|
|
if [ "$idle" -ge "$idle_s" ]; then
|
|
echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3; kill_job
|
|
exit 0
|
|
fi
|
|
done ) 3>&2 &
|
|
watchdog=$!
|
|
set +e
|
|
if [ "$host_mode" = 1 ]; then
|
|
# Own process group (set -m), started in the job's work directory, so the watchdog can signal all of it.
|
|
set -m
|
|
( cd "$work" && export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \
|
|
YGG_KILLED=$killed DOCKER_ARGS=$docker_args
|
|
[ -z "$nonce" ] || export YGG_NONCE=$nonce
|
|
exec bash -o pipefail -c "$cmd" ) </dev/null &
|
|
hostpid=$!
|
|
set +m
|
|
echo "$hostpid" >"$hostpg"
|
|
wait "$hostpid"
|
|
status=$?
|
|
# The watchdog killed the job: that is the verdict, whatever the command exited with (a host
|
|
# command's status comes from the host command, not from a killed container).
|
|
if [ -e "$killed" ]; then status=124; fi
|
|
else
|
|
# Priority is set inside the container: nice on the docker client does not reach a process
|
|
# containerd starts. Jobs run at nice 10 and idle IO class, so the swarm's daemons and turns
|
|
# on the same disk go first (2026-10-31: three verify jobs held IO pressure at ~75% full).
|
|
sudo docker run --rm --name "$cname" --label "$label" --cpus="$cpus" --memory="$mem" \
|
|
-v "$work:/src" -v /etc/machine-id:/etc/machine-id:ro \
|
|
-v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages \
|
|
-v eureka-sccache:/sccache \
|
|
-e BASH_ENV=/tmp/.ygg-bash-env ${nonce:+-e YGG_NONCE=$nonce} \
|
|
-e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \
|
|
$docker_args \
|
|
-w /src "$image" sh -c 'if command -v ionice >/dev/null 2>&1; then exec nice -n 10 ionice -c3 "$@"; else exec nice -n 10 "$@"; fi' sh \
|
|
bash -o pipefail -c "printf 'set -o pipefail\n' >/tmp/.ygg-bash-env; $cmd"
|
|
status=$?
|
|
fi
|
|
set -e
|
|
pkill -P "$watchdog" 2>/dev/null || true
|
|
kill "$watchdog" 2>/dev/null || true
|
|
# Reap it. The subshell's own stderr is /dev/null, so killing its sleep
|
|
# prints no "Terminated" (agents misread that as a failed job); the TIMEOUT
|
|
# notice goes out through fd 3.
|
|
wait "$watchdog" 2>/dev/null || true
|
|
echo "ygg-verify: exit $status" >&2
|
|
echo "__YGG_VERDICT__ $status"
|
|
exit $status
|
|
REMOTE
|
|
) | {
|
|
# The remote prints __YGG_VERDICT__ <status> as its last line. Strip it from the
|
|
# output and exit with it. If it never arrives the remote script died before its
|
|
# own exit line, so refuse to report a status that was never produced.
|
|
verdict=97
|
|
line=''
|
|
while IFS= read -r line; do
|
|
case "$line" in
|
|
"__YGG_VERDICT__ "*) verdict=${line#__YGG_VERDICT__ }; continue ;;
|
|
esac
|
|
printf '%s\n' "$line"
|
|
done
|
|
if [ "$verdict" = 97 ]; then
|
|
echo "ygg-verify: NO VERDICT from the remote job - treating as failure" >&2
|
|
exit 97
|
|
fi
|
|
exit "$verdict"
|
|
}
|