nice on the docker client never reached the job, which containerd starts.
On 2026-10-31 three verify jobs held Yggdrasil's IO pressure near 75% full
and the swarm's turns slowed; idle IO lets the swarm go first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A RUN_ON_HOST job whose TIMEOUT fired still exited 0: kill_job only killed
labelled containers and the status came from the host command (Soul F2). The
host command now runs in its own process group (set -m), in the job's work
directory (it ran in the ssh home), with stdin from /dev/null. kill_job sends
the group TERM, then KILL 20 s later, and the verdict is forced to 124 when the
watchdog killed the job, whatever the command exited with. The TIMEOUT/IDLE
notice now prints before the kill so the 20 s grace cannot swallow it.
Proved on Yggdrasil with the sleep-past-TIMEOUT probe (exit 124, cwd = the work
dir, no sleep left) and with a command that traps TERM and exits 0 (still 124);
probes recorded in tools/fixtures/mutation-probe/README.txt, with the expected
rows for the fresh and jsr reproducers.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Jest without a TTY prints "FAIL jt/add.test.js" with one space; the parser
matched two, so a real jest load failure read ERROR, not UNVIABLE. The old
`^ +● ` denylist also named process.exit's "● process.exit called with "1""
as a killing test (CAUGHT). Names now need a FAIL file header above them; a
crash with no named failing test is ERROR. The "no test counts recognised"
line names jest. The synthetic jest logs and mock are replaced by real jest
29.7.0 output captured on Yggdrasil from tools/fixtures/mutation-probe/jsr
(green, wrong, load, exit runs in logs/jest-real.log).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The git-status restore could not see ignored files, so a test keeping state under
target/ polluted the next mutant (an uncovered mutant read CAUGHT). The baselines now
run once in one container whose filesystem holds the revision; that container is
`docker commit`ted to a throwaway image and each mutant runs in a fresh --rm container
from it. restore(), dirtied() and tree_state() are deleted. ygg-verify.sh gains
RUN_ON_HOST=1 (the driver owns container topology; the watchdog kills by job label).
testparse.sh: LOADFAIL for file-level FAIL lines (no " > "), jest "●" names and jest
Tests: counts. Mock docker, jest and vitest-variant fixtures, soul2 reproducer.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Carried summary lines must start YGG-SUMMARY/<nonce>:, the nonce exported into the
job; others are counted and ignored. Carried lines no longer drop failure sections.
Red summaries say NAMES MISSING when counts exceed names. wait trusts a .done only
when its run id matches .run; a missing summary never yields a bare exit 1.
mutation-table.sh must print the nonce prefix (separate branch).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
SUMMARY=<log> keeps the full log and prints a bounded verdict summary;
wait <log> blocks at most 270 s and returns it once. mutation-table.sh runs
hand mutants in one verify job and returns a kill table. testparse.sh is the
one parser for both. Fixtures: real cargo and mutation logs from Yggdrasil
(tiny-crate), plus hand-written dotnet, vitest, pytest, exit-0-but-failed and
no-verdict logs. Every commit here builds; verified live on Yggdrasil.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
cargo-mutants --jobs N workers all built into /src/target and tested each
other's binaries, so parallel kill counts were noise (idunn-watchdog Soul
s3). Cargo's default target dir is already inside /src.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sccache 0.18.0 as RUSTC_WRAPPER with a shared eureka-sccache volume.
Idunn lib test build: 51 s cold, 29 s warm (83/83 hits). Slot wait, not
compilation, is the larger cost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exporting SHELLOPTS also exported a job's own `set -u` to every bash-wrapped
tool; kotlinc died on `JAVA_OPTS: unbound variable` (put-serve Hands, 2026-09-30).
Probed on Yggdrasil: `false | cat` in a child script still returns 1, and a
parent's `set -u` no longer reaches child shells.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Soul proved bash -o pipefail -c "bash job.sh" leaves the script without
pipefail; a failing piped test came back exit 0. Probe: false|cat exits 1,
true|cat exits 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 8 slots /proc/pressure/io showed full avg60=41% and live Odin fsyncs
stalled route challenges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ygg-verify.sh gains a kotlin alias tagged by kotlin.Dockerfile's hash; the
remote build step is generalised from rust.Dockerfile to <name>.Dockerfile.
PowerShell comes from the GitHub tarball because the Ubuntu 26.04 base has no
Microsoft apt package. Proven with CultLib ack Cut K0: green on hands/ack-K0,
red on a deliberately failing self-test.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Measured 2026-09-30 00:17 CEST: 40% CPU idle, 53 of 62 GiB available,
~20 jobs queued, running jobs ~1 GiB each. 8 x 6 GiB worst case still
leaves the services their memory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ack Cut 2 job piped a failing test run into cut and reported exit 0.
The rake was documented; agents kept stepping on it. pipefail now holds
for every job; the comment says what it still does not rescue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Idunn verification now runs cargo check --target x86_64-pc-windows-gnu
--bin idunn-host so a Windows-only compile break is caught on Yggdrasil.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Git LFS's pre-push hook rejected the scp-style host:path remote and stalled
every Aetheria verify (found by Imagination and Soul, 2026-09-30). The mirror
never needs LFS objects; jobs that need one smudge it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-09-30 three hung test binaries held all three slots for up to an hour
at ~2% CPU while Yggdrasil sat at load 1.5 of 16. The operator wants
Yggdrasil pinned. CPU caps now oversubscribe on purpose (jobs run niced);
memory is the ceiling, so the per-job cap drops to 8 GiB. The watchdog samples
docker stats each minute and kills a container under 5% of a core for IDLE
seconds (default 900), reporting it red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog subshell reported its own killed sleep as "Terminated" on
every job, successful or not. Agents read that as the job failing: one
Hands reported "the script reports exit 0 after Terminated even when the
job fails", which is false. Tested: exit 3 gives rc 3, and true gives rc 0.
Now the subshell's stderr is /dev/null and the TIMEOUT notice goes out
through fd 3. Checked: TIMEOUT=20 on sleep 60 prints TIMEOUT and exits
137, and a clean run prints no stray line. The watchdog is also reaped
with wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog added in 9359e9c ran in a subshell that inherited the slot
lock's file descriptor. When a job finished early, `kill $watchdog`
ended the subshell but not its `sleep` child. That orphaned sleep kept
the slot for up to TIMEOUT seconds. Two slots were held this way on
2026-09-23, and four jobs queued behind them.
The subshell now closes the lock fd before sleeping, and the cleanup
kills the watchdog's children before the watchdog itself.
Tested on Yggdrasil: a fast job exited 0, and a TIMEOUT=15 hang exited
137 with the TIMEOUT line. Afterwards no sleep process held any
slot lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Investigating a QUIC interop failure (native bridge client vs .NET managed
server, both on libmsquic, both over loopback) needs to test IPv6-disabled
Docker networking as a hypothesis, which the script had no way to express:
no docker-args hook existed. DOCKER_ARGS is forwarded verbatim into the
container's own `docker run`, word-split by the remote shell, appended right
before the image name — e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0'
or DOCKER_ARGS='--network host'. Empty by default, so every existing caller is
unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A Cut 4 test process sat at 0% CPU for 25 minutes on Yggdrasil. It had
no JS work left, but its event loop never exited. The Hands agent
waiting on the job was never woken, and nothing moved until Self went
looking.
The job's container is now named, and a watchdog kills it after
TIMEOUT seconds (default 3600). Killing the docker client alone would
have left the container running. A timed-out job prints a TIMEOUT line
and exits 137.
Tested on Yggdrasil: TIMEOUT=20 on `sleep 90` exited 137 with the
TIMEOUT line, and a fast job exited 0. No containers were left behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job piped into tail loses its exit status, because the command runs under
bash -c and the sentinel then reports the pipeline's zero rather than the
job's. That is the same false-green the CRLF fix closed, arriving by a
different road, and it cost a Soul run today.
The second is smaller and just as costly to rediscover: CultLib has no
Cargo.toml at its root, so a rust job must cd into the package first or cargo
fails in a way that looks like a broken image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul lost a run to CRLF and it still exited 0. The cause: this file's own line
endings travel with the heredoc, so the remote bash fails on `set -euo
pipefail\r` before `set -e` is in force, limps past the failure and exits
clean. A verification runner that can report success without running anything
is worse than no runner.
Two guards, because one was not enough to notice this for a day. The remote
now strips carriage returns before bash sees them, and it prints a verdict
sentinel the local side requires; a missing sentinel is treated as failure
rather than as silence. A .gitattributes keeps the file LF on disk so the
first guard is rarely needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator: "throw in a third slot, the other cores are mostly sitting idle".
Verification now takes 12 of Yggdrasil's 16 CPUs at most, and the live
services keep four.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator ruling 2026-09-22: better nothing than a harness that punishes
refactoring. Behavioural tests plus the ecosystem's mutation tools on a cut's
diff; Soul's hand mutants stay one-off probes. The stopgap image carries
cargo-mutants instead of pwsh and is tagged by its Dockerfile's hash.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A worktree's .git points at a host path the container cannot see, so
Ghostlight's build.rs failed asking git for the commit. Editing the running
script in place killed a job with 'unexpected EOF': bash reads scripts from disk
as it runs, so new versions go in by rename.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A job that found both slots busy blocked on slot 1 alone while slot 2 freed,
and ssh sessions whose remote end was gone hung on Starfire for over an hour
with no keepalive to notice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul found 36 Idunn tests fail without it, so Hands' numbers only reproduced
after a hand-made machine-id inside the container.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
