Git LFS's pre-push hook rejected the scp-style host:path remote and stalled
every Aetheria verify (found by Imagination and Soul, 2026-09-30). The mirror
never needs LFS objects; jobs that need one smudge it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-09-30 three hung test binaries held all three slots for up to an hour
at ~2% CPU while Yggdrasil sat at load 1.5 of 16. The operator wants
Yggdrasil pinned. CPU caps now oversubscribe on purpose (jobs run niced);
memory is the ceiling, so the per-job cap drops to 8 GiB. The watchdog samples
docker stats each minute and kills a container under 5% of a core for IDLE
seconds (default 900), reporting it red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog subshell reported its own killed sleep as "Terminated" on
every job, successful or not. Agents read that as the job failing: one
Hands reported "the script reports exit 0 after Terminated even when the
job fails", which is false. Tested: exit 3 gives rc 3, and true gives rc 0.
Now the subshell's stderr is /dev/null and the TIMEOUT notice goes out
through fd 3. Checked: TIMEOUT=20 on sleep 60 prints TIMEOUT and exits
137, and a clean run prints no stray line. The watchdog is also reaped
with wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog added in 9359e9c ran in a subshell that inherited the slot
lock's file descriptor. When a job finished early, `kill $watchdog`
ended the subshell but not its `sleep` child. That orphaned sleep kept
the slot for up to TIMEOUT seconds. Two slots were held this way on
2026-09-23, and four jobs queued behind them.
The subshell now closes the lock fd before sleeping, and the cleanup
kills the watchdog's children before the watchdog itself.
Tested on Yggdrasil: a fast job exited 0, and a TIMEOUT=15 hang exited
137 with the TIMEOUT line. Afterwards no sleep process held any
slot lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Investigating a QUIC interop failure (native bridge client vs .NET managed
server, both on libmsquic, both over loopback) needs to test IPv6-disabled
Docker networking as a hypothesis, which the script had no way to express:
no docker-args hook existed. DOCKER_ARGS is forwarded verbatim into the
container's own `docker run`, word-split by the remote shell, appended right
before the image name — e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0'
or DOCKER_ARGS='--network host'. Empty by default, so every existing caller is
unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A Cut 4 test process sat at 0% CPU for 25 minutes on Yggdrasil. It had
no JS work left, but its event loop never exited. The Hands agent
waiting on the job was never woken, and nothing moved until Self went
looking.
The job's container is now named, and a watchdog kills it after
TIMEOUT seconds (default 3600). Killing the docker client alone would
have left the container running. A timed-out job prints a TIMEOUT line
and exits 137.
Tested on Yggdrasil: TIMEOUT=20 on `sleep 90` exited 137 with the
TIMEOUT line, and a fast job exited 0. No containers were left behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job piped into tail loses its exit status, because the command runs under
bash -c and the sentinel then reports the pipeline's zero rather than the
job's. That is the same false-green the CRLF fix closed, arriving by a
different road, and it cost a Soul run today.
The second is smaller and just as costly to rediscover: CultLib has no
Cargo.toml at its root, so a rust job must cd into the package first or cargo
fails in a way that looks like a broken image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul lost a run to CRLF and it still exited 0. The cause: this file's own line
endings travel with the heredoc, so the remote bash fails on `set -euo
pipefail\r` before `set -e` is in force, limps past the failure and exits
clean. A verification runner that can report success without running anything
is worse than no runner.
Two guards, because one was not enough to notice this for a day. The remote
now strips carriage returns before bash sees them, and it prints a verdict
sentinel the local side requires; a missing sentinel is treated as failure
rather than as silence. A .gitattributes keeps the file LF on disk so the
first guard is rarely needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator: "throw in a third slot, the other cores are mostly sitting idle".
Verification now takes 12 of Yggdrasil's 16 CPUs at most, and the live
services keep four.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator ruling 2026-09-22: better nothing than a harness that punishes
refactoring. Behavioural tests plus the ecosystem's mutation tools on a cut's
diff; Soul's hand mutants stay one-off probes. The stopgap image carries
cargo-mutants instead of pwsh and is tagged by its Dockerfile's hash.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A worktree's .git points at a host path the container cannot see, so
Ghostlight's build.rs failed asking git for the commit. Editing the running
script in place killed a job with 'unexpected EOF': bash reads scripts from disk
as it runs, so new versions go in by rename.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A job that found both slots busy blocked on slot 1 alone while slot 2 freed,
and ssh sessions whose remote end was gone hung on Starfire for over an hour
with no keepalive to notice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul found 36 Idunn tests fail without it, so Hands' numbers only reproduced
after a hand-made machine-id inside the container.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
