Git LFS's pre-push hook rejected the scp-style host:path remote and stalled
every Aetheria verify (found by Imagination and Soul, 2026-09-30). The mirror
never needs LFS objects; jobs that need one smudge it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-09-30 three hung test binaries held all three slots for up to an hour
at ~2% CPU while Yggdrasil sat at load 1.5 of 16. The operator wants
Yggdrasil pinned. CPU caps now oversubscribe on purpose (jobs run niced);
memory is the ceiling, so the per-job cap drops to 8 GiB. The watchdog samples
docker stats each minute and kills a container under 5% of a core for IDLE
seconds (default 900), reporting it red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog subshell reported its own killed sleep as "Terminated" on
every job, successful or not. Agents read that as the job failing: one
Hands reported "the script reports exit 0 after Terminated even when the
job fails", which is false. Tested: exit 3 gives rc 3, and true gives rc 0.
Now the subshell's stderr is /dev/null and the TIMEOUT notice goes out
through fd 3. Checked: TIMEOUT=20 on sleep 60 prints TIMEOUT and exits
137, and a clean run prints no stray line. The watchdog is also reaped
with wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog added in 9359e9c ran in a subshell that inherited the slot
lock's file descriptor. When a job finished early, `kill $watchdog`
ended the subshell but not its `sleep` child. That orphaned sleep kept
the slot for up to TIMEOUT seconds. Two slots were held this way on
2026-09-23, and four jobs queued behind them.
The subshell now closes the lock fd before sleeping, and the cleanup
kills the watchdog's children before the watchdog itself.
Tested on Yggdrasil: a fast job exited 0, and a TIMEOUT=15 hang exited
137 with the TIMEOUT line. Afterwards no sleep process held any
slot lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Investigating a QUIC interop failure (native bridge client vs .NET managed
server, both on libmsquic, both over loopback) needs to test IPv6-disabled
Docker networking as a hypothesis, which the script had no way to express:
no docker-args hook existed. DOCKER_ARGS is forwarded verbatim into the
container's own `docker run`, word-split by the remote shell, appended right
before the image name — e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0'
or DOCKER_ARGS='--network host'. Empty by default, so every existing caller is
unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A Cut 4 test process sat at 0% CPU for 25 minutes on Yggdrasil. It had
no JS work left, but its event loop never exited. The Hands agent
waiting on the job was never woken, and nothing moved until Self went
looking.
The job's container is now named, and a watchdog kills it after
TIMEOUT seconds (default 3600). Killing the docker client alone would
have left the container running. A timed-out job prints a TIMEOUT line
and exits 137.
Tested on Yggdrasil: TIMEOUT=20 on `sleep 90` exited 137 with the
TIMEOUT line, and a fast job exited 0. No containers were left behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job piped into tail loses its exit status, because the command runs under
bash -c and the sentinel then reports the pipeline's zero rather than the
job's. That is the same false-green the CRLF fix closed, arriving by a
different road, and it cost a Soul run today.
The second is smaller and just as costly to rediscover: CultLib has no
Cargo.toml at its root, so a rust job must cd into the package first or cargo
fails in a way that looks like a broken image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul lost a run to CRLF and it still exited 0. The cause: this file's own line
endings travel with the heredoc, so the remote bash fails on `set -euo
pipefail\r` before `set -e` is in force, limps past the failure and exits
clean. A verification runner that can report success without running anything
is worse than no runner.
Two guards, because one was not enough to notice this for a day. The remote
now strips carriage returns before bash sees them, and it prints a verdict
sentinel the local side requires; a missing sentinel is treated as failure
rather than as silence. A .gitattributes keeps the file LF on disk so the
first guard is rarely needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator: "throw in a third slot, the other cores are mostly sitting idle".
Verification now takes 12 of Yggdrasil's 16 CPUs at most, and the live
services keep four.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator ruling 2026-09-22: better nothing than a harness that punishes
refactoring. Behavioural tests plus the ecosystem's mutation tools on a cut's
diff; Soul's hand mutants stay one-off probes. The stopgap image carries
cargo-mutants instead of pwsh and is tagged by its Dockerfile's hash.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A worktree's .git points at a host path the container cannot see, so
Ghostlight's build.rs failed asking git for the commit. Editing the running
script in place killed a job with 'unexpected EOF': bash reads scripts from disk
as it runs, so new versions go in by rename.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A job that found both slots busy blocked on slot 1 alone while slot 2 freed,
and ssh sessions whose remote end was gone hung on Starfire for over an hour
with no keepalive to notice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul found 36 Idunn tests fail without it, so Hands' numbers only reproduced
after a hand-made machine-id inside the container.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every hash comparison in the script (sidecar verification, mutant
verification, restore verification: Assert-TargetUnedited, Restore-
Targets, the startup repair loop, and M0) used PowerShell's plain -eq/
-ne, which is case-insensitive. Hashes are hex strings where case never
carries meaning here (Get-Hash and Get-BytesHash always produce a fixed
case each), so this was latent, not exploitable -- but it left every
"is this file still what I think it is" check one string-comparison
operator away from silently accepting a corrupted file whose hex
happened to differ only in case. Switched all eight sites to -ceq/-cne.
Hash case is chosen to matter, both for defence in depth and because
these functions are the harness's only proof that a restore or a
mutant write landed the intended bytes.
The no-op guard that rejects a mutation whose old and new text come out
identical also used -eq: `if ($old -eq $new)`. Under case-insensitive
comparison a mutant that differs from the source only in case, such as
`"query"` to `"Query"`, compared equal and was refused as a no-op even
though it is a real source change. Switched to -ceq.
Anchor matching (Measure-Sites, and the splice-site IndexOf that
follows it) already used [System.StringComparison]::Ordinal, which is
case-sensitive; checked and left unchanged.
Demonstrated in a scratch repo (demo-case) with a single-mutation suite
whose only edit is `"query"` -> `"Query"` against `fn f() -> query { 1
}`, and a fake test command that fails unless the unmutated lowercase
text is present (i.e. is killed by the mutation).
Before: "CaseDemo: replacement changes nothing in t.txt.", exit 1,
mutation never ran.
After: the mutation runs, the fake test fails against the mutated text,
verdict "killed", M0 green, exit 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
H1 (Assert-TargetUnedited, :~166-220) keeps a target's sidecar rather
than removing it when it catches an edit the harness did not make, so
the M0 original survives beside the lost edit. Startup repair (the
sidecar loop after "Repair a previous run that died mid-mutation") could
not tell that kept sidecar apart from a crash's: on the next run it
restored the M0 original over whatever an operator had since hand-
cleaned into the target, moved the operator's clean file to
`.eureka-mutation-overwritten` (clobbering H1's own copy there before
the F2 fix), printed "a previous run died mid-mutation", removed the
sidecar, and finished green. The hand-cleaned edit was gone one run
later with no non-zero exit to notice it by.
Get-H1MarkerPath names a marker, `<target>.eureka-h1-stopped`, that
Assert-TargetUnedited now writes beside the sidecar whenever it keeps
one. Startup repair now scans for these markers before touching any
sidecar; if any exist it refuses the whole pass, restores nothing,
prints which files to reconcile and where their marker is, and exits
non-zero. An operator deletes the marker (and the sidecar, once done
with it) by hand to let repair run again.
Demonstrated in a scratch repo (demo-f1): H1 fired (a fake test command
appended stray bytes to the target mid-run), leaving a sidecar, a
marker, and an overwritten copy. The target was then hand-cleaned to
"alpha-HANDS-EDIT" and the harness rerun.
Before (original script, ignoring the marker): exit 0, target silently
reverted to the M0 original ("alpha"), the hand-cleaned edit gone.
After (fixed script): exit 1, "Startup repair refuses to run: 1 file(s)
were stopped by H1 in a previous run, not crashed...", target still
"alpha-HANDS-EDIT", sidecar/marker/overwritten all untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Assert-TargetUnedited (tools/eureka-mutations.ps1:172-173 before this
change) and the startup sidecar repair (:342) both wrote an unconditional
"$path.eureka-mutation-overwritten" whenever they preserved bytes that
were about to be overwritten. A second EDIT LOST or a second dead-mid-
mutation repair against the same target silently destroyed the first
preserved copy -- and so did an operator's own file already sitting at
that name.
Get-UniqueOverwrittenPath now picks the first unused name in the family
(".eureka-mutation-overwritten", then ".eureka-mutation-overwritten.1",
".2", ...) before either site writes. Every preserved copy now survives
beside every other.
Demonstrated in a scratch repo (demo-f2): a file was planted at
t.txt.eureka-mutation-overwritten holding "PRECIOUS-PRE-EXISTING-COPY",
then an H1 edit-lost event was triggered against the same target.
Before: the planted file's content was replaced with the newly-lost
bytes; only one copy survived.
After: the planted file is untouched (still "PRECIOUS-PRE-EXISTING-COPY")
and the new loss lands at t.txt.eureka-mutation-overwritten.1; both
copies survive.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Restore-Targets never checked what a target's bytes actually were before
restoring over them, so a build step or stray edit landing mid-run or
between entries was overwritten without a trace (Soul's finding on the
Huginn memory-organ Cut 10 fourth fix batch, 2026-09-22; that same batch
had lost a Hands edit this way). Restore-Targets and the entry loop's
pre-write step now hash the target first: a match against the M0 original
or this entry's own mutant restores as before, anything else is saved to
<target>.eureka-mutation-overwritten, warned about loudly, and fails the
run with a non-zero exit and no further entries.
Demonstrated in the scratchpad with a throwaway repo whose test command
appends a line to the target mid-run: pre-fix the run went green and the
target came back pristine; post-fix it failed loudly and kept the
appended line. Recorded in references/changelog.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The mutation harness lived in Epiphany because that is the repo that needed it
first, and Huginn reached across for it. It takes the repository as a
parameter, so one copy serves every campaign and a fix to it fixes all of
them. A repo that cannot run it keeps its own runner and owes the same
contract by name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
