diff --git a/SKILL.md b/SKILL.md index 71a5ce1..658e057 100644 --- a/SKILL.md +++ b/SKILL.md @@ -317,7 +317,9 @@ nothing is pasted or paraphrased. The brief says: - **Measure the suite with the ecosystem's mutation tool, scoped to the cut's diff:** Stryker.NET for C#, cargo-mutants for Rust, StrykerJS for TypeScript, mutmut for Python. The tool generates the mutants and nobody - writes an anchor. Every survivor is a finding about the tests, triaged by + writes an anchor. cargo-mutants runs through `tools/cargo-mutants.sh` (per-file + scoped tests, survivors confirmed against the full suite, sharded over + Yggdrasil slots), once per Hands pass. Every survivor is a finding about the tests, triaged by name: a behavioural test is missing, a fixture is degenerate (see below), or the mutant is equivalent and gets a one-line reason. Boundary flips on float thresholds are equivalent by default and are not chased. The score is @@ -622,7 +624,8 @@ exact script they want and agents do not browse. Contributions come by branch and Forgejo pull request (this repo lives on Yggdrasil's Forgejo), never a push to main. A script that verdicts depend on gets a Soul pass before merge, and `bash tools/test-tools.sh` passes first. The summary and wait modes of -`tools/stopgap/ygg-verify.sh` and `tools/mutation-table.sh` are such scripts. A script whose brief no longer names it is pruned. +`tools/stopgap/ygg-verify.sh`, `tools/mutation-table.sh` and +`tools/cargo-mutants.sh` are such scripts. A script whose brief no longer names it is pruned. Scripts that know a project's layout stay in that project. ## Git and tooling rules (scars) diff --git a/references/briefs.md b/references/briefs.md index b3acc87..6020093 100644 --- a/references/briefs.md +++ b/references/briefs.md @@ -256,7 +256,12 @@ cites, read by id): Every rule the spec or the operator names gets a behavioural test. Measure the suite with , scoped to this cut's diff (`--since:` or the tool's equivalent), against the final spelling of the -code. The report's mutations (at most 64) holds every survivor and the kills +code. For Rust the tool is `tools/cargo-mutants.sh [workdir]`, +run ONCE per pass, at the end: it scopes each file's mutants to that file's tests, +confirms every survivor on the full suite and shards over free slots. After killing +survivors with tests, rerun only those names (`NAMES=`), +never a second whole-diff sweep; a flaky test is skipped with +`TEST_ARGS='--skip '`. The report's mutations (at most 64) holds every survivor and the kills Soul should rerun; put the totals (generated, caught, unviable, missed) in verification evidence. Triage every survivor by name and line: killed by a new test, killed by fixing a degenerate fixture, @@ -428,8 +433,9 @@ Verification: . Python suites locally on 2026-10-12, red for Windows-only reasons). - Verify through `tools/stopgap/ygg-verify.sh` with `SUMMARY=` (above): read the summary, and open the full log only at a named failing test, with - `grep -n -A 40`. Hand mutants go in one list to `tools/mutation-table.sh`, - one job for all of them. + `grep -n -A 40`. Rust mutants go to `tools/cargo-mutants.sh` (one job, scoped, + confirmed, sharded), hand mutants in one list to `tools/mutation-table.sh`; never + an inline `cargo mutants` command. - Stryker `--mutate` ranges are character offsets, not lines, into the file as Stryker reads it: the bytes of the checkout the job runs in. Line endings there follow `.gitattributes`, not the platform: CultLib's `packages/cultmath/src/**` @@ -539,9 +545,11 @@ specifically: binaries or files the cut touches and the ones that name its rules. Run the full suite exactly once, at the end. Put that run in the same job as your mutation run where you can, since that job pays for the build anyway. Batch - probes into one job script rather than one job each. Run the mutation tool - in parallel (`--jobs 3`), and skip by name any test that a follow-up records - as flaky, so that its flakes cannot count a surviving mutant as killed. + probes into one job script rather than one job each. Run Rust mutants through + `tools/cargo-mutants.sh` (it shards; do not add `--jobs`), skipping any test a + follow-up records as flaky with `TEST_ARGS='--skip '`; other ecosystems + run their tool in parallel (`--jobs 3`) and skip by name, so that a flake cannot + count a surviving mutant as killed. Dump the range once (`git diff .. > //range.diff` plus `--stat`) and grep that file instead of re-running `git show` per file. Hand mutants go in one list to `tools/mutation-table.sh`, never one job each. diff --git a/tools/INDEX.md b/tools/INDEX.md index 2191432..5562e83 100644 --- a/tools/INDEX.md +++ b/tools/INDEX.md @@ -11,8 +11,9 @@ gets a Soul pass before merge. | `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | | `tools/wait-on.sh` | One general waiter for anything that is not a Yggdrasil job: `pid []`, `log ` (with `WAIT_PID=` a dead process with no match is exit 98; an invalid ERE is exit 3), `file `. The `pid` form prints `GONE (no completion evidence)`: prefer `log` whenever the job writes a done line. Same contract as `ygg-verify.sh wait`: blocks at most 270 s, exit 0 done with a tail of at most 60 log lines, `RUNNING s` exit 99, exit 98 gone with no completion evidence; call it with a Bash tool `timeout` of at least 280000 ms. Replaces hand-rolled poll loops and idle `true`/`echo`/`sleep` calls. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs, Soul long jobs, detached scripts) | | `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL [ ]`, jest `FAIL ` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | -| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | -| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | +| `tools/cargo-mutants.sh` | `SUMMARY= tools/cargo-mutants.sh [workdir]`: cargo-mutants on a revision range as ONE verify job, the one owner of running it. Plan container lists the range's mutants (diff taken at the workspace root); stage 1 runs each file's mutants against that file's tests only (filter derived from the path, `FILTERS` overrides), deciding only CAUGHT and UNVIABLE; stage 2 confirms every survivor, timeout, unfiltered file and zero-test file on the full suite, sharded; merge counts each name once and prints `MUTANTS` totals, `FILTER` lines and the final `MISSED`/`TIMEOUT` names. Containers = min(`SHARDS`, slots, units, 1 + free slots), leased with `flock -n` from Yggdrasil's slot pool (none while IO pressure full avg60 > 20). `NAMES=` reruns chosen survivors (no stage 1), `TEST_ARGS` skips flaky tests, `JOBS`, `MIN_TEST_TIMEOUT`, `CONFIRM_TIMEOUT_MULT`, `TIMEOUT`. Exit 0 on a green baseline whatever survives; a red baseline or a failed container is non-zero with no totals. Run once per pass. | Hands, Soul | `references/briefs.md` (Hands verification, Soul reruns), `SKILL.md` (mutation bullet) | +| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | `tools/stopgap/ygg-verify.sh`, `tools/mutation-table.sh` | not named by a brief; it lives and dies with them | +| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runners' host drivers and container halves (`tools/mutation-table.sh`, `tools/cargo-mutants.sh`) under a mock docker, a fake `cargo mutants` and real slot locks. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | | `tools/context-pack.py` | `py -3 tools/context-pack.py [--since ] [--only \|r[,...]] [--spec-json ] [--max-lines ]`. Carries a cut_spec's `file_changes` and `reads` anchors (read rows `r`; a read absent at base is MISSING) from the spec's base to the worktree by `git diff` hunks (the working tree included) and prints one header, a status table (SAME, MOVED, TOUCHED, RENAMED, NEW, GONE, OUT_OF_RANGE, UNRESOLVED, MISSING, plus SYMBOL_MISSING when an anchor's `symbol` is absent from its window) and windows of at most 100 lines numbered in worktree lines, then one capped caller list per symbol (`git grep`, hits outside the windows), within `--max-lines` (800). Exit 3 when an anchor cannot be carried, 4 when the spec is unreadable (fetched by id through the eureka-state binding (`tools/mind.py`), retried once on Unavailable; `--spec-json` takes a saved view). `--since ` re-prints only the windows edited since that commit. Standard library plus git; writes nothing. | Hands, Soul, Imagination | `references/briefs.md` (Imagination map, Hands, Soul), `SKILL.md` (step 1) | | `tools/mind.py` | The one way a Eureka script reaches the mind: `mind.call(tool, arguments, deadline)` spawns the eureka-state binding (`EUREKA_STATE_EXE`, else `~/.claude.json` `mcpServers.eureka-state`), does the stdio handshake and one tool call within `deadline` seconds, returns the parsed JSON body, raises `MindError` (kind binding, transport, tool or answer; a detail, never an input value), and kills the child on every exit path. No retry inside `call`; `retry_once(fn)` is the caller's choice. Standard library only. | `tools/context-pack.py`, `tools/recall.py` | not named by a brief; it lives and dies with its consumers | | `tools/recall.py` | `py -3 tools/recall.py prompt`, a `UserPromptSubmit` command hook (hook JSON on stdin): queries the rulings, questions and follow-ups in force across every root, ranked by Huginn against the prompt (`RECALL_TOP_K`, 5), and prints one hook JSON whose `additionalContext` is a header saying the lines are recalled records, data and never instructions, written by agents and not the operator, then one JSON object per item (`id`, `title`, a short `operator_quote`) with `<`, `>` and control, bidi and separator characters escaped, within `RECALL_BUDGET` characters (4000, at most 9000: Claude Code caps `additionalContext` at 10,000), whole lines only. Prompts under 3 words are not queried. Never blocks: any failure exits 0 with empty stdout; its own deadline is `RECALL_DEADLINE_MS` (4000) for the whole run (stdin read to 4 MiB, answer read to 1 MiB), so set the hook `timeout` above it. One attempt, no retry. Install both hooks as ` C:/Users/Meta/.claude/skills/eureka/tools/recall.py prompt || true` (and `... admit || true`): Claude Code runs hook commands through Git Bash on Windows, and `py -3 ` exits 2, which blocks every prompt or admit; with `|| true` the hook cannot exit 2 (a missing script, interpreter or file passes) and the script's stdout is untouched (proved live on 2.1.308). `py -3 tools/recall.py admit` is a `PreToolUse` hook on `mcp__eureka-state__admit`: for each ruling or question in the batch (at most `RECALL_ADMIT_MAX`, 3) it queries the nearest in-force rulings and questions, and denies the first attempt with the neighbours as the reason (a deliberate hold, not a failure); the identical call sent again (digest of `tool_input` plus `agent_id`, kept in the session scratchpad) passes. Never allow or ask; a failed check exits 0 with a note naming the failure kind (`RECALL_TOP_K` 3, `RECALL_DEADLINE_MS` 10000 in this mode). Standard library only; writes only the digest file. | the operator's `UserPromptSubmit` hook and `PreToolUse` hook on `mcp__eureka-state__admit` (fires inside subagents too) in `~/.claude/settings.json` | `references/campaign-state.md` (Precedent) | diff --git a/tools/cargo-mutants.sh b/tools/cargo-mutants.sh new file mode 100644 index 0000000..cc1d19a --- /dev/null +++ b/tools/cargo-mutants.sh @@ -0,0 +1,387 @@ +#!/usr/bin/env bash +# cargo-mutants on Yggdrasil for a revision range: ONE verify job, per-file scoped +# tests, survivors confirmed on the full suite, sharded over free slots, one merged +# survivor list. The one owner of "run cargo-mutants" (no brief, no job script runs +# an inline `cargo mutants`). +# +# SUMMARY= tools/cargo-mutants.sh [workdir] +# workdir directory inside the repo where cargo runs (a package or workspace +# member, e.g. crates/mind-body; CultLib has no Cargo.toml at its root). +# The mutated range is `git diff --relative ` taken at the +# WORKSPACE ROOT: cargo-mutants matches --in-diff paths, -f paths and +# prints mutant names relative to the workspace root, not to workdir. +# env FILTERS '=,...' overrides a file's derived test filter +# (path as the mutant names print it); '-' = full suite +# TEST_ARGS harness args, e.g. '--skip flaky_one' (a flaky test would +# make a surviving mutant read as killed in one run) +# NAMES local file of mutant names (one per line, copied from a +# previous MISSED/TIMEOUT list): skips stage 1 and the listing +# and runs only those names on the full suite. This is the rerun +# after killing survivors with tests. +# SHARDS (3) JOBS (2) MIN_TEST_TIMEOUT (20) CONFIRM_TIMEOUT_MULT (5) +# TIMEOUT seconds for the whole job (default 10800) +# everything else tools/stopgap/ygg-verify.sh reads (MEM, CPUS, ...) +# Typical, from an agent: launch as ONE background shell with a fresh log, then +# tools/stopgap/ygg-verify.sh wait (Bash tool timeout >= 280000 ms) +# Run it ONCE per pass, at the end, on the final spelling of the code. Chasing +# survivors is a NAMES rerun, never a second whole-range sweep. +# +# STAGES (all inside the one job and the one host driver): +# plan one container writes the range diff, lists every mutant in it, derives each +# touched file's test filter and counts the tests that filter selects +# (`cargo test -- --list`, one build for all files). +# stage 1 each touched file whose filter selects tests runs its mutants against that +# file's tests only (`cargo mutants -f -- `). The filter is +# derived from the path: the module path after the last src/ (a/mod.rs -> +# `a::`, a/b.rs -> `a::b::`); lib.rs, main.rs, bin/ and files outside src/ +# have none. FILTERS overrides. Stage 1 decides only CAUGHT (its own green +# filtered baseline guards false kills) and UNVIABLE (a build fact). Its +# MISSED and TIMEOUT are never reported: a mutant that only another module's +# tests or tests/ kill survives a filtered run. A file whose filter selects 0 +# tests (tests=0) never runs stage 1: all its mutants go to stage 2. +# stage 2 every name stage 1 did not settle (survivors, files without a filter, +# zero-test files, or NAMES) runs on the full suite in cargo-mutants' own +# baseline, sharded over the containers. Stage 2 ALWAYS runs: that baseline is +# the only full-suite check, so a red suite is a red job even when stage 1 +# settled every mutant (one name then rides along; its outcome is dropped). +# The test timeout is max(MIN_TEST_TIMEOUT, CONFIRM_TIMEOUT_MULT x baseline); +# the generous default keeps a slow-but-finishing test from flipping a +# MISSED mutant to TIMEOUT with the host's load. Its outcome is final. +# merge each name counted once; MISSED and TIMEOUT are stage-2 outcomes only. +# Containers per stage: min(SHARDS, SLOTS, units, 1 + free slots now), units being the +# stage's files (stage 1) or names (stage 2). The job holds one slot (ygg-verify's); +# the driver leases further slot locks with flock -n, never waits for one, takes none +# while /proc/pressure/io full avg60 > 20, and releases them when each stage ends. A +# killed job (YGG_KILLED) starts nothing further. +# +# OUTPUT: the YGG-SUMMARY/$YGG_NONCE lines ygg-verify's summary carries; per-container +# logs stay on the host and go with the job. +# FILTER tests= mutants= one per stage-1 file (zero-test first) +# MUTANTS tested=N caught=C missed=M timeout=T unviable=U | stage1 filtered=F +# survivors=S confirmed_caught=X | containers=K | plan=..s stage1=..s/ stage2=..s/ | m +# MISSED / TIMEOUT final outcomes, at most 50 +# ERROR ... a red baseline, a container that failed, mutants with no outcome: +# the job exits non-zero and prints no MUTANTS line +# Exit 0 when every stage ran on a green baseline; survivors do not fail the job (the +# score is not a gate). cargo-mutants exits 2 (missed) and 3 (timeouts) normally; 4 is a +# red baseline; anything else is an error. +# +# DELETION LINE: this file dies with tools/stopgap/ygg-verify.sh when `idunn verify` +# ships a typed mutation step. Modes in this one file: no flag = launcher on the +# workstation; --host = the driver on the Yggdrasil host (ygg-verify.sh RUN_ON_HOST=1); +# --plan / --item = the work inside a container. +set -euo pipefail + +SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" +sum() { printf '%s: %s\n' "$SUMTAG" "$*"; } + +# ---- --escape: mutant names on stdin -> one anchored -F pattern per line ---------------------- +# Only regex metacharacters are escaped, so names holding * + ( ) | stay literal; ^...$ pins the whole name. +if [ "${1:-}" = --escape ]; then + exec perl -pe 'chomp; s/([\\.+*?()|\[\]{}^\$])/\\$1/g; $_ = "^$_\$\n"' +fi + +# ---- launcher (workstation) ------------------------------------------------- +if [ "${1:-}" != --host ] && [ "${1:-}" != --plan ] && [ "${1:-}" != --item ]; then + here=$(cd "$(dirname "$0")" && pwd) + repo=${1:?usage: cargo-mutants.sh [workdir]} + rev=${2:?revision}; base=${3:?base revision}; workdir=${4:-.} + : "${SHARDS:=3}" "${JOBS:=2}" "${MIN_TEST_TIMEOUT:=20}" "${CONFIRM_TIMEOUT_MULT:=5}" "${TIMEOUT:=10800}" + for v in SHARDS JOBS MIN_TEST_TIMEOUT TIMEOUT; do + case ${!v} in ''|*[!0-9]*|0) echo "cargo-mutants: $v must be a positive integer" >&2; exit 2 ;; esac + done + case $CONFIRM_TIMEOUT_MULT in ''|*[!0-9.]*|.*|*.|*.*.*) echo "cargo-mutants: CONFIRM_TIMEOUT_MULT must be a positive number" >&2; exit 2 ;; esac + if [ -n "${FILTERS:-}" ]; then + old=$IFS; IFS=, + for f in $FILTERS; do + case $f in ?*=?*) ;; *) IFS=$old; echo "cargo-mutants: FILTERS entries are =" >&2; exit 2 ;; esac + done + IFS=$old + fi + sha_rev=$(git -C "$repo" rev-parse --verify "$rev^{commit}") || { echo "cargo-mutants: unknown revision" >&2; exit 2; } + sha_base=$(git -C "$repo" rev-parse --verify "$base^{commit}") || { echo "cargo-mutants: unknown base revision" >&2; exit 2; } + git -C "$repo" merge-base --is-ancestor "$sha_base" "$sha_rev" || { echo "cargo-mutants: base is not an ancestor of rev (the verify mirror carries only rev's history)" >&2; exit 2; } + stage=$(mktemp -d) + trap 'rm -rf "$stage"' EXIT + cp "$here/cargo-mutants.sh" "$stage/" + files=(cargo-mutants.sh) + if [ -n "${NAMES:-}" ]; then + [ -r "$NAMES" ] || { echo "cargo-mutants: cannot read the NAMES file" >&2; exit 2; } + tr -d '\r' <"$NAMES" >"$stage/names.txt"; files+=(names.txt) + fi + bundle=$(tar -C "$stage" -cz "${files[@]}" | base64 | tr -d '\n') + # The driver runs on the host: it unpacks the bundle into its own directory (mounted read-only into every container) and removes it when done. + cmd=$(printf 'd=$(mktemp -d) && echo %s | base64 -d | tar -xz -C "$d" && WORKDIR=%q BASE=%q FILTERS=%q TEST_ARGS=%q SHARDS=%q JOBS=%q MIN_TEST_TIMEOUT=%q CONFIRM_TIMEOUT_MULT=%q CM_NAMES=%q CM_DIR="$d" bash "$d/cargo-mutants.sh" --host' \ + "$bundle" "$workdir" "$sha_base" "${FILTERS:-}" "${TEST_ARGS:-}" "$SHARDS" "$JOBS" "$MIN_TEST_TIMEOUT" "$CONFIRM_TIMEOUT_MULT" "${NAMES:+1}") + RUN_ON_HOST=1 TIMEOUT=$TIMEOUT "$here/stopgap/ygg-verify.sh" "$repo" "$sha_rev" rust "$cmd" + exit $? +fi + +# ---- driver (Yggdrasil host, inside the job's slot) -------------------------- +if [ "$1" = --host ]; then + : "${YGG_WORK:?not started by ygg-verify.sh RUN_ON_HOST=1}" "${YGG_IMAGE:?}" "${YGG_CNAME:?}" "${YGG_LABEL:?}" "${YGG_SLOTS:?}" "${YGG_ROOT:?}" + D=${CM_DIR:?} + set +e # every docker step is checked by hand: a container exits non-zero by design + export LC_ALL=C + t0=$(date +%s) + LEASE_FDS=() + unlease() { local lf; for lf in "${LEASE_FDS[@]}"; do exec {lf}>&-; done; LEASE_FDS=(); } + cleanup() { + local p; for p in $(jobs -p); do kill "$p" 2>/dev/null; done + unlease + # containers write as root + [ -z "$D" ] || [ "$D" = / ] || sudo rm -rf -- "$D" + } + trap cleanup EXIT; trap 'exit 143' TERM INT HUP + killed() { [ -n "${YGG_KILLED:-}" ] && [ -e "$YGG_KILLED" ]; } + now() { date +%s; } + # Every container carries the job label (the stopgap's watchdog kills by it) and the same caps. + common=(--label "$YGG_LABEL" "--cpus=${YGG_CPUS:-4}" "--memory=${YGG_MEM:-6g}" + -v "$YGG_WORK:/seed:ro" -v "$D:/cm:ro" -v /etc/machine-id:/etc/machine-id:ro + -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages -v eureka-sccache:/sccache + -e BASH_ENV=/tmp/.ygg-bash-env + -e "WORKDIR=${WORKDIR:-.}" -e "BASE=$BASE" -e "TEST_ARGS=${TEST_ARGS:-}" -e "FILTERS=${FILTERS:-}" -e "CM_NAMES=${CM_NAMES:-}" -e "JOBS=$JOBS" + -e "MIN_TEST_TIMEOUT=$MIN_TEST_TIMEOUT" -e "CONFIRM_TIMEOUT_MULT=$CONFIRM_TIMEOUT_MULT" + -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e 'GIT_CONFIG_VALUE_0=*') + [ -z "${YGG_NONCE:-}" ] || common+=(-e "YGG_NONCE=$YGG_NONCE") + pre="printf 'set -o pipefail\n' >/tmp/.ygg-bash-env" + # drun [-e flags]: one container (the half named by $MODE), its output to /stdout, its + # status in /hostrc. A killed job starts nothing (125). DOCKER_ARGS is a flag string, split on purpose. + # shellcheck disable=SC2086 + drun() { + local name=$1 out=$2 rc; shift 2 + killed && return 125 + mkdir -p "$out" + sudo nice -n 10 docker run --rm --name "$name" "${common[@]}" -v "$out:/out" "$@" -w /src ${DOCKER_ARGS:-} "$YGG_IMAGE" \ + bash -o pipefail -c "$pre; bash /cm/cargo-mutants.sh $MODE" >"$out/stdout" 2>&1 + rc=$?; echo "$rc" >"$out/hostrc"; return "$rc" + } + killed_exit() { sum "JOB KILLED by the watchdog before the run finished; no totals"; exit 124; } + mkdir -p "$D/items" "$D/plan" "$D/q" "$D/m" + + # ---- plan + tp=$(now) + MODE=--plan drun "$YGG_CNAME-plan" "$D/plan" + rc=$? + killed && killed_exit + if [ "$rc" != 0 ]; then + sum "ERROR the plan container exited $rc: the range diff or the mutant listing failed" + tail -n 6 "$D/plan/stdout" 2>/dev/null | sed "s|^|$SUMTAG: ERROR |" + exit 2 + fi + tp=$(( $(now) - tp )) + if [ "${CM_NAMES:-}" = 1 ]; then sort -u "$D/names.txt" >"$D/m/universe" + else sort -u "$D/plan/names.txt" >"$D/m/universe"; fi + for f in s1.caught s1.unviable s1.surv s1.names; do : >"$D/m/$f"; done + + # lease : the containers of a stage = min(SHARDS, SLOTS, units, 1 + free slots now); sets K and holds the leases + lease() { + local want=$1 i lf psi + [ "$want" -le "$SHARDS" ] || want=$SHARDS + [ "$want" -le "$YGG_SLOTS" ] || want=$YGG_SLOTS + K=1; [ "$want" -gt 1 ] || return 0 + psi=$(awk '$1 == "full" { for (i = 2; i <= NF; i++) if ($i ~ /^avg60=/) { sub("avg60=", "", $i); print $i } }' "${CM_PSI:-/proc/pressure/io}" 2>/dev/null) + if awk -v p="${psi:-0}" 'BEGIN { exit !(p > 20) }'; then return 0; fi + for i in $(seq 1 "$YGG_SLOTS"); do + [ "$K" -lt "$want" ] || break + exec {lf}>"$YGG_ROOT/slot-$i.lock" + if flock -n "$lf"; then LEASE_FDS+=("$lf"); K=$((K + 1)); else exec {lf}>&-; fi + done + } + # run_stage : K workers claim items 1..count by mkdir, in order (largest first) + run_stage() { + local stage=$1 count=$2 fn=$3 w pids=() + for w in $(seq 1 "$K"); do + ( for lf in "${LEASE_FDS[@]}"; do exec {lf}>&-; done # the lease belongs to the driver, never to a child (2026-09-23 inherited-fd scar) + i=0 + while [ "$i" -lt "$count" ]; do + i=$((i + 1)) + killed && break + [ ! -e "$D/failed" ] || break + mkdir "$D/q/$stage-$i" 2>/dev/null || continue + "$fn" "$i" + done ) & + pids+=($!) + done + wait "${pids[@]}" + } + container_error() { # container_error