tools/cargo-mutants.sh: one owner of cargo-mutants on Yggdrasil
One verify job per range: a plan container lists the mutants (diff taken at the workspace root), stage 1 runs each file's mutants against that file's tests only and decides only caught and unviable, stage 2 confirms every survivor, timeout, unfiltered or zero-test file on the full suite, sharded over free slot leases, and the merge prints one MUTANTS line and the final MISSED/TIMEOUT names. NAMES reruns chosen survivors; TEST_ARGS skips flaky tests. Deletes ygg-verify.sh's MISSED/TIMEOUT summary grep and its inline cargo mutants usage example; the host-mode export gains YGG_SLOTS and YGG_ROOT. Briefs and SKILL.md name the script; the mock docker grows /cm and /out, with a fake cargo and a fake flock for the new test-tools.sh cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
d2d64d0d19
commit
d4fba11da6
15 changed files with 703 additions and 25 deletions
7
SKILL.md
7
SKILL.md
|
|
@ -317,7 +317,9 @@ nothing is pasted or paraphrased. The brief says:
|
|||
- **Measure the suite with the ecosystem's mutation tool, scoped to the cut's
|
||||
diff:** Stryker.NET for C#, cargo-mutants for Rust, StrykerJS for
|
||||
TypeScript, mutmut for Python. The tool generates the mutants and nobody
|
||||
writes an anchor. Every survivor is a finding about the tests, triaged by
|
||||
writes an anchor. cargo-mutants runs through `tools/cargo-mutants.sh` (per-file
|
||||
scoped tests, survivors confirmed against the full suite, sharded over
|
||||
Yggdrasil slots), once per Hands pass. Every survivor is a finding about the tests, triaged by
|
||||
name: a behavioural test is missing, a fixture is degenerate (see below),
|
||||
or the mutant is equivalent and gets a one-line reason. Boundary flips on
|
||||
float thresholds are equivalent by default and are not chased. The score is
|
||||
|
|
@ -622,7 +624,8 @@ exact script they want and agents do not browse. Contributions come by branch
|
|||
and Forgejo pull request (this repo lives on Yggdrasil's Forgejo), never a push
|
||||
to main. A script that verdicts depend on gets a Soul pass before merge, and
|
||||
`bash tools/test-tools.sh` passes first. The summary and wait modes of
|
||||
`tools/stopgap/ygg-verify.sh` and `tools/mutation-table.sh` are such scripts. A script whose brief no longer names it is pruned.
|
||||
`tools/stopgap/ygg-verify.sh`, `tools/mutation-table.sh` and
|
||||
`tools/cargo-mutants.sh` are such scripts. A script whose brief no longer names it is pruned.
|
||||
Scripts that know a project's layout stay in that project.
|
||||
|
||||
## Git and tooling rules (scars)
|
||||
|
|
|
|||
|
|
@ -256,7 +256,12 @@ cites, read by id):
|
|||
Every rule the spec or the operator names gets a behavioural test. Measure the
|
||||
suite with <the ecosystem's mutation tool>, scoped to this cut's diff
|
||||
(`--since:<base>` or the tool's equivalent), against the final spelling of the
|
||||
code. The report's mutations (at most 64) holds every survivor and the kills
|
||||
code. For Rust the tool is `tools/cargo-mutants.sh <repo> <head> <base> [workdir]`,
|
||||
run ONCE per pass, at the end: it scopes each file's mutants to that file's tests,
|
||||
confirms every survivor on the full suite and shards over free slots. After killing
|
||||
survivors with tests, rerun only those names (`NAMES=<file of the MISSED names>`),
|
||||
never a second whole-diff sweep; a flaky test is skipped with
|
||||
`TEST_ARGS='--skip <name>'`. The report's mutations (at most 64) holds every survivor and the kills
|
||||
Soul should rerun; put the totals (generated, caught, unviable, missed) in
|
||||
verification evidence. Triage every survivor
|
||||
by name and line: killed by a new test, killed by fixing a degenerate fixture,
|
||||
|
|
@ -428,8 +433,9 @@ Verification: <exact builds, tests, captures, negative greps>.
|
|||
Python suites locally on 2026-10-12, red for Windows-only reasons).
|
||||
- Verify through `tools/stopgap/ygg-verify.sh` with `SUMMARY=<log>` (above):
|
||||
read the summary, and open the full log only at a named failing test, with
|
||||
`grep -n -A 40`. Hand mutants go in one list to `tools/mutation-table.sh`,
|
||||
one job for all of them.
|
||||
`grep -n -A 40`. Rust mutants go to `tools/cargo-mutants.sh` (one job, scoped,
|
||||
confirmed, sharded), hand mutants in one list to `tools/mutation-table.sh`; never
|
||||
an inline `cargo mutants` command.
|
||||
- Stryker `--mutate` ranges are character offsets, not lines, into the file as
|
||||
Stryker reads it: the bytes of the checkout the job runs in. Line endings there
|
||||
follow `.gitattributes`, not the platform: CultLib's `packages/cultmath/src/**`
|
||||
|
|
@ -539,9 +545,11 @@ specifically:
|
|||
binaries or files the cut touches and the ones that name its rules. Run the
|
||||
full suite exactly once, at the end. Put that run in the same job as your
|
||||
mutation run where you can, since that job pays for the build anyway. Batch
|
||||
probes into one job script rather than one job each. Run the mutation tool
|
||||
in parallel (`--jobs 3`), and skip by name any test that a follow-up records
|
||||
as flaky, so that its flakes cannot count a surviving mutant as killed.
|
||||
probes into one job script rather than one job each. Run Rust mutants through
|
||||
`tools/cargo-mutants.sh` (it shards; do not add `--jobs`), skipping any test a
|
||||
follow-up records as flaky with `TEST_ARGS='--skip <name>'`; other ecosystems
|
||||
run their tool in parallel (`--jobs 3`) and skip by name, so that a flake cannot
|
||||
count a surviving mutant as killed.
|
||||
Dump the range once (`git diff <base>..<head> > <scratchpad>/<agent>/range.diff` plus
|
||||
`--stat`) and grep that file instead of re-running `git show` per file. Hand
|
||||
mutants go in one list to `tools/mutation-table.sh`, never one job each.
|
||||
|
|
|
|||
|
|
@ -11,8 +11,9 @@ gets a Soul pass before merge.
|
|||
| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=<log>` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait <log>` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) |
|
||||
| `tools/wait-on.sh` | One general waiter for anything that is not a Yggdrasil job: `pid <pid> [<log>]`, `log <log> <done-regex>` (with `WAIT_PID=<pid>` a dead process with no match is exit 98; an invalid ERE is exit 3), `file <path>`. The `pid` form prints `GONE (no completion evidence)`: prefer `log` whenever the job writes a done line. Same contract as `ygg-verify.sh wait`: blocks at most 270 s, exit 0 done with a tail of at most 60 log lines, `RUNNING <n>s` exit 99, exit 98 gone with no completion evidence; call it with a Bash tool `timeout` of at least 280000 ms. Replaces hand-rolled poll loops and idle `true`/`echo`/`sleep` calls. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs, Soul long jobs, detached scripts) |
|
||||
| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL <file> [ <file> ]`, jest `FAIL <file>` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) |
|
||||
| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them |
|
||||
| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) |
|
||||
| `tools/cargo-mutants.sh` | `SUMMARY=<log> tools/cargo-mutants.sh <repo> <rev> <base> [workdir]`: cargo-mutants on a revision range as ONE verify job, the one owner of running it. Plan container lists the range's mutants (diff taken at the workspace root); stage 1 runs each file's mutants against that file's tests only (filter derived from the path, `FILTERS` overrides), deciding only CAUGHT and UNVIABLE; stage 2 confirms every survivor, timeout, unfiltered file and zero-test file on the full suite, sharded; merge counts each name once and prints `MUTANTS` totals, `FILTER` lines and the final `MISSED`/`TIMEOUT` names. Containers = min(`SHARDS`, slots, units, 1 + free slots), leased with `flock -n` from Yggdrasil's slot pool (none while IO pressure full avg60 > 20). `NAMES=<file>` reruns chosen survivors (no stage 1), `TEST_ARGS` skips flaky tests, `JOBS`, `MIN_TEST_TIMEOUT`, `CONFIRM_TIMEOUT_MULT`, `TIMEOUT`. Exit 0 on a green baseline whatever survives; a red baseline or a failed container is non-zero with no totals. Run once per pass. | Hands, Soul | `references/briefs.md` (Hands verification, Soul reruns), `SKILL.md` (mutation bullet) |
|
||||
| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | `tools/stopgap/ygg-verify.sh`, `tools/mutation-table.sh` | not named by a brief; it lives and dies with them |
|
||||
| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runners' host drivers and container halves (`tools/mutation-table.sh`, `tools/cargo-mutants.sh`) under a mock docker, a fake `cargo mutants` and real slot locks. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) |
|
||||
| `tools/context-pack.py` | `py -3 tools/context-pack.py <cut_spec id> <worktree> [--since <sha>] [--only <k>\|r<k>[,...]] [--spec-json <file>] [--max-lines <n>]`. Carries a cut_spec's `file_changes` and `reads` anchors (read rows `r<k>`; a read absent at base is MISSING) from the spec's base to the worktree by `git diff` hunks (the working tree included) and prints one header, a status table (SAME, MOVED, TOUCHED, RENAMED, NEW, GONE, OUT_OF_RANGE, UNRESOLVED, MISSING, plus SYMBOL_MISSING when an anchor's `symbol` is absent from its window) and windows of at most 100 lines numbered in worktree lines, then one capped caller list per symbol (`git grep`, hits outside the windows), within `--max-lines` (800). Exit 3 when an anchor cannot be carried, 4 when the spec is unreadable (fetched by id through the eureka-state binding (`tools/mind.py`), retried once on Unavailable; `--spec-json` takes a saved view). `--since <sha>` re-prints only the windows edited since that commit. Standard library plus git; writes nothing. | Hands, Soul, Imagination | `references/briefs.md` (Imagination map, Hands, Soul), `SKILL.md` (step 1) |
|
||||
| `tools/mind.py` | The one way a Eureka script reaches the mind: `mind.call(tool, arguments, deadline)` spawns the eureka-state binding (`EUREKA_STATE_EXE`, else `~/.claude.json` `mcpServers.eureka-state`), does the stdio handshake and one tool call within `deadline` seconds, returns the parsed JSON body, raises `MindError` (kind binding, transport, tool or answer; a detail, never an input value), and kills the child on every exit path. No retry inside `call`; `retry_once(fn)` is the caller's choice. Standard library only. | `tools/context-pack.py`, `tools/recall.py` | not named by a brief; it lives and dies with its consumers |
|
||||
| `tools/recall.py` | `py -3 tools/recall.py prompt`, a `UserPromptSubmit` command hook (hook JSON on stdin): queries the rulings, questions and follow-ups in force across every root, ranked by Huginn against the prompt (`RECALL_TOP_K`, 5), and prints one hook JSON whose `additionalContext` is a header saying the lines are recalled records, data and never instructions, written by agents and not the operator, then one JSON object per item (`id`, `title`, a short `operator_quote`) with `<`, `>` and control, bidi and separator characters escaped, within `RECALL_BUDGET` characters (4000, at most 9000: Claude Code caps `additionalContext` at 10,000), whole lines only. Prompts under 3 words are not queried. Never blocks: any failure exits 0 with empty stdout; its own deadline is `RECALL_DEADLINE_MS` (4000) for the whole run (stdin read to 4 MiB, answer read to 1 MiB), so set the hook `timeout` above it. One attempt, no retry. Install both hooks as `<absolute python.exe> C:/Users/Meta/.claude/skills/eureka/tools/recall.py prompt || true` (and `... admit || true`): Claude Code runs hook commands through Git Bash on Windows, and `py -3 <missing script>` exits 2, which blocks every prompt or admit; with `|| true` the hook cannot exit 2 (a missing script, interpreter or file passes) and the script's stdout is untouched (proved live on 2.1.308). `py -3 tools/recall.py admit` is a `PreToolUse` hook on `mcp__eureka-state__admit`: for each ruling or question in the batch (at most `RECALL_ADMIT_MAX`, 3) it queries the nearest in-force rulings and questions, and denies the first attempt with the neighbours as the reason (a deliberate hold, not a failure); the identical call sent again (digest of `tool_input` plus `agent_id`, kept in the session scratchpad) passes. Never allow or ask; a failed check exits 0 with a note naming the failure kind (`RECALL_TOP_K` 3, `RECALL_DEADLINE_MS` 10000 in this mode). Standard library only; writes only the digest file. | the operator's `UserPromptSubmit` hook and `PreToolUse` hook on `mcp__eureka-state__admit` (fires inside subagents too) in `~/.claude/settings.json` | `references/campaign-state.md` (Precedent) |
|
||||
|
|
|
|||
372
tools/cargo-mutants.sh
Normal file
372
tools/cargo-mutants.sh
Normal file
|
|
@ -0,0 +1,372 @@
|
|||
#!/usr/bin/env bash
|
||||
# cargo-mutants on Yggdrasil for a revision range: ONE verify job, per-file scoped
|
||||
# tests, survivors confirmed on the full suite, sharded over free slots, one merged
|
||||
# survivor list. The one owner of "run cargo-mutants" (no brief, no job script runs
|
||||
# an inline `cargo mutants`).
|
||||
#
|
||||
# SUMMARY=<log> tools/cargo-mutants.sh <local-repo> <rev> <base> [workdir]
|
||||
# workdir directory inside the repo where cargo runs (a package or workspace
|
||||
# member, e.g. crates/mind-body; CultLib has no Cargo.toml at its root).
|
||||
# The mutated range is `git diff --relative <base> <rev>` taken at the
|
||||
# WORKSPACE ROOT: cargo-mutants matches --in-diff paths, -f paths and
|
||||
# prints mutant names relative to the workspace root, not to workdir.
|
||||
# env FILTERS '<path>=<filter>,...' overrides a file's derived test filter
|
||||
# (path as the mutant names print it); '-' = full suite
|
||||
# TEST_ARGS harness args, e.g. '--skip flaky_one' (a flaky test would
|
||||
# make a surviving mutant read as killed in one run)
|
||||
# NAMES local file of mutant names (one per line, copied from a
|
||||
# previous MISSED/TIMEOUT list): skips stage 1 and the listing
|
||||
# and runs only those names on the full suite. This is the rerun
|
||||
# after killing survivors with tests.
|
||||
# SHARDS (3) JOBS (2) MIN_TEST_TIMEOUT (20) CONFIRM_TIMEOUT_MULT (2)
|
||||
# TIMEOUT seconds for the whole job (default 10800)
|
||||
# everything else tools/stopgap/ygg-verify.sh reads (MEM, CPUS, ...)
|
||||
# Typical, from an agent: launch as ONE background shell with a fresh log, then
|
||||
# tools/stopgap/ygg-verify.sh wait <log> (Bash tool timeout >= 280000 ms)
|
||||
# Run it ONCE per pass, at the end, on the final spelling of the code. Chasing
|
||||
# survivors is a NAMES rerun, never a second whole-range sweep.
|
||||
#
|
||||
# STAGES (all inside the one job and the one host driver):
|
||||
# plan one container writes the range diff and lists every mutant in it.
|
||||
# stage 1 each touched file with a test filter runs its mutants against that
|
||||
# file's tests only (`cargo mutants -f <file> -- <filter>`). The filter is
|
||||
# derived from the path: the module path after the last src/ (a/mod.rs ->
|
||||
# `a::`, a/b.rs -> `a::b::`); lib.rs, main.rs, bin/ and files outside src/
|
||||
# have none. FILTERS overrides. Stage 1 decides only CAUGHT (its own green
|
||||
# filtered baseline guards false kills) and UNVIABLE (a build fact). Its
|
||||
# MISSED and TIMEOUT are never reported: a mutant that only another module's
|
||||
# tests or tests/ kill survives a filtered run. A filter whose baseline ran
|
||||
# 0 tests settles nothing and all its file's mutants go to stage 2.
|
||||
# stage 2 every name stage 1 did not settle (survivors, files without a filter,
|
||||
# zero-test files, or NAMES) runs on the full suite in cargo-mutants' own
|
||||
# baseline, sharded over the containers, with the test timeout at
|
||||
# CONFIRM_TIMEOUT_MULT x that baseline. Its outcome is final.
|
||||
# merge each name counted once; MISSED and TIMEOUT are stage-2 outcomes only.
|
||||
# Containers per stage: min(SHARDS, SLOTS, units, 1 + free slots now), units being the
|
||||
# stage's files (stage 1) or names (stage 2). The job holds one slot (ygg-verify's);
|
||||
# the driver leases further slot locks with flock -n, never waits for one, takes none
|
||||
# while /proc/pressure/io full avg60 > 20, and releases them when each stage ends. A
|
||||
# killed job (YGG_KILLED) starts nothing further.
|
||||
#
|
||||
# OUTPUT: the YGG-SUMMARY/$YGG_NONCE lines ygg-verify's summary carries; per-container
|
||||
# logs stay on the host and go with the job.
|
||||
# FILTER <file> <filter> tests=<n> mutants=<m> one per stage-1 file (zero-test first)
|
||||
# MUTANTS tested=N caught=C missed=M timeout=T unviable=U | stage1 filtered=F
|
||||
# survivors=S confirmed_caught=X | containers=K | plan=..s stage1=..s stage2=..s | <m>m
|
||||
# MISSED <name> / TIMEOUT <name> final outcomes, at most 50
|
||||
# ERROR ... a red baseline, a container that failed, mutants with no outcome:
|
||||
# the job exits non-zero and prints no MUTANTS line
|
||||
# Exit 0 when every stage ran on a green baseline; survivors do not fail the job (the
|
||||
# score is not a gate). cargo-mutants exits 2 (missed) and 3 (timeouts) normally; 4 is a
|
||||
# red baseline; anything else is an error.
|
||||
#
|
||||
# DELETION LINE: this file dies with tools/stopgap/ygg-verify.sh when `idunn verify`
|
||||
# ships a typed mutation step. Modes in this one file: no flag = launcher on the
|
||||
# workstation; --host = the driver on the Yggdrasil host (ygg-verify.sh RUN_ON_HOST=1);
|
||||
# --plan / --item = the work inside a container.
|
||||
set -euo pipefail
|
||||
|
||||
SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}"
|
||||
sum() { printf '%s: %s\n' "$SUMTAG" "$*"; }
|
||||
|
||||
# ---- launcher (workstation) -------------------------------------------------
|
||||
if [ "${1:-}" != --host ] && [ "${1:-}" != --plan ] && [ "${1:-}" != --item ]; then
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
repo=${1:?usage: cargo-mutants.sh <local-repo> <rev> <base> [workdir]}
|
||||
rev=${2:?revision}; base=${3:?base revision}; workdir=${4:-.}
|
||||
: "${SHARDS:=3}" "${JOBS:=2}" "${MIN_TEST_TIMEOUT:=20}" "${CONFIRM_TIMEOUT_MULT:=2}" "${TIMEOUT:=10800}"
|
||||
for v in SHARDS JOBS MIN_TEST_TIMEOUT TIMEOUT; do
|
||||
case ${!v} in ''|*[!0-9]*|0) echo "cargo-mutants: $v must be a positive integer" >&2; exit 2 ;; esac
|
||||
done
|
||||
case $CONFIRM_TIMEOUT_MULT in ''|*[!0-9.]*|.*|*.|*.*.*) echo "cargo-mutants: CONFIRM_TIMEOUT_MULT must be a positive number" >&2; exit 2 ;; esac
|
||||
if [ -n "${FILTERS:-}" ]; then
|
||||
old=$IFS; IFS=,
|
||||
for f in $FILTERS; do
|
||||
case $f in ?*=?*) ;; *) IFS=$old; echo "cargo-mutants: FILTERS entries are <path>=<filter>" >&2; exit 2 ;; esac
|
||||
done
|
||||
IFS=$old
|
||||
fi
|
||||
sha_rev=$(git -C "$repo" rev-parse --verify "$rev^{commit}") || { echo "cargo-mutants: unknown revision" >&2; exit 2; }
|
||||
sha_base=$(git -C "$repo" rev-parse --verify "$base^{commit}") || { echo "cargo-mutants: unknown base revision" >&2; exit 2; }
|
||||
git -C "$repo" merge-base --is-ancestor "$sha_base" "$sha_rev" || { echo "cargo-mutants: base is not an ancestor of rev (the verify mirror carries only rev's history)" >&2; exit 2; }
|
||||
stage=$(mktemp -d)
|
||||
trap 'rm -rf "$stage"' EXIT
|
||||
cp "$here/cargo-mutants.sh" "$stage/"
|
||||
files=(cargo-mutants.sh)
|
||||
if [ -n "${NAMES:-}" ]; then
|
||||
[ -r "$NAMES" ] || { echo "cargo-mutants: cannot read the NAMES file" >&2; exit 2; }
|
||||
tr -d '\r' <"$NAMES" >"$stage/names.txt"; files+=(names.txt)
|
||||
fi
|
||||
bundle=$(tar -C "$stage" -cz "${files[@]}" | base64 | tr -d '\n')
|
||||
# The driver runs on the host: it unpacks the bundle into its own directory (mounted read-only into every container) and removes it when done.
|
||||
cmd=$(printf 'd=$(mktemp -d) && echo %s | base64 -d | tar -xz -C "$d" && WORKDIR=%q BASE=%q FILTERS=%q TEST_ARGS=%q SHARDS=%q JOBS=%q MIN_TEST_TIMEOUT=%q CONFIRM_TIMEOUT_MULT=%q CM_NAMES=%q CM_DIR="$d" bash "$d/cargo-mutants.sh" --host' \
|
||||
"$bundle" "$workdir" "$sha_base" "${FILTERS:-}" "${TEST_ARGS:-}" "$SHARDS" "$JOBS" "$MIN_TEST_TIMEOUT" "$CONFIRM_TIMEOUT_MULT" "${NAMES:+1}")
|
||||
RUN_ON_HOST=1 TIMEOUT=$TIMEOUT "$here/stopgap/ygg-verify.sh" "$repo" "$sha_rev" rust "$cmd"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
# ---- driver (Yggdrasil host, inside the job's slot) --------------------------
|
||||
if [ "$1" = --host ]; then
|
||||
: "${YGG_WORK:?not started by ygg-verify.sh RUN_ON_HOST=1}" "${YGG_IMAGE:?}" "${YGG_CNAME:?}" "${YGG_LABEL:?}" "${YGG_SLOTS:?}" "${YGG_ROOT:?}"
|
||||
D=${CM_DIR:?}
|
||||
set +e # every docker step is checked by hand: a container exits non-zero by design
|
||||
export LC_ALL=C
|
||||
t0=$(date +%s)
|
||||
LEASE_FDS=()
|
||||
unlease() { local lf; for lf in "${LEASE_FDS[@]}"; do exec {lf}>&-; done; LEASE_FDS=(); }
|
||||
cleanup() {
|
||||
local p; for p in $(jobs -p); do kill "$p" 2>/dev/null; done
|
||||
unlease
|
||||
# containers write as root
|
||||
[ -z "$D" ] || [ "$D" = / ] || sudo rm -rf -- "$D"
|
||||
}
|
||||
trap cleanup EXIT; trap 'exit 143' TERM INT HUP
|
||||
killed() { [ -n "${YGG_KILLED:-}" ] && [ -e "$YGG_KILLED" ]; }
|
||||
now() { date +%s; }
|
||||
# Every container carries the job label (the stopgap's watchdog kills by it) and the same caps.
|
||||
common=(--label "$YGG_LABEL" "--cpus=${YGG_CPUS:-4}" "--memory=${YGG_MEM:-6g}"
|
||||
-v "$YGG_WORK:/seed:ro" -v "$D:/cm:ro" -v /etc/machine-id:/etc/machine-id:ro
|
||||
-v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages -v eureka-sccache:/sccache
|
||||
-e BASH_ENV=/tmp/.ygg-bash-env
|
||||
-e "WORKDIR=${WORKDIR:-.}" -e "BASE=$BASE" -e "TEST_ARGS=${TEST_ARGS:-}" -e "JOBS=$JOBS"
|
||||
-e "MIN_TEST_TIMEOUT=$MIN_TEST_TIMEOUT" -e "CONFIRM_TIMEOUT_MULT=$CONFIRM_TIMEOUT_MULT"
|
||||
-e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e 'GIT_CONFIG_VALUE_0=*')
|
||||
[ -z "${YGG_NONCE:-}" ] || common+=(-e "YGG_NONCE=$YGG_NONCE")
|
||||
pre="printf 'set -o pipefail\n' >/tmp/.ygg-bash-env"
|
||||
# drun <name> <out dir> [-e flags]: one container (the half named by $MODE), its output to <out>/stdout, its
|
||||
# status in <out>/hostrc. A killed job starts nothing (125). DOCKER_ARGS is a flag string, split on purpose.
|
||||
# shellcheck disable=SC2086
|
||||
drun() {
|
||||
local name=$1 out=$2 rc; shift 2
|
||||
killed && return 125
|
||||
mkdir -p "$out"
|
||||
sudo nice -n 10 docker run --rm --name "$name" "${common[@]}" -v "$out:/out" "$@" -w /src ${DOCKER_ARGS:-} "$YGG_IMAGE" \
|
||||
bash -o pipefail -c "$pre; bash /cm/cargo-mutants.sh $MODE" >"$out/stdout" 2>&1
|
||||
rc=$?; echo "$rc" >"$out/hostrc"; return "$rc"
|
||||
}
|
||||
killed_exit() { sum "JOB KILLED by the watchdog before the run finished; no totals"; exit 124; }
|
||||
mkdir -p "$D/items" "$D/plan" "$D/q" "$D/m"
|
||||
|
||||
# ---- plan
|
||||
tp=$(now)
|
||||
MODE=--plan drun "$YGG_CNAME-plan" "$D/plan"
|
||||
rc=$?
|
||||
killed && killed_exit
|
||||
if [ "$rc" != 0 ]; then
|
||||
sum "ERROR the plan container exited $rc: the range diff or the mutant listing failed"
|
||||
tail -n 6 "$D/plan/stdout" 2>/dev/null | sed "s|^|$SUMTAG: ERROR |"
|
||||
exit 2
|
||||
fi
|
||||
tp=$(( $(now) - tp ))
|
||||
if [ "${CM_NAMES:-}" = 1 ]; then sort -u "$D/names.txt" >"$D/m/universe"
|
||||
else sort -u "$D/plan/names.txt" >"$D/m/universe"; fi
|
||||
for f in s1.caught s1.unviable s1.surv s1.names s1.tests; do : >"$D/m/$f"; done
|
||||
|
||||
# lease <units>: the containers of a stage = min(SHARDS, SLOTS, units, 1 + free slots now); sets K and holds the leases
|
||||
lease() {
|
||||
local want=$1 i lf psi
|
||||
[ "$want" -le "$SHARDS" ] || want=$SHARDS
|
||||
[ "$want" -le "$YGG_SLOTS" ] || want=$YGG_SLOTS
|
||||
K=1; [ "$want" -gt 1 ] || return 0
|
||||
psi=$(awk '$1 == "full" { for (i = 2; i <= NF; i++) if ($i ~ /^avg60=/) { sub("avg60=", "", $i); print $i } }' "${CM_PSI:-/proc/pressure/io}" 2>/dev/null)
|
||||
if awk -v p="${psi:-0}" 'BEGIN { exit !(p > 20) }'; then return 0; fi
|
||||
for i in $(seq 1 "$YGG_SLOTS"); do
|
||||
[ "$K" -lt "$want" ] || break
|
||||
exec {lf}>"$YGG_ROOT/slot-$i.lock"
|
||||
if flock -n "$lf"; then LEASE_FDS+=("$lf"); K=$((K + 1)); else exec {lf}>&-; fi
|
||||
done
|
||||
}
|
||||
# run_stage <stage> <count> <fn>: K workers claim items 1..count by mkdir, in order (largest first)
|
||||
run_stage() {
|
||||
local stage=$1 count=$2 fn=$3 w pids=()
|
||||
for w in $(seq 1 "$K"); do
|
||||
( for lf in "${LEASE_FDS[@]}"; do exec {lf}>&-; done # the lease belongs to the driver, never to a child (2026-09-23 inherited-fd scar)
|
||||
i=0
|
||||
while [ "$i" -lt "$count" ]; do
|
||||
i=$((i + 1))
|
||||
killed && break
|
||||
[ ! -e "$D/failed" ] || break
|
||||
mkdir "$D/q/$stage-$i" 2>/dev/null || continue
|
||||
"$fn" "$i"
|
||||
done ) &
|
||||
pids+=($!)
|
||||
done
|
||||
wait "${pids[@]}"
|
||||
}
|
||||
container_error() { # container_error <label> <out dir>
|
||||
local rc; rc=$(cat "$2/hostrc" 2>/dev/null)
|
||||
sum "ERROR $1 container exited ${rc:-?} (4 is a red baseline: the unmutated tests failed there)"
|
||||
tail -n 5 "$2/stdout" 2>/dev/null | sed "s|^|$SUMTAG: ERROR |"
|
||||
}
|
||||
|
||||
# ---- stage 1: files with a filter
|
||||
declare -A fover
|
||||
if [ -n "${FILTERS:-}" ]; then
|
||||
IFS=, read -ra fl <<<"$FILTERS"
|
||||
for e in "${fl[@]}"; do fover[${e%%=*}]=${e#*=}; done
|
||||
fi
|
||||
derive_filter() { # derive_filter <path>: the test filter of a file, nothing for the full suite
|
||||
local p=$1 rest
|
||||
if [ -n "${fover[$p]+x}" ]; then [ "${fover[$p]}" = - ] || echo "${fover[$p]}"; return 0; fi
|
||||
case $p in */src/*.rs|src/*.rs) ;; *) return 0 ;; esac
|
||||
rest=${p##*src/}; rest=${rest%.rs}
|
||||
case $rest in lib|main|bin/*|*/bin/*) return 0 ;; esac
|
||||
rest=${rest%/mod}
|
||||
echo "${rest//\//::}::"
|
||||
}
|
||||
S1_FILE=(); S1_FILTER=(); s1n=0
|
||||
if [ "${CM_NAMES:-}" != 1 ]; then
|
||||
# files by mutant count, largest first
|
||||
while read -r cnt file; do
|
||||
f=$(derive_filter "$file")
|
||||
[ -n "$f" ] || continue
|
||||
s1n=$((s1n + 1)); S1_FILE[$s1n]=$file; S1_FILTER[$s1n]=$f
|
||||
awk -F: -v f="$file" '$1 == f' "$D/m/universe" >"$D/items/s1-$s1n.names"
|
||||
echo "$cnt" >"$D/items/s1-$s1n.count"
|
||||
done < <(awk -F: '{ c[$1]++ } END { for (f in c) print c[f], f }' "$D/m/universe" | sort -k1,1nr -k2)
|
||||
fi
|
||||
s1_item() {
|
||||
local n=$1 rc
|
||||
MODE=--item drun "$YGG_CNAME-s1-$n" "$D/items/s1-$n/out" -e CM_STAGE=1 -e "CM_FILE=${S1_FILE[$n]}" -e "CM_FILTER=${S1_FILTER[$n]}"
|
||||
rc=$?
|
||||
if [ "$rc" != 0 ] && ! killed; then touch "$D/failed"; fi
|
||||
}
|
||||
K1=0; ts1=0
|
||||
if [ "$s1n" -gt 0 ]; then
|
||||
ts1=$(now)
|
||||
lease "$s1n"; K1=$K
|
||||
run_stage s1 "$s1n" s1_item
|
||||
unlease
|
||||
killed && killed_exit
|
||||
ts1=$(( $(now) - ts1 ))
|
||||
bad=0
|
||||
for n in $(seq 1 "$s1n"); do
|
||||
o="$D/items/s1-$n/out"
|
||||
if [ "$(cat "$o/hostrc" 2>/dev/null)" != 0 ]; then
|
||||
[ -e "$D/q/s1-$n" ] || continue # never started: the job is already red for another file
|
||||
container_error "stage 1 (${S1_FILE[$n]})" "$o"; bad=1; continue
|
||||
fi
|
||||
echo "$n $(cat "$o/tests" 2>/dev/null || echo 0)" >>"$D/m/s1.tests"
|
||||
done
|
||||
if [ "$bad" != 0 ] || [ -e "$D/failed" ]; then exit 1; fi
|
||||
fi
|
||||
# FILTER lines: the anomalies (0 tests) first, then the rest, at most 12 in all
|
||||
shown=0; hidden=0
|
||||
for pass in zero rest; do
|
||||
for n in $(seq 1 "$s1n"); do
|
||||
t=$(awk -v n="$n" '$1 == n { print $2 }' "$D/m/s1.tests"); t=${t:-0}
|
||||
if [ "$pass" = zero ] && [ "$t" != 0 ]; then continue; fi
|
||||
if [ "$pass" = rest ] && [ "$t" = 0 ]; then continue; fi
|
||||
if [ "$shown" -lt 12 ]; then sum "FILTER ${S1_FILE[$n]} ${S1_FILTER[$n]} tests=$t mutants=$(cat "$D/items/s1-$n.count")"; shown=$((shown + 1))
|
||||
else hidden=$((hidden + 1)); fi
|
||||
done
|
||||
done
|
||||
[ "$hidden" = 0 ] || sum "FILTER ... $hidden more files"
|
||||
for n in $(seq 1 "$s1n"); do
|
||||
t=$(awk -v n="$n" '$1 == n { print $2 }' "$D/m/s1.tests"); t=${t:-0}
|
||||
[ "$t" != 0 ] || continue # a filter that ran no test settled nothing
|
||||
o="$D/items/s1-$n/out"
|
||||
cat "$D/items/s1-$n.names" >>"$D/m/s1.names"
|
||||
cat "$o/caught.txt" >>"$D/m/s1.caught"; cat "$o/unviable.txt" >>"$D/m/s1.unviable"
|
||||
cat "$o/missed.txt" "$o/timeout.txt" >>"$D/m/s1.surv"
|
||||
done
|
||||
# only names of this run's range count; stage 1 settles caught and unviable, nothing else
|
||||
for f in s1.caught s1.unviable s1.surv s1.names; do
|
||||
sort -u "$D/m/$f" | comm -12 - "$D/m/universe" >"$D/m/$f.t" && mv "$D/m/$f.t" "$D/m/$f"
|
||||
done
|
||||
cat "$D/m/s1.caught" "$D/m/s1.unviable" | sort -u >"$D/m/s1.settled"
|
||||
comm -23 "$D/m/universe" "$D/m/s1.settled" >"$D/s2.txt"
|
||||
# only regex metacharacters are escaped, so names holding * + ( ) | stay literal; ^...$ pins the whole name
|
||||
perl -pe 'chomp; s/([\\.+*?()|\[\]{}^\$])/\\$1/g; $_ = "^$_\$\n"' "$D/s2.txt" >"$D/s2.re"
|
||||
|
||||
# ---- stage 2: the full suite on everything stage 1 did not settle
|
||||
s2n=$(wc -l <"$D/s2.txt" | tr -d ' ')
|
||||
K2=0; ts2=0
|
||||
for l in caught missed timeout unviable; do : >"$D/m/s2.$l"; done
|
||||
s2_shard() {
|
||||
local k=$1 rc
|
||||
MODE=--item drun "$YGG_CNAME-s2-$k" "$D/items/s2-$k/out" -e CM_STAGE=2 -e "CM_SHARD=$((k - 1))/$K2"
|
||||
rc=$?
|
||||
if [ "$rc" != 0 ] && ! killed; then touch "$D/failed"; fi
|
||||
}
|
||||
if [ "$s2n" -gt 0 ]; then
|
||||
ts2=$(now)
|
||||
lease "$s2n"; K2=$K
|
||||
run_stage s2 "$K2" s2_shard
|
||||
unlease
|
||||
killed && killed_exit
|
||||
ts2=$(( $(now) - ts2 ))
|
||||
bad=0
|
||||
for k in $(seq 1 "$K2"); do
|
||||
o="$D/items/s2-$k/out"
|
||||
if [ "$(cat "$o/hostrc" 2>/dev/null)" != 0 ]; then
|
||||
[ -e "$D/q/s2-$k" ] || continue
|
||||
container_error "stage 2 (shard $k of $K2)" "$o"; bad=1; continue
|
||||
fi
|
||||
for l in caught missed timeout unviable; do cat "$o/$l.txt" >>"$D/m/s2.$l"; done
|
||||
done
|
||||
if [ "$bad" != 0 ] || [ -e "$D/failed" ]; then exit 1; fi
|
||||
fi
|
||||
|
||||
# ---- merge: a name's outcome is stage 2's where it ran, else stage 1's; each name once
|
||||
for l in caught missed timeout unviable; do
|
||||
sort -u "$D/m/s2.$l" | comm -12 - "$D/s2.txt" >"$D/m/s2.$l.t" && mv "$D/m/s2.$l.t" "$D/m/s2.$l"
|
||||
done
|
||||
cat "$D/m/s2.caught" "$D/m/s2.missed" "$D/m/s2.timeout" "$D/m/s2.unviable" | sort -u >"$D/m/s2.all"
|
||||
nout=$(comm -23 "$D/s2.txt" "$D/m/s2.all" | wc -l | tr -d ' ')
|
||||
if [ "$nout" != 0 ]; then
|
||||
sum "ERROR $nout mutants selected for the full-suite run have no outcome in any shard (a name unknown to cargo-mutants at this revision, or a shard that died)"
|
||||
exit 1
|
||||
fi
|
||||
C=$(cat "$D/m/s1.caught" "$D/m/s2.caught" | sort -u | wc -l | tr -d ' ')
|
||||
U=$(cat "$D/m/s1.unviable" "$D/m/s2.unviable" | sort -u | wc -l | tr -d ' ')
|
||||
M=$(wc -l <"$D/m/s2.missed" | tr -d ' '); T=$(wc -l <"$D/m/s2.timeout" | tr -d ' ')
|
||||
S=$(wc -l <"$D/m/s1.surv" | tr -d ' '); F=$(wc -l <"$D/m/s1.names" | tr -d ' ')
|
||||
X=$(comm -12 "$D/m/s1.surv" "$D/m/s2.caught" | wc -l | tr -d ' ')
|
||||
K=$K1; [ "$K2" -le "$K" ] || K=$K2
|
||||
sum "MUTANTS tested=$((C + M + T + U)) caught=$C missed=$M timeout=$T unviable=$U | stage1 filtered=$F survivors=$S confirmed_caught=$X | containers=$K | plan=${tp}s stage1=${ts1}s stage2=${ts2}s | $(awk -v e="$(( $(now) - t0 ))" 'BEGIN { printf "%.1f", e / 60 }')m"
|
||||
{ sed 's/^/MISSED /' "$D/m/s2.missed"; sed 's/^/TIMEOUT /' "$D/m/s2.timeout"; } >"$D/m/surv.txt"
|
||||
total=$(wc -l <"$D/m/surv.txt" | tr -d ' ')
|
||||
head -n 50 "$D/m/surv.txt" | while IFS= read -r l; do sum "$l"; done
|
||||
if [ "$total" -gt 50 ]; then
|
||||
sum "... $((total - 50)) more MISSED/TIMEOUT names; they follow in the job log as cm-survivor lines"
|
||||
tail -n +51 "$D/m/surv.txt" | sed 's/^/cm-survivor: /'
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- inside a container -----------------------------------------------------
|
||||
set +e
|
||||
# overridable only so tools/test-tools.sh can run this half against a mock
|
||||
SRC=${CM_SRC:-/src}; CM=${CM_DIR:-/cm}; OUT=${CM_OUT:-/out}; SEED=${CM_SEED:-/seed}
|
||||
if [ -d "$SEED" ]; then mkdir -p "$SRC" && cp -a "$SEED/." "$SRC/" || { echo "cannot copy $SEED into $SRC"; exit 2; }; fi
|
||||
cd "$SRC/${WORKDIR:-.}" || { echo "BAD WORKDIR"; exit 2; }
|
||||
if [ "$1" = --plan ]; then
|
||||
wsroot=$(dirname "$(cargo locate-project --workspace --message-format plain)") || { echo "cargo locate-project failed"; exit 2; }
|
||||
git -C "$wsroot" diff --relative "$BASE" HEAD >"$OUT/d" || { echo "git diff of the range failed"; exit 2; }
|
||||
cargo mutants --list --in-diff "$OUT/d" >"$OUT/names.txt" 2>"$OUT/list.err" || { echo "cargo mutants --list failed"; tail -n 5 "$OUT/list.err"; exit 2; }
|
||||
exit 0
|
||||
fi
|
||||
# --item: stage 1 (CM_FILE with its CM_FILTER) or stage 2 (shard CM_SHARD of the names in $CM/s2.re)
|
||||
set -f; read -ra targs <<<"${TEST_ARGS:-}"; set +f
|
||||
o=${CM_TMP:-/tmp}/cmo
|
||||
args=(mutants --in-diff "$CM/plan/d" -o "$o" --jobs "$JOBS")
|
||||
tail_args=()
|
||||
if [ "$CM_STAGE" = 1 ]; then
|
||||
args+=(-f "$CM_FILE" --minimum-test-timeout "$MIN_TEST_TIMEOUT")
|
||||
tail_args=(-- "$CM_FILTER"); [ "${#targs[@]}" = 0 ] || tail_args+=(-- "${targs[@]}")
|
||||
else
|
||||
mapfile -t res <"$CM/s2.re"
|
||||
for r in "${res[@]}"; do args+=(-F "$r"); done
|
||||
args+=(--timeout-multiplier "$CONFIRM_TIMEOUT_MULT")
|
||||
case $CM_SHARD in */1) ;; *) args+=(--shard "$CM_SHARD") ;; esac
|
||||
[ "${#targs[@]}" = 0 ] || tail_args=(-- -- "${targs[@]}")
|
||||
fi
|
||||
cargo "${args[@]}" "${tail_args[@]}" >"$OUT/cargo.log" 2>&1
|
||||
rc=$?
|
||||
for l in caught missed timeout unviable; do cp "$o/mutants.out/$l.txt" "$OUT/$l.txt" 2>/dev/null || : >"$OUT/$l.txt"; done
|
||||
awk '/^running [0-9]+ tests?/ { s += $2 } END { print s + 0 }' "$o/mutants.out/log/baseline.log" >"$OUT/tests" 2>/dev/null || echo 0 >"$OUT/tests"
|
||||
echo "$rc" >"$OUT/rc"
|
||||
tail -n 30 "$OUT/cargo.log"
|
||||
# 0 clean, 2 missed and 3 timeouts are normal; 4 (red baseline) and everything else are errors
|
||||
case $rc in 0|2|3) exit 0 ;; *) exit "$rc" ;; esac
|
||||
2
tools/fixtures/cargo-mutants-crate/.gitignore
vendored
Normal file
2
tools/fixtures/cargo-mutants-crate/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
target/
|
||||
Cargo.lock
|
||||
4
tools/fixtures/cargo-mutants-crate/Cargo.toml
Normal file
4
tools/fixtures/cargo-mutants-crate/Cargo.toml
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
[package]
|
||||
name = "cmfix"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
18
tools/fixtures/cargo-mutants-crate/src/alpha/mod.rs
Normal file
18
tools/fixtures/cargo-mutants-crate/src/alpha/mod.rs
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
pub fn add(a: i32, b: i32) -> i32 {
|
||||
a + b
|
||||
}
|
||||
|
||||
/// Only tests/it.rs pins this one: a unit-filtered run cannot kill it.
|
||||
pub fn double(a: i32) -> i32 {
|
||||
a * 2
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn add_adds() {
|
||||
assert_eq!(add(2, 3), 5);
|
||||
}
|
||||
}
|
||||
18
tools/fixtures/cargo-mutants-crate/src/beta.rs
Normal file
18
tools/fixtures/cargo-mutants-crate/src/beta.rs
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
pub fn neg(a: i32) -> i32 {
|
||||
-a
|
||||
}
|
||||
|
||||
/// No test pins this one anywhere: a real survivor in every run.
|
||||
pub fn unpinned(a: i32) -> i32 {
|
||||
a + 1
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn neg_negates() {
|
||||
assert_eq!(neg(4), -4);
|
||||
}
|
||||
}
|
||||
2
tools/fixtures/cargo-mutants-crate/src/lib.rs
Normal file
2
tools/fixtures/cargo-mutants-crate/src/lib.rs
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
pub mod alpha;
|
||||
pub mod beta;
|
||||
4
tools/fixtures/cargo-mutants-crate/tests/it.rs
Normal file
4
tools/fixtures/cargo-mutants-crate/tests/it.rs
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
#[test]
|
||||
fn double_doubles() {
|
||||
assert_eq!(cmfix::alpha::double(4), 8);
|
||||
}
|
||||
64
tools/fixtures/mutation-mock/fakebin/cargo
Normal file
64
tools/fixtures/mutation-mock/fakebin/cargo
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
#!/usr/bin/env bash
|
||||
# Fake cargo for tools/test-tools.sh, only what tools/cargo-mutants.sh calls:
|
||||
# cargo locate-project --workspace --message-format plain
|
||||
# cargo mutants [--list] --in-diff <diff> [-f <file>] [-F <re>]... [--shard k/n] [-o <dir>] [...] [-- <filter> [-- <harness args>]]
|
||||
# FAKE_CM_TABLE: tab-separated <mutant name> <outcome under a filtered run> <outcome under the full suite>
|
||||
# (caught|missed|timeout|unviable). A run with -F is the full suite; a run with -f and a filter is filtered.
|
||||
# FAKE_CM_TESTS: tab-separated <filter> <tests it runs> (default 1; the unfiltered suite runs 5)
|
||||
# FAKE_CM_RED=1|2: the stage-1 or stage-2 run fails its baseline (exit 4, no output).
|
||||
# FAKE_CM_KILLS: a path a run creates (the watchdog's YGG_KILLED flag appearing mid-job).
|
||||
# FAKE_CM_LOG: appended one line per run (stage, filter, harness args, open fds from 10 up).
|
||||
# FAKE_WSROOT: the workspace root relative to the repository root, with a trailing slash ('' = the repository root).
|
||||
case ${1:-} in
|
||||
locate-project) echo "$(git rev-parse --show-toplevel)/${FAKE_WSROOT:-}Cargo.toml"; exit 0 ;;
|
||||
mutants) shift ;;
|
||||
*) echo "fake cargo: unsupported ${1:-}" >&2; exit 127 ;;
|
||||
esac
|
||||
list=0; diff=''; ffile=''; res=(); shard=''; out=''; tail=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
--list) list=1 ;;
|
||||
--in-diff) diff=$2; shift ;;
|
||||
-f) ffile=$2; shift ;;
|
||||
-F) res+=("$2"); shift ;;
|
||||
--shard) shard=$2; shift ;;
|
||||
-o) out=$2; shift ;;
|
||||
--jobs|--minimum-test-timeout|--timeout-multiplier) shift ;;
|
||||
--) shift; tail=("$@"); break ;;
|
||||
*) echo "fake cargo mutants: unknown argument $1" >&2; exit 64 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
[ -n "$diff" ] && [ -f "$diff" ] || { echo "fake cargo mutants: no readable --in-diff" >&2; exit 64; }
|
||||
# the mutants of the files the diff touches (paths as the diff prints them: relative to the workspace root)
|
||||
touched=$(sed -n 's|^+++ b/||p' "$diff")
|
||||
sel=$(awk -F'\t' 'NR == FNR { t[$0] = 1; next } { f = $1; sub(/:.*/, "", f); if (f in t) print $1 "\t" $2 "\t" $3 }' <(printf '%s\n' "$touched") "${FAKE_CM_TABLE:?}")
|
||||
if [ -n "$ffile" ]; then sel=$(awk -F'\t' -v f="$ffile" '{ n = $1; sub(/:.*/, "", n); if (n == f) print }' <<<"$sel"); fi
|
||||
if [ "${#res[@]}" -gt 0 ]; then
|
||||
keep=$(cut -f1 <<<"$sel" | perl -e '@r = map { qr/$_/ } @ARGV; while (<STDIN>) { chomp; for $r (@r) { if (/$r/) { print "$_\n"; last } } }' "${res[@]}")
|
||||
sel=$(awk -F'\t' 'NR == FNR { k[$0] = 1; next } $1 in k' <(printf '%s\n' "$keep") <(printf '%s\n' "$sel"))
|
||||
fi
|
||||
if [ -n "$shard" ]; then
|
||||
sel=$(awk -F'\t' -v k="${shard%/*}" -v n="${shard#*/}" '(NR - 1) % n == k' <<<"$sel")
|
||||
fi
|
||||
if [ "$list" = 1 ]; then cut -f1 <<<"$sel" | awk 'NF'; exit 0; fi
|
||||
stage=1; [ "${#res[@]}" -eq 0 ] || stage=2
|
||||
filter=''; harness=()
|
||||
if [ "${#tail[@]}" -gt 0 ] && [ "${tail[0]}" != -- ]; then filter=${tail[0]}; fi
|
||||
for i in "${!tail[@]}"; do if [ "${tail[$i]}" = -- ]; then harness=("${tail[@]:$((i + 1))}"); break; fi; done
|
||||
open=$(perl -MPOSIX -e '$n = 0; for (10 .. 40) { my $d = POSIX::dup($_); if (defined $d) { $n++; POSIX::close($d) } } print $n')
|
||||
[ -z "${FAKE_CM_KILLS:-}" ] || touch "$FAKE_CM_KILLS" # a watchdog kill arriving while a run is going
|
||||
echo "stage=$stage filter=$filter harness=${harness[*]} openfds=$open" >>"${FAKE_CM_LOG:?}"
|
||||
[ "${FAKE_CM_RED:-}" != "$stage" ] || { echo "baseline failed" >&2; exit 4; }
|
||||
tests=5
|
||||
if [ "$stage" = 1 ] && [ -n "$filter" ]; then tests=$(awk -F'\t' -v f="$filter" '$1 == f { print $2 }' "${FAKE_CM_TESTS:-/dev/null}"); tests=${tests:-1}; fi
|
||||
mkdir -p "$out/mutants.out/log"
|
||||
printf 'running 0 tests\nrunning %s tests\n' "$tests" >"$out/mutants.out/log/baseline.log"
|
||||
col=3; [ "$stage" = 2 ] || col=2
|
||||
for o in caught missed timeout unviable; do
|
||||
awk -F'\t' -v c="$col" -v o="$o" -v z="$tests" '{ x = $c; if (z == 0) x = "missed"; if (x == o) print $1 }' <<<"$sel" >"$out/mutants.out/$o.txt"
|
||||
done
|
||||
# a run that selected no mutant (the awk above printed empty lines): drop them
|
||||
for o in caught missed timeout unviable; do sed -i '/^$/d' "$out/mutants.out/$o.txt"; done
|
||||
if [ -s "$out/mutants.out/timeout.txt" ]; then exit 3; elif [ -s "$out/mutants.out/missed.txt" ]; then exit 2; fi
|
||||
exit 0
|
||||
|
|
@ -1,21 +1,22 @@
|
|||
#!/usr/bin/env bash
|
||||
# Mock docker for tools/test-tools.sh: a container is a directory, an image is a
|
||||
# copy of one, so `run` from a committed image really is copy-on-write. Implements
|
||||
# only what tools/mutation-table.sh --host calls: run, commit, cp, rm, rmi. The
|
||||
# in-container paths /mt and /seed map to the mounted host directories, and the
|
||||
# script's own /src and /tmp/mtm live inside the container directory.
|
||||
# only what tools/mutation-table.sh and tools/cargo-mutants.sh --host call: run, commit,
|
||||
# cp, rm, rmi. The in-container paths /mt, /cm, /out and /seed map to the mounted host
|
||||
# directories (/out is written through), and the script's own /src and /tmp/mtm live
|
||||
# inside the container directory.
|
||||
FD=${FAKE_DOCKER:?}; mkdir -p "$FD/c" "$FD/i"
|
||||
imgdir() { echo "$FD/i/${1//[:\/]/_}"; }
|
||||
cmd=$1; shift
|
||||
case $cmd in
|
||||
run)
|
||||
rm_after=0; name=''; mt=''; seed=/nonexistent; labels=0; envs=()
|
||||
rm_after=0; name=''; mt=''; cm=''; out=''; seed=/nonexistent; labels=0; envs=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
--rm) rm_after=1 ;;
|
||||
--name) name=$2; shift ;;
|
||||
--label) labels=$((labels + 1)); shift ;;
|
||||
-v) case $2 in *:/mt:ro) mt=${2%:/mt:ro} ;; *:/seed:ro) seed=${2%:/seed:ro} ;; esac; shift ;;
|
||||
-v) case $2 in *:/mt:ro) mt=${2%:/mt:ro} ;; *:/cm:ro) cm=${2%:/cm:ro} ;; *:/out) out=${2%:/out} ;; *:/seed:ro) seed=${2%:/seed:ro} ;; esac; shift ;;
|
||||
-e) case $2 in BASH_ENV=*) ;; *) envs+=("$2") ;; esac; shift ;;
|
||||
-w) shift ;;
|
||||
--*=*) ;;
|
||||
|
|
@ -29,9 +30,9 @@ case $cmd in
|
|||
c=$FD/c/$name; rm -rf "$c"
|
||||
if [ -d "$(imgdir "$image")" ]; then cp -a "$(imgdir "$image")" "$c"; else mkdir -p "$c"; fi
|
||||
echo "run $name $image" >>"$FD/log"
|
||||
script=${*: -1}; script=${script//\/mt\//$mt/}
|
||||
script=${*: -1}; script=${script//\/mt\//$mt/}; script=${script//\/cm\//$cm/}
|
||||
( for e in "${envs[@]}"; do export "$e"; done
|
||||
export MT_SRC=$c/src MT_M=$c/tmp/mtm MT_DIR=$mt MT_SEED=$seed
|
||||
export MT_SRC=$c/src MT_M=$c/tmp/mtm MT_DIR=$mt MT_SEED=$seed CM_SRC=$c/src CM_DIR=$cm CM_OUT=$out CM_SEED=$seed CM_TMP=$c/tmp
|
||||
bash -o pipefail -c "$script" )
|
||||
rc=$?
|
||||
[ "$rm_after" = 0 ] || rm -rf "$c"
|
||||
|
|
|
|||
4
tools/fixtures/mutation-mock/fakebin/flock
Normal file
4
tools/fixtures/mutation-mock/fakebin/flock
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
#!/usr/bin/env bash
|
||||
# Fake flock for tools/test-tools.sh (Git Bash has none): only `flock -n <fd>`, via perl's flock(2) on the inherited descriptor.
|
||||
[ "${1:-}" = -n ] && [ -n "${2:-}" ] || { echo "fake flock: unsupported arguments $*" >&2; exit 64; }
|
||||
exec perl -e 'use Fcntl qw(:flock); open(my $f, "+<&=", $ARGV[0]) or exit 2; exit(flock($f, LOCK_EX | LOCK_NB) ? 0 : 1)' "$2"
|
||||
|
|
@ -83,10 +83,12 @@
|
|||
# YGG_WORK (the clone), YGG_IMAGE, YGG_CPUS, YGG_MEM, YGG_CNAME (job name),
|
||||
# YGG_LABEL (every container it starts must carry `--label $YGG_LABEL`: the
|
||||
# watchdog kills containers by that label), YGG_KILLED (a path that exists once
|
||||
# the watchdog has killed the job: stop starting work) and DOCKER_ARGS. The
|
||||
# command owns the container topology and must remove what it creates.
|
||||
# tools/mutation-table.sh is the one user: it snapshots a baselined container
|
||||
# and runs each mutant in a fresh container from the snapshot.
|
||||
# the watchdog has killed the job: stop starting work), YGG_SLOTS and YGG_ROOT
|
||||
# (the slot pool: $YGG_ROOT/slot-<i>.lock for i in 1..$YGG_SLOTS, flock -n) and
|
||||
# DOCKER_ARGS. The command owns the container topology and must remove what it
|
||||
# creates. Two users: tools/mutation-table.sh snapshots a baselined container
|
||||
# and runs each mutant in a fresh container from the snapshot;
|
||||
# tools/cargo-mutants.sh also leases free slot locks for extra containers.
|
||||
# DOCKER_ARGS='<extra docker run flags>' passes through to the container's
|
||||
# own `docker run`, appended after the fixed flags and before the image name
|
||||
# (e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0' or
|
||||
|
|
@ -133,8 +135,8 @@
|
|||
# dotnet tool install -g dotnet-stryker)
|
||||
# any other value is used as an image name as-is.
|
||||
# Example:
|
||||
# ygg-verify.sh /f/Projects/Huginn HEAD rust \
|
||||
# 'cargo test --workspace && git diff BASE HEAD > /tmp/d && cargo mutants --in-diff /tmp/d'
|
||||
# ygg-verify.sh /f/Projects/Huginn HEAD rust 'cargo test --workspace'
|
||||
# Rust mutation runs use tools/cargo-mutants.sh, never an inline cargo mutants command.
|
||||
set -euo pipefail
|
||||
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
|
|
@ -186,7 +188,6 @@ ygg_summarize() {
|
|||
if [ "$untrusted" -gt 0 ]; then
|
||||
echo "$untrusted YGG-SUMMARY lines without this run's nonce were ignored (a test printed them, or the table printer predates the nonce); see the log"
|
||||
fi
|
||||
{ grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20'
|
||||
if [ "$status" != 0 ]; then
|
||||
if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ] && [ -z "$loads" ]; then
|
||||
echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command"
|
||||
|
|
@ -389,7 +390,7 @@ if [ "$host_mode" = 1 ]; then
|
|||
# Own process group (set -m), started in the job's work directory, so the watchdog can signal all of it.
|
||||
set -m
|
||||
( cd "$work" && export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \
|
||||
YGG_KILLED=$killed DOCKER_ARGS=$docker_args
|
||||
YGG_KILLED=$killed YGG_SLOTS=$slots YGG_ROOT=$root DOCKER_ARGS=$docker_args
|
||||
[ -z "$nonce" ] || export YGG_NONCE=$nonce
|
||||
exec bash -o pipefail -c "$cmd" ) </dev/null &
|
||||
hostpid=$!
|
||||
|
|
|
|||
|
|
@ -5,6 +5,9 @@
|
|||
# reproducer) are real Yggdrasil runs, 2026-10-01; soul-s5/*.log are real
|
||||
# dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per
|
||||
# log, split by "=== MARK <name>" lines); the rest are hand-written.
|
||||
# The mutation runners tools/mutation-table.sh and tools/cargo-mutants.sh (launcher, host driver
|
||||
# and container halves) run under a mock docker and, for cargo-mutants, a fake cargo and flock
|
||||
# (fixtures/mutation-mock/fakebin), with real slot locks.
|
||||
# Run: bash tools/test-tools.sh (no network, no builds)
|
||||
set -uo pipefail
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
|
|
@ -192,6 +195,179 @@ check "$(seg jest-real.log exit)" 1 "exit non-zero with no failing test parsed"
|
|||
# F4: the "nothing recognised" line names every framework the parser reads
|
||||
check "$(seg jest-real.log exit)" 1 "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)"
|
||||
|
||||
# ---- tools/cargo-mutants.sh: the launcher, host driver and container halves, through the mock docker
|
||||
# and fixtures/mutation-mock/fakebin/cargo (a fake `cargo mutants` driven by a table of mutant ->
|
||||
# outcome under a filtered run and under the full suite). The launcher runs for real: its bundle,
|
||||
# quoting and the environment it hands the driver are the production path; only ygg-verify.sh is
|
||||
# stubbed, by a script that does what RUN_ON_HOST=1 does (run the command inside the checked-out
|
||||
# revision). Slot locks are real flock(2) locks on files, taken by the driver and by the test.
|
||||
CMB=$here/fixtures/mutation-mock/fakebin
|
||||
cmrepo() { # cmrepo A|B: sets cm_repo and cm_wd. A: a workspace at the repo root, workdir crates/x. B: a standalone package pkg/ under the repo root.
|
||||
local r f c; r=$(mktemp -d); cm_repo=$r/repo; mkdir -p "$cm_repo"; git -C "$cm_repo" init -q
|
||||
c="git -C $cm_repo -c user.name=t -c user.email=t@t"
|
||||
if [ "$1" = A ]; then
|
||||
cm_wd=crates/x; mkdir -p "$cm_repo/crates/x/src/alpha" "$cm_repo/crates/x/src/bin"
|
||||
printf '[workspace]\nmembers = ["crates/x"]\n' >"$cm_repo/Cargo.toml"; printf '[package]\nname = "x"\n' >"$cm_repo/crates/x/Cargo.toml"
|
||||
: >"$cm_repo/crates/x/src/lib.rs"; $c add -A 2>/dev/null; $c commit -qm c1
|
||||
for f in src/lib.rs src/main.rs src/gamma.rs src/beta.rs src/alpha/mod.rs src/alpha/inner.rs src/bin/tool.rs build.rs; do echo "// $f" >"$cm_repo/crates/x/$f"; done
|
||||
else
|
||||
cm_wd=pkg; mkdir -p "$cm_repo/pkg/src/a" "$cm_repo/pkg/src/bin"
|
||||
printf '[package]\nname = "p"\n' >"$cm_repo/pkg/Cargo.toml"; : >"$cm_repo/pkg/src/lib.rs"; $c add -A 2>/dev/null; $c commit -qm c1
|
||||
for f in src/lib.rs src/main.rs src/bin/x.rs build.rs src/a/mod.rs src/a/b.rs; do echo "// $f" >"$cm_repo/pkg/$f"; done
|
||||
fi
|
||||
$c add -A 2>/dev/null; $c commit -qm c2
|
||||
}
|
||||
cmtable() { # cmtable A|B: sets cm_table (name <TAB> outcome under a filtered run <TAB> outcome under the full suite)
|
||||
cm_table=$(mktemp)
|
||||
if [ "$1" = A ]; then
|
||||
local p=crates/x/src
|
||||
printf '%s\n' \
|
||||
"$p/alpha/mod.rs:2:5: replace add -> i32 with 0 caught caught" \
|
||||
"$p/alpha/mod.rs:7:7: replace * with + in double missed caught" \
|
||||
"$p/alpha/mod.rs:9:9: replace && with || in check missed missed" \
|
||||
"$p/alpha/mod.rs:12:5: replace f(x) -> Vec<i32> with vec![1] caught caught" \
|
||||
"$p/alpha/mod.rs:15:5: replace id -> Option<i32> with None unviable unviable" \
|
||||
"$p/alpha/inner.rs:3:5: replace slow -> u32 with 0 timeout caught" \
|
||||
"$p/alpha/inner.rs:6:5: replace spin -> u32 with 1 missed timeout" \
|
||||
"$p/beta.rs:2:5: replace b1 -> i32 with 0 missed caught" \
|
||||
"$p/beta.rs:3:5: replace b2 -> i32 with 0 missed missed" \
|
||||
"$p/beta.rs:4:5: replace b3 -> i32 with 0 missed unviable" \
|
||||
"$p/gamma.rs:2:5: replace g -> i32 with 0 caught caught" \
|
||||
"$p/lib.rs:2:5: replace l1 -> i32 with 0 missed caught" \
|
||||
"$p/lib.rs:3:5: replace f(x) -> Vec<i32> with vec![1] missed missed" \
|
||||
"$p/main.rs:2:5: replace m -> i32 with 0 missed caught" \
|
||||
"crates/x/build.rs:2:5: replace bu -> i32 with 0 missed caught" \
|
||||
"$p/bin/tool.rs:2:5: replace t -> i32 with 0 missed caught" >"$cm_table"
|
||||
else
|
||||
printf '%s\n' "src/lib.rs:2:5: replace a -> i32 with 0 caught caught" "src/main.rs:2:5: replace a -> i32 with 0 caught caught" \
|
||||
"src/bin/x.rs:2:5: replace a -> i32 with 0 caught caught" "build.rs:2:5: replace a -> i32 with 0 caught caught" \
|
||||
"src/a/mod.rs:2:5: replace a -> i32 with 0 caught caught" "src/a/b.rs:2:5: replace a -> i32 with 0 caught caught" >"$cm_table"
|
||||
fi
|
||||
}
|
||||
# cm <A|B> [VAR=value]...: runs the launcher; sets cm_out, cm_rc, cm_root, cm_fd, cm_log. HOLD_OWN=1 holds the job's own slot as
|
||||
# ygg-verify does (a descriptor the driver inherits); HOLD_OTHERS=<n> holds n further slots as other jobs would (separate processes); PSI=<n> is the io pressure file's full avg60.
|
||||
cm() {
|
||||
local lay=$1; shift
|
||||
cmrepo "$lay"; cmtable "$lay"
|
||||
cm_root=$(dirname "$cm_repo"); cm_fd=$cm_root/fd; cm_log=$cm_root/cargo.log
|
||||
mkdir -p "$cm_fd" "$cm_root/tools/stopgap" "$cm_root/ygg" "$cm_root/tmp"
|
||||
cp "$here/cargo-mutants.sh" "$cm_root/tools/"
|
||||
printf '#!/usr/bin/env bash\n# test stub of ygg-verify.sh RUN_ON_HOST=1: <repo> <rev> <image> <cmd>; runs cmd as the host half inside the revision\ncd "$YGG_WORK" && exec bash -o pipefail -c "$4"\n' >"$cm_root/tools/stopgap/ygg-verify.sh"
|
||||
local holders=() i n own=0 others=0 psi=0.00 kv env=()
|
||||
for kv in "$@"; do case $kv in HOLD_OWN=*) own=${kv#*=} ;; HOLD_OTHERS=*) others=${kv#*=} ;; PSI=*) psi=${kv#*=} ;; *) env+=("$kv") ;; esac; done
|
||||
for i in $(seq 1 "$others"); do
|
||||
perl -e 'use Fcntl ":flock"; open(F, ">>", $ARGV[0]) or die; flock(F, LOCK_EX) or die; open(R, ">", "$ARGV[0].ready"); close R; sleep 120' "$cm_root/ygg/slot-$((i + 1)).lock" & holders+=($!)
|
||||
for n in $(seq 1 50); do [ -e "$cm_root/ygg/slot-$((i + 1)).lock.ready" ] && break; sleep 0.1; done
|
||||
done
|
||||
cm_out=$( ( if [ "$own" = 1 ]; then exec {ownfd}>"$cm_root/ygg/slot-1.lock"; PATH=$CMB:$PATH flock -n "$ownfd"; fi
|
||||
export PATH=$CMB:$PATH FAKE_DOCKER=$cm_fd FAKE_CM_TABLE=$cm_table FAKE_CM_LOG=$cm_log FAKE_CM_TESTS=$cm_root/tests.tsv \
|
||||
FAKE_WSROOT=$([ "$lay" = A ] || echo pkg/) YGG_WORK=$cm_repo YGG_IMAGE=rust YGG_CNAME=job1 YGG_LABEL=eureka-verify-job=job1 YGG_CPUS=1 YGG_MEM=1g \
|
||||
YGG_SLOTS=5 YGG_ROOT=$cm_root/ygg YGG_NONCE=n0nce1 CM_PSI=$cm_root/psi TMPDIR=$cm_root/tmp
|
||||
printf 'beta::\t0\n' >"$cm_root/tests.tsv"
|
||||
printf 'some avg10=0.00 avg60=0.00 avg300=0.00 total=0\nfull avg10=0.00 avg60=%s avg300=0.00 total=0\n' "$psi" >"$cm_root/psi"
|
||||
for kv in "${env[@]}"; do export "${kv?}"; done
|
||||
timeout 240 bash "$cm_root/tools/cargo-mutants.sh" "$cm_repo" HEAD HEAD~1 "$cm_wd" 2>&1 ) )
|
||||
cm_rc=$?
|
||||
for i in "${holders[@]}"; do kill "$i" 2>/dev/null; done; wait 2>/dev/null
|
||||
}
|
||||
cmcheck() { # cmcheck <name> <must-contain | !must-not-contain>...
|
||||
local name=$1 s; shift
|
||||
for s in "$@"; do
|
||||
case $s in
|
||||
'!'*) if printf '%s\n' "$cm_out" | grep -qF -- "${s#!}"; then echo "FAIL cargo-mutants $name: must not contain: ${s#!}"; printf '%s\n' "$cm_out" | tail -n 12; fail=1; return; fi ;;
|
||||
*) printf '%s\n' "$cm_out" | grep -qF -- "$s" || { echo "FAIL cargo-mutants $name: missing: $s"; printf '%s\n' "$cm_out" | tail -n 14; fail=1; return; } ;;
|
||||
esac
|
||||
done
|
||||
echo "ok cargo-mutants $name"
|
||||
}
|
||||
cmrc() { [ "$cm_rc" = "$2" ] || { echo "FAIL cargo-mutants $1: exit $cm_rc, want $2"; printf '%s\n' "$cm_out" | tail -n 8; fail=1; }; }
|
||||
cmclean() { # nothing the job made may outlive it: no container directory, no bundle, no staging directory
|
||||
local left; left=$(ls "$cm_fd/c" "$cm_root/tmp" 2>/dev/null | grep -v '^$' | grep -v ':$' | paste -sd, -)
|
||||
if [ -z "$left" ]; then echo "ok cargo-mutants $1: containers, bundle and staging removed"; else echo "FAIL cargo-mutants $1: left behind: $left"; fail=1; fi
|
||||
}
|
||||
cmlocks() { # cmlocks <case>: after the driver exits, every slot lock can be taken again by a fresh process
|
||||
local i bad=''
|
||||
for i in 1 2 3 4 5; do bash -c "exec {fd}>'$cm_root/ygg/slot-$i.lock'; PATH=$CMB:\$PATH flock -n \$fd" || bad="$bad $i"; done
|
||||
if [ -z "$bad" ]; then echo "ok cargo-mutants $1: every slot lock is free again"; else echo "FAIL cargo-mutants $1: slots still locked:$bad"; fail=1; fi
|
||||
}
|
||||
|
||||
# filter derivation, stage-1 survivors and timeouts confirmed, zero-test filter, no raw stage-1 lines, exit 2 and 3 normal
|
||||
cm A TEST_ARGS='--skip flaky'
|
||||
cmrc A 0
|
||||
cmcheck A-filters \
|
||||
'FILTER crates/x/src/alpha/mod.rs alpha:: tests=1 mutants=5' 'FILTER crates/x/src/alpha/inner.rs alpha::inner:: tests=1 mutants=2' \
|
||||
'FILTER crates/x/src/gamma.rs gamma:: tests=1 mutants=1' \
|
||||
'!FILTER crates/x/src/lib.rs' '!FILTER crates/x/src/main.rs' '!FILTER crates/x/src/bin/tool.rs' '!FILTER crates/x/build.rs'
|
||||
cmcheck A-zero-test-filter 'FILTER crates/x/src/beta.rs beta:: tests=0 mutants=3' 'caught=10 missed=3 timeout=1 unviable=2'
|
||||
cmcheck A-confirmed-and-counted 'MUTANTS tested=16 caught=10 missed=3 timeout=1 unviable=2 | stage1 filtered=8 survivors=4 confirmed_caught=2 | containers=3 |' \
|
||||
'!replace * with + in double' '!replace slow -> u32 with 0' '!replace add -> i32' \
|
||||
'MISSED crates/x/src/alpha/mod.rs:9:9: replace && with || in check' 'MISSED crates/x/src/beta.rs:3:5: replace b2' 'MISSED crates/x/src/lib.rs:3:5: replace f(x) -> Vec<i32> with vec![1]' \
|
||||
'TIMEOUT crates/x/src/alpha/inner.rs:6:5: replace spin'
|
||||
if [ "$(printf '%s\n' "$cm_out" | grep -c "^YGG-SUMMARY/n0nce1: MISSED ")" = 3 ]; then echo "ok cargo-mutants A-missed-once: 3 final MISSED lines"; else echo "FAIL cargo-mutants A-missed-once"; fail=1; fi
|
||||
if printf '%s\n' "$cm_out" | grep -qE '^(MISSED|TIMEOUT) '; then echo "FAIL cargo-mutants A-no-raw-lines: a MISSED or TIMEOUT line outside the nonce-tagged lines"; fail=1; else echo "ok cargo-mutants A-no-raw-lines"; fi
|
||||
if grep -qF 'stage=1 filter=alpha:: harness=--skip flaky' "$cm_log" && grep -qF 'stage=1 filter=alpha::inner:: harness=--skip flaky' "$cm_log" && grep -qF 'stage=2 filter= harness=--skip flaky' "$cm_log"
|
||||
then echo "ok cargo-mutants A-harness-args: TEST_ARGS follow the filter in stage 1 and stand alone in stage 2"; else echo "FAIL cargo-mutants A-harness-args"; sort -u "$cm_log"; fail=1; fi
|
||||
if [ "$(grep -c '^stage=2' "$cm_log")" = 3 ]; then echo "ok cargo-mutants A-stage2-sharded: 3 full-suite shards"; else echo "FAIL cargo-mutants A-stage2-sharded"; fail=1; fi
|
||||
cmclean A; cmlocks A
|
||||
|
||||
# derivation on a standalone package under the repo root: the diff is taken at the workspace root (--relative)
|
||||
cm B
|
||||
cmrc B 0
|
||||
cmcheck B-derivation 'FILTER src/a/mod.rs a:: tests=1 mutants=1' 'FILTER src/a/b.rs a::b:: tests=1 mutants=1' \
|
||||
'!FILTER src/lib.rs' '!FILTER src/main.rs' '!FILTER src/bin/x.rs' '!FILTER build.rs' 'MUTANTS tested=6 caught=6 missed=0'
|
||||
# FILTERS overrides the derived filter; '-' forces the full suite
|
||||
cm B 'FILTERS=src/a/b.rs=-,src/lib.rs=special::'
|
||||
cmcheck B-override 'FILTER src/lib.rs special:: tests=1 mutants=1' 'FILTER src/a/mod.rs a:: ' '!FILTER src/a/b.rs' 'tested=6 caught=6'
|
||||
|
||||
# slot lease: the job's own slot is held by ygg-verify (an inherited descriptor), others by other jobs
|
||||
cm A HOLD_OWN=1 HOLD_OTHERS=1; cmcheck lease-free 'containers=3 |'; cmlocks lease-free
|
||||
cm A HOLD_OWN=1 HOLD_OTHERS=3; cmcheck lease-one-free 'containers=2 |'
|
||||
cm A HOLD_OWN=1 HOLD_OTHERS=4; cmcheck lease-none-free 'containers=1 |' 'tested=16'; cmlocks lease-none-free
|
||||
open_none=$(grep -o 'openfds=[0-9]*' "$cm_log" | sort -u | paste -sd, -)
|
||||
cm A HOLD_OWN=1 HOLD_OTHERS=1; open_leased=$(grep -o 'openfds=[0-9]*' "$cm_log" | sort -u | paste -sd, -)
|
||||
if [ "$open_none" = "$open_leased" ]; then echo "ok cargo-mutants lease-not-inherited: containers see the same descriptors with 2 leases held as with none ($open_none)"; else echo "FAIL cargo-mutants lease-not-inherited: $open_none vs $open_leased"; fail=1; fi
|
||||
cm A PSI=41.00; cmcheck lease-io-pressure 'containers=1 |' 'tested=16'
|
||||
# a red baseline, in stage 1 or in stage 2, is not a clean run
|
||||
cm A FAKE_CM_RED=1
|
||||
[ "$cm_rc" != 0 ] || { echo "FAIL cargo-mutants red-stage1: exit 0"; fail=1; }
|
||||
cmcheck red-stage1 'ERROR stage 1 (' 'container exited 4' '!MUTANTS' '!MISSED' '!TIMEOUT'
|
||||
cm A FAKE_CM_RED=2
|
||||
[ "$cm_rc" != 0 ] || { echo "FAIL cargo-mutants red-stage2: exit 0"; fail=1; }
|
||||
cmcheck red-stage2 'ERROR stage 2 (shard' 'container exited 4' '!MUTANTS' '!MISSED' '!TIMEOUT'
|
||||
cmclean red-stage2
|
||||
# NAMES rerun: no stage-1 container, only the listed names, on the full suite
|
||||
cm_names=$(mktemp)
|
||||
printf '%s\n' 'crates/x/src/alpha/mod.rs:9:9: replace && with || in check' 'crates/x/src/lib.rs:3:5: replace f(x) -> Vec<i32> with vec![1]' 'crates/x/src/beta.rs:2:5: replace b1 -> i32 with 0' >"$cm_names"
|
||||
cm A "NAMES=$cm_names"
|
||||
cmrc names 0
|
||||
cmcheck names 'MUTANTS tested=3 caught=1 missed=2 timeout=0 unviable=0 | stage1 filtered=0 survivors=0 confirmed_caught=0' 'MISSED crates/x/src/alpha/mod.rs:9:9' '!FILTER'
|
||||
if grep -q 'run job1-s1' "$cm_fd/log" || grep -q '^stage=1' "$cm_log"; then echo "FAIL cargo-mutants names: a stage-1 container or a filtered run"; fail=1; else echo "ok cargo-mutants names: no stage-1 container"; fi
|
||||
# a name the range does not produce is an error that does not echo the name
|
||||
printf '%s\n' 'crates/x/src/lib.rs:99:9: replace no_such_secret_name -> i32 with 0' >"$cm_names"
|
||||
cm A "NAMES=$cm_names"
|
||||
[ "$cm_rc" != 0 ] || { echo "FAIL cargo-mutants names-unknown: exit 0"; fail=1; }
|
||||
cmcheck names-unknown 'ERROR 1 mutants selected' '!no_such_secret_name' '!MUTANTS'
|
||||
rm -f "$cm_names"
|
||||
# killed: a job the watchdog already killed starts nothing and leaves nothing
|
||||
cm_k=$(mktemp); cm A "YGG_KILLED=$cm_k"; rm -f "$cm_k"
|
||||
cmrc killed-before 124
|
||||
if grep -q '^run ' "$cm_fd/log" 2>/dev/null; then echo "FAIL cargo-mutants killed-before: a container started"; fail=1; else echo "ok cargo-mutants killed-before: no container started"; fi
|
||||
cmclean killed-before
|
||||
# the flag appears while stage 1 runs: stage 2 never starts
|
||||
cm_k=$(mktemp -u); cm A "YGG_KILLED=$cm_k" "FAKE_CM_KILLS=$cm_k"; rm -f "$cm_k"
|
||||
cmrc killed-during 124
|
||||
cmcheck killed-during 'JOB KILLED' '!MUTANTS'
|
||||
if grep -q 'run job1-s2' "$cm_fd/log"; then echo "FAIL cargo-mutants killed-during: a stage-2 container started"; fail=1; else echo "ok cargo-mutants killed-during: no stage-2 container"; fi
|
||||
cmclean killed-during
|
||||
# the launcher refuses bad input before any job starts, and never echoes the value it refused
|
||||
cmrepo A
|
||||
for v in 'SHARDS=0' 'SHARDS=x;SECRET' 'FILTERS=nopairSECRET' 'CONFIRM_TIMEOUT_MULT=2;SECRET'; do
|
||||
lo=$(env "$v" bash "$here/cargo-mutants.sh" "$cm_repo" HEAD HEAD~1 "$cm_wd" 2>&1); lrc=$?
|
||||
if [ "$lrc" = 2 ] && ! printf '%s' "$lo" | grep -q SECRET; then echo "ok cargo-mutants launcher refuses ${v%%=*}"; else echo "FAIL cargo-mutants launcher ${v%%=*}: exit $lrc, output: $lo"; fail=1; fi
|
||||
done
|
||||
lo=$(bash "$here/cargo-mutants.sh" "$cm_repo" HEAD~1 HEAD "$cm_wd" 2>&1); lrc=$?
|
||||
if [ "$lrc" = 2 ] && printf '%s' "$lo" | grep -q 'not an ancestor'; then echo "ok cargo-mutants launcher refuses a base that is not an ancestor"; else echo "FAIL cargo-mutants launcher base: exit $lrc, output: $lo"; fail=1; fi
|
||||
|
||||
# wait-on.sh: pid, log and file forms, and the vanished-process exit
|
||||
W=$here/wait-on.sh
|
||||
wo() { local rc; WAIT_MAX=${WM:-4} bash "$W" "$@" >"$T/wo.out" 2>&1; rc=$?; echo "$rc"; }
|
||||
|
|
|
|||
Loading…
Reference in a new issue