Verify containers share cargo's package-cache lock: mount the shared volume as CARGO_HOME #43

Merged

eureka merged 3 commits from eureka-substrate/mutants-sccache into main

2026-11-03 01:37:08 +00:00

eureka commented

2026-11-03 00:19:34 +00:00

Owner

Cause: not sccache. The verify containers mount the shared volume eureka-cargo-registry at /usr/local/cargo/registry, but CARGO_HOME=/usr/local/cargo, so cargo's package-cache lock ($CARGO_HOME/.package-cache) lives in each container's own layer. Sibling containers (cargo-mutants stages, concurrent ygg-verify jobs) therefore hold no common lock. Cargo unpacks a crate by removing a directory that lacks .cargo-ok and extracting again, so one container removes a directory another is unpacking or building from. Registry dependencies compile with the package root as cwd, so rustc then fails to spawn ("could not execute process ... (never executed)", ENOENT).

Reproduction, Yggdrasil, eureka-verify-rust:b99d30ec441e, 6 containers started together on one 17-crate project, an empty throwaway registry per trial, 3 trials per configuration (18 containers each):

registry-only mount, wrapper on (as now): 15 of 18 failed
registry-only mount, RUSTC_WRAPPER=: 15 of 18 failed (the wrapper is irrelevant)
volume mounted as CARGO_HOME, wrapper on: 0 of 18 failed
The reproduced symptom is the unpack-time form (failed to unpack package, failed to open .../.cargo-ok); the compile-time ENOENT in the live logs is the same directory removal seen by a container already building. One live symptom, a dep-info file missing in a container's own target dir, is not explained by this and is not reproduced.

Lock test: a container holding an flock on $CARGO_HOME/.package-cache of the shared volume made a second container's cargo fetch print "Blocking waiting for file lock on package cache" and wait for it.

Fix at the owner: tools/stopgap/ygg-verify.sh and tools/cargo-mutants.sh mount eureka-cargo-home at /cargo-home and set CARGO_HOME and PATH (restating the image's, since cargo's bin stays in /usr/local/cargo/bin) for the rust image. The retry without RUSTC_WRAPPER, its NOTE and diag plumbing and its two tests are gone; a failed build is red. Container errors keep a 25-line tail.

Cost once: the new volume starts empty, so crates are downloaded again on demand (the old eureka-cargo-registry holds 1.26 GB and is no longer mounted; remove it when convenient). tools/mutation-table.sh has the same registry-only mount and is not changed here.

Proof: a one-file-pair scoped cargo-mutants run through ygg-verify with sibling stage containers on the cold volume, green, 14 mutants tested.

🤖 Generated with Claude Code

Cause: not sccache. The verify containers mount the shared volume `eureka-cargo-registry` at `/usr/local/cargo/registry`, but `CARGO_HOME=/usr/local/cargo`, so cargo's package-cache lock (`$CARGO_HOME/.package-cache`) lives in each container's own layer. Sibling containers (cargo-mutants stages, concurrent ygg-verify jobs) therefore hold no common lock. Cargo unpacks a crate by removing a directory that lacks `.cargo-ok` and extracting again, so one container removes a directory another is unpacking or building from. Registry dependencies compile with the package root as cwd, so rustc then fails to spawn ("could not execute process ... (never executed)", ENOENT). Reproduction, Yggdrasil, `eureka-verify-rust:b99d30ec441e`, 6 containers started together on one 17-crate project, an empty throwaway registry per trial, 3 trials per configuration (18 containers each): - registry-only mount, wrapper on (as now): 15 of 18 failed - registry-only mount, `RUSTC_WRAPPER=`: 15 of 18 failed (the wrapper is irrelevant) - volume mounted as `CARGO_HOME`, wrapper on: 0 of 18 failed The reproduced symptom is the unpack-time form (`failed to unpack package`, `failed to open .../.cargo-ok`); the compile-time ENOENT in the live logs is the same directory removal seen by a container already building. One live symptom, a dep-info file missing in a container's own target dir, is not explained by this and is not reproduced. Lock test: a container holding an flock on `$CARGO_HOME/.package-cache` of the shared volume made a second container's `cargo fetch` print "Blocking waiting for file lock on package cache" and wait for it. Fix at the owner: `tools/stopgap/ygg-verify.sh` and `tools/cargo-mutants.sh` mount `eureka-cargo-home` at `/cargo-home` and set `CARGO_HOME` and `PATH` (restating the image's, since cargo's bin stays in `/usr/local/cargo/bin`) for the rust image. The retry without `RUSTC_WRAPPER`, its NOTE and diag plumbing and its two tests are gone; a failed build is red. Container errors keep a 25-line tail. Cost once: the new volume starts empty, so crates are downloaded again on demand (the old `eureka-cargo-registry` holds 1.26 GB and is no longer mounted; remove it when convenient). `tools/mutation-table.sh` has the same registry-only mount and is not changed here. Proof: a one-file-pair scoped cargo-mutants run through ygg-verify with sibling stage containers on the cold volume, green, 14 mutants tested. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

eureka added 1 commit

2026-11-03 00:19:34 +00:00

cargo-mutants.sh: a failed build under the image's rustc wrapper retries once without it e1e7962238

Stage containers sharing the eureka-sccache volume died mid-build with cargo's
"could not execute process ... sccache ... (never executed)" or a missing dep-info
file in their own target dir, though the image's sccache is present, executes, and
a lone container builds clean. The wrapper only caches, so a failed build with
RUSTC_WRAPPER set is retried once without it; the job summary carries a NOTE line
saying which way it went, the first attempt's log is kept, and a build dead either
way stays a red baseline (exit 4) with its cause. Container errors now tail 25
lines, not 5. Tests: dead wrapper recovers with NOTE; red with and without the
wrapper stays red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eureka added 1 commit

2026-11-03 00:50:32 +00:00

Verify containers share cargo's package-cache lock: the shared volume is CARGO_HOME, and the wrapper retry goes 83b00d2f54

A registry-only mount at /usr/local/cargo/registry left the package-cache lock under CARGO_HOME=/usr/local/cargo in each
container's own layer, so sibling containers (cargo-mutants stages, concurrent ygg-verify jobs) shared no lock and removed
each other's half-unpacked crate directories. Reproduced on Yggdrasil from an empty registry: 15 of 18 containers failed
with a registry mount, 15 of 18 with RUSTC_WRAPPER= too, 0 of 18 with the volume as CARGO_HOME.

ygg-verify.sh and cargo-mutants.sh mount eureka-cargo-home at /cargo-home and set CARGO_HOME and PATH for the rust image.
Removed: the retry without RUSTC_WRAPPER, its note and diag plumbing, and its two tests and the FAKE_CM_DEADWRAP switch.
Kept: container errors tail 25 lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eureka changed title from ~~cargo-mutants.sh: retry a failed build once without the rustc wrapper~~ to Verify containers share cargo's package-cache lock: mount the shared volume as CARGO_HOME

2026-11-03 00:50:56 +00:00

eureka added 1 commit

2026-11-03 01:17:31 +00:00

CARGO_HOME alone: the image's PATH already holds cargo/bin, and restating it clobbered the test mock's PATH 25b98a9699

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eureka merged commit 8b17ed5ff5 into main

2026-11-03 01:37:08 +00:00

eureka referenced this pull request from a commit

2026-11-03 01:37:09 +00:00

Merge pull request 'Verify containers share cargo's package-cache lock: mount the shared volume as CARGO_HOME' (#43) from eureka-substrate/mutants-sccache into main

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

GameCult/Eureka!43

No description provided.

Rows
Columns