Under CPU starvation (16 burners on 8 logical CPUs), all 5 rounds failed
at iteration 0 or 1 with "256 of 256 pollers returned before the close
began, so they were never in the race". The fixture contradicted itself:
the asserts-on settle waits up to 30s (plus a 50ms remainder) for every
poller to be counted inside the library, but each poller's own
next_event wait used a 5s timeout, so the first pollers timed out for
real while the settle was still waiting on the last ones. The race never
ran; the fixture just gave up on itself.
kPollTimeoutMs is now derived from the settle budget it must outlast in
every build configuration (asserts-on's counted deadline plus its fixed
remainder, and the no-seam #else's longer fixed window), named side by
side so they cannot drift apart again. A long poll timeout costs nothing
on a passing run, since the close is what actually ends each wait.
Also give the fixture's own failure to settle a distinct report. If
counted_deadline expires before every poller is counted, CloseRaceOnce
now says "fixture could not settle: N of M pollers counted inside after
30000 ms" instead of proceeding to a close that blames the pollers for a
race the fixture never set up.
holdclose, latecall and holdtimeout use the same counted-deadline idiom
but are not the same defect: their pollers are parked behind
cultmesh_quic_debug_hold_calls, so they cannot time out for real while
the settle waits, and were left alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Self's ruling of 2026-09-22 named six findings against Soul's eighth pass.
This lands the code side of all six:
- The wait-record seam (CULTMESH_QUIC_DEBUG_RECORD_WAIT) now wraps the whole
duration handed to `wait_for`, not the raw `timeout_ms` before it is
converted to milliseconds, so arithmetic applied after that conversion can
no longer live outside what the seam records. Its "unset" sentinel moves to
a separate `debug_wait_recorded` flag so a recorded -1 could never be
confused with nothing recorded.
- `pollhammer` also runs against a runtime holding a live connection to an
unroutable address (RFC 5737 TEST-NET-1), a best-effort stream on it, and a
real (never-connected) listener, closing the gap where the wait's predicate
was only ever exercised on an empty runtime.
- A new `zerotimeout` scenario pins that `timeout_ms <= 0` returns within
50 ms, probing 0, -1 and INT32_MIN.
- `closerace`'s settle no longer sleeps a guessed duration: where the
development seam exists it waits for `debug_peak_calls` to reach the poller
count (reset before any poller thread exists, so nothing can zero a count a
poller already raised) before a short fixed margin closes the last gap
between being counted and being inside the condition wait.
- `kGenerousLateToleranceMs` narrows from 800 ms to 400 ms so the wall-clock
scenarios can catch arithmetic applied outside the seam's own argument,
which `waitseam`'s equality check structurally cannot see.
scripts/mutate-cultmesh.mjs gains a plain `release` configuration on both
platforms (CULTMESH_QUIC_DEBUG_ASSERTS=OFF, CMAKE_BUILD_TYPE=Release) running
`polltimeout`, `pollbusy`, `pollhammer` and `zerotimeout`, and seven new
mutation entries: R1 (the release macro divides the duration by 10), W1-W3
(600 ms added, a 500 ms floor, and a 1000/909 scale, each applied outside the
seam's own argument), P1 (the idle predicate also ends on a live connection),
and Z1-Z2 (a 500 ms sleep and a 300 ms spin on a non-positive timeout). Two
stale comments are corrected: the scenario list above `assertScenarios` now
names `pollhammer` among the clock-sensitive entries, and a historical "1300"
probe value that no longer matches the test file is removed. The header's
section 7 intro now says the seam serves both the quiesce and the wait
recording, not quiesce alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Self's ruling of 2026-09-22 called it after seven Soul passes: a wall-clock
probe against cultmesh_quic_next_event's timeout can never tell the host's
argument from a clamp, floor, round, scale or later-poll fault that happens to
be the identity at the probe's own value, because there is always another
function that matches. Move the observation to where the rule is decided.
Add cultmesh_quic_debug_last_wait_ms / _reset_last_wait_ms, built exactly like
the existing hold seam: gated behind CULTMESH_QUIC_DEBUG_ASSERTS, absent from a
shipped build. CULTMESH_QUIC_DEBUG_RECORD_WAIT wraps the literal timeout_ms
handed to the condition wait as a pass-through, so what is recorded and what is
actually waited on are the same evaluation and cannot diverge except by
changing what the macro wraps.
Two scenarios:
- waitseam asserts the recorded wait equals the host's own argument, exactly,
over 0, 1, 15, 16, 39, 40, 999, 1000, 1001, 7300 and INT32_MAX (the header
states no narrower maximum), and over 40 polls on one runtime past the point
every prior scenario stopped asking. timeout_ms <= 0 never reaches the wait
at all, so its probe checks the honest converse — the seam stays untouched —
which is what catches a seam that records the raw argument on every call
instead of only the value it actually hands its wait.
- pollhammer runs one thread hammering every gate-touching entry point that
cannot itself queue an event, in a yield loop, against another thread's idle
poll, once on a clean runtime and once after a refused listener_open has
left runtime->error non-empty. The idle poll must still stay for its
timeout.
polltimeout, pollbusy and holdtimeout stay as generous-margin proof that the
recorded wait is really waited, not as the precision instrument they used to
have to be: the fastest-of-ten mechanism, the 15 ms probe's tight tolerance,
and the stated-limits table they rested on are gone, because waitseam now
kills every clamp, floor, round, scale, offset and later-poll mutant by
equality regardless of timing. The Windows priority raise these three
scenarios carried is gone with them: a flat 800 ms late tolerance already
covers the worst overshoot the old table recorded (127 ms at normal priority
under load), so raising the process above the load bought nothing the new
margin does not already have. holdtimeout's long probe moves to 1600 ms, above
the stated 1500 ms ceiling on the hold guard, with the hold kept armed to
3000 ms so a guard that parks a naturally-timed-out call anyway overshoots by
well more than the margin.
Verified locally on win32-x64: waitseam and pollhammer pass against the
unmutated bridge; a seam-loosening mutant (recording the raw argument outside
the wait's own branch) fails waitseam's 0 ms probe; an inline +25 ms offset on
the wait's argument fails waitseam by exact mismatch; the S5 hold-guard mutant
(timeout_ms > 1500) fails holdtimeout by 1.4 s overshoot; K1 (++wakes > 64), K2
(!error.empty()) and a no-predicate revert all fail pollhammer's idle poll
within tens of milliseconds. All were hand-reverted and diffed clean against
the working tree before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Soul's sixth pass (N4): `(woken || timeout_ms > 1000) && HELD` survived both
targets. The two existing hold entries only rearrange the guard's syntax,
and `holdtimeout`'s single 150 ms poll sits below any threshold such a guard
would carry.
`holdtimeout` now runs two idle polls under the armed hold, 150 ms and
1300 ms, and holds for 2200 ms. The new entry is that guard. Compiled in on
both targets, the 1300 ms poll is parked until the release and returns at
about 2200 ms, and every other scenario stays green. The harness kills it
on both targets.
With the two polls running together, the first poll's exit also wakes the
second. That gives the asserts build a second multi-threaded witness for the
predicate: the N1 revert and read-what-woke mutants die here as well as on
`pollbusy`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul's sixth pass (N1): every CallScope destructor wakes every waiter, so
without the predicate a poll leaves whenever another host thread makes any
call. Every scenario had a single-threaded host, and dropping the predicate
survived the whole matrix on both targets.
`pollbusy` is a public-API scenario and needs no development seam. A second
host thread calls `cultmesh_quic_last_error` every 20 ms for twice the
poll's timeout while an idle `poll(1000)` waits. It checks the timing first.
It then checks the other thread's call count, so a poll that passed without
ever being woken reads as the fixture failing.
Three harness entries:
- revert: no predicate, `woken` taken from `cv_status`;
- loosening: the wait reads what woke it and does not wait again;
- loosening: a hand-written loop that restarts the whole timeout on every
wake. Its lateness is a function of the input.
Each was compiled in and run through every scenario on both targets:
- revert and read-what-woke: `pollbusy` returns at 0 ms;
- re-arm: `pollbusy` returns at about 3000 ms.
The earlier scenarios all stay green under all three. The harness kills all
three on both targets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Soul's sixth pass (N2): with probes at 150, 200 and 5000 ms, `min(t,5000)`,
`min(t,4990)`, `t*11/10` and `t+70` survived both targets, `max(t,220)`
survived linux-x64, and `max(t,250)` and `t+80` died by 22 and 2 ms. "5000 is
above any plausible ceiling" was wrong: 5000 is a round ceiling.
- A 15 ms probe, asked ten times with the fastest checked against 30 ms. It
is the only place a floor or an added constant shows. Scheduler delay only
adds, so noise has to delay all ten attempts to fail the bridge.
- The long probe is 7300 ms, which is not a round number.
- A single poll gets 60 ms plus a sixteenth of the timeout.
- Each timed scenario prints its worst overshoot per probe.
- Three harness entries: a round clamp `min(t,5000)`, a floor
`max(t,100)` and an offset `t+100`.
Stressed with two busy threads per logical CPU (16 on 8), five rounds of
every timed scenario on both targets. At normal priority, Windows put a
single poll up to 127 ms late and the fastest of ten 15 ms attempts up to
87 ms late, and a 45 ms tolerance failed the unmutated bridge in four rounds
of five. The timed scenarios therefore raise their own process to
HIGH_PRIORITY_CLASS on win32-x64, which needs no privilege. With that, the
worst single-poll overshoot was 15 ms on win32-x64 and 41 ms on linux-x64,
at normal priority there. The fastest 15 ms attempt was 0 ms late on both.
The limit in source is measured, not asserted. Each boundary mutant was
compiled in and run on both targets:
- `t+25` survived and `t+40` died;
- `max(t,40)` survived and `max(t,50)` died, on linux-x64 by 5 ms;
- `t*106/100` survived and `t*108/100` died;
- `min(t,7285)` survived and `min(t,7275)` died.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last pass's only non-comment source change was held by nothing. Reverting the
hold's guard so it fires unconditionally survived all five scenarios, because no
scenario ever had a call in both states at once: the timeout scenario never arms
the hold, and the hold scenarios never let a timeout expire.
`holdtimeout` is the missing state. The hold is armed, the runtime is idle, and
one poll asks for 150 ms while the hold stays armed for 800. A call the wait woke
is the seam's to park; a call whose own timeout expired is on its way out, and
parking it would let `holdclose` report the bridge keeping a host call inside the
library when what kept it was the fixture. The measurement is which of those two
numbers the poll comes back at.
Two entries, and the loosening is the one worth reading: the same two conditions
in the other order. The hold blocks, so the order is the rule — asked first, it
parks the call and then decides whether it should have. Both die on
`holdtimeout` at about 800 ms and on nothing else; `holdclose` and `latecall`
stay green under both, which is the point.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two bands ruled out a constant and ruled out nothing else. Any mapping that
is the identity at 200 and 900 passed for free, and the late tolerance was a flat
300 ms, so anything within that of the truth passed at every value forever.
Three ordinary ones survived all five scenarios on both targets: clamping the
wait to at most a second, adding 150 ms to it, and scaling it by six fifths.
The clamp is the one that matters. Its failure is verbatim the bug the last pass
claimed to fix — a host asking for five seconds gets one, forever, with nothing
reporting it — and "cap the wait so a shutdown gets noticed" is the most ordinary
spelling that line will ever be given.
So the long probe goes to 5000, above any ceiling a bridge would plausibly clamp
to, which kills the clamp and the scale on the early check and the late one. And
the late tolerance becomes a fraction of what was asked for — 60 ms plus an
eighth — which is what kills the offset: 150 ms is 75% late at the 200 ms probe
and there is no flat gap left to hide in. Measured overshoot on either target is
about 10 ms, against 85 ms allowed at the short probe and 685 at the long.
The table gets the clamp as an entry of its own, because the whole lesson here is
that a table of constants says nothing about a wait computed from the argument,
and a table of constants would go green again the moment these numbers drifted
back. Its honest limit is stated where the tolerance is defined: a scaling
smaller than an eighth survives, and nothing here claims otherwise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three rules the bridge has always promised, none of which anything committed
could see.
The timeout was the structural one. Both scenarios end their waits with a close,
so the duration the bridge waits on governed nothing either of them observes: a
hard-coded 2000 ms survived the whole Linux matrix and Windows, and a host asking
for five seconds would have got two forever with nothing reporting it. The fix is
not another entry at another constant. A timeout's only observable is how long
the call takes, so `polltimeout` lets an idle runtime's poll be ended by its own
timeout and measures it, twice at different values, because no single constant
sits in both bands. The seam's hold moves behind that too: it parks only a call
the wait woke, so a poll whose own timeout expired is on its way out rather than
held there by the fixture and counted as the bridge's doing.
The refusal during a close (the header's section 4) had no scenario at all. A
call admitted after the closer has begun increments the in-flight count behind a
wait that has already read it. `latecall` holds the close open with the seam and
makes a call into it.
And the two-phase poll was written off as needing a listener, a credential and a
connection. It needs a closed loopback port: the failure always carries a reason,
which is an 87-byte payload, in about a millisecond. Moving the pop above the
copy crashes on both targets.
18 killed, 0 survived, 0 skipped on linux-x64; 16 killed, 0 survived on
win32-x64, where the two ThreadSanitizer entries are honestly skipped. Both runs
restored the source to the same sha256, b163cab7d48ebc93.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image the bridge is built and mutated in existed nowhere in the repository,
and the recipe that stood in for it installed a C++ toolchain and no JavaScript
runtime — so the container could not run the harness that checks the bridge it
had just built. It is a file now, pinned to the same debian:13 digest as the
release workflow, carrying ninja, Node, setarch and MsQuic's four runtime
dependencies.
The harness configures with `-G Ninja` on linux-x64, which is what
scripts/build-quic-native.sh uses. CMake's Linux default is Makefiles, the image
carries ninja rather than make, and the default configured nothing at all: the
first run in the committed container died in cmake. It said so, because the
control now prints what the child said, which is the whole point of that change.
The seccomp requirement is stated where someone hitting it will read it — the
bridge's README, beside the recipe — rather than being knowledge the runner
assumes its caller already has.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The closerace scenario pinned the order of the count and the notify but never
that a host call is counted at all. Deleting the call scope from
`cultmesh_quic_next_event` outright leaves every configuration green — the
pollers finish before the teardown reaches them — and so does inverting the
comparison that makes a positive timeout block, which empties the library before
the close even begins. A scenario that only sleeps and hopes cannot tell any of
that apart from the real thing.
So the bridge says it itself, in development builds only (section 7 of the
header, behind CULTMESH_QUIC_DEBUG_ASSERTS, absent from every shipped binary):
the peak number of host calls counted inside the library, the number counted
when the close began its wait, and a hold that parks a call inside the library
until the scenario releases it. A held call returns the 0 an empty queue would
have and touches the runtime no further, so a bridge that frees the runtime
under it is reported by the scenario rather than crashing inside it.
`holdclose` drives the quiesce through that seam: the calls are counted, they
were still counted when the close began, and the close did not return while they
were held. `closerace` keeps hunting the race and now also refuses to pass when
a poller left before the close began, which is a scenario that raced nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bridge's close path had no committed test. Whether a close is safe against
host threads already inside the library is the one thing about it a host cannot
establish by reading, and every check of it so far lived in a session scratchpad
and is gone.
The scenario runner under tests/ drives that path and nothing else: it parks
pollers in cultmesh_quic_next_event, closes the runtime under them, and asserts
that the close is what woke them. It consumes the bridge through the public
header, as a host does. CULTMESH_QUIC_BUILD_TESTS builds it, off by default,
because a delivery build has no business producing an executable.
CULTMESH_QUIC_DEBUG_ASSERTS, also off by default, makes the bridge state the
quiesce invariant itself rather than leaving it to whatever a sanitizer happens
to catch. Deleting the quiesce wait is otherwise falsifiable only by luck:
AddressSanitizer saw it 4 times in 48 attempts with pollers blocked mid-copy of
large frames, and never on release. With the assertion and pollers blocked it
fails on the first close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
