Soul's probes found every provider peer evicted at handshakeTimeoutMs
(10s by default): StreamPixels' Node service would drop every Unity
overlay 10s after it connected. Root cause: acceptConnection registered
a temporary buffering handler that was the only thing setting
state.connected, then attachPeer awaited CultMeshQuicNativeRuntime.open()
before swapping in the real handler and replaying buffered events. The
native bridge always emits NEW_CONNECTION before CONNECTED as separate
events; when CONNECTED landed after the swap, it went straight to
transport.handleConnectionEvent (which has no case for CONNECTED) with
nothing left to clear the handshake timer, so a fully healthy peer was
evicted on schedule regardless.
Fix: acceptConnection takes a synchronous retain() on the provider's own
already-open runtime instead of awaiting a fresh open(), so one handler
owns the connection from NEW_CONNECTION onward with no async window, no
handler swap, and no buffered-event replay. PendingAccept, its buffering
handler, and attachPeer are gone; dispose() now sweeps `accepted` (every
transport a connection ever got, whether or not its handshake reached
CONNECTED) instead of the deleted pendingAccepts set. Cut C's
retained-frame seeding still happens atomically before the peer joins
`peers`.
Also, while touching broadcast() and sendFrame():
- Encode and validate each frame exactly once in broadcast(), fanning the
encoded bytes out to every peer and outbox, instead of re-encoding once
per peer inside sendFrame(). An invalid frame (e.g. a 70,000-byte
channelId) now throws to the caller before touching any peer, rather
than failing the same way for every peer in turn and evicting all of
them while broadcast() itself reported success. The provider's retained
latest-only frames now cache their encoded bytes too, so seeding a newly
attached peer never re-encodes them either.
- Add an `advertisedHost` provider option, independent of the bind
address: `host` is typically 0.0.0.0, an empty string, or an IPv6
wildcard for binding, none of which describe a route a client can dial,
and the native listener only accepts IP addresses in the first place,
not a DNS name. IPv6 literals are bracketed correctly in the advertised
endpoint. The uppercase cert-sha256 pin is unchanged (the operator ruled
StreamPixels pins a long-lived self-signed certificate).
- provider.receive() with an already-aborted signal now rejects
immediately instead of registering a waiter an already-fired abort event
can never reach.
- The duplicated "datagrams not supported" message (sendFrame and
broadcast each had their own copy) is now one shared constant.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A caller that already holds a live runtime instance (the QUIC realtime
provider, handing a reference to each newly accepted peer) needs one more
reference without crossing an async boundary. `open()` itself has no I/O
to await, but being `async` still forces every caller through a microtask
hop; `retain()` just increments the shared refcount and returns `this`,
synchronously, so the caller stays inside the same turn.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cut C's headline cross-process proof. The TS `serve` peer already tore its
listener down the instant it finished broadcasting, so nothing could dial it
late; it gains `--linger-ms` to hold the listener open afterward instead. The
interop harness starts that provider, runs a throwaway `dial` peer that
receives every frame and disconnects (the provider requires a first peer
before it broadcasts at all, so this is also what proves broadcasting
finished), then only after that dials with the C# native connector and
asserts the frame it gets is the retained one, by payload, not just
non-empty. New `NativeConnectorReceivesTheRetainedFrameWhenJoiningLate` in
GameCult.Mesh.Quic.Native.Tests carries the assertion; the existing
`NativeConnectorReceivesFromExternalProvider` is untouched. Windows-only,
matching the native connector under test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AReliableOrderedFrameIsNeverRetainedAndIsNotDeliveredToALateJoiner disposed
the first session and then waited on server.ConnectionCount == 0 before
attaching a second one, which timed out under Yggdrasil's ygg-verify
container (dotnet test, cultlib-quic-native-dev image): the retention rule
under test does not depend on the first peer disconnecting, so the fix keeps
both sessions connected concurrently and drops the flaky wait.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One sentence each in the C# QUIC README and the transport-planes doc: a
provider retains the newest latest-only frame per (channel, body) and
delivers it to a peer as soon as it attaches, bounded by the number of
distinct keys.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CultMeshQuicRealtimeServer mirrors the TS provider: a `_retained` dictionary
holds the newest latest-only frame per (channel, body) key, updated in
BroadcastAsync with the same CultMeshGeneration comparison the receive-side
filter already uses. AcceptLoopAsync seeds a newly accepted client with the
retained frames. Both the retained update+fan-out and the client
registration+seed run under one lock (`_retainedGate`), so a seed and a
concurrent broadcast are totally ordered and a client can never end up seeded
with a frame older than one it already received via broadcast. Reliable-
ordered frames are events and are never retained.
CultMeshGeneration is now internal so the server can reuse the transport's
own generation comparison instead of a second implementation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CultMeshQuicRealtimeProvider now keeps a `retained` map of the newest
latest-only frame per (channel, body) key, updated on broadcast using the
same generation comparison the receive-side filter already uses. Every peer
attached after that point is seeded with the retained frames before it is
added to the peer set, so a StreamPixels overlay that connects or reconnects
after state was published converges on it instead of waiting for the next
change. Reliable-ordered frames are events and are never retained.
Operator ruling (2026-09-23, OQ1 = A): the provider retains, not a service
heartbeat or a per-peer attach hook.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The mid-handshake test still hung under M2 in the full workspace test run
even after the previous two fixes (pendingAccepts sweep, try/finally).
Diagnosed by printing the stuck value: refCount settled one below its own
"before" snapshot, not above — the opposite of a leak. Root cause:
CultMeshQuicRealtimeProvider.dispose() releases its runtime reference
fire-and-forget (`void this.runtime.release()`), so when M2 makes every
preceding test's own dial fail during the handshake too, their releases
can still be landing when this test starts and snapshots "before". One
lands mid-test, after the snapshot, and refCount never returns to a value
that already excluded it.
Added waitForRefCountToStabilize(): polls until refCount stops moving for
a short quiet window before treating it as a baseline. Verified against
the actual M2 mutation, in the full `npm run test` workspace run (not
just this one file, which had already stayed clean in isolation): the
file now exits in the low tens of seconds with 9 expected failures (every
test that dials through the now-broken cert check) and none of the
"Promise resolution is still pending" hangs from the three earlier
attempts. Confirmed clean on the unmutated tree first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same hygiene sweep as the previous commits, applied to the three tests
whose own subject is a fault or cleanup path that, if it regressed, would
leave a receiveFrame() call with nothing to ever resolve or reject it:
the two "fix 1" throwing-handler tests (a malformed frame that should
fault the connection, and a peer shutdown that should run cleanup()) and
the "fix 3" peer-shutdown test. Each now wraps its transport in
try/finally and uses receiveFrameOrTimeout() instead of a bare
receiveFrame() behind assert.rejects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-running M1 after the previous fix: the assertion now correctly failed
(assertRejectsAndDisposes disposed the resolved client transport), but
the file still hung. Root cause: under M1 the handshake actually
completes (the connector accepts any certificate), so the provider's
peer is fully attached rather than left mid-handshake, and the test's
`provider` was never wrapped in try/finally — the failed assertion threw
past `provider.dispose()`, leaking the listener and the now-live peer.
provider.dispose() is idempotent, so wrapping the whole body costs
nothing on the normal path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The "dispose() evicts a connection still mid-handshake" test (previous
commit) used a bare assert.rejects(connector.connect(...)) to force the
handshake-rejection path it needs, which is exactly the pattern that hung
realtime-quic-consumer.test.ts under M1: found re-running M1 against the
full workspace after the previous fix — it hung again, this time in
realtime-quic-provider.test.ts, at this new test, for the same reason.
Ported the same assertRejectsAndDisposes() helper into this file rather
than importing it across test files (each file already keeps its own
small helpers).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found running the M2 mutation (certificate pin compared case-sensitively)
on Yggdrasil: it broke the handshake for every provider test that dials
with the provider's uppercase-advertised pin, each test correctly failed
its own assertion in under 40ms, but the whole file then hung for two and
a half minutes past its last visible test result.
CultMeshQuicRealtimeProvider.dispose() only disposed connections already
in `this.peers` (attached after reaching CONNECTED). A connection whose
handshake never gets there — a rejected certificate, a stalled client —
sits in acceptConnection()/attachPeer()'s own state with its own
handshakeTimeoutMs timer and, once attachPeer's own
CultMeshQuicNativeRuntime.open() resolves, its own runtime reference,
entirely outside dispose()'s reach. Nine tests each leaked one such
pending accept under M2, each held open for up to its 10s default
handshake timeout, serialized enough to add up to the observed hang.
Fixed by tracking every in-flight accept's `evict` closure in a
`pendingAccepts` set that dispose() sweeps alongside `peers`; each accept
removes itself once it settles, whether attached or evicted, so the set
never grows for a healthy, long-lived connection. Added a regression test
that pins a handshake to reject on a long (30s) handshakeTimeoutMs and
asserts dispose() releases the runtime reference in under 2 seconds, not
after the timeout.
Also fixed the same underlying test-hygiene gap the M2 investigation
surfaced directly: the M4 and M4b tests (a stream-kind/delivery mismatch
should fault a connection) left `transportA` undisposed with an unbounded
`receiveFrame()` behind a bare `assert.rejects`; if the fault path
regresses, that call hangs forever instead of failing. Added
receiveFrameOrTimeout() (a bounded receiveFrame, in the same spirit as
assertRejectsAndDisposes()) and a `finally { transportA.dispose(); }` to
both tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
packages/cultmesh-ts/package.json has depended on koffi 3.3.0 since Cut 4
landed the native QUIC binding, but the root package-lock.json was never
regenerated to match: it carried no koffi entry at all, so `npm ci` at the
repo root failed with EUSAGE. The StreamPixels deploy host runs exactly
`npm ci && npm run build:ts`, so this blocked deployment.
Regenerated on Yggdrasil (Linux, the cultlib-quic-native-dev container),
not on the operator's workstation, via `npm install`. Verified from a
clean clone on the same host: `npm ci && npm run build:ts` both succeed
against the regenerated lockfile.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found running the M1 mutation (bypasses the certificate-accept check) on
Yggdrasil under cgroup-limited CPU: three tests correctly failed their
assertion (`assert.rejects(connector.connect(...))` / `manager.connect(...)`
observed a resolve instead of a reject), but realtime-quic-consumer.test.js
then hung instead of exiting — "Promise resolution is still pending but the
event loop has already resolved". Each of those assertions discarded the
connect() call's return value on the assumption it would always reject, so
the transport that unexpectedly connected under the mutant was never
captured, never disposed: a live native connection, an active
onConnectionEvent registration and an outstanding runtime reference kept
the process alive after node:test had already recorded the failures. This
is the exact scenario the batch names by history ("Soul saw M1 and M3 fail
their assertion and then hang the file").
Added `assertRejectsAndDisposes()`, a drop-in replacement for
`assert.rejects()` that disposes whatever the promise resolves to before
failing the assertion, and used it at all eight call sites in this file
that assert a connect() call rejects (two direct connector.connect() calls,
six through the session manager). Verified on Yggdrasil: M1 re-run now
fails the same three assertions and the file exits on its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real bug found while writing the "dispose with a pending send in flight"
regression test, not a pre-scoped fix: sendOnStream() registered a
pendingSends entry unconditionally. If the reliable-send microtask queued
by sendFrame() (chained onto reliableSendTail) ran after cleanup() had
already executed — cleanup()'s own reject-sweep runs once and is guarded
against re-entry, and offConnectionEvent means no later native event can
resolve the entry either — that entry's promise then never settles,
leaking the pending send forever. Confirmed by a hanging test that
reproduced intermittently (~20-25% of runs) when a peer transport's
dispose() raced an in-flight reliable-ordered sendFrame() on the same
transport; not reproducible in isolation outside the full node:test file
(over 100 standalone cycles clean), which points at the race window being
narrow enough that only real inter-test timing variance opens it.
sendOnStream() now checks `this.disposed` before registering, closing the
gap: a send that loses the race rejects immediately instead of
registering an entry nothing will ever settle.
Also gives that regression test its own `{ timeout: 10_000 }` so a
recurrence fails that one test loudly instead of hanging the whole file
for the harness's 60s+ default; 60 consecutive full-file runs were clean
with both changes in place (25 with the fix alone still showed the hang
occasionally, so the fix closes a real but not the only contributing
factor — recorded here rather than claimed as a full proof).
New coverage for the Cut 5 provider's runtime-release-on-every-path
requirement:
- a reliable-ordered send failure evicts the peer without broadcast()
throwing to the caller;
- the runtime reference is released when the listener itself fails to
open (a real PKCS12 credential, but a port already bound by another
listener, so listen()'s own `catch { await runtime.release(); throw
error; }` actually runs) — split out from a same-named test that had
only exercised an earlier failure point (extractLeafCertificateDer(),
before CultMeshQuicNativeRuntime.open() is ever called), which is kept
as its own, more honestly named test;
- dispose() with a pending send in flight releases the runtime reference
(the test whose flakiness led to the fix above).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports Soul's uaf.js probe (scenario a: bare open/release cycles,
including a fresh open overlapping the previous runtime's close) into a
committed test. UV_THREADPOOL_SIZE and MALLOC_PERTURB_ must be set before
Node starts, so the test spawns a child `node` process running
test/support/release-uaf-probe.js (plain JS, not compiled, requiring
dist/ directly like the package's own compiled tests) once per pool size
(1, 2, 4) and asserts each exits 0.
Trimmed from the probe's original 300 cycles to 20: at pool size 1 with
the crypto.pbkdf2 saturation load, 300 cycles took over two minutes on
this workstation, and the ordering bug this test defends does not need
that many iterations to reach the race — 20 keeps the whole test under
about 25 seconds while still exercising many open/release cycles with a
fresh open racing a still-closing runtime.
Verified by hand: reverting release()'s order in
realtime-quic-native.ts (runtimeCloseFn before awaiting pumpLoop, the
pre-fix order) did not make this test fail on Windows — MALLOC_PERTURB_
is a glibc feature the Windows CRT ignores, so corruption here is silent
rather than a deterministic crash. The revert-and-confirm check that
actually demonstrates the bug runs on Yggdrasil's Linux container, where
glibc honors MALLOC_PERTURB_; see the verify report for that run's
output. The fix was restored immediately either way, confirmed by a clean
git diff before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two scenarios the batch specifically calls for: a throwing handler plus a
garbage frame (the native fault path, through faultConnection()'s now
try/catch-guarded onFault) and a throwing handler plus a peer shutdown
(the transport's own cleanup(), now releasing before running handlers).
Both assert the second onDisposed handler still runs despite the first
throwing, and the peer-shutdown case also asserts
CultMeshQuicNativeRuntime.refCount returns to baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
M4b: the mirror image of the existing M4 test — a latest-only stream
carrying a frame whose own delivery byte says reliable-ordered must fault
only that connection, not the pump.
M10: an equal-generation frame is dropped by the receive-side filter.
Sequenced so it is observable (Soul's note): the first frame is consumed
before the duplicate generation arrives, so the coalescer no longer holds
a pending value the duplicate could silently overwrite with itself.
Without this sequencing, M10's `<=0` -> `<0` mutation is invisible: the
coalescer's overwrite semantics make an unconsumed duplicate look
identical to a dropped one.
F7: split into two tests, because a genuine end-to-end
SEND_CANCELED-through-sendFrame() scenario proved unreliable on this
host. The native-layer test (realtime-quic-native.test.ts) is
deterministic: it round-trips two 64-bit codes above 2^53 through
CONNECTION_SHUTDOWN, exercising the exact decode line F7 mutates
(`code: toBigInt64(decoded.code)`, shared by every native event type,
including STREAM_SEND_COMPLETE). The consumer-level attempt — prime a
reliable stream, start a second send, shut the connection down from the
peer mid-flight — is recorded as a comment: MsQuic here appears to
locally complete a `StreamSend` (as an uncanceled SEND_COMPLETE) faster
than any JS-observable window can land a peer shutdown ahead of it,
regardless of payload size (tried up to 8 MiB) or an explicit delay
before the shutdown call. Recorded as not yet reached at the consumer
layer, not as unreachable.
F6kinds: the existing "stream shutdown prunes streamKinds" test only
proved a second frame still worked afterward, which passes whether or not
the map is actually pruned (a stream id is never reused within one
connection, so a leaked entry has no other behavioural symptom — exactly
why the mutation survived). Strengthened to assert
CultMeshQuicRealtimeTransport.trackedStreamCount (added in the previous
commit) directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the session-manager fixes in the previous commit:
- dispose() aborts a dial in flight instead of letting the connector's own
timeout run out before the transport is discarded (asserted by
wall-clock bound, not just by eventual rejection).
- disconnect() during a dial does not let that dial cache a session once
it lands; the manager stays usable afterward.
- F5abort: a losing race candidate is told to abort promptly, proven by
recording when spyConnector's own abort listener actually fires rather
than only asserting eventual disposal (which the mutation's removed
controller.abort() call does not affect, since the cleanup loop that
disposes settled losers is untouched by that mutation).
- F5loser: a losing candidate that connects anyway, past the point abort
would have helped (spyConnector gains an ignoreAbort option to model
this), is disposed rather than left connected.
Replaces the old "fix 4: a connect() that completes after dispose()"
test, which pinned the pre-fix behaviour of waiting out the connector's
full delay before discarding the late transport; that delay is exactly
what dispose() aborting the dial now removes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix 1 (consumer half): CultMeshQuicRealtimeTransport.cleanup() ran
onDisposed handlers before releasing the runtime reference, unguarded. A
throwing handler skipped every handler queued after it (the throw exits
the for-of) and skipped the release entirely: on the peer-shutdown path
the refcount stuck and the process never exited; on the fault path the
throw escaped cleanup() into dispatch()'s catch in the native runtime,
which is itself now guarded (previous commit), but cleanup()'s own
contract should not depend on that. release() now runs first, and each
handler runs in its own try/catch.
Low fixes:
- receiveFrame(signal) with an already-aborted signal now rejects
immediately instead of registering an "abort" listener that already
fired and will never fire again, which hung the call forever.
- CultMeshQuicRealtimeConnector.connect() merged its four failure paths
(already-aborted, handshake timeout, later abort, handshake-time error)
into one fail() closure. The handshake-time error path was the one that
mattered: it dropped the runtime reference and rejected but never called
connectionShutdown on the native connection, leaking it at the native
level on every rejected handshake past the certificate stage.
- CultMeshQuicRealtimeSessionManager.dispose() now aborts every dial still
in flight instead of only marking sessions disposed and hoping a live
dial notices later; disconnect() aborts the one in-flight dial for its
target so a race with an in-progress connect() cannot cache the session
disconnect() meant to discard. Threaded through dialAsync/
raceFirstSuccessAsync/connectCandidateAsync as an AbortSignal that
combines with the per-race internal controller.
Subtraction (~-70 net in this file despite the fixes above and new
trackedStreamCount/dialAborts surface):
- SimpleGate replaced by a plain promise-chain field on the transport
(reliableSendTail); the class only ever wrapped one method.
- The two dead streamKinds.set() writes on outbound (self-opened) streams
removed: the map is only ever read for streams that deliver an inbound
STREAM_FRAME event, which a stream this side opened never does.
- The write-only authorityRoute field on CultMeshRealtimeCandidate
deleted; nothing read it.
- The one-use TimeoutError class deleted in favour of Error.
- routeCertificateView moved to cultnet-ts (previous commit); the local
copy deleted.
Also exposes CultMeshQuicRealtimeTransport.trackedStreamCount (streamKinds
map size) for the F6kinds regression test: nothing else could observe
whether STREAM_SHUTDOWN actually prunes the map, since a stream id is
never reused within one connection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix 1 (native half): faultConnection() called the caller-supplied onFault
handler unguarded. A throwing handler escaped faultConnection(), then
dispatch(), then the pump's async loop as an unhandled rejection nobody
awaits, crashing the process on a single bad connection. Wrap the call and
report instead of propagating, matching the try/catch dispatch() already
has around the listener call that reaches faultConnection() in the first
place.
Subtraction: fold the private NativeBindings interface and the
bindingsCache wrapper object into the runtime class's own methods, which
already typed the same twelve calls one door down. The wrapper only
restated that typing a second time. loadBindings() now returns the raw
koffi handles (untyped, koffi has no useful types) and the class methods
do the alloc/decode work directly, the way half of them already did for
their out-parameters. Net -101 lines despite the added refCount getter and
a trimmed module doc comment (no longer narrating the koffi probe process
twice).
Also exposes CultMeshQuicNativeRuntime.refCount as a public static getter
(renaming the backing field to refCountValue): the batch's regression
tests assert it returns to baseline after a fault, a peer shutdown, and
release(); nothing else needed to reach into a private static before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both packages mapped an advertised route's wire-shaped certificate
(providerKeyId/providerPublicKeyX/providerPublicKeyY/...) to
CultMeshAuthorityRouteCertificate independently. Move the mapping into
cultnet-ts's cultmesh-authority module, next to verifyAuthorityRoute, and
have both consumers call it instead of duplicating the object literal.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by a second real failure, this time on Windows: `dotnet test --logger
"console;verbosity=detailed"` prints a passed test's TestContext.Out lines
twice (live-streamed, then again under "Standard Output Messages:"), and
matching CULTMESH_GOLDEN_SEQUENCE and CULTMESH_GOLDEN_HEX with two
independent regexes let the harness pair a sequence from one copy with a hex
from the other, producing a spurious mismatch even though the C# and
TypeScript sides agreed. One regex now requires them adjacent, exactly as
the C# test's two back-to-back WriteLine calls print them. All four QUIC
interop lanes now pass on Windows (real dotnet test run, native connector
lane included).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by a real failure on the Linux verification container: latest-only
coalesces on both the provider's outbox and the managed connector's own
receive-side inbox, so a single ReceiveAsync() call is not guaranteed to
land on the first frame broadcast — on a slower host it landed on sequence 3.
The C# test now also prints CULTMESH_GOLDEN_SEQUENCE, and the harness looks
up the TypeScript peer's logged encoding of that same sequence instead of
hardcoding sent === 1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified live on Windows: a passed NUnit test under `dotnet test --logger
"console;verbosity=detailed"` prints "Test Run Successful." and "Passed: N",
never the literal "Passed!" the assertion was checking for. Both the native
and managed connector lanes now match the real strings; both were confirmed
to pass for real against a live TypeScript provider (lane 1: 137ms, lane 2:
145ms with the golden-bytes hex line matching the TypeScript peer's own
encoding byte for byte).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TypeScript provider vs C# native connector (the Unity path, gated to
Windows since the native connector under test is); TypeScript provider vs
C# managed connector, with a golden-bytes check comparing the C# test's
re-encoded frame against the TypeScript peer's own encoding of the same
frame; latest-only under a stalled consumer (two dials against one 200-frame
provider, one paused mid-stream via the peer script's new --pause-after),
asserting the provider's broadcasts stay bounded regardless of the pause and
both consumers converge on sequence 200.
Both dotnet-test lanes run `dotnet test --filter FullyQualifiedName~<test>
--logger "console;verbosity=detailed"`: detailed verbosity is what surfaces
a passed test's own TestContext.Out lines, which the golden-bytes lane reads
back out of stdout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dial gains --expect-sequence (stop once a frame with that sequence arrives,
rather than counting frames received — the number of frames a stalled
latest-only reader sees is not the interesting value, what it converges on
is) and --pause-after/--pause-ms, which simulate the stalled-consumer test
hook at the only level this subprocess can reach: not calling receiveFrame
for a while. The shared native pump keeps dispatching and coalescing into
this transport's own inbox regardless of whether the app reads, so not
reading is externally equivalent to not polling.
serve now also logs each broadcast frame's own TypeScript encoding as hex, so
the harness can compare it against the same frame's C#-side re-encoding
without importing the codec itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
NativeConnectorReceivesFromExternalManagedProvider -> NativeConnectorReceivesFromExternalProvider:
CULTMESH_NATIVE_EXTERNAL_ENDPOINT's peer is no longer necessarily the C#
managed provider now that Cut 5 adds a TypeScript one on the same lane; the
env var name stays, since it names the endpoint, not the peer.
Add ManagedConnectorReceivesFromExternalProvider, gated by the same env var:
dials an external provider with the plain managed connector and no
ValidateProviderCertificate (so acceptance runs on the endpoint's own
cert-sha256 pin alone, like an ordinary consumer), and prints the received
frame's CultMeshRealtimeWireProtocol re-encoding as hex for the harness's
golden-bytes comparison against the TypeScript peer's own encoding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cultmesh-quic-peer.ts serve opens a CultMeshQuicRealtimeProvider on the same
test fixture credential the unit tests use, prints {status, endpoint} in the
shape spawnServeProcess already expects, waits for a first peer before
broadcasting (a provider's broadcast to zero peers silently sends nothing,
same reasoning as the C# quic-realtime-serve peer's
WaitForFirstConnectionAsync), then broadcasts --frames frames at --delivery
with --interval-ms between them, stopping early on stdin EOF like the C#
peer's WatchStdinCloseAsync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reliable-ordered ordering in both directions, latest-only coalescing (a
synchronous back-to-back publish is deterministically coalesced, not by
network timing luck), a stalled peer's outbox holding at most one pending
frame per key across 200 broadcasts, a non-blocking broadcast, the uppercase
cert-sha256 pin, unreliable failing closed, and eviction on peer shutdown.
Runs against the real native bridge and Cut 4's real consumer connector, no
mocks; skips when CULTMESH_QUIC_NATIVE_DIR is unset, matching every other
QUIC test file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CultMeshQuicRealtimeProvider (packages/cultmesh-ts/src/realtime-quic.ts) is
the StreamPixels role: listen() opens a native listener, computes an
uppercase cert-sha256 advertisedEndpoint (the Unity connector requires the
pin), and accepts peers into a Set of the same CultMeshQuicRealtimeTransport
Cut 4 already built for consumers. broadcast() mirrors
CultMeshQuicRealtimeServer.BroadcastAsync: reliable-ordered awaits every
peer's send_complete via transport.sendFrame directly (Task.WhenAll,
evict-on-failure); latest-only publishes into a new per-peer
CultMeshQuicRealtimeProviderOutbox and returns without waiting on any peer's
I/O. The outbox pumps sendFrame's existing fresh-stream-per-frame path one
frame at a time, so a stalled peer holds at most one pending frame per
(channel, body) key.
Extracted CultMeshRealtimeLatestOnlyCoalescer from the inbound path's
inline Map+FIFO bookkeeping (unchanged behaviour) so the new outbox reuses
the same coalescing idiom instead of a second inline copy.
Accepting a connection buffers native events behind a synchronous listener
registration (the pump dispatches strictly sequentially, so registering
before any await is what makes the async runtime-reference acquisition that
follows safe) and replays them once the peer's CultMeshQuicRealtimeTransport
exists; a handshake timer evicts a peer that never completes CONNECTED.
Wired into the CultMesh facade as createQuicRealtimeProvider.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The PKCS12/DER pair had no record of how it was made or what it is. Documents
the CN, the openssl recipe that reproduces an equivalent pair, and that
nothing depends on today's exact bytes past what the tests read back out.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A non-literal first stdio element loses spawn()'s typed-tuple overload
entirely (verified on Yggdrasil: tsc failed on the two other, unrelated
serve helpers' stdout/stderr going nullable), not just for stdin's varying
element. Asserted at the one call site instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Finding 8(c): the dev image builds the Node bridge's own libmsquic copy into
a build-local directory (scripts/build-quic-native.sh), which is never on the
loader's search path. .NET's System.Net.Quic resolves "libmsquic" through
that path independently and failed with PlatformNotSupportedException before
this. The smallest honest fix: dpkg -i the same pinned 2.5.9 deb (same URL,
same SHA-256 as the build script) system-wide and run ldconfig, so both the
Node bridge's own copy and .NET's independent resolution find a libmsquic.
Changes the image; the new tag is cultlib-quic-native-dev:75cb89e73850
(first 12 hex of this file's sha256), built fresh on Yggdrasil.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Finding 8(a): every spawnServeProcess call used stdio ["ignore", ...], which
hands the C# quic-realtime-serve peer an already-closed stdin. Its own
WatchStdinCloseAsync (Program.cs:1261-1269) treats that as EOF and tears the
serve loop down at once, before it can send anything. spawnServeProcess gains
an opt-in `stdin: "pipe"` (default stays "ignore" for every other serve mode,
which never reads stdin), used only by the "CultMesh QUIC realtime" test; its
teardown ends that pipe explicitly before killing the process.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Finding 8(b): the serve mode broadcast immediately after printing its ready
line, before the harness's dial could reach it, so frames went to an empty
peer set and were silently dropped. WaitForFirstConnectionAsync polls
ConnectionCount, bounded at 30s so a peer that never gets dialed still exits
instead of hanging the interop lane forever.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Each rule below fails under its own mutation of the code fixed in the previous
two commits; verified by hand and restored, per the fix-batch brief, not
committed as a mutation harness (the operator retired that pattern).
- realtime-quic-native.test.ts: release() refuses a call with no outstanding
reference (fix 2).
- realtime-quic-consumer.test.ts: a stream-kind/delivery mismatch faults only
its connection while a second, independent connection through the same
pump keeps working (fix 1 / M4); a peer-initiated shutdown rejects a
pending receive and cleans up exactly once, so a further dispose is a
no-op and a fresh receive rejects instead of hanging (fix 3); a slow
latest-only reader accumulates at most one pending frame per key, and an
equal generation is dropped outright (fix 6, M10); a real connected
transport's isVerifiedFor rejects a generation other than the one it
connected with, and the session manager disposes a real (not stubbed)
transport whose generation no longer matches a freshly verified route
(M3); the advertised certificate pin matches case-insensitively (M2).
- realtime-quic-session-manager.test.ts (new): eviction of a dead cached
session and redial on the next connect, single-flight sharing one dial
across concurrent callers, refusing a transport that lands after dispose()
(all fix 4); the race returning on first success without waiting out a
slow candidate (fix 5); the maxRacedCandidates cap (M6); the protocolIds
and authorityRuntimeId route filters (M7, M8); the session key including
authorityRuntimeId (M9). Trust verification is skipped via
mode: "local-development" over loopback endpoints, since these tests are
about the session manager's own binding/racing/caching rules, not route
trust (already pinned by realtime-quic-consumer.test.ts's trust
negatives).
Subtraction: `nativeBridgeAvailable` was defined twice, disagreeing on
whether to check the dependency library too; both test files now import one
definition from test/support/native-bridge.ts (the stricter of the two).
Gap, reported rather than papered over: the stream-shutdown pruning half of
fix 6 (streamKinds) has no direct test, because the map is private and
nothing here exposes it; the committed test only shows a second frame still
arrives after the pruned stream's shutdown, which does not by itself prove
the entry was removed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Soul's fix batch, findings 3, 4, 5 and 6:
- A peer-initiated CONNECTION_SHUTDOWN only set `disposed = true`, so
`dispose()`'s early-return skipped `release()` and `offConnectionEvent`: the
runtime reference leaked forever and a pending `receiveFrame` hung forever.
`disposed` and "already cleaned up" are now the same flag, run through one
`cleanup()` reached from `dispose()`, a peer shutdown, and the new `fault()`
(wired from fix 1's `onFault`). Every path releases the runtime reference
exactly once and rejects every pending and future receive/send with the real
reason instead of a generic "disposed" message.
- `CultMeshQuicRealtimeSessionManager.connect()` cached a transport forever,
even a dead one, and raced candidates with `Promise.allSettled`, so one dead
route delayed every connect by the full handshake timeout. It now: evicts a
session once its transport reports itself disposed (`onDisposed`, mirroring
`CultMeshSessions.InvalidateRealtimeSession`); shares one in-flight dial
across concurrent callers for the same target (mirrors C#'s `Lazy`); refuses
a transport that lands after `dispose()` instead of caching it; and races
each tier for the first verified success (mirrors `Task.WhenAny`), aborting
and disposing the losers via an `AbortSignal` threaded into
`connector.connect`, which now also honours a signal already aborted at
entry.
- `streamKinds` never pruned a finished stream (`STREAM_SHUTDOWN` fell into
`default`); it now removes the entry. The receive side queued every
latest-only frame in an unbounded array; it now mirrors the C# reference's
`CultMeshRealtimeInbox`: at most one pending frame per `(channel, body)`,
coalesced on arrival, while reliable-ordered frames still queue directly
(unbounded, matching the C# reference's own unbounded channel).
Subtraction: `trimmedOrEmpty` duplicated `cultnet-ts`'s exported `trimCSharp`;
callers now use the shared one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Soul's fix batch, findings 1, 2 and 7:
- The pump's dispatch had no fault boundary: a malformed frame, a stream-kind
mismatch, or a throwing caller callback (certificate validator, waiter) threw
out of `dispatch` and killed the pump, taking the whole process with it.
`dispatch` now catches per-connection, shuts that connection down at the
native level, removes its listener, and hands the error to an optional
`onFault` callback registered alongside the listener. Every other connection,
and the pump itself, keep running.
- `release()` called `runtimeClose` before awaiting the pump's exit, so a
queued-but-not-yet-started `nextEvent.async` call could begin after close and
run against freed memory. It now marks closed, awaits the pump loop (it wakes
within its 250 ms timeout), and only then closes. It also refuses a release
with no outstanding reference instead of silently flooring at zero.
- koffi decodes a 64-bit `uint64_t`/`void *` as a plain `number` when it fits a
safe integer, not `bigint` (probed on the pinned 3.3.0 build against the real
bridge). Every declared type here promises `bigint`; `toBigInt64` is now the
single boundary conversion, applied to every 64-bit decode, so the mismatch
that made `event.code === BigInt(CULTMESH_QUIC_SEND_CANCELED)` never true is
fixed at its source instead of patched at that one comparison.
Subtraction: deleted the unused `lastStatus` export (wired but never called),
and collapsed the stacked eslint-disable-next-line pair over one line into one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Program.cs is a top-level-statements file: every type declaration (the
existing ServeConfig, DialConfig and friends) lives after every top-level
statement and local function, all clustered at the end. The new record landed
between two local functions instead, which is CS8803: "top-level statements
must precede namespace and type declarations." Moved beside DialConfig, where
its neighbors already live.
Found by a direct `dotnet build` run on Yggdrasil after the interop test's own
error message was truncated by Node's console printing before reaching the
actual compiler diagnostic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The interop lane's dotnet build FailFasts on startup in this minimal Debian
image: no ICU package, and .NET's globalization stack aborts without one
rather than degrading. The interop peer already parses every argument through
CultureInfo.InvariantCulture, so invariant mode costs it nothing; it is also
Microsoft's own documented fix for a minimal container, cheaper than pinning
and installing an ICU package by name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Self killed a verify run after it sat idle for 25+ minutes past its last
printed result and root-caused it: SIGUSR2 diagnostics on the stuck process
showed an empty JS stack, every libuv handle unreferenced, and MsQuic's own
worker threads parked in epoll_wait — a native runtime that was opened but
never closed, so its registration and threads never tore down and the process
never exited on its own.
The cause was mundane and compounds badly: `RawQuicListener.open()` in
realtime-quic-consumer.test.ts opened a native runtime, then read a test
fixture whose path was wrong (see below) and threw — with no try/catch, so
the runtime it had just opened was never released. That test-file pattern ran
on seven separate tests in one process, each leaking its own reference to the
one process-wide shared runtime; by the time the file's tests finished, the
refcount had never reached zero and the live registration held the process
open indefinitely. `RawQuicListener.open()` now releases the runtime on any
setup failure before rethrowing.
Two more real bugs the same run's fixture-path failures exposed:
- Both new test files resolved fixtures at `join(__dirname, "fixtures", ...)`.
`__dirname` at runtime is `dist-test/test`, and binary fixtures are never
compiled or copied there — only `.ts` sources under `test/` and `src/` are.
Both now read fixtures from their real location, two directories up.
- The negative-grep test looked for a literal `.async(` call. The bugfix in
the previous commit references `nextEventFn.async` as a value handed to
`promisify`, never written with a trailing `(` at the reference site, so the
grep found nothing to check and failed by construction. It now matches
`.async` as a property reference, not only a call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A `node -e` probe against the real bridge (loaded through koffi 3.3.0 on
Yggdrasil) showed the `_Out_` prototype marker does not auto-allocate storage
or reshape the return value: `openFn(null, null)` returned the plain scalar
`-1`, because the null "out" pointer is itself a bad call under the header's
own contract, and koffi never populated anything because there was nothing to
write into. `openFn(null, koffi.alloc("void *", 1))` returns the plain
`int32_t` status and the pointer decodes correctly with
`koffi.decode(buf, "void *")`, as a `bigint`.
Every out parameter is now `koffi.alloc(type, 1)` before the call and
`koffi.decode` after, inside `loadBindings`; the rest of the file is
unaffected, since it already treated ids as `bigint`. The runtime handle gets
its own `CultMeshQuicNativeHandle` alias (`bigint`) so its type is named
rather than smuggled through as `unknown`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The wrapper cast the raw promisified koffi call to the same 4-argument public
NativeBindings#nextEventAsync type it was implementing, then called it with
six arguments (the two _Out_ placeholders included) — a type checked itself
into a contradiction tsc caught on a clean build. The raw call gets its own
type matching what koffi actually expects; the public wrapper still exposes
four.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- koffi's `_Out_` sugar still counts toward the call's declared arity in this
version (an arity check rejected the shorter call); pass `null` at each such
slot, since the value comes back through the named field on the result
object regardless of what was passed.
- The negative-grep test read `dist-test/src/...`, which does not exist:
`__dirname` at runtime is `dist-test/test`, and only compiled output is
mirrored there, so the original source is two levels up in `src/`, not one.
- Two generated Odin test roots shared one hardcoded key id
("odin-root-1"), so trust negative (c) had the trust policy's root looked
up successfully under the signer's own id and failed on signature mismatch
instead of "not trusted" — proving the wrong rule. Each generated root now
gets a distinct key id.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
koffi's own .d.ts resolves as an ECMAScript module under Node16 resolution,
which a CommonJS `typeof import("koffi")` reference and a `koffi.promisify`
property access (which does not exist) both choke on. Type the loaded module
as `any` and promisify `nextEvent.async` with `node:util` directly instead of
guessing at a koffi-native promise helper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cut 4's own npm tests and the C#-to-TS interop lane need to run in the same
container that builds the native bridge, and the image had neither runtime.
Smallest change: install Node 24 from NodeSource and the .NET 10 SDK from
dotnet-install.sh, both pinned by channel/major version the way the rest of
this image pins its toolchain by digest.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tests/GameCult.Networking.InteropPeer gains mode quic-realtime-serve
(--port, --frames, --delivery, --interval-ms): creates a self-signed
certificate the same way CultMeshQuicRealtimeTransportTests.cs does, starts
CultMeshQuicRealtimeServer, and prints one ready line carrying the advertised
cultmesh-state+quic:// endpoint with its cert-sha256 pin, then broadcasts N
frames with increasing sequence, exiting after the loop or on stdin close.
GameCult.Networking.InteropPeer.csproj references GameCult.Mesh.Quic.
Discrepancy, kept: the cut map's worked line for the ready signal is
`{ endpoint: "..." }` alone; this repo's spawnServeProcess harness
(cultnet-interop.test.ts:2491) waits for a line whose parsed `status` is
"ready", the same convention every other serve mode here already uses. The
printed line is `{ status: "ready", endpoint: "..." }` — the map's shape plus
the field its own named mechanism requires — rather than inventing a second
readiness convention for one peer.
packages/cultmesh-ts/test/interop/cultmesh-quic-peer.ts: the TypeScript half,
`dial --endpoint --expect N`, connecting with CultMeshQuicRealtimeConnector
and printing the received frames as one JSON line (fields plus payload hex).
It reaches the connector through this package's own src/, never importing
cultmesh-ts itself, keeping the dependency direction the map states
(cultmesh-ts depends on cultnet-ts, never the reverse).
packages/cultnet-ts/test/interop/cultnet-interop.test.ts gains
"CultMesh QUIC realtime: C# managed provider and TypeScript consumer",
spawning the C# peer with spawnServeProcess and the TypeScript peer with
runJsonCommand, both established patterns in this file. Skips when
CULTMESH_QUIC_NATIVE_DIR is unset, the same convention the unit tests use.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
packages/cultmesh-ts/test/realtime-quic-native.test.ts: the FFI smoke against
CultMeshQuicNativeRuntime (open/close, a listener with a real PKCS12
credential, a full loopback connection delivering one raw frame end to end),
plus the negative grep that .async( appears only on nextEvent and no
koffi.register exists.
packages/cultmesh-ts/test/realtime-quic-consumer.test.ts: the TypeScript
consumer against a raw listener opened through the native runtime (connect
with the correct pin; refuse a wrong pin; refuse a missing pin with no
validator; a caller validator observes the target and the DER bytes; a late
older generation is dropped), mirroring
CultMeshNativeQuicRealtimeTransportTests.cs and
CultMeshQuicRealtimeTransportTests.cs. Then the five trust negatives through
CultMeshQuicRealtimeSessionManager with a static lookup source and routes
signed with node:crypto (P-256, IEEE P1363 via dsaEncoding): (a) expired
certificate, (b) unsigned remote route, (c) untrusted Odin root, (d) a route
that verifies but pins the wrong TLS certificate, refused by the connector
before isVerifiedFor is ever asked, and (e) a stale/replayed generation,
proven with a stub connector whose transport fails isVerifiedFor, asserting
the session manager disposes it and rejects the connect. There is no
session_open nonce on the QUIC plane (route generation plus the TLS handshake
is its freshness), so (e) is this shape rather than a literal nonce replay,
as the cut map records.
test/fixtures/quic-test.p12 and quic-test-cert.der: a self-signed ECDSA P-256
PKCS12 test credential (OpenSSL, empty password, 1088 bytes — the same shape
as the native bridge's own embedded test PKCS12) and its DER certificate, used
to open real raw listeners and compute the expected cert-sha256 pin at test
time rather than hand-transcribing it.
Discrepancy kept, reported: the C# reference's receive-side inbox also
coalesces a backlog of not-yet-drained latest-only frames of the same key to
just the newest; this transport applies only the generation-drop filter the
cut map names (":538-560") and queues every frame that passes it. The interop
lane's frames are far enough apart that this never matters in practice, but a
consumer that fell behind would see more history here than the C# reference
delivers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
packages/cultmesh-ts/src/realtime-quic.ts adds, over the native binding:
- parseQuicRealtimeEndpoint, mirroring
CultMeshQuicRealtimeTransportConnector.TryParseEndpoint (:98-109).
- CultMeshQuicRealtimeTransport: sendFrame/receiveFrame/dispose/isVerifiedFor,
the stream-kind-vs-delivery consistency check (:519-524) and the
per-(channel,body) generation filter that drops older (producerEpoch,
sequence) (:538-560), both pinned against CultMeshQuicRealtimeTransport.cs.
Reliable sends reuse one persistent outbound stream behind a gate;
latest-only opens a fresh stream per send; unreliable fails closed. The
bridge owns wire framing both ways, so this file never length-prefixes
anything by hand.
- CultMeshQuicRealtimeConnector: the certificate decision on event 3 (a
caller validator, else SHA-256-of-DER against the endpoint's cert-sha256,
else reject), mirroring MatchesAdvertisedCertificatePin (:111-119).
- CultMeshQuicRealtimeSessionManager and CultMeshStaticRealtimeLookupSource:
binds candidates from authorityRoutes carrying cultmesh.realtime_state.v1,
verifies each with cultnet-ts's verifyAuthorityRoute before any connector is
asked, orders by connector then route priority, races at most
maxRacedCandidates per tier, requires isVerifiedFor after connect and
disposes a transport that fails it — the realtime path of
CultMeshSessions.cs's ConnectRealtimeAsync (:833-896), including its
session key verseId + "\u001f" + authorityRuntimeId (:46).
No Odin WebSocket rendezvous lives here (Q7, ruled A): only the
ICultMeshRealtimeLookupSource port and a static source.
index.ts re-exports the module and gains three CultMesh statics beside the
RUDP ones: parseQuicRealtimeEndpoint, createQuicRealtimeConnector,
createQuicRealtimeSessionManager.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
