GameCult/CultLib
Watch
9
0
Fork
You've already forked CultLib
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Merge origin/main into hands/ack-cutD

Browse source
This commit is contained in:
Meta Vens 2026-09-30 10:58:23 +02:00
commit b8c00117b3
111 changed files with 14950 additions and 1648 deletions
Download patch file Download diff file Expand all files Collapse all files

10
contracts/cultnet/cultnet-distributed-database.md
View file

@ -104,6 +104,16 @@ Not implemented yet:
- cross-runtime rollback/resimulation helpers for simulation frames
- declared CRDT merge policies
## Error Codes
`cultnet.error.v0` carries an optional machine-readable `code`. The set of codes is a closed enum
(`cultnet.error.schema.json`): Rust and TypeScript refuse a message whose code they do not know, they do
not guess. Adding a code is therefore a coordinated change across every runtime, made together: the schema (both
copies), the C# `CultNetErrorMessage` factory, the Rust `CultNetErrorCode`, and a captured-bytes case in
`packages/cultnet-rs/tests/error_message.rs` for it. TypeScript reads the schema. The current codes are
`selection_invalid`, `cursor_stale`, `cursor_invalid`, `reference_outside_target`, `variant_unsupported` and
`unowned_schema` (a write, or a remote put or delete, for a schema and key no shard owns).
## Live Invariants
- CultCache owns document schema, identity, local indexes, and local diffing.

8
contracts/cultnet/cultnet.error.schema.json
View file

@ -26,9 +26,9 @@
}
},
"code": {
"description": "Machine-readable refusal code (R-N). Additive: a v0 peer that does not know this field ignores it.",
"description": "Machine-readable refusal code (R-N). Additive: a v0 peer that does not know this field ignores it. The set is a closed enum: Rust and TypeScript refuse a message whose code is not listed, so adding a code is one coordinated change across every runtime (this schema, the C# factory, the Rust CultNetErrorCode, and a captured-bytes case for each).",
"type": ["string", "null"],
"enum": [null, "selection_invalid", "cursor_stale", "cursor_invalid", "reference_outside_target", "variant_unsupported"]
"enum": [null, "selection_invalid", "cursor_stale", "cursor_invalid", "reference_outside_target", "variant_unsupported", "unowned_schema"]
},
"details": {
"description": "Code-specific structured detail (R-N). Every field is populated only for the code that defines it; the others are null, not omitted.",
@ -36,8 +36,8 @@
"additionalProperties": false,
"required": ["field", "value", "asOf", "current"],
"properties": {
"field": { "description": "selection_invalid: the selection field that failed. variant_unsupported: \"subscriptionId\", when a subscription was ended.", "type": ["string", "null"] },
"value": { "description": "selection_invalid: the offending value, when there is one string worth naming. variant_unsupported: the id of the subscription that ended.", "type": ["string", "null"] },
"field": { "description": "selection_invalid: the selection field that failed. variant_unsupported: \"subscriptionId\", when a subscription was ended. unowned_schema: \"schemaId\".", "type": ["string", "null"] },
"value": { "description": "selection_invalid: the offending value, when there is one string worth naming. variant_unsupported: the id of the subscription that ended. unowned_schema: the schema id no shard owns.", "type": ["string", "null"] },
"asOf": { "description": "cursor_stale: the cursor's own minted asOf.", "type": ["integer", "null"], "minimum": 0 },
"current": { "description": "cursor_stale: the answering server's current asOf.", "type": ["integer", "null"], "minimum": 0 }
}

26
docs/cultcache-publication-cut.md
View file

@ -813,4 +813,28 @@ Recorded, not forked:
a gap (`ApplyShardSnapshotResponseAsync` `:675-736`). It predates this campaign.
- **AetheriaEve `Program.cs:3722`.** It reads a committed write's `InvalidOperationException` as a denial. Under
Q-P5 A, Cut 4's log failures cannot reach it. Observer and `OnUpdate` exceptions still can, so the Cuts 1-2
recording stands.
recording stands.- **Cut 4 Soul, recorded 2026-09-30 (Self), not fixed in Cut 4:**
- **File log store is not crash-safe (hard prerequisite).** `CultNetFileShardMutationLogStore.Append` and
`CompactThrough` use `File.WriteAllBytes` (truncate, rewrite, no temp+rename, no fsync). A failure mid-write
truncates the log; every later append burns a sequence and a restart cannot construct the database. It also
rewrites the whole file per append, which Cut 4 now does under the cache gate (2.2 ms per write at 100
entries, 33 ms at 950, a 2.49 s stall). An atomic, append-only store must land before any live daemon
enables durable shard logs. Today only AetheriaEve's offline tools do; the live Aetheria daemon passes
`enableDurableShardLogs: false`.
- **A slow log append blocks cache readers** (`CultCache.Get` takes the gate). Accepted with a note in the
store's XML doc.
- **Crash between the store write and the log append** leaves the log lagging the store, and nothing detects
it; a restarted primary reuses that sequence (A2's divergence reached by a plain kill). The spec called this
lag detectable; for a crash it is not.
- **Cut 4 final Soul (4d), recorded 2026-09-30 (Self), not fixed in Cut 4:**
- **F2, read cost.** `CultNetDatabase.Serves` (`:1980-1989`) runs per row in `GetAll`/`GetByName`/`GetByIndex` for every
layout except a single owns-everything shard: a reflection `GetTrackedKey`, a gate lock and a shard re-scan per row.
`database.GetAll` measured 24.8x `cache.GetAll` on 20,000 rows under a key-prefixed shard. Fix: filter by key inside
the cache enumeration (the `rowFilter` path already has keys) and hoist the owns-everything check.
- **F4, dispose drain edges.** An observer that removed its stash entry before the drain can publish after `Dispose`
returns (dropped); a throwing `_cacheJournal.Dispose()` skips the drain. Neither double-publishes or deadlocks.
- **F5, refusal naming.** A write whose schema is owned but whose key is outside the shard's prefix also answers
`unowned_schema` with details `{field:"schemaId"}`; the peer cannot tell a key refusal from a schema refusal.
- **Pre-existing, separate task.** Python emits error code `unsupported_schema_version`
(`cultnet-py/.../interop_peer.py:415`, `cultmesh-py/.../server.py:161`), which is not in the error-code enum; TS and
Rust refuse the whole message.

51
docs/cultnet-rudp-ack-cut.md
View file

@ -35,10 +35,12 @@ it. One thing moves for them: Cut 2 merges only together with Cut F (the Cut 2 S
fragmented reliable frames (the default Rust `media` channel). Pass 3 absorbs it: Cut F's receiver drops
every incomplete fragment set a notice covers; Cut 2 merges only together with Cut F; C# gets the bound
it never had (Cut 2c). What a receiver does with a sender that never sends notices is **Q-A9**.
- **Open:** Q-A7 (the notice names one sequence, not a cumulative point), Q-A8 (reliable expiry in the
four runtimes that lack it) and Q-A9 (a capability bit, so Cut 2's rule applies only to peers that send
notices). Each has a recommendation. Q-A7 and Q-A9 block Cut F and with it Cut 2's merge; Q-A8 blocks
only Cut 7.
- **Ruled after pass 3 (operator, 2026-09-30), each the recommendation:**
- **Q-A7 A:** a notice names one sequence, not a cumulative point.
- **Q-A8 A:** C#, TypeScript, Python and Kotlin gain reliable expiry (Cut 7, K5).
- **Q-A9 B:** a capability bit on Connect and Accept; Cut 2's never-evict rule applies only to peers
that advertise it, and other peers keep pre-Cut-2 eviction.
Cut F is unblocked once Cuts 1 and 3 merge; the text below mapped these answers already.
- **Body.** CultLib `main` at `e9469cb3`. Every transport file is byte-identical to `3bf1c0c` there
(`git diff 3bf1c0c e9469cb3 -- src packages` is empty), so every `file:line` below is against both unless
@ -620,6 +622,12 @@ The ack cuts build on these behaviours and do not alter them:
drops what it owed. How a new generation is recognised (Cut 1c) needs one (Q-A6). The two are separable
because 1b does not forget peer state: the peer may not know the session ended, and forgetting what was
received would let its retransmits be delivered twice.
**Correction (Cut 1/1b Soul, 2026-09-30): they are separable as specs, not as merges.** 1b drops owed
writes, which leaves holes the sender will never fill; the receiver keeps `R` across the ending, so if the
next handshake does not reset it, every later ordered frame is held and acked, and flush succeeds while
nothing is delivered. Only 1c's reset on a non-repeated handshake removes that. **1b merges only together
with 1c.** Soul also found `R` seeded by the first reliable packet rather than the handshake; ruled:
`R` is seeded only by the handshake, and reliable data before the Accept is neither acked nor delivered.
- **What must hold after both.** A write is Acknowledged or Invalidated within the generation it was
issued in. A restarted client is admitted. A new server session's frames are delivered after a
reconnect. A retransmitted Connect is still a repeat.
@ -1541,3 +1549,38 @@ Each question stands alone.
`packetsDropped` until K1. Record the stats shape in the parity doc once both land.
- **The P4 probe at the old pins** predates its fix; rerun `09318d13`'s P4 against a pinned sender if
old-sender evidence is needed.
**Follow-up (media FEC Cut 3 Hands, 2026-09-30):** `channel_send_options` in `rudp.rs` has no test for the
`"schema"` and `"latest"` channel arms. Deleting either falls to the default and every test stays green. Pin their send
flags on the wire the way the media arm is now pinned (raw peer socket), in whichever cut next touches channel
profiles.
**Rulings after the Cut 1/1b/1c Soul pass (Self, 2026-09-30):**
- **Stale data after a reconnect.** A reliable sequence at or before the generation's seeded `R` (serial arithmetic
within the receive window) is a duplicate: acknowledged, not delivered. This is what §6/§8 already promised.
- **Stale Connect (RFC 5961's challenge ACK).** A Connect whose sequence precedes the current generation's Connect
within the receive window is stale: it is answered with an Ack naming the current generation and does not reset.
A client whose Connect gets no Accept before its connect attempt times out starts a fresh attempt with a newly
drawn initial sequence, so a restarted client whose random sequence lands in that window still connects.
- **Server mode.** A Connect from a different endpoint is never a repeat; the session predicate stays sequence-only.
(Hands' deviation 5 was rejected: pinned clients restarting on a new port were locked out.)
- **The random default belongs to the session.** A session draws its own initial sequence when none is set, so one
options object reused across sessions or reconnects never repeats a sequence.
- **Connect attempts (Self, 2026-09-30, after the Cut 1c fix-batch Soul pass).** While a Connect awaits its Accept, only an
Accept the client honours retires it; an Ack never does (an Ack naming the pending Connect had wedged the client for
good). A fresh attempt after the connect-attempt timeout uses exactly the abandoned sequence + (receive window − 1),
not a random draw: that keeps a late copy of the abandoned Connect inside the new one's stale window while escaping the
server's stale window (unless the server sits exactly there, in which case the next attempt escapes). Only a client's
first Connect draws randomly.
- **Recorded divergence (not this campaign's to fix yet).** Only Rust has a receiver receive-ahead window (1,024). TS,
C# and Python acknowledge a far-ahead reliable frame by name and hold it behind an ordered gap indefinitely, so a
stranded ordered channel is silent loss there and "still owed" in Rust. Map a receive window for the three runtimes.
- **Residuals after the final Cut 1c Soul pass (recorded 2026-09-30, low, not fixed):**
- A late copy of a Connect from two or more attempts back (reordered > ~3 s) still restarts the server; the +window−1
rule keeps only the immediately previous attempt stale. Rust then times out and reconnects; TS/C#/Python hit the
receive-window divergence above (silent loss behind the gap).
- A client pinning `initial_sequence` that restarts while its old data is still in flight after the restarted Connect
can be matched to the old session's resent Accept (byte-identical; the client cannot tell). Pinning forfeits restart
detection; the random default closes it. Document on the option.
- Repeated failed attempts walk the sequence up 4095 per 3 s and clamp at `u32::MAX−1` after ~18-27 days on one session;
make exhaustion a hard error the caller sees (the clamp removal mutant survives).

2
packages/cultmath/README.md
View file

@ -33,6 +33,8 @@ shader semantics.
- `shaders/CultMath.hlsl`, a canonical HLSL mirror include for shader-side
parity. HLSL already owns `float2`, `float3`, and `float4`; the include
exposes `cultmath_*` functions for shared semantics HLSL does not provide.
It includes `shaders/CultMath.Phacelle.hlsl` (MPL-2.0) from the same
directory, so ship both files.
- `Voronoi.SampleTones`, a C# batch surface that calls the Rust
`cultmath-core` native kernel when `cultmath_core` is available and falls back
to the managed parity path otherwise.

20
packages/cultmath/THIRD-PARTY-NOTICES.md
View file

@ -19,3 +19,23 @@ permutation and scale as `snoise(float3)`.
- Copyright 2011 Ashima Arts.
- Upstream: <https://github.com/ashima/webgl-noise>
- License: MIT.
## Phacelle Noise
`math.phacelle` and `cultmath_phacelle` follow Rune Skovbo Johansen's Phacelle
noise (Shadertoy `t3dyWl`; erosion filter `wXcfWn`; blog, March 2026), read
from the C# port `PhacelleNoise` in `lpmitchell/AdvancedTerrainErosion`
(`AdvancedTerrainErosion.cs`, converted to burstable C# by Luke Mitchell,
2026). The cell weight, the phase blend and the normalization follow it. The
3D-cell generalisation, the caller-supplied stripe wave vector, the exact
gradient and the exact pruning are new here.
- Copyright (c) 2025 Rune Skovbo Johansen.
- Upstream: <https://github.com/lpmitchell/AdvancedTerrainErosion>
- License: MPL-2.0 (<https://mozilla.org/MPL/2.0/>). MPL-2.0 is file-level, so it
covers exactly the files that carry the port and nothing else:
`src/CultMath/math.Phacelle.cs`, `src/CultMath/CultPhasor.cs`,
`shaders/CultMath.Phacelle.hlsl` and the Unity copy of that include. Each
carries the MPL-2.0 header and this attribution. The rest of CultMath is MIT,
so the `GameCult.Math` NuGet package declares `MIT AND MPL-2.0` and ships this
file. The Unity package as a whole is MPL-2.0.

39
packages/cultmath/docs/design.md
View file

@ -128,8 +128,8 @@ whose HLSL return type is a plain struct of `float`/`floatN`/`int`/`intN`
fields is compared to its C# counterpart field by field, recursing into each
field's own components, down to bit-for-bit scalars, matching each leaf's
declaration path (field names, not just position) and declared type as well
as its value. `CultCellular` (`math.cs`, `cellular`) is the first and, for
now, only consumer. A struct return with a mismatched, reordered-by-name, or
as its value. `CultCellular` (`math.cs`, `cellular`) and `CultPhasor`
(`math.Phacelle.cs`, `phacelle`) are its consumers. A struct return with a mismatched, reordered-by-name, or
retyped field does not get a second comparison path or a shape of its own;
it fails the same walk that already handles a bare vector return.
@ -264,6 +264,41 @@ together, laid out as `float4(gradient.xyz, value.w)`, with no value-only twin
which is the right domain for `pcg3d`/`pcg4d`'s O(1) hash inputs but would
turn an unclamped octave count into a multi-billion-iteration loop for these
two mirrors.
- `phacelle(float3 p, float3 side, float offset, float normalization)` is Rune
Skovbo Johansen's Phacelle noise ("phase + cell": a stripe pattern built by
blending one cosine/sine wave per jittered cell), generalised to 3D cells. It
returns `CultPhasor { cos, sin }`, each `(∇, value)`. `side` is the stripe
wave vector, whose length is 2π times the stripe frequency; in 3D the
perpendicular to the flow is not unique, so the caller chooses it (on a unit
sphere, `cross(p̂, flow)`). `offset` is the phase in cycles. It visits 4×4×4
cells with `pcg3d`-hashed jitter in `[-0.5, 0.5)`, weights each wave by
`max(0, exp(-2d²) - 0.01111)`, and normalizes the blended phasor as upstream
does (length 1 where the raw length is at least `1 - normalization`, scaled
by `1 / (1 - normalization)` below). The gradient is exact, weight
derivatives and normalization included, rather than upstream's
`∇cos ≈ -sin·side`, which ignores the weight gradient. It is discontinuous
where the raw length crosses the normalization threshold and where a cell's
weight leaves its support (d² = 2.24995, distance 1.49998); the value is
continuous at both. At the support edge `∇w` jumps from `-4·0.01111·v` to 0,
a per-cell jump of length `0.0444·|v|` (0.0667 at `|v| = 1.5`); measured over
random points, the output gradient jumps by a median 1.5% of its length and
at most about 27%. `normalization` is meant to lie in `[0, 1]`: 1 or more
saturates (every output is stretched to length 1), a negative value scales
every output down by `1 / (1 - normalization)`, and the normalized gradient
scales as `1 / |I|` for the blended phasor `I`, so it grows where the blend
nearly cancels. The total weight `W` is positive for every `pcg3d` input (the
smallest found by hill-climbing is 0.056); `W = 0`, and a NaN output, needs
adversarial jitter no hash yields.
Cells whose per-axis lower bound `Σ max(0, |local - g| - 0.5)²` is at least
2.25 lie outside the support and have weight exactly 0, so they are skipped,
which changes no output bit. Cells whose bound sits just under 2.25 can still
carry a nonzero weight, so the cut cannot be lowered; `PhacelleTests` pins
the cut with points next to such cells, against an internal overload with the
prune disabled, and the HLSL mirror test pins it on the shader side. The prune
keeps about 45 of the 64 cells on average (a saving of about 30%); about 14
have a nonzero weight. The port sits in its own MPL-2.0 files
(`math.Phacelle.cs`, `CultPhasor.cs`, `CultMath.Phacelle.hlsl`); see
`THIRD-PARTY-NOTICES.md`. No 2D variant exists; nothing consumes one.
Integer hashing uses the PCG hashes from Jarzynski and Olano, "Hash Functions
for GPU Rendering" (JCGT 9(3), 2020): `pcg(uint)` is O'Neill's RXS-M-XS 32/32

4
packages/cultmath/scripts/build-unity-package.ps1
View file

@ -78,7 +78,9 @@ $requiredTemplateFiles = @(
"Runtime\UnityBridge\UnityConversions.cs",
"Runtime\UnityBridge\UnityConversions.cs.meta",
"Shaders\CultMath.hlsl",
"Shaders\CultMath.hlsl.meta"
"Shaders\CultMath.hlsl.meta",
"Shaders\CultMath.Phacelle.hlsl",
"Shaders\CultMath.Phacelle.hlsl.meta"
)
foreach ($relativePath in $requiredTemplateFiles) {
if (-not (Test-Path -LiteralPath (Join-Path $templateRoot $relativePath) -PathType Leaf)) {

99
packages/cultmath/shaders/CultMath.Phacelle.hlsl Normal file
View file

@ -0,0 +1,99 @@
#ifndef CULTMATH_PHACELLE_HLSL
#define CULTMATH_PHACELLE_HLSL
// This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of
// the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.
//
// Phacelle noise. Copyright (c) 2025 Rune Skovbo Johansen (Shadertoy t3dyWl; MPL-2.0), read from the
// C# port PhacelleNoise in lpmitchell/AdvancedTerrainErosion, converted to burstable C# by Luke
// Mitchell (2026). Modified by GameCult: 3D cells, a caller-supplied stripe wave vector, an exact
// gradient and exact pruning. This file is an MPL-2.0 file; CultMath.hlsl, which includes it after
// its own definitions of CULTMATH_TAU and cultmath_pcg3d, is MIT (see THIRD-PARTY-NOTICES.md).
// Invariant 8's struct return shape for cultmath_phacelle, compared field by field and bit for bit
// against CultMath.CultPhasor, same field order.
struct CultPhasor
{
float4 cos;
float4 sin;
};
// Johansen's Phacelle noise on 3D cells. See math.phacelle's comment (math.Phacelle.cs) for the stripe wave
// vector, the weight, the normalization, the exact gradient and why the prune is exact. Mirrors
// the C# body operation for operation; the C#-only internal overload that switches the prune off
// has no shader twin.
CultPhasor cultmath_phacelle(float3 p, float3 side, float offset, float normalization)
{
float3 cell = floor(p);
float3 local = p - cell;
float phaseOffset = offset * CULTMATH_TAU;
float sumW = 0.0;
float sumCos = 0.0;
float sumSin = 0.0;
float3 gradW = float3(0.0, 0.0, 0.0);
float3 gradCos = float3(0.0, 0.0, 0.0);
float3 gradSin = float3(0.0, 0.0, 0.0);
for (int dz = -1; dz <= 2; dz++)
{
float gz = max(0.0, abs(local.z - (float)dz) - 0.5);
for (int dy = -1; dy <= 2; dy++)
{
float gy = max(0.0, abs(local.y - (float)dy) - 0.5);
for (int dx = -1; dx <= 2; dx++)
{
float gx = max(0.0, abs(local.x - (float)dx) - 0.5);
if (gx * gx + gy * gy + gz * gz >= 2.25)
continue;
float3 gridStep = float3(dx, dy, dz);
int3 hash = cultmath_pcg3d(int3(cell + gridStep));
float3 jitter = float3(
((uint)hash.x >> 8) * (1.0 / 16777216.0) - 0.5,
((uint)hash.y >> 8) * (1.0 / 16777216.0) - 0.5,
((uint)hash.z >> 8) * (1.0 / 16777216.0) - 0.5);
float3 v = local - gridStep - jitter;
float falloff = exp(-2.0 * dot(v, v));
float w = max(0.0, falloff - 0.01111);
float3 dw = w > 0.0 ? (-4.0 * falloff) * v : float3(0.0, 0.0, 0.0);
float phase = dot(v, side) + phaseOffset;
float c = cos(phase);
float s = sin(phase);
sumW += w;
sumCos += w * c;
sumSin += w * s;
gradW += dw;
gradCos += dw * c - (w * s) * side;
gradSin += dw * s + (w * c) * side;
}
}
}
float invW = 1.0 / sumW;
float rawCos = sumCos * invW;
float rawSin = sumSin * invW;
float3 gradRawCos = (gradCos - rawCos * gradW) * invW;
float3 gradRawSin = (gradSin - rawSin * gradW) * invW;
float rawLength = sqrt(rawCos * rawCos + rawSin * rawSin);
float floorLength = 1.0 - normalization;
CultPhasor result;
if (rawLength > floorLength)
{
float outCos = rawCos / rawLength;
float outSin = rawSin / rawLength;
float3 gradLength = (rawCos * gradRawCos + rawSin * gradRawSin) / rawLength;
result.cos = float4((gradRawCos - outCos * gradLength) / rawLength, outCos);
result.sin = float4((gradRawSin - outSin * gradLength) / rawLength, outSin);
}
else
{
result.cos = float4(gradRawCos / floorLength, rawCos / floorLength);
result.sin = float4(gradRawSin / floorLength, rawSin / floorLength);
}
return result;
}
#endif

7
packages/cultmath/shaders/CultMath.Phacelle.hlsl.meta Normal file
View file

@ -0,0 +1,7 @@
fileFormatVersion: 2
guid: e07a3b95c1d84f2aa6b83d4c5f19a0d7
ShaderIncludeImporter:
externalObjects: {}
userData:
assetBundleName:
assetBundleVariant:

2
packages/cultmath/shaders/CultMath.hlsl
View file

@ -503,6 +503,8 @@ CultCellular cultmath_cellular(float3 p)
return result;
}
#include "CultMath.Phacelle.hlsl"
float cultmath_value_noise(float2 position)
{
float2 cell = floor(position);

4
packages/cultmath/src/CultMath/CultMath.csproj
View file

@ -8,6 +8,8 @@
<RootNamespace>CultMath</RootNamespace>
<AssemblyName>CultMath</AssemblyName>
<PackageId>GameCult.Math</PackageId>
<!-- MIT overall; math.Phacelle.cs and CultPhasor.cs (and the two Phacelle HLSL files) are MPL-2.0. -->
<PackageLicenseExpression>MIT AND MPL-2.0</PackageLicenseExpression>
<Version>0.2.4</Version>
<Description>Clean-room C# math shaped for HLSL-style renderer code.</Description>
<Authors>GameCult</Authors>
@ -21,6 +23,8 @@
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
<None Include="..\..\shaders\CultMath.hlsl" Pack="true" PackagePath="contentFiles\any\any\shaders\CultMath.hlsl" CopyToOutputDirectory="PreserveNewest" />
<None Include="..\..\shaders\CultMath.Phacelle.hlsl" Pack="true" PackagePath="contentFiles\any\any\shaders\CultMath.Phacelle.hlsl" CopyToOutputDirectory="PreserveNewest" />
<None Include="..\..\THIRD-PARTY-NOTICES.md" Pack="true" PackagePath="\" />
</ItemGroup>
</Project>

27
packages/cultmath/src/CultMath/CultPhasor.cs Normal file
View file

@ -0,0 +1,27 @@
// This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of
// the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.
//
// The result type of math.phacelle (math.Phacelle.cs), Rune Skovbo Johansen's Phacelle noise
// (Copyright (c) 2025, MPL-2.0; see THIRD-PARTY-NOTICES.md).
namespace CultMath;
/// <summary>
/// The value-and-gradient result of <see cref="math.phacelle(float3, float3, float, float)"/>
/// (invariant 8's struct return shape). Both fields use the <c>float4(gradient.xyz, value.w)</c>
/// layout of a bare value-and-gradient return.
/// </summary>
public struct CultPhasor
{
/// <summary>(&#8711;cos, cos): the normalized cosine wave and its gradient.</summary>
public float4 cos;
/// <summary>(&#8711;sin, sin): the normalized sine wave and its gradient.</summary>
public float4 sin;
public CultPhasor(float4 cos, float4 sin)
{
this.cos = cos;
this.sin = sin;
}
}

2
packages/cultmath/src/CultMath/CultPhasor.cs.meta Normal file
View file

@ -0,0 +1,2 @@
fileFormatVersion: 2
guid: 807ae2c8a758cf31896da12f1e5047bd

127
packages/cultmath/src/CultMath/math.Phacelle.cs Normal file
View file

@ -0,0 +1,127 @@
// This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of
// the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.
//
// Phacelle noise. Copyright (c) 2025 Rune Skovbo Johansen (Shadertoy t3dyWl; MPL-2.0), read from the
// C# port PhacelleNoise in lpmitchell/AdvancedTerrainErosion, converted to burstable C# by Luke
// Mitchell (2026). Modified by GameCult: 3D cells, a caller-supplied stripe wave vector, an exact
// gradient and exact pruning. This file and CultPhasor.cs are the MPL-2.0 files of CultMath; the
// rest of the package is MIT (see THIRD-PARTY-NOTICES.md).
namespace CultMath;
public static partial class math
{
// Johansen's Phacelle noise (phase + cell), generalised to 3D cells with a caller-supplied
// stripe wave vector. Each of the 4x4x4 cells around p carries one jittered feature point
// (jitter in [-0.5, 0.5) per axis, from pcg3d over the integer cell coordinate, the same
// integer-hashing rule as cellular) and one cosine/sine wave that runs along `side` (the wave
// vector: its length is 2*pi times the stripe frequency, and the caller chooses which
// perpendicular it is; on a unit sphere that is cross(p_hat, flow)). The waves are blended with
// the bell weight max(0, exp(-2 d^2) - 0.01111) (exactly 0 at distance 1.5, the nearest a
// feature outside the 4x4x4 block can be in 2D, which is what keeps cell-grid lines out), and
// the blended phasor is normalized: its length is stretched to 1 wherever the raw length is at
// least 1 - normalization, and scaled by 1 / (1 - normalization) below that. `offset` is the
// phase in cycles (1 is a full turn). Value and normalization follow Johansen's
// PhacelleNoise as ported in lpmitchell/AdvancedTerrainErosion.
//
// `normalization` is meant to lie in [0, 1]. At 0 the output is the raw blend (a phasor of
// length at most 1); at 1 the floor is 0 and every output is stretched to length 1. Values of
// 1 or more saturate at that (the floor is never above the raw length), and a negative value
// scales every output down by 1 / (1 - normalization) < 1. The normalized gradient scales as
// 1 / |I| where I is the blended phasor (below), so it grows wherever the blend nearly
// cancels; a normalization near 1 lets that happen at every point.
//
// The gradient is exact: it carries the weight derivative (grad w = -4 v exp(-2 |v|^2) where
// w > 0) and the normalization, not upstream's convention grad(cos) ~ -sin * side, which
// ignores the weight gradient (Asura probe: 16.8 deg off at the median, 97.8 at the 95th
// percentile; the exact form is 0.00 and 0.03). With P = sum(w * (cos, sin)) and W = sum(w),
// I = P / W has grad I = (grad P - I * grad W) / W; above the threshold the output is I / |I|
// with gradient (grad I - out * grad|I|) / |I|, below it grad I / (1 - normalization). The
// value is continuous at both seams below; the gradient is not.
// * Where |I| crosses the threshold, the gradient jumps between the two branches.
// * Where a cell's weight leaves its support (d^2 = ln(1 / 0.01111) / 2 = 2.24995, distance
// 1.49998), grad w jumps from -4 * 0.01111 * v to 0: a per-cell jump of length
// 0.0444 * |v|, 0.0667 at |v| = 1.5. Measured over random points, the resulting jump in
// the output gradient is a median 1.5% of its length and at most about 27%.
//
// W is the sum of 64 weights and is positive for every input pcg3d produces (the smallest
// W found by hill-climbing the hash-driven cell contents is 0.056). It is 0, and the output
// NaN, only if all 64 weights vanish, which takes adversarial jitter no hash yields.
//
// Pruning is exact: per axis, |local - g| - 0.5 (floored at 0) lower-bounds |v| for any jitter,
// so a cell whose summed square is at least 2.25 lies outside the support (2.24995 < 2.25) and
// has weight exactly 0: it contributes nothing to any sum. Cells whose bound sits just under
// 2.25 can still have a nonzero weight, so the cut cannot be lowered. On average the prune
// keeps about 45 of the 64 cells (a saving of about 30%); only about 14 have a nonzero weight.
// Only the internal overload can switch the prune off; PhacelleTests uses it to pin bit equality.
public static CultPhasor phacelle(float3 p, float3 side, float offset, float normalization) =>
phacelle(p, side, offset, normalization, true);
internal static CultPhasor phacelle(float3 p, float3 side, float offset, float normalization, bool prune)
{
var cell = floor(p);
var local = p - cell;
var phaseOffset = offset * TAU;
var sumW = 0.0f;
var sumCos = 0.0f;
var sumSin = 0.0f;
var gradW = new float3(0.0f, 0.0f, 0.0f);
var gradCos = new float3(0.0f, 0.0f, 0.0f);
var gradSin = new float3(0.0f, 0.0f, 0.0f);
for (var dz = -1; dz <= 2; dz++)
{
var gz = max(0.0f, abs(local.z - (float)dz) - 0.5f);
for (var dy = -1; dy <= 2; dy++)
{
var gy = max(0.0f, abs(local.y - (float)dy) - 0.5f);
for (var dx = -1; dx <= 2; dx++)
{
var gx = max(0.0f, abs(local.x - (float)dx) - 0.5f);
if (prune && gx * gx + gy * gy + gz * gz >= 2.25f)
continue;
var gridStep = new float3(dx, dy, dz);
var hash = pcg3d(int3(cell + gridStep));
var jitter = new float3(cellular_unit(hash.x) - 0.5f, cellular_unit(hash.y) - 0.5f, cellular_unit(hash.z) - 0.5f);
var v = local - gridStep - jitter;
var falloff = exp(-2.0f * dot(v, v));
var w = max(0.0f, falloff - 0.01111f);
var dw = w > 0.0f ? (-4.0f * falloff) * v : new float3(0.0f, 0.0f, 0.0f);
var phase = dot(v, side) + phaseOffset;
var c = cos(phase);
var s = sin(phase);
sumW += w;
sumCos += w * c;
sumSin += w * s;
gradW += dw;
gradCos += dw * c - (w * s) * side;
gradSin += dw * s + (w * c) * side;
}
}
}
var invW = 1.0f / sumW;
var rawCos = sumCos * invW;
var rawSin = sumSin * invW;
var gradRawCos = (gradCos - rawCos * gradW) * invW;
var gradRawSin = (gradSin - rawSin * gradW) * invW;
var rawLength = sqrt(rawCos * rawCos + rawSin * rawSin);
var floorLength = 1.0f - normalization;
if (rawLength > floorLength)
{
var outCos = rawCos / rawLength;
var outSin = rawSin / rawLength;
var gradLength = (rawCos * gradRawCos + rawSin * gradRawSin) / rawLength;
return new CultPhasor(
new float4((gradRawCos - outCos * gradLength) / rawLength, outCos),
new float4((gradRawSin - outSin * gradLength) / rawLength, outSin));
}
return new CultPhasor(
new float4(gradRawCos / floorLength, rawCos / floorLength),
new float4(gradRawSin / floorLength, rawSin / floorLength));
}
}

2
packages/cultmath/src/CultMath/math.Phacelle.cs.meta Normal file
View file

@ -0,0 +1,2 @@
fileFormatVersion: 2
guid: 2c9d4e6b1a8f47d3b05e7a3c9d1f8e26

15
packages/cultmath/tests/CultMath.Tests/HlslMirrorTests.cs
View file

@ -7,7 +7,7 @@ public sealed class HlslMirrorTests
[Fact]
public void HlslMirrorDoesNotPublishGeometryKernels()
{
var source = File.ReadAllText(Path.Combine(GetCultMathRoot(), "shaders", "CultMath.hlsl"));
var source = HlslSourceCompatibilityTests.ReadShaderSource(GetCultMathRoot());
Assert.DoesNotContain("AdvancedErosionFilter.hlsl", source);
Assert.DoesNotContain("Planetary.hlsl", source);
Assert.DoesNotContain("spherical_erosion", source);
@ -17,7 +17,7 @@ public sealed class HlslMirrorTests
[Fact]
public void HlslMirrorPublishesCultMathPrimitives()
{
var include = File.ReadAllText(Path.Combine(GetCultMathRoot(), "shaders", "CultMath.hlsl"));
var include = HlslSourceCompatibilityTests.ReadShaderSource(GetCultMathRoot());
var requiredSymbols = new[]
{
"cultmath_radians",
@ -62,6 +62,17 @@ public sealed class HlslMirrorTests
Assert.DoesNotContain("cultmath_normalize", include);
}
[Theory]
[InlineData("CultMath.hlsl")]
[InlineData("CultMath.Phacelle.hlsl")]
public void UnityPackageShaderIsIdenticalToTheCanonicalShader(string name)
{
var root = GetCultMathRoot();
Assert.Equal(
File.ReadAllText(Path.Combine(root, "shaders", name)).ReplaceLineEndings("\n"),
File.ReadAllText(Path.Combine(root, "unity", "org.gamecult.cultmath", "Shaders", name)).ReplaceLineEndings("\n"));
}
private static string GetCultMathRoot()
{
var directory = new DirectoryInfo(AppContext.BaseDirectory);

73
packages/cultmath/tests/CultMath.Tests/HlslSourceCompatibilityTests.cs
View file

@ -30,6 +30,7 @@ public sealed class HlslSourceCompatibilityTests
Assert.NotNull(ShaderFunction(shader, "cultmath_snoise_grad", typeof(float3)));
Assert.NotNull(ShaderFunction(shader, "cultmath_fbm_grad", typeof(float3), typeof(int), typeof(float), typeof(float)));
Assert.NotNull(ShaderFunction(shader, "cultmath_ridged_grad", typeof(float3), typeof(int), typeof(float), typeof(float)));
Assert.NotNull(ShaderFunction(shader, "cultmath_phacelle", typeof(float3), typeof(float3), typeof(float), typeof(float)));
}
// HLSL functions carry no access modifier, so they compile as private instance methods.
@ -130,7 +131,63 @@ public sealed class HlslSourceCompatibilityTests
}
Assert.True(mismatches.Count == 0, $"{mismatches.Count} mismatches:{Environment.NewLine}{string.Join(Environment.NewLine, mismatches.Take(12))}");
Assert.Equal(33, compared.Count);
Assert.Equal(34, compared.Count);
}
/// <summary>
/// The generic comparison above passes about a hundred inputs through <c>cultmath_phacelle</c>, and
/// its prune only changes an output for the rare cell whose weight is nonzero but whose lower
/// bound sits just under the 2.25 cut (Soul's F6 lesson for cellular): a mirror whose cut is 2.0
/// survives a hundred inputs. A dense sweep of realistic arguments closes that gap.
/// </summary>
[Fact]
public void PhacelleMirrorMatchesCSharpBitForBitAcrossADenseSweep()
{
var (assembly, errors) = CompileShaderMirror();
Assert.True(assembly is not null, string.Join(Environment.NewLine, errors));
var shaderType = assembly!.GetType("CultMathHlsl.HlslShader")!;
var shader = Activator.CreateInstance(shaderType);
var mirror = ShaderFunction(shaderType, "cultmath_phacelle", typeof(float3), typeof(float3), typeof(float), typeof(float))!;
var random = new System.Random(0x5EED2);
float Next(float extent) => (random.NextSingle() * 2.0f - 1.0f) * extent;
for (var i = 0; i < 20000; i++)
{
var p = new float3(Next(50.0f), Next(50.0f), Next(50.0f));
var side = new float3(Next(9.0f), Next(9.0f), Next(9.0f));
var offset = random.NextSingle();
var normalization = random.NextSingle();
var expected = Values(phacelle(p, side, offset, normalization));
var actual = Values(mirror.Invoke(shader, new object[] { p, side, offset, normalization })!);
Assert.True(BitwiseEqual(expected, actual), $"mirror differs from C# at p = {p}, side = {side}");
}
}
/// <summary>
/// The dense sweep almost never meets a cell whose box bound is just under the 2.25 cut and whose
/// weight is still nonzero: a mirror cut of 2.245, 2.24 or 2.2 survives it. These points sit next
/// to such cells (<see cref="PhacelleTests.PruneCutPoints"/>), so the mirror must match C# bit for
/// bit, and must match the unpruned C# sum (the shader has no prune switch, so that is the
/// independent statement that the cut is exact).
/// </summary>
[Fact]
public void PhacelleMirrorMatchesCSharpBitForBitNextToCellsJustInsideTheCut()
{
var (assembly, errors) = CompileShaderMirror();
Assert.True(assembly is not null, string.Join(Environment.NewLine, errors));
var shaderType = assembly!.GetType("CultMathHlsl.HlslShader")!;
var shader = Activator.CreateInstance(shaderType);
var mirror = ShaderFunction(shaderType, "cultmath_phacelle", typeof(float3), typeof(float3), typeof(float), typeof(float))!;
foreach (var p in PhacelleTests.PruneCutPoints)
{
Assert.True(PhacelleTests.HasCellJustInsideTheCut(p), $"no cell just inside the cut at {p}");
var actual = Values(mirror.Invoke(shader, new object[] { p, PhacelleTests.PruneCutSide, PhacelleTests.PruneCutOffset, PhacelleTests.PruneCutNormalization })!);
var pruned = Values(phacelle(p, PhacelleTests.PruneCutSide, PhacelleTests.PruneCutOffset, PhacelleTests.PruneCutNormalization));
var unpruned = Values(phacelle(p, PhacelleTests.PruneCutSide, PhacelleTests.PruneCutOffset, PhacelleTests.PruneCutNormalization, false));
Assert.True(BitwiseEqual(pruned, actual), $"mirror differs from C# at p = {p}");
Assert.True(BitwiseEqual(unpruned, actual), $"mirror differs from the unpruned C# sum at p = {p}");
}
}
/// <summary>
@ -253,6 +310,18 @@ public sealed class HlslSourceCompatibilityTests
? (float.IsNaN(a) && float.IsNaN(b)) || BitConverter.SingleToInt32Bits(a) == BitConverter.SingleToInt32Bits(b)
: p.First.Value.Equals(p.Second.Value)));
/// <summary>
/// The text of <c>CultMath.hlsl</c> with each <c>#include "name"</c> replaced in place by the
/// included file from the same directory, as the shader compiler would resolve it. Only the
/// package's own includes exist, so a missing file is an error rather than a skip.
/// </summary>
internal static string ReadShaderSource(string cultMathRoot)
{
var shaders = Path.Combine(cultMathRoot, "shaders");
return Regex.Replace(File.ReadAllText(Path.Combine(shaders, "CultMath.hlsl")), @"(?m)^#include ""([^""]+)""[ \t]*\r?$",
match => File.ReadAllText(Path.Combine(shaders, match.Groups[1].Value)));
}
/// <summary>The documented HLSL-to-C# transformations, and nothing else.</summary>
internal static string TransformHlslToCSharp(string hlsl)
{
@ -281,7 +350,7 @@ public sealed class HlslSourceCompatibilityTests
private static (Assembly? Assembly, IReadOnlyList<string> Errors) CompileShaderMirror()
{
var hlsl = File.ReadAllText(Path.Combine(FindCultMathRoot(), "shaders", "CultMath.hlsl"));
var hlsl = ReadShaderSource(FindCultMathRoot());
var tree = CSharpSyntaxTree.ParseText(TransformHlslToCSharp(hlsl));
var references = ((string)AppContext.GetData("TRUSTED_PLATFORM_ASSEMBLIES")!)
.Split(Path.PathSeparator)

340
packages/cultmath/tests/CultMath.Tests/PhacelleTests.cs Normal file
View file

@ -0,0 +1,340 @@
using System;
using CultMath;
using Xunit;
namespace CultMath.Tests;
public sealed class PhacelleTests
{
// Every tolerance below is at least 5x the largest error the same test measures at its seed.
private const float GradientEps = 1.0e-3f;
private const float GradientTolerance = 5.0e-3f;
private const double ValueTolerance = 1.6e-5;
private const double ThresholdBand = 0.02;
private static float3 RandomPoint(System.Random random, float extent = 3.0f) => new(
(random.NextSingle() * 2.0f - 1.0f) * extent, (random.NextSingle() * 2.0f - 1.0f) * extent, (random.NextSingle() * 2.0f - 1.0f) * extent);
// A wave vector of length 2*pi times a stripe frequency in [0.5, 1.5], in a random direction.
private static float3 RandomSide(System.Random random)
{
float3 direction;
do
{
direction = new float3(random.NextSingle() - 0.5f, random.NextSingle() - 0.5f, random.NextSingle() - 0.5f);
}
while (math.length(direction) < 0.1f);
return math.normalize(direction) * (math.TAU * (0.5f + random.NextSingle()));
}
// ---- An independent double-precision evaluation: all 64 cells, no prune, no gradient. ----
//
// It shares only the hash (pcg3d over the integer cell coordinate, top 24 bits) and the constants
// the function is specified by: weight max(0, exp(-2 d^2) - 0.01111), jitter in [-0.5, 0.5).
private readonly record struct Reference(double Cos, double Sin, double RawLength, double SupportEdgeGap);
private static Reference Evaluate(float3 p, float3 side, float offset, float normalization)
{
double fx = Math.Floor(p.x), fy = Math.Floor(p.y), fz = Math.Floor(p.z);
double lx = p.x - fx, ly = p.y - fy, lz = p.z - fz;
double sumW = 0.0, sumCos = 0.0, sumSin = 0.0, edgeGap = double.MaxValue;
var supportRadius = Math.Sqrt(Math.Log(1.0 / 0.01111) / 2.0);
for (var dz = -1; dz <= 2; dz++)
for (var dy = -1; dy <= 2; dy++)
for (var dx = -1; dx <= 2; dx++)
{
var hash = math.pcg3d(new int3((int)fx + dx, (int)fy + dy, (int)fz + dz));
double vx = lx - dx - (math.cellular_unit(hash.x) - 0.5);
double vy = ly - dy - (math.cellular_unit(hash.y) - 0.5);
double vz = lz - dz - (math.cellular_unit(hash.z) - 0.5);
var d2 = vx * vx + vy * vy + vz * vz;
edgeGap = Math.Min(edgeGap, Math.Abs(Math.Sqrt(d2) - supportRadius));
var w = Math.Max(0.0, Math.Exp(-2.0 * d2) - 0.01111);
var phase = vx * side.x + vy * side.y + vz * side.z + (double)offset * math.TAU;
sumW += w;
sumCos += w * Math.Cos(phase);
sumSin += w * Math.Sin(phase);
}
double rawCos = sumCos / sumW, rawSin = sumSin / sumW;
var raw = Math.Sqrt(rawCos * rawCos + rawSin * rawSin);
var length = Math.Max(1.0 - normalization, raw);
return new Reference(rawCos / length, rawSin / length, raw, edgeGap);
}
private static float3 CentralDifference(Func<float3, float> f, float3 p, float eps) => new(
(f(p + new float3(eps, 0.0f, 0.0f)) - f(p - new float3(eps, 0.0f, 0.0f))) / (2.0f * eps),
(f(p + new float3(0.0f, eps, 0.0f)) - f(p - new float3(0.0f, eps, 0.0f))) / (2.0f * eps),
(f(p + new float3(0.0f, 0.0f, eps)) - f(p - new float3(0.0f, 0.0f, eps))) / (2.0f * eps));
private static float MaxComponentError(float3 numeric, float4 analytic) => MathF.Max(
MathF.Abs(numeric.x - analytic.x), MathF.Max(MathF.Abs(numeric.y - analytic.y), MathF.Abs(numeric.z - analytic.z)));
// ---- Value: the function is Johansen's, on 3D cells ----
[Theory]
[InlineData(0.0f)]
[InlineData(0.5f)]
[InlineData(0.9f)]
[InlineData(1.0f)]
public void ValueMatchesTheIndependentReferenceAndNormalizesAboveTheThreshold(float normalization)
{
var random = new System.Random(0x7ACE1 + (int)(normalization * 10.0f));
var maxError = 0.0;
var above = 0;
var below = 0;
for (var i = 0; i < 3000; i++)
{
var p = RandomPoint(random, 40.0f);
var side = RandomSide(random);
var offset = random.NextSingle();
var actual = math.phacelle(p, side, offset, normalization);
var expected = Evaluate(p, side, offset, normalization);
maxError = Math.Max(maxError, Math.Max(Math.Abs(actual.cos.w - expected.Cos), Math.Abs(actual.sin.w - expected.Sin)));
if (expected.RawLength >= 1.0 - normalization + ThresholdBand)
{
// The output magnitude is 1 wherever the raw magnitude reaches 1 - normalization.
above++;
var magnitude = Math.Sqrt((double)actual.cos.w * actual.cos.w + (double)actual.sin.w * actual.sin.w);
Assert.True(Math.Abs(magnitude - 1.0) < ValueTolerance, $"magnitude {magnitude} above the threshold at {p}");
}
else if (expected.RawLength <= 1.0 - normalization - ThresholdBand)
{
below++;
}
}
Assert.True(maxError < ValueTolerance, $"max value error against the reference {maxError}");
if (normalization > 0.0f)
Assert.True(above >= 500, $"only {above} points above the threshold");
if (normalization is > 0.0f and < 1.0f)
Assert.True(below >= 50, $"only {below} points below the threshold");
}
[Fact]
public void ValueIsPeriodicInOffsetAndAQuarterCycleRotatesCosIntoSin()
{
// offset is in cycles: a quarter turn maps cos to -sin and sin to cos, and a whole turn is the
// identity. The weights and the normalization do not depend on the phase, so this is exact
// up to float rounding.
var random = new System.Random(0x7ACE2);
for (var i = 0; i < 200; i++)
{
var p = RandomPoint(random);
var side = RandomSide(random);
var offset = random.NextSingle();
var basePhasor = math.phacelle(p, side, offset, 0.5f);
var quarter = math.phacelle(p, side, offset + 0.25f, 0.5f);
var whole = math.phacelle(p, side, offset + 1.0f, 0.5f);
Assert.True(MathF.Abs(quarter.cos.w + basePhasor.sin.w) < 2.0e-5f);
Assert.True(MathF.Abs(quarter.sin.w - basePhasor.cos.w) < 2.0e-5f);
Assert.True(MathF.Abs(whole.cos.w - basePhasor.cos.w) < 2.0e-5f);
Assert.True(MathF.Abs(whole.sin.w - basePhasor.sin.w) < 2.0e-5f);
}
}
// ---- Invariant 8: the gradient is exact ----
[Fact]
public void GradientOfCosAndSinMatchesCentralDifferencesOnBothSidesOfTheNormalizationThreshold()
{
// Above the threshold the output is I / |I|; below it, I / (1 - normalization). The gradient
// is discontinuous where |I| crosses the threshold, and at the edge of each cell's weight support
// (distance 1.49998, where max(0, ...) starts: a jump of 4 d 0.01111 in grad w), so a band
// around each is excluded (by the
// reference's raw length, since the output alone cannot say how close it is).
const float normalization = 0.5f;
var random = new System.Random(0x7ACE3);
var above = 0;
var below = 0;
var maxError = 0.0f;
for (var attempts = 0; attempts < 20000 && (above < 150 || below < 150); attempts++)
{
var p = RandomPoint(random);
var side = RandomSide(random);
var offset = random.NextSingle();
var reference = Evaluate(p, side, offset, normalization);
var distance = reference.RawLength - (1.0 - normalization);
if (Math.Abs(distance) < ThresholdBand || reference.SupportEdgeGap < 3.0 * GradientEps)
continue;
if (distance > 0.0 ? above >= 150 : below >= 150)
continue;
if (distance > 0.0) above++; else below++;
var analytic = math.phacelle(p, side, offset, normalization);
var numericCos = CentralDifference(q => math.phacelle(q, side, offset, normalization).cos.w, p, GradientEps);
var numericSin = CentralDifference(q => math.phacelle(q, side, offset, normalization).sin.w, p, GradientEps);
maxError = MathF.Max(maxError, MathF.Max(MaxComponentError(numericCos, analytic.cos), MaxComponentError(numericSin, analytic.sin)));
}
Assert.True(above >= 150 && below >= 150, $"{above} points above and {below} below the threshold");
Assert.True(maxError < GradientTolerance, $"max gradient error against central differences {maxError}");
}
// ---- Stripes run along the flow ----
[Fact]
public void OnTheUnitSphereStripesRunAlongTheFlow()
{
// side = cross(p_hat, flow) * 2*pi is tangent to the sphere and perpendicular to the flow, so the
// cosine should change across the stripes (along side) and hardly at all along them (along
// flow). Asura's probe measured 0.212 for this ratio, against 0.202 for the planar upstream.
const float radiusInCells = 10.0f;
var random = new System.Random(0x7ACE4);
var reference = math.normalize(new float3(0.3f, 0.5f, 0.8f));
var alongFlow = 0.0;
var acrossFlow = 0.0;
var used = 0;
while (used < 2000)
{
var hat = math.normalize(new float3(random.NextSingle() - 0.5f, random.NextSingle() - 0.5f, random.NextSingle() - 0.5f));
var tangent = reference - hat * math.dot(hat, reference);
if (math.length(tangent) < 0.2f)
continue;
var flow = math.normalize(tangent);
var sideDirection = math.cross(hat, flow);
var phasor = math.phacelle(hat * radiusInCells, sideDirection * math.TAU, 0.25f, 0.5f);
var gradient = new float3(phasor.cos.x, phasor.cos.y, phasor.cos.z);
alongFlow += Math.Abs(math.dot(gradient, flow));
acrossFlow += Math.Abs(math.dot(gradient, sideDirection));
used++;
}
var ratio = alongFlow / acrossFlow;
Assert.True(ratio <= 0.25, $"mean|d cos/d flow| / mean|d cos/d side| = {ratio}");
}
// ---- Continuity ----
[Fact]
public void ValueStepDifferenceShrinksTenfoldPerTenfoldStepIncludingAcrossCellPlanes()
{
// Pairs straddle an integer plane on each axis (where the 4x4x4 window moves by one cell and a
// weight that did not vanish at the window edge would jump) and also start at random points.
// Lipschitz continuity means the largest difference over a step scales with the step.
var random = new System.Random(0x7ACE5);
var steps = new[] { 1.0e-3f, 1.0e-4f, 1.0e-5f };
var maxDifference = new float[steps.Length];
for (var i = 0; i < 3000; i++)
{
var p = RandomPoint(random);
var side = RandomSide(random);
var offset = random.NextSingle();
var axis = i % 3;
if (i % 2 == 0)
p = SetAxis(p, axis, MathF.Round(GetAxis(p, axis)));
for (var k = 0; k < steps.Length; k++)
{
var low = math.phacelle(SetAxis(p, axis, GetAxis(p, axis) - steps[k] * 0.5f), side, offset, 0.5f);
var high = math.phacelle(SetAxis(p, axis, GetAxis(p, axis) + steps[k] * 0.5f), side, offset, 0.5f);
maxDifference[k] = MathF.Max(maxDifference[k], MathF.Max(MathF.Abs(low.cos.w - high.cos.w), MathF.Abs(low.sin.w - high.sin.w)));
}
}
Assert.True(maxDifference[1] * 8.0f <= maxDifference[0], $"1e-3: {maxDifference[0]}, 1e-4: {maxDifference[1]}");
Assert.True(maxDifference[2] * 8.0f <= maxDifference[1], $"1e-4: {maxDifference[1]}, 1e-5: {maxDifference[2]}");
}
private static float GetAxis(float3 v, int axis) => axis == 0 ? v.x : axis == 1 ? v.y : v.z;
private static float3 SetAxis(float3 v, int axis, float value) =>
axis == 0 ? new float3(value, v.y, v.z) : axis == 1 ? new float3(v.x, value, v.z) : new float3(v.x, v.y, value);
// ---- Exact pruning ----
[Fact]
public void PrunedOutputIsBitEqualToTheUnprunedSum()
{
var random = new System.Random(0x7ACE6);
for (var i = 0; i < 20000; i++)
{
var p = RandomPoint(random, 50.0f);
var side = RandomSide(random);
var offset = random.NextSingle();
var normalization = random.NextSingle();
var pruned = math.phacelle(p, side, offset, normalization);
var unpruned = math.phacelle(p, side, offset, normalization, false);
AssertBitEqual(unpruned.cos, pruned.cos);
AssertBitEqual(unpruned.sin, pruned.sin);
}
}
// Points next to a cell whose jitter sits at the far corner of its cube: the cell's box bound is
// just under the 2.25 cut (2.2451 to 2.25) while its real distance is still under the support
// radius (d^2 < 2.24995), so its weight is small but nonzero and a prune that cuts even slightly
// low (2.245, 2.24, 2.2) drops a contribution and changes output bits. Random points meet such
// a cell about once in 10^4 evaluations, which is why the random sweep above let those cuts
// through. Each was found once by an offline search (cells K in [-12, 12]^3 whose jitter on one
// axis lies within 0.0008 of the cube face; p is the feature moved 1.4996 out along that axis,
// so |p - feature| = 1.4996 and the bound is about 1.4988). The witness check below re-derives
// that property from the hash, so a change to pcg3d fails loudly instead of degenerating.
// Shared with the HLSL mirror test.
internal static readonly float3[] PruneCutPoints =
{
new(1.77850008f, -2.16020727f, -0.9991166f),
new(1.09346282f, -2.37882328f, -3.00082445f),
new(-6.39144945f, -3.99953747f, -0.610023975f),
new(-0.99940908f, 4.16302729f, 4.07277536f),
new(3.99912024f, 7.21890545f, 1.09053993f),
new(5.43959713f, 1.99937367f, -6.14929914f),
};
internal static readonly float3 PruneCutSide = new(2.1f, -3.7f, 1.3f);
internal const float PruneCutOffset = 0.3f;
internal const float PruneCutNormalization = 0.5f;
// True when some cell of the 4x4x4 window has a box bound in [2.2451, 2.25) and a positive weight.
internal static bool HasCellJustInsideTheCut(float3 p)
{
double fx = Math.Floor(p.x), fy = Math.Floor(p.y), fz = Math.Floor(p.z);
double lx = p.x - fx, ly = p.y - fy, lz = p.z - fz;
for (var dz = -1; dz <= 2; dz++)
for (var dy = -1; dy <= 2; dy++)
for (var dx = -1; dx <= 2; dx++)
{
var gx = Math.Max(0.0, Math.Abs(lx - dx) - 0.5);
var gy = Math.Max(0.0, Math.Abs(ly - dy) - 0.5);
var gz = Math.Max(0.0, Math.Abs(lz - dz) - 0.5);
var bound = gx * gx + gy * gy + gz * gz;
if (bound < 2.2451 || bound >= 2.25)
continue;
var hash = math.pcg3d(new int3((int)fx + dx, (int)fy + dy, (int)fz + dz));
double vx = lx - dx - (math.cellular_unit(hash.x) - 0.5);
double vy = ly - dy - (math.cellular_unit(hash.y) - 0.5);
double vz = lz - dz - (math.cellular_unit(hash.z) - 0.5);
if (Math.Exp(-2.0 * (vx * vx + vy * vy + vz * vz)) > 0.01111)
return true;
}
return false;
}
[Fact]
public void PrunedOutputIsBitEqualToTheUnprunedSumNextToCellsJustInsideTheCut()
{
foreach (var p in PruneCutPoints)
{
Assert.True(HasCellJustInsideTheCut(p), $"no cell just inside the cut at {p}: the fixture no longer exercises the prune");
var pruned = math.phacelle(p, PruneCutSide, PruneCutOffset, PruneCutNormalization);
var unpruned = math.phacelle(p, PruneCutSide, PruneCutOffset, PruneCutNormalization, false);
AssertBitEqual(unpruned.cos, pruned.cos);
AssertBitEqual(unpruned.sin, pruned.sin);
}
}
private static void AssertBitEqual(float4 expected, float4 actual)
{
Assert.Equal(BitConverter.SingleToInt32Bits(expected.x), BitConverter.SingleToInt32Bits(actual.x));
Assert.Equal(BitConverter.SingleToInt32Bits(expected.y), BitConverter.SingleToInt32Bits(actual.y));
Assert.Equal(BitConverter.SingleToInt32Bits(expected.z), BitConverter.SingleToInt32Bits(actual.z));
Assert.Equal(BitConverter.SingleToInt32Bits(expected.w), BitConverter.SingleToInt32Bits(actual.w));
}
}

4
packages/cultmath/unity/org.gamecult.cultmath/LICENSE.md
View file

@ -4,7 +4,7 @@ CultMath is licensed under the Mozilla Public License 2.0 (`MPL-2.0`).
The complete license text is available from the Mozilla Foundation at
<https://www.mozilla.org/MPL/2.0/>. Source and modification history for the
precompiled assembly in this package are available in the GameCult/CultMath
repository at the corresponding package tag.
precompiled assembly in this package are available in the GameCult/CultLib
repository (`packages/cultmath`) at the corresponding package tag.
Third-party code retains its original license. See `THIRD-PARTY-NOTICES.md`.

2
packages/cultmath/unity/org.gamecult.cultmath/README.md
View file

@ -2,7 +2,7 @@
This is CultMath's Unity 2021.3-compatible Git-UPM surface. It contains the
precompiled `CultMath.dll`, portable symbols, the numeric
`Shaders/CultMath.hlsl` mirror, and the `CultMath.UnityBridge` assembly.
`Shaders/CultMath.hlsl` mirror (with its `CultMath.Phacelle.hlsl` include), and the `CultMath.UnityBridge` assembly.
`CultMath.UnityBridge` is the only C# source Unity compiles here. It holds the
UnityEngine conversions the engine-free core cannot declare: `ToCultMath()` on

99
packages/cultmath/unity/org.gamecult.cultmath/Shaders/CultMath.Phacelle.hlsl Normal file
View file

@ -0,0 +1,99 @@
#ifndef CULTMATH_PHACELLE_HLSL
#define CULTMATH_PHACELLE_HLSL
// This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of
// the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.
//
// Phacelle noise. Copyright (c) 2025 Rune Skovbo Johansen (Shadertoy t3dyWl; MPL-2.0), read from the
// C# port PhacelleNoise in lpmitchell/AdvancedTerrainErosion, converted to burstable C# by Luke
// Mitchell (2026). Modified by GameCult: 3D cells, a caller-supplied stripe wave vector, an exact
// gradient and exact pruning. This file is an MPL-2.0 file; CultMath.hlsl, which includes it after
// its own definitions of CULTMATH_TAU and cultmath_pcg3d, is MIT (see THIRD-PARTY-NOTICES.md).
// Invariant 8's struct return shape for cultmath_phacelle, compared field by field and bit for bit
// against CultMath.CultPhasor, same field order.
struct CultPhasor
{
float4 cos;
float4 sin;
};
// Johansen's Phacelle noise on 3D cells. See math.phacelle's comment (math.Phacelle.cs) for the stripe wave
// vector, the weight, the normalization, the exact gradient and why the prune is exact. Mirrors
// the C# body operation for operation; the C#-only internal overload that switches the prune off
// has no shader twin.
CultPhasor cultmath_phacelle(float3 p, float3 side, float offset, float normalization)
{
float3 cell = floor(p);
float3 local = p - cell;
float phaseOffset = offset * CULTMATH_TAU;
float sumW = 0.0;
float sumCos = 0.0;
float sumSin = 0.0;
float3 gradW = float3(0.0, 0.0, 0.0);
float3 gradCos = float3(0.0, 0.0, 0.0);
float3 gradSin = float3(0.0, 0.0, 0.0);
for (int dz = -1; dz <= 2; dz++)
{
float gz = max(0.0, abs(local.z - (float)dz) - 0.5);
for (int dy = -1; dy <= 2; dy++)
{
float gy = max(0.0, abs(local.y - (float)dy) - 0.5);
for (int dx = -1; dx <= 2; dx++)
{
float gx = max(0.0, abs(local.x - (float)dx) - 0.5);
if (gx * gx + gy * gy + gz * gz >= 2.25)
continue;
float3 gridStep = float3(dx, dy, dz);
int3 hash = cultmath_pcg3d(int3(cell + gridStep));
float3 jitter = float3(
((uint)hash.x >> 8) * (1.0 / 16777216.0) - 0.5,
((uint)hash.y >> 8) * (1.0 / 16777216.0) - 0.5,
((uint)hash.z >> 8) * (1.0 / 16777216.0) - 0.5);
float3 v = local - gridStep - jitter;
float falloff = exp(-2.0 * dot(v, v));
float w = max(0.0, falloff - 0.01111);
float3 dw = w > 0.0 ? (-4.0 * falloff) * v : float3(0.0, 0.0, 0.0);
float phase = dot(v, side) + phaseOffset;
float c = cos(phase);
float s = sin(phase);
sumW += w;
sumCos += w * c;
sumSin += w * s;
gradW += dw;
gradCos += dw * c - (w * s) * side;
gradSin += dw * s + (w * c) * side;
}
}
}
float invW = 1.0 / sumW;
float rawCos = sumCos * invW;
float rawSin = sumSin * invW;
float3 gradRawCos = (gradCos - rawCos * gradW) * invW;
float3 gradRawSin = (gradSin - rawSin * gradW) * invW;
float rawLength = sqrt(rawCos * rawCos + rawSin * rawSin);
float floorLength = 1.0 - normalization;
CultPhasor result;
if (rawLength > floorLength)
{
float outCos = rawCos / rawLength;
float outSin = rawSin / rawLength;
float3 gradLength = (rawCos * gradRawCos + rawSin * gradRawSin) / rawLength;
result.cos = float4((gradRawCos - outCos * gradLength) / rawLength, outCos);
result.sin = float4((gradRawSin - outSin * gradLength) / rawLength, outSin);
}
else
{
result.cos = float4(gradRawCos / floorLength, rawCos / floorLength);
result.sin = float4(gradRawSin / floorLength, rawSin / floorLength);
}
return result;
}
#endif

7
packages/cultmath/unity/org.gamecult.cultmath/Shaders/CultMath.Phacelle.hlsl.meta Normal file
View file

@ -0,0 +1,7 @@
fileFormatVersion: 2
guid: 5b1c7e0a94d24f6c8a3e2d17f60b9c48
ShaderIncludeImporter:
externalObjects: {}
userData:
assetBundleName:
assetBundleVariant:

2
packages/cultmath/unity/org.gamecult.cultmath/Shaders/CultMath.hlsl
View file

@ -503,6 +503,8 @@ CultCellular cultmath_cellular(float3 p)
return result;
}
#include "CultMath.Phacelle.hlsl"
float cultmath_value_noise(float2 position)
{
float2 cell = floor(position);

20
packages/cultmath/unity/org.gamecult.cultmath/THIRD-PARTY-NOTICES.md
View file

@ -19,3 +19,23 @@ permutation and scale as `snoise(float3)`.
- Copyright 2011 Ashima Arts.
- Upstream: <https://github.com/ashima/webgl-noise>
- License: MIT.
## Phacelle Noise
`math.phacelle` and `cultmath_phacelle` follow Rune Skovbo Johansen's Phacelle
noise (Shadertoy `t3dyWl`; erosion filter `wXcfWn`; blog, March 2026), read
from the C# port `PhacelleNoise` in `lpmitchell/AdvancedTerrainErosion`
(`AdvancedTerrainErosion.cs`, converted to burstable C# by Luke Mitchell,
2026). The cell weight, the phase blend and the normalization follow it. The
3D-cell generalisation, the caller-supplied stripe wave vector, the exact
gradient and the exact pruning are new here.
- Copyright (c) 2025 Rune Skovbo Johansen.
- Upstream: <https://github.com/lpmitchell/AdvancedTerrainErosion>
- License: MPL-2.0 (<https://mozilla.org/MPL/2.0/>). MPL-2.0 is file-level, so it
covers exactly the files that carry the port and nothing else:
`src/CultMath/math.Phacelle.cs`, `src/CultMath/CultPhasor.cs`,
`shaders/CultMath.Phacelle.hlsl` and the Unity copy of that include. Each
carries the MPL-2.0 header and this attribution. The rest of CultMath is MIT,
so the `GameCult.Math` NuGet package declares `MIT AND MPL-2.0` and ships this
file. The Unity package as a whole is MPL-2.0.

4
packages/cultmesh-py/src/cultmesh_py/facade.py
View file

@ -157,7 +157,7 @@ class CultMesh:
bind_host: str = "127.0.0.1",
bind_port: int = 0,
socket: socket_module.socket | None = None,
initial_sequence: int = 1,
initial_sequence: int | None = None,
resend_delay_ms: int = 250,
transport_id: str = "rudp",
max_payload_bytes: int | None = None,
@ -191,7 +191,7 @@ class CultMesh:
bind_host: str = "127.0.0.1",
bind_port: int = 0,
socket: socket_module.socket | None = None,
initial_sequence: int = 1,
initial_sequence: int | None = None,
resend_delay_ms: int = 250,
transport_id: str = "rudp",
max_payload_bytes: int | None = None,

18
packages/cultmesh-py/src/cultmesh_py/server.py
View file

@ -232,15 +232,19 @@ class CultMeshLocalServer:
now_ms = _now_ms()
peer = peers.get(remote_addr)
if packet.packet_type == CultNetRudpPacketType.CONNECT:
peer = _RudpPeerConnection(
CultNetRudpSession(
CultNetRudpSessionOptions(
connection_id=self.rudp_connection_id,
resend_delay_ms=self.rudp_resend_delay_ms,
# The session decides whether this Connect repeats the one it
# accepted. Any other Connect is a new client session, with new
# subscriptions.
if peer is None or not peer.session.connect_repeats(packet):
peer = _RudpPeerConnection(
CultNetRudpSession(
CultNetRudpSessionOptions(
connection_id=self.rudp_connection_id,
resend_delay_ms=self.rudp_resend_delay_ms,
)
)
)
)
peers[remote_addr] = peer
peers[remote_addr] = peer
self._send_rudp_packet(
rudp_socket,
remote_addr,

159
packages/cultmesh-py/tests/test_rudp_server_generation.py Normal file
View file

@ -0,0 +1,159 @@
"""The CultMesh local server's RUDP loop asks each peer's session whether a Connect
repeats. A retransmitted Connect keeps the peer; a restarted client on the same
address is a new one, with none of the old one's subscriptions."""
from __future__ import annotations
import socket
import time
import unittest
from dataclasses import dataclass
import msgpack
from cultcache_py import define_database_entry_type
from cultmesh_py import create_node
from cultmesh_py.server import CultMeshLocalServer
from cultnet_py import (
CultNetRudpSendOptions,
CultNetRudpSession,
CultNetRudpSessionOptions,
document_put_raw,
)
from cultnet_py.transport import CultNetRudpPacketType, decode_rudp_packet, encode_rudp_packet
CONNECTION_ID = 0x10203091
SCHEMA_ID = "mesh.rudp_generation_note.v1"
RELIABLE = CultNetRudpSendOptions(reliable=True, ordered=True)
@dataclass
class Note:
body: str
NOTE = define_database_entry_type(
"mesh.rudp_generation_note",
[("body", 0)],
cls=Note,
schema_id=SCHEMA_ID,
schema_name="mesh.rudp_generation_note",
schema_version=SCHEMA_ID,
)
def wire(message: dict) -> bytes:
return msgpack.packb(message, use_bin_type=True)
class RudpServerGenerationTests(unittest.TestCase):
def setUp(self) -> None:
node = create_node(runtime_id="rudp-generation")
node.database.register_document(NOTE)
self.server = CultMeshLocalServer(node=node, rudp_connection_id=CONNECTION_ID)
self.server.start()
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.sock.bind(("127.0.0.1", 0))
self.sock.settimeout(0.02)
self.target = ("127.0.0.1", self.server.port)
# Every schema data packet the server sent, by sequence, whether or not the receiving
# session delivered it: a new server-side session numbers its packets afresh.
self.sent: dict[int, str] = {}
def tearDown(self) -> None:
self.sock.close()
self.server.stop()
def to_server(self, packet) -> None:
self.sock.sendto(encode_rudp_packet(packet), self.target)
def pump(self, session: CultNetRudpSession, until, seconds: float = 2.0) -> bool:
deadline = time.monotonic() + seconds
while time.monotonic() < deadline:
if until():
return True
try:
data, _ = self.sock.recvfrom(65535)
except TimeoutError:
continue
packet = decode_rudp_packet(data)
session.receive(packet, int(time.monotonic() * 1000))
if packet.reliable or packet.packet_type == CultNetRudpPacketType.ACCEPT:
self.to_server(session.create_ack_for_received(packet.sequence))
if packet.packet_type == CultNetRudpPacketType.DATA and packet.channel_id == "schema":
self.sent[packet.sequence] = msgpack.unpackb(packet.payload, raw=False).get("schemaVersion")
return until()
def count(self, schema_version: str) -> int:
return sum(1 for value in self.sent.values() if value == schema_version)
def new_session(self, initial_sequence: int) -> CultNetRudpSession:
return CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=initial_sequence))
def test_a_retransmitted_connect_keeps_the_peer_and_a_restarted_client_is_a_new_one(self) -> None:
first = self.new_session(50_000)
connect = first.create_connect(0, b"same")
self.to_server(connect)
self.assertTrue(self.pump(first, lambda: first.connected), "the first Accept")
hello = first.send(
"schema", wire({"schemaVersion": "cultnet.hello.v0", "runtimeId": "probe", "runtimeKind": "test"}), RELIABLE
)
self.to_server(hello)
self.assertTrue(self.pump(first, lambda: self.count("cultnet.hello.v0") == 1), "the first answer")
self.to_server(
first.send(
"schema",
wire({
"schemaVersion": "cultnet.database_subscribe.v0",
"messageId": "subscribe",
"subscriptionId": "old-incarnation",
"schemaIds": [SCHEMA_ID],
"includeSnapshot": False,
}),
RELIABLE,
)
)
self.pump(first, lambda: False, 0.2)
first_put = document_put_raw(
message_id="first-put",
key="note:a",
schema_id=SCHEMA_ID,
stored_at="2026-09-30T00:00:00Z",
payload=NOTE.encode_payload(Note("first")),
)
self.to_server(first.send("schema", first_put.to_bytes(), RELIABLE))
self.assertTrue(
self.pump(first, lambda: self.count("cultnet.database_change_raw.v0") == 1),
"the subscription did not notify its own connection",
)
# A retransmitted Connect repeats: the data already delivered is not delivered, or answered, again.
self.to_server(connect)
self.to_server(hello)
self.pump(first, lambda: False, 0.4)
self.assertEqual(self.count("cultnet.hello.v0"), 1, "the retransmitted Connect reset the peer")
# The same client restarted on the same address, connection id and payload: a new peer.
second = self.new_session(7)
self.to_server(second.create_connect(0, b"same"))
self.assertTrue(self.pump(second, lambda: second.connected), "the restarted client's Accept")
self.to_server(
second.send(
"schema", wire({"schemaVersion": "cultnet.hello.v0", "runtimeId": "probe", "runtimeKind": "test"}), RELIABLE
)
)
self.assertTrue(self.pump(second, lambda: self.count("cultnet.hello.v0") == 2), "the restarted client's answer")
# It inherited none of the old incarnation's subscriptions.
put = document_put_raw(
message_id="second-put",
key="note:a",
schema_id=SCHEMA_ID,
stored_at="2026-09-30T00:00:01Z",
payload=NOTE.encode_payload(Note("second")),
)
self.to_server(second.send("schema", put.to_bytes(), RELIABLE))
self.pump(second, lambda: False, 0.4)
self.assertEqual(self.count("cultnet.database_change_raw.v0"), 1, "the new peer inherited a subscription")
if __name__ == "__main__":
unittest.main()

4
packages/cultmesh-rs/src/lib.rs
View file

@ -961,7 +961,7 @@ fn connect_rudp_client(
mode: cultnet_rs::CultNetRudpSocketMode::Client,
remote_addr: Some(target),
connection_id: fresh_rudp_connection_epoch(),
initial_sequence: 1,
initial_sequence: None,
resend_delay_ms,
transport_id: Some(transport_id.to_string()),
max_payload_bytes: None,
@ -1406,7 +1406,7 @@ mod tests {
let mut session =
cultnet_rs::CultNetRudpSession::new(cultnet_rs::CultNetRudpSessionOptions {
connection_id: connect.connection_id,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 5,
max_pending_reliable_packets: Some(64),
});

16
packages/cultmesh-rs/src/rudp_document_server.rs
View file

@ -436,12 +436,20 @@ where
packet: &CultNetRudpPacket,
now: u64,
) -> Result<bool> {
// The session decides whether a Connect repeats the one it accepted. Any
// other Connect from this key is a new client session: the old one ends
// with its budget, and the new one starts fresh.
if self
.sessions
.get(&key)
.is_some_and(|entry| !entry.session.connect_repeats(packet))
{
self.sessions.remove(&key);
}
let reply = match self.sessions.get_mut(&key) {
// The same key is a retransmitted Connect for the existing epoch.
// A genuinely fresh client incarnation must choose a fresh id.
Some(entry) => {
entry.last_activity_monotonic_millis = now;
match entry.session.answer_repeated_connect(packet, now) {
match entry.session.accept_connect(packet, now, Vec::new()) {
Ok(reply) => reply,
Err(_) => {
self.end_refused_session(key)?;
@ -455,7 +463,7 @@ where
}
let mut session = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: key.connection_id,
initial_sequence: 1,
initial_sequence: None,
resend_delay_ms: duration_millis(self.options.resend_delay),
max_pending_reliable_packets: Some(
self.options.max_pending_reliable_packets_per_session,

107
packages/cultmesh-rs/tests/rudp_document_server.rs
View file

@ -160,7 +160,15 @@ fn server(
}
fn client(target: SocketAddr, id: u32) -> Result<CultNetRudpSocketTransportConnection> {
let socket = UdpSocket::bind("127.0.0.1:0")?;
client_on(UdpSocket::bind("127.0.0.1:0")?, target, id, 1)
}
fn client_on(
socket: UdpSocket,
target: SocketAddr,
id: u32,
initial_sequence: u32,
) -> Result<CultNetRudpSocketTransportConnection> {
socket.set_nonblocking(true)?;
CultNetRudpSocketTransportConnection::new(CultNetRudpSocketTransportOptions {
media_delivery: None,
@ -169,7 +177,7 @@ fn client(target: SocketAddr, id: u32) -> Result<CultNetRudpSocketTransportConne
mode: CultNetRudpSocketMode::Client,
remote_addr: Some(target),
connection_id: id,
initial_sequence: 1,
initial_sequence: Some(initial_sequence),
resend_delay_ms: 10,
transport_id: None,
max_payload_bytes: None,
@ -291,7 +299,7 @@ fn same_connection_id_is_peer_scoped_and_raw_bytes_are_untouched() -> Result<()>
}
#[test]
fn duplicate_connect_preserves_epoch_and_fresh_epoch_is_separate() -> Result<()> {
fn a_second_connect_from_a_client_replaces_its_session_and_a_fresh_epoch_is_separate() -> Result<()> {
let clock = Clock::new(45_000);
let sink = Sink::default();
let mut server = server(Default::default(), clock, sink.clone(), Source::default())?;
@ -325,6 +333,97 @@ fn duplicate_connect_preserves_epoch_and_fresh_epoch_is_separate() -> Result<()>
Ok(())
}
/// A client that restarts on the same address and connection id starts a new
/// session: its Connect carries a sequence the accepted one did not, so the
/// server does not mistake it for a retransmit, and what it sends is delivered.
#[test]
fn a_restarted_client_on_the_same_address_and_connection_id_is_admitted() -> Result<()> {
let sink = Sink::default();
let mut server = server(Default::default(), Clock::new(46_000), sink.clone(), Source::default())?;
let target = server.local_addr()?;
let shared = UdpSocket::bind("127.0.0.1:0")?;
let twin = shared.try_clone()?;
let mut first = client_on(shared, target, 93, 50_000)?;
connect(&mut server, &mut [&mut first])?;
for index in 0..3 {
send(
&mut first,
&CultNetMessage::DocumentPutRaw {
message_id: format!("before-{index}"),
document: document("before", vec![index]),
},
)?;
}
for _ in 0..40 {
server.poll_once()?;
first.receive_once()?;
}
assert_eq!(sink.0.lock().unwrap().receipts.len(), 3);
let mut restarted = client_on(twin, target, 93, 7)?;
connect(&mut server, &mut [&mut restarted])?;
assert_eq!(server.session_count(), 1);
send(
&mut restarted,
&CultNetMessage::DocumentPutRaw {
message_id: "after-restart".into(),
document: document("after", vec![9]),
},
)?;
for _ in 0..40 {
server.poll_once()?;
restarted.receive_once()?;
}
let receipts = sink.0.lock().unwrap();
assert!(
receipts.receipts.iter().any(|r| r.message_id == "after-restart"),
"the restarted client's frame was not delivered"
);
Ok(())
}
/// A restarted client is a new session, and starts with a new payload budget:
/// what the old one spent is not charged to it.
#[test]
fn a_restarted_client_starts_with_a_fresh_payload_budget() -> Result<()> {
let put = |id: &str| CultNetMessage::DocumentPutRaw {
message_id: id.into(),
document: document("budget", vec![7; 64]),
};
let encoded = encode_cultnet_message_to_vec(&put("put-a"), CultNetWireContract::CultNetSchemaV0)?;
let options = CultMeshRudpDocumentServerOptions {
max_admitted_payload_bytes: encoded.len() * 2,
max_admitted_payload_bytes_per_session: encoded.len(),
max_snapshot_response_bytes: encoded.len(),
..Default::default()
};
let sink = Sink::default();
let mut server = server(options, Clock::new(47_000), sink.clone(), Source::default())?;
let target = server.local_addr()?;
let shared = UdpSocket::bind("127.0.0.1:0")?;
let twin = shared.try_clone()?;
let mut first = client_on(shared, target, 94, 50_000)?;
connect(&mut server, &mut [&mut first])?;
send(&mut first, &put("put-a"))?;
for _ in 0..20 {
server.poll_once()?;
}
assert_eq!(sink.0.lock().unwrap().receipts.len(), 1);
let mut restarted = client_on(twin, target, 94, 7)?;
connect(&mut server, &mut [&mut restarted])?;
send(&mut restarted, &put("put-b"))?;
for _ in 0..20 {
server.poll_once()?;
}
assert_eq!(
sink.0.lock().unwrap().receipts.len(),
2,
"the restarted client was charged what the old session spent"
);
Ok(())
}
#[test]
fn application_rejection_is_nonfatal_peer_scoped_and_unacknowledged() -> Result<()> {
let clock = Clock::new(48_000);
@ -576,7 +675,7 @@ fn a_connect_storm_and_stray_frames_are_dropped_not_fatal() -> Result<()> {
let session = |id| {
CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: id,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 10,
max_pending_reliable_packets: None,
})

12
packages/cultmesh-ts/src/index.ts
View file

@ -5793,7 +5793,7 @@ export class CultMesh {
if (packet.packetType === "connect") {
const connectPayload = Uint8Array.from(packet.payload ?? []);
if (record && byteArraysEqual(record.connectPayload, connectPayload)) {
if (record?.session.connectRepeats(packet)) {
socket.send(
encodeRudpPacket(record.session.acceptConnect(packet, nowMs)),
remote.port,
@ -5846,7 +5846,7 @@ export class CultMesh {
}
if (packet.reliable) {
socket.send(
encodeRudpPacket(record.session.createAckFor(packet.sequence)),
encodeRudpPacket(record.session.createAckForReceived(packet.sequence)),
record.remote.port,
record.remote.address,
);
@ -6671,14 +6671,6 @@ function nonBlankOr(value?: string, fallback = ""): string {
return value && value.trim() ? value : fallback;
}
function byteArraysEqual(left: Uint8Array, right: Uint8Array): boolean {
if (left.byteLength !== right.byteLength) return false;
for (let index = 0; index < left.byteLength; index += 1) {
if (left[index] !== right[index]) return false;
}
return true;
}
function parseCultMeshLocalityKind(
value: string | undefined,
fallback: CultMeshLocalityKind,

75
packages/cultmesh-ts/test/provider-rudp-transport.test.ts
View file

@ -8,6 +8,7 @@ import {
decodeRudpPacket,
encodeRudpPacket,
type CultNetOperationRequestMessage,
type CultNetRudpPacket,
} from "cultnet-ts";
import {
@ -320,6 +321,80 @@ test("a replaced RUDP generation drops operations queued behind an active handle
}
});
test("a client restarted on the same address with the same Connect payload replaces its RUDP session, and a retransmitted Connect does not", async () => {
const connectionId = 0x43554c54;
let closedSessions = 0;
const server = CultMesh.createRudpDocumentServer("odin-restart-test", connectionId, {
bindPort: 0,
documents: new CultNetDocumentRegistry(),
onSessionClosed: () => { closedSessions += 1; },
onError: () => undefined,
});
await server.start();
const socket = createSocket("udp4");
const received: CultNetRudpPacket[] = [];
socket.on("message", wire => received.push(decodeRudpPacket(wire)));
const toServer = (packet: CultNetRudpPacket) =>
socket.send(encodeRudpPacket(packet), server.bind.port, "127.0.0.1");
const evidence = Buffer.from("same connect payload");
try {
const first = new CultNetRudpSession({ connectionId, initialSequence: 50_000 });
const connect = first.createConnect(Date.now(), evidence);
toServer(connect);
await waitFor(() => received.some(packet => packet.packetType === "accept"), "the first Accept");
// A retransmitted Connect repeats: the session stays.
toServer(connect);
await new Promise(resolve => setTimeout(resolve, 100));
assert.equal(closedSessions, 0, "a retransmitted Connect closed the session");
// The same client restarted: another sequence, the same payload.
received.length = 0;
const second = new CultNetRudpSession({ connectionId, initialSequence: 7 });
toServer(second.createConnect(Date.now(), evidence));
await waitFor(() => closedSessions === 1, "the old RUDP session to close");
await waitFor(() => received.some(packet => packet.packetType === "accept"), "the restarted client's Accept");
} finally {
socket.close();
server.close();
}
});
test("the RUDP document server does not acknowledge by name a reliable packet its session did not receive", async () => {
const connectionId = 0x43554c55;
const server = CultMesh.createRudpDocumentServer("odin-ack-test", connectionId, {
bindPort: 0,
documents: new CultNetDocumentRegistry(),
onError: () => undefined,
});
await server.start();
const socket = createSocket("udp4");
const received: CultNetRudpPacket[] = [];
socket.on("message", wire => received.push(decodeRudpPacket(wire)));
const toServer = (packet: CultNetRudpPacket) =>
socket.send(encodeRudpPacket(packet), server.bind.port, "127.0.0.1");
try {
const client = new CultNetRudpSession({ connectionId, initialSequence: 50_000 });
const connect = client.createConnect(Date.now());
toServer(connect);
await waitFor(() => received.some(packet => packet.packetType === "accept"), "the Accept");
received.length = 0;
// A reliable packet the session ignores (a client does not send an Accept).
// The session never received it, so the Ack must not name it.
const ignored: CultNetRudpPacket = { ...client.createAck(), packetType: "accept", reliable: true, sequence: connect.sequence + 5 };
toServer(ignored);
await waitFor(() => received.some(packet => packet.packetType === "ack"), "the acknowledgement");
const ack = received.find(packet => packet.packetType === "ack")!;
const names = ack.ack === ignored.sequence
|| Array.from({ length: 32 }, (_, bit) => bit).some(bit => (ack.ackMask & (1 << bit)) !== 0 && ack.ack - bit - 1 === ignored.sequence);
assert.equal(names, false, "the server acknowledged by name a packet its session refused");
} finally {
socket.close();
server.close();
}
});
function deferred<T>(): { promise: Promise<T>; resolve(value: T): void } {
let resolve!: (value: T) => void;
return { promise: new Promise<T>(done => resolve = done), resolve };

2
packages/cultnet-py/src/cultnet_py/client.py
View file

@ -393,7 +393,7 @@ def create_rudp_schema_transport(
runtime_id: str = "cultnet-python-rudp-client",
bind_host: str = "127.0.0.1",
bind_port: int = 0,
initial_sequence: int = 1,
initial_sequence: int | None = None,
resend_delay_ms: int = 25,
transport_id: str = "rudp",
max_payload_bytes: int | None = None,

425
packages/cultnet-py/src/cultnet_py/transport.py
View file

@ -1,5 +1,6 @@
from __future__ import annotations
import secrets
import socket
import threading
import time
@ -292,10 +293,45 @@ class CultNetRudpReceiveResult:
disconnect_reason: bytes = b""
# How long a client keeps retransmitting a Connect nobody answered before it
# abandons that attempt for a fresh one. A Connect the server answers with an
# Ack, never an Accept, is one the server judged stale (see `accept_connect`);
# retransmitting the same sequence would never change its mind.
_RUDP_CONNECT_ATTEMPT_MS = 3_000
def _draw_sequence() -> int:
"""A sequence drawn from a secure source in [1, 2^31). The Connect's
sequence is what tells a peer whether a Connect repeats one it already
accepted or starts a new session, so two sessions must not share one by
default."""
return 1 + secrets.randbelow(2**31 - 1)
def _at_or_before(sequence: int, mark: int, window: int) -> bool:
"""Whether `sequence` is at or before `mark` in serial order, within the
receive window. The compare is modular, so it holds across the wrap of the
32-bit space; a sequence further back than the window is the duplicate
test's below-window clause, and one ahead of the mark is never before it."""
return ((mark - sequence) & 0xFFFFFFFF) < window
def _packet_acknowledges(packet: "CultNetRudpPacket", sequence: int) -> bool:
if packet.ack == sequence:
return True
return any(
packet.ack_mask & (1 << bit) and packet.ack > bit and packet.ack - bit - 1 == sequence
for bit in range(32)
)
@dataclass(frozen=True)
class CultNetRudpSessionOptions:
connection_id: int
initial_sequence: int = 1
# The first sequence this session issues. Unset, the session draws its own
# at random: every session is a new identity to its peer, and an options
# object reused for several must not give them one sequence.
initial_sequence: int | None = None
resend_delay_ms: int = 250
max_pending_reliable_packets: int | None = None
@ -320,7 +356,7 @@ class CultNetRudpSocketTransportOptions:
mode: CultNetRudpSocketMode
connection_id: int
remote_addr: tuple[str, int] | None = None
initial_sequence: int = 1
initial_sequence: int | None = None
resend_delay_ms: int = 250
transport_id: str = "rudp"
max_payload_bytes: int | None = None
@ -409,20 +445,49 @@ class CultNetRudpSession:
if options.max_pending_reliable_packets is not None and options.max_pending_reliable_packets <= 0:
raise ValueError("RUDP max_pending_reliable_packets must be greater than zero")
self.max_pending_reliable_packets = options.max_pending_reliable_packets
self._next_sequence = _uint32(options.initial_sequence, "initial_sequence")
self._next_sequence = _uint32(
_draw_sequence() if options.initial_sequence is None else options.initial_sequence,
"initial_sequence",
)
if self._next_sequence == 0xFFFFFFFF:
raise ValueError("RUDP initial_sequence must leave room for a reliable packet")
self._next_sequenced_by_channel: dict[str, int] = {}
self._next_fragment_id = 1
self._connected = False
# Advances every time a generation ends. Everything issued in a
# generation (writes, flushes) belongs to it and dies with it.
self._generation = 0
# True from the end of a generation until the next Connect or Accept
# begins one. A flush started in that interval has no live generation
# to wait on.
self._ended = False
# The sequence of the Connect that started the current generation: sent
# by this side, or accepted from the peer. A Connect repeats exactly when
# the session is connected and the Connect carries this sequence.
self._connect_sequence: int | None = None
# A Connect this side sent is unanswered. Only then is an Accept honoured.
self._awaiting_accept = False
# When the unanswered Connect first went out; `due_resends` abandons it
# for a fresh attempt once _RUDP_CONNECT_ATTEMPT_MS has passed.
self._connect_started_at_ms = 0
# The payload of the Connect this side sent, kept so a fresh attempt can
# always be built whatever became of the pending packet.
self._connect_payload = b""
self._last_received_at_ms: int | None = None
self._highest_received_sequence: int | None = None
self._received_sequences: set[int] = set()
self._latest_sequenced_by_channel: dict[str, int] = {}
self._pending_reliable: dict[int, _PendingReliablePacket] = {}
self._queued_reliable: deque[CultNetRudpPacket] = deque()
self._ordered_next_sequence_by_channel: dict[str, int] = {}
self._ordered_buffers: dict[str, dict[int, tuple[CultNetRudpDeliveredFrame, int]]] = {}
# Every reliable sequence up to and including this one has been received
# since the peer state was last reset. Only the handshake seeds it: the
# peer's Connect on the accepting side, the peer's Accept on the
# connecting side. Reliable data that arrives before that is refused.
self._received_through: int | None = None
# Ordered frames received but not yet deliverable, keyed by first
# sequence. A frame is held for exactly one reason: a reliable sequence
# below it has not arrived.
self._ordered_held: dict[int, CultNetRudpDeliveredFrame] = {}
self._fragment_buffers: dict[tuple[str, int], dict[str, Any]] = {}
# Matches cultnet-rs max_pending_fragment_sets. A set that loses one
# fragment is never completed; without a bound it is kept for the life
@ -455,23 +520,49 @@ class CultNetRudpSession:
def outstanding_reliable_packet_count(self) -> int:
return len(self._pending_reliable) + len(self._queued_reliable)
def reset_peer_state(self) -> None:
"""Forgets everything learned from the peer; sequence numbers already issued stay issued."""
@property
def generation(self) -> int:
return self._generation
@property
def ended(self) -> bool:
return self._ended
def _end_session(self) -> None:
"""The one way a session generation ends. The session stops being
connected and what it still owed the peer dies with it: a write not yet
acknowledged is dropped, so no later session retransmits it or credits
an ack to it. What was learned from the peer is not touched: the peer
may not know the session ended, and forgetting what it sent would let its
retransmits be delivered twice."""
self._connected = False
self._ended = True
self._awaiting_accept = False
self._generation += 1
self._pending_reliable.clear()
self._queued_reliable.clear()
def reset_peer_state(self) -> None:
"""Ends the current generation and forgets everything learned from the peer; sequence numbers already issued stay issued."""
self._end_session()
self._last_received_at_ms = None
self._highest_received_sequence = None
self._received_sequences.clear()
self._next_sequenced_by_channel.clear()
self._latest_sequenced_by_channel.clear()
self._pending_reliable.clear()
self._queued_reliable.clear()
self._ordered_next_sequence_by_channel.clear()
self._ordered_buffers.clear()
self._received_through = None
self._ordered_held.clear()
self._fragment_buffers.clear()
self._fragment_sets_evicted = 0
def create_connect(self, now_ms: int = 0, payload: bytes = b"") -> CultNetRudpPacket:
# A session that has had a peer starts a new generation: nothing it
# learned from that peer describes the one this Connect reaches, and
# what it still owed that peer no longer takes room in the queue.
if self._connect_sequence is not None:
self.reset_peer_state()
self._ensure_reliable_capacity(1)
self._ended = False
packet = self._create_packet(
CultNetRudpPacketType.CONNECT,
"control",
@ -479,22 +570,75 @@ class CultNetRudpSession:
reliable=True,
ordered=True,
)
self._connect_sequence = packet.sequence
self._awaiting_accept = True
self._connect_started_at_ms = now_ms
self._connect_payload = packet.payload
self._track_reliable(packet, now_ms)
return packet
def connect_repeats(self, packet: CultNetRudpPacket) -> bool:
"""Whether `packet` is a Connect this session's current generation
already owns: a retransmit of the Connect that started it (the session
is connected and the sequence is that Connect's), or a stale copy of an
earlier attempt by the same client (a sequence before it, within the
receive window). Servers that keep one session per peer ask this to tell
a Connect that starts a new session from one that does not; anything
else that reaches `accept_connect` starts a new generation."""
return (
packet.packet_type == CultNetRudpPacketType.CONNECT
and self._connected
and self._connect_sequence is not None
and (
self._connect_sequence == packet.sequence
or self._connect_is_stale(packet.sequence, self._connect_sequence)
)
)
def _connect_is_stale(self, sequence: int, current: int) -> bool:
"""A Connect that precedes the current generation's within the receive
window is the client's earlier attempt, delayed in the network: a client
that retried never sends a lower sequence again. Restarting on it would
strand the client, which honours only the Accept for its newest Connect.
The rule is TCP's answer to a delayed SYN (RFC 5961's challenge ACK):
keep the connection and answer with an Ack. A restarted client whose
random initial sequence lands in this window is answered the same way
and abandons the attempt for a fresh draw (_RUDP_CONNECT_ATTEMPT_MS)."""
return sequence != current and _at_or_before(sequence, current, self.RECEIVED_SEQUENCE_WINDOW)
def accept_connect(
self,
packet: CultNetRudpPacket,
now_ms: int = 0,
payload: bytes = b"",
) -> CultNetRudpPacket:
"""Answers a Connect. A repeat of the accepted one queues nothing, so a
Connect storm cannot grow the reliable queue: the reply is the Accept
still awaiting acknowledgement, or an Ack once it was acknowledged. A
stale copy of an earlier attempt gets the same reply and changes nothing
else: it is not evidence the peer is alive. Any other Connect ends the
current generation, forgets the peer and accepts a new one."""
self._require_connection(packet)
if packet.packet_type != CultNetRudpPacketType.CONNECT:
raise ValueError(f"Expected RUDP connect packet, got {packet.packet_type.value}")
if self.connect_repeats(packet):
if self._connect_sequence == packet.sequence:
self._apply_acknowledgements(packet)
self._remember_received(packet.sequence)
self._last_received_at_ms = now_ms
for pending in self._pending_reliable.values():
if pending.packet.packet_type == CultNetRudpPacketType.ACCEPT:
pending.last_sent_at_ms = now_ms
return pending.packet
return self.create_ack()
self.reset_peer_state()
self._ensure_reliable_capacity(1)
self._remember_received(packet.sequence)
self._seed_received(packet.sequence)
self._last_received_at_ms = now_ms
self._connect_sequence = packet.sequence
self._connected = True
self._ended = False
response = self._create_packet(
CultNetRudpPacketType.ACCEPT,
"control",
@ -505,25 +649,6 @@ class CultNetRudpSession:
self._track_reliable(response, now_ms)
return response
def answer_repeated_connect(self, packet: CultNetRudpPacket, now_ms: int = 0) -> CultNetRudpPacket:
"""Answers a Connect from a peer this session already accepted.
The packet's sequence is remembered and nothing is queued, so a Connect
storm cannot grow the reliable queue. The reply is the Accept still awaiting
acknowledgement, or an Ack once it was acknowledged.
"""
self._require_connection(packet)
if packet.packet_type != CultNetRudpPacketType.CONNECT:
raise ValueError(f"Expected RUDP connect packet, got {packet.packet_type.value}")
self._apply_acknowledgements(packet)
self._remember_received(packet.sequence)
self._last_received_at_ms = now_ms
for pending in self._pending_reliable.values():
if pending.packet.packet_type == CultNetRudpPacketType.ACCEPT:
pending.last_sent_at_ms = now_ms
return pending.packet
return self.create_ack()
def send(
self,
channel_id: str,
@ -589,17 +714,28 @@ class CultNetRudpSession:
def receive(self, packet: CultNetRudpPacket, now_ms: int = 0) -> CultNetRudpReceiveResult:
self._require_connection(packet)
self._apply_acknowledgements(packet)
# An Accept counts only while this side's Connect is unanswered and the
# Accept names it. A late or duplicate one, or one from an earlier
# generation, must not seed the watermark or revive an ended session.
honours_accept = (
packet.packet_type == CultNetRudpPacketType.ACCEPT
and self._awaiting_accept
and self._connect_sequence is not None
and _packet_acknowledges(packet, self._connect_sequence)
)
if packet.packet_type == CultNetRudpPacketType.ACCEPT and not honours_accept:
return CultNetRudpReceiveResult()
# While this side's Connect awaits its Accept, only the Accept it
# honours retires it: an Ack that names the Connect (a server's reply to
# a repeat or a stale copy) says the server did not start a session.
if honours_accept or not self._awaiting_accept:
self._apply_acknowledgements(packet)
ready_to_send = self._promote_queued_reliable(now_ms)
self._last_received_at_ms = now_ms
expected_sequence_if_uninitialized = (
packet.sequence
if self._highest_received_sequence is None
else self._highest_received_sequence + 1
)
if packet.packet_type == CultNetRudpPacketType.ACCEPT:
self._remember_received(packet.sequence)
if honours_accept:
self._awaiting_accept = False
self._seed_received(packet.sequence)
self._connected = True
return CultNetRudpReceiveResult(ready_to_send=ready_to_send)
@ -617,7 +753,7 @@ class CultNetRudpSession:
)
if packet.packet_type == CultNetRudpPacketType.DISCONNECT:
self._connected = False
self._end_session()
return CultNetRudpReceiveResult(
ready_to_send=ready_to_send,
disconnected=True,
@ -627,37 +763,38 @@ class CultNetRudpSession:
if packet.packet_type != CultNetRudpPacketType.DATA:
return CultNetRudpReceiveResult(ready_to_send=ready_to_send)
duplicate = packet.reliable and (
packet.sequence in self._received_sequences or (
self._highest_received_sequence is not None
and packet.sequence < self._highest_received_sequence
and self._highest_received_sequence - packet.sequence >= self.RECEIVED_SEQUENCE_WINDOW
)
)
# Reliable data before the handshake has seeded the watermark has no place
# in the order: refuse it unremembered, so it is not acknowledged and the
# sender retransmits it once the handshake is done.
if packet.reliable and self._received_through is None:
return CultNetRudpReceiveResult(ready_to_send=ready_to_send)
duplicate = packet.reliable and self._was_received(packet.sequence)
if packet.reliable:
self._remember_received(packet.sequence)
if duplicate:
return CultNetRudpReceiveResult(ready_to_send=ready_to_send)
delivered: list[CultNetRudpDeliveredFrame] = []
reassembled = self._reassemble(packet)
if reassembled is None:
return CultNetRudpReceiveResult(ready_to_send=ready_to_send)
frame, ordered, next_sequence = reassembled
if not ordered and packet.sequenced:
newest_sequence = next_sequence - 1
latest_sequence = self._latest_sequenced_by_channel.get(frame.channel_id)
if latest_sequence is not None and newest_sequence <= latest_sequence:
return CultNetRudpReceiveResult(ready_to_send=ready_to_send)
self._latest_sequenced_by_channel[frame.channel_id] = newest_sequence
return CultNetRudpReceiveResult(delivered=(frame,), ready_to_send=ready_to_send)
if not ordered:
return CultNetRudpReceiveResult(delivered=(frame,), ready_to_send=ready_to_send)
return CultNetRudpReceiveResult(
delivered=tuple(
self._deliver_ordered(frame, next_sequence, expected_sequence_if_uninitialized)
),
ready_to_send=ready_to_send,
)
if reassembled is not None:
frame, ordered, next_sequence = reassembled
if ordered:
self._ordered_held[frame.sequence] = frame
elif not packet.sequenced:
delivered.append(frame)
else:
newest_sequence = next_sequence - 1
latest_sequence = self._latest_sequenced_by_channel.get(frame.channel_id)
if latest_sequence is None or newest_sequence > latest_sequence:
self._latest_sequenced_by_channel[frame.channel_id] = newest_sequence
delivered.append(frame)
# Any reliable packet may have advanced the watermark, a fragment or an
# unordered frame as much as an ordered one, so the drain runs after all
# of them.
delivered.extend(self._drain_ordered())
return CultNetRudpReceiveResult(delivered=tuple(delivered), ready_to_send=ready_to_send)
def create_ack(self) -> CultNetRudpPacket:
ack, ack_mask = self._ack_state()
@ -683,7 +820,11 @@ class CultNetRudpSession:
)
def create_ack_for_received(self, sequence: int) -> CultNetRudpPacket:
# What `receive` refused is not acknowledged by name (the ack carries only
# what was received), so its sender retransmits it.
received_sequence = _uint32(sequence, "received sequence")
if not self._was_received(received_sequence):
return self.create_ack()
ack, _ = self._ack_state()
if ack >= received_sequence and ack - received_sequence <= 32:
return self.create_ack()
@ -693,7 +834,7 @@ class CultNetRudpSession:
return self._create_packet(CultNetRudpPacketType.PING, "control", payload)
def create_disconnect(self, reason: bytes = b"") -> CultNetRudpPacket:
self._connected = False
self._end_session()
return self._create_packet(CultNetRudpPacketType.DISCONNECT, "control", reason)
def check_timeout(self, now_ms: int, timeout_ms: int) -> bool:
@ -701,10 +842,13 @@ class CultNetRudpSession:
return False
if now_ms - self._last_received_at_ms <= timeout_ms:
return False
self._connected = False
self._end_session()
return True
def due_resends(self, now_ms: int) -> tuple[CultNetRudpPacket, ...]:
fresh = self._abandon_unanswered_connect(now_ms)
if fresh is not None:
return (fresh,)
due: list[CultNetRudpPacket] = []
for pending in self._pending_reliable.values():
if now_ms - pending.last_sent_at_ms >= self.resend_delay_ms:
@ -712,6 +856,24 @@ class CultNetRudpSession:
due.append(pending.packet)
return tuple(sorted(due, key=lambda packet: packet.sequence))
def _abandon_unanswered_connect(self, now_ms: int) -> CultNetRudpPacket | None:
"""A Connect unanswered for _RUDP_CONNECT_ATTEMPT_MS is replaced, not
retransmitted further, by a Connect with the same payload and the
abandoned sequence plus the receive window less one. Nothing was ever
sent in an unanswered generation, so no sequence issued so far is owed.
The jump keeps a late copy of the abandoned Connect inside the new one's
stale window, so a server that took the new Connect answers the copy
with an Ack instead of restarting; and it leaves the stale window of
whatever generation the server holds, unless that generation sits
exactly at the jump, and then the next attempt leaves it. The first
Connect of a session is the only one drawn at random."""
if not self._awaiting_accept or now_ms - self._connect_started_at_ms < _RUDP_CONNECT_ATTEMPT_MS:
return None
if self._connect_sequence is None:
return None
self._next_sequence = min(self._connect_sequence + self.RECEIVED_SEQUENCE_WINDOW - 1, 0xFFFFFFFE)
return self.create_connect(now_ms, self._connect_payload)
def _create_packet(
self,
packet_type: CultNetRudpPacketType,
@ -806,8 +968,39 @@ class CultNetRudpSession:
if packet.ack_mask & (1 << bit):
self._pending_reliable.pop(packet.ack - bit - 1, None)
def _seed_received(self, sequence: int) -> None:
"""The handshake's one act on the watermark: the peer's Connect or Accept
is the first sequence of the session, whatever else it has sent."""
self._received_through = sequence
self._remember_received(sequence)
def _was_received(self, sequence: int) -> bool:
"""The one duplicate test: true for a sequence in the window that was
received, for one below the window, and for one at or before the
watermark. The watermark starts at the handshake's seed, so a frame the
peer sent in an earlier generation (every sequence it issued is below the
Connect that began this one) is a duplicate of something already
delivered and is acknowledged, never delivered again."""
return (
sequence in self._received_sequences
or (
self._highest_received_sequence is not None
and sequence < self._highest_received_sequence
and self._highest_received_sequence - sequence >= self.RECEIVED_SEQUENCE_WINDOW
)
or (
self._received_through is not None
and _at_or_before(sequence, self._received_through, self.RECEIVED_SEQUENCE_WINDOW)
)
)
def _remember_received(self, sequence: int) -> None:
self._received_sequences.add(sequence)
if self._received_through is not None:
through = self._received_through
while through + 1 in self._received_sequences:
through += 1
self._received_through = through
if self._highest_received_sequence is None or sequence > self._highest_received_sequence:
self._highest_received_sequence = sequence
if len(self._received_sequences) > self.RECEIVED_SEQUENCE_WINDOW:
@ -865,58 +1058,15 @@ class CultNetRudpSession:
max(sequences) + 1,
)
def _deliver_ordered(
self,
frame: CultNetRudpDeliveredFrame,
next_after_frame: int,
expected_sequence_if_uninitialized: int,
) -> list[CultNetRudpDeliveredFrame]:
next_sequence = self._ordered_next_sequence_by_channel.get(frame.channel_id)
if next_sequence is None:
next_sequence = min(expected_sequence_if_uninitialized, frame.sequence)
self._ordered_next_sequence_by_channel[frame.channel_id] = next_sequence
while (
frame.sequence > next_sequence
and next_sequence in self._received_sequences
and next_sequence not in self._ordered_buffers.get(frame.channel_id, {})
):
next_sequence += 1
self._ordered_next_sequence_by_channel[frame.channel_id] = next_sequence
if frame.sequence < next_sequence:
def _drain_ordered(self) -> list[CultNetRudpDeliveredFrame]:
"""Delivers, in sequence order, every held ordered frame whose first
sequence is at most one past the watermark. Ordered delivery has one
owner: the watermark. A frame this call does not deliver waits for the
sequence below it, and nothing else releases it."""
if self._received_through is None:
return []
if frame.sequence > next_sequence:
self._ordered_buffers.setdefault(frame.channel_id, {})[frame.sequence] = (frame, next_after_frame)
return []
self._ordered_next_sequence_by_channel[frame.channel_id] = next_after_frame
return [frame, *self._drain_ordered(frame.channel_id)]
def _drain_ordered(self, channel_id: str) -> list[CultNetRudpDeliveredFrame]:
delivered: list[CultNetRudpDeliveredFrame] = []
buffer = self._ordered_buffers.get(channel_id)
if buffer is None:
return delivered
while True:
next_sequence = self._ordered_next_sequence_by_channel[channel_id]
pending = buffer.pop(next_sequence, None)
if pending is None:
break
frame, next_after_frame = pending
delivered.append(frame)
self._ordered_next_sequence_by_channel[channel_id] = next_after_frame
self._skip_received_non_channel_sequences(channel_id)
return delivered
def _skip_received_non_channel_sequences(self, channel_id: str) -> None:
next_sequence = self._ordered_next_sequence_by_channel.get(channel_id)
while (
next_sequence is not None
and next_sequence in self._received_sequences
and next_sequence not in self._ordered_buffers.get(channel_id, {})
):
next_sequence += 1
self._ordered_next_sequence_by_channel[channel_id] = next_sequence
releasable = sorted(sequence for sequence in self._ordered_held if sequence <= self._received_through + 1)
return [self._ordered_held.pop(sequence) for sequence in releasable]
def _allocate_fragment_id(self) -> int:
fragment_id = self._next_fragment_id
@ -1033,6 +1183,10 @@ class CultNetRudpSocketTransportConnection:
if packet.connection_id != self.session.connection_id:
self._packets_dropped += 1
return None
# A Connect from another endpoint is a new client, whatever sequence it
# carries: the endpoint moves and the Connect starts a new generation.
# Only a Connect from the peer's own endpoint can be its repeat.
connect_from_new_endpoint = False
if self.remote_addr is None:
# Only a Connect claims the endpoint of a server-mode transport.
if self.mode == CultNetRudpSocketMode.SERVER and packet.packet_type != CultNetRudpPacketType.CONNECT:
@ -1040,18 +1194,21 @@ class CultNetRudpSocketTransportConnection:
return None
self.remote_addr = remote_addr
elif remote_addr != self.remote_addr:
self._packets_dropped += 1
return None
if self.mode == CultNetRudpSocketMode.SERVER and packet.packet_type == CultNetRudpPacketType.CONNECT:
self.remote_addr = remote_addr
connect_from_new_endpoint = True
else:
self._packets_dropped += 1
return None
if self.mode == CultNetRudpSocketMode.SERVER and packet.packet_type == CultNetRudpPacketType.CONNECT:
# A Connect from the peer this transport already accepted repeats:
# answer it with the Accept already owed and queue nothing.
# The session answers a repeat of the Connect it accepted with the
# Accept already owed, and anything else with a new generation.
try:
if self.session.connected:
reply = self.session.answer_repeated_connect(packet, _now_ms())
else:
reply = self.session.accept_connect(packet, _now_ms())
self.disconnect_reason = None
if connect_from_new_endpoint:
self.session.reset_peer_state()
reply = self.session.accept_connect(packet, _now_ms())
self.disconnect_reason = None
except ValueError:
self._packets_dropped += 1
return None
@ -1105,14 +1262,19 @@ class CultNetRudpSocketTransportConnection:
preserved = deque(self._delivered_frames)
self._delivered_frames.clear()
self.socket.settimeout(poll_timeout)
# A flush belongs to the generation it started in: whatever ends that
# generation, and whatever begins after it, the writes being waited on
# are gone.
generation = self.session.generation
try:
while True:
# An ended session forgot its unacknowledged writes; reporting
# them flushed would be a lie.
if self.disconnect_reason is not None:
if self.session.ended or self.session.generation != generation:
reason = self.disconnect_reason
raise ConnectionError(
"RUDP session ended before its reliable writes were acknowledged: "
+ self.disconnect_reason.decode("utf-8", errors="replace")
+ ("no reason given" if reason is None else reason.decode("utf-8", errors="replace"))
)
if self.session.outstanding_reliable_packet_count == 0:
return
@ -1137,7 +1299,10 @@ class CultNetRudpSocketTransportConnection:
self._send_packet(self.session.create_ping(payload))
def check_timeout(self, timeout_ms: int, now_ms: int | None = None) -> bool:
return self.session.check_timeout(_now_ms() if now_ms is None else now_ms, timeout_ms)
timed_out = self.session.check_timeout(_now_ms() if now_ms is None else now_ms, timeout_ms)
if timed_out:
self.disconnect_reason = b"session timed out"
return timed_out
def poll_resends(self) -> None:
for packet in self.session.due_resends(_now_ms()):

217
packages/cultnet-py/tests/test_cultnet.py
View file

@ -94,6 +94,17 @@ from cultnet_py import (
)
def acknowledges(packet, sequence: int) -> bool:
"""Whether the packet's ack field and mask cover the sequence."""
if packet.ack == sequence:
return True
return (
packet.ack > sequence
and packet.ack - sequence <= 32
and (packet.ack_mask & (1 << (packet.ack - sequence - 1))) != 0
)
def bind_udp_socket() -> socket.socket:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(("127.0.0.1", 0))
@ -616,12 +627,7 @@ class CultNetTests(unittest.TestCase):
receiver = CultNetRudpSession(
CultNetRudpSessionOptions(connection_id=7, initial_sequence=200, resend_delay_ms=100)
)
sender.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 7, 1, 0, 0, "control")
)
receiver.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 7, 2, 0, 0, "control")
)
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
first = sender.send(
"schema",
@ -638,27 +644,28 @@ class CultNetTests(unittest.TestCase):
b"third",
CultNetRudpSendOptions(reliable=True, ordered=True, now_ms=0),
)
self.assertEqual(sender.pending_reliable_sequences, (10, 11, 12))
self.assertEqual(sender.pending_reliable_sequences, (11, 12, 13))
receiver.receive(first)
receiver.receive(third)
ack_with_gap = receiver.create_ack()
self.assertEqual(ack_with_gap.ack, 12)
self.assertEqual(ack_with_gap.ack_mask, 0b10 | (1 << 9))
self.assertEqual(ack_with_gap.ack, 13)
self.assertEqual(ack_with_gap.ack_mask, 0b110)
sender.receive(ack_with_gap)
self.assertEqual(sender.pending_reliable_sequences, (11,))
self.assertEqual(sender.pending_reliable_sequences, (12,))
receiver.receive(second)
full_ack = receiver.create_ack()
self.assertEqual(full_ack.ack, 12)
self.assertEqual(full_ack.ack_mask, 0b11 | (1 << 9))
self.assertEqual(full_ack.ack, 13)
self.assertEqual(full_ack.ack_mask, 0b111)
sender.receive(full_ack)
self.assertEqual(sender.pending_reliable_sequences, ())
def test_cultnet_rudp_session_schedules_reliable_resends_until_acked(self) -> None:
session = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=99, initial_sequence=1, resend_delay_ms=100))
session.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 99, 50, 0, 0, "control")
CultNetRudpSession(CultNetRudpSessionOptions(connection_id=99, initial_sequence=900)).accept_connect(session.create_connect(0), 0),
0,
)
sent = session.send(
"schema",
@ -700,7 +707,7 @@ class CultNetTests(unittest.TestCase):
server = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=103, initial_sequence=100))
connect = client.create_connect(0, b"join")
accept = server.accept_connect(connect, 0)
reply = server.answer_repeated_connect(replace(connect, ack=accept.sequence), 1)
reply = server.accept_connect(replace(connect, ack=accept.sequence), 1)
self.assertEqual(reply.packet_type, CultNetRudpPacketType.ACK)
self.assertEqual(server.outstanding_reliable_packet_count, 0)
@ -708,17 +715,23 @@ class CultNetTests(unittest.TestCase):
session = CultNetRudpSession(
CultNetRudpSessionOptions(connection_id=102, initial_sequence=1, max_pending_reliable_packets=2)
)
session.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 102, 50, 0, 0, "control"))
session.receive(
CultNetRudpSession(CultNetRudpSessionOptions(connection_id=102, initial_sequence=900)).accept_connect(session.create_connect(0), 0),
0,
)
session.send("schema", b"first", CultNetRudpSendOptions(reliable=True, ordered=True))
session.send("schema", b"second", CultNetRudpSendOptions(reliable=True, ordered=True))
with self.assertRaisesRegex(ValueError, "reliable send queue is full"):
session.send("schema", b"third", CultNetRudpSendOptions(reliable=True, ordered=True))
self.assertEqual(session.pending_reliable_sequences, (1, 2))
self.assertEqual(session.pending_reliable_sequences, (2, 3))
fragmented = CultNetRudpSession(
CultNetRudpSessionOptions(connection_id=103, initial_sequence=1, max_pending_reliable_packets=3)
)
fragmented.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 103, 50, 0, 0, "control"))
fragmented.receive(
CultNetRudpSession(CultNetRudpSessionOptions(connection_id=103, initial_sequence=900)).accept_connect(fragmented.create_connect(0), 0),
0,
)
with self.assertRaisesRegex(ValueError, "reliable send queue is full"):
fragmented.send_many(
"schema",
@ -731,12 +744,7 @@ class CultNetTests(unittest.TestCase):
def test_cultnet_rudp_session_suppresses_duplicates_and_delivers_reliable_ordered_payloads(self) -> None:
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=123, initial_sequence=1))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=123, initial_sequence=100))
sender.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 123, 90, 0, 0, "control")
)
receiver.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 123, 91, 0, 0, "control")
)
sender.receive(receiver.accept_connect(sender.create_connect(0), 0))
first = sender.send("schema", b"first", CultNetRudpSendOptions(reliable=True, ordered=True))
second = sender.send("schema", b"second", CultNetRudpSendOptions(reliable=True, ordered=True))
@ -753,8 +761,7 @@ class CultNetTests(unittest.TestCase):
def test_cultnet_rudp_lossy_packets_cannot_create_reliable_ordered_gaps(self) -> None:
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=198, initial_sequence=1))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=198, initial_sequence=100))
sender.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 198, 90, 0, 0, "control"))
receiver.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 198, 91, 0, 0, "control"))
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
realtime = sender.send("realtime", b"discarded realtime", CultNetRudpSendOptions())
latest = sender.send("latest", b"discarded latest state", CultNetRudpSendOptions(sequenced=True))
@ -770,8 +777,7 @@ class CultNetTests(unittest.TestCase):
def test_cultnet_rudp_unreliable_sequenced_delivery_is_scoped_to_its_channel(self) -> None:
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=197, initial_sequence=50))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=197, initial_sequence=100))
sender.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 197, 90, 0, 0, "control"))
receiver.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 197, 91, 0, 0, "control"))
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
older = sender.send("latest", b"older", CultNetRudpSendOptions(sequenced=True))
newer = sender.send("latest", b"newer", CultNetRudpSendOptions(sequenced=True))
@ -786,7 +792,10 @@ class CultNetTests(unittest.TestCase):
def test_cultnet_rudp_rejects_unreliable_ordered_delivery(self) -> None:
session = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=196, initial_sequence=1))
session.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 196, 50, 0, 0, "control"))
session.receive(
CultNetRudpSession(CultNetRudpSessionOptions(connection_id=196, initial_sequence=900)).accept_connect(session.create_connect(0), 0),
0,
)
with self.assertRaises(ValueError) as raised:
session.send(
@ -799,12 +808,7 @@ class CultNetTests(unittest.TestCase):
def test_cultnet_rudp_session_skips_control_packets_while_ordering_schema_payloads(self) -> None:
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=124, initial_sequence=1))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=124, initial_sequence=100))
sender.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 124, 90, 0, 0, "control")
)
receiver.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 124, 91, 0, 0, "control")
)
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
first = sender.send("schema", b"first", CultNetRudpSendOptions(reliable=True, ordered=True))
control = sender.create_ack()
@ -817,12 +821,7 @@ class CultNetTests(unittest.TestCase):
def test_cultnet_rudp_session_fragments_and_reassembles_reliable_ordered_payloads(self) -> None:
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=456, initial_sequence=1))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=456, initial_sequence=100))
sender.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 456, 90, 0, 0, "control")
)
receiver.receive(
CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 456, 91, 0, 0, "control")
)
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
packets = sender.send_many(
"schema",
@ -847,8 +846,7 @@ class CultNetTests(unittest.TestCase):
connection_id = 457
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=1))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=100))
sender.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, connection_id, 0, 0, 0, "control"))
receiver.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, connection_id, 0, 0, 0, "control"))
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
fragment_count = CultNetRudpSession.RELIABLE_SEND_WINDOW_PACKETS + 17
payload = bytes(index % 251 for index in range(fragment_count * 8))
@ -1068,14 +1066,13 @@ class CultNetTests(unittest.TestCase):
# The peer is told, and the goodbye does not acknowledge the refused frame.
(goodbye,) = [p for p in drain_peer() if p.packet_type == CultNetRudpPacketType.DISCONNECT]
self.assertTrue(peer.receive(goodbye, 1).disconnected)
self.assertIn(poison.sequence, peer.pending_reliable_sequences)
self.assertFalse(acknowledges(goodbye, poison.sequence), "the goodbye acknowledged the refused frame")
# A retransmit is dropped, not acknowledged.
to_server(poison)
self.assertIsNone(server.receive_once())
self.assertEqual(server.stats.packets_dropped, 2)
self.assertEqual(drain_peer(), [])
self.assertIn(poison.sequence, peer.pending_reliable_sequences)
# A Connect claims the endpoint again.
again = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id))
@ -1145,6 +1142,135 @@ class CultNetTests(unittest.TestCase):
peer_socket.close()
server.close()
def test_cultnet_rudp_server_mode_admits_a_restarted_client_and_keeps_its_session_on_a_retransmit(self) -> None:
server, _, peer_socket, _, to_server, drain_peer = self._server_mode_pair(0x10203058)
connection_id = 0x10203058
reliable = CultNetRudpSendOptions(reliable=True, ordered=True)
try:
first = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=50_000))
connect = first.create_connect(0, b"same")
to_server(connect)
server.receive_once()
first.receive([p for p in drain_peer() if p.packet_type == CultNetRudpPacketType.ACCEPT][0], 0)
sent = [first.send("schema", f"c{index}".encode(), reliable) for index in range(3)]
delivered = []
for packet in sent:
to_server(packet)
delivered.append(server.receive_once().payload)
self.assertEqual(delivered, [b"c0", b"c1", b"c2"])
# A retransmitted Connect is a repeat: what was delivered stays delivered once.
to_server(connect)
server.receive_once()
to_server(sent[0])
self.assertIsNone(server.receive_once(), "the retransmitted Connect reset the session")
# The same client restarted on the same address, connection id and payload.
drain_peer()
second = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=7))
to_server(second.create_connect(0, b"same"))
server.receive_once()
second.receive([p for p in drain_peer() if p.packet_type == CultNetRudpPacketType.ACCEPT][0], 0)
to_server(second.send("schema", b"after restart", reliable))
self.assertEqual(server.receive_once().payload, b"after restart")
finally:
peer_socket.close()
server.close()
def test_cultnet_rudp_server_mode_admits_a_pinned_client_that_restarts_on_a_new_port(self) -> None:
server, server_socket, peer_socket, _, to_server, drain_peer = self._server_mode_pair(0x1020305B)
connection_id = 0x1020305B
reliable = CultNetRudpSendOptions(reliable=True, ordered=True)
other_socket = bind_udp_socket()
try:
first = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=1))
to_server(first.create_connect(0, b"join"))
server.receive_once()
first.receive([p for p in drain_peer() if p.packet_type == CultNetRudpPacketType.ACCEPT][0], 0)
to_server(first.send("schema", b"hello", reliable))
self.assertEqual(server.receive_once().payload, b"hello")
# The process restarted on a new port with the same pinned sequence.
second = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=1))
other_socket.sendto(encode_rudp_packet(second.create_connect(0, b"join")), server_socket.getsockname())
server.receive_once()
wire, _ = other_socket.recvfrom(65535)
second.receive(decode_rudp_packet(wire), 0)
self.assertTrue(second.connected, "the restarted client on a new port was not admitted")
other_socket.sendto(encode_rudp_packet(second.send("schema", b"after", reliable)), server_socket.getsockname())
self.assertEqual(server.receive_once().payload, b"after")
finally:
other_socket.close()
peer_socket.close()
server.close()
def test_cultnet_rudp_server_mode_admits_a_pinned_client_that_restarts_on_the_same_address(self) -> None:
server, _, peer_socket, _, to_server, drain_peer = self._server_mode_pair(0x1020305C)
connection_id = 0x1020305C
try:
first = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=1))
to_server(first.create_connect(0, b"join"))
server.receive_once()
accept = [p for p in drain_peer() if p.packet_type == CultNetRudpPacketType.ACCEPT][0]
first.receive(accept, 0)
to_server(first.create_ack_for_received(accept.sequence))
server.receive_once()
drain_peer()
# The process restarted on the same address with the same pinned sequence.
restarted = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=connection_id, initial_sequence=1))
to_server(restarted.create_connect(0, b"join"))
server.receive_once()
for packet in drain_peer():
restarted.receive(packet, 1)
self.assertFalse(restarted.connected)
to_server(restarted.due_resends(3_000)[0])
server.receive_once()
for packet in drain_peer():
restarted.receive(packet, 3_001)
self.assertTrue(restarted.connected, "the restarted pinned client was never admitted")
finally:
peer_socket.close()
server.close()
def test_cultnet_rudp_server_mode_admits_a_connect_from_a_new_endpoint(self) -> None:
server, server_socket, peer_socket, peer, to_server, drain_peer = self._server_mode_pair(0x10203059)
other_socket = bind_udp_socket()
try:
to_server(peer.create_connect(0, b"a"))
server.receive_once()
drain_peer()
other = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=0x10203059, initial_sequence=9))
other_socket.sendto(encode_rudp_packet(other.create_connect(0, b"b")), server_socket.getsockname())
server.receive_once()
wire, _ = other_socket.recvfrom(65535)
other.receive(decode_rudp_packet(wire), 0)
self.assertTrue(other.connected, "the new endpoint was not admitted")
other_socket.sendto(
encode_rudp_packet(other.send("schema", b"from b", CultNetRudpSendOptions(reliable=True, ordered=True))),
server_socket.getsockname(),
)
self.assertEqual(server.receive_once().payload, b"from b")
finally:
other_socket.close()
peer_socket.close()
server.close()
def test_cultnet_rudp_flush_after_a_timeout_names_the_timeout(self) -> None:
server, _, peer_socket, peer, to_server, drain_peer = self._server_mode_pair(0x1020305A)
try:
to_server(peer.create_connect(0, b"join"))
server.receive_once()
server.send("schema", b"unanswered")
self.assertTrue(server.check_timeout(0, now_ms=10**14))
with self.assertRaisesRegex(ConnectionError, "session timed out"):
server.flush_reliable(0.1)
finally:
peer_socket.close()
server.close()
def test_cultnet_rudp_session_rejects_limits_that_cannot_admit_a_connect(self) -> None:
with self.assertRaises(ValueError):
CultNetRudpSession(CultNetRudpSessionOptions(connection_id=1, initial_sequence=0xFFFFFFFF))
@ -2277,8 +2403,7 @@ class CultNetRudpFragmentBoundTests(unittest.TestCase):
def _pair(self) -> tuple[CultNetRudpSession, CultNetRudpSession]:
sender = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=311, initial_sequence=1))
receiver = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=311, initial_sequence=100))
sender.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 311, 90, 0, 0, "control"))
receiver.receive(CultNetRudpPacket(CultNetRudpPacketType.ACCEPT, 311, 91, 0, 0, "control"))
sender.receive(receiver.accept_connect(sender.create_connect(0), 0), 0)
return sender, receiver
def test_stranded_fragment_sets_are_bounded_and_evicted_oldest_first(self) -> None:

427
packages/cultnet-py/tests/test_rudp_connect_generation.py Normal file
View file

@ -0,0 +1,427 @@
"""A Connect's sequence says whether it repeats the one a session accepted, and
the handshake alone seeds the watermark that orders delivery."""
from __future__ import annotations
import unittest
from dataclasses import replace
from cultnet_py.transport import (
CultNetRudpPacketType,
CultNetRudpSendOptions,
CultNetRudpSession,
CultNetRudpSessionOptions,
CultNetRudpSocketMode,
CultNetRudpSocketTransportOptions,
)
CONNECTION_ID = 0x10203090
def session(initial_sequence: int) -> CultNetRudpSession:
return CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=initial_sequence))
def send(from_session: CultNetRudpSession, text: str, *, ordered: bool = True, channel: str = "schema"):
return from_session.send(channel, text.encode(), CultNetRudpSendOptions(reliable=True, ordered=ordered))
def names(result) -> list[str]:
return [frame.payload.decode() for frame in result.delivered]
def handshake(client: CultNetRudpSession, server: CultNetRudpSession):
connect = client.create_connect(0)
accept = server.accept_connect(connect, 0)
client.receive(accept, 0)
assert client.connected and server.connected
return connect, accept
class CultNetRudpConnectGenerationTests(unittest.TestCase):
def test_a_reconnect_on_the_same_session_is_not_stranded_by_a_write_the_ended_session_owed(self) -> None:
client, server = session(1), session(500)
handshake(client, server)
send(server, "lost")
server.create_disconnect(b"gone")
client.receive(server.accept_connect(client.create_connect(1), 1), 1)
self.assertEqual(names(client.receive(send(server, "after"), 2)), ["after"])
def test_a_server_that_accepts_a_new_client_forgets_the_old_clients_sequences(self) -> None:
old_client, server = session(50), session(500)
handshake(old_client, server)
for name in ("c1", "c2", "c3"):
self.assertEqual(names(server.receive(send(old_client, name), 1)), [name])
server.create_disconnect(b"restart")
new_client = session(2)
new_client.receive(server.accept_connect(new_client.create_connect(2), 2), 2)
self.assertEqual(names(server.receive(send(new_client, "fresh"), 3)), ["fresh"])
def test_a_retransmitted_connect_after_data_has_flowed_does_not_reset_the_session(self) -> None:
client, server = session(1), session(500)
connect, _ = handshake(client, server)
frame = send(client, "once")
self.assertEqual(names(server.receive(frame, 1)), ["once"])
self.assertTrue(server.connect_repeats(connect))
self.assertEqual(server.accept_connect(connect, 2).packet_type, CultNetRudpPacketType.ACK)
self.assertEqual(server.outstanding_reliable_packet_count, 0, "a repeat queued something")
self.assertEqual(
names(server.receive(frame, 3)), [], "a retransmitted Connect made the server forget what it had delivered"
)
def test_a_connect_with_another_sequence_is_not_a_repeat_whatever_it_carries(self) -> None:
server = session(500)
connect = session(1).create_connect(0, b"same payload")
server.accept_connect(connect, 0)
self.assertTrue(server.connect_repeats(connect))
self.assertFalse(server.connect_repeats(session(9).create_connect(0, b"same payload")))
self.assertFalse(server.connect_repeats(replace(connect, packet_type=CultNetRudpPacketType.PING)))
server.create_disconnect()
self.assertFalse(server.connect_repeats(connect), "an ended session repeats nothing")
def test_an_accept_that_does_not_name_the_pending_connect_is_ignored(self) -> None:
client, server = session(10), session(500)
connect = client.create_connect(0)
accept = server.accept_connect(connect, 0)
client.receive(replace(accept, ack=connect.sequence + 50, ack_mask=0), 1)
self.assertFalse(client.connected, "a stale Accept connected the session")
self.assertEqual(client.pending_reliable_sequences, (connect.sequence,))
client.receive(replace(accept, ack=connect.sequence + 3, ack_mask=1 << 2), 2)
self.assertTrue(client.connected, "the mask names the Connect")
def test_an_accept_after_a_local_disconnect_does_not_reconnect_the_session(self) -> None:
client, server = session(1), session(500)
_, accept = handshake(client, server)
client.create_disconnect(b"bye")
self.assertFalse(client.connected)
client.receive(accept, 5)
self.assertFalse(client.connected, "a late Accept revived an ended session")
def test_a_reconnect_forgets_what_the_old_server_sent(self) -> None:
client, old_server = session(1), session(500)
handshake(client, old_server)
for name in ("d1", "d2", "d3"):
self.assertEqual(names(client.receive(send(old_server, name), 1)), [name])
new_server = session(501)
client.receive(new_server.accept_connect(client.create_connect(2), 2), 2)
self.assertEqual(names(client.receive(send(new_server, "fresh"), 3)), ["fresh"])
def test_an_accept_for_an_abandoned_connect_does_not_connect_the_session(self) -> None:
client, server = session(1), session(500)
accept = server.accept_connect(client.create_connect(0), 0)
client.create_disconnect(b"never mind")
client.receive(accept, 1)
self.assertFalse(client.connected, "an Accept revived an abandoned Connect")
def test_a_full_queue_owed_to_a_vanished_peer_does_not_refuse_a_new_generation(self) -> None:
def bounded(initial_sequence: int) -> CultNetRudpSession:
return CultNetRudpSession(
CultNetRudpSessionOptions(
connection_id=CONNECTION_ID, initial_sequence=initial_sequence, max_pending_reliable_packets=1
)
)
server = bounded(500)
server.accept_connect(session(1).create_connect(0), 0)
self.assertEqual(server.outstanding_reliable_packet_count, 1, "the Accept is owed and never acknowledged")
following = session(9).create_connect(1)
server.accept_connect(following, 1)
self.assertTrue(server.connect_repeats(following))
client = bounded(1)
client.create_connect(0)
client.create_connect(1)
self.assertEqual(client.outstanding_reliable_packet_count, 1, "only the new Connect is owed")
def test_a_duplicate_accept_does_not_reseed_the_watermark(self) -> None:
client, server = session(1), session(500)
_, accept = handshake(client, server)
s1, s2 = send(server, "s1"), send(server, "s2")
client.receive(replace(accept, sequence=s2.sequence), 1)
self.assertEqual(names(client.receive(s2, 2)), [], "s1 is still missing")
self.assertEqual(names(client.receive(s1, 3)), ["s1", "s2"])
def test_reliable_data_before_the_accept_is_neither_delivered_nor_acknowledged(self) -> None:
client, server = session(1), session(500)
accept = server.accept_connect(client.create_connect(0), 0)
a, b = send(server, "A"), send(server, "B")
for early in (b, a):
self.assertEqual(names(client.receive(early, 1)), [])
ack = client.create_ack_for_received(early.sequence)
self.assertEqual((ack.ack, ack.ack_mask), (0, 0), "an unhandled packet was acknowledged")
client.receive(accept, 2)
self.assertTrue(client.connected)
self.assertEqual(server.outstanding_reliable_packet_count, 3, "nothing was acknowledged")
self.assertEqual(names(client.receive(a, 3)), ["A"])
self.assertEqual(names(client.receive(b, 3)), ["B"])
def test_ordered_frames_after_the_accept_are_delivered_in_order_whatever_arrives_first(self) -> None:
client, server = session(1), session(500)
handshake(client, server)
a, b = send(server, "A"), send(server, "B")
self.assertEqual(names(client.receive(b, 1)), [])
self.assertEqual(names(client.receive(a, 2)), ["A", "B"])
def test_an_unordered_reliable_frame_is_delivered_on_receipt_never_held_behind_a_gap(self) -> None:
client, server = session(1), session(500)
handshake(client, server)
send(server, "lost")
unordered = send(server, "now", ordered=False, channel="media")
ordered = send(server, "later")
self.assertEqual(names(client.receive(unordered, 1)), ["now"])
self.assertEqual(names(client.receive(ordered, 2)), [], "the ordered frame waits for the gap")
def test_a_peer_disconnect_does_not_forget_what_the_peer_sent(self) -> None:
client, server = session(1), session(500)
handshake(client, server)
s1, s2 = send(client, "s1"), send(client, "s2")
self.assertEqual(names(server.receive(s2, 1)), [], "s1 is missing, so s2 is held")
server.receive(client.create_disconnect(b"bye"), 2)
self.assertEqual(names(server.receive(s1, 3)), ["s1", "s2"], "the Disconnect forgot the held frame")
self.assertEqual(names(server.receive(s2, 4)), [], "the Disconnect forgot what was received")
def test_sessions_from_one_options_object_each_draw_their_own_initial_sequence(self) -> None:
self.assertIsNone(CultNetRudpSessionOptions(connection_id=1).initial_sequence)
self.assertIsNone(
CultNetRudpSocketTransportOptions(
runtime_id="r", socket=None, mode=CultNetRudpSocketMode.CLIENT, connection_id=1
).initial_sequence
)
options = CultNetRudpSessionOptions(connection_id=1)
firsts = {CultNetRudpSession(options).create_connect(0).sequence for _ in range(64)}
self.assertGreater(len(firsts), 32, "a default session does not draw its own sequence")
self.assertTrue(all(1 <= value < 2**31 for value in firsts), "a draw left [1, 2^31)")
# Ack Cut 1d: a frame from an earlier generation is a duplicate, a delayed
# Connect from an earlier attempt is stale, a Connect from a new endpoint is
# a new client.
@staticmethod
def names_sequence(ack, sequence: int) -> bool:
return ack.ack == sequence or any(
ack.ack_mask & (1 << bit) and ack.ack > bit and ack.ack - bit - 1 == sequence for bit in range(32)
)
def test_a_frame_from_the_clients_earlier_generation_is_acknowledged_not_delivered_again(self) -> None:
client, server = session(10), session(500)
handshake(client, server)
old = send(client, "old")
self.assertEqual(names(server.receive(old, 1)), ["old"])
connect = client.create_connect(2)
client.receive(server.accept_connect(connect, 2), 2)
self.assertLess(old.sequence, connect.sequence)
self.assertEqual(names(server.receive(old, 3)), [], "an earlier generation's frame was delivered again")
self.assertTrue(self.names_sequence(server.create_ack_for_received(old.sequence), old.sequence))
self.assertEqual(names(server.receive(send(client, "fresh"), 4)), ["fresh"])
def test_a_frame_from_the_servers_earlier_generation_is_acknowledged_not_delivered_again(self) -> None:
client, server = session(10), session(500)
handshake(client, server)
old = send(server, "old")
self.assertEqual(names(client.receive(old, 1)), ["old"])
client.receive(server.accept_connect(client.create_connect(2), 2), 2)
self.assertEqual(names(client.receive(old, 3)), [], "an earlier generation's frame was delivered again")
self.assertTrue(self.names_sequence(client.create_ack_for_received(old.sequence), old.sequence))
self.assertEqual(names(client.receive(send(server, "fresh"), 4)), ["fresh"])
def test_a_duplicate_below_the_receive_window_is_acknowledged_by_name(self) -> None:
client, server = session(10), session(500)
handshake(client, server)
first = send(client, "first")
self.assertEqual(names(server.receive(first, 1)), ["first"])
for _ in range(4_200):
server.receive(send(client, "x"), 1)
client.receive(server.create_ack(), 1)
self.assertEqual(names(server.receive(first, 2)), [])
self.assertEqual(
server.create_ack_for_received(first.sequence).ack,
first.sequence,
"a duplicate below the window was not acknowledged by name",
)
def test_a_delayed_connect_from_an_earlier_attempt_does_not_strand_the_client(self) -> None:
client, server = session(10), session(500)
earlier = client.create_connect(0)
retried = client.create_connect(300)
accept = server.accept_connect(retried, 301)
client.receive(accept, 302)
server.receive(client.create_ack_for_received(accept.sequence), 302)
self.assertTrue(client.connected)
a = send(client, "a")
self.assertEqual(names(server.receive(a, 303)), ["a"])
client.receive(server.create_ack_for_received(a.sequence), 303)
self.assertTrue(server.connect_repeats(earlier), "the delayed Connect starts nothing")
reply = server.accept_connect(earlier, 304)
self.assertEqual(reply.packet_type, CultNetRudpPacketType.ACK)
client.receive(reply, 305)
self.assertTrue(client.connected)
b = send(client, "b")
self.assertEqual(names(server.receive(b, 306)), ["b"], "the delayed Connect reset the server")
client.receive(server.create_ack_for_received(b.sequence), 307)
self.assertNotIn(b.sequence, client.pending_reliable_sequences)
def test_a_repeated_connect_refreshes_liveness_and_a_stale_one_does_not(self) -> None:
client, server = session(10), session(500)
earlier = client.create_connect(0)
current = client.create_connect(1)
server.accept_connect(current, 0)
server.accept_connect(current, 900)
self.assertFalse(server.check_timeout(1_000, 500), "a repeated Connect did not refresh liveness")
server.accept_connect(earlier, 1_400)
self.assertTrue(server.check_timeout(1_600, 500), "a stale Connect refreshed liveness")
def test_stale_connects_are_recognised_by_serial_arithmetic(self) -> None:
server = session(500)
server.accept_connect(session(3).create_connect(0), 0)
def connect(sequence: int):
return session(sequence).create_connect(0)
self.assertTrue(server.connect_repeats(connect(2**32 - 2)), "just before the wrap")
self.assertFalse(server.connect_repeats(connect(4)), "just after")
self.assertFalse(server.connect_repeats(connect(3 + 4_096)), "far after")
self.assertTrue(server.connect_repeats(connect(2**32 - 4_092)), "the window's edge")
self.assertFalse(server.connect_repeats(connect(2**32 - 4_093)), "past the window")
def test_a_restarted_client_whose_first_sequence_is_stale_connects_after_redrawing(self) -> None:
server, old = session(500), session(1_000)
old_accept = server.accept_connect(old.create_connect(0), 0)
old.receive(old_accept, 0)
restarted = session(900)
first = restarted.create_connect(0, b"join")
# The old client has not acknowledged its Accept yet, so that is the
# reply: it names the old Connect, not this one.
early = server.accept_connect(first, 1)
self.assertEqual(early.packet_type, CultNetRudpPacketType.ACCEPT)
restarted.receive(early, 1)
self.assertFalse(restarted.connected, "an Accept for another Connect connected the client")
server.receive(old.create_ack_for_received(old_accept.sequence), 1)
reply = server.accept_connect(first, 2)
self.assertEqual(reply.packet_type, CultNetRudpPacketType.ACK)
restarted.receive(reply, 2)
self.assertFalse(restarted.connected)
retransmitted = restarted.due_resends(1_000)
self.assertEqual(len(retransmitted), 1)
self.assertEqual(retransmitted[0].sequence, first.sequence, "the attempt is still young")
fresh = restarted.due_resends(3_500)
self.assertEqual(len(fresh), 1)
self.assertEqual(fresh[0].packet_type, CultNetRudpPacketType.CONNECT)
self.assertNotEqual(fresh[0].sequence, first.sequence, "the same Connect was retransmitted for ever")
self.assertEqual(fresh[0].payload, b"join", "the fresh attempt lost the Connect payload")
accept = server.accept_connect(fresh[0], 3_500)
self.assertEqual(accept.packet_type, CultNetRudpPacketType.ACCEPT)
restarted.receive(accept, 3_500)
self.assertTrue(restarted.connected)
self.assertEqual(names(server.receive(send(restarted, "hello"), 3_501)), ["hello"])
def test_an_answered_connect_is_never_replaced_by_the_attempt_timeout(self) -> None:
client, server = session(10), session(500)
connect, _ = handshake(client, server)
resent = client.due_resends(60_000)
self.assertTrue(
all(p.packet_type != CultNetRudpPacketType.CONNECT or p.sequence == connect.sequence for p in resent),
"a connected client started a new attempt",
)
self.assertTrue(client.connected)
def test_an_accepted_peer_that_goes_silent_times_out(self) -> None:
server = session(500)
server.accept_connect(session(10).create_connect(0), 0)
self.assertTrue(server.check_timeout(100_000, 1_000))
# Ack Cut 1d, batch 2: the connect-attempt timeout.
def test_an_ack_naming_the_pending_connect_does_not_retire_it(self) -> None:
server, old = session(500), session(1)
old_accept = server.accept_connect(old.create_connect(0), 0)
old.receive(old_accept, 0)
server.receive(old.create_ack_for_received(old_accept.sequence), 0)
restarted = session(1)
first = restarted.create_connect(0, b"join")
reply = server.accept_connect(first, 1)
self.assertEqual(reply.packet_type, CultNetRudpPacketType.ACK)
self.assertTrue(self.names_sequence(reply, first.sequence), "the Ack must name the pending Connect for this test to bite")
restarted.receive(reply, 1)
self.assertFalse(restarted.connected)
self.assertEqual(restarted.pending_reliable_sequences, (first.sequence,), "an Ack retired the Connect")
fresh = restarted.due_resends(3_000)
self.assertEqual(len(fresh), 1, "the client waits for ever with nothing to resend")
self.assertEqual(fresh[0].payload, b"join")
accept = server.accept_connect(fresh[0], 3_000)
self.assertEqual(accept.packet_type, CultNetRudpPacketType.ACCEPT)
restarted.receive(accept, 3_000)
self.assertTrue(restarted.connected)
def test_a_late_copy_of_an_abandoned_connect_is_stale_once_its_replacement_is_accepted(self) -> None:
server, client = session(500), session(10)
abandoned = client.create_connect(0)
fresh = client.due_resends(3_000)
self.assertEqual(len(fresh), 1)
self.assertEqual(fresh[0].sequence, abandoned.sequence + 4_095)
accept = server.accept_connect(fresh[0], 3_001)
client.receive(accept, 3_002)
server.receive(client.create_ack_for_received(accept.sequence), 3_002)
self.assertTrue(client.connected)
a = send(client, "a")
self.assertEqual(names(server.receive(a, 3_003)), ["a"])
client.receive(server.create_ack_for_received(a.sequence), 3_003)
self.assertTrue(server.connect_repeats(abandoned), "the late copy would restart the server")
reply = server.accept_connect(abandoned, 3_004)
self.assertEqual(reply.packet_type, CultNetRudpPacketType.ACK)
client.receive(reply, 3_005)
self.assertTrue(client.connected)
self.assertEqual(names(server.receive(send(client, "b"), 3_006)), ["b"])
def test_a_server_sitting_at_the_jump_is_left_by_the_next_attempt(self) -> None:
server, old = session(500), session(5_095)
old_accept = server.accept_connect(old.create_connect(0), 0)
old.receive(old_accept, 0)
server.receive(old.create_ack_for_received(old_accept.sequence), 0)
restarted = session(1_000)
first = restarted.create_connect(0, b"join")
restarted.receive(server.accept_connect(first, 1), 1)
self.assertFalse(restarted.connected)
second = restarted.due_resends(3_000)[0]
self.assertEqual(second.sequence, 5_095, "the jump lands on the server's generation")
reply = server.accept_connect(second, 3_001)
self.assertEqual(reply.packet_type, CultNetRudpPacketType.ACK)
restarted.receive(reply, 3_001)
self.assertFalse(restarted.connected)
third = restarted.due_resends(6_000)[0]
self.assertEqual(third.sequence, 5_095 + 4_095)
accept = server.accept_connect(third, 6_001)
self.assertEqual(accept.packet_type, CultNetRudpPacketType.ACCEPT)
restarted.receive(accept, 6_001)
self.assertTrue(restarted.connected)
if __name__ == "__main__":
unittest.main()

115
packages/cultnet-py/tests/test_rudp_ordered_delivery.py Normal file
View file

@ -0,0 +1,115 @@
"""Ordered delivery has one owner: the contiguous received-through watermark.
A frame is held for exactly one reason, a reliable sequence below it has not
arrived, and it is delivered in the call that fills the last such gap,
whichever channel the gap belonged to.
"""
from __future__ import annotations
import unittest
from cultnet_py.transport import (
CultNetRudpPacket,
CultNetRudpSendOptions,
CultNetRudpSession,
CultNetRudpSessionOptions,
)
def handshake() -> tuple[CultNetRudpSession, CultNetRudpSession]:
client = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=410, initial_sequence=1))
server = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=410, initial_sequence=500))
accept = server.accept_connect(client.create_connect(0), 0)
client.receive(accept, 0)
return client, server
def send(session: CultNetRudpSession, channel: str, payload: str, ordered: bool) -> CultNetRudpPacket:
return session.send(channel, payload.encode(), CultNetRudpSendOptions(reliable=True, ordered=ordered))
def names(result) -> list[str]:
return [frame.payload.decode() for frame in result.delivered]
class CultNetRudpOrderedDeliveryTests(unittest.TestCase):
def test_ordered_frame_waits_for_a_gap_filled_by_another_channel(self) -> None:
sender, receiver = handshake()
o1 = send(sender, "schema", "o1", True)
u = send(sender, "rel", "u", False)
o2 = send(sender, "schema", "o2", True)
o3 = send(sender, "schema", "o3", True)
o4 = send(sender, "schema", "o4", True)
self.assertEqual(names(receiver.receive(o1, 1)), ["o1"])
self.assertEqual(names(receiver.receive(o2, 2)), [], "u is missing")
self.assertEqual(names(receiver.receive(o3, 3)), [], "u is missing")
self.assertEqual(names(receiver.receive(o4, 4)), [], "u is missing")
self.assertEqual(names(receiver.receive(u, 5)), ["u", "o2", "o3", "o4"])
def test_two_ordered_channels_release_each_other_in_sequence_order(self) -> None:
sender, receiver = handshake()
a1 = send(sender, "schema", "A1", True)
b1 = send(sender, "other", "B1", True)
a2 = send(sender, "schema", "A2", True)
a3 = send(sender, "schema", "A3", True)
delivered = names(receiver.receive(a1, 1)) + names(receiver.receive(a2, 2)) + names(receiver.receive(a3, 3))
self.assertEqual(delivered, ["A1"], "B1 is missing, so A2 and A3 wait")
self.assertEqual(names(receiver.receive(b1, 4)), ["B1", "A2", "A3"])
def test_channel_first_used_after_other_traffic_loses_nothing(self) -> None:
sender, receiver = handshake()
c1 = send(sender, "late", "C1", True)
c2 = send(sender, "late", "C2", True)
x = send(sender, "schema", "X", True)
self.assertEqual(names(receiver.receive(x, 1)), [], "C1 is missing")
self.assertEqual(names(receiver.receive(c2, 2)), [], "C1 is missing")
self.assertEqual(names(receiver.receive(c1, 3)), ["C1", "C2", "X"])
def test_accept_seeds_the_watermark_of_the_connecting_side(self) -> None:
client, server = handshake()
s1 = send(server, "schema", "s1", True)
s2 = send(server, "schema", "s2", True)
self.assertEqual(names(client.receive(s2, 1)), [])
self.assertEqual(names(client.receive(s1, 2)), ["s1", "s2"])
def test_connect_seeds_the_watermark_of_the_accepting_side(self) -> None:
client, server = handshake()
first = send(client, "schema", "first", True)
second = send(client, "schema", "second", True)
self.assertEqual(names(server.receive(second, 1)), [])
self.assertEqual(names(server.receive(first, 2)), ["first", "second"])
def test_duplicate_of_a_held_frame_is_not_delivered_twice(self) -> None:
sender, receiver = handshake()
s1 = send(sender, "schema", "s1", True)
s2 = send(sender, "schema", "s2", True)
self.assertEqual(names(receiver.receive(s2, 1)), [])
self.assertEqual(names(receiver.receive(s2, 2)), [])
self.assertEqual(names(receiver.receive(s1, 3)), ["s1", "s2"])
self.assertEqual(names(receiver.receive(s2, 4)), [])
def test_reset_peer_state_forgets_held_frames_and_reseeds_the_watermark_from_the_next_handshake(self) -> None:
sender, receiver = handshake()
send(sender, "schema", "s1", True)
s2 = send(sender, "schema", "s2", True)
self.assertEqual(names(receiver.receive(s2, 1)), [])
receiver.reset_peer_state()
following = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=410, initial_sequence=100))
following.receive(receiver.accept_connect(following.create_connect(0), 2), 2)
d1 = send(following, "schema", "d1", True)
d2 = send(following, "schema", "d2", True)
# The held s2 died with the reset, and the watermark follows the new peer.
self.assertEqual(names(receiver.receive(d2, 3)), [])
self.assertEqual(names(receiver.receive(d1, 4)), ["d1", "d2"])
if __name__ == "__main__":
unittest.main()

223
packages/cultnet-py/tests/test_rudp_session_ending.py Normal file
View file

@ -0,0 +1,223 @@
"""A write belongs to the session generation it was issued in.
Every way a generation ends drops the writes it still owed, and a flush
belongs to the generation it started in.
"""
from __future__ import annotations
import socket
import threading
import time
import unittest
from cultnet_py.transport import (
CultNetRudpPacketType,
CultNetRudpSendOptions,
CultNetRudpSession,
CultNetRudpSessionOptions,
CultNetRudpSocketMode,
CultNetRudpSocketTransportConnection,
CultNetRudpSocketTransportOptions,
decode_rudp_packet,
encode_rudp_packet,
)
CONNECTION_ID = 0x10203070
def bind_udp_socket() -> socket.socket:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(("127.0.0.1", 0))
sock.settimeout(0.02)
return sock
def client_with_a_lost_write() -> tuple[CultNetRudpSession, CultNetRudpSession]:
client = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=1))
server = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=500))
client.receive(server.accept_connect(client.create_connect(0), 0), 0)
client.send("schema", b"owed to the old session", CultNetRudpSendOptions(reliable=True, ordered=True))
# A fragmented write larger than the send window leaves part of it queued.
client.send_many(
"schema", bytes(40 * 8), CultNetRudpSendOptions(reliable=True, ordered=True), max_fragment_bytes=8
)
assert client.outstanding_reliable_packet_count == 41
assert client.queued_reliable_packet_count > 0
return client, server
def drain(sock: socket.socket) -> list:
packets = []
while True:
try:
wire, _ = sock.recvfrom(65535)
except TimeoutError:
return packets
packets.append(decode_rudp_packet(wire))
def connected_client() -> tuple[CultNetRudpSocketTransportConnection, socket.socket, tuple]:
peer_socket = bind_udp_socket()
client = CultNetRudpSocketTransportConnection(
CultNetRudpSocketTransportOptions(
runtime_id="python-rudp-client",
socket=bind_udp_socket(),
mode=CultNetRudpSocketMode.CLIENT,
remote_addr=peer_socket.getsockname(),
connection_id=CONNECTION_ID,
)
)
client.connect(b"hello")
wire, client_addr = peer_socket.recvfrom(65535)
peer = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=500))
accept = peer.accept_connect(decode_rudp_packet(wire), 0)
peer_socket.sendto(encode_rudp_packet(accept), client_addr)
client.receive_once()
assert client.connected
return client, peer_socket, client_addr
class CultNetRudpSessionEndingTests(unittest.TestCase):
def test_every_way_a_session_ends_drops_the_writes_it_owed(self) -> None:
endings = {
"peer Disconnect": lambda client, server: client.receive(server.create_disconnect(b"bye"), 1),
"local disconnect": lambda client, server: client.create_disconnect(b"bye"),
"timeout": lambda client, server: self.assertTrue(client.check_timeout(1_000, 10)),
"reset": lambda client, server: client.reset_peer_state(),
}
for name, end in endings.items():
with self.subTest(ending=name):
client, server = client_with_a_lost_write()
end(client, server)
self.assertEqual(client.outstanding_reliable_packet_count, 0, "the write survived the end")
next_server = CultNetRudpSession(
CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=900)
)
next_server.accept_connect(client.create_connect(2_000), 2_000)
for resend in client.due_resends(60_000):
self.assertEqual(
next_server.receive(resend, 60_000).delivered,
(),
"the old session's write was delivered in the next session",
)
def test_a_flush_fails_after_a_local_disconnect_or_a_timeout(self) -> None:
client, peer_socket, _ = connected_client()
try:
client.send("schema", b"owed")
client.disconnect(b"bye")
with self.assertRaises(ConnectionError):
client.flush_reliable(0.2)
finally:
client.close()
peer_socket.close()
client, peer_socket, _ = connected_client()
try:
client.send("schema", b"owed")
time.sleep(0.005)
self.assertTrue(client.check_timeout(1))
with self.assertRaises(ConnectionError):
client.flush_reliable(0.2)
finally:
client.close()
peer_socket.close()
def test_a_flush_that_started_before_the_end_fails_even_after_a_later_connect(self) -> None:
client, peer_socket, _ = connected_client()
outcome: list[BaseException | None] = []
def flush() -> None:
try:
client.flush_reliable(5.0)
outcome.append(None)
except BaseException as error: # noqa: BLE001 - the outcome is the assertion
outcome.append(error)
try:
client.send("schema", b"owed")
worker = threading.Thread(target=flush)
worker.start()
time.sleep(0.05)
client.disconnect(b"bye")
client.connect(b"hello again")
worker.join(10)
self.assertEqual(len(outcome), 1)
self.assertIsInstance(outcome[0], ConnectionError)
finally:
client.close()
peer_socket.close()
def test_a_flush_started_after_a_reconnect_waits_on_the_new_session(self) -> None:
client, peer_socket, client_addr = connected_client()
try:
client.send("schema", b"owed to the old session")
client.disconnect(b"bye")
client.connect(b"hello again")
connect = [p for p in drain(peer_socket) if p.packet_type == CultNetRudpPacketType.CONNECT][-1]
accept = CultNetRudpSession(
CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=700)
).accept_connect(connect, 0)
peer_socket.sendto(encode_rudp_packet(accept), client_addr)
client.receive_once()
client.flush_reliable(0.5)
finally:
client.close()
peer_socket.close()
def test_a_flush_started_after_the_server_accepts_a_new_connect_waits_on_the_new_session(self) -> None:
server_socket = bind_udp_socket()
peer_socket = bind_udp_socket()
server = CultNetRudpSocketTransportConnection(
CultNetRudpSocketTransportOptions(
runtime_id="python-rudp-server",
socket=server_socket,
mode=CultNetRudpSocketMode.SERVER,
connection_id=CONNECTION_ID,
)
)
to_server = lambda packet: peer_socket.sendto(encode_rudp_packet(packet), server_socket.getsockname())
try:
first = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID))
to_server(first.create_connect(0))
server.receive_once()
to_server(first.create_disconnect())
server.receive_once()
following = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID))
to_server(following.create_connect(0))
server.receive_once()
accept = [p for p in drain(peer_socket) if p.packet_type == CultNetRudpPacketType.ACCEPT][-1]
following.receive(accept, 0)
to_server(following.create_ack())
server.receive_once()
server.flush_reliable(0.5)
finally:
server.close()
peer_socket.close()
def test_an_ending_does_not_forget_what_was_received_from_the_peer(self) -> None:
client = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=1))
server = CultNetRudpSession(CultNetRudpSessionOptions(connection_id=CONNECTION_ID, initial_sequence=500))
client.receive(server.accept_connect(client.create_connect(0), 0), 0)
ordered = CultNetRudpSendOptions(reliable=True, ordered=True)
s1 = client.send("schema", b"s1", ordered)
s2 = client.send("schema", b"s2", ordered)
self.assertEqual(server.receive(s2, 1).delivered, (), "s1 is missing, so s2 is held")
self.assertTrue(server.check_timeout(1_000, 10))
self.assertEqual(
[frame.payload for frame in server.receive(s1, 1_001).delivered],
[b"s1", b"s2"],
"the ending forgot the held frame",
)
self.assertEqual(server.receive(s2, 1_002).delivered, (), "the ending forgot what was received")
if __name__ == "__main__":
unittest.main()

6
packages/cultnet-rs/Cargo.toml
View file

@ -19,6 +19,7 @@ cultcache-rs = { path = "../cultcache-rs" }
ed25519-dalek = "2.2"
hmac = "0.12"
rand = "0.9"
reed-solomon-erasure = "6"
rmp-serde = "1.3"
rmpv = { version = "1.3", features = ["with-serde"] }
serde = { version = "1", features = ["derive"] }
@ -35,3 +36,8 @@ windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Securit
[dev-dependencies]
pretty_assertions = "1.4"
tempfile = "3"
[[example]]
name = "cultnet_impair"
path = "examples/cultnet_impair/main.rs"
test = true

8
packages/cultnet-rs/contracts/cultnet/cultnet.error.schema.json
View file

@ -26,9 +26,9 @@
}
},
"code": {
"description": "Machine-readable refusal code (R-N). Additive: a v0 peer that does not know this field ignores it.",
"description": "Machine-readable refusal code (R-N). Additive: a v0 peer that does not know this field ignores it. The set is a closed enum: Rust and TypeScript refuse a message whose code is not listed, so adding a code is one coordinated change across every runtime (this schema, the C# factory, the Rust CultNetErrorCode, and a captured-bytes case for each).",
"type": ["string", "null"],
"enum": [null, "selection_invalid", "cursor_stale", "cursor_invalid", "reference_outside_target", "variant_unsupported"]
"enum": [null, "selection_invalid", "cursor_stale", "cursor_invalid", "reference_outside_target", "variant_unsupported", "unowned_schema"]
},
"details": {
"description": "Code-specific structured detail (R-N). Every field is populated only for the code that defines it; the others are null, not omitted.",
@ -36,8 +36,8 @@
"additionalProperties": false,
"required": ["field", "value", "asOf", "current"],
"properties": {
"field": { "description": "selection_invalid: the selection field that failed. variant_unsupported: \"subscriptionId\", when a subscription was ended.", "type": ["string", "null"] },
"value": { "description": "selection_invalid: the offending value, when there is one string worth naming. variant_unsupported: the id of the subscription that ended.", "type": ["string", "null"] },
"field": { "description": "selection_invalid: the selection field that failed. variant_unsupported: \"subscriptionId\", when a subscription was ended. unowned_schema: \"schemaId\".", "type": ["string", "null"] },
"value": { "description": "selection_invalid: the offending value, when there is one string worth naming. variant_unsupported: the id of the subscription that ended. unowned_schema: the schema id no shard owns.", "type": ["string", "null"] },
"asOf": { "description": "cursor_stale: the cursor's own minted asOf.", "type": ["integer", "null"], "minimum": 0 },
"current": { "description": "cursor_stale: the answering server's current asOf.", "type": ["integer", "null"], "minimum": 0 }
}

336
packages/cultnet-rs/examples/cultnet_impair/main.rs Normal file
View file

@ -0,0 +1,336 @@
//! `cultnet-impair`: a deterministic bidirectional UDP proxy for pressure-testing
//! real CultNet RUDP sessions. It owns transport impairment only; what a stream
//! does with the loss is the stream's business.
//!
//! The proxy listens at the endpoint a client dials and forwards to `--upstream`
//! through one upstream socket per client, so the reverse ACK and feedback
//! traffic crosses the same schedule. The schedule is `policy.rs`.
//!
//! ```text
//! cargo run --release --example cultnet_impair -- \
//! --listen 0.0.0.0:17890 --upstream 127.0.0.1:17990 \
//! --profile tests/impairment/loss-1pct.toml --seed 424242 --metrics loss-1pct.csv
//! ```
//!
//! Profiles are a small TOML subset of unsigned integers; see
//! `tests/impairment/`. The receiver binds the upstream endpoint and the client
//! dials the proxy's; never one port for both. For a two-host test, run the proxy
//! on the receiver host so reverse traffic is impaired too.
mod policy;
use std::collections::HashMap;
use std::env;
use std::fs::{self, File};
use std::io::{self, Write};
use std::net::{SocketAddr, UdpSocket};
use std::path::{Path, PathBuf};
use std::thread;
use std::time::{Duration, Instant};
use policy::{Direction, Profile, Scheduler, Stats};
const MAX_DATAGRAM: usize = 65_535;
const MAX_CLIENT_FLOWS: usize = 64;
const CLIENT_FLOW_IDLE_TIMEOUT_MS: u64 = 30_000;
const MAX_DATAGRAMS_PER_FLOW_PER_TURN: usize = 16;
#[derive(Debug)]
struct UpstreamFlow {
socket: UdpSocket,
last_activity_ms: u64,
}
fn open_upstream_flow(upstream: SocketAddr, now_ms: u64) -> io::Result<UpstreamFlow> {
let socket = UdpSocket::bind("0.0.0.0:0")?;
socket.connect(upstream)?;
socket.set_nonblocking(true)?;
Ok(UpstreamFlow {
socket,
last_activity_ms: now_ms,
})
}
fn ensure_upstream_flow(
flows: &mut HashMap<SocketAddr, UpstreamFlow>,
client: SocketAddr,
upstream: SocketAddr,
now_ms: u64,
stats: &mut Stats,
) -> io::Result<()> {
if flows.contains_key(&client) {
return Ok(());
}
if flows.len() >= MAX_CLIENT_FLOWS
&& let Some(oldest) = flows
.iter()
.min_by_key(|(_, flow)| flow.last_activity_ms)
.map(|(address, _)| *address)
{
flows.remove(&oldest);
stats.flow_evicted += 1;
}
flows.insert(client, open_upstream_flow(upstream, now_ms)?);
stats.max_flows = stats.max_flows.max(flows.len());
Ok(())
}
fn reap_idle_flows(
flows: &mut HashMap<SocketAddr, UpstreamFlow>,
now_ms: u64,
stats: &mut Stats,
) {
let before = flows.len();
flows.retain(|_, flow| {
now_ms.saturating_sub(flow.last_activity_ms) < CLIENT_FLOW_IDLE_TIMEOUT_MS
});
stats.flow_evicted += (before - flows.len()) as u64;
}
#[derive(Debug)]
struct Options {
listen: SocketAddr,
upstream: SocketAddr,
profile: PathBuf,
seed: u64,
metrics: Option<PathBuf>,
}
fn parse_options() -> Result<Options, String> {
let mut args = env::args().skip(1);
let mut listen = None;
let mut upstream = None;
let mut profile = None;
let mut seed = 1;
let mut metrics = None;
while let Some(arg) = args.next() {
let mut value = || args.next().ok_or_else(|| format!("{arg} requires a value"));
match arg.as_str() {
"--listen" => {
listen = Some(
value()?
.parse()
.map_err(|_| "--listen must be a socket address")?,
)
}
"--upstream" => {
upstream = Some(
value()?
.parse()
.map_err(|_| "--upstream must be a socket address")?,
)
}
"--profile" => profile = Some(PathBuf::from(value()?)),
"--seed" => {
seed = value()?
.parse()
.map_err(|_| "--seed must be an unsigned integer")?
}
"--metrics" => metrics = Some(PathBuf::from(value()?)),
"--help" | "-h" => return Err(usage().into()),
_ => return Err(format!("unknown argument {arg}\n{}", usage())),
}
}
Ok(Options {
listen: listen.ok_or("--listen is required")?,
upstream: upstream.ok_or("--upstream is required")?,
profile: profile.ok_or("--profile is required")?,
seed,
metrics,
})
}
fn usage() -> &'static str {
"usage: cultnet-impair --listen HOST:PORT --upstream HOST:PORT --profile PATH [--seed N] [--metrics PATH]"
}
fn elapsed_ms(start: Instant) -> u64 {
start.elapsed().as_millis().try_into().unwrap_or(u64::MAX)
}
fn write_metrics(path: &Path, stats: &Stats, active_flows: usize) -> io::Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let mut file = File::create(path)?;
writeln!(
file,
"received,forwarded,dropped,duplicated,reordered,stalled,queue_overflow,max_queue,flow_evicted,max_flows,active_flows"
)?;
writeln!(
file,
"{},{},{},{},{},{},{},{},{},{},{}",
stats.received,
stats.forwarded,
stats.dropped,
stats.duplicated,
stats.reordered,
stats.stalled,
stats.queue_overflow,
stats.max_queue,
stats.flow_evicted,
stats.max_flows,
active_flows
)
}
fn run(options: Options) -> Result<(), String> {
let profile = Profile::load(&options.profile)?;
let client_socket = UdpSocket::bind(options.listen)
.map_err(|error| format!("binding {}: {error}", options.listen))?;
client_socket
.set_nonblocking(true)
.map_err(|error| error.to_string())?;
println!(
"cultnet-impair listen={} upstream={} seed={} profile={}",
options.listen,
options.upstream,
options.seed,
options.profile.display()
);
let start = Instant::now();
let mut scheduler = Scheduler::new(profile, options.seed);
let mut flows = HashMap::<SocketAddr, UpstreamFlow>::new();
let mut last_flow_reap_at = 0_u64;
let mut buffer = vec![0; MAX_DATAGRAM];
loop {
let now = elapsed_ms(start);
let mut did_work = false;
for _ in 0..MAX_DATAGRAMS_PER_FLOW_PER_TURN {
match client_socket.recv_from(&mut buffer) {
Ok((size, address)) => {
did_work = true;
ensure_upstream_flow(
&mut flows,
address,
options.upstream,
now,
&mut scheduler.stats,
)
.map_err(|error| format!("opening upstream flow for {address}: {error}"))?;
if let Some(flow) = flows.get_mut(&address) {
flow.last_activity_ms = now;
}
scheduler.admit(
now,
Direction::ClientToUpstream,
&buffer[..size],
address,
);
}
Err(error) if error.kind() == io::ErrorKind::WouldBlock => break,
Err(error) => return Err(error.to_string()),
}
}
for (address, flow) in &mut flows {
for _ in 0..MAX_DATAGRAMS_PER_FLOW_PER_TURN {
match flow.socket.recv(&mut buffer) {
Ok(size) => {
did_work = true;
flow.last_activity_ms = now;
scheduler.admit(
now,
Direction::UpstreamToClient,
&buffer[..size],
*address,
);
}
Err(error) if error.kind() == io::ErrorKind::WouldBlock => break,
Err(error) => return Err(error.to_string()),
}
}
}
while let Some(item) = scheduler.pop_due(now) {
did_work = true;
let result = match item.direction {
Direction::ClientToUpstream => match flows.get_mut(&item.client) {
Some(flow) => {
flow.last_activity_ms = now;
flow.socket.send(&item.bytes)
}
None => {
scheduler.stats.dropped += 1;
continue;
}
},
Direction::UpstreamToClient => client_socket.send_to(&item.bytes, item.client),
};
if result.is_ok() {
scheduler.stats.forwarded += 1;
}
}
if let Some(path) = options.metrics.as_deref() {
if now % 1000 == 0 {
let _ = write_metrics(path, &scheduler.stats, flows.len());
}
}
if now.saturating_sub(last_flow_reap_at) >= 1_000 {
reap_idle_flows(&mut flows, now, &mut scheduler.stats);
last_flow_reap_at = now;
}
if did_work {
thread::yield_now();
} else {
thread::sleep(Duration::from_millis(1));
}
}
}
fn main() {
match parse_options().and_then(run) {
Ok(()) => {}
Err(error) => {
eprintln!("{error}");
std::process::exit(2);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn independent_clients_keep_independent_upstream_udp_identities() {
let mut flows = HashMap::new();
let mut stats = Stats::default();
let upstream = "127.0.0.1:54321".parse().unwrap();
let first = "127.0.0.1:12001".parse().unwrap();
let second = "127.0.0.1:12002".parse().unwrap();
ensure_upstream_flow(&mut flows, first, upstream, 1, &mut stats).unwrap();
ensure_upstream_flow(&mut flows, second, upstream, 2, &mut stats).unwrap();
assert_eq!(flows.len(), 2);
assert_ne!(
flows[&first].socket.local_addr().unwrap(),
flows[&second].socket.local_addr().unwrap()
);
assert_eq!(stats.max_flows, 2);
assert_eq!(stats.flow_evicted, 0);
}
#[test]
fn idle_client_flows_are_reaped_without_touching_active_flows() {
let mut flows = HashMap::new();
let mut stats = Stats::default();
let upstream = "127.0.0.1:54321".parse().unwrap();
let idle = "127.0.0.1:12001".parse().unwrap();
let active = "127.0.0.1:12002".parse().unwrap();
ensure_upstream_flow(&mut flows, idle, upstream, 0, &mut stats).unwrap();
ensure_upstream_flow(
&mut flows,
active,
upstream,
CLIENT_FLOW_IDLE_TIMEOUT_MS - 1,
&mut stats,
)
.unwrap();
reap_idle_flows(&mut flows, CLIENT_FLOW_IDLE_TIMEOUT_MS, &mut stats);
assert!(!flows.contains_key(&idle));
assert!(flows.contains_key(&active));
assert_eq!(stats.flow_evicted, 1);
}
}

298
packages/cultnet-rs/examples/cultnet_impair/policy.rs Normal file
View file

@ -0,0 +1,298 @@
//! The impairment policy: what happens to each datagram, and nothing about
//! sockets. `Profile` says what to do, `Scheduler` decides per datagram and
//! holds the delayed ones, and every decision is a pure function of the
//! profile, the seed and the datagram's position, so a schedule replays
//! exactly. The proxy in `main.rs` and the loss-matrix test in
//! `tests/media_fec_loss.rs` both take their schedule from here.
use std::collections::VecDeque;
use std::fs;
use std::net::SocketAddr;
use std::path::Path;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Profile {
pub loss_basis_points: u32,
pub burst_every: u64,
pub burst_length: u64,
pub duplicate_every: u64,
pub reorder_every: u64,
pub reorder_delay_ms: u64,
pub delay_ms: u64,
pub jitter_ms: u64,
pub stall_at_ms: u64,
pub stall_for_ms: u64,
pub max_scheduled: usize,
}
impl Default for Profile {
fn default() -> Self {
Self {
loss_basis_points: 0,
burst_every: 0,
burst_length: 0,
duplicate_every: 0,
reorder_every: 0,
reorder_delay_ms: 0,
delay_ms: 0,
jitter_ms: 0,
stall_at_ms: 0,
stall_for_ms: 0,
max_scheduled: 4096,
}
}
}
impl Profile {
pub fn load(path: &Path) -> Result<Self, String> {
let text = fs::read_to_string(path)
.map_err(|error| format!("reading {}: {error}", path.display()))?;
let mut profile = Self::default();
for (index, raw) in text.lines().enumerate() {
let line = raw.split('#').next().unwrap_or("").trim();
if line.is_empty() || line.starts_with('[') {
continue;
}
let (key, value) = line
.split_once('=')
.ok_or_else(|| format!("{}:{}: expected key = value", path.display(), index + 1))?;
let key = key.trim();
let value = value.trim().trim_matches('"');
let number = value.parse::<u64>().map_err(|_| {
format!(
"{}:{}: {key} must be an unsigned integer",
path.display(),
index + 1
)
})?;
match key {
"loss_basis_points" => {
profile.loss_basis_points = number
.try_into()
.map_err(|_| "loss_basis_points is too large")?
}
"burst_every" => profile.burst_every = number,
"burst_length" => profile.burst_length = number,
"duplicate_every" => profile.duplicate_every = number,
"reorder_every" => profile.reorder_every = number,
"reorder_delay_ms" => profile.reorder_delay_ms = number,
"delay_ms" => profile.delay_ms = number,
"jitter_ms" => profile.jitter_ms = number,
"stall_at_ms" => profile.stall_at_ms = number,
"stall_for_ms" => profile.stall_for_ms = number,
"max_scheduled" => {
profile.max_scheduled = number
.try_into()
.map_err(|_| "max_scheduled is too large")?
}
_ => {
return Err(format!(
"{}:{}: unknown profile key {key}",
path.display(),
index + 1
));
}
}
}
if profile.loss_basis_points > 10_000 {
return Err("loss_basis_points must be <= 10000".into());
}
if profile.max_scheduled == 0 {
return Err("max_scheduled must be greater than zero".into());
}
Ok(profile)
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Direction {
ClientToUpstream,
UpstreamToClient,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Decision {
pub drop: bool,
pub copies: u8,
pub delay_ms: u64,
pub reordered: bool,
pub stalled: bool,
}
#[derive(Clone, Debug)]
pub struct DeterministicRng(u64);
impl DeterministicRng {
pub fn new(seed: u64) -> Self {
Self(if seed == 0 { 0x9e3779b97f4a7c15 } else { seed })
}
pub fn next(&mut self) -> u64 {
let mut x = self.0;
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
self.0 = x;
x
}
}
#[derive(Debug)]
pub struct Scheduler {
pub profile: Profile,
pub rng: DeterministicRng,
pub seen: u64,
pub burst_remaining: u64,
pub queue: VecDeque<Scheduled>,
pub stats: Stats,
}
#[derive(Debug)]
pub struct Scheduled {
pub due_ms: u64,
pub direction: Direction,
pub bytes: Vec<u8>,
pub client: SocketAddr,
}
#[derive(Debug, Default)]
pub struct Stats {
pub received: u64,
pub forwarded: u64,
pub dropped: u64,
pub duplicated: u64,
pub reordered: u64,
pub stalled: u64,
pub queue_overflow: u64,
pub max_queue: usize,
pub flow_evicted: u64,
pub max_flows: usize,
}
impl Scheduler {
pub fn new(profile: Profile, seed: u64) -> Self {
Self {
profile,
rng: DeterministicRng::new(seed),
seen: 0,
burst_remaining: 0,
queue: VecDeque::new(),
stats: Stats::default(),
}
}
pub fn decide(&mut self, now_ms: u64) -> Decision {
self.seen += 1;
let stalled = self.profile.stall_for_ms > 0
&& now_ms >= self.profile.stall_at_ms
&& now_ms
< self
.profile
.stall_at_ms
.saturating_add(self.profile.stall_for_ms);
if stalled {
return Decision {
drop: true,
copies: 0,
delay_ms: 0,
reordered: false,
stalled: true,
};
}
if self.burst_remaining == 0
&& self.profile.burst_every > 0
&& self.seen % self.profile.burst_every == 0
{
self.burst_remaining = self.profile.burst_length;
}
if self.burst_remaining > 0 {
self.burst_remaining -= 1;
return Decision {
drop: true,
copies: 0,
delay_ms: 0,
reordered: false,
stalled: false,
};
}
if self.profile.loss_basis_points > 0
&& self.rng.next() % 10_000 < u64::from(self.profile.loss_basis_points)
{
return Decision {
drop: true,
copies: 0,
delay_ms: 0,
reordered: false,
stalled: false,
};
}
let duplicate =
self.profile.duplicate_every > 0 && self.seen % self.profile.duplicate_every == 0;
let reordered =
self.profile.reorder_every > 0 && self.seen % self.profile.reorder_every == 0;
let jitter = if self.profile.jitter_ms == 0 {
0
} else {
self.rng.next() % (self.profile.jitter_ms + 1)
};
Decision {
drop: false,
copies: if duplicate { 2 } else { 1 },
delay_ms: self.profile.delay_ms
+ jitter
+ if reordered {
self.profile.reorder_delay_ms
} else {
0
},
reordered,
stalled: false,
}
}
pub fn admit(&mut self, now_ms: u64, direction: Direction, bytes: &[u8], client: SocketAddr) {
self.stats.received += 1;
let decision = self.decide(now_ms);
if decision.drop {
self.stats.dropped += 1;
if decision.stalled {
self.stats.stalled += 1;
}
return;
}
if decision.copies == 2 {
self.stats.duplicated += 1;
}
if decision.reordered {
self.stats.reordered += 1;
}
for copy in 0..decision.copies {
if self.queue.len() >= self.profile.max_scheduled {
self.stats.queue_overflow += 1;
self.stats.dropped += 1;
continue;
}
let delay = decision.delay_ms + u64::from(copy);
let item = Scheduled {
due_ms: now_ms.saturating_add(delay),
direction,
bytes: bytes.to_vec(),
client,
};
let position = self
.queue
.iter()
.position(|queued| queued.due_ms > item.due_ms)
.unwrap_or(self.queue.len());
self.queue.insert(position, item);
self.stats.max_queue = self.stats.max_queue.max(self.queue.len());
}
}
pub fn pop_due(&mut self, now_ms: u64) -> Option<Scheduled> {
if self.queue.front().is_some_and(|item| item.due_ms <= now_ms) {
self.queue.pop_front()
} else {
None
}
}
}

6
packages/cultnet-rs/examples/cultnet_interop_peer.rs
View file

@ -507,7 +507,7 @@ fn rudp_serve_once(options: &BTreeMap<String, String>) -> Result<()> {
let local_port = socket.local_addr()?.port();
let mut transport_options =
CultNetRudpSocketTransportOptions::server("rust-rudp-interop", socket, 0x22446688);
transport_options.initial_sequence = 100;
transport_options.initial_sequence = Some(100);
transport_options.resend_delay_ms = 25;
transport_options.max_fragment_bytes = max_fragment_bytes;
let mut transport = CultNetRudpSocketTransportConnection::new(transport_options)?;
@ -608,7 +608,7 @@ fn rudp_serve_message_once(options: &BTreeMap<String, String>) -> Result<()> {
let local_port = socket.local_addr()?.port();
let mut transport_options =
CultNetRudpSocketTransportOptions::server("rust-rudp-message-interop", socket, 0x22446689);
transport_options.initial_sequence = 100;
transport_options.initial_sequence = Some(100);
transport_options.resend_delay_ms = 25;
let mut transport = CultNetRudpSocketTransportConnection::new(transport_options)?;
@ -967,7 +967,7 @@ fn run_rudp_server(
let peer = peers.entry(remote_addr).or_insert_with(|| RudpPeerSession {
session: CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: RUDP_INTEROP_CONNECTION_ID,
initial_sequence: 100,
initial_sequence: Some(100),
resend_delay_ms: RUDP_INTEROP_RESEND_DELAY_MS,
max_pending_reliable_packets: None,
}),

7
packages/cultnet-rs/src/contracts.rs
View file

@ -250,8 +250,8 @@ impl CultNetShardDescriptor {
}
}
/// Machine-readable `cultnet.error.v0` refusal code (R-N). Mirrors the five strings
/// `contracts/cultnet/cultnet.error.schema.json` enumerates for `code`; a peer that receives a fifth
/// Machine-readable `cultnet.error.v0` refusal code (R-N). Mirrors the strings
/// `contracts/cultnet/cultnet.error.schema.json` enumerates for `code`; a peer that receives a
/// string it does not know refuses to decode rather than guess a meaning for it.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
@ -261,6 +261,7 @@ pub enum CultNetErrorCode {
CursorInvalid,
ReferenceOutsideTarget,
VariantUnsupported,
UnownedSchema,
}
impl CultNetErrorCode {
@ -271,6 +272,7 @@ impl CultNetErrorCode {
Self::CursorInvalid => "cursor_invalid",
Self::ReferenceOutsideTarget => "reference_outside_target",
Self::VariantUnsupported => "variant_unsupported",
Self::UnownedSchema => "unowned_schema",
}
}
@ -281,6 +283,7 @@ impl CultNetErrorCode {
"cursor_invalid" => Ok(Self::CursorInvalid),
"reference_outside_target" => Ok(Self::ReferenceOutsideTarget),
"variant_unsupported" => Ok(Self::VariantUnsupported),
"unowned_schema" => Ok(Self::UnownedSchema),
other => Err(anyhow!("ErrorMessage.Code {other:?} is not a recognized cultnet.error.v0 code")),
}
}

4
packages/cultnet-rs/src/cultmesh.rs
View file

@ -243,7 +243,7 @@ pub struct CultMeshRudpSocketOptions {
pub bind_host: String,
pub bind_port: u16,
pub read_timeout: Option<Duration>,
pub initial_sequence: u32,
pub initial_sequence: Option<u32>,
pub resend_delay_ms: u64,
pub max_payload_bytes: Option<u32>,
pub max_fragment_bytes: Option<u32>,
@ -256,7 +256,7 @@ impl Default for CultMeshRudpSocketOptions {
bind_host: "127.0.0.1".to_string(),
bind_port: 0,
read_timeout: Some(Duration::from_millis(20)),
initial_sequence: 1,
initial_sequence: None,
resend_delay_ms: 250,
max_payload_bytes: None,
max_fragment_bytes: None,

2
packages/cultnet-rs/src/lib.rs
View file

@ -2,6 +2,7 @@ mod content;
mod contracts;
mod idunn_deployment_brake;
mod idunn_lifecycle_brake;
mod media_fec;
mod media_stream_contracts;
mod media_stream_wire;
mod provider_health_contracts;
@ -24,6 +25,7 @@ pub use content::*;
pub use contracts::*;
pub use idunn_deployment_brake::*;
pub use idunn_lifecycle_brake::*;
pub use media_fec::*;
pub use media_stream_contracts::*;
pub use media_stream_wire::*;
pub use provider_health_contracts::*;

749
packages/cultnet-rs/src/media_fec.rs Normal file
View file

@ -0,0 +1,749 @@
//! Forward error correction for CultMesh Media Stream records.
//!
//! Pure functions over the records in [`crate::media_stream_contracts`]: no
//! sockets, no clock, no randomness. A producer calls [`protect_video_frame`]
//! and [`protect_audio_block`]; a consumer calls [`recover_video_block`] and
//! [`recover_audio_block`]. Both ends link this one implementation, so a
//! producer and a consumer cannot disagree about the math without one of them
//! failing to build against the same version of this crate.
//!
//! # The code
//!
//! Every block is a systematic Reed-Solomon code over GF(2^8): `k` data shards
//! are sent unchanged and `m` parity shards are computed from them. Any `k` of
//! the `k + m` shards recover the block (the code is MDS), so any `m` erasures
//! are repaired and any more are reported as [`MediaFecError::BeyondRepair`],
//! never as wrong bytes. The construction is named on every parity record by
//! [`MEDIA_FEC_SCHEME_RS_GF256_V1`] and backed by the `reed-solomon-erasure`
//! crate. The library is not the contract: the construction below is, and
//! `tests/media_fec.rs` pins the parity bytes of known inputs, so a library that
//! changes the matrix fails there instead of on the wire, and a decoder in
//! another runtime has vectors to meet.
//!
//! # The `rs-gf256-v1` construction
//!
//! Enough to implement it in another runtime without this crate.
//!
//! * The field is GF(2^8) over the polynomial `x^8 + x^4 + x^3 + x^2 + 1`
//! (`0x11D`). Addition is XOR; multiplication is carry-less multiplication
//! reduced by `0x11D`.
//! * For a block of `k` data shards and `m` parity shards, let `n = k + m`
//! (`n <= 256`) and build the `n x k` Vandermonde matrix `V` whose entry in
//! row `r`, column `c` is `r^c`, where the row number `r` (0 through
//! `n - 1`) is itself read as a field element and `0^0 = 1`. The evaluation
//! points are the integers 0 through `n - 1`, not powers of a generator.
//! * The encoding matrix is `E = V * inverse(top)`, where `top` is the first
//! `k` rows of `V`. Its first `k` rows are the identity, so data shards are
//! sent unchanged; its last `m` rows generate parity.
//! * Every shard of a block has the same length, `shard_payload_bytes`. A
//! shorter final video chunk is zero-padded to that length before encoding.
//! Byte `j` of parity shard `p` is the XOR over data shards `i` of
//! `E[k + p][i] * data[i][j]`.
//! * Any `k` surviving shards recover the block: invert the `k x k` submatrix
//! of `E` for the surviving rows and multiply.
//!
//! The known-answer vectors in `tests/media_fec.rs` were checked against an
//! independent implementation of exactly this text.
//!
//! # What a block is
//!
//! * Video: a frame's chunks split into `ceil(n / 16)` contiguous, near-equal
//! blocks, each protected by `max(2, ceil(k / 4))` parity shards. A block
//! never spans two frames, so one frame's deadline governs its recovery.
//! [`protect_video_frame`] returns the frame's records in send order: blocks
//! are interleaved, so a burst of consecutive losses lands as one loss per
//! block rather than many in one.
//! * Audio: `4` consecutive packets of one payload length, protected by `2`
//! parity shards.
//!
//! # One datagram
//!
//! Every record the protect functions return, data or parity, fits one
//! datagram once wrapped for the wire, so parity never reintroduces the
//! fragment amplification it exists to remove. The protect functions enforce
//! that against [`MediaFecPolicy::max_wire_bytes`]: they encode each record
//! with the caller's provenance and refuse the block if any is larger.
//! The default budget is [`MEDIA_FEC_MAX_WIRE_BYTES`], an IPv4 path with a
//! 1500-byte MTU. Pass a smaller budget when the path carries less: IPv6 over
//! 1500 leaves a UDP payload of 1452 bytes, and a WireGuard or Tailscale
//! tunnel about 1252, so subtract the 41 bytes of RUDP header and channel name
//! from the path's UDP payload limit.
//!
//! The budget is measured with the provenance given, so a producer whose
//! `stored_at` or identifiers grow later must pass the widest it will use.
//!
//! # Audio playout
//!
//! A recovered audio packet has the block's latest `deadline_ticks`, so a
//! packet recovered early in the block reports a deadline later than its own.
//! Audio consumers must gate playout on `pts_ticks`, which recovery derives
//! exactly, and use `deadline_ticks` only to give up on the block as a whole.
//!
//! # What this module does not own
//!
//! When to give up on a block is the receiver's deadline, and the `(k, m)`
//! choice is the producer's policy ([`MediaFecPolicy`]); this module only makes
//! and consumes shards.
use std::collections::BTreeMap;
use std::fmt;
use reed_solomon_erasure::galois_8::ReedSolomon;
use crate::media_stream_contracts::{
GameCultMediaAudioPacketRecord, GameCultMediaAudioParityShardRecord,
GameCultMediaVideoAccessUnitRecord, GameCultMediaVideoParityShardRecord,
};
use crate::media_stream_wire::{
GAMECULT_MEDIA_CHANNEL, GAMECULT_MEDIA_MAX_WIRE_BYTES, GameCultMediaWireRecord, MediaWireProvenance,
encode_media_wire_record, validate_audio_parity_record, validate_audio_record,
validate_video_parity_record, validate_video_record,
};
use crate::rudp::RUDP_FIXED_HEADER_BYTES;
/// Names the construction on every parity record: systematic Reed-Solomon over
/// GF(2^8), as built by `reed-solomon-erasure`'s `galois_8` field. A different
/// matrix is a different scheme id.
pub const MEDIA_FEC_SCHEME_RS_GF256_V1: &str = "rs-gf256-v1";
/// The largest UDP payload an IPv4 datagram carries on a 1500-byte MTU.
pub const MEDIA_IPV4_UDP_PAYLOAD_BYTES: usize = 1_472;
/// The default datagram budget: the most a wrapped media record may occupy so
/// that the RUDP header and the channel name still leave it in one IPv4
/// datagram on a 1500-byte MTU (1,431 bytes).
pub const MEDIA_FEC_MAX_WIRE_BYTES: usize =
MEDIA_IPV4_UDP_PAYLOAD_BYTES - RUDP_FIXED_HEADER_BYTES - GAMECULT_MEDIA_CHANNEL.len();
/// A producer's choice of `(k, m)` and datagram budget.
/// [`MediaFecPolicy::STANDARD`] is the ruled default; each record carries the
/// `(k, m)` it was made with, so a decoder needs no policy.
///
/// Build one by overriding fields of the default, `MediaFecPolicy {
/// max_wire_bytes: 1_211, ..MediaFecPolicy::STANDARD }`, not by naming every
/// field: a field added later then cannot break, or silently mis-set, the
/// caller. `max_wire_bytes` may not exceed
/// [`GAMECULT_MEDIA_MAX_WIRE_BYTES`], the ceiling a decoder holds every record
/// to.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct MediaFecPolicy {
/// The most data chunks in one video block.
pub video_max_block_data_shards: u16,
/// The fewest parity shards a video block carries.
pub video_min_parity_shards: u16,
/// A video block of `k` data shards carries at least `ceil(k / this)` parity
/// shards.
pub video_parity_divisor: u16,
/// Packets per audio block.
pub audio_data_shards: u16,
/// Parity shards per audio block.
pub audio_parity_shards: u16,
/// The most a wrapped record may occupy. The protect functions refuse a
/// block with any record over it. See the module docs for smaller paths.
pub max_wire_bytes: usize,
}
impl MediaFecPolicy {
/// Video `k <= 16` with `m = max(2, ceil(k / 4))`; audio 4+2; the IPv4
/// datagram budget.
pub const STANDARD: Self = Self {
video_max_block_data_shards: 16,
video_min_parity_shards: 2,
video_parity_divisor: 4,
audio_data_shards: 4,
audio_parity_shards: 2,
max_wire_bytes: MEDIA_FEC_MAX_WIRE_BYTES,
};
/// `m` for a video block of `data_shards` chunks.
pub fn video_parity_shards(&self, data_shards: u16) -> u16 {
data_shards
.div_ceil(self.video_parity_divisor.max(1))
.max(self.video_min_parity_shards)
}
/// How a frame of `chunk_count` chunks splits: `ceil(n / max)` blocks whose
/// sizes differ by at most one, larger blocks first.
pub fn video_block_sizes(&self, chunk_count: u16) -> Vec<u16> {
let max = self.video_max_block_data_shards.max(1);
let blocks = chunk_count.div_ceil(max);
if blocks == 0 {
return Vec::new();
}
let base = chunk_count / blocks;
let extra = chunk_count % blocks;
(0..blocks).map(|index| base + u16::from(index < extra)).collect()
}
fn validate(&self) -> Result<(), MediaFecError> {
let widest = self.video_max_block_data_shards;
if widest == 0
|| self.video_min_parity_shards == 0
|| self.video_parity_divisor == 0
|| self.audio_data_shards == 0
|| self.audio_parity_shards == 0
{
return Err(MediaFecError::invalid("media FEC policy fields must be non-zero"));
}
if usize::from(widest) + usize::from(self.video_parity_shards(widest)) > 256
|| usize::from(self.audio_data_shards) + usize::from(self.audio_parity_shards) > 256
{
return Err(MediaFecError::invalid("media FEC policy exceeds 256 shards per block"));
}
if self.max_wire_bytes > GAMECULT_MEDIA_MAX_WIRE_BYTES {
return Err(MediaFecError::invalid(format!(
"media FEC policy max_wire_bytes {} exceeds the {GAMECULT_MEDIA_MAX_WIRE_BYTES}-byte media record ceiling",
self.max_wire_bytes
)));
}
Ok(())
}
}
impl Default for MediaFecPolicy {
fn default() -> Self {
Self::STANDARD
}
}
/// Why a block could not be protected or recovered.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum MediaFecError {
/// Fewer than `k` of the block's shards are present. Nothing is returned:
/// a block that cannot be repaired is reported, never guessed at.
BeyondRepair { shards_present: usize, shards_needed: usize },
/// The inputs are not one well-formed block.
Invalid(String),
}
impl MediaFecError {
fn invalid(message: impl Into<String>) -> Self {
Self::Invalid(message.into())
}
}
impl fmt::Display for MediaFecError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::BeyondRepair { shards_present, shards_needed } => write!(
f,
"media FEC block is beyond repair: {shards_present} shards present, {shards_needed} needed"
),
Self::Invalid(message) => write!(f, "invalid media FEC block: {message}"),
}
}
}
impl std::error::Error for MediaFecError {}
// ---------------------------------------------------------------------------
// The code: equal-length shards in, equal-length shards out.
// ---------------------------------------------------------------------------
fn parity_shards(data: &[Vec<u8>], parity_count: usize) -> Result<Vec<Vec<u8>>, MediaFecError> {
let code = ReedSolomon::new(data.len(), parity_count)
.map_err(|error| MediaFecError::invalid(format!("{error:?}")))?;
let mut parity = vec![vec![0_u8; data[0].len()]; parity_count];
code.encode_sep(data, &mut parity)
.map_err(|error| MediaFecError::invalid(format!("{error:?}")))?;
Ok(parity)
}
/// `shards` holds the block's `k` data slots then `m` parity slots, `None` for
/// each missing. Returns the `k` data shards.
fn recover_data_shards(
data_shards: usize,
mut shards: Vec<Option<Vec<u8>>>,
) -> Result<Vec<Vec<u8>>, MediaFecError> {
let parity_shards = shards.len() - data_shards;
let present = shards.iter().flatten().count();
if present < data_shards {
return Err(MediaFecError::BeyondRepair {
shards_present: present,
shards_needed: data_shards,
});
}
let code = ReedSolomon::new(data_shards, parity_shards)
.map_err(|error| MediaFecError::invalid(format!("{error:?}")))?;
code.reconstruct_data(&mut shards)
.map_err(|error| MediaFecError::invalid(format!("{error:?}")))?;
Ok(shards
.into_iter()
.take(data_shards)
.map(|shard| shard.expect("reconstruct_data fills every data shard"))
.collect())
}
fn padded(payload: &[u8], shard_bytes: usize) -> Vec<u8> {
let mut shard = vec![0_u8; shard_bytes];
shard[..payload.len()].copy_from_slice(payload);
shard
}
fn from_anyhow(error: anyhow::Error) -> MediaFecError {
MediaFecError::Invalid(error.to_string())
}
/// Refuses a record that would not fit one datagram once wrapped.
fn fits_datagram(
record: &GameCultMediaWireRecord,
provenance: MediaWireProvenance<'_>,
policy: &MediaFecPolicy,
) -> Result<(), MediaFecError> {
let wire = encode_media_wire_record(record, provenance).map_err(from_anyhow)?;
if wire.len() > policy.max_wire_bytes {
return Err(MediaFecError::invalid(format!(
"a {} record wraps to {} bytes, over the {}-byte datagram budget: cut chunks smaller or raise max_wire_bytes",
record.schema_id(),
wire.len(),
policy.max_wire_bytes
)));
}
Ok(())
}
/// Where slot `slot` of an audio block lands: its packet id and presentation
/// time, or `None` if either overflows. The one place block arithmetic lives;
/// the protect path, the wire validator and recovery all ask it.
pub(crate) fn audio_slot(
base_packet_id: u64,
base_pts_ticks: i64,
packet_duration_ticks: u32,
slot: u64,
) -> Option<(u64, i64)> {
let packet_id = base_packet_id.checked_add(slot)?;
let pts_ticks = i64::from(packet_duration_ticks)
.checked_mul(i64::try_from(slot).ok()?)?
.checked_add(base_pts_ticks)?;
Some((packet_id, pts_ticks))
}
// ---------------------------------------------------------------------------
// Video
// ---------------------------------------------------------------------------
/// Protects one video frame. `chunks` are all of the frame's access-unit
/// records, in any order; the result is every data and parity record in send
/// order.
///
/// Within a block, data and parity alternate; across blocks, the sequence
/// rotates one record from each block at a time. Where every block is the same
/// size, no two consecutive `block_count` records share a block. Blocks that
/// differ in size run out at the end, and the rotation then cycles over the
/// blocks remaining.
///
/// Errors if `chunks` are not exactly one whole frame (mixed frames, a missing
/// or repeated chunk index, disagreeing framing), if a chunk is not shard
/// shaped (every chunk of a block but its last must be the same length, the
/// last no longer), or if any resulting record, wrapped with `provenance`,
/// exceeds `policy.max_wire_bytes`.
pub fn protect_video_frame(
chunks: &[GameCultMediaVideoAccessUnitRecord],
policy: &MediaFecPolicy,
provenance: MediaWireProvenance<'_>,
) -> Result<Vec<GameCultMediaWireRecord>, MediaFecError> {
policy.validate()?;
let first = chunks
.first()
.ok_or_else(|| MediaFecError::invalid("a video frame needs at least one chunk"))?;
let chunk_count = usize::from(first.chunk_count);
if chunks.len() != chunk_count {
return Err(MediaFecError::invalid(format!(
"frame {} declares {} chunks but {} were given",
first.frame_id,
chunk_count,
chunks.len()
)));
}
let mut by_index: BTreeMap<u16, &GameCultMediaVideoAccessUnitRecord> = BTreeMap::new();
for chunk in chunks {
validate_video_record(chunk).map_err(from_anyhow)?;
if !same_frame(first, chunk) {
return Err(MediaFecError::invalid(
"a block never spans frames: chunks disagree on stream, session, frame or framing",
));
}
if by_index.insert(chunk.chunk_index, chunk).is_some() {
return Err(MediaFecError::invalid(format!(
"chunk {} appears twice",
chunk.chunk_index
)));
}
}
let block_sizes = policy.video_block_sizes(first.chunk_count);
let block_count = block_sizes.len() as u16;
let mut lanes: Vec<Vec<GameCultMediaWireRecord>> = Vec::with_capacity(block_sizes.len());
let mut start = 0_u16;
for (block_index, &size) in block_sizes.iter().enumerate() {
let members: Vec<&GameCultMediaVideoAccessUnitRecord> = (start..start + size)
.map(|index| by_index[&index])
.collect();
let shard_bytes = members[0].payload.len();
let (last, body) = members.split_last().expect("block has a member");
if body.iter().any(|chunk| chunk.payload.len() != shard_bytes)
|| last.payload.len() > shard_bytes
{
return Err(MediaFecError::invalid(
"every chunk of a block but its last must share one length, and the last no longer",
));
}
let data: Vec<Vec<u8>> = members
.iter()
.map(|chunk| padded(&chunk.payload, shard_bytes))
.collect();
let parity_count = policy.video_parity_shards(size);
let parity = parity_shards(&data, usize::from(parity_count))?;
let parity_records: Vec<GameCultMediaWireRecord> = parity
.into_iter()
.enumerate()
.map(|(parity_index, payload)| {
GameCultMediaWireRecord::VideoParity(GameCultMediaVideoParityShardRecord {
stream_id: first.stream_id.clone(),
session_id: first.session_id.clone(),
frame_id: first.frame_id,
codec: first.codec.clone(),
pts_ticks: first.pts_ticks,
duration_ticks: first.duration_ticks,
timebase_num: first.timebase_num,
timebase_den: first.timebase_den,
keyframe: first.keyframe,
dependency_frame_id: first.dependency_frame_id,
deadline_ticks: first.deadline_ticks,
chunk_count: first.chunk_count,
fec_scheme: MEDIA_FEC_SCHEME_RS_GF256_V1.to_string(),
block_index: block_index as u16,
block_count,
block_data_start: start,
block_data_count: size,
parity_index: parity_index as u16,
parity_count,
shard_payload_bytes: shard_bytes as u32,
last_chunk_payload_bytes: last.payload.len() as u32,
payload,
})
})
.collect();
let data_records = members
.iter()
.map(|chunk| GameCultMediaWireRecord::Video((*chunk).clone()));
lanes.push(interleave(data_records.collect(), parity_records));
start += size;
}
let records = rotate(lanes);
for record in &records {
fits_datagram(record, provenance, policy)?;
}
Ok(records)
}
fn same_frame(a: &GameCultMediaVideoAccessUnitRecord, b: &GameCultMediaVideoAccessUnitRecord) -> bool {
a.stream_id == b.stream_id
&& a.session_id == b.session_id
&& a.frame_id == b.frame_id
&& a.codec == b.codec
&& a.pts_ticks == b.pts_ticks
&& a.duration_ticks == b.duration_ticks
&& a.timebase_num == b.timebase_num
&& a.timebase_den == b.timebase_den
&& a.keyframe == b.keyframe
&& a.dependency_frame_id == b.dependency_frame_id
&& a.deadline_ticks == b.deadline_ticks
&& a.chunk_count == b.chunk_count
}
/// Data and parity alternate: `D0 P0 D1 P1 ...`, the longer list finishing the
/// lane.
fn interleave<T>(data: Vec<T>, parity: Vec<T>) -> Vec<T> {
let mut data = data.into_iter();
let mut parity = parity.into_iter();
let mut lane = Vec::new();
loop {
let next_data = data.next();
let next_parity = parity.next();
if next_data.is_none() && next_parity.is_none() {
return lane;
}
lane.extend(next_data);
lane.extend(next_parity);
}
}
/// One record from each lane in turn, until every lane is empty.
fn rotate<T>(lanes: Vec<Vec<T>>) -> Vec<T> {
let mut lanes: Vec<_> = lanes.into_iter().map(Vec::into_iter).collect();
let mut out = Vec::new();
loop {
let before = out.len();
for lane in &mut lanes {
out.extend(lane.next());
}
if out.len() == before {
return out;
}
}
}
/// Rebuilds the data chunks a block lost.
///
/// `parity` are the block's parity shards that arrived (at least one, for its
/// geometry) and `data` its data chunks that arrived; each may be in any order
/// and may repeat. Returns the missing chunks, ordered by `chunk_index`: empty
/// if none were missing. Fails with [`MediaFecError::BeyondRepair`] if fewer
/// than `block_data_count` shards of the block are present, and with
/// [`MediaFecError::Invalid`] if the records are not from one block.
pub fn recover_video_block(
parity: &[GameCultMediaVideoParityShardRecord],
data: &[GameCultMediaVideoAccessUnitRecord],
) -> Result<Vec<GameCultMediaVideoAccessUnitRecord>, MediaFecError> {
let geometry = parity
.first()
.ok_or_else(|| MediaFecError::invalid("recovery needs at least one parity shard"))?;
let k = usize::from(geometry.block_data_count);
let m = usize::from(geometry.parity_count);
let start = usize::from(geometry.block_data_start);
let shard_bytes = geometry.shard_payload_bytes as usize;
let mut shards: Vec<Option<Vec<u8>>> = vec![None; k + m];
for shard in parity {
validate_video_parity_record(shard).map_err(from_anyhow)?;
if !same_block(geometry, shard) {
return Err(MediaFecError::invalid("parity shards are not from one block"));
}
shards[k + usize::from(shard.parity_index)].get_or_insert_with(|| shard.payload.clone());
}
for chunk in data {
validate_video_record(chunk).map_err(from_anyhow)?;
let index = usize::from(chunk.chunk_index);
let framing_agrees = chunk.stream_id == geometry.stream_id
&& chunk.session_id == geometry.session_id
&& chunk.frame_id == geometry.frame_id
&& chunk.chunk_count == geometry.chunk_count;
if !framing_agrees || index < start || index >= start + k {
return Err(MediaFecError::invalid("data chunk does not belong to the parity's block"));
}
let expected = if index == start + k - 1 {
geometry.last_chunk_payload_bytes as usize
} else {
shard_bytes
};
if chunk.payload.len() != expected {
return Err(MediaFecError::invalid(format!(
"chunk {index} is {} bytes but the block declares {expected}",
chunk.payload.len()
)));
}
shards[index - start].get_or_insert_with(|| padded(&chunk.payload, shard_bytes));
}
let missing: Vec<usize> = (0..k).filter(|&slot| shards[slot].is_none()).collect();
let recovered = recover_data_shards(k, shards)?;
Ok(missing
.into_iter()
.map(|slot| {
let mut payload = recovered[slot].clone();
if slot == k - 1 {
payload.truncate(geometry.last_chunk_payload_bytes as usize);
}
GameCultMediaVideoAccessUnitRecord {
stream_id: geometry.stream_id.clone(),
session_id: geometry.session_id.clone(),
frame_id: geometry.frame_id,
codec: geometry.codec.clone(),
pts_ticks: geometry.pts_ticks,
duration_ticks: geometry.duration_ticks,
timebase_num: geometry.timebase_num,
timebase_den: geometry.timebase_den,
keyframe: geometry.keyframe,
dependency_frame_id: geometry.dependency_frame_id,
deadline_ticks: geometry.deadline_ticks,
chunk_index: (start + slot) as u16,
chunk_count: geometry.chunk_count,
payload,
}
})
.collect())
}
fn same_block(a: &GameCultMediaVideoParityShardRecord, b: &GameCultMediaVideoParityShardRecord) -> bool {
a.stream_id == b.stream_id
&& a.session_id == b.session_id
&& a.frame_id == b.frame_id
&& a.codec == b.codec
&& a.pts_ticks == b.pts_ticks
&& a.duration_ticks == b.duration_ticks
&& a.timebase_num == b.timebase_num
&& a.timebase_den == b.timebase_den
&& a.keyframe == b.keyframe
&& a.dependency_frame_id == b.dependency_frame_id
&& a.deadline_ticks == b.deadline_ticks
&& a.chunk_count == b.chunk_count
&& a.fec_scheme == b.fec_scheme
&& a.block_index == b.block_index
&& a.block_count == b.block_count
&& a.block_data_start == b.block_data_start
&& a.block_data_count == b.block_data_count
&& a.parity_count == b.parity_count
&& a.shard_payload_bytes == b.shard_payload_bytes
&& a.last_chunk_payload_bytes == b.last_chunk_payload_bytes
}
// ---------------------------------------------------------------------------
// Audio
// ---------------------------------------------------------------------------
/// Protects one block of audio packets: exactly `policy.audio_data_shards`
/// packets with contiguous ids and presentation times, one payload length, and
/// one codec and timebase. Returns the parity records in `parity_index` order.
/// Errors if a parity record, wrapped with `provenance`, exceeds
/// `policy.max_wire_bytes`; the packets are smaller than their parity.
pub fn protect_audio_block(
packets: &[GameCultMediaAudioPacketRecord],
policy: &MediaFecPolicy,
provenance: MediaWireProvenance<'_>,
) -> Result<Vec<GameCultMediaAudioParityShardRecord>, MediaFecError> {
policy.validate()?;
if packets.len() != usize::from(policy.audio_data_shards) {
return Err(MediaFecError::invalid(format!(
"an audio block is {} packets, got {}",
policy.audio_data_shards,
packets.len()
)));
}
let first = &packets[0];
let shard_bytes = first.payload.len();
for (index, packet) in packets.iter().enumerate() {
validate_audio_record(packet).map_err(from_anyhow)?;
let expected =
audio_slot(first.packet_id, first.pts_ticks, first.duration_ticks, index as u64);
if packet.stream_id != first.stream_id
|| packet.session_id != first.session_id
|| packet.codec != first.codec
|| packet.timebase_num != first.timebase_num
|| packet.timebase_den != first.timebase_den
|| packet.duration_ticks != first.duration_ticks
|| Some((packet.packet_id, packet.pts_ticks)) != expected
|| packet.payload.len() != shard_bytes
{
return Err(MediaFecError::invalid(
"audio block packets must be contiguous and share codec, timebase, duration and payload length",
));
}
}
let data: Vec<Vec<u8>> = packets.iter().map(|packet| packet.payload.clone()).collect();
let deadline_ticks = packets.iter().map(|packet| packet.deadline_ticks).max().unwrap_or(0);
let parity: Vec<GameCultMediaAudioParityShardRecord> =
parity_shards(&data, usize::from(policy.audio_parity_shards))?
.into_iter()
.enumerate()
.map(|(parity_index, payload)| GameCultMediaAudioParityShardRecord {
stream_id: first.stream_id.clone(),
session_id: first.session_id.clone(),
codec: first.codec.clone(),
fec_scheme: MEDIA_FEC_SCHEME_RS_GF256_V1.to_string(),
base_packet_id: first.packet_id,
base_pts_ticks: first.pts_ticks,
packet_duration_ticks: first.duration_ticks,
timebase_num: first.timebase_num,
timebase_den: first.timebase_den,
deadline_ticks,
data_shard_count: policy.audio_data_shards,
parity_index: parity_index as u16,
parity_shard_count: policy.audio_parity_shards,
shard_payload_bytes: shard_bytes as u32,
payload,
})
.collect();
// A parity record repeats every field of a packet and adds the scheme and
// the geometry, so it is the larger of the block's records: if it fits, the
// packets do.
for shard in &parity {
fits_datagram(&GameCultMediaWireRecord::AudioParity(shard.clone()), provenance, policy)?;
}
Ok(parity)
}
/// Rebuilds the packets an audio block lost, ordered by `packet_id`. A
/// recovered packet takes its id and presentation time from the block's
/// arithmetic and its deadline from the block's, the latest in the block:
/// gate playout on `pts_ticks`, not `deadline_ticks`. Errors as [`recover_video_block`] does.
pub fn recover_audio_block(
parity: &[GameCultMediaAudioParityShardRecord],
data: &[GameCultMediaAudioPacketRecord],
) -> Result<Vec<GameCultMediaAudioPacketRecord>, MediaFecError> {
let geometry = parity
.first()
.ok_or_else(|| MediaFecError::invalid("recovery needs at least one parity shard"))?;
let k = usize::from(geometry.data_shard_count);
let m = usize::from(geometry.parity_shard_count);
let shard_bytes = geometry.shard_payload_bytes as usize;
let mut shards: Vec<Option<Vec<u8>>> = vec![None; k + m];
for shard in parity {
validate_audio_parity_record(shard).map_err(from_anyhow)?;
if !same_audio_block(geometry, shard) {
return Err(MediaFecError::invalid("parity shards are not from one block"));
}
shards[k + usize::from(shard.parity_index)].get_or_insert_with(|| shard.payload.clone());
}
for packet in data {
validate_audio_record(packet).map_err(from_anyhow)?;
let slot = packet
.packet_id
.checked_sub(geometry.base_packet_id)
.filter(|slot| *slot < k as u64)
.ok_or_else(|| MediaFecError::invalid("audio packet does not belong to the parity's block"))?
as usize;
if packet.stream_id != geometry.stream_id
|| packet.session_id != geometry.session_id
|| packet.payload.len() != shard_bytes
{
return Err(MediaFecError::invalid("audio packet disagrees with the parity's block"));
}
shards[slot].get_or_insert_with(|| packet.payload.clone());
}
let missing: Vec<usize> = (0..k).filter(|&slot| shards[slot].is_none()).collect();
let recovered = recover_data_shards(k, shards)?;
missing
.into_iter()
.map(|slot| {
let (packet_id, pts_ticks) = audio_slot(
geometry.base_packet_id,
geometry.base_pts_ticks,
geometry.packet_duration_ticks,
slot as u64,
)
.ok_or_else(|| MediaFecError::invalid("audio block runs past the packet id or pts range"))?;
Ok(GameCultMediaAudioPacketRecord {
stream_id: geometry.stream_id.clone(),
session_id: geometry.session_id.clone(),
packet_id,
codec: geometry.codec.clone(),
pts_ticks,
duration_ticks: geometry.packet_duration_ticks,
timebase_num: geometry.timebase_num,
timebase_den: geometry.timebase_den,
deadline_ticks: geometry.deadline_ticks,
payload: recovered[slot].clone(),
})
})
.collect()
}
fn same_audio_block(a: &GameCultMediaAudioParityShardRecord, b: &GameCultMediaAudioParityShardRecord) -> bool {
a.stream_id == b.stream_id
&& a.session_id == b.session_id
&& a.codec == b.codec
&& a.fec_scheme == b.fec_scheme
&& a.base_packet_id == b.base_packet_id
&& a.base_pts_ticks == b.base_pts_ticks
&& a.packet_duration_ticks == b.packet_duration_ticks
&& a.timebase_num == b.timebase_num
&& a.timebase_den == b.timebase_den
&& a.deadline_ticks == b.deadline_ticks
&& a.data_shard_count == b.data_shard_count
&& a.parity_shard_count == b.parity_shard_count
&& a.shard_payload_bytes == b.shard_payload_bytes
}

104
packages/cultnet-rs/src/media_stream_contracts.rs
View file

@ -36,15 +36,15 @@
//! Nothing here specifies a delivery guarantee. Whether a stream rides a
//! reliable or lossy CultNet channel, and whether it carries parity, is a
//! transport decision made per stream — not a property of the payload shape.
//! Video parity shards exist for producers that choose forward error correction;
//! audio currently has no parity record, and a producer emitting audio on a
//! lossy channel needs one.
//! Video and audio parity shards exist for producers that choose forward error
//! correction; [`crate::media_fec`] owns the code that makes and consumes them.
use cultcache_rs::DatabaseEntry;
pub const GAMECULT_MEDIA_VIDEO_ACCESS_UNIT_SCHEMA: &str = "gamecult.media_video_access_unit.v1";
pub const GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA: &str = "gamecult.media_video_parity_shard.v2";
pub const GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA: &str = "gamecult.media_video_parity_shard.v3";
pub const GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA: &str = "gamecult.media_audio_packet.v1";
pub const GAMECULT_MEDIA_AUDIO_PARITY_SHARD_SCHEMA: &str = "gamecult.media_audio_parity_shard.v1";
pub const GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA: &str = "gamecult.media_receiver_feedback.v1";
pub const GAMECULT_MEDIA_STREAM_ADVERTISEMENT_SCHEMA: &str =
"gamecult.media_stream_advertisement.v1";
@ -464,15 +464,25 @@ pub struct GameCultMediaVideoAccessUnitRecord {
pub payload: Vec<u8>,
}
/// Forward-error-correction parity for one video access unit. Carries the
/// framing fields of the access unit it protects so a receiver can recover
/// without having seen the original, plus the shard geometry needed to run
/// recovery: `parity_index` within `parity_count`, and the chunk sizes the
/// parity was computed over.
/// Forward-error-correction parity for one block of a video access unit.
///
/// A frame's `chunk_count` chunks are split into `block_count` contiguous
/// blocks, and each block carries `parity_count` parity shards computed over its
/// `block_data_count` data chunks (`block_data_start..block_data_start +
/// block_data_count`). Any `block_data_count` of a block's
/// `block_data_count + parity_count` shards recover it. The math is
/// [`crate::media_fec`]'s and is named by `fec_scheme`; a decoder that does not
/// know the scheme must drop the shard rather than guess.
///
/// It carries the framing fields of the access unit it protects, so a receiver
/// can rebuild a lost chunk without having seen any chunk of the frame.
/// `shard_payload_bytes` is the padded shard length; `last_chunk_payload_bytes`
/// is the true length of the block's final data chunk, the only one that may be
/// shorter.
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(
type = "gamecult.media_video_parity_shard",
schema = "gamecult.media_video_parity_shard.v2"
schema = "gamecult.media_video_parity_shard.v3"
)]
pub struct GameCultMediaVideoParityShardRecord {
#[cultcache(key = 0)]
@ -500,14 +510,26 @@ pub struct GameCultMediaVideoParityShardRecord {
#[cultcache(key = 11)]
pub chunk_count: u16,
#[cultcache(key = 12)]
pub parity_index: u16,
pub fec_scheme: String,
#[cultcache(key = 13)]
pub parity_count: u16,
pub block_index: u16,
#[cultcache(key = 14)]
pub chunk_payload_bytes: u32,
pub block_count: u16,
#[cultcache(key = 15)]
pub block_data_start: u16,
/// `k`: how many data chunks this block protects.
#[cultcache(key = 16)]
pub block_data_count: u16,
#[cultcache(key = 17)]
pub parity_index: u16,
/// `m`: how many parity shards this block carries.
#[cultcache(key = 18)]
pub parity_count: u16,
#[cultcache(key = 19)]
pub shard_payload_bytes: u32,
#[cultcache(key = 20)]
pub last_chunk_payload_bytes: u32,
#[cultcache(key = 16, bytes)]
#[cultcache(key = 21, bytes)]
pub payload: Vec<u8>,
}
@ -541,6 +563,60 @@ pub struct GameCultMediaAudioPacketRecord {
pub payload: Vec<u8>,
}
/// Forward-error-correction parity for a block of consecutive audio packets.
///
/// The block is `data_shard_count` packets with contiguous ids starting at
/// `base_packet_id`, all with one payload length (`shard_payload_bytes`) and
/// contiguous presentation times spaced `packet_duration_ticks` apart. A packet
/// recovered from parity takes its id and time from that arithmetic, and its
/// `deadline_ticks` from the block's, which is the latest in the block: the
/// per-packet deadline is not recoverable, so a recovered packet may be held
/// past its own but never past the block's.
///
/// Codecs whose frames vary in length pad each one to a shared shard length
/// before parity is computed; that framing belongs to the codec, not to this
/// record.
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(
type = "gamecult.media_audio_parity_shard",
schema = "gamecult.media_audio_parity_shard.v1"
)]
pub struct GameCultMediaAudioParityShardRecord {
#[cultcache(key = 0)]
pub stream_id: String,
#[cultcache(key = 1)]
pub session_id: String,
#[cultcache(key = 2)]
pub codec: String,
#[cultcache(key = 3)]
pub fec_scheme: String,
#[cultcache(key = 4)]
pub base_packet_id: u64,
#[cultcache(key = 5)]
pub base_pts_ticks: i64,
#[cultcache(key = 6)]
pub packet_duration_ticks: u32,
#[cultcache(key = 7)]
pub timebase_num: u32,
#[cultcache(key = 8)]
pub timebase_den: u32,
/// The latest `deadline_ticks` among the block's data packets.
#[cultcache(key = 9)]
pub deadline_ticks: i64,
/// `k`.
#[cultcache(key = 10)]
pub data_shard_count: u16,
#[cultcache(key = 11)]
pub parity_index: u16,
/// `m`.
#[cultcache(key = 12)]
pub parity_shard_count: u16,
#[cultcache(key = 13)]
pub shard_payload_bytes: u32,
#[cultcache(key = 14, bytes)]
pub payload: Vec<u8>,
}
/// What a receiver observed, published back to the producer so it can adapt:
/// what decoded, what is missing, what arrived too late, and how much queue and
/// jitter the receiver is carrying.

161
packages/cultnet-rs/src/media_stream_wire.rs
View file

@ -28,20 +28,32 @@ use crate::contracts::{
CultNetMessage, CultNetRawDocumentRecord, CultNetRawPayloadEncoding, CultNetWireContract,
decode_cultnet_message_from_slice, encode_cultnet_message_to_vec,
};
use crate::media_fec::{MEDIA_FEC_SCHEME_RS_GF256_V1, audio_slot};
use crate::media_stream_contracts::{
GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA, GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA,
GAMECULT_MEDIA_VIDEO_ACCESS_UNIT_SCHEMA, GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA,
GameCultMediaAudioPacketRecord, GameCultMediaReceiverFeedbackRecord,
GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA, GAMECULT_MEDIA_AUDIO_PARITY_SHARD_SCHEMA,
GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA, GAMECULT_MEDIA_VIDEO_ACCESS_UNIT_SCHEMA,
GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA, GameCultMediaAudioPacketRecord,
GameCultMediaAudioParityShardRecord, GameCultMediaReceiverFeedbackRecord,
GameCultMediaVideoAccessUnitRecord, GameCultMediaVideoParityShardRecord,
};
/// The CultNet channel video and receiver feedback ride.
/// The CultNet channel every media record rides: video, audio, their parity,
/// and receiver feedback. Whether it is reliable is the transport profile's
/// choice per session; stale media is dropped, not retransmitted.
pub const GAMECULT_MEDIA_CHANNEL: &str = "media";
/// The CultNet channel audio rides. Audio has its own channel so one session
/// can carry it reliably with a short expiry while video stays lossy under
/// parity; the RUDP transport profile owns each channel's delivery.
pub const GAMECULT_MEDIA_AUDIO_CHANNEL: &str = "audio";
/// The largest wrapped media record the decoder accepts, and the most the RUDP
/// `media` channel reassembles: 64 KiB.
///
/// It is the bound a receiver holds a peer to. A parity shard's length is
/// whatever its producer's [`MediaFecPolicy::max_wire_bytes`](crate::MediaFecPolicy::max_wire_bytes) allowed, and
/// recovery work grows with the square of the block size times that length, so
/// without a ceiling one hostile record could name any length RUDP would
/// reassemble (16 MiB). 64 KiB is one maximum UDP datagram: it covers every
/// path MTU in use, jumbo frames (9,000 bytes) with a wide margin, and a
/// record is still one message rather than a stream. A policy may not ask for
/// more; see [`MediaFecPolicy`](crate::MediaFecPolicy).
pub const GAMECULT_MEDIA_MAX_WIRE_BYTES: usize = 64 * 1024;
/// One media record on its way somewhere.
#[derive(Clone, Debug, PartialEq, Eq)]
@ -49,6 +61,7 @@ pub enum GameCultMediaWireRecord {
Video(GameCultMediaVideoAccessUnitRecord),
VideoParity(GameCultMediaVideoParityShardRecord),
Audio(GameCultMediaAudioPacketRecord),
AudioParity(GameCultMediaAudioParityShardRecord),
Feedback(GameCultMediaReceiverFeedbackRecord),
}
@ -84,6 +97,7 @@ impl GameCultMediaWireRecord {
Self::Video(_) => GAMECULT_MEDIA_VIDEO_ACCESS_UNIT_SCHEMA,
Self::VideoParity(_) => GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA,
Self::Audio(_) => GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA,
Self::AudioParity(_) => GAMECULT_MEDIA_AUDIO_PARITY_SHARD_SCHEMA,
Self::Feedback(_) => GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA,
}
}
@ -96,13 +110,21 @@ impl GameCultMediaWireRecord {
record.stream_id, record.session_id, record.frame_id, record.chunk_index
),
Self::VideoParity(record) => format!(
"{}:{}:video-parity:{}:{}",
record.stream_id, record.session_id, record.frame_id, record.parity_index
"{}:{}:video-parity:{}:{}:{}",
record.stream_id,
record.session_id,
record.frame_id,
record.block_index,
record.parity_index
),
Self::Audio(record) => format!(
"{}:{}:audio:{}",
record.stream_id, record.session_id, record.packet_id
),
Self::AudioParity(record) => format!(
"{}:{}:audio-parity:{}:{}",
record.stream_id, record.session_id, record.base_packet_id, record.parity_index
),
Self::Feedback(record) => format!(
"{}:{}:feedback:{}",
record.stream_id, record.session_id, record.receiver_id
@ -124,6 +146,7 @@ impl GameCultMediaWireRecord {
Self::Video(record) => rmp_serde::to_vec(record),
Self::VideoParity(record) => rmp_serde::to_vec(record),
Self::Audio(record) => rmp_serde::to_vec(record),
Self::AudioParity(record) => rmp_serde::to_vec(record),
Self::Feedback(record) => rmp_serde::to_vec(record),
}
.map_err(Into::into)
@ -159,11 +182,24 @@ pub fn encode_media_wire_record(
},
};
encode_cultnet_message_to_vec(&message, CultNetWireContract::CultNetSchemaV0).map_err(Into::into)
let wire = encode_cultnet_message_to_vec(&message, CultNetWireContract::CultNetSchemaV0)?;
if wire.len() > GAMECULT_MEDIA_MAX_WIRE_BYTES {
return Err(anyhow!(
"media record is {} bytes, over the {GAMECULT_MEDIA_MAX_WIRE_BYTES}-byte ceiling",
wire.len()
));
}
Ok(wire)
}
/// Unwraps a media record from the wire.
pub fn decode_media_wire_record(payload: &[u8]) -> Result<GameCultMediaWireRecord> {
if payload.len() > GAMECULT_MEDIA_MAX_WIRE_BYTES {
return Err(anyhow!(
"media record is {} bytes, over the {GAMECULT_MEDIA_MAX_WIRE_BYTES}-byte ceiling",
payload.len()
));
}
let message = decode_cultnet_message_from_slice(payload, CultNetWireContract::CultNetSchemaV0)?;
let CultNetMessage::DocumentPutRaw { document, .. } = message else {
return Err(anyhow!("expected cultnet.document_put_raw.v0"));
@ -191,6 +227,12 @@ pub fn decode_media_wire_record(payload: &[u8]) -> Result<GameCultMediaWireRecor
validate_audio_record(&record)?;
GameCultMediaWireRecord::Audio(record)
}
GAMECULT_MEDIA_AUDIO_PARITY_SHARD_SCHEMA => {
let record: GameCultMediaAudioParityShardRecord =
rmp_serde::from_slice(&document.payload)?;
validate_audio_parity_record(&record)?;
GameCultMediaWireRecord::AudioParity(record)
}
GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA => {
let record: GameCultMediaReceiverFeedbackRecord =
rmp_serde::from_slice(&document.payload)?;
@ -349,39 +391,114 @@ pub fn validate_video_parity_record(record: &GameCultMediaVideoParityShardRecord
"video parity media record deadline_ticks must not precede pts_ticks"
));
}
if record.fec_scheme != MEDIA_FEC_SCHEME_RS_GF256_V1 {
return Err(anyhow!(
"video parity media record fec_scheme {:?} is not supported",
record.fec_scheme
));
}
if record.chunk_count == 0 {
return Err(anyhow!(
"video parity media record chunk_count must be non-zero"
));
}
if record.block_count == 0 || record.block_index >= record.block_count {
return Err(anyhow!(
"video parity media record block_index {} is outside block_count {}",
record.block_index,
record.block_count
));
}
if record.block_data_count == 0
|| u32::from(record.block_data_start) + u32::from(record.block_data_count)
> u32::from(record.chunk_count)
{
return Err(anyhow!(
"video parity media record block does not lie inside the frame's chunks"
));
}
if record.parity_count == 0 || record.parity_index >= record.parity_count {
return Err(anyhow!(
"video parity media record has invalid parity stripe metadata"
"video parity media record parity_index {} is outside parity_count {}",
record.parity_index,
record.parity_count
));
}
if record.parity_count > record.chunk_count {
if usize::from(record.block_data_count) + usize::from(record.parity_count) > 256 {
return Err(anyhow!(
"video parity media record parity_count exceeds chunk_count"
"video parity media record block exceeds 256 shards"
));
}
if record.chunk_payload_bytes == 0 || record.last_chunk_payload_bytes == 0 {
if record.shard_payload_bytes == 0 {
return Err(anyhow!(
"video parity media record chunk lengths must be non-zero"
"video parity media record shard_payload_bytes must be non-zero"
));
}
if record.last_chunk_payload_bytes > record.chunk_payload_bytes {
if record.last_chunk_payload_bytes == 0
|| record.last_chunk_payload_bytes > record.shard_payload_bytes
{
return Err(anyhow!(
"video parity media record last chunk length exceeds regular chunk length"
"video parity media record last chunk length must be in 1..=shard_payload_bytes"
));
}
if record.payload.is_empty() {
if record.payload.len() != record.shard_payload_bytes as usize {
return Err(anyhow!(
"video parity media record payload must be non-empty"
"video parity media record payload does not match declared shard length"
));
}
if record.payload.len() > record.chunk_payload_bytes as usize {
Ok(())
}
pub fn validate_audio_parity_record(record: &GameCultMediaAudioParityShardRecord) -> Result<()> {
if record.stream_id.is_empty() || record.session_id.is_empty() || record.codec.is_empty() {
return Err(anyhow!(
"video parity media record payload exceeds declared chunk length"
"audio parity media record stream_id, session_id and codec must be non-empty"
));
}
if record.timebase_num == 0 || record.timebase_den == 0 || record.packet_duration_ticks == 0 {
return Err(anyhow!(
"audio parity media record timing metadata must be non-zero"
));
}
if record.fec_scheme != MEDIA_FEC_SCHEME_RS_GF256_V1 {
return Err(anyhow!(
"audio parity media record fec_scheme {:?} is not supported",
record.fec_scheme
));
}
if record.data_shard_count == 0
|| record.parity_shard_count == 0
|| record.parity_index >= record.parity_shard_count
|| usize::from(record.data_shard_count) + usize::from(record.parity_shard_count) > 256
{
return Err(anyhow!("audio parity media record stripe metadata is invalid"));
}
let Some((_, last_pts_ticks)) = audio_slot(
record.base_packet_id,
record.base_pts_ticks,
record.packet_duration_ticks,
u64::from(record.data_shard_count) - 1,
) else {
return Err(anyhow!(
"audio parity media record block runs past the packet id or pts range"
));
};
// Recovery gives every recovered packet the block's deadline, so it must
// not precede the last packet's pts or a recovered packet would fail
// `validate_audio_record`.
if record.deadline_ticks < last_pts_ticks {
return Err(anyhow!(
"audio parity media record deadline_ticks must not precede the block's last pts_ticks"
));
}
if record.shard_payload_bytes == 0 {
return Err(anyhow!(
"audio parity media record shard_payload_bytes must be non-zero"
));
}
if record.payload.len() != record.shard_payload_bytes as usize {
return Err(anyhow!(
"audio parity media record payload does not match declared shard length"
));
}
Ok(())

675
packages/cultnet-rs/src/rudp.rs
View file

File diff suppressed because it is too large Load diff

585
packages/cultnet-rs/tests/cultnet.rs
View file

@ -432,12 +432,6 @@ fn rudp_transport_profile_advertises_state_and_realtime_channels() {
CultNetTransportOrdering::Unordered,
Some(64)
),
(
"audio",
CultNetTransportDelivery::Reliable,
CultNetTransportOrdering::Unordered,
Some(64)
),
]
);
}
@ -558,13 +552,13 @@ fn rudp_reconnect_loop_consumes_shared_controller() -> Result<()> {
fn rudp_session_handshake_acks_reliable_connect_and_accept_packets() -> Result<()> {
let mut client = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 0x0a0b0c0d,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 50,
..CultNetRudpSessionOptions::default()
});
let mut server = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 0x0a0b0c0d,
initial_sequence: 100,
initial_sequence: Some(100),
resend_delay_ms: 50,
..CultNetRudpSessionOptions::default()
});
@ -595,52 +589,19 @@ fn rudp_session_handshake_acks_reliable_connect_and_accept_packets() -> Result<(
fn rudp_session_computes_ack_masks_and_clears_pending_reliable_packets() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 7,
initial_sequence: 10,
initial_sequence: Some(10),
resend_delay_ms: 100,
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 7,
initial_sequence: 200,
initial_sequence: Some(200),
resend_delay_ms: 100,
..CultNetRudpSessionOptions::default()
});
sender.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 7,
sequence: 1,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
receiver.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 7,
sequence: 2,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
let connect = sender.create_connect(0, Vec::new())?;
let accept = receiver.accept_connect(&connect, 0, Vec::new())?;
sender.receive(&accept, 0)?;
let options = CultNetRudpSendOptions {
reliable: true,
@ -650,20 +611,20 @@ fn rudp_session_computes_ack_masks_and_clears_pending_reliable_packets() -> Resu
let first = sender.send("schema", b"first".to_vec(), options.clone())?;
let second = sender.send("schema", b"second".to_vec(), options.clone())?;
let third = sender.send("schema", b"third".to_vec(), options)?;
assert_eq!(sender.pending_reliable_sequences(), vec![10, 11, 12]);
assert_eq!(sender.pending_reliable_sequences(), vec![11, 12, 13]);
receiver.receive(&first, 0)?;
receiver.receive(&third, 0)?;
let ack_with_gap = receiver.create_ack();
assert_eq!(ack_with_gap.ack, 12);
assert_eq!(ack_with_gap.ack_mask, 0b10 | (1 << 9));
assert_eq!(ack_with_gap.ack, 13);
assert_eq!(ack_with_gap.ack_mask, 0b110);
sender.receive(&ack_with_gap, 0)?;
assert_eq!(sender.pending_reliable_sequences(), vec![11]);
assert_eq!(sender.pending_reliable_sequences(), vec![12]);
receiver.receive(&second, 0)?;
let full_ack = receiver.create_ack();
assert_eq!(full_ack.ack, 12);
assert_eq!(full_ack.ack_mask, 0b11 | (1 << 9));
assert_eq!(full_ack.ack, 13);
assert_eq!(full_ack.ack_mask, 0b111);
sender.receive(&full_ack, 0)?;
assert!(sender.pending_reliable_sequences().is_empty());
Ok(())
@ -673,28 +634,11 @@ fn rudp_session_computes_ack_masks_and_clears_pending_reliable_packets() -> Resu
fn rudp_session_schedules_reliable_resends_until_acked() -> Result<()> {
let mut session = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 99,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 100,
..CultNetRudpSessionOptions::default()
});
session.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 99,
sequence: 50,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
session.assume_connected(0);
let sent = session.send(
"schema",
b"payload".to_vec(),
@ -743,12 +687,12 @@ fn rudp_session_schedules_reliable_resends_until_acked() -> Result<()> {
fn rudp_session_pings_and_detects_receive_timeout() -> Result<()> {
let mut client = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 101,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut server = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 101,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = client.create_connect(0, b"join".to_vec())?;
@ -774,12 +718,12 @@ fn rudp_session_pings_and_detects_receive_timeout() -> Result<()> {
fn rudp_lossy_packets_cannot_create_reliable_ordered_gaps() -> Result<()> {
let mut client = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 198,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut server = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 198,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = client.create_connect(0, Vec::new())?;
@ -830,12 +774,12 @@ fn rudp_lossy_packets_cannot_create_reliable_ordered_gaps() -> Result<()> {
fn rudp_unreliable_sequenced_delivery_is_scoped_to_its_channel() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 197,
initial_sequence: 50,
initial_sequence: Some(50),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 197,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = sender.create_connect(0, Vec::new())?;
@ -861,24 +805,7 @@ fn rudp_rejects_unreliable_ordered_delivery() -> Result<()> {
connection_id: 196,
..CultNetRudpSessionOptions::default()
});
session.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 196,
sequence: 50,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
session.assume_connected(0);
let error = session
.send(
"schema",
@ -901,28 +828,11 @@ fn rudp_rejects_unreliable_ordered_delivery() -> Result<()> {
fn rudp_session_bounds_pending_reliable_packets_before_enqueue() -> Result<()> {
let mut session = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 102,
initial_sequence: 1,
initial_sequence: Some(1),
max_pending_reliable_packets: Some(2),
..CultNetRudpSessionOptions::default()
});
session.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 102,
sequence: 50,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
session.assume_connected(0);
session.send(
"schema",
b"first".to_vec(),
@ -957,28 +867,11 @@ fn rudp_session_bounds_pending_reliable_packets_before_enqueue() -> Result<()> {
let mut fragmented = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 103,
initial_sequence: 1,
initial_sequence: Some(1),
max_pending_reliable_packets: Some(3),
..CultNetRudpSessionOptions::default()
});
fragmented.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 103,
sequence: 50,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
fragmented.assume_connected(0);
let error = fragmented
.send_many(
"schema",
@ -1001,50 +894,17 @@ fn rudp_session_suppresses_duplicates_and_delivers_reliable_ordered_payloads_in_
-> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 123,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 123,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
sender.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 123,
sequence: 90,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
receiver.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 123,
sequence: 91,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
let connect = sender.create_connect(0, Vec::new())?;
let accept = receiver.accept_connect(&connect, 0, Vec::new())?;
sender.receive(&accept, 0)?;
let options = CultNetRudpSendOptions {
reliable: true,
@ -1082,50 +942,17 @@ fn rudp_session_suppresses_duplicates_and_delivers_reliable_ordered_payloads_in_
fn rudp_session_skips_control_packets_while_ordering_schema_payloads() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 124,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 124,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
sender.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 124,
sequence: 90,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
receiver.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 124,
sequence: 91,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
let connect = sender.create_connect(0, Vec::new())?;
let accept = receiver.accept_connect(&connect, 0, Vec::new())?;
sender.receive(&accept, 0)?;
let options = CultNetRudpSendOptions {
reliable: true,
@ -1164,50 +991,17 @@ fn rudp_session_skips_control_packets_while_ordering_schema_payloads() -> Result
fn rudp_session_fragments_and_reassembles_reliable_ordered_payloads() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 456,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 456,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
sender.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 456,
sequence: 90,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
receiver.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id: 456,
sequence: 91,
ack: 0,
ack_mask: 0,
channel_id: "control".to_string(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
let connect = sender.create_connect(0, Vec::new())?;
let accept = receiver.accept_connect(&connect, 0, Vec::new())?;
sender.receive(&accept, 0)?;
let packets = sender.send_many(
"schema",
@ -1265,7 +1059,7 @@ fn rudp_socket_transport_handshakes_and_carries_reliable_ordered_schema_frames()
mode: CultNetRudpSocketMode::Server,
remote_addr: None,
connection_id,
initial_sequence: 100,
initial_sequence: Some(100),
resend_delay_ms: 25,
transport_id: None,
max_payload_bytes: None,
@ -1282,7 +1076,7 @@ fn rudp_socket_transport_handshakes_and_carries_reliable_ordered_schema_frames()
mode: CultNetRudpSocketMode::Client,
remote_addr: Some(server_addr),
connection_id,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 25,
transport_id: None,
max_payload_bytes: None,
@ -1359,7 +1153,7 @@ fn rudp_socket_transport_carries_fragmented_reliable_ordered_schema_frames() ->
mode: CultNetRudpSocketMode::Server,
remote_addr: None,
connection_id,
initial_sequence: 100,
initial_sequence: Some(100),
resend_delay_ms: 25,
transport_id: None,
max_payload_bytes: None,
@ -1376,7 +1170,7 @@ fn rudp_socket_transport_carries_fragmented_reliable_ordered_schema_frames() ->
mode: CultNetRudpSocketMode::Client,
remote_addr: Some(server_addr),
connection_id,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 25,
transport_id: None,
max_payload_bytes: None,
@ -1404,36 +1198,19 @@ fn rudp_session_advances_large_fragment_sets_through_a_bounded_reliable_window()
let connection_id = 0x1020_3042;
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 25,
max_pending_reliable_packets: None,
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id,
initial_sequence: 100,
initial_sequence: Some(100),
resend_delay_ms: 25,
max_pending_reliable_packets: None,
});
for session in [&mut sender, &mut receiver] {
session.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id,
sequence: 0,
ack: 0,
ack_mask: 0,
channel_id: "control".into(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
}
let connect = sender.create_connect(0, Vec::new())?;
let accept = receiver.accept_connect(&connect, 0, Vec::new())?;
sender.receive(&accept, 0)?;
let fragment_count = CULTNET_RUDP_RELIABLE_SEND_WINDOW_PACKETS + 17;
let payload = (0..fragment_count * 8)
@ -1491,7 +1268,7 @@ fn rudp_socket_flush_waits_for_large_fragment_delivery() -> Result<()> {
mode: CultNetRudpSocketMode::Server,
remote_addr: None,
connection_id,
initial_sequence: 100,
initial_sequence: Some(100),
resend_delay_ms: 25,
transport_id: None,
max_payload_bytes: None,
@ -1508,7 +1285,7 @@ fn rudp_socket_flush_waits_for_large_fragment_delivery() -> Result<()> {
mode: CultNetRudpSocketMode::Client,
remote_addr: Some(server_addr),
connection_id,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 25,
transport_id: None,
max_payload_bytes: None,
@ -2615,12 +2392,12 @@ fn schema_discovery_round_trips_over_legacy_gamecult_contract_when_inline_schema
fn rudp_sequenced_channel_tracking_is_bounded() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 401,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 401,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = sender.create_connect(0, Vec::new())?;
@ -2655,18 +2432,30 @@ fn rudp_sequenced_channel_tracking_is_bounded() -> Result<()> {
fn rudp_ordered_channel_tracking_is_bounded() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 402,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 402,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = sender.create_connect(0, Vec::new())?;
let accept = receiver.accept_connect(&connect, 1, Vec::new())?;
sender.receive(&accept, 2)?;
// Channels are tracked only while they hold a frame, so the bound is on
// channels held behind one missing sequence.
let gap = sender.send(
"gap",
b"lost".to_vec(),
CultNetRudpSendOptions {
reliable: true,
ordered: false,
..CultNetRudpSendOptions::default()
},
)?;
sender.receive(&receiver.create_ack_for(gap.sequence), 3)?;
let mut error = None;
for channel in 0..256u32 {
let packet = sender.send(
@ -2679,7 +2468,8 @@ fn rudp_ordered_channel_tracking_is_bounded() -> Result<()> {
},
)?;
match receiver.receive(&packet, 3) {
Ok(_) => {
Ok(result) => {
assert!(result.delivered.is_empty(), "the gap holds every frame");
sender.receive(&receiver.create_ack_for(packet.sequence), 3)?;
}
Err(refused) => {
@ -2700,12 +2490,12 @@ fn rudp_ordered_channel_tracking_is_bounded() -> Result<()> {
fn rudp_ordered_hold_buffer_is_bounded() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 403,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 403,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = sender.create_connect(0, Vec::new())?;
@ -2743,16 +2533,196 @@ fn rudp_ordered_hold_buffer_is_bounded() -> Result<()> {
Ok(())
}
fn rudp_ordered_pair() -> Result<(CultNetRudpSession, CultNetRudpSession)> {
let mut client = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 410,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut server = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 410,
initial_sequence: Some(500),
..CultNetRudpSessionOptions::default()
});
let connect = client.create_connect(0, Vec::new())?;
let accept = server.accept_connect(&connect, 0, Vec::new())?;
client.receive(&accept, 0)?;
Ok((client, server))
}
fn rudp_send_options(ordered: bool) -> CultNetRudpSendOptions {
CultNetRudpSendOptions {
reliable: true,
ordered,
..CultNetRudpSendOptions::default()
}
}
fn rudp_delivered_names(delivered: &[cultnet_rs::CultNetRudpDeliveredFrame]) -> Vec<String> {
delivered
.iter()
.map(|frame| String::from_utf8(frame.payload.clone()).unwrap())
.collect()
}
#[test]
fn rudp_ordered_frame_waits_for_a_gap_filled_by_another_channel() -> Result<()> {
let (mut sender, mut receiver) = rudp_ordered_pair()?;
let o1 = sender.send("schema", b"o1".to_vec(), rudp_send_options(true))?;
let u = sender.send("media", b"u".to_vec(), rudp_send_options(false))?;
let o2 = sender.send("schema", b"o2".to_vec(), rudp_send_options(true))?;
let o3 = sender.send("schema", b"o3".to_vec(), rudp_send_options(true))?;
let o4 = sender.send("schema", b"o4".to_vec(), rudp_send_options(true))?;
assert_eq!(rudp_delivered_names(&receiver.receive(&o1, 1)?.delivered), ["o1"]);
// `u` is lost, so nothing behind it may be delivered.
assert!(receiver.receive(&o2, 2)?.delivered.is_empty());
assert!(receiver.receive(&o3, 3)?.delivered.is_empty());
assert!(receiver.receive(&o4, 4)?.delivered.is_empty());
// The other channel's packet fills the gap and releases every ordered
// frame behind it in the same call.
assert_eq!(
rudp_delivered_names(&receiver.receive(&u, 5)?.delivered),
["u", "o2", "o3", "o4"]
);
Ok(())
}
#[test]
fn rudp_two_ordered_channels_release_each_other_in_sequence_order() -> Result<()> {
let (mut sender, mut receiver) = rudp_ordered_pair()?;
let a1 = sender.send("schema", b"A1".to_vec(), rudp_send_options(true))?;
let b1 = sender.send("other", b"B1".to_vec(), rudp_send_options(true))?;
let a2 = sender.send("schema", b"A2".to_vec(), rudp_send_options(true))?;
let a3 = sender.send("schema", b"A3".to_vec(), rudp_send_options(true))?;
let mut delivered = rudp_delivered_names(&receiver.receive(&a1, 1)?.delivered);
delivered.extend(rudp_delivered_names(&receiver.receive(&a2, 2)?.delivered));
delivered.extend(rudp_delivered_names(&receiver.receive(&a3, 3)?.delivered));
assert_eq!(delivered, ["A1"], "B1 is missing, so A2 and A3 wait");
assert_eq!(
rudp_delivered_names(&receiver.receive(&b1, 4)?.delivered),
["B1", "A2", "A3"]
);
Ok(())
}
#[test]
fn rudp_channel_first_used_after_other_traffic_loses_nothing() -> Result<()> {
let (mut sender, mut receiver) = rudp_ordered_pair()?;
let c1 = sender.send("late", b"C1".to_vec(), rudp_send_options(true))?;
let c2 = sender.send("late", b"C2".to_vec(), rudp_send_options(true))?;
let x = sender.send("schema", b"X".to_vec(), rudp_send_options(true))?;
// X, then C2, arrive while C1 is still in flight.
let mut delivered = rudp_delivered_names(&receiver.receive(&x, 1)?.delivered);
delivered.extend(rudp_delivered_names(&receiver.receive(&c2, 2)?.delivered));
assert!(delivered.is_empty(), "C1 is missing, so nothing is delivered");
assert_eq!(
rudp_delivered_names(&receiver.receive(&c1, 3)?.delivered),
["C1", "C2", "X"]
);
Ok(())
}
#[test]
fn rudp_accept_seeds_the_watermark_of_the_connecting_side() -> Result<()> {
let (mut client, mut server) = rudp_ordered_pair()?;
let s1 = server.send("schema", b"s1".to_vec(), rudp_send_options(true))?;
let s2 = server.send("schema", b"s2".to_vec(), rudp_send_options(true))?;
assert!(client.receive(&s2, 1)?.delivered.is_empty());
assert_eq!(
rudp_delivered_names(&client.receive(&s1, 2)?.delivered),
["s1", "s2"]
);
Ok(())
}
#[test]
fn rudp_reset_forgets_held_frames_and_reseeds_the_watermark_from_the_next_handshake() -> Result<()> {
let (mut sender, mut receiver) = rudp_ordered_pair()?;
sender.send("schema", b"s1".to_vec(), rudp_send_options(true))?;
let s2 = sender.send("schema", b"s2".to_vec(), rudp_send_options(true))?;
assert!(receiver.receive(&s2, 1)?.delivered.is_empty());
receiver.reset_peer_state();
let mut next = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 410,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let accept = receiver.accept_connect(&next.create_connect(0, Vec::new())?, 2, Vec::new())?;
next.receive(&accept, 2)?;
let d1 = next.send("schema", b"d1".to_vec(), rudp_send_options(true))?;
let d2 = next.send("schema", b"d2".to_vec(), rudp_send_options(true))?;
// The held s2 died with the reset, and the watermark follows the new peer.
assert!(receiver.receive(&d2, 3)?.delivered.is_empty());
assert_eq!(rudp_delivered_names(&receiver.receive(&d1, 4)?.delivered), ["d1", "d2"]);
Ok(())
}
#[test]
fn rudp_large_ordered_frame_without_a_gap_is_never_held_or_refused() -> Result<()> {
let (mut sender, mut receiver) = rudp_ordered_pair()?;
let packets = sender.send_many(
"schema",
vec![7u8; 5 * 1024 * 1024],
rudp_send_options(true),
Some(60_000),
)?;
let mut queue = packets;
let mut delivered = Vec::new();
while !queue.is_empty() {
let mut next = Vec::new();
for packet in &queue {
delivered.extend(receiver.receive(packet, 1)?.delivered);
let ack = receiver.create_ack_for_received(packet.sequence);
next.extend(sender.receive(&ack, 1)?.ready_to_send);
}
queue = next;
}
assert_eq!(delivered.len(), 1);
assert_eq!(delivered[0].payload.len(), 5 * 1024 * 1024);
Ok(())
}
#[test]
fn rudp_ordered_hold_buffer_is_bounded_in_bytes() -> Result<()> {
let (mut sender, mut receiver) = rudp_ordered_pair()?;
let withheld = sender.send("schema", b"first".to_vec(), rudp_send_options(true))?;
sender.receive(&receiver.create_ack_for(withheld.sequence), 1)?;
// Four 1 MiB frames fit the 4 MiB hold; the fifth does not.
let mut refused_at = None;
for step in 0..8u32 {
let packet = sender.send("schema", vec![7u8; 1024 * 1024], rudp_send_options(true))?;
match receiver.receive(&packet, 2) {
Ok(result) => {
assert!(result.delivered.is_empty(), "the gap holds every frame");
sender.receive(&receiver.create_ack_for(packet.sequence), 2)?;
}
Err(refusal) => {
assert!(refusal.to_string().contains("ordered hold buffer is full"), "{refusal}");
refused_at = Some(step);
break;
}
}
}
assert_eq!(refused_at, Some(4));
Ok(())
}
#[test]
fn rudp_far_ahead_sequence_cannot_starve_the_session() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 404,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 404,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = sender.create_connect(0, Vec::new())?;
@ -2801,12 +2771,12 @@ fn rudp_reliable_sequence_exhaustion_is_reported_not_fatal() -> Result<()> {
connection_id: 405,
// One sequence left: the handshake takes it, so the next send is the
// first allocation with nowhere to go.
initial_sequence: u32::MAX - 1,
initial_sequence: Some(u32::MAX - 1),
..CultNetRudpSessionOptions::default()
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 405,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = sender.create_connect(0, Vec::new())?;
@ -2870,7 +2840,7 @@ fn tcp_framed_receive_refuses_a_length_beyond_the_advertised_maximum() -> Result
fn rudp_reliable_expiry_drops_late_packets_and_leaves_durable_sends_alone() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 7,
initial_sequence: 10,
initial_sequence: Some(10),
resend_delay_ms: 100,
..CultNetRudpSessionOptions::default()
});
@ -2934,7 +2904,7 @@ fn rudp_reliable_expiry_drops_late_packets_and_leaves_durable_sends_alone() -> R
fn rudp_reliable_expiry_also_reclaims_the_send_backlog() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 9,
initial_sequence: 10,
initial_sequence: Some(10),
resend_delay_ms: 100,
..CultNetRudpSessionOptions::default()
});
@ -2983,30 +2953,13 @@ fn rudp_reliable_expiry_also_reclaims_the_send_backlog() -> Result<()> {
Ok(())
}
fn accept_rudp_session(session: &mut CultNetRudpSession, connection_id: u32) -> Result<()> {
session.receive(
&CultNetRudpPacket {
packet_type: CultNetRudpPacketType::Accept,
connection_id,
sequence: 0,
ack: 0,
ack_mask: 0,
channel_id: "control".into(),
reliable: false,
ordered: false,
sequenced: false,
fragment_id: 0,
fragment_index: 0,
fragment_count: 0,
payload: Vec::new(),
},
0,
)?;
fn accept_rudp_session(session: &mut CultNetRudpSession, _connection_id: u32) -> Result<()> {
session.assume_connected(0);
Ok(())
}
#[test]
fn rudp_profile_expires_only_the_media_channels() -> Result<()> {
fn rudp_profile_expires_only_the_media_channel() -> Result<()> {
let socket = std::net::UdpSocket::bind("127.0.0.1:0")?;
let peer = socket.local_addr()?;
let options = CultNetRudpSocketTransportOptions::client("expiry-profile", socket, peer, 11);
@ -3020,14 +2973,12 @@ fn rudp_profile_expires_only_the_media_channels() -> Result<()> {
.map(|channel| (channel.channel_id.as_str(), channel.reliable_expire_after_ms))
.collect::<Vec<_>>();
for media_channel in ["media", "audio"] {
assert!(
channels.contains(&(media_channel, Some(75))),
"the {media_channel} channel advertises its expiry so a peer can see it: {channels:?}"
);
}
assert!(
channels.contains(&("media", Some(75))),
"the media channel advertises its expiry so a peer can see it: {channels:?}"
);
for (channel_id, expiry) in &channels {
if !matches!(*channel_id, "media" | "audio") {
if *channel_id != "media" {
assert_eq!(
*expiry, None,
"state channel {channel_id} must never expire a reliable send"
@ -3040,12 +2991,12 @@ fn rudp_profile_expires_only_the_media_channels() -> Result<()> {
fn connected_rudp_pair() -> Result<(CultNetRudpSession, CultNetRudpSession)> {
let mut client = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 311,
initial_sequence: 1,
initial_sequence: Some(1),
..CultNetRudpSessionOptions::default()
});
let mut server = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 311,
initial_sequence: 100,
initial_sequence: Some(100),
..CultNetRudpSessionOptions::default()
});
let connect = client.create_connect(0, Vec::new())?;

40
packages/cultnet-rs/tests/error_message.rs
View file

@ -118,6 +118,46 @@ fn cases() -> Vec<Case> {
details: None,
},
},
Case {
name: "variant_unsupported_subscription_ended",
// C#: ForVariantUnsupported("variant unsupported here", "sub-1")
csharp_hex: "85AD736368656D6156657273696F6EB063756C746E65742E6572726F722E7630A56572726F72B876617269616E7420756E737570706F727465642068657265AB726F7574696E6748696E74C0A4636F6465B376617269616E745F756E737570706F72746564A764657461696C7384A56669656C64AE737562736372697074696F6E4964A576616C7565A57375622D31A461734F66C0A763757272656E74C0",
message: || CultNetMessage::Error {
error: "variant unsupported here".to_string(),
code: Some(CultNetErrorCode::VariantUnsupported),
details: Some(CultNetErrorDetails {
field: Some("subscriptionId".to_string()),
value: Some("sub-1".to_string()),
as_of: None,
current: None,
}),
},
},
Case {
name: "variant_unsupported_no_subscription",
// C#: ForVariantUnsupported("variant unsupported here") - no subscription ended, so Details stays null.
csharp_hex: "85AD736368656D6156657273696F6EB063756C746E65742E6572726F722E7630A56572726F72B876617269616E7420756E737570706F727465642068657265AB726F7574696E6748696E74C0A4636F6465B376617269616E745F756E737570706F72746564A764657461696C73C0",
message: || CultNetMessage::Error {
error: "variant unsupported here".to_string(),
code: Some(CultNetErrorCode::VariantUnsupported),
details: None,
},
},
Case {
name: "unowned_schema",
// C#: ForUnownedSchema(new CultNetUnownedSchemaException("some.schema.v1", new CultRecordKey("some:key")))
csharp_hex: "85AD736368656D6156657273696F6EB063756C746E65742E6572726F722E7630A56572726F72D95C756E6F776E65645F736368656D613A204E6F207368617264206F776E7320736368656D612027736F6D652E736368656D612E763127206B65792027736F6D653A6B6579273B207468652077726974652077617320726566757365642EAB726F7574696E6748696E74C0A4636F6465AE756E6F776E65645F736368656D61A764657461696C7384A56669656C64A8736368656D614964A576616C7565AE736F6D652E736368656D612E7631A461734F66C0A763757272656E74C0",
message: || CultNetMessage::Error {
error: "unowned_schema: No shard owns schema 'some.schema.v1' key 'some:key'; the write was refused.".to_string(),
code: Some(CultNetErrorCode::UnownedSchema),
details: Some(CultNetErrorDetails {
field: Some("schemaId".to_string()),
value: Some("some.schema.v1".to_string()),
as_of: None,
current: None,
}),
},
},
]
}

3
packages/cultnet-rs/tests/impairment/burst-4.toml Normal file
View file

@ -0,0 +1,3 @@
burst_every = 200
burst_length = 4
max_scheduled = 4096

3
packages/cultnet-rs/tests/impairment/burst-8.toml Normal file
View file

@ -0,0 +1,3 @@
burst_every = 200
burst_length = 8
max_scheduled = 4096

2
packages/cultnet-rs/tests/impairment/clean.toml Normal file
View file

@ -0,0 +1,2 @@
# No impairment. The proxy itself remains in the measured path.
max_scheduled = 4096

3
packages/cultnet-rs/tests/impairment/delay-jitter.toml Normal file
View file

@ -0,0 +1,3 @@
delay_ms = 5
jitter_ms = 10
max_scheduled = 4096

2
packages/cultnet-rs/tests/impairment/duplicate.toml Normal file
View file

@ -0,0 +1,2 @@
duplicate_every = 50
max_scheduled = 4096

2
packages/cultnet-rs/tests/impairment/loss-1pct.toml Normal file
View file

@ -0,0 +1,2 @@
loss_basis_points = 100
max_scheduled = 4096

2
packages/cultnet-rs/tests/impairment/loss-3pct.toml Normal file
View file

@ -0,0 +1,2 @@
loss_basis_points = 300
max_scheduled = 4096

2
packages/cultnet-rs/tests/impairment/loss-5pct.toml Normal file
View file

@ -0,0 +1,2 @@
loss_basis_points = 500
max_scheduled = 4096

37
packages/cultnet-rs/tests/impairment/loss-matrix.txt Normal file
View file

@ -0,0 +1,37 @@
profile chunks frames whole/no-fec whole/fec recovered beyond
clean 8 200 200 200 0 0
clean 16 200 200 200 0 0
clean 17 200 200 200 0 0
clean 24 200 200 200 0 0
clean 32 200 200 200 0 0
clean 48 200 200 200 0 0
loss-1pct 8 200 183 200 17 0
loss-1pct 16 200 170 200 30 0
loss-1pct 17 200 170 200 31 0
loss-1pct 24 200 154 200 48 0
loss-1pct 32 200 144 200 59 0
loss-1pct 48 200 137 200 75 0
loss-3pct 8 200 158 199 41 1
loss-3pct 16 200 126 200 74 0
loss-3pct 17 200 119 199 90 0
loss-3pct 24 200 101 200 116 0
loss-3pct 32 200 78 200 147 0
loss-3pct 48 200 57 200 208 0
loss-5pct 8 200 130 197 67 3
loss-5pct 16 200 93 199 106 1
loss-5pct 17 200 92 198 132 1
loss-5pct 24 200 59 198 180 2
loss-5pct 32 200 41 200 214 0
loss-5pct 48 200 21 199 318 1
burst-4 8 200 181 191 10 9
burst-4 16 200 161 200 39 0
burst-4 17 200 175 200 46 0
burst-4 24 200 161 200 68 0
burst-4 32 200 121 200 118 0
burst-4 48 200 121 200 177 0
burst-8 8 200 181 191 10 0
burst-8 16 200 161 181 20 19
burst-8 17 200 171 183 20 30
burst-8 24 200 161 171 19 49
burst-8 32 200 121 200 118 0
burst-8 48 200 121 200 197 0

3
packages/cultnet-rs/tests/impairment/reorder.toml Normal file
View file

@ -0,0 +1,3 @@
reorder_every = 20
reorder_delay_ms = 12
max_scheduled = 4096

3
packages/cultnet-rs/tests/impairment/stall-250ms.toml Normal file
View file

@ -0,0 +1,3 @@
stall_at_ms = 10000
stall_for_ms = 250
max_scheduled = 4096

219
packages/cultnet-rs/tests/impairment_policy.rs Normal file
View file

@ -0,0 +1,219 @@
//! The impairment policy is deterministic, or a loss experiment cannot be
//! replayed. These pin each decision the proxy example makes, against the same
//! `policy.rs` the example compiles.
#[allow(dead_code)]
#[path = "../examples/cultnet_impair/policy.rs"]
mod policy;
use std::net::SocketAddr;
use policy::{Direction, Profile, Scheduler};
fn client() -> SocketAddr {
"127.0.0.1:1234".parse().unwrap()
}
#[test]
fn iid_decisions_are_seed_reproducible() {
let profile = Profile {
loss_basis_points: 2500,
..Default::default()
};
let mut a = Scheduler::new(profile.clone(), 42);
let mut b = Scheduler::new(profile, 42);
let left: Vec<_> = (0..100).map(|_| a.decide(0)).collect();
let right: Vec<_> = (0..100).map(|_| b.decide(0)).collect();
assert_eq!(left, right);
assert!(left.iter().any(|d| d.drop));
assert!(left.iter().any(|d| !d.drop));
}
#[test]
fn burst_drop_is_exact() {
let profile = Profile {
burst_every: 5,
burst_length: 3,
..Default::default()
};
let mut scheduler = Scheduler::new(profile, 1);
let drops: Vec<_> = (1..=9).filter(|_| scheduler.decide(0).drop).collect();
assert_eq!(drops, vec![5, 6, 7], "a burst starts on the Nth datagram and runs burst_length");
}
#[test]
fn duplicate_delay_and_reorder_are_deterministic() {
let profile = Profile {
duplicate_every: 2,
reorder_every: 3,
reorder_delay_ms: 20,
delay_ms: 5,
..Default::default()
};
let mut scheduler = Scheduler::new(profile, 1);
assert_eq!(scheduler.decide(0).copies, 1);
assert_eq!(scheduler.decide(0).copies, 2);
let third = scheduler.decide(0);
assert!(third.reordered);
assert_eq!(third.delay_ms, 25);
}
#[test]
fn stall_drops_only_inside_window() {
let profile = Profile {
stall_at_ms: 100,
stall_for_ms: 50,
..Default::default()
};
let mut scheduler = Scheduler::new(profile, 1);
assert!(!scheduler.decide(99).drop);
let stalled = scheduler.decide(100);
assert!(stalled.drop && stalled.stalled);
assert!(scheduler.decide(149).drop);
assert!(!scheduler.decide(150).drop);
}
#[test]
fn reorder_delay_places_packet_behind_later_packet() {
let profile = Profile {
reorder_every: 2,
reorder_delay_ms: 20,
..Default::default()
};
let mut scheduler = Scheduler::new(profile, 1);
scheduler.admit(0, Direction::ClientToUpstream, b"first", client());
scheduler.admit(0, Direction::ClientToUpstream, b"held", client());
scheduler.admit(1, Direction::ClientToUpstream, b"later", client());
assert_eq!(scheduler.pop_due(1).unwrap().bytes, b"first");
assert_eq!(scheduler.pop_due(1).unwrap().bytes, b"later");
assert!(scheduler.pop_due(19).is_none());
assert_eq!(scheduler.pop_due(20).unwrap().bytes, b"held");
}
#[test]
fn scheduled_queue_is_bounded_and_counts_overflow() {
let profile = Profile {
delay_ms: 100,
max_scheduled: 2,
..Default::default()
};
let mut scheduler = Scheduler::new(profile, 1);
for value in 0..5 {
scheduler.admit(0, Direction::ClientToUpstream, &[value], client());
}
assert_eq!(scheduler.queue.len(), 2);
assert_eq!(scheduler.stats.max_queue, 2);
assert_eq!(scheduler.stats.queue_overflow, 3);
assert_eq!(scheduler.stats.dropped, 3);
}
fn profile_path(name: &str) -> std::path::PathBuf {
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/impairment")
.join(format!("{name}.toml"))
}
#[test]
fn every_committed_profile_loads_with_its_named_meaning() {
let load = |name: &str| Profile::load(&profile_path(name)).unwrap_or_else(|e| panic!("{name}: {e}"));
assert_eq!(load("clean"), Profile::default());
assert_eq!(load("loss-1pct").loss_basis_points, 100);
assert_eq!(load("loss-3pct").loss_basis_points, 300);
assert_eq!(load("loss-5pct").loss_basis_points, 500);
let burst4 = load("burst-4");
assert_eq!((burst4.burst_every, burst4.burst_length), (200, 4));
let burst8 = load("burst-8");
assert_eq!((burst8.burst_every, burst8.burst_length), (200, 8));
let jitter = load("delay-jitter");
assert_eq!((jitter.delay_ms, jitter.jitter_ms), (5, 10));
assert_eq!(load("duplicate").duplicate_every, 50);
let reorder = load("reorder");
assert_eq!((reorder.reorder_every, reorder.reorder_delay_ms), (20, 12));
let stall = load("stall-250ms");
assert_eq!((stall.stall_at_ms, stall.stall_for_ms), (10_000, 250));
}
#[test]
fn a_malformed_profile_is_refused_with_its_line() {
let dir = tempfile::tempdir().unwrap();
let write = |name: &str, text: &str| {
let path = dir.path().join(name);
std::fs::write(&path, text).unwrap();
Profile::load(&path).unwrap_err()
};
assert!(write("unknown.toml", "jitter = 3
").contains("unknown profile key jitter"));
assert!(write("word.toml", "delay_ms = soon
").contains("unsigned integer"));
assert!(write("shape.toml", "delay_ms
").contains("key = value"));
assert!(write("loss.toml", "loss_basis_points = 10001
").contains("<= 10000"));
assert!(write("queue.toml", "max_scheduled = 0
").contains("greater than zero"));
assert!(write("line.toml", "
delay_ms = x
").contains(":3:"));
}
#[test]
fn admission_counts_what_it_did_to_each_datagram() {
let mut scheduler = Scheduler::new(
Profile { duplicate_every: 2, reorder_every: 3, reorder_delay_ms: 10, ..Default::default() },
1,
);
for value in 0..6 {
scheduler.admit(0, Direction::ClientToUpstream, &[value], client());
}
// Datagrams 2, 4 and 6 are duplicated, 3 and 6 are reordered (6 is both).
assert_eq!(scheduler.stats.received, 6);
assert_eq!(scheduler.stats.duplicated, 3);
assert_eq!(scheduler.stats.reordered, 2);
assert_eq!(scheduler.queue.len(), 9);
let mut stalled = Scheduler::new(Profile { stall_at_ms: 0, stall_for_ms: 10, ..Default::default() }, 1);
stalled.admit(5, Direction::UpstreamToClient, b"x", client());
assert_eq!((stalled.stats.stalled, stalled.stats.dropped, stalled.queue.len()), (1, 1, 0));
}
#[test]
fn jitter_is_seeded_uniform_within_its_bound() {
let profile = Profile { delay_ms: 7, jitter_ms: 5, ..Default::default() };
let mut scheduler = Scheduler::new(profile, 9);
let delays: Vec<u64> = (0..300).map(|_| scheduler.decide(0).delay_ms).collect();
assert!(delays.iter().all(|delay| (7..=12).contains(delay)), "{delays:?}");
for delay in 7..=12 {
assert!(delays.contains(&delay), "delay {delay} never drawn");
}
}
#[test]
fn the_loss_ceiling_is_inclusive() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("all.toml");
std::fs::write(&path, "loss_basis_points = 10000\n").unwrap();
let profile = Profile::load(&path).unwrap();
let mut scheduler = Scheduler::new(profile, 1);
assert!((0..50).all(|_| scheduler.decide(0).drop), "10000 basis points drops everything");
}
/// The seeded stream is part of the contract: the same seed must drop the same
/// datagrams on every run, on every host. 100,000 draws of xorshift64 from seed
/// 42, taken modulo 10,000 and compared against 100, give 1,008.
#[test]
fn seeded_loss_drops_the_exact_datagrams_the_stream_dictates() {
let mut scheduler = Scheduler::new(Profile { loss_basis_points: 100, ..Default::default() }, 42);
let dropped = (0..100_000).filter(|_| scheduler.decide(0).drop).count();
assert_eq!(dropped, 1_008);
}
#[test]
fn datagrams_due_together_leave_in_arrival_order() {
let mut scheduler = Scheduler::new(Profile { delay_ms: 5, ..Default::default() }, 1);
for value in 0..4_u8 {
scheduler.admit(0, Direction::ClientToUpstream, &[value], client());
}
let order: Vec<u8> = std::iter::from_fn(|| scheduler.pop_due(5)).map(|item| item.bytes[0]).collect();
assert_eq!(order, vec![0, 1, 2, 3]);
}

118
packages/cultnet-rs/tests/media_channel_delivery.rs
View file

@ -90,3 +90,121 @@ fn the_other_channels_keep_their_fixed_delivery() {
assert_eq!(delivery_of("schema"), CultNetTransportDelivery::Reliable);
assert_eq!(delivery_of("realtime"), CultNetTransportDelivery::Unreliable);
}
// ---------------------------------------------------------------------------
// What the media channel puts on the wire
// ---------------------------------------------------------------------------
use std::thread::sleep;
use std::time::Duration;
use cultnet_rs::{
CultNetRudpPacket, CultNetRudpSession, CultNetRudpSessionOptions, decode_rudp_packet,
encode_rudp_packet,
};
/// A connected client transport and the raw socket standing in for its peer.
struct Link {
client: CultNetRudpSocketTransportConnection,
peer: UdpSocket,
}
impl Link {
fn open(delivery: Option<CultNetTransportDelivery>, expire_after_ms: Option<u64>) -> Self {
let peer = UdpSocket::bind("127.0.0.1:0").expect("binds");
peer.set_read_timeout(Some(Duration::from_millis(500))).unwrap();
let socket = UdpSocket::bind("127.0.0.1:0").expect("binds");
socket.set_read_timeout(Some(Duration::from_millis(300))).unwrap();
let mut options = CultNetRudpSocketTransportOptions::client(
"test-media",
socket,
peer.local_addr().unwrap(),
7,
);
options.media_delivery = delivery;
options.media_reliable_expire_after_ms = expire_after_ms;
options.resend_delay_ms = 10;
let mut client = CultNetRudpSocketTransportConnection::new(options).expect("opens");
client.connect(Vec::new()).expect("sends connect");
let mut wire = vec![0_u8; 2_048];
let (received, client_addr) = peer.recv_from(&mut wire).expect("the connect arrives");
let connect = decode_rudp_packet(&wire[..received]).expect("decodes");
let mut server = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 7,
..CultNetRudpSessionOptions::default()
});
let accept = server.accept_connect(&connect, 0, Vec::new()).expect("accepts");
peer.send_to(&encode_rudp_packet(&accept).unwrap(), client_addr).unwrap();
client.receive_once().expect("takes the accept");
assert!(client.connected());
let mut link = Self { client, peer };
link.datagrams(Duration::from_millis(100));
link
}
/// Everything the peer receives within `window`.
fn datagrams(&mut self, window: Duration) -> Vec<CultNetRudpPacket> {
self.peer.set_read_timeout(Some(window)).unwrap();
let mut packets = Vec::new();
let mut wire = vec![0_u8; 2_048];
while let Ok((received, _)) = self.peer.recv_from(&mut wire) {
packets.push(decode_rudp_packet(&wire[..received]).expect("decodes"));
}
packets
}
fn send_media(&mut self, window: Duration) -> Vec<CultNetRudpPacket> {
self.client.send("media", vec![1, 2, 3]).expect("sends");
self.datagrams(window)
}
}
fn media_only(packets: Vec<CultNetRudpPacket>) -> Vec<CultNetRudpPacket> {
packets.into_iter().filter(|packet| packet.channel_id == "media").collect()
}
/// The channel the profile describes as reliable is sent reliable, unordered
/// and unsequenced; asking for an unreliable lane sends it unreliable.
#[test]
fn the_media_channel_is_sent_with_the_delivery_the_profile_advertises() {
let mut reliable = Link::open(None, None);
let sent = media_only(reliable.send_media(Duration::from_millis(100)));
assert_eq!(sent.len(), 1);
assert!(sent[0].reliable, "the default media lane is reliable");
assert!(!sent[0].ordered && !sent[0].sequenced, "media is neither ordered nor sequenced");
let mut unreliable = Link::open(Some(CultNetTransportDelivery::Unreliable), None);
let sent = media_only(unreliable.send_media(Duration::from_millis(100)));
assert_eq!(sent.len(), 1);
assert!(!sent[0].reliable, "an unreliable media lane is not retransmitted");
assert!(!sent[0].ordered && !sent[0].sequenced);
}
/// A reliable media packet with an expiry is dropped once it has passed, and
/// one without is retransmitted until acknowledged. The session reclaims
/// expired packets on the next send, so the probe sends a second packet.
fn first_packet_is_resent_after_a_later_send(expire_after_ms: Option<u64>) -> bool {
let mut link = Link::open(None, expire_after_ms);
let first = media_only(link.send_media(Duration::from_millis(50)));
assert_eq!(first.len(), 1);
sleep(Duration::from_millis(150));
assert_eq!(media_only(link.send_media(Duration::from_millis(50))).len(), 1);
sleep(Duration::from_millis(15));
link.client.poll_resends().unwrap();
media_only(link.datagrams(Duration::from_millis(100)))
.iter()
.any(|packet| packet.sequence == first[0].sequence)
}
#[test]
fn the_media_channels_expiry_is_applied_to_its_packets() {
assert!(
!first_packet_is_resent_after_a_later_send(Some(20)),
"an expired media packet is dropped, not retransmitted"
);
assert!(
first_packet_is_resent_after_a_later_send(None),
"without an expiry the unacknowledged packet is retransmitted"
);
}

1143
packages/cultnet-rs/tests/media_fec.rs Normal file
View file

File diff suppressed because it is too large Load diff

208
packages/cultnet-rs/tests/media_fec_loss.rs Normal file
View file

@ -0,0 +1,208 @@
//! Video frames protected by the media FEC codec, sent through the impairment
//! schedules `cultnet_impair` applies to real sockets, with no sockets: each
//! record of a frame in send order takes the next datagram decision, and what
//! survives is handed to `recover_video_block`.
//!
//! Two kinds of claim. The guarantee is asserted: a block recovers exactly when
//! at most `m` of its records were lost, and a burst short enough that the send
//! order spreads it under `m` per block always leaves every frame whole. The
//! recovered-frame ratios are evidence, not assertions; `--nocapture` prints the
//! table and `tests/impairment/loss-matrix.txt` is that output (200 frames per cell, seed 20260930).
#[allow(dead_code)]
#[path = "../examples/cultnet_impair/policy.rs"]
mod policy;
use std::fmt::Write as _;
use std::path::Path;
use cultnet_rs::{
GameCultMediaVideoAccessUnitRecord, GameCultMediaWireRecord, MediaFecError, MediaFecPolicy,
MediaWireProvenance, protect_video_frame, recover_video_block,
};
use policy::{Profile, Scheduler};
const POLICY: MediaFecPolicy = MediaFecPolicy::STANDARD;
const SEED: u64 = 20_260_930;
const FRAMES: u64 = 200;
fn provenance() -> MediaWireProvenance<'static> {
MediaWireProvenance {
stored_at: "unix:1700000000000",
runtime_id: "raven-muninn-primary",
role: "muninn.media",
producer: "muninn",
}
}
fn frame(frame_id: u64, chunk_count: u16) -> Vec<GameCultMediaVideoAccessUnitRecord> {
(0..chunk_count)
.map(|index| GameCultMediaVideoAccessUnitRecord {
stream_id: "muninn.raven.av.rudp".to_string(),
session_id: "session-1".to_string(),
frame_id,
codec: "h264".to_string(),
pts_ticks: frame_id as i64 * 3_000,
duration_ticks: 3_000,
timebase_num: 1,
timebase_den: 90_000,
keyframe: false,
dependency_frame_id: frame_id.checked_sub(1),
deadline_ticks: frame_id as i64 * 3_000 + 22_500,
chunk_index: index,
chunk_count,
payload: (0..if index + 1 == chunk_count { 60 } else { 100 })
.map(|byte| (frame_id as usize * 31 + usize::from(index) * 7 + byte) as u8)
.collect(),
})
.collect()
}
#[derive(Default)]
struct Tally {
frames: u64,
whole_without_parity: u64,
whole_with_parity: u64,
blocks_recovered: u64,
blocks_beyond_repair: u64,
}
/// Runs one shape through one profile's schedule and asserts the per-block
/// guarantee on every block of every frame.
fn run(profile: &Profile, chunk_count: u16) -> Tally {
let mut scheduler = Scheduler::new(profile.clone(), SEED);
let sizes = POLICY.video_block_sizes(chunk_count);
let mut tally = Tally::default();
for frame_id in 0..FRAMES {
let original = frame(frame_id, chunk_count);
let records = protect_video_frame(&original, &POLICY, provenance()).unwrap();
let survivors: Vec<GameCultMediaWireRecord> = records
.into_iter()
.filter(|_| !scheduler.decide(0).drop)
.collect();
let mut whole = true;
let mut whole_without_parity = true;
let mut start = 0_u16;
for (block, &size) in sizes.iter().enumerate() {
let m = usize::from(POLICY.video_parity_shards(size));
let mut data = Vec::new();
let mut parity = Vec::new();
for record in &survivors {
match record {
GameCultMediaWireRecord::Video(chunk)
if chunk.chunk_index >= start && chunk.chunk_index < start + size =>
{
data.push(chunk.clone())
}
GameCultMediaWireRecord::VideoParity(shard) if usize::from(shard.block_index) == block => {
parity.push(shard.clone())
}
_ => {}
}
}
let lost = usize::from(size) + m - data.len() - parity.len();
let missing_data = usize::from(size) - data.len();
whole_without_parity &= missing_data == 0;
let block_whole = if missing_data == 0 {
true
} else if parity.is_empty() {
assert!(lost > m, "no parity survived, so more than m shards were lost");
false
} else {
match recover_video_block(&parity, &data) {
Ok(recovered) => {
assert!(lost <= m, "block {block} of frame {frame_id} recovered {lost} losses with m = {m}");
assert_eq!(recovered.len(), missing_data);
for chunk in &recovered {
assert_eq!(
chunk, &original[usize::from(chunk.chunk_index)],
"a recovered chunk must be the chunk that was sent"
);
}
tally.blocks_recovered += 1;
true
}
Err(MediaFecError::BeyondRepair { .. }) => {
assert!(lost > m, "block {block} of frame {frame_id} refused a recoverable {lost} losses with m = {m}");
tally.blocks_beyond_repair += 1;
false
}
Err(other) => panic!("{other}"),
}
};
whole &= block_whole;
start += size;
}
tally.frames += 1;
tally.whole_with_parity += u64::from(whole);
tally.whole_without_parity += u64::from(whole_without_parity);
assert!(whole || !whole_without_parity, "parity never makes a whole frame worse");
}
tally
}
fn profile(name: &str) -> Profile {
Profile::load(&Path::new(env!("CARGO_MANIFEST_DIR")).join(format!("tests/impairment/{name}.toml")))
.unwrap()
}
/// Every shape here splits into equal blocks, so a burst of `L` consecutive
/// datagrams costs each of `B` blocks at most `ceil(L / B)` of its records.
const EQUAL_BLOCK_SHAPES: [u16; 5] = [8, 16, 24, 32, 48];
#[test]
fn the_loss_matrix() {
let cells = ["clean", "loss-1pct", "loss-3pct", "loss-5pct", "burst-4", "burst-8"];
let shapes = [8_u16, 16, 17, 24, 32, 48];
let mut table = String::new();
writeln!(
table,
"{:<10} {:>6} {:>7} {:>13} {:>12} {:>10} {:>9}",
"profile", "chunks", "frames", "whole/no-fec", "whole/fec", "recovered", "beyond"
)
.unwrap();
for cell in cells {
let loaded = profile(cell);
for &chunks in &shapes {
let tally = run(&loaded, chunks);
writeln!(
table,
"{:<10} {:>6} {:>7} {:>13} {:>12} {:>10} {:>9}",
cell,
chunks,
tally.frames,
tally.whole_without_parity,
tally.whole_with_parity,
tally.blocks_recovered,
tally.blocks_beyond_repair
)
.unwrap();
if cell == "clean" {
assert_eq!(tally.whole_with_parity, tally.frames, "a clean link loses nothing");
}
let burst = match cell {
"burst-4" => 4_usize,
"burst-8" => 8,
_ => continue,
};
if EQUAL_BLOCK_SHAPES.contains(&chunks) {
let sizes = POLICY.video_block_sizes(chunks);
let per_block = burst.div_ceil(sizes.len());
let m = usize::from(POLICY.video_parity_shards(sizes[0]));
if per_block <= m {
assert_eq!(
tally.whole_with_parity, tally.frames,
"{cell} over {chunks} chunks costs each of {} blocks at most {per_block} of m = {m}: no frame may be lost",
sizes.len()
);
}
}
}
}
println!("{table}");
}

73
packages/cultnet-rs/tests/media_stream_contracts.rs
View file

@ -5,9 +5,10 @@
use cultcache_rs::DatabaseEntry;
use cultnet_rs::{
GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA, GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA,
GAMECULT_MEDIA_VIDEO_ACCESS_UNIT_SCHEMA, GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA,
GameCultMediaAudioPacketRecord, GameCultMediaReceiverFeedbackRecord,
GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA, GAMECULT_MEDIA_AUDIO_PARITY_SHARD_SCHEMA,
GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA, GAMECULT_MEDIA_VIDEO_ACCESS_UNIT_SCHEMA,
GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA, GameCultMediaAudioPacketRecord,
GameCultMediaAudioParityShardRecord, GameCultMediaReceiverFeedbackRecord,
GameCultMediaVideoAccessUnitRecord, GameCultMediaVideoParityShardRecord,
};
use rmpv::Value;
@ -129,9 +130,14 @@ fn parity_shards_carry_the_geometry_recovery_needs() {
dependency_frame_id: Some(4241),
deadline_ticks: 108_000,
chunk_count: 5,
fec_scheme: "rs-gf256-v1".to_string(),
block_index: 0,
block_count: 1,
block_data_start: 0,
block_data_count: 5,
parity_index: 1,
parity_count: 2,
chunk_payload_bytes: 1024,
shard_payload_bytes: 1024,
last_chunk_payload_bytes: 512,
payload: vec![9, 9, 9],
};
@ -141,13 +147,21 @@ fn parity_shards_carry_the_geometry_recovery_needs() {
panic!("parity shards must encode as a positional array");
};
assert_eq!(slots.len(), 17, "field count is wire contract");
assert_eq!(slots[12].as_u64(), Some(1), "slot 12 is parity_index");
assert_eq!(slots[13].as_u64(), Some(2), "slot 13 is parity_count");
assert_eq!(slots.len(), 22, "field count is wire contract");
assert_eq!(slots[12].as_str(), Some("rs-gf256-v1"), "slot 12 is fec_scheme");
assert_eq!(slots[13].as_u64(), Some(0), "slot 13 is block_index");
assert_eq!(slots[16].as_u64(), Some(5), "slot 16 is block_data_count (k)");
assert_eq!(slots[17].as_u64(), Some(1), "slot 17 is parity_index");
assert_eq!(slots[18].as_u64(), Some(2), "slot 18 is parity_count (m)");
assert_eq!(
slots[15].as_u64(),
slots[20].as_u64(),
Some(512),
"slot 15 is last_chunk_payload_bytes, which recovery needs to size the tail"
"slot 20 is last_chunk_payload_bytes, which recovery needs to size the tail"
);
assert_eq!(
slots[21].as_slice(),
Some([9, 9, 9].as_slice()),
"the parity payload is bin, not an array of integers"
);
let decoded: GameCultMediaVideoParityShardRecord =
@ -155,6 +169,42 @@ fn parity_shards_carry_the_geometry_recovery_needs() {
assert_eq!(decoded, shard);
}
#[test]
fn audio_parity_shards_carry_the_geometry_recovery_needs() {
let shard = GameCultMediaAudioParityShardRecord {
stream_id: "raven-primary-av".to_string(),
session_id: "session-1".to_string(),
codec: "opus".to_string(),
fec_scheme: "rs-gf256-v1".to_string(),
base_packet_id: 77,
base_pts_ticks: 960,
packet_duration_ticks: 960,
timebase_num: 1,
timebase_den: 48_000,
deadline_ticks: 4_800,
data_shard_count: 4,
parity_index: 1,
parity_shard_count: 2,
shard_payload_bytes: 3,
payload: vec![7, 8, 9],
};
let encoded = rmp_serde::to_vec(&shard).expect("encodes");
let Value::Array(slots) = decode_as_value(&encoded) else {
panic!("audio parity shards must encode as a positional array");
};
assert_eq!(slots.len(), 15, "field count is wire contract");
assert_eq!(slots[4].as_u64(), Some(77), "slot 4 is base_packet_id");
assert_eq!(slots[10].as_u64(), Some(4), "slot 10 is data_shard_count (k)");
assert_eq!(slots[12].as_u64(), Some(2), "slot 12 is parity_shard_count (m)");
assert_eq!(slots[14].as_slice(), Some([7, 8, 9].as_slice()), "the payload is bin");
let decoded: GameCultMediaAudioParityShardRecord =
rmp_serde::from_slice(&encoded).expect("round-trips");
assert_eq!(decoded, shard);
}
/// `missing_video_chunk_keys` was appended after producers were already writing
/// feedback. A record written without it must still decode, or the append was a
/// breaking change wearing a `default` attribute.
@ -205,6 +255,11 @@ fn published_schema_constants_match_the_derived_identity() {
GameCultMediaAudioPacketRecord::SCHEMA_NAME,
GAMECULT_MEDIA_AUDIO_PACKET_SCHEMA
);
assert_eq!(
GameCultMediaAudioParityShardRecord::SCHEMA_NAME,
GAMECULT_MEDIA_AUDIO_PARITY_SHARD_SCHEMA
);
assert_eq!(GAMECULT_MEDIA_VIDEO_PARITY_SHARD_SCHEMA, "gamecult.media_video_parity_shard.v3");
assert_eq!(
GameCultMediaReceiverFeedbackRecord::SCHEMA_NAME,
GAMECULT_MEDIA_RECEIVER_FEEDBACK_SCHEMA

463
packages/cultnet-rs/tests/media_stream_wire.rs
View file

@ -2,7 +2,10 @@
//! nothing on the wire.
use cultnet_rs::{
GAMECULT_MEDIA_CHANNEL, GameCultMediaAudioPacketRecord, GameCultMediaReceiverFeedbackRecord,
CultNetRudpDeliveredFrame, CultNetRudpSendOptions, CultNetRudpSession, CultNetRudpSessionOptions,
GAMECULT_MEDIA_MAX_WIRE_BYTES, RudpTransportProfileOptions, create_rudp_transport_profile,
GAMECULT_MEDIA_CHANNEL, GameCultMediaAudioPacketRecord, GameCultMediaAudioParityShardRecord,
GameCultMediaReceiverFeedbackRecord,
GameCultMediaVideoAccessUnitRecord, GameCultMediaVideoParityShardRecord,
GameCultMediaWireRecord, MediaWireProvenance, decode_media_wire_record,
encode_media_wire_record,
@ -116,12 +119,34 @@ fn every_variant_round_trips() {
dependency_frame_id: Some(0),
deadline_ticks: 1,
chunk_count: 4,
fec_scheme: "rs-gf256-v1".to_string(),
block_index: 0,
block_count: 1,
block_data_start: 0,
block_data_count: 4,
parity_index: 1,
parity_count: 2,
chunk_payload_bytes: 848,
last_chunk_payload_bytes: 400,
shard_payload_bytes: 3,
last_chunk_payload_bytes: 2,
payload: vec![1, 2, 3],
}),
GameCultMediaWireRecord::AudioParity(GameCultMediaAudioParityShardRecord {
stream_id: "s".to_string(),
session_id: "x".to_string(),
codec: "opus".to_string(),
fec_scheme: "rs-gf256-v1".to_string(),
base_packet_id: 8,
base_pts_ticks: 0,
packet_duration_ticks: 960,
timebase_num: 1,
timebase_den: 48_000,
deadline_ticks: 4_000,
data_shard_count: 4,
parity_index: 0,
parity_shard_count: 2,
shard_payload_bytes: 3,
payload: vec![4, 5, 6],
}),
GameCultMediaWireRecord::Audio(GameCultMediaAudioPacketRecord {
stream_id: "s".to_string(),
session_id: "x".to_string(),
@ -215,3 +240,435 @@ fn provenance_must_be_complete() {
fn the_media_channel_is_named_once() {
assert_eq!(GAMECULT_MEDIA_CHANNEL, "media");
}
fn video_parity() -> GameCultMediaVideoParityShardRecord {
GameCultMediaVideoParityShardRecord {
stream_id: "s".to_string(),
session_id: "x".to_string(),
frame_id: 7,
codec: "h264".to_string(),
pts_ticks: 0,
duration_ticks: 1,
timebase_num: 1,
timebase_den: 90_000,
keyframe: false,
dependency_frame_id: None,
deadline_ticks: 1,
chunk_count: 20,
fec_scheme: "rs-gf256-v1".to_string(),
block_index: 1,
block_count: 2,
block_data_start: 10,
block_data_count: 10,
parity_index: 1,
parity_count: 3,
shard_payload_bytes: 4,
last_chunk_payload_bytes: 2,
payload: vec![1, 2, 3, 4],
}
}
fn audio_parity() -> GameCultMediaAudioParityShardRecord {
GameCultMediaAudioParityShardRecord {
stream_id: "s".to_string(),
session_id: "x".to_string(),
codec: "opus".to_string(),
fec_scheme: "rs-gf256-v1".to_string(),
base_packet_id: 8,
base_pts_ticks: 0,
packet_duration_ticks: 960,
timebase_num: 1,
timebase_den: 48_000,
deadline_ticks: 4_000,
data_shard_count: 4,
parity_index: 1,
parity_shard_count: 2,
shard_payload_bytes: 3,
payload: vec![4, 5, 6],
}
}
fn decodes(record: GameCultMediaWireRecord) -> Result<GameCultMediaWireRecord, String> {
let wire = encode_media_wire_record(&record, provenance()).map_err(|error| error.to_string())?;
decode_media_wire_record(&wire).map_err(|error| error.to_string())
}
/// A parity key names the block as well as the shard, or two blocks' parity
/// index 0 would share an address.
#[test]
fn parity_record_keys_name_the_block_and_the_shard() {
let mut other_block = video_parity();
other_block.block_index = 0;
let mut other_shard = video_parity();
other_shard.parity_index = 0;
let keys = [
GameCultMediaWireRecord::VideoParity(video_parity()).record_key(),
GameCultMediaWireRecord::VideoParity(other_block).record_key(),
GameCultMediaWireRecord::VideoParity(other_shard).record_key(),
];
assert_eq!(keys[0], "s:x:video-parity:7:1:1");
assert_ne!(keys[0], keys[1]);
assert_ne!(keys[0], keys[2]);
let audio = GameCultMediaWireRecord::AudioParity(audio_parity()).record_key();
assert_eq!(audio, "s:x:audio-parity:8:1");
let mut other = audio_parity();
other.parity_index = 0;
assert_ne!(audio, GameCultMediaWireRecord::AudioParity(other).record_key());
}
#[test]
fn a_well_formed_parity_record_is_admitted() {
assert!(decodes(GameCultMediaWireRecord::VideoParity(video_parity())).is_ok());
assert!(decodes(GameCultMediaWireRecord::AudioParity(audio_parity())).is_ok());
}
/// The decoder is where a consumer's idea of a well-formed shard is enforced.
/// Each corruption below would otherwise reach the recovery arithmetic.
#[test]
fn a_malformed_video_parity_record_is_refused_with_its_reason() {
let cases: Vec<(&str, Box<dyn Fn(&mut GameCultMediaVideoParityShardRecord)>)> = vec![
("fec_scheme", Box::new(|r| r.fec_scheme = "xor-v0".to_string())),
("block_index", Box::new(|r| r.block_index = 2)),
("block_index", Box::new(|r| r.block_count = 0)),
("inside the frame", Box::new(|r| r.block_data_start = 11)),
("inside the frame", Box::new(|r| r.block_data_count = 0)),
("parity_index", Box::new(|r| r.parity_index = 3)),
("parity_index", Box::new(|r| r.parity_count = 0)),
("256 shards", Box::new(|r| {
r.block_data_count = 254;
r.chunk_count = 300;
r.parity_count = 3;
})),
("shard_payload_bytes", Box::new(|r| {
r.shard_payload_bytes = 0;
r.payload.clear();
})),
("last chunk length", Box::new(|r| r.last_chunk_payload_bytes = 0)),
("last chunk length", Box::new(|r| r.last_chunk_payload_bytes = 5)),
("declared shard length", Box::new(|r| r.payload.push(0))),
("chunk_count", Box::new(|r| r.chunk_count = 0)),
("256 shards", Box::new(|r| {
r.block_data_count = 256;
r.chunk_count = 300;
r.parity_count = 1;
r.parity_index = 0;
})),
("deadline_ticks", Box::new(|r| r.deadline_ticks = -1)),
];
for (expected, corrupt) in cases {
let mut record = video_parity();
corrupt(&mut record);
let error = decodes(GameCultMediaWireRecord::VideoParity(record)).unwrap_err();
assert!(error.contains(expected), "expected {expected:?} in {error:?}");
}
}
#[test]
fn a_malformed_audio_parity_record_is_refused_with_its_reason() {
let cases: Vec<(&str, Box<dyn Fn(&mut GameCultMediaAudioParityShardRecord)>)> = vec![
("fec_scheme", Box::new(|r| r.fec_scheme = String::new())),
("stripe", Box::new(|r| r.parity_index = 2)),
("stripe", Box::new(|r| r.data_shard_count = 0)),
("stripe", Box::new(|r| r.data_shard_count = 255)),
("timing", Box::new(|r| r.packet_duration_ticks = 0)),
("last pts_ticks", Box::new(|r| r.deadline_ticks = -1)),
("packet id or pts range", Box::new(|r| r.base_packet_id = u64::MAX - 2)),
("packet id or pts range", Box::new(|r| {
r.base_pts_ticks = i64::MAX - 960 * 3 + 1;
r.deadline_ticks = i64::MAX;
})),
("declared shard length", Box::new(|r| { r.payload.pop(); })),
("codec", Box::new(|r| r.codec = String::new())),
("stream_id", Box::new(|r| r.stream_id = String::new())),
("stream_id", Box::new(|r| r.session_id = String::new())),
("timing", Box::new(|r| r.timebase_num = 0)),
("timing", Box::new(|r| r.timebase_den = 0)),
];
for (expected, corrupt) in cases {
let mut record = audio_parity();
corrupt(&mut record);
let error = decodes(GameCultMediaWireRecord::AudioParity(record)).unwrap_err();
assert!(error.contains(expected), "expected {expected:?} in {error:?}");
}
}
/// The 256-shard ceiling and the deadline floor are inclusive on the side that
/// keeps a valid record valid.
#[test]
fn records_on_the_edge_of_each_bound_are_admitted() {
let mut video = video_parity();
video.chunk_count = 300;
video.block_data_count = 254;
video.parity_count = 2;
video.parity_index = 1;
assert!(decodes(GameCultMediaWireRecord::VideoParity(video)).is_ok(), "254 + 2 = 256 shards");
let mut audio = audio_parity();
audio.data_shard_count = 254;
audio.deadline_ticks = i64::MAX;
audio.parity_shard_count = 2;
assert!(decodes(GameCultMediaWireRecord::AudioParity(audio)).is_ok(), "254 + 2 = 256 shards");
let mut audio = audio_parity();
audio.deadline_ticks = audio.base_pts_ticks + 3 * i64::from(audio.packet_duration_ticks);
assert!(decodes(GameCultMediaWireRecord::AudioParity(audio)).is_ok(), "a deadline at the last pts");
let mut audio = audio_parity();
audio.base_packet_id = u64::MAX - 3;
assert!(decodes(GameCultMediaWireRecord::AudioParity(audio)).is_ok(), "the last packet id is u64::MAX");
let mut audio = audio_parity();
audio.base_pts_ticks = i64::MAX - 960 * 3;
audio.deadline_ticks = i64::MAX;
assert!(decodes(GameCultMediaWireRecord::AudioParity(audio)).is_ok(), "the last pts is i64::MAX");
}
// ---------------------------------------------------------------------------
// The ceiling on one media record
// ---------------------------------------------------------------------------
fn sized_audio_parity(payload_bytes: usize, deadline_ticks: i64) -> GameCultMediaAudioParityShardRecord {
GameCultMediaAudioParityShardRecord {
stream_id: "s".to_string(),
session_id: "x".to_string(),
codec: "opus".to_string(),
fec_scheme: "rs-gf256-v1".to_string(),
base_packet_id: 8,
base_pts_ticks: 0,
packet_duration_ticks: 100,
timebase_num: 1,
timebase_den: 48_000,
deadline_ticks,
data_shard_count: 4,
parity_index: 0,
parity_shard_count: 2,
shard_payload_bytes: payload_bytes as u32,
payload: vec![7; payload_bytes],
}
}
fn sized_video_parity(payload_bytes: usize) -> GameCultMediaVideoParityShardRecord {
GameCultMediaVideoParityShardRecord {
stream_id: "s".to_string(),
session_id: "x".to_string(),
frame_id: 1,
codec: "h264".to_string(),
pts_ticks: 0,
duration_ticks: 1,
timebase_num: 1,
timebase_den: 90_000,
keyframe: false,
dependency_frame_id: Some(0),
deadline_ticks: 1,
chunk_count: 4,
fec_scheme: "rs-gf256-v1".to_string(),
block_index: 0,
block_count: 1,
block_data_start: 0,
block_data_count: 4,
parity_index: 1,
parity_count: 2,
shard_payload_bytes: payload_bytes as u32,
last_chunk_payload_bytes: 2,
payload: vec![7; payload_bytes],
}
}
fn wire_len(record: &GameCultMediaWireRecord) -> usize {
encode_media_wire_record(record, provenance()).unwrap().len()
}
/// The payload length whose wrapped record is exactly `wire_bytes` long, found
/// from the record's own linear growth in its payload.
fn payload_for_wire_len(
wire_bytes: usize,
wrap: &dyn Fn(usize) -> GameCultMediaWireRecord,
) -> usize {
let probe = 1_000;
let payload = probe + wire_bytes - wire_len(&wrap(probe));
assert_eq!(wire_len(&wrap(payload)), wire_bytes, "the record grows one byte per payload byte here");
payload
}
/// RUDP would reassemble a 16 MiB record, and recovery work scales with the
/// shard length, so the decoder holds every record to one ceiling.
#[test]
fn a_media_record_one_byte_over_the_ceiling_is_refused_at_decode() {
let ceiling = GAMECULT_MEDIA_MAX_WIRE_BYTES;
let wraps: [(&str, Box<dyn Fn(usize) -> GameCultMediaWireRecord>); 2] = [
(
"audio parity",
Box::new(|payload| GameCultMediaWireRecord::AudioParity(sized_audio_parity(payload, 300))),
),
(
"video parity",
Box::new(|payload| GameCultMediaWireRecord::VideoParity(sized_video_parity(payload))),
),
];
for (name, wrap) in &wraps {
let at = payload_for_wire_len(ceiling, wrap.as_ref());
let wire = encode_media_wire_record(&wrap(at), provenance()).unwrap();
assert_eq!(wire.len(), ceiling);
decode_media_wire_record(&wire)
.unwrap_or_else(|error| panic!("{name} at the ceiling is refused: {error}"));
// The encoder refuses to make an over-ceiling record, so one byte of
// padding stands in for a peer that sent one.
let mut over = wire.clone();
over.push(0);
let error = decode_media_wire_record(&over).unwrap_err();
assert!(error.to_string().contains("ceiling"), "{name}: {error}");
}
}
#[test]
fn a_jumbo_frame_sized_record_is_accepted() {
let wire = encode_media_wire_record(
&GameCultMediaWireRecord::AudioParity(sized_audio_parity(8_800, 300)),
provenance(),
)
.unwrap();
assert!(wire.len() > 8_800 && wire.len() < GAMECULT_MEDIA_MAX_WIRE_BYTES);
decode_media_wire_record(&wire).expect("a jumbo-frame record decodes");
}
/// Recovery hands every recovered packet the block's deadline, so a parity
/// whose deadline precedes the last packet's pts would recover a packet that
/// fails `validate_audio_record`.
#[test]
fn audio_parity_must_not_have_a_deadline_before_its_last_packets_pts() {
// Four packets, 100 ticks each from 0: the last plays at 300.
let decodes = |deadline_ticks: i64| {
let wire = encode_media_wire_record(
&GameCultMediaWireRecord::AudioParity(sized_audio_parity(8, deadline_ticks)),
provenance(),
)
.unwrap();
decode_media_wire_record(&wire)
};
assert!(decodes(300).is_ok());
assert!(decodes(301).is_ok());
let error = decodes(299).unwrap_err();
assert!(error.to_string().contains("last pts_ticks"), "{error}");
let error = decodes(0).unwrap_err();
assert!(error.to_string().contains("last pts_ticks"), "{error}");
}
// ---------------------------------------------------------------------------
// The media channel's own payload cap
// ---------------------------------------------------------------------------
fn connected_session() -> CultNetRudpSession {
let mut session = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: 9,
..CultNetRudpSessionOptions::default()
});
session.assume_connected(0);
session
}
const FRAGMENT_BYTES: usize = 16 * 1024;
fn media_payload_limit(session_limit: Option<u32>) -> Option<u32> {
let profile = create_rudp_transport_profile(
"raven",
RudpTransportProfileOptions {
max_payload_bytes: session_limit,
..RudpTransportProfileOptions::default()
},
);
profile.transports[0]
.channels
.iter()
.find(|channel| channel.channel_id == GAMECULT_MEDIA_CHANNEL)
.expect("the profile advertises the media channel")
.max_payload_bytes
}
#[test]
fn the_media_channel_advertises_the_ceiling_or_a_tighter_session_cap() {
let ceiling = GAMECULT_MEDIA_MAX_WIRE_BYTES as u32;
assert_eq!(media_payload_limit(None), Some(ceiling));
assert_eq!(media_payload_limit(Some(16 * 1024 * 1024)), Some(ceiling));
assert_eq!(media_payload_limit(Some(4_096)), Some(4_096));
}
#[test]
fn a_session_refuses_to_send_more_than_the_ceiling_on_the_media_channel_only() {
let ceiling = GAMECULT_MEDIA_MAX_WIRE_BYTES;
let send = |channel: &str, payload: usize| {
connected_session().send_many(
channel,
vec![0; payload],
CultNetRudpSendOptions::default(),
Some(FRAGMENT_BYTES),
)
};
assert!(send(GAMECULT_MEDIA_CHANNEL, ceiling).is_ok());
let error = send(GAMECULT_MEDIA_CHANNEL, ceiling + 1).unwrap_err();
assert!(error.to_string().contains("max_payload_bytes"), "{error}");
assert!(send("schema", ceiling + 1).is_ok(), "other channels keep the session cap");
}
/// Reassembly is where memory is spent, so the receiver enforces the ceiling
/// itself, not only the sender.
#[test]
fn a_session_refuses_to_reassemble_more_than_the_ceiling_on_the_media_channel() {
let ceiling = GAMECULT_MEDIA_MAX_WIRE_BYTES;
let packets = connected_session()
.send_many(
"schema",
vec![0; ceiling + 1],
CultNetRudpSendOptions::default(),
Some(FRAGMENT_BYTES),
)
.unwrap();
assert!(packets.len() > 1);
let deliver = |channel: &str| -> anyhow::Result<Vec<CultNetRudpDeliveredFrame>> {
let mut receiver = connected_session();
let mut delivered = Vec::new();
for packet in &packets {
let mut packet = packet.clone();
packet.channel_id = channel.to_string();
delivered.extend(receiver.receive(&packet, 0)?.delivered);
}
Ok(delivered)
};
let frames = deliver("schema").expect("the same fragments reassemble on another channel");
assert_eq!(frames.len(), 1);
assert_eq!(frames[0].payload.len(), ceiling + 1);
let error = deliver(GAMECULT_MEDIA_CHANNEL).unwrap_err();
assert!(error.to_string().contains("max_payload_bytes"), "{error}");
}
/// A producer learns at encode time that a record is over the ceiling, rather
/// than the receiver dropping it.
#[test]
fn the_encoder_refuses_a_record_over_the_ceiling() {
let wrap = |payload: usize| GameCultMediaWireRecord::AudioParity(sized_audio_parity(payload, 300));
let at = payload_for_wire_len(GAMECULT_MEDIA_MAX_WIRE_BYTES, &wrap);
assert!(encode_media_wire_record(&wrap(at), provenance()).is_ok());
let error = encode_media_wire_record(&wrap(at + 1), provenance()).unwrap_err();
assert!(error.to_string().contains("ceiling"), "{error}");
}
/// A record that arrives unfragmented is held to the same ceiling.
#[test]
fn a_session_refuses_an_unfragmented_media_packet_over_the_ceiling() {
let packets = connected_session()
.send_many(
"schema",
vec![0; GAMECULT_MEDIA_MAX_WIRE_BYTES + 1],
CultNetRudpSendOptions::default(),
None,
)
.unwrap();
assert_eq!(packets.len(), 1, "sent whole");
let mut packet = packets[0].clone();
assert!(connected_session().receive(&packet, 0).is_ok());
packet.channel_id = GAMECULT_MEDIA_CHANNEL.to_string();
let error = connected_session().receive(&packet, 0).unwrap_err();
assert!(error.to_string().contains("max_payload_bytes"), "{error}");
}

1001
packages/cultnet-rs/tests/rudp_connect_generation.rs Normal file
View file

File diff suppressed because it is too large Load diff

22
packages/cultnet-rs/tests/rudp_server_hub.rs
View file

@ -168,7 +168,7 @@ fn hub_keeps_independent_peer_sessions_and_exposes_connect_evidence() -> Result<
}
#[test]
fn hub_fences_replaced_generations_and_does_not_replace_connect_retransmits() -> Result<()> {
fn hub_fences_replaced_generations() -> Result<()> {
let server_socket = socket()?;
let server_addr = server_socket.local_addr()?;
let mut hub = CultNetRudpServerHub::new(CultNetRudpServerHubOptions::new(
@ -189,12 +189,6 @@ fn hub_fences_replaced_generations_and_does_not_replace_connect_retransmits() ->
})?;
let original = connect(&mut hub, &mut provider, &original_evidence)?;
provider.connect(original_evidence.clone())?;
for _ in 0..3 {
assert!(hub.receive_event_once()?.is_none());
}
assert_eq!(hub.session(original.remote_addr), Some(&original));
provider.connect(replacement_evidence.clone())?;
let mut disconnected = None;
for _ in 0..20 {
@ -253,7 +247,7 @@ fn identical_connect_retransmit_reuses_pending_accept_after_loss() -> Result<()>
let client_socket = socket()?;
let mut client_session = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});
@ -292,13 +286,13 @@ fn identical_connect_retransmit_reuses_pending_accept_after_loss() -> Result<()>
fn exact_reliable_acks_clear_more_than_one_ack_window_of_fragments() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});
@ -352,7 +346,7 @@ fn exact_reliable_acks_clear_more_than_one_ack_window_of_fragments() -> Result<(
fn session_and_hub_reject_configured_memory_bounds() -> Result<()> {
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});
@ -360,7 +354,7 @@ fn session_and_hub_reject_configured_memory_bounds() -> Result<()> {
receiver.set_max_pending_fragment_sets(1)?;
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});
@ -428,13 +422,13 @@ fn session_and_hub_reject_configured_memory_bounds() -> Result<()> {
fn replay_history_is_bounded_and_idle_hub_sessions_expire() -> Result<()> {
let mut sender = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});
let mut receiver = CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id: CONNECTION_ID,
initial_sequence: 1,
initial_sequence: Some(1),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
});

232
packages/cultnet-rs/tests/rudp_stray_packets.rs
View file

@ -30,9 +30,13 @@ fn socket() -> Result<UdpSocket> {
}
fn raw_session(connection_id: u32) -> CultNetRudpSession {
raw_session_from(connection_id, 1)
}
fn raw_session_from(connection_id: u32, initial_sequence: u32) -> CultNetRudpSession {
CultNetRudpSession::new(CultNetRudpSessionOptions {
connection_id,
initial_sequence: 1,
initial_sequence: Some(initial_sequence),
resend_delay_ms: 250,
max_pending_reliable_packets: None,
})
@ -442,7 +446,7 @@ fn server_mode_ends_the_session_when_the_session_refuses_a_packet() -> Result<()
#[test]
fn constructors_reject_limits_that_make_a_connect_unadmittable() -> Result<()> {
let mut hub = CultNetRudpServerHubOptions::new("hub", socket()?, CONNECTION_ID);
hub.initial_sequence = u32::MAX;
hub.initial_sequence = Some(u32::MAX);
assert!(CultNetRudpServerHub::new(hub).is_err());
let mut hub = CultNetRudpServerHubOptions::new("hub", socket()?, CONNECTION_ID);
hub.max_pending_reliable_packets = Some(0);
@ -455,13 +459,13 @@ fn constructors_reject_limits_that_make_a_connect_unadmittable() -> Result<()> {
] {
let addr = "127.0.0.1:9".parse()?;
let mut options = build(socket()?, addr);
options.initial_sequence = u32::MAX;
options.initial_sequence = Some(u32::MAX);
assert!(CultNetRudpSocketTransportConnection::new(options).is_err());
let mut options = build(socket()?, addr);
options.max_pending_reliable_packets = Some(0);
assert!(CultNetRudpSocketTransportConnection::new(options).is_err());
let mut options = build(socket()?, addr);
options.initial_sequence = u32::MAX - 1;
options.initial_sequence = Some(u32::MAX - 1);
options.max_pending_reliable_packets = Some(1);
assert!(CultNetRudpSocketTransportConnection::new(options).is_ok());
}
@ -725,7 +729,7 @@ fn a_flush_fails_when_a_new_endpoint_replaces_the_peer_mid_wait() -> Result<()>
send_to(&peer_a, server_addr, &raw_session(CONNECTION_ID).create_connect(0, b"a".to_vec())?)?;
let _ = server.receive_once()?;
server.send_reliable("schema", b"for A".to_vec())?;
send_to(&peer_b, server_addr, &raw_session(CONNECTION_ID).create_connect(0, b"b".to_vec())?)?;
send_to(&peer_b, server_addr, &raw_session_from(CONNECTION_ID, 2).create_connect(0, b"b".to_vec())?)?;
let error = server.flush_reliable(Duration::from_millis(300)).expect_err("A's write was forgotten");
assert!(
@ -745,8 +749,224 @@ fn a_repeated_connect_acknowledges_what_it_carries() -> Result<()> {
let accept = server.accept_connect(&connect, 0, Vec::new())?;
let mut repeat = connect.clone();
repeat.ack = accept.sequence;
let reply = server.answer_repeated_connect(&repeat, 1)?;
let reply = server.accept_connect(&repeat, 1, Vec::new())?;
assert_eq!(reply.packet_type, CultNetRudpPacketType::Ack);
assert_eq!(server.outstanding_reliable_packet_count(), 0);
Ok(())
}
/// A connected client session with one ordered write the peer never received.
fn client_with_a_lost_write() -> Result<(CultNetRudpSession, CultNetRudpSession)> {
let mut client = raw_session(CONNECTION_ID);
let mut server = raw_session(CONNECTION_ID);
let accept = server.accept_connect(&client.create_connect(0, Vec::new())?, 0, Vec::new())?;
client.receive(&accept, 0)?;
client.send(
"schema",
b"owed to the old session".to_vec(),
CultNetRudpSendOptions {
reliable: true,
ordered: true,
..Default::default()
},
)?;
// A fragmented write larger than the send window leaves part of it queued.
client.send_many(
"schema",
vec![7u8; 40 * 8],
CultNetRudpSendOptions {
reliable: true,
ordered: true,
..Default::default()
},
Some(8),
)?;
assert_eq!(client.outstanding_reliable_packet_count(), 41);
assert!(client.queued_reliable_packet_count() > 0);
Ok((client, server))
}
/// A write belongs to the session it was issued in. Whichever way that session
/// ends, the write it still owed dies with it: it is not retransmitted into a
/// later session, where the new peer would deliver it.
#[test]
fn every_way_a_session_ends_drops_the_writes_it_owed() -> Result<()> {
type End = fn(&mut CultNetRudpSession, &mut CultNetRudpSession) -> Result<()>;
let endings: [(&str, End); 4] = [
("peer Disconnect", |client, server| {
client.receive(&server.create_disconnect(b"bye".to_vec()), 1)?;
Ok(())
}),
("local disconnect", |client, _| {
client.create_disconnect(b"bye".to_vec());
Ok(())
}),
("timeout", |client, _| {
assert!(client.check_timeout(1_000, 10));
Ok(())
}),
("refusal", |client, _| {
client.end_refused_session();
Ok(())
}),
];
for (name, end) in endings {
let (mut client, mut server) = client_with_a_lost_write()?;
end(&mut client, &mut server)?;
assert_eq!(client.outstanding_reliable_packet_count(), 0, "{name}: the write survived the end");
let mut next_server = raw_session(CONNECTION_ID);
let connect = client.create_connect(2_000, Vec::new())?;
next_server.accept_connect(&connect, 2_000, Vec::new())?;
for resend in client.due_resends(60_000) {
let result = next_server.receive(&resend, 60_000)?;
assert!(
result.delivered.is_empty(),
"{name}: the old session's write was delivered in the next session"
);
}
}
Ok(())
}
fn connected_client(server_socket: &UdpSocket, server_addr: SocketAddr) -> Result<(CultNetRudpSocketTransportConnection, CultNetRudpSession, SocketAddr)> {
let mut client = CultNetRudpSocketTransportConnection::new(
CultNetRudpSocketTransportOptions::client("client", socket()?, server_addr, CONNECTION_ID),
)?;
client.connect(b"hello".to_vec())?;
let mut buffer = vec![0_u8; 65_535];
let (received, client_addr) = server_socket.recv_from(&mut buffer)?;
let mut peer = raw_session(CONNECTION_ID);
let accept = peer.accept_connect(&decode_rudp_packet(&buffer[..received])?, 0, Vec::new())?;
send_to(server_socket, client_addr, &accept)?;
let _ = client.receive_once()?;
Ok((client, peer, client_addr))
}
/// A local `disconnect()` ends the session like any other end: the receipt of a
/// write the peer never acknowledged is Invalidated, stays so after a
/// reconnect, and the write is not carried into the new session.
#[test]
fn a_receipt_invalidated_by_a_local_disconnect_stays_invalidated_after_reconnecting() -> Result<()> {
let server = socket()?;
let server_addr = server.local_addr()?;
let (mut client, _peer, _) = connected_client(&server, server_addr)?;
let receipt = client.send_reliable("schema", b"written before the disconnect".to_vec())?;
assert_eq!(client.reliable_send_status(&receipt), CultNetRudpReliableSendStatus::Pending);
client.disconnect(b"bye".to_vec())?;
assert_eq!(client.reliable_send_status(&receipt), CultNetRudpReliableSendStatus::Invalidated);
assert_eq!(client.outstanding_reliable_packet_count(), 0, "the write survived the disconnect");
client.connect(b"hello again".to_vec())?;
assert_eq!(
client.reliable_send_status(&receipt),
CultNetRudpReliableSendStatus::Invalidated,
"reconnecting brought the write back"
);
assert_eq!(client.outstanding_reliable_packet_count(), 1, "only the new Connect is owed");
Ok(())
}
/// A flush that starts after the session ended has no live session to wait on.
#[test]
fn a_flush_fails_after_a_local_disconnect_or_a_timeout() -> Result<()> {
let server = socket()?;
let server_addr = server.local_addr()?;
let (mut client, _peer, _) = connected_client(&server, server_addr)?;
client.send_reliable("schema", b"owed".to_vec())?;
client.disconnect(b"bye".to_vec())?;
let error = client.flush_reliable(Duration::from_millis(200)).expect_err("the write was dropped");
assert!(error.to_string().contains("ended before its reliable writes were acknowledged"), "{error}");
let server = socket()?;
let server_addr = server.local_addr()?;
let (mut client, _peer, _) = connected_client(&server, server_addr)?;
client.send_reliable("schema", b"owed".to_vec())?;
std::thread::sleep(Duration::from_millis(5));
assert!(client.check_timeout(1));
let error = client.flush_reliable(Duration::from_millis(200)).expect_err("the write was dropped");
assert!(error.to_string().contains("ended before its reliable writes were acknowledged"), "{error}");
Ok(())
}
/// A flush that starts after a reconnect waits on the new session: the end of
/// the old one does not fail it.
#[test]
fn a_flush_started_after_a_reconnect_waits_on_the_new_session() -> Result<()> {
let server = socket()?;
let server_addr = server.local_addr()?;
let (mut client, _peer, client_addr) = connected_client(&server, server_addr)?;
client.send_reliable("schema", b"owed to the old session".to_vec())?;
client.disconnect(b"bye".to_vec())?;
client.connect(b"hello again".to_vec())?;
let connect = drain(&server)?
.into_iter()
.rfind(|packet| packet.packet_type == CultNetRudpPacketType::Connect)
.expect("the reconnect reached the peer");
let accept = raw_session(CONNECTION_ID).accept_connect(&connect, 0, Vec::new())?;
send_to(&server, client_addr, &accept)?;
let _ = client.receive_once()?;
client.flush_reliable(Duration::from_millis(500))?;
Ok(())
}
/// A server-mode transport that accepts a new Connect after its peer left has a
/// live session again, and a flush started in it waits on that session.
#[test]
fn a_flush_started_after_the_server_accepts_a_new_connect_waits_on_the_new_session() -> Result<()> {
let server_socket = socket()?;
let server_addr = server_socket.local_addr()?;
let mut server = CultNetRudpSocketTransportConnection::new(
CultNetRudpSocketTransportOptions::server("server", server_socket, CONNECTION_ID),
)?;
let peer = socket()?;
let mut first = raw_session(CONNECTION_ID);
send_to(&peer, server_addr, &first.create_connect(0, b"peer".to_vec())?)?;
let _ = server.receive_once()?;
send_to(&peer, server_addr, &first.create_disconnect(b"bye".to_vec()))?;
let _ = server.receive_once()?;
let mut next = raw_session(CONNECTION_ID);
send_to(&peer, server_addr, &next.create_connect(0, b"peer".to_vec())?)?;
let _ = server.receive_once()?;
let accept = drain(&peer)?
.into_iter()
.rfind(|packet| packet.packet_type == CultNetRudpPacketType::Accept)
.expect("the new Connect was accepted");
next.receive(&accept, 0)?;
send_to(&peer, server_addr, &next.create_ack_for_received(accept.sequence))?;
let _ = server.receive_once()?;
server.flush_reliable(Duration::from_millis(500))?;
Ok(())
}
/// The session ended, the peer did not notice: what it sent before the end is
/// still remembered, held frames included, so its retransmits are duplicates.
#[test]
fn an_ending_does_not_forget_what_was_received_from_the_peer() -> Result<()> {
let mut client = raw_session(CONNECTION_ID);
let mut server = raw_session(CONNECTION_ID);
let accept = server.accept_connect(&client.create_connect(0, Vec::new())?, 0, Vec::new())?;
client.receive(&accept, 0)?;
let ordered = CultNetRudpSendOptions {
reliable: true,
ordered: true,
..Default::default()
};
let s1 = client.send("schema", b"s1".to_vec(), ordered.clone())?;
let s2 = client.send("schema", b"s2".to_vec(), ordered)?;
assert!(server.receive(&s2, 1)?.delivered.is_empty(), "s1 is missing, so s2 is held");
assert!(server.check_timeout(1_000, 10));
let released = server.receive(&s1, 1_001)?.delivered;
assert_eq!(
released.iter().map(|frame| frame.payload.clone()).collect::<Vec<_>>(),
[b"s1".to_vec(), b"s2".to_vec()],
"the ending forgot the held frame"
);
assert!(server.receive(&s2, 1_002)?.delivered.is_empty(), "the ending forgot what was received");
Ok(())
}

443
packages/cultnet-ts/src/rudp.ts
View file

@ -1,3 +1,4 @@
import { randomInt } from "node:crypto";
import { EventEmitter } from "node:events";
import { type RemoteInfo, type Socket } from "node:dgram";
@ -18,6 +19,34 @@ const RUDP_FIXED_HEADER_BYTES = 36;
const MAX_CHANNEL_ID_BYTES = 255;
export const CULTNET_RUDP_RELIABLE_SEND_WINDOW_PACKETS = 32;
const RUDP_RECEIVED_SEQUENCE_WINDOW = 4_096;
/// How long a client keeps retransmitting a Connect nobody answered before it
/// abandons that attempt for a fresh one. A Connect the server answers with an
/// Ack, never an Accept, is one the server judged stale (see `acceptConnect`);
/// retransmitting the same sequence would never change its mind.
const RUDP_CONNECT_ATTEMPT_MS = 3_000;
/// A sequence drawn from a secure source in [1, 2^31). The Connect's sequence
/// is what tells a peer whether a Connect repeats one it already accepted or
/// starts a new session, so two sessions must not share one by default.
function drawSequence(): number {
return randomInt(1, 2 ** 31);
}
/// Whether `sequence` is at or before `mark` in serial order, within the
/// receive window. The compare is modular, so it holds across the wrap of the
/// 32-bit space; a sequence further back than the window is the duplicate
/// test's below-window clause, and one ahead of the mark is never before it.
function atOrBefore(sequence: number, mark: number): boolean {
return ((mark - sequence) >>> 0) < RUDP_RECEIVED_SEQUENCE_WINDOW;
}
function packetAcknowledges(packet: CultNetRudpPacket, sequence: number): boolean {
if (packet.ack === sequence) return true;
for (let bit = 0; bit < 32; bit += 1) {
if ((packet.ackMask & (1 << bit)) !== 0 && packet.ack - bit - 1 === sequence) return true;
}
return false;
}
export type CultNetRudpPacketType =
| "connect"
@ -72,11 +101,6 @@ type PendingReliablePacket = {
lastSentAtMs: number;
};
type PendingOrderedFrame = {
frame: CultNetRudpDeliveredFrame;
nextSequence: number;
};
type FragmentBuffer = {
channelId: string;
reliable: boolean;
@ -150,6 +174,24 @@ export class CultNetRudpSession {
readonly #nextSequencedByChannel = new Map<string, number>();
#nextFragmentId = 1;
#connected = false;
/// Advances every time a generation ends. Everything issued in a generation
/// (writes, flushes) belongs to it and dies with it.
#generation = 0;
/// True from the end of a generation until the next Connect or Accept begins
/// one. A flush started in that interval has no live generation to wait on.
#ended = false;
/// The sequence of the Connect that started the current generation: sent by
/// this side, or accepted from the peer. A Connect repeats exactly when the
/// session is connected and the Connect carries this sequence.
#connectSequence: number | undefined;
/// A Connect this side sent is unanswered. Only then is an Accept honoured.
#awaitingAccept = false;
/// The payload of the Connect this side sent, kept so a fresh attempt can
/// always be built whatever became of the pending packet.
#connectPayload = new Uint8Array();
/// When the unanswered Connect first went out; `dueResends` abandons it for a
/// fresh attempt once `RUDP_CONNECT_ATTEMPT_MS` has passed.
#connectStartedAtMs = 0;
readonly #maxPendingReliablePackets: number | undefined;
#lastReceivedAtMs: number | undefined;
#highestReceivedSequence: number | undefined;
@ -157,8 +199,15 @@ export class CultNetRudpSession {
readonly #latestSequencedByChannel = new Map<string, number>();
readonly #pendingReliable = new Map<number, PendingReliablePacket>();
readonly #queuedReliable: CultNetRudpPacket[] = [];
readonly #orderedNextSequenceByChannel = new Map<string, number>();
readonly #orderedBuffers = new Map<string, Map<number, PendingOrderedFrame>>();
/// Every reliable sequence up to and including this one has been received
/// since the peer state was last reset. Only the handshake seeds it: the
/// peer's Connect on the accepting side, the peer's Accept on the connecting
/// side. Reliable data that arrives before that is refused.
#receivedThrough: number | undefined;
/// Ordered frames received but not yet deliverable, keyed by first sequence.
/// A frame is held for exactly one reason: a reliable sequence below it has
/// not arrived.
readonly #orderedHeld = new Map<number, CultNetRudpDeliveredFrame>();
readonly #fragmentBuffers = new Map<string, FragmentBuffer>();
/// Matches cultnet-rs's `max_pending_fragment_sets`. A fragment set is only
/// removed on successful reassembly, so a set that loses one fragment is
@ -169,7 +218,7 @@ export class CultNetRudpSession {
constructor(options: CultNetRudpSessionOptions) {
this.connectionId = toUint32(options.connectionId, "connectionId");
this.#nextSequence = toUint32(options.initialSequence ?? 1, "initialSequence");
this.#nextSequence = toUint32(options.initialSequence ?? drawSequence(), "initialSequence");
if (this.#nextSequence === 0xffff_ffff) {
throw new Error("RUDP initialSequence must leave room for a reliable packet.");
}
@ -211,23 +260,53 @@ export class CultNetRudpSession {
return this.#lastReceivedAtMs;
}
resetPeerState(): void {
get generation(): number {
return this.#generation;
}
get ended(): boolean {
return this.#ended;
}
/// The one way a session generation ends. The session stops being connected
/// and what it still owed the peer dies with it: a write not yet acknowledged
/// is dropped, so no later session retransmits it or credits an ack to it.
/// What was learned from the peer is not touched: the peer may not know the
/// session ended, and forgetting what it sent would let its retransmits be
/// delivered twice.
#endSession(): void {
this.#connected = false;
this.#ended = true;
this.#awaitingAccept = false;
this.#generation += 1;
this.#pendingReliable.clear();
this.#queuedReliable.splice(0);
}
/// Ends the current generation and forgets everything learned from the peer;
/// sequence numbers already issued stay issued.
resetPeerState(): void {
this.#endSession();
this.#lastReceivedAtMs = undefined;
this.#highestReceivedSequence = undefined;
this.#receivedSequences.clear();
this.#nextSequencedByChannel.clear();
this.#latestSequencedByChannel.clear();
this.#pendingReliable.clear();
this.#queuedReliable.splice(0);
this.#orderedNextSequenceByChannel.clear();
this.#orderedBuffers.clear();
this.#receivedThrough = undefined;
this.#orderedHeld.clear();
this.#fragmentBuffers.clear();
this.#fragmentSetsEvicted = 0;
}
createConnect(nowMs = 0, payload = new Uint8Array()): CultNetRudpPacket {
// A session that has had a peer starts a new generation: nothing it learned
// from that peer describes the one this Connect reaches, and what it still
// owed that peer no longer takes room in the queue.
if (this.#connectSequence !== undefined) {
this.resetPeerState();
}
this.#ensureReliableCapacity(1);
this.#ended = false;
const packet = this.#createPacket({
packetType: "connect",
channelId: "control",
@ -235,24 +314,68 @@ export class CultNetRudpSession {
ordered: true,
payload,
});
this.#connectSequence = packet.sequence;
this.#awaitingAccept = true;
this.#connectStartedAtMs = nowMs;
this.#connectPayload = new Uint8Array(payload);
this.#trackReliable(packet, nowMs);
return packet;
}
/// Whether `packet` is a Connect this session's current generation already
/// owns: a retransmit of the Connect that started it (the session is
/// connected and the sequence is that Connect's), or a stale copy of an
/// earlier attempt by the same client (a sequence before it, within the
/// receive window). Servers that keep one session per peer ask this to tell a
/// Connect that starts a new session from one that does not; anything else
/// that reaches `acceptConnect` starts a new generation.
connectRepeats(packet: CultNetRudpPacket): boolean {
return packet.packetType === "connect"
&& this.#connected
&& this.#connectSequence !== undefined
&& (this.#connectSequence === packet.sequence
|| this.#connectIsStale(packet.sequence, this.#connectSequence));
}
/// A Connect that precedes the current generation's within the receive window
/// is the client's earlier attempt, delayed in the network: a client that
/// retried never sends a lower sequence again. Restarting on it would strand
/// the client, which honours only the Accept for its newest Connect. The rule
/// is TCP's answer to a delayed SYN (RFC 5961's challenge ACK): keep the
/// connection and answer with an Ack. A restarted client whose random initial
/// sequence lands in this window is answered the same way and abandons the
/// attempt for a fresh draw (`RUDP_CONNECT_ATTEMPT_MS`).
#connectIsStale(sequence: number, current: number): boolean {
return sequence !== current && atOrBefore(sequence, current);
}
/// Answers a Connect. A repeat of the accepted one queues nothing, so a
/// Connect storm cannot grow the reliable queue: the reply is the Accept
/// still awaiting acknowledgement, or an Ack once it was acknowledged. A
/// stale copy of an earlier attempt gets the same reply and changes nothing
/// else: it is not evidence the peer is alive. Any other Connect ends the
/// current generation, forgets the peer and accepts a new one.
acceptConnect(packet: CultNetRudpPacket, nowMs = 0, payload = new Uint8Array()): CultNetRudpPacket {
this.#requireConnection(packet);
if (packet.packetType !== "connect") {
throw new Error(`Expected RUDP connect packet, got ${packet.packetType}.`);
}
// A Connect from a peer this session already accepted repeats.
if (this.#connected) {
return this.answerRepeatedConnect(packet, nowMs);
if (this.connectRepeats(packet)) {
if (this.#connectSequence === packet.sequence) {
this.#applyAcknowledgements(packet);
this.#rememberReceived(packet.sequence);
this.#lastReceivedAtMs = nowMs;
}
return this.#pendingAcceptForResend(nowMs) ?? this.createAck();
}
this.resetPeerState();
this.#ensureReliableCapacity(1);
this.#rememberReceived(packet.sequence);
this.#seedReceived(packet.sequence);
this.#lastReceivedAtMs = nowMs;
this.#connectSequence = packet.sequence;
this.#connected = true;
this.#ended = false;
const response = this.#createPacket({
packetType: "accept",
channelId: "control",
@ -264,24 +387,12 @@ export class CultNetRudpSession {
return response;
}
/**
* Answers a Connect from a peer this session has already accepted. The packet's
* sequence is remembered and nothing is queued, so a Connect storm cannot grow
* the reliable queue. The reply is the Accept still awaiting acknowledgement,
* or an Ack once it was acknowledged.
*/
answerRepeatedConnect(packet: CultNetRudpPacket, nowMs = 0): CultNetRudpPacket {
this.#requireConnection(packet);
if (packet.packetType !== "connect") {
throw new Error(`Expected RUDP connect packet, got ${packet.packetType}.`);
}
this.#applyAcknowledgements(packet);
this.#rememberReceived(packet.sequence);
this.#lastReceivedAtMs = nowMs;
/// The Accept still awaiting acknowledgement, resent.
#pendingAcceptForResend(nowMs: number): CultNetRudpPacket | undefined {
const pendingAccept = [...this.#pendingReliable.values()]
.find(pending => pending.packet.packetType === "accept");
if (!pendingAccept) {
return this.createAck();
return undefined;
}
pendingAccept.lastSentAtMs = nowMs;
return {
@ -369,15 +480,28 @@ export class CultNetRudpSession {
receive(packet: CultNetRudpPacket, nowMs = 0): CultNetRudpReceiveResult {
this.#requireConnection(packet);
this.#applyAcknowledgements(packet);
// An Accept counts only while this side's Connect is unanswered and the
// Accept names it. A late or duplicate one, or one from an earlier
// generation, must not seed the watermark or revive an ended session.
const honoursAccept = packet.packetType === "accept"
&& this.#awaitingAccept
&& this.#connectSequence !== undefined
&& packetAcknowledges(packet, this.#connectSequence);
if (packet.packetType === "accept" && !honoursAccept) {
return { delivered: [], readyToSend: [] };
}
// While this side's Connect awaits its Accept, only the Accept it honours
// retires it: an Ack that names the Connect (a server's reply to a repeat or
// a stale copy) says the server did not start a session.
if (honoursAccept || !this.#awaitingAccept) {
this.#applyAcknowledgements(packet);
}
const readyToSend = this.#promoteQueuedReliable(nowMs);
this.#lastReceivedAtMs = nowMs;
const expectedSequenceIfUninitialized = this.#highestReceivedSequence === undefined
? packet.sequence
: this.#highestReceivedSequence + 1;
if (packet.packetType === "accept") {
this.#rememberReceived(packet.sequence);
if (honoursAccept) {
this.#awaitingAccept = false;
this.#seedReceived(packet.sequence);
this.#connected = true;
return { delivered: [], readyToSend };
}
@ -404,7 +528,7 @@ export class CultNetRudpSession {
}
if (packet.packetType === "disconnect") {
this.#connected = false;
this.#endSession();
return {
delivered: [],
readyToSend,
@ -417,39 +541,41 @@ export class CultNetRudpSession {
return { delivered: [], readyToSend };
}
const isDuplicate = (packet.reliable ?? false)
&& (this.#receivedSequences.has(packet.sequence)
|| (this.#highestReceivedSequence !== undefined
&& packet.sequence < this.#highestReceivedSequence
&& this.#highestReceivedSequence - packet.sequence >= RUDP_RECEIVED_SEQUENCE_WINDOW));
// Reliable data before the handshake has seeded the watermark has no place
// in the order: refuse it unremembered, so it is not acknowledged and the
// sender retransmits it once the handshake is done.
if ((packet.reliable ?? false) && this.#receivedThrough === undefined) {
return { delivered: [], readyToSend };
}
const isDuplicate = (packet.reliable ?? false) && this.#wasReceived(packet.sequence);
if (packet.reliable ?? false) this.#rememberReceived(packet.sequence);
if (isDuplicate) {
return { delivered: [], readyToSend };
}
const delivered: CultNetRudpDeliveredFrame[] = [];
const reassembled = this.#reassemble(packet);
if (!reassembled) {
return { delivered: [], readyToSend };
}
if (!reassembled.ordered && (packet.sequenced ?? false)) {
const newestSequence = reassembled.nextSequence - 1;
const latestSequence = this.#latestSequencedByChannel.get(reassembled.frame.channelId);
if (latestSequence !== undefined && newestSequence <= latestSequence) {
return { delivered: [], readyToSend };
if (reassembled) {
if (reassembled.ordered) {
this.#orderedHeld.set(reassembled.frame.sequence, reassembled.frame);
} else if (!(packet.sequenced ?? false)) {
delivered.push(reassembled.frame);
} else {
const newestSequence = reassembled.nextSequence - 1;
const latestSequence = this.#latestSequencedByChannel.get(reassembled.frame.channelId);
if (latestSequence === undefined || newestSequence > latestSequence) {
this.#latestSequencedByChannel.set(reassembled.frame.channelId, newestSequence);
delivered.push(reassembled.frame);
}
}
this.#latestSequencedByChannel.set(reassembled.frame.channelId, newestSequence);
return { delivered: [reassembled.frame], readyToSend };
}
if (!reassembled.ordered) {
return { delivered: [reassembled.frame], readyToSend };
}
return {
delivered: this.#deliverOrdered(reassembled.frame, reassembled.nextSequence, expectedSequenceIfUninitialized),
readyToSend,
};
// Any reliable packet may have advanced the watermark, a fragment or an
// unordered frame as much as an ordered one, so the drain runs after all of
// them.
delivered.push(...this.#drainOrdered());
return { delivered, readyToSend };
}
createAck(): CultNetRudpPacket {
@ -490,7 +616,12 @@ export class CultNetRudpSession {
}
createAckForReceived(sequence: number): CultNetRudpPacket {
// What `receive` refused is not acknowledged by name (the ack carries only
// what was received), so its sender retransmits it.
const receivedSequence = toUint32(sequence, "received sequence");
if (!this.#wasReceived(receivedSequence)) {
return this.createAck();
}
const { ack } = this.#ackState();
return ack >= receivedSequence && ack - receivedSequence <= 32
? this.createAck()
@ -506,7 +637,7 @@ export class CultNetRudpSession {
}
createDisconnect(reason = new Uint8Array()): CultNetRudpPacket {
this.#connected = false;
this.#endSession();
return this.#createPacket({
packetType: "disconnect",
channelId: "control",
@ -521,11 +652,15 @@ export class CultNetRudpSession {
if (nowMs - this.#lastReceivedAtMs <= timeoutMs) {
return false;
}
this.#connected = false;
this.#endSession();
return true;
}
dueResends(nowMs: number): CultNetRudpPacket[] {
const fresh = this.#abandonUnansweredConnect(nowMs);
if (fresh) {
return [fresh];
}
const due: CultNetRudpPacket[] = [];
for (const pending of this.#pendingReliable.values()) {
if (nowMs - pending.lastSentAtMs >= this.resendDelayMs) {
@ -536,6 +671,27 @@ export class CultNetRudpSession {
return due.sort((left, right) => left.sequence - right.sequence);
}
/// A Connect unanswered for `RUDP_CONNECT_ATTEMPT_MS` is replaced, not
/// retransmitted further, by a Connect with the same payload and the
/// abandoned sequence plus the receive window less one. Nothing was ever sent
/// in an unanswered generation, so no sequence issued so far is owed. The jump
/// keeps a late copy of the abandoned Connect inside the new one's stale
/// window, so a server that took the new Connect answers the copy with an Ack
/// instead of restarting; and it leaves the stale window of whatever
/// generation the server holds, unless that generation sits exactly at the
/// jump, and then the next attempt leaves it. The first Connect of a session
/// is the only one drawn at random.
#abandonUnansweredConnect(nowMs: number): CultNetRudpPacket | undefined {
if (!this.#awaitingAccept || nowMs - this.#connectStartedAtMs < RUDP_CONNECT_ATTEMPT_MS) {
return undefined;
}
if (this.#connectSequence === undefined) {
return undefined;
}
this.#nextSequence = Math.min(this.#connectSequence + RUDP_RECEIVED_SEQUENCE_WINDOW - 1, 0xffff_fffe);
return this.createConnect(nowMs, this.#connectPayload);
}
#createPacket(packet: {
packetType: CultNetRudpPacketType;
channelId: string;
@ -616,8 +772,34 @@ export class CultNetRudpSession {
}
}
/// The handshake's one act on the watermark: the peer's Connect or Accept is
/// the first sequence of the session, whatever else it has sent.
#seedReceived(sequence: number): void {
this.#receivedThrough = sequence;
this.#rememberReceived(sequence);
}
/// The one duplicate test: true for a sequence in the window that was
/// received, for one below the window, and for one at or before the
/// watermark. The watermark starts at the handshake's seed, so a frame the
/// peer sent in an earlier generation (every sequence it issued is below the
/// Connect that began this one) is a duplicate of something already delivered
/// and is acknowledged, never delivered again.
#wasReceived(sequence: number): boolean {
return this.#receivedSequences.has(sequence)
|| (this.#highestReceivedSequence !== undefined
&& sequence < this.#highestReceivedSequence
&& this.#highestReceivedSequence - sequence >= RUDP_RECEIVED_SEQUENCE_WINDOW)
|| (this.#receivedThrough !== undefined && atOrBefore(sequence, this.#receivedThrough));
}
#rememberReceived(sequence: number): void {
this.#receivedSequences.add(sequence);
if (this.#receivedThrough !== undefined) {
let through = this.#receivedThrough;
while (this.#receivedSequences.has(through + 1)) through += 1;
this.#receivedThrough = through;
}
if (this.#highestReceivedSequence === undefined || sequence > this.#highestReceivedSequence) {
this.#highestReceivedSequence = sequence;
}
@ -734,68 +916,20 @@ export class CultNetRudpSession {
};
}
#deliverOrdered(
frame: CultNetRudpDeliveredFrame,
nextSequence: number,
expectedSequenceIfUninitialized: number,
): CultNetRudpDeliveredFrame[] {
let next = this.#orderedNextSequenceByChannel.get(frame.channelId);
if (next === undefined) {
next = Math.min(expectedSequenceIfUninitialized, frame.sequence);
this.#orderedNextSequenceByChannel.set(frame.channelId, next);
}
while (frame.sequence > next && this.#receivedSequences.has(next) && !this.#orderedBuffers.get(frame.channelId)?.has(next)) {
next += 1;
this.#orderedNextSequenceByChannel.set(frame.channelId, next);
}
if (frame.sequence < next) {
return [];
}
if (frame.sequence > next) {
let buffer = this.#orderedBuffers.get(frame.channelId);
if (!buffer) {
buffer = new Map();
this.#orderedBuffers.set(frame.channelId, buffer);
}
buffer.set(frame.sequence, { frame, nextSequence });
return [];
}
this.#orderedNextSequenceByChannel.set(frame.channelId, nextSequence);
return [
frame,
...this.#drainOrdered(frame.channelId),
];
}
#drainOrdered(channelId: string): CultNetRudpDeliveredFrame[] {
const delivered: CultNetRudpDeliveredFrame[] = [];
const buffer = this.#orderedBuffers.get(channelId);
if (!buffer) {
return delivered;
}
let next = this.#orderedNextSequenceByChannel.get(channelId);
while (next !== undefined && buffer.has(next)) {
const pending = buffer.get(next)!;
buffer.delete(next);
delivered.push(pending.frame);
next = pending.nextSequence;
this.#orderedNextSequenceByChannel.set(channelId, next);
this.#skipReceivedNonChannelSequences(channelId);
}
return delivered;
}
#skipReceivedNonChannelSequences(channelId: string): void {
let next = this.#orderedNextSequenceByChannel.get(channelId);
while (next !== undefined && this.#receivedSequences.has(next) && !this.#orderedBuffers.get(channelId)?.has(next)) {
next += 1;
this.#orderedNextSequenceByChannel.set(channelId, next);
}
/// Delivers, in sequence order, every held ordered frame whose first sequence
/// is at most one past the watermark. Ordered delivery has one owner: the
/// watermark. A frame this call does not deliver waits for the sequence below
/// it, and nothing else releases it.
#drainOrdered(): CultNetRudpDeliveredFrame[] {
if (this.#receivedThrough === undefined) return [];
const releasable = [...this.#orderedHeld.keys()]
.filter((sequence) => sequence <= this.#receivedThrough! + 1)
.sort((left, right) => left - right);
return releasable.map((sequence) => {
const frame = this.#orderedHeld.get(sequence)!;
this.#orderedHeld.delete(sequence);
return frame;
});
}
#allocateFragmentId(): number {
@ -824,12 +958,8 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
#remoteHost: string | undefined;
#remotePort: number | undefined;
#closed = false;
// A flush belongs to the session it started in. Each session start advances
// the generation; each session end records the generation it ended and why.
// A reconnect from a disconnect listener starts a new generation but cannot
// un-end the old one.
#generation = 0;
#endedGeneration = -1;
// Why the last generation ended. A report only: the session owns whether it
// ended.
#endedReason: Uint8Array | undefined;
readonly #stats: CultNetTransportStats = {
bytesReceived: 0,
@ -892,7 +1022,6 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
if (this.#mode !== "client") {
throw new Error("Only a client RUDP socket transport can initiate connect.");
}
this.#generation += 1;
this.#sendPacket(this.#session.createConnect(Date.now(), payload));
}
@ -910,11 +1039,14 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
async flush(timeoutMs = 30_000): Promise<void> {
const deadline = Date.now() + Math.max(0, timeoutMs);
const generation = this.#generation;
// A flush belongs to the generation it started in: whatever ends that
// generation, and whatever begins after it (a reconnect from a disconnect
// listener included), the writes being waited on are gone.
const generation = this.#session.generation;
for (;;) {
// An ended session forgot its unacknowledged writes; reporting them
// flushed would be a lie, even if a new session has since begun.
if (this.#endedGeneration >= generation) {
// flushed would be a lie.
if (this.#session.ended || this.#session.generation !== generation) {
throw new Error(
`RUDP session ended before its reliable writes were acknowledged: ${Buffer.from(this.#endedReason ?? new Uint8Array()).toString("utf8")}`,
);
@ -934,11 +1066,6 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
}
}
#endGeneration(reason: Uint8Array): void {
this.#endedGeneration = this.#generation;
this.#endedReason = reason;
}
ping(payload = new Uint8Array()): void {
this.#sendPacket(this.#session.createPing(payload));
}
@ -946,7 +1073,7 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
checkTimeout(timeoutMs: number, nowMs = Date.now()): boolean {
const timedOut = this.#session.checkTimeout(nowMs, timeoutMs);
if (timedOut) {
this.#endGeneration(Buffer.from("session timed out", "utf8"));
this.#endedReason = Buffer.from("session timed out", "utf8");
this.emit("timeout");
this.emit("close");
}
@ -973,7 +1100,7 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
// The goodbye is built after the reset, or its ack field would acknowledge
// the very frame the session refused.
this.#session.resetPeerState();
this.#endGeneration(reason);
this.#endedReason = reason;
try {
this.#sendPacket(this.#session.createDisconnect(reason));
} catch {
@ -1008,7 +1135,10 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
return;
}
let movedEndpoint = false;
// A Connect from another endpoint is a new client, whatever sequence it
// carries: the endpoint moves and the Connect starts a new generation. Only
// a Connect from the peer's own endpoint can be its repeat.
let connectFromNewEndpoint = false;
if (!this.#remoteHost || this.#remotePort === undefined) {
if (this.#mode === "server" && packet.packetType !== "connect") {
this.#stats.packetsDropped += 1;
@ -1018,9 +1148,9 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
this.#remotePort = remote.port;
} else if (remote.address !== this.#remoteHost || remote.port !== this.#remotePort) {
if (this.#mode === "server" && packet.packetType === "connect") {
movedEndpoint = true;
this.#remoteHost = remote.address;
this.#remotePort = remote.port;
connectFromNewEndpoint = true;
} else {
this.#stats.packetsDropped += 1;
return;
@ -1028,22 +1158,19 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
}
if (this.#mode === "server" && packet.packetType === "connect") {
// Only a Connect from the endpoint already accepted repeats, and the
// session answers it with the Accept owed. A Connect from another
// endpoint replaces the peer: the old session's writes die with it.
const repeated = this.#session.connected && !movedEndpoint;
if (!repeated) {
// A timed-out session already ended its generation; any other
// predecessor ends here, connected or not.
if (this.#endedGeneration < this.#generation) {
this.#endGeneration(Buffer.from("replaced by a new Connect", "utf8"));
}
// The session decides whether this Connect repeats the one it accepted; a
// repeat is answered with the Accept owed, anything else replaces the
// peer and the old session's writes die with it. A timed-out session
// already ended its generation; any other predecessor ends here.
if ((connectFromNewEndpoint || !this.#session.connectRepeats(packet)) && !this.#session.ended) {
this.#endedReason = Buffer.from("replaced by a new Connect", "utf8");
}
if (connectFromNewEndpoint) {
this.#session.resetPeerState();
}
let accept: CultNetRudpPacket;
try {
accept = this.#session.acceptConnect(packet, Date.now());
if (!repeated) this.#generation += 1;
} catch {
this.#stats.packetsDropped += 1;
return;
@ -1081,7 +1208,7 @@ export class CultNetRudpSocketTransportConnection extends EventEmitter implement
} satisfies CultNetTransportFrame);
}
if (result.disconnected) {
this.#endGeneration(result.disconnectReason ?? new Uint8Array());
this.#endedReason = result.disconnectReason ?? new Uint8Array();
this.emit("disconnect", { reason: result.disconnectReason ?? new Uint8Array() });
this.emit("close");
return;

245
packages/cultnet-ts/test/cultnet.test.ts
View file

@ -8,7 +8,7 @@ import { join } from "node:path";
import { tmpdir } from "node:os";
import { z } from "zod";
import { encode } from "@msgpack/msgpack";
import { decode as decodeMsgpack, encode } from "@msgpack/msgpack";
import {
CultCache,
SingleFileMessagePackBackingStore,
@ -166,6 +166,14 @@ function createDuplexPair(): { a: Duplex; b: Duplex } {
return { a, b };
}
/// Whether the packet's ack field and mask cover the sequence.
function acknowledges(packet: CultNetRudpPacket, sequence: number): boolean {
if (packet.ack === sequence) return true;
return packet.ack > sequence
&& packet.ack - sequence <= 32
&& (packet.ackMask & (1 << (packet.ack - sequence - 1))) !== 0;
}
async function bindUdpSocket(): Promise<Socket> {
const socket = dgram.createSocket("udp4");
await new Promise<void>((resolve) => socket.bind(0, "127.0.0.1", resolve));
@ -500,33 +508,32 @@ test("rudp session handshake acks reliable connect and accept packets", () => {
test("rudp session computes ack masks and clears pending reliable packets", () => {
const sender = new CultNetRudpSession({ connectionId: 7, initialSequence: 10, resendDelayMs: 100 });
const receiver = new CultNetRudpSession({ connectionId: 7, initialSequence: 200, resendDelayMs: 100 });
sender.receive({ packetType: "accept", connectionId: 7, sequence: 1, ack: 0, ackMask: 0, channelId: "control" });
receiver.receive({ packetType: "accept", connectionId: 7, sequence: 2, ack: 0, ackMask: 0, channelId: "control" });
sender.receive(receiver.acceptConnect(sender.createConnect(0), 0), 0);
const first = sender.send("schema", Buffer.from("first"), { reliable: true, ordered: true, nowMs: 0 });
const second = sender.send("schema", Buffer.from("second"), { reliable: true, ordered: true, nowMs: 0 });
const third = sender.send("schema", Buffer.from("third"), { reliable: true, ordered: true, nowMs: 0 });
assert.deepEqual(sender.pendingReliableSequences, [10, 11, 12]);
assert.deepEqual(sender.pendingReliableSequences, [11, 12, 13]);
receiver.receive(first);
receiver.receive(third);
const ackWithGap = receiver.createAck();
assert.equal(ackWithGap.ack, 12);
assert.equal(ackWithGap.ackMask, 0b10 | (1 << 9));
assert.equal(ackWithGap.ack, 13);
assert.equal(ackWithGap.ackMask, 0b110);
sender.receive(ackWithGap);
assert.deepEqual(sender.pendingReliableSequences, [11]);
assert.deepEqual(sender.pendingReliableSequences, [12]);
receiver.receive(second);
const fullAck = receiver.createAck();
assert.equal(fullAck.ack, 12);
assert.equal(fullAck.ackMask, 0b11 | (1 << 9));
assert.equal(fullAck.ack, 13);
assert.equal(fullAck.ackMask, 0b111);
sender.receive(fullAck);
assert.deepEqual(sender.pendingReliableSequences, []);
});
test("rudp session schedules reliable resends until acked", () => {
const session = new CultNetRudpSession({ connectionId: 99, initialSequence: 1, resendDelayMs: 100 });
session.receive({ packetType: "accept", connectionId: 99, sequence: 50, ack: 0, ackMask: 0, channelId: "control" });
session.receive(new CultNetRudpSession({ connectionId: 99, initialSequence: 900 }).acceptConnect(session.createConnect(0), 0), 0);
const sent = session.send("schema", Buffer.from("payload"), { reliable: true, ordered: true, nowMs: 10 });
assert.deepEqual(session.dueResends(90), []);
@ -597,7 +604,7 @@ test("rudp unreliable sequenced delivery is scoped to its channel", () => {
test("rudp rejects unreliable ordered delivery", () => {
const session = new CultNetRudpSession({ connectionId: 196 });
session.receive({ packetType: "accept", connectionId: 196, sequence: 50, ack: 0, ackMask: 0, channelId: "control" });
session.receive(new CultNetRudpSession({ connectionId: 196, initialSequence: 900 }).acceptConnect(session.createConnect(0), 0), 0);
assert.throws(
() => session.send("schema", Buffer.from("cannot order what will not retransmit"), { ordered: true }),
/ordered delivery requires reliability/,
@ -606,7 +613,7 @@ test("rudp rejects unreliable ordered delivery", () => {
test("rudp session bounds pending reliable packets before enqueue", () => {
const session = new CultNetRudpSession({ connectionId: 102, initialSequence: 1, maxPendingReliablePackets: 2 });
session.receive({ packetType: "accept", connectionId: 102, sequence: 50, ack: 0, ackMask: 0, channelId: "control" });
session.receive(new CultNetRudpSession({ connectionId: 102, initialSequence: 900 }).acceptConnect(session.createConnect(0), 0), 0);
session.send("schema", Buffer.from("first"), { reliable: true, ordered: true });
session.send("schema", Buffer.from("second"), { reliable: true, ordered: true });
@ -614,10 +621,10 @@ test("rudp session bounds pending reliable packets before enqueue", () => {
() => session.send("schema", Buffer.from("third"), { reliable: true, ordered: true }),
/reliable send queue is full/,
);
assert.deepEqual(session.pendingReliableSequences, [1, 2]);
assert.deepEqual(session.pendingReliableSequences, [2, 3]);
const fragmented = new CultNetRudpSession({ connectionId: 103, initialSequence: 1, maxPendingReliablePackets: 3 });
fragmented.receive({ packetType: "accept", connectionId: 103, sequence: 50, ack: 0, ackMask: 0, channelId: "control" });
fragmented.receive(new CultNetRudpSession({ connectionId: 103, initialSequence: 900 }).acceptConnect(fragmented.createConnect(0), 0), 0);
assert.throws(
() => fragmented.sendMany("schema", Buffer.from("fragment-me"), { reliable: true, ordered: true, maxFragmentBytes: 3 }),
/reliable send queue is full/,
@ -628,8 +635,7 @@ test("rudp session bounds pending reliable packets before enqueue", () => {
test("rudp session suppresses duplicates and delivers reliable ordered payloads in sequence", () => {
const sender = new CultNetRudpSession({ connectionId: 123, initialSequence: 1 });
const receiver = new CultNetRudpSession({ connectionId: 123, initialSequence: 100 });
sender.receive({ packetType: "accept", connectionId: 123, sequence: 90, ack: 0, ackMask: 0, channelId: "control" });
receiver.receive({ packetType: "accept", connectionId: 123, sequence: 91, ack: 0, ackMask: 0, channelId: "control" });
sender.receive(receiver.acceptConnect(sender.createConnect(0), 0));
const first = sender.send("schema", Buffer.from("first"), { reliable: true, ordered: true });
const second = sender.send("schema", Buffer.from("second"), { reliable: true, ordered: true });
@ -647,8 +653,7 @@ test("rudp session suppresses duplicates and delivers reliable ordered payloads
test("rudp session skips received control packets while ordering schema payloads", () => {
const sender = new CultNetRudpSession({ connectionId: 124, initialSequence: 1 });
const receiver = new CultNetRudpSession({ connectionId: 124, initialSequence: 100 });
sender.receive({ packetType: "accept", connectionId: 124, sequence: 90, ack: 0, ackMask: 0, channelId: "control" });
receiver.receive({ packetType: "accept", connectionId: 124, sequence: 91, ack: 0, ackMask: 0, channelId: "control" });
sender.receive(receiver.acceptConnect(sender.createConnect(0), 0), 0);
const first = sender.send("schema", Buffer.from("first"), { reliable: true, ordered: true });
const control = sender.createAck();
@ -665,8 +670,7 @@ test("rudp session skips received control packets while ordering schema payloads
test("rudp session fragments and reassembles reliable ordered payloads", () => {
const sender = new CultNetRudpSession({ connectionId: 456, initialSequence: 1 });
const receiver = new CultNetRudpSession({ connectionId: 456, initialSequence: 100 });
sender.receive({ packetType: "accept", connectionId: 456, sequence: 90, ack: 0, ackMask: 0, channelId: "control" });
receiver.receive({ packetType: "accept", connectionId: 456, sequence: 91, ack: 0, ackMask: 0, channelId: "control" });
sender.receive(receiver.acceptConnect(sender.createConnect(0), 0), 0);
const packets = sender.sendMany("schema", Buffer.from("fragment-me-please"), {
reliable: true,
@ -1375,8 +1379,7 @@ test("rudp session advances large fragment sets through a bounded reliable windo
const connectionId = 457;
const sender = new CultNetRudpSession({ connectionId, initialSequence: 1, resendDelayMs: 25 });
const receiver = new CultNetRudpSession({ connectionId, initialSequence: 100, resendDelayMs: 25 });
sender.receive({ packetType: "accept", connectionId, sequence: 0, ack: 0, ackMask: 0, channelId: "control" });
receiver.receive({ packetType: "accept", connectionId, sequence: 0, ack: 0, ackMask: 0, channelId: "control" });
sender.receive(receiver.acceptConnect(sender.createConnect(0), 0), 0);
const fragmentCount = CULTNET_RUDP_RELIABLE_SEND_WINDOW_PACKETS + 17;
const payload = Uint8Array.from(
@ -1526,14 +1529,13 @@ test("RUDP operation service correlates typed payload envelopes", async () => {
test("rudp sequence-neutral acknowledgements interoperate with ordered receivers", () => {
const sender = new CultNetRudpSession({ connectionId: 125, initialSequence: 1 });
const receiver = new CultNetRudpSession({ connectionId: 125, initialSequence: 100 });
sender.receive({ packetType: "accept", connectionId: 125, sequence: 90, ack: 0, ackMask: 0, channelId: "control" });
receiver.receive({ packetType: "accept", connectionId: 125, sequence: 91, ack: 0, ackMask: 0, channelId: "control" });
sender.receive(receiver.acceptConnect(sender.createConnect(0), 0), 0);
const ack = sender.createAck();
const request = sender.send("schema", Buffer.from("snapshot"), { reliable: true, ordered: true });
assert.equal(ack.sequence, 0);
assert.equal(request.sequence, 1);
assert.equal(request.sequence, 2);
assert.deepEqual(receiver.receive(ack).delivered, []);
assert.deepEqual(receiver.receive(request).delivered.map(frame => Buffer.from(frame.payload).toString("utf8")), ["snapshot"]);
});
@ -1832,7 +1834,7 @@ test("rudp receiver evicts stranded fragment sets instead of refusing new ones",
// Heimdall fragmented envelopes (max_fragment_bytes = 2048), so on the live
// auth path this map grew for every lost fragment and was never swept.
const receiver = new CultNetRudpSession({ connectionId: 9, initialSequence: 1, resendDelayMs: 50 });
receiver.receive({ packetType: "accept", connectionId: 9, sequence: 1, ack: 0, ackMask: 0, channelId: "control" });
receiver.acceptConnect(new CultNetRudpSession({ connectionId: 9, initialSequence: 1 }).createConnect(0), 0);
// 200 fragment sets that each lose their second fragment.
for (let id = 1; id <= 200; id += 1) {
@ -1976,7 +1978,7 @@ test("rudp socket transport ends the session that refuses a reliable frame and n
// neither in the goodbye nor after it retransmits.
const goodbye = received.find((p) => p.packetType === "disconnect")!;
assert.equal(peer.receive(goodbye, 1).disconnected, true);
assert.ok(peer.pendingReliableSequences.includes(poison.sequence), "the refused frame was acknowledged");
assert.equal(acknowledges(goodbye, poison.sequence), false, "the goodbye acknowledged the refused frame");
const before = received.length;
toServer(poison);
await waitFor(() => server.stats.packetsDropped === 2, "the retransmit dropped");
@ -2353,7 +2355,7 @@ test("a repeated Connect acknowledges what it carries", () => {
const server = new CultNetRudpSession({ connectionId });
const connect = client.createConnect(0);
const accept = server.acceptConnect(connect, 0);
const reply = server.answerRepeatedConnect({ ...connect, ack: accept.sequence }, 1);
const reply = server.acceptConnect({ ...connect, ack: accept.sequence }, 1);
assert.equal(reply.packetType, "ack");
assert.equal(server.outstandingReliablePacketCount, 0);
});
@ -2490,3 +2492,190 @@ test("operation service sweeps a session that ran a handler once it idles out",
await server.close();
}
});
test("a flush started after a reconnect waits on the new session", async () => {
const peerSocket = await bindUdpSocket();
const clientSocket = await bindUdpSocket();
const connectionId = 0x1020305b;
const client = new CultNetRudpSocketTransportConnection({
runtimeId: "rudp-client",
socket: clientSocket,
mode: "client",
remoteHost: "127.0.0.1",
remotePort: udpPort(peerSocket),
connectionId,
resendDelayMs: 1000,
resendPollMs: 1000,
});
const received: CultNetRudpPacket[] = [];
let clientPort = 0;
peerSocket.on("message", (wire, remote) => {
clientPort = remote.port;
received.push(decodeRudpPacket(wire));
});
try {
client.connect();
await waitFor(() => received.some((p) => p.packetType === "connect"), "the Connect");
const first = new CultNetRudpSession({ connectionId });
peerSocket.send(encodeRudpPacket(first.acceptConnect(received[0]!, 0)), clientPort, "127.0.0.1");
await waitFor(() => client.connected, "the client connected");
client.send("schema", Buffer.from("owed to the old session"));
client.checkTimeout(50, Date.now() + 10_000);
assert.equal(client.connected, false);
client.connect();
await waitFor(() => received.filter((p) => p.packetType === "connect").length === 2, "the reconnect");
const second = new CultNetRudpSession({ connectionId });
const reconnect = received.filter((p) => p.packetType === "connect")[1]!;
peerSocket.send(encodeRudpPacket(second.acceptConnect(reconnect, 0)), clientPort, "127.0.0.1");
await waitFor(() => client.connected, "the client reconnected");
await client.flush(500);
} finally {
peerSocket.close();
client.close();
}
});
test("a flush started after the server accepts a new Connect waits on the new session", async () => {
const serverSocket = await bindUdpSocket();
const peerSocket = await bindUdpSocket();
const connectionId = 0x1020305c;
const server = new CultNetRudpSocketTransportConnection({
runtimeId: "rudp-server",
socket: serverSocket,
mode: "server",
connectionId,
resendDelayMs: 1000,
resendPollMs: 1000,
});
const received: CultNetRudpPacket[] = [];
peerSocket.on("message", (wire) => received.push(decodeRudpPacket(wire)));
const toServer = (packet: CultNetRudpPacket) =>
peerSocket.send(encodeRudpPacket(packet), udpPort(serverSocket), "127.0.0.1");
try {
const first = new CultNetRudpSession({ connectionId });
toServer(first.createConnect(0));
await waitFor(() => received.some((p) => p.packetType === "accept"), "the first Accept");
toServer(first.createDisconnect(Buffer.from("bye")));
await waitFor(() => !server.connected, "the first session ended");
const next = new CultNetRudpSession({ connectionId });
toServer(next.createConnect(0));
await waitFor(() => received.filter((p) => p.packetType === "accept").length === 2, "the second Accept");
const accept = received.filter((p) => p.packetType === "accept")[1]!;
next.receive(accept, 0);
toServer(next.createAckForReceived(accept.sequence));
await waitFor(() => server.outstandingReliablePacketCount === 0, "the second Accept acknowledged");
await server.flush(500);
} finally {
peerSocket.close();
server.close();
}
});
test("server-mode transport admits a restarted client and keeps its session when a Connect is retransmitted", async () => {
const serverSocket = await bindUdpSocket();
const peerSocket = await bindUdpSocket();
const connectionId = 0x10203059;
const server = new CultNetRudpSocketTransportConnection({
runtimeId: "rudp-server",
socket: serverSocket,
mode: "server",
connectionId,
resendPollMs: 1000,
});
let received: CultNetRudpPacket[] = [];
peerSocket.on("message", (wire) => received.push(decodeRudpPacket(wire)));
const frames: string[] = [];
server.on("frame", (frame: { payload: Uint8Array }) => frames.push(Buffer.from(frame.payload).toString("utf8")));
const toServer = (packet: CultNetRudpPacket) =>
peerSocket.send(encodeRudpPacket(packet), udpPort(serverSocket), "127.0.0.1");
const reliable = { reliable: true, ordered: true };
try {
const first = new CultNetRudpSession({ connectionId, initialSequence: 50_000 });
const connect = first.createConnect(0, Buffer.from("same"));
toServer(connect);
await waitFor(() => received.some((p) => p.packetType === "accept"), "the first Accept");
first.receive(received.find((p) => p.packetType === "accept")!, 0);
const sent = [0, 1, 2].map((index) => first.send("schema", Buffer.from(`c${index}`), reliable));
sent.forEach(toServer);
await waitFor(() => frames.length === 3, "the first client's frames");
// A retransmitted Connect is a repeat: what was delivered stays delivered once.
toServer(connect);
toServer(sent[0]!);
await new Promise((resolve) => setTimeout(resolve, 100));
assert.deepEqual(frames, ["c0", "c1", "c2"]);
// The same client restarted on the same address, connection id and payload.
received = [];
const second = new CultNetRudpSession({ connectionId, initialSequence: 7 });
toServer(second.createConnect(0, Buffer.from("same")));
await waitFor(() => received.some((p) => p.packetType === "accept"), "the restarted client's Accept");
second.receive(received.find((p) => p.packetType === "accept")!, 0);
toServer(second.send("schema", Buffer.from("after restart"), reliable));
await waitFor(() => frames.includes("after restart"), "the restarted client's frame");
} finally {
peerSocket.close();
server.close();
}
});
test("RUDP operation service admits a restarted client on the same address", async () => {
const server = await startCultNetOperationServer({
runtimeId: "sai-sidecar",
handler: request => ({
schemaVersion: "cultnet.operation_response.v0",
messageId: request.messageId,
serviceId: request.serviceId,
operation: request.operation,
status: "ok",
payloadSchema: "gamecult.eve.plugin_abi.response.v1",
payloadEncoding: "messagepack-base64",
payload: request.payload,
diagnostics: [],
sourceRuntimeId: "sai-sidecar",
}),
});
const connectionId = 0x43554c54;
const port = Number(new URL(server.endpoint).port);
const socket = await bindUdpSocket();
let session = new CultNetRudpSession({ connectionId, initialSequence: 50_000 });
const answered: string[] = [];
socket.on("message", (wire) => {
const packet = decodeRudpPacket(wire);
const result = session.receive(packet, Date.now());
if (packet.packetType === "accept" || packet.reliable) {
socket.send(encodeRudpPacket(session.createAckForReceived(packet.sequence)), port, "127.0.0.1");
}
for (const frame of result.delivered) {
answered.push((decodeMsgpack(frame.payload) as { messageId: string }).messageId);
}
});
const toServer = (packet: CultNetRudpPacket) => socket.send(encodeRudpPacket(packet), port, "127.0.0.1");
const request = (messageId: string) => Buffer.from(encode(encodeCultNetMessageForWire({
schemaVersion: "cultnet.operation_request.v0",
messageId,
serviceId: "sai.vn",
operation: "describe",
payloadSchema: "gamecult.eve.plugin_abi.request.v1",
payloadEncoding: "messagepack-base64",
payload: "gaZzY2hlbWE=",
} satisfies CultNetOperationRequestMessage)));
try {
toServer(session.createConnect(Date.now()));
await waitFor(() => session.connected, "the first Accept");
toServer(session.send("schema", request("before-restart"), { reliable: true, ordered: true }));
await waitFor(() => answered.includes("before-restart"), "the first answer");
session = new CultNetRudpSession({ connectionId, initialSequence: 7 });
toServer(session.createConnect(Date.now()));
await waitFor(() => session.connected, "the restarted client's Accept");
toServer(session.send("schema", request("after-restart"), { reliable: true, ordered: true }));
await waitFor(() => answered.includes("after-restart"), "the restarted client's answer");
} finally {
socket.close();
await server.close();
}
});

535
packages/cultnet-ts/test/rudp-connect-generation.test.ts Normal file
View file

@ -0,0 +1,535 @@
// A Connect's sequence says whether it repeats the one a session accepted, and
// the handshake alone seeds the watermark that orders delivery.
import assert from "node:assert/strict";
import test from "node:test";
import { createSocket, type Socket } from "node:dgram";
import { once } from "node:events";
import {
CultNetRudpSession,
CultNetRudpSocketTransportConnection,
decodeRudpPacket,
encodeRudpPacket,
type CultNetRudpPacket,
} from "../src";
const connectionId = 0x10203080;
const enc = (text: string) => new TextEncoder().encode(text);
const dec = (bytes: Uint8Array) => new TextDecoder().decode(bytes);
const session = (initialSequence: number) => new CultNetRudpSession({ connectionId, initialSequence });
const send = (from: CultNetRudpSession, text: string) => from.send("schema", enc(text), { reliable: true, ordered: true });
const names = (result: { delivered: { payload: Uint8Array }[] }) => result.delivered.map((frame) => dec(frame.payload));
function handshake(client: CultNetRudpSession, server: CultNetRudpSession): { connect: CultNetRudpPacket; accept: CultNetRudpPacket } {
const connect = client.createConnect(0);
const accept = server.acceptConnect(connect, 0);
client.receive(accept, 0);
assert.ok(client.connected && server.connected);
return { connect, accept };
}
test("a reconnect on the same session is not stranded by a write the ended session owed", () => {
const client = session(1);
const server = session(500);
handshake(client, server);
send(server, "lost");
server.createDisconnect(enc("gone"));
client.receive(server.acceptConnect(client.createConnect(1), 1), 1);
assert.deepEqual(names(client.receive(send(server, "after"), 2)), ["after"]);
});
test("a server that accepts a new client forgets the old client's sequences", () => {
const oldClient = session(50);
const server = session(500);
handshake(oldClient, server);
for (const name of ["c1", "c2", "c3"]) {
assert.deepEqual(names(server.receive(send(oldClient, name), 1)), [name]);
}
server.createDisconnect(enc("restart"));
const newClient = session(2);
newClient.receive(server.acceptConnect(newClient.createConnect(2), 2), 2);
assert.deepEqual(names(server.receive(send(newClient, "fresh"), 3)), ["fresh"]);
});
test("a retransmitted Connect after data has flowed does not reset the session", () => {
const client = session(1);
const server = session(500);
const { connect } = handshake(client, server);
const frame = send(client, "once");
assert.deepEqual(names(server.receive(frame, 1)), ["once"]);
assert.equal(server.connectRepeats(connect), true);
assert.equal(server.acceptConnect(connect, 2).packetType, "ack", "the Accept was already acknowledged");
assert.equal(server.outstandingReliablePacketCount, 0, "a repeat queued something");
assert.deepEqual(server.receive(frame, 3).delivered, [], "a retransmitted Connect made the server forget what it had delivered");
});
test("a Connect with another sequence is not a repeat whatever it carries", () => {
const server = session(500);
const connect = session(1).createConnect(0, enc("same payload"));
server.acceptConnect(connect, 0);
assert.equal(server.connectRepeats(connect), true);
assert.equal(server.connectRepeats(session(9).createConnect(0, enc("same payload"))), false);
assert.equal(server.connectRepeats({ ...connect, packetType: "ping" }), false);
server.createDisconnect();
assert.equal(server.connectRepeats(connect), false, "an ended session repeats nothing");
});
test("an Accept that does not name the pending Connect is ignored", () => {
const client = session(10);
const server = session(500);
const connect = client.createConnect(0);
const accept = server.acceptConnect(connect, 0);
client.receive({ ...accept, ack: connect.sequence + 50, ackMask: 0 }, 1);
assert.equal(client.connected, false, "a stale Accept connected the session");
assert.deepEqual(client.pendingReliableSequences, [connect.sequence]);
client.receive({ ...accept, ack: connect.sequence + 3, ackMask: 1 << 2 }, 2);
assert.equal(client.connected, true, "the mask names the Connect");
});
test("an Accept after a local disconnect does not reconnect the session", () => {
const client = session(1);
const server = session(500);
const { accept } = handshake(client, server);
client.createDisconnect(enc("bye"));
assert.equal(client.connected, false);
client.receive(accept, 5);
assert.equal(client.connected, false, "a late Accept revived an ended session");
});
test("a reconnect forgets what the old server sent", () => {
const client = session(1);
const oldServer = session(500);
handshake(client, oldServer);
for (const name of ["d1", "d2", "d3"]) {
assert.deepEqual(names(client.receive(send(oldServer, name), 1)), [name]);
}
const newServer = session(501);
client.receive(newServer.acceptConnect(client.createConnect(2), 2), 2);
assert.deepEqual(names(client.receive(send(newServer, "fresh"), 3)), ["fresh"]);
});
test("an Accept for an abandoned Connect does not connect the session", () => {
const client = session(1);
const server = session(500);
const accept = server.acceptConnect(client.createConnect(0), 0);
client.createDisconnect(enc("never mind"));
client.receive(accept, 1);
assert.equal(client.connected, false, "an Accept revived an abandoned Connect");
});
test("a full queue owed to a vanished peer does not refuse a new generation", () => {
const bounded = (initialSequence: number) => new CultNetRudpSession({ connectionId, initialSequence, maxPendingReliablePackets: 1 });
const server = bounded(500);
server.acceptConnect(session(1).createConnect(0), 0);
assert.equal(server.outstandingReliablePacketCount, 1, "the Accept is owed and never acknowledged");
const next = session(9).createConnect(1);
server.acceptConnect(next, 1);
assert.equal(server.connectRepeats(next), true);
const client = bounded(1);
client.createConnect(0);
client.createConnect(1);
assert.equal(client.outstandingReliablePacketCount, 1, "only the new Connect is owed");
});
test("a duplicate Accept does not reseed the watermark", () => {
const client = session(1);
const server = session(500);
const { accept } = handshake(client, server);
const s1 = send(server, "s1");
const s2 = send(server, "s2");
client.receive({ ...accept, sequence: s2.sequence }, 1);
assert.deepEqual(client.receive(s2, 2).delivered, [], "s1 is still missing");
assert.deepEqual(names(client.receive(s1, 3)), ["s1", "s2"]);
});
test("reliable data before the Accept is neither delivered nor acknowledged", () => {
const client = session(1);
const server = session(500);
const accept = server.acceptConnect(client.createConnect(0), 0);
const a = send(server, "A");
const b = send(server, "B");
for (const early of [b, a]) {
assert.deepEqual(client.receive(early, 1).delivered, []);
const ack = client.createAckForReceived(early.sequence);
assert.deepEqual([ack.ack, ack.ackMask], [0, 0], "an unhandled packet was acknowledged");
}
client.receive(accept, 2);
assert.ok(client.connected);
assert.equal(server.outstandingReliablePacketCount, 3, "nothing was acknowledged");
assert.deepEqual(names(client.receive(a, 3)), ["A"]);
assert.deepEqual(names(client.receive(b, 3)), ["B"]);
});
test("ordered frames after the Accept are delivered in order whatever arrives first", () => {
const client = session(1);
const server = session(500);
handshake(client, server);
const a = send(server, "A");
const b = send(server, "B");
assert.deepEqual(client.receive(b, 1).delivered, []);
assert.deepEqual(names(client.receive(a, 2)), ["A", "B"]);
});
test("an unordered reliable frame is delivered on receipt, never held behind a gap", () => {
const client = session(1);
const server = session(500);
handshake(client, server);
send(server, "lost");
const unordered = server.send("media", enc("now"), { reliable: true, ordered: false });
const ordered = send(server, "later");
assert.deepEqual(names(client.receive(unordered, 1)), ["now"]);
assert.deepEqual(client.receive(ordered, 2).delivered, [], "the ordered frame waits for the gap");
});
test("a peer Disconnect does not forget what the peer sent", () => {
const client = session(1);
const server = session(500);
handshake(client, server);
const s1 = send(client, "s1");
const s2 = send(client, "s2");
assert.deepEqual(server.receive(s2, 1).delivered, [], "s1 is missing, so s2 is held");
server.receive(client.createDisconnect(enc("bye")), 2);
assert.deepEqual(names(server.receive(s1, 3)), ["s1", "s2"], "the Disconnect forgot the held frame");
assert.deepEqual(server.receive(s2, 4).delivered, [], "the Disconnect forgot what was received");
});
test("sessions from one options object each draw their own initial sequence from one to two to the thirty first", () => {
const options = { connectionId };
const firsts = new Set(Array.from({ length: 64 }, () => new CultNetRudpSession(options).createConnect(0).sequence));
assert.ok(firsts.size > 32, `a default session does not draw its own sequence: ${[...firsts]}`);
assert.ok([...firsts].every((value) => value >= 1 && value < 2 ** 31), `left [1, 2^31): ${[...firsts]}`);
});
// Ack Cut 1d: a frame from an earlier generation is a duplicate, a delayed
// Connect from an earlier attempt is stale, a Connect from a new endpoint is a
// new client.
const namesSequence = (ack: CultNetRudpPacket, sequence: number) =>
ack.ack === sequence
|| Array.from({ length: 32 }, (_, bit) => bit).some(
(bit) => (ack.ackMask & (1 << bit)) !== 0 && ack.ack - bit - 1 === sequence,
);
test("a frame from the client's earlier generation is acknowledged, not delivered again", () => {
const client = session(10);
const server = session(500);
handshake(client, server);
const old = send(client, "old");
assert.deepEqual(names(server.receive(old, 1)), ["old"]);
const connect = client.createConnect(2);
client.receive(server.acceptConnect(connect, 2), 2);
assert.ok(old.sequence < connect.sequence);
assert.deepEqual(server.receive(old, 3).delivered, [], "an earlier generation's frame was delivered again");
assert.ok(namesSequence(server.createAckForReceived(old.sequence), old.sequence), "the duplicate was not acknowledged");
assert.deepEqual(names(server.receive(send(client, "fresh"), 4)), ["fresh"]);
});
test("a frame from the server's earlier generation is acknowledged, not delivered again", () => {
const client = session(10);
const server = session(500);
handshake(client, server);
const old = send(server, "old");
assert.deepEqual(names(client.receive(old, 1)), ["old"]);
client.receive(server.acceptConnect(client.createConnect(2), 2), 2);
assert.deepEqual(client.receive(old, 3).delivered, [], "an earlier generation's frame was delivered again");
assert.ok(namesSequence(client.createAckForReceived(old.sequence), old.sequence));
assert.deepEqual(names(client.receive(send(server, "fresh"), 4)), ["fresh"]);
});
test("a duplicate below the receive window is acknowledged by name", () => {
const client = session(10);
const server = session(500);
handshake(client, server);
const first = send(client, "first");
assert.deepEqual(names(server.receive(first, 1)), ["first"]);
for (let index = 0; index < 4_200; index += 1) {
server.receive(send(client, "x"), 1);
client.receive(server.createAck(), 1);
}
assert.deepEqual(server.receive(first, 2).delivered, []);
assert.equal(server.createAckForReceived(first.sequence).ack, first.sequence, "a duplicate below the window was not acknowledged by name");
});
test("a delayed Connect from an earlier attempt does not strand the client", () => {
const client = session(10);
const server = session(500);
const earlier = client.createConnect(0);
const retried = client.createConnect(300);
const accept = server.acceptConnect(retried, 301);
client.receive(accept, 302);
server.receive(client.createAckForReceived(accept.sequence), 302);
assert.ok(client.connected);
const a = send(client, "a");
assert.deepEqual(names(server.receive(a, 303)), ["a"]);
client.receive(server.createAckForReceived(a.sequence), 303);
assert.equal(server.connectRepeats(earlier), true, "the delayed Connect starts nothing");
const reply = server.acceptConnect(earlier, 304);
assert.equal(reply.packetType, "ack");
client.receive(reply, 305);
assert.ok(client.connected);
const b = send(client, "b");
assert.deepEqual(names(server.receive(b, 306)), ["b"], "the delayed Connect reset the server");
client.receive(server.createAckForReceived(b.sequence), 307);
assert.ok(!client.pendingReliableSequences.includes(b.sequence));
});
test("a repeated Connect refreshes liveness and a stale one does not", () => {
const client = session(10);
const server = session(500);
const earlier = client.createConnect(0);
const current = client.createConnect(1);
server.acceptConnect(current, 0);
server.acceptConnect(current, 900);
assert.equal(server.checkTimeout(1_000, 500), false, "a repeated Connect did not refresh liveness");
server.acceptConnect(earlier, 1_400);
assert.equal(server.checkTimeout(1_600, 500), true, "a stale Connect refreshed liveness");
});
test("stale Connects are recognised by serial arithmetic", () => {
const server = session(500);
server.acceptConnect(session(3).createConnect(0), 0);
const connect = (sequence: number) => session(sequence).createConnect(0);
assert.equal(server.connectRepeats(connect(2 ** 32 - 2)), true, "just before the wrap");
assert.equal(server.connectRepeats(connect(4)), false, "just after");
assert.equal(server.connectRepeats(connect(3 + 4_096)), false, "far after");
assert.equal(server.connectRepeats(connect(2 ** 32 - 4_092)), true, "the window's edge");
assert.equal(server.connectRepeats(connect(2 ** 32 - 4_093)), false, "past the window");
});
test("a restarted client whose first sequence is stale connects after redrawing", () => {
const server = session(500);
const old = session(1_000);
const oldAccept = server.acceptConnect(old.createConnect(0), 0);
old.receive(oldAccept, 0);
const restarted = session(900);
const first = restarted.createConnect(0, enc("join"));
// The old client has not acknowledged its Accept yet, so that is the reply:
// it names the old Connect, not this one.
const early = server.acceptConnect(first, 1);
assert.equal(early.packetType, "accept");
restarted.receive(early, 1);
assert.equal(restarted.connected, false, "an Accept for another Connect connected the client");
server.receive(old.createAckForReceived(oldAccept.sequence), 1);
const reply = server.acceptConnect(first, 2);
assert.equal(reply.packetType, "ack");
restarted.receive(reply, 2);
assert.equal(restarted.connected, false);
const retransmitted = restarted.dueResends(1_000);
assert.equal(retransmitted.length, 1);
assert.equal(retransmitted[0]!.sequence, first.sequence, "the attempt is still young");
const fresh = restarted.dueResends(3_500);
assert.equal(fresh.length, 1);
assert.equal(fresh[0]!.packetType, "connect");
assert.notEqual(fresh[0]!.sequence, first.sequence, "the same Connect was retransmitted for ever");
assert.equal(dec(fresh[0]!.payload ?? new Uint8Array()), "join", "the fresh attempt lost the Connect payload");
const accept = server.acceptConnect(fresh[0]!, 3_500);
assert.equal(accept.packetType, "accept");
restarted.receive(accept, 3_500);
assert.ok(restarted.connected);
assert.deepEqual(names(server.receive(send(restarted, "hello"), 3_501)), ["hello"]);
});
test("an answered Connect is never replaced by the attempt timeout", () => {
const client = session(10);
const server = session(500);
const { connect } = handshake(client, server);
const resent = client.dueResends(60_000);
assert.ok(resent.every((packet) => packet.packetType !== "connect" || packet.sequence === connect.sequence), "a connected client started a new attempt");
assert.ok(client.connected);
});
async function bind(): Promise<Socket> {
const socket = createSocket("udp4");
socket.bind(0, "127.0.0.1");
await once(socket, "listening");
return socket;
}
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
test("server mode admits a pinned client that restarts on a new port", async () => {
const serverSocket = await bind();
const a = await bind();
const b = await bind();
const server = new CultNetRudpSocketTransportConnection({
runtimeId: "srv", socket: serverSocket, mode: "server", connectionId, resendPollMs: 20, resendDelayMs: 20,
});
const frames: string[] = [];
server.on("frame", (frame: { payload: Uint8Array }) => frames.push(dec(frame.payload)));
const port = (serverSocket.address() as { port: number }).port;
const atA: CultNetRudpPacket[] = [];
const atB: CultNetRudpPacket[] = [];
a.on("message", (wire) => atA.push(decodeRudpPacket(wire)));
b.on("message", (wire) => atB.push(decodeRudpPacket(wire)));
try {
const first = session(1);
a.send(encodeRudpPacket(first.createConnect(0, enc("join"))), port, "127.0.0.1");
await sleep(100);
const accept = atA.find((packet) => packet.packetType === "accept");
assert.ok(accept);
first.receive(accept, 0);
a.send(encodeRudpPacket(first.createAckForReceived(accept.sequence)), port, "127.0.0.1");
a.send(encodeRudpPacket(send(first, "hello")), port, "127.0.0.1");
await sleep(100);
assert.deepEqual(frames, ["hello"]);
const second = session(1); // the process restarted on a new port, pinned
b.send(encodeRudpPacket(second.createConnect(0, enc("join"))), port, "127.0.0.1");
await sleep(100);
const acceptB = atB.find((packet) => packet.packetType === "accept");
assert.ok(acceptB, `the restarted client on a new port was not admitted: ${JSON.stringify(atB.map((packet) => packet.packetType))}`);
second.receive(acceptB, 1);
assert.ok(second.connected);
b.send(encodeRudpPacket(send(second, "after")), port, "127.0.0.1");
await sleep(100);
assert.deepEqual(frames, ["hello", "after"]);
} finally {
server.close();
a.close();
b.close();
}
});
test("an accepted peer that goes silent times out", () => {
const server = session(500);
server.acceptConnect(session(10).createConnect(0), 0);
assert.equal(server.checkTimeout(100_000, 1_000), true);
});
// Ack Cut 1d, batch 2: the connect-attempt timeout.
test("an Ack naming the pending Connect does not retire it", () => {
const server = session(500);
const old = session(1);
const oldAccept = server.acceptConnect(old.createConnect(0), 0);
old.receive(oldAccept, 0);
server.receive(old.createAckForReceived(oldAccept.sequence), 0);
const restarted = session(1);
const first = restarted.createConnect(0, enc("join"));
const reply = server.acceptConnect(first, 1);
assert.equal(reply.packetType, "ack");
assert.ok(namesSequence(reply, first.sequence), "the Ack must name the pending Connect for this test to bite");
restarted.receive(reply, 1);
assert.equal(restarted.connected, false);
assert.deepEqual(restarted.pendingReliableSequences, [first.sequence], "an Ack retired the Connect");
const fresh = restarted.dueResends(3_000);
assert.equal(fresh.length, 1, "the client waits for ever with nothing to resend");
assert.equal(dec(fresh[0]!.payload ?? new Uint8Array()), "join");
const accept = server.acceptConnect(fresh[0]!, 3_000);
assert.equal(accept.packetType, "accept");
restarted.receive(accept, 3_000);
assert.ok(restarted.connected);
});
test("a late copy of an abandoned Connect is stale once its replacement is accepted", () => {
const server = session(500);
const client = session(10);
const abandoned = client.createConnect(0);
const fresh = client.dueResends(3_000);
assert.equal(fresh.length, 1);
assert.equal(fresh[0]!.sequence, abandoned.sequence + 4_095);
const accept = server.acceptConnect(fresh[0]!, 3_001);
client.receive(accept, 3_002);
server.receive(client.createAckForReceived(accept.sequence), 3_002);
assert.ok(client.connected);
const a = send(client, "a");
assert.deepEqual(names(server.receive(a, 3_003)), ["a"]);
client.receive(server.createAckForReceived(a.sequence), 3_003);
assert.equal(server.connectRepeats(abandoned), true, "the late copy would restart the server");
const reply = server.acceptConnect(abandoned, 3_004);
assert.equal(reply.packetType, "ack");
client.receive(reply, 3_005);
assert.ok(client.connected);
assert.deepEqual(names(server.receive(send(client, "b"), 3_006)), ["b"]);
});
test("a server sitting at the jump is left by the next attempt", () => {
const server = session(500);
const old = session(5_095);
const oldAccept = server.acceptConnect(old.createConnect(0), 0);
old.receive(oldAccept, 0);
server.receive(old.createAckForReceived(oldAccept.sequence), 0);
const restarted = session(1_000);
const first = restarted.createConnect(0, enc("join"));
restarted.receive(server.acceptConnect(first, 1), 1);
assert.equal(restarted.connected, false);
const second = restarted.dueResends(3_000)[0]!;
assert.equal(second.sequence, 5_095, "the jump lands on the server's generation");
const reply = server.acceptConnect(second, 3_001);
assert.equal(reply.packetType, "ack");
restarted.receive(reply, 3_001);
assert.equal(restarted.connected, false);
const third = restarted.dueResends(6_000)[0]!;
assert.equal(third.sequence, 5_095 + 4_095);
const accept = server.acceptConnect(third, 6_001);
assert.equal(accept.packetType, "accept");
restarted.receive(accept, 6_001);
assert.ok(restarted.connected);
});
test("server mode admits a pinned client that restarts on the same address", async () => {
const serverSocket = await bind();
const peer = await bind();
const server = new CultNetRudpSocketTransportConnection({
runtimeId: "srv", socket: serverSocket, mode: "server", connectionId, resendPollMs: 20, resendDelayMs: 20,
});
const port = (serverSocket.address() as { port: number }).port;
const received: CultNetRudpPacket[] = [];
peer.on("message", (wire) => received.push(decodeRudpPacket(wire)));
const take = async (type: string) => {
for (let index = 0; index < 50; index += 1) {
const found = received.findIndex((packet) => packet.packetType === type);
if (found >= 0) return received.splice(found, 1)[0]!;
await sleep(10);
}
throw new Error(`no ${type} arrived`);
};
try {
const first = session(1);
peer.send(encodeRudpPacket(first.createConnect(0, enc("join"))), port, "127.0.0.1");
const accept = await take("accept");
first.receive(accept, 0);
peer.send(encodeRudpPacket(first.createAckForReceived(accept.sequence)), port, "127.0.0.1");
await sleep(50);
received.length = 0;
const restarted = session(1);
peer.send(encodeRudpPacket(restarted.createConnect(0, enc("join"))), port, "127.0.0.1");
restarted.receive(await take("ack"), 1);
assert.equal(restarted.connected, false);
peer.send(encodeRudpPacket(restarted.dueResends(3_000)[0]!), port, "127.0.0.1");
restarted.receive(await take("accept"), 3_001);
assert.ok(restarted.connected, "the restarted pinned client was never admitted");
} finally {
server.close();
peer.close();
}
});

110
packages/cultnet-ts/test/rudp-ordered-delivery.test.ts Normal file
View file

@ -0,0 +1,110 @@
// Ordered delivery has one owner: the contiguous received-through watermark. A
// frame is held for exactly one reason, a reliable sequence below it has not
// arrived, and it is delivered in the call that fills the last such gap,
// whichever channel the gap belonged to.
import assert from "node:assert/strict";
import test from "node:test";
import { CultNetRudpSession } from "../src";
const enc = (text: string) => new TextEncoder().encode(text);
const dec = (bytes: Uint8Array) => new TextDecoder().decode(bytes);
function handshake(): { client: CultNetRudpSession; server: CultNetRudpSession } {
const client = new CultNetRudpSession({ connectionId: 410, initialSequence: 1 });
const server = new CultNetRudpSession({ connectionId: 410, initialSequence: 500 });
const accept = server.acceptConnect(client.createConnect(0), 0);
client.receive(accept, 0);
return { client, server };
}
const names = (result: { delivered: { payload: Uint8Array }[] }) => result.delivered.map((frame) => dec(frame.payload));
const send = (session: CultNetRudpSession, channel: string, payload: string, ordered: boolean) =>
session.send(channel, enc(payload), { reliable: true, ordered });
test("an ordered frame waits for a gap filled by another channel", () => {
const { client: sender, server: receiver } = handshake();
const o1 = send(sender, "schema", "o1", true);
const u = send(sender, "rel", "u", false);
const o2 = send(sender, "schema", "o2", true);
const o3 = send(sender, "schema", "o3", true);
const o4 = send(sender, "schema", "o4", true);
assert.deepEqual(names(receiver.receive(o1, 1)), ["o1"]);
assert.deepEqual(names(receiver.receive(o2, 2)), [], "u is missing");
assert.deepEqual(names(receiver.receive(o3, 3)), [], "u is missing");
assert.deepEqual(names(receiver.receive(o4, 4)), [], "u is missing");
assert.deepEqual(names(receiver.receive(u, 5)), ["u", "o2", "o3", "o4"]);
});
test("two ordered channels release each other in sequence order", () => {
const { client: sender, server: receiver } = handshake();
const a1 = send(sender, "schema", "A1", true);
const b1 = send(sender, "other", "B1", true);
const a2 = send(sender, "schema", "A2", true);
const a3 = send(sender, "schema", "A3", true);
const delivered = [
...names(receiver.receive(a1, 1)),
...names(receiver.receive(a2, 2)),
...names(receiver.receive(a3, 3)),
];
assert.deepEqual(delivered, ["A1"], "B1 is missing, so A2 and A3 wait");
assert.deepEqual(names(receiver.receive(b1, 4)), ["B1", "A2", "A3"]);
});
test("a channel first used after other traffic loses nothing", () => {
const { client: sender, server: receiver } = handshake();
const c1 = send(sender, "late", "C1", true);
const c2 = send(sender, "late", "C2", true);
const x = send(sender, "schema", "X", true);
assert.deepEqual(names(receiver.receive(x, 1)), [], "C1 is missing");
assert.deepEqual(names(receiver.receive(c2, 2)), [], "C1 is missing");
assert.deepEqual(names(receiver.receive(c1, 3)), ["C1", "C2", "X"]);
});
test("Accept seeds the watermark of the connecting side", () => {
const { client, server } = handshake();
const s1 = send(server, "schema", "s1", true);
const s2 = send(server, "schema", "s2", true);
assert.deepEqual(names(client.receive(s2, 1)), []);
assert.deepEqual(names(client.receive(s1, 2)), ["s1", "s2"]);
});
test("Connect seeds the watermark of the accepting side", () => {
const { client, server } = handshake();
const first = send(client, "schema", "first", true);
const second = send(client, "schema", "second", true);
assert.deepEqual(names(server.receive(second, 1)), []);
assert.deepEqual(names(server.receive(first, 2)), ["first", "second"]);
});
test("a duplicate of a held frame is not delivered twice", () => {
const { client: sender, server: receiver } = handshake();
const s1 = send(sender, "schema", "s1", true);
const s2 = send(sender, "schema", "s2", true);
assert.deepEqual(names(receiver.receive(s2, 1)), []);
assert.deepEqual(names(receiver.receive(s2, 2)), []);
assert.deepEqual(names(receiver.receive(s1, 3)), ["s1", "s2"]);
assert.deepEqual(names(receiver.receive(s2, 4)), []);
});
test("resetting peer state forgets held frames and reseeds the watermark from the next handshake", () => {
const { client: sender, server: receiver } = handshake();
send(sender, "schema", "s1", true);
const s2 = send(sender, "schema", "s2", true);
assert.deepEqual(names(receiver.receive(s2, 1)), []);
receiver.resetPeerState();
const next = new CultNetRudpSession({ connectionId: 410, initialSequence: 100 });
next.receive(receiver.acceptConnect(next.createConnect(0), 2), 2);
const d1 = send(next, "schema", "d1", true);
const d2 = send(next, "schema", "d2", true);
// The held s2 died with the reset, and the watermark follows the new peer.
assert.deepEqual(names(receiver.receive(d2, 3)), []);
assert.deepEqual(names(receiver.receive(d1, 4)), ["d1", "d2"]);
});

72
packages/cultnet-ts/test/rudp-session-ending.test.ts Normal file
View file

@ -0,0 +1,72 @@
// A write belongs to the session generation it was issued in. Every way a
// generation ends drops the writes it still owed, so none is retransmitted
// into a later session where the new peer would deliver it.
import assert from "node:assert/strict";
import test from "node:test";
import { CultNetRudpSession } from "../src";
const connectionId = 0x10203070;
const enc = (text: string) => new TextEncoder().encode(text);
function clientWithALostWrite(): { client: CultNetRudpSession; server: CultNetRudpSession } {
const client = new CultNetRudpSession({ connectionId, initialSequence: 1 });
const server = new CultNetRudpSession({ connectionId, initialSequence: 500 });
client.receive(server.acceptConnect(client.createConnect(0), 0), 0);
client.send("schema", enc("owed to the old session"), { reliable: true, ordered: true });
// A fragmented write larger than the send window leaves part of it queued.
client.sendMany("schema", new Uint8Array(40 * 8), { reliable: true, ordered: true, maxFragmentBytes: 8 });
assert.equal(client.outstandingReliablePacketCount, 41);
assert.ok(client.queuedReliablePacketCount > 0);
return { client, server };
}
const endings: Record<string, (client: CultNetRudpSession, server: CultNetRudpSession) => void> = {
"peer Disconnect": (client, server) => {
client.receive(server.createDisconnect(enc("bye")), 1);
},
"local disconnect": (client) => {
client.createDisconnect(enc("bye"));
},
timeout: (client) => {
assert.equal(client.checkTimeout(1_000, 10), true);
},
reset: (client) => {
client.resetPeerState();
},
};
for (const [name, end] of Object.entries(endings)) {
test(`a ${name} drops the writes the session owed`, () => {
const { client, server } = clientWithALostWrite();
end(client, server);
assert.equal(client.outstandingReliablePacketCount, 0, "the write survived the end");
const nextServer = new CultNetRudpSession({ connectionId, initialSequence: 900 });
nextServer.acceptConnect(client.createConnect(2_000), 2_000);
for (const resend of client.dueResends(60_000)) {
assert.deepEqual(
nextServer.receive(resend, 60_000).delivered,
[],
"the old session's write was delivered in the next session",
);
}
});
}
test("an ending does not forget what was received from the peer", () => {
const client = new CultNetRudpSession({ connectionId, initialSequence: 1 });
const server = new CultNetRudpSession({ connectionId, initialSequence: 500 });
client.receive(server.acceptConnect(client.createConnect(0), 0), 0);
const s1 = client.send("schema", enc("s1"), { reliable: true, ordered: true });
const s2 = client.send("schema", enc("s2"), { reliable: true, ordered: true });
assert.deepEqual(server.receive(s2, 1).delivered, [], "s1 is missing, so s2 is held");
assert.equal(server.checkTimeout(1_000, 10), true);
assert.deepEqual(
server.receive(s1, 1_001).delivered.map((frame) => new TextDecoder().decode(frame.payload)),
["s1", "s2"],
"the ending forgot the held frame",
);
assert.deepEqual(server.receive(s2, 1_002).delivered, [], "the ending forgot what was received");
});

11
src/GameCult.Caching.MessagePack/CultMathJson.cs
View file

@ -14,7 +14,7 @@ namespace GameCult.Caching.MessagePack;
/// The canonical System.Text.Json shape of every public CultMath value type, the JSON twin of
/// <see cref="CultMathResolver"/>. Vectors, quaternions and colors are objects keyed by their
/// field names (<c>{"x":1,"y":2}</c>), matrices are arrays of row objects, <c>rect</c> is
/// <c>{"min":..,"max":..}</c>, and <c>CultCellular</c> is <c>{"nearest":..,"edge":..,"id":..}</c>.
/// <c>{"min":..,"max":..}</c>, <c>CultCellular</c> is <c>{"nearest":..,"edge":..,"id":..}</c>, and <c>CultPhasor</c> is <c>{"cos":..,"sin":..}</c>.
/// Property names are fixed: <c>PropertyNamingPolicy</c> does not rename them, and
/// <c>PropertyNameCaseInsensitive</c> makes reading case-insensitive.
/// <para>
@ -481,7 +481,7 @@ public static class CultMathJson
Add<Random, uint>(UIntScalar.Instance, new Fields<uint>(("state", Val<uint>())),
a => new Random { state = a[0] }, (v, a) => a[0] = v.state),
// Compositions: matrices are arrays of row objects, rect is {min, max}, CultCellular is {nearest, edge, id}.
// Compositions: matrices are arrays of row objects, rect is {min, max}, CultCellular is {nearest, edge, id}, CultPhasor is {cos, sin}.
Add<float2x2, float>(f, new Rows<float>(2, Xy<float>()),
a => new float2x2(a[0], a[1], a[2], a[3]),
(v, a) => { a[0] = v[0].x; a[1] = v[0].y; a[2] = v[1].x; a[3] = v[1].y; }),
@ -507,6 +507,13 @@ public static class CultMathJson
a[4] = v.edge.x; a[5] = v.edge.y; a[6] = v.edge.z; a[7] = v.edge.w;
a[8] = v.id;
}),
Add<CultPhasor, float>(f, new Fields<float>(("cos", Xyzw<float>()), ("sin", Xyzw<float>())),
a => new CultPhasor(new float4(a[0], a[1], a[2], a[3]), new float4(a[4], a[5], a[6], a[7])),
(v, a) =>
{
a[0] = v.cos.x; a[1] = v.cos.y; a[2] = v.cos.z; a[3] = v.cos.w;
a[4] = v.sin.x; a[5] = v.sin.y; a[6] = v.sin.z; a[7] = v.sin.w;
}),
};
var map = new Dictionary<Type, JsonConverter>();

17
src/GameCult.Caching.MessagePack/CultMathResolver.cs
View file

@ -180,7 +180,7 @@ public sealed class CultMathResolver : IFormatterResolver
return value;
}),
// Compositions: matrices are rows, rect is [min, max], CultCellular is [nearest, edge, id].
// Compositions: matrices are rows, rect is [min, max], CultCellular is [nearest, edge, id], CultPhasor is [cos, sin].
Shape<float2x2>((ref MessagePackWriter w, float2x2 v) => { w.WriteArrayHeader(2); PutFloat2(ref w, v[0]); PutFloat2(ref w, v[1]); },
(ref MessagePackReader r) =>
{
@ -221,6 +221,21 @@ public sealed class CultMathResolver : IFormatterResolver
}
return value;
}),
Shape<CultPhasor>((ref MessagePackWriter w, CultPhasor v) => { w.WriteArrayHeader(2); PutFloat4(ref w, v.cos); PutFloat4(ref w, v.sin); },
(ref MessagePackReader r) =>
{
var value = default(CultPhasor);
for (int i = 0, n = r.ReadArrayHeader(); i < n; i++)
{
switch (i)
{
case 0: value.cos = GetFloat4(ref r); break;
case 1: value.sin = GetFloat4(ref r); break;
default: r.Skip(); break;
}
}
return value;
}),
};
var map = new Dictionary<Type, object>();

1
src/GameCult.Caching.MessagePack/README.md
View file

@ -93,6 +93,7 @@ registered ahead of these wins.
| `float2x2`, `float3x3` | array of row vectors (row-major) | array of row objects |
| `rect` | `[min, max]` of `float2` | `{"min":{..},"max":{..}}` |
| `CultCellular` | `[nearest, edge, id]` | `{"nearest":{..},"edge":{..},"id":..}` |
| `CultPhasor` | `[cos, sin]` of `float4` | `{"cos":{..},"sin":{..}}` |
Floats are MessagePack float32 (`0xCA`), doubles float64 (`0xCB`), integers the
shortest MessagePack integer.

22
src/GameCult.Caching/Contracts/cultcache-store-composition.md
View file

@ -47,10 +47,20 @@ which route wrote it.
- Changes published through `Watch`/`WatchRecord` carry a `Sequence`: a
per-cache, in-memory number assigned when the cache admits the change, under
its gate. It increases in admission order and is not persisted.
`OnUpdate(previous, current)` carries none. Streams CultNet derives from
`OnUpdate` (`CultNetDatabase.WatchAllChanges`, the subscription server,
database-backed Mesh handles) have no ordering guarantee and no stale
protection.
`OnUpdate(previous, current)` carries none. CultNet derives its streams
(`CultNetDatabase.WatchAllChanges`, the subscription server, database-backed
Mesh handles) from `Watch` plus one journal (`AddJournal`), so a load, a
commit and a variant dependent's re-resolution all reach subscribers and the
mutation log. The journal runs under the gate in `Sequence` order, so the
shard log is written in cache order; the `Watch` observer publishes what the
journal decided. A database write door only hands the journal the wire message
or replicated entry for the change it admitted. A replica publishes but never
mints log sequences. A primary logs every change it admits, loads and pulls
included; a store attached after the database exists has its hydration logged
too. A primary that cannot log a change compacts through that change's
sequence and throws `CultNetShardLogException` to the writer after publication:
the commit stands, and a replica behind it resynchronizes from a snapshot. The
streams carry no `Sequence` and have no stale protection.
- `GetWithSequence(key)` returns the document and the cache's current
`Sequence`, read together under the gate: every change with a `Sequence` at
or below it is reflected in the document.
@ -126,6 +136,10 @@ which route wrote it.
needs the cache or its gate: another thread's write, a lock a writer holds.
Its own leaf locks, taken only inside the journal, are the only locks that may
nest under the gate.
- A shard mutation log store (`ICultNetShardMutationLogStore`) runs inside the
CultNet journal, under the cache's gate: it must not call the cache or wait
for a thread that does. A store that writes its own cache throws through the
journal's reentry guard, and the admission stands.
- A thread holds one cache's gate at a time: entering a cache's hold inside
another cache's hold throws.
- A store doing I/O on behalf of its cache (pull, flush, commit) holds the gate,

13
src/GameCult.Caching/CultCache.cs
View file

@ -2124,7 +2124,15 @@ namespace GameCult.Caching
Remove(handle.Key);
}
// false: a condition failed; nothing was written, changed in memory, or published.
/// <summary>
/// Commits the staged batch. Returns false when a condition failed: nothing was written, changed in memory, or published.
/// </summary>
/// <remarks>
/// A commit that succeeds and then fails in a journal or observer still stands and is published, and the writer
/// receives the failure. One failure arrives as itself, several as an <see cref="AggregateException"/>. A writer that
/// wants a particular failure (a CultNetShardLogException, when a database's log refused the commit) catches that type,
/// or unwraps <see cref="AggregateException.InnerExceptions"/> and looks for it.
/// </remarks>
public bool Commit(Action<CultCacheBatch> stage)
{
return Land(stage, wait: true) == CultCommitOutcome.Committed;
@ -2427,7 +2435,8 @@ namespace GameCult.Caching
// run under the gate, and a write an observer makes is a new outermost hold. Cross-thread delivery order is not
// guaranteed; each change carries the Sequence it was admitted with. Journals run before that, under the gate,
// in Sequence order (AddJournal); no hold may be entered on this cache while one runs. Every change reaches every observer even
// if an observer or an OnUpdate handler throws; then the first exception (an AggregateException for several) is rethrown.
// if an observer or an OnUpdate handler throws; then the exception (an AggregateException for several, whatever their
// origin: a journal, an observer, a second database) is rethrown. A writer looking for one type unwraps InnerExceptions.
// If the body threw, its exception wins.
internal T Held<T>(Func<T> body)
{

18
src/GameCult.Geometry/CultGeometryDocuments.cs
View file

@ -153,8 +153,8 @@ namespace GameCult.Geometry
public string RequestId { get; set; } = string.Empty;
[Key(1)]
[CultReference(typeof(CultGeometryDomainDocument))]
public string DomainKey { get; set; } = string.Empty;
[MessagePackFormatter(typeof(CultRecordRefFormatter<CultGeometryDomainDocument>))]
public CultRecordRef<CultGeometryDomainDocument> DomainKey { get; set; }
[Key(2)]
[CultIndex]
@ -200,7 +200,7 @@ namespace GameCult.Geometry
{
if (request == null) throw new ArgumentNullException(nameof(request));
return new CultRecordKey("geometry:request:" + StableHash(
request.DomainKey,
request.DomainKey.Key.Value,
request.WorkerGroup,
StableVector(request.CameraPosition),
StableVector(request.FrustumMin),
@ -228,8 +228,8 @@ namespace GameCult.Geometry
public string CutId { get; set; } = string.Empty;
[Key(1)]
[CultReference(typeof(CultGeometryBuildRequest))]
public string RequestKey { get; set; } = string.Empty;
[MessagePackFormatter(typeof(CultRecordRefFormatter<CultGeometryBuildRequest>))]
public CultRecordRef<CultGeometryBuildRequest> RequestKey { get; set; }
[Key(2)]
public string[] SelectedNodes { get; set; } = Array.Empty<string>();
@ -247,7 +247,7 @@ namespace GameCult.Geometry
{
if (cut == null) throw new ArgumentNullException(nameof(cut));
return new CultRecordKey("geometry:cut:" + StableHash(
cut.RequestKey,
cut.RequestKey.Key.Value,
cut.CutId,
StableArray(cut.SelectedNodes),
StableArray(cut.DeferredChildRequests),
@ -313,8 +313,8 @@ namespace GameCult.Geometry
public string ChunkId { get; set; } = string.Empty;
[Key(1)]
[CultReference(typeof(CultGeometrySelectedCutManifest))]
public string CutKey { get; set; } = string.Empty;
[MessagePackFormatter(typeof(CultRecordRefFormatter<CultGeometrySelectedCutManifest>))]
public CultRecordRef<CultGeometrySelectedCutManifest> CutKey { get; set; }
[Key(2)]
[CultIndex]
@ -357,7 +357,7 @@ namespace GameCult.Geometry
{
if (chunk == null) throw new ArgumentNullException(nameof(chunk));
return new CultRecordKey("geometry:chunk:" + StableHash(
chunk.CutKey,
chunk.CutKey.Key.Value,
chunk.ChunkId,
chunk.SelectedCutId,
StableArray(chunk.SourceDomainKeys),

1
src/GameCult.Geometry/GameCult.Geometry.csproj
View file

@ -10,6 +10,7 @@
<ItemGroup>
<ProjectReference Include="..\..\packages\cultmath\src\CultMath\CultMath.csproj" />
<ProjectReference Include="..\GameCult.Caching\GameCult.Caching.csproj" />
<ProjectReference Include="..\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<PackageReference Include="MessagePack" Version="3.1.7" />
<PackageReference Include="System.Text.Json" Version="9.0.10" />
</ItemGroup>

7
src/GameCult.Mesh/CultMesh.cs
View file

@ -543,8 +543,8 @@ namespace GameCult.Mesh
public int BindPort { get; set; }
/// <summary>Gets or sets a caller-owned bound socket.</summary>
public Socket? Socket { get; set; }
/// <summary>Gets or sets the first local packet sequence.</summary>
public uint InitialSequence { get; set; } = 1;
/// <summary>Gets or sets the first local packet sequence. Unset, each session draws its own at random.</summary>
public uint? InitialSequence { get; set; }
/// <summary>Gets or sets the reliable resend delay in milliseconds.</summary>
public long ResendDelayMs { get; set; } = 250;
/// <summary>Gets or sets the advertised transport id.</summary>
@ -3379,7 +3379,8 @@ namespace GameCult.Mesh
if (document != null)
return document;
var untyped = database.Cache.Get(key);
// Through the database, which serves only what its shards own; the cache behind it may hold more.
var untyped = await database.GetAsync<object>(key).ConfigureAwait(false);
if (untyped != null && IsSameCultDocumentSchema<TDocument>(untyped.GetType()))
return ConvertUntypedDocument<TDocument>(untyped);

935
src/GameCult.Networking/CultNetDatabase.cs
View file

File diff suppressed because it is too large Load diff

33
src/GameCult.Networking/CultNetDatabaseServer.cs
View file

@ -112,10 +112,13 @@ namespace GameCult.Networking
request);
}
// A database serves only what its shards own.
return _database.Documents.CreateRawSnapshotResponse(
_database.Cache,
string.IsNullOrWhiteSpace(request.MessageId) ? Guid.NewGuid().ToString("N") : request.MessageId,
request);
request,
options: null,
rowFilter: _database.Owns);
}
/// <summary>
@ -135,6 +138,9 @@ namespace GameCult.Networking
// R-Q: asOf is one shard-log watermark, so a selection whose matched rows span more than
// one shard's log is refused rather than answered against a watermark that is not
// exact for all of them.
// A row whose schema no shard owns is not part of what this database serves: it is left out, not an error.
options: null,
rowFilter: _database.Owns,
shardIdOf: (schemaId, key) => _database.ResolveShard(schemaId, key).ShardId,
cursorKey: _database.CursorKey,
// S-9: once the matched rows resolve to one shard, asOf is that shard's own watermark,
@ -312,6 +318,16 @@ namespace GameCult.Networking
peer.SendCultNet(CreateRoutingError(ex));
}
}
catch (CultNetUnownedSchemaException ex)
{
_server.Logger.LogWarning($"CultNet raw put refused: {ex.Message}");
peer.SendCultNet(CultNetErrorMessage.ForUnownedSchema(ex));
}
catch (Exception ex) when (CommittedWithLogFailure(ex))
{
// The write committed and was published; only its shard log failed, so the peer is not told it was refused.
_server.Logger.LogWarning($"CultNet raw put committed but its shard log failed: {ex.Message}");
}
catch (Exception ex)
{
_server.Logger.LogError($"CultNet raw put failed: {ex.Message}");
@ -332,6 +348,15 @@ namespace GameCult.Networking
peer.SendCultNet(CreateRoutingError(ex));
}
}
catch (CultNetUnownedSchemaException ex)
{
_server.Logger.LogWarning($"CultNet raw delete refused: {ex.Message}");
peer.SendCultNet(CultNetErrorMessage.ForUnownedSchema(ex));
}
catch (Exception ex) when (CommittedWithLogFailure(ex))
{
_server.Logger.LogWarning($"CultNet raw delete committed but its shard log failed: {ex.Message}");
}
catch (Exception ex)
{
peer.SendCultNet(new CultNetErrorMessage { Error = ex.Message });
@ -604,6 +629,12 @@ namespace GameCult.Networking
: messageId;
}
// A CultNetShardLogException means the write committed and was published (the writer hears of a failed log afterwards);
// the cache wraps it in an AggregateException when something else failed in the same hold.
private static bool CommittedWithLogFailure(Exception exception) =>
exception is CultNetShardLogException ||
exception is AggregateException aggregate && aggregate.Flatten().InnerExceptions.Any(inner => inner is CultNetShardLogException);
private static CultNetErrorMessage CreateRoutingError(CultNetShardAuthorityException exception)
{
return new CultNetErrorMessage

3
src/GameCult.Networking/CultNetDatabaseSubscriptionServer.cs
View file

@ -421,7 +421,8 @@ namespace GameCult.Networking
documentSelection,
ordinalOf: (schemaId, key) => _database.LastWriteSequence(schemaId, key) ?? 0,
asOf: _database.CurrentAsOf(),
options: null);
options: null,
rowFilter: _database.Owns);
// The snapshot is one row per record key by construction, so the duplicate-key throw this
// used to guard can no longer fire (docs/cultnet-selection-cut.md, S2-2); deleted rather
// than tested around.

31
src/GameCult.Networking/CultNetDocumentRegistry.cs
View file

@ -280,7 +280,16 @@ namespace GameCult.Networking
CultCache cache,
string messageId,
CultNetSnapshotRequestMessage? filter = null,
CultNetDocumentMessageOptions? options = null)
CultNetDocumentMessageOptions? options = null) =>
CreateRawSnapshotResponse(cache, messageId, filter, options, rowFilter: null);
// rowFilter leaves out the cached rows it rejects before evaluation (a database serves only what its shards own).
internal CultNetSnapshotResponseRawMessage CreateRawSnapshotResponse(
CultCache cache,
string messageId,
CultNetSnapshotRequestMessage? filter,
CultNetDocumentMessageOptions? options,
Func<CultDocumentDescriptor, CultRecordKey, bool>? rowFilter)
{
if (cache == null) throw new ArgumentNullException(nameof(cache));
var lowSchemas = CultNetV0SelectionLowering.Lower(filter?.SchemaIds);
@ -323,7 +332,7 @@ namespace GameCult.Networking
ordinalOf = static (_, _) => 0;
}
var page = SelectAll(cache, selection, ordinalOf, asOf: 0, options);
var page = SelectAll(cache, selection, ordinalOf, asOf: 0, options, rowFilter);
return new CultNetSnapshotResponseRawMessage
{
@ -348,13 +357,27 @@ namespace GameCult.Networking
CultNetDocumentMessageOptions? options = null,
Func<string, CultRecordKey, string>? shardIdOf = null,
CultNetSelectionCursorKey? cursorKey = null,
Func<string, ulong>? asOfForShard = null)
Func<string, ulong>? asOfForShard = null) =>
CreateSelectionResponse(cache, messageId, selection, ordinalOf, asOf, options, shardIdOf, cursorKey, asOfForShard, rowFilter: null);
// rowFilter leaves out the cached rows it rejects before evaluation (a database serves only what its shards own).
internal CultNetSnapshotResponseRawV1Message CreateSelectionResponse(
CultCache cache,
string messageId,
CultNetSelection selection,
Func<string, CultRecordKey, long> ordinalOf,
ulong asOf,
CultNetDocumentMessageOptions? options,
Func<string, CultRecordKey, string>? shardIdOf,
CultNetSelectionCursorKey? cursorKey,
Func<string, ulong>? asOfForShard,
Func<CultDocumentDescriptor, CultRecordKey, bool>? rowFilter)
{
if (cache == null) throw new ArgumentNullException(nameof(cache));
if (selection == null) throw new ArgumentNullException(nameof(selection));
if (ordinalOf == null) throw new ArgumentNullException(nameof(ordinalOf));
var page = SelectPage(cache, selection, ordinalOf, asOf, options, shardIdOf: shardIdOf, cursorKey: cursorKey, asOfForShard: asOfForShard);
var page = SelectPage(cache, selection, ordinalOf, asOf, options, rowFilter, shardIdOf, cursorKey, asOfForShard);
return new CultNetSnapshotResponseRawV1Message
{
MessageId = RequireNonEmpty(messageId, nameof(messageId)),

5
src/GameCult.Networking/CultNetRudpSchemaServer.cs
View file

@ -43,9 +43,10 @@ namespace GameCult.Networking
/// </summary>
public uint ConnectionId { get; set; } = 0x43554c54;
/// <summary>
/// Gets or sets the first local packet sequence for each accepted peer.
/// Gets or sets the first local packet sequence for each accepted peer. Unset, each peer's
/// session draws its own at random.
/// </summary>
public uint InitialSequence { get; set; } = 100;
public uint? InitialSequence { get; set; }
/// <summary>
/// Gets or sets the resend delay in milliseconds.
/// </summary>

13
src/GameCult.Networking/CultNetSchemaMessages.cs
View file

@ -511,7 +511,7 @@ namespace GameCult.Networking
[Key("routingHint")] public CultNetShardRoutingHint? RoutingHint { get; set; }
/// <summary>
/// Gets or sets the machine-readable refusal code (R-N), e.g. <c>selection_invalid</c>,
/// <c>cursor_stale</c>, <c>cursor_invalid</c>, <c>reference_outside_target</c>. Additive to
/// <c>cursor_stale</c>, <c>cursor_invalid</c>, <c>reference_outside_target</c>, <c>unowned_schema</c>. Additive to
/// <c>cultnet.error.v0</c>; a v0 peer that does not know this field ignores it.
/// </summary>
[Key("code")] public string? Code { get; set; }
@ -532,6 +532,17 @@ namespace GameCult.Networking
Details = new CultNetErrorDetails { Field = ex.Field, Value = ex.Value }
};
/// <summary>
/// Builds the wire error for a write no shard owns: code <c>unowned_schema</c>, details <c>{ field: "schemaId", value }</c>.
/// </summary>
public static CultNetErrorMessage ForUnownedSchema(CultNetUnownedSchemaException ex) =>
new CultNetErrorMessage
{
Error = $"unowned_schema: {ex.Message}",
Code = "unowned_schema",
Details = new CultNetErrorDetails { Field = "schemaId", Value = ex.SchemaId }
};
/// <summary>
/// Builds the wire error for a cursor refusal (R-N): code <c>cursor_stale</c> with details
/// <c>{ asOf, current }</c>, or code <c>cursor_invalid</c> with no details.

5
src/GameCult.Networking/CultNetShardMutationLogStorage.cs
View file

@ -10,6 +10,11 @@ namespace GameCult.Networking
/// <summary>
/// Stores accepted shard-log entries in their replica catch-up wire shape.
/// </summary>
/// <remarks>
/// A database calls its store from a cache journal, under that cache's gate, so a slow store slows every writer of
/// the cache and blocks its readers too, because reads take the same gate. A store must not call the cache, or wait for a thread that does. Appends must throw rather than
/// swallow a failure: the database burns the sequence and compacts past it, and tells the writer.
/// </remarks>
public interface ICultNetShardMutationLogStore
{
/// <summary>

512
src/GameCult.Networking/CultNetTransport.cs
View file

@ -775,9 +775,11 @@ namespace GameCult.Networking
/// </summary>
public uint ConnectionId { get; set; }
/// <summary>
/// Gets or sets the first local packet sequence.
/// Gets or sets the first local packet sequence. Unset, each session draws its own at random:
/// every session is a new identity to its peer, and an options object reused for several
/// must not give them one sequence.
/// </summary>
public uint InitialSequence { get; set; } = 1;
public uint? InitialSequence { get; set; }
/// <summary>
/// Gets or sets the resend delay in milliseconds.
/// </summary>
@ -852,9 +854,11 @@ namespace GameCult.Networking
/// </summary>
public uint ConnectionId { get; set; }
/// <summary>
/// Gets or sets the first local packet sequence.
/// Gets or sets the first local packet sequence. Unset, each session draws its own at random:
/// every session is a new identity to its peer, and an options object reused for several
/// must not give them one sequence.
/// </summary>
public uint InitialSequence { get; set; } = 1;
public uint? InitialSequence { get; set; }
/// <summary>
/// Gets or sets the resend delay in milliseconds.
/// </summary>
@ -889,18 +893,17 @@ namespace GameCult.Networking
/// <summary>Maximum reliable packets admitted to the wire before acknowledgements advance the window.</summary>
public const int ReliableSendWindowPackets = 32;
private const int ReceivedSequenceWindow = 4096;
// How long a client keeps retransmitting a Connect nobody answered before it abandons that
// attempt for a fresh one. A Connect the server answers with an Ack, never an Accept, is one
// the server judged stale (see AcceptConnect); retransmitting the same sequence would never
// change its mind.
private const long ConnectAttemptMs = 3_000;
private sealed class PendingReliablePacket
{
public CultNetRudpPacket Packet { get; set; } = new CultNetRudpPacket();
public long LastSentAtMs { get; set; }
}
private sealed class PendingOrderedFrame
{
public CultNetRudpDeliveredFrame Frame { get; set; } = new CultNetRudpDeliveredFrame();
public uint NextSequence { get; set; }
}
private sealed class FragmentBuffer
{
public string ChannelId { get; set; } = string.Empty;
@ -915,6 +918,24 @@ namespace GameCult.Networking
private ushort _nextFragmentId = 1;
private readonly int? _maxPendingReliablePackets;
private volatile bool _connected;
// Advances every time a generation ends. Everything issued in a generation (writes, flushes)
// belongs to it and dies with it.
private long _generation;
// True from the end of a generation until the next Connect or Accept begins one. A flush started
// in that interval has no live generation to wait on.
private bool _ended;
// The sequence of the Connect that started the current generation: sent by this side, or
// accepted from the peer. A Connect repeats exactly when the session is connected and the
// Connect carries this sequence.
private uint? _connectSequence;
// A Connect this side sent is unanswered. Only then is an Accept honoured.
private bool _awaitingAccept;
// When the unanswered Connect first went out; DueResends abandons it for a fresh attempt once
// ConnectAttemptMs has passed.
private long _connectStartedAtMs;
// The payload of the Connect this side sent, kept so a fresh attempt can always be built
// whatever became of the pending packet.
private byte[] _connectPayload = Array.Empty<byte>();
private long? _lastReceivedAtMs;
private uint? _highestReceivedSequence;
private readonly HashSet<uint> _receivedSequences = new HashSet<uint>();
@ -922,9 +943,14 @@ namespace GameCult.Networking
private readonly object _pendingReliableGate = new object();
private readonly Dictionary<uint, PendingReliablePacket> _pendingReliable = new Dictionary<uint, PendingReliablePacket>();
private readonly Queue<CultNetRudpPacket> _queuedReliable = new Queue<CultNetRudpPacket>();
private readonly Dictionary<string, uint> _orderedNextSequenceByChannel = new Dictionary<string, uint>(StringComparer.Ordinal);
private readonly Dictionary<string, SortedDictionary<uint, PendingOrderedFrame>> _orderedBuffers =
new Dictionary<string, SortedDictionary<uint, PendingOrderedFrame>>(StringComparer.Ordinal);
// Every reliable sequence up to and including this one has been received since the peer state
// was last reset. Only the handshake seeds it: the peer's Connect on the accepting side, the
// peer's Accept on the connecting side. Reliable data that arrives before that is refused.
private uint? _receivedThrough;
// Ordered frames received but not yet deliverable, keyed by first sequence. A frame is held for
// exactly one reason: a reliable sequence below it has not arrived.
private readonly SortedDictionary<uint, CultNetRudpDeliveredFrame> _orderedHeld =
new SortedDictionary<uint, CultNetRudpDeliveredFrame>();
private readonly Dictionary<string, FragmentBuffer> _fragmentBuffers = new Dictionary<string, FragmentBuffer>(StringComparer.Ordinal);
/// <summary>
@ -936,18 +962,39 @@ namespace GameCult.Networking
ValidateLimits(options.InitialSequence, options.MaxPendingReliablePackets);
ConnectionId = options.ConnectionId;
_nextSequence = options.InitialSequence;
_nextSequence = options.InitialSequence ?? DrawSequence();
ResendDelayMs = options.ResendDelayMs;
_maxPendingReliablePackets = options.MaxPendingReliablePackets;
}
/// <summary>
/// A sequence drawn from a secure source in [1, 2^31). The Connect's sequence is what tells a
/// peer whether a Connect repeats one it already accepted or starts a new session, so two
/// sessions must not share one by default.
/// </summary>
private static uint DrawSequence()
{
const uint ceiling = 1u << 31;
var bytes = new byte[4];
using (var random = System.Security.Cryptography.RandomNumberGenerator.Create())
random.GetBytes(bytes);
return 1u + BitConverter.ToUInt32(bytes, 0) % (ceiling - 1u);
}
// Whether `sequence` is at or before `mark` in serial order, within the receive window. The
// compare is modular, so it holds across the wrap of the 32-bit space; a sequence further back
// than the window is the duplicate test's below-window clause, and one ahead of the mark is
// never before it.
private static bool AtOrBefore(uint sequence, uint mark) =>
unchecked(mark - sequence) < ReceivedSequenceWindow;
internal static readonly byte[] RefusedPacketReason = Encoding.UTF8.GetBytes("session refused a packet");
/// <summary>
/// A session cannot admit a Connect when its sequence space starts exhausted or its reliable
/// queue holds nothing, so neither is a usable configuration.
/// </summary>
internal static void ValidateLimits(uint initialSequence, int? maxPendingReliablePackets)
internal static void ValidateLimits(uint? initialSequence, int? maxPendingReliablePackets)
{
if (initialSequence == uint.MaxValue)
{
@ -972,6 +1019,8 @@ namespace GameCult.Networking
/// Gets whether the session has completed the connect/accept handshake.
/// </summary>
public bool Connected => _connected;
internal long Generation => Interlocked.Read(ref _generation);
internal bool Ended => _ended;
/// <summary>
/// Gets the logical time of the last received packet.
/// </summary>
@ -1007,14 +1056,57 @@ namespace GameCult.Networking
/// </summary>
public CultNetRudpPacket CreateConnect(long nowMs = 0, byte[]? payload = null)
{
// A session that has had a peer starts a new generation: nothing it learned from that
// peer describes the one this Connect reaches, and what it still owed that peer no
// longer takes room in the queue.
if (_connectSequence.HasValue)
ResetPeerState();
EnsureReliableCapacity(1);
_ended = false;
var packet = CreatePacket(CultNetRudpPacketType.Connect, "control", payload ?? Array.Empty<byte>(), reliable: true, ordered: true, sequenced: false);
_connectSequence = packet.Sequence;
_awaitingAccept = true;
_connectStartedAtMs = nowMs;
_connectPayload = packet.Payload ?? Array.Empty<byte>();
TrackReliable(packet, nowMs);
return packet;
}
/// <summary>
/// Accepts a connect packet and returns a reliable ordered accept packet.
/// Whether <paramref name="packet"/> is a Connect this session's current generation already owns:
/// a retransmit of the Connect that started it (the session is connected and the sequence is
/// that Connect's), or a stale copy of an earlier attempt by the same client (a sequence before
/// it, within the receive window). Servers that keep one session per peer ask this to tell a
/// Connect that starts a new session from one that does not; anything else that reaches
/// <see cref="AcceptConnect"/> starts a new generation.
/// </summary>
public bool ConnectRepeats(CultNetRudpPacket packet)
{
if (packet == null) throw new ArgumentNullException(nameof(packet));
return packet.PacketType == CultNetRudpPacketType.Connect
&& _connected
&& _connectSequence.HasValue
&& (_connectSequence.Value == packet.Sequence
|| ConnectIsStale(packet.Sequence, _connectSequence.Value));
}
// A Connect that precedes the current generation's within the receive window is the client's
// earlier attempt, delayed in the network: a client that retried never sends a lower sequence
// again. Restarting on it would strand the client, which honours only the Accept for its newest
// Connect. The rule is TCP's answer to a delayed SYN (RFC 5961's challenge ACK): keep the
// connection and answer with an Ack. A restarted client whose random initial sequence lands in
// this window is answered the same way and abandons the attempt for a fresh draw
// (ConnectAttemptMs).
private static bool ConnectIsStale(uint sequence, uint current) =>
sequence != current && AtOrBefore(sequence, current);
/// <summary>
/// Answers a connect packet. A repeat of the accepted connect queues nothing, so a connect storm
/// cannot grow the reliable queue: the reply is the accept still awaiting acknowledgement, or an
/// ack once it was acknowledged. A stale copy of an earlier attempt gets the same reply and
/// changes nothing else: it is not evidence the peer is alive. Any other connect ends the
/// current generation, forgets the peer and accepts a new one with a reliable ordered accept
/// packet.
/// </summary>
public CultNetRudpPacket AcceptConnect(CultNetRudpPacket packet, long nowMs = 0, byte[]? payload = null)
{
@ -1024,30 +1116,31 @@ namespace GameCult.Networking
throw new InvalidOperationException($"Expected RUDP connect packet, got {packet.PacketType}.");
}
if (ConnectRepeats(packet))
{
if (_connectSequence == packet.Sequence)
{
ApplyAcknowledgements(packet);
RememberReceived(packet.Sequence);
_lastReceivedAtMs = nowMs;
}
return PendingAcceptForResend(nowMs) ?? CreateAck();
}
ResetPeerState();
EnsureReliableCapacity(1);
RememberReceived(packet.Sequence);
SeedReceived(packet.Sequence);
_lastReceivedAtMs = nowMs;
_connectSequence = packet.Sequence;
_connected = true;
_ended = false;
var response = CreatePacket(CultNetRudpPacketType.Accept, "control", payload ?? Array.Empty<byte>(), reliable: true, ordered: true, sequenced: false);
TrackReliable(response, nowMs);
return response;
}
/// <summary>
/// Answers a connect from a peer this session has already accepted. The packet's sequence is
/// remembered and nothing is queued, so a connect storm cannot grow the reliable queue. The reply
/// is the accept still awaiting acknowledgement, or an ack once it was acknowledged.
/// </summary>
public CultNetRudpPacket AnswerRepeatedConnect(CultNetRudpPacket packet, long nowMs = 0)
// The Accept still awaiting acknowledgement, resent.
private CultNetRudpPacket? PendingAcceptForResend(long nowMs)
{
RequireConnection(packet);
if (packet.PacketType != CultNetRudpPacketType.Connect)
{
throw new InvalidOperationException($"Expected RUDP connect packet, got {packet.PacketType}.");
}
ApplyAcknowledgements(packet);
RememberReceived(packet.Sequence);
_lastReceivedAtMs = nowMs;
lock (_pendingReliableGate)
{
foreach (var pending in _pendingReliable.Values)
@ -1059,7 +1152,7 @@ namespace GameCult.Networking
}
}
return CreateAck();
return null;
}
/// <summary>
@ -1152,16 +1245,29 @@ namespace GameCult.Networking
public CultNetRudpReceiveResult Receive(CultNetRudpPacket packet, long nowMs = 0)
{
RequireConnection(packet);
ApplyAcknowledgements(packet);
// An Accept counts only while this side's Connect is unanswered and the Accept names it.
// A late or duplicate one, or one from an earlier generation, must not seed the watermark
// or revive an ended session.
var honoursAccept = packet.PacketType == CultNetRudpPacketType.Accept
&& _awaitingAccept
&& _connectSequence.HasValue
&& PacketAcknowledges(packet, _connectSequence.Value);
if (packet.PacketType == CultNetRudpPacketType.Accept && !honoursAccept)
{
return new CultNetRudpReceiveResult();
}
// While this side's Connect awaits its Accept, only the Accept it honours retires it: an Ack
// that names the Connect (a server's reply to a repeat or a stale copy) says the server did
// not start a session.
if (honoursAccept || !_awaitingAccept)
ApplyAcknowledgements(packet);
var readyToSend = PromoteQueuedReliable(nowMs);
_lastReceivedAtMs = nowMs;
var expectedSequenceIfUninitialized = _highestReceivedSequence.HasValue
? _highestReceivedSequence.Value + 1
: packet.Sequence;
if (packet.PacketType == CultNetRudpPacketType.Accept)
if (honoursAccept)
{
RememberReceived(packet.Sequence);
_awaitingAccept = false;
SeedReceived(packet.Sequence);
_connected = true;
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend };
}
@ -1196,7 +1302,7 @@ namespace GameCult.Networking
if (packet.PacketType == CultNetRudpPacketType.Disconnect)
{
_connected = false;
EndSession();
return new CultNetRudpReceiveResult
{
ReadyToSend = readyToSend,
@ -1210,11 +1316,15 @@ namespace GameCult.Networking
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend };
}
var duplicate = packet.Reliable
&& (_receivedSequences.Contains(packet.Sequence)
|| (_highestReceivedSequence.HasValue
&& packet.Sequence < _highestReceivedSequence.Value
&& _highestReceivedSequence.Value - packet.Sequence >= ReceivedSequenceWindow));
// Reliable data before the handshake has seeded the watermark has no place in the order:
// refuse it unremembered, so it is not acknowledged and the sender retransmits it once
// the handshake is done.
if (packet.Reliable && !_receivedThrough.HasValue)
{
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend };
}
var duplicate = packet.Reliable && WasReceived(packet.Sequence);
if (packet.Reliable)
{
RememberReceived(packet.Sequence);
@ -1224,30 +1334,34 @@ namespace GameCult.Networking
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend };
}
var delivered = new List<CultNetRudpDeliveredFrame>();
var reassembled = Reassemble(packet);
if (reassembled == null)
if (reassembled != null)
{
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend };
}
if (!reassembled.Ordered && packet.Sequenced)
{
var newestSequence = reassembled.NextSequence - 1;
if (_latestSequencedByChannel.TryGetValue(reassembled.Frame.ChannelId, out var latestSequence)
&& newestSequence <= latestSequence)
if (reassembled.Ordered)
{
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend };
_orderedHeld[reassembled.Frame.Sequence] = reassembled.Frame;
}
else if (!packet.Sequenced)
{
delivered.Add(reassembled.Frame);
}
else
{
var newestSequence = reassembled.NextSequence - 1;
if (!_latestSequencedByChannel.TryGetValue(reassembled.Frame.ChannelId, out var latestSequence)
|| newestSequence > latestSequence)
{
_latestSequencedByChannel[reassembled.Frame.ChannelId] = newestSequence;
delivered.Add(reassembled.Frame);
}
}
_latestSequencedByChannel[reassembled.Frame.ChannelId] = newestSequence;
}
return new CultNetRudpReceiveResult
{
ReadyToSend = readyToSend,
Delivered = reassembled.Ordered
? DeliverOrdered(reassembled.Frame, reassembled.NextSequence, expectedSequenceIfUninitialized)
: new[] { reassembled.Frame }
};
// Any reliable packet may have advanced the watermark, a fragment or an unordered frame as
// much as an ordered one, so the drain runs after all of them.
delivered.AddRange(DrainOrdered());
return new CultNetRudpReceiveResult { ReadyToSend = readyToSend, Delivered = delivered };
}
/// <summary>
@ -1279,6 +1393,10 @@ namespace GameCult.Networking
/// </summary>
public CultNetRudpPacket CreateAckForReceived(uint receivedSequence)
{
// What Receive refused is not acknowledged by name (the ack carries only what was
// received), so its sender retransmits it.
if (!WasReceived(receivedSequence))
return CreateAck();
var (ack, _) = AckState();
return ack >= receivedSequence && ack - receivedSequence <= 32
? CreateAck()
@ -1294,23 +1412,39 @@ namespace GameCult.Networking
}
/// <summary>
/// Forgets everything learned from the peer; sequence numbers already issued stay issued.
/// The one way a session generation ends. The session stops being connected and what it still
/// owed the peer dies with it: a write not yet acknowledged is dropped, so no later session
/// retransmits it or credits an ack to it. What was learned from the peer is not touched: the
/// peer may not know the session ended, and forgetting what it sent would let its retransmits
/// be delivered twice.
/// </summary>
public void ResetPeerState()
private void EndSession()
{
_connected = false;
_lastReceivedAtMs = null;
_highestReceivedSequence = null;
_receivedSequences.Clear();
_nextSequencedByChannel.Clear();
_latestSequencedByChannel.Clear();
_ended = true;
_awaitingAccept = false;
Interlocked.Increment(ref _generation);
lock (_pendingReliableGate)
{
_pendingReliable.Clear();
_queuedReliable.Clear();
}
_orderedNextSequenceByChannel.Clear();
_orderedBuffers.Clear();
}
/// <summary>
/// Ends the current generation and forgets everything learned from the peer; sequence numbers
/// already issued stay issued.
/// </summary>
public void ResetPeerState()
{
EndSession();
_lastReceivedAtMs = null;
_highestReceivedSequence = null;
_receivedSequences.Clear();
_nextSequencedByChannel.Clear();
_latestSequencedByChannel.Clear();
_receivedThrough = null;
_orderedHeld.Clear();
_fragmentBuffers.Clear();
}
@ -1330,7 +1464,7 @@ namespace GameCult.Networking
/// </summary>
public CultNetRudpPacket CreateDisconnect(byte[]? reason = null)
{
_connected = false;
EndSession();
return CreatePacket(CultNetRudpPacketType.Disconnect, "control", reason ?? Array.Empty<byte>(), reliable: false, ordered: false, sequenced: false);
}
@ -1347,7 +1481,7 @@ namespace GameCult.Networking
{
return false;
}
_connected = false;
EndSession();
return true;
}
@ -1356,6 +1490,9 @@ namespace GameCult.Networking
/// </summary>
public IReadOnlyList<CultNetRudpPacket> DueResends(long nowMs)
{
var fresh = AbandonUnansweredConnect(nowMs);
if (fresh != null)
return new[] { fresh };
var due = new List<CultNetRudpPacket>();
lock (_pendingReliableGate)
{
@ -1522,9 +1659,69 @@ namespace GameCult.Networking
}
}
private static bool PacketAcknowledges(CultNetRudpPacket packet, uint sequence)
{
if (packet.Ack == sequence)
return true;
for (var bit = 0; bit < 32; bit++)
{
if ((packet.AckMask & (1u << bit)) != 0 && packet.Ack > bit && packet.Ack - (uint)bit - 1 == sequence)
return true;
}
return false;
}
// A Connect unanswered for ConnectAttemptMs is replaced, not retransmitted further, by a Connect
// with the same payload and the abandoned sequence plus the receive window less one. Nothing
// was ever sent in an unanswered generation, so no sequence issued so far is owed. The jump
// keeps a late copy of the abandoned Connect inside the new one's stale window, so a server
// that took the new Connect answers the copy with an Ack instead of restarting; and it leaves
// the stale window of whatever generation the server holds, unless that generation sits
// exactly at the jump, and then the next attempt leaves it. The first Connect of a session is
// the only one drawn at random.
private CultNetRudpPacket? AbandonUnansweredConnect(long nowMs)
{
if (!_awaitingAccept || nowMs - _connectStartedAtMs < ConnectAttemptMs || !_connectSequence.HasValue)
return null;
var jumped = (ulong)_connectSequence.Value + ReceivedSequenceWindow - 1;
_nextSequence = (uint)Math.Min(jumped, uint.MaxValue - 1);
return CreateConnect(nowMs, _connectPayload);
}
// The handshake's one act on the watermark: the peer's Connect or Accept is the first sequence
// of the session, whatever else it has sent.
private void SeedReceived(uint sequence)
{
_receivedThrough = sequence;
RememberReceived(sequence);
}
// The one duplicate test: true for a sequence in the window that was received, for one below
// the window, and for one at or before the watermark. The watermark starts at the handshake's
// seed, so a frame the peer sent in an earlier generation (every sequence it issued is below
// the Connect that began this one) is a duplicate of something already delivered and is
// acknowledged, never delivered again.
private bool WasReceived(uint sequence)
{
return _receivedSequences.Contains(sequence)
|| (_highestReceivedSequence.HasValue
&& sequence < _highestReceivedSequence.Value
&& _highestReceivedSequence.Value - sequence >= ReceivedSequenceWindow)
|| (_receivedThrough.HasValue && AtOrBefore(sequence, _receivedThrough.Value));
}
private void RememberReceived(uint sequence)
{
_receivedSequences.Add(sequence);
if (_receivedThrough.HasValue)
{
var through = _receivedThrough.Value;
while (through != uint.MaxValue && _receivedSequences.Contains(through + 1))
{
through++;
}
_receivedThrough = through;
}
if (!_highestReceivedSequence.HasValue || sequence > _highestReceivedSequence.Value)
{
_highestReceivedSequence = sequence;
@ -1632,80 +1829,35 @@ namespace GameCult.Networking
};
}
private IReadOnlyList<CultNetRudpDeliveredFrame> DeliverOrdered(
CultNetRudpDeliveredFrame frame,
uint nextSequenceAfterFrame,
uint expectedSequenceIfUninitialized)
{
if (!_orderedNextSequenceByChannel.TryGetValue(frame.ChannelId, out var next))
{
next = Math.Min(expectedSequenceIfUninitialized, frame.Sequence);
_orderedNextSequenceByChannel[frame.ChannelId] = next;
}
while (frame.Sequence > next
&& _receivedSequences.Contains(next)
&& (!_orderedBuffers.TryGetValue(frame.ChannelId, out var pendingBuffer) || !pendingBuffer.ContainsKey(next)))
{
next++;
_orderedNextSequenceByChannel[frame.ChannelId] = next;
}
if (frame.Sequence < next)
{
return Array.Empty<CultNetRudpDeliveredFrame>();
}
if (frame.Sequence > next)
{
if (!_orderedBuffers.TryGetValue(frame.ChannelId, out var buffer))
{
buffer = new SortedDictionary<uint, PendingOrderedFrame>();
_orderedBuffers[frame.ChannelId] = buffer;
}
buffer[frame.Sequence] = new PendingOrderedFrame { Frame = frame, NextSequence = nextSequenceAfterFrame };
return Array.Empty<CultNetRudpDeliveredFrame>();
}
_orderedNextSequenceByChannel[frame.ChannelId] = nextSequenceAfterFrame;
return new[] { frame }.Concat(DrainOrdered(frame.ChannelId)).ToArray();
}
private IReadOnlyList<CultNetRudpDeliveredFrame> DrainOrdered(string channelId)
/// <summary>
/// Delivers, in sequence order, every held ordered frame whose first sequence is at most one past
/// the watermark. Ordered delivery has one owner: the watermark. A frame this call does not
/// deliver waits for the sequence below it, and nothing else releases it.
/// </summary>
private IReadOnlyList<CultNetRudpDeliveredFrame> DrainOrdered()
{
var delivered = new List<CultNetRudpDeliveredFrame>();
if (!_orderedBuffers.TryGetValue(channelId, out var buffer))
if (!_receivedThrough.HasValue)
{
return delivered;
}
while (_orderedNextSequenceByChannel.TryGetValue(channelId, out var next) && buffer.TryGetValue(next, out var pending))
var through = _receivedThrough.Value;
foreach (var pair in _orderedHeld)
{
buffer.Remove(next);
delivered.Add(pending.Frame);
_orderedNextSequenceByChannel[channelId] = pending.NextSequence;
SkipReceivedNonChannelSequences(channelId);
if (through != uint.MaxValue && pair.Key > through + 1)
{
break;
}
delivered.Add(pair.Value);
}
foreach (var frame in delivered)
{
_orderedHeld.Remove(frame.Sequence);
}
return delivered;
}
private void SkipReceivedNonChannelSequences(string channelId)
{
if (!_orderedNextSequenceByChannel.TryGetValue(channelId, out var next))
{
return;
}
while (_receivedSequences.Contains(next)
&& (!_orderedBuffers.TryGetValue(channelId, out var buffer) || !buffer.ContainsKey(next)))
{
next++;
_orderedNextSequenceByChannel[channelId] = next;
}
}
private ushort AllocateFragmentId()
{
var fragmentId = _nextFragmentId;
@ -1924,18 +2076,23 @@ namespace GameCult.Networking
var preserved = new List<CultNetTransportFrame>();
while (_deliveredFrames.Count > 0)
preserved.Add(_deliveredFrames.Dequeue());
// A flush belongs to the generation it started in: whatever ends that generation, and
// whatever begins after it, the writes being waited on are gone.
long generation;
lock (_sessionGate)
generation = _session.Generation;
try
{
while (true)
{
// An ended session forgot its unacknowledged writes; reporting them
// flushed would be a lie.
var ended = DisconnectReason;
if (ended != null)
throw new InvalidOperationException(
"RUDP session ended before its reliable writes were acknowledged: " + Encoding.UTF8.GetString(ended));
lock (_sessionGate)
{
// An ended session forgot its unacknowledged writes; reporting them flushed
// would be a lie.
if (_session.Ended || _session.Generation != generation)
throw new InvalidOperationException(
"RUDP session ended before its reliable writes were acknowledged: " +
(DisconnectReason == null ? "no reason given" : Encoding.UTF8.GetString(DisconnectReason)));
if (_session.OutstandingReliablePacketCount == 0)
return;
}
@ -2178,7 +2335,6 @@ namespace GameCult.Networking
error.SocketErrorCode == SocketError.ConnectionAborted ||
error.SocketErrorCode == SocketError.Shutdown)
{
DisconnectReason = Array.Empty<byte>();
return false;
}
@ -2205,6 +2361,10 @@ namespace GameCult.Networking
_stats.PacketsDropped++;
return true;
}
// A Connect from another endpoint is a new client, whatever sequence it carries: the endpoint
// moves and the Connect starts a new generation. Only a Connect from the peer's own
// endpoint can be its repeat.
var connectFromNewEndpoint = false;
if (_remoteEndPoint == null)
{
// Only a Connect claims the endpoint of a server-mode transport.
@ -2218,8 +2378,16 @@ namespace GameCult.Networking
}
else if (!_remoteEndPoint.Equals(remote))
{
_stats.PacketsDropped++;
return true;
if (_mode == CultNetRudpSocketMode.Server && packet.PacketType == CultNetRudpPacketType.Connect)
{
_remoteEndPoint = remote;
connectFromNewEndpoint = true;
}
else
{
_stats.PacketsDropped++;
return true;
}
}
TracePacket("rx", packet, received, remote);
@ -2230,17 +2398,12 @@ namespace GameCult.Networking
CultNetRudpPacket accept;
try
{
// A connect from the peer this transport already accepted repeats: answer it
// with the accept already owed and queue nothing.
if (_session.Connected)
{
accept = _session.AnswerRepeatedConnect(packet, NowMs());
}
else
{
accept = _session.AcceptConnect(packet, NowMs());
DisconnectReason = null;
}
// The session answers a repeat of the connect it accepted with the accept
// already owed, and anything else with a new generation.
if (connectFromNewEndpoint)
_session.ResetPeerState();
accept = _session.AcceptConnect(packet, NowMs());
DisconnectReason = null;
}
catch (InvalidOperationException)
{
@ -2406,9 +2569,10 @@ namespace GameCult.Networking
/// </summary>
public uint ConnectionId { get; set; }
/// <summary>
/// Gets or sets the first local packet sequence for each accepted peer session.
/// Gets or sets the first local packet sequence for each accepted peer session. Unset, each
/// session draws its own at random.
/// </summary>
public uint InitialSequence { get; set; } = 100;
public uint? InitialSequence { get; set; }
/// <summary>
/// Gets or sets the resend delay in milliseconds.
/// </summary>
@ -2500,7 +2664,7 @@ namespace GameCult.Networking
private const int WireBurstPackets = 32;
private readonly Socket _socket;
private readonly uint _connectionId;
private readonly uint _initialSequence;
private readonly uint? _initialSequence;
private readonly long _resendDelayMs;
private readonly int? _maxFragmentBytes;
private readonly int? _maxPendingReliablePackets;
@ -2675,25 +2839,29 @@ namespace GameCult.Networking
{
var connectPayload = packet.Payload ?? Array.Empty<byte>();
_peers.TryGetValue(peerKey, out var admitted);
// The same payload from an admitted peer is a repeated Connect: answer it with the
// Accept already owed and queue nothing. A different payload is a new generation.
if (admitted != null && admitted.ConnectPayload.AsSpan().SequenceEqual(connectPayload))
// The admitted peer's session decides whether this Connect repeats the one it accepted:
// answer a repeat with the Accept already owed and queue nothing. Any other Connect is
// a new generation.
if (admitted != null)
{
lock (admitted.SessionGate)
{
CultNetRudpPacket reply;
try
if (admitted.Session.ConnectRepeats(packet))
{
reply = admitted.Session.AnswerRepeatedConnect(packet, NowMs());
}
catch (InvalidOperationException)
{
_stats.PacketsDropped++;
CultNetRudpPacket reply;
try
{
reply = admitted.Session.AcceptConnect(packet, NowMs(), _acceptPayload);
}
catch (InvalidOperationException)
{
_stats.PacketsDropped++;
return true;
}
SendPacket(admitted.RemoteEndPoint, reply);
return true;
}
SendPacket(admitted.RemoteEndPoint, reply);
}
return true;
}
var peer = new CultNetRudpSocketServerPeer(

3
tests/GameCult.Caching.Tests/CultMathJsonTests.cs
View file

@ -43,6 +43,8 @@ namespace GameCult.Caching.Tests
Json(new rect(5f, 7f, -1f, -2f)).Should().Be("""{"min":{"x":-1,"y":-2},"max":{"x":5,"y":7}}""");
Json(new CultCellular(new float4(1f, 2f, 3f, 4f), new float4(5f, 6f, 7f, 8f), 0.25f)).Should().Be(
"""{"nearest":{"x":1,"y":2,"z":3,"w":4},"edge":{"x":5,"y":6,"z":7,"w":8},"id":0.25}""");
Json(new CultPhasor(new float4(1f, 2f, 3f, 4f), new float4(5f, 6f, 7f, 8f))).Should().Be(
"""{"cos":{"x":1,"y":2,"z":3,"w":4},"sin":{"x":5,"y":6,"z":7,"w":8}}""");
}
[Test]
@ -66,6 +68,7 @@ namespace GameCult.Caching.Tests
RoundTrip(new float3x3(1f, 2f, 3f, 4f, 5f, 6f, 7f, 8f, 9f));
RoundTrip(new rect(-1f, -2f, 5f, 7f));
RoundTrip(new CultCellular(new float4(1, 2, 3, 4), new float4(5, 6, 7, 8), 0.25f));
RoundTrip(new CultPhasor(new float4(1, 2, 3, 4), new float4(5, 6, 7, 8)));
}
[Test]

5
tests/GameCult.Caching.Tests/CultMathSerializationTests.cs
View file

@ -101,6 +101,7 @@ namespace GameCult.Caching.Tests
RoundTrip(new float3x3(1f, 2f, 3f, 4f, 5f, 6f, 7f, 8f, 9f));
RoundTrip(new rect(-1f, -2f, 5f, 7f));
RoundTrip(new CultCellular(new float4(1, 2, 3, 4), new float4(5, 6, 7, 8), 0.25f));
RoundTrip(new CultPhasor(new float4(1, 2, 3, 4), new float4(5, 6, 7, 8)));
}
[Test]
@ -132,6 +133,10 @@ namespace GameCult.Caching.Tests
"94CA3F800000CA40000000CA40400000CA40800000" +
"94CA40A00000CA40C00000CA40E00000CA41000000" +
"CA3E800000");
Hex(new CultPhasor(new float4(1f, 2f, 3f, 4f), new float4(5f, 6f, 7f, 8f))).Should().Be(
"92" +
"94CA3F800000CA40000000CA40400000CA40800000" +
"94CA40A00000CA40C00000CA40E00000CA41000000");
}
[Test]

25
tests/GameCult.Geometry.Tests/GeometryDocumentTests.cs
View file

@ -10,6 +10,7 @@ using R3;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;
@ -104,7 +105,7 @@ namespace GameCult.Geometry.Tests
var domainKey = CultGeometryDomainDocument.CreateRecordKey(domain);
var request = new CultGeometryBuildRequest
{
DomainKey = domainKey.Value,
DomainKey = new CultRecordRef<CultGeometryDomainDocument>(domainKey),
WorkerGroup = "ragnarok-column-workers",
CameraPosition = [36f, -42f, 30f],
FrustumMin = [-75f, -75f, -30f],
@ -129,6 +130,26 @@ namespace GameCult.Geometry.Tests
first.Value.Should().StartWith("geometry:request:");
}
// The geometry documents reference each other by record key. The reference members are
// CultRecordRef<T> so the cache can walk them, and they stay a bare msgpack string on the wire.
[Test]
public void GeometryReferences_AreWalkableByTheCache_AndStayStringsOnTheWire()
{
var cache = new CultCache();
var descriptor = cache.Registry.GetRequired<CultGeometryBuildRequest>();
descriptor.DeclaredMembers.Should().Contain(m => m.MemberName == "DomainKey" && m.IsReference && !m.IsMany);
var chunk = SampleChunk();
var references = cache.Registry.GetRequired<CultGeometryChunkArtifact>().ReferencesOf(chunk, "CutKey").ToArray();
references.Should().ContainSingle().Which.Target.Value.Should().Be("geometry:cut:test");
var payload = CultDocumentMessagePackSerialization.SerializeUntyped(chunk, typeof(CultGeometryChunkArtifact));
var reader = new MessagePackReader(payload);
reader.ReadArrayHeader();
reader.ReadString().Should().Be(chunk.ChunkId);
reader.ReadString().Should().Be("geometry:cut:test");
}
[Test]
public void ChunkArtifact_RoundTrips_ThroughCultCacheMessagePack()
{
@ -382,7 +403,7 @@ namespace GameCult.Geometry.Tests
return new CultGeometryChunkArtifact
{
ChunkId = "chunk/ragnarok-column/column-00",
CutKey = "geometry:cut:test",
CutKey = new CultRecordRef<CultGeometrySelectedCutManifest>(new CultRecordKey("geometry:cut:test")),
SelectedCutId = "cut-test",
BoundsMin = [-1f, -1f, 0f],
BoundsMax = [1f, 1f, 1f],

26
tests/GameCult.Mesh.Tests/CultMeshStreamingTests.cs
View file

@ -938,6 +938,32 @@ public sealed class CultMeshStreamingTests
await act.Should().ThrowAsync<NotSupportedException>();
}
// A database serves only what its shards own, including when a handle reads a same-schema instance of another CLR type
// out of the cache behind it.
[TestCase("tests.mesh_note.v1", true)]
[TestCase("tests.unrelated.v1", false)]
public async Task DocumentHandle_ReadsASameSchemaInstanceOnlyWhenAShardOwnsIt(string ownedSchemaId, bool served)
{
var cache = new CultCache();
var key = new CultRecordKey("note:one");
await cache.UpsertAsync(new MeshNoteDocument { Schema = "tests.mesh_note.v1", Text = "cached" }, new CultRecordHandle<MeshNoteDocument>(key));
var database = new CultNetDatabase(cache, new CultNetDatabaseOptions
{
RuntimeId = "reader",
Shards = new[] { new CultNetShardDescriptor("notes", "reader", epoch: 1, isPrimary: true, schemaIds: new[] { ownedSchemaId }) }
});
var handle = CultMesh.Document<MeshNoteAliasDocument>(database, key, CultMesh.Verse("starbridge", "reader"));
if (served)
{
(await handle.LatestAsync()).Text.Should().Be("cached");
}
else
{
await FluentActions.Awaiting(() => handle.LatestAsync()).Should().ThrowAsync<KeyNotFoundException>();
}
}
[Test]
public async Task DocumentHandle_SubmitsPredictionsThroughCultNetDatabase()
{

1
tests/GameCult.Networking.InteropPeer/Program.cs
View file

@ -873,6 +873,7 @@ static void RudpServeOnce(Dictionary<string, string> options)
}
if (disconnectReason != null)
{
transport.FlushReliable(TimeSpan.FromSeconds(2));
transport.Disconnect(disconnectReason);
}
PollRudpAfterSend(transport, TimeSpan.FromMilliseconds(250));

1199
tests/GameCult.Networking.Tests/CultNetJournalTests.cs Normal file
View file

File diff suppressed because it is too large Load diff

Some files were not shown because too many files have changed in this diff Show more