GameCult/Mimir
Watch
9
0
Fork
You've already forked Mimir
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Harden Muninn OBS recovery from Moonlight field study

Browse source
This commit is contained in:
Meta Vens 2026-07-17 05:48:31 +01:00
commit 1221583c23
10 changed files with 438 additions and 126 deletions
Download patch file Download diff file Expand all files Collapse all files

35
docs/implementation-plan.md
View file

@ -452,12 +452,15 @@ The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot
`8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media
deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware
repair, and decode-chain-owned keyframe pressure. The production video path now
uses typed V3 Cauchy GF(256) FEC with sixteen parity shards per access unit;
the OBS receiver reconstructs arbitrary erasure sets up to the received parity
count, while V2 stripe parity remains a distinct non-authoritative schema.
Sender access units enter the handoff queue independently, and CultMesh catalog
publication runs outside the realtime media loop. The remaining work is the
full direct/proxy impairment matrix, Opus FEC/PLC, and long-duration mixed soak.
uses typed V4 Cauchy GF(256) FEC in independent 8-data/8-parity blocks. Each
block schedules data and parity as separate lanes, including fixed protection
for a short tail block. Canonical video and parity use CultNet's unreliable
`realtime` lane; selective repair and IDR recovery remain deadline-bound.
Canonical audio and its parity also use `realtime`; 4+2 FEC, reorder, and
concealment own continuity without an ACK/retransmit window. Sender access units enter the
handoff queue independently, and CultMesh catalog publication runs outside the
realtime media loop. The remaining work is the full direct/proxy impairment
matrix, Opus FEC/PLC, and long-duration mixed soak.
The socket harness and bounded PCM loss recovery have now landed. Odin's
`cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/
@ -469,9 +472,11 @@ Both sides use the experimental CultLib/CultMesh snapshot lineage used by Eve,
Aetheria, and VoidBot; no stable-branch transport shim was introduced.
Receiver pressure now closes into the long-lived NVENC encoder as bounded AIMD
bitrate control. Late/decode/repair/queue pressure backs off by 15 percent;
clean recovery adds five percent of the configured ceiling only after two
stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a
bitrate control. Startup begins at half of the configured encoder ceiling
because fixed block parity can roughly double wire rate, and additive recovery
is capped at half of that configured ceiling. Late/decode/repair/queue
pressure backs off by 15 percent; clean recovery adds one-fiftieth of the safe
cap only after ten stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a
restart and emitted the required transition IDR. The exact remaining completion
gate is tracked in
`docs/research/moonlight-reliability-acceptance-2026-07-16.md`; no cross-host
@ -485,11 +490,13 @@ with short silence concealment. The next audio cut is Opus with explicit
FEC/PLC for variable-rate compressed audio; PCM now has bounded 4+2 erasure
recovery plus concealment. The controllable video encoder owner now exists and
forces the next NVENC frame to IDR without restarting the video session. A live
D3D11 desktop proof verified the command-generated IDR. The remaining field
cut is to install its bundle on Raven, route the existing activation task
through `--video-encoder`, and run the named impairment profiles. Raven's
WireGuard SSH route timed out during the first deployment attempt, so no
interactive workstation task was disturbed.
D3D11 desktop proof verified the command-generated IDR. The Raven bundle and
OBS receiver are now deployed over the direct LAN route. Raven activation is an
interactive-token scheduled task whose PowerShell action uses
`-WindowStyle Hidden`; no WireGuard path or foreground terminal is part of the
runtime. The active field cut uses D3D11/NVENC video, WASAPI loopback audio,
typed CultNet media, and the experimental CultLib lineage. The named direct and
proxy impairment profiles and long soak remain the completion gate.
1. Replace the frame-event diagnostic bridge with concrete direct capture
drivers for Leap stereo IR first, then the

49
docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md
View file

@ -17,20 +17,24 @@ and keyframe requests. Sleipnir already rejects regressing sequence/timestamp
pairs, expires reliable HID sends after 25 ms, reports stage latency, and
neutralizes stale output.
The weakness is that these parts do not yet form one closed realtime policy:
The 2026-07-17 field cut closes most of the original mechanism gaps:
- video already uses unreliable media delivery with parity/manual repair, but
audio and HID still lean on generic expiring reliability and none of the
lanes yet forms one closed media/input policy;
- the default Muninn media latency budget is 2000 ms, which is an archive-shaped
tolerance wearing a realtime badge;
- receiver feedback is measured and repair/keyframe pressure is recorded, but
the encoder and sender do not visibly adapt bitrate, parity, or pacing from it;
- Sleipnir coalesces whole controller snapshots implicitly at the producer, but
its protocol does not name which values are replaceable state and which are
non-replaceable edges;
- wall-clock source age is reported as latency even though no explicit clock
synchronization contract proves that subtraction valid across hosts.
- canonical video and video parity use the unreliable `realtime` lane; typed
8+8 GF(256) blocks, selective repair, and decode-chain IDR recovery own loss;
- canonical 10 ms PCM audio uses `realtime`, with 4+2 FEC, a 40 ms reorder
owner, and short concealment; reliable delivery is reserved for control;
- the consumer deadline is 100 ms by default and 250 ms in the Raven field
profile; queues, retransmits, repair material, and assemblies derive from it;
- receiver pressure actuates the live NVENC encoder through multiplicative
backoff and conservative additive recovery; startup begins at half the
requested ceiling because fixed video parity can roughly double wire load;
- Sleipnir latest state and ordered edges have distinct epoch/sequence and
application-ack semantics, including bounded overflow rebase.
The remaining claim boundary is evidence: direct and impaired Raven-to-
Starfire soaks, reconnect repetition, A/V skew measurement, and a live virtual
HID exercise. The machine is no longer missing a named owner for these paths;
it still has to prove those owners under pressure.
Do not transplant Moonlight wholesale. Distill its ownership model into the
existing typed Muninn/Sleipnir pipeline.
@ -134,12 +138,12 @@ Source: [Moonlight frame-pacing FAQ](https://github.com/moonlight-stream/moonlig
| Concern | Moonlight | Muninn / Sleipnir now | Distilled judgment |
|---|---|---|---|
| Video unit | Encoded frame, FEC blocks, dependency recovery | Encoded access unit, chunks/parity, frame/dependency IDs, deadlines | The conceptual unit is already correct. Keep it. |
| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Parity, missing-chunk feedback, repair cache, keyframe pressure | Close the loop into encoder action and measure whether repair beats deadline. |
| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Typed audio packets and deadlines over generic expiring reliable delivery | Add an audio continuity policy with bounded FEC/reorder/concealment ownership. |
| Congestion | Stream-specific queues and feedback behavior | Sender queue bounds and receiver feedback stats, fixed configured bitrate | Feedback must alter pacing/bitrate/parity or it is only an autopsy report. |
| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Default media budget 2000 ms; Sleipnir reliable expiry 25 ms | Replace one broad media budget with video/audio/input budgets derived from consumption deadlines. |
| Input state | Coalesces replaceable motion/sensor/analog values | Producer emits latest full controller state; receiver filters regressions | Make coalescing and edge preservation typed protocol semantics, not an accident of polling. |
| Input edges | Button/type changes terminate batches; text is ordered | Whole snapshots contain buttons and axes | Add edge sequence/ack or a compact transition log so a press/release cannot vanish between snapshots. |
| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Typed 8+8 block FEC, deadline-bound repair, completed-frame tombstones, IDR actuation | Same ownership pattern; field loss ratios and parity overhead remain tunable. |
| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Realtime 4+2 PCM FEC, 40 ms reorder, bounded concealment | Mechanism is closed; Opus FEC/PLC is a later efficiency/quality cut. |
| Congestion | Stream-specific queues and feedback behavior | Half-ceiling startup, 15% pressure backoff, bounded additive ramp, fresh media ahead of repair | Closed loop exists; impairment evidence decides its final constants. |
| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Consumer-owned 100 ms default/250 ms Raven proof; 25 ms HID expiry | Correct owner and bounded values; glass-to-glass measurement remains. |
| Input state | Coalesces replaceable motion/sensor/analog values | Epoch-scoped latest state supersedes stale state | Structural rather than polling-accidental. |
| Input edges | Button/type changes terminate batches; text is ordered | Ordered edge sequence, application ACK, bounded replay, epoch rebase | Structural; live virtual-HID impairment proof remains. |
| Staleness | Queue construction prevents much stale work | Receiver drops regressing timestamps and neutralizes after timeout | Keep the guard, but move freshness authority earlier into enqueue/supersession. |
| Timing | RTP/media clocks and local queue timing | source Unix nanoseconds plus receiver instants | Use monotonic source time plus an explicit clock-offset/uncertainty model; do not call raw wall-clock subtraction latency. |
| Security | Authenticated encryption in modern stream generations | CultNet typed transport policy varies by lane | Preserve typed schemas, but specify replay/authentication guarantees for hot media and HID lanes. |
@ -252,3 +256,10 @@ loss:
Muninn and Sleipnir already have enough organs to become strong. The next work
is to give each signal one explicit survival law and make the transport obey it.
The field comparison also exposed a generic transport invariant Moonlight's
long-running sessions force us to respect: bounded wire identifiers must wrap
deliberately. CultNet's fragmented-message ID previously saturated at the
maximum `u16`, so fragmented audio became unreassemblable after the boundary.
Wrap-to-one plus a boundary test removed the repeatable minute-eight collapse;
a subsequent 10-minute direct LAN run remained clean across the boundary.

30
docs/research/moonlight-reliability-acceptance-2026-07-16.md
View file

@ -10,14 +10,15 @@ permission to call the field path done.
|---|---|---|---|
| Experimental CultMesh/CultNet lineage | Odin vendored CultLib snapshot `8965f3c0` | Snapshot provenance plus CultNet/Muninn/Sleipnir suites | proved locally |
| Low-delay NVENC | Muninn controllable encoder | D3D11 desktop capture, no B frames/lookahead, two-frame VBV, clean Annex-B decode | proved locally |
| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default, bounded queues/cache, late-frame repair refusal | proved structurally |
| Recoverable video loss avoids decoder reset | Mimir parity/repair, Odin repair cache | Production XOR parity; live early-feedback fixture; repair expiry tests | proved locally |
| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default/250 ms Raven profile, bounded queues/cache/reliable expiry, late-frame repair refusal | proved structurally |
| Recoverable video loss avoids decoder reset | Mimir V4 block FEC/repair, Odin repair cache | Exhaustive 2..80 chunk, burst 1..8 tests in Rust/C++; completed-frame tombstones | proved locally |
| Decoder-chain loss forces immediate IDR | Muninn feedback owner and native encoder | Live 2560x1440 command produced IDR at frame 22 under a 600-frame GOP | proved locally |
| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Live 12-to-6 Mbps reconfiguration produced clean IDR at frame 35 | proved locally |
| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Half-ceiling start and protected cap; live 12-to-6 Mbps reconfiguration produced clean IDR | proved locally |
| Audio survives small erasure bursts | Odin 4+2 GF(256), Mimir FEC/reorder/concealment | Exhaustive one/two-erasure Rust and C++ tests; production DLL build | proved locally |
| Latest input state supersedes stale state | Muninn epoch/state sequence, Sleipnir cursor | Duplicate/reorder/stale epoch tests | proved locally |
| Button edges are not lost or retained forever | Muninn replay/app ACK/epoch rebase, Sleipnir ordered cursor | Quick-tap, ACK, overflow rebase, sparse-gap tests | proved locally |
| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | No current run: Raven WireGuard SSH/TCP 22 timed out | **missing** |
| Raven runtime is LAN-only and non-interrupting | Hidden Raven scheduled task, direct LAN SSH/media | One coherent Muninn/WASAPI/NVENC/FFmpeg tree; no WireGuard route; `-WindowStyle Hidden` | proved operationally |
| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | Direct LAN clean run 05:34:17–05:44:37: 79,200 records, 697,450,085 bytes, 18,588 recovered shards, zero decode/assembly/media/audio failures and zero sender capacity eviction; proxy matrix and long soak remain | **direct clean passed** |
## Field acceptance matrix
@ -38,6 +39,27 @@ timeline, not only proxy CSV counters.
| reconnect | old input epoch cannot act; neutral/current state within 100 ms; decoded video within 500 ms |
| mixed 60-minute soak | stable memory/handle/thread counts and no progressive A/V drift |
The first seeded 1% proxy run is a failed acceptance result, not a pass: 987 of
95,741 datagrams were dropped; in roughly one minute the receiver recorded 38
expired video assemblies and 56 skipped audio packets while remaining bounded
and avoiding decoder reset. CultNet fragmentation currently sits below
application-level FEC, so one lost transport fragment discards an entire FEC
shard. The next reliability cut is to make protected media shards fit one wire
datagram or move fragmentation above the erasure-code boundary.
## Field scars
- A fragmented CultNet RUDP session failed repeatably after roughly eight to
nine minutes because the `u16` fragment-set ID used saturating increment and
became permanently stuck at 65,535. Every later fragmented audio document
then collided in one receiver reassembly key. The allocator now wraps from
65,535 to 1, with a boundary test. The clean field run above crossed that
boundary without a single audio reorder, skip, stale packet, or failure.
- Audio and video producer ingress now use separate bounded channels. Video
capture can no longer block audio ingestion before the audio-priority send
scheduler sees it. Capacity-eviction counters proved zero during the clean
acceptance run.
## Completion rule
Moonlight-level performance is reasonably expected only after all local rows

260
native/obs_stem_source/mimir_obs_muninn_source.cpp
View file

@ -224,13 +224,28 @@ struct MuninnMediaPayload {
std::optional<uint64_t> dependency_frame_id;
uint16_t chunk_index = 0;
uint16_t chunk_count = 1;
uint16_t block_index = 0;
uint16_t block_count = 0;
uint16_t block_data_start = 0;
uint16_t block_data_count = 0;
uint16_t parity_index = 0;
uint16_t parity_count = 0;
bool video_fec_cauchy = false;
bool video_fec_blocked = false;
std::vector<uint32_t> chunk_payload_lengths;
std::vector<uint8_t> payload;
};
struct MuninnVideoFecBlockAssembly {
uint16_t block_index = 0;
uint16_t block_count = 0;
uint16_t data_start = 0;
uint16_t data_count = 0;
uint16_t parity_count = 0;
std::vector<uint32_t> chunk_payload_lengths;
std::map<uint16_t, std::vector<uint8_t>> parity_payloads;
};
struct MuninnVideoFrameAssembly {
std::string stream_id;
std::string session_id;
@ -241,10 +256,7 @@ struct MuninnVideoFrameAssembly {
uint16_t chunks_received = 0;
std::chrono::steady_clock::time_point started_at;
bool repair_requested = false;
std::vector<uint32_t> chunk_payload_lengths;
uint16_t parity_count = 0;
bool video_fec_cauchy = false;
std::map<uint16_t, std::vector<uint8_t>> parity_payloads;
std::map<uint16_t, MuninnVideoFecBlockAssembly> fec_blocks;
std::vector<std::vector<uint8_t>> chunks;
};
@ -255,9 +267,14 @@ struct MuninnVideoParityCacheEntry {
bool keyframe = false;
std::optional<uint64_t> dependency_frame_id;
uint16_t chunk_count = 0;
uint16_t block_index = 0;
uint16_t block_count = 0;
uint16_t block_data_start = 0;
uint16_t block_data_count = 0;
uint16_t parity_index = 0;
uint16_t parity_count = 0;
bool video_fec_cauchy = false;
bool video_fec_blocked = false;
std::chrono::steady_clock::time_point received_at;
std::vector<uint32_t> chunk_payload_lengths;
std::vector<uint8_t> payload;
@ -695,6 +712,47 @@ static std::optional<MuninnMediaPayload> decode_muninn_media_payload(const uint8
media.payload = record.read_bytes();
return media;
}
if (schema_id == "muninn.media_video_parity_shard.v4") {
if (record_len < 21) {
return std::nullopt;
}
MuninnMediaPayload media;
media.kind = MuninnMediaPayload::Kind::VideoParity;
media.stream_id = record.read_string();
media.session_id = record.read_string();
media.frame_id = record.read_unsigned();
record.skip(); // codec
record.skip(); // pts_ticks
record.skip(); // duration_ticks
record.skip(); // timebase_num
record.skip(); // timebase_den
media.keyframe = record.read_bool();
media.dependency_frame_id = record.read_optional_unsigned();
record.skip(); // deadline_ticks
media.chunk_count = static_cast<uint16_t>(record.read_unsigned());
media.block_index = static_cast<uint16_t>(record.read_unsigned());
media.block_count = static_cast<uint16_t>(record.read_unsigned());
media.block_data_start = static_cast<uint16_t>(record.read_unsigned());
media.block_data_count = static_cast<uint16_t>(record.read_unsigned());
media.parity_index = static_cast<uint16_t>(record.read_unsigned());
media.parity_count = static_cast<uint16_t>(record.read_unsigned());
media.video_fec_cauchy = true;
media.video_fec_blocked = true;
const uint32_t chunk_payload_bytes = static_cast<uint32_t>(record.read_unsigned());
const uint32_t last_chunk_payload_bytes = static_cast<uint32_t>(record.read_unsigned());
if (media.chunk_count == 0 || media.block_count == 0 ||
media.block_index >= media.block_count || media.block_data_count == 0 ||
media.block_data_start + media.block_data_count > media.chunk_count ||
media.parity_count < media.block_data_count || media.parity_count > 8 ||
media.parity_index >= media.parity_count || chunk_payload_bytes == 0 ||
last_chunk_payload_bytes == 0 || last_chunk_payload_bytes > chunk_payload_bytes) {
return std::nullopt;
}
media.chunk_payload_lengths.assign(media.block_data_count, chunk_payload_bytes);
media.chunk_payload_lengths.back() = last_chunk_payload_bytes;
media.payload = record.read_bytes();
return media;
}
if (schema_id == "muninn.media_audio_packet.v1") {
if (record_len < 10) {
return std::nullopt;
@ -984,6 +1042,7 @@ private:
}
const uint8_t version = packet[4];
const uint8_t packet_type = packet[5];
const bool reliable = (packet[6] & 0x01) != 0;
const uint8_t header_bytes = packet[7];
const uint32_t connection_id = read_be32(packet + 8);
const uint32_t sequence = read_be32(packet + 12);
@ -1009,6 +1068,26 @@ private:
}
if (packet_type == 1) {
const auto connect_endpoint =
std::make_pair(remote.sin_addr.s_addr, remote.sin_port);
auto &last_connect_sequence = is_audio_connection
? last_audio_connect_sequence_
: last_video_connect_sequence_;
auto &last_connect_endpoint = is_audio_connection
? last_audio_connect_endpoint_
: last_video_connect_endpoint_;
if (last_connect_sequence == sequence &&
last_connect_endpoint == connect_endpoint) {
connection_ack_tracker.remember(sequence);
const auto [ack, ack_mask] = connection_ack_tracker.state();
send_control_packet(2, 0x03, connection_id, bridge_sequence++, ack, ack_mask, remote);
blog(LOG_INFO,
"Muninn RUDP bridge idempotently accepted duplicate %s connection sequence %u",
is_audio_connection ? "audio" : "video", sequence);
return;
}
last_connect_sequence = sequence;
last_connect_endpoint = connect_endpoint;
const bool was_connected = media_connected_;
video_assemblies_.clear();
video_parity_cache_.clear();
@ -1054,13 +1133,18 @@ private:
return;
}
if (is_video_connection) {
observe_media_sequence(sequence);
}
if (packet_type == 3) {
auto &logged_first_data = is_audio_connection ? logged_first_audio_data_ : logged_first_video_data_;
if (!logged_first_data) {
logged_first_data = true;
blog(LOG_INFO,
"Muninn RUDP bridge first %s data sequence=%u payload=%u fragment_id=%u fragment_index=%u fragment_count=%u datagram=%zu",
is_audio_connection ? "audio" : "video", sequence, payload_len,
"Muninn RUDP bridge first %s data sequence=%u reliable=%s payload=%u fragment_id=%u fragment_index=%u fragment_count=%u datagram=%zu",
is_audio_connection ? "audio" : "video", sequence,
reliable ? "true" : "false", payload_len,
static_cast<unsigned int>(read_be16(packet + 24)),
static_cast<unsigned int>(read_be16(packet + 26)),
static_cast<unsigned int>(fragment_count), size);
@ -1102,17 +1186,22 @@ private:
blog(LOG_INFO, "Muninn RUDP bridge adopted existing media sender at sequence %u", sequence);
}
connection_ack_tracker.remember(sequence);
if (is_video_connection) {
observe_media_sequence(sequence);
}
const std::string channel_id(
reinterpret_cast<const char *>(packet + 36),
reinterpret_cast<const char *>(packet + 36 + channel_len));
if (channel_id != parts_.channel_id) {
const auto [ack, ack_mask] = connection_ack_tracker.state();
send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote);
const bool admitted_media_channel = channel_id == parts_.channel_id ||
(parts_.channel_id == "media" && channel_id == "realtime");
if (!admitted_media_channel) {
if (reliable) {
send_control_packet(4, 0x00, connection_id, bridge_sequence++, sequence, 0, remote);
}
return;
}
if (reliable) {
// Receipt belongs to the transport layer. Do not hold its ACK
// behind MessagePack decoding, FEC recovery, or decoder I/O.
send_control_packet(4, 0x00, connection_id, bridge_sequence++, sequence, 0, remote);
}
std::optional<std::pair<uint32_t, std::vector<uint8_t>>> delivered;
if (fragment_count == 0) {
delivered = std::make_pair(
@ -1138,8 +1227,6 @@ private:
audio_addr,
bridge_sequence);
}
const auto [ack, ack_mask] = connection_ack_tracker.state();
send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote);
}
}
@ -1567,11 +1654,21 @@ private:
std::optional<std::vector<uint8_t>> assemble_video_parity_payload(const MuninnMediaPayload &media, uint32_t &bridge_sequence)
{
++video_parity_received_;
const uint16_t block_index = media.video_fec_blocked ? media.block_index : 0;
const uint16_t block_count = media.video_fec_blocked ? media.block_count : 1;
const uint16_t block_data_start = media.video_fec_blocked ? media.block_data_start : 0;
const uint16_t block_data_count = media.video_fec_blocked ? media.block_data_count : media.chunk_count;
if (media.chunk_count == 0 ||
block_count == 0 ||
block_index >= block_count ||
block_data_count == 0 ||
static_cast<uint32_t>(block_data_start) + block_data_count > media.chunk_count ||
media.parity_count == 0 ||
media.parity_index >= media.parity_count ||
media.parity_count > media.chunk_count ||
media.chunk_payload_lengths.size() != media.chunk_count ||
(media.video_fec_blocked
? (media.parity_count < block_data_count || media.parity_count > 8)
: media.parity_count > block_data_count) ||
media.chunk_payload_lengths.size() != block_data_count ||
media.payload.empty()) {
++payloads_decoded_dropped_;
++video_parity_unusable_;
@ -1599,24 +1696,27 @@ private:
log_bridge_pressure("dropped video parity with mixed metadata");
return std::nullopt;
}
assembly.chunk_payload_lengths = media.chunk_payload_lengths;
if (assembly.parity_count != 0 && assembly.parity_count != media.parity_count) {
auto &block = assembly.fec_blocks[block_index];
if (block.data_count == 0) {
block.block_index = block_index;
block.block_count = block_count;
block.data_start = block_data_start;
block.data_count = block_data_count;
block.parity_count = media.parity_count;
block.chunk_payload_lengths = media.chunk_payload_lengths;
}
if (block.block_count != block_count ||
block.data_start != block_data_start ||
block.data_count != block_data_count ||
block.parity_count != media.parity_count ||
block.chunk_payload_lengths != media.chunk_payload_lengths) {
video_assemblies_.erase(key);
++payloads_decoded_dropped_;
++video_parity_unusable_;
log_bridge_pressure("dropped video parity with mixed stripe count");
log_bridge_pressure("dropped video parity with mixed block metadata");
return std::nullopt;
}
if (assembly.parity_count != 0 && assembly.video_fec_cauchy != media.video_fec_cauchy) {
video_assemblies_.erase(key);
++payloads_decoded_dropped_;
++video_parity_unusable_;
log_bridge_pressure("dropped video parity with mixed FEC scheme");
return std::nullopt;
}
assembly.parity_count = media.parity_count;
assembly.video_fec_cauchy = media.video_fec_cauchy;
assembly.parity_payloads[media.parity_index] = media.payload;
block.parity_payloads[media.parity_index] = media.payload;
if (try_recover_video_assembly_from_parity(assembly)) {
++video_parity_recovered_;
return finish_video_assembly(key, assembly);
@ -1634,9 +1734,14 @@ private:
entry.keyframe = media.keyframe;
entry.dependency_frame_id = media.dependency_frame_id;
entry.chunk_count = media.chunk_count;
entry.block_index = media.video_fec_blocked ? media.block_index : 0;
entry.block_count = media.video_fec_blocked ? media.block_count : 1;
entry.block_data_start = media.video_fec_blocked ? media.block_data_start : 0;
entry.block_data_count = media.video_fec_blocked ? media.block_data_count : media.chunk_count;
entry.parity_index = media.parity_index;
entry.parity_count = media.parity_count;
entry.video_fec_cauchy = media.video_fec_cauchy;
entry.video_fec_blocked = media.video_fec_blocked;
entry.received_at = std::chrono::steady_clock::now();
entry.chunk_payload_lengths = media.chunk_payload_lengths;
entry.payload = media.payload;
@ -1655,15 +1760,34 @@ private:
if (entry.chunk_count == assembly.chunk_count &&
entry.keyframe == assembly.keyframe &&
entry.dependency_frame_id == assembly.dependency_frame_id &&
entry.block_count != 0 &&
entry.block_index < entry.block_count &&
entry.block_data_count != 0 &&
static_cast<uint32_t>(entry.block_data_start) + entry.block_data_count <= assembly.chunk_count &&
entry.parity_count != 0 &&
entry.parity_index < entry.parity_count &&
(assembly.parity_count == 0 ||
(assembly.parity_count == entry.parity_count &&
assembly.video_fec_cauchy == entry.video_fec_cauchy))) {
assembly.chunk_payload_lengths = entry.chunk_payload_lengths;
assembly.parity_count = entry.parity_count;
assembly.video_fec_cauchy = entry.video_fec_cauchy;
assembly.parity_payloads[entry.parity_index] = entry.payload;
(entry.video_fec_blocked
? (entry.parity_count >= entry.block_data_count && entry.parity_count <= 8)
: entry.parity_count <= entry.block_data_count) &&
entry.chunk_payload_lengths.size() == entry.block_data_count) {
auto &block = assembly.fec_blocks[entry.block_index];
if (block.data_count == 0) {
block.block_index = entry.block_index;
block.block_count = entry.block_count;
block.data_start = entry.block_data_start;
block.data_count = entry.block_data_count;
block.parity_count = entry.parity_count;
block.chunk_payload_lengths = entry.chunk_payload_lengths;
}
if (block.block_count == entry.block_count &&
block.data_start == entry.block_data_start &&
block.data_count == entry.block_data_count &&
block.parity_count == entry.parity_count &&
block.chunk_payload_lengths == entry.chunk_payload_lengths) {
block.parity_payloads[entry.parity_index] = entry.payload;
} else {
++video_parity_unusable_;
}
} else {
++video_parity_unusable_;
}
@ -1707,22 +1831,35 @@ private:
bool try_recover_video_assembly_from_parity(MuninnVideoFrameAssembly &assembly)
{
if (assembly.parity_count == 0 ||
!assembly.video_fec_cauchy ||
assembly.parity_payloads.empty() ||
assembly.chunk_payload_lengths.size() != assembly.chunk_count ||
assembly.chunks.size() != assembly.chunk_count) {
if (assembly.fec_blocks.empty() || assembly.chunks.size() != assembly.chunk_count) {
return false;
}
const uint16_t missing = assembly.chunk_count - assembly.chunks_received;
if (!muninn_video_fec::recover(
assembly.chunks,
assembly.chunk_payload_lengths,
assembly.parity_count,
assembly.parity_payloads)) {
return false;
for (auto &[_, block] : assembly.fec_blocks) {
if (block.data_count == 0 || block.parity_count == 0 ||
block.chunk_payload_lengths.size() != block.data_count ||
static_cast<uint32_t>(block.data_start) + block.data_count > assembly.chunk_count) {
continue;
}
uint16_t missing = 0;
for (uint16_t local_index = 0; local_index < block.data_count; ++local_index) {
const auto &chunk = assembly.chunks[block.data_start + local_index];
if (chunk.empty()) {
++missing;
}
}
if (missing == 0 || missing > block.parity_payloads.size()) {
continue;
}
if (!muninn_video_fec::recover_block(
assembly.chunks,
block.data_start,
block.chunk_payload_lengths,
block.parity_count,
block.parity_payloads)) {
continue;
}
assembly.chunks_received += missing;
}
assembly.chunks_received += missing;
return assembly.chunks_received == assembly.chunk_count;
}
@ -1831,8 +1968,22 @@ private:
std::chrono::steady_clock::time_point now) const
{
const auto age_ms = std::chrono::duration_cast<std::chrono::milliseconds>(now - assembly.started_at).count();
std::ostringstream fec_state;
for (const auto &[block_index, block] : assembly.fec_blocks) {
size_t missing = 0;
for (uint16_t local = 0; local < block.data_count; ++local) {
const size_t global = static_cast<size_t>(block.data_start) + local;
if (global >= assembly.chunks.size() || assembly.chunks[global].empty()) {
++missing;
}
}
if (fec_state.tellp() > 0) {
fec_state << ',';
}
fec_state << block_index << ':' << missing << '/' << block.parity_payloads.size();
}
blog(LOG_WARNING,
"Muninn RUDP bridge expired video frame id=%llu keyframe=%s dependency=%s chunks=%u/%u missing=%zu age_ms=%lld repair_requested=%s parity_present=%s",
"Muninn RUDP bridge expired video frame id=%llu keyframe=%s dependency=%s chunks=%u/%u missing=%zu age_ms=%lld repair_requested=%s parity_present=%s fec_missing_available=%s",
static_cast<unsigned long long>(assembly.frame_id),
assembly.keyframe ? "true" : "false",
assembly.dependency_frame_id.has_value() ? "true" : "false",
@ -1841,7 +1992,8 @@ private:
missing_chunk_keys.size(),
static_cast<long long>(age_ms),
assembly.repair_requested ? "true" : "false",
assembly.parity_payloads.empty() ? "false" : "true");
assembly.fec_blocks.empty() ? "false" : "true",
fec_state.str().c_str());
}
void request_video_chunk_repair_if_due(MuninnVideoFrameAssembly &assembly,
@ -2042,6 +2194,10 @@ private:
SOCKET socket_ = INVALID_SOCKET;
std::optional<sockaddr_in> last_remote_;
std::optional<sockaddr_in> last_audio_remote_;
std::optional<uint32_t> last_video_connect_sequence_;
std::optional<uint32_t> last_audio_connect_sequence_;
std::optional<std::pair<uint32_t, uint16_t>> last_video_connect_endpoint_;
std::optional<std::pair<uint32_t, uint16_t>> last_audio_connect_endpoint_;
#endif
std::atomic<bool> running_{false};
std::atomic<uint64_t> started_ms_{0};
@ -2781,7 +2937,7 @@ static std::string expected_rudp_stream_url(const MuninnSource *ctx, const std::
{
const auto reliable_expire_after_ms = std::max<uint32_t>(1, ctx->rudp_latency_budget_ms);
return "rudp://" + ctx->media_target_host + ":" + std::to_string(ctx->media_port) + "/" + stream_id +
"?channel=realtime&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" +
"?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" +
std::to_string(reliable_expire_after_ms) + "&assembly_deadline_ms=" + std::to_string(ctx->rudp_latency_budget_ms) +
"&gap_wait_ms=16&audio_reorder_ms=" + std::to_string(ctx->rudp_audio_reorder_ms);
}

2
native/obs_stem_source/muninn_rudp_url.h
View file

@ -18,7 +18,7 @@ inline constexpr uint32_t DefaultSenderResendDelayMs = 5;
inline constexpr uint32_t MaxLatencyBudgetMs = 500;
struct RudpUrlParts {
std::string channel_id = "realtime";
std::string channel_id = "media";
uint16_t port = 0;
uint16_t video_local_port = 0;
uint16_t audio_local_port = 0;

20
native/obs_stem_source/muninn_video_fec.h
View file

@ -158,4 +158,24 @@ inline bool recover(std::vector<std::vector<uint8_t>> &chunks,
return true;
}
inline bool recover_block(std::vector<std::vector<uint8_t>> &frame_chunks,
uint16_t data_start,
const std::vector<uint32_t> &block_lengths,
uint16_t parity_count,
const std::map<uint16_t, std::vector<uint8_t>> &parity)
{
if (block_lengths.empty() ||
static_cast<size_t>(data_start) + block_lengths.size() > frame_chunks.size())
return false;
std::vector<std::vector<uint8_t>> block_chunks;
block_chunks.reserve(block_lengths.size());
for (size_t index = 0; index < block_lengths.size(); ++index)
block_chunks.push_back(frame_chunks[data_start + index]);
if (!recover(block_chunks, block_lengths, parity_count, parity))
return false;
for (size_t index = 0; index < block_lengths.size(); ++index)
frame_chunks[data_start + index] = std::move(block_chunks[index]);
return true;
}
} // namespace muninn_video_fec

2
native/obs_stem_source/tests/muninn_rudp_url_tests.cpp
View file

@ -42,7 +42,7 @@ int main()
const auto fallback = muninn_rudp_url::parse("rudp://192.168.1.66:5204/muninn.raven.av.rudp");
require(fallback.has_value(), "minimal RUDP URL did not parse");
require(fallback->channel_id == "realtime", "minimal RUDP URL did not use the experimental realtime lane");
require(fallback->channel_id == "media", "minimal RUDP URL did not use bounded reliable media delivery");
require(fallback->connection_id == muninn_rudp_url::DefaultMediaRudpConnectionId,
"minimal RUDP URL did not keep the default connection id");
require(fallback->audio_connection_id == muninn_rudp_url::DefaultAudioRudpConnectionId,

61
native/obs_stem_source/tests/muninn_video_fec_tests.cpp
View file

@ -46,5 +46,66 @@ int main()
for (uint16_t index = 0; index <= parity_count; ++index)
excessive[index].clear();
require(!muninn_video_fec::recover(excessive, lengths, parity_count, parity));
constexpr uint16_t block_data = 8;
constexpr uint16_t frame_data = 80;
constexpr uint16_t block_count = frame_data / block_data;
constexpr size_t block_shard_bytes = 96;
std::vector<std::vector<uint8_t>> frame(frame_data, std::vector<uint8_t>(block_shard_bytes));
for (uint16_t data = 0; data < frame_data; ++data) {
for (size_t offset = 0; offset < block_shard_bytes; ++offset)
frame[data][offset] = static_cast<uint8_t>((data * 31 + offset * 13) & 0xff);
}
std::vector<std::map<uint16_t, std::vector<uint8_t>>> block_parity(block_count);
for (uint16_t block = 0; block < block_count; ++block) {
for (uint16_t row = 0; row < block_data; ++row) {
auto &shard = block_parity[block][row];
shard.assign(block_shard_bytes, 0);
for (uint16_t local = 0; local < block_data; ++local) {
const auto &data = frame[block * block_data + local];
const uint8_t coefficient = muninn_video_fec::coefficient(row, block_data, local);
for (size_t offset = 0; offset < block_shard_bytes; ++offset)
shard[offset] ^= muninn_video_fec::multiply(data[offset], coefficient);
}
}
}
struct WireShard {
bool parity;
uint16_t block;
uint16_t local;
};
std::vector<WireShard> wire;
for (uint16_t local = 0; local < block_data; ++local) {
for (uint16_t block = 0; block < block_count; ++block)
wire.push_back({false, block, local});
for (uint16_t block = 0; block < block_count; ++block)
wire.push_back({true, block, local});
}
require(wire.size() == frame_data * 2);
const std::vector<uint32_t> block_lengths(block_data, block_shard_bytes);
for (size_t burst = 1; burst <= 54; ++burst) {
for (size_t start = 0; start + burst <= wire.size(); ++start) {
auto damaged = frame;
auto available_parity = block_parity;
for (size_t index = start; index < start + burst; ++index) {
const auto shard = wire[index];
if (shard.parity)
available_parity[shard.block].erase(shard.local);
else
damaged[shard.block * block_data + shard.local].clear();
}
const auto started = std::chrono::steady_clock::now();
for (uint16_t block = 0; block < block_count; ++block) {
require(muninn_video_fec::recover_block(
damaged,
block * block_data,
block_lengths,
block_data,
available_parity[block]));
}
require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50));
require(damaged == frame);
}
}
return 0;
}

36
notes/current-system-map.md
View file

@ -337,6 +337,42 @@ not become the synchronized program authority.
## CultMesh Media Bridge
The live Raven-to-Starfire program path is Muninn's typed CultNet media body:
```mermaid
flowchart TD
A["Raven D3D11 desktop capture"] --> B["Long-lived NVENC encoder"]
C["Raven WASAPI loopback"] --> D["10 ms float PCM packetizer"]
B --> E["V4 8+8 block FEC and realtime video lane"]
D --> F["4+2 audio FEC and realtime audio lane"]
E --> G["Starfire OBS Muninn receiver udp/5200"]
F --> G
G --> H["Frame assembly / FEC / IDR feedback"]
G --> I["40 ms audio reorder / concealment"]
H --> J["OBS-local video decoder socket"]
I --> K["OBS-local audio decoder socket"]
```
Owner: Muninn owns Raven capture children, access-unit/audio packetization,
CultNet delivery, repair material, and encoder feedback response. The Mimir OBS
plugin owns receipt admission, frame/FEC assembly, audio continuity, and local
OBS lowering. OBS owns decode and composition. Odin/CultMesh discovery names
the provider surface; neither discovery nor the compatibility decoder sockets
own hot media truth.
Invariant: video packets are useful only before their assembly deadline.
Canonical video and V4 parity use unreliable `realtime`; block FEC, selective
repair, and IDR recovery own loss. Canonical audio and 4+2 parity also use
`realtime`; reorder and bounded concealment own playout continuity. Completed
video frames are tombstoned so late shards cannot
become a second assembly owner. Raven uses direct LAN only and starts through a
hidden interactive-token scheduled task; no WireGuard hop or foreground
terminal participates.
The older `Mimir.CultMeshMedia` relay description below is retained as a
non-authoritative historical/body-bridge map. It does not own the active Muninn
OBS feed.
```mermaid
flowchart TD
A["Raven FFmpeg desktop + WASAPI loopback mux"] --> B["Mimir.CultMeshMedia send"]

67
state/map.yaml
View file

@ -87,41 +87,32 @@ invariants:
are temporary smoke/bootstrap edges only.
accepted_design:
- Muninn video FEC is now typed as `muninn.media_video_parity_shard.v3`.
Odin emits 16 Cauchy GF(256) parity shards around each multi-chunk access
unit, so any erasure set up to the received parity count is recoverable;
Mimir's OBS receiver owns the matching bounded matrix recovery before the
assembly deadline. V2 stripe parity is never reinterpreted as V3. Native
receiver tests prove consecutive bursts from one through sixteen shards,
including production-sized 64 by 1400-byte frames under a 50 ms per-frame
recovery bound. Odin's 180 Muninn tests pass. A per-byte elimination draft
was rejected after one recovery stalled the receiver thread; production
inverts the coefficient matrix once per frame and uses table-driven field
arithmetic. Direct Raven LAN evidence exposed two distinct faults. First,
successful early FEC completion did not tombstone the frame, so late
original chunks resurrected a partial ghost assembly and manufactured loss
feedback; the receiver now remembers completed frame keys and ignores that
residue. A seven-minute proof then kept every video failure counter at zero
while recovering 12,504 frames. Second, a genuine 24-datagram tail burst
left one frame at 18/33 chunks: the leading parity bookend supplied only
eight equations and the trailing eight were lost with fifteen data shards.
Repair retransmission is capped behind fresh media, IDR requests are
coalesced to 500 ms, and video datagrams use a 50 microsecond spin pacer,
but Moonlight-style independently recoverable FEC blocks remain required
before the clean ten-minute gate can pass. An experimental weighted audio/
video scheduler was deleted after its accumulated audio debt caused bounded
queue eviction; audio retains strict latency priority.
Normal frames are forwarded immediately; the recovery deadline is not a
playout buffer. The ten-minute and impaired acceptance gates remain open.
- Muninn video FEC is typed as `muninn.media_video_parity_shard.v4`.
Odin divides each access unit into independent blocks of at most eight data
shards and emits eight Cauchy GF(256) parity rows for every block, including
a short tail. Data/parity lanes are round-robin scheduled per block. Mimir's
OBS receiver owns bounded block recovery before the assembly deadline,
forwards only the canonical completed frame, and tombstones completed frame
keys so late shards cannot resurrect ghost assemblies. V2/V3 remain legacy
decode shapes and never own V4 behavior. Rust tests exhaust every burst of
one through eight packets at every alignment for chunk counts 2 through 80;
native tests prove the matching receiver. Canonical video and parity now use
CultNet `realtime` without per-packet reliable queues. Repair is selective
and behind fresh media; unrecoverable dependency damage requests a live
NVENC IDR. Normal frames forward immediately; the recovery deadline is not
a playout buffer. Direct Raven LAN validation is active; impaired and soak
gates remain open until recorded in the acceptance ledger.
- Muninn now closes receiver congestion pressure into the live NVENC context.
Every 500 ms it applies multiplicative 15 percent backoff for late frames,
requested recovery, deferred repair, sender queue loss, or combined
jitter/decode pressure above 75 percent of the consumer deadline. Recovery
adds only five percent of the configured ceiling after each two stable
seconds, with a one-quarter/1 Mbps floor. A live 2560x1440 D3D11 proof
adds one-fiftieth of a safe cap after each ten stable seconds. The safe cap
is half of the requested encoder ceiling, with a one-quarter/1 Mbps
floor. Startup begins at half ceiling because fixed block parity can roughly
double wire rate. A live 2560x1440 D3D11 proof
changed 12 Mbps to 6 Mbps at frame 35; FFmpeg/NVENC reconfigured the same
session, emitted an IDR at frame 35, and the Annex-B stream decoded cleanly.
Muninn's full suite now passes 180 tests.
Muninn's full suite now passes 184 tests.
- Receiver-driven video recovery now reaches the actual NVENC frame owner.
Odin's `native/muninn-video-encoder` is a long-lived libavdevice desktop
capture and `h264_nvenc` process; a new decode-chain invalidation edge makes
@ -130,8 +121,9 @@ accepted_design:
requested second IDR at frame 22, and its Annex-B stream decoded cleanly.
The normal quarter-second scheduled IDR remains independent fallback.
The repeatable bundle includes the OBS experimental FFmpeg dependency
snapshot. Raven installation and real impairment acceptance remain pending
because Raven's documented WireGuard SSH route timed out on 2026-07-16.
snapshot. Raven now runs the bundle over direct LAN from a hidden scheduled
task; WireGuard is not a runtime or administration path. Real impairment
acceptance remains pending.
- Muninn input now bounds disconnected unacknowledged button history at 256
edges. Overflow rotates the controller epoch and rebases Sleipnir from the
next complete latest-state frame, rather than dropping an arbitrary press
@ -140,6 +132,8 @@ accepted_design:
- The 2026-07-16 loss-recovery cut is now live on both sides of the Muninn
audio boundary. Odin emits typed `muninn.media_audio_parity_shard.v1`
records using fixed 4+2 GF(256) parity for constant-size 10 ms PCM packets.
Canonical audio and parity use `realtime`; 4+2 FEC, reorder, and bounded
concealment own continuity without a media ACK/retransmit window.
Mimir's production OBS receiver retains a bounded recent data window,
reconstructs up to two erased packets per block before the 40 ms playout
deadline, and then hands recovered packets to the same reorder queue that
@ -168,11 +162,16 @@ accepted_design:
Sleipnir against the experimental CultLib runtime snapshot at `8965f3c0`.
HID latest state and ordered button edges have separate epoch/sequence and
application-ack semantics; wall timestamps no longer order input. Muninn
media uses CultNet's unreliable `realtime` lane rather than the experimental
reliable-without-expiry `media` lane, defaults LAN usefulness to 100 ms,
video and audio bodies use CultNet's unreliable `realtime` lane; reliable
delivery remains control/feedback authority. LAN usefulness
defaults to 100 ms,
bounds producer/pending queues, expires repair material, refuses repair for
receiver-late frames, and reserves keyframe requests for decode-chain
invalidation. Odin records the live authority and remaining gaps in
invalidation. CultNet fragmented-message IDs wrap from 65535 to 1;
saturation at 65535 previously caused every later fragmented audio
document to collide after roughly eight minutes. Audio and video producer
ingress are separately bounded before the audio-priority scheduler, with
explicit eviction counts. Odin records the live authority and remaining gaps in
`docs/realtime-delivery-authority.md`.
- The Moonlight comparison at
`docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md` establishes