12 KiB
Muninn Media Streaming
Muninn live A/V is the stress organ for CultMesh transport. The goal is not "make OBS show something once." The goal is encoded game video and audio moving between nearby PCs at LAN speed with bounded latency, observable ownership, and transport behavior that teaches CultMesh how to carry hot media.
Objective
Raven Muninn captures screen video and Realtek loopback audio, encodes them with hardware-first low-latency settings, publishes them through CultNet/CultMesh media streams, and lets the Muninn OBS plugin render the stream without owning capture, daemon lifecycle, or transport truth.
Current Mechanism
The current Raven A/V path is a compatibility path:
- OBS reads Raven's
muninn.obs_stream_catalogfrom the synced CultCache store. - The OBS plugin sends a typed
muninn.capture_stream_commandto Raven's Muninn daemon over CultNet RUDP. - Raven Muninn
serveowns command acceptance and spawns a daemon-owned activation child for the requested stream. - The activation child starts WASAPI loopback capture and FFmpeg.
- FFmpeg muxes encoded video and audio into MPEG-TS on stdout.
- Muninn slices that byte stream into fixed-size chunks and sends those chunks
over the RUDP
mediachannel. - The OBS plugin forwards received bytes into a local FFmpeg source.
That path proved activation, capture, hardware encode, and cross-machine delivery. It is not the final media architecture. MPEG-TS byte chunks over an unordered lossy hot path give CultMesh no frame identity, no deadline, no media dependency graph, and no clean way to choose between retransmit, conceal, discard, keyframe request, or audio/video resync.
Invariants
- Muninn owns local capture and stream activation for the machine body where the sensors live.
- OBS and Mimir are consumers. They may request a stream and report receiver health, but they do not start capture by implication and do not own Raven daemon state.
- Idunn owns daemon supervision and health pressure. It does not infer that liveness means screen/audio capture should be burning.
- CultMesh owns live media stream semantics: stream identity, frame identity, timestamps, deadlines, dependencies, channel policy, and receiver feedback.
- CultCache records are receipts, catalog entries, and operator/debug state. They are not the hot media lane.
- Audio and video have separate clocks and recovery policy. They may share a session, but they must not be hidden inside an opaque transport byte soup.
- Transport reliability is deadline-bound. Late media is damage, not treasure.
Intended Change
Replace "MPEG-TS stdout sliced into RUDP packets" with "codec access units and audio packets published as typed CultMesh media frames."
The stream owner should emit media documents shaped around decisions the transport and decoder actually need:
muninn.media_video_access_unit.v1
stream_id
session_id
frame_id
codec
pts_ticks
duration_ticks
timebase_num
timebase_den
keyframe
dependency_frame_id
deadline_ticks
chunk_index
chunk_count
payload
muninn.media_audio_packet.v1
stream_id
session_id
packet_id
codec
pts_ticks
duration_ticks
timebase_num
timebase_den
deadline_ticks
payload
muninn.media_receiver_feedback.v1
stream_id
session_id
receiver_id
highest_decodable_frame_id
missing_frame_ids
missing_video_chunk_keys
late_frame_ids
requested_keyframe
jitter_us
decode_queue_us
The exact schema names can move when promoted into the shared document catalog, but the ownership shape should not: frame/access-unit identity is load-bearing.
Authority Map
- Owner: Muninn capture runtime owns source capture, encode configuration, and emission of video access units plus audio packets for a requested stream.
- Inputs: explicit
muninn.capture_stream_command, local capture devices, encoder capabilities, receiver feedback, stream policy, and Idunn-supervised daemon runtime state. - Outputs: typed active stream receipts in CultCache, CultMesh media frames over CultNet RUDP, and typed receiver/transport health receipts.
- Derived state: OBS catalog entries are discovery hints; local FFmpeg bridge URLs are compatibility lowering details; packet counters and logs are observability only.
- Forbidden writers: OBS source settings, local UDP bridge behavior, scheduled task wrappers, health scripts, and replayed command receipts must not decide capture state, stream identity, frame order, or media repair policy.
- Shared paths: direct operator requests, OBS plugin requests, future Mimir requests, restart recovery, and reconnect recovery must all publish the same typed capture command and consume the same active stream receipts.
- Deletion line: delete the hot-path assumption that a media stream is an MPEG-TS byte stream. Keep it only as a named compatibility lowering until the OBS receiver can consume typed media frames directly.
Codec Direction
Use vendor hardware encode before inventing a codec. NVIDIA's practical answer in this space is NVENC/NVDEC, low-latency encode presets, high LAN bitrate, short GOP or intra-refresh, no B-frames, no lookahead, and application-level transport that understands frame deadlines.
Shipping order:
- H.264 NVENC low-latency as the compatibility baseline for broad decoder support, simple failure analysis, and OBS bridge stability.
- HEVC NVENC low-latency when Raven encode and OBS decode both prove the target latency budget.
- AV1 NVENC only after the exact Raven-to-OBS path proves encode, decode, and queue latency inside budget. AV1 can be the quality/bandwidth winner, but it must earn the low-latency lane on our hardware instead of getting promoted by codec fashion.
The codec should produce elementary video access units, not a muxed MPEG-TS program. Audio should use Opus low-delay packets unless the OBS lowering needs a temporary PCM/AAC compatibility bridge.
Recommended first LAN profile:
- Profile id:
muninn.rudp.low_latency_h264_lan.v1. - Video: H.264 NVENC,
p1, ultra-low-latency tune, CBR high enough that LAN bandwidth is not the constraint, no B-frames, no lookahead, short GOP or intra-refresh, periodic IDR/keyframe on feedback pressure. - VBV: derive
bufsizefrom roughly one frame of bitrate (bitrate_kbps / framerate) instead of a large fixed encoder reservoir. The point is bounded latency, not hoarding compressed video in a nicer hat. - Audio: Opus low-delay when OBS lowering can consume it directly; otherwise a temporary PCM/AAC bridge with separate packet identity and clock.
- Transport: payload elementary access units as typed media records, never a muxed container as the unit of truth.
- Packet size: default typed media chunks are
480bytes. That leaves headroom for CultNet/MessagePack/RUDP headers inside a normal Ethernet MTU so the OBS bridge does not depend on IP fragmentation or private RUDP fragment reassembly. The sender also sets CultNet RUDPmax_fragment_bytesto1431bytes (1472byte IPv4 UDP payload budget minus36RUDP fixed header bytes minus the five-bytemediachannel id), so oversized typed records fragment in CultNet deliberately instead of falling through to accidental IP fragmentation. - Resend cadence: sender RUDP media resends are scheduled every
5ms while the packet is still inside the75ms reliable-expiry budget. - Control: sender may adapt bitrate, keyframe cadence, chunk size, and playout budget from receiver feedback; the receiver must not silently stretch latency to preserve visual perfection.
- Receiver URL policy: RUDP stream URLs carry
profile,sender_resend_delay_ms,reliable_expire_after_ms,assembly_deadline_ms, andgap_wait_msso the OBS bridge receives timing budget from Muninn's stream profile instead of hard-coding receiver policy privately.
Motion Vectors And Reconstruction
Do not make CultMesh a bespoke game-streaming codec in the first shipping path. The motion-vector/delta idea is real, but it belongs as an augmentation around the hardware codec, not as the primary authority.
NVIDIA's relevant public pattern is:
- NVENC/NVDEC for hardware video encode and decode.
- Low-latency encode tuning, high refresh rate, adaptive bitrate, and explicit network/frame timing for cloud gaming.
- Foveated streaming for bandwidth reduction when a trusted focus region exists.
- Optical Flow / motion estimation hardware for interpolation, extrapolation, analysis, or concealment experiments.
For Muninn, that means:
- If we only have desktop capture pixels, trust NVENC for motion estimation and compression. Do not compute our own patch stream before the codec has failed a measured requirement.
- If a game or renderer we control can expose motion vectors, depth, regions of interest, or dirty rectangles, publish them as optional sideband media hints. They may guide encoder ROI, decoder concealment, frame extrapolation, or foveated quality allocation. They do not replace the encoded video access unit unless a separate research path proves that decoder under the same OBS latency budget.
- Stochastic reconstruction is allowed only inside a bounded playout budget. When the deadline expires, the receiver displays the best available frame, conceals damage, or requests a keyframe. It must not preserve dead packets by growing latency.
Useful references for this direction:
- NVIDIA Video Codec SDK presets describe low-latency and ultra-low-latency tuning plus preset tradeoffs for encoder tools such as GOP structure, B-frames, and lookahead: https://developer.nvidia.com/blog/introducing-video-codec-sdk-10-presets/
- NVIDIA CloudXR publicly frames low-latency spatial streaming around NVENC, adaptive bitrate, foveated streaming, diagnostics, and bidirectional data channels: https://docs.nvidia.com/cloudxr-sdk/latest/overview/overview.html
- NVIDIA's Optical Flow SDK exposes hardware motion vectors for real-time interpolation/extrapolation and related video processing: https://developer.nvidia.com/optical-flow-sdk
CultNet RUDP Media Policy
The RUDP media channel should become a media-aware lane, not merely a reliable
unordered byte lane.
Required behavior:
- Frame-level fragmentation and reassembly keyed by
(stream_id, session_id, frame_id). - Deadline-aware retransmission: resend while useful, drop when the playout budget is gone.
- Receiver feedback that can NACK missing chunks, request a keyframe, and report jitter/decode queue pressure.
- Small playout buffer chosen by policy, for example 8-20 ms on a wired LAN.
- Optional FEC/parity for video chunks where one repair packet beats a resend.
- Separate audio policy that favors continuity and bounded drift over perfect packet recovery.
OBS Lowering
The OBS plugin should lower typed media frames into OBS-friendly video/audio. The clean target is direct decode from elementary streams using libavcodec or NVDEC where available, then hand decoded video frames and audio packets to OBS.
Temporary bridge allowed:
- Reconstruct an elementary stream from typed video access units.
- Feed the local FFmpeg source only as a compatibility decoder.
- Keep audio as a separate OBS audio path so audio sync is explicit instead of being hidden inside a damaged MPEG-TS stream.
The compatibility bridge must not own stream truth. It is a lowering target, not the protocol.
Cut Plan
- Promote media-frame schemas in Odin/CultLib and add unit tests for encode/decode round trips.
- Teach Muninn activation to emit video access units and audio packets instead of arbitrary MPEG-TS byte chunks.
- Teach CultNet RUDP media receive to reassemble by frame and deadline, with explicit NACK/keyframe feedback.
- Teach the OBS plugin to consume typed media frames and feed OBS through a direct decoder or a narrowly named compatibility elementary-stream bridge.
- Remove the MPEG-TS byte-chunk hot path once the typed receiver is stable.
Verification
The rebuild is not proven by "OBS shows video once." Minimum proof:
- Raven
serveremains idle until an explicit capture command arrives. - OBS request activates Raven without manual Raven terminals.
- Sender logs and receiver telemetry agree on stream id, session id, frame id, codec, PTS, deadlines, drops, NACKs, and keyframe requests.
- A forced packet drop causes bounded damage and recovery, not OBS freeze.
- Audio remains present as an OBS audio source and stays within the chosen sync budget.
- Restarting OBS does not strand Raven capture or poison the next request with stale command receipts.
- Restarting Raven Muninn restores catalog/health without starting capture until requested.