Muninn/docs/muninn-media-streaming.md

Muninn Media Streaming

Muninn live A/V is the stress organ for CultMesh transport. The goal is not "make OBS show something once." The goal is encoded game video and audio moving between nearby PCs at LAN speed with bounded latency, observable ownership, and transport behavior that teaches CultMesh how to carry hot media.

Objective

Raven Muninn captures screen video and Realtek loopback audio, encodes them with hardware-first low-latency settings, publishes them through CultNet/CultMesh media streams, and lets the Muninn OBS plugin render the stream without owning capture, daemon lifecycle, or transport truth.

Current Mechanism

The current Raven A/V path is a compatibility path:

1. OBS reads Raven's muninn.obs_stream_catalog from the synced CultCache store.
2. The OBS plugin sends a typed muninn.capture_stream_command to Raven's Muninn daemon over CultNet RUDP.
3. Raven Muninn serve owns command acceptance and spawns a daemon-owned activation child for the requested stream.
4. The activation child starts WASAPI loopback capture and FFmpeg.
5. FFmpeg muxes encoded video and audio into MPEG-TS on stdout.
6. Muninn slices that byte stream into fixed-size chunks and sends those chunks over the RUDP media channel.
7. The OBS plugin forwards received bytes into a local FFmpeg source.

That path proved activation, capture, hardware encode, and cross-machine delivery. It is not the final media architecture. MPEG-TS byte chunks over an unordered lossy hot path give CultMesh no frame identity, no deadline, no media dependency graph, and no clean way to choose between retransmit, conceal, discard, keyframe request, or audio/video resync.

Invariants

• Muninn owns local capture and stream activation for the machine body where the sensors live.
• OBS and Mimir are consumers. They may request a stream and report receiver health, but they do not start capture by implication and do not own Raven daemon state.
• Idunn owns daemon supervision and health pressure. It does not infer that liveness means screen/audio capture should be burning.
• CultMesh owns live media stream semantics: stream identity, frame identity, timestamps, deadlines, dependencies, channel policy, and receiver feedback.
• CultCache records are receipts, catalog entries, and operator/debug state. They are not the hot media lane.
• Audio and video have separate clocks and recovery policy. They may share a session, but they must not be hidden inside an opaque transport byte soup.
• Transport reliability is deadline-bound. Late media is damage, not treasure.

Intended Change

Replace "MPEG-TS stdout sliced into RUDP packets" with "codec access units and audio packets published as typed CultMesh media frames."

The stream owner should emit media documents shaped around decisions the transport and decoder actually need:

muninn.media_video_access_unit.v1
  stream_id
  session_id
  frame_id
  codec
  pts_ticks
  duration_ticks
  timebase_num
  timebase_den
  keyframe
  dependency_frame_id
  deadline_ticks
  chunk_index
  chunk_count
  payload

muninn.media_audio_packet.v1
  stream_id
  session_id
  packet_id
  codec
  pts_ticks
  duration_ticks
  timebase_num
  timebase_den
  deadline_ticks
  payload

muninn.media_receiver_feedback.v1
  stream_id
  session_id
  receiver_id
  highest_decodable_frame_id
  missing_frame_ids
  missing_video_chunk_keys
  late_frame_ids
  requested_keyframe
  jitter_us
  decode_queue_us

The exact schema names can move when promoted into the shared document catalog, but the ownership shape should not: frame/access-unit identity is load-bearing.

Authority Map

• Owner: Muninn capture runtime owns source capture, encode configuration, and emission of video access units plus audio packets for a requested stream.
• Inputs: explicit muninn.capture_stream_command, local capture devices, encoder capabilities, receiver feedback, stream policy, and Idunn-supervised daemon runtime state.
• Outputs: typed active stream receipts in CultCache, CultMesh media frames over CultNet RUDP, and typed receiver/transport health receipts.
• Derived state: OBS catalog entries are discovery hints; local FFmpeg bridge URLs are compatibility lowering details; packet counters and logs are observability only.
• Forbidden writers: OBS source settings, local UDP bridge behavior, scheduled task wrappers, health scripts, and replayed command receipts must not decide capture state, stream identity, frame order, or media repair policy.
• Shared paths: direct operator requests, OBS plugin requests, future Mimir requests, restart recovery, and reconnect recovery must all publish the same typed capture command and consume the same active stream receipts.
• Deletion line: delete the hot-path assumption that a media stream is an MPEG-TS byte stream. Keep it only as a named compatibility lowering until the OBS receiver can consume typed media frames directly.

Boring Stream Discovery Contract

The desired operator experience is boring: OBS asks the Verse for Muninns, the plugin shows live sources, and a selected source either activates or reports the specific owner that refused it. No local plugin fallback may invent Raven devices. A fake display:0 or Realtek loopback row is worse than an empty list because it makes stale state look selectable.

Live ownership:

• Owner: Muninn serve owns source inventory for the body where the devices exist. It publishes video and audio source ids, labels, command boundary, media profile, and current active sessions.
• Inputs: local display/audio enumeration, explicit configured source hints, daemon health, and accepted stream commands.
• Outputs: muninn.telemetry_surface.v1, gamecult.eve.provider_advertisement.v1, muninn.command_boundary.v1, muninn.transport_profile.v1, active muninn.capture_stream.v1 receipts, and the temporary muninn.obs_stream_catalog.v1 compatibility record.
• Derived state: the OBS dropdown is a lowering of discovered Muninn state. It is not an inventory owner, activation owner, or health owner.
• Forbidden writers: OBS defaults, local hard-coded device names, synced stale CultCache mirrors, scheduled-task launchers, and previous activation receipts must not create selectable devices or active streams.
• Shared paths: initial OBS load, refresh button, periodic refresh, source selection, reconnect, and OBS restart must all read the same live-discovered source list and issue the same typed capture command when activation is needed.
• Deletion line: the plugin may display discovery missing, but it must not synthesize Display 1 or Realtek loopback as if Raven advertised them. Until Odin/Verse discovery is the direct plugin input, the OBS catalog remains a compatibility projection of Muninn-owned live state, not its own truth. Compatibility store fallback is allowed only as availability fallback: the plugin must use the first current store that yields Muninn telemetry and must not merge stale stores into a more attractive fake inventory. A store older than the Muninn health freshness budget is not a discovery source; it is an outage receipt.

This contract also splits the media session truth:

• Video session: selected video source id, encoder profile, bitrate, frame timestamps, access-unit ids, keyframe policy, feedback pressure, and video receiver queue.
• Audio session: selected audio source id, sample format, packet timestamps, reorder budget, continuity/drop policy, and OBS audio output cadence.
• Combined OBS source: a convenience lowering that may request one video session and one audio session together. It does not merge their clocks or transport state into one opaque "A/V target".

Known-Good Checkpoint (2026-06-23)

The current system is in a usable compatibility state worth preserving before the next transport experiment.

What is working:

• Raven Muninn runs continuously as the capture owner and accepts stream activation over CultNet RUDP.
• OBS on Starfire shows the live-discovered Muninn source inventory instead of synthesizing fake devices from stale local defaults.
• Video and audio sessions activate separately through the combined OBS source lowering and the stream comes up cleanly again after reconnect.
• The OBS plugin currently receives the temporary live muninn.obs_stream_catalog.v1 projection over UDP 17874 and uses it as the live discovery source before any CultCache fallback.

Current compatibility details:

• Raven command ingress is currently bound to 0.0.0.0:17883.
• Starfire OBS listens for the temporary live OBS catalog on UDP 17874.
• The daemon serve loop must continue publishing the OBS catalog while idle; otherwise OBS falls back to stale store state and reports discovery-missing or shows dead inventory.
• The current OBS plugin is still a compatibility client. It does not yet host the Rust CultMesh runtime directly and it is not yet consuming Odin/Verse discovery as a first-class Verse client.

Known debt that remains intentionally visible:

• Idunn health publication can still time out independently of the media path. That is operational debt, not proof that Muninn discovery or streaming is down.
• The OBS catalog is still a temporary projection. The boring end state is live Verse discovery with no plugin-owned catalog receiver and no stale store merge behavior.

Codec Direction

Use vendor hardware encode before inventing a codec. NVIDIA's practical answer in this space is NVENC/NVDEC, low-latency encode presets, high LAN bitrate, short GOP or intra-refresh, no B-frames, no lookahead, and application-level transport that understands frame deadlines.

Shipping order:

1. H.264 NVENC low-latency as the compatibility baseline for broad decoder support, simple failure analysis, and OBS bridge stability.
2. HEVC NVENC low-latency when Raven encode and OBS decode both prove the target latency budget.
3. AV1 NVENC only after the exact Raven-to-OBS path proves encode, decode, and queue latency inside budget. AV1 can be the quality/bandwidth winner, but it must earn the low-latency lane on our hardware instead of getting promoted by codec fashion.

The codec should produce elementary video access units, not a muxed MPEG-TS program. Audio should use Opus low-delay packets unless the OBS lowering needs a temporary PCM/AAC compatibility bridge.

Recommended first LAN profile:

• Profile id: muninn.rudp.low_latency_h264_lan.v1.
• Video: H.264 NVENC, p5 (main.rs:9504), ultra-low-latency tune, CBR high enough that LAN bandwidth is not the constraint, no B-frames, no lookahead, short GOP or intra-refresh, keyframe actuation on feedback pressure (not built: keyframe requests are only counted and logged today; Cut 6 builds it).
• VBV: derive bufsize from roughly one frame of bitrate (bitrate_kbps / framerate) instead of a large fixed encoder reservoir. The point is bounded latency, not hoarding compressed video in a nicer hat.
• Audio: Opus low-delay when OBS lowering can consume it directly; otherwise a temporary PCM/AAC bridge with separate packet identity and clock.
• Transport: payload elementary access units as typed media records, never a muxed container as the unit of truth.
• Packet size: default typed media chunks are 848 bytes (main.rs:81). That leaves headroom for CultNet/MessagePack/RUDP headers inside a normal Ethernet MTU so the OBS bridge does not depend on IP fragmentation or private RUDP fragment reassembly. The sender also sets CultNet RUDP max_fragment_bytes to 1431 bytes (1472 byte IPv4 UDP payload budget minus 36 RUDP fixed header bytes minus the five-byte media channel id), so oversized typed records fragment in CultNet deliberately instead of falling through to accidental IP fragmentation.
• Resend cadence: sender RUDP media resends are scheduled every 30 ms (main.rs:94) while the packet is still inside its expiry, which is the request's latency budget (a request that names none gets 250 ms, main.rs:98, and the advertisement offers the same as default_latency_budget_ms).
• Send queue: media_send.rs owns it. The unit is a group, one access unit with its parity or one audio packet, stamped produced_at + latency budget when the packetizer emits it. A request's latency budget is clamped to 1..=2000 ms. The send loop moves one payload per step and picks it afresh each time: audio, then repairs, then video, so audio never waits behind more than the payload in progress, full socket or not. The group deadline is checked before every payload; a group that outlives it mid-send is abandoned and counted groups_cut_short. Queues are capped per kind (audio 256, video 512 groups) and drop the oldest group, counted as groups_dropped_audio and groups_dropped_video. Every access unit the sender drops (queue cap, expiry, packetizer overflow) has already consumed its frame id, so the receiver sees a gap; an overflow also discards the dropped frame's remaining slices instead of emitting them as a frame. A repair keeps the deadline of the frame it repairs, goes only to the receiver that asked, and is not sent after that deadline.
• Control: sender may adapt bitrate, keyframe cadence, chunk size, and playout budget from receiver feedback; the receiver must not silently stretch latency to preserve visual perfection.
• Receiver URL policy: RUDP stream URLs carry profile, sender_resend_delay_ms, reliable_expire_after_ms, assembly_deadline_ms, and gap_wait_ms so the OBS bridge receives timing budget from Muninn's stream profile instead of hard-coding receiver policy privately.

Motion Vectors And Reconstruction

Do not make CultMesh a bespoke game-streaming codec in the first shipping path. The motion-vector/delta idea is real, but it belongs as an augmentation around the hardware codec, not as the primary authority.

NVIDIA's relevant public pattern is:

• NVENC/NVDEC for hardware video encode and decode.
• Low-latency encode tuning, high refresh rate, adaptive bitrate, and explicit network/frame timing for cloud gaming.
• Foveated streaming for bandwidth reduction when a trusted focus region exists.
• Optical Flow / motion estimation hardware for interpolation, extrapolation, analysis, or concealment experiments.

For Muninn, that means:

• If we only have desktop capture pixels, trust NVENC for motion estimation and compression. Do not compute our own patch stream before the codec has failed a measured requirement.
• If a game or renderer we control can expose motion vectors, depth, regions of interest, or dirty rectangles, publish them as optional sideband media hints. They may guide encoder ROI, decoder concealment, frame extrapolation, or foveated quality allocation. They do not replace the encoded video access unit unless a separate research path proves that decoder under the same OBS latency budget.
• Stochastic reconstruction is allowed only inside a bounded playout budget. When the deadline expires, the receiver displays the best available frame, conceals damage, or requests a keyframe. It must not preserve dead packets by growing latency.

Useful references for this direction:

• NVIDIA Video Codec SDK presets describe low-latency and ultra-low-latency tuning plus preset tradeoffs for encoder tools such as GOP structure, B-frames, and lookahead: https://developer.nvidia.com/blog/introducing-video-codec-sdk-10-presets/
• NVIDIA CloudXR publicly frames low-latency spatial streaming around NVENC, adaptive bitrate, foveated streaming, diagnostics, and bidirectional data channels: https://docs.nvidia.com/cloudxr-sdk/latest/overview/overview.html
• NVIDIA's Optical Flow SDK exposes hardware motion vectors for real-time interpolation/extrapolation and related video processing: https://developer.nvidia.com/optical-flow-sdk

CultNet RUDP Media Policy

The RUDP media channel should become a media-aware lane, not merely a reliable unordered byte lane.

Required behavior:

• Frame-level fragmentation and reassembly keyed by (stream_id, session_id, frame_id).
• Deadline-aware retransmission: resend while useful, drop when the playout budget is gone.
• Receiver feedback that can NACK missing chunks, request a keyframe, and report jitter/decode queue pressure.
• Small playout buffer chosen by policy, for example 8-20 ms on a wired LAN.
• Optional FEC/parity for video chunks where one repair packet beats a resend.
• Separate audio policy that favors continuity and bounded drift over perfect packet recovery.

OBS Lowering

The OBS plugin should lower typed media frames into OBS-friendly video/audio. The clean target is direct decode from elementary streams using libavcodec or NVDEC where available, then hand decoded video frames and audio packets to OBS.

Temporary bridge allowed:

• Reconstruct an elementary stream from typed video access units.
• Feed the local FFmpeg source only as a compatibility decoder.
• Keep audio as a separate OBS audio path so audio sync is explicit instead of being hidden inside a damaged MPEG-TS stream.

The compatibility bridge must not own stream truth. It is a lowering target, not the protocol.

Cut Plan

1. Promote media-frame schemas in Odin/CultLib and add unit tests for encode/decode round trips.
2. Teach Muninn activation to emit video access units and audio packets instead of arbitrary MPEG-TS byte chunks.
3. Teach CultNet RUDP media receive to reassemble by frame and deadline, with explicit NACK/keyframe feedback.
4. Teach the OBS plugin to consume typed media frames and feed OBS through a direct decoder or a narrowly named compatibility elementary-stream bridge.
5. Remove the MPEG-TS byte-chunk hot path once the typed receiver is stable.

Verification

The rebuild is not proven by "OBS shows video once." Minimum proof:

• Raven serve remains idle until an explicit capture command arrives.
• OBS request activates Raven without manual Raven terminals.
• Sender logs and receiver telemetry agree on stream id, session id, frame id, codec, PTS, deadlines, drops, NACKs, and keyframe requests.
• A forced packet drop causes bounded damage and recovery, not OBS freeze.
• Audio remains present as an OBS audio source and stays within the chosen sync budget.
• Restarting OBS does not strand Raven capture or poison the next request with stale command receipts.
• Restarting Raven Muninn restores catalog/health without starting capture until requested.