Odin/docs/muninn-media-streaming.md

# Muninn Media Streaming

Muninn live A/V is the stress organ for CultMesh transport. The goal is not
"make OBS show something once." The goal is encoded game video and audio moving
between nearby PCs at LAN speed with bounded latency, observable ownership, and
transport behavior that teaches CultMesh how to carry hot media.

## Objective

Raven Muninn captures screen video and Realtek loopback audio, encodes them with
hardware-first low-latency settings, publishes them through CultNet/CultMesh
media streams, and lets the Muninn OBS plugin render the stream without owning
capture, daemon lifecycle, or transport truth.

## Current Mechanism

The current Raven A/V path is a compatibility path:

1. OBS reads Raven's `muninn.obs_stream_catalog` from the synced CultCache
   store.
2. The OBS plugin sends a typed `muninn.capture_stream_command` to Raven's
   Muninn daemon over CultNet RUDP.
3. Raven Muninn `serve` owns command acceptance and spawns a daemon-owned
   activation child for the requested stream.
4. The activation child starts WASAPI loopback capture and FFmpeg.
5. FFmpeg muxes encoded video and audio into MPEG-TS on stdout.
6. Muninn slices that byte stream into fixed-size chunks and sends those chunks
   over the RUDP `media` channel.
7. The OBS plugin forwards received bytes into a local FFmpeg source.

That path proved activation, capture, hardware encode, and cross-machine
delivery. It is not the final media architecture. MPEG-TS byte chunks over an
unordered lossy hot path give CultMesh no frame identity, no deadline, no media
dependency graph, and no clean way to choose between retransmit, conceal,
discard, keyframe request, or audio/video resync.

## Invariants

- Muninn owns local capture and stream activation for the machine body where
  the sensors live.
- OBS and Mimir are consumers. They may request a stream and report receiver
  health, but they do not start capture by implication and do not own Raven
  daemon state.
- Idunn owns daemon supervision and health pressure. It does not infer that
  liveness means screen/audio capture should be burning.
- CultMesh owns live media stream semantics: stream identity, frame identity,
  timestamps, deadlines, dependencies, channel policy, and receiver feedback.
- CultCache records are receipts, catalog entries, and operator/debug state.
  They are not the hot media lane.
- Audio and video have separate clocks and recovery policy. They may share a
  session, but they must not be hidden inside an opaque transport byte soup.
- Transport reliability is deadline-bound. Late media is damage, not treasure.

## Intended Change

Replace "MPEG-TS stdout sliced into RUDP packets" with "codec access units and
audio packets published as typed CultMesh media frames."

The stream owner should emit media documents shaped around decisions the
transport and decoder actually need:

```text
muninn.media_video_access_unit.v1
  stream_id
  session_id
  frame_id
  codec
  pts_ticks
  duration_ticks
  timebase_num
  timebase_den
  keyframe
  dependency_frame_id
  deadline_ticks
  chunk_index
  chunk_count
  payload

muninn.media_audio_packet.v1
  stream_id
  session_id
  packet_id
  codec
  pts_ticks
  duration_ticks
  timebase_num
  timebase_den
  deadline_ticks
  payload

muninn.media_receiver_feedback.v1
  stream_id
  session_id
  receiver_id
  highest_decodable_frame_id
  missing_frame_ids
  missing_video_chunk_keys
  late_frame_ids
  requested_keyframe
  jitter_us
  decode_queue_us
```

The exact schema names can move when promoted into the shared document catalog,
but the ownership shape should not: frame/access-unit identity is load-bearing.

## Authority Map

- Owner: Muninn capture runtime owns source capture, encode configuration, and
  emission of video access units plus audio packets for a requested stream.
- Inputs: explicit `muninn.capture_stream_command`, local capture devices,
  encoder capabilities, receiver feedback, stream policy, and Idunn-supervised
  daemon runtime state.
- Outputs: typed active stream receipts in CultCache, CultMesh media frames over
  CultNet RUDP, and typed receiver/transport health receipts.
- Derived state: OBS catalog entries are discovery hints; local FFmpeg bridge
  URLs are compatibility lowering details; packet counters and logs are
  observability only.
- Forbidden writers: OBS source settings, local UDP bridge behavior, scheduled
  task wrappers, health scripts, and replayed command receipts must not decide
  capture state, stream identity, frame order, or media repair policy.
- Shared paths: direct operator requests, OBS plugin requests, future Mimir
  requests, restart recovery, and reconnect recovery must all publish the same
  typed capture command and consume the same active stream receipts.
- Deletion line: delete the hot-path assumption that a media stream is an
  MPEG-TS byte stream. Keep it only as a named compatibility lowering until the
  OBS receiver can consume typed media frames directly.

## Boring Stream Discovery Contract

The desired operator experience is boring: OBS asks the Verse for Muninns, the
plugin shows live sources, and a selected source either activates or reports the
specific owner that refused it. No local plugin fallback may invent Raven
devices. A fake `display:0` or Realtek loopback row is worse than an empty list
because it makes stale state look selectable.

Live ownership:

- Owner: Muninn `serve` owns source inventory for the body where the devices
  exist. It publishes video and audio source ids, labels, command boundary,
  media profile, and current active sessions.
- Inputs: local display/audio enumeration, explicit configured source hints,
  daemon health, and accepted stream commands.
- Outputs: `muninn.telemetry_surface.v1`, `gamecult.eve.provider_advertisement.v1`,
  `muninn.command_boundary.v1`, `muninn.transport_profile.v1`, active
  `muninn.capture_stream.v1` receipts, and the temporary
  `muninn.obs_stream_catalog.v1` compatibility record.
- Derived state: the OBS dropdown is a lowering of discovered Muninn state. It
  is not an inventory owner, activation owner, or health owner.
- Forbidden writers: OBS defaults, local hard-coded device names, synced stale
  CultCache mirrors, scheduled-task launchers, and previous activation receipts
  must not create selectable devices or active streams.
- Shared paths: initial OBS load, refresh button, periodic refresh, source
  selection, reconnect, and OBS restart must all read the same live-discovered
  source list and issue the same typed capture command when activation is
  needed.
- Deletion line: the plugin may display `discovery missing`, but it must not
  synthesize `Display 1` or `Realtek loopback` as if Raven advertised them.
  Until Odin/Verse discovery is the direct plugin input, the OBS catalog remains
  a compatibility projection of Muninn-owned live state, not its own truth.
  Compatibility store fallback is allowed only as availability fallback: the
  plugin must use the first current store that yields Muninn telemetry and must
  not merge stale stores into a more attractive fake inventory.
  A store older than the Muninn health freshness budget is not a discovery
  source; it is an outage receipt.

This contract also splits the media session truth:

- Video session: selected video source id, encoder profile, bitrate, frame
  timestamps, access-unit ids, keyframe policy, feedback pressure, and video
  receiver queue.
- Audio session: selected audio source id, sample format, packet timestamps,
  reorder budget, continuity/drop policy, and OBS audio output cadence.
- Combined OBS source: a convenience lowering that may request one video session
  and one audio session together. It does not merge their clocks or transport
  state into one opaque "A/V target".

## Known-Good Checkpoint (2026-06-23)

The current system is in a usable compatibility state worth preserving before
the next transport experiment.

What is working:

- Raven Muninn runs continuously as the capture owner and accepts stream
  activation over CultNet RUDP.
- OBS on Starfire shows the live-discovered Muninn source inventory instead of
  synthesizing fake devices from stale local defaults.
- Video and audio sessions activate separately through the combined OBS source
  lowering and the stream comes up cleanly again after reconnect.
- The OBS plugin currently receives the temporary live
  `muninn.obs_stream_catalog.v1` projection over UDP `17874` and uses it as the
  live discovery source before any CultCache fallback.

Current compatibility details:

- Raven command ingress is currently bound to `0.0.0.0:17883`.
- Starfire OBS listens for the temporary live OBS catalog on UDP `17874`.
- The daemon `serve` loop must continue publishing the OBS catalog while idle;
  otherwise OBS falls back to stale store state and reports `discovery-missing`
  or shows dead inventory.
- The current OBS plugin is still a compatibility client. It does not yet host
  the Rust CultMesh runtime directly and it is not yet consuming Odin/Verse
  discovery as a first-class Verse client.

Known debt that remains intentionally visible:

- Idunn health publication can still time out independently of the media path.
  That is operational debt, not proof that Muninn discovery or streaming is
  down.
- The OBS catalog is still a temporary projection. The boring end state is live
  Verse discovery with no plugin-owned catalog receiver and no stale store
  merge behavior.

## Codec Direction

Use vendor hardware encode before inventing a codec. NVIDIA's practical answer
in this space is NVENC/NVDEC, low-latency encode presets, high LAN bitrate,
short GOP or intra-refresh, no B-frames, no lookahead, and application-level
transport that understands frame deadlines.

Shipping order:

1. H.264 NVENC low-latency as the compatibility baseline for broad decoder
   support, simple failure analysis, and OBS bridge stability.
2. HEVC NVENC low-latency when Raven encode and OBS decode both prove the
   target latency budget.
3. AV1 NVENC only after the exact Raven-to-OBS path proves encode, decode, and
   queue latency inside budget. AV1 can be the quality/bandwidth winner, but it
   must earn the low-latency lane on our hardware instead of getting promoted by
   codec fashion.

The codec should produce elementary video access units, not a muxed MPEG-TS
program. Audio should use Opus low-delay packets unless the OBS lowering needs
a temporary PCM/AAC compatibility bridge.

Recommended first LAN profile:

- Profile id: `muninn.rudp.low_latency_h264_lan.v1`.
- Video: H.264 NVENC, `p1`, ultra-low-latency tune, CBR high enough that LAN
  bandwidth is not the constraint, no B-frames, no lookahead, short GOP or
  intra-refresh, periodic IDR/keyframe on feedback pressure.
- VBV: derive `bufsize` from roughly one frame of bitrate
  (`bitrate_kbps / framerate`) instead of a large fixed encoder reservoir.
  The point is bounded latency, not hoarding compressed video in a nicer hat.
- Audio: Opus low-delay when OBS lowering can consume it directly; otherwise a
  temporary PCM/AAC bridge with separate packet identity and clock.
- Transport: payload elementary access units as typed media records, never a
  muxed container as the unit of truth.
- Packet size: default typed media chunks are `480` bytes. That leaves headroom
  for CultNet/MessagePack/RUDP headers inside a normal Ethernet MTU so the OBS
  bridge does not depend on IP fragmentation or private RUDP fragment
  reassembly. The sender also sets CultNet RUDP `max_fragment_bytes` to `1431`
  bytes (`1472` byte IPv4 UDP payload budget minus `36` RUDP fixed header bytes
  minus the five-byte `media` channel id), so oversized typed records fragment
  in CultNet deliberately instead of falling through to accidental IP
  fragmentation.
- Resend cadence: sender RUDP media resends are scheduled every `5` ms while
  the packet is still inside the `75` ms reliable-expiry budget.
- Control: sender may adapt bitrate, keyframe cadence, chunk size, and playout
  budget from receiver feedback; the receiver must not silently stretch latency
  to preserve visual perfection.
- Receiver URL policy: RUDP stream URLs carry `profile`,
  `sender_resend_delay_ms`, `reliable_expire_after_ms`,
  `assembly_deadline_ms`, and `gap_wait_ms` so the OBS bridge receives timing
  budget from Muninn's stream profile instead of hard-coding receiver policy
  privately.

### Motion Vectors And Reconstruction

Do not make CultMesh a bespoke game-streaming codec in the first shipping path.
The motion-vector/delta idea is real, but it belongs as an augmentation around
the hardware codec, not as the primary authority.

NVIDIA's relevant public pattern is:

- NVENC/NVDEC for hardware video encode and decode.
- Low-latency encode tuning, high refresh rate, adaptive bitrate, and explicit
  network/frame timing for cloud gaming.
- Foveated streaming for bandwidth reduction when a trusted focus region exists.
- Optical Flow / motion estimation hardware for interpolation, extrapolation,
  analysis, or concealment experiments.

For Muninn, that means:

- If we only have desktop capture pixels, trust NVENC for motion estimation and
  compression. Do not compute our own patch stream before the codec has failed a
  measured requirement.
- If a game or renderer we control can expose motion vectors, depth, regions of
  interest, or dirty rectangles, publish them as optional sideband media hints.
  They may guide encoder ROI, decoder concealment, frame extrapolation, or
  foveated quality allocation. They do not replace the encoded video access
  unit unless a separate research path proves that decoder under the same OBS
  latency budget.
- Stochastic reconstruction is allowed only inside a bounded playout budget.
  When the deadline expires, the receiver displays the best available frame,
  conceals damage, or requests a keyframe. It must not preserve dead packets by
  growing latency.

Useful references for this direction:

- NVIDIA Video Codec SDK presets describe low-latency and ultra-low-latency
  tuning plus preset tradeoffs for encoder tools such as GOP structure,
  B-frames, and lookahead:
  <https://developer.nvidia.com/blog/introducing-video-codec-sdk-10-presets/>
- NVIDIA CloudXR publicly frames low-latency spatial streaming around NVENC,
  adaptive bitrate, foveated streaming, diagnostics, and bidirectional data
  channels:
  <https://docs.nvidia.com/cloudxr-sdk/latest/overview/overview.html>
- NVIDIA's Optical Flow SDK exposes hardware motion vectors for real-time
  interpolation/extrapolation and related video processing:
  <https://developer.nvidia.com/optical-flow-sdk>

## CultNet RUDP Media Policy

The RUDP `media` channel should become a media-aware lane, not merely a reliable
unordered byte lane.

Required behavior:

- Frame-level fragmentation and reassembly keyed by `(stream_id, session_id,
  frame_id)`.
- Deadline-aware retransmission: resend while useful, drop when the playout
  budget is gone.
- Receiver feedback that can NACK missing chunks, request a keyframe, and
  report jitter/decode queue pressure.
- Small playout buffer chosen by policy, for example 8-20 ms on a wired LAN.
- Optional FEC/parity for video chunks where one repair packet beats a resend.
- Separate audio policy that favors continuity and bounded drift over perfect
  packet recovery.

## OBS Lowering

The OBS plugin should lower typed media frames into OBS-friendly video/audio.
The clean target is direct decode from elementary streams using libavcodec or
NVDEC where available, then hand decoded video frames and audio packets to OBS.

Temporary bridge allowed:

- Reconstruct an elementary stream from typed video access units.
- Feed the local FFmpeg source only as a compatibility decoder.
- Keep audio as a separate OBS audio path so audio sync is explicit instead of
  being hidden inside a damaged MPEG-TS stream.

The compatibility bridge must not own stream truth. It is a lowering target,
not the protocol.

## Cut Plan

1. Promote media-frame schemas in Odin/CultLib and add unit tests for
   encode/decode round trips.
2. Teach Muninn activation to emit video access units and audio packets instead
   of arbitrary MPEG-TS byte chunks.
3. Teach CultNet RUDP media receive to reassemble by frame and deadline, with
   explicit NACK/keyframe feedback.
4. Teach the OBS plugin to consume typed media frames and feed OBS through a
   direct decoder or a narrowly named compatibility elementary-stream bridge.
5. Remove the MPEG-TS byte-chunk hot path once the typed receiver is stable.

## Verification

The rebuild is not proven by "OBS shows video once." Minimum proof:

- Raven `serve` remains idle until an explicit capture command arrives.
- OBS request activates Raven without manual Raven terminals.
- Sender logs and receiver telemetry agree on stream id, session id, frame id,
  codec, PTS, deadlines, drops, NACKs, and keyframe requests.
- A forced packet drop causes bounded damage and recovery, not OBS freeze.
- Audio remains present as an OBS audio source and stays within the chosen
  sync budget.
- Restarting OBS does not strand Raven capture or poison the next request with
  stale command receipts.
- Restarting Raven Muninn restores catalog/health without starting capture
  until requested.