Mimir/state/map.yaml

workspace:
  name: Mimir
  status: active
  focus: direct-driver-aquarium-stream-app
  implementation_name: Mimir public surface / native runtime contracts
  premise: >
    Mimir is the local realtime field machine: cameras, microphones, speakers,
    network feeds, and timing signals become one coherent OBS-facing volumetric
    program surface. The live path is a C# Aquarium-hosted app with in-memory
    rolling buffers and native capture/runtime boundaries.

objective:
  primary: >
    Build one synchronized spatial stream machine from six cameras, six
    microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP,
    and OBS output.
  current: >
    Implement direct camera/audio capture into Mimir.Runtime buffers, starting
    with Leap stereo IR as the timing-camera candidate, then expose runtime
    health/settings/output controls through Aquarium UI.

constraints:
  - Sender is Windows.
  - Receiver uses OBS.
  - Video encode must use NVENC when the bridge path is active.
  - Audio sources must remain separately controllable in OBS.
  - V1 bridge is LAN-only.
  - Persistent state must clarify rehydrate, not become transcript tax.
  - Live ingest belongs to Mimir.Runtime, native drivers, Aquarium, and
    Faust/native DSP.
  - Edge JSON is schema/diagnostic only; live process data is typed or in
    memory.
  - Networked feeds are producers into the live runtime, not independent clock
    authorities.
  - Mimir explicitly trades latency and memory for throughput and field quality:
    the default five-second buffer is compute budget for stream alignment,
    GPU/audio analysis, and volumetric sound-field extraction.

invariants:
  - OBS composition authority stays in OBS.
  - FFmpeg owns capture, encoding, and transport only for bridge utilities.
  - One stable bridge port equals one OBS source endpoint.
  - One rolling live window owns the timing edge and expiry.
  - The rolling window is not accidental latency. It must preserve enough time
    to line up feeds and crunch data before OBS-facing output is emitted.
  - Typed views may index the rolling buffer, but no per-kind ring owns
    retention.
  - Aquarium owns production UI, GPU fusion, D3D12 interop, and Spout2 output.
  - Faust/native DSP owns hot audio alignment, separation, spatialization, and
    stems.
  - Local secrets remain untracked.

accepted_design:
  - `Mimir.slnx` owns the app surface.
  - `src/Mimir.App` launches Aquarium Engine as host/window/D3D12 library.
  - `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
    `MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`,
    direct native ingest, process bridge ingest, and video capture driver seams.
  - `MimirVideoFrameDescriptor` carries width, height, pixel format, stride,
    device timestamp, and optional native/GPU handle.
  - `MimirAudioBlockDescriptor` carries sample rate, channels, sample format,
    frame count, timestamp, and optional native handle.
  - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for
    JSON-line video frame and audio block metadata from native probes into
    runtime rolling buffers; it does not carry pixels/audio payloads or own the
    final local-device hot path.
  - The diagnostic audio path can optionally carry base64 float sample payloads
    for sync smoke tests. `MimirChirpletTimeline` owns the structured
    birdsong-like timeline fingerprint: an order-3 de Bruijn sequence over 32
    separated chirp symbols, plus PCM segment rendering, matched timing atoms,
    symbol/event decoding, and per-band response kernels. Any three consecutive
    correctly detected symbols identify a timeline event within the current
    operating horizon. `MimirAudioSynchronizationAnalyzer` now treats
    `loopback-scarlett-speakers` as timing authority, uses coarse chirplet
    energy for acquisition, and refines delay from matched timeline event
    indices when symbol decoding succeeds. Positive delay means the
    candidate mic is late relative to loopback. Reports include rounded delay,
    fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
    owns the smoothed per-source belief: latest delay, smoothed delay,
    confidence, per-band response evidence, and delay-slope/SRO estimate in
    ppm. `MimirRuntime` now runs this analysis online on a fixed cadence and
    `MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry.
    `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
    provisional loopback-referenced aligned mono frame for channels that clear
    the confidence gate.
  - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
    seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
    texture capture.
  - `native/reservoir` owns the lower shared-edge rolling-buffer invariant for
    Aquarium/Faust integration.
  - `config/perfect-machine.example.json` declares live owners, sensors,
    reservoir views, and runtime policy.
  - `.voidbot` state is part of repo memory and must be committed separately
    whenever dirty.

active_subgoals:
  - Implement the first concrete Leap direct capture driver and measure cadence.
  - Add remaining local camera drivers through `IMimirVideoCaptureDriver`.
  - Add native audio capture workers for mic, loopback, and network audio feeds.
  - Bind Aquarium UI to stream health, buffer depth, source timestamps,
    settings, and output management.
  - Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
    replaces them.

current_status:
  summary: >
    The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS
    bridge scripts, and persistence surfaces. Script-era packages, tests,
    helper tools, dependency manifests, and stale docs have been removed from
    the working tree. Research archives remain as reference-only memory for raw
    native/GPU implementation shape. A raw PS3 Eye libusb/WinUSB cadence probe
    measured both PS3 Eyes simultaneously at about 183 fps each at 320x240 and
    57-59 fps each at 640x480. A native Kernel Streaming LeapUVC probe can pull
    frames through usbvideo.sys without Media Foundation, DirectShow, OpenCV, or
    frame processing; after discarding a two-second warm-up, the useful 640x240
    stereo IR mode advertises 115 fps and currently measures about 110.8 fps
    with eight queued asynchronous KS reads. The half-height 640x120 mode
    advertises 214 fps and currently measures about 205.9 fps. Plain Kiyo is
    usable through KS when manual exposure is forced shorter: 1080p MJPG reaches
    about 30 fps and 720p MJPG reaches about 51.5 fps. Kiyo Pro advertises 60
    fps MJPG at 720p/1080p but currently delivers only about 25 fps through the
    same KS path even with manual exposure tuning. Standard KS low-light
    compensation off does not restore 60 fps, powerline 60Hz only raises Pro
    cadence to about 29 fps, and standard extended Video HDR is not exposed on
    this KS path. The controls-only probe now parses USB VideoControl
    descriptors and found the Kiyo Pro's two vendor extension units. One
    extension unit answers writable KS selectors. Moving the Pro directly to a
    motherboard USB3 port changed the path to `root_hub30`, `bcdUSB=0x0320`,
    and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still
    reports `UsbHighSpeed` rather than `UsbSuperSpeed`, and baseline cadence is
    still about 25 fps across every tested format. The first simultaneous local
    camera pull only had three visible devices: LeapUVC, Kiyo Pro, and one PS3
    Eye. The KS multi-stream harness plus raw PS3 Eye probe pulled all three at
    once, but Leap cadence collapsed under shared load: with PS3 Eye at
    640x480@60 the set delivered Leap 40.12 fps, Kiyo Pro 24.99 fps, and PS3
    Eye 56.06 fps; with PS3 Eye at 320x240@187 it delivered Leap 81.10 fps,
    Kiyo Pro 25.33 fps, and PS3 Eye 185.74 fps. After port swapping, five
    camera devices / six optical sensors were visible and pulled at once:
    LeapUVC's packed stereo IR pair, Kiyo Pro, regular Kiyo, and two PS3 Eyes.
    With both Eyes at 640x480@60, the set delivered Leap stereo 57.33 fps,
    Kiyo Pro 24.97 fps, Kiyo 30.30 fps, PS3 Eye 0 59.59 fps, and PS3 Eye 1
    57.60 fps. With both Eyes at 320x240@187, it delivered Leap stereo 81.05
    fps, Kiyo Pro 24.97 fps, Kiyo 30.28 fps, PS3 Eye 0 187.60 fps, and PS3
    Eye 1 187.03 fps. Kiyo Pro still negotiates high-speed USB, and Leap
    remains topology-sensitive under shared load. A follow-up moved Leap alone
    onto the blue rear USB3 group/root_hub30#5 while both Kiyos shared
    root_hub30#7&18eea21b and one visible PS3 Eye used root_hub30#7&124f193f;
    Leap still delivered only 80.99 fps with the other visible cameras active,
    so the remaining loss is not explained by same-root-hub contention alone.
    Splitting Kiyo Pro and one PS3 Eye to Raven left the Mimir host with
    LeapUVC, regular Kiyo, and one PS3 Eye, but local cadence still stayed low:
    Leap delivered 81.34 fps with Kiyo+Eye, 82.77 fps with Kiyo only, and only
    84.18 fps in a same-session Leap-only baseline. The previous ~110.8 fps
    Leap result was not reproduced until the Focusrite Scarlett Solo was moved
    away from the Leap USB neighborhood. After that, Leap-only baseline rose to
    95.48 fps, best full-height scenario to 101.67 fps, and the local split set
    delivered Leap 105.10 fps, regular Kiyo 30.17 fps, and one PS3 Eye 185.81
    fps. The split-host layout is viable if the Leap's USB neighborhood stays
    free of the Focusrite and other heavy devices. Raven's PS3 Eye raw interface
    now has the same libwdi WinUSB binding as the local Eye and can be opened by
    the copied cadence probe, but Raven Eye cadence is currently unusable:
    320x240@187 delivered 132.24 fps then 78.93 fps, and 640x480@60 delivered
    6.32 fps with multi-second gaps. Moving that Eye back to the Mimir host
    produced the current clean local topology: LeapUVC, regular Kiyo, and two
    PS3 Eyes run in parallel while Kiyo Pro stays off-host. With both Eyes at
    320x240@187, Leap delivered 106.07 fps, Kiyo 30.33 fps, and the Eyes 186.57
    / 186.14 fps. With both Eyes at 640x480@60, Leap delivered 108.07 fps, Kiyo
    30.34 fps, and the Eyes 57.79 / 58.08 fps. Moving Kiyo Pro back to the
    Mimir host while regular Kiyo was absent kept Leap healthy: with both Eyes
    at 320x240@187, Leap delivered 110.80 fps, Kiyo Pro 24.97 fps, and the Eyes
    187.46 / 186.89 fps; with both Eyes at 640x480@60, Leap delivered 110.72
    fps, Kiyo Pro 24.87 fps, and the Eyes 59.91 / 59.19 fps. Kiyo Pro still
    negotiates UsbHighSpeed(2), but it does not poison Leap cadence in the
    current layout. Moving regular Kiyo from front panel to a rear port made
    all five camera devices / six optical sensors visible locally. With both
    Eyes at 320x240@187, the full set delivered Leap 110.84 fps, Kiyo Pro 25.08
    fps, Kiyo 30.18 fps, PS3 Eye 0 187.59 fps, and PS3 Eye 1 187.02 fps. With
    both Eyes at 640x480@60, it delivered Leap 110.76 fps, Kiyo Pro 25.14 fps,
    Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
    the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
  next_action: >
    Harden the online state filter before adding the synchronization actuator.
    Current app testing proves Aquarium WASAPI output wakes Scarlett speaker
    loopback, live mic buffers stay populated, runtime sync analysis runs on
    cadence, and the continuous chirplet timeline can produce confident sync
    states in the app. Delay estimates are still jumpy, so the actuator needs
    outlier rejection and better SRO smoothing before it drives variable-rate
    resampling and fractional-delay correction per non-reference stream. Replace
    the diagnostic JSON-line process bridge with direct native capture drivers that push buffer handles into
    Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned
    audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when
    9-16 kHz calibration chirplets are playing. Treat PS3 Eye audio as
    enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from
    the Leap USB neighborhood and treat Kiyo Pro as a ~25 fps source until its
    SuperSpeed negotiation is fixed.