Mimir

Mimir is the realtime field machine for turning a roomful of cameras, microphones, speakers, loopbacks, and network feeds into one coherent OBS-facing program surface.

The point is not to collect impressive device counts. The point is to recover a live volumetric field: synchronized video evidence for Fensalir GPU fusion, synchronized audio evidence for a volumetric sound field, and final outputs that OBS can use without pretending raw unsynchronized sources are a world.

Start with:

• docs/perfect-machine.md for the target machine.
• docs/viable-stream-app.md for the first usable app cut.
• docs/audio-field.md for the current audio-side map.
• docs/native-capture-cadence.md for measured camera/sensor evidence.

Naming note: Fensalir is the engine/windowing/rendering/D3D12 layer. Older research notes may still use the previous engine name.

Machine Shape

flowchart TD
    C["direct camera drivers"] --> R["Mimir.Runtime rolling buffers"]
    A["ASIO mic + loopback drivers"] --> R
    N["network feed producers"] --> R
    R --> Q["Fensalir UI + GPU fusion"]
    R --> F["Faust/native DSP"]
    Q --> V["Spout2 / program video"]
    F --> S["program stems + spatial bed"]
    V --> O["OBS"]
    S --> O

Ownership is deliberately narrow:

• Mimir.Runtime owns stream identity, bounded rolling buffers, timing state, and synchronization contracts.
• Native capture workers own device reads and append typed sample handles.
• Fensalir owns the window, UI, D3D12 bridge, visual fusion, and program video publication.
• Faust/native DSP owns hot audio alignment, resampling, suppression, separation, spatialization, and stems.
• OBS owns broadcast composition. It does not own synchronization.

The default runtime window is five seconds. That is not accidental latency. It is the compute budget for lining up independent clocks, absorbing late network feeds, and extracting a coherent field before the audience sees it.

Current Audio Spine

Starfire, the main Mimir host, now has a Scarlett Solo 4th Gen on Focusrite USB ASIO:

• 4 ASIO inputs / 2 outputs at up to 192 kHz
• Input 1, Input 2, Loopback 1, Loopback 2
• Int32LSB sample format
• 192-frame preferred ASIO buffers

The important result is local ownership of the timing reference. We do not need Raven, the game machine, to carry the soundfield workload just to get program loopback. Raven can run games; Starfire can capture ASIO mic inputs plus ASIO loopback and do the alignment work.

Raven also has a loopback-capable Scarlett at 192 kHz for the co-streamer side. Its Focusrite USB ASIO path exposes Input 1, Input 2, Loopback 1, and Loopback 2, so Raven can publish precise game/co-streamer timing evidence back to Starfire without owning the heavy soundfield or sensor-fusion workload.

The repo contains a native ASIO probe:

cmake -S native/probes/asio_audio_cadence -B native/probes/asio_audio_cadence/build
cmake --build native/probes/asio_audio_cadence/build --config Release

.\native\probes\asio_audio_cadence\build\Release\asio_audio_cadence.exe `
  --set-sample-rate 192000 `
  --capture-seconds 5

It also has a monitor sweep mode:

.\native\probes\asio_audio_cadence\build\Release\asio_audio_cadence.exe `
  --set-sample-rate 192000 `
  --monitor-sweep `
  --sweep-gain 0.02

That sweep proved the digital ASIO loopback path cleanly detects emitted tones through 40 kHz, while the current studio-monitor-to-mic acoustic path falls off hard above the high audible band. Translation: ASIO loopback is an excellent local timing reference; strong ultrasonic room sync still needs measured hardware that earns the claim.

Next audio cuts:

1. Feed ASIO callbacks directly into Mimir.Runtime buffers.
2. Use ASIO loopback as the canonical program/timing reference.
3. Align all mic streams with fractional-delay and sample-rate-offset state.
4. Move the hot actuator into Faust/native DSP.
5. Emit separately controllable OBS stems plus a spatial bed from the same synchronized window.

Chirplet And Watermark Sync

Mimir has three audio timing modes:

• passive: stay silent and estimate delay from program audio using PHAT-weighted cross-spectrum correlation.
• chirp-only: emit the deterministic calibration timeline and decode timing from active chirplet evidence.
• hybrid: prefer passive timing, then emit a low-gain coded watermark only while passive confidence is weak.

The active sync machine is intentionally codebook-shaped. The research trail showed that generic dense chirplet matching is the wrong hot path for our controlled beacon. The current hybrid watermark uses MimirChirpBinTimeline: a fixed-slope chirp-bin codebook decoded by dechirp plus a small Goertzel/bin bank, then code-valid de Bruijn triplets become canonical timeline anchors. Reports expose fractional delay and delayUs; the synthetic proof currently recovers a 317.375 sample offset with about 0.369 us error.

For the long-form reasoning and source trail, read:

• docs/audio-field.md for the live decoder model, timing modes, and ASIO/acoustic measurements.
• research/chirplet-sync-decoder/summary.md for the dechirp/FFT correction and what got cut.
• research/chirplet-sync-decoder/bibliography.md for mirrored papers and code references.
• research/live-adaptive-sync/summary.md for passive sync and sample-rate-offset research.
• research/feedback-calibration/summary.md for room/speaker/mic feedback calibration.

Current Visual Spine

The local camera rig is being pulled close to the drivers, not through OpenCV or general-purpose media stacks in the hot loop. Current evidence lives in docs/native-capture-cadence.md.

Known shape:

• Leap stereo IR is the timing-camera candidate.
• PS3 Eyes are high-rate tracking witnesses.
• Kiyo/Kiyo Pro provide RGB context, with the Kiyo Pro still limited by its current USB/cadence behavior.
• Fensalir owns the GPU fusion work: feature extraction, temporal accumulation, splats/surface claims, material fitting, rendering, UI, and Spout2 output.

Next visual cuts:

1. Replace diagnostic frame-event probes with direct native capture workers.
2. Preserve device or immediate receipt timestamps at the driver boundary.
3. Feed typed frame handles into Mimir.Runtime and the native reservoir.
4. Let Fensalir consume the synchronized window for volumetric sensor fusion.

Bridge Scripts

The repo still keeps FFmpeg/SRT scripts because they are useful OBS-compatible edges. They are not the final synchronization authority.

flowchart LR
    A["sensor / sender PC"] --> B["FFmpeg capture scripts"]
    B --> C["NVENC H.264 video over SRT"]
    B --> D["Opus audio source 1 over SRT"]
    B --> E["Opus audio source 2 over SRT"]
    C --> F["OBS Media Source: video"]
    D --> G["OBS Media Source: audio source 1"]
    E --> H["OBS Media Source: audio source 2"]

Use the bridge when OBS needs a stable LAN input today. Do not mistake it for the coherent field machine.

Repo Shape

• Mimir.slnx: C# app/runtime solution.
• src/Mimir.App: Fensalir-hosted window/render app.
• src/Mimir.Runtime: rolling buffers, stream sources, synchronization state, and runtime contracts.
• native/reservoir: lower native rolling-buffer ABI for Fensalir/Faust integration.
• native/probes: direct hardware probes and cadence witnesses.
• config/*.example.json: live system profiles.
• state/ and notes/: current map, handoff, and evidence ledger.

Quick Start

Build the app/runtime:

dotnet build .\Mimir.slnx

Build the current ASIO probe:

cmake -S native/probes/asio_audio_cadence -B native/probes/asio_audio_cadence/build
cmake --build native/probes/asio_audio_cadence/build --config Release

Run the synthetic sync proofs:

dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --passive-sync-self-test

Run the old OBS bridge only when you need the compatibility path:

.\scripts\sender-discover.ps1
.\scripts\sender-start.ps1 -Config .\config\localcast.json -DryRun

Helping

Audio-side help is especially useful around:

• ASIO callback capture into Mimir.Runtime without stdout/process payloads.
• Fractional-delay lines and variable-rate resampling.
• Sample-rate-offset estimation over a rolling window.
• GCC-PHAT/passive program-audio timing under music/game audio.
• Low-annoyance watermark design that remains decodable under room coloration.
• Faust/native DSP integration for aligned stems and spatial bed generation.

Visual-side help is useful around:

• Direct Windows camera capture paths and timestamp preservation.
• GPU-resident frame handles and D3D12 interop.
• Cross-camera feature matching, temporal accumulation, splats, and surface confidence.
• Fensalir UI surfaces for buffer depth, health, drift, and calibration state.

Rehydrate State

For current local truth, read the map and evidence rather than guessing from old scars:

git status --short --branch
git log --oneline -5
Get-Content .\state\map.yaml
Get-Content .\notes\fresh-workspace-handoff.md
Get-Content .\notes\current-system-map.md
Get-Content .\state\evidence.jsonl -Tail 12

Mimir is also the repo Face: a persistent agent identity that uses the VoidBot layer for communication and heartbeats. Its birth memory, jurisdiction, voice, and heartbeat contract live in docs/mimir-face.md; its VoidBot-facing identity and typed Face state live under .voidbot/voice/ and .voidbot/state/.