workspace: name: Mimir status: active focus: direct-driver-fensalir-stream-app implementation_name: Mimir public surface / native runtime contracts premise: > Mimir is the local realtime field machine: cameras, microphones, speakers, network feeds, and timing signals become one coherent OBS-facing volumetric program surface. The live path is a C# Fensalir-hosted app with in-memory rolling buffers and native capture/runtime boundaries. objective: primary: > Build one synchronized spatial stream machine from six cameras, six microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP, and OBS output. current: > Implement direct camera/audio capture into Mimir.Runtime buffers, starting with Leap stereo IR as the timing-camera candidate, then expose runtime health/settings/output controls through Fensalir UI. constraints: - Sender is Windows. - Receiver uses OBS. - Video encode must use NVENC when the bridge path is active. - Audio sources must remain separately controllable in OBS. - V1 bridge is LAN-only. - Persistent state must clarify rehydrate, not become transcript tax. - Live ingest belongs to Mimir.Runtime, native drivers, Fensalir, and Faust/native DSP. - Edge JSON is schema/diagnostic only; live process data is typed or in memory. - Networked feeds are producers into the live runtime, not independent clock authorities. - Mimir explicitly trades latency and memory for throughput and field quality: the default five-second buffer is compute budget for stream alignment, GPU/audio analysis, and volumetric sound-field extraction. invariants: - OBS composition authority stays in OBS. - FFmpeg owns capture, encoding, and transport only for bridge utilities. - One stable bridge port equals one OBS source endpoint. - One rolling live window owns the timing edge and expiry. - The rolling window is not accidental latency. It must preserve enough time to line up feeds and crunch data before OBS-facing output is emitted. - Typed views may index the rolling buffer, but no per-kind ring owns retention. - Fensalir owns production UI, GPU fusion, D3D12 interop, and Spout2 output. - Faust/native DSP owns hot audio alignment, separation, spatialization, and stems. - Local secrets remain untracked. - Mimir is a CultMesh app. Codebooks, schedules, calibration receipts, path response, and compute-offload work need typed mesh-syncable state surfaces so phones, microcontrollers, Raven, and Starfire can cooperate without becoming independent timing authorities. accepted_design: - `Mimir.slnx` owns the app surface. - `src/Mimir.App` launches Fensalir as host/window/D3D12 library. - `src/Mimir.Runtime` owns `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`, direct native ingest, process bridge ingest, and video capture driver seams. - `MimirVideoFrameDescriptor` carries width, height, pixel format, stride, device timestamp, and optional native/GPU handle. - `MimirAudioBlockDescriptor` carries sample rate, channels, sample format, frame count, timestamp, and optional native handle. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the final local-device hot path. - `MimirBioacousticTimeline` is the active runtime watermark. It renders a low-gain birdsong-like word language with 128 self-identifying word positions, left-speaker and right-speaker variants, four formant-rich syllables per word, rhythm variation, and direct word identity instead of a de Bruijn grammar. `MimirRuntime` emits separate panned vocabularies per speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as `evidence=bioacoustic`; the old chirp-bin path remains a calibration/reference artifact with ASIO tooling. - The diagnostic audio path can optionally carry base64 float sample payloads for sync smoke tests. `MimirChirpletTimeline` owns the older structured chirplet reference fingerprint: an order-3 de Bruijn sequence over 32 time/frequency constellation symbols, plus PCM segment rendering, matched timing atoms, symbol/event decoding, and per-band response kernels. Symbol identity is carried by start band, glide shape, duration, and following inter-chirp gap. Any three consecutive correctly detected symbols identify a timeline event within the current operating horizon. `MimirChirpletSymbolCodebook` owns the 32 symbol definitions so every symbol has a unique chirp shape, with rhythm as additional evidence. `MimirChirpletStreamDecoder` is the first constrained chirplet-transform receiver: it owns a bounded PCM window, emits transform frames with multiple phase-invariant symbol candidates and refined per-candidate sample offsets, decodes code-valid triplet anchors through gap/clock coherence, and fits a per-source sample clock from those anchors. Its synthetic smoke path is `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`. It currently decodes all 15 emitted chirps in a two-second window into 13 anchors for events 0-12 with a 47999.999990 Hz clock fit and 0.000014 sample MAE. `MimirAudioSynchronizationAnalyzer` now treats `loopback-scarlett-speakers` as timing authority and derives delay only from matched decoded timeline anchors. Positive delay means the candidate mic is late relative to loopback. Reports include rounded delay, fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub` caches the latest per-source reports and `MimirAudioSynchronizationStateTracker` owns the smoothed per-source belief: latest delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in ppm. `MimirRuntime` now runs this analysis online as a bounded rotating service on a fixed cadence, while UI and telemetry read cached reports/states instead of invoking analysis. `MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry. `MimirAudioSynchronizationSettings.Mode` now owns the active evidence policy: `chirp-only` emits the calibration witness, `passive` stays silent and uses program-audio phase correlation, and `hybrid` emits active pilot chunks only while passive confidence is weak. `MimirPassiveAudioSynchronizationEstimator` is the first passive path: it uses PHAT-weighted cross-spectrum correlation between loopback and mic windows before any watermark is needed. Negative passive lags are contradictory evidence and carry zero confidence; single-window passive confidence is capped below certainty until repeated state earns it. `MimirBioacousticTimeline` is now the active calibration witness for both chirp-only and hybrid: 128 self-identifying word positions, left/right speaker variants, four formant-rich syllables per word, rhythm variation, and direct word identity so any correctly decoded word identifies one canonical timeline location. The analyzer refines final active delay with constrained local waveform correlation around the decoded anchor delay. Each detected motif carries per-band response evidence, and the next calibration cut should use those profiles to tune contours, formants, usable bands, gain, and rhythm weighting. `Mimir.BufferSmoke --bioacoustic-cepstral-smoke` now proves the non-brute-force receiver direction: degraded mel-cepstral calls are decoded by an augmented MFCC/projection-hash word index before any expensive confirmation. `Mimir.BufferSmoke --bioacoustic-train` now runs a receipt-backed hypothesis panel over indexed cepstral decoder shapes and writes typed CultCache results plus pre-warp/post-warp/reconstructed WAV artifacts under `artifacts/bioacoustic-training/`. The first receipt `bioacoustic-20260524-103438` showed compact indexing fastest on clean audio and exposed timing error under domain warp. The corrected receipt `bioacoustic-20260524-150729` adds global clock hypothesis scoring over a longer window; compact indexing wins clean throughput, baseline indexing remains steadier under degradation, highband indexing has warped-path value, and anchor density needs its own metric. `MimirChirpBinTimeline` remains the controlled chirp-bin calibration/reference artifact with ASIO tooling and `MimirChirpBinCalibrationModel`; keep those captures as response evidence, not the active runtime sound. Reports and sync states expose delay in microseconds as well as fractional samples. Hybrid emits the bioacoustic witness as low-gain half-second bursts while passive confidence is weak, never as catch-up backlog. The analyzer no longer applies the passive two-second window floor to active fallback; `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 48000` recovers a 317.375-sample synthetic delay with printed 0.000 us error using `evidence=bioacoustic`, while `--hybrid-sync-self-test --sample-rate 48000` correctly accepts strong passive evidence at 2.772 us error. The synthetic standalone receiver proof `--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5` recovers delayed source time below printed microsecond precision using only the codebook/schedule. `--bioacoustic-actuator-self-test --sample-rate 48000 --delay-samples 317.375` now proves the first measurement-to-sample-movement loop: estimate bioacoustic delay, apply fractional correction, and remeasure residual below printed microsecond precision. `faust/mimir_alignment_actuator.dsp` is the first Faust-owned actuator surface for six channels of bounded fractional delay/gain. The synthetic codebook proof is `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`. `Mimir.BufferSmoke --render-chirp-bin-f32` plus `native/probes/asio_audio_cadence --play-f32-mono --record-f32-interleaved` now provides a real Scarlett ASIO artifact proof for active mode. At 192 kHz, Loopback 1 -> Loopback 2 decoded at 0.000 us with 12 anchors and 0.999 confidence. `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` now renders or ingests a calibration capture and persists the response/confusion/delay model under `calibration/chirp-bin/`; `--analyze-asio-f32 --calibration` loads it back into the chirp-bin reference decoder. Physical inputs still fail accepted pairwise timing against loopback in the stored artifact, but decoded mic channels now leave calibration models; physical input 1 produced two reliable symbols and a different strongest measured band set. `native/asio_capture` and `MimirAsioStreamSource` are now the runtime Scarlett path: Focusrite ASIO callbacks feed sample-bearing 192 kHz Float32 blocks into `Mimir.Runtime` in process. `config/mimir-runtime.asio.example.json` proves continuous ASIO runtime ingest: a two-second BufferSmoke run at 192 kHz ingested more than 12,000 sample-bearing blocks across `asio-ch0` through `asio-ch3` and retained 2,048 blocks per channel, all from the Focusrite ASIO callback stream. `Mimir.BufferSmoke --calibrate-contestant-asio-f32` now persists the active packet-song physical response model under `calibration/bioacoustic/`, including per-channel delay search, polarity, payload reliability, response-normalization bands, gain, confidence, and pairwise propagation delay. The latest canary-packet Scarlett receipt runs at 10.7x realtime across four 192 kHz channels: loopback is 37/37 with 2.524 us MAE, co-streamer shotgun is 37/37 payload with 58.785 us MAE, and the cardioid is 26/37 with 90.558 us MAE. This is a serious calibration surface, not yet true physical microsecond sync. Acoustic robustness remains open. The old arbitrary-chirplet matcher is a diagnostic/reference artifact only, not the runtime hot path. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared texture capture. - `native/reservoir` owns the lower shared-edge rolling-buffer invariant for Fensalir/Faust integration. - `config/perfect-machine.example.json` declares live owners, sensors, reservoir views, and runtime policy. - `src/Mimir.Runtime/Synchronization` now includes the Perfect Machine module library distilled from the research notes: node profiles, decoder/language profiles, path learning, benchmark panels, actuator strategies, capture and ingest strategies, reservoir strategies, field profiles, compute offload, OBS publication, CultMesh contracts, Fensalir lowering, and exportable manifest generation. - `docs/perfect-machine-module-library.md` indexes those reusable runtime modules and their smoke commands. - `docs/code-algorithm-map.md` is the source-level ownership and algorithm map for the current repo code surface. - `docs/perfect-machine-domain-index.md` indexes Mimir's full problem domain, current evidence taxonomy, hot paths, and implementation cuts. - `research/perfect-machine-study-2026-05-23/` preserves the architecture rumination, optimization ledger, research references, source/hotspot audits, Fensalir/native boundary maps, benchmark plan, calibration-session spec, distributed receiver spec, failure-mode ledger, data dictionary, low-level implementation notes, and sample code sketches for streaming chirp-bin decode, AVX2/GPU scoring, batched native ABI, fractional-delay/SRO actuation, acoustic source localization, Mimir-to-Fensalir lowering, and native capture rings. The corrected deep-study pass ran from `2026-05-23T23:54:42.9041588+01:00` through at least `2026-05-24T01:54:56.9747630+01:00`; use `research/perfect-machine-study-2026-05-23/reading-guide.md` as the re-entry index. - `.voidbot` state is part of repo memory and must be committed separately whenever dirty. active_subgoals: - Implement the first concrete Leap direct capture driver and measure cadence. - Add remaining local camera drivers through `IMimirVideoCaptureDriver`. - Use the verified in-process Focusrite ASIO source for local Scarlett mic and loopback ingest; WASAPI remains a diagnostic witness. - Add network audio feed producers that enter the same runtime clock/rebuffer machinery instead of becoming independent authorities. - Bind Fensalir UI to stream health, buffer depth, source timestamps, settings, and output management. - Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output replaces them. current_status: summary: > The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS bridge scripts, and persistence surfaces. Script-era packages, tests, helper tools, dependency manifests, and stale docs have been removed from the working tree. Research archives remain as reference-only memory for raw native/GPU implementation shape. A raw PS3 Eye libusb/WinUSB cadence probe measured both PS3 Eyes simultaneously at about 183 fps each at 320x240 and 57-59 fps each at 640x480. A native Kernel Streaming LeapUVC probe can pull frames through usbvideo.sys without Media Foundation, DirectShow, OpenCV, or frame processing; after discarding a two-second warm-up, the useful 640x240 stereo IR mode advertises 115 fps and currently measures about 110.8 fps with eight queued asynchronous KS reads. The half-height 640x120 mode advertises 214 fps and currently measures about 205.9 fps. Plain Kiyo is usable through KS when manual exposure is forced shorter: 1080p MJPG reaches about 30 fps and 720p MJPG reaches about 51.5 fps. Kiyo Pro advertises 60 fps MJPG at 720p/1080p but currently delivers only about 25 fps through the same KS path even with manual exposure tuning. Standard KS low-light compensation off does not restore 60 fps, powerline 60Hz only raises Pro cadence to about 29 fps, and standard extended Video HDR is not exposed on this KS path. The controls-only probe now parses USB VideoControl descriptors and found the Kiyo Pro's two vendor extension units. One extension unit answers writable KS selectors. Moving the Pro directly to a motherboard USB3 port changed the path to `root_hub30`, `bcdUSB=0x0320`, and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports `UsbHighSpeed` rather than `UsbSuperSpeed`, and baseline cadence is still about 25 fps across every tested format. The first simultaneous local camera pull only had three visible devices: LeapUVC, Kiyo Pro, and one PS3 Eye. The KS multi-stream harness plus raw PS3 Eye probe pulled all three at once, but Leap cadence collapsed under shared load: with PS3 Eye at 640x480@60 the set delivered Leap 40.12 fps, Kiyo Pro 24.99 fps, and PS3 Eye 56.06 fps; with PS3 Eye at 320x240@187 it delivered Leap 81.10 fps, Kiyo Pro 25.33 fps, and PS3 Eye 185.74 fps. After port swapping, five camera devices / six optical sensors were visible and pulled at once: LeapUVC's packed stereo IR pair, Kiyo Pro, regular Kiyo, and two PS3 Eyes. With both Eyes at 640x480@60, the set delivered Leap stereo 57.33 fps, Kiyo Pro 24.97 fps, Kiyo 30.30 fps, PS3 Eye 0 59.59 fps, and PS3 Eye 1 57.60 fps. With both Eyes at 320x240@187, it delivered Leap stereo 81.05 fps, Kiyo Pro 24.97 fps, Kiyo 30.28 fps, PS3 Eye 0 187.60 fps, and PS3 Eye 1 187.03 fps. Kiyo Pro still negotiates high-speed USB, and Leap remains topology-sensitive under shared load. A follow-up moved Leap alone onto the blue rear USB3 group/root_hub30#5 while both Kiyos shared root_hub30#7&18eea21b and one visible PS3 Eye used root_hub30#7&124f193f; Leap still delivered only 80.99 fps with the other visible cameras active, so the remaining loss is not explained by same-root-hub contention alone. Splitting Kiyo Pro and one PS3 Eye to Raven left the Mimir host with LeapUVC, regular Kiyo, and one PS3 Eye, but local cadence still stayed low: Leap delivered 81.34 fps with Kiyo+Eye, 82.77 fps with Kiyo only, and only 84.18 fps in a same-session Leap-only baseline. The previous ~110.8 fps Leap result was not reproduced until the Focusrite Scarlett Solo was moved away from the Leap USB neighborhood. After that, Leap-only baseline rose to 95.48 fps, best full-height scenario to 101.67 fps, and the local split set delivered Leap 105.10 fps, regular Kiyo 30.17 fps, and one PS3 Eye 185.81 fps. The split-host layout is viable if the Leap's USB neighborhood stays free of the Focusrite and other heavy devices. Raven's PS3 Eye raw interface now has the same libwdi WinUSB binding as the local Eye and can be opened by the copied cadence probe, but Raven Eye cadence is currently unusable: 320x240@187 delivered 132.24 fps then 78.93 fps, and 640x480@60 delivered 6.32 fps with multi-second gaps. Moving that Eye back to the Mimir host produced the current clean local topology: LeapUVC, regular Kiyo, and two PS3 Eyes run in parallel while Kiyo Pro stays off-host. With both Eyes at 320x240@187, Leap delivered 106.07 fps, Kiyo 30.33 fps, and the Eyes 186.57 / 186.14 fps. With both Eyes at 640x480@60, Leap delivered 108.07 fps, Kiyo 30.34 fps, and the Eyes 57.79 / 58.08 fps. Moving Kiyo Pro back to the Mimir host while regular Kiyo was absent kept Leap healthy: with both Eyes at 320x240@187, Leap delivered 110.80 fps, Kiyo Pro 24.97 fps, and the Eyes 187.46 / 186.89 fps; with both Eyes at 640x480@60, Leap delivered 110.72 fps, Kiyo Pro 24.87 fps, and the Eyes 59.91 / 59.19 fps. Kiyo Pro still negotiates UsbHighSpeed(2), but it does not poison Leap cadence in the current layout. Moving regular Kiyo from front panel to a rear port made all five camera devices / six optical sensors visible locally. With both Eyes at 320x240@187, the full set delivered Leap 110.84 fps, Kiyo Pro 25.08 fps, Kiyo 30.18 fps, PS3 Eye 0 187.59 fps, and PS3 Eye 1 187.02 fps. With both Eyes at 640x480@60, it delivered Leap 110.76 fps, Kiyo Pro 25.14 fps, Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > Use the new bioacoustic timeline watermark as the active acoustic sync direction. Cut the next implementation in this order: streaming indexed log-mel/MFCC motif decoder state with a global clock/path hypothesis fed by `--bioacoustic-train` receipts, calibration-session command/replay proof that learns per-output/mic motif/formant response, fractional-delay/SRO actuator, then native camera payload handles and Fensalir lowering. Keep chirp-bin calibration captures as reference evidence, not the runtime target. Current app testing proves Fensalir WASAPI output wakes Scarlett speaker loopback, live mic buffers stay populated, runtime sync analysis now runs as a bounded rotating service without UI/telemetry re-entry, and the bioacoustic motif timeline produces deterministic synthetic active sync. The next invariant is self-identifying motif words producing stable canonical anchors and clock fits through real loopback and microphones, so the actuator can use decoded time instead of filtered lag guesses. Replace the remaining diagnostic JSON-line process bridges with direct native capture drivers that push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when calibration witnesses are playing. Treat PS3 Eye audio as enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from the Leap USB neighborhood. The Scarlett Solo 4th Gen is now attached to Starfire, and `native/probes/asio_audio_cadence` proves local Focusrite USB ASIO access with 4 inputs / 2 outputs including Loopback 1/2, 192-frame preferred buffers, 44.1-192 kHz support, and nonzero 4-channel Int32LSB input callbacks at 192 kHz. Raven also has a 192 kHz loopback-capable Scarlett for co-streamer/game timing evidence, but Starfire owns the heavy soundfield and sensor-fusion work. Treat Kiyo Pro as a ~25 fps source until its SuperSpeed negotiation is fixed.