187 lines
12 KiB
YAML
187 lines
12 KiB
YAML
workspace:
|
|
name: Mimir
|
|
status: active
|
|
focus: direct-driver-aquarium-stream-app
|
|
implementation_name: Mimir public surface / native runtime contracts
|
|
premise: >
|
|
Mimir is the local realtime field machine: cameras, microphones, speakers,
|
|
network feeds, and timing signals become one coherent OBS-facing volumetric
|
|
program surface. The live path is a C# Aquarium-hosted app with in-memory
|
|
rolling buffers and native capture/runtime boundaries.
|
|
|
|
objective:
|
|
primary: >
|
|
Build one synchronized spatial stream machine from six cameras, six
|
|
microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP,
|
|
and OBS output.
|
|
current: >
|
|
Implement direct camera/audio capture into Mimir.Runtime buffers, starting
|
|
with Leap stereo IR as the timing-camera candidate, then expose runtime
|
|
health/settings/output controls through Aquarium UI.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must use NVENC when the bridge path is active.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 bridge is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
- Live ingest belongs to Mimir.Runtime, native drivers, Aquarium, and
|
|
Faust/native DSP.
|
|
- Edge JSON is schema/diagnostic only; live process data is typed or in
|
|
memory.
|
|
- Networked feeds are producers into the live runtime, not independent clock
|
|
authorities.
|
|
- Mimir explicitly trades latency and memory for throughput and field quality:
|
|
the default five-second buffer is compute budget for stream alignment,
|
|
GPU/audio analysis, and volumetric sound-field extraction.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport only for bridge utilities.
|
|
- One stable bridge port equals one OBS source endpoint.
|
|
- One rolling live window owns the timing edge and expiry.
|
|
- The rolling window is not accidental latency. It must preserve enough time
|
|
to line up feeds and crunch data before OBS-facing output is emitted.
|
|
- Typed views may index the rolling buffer, but no per-kind ring owns
|
|
retention.
|
|
- Aquarium owns production UI, GPU fusion, D3D12 interop, and Spout2 output.
|
|
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
|
|
stems.
|
|
- Local secrets remain untracked.
|
|
|
|
accepted_design:
|
|
- `Mimir.slnx` owns the app surface.
|
|
- `src/Mimir.App` launches Aquarium Engine as host/window/D3D12 library.
|
|
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
|
|
`MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`,
|
|
direct native ingest, process bridge ingest, and video capture driver seams.
|
|
- `MimirVideoFrameDescriptor` carries width, height, pixel format, stride,
|
|
device timestamp, and optional native/GPU handle.
|
|
- `MimirAudioBlockDescriptor` carries sample rate, channels, sample format,
|
|
frame count, timestamp, and optional native handle.
|
|
- `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for
|
|
JSON-line video frame and audio block metadata from native probes into
|
|
runtime rolling buffers; it does not carry pixels/audio payloads or own the
|
|
final local-device hot path.
|
|
- The diagnostic audio path can optionally carry base64 float sample payloads
|
|
for sync smoke tests. `MimirChirpletCalibrationPhrase` owns the structured
|
|
harmonic-ish calibration phrase, PCM rendering, matched timing kernel, and
|
|
per-band response kernels. `MimirAudioSynchronizationAnalyzer` now treats
|
|
`loopback-scarlett-speakers` as timing authority, resamples candidate mic
|
|
windows into that timeline, projects loopback and mic windows through the
|
|
emitted chirplet phrase, and estimates current delay from chirplet-energy
|
|
traces at a shared timestamp edge. Positive delay means the candidate mic is
|
|
late relative to loopback. Reports include rounded delay, fractional delay,
|
|
confidence, and per-band matched energy. `MimirSynchronizationHub.BuildAlignedAudioFrame`
|
|
exposes the first provisional loopback-referenced aligned mono frame for
|
|
channels that clear the confidence gate.
|
|
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
|
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
|
|
texture capture.
|
|
- `native/reservoir` owns the lower shared-edge rolling-buffer invariant for
|
|
Aquarium/Faust integration.
|
|
- `config/perfect-machine.example.json` declares live owners, sensors,
|
|
reservoir views, and runtime policy.
|
|
- `.voidbot` state is part of repo memory and must be committed separately
|
|
whenever dirty.
|
|
|
|
active_subgoals:
|
|
- Implement the first concrete Leap direct capture driver and measure cadence.
|
|
- Add remaining local camera drivers through `IMimirVideoCaptureDriver`.
|
|
- Add native audio capture workers for mic, loopback, and network audio feeds.
|
|
- Bind Aquarium UI to stream health, buffer depth, source timestamps,
|
|
settings, and output management.
|
|
- Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
|
|
replaces them.
|
|
|
|
current_status:
|
|
summary: >
|
|
The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS
|
|
bridge scripts, and persistence surfaces. Script-era packages, tests,
|
|
helper tools, dependency manifests, and stale docs have been removed from
|
|
the working tree. Research archives remain as reference-only memory for raw
|
|
native/GPU implementation shape. A raw PS3 Eye libusb/WinUSB cadence probe
|
|
measured both PS3 Eyes simultaneously at about 183 fps each at 320x240 and
|
|
57-59 fps each at 640x480. A native Kernel Streaming LeapUVC probe can pull
|
|
frames through usbvideo.sys without Media Foundation, DirectShow, OpenCV, or
|
|
frame processing; after discarding a two-second warm-up, the useful 640x240
|
|
stereo IR mode advertises 115 fps and currently measures about 110.8 fps
|
|
with eight queued asynchronous KS reads. The half-height 640x120 mode
|
|
advertises 214 fps and currently measures about 205.9 fps. Plain Kiyo is
|
|
usable through KS when manual exposure is forced shorter: 1080p MJPG reaches
|
|
about 30 fps and 720p MJPG reaches about 51.5 fps. Kiyo Pro advertises 60
|
|
fps MJPG at 720p/1080p but currently delivers only about 25 fps through the
|
|
same KS path even with manual exposure tuning. Standard KS low-light
|
|
compensation off does not restore 60 fps, powerline 60Hz only raises Pro
|
|
cadence to about 29 fps, and standard extended Video HDR is not exposed on
|
|
this KS path. The controls-only probe now parses USB VideoControl
|
|
descriptors and found the Kiyo Pro's two vendor extension units. One
|
|
extension unit answers writable KS selectors. Moving the Pro directly to a
|
|
motherboard USB3 port changed the path to `root_hub30`, `bcdUSB=0x0320`,
|
|
and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still
|
|
reports `UsbHighSpeed` rather than `UsbSuperSpeed`, and baseline cadence is
|
|
still about 25 fps across every tested format. The first simultaneous local
|
|
camera pull only had three visible devices: LeapUVC, Kiyo Pro, and one PS3
|
|
Eye. The KS multi-stream harness plus raw PS3 Eye probe pulled all three at
|
|
once, but Leap cadence collapsed under shared load: with PS3 Eye at
|
|
640x480@60 the set delivered Leap 40.12 fps, Kiyo Pro 24.99 fps, and PS3
|
|
Eye 56.06 fps; with PS3 Eye at 320x240@187 it delivered Leap 81.10 fps,
|
|
Kiyo Pro 25.33 fps, and PS3 Eye 185.74 fps. After port swapping, five
|
|
camera devices / six optical sensors were visible and pulled at once:
|
|
LeapUVC's packed stereo IR pair, Kiyo Pro, regular Kiyo, and two PS3 Eyes.
|
|
With both Eyes at 640x480@60, the set delivered Leap stereo 57.33 fps,
|
|
Kiyo Pro 24.97 fps, Kiyo 30.30 fps, PS3 Eye 0 59.59 fps, and PS3 Eye 1
|
|
57.60 fps. With both Eyes at 320x240@187, it delivered Leap stereo 81.05
|
|
fps, Kiyo Pro 24.97 fps, Kiyo 30.28 fps, PS3 Eye 0 187.60 fps, and PS3
|
|
Eye 1 187.03 fps. Kiyo Pro still negotiates high-speed USB, and Leap
|
|
remains topology-sensitive under shared load. A follow-up moved Leap alone
|
|
onto the blue rear USB3 group/root_hub30#5 while both Kiyos shared
|
|
root_hub30#7&18eea21b and one visible PS3 Eye used root_hub30#7&124f193f;
|
|
Leap still delivered only 80.99 fps with the other visible cameras active,
|
|
so the remaining loss is not explained by same-root-hub contention alone.
|
|
Splitting Kiyo Pro and one PS3 Eye to Raven left the Mimir host with
|
|
LeapUVC, regular Kiyo, and one PS3 Eye, but local cadence still stayed low:
|
|
Leap delivered 81.34 fps with Kiyo+Eye, 82.77 fps with Kiyo only, and only
|
|
84.18 fps in a same-session Leap-only baseline. The previous ~110.8 fps
|
|
Leap result was not reproduced until the Focusrite Scarlett Solo was moved
|
|
away from the Leap USB neighborhood. After that, Leap-only baseline rose to
|
|
95.48 fps, best full-height scenario to 101.67 fps, and the local split set
|
|
delivered Leap 105.10 fps, regular Kiyo 30.17 fps, and one PS3 Eye 185.81
|
|
fps. The split-host layout is viable if the Leap's USB neighborhood stays
|
|
free of the Focusrite and other heavy devices. Raven's PS3 Eye raw interface
|
|
now has the same libwdi WinUSB binding as the local Eye and can be opened by
|
|
the copied cadence probe, but Raven Eye cadence is currently unusable:
|
|
320x240@187 delivered 132.24 fps then 78.93 fps, and 640x480@60 delivered
|
|
6.32 fps with multi-second gaps. Moving that Eye back to the Mimir host
|
|
produced the current clean local topology: LeapUVC, regular Kiyo, and two
|
|
PS3 Eyes run in parallel while Kiyo Pro stays off-host. With both Eyes at
|
|
320x240@187, Leap delivered 106.07 fps, Kiyo 30.33 fps, and the Eyes 186.57
|
|
/ 186.14 fps. With both Eyes at 640x480@60, Leap delivered 108.07 fps, Kiyo
|
|
30.34 fps, and the Eyes 57.79 / 58.08 fps. Moving Kiyo Pro back to the
|
|
Mimir host while regular Kiyo was absent kept Leap healthy: with both Eyes
|
|
at 320x240@187, Leap delivered 110.80 fps, Kiyo Pro 24.97 fps, and the Eyes
|
|
187.46 / 186.89 fps; with both Eyes at 640x480@60, Leap delivered 110.72
|
|
fps, Kiyo Pro 24.87 fps, and the Eyes 59.91 / 59.19 fps. Kiyo Pro still
|
|
negotiates UsbHighSpeed(2), but it does not poison Leap cadence in the
|
|
current layout. Moving regular Kiyo from front panel to a rear port made
|
|
all five camera devices / six optical sensors visible locally. With both
|
|
Eyes at 320x240@187, the full set delivered Leap 110.84 fps, Kiyo Pro 25.08
|
|
fps, Kiyo 30.18 fps, PS3 Eye 0 187.59 fps, and PS3 Eye 1 187.02 fps. With
|
|
both Eyes at 640x480@60, it delivered Leap 110.76 fps, Kiyo Pro 25.14 fps,
|
|
Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
|
|
the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
|
|
next_action: >
|
|
Add the synchronization actuator after the current chirplet measurement
|
|
loop: smooth delay/SRO estimates, then drive variable-rate resampling and
|
|
fractional-delay correction per non-reference stream. Replace the
|
|
diagnostic JSON-line process bridge with direct native capture drivers that
|
|
push buffer handles into Mimir.Runtime. Current chirplet-backed
|
|
Mimir.BufferSmoke proves loopback-referenced aligned audio for Scarlett
|
|
loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when 9-16 kHz
|
|
calibration chirplets are playing. After a later replug, both PS3 Eye audio
|
|
endpoints emitted 480-frame WASAPI blocks into runtime buffers again while
|
|
both PS3 Eye camera buffers were live. Treat PS3 Eye audio as
|
|
enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from
|
|
the Leap USB neighborhood and treat Kiyo Pro as a ~25 fps source until its
|
|
SuperSpeed negotiation is fixed.
|