107 lines
5.6 KiB
YAML
107 lines
5.6 KiB
YAML
workspace:
|
|
name: Mimir
|
|
status: active
|
|
focus: direct-driver-aquarium-stream-app
|
|
implementation_name: Mimir public surface / native runtime contracts
|
|
premise: >
|
|
Mimir is the local realtime field machine: cameras, microphones, speakers,
|
|
network feeds, and timing signals become one coherent OBS-facing volumetric
|
|
program surface. The live path is a C# Aquarium-hosted app with in-memory
|
|
rolling buffers and native capture/runtime boundaries.
|
|
|
|
objective:
|
|
primary: >
|
|
Build one synchronized spatial stream machine from six cameras, six
|
|
microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP,
|
|
and OBS output.
|
|
current: >
|
|
Implement direct camera/audio capture into Mimir.Runtime buffers, starting
|
|
with Leap stereo IR as the timing-camera candidate, then expose runtime
|
|
health/settings/output controls through Aquarium UI.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must use NVENC when the bridge path is active.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 bridge is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
- Live ingest belongs to Mimir.Runtime, native drivers, Aquarium, and
|
|
Faust/native DSP.
|
|
- Edge JSON is schema/diagnostic only; live process data is typed or in
|
|
memory.
|
|
- Networked feeds are producers into the live runtime, not independent clock
|
|
authorities.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport only for bridge utilities.
|
|
- One stable bridge port equals one OBS source endpoint.
|
|
- One rolling live window owns the timing edge and expiry.
|
|
- Typed views may index the rolling buffer, but no per-kind ring owns
|
|
retention.
|
|
- Aquarium owns production UI, GPU fusion, D3D12 interop, and Spout2 output.
|
|
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
|
|
stems.
|
|
- Local secrets remain untracked.
|
|
|
|
accepted_design:
|
|
- `Mimir.slnx` owns the app surface.
|
|
- `src/Mimir.App` launches Aquarium Engine as host/window/D3D12 library.
|
|
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
|
|
`MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`,
|
|
direct native ingest, process bridge ingest, and video capture driver seams.
|
|
- `MimirVideoFrameDescriptor` carries width, height, pixel format, stride,
|
|
device timestamp, and optional native/GPU handle.
|
|
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
|
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
|
|
texture capture.
|
|
- `native/reservoir` owns the lower shared-edge rolling-buffer invariant for
|
|
Aquarium/Faust integration.
|
|
- `config/perfect-machine.example.json` declares live owners, sensors,
|
|
reservoir views, and runtime policy.
|
|
- `.voidbot` state is part of repo memory and must be committed separately
|
|
whenever dirty.
|
|
|
|
active_subgoals:
|
|
- Implement the first concrete Leap direct capture driver and measure cadence.
|
|
- Add remaining local camera drivers through `IMimirVideoCaptureDriver`.
|
|
- Add native audio capture workers for mic, loopback, and network audio feeds.
|
|
- Bind Aquarium UI to stream health, buffer depth, source timestamps,
|
|
settings, and output management.
|
|
- Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
|
|
replaces them.
|
|
|
|
current_status:
|
|
summary: >
|
|
The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS
|
|
bridge scripts, and persistence surfaces. Script-era packages, tests,
|
|
helper tools, dependency manifests, and stale docs have been removed from
|
|
the working tree. Research archives remain as reference-only memory for raw
|
|
native/GPU implementation shape. A raw PS3 Eye libusb/WinUSB cadence probe
|
|
measured both PS3 Eyes simultaneously at about 183 fps each at 320x240 and
|
|
57-59 fps each at 640x480. A native Kernel Streaming LeapUVC probe can pull
|
|
frames through usbvideo.sys without Media Foundation, DirectShow, OpenCV, or
|
|
frame processing; after discarding a two-second warm-up, the useful 640x240
|
|
stereo IR mode advertises 115 fps and currently measures about 110.8 fps
|
|
with eight queued asynchronous KS reads. The half-height 640x120 mode
|
|
advertises 214 fps and currently measures about 205.9 fps. Plain Kiyo is
|
|
usable through KS when manual exposure is forced shorter: 1080p MJPG reaches
|
|
about 30 fps and 720p MJPG reaches about 51.5 fps. Kiyo Pro advertises 60
|
|
fps MJPG at 720p/1080p but currently delivers only about 25 fps through the
|
|
same KS path even with manual exposure tuning. Standard KS low-light
|
|
compensation off does not restore 60 fps, powerline 60Hz only raises Pro
|
|
cadence to about 29 fps, and standard extended Video HDR is not exposed on
|
|
this KS path. The controls-only probe now parses USB VideoControl
|
|
descriptors and found the Kiyo Pro's two vendor extension units. One
|
|
extension unit answers writable KS selectors. Moving the Pro directly to a
|
|
motherboard USB3 port changed the path to `root_hub30`, `bcdUSB=0x0320`,
|
|
and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still
|
|
reports `UsbHighSpeed` rather than `UsbSuperSpeed`, and baseline cadence is
|
|
still about 25 fps across every tested format.
|
|
next_action: >
|
|
Turn the warmed, queued Leap KS path into an `IMimirVideoCaptureDriver`,
|
|
preserve direct control hooks for timing-camera state, and decide whether
|
|
Kiyo Pro reaches 60 fps after confirming a true `UsbSuperSpeed` cable/port
|
|
path; if it remains locked at 25 fps there, inspect explicit frame-interval
|
|
negotiation or vendor selectors.
|