111 lines
6 KiB
Markdown
111 lines
6 KiB
Markdown
# Implementation Plan
|
|
|
|
## Current Cut Line
|
|
|
|
Mimir is now a C# app/runtime plus native reservoir project. The live stream
|
|
machine must be direct-driver and native-buffer first:
|
|
|
|
- `src/Mimir.App` hosts Aquarium Engine for windowing, rendering, and the D3D12
|
|
bridge.
|
|
- `src/Mimir.Runtime` owns stream descriptors, source polling, direct push
|
|
ingest, one rolling buffer per configured audio/video stream, and the
|
|
synchronization hub Aquarium can inspect.
|
|
- The default five-second window is an intentional latency/memory trade: use it
|
|
to line up streams and extract the volumetric audio/video field before OBS
|
|
sees program output.
|
|
- `native/reservoir` owns the lower native rolling-buffer invariant for
|
|
Aquarium/Faust integration.
|
|
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN OBS feeds. It is not
|
|
the synchronized program authority.
|
|
|
|
The old script stack is gone. Do not add a compatibility edge unless it protects
|
|
a named invariant that the native runtime cannot protect yet.
|
|
|
|
## Implemented
|
|
|
|
- Mimir public identity, branding, and Face memory.
|
|
- `Mimir.slnx` with `src/Mimir.App` and `src/Mimir.Runtime`.
|
|
- Aquarium host bootstrapping from `Mimir.App`.
|
|
- `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and
|
|
`IMimirStreamSource`.
|
|
- Configurable five-second default rolling buffers for local and network audio
|
|
and video streams.
|
|
- `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers.
|
|
- `MimirProcessStreamSource` for bridge/network command edges.
|
|
- `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata
|
|
from native probes into the same rolling buffers Aquarium inspects. One probe
|
|
process can accept multiple emitted `sourceId` values so it does not reopen
|
|
the same camera set once per stream.
|
|
- `src/Mimir.BufferSmoke` loads the runtime config, polls the synchronization
|
|
hub, and prints the actual rolling buffers. Use `--require-samples` when an
|
|
empty declared sensor buffer should fail the run.
|
|
- `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback
|
|
block metadata and emits `audio-block` JSON events for the diagnostic runtime
|
|
adapter.
|
|
- `MimirChirpletTimeline` owns the structured birdsong-like calibration stream,
|
|
PCM segment rendering, matched timing trace, and per-band response kernels.
|
|
The default timeline is an order-3 de Bruijn symbol sequence over 32 separated
|
|
chirp symbols, so any three consecutive correctly detected symbols identify a
|
|
timeline event inside the current operating horizon.
|
|
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
|
|
sample-bearing audio blocks are resampled into the Scarlett loopback timeline
|
|
and compared by chirplet-energy delay estimation with fractional peak
|
|
interpolation. When timeline symbols are decoded, matched event indices refine
|
|
the delay estimate and are exposed in telemetry.
|
|
- `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
|
provisional aligned mono frame for loopback-referenced audio channels that
|
|
clear the confidence gate.
|
|
- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source
|
|
sync state: latest fractional delay, smoothed delay, confidence, per-band
|
|
response evidence, and delay-slope/SRO estimate in ppm.
|
|
- `MimirRuntime` updates audio sync analysis online and can emit live sync
|
|
telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`.
|
|
- `MimirRuntime` continuously queues chirplet timeline PCM through Aquarium
|
|
audio so the loopback and acoustic mic buffers carry a shared timing witness.
|
|
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
|
|
timestamp, and native/GPU handle metadata.
|
|
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
|
|
driver-facing seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or
|
|
shared texture capture.
|
|
- `native/reservoir` with one shared-edge rolling buffer, typed views, C ABI,
|
|
source-id hashing, producer helpers, and typed audio/render payload
|
|
descriptors.
|
|
- Windows bridge scripts for sender discovery/start/stop and simple OBS Media
|
|
Source ingest.
|
|
- Documentation for OBS receiver setup, native rebuild boundaries, the viable
|
|
stream app, and the Mimir Face.
|
|
|
|
## Temporary
|
|
|
|
- Audio and video may still traverse separate OBS/SRT endpoints during bridge
|
|
testing so OBS can preserve independent controls.
|
|
- Process-backed stream sources are only acceptable for network bridge feeds or
|
|
diagnostics. Six-camera local ingest belongs behind direct capture drivers.
|
|
- Frame-event process sources are diagnostic only. They prove source cadence and
|
|
runtime plumbing without dragging stdout bytes into the pixel hot loop.
|
|
- Calibration artifacts may remain on disk as evidence, but live state must be
|
|
in memory inside Mimir/Aquarium/native runtime surfaces.
|
|
|
|
## Next
|
|
|
|
1. Replace the frame-event diagnostic bridge with concrete direct capture
|
|
drivers for Leap stereo IR first, then the
|
|
other cameras.
|
|
2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
|
|
frame cadence in the rolling buffers.
|
|
3. Replace the WASAPI frame-event diagnostic bridge with native audio capture
|
|
workers for local mic, loopback, and network audio feeds.
|
|
4. Add the synchronization actuator: drive a variable-rate resampler and
|
|
fractional delay line per non-reference stream from the smoothed
|
|
`MimirAudioSynchronizationState`. The current aligned frame is integer-delay
|
|
only and should not pretend to be the final DSP path.
|
|
First, harden the online state filter so the continuous chirplet lock stops
|
|
producing jumpy delay estimates.
|
|
5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
|
|
source timestamps, and output settings are visible and adjustable.
|
|
6. Move GPU feature extraction, fusion, material fitting, render budgeting, and
|
|
Spout2 publication into Aquarium.
|
|
7. Move mic alignment, room suppression, voice separation, spatialization, and
|
|
stem generation into Faust/native DSP.
|
|
8. Keep the OBS bridge witness ledger as evidence before expanding receiver
|
|
machinery.
|