Mimir/docs/implementation-plan.md

171 lines
10 KiB
Markdown
Raw Normal View History

2026-05-21 18:48:47 +01:00
# Implementation Plan
2026-05-18 10:56:01 +01:00
## Current Cut Line
2026-05-21 18:48:47 +01:00
Mimir is now a C# app/runtime plus native reservoir project. The live stream
machine must be direct-driver and native-buffer first:
2026-05-23 18:12:31 +01:00
- `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12
2026-05-21 18:48:47 +01:00
bridge.
- `src/Mimir.Runtime` owns stream descriptors, source polling, direct push
ingest, one rolling buffer per configured audio/video stream, and the
2026-05-23 18:12:31 +01:00
synchronization hub Fensalir can inspect.
- The default five-second window is an intentional latency/memory trade: use it
to line up streams and extract the volumetric audio/video field before OBS
sees program output.
2026-05-21 18:48:47 +01:00
- `native/reservoir` owns the lower native rolling-buffer invariant for
2026-05-23 18:12:31 +01:00
Fensalir/Faust integration.
2026-05-21 18:48:47 +01:00
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN OBS feeds. It is not
the synchronized program authority.
2026-05-21 18:48:47 +01:00
The old script stack is gone. Do not add a compatibility edge unless it protects
a named invariant that the native runtime cannot protect yet.
2026-05-21 10:27:09 +01:00
2026-05-18 10:56:01 +01:00
## Implemented
2026-05-21 18:48:47 +01:00
- Mimir public identity, branding, and Face memory.
- `Mimir.slnx` with `src/Mimir.App` and `src/Mimir.Runtime`.
2026-05-23 18:12:31 +01:00
- Fensalir host bootstrapping from `Mimir.App`.
2026-05-21 18:48:47 +01:00
- `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and
`IMimirStreamSource`.
- Configurable five-second default rolling buffers for local and network audio
and video streams.
- `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers.
- `MimirProcessStreamSource` for bridge/network command edges.
- `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata
2026-05-23 18:12:31 +01:00
from native probes into the same rolling buffers Fensalir inspects. One probe
process can accept multiple emitted `sourceId` values so it does not reopen
the same camera set once per stream.
- `src/Mimir.BufferSmoke` loads the runtime config, polls the synchronization
hub, and prints the actual rolling buffers. Use `--require-samples` when an
2026-05-22 16:46:50 +01:00
empty declared sensor buffer should fail the run. Use `--chirplet-self-test`
to render the canonical timeline into memory and verify that the constrained
decoder recovers sub-frame anchors without hardware. Use
`--standalone-chirp-bin-self-test` to verify that a receiver with only the
codebook/schedule can recover canonical source offset from delayed audio.
- `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback
block metadata and emits `audio-block` JSON events for the diagnostic runtime
2026-05-23 10:40:14 +01:00
adapter. It can probe requested shared/exclusive formats so driver state is
explicit, but Scarlett production capture belongs on ASIO.
2026-05-23 16:02:42 +01:00
- `native/probes/asio_audio_cadence` opens the registered Focusrite ASIO COM
driver, reports channel counts, buffer sizing, supported sample rates, and can
2026-05-23 17:07:06 +01:00
run a short input callback capture. Current Starfire Focusrite USB ASIO proof
with the Scarlett Solo 4th Gen shows 4 inputs / 2 outputs, including
`Loopback 1/2`, 192-frame preferred buffers, 44.1-192 kHz support, and
2026-05-23 17:57:09 +01:00
nonzero 4-channel `Int32LSB` input callbacks at 192 kHz. `--monitor-sweep`
emits low-gain ASIO output bursts and synchronously measures loopback/mic
2026-05-23 19:04:16 +01:00
response per frequency so ultrasonic acoustic claims stay measured. The probe
can also play raw mono Float32 timeline audio with `--play-f32-mono` and
capture raw interleaved Float32 ASIO input with `--record-f32-interleaved`.
2026-05-22 13:24:38 +01:00
- `MimirChirpletTimeline` owns the structured birdsong-like calibration stream,
PCM segment rendering, matched timing trace, and per-band response kernels.
The default timeline is an order-3 de Bruijn symbol sequence over 32
time/frequency constellation symbols, so any three consecutive correctly
detected symbols identify a timeline event inside the current operating
horizon. Symbol identity is carried by start band, glide shape, duration, and
following inter-chirp gap.
2026-05-22 15:53:57 +01:00
- `MimirChirpletSymbolCodebook` owns the 32 symbol definitions. Each symbol has
a unique chirp shape, with inter-chirp rhythm as additional code evidence.
2026-05-22 14:53:26 +01:00
- `MimirChirpletStreamDecoder` is the first constrained chirplet-transform
2026-05-22 15:00:12 +01:00
receiver. It owns a bounded PCM window, emits transform frames with multiple
2026-05-22 16:46:50 +01:00
phase-invariant symbol candidates and per-candidate refined sample offsets,
decodes code-valid triplet anchors through a local trellis that requires gap
and clock coherence, and fits a per-source sample clock from those anchors.
2026-05-22 01:22:36 +01:00
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
2026-05-22 14:53:26 +01:00
sample-bearing audio blocks are resampled into the Scarlett loopback timeline.
2026-05-22 15:43:16 +01:00
The analyzer derives delay only from matched decoded triplet timeline anchors.
A source without at least three matched anchors has no timing report for that
2026-05-23 10:40:14 +01:00
window. It accepts Float32, Int16, Int24, and Int32 PCM windows so ASIO/native
capture can feed true interface formats without a pre-conversion shim.
2026-05-22 12:46:51 +01:00
- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source
sync state: latest fractional delay, smoothed delay, confidence, per-band
response evidence, and delay-slope/SRO estimate in ppm.
2026-05-22 14:16:50 +01:00
- `MimirRuntime` updates audio sync analysis online as a bounded rotating
service and can emit live sync telemetry with
`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry read cached reports/states;
they do not run synchronization analysis.
2026-05-23 18:12:31 +01:00
- `MimirRuntime` queues chirplet timeline PCM through Fensalir audio when the
active timing witness is allowed. `MimirAudioSynchronizationSettings.Mode`
selects `chirp-only`, `passive`, or `hybrid`; passive disables active
2026-05-23 19:22:25 +01:00
emission, chirp-only emits the active witness continuously, and hybrid emits
the active pilot only while passive confidence is below threshold.
- `MimirPassiveAudioSynchronizationEstimator` is the first program-audio timing
path. It estimates loopback-to-mic delay with PHAT-weighted cross-spectrum
correlation so music can act as the default timing witness before any audible
watermark is needed.
2026-05-23 19:22:25 +01:00
- `MimirChirpBinTimeline` is the active calibration path for both `chirp-only`
and `hybrid`. It renders a fixed-slope chirp-bin codebook and decodes symbols
with cheap event-energy proposals, dechirp plus fixed Goertzel bins, and the
same de Bruijn triplet timeline-anchor machine. The detector keeps
time/frequency ambiguity as candidate symbol/offset pairs so code constraints
can choose the coherent path. The analyzer refines the final fractional delay
with constrained local waveform correlation around the decoded active delay.
Reports/states expose delay in microseconds as well as fractional samples.
Hybrid emits this as low-gain half-second bursts every two seconds only while
passive confidence is weak.
2026-05-23 10:27:16 +01:00
Use `--chirp-bin-self-test` to prove the codebook/decoder and
`--hybrid-sync-self-test` to prove that the analyzer can recover a fractional
delay from a one-second rolling-buffer chirp-bin window. The current synthetic
microsecond proof recovers a 317.375-sample delay with 0.369 us error.
2026-05-23 19:22:25 +01:00
- `Mimir.BufferSmoke --render-chirp-bin-f32` and `--analyze-asio-f32` provide
the current active Scarlett artifact proof. A 192 kHz chirp-bin run decoded
Focusrite `Loopback 1 -> Loopback 2` at `0.000 us` with 12 matched anchors and
0.999 confidence. Physical inputs did not produce matched anchors in that run,
so acoustic robustness remains separate from the clean loopback timing proof.
A standalone 192 kHz synthetic receiver test recovers a 500 ms delayed audio
stream to below printed microsecond precision using only the chirp-bin
codebook and schedule state.
2026-05-21 18:48:47 +01:00
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
timestamp, and native/GPU handle metadata.
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
driver-facing seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or
shared texture capture.
- `native/reservoir` with one shared-edge rolling buffer, typed views, C ABI,
source-id hashing, producer helpers, and typed audio/render payload
descriptors.
- Windows bridge scripts for sender discovery/start/stop and simple OBS Media
Source ingest.
- Documentation for OBS receiver setup, native rebuild boundaries, the viable
stream app, and the Mimir Face.
2026-05-18 10:56:01 +01:00
## Temporary
2026-05-21 18:48:47 +01:00
- Audio and video may still traverse separate OBS/SRT endpoints during bridge
testing so OBS can preserve independent controls.
- Process-backed stream sources are only acceptable for network bridge feeds or
diagnostics. Six-camera local ingest belongs behind direct capture drivers.
- Frame-event process sources are diagnostic only. They prove source cadence and
runtime plumbing without dragging stdout bytes into the pixel hot loop.
2026-05-21 18:48:47 +01:00
- Calibration artifacts may remain on disk as evidence, but live state must be
2026-05-23 18:12:31 +01:00
in memory inside Mimir/Fensalir/native runtime surfaces.
2026-05-18 10:56:01 +01:00
## Next
2026-05-22 01:01:12 +01:00
1. Replace the frame-event diagnostic bridge with concrete direct capture
drivers for Leap stereo IR first, then the
2026-05-21 18:48:47 +01:00
other cameras.
2026-05-22 01:01:12 +01:00
2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
2026-05-21 18:48:47 +01:00
frame cadence in the rolling buffers.
2026-05-23 16:02:42 +01:00
3. Replace the WASAPI frame-event diagnostic bridge with native audio capture
workers for local mic, loopback, and network audio feeds. Use the verified
Focusrite ASIO path for Scarlett capture; WASAPI remains a diagnostic
witness, not the Scarlett hot path.
2026-05-23 19:22:25 +01:00
4. Prove the active chirp-bin path through real microphones, not only Scarlett
loopback; tune acoustic bands/gain/code spacing from measured mic response
without weakening the standalone codebook/schedule receiver invariant.
2026-05-23 19:04:16 +01:00
5. Add the synchronization actuator: drive a variable-rate resampler and
2026-05-22 12:46:51 +01:00
fractional delay line per non-reference stream from the smoothed
2026-05-22 16:46:50 +01:00
`MimirAudioSynchronizationState`. First, prove the constrained chirplet
decoder through real loopback and microphone paths so every correctly heard
triplet becomes a deterministic timeline anchor before the actuator moves
samples.
2026-05-23 19:04:16 +01:00
6. Prove the chirp-bin hybrid fallback through real loopback and microphones
2026-05-22 19:50:02 +01:00
with probe durations long enough to keep loopback and mic windows live.
2026-05-23 19:04:16 +01:00
7. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence,
2026-05-21 18:48:47 +01:00
source timestamps, and output settings are visible and adjustable.
2026-05-23 19:04:16 +01:00
8. Move GPU feature extraction, fusion, material fitting, render budgeting, and
2026-05-23 18:12:31 +01:00
Spout2 publication into Fensalir.
2026-05-23 19:04:16 +01:00
9. Move mic alignment, room suppression, voice separation, spatialization, and
2026-05-21 18:48:47 +01:00
stem generation into Faust/native DSP.
2026-05-23 19:04:16 +01:00
10. Keep the OBS bridge witness ledger as evidence before expanding receiver
2026-05-21 18:48:47 +01:00
machinery.