372 lines
21 KiB
Markdown
372 lines
21 KiB
Markdown
# Audio Field
|
|
|
|
Mimir's audio field is six microphones, loopback/program reference, and two
|
|
calibration speakers aligned into one presentation timeline.
|
|
|
|
## Live Target
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A["mic / loopback capture drivers"] --> B["Mimir.Runtime audio buffers"]
|
|
C["speaker probe scheduler"] --> B
|
|
B --> D["native alignment + phase state"]
|
|
D --> E["Faust/native DSP"]
|
|
E --> F["host voice"]
|
|
E --> G["co-streamer voice"]
|
|
E --> H["ambient / transients"]
|
|
E --> I["loopback stems"]
|
|
E --> J["spatial bed"]
|
|
F --> K["OBS"]
|
|
G --> K
|
|
H --> K
|
|
I --> K
|
|
J --> K
|
|
```
|
|
|
|
## Invariants
|
|
|
|
- Starfire Realtek is the active chirp speaker and loopback-measurement path in
|
|
the current routing. The Windows system default output is now Scarlett for
|
|
monitoring, so default-output chirps are wrong unless they explicitly target
|
|
Realtek.
|
|
- Scarlett ASIO is the professional mic and monitor interface. Its outputs are
|
|
not the current active chirp-emission authority.
|
|
- Focusrite dialogue mics are the voice anchors.
|
|
- Camera mics are spatial/context witnesses.
|
|
- Loopback/program audio is timing evidence where available; it outranks
|
|
acoustic mics for clock/timing because it is the emitted program surface.
|
|
- Distributed inputs must be aligned and resampled before they become program
|
|
stems.
|
|
- The five-second runtime window is allowed to be spent on alignment,
|
|
resampling, separation, and spatial-field extraction. Low latency loses to a
|
|
coherent volumetric sound field here.
|
|
- Probe signals are budgeted telemetry, not a permanent audio bed.
|
|
- Faust/native DSP owns the hot separation and spatialization graph.
|
|
|
|
## Synchronization Modes
|
|
|
|
`MimirAudioSynchronizationSettings.Mode` is the runtime authority for what
|
|
timing evidence Mimir is allowed to emit:
|
|
|
|
- `chirp-only`: emit the deterministic calibration timeline and decode timing
|
|
only from that active witness. This is the lab/debug mode and the fallback for
|
|
silent program material.
|
|
- `passive`: do not emit calibration audio. Use loopback/program audio as the
|
|
timing witness by estimating delay between the loopback buffer and each mic
|
|
buffer.
|
|
- `hybrid`: prefer passive program-audio evidence when confidence is high, then
|
|
emit a watermark when confidence falls. The passive side uses bounded
|
|
GCC-PHAT-style phase correlation.
|
|
|
|
The mode belongs to the runtime, not the decoder. The decoder should consume
|
|
known timing evidence; it should not decide whether Mimir is allowed to make
|
|
sound.
|
|
|
|
The passive estimator is the first real program-audio path, not the final DSP
|
|
actuator. It removes DC, pre-emphasizes the window, applies a Hann taper, runs a
|
|
PHAT-weighted cross spectrum, and reports the strongest loopback-to-mic lag
|
|
inside a bounded window. Positive delay still means the candidate mic is late
|
|
relative to loopback. Negative lags are treated as contradictory passive
|
|
evidence and carry zero confidence. A single passive window is also capped below
|
|
certainty; full confidence belongs to repeated coherent state over time, not
|
|
one attractive correlation peak.
|
|
|
|
The active calibration path is deliberately receiver-cheap in both `chirp-only`
|
|
and `hybrid`. `MimirBioacousticTimeline` is the live runtime witness: it emits
|
|
low-gain log-frequency words made from short formant-rich syllables, rhythm
|
|
offsets, and speaker-specific variants. Acquisition is an energy proposal pass
|
|
plus bounded motif matching; timeline placement comes from direct word identity,
|
|
and a constrained local waveform correlation around the decoded delay
|
|
provides the final fractional offset. The decoder keeps per-band response
|
|
evidence for each detected motif so the same stream can calibrate timing and the
|
|
speaker/room/mic transfer function.
|
|
|
|
`MimirChirpBinTimeline` remains as the older controlled calibration/reference
|
|
artifact. Its full bin-energy surface and `MimirChirpBinCalibrationModel` still
|
|
matter for measured usable bands, expected-symbol versus observed-bin confusion,
|
|
timing residuals, delay hypotheses, phase summaries, and adaptive codebook
|
|
plans. That evidence is allowed to shape the bioacoustic receiver next; it is
|
|
not the active runtime sound.
|
|
|
|
The raw profile is kept even when a timing report is rejected, because a failed
|
|
sync window can still teach us which bands survived the speaker/room/mic path.
|
|
`MimirBioacousticProbeScheduler` is now the runtime-side cadence owner for that
|
|
feedback loop: it consumes smoothed sync states plus per-band frequency response
|
|
residuals, schedules a targeted probe only when confidence pressure and duty
|
|
budget allow it, and publishes aggregate/per-source sync and frequency-response
|
|
confidence telemetry. `MimirSynchronizationHub.UpdateBioacousticProbeSchedule`
|
|
is the live hub surface; `MimirRuntime` prints
|
|
`mimir-bioacoustic-probe-schedule` and `mimir-bioacoustic-probe-source` lines so
|
|
operators can watch confidence climb as probes are spent. The scheduler owns
|
|
when to spend active sound. It does not own the output device: Starfire Realtek
|
|
still must be selected explicitly by the eventual live renderer.
|
|
The first explicit Realtek renderer proof now lives in the repo tooling:
|
|
`Mimir.WasapiLoopback` can enumerate render endpoints, select one by friendly
|
|
name substring, and play mono Float32 probe audio through shared-mode WASAPI.
|
|
`Mimir.BufferSmoke --targeted-bioacoustic-probe-realtk-live` renders a targeted
|
|
probe plan, starts Scarlett ASIO capture, plays the probe through
|
|
`Speakers (Realtek(R) Audio)`, and scores the captured shotgun/cardioid bands.
|
|
The targeted probe planner now caps default acoustic probe selection at 12 kHz
|
|
and mildly biases low/mid measurable bands, because commodity speakers and the
|
|
current Scarlett mics have little reason to carry useful calibration evidence
|
|
above that range. The 2026-06-01 proof at gain `0.75` selected the Realtek speaker endpoint and
|
|
captured `artifacts/wasapi/targeted-bioacoustic-realtk-live/20260601-033851`.
|
|
The physical mics heard the burst, but the scored confidence still concentrated
|
|
in low/mid evidence: shotgun peak `0.407887`, cardioid peak `0.074494`, 18.6 kHz
|
|
confidence near zero, and the best reported band was cardioid 3.6 kHz at
|
|
confidence `0.098`. Treat high-band probes as poor questions for this route
|
|
until a better speaker/mic/path proves otherwise.
|
|
The live exploration path is scheduler-driven, not a blind sweep:
|
|
`Mimir.BufferSmoke --bioacoustic-realtk-scheduled-calibration-live` repeatedly
|
|
asks `MimirBioacousticProbeScheduler` for the next probe plan, emits that plan
|
|
through Realtek, captures Scarlett, updates residuals, and asks again. This is
|
|
a live physical I/O diagnostic harness seeded from the current Scarlett residual
|
|
model; the production runtime still needs to feed measured confidence state into
|
|
the same scheduler/emitter path. The 2026-06-01 receipt at
|
|
`artifacts/wasapi/realtk-scheduled-calibration-live/20260601-035354` ran four
|
|
scheduled probe steps at gain `1.15` and wrote `scheduled-response-bands.csv`
|
|
plus `balance-plan.md`. Local peak-versus-percentile-floor scoring found strong
|
|
usable response near `910 Hz`, `3.6 kHz`, and `9.84-9.87 kHz` on both Scarlett
|
|
mics.
|
|
The scheduler and planner share the same measurable frequency boundary
|
|
(`120 Hz` through `12 kHz` by default). Residuals above that window may remain
|
|
diagnostic scars, but they no longer own live probe cadence or the weakest-band
|
|
status when the current room path cannot answer them.
|
|
The receiver must consume persisted models to weight reliable symbols,
|
|
downweight dead bands, apply phase-coherence weighting, apply first-order
|
|
group-delay correction, and search joint global delay/frequency-shift hypotheses
|
|
before selecting the coherent anchor path. If the measured path only leaves a
|
|
smaller alphabet, Mimir should use that reliable symbol set and raise the
|
|
sequence order so the timeline remains unique without pretending dead bands
|
|
still carry code.
|
|
The first concrete two-mic cancellation owner is
|
|
`MimirCalibratedAudioCompositeBuilder`: after fractional alignment and measured
|
|
band correction, it scores calibrated-band coherence across mics, suppresses
|
|
incoherent measured-band components, then performs the confidence/noise-weighted
|
|
sum. `Mimir.BufferSmoke --dual-mic-coherence-cancellation-smoke` proves the
|
|
contract on synthetic shotgun/cardioid witnesses: coherent shared tones survive,
|
|
a mic-local `3.52 kHz` interferer is suppressed, and SNR rises from `1.366 dB`
|
|
to `16.304 dB`. This is still the C# diagnostic/offline contract; Faust/native
|
|
DSP owns the production hot lane.
|
|
The first listenable bridge receipt is
|
|
`scripts/record-kiyo-dual-mic-composite.ps1`. It records Kiyo video and
|
|
Scarlett ASIO raw channels together, converts Scarlett channels 0/1 through
|
|
the offline Faust chain with `--asio-dual-mic-faust-cleaner-wav`, then muxes
|
|
the dialogue WAV with the Kiyo video. That command compiles and runs
|
|
`faust/mimir_alignment_actuator.dsp` first, then feeds the aligned
|
|
shotgun/cardioid stems into `faust/mimir_dual_mic_dialogue_cleaner.dsp`; it is a
|
|
listening receipt for the real DSP chain, not the older C# composite audition.
|
|
The fresh 2026-06-01 receipt at
|
|
`artifacts/runtime/kiyo-dual-mic-composite/kiyo-dual-mic-composite-latest.mp4`
|
|
contains 10.000 s of 640x480 30 fps Kiyo video and 10.000 s of mono AAC from the
|
|
Faust dialogue stem. Run `20260601-051044` estimated the cardioid 359.398
|
|
samples late at 48 kHz (7487.452 us), applied holdback 359.398/0.000 samples to
|
|
shotgun/cardioid before cleanup, wrote a residual WAV, and reported dialogue
|
|
RMS `0.002678` before receipt normalization. The first muxed receipt was almost
|
|
inaudible because `NormalizePeak` only attenuated hot samples; the regenerated
|
|
latest MP4 now peaks the dialogue WAV at -0.724 dBFS with RMS -14.202 dBFS. The
|
|
hot owner is
|
|
`faust/mimir_dual_mic_dialogue_cleaner.dsp`: a two-input/two-output Faust graph
|
|
for aligned shotgun/cardioid witnesses that emits a dialogue composite stem and
|
|
a rejected residual witness using smooth multiband voice shaping, witness
|
|
subtraction, and slow downward expansion. `MimirRuntime` now queues that Faust
|
|
program as `dual-mic-dialogue-cleaner` with `host_voice` and residual `ambient`
|
|
output stems, but only feeds it from the alignment actuator's published
|
|
`aligned_source_*` stem frame. Alignment is therefore a structural precursor to
|
|
cleanup, not an optional sidecar. The routing invariant is covered by
|
|
`Mimir.BufferSmoke --dialogue-cleaner-routing-smoke`: alignment frames queue the
|
|
cleaner DSP and cleaner input block, while cleaner output frames are the
|
|
publishable `NativeProgram` stems. Treat the C# bridge as an audition/scoring
|
|
receipt, not the final native live DSP path.
|
|
The synthetic invariant is
|
|
`dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`;
|
|
it renders the bioacoustic timeline, decodes direct word anchors, and requires a
|
|
stable source clock. The standalone invariant is
|
|
`--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5`;
|
|
it proves that a delayed receiver can recover canonical source time from the
|
|
known codebook and schedule alone. `--bioacoustic-cepstral-smoke` proves the
|
|
next receiver shape: degraded mel-cepstral calls are decoded through an
|
|
augmented MFCC/projection-hash word index, not through dense waveform
|
|
brute-force. `chirp-only` emits this witness continuously;
|
|
`hybrid` still prefers passive program-audio evidence, then emits the low-gain
|
|
bioacoustic witness only when passive confidence is weak. Default hybrid
|
|
watermark gain is intentionally low (`watermarkGain`, or `MIMIR_WATERMARK_GAIN`).
|
|
|
|
Research notes:
|
|
|
|
- [[research/chirplet-sync-decoder/summary|Chirplet Sync Decoder Summary]]
|
|
explains why the hot hybrid path
|
|
moved from dense chirplet matching to dechirp plus FFT/Goertzel bin decoding.
|
|
- [[research/chirplet-sync-decoder/bibliography|Chirplet Sync Decoder Bibliography]]
|
|
links the mirrored chirplet,
|
|
LoRa/chirp-decoder, and fast-transform references.
|
|
- [[research/live-adaptive-sync/summary|Live Adaptive Sync Summary]] covers
|
|
passive program-audio sync,
|
|
GCC-PHAT, and sample-rate-offset estimation.
|
|
- [[research/feedback-calibration/summary|Feedback Calibration Summary]] covers
|
|
online speaker/room/mic
|
|
response calibration.
|
|
|
|
## Next Cut
|
|
|
|
The current diagnostic witness is `native/probes/wasapi_audio_cadence`, which
|
|
emits timestamped WASAPI `audio-block` metadata into `Mimir.Runtime` through the
|
|
frame-event adapter. It can request and verify specific shared or exclusive
|
|
formats for device diagnosis, but it is not the Focusrite hot path. Scarlett
|
|
capture belongs on ASIO so the runtime can use the interface clock and the
|
|
24-bit/192 kHz modes instead of the Windows shared engine. The probe has proven
|
|
Focusrite mic, Kiyo Pro mic, Kiyo mic, both USB Camera / PS3 Eye mics, and
|
|
Scarlett speaker loopback in rolling buffers when loopback audio is actively
|
|
playing. One PS3 Eye mic previously enumerated but produced zero WASAPI packets
|
|
until that Eye was unplugged and replugged.
|
|
|
|
`native/probes/asio_audio_cadence` is the first direct ASIO witness. It opens
|
|
the registered `Focusrite USB ASIO` COM driver, reports channel counts, buffer
|
|
size, sample-rate support, input sample type, and can run a short callback
|
|
capture. With the Scarlett Solo 4th Gen attached to Starfire, Focusrite USB
|
|
ASIO exposes 4 inputs and 2 outputs: `Input 1`, `Input 2`, `Loopback 1`, and
|
|
`Loopback 2`. It accepts 44.1 through 192 kHz, reports a 192-frame preferred
|
|
buffer, and captures nonzero 4-channel `Int32LSB` input callbacks at 192 kHz.
|
|
Raven also has a loopback-capable Scarlett ASIO path at 192 kHz for
|
|
co-streamer/game timing evidence, but Starfire owns the heavy local alignment,
|
|
soundfield, and sensor-fusion work.
|
|
The runtime hot path is now `native/asio_capture` plus
|
|
`MimirAsioStreamSource`. The native DLL owns the Focusrite ASIO COM driver and
|
|
keeps callback buffers in process; the managed source drains one 192 kHz Float32
|
|
block per ASIO input channel into `Mimir.Runtime` buffers (`asio-ch0` through
|
|
`asio-ch3`). The older `--emit-json-blocks` worker path remains a diagnostic
|
|
probe, not the live Scarlett ingest path.
|
|
|
|
The probe also has a `--monitor-sweep` mode for measuring the acoustic ceiling.
|
|
At 192 kHz, ASIO loopback detected emitted 8-40 kHz tones cleanly, but the
|
|
current studio-monitor-to-Scarlett-mic acoustic path was already weak by
|
|
14-16 kHz and very weak above 20 kHz. Treat ultrasonic acoustic sync as
|
|
unproven until a measured transducer/mic path earns it.
|
|
|
|
The active bioacoustic timing proof now runs in memory through the same runtime
|
|
analyzer path. `--bioacoustic-self-test` proves direct word anchors,
|
|
`--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5`
|
|
proves a receiver can recover canonical source time from delayed audio alone,
|
|
and `--chirp-only-sync-self-test --sample-rate 48000` recovers a 317.375-sample
|
|
delay with printed 0.000 us error using `evidence=bioacoustic`.
|
|
`--bioacoustic-cepstral-smoke` runs an automated degradation panel over
|
|
mel-cepstral warping and separable 5-tap blur; the current indexed MFCC decoder
|
|
recovers word identity through the panel, while timing error remains a visible
|
|
pressure for the global clock-hypothesis stage. The live ingest
|
|
proof now feeds ASIO callbacks into `Mimir.Runtime` without a process/stdout
|
|
bridge: a two-second BufferSmoke run at 192 kHz ingested more than 12,000
|
|
sample-bearing blocks and retained 2,048 blocks per channel across `asio-ch0`
|
|
through `asio-ch3`.
|
|
|
|
The full probe runtime config now enables sample-bearing blocks for every local
|
|
audio source. `MimirBioacousticTimeline` owns the emitted calibration stream and
|
|
the motif-matching shape used to analyze it. The default timeline is a
|
|
deterministic set of self-identifying log-frequency words with separate left and
|
|
right speaker variants. Any correctly detected word identifies the event index
|
|
inside the current operating horizon, so a receiver can place its audio window
|
|
on the canonical timeline without being handed Mimir's runtime clock. Mimir queues
|
|
half-second PCM segments ahead of the audio cursor. Each motif is a short phrase
|
|
of formant-rich syllables with log-frequency contour and rhythm offsets, so the
|
|
code is carried by both spectral shape and timing.
|
|
|
|
The intended decoder is a compact constrained song-contour transform, not a
|
|
generic time-frequency explorer and not an outlier filter around bad guesses.
|
|
Mimir owns the emitter, so the receiver keeps its ear open for known song words
|
|
and maps each successful call directly to canonical time. A word is not a
|
|
single scalar event: syllable onsets, bends, formants, rhythm offsets, payload
|
|
ornaments, speaker tint, and log-mel contour all expose local time/frequency
|
|
constraints. One call can pin a cluster of anchors at once. A decoded anchor
|
|
means observed sample offset `S` corresponds to emitted event time `T`. A stream
|
|
of anchors fits the source clock directly:
|
|
|
|
```text
|
|
observed_sample = source_offset + canonical_seconds * effective_sample_rate
|
|
```
|
|
|
|
Delay and sampling-rate offset are derived by comparing the loopback clock fit
|
|
to each mic clock fit over common canonical time. The state tracker is not
|
|
allowed to launder invalid words into plausible timing. If a stream cannot
|
|
produce at least one matched canonical song word, it has not decoded timing for
|
|
that window. Reports carry rounded integer delay, fractional delay, and the
|
|
count/confidence of contour anchors used to derive the report.
|
|
|
|
The same song starts the frequency-response path. Each active bioacoustic decode
|
|
carries per-band and per-contour response evidence for the source set, whether
|
|
or not each source earns a timing report. The older chirp-bin calibration model
|
|
remains the controlled ASIO reference surface for response/confusion/delay
|
|
captures:
|
|
use `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` to render that reference
|
|
timeline, optionally capture it through the ASIO worker with `--capture-asio`,
|
|
compute the response/confusion/delay model, and persist it for decoder tuning.
|
|
|
|
`MimirAudioSynchronizationStateTracker` turns continuous observations into
|
|
state. It confidence-gates reports, smooths fractional delay per source, and
|
|
estimates delay slope as sampling-rate offset in ppm. This is the control input
|
|
for the coming actuator. The state can survive a brief weak report, but it is
|
|
not a license to run blind: loopback must keep receiving the emitted timeline or
|
|
fresh reports will stop.
|
|
|
|
The actual Mimir app path now runs this online: `MimirRuntime.Update` keeps the
|
|
bioacoustic timeline queued, polls sources, and updates sync analysis on a fixed cadence.
|
|
`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current
|
|
runtime testing proves Fensalir output wakes the Scarlett loopback and the mic
|
|
buffers stay live. The next failure is physical capture proof: the same
|
|
canonical song-contour anchors need to stay stable through loopback, room mics, device
|
|
clocks, and codec/network paths.
|
|
|
|
## Retained Reference Calibration
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A["MimirChirpletTimeline"] --> B["Fensalir audio output"]
|
|
B --> C["Scarlett speaker loopback"]
|
|
B --> D["room + speakers + mics"]
|
|
C --> E["loopback rolling buffer"]
|
|
D --> F["mic rolling buffers"]
|
|
E --> G["matched reference traces"]
|
|
F --> G
|
|
G --> H["symbol likelihood events"]
|
|
H --> L["timeline anchors"]
|
|
L --> M["per-source clock fit"]
|
|
M --> N["delay + SRO"]
|
|
G --> I["per-source calibration profile"]
|
|
N --> J["fractional delay / resampler actuator"]
|
|
I --> K["frequency response normalization"]
|
|
```
|
|
|
|
The retained chirplet/chirp-bin timelines own three reference facts:
|
|
|
|
- **Emission**: the PCM that Fensalir sends to the speakers.
|
|
- **Timing witness**: the matched-filter atom bank used to find the stream in
|
|
loopback and mic buffers.
|
|
- **Response witness**: the per-band atoms used to measure how strongly each
|
|
mic hears each emitted band.
|
|
|
|
Continuous active song evidence should give both the current delay and the drift/SRO by
|
|
watching delay change over time. Per-band energy over the same stream gives the
|
|
normalization curve. The important constraint is that all three measurements
|
|
must be tied to the same emitted timeline, not three separately invented probes.
|
|
The current ASIO artifact proves why this matters: Scarlett loopback channel 2
|
|
and loopback channel 3 both decode 12 anchors with 0.996 clock confidence, while
|
|
physical input 1 decodes a noisier independent clock and a different strongest
|
|
band set but still fails pairwise timing because no canonical events overlap
|
|
the loopback path. That is not a blank failure anymore; it is calibration data
|
|
for the next codebook/weighting cut.
|
|
|
|
The active symbol layer is now the song contour. It does not rely on one fixed
|
|
frequency shelf or a coded sequence: syllable timing, contour bends, formants,
|
|
payload ornaments, speaker tint, and local spectral flux all contribute so poor
|
|
mic response does not erase the whole code. The old chirplet self-test remains
|
|
available as historical proof of sub-frame math, but it must not retake
|
|
authority over the active runtime path.
|
|
|
|
The older `MimirChirpletTimeline` matcher is a diagnostic/reference artifact,
|
|
not the active runtime path. `BuildChirpletEnergyTrace` still behaves like a
|
|
dense sliding matched-filter bank and should not retake authority.
|
|
|
|
Next, use the in-process ASIO source as the local Scarlett authority, prove
|
|
stable acoustic anchors against real mics, then expose buffer depth, clock
|
|
state, delay estimates, and stem routing in Fensalir UI. The analyzer accepts
|
|
Float32, Int16, Int24, and Int32 PCM windows so direct driver paths can preserve
|
|
real interface formats before Faust/native DSP owns the hot resampling and
|
|
alignment.
|