Mimir/docs/audio-field.md
2026-05-22 19:50:02 +01:00

202 lines
10 KiB
Markdown

# Audio Field
Mimir's audio field is six microphones, loopback/program reference, and two
calibration speakers aligned into one presentation timeline.
## Live Target
```mermaid
flowchart TD
A["mic / loopback capture drivers"] --> B["Mimir.Runtime audio buffers"]
C["speaker probe scheduler"] --> B
B --> D["native alignment + phase state"]
D --> E["Faust/native DSP"]
E --> F["host voice"]
E --> G["co-streamer voice"]
E --> H["ambient / transients"]
E --> I["loopback stems"]
E --> J["spatial bed"]
F --> K["OBS"]
G --> K
H --> K
I --> K
J --> K
```
## Invariants
- Scarlett speaker loopback is the timing authority when calibration chirplets
are playing.
- Focusrite dialogue mics are the voice anchors.
- Camera mics are spatial/context witnesses.
- Loopback/program audio is timing evidence where available; it outranks
acoustic mics for clock/timing because it is the emitted program surface.
- Distributed inputs must be aligned and resampled before they become program
stems.
- The five-second runtime window is allowed to be spent on alignment,
resampling, separation, and spatial-field extraction. Low latency loses to a
coherent volumetric sound field here.
- Probe signals are budgeted telemetry, not a permanent audio bed.
- Faust/native DSP owns the hot separation and spatialization graph.
## Synchronization Modes
`MimirAudioSynchronizationSettings.Mode` is the runtime authority for what
timing evidence Mimir is allowed to emit:
- `chirp-only`: emit the deterministic calibration timeline and decode timing
only from that active witness. This is the lab/debug mode and the fallback for
silent program material.
- `passive`: do not emit calibration audio. Use loopback/program audio as the
timing witness by estimating delay between the loopback buffer and each mic
buffer.
- `hybrid`: prefer passive program-audio evidence when confidence is high, then
emit a watermark when confidence falls. The passive side uses bounded
GCC-PHAT-style phase correlation. The active fallback now uses
`MimirChirpBinTimeline`: a fixed-slope chirp-bin watermark whose symbols are
decoded by one dechirp and a fixed Goertzel bin bank instead of dense sliding
matched filters.
The mode belongs to the runtime, not the decoder. The decoder should consume
known timing evidence; it should not decide whether Mimir is allowed to make
sound.
The passive estimator is the first real program-audio path, not the final DSP
actuator. It removes DC, pre-emphasizes the window, applies a Hann taper, runs a
PHAT-weighted cross spectrum, and reports the strongest loopback-to-mic lag
inside a bounded window. Positive delay still means the candidate mic is late
relative to loopback. Negative lags are treated as contradictory passive
evidence and carry zero confidence. A single passive window is also capped below
certainty; full confidence belongs to repeated coherent state over time, not
one attractive correlation peak.
The active watermark path is deliberately receiver-cheap. `MimirChirpBinTimeline`
uses a common chirp duration, chirp slope, and Hann window for every symbol.
Symbol identity is the dechirped frequency bin. The synthetic invariant is
`dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test`;
it renders the chirp-bin timeline, decodes by dechirp plus Goertzel bins, and
requires code-valid triplet anchors plus a stable source clock.
## Next Cut
The current diagnostic witness is `native/probes/wasapi_audio_cadence`, which
emits timestamped WASAPI `audio-block` metadata into `Mimir.Runtime` through the
frame-event adapter. It has proven Focusrite mic, Kiyo Pro mic, Kiyo mic, both
USB Camera / PS3 Eye mics, and Scarlett speaker loopback in rolling buffers when
loopback audio is actively playing. One PS3 Eye mic previously enumerated but
produced zero WASAPI packets until that Eye was unplugged and replugged.
The full probe runtime config now enables sample-bearing blocks for every local
audio source. `MimirChirpletTimeline` owns the emitted calibration stream and
the matched-filter shape used to analyze it. The default timeline is a
deterministic order-3 de Bruijn symbol sequence over 32 chirp symbols. Any three
consecutive correctly detected symbols identify the event index inside the
current operating horizon, so a receiver can place its audio window on the
canonical timeline without being handed Mimir's runtime clock. Mimir queues
half-second PCM segments ahead of the audio cursor. Each symbol is a small
time/frequency constellation: start band, glide direction/range, duration, and
the following inter-chirp gap all carry code. The point is not ornament. The
timing code is carried by both frequency and rhythm, so it behaves more like a
quiet birdsong texture than a repeated sweep.
The intended decoder is a constrained chirplet transform, not a generic
time-frequency explorer and not an outlier filter around bad guesses. Mimir owns
the emitter, so the receiver projects each mic stream against the known chirplet
dictionary, produces transform frames with multiple symbol candidates, and
scores candidate triplets through the de Bruijn map. A triplet only becomes a
canonical timeline anchor when its symbol likelihoods, measured inter-chirp
gaps, and neighboring anchors agree on one local sample clock. A decoded anchor
means observed sample offset `S` corresponds to emitted event time `T`. A stream
of anchors fits the source clock directly:
```text
observed_sample = source_offset + canonical_seconds * effective_sample_rate
```
Delay and sampling-rate offset are derived by comparing the loopback clock fit
to each mic clock fit over common canonical time. The state tracker is not
allowed to launder invalid codewords into plausible timing. If a stream cannot
produce at least three matched canonical anchors, it has not decoded timing for
that window. Reports carry rounded integer delay, fractional delay, and the
count/confidence of timeline-symbol anchors used to derive the report.
The same timeline also starts the frequency-response path. Each report includes
per-band matched energy for the chirplet atoms. That is not a finished room/mic
normalizer yet, but it is the live surface that will become response-curve
estimation: loopback carries what was emitted, each mic carries what survived
speaker, air, room, and capsule, and the ratio over the continuous chirplet
timeline becomes gain/phase correction evidence.
`MimirAudioSynchronizationStateTracker` turns continuous observations into
state. It confidence-gates reports, smooths fractional delay per source, and
estimates delay slope as sampling-rate offset in ppm. This is the control input
for the coming actuator. The state can survive a brief weak report, but it is
not a license to run blind: loopback must keep receiving the emitted timeline or
fresh reports will stop.
The actual Mimir app path now runs this online: `MimirRuntime.Update` keeps the
chirplet timeline queued, polls sources, and updates sync analysis on a fixed cadence.
`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current
runtime testing proves Aquarium output wakes the Scarlett loopback and the mic
buffers stay live. The decoder now uses quadrature chirplet atoms so symbol
classification is phase-invariant at the transform layer. The next failure is
physical capture proof: the same canonical anchors need to stay stable through
loopback, room mics, device clocks, and codec/network paths.
## Chirplet Calibration Model
```mermaid
flowchart TD
A["MimirChirpletTimeline"] --> B["Aquarium audio output"]
B --> C["Scarlett speaker loopback"]
B --> D["room + speakers + mics"]
C --> E["loopback rolling buffer"]
D --> F["mic rolling buffers"]
E --> G["matched chirplet traces"]
F --> G
G --> H["symbol likelihood events"]
H --> L["triplet timeline anchors"]
L --> M["per-source clock fit"]
M --> N["delay + SRO"]
G --> I["per-band response estimates"]
N --> J["fractional delay / resampler actuator"]
I --> K["frequency response normalization"]
```
The chirplet timeline owns three facts:
- **Emission**: the PCM that Aquarium sends to the speakers.
- **Timing witness**: the matched-filter atom bank used to find the stream in
loopback and mic buffers.
- **Response witness**: the per-band atoms used to measure how strongly each
mic hears each emitted band.
Continuous chirplet evidence gives both the current delay and the drift/SRO by
watching delay change over time. Per-band energy over the same stream gives the
normalization curve. The important constraint is that all three measurements
must be tied to the same emitted timeline, not three separately invented probes.
The symbol layer is intentionally redundant. It does not rely on one fixed
frequency shelf: timing gaps, chirp duration, start band, and glide shape all
contribute so poor mic frequency response does not erase the whole code.
`MimirChirpletSymbolCodebook` owns the symbol definitions so timeline ordering
does not smuggle acoustic shape decisions into bit arithmetic. Every symbol has
a unique chirp shape; rhythm remains additional evidence, not a substitute for
symbol separability. The transform uses sine/cosine chirplet kernels for each
symbol, so timing does not depend on the receiver preserving the emitter's
absolute phase. Run `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`
to render two seconds of canonical timeline audio and decode it back to
timeline anchors. Current synthetic proof detects all 15 emitted chirps, keeps
13 possible triplet anchors for events 0 through 12, fits the clock at
47999.999990 Hz, and holds mean absolute anchor error to 0.000014 samples. Real
device runs still depend on loopback capture staying live; the local Scarlett
loopback has intermittently stopped advancing during short headless sniffs.
The older `MimirChirpletTimeline` matcher is now the chirp-only lab path, not
the hybrid fallback. `BuildChirpletEnergyTrace` still behaves like a dense
sliding matched-filter bank and should not retake the hot path. The active
hybrid watermark belongs to the chirp-bin machine.
Next, replace the diagnostic bridge with native audio capture workers that
append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state,
delay estimates, and stem routing in Aquarium UI.