From bd5b3a4fc2c667c72a7f0f63fdc168182a4de171 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 25 May 2026 00:54:43 +0100 Subject: [PATCH] Map sonar DSP recursive refinement path --- notes/current-system-map.md | 6 +- ...nar-dsp-recursive-refinement-2026-05-25.md | 191 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 6 +- 4 files changed, 202 insertions(+), 2 deletions(-) create mode 100644 research/sonar-dsp-recursive-refinement-2026-05-25.md diff --git a/notes/current-system-map.md b/notes/current-system-map.md index a5a7732..55360e4 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -238,7 +238,11 @@ A naive recursive waveform phase-lock pass was also rejected: it improved one cardioid receipt but worsened the shotgun and locked a fresh cardioid run onto a later reflection lobe. Recursive refinement remains a good architecture only after the path model can distinguish direct arrival, phase/group delay, and room -reflection energy. +reflection energy. The follow-up sonar/DSP research note at +`research/sonar-dsp-recursive-refinement-2026-05-25.md` maps that correction: +use complex matched filters, phase-slope/group-delay fitting, acquisition versus +tracking loop bandwidths, and sparse multipath residuals before attempting +another recursive fitter. The first actuator proof now exists: `faust/mimir_alignment_actuator.dsp` owns six channels of bounded fractional delay/gain controls for Faust/native DSP, and `Mimir.BufferSmoke --bioacoustic-actuator-self-test --sample-rate 48000 diff --git a/research/sonar-dsp-recursive-refinement-2026-05-25.md b/research/sonar-dsp-recursive-refinement-2026-05-25.md new file mode 100644 index 0000000..d673cf9 --- /dev/null +++ b/research/sonar-dsp-recursive-refinement-2026-05-25.md @@ -0,0 +1,191 @@ +# Sonar DSP Notes For Recursive Bioacoustic Refinement + +## Question + +Can Mimir use iterative refinement in the decoder by transforming a sampled song +contour back toward the emitted chirp, resampling, and repeating until phase is +locked? + +Short answer: yes, but not as plain waveform correlation. The failed local +experiment rejected the naive objective function, not recursive refinement. + +## Sources Checked + +- liquid-dsp: open-source C DSP library for SDR-style filters, transforms, + synchronizers, modulators, demodulators, and frame processing. + +- liquid-dsp release notes: `qdsync` detects frames and estimates/corrects + carrier frequency, carrier phase, and timing offset before callback delivery. + +- Underwater acoustic modem paper using liquid-dsp: packet starts are found + with chirp matched filters; later flexframe receivers use PN synchronization, + equalization, PLL, AGC, and acquisition/tracking loop bandwidth changes. + +- pyAPRiL passive radar: explicit reference/surveillance processing, + clutter-cancellation families, frequency-domain/batched cross-correlation, + CFAR, and range-Doppler processing. + +- GNU Radio correlate access code / stream tags: synchronization emits tags at + precise stream positions so downstream blocks process the same sample truth. + + +- OpenTDS / time-delay spectrometry: swept source plus tracking band-pass can + isolate direct sound from lower-frequency delayed reflections, then derive + magnitude and phase response. + +- Signalsmith DSP: high-quality fractional delay/resampling primitives include + Lagrange, polyphase, and Kaiser-sinc interpolation with measured delay error. + +- Open Echo: open-source sonar stack captures raw echo data and emphasizes + transducer/receiver filtering as part of the measurement system. + +- ahoi acoustic modem docs: receiver front-end has preamp plus band-pass stage + adapted to the used frequency band before DSP sees samples. + + +## What The Failed Mimir Pass Got Wrong + +The reverted recursive pass treated the best waveform correlation after +fractional resampling as the truth. In a reflective room, that is not a timing +estimator. It is a reflection selector. + +Observed failure: + +- stored cardioid improved from 7.576 us to 5.141 us; +- stored shotgun worsened from 6.578 us to 10.236 us; +- fresh cardioid locked to a later reflection lobe at 65.666 us instead of the + prior 18.930 us. + +This means the correct recursion variable is not "delay that maximizes raw +correlation." It is a channel hypothesis: + +```text +direct-path delay ++ sample-rate offset / clock drift ++ frequency-dependent phase/group delay ++ complex path gain per contour/formant band ++ sparse reflection taps with later arrival and lower authority +``` + +The direct path owns sync. Reflections are channel evidence, not alternate +timelines. + +## Better Shape + +```text +rolling ASIO window +-> analytic / complex bandbank or short chirp matched-filter bank +-> per-anchor complex response: magnitude, phase, local slope, confidence +-> ambiguity candidates over delay and small frequency/clock offsets +-> direct-path hypothesis tracker +-> sparse multipath residual model +-> recursive refinement: + 1. predict direct-path anchor phases from current hypothesis + 2. dechirp / derotate each contour into baseband + 3. score phase slope and group-delay residuals, not only magnitude + 4. update delay/SRO/group-delay with bounded loop bandwidth + 5. explain later energy as reflection taps + 6. repeat until the direct-path innovation is small +-> output timing anchor, response model, confidence, and residual reflection map +``` + +This is closer to sonar/acoustic modem practice: + +- matched filters acquire the preamble or contour; +- PLL/timing loops refine phase and sample timing; +- equalizers/channel estimators explain multipath; +- CFAR/thresholding prevents noise-floor peaks from becoming detections; +- stream tags carry exact sample-position decisions forward. + +## Concrete Mimir Implementation Cut + +### 1. Replace Raw Waveform Scoring With Complex Matched Filters + +For each shaped contour anchor, build a complex reference template: + +```text +template[n] = envelope[n] * exp(j * phase[n]) +``` + +Maintain streaming I/Q accumulators per anchor family: + +```text +z = sum window[n] * conj(template[n]) +energy = sum abs(window[n])^2 +score = abs(z)^2 / energy +phase = atan2(imag(z), real(z)) +``` + +The magnitude says "this contour exists." The phase and phase slope say "the +direct-path hypothesis is early/late." + +### 2. Estimate Delay From Phase Slope + +Across frequency-separated anchors: + +```text +phase(f) ~= phase0 - 2*pi*f*delay + group_delay_error(f) +``` + +Fit delay from the wrapped phase slope after unwrapping over the usable +anchor set. Weight by learned path response. This is the missing bridge between +"subsample interpolation" and physically meaningful phase lock. + +### 3. Track, Do Not Reacquire Every Window + +Use two loop bandwidths: + +- acquisition: broader ambiguity search, more tolerance, accepts less precise + anchors; +- tracking: narrow predicted window around the existing path, rejects peaks + that require impossible acceleration, clock drift, or propagation jumps. + +This is how recursion becomes stable: each pass refines a live channel state +instead of discovering a new room every frame. + +### 4. Model Multipath Explicitly + +After subtracting the predicted direct-path complex template, keep a small +sparse list of later residual taps: + +```text +tap = delay_after_direct, complex_gain_by_band, decay/confidence +``` + +Those taps are useful for room mapping and frequency-response normalization, but +they cannot vote the direct-path delay backward or forward unless the direct +path is absent. + +### 5. Use Real Fractional Delay Tools + +Linear interpolation is not good enough for the last mile. The actuator and +decoder should use at least Lagrange or polyphase/Kaiser-sinc fractional delay +when comparing/resampling high-rate anchors. Signalsmith's public DSP library is +a useful reference for delay-line quality gates. + +## What To Build Next + +1. A `MimirComplexContourMatchedFilter` over the canary packet anchors. +2. A per-mic `MimirAcousticPathTracker` with: + - direct delay; + - SRO ppm; + - phase intercept; + - per-band complex gain; + - sparse reflection taps. +3. A calibration replay command that runs both stored ASIO receipts and reports: + - direct delay MAE; + - phase-slope residual; + - reflection-tap energy; + - usable anchor bands; + - convergence time; + - realtime factor. +4. Only then reintroduce recursive refinement as a bounded tracker update. + +## Current Belief + +Iterative refinement is absolutely worth pursuing. The correct version is a +complex matched-filter / phase-slope / channel-tracker machine, not a raw +waveform-correlation machine. "Nanosecond" remains a dangerous word in air with +consumer-ish acoustic paths, but the route toward lower microseconds is real: +make phase/group delay first-class and stop letting reflections masquerade as +time. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index afc5864..d19eb1d 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -99,3 +99,4 @@ {"ts": "2026-05-25T00:08:44+01:00", "type": "experiment", "status": "partial", "note": "Built the first anchor-rich canary packet: timing chips, formant pivots, harmonic-envelope notches, payload ornaments, renderer-level template caching, and per-event intra-call anchor measurements in the physical calibration receipt. Fresh 192 kHz ASIO receipt calibration/bioacoustic/scarlett-canary-packet-anchor-rich-192k.json clears 10x realtime and records hundreds of anchors per channel. Loopback improves to about 2.23 us MAE with anchor residuals under 0.3 samples. Physical mics did not improve yet: shotgun 36/37 payload=0.973 at 56.365 us MAE, cardioid 27/37 payload=0.730 at 101.522 us MAE. The new anchors expose path bias; the next cut should shape/weight anchors for acoustic survivability instead of trusting every formant feature equally."} {"ts": "2026-05-25T00:42:12+01:00", "type": "experiment", "status": "partial", "note": "Added path-level loopback truth to the anchor-rich calibration: matched mic anchors are compared directly against loopback anchors, then event-local waveform correlation against the captured loopback packet chooses the path delay. Release calibration clears the realtime budget at 21.1x on the stored anchor-rich capture. Best stored-result path precision improved to cardioid 7.576 us and shotgun 6.578 us. A fresh room capture at calibration/bioacoustic/scarlett-canary-packet-anchor-rich-latest-192k.json keeps shotgun near 5.916 us but worsens cardioid to 18.930 us. The razor timing-chip emitter mutation was rejected because it reduced physical reliability. Current floor is single-digit microseconds for the shotgun, not yet the one-microsecond target; next required cut is phase/group-delay correction or a direct-path anchor family that survives both mics."} {"ts": "2026-05-25T01:05:00+01:00", "type": "experiment", "status": "rejected", "note": "Tested a recursive event-local waveform delay fitter: start from the current path hypothesis, resample candidate song windows against loopback, shrink the residual search radius, and weight events by phase-lock consistency. The naive time-domain version improved one stored cardioid path from 7.576 us to 5.141 us, but it also worsened the stored shotgun from 6.578 us to 10.236 us and locked the fresh cardioid onto a later reflection lobe at 65.666 us instead of the prior 18.930 us. Reverted the code and receipts. Lesson: recursive refinement is a valid shape only when constrained by a real phase/group-delay/path model; plain waveform resampling is not sufficient and nanosecond claims are physically dishonest in the current acoustic path."} +{"ts": "2026-05-25T01:25:00+01:00", "type": "research", "status": "ok", "note": "Looked at open realtime DSP/sonar-adjacent systems after the recursive-fit rejection. liquid-dsp, an underwater acoustic modem using liquid-dsp, pyAPRiL passive radar, GNU Radio stream/tag synchronization, OpenTDS, Signalsmith DSP, Open Echo, and ahoi all point the same way: acquisition should use matched filters/ambiguity search; refinement should track complex phase, timing, SRO, and channel response; multipath should be modeled as later residual taps; and exact stream positions should be propagated as tags/state. Added research/sonar-dsp-recursive-refinement-2026-05-25.md. The next implementation cut should be a complex contour matched-filter bank plus direct-path acoustic tracker, not another raw waveform recursion pass."} diff --git a/state/map.yaml b/state/map.yaml index 37109a1..3c456bb 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -212,7 +212,11 @@ accepted_design: improve one cardioid receipt while worsening the shotgun and locking a fresh cardioid run onto a later reflection lobe. Recursion is only allowed back as a constrained direct-path/group-delay model, not as unconstrained waveform - resampling with prettier decimals. + resampling with prettier decimals. The sonar/DSP research note + `research/sonar-dsp-recursive-refinement-2026-05-25.md` points the next + decoder cut at complex contour matched filters, phase-slope/group-delay + fitting, acquisition/tracking loop bandwidths, and sparse multipath + residuals. Acoustic robustness remains open. The old arbitrary-chirplet matcher is a diagnostic/reference artifact only, not the runtime hot path.