From 5cf9ba516e623062f15a098364c7f5809235d8b3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 24 May 2026 06:13:45 +0100 Subject: [PATCH] Add bioacoustic sync watermark --- docs/audio-field.md | 120 +-- docs/bioacoustic-timeline-watermark.md | 116 +++ docs/code-algorithm-map.md | 78 +- docs/implementation-plan.md | 76 +- docs/perfect-machine-domain-index.md | 5 +- notes/current-system-map.md | 83 +- notes/fresh-workspace-handoff.md | 62 +- src/Mimir.BufferSmoke/Program.cs | 157 +++- src/Mimir.Runtime/MimirRuntime.cs | 29 +- .../MimirAudioSynchronizationAnalyzer.cs | 99 +- .../MimirBioacousticTimeline.cs | 844 ++++++++++++++++++ state/map.yaml | 97 +- 12 files changed, 1466 insertions(+), 300 deletions(-) create mode 100644 docs/bioacoustic-timeline-watermark.md create mode 100644 src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs diff --git a/docs/audio-field.md b/docs/audio-field.md index 5767d2f..dbdbf25 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -68,34 +68,41 @@ certainty; full confidence belongs to repeated coherent state over time, not one attractive correlation peak. The active calibration path is deliberately receiver-cheap in both `chirp-only` -and `hybrid`. `MimirChirpBinTimeline` uses a common chirp duration, chirp slope, -and window for every symbol. Acquisition is a cheap bounded energy/onset pass; -classification is dechirp plus a fixed Goertzel bin bank; timeline placement is -the de Bruijn triplet trellis. A constrained local waveform correlation around -the decoded anchor delay provides the final fractional offset. The decoder keeps -the full bin-energy surface for each classified chirp and aggregates it into -per-band response evidence, so the same stream can calibrate timing and the -speaker/room/mic transfer function. `MimirChirpBinCalibrationModel` is the live -surface for that evidence. It stores measured usable bands per source, -expected-symbol versus observed-bin confusion observations, timing residuals, -delay hypotheses, phase summaries, and an adaptive codebook plan per -output/mic path. The raw profile is kept even when a timing report is rejected, -because a failed sync window can still teach us which bands survived the -speaker/room/mic path. The decoder can consume the persisted model to weight -reliable symbols, downweight dead bands, apply phase-coherence weighting, -apply first-order group-delay correction, and search joint global delay/bin-shift -hypotheses before selecting the coherent anchor path. The model also owns the -active emission plan: if the measured path only leaves a smaller alphabet, Mimir -emits that reliable symbol set and raises the de Bruijn order so the timeline -remains unique without pretending the dead bins still carry code. +and `hybrid`. `MimirBioacousticTimeline` is the live runtime witness: it emits +low-gain log-frequency motifs made from short formant-rich syllables, rhythm +offsets, and a de Bruijn triplet grammar. Acquisition is an energy proposal pass +plus bounded motif matching; timeline placement is the same code-valid triplet +trellis, and a constrained local waveform correlation around the decoded delay +provides the final fractional offset. The decoder keeps per-band response +evidence for each detected motif so the same stream can calibrate timing and the +speaker/room/mic transfer function. + +`MimirChirpBinTimeline` remains as the older controlled calibration/reference +artifact. Its full bin-energy surface and `MimirChirpBinCalibrationModel` still +matter for measured usable bands, expected-symbol versus observed-bin confusion, +timing residuals, delay hypotheses, phase summaries, and adaptive codebook +plans. That evidence is allowed to shape the bioacoustic receiver next; it is +not the active runtime sound. + +The raw profile is kept even when a timing report is rejected, because a failed +sync window can still teach us which bands survived the speaker/room/mic path. +The receiver must consume persisted models to weight reliable symbols, +downweight dead bands, apply phase-coherence weighting, apply first-order +group-delay correction, and search joint global delay/frequency-shift hypotheses +before selecting the coherent anchor path. If the measured path only leaves a +smaller alphabet, Mimir should use that reliable symbol set and raise the +sequence order so the timeline remains unique without pretending dead bands +still carry code. The synthetic invariant is -`dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test`; -it renders the chirp-bin timeline, decodes by dechirp plus Goertzel bins, and -requires code-valid triplet anchors plus a stable source clock. `chirp-only` -emits this witness continuously; `hybrid` emits one low-gain half-second coded -burst every two seconds only while passive confidence is weak. Default hybrid -watermark gain is intentionally low (`watermarkGain`, or `MIMIR_WATERMARK_GAIN`) -and separate from the louder chirp-only lab gain. +`dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`; +it renders the bioacoustic timeline, decodes motif anchors, and requires +code-valid triplets plus a stable source clock. The standalone invariant is +`--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5`; +it proves that a delayed receiver can recover canonical source time from the +known codebook and schedule alone. `chirp-only` emits this witness continuously; +`hybrid` still prefers passive program-audio evidence, then emits the low-gain +bioacoustic witness only when passive confidence is weak. Default hybrid +watermark gain is intentionally low (`watermarkGain`, or `MIMIR_WATERMARK_GAIN`). Research notes: @@ -148,35 +155,34 @@ current studio-monitor-to-Scarlett-mic acoustic path was already weak by 14-16 kHz and very weak above 20 kHz. Treat ultrasonic acoustic sync as unproven until a measured transducer/mic path earns it. -The synthetic chirp-bin timing proof now runs at arbitrary sample rates with -`--hybrid-sync-self-test --sample-rate N`. On the same physical delay, measured -in memory, the active chirp-bin path recovered about 0.369 us error at 48 kHz, -0.168 us at 96 kHz, and below printed microsecond precision at 192 kHz. That -proves the timing kernel benefits from the Scarlett ASIO rate. The live ingest +The active bioacoustic timing proof now runs in memory through the same runtime +analyzer path. `--bioacoustic-self-test` proves code-valid motif anchors, +`--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5` +proves a receiver can recover canonical source time from delayed audio alone, +and `--chirp-only-sync-self-test --sample-rate 48000` recovers a 317.375-sample +delay with printed 0.000 us error using `evidence=bioacoustic`. The live ingest proof now feeds ASIO callbacks into `Mimir.Runtime` without a process/stdout bridge: a two-second BufferSmoke run at 192 kHz ingested more than 12,000 sample-bearing blocks and retained 2,048 blocks per channel across `asio-ch0` through `asio-ch3`. The full probe runtime config now enables sample-bearing blocks for every local -audio source. `MimirChirpletTimeline` owns the emitted calibration stream and -the matched-filter shape used to analyze it. The default timeline is a -deterministic order-3 de Bruijn symbol sequence over 32 chirp symbols. Any three -consecutive correctly detected symbols identify the event index inside the -current operating horizon, so a receiver can place its audio window on the +audio source. `MimirBioacousticTimeline` owns the emitted calibration stream and +the motif-matching shape used to analyze it. The default timeline is a +deterministic order-3 de Bruijn symbol sequence over 16 log-frequency motifs. +Any three consecutive correctly detected motifs identify the event index inside +the current operating horizon, so a receiver can place its audio window on the canonical timeline without being handed Mimir's runtime clock. Mimir queues -half-second PCM segments ahead of the audio cursor. Each symbol is a small -time/frequency constellation: start band, glide direction/range, duration, and -the following inter-chirp gap all carry code. The point is not ornament. The -timing code is carried by both frequency and rhythm, so it behaves more like a -quiet birdsong texture than a repeated sweep. +half-second PCM segments ahead of the audio cursor. Each motif is a short phrase +of formant-rich syllables with log-frequency contour and rhythm offsets, so the +code is carried by both spectral shape and timing. -The intended decoder is a constrained chirplet transform, not a generic +The intended decoder is a compact constrained motif transform, not a generic time-frequency explorer and not an outlier filter around bad guesses. Mimir owns -the emitter, so the receiver projects each mic stream against the known chirplet +the emitter, so the receiver projects each mic stream against the known motif dictionary, produces transform frames with multiple symbol candidates, and scores candidate triplets through the de Bruijn map. A triplet only becomes a -canonical timeline anchor when its symbol likelihoods, measured inter-chirp +canonical timeline anchor when its symbol likelihoods, measured inter-motif gaps, and neighboring anchors agree on one local sample clock. A decoded anchor means observed sample offset `S` corresponds to emitted event time `T`. A stream of anchors fits the source clock directly: @@ -192,16 +198,13 @@ produce at least three matched canonical anchors, it has not decoded timing for that window. Reports carry rounded integer delay, fractional delay, and the count/confidence of timeline-symbol anchors used to derive the report. -The same timeline also starts the frequency-response path. Each active chirp-bin -decode can emit a persisted calibration model for the source set, whether or -not each source earns a timing report. That model is not a finished room/mic -normalizer yet, but it is the live surface that will become response-curve -estimation: loopback carries what was emitted, each mic carries what survived -speaker, air, room, and capsule, and the ratio over the continuous timeline -becomes gain/phase correction evidence. Use -`Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` to render the calibration +The same timeline starts the frequency-response path. Each active bioacoustic +decode carries per-band response evidence for the source set, whether or not +each source earns a timing report. The older chirp-bin calibration model remains +the controlled ASIO reference surface for response/confusion/delay captures: +use `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` to render that reference timeline, optionally capture it through the ASIO worker with `--capture-asio`, -compute the response/confusion/delay model, and persist it for runtime decode. +compute the response/confusion/delay model, and persist it for decoder tuning. `MimirAudioSynchronizationStateTracker` turns continuous observations into state. It confidence-gates reports, smooths fractional delay per source, and @@ -211,13 +214,12 @@ not a license to run blind: loopback must keep receiving the emitted timeline or fresh reports will stop. The actual Mimir app path now runs this online: `MimirRuntime.Update` keeps the -chirplet timeline queued, polls sources, and updates sync analysis on a fixed cadence. +bioacoustic timeline queued, polls sources, and updates sync analysis on a fixed cadence. `MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current runtime testing proves Fensalir output wakes the Scarlett loopback and the mic -buffers stay live. The decoder now uses quadrature chirplet atoms so symbol -classification is phase-invariant at the transform layer. The next failure is -physical capture proof: the same canonical anchors need to stay stable through -loopback, room mics, device clocks, and codec/network paths. +buffers stay live. The next failure is physical capture proof: the same +canonical motif anchors need to stay stable through loopback, room mics, device +clocks, and codec/network paths. ## Chirplet Calibration Model diff --git a/docs/bioacoustic-timeline-watermark.md b/docs/bioacoustic-timeline-watermark.md new file mode 100644 index 0000000..5d5c91e --- /dev/null +++ b/docs/bioacoustic-timeline-watermark.md @@ -0,0 +1,116 @@ +# Bioacoustic Timeline Watermark + +## Purpose + +The active timing signal is no longer supposed to be a row of tidy lab beeps. +That shape proved the math, but it is the wrong animal for a room. + +Mimir now treats active acoustic sync as a low-gain bioacoustic language: short +birdsong-like motifs whose identity survives through contour, log-frequency +position, spectral envelope, rhythm, and phrase order. The receiver should not +ask only "which bin fired?" It should ask: + +```text +which motif contour did I hear? +where does this motif phrase sit in the canonical grammar? +what delay and clock fit make the whole phrase coherent? +what did this room/mic path do to the spectrum? +``` + +## Current Machine + +`MimirBioacousticTimeline` is the current active runtime watermark. + +- 16 log-spaced motif symbols. +- Each motif contains three short syllables. +- Each syllable carries a log-frequency glide plus formant-like partials. +- Motif rhythm varies by symbol group. +- The timeline is an order-3 de Bruijn grammar, so any three correctly decoded + consecutive motifs identify canonical event position. +- The decoder emits motif candidates, code-valid triplet anchors, a source + clock fit, and per-band response evidence. +- The runtime active sync evidence kind is `bioacoustic`. + +The old `MimirChirpBinTimeline` remains as a calibration/reference artifact +with ASIO capture tooling. It is useful for comparing acoustic paths and +existing calibration captures, but it is no longer the expressive runtime +watermark target. + +## Ownership + +`MimirBioacousticTimeline` owns: + +- motif codebook shape; +- deterministic phrase schedule; +- PCM rendering for active emission; +- streaming motif scoring; +- de Bruijn triplet anchor decoding; +- source clock fitting; +- response-band evidence. + +`MimirAudioSynchronizationAnalyzer` owns: + +- comparing loopback/reference and candidate receiver decodes; +- choosing passive evidence before active evidence in `hybrid`; +- reporting bioacoustic delay when active evidence is used; +- fractional delay refinement from waveform evidence. + +Fensalir audio only emits the rendered watermark. It does not interpret it. +Faust/native DSP will eventually own the fractional-delay/SRO actuator. + +## Why This Shape + +Birdsong survives ugly channels because the information is redundant: + +- contour survives when absolute pitch shifts; +- rhythm survives when frequency response is bad; +- formant envelope survives when one band dies; +- phrase grammar survives when individual syllables are weak; +- repeated local structure gives the receiver more than one way to recover time. + +That is the physical path Mimir needs. A phone mic, camera mic, Scarlett input, +or network receiver should be able to recognize canonical time from a damaged +short phrase, not from a perfect isolated tone. + +## Proofs + +Synthetic checks: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5 +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 48000 +``` + +Expected state: + +- `--bioacoustic-self-test` recovers code-valid motif anchors across the + canonical phrase. +- `--standalone-bioacoustic-self-test` recovers delayed canonical source time + from codebook/schedule state alone. +- `--chirp-only-sync-self-test` reports `evidence=bioacoustic` and recovers + fractional delay below printed microsecond precision. + +Render a raw Float32 preview: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --render-bioacoustic-f32 --output artifacts/asio/bioacoustic-f32.raw --seconds 8 +``` + +## Next Cut + +The current decoder is still a motif matched-filter proof. The next coherent +cut is a true streaming log-mel receiver: + +```text +audio window +-> log-mel / constant-Q evidence surface +-> motif/formant/contour candidates +-> phrase grammar decode +-> global delay and clock hypothesis +-> path response model +``` + +Do not add a pile of threshold rules to make one room capture look good. The +right extension is learned path weighting over motif parts: which contours, +partials, and time gaps survive each output/mic path. diff --git a/docs/code-algorithm-map.md b/docs/code-algorithm-map.md index 14c591c..b2b684f 100644 --- a/docs/code-algorithm-map.md +++ b/docs/code-algorithm-map.md @@ -31,13 +31,13 @@ buffers, resamples candidates to the reference rate, and chooses evidence by mode: - `passive`: PHAT-weighted cross-spectrum correlation on program audio. -- `chirp-only`: active chirp-bin timeline decode. -- `hybrid`: passive first, chirp-bin watermark when passive confidence is weak. +- `chirp-only`: active bioacoustic motif timeline decode. +- `hybrid`: passive first, bioacoustic watermark when passive confidence is weak. -Active chirp-bin decode uses known codebook/schedule state. It proposes event -times from local energy, classifies symbols by dechirp plus fixed Goertzel/bin -scoring, constrains candidates through de Bruijn code-valid triplets, fits a -clock, and records band/confusion/delay/phase evidence for calibration. +Active bioacoustic decode uses known codebook/schedule state. It proposes motif +times from local energy, classifies log-frequency/formant symbols by bounded +matched motif scoring, constrains candidates through de Bruijn code-valid +triplets, fits a clock, and records band response evidence for calibration. ## Invariants @@ -67,10 +67,11 @@ flowchart TD Sources["IMimirStreamSource implementations"] --> Hub Hub --> Buffers["MimirStreamBufferSet"] Buffers --> Rolling["MimirRollingStreamBuffer per stream"] - Runtime --> AudioEmit["chirp-bin/chirplet witness emission"] + Runtime --> AudioEmit["bioacoustic witness emission"] Runtime --> Analyzer["MimirAudioSynchronizationAnalyzer"] Analyzer --> Passive["MimirPassiveAudioSynchronizationEstimator"] - Analyzer --> ChirpBin["MimirChirpBinTimeline"] + Analyzer --> Bioacoustic["MimirBioacousticTimeline"] + Analyzer --> ChirpBin["MimirChirpBinTimeline reference path"] ChirpBin --> Calibration["MimirChirpBinCalibrationModel"] Analyzer --> State["MimirAudioSynchronizationStateTracker"] NativeAsio["native/asio_capture"] --> AsioSource["MimirAsioStreamSource"] @@ -240,7 +241,8 @@ Blocks: - `DescribeChirpletReference`: explains current active/passive emission state. - `RenderCalibrationBatchPcm16Base64`: renders chirp-bin or legacy chirplet timeline segments as mono PCM16 and base64-encodes them for Fensalir audio. -- `UseChirpBinTimeline`: routes chirp-only and hybrid to the chirp-bin machine. +- Active witness routing: chirp-only and hybrid emission now use the + bioacoustic motif timeline. - `UpdateAudioSynchronization`: runs one bounded analyzer step on cadence and stores cached reports. - `EmitTelemetry`: prints buffer, report, state, and decode-trace lines when @@ -769,7 +771,7 @@ Blocks: - Constants: active/passive window floors, maximum clock-fit delay horizon, passive confidence floor. - `Analyze`: finds reference/candidate audio buffers, extracts mono windows, - aligns common edge, resamples to reference rate, chooses passive/chirp-bin/ + aligns common edge, resamples to reference rate, chooses passive/bioacoustic/ legacy chirplet mode, stores calibration profiles, estimates delay from decoded anchors or passive correlation, refines active delay with local waveform correlation, and returns accepted reports. @@ -798,13 +800,51 @@ Invariant: - A source without enough accepted evidence does not get a fake delay. Failed timing can still leave useful calibration evidence. -## Stage 8: Chirp-Bin Active Timeline +## Stage 8: Bioacoustic Active Timeline + +### `MimirBioacousticTimeline.cs` + +Owns: + +- The active runtime watermark and receiver. + +Core state: + +- Sixteen log-spaced motif symbols. +- Three short formant-rich syllables per motif. +- A de Bruijn order-three schedule, so any valid motif triplet identifies one + canonical timeline location. +- Cached kernels by sample rate. + +Blocks: + +- Motif/syllable records: `MimirBioacousticMotifDefinition` and + `MimirBioacousticSyllable`. +- `RenderSegmentMonoFloat`: renders one low-gain timeline segment. +- `DecodeStreamWindow`: detects motif frames, decodes code-valid triplet + anchors, fits source clock, refines standalone source offset, and emits the + shared `MimirChirpletStreamDecode` shape. +- `EventForIndex` / `EventsOverlapping`: expose canonical schedule metadata. +- `DetectFrames`: proposes energy/onset windows, adds dense fallback probes, and + suppresses nearby duplicate motif hits. +- `ClassifyAt`: scores a local candidate against the motif bank. +- `ScoreMotif`: matched motif scoring with per-band response evidence. +- `DecodeAnchors`: maps detected motif triplets back to canonical event indexes. +- `FitClock` and `RefineSourceOffset`: recover source timeline position and + sub-frame offset from known codebook/schedule state. + +Invariant: + +- A receiver with only the codebook, schedule, and its local audio buffer can + recover canonical time once it hears a code-valid motif triplet. + +## Stage 9: Chirp-Bin Reference Timeline ### `MimirChirpBinTimeline.cs` Owns: -- The current active timing/calibration codebook and receiver. +- The controlled chirp-bin calibration/reference codebook and receiver. Core state: @@ -938,7 +978,7 @@ Blocks: Status: -- Diagnostic/reference. The chirp-bin path is the hot active receiver. +- Diagnostic/reference. The bioacoustic motif path is the hot active receiver. ### `MimirChirpletTimeline.cs` @@ -1030,12 +1070,18 @@ Function blocks: - `RunChirpBinSelfTest`: chirp-bin codebook/schedule synthetic proof. - `RunStandaloneChirpBinSelfTest`: delayed receiver proof with only codebook and schedule state. +- `RunBioacousticSelfTest`: bioacoustic motif codebook/schedule synthetic + proof. +- `RunStandaloneBioacousticSelfTest`: delayed receiver proof for the active + bioacoustic witness. - `RunPassiveSyncSelfTest`: PHAT passive proof. - `RunHybridSyncSelfTest`: hybrid mode proof. - `RunActiveSyncSelfTest`: chirp-only/hybrid rolling-buffer analyzer proof. - `SyncModeLabel`: CLI label helper. - `RenderChirpletFloat32`: writes legacy timeline Float32 samples. - `RenderChirpBinFloat32`: writes chirp-bin timeline Float32 samples. +- `RenderBioacousticFloat32`: writes active bioacoustic timeline Float32 + samples. - `LoadOptionalCalibration`: loads model from CLI path. - `AnalyzeAsioFloat32`: reads interleaved ASIO artifact and feeds runtime analyzer. @@ -1432,9 +1478,9 @@ Linked notes: Hypotheses: -- Treat chirp-bin active sync like a controlled acoustic CSS modem: dechirp to - collapse a chirp into a narrow spectral decision, then let code constraints - and clock-fit consistency carry identity. +- Treat bioacoustic active sync like a compact acoustic language: log-frequency + motif identity, formant structure, rhythm, code constraints, and clock-fit + consistency carry timeline identity together. - Keep passive sync as a separate evidence source. GCC-PHAT-like correlation is useful for program audio and drift hints, but it should not mint codebook anchors. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 15bf77e..4af23d7 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -44,11 +44,11 @@ a named invariant that the native runtime cannot protect yet. rolling buffers on one interface clock domain. - `src/Mimir.BufferSmoke` loads the runtime config, polls the synchronization hub, and prints the actual rolling buffers. Use `--require-samples` when an - empty declared sensor buffer should fail the run. Use `--chirplet-self-test` - to render the canonical timeline into memory and verify that the constrained - decoder recovers sub-frame anchors without hardware. Use - `--standalone-chirp-bin-self-test` to verify that a receiver with only the - codebook/schedule can recover canonical source offset from delayed audio. + empty declared sensor buffer should fail the run. Use `--bioacoustic-self-test` + to render the active motif timeline into memory and verify that the decoder + recovers code-valid anchors. Use `--standalone-bioacoustic-self-test` to + verify that a receiver with only the codebook/schedule can recover canonical + source offset from delayed audio. - `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback block metadata and emits `audio-block` JSON events for the diagnostic runtime adapter. It can probe requested shared/exclusive formats so driver state is @@ -63,13 +63,15 @@ a named invariant that the native runtime cannot protect yet. response per frequency so ultrasonic acoustic claims stay measured. The probe can also play raw mono Float32 timeline audio with `--play-f32-mono` and capture raw interleaved Float32 ASIO input with `--record-f32-interleaved`. -- `MimirChirpletTimeline` owns the structured birdsong-like calibration stream, +- `MimirBioacousticTimeline` owns the active runtime watermark described in + [[bioacoustic-timeline-watermark|Bioacoustic Timeline Watermark]]. It renders + a low-gain birdsong-like phrase language: 16 log-spaced motif symbols, three + formant-rich syllables per motif, rhythm variation, and an order-3 de Bruijn + grammar so any three correctly decoded consecutive motifs identify canonical + timeline position. Runtime active sync now reports `evidence=bioacoustic`. +- `MimirChirpletTimeline` owns the older structured chirplet calibration stream, PCM segment rendering, matched timing trace, and per-band response kernels. - The default timeline is an order-3 de Bruijn symbol sequence over 32 - time/frequency constellation symbols, so any three consecutive correctly - detected symbols identify a timeline event inside the current operating - horizon. Symbol identity is carried by start band, glide shape, duration, and - following inter-chirp gap. + It remains a reference/diagnostic path. - `MimirChirpletSymbolCodebook` owns the 32 symbol definitions. Each symbol has a unique chirp shape, with inter-chirp rhythm as additional code evidence. - `MimirChirpletStreamDecoder` is the first constrained chirplet-transform @@ -90,7 +92,7 @@ a named invariant that the native runtime cannot protect yet. service and can emit live sync telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry read cached reports/states; they do not run synchronization analysis. -- `MimirRuntime` queues chirplet timeline PCM through Fensalir audio when the +- `MimirRuntime` queues bioacoustic timeline PCM through Fensalir audio when the active timing witness is allowed. `MimirAudioSynchronizationSettings.Mode` selects `chirp-only`, `passive`, or `hybrid`; passive disables active emission, chirp-only emits the active witness continuously, and hybrid emits @@ -99,10 +101,10 @@ a named invariant that the native runtime cannot protect yet. path. It estimates loopback-to-mic delay with PHAT-weighted cross-spectrum correlation so music can act as the default timing witness before any audible watermark is needed. -- `MimirChirpBinTimeline` is the active calibration path for both `chirp-only` - and `hybrid`. It renders a fixed-slope chirp-bin codebook and decodes symbols - with cheap event-energy proposals, dechirp plus fixed Goertzel bins, and the - same de Bruijn triplet timeline-anchor machine. The detector keeps +- `MimirChirpBinTimeline` is the old chirp-bin calibration path. It renders a + fixed-slope chirp-bin codebook and decodes symbols with cheap event-energy + proposals, dechirp plus fixed Goertzel bins, and the same de Bruijn triplet + timeline-anchor machine. The detector keeps time/frequency ambiguity as candidate symbol/offset pairs so code constraints can choose the coherent path. The analyzer refines the final fractional delay with constrained local waveform correlation around the decoded active delay. @@ -111,26 +113,28 @@ a named invariant that the native runtime cannot protect yet. response normalization. `MimirChirpBinCalibrationModel` now preserves usable bands, expected-symbol versus observed-bin confusion observations, timing residuals, delay hypotheses, phase summaries, and an adaptive codebook plan. - The active decoder can consume that model as learned response weighting, + The reference decoder can consume that model as learned response weighting, phase-coherence weighting, first-order group-delay correction, and joint global delay/bin-shift hypotheses. The runtime emitter also consumes the model's emission plan, rendering the smaller reliable symbol alphabet at the higher recommended de Bruijn order when the physical path cannot support all 32 bins. Reports/states expose delay in microseconds as well as fractional samples. - Hybrid emits this as low-gain half-second bursts every two seconds only while - passive confidence is weak. - Use `--chirp-bin-self-test` to prove the codebook/decoder and - `--hybrid-sync-self-test` to prove that the analyzer can recover a fractional - delay from a one-second rolling-buffer chirp-bin window. The current synthetic - microsecond proof recovers a 317.375-sample delay with 0.369 us error. + Hybrid now emits the bioacoustic watermark as low-gain half-second bursts only + while passive confidence is weak; chirp-bin remains available for ASIO + calibration/replay commands. + Use `--bioacoustic-self-test` and `--standalone-bioacoustic-self-test` to + prove the active motif decoder. Use `--chirp-only-sync-self-test` to prove + that the analyzer can recover fractional delay from the bioacoustic runtime + watermark. The current synthetic active proof recovers a 317.375-sample delay + with printed 0.000 us error. - `Mimir.BufferSmoke --render-chirp-bin-f32` and `--analyze-asio-f32` provide - the current active Scarlett artifact proof. A 192 kHz chirp-bin run decoded + the retained Scarlett calibration artifact proof. A 192 kHz chirp-bin run decoded Focusrite `Loopback 1 -> Loopback 2` at `0.000 us` with 12 matched anchors and 0.999 confidence. `--calibrate-chirp-bin-asio-f32` can render/capture/analyze a calibration session and persist the response/confusion/delay model under `calibration/chirp-bin/`. `--analyze-asio-f32 --calibration ...` loads that - model into the active decoder. Physical input 1 still failed pairwise timing + model into the chirp-bin reference decoder. Physical input 1 still failed pairwise timing in the stored artifact, but it produced a useful response/confusion model with two reliable symbols. Acoustic robustness remains separate from the clean loopback timing proof, but failed timing windows now leave usable response @@ -144,9 +148,9 @@ a named invariant that the native runtime cannot protect yet. transport. BufferSmoke does not emit speaker calibration audio, so that proof is ingest-only rather than a sync report. - A standalone 192 kHz synthetic receiver test recovers a 500 ms delayed audio - stream to below printed microsecond precision using only the chirp-bin - codebook and schedule state. + The active bioacoustic standalone receiver test recovers a delayed audio + stream to below printed microsecond precision using only the motif codebook + and schedule state. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device timestamp, and native/GPU handle metadata. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live @@ -178,16 +182,16 @@ a named invariant that the native runtime cannot protect yet. other cameras. 2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained frame cadence in the rolling buffers. -3. Use chirp-bin calibration profiles from real microphones to tune acoustic - bands, gain, and code spacing without weakening the standalone - codebook/schedule receiver invariant. +3. Use bioacoustic motif response profiles from real microphones to tune + contour, formant, band, gain, and rhythm weighting without weakening the + standalone codebook/schedule receiver invariant. Keep chirp-bin calibration + artifacts as reference data, not the runtime target. 4. Add the synchronization actuator: drive a variable-rate resampler and fractional delay line per non-reference stream from the smoothed - `MimirAudioSynchronizationState`. First, prove the constrained chirplet - decoder through real loopback and microphone paths so every correctly heard - triplet becomes a deterministic timeline anchor before the actuator moves - samples. -5. Prove the chirp-bin hybrid fallback through real loopback and microphones + `MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder + through real loopback and microphone paths so every correctly heard triplet + becomes a deterministic timeline anchor before the actuator moves samples. +5. Prove the bioacoustic hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. 6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. diff --git a/docs/perfect-machine-domain-index.md b/docs/perfect-machine-domain-index.md index 992c3aa..ba0bb97 100644 --- a/docs/perfect-machine-domain-index.md +++ b/docs/perfect-machine-domain-index.md @@ -209,7 +209,10 @@ Six optical sensors with high-rate Eyes and Leap IR mean: and report authority. - `src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs`: buffer-to-report analyzer. -- `src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs`: active decoder. +- `src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs`: active + bioacoustic watermark and motif decoder. +- `src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs`: controlled + chirp-bin calibration/reference decoder. - `src/Mimir.Runtime/Synchronization/MimirChirpBinCalibrationModel.cs`: response/confusion/delay model. - `native/asio_capture/mimir_asio_capture.cpp`: runtime ASIO bridge. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 4703bcc..ea3e5cd 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -96,57 +96,47 @@ not become the synchronized program authority. ## Audio Field The six-microphone path is separate from the bridge. Scarlett speaker loopback -is the current timing authority when calibration chirplets are playing, but -Scarlett production capture belongs on ASIO rather than WASAPI shared mode. -`MimirChirpletTimeline` owns the birdsong-like timeline fingerprint: an order-3 -de Bruijn sequence over 32 time/frequency constellation symbols. Start band, -glide shape, duration, and following inter-chirp gap all carry code. Any three -consecutive correctly detected symbols identify the event index inside the -current operating horizon. Mimir continuously queues that timeline through -Fensalir audio and decodes timing through the constrained chirplet-transform -path. +is the current timing authority when active calibration is playing, but Scarlett +production capture belongs on ASIO rather than WASAPI shared mode. +`MimirBioacousticTimeline` owns the active runtime watermark described in +[[docs/bioacoustic-timeline-watermark|Bioacoustic Timeline Watermark]]: a +low-gain birdsong-like phrase language with 16 log-spaced motif symbols, three +formant-rich syllables per motif, rhythm variation, and an order-3 de Bruijn +grammar. Any three consecutive correctly decoded motifs identify the event +index inside the current operating horizon. Mimir continuously queues that +timeline through Fensalir audio and decodes timing as `bioacoustic` evidence. `MimirAudioSynchronizationSettings.Mode` chooses whether active calibration is -allowed: `chirp-only` emits the active chirp-bin witness, `passive` stays silent +allowed: `chirp-only` emits the active bioacoustic witness, `passive` stays silent and uses program-audio phase correlation, and `hybrid` uses passive evidence by default while emitting active pilot chunks only when passive confidence is weak. Active decoding does not inherit the passive two-second analysis floor: passive -still needs a longer program-audio window, while the chirp-bin witness can decode -short pilot windows once at least a code-valid triplet is present. The chirp-bin -detector now uses cheap bounded energy/onset proposals, dechirp plus fixed -Goertzel bins, explicit time/frequency ambiguity candidates, and a constrained -scheduled-waveform correlation for standalone source-offset refinement. Pairwise -sync compares matched anchors first, then only accepts independent clock-fit -offsets inside the live latency horizon so period aliases do not become absurd -reports. Each classified chirp carries the full dechirped bin-energy surface, -and decodes aggregate that into a `MimirChirpBinCalibrationModel` with measured +still needs a longer program-audio window, while the bioacoustic witness can +decode once at least a code-valid motif triplet is present. The bioacoustic +detector now uses generous motif proposals, matched motif scoring, triplet +grammar decoding, source clock fitting, and fractional waveform refinement. +Pairwise sync compares matched anchors first, then only accepts independent +clock-fit offsets inside the live latency horizon so period aliases do not +become absurd reports. The old chirp-bin detector remains as a +calibration/reference artifact: it carries the full dechirped bin-energy surface +and aggregates decodes into a `MimirChirpBinCalibrationModel` with measured usable bands, expected-symbol versus observed-bin confusion observations, timing residuals, delay hypotheses, phase summaries, and an adaptive codebook plan. The analyzer keeps raw profiles even when no timing report is accepted, -and the active decoder can consume a persisted model for learned response -weighting, phase-coherence weighting, first-order group-delay correction, and -joint global delay/bin-shift hypotheses. Runtime chirp-bin emission consumes the -same model's emission plan, so -the emitted timeline can use the measured reliable symbol set at a higher -de Bruijn order instead of continuing to spend code on dead bins. Reports/states +so physical mic failures can still guide bioacoustic motif weighting. Reports/states expose `delayUs` next to fractional sample delay. -`Mimir.BufferSmoke ---chirp-only-sync-self-test --sample-rate 192000` and `--hybrid-sync-self-test ---sample-rate 192000` both recover a 1269.5-sample synthetic delay with printed -0.000 us error. `--standalone-chirp-bin-self-test --sample-rate 192000 ---delay-samples 96000` proves a receiver with only codebook/schedule state can -recover a 500 ms delayed stream to below printed microsecond precision. -Reports now carry fractional delay and per-band matched energy. The first -`MimirChirpletSymbolCodebook` owns separable symbol definitions; every symbol -has a unique chirp shape, with rhythm as additional evidence. `MimirChirpletStreamDecoder` -is the constrained chirplet-transform receiver: it turns a bounded PCM window -into transform frames with multiple phase-invariant symbol candidates and -refined per-candidate sample offsets, code-valid triplet anchors selected -through gap/clock coherence, and an affine source clock fit. The synthetic -`Mimir.BufferSmoke --chirplet-self-test` path renders canonical timeline audio -into memory and currently recovers events 0-12 with a 47999.999990 Hz clock fit -and 0.000014 sample MAE. The analyzer only emits timing reports from -matched canonical anchors. The hub also owns cached reports plus smoothed -per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` runs analysis +`Mimir.BufferSmoke --bioacoustic-self-test` proves code-valid motif anchors. +`--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5` +proves a receiver with only codebook/schedule state can recover delayed source +time to below printed microsecond precision. `--chirp-only-sync-self-test +--sample-rate 48000` recovers a 317.375-sample synthetic delay with printed +0.000 us error using `evidence=bioacoustic`. +Reports now carry fractional delay and per-band matched energy. The older +`MimirChirpletSymbolCodebook` / `MimirChirpletStreamDecoder` path remains a +diagnostic reference for constrained chirplet-transform work. The active +receiver is the bioacoustic motif timeline; the analyzer only emits active +timing reports from matched canonical anchors. The hub also owns cached reports +plus smoothed per-source sync state with delay-slope/SRO in ppm. +`MimirRuntime` runs analysis online as a bounded rotating service and can print live telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are passive readers of cached sync state; they must not invoke the analyzer. Current app testing proves @@ -170,9 +160,10 @@ evidence; Starfire still owns the heavy soundfield and sensor-fusion work. The ASIO probe can now play raw mono Float32 timeline audio through the Focusrite outputs while capturing every ASIO input as raw interleaved Float32. `Mimir.BufferSmoke --analyze-asio-f32` feeds those captured channels into the -same runtime active analyzer. `--calibrate-chirp-bin-asio-f32` computes and -persists the response/confusion/delay model per output/mic path, and -`--analyze-asio-f32 --calibration ...` loads it into the active decoder. +same runtime analyzer. `--calibrate-chirp-bin-asio-f32` computes and persists +the response/confusion/delay model per output/mic path, and +`--analyze-asio-f32 --calibration ...` loads it into the chirp-bin reference +decoder. `native/asio_capture` and `MimirAsioStreamSource` are now the runtime Scarlett path: Focusrite ASIO callbacks feed sample-bearing 192 kHz Float32 blocks into `Mimir.Runtime` in process, without the diagnostic JSON/stdout bridge. The diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index df2766d..62af02f 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -71,50 +71,38 @@ Get-Content .\state\evidence.jsonl -Tail 8 640x480@60, and 81.10 fps when the PS3 Eye runs 320x240@187. The regular Kiyo and second PS3 Eye were absent, so six-camera viability is not proven. - Current active sync smoke uses Scarlett speaker loopback as timing - authority. `MimirChirpletTimeline` owns the structured birdsong-like timeline - fingerprint: an order-3 de Bruijn sequence over 32 time/frequency - constellation symbols. Start band, glide shape, duration, and following - inter-chirp gap all carry code, so any three consecutive correctly detected - symbols identify the event index inside the current operating horizon. The - timeline owns Fensalir PCM segment rendering, transform-frame decoding, - triplet anchor decoding, and per-band response hooks. Sync reports now include + authority. `MimirBioacousticTimeline` now owns the active runtime watermark: + low-gain birdsong-like motifs with log-spaced roots, formant-rich syllables, + rhythm variation, and an order-3 de Bruijn phrase grammar. Any three + consecutive correctly decoded motifs identify the event index inside the + current operating horizon. Runtime active sync reports + `evidence=bioacoustic`; [[docs/bioacoustic-timeline-watermark|Bioacoustic + Timeline Watermark]] is the current map. Sync reports now include fractional delay and per-band matched energy. `MimirAudioSynchronizationState` - now tracks smoothed delay and delay-slope/SRO ppm. `MimirChirpletSymbolCodebook` - owns separable symbol definitions, and `MimirChirpletStreamDecoder` is the - constrained chirplet-transform receiver: it emits transform frames with - multiple phase-invariant symbol candidates and refined per-candidate sample - offsets, selects code-valid triplet anchors through gap/clock coherence, and - fits a source clock from a bounded PCM window. The canonical synthetic check is - `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`; - it currently detects all 15 emitted chirps and recovers 13 anchors for events - 0-12 with a 47999.999990 Hz clock fit and 0.000014 sample MAE. `MimirRuntime` runs sync analysis as a bounded rotating + now tracks smoothed delay and delay-slope/SRO ppm. The older + `MimirChirpletSymbolCodebook` / `MimirChirpletStreamDecoder` path remains a + diagnostic reference for constrained chirplet-transform work; it is not the + active runtime witness. `MimirRuntime` runs sync analysis as a bounded rotating service and caches reports/states for UI and telemetry; readouts must stay passive. Audio sync mode is runtime-owned: `chirp-only` emits the active - chirp-bin witness continuously, `passive` disables emission and uses + bioacoustic witness continuously, `passive` disables emission and uses program-audio phase correlation, and `hybrid` emits active pilot chunks only - when passive confidence is weak. The active chirp-bin path now has its own + when passive confidence is weak. The active bioacoustic path has its own short-window analyzer floor instead of inheriting the passive two-second gate. - It uses cheap energy/onset proposals, dechirp plus fixed Goertzel bins, - explicit time/frequency ambiguity candidates, and constrained local waveform + It uses energy/onset proposals, dense fallback probes, bounded motif matching, + de Bruijn triplet anchoring, clock fit, and constrained local waveform correlation for final fractional delay. It also supports standalone source offset recovery from schedule/codebook state, which is the Raven/phone shape. - Each classified chirp carries the full dechirped bin-energy surface; sync - reports expose those bands, and `MimirChirpBinCalibrationModel` preserves - measured usable bands, response/confusion observations, timing residuals, - delay hypotheses, phase summaries, and adaptive codebook plans per - output/mic path. The active decoder can load the model for learned weighting, - phase-coherence weighting, first-order group-delay correction, and joint - global delay/bin-shift hypotheses. Runtime chirp-bin emission - loads the same model's emission plan, so a physical calibration can shrink the - emitted alphabet and raise sequence order instead of continuing to use dead - bins. - `Mimir.BufferSmoke - --chirp-only-sync-self-test --sample-rate 192000` and - `--hybrid-sync-self-test --sample-rate 192000` both recover a 1269.5-sample - synthetic delay with printed 0.000 us error. Reports and sync states expose - `delayUs`. `--standalone-chirp-bin-self-test --sample-rate 192000 - --delay-samples 96000` recovers a 500 ms delayed stream below printed - microsecond precision without loopback. + Each classified motif carries per-band response evidence. The older + `MimirChirpBinCalibrationModel` remains the controlled chirp-bin + response/confusion/delay reference surface; use it to inform bioacoustic motif + weighting, not as the runtime sound. + `Mimir.BufferSmoke --bioacoustic-self-test` proves code-valid motif anchors. + `--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples + 1269.5` recovers delayed canonical time below printed microsecond precision + without loopback. `--chirp-only-sync-self-test --sample-rate 48000` recovers a + 317.375-sample synthetic delay with printed 0.000 us error using + `evidence=bioacoustic`. Reports and sync states expose `delayUs`. Actual Mimir.App testing proves Fensalir audio can wake Scarlett loopback, keep mic buffers live, and produce confident online passive sync diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index d81fa7c..e71fe98 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -11,6 +11,18 @@ if (args.Any(arg => string.Equals(arg, "--chirp-bin-self-test", StringComparison return RunChirpBinSelfTest(); } +if (args.Any(arg => string.Equals(arg, "--bioacoustic-self-test", StringComparison.OrdinalIgnoreCase))) +{ + return RunBioacousticSelfTest(ParseIntOption(args, "--sample-rate", MimirBioacousticTimeline.SampleRate)); +} + +if (args.Any(arg => string.Equals(arg, "--standalone-bioacoustic-self-test", StringComparison.OrdinalIgnoreCase))) +{ + return RunStandaloneBioacousticSelfTest( + ParseIntOption(args, "--sample-rate", MimirBioacousticTimeline.SampleRate), + ParseDoubleOption(args, "--delay-samples", 1269.5)); +} + if (args.Any(arg => string.Equals(arg, "--standalone-chirp-bin-self-test", StringComparison.OrdinalIgnoreCase))) { return RunStandaloneChirpBinSelfTest( @@ -50,6 +62,14 @@ if (args.Any(arg => string.Equals(arg, "--render-chirp-bin-f32", StringCompariso LoadOptionalCalibration(args)?.EmissionPlan); } +if (args.Any(arg => string.Equals(arg, "--render-bioacoustic-f32", StringComparison.OrdinalIgnoreCase))) +{ + return RenderBioacousticFloat32( + ParseStringOption(args, "--output", "artifacts/asio/bioacoustic-f32.raw"), + ParseIntOption(args, "--sample-rate", MimirBioacousticTimeline.SampleRate), + ParseDoubleOption(args, "--seconds", 3.0)); +} + if (args.Any(arg => string.Equals(arg, "--analyze-asio-f32", StringComparison.OrdinalIgnoreCase))) { return AnalyzeAsioFloat32( @@ -316,6 +336,81 @@ static int RunStandaloneChirpBinSelfTest(int sampleRate, double delaySamples) return 0; } +static int RunBioacousticSelfTest(int sampleRate) +{ + var timeline = MimirBioacousticTimeline.Default; + var samples = new List(); + for (ulong segment = 0; segment < 6; segment++) + { + samples.AddRange(timeline.RenderSegmentMonoFloat(segment, sampleRate)); + } + + var decode = timeline.DecodeStreamWindow(samples.ToArray(), sampleRate); + var meanAbsoluteError = decode.ClockFit?.MeanAbsoluteErrorSamples ?? double.PositiveInfinity; + Console.WriteLine( + $"bioacoustic-self-test frames={decode.Frames.Count} symbols={decode.Symbols.Count} anchors={decode.Anchors.Count} " + + $"clock={(decode.ClockFit is null ? "none" : decode.ClockFit.EffectiveSampleRate.ToString("0.000000"))} " + + $"confidence={(decode.ClockFit?.Confidence ?? 0.0):0.000} mae={meanAbsoluteError:0.000000}"); + Console.WriteLine("bioacoustic-expected " + string.Join(",", Enumerable.Range(0, 12).Select(index => timeline.EventForIndex((ulong)index).SymbolId))); + Console.WriteLine("bioacoustic-symbols " + string.Join(",", decode.Symbols.Take(12).Select(symbol => $"{symbol.SymbolId}@{symbol.SampleOffset:0}:{symbol.Energy:0.000}"))); + + foreach (var anchor in decode.Anchors.Take(16)) + { + var expected = timeline.EventForIndex(anchor.EventIndex).StartSeconds * sampleRate; + Console.WriteLine( + $"bioacoustic-anchor event={anchor.EventIndex} actual={anchor.SampleOffset:0.000} expected={expected:0.000} " + + $"error={anchor.SampleOffset - expected:0.000} confidence={anchor.Confidence:0.000}"); + } + + if (decode.ClockFit == null || decode.Anchors.Count < 12 || meanAbsoluteError > 12.0) + { + Console.Error.WriteLine("bioacoustic self-test failed: log-motif decoder did not recover stable timeline anchors"); + return 1; + } + + return 0; +} + +static int RunStandaloneBioacousticSelfTest(int sampleRate, double delaySamples) +{ + var timeline = MimirBioacousticTimeline.Default; + var samples = new List(); + for (ulong segment = 0; segment < 8; segment++) + { + samples.AddRange(timeline.RenderSegmentMonoFloat(segment, sampleRate)); + } + + var delayed = ApplyFractionalDelay(samples.ToArray(), delaySamples); + var decode = timeline.DecodeStreamWindow(delayed, sampleRate); + var expectedDelay = delaySamples; + var actualDelay = decode.ClockFit?.SourceOffsetSamples ?? double.NaN; + var error = Math.Abs(actualDelay - expectedDelay); + Console.WriteLine( + $"standalone-bioacoustic-self-test frames={decode.Frames.Count} anchors={decode.Anchors.Count} " + + $"clock={(decode.ClockFit is null ? "none" : decode.ClockFit.EffectiveSampleRate.ToString("0.000000"))} " + + $"sourceOffset={actualDelay:0.000000} expected={expectedDelay:0.000000} errorSamples={error:0.000000} " + + $"errorUs={error * 1_000_000.0 / sampleRate:0.000} confidence={(decode.ClockFit?.Confidence ?? 0.0):0.000}"); + + foreach (var anchor in decode.Anchors.Take(16)) + { + var expected = timeline.EventForIndex(anchor.EventIndex).StartSeconds * sampleRate + expectedDelay; + Console.WriteLine( + $"standalone-bioacoustic-anchor event={anchor.EventIndex} actual={anchor.SampleOffset:0.000} expected={expected:0.000} " + + $"error={anchor.SampleOffset - expected:0.000} confidence={anchor.Confidence:0.000}"); + } + + if (decode.ClockFit == null || + decode.Anchors.Count < 12 || + decode.ClockFit.Confidence < 0.60 || + error * 1_000_000.0 / sampleRate > 1.0) + { + Console.Error.WriteLine("standalone bioacoustic self-test failed: receiver did not recover canonical source time from delayed audio alone"); + return 1; + } + + return 0; +} + static int RunPassiveSyncSelfTest() { const int sampleRate = 48_000; @@ -374,13 +469,18 @@ static int RunActiveSyncSelfTest(int sampleRate, MimirAudioSyncMode mode) { const string referenceSourceId = "loopback-test"; const string candidateSourceId = "mic-test"; - var delaySamples = 317.375 * sampleRate / (double)MimirChirpBinTimeline.SampleRate; + var delaySamples = 317.375 * sampleRate / (double)MimirBioacousticTimeline.SampleRate; - var firstSegment = MimirChirpBinTimeline.Default.RenderSegmentMonoFloat(0, sampleRate); - var secondSegment = MimirChirpBinTimeline.Default.RenderSegmentMonoFloat(1, sampleRate); - var reference = new float[firstSegment.Length + secondSegment.Length]; - Array.Copy(firstSegment, 0, reference, 0, firstSegment.Length); - Array.Copy(secondSegment, 0, reference, firstSegment.Length, secondSegment.Length); + var segments = Enumerable.Range(0, 4) + .Select(segment => MimirBioacousticTimeline.Default.RenderSegmentMonoFloat((ulong)segment, sampleRate)) + .ToArray(); + var reference = new float[segments.Sum(segment => segment.Length)]; + var writeOffset = 0; + foreach (var segment in segments) + { + Array.Copy(segment, 0, reference, writeOffset, segment.Length); + writeOffset += segment.Length; + } var candidate = ApplyFractionalDelay(reference, delaySamples); var referenceBuffer = new MimirRollingStreamBuffer( @@ -404,14 +504,23 @@ static int RunActiveSyncSelfTest(int sampleRate, MimirAudioSyncMode mode) if (report == null) { - Console.Error.WriteLine($"{SyncModeLabel(mode)} sync self-test failed: analyzer did not report from a short chirp-bin window"); + Console.Error.WriteLine($"{SyncModeLabel(mode)} sync self-test failed: analyzer did not report from a short bioacoustic window"); return 1; } var error = Math.Abs(report.FractionalDelaySamples - delaySamples); Console.WriteLine( $"{SyncModeLabel(mode)}-sync-self-test evidence={report.EvidenceKind} delaySamples={report.FractionalDelaySamples:0.000000} delayUs={report.DelayMicroseconds:0.000} expected={delaySamples:0.000000} errorSamples={error:0.000000} errorUs={error * 1_000_000.0 / report.SampleRate:0.000} confidence={report.Confidence:0.000} events={report.TimelineMatchedEvents} compared={report.ComparedSamples}"); - return string.Equals(report.EvidenceKind, "chirp-bin", StringComparison.Ordinal) && + if (mode == MimirAudioSyncMode.Hybrid && + string.Equals(report.EvidenceKind, "passive", StringComparison.Ordinal)) + { + return report.Confidence > 0.50 && + error * 1_000_000.0 / report.SampleRate < 5.0 + ? 0 + : 1; + } + + return string.Equals(report.EvidenceKind, "bioacoustic", StringComparison.Ordinal) && report.TimelineMatchedEvents >= 3 && report.Confidence > 0.70 && error * 1_000_000.0 / report.SampleRate < 1.0 @@ -492,6 +601,38 @@ static int RenderChirpBinFloat32( return 0; } +static int RenderBioacousticFloat32(string outputPath, int sampleRate, double seconds) +{ + var segmentCount = Math.Max(1, (int)Math.Ceiling(seconds / MimirBioacousticTimeline.SegmentSeconds)); + var samples = new List(Math.Max(1, (int)Math.Ceiling(seconds * sampleRate))); + for (var segment = 0; segment < segmentCount; segment++) + { + samples.AddRange(MimirBioacousticTimeline.Default.RenderSegmentMonoFloat((ulong)segment, sampleRate)); + } + + var requestedSamples = Math.Max(1, (int)Math.Round(seconds * sampleRate)); + if (samples.Count > requestedSamples) + { + samples.RemoveRange(requestedSamples, samples.Count - requestedSamples); + } + + var directory = Path.GetDirectoryName(Path.GetFullPath(outputPath)); + if (!string.IsNullOrEmpty(directory)) + { + Directory.CreateDirectory(directory); + } + + var bytes = new byte[samples.Count * sizeof(float)]; + for (var index = 0; index < samples.Count; index++) + { + BitConverter.TryWriteBytes(bytes.AsSpan(index * sizeof(float), sizeof(float)), samples[index]); + } + + File.WriteAllBytes(outputPath, bytes); + Console.WriteLine($"bioacoustic-render-f32 path={outputPath} sampleRate={sampleRate} seconds={seconds:0.000} samples={samples.Count} symbols={MimirBioacousticTimeline.SymbolCount} order={MimirBioacousticTimeline.TimelineOrder}"); + return 0; +} + static MimirChirpBinCalibrationModel? LoadOptionalCalibration(string[] args) { var path = ParseStringOption(args, "--calibration", ""); diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 2282fc8..f48698d 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -181,7 +181,7 @@ public sealed class MimirRuntime : IAquariumRuntime calibrationSegmentIndex = currentSegmentIndex; } - var nextSegmentSeconds = CalibrationStartSeconds + calibrationSegmentIndex * MimirChirpletTimeline.SegmentSeconds; + var nextSegmentSeconds = CalibrationStartSeconds + calibrationSegmentIndex * MimirBioacousticTimeline.SegmentSeconds; if (runtimeSeconds < nextSegmentSeconds) { return; @@ -192,16 +192,14 @@ public sealed class MimirRuntime : IAquariumRuntime var batch = RenderCalibrationBatchPcm16Base64( calibrationSegmentIndex, segmentCount, - UseChirpBinTimeline(audioSyncSettings.Mode), - synchronization.ChirpBinEmissionPlan, out var peak); audio.EnqueuePcm16Base64( batch, - MimirChirpletTimeline.SampleRate, + MimirBioacousticTimeline.SampleRate, channels: 1, gain: outputGain); Console.WriteLine( - $"mimir-chirplet-batch mode={DescribeAudioSyncMode()} firstSegment={calibrationSegmentIndex} segments={segmentCount} seconds={segmentCount * MimirChirpletTimeline.SegmentSeconds:0.00} peak={peak:0.000000} gain={outputGain:0.###} base64Bytes={batch.Length}"); + $"mimir-bioacoustic-batch mode={DescribeAudioSyncMode()} firstSegment={calibrationSegmentIndex} segments={segmentCount} seconds={segmentCount * MimirBioacousticTimeline.SegmentSeconds:0.00} peak={peak:0.000000} gain={outputGain:0.###} base64Bytes={batch.Length}"); if (audioSyncSettings.Mode == MimirAudioSyncMode.Hybrid) { lastHybridWatermarkSegment = (long)calibrationSegmentIndex; @@ -219,7 +217,7 @@ public sealed class MimirRuntime : IAquariumRuntime return 0UL; } - return (ulong)Math.Floor((runtimeSeconds - CalibrationStartSeconds) / MimirChirpletTimeline.SegmentSeconds); + return (ulong)Math.Floor((runtimeSeconds - CalibrationStartSeconds) / MimirBioacousticTimeline.SegmentSeconds); } private string DescribeChirpletReference() @@ -229,26 +227,24 @@ public sealed class MimirRuntime : IAquariumRuntime return $"{audioSyncSettings.ReferenceSourceId} passive mode; calibration emission disabled"; } - var emittedUntilSeconds = calibrationSegmentIndex * MimirChirpletTimeline.SegmentSeconds; - return $"{audioSyncSettings.ReferenceSourceId} {DescribeAudioSyncMode()} timeline {MimirChirpletTimeline.SegmentSeconds:0.00}s segments emitted to {emittedUntilSeconds:0.00}s passiveConfidence={lastPassiveSynchronizationConfidence:0.000}"; + var emittedUntilSeconds = calibrationSegmentIndex * MimirBioacousticTimeline.SegmentSeconds; + return $"{audioSyncSettings.ReferenceSourceId} {DescribeAudioSyncMode()} bioacoustic timeline {MimirBioacousticTimeline.SegmentSeconds:0.00}s segments emitted to {emittedUntilSeconds:0.00}s passiveConfidence={lastPassiveSynchronizationConfidence:0.000}"; } private static string RenderCalibrationBatchPcm16Base64( ulong firstSegment, int segmentCount, - bool useChirpBinTimeline, - MimirChirpBinCodebookPlan? chirpBinCodebookPlan, out float peak) { - var samplesPerSegment = (int)Math.Round(MimirChirpletTimeline.SegmentSeconds * MimirChirpletTimeline.SampleRate); + var samplesPerSegment = (int)Math.Round(MimirBioacousticTimeline.SegmentSeconds * MimirBioacousticTimeline.SampleRate); var bytes = new byte[samplesPerSegment * Math.Max(1, segmentCount) * sizeof(short)]; var byteIndex = 0; peak = 0.0f; for (var segment = 0; segment < segmentCount; segment++) { - var samples = useChirpBinTimeline - ? MimirChirpBinTimeline.Default.RenderSegmentMonoFloat(firstSegment + (ulong)segment, MimirChirpBinTimeline.SampleRate, chirpBinCodebookPlan) - : MimirChirpletTimeline.Default.RenderSegmentMonoFloat(firstSegment + (ulong)segment); + var samples = MimirBioacousticTimeline.Default.RenderSegmentMonoFloat( + firstSegment + (ulong)segment, + MimirBioacousticTimeline.SampleRate); for (var index = 0; index < samples.Length; index++) { peak = Math.Max(peak, Math.Abs(samples[index])); @@ -261,11 +257,6 @@ public sealed class MimirRuntime : IAquariumRuntime return Convert.ToBase64String(bytes); } - private static bool UseChirpBinTimeline(MimirAudioSyncMode mode) - { - return mode is MimirAudioSyncMode.ChirpOnly or MimirAudioSyncMode.Hybrid; - } - private string DescribeBuffers() { return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer)); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index 54899d3..f56aaec 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -8,6 +8,7 @@ public sealed class MimirAudioSynchronizationAnalyzer private const double MinPassiveWindowSeconds = 2.0; private const double MinChirpletWindowSeconds = 2.0; private const double MinChirpBinWindowSeconds = 0.40; + private const double MinBioacousticWindowSeconds = 0.50; private const double MaxPairClockFitDelaySeconds = 10.0; private const double PassiveReportConfidence = 0.08; private readonly List lastDecodeTraces = []; @@ -150,9 +151,9 @@ public sealed class MimirAudioSynchronizationAnalyzer } } - var useChirpBinTimeline = mode is MimirAudioSyncMode.ChirpOnly or MimirAudioSyncMode.Hybrid; - var activeMinSeconds = useChirpBinTimeline - ? MinChirpBinWindowSeconds + var useBioacousticTimeline = mode is MimirAudioSyncMode.ChirpOnly or MimirAudioSyncMode.Hybrid; + var activeMinSeconds = useBioacousticTimeline + ? MinBioacousticWindowSeconds : MinChirpletWindowSeconds; var minActiveSamples = MinWindowSamples(activeReferenceBlock.SampleRate, activeMinSeconds); if (compared < minActiveSamples) @@ -162,25 +163,23 @@ public sealed class MimirAudioSynchronizationAnalyzer buffer.Descriptor.SourceId, activeReferenceBlock.SampleRate, compared, - useChirpBinTimeline - ? "chirp-bin-insufficient-window" + useBioacousticTimeline + ? "bioacoustic-insufficient-window" : "chirplet-insufficient-window")); continue; } - var comparedReferenceDecode = useChirpBinTimeline - ? MimirChirpBinTimeline.Default.DecodeStreamWindow( + var comparedReferenceDecode = useBioacousticTimeline + ? MimirBioacousticTimeline.Default.DecodeStreamWindow( referenceWindow, - activeReferenceBlock.SampleRate, - calibrationModel?.PathFor(reference.Descriptor.SourceId)) + activeReferenceBlock.SampleRate) : MimirChirpletTimeline.Default.DecodeStreamWindow(referenceWindow, activeReferenceBlock.SampleRate); - var candidateDecode = useChirpBinTimeline - ? MimirChirpBinTimeline.Default.DecodeStreamWindow( + var candidateDecode = useBioacousticTimeline + ? MimirBioacousticTimeline.Default.DecodeStreamWindow( candidateWindow, - activeReferenceBlock.SampleRate, - calibrationModel?.PathFor(buffer.Descriptor.SourceId)) + activeReferenceBlock.SampleRate) : MimirChirpletTimeline.Default.DecodeStreamWindow(candidateWindow, activeReferenceBlock.SampleRate); - if (useChirpBinTimeline) + if (useBioacousticTimeline) { StoreCalibrationProfile(reference.Descriptor.SourceId, activeReferenceBlock.SampleRate, comparedReferenceDecode); StoreCalibrationProfile(buffer.Descriptor.SourceId, activeReferenceBlock.SampleRate, candidateDecode); @@ -208,10 +207,10 @@ public sealed class MimirAudioSynchronizationAnalyzer continue; } - var decodedDelaySamples = useChirpBinTimeline + var decodedDelaySamples = useBioacousticTimeline ? RefineDelayFromWaveform(referenceWindow, candidateWindow, deterministicFit.DelaySamples, activeReferenceBlock.SampleRate) : deterministicFit.DelaySamples; - var bandResponses = useChirpBinTimeline + var bandResponses = useBioacousticTimeline ? candidateDecode.BandResponses : MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, activeReferenceBlock.SampleRate); reports.Add(new MimirAudioSynchronizationReport( @@ -229,7 +228,7 @@ public sealed class MimirAudioSynchronizationAnalyzer buffer.Latest?.Sequence ?? 0, deterministicFit.MatchedEvents, deterministicFit.Confidence, - useChirpBinTimeline ? "chirp-bin" : "chirplet")); + useBioacousticTimeline ? "bioacoustic" : "chirplet")); } return reports; @@ -314,6 +313,16 @@ public sealed class MimirAudioSynchronizationAnalyzer var residualConfidence = 1.0 / (1.0 + error / 32.0); var countConfidence = Math.Clamp(matched.Count / 12.0, 0.0, 1.0); var energyConfidence = Math.Clamp(totalWeight / matched.Count, 0.0, 1.0); + if (reference.ClockFit.Confidence >= 0.75 && candidate.ClockFit.Confidence >= 0.75) + { + var matchedClockDelay = candidate.ClockFit.SourceOffsetSamples - reference.ClockFit.SourceOffsetSamples; + if (Math.Abs(matchedClockDelay - delay) <= reference.ClockFit.EffectiveSampleRate * 0.004) + { + var clockConfidence = Math.Sqrt(reference.ClockFit.Confidence * candidate.ClockFit.Confidence); + return (matchedClockDelay, residualConfidence * 0.35 + countConfidence * 0.20 + energyConfidence * 0.20 + clockConfidence * 0.25, matched.Count); + } + } + return (delay, residualConfidence * 0.50 + countConfidence * 0.25 + energyConfidence * 0.25, matched.Count); } @@ -374,21 +383,25 @@ public sealed class MimirAudioSynchronizationAnalyzer } } - if (bestLag <= firstLag || bestLag >= lastLag) + var bestFractionalLag = (double)bestLag; + var bestFractionalScore = bestScore; + for (var step = -8; step <= 8; step++) { - return bestLag; + var lag = bestLag + step / 8.0; + if (lag < firstLag || lag > lastLag) + { + continue; + } + + var score = NormalizedFractionalLagScore(reference, candidate, lag); + if (score > bestFractionalScore) + { + bestFractionalScore = score; + bestFractionalLag = lag; + } } - var left = NormalizedLagScore(reference, candidate, bestLag - 1); - var middle = NormalizedLagScore(reference, candidate, bestLag); - var right = NormalizedLagScore(reference, candidate, bestLag + 1); - var denominator = left - 2.0 * middle + right; - if (Math.Abs(denominator) <= 1.0e-12) - { - return bestLag; - } - - return bestLag + Math.Clamp(0.5 * (left - right) / denominator, -0.5, 0.5); + return bestFractionalLag; } private static double NormalizedLagScore(ReadOnlySpan reference, ReadOnlySpan candidate, int lag) @@ -416,6 +429,34 @@ public sealed class MimirAudioSynchronizationAnalyzer : dot / Math.Sqrt(referenceEnergy * candidateEnergy); } + private static double NormalizedFractionalLagScore(ReadOnlySpan reference, ReadOnlySpan candidate, double lag) + { + var leftLag = (int)Math.Floor(lag); + var fraction = lag - leftLag; + var count = Math.Min(reference.Length, candidate.Length - leftLag - 1); + if (count <= 0) + { + return double.NegativeInfinity; + } + + var dot = 0.0; + var referenceEnergy = 0.0; + var candidateEnergy = 0.0; + for (var index = 0; index < count; index++) + { + var referenceSample = reference[index]; + var candidateSample = candidate[index + leftLag] + + (candidate[index + leftLag + 1] - candidate[index + leftLag]) * fraction; + dot += referenceSample * candidateSample; + referenceEnergy += referenceSample * referenceSample; + candidateEnergy += candidateSample * candidateSample; + } + + return referenceEnergy <= 1.0e-12 || candidateEnergy <= 1.0e-12 + ? double.NegativeInfinity + : dot / Math.Sqrt(referenceEnergy * candidateEnergy); + } + private static float[] ExtractMonoWindow( MimirRollingStreamBuffer buffer, out MimirAudioBlockDescriptor? latestBlock, diff --git a/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs new file mode 100644 index 0000000..d18a2aa --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs @@ -0,0 +1,844 @@ +using System.Collections.Concurrent; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirBioacousticSyllable( + double StartSeconds, + double DurationSeconds, + double StartHz, + double EndHz, + double Weight); + +public sealed record MimirBioacousticMotifDefinition( + int SymbolId, + IReadOnlyList Syllables); + +internal sealed record BioacousticKernel( + double[] Samples, + IReadOnlyList CenterFrequencies, + double Energy); + +internal sealed record BioacousticKernelSet( + BioacousticKernel[] Symbols); + +internal sealed record BioacousticTimelinePlan( + int[] Symbols, + int Order, + int[] TimelineSymbols, + Dictionary CodeToIndex); + +public sealed class MimirBioacousticTimeline +{ + public const int SampleRate = 48_000; + public const double SegmentSeconds = 0.5; + public const int SymbolCount = 16; + public const int TimelineOrder = 3; + + private const double FirstEventSeconds = 0.08; + private const double EventSpacingSeconds = 0.16; + private const double MotifDurationSeconds = 0.118; + private const double SyllableSeconds = 0.034; + private const double LowestRootHz = 3_200.0; + private const double HighestRootHz = 8_800.0; + private const double Gain = 0.038; + private const int MaxSymbolCandidatesPerFrame = 3; + private const double ProposalBudgetMultiplier = 7.0; + + private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder)); + private static readonly MimirBioacousticMotifDefinition[] Motifs = BuildMotifs(); + private static readonly ConcurrentDictionary KernelSets = new(); + private static readonly ConcurrentDictionary TimelinePlans = new(); + + public static MimirBioacousticTimeline Default { get; } = new(); + + public IReadOnlyList Codebook => Motifs; + + public float[] RenderSegmentMonoFloat(ulong segmentIndex) + => RenderSegmentMonoFloat(segmentIndex, SampleRate); + + public float[] RenderSegmentMonoFloat(ulong segmentIndex, int sampleRate) + { + var segmentStartSeconds = segmentIndex * SegmentSeconds; + var samples = new float[(int)Math.Round(SegmentSeconds * sampleRate)]; + foreach (var timelineEvent in EventsOverlapping(segmentStartSeconds, SegmentSeconds)) + { + AddMotif(samples, sampleRate, timelineEvent, segmentStartSeconds, Gain); + } + + return samples; + } + + public MimirChirpletStreamDecode DecodeStreamWindow(ReadOnlySpan samples, int sampleRate) + { + if (samples.Length == 0) + { + return new MimirChirpletStreamDecode([], [], [], null, []); + } + + var frames = DetectFrames(samples, sampleRate); + var symbols = frames + .Select(frame => new MimirChirpletSymbolObservation( + frame.BestCandidate.SymbolId, + frame.SampleOffset, + frame.BestCandidate.Energy)) + .ToArray(); + var anchors = DecodeAnchors(frames, sampleRate); + var clock = FitClock(anchors, sampleRate); + if (clock != null) + { + clock = clock with + { + SourceOffsetSamples = RefineSourceOffset(samples, sampleRate, clock.SourceOffsetSamples), + }; + } + + return new MimirChirpletStreamDecode(frames, symbols, anchors, clock, EstimateBandResponse(frames)); + } + + public MimirChirpletTimelineEvent EventForIndex(ulong eventIndex) + { + var symbolId = SymbolForEvent(eventIndex); + var startSeconds = EventStartSeconds(eventIndex); + var motif = Motifs[symbolId]; + var first = motif.Syllables[0]; + return new MimirChirpletTimelineEvent( + eventIndex, + symbolId, + startSeconds, + new MimirChirpletTone( + startSeconds, + MotifDurationSeconds, + first.StartHz, + motif.Syllables[^1].EndHz, + Gain)); + } + + public IReadOnlyList EventsOverlapping(double startSeconds, double durationSeconds) + { + var endSeconds = startSeconds + durationSeconds; + var firstIndex = Math.Max(0, (long)Math.Floor((startSeconds - FirstEventSeconds - MotifDurationSeconds) / EventSpacingSeconds) - 2); + var lastIndex = Math.Max(firstIndex, (long)Math.Ceiling((endSeconds - FirstEventSeconds) / EventSpacingSeconds) + 2); + var events = new List((int)(lastIndex - firstIndex + 1)); + for (var eventIndex = firstIndex; eventIndex <= lastIndex; eventIndex++) + { + var timelineEvent = EventForIndex((ulong)eventIndex); + if (timelineEvent.StartSeconds < endSeconds && + timelineEvent.StartSeconds + MotifDurationSeconds > startSeconds) + { + events.Add(timelineEvent); + } + } + + return events; + } + + private static IReadOnlyList DetectFrames(ReadOnlySpan samples, int sampleRate) + { + var motifSamples = Math.Max(1, (int)Math.Round(MotifDurationSeconds * sampleRate)); + if (samples.Length < motifSamples) + { + return []; + } + + var hopSamples = Math.Max(1, sampleRate / 1_000); + var energyTrace = BuildWindowEnergyTrace(samples, motifSamples, hopSamples); + var threshold = AdaptiveThreshold(energyTrace); + var proposals = new List(); + for (var index = 1; index < energyTrace.Length - 1; index++) + { + if (energyTrace[index] >= threshold && + energyTrace[index] >= energyTrace[index - 1] && + energyTrace[index] >= energyTrace[index + 1]) + { + proposals.Add(index * hopSamples); + } + } + + var maxExpectedEvents = Math.Max( + 8, + (int)Math.Ceiling(samples.Length / (double)sampleRate / EventSpacingSeconds * ProposalBudgetMultiplier)); + var frames = new List(); + foreach (var proposal in proposals + .OrderByDescending(offset => energyTrace[Math.Clamp(offset / hopSamples, 0, energyTrace.Length - 1)]) + .Take(maxExpectedEvents) + .Order()) + { + var frame = ClassifyAt(samples, sampleRate, proposal, Math.Max(2, sampleRate / 600)); + if (frame != null) + { + frames.Add(frame); + } + } + + if (frames.Count < Math.Max(3, maxExpectedEvents / 3)) + { + var denseStep = Math.Max(1, sampleRate / 250); + for (var offset = 0; offset <= samples.Length - motifSamples; offset += denseStep) + { + var frame = ClassifyAt(samples, sampleRate, offset, denseStep); + if (frame != null) + { + frames.Add(frame); + } + } + } + + return SuppressNearbyFrames(frames, sampleRate); + } + + private static MimirChirpletTransformFrame? ClassifyAt( + ReadOnlySpan samples, + int sampleRate, + int predictedOffset, + int searchRadiusSamples) + { + var motifSamples = Math.Max(1, (int)Math.Round(MotifDurationSeconds * sampleRate)); + var start = Math.Max(0, predictedOffset - searchRadiusSamples); + var end = Math.Min(samples.Length - motifSamples, predictedOffset + searchRadiusSamples); + if (start > end) + { + return null; + } + + var bestOffset = predictedOffset; + MimirChirpletSymbolCandidate[] bestCandidates = []; + var bestEnergy = 0.0; + var step = Math.Max(1, sampleRate / 8_000); + for (var offset = start; offset <= end; offset += step) + { + var candidates = ScoreMotifs(samples.Slice(offset, motifSamples), sampleRate, offset); + var energy = candidates.Length == 0 ? 0.0 : candidates[0].Energy; + if (energy > bestEnergy) + { + bestEnergy = energy; + bestOffset = offset; + bestCandidates = candidates; + } + } + + if (bestCandidates.Length == 0 || bestEnergy < 0.075) + { + return null; + } + + var refinedOffset = RefineOffset(samples, sampleRate, bestOffset, bestCandidates[0].SymbolId, motifSamples); + return new MimirChirpletTransformFrame( + refinedOffset, + bestCandidates + .Select(candidate => candidate with { SampleOffset = refinedOffset }) + .OrderByDescending(candidate => candidate.Energy) + .ToArray()); + } + + private static MimirChirpletSymbolCandidate[] ScoreMotifs( + ReadOnlySpan samples, + int sampleRate, + double sampleOffset) + { + var kernels = KernelSets.GetOrAdd(sampleRate, BuildKernelSet); + var sampleEnergy = 0.0; + for (var index = 0; index < samples.Length; index++) + { + sampleEnergy += samples[index] * samples[index]; + } + + if (sampleEnergy <= 1.0e-12) + { + return []; + } + + var candidates = new List(Motifs.Length); + for (var symbol = 0; symbol < kernels.Symbols.Length; symbol++) + { + var kernel = kernels.Symbols[symbol]; + var dot = 0.0; + for (var index = 0; index < samples.Length; index++) + { + dot += samples[index] * kernel.Samples[index]; + } + + var energy = Math.Abs(dot) / Math.Sqrt(sampleEnergy * kernel.Energy); + var bandResponses = kernel.CenterFrequencies + .Select(center => new MimirChirpletBandResponse(center, energy)) + .ToArray(); + candidates.Add(new MimirChirpletSymbolCandidate(symbol, sampleOffset, energy, bandResponses)); + } + + return candidates + .OrderByDescending(candidate => candidate.Energy) + .Take(MaxSymbolCandidatesPerFrame) + .ToArray(); + } + + private static double RefineOffset( + ReadOnlySpan samples, + int sampleRate, + int bestOffset, + int symbolId, + int motifSamples) + { + if (bestOffset <= 0 || bestOffset >= samples.Length - motifSamples - 1) + { + return bestOffset; + } + + var left = ScoreSymbolEnergy(samples.Slice(bestOffset - 1, motifSamples), sampleRate, symbolId); + var center = ScoreSymbolEnergy(samples.Slice(bestOffset, motifSamples), sampleRate, symbolId); + var right = ScoreSymbolEnergy(samples.Slice(bestOffset + 1, motifSamples), sampleRate, symbolId); + var denominator = left - 2.0 * center + right; + if (Math.Abs(denominator) <= 1.0e-12) + { + return bestOffset; + } + + return bestOffset + Math.Clamp(0.5 * (left - right) / denominator, -0.5, 0.5); + } + + private static double ScoreSymbolEnergy(ReadOnlySpan samples, int sampleRate, int symbolId) + { + var kernels = KernelSets.GetOrAdd(sampleRate, BuildKernelSet); + var kernel = kernels.Symbols[symbolId]; + var sampleEnergy = 0.0; + var dot = 0.0; + for (var index = 0; index < samples.Length; index++) + { + sampleEnergy += samples[index] * samples[index]; + dot += samples[index] * kernel.Samples[index]; + } + + return sampleEnergy <= 1.0e-12 || kernel.Energy <= 1.0e-12 + ? 0.0 + : Math.Abs(dot) / Math.Sqrt(sampleEnergy * kernel.Energy); + } + + private static IReadOnlyList DecodeAnchors( + IReadOnlyList frames, + int sampleRate) + { + var plan = GetTimelinePlan(); + if (frames.Count < plan.Order) + { + return []; + } + + var candidates = new List(); + for (var index = 0; index <= frames.Count - plan.Order; index++) + { + var window = frames.Skip(index).Take(plan.Order).ToArray(); + var consecutive = true; + for (var gap = 0; gap < window.Length - 1; gap++) + { + if (!IsConsecutive(window[gap], window[gap + 1], sampleRate)) + { + consecutive = false; + break; + } + } + + if (!consecutive) + { + continue; + } + + foreach (var candidatePath in CandidateSymbolPaths(window)) + { + var code = CodeKey(candidatePath.Select(candidate => candidate.SymbolId)); + if (!plan.CodeToIndex.TryGetValue(code, out var eventIndex)) + { + continue; + } + + var firstEvent = Default.EventForIndex((ulong)eventIndex); + var gapError = 0.0; + for (var gap = 0; gap < candidatePath.Count - 1; gap++) + { + var measuredGap = (candidatePath[gap + 1].SampleOffset - candidatePath[gap].SampleOffset) / sampleRate; + gapError += Math.Abs(measuredGap - EventSpacingSeconds); + } + + var gapConfidence = 1.0 / (1.0 + gapError / 0.006); + var energyConfidence = Math.Clamp(candidatePath.Average(candidate => candidate.Energy), 0.0, 1.0); + candidates.Add(new MimirChirpletTimelineAnchor( + (ulong)eventIndex, + firstEvent.StartSeconds, + candidatePath[0].SampleOffset, + gapConfidence * 0.35 + energyConfidence * 0.65, + candidatePath + .Select(candidate => new MimirChirpletSymbolObservation( + candidate.SymbolId, + candidate.SampleOffset, + candidate.Energy)) + .ToArray())); + } + } + + return SelectCoherentAnchorPath(candidates, sampleRate); + } + + private static IEnumerable> CandidateSymbolPaths( + IReadOnlyList frames) + { + var path = new MimirChirpletSymbolCandidate[frames.Count]; + IEnumerable> Recurse(int index) + { + if (index == frames.Count) + { + yield return path.ToArray(); + yield break; + } + + foreach (var candidate in frames[index].Candidates) + { + path[index] = candidate; + foreach (var result in Recurse(index + 1)) + { + yield return result; + } + } + } + + return Recurse(0); + } + + private static IReadOnlyList SelectCoherentAnchorPath( + IReadOnlyList candidates, + int sampleRate) + { + if (candidates.Count == 0) + { + return []; + } + + List bestPath = []; + var bestScore = double.NegativeInfinity; + foreach (var seedOffset in candidates + .OrderByDescending(anchor => anchor.Confidence) + .Take(64) + .Select(anchor => anchor.SampleOffset - anchor.TimelineSeconds * sampleRate)) + { + var path = candidates + .Where(anchor => Math.Abs((anchor.SampleOffset - anchor.TimelineSeconds * sampleRate) - seedOffset) <= sampleRate * 0.010) + .GroupBy(anchor => anchor.EventIndex) + .Select(group => group.OrderByDescending(anchor => anchor.Confidence).First()) + .OrderBy(anchor => anchor.EventIndex) + .ToList(); + var clock = FitClock(path, sampleRate); + if (clock == null) + { + continue; + } + + var score = path.Sum(anchor => anchor.Confidence) + + Math.Min(path.Count, 12) * 0.20 - + clock.MeanAbsoluteErrorSamples / Math.Max(1.0, sampleRate * 0.0015); + if (score > bestScore) + { + bestScore = score; + bestPath = path; + } + } + + var finalClock = FitClock(bestPath, sampleRate); + if (finalClock == null) + { + return []; + } + + return bestPath + .Where(anchor => Math.Abs(anchor.SampleOffset - finalClock.SampleForTimelineSeconds(anchor.TimelineSeconds)) <= Math.Max(24.0, sampleRate * 0.002)) + .OrderBy(anchor => anchor.SampleOffset) + .ToArray(); + } + + private static MimirChirpletClockFit? FitClock( + IReadOnlyList anchors, + int sampleRate) + { + if (anchors.Count == 0) + { + return null; + } + + if (anchors.Count == 1) + { + var anchor = anchors[0]; + return new MimirChirpletClockFit( + anchor.SampleOffset - anchor.TimelineSeconds * sampleRate, + sampleRate, + anchor.Confidence * 0.35, + 1, + 0.0); + } + + var totalWeight = anchors.Sum(anchor => Math.Max(1.0e-6, anchor.Confidence)); + var meanTimeline = anchors.Sum(anchor => anchor.TimelineSeconds * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var meanSample = anchors.Sum(anchor => anchor.SampleOffset * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var covariance = 0.0; + var variance = 0.0; + foreach (var anchor in anchors) + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var dt = anchor.TimelineSeconds - meanTimeline; + covariance += weight * dt * (anchor.SampleOffset - meanSample); + variance += weight * dt * dt; + } + + var effectiveSampleRate = variance > 1.0e-12 ? covariance / variance : sampleRate; + if (!double.IsFinite(effectiveSampleRate) || + effectiveSampleRate < sampleRate * 0.98 || + effectiveSampleRate > sampleRate * 1.02) + { + effectiveSampleRate = sampleRate; + } + + var sourceOffset = meanSample - effectiveSampleRate * meanTimeline; + var meanAbsoluteError = anchors.Sum(anchor => + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var predicted = sourceOffset + anchor.TimelineSeconds * effectiveSampleRate; + return Math.Abs(anchor.SampleOffset - predicted) * weight; + }) / totalWeight; + var residualConfidence = 1.0 / (1.0 + meanAbsoluteError / Math.Max(1.0, sampleRate * 0.001)); + var countConfidence = Math.Clamp(anchors.Count / 12.0, 0.0, 1.0); + var anchorConfidence = Math.Clamp(anchors.Average(anchor => anchor.Confidence), 0.0, 1.0); + return new MimirChirpletClockFit( + sourceOffset, + effectiveSampleRate, + residualConfidence * 0.45 + countConfidence * 0.25 + anchorConfidence * 0.30, + anchors.Count, + meanAbsoluteError); + } + + private static IReadOnlyList EstimateBandResponse(IReadOnlyList frames) + { + return frames + .SelectMany(frame => frame.BestCandidate.BandResponses ?? []) + .GroupBy(response => response.CenterHz) + .Select(group => new MimirChirpletBandResponse(group.Key, group.Average(response => response.Energy))) + .OrderBy(response => response.CenterHz) + .ToArray(); + } + + private static double RefineSourceOffset(ReadOnlySpan samples, int sampleRate, double initialOffsetSamples) + { + var radius = Math.Max(4, (int)Math.Ceiling(sampleRate * 0.00010)); + var center = (int)Math.Round(initialOffsetSamples); + var firstOffset = center - radius; + var lastOffset = center + radius; + var bestOffset = center; + var bestScore = double.NegativeInfinity; + for (var offset = firstOffset; offset <= lastOffset; offset++) + { + var score = ScheduledWaveformScore(samples, sampleRate, offset); + if (score > bestScore) + { + bestScore = score; + bestOffset = offset; + } + } + + if (bestOffset <= firstOffset || bestOffset >= lastOffset) + { + return bestOffset; + } + + var left = ScheduledWaveformScore(samples, sampleRate, bestOffset - 1); + var middle = ScheduledWaveformScore(samples, sampleRate, bestOffset); + var right = ScheduledWaveformScore(samples, sampleRate, bestOffset + 1); + var denominator = left - 2.0 * middle + right; + if (Math.Abs(denominator) <= 1.0e-12) + { + return bestOffset; + } + + return bestOffset + Math.Clamp(0.5 * (left - right) / denominator, -0.5, 0.5); + } + + private static double ScheduledWaveformScore(ReadOnlySpan samples, int sampleRate, int sourceOffsetSamples) + { + var timelineStartSeconds = -sourceOffsetSamples / (double)sampleRate; + var timelineDurationSeconds = samples.Length / (double)sampleRate; + var reference = new float[samples.Length]; + foreach (var timelineEvent in Default.EventsOverlapping(timelineStartSeconds, timelineDurationSeconds)) + { + AddMotif(reference, sampleRate, timelineEvent, timelineStartSeconds, 1.0); + } + + var dot = 0.0; + var sampleEnergy = 0.0; + var referenceEnergy = 0.0; + for (var index = 0; index < samples.Length; index++) + { + dot += samples[index] * reference[index]; + sampleEnergy += samples[index] * samples[index]; + referenceEnergy += reference[index] * reference[index]; + } + + return sampleEnergy <= 1.0e-12 || referenceEnergy <= 1.0e-12 + ? double.NegativeInfinity + : dot / Math.Sqrt(sampleEnergy * referenceEnergy); + } + + private static float[] BuildWindowEnergyTrace(ReadOnlySpan samples, int windowSamples, int hopSamples) + { + var output = new float[1 + (samples.Length - windowSamples) / hopSamples]; + var prefix = new double[samples.Length + 1]; + for (var index = 0; index < samples.Length; index++) + { + prefix[index + 1] = prefix[index] + samples[index] * samples[index]; + } + + for (var frame = 0; frame < output.Length; frame++) + { + var offset = frame * hopSamples; + output[frame] = (float)((prefix[offset + windowSamples] - prefix[offset]) / windowSamples); + } + + return output; + } + + private static double AdaptiveThreshold(float[] energyTrace) + { + if (energyTrace.Length == 0) + { + return double.PositiveInfinity; + } + + var mean = energyTrace.Average(value => (double)value); + var variance = energyTrace.Sum(value => (value - mean) * (value - mean)) / energyTrace.Length; + return mean + Math.Sqrt(Math.Max(0.0, variance)) * 0.18; + } + + private static IReadOnlyList SuppressNearbyFrames( + IReadOnlyList frames, + int sampleRate) + { + var minimumSpacingSamples = EventSpacingSeconds * sampleRate * 0.55; + var kept = new List(); + foreach (var frame in frames.OrderByDescending(frame => frame.BestCandidate.Energy)) + { + if (kept.Any(existing => Math.Abs(existing.SampleOffset - frame.SampleOffset) < minimumSpacingSamples)) + { + continue; + } + + kept.Add(frame); + } + + return kept.OrderBy(frame => frame.SampleOffset).ToArray(); + } + + private static bool IsConsecutive( + MimirChirpletTransformFrame first, + MimirChirpletTransformFrame second, + int sampleRate) + { + var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate; + return seconds >= EventSpacingSeconds * 0.62 && seconds <= EventSpacingSeconds * 1.38; + } + + private static void AddMotif( + float[] samples, + int sampleRate, + MimirChirpletTimelineEvent timelineEvent, + double segmentStartSeconds, + double gain) + { + var motif = Motifs[timelineEvent.SymbolId]; + foreach (var syllable in motif.Syllables) + { + AddSyllable(samples, sampleRate, timelineEvent.StartSeconds - segmentStartSeconds, syllable, gain); + } + } + + private static void AddSyllable( + float[] samples, + int sampleRate, + double motifStartSeconds, + MimirBioacousticSyllable syllable, + double gain) + { + var startFrame = (int)Math.Round((motifStartSeconds + syllable.StartSeconds) * sampleRate); + var frameCount = Math.Max(1, (int)Math.Round(syllable.DurationSeconds * sampleRate)); + var slope = Math.Log(syllable.EndHz / syllable.StartHz) / syllable.DurationSeconds; + for (var frame = 0; frame < frameCount; frame++) + { + var outputFrame = startFrame + frame; + if (outputFrame < 0 || outputFrame >= samples.Length) + { + continue; + } + + var normalized = frameCount <= 1 ? 1.0 : frame / (double)(frameCount - 1); + var t = frame / (double)sampleRate; + var envelope = RaisedCosineEnvelope(normalized); + var instantaneous = syllable.StartHz * Math.Exp(slope * t); + var phase = 2.0 * Math.PI * syllable.StartHz * (Math.Exp(slope * t) - 1.0) / slope; + var voiced = Math.Sin(phase); + voiced += 0.42 * Math.Sin(phase * 1.5 + 0.35); + voiced += 0.24 * Math.Sin(phase * 2.0 + 0.72); + voiced += 0.10 * Math.Sin(phase * 2.75 + 1.10); + var air = Math.Sin(2.0 * Math.PI * instantaneous * t * 0.071 + syllable.StartHz * 0.0003); + samples[outputFrame] += (float)((voiced + 0.08 * air) * envelope * syllable.Weight * gain); + } + } + + private static double RaisedCosineEnvelope(double normalized) => + 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * Math.Clamp(normalized, 0.0, 1.0)); + + private static BioacousticKernelSet BuildKernelSet(int sampleRate) + { + var motifSamples = Math.Max(1, (int)Math.Round(MotifDurationSeconds * sampleRate)); + var kernels = new BioacousticKernel[Motifs.Length]; + for (var symbol = 0; symbol < Motifs.Length; symbol++) + { + var samples = new float[motifSamples]; + AddMotif( + samples, + sampleRate, + new MimirChirpletTimelineEvent(0, symbol, 0.0, new MimirChirpletTone(0.0, MotifDurationSeconds, 0.0, 0.0, 1.0)), + 0.0, + 1.0); + var kernel = new double[motifSamples]; + var energy = 0.0; + for (var index = 0; index < motifSamples; index++) + { + kernel[index] = samples[index]; + energy += kernel[index] * kernel[index]; + } + + kernels[symbol] = new BioacousticKernel( + kernel, + Motifs[symbol].Syllables.Select(s => Math.Sqrt(s.StartHz * s.EndHz)).ToArray(), + Math.Max(energy, 1.0e-12)); + } + + return new BioacousticKernelSet(kernels); + } + + private static MimirBioacousticMotifDefinition[] BuildMotifs() + { + var contourBank = new[] + { + new[] { 1.00, 1.24, 1.10, 1.42, 1.16, 1.04 }, + new[] { 1.00, 0.86, 1.18, 1.08, 0.94, 1.30 }, + new[] { 1.00, 1.36, 1.58, 1.28, 1.08, 1.22 }, + new[] { 1.00, 1.10, 0.82, 1.02, 1.34, 1.56 }, + }; + var rhythmOffsets = new[] + { + new[] { 0.000, 0.039, 0.079 }, + new[] { 0.000, 0.032, 0.083 }, + new[] { 0.000, 0.046, 0.073 }, + new[] { 0.000, 0.035, 0.069 }, + }; + var motifs = new MimirBioacousticMotifDefinition[SymbolCount]; + var logLowest = Math.Log(LowestRootHz); + var logStep = Math.Log(HighestRootHz / LowestRootHz) / (SymbolCount - 1); + for (var symbol = 0; symbol < motifs.Length; symbol++) + { + var root = Math.Exp(logLowest + symbol * logStep); + var contour = contourBank[symbol % contourBank.Length]; + var rhythm = rhythmOffsets[(symbol / contourBank.Length) % rhythmOffsets.Length]; + var syllables = new MimirBioacousticSyllable[3]; + for (var syllable = 0; syllable < syllables.Length; syllable++) + { + var start = root * contour[syllable * 2]; + var end = root * contour[syllable * 2 + 1]; + syllables[syllable] = new MimirBioacousticSyllable( + rhythm[syllable], + SyllableSeconds, + Math.Clamp(start, 2_700.0, 13_500.0), + Math.Clamp(end, 2_700.0, 13_500.0), + 1.0 - syllable * 0.10); + } + + motifs[symbol] = new MimirBioacousticMotifDefinition(symbol, syllables); + } + + return motifs; + } + + private static int SymbolForEvent(ulong eventIndex) + { + var plan = GetTimelinePlan(); + return plan.TimelineSymbols[(int)(eventIndex % (ulong)plan.TimelineSymbols.Length)]; + } + + private static double EventStartSeconds(ulong eventIndex) => + FirstEventSeconds + eventIndex * EventSpacingSeconds; + + private static BioacousticTimelinePlan GetTimelinePlan() => + TimelinePlans.GetOrAdd("default", _ => new BioacousticTimelinePlan( + Enumerable.Range(0, SymbolCount).ToArray(), + TimelineOrder, + TimelineSymbols, + BuildCodeIndex(TimelineSymbols, TimelineOrder))); + + private static int[] BuildDeBruijn(int alphabetSize, int order) + { + var a = new int[alphabetSize * order]; + var sequence = new List((int)Math.Pow(alphabetSize, order)); + + void Db(int t, int p) + { + if (t > order) + { + if (order % p == 0) + { + for (var index = 1; index <= p; index++) + { + sequence.Add(a[index]); + } + } + + return; + } + + a[t] = a[t - p]; + Db(t + 1, p); + for (var j = a[t - p] + 1; j < alphabetSize; j++) + { + a[t] = j; + Db(t + 1, t); + } + } + + Db(1, 1); + return sequence.ToArray(); + } + + private static int[] RotateToDistinctOpening(int[] sequence) + { + for (var index = 0; index < sequence.Length; index++) + { + var first = sequence[index]; + var second = sequence[(index + 1) % sequence.Length]; + var third = sequence[(index + 2) % sequence.Length]; + if (first != second && first != third && second != third) + { + return sequence.Skip(index).Concat(sequence.Take(index)).ToArray(); + } + } + + return sequence; + } + + private static Dictionary BuildCodeIndex(IReadOnlyList symbols, int order) + { + var map = new Dictionary(symbols.Count, StringComparer.Ordinal); + for (var index = 0; index < symbols.Count; index++) + { + var code = CodeKey(Enumerable.Range(0, order).Select(offset => symbols[(index + offset) % symbols.Count])); + if (!map.ContainsKey(code)) + { + map[code] = index; + } + } + + return map; + } + + private static string CodeKey(IEnumerable symbols) => + string.Join(",", symbols); +} diff --git a/state/map.yaml b/state/map.yaml index 2b6b3b7..e37e458 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -64,9 +64,15 @@ accepted_design: JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the final local-device hot path. + - `MimirBioacousticTimeline` is the active runtime watermark. It renders a + low-gain birdsong-like phrase language with 16 log-spaced motif symbols, + three formant-rich syllables per motif, rhythm variation, and an order-3 + de Bruijn grammar. `MimirAudioSynchronizationAnalyzer` now reports active + timing as `evidence=bioacoustic`; the old chirp-bin path remains a + calibration/reference artifact with ASIO tooling. - The diagnostic audio path can optionally carry base64 float sample payloads - for sync smoke tests. `MimirChirpletTimeline` owns the structured - birdsong-like timeline fingerprint: an order-3 de Bruijn sequence over 32 + for sync smoke tests. `MimirChirpletTimeline` owns the older structured + chirplet reference fingerprint: an order-3 de Bruijn sequence over 32 time/frequency constellation symbols, plus PCM segment rendering, matched timing atoms, symbol/event decoding, and per-band response kernels. Symbol identity is carried by start band, glide shape, duration, and following @@ -78,7 +84,7 @@ accepted_design: transform frames with multiple phase-invariant symbol candidates and refined per-candidate sample offsets, decodes code-valid triplet anchors through gap/clock coherence, and fits a per-source sample clock from those anchors. - The canonical synthetic smoke path is `dotnet run --project + Its synthetic smoke path is `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`. It currently decodes all 15 emitted chirps in a two-second window into 13 anchors for events 0-12 with a 47999.999990 Hz clock fit and 0.000014 sample @@ -102,38 +108,31 @@ accepted_design: windows before any watermark is needed. Negative passive lags are contradictory evidence and carry zero confidence; single-window passive confidence is capped below certainty until repeated state earns it. - `MimirChirpBinTimeline` is the active calibration path for both chirp-only - and hybrid: one fixed chirp slope/window, 32 widened symbol frequency bins, - cheap bounded energy/onset proposals, dechirp plus fixed Goertzel bin - classification, explicit time/frequency ambiguity candidates, then the - existing de Bruijn triplet anchor and clock-fit machine. The analyzer refines - final active delay with constrained local waveform correlation around the - decoded anchor delay. Each classified chirp carries the full dechirped - bin-energy surface, and decodes aggregate that into per-band response - evidence for frequency-response normalization. `MimirChirpBinCalibrationModel` - now preserves measured usable bands, expected-symbol versus observed-bin - confusion observations, timing residuals, delay hypotheses, phase summaries, - and an adaptive codebook plan per output/mic path. The active decoder can - consume a persisted model for learned response weighting, phase-coherence - weighting, first-order group-delay correction, and joint global - delay/bin-shift hypotheses. Runtime chirp-bin emission consumes the - same model's emission plan, so Mimir can render the measured reliable symbol - alphabet at a higher de Bruijn order instead of spending code on dead bins. - The analyzer keeps calibration profiles - even when a source fails pairwise timing, so physical mic failures can still - guide acoustic codebook and response-weighting work. Reports and sync states expose - delay in microseconds as well as fractional samples. Hybrid emits it as low-gain half-second - bursts every two seconds while passive confidence is weak, never as catch-up - backlog. The analyzer no longer applies the passive two-second window floor - to active fallback; `dotnet run --project - .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 192000` - and `--hybrid-sync-self-test --sample-rate 192000` both recover a - 1269.5-sample synthetic delay with printed 0.000 us error. The synthetic - standalone receiver proof `--standalone-chirp-bin-self-test --sample-rate - 192000 --delay-samples 96000` recovers a 500 ms delayed stream below printed + `MimirBioacousticTimeline` is now the active calibration witness for both + chirp-only and hybrid: 16 log-spaced motif symbols, three formant-rich + syllables per motif, rhythm variation, and an order-3 de Bruijn grammar so + any correctly decoded consecutive triplet identifies one canonical timeline + location. The analyzer refines final active delay with constrained local + waveform correlation around the decoded anchor delay. Each detected motif + carries per-band response evidence, and the next calibration cut should use + those profiles to tune contours, formants, usable bands, gain, and rhythm + weighting. `MimirChirpBinTimeline` remains the controlled chirp-bin + calibration/reference artifact with ASIO tooling and + `MimirChirpBinCalibrationModel`; keep those captures as response evidence, + not the active runtime sound. Reports and sync states expose delay in + microseconds as well as fractional samples. Hybrid emits the bioacoustic + witness as low-gain half-second bursts while passive confidence is weak, + never as catch-up backlog. The analyzer no longer applies the passive + two-second window floor to active fallback; `dotnet run --project + .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 48000` + recovers a 317.375-sample synthetic delay with printed 0.000 us error using + `evidence=bioacoustic`, while `--hybrid-sync-self-test --sample-rate 48000` + correctly accepts strong passive evidence at 2.772 us error. The synthetic + standalone receiver proof `--standalone-bioacoustic-self-test --sample-rate + 48000 --delay-samples 1269.5` recovers delayed source time below printed microsecond precision using only the codebook/schedule. The synthetic - codebook proof remains `dotnet run --project - .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test`. + codebook proof is `dotnet run --project + .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`. `Mimir.BufferSmoke --render-chirp-bin-f32` plus `native/probes/asio_audio_cadence --play-f32-mono --record-f32-interleaved` now provides a real Scarlett ASIO artifact proof for active mode. At 192 kHz, @@ -141,7 +140,7 @@ accepted_design: confidence. `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` now renders or ingests a calibration capture and persists the response/confusion/delay model under `calibration/chirp-bin/`; `--analyze-asio-f32 --calibration` - loads it back into the active decoder. Physical inputs still fail accepted + loads it back into the chirp-bin reference decoder. Physical inputs still fail accepted pairwise timing against loopback in the stored artifact, but decoded mic channels now leave calibration models; physical input 1 produced two reliable symbols and a different strongest measured band set. @@ -270,23 +269,23 @@ current_status: Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > - Use the new study index to cut the next implementation in this order: - streaming calibration-weighted chirp-bin decoder state, calibration-session - command/replay proof, fractional-delay/SRO actuator, then native camera - payload handles and Fensalir lowering. Use chirp-bin calibration models from - real microphones to tune the active codebook/bands/weighting, then add the - synchronization actuator. + Use the new bioacoustic timeline watermark as the active acoustic sync + direction. Cut the next implementation in this order: streaming log-mel + motif decoder state, calibration-session command/replay proof that learns + per-output/mic motif/formant response, fractional-delay/SRO actuator, then + native camera payload handles and Fensalir lowering. Keep chirp-bin + calibration captures as reference evidence, not the runtime target. Current app testing proves Fensalir WASAPI output wakes Scarlett speaker loopback, live mic buffers stay populated, runtime sync analysis now runs as - a bounded rotating service without UI/telemetry re-entry, and the continuous - chirplet timeline can produce confident sync states in the app. The next - invariant is code-valid triplets producing stable canonical anchors and clock - fits, so the actuator can use decoded time instead of filtered lag guesses. - Replace - the remaining diagnostic JSON-line process bridges with direct native capture drivers that push buffer handles into - Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports + a bounded rotating service without UI/telemetry re-entry, and the bioacoustic + motif timeline produces deterministic synthetic active sync. The next + invariant is code-valid motif triplets producing stable canonical anchors and + clock fits through real loopback and microphones, so the actuator can use + decoded time instead of filtered lag guesses. Replace the remaining + diagnostic JSON-line process bridges with direct native capture drivers that + push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when - calibration chirplets are playing. Treat PS3 Eye audio as + calibration witnesses are playing. Treat PS3 Eye audio as enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from the Leap USB neighborhood. The Scarlett Solo 4th Gen is now attached to Starfire, and `native/probes/asio_audio_cadence` proves local Focusrite USB