From f134ea6bca76863c4ba7794a12b79b432d3d5937 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 24 May 2026 23:31:41 +0100 Subject: [PATCH] Make song contours the active sync authority --- docs/audio-field.md | 58 ++++++++----------- docs/bioacoustic-timeline-watermark.md | 58 ++++++++++--------- docs/code-algorithm-map.md | 7 ++- docs/implementation-plan.md | 18 +++--- notes/current-system-map.md | 10 +++- notes/fresh-workspace-handoff.md | 22 ++++--- .../MimirBioacousticTimeline.cs | 1 - state/evidence.jsonl | 1 + state/map.yaml | 21 ++++--- 9 files changed, 108 insertions(+), 88 deletions(-) diff --git a/docs/audio-field.md b/docs/audio-field.md index 6646b04..fb48f70 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -184,13 +184,13 @@ half-second PCM segments ahead of the audio cursor. Each motif is a short phrase of formant-rich syllables with log-frequency contour and rhythm offsets, so the code is carried by both spectral shape and timing. -The intended decoder is a compact constrained motif transform, not a generic -time-frequency explorer and not an outlier filter around bad guesses. Mimir owns -the emitter, so the receiver projects each mic stream against the known motif -dictionary, produces transform frames with multiple word candidates, and maps -the best coherent word identities directly to canonical events. A word becomes a -canonical timeline anchor when its motif likelihood and neighboring anchors -agree on one local sample clock. A decoded anchor +The intended decoder is a compact constrained song-contour transform, not a +generic time-frequency explorer and not an outlier filter around bad guesses. +Mimir owns the emitter, so the receiver keeps its ear open for known song words +and maps each successful call directly to canonical time. A word is not a +single scalar event: syllable onsets, bends, formants, rhythm offsets, payload +ornaments, speaker tint, and log-mel contour all expose local time/frequency +constraints. One call can pin a cluster of anchors at once. A decoded anchor means observed sample offset `S` corresponds to emitted event time `T`. A stream of anchors fits the source clock directly: @@ -201,14 +201,15 @@ observed_sample = source_offset + canonical_seconds * effective_sample_rate Delay and sampling-rate offset are derived by comparing the loopback clock fit to each mic clock fit over common canonical time. The state tracker is not allowed to launder invalid words into plausible timing. If a stream cannot -produce at least one matched canonical anchor, it has not decoded timing for +produce at least one matched canonical song word, it has not decoded timing for that window. Reports carry rounded integer delay, fractional delay, and the -count/confidence of timeline-symbol anchors used to derive the report. +count/confidence of contour anchors used to derive the report. -The same timeline starts the frequency-response path. Each active bioacoustic -decode carries per-band response evidence for the source set, whether or not -each source earns a timing report. The older chirp-bin calibration model remains -the controlled ASIO reference surface for response/confusion/delay captures: +The same song starts the frequency-response path. Each active bioacoustic decode +carries per-band and per-contour response evidence for the source set, whether +or not each source earns a timing report. The older chirp-bin calibration model +remains the controlled ASIO reference surface for response/confusion/delay +captures: use `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` to render that reference timeline, optionally capture it through the ASIO worker with `--capture-asio`, compute the response/confusion/delay model, and persist it for decoder tuning. @@ -225,10 +226,10 @@ bioacoustic timeline queued, polls sources, and updates sync analysis on a fixed `MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current runtime testing proves Fensalir output wakes the Scarlett loopback and the mic buffers stay live. The next failure is physical capture proof: the same -canonical motif anchors need to stay stable through loopback, room mics, device +canonical song-contour anchors need to stay stable through loopback, room mics, device clocks, and codec/network paths. -## Chirplet Calibration Model +## Retained Reference Calibration ```mermaid flowchart TD @@ -237,7 +238,7 @@ flowchart TD B --> D["room + speakers + mics"] C --> E["loopback rolling buffer"] D --> F["mic rolling buffers"] - E --> G["matched chirplet traces"] + E --> G["matched reference traces"] F --> G G --> H["symbol likelihood events"] H --> L["timeline anchors"] @@ -248,7 +249,7 @@ flowchart TD I --> K["frequency response normalization"] ``` -The chirplet timeline owns three facts: +The retained chirplet/chirp-bin timelines own three reference facts: - **Emission**: the PCM that Fensalir sends to the speakers. - **Timing witness**: the matched-filter atom bank used to find the stream in @@ -256,7 +257,7 @@ The chirplet timeline owns three facts: - **Response witness**: the per-band atoms used to measure how strongly each mic hears each emitted band. -Continuous chirplet evidence gives both the current delay and the drift/SRO by +Continuous active song evidence should give both the current delay and the drift/SRO by watching delay change over time. Per-band energy over the same stream gives the normalization curve. The important constraint is that all three measurements must be tied to the same emitted timeline, not three separately invented probes. @@ -267,21 +268,12 @@ band set but still fails pairwise timing because no canonical events overlap the loopback path. That is not a blank failure anymore; it is calibration data for the next codebook/weighting cut. -The symbol layer is intentionally redundant. It does not rely on one fixed -frequency shelf: timing gaps, chirp duration, start band, and glide shape all -contribute so poor mic frequency response does not erase the whole code. -`MimirChirpletSymbolCodebook` owns the symbol definitions so timeline ordering -does not smuggle acoustic shape decisions into bit arithmetic. Every symbol has -a unique chirp shape; rhythm remains additional evidence, not a substitute for -symbol separability. The transform uses sine/cosine chirplet kernels for each -symbol, so timing does not depend on the receiver preserving the emitter's -absolute phase. Run `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test` -to render two seconds of canonical timeline audio and decode it back to -timeline anchors. Current synthetic proof detects all 15 emitted chirps, keeps -13 possible triplet anchors for events 0 through 12, fits the clock at -47999.999990 Hz, and holds mean absolute anchor error to 0.000014 samples. Real -device runs still depend on loopback capture staying live; the local Scarlett -loopback has intermittently stopped advancing during short headless sniffs. +The active symbol layer is now the song contour. It does not rely on one fixed +frequency shelf or a coded sequence: syllable timing, contour bends, formants, +payload ornaments, speaker tint, and local spectral flux all contribute so poor +mic response does not erase the whole code. The old chirplet self-test remains +available as historical proof of sub-frame math, but it must not retake +authority over the active runtime path. The older `MimirChirpletTimeline` matcher is a diagnostic/reference artifact, not the active runtime path. `BuildChirpletEnergyTrace` still behaves like a diff --git a/docs/bioacoustic-timeline-watermark.md b/docs/bioacoustic-timeline-watermark.md index ce5ac4b..e22c4b4 100644 --- a/docs/bioacoustic-timeline-watermark.md +++ b/docs/bioacoustic-timeline-watermark.md @@ -12,9 +12,10 @@ ask only "which bin fired?" It should ask: ```text which motif contour did I hear? -which self-identifying word did that contour encode? +which self-identifying song word did that contour encode? which speaker vocabulary emitted it? -what delay and clock fit make the heard words coherent? +which time and frequency anchors did that one call expose? +what delay and clock fit make the heard anchors coherent? what did this room/mic path do to the spectrum? ``` @@ -27,10 +28,15 @@ what did this room/mic path do to the spectrum? - Each word contains four short syllables. - Each syllable carries a log-frequency glide plus formant-like partials. - Motif rhythm, contour, syllable duration, root, and speaker tint vary by word. -- No de Bruijn grammar is used in the active bioacoustic path. A word identifies - its canonical event directly inside the current operating horizon. -- The decoder emits motif candidates, direct word anchors, a source clock fit, - and per-band response evidence. +- No de Bruijn grammar is used in the active bioacoustic path. One heard song + word identifies its canonical event directly inside the current operating + horizon. +- The decoder emits motif candidates, direct song-word anchors, a source clock + fit, and per-band response evidence. +- A song word is not one scalar chirp. It is a contour packet: multiple + syllable onsets, bends, formants, rhythm offsets, payload ornaments, and + speaker-colored spectral features all become anchors in log-mel space. One + successful call can pin a cluster of time/frequency constraints at once. - Runtime emission sends the left vocabulary to the left speaker and the right vocabulary to the right speaker, so microphones can learn each side of the room separately. @@ -45,11 +51,11 @@ watermark target. `MimirBioacousticTimeline` owns: -- motif codebook shape; +- song-word codebook shape; - deterministic phrase schedule; - PCM rendering for active emission; - streaming motif scoring; -- direct word anchor decoding; +- direct contour-anchor decoding; - source clock fitting; - response-band evidence. @@ -72,7 +78,7 @@ Birdsong survives ugly channels because the information is redundant: - formant envelope survives when one band dies; - speaker-specific variants expose room asymmetry; - repeated local structure gives the receiver more than one way to recover time - and response. + response, and local payload identity. That is the physical path Mimir needs. A phone mic, camera mic, Scarlett input, or network receiver should be able to recognize canonical time from a damaged @@ -112,34 +118,34 @@ dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --rende ## Next Cut -The current runtime decoder still uses a motif matched-filter proof. The smoke -harness now carries the next receiver shape: indexed MFCC/log-mel word -fingerprints with path-degradation augmentation. The next coherent cut is to -promote that shape into a true streaming log-mel receiver: +The current physical receipt path uses the canary packet song as the first +serious contour-anchor witness. The next coherent cut is to promote that shape +into the live streaming log-mel receiver: ```text audio window -> log-mel / constant-Q evidence surface --> motif/formant/contour candidates --> projection-hash / ANN candidate retrieval --> direct word identity decode over a tiny candidate set --> global delay and clock hypothesis +-> contour packet proposals +-> direct song-word identity over a tiny candidate set +-> intra-call time/frequency anchor extraction +-> global delay, clock, and path hypothesis -> path response model ``` Do not add a pile of threshold rules to make one room capture look good. The -right extension is learned path weighting over motif parts: which contours, -partials, and time gaps survive each output/mic path. +right extension is learned path weighting over contour parts: which syllable +onsets, bends, formants, payload ornaments, partials, and time gaps survive each +output/mic path. The latest training receipt says identity can survive the current mel-cepstral degradation panel, but timing collapses under warped domains. That points the next cut at a global delay/clock/path hypothesis over detected words, not a larger brute-force dictionary. -That cut has started in the harness. `--bioacoustic-train` now persists a -global clock hypothesis per result and scores timing through its confidence -rather than raw absolute word offset. The latest corrected receipt is -`artifacts/bioacoustic-training/bioacoustic-20260524-150729/`; it shows compact -indexing still wins clean throughput, baseline indexing remains steadier under -degradation, highband indexing has real warped-path value, and anchor density -must become its own benchmark metric. +That cut has started in the harness. `--bioacoustic-train` persists a global +clock hypothesis per result, and `--calibrate-contestant-asio-f32` now persists +the first physical packet-song response model. The current Scarlett canary +receipt clears 10x realtime and proves that a single song word can own local +identity; the remaining work is to make each syllable/formant contour emit +enough stable anchors that physical mic residuals collapse from tens of +microseconds toward the one-microsecond target. diff --git a/docs/code-algorithm-map.md b/docs/code-algorithm-map.md index 72299b8..914553b 100644 --- a/docs/code-algorithm-map.md +++ b/docs/code-algorithm-map.md @@ -837,7 +837,10 @@ Blocks: Invariant: - A receiver with only the codebook, schedule, and its local audio buffer can - recover canonical time once it hears a valid motif word. + recover canonical time once it hears a valid song word. +- The active path is not sequence-coded. A song word's log-mel contour, + syllable timing, formants, payload ornaments, speaker tint, and spectral flux + are the local anchor field. ## Stage 9: Chirp-Bin Reference Timeline @@ -846,6 +849,8 @@ Invariant: Owns: - The controlled chirp-bin calibration/reference codebook and receiver. +- Historical sequence-coded reference machinery only. This is not the active + packet-song receiver. Core state: diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index bf42897..4c5c36f 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -70,7 +70,10 @@ a named invariant that the native runtime cannot protect yet. a low-gain birdsong-like word language: 128 self-identifying word positions, left-speaker and right-speaker variants, four formant-rich syllables per word, rhythm variation, and direct word identity so a correctly decoded word - identifies canonical timeline position. Runtime active sync now reports + identifies canonical timeline position. The active receiver is no longer a + sequence decoder: each song contour should expose multiple time/frequency + anchors through syllable timing, bends, formants, payload ornaments, rhythm, + speaker tint, and log-mel shape. Runtime active sync now reports `evidence=bioacoustic`. - `MimirChirpletTimeline` owns the older structured chirplet calibration stream, PCM segment rendering, matched timing trace, and per-band response kernels. @@ -215,13 +218,12 @@ a named invariant that the native runtime cannot protect yet. other cameras. 2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained frame cadence in the rolling buffers. -3. Use bioacoustic motif response profiles from real microphones to tune - contour, formant, band, gain, rhythm weighting, and left/right speaker - separation without weakening the standalone word-identity receiver invariant. - Keep chirp-bin calibration artifacts as reference data, not the runtime - target. Promote the `--bioacoustic-train` receipt shape into a real - calibration loop: detected words should feed a global delay/clock/path - hypothesis and learned per-output/mic response, not per-case threshold rules. +3. Promote the packet-song physical calibration receipt into the runtime + receiver. The live decoder should keep its ear open for self-identifying + song contours, extract intra-call time/frequency anchors from log-mel parts, + apply learned per-output/mic path weighting, and feed a global + delay/clock/path hypothesis. Keep chirp-bin calibration artifacts as + reference data, not the runtime target. 4. Add the synchronization actuator: drive a variable-rate resampler and fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder diff --git a/notes/current-system-map.md b/notes/current-system-map.md index f22aebb..7dbb061 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -116,9 +116,13 @@ and uses program-audio phase correlation, and `hybrid` uses passive evidence by default while emitting active pilot chunks only when passive confidence is weak. Active decoding does not inherit the passive two-second analysis floor: passive still needs a longer program-audio window, while the bioacoustic witness can -decode once at least one self-identifying word is present. The bioacoustic -detector now uses bounded motif proposals, matched motif scoring, direct word -anchors, source clock fitting, and fractional waveform refinement. +decode once at least one self-identifying song word is present. The active +decoder is a song-contour anchor machine, not a de Bruijn sequence receiver: +one call carries enough contour, syllable timing, formant, payload, rhythm, and +speaker-tint evidence to identify canonical time inside the operating horizon +and pin multiple time/frequency anchors at once. The bioacoustic detector now +uses bounded motif proposals, matched motif scoring, direct word anchors, +source clock fitting, and fractional waveform refinement. Pairwise sync compares matched anchors first, then only accepts independent clock-fit offsets inside the live latency horizon so period aliases do not become absurd reports. The old chirp-bin detector remains as a diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index bd4bdf1..f164d87 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -89,7 +89,11 @@ Get-Content .\state\evidence.jsonl -Tail 8 program-audio phase correlation, and `hybrid` emits active pilot chunks only when passive confidence is weak. The active bioacoustic path has its own short-window analyzer floor instead of inheriting the passive two-second gate. - It uses energy/onset proposals, dense fallback probes, bounded motif matching, + The active receiver is now a song-contour anchor machine, not a de Bruijn + sequence receiver: one heard call should carry enough syllable timing, + formant, payload, rhythm, speaker tint, and log-mel contour evidence to + identify canonical time and pin multiple time/frequency anchors at once. It + uses energy/onset proposals, dense fallback probes, bounded motif matching, direct word anchoring, clock fit, and constrained local waveform correlation for final fractional delay. It also supports standalone source offset recovery from schedule/codebook state, which is the Raven/phone shape. @@ -141,10 +145,12 @@ Get-Content .\state\evidence.jsonl -Tail 8 The corrected two-hour study pass ran from `2026-05-23T23:54:42.9041588+01:00` to at least -`2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement: -streaming calibration-weighted chirp-bin decoder state, calibration-session -command/replay proof, fractional-delay/SRO actuator, then native camera payload -handles and Fensalir contract lowering. Keep loopback as timing authority and -keep Scarlett capture on the in-process ASIO source. Do not call synchronization -analysis from UI/telemetry readouts. Do not restore deleted script -infrastructure because a stale doc once missed it. +`2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement the +active song-contour receiver: promote packet-song physical calibration from the +BufferSmoke receipt into runtime state, extract intra-call time/frequency +anchors from log-mel contour parts, apply learned path weighting, drive the +fractional-delay/SRO actuator, then continue native camera payload handles and +Fensalir contract lowering. Keep loopback as timing authority and keep Scarlett +capture on the in-process ASIO source. Do not call synchronization analysis +from UI/telemetry readouts. Do not restore deleted script infrastructure +because a stale doc once missed it. diff --git a/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs index f5e3c41..b5e4ff4 100644 --- a/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs @@ -34,7 +34,6 @@ public sealed class MimirBioacousticTimeline public const int WordCount = 128; public const int SpeakerCount = 2; public const int SymbolCount = WordCount * SpeakerCount; - public const int TimelineOrder = 1; private const double FirstEventSeconds = 0.08; private const double EventSpacingSeconds = 0.16; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index fb514a7..27b32e7 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -94,3 +94,4 @@ {"ts": "2026-05-24T23:04:57+01:00", "type": "experiment", "status": "partial", "note": "First real-world packet razor ASIO test: rendered canary-packet-trill at 192 kHz, played through Focusrite ASIO, captured four inputs. Corrected probe needed --set-sample-rate 192000; first run stayed at 44.1 kHz. Loopback channels 2/3 correlated at +1623 samples and decoded 37/37 events with payload=1.000, timing=1.000, delay=8453.125 us. Acoustic channel 1 decoded only 1/37 and channel 0 none, so room/mic path still needs acquisition/level/channel work."} {"ts": "2026-05-24T23:10:02+01:00", "type": "experiment", "status": "partial", "note": "Second physical 192 kHz packet run with cardioid and co-streamer shotgun: loopback channels 2/3 still decode 37/37 payload=1.000 at offset 1623 samples. Shotgun channel 1 locks with per-channel offset 2122 samples: 37/37 payload=1.000, timing=0.828, delay=11166.397 us. Cardioid channel 0 partially decodes at offset 2519: 26/37 payload=0.595. Added polarity-invariant scoring for the streaming packet-local path because mic correlations were inverted."} {"ts": "2026-05-24T23:22:46+01:00", "type": "experiment", "status": "partial", "note": "Promoted canary-packet physical ASIO calibration into a persisted bioacoustic response model at calibration/bioacoustic/scarlett-canary-packet-192k-rerun.json. The command clears the hot-loop budget at 10.7x realtime across four 192 kHz channels after template caching and two-sample coarse search with sub-sample refinement. Loopback channels remain 37/37 with 2.524 us MAE. Co-streamer shotgun reaches 37/37 payload=1.000 with 58.785 us MAE and 2.940 ms acoustic delay relative to loopback. Cardioid reaches 26/37 payload=0.703 with 90.558 us MAE and 4.636 ms acoustic delay. This is exact enough to build response normalization and propagation-delay profiles, but not yet true microsecond sync for physical mics."} +{"ts": "2026-05-24T23:35:51+01:00", "type": "decision", "status": "ok", "note": "Active bioacoustic sync is no longer sequence-coded. The live receiver should keep its ear open for self-identifying song contours; one heard call carries enough log-mel contour, syllable timing, formant, rhythm, payload, and speaker-tint evidence to pin multiple time/frequency anchors at once. De Bruijn/triplet machinery is demoted to retained chirplet/chirp-bin reference artifacts only, not runtime authority."} diff --git a/state/map.yaml b/state/map.yaml index 0dfc674..362037a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -15,9 +15,10 @@ objective: microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP, and OBS output. current: > - Implement direct camera/audio capture into Mimir.Runtime buffers, starting - with Leap stereo IR as the timing-camera candidate, then expose runtime - health/settings/output controls through Fensalir UI. + Promote the active bioacoustic packet-song receiver into runtime state: + self-identifying song contours should pin multiple time/frequency anchors + per heard call, learn per-path response, and drive the audio actuator while + direct camera/audio capture continues feeding Mimir.Runtime buffers. constraints: - Sender is Windows. @@ -72,8 +73,11 @@ accepted_design: low-gain birdsong-like word language with 128 self-identifying word positions, left-speaker and right-speaker variants, four formant-rich syllables per word, rhythm variation, and direct word identity instead of a - de Bruijn grammar. `MimirRuntime` emits separate panned vocabularies per - speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as + de Bruijn grammar. The active receiver is a song-contour anchor machine: + one heard call should expose syllable timing, contour bends, formants, + payload ornaments, speaker tint, and log-mel evidence as a cluster of + time/frequency anchors. `MimirRuntime` emits separate panned vocabularies + per speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as `evidence=bioacoustic`; the old chirp-bin path remains a calibration/reference artifact with ASIO tooling. - The diagnostic audio path can optionally carry base64 float sample payloads @@ -320,9 +324,10 @@ current_status: loopback, live mic buffers stay populated, runtime sync analysis now runs as a bounded rotating service without UI/telemetry re-entry, and the bioacoustic motif timeline produces deterministic synthetic active sync. The next - invariant is self-identifying motif words producing stable canonical anchors - and clock fits through real loopback and microphones, so the actuator can use - decoded time instead of filtered lag guesses. Replace the remaining + invariant is self-identifying song contours producing stable intra-call + time/frequency anchors and clock fits through real loopback and microphones, + so the actuator can use decoded time instead of filtered lag guesses. + Replace the remaining diagnostic JSON-line process bridges with direct native capture drivers that push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when