Make song contours the active sync authority

This commit is contained in:
Meta Vens 2026-05-24 23:31:41 +01:00
commit f134ea6bca
9 changed files with 108 additions and 88 deletions

View file

@ -184,13 +184,13 @@ half-second PCM segments ahead of the audio cursor. Each motif is a short phrase
of formant-rich syllables with log-frequency contour and rhythm offsets, so the
code is carried by both spectral shape and timing.
The intended decoder is a compact constrained motif transform, not a generic
time-frequency explorer and not an outlier filter around bad guesses. Mimir owns
the emitter, so the receiver projects each mic stream against the known motif
dictionary, produces transform frames with multiple word candidates, and maps
the best coherent word identities directly to canonical events. A word becomes a
canonical timeline anchor when its motif likelihood and neighboring anchors
agree on one local sample clock. A decoded anchor
The intended decoder is a compact constrained song-contour transform, not a
generic time-frequency explorer and not an outlier filter around bad guesses.
Mimir owns the emitter, so the receiver keeps its ear open for known song words
and maps each successful call directly to canonical time. A word is not a
single scalar event: syllable onsets, bends, formants, rhythm offsets, payload
ornaments, speaker tint, and log-mel contour all expose local time/frequency
constraints. One call can pin a cluster of anchors at once. A decoded anchor
means observed sample offset `S` corresponds to emitted event time `T`. A stream
of anchors fits the source clock directly:
@ -201,14 +201,15 @@ observed_sample = source_offset + canonical_seconds * effective_sample_rate
Delay and sampling-rate offset are derived by comparing the loopback clock fit
to each mic clock fit over common canonical time. The state tracker is not
allowed to launder invalid words into plausible timing. If a stream cannot
produce at least one matched canonical anchor, it has not decoded timing for
produce at least one matched canonical song word, it has not decoded timing for
that window. Reports carry rounded integer delay, fractional delay, and the
count/confidence of timeline-symbol anchors used to derive the report.
count/confidence of contour anchors used to derive the report.
The same timeline starts the frequency-response path. Each active bioacoustic
decode carries per-band response evidence for the source set, whether or not
each source earns a timing report. The older chirp-bin calibration model remains
the controlled ASIO reference surface for response/confusion/delay captures:
The same song starts the frequency-response path. Each active bioacoustic decode
carries per-band and per-contour response evidence for the source set, whether
or not each source earns a timing report. The older chirp-bin calibration model
remains the controlled ASIO reference surface for response/confusion/delay
captures:
use `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` to render that reference
timeline, optionally capture it through the ASIO worker with `--capture-asio`,
compute the response/confusion/delay model, and persist it for decoder tuning.
@ -225,10 +226,10 @@ bioacoustic timeline queued, polls sources, and updates sync analysis on a fixed
`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current
runtime testing proves Fensalir output wakes the Scarlett loopback and the mic
buffers stay live. The next failure is physical capture proof: the same
canonical motif anchors need to stay stable through loopback, room mics, device
canonical song-contour anchors need to stay stable through loopback, room mics, device
clocks, and codec/network paths.
## Chirplet Calibration Model
## Retained Reference Calibration
```mermaid
flowchart TD
@ -237,7 +238,7 @@ flowchart TD
B --> D["room + speakers + mics"]
C --> E["loopback rolling buffer"]
D --> F["mic rolling buffers"]
E --> G["matched chirplet traces"]
E --> G["matched reference traces"]
F --> G
G --> H["symbol likelihood events"]
H --> L["timeline anchors"]
@ -248,7 +249,7 @@ flowchart TD
I --> K["frequency response normalization"]
```
The chirplet timeline owns three facts:
The retained chirplet/chirp-bin timelines own three reference facts:
- **Emission**: the PCM that Fensalir sends to the speakers.
- **Timing witness**: the matched-filter atom bank used to find the stream in
@ -256,7 +257,7 @@ The chirplet timeline owns three facts:
- **Response witness**: the per-band atoms used to measure how strongly each
mic hears each emitted band.
Continuous chirplet evidence gives both the current delay and the drift/SRO by
Continuous active song evidence should give both the current delay and the drift/SRO by
watching delay change over time. Per-band energy over the same stream gives the
normalization curve. The important constraint is that all three measurements
must be tied to the same emitted timeline, not three separately invented probes.
@ -267,21 +268,12 @@ band set but still fails pairwise timing because no canonical events overlap
the loopback path. That is not a blank failure anymore; it is calibration data
for the next codebook/weighting cut.
The symbol layer is intentionally redundant. It does not rely on one fixed
frequency shelf: timing gaps, chirp duration, start band, and glide shape all
contribute so poor mic frequency response does not erase the whole code.
`MimirChirpletSymbolCodebook` owns the symbol definitions so timeline ordering
does not smuggle acoustic shape decisions into bit arithmetic. Every symbol has
a unique chirp shape; rhythm remains additional evidence, not a substitute for
symbol separability. The transform uses sine/cosine chirplet kernels for each
symbol, so timing does not depend on the receiver preserving the emitter's
absolute phase. Run `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`
to render two seconds of canonical timeline audio and decode it back to
timeline anchors. Current synthetic proof detects all 15 emitted chirps, keeps
13 possible triplet anchors for events 0 through 12, fits the clock at
47999.999990 Hz, and holds mean absolute anchor error to 0.000014 samples. Real
device runs still depend on loopback capture staying live; the local Scarlett
loopback has intermittently stopped advancing during short headless sniffs.
The active symbol layer is now the song contour. It does not rely on one fixed
frequency shelf or a coded sequence: syllable timing, contour bends, formants,
payload ornaments, speaker tint, and local spectral flux all contribute so poor
mic response does not erase the whole code. The old chirplet self-test remains
available as historical proof of sub-frame math, but it must not retake
authority over the active runtime path.
The older `MimirChirpletTimeline` matcher is a diagnostic/reference artifact,
not the active runtime path. `BuildChirpletEnergyTrace` still behaves like a

View file

@ -12,9 +12,10 @@ ask only "which bin fired?" It should ask:
```text
which motif contour did I hear?
which self-identifying word did that contour encode?
which self-identifying song word did that contour encode?
which speaker vocabulary emitted it?
what delay and clock fit make the heard words coherent?
which time and frequency anchors did that one call expose?
what delay and clock fit make the heard anchors coherent?
what did this room/mic path do to the spectrum?
```
@ -27,10 +28,15 @@ what did this room/mic path do to the spectrum?
- Each word contains four short syllables.
- Each syllable carries a log-frequency glide plus formant-like partials.
- Motif rhythm, contour, syllable duration, root, and speaker tint vary by word.
- No de Bruijn grammar is used in the active bioacoustic path. A word identifies
its canonical event directly inside the current operating horizon.
- The decoder emits motif candidates, direct word anchors, a source clock fit,
and per-band response evidence.
- No de Bruijn grammar is used in the active bioacoustic path. One heard song
word identifies its canonical event directly inside the current operating
horizon.
- The decoder emits motif candidates, direct song-word anchors, a source clock
fit, and per-band response evidence.
- A song word is not one scalar chirp. It is a contour packet: multiple
syllable onsets, bends, formants, rhythm offsets, payload ornaments, and
speaker-colored spectral features all become anchors in log-mel space. One
successful call can pin a cluster of time/frequency constraints at once.
- Runtime emission sends the left vocabulary to the left speaker and the right
vocabulary to the right speaker, so microphones can learn each side of the
room separately.
@ -45,11 +51,11 @@ watermark target.
`MimirBioacousticTimeline` owns:
- motif codebook shape;
- song-word codebook shape;
- deterministic phrase schedule;
- PCM rendering for active emission;
- streaming motif scoring;
- direct word anchor decoding;
- direct contour-anchor decoding;
- source clock fitting;
- response-band evidence.
@ -72,7 +78,7 @@ Birdsong survives ugly channels because the information is redundant:
- formant envelope survives when one band dies;
- speaker-specific variants expose room asymmetry;
- repeated local structure gives the receiver more than one way to recover time
and response.
response, and local payload identity.
That is the physical path Mimir needs. A phone mic, camera mic, Scarlett input,
or network receiver should be able to recognize canonical time from a damaged
@ -112,34 +118,34 @@ dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --rende
## Next Cut
The current runtime decoder still uses a motif matched-filter proof. The smoke
harness now carries the next receiver shape: indexed MFCC/log-mel word
fingerprints with path-degradation augmentation. The next coherent cut is to
promote that shape into a true streaming log-mel receiver:
The current physical receipt path uses the canary packet song as the first
serious contour-anchor witness. The next coherent cut is to promote that shape
into the live streaming log-mel receiver:
```text
audio window
-> log-mel / constant-Q evidence surface
-> motif/formant/contour candidates
-> projection-hash / ANN candidate retrieval
-> direct word identity decode over a tiny candidate set
-> global delay and clock hypothesis
-> contour packet proposals
-> direct song-word identity over a tiny candidate set
-> intra-call time/frequency anchor extraction
-> global delay, clock, and path hypothesis
-> path response model
```
Do not add a pile of threshold rules to make one room capture look good. The
right extension is learned path weighting over motif parts: which contours,
partials, and time gaps survive each output/mic path.
right extension is learned path weighting over contour parts: which syllable
onsets, bends, formants, payload ornaments, partials, and time gaps survive each
output/mic path.
The latest training receipt says identity can survive the current
mel-cepstral degradation panel, but timing collapses under warped domains. That
points the next cut at a global delay/clock/path hypothesis over detected words,
not a larger brute-force dictionary.
That cut has started in the harness. `--bioacoustic-train` now persists a
global clock hypothesis per result and scores timing through its confidence
rather than raw absolute word offset. The latest corrected receipt is
`artifacts/bioacoustic-training/bioacoustic-20260524-150729/`; it shows compact
indexing still wins clean throughput, baseline indexing remains steadier under
degradation, highband indexing has real warped-path value, and anchor density
must become its own benchmark metric.
That cut has started in the harness. `--bioacoustic-train` persists a global
clock hypothesis per result, and `--calibrate-contestant-asio-f32` now persists
the first physical packet-song response model. The current Scarlett canary
receipt clears 10x realtime and proves that a single song word can own local
identity; the remaining work is to make each syllable/formant contour emit
enough stable anchors that physical mic residuals collapse from tens of
microseconds toward the one-microsecond target.

View file

@ -837,7 +837,10 @@ Blocks:
Invariant:
- A receiver with only the codebook, schedule, and its local audio buffer can
recover canonical time once it hears a valid motif word.
recover canonical time once it hears a valid song word.
- The active path is not sequence-coded. A song word's log-mel contour,
syllable timing, formants, payload ornaments, speaker tint, and spectral flux
are the local anchor field.
## Stage 9: Chirp-Bin Reference Timeline
@ -846,6 +849,8 @@ Invariant:
Owns:
- The controlled chirp-bin calibration/reference codebook and receiver.
- Historical sequence-coded reference machinery only. This is not the active
packet-song receiver.
Core state:

View file

@ -70,7 +70,10 @@ a named invariant that the native runtime cannot protect yet.
a low-gain birdsong-like word language: 128 self-identifying word positions,
left-speaker and right-speaker variants, four formant-rich syllables per word,
rhythm variation, and direct word identity so a correctly decoded word
identifies canonical timeline position. Runtime active sync now reports
identifies canonical timeline position. The active receiver is no longer a
sequence decoder: each song contour should expose multiple time/frequency
anchors through syllable timing, bends, formants, payload ornaments, rhythm,
speaker tint, and log-mel shape. Runtime active sync now reports
`evidence=bioacoustic`.
- `MimirChirpletTimeline` owns the older structured chirplet calibration stream,
PCM segment rendering, matched timing trace, and per-band response kernels.
@ -215,13 +218,12 @@ a named invariant that the native runtime cannot protect yet.
other cameras.
2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
frame cadence in the rolling buffers.
3. Use bioacoustic motif response profiles from real microphones to tune
contour, formant, band, gain, rhythm weighting, and left/right speaker
separation without weakening the standalone word-identity receiver invariant.
Keep chirp-bin calibration artifacts as reference data, not the runtime
target. Promote the `--bioacoustic-train` receipt shape into a real
calibration loop: detected words should feed a global delay/clock/path
hypothesis and learned per-output/mic response, not per-case threshold rules.
3. Promote the packet-song physical calibration receipt into the runtime
receiver. The live decoder should keep its ear open for self-identifying
song contours, extract intra-call time/frequency anchors from log-mel parts,
apply learned per-output/mic path weighting, and feed a global
delay/clock/path hypothesis. Keep chirp-bin calibration artifacts as
reference data, not the runtime target.
4. Add the synchronization actuator: drive a variable-rate resampler and
fractional delay line per non-reference stream from the smoothed
`MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder

View file

@ -116,9 +116,13 @@ and uses program-audio phase correlation, and `hybrid` uses passive evidence by
default while emitting active pilot chunks only when passive confidence is weak.
Active decoding does not inherit the passive two-second analysis floor: passive
still needs a longer program-audio window, while the bioacoustic witness can
decode once at least one self-identifying word is present. The bioacoustic
detector now uses bounded motif proposals, matched motif scoring, direct word
anchors, source clock fitting, and fractional waveform refinement.
decode once at least one self-identifying song word is present. The active
decoder is a song-contour anchor machine, not a de Bruijn sequence receiver:
one call carries enough contour, syllable timing, formant, payload, rhythm, and
speaker-tint evidence to identify canonical time inside the operating horizon
and pin multiple time/frequency anchors at once. The bioacoustic detector now
uses bounded motif proposals, matched motif scoring, direct word anchors,
source clock fitting, and fractional waveform refinement.
Pairwise sync compares matched anchors first, then only accepts independent
clock-fit offsets inside the live latency horizon so period aliases do not
become absurd reports. The old chirp-bin detector remains as a

View file

@ -89,7 +89,11 @@ Get-Content .\state\evidence.jsonl -Tail 8
program-audio phase correlation, and `hybrid` emits active pilot chunks only
when passive confidence is weak. The active bioacoustic path has its own
short-window analyzer floor instead of inheriting the passive two-second gate.
It uses energy/onset proposals, dense fallback probes, bounded motif matching,
The active receiver is now a song-contour anchor machine, not a de Bruijn
sequence receiver: one heard call should carry enough syllable timing,
formant, payload, rhythm, speaker tint, and log-mel contour evidence to
identify canonical time and pin multiple time/frequency anchors at once. It
uses energy/onset proposals, dense fallback probes, bounded motif matching,
direct word anchoring, clock fit, and constrained local waveform
correlation for final fractional delay. It also supports standalone source
offset recovery from schedule/codebook state, which is the Raven/phone shape.
@ -141,10 +145,12 @@ Get-Content .\state\evidence.jsonl -Tail 8
The corrected two-hour study pass ran from
`2026-05-23T23:54:42.9041588+01:00` to at least
`2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement:
streaming calibration-weighted chirp-bin decoder state, calibration-session
command/replay proof, fractional-delay/SRO actuator, then native camera payload
handles and Fensalir contract lowering. Keep loopback as timing authority and
keep Scarlett capture on the in-process ASIO source. Do not call synchronization
analysis from UI/telemetry readouts. Do not restore deleted script
infrastructure because a stale doc once missed it.
`2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement the
active song-contour receiver: promote packet-song physical calibration from the
BufferSmoke receipt into runtime state, extract intra-call time/frequency
anchors from log-mel contour parts, apply learned path weighting, drive the
fractional-delay/SRO actuator, then continue native camera payload handles and
Fensalir contract lowering. Keep loopback as timing authority and keep Scarlett
capture on the in-process ASIO source. Do not call synchronization analysis
from UI/telemetry readouts. Do not restore deleted script infrastructure
because a stale doc once missed it.

View file

@ -34,7 +34,6 @@ public sealed class MimirBioacousticTimeline
public const int WordCount = 128;
public const int SpeakerCount = 2;
public const int SymbolCount = WordCount * SpeakerCount;
public const int TimelineOrder = 1;
private const double FirstEventSeconds = 0.08;
private const double EventSpacingSeconds = 0.16;

View file

@ -94,3 +94,4 @@
{"ts": "2026-05-24T23:04:57+01:00", "type": "experiment", "status": "partial", "note": "First real-world packet razor ASIO test: rendered canary-packet-trill at 192 kHz, played through Focusrite ASIO, captured four inputs. Corrected probe needed --set-sample-rate 192000; first run stayed at 44.1 kHz. Loopback channels 2/3 correlated at +1623 samples and decoded 37/37 events with payload=1.000, timing=1.000, delay=8453.125 us. Acoustic channel 1 decoded only 1/37 and channel 0 none, so room/mic path still needs acquisition/level/channel work."}
{"ts": "2026-05-24T23:10:02+01:00", "type": "experiment", "status": "partial", "note": "Second physical 192 kHz packet run with cardioid and co-streamer shotgun: loopback channels 2/3 still decode 37/37 payload=1.000 at offset 1623 samples. Shotgun channel 1 locks with per-channel offset 2122 samples: 37/37 payload=1.000, timing=0.828, delay=11166.397 us. Cardioid channel 0 partially decodes at offset 2519: 26/37 payload=0.595. Added polarity-invariant scoring for the streaming packet-local path because mic correlations were inverted."}
{"ts": "2026-05-24T23:22:46+01:00", "type": "experiment", "status": "partial", "note": "Promoted canary-packet physical ASIO calibration into a persisted bioacoustic response model at calibration/bioacoustic/scarlett-canary-packet-192k-rerun.json. The command clears the hot-loop budget at 10.7x realtime across four 192 kHz channels after template caching and two-sample coarse search with sub-sample refinement. Loopback channels remain 37/37 with 2.524 us MAE. Co-streamer shotgun reaches 37/37 payload=1.000 with 58.785 us MAE and 2.940 ms acoustic delay relative to loopback. Cardioid reaches 26/37 payload=0.703 with 90.558 us MAE and 4.636 ms acoustic delay. This is exact enough to build response normalization and propagation-delay profiles, but not yet true microsecond sync for physical mics."}
{"ts": "2026-05-24T23:35:51+01:00", "type": "decision", "status": "ok", "note": "Active bioacoustic sync is no longer sequence-coded. The live receiver should keep its ear open for self-identifying song contours; one heard call carries enough log-mel contour, syllable timing, formant, rhythm, payload, and speaker-tint evidence to pin multiple time/frequency anchors at once. De Bruijn/triplet machinery is demoted to retained chirplet/chirp-bin reference artifacts only, not runtime authority."}

View file

@ -15,9 +15,10 @@ objective:
microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP,
and OBS output.
current: >
Implement direct camera/audio capture into Mimir.Runtime buffers, starting
with Leap stereo IR as the timing-camera candidate, then expose runtime
health/settings/output controls through Fensalir UI.
Promote the active bioacoustic packet-song receiver into runtime state:
self-identifying song contours should pin multiple time/frequency anchors
per heard call, learn per-path response, and drive the audio actuator while
direct camera/audio capture continues feeding Mimir.Runtime buffers.
constraints:
- Sender is Windows.
@ -72,8 +73,11 @@ accepted_design:
low-gain birdsong-like word language with 128 self-identifying word
positions, left-speaker and right-speaker variants, four formant-rich
syllables per word, rhythm variation, and direct word identity instead of a
de Bruijn grammar. `MimirRuntime` emits separate panned vocabularies per
speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as
de Bruijn grammar. The active receiver is a song-contour anchor machine:
one heard call should expose syllable timing, contour bends, formants,
payload ornaments, speaker tint, and log-mel evidence as a cluster of
time/frequency anchors. `MimirRuntime` emits separate panned vocabularies
per speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as
`evidence=bioacoustic`; the old chirp-bin path remains a
calibration/reference artifact with ASIO tooling.
- The diagnostic audio path can optionally carry base64 float sample payloads
@ -320,9 +324,10 @@ current_status:
loopback, live mic buffers stay populated, runtime sync analysis now runs as
a bounded rotating service without UI/telemetry re-entry, and the bioacoustic
motif timeline produces deterministic synthetic active sync. The next
invariant is self-identifying motif words producing stable canonical anchors
and clock fits through real loopback and microphones, so the actuator can use
decoded time instead of filtered lag guesses. Replace the remaining
invariant is self-identifying song contours producing stable intra-call
time/frequency anchors and clock fits through real loopback and microphones,
so the actuator can use decoded time instead of filtered lag guesses.
Replace the remaining
diagnostic JSON-line process bridges with direct native capture drivers that
push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports
for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when