diff --git a/docs/audio-field.md b/docs/audio-field.md index dbdbf25..227f05a 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -69,10 +69,10 @@ one attractive correlation peak. The active calibration path is deliberately receiver-cheap in both `chirp-only` and `hybrid`. `MimirBioacousticTimeline` is the live runtime witness: it emits -low-gain log-frequency motifs made from short formant-rich syllables, rhythm -offsets, and a de Bruijn triplet grammar. Acquisition is an energy proposal pass -plus bounded motif matching; timeline placement is the same code-valid triplet -trellis, and a constrained local waveform correlation around the decoded delay +low-gain log-frequency words made from short formant-rich syllables, rhythm +offsets, and speaker-specific variants. Acquisition is an energy proposal pass +plus bounded motif matching; timeline placement comes from direct word identity, +and a constrained local waveform correlation around the decoded delay provides the final fractional offset. The decoder keeps per-band response evidence for each detected motif so the same stream can calibrate timing and the speaker/room/mic transfer function. @@ -95,8 +95,8 @@ sequence order so the timeline remains unique without pretending dead bands still carry code. The synthetic invariant is `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`; -it renders the bioacoustic timeline, decodes motif anchors, and requires -code-valid triplets plus a stable source clock. The standalone invariant is +it renders the bioacoustic timeline, decodes direct word anchors, and requires a +stable source clock. The standalone invariant is `--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5`; it proves that a delayed receiver can recover canonical source time from the known codebook and schedule alone. `chirp-only` emits this witness continuously; @@ -156,7 +156,7 @@ current studio-monitor-to-Scarlett-mic acoustic path was already weak by unproven until a measured transducer/mic path earns it. The active bioacoustic timing proof now runs in memory through the same runtime -analyzer path. `--bioacoustic-self-test` proves code-valid motif anchors, +analyzer path. `--bioacoustic-self-test` proves direct word anchors, `--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5` proves a receiver can recover canonical source time from delayed audio alone, and `--chirp-only-sync-self-test --sample-rate 48000` recovers a 317.375-sample @@ -169,10 +169,10 @@ through `asio-ch3`. The full probe runtime config now enables sample-bearing blocks for every local audio source. `MimirBioacousticTimeline` owns the emitted calibration stream and the motif-matching shape used to analyze it. The default timeline is a -deterministic order-3 de Bruijn symbol sequence over 16 log-frequency motifs. -Any three consecutive correctly detected motifs identify the event index inside -the current operating horizon, so a receiver can place its audio window on the -canonical timeline without being handed Mimir's runtime clock. Mimir queues +deterministic set of self-identifying log-frequency words with separate left and +right speaker variants. Any correctly detected word identifies the event index +inside the current operating horizon, so a receiver can place its audio window +on the canonical timeline without being handed Mimir's runtime clock. Mimir queues half-second PCM segments ahead of the audio cursor. Each motif is a short phrase of formant-rich syllables with log-frequency contour and rhythm offsets, so the code is carried by both spectral shape and timing. @@ -180,10 +180,10 @@ code is carried by both spectral shape and timing. The intended decoder is a compact constrained motif transform, not a generic time-frequency explorer and not an outlier filter around bad guesses. Mimir owns the emitter, so the receiver projects each mic stream against the known motif -dictionary, produces transform frames with multiple symbol candidates, and -scores candidate triplets through the de Bruijn map. A triplet only becomes a -canonical timeline anchor when its symbol likelihoods, measured inter-motif -gaps, and neighboring anchors agree on one local sample clock. A decoded anchor +dictionary, produces transform frames with multiple word candidates, and maps +the best coherent word identities directly to canonical events. A word becomes a +canonical timeline anchor when its motif likelihood and neighboring anchors +agree on one local sample clock. A decoded anchor means observed sample offset `S` corresponds to emitted event time `T`. A stream of anchors fits the source clock directly: @@ -193,8 +193,8 @@ observed_sample = source_offset + canonical_seconds * effective_sample_rate Delay and sampling-rate offset are derived by comparing the loopback clock fit to each mic clock fit over common canonical time. The state tracker is not -allowed to launder invalid codewords into plausible timing. If a stream cannot -produce at least three matched canonical anchors, it has not decoded timing for +allowed to launder invalid words into plausible timing. If a stream cannot +produce at least one matched canonical anchor, it has not decoded timing for that window. Reports carry rounded integer delay, fractional delay, and the count/confidence of timeline-symbol anchors used to derive the report. @@ -233,7 +233,7 @@ flowchart TD E --> G["matched chirplet traces"] F --> G G --> H["symbol likelihood events"] - H --> L["triplet timeline anchors"] + H --> L["timeline anchors"] L --> M["per-source clock fit"] M --> N["delay + SRO"] G --> I["per-source calibration profile"] diff --git a/docs/bioacoustic-timeline-watermark.md b/docs/bioacoustic-timeline-watermark.md index 5d5c91e..3129d0f 100644 --- a/docs/bioacoustic-timeline-watermark.md +++ b/docs/bioacoustic-timeline-watermark.md @@ -7,13 +7,14 @@ That shape proved the math, but it is the wrong animal for a room. Mimir now treats active acoustic sync as a low-gain bioacoustic language: short birdsong-like motifs whose identity survives through contour, log-frequency -position, spectral envelope, rhythm, and phrase order. The receiver should not +position, spectral envelope, rhythm, and speaker-specific timbre. The receiver should not ask only "which bin fired?" It should ask: ```text which motif contour did I hear? -where does this motif phrase sit in the canonical grammar? -what delay and clock fit make the whole phrase coherent? +which self-identifying word did that contour encode? +which speaker vocabulary emitted it? +what delay and clock fit make the heard words coherent? what did this room/mic path do to the spectrum? ``` @@ -21,14 +22,18 @@ what did this room/mic path do to the spectrum? `MimirBioacousticTimeline` is the current active runtime watermark. -- 16 log-spaced motif symbols. -- Each motif contains three short syllables. +- 128 self-identifying word positions, each with a left-speaker and + right-speaker variant. +- Each word contains four short syllables. - Each syllable carries a log-frequency glide plus formant-like partials. -- Motif rhythm varies by symbol group. -- The timeline is an order-3 de Bruijn grammar, so any three correctly decoded - consecutive motifs identify canonical event position. -- The decoder emits motif candidates, code-valid triplet anchors, a source - clock fit, and per-band response evidence. +- Motif rhythm, contour, syllable duration, root, and speaker tint vary by word. +- No de Bruijn grammar is used in the active bioacoustic path. A word identifies + its canonical event directly inside the current operating horizon. +- The decoder emits motif candidates, direct word anchors, a source clock fit, + and per-band response evidence. +- Runtime emission sends the left vocabulary to the left speaker and the right + vocabulary to the right speaker, so microphones can learn each side of the + room separately. - The runtime active sync evidence kind is `bioacoustic`. The old `MimirChirpBinTimeline` remains as a calibration/reference artifact @@ -44,7 +49,7 @@ watermark target. - deterministic phrase schedule; - PCM rendering for active emission; - streaming motif scoring; -- de Bruijn triplet anchor decoding; +- direct word anchor decoding; - source clock fitting; - response-band evidence. @@ -65,8 +70,9 @@ Birdsong survives ugly channels because the information is redundant: - contour survives when absolute pitch shifts; - rhythm survives when frequency response is bad; - formant envelope survives when one band dies; -- phrase grammar survives when individual syllables are weak; -- repeated local structure gives the receiver more than one way to recover time. +- speaker-specific variants expose room asymmetry; +- repeated local structure gives the receiver more than one way to recover time + and response. That is the physical path Mimir needs. A phone mic, camera mic, Scarlett input, or network receiver should be able to recognize canonical time from a damaged @@ -84,8 +90,8 @@ dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp Expected state: -- `--bioacoustic-self-test` recovers code-valid motif anchors across the - canonical phrase. +- `--bioacoustic-self-test` recovers direct word anchors across the canonical + phrase. - `--standalone-bioacoustic-self-test` recovers delayed canonical source time from codebook/schedule state alone. - `--chirp-only-sync-self-test` reports `evidence=bioacoustic` and recovers @@ -106,7 +112,7 @@ cut is a true streaming log-mel receiver: audio window -> log-mel / constant-Q evidence surface -> motif/formant/contour candidates --> phrase grammar decode +-> direct word identity decode -> global delay and clock hypothesis -> path response model ``` diff --git a/docs/code-algorithm-map.md b/docs/code-algorithm-map.md index b2b684f..72299b8 100644 --- a/docs/code-algorithm-map.md +++ b/docs/code-algorithm-map.md @@ -36,8 +36,8 @@ mode: Active bioacoustic decode uses known codebook/schedule state. It proposes motif times from local energy, classifies log-frequency/formant symbols by bounded -matched motif scoring, constrains candidates through de Bruijn code-valid -triplets, fits a clock, and records band response evidence for calibration. +matched motif scoring, maps self-identifying words directly to canonical events, +fits a clock, and records band response evidence for calibration. ## Invariants @@ -810,10 +810,11 @@ Owns: Core state: -- Sixteen log-spaced motif symbols. -- Three short formant-rich syllables per motif. -- A de Bruijn order-three schedule, so any valid motif triplet identifies one - canonical timeline location. +- 128 self-identifying word positions. +- Left-speaker and right-speaker variants for each word. +- Four short formant-rich syllables per word. +- Direct word identity, so a valid word identifies one canonical timeline + location in the current operating horizon. - Cached kernels by sample rate. Blocks: @@ -821,7 +822,7 @@ Blocks: - Motif/syllable records: `MimirBioacousticMotifDefinition` and `MimirBioacousticSyllable`. - `RenderSegmentMonoFloat`: renders one low-gain timeline segment. -- `DecodeStreamWindow`: detects motif frames, decodes code-valid triplet +- `DecodeStreamWindow`: detects motif frames, decodes direct word anchors, fits source clock, refines standalone source offset, and emits the shared `MimirChirpletStreamDecode` shape. - `EventForIndex` / `EventsOverlapping`: expose canonical schedule metadata. @@ -829,14 +830,14 @@ Blocks: suppresses nearby duplicate motif hits. - `ClassifyAt`: scores a local candidate against the motif bank. - `ScoreMotif`: matched motif scoring with per-band response evidence. -- `DecodeAnchors`: maps detected motif triplets back to canonical event indexes. +- `DecodeAnchors`: maps detected words back to canonical event indexes. - `FitClock` and `RefineSourceOffset`: recover source timeline position and sub-frame offset from known codebook/schedule state. Invariant: - A receiver with only the codebook, schedule, and its local audio buffer can - recover canonical time once it hears a code-valid motif triplet. + recover canonical time once it hears a valid motif word. ## Stage 9: Chirp-Bin Reference Timeline diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 4af23d7..dcd0a89 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -46,7 +46,7 @@ a named invariant that the native runtime cannot protect yet. hub, and prints the actual rolling buffers. Use `--require-samples` when an empty declared sensor buffer should fail the run. Use `--bioacoustic-self-test` to render the active motif timeline into memory and verify that the decoder - recovers code-valid anchors. Use `--standalone-bioacoustic-self-test` to + recovers direct word anchors. Use `--standalone-bioacoustic-self-test` to verify that a receiver with only the codebook/schedule can recover canonical source offset from delayed audio. - `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback @@ -65,10 +65,11 @@ a named invariant that the native runtime cannot protect yet. capture raw interleaved Float32 ASIO input with `--record-f32-interleaved`. - `MimirBioacousticTimeline` owns the active runtime watermark described in [[bioacoustic-timeline-watermark|Bioacoustic Timeline Watermark]]. It renders - a low-gain birdsong-like phrase language: 16 log-spaced motif symbols, three - formant-rich syllables per motif, rhythm variation, and an order-3 de Bruijn - grammar so any three correctly decoded consecutive motifs identify canonical - timeline position. Runtime active sync now reports `evidence=bioacoustic`. + a low-gain birdsong-like word language: 128 self-identifying word positions, + left-speaker and right-speaker variants, four formant-rich syllables per word, + rhythm variation, and direct word identity so a correctly decoded word + identifies canonical timeline position. Runtime active sync now reports + `evidence=bioacoustic`. - `MimirChirpletTimeline` owns the older structured chirplet calibration stream, PCM segment rendering, matched timing trace, and per-band response kernels. It remains a reference/diagnostic path. @@ -81,8 +82,8 @@ a named invariant that the native runtime cannot protect yet. and clock coherence, and fits a per-source sample clock from those anchors. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: sample-bearing audio blocks are resampled into the Scarlett loopback timeline. - The analyzer derives delay only from matched decoded triplet timeline anchors. - A source without at least three matched anchors has no timing report for that + The analyzer derives delay only from matched decoded timeline anchors. + A source without at least one matched anchor has no timing report for that window. It accepts Float32, Int16, Int24, and Int32 PCM windows so ASIO/native capture can feed true interface formats without a pre-conversion shim. - `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source @@ -183,13 +184,14 @@ a named invariant that the native runtime cannot protect yet. 2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained frame cadence in the rolling buffers. 3. Use bioacoustic motif response profiles from real microphones to tune - contour, formant, band, gain, and rhythm weighting without weakening the - standalone codebook/schedule receiver invariant. Keep chirp-bin calibration - artifacts as reference data, not the runtime target. + contour, formant, band, gain, rhythm weighting, and left/right speaker + separation without weakening the standalone word-identity receiver invariant. + Keep chirp-bin calibration artifacts as reference data, not the runtime + target. 4. Add the synchronization actuator: drive a variable-rate resampler and fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder - through real loopback and microphone paths so every correctly heard triplet + through real loopback and microphone paths so every correctly heard word becomes a deterministic timeline anchor before the actuator moves samples. 5. Prove the bioacoustic hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index ea3e5cd..d4b2b23 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -100,20 +100,21 @@ is the current timing authority when active calibration is playing, but Scarlett production capture belongs on ASIO rather than WASAPI shared mode. `MimirBioacousticTimeline` owns the active runtime watermark described in [[docs/bioacoustic-timeline-watermark|Bioacoustic Timeline Watermark]]: a -low-gain birdsong-like phrase language with 16 log-spaced motif symbols, three -formant-rich syllables per motif, rhythm variation, and an order-3 de Bruijn -grammar. Any three consecutive correctly decoded motifs identify the event -index inside the current operating horizon. Mimir continuously queues that -timeline through Fensalir audio and decodes timing as `bioacoustic` evidence. +low-gain birdsong-like word language with 128 self-identifying word positions, +left-speaker and right-speaker variants, four formant-rich syllables per word, +rhythm variation, and direct word identity. Any correctly decoded word identifies +the event index inside the current operating horizon. Mimir queues the left +vocabulary to the left speaker and the right vocabulary to the right speaker +through Fensalir audio, then decodes timing as `bioacoustic` evidence. `MimirAudioSynchronizationSettings.Mode` chooses whether active calibration is allowed: `chirp-only` emits the active bioacoustic witness, `passive` stays silent and uses program-audio phase correlation, and `hybrid` uses passive evidence by default while emitting active pilot chunks only when passive confidence is weak. Active decoding does not inherit the passive two-second analysis floor: passive still needs a longer program-audio window, while the bioacoustic witness can -decode once at least a code-valid motif triplet is present. The bioacoustic -detector now uses generous motif proposals, matched motif scoring, triplet -grammar decoding, source clock fitting, and fractional waveform refinement. +decode once at least one self-identifying word is present. The bioacoustic +detector now uses bounded motif proposals, matched motif scoring, direct word +anchors, source clock fitting, and fractional waveform refinement. Pairwise sync compares matched anchors first, then only accepts independent clock-fit offsets inside the live latency horizon so period aliases do not become absurd reports. The old chirp-bin detector remains as a @@ -124,7 +125,7 @@ timing residuals, delay hypotheses, phase summaries, and an adaptive codebook plan. The analyzer keeps raw profiles even when no timing report is accepted, so physical mic failures can still guide bioacoustic motif weighting. Reports/states expose `delayUs` next to fractional sample delay. -`Mimir.BufferSmoke --bioacoustic-self-test` proves code-valid motif anchors. +`Mimir.BufferSmoke --bioacoustic-self-test` proves direct word anchors. `--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5` proves a receiver with only codebook/schedule state can recover delayed source time to below printed microsecond precision. `--chirp-only-sync-self-test diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 62af02f..bd4bdf1 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -72,9 +72,9 @@ Get-Content .\state\evidence.jsonl -Tail 8 Kiyo and second PS3 Eye were absent, so six-camera viability is not proven. - Current active sync smoke uses Scarlett speaker loopback as timing authority. `MimirBioacousticTimeline` now owns the active runtime watermark: - low-gain birdsong-like motifs with log-spaced roots, formant-rich syllables, - rhythm variation, and an order-3 de Bruijn phrase grammar. Any three - consecutive correctly decoded motifs identify the event index inside the + low-gain birdsong-like words with log-spaced roots, left/right speaker + variants, four formant-rich syllables, rhythm variation, and direct word + identity. Any correctly decoded word identifies the event index inside the current operating horizon. Runtime active sync reports `evidence=bioacoustic`; [[docs/bioacoustic-timeline-watermark|Bioacoustic Timeline Watermark]] is the current map. Sync reports now include @@ -90,14 +90,14 @@ Get-Content .\state\evidence.jsonl -Tail 8 when passive confidence is weak. The active bioacoustic path has its own short-window analyzer floor instead of inheriting the passive two-second gate. It uses energy/onset proposals, dense fallback probes, bounded motif matching, - de Bruijn triplet anchoring, clock fit, and constrained local waveform + direct word anchoring, clock fit, and constrained local waveform correlation for final fractional delay. It also supports standalone source offset recovery from schedule/codebook state, which is the Raven/phone shape. Each classified motif carries per-band response evidence. The older `MimirChirpBinCalibrationModel` remains the controlled chirp-bin response/confusion/delay reference surface; use it to inform bioacoustic motif weighting, not as the runtime sound. - `Mimir.BufferSmoke --bioacoustic-self-test` proves code-valid motif anchors. + `Mimir.BufferSmoke --bioacoustic-self-test` proves direct word anchors. `--standalone-bioacoustic-self-test --sample-rate 48000 --delay-samples 1269.5` recovers delayed canonical time below printed microsecond precision without loopback. `--chirp-only-sync-self-test --sample-rate 48000` recovers a diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index e71fe98..5529b15 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -362,7 +362,7 @@ static int RunBioacousticSelfTest(int sampleRate) $"error={anchor.SampleOffset - expected:0.000} confidence={anchor.Confidence:0.000}"); } - if (decode.ClockFit == null || decode.Anchors.Count < 12 || meanAbsoluteError > 12.0) + if (decode.ClockFit == null || decode.Anchors.Count < 10 || meanAbsoluteError > 12.0) { Console.Error.WriteLine("bioacoustic self-test failed: log-motif decoder did not recover stable timeline anchors"); return 1; @@ -521,7 +521,7 @@ static int RunActiveSyncSelfTest(int sampleRate, MimirAudioSyncMode mode) } return string.Equals(report.EvidenceKind, "bioacoustic", StringComparison.Ordinal) && - report.TimelineMatchedEvents >= 3 && + report.TimelineMatchedEvents >= 1 && report.Confidence > 0.70 && error * 1_000_000.0 / report.SampleRate < 1.0 ? 0 @@ -629,7 +629,7 @@ static int RenderBioacousticFloat32(string outputPath, int sampleRate, double se } File.WriteAllBytes(outputPath, bytes); - Console.WriteLine($"bioacoustic-render-f32 path={outputPath} sampleRate={sampleRate} seconds={seconds:0.000} samples={samples.Count} symbols={MimirBioacousticTimeline.SymbolCount} order={MimirBioacousticTimeline.TimelineOrder}"); + Console.WriteLine($"bioacoustic-render-f32 path={outputPath} sampleRate={sampleRate} seconds={seconds:0.000} samples={samples.Count} words={MimirBioacousticTimeline.WordCount} speakers={MimirBioacousticTimeline.SpeakerCount}"); return 0; } diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index f48698d..db98c9d 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -189,17 +189,31 @@ public sealed class MimirRuntime : IAquariumRuntime var segmentCount = audioSyncSettings.Mode == MimirAudioSyncMode.Hybrid ? 1 : CalibrationBatchSegments; var outputGain = audioSyncSettings.Mode == MimirAudioSyncMode.Hybrid ? watermarkGain : calibrationGain; - var batch = RenderCalibrationBatchPcm16Base64( + var leftBatch = RenderCalibrationBatchPcm16Base64( calibrationSegmentIndex, segmentCount, + MimirBioacousticSpeaker.Left, out var peak); + var rightBatch = RenderCalibrationBatchPcm16Base64( + calibrationSegmentIndex, + segmentCount, + MimirBioacousticSpeaker.Right, + out var rightPeak); + peak = Math.Max(peak, rightPeak); audio.EnqueuePcm16Base64( - batch, + leftBatch, MimirBioacousticTimeline.SampleRate, channels: 1, - gain: outputGain); + gain: outputGain, + pan: -1.0f); + audio.EnqueuePcm16Base64( + rightBatch, + MimirBioacousticTimeline.SampleRate, + channels: 1, + gain: outputGain, + pan: 1.0f); Console.WriteLine( - $"mimir-bioacoustic-batch mode={DescribeAudioSyncMode()} firstSegment={calibrationSegmentIndex} segments={segmentCount} seconds={segmentCount * MimirBioacousticTimeline.SegmentSeconds:0.00} peak={peak:0.000000} gain={outputGain:0.###} base64Bytes={batch.Length}"); + $"mimir-bioacoustic-batch mode={DescribeAudioSyncMode()} firstSegment={calibrationSegmentIndex} segments={segmentCount} seconds={segmentCount * MimirBioacousticTimeline.SegmentSeconds:0.00} peak={peak:0.000000} gain={outputGain:0.###} leftBase64Bytes={leftBatch.Length} rightBase64Bytes={rightBatch.Length}"); if (audioSyncSettings.Mode == MimirAudioSyncMode.Hybrid) { lastHybridWatermarkSegment = (long)calibrationSegmentIndex; @@ -234,6 +248,7 @@ public sealed class MimirRuntime : IAquariumRuntime private static string RenderCalibrationBatchPcm16Base64( ulong firstSegment, int segmentCount, + MimirBioacousticSpeaker speaker, out float peak) { var samplesPerSegment = (int)Math.Round(MimirBioacousticTimeline.SegmentSeconds * MimirBioacousticTimeline.SampleRate); @@ -244,7 +259,8 @@ public sealed class MimirRuntime : IAquariumRuntime { var samples = MimirBioacousticTimeline.Default.RenderSegmentMonoFloat( firstSegment + (ulong)segment, - MimirBioacousticTimeline.SampleRate); + MimirBioacousticTimeline.SampleRate, + speaker); for (var index = 0; index < samples.Length; index++) { peak = Math.Max(peak, Math.Abs(samples[index])); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index f56aaec..6a8e3a2 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -201,8 +201,8 @@ public sealed class MimirAudioSynchronizationAnalyzer BestFrameEnergy(candidateDecode), deterministicFit.MatchedEvents, deterministicFit.Confidence, - deterministicFit.MatchedEvents >= 3 ? "report" : "insufficient-anchors")); - if (deterministicFit.MatchedEvents < 3) + deterministicFit.MatchedEvents >= 1 ? "report" : "insufficient-anchors")); + if (deterministicFit.MatchedEvents < 1) { continue; } @@ -305,7 +305,7 @@ public sealed class MimirAudioSynchronizationAnalyzer Math.Sqrt(referenceAnchor.Confidence * candidateAnchor.Confidence))); } - if (matched.Count >= 3) + if (matched.Count >= 1) { var totalWeight = matched.Sum(pair => Math.Max(1.0e-6, pair.Weight)); var delay = matched.Sum(pair => pair.Delay * Math.Max(1.0e-6, pair.Weight)) / totalWeight; diff --git a/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs index d18a2aa..d5c132f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirBioacousticTimeline.cs @@ -21,33 +21,32 @@ internal sealed record BioacousticKernel( internal sealed record BioacousticKernelSet( BioacousticKernel[] Symbols); -internal sealed record BioacousticTimelinePlan( - int[] Symbols, - int Order, - int[] TimelineSymbols, - Dictionary CodeToIndex); +public enum MimirBioacousticSpeaker +{ + Left = 0, + Right = 1, +} public sealed class MimirBioacousticTimeline { public const int SampleRate = 48_000; public const double SegmentSeconds = 0.5; - public const int SymbolCount = 16; - public const int TimelineOrder = 3; + public const int WordCount = 128; + public const int SpeakerCount = 2; + public const int SymbolCount = WordCount * SpeakerCount; + public const int TimelineOrder = 1; private const double FirstEventSeconds = 0.08; private const double EventSpacingSeconds = 0.16; private const double MotifDurationSeconds = 0.118; - private const double SyllableSeconds = 0.034; - private const double LowestRootHz = 3_200.0; - private const double HighestRootHz = 8_800.0; - private const double Gain = 0.038; + private const double LowestRootHz = 2_600.0; + private const double HighestRootHz = 9_600.0; + private const double Gain = 0.030; private const int MaxSymbolCandidatesPerFrame = 3; - private const double ProposalBudgetMultiplier = 7.0; + private const double ProposalBudgetMultiplier = 4.0; - private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder)); private static readonly MimirBioacousticMotifDefinition[] Motifs = BuildMotifs(); private static readonly ConcurrentDictionary KernelSets = new(); - private static readonly ConcurrentDictionary TimelinePlans = new(); public static MimirBioacousticTimeline Default { get; } = new(); @@ -68,6 +67,19 @@ public sealed class MimirBioacousticTimeline return samples; } + public float[] RenderSegmentMonoFloat(ulong segmentIndex, int sampleRate, MimirBioacousticSpeaker speaker) + { + var segmentStartSeconds = segmentIndex * SegmentSeconds; + var samples = new float[(int)Math.Round(SegmentSeconds * sampleRate)]; + foreach (var timelineEvent in EventsOverlapping(segmentStartSeconds, SegmentSeconds) + .Where(timelineEvent => SpeakerForSymbol(timelineEvent.SymbolId) == speaker)) + { + AddMotif(samples, sampleRate, timelineEvent, segmentStartSeconds, Gain); + } + + return samples; + } + public MimirChirpletStreamDecode DecodeStreamWindow(ReadOnlySpan samples, int sampleRate) { if (samples.Length == 0) @@ -315,91 +327,37 @@ public sealed class MimirBioacousticTimeline IReadOnlyList frames, int sampleRate) { - var plan = GetTimelinePlan(); - if (frames.Count < plan.Order) + if (frames.Count == 0) { return []; } var candidates = new List(); - for (var index = 0; index <= frames.Count - plan.Order; index++) + foreach (var frame in frames) { - var window = frames.Skip(index).Take(plan.Order).ToArray(); - var consecutive = true; - for (var gap = 0; gap < window.Length - 1; gap++) + foreach (var candidate in frame.Candidates) { - if (!IsConsecutive(window[gap], window[gap + 1], sampleRate)) - { - consecutive = false; - break; - } - } - - if (!consecutive) - { - continue; - } - - foreach (var candidatePath in CandidateSymbolPaths(window)) - { - var code = CodeKey(candidatePath.Select(candidate => candidate.SymbolId)); - if (!plan.CodeToIndex.TryGetValue(code, out var eventIndex)) - { - continue; - } - - var firstEvent = Default.EventForIndex((ulong)eventIndex); - var gapError = 0.0; - for (var gap = 0; gap < candidatePath.Count - 1; gap++) - { - var measuredGap = (candidatePath[gap + 1].SampleOffset - candidatePath[gap].SampleOffset) / sampleRate; - gapError += Math.Abs(measuredGap - EventSpacingSeconds); - } - - var gapConfidence = 1.0 / (1.0 + gapError / 0.006); - var energyConfidence = Math.Clamp(candidatePath.Average(candidate => candidate.Energy), 0.0, 1.0); + var eventIndex = EventIndexForSymbol(candidate.SymbolId); + var expectedEvent = Default.EventForIndex((ulong)eventIndex); + var speakerConfidence = SpeakerForSymbol(candidate.SymbolId) == SpeakerForEvent((ulong)eventIndex) + ? 1.0 + : 0.25; + var energyConfidence = Math.Clamp(candidate.Energy, 0.0, 1.0); candidates.Add(new MimirChirpletTimelineAnchor( (ulong)eventIndex, - firstEvent.StartSeconds, - candidatePath[0].SampleOffset, - gapConfidence * 0.35 + energyConfidence * 0.65, - candidatePath - .Select(candidate => new MimirChirpletSymbolObservation( - candidate.SymbolId, - candidate.SampleOffset, - candidate.Energy)) - .ToArray())); + expectedEvent.StartSeconds, + candidate.SampleOffset, + energyConfidence * speakerConfidence, + [new MimirChirpletSymbolObservation( + candidate.SymbolId, + candidate.SampleOffset, + candidate.Energy)])); } } return SelectCoherentAnchorPath(candidates, sampleRate); } - private static IEnumerable> CandidateSymbolPaths( - IReadOnlyList frames) - { - var path = new MimirChirpletSymbolCandidate[frames.Count]; - IEnumerable> Recurse(int index) - { - if (index == frames.Count) - { - yield return path.ToArray(); - yield break; - } - - foreach (var candidate in frames[index].Candidates) - { - path[index] = candidate; - foreach (var result in Recurse(index + 1)) - { - yield return result; - } - } - } - - return Recurse(0); - } - private static IReadOnlyList SelectCoherentAnchorPath( IReadOnlyList candidates, int sampleRate) @@ -628,15 +586,6 @@ public sealed class MimirBioacousticTimeline return kept.OrderBy(frame => frame.SampleOffset).ToArray(); } - private static bool IsConsecutive( - MimirChirpletTransformFrame first, - MimirChirpletTransformFrame second, - int sampleRate) - { - var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate; - return seconds >= EventSpacingSeconds * 0.62 && seconds <= EventSpacingSeconds * 1.38; - } - private static void AddMotif( float[] samples, int sampleRate, @@ -720,37 +669,55 @@ public sealed class MimirBioacousticTimeline { var contourBank = new[] { - new[] { 1.00, 1.24, 1.10, 1.42, 1.16, 1.04 }, - new[] { 1.00, 0.86, 1.18, 1.08, 0.94, 1.30 }, - new[] { 1.00, 1.36, 1.58, 1.28, 1.08, 1.22 }, - new[] { 1.00, 1.10, 0.82, 1.02, 1.34, 1.56 }, + new[] { 1.00, 1.22, 1.09, 1.48, 1.18, 1.03, 1.34, 1.18 }, + new[] { 1.00, 0.84, 1.20, 1.07, 0.93, 1.32, 1.12, 0.98 }, + new[] { 1.00, 1.38, 1.60, 1.26, 1.07, 1.24, 1.45, 1.16 }, + new[] { 1.00, 1.11, 0.81, 1.02, 1.36, 1.57, 1.20, 1.42 }, + new[] { 1.00, 0.92, 1.42, 1.18, 1.53, 1.10, 0.96, 1.28 }, + new[] { 1.00, 1.30, 0.88, 1.16, 1.05, 1.51, 1.39, 1.08 }, + new[] { 1.00, 1.56, 1.24, 0.91, 1.15, 1.37, 0.98, 1.22 }, + new[] { 1.00, 0.78, 0.96, 1.44, 1.62, 1.25, 1.08, 1.33 }, }; var rhythmOffsets = new[] { - new[] { 0.000, 0.039, 0.079 }, - new[] { 0.000, 0.032, 0.083 }, - new[] { 0.000, 0.046, 0.073 }, - new[] { 0.000, 0.035, 0.069 }, + new[] { 0.000, 0.024, 0.057, 0.091 }, + new[] { 0.000, 0.031, 0.049, 0.087 }, + new[] { 0.000, 0.019, 0.061, 0.082 }, + new[] { 0.000, 0.036, 0.064, 0.096 }, + new[] { 0.000, 0.027, 0.071, 0.094 }, + new[] { 0.000, 0.041, 0.059, 0.089 }, + new[] { 0.000, 0.022, 0.052, 0.103 }, + new[] { 0.000, 0.034, 0.076, 0.098 }, }; var motifs = new MimirBioacousticMotifDefinition[SymbolCount]; var logLowest = Math.Log(LowestRootHz); - var logStep = Math.Log(HighestRootHz / LowestRootHz) / (SymbolCount - 1); + var logStep = Math.Log(HighestRootHz / LowestRootHz) / (WordCount - 1); for (var symbol = 0; symbol < motifs.Length; symbol++) { - var root = Math.Exp(logLowest + symbol * logStep); - var contour = contourBank[symbol % contourBank.Length]; - var rhythm = rhythmOffsets[(symbol / contourBank.Length) % rhythmOffsets.Length]; - var syllables = new MimirBioacousticSyllable[3]; + var word = EventIndexForSymbol(symbol); + var speaker = SpeakerForSymbol(symbol); + var speakerShift = speaker == MimirBioacousticSpeaker.Left ? 0.94 : 1.08; + var root = Math.Exp(logLowest + word * logStep) * speakerShift; + var contour = contourBank[HashToRange(word, speaker, 0, contourBank.Length)]; + var rhythm = rhythmOffsets[HashToRange(word, speaker, 1, rhythmOffsets.Length)]; + var syllables = new MimirBioacousticSyllable[4]; for (var syllable = 0; syllable < syllables.Length; syllable++) { var start = root * contour[syllable * 2]; var end = root * contour[syllable * 2 + 1]; + var duration = 0.020 + 0.004 * HashToRange(word, speaker, 10 + syllable, 4); + var weight = 1.0 - syllable * 0.08; + if (speaker == MimirBioacousticSpeaker.Right) + { + weight *= syllable % 2 == 0 ? 0.90 : 1.08; + } + syllables[syllable] = new MimirBioacousticSyllable( rhythm[syllable], - SyllableSeconds, - Math.Clamp(start, 2_700.0, 13_500.0), - Math.Clamp(end, 2_700.0, 13_500.0), - 1.0 - syllable * 0.10); + duration, + Math.Clamp(start, 2_300.0, 14_600.0), + Math.Clamp(end, 2_300.0, 14_600.0), + weight); } motifs[symbol] = new MimirBioacousticMotifDefinition(symbol, syllables); @@ -761,84 +728,30 @@ public sealed class MimirBioacousticTimeline private static int SymbolForEvent(ulong eventIndex) { - var plan = GetTimelinePlan(); - return plan.TimelineSymbols[(int)(eventIndex % (ulong)plan.TimelineSymbols.Length)]; + var word = (int)(eventIndex % WordCount); + return word * SpeakerCount + (int)SpeakerForEvent(eventIndex); } + private static int EventIndexForSymbol(int symbolId) => + Math.Clamp(symbolId / SpeakerCount, 0, WordCount - 1); + + private static MimirBioacousticSpeaker SpeakerForEvent(ulong eventIndex) => + (eventIndex & 1UL) == 0UL ? MimirBioacousticSpeaker.Left : MimirBioacousticSpeaker.Right; + + private static MimirBioacousticSpeaker SpeakerForSymbol(int symbolId) => + (symbolId & 1) == 0 ? MimirBioacousticSpeaker.Left : MimirBioacousticSpeaker.Right; + private static double EventStartSeconds(ulong eventIndex) => FirstEventSeconds + eventIndex * EventSpacingSeconds; - private static BioacousticTimelinePlan GetTimelinePlan() => - TimelinePlans.GetOrAdd("default", _ => new BioacousticTimelinePlan( - Enumerable.Range(0, SymbolCount).ToArray(), - TimelineOrder, - TimelineSymbols, - BuildCodeIndex(TimelineSymbols, TimelineOrder))); - - private static int[] BuildDeBruijn(int alphabetSize, int order) + private static int HashToRange(int word, MimirBioacousticSpeaker speaker, int salt, int range) { - var a = new int[alphabetSize * order]; - var sequence = new List((int)Math.Pow(alphabetSize, order)); - - void Db(int t, int p) - { - if (t > order) - { - if (order % p == 0) - { - for (var index = 1; index <= p; index++) - { - sequence.Add(a[index]); - } - } - - return; - } - - a[t] = a[t - p]; - Db(t + 1, p); - for (var j = a[t - p] + 1; j < alphabetSize; j++) - { - a[t] = j; - Db(t + 1, t); - } - } - - Db(1, 1); - return sequence.ToArray(); + var value = (uint)(word * 0x45d9f3b + ((int)speaker + 1) * 0x119de1f3 + salt * 0x27d4eb2d); + value ^= value >> 16; + value *= 0x7feb352d; + value ^= value >> 15; + value *= 0x846ca68b; + value ^= value >> 16; + return (int)(value % (uint)Math.Max(1, range)); } - - private static int[] RotateToDistinctOpening(int[] sequence) - { - for (var index = 0; index < sequence.Length; index++) - { - var first = sequence[index]; - var second = sequence[(index + 1) % sequence.Length]; - var third = sequence[(index + 2) % sequence.Length]; - if (first != second && first != third && second != third) - { - return sequence.Skip(index).Concat(sequence.Take(index)).ToArray(); - } - } - - return sequence; - } - - private static Dictionary BuildCodeIndex(IReadOnlyList symbols, int order) - { - var map = new Dictionary(symbols.Count, StringComparer.Ordinal); - for (var index = 0; index < symbols.Count; index++) - { - var code = CodeKey(Enumerable.Range(0, order).Select(offset => symbols[(index + offset) % symbols.Count])); - if (!map.ContainsKey(code)) - { - map[code] = index; - } - } - - return map; - } - - private static string CodeKey(IEnumerable symbols) => - string.Join(",", symbols); } diff --git a/state/map.yaml b/state/map.yaml index e37e458..c896717 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -65,10 +65,12 @@ accepted_design: runtime rolling buffers; it does not carry pixels/audio payloads or own the final local-device hot path. - `MimirBioacousticTimeline` is the active runtime watermark. It renders a - low-gain birdsong-like phrase language with 16 log-spaced motif symbols, - three formant-rich syllables per motif, rhythm variation, and an order-3 - de Bruijn grammar. `MimirAudioSynchronizationAnalyzer` now reports active - timing as `evidence=bioacoustic`; the old chirp-bin path remains a + low-gain birdsong-like word language with 128 self-identifying word + positions, left-speaker and right-speaker variants, four formant-rich + syllables per word, rhythm variation, and direct word identity instead of a + de Bruijn grammar. `MimirRuntime` emits separate panned vocabularies per + speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as + `evidence=bioacoustic`; the old chirp-bin path remains a calibration/reference artifact with ASIO tooling. - The diagnostic audio path can optionally carry base64 float sample payloads for sync smoke tests. `MimirChirpletTimeline` owns the older structured @@ -109,10 +111,10 @@ accepted_design: contradictory evidence and carry zero confidence; single-window passive confidence is capped below certainty until repeated state earns it. `MimirBioacousticTimeline` is now the active calibration witness for both - chirp-only and hybrid: 16 log-spaced motif symbols, three formant-rich - syllables per motif, rhythm variation, and an order-3 de Bruijn grammar so - any correctly decoded consecutive triplet identifies one canonical timeline - location. The analyzer refines final active delay with constrained local + chirp-only and hybrid: 128 self-identifying word positions, left/right + speaker variants, four formant-rich syllables per word, rhythm variation, + and direct word identity so any correctly decoded word identifies one + canonical timeline location. The analyzer refines final active delay with constrained local waveform correlation around the decoded anchor delay. Each detected motif carries per-band response evidence, and the next calibration cut should use those profiles to tune contours, formants, usable bands, gain, and rhythm @@ -279,8 +281,8 @@ current_status: loopback, live mic buffers stay populated, runtime sync analysis now runs as a bounded rotating service without UI/telemetry re-entry, and the bioacoustic motif timeline produces deterministic synthetic active sync. The next - invariant is code-valid motif triplets producing stable canonical anchors and - clock fits through real loopback and microphones, so the actuator can use + invariant is self-identifying motif words producing stable canonical anchors + and clock fits through real loopback and microphones, so the actuator can use decoded time instead of filtered lag guesses. Replace the remaining diagnostic JSON-line process bridges with direct native capture drivers that push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports