diff --git a/docs/audio-field.md b/docs/audio-field.md index c3d991b..b32036b 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -64,10 +64,12 @@ quiet birdsong texture than a repeated sweep. The intended decoder is a constrained chirplet transform, not a generic time-frequency explorer and not an outlier filter around bad guesses. Mimir owns the emitter, so the receiver projects each mic stream against the known chirplet -dictionary, produces symbol likelihood events, and decodes every valid -consecutive triplet through the de Bruijn map. A decoded triplet is a canonical -timeline anchor: observed sample offset `S` corresponds to emitted event time -`T`. A stream of anchors fits the source clock directly: +dictionary, produces transform frames with multiple symbol candidates, and +scores candidate triplets through the de Bruijn map. A triplet only becomes a +canonical timeline anchor when its symbol likelihoods, measured inter-chirp +gaps, and neighboring anchors agree on one local sample clock. A decoded anchor +means observed sample offset `S` corresponds to emitted event time `T`. A stream +of anchors fits the source clock directly: ```text observed_sample = source_offset + canonical_seconds * effective_sample_rate diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 5a0780f..17d3330 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -50,9 +50,10 @@ a named invariant that the native runtime cannot protect yet. horizon. Symbol identity is carried by start band, glide shape, duration, and following inter-chirp gap. - `MimirChirpletStreamDecoder` is the first constrained chirplet-transform - receiver. It owns a bounded PCM window, emits symbol observations, decodes - code-valid triplet anchors onto the canonical timeline, and fits a per-source - sample clock from those anchors. + receiver. It owns a bounded PCM window, emits transform frames with multiple + symbol candidates, decodes code-valid triplet anchors through a local trellis + that requires gap and clock coherence, and fits a per-source sample clock from + those anchors. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: sample-bearing audio blocks are resampled into the Scarlett loopback timeline. The analyzer now prefers decoded triplet timeline anchors and derives delay diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 96c86e1..88f4787 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -101,10 +101,11 @@ matched event indices when the symbol decoder succeeds, and can build a provisional aligned mono frame for channels that clear the confidence gate. Reports now carry fractional delay and per-band matched energy. The first `MimirChirpletStreamDecoder` exists as the constrained chirplet-transform -receiver: it turns a bounded PCM window into symbol observations, every -code-valid triplet anchor it can prove, and an affine source clock fit. The -analyzer prefers matched canonical anchors before falling back to the older -chirplet-energy lag search. The hub also owns cached reports plus smoothed +receiver: it turns a bounded PCM window into transform frames with multiple +symbol candidates, code-valid triplet anchors selected through gap/clock +coherence, and an affine source clock fit. The analyzer prefers matched +canonical anchors before falling back to the older chirplet-energy lag search. +The hub also owns cached reports plus smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` runs analysis online as a bounded rotating service and can print live telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are passive readers of cached diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index d85f1ee..418036d 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -72,9 +72,10 @@ Get-Content .\state\evidence.jsonl -Tail 8 symbol/event decoding, and per-band response hooks. Sync reports now include fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and delay-slope/SRO ppm. `MimirChirpletStreamDecoder` - is the first constrained chirplet-transform receiver: it emits symbol - observations, code-valid triplet anchors, and a source clock fit from a - bounded PCM window. `MimirRuntime` runs sync analysis as a bounded rotating + is the first constrained chirplet-transform receiver: it emits transform + frames with multiple symbol candidates, selects code-valid triplet anchors + through gap/clock coherence, and fits a source clock from a bounded PCM + window. `MimirRuntime` runs sync analysis as a bounded rotating service and caches reports/states for UI and telemetry; readouts must stay passive. Actual Mimir.App testing proves Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and produces confident online sync states, diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index bb3bd09..e0cf6c2 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -76,7 +76,7 @@ public sealed class MimirAudioSynchronizationAnalyzer var referenceDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(referenceWindow, referenceBlock.SampleRate); var candidateDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(candidateWindow, referenceBlock.SampleRate); var deterministicFit = EstimateDelayFromDecodedTimeline(referenceDecode, candidateDecode); - if (deterministicFit.MatchedEvents >= 1) + if (deterministicFit.MatchedEvents >= 3) { var decodedDelaySamples = deterministicFit.DelaySamples; var decodedBandResponses = MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate); @@ -189,7 +189,7 @@ public sealed class MimirAudioSynchronizationAnalyzer Math.Sqrt(referenceAnchor.Confidence * candidateAnchor.Confidence))); } - if (matched.Count >= 1) + if (matched.Count >= 3) { var totalWeight = matched.Sum(pair => Math.Max(1.0e-6, pair.Weight)); var delay = matched.Sum(pair => pair.Delay * Math.Max(1.0e-6, pair.Weight)) / totalWeight; diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs index 0ed2d59..8720e97 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs @@ -1,5 +1,17 @@ namespace Mimir.Runtime.Synchronization; +public sealed record MimirChirpletSymbolCandidate( + int SymbolId, + double Energy); + +public sealed record MimirChirpletTransformFrame( + double SampleOffset, + IReadOnlyList Candidates) +{ + public MimirChirpletSymbolCandidate BestCandidate => + Candidates.OrderByDescending(candidate => candidate.Energy).First(); +} + public sealed record MimirChirpletSymbolObservation( int SymbolId, double SampleOffset, @@ -24,6 +36,7 @@ public sealed record MimirChirpletClockFit( } public sealed record MimirChirpletStreamDecode( + IReadOnlyList Frames, IReadOnlyList Symbols, IReadOnlyList Anchors, MimirChirpletClockFit? ClockFit); diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs index 38094f0..83de69f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs @@ -74,6 +74,9 @@ public sealed class MimirChirpletStreamDecoder var clock = decode.ClockFit == null ? null : decode.ClockFit with { SourceOffsetSamples = decode.ClockFit.SourceOffsetSamples + originSample }; - return new MimirChirpletStreamDecode(symbols, anchors, clock); + var frames = decode.Frames + .Select(frame => frame with { SampleOffset = frame.SampleOffset + originSample }) + .ToArray(); + return new MimirChirpletStreamDecode(frames, symbols, anchors, clock); } } diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs index bcf2a31..2beba7b 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs @@ -46,6 +46,7 @@ public sealed class MimirChirpletTimeline private const double MaxEventGapSeconds = 0.162; private const double MaxEventDurationSeconds = 0.078; private const double Gain = 0.090; + private const int MaxSymbolCandidatesPerFrame = 8; private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder)); private static readonly Dictionary TripleToIndex = BuildTripleIndex(TimelineSymbols); @@ -138,18 +139,19 @@ public sealed class MimirChirpletTimeline { if (samples.Length == 0) { - return new MimirChirpletStreamDecode([], [], null); + return new MimirChirpletStreamDecode([], [], [], null); } - var symbols = DetectSymbolEvents(samples, sampleRate, Math.Max(1, sampleRate / 200)) - .Select(observation => new MimirChirpletSymbolObservation( - observation.SymbolId, - observation.SampleOffset, - observation.Energy)) + var frames = DetectTransformFrames(samples, sampleRate, Math.Max(1, sampleRate / 200)); + var symbols = frames + .Select(frame => new MimirChirpletSymbolObservation( + frame.BestCandidate.SymbolId, + frame.SampleOffset, + frame.BestCandidate.Energy)) .ToArray(); - var anchors = PlaceAllTripletAnchors(symbols, sampleRate); + var anchors = DecodeTrellisAnchors(frames, sampleRate); var clock = FitClock(anchors, sampleRate); - return new MimirChirpletStreamDecode(symbols, anchors, clock); + return new MimirChirpletStreamDecode(frames, symbols, anchors, clock); } public MimirChirpletTimelinePlacement DecodeReferenceWindow( @@ -277,6 +279,102 @@ public sealed class MimirChirpletTimeline return SuppressNearbyDetections(candidates, sampleRate); } + private IReadOnlyList DetectTransformFrames( + ReadOnlySpan samples, + int sampleRate, + int hopSamples) + { + if (samples.Length == 0) + { + return []; + } + + var coarse = ContrastNormalize(BuildEnvelopeEnergyTrace(samples, sampleRate, hopSamples)); + if (coarse.Length <= 2) + { + return []; + } + + var frames = new List(); + for (var frame = 1; frame < coarse.Length - 1; frame++) + { + if (coarse[frame] < 0.055 || + coarse[frame] < coarse[frame - 1] || + coarse[frame] < coarse[frame + 1]) + { + continue; + } + + var transformFrame = ClassifySymbolsAt( + samples, + sampleRate, + frame * hopSamples, + Math.Max(hopSamples * 3, sampleRate / 100)); + if (transformFrame != null) + { + frames.Add(transformFrame); + } + } + + return SuppressNearbyFrames(frames, sampleRate); + } + + private MimirChirpletTransformFrame? ClassifySymbolsAt( + ReadOnlySpan samples, + int sampleRate, + int predictedOffset, + int searchRadiusSamples) + { + var candidates = new List<(MimirChirpletSymbolCandidate Candidate, int Offset)>(SymbolTones.Length); + for (var symbol = 0; symbol < SymbolTones.Length; symbol++) + { + var tone = SymbolTones[symbol]; + var kernel = sampleRate == SampleRate ? symbolKernels[symbol] : RenderToneKernel(tone, sampleRate); + if (kernel.Length == 0 || samples.Length < kernel.Length) + { + continue; + } + + var start = Math.Max(0, predictedOffset - searchRadiusSamples); + var end = Math.Min(samples.Length - kernel.Length, predictedOffset + searchRadiusSamples); + var bestEnergy = 0.0; + var bestOffset = predictedOffset; + var step = Math.Max(1, searchRadiusSamples / 8); + for (var offset = start; offset <= end; offset += step) + { + var energy = DirectionalMatchedEnergy(samples.Slice(offset, kernel.Length), kernel); + if (energy > bestEnergy) + { + bestEnergy = energy; + bestOffset = offset; + } + } + + if (bestEnergy >= 0.08) + { + candidates.Add((new MimirChirpletSymbolCandidate(symbol, bestEnergy), bestOffset)); + } + } + + if (candidates.Count == 0) + { + return null; + } + + var ordered = candidates + .OrderByDescending(candidate => candidate.Candidate.Energy) + .Take(MaxSymbolCandidatesPerFrame) + .ToArray(); + if (ordered[0].Candidate.Energy < 0.25) + { + return null; + } + + return new MimirChirpletTransformFrame( + ordered[0].Offset, + ordered.Select(candidate => candidate.Candidate).ToArray()); + } + private MimirChirpletEventObservation? ClassifySymbolAt( ReadOnlySpan samples, int sampleRate, @@ -338,6 +436,30 @@ public sealed class MimirChirpletTimeline return kept.OrderBy(observation => observation.SampleOffset).ToArray(); } + private static IReadOnlyList SuppressNearbyFrames( + IReadOnlyList frames, + int sampleRate) + { + if (frames.Count == 0) + { + return []; + } + + var minimumSpacingSamples = MinEventGapSeconds * sampleRate * 0.55; + var kept = new List(); + foreach (var frame in frames.OrderByDescending(frame => frame.BestCandidate.Energy)) + { + if (kept.Any(existing => Math.Abs(existing.SampleOffset - frame.SampleOffset) < minimumSpacingSamples)) + { + continue; + } + + kept.Add(frame); + } + + return kept.OrderBy(frame => frame.SampleOffset).ToArray(); + } + private static MimirChirpletTimelinePlacement PlaceObservedSymbols( IReadOnlyList observations, int sampleRate) @@ -442,6 +564,124 @@ public sealed class MimirChirpletTimeline .ToArray(); } + private static IReadOnlyList DecodeTrellisAnchors( + IReadOnlyList frames, + int sampleRate) + { + if (frames.Count < TimelineOrder) + { + return []; + } + + var candidates = new List(); + for (var index = 0; index <= frames.Count - TimelineOrder; index++) + { + var firstFrame = frames[index]; + var secondFrame = frames[index + 1]; + var thirdFrame = frames[index + 2]; + if (!IsConsecutiveByTime(firstFrame, secondFrame, sampleRate) || + !IsConsecutiveByTime(secondFrame, thirdFrame, sampleRate)) + { + continue; + } + + foreach (var first in firstFrame.Candidates) + foreach (var second in secondFrame.Candidates) + foreach (var third in thirdFrame.Candidates) + { + var code = TripleCode(first.SymbolId, second.SymbolId, third.SymbolId); + if (!TripleToIndex.TryGetValue(code, out var eventIndex)) + { + continue; + } + + var firstEvent = Default.EventForIndex((ulong)eventIndex); + var secondEvent = Default.EventForIndex((ulong)eventIndex + 1UL); + var thirdEvent = Default.EventForIndex((ulong)eventIndex + 2UL); + var measuredGapA = (secondFrame.SampleOffset - firstFrame.SampleOffset) / sampleRate; + var measuredGapB = (thirdFrame.SampleOffset - secondFrame.SampleOffset) / sampleRate; + var expectedGapA = secondEvent.StartSeconds - firstEvent.StartSeconds; + var expectedGapB = thirdEvent.StartSeconds - secondEvent.StartSeconds; + var gapError = Math.Abs(measuredGapA - expectedGapA) + Math.Abs(measuredGapB - expectedGapB); + var gapConfidence = 1.0 / (1.0 + gapError / 0.004); + if (gapConfidence < 0.50) + { + continue; + } + + var energyConfidence = Math.Clamp((first.Energy + second.Energy + third.Energy) / 3.0, 0.0, 1.0); + candidates.Add(new MimirChirpletTimelineAnchor( + (ulong)eventIndex, + firstEvent.StartSeconds, + firstFrame.SampleOffset, + gapConfidence * 0.70 + energyConfidence * 0.30, + [ + new MimirChirpletSymbolObservation(first.SymbolId, firstFrame.SampleOffset, first.Energy), + new MimirChirpletSymbolObservation(second.SymbolId, secondFrame.SampleOffset, second.Energy), + new MimirChirpletSymbolObservation(third.SymbolId, thirdFrame.SampleOffset, third.Energy), + ])); + } + } + + return SelectCoherentAnchorPath(candidates, sampleRate); + } + + private static IReadOnlyList SelectCoherentAnchorPath( + IReadOnlyList candidates, + int sampleRate) + { + if (candidates.Count == 0) + { + return []; + } + + List bestPath = []; + var bestScore = double.NegativeInfinity; + foreach (var seed in candidates.OrderByDescending(anchor => anchor.Confidence).Take(64)) + { + var seedOffset = seed.SampleOffset - seed.TimelineSeconds * sampleRate; + var path = candidates + .Where(anchor => + Math.Abs((anchor.SampleOffset - anchor.TimelineSeconds * sampleRate) - seedOffset) <= sampleRate * 0.008) + .GroupBy(anchor => anchor.EventIndex) + .Select(group => group.OrderByDescending(anchor => anchor.Confidence).First()) + .OrderBy(anchor => anchor.EventIndex) + .ToList(); + var clock = FitClock(path, sampleRate); + if (clock == null) + { + continue; + } + + var score = path.Sum(anchor => anchor.Confidence) + + Math.Min(path.Count, 12) * 0.20 - + clock.MeanAbsoluteErrorSamples / Math.Max(1.0, sampleRate * 0.001); + if (score > bestScore) + { + bestScore = score; + bestPath = path; + } + } + + if (bestPath.Count == 0) + { + return []; + } + + var finalClock = FitClock(bestPath, sampleRate); + if (finalClock == null) + { + return []; + } + + return bestPath + .Where(anchor => + Math.Abs(anchor.SampleOffset - finalClock.SampleForTimelineSeconds(anchor.TimelineSeconds)) <= + Math.Max(24.0, sampleRate * 0.002)) + .OrderBy(anchor => anchor.SampleOffset) + .ToArray(); + } + private static MimirChirpletClockFit? FitClock( IReadOnlyList anchors, int sampleRate) @@ -519,6 +759,15 @@ public sealed class MimirChirpletTimeline return seconds >= MinEventGapSeconds * 0.55 && seconds <= MaxEventGapSeconds * 1.45; } + private static bool IsConsecutiveByTime( + MimirChirpletTransformFrame first, + MimirChirpletTransformFrame second, + int sampleRate) + { + var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate; + return seconds >= MinEventGapSeconds * 0.55 && seconds <= MaxEventGapSeconds * 1.45; + } + private static MimirChirpletEventObservation PlaceObservation( MimirChirpletEventObservation observation, ulong eventIndex) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 8151582..ab4cbfe 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -44,3 +44,4 @@ {"ts":"2026-05-22T14:48:00+01:00","type":"verification","status":"partial","note":"Attempted a 20-second real-world Mimir.App chirplet sync run with all configured local mics. The headless path only emitted one telemetry tick before runtime updates stopped. Switching to the normal app path produced telemetry through t=4.26s with all six audio buffers live and sync states for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Timeline-symbol refinement appeared on Focusrite, Kiyo, and one PS3 Eye, but delay estimates were unstable and the app did not continue telemetry to 20 seconds. Replaced the coarse acquisition trace with envelope energy to avoid matched-filter scans in the hot acquisition path; symbol decoding remains the refinement step."} {"ts":"2026-05-22T15:20:00+01:00","type":"implementation","status":"partial","note":"Removed hidden re-entrant synchronization analysis from the live app path. MimirRuntime now calls a bounded rotating AnalyzeAudioSynchronizationStep every 0.1s, MimirSynchronizationHub caches per-source reports/states, and UI/telemetry read cached state instead of invoking the analyzer or BuildAlignedAudioFrame. FitDelayFromEvents now collapses duplicate candidate event observations by strongest energy, fixing the real app crash 'An item with the same key has already been added. Key: 0'. Dotnet build passed. A visible Mimir.App run with all configured mics no longer crashed at the earlier 4s wall and produced telemetry through runtime t=17.28s in a 22s wall-time harness; it did not reach a full 20 runtime seconds because startup consumed wall time. All non-loopback mics produced cached reports/states, but delay estimates remained unstable and timeline event refinement was intermittent, so the next live invariant is robust outlier rejection/state filtering before any actuator drives resampling."} {"ts":"2026-05-22T16:05:00+01:00","type":"implementation","status":"partial","note":"Documented the intended decoder as a constrained chirplet transform: known dictionary projection to symbol observations, code-valid de Bruijn triplets to canonical timeline anchors, anchors to affine source clock fits, and delay/SRO from comparing loopback and mic clock fits. Added MimirChirpletStreamDecode records and MimirChirpletStreamDecoder with a bounded PCM window, absolute sample offsets, triplet anchors, and clock fits. MimirAudioSynchronizationAnalyzer now prefers matched decoded timeline anchors before falling back to the older chirplet-energy lag search. Synthetic self-test over two seconds of emitted timeline audio currently decodes 12 symbols and one correct anchor for event 9 at sample 47520 with clock 48000 Hz; this proves the new spine but also proves the symbol likelihood/codebook layer is not finished enough to place every triplet yet."} +{"ts":"2026-05-22T16:35:00+01:00","type":"implementation","status":"partial","note":"Replaced one-best-symbol triplet placement with a constrained transform/trellis pass. DecodeStreamWindow now emits transform frames with multiple symbol candidates per chirp, enumerates candidate triplets through the de Bruijn map, scores measured inter-chirp gaps, and keeps the coherent local clock path rather than accepting any random valid triplet. Synthetic two-second emitted timeline check improved from one anchor to 11 coherent anchors, including events 0-9, with fitted clock 47978.350 Hz, confidence 0.885, and 8.741 sample MAE. This is the right decoder shape; remaining work is symbol/codebook separation and sub-frame timing precision so the fit lands exactly at 48000 Hz and covers every emitted triplet."} diff --git a/state/map.yaml b/state/map.yaml index 976bbbd..0bc893b 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -73,9 +73,9 @@ accepted_design: inter-chirp gap. Any three consecutive correctly detected symbols identify a timeline event within the current operating horizon. `MimirChirpletStreamDecoder` is the first constrained chirplet-transform receiver: it owns a bounded PCM - window, emits symbol observations, decodes code-valid triplet anchors onto - the canonical timeline, and fits a per-source sample clock from those - anchors. `MimirAudioSynchronizationAnalyzer` now treats + window, emits transform frames with multiple symbol candidates, decodes + code-valid triplet anchors through gap/clock coherence, and fits a + per-source sample clock from those anchors. `MimirAudioSynchronizationAnalyzer` now treats `loopback-scarlett-speakers` as timing authority, prefers delay from matched decoded timeline anchors, and keeps the older chirplet-energy lag search as a diagnostic fallback while the decoder is cut over. Positive delay means