From ea09212d4952c0191d1bcecb1b1ed3be11ba087b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 22 May 2026 14:53:26 +0100 Subject: [PATCH] Start chirplet timeline decoder --- docs/audio-field.md | 33 ++-- docs/implementation-plan.md | 19 ++- notes/current-system-map.md | 25 +-- notes/fresh-workspace-handoff.md | 19 ++- .../MimirAudioSynchronizationAnalyzer.cs | 84 +++++++++- .../MimirChirpletStreamDecode.cs | 29 ++++ .../MimirChirpletStreamDecoder.cs | 79 +++++++++ .../Synchronization/MimirChirpletTimeline.cs | 151 +++++++++++++++++- state/evidence.jsonl | 1 + state/map.yaml | 27 ++-- 10 files changed, 415 insertions(+), 52 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs create mode 100644 src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs diff --git a/docs/audio-field.md b/docs/audio-field.md index 3b37f6c..c3d991b 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -61,13 +61,24 @@ the following inter-chirp gap all carry code. The point is not ornament. The timing code is carried by both frequency and rhythm, so it behaves more like a quiet birdsong texture than a repeated sweep. -`MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the -loopback sample-rate timeline, uses cheap coarse chirplet energy for acquisition, -then lets the symbol decoder refine delay from matched event indices. The lag -search is wide enough for ordinary remote/network delay, not just local -speaker-to-mic delay. Reports carry rounded integer delay, fractional delay from -parabolic peak interpolation, and the count/confidence of timeline-symbol events -used to refine the report. +The intended decoder is a constrained chirplet transform, not a generic +time-frequency explorer and not an outlier filter around bad guesses. Mimir owns +the emitter, so the receiver projects each mic stream against the known chirplet +dictionary, produces symbol likelihood events, and decodes every valid +consecutive triplet through the de Bruijn map. A decoded triplet is a canonical +timeline anchor: observed sample offset `S` corresponds to emitted event time +`T`. A stream of anchors fits the source clock directly: + +```text +observed_sample = source_offset + canonical_seconds * effective_sample_rate +``` + +Delay and sampling-rate offset are derived by comparing the loopback clock fit +to each mic clock fit over common canonical time. Wide cross-correlation is only +a diagnostic fallback while the transform decoder is being cut over. The state +tracker is not allowed to launder invalid codewords into plausible timing. +Reports carry rounded integer delay, fractional delay, and the count/confidence +of timeline-symbol anchors used to derive the report. `MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned mono frame: loopback is always channel zero, and other channels enter only when @@ -111,10 +122,12 @@ flowchart TD D --> F["mic rolling buffers"] E --> G["matched chirplet traces"] F --> G - G --> H["delay observations"] - H --> L["smoothed sync state + SRO"] + G --> H["symbol likelihood events"] + H --> L["triplet timeline anchors"] + L --> M["per-source clock fit"] + M --> N["delay + SRO"] G --> I["per-band response estimates"] - L --> J["fractional delay / resampler actuator"] + N --> J["fractional delay / resampler actuator"] I --> K["frequency response normalization"] ``` diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index f3d14fb..5a0780f 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -49,11 +49,15 @@ a named invariant that the native runtime cannot protect yet. detected symbols identify a timeline event inside the current operating horizon. Symbol identity is carried by start band, glide shape, duration, and following inter-chirp gap. +- `MimirChirpletStreamDecoder` is the first constrained chirplet-transform + receiver. It owns a bounded PCM window, emits symbol observations, decodes + code-valid triplet anchors onto the canonical timeline, and fits a per-source + sample clock from those anchors. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: - sample-bearing audio blocks are resampled into the Scarlett loopback timeline - and compared by chirplet-energy delay estimation with fractional peak - interpolation. When timeline symbols are decoded, matched event indices refine - the delay estimate and are exposed in telemetry. + sample-bearing audio blocks are resampled into the Scarlett loopback timeline. + The analyzer now prefers decoded triplet timeline anchors and derives delay + from matched canonical events. The older chirplet-energy lag search remains a + diagnostic fallback while the decoder is being cut over. - `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional aligned mono frame for loopback-referenced audio channels that clear the confidence gate. @@ -103,10 +107,9 @@ a named invariant that the native runtime cannot protect yet. fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. The current aligned frame is integer-delay only and should not pretend to be the final DSP path. - First, harden the online state filter so the continuous chirplet lock stops - producing jumpy delay estimates. The runtime sync cadence is now bounded and - survives longer real app runs, but the state filter still accepts outliers - too readily. + First, finish the constrained chirplet-transform decoder so every correctly + heard triplet becomes a deterministic timeline anchor and the state tracker no + longer has to launder ambiguous lag guesses. 5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 6. Move GPU feature extraction, fusion, material fitting, render budgeting, and diff --git a/notes/current-system-map.md b/notes/current-system-map.md index e41fdcf..96c86e1 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -99,16 +99,21 @@ current operating horizon. Mimir continuously queues that timeline through Aquarium audio, uses coarse chirplet energy for acquisition, refines delay from matched event indices when the symbol decoder succeeds, and can build a provisional aligned mono frame for channels that clear the confidence gate. -Reports now carry fractional delay and per-band matched energy. The hub also -owns cached reports plus smoothed per-source sync state with delay-slope/SRO in -ppm. `MimirRuntime` runs analysis online as a bounded rotating service and can -print live telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are -passive readers of cached sync state; they must not invoke the analyzer. Current -app testing proves loopback wakeup, live mic buffers, and confident online sync -states, but delay estimates are still jumpy. Camera mics are spatial/context -witnesses; Focusrite devices are dialogue anchors. The current aligned frame is -integer-delay only; fractional delay and the hot resampler still belong in -Faust/native DSP. +Reports now carry fractional delay and per-band matched energy. The first +`MimirChirpletStreamDecoder` exists as the constrained chirplet-transform +receiver: it turns a bounded PCM window into symbol observations, every +code-valid triplet anchor it can prove, and an affine source clock fit. The +analyzer prefers matched canonical anchors before falling back to the older +chirplet-energy lag search. The hub also owns cached reports plus smoothed +per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` runs analysis +online as a bounded rotating service and can print live telemetry with +`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are passive readers of cached +sync state; they must not invoke the analyzer. Current app testing proves +loopback wakeup, live mic buffers, and confident online sync states, but the +decoder is not finished until arbitrary valid triplets produce stable anchors in +real mic streams. Camera mics are spatial/context witnesses; Focusrite devices +are dialogue anchors. The current aligned frame is integer-delay only; +fractional delay and the hot resampler still belong in Faust/native DSP. ## Visual Fusion diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 85970d1..d85f1ee 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -71,19 +71,22 @@ Get-Content .\state\evidence.jsonl -Tail 8 timeline owns Aquarium PCM segment rendering, coarse acquisition, symbol/event decoding, and per-band response hooks. Sync reports now include fractional delay and per-band matched energy. `MimirAudioSynchronizationState` - now tracks smoothed delay and delay-slope/SRO ppm. `MimirRuntime` runs sync - analysis as a bounded rotating service and caches reports/states for UI and - telemetry; readouts must stay passive. Actual Mimir.App testing proves - Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and produces - confident online sync states, but delay estimates are still jumpy. PS3 Eye - audio is + now tracks smoothed delay and delay-slope/SRO ppm. `MimirChirpletStreamDecoder` + is the first constrained chirplet-transform receiver: it emits symbol + observations, code-valid triplet anchors, and a source clock fit from a + bounded PCM window. `MimirRuntime` runs sync analysis as a bounded rotating + service and caches reports/states for UI and telemetry; readouts must stay + passive. Actual Mimir.App testing proves Aquarium audio wakes Scarlett + loopback, keeps mic buffers live, and produces confident online sync states, + but the decoder still needs stronger symbol likelihoods/codebook distance + before arbitrary real triplets can replace the lag fallback. PS3 Eye audio is enumeration/runtime-fragile: one run saw both mic buffers empty while both cameras were live, then a replug made both PS3 Eye mic buffers emit 480-frame WASAPI blocks again. ## Immediate Re-entry Instruction -Harden the online sync state filter, then turn those delay reports into a -smoothed SRO/fractional-delay actuator. Keep loopback as timing authority. Do +Finish the constrained chirplet-transform decoder, then turn decoded clock fits +into the SRO/fractional-delay actuator. Keep loopback as timing authority. Do not call synchronization analysis from UI/telemetry readouts. Do not restore deleted script infrastructure because a stale doc once missed it. diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index 640b3c8..bb3bd09 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -38,8 +38,6 @@ public sealed class MimirAudioSynchronizationAnalyzer return []; } - MimirChirpletTimelinePlacement? referencePlacement = null; - var reports = new List(); foreach (var buffer in audioBuffers) { @@ -75,6 +73,31 @@ public sealed class MimirAudioSynchronizationAnalyzer var referenceWindow = referenceSamples.AsSpan(^compared..); var candidateWindow = candidateSamples.AsSpan(^compared..); + var referenceDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(referenceWindow, referenceBlock.SampleRate); + var candidateDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(candidateWindow, referenceBlock.SampleRate); + var deterministicFit = EstimateDelayFromDecodedTimeline(referenceDecode, candidateDecode); + if (deterministicFit.MatchedEvents >= 1) + { + var decodedDelaySamples = deterministicFit.DelaySamples; + var decodedBandResponses = MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate); + reports.Add(new MimirAudioSynchronizationReport( + reference.Descriptor.SourceId, + buffer.Descriptor.SourceId, + referenceBlock.SampleRate, + (int)Math.Round(decodedDelaySamples), + decodedDelaySamples, + decodedDelaySamples * 1000.0 / referenceBlock.SampleRate, + deterministicFit.Confidence, + decodedBandResponses, + commonEndNs, + compared, + reference.Latest?.Sequence ?? 0, + buffer.Latest?.Sequence ?? 0, + deterministicFit.MatchedEvents, + deterministicFit.Confidence)); + continue; + } + var referenceWindowStartSample = referenceSamples.Length - compared; var referenceSync = MimirChirpletTimeline.Default.BuildTimelineEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples); var candidateSync = MimirChirpletTimeline.Default.BuildTimelineEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples); @@ -90,7 +113,7 @@ public sealed class MimirAudioSynchronizationAnalyzer var timelineConfidence = 0.0; if (confidence >= 0.03 && double.IsFinite(approximateTimelineSeconds)) { - referencePlacement ??= MimirChirpletTimeline.Default.DecodeReferenceWindow( + var referencePlacement = MimirChirpletTimeline.Default.DecodeReferenceWindow( referenceSamples, referenceBlock.SampleRate, approximateTimelineSeconds); @@ -140,6 +163,61 @@ public sealed class MimirAudioSynchronizationAnalyzer return reports; } + private static (double DelaySamples, double Confidence, int MatchedEvents) EstimateDelayFromDecodedTimeline( + MimirChirpletStreamDecode reference, + MimirChirpletStreamDecode candidate) + { + if (reference.Anchors.Count == 0 || + candidate.Anchors.Count == 0 || + reference.ClockFit == null || + candidate.ClockFit == null) + { + return (0.0, 0.0, 0); + } + + var candidateByEvent = candidate.Anchors.ToDictionary(anchor => anchor.EventIndex); + var matched = new List<(double Delay, double Weight)>(); + foreach (var referenceAnchor in reference.Anchors) + { + if (!candidateByEvent.TryGetValue(referenceAnchor.EventIndex, out var candidateAnchor)) + { + continue; + } + + matched.Add(( + candidateAnchor.SampleOffset - referenceAnchor.SampleOffset, + Math.Sqrt(referenceAnchor.Confidence * candidateAnchor.Confidence))); + } + + if (matched.Count >= 1) + { + var totalWeight = matched.Sum(pair => Math.Max(1.0e-6, pair.Weight)); + var delay = matched.Sum(pair => pair.Delay * Math.Max(1.0e-6, pair.Weight)) / totalWeight; + var error = matched.Sum(pair => Math.Abs(pair.Delay - delay) * Math.Max(1.0e-6, pair.Weight)) / totalWeight; + var residualConfidence = 1.0 / (1.0 + error / 32.0); + var countConfidence = Math.Clamp(matched.Count / 12.0, 0.0, 1.0); + var energyConfidence = Math.Clamp(totalWeight / matched.Count, 0.0, 1.0); + return (delay, residualConfidence * 0.50 + countConfidence * 0.25 + energyConfidence * 0.25, matched.Count); + } + + var referenceFirst = reference.Anchors.Min(anchor => anchor.TimelineSeconds); + var referenceLast = reference.Anchors.Max(anchor => anchor.TimelineSeconds); + var candidateFirst = candidate.Anchors.Min(anchor => anchor.TimelineSeconds); + var candidateLast = candidate.Anchors.Max(anchor => anchor.TimelineSeconds); + var start = Math.Max(referenceFirst, candidateFirst); + var end = Math.Min(referenceLast, candidateLast); + if (end <= start) + { + return (0.0, 0.0, matched.Count); + } + + var timelineSeconds = (start + end) * 0.5; + var clockDelay = candidate.ClockFit.SampleForTimelineSeconds(timelineSeconds) - + reference.ClockFit.SampleForTimelineSeconds(timelineSeconds); + var confidence = Math.Sqrt(reference.ClockFit.Confidence * candidate.ClockFit.Confidence) * 0.65; + return (clockDelay, confidence, Math.Min(reference.ClockFit.AnchorCount, candidate.ClockFit.AnchorCount)); + } + public MimirAlignedAudioFrame? BuildAlignedFrame( IEnumerable buffers, string referenceSourceId, diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs new file mode 100644 index 0000000..0ed2d59 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecode.cs @@ -0,0 +1,29 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirChirpletSymbolObservation( + int SymbolId, + double SampleOffset, + double Energy); + +public sealed record MimirChirpletTimelineAnchor( + ulong EventIndex, + double TimelineSeconds, + double SampleOffset, + double Confidence, + IReadOnlyList Symbols); + +public sealed record MimirChirpletClockFit( + double SourceOffsetSamples, + double EffectiveSampleRate, + double Confidence, + int AnchorCount, + double MeanAbsoluteErrorSamples) +{ + public double SampleForTimelineSeconds(double timelineSeconds) => + SourceOffsetSamples + timelineSeconds * EffectiveSampleRate; +} + +public sealed record MimirChirpletStreamDecode( + IReadOnlyList Symbols, + IReadOnlyList Anchors, + MimirChirpletClockFit? ClockFit); diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs new file mode 100644 index 0000000..38094f0 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletStreamDecoder.cs @@ -0,0 +1,79 @@ +using System.Runtime.InteropServices; + +namespace Mimir.Runtime.Synchronization; + +public sealed class MimirChirpletStreamDecoder +{ + private readonly MimirChirpletTimeline timeline; + private readonly int sampleRate; + private readonly int maxSamples; + private readonly List samples; + private long originSample; + + public MimirChirpletStreamDecoder( + MimirChirpletTimeline? timeline = null, + int sampleRate = MimirChirpletTimeline.SampleRate, + TimeSpan? windowDuration = null) + { + this.timeline = timeline ?? MimirChirpletTimeline.Default; + this.sampleRate = sampleRate; + maxSamples = Math.Max( + sampleRate / 2, + (int)Math.Round((windowDuration ?? TimeSpan.FromSeconds(5)).TotalSeconds * sampleRate)); + samples = new List(maxSamples); + } + + public long OriginSample => originSample; + + public long NextSample => originSample + samples.Count; + + public MimirChirpletStreamDecode Decode() => + WithAbsoluteOffsets(timeline.DecodeStreamWindow(CollectionsMarshal.AsSpan(samples), sampleRate)); + + public MimirChirpletStreamDecode Append(ReadOnlySpan block) + { + if (block.Length == 0) + { + return Decode(); + } + + foreach (var sample in block) + { + samples.Add(sample); + } + + var overflow = samples.Count - maxSamples; + if (overflow > 0) + { + samples.RemoveRange(0, overflow); + originSample += overflow; + } + + return Decode(); + } + + private MimirChirpletStreamDecode WithAbsoluteOffsets(MimirChirpletStreamDecode decode) + { + if (originSample == 0) + { + return decode; + } + + var symbols = decode.Symbols + .Select(symbol => symbol with { SampleOffset = symbol.SampleOffset + originSample }) + .ToArray(); + var anchors = decode.Anchors + .Select(anchor => anchor with + { + SampleOffset = anchor.SampleOffset + originSample, + Symbols = anchor.Symbols + .Select(symbol => symbol with { SampleOffset = symbol.SampleOffset + originSample }) + .ToArray(), + }) + .ToArray(); + var clock = decode.ClockFit == null + ? null + : decode.ClockFit with { SourceOffsetSamples = decode.ClockFit.SourceOffsetSamples + originSample }; + return new MimirChirpletStreamDecode(symbols, anchors, clock); + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs index db90dee..bcf2a31 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs @@ -134,6 +134,24 @@ public sealed class MimirChirpletTimeline return responses; } + public MimirChirpletStreamDecode DecodeStreamWindow(ReadOnlySpan samples, int sampleRate) + { + if (samples.Length == 0) + { + return new MimirChirpletStreamDecode([], [], null); + } + + var symbols = DetectSymbolEvents(samples, sampleRate, Math.Max(1, sampleRate / 200)) + .Select(observation => new MimirChirpletSymbolObservation( + observation.SymbolId, + observation.SampleOffset, + observation.Energy)) + .ToArray(); + var anchors = PlaceAllTripletAnchors(symbols, sampleRate); + var clock = FitClock(anchors, sampleRate); + return new MimirChirpletStreamDecode(symbols, anchors, clock); + } + public MimirChirpletTimelinePlacement DecodeReferenceWindow( ReadOnlySpan samples, int sampleRate, @@ -245,7 +263,11 @@ public sealed class MimirChirpletTimeline continue; } - var observation = ClassifySymbolAt(samples, sampleRate, frame * hopSamples, Math.Max(1, hopSamples)); + var observation = ClassifySymbolAt( + samples, + sampleRate, + frame * hopSamples, + Math.Max(hopSamples * 3, sampleRate / 100)); if (observation != null) { candidates.Add(observation); @@ -287,7 +309,7 @@ public sealed class MimirChirpletTimeline } } - return bestSymbol < 0 || bestEnergy < 0.055 + return bestSymbol < 0 || bestEnergy < 0.25 ? null : new MimirChirpletEventObservation(0, bestSymbol, 0.0, bestOffset, bestEnergy); } @@ -363,6 +385,122 @@ public sealed class MimirChirpletTimeline : FitTimelinePlacement(best, sampleRate); } + private static IReadOnlyList PlaceAllTripletAnchors( + IReadOnlyList symbols, + int sampleRate) + { + if (symbols.Count < TimelineOrder) + { + return []; + } + + var anchors = new List(); + for (var index = 0; index <= symbols.Count - TimelineOrder; index++) + { + var first = symbols[index]; + var second = symbols[index + 1]; + var third = symbols[index + 2]; + if (!IsConsecutiveByTime(first, second, sampleRate) || + !IsConsecutiveByTime(second, third, sampleRate)) + { + continue; + } + + var code = TripleCode(first.SymbolId, second.SymbolId, third.SymbolId); + if (!TripleToIndex.TryGetValue(code, out var eventIndex)) + { + continue; + } + + var timelineEvent = Default.EventForIndex((ulong)eventIndex); + var predictedSecond = Default.EventForIndex((ulong)eventIndex + 1UL); + var predictedThird = Default.EventForIndex((ulong)eventIndex + 2UL); + var measuredGapA = (second.SampleOffset - first.SampleOffset) / sampleRate; + var measuredGapB = (third.SampleOffset - second.SampleOffset) / sampleRate; + var expectedGapA = predictedSecond.StartSeconds - timelineEvent.StartSeconds; + var expectedGapB = predictedThird.StartSeconds - predictedSecond.StartSeconds; + var gapError = Math.Abs(measuredGapA - expectedGapA) + Math.Abs(measuredGapB - expectedGapB); + var gapConfidence = 1.0 / (1.0 + gapError / 0.006); + if (gapConfidence < 0.45) + { + continue; + } + + var energyConfidence = Math.Clamp((first.Energy + second.Energy + third.Energy) / 3.0, 0.0, 1.0); + anchors.Add(new MimirChirpletTimelineAnchor( + (ulong)eventIndex, + timelineEvent.StartSeconds, + first.SampleOffset, + gapConfidence * 0.65 + energyConfidence * 0.35, + [first, second, third])); + } + + return anchors + .GroupBy(anchor => anchor.EventIndex) + .Select(group => group.OrderByDescending(anchor => anchor.Confidence).First()) + .OrderBy(anchor => anchor.SampleOffset) + .ToArray(); + } + + private static MimirChirpletClockFit? FitClock( + IReadOnlyList anchors, + int sampleRate) + { + if (anchors.Count == 0) + { + return null; + } + + if (anchors.Count == 1) + { + var anchor = anchors[0]; + return new MimirChirpletClockFit( + anchor.SampleOffset - anchor.TimelineSeconds * sampleRate, + sampleRate, + anchor.Confidence * 0.35, + 1, + 0.0); + } + + var totalWeight = anchors.Sum(anchor => Math.Max(1.0e-6, anchor.Confidence)); + var meanTimeline = anchors.Sum(anchor => anchor.TimelineSeconds * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var meanSample = anchors.Sum(anchor => anchor.SampleOffset * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var covariance = 0.0; + var variance = 0.0; + foreach (var anchor in anchors) + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var dt = anchor.TimelineSeconds - meanTimeline; + covariance += weight * dt * (anchor.SampleOffset - meanSample); + variance += weight * dt * dt; + } + + var effectiveSampleRate = variance > 1.0e-12 ? covariance / variance : sampleRate; + if (!double.IsFinite(effectiveSampleRate) || + effectiveSampleRate < sampleRate * 0.90 || + effectiveSampleRate > sampleRate * 1.10) + { + effectiveSampleRate = sampleRate; + } + + var sourceOffset = meanSample - effectiveSampleRate * meanTimeline; + var meanAbsoluteError = anchors.Sum(anchor => + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var predicted = sourceOffset + anchor.TimelineSeconds * effectiveSampleRate; + return Math.Abs(anchor.SampleOffset - predicted) * weight; + }) / totalWeight; + var residualConfidence = 1.0 / (1.0 + meanAbsoluteError / Math.Max(1.0, sampleRate * 0.0015)); + var countConfidence = Math.Clamp(anchors.Count / 12.0, 0.0, 1.0); + var anchorConfidence = Math.Clamp(anchors.Average(anchor => anchor.Confidence), 0.0, 1.0); + return new MimirChirpletClockFit( + sourceOffset, + effectiveSampleRate, + residualConfidence * 0.45 + countConfidence * 0.25 + anchorConfidence * 0.30, + anchors.Count, + meanAbsoluteError); + } + private static bool IsConsecutiveByTime( MimirChirpletEventObservation first, MimirChirpletEventObservation second, @@ -372,6 +510,15 @@ public sealed class MimirChirpletTimeline return seconds >= MinEventGapSeconds * 0.55 && seconds <= MaxEventGapSeconds * 1.45; } + private static bool IsConsecutiveByTime( + MimirChirpletSymbolObservation first, + MimirChirpletSymbolObservation second, + int sampleRate) + { + var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate; + return seconds >= MinEventGapSeconds * 0.55 && seconds <= MaxEventGapSeconds * 1.45; + } + private static MimirChirpletEventObservation PlaceObservation( MimirChirpletEventObservation observation, ulong eventIndex) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 80bc4a4..8151582 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -43,3 +43,4 @@ {"ts":"2026-05-22T14:32:00+01:00","type":"implementation","status":"ok","note":"Reworked MimirChirpletTimeline symbols from fixed frequency slots into time/frequency constellations. The order-3 de Bruijn sequence still gives three-chirp timeline placement, but each symbol now carries identity through start band, glide shape, duration, and following inter-chirp gap. Event start times are precomputed over the de Bruijn period instead of using a fixed stride. Synthetic verification rendered two seconds of canonical timeline audio and decoded interior events 9, 10, and 11 back to a zero-second window start, proving placement does not rely on the opening triplet."} {"ts":"2026-05-22T14:48:00+01:00","type":"verification","status":"partial","note":"Attempted a 20-second real-world Mimir.App chirplet sync run with all configured local mics. The headless path only emitted one telemetry tick before runtime updates stopped. Switching to the normal app path produced telemetry through t=4.26s with all six audio buffers live and sync states for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Timeline-symbol refinement appeared on Focusrite, Kiyo, and one PS3 Eye, but delay estimates were unstable and the app did not continue telemetry to 20 seconds. Replaced the coarse acquisition trace with envelope energy to avoid matched-filter scans in the hot acquisition path; symbol decoding remains the refinement step."} {"ts":"2026-05-22T15:20:00+01:00","type":"implementation","status":"partial","note":"Removed hidden re-entrant synchronization analysis from the live app path. MimirRuntime now calls a bounded rotating AnalyzeAudioSynchronizationStep every 0.1s, MimirSynchronizationHub caches per-source reports/states, and UI/telemetry read cached state instead of invoking the analyzer or BuildAlignedAudioFrame. FitDelayFromEvents now collapses duplicate candidate event observations by strongest energy, fixing the real app crash 'An item with the same key has already been added. Key: 0'. Dotnet build passed. A visible Mimir.App run with all configured mics no longer crashed at the earlier 4s wall and produced telemetry through runtime t=17.28s in a 22s wall-time harness; it did not reach a full 20 runtime seconds because startup consumed wall time. All non-loopback mics produced cached reports/states, but delay estimates remained unstable and timeline event refinement was intermittent, so the next live invariant is robust outlier rejection/state filtering before any actuator drives resampling."} +{"ts":"2026-05-22T16:05:00+01:00","type":"implementation","status":"partial","note":"Documented the intended decoder as a constrained chirplet transform: known dictionary projection to symbol observations, code-valid de Bruijn triplets to canonical timeline anchors, anchors to affine source clock fits, and delay/SRO from comparing loopback and mic clock fits. Added MimirChirpletStreamDecode records and MimirChirpletStreamDecoder with a bounded PCM window, absolute sample offsets, triplet anchors, and clock fits. MimirAudioSynchronizationAnalyzer now prefers matched decoded timeline anchors before falling back to the older chirplet-energy lag search. Synthetic self-test over two seconds of emitted timeline audio currently decodes 12 symbols and one correct anchor for event 9 at sample 47520 with clock 48000 Hz; this proves the new spine but also proves the symbol likelihood/codebook layer is not finished enough to place every triplet yet."} diff --git a/state/map.yaml b/state/map.yaml index 92c1101..976bbbd 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -71,12 +71,16 @@ accepted_design: timing atoms, symbol/event decoding, and per-band response kernels. Symbol identity is carried by start band, glide shape, duration, and following inter-chirp gap. Any three consecutive correctly detected symbols identify a - timeline event within the current operating horizon. `MimirAudioSynchronizationAnalyzer` - now treats `loopback-scarlett-speakers` as timing authority, uses coarse - chirplet energy for acquisition, and refines delay from matched timeline - event indices when symbol decoding succeeds. Positive delay means the - candidate mic is late relative to loopback. Reports include rounded delay, - fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub` + timeline event within the current operating horizon. `MimirChirpletStreamDecoder` + is the first constrained chirplet-transform receiver: it owns a bounded PCM + window, emits symbol observations, decodes code-valid triplet anchors onto + the canonical timeline, and fits a per-source sample clock from those + anchors. `MimirAudioSynchronizationAnalyzer` now treats + `loopback-scarlett-speakers` as timing authority, prefers delay from matched + decoded timeline anchors, and keeps the older chirplet-energy lag search as + a diagnostic fallback while the decoder is cut over. Positive delay means + the candidate mic is late relative to loopback. Reports include rounded + delay, fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub` caches the latest per-source reports and `MimirAudioSynchronizationStateTracker` owns the smoothed per-source belief: latest delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in @@ -183,14 +187,15 @@ current_status: Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > - Harden the online state filter before adding the synchronization actuator. + Finish the constrained chirplet-transform decoder before adding the + synchronization actuator. Current app testing proves Aquarium WASAPI output wakes Scarlett speaker loopback, live mic buffers stay populated, runtime sync analysis now runs as a bounded rotating service without UI/telemetry re-entry, and the continuous - chirplet timeline can produce confident sync states in the app. Delay - estimates are still jumpy, so the actuator needs outlier rejection and better - SRO smoothing before it drives variable-rate resampling and fractional-delay - correction per non-reference stream. Replace + chirplet timeline can produce confident sync states in the app. The next + invariant is code-valid triplets producing stable canonical anchors and clock + fits, so the actuator can use decoded time instead of filtered lag guesses. + Replace the diagnostic JSON-line process bridge with direct native capture drivers that push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when