diff --git a/docs/audio-field.md b/docs/audio-field.md index 923f314..379b83a 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -50,21 +50,23 @@ produced zero WASAPI packets until that Eye was unplugged and replugged. The full probe runtime config now enables sample-bearing blocks for every local audio source. `MimirChirpletTimeline` owns the emitted calibration stream and -the matched-filter shape used to analyze it. The default timeline is -deterministic and unbounded: event `N` is generated directly from `N`, with no -registry, remembered random state, or repeat cycle. Mimir queues half-second PCM -segments ahead of the audio cursor, and each segment contains short asymmetric -chirplets chosen from a harmonic high-frequency vocabulary. The point is not -ornament. The timing code is carried by both frequency and rhythm, so it behaves -more like a quiet birdsong texture than a repeated sweep. +the matched-filter shape used to analyze it. The default timeline is a +deterministic order-3 de Bruijn symbol sequence over 32 chirp symbols. Any three +consecutive correctly detected symbols identify the event index inside the +current operating horizon, so a receiver can place its audio window on the +canonical timeline without being handed Mimir's runtime clock. Mimir queues +half-second PCM segments ahead of the audio cursor, and each segment contains +short asymmetric chirplets chosen from separated high-frequency slots. The point +is not ornament. The timing code is carried by both frequency and rhythm, so it +behaves more like a quiet birdsong texture than a repeated sweep. `MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the -loopback sample-rate timeline, projects loopback and candidate windows through a -continuous chirplet atom bank, contrast-normalizes the resulting energy traces, -and estimates current delay against `loopback-scarlett-speakers`. The lag search -is wide enough for ordinary remote/network delay, not just local speaker-to-mic -delay. Reports carry both rounded integer delay and fractional delay from -parabolic peak interpolation over the chirplet correlation peak. +loopback sample-rate timeline, uses cheap coarse chirplet energy for acquisition, +then lets the symbol decoder refine delay from matched event indices. The lag +search is wide enough for ordinary remote/network delay, not just local +speaker-to-mic delay. Reports carry rounded integer delay, fractional delay from +parabolic peak interpolation, and the count/confidence of timeline-symbol events +used to refine the report. `MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned mono frame: loopback is always channel zero, and other channels enter only when @@ -128,6 +130,14 @@ watching delay change over time. Per-band energy over the same stream gives the normalization curve. The important constraint is that all three measurements must be tied to the same emitted timeline, not three separately invented probes. +The symbol layer is intentionally blunt. Chirp symbols use separated frequency +slots instead of subtle variants sharing the same center frequency, because a +wrong symbol is worse than a noisy delay. A synthetic runtime check rendered two +seconds of canonical timeline audio and decoded events 0, 1, and 2 back to a +window start within about 1.5 ms. Real device runs still depend on loopback +capture staying live; the local Scarlett loopback has intermittently stopped +advancing during short headless sniffs. + Next, replace the diagnostic bridge with native audio capture workers that append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state, delay estimates, and stem routing in Aquarium UI. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 3fe0f8c..944fee2 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -44,12 +44,14 @@ a named invariant that the native runtime cannot protect yet. adapter. - `MimirChirpletTimeline` owns the structured birdsong-like calibration stream, PCM segment rendering, matched timing trace, and per-band response kernels. - The default timeline is deterministic and unbounded: chirplet event `N` is - generated from `N`, with no repeat cycle or remembered random state. + The default timeline is an order-3 de Bruijn symbol sequence over 32 separated + chirp symbols, so any three consecutive correctly detected symbols identify a + timeline event inside the current operating horizon. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: sample-bearing audio blocks are resampled into the Scarlett loopback timeline and compared by chirplet-energy delay estimation with fractional peak - interpolation. + interpolation. When timeline symbols are decoded, matched event indices refine + the delay estimate and are exposed in telemetry. - `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional aligned mono frame for loopback-referenced audio channels that clear the confidence gate. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 5a08594..52480a0 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -91,12 +91,13 @@ not become the synchronized program authority. The six-microphone path is separate from the bridge. Scarlett speaker loopback is the current timing authority when calibration chirplets are playing. -`MimirChirpletTimeline` owns the birdsong-like timeline fingerprint: a -deterministic unbounded chirplet stream where event `N` is generated from `N`, -with no repeat cycle or remembered random state. Mimir continuously queues that -timeline through Aquarium audio, compares mic buffers against loopback by -chirplet-energy delay estimation, and can build a provisional aligned mono -frame for channels that clear the confidence gate. +`MimirChirpletTimeline` owns the birdsong-like timeline fingerprint: an order-3 +de Bruijn sequence over 32 separated chirp symbols. Any three consecutive +correctly detected symbols identify the event index inside the current operating +horizon. Mimir continuously queues that timeline through Aquarium audio, uses +coarse chirplet energy for acquisition, refines delay from matched event +indices when the symbol decoder succeeds, and can build a provisional aligned +mono frame for channels that clear the confidence gate. Reports now carry fractional delay and per-band matched energy. The hub also owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` runs the analysis online and can print live telemetry with diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 8839e1f..5f6ca76 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -64,10 +64,11 @@ Get-Content .\state\evidence.jsonl -Tail 8 Kiyo and second PS3 Eye were absent, so six-camera viability is not proven. - Current chirplet-backed smoke uses Scarlett speaker loopback as timing authority. `MimirChirpletTimeline` owns the structured birdsong-like timeline - fingerprint: deterministic unbounded chirplet events where event `N` is - generated from `N`, with no repeat cycle or remembered random state. The - timeline owns Aquarium PCM segment rendering, matched timing trace, and - per-band response hooks. Sync reports now include + fingerprint: an order-3 de Bruijn sequence over 32 separated chirp symbols, + so any three consecutive correctly detected symbols identify the event index + inside the current operating horizon. The timeline owns Aquarium PCM segment + rendering, coarse acquisition, symbol/event decoding, and per-band response + hooks. Sync reports now include fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing proves Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index a43f10c..987ab80 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -164,7 +164,7 @@ public sealed class MimirRuntime : IAquariumRuntime return; } - lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference); + lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds); nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds; } @@ -184,7 +184,7 @@ public sealed class MimirRuntime : IAquariumRuntime foreach (var report in lastAudioSynchronizationReports.OrderBy(report => report.SourceId, StringComparer.Ordinal)) { Console.WriteLine( - $"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000}"); + $"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} timelineEvents={report.TimelineMatchedEvents} timelineConfidence={report.TimelineConfidence:0.000}"); } foreach (var state in states) @@ -229,10 +229,10 @@ public sealed class MimirRuntime : IAquariumRuntime private string DescribeAudioSync() { - var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference); + var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds); return reports.Count == 0 ? "no payload windows" - : string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00}")); + : string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00} events={report.TimelineMatchedEvents}")); } private string DescribeAlignedAudio() diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index df44eda..10028cb 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -4,13 +4,14 @@ namespace Mimir.Runtime.Synchronization; public sealed class MimirAudioSynchronizationAnalyzer { - private const int MaxWindowSamples = 48_000 * 5; + private const int MaxWindowSamples = 48_000 * 2; private const int MaxLagSamples = 48_000; private const int ChirpletHopSamples = 16; public IReadOnlyList Analyze( IEnumerable buffers, - string referenceSourceId) + string referenceSourceId, + double approximateTimelineSeconds) { var audioBuffers = buffers .Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio) @@ -28,6 +29,16 @@ public sealed class MimirAudioSynchronizationAnalyzer return []; } + var referenceSamples = ExtractMonoWindow(reference, out var referenceBlock); + if (referenceSamples.Length == 0 || + referenceBlock == null || + referenceBlock.SampleFormat != MimirAudioSampleFormat.Float32) + { + return []; + } + + MimirChirpletTimelinePlacement? referencePlacement = null; + var reports = new List(); foreach (var buffer in audioBuffers) { @@ -37,20 +48,13 @@ public sealed class MimirAudioSynchronizationAnalyzer } var commonEndNs = Math.Min(referenceLatest.Value.TimestampNs, buffer.Latest?.TimestampNs ?? 0); - var referenceSamples = ExtractMonoWindow(reference, out var referenceBlock, commonEndNs); - var candidateSamples = ExtractMonoWindow(buffer, out var candidateBlock, commonEndNs); + var candidateSamples = ExtractMonoWindow(buffer, out var candidateBlock); if (candidateSamples.Length == 0 || candidateBlock == null) { continue; } - if (referenceSamples.Length == 0 || referenceBlock == null) - { - continue; - } - - if (candidateBlock.SampleFormat != MimirAudioSampleFormat.Float32 || - referenceBlock.SampleFormat != MimirAudioSampleFormat.Float32) + if (candidateBlock.SampleFormat != MimirAudioSampleFormat.Float32) { continue; } @@ -65,6 +69,7 @@ public sealed class MimirAudioSynchronizationAnalyzer var referenceWindow = referenceSamples.AsSpan(^compared..); var candidateWindow = candidateSamples.AsSpan(^compared..); + var referenceWindowStartSample = referenceSamples.Length - compared; var referenceSync = MimirChirpletTimeline.Default.BuildTimelineEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples); var candidateSync = MimirChirpletTimeline.Default.BuildTimelineEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples); if (referenceSync.Length < 16 || candidateSync.Length < 16) @@ -75,6 +80,38 @@ public sealed class MimirAudioSynchronizationAnalyzer var maxLag = Math.Min(MaxLagSamples / ChirpletHopSamples, Math.Min(referenceSync.Length, candidateSync.Length) / 2); var (delayHops, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag); var fractionalDelaySamples = delayHops * ChirpletHopSamples; + var timelineMatchedEvents = 0; + var timelineConfidence = 0.0; + if (confidence >= 0.03 && double.IsFinite(approximateTimelineSeconds)) + { + referencePlacement ??= MimirChirpletTimeline.Default.DecodeReferenceWindow( + referenceSamples, + referenceBlock.SampleRate, + approximateTimelineSeconds); + var adjustedReferenceObservations = referencePlacement.Observations + .Where(observation => + observation.SampleOffset >= referenceWindowStartSample && + observation.SampleOffset < referenceWindowStartSample + compared) + .Select(observation => observation with + { + SampleOffset = observation.SampleOffset - referenceWindowStartSample, + }) + .ToArray(); + var candidateObservations = MimirChirpletTimeline.Default.ObserveKnownEvents( + candidateWindow, + referenceBlock.SampleRate, + adjustedReferenceObservations, + fractionalDelaySamples); + var eventFit = FitDelayFromEvents(adjustedReferenceObservations, candidateObservations, referenceBlock.SampleRate); + if (eventFit.MatchedEvents >= 3) + { + fractionalDelaySamples = eventFit.DelaySamples; + timelineMatchedEvents = eventFit.MatchedEvents; + timelineConfidence = eventFit.Confidence; + confidence = Math.Clamp((confidence * 0.35) + (eventFit.Confidence * 0.65), 0.0, 1.0); + } + } + var delaySamples = (int)Math.Round(fractionalDelaySamples); var bandResponses = MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate); reports.Add(new MimirAudioSynchronizationReport( @@ -89,7 +126,9 @@ public sealed class MimirAudioSynchronizationAnalyzer commonEndNs, compared, reference.Latest?.Sequence ?? 0, - buffer.Latest?.Sequence ?? 0)); + buffer.Latest?.Sequence ?? 0, + timelineMatchedEvents, + timelineConfidence)); } return reports; @@ -115,7 +154,7 @@ public sealed class MimirAudioSynchronizationAnalyzer return null; } - var reports = Analyze(audioBuffers.Values, referenceSourceId) + var reports = Analyze(audioBuffers.Values, referenceSourceId, approximateTimelineSeconds: double.PositiveInfinity) .Where(report => report.Confidence >= minimumConfidence) .ToDictionary(report => report.SourceId, StringComparer.Ordinal); var alignedReports = reports.Values @@ -314,6 +353,51 @@ public sealed class MimirAudioSynchronizationAnalyzer return (bestLag + offset, Math.Max(0.0, Math.Min(1.0, best))); } + private static (double DelaySamples, double Confidence, int MatchedEvents) FitDelayFromEvents( + IReadOnlyList referenceObservations, + IReadOnlyList candidateObservations, + int sampleRate) + { + if (referenceObservations.Count == 0 || candidateObservations.Count == 0) + { + return (0.0, 0.0, 0); + } + + var candidateByEvent = candidateObservations.ToDictionary(observation => observation.EventIndex); + var delays = new List<(double Delay, double Weight)>(); + foreach (var reference in referenceObservations) + { + if (!candidateByEvent.TryGetValue(reference.EventIndex, out var candidate)) + { + continue; + } + + var weight = Math.Sqrt(Math.Max(0.0, reference.Energy) * Math.Max(0.0, candidate.Energy)); + delays.Add((candidate.SampleOffset - reference.SampleOffset, weight)); + } + + if (delays.Count < 3) + { + return (0.0, 0.0, delays.Count); + } + + var ordered = delays.OrderBy(pair => pair.Delay).ToArray(); + var trim = ordered.Length >= 8 ? ordered.Length / 8 : 0; + var kept = ordered.Skip(trim).Take(ordered.Length - trim * 2).ToArray(); + var totalWeight = kept.Sum(pair => pair.Weight); + if (totalWeight <= 1.0e-12) + { + return (0.0, 0.0, kept.Length); + } + + var delay = kept.Sum(pair => pair.Delay * pair.Weight) / totalWeight; + var meanAbsoluteError = kept.Sum(pair => Math.Abs(pair.Delay - delay) * pair.Weight) / totalWeight; + var scatterConfidence = 1.0 / (1.0 + meanAbsoluteError / Math.Max(1.0, sampleRate * 0.002)); + var countConfidence = Math.Clamp(kept.Length / 12.0, 0.0, 1.0); + var energyConfidence = Math.Clamp(totalWeight / kept.Length, 0.0, 1.0); + return (delay, scatterConfidence * 0.55 + countConfidence * 0.25 + energyConfidence * 0.20, kept.Length); + } + private static double ParabolicPeakOffset(double left, double center, double right) { if (!double.IsFinite(left) || !double.IsFinite(center) || !double.IsFinite(right)) diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs index 9d6c1f5..5d3415f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs @@ -12,4 +12,6 @@ public sealed record MimirAudioSynchronizationReport( long AnalysisTimestampNs, int ComparedSamples, ulong ReferenceSequence, - ulong SourceSequence); + ulong SourceSequence, + int TimelineMatchedEvents = 0, + double TimelineConfidence = 0.0); diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs index 061e0ea..783b26f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletTimeline.cs @@ -14,44 +14,57 @@ public sealed record MimirChirpletBandResponse( double CenterHz, double Energy); +public sealed record MimirChirpletTimelineEvent( + ulong Index, + int SymbolId, + double StartSeconds, + MimirChirpletTone Tone); + +public sealed record MimirChirpletEventObservation( + ulong EventIndex, + int SymbolId, + double TimelineSeconds, + double SampleOffset, + double Energy); + +public sealed record MimirChirpletTimelinePlacement( + double TimelineSecondsAtWindowStart, + double Confidence, + IReadOnlyList Observations); + public sealed class MimirChirpletTimeline { public const int SampleRate = 48_000; public const double SegmentSeconds = 0.5; public const double QueueLeadSeconds = 1.0; + public const int SymbolCount = 32; + public const int TimelineOrder = 3; + public static readonly int TimelinePeriod = (int)Math.Pow(SymbolCount, TimelineOrder); + private const double FirstEventSeconds = 0.08; private const double EventStrideSeconds = 0.118; - private const double MaxEventDurationSeconds = 0.110; - private const double Gain = 0.095; + private const double MaxEventDurationSeconds = 0.082; + private const double Gain = 0.090; - private static readonly MimirChirpletTone[] AnalysisAtoms = + private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder)); + private static readonly Dictionary TripleToIndex = BuildTripleIndex(TimelineSymbols); + private static readonly MimirChirpletTone[] SymbolTones = BuildSymbolTones(); + private static readonly MimirChirpletTone[] CoarseTones = [ - new(0.0, 0.050, 6_900.0, 7_500.0, 0.70), - new(0.0, 0.055, 8_000.0, 8_900.0, 0.75), - new(0.0, 0.060, 9_600.0, 8_800.0, 0.75), - new(0.0, 0.060, 10_600.0, 11_700.0, 0.85), - new(0.0, 0.065, 12_400.0, 13_500.0, 0.90), - new(0.0, 0.070, 14_400.0, 15_800.0, 0.80), + new(0.0, 0.048, 6_700.0, 7_200.0, 0.75), + new(0.0, 0.052, 7_700.0, 8_400.0, 0.80), + new(0.0, 0.056, 8_900.0, 9_800.0, 0.85), + new(0.0, 0.060, 10_300.0, 11_400.0, 0.85), + new(0.0, 0.064, 12_100.0, 13_500.0, 0.90), + new(0.0, 0.068, 14_200.0, 15_900.0, 0.80), ]; - private static readonly double[] HarmonicRatios = - [ - 1.0, - 9.0 / 8.0, - 5.0 / 4.0, - 4.0 / 3.0, - 3.0 / 2.0, - 5.0 / 3.0, - 15.0 / 8.0, - 2.0, - 9.0 / 4.0, - 5.0 / 2.0, - ]; - - private readonly IReadOnlyList atomKernels; + private readonly IReadOnlyList symbolKernels; + private readonly IReadOnlyList coarseKernels; private MimirChirpletTimeline() { - atomKernels = AnalysisAtoms.Select(atom => RenderToneKernel(atom, SampleRate)).ToArray(); + symbolKernels = SymbolTones.Select(symbol => RenderToneKernel(symbol, SampleRate)).ToArray(); + coarseKernels = CoarseTones.Select(tone => RenderToneKernel(tone, SampleRate)).ToArray(); } public static MimirChirpletTimeline Default { get; } = new(); @@ -60,9 +73,9 @@ public sealed class MimirChirpletTimeline { var segmentStartSeconds = segmentIndex * SegmentSeconds; var samples = new float[(int)Math.Round(SegmentSeconds * SampleRate)]; - foreach (var tone in TonesOverlapping(segmentStartSeconds, SegmentSeconds)) + foreach (var timelineEvent in EventsOverlapping(segmentStartSeconds, SegmentSeconds)) { - AddTone(samples, SampleRate, tone, segmentStartSeconds); + AddTone(samples, SampleRate, timelineEvent.Tone, segmentStartSeconds); } var peak = samples.Select(Math.Abs).DefaultIfEmpty(0.0f).Max(); @@ -96,37 +109,7 @@ public sealed class MimirChirpletTimeline public float[] BuildTimelineEnergyTrace(ReadOnlySpan samples, int sampleRate, int hopSamples) { - var traces = new List(); - for (var index = 0; index < AnalysisAtoms.Length; index++) - { - var atom = AnalysisAtoms[index]; - var kernel = sampleRate == SampleRate ? atomKernels[index] : RenderToneKernel(atom, sampleRate); - var trace = BuildMatchedEnergyTrace(samples, kernel, hopSamples); - if (trace.Length > 0) - { - traces.Add(trace); - } - } - - if (traces.Count == 0) - { - return []; - } - - var length = traces.Min(trace => trace.Length); - var output = new float[length]; - for (var frame = 0; frame < length; frame++) - { - var best = 0.0f; - foreach (var trace in traces) - { - best = Math.Max(best, trace[frame]); - } - - output[frame] = best; - } - - return ContrastNormalize(output); + return ContrastNormalize(BuildCoarseEnergyTrace(samples, sampleRate, hopSamples)); } public IReadOnlyList EstimateBandResponse(ReadOnlySpan samples, int sampleRate) @@ -136,63 +119,445 @@ public sealed class MimirChirpletTimeline return []; } - var responses = new List(AnalysisAtoms.Length); - for (var index = 0; index < AnalysisAtoms.Length; index++) + var responses = new List(SymbolTones.Length); + for (var symbol = 0; symbol < SymbolTones.Length; symbol++) { - var atom = AnalysisAtoms[index]; - var kernel = sampleRate == SampleRate ? atomKernels[index] : RenderToneKernel(atom, sampleRate); + var tone = SymbolTones[symbol]; + var kernel = sampleRate == SampleRate ? symbolKernels[symbol] : RenderToneKernel(tone, sampleRate); var energy = MaxMatchedEnergy(samples, kernel, Math.Max(1, sampleRate / 1_000)); - responses.Add(new MimirChirpletBandResponse(atom.CenterHz, energy)); + responses.Add(new MimirChirpletBandResponse(tone.CenterHz, energy)); } return responses; } - private static IEnumerable TonesOverlapping(double startSeconds, double durationSeconds) + public MimirChirpletTimelinePlacement DecodeReferenceWindow( + ReadOnlySpan samples, + int sampleRate, + double approximateWindowEndTimelineSeconds) + { + if (samples.Length == 0) + { + return new MimirChirpletTimelinePlacement(0.0, 0.0, []); + } + + var observed = DetectSymbolEvents(samples, sampleRate, Math.Max(1, sampleRate / 200)); + var placement = PlaceObservedSymbols(observed, sampleRate); + if (placement.Observations.Count >= 3 || !double.IsFinite(approximateWindowEndTimelineSeconds)) + { + return placement; + } + + var windowSeconds = samples.Length / (double)sampleRate; + var approximateStartSeconds = Math.Max(0.0, approximateWindowEndTimelineSeconds - windowSeconds); + var fallback = new List(); + foreach (var timelineEvent in EventsOverlapping(approximateStartSeconds - 0.25, windowSeconds + 0.5)) + { + var predictedSample = (timelineEvent.StartSeconds - approximateStartSeconds) * sampleRate; + var observation = ObserveEvent(samples, sampleRate, timelineEvent, predictedSample, searchRadiusSamples: Math.Max(8, sampleRate / 2)); + if (observation != null) + { + fallback.Add(observation); + } + } + + if (fallback.Count < 3) + { + return new MimirChirpletTimelinePlacement(approximateStartSeconds, 0.0, fallback); + } + + return FitTimelinePlacement(fallback, sampleRate); + } + + public IReadOnlyList ObserveKnownEvents( + ReadOnlySpan samples, + int sampleRate, + IEnumerable referenceObservations, + double coarseDelaySamples) + { + var observations = new List(); + foreach (var reference in referenceObservations) + { + var timelineEvent = EventForIndex(reference.EventIndex); + var predictedSample = reference.SampleOffset + coarseDelaySamples; + var observation = ObserveEvent(samples, sampleRate, timelineEvent, predictedSample, searchRadiusSamples: Math.Max(16, sampleRate / 10)); + if (observation != null) + { + observations.Add(observation); + } + } + + return observations; + } + + public MimirChirpletTimelineEvent EventForIndex(ulong eventIndex) + { + var symbolId = SymbolForEvent(eventIndex); + var tone = SymbolTones[symbolId] with { StartSeconds = EventStartSeconds(eventIndex) }; + return new MimirChirpletTimelineEvent(eventIndex, symbolId, tone.StartSeconds, tone); + } + + public IReadOnlyList EventsOverlapping(double startSeconds, double durationSeconds) { var endSeconds = startSeconds + durationSeconds; var firstIndex = Math.Max(0, (long)Math.Floor((startSeconds - FirstEventSeconds - MaxEventDurationSeconds) / EventStrideSeconds) - 2); var lastIndex = Math.Max(firstIndex, (long)Math.Ceiling((endSeconds - FirstEventSeconds) / EventStrideSeconds) + 2); + var events = new List((int)(lastIndex - firstIndex + 1)); for (var eventIndex = firstIndex; eventIndex <= lastIndex; eventIndex++) { - var tone = ToneForEvent((ulong)eventIndex); - if (tone.StartSeconds < endSeconds && tone.StartSeconds + tone.DurationSeconds > startSeconds) + var timelineEvent = EventForIndex((ulong)eventIndex); + if (timelineEvent.StartSeconds < endSeconds && + timelineEvent.StartSeconds + timelineEvent.Tone.DurationSeconds > startSeconds) { - yield return tone; + events.Add(timelineEvent); } } + + return events; } - private static MimirChirpletTone ToneForEvent(ulong eventIndex) + private IReadOnlyList DetectSymbolEvents( + ReadOnlySpan samples, + int sampleRate, + int hopSamples) { - var seed = Mix(eventIndex); - var startJitter = Unit(seed) * 0.034 - 0.017; - var duration = 0.045 + Unit(seed >> 10) * 0.052; - var octaveJitter = Unit(seed >> 20) * 0.028 - 0.014; - var glideScale = 0.72 + Unit(seed >> 31) * 0.66; - var gain = 0.72 + Unit(seed >> 42) * 0.34; - var descending = ((seed >> 57) & 1UL) != 0; - var degree = (int)((seed >> 58) % 10); - var center = Math.Clamp(6_800.0 * HarmonicRatios[degree] * (1.0 + octaveJitter), 6_500.0, 16_300.0); - var halfWidth = Math.Clamp(center * 0.035 * glideScale, 220.0, 820.0); - var low = Math.Clamp(center - halfWidth, 6_300.0, 16_500.0); - var high = Math.Clamp(center + halfWidth, 6_300.0, 16_500.0); - var start = FirstEventSeconds + eventIndex * EventStrideSeconds + startJitter; - return descending - ? new MimirChirpletTone(start, duration, high, low, gain) - : new MimirChirpletTone(start, duration, low, high, gain); + if (samples.Length == 0) + { + return []; + } + + var coarse = ContrastNormalize(BuildEnvelopeEnergyTrace(samples, sampleRate, hopSamples)); + if (coarse.Length <= 2) + { + return []; + } + + var candidates = new List(); + for (var frame = 1; frame < coarse.Length - 1; frame++) + { + if (coarse[frame] < 0.055 || + coarse[frame] < coarse[frame - 1] || + coarse[frame] < coarse[frame + 1]) + { + continue; + } + + var observation = ClassifySymbolAt(samples, sampleRate, frame * hopSamples, Math.Max(1, hopSamples)); + if (observation != null) + { + candidates.Add(observation); + } + } + + return SuppressNearbyDetections(candidates, sampleRate); } - private static ulong Mix(ulong value) + private MimirChirpletEventObservation? ClassifySymbolAt( + ReadOnlySpan samples, + int sampleRate, + int predictedOffset, + int searchRadiusSamples) { - value += 0x9E3779B97F4A7C15UL; - value = (value ^ (value >> 30)) * 0xBF58476D1CE4E5B9UL; - value = (value ^ (value >> 27)) * 0x94D049BB133111EBUL; - return value ^ (value >> 31); + var bestSymbol = -1; + var bestOffset = predictedOffset; + var bestEnergy = 0.0; + for (var symbol = 0; symbol < SymbolTones.Length; symbol++) + { + var tone = SymbolTones[symbol]; + var kernel = sampleRate == SampleRate ? symbolKernels[symbol] : RenderToneKernel(tone, sampleRate); + if (kernel.Length == 0 || samples.Length < kernel.Length) + { + continue; + } + + var start = Math.Max(0, predictedOffset - searchRadiusSamples); + var end = Math.Min(samples.Length - kernel.Length, predictedOffset + searchRadiusSamples); + for (var offset = start; offset <= end; offset += Math.Max(1, searchRadiusSamples / 2)) + { + var energy = DirectionalMatchedEnergy(samples.Slice(offset, kernel.Length), kernel); + if (energy > bestEnergy) + { + bestEnergy = energy; + bestSymbol = symbol; + bestOffset = offset; + } + } + } + + return bestSymbol < 0 || bestEnergy < 0.055 + ? null + : new MimirChirpletEventObservation(0, bestSymbol, 0.0, bestOffset, bestEnergy); } - private static double Unit(ulong value) => - (value & ((1UL << 53) - 1)) / (double)(1UL << 53); + private static IReadOnlyList SuppressNearbyDetections( + IReadOnlyList observations, + int sampleRate) + { + if (observations.Count == 0) + { + return []; + } + + var minimumSpacingSamples = EventStrideSeconds * sampleRate * 0.55; + var kept = new List(); + foreach (var observation in observations.OrderByDescending(observation => observation.Energy)) + { + if (kept.Any(existing => Math.Abs(existing.SampleOffset - observation.SampleOffset) < minimumSpacingSamples)) + { + continue; + } + + kept.Add(observation); + } + + return kept.OrderBy(observation => observation.SampleOffset).ToArray(); + } + + private static MimirChirpletTimelinePlacement PlaceObservedSymbols( + IReadOnlyList observations, + int sampleRate) + { + if (observations.Count < 3) + { + return new MimirChirpletTimelinePlacement(0.0, 0.0, observations); + } + + var best = Array.Empty(); + var bestConfidence = 0.0; + for (var index = 0; index <= observations.Count - 3; index++) + { + var first = observations[index]; + var second = observations[index + 1]; + var third = observations[index + 2]; + if (!IsConsecutiveByTime(first, second, sampleRate) || + !IsConsecutiveByTime(second, third, sampleRate)) + { + continue; + } + + var code = TripleCode(first.SymbolId, second.SymbolId, third.SymbolId); + if (!TripleToIndex.TryGetValue(code, out var eventIndex)) + { + continue; + } + + var placed = new[] + { + PlaceObservation(first, (ulong)eventIndex), + PlaceObservation(second, (ulong)eventIndex + 1UL), + PlaceObservation(third, (ulong)eventIndex + 2UL), + }; + var confidence = placed.Average(observation => observation.Energy); + if (confidence > bestConfidence) + { + bestConfidence = confidence; + best = placed; + } + } + + return best.Length == 0 + ? new MimirChirpletTimelinePlacement(0.0, 0.0, observations) + : FitTimelinePlacement(best, sampleRate); + } + + private static bool IsConsecutiveByTime( + MimirChirpletEventObservation first, + MimirChirpletEventObservation second, + int sampleRate) + { + var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate; + return seconds >= EventStrideSeconds * 0.55 && seconds <= EventStrideSeconds * 1.45; + } + + private static MimirChirpletEventObservation PlaceObservation( + MimirChirpletEventObservation observation, + ulong eventIndex) + { + var timelineEvent = Default.EventForIndex(eventIndex); + return observation with + { + EventIndex = eventIndex, + SymbolId = timelineEvent.SymbolId, + TimelineSeconds = timelineEvent.StartSeconds, + }; + } + + private static MimirChirpletTimelinePlacement FitTimelinePlacement( + IReadOnlyList observations, + int sampleRate) + { + if (observations.Count == 0) + { + return new MimirChirpletTimelinePlacement(0.0, 0.0, []); + } + + var offsets = observations + .Select(observation => observation.TimelineSeconds - observation.SampleOffset / sampleRate) + .ToArray(); + var timelineStart = Median(offsets); + var residualSeconds = observations + .Average(observation => Math.Abs(observation.TimelineSeconds - observation.SampleOffset / sampleRate - timelineStart)); + var residualConfidence = 1.0 / (1.0 + residualSeconds / 0.004); + var countConfidence = Math.Clamp(observations.Count / 6.0, 0.0, 1.0); + var energyConfidence = Math.Clamp(observations.Average(observation => observation.Energy), 0.0, 1.0); + return new MimirChirpletTimelinePlacement( + timelineStart, + residualConfidence * 0.45 + countConfidence * 0.25 + energyConfidence * 0.30, + observations); + } + + private static MimirChirpletEventObservation? ObserveEvent( + ReadOnlySpan samples, + int sampleRate, + MimirChirpletTimelineEvent timelineEvent, + double predictedSample, + int searchRadiusSamples) + { + var kernel = RenderToneKernel(timelineEvent.Tone with { StartSeconds = 0.0 }, sampleRate); + if (kernel.Length == 0 || samples.Length < kernel.Length) + { + return null; + } + + var predictedOffset = (int)Math.Round(predictedSample); + var start = Math.Max(0, predictedOffset - searchRadiusSamples); + var end = Math.Min(samples.Length - kernel.Length, predictedOffset + searchRadiusSamples); + if (end < start) + { + return null; + } + + var bestOffset = start; + var bestEnergy = 0.0; + var step = Math.Max(1, sampleRate / 4_000); + for (var offset = start; offset <= end; offset += step) + { + var energy = DirectionalMatchedEnergy(samples.Slice(offset, kernel.Length), kernel); + if (energy > bestEnergy) + { + bestEnergy = energy; + bestOffset = offset; + } + } + + if (bestEnergy < 0.045) + { + return null; + } + + return new MimirChirpletEventObservation( + timelineEvent.Index, + timelineEvent.SymbolId, + timelineEvent.StartSeconds, + bestOffset, + bestEnergy); + } + + private static int SymbolForEvent(ulong eventIndex) => + TimelineSymbols[(int)(eventIndex % (ulong)TimelineSymbols.Length)]; + + private static double EventStartSeconds(ulong eventIndex) => + FirstEventSeconds + eventIndex * EventStrideSeconds; + + private static MimirChirpletTone[] BuildSymbolTones() + { + var tones = new MimirChirpletTone[SymbolCount]; + for (var symbol = 0; symbol < tones.Length; symbol++) + { + var baseHz = 6_500.0 * Math.Pow(16_400.0 / 6_500.0, symbol / (double)(SymbolCount - 1)); + var duration = 0.055; + var width = 190.0; + var descending = (symbol & 1) != 0; + var low = Math.Clamp(baseHz - width * 0.5, 6_300.0, 16_200.0); + var high = Math.Clamp(baseHz + width * 0.5, 6_400.0, 16_500.0); + tones[symbol] = descending + ? new MimirChirpletTone(0.0, duration, high, low, 0.85) + : new MimirChirpletTone(0.0, duration, low, high, 0.85); + } + + return tones; + } + + private static int[] BuildDeBruijn(int alphabetSize, int order) + { + var a = new int[alphabetSize * order]; + var sequence = new List(TimelinePeriod); + + void Db(int t, int p) + { + if (t > order) + { + if (order % p == 0) + { + for (var index = 1; index <= p; index++) + { + sequence.Add(a[index]); + } + } + + return; + } + + a[t] = a[t - p]; + Db(t + 1, p); + for (var j = a[t - p] + 1; j < alphabetSize; j++) + { + a[t] = j; + Db(t + 1, t); + } + } + + Db(1, 1); + return sequence.ToArray(); + } + + private static int[] RotateToDistinctOpening(int[] sequence) + { + for (var index = 0; index < sequence.Length; index++) + { + var first = sequence[index]; + var second = sequence[(index + 1) % sequence.Length]; + var third = sequence[(index + 2) % sequence.Length]; + if (first != second && first != third && second != third) + { + return sequence.Skip(index).Concat(sequence.Take(index)).ToArray(); + } + } + + return sequence; + } + + private static Dictionary BuildTripleIndex(IReadOnlyList symbols) + { + var map = new Dictionary(symbols.Count); + for (var index = 0; index < symbols.Count; index++) + { + var code = TripleCode( + symbols[index], + symbols[(index + 1) % symbols.Count], + symbols[(index + 2) % symbols.Count]); + map[code] = index; + } + + return map; + } + + private static int TripleCode(int first, int second, int third) => + first * SymbolCount * SymbolCount + second * SymbolCount + third; + + private static double Median(double[] values) + { + if (values.Length == 0) + { + return 0.0; + } + + Array.Sort(values); + var middle = values.Length / 2; + return values.Length % 2 == 0 + ? (values[middle - 1] + values[middle]) * 0.5 + : values[middle]; + } private static float[] RenderToneKernel(MimirChirpletTone tone, int sampleRate) { @@ -223,23 +588,6 @@ public sealed class MimirChirpletTimeline } } - private static float[] BuildMatchedEnergyTrace(ReadOnlySpan samples, ReadOnlySpan kernel, int hopSamples) - { - if (samples.Length < kernel.Length || kernel.Length == 0) - { - return []; - } - - var output = new float[1 + (samples.Length - kernel.Length) / hopSamples]; - for (var frame = 0; frame < output.Length; frame++) - { - var offset = frame * hopSamples; - output[frame] = (float)MatchedEnergy(samples.Slice(offset, kernel.Length), kernel); - } - - return output; - } - private static double MatchedEnergy(ReadOnlySpan samples, ReadOnlySpan kernel) { var count = Math.Min(samples.Length, kernel.Length); @@ -264,6 +612,118 @@ public sealed class MimirChirpletTimeline return denominator > 1.0e-12 ? Math.Abs(dot) / denominator : 0.0; } + private static double DirectionalMatchedEnergy(ReadOnlySpan samples, ReadOnlySpan kernel) + { + var count = Math.Min(samples.Length, kernel.Length); + if (count == 0) + { + return 0.0; + } + + var dot = 0.0; + var sampleEnergy = 0.0; + var kernelEnergy = 0.0; + for (var index = 0; index < count; index++) + { + var sample = samples[index]; + var basis = kernel[index]; + dot += sample * basis; + sampleEnergy += sample * sample; + kernelEnergy += basis * basis; + } + + var denominator = Math.Sqrt(sampleEnergy * kernelEnergy); + return denominator > 1.0e-12 ? Math.Max(0.0, dot / denominator) : 0.0; + } + + private float[] BuildCoarseEnergyTrace(ReadOnlySpan samples, int sampleRate, int hopSamples) + { + var traces = new List(); + for (var index = 0; index < CoarseTones.Length; index++) + { + var tone = CoarseTones[index]; + var kernel = sampleRate == SampleRate ? coarseKernels[index] : RenderToneKernel(tone, sampleRate); + var trace = BuildMatchedEnergyTrace(samples, kernel, hopSamples); + if (trace.Length > 0) + { + traces.Add(trace); + } + } + + if (traces.Count == 0) + { + return []; + } + + var length = traces.Min(trace => trace.Length); + var output = new float[length]; + for (var frame = 0; frame < output.Length; frame++) + { + var best = 0.0f; + foreach (var trace in traces) + { + best = Math.Max(best, trace[frame]); + } + + output[frame] = best; + } + + return output; + } + + private static float[] BuildEnvelopeEnergyTrace(ReadOnlySpan samples, int sampleRate, int hopSamples) + { + var windowSamples = Math.Max(1, (int)Math.Round(MaxEventDurationSeconds * sampleRate)); + if (samples.Length < windowSamples) + { + return []; + } + + var output = new float[1 + (samples.Length - windowSamples) / hopSamples]; + var energy = 0.0; + for (var index = 0; index < windowSamples; index++) + { + energy += samples[index] * samples[index]; + } + + output[0] = (float)Math.Sqrt(energy / windowSamples); + for (var frame = 1; frame < output.Length; frame++) + { + var previousStart = (frame - 1) * hopSamples; + var nextStart = frame * hopSamples; + for (var index = previousStart; index < nextStart; index++) + { + energy -= samples[index] * samples[index]; + } + + for (var index = previousStart + windowSamples; index < nextStart + windowSamples; index++) + { + energy += samples[index] * samples[index]; + } + + output[frame] = (float)Math.Sqrt(Math.Max(0.0, energy) / windowSamples); + } + + return output; + } + + private static float[] BuildMatchedEnergyTrace(ReadOnlySpan samples, ReadOnlySpan kernel, int hopSamples) + { + if (samples.Length < kernel.Length || kernel.Length == 0) + { + return []; + } + + var output = new float[1 + (samples.Length - kernel.Length) / hopSamples]; + for (var frame = 0; frame < output.Length; frame++) + { + var offset = frame * hopSamples; + output[frame] = (float)MatchedEnergy(samples.Slice(offset, kernel.Length), kernel); + } + + return output; + } + private static double MaxMatchedEnergy(ReadOnlySpan samples, ReadOnlySpan kernel, int hopSamples) { if (samples.Length < kernel.Length || kernel.Length == 0) diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs index 3136be2..80fb54f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs @@ -62,10 +62,12 @@ public sealed class MimirSynchronizationHub : IDisposable return $"{video} video / {audio} audio buffers"; } - public IReadOnlyList AnalyzeAudioSynchronization(string referenceSourceId) + public IReadOnlyList AnalyzeAudioSynchronization( + string referenceSourceId, + double approximateTimelineSeconds = double.PositiveInfinity) { ObjectDisposedException.ThrowIf(disposed, this); - var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId); + var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId, approximateTimelineSeconds); audioSynchronizationState.Update(reports); return reports; } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 3f7143b..438e1c2 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -39,3 +39,4 @@ {"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."} {"ts":"2026-05-22T13:12:32+01:00","type":"verification","status":"partial","note":"Actual Mimir.App runtime test with MIMIR_RUNTIME_CONFIG=config/mimir-runtime.probes.example.json and MIMIR_SYNC_TELEMETRY_SECONDS enabled proved Aquarium WASAPI output wakes the Scarlett loopback in the real app path. Headless run showed loopbackCount rising to 57 with mic buffers live; visible-window run showed loopbackCount 77 with Focusrite/Kiyo/Kiyo Pro/PS3 Eye mic buffers live. The runtime now updates sync analysis on a 0.5s cadence and telemetry prints loopback, reports, states, and audio buffer counts. No confident acoustic sync state locked in these app runs: reports were absent until enough phrase audio accumulated, then a live report appeared with confidence 0.000. Treat this as online emission/capture proven, online acoustic lock not yet solved."} {"ts":"2026-05-22T13:34:00+01:00","type":"implementation","status":"ok","note":"Replaced the discrete 16-phrase chirplet cycle with MimirChirpletTimeline, a deterministic unbounded chirplet stream. MimirRuntime now continuously queues uncached 0.5-second PCM segments through Aquarium audio with 1 second of lead, and MimirAudioSynchronizationAnalyzer extracts continuous chirplet atom energy traces from loopback and mic windows rather than searching a finite phrase cycle. Short headless Mimir.App telemetry runs proved the actual runtime path now locks online: loopback and all six audio buffers were live, reports were produced, and sync states appeared for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Delay estimates are still jumpy enough that the fractional-delay/SRO actuator should wait for better state filtering."} +{"ts":"2026-05-22T14:10:00+01:00","type":"implementation","status":"partial","note":"Changed MimirChirpletTimeline from anonymous deterministic texture to an explicit order-3 de Bruijn symbol timeline over 32 separated chirp symbols. Any three consecutive correctly detected symbols identify a timeline event within the current operating horizon; reports now expose timelineMatchedEvents and timelineConfidence when event-index refinement is used. A synthetic check rendered two seconds of canonical timeline audio and decoded events 0, 1, and 2 back to a window start of -0.0015 seconds. Real headless app sniff did not provide a clean physical proof because Scarlett loopback capture again stopped advancing early; build remains clean."} diff --git a/state/map.yaml b/state/map.yaml index ff6e312..209b96e 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -66,13 +66,14 @@ accepted_design: final local-device hot path. - The diagnostic audio path can optionally carry base64 float sample payloads for sync smoke tests. `MimirChirpletTimeline` owns the structured - birdsong-like timeline fingerprint: deterministic unbounded chirplet events - where event `N` is generated from `N`, plus PCM segment rendering, matched - timing atoms, and per-band response kernels. `MimirAudioSynchronizationAnalyzer` - now treats `loopback-scarlett-speakers` as timing authority, resamples - candidate mic windows into that timeline, projects loopback and mic windows - through continuous chirplet energy traces, and estimates current delay from - chirplet-energy traces at a shared timestamp edge. Positive delay means the + birdsong-like timeline fingerprint: an order-3 de Bruijn sequence over 32 + separated chirp symbols, plus PCM segment rendering, matched timing atoms, + symbol/event decoding, and per-band response kernels. Any three consecutive + correctly detected symbols identify a timeline event within the current + operating horizon. `MimirAudioSynchronizationAnalyzer` now treats + `loopback-scarlett-speakers` as timing authority, uses coarse chirplet + energy for acquisition, and refines delay from matched timeline event + indices when symbol decoding succeeds. Positive delay means the candidate mic is late relative to loopback. Reports include rounded delay, fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker` owns the smoothed per-source belief: latest delay, smoothed delay,