diff --git a/docs/audio-field.md b/docs/audio-field.md index 16c5f10..5cab302 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -51,12 +51,11 @@ timing evidence Mimir is allowed to emit: timing witness by estimating delay between the loopback buffer and each mic buffer. - `hybrid`: prefer passive program-audio evidence when confidence is high, then - emit a watermark when confidence falls. Today the passive side uses bounded - GCC-PHAT-style phase correlation and the fallback still uses the old active - chirplet pilot in half-second chunks. The intended watermark is the - dechirp/FFT-friendly chirp-bin codebook from the decoder research notes, - shaped and low-gain enough to sit inside the program audio instead of - announcing itself like a lab sweep. + emit a watermark when confidence falls. The passive side uses bounded + GCC-PHAT-style phase correlation. The active fallback now uses + `MimirChirpBinTimeline`: a fixed-slope chirp-bin watermark whose symbols are + decoded by one dechirp and a fixed Goertzel bin bank instead of dense sliding + matched filters. The mode belongs to the runtime, not the decoder. The decoder should consume known timing evidence; it should not decide whether Mimir is allowed to make @@ -71,6 +70,13 @@ evidence and carry zero confidence. A single passive window is also capped below certainty; full confidence belongs to repeated coherent state over time, not one attractive correlation peak. +The active watermark path is deliberately receiver-cheap. `MimirChirpBinTimeline` +uses a common chirp duration, chirp slope, and Hann window for every symbol. +Symbol identity is the dechirped frequency bin. The synthetic invariant is +`dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test`; +it renders the chirp-bin timeline, decodes by dechirp plus Goertzel bins, and +requires code-valid triplet anchors plus a stable source clock. + ## Next Cut The current diagnostic witness is `native/probes/wasapi_audio_cadence`, which @@ -186,13 +192,10 @@ timeline anchors. Current synthetic proof detects all 15 emitted chirps, keeps device runs still depend on loopback capture staying live; the local Scarlett loopback has intermittently stopped advancing during short headless sniffs. -This matcher is a proof, not the final hot path. `BuildChirpletEnergyTrace` -still behaves like a dense sliding matched-filter bank. The current research -note at `research/chirplet-sync-decoder/summary.md` points at the better -receiver shape: design the emitted symbol codebook so a mic can dechirp a -candidate event window and classify symbols by FFT or Goertzel bins. The -transmitter is ours; do not preserve arbitrary symbol shapes if they force the -receiver to spend a CPU core proving they exist. +The older `MimirChirpletTimeline` matcher is now the chirp-only lab path, not +the hybrid fallback. `BuildChirpletEnergyTrace` still behaves like a dense +sliding matched-filter bank and should not retake the hot path. The active +hybrid watermark belongs to the chirp-bin machine. Next, replace the diagnostic bridge with native audio capture workers that append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state, diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 8a750a8..7c07709 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -79,6 +79,10 @@ a named invariant that the native runtime cannot protect yet. path. It estimates loopback-to-mic delay with PHAT-weighted cross-spectrum correlation so music can act as the default timing witness before any audible watermark is needed. +- `MimirChirpBinTimeline` is the active hybrid watermark path. It renders a + fixed-slope chirp-bin codebook and decodes symbols with one dechirp plus + fixed Goertzel bins, then feeds the same de Bruijn triplet timeline-anchor + machine. Use `--chirp-bin-self-test` to prove this path without hardware. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device timestamp, and native/GPU handle metadata. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live @@ -118,10 +122,8 @@ a named invariant that the native runtime cannot protect yet. decoder through real loopback and microphone paths so every correctly heard triplet becomes a deterministic timeline anchor before the actuator moves samples. -5. Replace the active chirplet fallback in hybrid mode with shaped watermark - emission when passive confidence drops. The watermark codebook should be - designed for dechirp/FFT or Goertzel classification, not dense sliding - matched filters. +5. Prove the chirp-bin hybrid fallback through real loopback and microphones + with probe durations long enough to keep loopback and mic windows live. 6. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 7. Move GPU feature extraction, fusion, material fitting, render budgeting, and diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 48ed7b1..5e20472 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -5,6 +5,11 @@ if (args.Any(arg => string.Equals(arg, "--chirplet-self-test", StringComparison. return RunChirpletSelfTest(); } +if (args.Any(arg => string.Equals(arg, "--chirp-bin-self-test", StringComparison.OrdinalIgnoreCase))) +{ + return RunChirpBinSelfTest(); +} + if (args.Any(arg => string.Equals(arg, "--passive-sync-self-test", StringComparison.OrdinalIgnoreCase))) { return RunPassiveSyncSelfTest(); @@ -162,6 +167,41 @@ static int RunChirpletSelfTest() return 0; } +static int RunChirpBinSelfTest() +{ + var timeline = MimirChirpBinTimeline.Default; + var samples = new List(); + for (ulong segment = 0; segment < 6; segment++) + { + samples.AddRange(timeline.RenderSegmentMonoFloat(segment)); + } + + var decode = timeline.DecodeStreamWindow(samples.ToArray(), MimirChirpBinTimeline.SampleRate); + var meanAbsoluteError = decode.ClockFit?.MeanAbsoluteErrorSamples ?? double.PositiveInfinity; + Console.WriteLine( + $"chirp-bin-self-test frames={decode.Frames.Count} symbols={decode.Symbols.Count} anchors={decode.Anchors.Count} " + + $"clock={(decode.ClockFit is null ? "none" : decode.ClockFit.EffectiveSampleRate.ToString("0.000000"))} " + + $"confidence={(decode.ClockFit?.Confidence ?? 0.0):0.000} mae={meanAbsoluteError:0.000000}"); + Console.WriteLine("chirp-bin-expected " + string.Join(",", Enumerable.Range(0, 12).Select(index => timeline.EventForIndex((ulong)index).SymbolId))); + Console.WriteLine("chirp-bin-symbols " + string.Join(",", decode.Symbols.Take(12).Select(symbol => $"{symbol.SymbolId}@{symbol.SampleOffset:0}:{symbol.Energy:0.000}"))); + + foreach (var anchor in decode.Anchors.Take(16)) + { + var expected = timeline.EventForIndex(anchor.EventIndex).StartSeconds * MimirChirpBinTimeline.SampleRate; + Console.WriteLine( + $"chirp-bin-anchor event={anchor.EventIndex} actual={anchor.SampleOffset:0.000} expected={expected:0.000} " + + $"error={anchor.SampleOffset - expected:0.000} confidence={anchor.Confidence:0.000}"); + } + + if (decode.ClockFit == null || decode.Anchors.Count < 16 || meanAbsoluteError > 2.0) + { + Console.Error.WriteLine("chirp-bin self-test failed: dechirp/bin decoder did not recover stable timeline anchors"); + return 1; + } + + return 0; +} + static int RunPassiveSyncSelfTest() { const int sampleRate = 48_000; diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 9a7d49a..4f65303 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -156,14 +156,18 @@ public sealed class MimirRuntime : IAquariumRuntime } var segmentCount = audioSyncSettings.Mode == MimirAudioSyncMode.Hybrid ? 1 : CalibrationBatchSegments; - var batch = RenderCalibrationBatchPcm16Base64(calibrationSegmentIndex, segmentCount, out var peak); + var batch = RenderCalibrationBatchPcm16Base64( + calibrationSegmentIndex, + segmentCount, + audioSyncSettings.Mode == MimirAudioSyncMode.Hybrid, + out var peak); visualRuntime.Audio.EnqueuePcm16Base64( batch, MimirChirpletTimeline.SampleRate, channels: 1, gain: calibrationGain); Console.WriteLine( - $"mimir-chirplet-batch firstSegment={calibrationSegmentIndex} segments={segmentCount} seconds={segmentCount * MimirChirpletTimeline.SegmentSeconds:0.00} peak={peak:0.000000} gain={calibrationGain:0.###} base64Bytes={batch.Length}"); + $"mimir-chirplet-batch mode={DescribeAudioSyncMode()} firstSegment={calibrationSegmentIndex} segments={segmentCount} seconds={segmentCount * MimirChirpletTimeline.SegmentSeconds:0.00} peak={peak:0.000000} gain={calibrationGain:0.###} base64Bytes={batch.Length}"); calibrationSegmentIndex += (ulong)segmentCount; } @@ -178,7 +182,11 @@ public sealed class MimirRuntime : IAquariumRuntime return $"{audioSyncSettings.ReferenceSourceId} {DescribeAudioSyncMode()} timeline {MimirChirpletTimeline.SegmentSeconds:0.00}s segments emitted to {emittedUntilSeconds:0.00}s passiveConfidence={lastPassiveSynchronizationConfidence:0.000}"; } - private static string RenderCalibrationBatchPcm16Base64(ulong firstSegment, int segmentCount, out float peak) + private static string RenderCalibrationBatchPcm16Base64( + ulong firstSegment, + int segmentCount, + bool useChirpBinTimeline, + out float peak) { var samplesPerSegment = (int)Math.Round(MimirChirpletTimeline.SegmentSeconds * MimirChirpletTimeline.SampleRate); var bytes = new byte[samplesPerSegment * Math.Max(1, segmentCount) * sizeof(short)]; @@ -186,7 +194,9 @@ public sealed class MimirRuntime : IAquariumRuntime peak = 0.0f; for (var segment = 0; segment < segmentCount; segment++) { - var samples = MimirChirpletTimeline.Default.RenderSegmentMonoFloat(firstSegment + (ulong)segment); + var samples = useChirpBinTimeline + ? MimirChirpBinTimeline.Default.RenderSegmentMonoFloat(firstSegment + (ulong)segment) + : MimirChirpletTimeline.Default.RenderSegmentMonoFloat(firstSegment + (ulong)segment); for (var index = 0; index < samples.Length; index++) { peak = Math.Max(peak, Math.Abs(samples[index])); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index 1332ac9..635eab7 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -152,8 +152,12 @@ public sealed class MimirAudioSynchronizationAnalyzer } } - var comparedReferenceDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(referenceWindow, activeReferenceBlock.SampleRate); - var candidateDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(candidateWindow, activeReferenceBlock.SampleRate); + var comparedReferenceDecode = mode == MimirAudioSyncMode.Hybrid + ? MimirChirpBinTimeline.Default.DecodeStreamWindow(referenceWindow, activeReferenceBlock.SampleRate) + : MimirChirpletTimeline.Default.DecodeStreamWindow(referenceWindow, activeReferenceBlock.SampleRate); + var candidateDecode = mode == MimirAudioSyncMode.Hybrid + ? MimirChirpBinTimeline.Default.DecodeStreamWindow(candidateWindow, activeReferenceBlock.SampleRate) + : MimirChirpletTimeline.Default.DecodeStreamWindow(candidateWindow, activeReferenceBlock.SampleRate); var deterministicFit = EstimateDelayFromDecodedTimeline(comparedReferenceDecode, candidateDecode); lastDecodeTraces.Add(new MimirAudioSynchronizationDecodeTrace( reference.Descriptor.SourceId, @@ -177,7 +181,9 @@ public sealed class MimirAudioSynchronizationAnalyzer } var decodedDelaySamples = deterministicFit.DelaySamples; - var bandResponses = MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, activeReferenceBlock.SampleRate); + var bandResponses = mode == MimirAudioSyncMode.Hybrid + ? [] + : MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, activeReferenceBlock.SampleRate); reports.Add(new MimirAudioSynchronizationReport( reference.Descriptor.SourceId, buffer.Descriptor.SourceId, @@ -193,7 +199,7 @@ public sealed class MimirAudioSynchronizationAnalyzer buffer.Latest?.Sequence ?? 0, deterministicFit.MatchedEvents, deterministicFit.Confidence, - "chirplet")); + mode == MimirAudioSyncMode.Hybrid ? "chirp-bin" : "chirplet")); } return reports; diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs new file mode 100644 index 0000000..d3d425c --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs @@ -0,0 +1,596 @@ +using System.Numerics; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirChirpBinSymbolDefinition( + int SymbolId, + double OffsetHz); + +public sealed class MimirChirpBinTimeline +{ + public const int SampleRate = 48_000; + public const double SegmentSeconds = 0.5; + public const int SymbolCount = 32; + public const int TimelineOrder = 3; + + private const double FirstEventSeconds = 0.08; + private const double EventSpacingSeconds = 0.125; + private const double ChirpDurationSeconds = 0.064; + private const double BaseStartHz = 7_600.0; + private const double BaseEndHz = 14_800.0; + private const double BinSpacingHz = 62.5; + private const double Gain = 0.070; + private const int MaxSymbolCandidatesPerFrame = 6; + + private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder)); + private static readonly Dictionary TripleToIndex = BuildTripleIndex(TimelineSymbols); + private static readonly MimirChirpBinSymbolDefinition[] Symbols = BuildSymbols(); + + public static MimirChirpBinTimeline Default { get; } = new(); + + public IReadOnlyList Codebook => Symbols; + + public float[] RenderSegmentMonoFloat(ulong segmentIndex) + { + var segmentStartSeconds = segmentIndex * SegmentSeconds; + var samples = new float[(int)Math.Round(SegmentSeconds * SampleRate)]; + foreach (var timelineEvent in EventsOverlapping(segmentStartSeconds, SegmentSeconds)) + { + AddChirp(samples, SampleRate, timelineEvent, segmentStartSeconds); + } + + return samples; + } + + public MimirChirpletStreamDecode DecodeStreamWindow(ReadOnlySpan samples, int sampleRate) + { + if (samples.Length == 0) + { + return new MimirChirpletStreamDecode([], [], [], null); + } + + var frames = DetectFrames(samples, sampleRate); + var symbols = frames + .Select(frame => new MimirChirpletSymbolObservation( + frame.BestCandidate.SymbolId, + frame.SampleOffset, + frame.BestCandidate.Energy)) + .ToArray(); + var anchors = DecodeAnchors(frames, sampleRate); + var clock = FitClock(anchors, sampleRate); + return new MimirChirpletStreamDecode(frames, symbols, anchors, clock); + } + + public MimirChirpletTimelineEvent EventForIndex(ulong eventIndex) + { + var symbolId = SymbolForEvent(eventIndex); + var startSeconds = EventStartSeconds(eventIndex); + var symbol = Symbols[symbolId]; + return new MimirChirpletTimelineEvent( + eventIndex, + symbolId, + startSeconds, + new MimirChirpletTone( + startSeconds, + ChirpDurationSeconds, + BaseStartHz + symbol.OffsetHz, + BaseEndHz + symbol.OffsetHz, + Gain)); + } + + public IReadOnlyList EventsOverlapping(double startSeconds, double durationSeconds) + { + var endSeconds = startSeconds + durationSeconds; + var firstIndex = Math.Max(0, (long)Math.Floor((startSeconds - FirstEventSeconds - ChirpDurationSeconds) / EventSpacingSeconds) - 2); + var lastIndex = Math.Max(firstIndex, (long)Math.Ceiling((endSeconds - FirstEventSeconds) / EventSpacingSeconds) + 2); + var events = new List((int)(lastIndex - firstIndex + 1)); + for (var eventIndex = firstIndex; eventIndex <= lastIndex; eventIndex++) + { + var timelineEvent = EventForIndex((ulong)eventIndex); + if (timelineEvent.StartSeconds < endSeconds && + timelineEvent.StartSeconds + ChirpDurationSeconds > startSeconds) + { + events.Add(timelineEvent); + } + } + + return events; + } + + private static IReadOnlyList DetectFrames(ReadOnlySpan samples, int sampleRate) + { + var chirpSamples = Math.Max(1, (int)Math.Round(ChirpDurationSeconds * sampleRate)); + if (samples.Length < chirpSamples) + { + return []; + } + + var hopSamples = Math.Max(1, sampleRate / 400); + var energyTrace = BuildEnergyTrace(samples, chirpSamples, hopSamples); + var threshold = AdaptiveThreshold(energyTrace); + var proposals = new List(); + for (var index = 1; index < energyTrace.Length - 1; index++) + { + if (energyTrace[index] >= threshold && + energyTrace[index] >= energyTrace[index - 1] && + energyTrace[index] >= energyTrace[index + 1]) + { + proposals.Add(index * hopSamples); + } + } + + var frames = new List(); + foreach (var proposal in proposals) + { + var frame = ClassifyAt(samples, sampleRate, proposal, sampleRate / 120); + if (frame != null) + { + frames.Add(frame); + } + } + + return SuppressNearbyFrames(frames, sampleRate); + } + + private static MimirChirpletTransformFrame? ClassifyAt( + ReadOnlySpan samples, + int sampleRate, + int predictedOffset, + int searchRadiusSamples) + { + var chirpSamples = Math.Max(1, (int)Math.Round(ChirpDurationSeconds * sampleRate)); + if (samples.Length < chirpSamples) + { + return null; + } + + var start = Math.Max(0, predictedOffset - searchRadiusSamples); + var end = Math.Min(samples.Length - chirpSamples, predictedOffset + searchRadiusSamples); + var bestOffset = predictedOffset; + MimirChirpletSymbolCandidate[] bestCandidates = []; + var bestEnergy = 0.0; + var step = Math.Max(2, searchRadiusSamples / 24); + for (var offset = start; offset <= end; offset += step) + { + var candidates = ScoreBins(samples.Slice(offset, chirpSamples), sampleRate, offset); + var energy = candidates.Length == 0 ? 0.0 : candidates[0].Energy; + if (energy > bestEnergy) + { + bestEnergy = energy; + bestOffset = offset; + bestCandidates = candidates; + } + } + + var localStart = Math.Max(start, bestOffset - step); + var localEnd = Math.Min(end, bestOffset + step); + for (var offset = localStart; offset <= localEnd; offset++) + { + var candidates = ScoreBins(samples.Slice(offset, chirpSamples), sampleRate, offset); + var energy = candidates.Length == 0 ? 0.0 : candidates[0].Energy; + if (energy > bestEnergy) + { + bestEnergy = energy; + bestOffset = offset; + bestCandidates = candidates; + } + } + + if (bestCandidates.Length == 0 || bestEnergy < 0.11) + { + return null; + } + + return new MimirChirpletTransformFrame(bestOffset, bestCandidates); + } + + private static MimirChirpletSymbolCandidate[] ScoreBins( + ReadOnlySpan samples, + int sampleRate, + int sampleOffset) + { + var scores = new MimirChirpletSymbolCandidate[Symbols.Length]; + var sampleEnergy = 0.0; + var referenceEnergy = 0.0; + for (var index = 0; index < samples.Length; index++) + { + sampleEnergy += samples[index] * samples[index]; + var normalized = samples.Length <= 1 ? 1.0 : index / (double)(samples.Length - 1); + var window = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); + referenceEnergy += window * window * 0.5; + } + + if (sampleEnergy <= 1.0e-12 || referenceEnergy <= 1.0e-12) + { + return []; + } + + for (var symbol = 0; symbol < Symbols.Length; symbol++) + { + var score = GoertzelDechirpedBin(samples, sampleRate, Symbols[symbol].OffsetHz); + scores[symbol] = new MimirChirpletSymbolCandidate( + symbol, + sampleOffset, + 2.0 * score / Math.Sqrt(sampleEnergy * referenceEnergy)); + } + + return scores + .OrderByDescending(candidate => candidate.Energy) + .Take(MaxSymbolCandidatesPerFrame) + .ToArray(); + } + + private static double GoertzelDechirpedBin(ReadOnlySpan samples, int sampleRate, double offsetHz) + { + var real = 0.0; + var imaginary = 0.0; + for (var index = 0; index < samples.Length; index++) + { + var t = index / (double)sampleRate; + var normalized = samples.Length <= 1 ? 1.0 : index / (double)(samples.Length - 1); + var window = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); + var basePhase = BasePhase(t); + var binPhase = 2.0 * Math.PI * offsetHz * t; + var phase = -(basePhase + binPhase); + var value = samples[index] * window; + real += value * Math.Cos(phase); + imaginary += value * Math.Sin(phase); + } + + return Math.Sqrt(real * real + imaginary * imaginary); + } + + private static IReadOnlyList DecodeAnchors( + IReadOnlyList frames, + int sampleRate) + { + if (frames.Count < TimelineOrder) + { + return []; + } + + var candidates = new List(); + for (var index = 0; index <= frames.Count - TimelineOrder; index++) + { + var firstFrame = frames[index]; + var secondFrame = frames[index + 1]; + var thirdFrame = frames[index + 2]; + if (!IsConsecutive(firstFrame, secondFrame, sampleRate) || + !IsConsecutive(secondFrame, thirdFrame, sampleRate)) + { + continue; + } + + foreach (var first in firstFrame.Candidates) + foreach (var second in secondFrame.Candidates) + foreach (var third in thirdFrame.Candidates) + { + var code = TripleCode(first.SymbolId, second.SymbolId, third.SymbolId); + if (!TripleToIndex.TryGetValue(code, out var eventIndex)) + { + continue; + } + + var firstEvent = Default.EventForIndex((ulong)eventIndex); + var measuredGapA = (second.SampleOffset - first.SampleOffset) / sampleRate; + var measuredGapB = (third.SampleOffset - second.SampleOffset) / sampleRate; + var gapError = Math.Abs(measuredGapA - EventSpacingSeconds) + Math.Abs(measuredGapB - EventSpacingSeconds); + var gapConfidence = 1.0 / (1.0 + gapError / 0.003); + if (gapConfidence < 0.55) + { + continue; + } + + var energyConfidence = Math.Clamp((first.Energy + second.Energy + third.Energy) / 3.0, 0.0, 1.0); + candidates.Add(new MimirChirpletTimelineAnchor( + (ulong)eventIndex, + firstEvent.StartSeconds, + first.SampleOffset, + gapConfidence * 0.60 + energyConfidence * 0.40, + [ + new MimirChirpletSymbolObservation(first.SymbolId, first.SampleOffset, first.Energy), + new MimirChirpletSymbolObservation(second.SymbolId, second.SampleOffset, second.Energy), + new MimirChirpletSymbolObservation(third.SymbolId, third.SampleOffset, third.Energy), + ])); + } + } + + return SelectCoherentAnchorPath(candidates, sampleRate); + } + + private static IReadOnlyList SelectCoherentAnchorPath( + IReadOnlyList candidates, + int sampleRate) + { + if (candidates.Count == 0) + { + return []; + } + + List bestPath = []; + var bestScore = double.NegativeInfinity; + foreach (var seed in candidates.OrderByDescending(anchor => anchor.Confidence).Take(64)) + { + var seedOffset = seed.SampleOffset - seed.TimelineSeconds * sampleRate; + var path = candidates + .Where(anchor => Math.Abs((anchor.SampleOffset - anchor.TimelineSeconds * sampleRate) - seedOffset) <= sampleRate * 0.006) + .GroupBy(anchor => anchor.EventIndex) + .Select(group => group.OrderByDescending(anchor => anchor.Confidence).First()) + .OrderBy(anchor => anchor.EventIndex) + .ToList(); + var clock = FitClock(path, sampleRate); + if (clock == null) + { + continue; + } + + var score = path.Sum(anchor => anchor.Confidence) + + Math.Min(path.Count, 12) * 0.20 - + clock.MeanAbsoluteErrorSamples / Math.Max(1.0, sampleRate * 0.001); + if (score > bestScore) + { + bestScore = score; + bestPath = path; + } + } + + var finalClock = FitClock(bestPath, sampleRate); + if (finalClock == null) + { + return []; + } + + return bestPath + .Where(anchor => Math.Abs(anchor.SampleOffset - finalClock.SampleForTimelineSeconds(anchor.TimelineSeconds)) <= Math.Max(16.0, sampleRate * 0.0015)) + .OrderBy(anchor => anchor.SampleOffset) + .ToArray(); + } + + private static MimirChirpletClockFit? FitClock( + IReadOnlyList anchors, + int sampleRate) + { + if (anchors.Count == 0) + { + return null; + } + + if (anchors.Count == 1) + { + var anchor = anchors[0]; + return new MimirChirpletClockFit( + anchor.SampleOffset - anchor.TimelineSeconds * sampleRate, + sampleRate, + anchor.Confidence * 0.35, + 1, + 0.0); + } + + var totalWeight = anchors.Sum(anchor => Math.Max(1.0e-6, anchor.Confidence)); + var meanTimeline = anchors.Sum(anchor => anchor.TimelineSeconds * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var meanSample = anchors.Sum(anchor => anchor.SampleOffset * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var covariance = 0.0; + var variance = 0.0; + foreach (var anchor in anchors) + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var dt = anchor.TimelineSeconds - meanTimeline; + covariance += weight * dt * (anchor.SampleOffset - meanSample); + variance += weight * dt * dt; + } + + var effectiveSampleRate = variance > 1.0e-12 ? covariance / variance : sampleRate; + if (!double.IsFinite(effectiveSampleRate) || + effectiveSampleRate < sampleRate * 0.98 || + effectiveSampleRate > sampleRate * 1.02) + { + effectiveSampleRate = sampleRate; + } + + var sourceOffset = meanSample - effectiveSampleRate * meanTimeline; + var meanAbsoluteError = anchors.Sum(anchor => + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var predicted = sourceOffset + anchor.TimelineSeconds * effectiveSampleRate; + return Math.Abs(anchor.SampleOffset - predicted) * weight; + }) / totalWeight; + var residualConfidence = 1.0 / (1.0 + meanAbsoluteError / Math.Max(1.0, sampleRate * 0.00075)); + var countConfidence = Math.Clamp(anchors.Count / 12.0, 0.0, 1.0); + var anchorConfidence = Math.Clamp(anchors.Average(anchor => anchor.Confidence), 0.0, 1.0); + return new MimirChirpletClockFit( + sourceOffset, + effectiveSampleRate, + residualConfidence * 0.45 + countConfidence * 0.25 + anchorConfidence * 0.30, + anchors.Count, + meanAbsoluteError); + } + + private static float[] BuildEnergyTrace(ReadOnlySpan samples, int windowSamples, int hopSamples) + { + var output = new float[1 + (samples.Length - windowSamples) / hopSamples]; + var energy = 0.0; + for (var index = 0; index < windowSamples; index++) + { + energy += samples[index] * samples[index]; + } + + output[0] = (float)Math.Sqrt(energy / windowSamples); + for (var frame = 1; frame < output.Length; frame++) + { + var previousStart = (frame - 1) * hopSamples; + var nextStart = frame * hopSamples; + for (var index = previousStart; index < nextStart; index++) + { + energy -= samples[index] * samples[index]; + } + + for (var index = previousStart + windowSamples; index < nextStart + windowSamples; index++) + { + energy += samples[index] * samples[index]; + } + + output[frame] = (float)Math.Sqrt(Math.Max(0.0, energy) / windowSamples); + } + + return output; + } + + private static double AdaptiveThreshold(float[] energyTrace) + { + if (energyTrace.Length == 0) + { + return double.PositiveInfinity; + } + + var mean = energyTrace.Average(value => (double)value); + var variance = energyTrace.Sum(value => (value - mean) * (value - mean)) / energyTrace.Length; + return mean + Math.Sqrt(Math.Max(0.0, variance)) * 0.75; + } + + private static IReadOnlyList SuppressNearbyFrames( + IReadOnlyList frames, + int sampleRate) + { + var minimumSpacingSamples = EventSpacingSeconds * sampleRate * 0.65; + var kept = new List(); + foreach (var frame in frames.OrderByDescending(frame => frame.BestCandidate.Energy)) + { + if (kept.Any(existing => Math.Abs(existing.SampleOffset - frame.SampleOffset) < minimumSpacingSamples)) + { + continue; + } + + kept.Add(frame); + } + + return kept.OrderBy(frame => frame.SampleOffset).ToArray(); + } + + private static bool IsConsecutive( + MimirChirpletTransformFrame first, + MimirChirpletTransformFrame second, + int sampleRate) + { + var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate; + return seconds >= EventSpacingSeconds * 0.70 && seconds <= EventSpacingSeconds * 1.30; + } + + private static void AddChirp( + float[] samples, + int sampleRate, + MimirChirpletTimelineEvent timelineEvent, + double segmentStartSeconds) + { + var symbol = Symbols[timelineEvent.SymbolId]; + var startFrame = (int)Math.Round((timelineEvent.StartSeconds - segmentStartSeconds) * sampleRate); + var frameCount = Math.Max(1, (int)Math.Round(ChirpDurationSeconds * sampleRate)); + for (var frame = 0; frame < frameCount; frame++) + { + var outputFrame = startFrame + frame; + if (outputFrame < 0 || outputFrame >= samples.Length) + { + continue; + } + + var normalized = frameCount <= 1 ? 1.0 : frame / (double)(frameCount - 1); + var t = frame / (double)sampleRate; + var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); + var phase = BasePhase(t) + 2.0 * Math.PI * symbol.OffsetHz * t; + samples[outputFrame] += (float)(Math.Sin(phase) * envelope * Gain); + } + } + + private static double BasePhase(double t) + { + var slope = (BaseEndHz - BaseStartHz) / ChirpDurationSeconds; + return 2.0 * Math.PI * (BaseStartHz * t + 0.5 * slope * t * t); + } + + private static MimirChirpBinSymbolDefinition[] BuildSymbols() + { + var symbols = new MimirChirpBinSymbolDefinition[SymbolCount]; + var firstOffset = -(SymbolCount - 1) * BinSpacingHz * 0.5; + for (var symbol = 0; symbol < symbols.Length; symbol++) + { + symbols[symbol] = new MimirChirpBinSymbolDefinition(symbol, firstOffset + symbol * BinSpacingHz); + } + + return symbols; + } + + private static int SymbolForEvent(ulong eventIndex) => + TimelineSymbols[(int)(eventIndex % (ulong)TimelineSymbols.Length)]; + + private static double EventStartSeconds(ulong eventIndex) => + FirstEventSeconds + eventIndex * EventSpacingSeconds; + + private static int[] BuildDeBruijn(int alphabetSize, int order) + { + var a = new int[alphabetSize * order]; + var sequence = new List((int)Math.Pow(alphabetSize, order)); + + void Db(int t, int p) + { + if (t > order) + { + if (order % p == 0) + { + for (var index = 1; index <= p; index++) + { + sequence.Add(a[index]); + } + } + + return; + } + + a[t] = a[t - p]; + Db(t + 1, p); + for (var j = a[t - p] + 1; j < alphabetSize; j++) + { + a[t] = j; + Db(t + 1, t); + } + } + + Db(1, 1); + return sequence.ToArray(); + } + + private static int[] RotateToDistinctOpening(int[] sequence) + { + for (var index = 0; index < sequence.Length; index++) + { + var first = sequence[index]; + var second = sequence[(index + 1) % sequence.Length]; + var third = sequence[(index + 2) % sequence.Length]; + if (first != second && first != third && second != third) + { + return sequence.Skip(index).Concat(sequence.Take(index)).ToArray(); + } + } + + return sequence; + } + + private static Dictionary BuildTripleIndex(IReadOnlyList symbols) + { + var map = new Dictionary(symbols.Count); + for (var index = 0; index < symbols.Count; index++) + { + var code = TripleCode( + symbols[index], + symbols[(index + 1) % symbols.Count], + symbols[(index + 2) % symbols.Count]); + if (!map.ContainsKey(code)) + { + map[code] = index; + } + } + + return map; + } + + private static int TripleCode(int first, int second, int third) => + first * SymbolCount * SymbolCount + second * SymbolCount + third; +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 7e6eac0..6077848 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -56,3 +56,4 @@ {"ts": "2026-05-22T19:08:10+01:00", "type": "implementation", "status": "ok", "note": "Renamed the public no-chirp mode to passive and started the real hybrid sync machine. Added MimirPassiveAudioSynchronizationEstimator using PHAT-weighted cross-spectrum delay estimation over loopback/program audio; MimirAudioSynchronizationAnalyzer now reports passive evidence in passive/hybrid modes and falls back to chirplet anchors when hybrid passive confidence is weak. Hybrid runtime emission now queues short active pilot chunks only while passive confidence stays below threshold. dotnet build, passive sync self-test, and chirplet self-test passed."} {"ts": "2026-05-22T19:08:56+01:00", "type": "verification", "status": "partial", "note": "Passive live smoke with MIMIR_AUDIO_SYNC_MODE=passive and config/mimir-runtime.probes.example.json emitted no calibration audio and produced loopback/program-audio reports: Focusrite delaySamples=6653 confidence=0.805, Kiyo Pro delaySamples=6804 confidence=0.227, USB Camera 0 delaySamples=-882 confidence=1.000 over 88200 compared 44.1k samples. This proves the passive path can lock on real loopback/music, but the negative Eye delay needs timestamp/window-sign inspection before driving an actuator."} {"ts": "2026-05-22T19:33:21+01:00", "type": "verification", "status": "ok", "note": "Fixed passive sync confidence bug. Negative passive lags now produce status=passive-negative-lag with confidence=0, single-window passive confidence is capped at 0.85, and hub invalidates cached reports/state for negative-lag traces. Passive self-test now proves both positive delay recovery and reversed-input zero confidence. A 14s live passive smoke showed negative Focusrite evidence at confidence 0 without leaving a report/state, while positive passive reports stayed capped at 0.85."} +{"ts": "2026-05-22T19:49:43+01:00", "type": "implementation", "status": "partial", "note": "Built the first chirp-bin active watermark machine. MimirChirpBinTimeline renders a fixed-slope/window 32-bin chirp codebook and decodes with one dechirp plus fixed Goertzel bin scoring, then feeds de Bruijn triplet anchors and clock fitting. Hybrid fallback now renders/decodes chirp-bin evidence instead of the old dense chirplet matcher. Synthetic --chirp-bin-self-test passed with 23 frames, 21 anchors, 48000 Hz clock, and zero MAE; old --chirplet-self-test still passed. A real hybrid app run emitted chirp-bin batches, but the current probe config's 30s capture limit left loopback/mic windows stale before acoustic chirp-bin reports could be proven."} diff --git a/state/map.yaml b/state/map.yaml index c077219..06919a0 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -102,6 +102,11 @@ accepted_design: windows before any watermark is needed. Negative passive lags are contradictory evidence and carry zero confidence; single-window passive confidence is capped below certainty until repeated state earns it. + `MimirChirpBinTimeline` is the active hybrid watermark path: one fixed chirp + slope/window, 32 symbol frequency bins, dechirp plus fixed Goertzel bin + scoring, then the existing de Bruijn triplet anchor and clock-fit machine. + The synthetic proof is `dotnet run --project + .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test`. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared texture capture.