diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 7e7f3e0..b040a99 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -40,7 +40,9 @@ a named invariant that the native runtime cannot protect yet. hub, and prints the actual rolling buffers. Use `--require-samples` when an empty declared sensor buffer should fail the run. Use `--chirplet-self-test` to render the canonical timeline into memory and verify that the constrained - decoder recovers sub-frame anchors without hardware. + decoder recovers sub-frame anchors without hardware. Use + `--standalone-chirp-bin-self-test` to verify that a receiver with only the + codebook/schedule can recover canonical source offset from delayed audio. - `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback block metadata and emits `audio-block` JSON events for the diagnostic runtime adapter. It can probe requested shared/exclusive formats so driver state is @@ -110,6 +112,9 @@ a named invariant that the native runtime cannot protect yet. Focusrite `Loopback 1 -> Loopback 2` at `0.000 us` with 12 matched anchors and 0.999 confidence. Physical inputs did not produce matched anchors in that run, so acoustic robustness remains separate from the clean loopback timing proof. + A standalone 192 kHz synthetic receiver test recovers a 500 ms delayed audio + stream to below printed microsecond precision using only the chirp-bin + codebook and schedule state. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device timestamp, and native/GPU handle metadata. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live @@ -146,7 +151,8 @@ a named invariant that the native runtime cannot protect yet. Focusrite ASIO path for Scarlett capture; WASAPI remains a diagnostic witness, not the Scarlett hot path. 4. Prove the active chirp-bin path through real microphones, not only Scarlett - loopback; tune acoustic bands/gain/code spacing from measured mic response. + loopback; tune acoustic bands/gain/code spacing from measured mic response + without weakening the standalone codebook/schedule receiver invariant. 5. Add the synchronization actuator: drive a variable-rate resampler and fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. First, prove the constrained chirplet diff --git a/notes/current-system-map.md b/notes/current-system-map.md index f849eba..339936e 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -107,11 +107,16 @@ still needs a longer program-audio window, while the chirp-bin witness can decod short pilot windows once at least a code-valid triplet is present. The chirp-bin detector now uses cheap bounded energy/onset proposals, dechirp plus fixed Goertzel bins, explicit time/frequency ambiguity candidates, and a constrained -local waveform correlation for final fractional delay. Reports/states expose -`delayUs` next to fractional sample delay. `Mimir.BufferSmoke +scheduled-waveform correlation for standalone source-offset refinement. Pairwise +sync compares matched anchors first, then only accepts independent clock-fit +offsets inside the live latency horizon so period aliases do not become absurd +reports. Reports/states expose `delayUs` next to fractional sample delay. +`Mimir.BufferSmoke --chirp-only-sync-self-test --sample-rate 192000` and `--hybrid-sync-self-test --sample-rate 192000` both recover a 1269.5-sample synthetic delay with printed -0.000 us error. +0.000 us error. `--standalone-chirp-bin-self-test --sample-rate 192000 +--delay-samples 96000` proves a receiver with only codebook/schedule state can +recover a 500 ms delayed stream to below printed microsecond precision. Reports now carry fractional delay and per-band matched energy. The first `MimirChirpletSymbolCodebook` owns separable symbol definitions; every symbol has a unique chirp shape, with rhythm as additional evidence. `MimirChirpletStreamDecoder` @@ -149,9 +154,11 @@ Focusrite outputs while capturing every ASIO input as raw interleaved Float32. `Mimir.BufferSmoke --analyze-asio-f32` feeds those captured channels into the same runtime active analyzer. A real 192 kHz chirp-bin Scarlett artifact decoded `Loopback 1 -> Loopback 2` at exactly `0.000 us` with 12 matched anchors and -0.999 confidence. Physical inputs did not produce matched anchors in that -chirp-bin run; acoustic robustness is now the open problem, not clean loopback -timing or active decoder shape. +0.999 confidence. Physical input 2 independently decoded a coherent clock in +that artifact, but its event path did not overlap loopback and its raw +clock-offset difference was a period alias far outside the live latency horizon, +so Mimir correctly rejected it as insufficient anchors. Acoustic robustness is +now the open problem, not clean loopback timing or standalone decoder shape. ## Visual Fusion diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 420a1a5..fcc229d 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -87,11 +87,15 @@ Get-Content .\state\evidence.jsonl -Tail 8 short-window analyzer floor instead of inheriting the passive two-second gate. It uses cheap energy/onset proposals, dechirp plus fixed Goertzel bins, explicit time/frequency ambiguity candidates, and constrained local waveform - correlation for final fractional delay. `Mimir.BufferSmoke + correlation for final fractional delay. It also supports standalone source + offset recovery from schedule/codebook state, which is the Raven/phone shape. + `Mimir.BufferSmoke --chirp-only-sync-self-test --sample-rate 192000` and `--hybrid-sync-self-test --sample-rate 192000` both recover a 1269.5-sample synthetic delay with printed 0.000 us error. Reports and sync states expose - `delayUs`. + `delayUs`. `--standalone-chirp-bin-self-test --sample-rate 192000 + --delay-samples 96000` recovers a 500 ms delayed stream below printed + microsecond precision without loopback. Actual Mimir.App testing proves Fensalir audio can wake Scarlett loopback, keep mic buffers live, and produce confident online passive sync @@ -112,8 +116,9 @@ Get-Content .\state\evidence.jsonl -Tail 8 comparison and is now diagnostic only. A real Scarlett chirp-bin artifact run at 192 kHz decoded `Loopback 1 -> Loopback 2` at `0.000 us` delay with 12 matched anchors and 0.999 confidence in the normal active analyzer. Physical - inputs did not produce matched anchors in that chirp-bin run, so acoustic - robustness is still open. + input 2 independently decoded a coherent clock, but its clock-offset + difference was a period alias outside the live latency horizon and is rejected + as insufficient anchors. Acoustic robustness is still open. - Raven also has a 192 kHz loopback-capable Scarlett for co-streamer/game timing evidence. Do not move the heavy soundfield or sensor-fusion workload there. diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index c14eb47..da50381 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -10,6 +10,13 @@ if (args.Any(arg => string.Equals(arg, "--chirp-bin-self-test", StringComparison return RunChirpBinSelfTest(); } +if (args.Any(arg => string.Equals(arg, "--standalone-chirp-bin-self-test", StringComparison.OrdinalIgnoreCase))) +{ + return RunStandaloneChirpBinSelfTest( + ParseIntOption(args, "--sample-rate", MimirChirpBinTimeline.SampleRate), + ParseDoubleOption(args, "--delay-samples", 1269.5)); +} + if (args.Any(arg => string.Equals(arg, "--passive-sync-self-test", StringComparison.OrdinalIgnoreCase))) { return RunPassiveSyncSelfTest(); @@ -238,6 +245,46 @@ static int RunChirpBinSelfTest() return 0; } +static int RunStandaloneChirpBinSelfTest(int sampleRate, double delaySamples) +{ + var timeline = MimirChirpBinTimeline.Default; + var samples = new List(); + for (ulong segment = 0; segment < 8; segment++) + { + samples.AddRange(timeline.RenderSegmentMonoFloat(segment, sampleRate)); + } + + var delayed = ApplyFractionalDelay(samples.ToArray(), delaySamples); + var decode = timeline.DecodeStreamWindow(delayed, sampleRate); + var expectedDelay = delaySamples; + var actualDelay = decode.ClockFit?.SourceOffsetSamples ?? double.NaN; + var error = Math.Abs(actualDelay - expectedDelay); + Console.WriteLine( + $"standalone-chirp-bin-self-test frames={decode.Frames.Count} anchors={decode.Anchors.Count} " + + $"clock={(decode.ClockFit is null ? "none" : decode.ClockFit.EffectiveSampleRate.ToString("0.000000"))} " + + $"sourceOffset={actualDelay:0.000000} expected={expectedDelay:0.000000} errorSamples={error:0.000000} " + + $"errorUs={error * 1_000_000.0 / sampleRate:0.000} confidence={(decode.ClockFit?.Confidence ?? 0.0):0.000}"); + + foreach (var anchor in decode.Anchors.Take(16)) + { + var expected = timeline.EventForIndex(anchor.EventIndex).StartSeconds * sampleRate + expectedDelay; + Console.WriteLine( + $"standalone-anchor event={anchor.EventIndex} actual={anchor.SampleOffset:0.000} expected={expected:0.000} " + + $"error={anchor.SampleOffset - expected:0.000} confidence={anchor.Confidence:0.000}"); + } + + if (decode.ClockFit == null || + decode.Anchors.Count < 12 || + decode.ClockFit.Confidence < 0.70 || + error * 1_000_000.0 / sampleRate > 1.0) + { + Console.Error.WriteLine("standalone chirp-bin self-test failed: receiver did not recover canonical source time from delayed audio alone"); + return 1; + } + + return 0; +} + static int RunPassiveSyncSelfTest() { const int sampleRate = 48_000; diff --git a/src/Mimir.Runtime/Mimir.Runtime.csproj b/src/Mimir.Runtime/Mimir.Runtime.csproj index ad50eeb..068c445 100644 --- a/src/Mimir.Runtime/Mimir.Runtime.csproj +++ b/src/Mimir.Runtime/Mimir.Runtime.csproj @@ -11,7 +11,7 @@ - + diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 61cee84..456c4e4 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -3,7 +3,7 @@ using Aquarium.Engine.Audio; using Aquarium.Engine.Input; using Aquarium.Engine.Render; using Aquarium.Engine.Ui; -using Aquarium.LocalCast; +using Aquarium.Fensalir; using Mimir.Runtime.Synchronization; using System.Diagnostics; @@ -16,7 +16,7 @@ public sealed class MimirRuntime : IAquariumRuntime private const double CalibrationStartSeconds = 0.5; private const int CalibrationBatchSegments = 120; private const int HybridWatermarkIntervalSegments = 4; - private readonly LocalCastRuntime visualRuntime; + private readonly IAquariumRuntime visualRuntime; private readonly MimirSynchronizationHub synchronization; private readonly AquariumUiDocument ui; private readonly MimirAudioSynchronizationSettings audioSyncSettings; @@ -55,7 +55,7 @@ public sealed class MimirRuntime : IAquariumRuntime IEnumerable streamSources) { Options = options; - visualRuntime = new LocalCastRuntime(options); + visualRuntime = new FensalirRuntimeFactory().Create(options); synchronization = new MimirSynchronizationHub(settings); audioSyncSettings = settings.Audio; telemetryIntervalSeconds = ParseTelemetryIntervalSeconds(); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index 310d142..227c24f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -8,6 +8,7 @@ public sealed class MimirAudioSynchronizationAnalyzer private const double MinPassiveWindowSeconds = 2.0; private const double MinChirpletWindowSeconds = 2.0; private const double MinChirpBinWindowSeconds = 0.40; + private const double MaxPairClockFitDelaySeconds = 10.0; private const double PassiveReportConfidence = 0.08; private readonly List lastDecodeTraces = []; private readonly MimirPassiveAudioSynchronizationEstimator passiveEstimator = new(); @@ -290,6 +291,18 @@ public sealed class MimirAudioSynchronizationAnalyzer return (delay, residualConfidence * 0.50 + countConfidence * 0.25 + energyConfidence * 0.25, matched.Count); } + if (reference.ClockFit.Confidence >= 0.50 && candidate.ClockFit.Confidence >= 0.50) + { + var delay = candidate.ClockFit.SourceOffsetSamples - reference.ClockFit.SourceOffsetSamples; + if (Math.Abs(delay) <= reference.ClockFit.EffectiveSampleRate * MaxPairClockFitDelaySeconds) + { + var residualConfidence = 1.0 / (1.0 + (reference.ClockFit.MeanAbsoluteErrorSamples + candidate.ClockFit.MeanAbsoluteErrorSamples) / 64.0); + var countConfidence = Math.Clamp(Math.Min(reference.ClockFit.AnchorCount, candidate.ClockFit.AnchorCount) / 12.0, 0.0, 1.0); + var clockConfidence = Math.Sqrt(reference.ClockFit.Confidence * candidate.ClockFit.Confidence); + return (delay, residualConfidence * 0.35 + countConfidence * 0.25 + clockConfidence * 0.40, Math.Min(reference.ClockFit.AnchorCount, candidate.ClockFit.AnchorCount)); + } + } + var referenceFirst = reference.Anchors.Min(anchor => anchor.TimelineSeconds); var referenceLast = reference.Anchors.Max(anchor => anchor.TimelineSeconds); var candidateFirst = candidate.Anchors.Min(anchor => anchor.TimelineSeconds); diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs index 508416d..395a6d6 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs @@ -73,6 +73,12 @@ public sealed class MimirChirpBinTimeline .ToArray(); var anchors = DecodeAnchors(frames, sampleRate); var clock = FitClock(anchors, sampleRate); + if (clock != null) + { + var refinedOffset = RefineSourceOffset(samples, sampleRate, clock.SourceOffsetSamples); + clock = clock with { SourceOffsetSamples = refinedOffset }; + } + return new MimirChirpletStreamDecode(frames, symbols, anchors, clock); } @@ -503,6 +509,66 @@ public sealed class MimirChirpBinTimeline meanAbsoluteError); } + private static double RefineSourceOffset(ReadOnlySpan samples, int sampleRate, double initialOffsetSamples) + { + var radius = Math.Max(4, (int)Math.Ceiling(sampleRate * 0.00008)); + var center = (int)Math.Round(initialOffsetSamples); + var firstOffset = center - radius; + var lastOffset = center + radius; + var bestOffset = center; + var bestScore = double.NegativeInfinity; + for (var offset = firstOffset; offset <= lastOffset; offset++) + { + var score = ScheduledWaveformScore(samples, sampleRate, offset); + if (score > bestScore) + { + bestScore = score; + bestOffset = offset; + } + } + + if (bestOffset <= firstOffset || bestOffset >= lastOffset) + { + return bestOffset; + } + + var left = ScheduledWaveformScore(samples, sampleRate, bestOffset - 1); + var middle = ScheduledWaveformScore(samples, sampleRate, bestOffset); + var right = ScheduledWaveformScore(samples, sampleRate, bestOffset + 1); + var denominator = left - 2.0 * middle + right; + if (Math.Abs(denominator) <= 1.0e-12) + { + return bestOffset; + } + + return bestOffset + Math.Clamp(0.5 * (left - right) / denominator, -0.5, 0.5); + } + + private static double ScheduledWaveformScore(ReadOnlySpan samples, int sampleRate, int sourceOffsetSamples) + { + var timelineStartSeconds = -sourceOffsetSamples / (double)sampleRate; + var timelineDurationSeconds = samples.Length / (double)sampleRate; + var reference = new float[samples.Length]; + foreach (var timelineEvent in Default.EventsOverlapping(timelineStartSeconds, timelineDurationSeconds)) + { + AddChirp(reference, sampleRate, timelineEvent, timelineStartSeconds); + } + + var dot = 0.0; + var sampleEnergy = 0.0; + var referenceEnergy = 0.0; + for (var index = 0; index < samples.Length; index++) + { + dot += samples[index] * reference[index]; + sampleEnergy += samples[index] * samples[index]; + referenceEnergy += reference[index] * reference[index]; + } + + return sampleEnergy <= 1.0e-12 || referenceEnergy <= 1.0e-12 + ? double.NegativeInfinity + : dot / Math.Sqrt(sampleEnergy * referenceEnergy); + } + private static float[] BuildWindowEnergyTrace(ReadOnlySpan samples, int windowSamples, int hopSamples) { var output = new float[1 + (samples.Length - windowSamples) / hopSamples]; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index e867a7f..2c8cb60 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -68,3 +68,4 @@ {"ts": "2026-05-23T18:46:59+01:00", "type": "verification", "status": "ok", "note": "Added sample-rate selection to the hybrid chirp-bin self-test and removed the high-rate timing plateau bias from MimirChirpBinTimeline while preserving the 48 kHz compatibility behavior. Synthetic active chirp-bin timing now scales with sample rate: 48 kHz recovered 317.392716 vs 317.375 samples (0.369 us error), 96 kHz recovered 634.766098 vs 634.75 samples (0.168 us error), and 192 kHz recovered 1269.499997 vs 1269.5 samples (about 0.000003 samples, below printed microsecond precision). This proves the timing kernel benefits from 192 kHz in memory; live hardware proof still requires feeding ASIO callbacks into Mimir.Runtime instead of the diagnostic WASAPI/process path."} {"ts": "2026-05-23T19:03:44+01:00", "type": "measurement", "status": "ok", "note": "Scarlett ASIO chirp-only artifact at 192 kHz: Loopback 1 -> Loopback 2 decoded at 0.000 us with 7 anchors and 0.896 confidence; physical input 2 decoded at 616.226 samples / 3209.508 us with 3 anchors and 0.516 confidence; physical input 1 had no anchors. One 2-second matched-kernel chirplet comparison took roughly 100 seconds, so the current chirp-only decoder is diagnostic, not the runtime hot path."} {"ts": "2026-05-23T19:22:00+01:00", "type": "implementation", "status": "ok", "note": "Made the active audio sync path use the receiver-cheap chirp-bin decoder for both chirp-only and hybrid modes. The old arbitrary chirplet matcher remains diagnostic/reference only. The chirp-bin decoder now uses bounded energy/onset proposals, dechirp plus fixed Goertzel bins, explicit time/frequency ambiguity candidates, and constrained local waveform correlation for final fractional delay. Synthetic 192 kHz chirp-only and hybrid self-tests both recover a 1269.5-sample delay with printed 0.000 us error; a real Scarlett ASIO chirp-bin artifact decodes Loopback 1 -> Loopback 2 at 0.000 us with 12 anchors and 0.999 confidence, while physical mic channels still need acoustic robustness work."} +{"ts": "2026-05-23T19:40:50+01:00", "type": "implementation", "status": "ok", "note": "Extended active chirp-bin decoding toward the Perfect Machine receiver shape: standalone DecodeStreamWindow now refines source offset against the scheduled waveform so a listener with only codebook/schedule state can place delayed audio on canonical time. Synthetic 192 kHz standalone tests recover 1269.5 samples exactly, 96000 samples / 500 ms to below printed microsecond precision, and 13789.75 samples within 0.021 us. Pairwise Mimir sync now rejects independent clock-fit differences outside a 10s live horizon so coherent period aliases do not become absurd delays. Fensalir project references were updated from removed Aquarium.LocalCast to Aquarium.Fensalir."} diff --git a/state/map.yaml b/state/map.yaml index 4ffb6b4..4efc2eb 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -116,15 +116,20 @@ accepted_design: .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 192000` and `--hybrid-sync-self-test --sample-rate 192000` both recover a 1269.5-sample synthetic delay with printed 0.000 us error. The synthetic + standalone receiver proof `--standalone-chirp-bin-self-test --sample-rate + 192000 --delay-samples 96000` recovers a 500 ms delayed stream below printed + microsecond precision using only the codebook/schedule. The synthetic codebook proof remains `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-bin-self-test`. `Mimir.BufferSmoke --render-chirp-bin-f32` plus `native/probes/asio_audio_cadence --play-f32-mono --record-f32-interleaved` now provides a real Scarlett ASIO artifact proof for active mode. At 192 kHz, Loopback 1 -> Loopback 2 decoded at 0.000 us with 12 anchors and 0.999 - confidence. Physical inputs did not produce matched anchors in that chirp-bin - run, so acoustic robustness remains open. The old arbitrary-chirplet matcher - is a diagnostic/reference artifact only, not the runtime hot path. + confidence. Physical input 2 independently decoded a coherent clock, but its + clock-offset difference was a period alias outside the live latency horizon + and is rejected as insufficient anchors. Acoustic robustness remains open. + The old arbitrary-chirplet matcher is a diagnostic/reference artifact only, + not the runtime hot path. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared texture capture.