From 1cf46ef97ba3f35ad4ff5529e8d7416e5fbc40e6 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 22 May 2026 12:46:51 +0100 Subject: [PATCH] Track chirplet audio sync state --- docs/audio-field.md | 12 ++- docs/implementation-plan.md | 11 ++- notes/current-system-map.md | 7 +- notes/fresh-workspace-handoff.md | 3 +- src/Mimir.BufferSmoke/Program.cs | 13 +++ src/Mimir.Runtime/MimirRuntime.cs | 9 ++ .../MimirAudioSynchronizationAnalyzer.cs | 1 + .../MimirAudioSynchronizationReport.cs | 1 + .../MimirAudioSynchronizationState.cs | 89 +++++++++++++++++++ .../MimirSynchronizationHub.cs | 8 +- state/evidence.jsonl | 1 + state/map.yaml | 9 +- 12 files changed, 150 insertions(+), 14 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationState.cs diff --git a/docs/audio-field.md b/docs/audio-field.md index a059575..2eb6b8e 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -78,6 +78,13 @@ estimation: loopback carries what was emitted, each mic carries what survived speaker, air, room, and capsule, and the ratio over repeated chirplet phrases becomes gain/phase correction evidence. +`MimirAudioSynchronizationStateTracker` turns per-phrase observations into +state. It confidence-gates reports, smooths fractional delay per source, and +estimates delay slope as sampling-rate offset in ppm. This is the control input +for the coming actuator. The state can survive a brief weak report, but it is +not a license to run blind: loopback must keep receiving the emitted phrase or +fresh reports will stop. + ## Chirplet Calibration Model ```mermaid @@ -89,9 +96,10 @@ flowchart TD D --> F["mic rolling buffers"] E --> G["matched chirplet traces"] F --> G - G --> H["delay + SRO estimates"] + G --> H["delay observations"] + H --> L["smoothed sync state + SRO"] G --> I["per-band response estimates"] - H --> J["fractional delay / resampler actuator"] + L --> J["fractional delay / resampler actuator"] I --> K["frequency response normalization"] ``` diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index a5bf8e4..91cd7a1 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -51,6 +51,9 @@ a named invariant that the native runtime cannot protect yet. - `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional aligned mono frame for loopback-referenced audio channels that clear the confidence gate. +- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source + sync state: latest fractional delay, smoothed delay, confidence, per-band + response evidence, and delay-slope/SRO estimate in ppm. - `MimirRuntime` emits short calibration chirplets through Aquarium audio so the loopback and acoustic mic buffers carry a shared timing witness. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device @@ -86,10 +89,10 @@ a named invariant that the native runtime cannot protect yet. frame cadence in the rolling buffers. 3. Replace the WASAPI frame-event diagnostic bridge with native audio capture workers for local mic, loopback, and network audio feeds. -4. Add the synchronization actuator: smooth delay/SRO estimates, then drive a - variable-rate resampler and fractional delay line per non-reference stream. - The current aligned frame is integer-delay only and should not pretend to be - the final DSP path. +4. Add the synchronization actuator: drive a variable-rate resampler and + fractional delay line per non-reference stream from the smoothed + `MimirAudioSynchronizationState`. The current aligned frame is integer-delay + only and should not pretend to be the final DSP path. 5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 6. Move GPU feature extraction, fusion, material fitting, render budgeting, and diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 7c69c27..5ba15da 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -96,10 +96,11 @@ is the current timing authority when calibration chirplets are playing. Mimir emits that phrase through Aquarium audio, compares mic buffers against loopback by chirplet-energy delay estimation, and can build a provisional aligned mono frame for channels that clear the confidence gate. Reports now -carry fractional delay and per-band matched energy. Camera mics are +carry fractional delay and per-band matched energy. The hub also owns smoothed +per-source sync state with delay-slope/SRO in ppm. Camera mics are spatial/context witnesses; Focusrite devices are dialogue anchors. The current -aligned frame is integer-delay only; SRO smoothing, fractional delay, and the -hot resampler still belong in Faust/native DSP. +aligned frame is integer-delay only; fractional delay and the hot resampler +still belong in Faust/native DSP. ## Visual Fusion diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 4eb114b..1b09c77 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -66,7 +66,8 @@ Get-Content .\state\evidence.jsonl -Tail 8 authority. `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, the Aquarium PCM rendering, matched timing trace, and per-band response hooks. Sync reports now include - fractional delay and per-band matched energy. PS3 Eye audio is + fractional delay and per-band matched energy. `MimirAudioSynchronizationState` + now tracks smoothed delay and delay-slope/SRO ppm. PS3 Eye audio is enumeration/runtime-fragile: one run saw both mic buffers empty while both cameras were live, then a replug made both PS3 Eye mic buffers emit 480-frame WASAPI blocks again. diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 5753a66..4561ef7 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -14,9 +14,16 @@ foreach (var source in configuration.Value.Sources) } var deadline = DateTimeOffset.UtcNow + duration; +var nextSyncPoll = DateTimeOffset.UtcNow + TimeSpan.FromSeconds(1); while (DateTimeOffset.UtcNow < deadline) { hub.PollSources(maxSamplesPerSource: 4096); + if (DateTimeOffset.UtcNow >= nextSyncPoll) + { + hub.AnalyzeAudioSynchronization(syncReference); + nextSyncPoll += TimeSpan.FromSeconds(1); + } + await Task.Delay(pollDelay).ConfigureAwait(false); } @@ -55,6 +62,12 @@ foreach (var report in reports.OrderBy(report => report.SourceId, StringComparer $"sync {report.ReferenceSourceId}->{report.SourceId}: delaySamples={report.DelaySamples} fractionalDelaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} bands={DescribeBands(report.BandResponses)} compared={report.ComparedSamples}"); } +foreach (var state in hub.AudioSynchronizationStates) +{ + Console.WriteLine( + $"sync-state {state.ReferenceSourceId}->{state.SourceId}: smoothedDelaySamples={state.SmoothedDelaySamples:0.000} delayMs={state.DelayMilliseconds:0.000} sroPpm={state.SamplingRateOffsetPpm:0.000} confidence={state.Confidence:0.000} bands={DescribeBands(state.BandResponses)}"); +} + var aligned = hub.BuildAlignedAudioFrame(syncReference); if (aligned != null) { diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 84857ef..2a2686b 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -117,6 +117,7 @@ public sealed class MimirRuntime : IAquariumRuntime panel.Readout("Ingested", () => $"{synchronization.IngestedSamples}"); panel.Readout("Buffer details", DescribeBuffers); panel.Readout("Audio sync", DescribeAudioSync); + panel.Readout("Audio sync state", DescribeAudioSyncState); panel.Readout("Aligned audio", DescribeAlignedAudio); panel.Readout("Chirplet reference", () => $"{DefaultAudioSyncReference} every {CalibrationPhrase.IntervalSeconds:0.0}s"); }); @@ -171,4 +172,12 @@ public sealed class MimirRuntime : IAquariumRuntime ? "no aligned frame" : $"{frame.Channels.Count}ch {frame.SampleRate}Hz {frame.FrameCount} frames"; } + + private string DescribeAudioSyncState() + { + var states = synchronization.AudioSynchronizationStates; + return states.Count == 0 + ? "no sync state" + : string.Join(" | ", states.Select(state => $"{state.SourceId}: {state.SmoothedDelaySamples:0.0} samples sro={state.SamplingRateOffsetPpm:0.0}ppm c={state.Confidence:0.00}")); + } } diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index d952f41..eb5735d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -87,6 +87,7 @@ public sealed class MimirAudioSynchronizationAnalyzer fractionalDelaySamples * 1000.0 / referenceBlock.SampleRate, confidence, bandResponses, + commonEndNs, compared, reference.Latest?.Sequence ?? 0, buffer.Latest?.Sequence ?? 0)); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs index 18df0cc..9d6c1f5 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs @@ -9,6 +9,7 @@ public sealed record MimirAudioSynchronizationReport( double DelayMilliseconds, double Confidence, IReadOnlyList BandResponses, + long AnalysisTimestampNs, int ComparedSamples, ulong ReferenceSequence, ulong SourceSequence); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationState.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationState.cs new file mode 100644 index 0000000..bc96321 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationState.cs @@ -0,0 +1,89 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirAudioSynchronizationState( + string ReferenceSourceId, + string SourceId, + int SampleRate, + double LatestDelaySamples, + double SmoothedDelaySamples, + double DelayMilliseconds, + double SamplingRateOffsetPpm, + double Confidence, + IReadOnlyList BandResponses, + long UpdatedAtNs, + ulong ReferenceSequence, + ulong SourceSequence); + +public sealed class MimirAudioSynchronizationStateTracker +{ + private const double MinimumUpdateConfidence = 0.05; + private readonly Dictionary states = new(StringComparer.Ordinal); + + public IReadOnlyList States => + states.Values.OrderBy(state => state.SourceId, StringComparer.Ordinal).ToArray(); + + public void Update(IEnumerable reports) + { + foreach (var report in reports) + { + if (report.Confidence < MinimumUpdateConfidence) + { + continue; + } + + if (!states.TryGetValue(report.SourceId, out var previous)) + { + states[report.SourceId] = NewState(report, report.FractionalDelaySamples, 0.0); + continue; + } + + if (previous.ReferenceSequence == report.ReferenceSequence && + previous.SourceSequence == report.SourceSequence) + { + continue; + } + + var alpha = Math.Clamp(report.Confidence * 0.35, 0.04, 0.30); + var smoothedDelay = previous.SmoothedDelaySamples + + (report.FractionalDelaySamples - previous.SmoothedDelaySamples) * alpha; + var measuredSro = EstimateSroPpm(previous, smoothedDelay, report); + var sroAlpha = Math.Clamp(report.Confidence * 0.20, 0.02, 0.18); + var smoothedSro = previous.SamplingRateOffsetPpm + + (measuredSro - previous.SamplingRateOffsetPpm) * sroAlpha; + states[report.SourceId] = NewState(report, smoothedDelay, smoothedSro); + } + } + + private static MimirAudioSynchronizationState NewState( + MimirAudioSynchronizationReport report, + double smoothedDelaySamples, + double samplingRateOffsetPpm) => + new( + report.ReferenceSourceId, + report.SourceId, + report.SampleRate, + report.FractionalDelaySamples, + smoothedDelaySamples, + smoothedDelaySamples * 1000.0 / report.SampleRate, + samplingRateOffsetPpm, + report.Confidence, + report.BandResponses, + report.AnalysisTimestampNs, + report.ReferenceSequence, + report.SourceSequence); + + private static double EstimateSroPpm( + MimirAudioSynchronizationState previous, + double smoothedDelaySamples, + MimirAudioSynchronizationReport report) + { + var dtSeconds = (report.AnalysisTimestampNs - previous.UpdatedAtNs) / 1_000_000_000.0; + if (dtSeconds <= 0.050) + { + return previous.SamplingRateOffsetPpm; + } + + var delayDeltaSamples = smoothedDelaySamples - previous.SmoothedDelaySamples; + return delayDeltaSamples / (dtSeconds * report.SampleRate) * 1_000_000.0; + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs index 7fdd18f..3136be2 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs @@ -4,6 +4,7 @@ public sealed class MimirSynchronizationHub : IDisposable { private readonly List sources = []; private readonly MimirAudioSynchronizationAnalyzer audioSynchronization = new(); + private readonly MimirAudioSynchronizationStateTracker audioSynchronizationState = new(); private ulong ingestedSamples; public MimirSynchronizationHub(MimirSynchronizationSettings settings) @@ -24,6 +25,9 @@ public sealed class MimirSynchronizationHub : IDisposable public int SourceCount => sources.Count; + public IReadOnlyList AudioSynchronizationStates => + audioSynchronizationState.States; + public void AddSource(IMimirStreamSource source) { ObjectDisposedException.ThrowIf(disposed, this); @@ -61,7 +65,9 @@ public sealed class MimirSynchronizationHub : IDisposable public IReadOnlyList AnalyzeAudioSynchronization(string referenceSourceId) { ObjectDisposedException.ThrowIf(disposed, this); - return audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId); + var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId); + audioSynchronizationState.Update(reports); + return reports; } public MimirAlignedAudioFrame? BuildAlignedAudioFrame( diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 95d4e6a..3a7b6ee 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -35,3 +35,4 @@ {"ts":"2026-05-22T11:58:15+01:00","type":"verification","status":"ok","note":"Delay signs were suspect because MimirAudioSynchronizationAnalyzer compared latest sample tails from buffers with different edge timestamps. The analyzer now estimates each pair at a shared timestamp edge and BuildAlignedAudioFrame chooses an output edge all positive-late channels can supply. Controlled 9-16 kHz chirplet smoke after the fix reported positive loopback-relative PS3 Eye delay: mic-usb-camera-1 delaySamples=704, delayMs=15.964, confidence=0.166, and the aligned frame contained loopback plus mic-usb-camera-1."} {"ts":"2026-05-22T12:38:53+01:00","type":"implementation","status":"ok","note":"Documented the chirplet calibration model and started the structured implementation. MimirChirpletCalibrationPhrase now owns a harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, PCM rendering for Aquarium emission, matched chirplet timing traces, and per-band matched-energy response hooks. MimirRuntime emits that shared phrase instead of a separate hard-coded sweep, and MimirAudioSynchronizationAnalyzer uses the same phrase for loopback-referenced delay analysis with 16-sample hops and parabolic fractional peak interpolation. BufferSmoke output now includes fractionalDelaySamples and per-band energy."} {"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."} +{"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."} diff --git a/state/map.yaml b/state/map.yaml index 6af4e2d..8c28235 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -73,9 +73,12 @@ accepted_design: emitted chirplet phrase, and estimates current delay from chirplet-energy traces at a shared timestamp edge. Positive delay means the candidate mic is late relative to loopback. Reports include rounded delay, fractional delay, - confidence, and per-band matched energy. `MimirSynchronizationHub.BuildAlignedAudioFrame` - exposes the first provisional loopback-referenced aligned mono frame for - channels that clear the confidence gate. + confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker` + owns the smoothed per-source belief: latest delay, smoothed delay, + confidence, per-band response evidence, and delay-slope/SRO estimate in + ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first + provisional loopback-referenced aligned mono frame for channels that clear + the confidence gate. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared texture capture.