From 200186d4d57cf2af610673935f2d2e89757b2c7d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 22 May 2026 14:16:50 +0100 Subject: [PATCH] Bound runtime sync analysis --- docs/implementation-plan.md | 10 ++-- notes/current-system-map.md | 16 +++--- notes/fresh-workspace-handoff.md | 13 +++-- src/Mimir.Runtime/MimirRuntime.cs | 15 +++--- .../MimirAudioSynchronizationAnalyzer.cs | 14 ++++- .../MimirSynchronizationHub.cs | 52 ++++++++++++++++++- state/evidence.jsonl | 1 + state/map.yaml | 28 +++++----- 8 files changed, 112 insertions(+), 37 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 96c43c5..f3d14fb 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -60,8 +60,10 @@ a named invariant that the native runtime cannot protect yet. - `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source sync state: latest fractional delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in ppm. -- `MimirRuntime` updates audio sync analysis online and can emit live sync - telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. +- `MimirRuntime` updates audio sync analysis online as a bounded rotating + service and can emit live sync telemetry with + `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry read cached reports/states; + they do not run synchronization analysis. - `MimirRuntime` continuously queues chirplet timeline PCM through Aquarium audio so the loopback and acoustic mic buffers carry a shared timing witness. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device @@ -102,7 +104,9 @@ a named invariant that the native runtime cannot protect yet. `MimirAudioSynchronizationState`. The current aligned frame is integer-delay only and should not pretend to be the final DSP path. First, harden the online state filter so the continuous chirplet lock stops - producing jumpy delay estimates. + producing jumpy delay estimates. The runtime sync cadence is now bounded and + survives longer real app runs, but the state filter still accepts outliers + too readily. 5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 6. Move GPU feature extraction, fusion, material fitting, render budgeting, and diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 853bcc8..e41fdcf 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -100,13 +100,15 @@ Aquarium audio, uses coarse chirplet energy for acquisition, refines delay from matched event indices when the symbol decoder succeeds, and can build a provisional aligned mono frame for channels that clear the confidence gate. Reports now carry fractional delay and per-band matched energy. The hub also -owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` -runs the analysis online and can print live telemetry with -`MIMIR_SYNC_TELEMETRY_SECONDS`. Current app testing proves loopback wakeup, live -mic buffers, and confident online sync states, but delay estimates are still -jumpy. Camera mics are spatial/context witnesses; Focusrite devices are dialogue -anchors. The current aligned frame is integer-delay only; fractional delay and -the hot resampler still belong in Faust/native DSP. +owns cached reports plus smoothed per-source sync state with delay-slope/SRO in +ppm. `MimirRuntime` runs analysis online as a bounded rotating service and can +print live telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are +passive readers of cached sync state; they must not invoke the analyzer. Current +app testing proves loopback wakeup, live mic buffers, and confident online sync +states, but delay estimates are still jumpy. Camera mics are spatial/context +witnesses; Focusrite devices are dialogue anchors. The current aligned frame is +integer-delay only; fractional delay and the hot resampler still belong in +Faust/native DSP. ## Visual Fusion diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 2d82cc5..85970d1 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -71,10 +71,12 @@ Get-Content .\state\evidence.jsonl -Tail 8 timeline owns Aquarium PCM segment rendering, coarse acquisition, symbol/event decoding, and per-band response hooks. Sync reports now include fractional delay and per-band matched energy. `MimirAudioSynchronizationState` - now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing - proves Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and - produces confident online sync states, but delay estimates are still jumpy. - PS3 Eye audio is + now tracks smoothed delay and delay-slope/SRO ppm. `MimirRuntime` runs sync + analysis as a bounded rotating service and caches reports/states for UI and + telemetry; readouts must stay passive. Actual Mimir.App testing proves + Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and produces + confident online sync states, but delay estimates are still jumpy. PS3 Eye + audio is enumeration/runtime-fragile: one run saw both mic buffers empty while both cameras were live, then a replug made both PS3 Eye mic buffers emit 480-frame WASAPI blocks again. @@ -83,4 +85,5 @@ Get-Content .\state\evidence.jsonl -Tail 8 Harden the online sync state filter, then turn those delay reports into a smoothed SRO/fractional-delay actuator. Keep loopback as timing authority. Do -not restore deleted script infrastructure because a stale doc once missed it. +not call synchronization analysis from UI/telemetry readouts. Do not restore +deleted script infrastructure because a stale doc once missed it. diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 987ab80..a1eb5e1 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -11,7 +11,7 @@ namespace Mimir.Runtime; public sealed class MimirRuntime : IAquariumRuntime { private const string DefaultAudioSyncReference = "loopback-scarlett-speakers"; - private const float AudioSyncUpdateIntervalSeconds = 0.5f; + private const float AudioSyncUpdateIntervalSeconds = 0.1f; private readonly LocalCastRuntime visualRuntime; private readonly MimirSynchronizationHub synchronization; private readonly AquariumUiDocument ui; @@ -164,7 +164,8 @@ public sealed class MimirRuntime : IAquariumRuntime return; } - lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds); + synchronization.AnalyzeAudioSynchronizationStep(DefaultAudioSyncReference, runtimeSeconds); + lastAudioSynchronizationReports = synchronization.AudioSynchronizationReports; nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds; } @@ -229,7 +230,7 @@ public sealed class MimirRuntime : IAquariumRuntime private string DescribeAudioSync() { - var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds); + var reports = synchronization.AudioSynchronizationReports; return reports.Count == 0 ? "no payload windows" : string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00} events={report.TimelineMatchedEvents}")); @@ -237,10 +238,10 @@ public sealed class MimirRuntime : IAquariumRuntime private string DescribeAlignedAudio() { - var frame = synchronization.BuildAlignedAudioFrame(DefaultAudioSyncReference); - return frame == null - ? "no aligned frame" - : $"{frame.Channels.Count}ch {frame.SampleRate}Hz {frame.FrameCount} frames"; + var states = synchronization.AudioSynchronizationStates; + return states.Count == 0 + ? "no aligned state" + : $"{states.Count + 1}ch state-ready"; } private string DescribeAudioSyncState() diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index 10028cb..640b3c8 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -11,7 +11,8 @@ public sealed class MimirAudioSynchronizationAnalyzer public IReadOnlyList Analyze( IEnumerable buffers, string referenceSourceId, - double approximateTimelineSeconds) + double approximateTimelineSeconds, + IReadOnlySet? candidateSourceIds = null) { var audioBuffers = buffers .Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio) @@ -47,6 +48,11 @@ public sealed class MimirAudioSynchronizationAnalyzer continue; } + if (candidateSourceIds != null && !candidateSourceIds.Contains(buffer.Descriptor.SourceId)) + { + continue; + } + var commonEndNs = Math.Min(referenceLatest.Value.TimestampNs, buffer.Latest?.TimestampNs ?? 0); var candidateSamples = ExtractMonoWindow(buffer, out var candidateBlock); if (candidateSamples.Length == 0 || candidateBlock == null) @@ -363,7 +369,11 @@ public sealed class MimirAudioSynchronizationAnalyzer return (0.0, 0.0, 0); } - var candidateByEvent = candidateObservations.ToDictionary(observation => observation.EventIndex); + var candidateByEvent = candidateObservations + .GroupBy(observation => observation.EventIndex) + .ToDictionary( + group => group.Key, + group => group.OrderByDescending(observation => observation.Energy).First()); var delays = new List<(double Delay, double Weight)>(); foreach (var reference in referenceObservations) { diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs index 80fb54f..265ad98 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs @@ -5,7 +5,9 @@ public sealed class MimirSynchronizationHub : IDisposable private readonly List sources = []; private readonly MimirAudioSynchronizationAnalyzer audioSynchronization = new(); private readonly MimirAudioSynchronizationStateTracker audioSynchronizationState = new(); + private readonly Dictionary audioSynchronizationReports = new(StringComparer.Ordinal); private ulong ingestedSamples; + private int nextAudioSynchronizationCandidate; public MimirSynchronizationHub(MimirSynchronizationSettings settings) { @@ -28,6 +30,9 @@ public sealed class MimirSynchronizationHub : IDisposable public IReadOnlyList AudioSynchronizationStates => audioSynchronizationState.States; + public IReadOnlyList AudioSynchronizationReports => + audioSynchronizationReports.Values.OrderBy(report => report.SourceId, StringComparer.Ordinal).ToArray(); + public void AddSource(IMimirStreamSource source) { ObjectDisposedException.ThrowIf(disposed, this); @@ -68,7 +73,42 @@ public sealed class MimirSynchronizationHub : IDisposable { ObjectDisposedException.ThrowIf(disposed, this); var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId, approximateTimelineSeconds); - audioSynchronizationState.Update(reports); + StoreAudioSynchronizationReports(reports); + return reports; + } + + public IReadOnlyList AnalyzeAudioSynchronizationStep( + string referenceSourceId, + double approximateTimelineSeconds, + int maxCandidates = 1) + { + ObjectDisposedException.ThrowIf(disposed, this); + var candidates = Buffers.Buffers + .Where(buffer => + buffer.Descriptor.Kind == MimirStreamKind.Audio && + !string.Equals(buffer.Descriptor.SourceId, referenceSourceId, StringComparison.Ordinal) && + buffer.Latest?.AudioBlock != null) + .OrderBy(buffer => buffer.Descriptor.SourceId, StringComparer.Ordinal) + .Select(buffer => buffer.Descriptor.SourceId) + .ToArray(); + if (candidates.Length == 0) + { + return []; + } + + var selected = new HashSet(StringComparer.Ordinal); + for (var count = 0; count < Math.Min(maxCandidates, candidates.Length); count++) + { + selected.Add(candidates[nextAudioSynchronizationCandidate % candidates.Length]); + nextAudioSynchronizationCandidate++; + } + + var reports = audioSynchronization.Analyze( + Buffers.Buffers, + referenceSourceId, + approximateTimelineSeconds, + selected); + StoreAudioSynchronizationReports(reports); return reports; } @@ -87,6 +127,16 @@ public sealed class MimirSynchronizationHub : IDisposable private bool disposed; + private void StoreAudioSynchronizationReports(IReadOnlyList reports) + { + foreach (var report in reports) + { + audioSynchronizationReports[report.SourceId] = report; + } + + audioSynchronizationState.Update(reports); + } + public void Dispose() { if (disposed) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 236a972..80bc4a4 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -42,3 +42,4 @@ {"ts":"2026-05-22T14:10:00+01:00","type":"implementation","status":"partial","note":"Changed MimirChirpletTimeline from anonymous deterministic texture to an explicit order-3 de Bruijn symbol timeline over 32 separated chirp symbols. Any three consecutive correctly detected symbols identify a timeline event within the current operating horizon; reports now expose timelineMatchedEvents and timelineConfidence when event-index refinement is used. A synthetic check rendered two seconds of canonical timeline audio and decoded events 0, 1, and 2 back to a window start of -0.0015 seconds. Real headless app sniff did not provide a clean physical proof because Scarlett loopback capture again stopped advancing early; build remains clean."} {"ts":"2026-05-22T14:32:00+01:00","type":"implementation","status":"ok","note":"Reworked MimirChirpletTimeline symbols from fixed frequency slots into time/frequency constellations. The order-3 de Bruijn sequence still gives three-chirp timeline placement, but each symbol now carries identity through start band, glide shape, duration, and following inter-chirp gap. Event start times are precomputed over the de Bruijn period instead of using a fixed stride. Synthetic verification rendered two seconds of canonical timeline audio and decoded interior events 9, 10, and 11 back to a zero-second window start, proving placement does not rely on the opening triplet."} {"ts":"2026-05-22T14:48:00+01:00","type":"verification","status":"partial","note":"Attempted a 20-second real-world Mimir.App chirplet sync run with all configured local mics. The headless path only emitted one telemetry tick before runtime updates stopped. Switching to the normal app path produced telemetry through t=4.26s with all six audio buffers live and sync states for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Timeline-symbol refinement appeared on Focusrite, Kiyo, and one PS3 Eye, but delay estimates were unstable and the app did not continue telemetry to 20 seconds. Replaced the coarse acquisition trace with envelope energy to avoid matched-filter scans in the hot acquisition path; symbol decoding remains the refinement step."} +{"ts":"2026-05-22T15:20:00+01:00","type":"implementation","status":"partial","note":"Removed hidden re-entrant synchronization analysis from the live app path. MimirRuntime now calls a bounded rotating AnalyzeAudioSynchronizationStep every 0.1s, MimirSynchronizationHub caches per-source reports/states, and UI/telemetry read cached state instead of invoking the analyzer or BuildAlignedAudioFrame. FitDelayFromEvents now collapses duplicate candidate event observations by strongest energy, fixing the real app crash 'An item with the same key has already been added. Key: 0'. Dotnet build passed. A visible Mimir.App run with all configured mics no longer crashed at the earlier 4s wall and produced telemetry through runtime t=17.28s in a 22s wall-time harness; it did not reach a full 20 runtime seconds because startup consumed wall time. All non-loopback mics produced cached reports/states, but delay estimates remained unstable and timeline event refinement was intermittent, so the next live invariant is robust outlier rejection/state filtering before any actuator drives resampling."} diff --git a/state/map.yaml b/state/map.yaml index 1ebd3a4..92c1101 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -71,16 +71,19 @@ accepted_design: timing atoms, symbol/event decoding, and per-band response kernels. Symbol identity is carried by start band, glide shape, duration, and following inter-chirp gap. Any three consecutive correctly detected symbols identify a - timeline event within the current operating horizon. `MimirAudioSynchronizationAnalyzer` now treats - `loopback-scarlett-speakers` as timing authority, uses coarse chirplet - energy for acquisition, and refines delay from matched timeline event - indices when symbol decoding succeeds. Positive delay means the + timeline event within the current operating horizon. `MimirAudioSynchronizationAnalyzer` + now treats `loopback-scarlett-speakers` as timing authority, uses coarse + chirplet energy for acquisition, and refines delay from matched timeline + event indices when symbol decoding succeeds. Positive delay means the candidate mic is late relative to loopback. Reports include rounded delay, - fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker` + fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub` + caches the latest per-source reports and `MimirAudioSynchronizationStateTracker` owns the smoothed per-source belief: latest delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in - ppm. `MimirRuntime` now runs this analysis online on a fixed cadence and - `MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry. + ppm. `MimirRuntime` now runs this analysis online as a bounded rotating + service on a fixed cadence, while UI and telemetry read cached reports/states + instead of invoking analysis. `MIMIR_SYNC_TELEMETRY_SECONDS` prints live + buffer/report/state telemetry. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional loopback-referenced aligned mono frame for channels that clear the confidence gate. @@ -182,11 +185,12 @@ current_status: next_action: > Harden the online state filter before adding the synchronization actuator. Current app testing proves Aquarium WASAPI output wakes Scarlett speaker - loopback, live mic buffers stay populated, runtime sync analysis runs on - cadence, and the continuous chirplet timeline can produce confident sync - states in the app. Delay estimates are still jumpy, so the actuator needs - outlier rejection and better SRO smoothing before it drives variable-rate - resampling and fractional-delay correction per non-reference stream. Replace + loopback, live mic buffers stay populated, runtime sync analysis now runs as + a bounded rotating service without UI/telemetry re-entry, and the continuous + chirplet timeline can produce confident sync states in the app. Delay + estimates are still jumpy, so the actuator needs outlier rejection and better + SRO smoothing before it drives variable-rate resampling and fractional-delay + correction per non-reference stream. Replace the diagnostic JSON-line process bridge with direct native capture drivers that push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when