From da9915b6e580c8aa5e8f8a56a684694931c9ccf4 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 22 May 2026 13:15:38 +0100 Subject: [PATCH] Test live runtime chirplet sync --- docs/audio-field.md | 11 +++- docs/implementation-plan.md | 8 ++- notes/current-system-map.md | 7 ++- notes/fresh-workspace-handoff.md | 18 +++--- src/Mimir.Runtime/MimirRuntime.cs | 63 +++++++++++++++++++ .../MimirChirpletCalibrationPhrase.cs | 16 ++--- state/evidence.jsonl | 1 + state/map.yaml | 25 ++++---- 8 files changed, 116 insertions(+), 33 deletions(-) diff --git a/docs/audio-field.md b/docs/audio-field.md index 325b32b..fa020df 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -53,8 +53,8 @@ audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration phrase and the matched-filter shape used to analyze it. The default timeline is a stateless 16-phrase cycle. Phrase `N` is generated directly from `N`, with no registry or remembered random state. Each phrase is a spread-out asymmetric -motif: six short chirplets over about 1.38 seconds, with nonuniform gaps and -high-frequency bands. A new phrase fires every 3.25 seconds through Aquarium +motif: six short chirplets over about 0.85 seconds, with nonuniform gaps and +high-frequency bands. A new phrase fires every 2.25 seconds through Aquarium audio. The point is not ornament. The timing code is carried by both frequency and rhythm, so it behaves more like a small birdsong signature than a repeated sweep. The indexed phrase sequence has lower ambiguity when remote feeds add @@ -91,6 +91,13 @@ for the coming actuator. The state can survive a brief weak report, but it is not a license to run blind: loopback must keep receiving the emitted phrase or fresh reports will stop. +The actual Mimir app path now runs this online: `MimirRuntime.Update` emits the +phrase sequence, polls sources, and updates sync analysis on a fixed cadence. +`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current +runtime testing proves Aquarium output wakes the Scarlett loopback and the mic +buffers stay live, but the app has not yet produced a confident acoustic lock. +That next failure is calibration, not plumbing. + ## Chirplet Calibration Model ```mermaid diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 37484e2..7a23fa1 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -45,8 +45,8 @@ a named invariant that the native runtime cannot protect yet. - `MimirChirpletCalibrationPhrase` owns the structured birdsong-like calibration phrase, PCM rendering, matched timing trace, and per-band response kernels. The default timeline is a stateless 16-phrase cycle; phrase `N` is generated - from `N`, spreads six chirplets across about 1.38 seconds, and fires every - 3.25 seconds to reduce repeated-sweep ambiguity. + from `N`, spreads six chirplets across about 0.85 seconds, and fires every + 2.25 seconds to reduce repeated-sweep ambiguity. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: sample-bearing audio blocks are resampled into the Scarlett loopback timeline and compared by chirplet-energy delay estimation with fractional peak @@ -57,6 +57,8 @@ a named invariant that the native runtime cannot protect yet. - `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source sync state: latest fractional delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in ppm. +- `MimirRuntime` updates audio sync analysis online and can emit live sync + telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. - `MimirRuntime` emits short calibration chirplets through Aquarium audio so the loopback and acoustic mic buffers carry a shared timing witness. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device @@ -96,6 +98,8 @@ a named invariant that the native runtime cannot protect yet. fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. The current aligned frame is integer-delay only and should not pretend to be the final DSP path. + First, make the live app acoustic lock robust enough to populate sync state + from the online chirplet timeline, not just from smoke harnesses. 5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 6. Move GPU feature extraction, fusion, material fitting, render budgeting, and diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 21b45c0..f9aecb2 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -93,12 +93,15 @@ The six-microphone path is separate from the bridge. Scarlett speaker loopback is the current timing authority when calibration chirplets are playing. `MimirChirpletCalibrationPhrase` owns the birdsong-like timeline fingerprint: a stateless 16-phrase cycle where phrase `N` is generated from `N`; each phrase -spreads six asymmetric chirplets across about 1.38 seconds and fires every 3.25 +spreads six asymmetric chirplets across about 0.85 seconds and fires every 2.25 seconds. Mimir emits that phrase sequence through Aquarium audio, compares mic buffers against loopback by chirplet-energy delay estimation, and can build a provisional aligned mono frame for channels that clear the confidence gate. Reports now carry fractional delay and per-band matched energy. The hub also -owns smoothed per-source sync state with delay-slope/SRO in ppm. Camera mics are +owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` +runs the analysis online and can print live telemetry with +`MIMIR_SYNC_TELEMETRY_SECONDS`. Current app testing proves loopback wakeup and +live mic buffers, but not yet a confident acoustic lock. Camera mics are spatial/context witnesses; Focusrite devices are dialogue anchors. The current aligned frame is integer-delay only; fractional delay and the hot resampler still belong in Faust/native DSP. diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index fe0d215..3e7e6d4 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -65,18 +65,20 @@ Get-Content .\state\evidence.jsonl -Tail 8 - Current chirplet-backed smoke uses Scarlett speaker loopback as timing authority. `MimirChirpletCalibrationPhrase` owns the structured birdsong-like timeline fingerprint: a stateless 16-phrase cycle where phrase `N` is - generated from `N`; each phrase has six asymmetric chirplets over about 1.38 - seconds and fires every 3.25 seconds. The phrase owns Aquarium PCM rendering, + generated from `N`; each phrase has six asymmetric chirplets over about 0.85 + seconds and fires every 2.25 seconds. The phrase owns Aquarium PCM rendering, matched timing trace, and per-band response hooks. Sync reports now include - fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and - delay-slope/SRO ppm. PS3 Eye audio is + fractional delay and per-band matched energy. `MimirAudioSynchronizationState` + now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing + proves Aquarium audio wakes Scarlett loopback and keeps mic buffers live, but + the online acoustic report has not cleared the confidence gate yet. PS3 Eye audio is enumeration/runtime-fragile: one run saw both mic buffers empty while both cameras were live, then a replug made both PS3 Eye mic buffers emit 480-frame WASAPI blocks again. ## Immediate Re-entry Instruction -Replace the diagnostic audio JSON process bridge with native audio capture -workers, then turn chirplet delay reports into a smoothed SRO/fractional-delay -actuator. Keep loopback as timing authority. Do not restore deleted script -infrastructure because a stale doc once missed it. +Make the actual Mimir.App path produce a confident online acoustic lock from the +chirplet timeline, then turn those delay reports into a smoothed +SRO/fractional-delay actuator. Keep loopback as timing authority. Do not restore +deleted script infrastructure because a stale doc once missed it. diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 5702fb7..b31beb9 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -11,14 +11,19 @@ namespace Mimir.Runtime; public sealed class MimirRuntime : IAquariumRuntime { private const string DefaultAudioSyncReference = "loopback-scarlett-speakers"; + private const float AudioSyncUpdateIntervalSeconds = 0.5f; private readonly LocalCastRuntime visualRuntime; private readonly MimirSynchronizationHub synchronization; private readonly AquariumUiDocument ui; + private readonly float telemetryIntervalSeconds; private int lastPollCount; private float runtimeSeconds; + private float nextAudioSyncSeconds = AudioSyncUpdateIntervalSeconds; + private float nextTelemetrySeconds; private float nextCalibrationChirpSeconds = (float)MimirChirpletCalibrationPhrase.Default.FirstFireSeconds; private ulong calibrationPhraseIndex; private readonly Dictionary calibrationPhraseCache = new(); + private IReadOnlyList lastAudioSynchronizationReports = []; public MimirRuntime(AquariumRuntimeOptions options) : this(options, MimirRuntimeConfiguration.Load()) @@ -43,6 +48,8 @@ public sealed class MimirRuntime : IAquariumRuntime Options = options; visualRuntime = new LocalCastRuntime(options); synchronization = new MimirSynchronizationHub(settings); + telemetryIntervalSeconds = ParseTelemetryIntervalSeconds(); + nextTelemetrySeconds = telemetryIntervalSeconds; foreach (var source in streamSources) { synchronization.AddSource(source); @@ -85,6 +92,8 @@ public sealed class MimirRuntime : IAquariumRuntime runtimeSeconds += Math.Max(deltaSeconds, 0.0f); QueueCalibrationChirps(); lastPollCount = synchronization.PollSources(); + UpdateAudioSynchronization(); + EmitTelemetry(); visualRuntime.Update(deltaSeconds, input); } @@ -161,6 +170,60 @@ public sealed class MimirRuntime : IAquariumRuntime return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer)); } + private void UpdateAudioSynchronization() + { + if (runtimeSeconds < nextAudioSyncSeconds) + { + return; + } + + lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference); + nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds; + } + + private void EmitTelemetry() + { + if (telemetryIntervalSeconds <= 0.0f || runtimeSeconds < nextTelemetrySeconds) + { + return; + } + + var loopback = synchronization.Buffers.Buffers.FirstOrDefault(buffer => + string.Equals(buffer.Descriptor.SourceId, DefaultAudioSyncReference, StringComparison.Ordinal)); + var states = synchronization.AudioSynchronizationStates; + Console.WriteLine( + $"mimir-sync-telemetry t={runtimeSeconds:0.00}s loopbackCount={loopback?.Count ?? 0} loopbackEdgeNs={loopback?.EdgeNs ?? 0} reports={lastAudioSynchronizationReports.Count} states={states.Count} aligned={DescribeAlignedAudio()}"); + Console.WriteLine($"mimir-sync-buffers {DescribeAudioBuffers()}"); + foreach (var report in lastAudioSynchronizationReports.OrderBy(report => report.SourceId, StringComparer.Ordinal)) + { + Console.WriteLine( + $"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000}"); + } + + foreach (var state in states) + { + Console.WriteLine( + $"mimir-sync-state {state.ReferenceSourceId}->{state.SourceId} delaySamples={state.SmoothedDelaySamples:0.000} delayMs={state.DelayMilliseconds:0.000} sroPpm={state.SamplingRateOffsetPpm:0.000} confidence={state.Confidence:0.000}"); + } + + nextTelemetrySeconds += telemetryIntervalSeconds; + } + + private static float ParseTelemetryIntervalSeconds() + { + return float.TryParse(Environment.GetEnvironmentVariable("MIMIR_SYNC_TELEMETRY_SECONDS"), out var seconds) + ? Math.Clamp(seconds, 0.0f, 60.0f) + : 0.0f; + } + + private string DescribeAudioBuffers() + { + return string.Join(" | ", synchronization.Buffers.Buffers + .Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio) + .OrderBy(buffer => buffer.Descriptor.SourceId, StringComparer.Ordinal) + .Select(buffer => $"{buffer.Descriptor.SourceId}:{buffer.Count}@{buffer.EdgeNs}")); + } + private static string DescribeBuffer(MimirRollingStreamBuffer buffer) { var latest = buffer.Latest; diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs index af6af7a..b65fed5 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs @@ -48,7 +48,7 @@ public sealed class MimirChirpletCalibrationPhrase public static MimirChirpletCalibrationPhrase ForIndex(ulong phraseIndex) => new( sampleRate: 48_000, - intervalSeconds: 3.25, + intervalSeconds: 2.25, firstFireSeconds: 0.75, gain: 0.13, tones: GenerateTones(phraseIndex)); @@ -161,12 +161,12 @@ public sealed class MimirChirpletCalibrationPhrase { var baseTones = new[] { - new MimirChirpletTone(0.000, 0.070, 7_040.0, 7_680.0, 0.80), - new MimirChirpletTone(0.190, 0.060, 9_600.0, 8_800.0, 0.75), - new MimirChirpletTone(0.410, 0.080, 11_800.0, 12_800.0, 0.90), - new MimirChirpletTone(0.735, 0.065, 8_200.0, 9_000.0, 0.60), - new MimirChirpletTone(0.980, 0.095, 14_500.0, 15_800.0, 0.80), - new MimirChirpletTone(1.310, 0.070, 10_800.0, 10_200.0, 0.70), + new MimirChirpletTone(0.000, 0.060, 7_040.0, 7_680.0, 0.80), + new MimirChirpletTone(0.125, 0.055, 9_600.0, 8_800.0, 0.75), + new MimirChirpletTone(0.260, 0.065, 11_800.0, 12_800.0, 0.90), + new MimirChirpletTone(0.435, 0.055, 8_200.0, 9_000.0, 0.60), + new MimirChirpletTone(0.610, 0.075, 14_500.0, 15_800.0, 0.80), + new MimirChirpletTone(0.790, 0.060, 10_800.0, 10_200.0, 0.70), }; var tones = new List(baseTones.Length); var previousEnd = -1.0; @@ -174,7 +174,7 @@ public sealed class MimirChirpletCalibrationPhrase { var baseTone = baseTones[index]; var seed = Mix(phraseIndex ^ ((ulong)index * 0x9E3779B97F4A7C15UL)); - var timeJitter = Unit(seed) * 0.050 - 0.025; + var timeJitter = Unit(seed) * 0.030 - 0.015; var durationJitter = Unit(seed >> 11) * 0.020 - 0.010; var bandJitter = Unit(seed >> 22) * 480.0 - 240.0; var glideScale = 0.88 + Unit(seed >> 33) * 0.24; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 69ec769..ef9d325 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -37,3 +37,4 @@ {"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."} {"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."} {"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."} +{"ts":"2026-05-22T13:12:32+01:00","type":"verification","status":"partial","note":"Actual Mimir.App runtime test with MIMIR_RUNTIME_CONFIG=config/mimir-runtime.probes.example.json and MIMIR_SYNC_TELEMETRY_SECONDS enabled proved Aquarium WASAPI output wakes the Scarlett loopback in the real app path. Headless run showed loopbackCount rising to 57 with mic buffers live; visible-window run showed loopbackCount 77 with Focusrite/Kiyo/Kiyo Pro/PS3 Eye mic buffers live. The runtime now updates sync analysis on a 0.5s cadence and telemetry prints loopback, reports, states, and audio buffer counts. No confident acoustic sync state locked in these app runs: reports were absent until enough phrase audio accumulated, then a live report appeared with confidence 0.000. Treat this as online emission/capture proven, online acoustic lock not yet solved."} diff --git a/state/map.yaml b/state/map.yaml index f8f5b4c..f64f7ff 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -77,7 +77,9 @@ accepted_design: fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker` owns the smoothed per-source belief: latest delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in - ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first + ppm. `MimirRuntime` now runs this analysis online on a fixed cadence and + `MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry. + `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional loopback-referenced aligned mono frame for channels that clear the confidence gate. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live @@ -176,16 +178,17 @@ current_status: Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > - Add the synchronization actuator after the current chirplet measurement - loop: smooth delay/SRO estimates, then drive variable-rate resampling and - fractional-delay correction per non-reference stream. Replace the - diagnostic JSON-line process bridge with direct native capture drivers that - push buffer handles into Mimir.Runtime. Current chirplet-backed - Mimir.BufferSmoke proves loopback-referenced aligned audio for Scarlett - loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when 9-16 kHz - calibration chirplets are playing. After a later replug, both PS3 Eye audio - endpoints emitted 480-frame WASAPI blocks into runtime buffers again while - both PS3 Eye camera buffers were live. Treat PS3 Eye audio as + Make the actual Mimir.App path produce a confident online acoustic lock from + the chirplet timeline before adding the synchronization actuator. Current + app testing proves Aquarium WASAPI output wakes Scarlett speaker loopback, + live mic buffers stay populated, and runtime sync analysis runs on cadence, + but the live acoustic report has not cleared the confidence gate. Once that + lock is real, drive variable-rate resampling and fractional-delay correction + per non-reference stream. Replace the diagnostic JSON-line process bridge + with direct native capture drivers that push buffer handles into + Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned + audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when + 9-16 kHz calibration chirplets are playing. Treat PS3 Eye audio as enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from the Leap USB neighborhood and treat Kiyo Pro as a ~25 fps source until its SuperSpeed negotiation is fixed.