From 7f804ad5da8c1c324b968164acdc20e0dbebcd4d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 22 May 2026 12:54:56 +0100 Subject: [PATCH] Generate stateless chirplet timeline fingerprints --- docs/audio-field.md | 28 +++-- docs/implementation-plan.md | 5 +- notes/current-system-map.md | 15 +-- notes/fresh-workspace-handoff.md | 10 +- src/Mimir.Runtime/MimirRuntime.cs | 34 ++++-- .../MimirAudioSynchronizationAnalyzer.cs | 9 +- .../MimirChirpletCalibrationPhrase.cs | 100 ++++++++++++++++-- state/evidence.jsonl | 1 + state/map.yaml | 17 +-- 9 files changed, 169 insertions(+), 50 deletions(-) diff --git a/docs/audio-field.md b/docs/audio-field.md index 2eb6b8e..325b32b 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -50,17 +50,23 @@ produced zero WASAPI packets until that Eye was unplugged and replugged. The full probe runtime config now enables sample-bearing blocks for every local audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration -phrase and the matched-filter shape used to analyze it. The default phrase is a -short harmonic-ish pattern around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, repeated -every 1.5 seconds through Aquarium audio. That makes the telemetry less like a -single sterile squeal and more like a small, identifiable timing signature. +phrase and the matched-filter shape used to analyze it. The default timeline is +a stateless 16-phrase cycle. Phrase `N` is generated directly from `N`, with no +registry or remembered random state. Each phrase is a spread-out asymmetric +motif: six short chirplets over about 1.38 seconds, with nonuniform gaps and +high-frequency bands. A new phrase fires every 3.25 seconds through Aquarium +audio. The point is not ornament. The timing code is carried by both frequency +and rhythm, so it behaves more like a small birdsong signature than a repeated +sweep. The indexed phrase sequence has lower ambiguity when remote feeds add +network/encoding latency. `MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the loopback sample-rate timeline, projects loopback and candidate windows through the same phrase, contrast-normalizes the resulting energy traces, and estimates -current delay against `loopback-scarlett-speakers`. Reports carry both rounded -integer delay and fractional delay from parabolic peak interpolation over the -chirplet correlation peak. +current delay against `loopback-scarlett-speakers`. The lag search is wide +enough for ordinary remote/network delay, not just local speaker-to-mic delay. +Reports carry both rounded integer delay and fractional delay from parabolic +peak interpolation over the chirplet correlation peak. `MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned mono frame: loopback is always channel zero, and other channels enter only when @@ -116,6 +122,14 @@ watching delay change over time. Per-band energy over many phrases gives the normalization curve. The important constraint is that all three measurements must be tied to the same emitted phrase, not three separately invented probes. +The phrase is no longer static. Mimir currently rotates a deterministic +16-phrase cycle and the analyzer searches that cycle. That is enough for a +first timeline fingerprint and ordinary remote/network delay. For higher-latency +or recorded/replayed sources, the next extension should lengthen the code or +make the phrase index explicitly recoverable from the chirplet rhythm so Mimir +can distinguish phrase N from phrase N+16 without relying only on wall-clock +arrival. + Next, replace the diagnostic bridge with native audio capture workers that append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state, delay estimates, and stem routing in Aquarium UI. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 91cd7a1..37484e2 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -42,8 +42,11 @@ a named invariant that the native runtime cannot protect yet. - `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback block metadata and emits `audio-block` JSON events for the diagnostic runtime adapter. -- `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish calibration +- `MimirChirpletCalibrationPhrase` owns the structured birdsong-like calibration phrase, PCM rendering, matched timing trace, and per-band response kernels. + The default timeline is a stateless 16-phrase cycle; phrase `N` is generated + from `N`, spreads six chirplets across about 1.38 seconds, and fires every + 3.25 seconds to reduce repeated-sweep ambiguity. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: sample-bearing audio blocks are resampled into the Scarlett loopback timeline and compared by chirplet-energy delay estimation with fractional peak diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 5ba15da..21b45c0 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -91,13 +91,14 @@ not become the synchronized program authority. The six-microphone path is separate from the bridge. Scarlett speaker loopback is the current timing authority when calibration chirplets are playing. -`MimirChirpletCalibrationPhrase` owns the harmonic-ish calibration phrase around -8 kHz, 10 kHz, 12 kHz, and 16 kHz, plus the matched timing and response kernels. -Mimir emits that phrase through Aquarium audio, compares mic buffers against -loopback by chirplet-energy delay estimation, and can build a provisional -aligned mono frame for channels that clear the confidence gate. Reports now -carry fractional delay and per-band matched energy. The hub also owns smoothed -per-source sync state with delay-slope/SRO in ppm. Camera mics are +`MimirChirpletCalibrationPhrase` owns the birdsong-like timeline fingerprint: +a stateless 16-phrase cycle where phrase `N` is generated from `N`; each phrase +spreads six asymmetric chirplets across about 1.38 seconds and fires every 3.25 +seconds. Mimir emits that phrase sequence through Aquarium audio, compares mic +buffers against loopback by chirplet-energy delay estimation, and can build a +provisional aligned mono frame for channels that clear the confidence gate. +Reports now carry fractional delay and per-band matched energy. The hub also +owns smoothed per-source sync state with delay-slope/SRO in ppm. Camera mics are spatial/context witnesses; Focusrite devices are dialogue anchors. The current aligned frame is integer-delay only; fractional delay and the hot resampler still belong in Faust/native DSP. diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 1b09c77..fe0d215 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -63,11 +63,13 @@ Get-Content .\state\evidence.jsonl -Tail 8 640x480@60, and 81.10 fps when the PS3 Eye runs 320x240@187. The regular Kiyo and second PS3 Eye were absent, so six-camera viability is not proven. - Current chirplet-backed smoke uses Scarlett speaker loopback as timing - authority. `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish - phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, the Aquarium PCM rendering, + authority. `MimirChirpletCalibrationPhrase` owns the structured birdsong-like + timeline fingerprint: a stateless 16-phrase cycle where phrase `N` is + generated from `N`; each phrase has six asymmetric chirplets over about 1.38 + seconds and fires every 3.25 seconds. The phrase owns Aquarium PCM rendering, matched timing trace, and per-band response hooks. Sync reports now include - fractional delay and per-band matched energy. `MimirAudioSynchronizationState` - now tracks smoothed delay and delay-slope/SRO ppm. PS3 Eye audio is + fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and + delay-slope/SRO ppm. PS3 Eye audio is enumeration/runtime-fragile: one run saw both mic buffers empty while both cameras were live, then a replug made both PS3 Eye mic buffers emit 480-frame WASAPI blocks again. diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 2a2686b..5702fb7 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -11,14 +11,14 @@ namespace Mimir.Runtime; public sealed class MimirRuntime : IAquariumRuntime { private const string DefaultAudioSyncReference = "loopback-scarlett-speakers"; - private static readonly MimirChirpletCalibrationPhrase CalibrationPhrase = MimirChirpletCalibrationPhrase.Default; private readonly LocalCastRuntime visualRuntime; private readonly MimirSynchronizationHub synchronization; private readonly AquariumUiDocument ui; private int lastPollCount; private float runtimeSeconds; - private float nextCalibrationChirpSeconds = (float)CalibrationPhrase.FirstFireSeconds; - private string? calibrationPhrasePcm16Base64; + private float nextCalibrationChirpSeconds = (float)MimirChirpletCalibrationPhrase.Default.FirstFireSeconds; + private ulong calibrationPhraseIndex; + private readonly Dictionary calibrationPhraseCache = new(); public MimirRuntime(AquariumRuntimeOptions options) : this(options, MimirRuntimeConfiguration.Load()) @@ -119,7 +119,7 @@ public sealed class MimirRuntime : IAquariumRuntime panel.Readout("Audio sync", DescribeAudioSync); panel.Readout("Audio sync state", DescribeAudioSyncState); panel.Readout("Aligned audio", DescribeAlignedAudio); - panel.Readout("Chirplet reference", () => $"{DefaultAudioSyncReference} every {CalibrationPhrase.IntervalSeconds:0.0}s"); + panel.Readout("Chirplet reference", DescribeChirpletReference); }); } @@ -127,15 +127,35 @@ public sealed class MimirRuntime : IAquariumRuntime { while (runtimeSeconds >= nextCalibrationChirpSeconds) { + var cycleIndex = calibrationPhraseIndex % MimirChirpletCalibrationPhrase.PhraseCycleLength; + var phrase = MimirChirpletCalibrationPhrase.ForIndex(cycleIndex); visualRuntime.Audio.EnqueuePcm16Base64( - calibrationPhrasePcm16Base64 ??= CalibrationPhrase.RenderPcm16Base64(), - CalibrationPhrase.SampleRate, + Pcm16Base64ForPhrase(cycleIndex, phrase), + phrase.SampleRate, channels: 1, gain: 1.0f); - nextCalibrationChirpSeconds += (float)CalibrationPhrase.IntervalSeconds; + calibrationPhraseIndex++; + nextCalibrationChirpSeconds += (float)phrase.IntervalSeconds; } } + private string Pcm16Base64ForPhrase(ulong cycleIndex, MimirChirpletCalibrationPhrase phrase) + { + if (!calibrationPhraseCache.TryGetValue(cycleIndex, out var pcm16Base64)) + { + pcm16Base64 = phrase.RenderPcm16Base64(); + calibrationPhraseCache[cycleIndex] = pcm16Base64; + } + + return pcm16Base64; + } + + private string DescribeChirpletReference() + { + var phrase = MimirChirpletCalibrationPhrase.Default; + return $"{DefaultAudioSyncReference} every {phrase.IntervalSeconds:0.00}s code {calibrationPhraseIndex % MimirChirpletCalibrationPhrase.PhraseCycleLength}/{MimirChirpletCalibrationPhrase.PhraseCycleLength}"; + } + private string DescribeBuffers() { return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer)); diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index eb5735d..b1bc455 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -5,9 +5,8 @@ namespace Mimir.Runtime.Synchronization; public sealed class MimirAudioSynchronizationAnalyzer { private const int MaxWindowSamples = 48_000 * 5; - private const int MaxLagSamples = 2_400; + private const int MaxLagSamples = 48_000; private const int ChirpletHopSamples = 16; - private readonly MimirChirpletCalibrationPhrase chirpletPhrase = MimirChirpletCalibrationPhrase.Default; public IReadOnlyList Analyze( IEnumerable buffers, @@ -66,8 +65,8 @@ public sealed class MimirAudioSynchronizationAnalyzer var referenceWindow = referenceSamples.AsSpan(^compared..); var candidateWindow = candidateSamples.AsSpan(^compared..); - var referenceSync = chirpletPhrase.BuildEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples); - var candidateSync = chirpletPhrase.BuildEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples); + var referenceSync = MimirChirpletCalibrationPhrase.BuildTimelineEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples); + var candidateSync = MimirChirpletCalibrationPhrase.BuildTimelineEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples); if (referenceSync.Length < 16 || candidateSync.Length < 16) { continue; @@ -77,7 +76,7 @@ public sealed class MimirAudioSynchronizationAnalyzer var (delayHops, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag); var fractionalDelaySamples = delayHops * ChirpletHopSamples; var delaySamples = (int)Math.Round(fractionalDelaySamples); - var bandResponses = chirpletPhrase.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate); + var bandResponses = MimirChirpletCalibrationPhrase.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate); reports.Add(new MimirAudioSynchronizationReport( reference.Descriptor.SourceId, buffer.Descriptor.SourceId, diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs index 9f8e40c..af6af7a 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs @@ -16,6 +16,7 @@ public sealed record MimirChirpletBandResponse( public sealed class MimirChirpletCalibrationPhrase { + public const int PhraseCycleLength = 16; private readonly float[] kernel; private readonly IReadOnlyList toneKernels; @@ -38,18 +39,46 @@ public sealed class MimirChirpletCalibrationPhrase toneKernels = tones.Select(tone => RenderToneKernel(tone, sampleRate, DurationSeconds)).ToArray(); } - public static MimirChirpletCalibrationPhrase Default { get; } = new( + public static MimirChirpletCalibrationPhrase Default { get; } = ForIndex(0); + + public static IReadOnlyList AnalysisCycle { get; } = + Enumerable.Range(0, PhraseCycleLength) + .Select(index => ForIndex((ulong)index)) + .ToArray(); + + public static MimirChirpletCalibrationPhrase ForIndex(ulong phraseIndex) => new( sampleRate: 48_000, - intervalSeconds: 1.5, - firstFireSeconds: 0.5, - gain: 0.125, - tones: - [ - new(0.000, 0.055, 7_680.0, 8_320.0), - new(0.085, 0.055, 9_600.0, 10_400.0), - new(0.170, 0.055, 11_520.0, 12_480.0), - new(0.255, 0.055, 15_360.0, 16_000.0), - ]); + intervalSeconds: 3.25, + firstFireSeconds: 0.75, + gain: 0.13, + tones: GenerateTones(phraseIndex)); + + public static float[] BuildTimelineEnergyTrace(ReadOnlySpan samples, int sampleRate, int hopSamples) + { + var traces = new List(); + foreach (var phrase in AnalysisCycle) + { + var trace = phrase.BuildEnergyTrace(samples, sampleRate, hopSamples); + if (trace.Length > 0) + { + traces.Add(trace); + } + } + + if (traces.Count == 0) + { + return []; + } + + var length = traces.Min(trace => trace.Length); + var output = new float[length]; + for (var index = 0; index < length; index++) + { + output[index] = traces.Max(trace => trace[index]); + } + + return output; + } public int SampleRate { get; } @@ -128,6 +157,55 @@ public sealed class MimirChirpletCalibrationPhrase return samples; } + private static IReadOnlyList GenerateTones(ulong phraseIndex) + { + var baseTones = new[] + { + new MimirChirpletTone(0.000, 0.070, 7_040.0, 7_680.0, 0.80), + new MimirChirpletTone(0.190, 0.060, 9_600.0, 8_800.0, 0.75), + new MimirChirpletTone(0.410, 0.080, 11_800.0, 12_800.0, 0.90), + new MimirChirpletTone(0.735, 0.065, 8_200.0, 9_000.0, 0.60), + new MimirChirpletTone(0.980, 0.095, 14_500.0, 15_800.0, 0.80), + new MimirChirpletTone(1.310, 0.070, 10_800.0, 10_200.0, 0.70), + }; + var tones = new List(baseTones.Length); + var previousEnd = -1.0; + for (var index = 0; index < baseTones.Length; index++) + { + var baseTone = baseTones[index]; + var seed = Mix(phraseIndex ^ ((ulong)index * 0x9E3779B97F4A7C15UL)); + var timeJitter = Unit(seed) * 0.050 - 0.025; + var durationJitter = Unit(seed >> 11) * 0.020 - 0.010; + var bandJitter = Unit(seed >> 22) * 480.0 - 240.0; + var glideScale = 0.88 + Unit(seed >> 33) * 0.24; + var gainScale = 0.85 + Unit(seed >> 44) * 0.30; + var flips = ((seed >> 57) & 1UL) != 0; + var duration = Math.Clamp(baseTone.DurationSeconds + durationJitter, 0.045, 0.115); + var start = Math.Max(baseTone.StartSeconds + timeJitter, previousEnd + 0.060); + var center = Math.Clamp(baseTone.CenterHz + bandJitter, 6_800.0, 16_200.0); + var halfWidth = Math.Abs(baseTone.EndHz - baseTone.StartHz) * 0.5 * glideScale; + var low = Math.Clamp(center - halfWidth, 6_700.0, 16_400.0); + var high = Math.Clamp(center + halfWidth, 6_700.0, 16_400.0); + tones.Add(flips + ? new MimirChirpletTone(start, duration, high, low, baseTone.Gain * gainScale) + : new MimirChirpletTone(start, duration, low, high, baseTone.Gain * gainScale)); + previousEnd = start + duration; + } + + return tones; + } + + private static ulong Mix(ulong value) + { + value += 0x9E3779B97F4A7C15UL; + value = (value ^ (value >> 30)) * 0xBF58476D1CE4E5B9UL; + value = (value ^ (value >> 27)) * 0x94D049BB133111EBUL; + return value ^ (value >> 31); + } + + private static double Unit(ulong value) => + (value & ((1UL << 53) - 1)) / (double)(1UL << 53); + private static float[] RenderToneKernel(MimirChirpletTone tone, int sampleRate, double phraseDurationSeconds) { var frameCount = Math.Max(1, (int)Math.Ceiling(phraseDurationSeconds * sampleRate)); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 3a7b6ee..69ec769 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -36,3 +36,4 @@ {"ts":"2026-05-22T12:38:53+01:00","type":"implementation","status":"ok","note":"Documented the chirplet calibration model and started the structured implementation. MimirChirpletCalibrationPhrase now owns a harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, PCM rendering for Aquarium emission, matched chirplet timing traces, and per-band matched-energy response hooks. MimirRuntime emits that shared phrase instead of a separate hard-coded sweep, and MimirAudioSynchronizationAnalyzer uses the same phrase for loopback-referenced delay analysis with 16-sample hops and parabolic fractional peak interpolation. BufferSmoke output now includes fractionalDelaySamples and per-band energy."} {"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."} {"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."} +{"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."} diff --git a/state/map.yaml b/state/map.yaml index 8c28235..f8f5b4c 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -66,14 +66,15 @@ accepted_design: final local-device hot path. - The diagnostic audio path can optionally carry base64 float sample payloads for sync smoke tests. `MimirChirpletCalibrationPhrase` owns the structured - harmonic-ish calibration phrase, PCM rendering, matched timing kernel, and - per-band response kernels. `MimirAudioSynchronizationAnalyzer` now treats - `loopback-scarlett-speakers` as timing authority, resamples candidate mic - windows into that timeline, projects loopback and mic windows through the - emitted chirplet phrase, and estimates current delay from chirplet-energy - traces at a shared timestamp edge. Positive delay means the candidate mic is - late relative to loopback. Reports include rounded delay, fractional delay, - confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker` + birdsong-like timeline fingerprint: a stateless 16-phrase cycle where + phrase `N` is generated from `N`, plus PCM rendering, matched timing kernel, + and per-band response kernels. `MimirAudioSynchronizationAnalyzer` now + treats `loopback-scarlett-speakers` as timing authority, resamples candidate + mic windows into that timeline, projects loopback and mic windows through + the emitted chirplet phrase cycle, and estimates current delay from + chirplet-energy traces at a shared timestamp edge. Positive delay means the + candidate mic is late relative to loopback. Reports include rounded delay, + fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker` owns the smoothed per-source belief: latest delay, smoothed delay, confidence, per-band response evidence, and delay-slope/SRO estimate in ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first