diff --git a/docs/audio-field.md b/docs/audio-field.md index 6a55e62..a059575 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -49,19 +49,64 @@ loopback audio is actively playing. One PS3 Eye mic previously enumerated but produced zero WASAPI packets until that Eye was unplugged and replugged. The full probe runtime config now enables sample-bearing blocks for every local -audio source. `MimirRuntime` emits short 9-16 kHz calibration chirplets through -Aquarium audio every 1.5 seconds. `MimirAudioSynchronizationAnalyzer` resamples -candidate mic windows into the loopback sample-rate timeline, projects them -through the same chirplet shape, contrast-normalizes the resulting energy -trace, and estimates current delay against `loopback-scarlett-speakers`. +audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration +phrase and the matched-filter shape used to analyze it. The default phrase is a +short harmonic-ish pattern around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, repeated +every 1.5 seconds through Aquarium audio. That makes the telemetry less like a +single sterile squeal and more like a small, identifiable timing signature. + +`MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the +loopback sample-rate timeline, projects loopback and candidate windows through +the same phrase, contrast-normalizes the resulting energy traces, and estimates +current delay against `loopback-scarlett-speakers`. Reports carry both rounded +integer delay and fractional delay from parabolic peak interpolation over the +chirplet correlation peak. `MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned mono frame: loopback is always channel zero, and other channels enter only when their chirplet confidence clears the gate. Delay estimates compare loopback and candidate mic windows at a shared timestamp edge; positive delay means the -candidate mic is late relative to loopback. This is still integer-delay -alignment at 64-sample chirplet-hop granularity. SRO smoothing, fractional-delay -correction, and the hot resampler are the actuator still ahead of us. +candidate mic is late relative to loopback. The current chirplet trace uses a +16-sample hop and parabolic peak interpolation. The aligned-frame application +still rounds to integer samples; the fractional estimate exists so the next +actuator can drive a real fractional-delay line. + +The same phrase also starts the frequency-response path. Each report includes +per-band matched energy for the phrase tones. That is not a finished room/mic +normalizer yet, but it is the live surface that will become response-curve +estimation: loopback carries what was emitted, each mic carries what survived +speaker, air, room, and capsule, and the ratio over repeated chirplet phrases +becomes gain/phase correction evidence. + +## Chirplet Calibration Model + +```mermaid +flowchart TD + A["MimirChirpletCalibrationPhrase"] --> B["Aquarium audio output"] + B --> C["Scarlett speaker loopback"] + B --> D["room + speakers + mics"] + C --> E["loopback rolling buffer"] + D --> F["mic rolling buffers"] + E --> G["matched chirplet traces"] + F --> G + G --> H["delay + SRO estimates"] + G --> I["per-band response estimates"] + H --> J["fractional delay / resampler actuator"] + I --> K["frequency response normalization"] +``` + +The calibration phrase owns three facts: + +- **Emission**: the PCM that Aquarium sends to the speakers. +- **Timing witness**: the matched-filter kernel used to find the phrase in + loopback and mic buffers. +- **Response witness**: the per-tone kernels used to measure how strongly each + mic hears each emitted band. + +One chirplet phrase gives a delay estimate. Repeated phrases give drift/SRO by +watching delay change over time. Per-band energy over many phrases gives the +normalization curve. The important constraint is that all three measurements +must be tied to the same emitted phrase, not three separately invented probes. Next, replace the diagnostic bridge with native audio capture workers that append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state, diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 78f4953..a5bf8e4 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -42,9 +42,12 @@ a named invariant that the native runtime cannot protect yet. - `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback block metadata and emits `audio-block` JSON events for the diagnostic runtime adapter. +- `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish calibration + phrase, PCM rendering, matched timing trace, and per-band response kernels. - `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement: sample-bearing audio blocks are resampled into the Scarlett loopback timeline - and compared by chirplet-energy delay estimation. + and compared by chirplet-energy delay estimation with fractional peak + interpolation. - `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional aligned mono frame for loopback-referenced audio channels that clear the confidence gate. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 9516bda..7c69c27 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -90,13 +90,16 @@ not become the synchronized program authority. ## Audio Field The six-microphone path is separate from the bridge. Scarlett speaker loopback -is the current timing authority when calibration chirplets are playing. Mimir -emits short 9-16 kHz chirplets through Aquarium audio, compares mic buffers -against loopback by chirplet-energy delay estimation, and can build a -provisional aligned mono frame for channels that clear the confidence gate. -Camera mics are spatial/context witnesses; Focusrite devices are dialogue -anchors. The current aligned frame is integer-delay only; SRO smoothing, -fractional delay, and the hot resampler still belong in Faust/native DSP. +is the current timing authority when calibration chirplets are playing. +`MimirChirpletCalibrationPhrase` owns the harmonic-ish calibration phrase around +8 kHz, 10 kHz, 12 kHz, and 16 kHz, plus the matched timing and response kernels. +Mimir emits that phrase through Aquarium audio, compares mic buffers against +loopback by chirplet-energy delay estimation, and can build a provisional +aligned mono frame for channels that clear the confidence gate. Reports now +carry fractional delay and per-band matched energy. Camera mics are +spatial/context witnesses; Focusrite devices are dialogue anchors. The current +aligned frame is integer-delay only; SRO smoothing, fractional delay, and the +hot resampler still belong in Faust/native DSP. ## Visual Fusion diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index acb6b7d..4eb114b 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -63,11 +63,13 @@ Get-Content .\state\evidence.jsonl -Tail 8 640x480@60, and 81.10 fps when the PS3 Eye runs 320x240@187. The regular Kiyo and second PS3 Eye were absent, so six-camera viability is not proven. - Current chirplet-backed smoke uses Scarlett speaker loopback as timing - authority and can build an aligned audio frame containing loopback, Focusrite - mic, Kiyo mic, and Kiyo Pro mic when 9-16 kHz calibration chirplets are - playing. PS3 Eye audio is enumeration/runtime-fragile: one run saw both mic - buffers empty while both cameras were live, then a replug made both PS3 Eye - mic buffers emit 480-frame WASAPI blocks again. + authority. `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish + phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, the Aquarium PCM rendering, + matched timing trace, and per-band response hooks. Sync reports now include + fractional delay and per-band matched energy. PS3 Eye audio is + enumeration/runtime-fragile: one run saw both mic buffers empty while both + cameras were live, then a replug made both PS3 Eye mic buffers emit + 480-frame WASAPI blocks again. ## Immediate Re-entry Instruction diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index e50287d..5753a66 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -52,7 +52,7 @@ var reports = hub.AnalyzeAudioSynchronization(syncReference); foreach (var report in reports.OrderBy(report => report.SourceId, StringComparer.Ordinal)) { Console.WriteLine( - $"sync {report.ReferenceSourceId}->{report.SourceId}: delaySamples={report.DelaySamples} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} compared={report.ComparedSamples}"); + $"sync {report.ReferenceSourceId}->{report.SourceId}: delaySamples={report.DelaySamples} fractionalDelaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} bands={DescribeBands(report.BandResponses)} compared={report.ComparedSamples}"); } var aligned = hub.BuildAlignedAudioFrame(syncReference); @@ -63,7 +63,7 @@ if (aligned != null) foreach (var channel in aligned.Channels) { Console.WriteLine( - $"aligned-channel {channel.SourceId}: delaySamples={channel.DelaySamples} confidence={channel.Confidence:0.000}"); + $"aligned-channel {channel.SourceId}: delaySamples={channel.DelaySamples} fractionalDelaySamples={channel.FractionalDelaySamples:0.000} confidence={channel.Confidence:0.000}"); } } @@ -104,6 +104,13 @@ static string ParseStringOption(IReadOnlyList args, string name, string return fallback; } +static string DescribeBands(IReadOnlyList bands) +{ + return bands.Count == 0 + ? "none" + : string.Join(",", bands.Select(band => $"{band.CenterHz:0}Hz:{band.Energy:0.000}")); +} + internal sealed class DisposableConfiguration : IDisposable { private readonly List ownedSources; diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index ad8d021..84857ef 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -11,17 +11,14 @@ namespace Mimir.Runtime; public sealed class MimirRuntime : IAquariumRuntime { private const string DefaultAudioSyncReference = "loopback-scarlett-speakers"; - private const int CalibrationChirpSampleRate = 48_000; - private const float CalibrationChirpIntervalSeconds = 1.5f; - private const float CalibrationChirpDurationSeconds = 0.08f; - private const float CalibrationChirpGain = 0.125f; + private static readonly MimirChirpletCalibrationPhrase CalibrationPhrase = MimirChirpletCalibrationPhrase.Default; private readonly LocalCastRuntime visualRuntime; private readonly MimirSynchronizationHub synchronization; private readonly AquariumUiDocument ui; private int lastPollCount; private float runtimeSeconds; - private float nextCalibrationChirpSeconds = 0.5f; - private string? calibrationChirpPcm16Base64; + private float nextCalibrationChirpSeconds = (float)CalibrationPhrase.FirstFireSeconds; + private string? calibrationPhrasePcm16Base64; public MimirRuntime(AquariumRuntimeOptions options) : this(options, MimirRuntimeConfiguration.Load()) @@ -121,7 +118,7 @@ public sealed class MimirRuntime : IAquariumRuntime panel.Readout("Buffer details", DescribeBuffers); panel.Readout("Audio sync", DescribeAudioSync); panel.Readout("Aligned audio", DescribeAlignedAudio); - panel.Readout("Chirp reference", () => $"{DefaultAudioSyncReference} every {CalibrationChirpIntervalSeconds:0.0}s"); + panel.Readout("Chirplet reference", () => $"{DefaultAudioSyncReference} every {CalibrationPhrase.IntervalSeconds:0.0}s"); }); } @@ -130,35 +127,14 @@ public sealed class MimirRuntime : IAquariumRuntime while (runtimeSeconds >= nextCalibrationChirpSeconds) { visualRuntime.Audio.EnqueuePcm16Base64( - calibrationChirpPcm16Base64 ??= BuildCalibrationChirpPcm16Base64(), - CalibrationChirpSampleRate, + calibrationPhrasePcm16Base64 ??= CalibrationPhrase.RenderPcm16Base64(), + CalibrationPhrase.SampleRate, channels: 1, gain: 1.0f); - nextCalibrationChirpSeconds += CalibrationChirpIntervalSeconds; + nextCalibrationChirpSeconds += (float)CalibrationPhrase.IntervalSeconds; } } - private static string BuildCalibrationChirpPcm16Base64() - { - var frameCount = Math.Max(1, (int)Math.Round(CalibrationChirpSampleRate * CalibrationChirpDurationSeconds)); - var bytes = new byte[frameCount * sizeof(short)]; - const double startHz = 9_000.0; - const double endHz = 16_000.0; - for (var frame = 0; frame < frameCount; frame++) - { - var t = frame / (double)CalibrationChirpSampleRate; - var normalized = frameCount <= 1 ? 1.0 : frame / (double)(frameCount - 1); - var phase = 2.0 * Math.PI * (startHz * t + 0.5 * (endHz - startHz) * t * normalized); - var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); - var value = Math.Clamp(Math.Sin(phase) * envelope * CalibrationChirpGain, -1.0, 1.0); - var sample = (short)Math.Round(value * short.MaxValue); - bytes[frame * sizeof(short)] = (byte)(sample & 0xff); - bytes[frame * sizeof(short) + 1] = (byte)((sample >> 8) & 0xff); - } - - return Convert.ToBase64String(bytes); - } - private string DescribeBuffers() { return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer)); @@ -185,7 +161,7 @@ public sealed class MimirRuntime : IAquariumRuntime var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference); return reports.Count == 0 ? "no payload windows" - : string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.DelaySamples} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00}")); + : string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00}")); } private string DescribeAlignedAudio() diff --git a/src/Mimir.Runtime/Synchronization/MimirAlignedAudioFrame.cs b/src/Mimir.Runtime/Synchronization/MimirAlignedAudioFrame.cs index f68f9be..1fdf6b0 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAlignedAudioFrame.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAlignedAudioFrame.cs @@ -3,6 +3,7 @@ namespace Mimir.Runtime.Synchronization; public sealed record MimirAlignedAudioChannel( string SourceId, int DelaySamples, + double FractionalDelaySamples, double Confidence); public sealed record MimirAlignedAudioFrame( diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs index 70a35c3..d952f41 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationAnalyzer.cs @@ -6,10 +6,8 @@ public sealed class MimirAudioSynchronizationAnalyzer { private const int MaxWindowSamples = 48_000 * 5; private const int MaxLagSamples = 2_400; - private const int ChirpletHopSamples = 64; - private const double ChirpletDurationSeconds = 0.08; - private const double ChirpletStartHz = 9_000.0; - private const double ChirpletEndHz = 16_000.0; + private const int ChirpletHopSamples = 16; + private readonly MimirChirpletCalibrationPhrase chirpletPhrase = MimirChirpletCalibrationPhrase.Default; public IReadOnlyList Analyze( IEnumerable buffers, @@ -66,23 +64,29 @@ public sealed class MimirAudioSynchronizationAnalyzer continue; } - var referenceSync = BuildChirpletEnergyTrace(referenceSamples.AsSpan(^compared..), referenceBlock.SampleRate); - var candidateSync = BuildChirpletEnergyTrace(candidateSamples.AsSpan(^compared..), referenceBlock.SampleRate); + var referenceWindow = referenceSamples.AsSpan(^compared..); + var candidateWindow = candidateSamples.AsSpan(^compared..); + var referenceSync = chirpletPhrase.BuildEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples); + var candidateSync = chirpletPhrase.BuildEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples); if (referenceSync.Length < 16 || candidateSync.Length < 16) { continue; } var maxLag = Math.Min(MaxLagSamples / ChirpletHopSamples, Math.Min(referenceSync.Length, candidateSync.Length) / 2); - var (delaySamples, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag); - delaySamples *= ChirpletHopSamples; + var (delayHops, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag); + var fractionalDelaySamples = delayHops * ChirpletHopSamples; + var delaySamples = (int)Math.Round(fractionalDelaySamples); + var bandResponses = chirpletPhrase.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate); reports.Add(new MimirAudioSynchronizationReport( reference.Descriptor.SourceId, buffer.Descriptor.SourceId, referenceBlock.SampleRate, delaySamples, - delaySamples * 1000.0 / referenceBlock.SampleRate, + fractionalDelaySamples, + fractionalDelaySamples * 1000.0 / referenceBlock.SampleRate, confidence, + bandResponses, compared, reference.Latest?.Sequence ?? 0, buffer.Latest?.Sequence ?? 0)); @@ -142,7 +146,7 @@ public sealed class MimirAudioSynchronizationAnalyzer var maxPositiveDelay = Math.Max(0, alignedReports.Select(report => report.DelaySamples).DefaultIfEmpty(0).Max()); var channels = new List { - new(referenceSourceId, 0, 1.0), + new(referenceSourceId, 0, 0.0, 1.0), }; var samples = new List { @@ -171,7 +175,11 @@ public sealed class MimirAudioSynchronizationAnalyzer } samples.Add(TailEndingBefore(candidateSamples, frameCount, trimSamples)); - channels.Add(new MimirAlignedAudioChannel(report.SourceId, report.DelaySamples, report.Confidence)); + channels.Add(new MimirAlignedAudioChannel( + report.SourceId, + report.DelaySamples, + report.FractionalDelaySamples, + report.Confidence)); } return new MimirAlignedAudioFrame( @@ -276,81 +284,12 @@ public sealed class MimirAudioSynchronizationAnalyzer return output; } - private static float[] BuildChirpletEnergyTrace(ReadOnlySpan samples, int sampleRate) - { - var kernelLength = Math.Max(8, (int)Math.Round(sampleRate * ChirpletDurationSeconds)); - if (samples.Length < kernelLength) - { - return []; - } - - var output = new float[1 + (samples.Length - kernelLength) / ChirpletHopSamples]; - for (var frame = 0; frame < output.Length; frame++) - { - var offset = frame * ChirpletHopSamples; - var real = 0.0; - var imag = 0.0; - var energy = 0.0; - for (var index = 0; index < kernelLength; index++) - { - var normalized = kernelLength <= 1 ? 1.0 : index / (double)(kernelLength - 1); - var time = index / (double)sampleRate; - var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); - var phase = 2.0 * Math.PI * (ChirpletStartHz * time + 0.5 * (ChirpletEndHz - ChirpletStartHz) * time * normalized); - var sample = samples[offset + index] * envelope; - real += sample * Math.Cos(phase); - imag -= sample * Math.Sin(phase); - energy += sample * sample; - } - - var magnitude = Math.Sqrt(real * real + imag * imag); - output[frame] = energy > 1.0e-12 ? (float)(magnitude / Math.Sqrt(energy)) : 0.0f; - } - - return ContrastNormalize(output); - } - - private static float[] ContrastNormalize(float[] samples) - { - if (samples.Length == 0) - { - return samples; - } - - var mean = 0.0; - for (var index = 0; index < samples.Length; index++) - { - mean += samples[index]; - } - - mean /= samples.Length; - var variance = 0.0; - for (var index = 0; index < samples.Length; index++) - { - var centered = samples[index] - mean; - variance += centered * centered; - } - - var deviation = Math.Sqrt(variance / samples.Length); - if (deviation <= 1.0e-12) - { - return samples; - } - - var output = new float[samples.Length]; - for (var index = 0; index < samples.Length; index++) - { - var z = (samples[index] - mean) / deviation; - output[index] = z > 0.0 ? (float)(z * z) : 0.0f; - } - - return output; - } - - private static (int DelaySamples, double Confidence) EstimateDelay(ReadOnlySpan reference, ReadOnlySpan candidate, int maxLag) + private static (double DelaySamples, double Confidence) EstimateDelay(ReadOnlySpan reference, ReadOnlySpan candidate, int maxLag) { var bestLag = 0; var best = double.NegativeInfinity; + var previous = double.NegativeInfinity; + var next = double.NegativeInfinity; for (var lag = -maxLag; lag <= maxLag; lag++) { var score = NormalizedCorrelation(reference, candidate, lag); @@ -361,7 +300,34 @@ public sealed class MimirAudioSynchronizationAnalyzer } } - return (bestLag, Math.Max(0.0, Math.Min(1.0, best))); + if (bestLag > -maxLag) + { + previous = NormalizedCorrelation(reference, candidate, bestLag - 1); + } + + if (bestLag < maxLag) + { + next = NormalizedCorrelation(reference, candidate, bestLag + 1); + } + + var offset = ParabolicPeakOffset(previous, best, next); + return (bestLag + offset, Math.Max(0.0, Math.Min(1.0, best))); + } + + private static double ParabolicPeakOffset(double left, double center, double right) + { + if (!double.IsFinite(left) || !double.IsFinite(center) || !double.IsFinite(right)) + { + return 0.0; + } + + var denominator = left - 2.0 * center + right; + if (Math.Abs(denominator) <= 1.0e-12) + { + return 0.0; + } + + return Math.Clamp(0.5 * (left - right) / denominator, -0.5, 0.5); } private static double NormalizedCorrelation(ReadOnlySpan reference, ReadOnlySpan candidate, int lag) diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs index 65dfd9c..18df0cc 100644 --- a/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs +++ b/src/Mimir.Runtime/Synchronization/MimirAudioSynchronizationReport.cs @@ -5,8 +5,10 @@ public sealed record MimirAudioSynchronizationReport( string SourceId, int SampleRate, int DelaySamples, + double FractionalDelaySamples, double DelayMilliseconds, double Confidence, + IReadOnlyList BandResponses, int ComparedSamples, ulong ReferenceSequence, ulong SourceSequence); diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs b/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs new file mode 100644 index 0000000..9f8e40c --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirChirpletCalibrationPhrase.cs @@ -0,0 +1,246 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirChirpletTone( + double StartSeconds, + double DurationSeconds, + double StartHz, + double EndHz, + double Gain = 1.0) +{ + public double CenterHz => (StartHz + EndHz) * 0.5; +} + +public sealed record MimirChirpletBandResponse( + double CenterHz, + double Energy); + +public sealed class MimirChirpletCalibrationPhrase +{ + private readonly float[] kernel; + private readonly IReadOnlyList toneKernels; + + public MimirChirpletCalibrationPhrase( + int sampleRate, + double intervalSeconds, + double firstFireSeconds, + double gain, + IReadOnlyList tones) + { + SampleRate = sampleRate; + IntervalSeconds = intervalSeconds; + FirstFireSeconds = firstFireSeconds; + Gain = gain; + Tones = tones; + DurationSeconds = tones.Count == 0 + ? 0.0 + : tones.Max(tone => tone.StartSeconds + tone.DurationSeconds); + kernel = RenderMonoFloat(sampleRate); + toneKernels = tones.Select(tone => RenderToneKernel(tone, sampleRate, DurationSeconds)).ToArray(); + } + + public static MimirChirpletCalibrationPhrase Default { get; } = new( + sampleRate: 48_000, + intervalSeconds: 1.5, + firstFireSeconds: 0.5, + gain: 0.125, + tones: + [ + new(0.000, 0.055, 7_680.0, 8_320.0), + new(0.085, 0.055, 9_600.0, 10_400.0), + new(0.170, 0.055, 11_520.0, 12_480.0), + new(0.255, 0.055, 15_360.0, 16_000.0), + ]); + + public int SampleRate { get; } + + public double IntervalSeconds { get; } + + public double FirstFireSeconds { get; } + + public double Gain { get; } + + public double DurationSeconds { get; } + + public IReadOnlyList Tones { get; } + + public string RenderPcm16Base64() + { + var samples = RenderMonoFloat(SampleRate); + var bytes = new byte[samples.Length * sizeof(short)]; + for (var index = 0; index < samples.Length; index++) + { + var sample = (short)Math.Round(Math.Clamp(samples[index], -1.0f, 1.0f) * short.MaxValue); + bytes[index * sizeof(short)] = (byte)(sample & 0xff); + bytes[index * sizeof(short) + 1] = (byte)((sample >> 8) & 0xff); + } + + return Convert.ToBase64String(bytes); + } + + public float[] BuildEnergyTrace(ReadOnlySpan samples, int sampleRate, int hopSamples) + { + var analysisKernel = sampleRate == SampleRate ? kernel : RenderMonoFloat(sampleRate); + return BuildMatchedEnergyTrace(samples, analysisKernel, hopSamples); + } + + public IReadOnlyList EstimateBandResponse(ReadOnlySpan samples, int sampleRate) + { + if (samples.Length == 0 || Tones.Count == 0) + { + return []; + } + + var responses = new List(Tones.Count); + for (var index = 0; index < Tones.Count; index++) + { + var tone = Tones[index]; + var toneKernel = sampleRate == SampleRate + ? toneKernels[index] + : RenderToneKernel(tone, sampleRate, DurationSeconds); + var energy = MaxMatchedEnergy(samples, toneKernel, Math.Max(1, sampleRate / 1_000)); + responses.Add(new MimirChirpletBandResponse(tone.CenterHz, energy)); + } + + return responses; + } + + private float[] RenderMonoFloat(int sampleRate) + { + var frameCount = Math.Max(1, (int)Math.Ceiling(DurationSeconds * sampleRate)); + var samples = new float[frameCount]; + foreach (var tone in Tones) + { + AddTone(samples, sampleRate, tone); + } + + var peak = samples.Select(Math.Abs).DefaultIfEmpty(0.0f).Max(); + if (peak <= 1.0e-9f) + { + return samples; + } + + var scale = (float)(Gain / peak); + for (var index = 0; index < samples.Length; index++) + { + samples[index] *= scale; + } + + return samples; + } + + private static float[] RenderToneKernel(MimirChirpletTone tone, int sampleRate, double phraseDurationSeconds) + { + var frameCount = Math.Max(1, (int)Math.Ceiling(phraseDurationSeconds * sampleRate)); + var samples = new float[frameCount]; + AddTone(samples, sampleRate, tone); + return samples; + } + + private static void AddTone(float[] samples, int sampleRate, MimirChirpletTone tone) + { + var startFrame = Math.Max(0, (int)Math.Round(tone.StartSeconds * sampleRate)); + var frameCount = Math.Max(1, (int)Math.Round(tone.DurationSeconds * sampleRate)); + for (var frame = 0; frame < frameCount && startFrame + frame < samples.Length; frame++) + { + var normalized = frameCount <= 1 ? 1.0 : frame / (double)(frameCount - 1); + var t = frame / (double)sampleRate; + var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); + var phase = 2.0 * Math.PI * (tone.StartHz * t + 0.5 * (tone.EndHz - tone.StartHz) * t * normalized); + samples[startFrame + frame] += (float)(Math.Sin(phase) * envelope * tone.Gain); + } + } + + private static float[] BuildMatchedEnergyTrace(ReadOnlySpan samples, ReadOnlySpan kernel, int hopSamples) + { + if (samples.Length < kernel.Length || kernel.Length == 0) + { + return []; + } + + var output = new float[1 + (samples.Length - kernel.Length) / hopSamples]; + for (var frame = 0; frame < output.Length; frame++) + { + var offset = frame * hopSamples; + output[frame] = (float)MatchedEnergy(samples.Slice(offset, kernel.Length), kernel); + } + + return ContrastNormalize(output); + } + + private static double MatchedEnergy(ReadOnlySpan samples, ReadOnlySpan kernel) + { + var count = Math.Min(samples.Length, kernel.Length); + if (count == 0) + { + return 0.0; + } + + var dot = 0.0; + var sampleEnergy = 0.0; + var kernelEnergy = 0.0; + for (var index = 0; index < count; index++) + { + var sample = samples[index]; + var basis = kernel[index]; + dot += sample * basis; + sampleEnergy += sample * sample; + kernelEnergy += basis * basis; + } + + var denominator = Math.Sqrt(sampleEnergy * kernelEnergy); + return denominator > 1.0e-12 ? Math.Abs(dot) / denominator : 0.0; + } + + private static double MaxMatchedEnergy(ReadOnlySpan samples, ReadOnlySpan kernel, int hopSamples) + { + if (samples.Length < kernel.Length || kernel.Length == 0) + { + return 0.0; + } + + var best = 0.0; + for (var offset = 0; offset <= samples.Length - kernel.Length; offset += hopSamples) + { + best = Math.Max(best, MatchedEnergy(samples.Slice(offset, kernel.Length), kernel)); + } + + return best; + } + + private static float[] ContrastNormalize(float[] samples) + { + if (samples.Length == 0) + { + return samples; + } + + var mean = 0.0; + for (var index = 0; index < samples.Length; index++) + { + mean += samples[index]; + } + + mean /= samples.Length; + var variance = 0.0; + for (var index = 0; index < samples.Length; index++) + { + var centered = samples[index] - mean; + variance += centered * centered; + } + + var deviation = Math.Sqrt(variance / samples.Length); + if (deviation <= 1.0e-12) + { + return samples; + } + + var output = new float[samples.Length]; + for (var index = 0; index < samples.Length; index++) + { + var z = (samples[index] - mean) / deviation; + output[index] = z > 0.0 ? (float)(z * z) : 0.0f; + } + + return output; + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 81234ca..95d4e6a 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -33,3 +33,5 @@ {"ts":"2026-05-22T11:39:31+01:00","type":"verification","status":"partial","note":"Chirplet-backed sync smoke used Scarlett speaker loopback as timing authority with 9-16 kHz calibration chirps playing, resampled 48 kHz mics into the 44.1 kHz loopback timeline, and built an aligned audio frame with 4 channels: loopback, mic-focusrite-local delaySamples=-768 confidence=0.160, mic-kiyo delaySamples=-256 confidence=0.119, and mic-kiyo-pro delaySamples=-1664 confidence=0.118. Both PS3 Eye audio buffers were empty in this run while both PS3 Eye camera buffers were live, so PS3 Eye audio remains enumeration/runtime-fragile rather than solved."} {"ts":"2026-05-22T11:48:37+01:00","type":"verification","status":"ok","note":"After replugging the PS3 Eyes, full Mimir.BufferSmoke with --require-samples proved both PS3 Eye audio buffers were live again: mic-usb-camera-0 count=324 and mic-usb-camera-1 count=210, each reporting 2ch 48000Hz Float32 480-frame WASAPI blocks. Both PS3 Eye camera buffers were live at the same time. Loopback-referenced sync reports included mic-usb-camera-0 delaySamples=2368 confidence=0.095 and mic-usb-camera-1 delaySamples=-2304 confidence=0.152; only mic-usb-camera-1 cleared the current aligned-frame confidence gate in that run."} {"ts":"2026-05-22T11:58:15+01:00","type":"verification","status":"ok","note":"Delay signs were suspect because MimirAudioSynchronizationAnalyzer compared latest sample tails from buffers with different edge timestamps. The analyzer now estimates each pair at a shared timestamp edge and BuildAlignedAudioFrame chooses an output edge all positive-late channels can supply. Controlled 9-16 kHz chirplet smoke after the fix reported positive loopback-relative PS3 Eye delay: mic-usb-camera-1 delaySamples=704, delayMs=15.964, confidence=0.166, and the aligned frame contained loopback plus mic-usb-camera-1."} +{"ts":"2026-05-22T12:38:53+01:00","type":"implementation","status":"ok","note":"Documented the chirplet calibration model and started the structured implementation. MimirChirpletCalibrationPhrase now owns a harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, PCM rendering for Aquarium emission, matched chirplet timing traces, and per-band matched-energy response hooks. MimirRuntime emits that shared phrase instead of a separate hard-coded sweep, and MimirAudioSynchronizationAnalyzer uses the same phrase for loopback-referenced delay analysis with 16-sample hops and parabolic fractional peak interpolation. BufferSmoke output now includes fractionalDelaySamples and per-band energy."} +{"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."} diff --git a/state/map.yaml b/state/map.yaml index e9e3abd..6af4e2d 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -65,12 +65,15 @@ accepted_design: runtime rolling buffers; it does not carry pixels/audio payloads or own the final local-device hot path. - The diagnostic audio path can optionally carry base64 float sample payloads - for sync smoke tests. `MimirAudioSynchronizationAnalyzer` now treats + for sync smoke tests. `MimirChirpletCalibrationPhrase` owns the structured + harmonic-ish calibration phrase, PCM rendering, matched timing kernel, and + per-band response kernels. `MimirAudioSynchronizationAnalyzer` now treats `loopback-scarlett-speakers` as timing authority, resamples candidate mic windows into that timeline, projects loopback and mic windows through the - emitted chirplet shape, and estimates current delay from chirplet-energy + emitted chirplet phrase, and estimates current delay from chirplet-energy traces at a shared timestamp edge. Positive delay means the candidate mic is - late relative to loopback. `MimirSynchronizationHub.BuildAlignedAudioFrame` + late relative to loopback. Reports include rounded delay, fractional delay, + confidence, and per-band matched energy. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first provisional loopback-referenced aligned mono frame for channels that clear the confidence gate. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live