Add structured chirplet calibration phrase
This commit is contained in:
Meta Vens
parent
5ab8e80cf0
commit
bbb052a8ba
12 changed files with 392 additions and 136 deletions
|
|
@ -49,19 +49,64 @@ loopback audio is actively playing. One PS3 Eye mic previously enumerated but
|
|||
produced zero WASAPI packets until that Eye was unplugged and replugged.
|
||||
|
||||
The full probe runtime config now enables sample-bearing blocks for every local
|
||||
audio source. `MimirRuntime` emits short 9-16 kHz calibration chirplets through
|
||||
Aquarium audio every 1.5 seconds. `MimirAudioSynchronizationAnalyzer` resamples
|
||||
candidate mic windows into the loopback sample-rate timeline, projects them
|
||||
through the same chirplet shape, contrast-normalizes the resulting energy
|
||||
trace, and estimates current delay against `loopback-scarlett-speakers`.
|
||||
audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration
|
||||
phrase and the matched-filter shape used to analyze it. The default phrase is a
|
||||
short harmonic-ish pattern around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, repeated
|
||||
every 1.5 seconds through Aquarium audio. That makes the telemetry less like a
|
||||
single sterile squeal and more like a small, identifiable timing signature.
|
||||
|
||||
`MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the
|
||||
loopback sample-rate timeline, projects loopback and candidate windows through
|
||||
the same phrase, contrast-normalizes the resulting energy traces, and estimates
|
||||
current delay against `loopback-scarlett-speakers`. Reports carry both rounded
|
||||
integer delay and fractional delay from parabolic peak interpolation over the
|
||||
chirplet correlation peak.
|
||||
|
||||
`MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned
|
||||
mono frame: loopback is always channel zero, and other channels enter only when
|
||||
their chirplet confidence clears the gate. Delay estimates compare loopback and
|
||||
candidate mic windows at a shared timestamp edge; positive delay means the
|
||||
candidate mic is late relative to loopback. This is still integer-delay
|
||||
alignment at 64-sample chirplet-hop granularity. SRO smoothing, fractional-delay
|
||||
correction, and the hot resampler are the actuator still ahead of us.
|
||||
candidate mic is late relative to loopback. The current chirplet trace uses a
|
||||
16-sample hop and parabolic peak interpolation. The aligned-frame application
|
||||
still rounds to integer samples; the fractional estimate exists so the next
|
||||
actuator can drive a real fractional-delay line.
|
||||
|
||||
The same phrase also starts the frequency-response path. Each report includes
|
||||
per-band matched energy for the phrase tones. That is not a finished room/mic
|
||||
normalizer yet, but it is the live surface that will become response-curve
|
||||
estimation: loopback carries what was emitted, each mic carries what survived
|
||||
speaker, air, room, and capsule, and the ratio over repeated chirplet phrases
|
||||
becomes gain/phase correction evidence.
|
||||
|
||||
## Chirplet Calibration Model
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["MimirChirpletCalibrationPhrase"] --> B["Aquarium audio output"]
|
||||
B --> C["Scarlett speaker loopback"]
|
||||
B --> D["room + speakers + mics"]
|
||||
C --> E["loopback rolling buffer"]
|
||||
D --> F["mic rolling buffers"]
|
||||
E --> G["matched chirplet traces"]
|
||||
F --> G
|
||||
G --> H["delay + SRO estimates"]
|
||||
G --> I["per-band response estimates"]
|
||||
H --> J["fractional delay / resampler actuator"]
|
||||
I --> K["frequency response normalization"]
|
||||
```
|
||||
|
||||
The calibration phrase owns three facts:
|
||||
|
||||
- **Emission**: the PCM that Aquarium sends to the speakers.
|
||||
- **Timing witness**: the matched-filter kernel used to find the phrase in
|
||||
loopback and mic buffers.
|
||||
- **Response witness**: the per-tone kernels used to measure how strongly each
|
||||
mic hears each emitted band.
|
||||
|
||||
One chirplet phrase gives a delay estimate. Repeated phrases give drift/SRO by
|
||||
watching delay change over time. Per-band energy over many phrases gives the
|
||||
normalization curve. The important constraint is that all three measurements
|
||||
must be tied to the same emitted phrase, not three separately invented probes.
|
||||
|
||||
Next, replace the diagnostic bridge with native audio capture workers that
|
||||
append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state,
|
||||
|
|
|
|||
|
|
@ -42,9 +42,12 @@ a named invariant that the native runtime cannot protect yet.
|
|||
- `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback
|
||||
block metadata and emits `audio-block` JSON events for the diagnostic runtime
|
||||
adapter.
|
||||
- `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish calibration
|
||||
phrase, PCM rendering, matched timing trace, and per-band response kernels.
|
||||
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
|
||||
sample-bearing audio blocks are resampled into the Scarlett loopback timeline
|
||||
and compared by chirplet-energy delay estimation.
|
||||
and compared by chirplet-energy delay estimation with fractional peak
|
||||
interpolation.
|
||||
- `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
||||
provisional aligned mono frame for loopback-referenced audio channels that
|
||||
clear the confidence gate.
|
||||
|
|
|
|||
|
|
@ -90,13 +90,16 @@ not become the synchronized program authority.
|
|||
## Audio Field
|
||||
|
||||
The six-microphone path is separate from the bridge. Scarlett speaker loopback
|
||||
is the current timing authority when calibration chirplets are playing. Mimir
|
||||
emits short 9-16 kHz chirplets through Aquarium audio, compares mic buffers
|
||||
against loopback by chirplet-energy delay estimation, and can build a
|
||||
provisional aligned mono frame for channels that clear the confidence gate.
|
||||
Camera mics are spatial/context witnesses; Focusrite devices are dialogue
|
||||
anchors. The current aligned frame is integer-delay only; SRO smoothing,
|
||||
fractional delay, and the hot resampler still belong in Faust/native DSP.
|
||||
is the current timing authority when calibration chirplets are playing.
|
||||
`MimirChirpletCalibrationPhrase` owns the harmonic-ish calibration phrase around
|
||||
8 kHz, 10 kHz, 12 kHz, and 16 kHz, plus the matched timing and response kernels.
|
||||
Mimir emits that phrase through Aquarium audio, compares mic buffers against
|
||||
loopback by chirplet-energy delay estimation, and can build a provisional
|
||||
aligned mono frame for channels that clear the confidence gate. Reports now
|
||||
carry fractional delay and per-band matched energy. Camera mics are
|
||||
spatial/context witnesses; Focusrite devices are dialogue anchors. The current
|
||||
aligned frame is integer-delay only; SRO smoothing, fractional delay, and the
|
||||
hot resampler still belong in Faust/native DSP.
|
||||
|
||||
## Visual Fusion
|
||||
|
||||
|
|
|
|||
|
|
@ -63,11 +63,13 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
640x480@60, and 81.10 fps when the PS3 Eye runs 320x240@187. The regular
|
||||
Kiyo and second PS3 Eye were absent, so six-camera viability is not proven.
|
||||
- Current chirplet-backed smoke uses Scarlett speaker loopback as timing
|
||||
authority and can build an aligned audio frame containing loopback, Focusrite
|
||||
mic, Kiyo mic, and Kiyo Pro mic when 9-16 kHz calibration chirplets are
|
||||
playing. PS3 Eye audio is enumeration/runtime-fragile: one run saw both mic
|
||||
buffers empty while both cameras were live, then a replug made both PS3 Eye
|
||||
mic buffers emit 480-frame WASAPI blocks again.
|
||||
authority. `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish
|
||||
phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, the Aquarium PCM rendering,
|
||||
matched timing trace, and per-band response hooks. Sync reports now include
|
||||
fractional delay and per-band matched energy. PS3 Eye audio is
|
||||
enumeration/runtime-fragile: one run saw both mic buffers empty while both
|
||||
cameras were live, then a replug made both PS3 Eye mic buffers emit
|
||||
480-frame WASAPI blocks again.
|
||||
|
||||
## Immediate Re-entry Instruction
|
||||
|
||||
|
|
|
|||
|
|
@ -52,7 +52,7 @@ var reports = hub.AnalyzeAudioSynchronization(syncReference);
|
|||
foreach (var report in reports.OrderBy(report => report.SourceId, StringComparer.Ordinal))
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"sync {report.ReferenceSourceId}->{report.SourceId}: delaySamples={report.DelaySamples} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} compared={report.ComparedSamples}");
|
||||
$"sync {report.ReferenceSourceId}->{report.SourceId}: delaySamples={report.DelaySamples} fractionalDelaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} bands={DescribeBands(report.BandResponses)} compared={report.ComparedSamples}");
|
||||
}
|
||||
|
||||
var aligned = hub.BuildAlignedAudioFrame(syncReference);
|
||||
|
|
@ -63,7 +63,7 @@ if (aligned != null)
|
|||
foreach (var channel in aligned.Channels)
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"aligned-channel {channel.SourceId}: delaySamples={channel.DelaySamples} confidence={channel.Confidence:0.000}");
|
||||
$"aligned-channel {channel.SourceId}: delaySamples={channel.DelaySamples} fractionalDelaySamples={channel.FractionalDelaySamples:0.000} confidence={channel.Confidence:0.000}");
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -104,6 +104,13 @@ static string ParseStringOption(IReadOnlyList<string> args, string name, string
|
|||
return fallback;
|
||||
}
|
||||
|
||||
static string DescribeBands(IReadOnlyList<MimirChirpletBandResponse> bands)
|
||||
{
|
||||
return bands.Count == 0
|
||||
? "none"
|
||||
: string.Join(",", bands.Select(band => $"{band.CenterHz:0}Hz:{band.Energy:0.000}"));
|
||||
}
|
||||
|
||||
internal sealed class DisposableConfiguration : IDisposable
|
||||
{
|
||||
private readonly List<IMimirStreamSource> ownedSources;
|
||||
|
|
|
|||
|
|
@ -11,17 +11,14 @@ namespace Mimir.Runtime;
|
|||
public sealed class MimirRuntime : IAquariumRuntime
|
||||
{
|
||||
private const string DefaultAudioSyncReference = "loopback-scarlett-speakers";
|
||||
private const int CalibrationChirpSampleRate = 48_000;
|
||||
private const float CalibrationChirpIntervalSeconds = 1.5f;
|
||||
private const float CalibrationChirpDurationSeconds = 0.08f;
|
||||
private const float CalibrationChirpGain = 0.125f;
|
||||
private static readonly MimirChirpletCalibrationPhrase CalibrationPhrase = MimirChirpletCalibrationPhrase.Default;
|
||||
private readonly LocalCastRuntime visualRuntime;
|
||||
private readonly MimirSynchronizationHub synchronization;
|
||||
private readonly AquariumUiDocument ui;
|
||||
private int lastPollCount;
|
||||
private float runtimeSeconds;
|
||||
private float nextCalibrationChirpSeconds = 0.5f;
|
||||
private string? calibrationChirpPcm16Base64;
|
||||
private float nextCalibrationChirpSeconds = (float)CalibrationPhrase.FirstFireSeconds;
|
||||
private string? calibrationPhrasePcm16Base64;
|
||||
|
||||
public MimirRuntime(AquariumRuntimeOptions options)
|
||||
: this(options, MimirRuntimeConfiguration.Load())
|
||||
|
|
@ -121,7 +118,7 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
panel.Readout("Buffer details", DescribeBuffers);
|
||||
panel.Readout("Audio sync", DescribeAudioSync);
|
||||
panel.Readout("Aligned audio", DescribeAlignedAudio);
|
||||
panel.Readout("Chirp reference", () => $"{DefaultAudioSyncReference} every {CalibrationChirpIntervalSeconds:0.0}s");
|
||||
panel.Readout("Chirplet reference", () => $"{DefaultAudioSyncReference} every {CalibrationPhrase.IntervalSeconds:0.0}s");
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -130,35 +127,14 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
while (runtimeSeconds >= nextCalibrationChirpSeconds)
|
||||
{
|
||||
visualRuntime.Audio.EnqueuePcm16Base64(
|
||||
calibrationChirpPcm16Base64 ??= BuildCalibrationChirpPcm16Base64(),
|
||||
CalibrationChirpSampleRate,
|
||||
calibrationPhrasePcm16Base64 ??= CalibrationPhrase.RenderPcm16Base64(),
|
||||
CalibrationPhrase.SampleRate,
|
||||
channels: 1,
|
||||
gain: 1.0f);
|
||||
nextCalibrationChirpSeconds += CalibrationChirpIntervalSeconds;
|
||||
nextCalibrationChirpSeconds += (float)CalibrationPhrase.IntervalSeconds;
|
||||
}
|
||||
}
|
||||
|
||||
private static string BuildCalibrationChirpPcm16Base64()
|
||||
{
|
||||
var frameCount = Math.Max(1, (int)Math.Round(CalibrationChirpSampleRate * CalibrationChirpDurationSeconds));
|
||||
var bytes = new byte[frameCount * sizeof(short)];
|
||||
const double startHz = 9_000.0;
|
||||
const double endHz = 16_000.0;
|
||||
for (var frame = 0; frame < frameCount; frame++)
|
||||
{
|
||||
var t = frame / (double)CalibrationChirpSampleRate;
|
||||
var normalized = frameCount <= 1 ? 1.0 : frame / (double)(frameCount - 1);
|
||||
var phase = 2.0 * Math.PI * (startHz * t + 0.5 * (endHz - startHz) * t * normalized);
|
||||
var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized);
|
||||
var value = Math.Clamp(Math.Sin(phase) * envelope * CalibrationChirpGain, -1.0, 1.0);
|
||||
var sample = (short)Math.Round(value * short.MaxValue);
|
||||
bytes[frame * sizeof(short)] = (byte)(sample & 0xff);
|
||||
bytes[frame * sizeof(short) + 1] = (byte)((sample >> 8) & 0xff);
|
||||
}
|
||||
|
||||
return Convert.ToBase64String(bytes);
|
||||
}
|
||||
|
||||
private string DescribeBuffers()
|
||||
{
|
||||
return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer));
|
||||
|
|
@ -185,7 +161,7 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference);
|
||||
return reports.Count == 0
|
||||
? "no payload windows"
|
||||
: string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.DelaySamples} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00}"));
|
||||
: string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00}"));
|
||||
}
|
||||
|
||||
private string DescribeAlignedAudio()
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ namespace Mimir.Runtime.Synchronization;
|
|||
public sealed record MimirAlignedAudioChannel(
|
||||
string SourceId,
|
||||
int DelaySamples,
|
||||
double FractionalDelaySamples,
|
||||
double Confidence);
|
||||
|
||||
public sealed record MimirAlignedAudioFrame(
|
||||
|
|
|
|||
|
|
@ -6,10 +6,8 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
{
|
||||
private const int MaxWindowSamples = 48_000 * 5;
|
||||
private const int MaxLagSamples = 2_400;
|
||||
private const int ChirpletHopSamples = 64;
|
||||
private const double ChirpletDurationSeconds = 0.08;
|
||||
private const double ChirpletStartHz = 9_000.0;
|
||||
private const double ChirpletEndHz = 16_000.0;
|
||||
private const int ChirpletHopSamples = 16;
|
||||
private readonly MimirChirpletCalibrationPhrase chirpletPhrase = MimirChirpletCalibrationPhrase.Default;
|
||||
|
||||
public IReadOnlyList<MimirAudioSynchronizationReport> Analyze(
|
||||
IEnumerable<MimirRollingStreamBuffer> buffers,
|
||||
|
|
@ -66,23 +64,29 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
continue;
|
||||
}
|
||||
|
||||
var referenceSync = BuildChirpletEnergyTrace(referenceSamples.AsSpan(^compared..), referenceBlock.SampleRate);
|
||||
var candidateSync = BuildChirpletEnergyTrace(candidateSamples.AsSpan(^compared..), referenceBlock.SampleRate);
|
||||
var referenceWindow = referenceSamples.AsSpan(^compared..);
|
||||
var candidateWindow = candidateSamples.AsSpan(^compared..);
|
||||
var referenceSync = chirpletPhrase.BuildEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples);
|
||||
var candidateSync = chirpletPhrase.BuildEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples);
|
||||
if (referenceSync.Length < 16 || candidateSync.Length < 16)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var maxLag = Math.Min(MaxLagSamples / ChirpletHopSamples, Math.Min(referenceSync.Length, candidateSync.Length) / 2);
|
||||
var (delaySamples, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag);
|
||||
delaySamples *= ChirpletHopSamples;
|
||||
var (delayHops, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag);
|
||||
var fractionalDelaySamples = delayHops * ChirpletHopSamples;
|
||||
var delaySamples = (int)Math.Round(fractionalDelaySamples);
|
||||
var bandResponses = chirpletPhrase.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate);
|
||||
reports.Add(new MimirAudioSynchronizationReport(
|
||||
reference.Descriptor.SourceId,
|
||||
buffer.Descriptor.SourceId,
|
||||
referenceBlock.SampleRate,
|
||||
delaySamples,
|
||||
delaySamples * 1000.0 / referenceBlock.SampleRate,
|
||||
fractionalDelaySamples,
|
||||
fractionalDelaySamples * 1000.0 / referenceBlock.SampleRate,
|
||||
confidence,
|
||||
bandResponses,
|
||||
compared,
|
||||
reference.Latest?.Sequence ?? 0,
|
||||
buffer.Latest?.Sequence ?? 0));
|
||||
|
|
@ -142,7 +146,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
var maxPositiveDelay = Math.Max(0, alignedReports.Select(report => report.DelaySamples).DefaultIfEmpty(0).Max());
|
||||
var channels = new List<MimirAlignedAudioChannel>
|
||||
{
|
||||
new(referenceSourceId, 0, 1.0),
|
||||
new(referenceSourceId, 0, 0.0, 1.0),
|
||||
};
|
||||
var samples = new List<float[]>
|
||||
{
|
||||
|
|
@ -171,7 +175,11 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
}
|
||||
|
||||
samples.Add(TailEndingBefore(candidateSamples, frameCount, trimSamples));
|
||||
channels.Add(new MimirAlignedAudioChannel(report.SourceId, report.DelaySamples, report.Confidence));
|
||||
channels.Add(new MimirAlignedAudioChannel(
|
||||
report.SourceId,
|
||||
report.DelaySamples,
|
||||
report.FractionalDelaySamples,
|
||||
report.Confidence));
|
||||
}
|
||||
|
||||
return new MimirAlignedAudioFrame(
|
||||
|
|
@ -276,81 +284,12 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
return output;
|
||||
}
|
||||
|
||||
private static float[] BuildChirpletEnergyTrace(ReadOnlySpan<float> samples, int sampleRate)
|
||||
{
|
||||
var kernelLength = Math.Max(8, (int)Math.Round(sampleRate * ChirpletDurationSeconds));
|
||||
if (samples.Length < kernelLength)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
var output = new float[1 + (samples.Length - kernelLength) / ChirpletHopSamples];
|
||||
for (var frame = 0; frame < output.Length; frame++)
|
||||
{
|
||||
var offset = frame * ChirpletHopSamples;
|
||||
var real = 0.0;
|
||||
var imag = 0.0;
|
||||
var energy = 0.0;
|
||||
for (var index = 0; index < kernelLength; index++)
|
||||
{
|
||||
var normalized = kernelLength <= 1 ? 1.0 : index / (double)(kernelLength - 1);
|
||||
var time = index / (double)sampleRate;
|
||||
var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized);
|
||||
var phase = 2.0 * Math.PI * (ChirpletStartHz * time + 0.5 * (ChirpletEndHz - ChirpletStartHz) * time * normalized);
|
||||
var sample = samples[offset + index] * envelope;
|
||||
real += sample * Math.Cos(phase);
|
||||
imag -= sample * Math.Sin(phase);
|
||||
energy += sample * sample;
|
||||
}
|
||||
|
||||
var magnitude = Math.Sqrt(real * real + imag * imag);
|
||||
output[frame] = energy > 1.0e-12 ? (float)(magnitude / Math.Sqrt(energy)) : 0.0f;
|
||||
}
|
||||
|
||||
return ContrastNormalize(output);
|
||||
}
|
||||
|
||||
private static float[] ContrastNormalize(float[] samples)
|
||||
{
|
||||
if (samples.Length == 0)
|
||||
{
|
||||
return samples;
|
||||
}
|
||||
|
||||
var mean = 0.0;
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
mean += samples[index];
|
||||
}
|
||||
|
||||
mean /= samples.Length;
|
||||
var variance = 0.0;
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
var centered = samples[index] - mean;
|
||||
variance += centered * centered;
|
||||
}
|
||||
|
||||
var deviation = Math.Sqrt(variance / samples.Length);
|
||||
if (deviation <= 1.0e-12)
|
||||
{
|
||||
return samples;
|
||||
}
|
||||
|
||||
var output = new float[samples.Length];
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
var z = (samples[index] - mean) / deviation;
|
||||
output[index] = z > 0.0 ? (float)(z * z) : 0.0f;
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
private static (int DelaySamples, double Confidence) EstimateDelay(ReadOnlySpan<float> reference, ReadOnlySpan<float> candidate, int maxLag)
|
||||
private static (double DelaySamples, double Confidence) EstimateDelay(ReadOnlySpan<float> reference, ReadOnlySpan<float> candidate, int maxLag)
|
||||
{
|
||||
var bestLag = 0;
|
||||
var best = double.NegativeInfinity;
|
||||
var previous = double.NegativeInfinity;
|
||||
var next = double.NegativeInfinity;
|
||||
for (var lag = -maxLag; lag <= maxLag; lag++)
|
||||
{
|
||||
var score = NormalizedCorrelation(reference, candidate, lag);
|
||||
|
|
@ -361,7 +300,34 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
}
|
||||
}
|
||||
|
||||
return (bestLag, Math.Max(0.0, Math.Min(1.0, best)));
|
||||
if (bestLag > -maxLag)
|
||||
{
|
||||
previous = NormalizedCorrelation(reference, candidate, bestLag - 1);
|
||||
}
|
||||
|
||||
if (bestLag < maxLag)
|
||||
{
|
||||
next = NormalizedCorrelation(reference, candidate, bestLag + 1);
|
||||
}
|
||||
|
||||
var offset = ParabolicPeakOffset(previous, best, next);
|
||||
return (bestLag + offset, Math.Max(0.0, Math.Min(1.0, best)));
|
||||
}
|
||||
|
||||
private static double ParabolicPeakOffset(double left, double center, double right)
|
||||
{
|
||||
if (!double.IsFinite(left) || !double.IsFinite(center) || !double.IsFinite(right))
|
||||
{
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
var denominator = left - 2.0 * center + right;
|
||||
if (Math.Abs(denominator) <= 1.0e-12)
|
||||
{
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
return Math.Clamp(0.5 * (left - right) / denominator, -0.5, 0.5);
|
||||
}
|
||||
|
||||
private static double NormalizedCorrelation(ReadOnlySpan<float> reference, ReadOnlySpan<float> candidate, int lag)
|
||||
|
|
|
|||
|
|
@ -5,8 +5,10 @@ public sealed record MimirAudioSynchronizationReport(
|
|||
string SourceId,
|
||||
int SampleRate,
|
||||
int DelaySamples,
|
||||
double FractionalDelaySamples,
|
||||
double DelayMilliseconds,
|
||||
double Confidence,
|
||||
IReadOnlyList<MimirChirpletBandResponse> BandResponses,
|
||||
int ComparedSamples,
|
||||
ulong ReferenceSequence,
|
||||
ulong SourceSequence);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,246 @@
|
|||
namespace Mimir.Runtime.Synchronization;
|
||||
|
||||
public sealed record MimirChirpletTone(
|
||||
double StartSeconds,
|
||||
double DurationSeconds,
|
||||
double StartHz,
|
||||
double EndHz,
|
||||
double Gain = 1.0)
|
||||
{
|
||||
public double CenterHz => (StartHz + EndHz) * 0.5;
|
||||
}
|
||||
|
||||
public sealed record MimirChirpletBandResponse(
|
||||
double CenterHz,
|
||||
double Energy);
|
||||
|
||||
public sealed class MimirChirpletCalibrationPhrase
|
||||
{
|
||||
private readonly float[] kernel;
|
||||
private readonly IReadOnlyList<float[]> toneKernels;
|
||||
|
||||
public MimirChirpletCalibrationPhrase(
|
||||
int sampleRate,
|
||||
double intervalSeconds,
|
||||
double firstFireSeconds,
|
||||
double gain,
|
||||
IReadOnlyList<MimirChirpletTone> tones)
|
||||
{
|
||||
SampleRate = sampleRate;
|
||||
IntervalSeconds = intervalSeconds;
|
||||
FirstFireSeconds = firstFireSeconds;
|
||||
Gain = gain;
|
||||
Tones = tones;
|
||||
DurationSeconds = tones.Count == 0
|
||||
? 0.0
|
||||
: tones.Max(tone => tone.StartSeconds + tone.DurationSeconds);
|
||||
kernel = RenderMonoFloat(sampleRate);
|
||||
toneKernels = tones.Select(tone => RenderToneKernel(tone, sampleRate, DurationSeconds)).ToArray();
|
||||
}
|
||||
|
||||
public static MimirChirpletCalibrationPhrase Default { get; } = new(
|
||||
sampleRate: 48_000,
|
||||
intervalSeconds: 1.5,
|
||||
firstFireSeconds: 0.5,
|
||||
gain: 0.125,
|
||||
tones:
|
||||
[
|
||||
new(0.000, 0.055, 7_680.0, 8_320.0),
|
||||
new(0.085, 0.055, 9_600.0, 10_400.0),
|
||||
new(0.170, 0.055, 11_520.0, 12_480.0),
|
||||
new(0.255, 0.055, 15_360.0, 16_000.0),
|
||||
]);
|
||||
|
||||
public int SampleRate { get; }
|
||||
|
||||
public double IntervalSeconds { get; }
|
||||
|
||||
public double FirstFireSeconds { get; }
|
||||
|
||||
public double Gain { get; }
|
||||
|
||||
public double DurationSeconds { get; }
|
||||
|
||||
public IReadOnlyList<MimirChirpletTone> Tones { get; }
|
||||
|
||||
public string RenderPcm16Base64()
|
||||
{
|
||||
var samples = RenderMonoFloat(SampleRate);
|
||||
var bytes = new byte[samples.Length * sizeof(short)];
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
var sample = (short)Math.Round(Math.Clamp(samples[index], -1.0f, 1.0f) * short.MaxValue);
|
||||
bytes[index * sizeof(short)] = (byte)(sample & 0xff);
|
||||
bytes[index * sizeof(short) + 1] = (byte)((sample >> 8) & 0xff);
|
||||
}
|
||||
|
||||
return Convert.ToBase64String(bytes);
|
||||
}
|
||||
|
||||
public float[] BuildEnergyTrace(ReadOnlySpan<float> samples, int sampleRate, int hopSamples)
|
||||
{
|
||||
var analysisKernel = sampleRate == SampleRate ? kernel : RenderMonoFloat(sampleRate);
|
||||
return BuildMatchedEnergyTrace(samples, analysisKernel, hopSamples);
|
||||
}
|
||||
|
||||
public IReadOnlyList<MimirChirpletBandResponse> EstimateBandResponse(ReadOnlySpan<float> samples, int sampleRate)
|
||||
{
|
||||
if (samples.Length == 0 || Tones.Count == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
var responses = new List<MimirChirpletBandResponse>(Tones.Count);
|
||||
for (var index = 0; index < Tones.Count; index++)
|
||||
{
|
||||
var tone = Tones[index];
|
||||
var toneKernel = sampleRate == SampleRate
|
||||
? toneKernels[index]
|
||||
: RenderToneKernel(tone, sampleRate, DurationSeconds);
|
||||
var energy = MaxMatchedEnergy(samples, toneKernel, Math.Max(1, sampleRate / 1_000));
|
||||
responses.Add(new MimirChirpletBandResponse(tone.CenterHz, energy));
|
||||
}
|
||||
|
||||
return responses;
|
||||
}
|
||||
|
||||
private float[] RenderMonoFloat(int sampleRate)
|
||||
{
|
||||
var frameCount = Math.Max(1, (int)Math.Ceiling(DurationSeconds * sampleRate));
|
||||
var samples = new float[frameCount];
|
||||
foreach (var tone in Tones)
|
||||
{
|
||||
AddTone(samples, sampleRate, tone);
|
||||
}
|
||||
|
||||
var peak = samples.Select(Math.Abs).DefaultIfEmpty(0.0f).Max();
|
||||
if (peak <= 1.0e-9f)
|
||||
{
|
||||
return samples;
|
||||
}
|
||||
|
||||
var scale = (float)(Gain / peak);
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
samples[index] *= scale;
|
||||
}
|
||||
|
||||
return samples;
|
||||
}
|
||||
|
||||
private static float[] RenderToneKernel(MimirChirpletTone tone, int sampleRate, double phraseDurationSeconds)
|
||||
{
|
||||
var frameCount = Math.Max(1, (int)Math.Ceiling(phraseDurationSeconds * sampleRate));
|
||||
var samples = new float[frameCount];
|
||||
AddTone(samples, sampleRate, tone);
|
||||
return samples;
|
||||
}
|
||||
|
||||
private static void AddTone(float[] samples, int sampleRate, MimirChirpletTone tone)
|
||||
{
|
||||
var startFrame = Math.Max(0, (int)Math.Round(tone.StartSeconds * sampleRate));
|
||||
var frameCount = Math.Max(1, (int)Math.Round(tone.DurationSeconds * sampleRate));
|
||||
for (var frame = 0; frame < frameCount && startFrame + frame < samples.Length; frame++)
|
||||
{
|
||||
var normalized = frameCount <= 1 ? 1.0 : frame / (double)(frameCount - 1);
|
||||
var t = frame / (double)sampleRate;
|
||||
var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized);
|
||||
var phase = 2.0 * Math.PI * (tone.StartHz * t + 0.5 * (tone.EndHz - tone.StartHz) * t * normalized);
|
||||
samples[startFrame + frame] += (float)(Math.Sin(phase) * envelope * tone.Gain);
|
||||
}
|
||||
}
|
||||
|
||||
private static float[] BuildMatchedEnergyTrace(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel, int hopSamples)
|
||||
{
|
||||
if (samples.Length < kernel.Length || kernel.Length == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
var output = new float[1 + (samples.Length - kernel.Length) / hopSamples];
|
||||
for (var frame = 0; frame < output.Length; frame++)
|
||||
{
|
||||
var offset = frame * hopSamples;
|
||||
output[frame] = (float)MatchedEnergy(samples.Slice(offset, kernel.Length), kernel);
|
||||
}
|
||||
|
||||
return ContrastNormalize(output);
|
||||
}
|
||||
|
||||
private static double MatchedEnergy(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel)
|
||||
{
|
||||
var count = Math.Min(samples.Length, kernel.Length);
|
||||
if (count == 0)
|
||||
{
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
var dot = 0.0;
|
||||
var sampleEnergy = 0.0;
|
||||
var kernelEnergy = 0.0;
|
||||
for (var index = 0; index < count; index++)
|
||||
{
|
||||
var sample = samples[index];
|
||||
var basis = kernel[index];
|
||||
dot += sample * basis;
|
||||
sampleEnergy += sample * sample;
|
||||
kernelEnergy += basis * basis;
|
||||
}
|
||||
|
||||
var denominator = Math.Sqrt(sampleEnergy * kernelEnergy);
|
||||
return denominator > 1.0e-12 ? Math.Abs(dot) / denominator : 0.0;
|
||||
}
|
||||
|
||||
private static double MaxMatchedEnergy(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel, int hopSamples)
|
||||
{
|
||||
if (samples.Length < kernel.Length || kernel.Length == 0)
|
||||
{
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
var best = 0.0;
|
||||
for (var offset = 0; offset <= samples.Length - kernel.Length; offset += hopSamples)
|
||||
{
|
||||
best = Math.Max(best, MatchedEnergy(samples.Slice(offset, kernel.Length), kernel));
|
||||
}
|
||||
|
||||
return best;
|
||||
}
|
||||
|
||||
private static float[] ContrastNormalize(float[] samples)
|
||||
{
|
||||
if (samples.Length == 0)
|
||||
{
|
||||
return samples;
|
||||
}
|
||||
|
||||
var mean = 0.0;
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
mean += samples[index];
|
||||
}
|
||||
|
||||
mean /= samples.Length;
|
||||
var variance = 0.0;
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
var centered = samples[index] - mean;
|
||||
variance += centered * centered;
|
||||
}
|
||||
|
||||
var deviation = Math.Sqrt(variance / samples.Length);
|
||||
if (deviation <= 1.0e-12)
|
||||
{
|
||||
return samples;
|
||||
}
|
||||
|
||||
var output = new float[samples.Length];
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
var z = (samples[index] - mean) / deviation;
|
||||
output[index] = z > 0.0 ? (float)(z * z) : 0.0f;
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
}
|
||||
|
|
@ -33,3 +33,5 @@
|
|||
{"ts":"2026-05-22T11:39:31+01:00","type":"verification","status":"partial","note":"Chirplet-backed sync smoke used Scarlett speaker loopback as timing authority with 9-16 kHz calibration chirps playing, resampled 48 kHz mics into the 44.1 kHz loopback timeline, and built an aligned audio frame with 4 channels: loopback, mic-focusrite-local delaySamples=-768 confidence=0.160, mic-kiyo delaySamples=-256 confidence=0.119, and mic-kiyo-pro delaySamples=-1664 confidence=0.118. Both PS3 Eye audio buffers were empty in this run while both PS3 Eye camera buffers were live, so PS3 Eye audio remains enumeration/runtime-fragile rather than solved."}
|
||||
{"ts":"2026-05-22T11:48:37+01:00","type":"verification","status":"ok","note":"After replugging the PS3 Eyes, full Mimir.BufferSmoke with --require-samples proved both PS3 Eye audio buffers were live again: mic-usb-camera-0 count=324 and mic-usb-camera-1 count=210, each reporting 2ch 48000Hz Float32 480-frame WASAPI blocks. Both PS3 Eye camera buffers were live at the same time. Loopback-referenced sync reports included mic-usb-camera-0 delaySamples=2368 confidence=0.095 and mic-usb-camera-1 delaySamples=-2304 confidence=0.152; only mic-usb-camera-1 cleared the current aligned-frame confidence gate in that run."}
|
||||
{"ts":"2026-05-22T11:58:15+01:00","type":"verification","status":"ok","note":"Delay signs were suspect because MimirAudioSynchronizationAnalyzer compared latest sample tails from buffers with different edge timestamps. The analyzer now estimates each pair at a shared timestamp edge and BuildAlignedAudioFrame chooses an output edge all positive-late channels can supply. Controlled 9-16 kHz chirplet smoke after the fix reported positive loopback-relative PS3 Eye delay: mic-usb-camera-1 delaySamples=704, delayMs=15.964, confidence=0.166, and the aligned frame contained loopback plus mic-usb-camera-1."}
|
||||
{"ts":"2026-05-22T12:38:53+01:00","type":"implementation","status":"ok","note":"Documented the chirplet calibration model and started the structured implementation. MimirChirpletCalibrationPhrase now owns a harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, PCM rendering for Aquarium emission, matched chirplet timing traces, and per-band matched-energy response hooks. MimirRuntime emits that shared phrase instead of a separate hard-coded sweep, and MimirAudioSynchronizationAnalyzer uses the same phrase for loopback-referenced delay analysis with 16-sample hops and parabolic fractional peak interpolation. BufferSmoke output now includes fractionalDelaySamples and per-band energy."}
|
||||
{"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."}
|
||||
|
|
|
|||
|
|
@ -65,12 +65,15 @@ accepted_design:
|
|||
runtime rolling buffers; it does not carry pixels/audio payloads or own the
|
||||
final local-device hot path.
|
||||
- The diagnostic audio path can optionally carry base64 float sample payloads
|
||||
for sync smoke tests. `MimirAudioSynchronizationAnalyzer` now treats
|
||||
for sync smoke tests. `MimirChirpletCalibrationPhrase` owns the structured
|
||||
harmonic-ish calibration phrase, PCM rendering, matched timing kernel, and
|
||||
per-band response kernels. `MimirAudioSynchronizationAnalyzer` now treats
|
||||
`loopback-scarlett-speakers` as timing authority, resamples candidate mic
|
||||
windows into that timeline, projects loopback and mic windows through the
|
||||
emitted chirplet shape, and estimates current delay from chirplet-energy
|
||||
emitted chirplet phrase, and estimates current delay from chirplet-energy
|
||||
traces at a shared timestamp edge. Positive delay means the candidate mic is
|
||||
late relative to loopback. `MimirSynchronizationHub.BuildAlignedAudioFrame`
|
||||
late relative to loopback. Reports include rounded delay, fractional delay,
|
||||
confidence, and per-band matched energy. `MimirSynchronizationHub.BuildAlignedAudioFrame`
|
||||
exposes the first provisional loopback-referenced aligned mono frame for
|
||||
channels that clear the confidence gate.
|
||||
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
||||
|
|
|
|||
Loading…
Reference in a new issue