Generate stateless chirplet timeline fingerprints

This commit is contained in:
Meta Vens 2026-05-22 12:54:56 +01:00
commit 7f804ad5da
9 changed files with 169 additions and 50 deletions

View file

@ -50,17 +50,23 @@ produced zero WASAPI packets until that Eye was unplugged and replugged.
The full probe runtime config now enables sample-bearing blocks for every local
audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration
phrase and the matched-filter shape used to analyze it. The default phrase is a
short harmonic-ish pattern around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, repeated
every 1.5 seconds through Aquarium audio. That makes the telemetry less like a
single sterile squeal and more like a small, identifiable timing signature.
phrase and the matched-filter shape used to analyze it. The default timeline is
a stateless 16-phrase cycle. Phrase `N` is generated directly from `N`, with no
registry or remembered random state. Each phrase is a spread-out asymmetric
motif: six short chirplets over about 1.38 seconds, with nonuniform gaps and
high-frequency bands. A new phrase fires every 3.25 seconds through Aquarium
audio. The point is not ornament. The timing code is carried by both frequency
and rhythm, so it behaves more like a small birdsong signature than a repeated
sweep. The indexed phrase sequence has lower ambiguity when remote feeds add
network/encoding latency.
`MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the
loopback sample-rate timeline, projects loopback and candidate windows through
the same phrase, contrast-normalizes the resulting energy traces, and estimates
current delay against `loopback-scarlett-speakers`. Reports carry both rounded
integer delay and fractional delay from parabolic peak interpolation over the
chirplet correlation peak.
current delay against `loopback-scarlett-speakers`. The lag search is wide
enough for ordinary remote/network delay, not just local speaker-to-mic delay.
Reports carry both rounded integer delay and fractional delay from parabolic
peak interpolation over the chirplet correlation peak.
`MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned
mono frame: loopback is always channel zero, and other channels enter only when
@ -116,6 +122,14 @@ watching delay change over time. Per-band energy over many phrases gives the
normalization curve. The important constraint is that all three measurements
must be tied to the same emitted phrase, not three separately invented probes.
The phrase is no longer static. Mimir currently rotates a deterministic
16-phrase cycle and the analyzer searches that cycle. That is enough for a
first timeline fingerprint and ordinary remote/network delay. For higher-latency
or recorded/replayed sources, the next extension should lengthen the code or
make the phrase index explicitly recoverable from the chirplet rhythm so Mimir
can distinguish phrase N from phrase N+16 without relying only on wall-clock
arrival.
Next, replace the diagnostic bridge with native audio capture workers that
append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state,
delay estimates, and stem routing in Aquarium UI.

View file

@ -42,8 +42,11 @@ a named invariant that the native runtime cannot protect yet.
- `native/probes/wasapi_audio_cadence` captures WASAPI mic or render-loopback
block metadata and emits `audio-block` JSON events for the diagnostic runtime
adapter.
- `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish calibration
- `MimirChirpletCalibrationPhrase` owns the structured birdsong-like calibration
phrase, PCM rendering, matched timing trace, and per-band response kernels.
The default timeline is a stateless 16-phrase cycle; phrase `N` is generated
from `N`, spreads six chirplets across about 1.38 seconds, and fires every
3.25 seconds to reduce repeated-sweep ambiguity.
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
sample-bearing audio blocks are resampled into the Scarlett loopback timeline
and compared by chirplet-energy delay estimation with fractional peak

View file

@ -91,13 +91,14 @@ not become the synchronized program authority.
The six-microphone path is separate from the bridge. Scarlett speaker loopback
is the current timing authority when calibration chirplets are playing.
`MimirChirpletCalibrationPhrase` owns the harmonic-ish calibration phrase around
8 kHz, 10 kHz, 12 kHz, and 16 kHz, plus the matched timing and response kernels.
Mimir emits that phrase through Aquarium audio, compares mic buffers against
loopback by chirplet-energy delay estimation, and can build a provisional
aligned mono frame for channels that clear the confidence gate. Reports now
carry fractional delay and per-band matched energy. The hub also owns smoothed
per-source sync state with delay-slope/SRO in ppm. Camera mics are
`MimirChirpletCalibrationPhrase` owns the birdsong-like timeline fingerprint:
a stateless 16-phrase cycle where phrase `N` is generated from `N`; each phrase
spreads six asymmetric chirplets across about 1.38 seconds and fires every 3.25
seconds. Mimir emits that phrase sequence through Aquarium audio, compares mic
buffers against loopback by chirplet-energy delay estimation, and can build a
provisional aligned mono frame for channels that clear the confidence gate.
Reports now carry fractional delay and per-band matched energy. The hub also
owns smoothed per-source sync state with delay-slope/SRO in ppm. Camera mics are
spatial/context witnesses; Focusrite devices are dialogue anchors. The current
aligned frame is integer-delay only; fractional delay and the hot resampler
still belong in Faust/native DSP.

View file

@ -63,11 +63,13 @@ Get-Content .\state\evidence.jsonl -Tail 8
640x480@60, and 81.10 fps when the PS3 Eye runs 320x240@187. The regular
Kiyo and second PS3 Eye were absent, so six-camera viability is not proven.
- Current chirplet-backed smoke uses Scarlett speaker loopback as timing
authority. `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish
phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, the Aquarium PCM rendering,
authority. `MimirChirpletCalibrationPhrase` owns the structured birdsong-like
timeline fingerprint: a stateless 16-phrase cycle where phrase `N` is
generated from `N`; each phrase has six asymmetric chirplets over about 1.38
seconds and fires every 3.25 seconds. The phrase owns Aquarium PCM rendering,
matched timing trace, and per-band response hooks. Sync reports now include
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
now tracks smoothed delay and delay-slope/SRO ppm. PS3 Eye audio is
fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and
delay-slope/SRO ppm. PS3 Eye audio is
enumeration/runtime-fragile: one run saw both mic buffers empty while both
cameras were live, then a replug made both PS3 Eye mic buffers emit
480-frame WASAPI blocks again.

View file

@ -11,14 +11,14 @@ namespace Mimir.Runtime;
public sealed class MimirRuntime : IAquariumRuntime
{
private const string DefaultAudioSyncReference = "loopback-scarlett-speakers";
private static readonly MimirChirpletCalibrationPhrase CalibrationPhrase = MimirChirpletCalibrationPhrase.Default;
private readonly LocalCastRuntime visualRuntime;
private readonly MimirSynchronizationHub synchronization;
private readonly AquariumUiDocument ui;
private int lastPollCount;
private float runtimeSeconds;
private float nextCalibrationChirpSeconds = (float)CalibrationPhrase.FirstFireSeconds;
private string? calibrationPhrasePcm16Base64;
private float nextCalibrationChirpSeconds = (float)MimirChirpletCalibrationPhrase.Default.FirstFireSeconds;
private ulong calibrationPhraseIndex;
private readonly Dictionary<ulong, string> calibrationPhraseCache = new();
public MimirRuntime(AquariumRuntimeOptions options)
: this(options, MimirRuntimeConfiguration.Load())
@ -119,7 +119,7 @@ public sealed class MimirRuntime : IAquariumRuntime
panel.Readout("Audio sync", DescribeAudioSync);
panel.Readout("Audio sync state", DescribeAudioSyncState);
panel.Readout("Aligned audio", DescribeAlignedAudio);
panel.Readout("Chirplet reference", () => $"{DefaultAudioSyncReference} every {CalibrationPhrase.IntervalSeconds:0.0}s");
panel.Readout("Chirplet reference", DescribeChirpletReference);
});
}
@ -127,15 +127,35 @@ public sealed class MimirRuntime : IAquariumRuntime
{
while (runtimeSeconds >= nextCalibrationChirpSeconds)
{
var cycleIndex = calibrationPhraseIndex % MimirChirpletCalibrationPhrase.PhraseCycleLength;
var phrase = MimirChirpletCalibrationPhrase.ForIndex(cycleIndex);
visualRuntime.Audio.EnqueuePcm16Base64(
calibrationPhrasePcm16Base64 ??= CalibrationPhrase.RenderPcm16Base64(),
CalibrationPhrase.SampleRate,
Pcm16Base64ForPhrase(cycleIndex, phrase),
phrase.SampleRate,
channels: 1,
gain: 1.0f);
nextCalibrationChirpSeconds += (float)CalibrationPhrase.IntervalSeconds;
calibrationPhraseIndex++;
nextCalibrationChirpSeconds += (float)phrase.IntervalSeconds;
}
}
private string Pcm16Base64ForPhrase(ulong cycleIndex, MimirChirpletCalibrationPhrase phrase)
{
if (!calibrationPhraseCache.TryGetValue(cycleIndex, out var pcm16Base64))
{
pcm16Base64 = phrase.RenderPcm16Base64();
calibrationPhraseCache[cycleIndex] = pcm16Base64;
}
return pcm16Base64;
}
private string DescribeChirpletReference()
{
var phrase = MimirChirpletCalibrationPhrase.Default;
return $"{DefaultAudioSyncReference} every {phrase.IntervalSeconds:0.00}s code {calibrationPhraseIndex % MimirChirpletCalibrationPhrase.PhraseCycleLength}/{MimirChirpletCalibrationPhrase.PhraseCycleLength}";
}
private string DescribeBuffers()
{
return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer));

View file

@ -5,9 +5,8 @@ namespace Mimir.Runtime.Synchronization;
public sealed class MimirAudioSynchronizationAnalyzer
{
private const int MaxWindowSamples = 48_000 * 5;
private const int MaxLagSamples = 2_400;
private const int MaxLagSamples = 48_000;
private const int ChirpletHopSamples = 16;
private readonly MimirChirpletCalibrationPhrase chirpletPhrase = MimirChirpletCalibrationPhrase.Default;
public IReadOnlyList<MimirAudioSynchronizationReport> Analyze(
IEnumerable<MimirRollingStreamBuffer> buffers,
@ -66,8 +65,8 @@ public sealed class MimirAudioSynchronizationAnalyzer
var referenceWindow = referenceSamples.AsSpan(^compared..);
var candidateWindow = candidateSamples.AsSpan(^compared..);
var referenceSync = chirpletPhrase.BuildEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples);
var candidateSync = chirpletPhrase.BuildEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples);
var referenceSync = MimirChirpletCalibrationPhrase.BuildTimelineEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples);
var candidateSync = MimirChirpletCalibrationPhrase.BuildTimelineEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples);
if (referenceSync.Length < 16 || candidateSync.Length < 16)
{
continue;
@ -77,7 +76,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
var (delayHops, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag);
var fractionalDelaySamples = delayHops * ChirpletHopSamples;
var delaySamples = (int)Math.Round(fractionalDelaySamples);
var bandResponses = chirpletPhrase.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate);
var bandResponses = MimirChirpletCalibrationPhrase.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate);
reports.Add(new MimirAudioSynchronizationReport(
reference.Descriptor.SourceId,
buffer.Descriptor.SourceId,

View file

@ -16,6 +16,7 @@ public sealed record MimirChirpletBandResponse(
public sealed class MimirChirpletCalibrationPhrase
{
public const int PhraseCycleLength = 16;
private readonly float[] kernel;
private readonly IReadOnlyList<float[]> toneKernels;
@ -38,18 +39,46 @@ public sealed class MimirChirpletCalibrationPhrase
toneKernels = tones.Select(tone => RenderToneKernel(tone, sampleRate, DurationSeconds)).ToArray();
}
public static MimirChirpletCalibrationPhrase Default { get; } = new(
public static MimirChirpletCalibrationPhrase Default { get; } = ForIndex(0);
public static IReadOnlyList<MimirChirpletCalibrationPhrase> AnalysisCycle { get; } =
Enumerable.Range(0, PhraseCycleLength)
.Select(index => ForIndex((ulong)index))
.ToArray();
public static MimirChirpletCalibrationPhrase ForIndex(ulong phraseIndex) => new(
sampleRate: 48_000,
intervalSeconds: 1.5,
firstFireSeconds: 0.5,
gain: 0.125,
tones:
[
new(0.000, 0.055, 7_680.0, 8_320.0),
new(0.085, 0.055, 9_600.0, 10_400.0),
new(0.170, 0.055, 11_520.0, 12_480.0),
new(0.255, 0.055, 15_360.0, 16_000.0),
]);
intervalSeconds: 3.25,
firstFireSeconds: 0.75,
gain: 0.13,
tones: GenerateTones(phraseIndex));
public static float[] BuildTimelineEnergyTrace(ReadOnlySpan<float> samples, int sampleRate, int hopSamples)
{
var traces = new List<float[]>();
foreach (var phrase in AnalysisCycle)
{
var trace = phrase.BuildEnergyTrace(samples, sampleRate, hopSamples);
if (trace.Length > 0)
{
traces.Add(trace);
}
}
if (traces.Count == 0)
{
return [];
}
var length = traces.Min(trace => trace.Length);
var output = new float[length];
for (var index = 0; index < length; index++)
{
output[index] = traces.Max(trace => trace[index]);
}
return output;
}
public int SampleRate { get; }
@ -128,6 +157,55 @@ public sealed class MimirChirpletCalibrationPhrase
return samples;
}
private static IReadOnlyList<MimirChirpletTone> GenerateTones(ulong phraseIndex)
{
var baseTones = new[]
{
new MimirChirpletTone(0.000, 0.070, 7_040.0, 7_680.0, 0.80),
new MimirChirpletTone(0.190, 0.060, 9_600.0, 8_800.0, 0.75),
new MimirChirpletTone(0.410, 0.080, 11_800.0, 12_800.0, 0.90),
new MimirChirpletTone(0.735, 0.065, 8_200.0, 9_000.0, 0.60),
new MimirChirpletTone(0.980, 0.095, 14_500.0, 15_800.0, 0.80),
new MimirChirpletTone(1.310, 0.070, 10_800.0, 10_200.0, 0.70),
};
var tones = new List<MimirChirpletTone>(baseTones.Length);
var previousEnd = -1.0;
for (var index = 0; index < baseTones.Length; index++)
{
var baseTone = baseTones[index];
var seed = Mix(phraseIndex ^ ((ulong)index * 0x9E3779B97F4A7C15UL));
var timeJitter = Unit(seed) * 0.050 - 0.025;
var durationJitter = Unit(seed >> 11) * 0.020 - 0.010;
var bandJitter = Unit(seed >> 22) * 480.0 - 240.0;
var glideScale = 0.88 + Unit(seed >> 33) * 0.24;
var gainScale = 0.85 + Unit(seed >> 44) * 0.30;
var flips = ((seed >> 57) & 1UL) != 0;
var duration = Math.Clamp(baseTone.DurationSeconds + durationJitter, 0.045, 0.115);
var start = Math.Max(baseTone.StartSeconds + timeJitter, previousEnd + 0.060);
var center = Math.Clamp(baseTone.CenterHz + bandJitter, 6_800.0, 16_200.0);
var halfWidth = Math.Abs(baseTone.EndHz - baseTone.StartHz) * 0.5 * glideScale;
var low = Math.Clamp(center - halfWidth, 6_700.0, 16_400.0);
var high = Math.Clamp(center + halfWidth, 6_700.0, 16_400.0);
tones.Add(flips
? new MimirChirpletTone(start, duration, high, low, baseTone.Gain * gainScale)
: new MimirChirpletTone(start, duration, low, high, baseTone.Gain * gainScale));
previousEnd = start + duration;
}
return tones;
}
private static ulong Mix(ulong value)
{
value += 0x9E3779B97F4A7C15UL;
value = (value ^ (value >> 30)) * 0xBF58476D1CE4E5B9UL;
value = (value ^ (value >> 27)) * 0x94D049BB133111EBUL;
return value ^ (value >> 31);
}
private static double Unit(ulong value) =>
(value & ((1UL << 53) - 1)) / (double)(1UL << 53);
private static float[] RenderToneKernel(MimirChirpletTone tone, int sampleRate, double phraseDurationSeconds)
{
var frameCount = Math.Max(1, (int)Math.Ceiling(phraseDurationSeconds * sampleRate));

View file

@ -36,3 +36,4 @@
{"ts":"2026-05-22T12:38:53+01:00","type":"implementation","status":"ok","note":"Documented the chirplet calibration model and started the structured implementation. MimirChirpletCalibrationPhrase now owns a harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, PCM rendering for Aquarium emission, matched chirplet timing traces, and per-band matched-energy response hooks. MimirRuntime emits that shared phrase instead of a separate hard-coded sweep, and MimirAudioSynchronizationAnalyzer uses the same phrase for loopback-referenced delay analysis with 16-sample hops and parabolic fractional peak interpolation. BufferSmoke output now includes fractionalDelaySamples and per-band energy."}
{"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."}
{"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."}
{"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."}

View file

@ -66,14 +66,15 @@ accepted_design:
final local-device hot path.
- The diagnostic audio path can optionally carry base64 float sample payloads
for sync smoke tests. `MimirChirpletCalibrationPhrase` owns the structured
harmonic-ish calibration phrase, PCM rendering, matched timing kernel, and
per-band response kernels. `MimirAudioSynchronizationAnalyzer` now treats
`loopback-scarlett-speakers` as timing authority, resamples candidate mic
windows into that timeline, projects loopback and mic windows through the
emitted chirplet phrase, and estimates current delay from chirplet-energy
traces at a shared timestamp edge. Positive delay means the candidate mic is
late relative to loopback. Reports include rounded delay, fractional delay,
confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
birdsong-like timeline fingerprint: a stateless 16-phrase cycle where
phrase `N` is generated from `N`, plus PCM rendering, matched timing kernel,
and per-band response kernels. `MimirAudioSynchronizationAnalyzer` now
treats `loopback-scarlett-speakers` as timing authority, resamples candidate
mic windows into that timeline, projects loopback and mic windows through
the emitted chirplet phrase cycle, and estimates current delay from
chirplet-energy traces at a shared timestamp edge. Positive delay means the
candidate mic is late relative to loopback. Reports include rounded delay,
fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
owns the smoothed per-source belief: latest delay, smoothed delay,
confidence, per-band response evidence, and delay-slope/SRO estimate in
ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first