Decode chirplet timeline symbols

This commit is contained in:
Meta Vens 2026-05-22 13:51:49 +01:00
commit 34bd9d7c8c
11 changed files with 711 additions and 147 deletions

View file

@ -50,21 +50,23 @@ produced zero WASAPI packets until that Eye was unplugged and replugged.
The full probe runtime config now enables sample-bearing blocks for every local
audio source. `MimirChirpletTimeline` owns the emitted calibration stream and
the matched-filter shape used to analyze it. The default timeline is
deterministic and unbounded: event `N` is generated directly from `N`, with no
registry, remembered random state, or repeat cycle. Mimir queues half-second PCM
segments ahead of the audio cursor, and each segment contains short asymmetric
chirplets chosen from a harmonic high-frequency vocabulary. The point is not
ornament. The timing code is carried by both frequency and rhythm, so it behaves
more like a quiet birdsong texture than a repeated sweep.
the matched-filter shape used to analyze it. The default timeline is a
deterministic order-3 de Bruijn symbol sequence over 32 chirp symbols. Any three
consecutive correctly detected symbols identify the event index inside the
current operating horizon, so a receiver can place its audio window on the
canonical timeline without being handed Mimir's runtime clock. Mimir queues
half-second PCM segments ahead of the audio cursor, and each segment contains
short asymmetric chirplets chosen from separated high-frequency slots. The point
is not ornament. The timing code is carried by both frequency and rhythm, so it
behaves more like a quiet birdsong texture than a repeated sweep.
`MimirAudioSynchronizationAnalyzer` resamples candidate mic windows into the
loopback sample-rate timeline, projects loopback and candidate windows through a
continuous chirplet atom bank, contrast-normalizes the resulting energy traces,
and estimates current delay against `loopback-scarlett-speakers`. The lag search
is wide enough for ordinary remote/network delay, not just local speaker-to-mic
delay. Reports carry both rounded integer delay and fractional delay from
parabolic peak interpolation over the chirplet correlation peak.
loopback sample-rate timeline, uses cheap coarse chirplet energy for acquisition,
then lets the symbol decoder refine delay from matched event indices. The lag
search is wide enough for ordinary remote/network delay, not just local
speaker-to-mic delay. Reports carry rounded integer delay, fractional delay from
parabolic peak interpolation, and the count/confidence of timeline-symbol events
used to refine the report.
`MimirSynchronizationHub.BuildAlignedAudioFrame` returns a provisional aligned
mono frame: loopback is always channel zero, and other channels enter only when
@ -128,6 +130,14 @@ watching delay change over time. Per-band energy over the same stream gives the
normalization curve. The important constraint is that all three measurements
must be tied to the same emitted timeline, not three separately invented probes.
The symbol layer is intentionally blunt. Chirp symbols use separated frequency
slots instead of subtle variants sharing the same center frequency, because a
wrong symbol is worse than a noisy delay. A synthetic runtime check rendered two
seconds of canonical timeline audio and decoded events 0, 1, and 2 back to a
window start within about 1.5 ms. Real device runs still depend on loopback
capture staying live; the local Scarlett loopback has intermittently stopped
advancing during short headless sniffs.
Next, replace the diagnostic bridge with native audio capture workers that
append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state,
delay estimates, and stem routing in Aquarium UI.

View file

@ -44,12 +44,14 @@ a named invariant that the native runtime cannot protect yet.
adapter.
- `MimirChirpletTimeline` owns the structured birdsong-like calibration stream,
PCM segment rendering, matched timing trace, and per-band response kernels.
The default timeline is deterministic and unbounded: chirplet event `N` is
generated from `N`, with no repeat cycle or remembered random state.
The default timeline is an order-3 de Bruijn symbol sequence over 32 separated
chirp symbols, so any three consecutive correctly detected symbols identify a
timeline event inside the current operating horizon.
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
sample-bearing audio blocks are resampled into the Scarlett loopback timeline
and compared by chirplet-energy delay estimation with fractional peak
interpolation.
interpolation. When timeline symbols are decoded, matched event indices refine
the delay estimate and are exposed in telemetry.
- `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
provisional aligned mono frame for loopback-referenced audio channels that
clear the confidence gate.

View file

@ -91,12 +91,13 @@ not become the synchronized program authority.
The six-microphone path is separate from the bridge. Scarlett speaker loopback
is the current timing authority when calibration chirplets are playing.
`MimirChirpletTimeline` owns the birdsong-like timeline fingerprint: a
deterministic unbounded chirplet stream where event `N` is generated from `N`,
with no repeat cycle or remembered random state. Mimir continuously queues that
timeline through Aquarium audio, compares mic buffers against loopback by
chirplet-energy delay estimation, and can build a provisional aligned mono
frame for channels that clear the confidence gate.
`MimirChirpletTimeline` owns the birdsong-like timeline fingerprint: an order-3
de Bruijn sequence over 32 separated chirp symbols. Any three consecutive
correctly detected symbols identify the event index inside the current operating
horizon. Mimir continuously queues that timeline through Aquarium audio, uses
coarse chirplet energy for acquisition, refines delay from matched event
indices when the symbol decoder succeeds, and can build a provisional aligned
mono frame for channels that clear the confidence gate.
Reports now carry fractional delay and per-band matched energy. The hub also
owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime`
runs the analysis online and can print live telemetry with

View file

@ -64,10 +64,11 @@ Get-Content .\state\evidence.jsonl -Tail 8
Kiyo and second PS3 Eye were absent, so six-camera viability is not proven.
- Current chirplet-backed smoke uses Scarlett speaker loopback as timing
authority. `MimirChirpletTimeline` owns the structured birdsong-like timeline
fingerprint: deterministic unbounded chirplet events where event `N` is
generated from `N`, with no repeat cycle or remembered random state. The
timeline owns Aquarium PCM segment rendering, matched timing trace, and
per-band response hooks. Sync reports now include
fingerprint: an order-3 de Bruijn sequence over 32 separated chirp symbols,
so any three consecutive correctly detected symbols identify the event index
inside the current operating horizon. The timeline owns Aquarium PCM segment
rendering, coarse acquisition, symbol/event decoding, and per-band response
hooks. Sync reports now include
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing
proves Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and

View file

@ -164,7 +164,7 @@ public sealed class MimirRuntime : IAquariumRuntime
return;
}
lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference);
lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds);
nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds;
}
@ -184,7 +184,7 @@ public sealed class MimirRuntime : IAquariumRuntime
foreach (var report in lastAudioSynchronizationReports.OrderBy(report => report.SourceId, StringComparer.Ordinal))
{
Console.WriteLine(
$"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000}");
$"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} timelineEvents={report.TimelineMatchedEvents} timelineConfidence={report.TimelineConfidence:0.000}");
}
foreach (var state in states)
@ -229,10 +229,10 @@ public sealed class MimirRuntime : IAquariumRuntime
private string DescribeAudioSync()
{
var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference);
var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds);
return reports.Count == 0
? "no payload windows"
: string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00}"));
: string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00} events={report.TimelineMatchedEvents}"));
}
private string DescribeAlignedAudio()

View file

@ -4,13 +4,14 @@ namespace Mimir.Runtime.Synchronization;
public sealed class MimirAudioSynchronizationAnalyzer
{
private const int MaxWindowSamples = 48_000 * 5;
private const int MaxWindowSamples = 48_000 * 2;
private const int MaxLagSamples = 48_000;
private const int ChirpletHopSamples = 16;
public IReadOnlyList<MimirAudioSynchronizationReport> Analyze(
IEnumerable<MimirRollingStreamBuffer> buffers,
string referenceSourceId)
string referenceSourceId,
double approximateTimelineSeconds)
{
var audioBuffers = buffers
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio)
@ -28,6 +29,16 @@ public sealed class MimirAudioSynchronizationAnalyzer
return [];
}
var referenceSamples = ExtractMonoWindow(reference, out var referenceBlock);
if (referenceSamples.Length == 0 ||
referenceBlock == null ||
referenceBlock.SampleFormat != MimirAudioSampleFormat.Float32)
{
return [];
}
MimirChirpletTimelinePlacement? referencePlacement = null;
var reports = new List<MimirAudioSynchronizationReport>();
foreach (var buffer in audioBuffers)
{
@ -37,20 +48,13 @@ public sealed class MimirAudioSynchronizationAnalyzer
}
var commonEndNs = Math.Min(referenceLatest.Value.TimestampNs, buffer.Latest?.TimestampNs ?? 0);
var referenceSamples = ExtractMonoWindow(reference, out var referenceBlock, commonEndNs);
var candidateSamples = ExtractMonoWindow(buffer, out var candidateBlock, commonEndNs);
var candidateSamples = ExtractMonoWindow(buffer, out var candidateBlock);
if (candidateSamples.Length == 0 || candidateBlock == null)
{
continue;
}
if (referenceSamples.Length == 0 || referenceBlock == null)
{
continue;
}
if (candidateBlock.SampleFormat != MimirAudioSampleFormat.Float32 ||
referenceBlock.SampleFormat != MimirAudioSampleFormat.Float32)
if (candidateBlock.SampleFormat != MimirAudioSampleFormat.Float32)
{
continue;
}
@ -65,6 +69,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
var referenceWindow = referenceSamples.AsSpan(^compared..);
var candidateWindow = candidateSamples.AsSpan(^compared..);
var referenceWindowStartSample = referenceSamples.Length - compared;
var referenceSync = MimirChirpletTimeline.Default.BuildTimelineEnergyTrace(referenceWindow, referenceBlock.SampleRate, ChirpletHopSamples);
var candidateSync = MimirChirpletTimeline.Default.BuildTimelineEnergyTrace(candidateWindow, referenceBlock.SampleRate, ChirpletHopSamples);
if (referenceSync.Length < 16 || candidateSync.Length < 16)
@ -75,6 +80,38 @@ public sealed class MimirAudioSynchronizationAnalyzer
var maxLag = Math.Min(MaxLagSamples / ChirpletHopSamples, Math.Min(referenceSync.Length, candidateSync.Length) / 2);
var (delayHops, confidence) = EstimateDelay(referenceSync, candidateSync, maxLag);
var fractionalDelaySamples = delayHops * ChirpletHopSamples;
var timelineMatchedEvents = 0;
var timelineConfidence = 0.0;
if (confidence >= 0.03 && double.IsFinite(approximateTimelineSeconds))
{
referencePlacement ??= MimirChirpletTimeline.Default.DecodeReferenceWindow(
referenceSamples,
referenceBlock.SampleRate,
approximateTimelineSeconds);
var adjustedReferenceObservations = referencePlacement.Observations
.Where(observation =>
observation.SampleOffset >= referenceWindowStartSample &&
observation.SampleOffset < referenceWindowStartSample + compared)
.Select(observation => observation with
{
SampleOffset = observation.SampleOffset - referenceWindowStartSample,
})
.ToArray();
var candidateObservations = MimirChirpletTimeline.Default.ObserveKnownEvents(
candidateWindow,
referenceBlock.SampleRate,
adjustedReferenceObservations,
fractionalDelaySamples);
var eventFit = FitDelayFromEvents(adjustedReferenceObservations, candidateObservations, referenceBlock.SampleRate);
if (eventFit.MatchedEvents >= 3)
{
fractionalDelaySamples = eventFit.DelaySamples;
timelineMatchedEvents = eventFit.MatchedEvents;
timelineConfidence = eventFit.Confidence;
confidence = Math.Clamp((confidence * 0.35) + (eventFit.Confidence * 0.65), 0.0, 1.0);
}
}
var delaySamples = (int)Math.Round(fractionalDelaySamples);
var bandResponses = MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate);
reports.Add(new MimirAudioSynchronizationReport(
@ -89,7 +126,9 @@ public sealed class MimirAudioSynchronizationAnalyzer
commonEndNs,
compared,
reference.Latest?.Sequence ?? 0,
buffer.Latest?.Sequence ?? 0));
buffer.Latest?.Sequence ?? 0,
timelineMatchedEvents,
timelineConfidence));
}
return reports;
@ -115,7 +154,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
return null;
}
var reports = Analyze(audioBuffers.Values, referenceSourceId)
var reports = Analyze(audioBuffers.Values, referenceSourceId, approximateTimelineSeconds: double.PositiveInfinity)
.Where(report => report.Confidence >= minimumConfidence)
.ToDictionary(report => report.SourceId, StringComparer.Ordinal);
var alignedReports = reports.Values
@ -314,6 +353,51 @@ public sealed class MimirAudioSynchronizationAnalyzer
return (bestLag + offset, Math.Max(0.0, Math.Min(1.0, best)));
}
private static (double DelaySamples, double Confidence, int MatchedEvents) FitDelayFromEvents(
IReadOnlyList<MimirChirpletEventObservation> referenceObservations,
IReadOnlyList<MimirChirpletEventObservation> candidateObservations,
int sampleRate)
{
if (referenceObservations.Count == 0 || candidateObservations.Count == 0)
{
return (0.0, 0.0, 0);
}
var candidateByEvent = candidateObservations.ToDictionary(observation => observation.EventIndex);
var delays = new List<(double Delay, double Weight)>();
foreach (var reference in referenceObservations)
{
if (!candidateByEvent.TryGetValue(reference.EventIndex, out var candidate))
{
continue;
}
var weight = Math.Sqrt(Math.Max(0.0, reference.Energy) * Math.Max(0.0, candidate.Energy));
delays.Add((candidate.SampleOffset - reference.SampleOffset, weight));
}
if (delays.Count < 3)
{
return (0.0, 0.0, delays.Count);
}
var ordered = delays.OrderBy(pair => pair.Delay).ToArray();
var trim = ordered.Length >= 8 ? ordered.Length / 8 : 0;
var kept = ordered.Skip(trim).Take(ordered.Length - trim * 2).ToArray();
var totalWeight = kept.Sum(pair => pair.Weight);
if (totalWeight <= 1.0e-12)
{
return (0.0, 0.0, kept.Length);
}
var delay = kept.Sum(pair => pair.Delay * pair.Weight) / totalWeight;
var meanAbsoluteError = kept.Sum(pair => Math.Abs(pair.Delay - delay) * pair.Weight) / totalWeight;
var scatterConfidence = 1.0 / (1.0 + meanAbsoluteError / Math.Max(1.0, sampleRate * 0.002));
var countConfidence = Math.Clamp(kept.Length / 12.0, 0.0, 1.0);
var energyConfidence = Math.Clamp(totalWeight / kept.Length, 0.0, 1.0);
return (delay, scatterConfidence * 0.55 + countConfidence * 0.25 + energyConfidence * 0.20, kept.Length);
}
private static double ParabolicPeakOffset(double left, double center, double right)
{
if (!double.IsFinite(left) || !double.IsFinite(center) || !double.IsFinite(right))

View file

@ -12,4 +12,6 @@ public sealed record MimirAudioSynchronizationReport(
long AnalysisTimestampNs,
int ComparedSamples,
ulong ReferenceSequence,
ulong SourceSequence);
ulong SourceSequence,
int TimelineMatchedEvents = 0,
double TimelineConfidence = 0.0);

View file

@ -14,44 +14,57 @@ public sealed record MimirChirpletBandResponse(
double CenterHz,
double Energy);
public sealed record MimirChirpletTimelineEvent(
ulong Index,
int SymbolId,
double StartSeconds,
MimirChirpletTone Tone);
public sealed record MimirChirpletEventObservation(
ulong EventIndex,
int SymbolId,
double TimelineSeconds,
double SampleOffset,
double Energy);
public sealed record MimirChirpletTimelinePlacement(
double TimelineSecondsAtWindowStart,
double Confidence,
IReadOnlyList<MimirChirpletEventObservation> Observations);
public sealed class MimirChirpletTimeline
{
public const int SampleRate = 48_000;
public const double SegmentSeconds = 0.5;
public const double QueueLeadSeconds = 1.0;
public const int SymbolCount = 32;
public const int TimelineOrder = 3;
public static readonly int TimelinePeriod = (int)Math.Pow(SymbolCount, TimelineOrder);
private const double FirstEventSeconds = 0.08;
private const double EventStrideSeconds = 0.118;
private const double MaxEventDurationSeconds = 0.110;
private const double Gain = 0.095;
private const double MaxEventDurationSeconds = 0.082;
private const double Gain = 0.090;
private static readonly MimirChirpletTone[] AnalysisAtoms =
private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder));
private static readonly Dictionary<int, int> TripleToIndex = BuildTripleIndex(TimelineSymbols);
private static readonly MimirChirpletTone[] SymbolTones = BuildSymbolTones();
private static readonly MimirChirpletTone[] CoarseTones =
[
new(0.0, 0.050, 6_900.0, 7_500.0, 0.70),
new(0.0, 0.055, 8_000.0, 8_900.0, 0.75),
new(0.0, 0.060, 9_600.0, 8_800.0, 0.75),
new(0.0, 0.060, 10_600.0, 11_700.0, 0.85),
new(0.0, 0.065, 12_400.0, 13_500.0, 0.90),
new(0.0, 0.070, 14_400.0, 15_800.0, 0.80),
new(0.0, 0.048, 6_700.0, 7_200.0, 0.75),
new(0.0, 0.052, 7_700.0, 8_400.0, 0.80),
new(0.0, 0.056, 8_900.0, 9_800.0, 0.85),
new(0.0, 0.060, 10_300.0, 11_400.0, 0.85),
new(0.0, 0.064, 12_100.0, 13_500.0, 0.90),
new(0.0, 0.068, 14_200.0, 15_900.0, 0.80),
];
private static readonly double[] HarmonicRatios =
[
1.0,
9.0 / 8.0,
5.0 / 4.0,
4.0 / 3.0,
3.0 / 2.0,
5.0 / 3.0,
15.0 / 8.0,
2.0,
9.0 / 4.0,
5.0 / 2.0,
];
private readonly IReadOnlyList<float[]> atomKernels;
private readonly IReadOnlyList<float[]> symbolKernels;
private readonly IReadOnlyList<float[]> coarseKernels;
private MimirChirpletTimeline()
{
atomKernels = AnalysisAtoms.Select(atom => RenderToneKernel(atom, SampleRate)).ToArray();
symbolKernels = SymbolTones.Select(symbol => RenderToneKernel(symbol, SampleRate)).ToArray();
coarseKernels = CoarseTones.Select(tone => RenderToneKernel(tone, SampleRate)).ToArray();
}
public static MimirChirpletTimeline Default { get; } = new();
@ -60,9 +73,9 @@ public sealed class MimirChirpletTimeline
{
var segmentStartSeconds = segmentIndex * SegmentSeconds;
var samples = new float[(int)Math.Round(SegmentSeconds * SampleRate)];
foreach (var tone in TonesOverlapping(segmentStartSeconds, SegmentSeconds))
foreach (var timelineEvent in EventsOverlapping(segmentStartSeconds, SegmentSeconds))
{
AddTone(samples, SampleRate, tone, segmentStartSeconds);
AddTone(samples, SampleRate, timelineEvent.Tone, segmentStartSeconds);
}
var peak = samples.Select(Math.Abs).DefaultIfEmpty(0.0f).Max();
@ -96,37 +109,7 @@ public sealed class MimirChirpletTimeline
public float[] BuildTimelineEnergyTrace(ReadOnlySpan<float> samples, int sampleRate, int hopSamples)
{
var traces = new List<float[]>();
for (var index = 0; index < AnalysisAtoms.Length; index++)
{
var atom = AnalysisAtoms[index];
var kernel = sampleRate == SampleRate ? atomKernels[index] : RenderToneKernel(atom, sampleRate);
var trace = BuildMatchedEnergyTrace(samples, kernel, hopSamples);
if (trace.Length > 0)
{
traces.Add(trace);
}
}
if (traces.Count == 0)
{
return [];
}
var length = traces.Min(trace => trace.Length);
var output = new float[length];
for (var frame = 0; frame < length; frame++)
{
var best = 0.0f;
foreach (var trace in traces)
{
best = Math.Max(best, trace[frame]);
}
output[frame] = best;
}
return ContrastNormalize(output);
return ContrastNormalize(BuildCoarseEnergyTrace(samples, sampleRate, hopSamples));
}
public IReadOnlyList<MimirChirpletBandResponse> EstimateBandResponse(ReadOnlySpan<float> samples, int sampleRate)
@ -136,63 +119,445 @@ public sealed class MimirChirpletTimeline
return [];
}
var responses = new List<MimirChirpletBandResponse>(AnalysisAtoms.Length);
for (var index = 0; index < AnalysisAtoms.Length; index++)
var responses = new List<MimirChirpletBandResponse>(SymbolTones.Length);
for (var symbol = 0; symbol < SymbolTones.Length; symbol++)
{
var atom = AnalysisAtoms[index];
var kernel = sampleRate == SampleRate ? atomKernels[index] : RenderToneKernel(atom, sampleRate);
var tone = SymbolTones[symbol];
var kernel = sampleRate == SampleRate ? symbolKernels[symbol] : RenderToneKernel(tone, sampleRate);
var energy = MaxMatchedEnergy(samples, kernel, Math.Max(1, sampleRate / 1_000));
responses.Add(new MimirChirpletBandResponse(atom.CenterHz, energy));
responses.Add(new MimirChirpletBandResponse(tone.CenterHz, energy));
}
return responses;
}
private static IEnumerable<MimirChirpletTone> TonesOverlapping(double startSeconds, double durationSeconds)
public MimirChirpletTimelinePlacement DecodeReferenceWindow(
ReadOnlySpan<float> samples,
int sampleRate,
double approximateWindowEndTimelineSeconds)
{
if (samples.Length == 0)
{
return new MimirChirpletTimelinePlacement(0.0, 0.0, []);
}
var observed = DetectSymbolEvents(samples, sampleRate, Math.Max(1, sampleRate / 200));
var placement = PlaceObservedSymbols(observed, sampleRate);
if (placement.Observations.Count >= 3 || !double.IsFinite(approximateWindowEndTimelineSeconds))
{
return placement;
}
var windowSeconds = samples.Length / (double)sampleRate;
var approximateStartSeconds = Math.Max(0.0, approximateWindowEndTimelineSeconds - windowSeconds);
var fallback = new List<MimirChirpletEventObservation>();
foreach (var timelineEvent in EventsOverlapping(approximateStartSeconds - 0.25, windowSeconds + 0.5))
{
var predictedSample = (timelineEvent.StartSeconds - approximateStartSeconds) * sampleRate;
var observation = ObserveEvent(samples, sampleRate, timelineEvent, predictedSample, searchRadiusSamples: Math.Max(8, sampleRate / 2));
if (observation != null)
{
fallback.Add(observation);
}
}
if (fallback.Count < 3)
{
return new MimirChirpletTimelinePlacement(approximateStartSeconds, 0.0, fallback);
}
return FitTimelinePlacement(fallback, sampleRate);
}
public IReadOnlyList<MimirChirpletEventObservation> ObserveKnownEvents(
ReadOnlySpan<float> samples,
int sampleRate,
IEnumerable<MimirChirpletEventObservation> referenceObservations,
double coarseDelaySamples)
{
var observations = new List<MimirChirpletEventObservation>();
foreach (var reference in referenceObservations)
{
var timelineEvent = EventForIndex(reference.EventIndex);
var predictedSample = reference.SampleOffset + coarseDelaySamples;
var observation = ObserveEvent(samples, sampleRate, timelineEvent, predictedSample, searchRadiusSamples: Math.Max(16, sampleRate / 10));
if (observation != null)
{
observations.Add(observation);
}
}
return observations;
}
public MimirChirpletTimelineEvent EventForIndex(ulong eventIndex)
{
var symbolId = SymbolForEvent(eventIndex);
var tone = SymbolTones[symbolId] with { StartSeconds = EventStartSeconds(eventIndex) };
return new MimirChirpletTimelineEvent(eventIndex, symbolId, tone.StartSeconds, tone);
}
public IReadOnlyList<MimirChirpletTimelineEvent> EventsOverlapping(double startSeconds, double durationSeconds)
{
var endSeconds = startSeconds + durationSeconds;
var firstIndex = Math.Max(0, (long)Math.Floor((startSeconds - FirstEventSeconds - MaxEventDurationSeconds) / EventStrideSeconds) - 2);
var lastIndex = Math.Max(firstIndex, (long)Math.Ceiling((endSeconds - FirstEventSeconds) / EventStrideSeconds) + 2);
var events = new List<MimirChirpletTimelineEvent>((int)(lastIndex - firstIndex + 1));
for (var eventIndex = firstIndex; eventIndex <= lastIndex; eventIndex++)
{
var tone = ToneForEvent((ulong)eventIndex);
if (tone.StartSeconds < endSeconds && tone.StartSeconds + tone.DurationSeconds > startSeconds)
var timelineEvent = EventForIndex((ulong)eventIndex);
if (timelineEvent.StartSeconds < endSeconds &&
timelineEvent.StartSeconds + timelineEvent.Tone.DurationSeconds > startSeconds)
{
yield return tone;
events.Add(timelineEvent);
}
}
return events;
}
private static MimirChirpletTone ToneForEvent(ulong eventIndex)
private IReadOnlyList<MimirChirpletEventObservation> DetectSymbolEvents(
ReadOnlySpan<float> samples,
int sampleRate,
int hopSamples)
{
var seed = Mix(eventIndex);
var startJitter = Unit(seed) * 0.034 - 0.017;
var duration = 0.045 + Unit(seed >> 10) * 0.052;
var octaveJitter = Unit(seed >> 20) * 0.028 - 0.014;
var glideScale = 0.72 + Unit(seed >> 31) * 0.66;
var gain = 0.72 + Unit(seed >> 42) * 0.34;
var descending = ((seed >> 57) & 1UL) != 0;
var degree = (int)((seed >> 58) % 10);
var center = Math.Clamp(6_800.0 * HarmonicRatios[degree] * (1.0 + octaveJitter), 6_500.0, 16_300.0);
var halfWidth = Math.Clamp(center * 0.035 * glideScale, 220.0, 820.0);
var low = Math.Clamp(center - halfWidth, 6_300.0, 16_500.0);
var high = Math.Clamp(center + halfWidth, 6_300.0, 16_500.0);
var start = FirstEventSeconds + eventIndex * EventStrideSeconds + startJitter;
return descending
? new MimirChirpletTone(start, duration, high, low, gain)
: new MimirChirpletTone(start, duration, low, high, gain);
if (samples.Length == 0)
{
return [];
}
var coarse = ContrastNormalize(BuildEnvelopeEnergyTrace(samples, sampleRate, hopSamples));
if (coarse.Length <= 2)
{
return [];
}
var candidates = new List<MimirChirpletEventObservation>();
for (var frame = 1; frame < coarse.Length - 1; frame++)
{
if (coarse[frame] < 0.055 ||
coarse[frame] < coarse[frame - 1] ||
coarse[frame] < coarse[frame + 1])
{
continue;
}
var observation = ClassifySymbolAt(samples, sampleRate, frame * hopSamples, Math.Max(1, hopSamples));
if (observation != null)
{
candidates.Add(observation);
}
}
return SuppressNearbyDetections(candidates, sampleRate);
}
private static ulong Mix(ulong value)
private MimirChirpletEventObservation? ClassifySymbolAt(
ReadOnlySpan<float> samples,
int sampleRate,
int predictedOffset,
int searchRadiusSamples)
{
value += 0x9E3779B97F4A7C15UL;
value = (value ^ (value >> 30)) * 0xBF58476D1CE4E5B9UL;
value = (value ^ (value >> 27)) * 0x94D049BB133111EBUL;
return value ^ (value >> 31);
var bestSymbol = -1;
var bestOffset = predictedOffset;
var bestEnergy = 0.0;
for (var symbol = 0; symbol < SymbolTones.Length; symbol++)
{
var tone = SymbolTones[symbol];
var kernel = sampleRate == SampleRate ? symbolKernels[symbol] : RenderToneKernel(tone, sampleRate);
if (kernel.Length == 0 || samples.Length < kernel.Length)
{
continue;
}
var start = Math.Max(0, predictedOffset - searchRadiusSamples);
var end = Math.Min(samples.Length - kernel.Length, predictedOffset + searchRadiusSamples);
for (var offset = start; offset <= end; offset += Math.Max(1, searchRadiusSamples / 2))
{
var energy = DirectionalMatchedEnergy(samples.Slice(offset, kernel.Length), kernel);
if (energy > bestEnergy)
{
bestEnergy = energy;
bestSymbol = symbol;
bestOffset = offset;
}
}
}
return bestSymbol < 0 || bestEnergy < 0.055
? null
: new MimirChirpletEventObservation(0, bestSymbol, 0.0, bestOffset, bestEnergy);
}
private static double Unit(ulong value) =>
(value & ((1UL << 53) - 1)) / (double)(1UL << 53);
private static IReadOnlyList<MimirChirpletEventObservation> SuppressNearbyDetections(
IReadOnlyList<MimirChirpletEventObservation> observations,
int sampleRate)
{
if (observations.Count == 0)
{
return [];
}
var minimumSpacingSamples = EventStrideSeconds * sampleRate * 0.55;
var kept = new List<MimirChirpletEventObservation>();
foreach (var observation in observations.OrderByDescending(observation => observation.Energy))
{
if (kept.Any(existing => Math.Abs(existing.SampleOffset - observation.SampleOffset) < minimumSpacingSamples))
{
continue;
}
kept.Add(observation);
}
return kept.OrderBy(observation => observation.SampleOffset).ToArray();
}
private static MimirChirpletTimelinePlacement PlaceObservedSymbols(
IReadOnlyList<MimirChirpletEventObservation> observations,
int sampleRate)
{
if (observations.Count < 3)
{
return new MimirChirpletTimelinePlacement(0.0, 0.0, observations);
}
var best = Array.Empty<MimirChirpletEventObservation>();
var bestConfidence = 0.0;
for (var index = 0; index <= observations.Count - 3; index++)
{
var first = observations[index];
var second = observations[index + 1];
var third = observations[index + 2];
if (!IsConsecutiveByTime(first, second, sampleRate) ||
!IsConsecutiveByTime(second, third, sampleRate))
{
continue;
}
var code = TripleCode(first.SymbolId, second.SymbolId, third.SymbolId);
if (!TripleToIndex.TryGetValue(code, out var eventIndex))
{
continue;
}
var placed = new[]
{
PlaceObservation(first, (ulong)eventIndex),
PlaceObservation(second, (ulong)eventIndex + 1UL),
PlaceObservation(third, (ulong)eventIndex + 2UL),
};
var confidence = placed.Average(observation => observation.Energy);
if (confidence > bestConfidence)
{
bestConfidence = confidence;
best = placed;
}
}
return best.Length == 0
? new MimirChirpletTimelinePlacement(0.0, 0.0, observations)
: FitTimelinePlacement(best, sampleRate);
}
private static bool IsConsecutiveByTime(
MimirChirpletEventObservation first,
MimirChirpletEventObservation second,
int sampleRate)
{
var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate;
return seconds >= EventStrideSeconds * 0.55 && seconds <= EventStrideSeconds * 1.45;
}
private static MimirChirpletEventObservation PlaceObservation(
MimirChirpletEventObservation observation,
ulong eventIndex)
{
var timelineEvent = Default.EventForIndex(eventIndex);
return observation with
{
EventIndex = eventIndex,
SymbolId = timelineEvent.SymbolId,
TimelineSeconds = timelineEvent.StartSeconds,
};
}
private static MimirChirpletTimelinePlacement FitTimelinePlacement(
IReadOnlyList<MimirChirpletEventObservation> observations,
int sampleRate)
{
if (observations.Count == 0)
{
return new MimirChirpletTimelinePlacement(0.0, 0.0, []);
}
var offsets = observations
.Select(observation => observation.TimelineSeconds - observation.SampleOffset / sampleRate)
.ToArray();
var timelineStart = Median(offsets);
var residualSeconds = observations
.Average(observation => Math.Abs(observation.TimelineSeconds - observation.SampleOffset / sampleRate - timelineStart));
var residualConfidence = 1.0 / (1.0 + residualSeconds / 0.004);
var countConfidence = Math.Clamp(observations.Count / 6.0, 0.0, 1.0);
var energyConfidence = Math.Clamp(observations.Average(observation => observation.Energy), 0.0, 1.0);
return new MimirChirpletTimelinePlacement(
timelineStart,
residualConfidence * 0.45 + countConfidence * 0.25 + energyConfidence * 0.30,
observations);
}
private static MimirChirpletEventObservation? ObserveEvent(
ReadOnlySpan<float> samples,
int sampleRate,
MimirChirpletTimelineEvent timelineEvent,
double predictedSample,
int searchRadiusSamples)
{
var kernel = RenderToneKernel(timelineEvent.Tone with { StartSeconds = 0.0 }, sampleRate);
if (kernel.Length == 0 || samples.Length < kernel.Length)
{
return null;
}
var predictedOffset = (int)Math.Round(predictedSample);
var start = Math.Max(0, predictedOffset - searchRadiusSamples);
var end = Math.Min(samples.Length - kernel.Length, predictedOffset + searchRadiusSamples);
if (end < start)
{
return null;
}
var bestOffset = start;
var bestEnergy = 0.0;
var step = Math.Max(1, sampleRate / 4_000);
for (var offset = start; offset <= end; offset += step)
{
var energy = DirectionalMatchedEnergy(samples.Slice(offset, kernel.Length), kernel);
if (energy > bestEnergy)
{
bestEnergy = energy;
bestOffset = offset;
}
}
if (bestEnergy < 0.045)
{
return null;
}
return new MimirChirpletEventObservation(
timelineEvent.Index,
timelineEvent.SymbolId,
timelineEvent.StartSeconds,
bestOffset,
bestEnergy);
}
private static int SymbolForEvent(ulong eventIndex) =>
TimelineSymbols[(int)(eventIndex % (ulong)TimelineSymbols.Length)];
private static double EventStartSeconds(ulong eventIndex) =>
FirstEventSeconds + eventIndex * EventStrideSeconds;
private static MimirChirpletTone[] BuildSymbolTones()
{
var tones = new MimirChirpletTone[SymbolCount];
for (var symbol = 0; symbol < tones.Length; symbol++)
{
var baseHz = 6_500.0 * Math.Pow(16_400.0 / 6_500.0, symbol / (double)(SymbolCount - 1));
var duration = 0.055;
var width = 190.0;
var descending = (symbol & 1) != 0;
var low = Math.Clamp(baseHz - width * 0.5, 6_300.0, 16_200.0);
var high = Math.Clamp(baseHz + width * 0.5, 6_400.0, 16_500.0);
tones[symbol] = descending
? new MimirChirpletTone(0.0, duration, high, low, 0.85)
: new MimirChirpletTone(0.0, duration, low, high, 0.85);
}
return tones;
}
private static int[] BuildDeBruijn(int alphabetSize, int order)
{
var a = new int[alphabetSize * order];
var sequence = new List<int>(TimelinePeriod);
void Db(int t, int p)
{
if (t > order)
{
if (order % p == 0)
{
for (var index = 1; index <= p; index++)
{
sequence.Add(a[index]);
}
}
return;
}
a[t] = a[t - p];
Db(t + 1, p);
for (var j = a[t - p] + 1; j < alphabetSize; j++)
{
a[t] = j;
Db(t + 1, t);
}
}
Db(1, 1);
return sequence.ToArray();
}
private static int[] RotateToDistinctOpening(int[] sequence)
{
for (var index = 0; index < sequence.Length; index++)
{
var first = sequence[index];
var second = sequence[(index + 1) % sequence.Length];
var third = sequence[(index + 2) % sequence.Length];
if (first != second && first != third && second != third)
{
return sequence.Skip(index).Concat(sequence.Take(index)).ToArray();
}
}
return sequence;
}
private static Dictionary<int, int> BuildTripleIndex(IReadOnlyList<int> symbols)
{
var map = new Dictionary<int, int>(symbols.Count);
for (var index = 0; index < symbols.Count; index++)
{
var code = TripleCode(
symbols[index],
symbols[(index + 1) % symbols.Count],
symbols[(index + 2) % symbols.Count]);
map[code] = index;
}
return map;
}
private static int TripleCode(int first, int second, int third) =>
first * SymbolCount * SymbolCount + second * SymbolCount + third;
private static double Median(double[] values)
{
if (values.Length == 0)
{
return 0.0;
}
Array.Sort(values);
var middle = values.Length / 2;
return values.Length % 2 == 0
? (values[middle - 1] + values[middle]) * 0.5
: values[middle];
}
private static float[] RenderToneKernel(MimirChirpletTone tone, int sampleRate)
{
@ -223,23 +588,6 @@ public sealed class MimirChirpletTimeline
}
}
private static float[] BuildMatchedEnergyTrace(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel, int hopSamples)
{
if (samples.Length < kernel.Length || kernel.Length == 0)
{
return [];
}
var output = new float[1 + (samples.Length - kernel.Length) / hopSamples];
for (var frame = 0; frame < output.Length; frame++)
{
var offset = frame * hopSamples;
output[frame] = (float)MatchedEnergy(samples.Slice(offset, kernel.Length), kernel);
}
return output;
}
private static double MatchedEnergy(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel)
{
var count = Math.Min(samples.Length, kernel.Length);
@ -264,6 +612,118 @@ public sealed class MimirChirpletTimeline
return denominator > 1.0e-12 ? Math.Abs(dot) / denominator : 0.0;
}
private static double DirectionalMatchedEnergy(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel)
{
var count = Math.Min(samples.Length, kernel.Length);
if (count == 0)
{
return 0.0;
}
var dot = 0.0;
var sampleEnergy = 0.0;
var kernelEnergy = 0.0;
for (var index = 0; index < count; index++)
{
var sample = samples[index];
var basis = kernel[index];
dot += sample * basis;
sampleEnergy += sample * sample;
kernelEnergy += basis * basis;
}
var denominator = Math.Sqrt(sampleEnergy * kernelEnergy);
return denominator > 1.0e-12 ? Math.Max(0.0, dot / denominator) : 0.0;
}
private float[] BuildCoarseEnergyTrace(ReadOnlySpan<float> samples, int sampleRate, int hopSamples)
{
var traces = new List<float[]>();
for (var index = 0; index < CoarseTones.Length; index++)
{
var tone = CoarseTones[index];
var kernel = sampleRate == SampleRate ? coarseKernels[index] : RenderToneKernel(tone, sampleRate);
var trace = BuildMatchedEnergyTrace(samples, kernel, hopSamples);
if (trace.Length > 0)
{
traces.Add(trace);
}
}
if (traces.Count == 0)
{
return [];
}
var length = traces.Min(trace => trace.Length);
var output = new float[length];
for (var frame = 0; frame < output.Length; frame++)
{
var best = 0.0f;
foreach (var trace in traces)
{
best = Math.Max(best, trace[frame]);
}
output[frame] = best;
}
return output;
}
private static float[] BuildEnvelopeEnergyTrace(ReadOnlySpan<float> samples, int sampleRate, int hopSamples)
{
var windowSamples = Math.Max(1, (int)Math.Round(MaxEventDurationSeconds * sampleRate));
if (samples.Length < windowSamples)
{
return [];
}
var output = new float[1 + (samples.Length - windowSamples) / hopSamples];
var energy = 0.0;
for (var index = 0; index < windowSamples; index++)
{
energy += samples[index] * samples[index];
}
output[0] = (float)Math.Sqrt(energy / windowSamples);
for (var frame = 1; frame < output.Length; frame++)
{
var previousStart = (frame - 1) * hopSamples;
var nextStart = frame * hopSamples;
for (var index = previousStart; index < nextStart; index++)
{
energy -= samples[index] * samples[index];
}
for (var index = previousStart + windowSamples; index < nextStart + windowSamples; index++)
{
energy += samples[index] * samples[index];
}
output[frame] = (float)Math.Sqrt(Math.Max(0.0, energy) / windowSamples);
}
return output;
}
private static float[] BuildMatchedEnergyTrace(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel, int hopSamples)
{
if (samples.Length < kernel.Length || kernel.Length == 0)
{
return [];
}
var output = new float[1 + (samples.Length - kernel.Length) / hopSamples];
for (var frame = 0; frame < output.Length; frame++)
{
var offset = frame * hopSamples;
output[frame] = (float)MatchedEnergy(samples.Slice(offset, kernel.Length), kernel);
}
return output;
}
private static double MaxMatchedEnergy(ReadOnlySpan<float> samples, ReadOnlySpan<float> kernel, int hopSamples)
{
if (samples.Length < kernel.Length || kernel.Length == 0)

View file

@ -62,10 +62,12 @@ public sealed class MimirSynchronizationHub : IDisposable
return $"{video} video / {audio} audio buffers";
}
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronization(string referenceSourceId)
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronization(
string referenceSourceId,
double approximateTimelineSeconds = double.PositiveInfinity)
{
ObjectDisposedException.ThrowIf(disposed, this);
var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId);
var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId, approximateTimelineSeconds);
audioSynchronizationState.Update(reports);
return reports;
}

View file

@ -39,3 +39,4 @@
{"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."}
{"ts":"2026-05-22T13:12:32+01:00","type":"verification","status":"partial","note":"Actual Mimir.App runtime test with MIMIR_RUNTIME_CONFIG=config/mimir-runtime.probes.example.json and MIMIR_SYNC_TELEMETRY_SECONDS enabled proved Aquarium WASAPI output wakes the Scarlett loopback in the real app path. Headless run showed loopbackCount rising to 57 with mic buffers live; visible-window run showed loopbackCount 77 with Focusrite/Kiyo/Kiyo Pro/PS3 Eye mic buffers live. The runtime now updates sync analysis on a 0.5s cadence and telemetry prints loopback, reports, states, and audio buffer counts. No confident acoustic sync state locked in these app runs: reports were absent until enough phrase audio accumulated, then a live report appeared with confidence 0.000. Treat this as online emission/capture proven, online acoustic lock not yet solved."}
{"ts":"2026-05-22T13:34:00+01:00","type":"implementation","status":"ok","note":"Replaced the discrete 16-phrase chirplet cycle with MimirChirpletTimeline, a deterministic unbounded chirplet stream. MimirRuntime now continuously queues uncached 0.5-second PCM segments through Aquarium audio with 1 second of lead, and MimirAudioSynchronizationAnalyzer extracts continuous chirplet atom energy traces from loopback and mic windows rather than searching a finite phrase cycle. Short headless Mimir.App telemetry runs proved the actual runtime path now locks online: loopback and all six audio buffers were live, reports were produced, and sync states appeared for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Delay estimates are still jumpy enough that the fractional-delay/SRO actuator should wait for better state filtering."}
{"ts":"2026-05-22T14:10:00+01:00","type":"implementation","status":"partial","note":"Changed MimirChirpletTimeline from anonymous deterministic texture to an explicit order-3 de Bruijn symbol timeline over 32 separated chirp symbols. Any three consecutive correctly detected symbols identify a timeline event within the current operating horizon; reports now expose timelineMatchedEvents and timelineConfidence when event-index refinement is used. A synthetic check rendered two seconds of canonical timeline audio and decoded events 0, 1, and 2 back to a window start of -0.0015 seconds. Real headless app sniff did not provide a clean physical proof because Scarlett loopback capture again stopped advancing early; build remains clean."}

View file

@ -66,13 +66,14 @@ accepted_design:
final local-device hot path.
- The diagnostic audio path can optionally carry base64 float sample payloads
for sync smoke tests. `MimirChirpletTimeline` owns the structured
birdsong-like timeline fingerprint: deterministic unbounded chirplet events
where event `N` is generated from `N`, plus PCM segment rendering, matched
timing atoms, and per-band response kernels. `MimirAudioSynchronizationAnalyzer`
now treats `loopback-scarlett-speakers` as timing authority, resamples
candidate mic windows into that timeline, projects loopback and mic windows
through continuous chirplet energy traces, and estimates current delay from
chirplet-energy traces at a shared timestamp edge. Positive delay means the
birdsong-like timeline fingerprint: an order-3 de Bruijn sequence over 32
separated chirp symbols, plus PCM segment rendering, matched timing atoms,
symbol/event decoding, and per-band response kernels. Any three consecutive
correctly detected symbols identify a timeline event within the current
operating horizon. `MimirAudioSynchronizationAnalyzer` now treats
`loopback-scarlett-speakers` as timing authority, uses coarse chirplet
energy for acquisition, and refines delay from matched timeline event
indices when symbol decoding succeeds. Positive delay means the
candidate mic is late relative to loopback. Reports include rounded delay,
fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
owns the smoothed per-source belief: latest delay, smoothed delay,