Decode chirplet paths through trellis

This commit is contained in:
Meta Vens 2026-05-22 15:00:12 +01:00
commit c298e0922d
10 changed files with 299 additions and 28 deletions

View file

@ -64,10 +64,12 @@ quiet birdsong texture than a repeated sweep.
The intended decoder is a constrained chirplet transform, not a generic
time-frequency explorer and not an outlier filter around bad guesses. Mimir owns
the emitter, so the receiver projects each mic stream against the known chirplet
dictionary, produces symbol likelihood events, and decodes every valid
consecutive triplet through the de Bruijn map. A decoded triplet is a canonical
timeline anchor: observed sample offset `S` corresponds to emitted event time
`T`. A stream of anchors fits the source clock directly:
dictionary, produces transform frames with multiple symbol candidates, and
scores candidate triplets through the de Bruijn map. A triplet only becomes a
canonical timeline anchor when its symbol likelihoods, measured inter-chirp
gaps, and neighboring anchors agree on one local sample clock. A decoded anchor
means observed sample offset `S` corresponds to emitted event time `T`. A stream
of anchors fits the source clock directly:
```text
observed_sample = source_offset + canonical_seconds * effective_sample_rate

View file

@ -50,9 +50,10 @@ a named invariant that the native runtime cannot protect yet.
horizon. Symbol identity is carried by start band, glide shape, duration, and
following inter-chirp gap.
- `MimirChirpletStreamDecoder` is the first constrained chirplet-transform
receiver. It owns a bounded PCM window, emits symbol observations, decodes
code-valid triplet anchors onto the canonical timeline, and fits a per-source
sample clock from those anchors.
receiver. It owns a bounded PCM window, emits transform frames with multiple
symbol candidates, decodes code-valid triplet anchors through a local trellis
that requires gap and clock coherence, and fits a per-source sample clock from
those anchors.
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
sample-bearing audio blocks are resampled into the Scarlett loopback timeline.
The analyzer now prefers decoded triplet timeline anchors and derives delay

View file

@ -101,10 +101,11 @@ matched event indices when the symbol decoder succeeds, and can build a
provisional aligned mono frame for channels that clear the confidence gate.
Reports now carry fractional delay and per-band matched energy. The first
`MimirChirpletStreamDecoder` exists as the constrained chirplet-transform
receiver: it turns a bounded PCM window into symbol observations, every
code-valid triplet anchor it can prove, and an affine source clock fit. The
analyzer prefers matched canonical anchors before falling back to the older
chirplet-energy lag search. The hub also owns cached reports plus smoothed
receiver: it turns a bounded PCM window into transform frames with multiple
symbol candidates, code-valid triplet anchors selected through gap/clock
coherence, and an affine source clock fit. The analyzer prefers matched
canonical anchors before falling back to the older chirplet-energy lag search.
The hub also owns cached reports plus smoothed
per-source sync state with delay-slope/SRO in ppm. `MimirRuntime` runs analysis
online as a bounded rotating service and can print live telemetry with
`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are passive readers of cached

View file

@ -72,9 +72,10 @@ Get-Content .\state\evidence.jsonl -Tail 8
symbol/event decoding, and per-band response hooks. Sync reports now include
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
now tracks smoothed delay and delay-slope/SRO ppm. `MimirChirpletStreamDecoder`
is the first constrained chirplet-transform receiver: it emits symbol
observations, code-valid triplet anchors, and a source clock fit from a
bounded PCM window. `MimirRuntime` runs sync analysis as a bounded rotating
is the first constrained chirplet-transform receiver: it emits transform
frames with multiple symbol candidates, selects code-valid triplet anchors
through gap/clock coherence, and fits a source clock from a bounded PCM
window. `MimirRuntime` runs sync analysis as a bounded rotating
service and caches reports/states for UI and telemetry; readouts must stay
passive. Actual Mimir.App testing proves Aquarium audio wakes Scarlett
loopback, keeps mic buffers live, and produces confident online sync states,

View file

@ -76,7 +76,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
var referenceDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(referenceWindow, referenceBlock.SampleRate);
var candidateDecode = MimirChirpletTimeline.Default.DecodeStreamWindow(candidateWindow, referenceBlock.SampleRate);
var deterministicFit = EstimateDelayFromDecodedTimeline(referenceDecode, candidateDecode);
if (deterministicFit.MatchedEvents >= 1)
if (deterministicFit.MatchedEvents >= 3)
{
var decodedDelaySamples = deterministicFit.DelaySamples;
var decodedBandResponses = MimirChirpletTimeline.Default.EstimateBandResponse(candidateWindow, referenceBlock.SampleRate);
@ -189,7 +189,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
Math.Sqrt(referenceAnchor.Confidence * candidateAnchor.Confidence)));
}
if (matched.Count >= 1)
if (matched.Count >= 3)
{
var totalWeight = matched.Sum(pair => Math.Max(1.0e-6, pair.Weight));
var delay = matched.Sum(pair => pair.Delay * Math.Max(1.0e-6, pair.Weight)) / totalWeight;

View file

@ -1,5 +1,17 @@
namespace Mimir.Runtime.Synchronization;
public sealed record MimirChirpletSymbolCandidate(
int SymbolId,
double Energy);
public sealed record MimirChirpletTransformFrame(
double SampleOffset,
IReadOnlyList<MimirChirpletSymbolCandidate> Candidates)
{
public MimirChirpletSymbolCandidate BestCandidate =>
Candidates.OrderByDescending(candidate => candidate.Energy).First();
}
public sealed record MimirChirpletSymbolObservation(
int SymbolId,
double SampleOffset,
@ -24,6 +36,7 @@ public sealed record MimirChirpletClockFit(
}
public sealed record MimirChirpletStreamDecode(
IReadOnlyList<MimirChirpletTransformFrame> Frames,
IReadOnlyList<MimirChirpletSymbolObservation> Symbols,
IReadOnlyList<MimirChirpletTimelineAnchor> Anchors,
MimirChirpletClockFit? ClockFit);

View file

@ -74,6 +74,9 @@ public sealed class MimirChirpletStreamDecoder
var clock = decode.ClockFit == null
? null
: decode.ClockFit with { SourceOffsetSamples = decode.ClockFit.SourceOffsetSamples + originSample };
return new MimirChirpletStreamDecode(symbols, anchors, clock);
var frames = decode.Frames
.Select(frame => frame with { SampleOffset = frame.SampleOffset + originSample })
.ToArray();
return new MimirChirpletStreamDecode(frames, symbols, anchors, clock);
}
}

View file

@ -46,6 +46,7 @@ public sealed class MimirChirpletTimeline
private const double MaxEventGapSeconds = 0.162;
private const double MaxEventDurationSeconds = 0.078;
private const double Gain = 0.090;
private const int MaxSymbolCandidatesPerFrame = 8;
private static readonly int[] TimelineSymbols = RotateToDistinctOpening(BuildDeBruijn(SymbolCount, TimelineOrder));
private static readonly Dictionary<int, int> TripleToIndex = BuildTripleIndex(TimelineSymbols);
@ -138,18 +139,19 @@ public sealed class MimirChirpletTimeline
{
if (samples.Length == 0)
{
return new MimirChirpletStreamDecode([], [], null);
return new MimirChirpletStreamDecode([], [], [], null);
}
var symbols = DetectSymbolEvents(samples, sampleRate, Math.Max(1, sampleRate / 200))
.Select(observation => new MimirChirpletSymbolObservation(
observation.SymbolId,
observation.SampleOffset,
observation.Energy))
var frames = DetectTransformFrames(samples, sampleRate, Math.Max(1, sampleRate / 200));
var symbols = frames
.Select(frame => new MimirChirpletSymbolObservation(
frame.BestCandidate.SymbolId,
frame.SampleOffset,
frame.BestCandidate.Energy))
.ToArray();
var anchors = PlaceAllTripletAnchors(symbols, sampleRate);
var anchors = DecodeTrellisAnchors(frames, sampleRate);
var clock = FitClock(anchors, sampleRate);
return new MimirChirpletStreamDecode(symbols, anchors, clock);
return new MimirChirpletStreamDecode(frames, symbols, anchors, clock);
}
public MimirChirpletTimelinePlacement DecodeReferenceWindow(
@ -277,6 +279,102 @@ public sealed class MimirChirpletTimeline
return SuppressNearbyDetections(candidates, sampleRate);
}
private IReadOnlyList<MimirChirpletTransformFrame> DetectTransformFrames(
ReadOnlySpan<float> samples,
int sampleRate,
int hopSamples)
{
if (samples.Length == 0)
{
return [];
}
var coarse = ContrastNormalize(BuildEnvelopeEnergyTrace(samples, sampleRate, hopSamples));
if (coarse.Length <= 2)
{
return [];
}
var frames = new List<MimirChirpletTransformFrame>();
for (var frame = 1; frame < coarse.Length - 1; frame++)
{
if (coarse[frame] < 0.055 ||
coarse[frame] < coarse[frame - 1] ||
coarse[frame] < coarse[frame + 1])
{
continue;
}
var transformFrame = ClassifySymbolsAt(
samples,
sampleRate,
frame * hopSamples,
Math.Max(hopSamples * 3, sampleRate / 100));
if (transformFrame != null)
{
frames.Add(transformFrame);
}
}
return SuppressNearbyFrames(frames, sampleRate);
}
private MimirChirpletTransformFrame? ClassifySymbolsAt(
ReadOnlySpan<float> samples,
int sampleRate,
int predictedOffset,
int searchRadiusSamples)
{
var candidates = new List<(MimirChirpletSymbolCandidate Candidate, int Offset)>(SymbolTones.Length);
for (var symbol = 0; symbol < SymbolTones.Length; symbol++)
{
var tone = SymbolTones[symbol];
var kernel = sampleRate == SampleRate ? symbolKernels[symbol] : RenderToneKernel(tone, sampleRate);
if (kernel.Length == 0 || samples.Length < kernel.Length)
{
continue;
}
var start = Math.Max(0, predictedOffset - searchRadiusSamples);
var end = Math.Min(samples.Length - kernel.Length, predictedOffset + searchRadiusSamples);
var bestEnergy = 0.0;
var bestOffset = predictedOffset;
var step = Math.Max(1, searchRadiusSamples / 8);
for (var offset = start; offset <= end; offset += step)
{
var energy = DirectionalMatchedEnergy(samples.Slice(offset, kernel.Length), kernel);
if (energy > bestEnergy)
{
bestEnergy = energy;
bestOffset = offset;
}
}
if (bestEnergy >= 0.08)
{
candidates.Add((new MimirChirpletSymbolCandidate(symbol, bestEnergy), bestOffset));
}
}
if (candidates.Count == 0)
{
return null;
}
var ordered = candidates
.OrderByDescending(candidate => candidate.Candidate.Energy)
.Take(MaxSymbolCandidatesPerFrame)
.ToArray();
if (ordered[0].Candidate.Energy < 0.25)
{
return null;
}
return new MimirChirpletTransformFrame(
ordered[0].Offset,
ordered.Select(candidate => candidate.Candidate).ToArray());
}
private MimirChirpletEventObservation? ClassifySymbolAt(
ReadOnlySpan<float> samples,
int sampleRate,
@ -338,6 +436,30 @@ public sealed class MimirChirpletTimeline
return kept.OrderBy(observation => observation.SampleOffset).ToArray();
}
private static IReadOnlyList<MimirChirpletTransformFrame> SuppressNearbyFrames(
IReadOnlyList<MimirChirpletTransformFrame> frames,
int sampleRate)
{
if (frames.Count == 0)
{
return [];
}
var minimumSpacingSamples = MinEventGapSeconds * sampleRate * 0.55;
var kept = new List<MimirChirpletTransformFrame>();
foreach (var frame in frames.OrderByDescending(frame => frame.BestCandidate.Energy))
{
if (kept.Any(existing => Math.Abs(existing.SampleOffset - frame.SampleOffset) < minimumSpacingSamples))
{
continue;
}
kept.Add(frame);
}
return kept.OrderBy(frame => frame.SampleOffset).ToArray();
}
private static MimirChirpletTimelinePlacement PlaceObservedSymbols(
IReadOnlyList<MimirChirpletEventObservation> observations,
int sampleRate)
@ -442,6 +564,124 @@ public sealed class MimirChirpletTimeline
.ToArray();
}
private static IReadOnlyList<MimirChirpletTimelineAnchor> DecodeTrellisAnchors(
IReadOnlyList<MimirChirpletTransformFrame> frames,
int sampleRate)
{
if (frames.Count < TimelineOrder)
{
return [];
}
var candidates = new List<MimirChirpletTimelineAnchor>();
for (var index = 0; index <= frames.Count - TimelineOrder; index++)
{
var firstFrame = frames[index];
var secondFrame = frames[index + 1];
var thirdFrame = frames[index + 2];
if (!IsConsecutiveByTime(firstFrame, secondFrame, sampleRate) ||
!IsConsecutiveByTime(secondFrame, thirdFrame, sampleRate))
{
continue;
}
foreach (var first in firstFrame.Candidates)
foreach (var second in secondFrame.Candidates)
foreach (var third in thirdFrame.Candidates)
{
var code = TripleCode(first.SymbolId, second.SymbolId, third.SymbolId);
if (!TripleToIndex.TryGetValue(code, out var eventIndex))
{
continue;
}
var firstEvent = Default.EventForIndex((ulong)eventIndex);
var secondEvent = Default.EventForIndex((ulong)eventIndex + 1UL);
var thirdEvent = Default.EventForIndex((ulong)eventIndex + 2UL);
var measuredGapA = (secondFrame.SampleOffset - firstFrame.SampleOffset) / sampleRate;
var measuredGapB = (thirdFrame.SampleOffset - secondFrame.SampleOffset) / sampleRate;
var expectedGapA = secondEvent.StartSeconds - firstEvent.StartSeconds;
var expectedGapB = thirdEvent.StartSeconds - secondEvent.StartSeconds;
var gapError = Math.Abs(measuredGapA - expectedGapA) + Math.Abs(measuredGapB - expectedGapB);
var gapConfidence = 1.0 / (1.0 + gapError / 0.004);
if (gapConfidence < 0.50)
{
continue;
}
var energyConfidence = Math.Clamp((first.Energy + second.Energy + third.Energy) / 3.0, 0.0, 1.0);
candidates.Add(new MimirChirpletTimelineAnchor(
(ulong)eventIndex,
firstEvent.StartSeconds,
firstFrame.SampleOffset,
gapConfidence * 0.70 + energyConfidence * 0.30,
[
new MimirChirpletSymbolObservation(first.SymbolId, firstFrame.SampleOffset, first.Energy),
new MimirChirpletSymbolObservation(second.SymbolId, secondFrame.SampleOffset, second.Energy),
new MimirChirpletSymbolObservation(third.SymbolId, thirdFrame.SampleOffset, third.Energy),
]));
}
}
return SelectCoherentAnchorPath(candidates, sampleRate);
}
private static IReadOnlyList<MimirChirpletTimelineAnchor> SelectCoherentAnchorPath(
IReadOnlyList<MimirChirpletTimelineAnchor> candidates,
int sampleRate)
{
if (candidates.Count == 0)
{
return [];
}
List<MimirChirpletTimelineAnchor> bestPath = [];
var bestScore = double.NegativeInfinity;
foreach (var seed in candidates.OrderByDescending(anchor => anchor.Confidence).Take(64))
{
var seedOffset = seed.SampleOffset - seed.TimelineSeconds * sampleRate;
var path = candidates
.Where(anchor =>
Math.Abs((anchor.SampleOffset - anchor.TimelineSeconds * sampleRate) - seedOffset) <= sampleRate * 0.008)
.GroupBy(anchor => anchor.EventIndex)
.Select(group => group.OrderByDescending(anchor => anchor.Confidence).First())
.OrderBy(anchor => anchor.EventIndex)
.ToList();
var clock = FitClock(path, sampleRate);
if (clock == null)
{
continue;
}
var score = path.Sum(anchor => anchor.Confidence) +
Math.Min(path.Count, 12) * 0.20 -
clock.MeanAbsoluteErrorSamples / Math.Max(1.0, sampleRate * 0.001);
if (score > bestScore)
{
bestScore = score;
bestPath = path;
}
}
if (bestPath.Count == 0)
{
return [];
}
var finalClock = FitClock(bestPath, sampleRate);
if (finalClock == null)
{
return [];
}
return bestPath
.Where(anchor =>
Math.Abs(anchor.SampleOffset - finalClock.SampleForTimelineSeconds(anchor.TimelineSeconds)) <=
Math.Max(24.0, sampleRate * 0.002))
.OrderBy(anchor => anchor.SampleOffset)
.ToArray();
}
private static MimirChirpletClockFit? FitClock(
IReadOnlyList<MimirChirpletTimelineAnchor> anchors,
int sampleRate)
@ -519,6 +759,15 @@ public sealed class MimirChirpletTimeline
return seconds >= MinEventGapSeconds * 0.55 && seconds <= MaxEventGapSeconds * 1.45;
}
private static bool IsConsecutiveByTime(
MimirChirpletTransformFrame first,
MimirChirpletTransformFrame second,
int sampleRate)
{
var seconds = (second.SampleOffset - first.SampleOffset) / sampleRate;
return seconds >= MinEventGapSeconds * 0.55 && seconds <= MaxEventGapSeconds * 1.45;
}
private static MimirChirpletEventObservation PlaceObservation(
MimirChirpletEventObservation observation,
ulong eventIndex)

View file

@ -44,3 +44,4 @@
{"ts":"2026-05-22T14:48:00+01:00","type":"verification","status":"partial","note":"Attempted a 20-second real-world Mimir.App chirplet sync run with all configured local mics. The headless path only emitted one telemetry tick before runtime updates stopped. Switching to the normal app path produced telemetry through t=4.26s with all six audio buffers live and sync states for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Timeline-symbol refinement appeared on Focusrite, Kiyo, and one PS3 Eye, but delay estimates were unstable and the app did not continue telemetry to 20 seconds. Replaced the coarse acquisition trace with envelope energy to avoid matched-filter scans in the hot acquisition path; symbol decoding remains the refinement step."}
{"ts":"2026-05-22T15:20:00+01:00","type":"implementation","status":"partial","note":"Removed hidden re-entrant synchronization analysis from the live app path. MimirRuntime now calls a bounded rotating AnalyzeAudioSynchronizationStep every 0.1s, MimirSynchronizationHub caches per-source reports/states, and UI/telemetry read cached state instead of invoking the analyzer or BuildAlignedAudioFrame. FitDelayFromEvents now collapses duplicate candidate event observations by strongest energy, fixing the real app crash 'An item with the same key has already been added. Key: 0'. Dotnet build passed. A visible Mimir.App run with all configured mics no longer crashed at the earlier 4s wall and produced telemetry through runtime t=17.28s in a 22s wall-time harness; it did not reach a full 20 runtime seconds because startup consumed wall time. All non-loopback mics produced cached reports/states, but delay estimates remained unstable and timeline event refinement was intermittent, so the next live invariant is robust outlier rejection/state filtering before any actuator drives resampling."}
{"ts":"2026-05-22T16:05:00+01:00","type":"implementation","status":"partial","note":"Documented the intended decoder as a constrained chirplet transform: known dictionary projection to symbol observations, code-valid de Bruijn triplets to canonical timeline anchors, anchors to affine source clock fits, and delay/SRO from comparing loopback and mic clock fits. Added MimirChirpletStreamDecode records and MimirChirpletStreamDecoder with a bounded PCM window, absolute sample offsets, triplet anchors, and clock fits. MimirAudioSynchronizationAnalyzer now prefers matched decoded timeline anchors before falling back to the older chirplet-energy lag search. Synthetic self-test over two seconds of emitted timeline audio currently decodes 12 symbols and one correct anchor for event 9 at sample 47520 with clock 48000 Hz; this proves the new spine but also proves the symbol likelihood/codebook layer is not finished enough to place every triplet yet."}
{"ts":"2026-05-22T16:35:00+01:00","type":"implementation","status":"partial","note":"Replaced one-best-symbol triplet placement with a constrained transform/trellis pass. DecodeStreamWindow now emits transform frames with multiple symbol candidates per chirp, enumerates candidate triplets through the de Bruijn map, scores measured inter-chirp gaps, and keeps the coherent local clock path rather than accepting any random valid triplet. Synthetic two-second emitted timeline check improved from one anchor to 11 coherent anchors, including events 0-9, with fitted clock 47978.350 Hz, confidence 0.885, and 8.741 sample MAE. This is the right decoder shape; remaining work is symbol/codebook separation and sub-frame timing precision so the fit lands exactly at 48000 Hz and covers every emitted triplet."}

View file

@ -73,9 +73,9 @@ accepted_design:
inter-chirp gap. Any three consecutive correctly detected symbols identify a
timeline event within the current operating horizon. `MimirChirpletStreamDecoder`
is the first constrained chirplet-transform receiver: it owns a bounded PCM
window, emits symbol observations, decodes code-valid triplet anchors onto
the canonical timeline, and fits a per-source sample clock from those
anchors. `MimirAudioSynchronizationAnalyzer` now treats
window, emits transform frames with multiple symbol candidates, decodes
code-valid triplet anchors through gap/clock coherence, and fits a
per-source sample clock from those anchors. `MimirAudioSynchronizationAnalyzer` now treats
`loopback-scarlett-speakers` as timing authority, prefers delay from matched
decoded timeline anchors, and keeps the older chirplet-energy lag search as
a diagnostic fallback while the decoder is cut over. Positive delay means