Track chirplet audio sync state
This commit is contained in:
Meta Vens
parent
bbb052a8ba
commit
1cf46ef97b
12 changed files with 150 additions and 14 deletions
|
|
@ -78,6 +78,13 @@ estimation: loopback carries what was emitted, each mic carries what survived
|
|||
speaker, air, room, and capsule, and the ratio over repeated chirplet phrases
|
||||
becomes gain/phase correction evidence.
|
||||
|
||||
`MimirAudioSynchronizationStateTracker` turns per-phrase observations into
|
||||
state. It confidence-gates reports, smooths fractional delay per source, and
|
||||
estimates delay slope as sampling-rate offset in ppm. This is the control input
|
||||
for the coming actuator. The state can survive a brief weak report, but it is
|
||||
not a license to run blind: loopback must keep receiving the emitted phrase or
|
||||
fresh reports will stop.
|
||||
|
||||
## Chirplet Calibration Model
|
||||
|
||||
```mermaid
|
||||
|
|
@ -89,9 +96,10 @@ flowchart TD
|
|||
D --> F["mic rolling buffers"]
|
||||
E --> G["matched chirplet traces"]
|
||||
F --> G
|
||||
G --> H["delay + SRO estimates"]
|
||||
G --> H["delay observations"]
|
||||
H --> L["smoothed sync state + SRO"]
|
||||
G --> I["per-band response estimates"]
|
||||
H --> J["fractional delay / resampler actuator"]
|
||||
L --> J["fractional delay / resampler actuator"]
|
||||
I --> K["frequency response normalization"]
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -51,6 +51,9 @@ a named invariant that the native runtime cannot protect yet.
|
|||
- `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
||||
provisional aligned mono frame for loopback-referenced audio channels that
|
||||
clear the confidence gate.
|
||||
- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source
|
||||
sync state: latest fractional delay, smoothed delay, confidence, per-band
|
||||
response evidence, and delay-slope/SRO estimate in ppm.
|
||||
- `MimirRuntime` emits short calibration chirplets through Aquarium audio so the
|
||||
loopback and acoustic mic buffers carry a shared timing witness.
|
||||
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
|
||||
|
|
@ -86,10 +89,10 @@ a named invariant that the native runtime cannot protect yet.
|
|||
frame cadence in the rolling buffers.
|
||||
3. Replace the WASAPI frame-event diagnostic bridge with native audio capture
|
||||
workers for local mic, loopback, and network audio feeds.
|
||||
4. Add the synchronization actuator: smooth delay/SRO estimates, then drive a
|
||||
variable-rate resampler and fractional delay line per non-reference stream.
|
||||
The current aligned frame is integer-delay only and should not pretend to be
|
||||
the final DSP path.
|
||||
4. Add the synchronization actuator: drive a variable-rate resampler and
|
||||
fractional delay line per non-reference stream from the smoothed
|
||||
`MimirAudioSynchronizationState`. The current aligned frame is integer-delay
|
||||
only and should not pretend to be the final DSP path.
|
||||
5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
|
||||
source timestamps, and output settings are visible and adjustable.
|
||||
6. Move GPU feature extraction, fusion, material fitting, render budgeting, and
|
||||
|
|
|
|||
|
|
@ -96,10 +96,11 @@ is the current timing authority when calibration chirplets are playing.
|
|||
Mimir emits that phrase through Aquarium audio, compares mic buffers against
|
||||
loopback by chirplet-energy delay estimation, and can build a provisional
|
||||
aligned mono frame for channels that clear the confidence gate. Reports now
|
||||
carry fractional delay and per-band matched energy. Camera mics are
|
||||
carry fractional delay and per-band matched energy. The hub also owns smoothed
|
||||
per-source sync state with delay-slope/SRO in ppm. Camera mics are
|
||||
spatial/context witnesses; Focusrite devices are dialogue anchors. The current
|
||||
aligned frame is integer-delay only; SRO smoothing, fractional delay, and the
|
||||
hot resampler still belong in Faust/native DSP.
|
||||
aligned frame is integer-delay only; fractional delay and the hot resampler
|
||||
still belong in Faust/native DSP.
|
||||
|
||||
## Visual Fusion
|
||||
|
||||
|
|
|
|||
|
|
@ -66,7 +66,8 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
authority. `MimirChirpletCalibrationPhrase` owns the structured harmonic-ish
|
||||
phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, the Aquarium PCM rendering,
|
||||
matched timing trace, and per-band response hooks. Sync reports now include
|
||||
fractional delay and per-band matched energy. PS3 Eye audio is
|
||||
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
|
||||
now tracks smoothed delay and delay-slope/SRO ppm. PS3 Eye audio is
|
||||
enumeration/runtime-fragile: one run saw both mic buffers empty while both
|
||||
cameras were live, then a replug made both PS3 Eye mic buffers emit
|
||||
480-frame WASAPI blocks again.
|
||||
|
|
|
|||
|
|
@ -14,9 +14,16 @@ foreach (var source in configuration.Value.Sources)
|
|||
}
|
||||
|
||||
var deadline = DateTimeOffset.UtcNow + duration;
|
||||
var nextSyncPoll = DateTimeOffset.UtcNow + TimeSpan.FromSeconds(1);
|
||||
while (DateTimeOffset.UtcNow < deadline)
|
||||
{
|
||||
hub.PollSources(maxSamplesPerSource: 4096);
|
||||
if (DateTimeOffset.UtcNow >= nextSyncPoll)
|
||||
{
|
||||
hub.AnalyzeAudioSynchronization(syncReference);
|
||||
nextSyncPoll += TimeSpan.FromSeconds(1);
|
||||
}
|
||||
|
||||
await Task.Delay(pollDelay).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
|
|
@ -55,6 +62,12 @@ foreach (var report in reports.OrderBy(report => report.SourceId, StringComparer
|
|||
$"sync {report.ReferenceSourceId}->{report.SourceId}: delaySamples={report.DelaySamples} fractionalDelaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000} bands={DescribeBands(report.BandResponses)} compared={report.ComparedSamples}");
|
||||
}
|
||||
|
||||
foreach (var state in hub.AudioSynchronizationStates)
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"sync-state {state.ReferenceSourceId}->{state.SourceId}: smoothedDelaySamples={state.SmoothedDelaySamples:0.000} delayMs={state.DelayMilliseconds:0.000} sroPpm={state.SamplingRateOffsetPpm:0.000} confidence={state.Confidence:0.000} bands={DescribeBands(state.BandResponses)}");
|
||||
}
|
||||
|
||||
var aligned = hub.BuildAlignedAudioFrame(syncReference);
|
||||
if (aligned != null)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -117,6 +117,7 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
panel.Readout("Ingested", () => $"{synchronization.IngestedSamples}");
|
||||
panel.Readout("Buffer details", DescribeBuffers);
|
||||
panel.Readout("Audio sync", DescribeAudioSync);
|
||||
panel.Readout("Audio sync state", DescribeAudioSyncState);
|
||||
panel.Readout("Aligned audio", DescribeAlignedAudio);
|
||||
panel.Readout("Chirplet reference", () => $"{DefaultAudioSyncReference} every {CalibrationPhrase.IntervalSeconds:0.0}s");
|
||||
});
|
||||
|
|
@ -171,4 +172,12 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
? "no aligned frame"
|
||||
: $"{frame.Channels.Count}ch {frame.SampleRate}Hz {frame.FrameCount} frames";
|
||||
}
|
||||
|
||||
private string DescribeAudioSyncState()
|
||||
{
|
||||
var states = synchronization.AudioSynchronizationStates;
|
||||
return states.Count == 0
|
||||
? "no sync state"
|
||||
: string.Join(" | ", states.Select(state => $"{state.SourceId}: {state.SmoothedDelaySamples:0.0} samples sro={state.SamplingRateOffsetPpm:0.0}ppm c={state.Confidence:0.00}"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -87,6 +87,7 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
fractionalDelaySamples * 1000.0 / referenceBlock.SampleRate,
|
||||
confidence,
|
||||
bandResponses,
|
||||
commonEndNs,
|
||||
compared,
|
||||
reference.Latest?.Sequence ?? 0,
|
||||
buffer.Latest?.Sequence ?? 0));
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ public sealed record MimirAudioSynchronizationReport(
|
|||
double DelayMilliseconds,
|
||||
double Confidence,
|
||||
IReadOnlyList<MimirChirpletBandResponse> BandResponses,
|
||||
long AnalysisTimestampNs,
|
||||
int ComparedSamples,
|
||||
ulong ReferenceSequence,
|
||||
ulong SourceSequence);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,89 @@
|
|||
namespace Mimir.Runtime.Synchronization;
|
||||
|
||||
public sealed record MimirAudioSynchronizationState(
|
||||
string ReferenceSourceId,
|
||||
string SourceId,
|
||||
int SampleRate,
|
||||
double LatestDelaySamples,
|
||||
double SmoothedDelaySamples,
|
||||
double DelayMilliseconds,
|
||||
double SamplingRateOffsetPpm,
|
||||
double Confidence,
|
||||
IReadOnlyList<MimirChirpletBandResponse> BandResponses,
|
||||
long UpdatedAtNs,
|
||||
ulong ReferenceSequence,
|
||||
ulong SourceSequence);
|
||||
|
||||
public sealed class MimirAudioSynchronizationStateTracker
|
||||
{
|
||||
private const double MinimumUpdateConfidence = 0.05;
|
||||
private readonly Dictionary<string, MimirAudioSynchronizationState> states = new(StringComparer.Ordinal);
|
||||
|
||||
public IReadOnlyList<MimirAudioSynchronizationState> States =>
|
||||
states.Values.OrderBy(state => state.SourceId, StringComparer.Ordinal).ToArray();
|
||||
|
||||
public void Update(IEnumerable<MimirAudioSynchronizationReport> reports)
|
||||
{
|
||||
foreach (var report in reports)
|
||||
{
|
||||
if (report.Confidence < MinimumUpdateConfidence)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!states.TryGetValue(report.SourceId, out var previous))
|
||||
{
|
||||
states[report.SourceId] = NewState(report, report.FractionalDelaySamples, 0.0);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (previous.ReferenceSequence == report.ReferenceSequence &&
|
||||
previous.SourceSequence == report.SourceSequence)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var alpha = Math.Clamp(report.Confidence * 0.35, 0.04, 0.30);
|
||||
var smoothedDelay = previous.SmoothedDelaySamples +
|
||||
(report.FractionalDelaySamples - previous.SmoothedDelaySamples) * alpha;
|
||||
var measuredSro = EstimateSroPpm(previous, smoothedDelay, report);
|
||||
var sroAlpha = Math.Clamp(report.Confidence * 0.20, 0.02, 0.18);
|
||||
var smoothedSro = previous.SamplingRateOffsetPpm +
|
||||
(measuredSro - previous.SamplingRateOffsetPpm) * sroAlpha;
|
||||
states[report.SourceId] = NewState(report, smoothedDelay, smoothedSro);
|
||||
}
|
||||
}
|
||||
|
||||
private static MimirAudioSynchronizationState NewState(
|
||||
MimirAudioSynchronizationReport report,
|
||||
double smoothedDelaySamples,
|
||||
double samplingRateOffsetPpm) =>
|
||||
new(
|
||||
report.ReferenceSourceId,
|
||||
report.SourceId,
|
||||
report.SampleRate,
|
||||
report.FractionalDelaySamples,
|
||||
smoothedDelaySamples,
|
||||
smoothedDelaySamples * 1000.0 / report.SampleRate,
|
||||
samplingRateOffsetPpm,
|
||||
report.Confidence,
|
||||
report.BandResponses,
|
||||
report.AnalysisTimestampNs,
|
||||
report.ReferenceSequence,
|
||||
report.SourceSequence);
|
||||
|
||||
private static double EstimateSroPpm(
|
||||
MimirAudioSynchronizationState previous,
|
||||
double smoothedDelaySamples,
|
||||
MimirAudioSynchronizationReport report)
|
||||
{
|
||||
var dtSeconds = (report.AnalysisTimestampNs - previous.UpdatedAtNs) / 1_000_000_000.0;
|
||||
if (dtSeconds <= 0.050)
|
||||
{
|
||||
return previous.SamplingRateOffsetPpm;
|
||||
}
|
||||
|
||||
var delayDeltaSamples = smoothedDelaySamples - previous.SmoothedDelaySamples;
|
||||
return delayDeltaSamples / (dtSeconds * report.SampleRate) * 1_000_000.0;
|
||||
}
|
||||
}
|
||||
|
|
@ -4,6 +4,7 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
{
|
||||
private readonly List<IMimirStreamSource> sources = [];
|
||||
private readonly MimirAudioSynchronizationAnalyzer audioSynchronization = new();
|
||||
private readonly MimirAudioSynchronizationStateTracker audioSynchronizationState = new();
|
||||
private ulong ingestedSamples;
|
||||
|
||||
public MimirSynchronizationHub(MimirSynchronizationSettings settings)
|
||||
|
|
@ -24,6 +25,9 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
|
||||
public int SourceCount => sources.Count;
|
||||
|
||||
public IReadOnlyList<MimirAudioSynchronizationState> AudioSynchronizationStates =>
|
||||
audioSynchronizationState.States;
|
||||
|
||||
public void AddSource(IMimirStreamSource source)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
|
|
@ -61,7 +65,9 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronization(string referenceSourceId)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
return audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId);
|
||||
var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId);
|
||||
audioSynchronizationState.Update(reports);
|
||||
return reports;
|
||||
}
|
||||
|
||||
public MimirAlignedAudioFrame? BuildAlignedAudioFrame(
|
||||
|
|
|
|||
|
|
@ -35,3 +35,4 @@
|
|||
{"ts":"2026-05-22T11:58:15+01:00","type":"verification","status":"ok","note":"Delay signs were suspect because MimirAudioSynchronizationAnalyzer compared latest sample tails from buffers with different edge timestamps. The analyzer now estimates each pair at a shared timestamp edge and BuildAlignedAudioFrame chooses an output edge all positive-late channels can supply. Controlled 9-16 kHz chirplet smoke after the fix reported positive loopback-relative PS3 Eye delay: mic-usb-camera-1 delaySamples=704, delayMs=15.964, confidence=0.166, and the aligned frame contained loopback plus mic-usb-camera-1."}
|
||||
{"ts":"2026-05-22T12:38:53+01:00","type":"implementation","status":"ok","note":"Documented the chirplet calibration model and started the structured implementation. MimirChirpletCalibrationPhrase now owns a harmonic-ish phrase around 8 kHz, 10 kHz, 12 kHz, and 16 kHz, PCM rendering for Aquarium emission, matched chirplet timing traces, and per-band matched-energy response hooks. MimirRuntime emits that shared phrase instead of a separate hard-coded sweep, and MimirAudioSynchronizationAnalyzer uses the same phrase for loopback-referenced delay analysis with 16-sample hops and parabolic fractional peak interpolation. BufferSmoke output now includes fractionalDelaySamples and per-band energy."}
|
||||
{"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."}
|
||||
{"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."}
|
||||
|
|
|
|||
|
|
@ -73,9 +73,12 @@ accepted_design:
|
|||
emitted chirplet phrase, and estimates current delay from chirplet-energy
|
||||
traces at a shared timestamp edge. Positive delay means the candidate mic is
|
||||
late relative to loopback. Reports include rounded delay, fractional delay,
|
||||
confidence, and per-band matched energy. `MimirSynchronizationHub.BuildAlignedAudioFrame`
|
||||
exposes the first provisional loopback-referenced aligned mono frame for
|
||||
channels that clear the confidence gate.
|
||||
confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
|
||||
owns the smoothed per-source belief: latest delay, smoothed delay,
|
||||
confidence, per-band response evidence, and delay-slope/SRO estimate in
|
||||
ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
||||
provisional loopback-referenced aligned mono frame for channels that clear
|
||||
the confidence gate.
|
||||
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
||||
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
|
||||
texture capture.
|
||||
|
|
|
|||
Loading…
Reference in a new issue