Test live runtime chirplet sync

This commit is contained in:
Meta Vens 2026-05-22 13:15:38 +01:00
commit da9915b6e5
8 changed files with 116 additions and 33 deletions

View file

@ -53,8 +53,8 @@ audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration
phrase and the matched-filter shape used to analyze it. The default timeline is
a stateless 16-phrase cycle. Phrase `N` is generated directly from `N`, with no
registry or remembered random state. Each phrase is a spread-out asymmetric
motif: six short chirplets over about 1.38 seconds, with nonuniform gaps and
high-frequency bands. A new phrase fires every 3.25 seconds through Aquarium
motif: six short chirplets over about 0.85 seconds, with nonuniform gaps and
high-frequency bands. A new phrase fires every 2.25 seconds through Aquarium
audio. The point is not ornament. The timing code is carried by both frequency
and rhythm, so it behaves more like a small birdsong signature than a repeated
sweep. The indexed phrase sequence has lower ambiguity when remote feeds add
@ -91,6 +91,13 @@ for the coming actuator. The state can survive a brief weak report, but it is
not a license to run blind: loopback must keep receiving the emitted phrase or
fresh reports will stop.
The actual Mimir app path now runs this online: `MimirRuntime.Update` emits the
phrase sequence, polls sources, and updates sync analysis on a fixed cadence.
`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current
runtime testing proves Aquarium output wakes the Scarlett loopback and the mic
buffers stay live, but the app has not yet produced a confident acoustic lock.
That next failure is calibration, not plumbing.
## Chirplet Calibration Model
```mermaid

View file

@ -45,8 +45,8 @@ a named invariant that the native runtime cannot protect yet.
- `MimirChirpletCalibrationPhrase` owns the structured birdsong-like calibration
phrase, PCM rendering, matched timing trace, and per-band response kernels.
The default timeline is a stateless 16-phrase cycle; phrase `N` is generated
from `N`, spreads six chirplets across about 1.38 seconds, and fires every
3.25 seconds to reduce repeated-sweep ambiguity.
from `N`, spreads six chirplets across about 0.85 seconds, and fires every
2.25 seconds to reduce repeated-sweep ambiguity.
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
sample-bearing audio blocks are resampled into the Scarlett loopback timeline
and compared by chirplet-energy delay estimation with fractional peak
@ -57,6 +57,8 @@ a named invariant that the native runtime cannot protect yet.
- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source
sync state: latest fractional delay, smoothed delay, confidence, per-band
response evidence, and delay-slope/SRO estimate in ppm.
- `MimirRuntime` updates audio sync analysis online and can emit live sync
telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`.
- `MimirRuntime` emits short calibration chirplets through Aquarium audio so the
loopback and acoustic mic buffers carry a shared timing witness.
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
@ -96,6 +98,8 @@ a named invariant that the native runtime cannot protect yet.
fractional delay line per non-reference stream from the smoothed
`MimirAudioSynchronizationState`. The current aligned frame is integer-delay
only and should not pretend to be the final DSP path.
First, make the live app acoustic lock robust enough to populate sync state
from the online chirplet timeline, not just from smoke harnesses.
5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
source timestamps, and output settings are visible and adjustable.
6. Move GPU feature extraction, fusion, material fitting, render budgeting, and

View file

@ -93,12 +93,15 @@ The six-microphone path is separate from the bridge. Scarlett speaker loopback
is the current timing authority when calibration chirplets are playing.
`MimirChirpletCalibrationPhrase` owns the birdsong-like timeline fingerprint:
a stateless 16-phrase cycle where phrase `N` is generated from `N`; each phrase
spreads six asymmetric chirplets across about 1.38 seconds and fires every 3.25
spreads six asymmetric chirplets across about 0.85 seconds and fires every 2.25
seconds. Mimir emits that phrase sequence through Aquarium audio, compares mic
buffers against loopback by chirplet-energy delay estimation, and can build a
provisional aligned mono frame for channels that clear the confidence gate.
Reports now carry fractional delay and per-band matched energy. The hub also
owns smoothed per-source sync state with delay-slope/SRO in ppm. Camera mics are
owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime`
runs the analysis online and can print live telemetry with
`MIMIR_SYNC_TELEMETRY_SECONDS`. Current app testing proves loopback wakeup and
live mic buffers, but not yet a confident acoustic lock. Camera mics are
spatial/context witnesses; Focusrite devices are dialogue anchors. The current
aligned frame is integer-delay only; fractional delay and the hot resampler
still belong in Faust/native DSP.

View file

@ -65,18 +65,20 @@ Get-Content .\state\evidence.jsonl -Tail 8
- Current chirplet-backed smoke uses Scarlett speaker loopback as timing
authority. `MimirChirpletCalibrationPhrase` owns the structured birdsong-like
timeline fingerprint: a stateless 16-phrase cycle where phrase `N` is
generated from `N`; each phrase has six asymmetric chirplets over about 1.38
seconds and fires every 3.25 seconds. The phrase owns Aquarium PCM rendering,
generated from `N`; each phrase has six asymmetric chirplets over about 0.85
seconds and fires every 2.25 seconds. The phrase owns Aquarium PCM rendering,
matched timing trace, and per-band response hooks. Sync reports now include
fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and
delay-slope/SRO ppm. PS3 Eye audio is
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing
proves Aquarium audio wakes Scarlett loopback and keeps mic buffers live, but
the online acoustic report has not cleared the confidence gate yet. PS3 Eye audio is
enumeration/runtime-fragile: one run saw both mic buffers empty while both
cameras were live, then a replug made both PS3 Eye mic buffers emit
480-frame WASAPI blocks again.
## Immediate Re-entry Instruction
Replace the diagnostic audio JSON process bridge with native audio capture
workers, then turn chirplet delay reports into a smoothed SRO/fractional-delay
actuator. Keep loopback as timing authority. Do not restore deleted script
infrastructure because a stale doc once missed it.
Make the actual Mimir.App path produce a confident online acoustic lock from the
chirplet timeline, then turn those delay reports into a smoothed
SRO/fractional-delay actuator. Keep loopback as timing authority. Do not restore
deleted script infrastructure because a stale doc once missed it.

View file

@ -11,14 +11,19 @@ namespace Mimir.Runtime;
public sealed class MimirRuntime : IAquariumRuntime
{
private const string DefaultAudioSyncReference = "loopback-scarlett-speakers";
private const float AudioSyncUpdateIntervalSeconds = 0.5f;
private readonly LocalCastRuntime visualRuntime;
private readonly MimirSynchronizationHub synchronization;
private readonly AquariumUiDocument ui;
private readonly float telemetryIntervalSeconds;
private int lastPollCount;
private float runtimeSeconds;
private float nextAudioSyncSeconds = AudioSyncUpdateIntervalSeconds;
private float nextTelemetrySeconds;
private float nextCalibrationChirpSeconds = (float)MimirChirpletCalibrationPhrase.Default.FirstFireSeconds;
private ulong calibrationPhraseIndex;
private readonly Dictionary<ulong, string> calibrationPhraseCache = new();
private IReadOnlyList<MimirAudioSynchronizationReport> lastAudioSynchronizationReports = [];
public MimirRuntime(AquariumRuntimeOptions options)
: this(options, MimirRuntimeConfiguration.Load())
@ -43,6 +48,8 @@ public sealed class MimirRuntime : IAquariumRuntime
Options = options;
visualRuntime = new LocalCastRuntime(options);
synchronization = new MimirSynchronizationHub(settings);
telemetryIntervalSeconds = ParseTelemetryIntervalSeconds();
nextTelemetrySeconds = telemetryIntervalSeconds;
foreach (var source in streamSources)
{
synchronization.AddSource(source);
@ -85,6 +92,8 @@ public sealed class MimirRuntime : IAquariumRuntime
runtimeSeconds += Math.Max(deltaSeconds, 0.0f);
QueueCalibrationChirps();
lastPollCount = synchronization.PollSources();
UpdateAudioSynchronization();
EmitTelemetry();
visualRuntime.Update(deltaSeconds, input);
}
@ -161,6 +170,60 @@ public sealed class MimirRuntime : IAquariumRuntime
return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer));
}
private void UpdateAudioSynchronization()
{
if (runtimeSeconds < nextAudioSyncSeconds)
{
return;
}
lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference);
nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds;
}
private void EmitTelemetry()
{
if (telemetryIntervalSeconds <= 0.0f || runtimeSeconds < nextTelemetrySeconds)
{
return;
}
var loopback = synchronization.Buffers.Buffers.FirstOrDefault(buffer =>
string.Equals(buffer.Descriptor.SourceId, DefaultAudioSyncReference, StringComparison.Ordinal));
var states = synchronization.AudioSynchronizationStates;
Console.WriteLine(
$"mimir-sync-telemetry t={runtimeSeconds:0.00}s loopbackCount={loopback?.Count ?? 0} loopbackEdgeNs={loopback?.EdgeNs ?? 0} reports={lastAudioSynchronizationReports.Count} states={states.Count} aligned={DescribeAlignedAudio()}");
Console.WriteLine($"mimir-sync-buffers {DescribeAudioBuffers()}");
foreach (var report in lastAudioSynchronizationReports.OrderBy(report => report.SourceId, StringComparer.Ordinal))
{
Console.WriteLine(
$"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000}");
}
foreach (var state in states)
{
Console.WriteLine(
$"mimir-sync-state {state.ReferenceSourceId}->{state.SourceId} delaySamples={state.SmoothedDelaySamples:0.000} delayMs={state.DelayMilliseconds:0.000} sroPpm={state.SamplingRateOffsetPpm:0.000} confidence={state.Confidence:0.000}");
}
nextTelemetrySeconds += telemetryIntervalSeconds;
}
private static float ParseTelemetryIntervalSeconds()
{
return float.TryParse(Environment.GetEnvironmentVariable("MIMIR_SYNC_TELEMETRY_SECONDS"), out var seconds)
? Math.Clamp(seconds, 0.0f, 60.0f)
: 0.0f;
}
private string DescribeAudioBuffers()
{
return string.Join(" | ", synchronization.Buffers.Buffers
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio)
.OrderBy(buffer => buffer.Descriptor.SourceId, StringComparer.Ordinal)
.Select(buffer => $"{buffer.Descriptor.SourceId}:{buffer.Count}@{buffer.EdgeNs}"));
}
private static string DescribeBuffer(MimirRollingStreamBuffer buffer)
{
var latest = buffer.Latest;

View file

@ -48,7 +48,7 @@ public sealed class MimirChirpletCalibrationPhrase
public static MimirChirpletCalibrationPhrase ForIndex(ulong phraseIndex) => new(
sampleRate: 48_000,
intervalSeconds: 3.25,
intervalSeconds: 2.25,
firstFireSeconds: 0.75,
gain: 0.13,
tones: GenerateTones(phraseIndex));
@ -161,12 +161,12 @@ public sealed class MimirChirpletCalibrationPhrase
{
var baseTones = new[]
{
new MimirChirpletTone(0.000, 0.070, 7_040.0, 7_680.0, 0.80),
new MimirChirpletTone(0.190, 0.060, 9_600.0, 8_800.0, 0.75),
new MimirChirpletTone(0.410, 0.080, 11_800.0, 12_800.0, 0.90),
new MimirChirpletTone(0.735, 0.065, 8_200.0, 9_000.0, 0.60),
new MimirChirpletTone(0.980, 0.095, 14_500.0, 15_800.0, 0.80),
new MimirChirpletTone(1.310, 0.070, 10_800.0, 10_200.0, 0.70),
new MimirChirpletTone(0.000, 0.060, 7_040.0, 7_680.0, 0.80),
new MimirChirpletTone(0.125, 0.055, 9_600.0, 8_800.0, 0.75),
new MimirChirpletTone(0.260, 0.065, 11_800.0, 12_800.0, 0.90),
new MimirChirpletTone(0.435, 0.055, 8_200.0, 9_000.0, 0.60),
new MimirChirpletTone(0.610, 0.075, 14_500.0, 15_800.0, 0.80),
new MimirChirpletTone(0.790, 0.060, 10_800.0, 10_200.0, 0.70),
};
var tones = new List<MimirChirpletTone>(baseTones.Length);
var previousEnd = -1.0;
@ -174,7 +174,7 @@ public sealed class MimirChirpletCalibrationPhrase
{
var baseTone = baseTones[index];
var seed = Mix(phraseIndex ^ ((ulong)index * 0x9E3779B97F4A7C15UL));
var timeJitter = Unit(seed) * 0.050 - 0.025;
var timeJitter = Unit(seed) * 0.030 - 0.015;
var durationJitter = Unit(seed >> 11) * 0.020 - 0.010;
var bandJitter = Unit(seed >> 22) * 480.0 - 240.0;
var glideScale = 0.88 + Unit(seed >> 33) * 0.24;

View file

@ -37,3 +37,4 @@
{"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."}
{"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."}
{"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."}
{"ts":"2026-05-22T13:12:32+01:00","type":"verification","status":"partial","note":"Actual Mimir.App runtime test with MIMIR_RUNTIME_CONFIG=config/mimir-runtime.probes.example.json and MIMIR_SYNC_TELEMETRY_SECONDS enabled proved Aquarium WASAPI output wakes the Scarlett loopback in the real app path. Headless run showed loopbackCount rising to 57 with mic buffers live; visible-window run showed loopbackCount 77 with Focusrite/Kiyo/Kiyo Pro/PS3 Eye mic buffers live. The runtime now updates sync analysis on a 0.5s cadence and telemetry prints loopback, reports, states, and audio buffer counts. No confident acoustic sync state locked in these app runs: reports were absent until enough phrase audio accumulated, then a live report appeared with confidence 0.000. Treat this as online emission/capture proven, online acoustic lock not yet solved."}

View file

@ -77,7 +77,9 @@ accepted_design:
fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
owns the smoothed per-source belief: latest delay, smoothed delay,
confidence, per-band response evidence, and delay-slope/SRO estimate in
ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
ppm. `MimirRuntime` now runs this analysis online on a fixed cadence and
`MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry.
`MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
provisional loopback-referenced aligned mono frame for channels that clear
the confidence gate.
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
@ -176,16 +178,17 @@ current_status:
Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
next_action: >
Add the synchronization actuator after the current chirplet measurement
loop: smooth delay/SRO estimates, then drive variable-rate resampling and
fractional-delay correction per non-reference stream. Replace the
diagnostic JSON-line process bridge with direct native capture drivers that
push buffer handles into Mimir.Runtime. Current chirplet-backed
Mimir.BufferSmoke proves loopback-referenced aligned audio for Scarlett
loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when 9-16 kHz
calibration chirplets are playing. After a later replug, both PS3 Eye audio
endpoints emitted 480-frame WASAPI blocks into runtime buffers again while
both PS3 Eye camera buffers were live. Treat PS3 Eye audio as
Make the actual Mimir.App path produce a confident online acoustic lock from
the chirplet timeline before adding the synchronization actuator. Current
app testing proves Aquarium WASAPI output wakes Scarlett speaker loopback,
live mic buffers stay populated, and runtime sync analysis runs on cadence,
but the live acoustic report has not cleared the confidence gate. Once that
lock is real, drive variable-rate resampling and fractional-delay correction
per non-reference stream. Replace the diagnostic JSON-line process bridge
with direct native capture drivers that push buffer handles into
Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned
audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when
9-16 kHz calibration chirplets are playing. Treat PS3 Eye audio as
enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from
the Leap USB neighborhood and treat Kiyo Pro as a ~25 fps source until its
SuperSpeed negotiation is fixed.