Test live runtime chirplet sync
This commit is contained in:
parent
7f804ad5da
commit
da9915b6e5
8 changed files with 116 additions and 33 deletions
|
|
@ -53,8 +53,8 @@ audio source. `MimirChirpletCalibrationPhrase` owns the emitted calibration
|
|||
phrase and the matched-filter shape used to analyze it. The default timeline is
|
||||
a stateless 16-phrase cycle. Phrase `N` is generated directly from `N`, with no
|
||||
registry or remembered random state. Each phrase is a spread-out asymmetric
|
||||
motif: six short chirplets over about 1.38 seconds, with nonuniform gaps and
|
||||
high-frequency bands. A new phrase fires every 3.25 seconds through Aquarium
|
||||
motif: six short chirplets over about 0.85 seconds, with nonuniform gaps and
|
||||
high-frequency bands. A new phrase fires every 2.25 seconds through Aquarium
|
||||
audio. The point is not ornament. The timing code is carried by both frequency
|
||||
and rhythm, so it behaves more like a small birdsong signature than a repeated
|
||||
sweep. The indexed phrase sequence has lower ambiguity when remote feeds add
|
||||
|
|
@ -91,6 +91,13 @@ for the coming actuator. The state can survive a brief weak report, but it is
|
|||
not a license to run blind: loopback must keep receiving the emitted phrase or
|
||||
fresh reports will stop.
|
||||
|
||||
The actual Mimir app path now runs this online: `MimirRuntime.Update` emits the
|
||||
phrase sequence, polls sources, and updates sync analysis on a fixed cadence.
|
||||
`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current
|
||||
runtime testing proves Aquarium output wakes the Scarlett loopback and the mic
|
||||
buffers stay live, but the app has not yet produced a confident acoustic lock.
|
||||
That next failure is calibration, not plumbing.
|
||||
|
||||
## Chirplet Calibration Model
|
||||
|
||||
```mermaid
|
||||
|
|
|
|||
|
|
@ -45,8 +45,8 @@ a named invariant that the native runtime cannot protect yet.
|
|||
- `MimirChirpletCalibrationPhrase` owns the structured birdsong-like calibration
|
||||
phrase, PCM rendering, matched timing trace, and per-band response kernels.
|
||||
The default timeline is a stateless 16-phrase cycle; phrase `N` is generated
|
||||
from `N`, spreads six chirplets across about 1.38 seconds, and fires every
|
||||
3.25 seconds to reduce repeated-sweep ambiguity.
|
||||
from `N`, spreads six chirplets across about 0.85 seconds, and fires every
|
||||
2.25 seconds to reduce repeated-sweep ambiguity.
|
||||
- `MimirAudioSynchronizationAnalyzer` ports the first live sync measurement:
|
||||
sample-bearing audio blocks are resampled into the Scarlett loopback timeline
|
||||
and compared by chirplet-energy delay estimation with fractional peak
|
||||
|
|
@ -57,6 +57,8 @@ a named invariant that the native runtime cannot protect yet.
|
|||
- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source
|
||||
sync state: latest fractional delay, smoothed delay, confidence, per-band
|
||||
response evidence, and delay-slope/SRO estimate in ppm.
|
||||
- `MimirRuntime` updates audio sync analysis online and can emit live sync
|
||||
telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`.
|
||||
- `MimirRuntime` emits short calibration chirplets through Aquarium audio so the
|
||||
loopback and acoustic mic buffers carry a shared timing witness.
|
||||
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
|
||||
|
|
@ -96,6 +98,8 @@ a named invariant that the native runtime cannot protect yet.
|
|||
fractional delay line per non-reference stream from the smoothed
|
||||
`MimirAudioSynchronizationState`. The current aligned frame is integer-delay
|
||||
only and should not pretend to be the final DSP path.
|
||||
First, make the live app acoustic lock robust enough to populate sync state
|
||||
from the online chirplet timeline, not just from smoke harnesses.
|
||||
5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
|
||||
source timestamps, and output settings are visible and adjustable.
|
||||
6. Move GPU feature extraction, fusion, material fitting, render budgeting, and
|
||||
|
|
|
|||
|
|
@ -93,12 +93,15 @@ The six-microphone path is separate from the bridge. Scarlett speaker loopback
|
|||
is the current timing authority when calibration chirplets are playing.
|
||||
`MimirChirpletCalibrationPhrase` owns the birdsong-like timeline fingerprint:
|
||||
a stateless 16-phrase cycle where phrase `N` is generated from `N`; each phrase
|
||||
spreads six asymmetric chirplets across about 1.38 seconds and fires every 3.25
|
||||
spreads six asymmetric chirplets across about 0.85 seconds and fires every 2.25
|
||||
seconds. Mimir emits that phrase sequence through Aquarium audio, compares mic
|
||||
buffers against loopback by chirplet-energy delay estimation, and can build a
|
||||
provisional aligned mono frame for channels that clear the confidence gate.
|
||||
Reports now carry fractional delay and per-band matched energy. The hub also
|
||||
owns smoothed per-source sync state with delay-slope/SRO in ppm. Camera mics are
|
||||
owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime`
|
||||
runs the analysis online and can print live telemetry with
|
||||
`MIMIR_SYNC_TELEMETRY_SECONDS`. Current app testing proves loopback wakeup and
|
||||
live mic buffers, but not yet a confident acoustic lock. Camera mics are
|
||||
spatial/context witnesses; Focusrite devices are dialogue anchors. The current
|
||||
aligned frame is integer-delay only; fractional delay and the hot resampler
|
||||
still belong in Faust/native DSP.
|
||||
|
|
|
|||
|
|
@ -65,18 +65,20 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
- Current chirplet-backed smoke uses Scarlett speaker loopback as timing
|
||||
authority. `MimirChirpletCalibrationPhrase` owns the structured birdsong-like
|
||||
timeline fingerprint: a stateless 16-phrase cycle where phrase `N` is
|
||||
generated from `N`; each phrase has six asymmetric chirplets over about 1.38
|
||||
seconds and fires every 3.25 seconds. The phrase owns Aquarium PCM rendering,
|
||||
generated from `N`; each phrase has six asymmetric chirplets over about 0.85
|
||||
seconds and fires every 2.25 seconds. The phrase owns Aquarium PCM rendering,
|
||||
matched timing trace, and per-band response hooks. Sync reports now include
|
||||
fractional delay and per-band matched energy. `MimirAudioSynchronizationState` now tracks smoothed delay and
|
||||
delay-slope/SRO ppm. PS3 Eye audio is
|
||||
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
|
||||
now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing
|
||||
proves Aquarium audio wakes Scarlett loopback and keeps mic buffers live, but
|
||||
the online acoustic report has not cleared the confidence gate yet. PS3 Eye audio is
|
||||
enumeration/runtime-fragile: one run saw both mic buffers empty while both
|
||||
cameras were live, then a replug made both PS3 Eye mic buffers emit
|
||||
480-frame WASAPI blocks again.
|
||||
|
||||
## Immediate Re-entry Instruction
|
||||
|
||||
Replace the diagnostic audio JSON process bridge with native audio capture
|
||||
workers, then turn chirplet delay reports into a smoothed SRO/fractional-delay
|
||||
actuator. Keep loopback as timing authority. Do not restore deleted script
|
||||
infrastructure because a stale doc once missed it.
|
||||
Make the actual Mimir.App path produce a confident online acoustic lock from the
|
||||
chirplet timeline, then turn those delay reports into a smoothed
|
||||
SRO/fractional-delay actuator. Keep loopback as timing authority. Do not restore
|
||||
deleted script infrastructure because a stale doc once missed it.
|
||||
|
|
|
|||
|
|
@ -11,14 +11,19 @@ namespace Mimir.Runtime;
|
|||
public sealed class MimirRuntime : IAquariumRuntime
|
||||
{
|
||||
private const string DefaultAudioSyncReference = "loopback-scarlett-speakers";
|
||||
private const float AudioSyncUpdateIntervalSeconds = 0.5f;
|
||||
private readonly LocalCastRuntime visualRuntime;
|
||||
private readonly MimirSynchronizationHub synchronization;
|
||||
private readonly AquariumUiDocument ui;
|
||||
private readonly float telemetryIntervalSeconds;
|
||||
private int lastPollCount;
|
||||
private float runtimeSeconds;
|
||||
private float nextAudioSyncSeconds = AudioSyncUpdateIntervalSeconds;
|
||||
private float nextTelemetrySeconds;
|
||||
private float nextCalibrationChirpSeconds = (float)MimirChirpletCalibrationPhrase.Default.FirstFireSeconds;
|
||||
private ulong calibrationPhraseIndex;
|
||||
private readonly Dictionary<ulong, string> calibrationPhraseCache = new();
|
||||
private IReadOnlyList<MimirAudioSynchronizationReport> lastAudioSynchronizationReports = [];
|
||||
|
||||
public MimirRuntime(AquariumRuntimeOptions options)
|
||||
: this(options, MimirRuntimeConfiguration.Load())
|
||||
|
|
@ -43,6 +48,8 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
Options = options;
|
||||
visualRuntime = new LocalCastRuntime(options);
|
||||
synchronization = new MimirSynchronizationHub(settings);
|
||||
telemetryIntervalSeconds = ParseTelemetryIntervalSeconds();
|
||||
nextTelemetrySeconds = telemetryIntervalSeconds;
|
||||
foreach (var source in streamSources)
|
||||
{
|
||||
synchronization.AddSource(source);
|
||||
|
|
@ -85,6 +92,8 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
runtimeSeconds += Math.Max(deltaSeconds, 0.0f);
|
||||
QueueCalibrationChirps();
|
||||
lastPollCount = synchronization.PollSources();
|
||||
UpdateAudioSynchronization();
|
||||
EmitTelemetry();
|
||||
visualRuntime.Update(deltaSeconds, input);
|
||||
}
|
||||
|
||||
|
|
@ -161,6 +170,60 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer));
|
||||
}
|
||||
|
||||
private void UpdateAudioSynchronization()
|
||||
{
|
||||
if (runtimeSeconds < nextAudioSyncSeconds)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference);
|
||||
nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds;
|
||||
}
|
||||
|
||||
private void EmitTelemetry()
|
||||
{
|
||||
if (telemetryIntervalSeconds <= 0.0f || runtimeSeconds < nextTelemetrySeconds)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var loopback = synchronization.Buffers.Buffers.FirstOrDefault(buffer =>
|
||||
string.Equals(buffer.Descriptor.SourceId, DefaultAudioSyncReference, StringComparison.Ordinal));
|
||||
var states = synchronization.AudioSynchronizationStates;
|
||||
Console.WriteLine(
|
||||
$"mimir-sync-telemetry t={runtimeSeconds:0.00}s loopbackCount={loopback?.Count ?? 0} loopbackEdgeNs={loopback?.EdgeNs ?? 0} reports={lastAudioSynchronizationReports.Count} states={states.Count} aligned={DescribeAlignedAudio()}");
|
||||
Console.WriteLine($"mimir-sync-buffers {DescribeAudioBuffers()}");
|
||||
foreach (var report in lastAudioSynchronizationReports.OrderBy(report => report.SourceId, StringComparer.Ordinal))
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"mimir-sync-report {report.ReferenceSourceId}->{report.SourceId} delaySamples={report.FractionalDelaySamples:0.000} delayMs={report.DelayMilliseconds:0.000} confidence={report.Confidence:0.000}");
|
||||
}
|
||||
|
||||
foreach (var state in states)
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"mimir-sync-state {state.ReferenceSourceId}->{state.SourceId} delaySamples={state.SmoothedDelaySamples:0.000} delayMs={state.DelayMilliseconds:0.000} sroPpm={state.SamplingRateOffsetPpm:0.000} confidence={state.Confidence:0.000}");
|
||||
}
|
||||
|
||||
nextTelemetrySeconds += telemetryIntervalSeconds;
|
||||
}
|
||||
|
||||
private static float ParseTelemetryIntervalSeconds()
|
||||
{
|
||||
return float.TryParse(Environment.GetEnvironmentVariable("MIMIR_SYNC_TELEMETRY_SECONDS"), out var seconds)
|
||||
? Math.Clamp(seconds, 0.0f, 60.0f)
|
||||
: 0.0f;
|
||||
}
|
||||
|
||||
private string DescribeAudioBuffers()
|
||||
{
|
||||
return string.Join(" | ", synchronization.Buffers.Buffers
|
||||
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio)
|
||||
.OrderBy(buffer => buffer.Descriptor.SourceId, StringComparer.Ordinal)
|
||||
.Select(buffer => $"{buffer.Descriptor.SourceId}:{buffer.Count}@{buffer.EdgeNs}"));
|
||||
}
|
||||
|
||||
private static string DescribeBuffer(MimirRollingStreamBuffer buffer)
|
||||
{
|
||||
var latest = buffer.Latest;
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ public sealed class MimirChirpletCalibrationPhrase
|
|||
|
||||
public static MimirChirpletCalibrationPhrase ForIndex(ulong phraseIndex) => new(
|
||||
sampleRate: 48_000,
|
||||
intervalSeconds: 3.25,
|
||||
intervalSeconds: 2.25,
|
||||
firstFireSeconds: 0.75,
|
||||
gain: 0.13,
|
||||
tones: GenerateTones(phraseIndex));
|
||||
|
|
@ -161,12 +161,12 @@ public sealed class MimirChirpletCalibrationPhrase
|
|||
{
|
||||
var baseTones = new[]
|
||||
{
|
||||
new MimirChirpletTone(0.000, 0.070, 7_040.0, 7_680.0, 0.80),
|
||||
new MimirChirpletTone(0.190, 0.060, 9_600.0, 8_800.0, 0.75),
|
||||
new MimirChirpletTone(0.410, 0.080, 11_800.0, 12_800.0, 0.90),
|
||||
new MimirChirpletTone(0.735, 0.065, 8_200.0, 9_000.0, 0.60),
|
||||
new MimirChirpletTone(0.980, 0.095, 14_500.0, 15_800.0, 0.80),
|
||||
new MimirChirpletTone(1.310, 0.070, 10_800.0, 10_200.0, 0.70),
|
||||
new MimirChirpletTone(0.000, 0.060, 7_040.0, 7_680.0, 0.80),
|
||||
new MimirChirpletTone(0.125, 0.055, 9_600.0, 8_800.0, 0.75),
|
||||
new MimirChirpletTone(0.260, 0.065, 11_800.0, 12_800.0, 0.90),
|
||||
new MimirChirpletTone(0.435, 0.055, 8_200.0, 9_000.0, 0.60),
|
||||
new MimirChirpletTone(0.610, 0.075, 14_500.0, 15_800.0, 0.80),
|
||||
new MimirChirpletTone(0.790, 0.060, 10_800.0, 10_200.0, 0.70),
|
||||
};
|
||||
var tones = new List<MimirChirpletTone>(baseTones.Length);
|
||||
var previousEnd = -1.0;
|
||||
|
|
@ -174,7 +174,7 @@ public sealed class MimirChirpletCalibrationPhrase
|
|||
{
|
||||
var baseTone = baseTones[index];
|
||||
var seed = Mix(phraseIndex ^ ((ulong)index * 0x9E3779B97F4A7C15UL));
|
||||
var timeJitter = Unit(seed) * 0.050 - 0.025;
|
||||
var timeJitter = Unit(seed) * 0.030 - 0.015;
|
||||
var durationJitter = Unit(seed >> 11) * 0.020 - 0.010;
|
||||
var bandJitter = Unit(seed >> 22) * 480.0 - 240.0;
|
||||
var glideScale = 0.88 + Unit(seed >> 33) * 0.24;
|
||||
|
|
|
|||
|
|
@ -37,3 +37,4 @@
|
|||
{"ts":"2026-05-22T12:42:00+01:00","type":"verification","status":"ok","note":"Built Mimir.slnx with zero warnings after structured chirplet changes. A 4-second Mimir.BufferSmoke run against the full probe config printed fractional delays and band energy, and built a 5-channel aligned frame: loopback plus mic-focusrite-local fractionalDelaySamples=2239.233 confidence=0.105, mic-kiyo fractionalDelaySamples=1859.505 confidence=0.496, mic-usb-camera-0 fractionalDelaySamples=2338.772 confidence=0.395, and mic-usb-camera-1 fractionalDelaySamples=1934.795 confidence=0.183. This run verifies report plumbing; the final response normalizer and fractional-delay actuator are still future work."}
|
||||
{"ts":"2026-05-22T12:46:18+01:00","type":"implementation","status":"ok","note":"Added MimirAudioSynchronizationStateTracker and MimirAudioSynchronizationState. The hub now updates confidence-gated per-source sync state from chirplet reports, smoothing fractional delay and estimating delay-slope/SRO in ppm. Mimir.BufferSmoke polls sync during the run and prints sync-state lines; Aquarium UI exposes the smoothed state. Verification built cleanly and a phrase-backed smoke printed sync-state loopback-scarlett-speakers->mic-usb-camera-0 smoothedDelaySamples=47.397 delayMs=1.075 sroPpm=0.000 confidence=0.216. Final fresh reports can still disappear when loopback rendering stops or falls outside the shared buffer edge, so active loopback remains a live precondition."}
|
||||
{"ts":"2026-05-22T12:53:02+01:00","type":"implementation","status":"ok","note":"Changed chirplet calibration from a single repeated motif to a stateless timeline fingerprint. MimirChirpletCalibrationPhrase.ForIndex(N) deterministically generates phrase N; runtime emits N mod 16 every 3.25 seconds, and the analyzer searches the 16-phrase cycle. Each phrase is a birdsong-like asymmetric motif with six chirplets spread over about 1.38 seconds. Mimir.slnx built cleanly. A short BufferSmoke run completed without analyzer errors, but loopback emitted zero packets during that smoke, so it verified plumbing only, not fresh delay estimates."}
|
||||
{"ts":"2026-05-22T13:12:32+01:00","type":"verification","status":"partial","note":"Actual Mimir.App runtime test with MIMIR_RUNTIME_CONFIG=config/mimir-runtime.probes.example.json and MIMIR_SYNC_TELEMETRY_SECONDS enabled proved Aquarium WASAPI output wakes the Scarlett loopback in the real app path. Headless run showed loopbackCount rising to 57 with mic buffers live; visible-window run showed loopbackCount 77 with Focusrite/Kiyo/Kiyo Pro/PS3 Eye mic buffers live. The runtime now updates sync analysis on a 0.5s cadence and telemetry prints loopback, reports, states, and audio buffer counts. No confident acoustic sync state locked in these app runs: reports were absent until enough phrase audio accumulated, then a live report appeared with confidence 0.000. Treat this as online emission/capture proven, online acoustic lock not yet solved."}
|
||||
|
|
|
|||
|
|
@ -77,7 +77,9 @@ accepted_design:
|
|||
fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
|
||||
owns the smoothed per-source belief: latest delay, smoothed delay,
|
||||
confidence, per-band response evidence, and delay-slope/SRO estimate in
|
||||
ppm. `MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
||||
ppm. `MimirRuntime` now runs this analysis online on a fixed cadence and
|
||||
`MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry.
|
||||
`MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
||||
provisional loopback-referenced aligned mono frame for channels that clear
|
||||
the confidence gate.
|
||||
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
||||
|
|
@ -176,16 +178,17 @@ current_status:
|
|||
Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
|
||||
the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
|
||||
next_action: >
|
||||
Add the synchronization actuator after the current chirplet measurement
|
||||
loop: smooth delay/SRO estimates, then drive variable-rate resampling and
|
||||
fractional-delay correction per non-reference stream. Replace the
|
||||
diagnostic JSON-line process bridge with direct native capture drivers that
|
||||
push buffer handles into Mimir.Runtime. Current chirplet-backed
|
||||
Mimir.BufferSmoke proves loopback-referenced aligned audio for Scarlett
|
||||
loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when 9-16 kHz
|
||||
calibration chirplets are playing. After a later replug, both PS3 Eye audio
|
||||
endpoints emitted 480-frame WASAPI blocks into runtime buffers again while
|
||||
both PS3 Eye camera buffers were live. Treat PS3 Eye audio as
|
||||
Make the actual Mimir.App path produce a confident online acoustic lock from
|
||||
the chirplet timeline before adding the synchronization actuator. Current
|
||||
app testing proves Aquarium WASAPI output wakes Scarlett speaker loopback,
|
||||
live mic buffers stay populated, and runtime sync analysis runs on cadence,
|
||||
but the live acoustic report has not cleared the confidence gate. Once that
|
||||
lock is real, drive variable-rate resampling and fractional-delay correction
|
||||
per non-reference stream. Replace the diagnostic JSON-line process bridge
|
||||
with direct native capture drivers that push buffer handles into
|
||||
Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned
|
||||
audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when
|
||||
9-16 kHz calibration chirplets are playing. Treat PS3 Eye audio as
|
||||
enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from
|
||||
the Leap USB neighborhood and treat Kiyo Pro as a ~25 fps source until its
|
||||
SuperSpeed negotiation is fixed.
|
||||
|
|
|
|||
Loading…
Reference in a new issue