Bound runtime sync analysis
This commit is contained in:
Meta Vens
parent
c64fd9050e
commit
200186d4d5
8 changed files with 112 additions and 37 deletions
|
|
@ -60,8 +60,10 @@ a named invariant that the native runtime cannot protect yet.
|
|||
- `MimirAudioSynchronizationStateTracker` owns the first smoothed per-source
|
||||
sync state: latest fractional delay, smoothed delay, confidence, per-band
|
||||
response evidence, and delay-slope/SRO estimate in ppm.
|
||||
- `MimirRuntime` updates audio sync analysis online and can emit live sync
|
||||
telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`.
|
||||
- `MimirRuntime` updates audio sync analysis online as a bounded rotating
|
||||
service and can emit live sync telemetry with
|
||||
`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry read cached reports/states;
|
||||
they do not run synchronization analysis.
|
||||
- `MimirRuntime` continuously queues chirplet timeline PCM through Aquarium
|
||||
audio so the loopback and acoustic mic buffers carry a shared timing witness.
|
||||
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
|
||||
|
|
@ -102,7 +104,9 @@ a named invariant that the native runtime cannot protect yet.
|
|||
`MimirAudioSynchronizationState`. The current aligned frame is integer-delay
|
||||
only and should not pretend to be the final DSP path.
|
||||
First, harden the online state filter so the continuous chirplet lock stops
|
||||
producing jumpy delay estimates.
|
||||
producing jumpy delay estimates. The runtime sync cadence is now bounded and
|
||||
survives longer real app runs, but the state filter still accepts outliers
|
||||
too readily.
|
||||
5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
|
||||
source timestamps, and output settings are visible and adjustable.
|
||||
6. Move GPU feature extraction, fusion, material fitting, render budgeting, and
|
||||
|
|
|
|||
|
|
@ -100,13 +100,15 @@ Aquarium audio, uses coarse chirplet energy for acquisition, refines delay from
|
|||
matched event indices when the symbol decoder succeeds, and can build a
|
||||
provisional aligned mono frame for channels that clear the confidence gate.
|
||||
Reports now carry fractional delay and per-band matched energy. The hub also
|
||||
owns smoothed per-source sync state with delay-slope/SRO in ppm. `MimirRuntime`
|
||||
runs the analysis online and can print live telemetry with
|
||||
`MIMIR_SYNC_TELEMETRY_SECONDS`. Current app testing proves loopback wakeup, live
|
||||
mic buffers, and confident online sync states, but delay estimates are still
|
||||
jumpy. Camera mics are spatial/context witnesses; Focusrite devices are dialogue
|
||||
anchors. The current aligned frame is integer-delay only; fractional delay and
|
||||
the hot resampler still belong in Faust/native DSP.
|
||||
owns cached reports plus smoothed per-source sync state with delay-slope/SRO in
|
||||
ppm. `MimirRuntime` runs analysis online as a bounded rotating service and can
|
||||
print live telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are
|
||||
passive readers of cached sync state; they must not invoke the analyzer. Current
|
||||
app testing proves loopback wakeup, live mic buffers, and confident online sync
|
||||
states, but delay estimates are still jumpy. Camera mics are spatial/context
|
||||
witnesses; Focusrite devices are dialogue anchors. The current aligned frame is
|
||||
integer-delay only; fractional delay and the hot resampler still belong in
|
||||
Faust/native DSP.
|
||||
|
||||
## Visual Fusion
|
||||
|
||||
|
|
|
|||
|
|
@ -71,10 +71,12 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
timeline owns Aquarium PCM segment rendering, coarse acquisition,
|
||||
symbol/event decoding, and per-band response hooks. Sync reports now include
|
||||
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
|
||||
now tracks smoothed delay and delay-slope/SRO ppm. Actual Mimir.App testing
|
||||
proves Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and
|
||||
produces confident online sync states, but delay estimates are still jumpy.
|
||||
PS3 Eye audio is
|
||||
now tracks smoothed delay and delay-slope/SRO ppm. `MimirRuntime` runs sync
|
||||
analysis as a bounded rotating service and caches reports/states for UI and
|
||||
telemetry; readouts must stay passive. Actual Mimir.App testing proves
|
||||
Aquarium audio wakes Scarlett loopback, keeps mic buffers live, and produces
|
||||
confident online sync states, but delay estimates are still jumpy. PS3 Eye
|
||||
audio is
|
||||
enumeration/runtime-fragile: one run saw both mic buffers empty while both
|
||||
cameras were live, then a replug made both PS3 Eye mic buffers emit
|
||||
480-frame WASAPI blocks again.
|
||||
|
|
@ -83,4 +85,5 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
|
||||
Harden the online sync state filter, then turn those delay reports into a
|
||||
smoothed SRO/fractional-delay actuator. Keep loopback as timing authority. Do
|
||||
not restore deleted script infrastructure because a stale doc once missed it.
|
||||
not call synchronization analysis from UI/telemetry readouts. Do not restore
|
||||
deleted script infrastructure because a stale doc once missed it.
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ namespace Mimir.Runtime;
|
|||
public sealed class MimirRuntime : IAquariumRuntime
|
||||
{
|
||||
private const string DefaultAudioSyncReference = "loopback-scarlett-speakers";
|
||||
private const float AudioSyncUpdateIntervalSeconds = 0.5f;
|
||||
private const float AudioSyncUpdateIntervalSeconds = 0.1f;
|
||||
private readonly LocalCastRuntime visualRuntime;
|
||||
private readonly MimirSynchronizationHub synchronization;
|
||||
private readonly AquariumUiDocument ui;
|
||||
|
|
@ -164,7 +164,8 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
return;
|
||||
}
|
||||
|
||||
lastAudioSynchronizationReports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds);
|
||||
synchronization.AnalyzeAudioSynchronizationStep(DefaultAudioSyncReference, runtimeSeconds);
|
||||
lastAudioSynchronizationReports = synchronization.AudioSynchronizationReports;
|
||||
nextAudioSyncSeconds += AudioSyncUpdateIntervalSeconds;
|
||||
}
|
||||
|
||||
|
|
@ -229,7 +230,7 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
|
||||
private string DescribeAudioSync()
|
||||
{
|
||||
var reports = synchronization.AnalyzeAudioSynchronization(DefaultAudioSyncReference, runtimeSeconds);
|
||||
var reports = synchronization.AudioSynchronizationReports;
|
||||
return reports.Count == 0
|
||||
? "no payload windows"
|
||||
: string.Join(" | ", reports.Select(report => $"{report.SourceId}: {report.FractionalDelaySamples:0.0} samples {report.DelayMilliseconds:0.00}ms c={report.Confidence:0.00} events={report.TimelineMatchedEvents}"));
|
||||
|
|
@ -237,10 +238,10 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
|
||||
private string DescribeAlignedAudio()
|
||||
{
|
||||
var frame = synchronization.BuildAlignedAudioFrame(DefaultAudioSyncReference);
|
||||
return frame == null
|
||||
? "no aligned frame"
|
||||
: $"{frame.Channels.Count}ch {frame.SampleRate}Hz {frame.FrameCount} frames";
|
||||
var states = synchronization.AudioSynchronizationStates;
|
||||
return states.Count == 0
|
||||
? "no aligned state"
|
||||
: $"{states.Count + 1}ch state-ready";
|
||||
}
|
||||
|
||||
private string DescribeAudioSyncState()
|
||||
|
|
|
|||
|
|
@ -11,7 +11,8 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
public IReadOnlyList<MimirAudioSynchronizationReport> Analyze(
|
||||
IEnumerable<MimirRollingStreamBuffer> buffers,
|
||||
string referenceSourceId,
|
||||
double approximateTimelineSeconds)
|
||||
double approximateTimelineSeconds,
|
||||
IReadOnlySet<string>? candidateSourceIds = null)
|
||||
{
|
||||
var audioBuffers = buffers
|
||||
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio)
|
||||
|
|
@ -47,6 +48,11 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
continue;
|
||||
}
|
||||
|
||||
if (candidateSourceIds != null && !candidateSourceIds.Contains(buffer.Descriptor.SourceId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var commonEndNs = Math.Min(referenceLatest.Value.TimestampNs, buffer.Latest?.TimestampNs ?? 0);
|
||||
var candidateSamples = ExtractMonoWindow(buffer, out var candidateBlock);
|
||||
if (candidateSamples.Length == 0 || candidateBlock == null)
|
||||
|
|
@ -363,7 +369,11 @@ public sealed class MimirAudioSynchronizationAnalyzer
|
|||
return (0.0, 0.0, 0);
|
||||
}
|
||||
|
||||
var candidateByEvent = candidateObservations.ToDictionary(observation => observation.EventIndex);
|
||||
var candidateByEvent = candidateObservations
|
||||
.GroupBy(observation => observation.EventIndex)
|
||||
.ToDictionary(
|
||||
group => group.Key,
|
||||
group => group.OrderByDescending(observation => observation.Energy).First());
|
||||
var delays = new List<(double Delay, double Weight)>();
|
||||
foreach (var reference in referenceObservations)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -5,7 +5,9 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
private readonly List<IMimirStreamSource> sources = [];
|
||||
private readonly MimirAudioSynchronizationAnalyzer audioSynchronization = new();
|
||||
private readonly MimirAudioSynchronizationStateTracker audioSynchronizationState = new();
|
||||
private readonly Dictionary<string, MimirAudioSynchronizationReport> audioSynchronizationReports = new(StringComparer.Ordinal);
|
||||
private ulong ingestedSamples;
|
||||
private int nextAudioSynchronizationCandidate;
|
||||
|
||||
public MimirSynchronizationHub(MimirSynchronizationSettings settings)
|
||||
{
|
||||
|
|
@ -28,6 +30,9 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
public IReadOnlyList<MimirAudioSynchronizationState> AudioSynchronizationStates =>
|
||||
audioSynchronizationState.States;
|
||||
|
||||
public IReadOnlyList<MimirAudioSynchronizationReport> AudioSynchronizationReports =>
|
||||
audioSynchronizationReports.Values.OrderBy(report => report.SourceId, StringComparer.Ordinal).ToArray();
|
||||
|
||||
public void AddSource(IMimirStreamSource source)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
|
|
@ -68,7 +73,42 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
var reports = audioSynchronization.Analyze(Buffers.Buffers, referenceSourceId, approximateTimelineSeconds);
|
||||
audioSynchronizationState.Update(reports);
|
||||
StoreAudioSynchronizationReports(reports);
|
||||
return reports;
|
||||
}
|
||||
|
||||
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronizationStep(
|
||||
string referenceSourceId,
|
||||
double approximateTimelineSeconds,
|
||||
int maxCandidates = 1)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
var candidates = Buffers.Buffers
|
||||
.Where(buffer =>
|
||||
buffer.Descriptor.Kind == MimirStreamKind.Audio &&
|
||||
!string.Equals(buffer.Descriptor.SourceId, referenceSourceId, StringComparison.Ordinal) &&
|
||||
buffer.Latest?.AudioBlock != null)
|
||||
.OrderBy(buffer => buffer.Descriptor.SourceId, StringComparer.Ordinal)
|
||||
.Select(buffer => buffer.Descriptor.SourceId)
|
||||
.ToArray();
|
||||
if (candidates.Length == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
var selected = new HashSet<string>(StringComparer.Ordinal);
|
||||
for (var count = 0; count < Math.Min(maxCandidates, candidates.Length); count++)
|
||||
{
|
||||
selected.Add(candidates[nextAudioSynchronizationCandidate % candidates.Length]);
|
||||
nextAudioSynchronizationCandidate++;
|
||||
}
|
||||
|
||||
var reports = audioSynchronization.Analyze(
|
||||
Buffers.Buffers,
|
||||
referenceSourceId,
|
||||
approximateTimelineSeconds,
|
||||
selected);
|
||||
StoreAudioSynchronizationReports(reports);
|
||||
return reports;
|
||||
}
|
||||
|
||||
|
|
@ -87,6 +127,16 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
|
||||
private bool disposed;
|
||||
|
||||
private void StoreAudioSynchronizationReports(IReadOnlyList<MimirAudioSynchronizationReport> reports)
|
||||
{
|
||||
foreach (var report in reports)
|
||||
{
|
||||
audioSynchronizationReports[report.SourceId] = report;
|
||||
}
|
||||
|
||||
audioSynchronizationState.Update(reports);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (disposed)
|
||||
|
|
|
|||
|
|
@ -42,3 +42,4 @@
|
|||
{"ts":"2026-05-22T14:10:00+01:00","type":"implementation","status":"partial","note":"Changed MimirChirpletTimeline from anonymous deterministic texture to an explicit order-3 de Bruijn symbol timeline over 32 separated chirp symbols. Any three consecutive correctly detected symbols identify a timeline event within the current operating horizon; reports now expose timelineMatchedEvents and timelineConfidence when event-index refinement is used. A synthetic check rendered two seconds of canonical timeline audio and decoded events 0, 1, and 2 back to a window start of -0.0015 seconds. Real headless app sniff did not provide a clean physical proof because Scarlett loopback capture again stopped advancing early; build remains clean."}
|
||||
{"ts":"2026-05-22T14:32:00+01:00","type":"implementation","status":"ok","note":"Reworked MimirChirpletTimeline symbols from fixed frequency slots into time/frequency constellations. The order-3 de Bruijn sequence still gives three-chirp timeline placement, but each symbol now carries identity through start band, glide shape, duration, and following inter-chirp gap. Event start times are precomputed over the de Bruijn period instead of using a fixed stride. Synthetic verification rendered two seconds of canonical timeline audio and decoded interior events 9, 10, and 11 back to a zero-second window start, proving placement does not rely on the opening triplet."}
|
||||
{"ts":"2026-05-22T14:48:00+01:00","type":"verification","status":"partial","note":"Attempted a 20-second real-world Mimir.App chirplet sync run with all configured local mics. The headless path only emitted one telemetry tick before runtime updates stopped. Switching to the normal app path produced telemetry through t=4.26s with all six audio buffers live and sync states for Focusrite, Kiyo, Kiyo Pro, and both PS3 Eye mics. Timeline-symbol refinement appeared on Focusrite, Kiyo, and one PS3 Eye, but delay estimates were unstable and the app did not continue telemetry to 20 seconds. Replaced the coarse acquisition trace with envelope energy to avoid matched-filter scans in the hot acquisition path; symbol decoding remains the refinement step."}
|
||||
{"ts":"2026-05-22T15:20:00+01:00","type":"implementation","status":"partial","note":"Removed hidden re-entrant synchronization analysis from the live app path. MimirRuntime now calls a bounded rotating AnalyzeAudioSynchronizationStep every 0.1s, MimirSynchronizationHub caches per-source reports/states, and UI/telemetry read cached state instead of invoking the analyzer or BuildAlignedAudioFrame. FitDelayFromEvents now collapses duplicate candidate event observations by strongest energy, fixing the real app crash 'An item with the same key has already been added. Key: 0'. Dotnet build passed. A visible Mimir.App run with all configured mics no longer crashed at the earlier 4s wall and produced telemetry through runtime t=17.28s in a 22s wall-time harness; it did not reach a full 20 runtime seconds because startup consumed wall time. All non-loopback mics produced cached reports/states, but delay estimates remained unstable and timeline event refinement was intermittent, so the next live invariant is robust outlier rejection/state filtering before any actuator drives resampling."}
|
||||
|
|
|
|||
|
|
@ -71,16 +71,19 @@ accepted_design:
|
|||
timing atoms, symbol/event decoding, and per-band response kernels. Symbol
|
||||
identity is carried by start band, glide shape, duration, and following
|
||||
inter-chirp gap. Any three consecutive correctly detected symbols identify a
|
||||
timeline event within the current operating horizon. `MimirAudioSynchronizationAnalyzer` now treats
|
||||
`loopback-scarlett-speakers` as timing authority, uses coarse chirplet
|
||||
energy for acquisition, and refines delay from matched timeline event
|
||||
indices when symbol decoding succeeds. Positive delay means the
|
||||
timeline event within the current operating horizon. `MimirAudioSynchronizationAnalyzer`
|
||||
now treats `loopback-scarlett-speakers` as timing authority, uses coarse
|
||||
chirplet energy for acquisition, and refines delay from matched timeline
|
||||
event indices when symbol decoding succeeds. Positive delay means the
|
||||
candidate mic is late relative to loopback. Reports include rounded delay,
|
||||
fractional delay, confidence, and per-band matched energy. `MimirAudioSynchronizationStateTracker`
|
||||
fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub`
|
||||
caches the latest per-source reports and `MimirAudioSynchronizationStateTracker`
|
||||
owns the smoothed per-source belief: latest delay, smoothed delay,
|
||||
confidence, per-band response evidence, and delay-slope/SRO estimate in
|
||||
ppm. `MimirRuntime` now runs this analysis online on a fixed cadence and
|
||||
`MIMIR_SYNC_TELEMETRY_SECONDS` prints live buffer/report/state telemetry.
|
||||
ppm. `MimirRuntime` now runs this analysis online as a bounded rotating
|
||||
service on a fixed cadence, while UI and telemetry read cached reports/states
|
||||
instead of invoking analysis. `MIMIR_SYNC_TELEMETRY_SECONDS` prints live
|
||||
buffer/report/state telemetry.
|
||||
`MimirSynchronizationHub.BuildAlignedAudioFrame` exposes the first
|
||||
provisional loopback-referenced aligned mono frame for channels that clear
|
||||
the confidence gate.
|
||||
|
|
@ -182,11 +185,12 @@ current_status:
|
|||
next_action: >
|
||||
Harden the online state filter before adding the synchronization actuator.
|
||||
Current app testing proves Aquarium WASAPI output wakes Scarlett speaker
|
||||
loopback, live mic buffers stay populated, runtime sync analysis runs on
|
||||
cadence, and the continuous chirplet timeline can produce confident sync
|
||||
states in the app. Delay estimates are still jumpy, so the actuator needs
|
||||
outlier rejection and better SRO smoothing before it drives variable-rate
|
||||
resampling and fractional-delay correction per non-reference stream. Replace
|
||||
loopback, live mic buffers stay populated, runtime sync analysis now runs as
|
||||
a bounded rotating service without UI/telemetry re-entry, and the continuous
|
||||
chirplet timeline can produce confident sync states in the app. Delay
|
||||
estimates are still jumpy, so the actuator needs outlier rejection and better
|
||||
SRO smoothing before it drives variable-rate resampling and fractional-delay
|
||||
correction per non-reference stream. Replace
|
||||
the diagnostic JSON-line process bridge with direct native capture drivers that push buffer handles into
|
||||
Mimir.Runtime. Mimir.BufferSmoke still proves loopback-referenced aligned
|
||||
audio for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when
|
||||
|
|
|
|||
Loading…
Reference in a new issue