Add audio synchronization mode policy

This commit is contained in:
Meta Vens 2026-05-22 18:49:01 +01:00
commit 7e4348bfe7
12 changed files with 205 additions and 18 deletions

View file

@ -1,5 +1,10 @@
{
"bufferSeconds": 5.0,
"audioSync": {
"mode": "hybrid",
"referenceSourceId": "loopback-scarlett-speakers",
"calibrationGain": 2.0
},
"streams": [
{
"sourceId": "leap-stereo-ir",

View file

@ -1,5 +1,10 @@
{
"bufferSeconds": 5.0,
"audioSync": {
"mode": "hybrid",
"referenceSourceId": "loopback-scarlett-speakers",
"calibrationGain": 2.0
},
"streams": [
{
"sourceId": "local-ks-probe",

View file

@ -1,5 +1,10 @@
{
"bufferSeconds": 5.0,
"audioSync": {
"mode": "chirp-only",
"referenceSourceId": "loopback-scarlett-speakers",
"calibrationGain": 2.0
},
"streams": [
{
"sourceId": "mic-focusrite-local",

View file

@ -39,6 +39,30 @@ flowchart TD
- Probe signals are budgeted telemetry, not a permanent audio bed.
- Faust/native DSP owns the hot separation and spatialization graph.
## Synchronization Modes
`MimirAudioSynchronizationSettings.Mode` is the runtime authority for what
timing evidence Mimir is allowed to emit:
- `chirp-only`: emit the deterministic calibration timeline and decode timing
only from that active witness. This is the lab/debug mode and the fallback for
silent program material.
- `no-chirp`: do not emit calibration audio. This is the passive/program-audio
mode. The runtime exposes it now, but the passive loopback/music estimator is
not wired yet, so current code intentionally reports that no-chirp sync is
unavailable instead of pretending the chirplet decoder can read arbitrary
music.
- `hybrid`: prefer passive program-audio evidence when confidence is high, then
emit a shaped watermark when confidence falls. Today this still behaves as an
active pilot because the passive estimator and confidence gate have not been
cut. The intended watermark is the dechirp/FFT-friendly chirp-bin codebook
from the decoder research notes, shaped and low-gain enough to sit inside the
program audio instead of announcing itself like a lab sweep.
The mode belongs to the runtime, not the decoder. The decoder should consume
known timing evidence; it should not decide whether Mimir is allowed to make
sound.
## Next Cut
The current diagnostic witness is `native/probes/wasapi_audio_cadence`, which

View file

@ -72,6 +72,10 @@ a named invariant that the native runtime cannot protect yet.
they do not run synchronization analysis.
- `MimirRuntime` continuously queues chirplet timeline PCM through Aquarium
audio so the loopback and acoustic mic buffers carry a shared timing witness.
`MimirAudioSynchronizationSettings.Mode` selects `chirp-only`, `no-chirp`, or
`hybrid`; no-chirp disables active emission until the passive program-audio
estimator exists, and hybrid currently uses the active pilot as the fallback
path.
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
timestamp, and native/GPU handle metadata.
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
@ -111,11 +115,15 @@ a named invariant that the native runtime cannot protect yet.
decoder through real loopback and microphone paths so every correctly heard
triplet becomes a deterministic timeline anchor before the actuator moves
samples.
5. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
5. Replace the active-pilot-only hybrid path with passive loopback/program-audio
confidence plus shaped watermark emission when confidence drops. The
watermark codebook should be designed for dechirp/FFT or Goertzel
classification, not dense sliding matched filters.
6. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
source timestamps, and output settings are visible and adjustable.
6. Move GPU feature extraction, fusion, material fitting, render budgeting, and
7. Move GPU feature extraction, fusion, material fitting, render budgeting, and
Spout2 publication into Aquarium.
7. Move mic alignment, room suppression, voice separation, spatialization, and
8. Move mic alignment, room suppression, voice separation, spatialization, and
stem generation into Faust/native DSP.
8. Keep the OBS bridge witness ledger as evidence before expanding receiver
9. Keep the OBS bridge witness ledger as evidence before expanding receiver
machinery.

View file

@ -97,6 +97,10 @@ consecutive correctly detected symbols identify the event index inside the
current operating horizon. Mimir continuously queues that timeline through
Aquarium audio and decodes timing through the constrained chirplet-transform
path.
`MimirAudioSynchronizationSettings.Mode` chooses whether active calibration is
allowed: `chirp-only` emits the witness, `no-chirp` stays silent and waits for
the passive program-audio estimator, and `hybrid` currently uses the active
pilot until the passive confidence gate and shaped watermark are built.
Reports now carry fractional delay and per-band matched energy. The first
`MimirChirpletSymbolCodebook` owns separable symbol definitions; every symbol
has a unique chirp shape, with rhythm as additional evidence. `MimirChirpletStreamDecoder`

View file

@ -80,7 +80,10 @@ Get-Content .\state\evidence.jsonl -Tail 8
it currently detects all 15 emitted chirps and recovers 13 anchors for events
0-12 with a 47999.999990 Hz clock fit and 0.000014 sample MAE. `MimirRuntime` runs sync analysis as a bounded rotating
service and caches reports/states for UI and telemetry; readouts must stay
passive. Actual Mimir.App testing proves Aquarium audio wakes Scarlett
passive. Audio sync mode is runtime-owned: `chirp-only` emits active
calibration, `no-chirp` disables emission and awaits passive program-audio
sync, and `hybrid` currently uses the active pilot until shaped watermark
confidence gating exists. Actual Mimir.App testing proves Aquarium audio wakes Scarlett
loopback, keeps mic buffers live, and produces confident online sync states,
but the decoder still needs stronger symbol likelihoods/codebook distance
before arbitrary real triplets can drive the actuator directly. PS3 Eye audio is

View file

@ -11,14 +11,13 @@ namespace Mimir.Runtime;
public sealed class MimirRuntime : IAquariumRuntime
{
private const string DefaultAudioSyncReference = "loopback-scarlett-speakers";
private const float DefaultAudioSyncUpdateIntervalSeconds = 0.1f;
private const double CalibrationStartSeconds = 0.5;
private const int CalibrationBatchSegments = 120;
private const float DefaultCalibrationGain = 2.0f;
private readonly LocalCastRuntime visualRuntime;
private readonly MimirSynchronizationHub synchronization;
private readonly AquariumUiDocument ui;
private readonly MimirAudioSynchronizationSettings audioSyncSettings;
private readonly float telemetryIntervalSeconds;
private readonly float audioSyncUpdateIntervalSeconds;
private readonly float calibrationGain;
@ -53,9 +52,10 @@ public sealed class MimirRuntime : IAquariumRuntime
Options = options;
visualRuntime = new LocalCastRuntime(options);
synchronization = new MimirSynchronizationHub(settings);
audioSyncSettings = settings.Audio;
telemetryIntervalSeconds = ParseTelemetryIntervalSeconds();
audioSyncUpdateIntervalSeconds = ParseAudioSyncIntervalSeconds();
calibrationGain = ParseCalibrationGain();
calibrationGain = settings.Audio.CalibrationGain;
nextAudioSyncSeconds = audioSyncUpdateIntervalSeconds;
nextTelemetrySeconds = telemetryIntervalSeconds;
foreach (var source in streamSources)
@ -91,7 +91,7 @@ public sealed class MimirRuntime : IAquariumRuntime
public void Start()
{
Console.WriteLine($"Mimir runtime sync buffers: {synchronization.Summary()} @ {synchronization.Settings.BufferDuration.TotalSeconds:0.###}s");
Console.WriteLine($"Mimir runtime sync buffers: {synchronization.Summary()} @ {synchronization.Settings.BufferDuration.TotalSeconds:0.###}s audioSync={audioSyncSettings.Mode} reference={audioSyncSettings.ReferenceSourceId}");
visualRuntime.Start();
}
@ -142,6 +142,11 @@ public sealed class MimirRuntime : IAquariumRuntime
private void QueueCalibrationTimeline()
{
if (!ShouldEmitCalibrationTimeline())
{
return;
}
var nextSegmentSeconds = CalibrationStartSeconds + calibrationSegmentIndex * MimirChirpletTimeline.SegmentSeconds;
if (runtimeSeconds < nextSegmentSeconds)
{
@ -161,8 +166,13 @@ public sealed class MimirRuntime : IAquariumRuntime
private string DescribeChirpletReference()
{
if (!ShouldEmitCalibrationTimeline())
{
return $"{audioSyncSettings.ReferenceSourceId} no-chirp mode; calibration emission disabled";
}
var emittedUntilSeconds = calibrationSegmentIndex * MimirChirpletTimeline.SegmentSeconds;
return $"{DefaultAudioSyncReference} continuous timeline {MimirChirpletTimeline.SegmentSeconds:0.00}s segments emitted to {emittedUntilSeconds:0.00}s";
return $"{audioSyncSettings.ReferenceSourceId} {DescribeAudioSyncMode()} timeline {MimirChirpletTimeline.SegmentSeconds:0.00}s segments emitted to {emittedUntilSeconds:0.00}s";
}
private static string RenderCalibrationBatchPcm16Base64(ulong firstSegment, int segmentCount, out float peak)
@ -198,8 +208,16 @@ public sealed class MimirRuntime : IAquariumRuntime
return;
}
if (audioSyncSettings.Mode == MimirAudioSyncMode.NoChirp)
{
lastAudioSyncAnalysisMilliseconds = 0.0;
lastAudioSynchronizationReports = [];
nextAudioSyncSeconds = runtimeSeconds + audioSyncUpdateIntervalSeconds;
return;
}
var stopwatch = Stopwatch.StartNew();
synchronization.AnalyzeAudioSynchronizationStep(DefaultAudioSyncReference);
synchronization.AnalyzeAudioSynchronizationStep(audioSyncSettings.ReferenceSourceId);
stopwatch.Stop();
lastAudioSyncAnalysisMilliseconds = stopwatch.Elapsed.TotalMilliseconds;
lastAudioSynchronizationReports = synchronization.AudioSynchronizationReports;
@ -214,10 +232,10 @@ public sealed class MimirRuntime : IAquariumRuntime
}
var loopback = synchronization.Buffers.Buffers.FirstOrDefault(buffer =>
string.Equals(buffer.Descriptor.SourceId, DefaultAudioSyncReference, StringComparison.Ordinal));
string.Equals(buffer.Descriptor.SourceId, audioSyncSettings.ReferenceSourceId, StringComparison.Ordinal));
var states = synchronization.AudioSynchronizationStates;
Console.WriteLine(
$"mimir-sync-telemetry t={runtimeSeconds:0.00}s loopbackCount={loopback?.Count ?? 0} loopbackEdgeNs={loopback?.EdgeNs ?? 0} reports={lastAudioSynchronizationReports.Count} states={states.Count} analyzeMs={lastAudioSyncAnalysisMilliseconds:0.0} aligned={DescribeAlignedAudio()}");
$"mimir-sync-telemetry t={runtimeSeconds:0.00}s audioSync={audioSyncSettings.Mode} loopbackCount={loopback?.Count ?? 0} loopbackEdgeNs={loopback?.EdgeNs ?? 0} reports={lastAudioSynchronizationReports.Count} states={states.Count} analyzeMs={lastAudioSyncAnalysisMilliseconds:0.0} aligned={DescribeAlignedAudio()}");
Console.WriteLine($"mimir-sync-buffers {DescribeAudioBuffers()}");
foreach (var report in lastAudioSynchronizationReports.OrderBy(report => report.SourceId, StringComparer.Ordinal))
{
@ -254,11 +272,20 @@ public sealed class MimirRuntime : IAquariumRuntime
: DefaultAudioSyncUpdateIntervalSeconds;
}
private static float ParseCalibrationGain()
private bool ShouldEmitCalibrationTimeline()
{
return float.TryParse(Environment.GetEnvironmentVariable("MIMIR_CHIRPLET_GAIN"), out var gain)
? Math.Clamp(gain, 0.05f, 4.0f)
: DefaultCalibrationGain;
return audioSyncSettings.Mode != MimirAudioSyncMode.NoChirp;
}
private string DescribeAudioSyncMode()
{
return audioSyncSettings.Mode switch
{
MimirAudioSyncMode.ChirpOnly => "chirp-only",
MimirAudioSyncMode.NoChirp => "no-chirp",
MimirAudioSyncMode.Hybrid => "hybrid active-pilot",
_ => audioSyncSettings.Mode.ToString(),
};
}
private string DescribeAudioBuffers()
@ -287,6 +314,11 @@ public sealed class MimirRuntime : IAquariumRuntime
private string DescribeAudioSync()
{
if (audioSyncSettings.Mode == MimirAudioSyncMode.NoChirp)
{
return "no-chirp mode; passive estimator not wired yet";
}
var reports = synchronization.AudioSynchronizationReports;
return reports.Count == 0
? "no payload windows"

View file

@ -29,6 +29,8 @@ public sealed class MimirRuntimeConfiguration
var bufferDuration = model.BufferSeconds > 0.0
? TimeSpan.FromSeconds(Math.Clamp(model.BufferSeconds, 0.25, 60.0))
: MimirSynchronizationSettings.FromEnvironment().BufferDuration;
var audio = (model.AudioSync?.ToSettings(new MimirAudioSynchronizationSettings()) ?? new MimirAudioSynchronizationSettings())
.WithEnvironmentOverrides();
var sourceModels = model.Streams
.Where(stream => stream.Enabled)
@ -47,6 +49,7 @@ public sealed class MimirRuntimeConfiguration
Settings = new MimirSynchronizationSettings
{
BufferDuration = bufferDuration,
Audio = audio,
Streams = streams,
},
Sources = sources,
@ -183,9 +186,34 @@ public sealed class MimirRuntimeConfigFile
{
public double BufferSeconds { get; set; } = 5.0;
public MimirAudioSyncConfig? AudioSync { get; set; }
public List<MimirStreamConfig> Streams { get; set; } = [];
}
public sealed class MimirAudioSyncConfig
{
public string Mode { get; set; } = "";
public string ReferenceSourceId { get; set; } = "";
public float CalibrationGain { get; set; } = float.NaN;
public MimirAudioSynchronizationSettings ToSettings(MimirAudioSynchronizationSettings fallback)
{
return new MimirAudioSynchronizationSettings
{
Mode = MimirAudioSynchronizationSettings.ParseMode(Mode, fallback.Mode),
ReferenceSourceId = string.IsNullOrWhiteSpace(ReferenceSourceId)
? fallback.ReferenceSourceId
: ReferenceSourceId.Trim(),
CalibrationGain = float.IsFinite(CalibrationGain)
? Math.Clamp(CalibrationGain, 0.0f, 4.0f)
: fallback.CalibrationGain,
};
}
}
public sealed class MimirStreamConfig
{
public string SourceId { get; set; } = "";

View file

@ -4,6 +4,8 @@ public sealed class MimirSynchronizationSettings
{
public TimeSpan BufferDuration { get; init; } = TimeSpan.FromSeconds(5);
public MimirAudioSynchronizationSettings Audio { get; init; } = new();
public IReadOnlyList<MimirStreamDescriptor> Streams { get; init; } = [];
public static MimirSynchronizationSettings FromEnvironment()
@ -18,6 +20,7 @@ public sealed class MimirSynchronizationSettings
return new MimirSynchronizationSettings
{
BufferDuration = duration,
Audio = MimirAudioSynchronizationSettings.FromEnvironment(),
Streams = streams,
};
}
@ -48,3 +51,68 @@ public sealed class MimirSynchronizationSettings
}
}
}
public sealed class MimirAudioSynchronizationSettings
{
public const string DefaultReferenceSourceId = "loopback-scarlett-speakers";
public const float DefaultCalibrationGain = 2.0f;
public MimirAudioSyncMode Mode { get; init; } = MimirAudioSyncMode.Hybrid;
public string ReferenceSourceId { get; init; } = DefaultReferenceSourceId;
public float CalibrationGain { get; init; } = DefaultCalibrationGain;
public static MimirAudioSynchronizationSettings FromEnvironment()
{
return new MimirAudioSynchronizationSettings().WithEnvironmentOverrides();
}
public MimirAudioSynchronizationSettings WithEnvironmentOverrides()
{
return new MimirAudioSynchronizationSettings
{
Mode = ParseMode(Environment.GetEnvironmentVariable("MIMIR_AUDIO_SYNC_MODE"), Mode),
ReferenceSourceId = ReadReferenceSourceId(ReferenceSourceId),
CalibrationGain = ReadCalibrationGain(CalibrationGain),
};
}
public static MimirAudioSyncMode ParseMode(string? value, MimirAudioSyncMode fallback)
{
if (string.IsNullOrWhiteSpace(value))
{
return fallback;
}
return value.Trim().ToLowerInvariant() switch
{
"chirp-only" or "chirponly" or "chirp" or "active" => MimirAudioSyncMode.ChirpOnly,
"no-chirp" or "nochirp" or "passive" or "passive-only" or "passiveonly" => MimirAudioSyncMode.NoChirp,
"hybrid" => MimirAudioSyncMode.Hybrid,
_ => fallback,
};
}
private static string ReadReferenceSourceId(string fallback)
{
var sourceId = Environment.GetEnvironmentVariable("MIMIR_AUDIO_SYNC_REFERENCE");
return string.IsNullOrWhiteSpace(sourceId)
? fallback
: sourceId.Trim();
}
private static float ReadCalibrationGain(float fallback)
{
return float.TryParse(Environment.GetEnvironmentVariable("MIMIR_CHIRPLET_GAIN"), out var gain)
? Math.Clamp(gain, 0.0f, 4.0f)
: fallback;
}
}
public enum MimirAudioSyncMode
{
ChirpOnly,
NoChirp,
Hybrid,
}

View file

@ -52,3 +52,4 @@
{"ts": "2026-05-22T17:34:45+01:00", "type": "verification", "status": "ok", "note": "Aquarium WASAPI resampler used int multiplication for source.Length * targetSampleRate; a 60s 48k chirplet batch overflowed and queued one 44.1k output frame. Switching to double-length arithmetic restored 2,646,000 output frames and audible chirplets."}
{"ts": "2026-05-22T18:08:51+01:00", "type": "verification", "status": "partial", "note": "Release live chirplet decoder test: Scarlett loopback->Scarlett mic produced repeated runtime reports in ~230-335 ms per 2s analysis window, typically 3-5 matched timeline events. Pair tests also produced reports for Kiyo mic. Kiyo Pro and both USB Camera mics showed chirplet energy but failed to form valid triplet anchors in this pass. Six-audio-source fan-out exposed process-probe buffer continuity/overlap failures, with many candidates rejected as insufficient-window."}
{"ts": "2026-05-22T18:34:16+01:00", "type": "research", "status": "ok", "note": "Archived chirplet decoder research under research/chirplet-sync-decoder. Distilled conclusion: current BuildChirpletEnergyTrace is dense sliding matched filtering; for Mimir's controlled sync beacon, redesign symbol codebook for dechirp+FFT/Goertzel classification so code constraints shape the receiver before expensive scoring."}
{"ts": "2026-05-22T18:47:44+01:00", "type": "implementation", "status": "ok", "note": "Added runtime-owned audio sync mode selection for chirp-only, no-chirp, and hybrid. Config/env now expose MIMIR_AUDIO_SYNC_MODE plus audioSync.mode/referenceSourceId/calibrationGain; no-chirp disables active calibration and reports passive sync as not wired yet, while hybrid currently uses the active pilot until passive confidence gating and shaped watermark emission are implemented. dotnet build and chirplet self-test passed."}

View file

@ -93,7 +93,11 @@ accepted_design:
ppm. `MimirRuntime` now runs this analysis online as a bounded rotating
service on a fixed cadence, while UI and telemetry read cached reports/states
instead of invoking analysis. `MIMIR_SYNC_TELEMETRY_SECONDS` prints live
buffer/report/state telemetry.
buffer/report/state telemetry. `MimirAudioSynchronizationSettings.Mode`
now owns the active evidence policy: `chirp-only` emits the calibration
witness, `no-chirp` stays silent until passive program-audio sync exists,
and `hybrid` currently uses the active pilot until the passive confidence
gate and shaped watermark are built.
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
texture capture.