GameCult/Mimir
Watch
9
0
Fork
You've already forked Mimir
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add calibrated audio composite diagnostic

Browse source
This commit is contained in:
Meta Vens 2026-06-01 02:56:03 +01:00
commit 8df851850c
6 changed files with 441 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/implementation-plan.md
View file

@ -783,6 +783,11 @@ a named invariant that the native runtime cannot protect yet.
aggregation, calibrated projection constants, and then a global
residual/calibration owner.
9. Move mic alignment, room suppression, voice separation, spatialization, and
stem generation into Faust/native DSP.
stem generation into Faust/native DSP. The C# diagnostic bridge now exists
as `MimirCalibratedAudioCompositeBuilder`: it consumes synchronization
states and measured band responses, emits a corrected composite plus scores,
and is proved by `--calibrated-audio-composite-smoke`. Treat that as the
contract and offline scorer for the native/Faust implementation, not as a
parallel live mixer authority.
10. Keep the OBS bridge witness ledger as evidence before expanding receiver
machinery.

12
notes/current-system-map.md
View file

@ -294,6 +294,18 @@ extracts Float32 windows from rolling audio buffers to publish
with about 0.219 us error from rolling buffers. The next real-world proof is to
run that lane through live Scarlett loopback and mics, not only stored ASIO
artifacts.
`MimirCalibratedAudioCompositeBuilder` is the first C# diagnostic owner for
turning that acoustic evidence into a corrected composite stream. It consumes
sample windows plus `MimirAudioSynchronizationState` delay/confidence and
per-band response evidence, applies fractional alignment, sparse measured-band
equalization, and confidence/noise-weighted summing, then emits composite
samples plus source flatness/weight reports. This is not the hot room-DSP
authority and it does not make room nulls disappear; Faust/native DSP still
owns the production alignment, suppression, separation, and stem lane.
`Mimir.BufferSmoke --calibrated-audio-composite-smoke` proves the contract on
synthetic delayed, colored, noisy mic paths: the uncalibrated mix measures
-3.171 dB SNR, the calibrated composite measures 15.135 dB SNR, and response
flatness improves from 0.548 to 1.000.
That real-world proof now has a first receipt:
`calibration/bioacoustic/complex-contour-live-20260525-063229.md`. A freshly
rendered canary-packet witness was played through Focusrite ASIO and captured

136
src/Mimir.BufferSmoke/Program.cs
View file

@ -75,6 +75,11 @@ if (args.Any(arg => string.Equals(arg, "--audio-actuator-bank-smoke", StringComp
return RunAudioActuatorBankSmoke();
}
if (args.Any(arg => string.Equals(arg, "--calibrated-audio-composite-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunCalibratedAudioCompositeSmoke();
}
if (args.Any(arg => string.Equals(arg, "--synchronized-buffer-planner-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunSynchronizedBufferPlannerSmoke();
@ -3323,6 +3328,43 @@ static int RunAudioActuatorBankSmoke()
: 1;
}
static int RunCalibratedAudioCompositeSmoke()
{
const int sampleRate = 48_000;
const int frameCount = sampleRate;
var bands = new[] { 220.0, 440.0, 880.0, 1760.0, 3520.0 };
var reference = RenderCalibrationToneStack(frameCount, sampleRate, bands, [1.0, 1.0, 1.0, 1.0, 1.0], noiseScale: 0.0, seed: 7);
var micA = RenderCalibrationToneStack(frameCount, sampleRate, bands, [0.45, 0.70, 1.25, 0.95, 0.35], noiseScale: 0.035, seed: 11);
var micB = RenderCalibrationToneStack(frameCount, sampleRate, bands, [1.15, 0.55, 0.40, 1.35, 0.80], noiseScale: 0.030, seed: 17);
var micC = RenderCalibrationToneStack(frameCount, sampleRate, bands, [0.75, 1.30, 0.85, 0.50, 1.10], noiseScale: 0.040, seed: 23);
micA = ApplyFractionalDelay(micA, 21.375);
micB = ApplyFractionalDelay(micB, -13.625);
micC = ApplyFractionalDelay(micC, 38.25);
var sources = new[]
{
NewCompositeSource("shotgun", micA, 21.375, 0.92, bands, [0.45, 0.70, 1.25, 0.95, 0.35], 0.035),
NewCompositeSource("cardioid", micB, -13.625, 0.88, bands, [1.15, 0.55, 0.40, 1.35, 0.80], 0.030),
NewCompositeSource("kiyo-pro-mic", micC, 38.25, 0.74, bands, [0.75, 1.30, 0.85, 0.50, 1.10], 0.040),
};
var uncalibrated = AverageSignals([micA, micB, micC]);
var result = new MimirCalibratedAudioCompositeBuilder().Build(sources);
var uncalibratedSnr = EstimateSnrDb(reference, uncalibrated);
var calibratedSnr = EstimateSnrDb(reference, result.Samples);
var flatnessGain = result.ResponseFlatnessAfter - result.ResponseFlatnessBefore;
Console.WriteLine(
$"calibrated-audio-composite-smoke sources={result.SourceReports.Count} compositeRms={result.CompositeRms:0.000000} " +
$"uncalibratedSnrDb={uncalibratedSnr:0.000} calibratedSnrDb={calibratedSnr:0.000} " +
$"flatnessBefore={result.ResponseFlatnessBefore:0.000} flatnessAfter={result.ResponseFlatnessAfter:0.000} flatnessGain={flatnessGain:0.000} " +
$"bands={string.Join(",", result.SourceReports.Select(report => $"{report.SourceId}:{report.CorrectedBandCount}"))}");
return result.SourceReports.Count == 3 &&
calibratedSnr > uncalibratedSnr + 2.0 &&
result.ResponseFlatnessAfter > result.ResponseFlatnessBefore + 0.15
? 0
: 1;
}
static int RunSynchronizedBufferPlannerSmoke()
{
const long t0 = 1_000_000_000;
@ -3572,6 +3614,100 @@ static MimirAudioSynchronizationState NewSyncState(string sourceId, double delay
10,
10);
static MimirCalibratedAudioCompositeSource NewCompositeSource(
string sourceId,
float[] samples,
double delaySamples,
double confidence,
IReadOnlyList<double> bandCenters,
IReadOnlyList<double> gains,
double noiseRms)
{
var responses = bandCenters
.Select((center, index) => new MimirChirpletBandResponse(center, gains[index] * gains[index]))
.ToArray();
var state = new MimirAudioSynchronizationState(
"loopback-scarlett-speakers",
sourceId,
48_000,
delaySamples,
delaySamples,
delaySamples * 1000.0 / 48_000,
0.0,
confidence,
responses,
1_000_000_000L,
10,
10);
return new MimirCalibratedAudioCompositeSource(sourceId, 48_000, samples, state, responses, noiseRms);
}
static float[] RenderCalibrationToneStack(
int frameCount,
int sampleRate,
IReadOnlyList<double> bands,
IReadOnlyList<double> gains,
double noiseScale,
int seed)
{
var random = new Random(seed);
var samples = new float[frameCount];
var normalization = 0.16 / Math.Max(1, bands.Count);
for (var band = 0; band < bands.Count; band++)
{
var omega = 2.0 * Math.PI * bands[band] / sampleRate;
for (var index = 0; index < samples.Length; index++)
{
samples[index] += (float)(Math.Sin(omega * index) * gains[band] * normalization);
}
}
if (noiseScale > 0.0)
{
for (var index = 0; index < samples.Length; index++)
{
samples[index] += (float)((random.NextDouble() * 2.0 - 1.0) * noiseScale);
}
}
return samples;
}
static float[] AverageSignals(IReadOnlyList<float[]> signals)
{
var frameCount = signals.Min(signal => signal.Length);
var output = new float[frameCount];
foreach (var signal in signals)
{
for (var index = 0; index < frameCount; index++)
{
output[index] += signal[index] / signals.Count;
}
}
return output;
}
static double EstimateSnrDb(IReadOnlyList<float> reference, IReadOnlyList<float> candidate)
{
var frameCount = Math.Min(reference.Count, candidate.Count);
var signal = 0.0;
var error = 0.0;
var referenceRms = RootMeanSquare(reference);
var candidateRms = RootMeanSquare(candidate);
var gain = candidateRms <= 1.0e-12 ? 1.0 : referenceRms / candidateRms;
for (var index = 0; index < frameCount; index++)
{
var expected = reference[index];
var actual = candidate[index] * gain;
signal += expected * expected;
var delta = expected - actual;
error += delta * delta;
}
return 10.0 * Math.Log10(Math.Max(1.0e-18, signal) / Math.Max(1.0e-18, error));
}
static MimirRollingStreamBuffer NewVideoBuffer(
string sourceId,
MimirStreamOrigin origin,

275
src/Mimir.Runtime/Synchronization/MimirCalibratedAudioComposite.cs Normal file
View file

@ -0,0 +1,275 @@
namespace Mimir.Runtime.Synchronization;
public sealed record MimirCalibratedAudioCompositeOptions(
double MinimumConfidence = 0.05,
double TargetBandEnergy = 0.70,
double MinimumBandEnergy = 0.01,
double MinimumBandGain = 0.20,
double MaximumBandGain = 4.00,
double NoiseFloorRms = 1.0e-5);
public sealed record MimirCalibratedAudioCompositeSource(
string SourceId,
int SampleRate,
float[] Samples,
MimirAudioSynchronizationState? SynchronizationState,
IReadOnlyList<MimirChirpletBandResponse> BandResponses,
double NoiseRms = 0.0);
public sealed record MimirCalibratedAudioCompositeSourceReport(
string SourceId,
double AppliedDelaySamples,
double AppliedGain,
double Weight,
int CorrectedBandCount,
double ResponseFlatnessBefore,
double ResponseFlatnessAfter,
double Confidence,
double NoiseRms);
public sealed record MimirCalibratedAudioCompositeResult(
int SampleRate,
float[] Samples,
IReadOnlyList<MimirCalibratedAudioCompositeSourceReport> SourceReports,
double CompositeRms,
double SourceMeanRms,
double ResponseFlatnessBefore,
double ResponseFlatnessAfter);
public sealed class MimirCalibratedAudioCompositeBuilder(MimirCalibratedAudioCompositeOptions? options = null)
{
private readonly MimirCalibratedAudioCompositeOptions options = options ?? new();
public MimirCalibratedAudioCompositeResult Build(IReadOnlyList<MimirCalibratedAudioCompositeSource> sources)
{
var accepted = sources
.Where(source => source.Samples.Length > 0)
.ToArray();
if (accepted.Length == 0)
{
return new MimirCalibratedAudioCompositeResult(0, [], [], 0.0, 0.0, 0.0, 0.0);
}
var sampleRate = accepted[0].SampleRate;
if (accepted.Any(source => source.SampleRate != sampleRate))
{
throw new ArgumentException("Calibrated audio composite requires one sample rate. Resampling belongs to the native DSP actuator.");
}
var frameCount = accepted.Min(source => source.Samples.Length);
var output = new float[frameCount];
var reports = new List<MimirCalibratedAudioCompositeSourceReport>(accepted.Length);
var totalWeight = 0.0;
var sourceRmsTotal = 0.0;
var beforeFlatnessTotal = 0.0;
var afterFlatnessTotal = 0.0;
foreach (var source in accepted)
{
var confidence = source.SynchronizationState?.Confidence ?? 1.0;
if (confidence < options.MinimumConfidence)
{
continue;
}
var delaySamples = source.SynchronizationState?.SmoothedDelaySamples ?? 0.0;
var aligned = ApplyFractionalDelay(source.Samples, -delaySamples, frameCount);
var beforeFlatness = ResponseFlatness(source.BandResponses);
var corrected = ApplySparseBandEqualization(aligned, sampleRate, source.BandResponses, out var correctedBands);
var afterResponses = EstimateCorrectedResponses(source.BandResponses);
var afterFlatness = ResponseFlatness(afterResponses);
var appliedGain = OverallGain(source.BandResponses);
if (Math.Abs(appliedGain - 1.0) > 1.0e-6)
{
for (var index = 0; index < corrected.Length; index++)
{
corrected[index] *= (float)appliedGain;
}
}
var noise = Math.Max(options.NoiseFloorRms, source.NoiseRms);
var responseWeight = Math.Clamp(afterFlatness <= 0.0 ? 1.0 : afterFlatness, 0.10, 1.0);
var weight = confidence * responseWeight / (noise * noise);
totalWeight += weight;
sourceRmsTotal += RootMeanSquare(source.Samples.AsSpan(0, frameCount));
beforeFlatnessTotal += beforeFlatness;
afterFlatnessTotal += afterFlatness;
for (var index = 0; index < frameCount; index++)
{
output[index] += (float)(corrected[index] * weight);
}
reports.Add(new MimirCalibratedAudioCompositeSourceReport(
source.SourceId,
delaySamples,
appliedGain,
weight,
correctedBands,
beforeFlatness,
afterFlatness,
confidence,
source.NoiseRms));
}
if (totalWeight > 0.0)
{
for (var index = 0; index < output.Length; index++)
{
output[index] = (float)Math.Clamp(output[index] / totalWeight, -1.0, 1.0);
}
}
var reportCount = Math.Max(1, reports.Count);
return new MimirCalibratedAudioCompositeResult(
sampleRate,
output,
reports,
RootMeanSquare(output),
sourceRmsTotal / reportCount,
beforeFlatnessTotal / reportCount,
afterFlatnessTotal / reportCount);
}
private float[] ApplySparseBandEqualization(
ReadOnlySpan<float> samples,
int sampleRate,
IReadOnlyList<MimirChirpletBandResponse> responses,
out int correctedBands)
{
var output = samples.ToArray();
correctedBands = 0;
foreach (var response in responses)
{
if (response.CenterHz <= 0.0 ||
response.CenterHz >= sampleRate * 0.48 ||
response.Energy < options.MinimumBandEnergy)
{
continue;
}
var gain = Math.Clamp(
Math.Sqrt(options.TargetBandEnergy / Math.Max(options.MinimumBandEnergy, response.Energy)),
options.MinimumBandGain,
options.MaximumBandGain);
if (Math.Abs(gain - 1.0) < 0.02)
{
continue;
}
AddBandGainDelta(output, samples, sampleRate, response.CenterHz, gain - 1.0);
correctedBands++;
}
return output;
}
private static void AddBandGainDelta(float[] output, ReadOnlySpan<float> samples, int sampleRate, double centerHz, double gainDelta)
{
var sine = 0.0;
var cosine = 0.0;
var omega = 2.0 * Math.PI * centerHz / sampleRate;
for (var index = 0; index < samples.Length; index++)
{
var phase = omega * index;
sine += samples[index] * Math.Sin(phase);
cosine += samples[index] * Math.Cos(phase);
}
var scale = 2.0 / Math.Max(1, samples.Length);
sine *= scale;
cosine *= scale;
for (var index = 0; index < output.Length; index++)
{
var phase = omega * index;
var component = sine * Math.Sin(phase) + cosine * Math.Cos(phase);
output[index] = (float)Math.Clamp(output[index] + component * gainDelta, -4.0, 4.0);
}
}
private double OverallGain(IReadOnlyList<MimirChirpletBandResponse> responses)
{
var usable = responses
.Where(response => response.Energy >= options.MinimumBandEnergy)
.Select(response => response.Energy)
.Order()
.ToArray();
if (usable.Length == 0)
{
return 1.0;
}
var median = usable[usable.Length / 2];
return Math.Clamp(
Math.Sqrt(options.TargetBandEnergy / Math.Max(options.MinimumBandEnergy, median)),
options.MinimumBandGain,
options.MaximumBandGain);
}
private IReadOnlyList<MimirChirpletBandResponse> EstimateCorrectedResponses(IReadOnlyList<MimirChirpletBandResponse> responses) =>
responses
.Select(response =>
{
if (response.Energy < options.MinimumBandEnergy)
{
return response;
}
var gain = Math.Clamp(
Math.Sqrt(options.TargetBandEnergy / Math.Max(options.MinimumBandEnergy, response.Energy)),
options.MinimumBandGain,
options.MaximumBandGain);
return response with { Energy = response.Energy * gain * gain };
})
.ToArray();
private static float[] ApplyFractionalDelay(ReadOnlySpan<float> source, double delaySamples, int frameCount)
{
var delayed = new float[frameCount];
for (var index = 0; index < delayed.Length; index++)
{
var sourcePosition = index - delaySamples;
if (sourcePosition < 0.0 || sourcePosition >= source.Length - 1)
{
continue;
}
var left = (int)Math.Floor(sourcePosition);
var fraction = sourcePosition - left;
delayed[index] = (float)(source[left] + (source[left + 1] - source[left]) * fraction);
}
return delayed;
}
private static double RootMeanSquare(ReadOnlySpan<float> samples)
{
if (samples.Length == 0)
{
return 0.0;
}
var sum = 0.0;
for (var index = 0; index < samples.Length; index++)
{
sum += samples[index] * samples[index];
}
return Math.Sqrt(sum / samples.Length);
}
private static double ResponseFlatness(IReadOnlyList<MimirChirpletBandResponse> responses)
{
var energies = responses
.Where(response => response.Energy > 1.0e-9)
.Select(response => Math.Log(response.Energy))
.ToArray();
if (energies.Length <= 1)
{
return energies.Length;
}
var mean = energies.Average();
var variance = energies.Sum(value => (value - mean) * (value - mean)) / energies.Length;
return 1.0 / (1.0 + Math.Sqrt(variance));
}
}

1
state/evidence.jsonl
View file

@ -307,3 +307,4 @@
{"ts": "2026-06-01T02:30:26+01:00", "type": "decision", "status": "ok", "note": "Raven stream proof loopback capture now prefers compiled Mimir.WasapiLoopback over the PowerShell WASAPI script. Current winget and BtbN FFmpeg builds on Raven expose no native wasapi input, and DirectShow has no Realtek Stereo Mix. Live helper proof parsed Raven Realtek as WAVE_FORMAT_EXTENSIBLE IEEE float 48 kHz stereo and produced a 10 s direct NVENC/AAC sample with no loopback clip candidates at artifacts/runtime/stream-proof/raven-nvenc-direct-wasapi-helper-20260601.mp4."}
{"ts": "2026-06-01T02:42:04+01:00", "type": "decision", "status": "ok", "note": "Raven video bottleneck was not NVENC. BtbN FFmpeg ddagrab through Raven's interactive desktop session records Desktop Duplication API frames into h264_nvenc at ~56.8 fps for 1080p60 with Mimir.WasapiLoopback AAC audio in the same local MP4. The remaining sparse-frame direct proof is the TCP/SSH proof-capture path, not Raven capture/encode. start-raven-av-sender now defaults to ddagrab and keeps gdigrab only as an explicit fallback."}
{"ts": "2026-06-01T02:45:51+01:00", "type": "verification", "status": "ok", "note": "Direct LAN SRT Raven proof is healthy after removing the SSH tunnel. Starfire listened on srt://0.0.0.0:5204 while Raven called 192.168.1.66:5204 with ddagrab + h264_nvenc + Mimir.WasapiLoopback AAC; artifact artifacts/runtime/stream-proof/raven-nvenc-ddagrab-lan-srt-20260601-024525.mp4 contains 564 video frames over 10.005 s (~56.5 fps) plus 48 kHz stereo AAC. The earlier sparse-frame recordings were the TCP/SSH proof path, not Raven GPU/capture/encode or LAN SRT."}
{"ts": "2026-06-01T02:54:52+01:00", "type": "verification", "status": "ok", "note": "Calibrated audio composite contract now exists as MimirCalibratedAudioCompositeBuilder. The smoke consumes synthetic delayed/noisy/colored mic windows plus sync states and band responses, applies fractional alignment plus sparse measured-band equalization and confidence/noise weighting, and improves SNR from -3.171 dB uncalibrated to 15.135 dB calibrated while response flatness rises from 0.548 to 1.000. This is an offline/diagnostic contract for the future Faust/native hot lane, not a claim that live room nulls are perfectly flattened."}

11
state/map.yaml
View file

@ -633,6 +633,17 @@ accepted_design:
rolling buffers, and publishes `evidence=complex-contour` reports. The
synthetic runtime-shaped proof `--complex-contour-runtime-self-test` at
192 kHz recovers a 693.5-sample delay with about 0.219 us error.
`MimirCalibratedAudioCompositeBuilder` is the first diagnostic composite
owner for corrected mic streams: it consumes audio samples plus
`MimirAudioSynchronizationState` delay/confidence and measured band
responses, applies fractional alignment, sparse measured-band
equalization, and confidence/noise-weighted summing, then emits composite
samples and flatness/weight reports. `Mimir.BufferSmoke
--calibrated-audio-composite-smoke` proves the contract on synthetic
delayed/noisy/colored mic paths, improving SNR from -3.171 dB uncalibrated
to 15.135 dB calibrated and response flatness from 0.548 to 1.000.
Faust/native DSP still owns the production hot lane; this C# path is a
contract, scorer, and offline diagnostic, not a competing live mixer.
A fresh live Scarlett proof is preserved at
`calibration/bioacoustic/complex-contour-live-20260525-063229.md`:
Focusrite ASIO playback/capture measured Loopback 1 -> Loopback 2 at