Record Kiyo with dual mic composite

This commit is contained in:
Meta Vens 2026-06-01 04:25:18 +01:00
commit 10090ee44c
7 changed files with 423 additions and 1 deletions

View file

@ -148,6 +148,15 @@ contract on synthetic shotgun/cardioid witnesses: coherent shared tones survive,
a mic-local `3.52 kHz` interferer is suppressed, and SNR rises from `1.366 dB`
to `16.304 dB`. This is still the C# diagnostic/offline contract; Faust/native
DSP owns the production hot lane.
The first listenable bridge receipt is
`scripts/record-kiyo-dual-mic-composite.ps1`. It records Kiyo video and
Scarlett ASIO raw channels together, converts Scarlett channels 0/1 into a
computed dual-mic composite WAV with `--asio-dual-mic-composite-wav`, then muxes
that WAV with the Kiyo video. The 2026-06-01 receipt at
`artifacts/runtime/kiyo-dual-mic-composite/kiyo-dual-mic-composite-latest.mp4`
contains 10.000 s of 640x480 30 fps Kiyo video and 9.994 s of mono AAC from the
computed composite. Treat it as a bridge proof of audibility/sync, not the final
native live DSP path.
The synthetic invariant is
`dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`;
it renders the bioacoustic timeline, decodes direct word anchors, and requires a

View file

@ -814,7 +814,11 @@ a named invariant that the native runtime cannot protect yet.
calibrated-band coherence and suppresses incoherent measured-band components
before summing; `--dual-mic-coherence-cancellation-smoke` proves the two-mic
case by cutting a mic-local 3.52 kHz interferer while preserving coherent
shared tones. Treat that as the contract and offline scorer for the
shared tones. `scripts/record-kiyo-dual-mic-composite.ps1` is the first
listenable bridge: it records Kiyo video plus Scarlett ASIO raw channels,
renders the channels 0/1 composite WAV with `--asio-dual-mic-composite-wav`,
and muxes the computed mono composite into the Kiyo MP4. Treat that as the
contract and offline scorer for the
native/Faust implementation, not as a parallel live mixer authority. The
next matching probe loop also exists as
`MimirTargetedBioacousticProbePlanner`: residuals and mic pathology notes

View file

@ -317,6 +317,15 @@ two-witness cancellation cut: shared tones keep about 0.999 coherence, a
mic-local 3.52 kHz interferer scores about 0.001 coherence and 0.948
suppression, its measured energy falls from 0.039195 to 0.001306, and SNR rises
from 1.366 dB to 16.304 dB.
`scripts/record-kiyo-dual-mic-composite.ps1` is the first listenable bridge
receipt for that path. It starts Kiyo FFmpeg capture and Scarlett ASIO raw
capture together, converts ASIO channels 0/1 to a computed composite WAV through
`--asio-dual-mic-composite-wav`, then muxes the composite WAV with Kiyo video.
The 2026-06-01 receipt
`artifacts/runtime/kiyo-dual-mic-composite/kiyo-dual-mic-composite-latest.mp4`
has 10.000 s of 640x480 30 fps Kiyo video and 9.994 s of mono AAC composite
audio; the composite report estimated channel-1 delay at about -141.019 samples
(-2937.889 us).
`MimirTargetedBioacousticProbePlanner` is the matching "ask the room a better
question" owner. It consumes per-source spectral residuals and optional source
pathology notes, then emits gain-bounded shaped probe bands for the

View file

@ -0,0 +1,158 @@
param(
[int]$DurationSeconds = 15,
[string]$OutputPath = "artifacts\runtime\kiyo-dual-mic-composite\kiyo-dual-mic-composite.mp4",
[string]$LogRoot = "artifacts\runtime\kiyo-dual-mic-composite",
[string]$FfmpegPath = "ffmpeg",
[string]$AsioProbePath = "native\probes\asio_audio_cadence\build\Release\asio_audio_cadence.exe",
[string]$KiyoVideoDevice = "USB Video Device",
[int]$KiyoWidth = 640,
[int]$KiyoHeight = 480,
[int]$KiyoFrameRate = 30,
[int]$SampleRate = 48000,
[int]$AsioChannels = 4,
[int]$ShotgunChannel = 0,
[int]$CardioidChannel = 1,
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
function Quote-ProcessArgument([string]$Value) {
if ($Value.Length -eq 0) {
return '""'
}
if ($Value -notmatch '[\s"]') {
return $Value
}
return '"' + $Value.Replace('"', '\"') + '"'
}
function Start-LoggedProcess([string]$FileName, [string[]]$Arguments, [string]$StdoutPath, [string]$StderrPath) {
$startInfo = [System.Diagnostics.ProcessStartInfo]::new()
$startInfo.FileName = $FileName
$startInfo.UseShellExecute = $false
$startInfo.RedirectStandardOutput = $true
$startInfo.RedirectStandardError = $true
$startInfo.Arguments = ($Arguments | ForEach-Object { Quote-ProcessArgument $_ }) -join " "
$process = [System.Diagnostics.Process]::Start($startInfo)
if (-not $process) {
throw "Could not start $FileName"
}
$stdoutStream = [System.IO.File]::Open($StdoutPath, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write, [System.IO.FileShare]::Read)
$stderrStream = [System.IO.File]::Open($StderrPath, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write, [System.IO.FileShare]::Read)
return [pscustomobject]@{
Process = $process
StdoutTask = $process.StandardOutput.BaseStream.CopyToAsync($stdoutStream)
StderrTask = $process.StandardError.BaseStream.CopyToAsync($stderrStream)
StdoutStream = $stdoutStream
StderrStream = $stderrStream
}
}
function Wait-LoggedProcess($Handle, [int]$TimeoutSeconds, [string]$Name) {
if (-not $Handle.Process.WaitForExit($TimeoutSeconds * 1000)) {
try { $Handle.Process.Kill($true) } catch {}
throw "$Name timed out after ${TimeoutSeconds}s."
}
[void]$Handle.StdoutTask.GetAwaiter().GetResult()
[void]$Handle.StderrTask.GetAwaiter().GetResult()
$Handle.StdoutStream.Dispose()
$Handle.StderrStream.Dispose()
if ($Handle.Process.ExitCode -ne 0) {
throw "$Name failed with exit code $($Handle.Process.ExitCode)."
}
}
$repo = Split-Path -Parent $PSScriptRoot
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$absoluteLogRoot = if ([System.IO.Path]::IsPathRooted($LogRoot)) { $LogRoot } else { Join-Path $repo $LogRoot }
$runDirectory = Join-Path $absoluteLogRoot $timestamp
New-Item -ItemType Directory -Force -Path $runDirectory | Out-Null
$absoluteOutputPath = if ([System.IO.Path]::IsPathRooted($OutputPath)) { $OutputPath } else { Join-Path $repo $OutputPath }
$outputDirectory = Split-Path -Parent $absoluteOutputPath
New-Item -ItemType Directory -Force -Path $outputDirectory | Out-Null
$asioRaw = Join-Path $runDirectory "scarlett-f32-interleaved.raw"
$kiyoMkv = Join-Path $runDirectory "kiyo-video.mkv"
$compositeWav = Join-Path $runDirectory "dual-mic-composite.wav"
$absoluteAsioProbe = if ([System.IO.Path]::IsPathRooted($AsioProbePath)) { $AsioProbePath } else { Join-Path $repo $AsioProbePath }
$asioArgs = @(
"--set-sample-rate", $SampleRate.ToString(),
"--record-f32-interleaved", $asioRaw,
"--capture-seconds", $DurationSeconds.ToString()
)
$kiyoArgs = @(
"-hide_banner",
"-y",
"-thread_queue_size", "1024",
"-rtbufsize", "256M",
"-f", "dshow",
"-video_size", "${KiyoWidth}x${KiyoHeight}",
"-framerate", $KiyoFrameRate.ToString(),
"-i", "video=$KiyoVideoDevice",
"-t", $DurationSeconds.ToString(),
"-r", $KiyoFrameRate.ToString(),
"-c:v", "h264_nvenc",
"-preset", "p4",
"-tune", "ll",
"-pix_fmt", "yuv420p",
$kiyoMkv
)
Write-Host "Kiyo + computed dual-mic composite recording:"
Write-Host " Kiyo: $KiyoVideoDevice ${KiyoWidth}x${KiyoHeight}@$KiyoFrameRate"
Write-Host " ASIO: $absoluteAsioProbe sampleRate=$SampleRate channels=$AsioChannels shotgun=$ShotgunChannel cardioid=$CardioidChannel"
Write-Host " Run dir: $runDirectory"
Write-Host " Output: $absoluteOutputPath"
if ($DryRun) {
Write-Host ($absoluteAsioProbe + " " + ($asioArgs -join " "))
Write-Host ($FfmpegPath + " " + ($kiyoArgs -join " "))
exit 0
}
$asio = Start-LoggedProcess $absoluteAsioProbe $asioArgs (Join-Path $runDirectory "asio.out.log") (Join-Path $runDirectory "asio.err.log")
$kiyo = Start-LoggedProcess $FfmpegPath $kiyoArgs (Join-Path $runDirectory "kiyo.out.log") (Join-Path $runDirectory "kiyo.err.log")
Wait-LoggedProcess $asio ($DurationSeconds + 15) "ASIO capture"
Wait-LoggedProcess $kiyo ($DurationSeconds + 15) "Kiyo capture"
dotnet run --project (Join-Path $repo "src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj") -- `
--asio-dual-mic-composite-wav `
--input $asioRaw `
--output $compositeWav `
--sample-rate $SampleRate `
--channels $AsioChannels `
--shotgun-channel $ShotgunChannel `
--cardioid-channel $CardioidChannel
if ($LASTEXITCODE -ne 0) {
throw "Dual-mic composite WAV generation failed."
}
$muxArgs = @(
"-hide_banner",
"-y",
"-i", $kiyoMkv,
"-i", $compositeWav,
"-map", "0:v:0",
"-map", "1:a:0",
"-c:v", "copy",
"-c:a", "aac",
"-b:a", "192k",
"-ar", "48000",
"-movflags", "+faststart",
$absoluteOutputPath
)
$mux = Start-LoggedProcess $FfmpegPath $muxArgs (Join-Path $runDirectory "mux.out.log") (Join-Path $runDirectory "mux.err.log")
Wait-LoggedProcess $mux 60 "Mux"
Write-Host "Recording complete:"
Write-Host " $absoluteOutputPath"
Write-Host " composite wav: $compositeWav"
Write-Host " report: $([System.IO.Path]::ChangeExtension($compositeWav, '.json'))"

View file

@ -85,6 +85,17 @@ if (args.Any(arg => string.Equals(arg, "--dual-mic-coherence-cancellation-smoke"
return RunDualMicCoherenceCancellationSmoke();
}
if (args.Any(arg => string.Equals(arg, "--asio-dual-mic-composite-wav", StringComparison.OrdinalIgnoreCase)))
{
return RunAsioDualMicCompositeWave(
ParseStringOption(args, "--input", ""),
ParseStringOption(args, "--output", ""),
ParseIntOption(args, "--sample-rate", 48_000),
ParseIntOption(args, "--channels", 4),
ParseIntOption(args, "--shotgun-channel", 0),
ParseIntOption(args, "--cardioid-channel", 1));
}
if (args.Any(arg => string.Equals(arg, "--targeted-bioacoustic-probe-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunTargetedBioacousticProbeSmoke();
@ -3473,6 +3484,82 @@ static int RunDualMicCoherenceCancellationSmoke()
: 1;
}
static int RunAsioDualMicCompositeWave(
string inputPath,
string outputPath,
int sampleRate,
int channels,
int shotgunChannel,
int cardioidChannel)
{
if (string.IsNullOrWhiteSpace(inputPath) || !File.Exists(inputPath))
{
Console.Error.WriteLine("asio-dual-mic-composite-wav failed: --input must name an existing f32 interleaved ASIO capture.");
return 1;
}
if (string.IsNullOrWhiteSpace(outputPath))
{
Console.Error.WriteLine("asio-dual-mic-composite-wav failed: --output is required.");
return 1;
}
if (shotgunChannel < 0 || cardioidChannel < 0 || shotgunChannel >= channels || cardioidChannel >= channels)
{
Console.Error.WriteLine($"asio-dual-mic-composite-wav failed: channel indices must fit 0..{channels - 1}.");
return 1;
}
var channelSamples = ReadInterleavedFloat32(inputPath, channels, out var frameCount);
if (frameCount == 0)
{
Console.Error.WriteLine("asio-dual-mic-composite-wav failed: capture has no frames.");
return 1;
}
var shotgun = channelSamples[shotgunChannel];
var cardioid = channelSamples[cardioidChannel];
var delay = EstimateRelativeDelaySamples(shotgun, cardioid, maxLagSamples: Math.Min(sampleRate / 100, 960));
var bands = BuildCompositeProbeBands(sampleRate);
var shotgunResponses = EstimateBandResponses(shotgun, sampleRate, bands);
var cardioidResponses = EstimateBandResponses(cardioid, sampleRate, bands);
var result = new MimirCalibratedAudioCompositeBuilder(new MimirCalibratedAudioCompositeOptions(
MinimumConfidence: 0.05,
TargetBandEnergy: MedianPositiveEnergy(shotgunResponses.Concat(cardioidResponses).Select(static response => response.Energy)),
MinimumBandEnergy: 1.0e-8,
MinimumBandGain: 0.65,
MaximumBandGain: 1.60,
NoiseFloorRms: 1.0e-5,
MinimumCoherence: 0.42,
IncoherentBandSuppression: 0.62)).Build(
[
NewCompositeSourceFromResponses("asio-ch0-shotgun", shotgun, sampleRate, delaySamples: 0.0, confidence: 0.90, shotgunResponses, NoiseEstimate(shotgun)),
NewCompositeSourceFromResponses("asio-ch1-cardioid", cardioid, sampleRate, delaySamples: delay, confidence: 0.90, cardioidResponses, NoiseEstimate(cardioid)),
]);
var normalized = NormalizePeak(result.Samples, peak: 0.92f);
WriteWave(outputPath, normalized, sampleRate);
var reportPath = Path.ChangeExtension(outputPath, ".json");
var report = new
{
input = Path.GetFullPath(inputPath),
output = Path.GetFullPath(outputPath),
sampleRate,
frameCount,
shotgunChannel,
cardioidChannel,
estimatedCardioidDelaySamples = delay,
estimatedCardioidDelayMicroseconds = delay * 1_000_000.0 / sampleRate,
compositeRms = result.CompositeRms,
sourceReports = result.SourceReports,
bandReports = result.BandReports,
};
File.WriteAllText(reportPath, JsonSerializer.Serialize(report, new JsonSerializerOptions { WriteIndented = true }));
Console.WriteLine(
$"asio-dual-mic-composite-wav input={inputPath} output={outputPath} report={reportPath} frames={frameCount} " +
$"delaySamples={delay:0.000} delayUs={delay * 1_000_000.0 / sampleRate:0.000} bands={result.BandReports.Count} rms={result.CompositeRms:0.000000}");
return 0;
}
static int RunTargetedBioacousticProbeSmoke()
{
var residuals = new[]
@ -4143,6 +4230,31 @@ static MimirCalibratedAudioCompositeSource NewCompositeSource(
return new MimirCalibratedAudioCompositeSource(sourceId, 48_000, samples, state, responses, noiseRms);
}
static MimirCalibratedAudioCompositeSource NewCompositeSourceFromResponses(
string sourceId,
float[] samples,
int sampleRate,
double delaySamples,
double confidence,
IReadOnlyList<MimirChirpletBandResponse> responses,
double noiseRms)
{
var state = new MimirAudioSynchronizationState(
"scarlett-asio-direct",
sourceId,
sampleRate,
delaySamples,
delaySamples,
delaySamples * 1000.0 / sampleRate,
0.0,
confidence,
responses,
1_000_000_000L,
1,
1);
return new MimirCalibratedAudioCompositeSource(sourceId, sampleRate, samples, state, responses, noiseRms);
}
static MimirAudioCalibrationBandResidual[] ScarlettProfessionalMicProbeResiduals() =>
[
new MimirAudioCalibrationBandResidual("asio-ch0-shotgun", 240.0, 0.91, 0.4, ResponseEnergy: 0.62),
@ -4429,6 +4541,126 @@ static double ToneEnergy(ReadOnlySpan<float> samples, int sampleRate, double cen
return Math.Sqrt(sine * sine + cosine * cosine) * 2.0 / Math.Max(1, count);
}
static double EstimateRelativeDelaySamples(IReadOnlyList<float> reference, IReadOnlyList<float> candidate, int maxLagSamples)
{
var frameCount = Math.Min(reference.Count, candidate.Count);
if (frameCount == 0)
{
return 0.0;
}
var bestLag = 0;
var bestScore = double.NegativeInfinity;
for (var lag = -maxLagSamples; lag <= maxLagSamples; lag++)
{
var score = 0.0;
var refEnergy = 0.0;
var candEnergy = 0.0;
var start = Math.Max(0, -lag);
var end = Math.Min(frameCount, frameCount - lag);
for (var index = start; index < end; index++)
{
var left = reference[index];
var right = candidate[index + lag];
score += left * right;
refEnergy += left * left;
candEnergy += right * right;
}
var normalized = score / Math.Sqrt(Math.Max(1.0e-18, refEnergy * candEnergy));
if (normalized > bestScore)
{
bestScore = normalized;
bestLag = lag;
}
}
if (bestLag <= -maxLagSamples || bestLag >= maxLagSamples)
{
return bestLag;
}
var leftScore = DelayCorrelation(reference, candidate, bestLag - 1);
var centerScore = DelayCorrelation(reference, candidate, bestLag);
var rightScore = DelayCorrelation(reference, candidate, bestLag + 1);
var denominator = leftScore - 2.0 * centerScore + rightScore;
var fractional = Math.Abs(denominator) <= 1.0e-12 ? 0.0 : 0.5 * (leftScore - rightScore) / denominator;
return bestLag + Math.Clamp(fractional, -0.5, 0.5);
}
static double DelayCorrelation(IReadOnlyList<float> reference, IReadOnlyList<float> candidate, int lag)
{
var frameCount = Math.Min(reference.Count, candidate.Count);
var score = 0.0;
var refEnergy = 0.0;
var candEnergy = 0.0;
var start = Math.Max(0, -lag);
var end = Math.Min(frameCount, frameCount - lag);
for (var index = start; index < end; index++)
{
var left = reference[index];
var right = candidate[index + lag];
score += left * right;
refEnergy += left * left;
candEnergy += right * right;
}
return score / Math.Sqrt(Math.Max(1.0e-18, refEnergy * candEnergy));
}
static double[] BuildCompositeProbeBands(int sampleRate)
{
var nyquistGuard = sampleRate * 0.45;
return new[] { 160.0, 250.0, 400.0, 630.0, 1_000.0, 1_600.0, 2_500.0, 3_600.0, 5_600.0, 8_000.0, 10_000.0 }
.Where(frequency => frequency < nyquistGuard)
.ToArray();
}
static MimirChirpletBandResponse[] EstimateBandResponses(IReadOnlyList<float> samples, int sampleRate, IReadOnlyList<double> bands) =>
bands
.Select(centerHz =>
{
var magnitude = ToneEnergy(samples is float[] array ? array : samples.ToArray(), sampleRate, centerHz);
return new MimirChirpletBandResponse(centerHz, magnitude * magnitude);
})
.ToArray();
static double MedianPositiveEnergy(IEnumerable<double> energies)
{
var sorted = energies
.Where(static energy => energy > 1.0e-10)
.Order()
.ToArray();
return sorted.Length == 0 ? 0.01 : Math.Clamp(sorted[sorted.Length / 2], 0.0001, 1.0);
}
static double NoiseEstimate(IReadOnlyList<float> samples)
{
var sorted = samples
.Select(static sample => Math.Abs(sample))
.Order()
.ToArray();
if (sorted.Length == 0)
{
return 1.0e-5;
}
var percentile = sorted[Math.Clamp((int)Math.Round((sorted.Length - 1) * 0.20), 0, sorted.Length - 1)];
return Math.Clamp(percentile * 1.4826, 1.0e-5, 0.25);
}
static float[] NormalizePeak(IReadOnlyList<float> samples, float peak)
{
var max = samples.Select(static sample => Math.Abs(sample)).DefaultIfEmpty(0.0f).Max();
if (max <= peak || max <= 1.0e-9f)
{
return samples.ToArray();
}
var scale = peak / max;
return samples.Select(sample => sample * scale).ToArray();
}
static float[] RenderCalibrationToneStack(
int frameCount,
int sampleRate,

View file

@ -317,3 +317,4 @@
{"ts": "2026-06-01T03:54:22+01:00", "type": "verification", "status": "ok", "note": "Ran scheduler-driven Realtek calibration under artifacts/wasapi/realtk-scheduled-calibration-live/20260601-035354. The runner used MimirBioacousticProbeScheduler to choose four targeted probe steps, emitted through Speakers (Realtek(R) Audio) at gain 1.15, captured Scarlett ASIO, and wrote scheduled-response-bands.csv plus balance-plan.md. Local peak-vs-percentile-floor scoring found strong usable response around 910 Hz, 3.6 kHz, and 9.84-9.87 kHz on both shotgun and cardioid; both channels scored high SNR in those asked bands."}
{"ts": "2026-06-01T04:01:53+01:00", "type": "verification", "status": "ok", "note": "Bioacoustic scheduler now shares the targeted probe planner's measurable frequency boundary. The scheduler filters residual confidence to schedulable bands before choosing cadence/status and passes the same 120 Hz-12 kHz window to the planner, so unmeasurable 18.5 kHz residuals no longer keep live status pinned to a band the Realtek/speaker/Scarlett path cannot answer. Verified with --bioacoustic-probe-scheduler-smoke and --targeted-bioacoustic-probe-smoke."}
{"ts": "2026-06-01T04:13:53+01:00", "type": "verification", "status": "ok", "note": "Added calibrated-band coherence suppression to MimirCalibratedAudioCompositeBuilder. The composite path now aligns sources, applies measured-band correction, scores phase/energy coherence per calibrated band, suppresses incoherent measured-band components, and reports source/band coherence. Verified with --dual-mic-coherence-cancellation-smoke: shared tones held ~0.999 coherence, a mic-local 3.52 kHz interferer scored ~0.001 coherence with ~0.948 suppression, poison energy fell 0.039195 to 0.001306, and SNR rose 1.366 dB to 16.304 dB."}
{"ts": "2026-06-01T04:24:57+01:00", "type": "verification", "status": "ok", "note": "Recorded a listenable Kiyo plus computed dual-mic composite bridge receipt at artifacts/runtime/kiyo-dual-mic-composite/kiyo-dual-mic-composite-latest.mp4. The script record-kiyo-dual-mic-composite.ps1 started Kiyo FFmpeg capture and Scarlett ASIO raw capture together, generated dual-mic-composite.wav via --asio-dual-mic-composite-wav, then muxed it. ffprobe verified 10.000 s of 640x480 30 fps H.264 video plus 9.994 s of 48 kHz mono AAC. The composite report estimated cardioid delay -141.019 samples (-2937.889 us) relative to shotgun and emitted 11 coherence band reports."}

View file

@ -649,6 +649,15 @@ accepted_design:
mic-local 3.52 kHz interferer scores about 0.001 coherence and 0.948
suppression, its measured energy falls from 0.039195 to 0.001306, and SNR
rises from 1.366 dB to 16.304 dB.
`scripts/record-kiyo-dual-mic-composite.ps1` is the first listenable bridge
receipt for that path. It starts Kiyo FFmpeg capture and Scarlett ASIO raw
capture together, converts ASIO channels 0/1 to a computed composite WAV
through `--asio-dual-mic-composite-wav`, then muxes the composite WAV with
Kiyo video. The 2026-06-01 receipt
`artifacts/runtime/kiyo-dual-mic-composite/kiyo-dual-mic-composite-latest.mp4`
has 10.000 s of 640x480 30 fps Kiyo video and 9.994 s of mono AAC composite
audio; the composite report estimated channel-1 delay at about -141.019
samples (-2937.889 us).
Faust/native DSP still owns the production hot lane; this C# path is a
contract, scorer, and offline diagnostic, not a competing live mixer.
`MimirTargetedBioacousticProbePlanner` is now the residual-driven probe