Publish runtime audio actuator commands

This commit is contained in:
Meta Vens 2026-05-27 18:03:25 +01:00
commit 03337cb62d
7 changed files with 169 additions and 6 deletions

View file

@ -136,10 +136,17 @@ a named invariant that the native runtime cannot protect yet.
- `faust/mimir_alignment_actuator.dsp` is the first Faust-owned sample movement
surface: six channels of bounded fractional delay and gain controls. Mimir
estimates delay/SRO; Faust/native DSP applies correction.
- `MimirAlignmentActuatorBank` is the runtime control-plane owner for those
estimates: it converts smoothed per-source sync state into nonnegative
holdback commands, preserves stable Faust `sourceN` slots for live sources,
reports source overflow, and keeps the reference holdback explicit instead of
pretending a measured-late stream can be advanced by a positive delay line.
- `MimirRuntime` updates audio sync analysis online as a bounded rotating
service and can emit live sync telemetry with
`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry read cached reports/states;
they do not run synchronization analysis.
they do not run synchronization analysis. The same cadence now publishes the
latest actuator command frame for telemetry/readout; Faust/native DSP still
owns actual sample movement.
- `MimirRuntime` no longer submits the legacy direct `AquariumSplineFrame`
spectrum dashboard. Live spectrum visualization authority is the
`AquariumFieldEvidenceFrame`: Mimir declares the rolling resource, emits a

View file

@ -301,7 +301,14 @@ six channels of bounded fractional delay/gain controls for Faust/native DSP,
and `Mimir.BufferSmoke --bioacoustic-actuator-self-test --sample-rate 48000
--delay-samples 317.375` proves the control loop shape by estimating a
bioacoustic delay, applying fractional correction, and remeasuring residual
below printed microsecond precision.
below printed microsecond precision. `MimirAlignmentActuatorBank` is now the
runtime command owner above that Faust surface: it converts smoothed sync states
into nonnegative holdback commands, keeps source-to-`sourceN` slots stable,
reports overflow beyond the six-source profile, and exposes the reference
holdback separately. `MimirRuntime` updates that command frame on the sync
analysis cadence and telemetry can print the current DSP targets. This is
control-plane publication only; Faust/native DSP still has to be wired as the
hot sample mover.
## Visual Fusion

View file

@ -68,6 +68,11 @@ if (args.Any(arg => string.Equals(arg, "--bioacoustic-actuator-self-test", Strin
ParseDoubleOption(args, "--delay-samples", 317.375));
}
if (args.Any(arg => string.Equals(arg, "--audio-actuator-bank-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunAudioActuatorBankSmoke();
}
if (args.Any(arg => string.Equals(arg, "--complex-contour-tracker-self-test", StringComparison.OrdinalIgnoreCase)))
{
return RunComplexContourTrackerSelfTest(
@ -1825,6 +1830,54 @@ static int RunBioacousticActuatorSelfTest(int sampleRate, double delaySamples)
: 1;
}
static int RunAudioActuatorBankSmoke()
{
var states = new[]
{
NewSyncState("mic-0-late", 120.0, 0.86),
NewSyncState("mic-1-early", -30.0, 0.90),
NewSyncState("mic-2-mid", 80.0, 0.75),
NewSyncState("mic-3-mid", 70.0, 0.75),
NewSyncState("mic-4-mid", 60.0, 0.75),
NewSyncState("mic-5-mid", 50.0, 0.75),
NewSyncState("mic-6-overflow", 40.0, 0.75),
};
var bank = new MimirAlignmentActuatorBank();
var frame = bank.Update(states, dtSeconds: 0.5);
var early = frame.Commands.First(command => string.Equals(command.SourceId, "mic-1-early", StringComparison.Ordinal));
var late = frame.Commands.First(command => string.Equals(command.SourceId, "mic-0-late", StringComparison.Ordinal));
var mid = frame.Commands.First(command => string.Equals(command.SourceId, "mic-2-mid", StringComparison.Ordinal));
Console.WriteLine(
$"audio-actuator-bank-smoke reference={frame.ReferenceSourceId} referenceHoldback={frame.ReferenceHoldbackSamples:0.000} commands={frame.Commands.Count} truncated={frame.TruncatedSourceCount} " +
$"lateDelay={late.TargetDelaySamples:0.000} midDelay={mid.TargetDelaySamples:0.000} earlyDelay={early.TargetDelaySamples:0.000} ratio={early.ResampleRatio:0.000000000}");
return string.Equals(frame.ReferenceSourceId, "loopback-scarlett-speakers", StringComparison.Ordinal) &&
Math.Abs(frame.ReferenceHoldbackSamples - 120.0) < 0.001 &&
frame.Commands.Count == MimirAlignmentActuatorProfile.SixSourceFaust.SourceCount &&
frame.TruncatedSourceCount == 1 &&
late.TargetDelaySamples <= mid.TargetDelaySamples &&
mid.TargetDelaySamples < early.TargetDelaySamples &&
early.FaustControls.ContainsKey("source1/delay_samples")
? 0
: 1;
}
static MimirAudioSynchronizationState NewSyncState(string sourceId, double delaySamples, double confidence) =>
new(
"loopback-scarlett-speakers",
sourceId,
192_000,
delaySamples,
delaySamples,
delaySamples * 1000.0 / 192_000,
0.0,
confidence,
[],
1_000_000_000L,
10,
10);
static int RunComplexContourTrackerSelfTest(int sampleRate, double delaySamples, double reflectionDelaySamples)
{
var renderer = new MimirBioacousticContestantRenderer(MimirBioacousticContestants.CanaryPacketTrill);

View file

@ -38,6 +38,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
private readonly MimirSynchronizationHub synchronization;
private readonly MimirFensalirFieldLowering fieldLowering;
private readonly MimirAudioSpectrumAnalyzer spectrumAnalyzer;
private readonly MimirAlignmentActuatorBank audioActuatorBank = new();
private readonly IReadOnlyList<MimirStreamSourceFactory> sourceFactories;
private readonly AquariumUiDocument ui;
private readonly AquariumAudioDocument audio = new();
@ -74,6 +75,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
private long spectrumHistorySequence;
private readonly Queue<MimirSpectrumHistoryFrame> spectrumHistory = new();
private AquariumRuntimeServices runtimeServices = AquariumRuntimeServices.Empty;
private MimirActuatorFrame lastAudioActuatorFrame = MimirActuatorFrame.Empty;
public MimirRuntime(AquariumRuntimeOptions options)
: this(options, MimirRuntimeConfiguration.Load())
@ -898,6 +900,9 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
lastAudioSyncAnalysisMilliseconds = stopwatch.Elapsed.TotalMilliseconds;
lastAudioSynchronizationReports = synchronization.AudioSynchronizationReports;
UpdatePassiveSynchronizationConfidence(lastAudioSynchronizationReports);
lastAudioActuatorFrame = audioActuatorBank.Update(
synchronization.AudioSynchronizationStates,
audioSyncUpdateIntervalSeconds);
nextAudioSyncSeconds = runtimeSeconds + audioSyncUpdateIntervalSeconds;
}
@ -983,6 +988,17 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
$"mimir-sync-state {state.ReferenceSourceId}->{state.SourceId} delaySamples={state.SmoothedDelaySamples:0.000000} delayUs={state.DelayMicroseconds:0.000} delayMs={state.DelayMilliseconds:0.000} sroPpm={state.SamplingRateOffsetPpm:0.000} confidence={state.Confidence:0.000}");
}
if (lastAudioActuatorFrame.Commands.Count > 0)
{
Console.WriteLine(
$"mimir-audio-actuator reference={lastAudioActuatorFrame.ReferenceSourceId} referenceHoldbackSamples={lastAudioActuatorFrame.ReferenceHoldbackSamples:0.000000} commands={lastAudioActuatorFrame.Commands.Count} truncated={lastAudioActuatorFrame.TruncatedSourceCount}");
foreach (var command in lastAudioActuatorFrame.Commands.OrderBy(command => command.SourceId, StringComparer.Ordinal))
{
Console.WriteLine(
$"mimir-audio-actuator-command source={command.SourceId} delaySamples={command.TargetDelaySamples:0.000000} ratio={command.ResampleRatio:0.000000000} confidence={command.Confidence:0.000} controls={command.FaustControls.Count}");
}
}
foreach (var trace in synchronization.AudioSynchronizationDecodeTraces.OrderBy(trace => trace.SourceId, StringComparer.Ordinal))
{
Console.WriteLine(
@ -1261,10 +1277,10 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
private string DescribeAlignedAudio()
{
var states = synchronization.AudioSynchronizationStates;
return states.Count == 0
var commands = lastAudioActuatorFrame.Commands;
return commands.Count == 0
? "no aligned state"
: $"{states.Count + 1}ch state-ready";
: $"{commands.Count + 1}ch commands-ready refHold={lastAudioActuatorFrame.ReferenceHoldbackSamples:0.000} samples";
}
private string DescribeAudioSyncState()

View file

@ -24,6 +24,15 @@ public sealed record MimirActuatorCommand(
double Confidence,
IReadOnlyDictionary<string, float> FaustControls);
public sealed record MimirActuatorFrame(
string ReferenceSourceId,
double ReferenceHoldbackSamples,
IReadOnlyList<MimirActuatorCommand> Commands,
int TruncatedSourceCount)
{
public static MimirActuatorFrame Empty { get; } = new("", 0.0, [], 0);
}
public sealed record MimirActuatorControllerOptions(
double MinimumConfidence = 0.05,
double MinimumDtSeconds = 0.001,
@ -86,3 +95,68 @@ public sealed class MimirSroPllActuatorController(
public double Confidence { get; set; }
}
}
public sealed class MimirAlignmentActuatorBank(
MimirAlignmentActuatorProfile? profile = null,
MimirSroPllActuatorController? controller = null)
{
private readonly MimirAlignmentActuatorProfile profile = profile ?? MimirAlignmentActuatorProfile.SixSourceFaust;
private readonly MimirSroPllActuatorController controller = controller ?? new(profile);
private readonly Dictionary<string, int> sourceSlots = new(StringComparer.Ordinal);
public MimirActuatorFrame Update(IReadOnlyList<MimirAudioSynchronizationState> states, double dtSeconds)
{
if (states.Count == 0)
{
return MimirActuatorFrame.Empty;
}
var ordered = states
.Where(state => !string.IsNullOrWhiteSpace(state.SourceId))
.OrderBy(state => state.SourceId, StringComparer.Ordinal)
.ToArray();
if (ordered.Length == 0)
{
return MimirActuatorFrame.Empty;
}
var referenceSourceId = ordered[0].ReferenceSourceId;
var referenceHoldbackSamples = Math.Clamp(
ordered.Max(state => Math.Max(0.0, state.SmoothedDelaySamples)),
0.0,
profile.MaxDelaySamples);
var commands = new List<MimirActuatorCommand>(Math.Min(ordered.Length, profile.SourceCount));
var truncatedSourceCount = 0;
foreach (var state in ordered)
{
if (!sourceSlots.TryGetValue(state.SourceId, out var sourceIndex))
{
if (sourceSlots.Count >= profile.SourceCount)
{
truncatedSourceCount++;
continue;
}
sourceIndex = sourceSlots.Count;
sourceSlots[state.SourceId] = sourceIndex;
}
var holdbackSamples = Math.Clamp(
referenceHoldbackSamples - state.SmoothedDelaySamples,
0.0,
profile.MaxDelaySamples);
commands.Add(controller.Update(
state.SourceId,
sourceIndex,
holdbackSamples,
state.Confidence,
dtSeconds));
}
return new MimirActuatorFrame(
referenceSourceId,
referenceHoldbackSamples,
commands,
truncatedSourceCount);
}
}

View file

@ -183,3 +183,4 @@
{"ts": "2026-05-27T16:35:32+01:00", "type": "verification", "status": "ok", "note": "Media Foundation GPU decode is now a Mimir-owned producer path. native/camera_capture/mimir_mf_gpu_capture.dll runs SourceReader with a D3D11 device manager, selects Kiyo Pro compressed modes, decodes MJPG/H264 on the GPU, copies the decoded GPU surface into a shared BGRA D3D11 texture, and MimirMediaFoundationGpuVideoCaptureDriver publishes a shared-d3d11-texture handle with empty live CPU payload bytes. Kiyo Pro 1920x1080 MJPG->RGB32 and H264->RGB32 smokes passed and lowered to valid Fensalir camera resources. Direct shared NV12 texture creation failed with E_INVALIDARG, so planar NV12 remains future plane-aware interop or GPU bridge-copy work."}
{"ts": "2026-05-27T17:19:41+01:00", "type": "verification", "status": "ok", "note": "The MF GPU direct texture path now carries D3D11/D3D12 synchronization instead of relying on flush timing. native/camera_capture/mimir_mf_gpu_capture.dll creates a shared D3D11 fence, signals it after copying the decoded GPU surface into the shared BGRA texture, and MimirMediaFoundationGpuVideoCaptureDriver carries the fence handle/value through MimirVideoFrameDescriptor into Fensalir field resource declarations. Kiyo Pro 1920x1080 MJPG->RGB32 and H264->RGB32 smokes passed with zero live CPU payload bytes, nonzero shared texture handles, nonzero producer fence handles, and valid Fensalir camera resources."}
{"ts": "2026-05-27T17:53:38+01:00", "type": "decision", "status": "ok", "note": "Synchronized camera Texture2D resources are not a mandate to invent production visual fusion now. Camera image claims remain debug-view-only until the renderer and audio work are fixed; future camera fusion/feature-extraction lowerings need a separate selected policy decision."}
{"ts": "2026-05-27T18:03:11+01:00", "type": "decision", "status": "ok", "note": "Runtime audio sync states now derive explicit actuator command frames through MimirAlignmentActuatorBank: measured delay remains analyzer truth, the bank converts offsets into nonnegative Faust holdback controls with stable source slots, reference holdback, and overflow reporting. This is control-plane publication only; Faust/native DSP still owns live sample movement."}

View file

@ -179,7 +179,12 @@ accepted_design:
measurement-to-sample-movement loop: estimate bioacoustic delay, apply
fractional correction, and remeasure residual below printed microsecond
precision. `faust/mimir_alignment_actuator.dsp` is the first Faust-owned
actuator surface for six channels of bounded fractional delay/gain. The synthetic
actuator surface for six channels of bounded fractional delay/gain.
`MimirAlignmentActuatorBank` now converts runtime sync states into explicit
nonnegative Faust holdback commands with stable source slots, a separate
reference holdback, and overflow reporting; this publishes DSP control
targets but does not move live samples until the Faust/native hot path is
wired. The synthetic
codebook proof is `dotnet run --project
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`.
`Mimir.BufferSmoke --render-chirp-bin-f32` plus