Add Fensalir program control surface
This commit is contained in:
Meta Vens
parent
bcb85468c8
commit
293fbd1cf8
8 changed files with 585 additions and 7 deletions
|
|
@ -201,6 +201,15 @@ a named invariant that the native runtime cannot protect yet.
|
|||
The display feed remains derived evidence, not an independent timing owner.
|
||||
`Mimir.BufferSmoke --synchronized-buffer-planner-smoke` proves the aligned
|
||||
local-camera, Raven-display, loopback, and mic buffer shape.
|
||||
- `MimirPresentationControlState` owns operator intent for the Fensalir program
|
||||
surface: video feed visibility/solo/opacity/layer order, audio mute/solo/gain,
|
||||
and global LUT preset selection. The `Mimir Program` UI panel is deliberately
|
||||
compact and fadeable. Video choices filter production surface intents before
|
||||
Fensalir composition; audio choices modify Faust gain controls and the samples
|
||||
sent through Fensalir streaming DSP. LUT presets are typed postprocess state
|
||||
with preset paths and strength; until the renderer grows LUT texture sampling,
|
||||
preset exposure and bloom are mapped into existing `GraphicsSettings`.
|
||||
`Mimir.BufferSmoke --presentation-control-smoke` proves the state owner.
|
||||
- `MimirRuntime` no longer submits the legacy direct `AquariumSplineFrame`
|
||||
spectrum dashboard. Live spectrum visualization authority is the
|
||||
`AquariumFieldEvidenceFrame`: Mimir declares the rolling resource, emits a
|
||||
|
|
|
|||
42
docs/program-control-surface.md
Normal file
42
docs/program-control-surface.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Program Control Surface
|
||||
|
||||
Mimir now exposes a compact Fensalir-hosted program panel for final presentation
|
||||
intent.
|
||||
|
||||
Authority map:
|
||||
|
||||
- Owner: `MimirPresentationControlState` owns operator intent for program video,
|
||||
audio mix, and global color preset.
|
||||
- Inputs: runtime rolling buffers, aligned audio lanes, LUT preset selection,
|
||||
feed visibility, feed opacity, stem mute/solo, and stem gain.
|
||||
- Outputs: filtered video surface intents, Faust/native DSP gain controls,
|
||||
sample gain applied before Fensalir streaming DSP, and a postprocess control
|
||||
snapshot.
|
||||
- Derived state: feed lists are derived from active rolling buffers. LUT preset
|
||||
exposure and bloom are currently mapped into `GraphicsSettings`.
|
||||
- Forbidden writers: OBS, bridge endpoints, debug readouts, and network arrival
|
||||
timestamps must not choose program composition.
|
||||
- Shared paths: local cameras, Raven display capture, and future network video
|
||||
feeds all enter rolling buffers first, then the presentation state decides
|
||||
whether they contribute to production surface intent.
|
||||
- Deletion line: direct compositor shortcuts outside Fensalir FieldEvidence and
|
||||
audio DSP are diagnostic only.
|
||||
|
||||
The current UI lives in the `Mimir Program` panel:
|
||||
|
||||
- `Video`: select feed, visible, solo, opacity, layer ordering.
|
||||
- `Audio`: select stem/source, mute, solo, gain.
|
||||
- `Color`: LUT preset and LUT strength.
|
||||
|
||||
The shader-side LUT path is explicit state, not finished renderer authority yet.
|
||||
`MimirLookupTablePreset` carries `LutPath`, strength, contrast, saturation,
|
||||
temperature, and tint so Fensalir can consume the same contract when the global
|
||||
post shader grows real LUT sampling. Until that renderer cut lands, Mimir maps
|
||||
the preset exposure and bloom values into existing Fensalir `GraphicsSettings`.
|
||||
|
||||
Smoke proof:
|
||||
|
||||
```powershell
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --presentation-control-smoke
|
||||
```
|
||||
|
||||
|
|
@ -374,6 +374,14 @@ shape is explicit: Raven controls both display pixels and an audio timing signal
|
|||
routed into Scarlett, so Scarlett-decoded Raven audio evidence can earn a
|
||||
`raven-sync` correction that is applied to the `raven-display` buffer. Network
|
||||
arrival timestamps remain metadata, not timing authority.
|
||||
`MimirPresentationControlState` now owns the Fensalir program-control intent:
|
||||
video feed visibility/solo/opacity/layer order, audio mute/solo/gain, and
|
||||
global LUT preset selection. The `Mimir Program` panel exposes those controls
|
||||
without giving OBS or bridge endpoints composition authority. Video controls
|
||||
filter production surface intents before Fensalir composition; audio controls
|
||||
modify Faust gain controls and sample gain before Fensalir streaming DSP. LUT
|
||||
presets are typed postprocess state, with exposure/bloom currently mapped into
|
||||
`GraphicsSettings` and LUT texture sampling left as the next renderer hook.
|
||||
|
||||
## Visual Fusion
|
||||
|
||||
|
|
|
|||
|
|
@ -79,6 +79,11 @@ if (args.Any(arg => string.Equals(arg, "--synchronized-buffer-planner-smoke", St
|
|||
return RunSynchronizedBufferPlannerSmoke();
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--presentation-control-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunPresentationControlSmoke();
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--audio-stem-publication-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunAudioStemPublicationSmoke();
|
||||
|
|
@ -1935,6 +1940,41 @@ static int RunSynchronizedBufferPlannerSmoke()
|
|||
: 1;
|
||||
}
|
||||
|
||||
static int RunPresentationControlSmoke()
|
||||
{
|
||||
var controls = new MimirPresentationControlState();
|
||||
var camera = NewVideoBuffer("camera-main", MimirStreamOrigin.LocalDevice, 1_000_000_000, 1280, 720, MimirVideoPixelFormat.Bgra8, 1);
|
||||
var display = NewVideoBuffer("raven-display", MimirStreamOrigin.Network, 1_000_000_000, 1920, 1080, MimirVideoPixelFormat.Bgra8, 2, "raven-sync");
|
||||
var mic = NewAudioBlockBuffer("asio-ch0", 1_000_000_000, 48_000, 480, 3);
|
||||
var loopback = NewAudioBlockBuffer("asio-ch2", 1_000_000_000, 48_000, 480, 4);
|
||||
controls.SyncFromBuffers([camera, display, mic, loopback]);
|
||||
|
||||
controls.SelectedVideoIndex = 1;
|
||||
controls.SelectedVideo!.Solo = true;
|
||||
controls.SelectedVideo.Opacity = 0.42f;
|
||||
controls.SelectedAudioIndex = 0;
|
||||
controls.SelectedAudio!.Muted = true;
|
||||
controls.SelectLutPreset(2);
|
||||
controls.LutStrength = 0.65f;
|
||||
controls.RefreshPostprocess();
|
||||
|
||||
Console.WriteLine(
|
||||
$"presentation-control-smoke video={controls.VideoFeeds.Count} audio={controls.AudioFeeds.Count} " +
|
||||
$"cameraIncluded={controls.IncludesVideo("camera-main")} displayIncluded={controls.IncludesVideo("raven-display")} displayOpacity={controls.VideoOpacity("raven-display"):0.00} " +
|
||||
$"micGain={controls.AudioGain("asio-ch0"):0.00} lut={controls.Postprocess.PresetId} strength={controls.Postprocess.LutStrength:0.00}");
|
||||
|
||||
return controls.VideoFeeds.Count == 2 &&
|
||||
controls.AudioFeeds.Count == 2 &&
|
||||
!controls.IncludesVideo("camera-main") &&
|
||||
controls.IncludesVideo("raven-display") &&
|
||||
Math.Abs(controls.VideoOpacity("raven-display") - 0.42f) < 0.001f &&
|
||||
controls.AudioGain("asio-ch0") == 0.0f &&
|
||||
string.Equals(controls.Postprocess.PresetId, "soft-film", StringComparison.Ordinal) &&
|
||||
Math.Abs(controls.Postprocess.LutStrength - 0.65f) < 0.001f
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
||||
static int RunAudioStemPublicationSmoke()
|
||||
{
|
||||
var publication = new MimirObsStemPublicationState(MimirObsPublicationConfigurations.AlignmentActuatorStemBus);
|
||||
|
|
|
|||
|
|
@ -40,6 +40,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
private readonly MimirFensalirFieldLowering fieldLowering;
|
||||
private readonly MimirAudioSpectrumAnalyzer spectrumAnalyzer;
|
||||
private readonly MimirAlignmentActuatorBank audioActuatorBank = new();
|
||||
private readonly MimirPresentationControlState presentationControls = new();
|
||||
private readonly MimirObsStemPublicationState obsStemPublication = new(MimirObsPublicationConfigurations.AlignmentActuatorStemBus);
|
||||
private readonly MimirObsStemSharedMemoryPublisher? obsStemPublisher;
|
||||
private readonly IReadOnlyList<MimirStreamSourceFactory> sourceFactories;
|
||||
|
|
@ -141,7 +142,9 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
RenderDebugMode: 0,
|
||||
SceneExposure: 1.0f,
|
||||
BloomIntensity: 0.075f,
|
||||
BloomVeilIntensity: 0.0f);
|
||||
BloomVeilIntensity: 0.0f,
|
||||
FieldReservoirMode: GraphicsSettings.FieldReservoirModeNativeDomain,
|
||||
FieldReservoirScale: 0.5f);
|
||||
}
|
||||
|
||||
obsStemPublisher = CreateObsStemPublisher();
|
||||
|
|
@ -151,6 +154,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
string.Equals(profile.Id, settings.Audio.BioacousticWitnessProfileId, StringComparison.OrdinalIgnoreCase))
|
||||
?? MimirBioacousticContestants.CanaryPacketTrill)
|
||||
: null;
|
||||
presentationControls.SyncFromBuffers(synchronization.Buffers.Buffers);
|
||||
nextAudioSyncSeconds = audioSyncUpdateIntervalSeconds;
|
||||
nextTelemetrySeconds = telemetryIntervalSeconds;
|
||||
ui = CreateUi();
|
||||
|
|
@ -274,7 +278,12 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
continue;
|
||||
}
|
||||
|
||||
intents.Add(BuildSurfaceIntent(window, observation));
|
||||
if (!presentationControls.IncludesVideo(window.StreamId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
intents.Add(BuildSurfaceIntent(window, observation, presentationControls.VideoOpacity(window.StreamId)));
|
||||
}
|
||||
|
||||
var frame = fieldLowering.BuildFieldEvidenceFrame(windows, observations, constraints, intents);
|
||||
|
|
@ -299,7 +308,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
}
|
||||
}
|
||||
|
||||
private static MimirSurfaceIntent BuildSurfaceIntent(MimirRollingStreamWindow window, MimirObservation observation)
|
||||
private static MimirSurfaceIntent BuildSurfaceIntent(MimirRollingStreamWindow window, MimirObservation observation, float opacity = 1.0f)
|
||||
{
|
||||
if (window.SourceKind == MimirStreamKind.Audio)
|
||||
{
|
||||
|
|
@ -314,7 +323,15 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
Purpose: MimirSurfaceIntentPurpose.Debug);
|
||||
}
|
||||
|
||||
return MimirFensalirBridgeMapper.MapDefaultSurfaceIntent(window, observation);
|
||||
return new MimirSurfaceIntent(
|
||||
IntentKey: $"{window.WindowId}:surface",
|
||||
SourceObservationKeys: [observation.ObservationKey],
|
||||
Domain: MimirSurfaceDomain.CameraImage,
|
||||
Axes: new MimirSurfaceAxes("image-x", "image-y", "time-age", "stream"),
|
||||
SupportPolicy: new MimirSurfaceSupportPolicy("program-composite", 0.0, window.Duration.TotalSeconds),
|
||||
MaterialGraph: new MimirSurfaceMaterialIntent("program-video", "source-evidence", Math.Clamp(opacity, 0.0f, 1.0f)),
|
||||
UpdateBudget: new MimirSurfaceUpdateBudget(0.0, 1, Math.Max(0, window.Payload.ByteLength)),
|
||||
Purpose: MimirSurfaceIntentPurpose.Production);
|
||||
}
|
||||
|
||||
private AquariumFieldEvidenceFrame AddSpectrumFieldEvidence(AquariumFieldEvidenceFrame frame)
|
||||
|
|
@ -690,6 +707,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
public void RegisterStreamSource(IMimirStreamSource source)
|
||||
{
|
||||
synchronization.AddSource(source);
|
||||
presentationControls.SyncFromBuffers(synchronization.Buffers.Buffers);
|
||||
}
|
||||
|
||||
public void Start()
|
||||
|
|
@ -707,6 +725,8 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
|
||||
runtimeSeconds += Math.Max(deltaSeconds, 0.0f);
|
||||
lastPollCount = synchronization.PollSources();
|
||||
presentationControls.SyncFromBuffers(synchronization.Buffers.Buffers);
|
||||
ApplyPresentationPostprocess();
|
||||
UpdateObsStemPublication();
|
||||
QueueCalibrationTimeline();
|
||||
UpdateAudioSpectra();
|
||||
|
|
@ -747,7 +767,70 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
private AquariumUiDocument CreateUi()
|
||||
{
|
||||
return new AquariumUiDocument()
|
||||
.Panel("Mimir Sync", 18.0f, 82.0f, 390.0f, panel =>
|
||||
.Panel("Mimir Program", 18.0f, 82.0f, 360.0f, fadeWhenMouseDistant: true, panel =>
|
||||
{
|
||||
panel.Section("Video");
|
||||
panel.Options("Feed", () => presentationControls.SelectedVideoIndex, value => presentationControls.SelectedVideoIndex = value, VideoFeedOptions(), "Selects the video feed edited by the controls below.");
|
||||
panel.Toggle("Visible", () => presentationControls.SelectedVideo?.Enabled ?? false, value =>
|
||||
{
|
||||
if (presentationControls.SelectedVideo is { } feed)
|
||||
{
|
||||
feed.Enabled = value;
|
||||
}
|
||||
});
|
||||
panel.Toggle("Solo", () => presentationControls.SelectedVideo?.Solo ?? false, value =>
|
||||
{
|
||||
if (presentationControls.SelectedVideo is { } feed)
|
||||
{
|
||||
feed.Solo = value;
|
||||
}
|
||||
});
|
||||
panel.Slider("Opacity", () => presentationControls.SelectedVideo?.Opacity ?? 0.0f, value =>
|
||||
{
|
||||
if (presentationControls.SelectedVideo is { } feed)
|
||||
{
|
||||
feed.Opacity = Math.Clamp(value, 0.0f, 1.0f);
|
||||
}
|
||||
}, 0.0f, 1.0f, "0.00");
|
||||
panel.Button("Layer Up", () => presentationControls.MoveSelectedVideo(-1));
|
||||
panel.Button("Layer Down", () => presentationControls.MoveSelectedVideo(1));
|
||||
panel.Readout("Composite", DescribeProgramVideo);
|
||||
|
||||
panel.Section("Audio");
|
||||
panel.Options("Stem", () => presentationControls.SelectedAudioIndex, value => presentationControls.SelectedAudioIndex = value, AudioFeedOptions(), "Selects the audio lane edited by the controls below.");
|
||||
panel.Toggle("Mute", () => presentationControls.SelectedAudio?.Muted ?? false, value =>
|
||||
{
|
||||
if (presentationControls.SelectedAudio is { } feed)
|
||||
{
|
||||
feed.Muted = value;
|
||||
}
|
||||
});
|
||||
panel.Toggle("Solo Stem", () => presentationControls.SelectedAudio?.Solo ?? false, value =>
|
||||
{
|
||||
if (presentationControls.SelectedAudio is { } feed)
|
||||
{
|
||||
feed.Solo = value;
|
||||
}
|
||||
});
|
||||
panel.Slider("Gain", () => presentationControls.SelectedAudio?.Gain ?? 0.0f, value =>
|
||||
{
|
||||
if (presentationControls.SelectedAudio is { } feed)
|
||||
{
|
||||
feed.Gain = Math.Clamp(value, 0.0f, 2.0f);
|
||||
}
|
||||
}, 0.0f, 2.0f, "0.00");
|
||||
panel.Readout("Mix", DescribeProgramAudio);
|
||||
|
||||
panel.Section("Color");
|
||||
panel.Options("LUT", () => presentationControls.SelectedLutPresetIndex, presentationControls.SelectLutPreset, LutPresetOptions(), "Global color lookup table preset for the Fensalir post pass.");
|
||||
panel.Slider("LUT Strength", () => presentationControls.LutStrength, value =>
|
||||
{
|
||||
presentationControls.LutStrength = Math.Clamp(value, 0.0f, 1.0f);
|
||||
presentationControls.RefreshPostprocess();
|
||||
}, 0.0f, 1.0f, "0.00");
|
||||
panel.Readout("Post", DescribePostprocess);
|
||||
})
|
||||
.Panel("Mimir Sync", 392.0f, 82.0f, 390.0f, fadeWhenMouseDistant: true, panel =>
|
||||
{
|
||||
panel.Section("Rolling Buffers");
|
||||
panel.Readout("Window", () => $"{synchronization.Settings.BufferDuration.TotalSeconds:0.###}s");
|
||||
|
|
@ -817,6 +900,18 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
return app.Plan;
|
||||
}
|
||||
|
||||
private void ApplyPresentationPostprocess()
|
||||
{
|
||||
presentationControls.RefreshPostprocess();
|
||||
var post = presentationControls.Postprocess;
|
||||
GraphicsSettings = GraphicsSettings with
|
||||
{
|
||||
SceneExposure = Math.Clamp(post.Exposure, GraphicsSettings.MinSceneExposure, GraphicsSettings.MaxSceneExposure),
|
||||
BloomIntensity = Math.Clamp(post.BloomIntensity, GraphicsSettings.MinBloomIntensity, GraphicsSettings.MaxBloomIntensity),
|
||||
BloomVeilIntensity = Math.Clamp(post.BloomVeilIntensity, GraphicsSettings.MinBloomVeilIntensity, GraphicsSettings.MaxBloomVeilIntensity),
|
||||
};
|
||||
}
|
||||
|
||||
private void QueueCalibrationTimeline()
|
||||
{
|
||||
if (!ShouldEmitCalibrationTimeline())
|
||||
|
|
@ -942,6 +1037,61 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
return string.Join(" | ", synchronization.Buffers.Buffers.Select(DescribeBuffer));
|
||||
}
|
||||
|
||||
private IReadOnlyList<AquariumUiOption> VideoFeedOptions()
|
||||
{
|
||||
var feeds = presentationControls.VideoFeeds;
|
||||
return feeds.Count == 0
|
||||
? [new AquariumUiOption(0, "No video")]
|
||||
: feeds.Select((feed, index) => new AquariumUiOption(index, $"{feed.Layer + 1}. {feed.DisplayName}")).ToArray();
|
||||
}
|
||||
|
||||
private IReadOnlyList<AquariumUiOption> AudioFeedOptions()
|
||||
{
|
||||
var feeds = presentationControls.AudioFeeds;
|
||||
return feeds.Count == 0
|
||||
? [new AquariumUiOption(0, "No audio")]
|
||||
: feeds.Select((feed, index) => new AquariumUiOption(index, feed.DisplayName)).ToArray();
|
||||
}
|
||||
|
||||
private IReadOnlyList<AquariumUiOption> LutPresetOptions() =>
|
||||
presentationControls.LutPresets
|
||||
.Select((preset, index) => new AquariumUiOption(index, preset.DisplayName))
|
||||
.ToArray();
|
||||
|
||||
private string DescribeProgramVideo()
|
||||
{
|
||||
var feeds = presentationControls.VideoFeeds;
|
||||
if (feeds.Count == 0)
|
||||
{
|
||||
return "no video buffers";
|
||||
}
|
||||
|
||||
var active = feeds
|
||||
.Where(feed => presentationControls.IncludesVideo(feed.SourceId))
|
||||
.Select(feed => $"{feed.Layer + 1}:{feed.SourceId}@{feed.Opacity:0.00}");
|
||||
return string.Join(" | ", active.DefaultIfEmpty("black"));
|
||||
}
|
||||
|
||||
private string DescribeProgramAudio()
|
||||
{
|
||||
var feeds = presentationControls.AudioFeeds;
|
||||
if (feeds.Count == 0)
|
||||
{
|
||||
return "no audio buffers";
|
||||
}
|
||||
|
||||
var active = feeds
|
||||
.Select(feed => $"{feed.SourceId}:{presentationControls.AudioGain(feed.SourceId):0.00}")
|
||||
.ToArray();
|
||||
return string.Join(" | ", active);
|
||||
}
|
||||
|
||||
private string DescribePostprocess()
|
||||
{
|
||||
var post = presentationControls.Postprocess;
|
||||
return $"{post.PresetName} strength={post.LutStrength:0.00} exp={post.Exposure:0.000} contrast={post.Contrast:0.00} sat={post.Saturation:0.00} lut={post.LutPath}";
|
||||
}
|
||||
|
||||
private void UpdateAudioSynchronization()
|
||||
{
|
||||
if (runtimeSeconds < nextAudioSyncSeconds)
|
||||
|
|
@ -984,7 +1134,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
command.TargetDelaySamples,
|
||||
command.ResampleRatio,
|
||||
command.Confidence,
|
||||
command.FaustControls))
|
||||
ApplyPresentationGain(command)))
|
||||
.ToArray(),
|
||||
frame.TruncatedSourceCount,
|
||||
++audioActuatorFrameSequence));
|
||||
|
|
@ -1053,7 +1203,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
continue;
|
||||
}
|
||||
|
||||
var samples = ExtractMonoAudioBlock(latest, block);
|
||||
var samples = ApplyGain(ExtractMonoAudioBlock(latest, block), presentationControls.AudioGain(commands[index].SourceId));
|
||||
if (samples.Length == 0)
|
||||
{
|
||||
continue;
|
||||
|
|
@ -1091,6 +1241,18 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
sequence));
|
||||
}
|
||||
|
||||
private IReadOnlyDictionary<string, float> ApplyPresentationGain(MimirActuatorCommand command)
|
||||
{
|
||||
var gain = presentationControls.AudioGain(command.SourceId);
|
||||
var controls = new Dictionary<string, float>(command.FaustControls, StringComparer.Ordinal);
|
||||
foreach (var key in controls.Keys.Where(key => key.EndsWith("/gain", StringComparison.Ordinal)).ToArray())
|
||||
{
|
||||
controls[key] = Math.Clamp(controls[key] * gain, 0.0f, 2.0f);
|
||||
}
|
||||
|
||||
return controls;
|
||||
}
|
||||
|
||||
private void UpdateObsStemPublication()
|
||||
{
|
||||
var consumed = false;
|
||||
|
|
@ -1298,6 +1460,23 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec
|
|||
return output;
|
||||
}
|
||||
|
||||
private static float[] ApplyGain(float[] samples, float gain)
|
||||
{
|
||||
var safeGain = Math.Clamp(gain, 0.0f, 2.0f);
|
||||
if (samples.Length == 0 || Math.Abs(safeGain - 1.0f) < 0.0001f)
|
||||
{
|
||||
return samples;
|
||||
}
|
||||
|
||||
var output = new float[samples.Length];
|
||||
for (var index = 0; index < samples.Length; index++)
|
||||
{
|
||||
output[index] = Math.Clamp(samples[index] * safeGain, -1.0f, 1.0f);
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
private static int FaustSourceIndex(MimirActuatorCommand command)
|
||||
{
|
||||
foreach (var key in command.FaustControls.Keys)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,292 @@
|
|||
namespace Mimir.Runtime.Synchronization;
|
||||
|
||||
public sealed record MimirLookupTablePreset(
|
||||
string Id,
|
||||
string DisplayName,
|
||||
string LutPath,
|
||||
float Exposure,
|
||||
float Contrast,
|
||||
float Saturation,
|
||||
float Temperature,
|
||||
float Tint,
|
||||
float BloomIntensity,
|
||||
float BloomVeilIntensity)
|
||||
{
|
||||
public static IReadOnlyList<MimirLookupTablePreset> BuiltIn { get; } =
|
||||
[
|
||||
new(
|
||||
"neutral",
|
||||
"Neutral",
|
||||
"luts/neutral.cube",
|
||||
Exposure: 0.16f,
|
||||
Contrast: 1.00f,
|
||||
Saturation: 1.00f,
|
||||
Temperature: 0.00f,
|
||||
Tint: 0.00f,
|
||||
BloomIntensity: 0.072f,
|
||||
BloomVeilIntensity: 0.014f),
|
||||
new(
|
||||
"clean-rec709",
|
||||
"Clean Rec.709",
|
||||
"luts/clean-rec709.cube",
|
||||
Exposure: 0.18f,
|
||||
Contrast: 1.04f,
|
||||
Saturation: 0.98f,
|
||||
Temperature: 0.00f,
|
||||
Tint: 0.00f,
|
||||
BloomIntensity: 0.045f,
|
||||
BloomVeilIntensity: 0.006f),
|
||||
new(
|
||||
"soft-film",
|
||||
"Soft Film",
|
||||
"luts/soft-film.cube",
|
||||
Exposure: 0.15f,
|
||||
Contrast: 0.94f,
|
||||
Saturation: 1.08f,
|
||||
Temperature: 0.08f,
|
||||
Tint: 0.02f,
|
||||
BloomIntensity: 0.095f,
|
||||
BloomVeilIntensity: 0.022f),
|
||||
new(
|
||||
"cool-monitor",
|
||||
"Cool Monitor",
|
||||
"luts/cool-monitor.cube",
|
||||
Exposure: 0.17f,
|
||||
Contrast: 1.02f,
|
||||
Saturation: 0.92f,
|
||||
Temperature: -0.10f,
|
||||
Tint: 0.01f,
|
||||
BloomIntensity: 0.060f,
|
||||
BloomVeilIntensity: 0.010f),
|
||||
new(
|
||||
"warm-room",
|
||||
"Warm Room",
|
||||
"luts/warm-room.cube",
|
||||
Exposure: 0.14f,
|
||||
Contrast: 0.98f,
|
||||
Saturation: 1.06f,
|
||||
Temperature: 0.12f,
|
||||
Tint: -0.01f,
|
||||
BloomIntensity: 0.085f,
|
||||
BloomVeilIntensity: 0.018f),
|
||||
];
|
||||
}
|
||||
|
||||
public sealed class MimirVideoPresentationControl
|
||||
{
|
||||
public MimirVideoPresentationControl(string sourceId, string displayName, int layer)
|
||||
{
|
||||
SourceId = sourceId;
|
||||
DisplayName = string.IsNullOrWhiteSpace(displayName) ? sourceId : displayName;
|
||||
Layer = Math.Max(0, layer);
|
||||
}
|
||||
|
||||
public string SourceId { get; }
|
||||
|
||||
public string DisplayName { get; private set; }
|
||||
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
public bool Solo { get; set; }
|
||||
|
||||
public float Opacity { get; set; } = 1.0f;
|
||||
|
||||
public int Layer { get; set; }
|
||||
|
||||
public void RefreshLabel(string displayName)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(displayName))
|
||||
{
|
||||
DisplayName = displayName;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class MimirAudioPresentationControl
|
||||
{
|
||||
public MimirAudioPresentationControl(string sourceId, string displayName)
|
||||
{
|
||||
SourceId = sourceId;
|
||||
DisplayName = string.IsNullOrWhiteSpace(displayName) ? sourceId : displayName;
|
||||
}
|
||||
|
||||
public string SourceId { get; }
|
||||
|
||||
public string DisplayName { get; private set; }
|
||||
|
||||
public bool Muted { get; set; }
|
||||
|
||||
public bool Solo { get; set; }
|
||||
|
||||
public float Gain { get; set; } = 1.0f;
|
||||
|
||||
public void RefreshLabel(string displayName)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(displayName))
|
||||
{
|
||||
DisplayName = displayName;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record MimirPostprocessControlSnapshot(
|
||||
string PresetId,
|
||||
string PresetName,
|
||||
string LutPath,
|
||||
float LutStrength,
|
||||
float Exposure,
|
||||
float Contrast,
|
||||
float Saturation,
|
||||
float Temperature,
|
||||
float Tint,
|
||||
float BloomIntensity,
|
||||
float BloomVeilIntensity);
|
||||
|
||||
public sealed class MimirPresentationControlState
|
||||
{
|
||||
private readonly Dictionary<string, MimirVideoPresentationControl> video = new(StringComparer.Ordinal);
|
||||
private readonly Dictionary<string, MimirAudioPresentationControl> audio = new(StringComparer.Ordinal);
|
||||
|
||||
public IReadOnlyList<MimirLookupTablePreset> LutPresets => MimirLookupTablePreset.BuiltIn;
|
||||
|
||||
public int SelectedVideoIndex { get; set; }
|
||||
|
||||
public int SelectedAudioIndex { get; set; }
|
||||
|
||||
public int SelectedLutPresetIndex { get; private set; }
|
||||
|
||||
public float LutStrength { get; set; } = 1.0f;
|
||||
|
||||
public MimirPostprocessControlSnapshot Postprocess { get; private set; } =
|
||||
SnapshotFor(MimirLookupTablePreset.BuiltIn[0], 1.0f);
|
||||
|
||||
public IReadOnlyList<MimirVideoPresentationControl> VideoFeeds =>
|
||||
video.Values.OrderBy(feed => feed.Layer).ThenBy(feed => feed.SourceId, StringComparer.Ordinal).ToArray();
|
||||
|
||||
public IReadOnlyList<MimirAudioPresentationControl> AudioFeeds =>
|
||||
audio.Values.OrderBy(feed => feed.SourceId, StringComparer.Ordinal).ToArray();
|
||||
|
||||
public MimirVideoPresentationControl? SelectedVideo =>
|
||||
VideoFeeds.Count == 0 ? null : VideoFeeds[Math.Clamp(SelectedVideoIndex, 0, VideoFeeds.Count - 1)];
|
||||
|
||||
public MimirAudioPresentationControl? SelectedAudio =>
|
||||
AudioFeeds.Count == 0 ? null : AudioFeeds[Math.Clamp(SelectedAudioIndex, 0, AudioFeeds.Count - 1)];
|
||||
|
||||
public void SyncFromBuffers(IEnumerable<MimirRollingStreamBuffer> buffers)
|
||||
{
|
||||
var videoIndex = video.Count;
|
||||
foreach (var buffer in buffers)
|
||||
{
|
||||
if (buffer.Descriptor.Kind == MimirStreamKind.Video)
|
||||
{
|
||||
if (!video.TryGetValue(buffer.Descriptor.SourceId, out var control))
|
||||
{
|
||||
control = new MimirVideoPresentationControl(buffer.Descriptor.SourceId, buffer.Descriptor.Label, videoIndex++);
|
||||
video.Add(buffer.Descriptor.SourceId, control);
|
||||
}
|
||||
|
||||
control.RefreshLabel(buffer.Descriptor.Label);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (buffer.Descriptor.Kind == MimirStreamKind.Audio)
|
||||
{
|
||||
if (!audio.TryGetValue(buffer.Descriptor.SourceId, out var control))
|
||||
{
|
||||
control = new MimirAudioPresentationControl(buffer.Descriptor.SourceId, buffer.Descriptor.Label);
|
||||
audio.Add(buffer.Descriptor.SourceId, control);
|
||||
}
|
||||
|
||||
control.RefreshLabel(buffer.Descriptor.Label);
|
||||
}
|
||||
}
|
||||
|
||||
SelectedVideoIndex = ClampIndex(SelectedVideoIndex, VideoFeeds.Count);
|
||||
SelectedAudioIndex = ClampIndex(SelectedAudioIndex, AudioFeeds.Count);
|
||||
}
|
||||
|
||||
public bool IncludesVideo(string sourceId)
|
||||
{
|
||||
if (!video.TryGetValue(sourceId, out var control))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
var hasSolo = video.Values.Any(feed => feed.Solo);
|
||||
return hasSolo ? control.Solo : control.Enabled;
|
||||
}
|
||||
|
||||
public float VideoOpacity(string sourceId) =>
|
||||
video.TryGetValue(sourceId, out var control)
|
||||
? Math.Clamp(control.Opacity, 0.0f, 1.0f)
|
||||
: 1.0f;
|
||||
|
||||
public float AudioGain(string sourceId)
|
||||
{
|
||||
if (!audio.TryGetValue(sourceId, out var control))
|
||||
{
|
||||
return 1.0f;
|
||||
}
|
||||
|
||||
var hasSolo = audio.Values.Any(feed => feed.Solo);
|
||||
if (control.Muted || (hasSolo && !control.Solo))
|
||||
{
|
||||
return 0.0f;
|
||||
}
|
||||
|
||||
return Math.Clamp(control.Gain, 0.0f, 2.0f);
|
||||
}
|
||||
|
||||
public void MoveSelectedVideo(int delta)
|
||||
{
|
||||
var selected = SelectedVideo;
|
||||
if (selected == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
selected.Layer = Math.Clamp(selected.Layer + delta, 0, Math.Max(0, video.Count - 1));
|
||||
NormalizeVideoLayers();
|
||||
SelectedVideoIndex = VideoFeeds.ToList().FindIndex(feed => string.Equals(feed.SourceId, selected.SourceId, StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
public void SelectLutPreset(int index)
|
||||
{
|
||||
SelectedLutPresetIndex = ClampIndex(index, LutPresets.Count);
|
||||
Postprocess = SnapshotFor(LutPresets[SelectedLutPresetIndex], LutStrength);
|
||||
}
|
||||
|
||||
public void RefreshPostprocess()
|
||||
{
|
||||
Postprocess = SnapshotFor(LutPresets[SelectedLutPresetIndex], LutStrength);
|
||||
}
|
||||
|
||||
private void NormalizeVideoLayers()
|
||||
{
|
||||
var ordered = video.Values
|
||||
.OrderBy(feed => feed.Layer)
|
||||
.ThenBy(feed => feed.SourceId, StringComparer.Ordinal)
|
||||
.ToArray();
|
||||
for (var index = 0; index < ordered.Length; index++)
|
||||
{
|
||||
ordered[index].Layer = index;
|
||||
}
|
||||
}
|
||||
|
||||
private static int ClampIndex(int index, int count) =>
|
||||
count <= 0 ? 0 : Math.Clamp(index, 0, count - 1);
|
||||
|
||||
private static MimirPostprocessControlSnapshot SnapshotFor(MimirLookupTablePreset preset, float strength) =>
|
||||
new(
|
||||
preset.Id,
|
||||
preset.DisplayName,
|
||||
preset.LutPath,
|
||||
Math.Clamp(strength, 0.0f, 1.0f),
|
||||
preset.Exposure,
|
||||
preset.Contrast,
|
||||
preset.Saturation,
|
||||
preset.Temperature,
|
||||
preset.Tint,
|
||||
preset.BloomIntensity,
|
||||
preset.BloomVeilIntensity);
|
||||
}
|
||||
|
|
@ -228,3 +228,4 @@
|
|||
{"timestamp":"2026-05-29T01:46:00+01:00","type":"decision","status":"ok","note":"Fensalir TubeField spectrum rendering now uses rolling-buffer column packets as reservoir candidate authority rather than generated segment packets. Expansion emits one GPU-resident column packet per logical history/source column, ReSTIR evaluates Catmull-Rom SDF/material samples directly from the source buffer, and generated mesh draw args are zeroed for the live path. Verified through Mimir.App headless synthetic spectrum capture artifacts/runtime/mimir-column-packet-final-headless.png."}
|
||||
{"ts": "2026-05-29T22:23:04+01:00", "type": "decision", "status": "ok", "note": "Raven display/audio sync uses shared clock-domain correction: Raven display frames can be tagged raven-sync, while Raven audio routed into Scarlett earns the timing offset applied by MimirSynchronizedBufferPlanner; network arrival timestamps remain metadata."}
|
||||
{"ts": "2026-05-29T23:50:56+01:00", "type": "decision", "status": "ok", "note": "Fast stereo depth is mapped as a future Fensalir-owned D3D12 compute lane, with libSGM as permissive SGM provenance rather than a CUDA dependency. External TensorRT/monocular depth projects remain reference evidence only; live depth must consume Mimir-declared texture resources, calibration, and fences, then emit GPU disparity/depth/confidence resources and FieldEvidence claims."}
|
||||
{"ts": "2026-05-29T23:52:11+01:00", "type": "decision", "status": "ok", "note": "Mimir Program panel now owns Fensalir presentation intent: video visibility/solo/opacity/layer filters production surface intents, audio mute/solo/gain feeds Faust controls and streaming samples, and LUT presets are typed postprocess state with exposure/bloom mapped to current GraphicsSettings until renderer LUT sampling lands."}
|
||||
|
|
|
|||
|
|
@ -230,6 +230,13 @@ accepted_design:
|
|||
shared `clockDomainId`; the Raven path uses audio evidence routed into
|
||||
Scarlett to earn a `raven-sync` correction for Raven display frames.
|
||||
Network arrival timestamps remain metadata only.
|
||||
`MimirPresentationControlState` owns program-control intent for Fensalir:
|
||||
video feed visibility/solo/opacity/layer order, audio mute/solo/gain, and
|
||||
global LUT preset selection. The `Mimir Program` panel filters video
|
||||
production surface intents and applies audio mix gain before/inside the
|
||||
Faust streaming DSP path. LUT presets are typed postprocess state; exposure
|
||||
and bloom map into current Fensalir `GraphicsSettings`, while actual LUT
|
||||
texture sampling remains the named renderer hook.
|
||||
`scripts/build-obs-stem-plugin.ps1` stages the upstream OBS plugin template
|
||||
SDK under `artifacts/obs-sdk/` and builds the plugin DLL locally.
|
||||
The synthetic
|
||||
|
|
|
|||
Loading…
Reference in a new issue