GameCult/Mimir
Watch
9
0
Fork
You've already forked Mimir
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Lease camera textures from Fensalir

Browse source
This commit is contained in:
Meta Vens 2026-05-27 13:29:48 +01:00
commit daf1d4ef8a
10 changed files with 284 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

22
docs/implementation-plan.md
View file

@ -45,7 +45,12 @@ Payload handles are now demoted to names inside a resource authority. Mimir
declares live native/GPU payloads as typed Fensalir resources with shape,
residency, shader access, validity, version, and native handle metadata; claims
and lowering requests reference those resource keys. A string handle without a
matching resource declaration is not payload truth.
matching resource declaration is not payload truth. For live camera images, the
preferred authority is Fensalir-owned texture leasing: Mimir asks the engine
broker for a keyed `Texture2D` lease, writes decoded frames into the returned
shared D3D12 texture, signals the returned fence, and commits that fence value
before the resource key is sampled by shader lowerings. Foreign shared handles
remain import edges only.
The old script stack is gone. Do not add a compatibility edge unless it protects
a named invariant that the native runtime cannot protect yet.
@ -176,14 +181,19 @@ a named invariant that the native runtime cannot protect yet.
`TubeField` packet with no deferred requests.
- `MimirFensalirFieldLowering.BuildCameraObservationFrame` is the current
camera bridge proof. It lowers latest video buffers into FieldEvidence
observation claims, declares GPU/native video payloads as shared `Texture2D`
resources, and creates camera surface intent only when a resource exists.
observation claims, declares engine-owned or imported video payloads as
`Texture2D` resources, and creates camera surface intent only when a resource
exists. `MimirFensalirTextureLeaseClient` is the producer-side API for asking
Fensalir for a keyed D3D12 texture/fence lease before decoded camera frames
are written. `MimirRuntime` accepts `AquariumRuntimeServices` and commits
producer fence values for leased video payloads before Fensalir samples them.
Metadata-only cadence frames remain observations with empty payload handles;
they do not become fake render requests. The old direct
`AquariumGpuSensorFrame` builder has been removed from Mimir's proof path.
Fensalir can now resolve those shared `Texture2D` resources by native D3D12
handle and accepts Mimir video format names. `Mimir.BufferSmoke
--fensalir-camera-observation-smoke` verifies the split.
Fensalir can still resolve imported shared `Texture2D` resources by native
D3D12 handle and accepts Mimir video format names.
`Mimir.BufferSmoke --fensalir-camera-observation-smoke` verifies the split,
and `--fensalir-texture-lease-smoke` verifies the engine-owned lease path.
- Mimir's active proof path no longer uses the direct
`AquariumAcousticFieldFrame` builder either. Sync states lower into
FieldEvidence calibration constraints through `MapCalibrationConstraints`,

19
notes/current-system-map.md
View file

@ -326,14 +326,17 @@ shader lowering. Mimir-declared rendering buffers are Fensalir GPU resources in
the same runtime; native/shared handle metadata is an import edge, not a
separate authority regime. Rendering-relevant buffers move to GPU residency as
early as possible and stay there; CPU readback/tessellation is diagnostic only.
Camera descriptors now enter this bridge too: latest video buffers lower into
FieldEvidence camera observations, GPU/native handles declare shared
`Texture2D` resources, and metadata-only cadence frames do not create camera
surface intents or fake payload requests. The old direct `AquariumGpuSensorFrame`
bridge proof has been removed from Mimir's active proof path. Fensalir can now
resolve those shared `Texture2D` resources by native D3D12 handle and accepts
Mimir video format names. Camera image claims currently defer until Fensalir
owns a selected visual-fusion lowering for camera textures.
Camera descriptors now enter this bridge too. The preferred hot path is
Fensalir-owned texture leasing: the runtime receives `AquariumRuntimeServices`,
asks Fensalir's field resource broker for a keyed D3D12 `Texture2D` lease, the
camera/decode producer writes that shared texture and signals the producer
fence, then Mimir carries the same resource key through FieldEvidence. Fensalir
waits on committed producer fence values before resolving shader reads. Shared
foreign texture handles remain an import edge, not the primary camera authority.
Metadata-only cadence frames do not create camera surface intents or fake
payload requests. The old direct `AquariumGpuSensorFrame` bridge proof has been
removed from Mimir's active proof path. Camera image claims currently defer
until Fensalir owns a selected visual-fusion lowering for camera textures.
The current smokes prove planning, not visible packet rendering:
`--fensalir-field-evidence-smoke` now produces one planned resource-backed
`TubeField` packet from Mimir's spectrum intent, and

5
notes/fresh-workspace-handoff.md
View file

@ -43,6 +43,11 @@ Get-Content .\state\evidence.jsonl -Tail 8
- `native/reservoir` owns the lower shared-edge typed-handle invariant for
Fensalir/Faust binding work.
- Fensalir owns production GPU fusion, UI, and Spout2 publication.
- Live camera image buffers should use Fensalir-owned texture leases when they
are rendering inputs: Mimir asks the engine broker for a keyed D3D12
Texture2D/fence lease, writes decoded frames into that texture, commits the
producer fence value, and lowers the same resource key through FieldEvidence.
Foreign shared texture handles are import edges, not the primary hot path.
- The current Mimir/Fensalir rendering teardown map is
[[docs/fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild
Migration]]. Mimir's production visual output path must publish typed

120
src/Mimir.BufferSmoke/Program.cs
View file

@ -108,6 +108,11 @@ if (args.Any(arg => string.Equals(arg, "--fensalir-camera-observation-smoke", St
return RunFensalirCameraObservationSmoke();
}
if (args.Any(arg => string.Equals(arg, "--fensalir-texture-lease-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunFensalirTextureLeaseSmoke();
}
if (args.Any(arg => string.Equals(arg, "--mimir-spectrum-upload-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunMimirSpectrumUploadSmoke();
@ -718,6 +723,78 @@ static int RunFensalirCameraObservationSmoke()
: 1;
}
static int RunFensalirTextureLeaseSmoke()
{
var broker = new FakeFieldResourceBroker();
var client = new MimirFensalirTextureLeaseClient(broker);
var requested = new MimirFensalirTextureLeaseRequest(
"kiyo-pro-rgb",
1920,
1080,
MimirVideoPixelFormat.Bgra8,
1920 * 4,
DeviceTimestampNs: 1_000_000_000L,
Version: 17);
if (!client.TryLeaseTexture2D(requested, out var lease))
{
Console.Error.WriteLine("fensalir-texture-lease-smoke failed to lease texture");
return 1;
}
var source = new MimirNativeIngestStreamSource(
new MimirStreamDescriptor("kiyo-pro-rgb", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice));
using (var runtime = new MimirRuntime(
new AquariumRuntimeOptions(Headless: true, CultCachePath: null),
new MimirSynchronizationSettings(),
[source]))
{
runtime.AttachServices(new AquariumRuntimeServices(broker));
runtime.OnSceneReady();
source.PushVideoFrame(lease.Frame with { ProducerFenceValue = 41 }, 1_000_000_010L);
runtime.Update(0.016f, new InputState());
_ = runtime.Frame;
}
var buffer = new MimirRollingStreamBuffer(
new MimirStreamDescriptor("kiyo-pro-rgb", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice),
TimeSpan.FromSeconds(5));
buffer.Append(new MimirStreamSample(
"kiyo-pro-rgb",
MimirStreamKind.Video,
MimirStreamOrigin.LocalDevice,
TimestampNs: 1_000_000_000L,
ArrivalNs: 1_000_000_010L,
Sequence: 17,
PayloadHandle: 0,
ByteLength: 1920 * 1080 * 4,
Data: default,
VideoFrame: lease.Frame with { ProducerFenceValue = 41 }));
var frame = new MimirFensalirFieldLowering().BuildCameraObservationFrame([buffer]);
var validation = AquariumFieldEvidenceValidator.Validate(frame);
var expectedResourceKey = MimirFensalirTextureLeaseClient.ResourceKeyForSource("kiyo-pro-rgb");
Console.WriteLine(
$"fensalir-texture-lease-smoke leased={lease.IsValid} committed={broker.CommittedFenceValue > 0} resources={frame.Resources.Count} claims={frame.Claims.Count} errors={validation.HasErrors}");
return lease.IsValid &&
broker.CommittedResourceKey == expectedResourceKey &&
broker.CommittedVersion == 0 &&
broker.CommittedFenceValue == 41 &&
lease.Frame.ResourceKey == expectedResourceKey &&
lease.Frame.NativeHandle == 0xCAFE &&
lease.Frame.NativeHandleKind == "fensalir-owned-d3d12-texture2d" &&
frame.Resources.Count == 1 &&
frame.Resources[0].ResourceKey == expectedResourceKey &&
frame.Resources[0].NativeHandle == new IntPtr(0xCAFE) &&
frame.Resources[0].NativeHandleKind == "fensalir-owned-d3d12-texture2d" &&
frame.Claims.Any(claim => claim.PayloadHandle == expectedResourceKey) &&
!validation.HasErrors
? 0
: 1;
}
static int RunMimirSpectrumUploadSmoke()
{
var previousSyntheticPreview = Environment.GetEnvironmentVariable("MIMIR_SYNTHETIC_SPECTRUM_PREVIEW");
@ -5948,3 +6025,46 @@ public sealed record ComplexContourChannelModelEvaluationCase(
double DeltaAbsolutePredictionErrorMicroseconds,
double DeltaMeanAbsoluteErrorSamples,
double DeltaMeanAbsolutePhaseErrorRadians);
sealed class FakeFieldResourceBroker : IAquariumFieldResourceBroker
{
public string CommittedResourceKey { get; private set; } = "";
public ulong CommittedVersion { get; private set; }
public ulong CommittedFenceValue { get; private set; }
public AquariumFieldResourceLease LeaseTexture2D(AquariumTexture2DLeaseRequest request)
{
var declaration = new AquariumFieldResourceDeclaration(
request.ResourceKey,
AquariumFieldResourceKind.Texture2D,
AquariumFieldResourceResidency.SharedGpu,
AquariumFieldShaderAccess.ShaderResource,
request.Format,
request.Width,
request.Height,
DepthOrCount: 1,
StrideBytes: AquariumFieldResourceDeclaration.FormatStrideBytes(request.Format),
request.ValidFromNs,
request.ValidUntilNs,
request.Version,
new IntPtr(0xCAFE),
"fensalir-owned-d3d12-texture2d");
return new AquariumFieldResourceLease(
declaration,
new IntPtr(0xCAFE),
"fensalir-owned-d3d12-texture2d",
new IntPtr(0xF00D),
request.Version,
true);
}
public bool CommitLeaseVersion(string resourceKey, ulong version, ulong producerFenceValue)
{
CommittedResourceKey = resourceKey;
CommittedVersion = version;
CommittedFenceValue = producerFenceValue;
return true;
}
}

27
src/Mimir.Runtime/MimirRuntime.cs
View file

@ -11,7 +11,7 @@ using System.Text;
namespace Mimir.Runtime;
public sealed class MimirRuntime : IAquariumRuntime
public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesReceiver
{
private const float DefaultAudioSyncUpdateIntervalSeconds = 0.1f;
private const double HybridPassiveConfidenceThreshold = 0.12;
@ -73,6 +73,7 @@ public sealed class MimirRuntime : IAquariumRuntime
private IReadOnlyList<MimirAudioSpectrumSnapshot> lastAudioSpectra = [];
private long spectrumHistorySequence;
private readonly Queue<MimirSpectrumHistoryFrame> spectrumHistory = new();
private AquariumRuntimeServices runtimeServices = AquariumRuntimeServices.Empty;
public MimirRuntime(AquariumRuntimeOptions options)
: this(options, MimirRuntimeConfiguration.Load())
@ -145,6 +146,11 @@ public sealed class MimirRuntime : IAquariumRuntime
public AquariumAudioDocument Audio => audio;
public void AttachServices(AquariumRuntimeServices services)
{
runtimeServices = services;
}
private AquariumFrame CreateFrame()
{
var channelCount = Math.Max(1, lastAudioSpectra.Count);
@ -186,6 +192,7 @@ public sealed class MimirRuntime : IAquariumRuntime
var observations = new List<MimirObservation>(buffers.Count);
MimirFensalirBridgeMapper.MapWindows(buffers, windows);
MimirFensalirBridgeMapper.MapLatestObservations(buffers, observations);
CommitProducerLeases(windows);
var constraints = new List<MimirCalibrationConstraint>(synchronization.AudioSynchronizationStates.Count);
MimirFensalirBridgeMapper.MapCalibrationConstraints(synchronization.AudioSynchronizationStates, constraints);
@ -216,6 +223,24 @@ public sealed class MimirRuntime : IAquariumRuntime
return AddSpectrumFieldEvidence(frame);
}
private void CommitProducerLeases(IReadOnlyList<MimirRollingStreamWindow> windows)
{
foreach (var window in windows)
{
if (window.SourceKind != MimirStreamKind.Video ||
string.IsNullOrWhiteSpace(window.Payload.ResourceKey) ||
window.Payload.ProducerFenceValue == 0)
{
continue;
}
runtimeServices.FieldResources.CommitLeaseVersion(
window.Payload.ResourceKey,
window.SequenceId,
window.Payload.ProducerFenceValue);
}
}
private static MimirSurfaceIntent BuildSurfaceIntent(MimirRollingStreamWindow window, MimirObservation observation)
{
if (window.SourceKind == MimirStreamKind.Audio)

8
src/Mimir.Runtime/Synchronization/MimirFensalirBridgeContracts.cs
View file

@ -48,7 +48,8 @@ public readonly record struct MimirBridgePayloadView(
ulong NativeHandle,
string NativeHandleKind,
int ByteLength,
string ResourceKey = "")
string ResourceKey = "",
ulong ProducerFenceValue = 0)
{
public bool HasResource => NativeHandle != 0 || !string.IsNullOrWhiteSpace(ResourceKey);
}
@ -361,11 +362,14 @@ public static class MimirFensalirBridgeMapper
{
var nativeHandle = sample.VideoFrame?.NativeHandle ?? sample.AudioBlock?.NativeHandle ?? sample.PayloadHandle;
var nativeHandleKind = sample.VideoFrame?.NativeHandleKind ?? sample.AudioBlock?.NativeHandleKind ?? "";
var resourceKey = sample.VideoFrame?.ResourceKey ?? "";
var producerFenceValue = sample.VideoFrame?.ProducerFenceValue ?? 0;
return new MimirBridgePayloadView(
nativeHandle,
nativeHandleKind,
sample.ByteLength,
ResourceKeyForPayload(nativeHandle, nativeHandleKind));
string.IsNullOrWhiteSpace(resourceKey) ? ResourceKeyForPayload(nativeHandle, nativeHandleKind) : resourceKey,
producerFenceValue);
}
public static string ResourceKeyForPayload(MimirBridgePayloadView payload) =>

86
src/Mimir.Runtime/Synchronization/MimirFensalirTextureLeaseClient.cs Normal file
View file

@ -0,0 +1,86 @@
using Aquarium.Engine.Render;
namespace Mimir.Runtime.Synchronization;
public readonly record struct MimirFensalirTextureLeaseRequest(
string SourceId,
int Width,
int Height,
MimirVideoPixelFormat PixelFormat,
int StrideBytes,
long DeviceTimestampNs,
ulong Version,
AquariumFieldShaderAccess ProducerAccess = AquariumFieldShaderAccess.ShaderResource);
public readonly record struct MimirFensalirTextureLease(
MimirVideoFrameDescriptor Frame,
IntPtr NativeHandle,
IntPtr ProducerFenceHandle,
AquariumFieldResourceDeclaration Declaration)
{
public bool IsValid => NativeHandle != IntPtr.Zero && Frame.NativeHandle != 0 && !string.IsNullOrWhiteSpace(Frame.ResourceKey);
}
public sealed class MimirFensalirTextureLeaseClient(IAquariumFieldResourceBroker broker)
{
public bool TryLeaseTexture2D(MimirFensalirTextureLeaseRequest request, out MimirFensalirTextureLease lease)
{
lease = default;
if (string.IsNullOrWhiteSpace(request.SourceId) ||
request.Width <= 0 ||
request.Height <= 0 ||
request.StrideBytes <= 0)
{
return false;
}
var resourceKey = ResourceKeyForSource(request.SourceId);
var engineLease = broker.LeaseTexture2D(new AquariumTexture2DLeaseRequest(
resourceKey,
request.Width,
request.Height,
FormatForPixelFormat(request.PixelFormat),
request.ProducerAccess,
request.Version));
if (!engineLease.IsValid || engineLease.NativeHandle == IntPtr.Zero)
{
return false;
}
var frame = new MimirVideoFrameDescriptor(
request.Width,
request.Height,
request.PixelFormat,
request.StrideBytes,
request.DeviceTimestampNs,
NativeHandle: unchecked((ulong)engineLease.NativeHandle.ToInt64()),
NativeHandleKind: engineLease.NativeHandleKind,
ResourceKey: resourceKey,
ProducerFenceValue: 0);
lease = new MimirFensalirTextureLease(
frame,
engineLease.NativeHandle,
engineLease.ProducerFenceHandle,
engineLease.Declaration);
return lease.IsValid;
}
public bool Commit(string resourceKey, ulong version, ulong producerFenceValue) =>
broker.CommitLeaseVersion(resourceKey, version, producerFenceValue);
public static string ResourceKeyForSource(string sourceId)
{
var normalized = sourceId.Trim().ToLowerInvariant().Replace('\\', '/');
return $"mimir:resource:camera:{normalized}:texture2d";
}
private static string FormatForPixelFormat(MimirVideoPixelFormat pixelFormat) =>
pixelFormat switch
{
MimirVideoPixelFormat.Gray8 or MimirVideoPixelFormat.R8 => "R8Unorm",
MimirVideoPixelFormat.Rg8 or MimirVideoPixelFormat.LeapStereoIr => "R8G8_UNorm",
MimirVideoPixelFormat.Bgra8 => "Bgra8",
MimirVideoPixelFormat.Nv12 => "Nv12",
_ => "Rgba8Unorm",
};
}

4
src/Mimir.Runtime/Synchronization/MimirVideoFrameDescriptor.cs
View file

@ -22,4 +22,6 @@ public sealed record MimirVideoFrameDescriptor(
int StrideBytes,
long DeviceTimestampNs,
ulong NativeHandle = 0,
string NativeHandleKind = "");
string NativeHandleKind = "",
string ResourceKey = "",
ulong ProducerFenceValue = 0);

1
state/evidence.jsonl
View file

@ -173,3 +173,4 @@
{"ts": "2026-05-27T12:43:45+01:00", "type": "implementation", "status": "ok", "note": "Camera descriptors now lower through FieldEvidence instead of the retired direct GPU sensor frame bridge. GPU/native video handles declare shared Texture2D resources and camera claims; metadata-only cadence frames remain observations with no render payload, so missing pixels cannot become fake latest-window requests. --fensalir-camera-observation-smoke verifies the split and camera image claims defer until Fensalir owns a selected visual-fusion lowering."}
{"ts": "2026-05-27T12:49:27+01:00", "type": "implementation", "status": "ok", "note": "Fensalir now resolves shared Texture2D field resources by native D3D12 handle and accepts Mimir video format names (Bgra8, Gray8/R8, Rg8, Nv12, LeapStereoIr). The camera observation bridge therefore produces resource declarations that can become real engine GPU resources before a selected camera/feature lowering exists."}
{"ts": "2026-05-27T12:54:07+01:00", "type": "implementation", "status": "ok", "note": "Mimir's proof path no longer uses the direct AquariumAcousticFieldFrame builder. Sync states lower into FieldEvidence calibration constraints through MapCalibrationConstraints, and the profile/benchmark smokes now measure the claim/defer path until Fensalir owns a selected acoustic visualization/fusion lowering."}
{"ts": "2026-05-27T13:26:42+01:00", "type": "decision", "status": "ok", "note": "Fensalir-owned Texture2D leases are the preferred live camera payload authority: Mimir receives AquariumRuntimeServices, requests keyed D3D12 texture/fence leases, writes decoded frames into the engine-owned shared texture, commits producer fence values, and lowers the resource key through FieldEvidence. Imported shared texture handles remain fallback/import edges."}

15
state/map.yaml
View file

@ -305,12 +305,17 @@ accepted_design:
Texture2D resources, and metadata-only cadence frames do not create camera
surface intents or fake payload requests. The old direct
`AquariumGpuSensorFrame` builder is no longer the active proof path;
Fensalir can now resolve those shared Texture2D resources by native D3D12
Fensalir can now resolve imported shared Texture2D resources by native D3D12
handle and accepts Mimir video format names such as Bgra8, Gray8, Rg8,
Nv12, and LeapStereoIr. `Mimir.BufferSmoke
--fensalir-camera-observation-smoke` verifies the split. Camera image
claims currently defer until Fensalir has a selected visual fusion/feature
extraction lowering for camera textures. Fensalir now owns the first in-process
Nv12, and LeapStereoIr. The preferred live camera path is now
Fensalir-owned texture leasing: Mimir receives `AquariumRuntimeServices`,
asks the engine field resource broker for a keyed Texture2D lease, writes
decoded frames into the returned shared D3D12 texture, signals the returned
producer fence, and commits that fence value before the resource key reaches
shader sampling. `Mimir.BufferSmoke --fensalir-camera-observation-smoke`
verifies the split, and `--fensalir-texture-lease-smoke` verifies the
engine-owned lease path. Camera image claims currently defer until Fensalir
has a selected visual fusion/feature extraction lowering for camera textures. Fensalir now owns the first in-process
D3D12 field resource registry: declared shared structured/curve buffers
import/alias GPU-resident resources by handle, while Fensalir-owned
resources allocate GPU slots only when Fensalir is the producer.