diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index f80d384..2b40351 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -45,7 +45,12 @@ Payload handles are now demoted to names inside a resource authority. Mimir declares live native/GPU payloads as typed Fensalir resources with shape, residency, shader access, validity, version, and native handle metadata; claims and lowering requests reference those resource keys. A string handle without a -matching resource declaration is not payload truth. +matching resource declaration is not payload truth. For live camera images, the +preferred authority is Fensalir-owned texture leasing: Mimir asks the engine +broker for a keyed `Texture2D` lease, writes decoded frames into the returned +shared D3D12 texture, signals the returned fence, and commits that fence value +before the resource key is sampled by shader lowerings. Foreign shared handles +remain import edges only. The old script stack is gone. Do not add a compatibility edge unless it protects a named invariant that the native runtime cannot protect yet. @@ -176,14 +181,19 @@ a named invariant that the native runtime cannot protect yet. `TubeField` packet with no deferred requests. - `MimirFensalirFieldLowering.BuildCameraObservationFrame` is the current camera bridge proof. It lowers latest video buffers into FieldEvidence - observation claims, declares GPU/native video payloads as shared `Texture2D` - resources, and creates camera surface intent only when a resource exists. + observation claims, declares engine-owned or imported video payloads as + `Texture2D` resources, and creates camera surface intent only when a resource + exists. `MimirFensalirTextureLeaseClient` is the producer-side API for asking + Fensalir for a keyed D3D12 texture/fence lease before decoded camera frames + are written. `MimirRuntime` accepts `AquariumRuntimeServices` and commits + producer fence values for leased video payloads before Fensalir samples them. Metadata-only cadence frames remain observations with empty payload handles; they do not become fake render requests. The old direct `AquariumGpuSensorFrame` builder has been removed from Mimir's proof path. - Fensalir can now resolve those shared `Texture2D` resources by native D3D12 - handle and accepts Mimir video format names. `Mimir.BufferSmoke - --fensalir-camera-observation-smoke` verifies the split. + Fensalir can still resolve imported shared `Texture2D` resources by native + D3D12 handle and accepts Mimir video format names. + `Mimir.BufferSmoke --fensalir-camera-observation-smoke` verifies the split, + and `--fensalir-texture-lease-smoke` verifies the engine-owned lease path. - Mimir's active proof path no longer uses the direct `AquariumAcousticFieldFrame` builder either. Sync states lower into FieldEvidence calibration constraints through `MapCalibrationConstraints`, diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 80c40ce..6e3fa9e 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -326,14 +326,17 @@ shader lowering. Mimir-declared rendering buffers are Fensalir GPU resources in the same runtime; native/shared handle metadata is an import edge, not a separate authority regime. Rendering-relevant buffers move to GPU residency as early as possible and stay there; CPU readback/tessellation is diagnostic only. -Camera descriptors now enter this bridge too: latest video buffers lower into -FieldEvidence camera observations, GPU/native handles declare shared -`Texture2D` resources, and metadata-only cadence frames do not create camera -surface intents or fake payload requests. The old direct `AquariumGpuSensorFrame` -bridge proof has been removed from Mimir's active proof path. Fensalir can now -resolve those shared `Texture2D` resources by native D3D12 handle and accepts -Mimir video format names. Camera image claims currently defer until Fensalir -owns a selected visual-fusion lowering for camera textures. +Camera descriptors now enter this bridge too. The preferred hot path is +Fensalir-owned texture leasing: the runtime receives `AquariumRuntimeServices`, +asks Fensalir's field resource broker for a keyed D3D12 `Texture2D` lease, the +camera/decode producer writes that shared texture and signals the producer +fence, then Mimir carries the same resource key through FieldEvidence. Fensalir +waits on committed producer fence values before resolving shader reads. Shared +foreign texture handles remain an import edge, not the primary camera authority. +Metadata-only cadence frames do not create camera surface intents or fake +payload requests. The old direct `AquariumGpuSensorFrame` bridge proof has been +removed from Mimir's active proof path. Camera image claims currently defer +until Fensalir owns a selected visual-fusion lowering for camera textures. The current smokes prove planning, not visible packet rendering: `--fensalir-field-evidence-smoke` now produces one planned resource-backed `TubeField` packet from Mimir's spectrum intent, and diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 043efa7..a62fc76 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -43,6 +43,11 @@ Get-Content .\state\evidence.jsonl -Tail 8 - `native/reservoir` owns the lower shared-edge typed-handle invariant for Fensalir/Faust binding work. - Fensalir owns production GPU fusion, UI, and Spout2 publication. +- Live camera image buffers should use Fensalir-owned texture leases when they + are rendering inputs: Mimir asks the engine broker for a keyed D3D12 + Texture2D/fence lease, writes decoded frames into that texture, commits the + producer fence value, and lowers the same resource key through FieldEvidence. + Foreign shared texture handles are import edges, not the primary hot path. - The current Mimir/Fensalir rendering teardown map is [[docs/fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild Migration]]. Mimir's production visual output path must publish typed diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 9856d08..659ac92 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -108,6 +108,11 @@ if (args.Any(arg => string.Equals(arg, "--fensalir-camera-observation-smoke", St return RunFensalirCameraObservationSmoke(); } +if (args.Any(arg => string.Equals(arg, "--fensalir-texture-lease-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunFensalirTextureLeaseSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--mimir-spectrum-upload-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMimirSpectrumUploadSmoke(); @@ -718,6 +723,78 @@ static int RunFensalirCameraObservationSmoke() : 1; } +static int RunFensalirTextureLeaseSmoke() +{ + var broker = new FakeFieldResourceBroker(); + var client = new MimirFensalirTextureLeaseClient(broker); + var requested = new MimirFensalirTextureLeaseRequest( + "kiyo-pro-rgb", + 1920, + 1080, + MimirVideoPixelFormat.Bgra8, + 1920 * 4, + DeviceTimestampNs: 1_000_000_000L, + Version: 17); + + if (!client.TryLeaseTexture2D(requested, out var lease)) + { + Console.Error.WriteLine("fensalir-texture-lease-smoke failed to lease texture"); + return 1; + } + + var source = new MimirNativeIngestStreamSource( + new MimirStreamDescriptor("kiyo-pro-rgb", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice)); + using (var runtime = new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: null), + new MimirSynchronizationSettings(), + [source])) + { + runtime.AttachServices(new AquariumRuntimeServices(broker)); + runtime.OnSceneReady(); + source.PushVideoFrame(lease.Frame with { ProducerFenceValue = 41 }, 1_000_000_010L); + runtime.Update(0.016f, new InputState()); + _ = runtime.Frame; + } + + var buffer = new MimirRollingStreamBuffer( + new MimirStreamDescriptor("kiyo-pro-rgb", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice), + TimeSpan.FromSeconds(5)); + buffer.Append(new MimirStreamSample( + "kiyo-pro-rgb", + MimirStreamKind.Video, + MimirStreamOrigin.LocalDevice, + TimestampNs: 1_000_000_000L, + ArrivalNs: 1_000_000_010L, + Sequence: 17, + PayloadHandle: 0, + ByteLength: 1920 * 1080 * 4, + Data: default, + VideoFrame: lease.Frame with { ProducerFenceValue = 41 })); + + var frame = new MimirFensalirFieldLowering().BuildCameraObservationFrame([buffer]); + var validation = AquariumFieldEvidenceValidator.Validate(frame); + var expectedResourceKey = MimirFensalirTextureLeaseClient.ResourceKeyForSource("kiyo-pro-rgb"); + + Console.WriteLine( + $"fensalir-texture-lease-smoke leased={lease.IsValid} committed={broker.CommittedFenceValue > 0} resources={frame.Resources.Count} claims={frame.Claims.Count} errors={validation.HasErrors}"); + + return lease.IsValid && + broker.CommittedResourceKey == expectedResourceKey && + broker.CommittedVersion == 0 && + broker.CommittedFenceValue == 41 && + lease.Frame.ResourceKey == expectedResourceKey && + lease.Frame.NativeHandle == 0xCAFE && + lease.Frame.NativeHandleKind == "fensalir-owned-d3d12-texture2d" && + frame.Resources.Count == 1 && + frame.Resources[0].ResourceKey == expectedResourceKey && + frame.Resources[0].NativeHandle == new IntPtr(0xCAFE) && + frame.Resources[0].NativeHandleKind == "fensalir-owned-d3d12-texture2d" && + frame.Claims.Any(claim => claim.PayloadHandle == expectedResourceKey) && + !validation.HasErrors + ? 0 + : 1; +} + static int RunMimirSpectrumUploadSmoke() { var previousSyntheticPreview = Environment.GetEnvironmentVariable("MIMIR_SYNTHETIC_SPECTRUM_PREVIEW"); @@ -5948,3 +6025,46 @@ public sealed record ComplexContourChannelModelEvaluationCase( double DeltaAbsolutePredictionErrorMicroseconds, double DeltaMeanAbsoluteErrorSamples, double DeltaMeanAbsolutePhaseErrorRadians); + +sealed class FakeFieldResourceBroker : IAquariumFieldResourceBroker +{ + public string CommittedResourceKey { get; private set; } = ""; + + public ulong CommittedVersion { get; private set; } + + public ulong CommittedFenceValue { get; private set; } + + public AquariumFieldResourceLease LeaseTexture2D(AquariumTexture2DLeaseRequest request) + { + var declaration = new AquariumFieldResourceDeclaration( + request.ResourceKey, + AquariumFieldResourceKind.Texture2D, + AquariumFieldResourceResidency.SharedGpu, + AquariumFieldShaderAccess.ShaderResource, + request.Format, + request.Width, + request.Height, + DepthOrCount: 1, + StrideBytes: AquariumFieldResourceDeclaration.FormatStrideBytes(request.Format), + request.ValidFromNs, + request.ValidUntilNs, + request.Version, + new IntPtr(0xCAFE), + "fensalir-owned-d3d12-texture2d"); + return new AquariumFieldResourceLease( + declaration, + new IntPtr(0xCAFE), + "fensalir-owned-d3d12-texture2d", + new IntPtr(0xF00D), + request.Version, + true); + } + + public bool CommitLeaseVersion(string resourceKey, ulong version, ulong producerFenceValue) + { + CommittedResourceKey = resourceKey; + CommittedVersion = version; + CommittedFenceValue = producerFenceValue; + return true; + } +} diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index af29eee..7b8c9fb 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -11,7 +11,7 @@ using System.Text; namespace Mimir.Runtime; -public sealed class MimirRuntime : IAquariumRuntime +public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesReceiver { private const float DefaultAudioSyncUpdateIntervalSeconds = 0.1f; private const double HybridPassiveConfidenceThreshold = 0.12; @@ -73,6 +73,7 @@ public sealed class MimirRuntime : IAquariumRuntime private IReadOnlyList lastAudioSpectra = []; private long spectrumHistorySequence; private readonly Queue spectrumHistory = new(); + private AquariumRuntimeServices runtimeServices = AquariumRuntimeServices.Empty; public MimirRuntime(AquariumRuntimeOptions options) : this(options, MimirRuntimeConfiguration.Load()) @@ -145,6 +146,11 @@ public sealed class MimirRuntime : IAquariumRuntime public AquariumAudioDocument Audio => audio; + public void AttachServices(AquariumRuntimeServices services) + { + runtimeServices = services; + } + private AquariumFrame CreateFrame() { var channelCount = Math.Max(1, lastAudioSpectra.Count); @@ -186,6 +192,7 @@ public sealed class MimirRuntime : IAquariumRuntime var observations = new List(buffers.Count); MimirFensalirBridgeMapper.MapWindows(buffers, windows); MimirFensalirBridgeMapper.MapLatestObservations(buffers, observations); + CommitProducerLeases(windows); var constraints = new List(synchronization.AudioSynchronizationStates.Count); MimirFensalirBridgeMapper.MapCalibrationConstraints(synchronization.AudioSynchronizationStates, constraints); @@ -216,6 +223,24 @@ public sealed class MimirRuntime : IAquariumRuntime return AddSpectrumFieldEvidence(frame); } + private void CommitProducerLeases(IReadOnlyList windows) + { + foreach (var window in windows) + { + if (window.SourceKind != MimirStreamKind.Video || + string.IsNullOrWhiteSpace(window.Payload.ResourceKey) || + window.Payload.ProducerFenceValue == 0) + { + continue; + } + + runtimeServices.FieldResources.CommitLeaseVersion( + window.Payload.ResourceKey, + window.SequenceId, + window.Payload.ProducerFenceValue); + } + } + private static MimirSurfaceIntent BuildSurfaceIntent(MimirRollingStreamWindow window, MimirObservation observation) { if (window.SourceKind == MimirStreamKind.Audio) diff --git a/src/Mimir.Runtime/Synchronization/MimirFensalirBridgeContracts.cs b/src/Mimir.Runtime/Synchronization/MimirFensalirBridgeContracts.cs index 34c0819..e194980 100644 --- a/src/Mimir.Runtime/Synchronization/MimirFensalirBridgeContracts.cs +++ b/src/Mimir.Runtime/Synchronization/MimirFensalirBridgeContracts.cs @@ -48,7 +48,8 @@ public readonly record struct MimirBridgePayloadView( ulong NativeHandle, string NativeHandleKind, int ByteLength, - string ResourceKey = "") + string ResourceKey = "", + ulong ProducerFenceValue = 0) { public bool HasResource => NativeHandle != 0 || !string.IsNullOrWhiteSpace(ResourceKey); } @@ -361,11 +362,14 @@ public static class MimirFensalirBridgeMapper { var nativeHandle = sample.VideoFrame?.NativeHandle ?? sample.AudioBlock?.NativeHandle ?? sample.PayloadHandle; var nativeHandleKind = sample.VideoFrame?.NativeHandleKind ?? sample.AudioBlock?.NativeHandleKind ?? ""; + var resourceKey = sample.VideoFrame?.ResourceKey ?? ""; + var producerFenceValue = sample.VideoFrame?.ProducerFenceValue ?? 0; return new MimirBridgePayloadView( nativeHandle, nativeHandleKind, sample.ByteLength, - ResourceKeyForPayload(nativeHandle, nativeHandleKind)); + string.IsNullOrWhiteSpace(resourceKey) ? ResourceKeyForPayload(nativeHandle, nativeHandleKind) : resourceKey, + producerFenceValue); } public static string ResourceKeyForPayload(MimirBridgePayloadView payload) => diff --git a/src/Mimir.Runtime/Synchronization/MimirFensalirTextureLeaseClient.cs b/src/Mimir.Runtime/Synchronization/MimirFensalirTextureLeaseClient.cs new file mode 100644 index 0000000..1105511 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirFensalirTextureLeaseClient.cs @@ -0,0 +1,86 @@ +using Aquarium.Engine.Render; + +namespace Mimir.Runtime.Synchronization; + +public readonly record struct MimirFensalirTextureLeaseRequest( + string SourceId, + int Width, + int Height, + MimirVideoPixelFormat PixelFormat, + int StrideBytes, + long DeviceTimestampNs, + ulong Version, + AquariumFieldShaderAccess ProducerAccess = AquariumFieldShaderAccess.ShaderResource); + +public readonly record struct MimirFensalirTextureLease( + MimirVideoFrameDescriptor Frame, + IntPtr NativeHandle, + IntPtr ProducerFenceHandle, + AquariumFieldResourceDeclaration Declaration) +{ + public bool IsValid => NativeHandle != IntPtr.Zero && Frame.NativeHandle != 0 && !string.IsNullOrWhiteSpace(Frame.ResourceKey); +} + +public sealed class MimirFensalirTextureLeaseClient(IAquariumFieldResourceBroker broker) +{ + public bool TryLeaseTexture2D(MimirFensalirTextureLeaseRequest request, out MimirFensalirTextureLease lease) + { + lease = default; + if (string.IsNullOrWhiteSpace(request.SourceId) || + request.Width <= 0 || + request.Height <= 0 || + request.StrideBytes <= 0) + { + return false; + } + + var resourceKey = ResourceKeyForSource(request.SourceId); + var engineLease = broker.LeaseTexture2D(new AquariumTexture2DLeaseRequest( + resourceKey, + request.Width, + request.Height, + FormatForPixelFormat(request.PixelFormat), + request.ProducerAccess, + request.Version)); + if (!engineLease.IsValid || engineLease.NativeHandle == IntPtr.Zero) + { + return false; + } + + var frame = new MimirVideoFrameDescriptor( + request.Width, + request.Height, + request.PixelFormat, + request.StrideBytes, + request.DeviceTimestampNs, + NativeHandle: unchecked((ulong)engineLease.NativeHandle.ToInt64()), + NativeHandleKind: engineLease.NativeHandleKind, + ResourceKey: resourceKey, + ProducerFenceValue: 0); + lease = new MimirFensalirTextureLease( + frame, + engineLease.NativeHandle, + engineLease.ProducerFenceHandle, + engineLease.Declaration); + return lease.IsValid; + } + + public bool Commit(string resourceKey, ulong version, ulong producerFenceValue) => + broker.CommitLeaseVersion(resourceKey, version, producerFenceValue); + + public static string ResourceKeyForSource(string sourceId) + { + var normalized = sourceId.Trim().ToLowerInvariant().Replace('\\', '/'); + return $"mimir:resource:camera:{normalized}:texture2d"; + } + + private static string FormatForPixelFormat(MimirVideoPixelFormat pixelFormat) => + pixelFormat switch + { + MimirVideoPixelFormat.Gray8 or MimirVideoPixelFormat.R8 => "R8Unorm", + MimirVideoPixelFormat.Rg8 or MimirVideoPixelFormat.LeapStereoIr => "R8G8_UNorm", + MimirVideoPixelFormat.Bgra8 => "Bgra8", + MimirVideoPixelFormat.Nv12 => "Nv12", + _ => "Rgba8Unorm", + }; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirVideoFrameDescriptor.cs b/src/Mimir.Runtime/Synchronization/MimirVideoFrameDescriptor.cs index 0726dfb..331496e 100644 --- a/src/Mimir.Runtime/Synchronization/MimirVideoFrameDescriptor.cs +++ b/src/Mimir.Runtime/Synchronization/MimirVideoFrameDescriptor.cs @@ -22,4 +22,6 @@ public sealed record MimirVideoFrameDescriptor( int StrideBytes, long DeviceTimestampNs, ulong NativeHandle = 0, - string NativeHandleKind = ""); + string NativeHandleKind = "", + string ResourceKey = "", + ulong ProducerFenceValue = 0); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 6a79e09..97467f0 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -173,3 +173,4 @@ {"ts": "2026-05-27T12:43:45+01:00", "type": "implementation", "status": "ok", "note": "Camera descriptors now lower through FieldEvidence instead of the retired direct GPU sensor frame bridge. GPU/native video handles declare shared Texture2D resources and camera claims; metadata-only cadence frames remain observations with no render payload, so missing pixels cannot become fake latest-window requests. --fensalir-camera-observation-smoke verifies the split and camera image claims defer until Fensalir owns a selected visual-fusion lowering."} {"ts": "2026-05-27T12:49:27+01:00", "type": "implementation", "status": "ok", "note": "Fensalir now resolves shared Texture2D field resources by native D3D12 handle and accepts Mimir video format names (Bgra8, Gray8/R8, Rg8, Nv12, LeapStereoIr). The camera observation bridge therefore produces resource declarations that can become real engine GPU resources before a selected camera/feature lowering exists."} {"ts": "2026-05-27T12:54:07+01:00", "type": "implementation", "status": "ok", "note": "Mimir's proof path no longer uses the direct AquariumAcousticFieldFrame builder. Sync states lower into FieldEvidence calibration constraints through MapCalibrationConstraints, and the profile/benchmark smokes now measure the claim/defer path until Fensalir owns a selected acoustic visualization/fusion lowering."} +{"ts": "2026-05-27T13:26:42+01:00", "type": "decision", "status": "ok", "note": "Fensalir-owned Texture2D leases are the preferred live camera payload authority: Mimir receives AquariumRuntimeServices, requests keyed D3D12 texture/fence leases, writes decoded frames into the engine-owned shared texture, commits producer fence values, and lowers the resource key through FieldEvidence. Imported shared texture handles remain fallback/import edges."} diff --git a/state/map.yaml b/state/map.yaml index c4310ad..55ce88a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -305,12 +305,17 @@ accepted_design: Texture2D resources, and metadata-only cadence frames do not create camera surface intents or fake payload requests. The old direct `AquariumGpuSensorFrame` builder is no longer the active proof path; - Fensalir can now resolve those shared Texture2D resources by native D3D12 + Fensalir can now resolve imported shared Texture2D resources by native D3D12 handle and accepts Mimir video format names such as Bgra8, Gray8, Rg8, - Nv12, and LeapStereoIr. `Mimir.BufferSmoke - --fensalir-camera-observation-smoke` verifies the split. Camera image - claims currently defer until Fensalir has a selected visual fusion/feature - extraction lowering for camera textures. Fensalir now owns the first in-process + Nv12, and LeapStereoIr. The preferred live camera path is now + Fensalir-owned texture leasing: Mimir receives `AquariumRuntimeServices`, + asks the engine field resource broker for a keyed Texture2D lease, writes + decoded frames into the returned shared D3D12 texture, signals the returned + producer fence, and commits that fence value before the resource key reaches + shader sampling. `Mimir.BufferSmoke --fensalir-camera-observation-smoke` + verifies the split, and `--fensalir-texture-lease-smoke` verifies the + engine-owned lease path. Camera image claims currently defer until Fensalir + has a selected visual fusion/feature extraction lowering for camera textures. Fensalir now owns the first in-process D3D12 field resource registry: declared shared structured/curve buffers import/alias GPU-resident resources by handle, while Fensalir-owned resources allocate GPU slots only when Fensalir is the producer.