From 2c197cb3ae740bd8cfcd1eca0ae3d2ecb1df99b8 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sat, 30 May 2026 21:54:58 +0100 Subject: [PATCH] Lower live Leap packed stereo to depth socket --- docs/implementation-plan.md | 28 ++++---- docs/perfect-machine-domain-index.md | 5 ++ notes/current-system-map.md | 20 +++--- notes/fresh-workspace-handoff.md | 13 ++-- src/Mimir.BufferSmoke/Program.cs | 71 +++++++++++++++++++ src/Mimir.Runtime/MimirRuntime.cs | 55 +++++++++++++- .../MimirFensalirFieldLowering.cs | 70 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 20 +++--- 9 files changed, 247 insertions(+), 36 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 7d32713..1293b0f 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -41,19 +41,21 @@ The research ledger is That note is not a dependency grant: CUDA implementations, TensorRT ports, and monocular model demos are provenance only. The live owner is Fensalir D3D12 compute over Mimir-declared texture resources and calibration state. -The first contract slice now exists in `MimirStereoDepthConfigurations` and -`MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame`: a calibrated -stereo pair profile references caller-declared shader-readable left/right input -textures, emits a GPU-resident compute-writable R16F disparity `SurfacePage`, -an R8 confidence texture, a Height FieldEvidence claim planned to the -`SurfacePage` backend, and an `AquariumFieldStereoDepthLowering` sidecar tying -profile, calibration, camera pair, inputs, disparity output, disparity -settings, SGM constants, and depth range together. The libSGM-provenance profile -currently publishes min disparity 0, 128 disparities, four aggregation paths, -census radius 2, P1 8, and P2 96. This is the socket for the kernel, not the -kernel. Fensalir's renderer now reports whether those lowerings are -dispatch-ready after planning/resource resolution, without writing placeholder -depth values. +The first live Leap slice now exists in `MimirStereoDepthConfigurations` and +`MimirFensalirFieldLowering`. The generic +`BuildStereoDepthCandidateFrame` still lowers caller-declared rectified +left/right textures into a GPU-resident compute-writable R16F disparity +`SurfacePage`, an R8 confidence texture, a Height FieldEvidence claim, and an +`AquariumFieldStereoDepthLowering` sidecar. The live +`BuildLeapPackedStereoDepthCandidateFrame` path now recognizes the current +`LeapStereoIr` rolling video window, reuses its declared packed R8G8 texture as +both left and right input resources, names the R/G lane observations, and emits +the same Height/depth lowering automatically from the runtime frame. The +calibration id is explicitly pending, so the output is disparity evidence, not +metric geometry truth. The libSGM-provenance profile currently publishes min +disparity 0, 128 disparities, four aggregation paths, census radius 2, P1 8, +and P2 96. Fensalir has the first packed-Leap D3D12 compute kernel installed as +a crude SAD/block-match disparity writer; it is not full SGM yet. Those claims are not inherently pixel-sized. Pixel-level resolve consumes the reservoir, but claim support is chosen from the represented field. Smooth diff --git a/docs/perfect-machine-domain-index.md b/docs/perfect-machine-domain-index.md index 0854bb7..7f77feb 100644 --- a/docs/perfect-machine-domain-index.md +++ b/docs/perfect-machine-domain-index.md @@ -85,6 +85,11 @@ letting OBS become the synchronization authority. ### Visual Evidence - Leap stereo IR: timing/depth candidate, topology-sensitive but high-value. + The live `LeapStereoIr` resource is a packed R8G8 texture; Mimir now lowers + it into a stereo-depth FieldEvidence claim by treating R/G as the left/right + observation lanes and asking Fensalir for an R16F disparity `SurfacePage`. + The first installed Fensalir kernel is a packed block matcher, not calibrated + metric depth or full SGM. - PS3 Eyes: high-rate tracking witnesses. - Kiyo/Kiyo Pro: RGB context and texture, with Kiyo Pro currently cadence/USB limited. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 750e548..314d190 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -408,20 +408,22 @@ resources, and explicit fences. CUDA, TensorRT, and monocular depth demos remain research references only. The first live contract exists: `MimirStereoDepthConfigurations` names the libSGM-provenance D3D12 SGM profile, and -`MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame` lowers a synthetic -rectified stereo result by referencing caller-declared shader-readable -left/right input textures, then emitting a GPU-resident compute-writable +`MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame` lowers caller- +declared rectified stereo input textures into a GPU-resident compute-writable disparity SurfacePage, confidence texture, Height FieldEvidence claim, and -`AquariumFieldStereoDepthLowering` sidecar. The lowering sidecar is the dispatch contract: it binds the +`AquariumFieldStereoDepthLowering` sidecar. The live +`BuildLeapPackedStereoDepthCandidateFrame` path now recognizes a live +`LeapStereoIr` rolling video window, reuses its declared packed R8G8 texture as +both left and right resource identity, names the left/right observations as +packed R/G lanes, and emits the same depth lowering automatically from the +runtime frame. The lowering sidecar is the dispatch contract: it binds the libSGM-provenance profile, calibration id, camera pair, inputs, disparity output, min disparity, disparity levels, aggregation paths, census radius, P1/P2 smoothness penalties, and depth range. The current D3D12 SGM profile publishes min disparity 0, 128 disparities, four paths, census radius 2, P1 8, -and P2 96. The kernel itself is still future work. Fensalir's D3D12 renderer now counts stereo-depth -lowerings as dispatch-ready only when the Height claim is planned and the -left/right input textures plus UAV disparity `SurfacePage` resolve through its -field resource registry; it logs that readiness separately from the absent -kernel. +and P2 96. Fensalir now has a first packed-Leap D3D12 kernel that writes R16F +disparity through a crude SAD/block-match pass. This is intentionally a small +kernel-shaped proof, not full libSGM-style SGM or calibrated metric depth yet. Fensalir must not be treated as a traditional rendering pipeline. Mimir does not ask it to "draw a thing" and hope post-processing makes the result true. diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 0fea3d5..79b665e 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -92,12 +92,15 @@ Get-Content .\state\evidence.jsonl -Tail 8 future Fensalir-owned D3D12 stereo SGM compute lane. Do not import CUDA or TensorRT as live depth authority. The first code contract is in `MimirStereoDepthConfigurations` and - `MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame`; run + `MimirFensalirFieldLowering`. Run `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --stereo-depth-contract-smoke` - to verify caller-declared synthetic rectified left/right input textures, - compute-writable R16F disparity `SurfacePage`, confidence texture, Height - claim, and `AquariumFieldStereoDepthLowering` dispatch metadata. This proves - the lane shape, not the SGM kernel. + to verify caller-declared rectified left/right input textures, or + `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --leap-packed-stereo-depth-smoke` + to verify live `LeapStereoIr` packed R8G8 lowering. The Leap path reuses the + same declared texture as left/right resource identity and treats R/G as the + packed observation lanes. Fensalir now has a first packed-Leap D3D12 + SAD/block-match disparity writer, not full libSGM-style SGM or calibrated + metric depth. - Fensalir also owns the EVE-facing dashboard pixels through `Global\MimirFensalirProgramTexture`. `src/Mimir.EveRelay` opens that shared texture, encodes H.264 Annex-B with NVENC by default, and serves EveCanvas diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index dc4e17e..6dcfcc8 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -149,6 +149,11 @@ if (args.Any(arg => string.Equals(arg, "--stereo-depth-contract-smoke", StringCo return RunStereoDepthContractSmoke(); } +if (args.Any(arg => string.Equals(arg, "--leap-packed-stereo-depth-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunLeapPackedStereoDepthSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--ks-camera-driver-smoke", StringComparison.OrdinalIgnoreCase))) { return RunKsCameraDriverSmoke(args); @@ -2396,6 +2401,72 @@ static int RunStereoDepthContractSmoke() : 1; } +static int RunLeapPackedStereoDepthSmoke() +{ + const string sourceId = "leap-stereo-ir"; + var frameDescriptor = new MimirVideoFrameDescriptor( + Width: 640, + Height: 240, + PixelFormat: MimirVideoPixelFormat.LeapStereoIr, + StrideBytes: 1280, + DeviceTimestampNs: 1_000_000_000L, + NativeHandle: 0xCAFE, + NativeHandleKind: "fensalir-owned-d3d12-texture2d", + ResourceKey: MimirFensalirTextureLeaseClient.ResourceKeyForSource(sourceId), + ProducerFenceHandle: 0xD00D, + ProducerFenceValue: 7); + var sample = new MimirStreamSample( + sourceId, + MimirStreamKind.Video, + MimirStreamOrigin.LocalDevice, + TimestampNs: frameDescriptor.DeviceTimestampNs, + ArrivalNs: frameDescriptor.DeviceTimestampNs, + Sequence: 7, + PayloadHandle: frameDescriptor.NativeHandle, + ByteLength: 0, + VideoFrame: frameDescriptor); + var buffer = new MimirRollingStreamBuffer( + new MimirStreamDescriptor(sourceId, MimirStreamKind.Video, MimirStreamOrigin.LocalDevice), + TimeSpan.FromSeconds(5)); + buffer.Append(sample); + + var lowerer = new MimirFensalirFieldLowering(); + var cameraFrame = lowerer.BuildCameraObservationFrame([buffer]); + var windows = new List(); + MimirFensalirBridgeMapper.MapWindows([buffer], windows); + var stereoFrame = lowerer.BuildLeapPackedStereoDepthCandidateFrame(windows, cameraFrame.Resources); + var validation = AquariumFieldEvidenceValidator.Validate(stereoFrame); + var plan = AquariumFieldLoweringPlanner.Plan(stereoFrame); + var lowering = stereoFrame.StereoDepthLowerings.FirstOrDefault(); + var packedResource = stereoFrame.Resources.FirstOrDefault(resource => + string.Equals(resource.ResourceKey, frameDescriptor.ResourceKey, StringComparison.Ordinal)); + var disparity = stereoFrame.Resources.FirstOrDefault(resource => + string.Equals(resource.ResourceKey, "mimir:resource:stereo-depth:leap-stereo-ir:disparity-r16f", StringComparison.Ordinal)); + + Console.WriteLine( + $"leap-packed-stereo-depth resources={stereoFrame.Resources.Count} claims={stereoFrame.Claims.Count} lowerings={stereoFrame.StereoDepthLowerings.Count} planned={plan.Packets.Count} deferred={plan.DeferredRequests.Count} errors={validation.HasErrors}"); + Console.WriteLine( + $"leap-packed-stereo-depth-input packed={packedResource.ResourceKey} format={packedResource.Format} left={lowering.LeftResourceKey} right={lowering.RightResourceKey} calibration={lowering.CalibrationKey}"); + Console.WriteLine( + $"leap-packed-stereo-depth-output disparity={disparity.ResourceKey} access={disparity.Access} format={disparity.Format} minDisparity={lowering.MinDisparity} disparities={lowering.DisparityLevels} paths={lowering.AggregationPathCount}"); + + return !validation.HasErrors && + stereoFrame.Resources.Count == 3 && + stereoFrame.Claims.Count == 1 && + stereoFrame.StereoDepthLowerings.Count == 1 && + packedResource.Format == "LeapStereoIr" && + lowering.LeftResourceKey == frameDescriptor.ResourceKey && + lowering.RightResourceKey == frameDescriptor.ResourceKey && + lowering.CalibrationKey == "leapuvc-packed-stereo-ir-calibration-pending" && + disparity.Kind == AquariumFieldResourceKind.SurfacePage && + disparity.Access == AquariumFieldShaderAccess.UnorderedAccess && + plan.Packets.Count == 1 && + plan.Packets[0].Backend == AquariumFieldBackendKind.SurfacePage && + plan.DeferredRequests.Count == 0 + ? 0 + : 1; +} + static int RunPerfectMachineProfileSmoke() { var profiles = MimirPerfectMachineProfiles.All; diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index e841f8d..bfbab0a 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -321,7 +321,8 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec } var frame = fieldLowering.BuildFieldEvidenceFrame(windows, observations, constraints, intents); - return AddSpectrumFieldEvidence(frame); + frame = AddSpectrumFieldEvidence(frame); + return AddLeapStereoDepthFieldEvidence(frame, windows); } private void CommitProducerLeases(IReadOnlyList windows) @@ -368,6 +369,58 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec Purpose: MimirSurfaceIntentPurpose.Production); } + private AquariumFieldEvidenceFrame AddLeapStereoDepthFieldEvidence( + AquariumFieldEvidenceFrame frame, + IReadOnlyList windows) + { + var stereoFrame = fieldLowering.BuildLeapPackedStereoDepthCandidateFrame(windows, frame.Resources); + if (!stereoFrame.HasInput) + { + return frame; + } + + return new AquariumFieldEvidenceFrame + { + Domains = [.. frame.Domains, .. stereoFrame.Domains], + Claims = [.. frame.Claims, .. stereoFrame.Claims], + Candidates = [.. frame.Candidates, .. stereoFrame.Candidates], + BackendPackets = [.. frame.BackendPackets, .. stereoFrame.BackendPackets], + Resources = MergeResources(frame.Resources, stereoFrame.Resources), + ResourceUploads = [.. frame.ResourceUploads, .. stereoFrame.ResourceUploads], + TubeSplineLowerings = [.. frame.TubeSplineLowerings, .. stereoFrame.TubeSplineLowerings], + StereoDepthLowerings = [.. frame.StereoDepthLowerings, .. stereoFrame.StereoDepthLowerings], + AccumulationWindowSeconds = frame.AccumulationWindowSeconds, + PresentationDelaySeconds = frame.PresentationDelaySeconds, + }; + } + + private static IReadOnlyList MergeResources( + IReadOnlyList first, + IReadOnlyList second) + { + if (first.Count == 0) + { + return second; + } + + if (second.Count == 0) + { + return first; + } + + var resources = new List(first.Count + second.Count); + var seen = new HashSet(StringComparer.Ordinal); + foreach (var resource in first.Concat(second)) + { + if (resource.HasIdentity && seen.Add(resource.ResourceKey)) + { + resources.Add(resource); + } + } + + return resources; + } + private AquariumFieldEvidenceFrame AddSpectrumFieldEvidence(AquariumFieldEvidenceFrame frame) { if (lastAudioSpectra.Count == 0) diff --git a/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs b/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs index 546839b..c41736d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs +++ b/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs @@ -180,6 +180,63 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt }; } + public AquariumFieldEvidenceFrame BuildLeapPackedStereoDepthCandidateFrame( + IEnumerable windows, + IEnumerable inputResources) + { + var resourcesByKey = inputResources + .Where(static resource => resource.HasIdentity) + .GroupBy(static resource => resource.ResourceKey, StringComparer.Ordinal) + .ToDictionary(static group => group.Key, static group => group.First(), StringComparer.Ordinal); + var leapWindow = windows + .Where(static window => + window.Status == MimirBridgeWindowStatus.Live && + window.SourceKind == MimirStreamKind.Video && + window.SampleDescriptor.PixelFormat == MimirVideoPixelFormat.LeapStereoIr && + window.Payload.HasResource && + !string.IsNullOrWhiteSpace(window.Payload.ResourceKey)) + .OrderByDescending(static window => window.EdgeNs) + .FirstOrDefault(); + if (string.IsNullOrWhiteSpace(leapWindow.WindowId)) + { + return AquariumFieldEvidenceFrame.Empty; + } + + var packedResourceKey = MimirFensalirBridgeMapper.ResourceKeyForPayload(leapWindow.Payload); + if (!resourcesByKey.TryGetValue(packedResourceKey, out var packedInput)) + { + return AquariumFieldEvidenceFrame.Empty; + } + + var profile = MimirStereoDepthConfigurations.D3D12SgmLibSgmProvenance; + var sourceSegment = SanitizeResourceSegment(leapWindow.StreamId); + var candidate = new MimirStereoDepthFieldCandidate( + CandidateKey: $"live-{sourceSegment}-{leapWindow.SequenceId}", + CalibrationId: "leapuvc-packed-stereo-ir-calibration-pending", + CameraPairId: leapWindow.StreamId, + ProducerKey: profile.Id, + ProfileId: profile.Id, + LeftObservationKey: $"{leapWindow.WindowId}:left-packed-r", + RightObservationKey: $"{leapWindow.WindowId}:right-packed-g", + LeftResourceKey: packedResourceKey, + RightResourceKey: packedResourceKey, + DisparityResourceKey: $"mimir:resource:stereo-depth:{sourceSegment}:disparity-r16f", + ConfidenceResourceKey: $"mimir:resource:stereo-depth:{sourceSegment}:confidence-r8", + Width: Math.Max(1, leapWindow.SampleDescriptor.Width), + Height: Math.Max(1, leapWindow.SampleDescriptor.Height), + MinDisparity: profile.MinDisparity, + DisparityLevels: profile.DisparityLevels, + AggregationPathCount: profile.AggregationPathCount, + CensusRadius: profile.CensusRadius, + SmoothnessPenaltySmall: profile.SmoothnessPenaltySmall, + SmoothnessPenaltyLarge: profile.SmoothnessPenaltyLarge, + MinDepthMeters: 0.20, + MaxDepthMeters: 4.0, + Confidence: 0.50, + ObservedTimeNs: leapWindow.EdgeNs); + return BuildStereoDepthCandidateFrame([candidate], [packedInput]); + } + public AquariumFieldEvidenceFrame BuildStereoDepthCandidateFrame( IEnumerable depthCandidates, IEnumerable? inputResources = null) @@ -951,6 +1008,19 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt private static string DomainKeyForSurfaceIntent(string intentKey) => $"mimir:intent:{intentKey}"; + private static string SanitizeResourceSegment(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "unknown"; + } + + var chars = value.Trim().ToLowerInvariant() + .Select(static character => char.IsLetterOrDigit(character) ? character : '-') + .ToArray(); + return chars.Length == 0 ? "unknown" : new string(chars); + } + private static float Clamp01(float value) => Math.Clamp(value, 0.0f, 1.0f); private static uint StableSeed(string key) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index e2823db..5f61cb1 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -236,3 +236,4 @@ {"ts":"2026-05-30T00:39:40+01:00","type":"decision","status":"ok","note":"Stereo-depth lowerer does not fabricate shared input handles. BuildStereoDepthCandidateFrame accepts caller-declared input resources, then emits the UAV disparity SurfacePage, confidence texture, Height claim, and AquariumFieldStereoDepthLowering sidecar that references those real texture resource keys. The BufferSmoke proof supplies synthetic input declarations only for contract validation."} {"ts":"2026-05-30T00:44:38+01:00","type":"decision","status":"ok","note":"Fensalir now owns stereo-depth dispatch readiness accounting before the SGM kernel exists: it only counts a lowering ready when the Height claim is planned and left/right textures plus the UAV disparity SurfacePage resolve. Mimir remains responsible for real input texture resource identity; no placeholder depth output is written."} {"ts":"2026-05-30T00:48:04+01:00","type":"implementation","status":"ok","note":"Stereo-depth profile/candidate/lowering now carries SGM constants through Mimir into the Fensalir sidecar: min disparity 0, 128 disparities, four aggregation paths, census radius 2, P1 8, and P2 96 for the libSGM-provenance D3D12 profile. Verified Mimir build, --stereo-depth-contract-smoke, and --perfect-machine-profile-smoke."} +{"ts":"2026-05-30T21:58:00+01:00","type":"implementation","status":"ok","note":"Mimir now lowers live Leap packed stereo IR into the stereo-depth FieldEvidence lane. MimirRuntime merges a Leap depth frame when a live LeapStereoIr rolling window has a declared GPU texture resource; BuildLeapPackedStereoDepthCandidateFrame treats the packed R8G8 texture as left/right resource identity with R/G observation lanes, emits the pending-calibration Height claim, UAV R16F disparity SurfacePage, R8 confidence texture, and AquariumFieldStereoDepthLowering sidecar. Verified with Mimir build, --leap-packed-stereo-depth-smoke, and --stereo-depth-contract-smoke."} diff --git a/state/map.yaml b/state/map.yaml index 702ea7d..c287b11 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -61,16 +61,20 @@ invariants: `MimirStereoDepthConfigurations` names the libSGM-provenance D3D12 SGM profile, and `MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame` references caller-declared shader-readable left/right Texture2D resources, - then lowers synthetic rectified stereo depth into a compute-writable R16F - disparity SurfacePage, R8 confidence Texture2D, a Height FieldEvidence claim, and an + then lowers rectified stereo depth into a compute-writable R16F disparity + SurfacePage, R8 confidence Texture2D, a Height FieldEvidence claim, and an `AquariumFieldStereoDepthLowering` sidecar tying profile, calibration, camera pair, inputs, disparity output, SGM constants, and depth range - together. The current libSGM-provenance D3D12 SGM profile publishes min - disparity 0, 128 disparities, four aggregation paths, census radius 2, P1 - 8, and P2 96. The actual - SGM HLSL/D3D12 kernel is still future work. Fensalir now reports - stereo-depth dispatch readiness only after the planned claim and all input/ - output resources resolve, without writing placeholder depth. + together. `BuildLeapPackedStereoDepthCandidateFrame` is the live Leap cut: + it recognizes a live `LeapStereoIr` rolling video window, reuses the + declared packed R8G8 texture as both left and right resource identity, + names the left/right observations as packed R/G lanes, and emits the same + depth lowering from the runtime frame. The current libSGM-provenance D3D12 + SGM profile publishes min disparity 0, 128 disparities, four aggregation + paths, census radius 2, P1 8, and P2 96. Fensalir now has a first + packed-Leap D3D12 SAD/block-match kernel that writes R16F disparity through + the typed SurfacePage socket; full SGM and calibrated metric depth remain + future work. - Fensalir packet payload handles are not authority. Live payload references must resolve through declared typed GPU/native resources with shape, residency, shader access, validity, version, and handle metadata before