From 4fadcfff14cd836ddf32dff4fc269a67c3e025bd Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sat, 30 May 2026 22:54:59 +0100 Subject: [PATCH] Add Leap point cloud field socket --- docs/implementation-plan.md | 16 +- notes/current-system-map.md | 12 + notes/fresh-workspace-handoff.md | 13 +- src/Mimir.BufferSmoke/Program.cs | 94 ++++++ src/Mimir.Runtime/MimirRuntime.cs | 21 +- .../MimirFensalirFieldLowering.cs | 267 ++++++++++++++++-- .../MimirStereoDepthConfiguration.cs | 82 ++++++ state/evidence.jsonl | 1 + state/map.yaml | 61 ++-- 9 files changed, 494 insertions(+), 73 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index e2f0c7d..6b6dd08 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -56,6 +56,15 @@ metric geometry truth. The libSGM-provenance profile currently publishes min disparity 0, 128 disparities, four aggregation paths, census radius 2, P1 8, and P2 96. Fensalir has the first packed-Leap D3D12 compute kernel installed as a crude SAD/block-match disparity writer; it is not full SGM yet. +The next socket now exists too: `MimirPointCloudConfigurations` records the +Leap disparity-to-point-cloud projection profile, and +`BuildLeapPackedStereoPointCloudCandidateFrame` derives a GPU-resident +`FieldMesh` point-list resource from the live Leap disparity resource. The +runtime merges that Mesh claim alongside the stereo-depth claim, and +`Mimir.BufferSmoke --leap-point-cloud-root-smoke` proves the contract plans as +one `SurfacePage` packet plus one `Mesh` packet with no deferred requests. This +is the first point-cloud root, not yet visible rendered points: Fensalir still +must implement the D3D12 disparity-to-vertex lowering and the editor draw path. Those claims are not inherently pixel-sized. Pixel-level resolve consumes the reservoir, but claim support is chosen from the represented field. Smooth @@ -565,9 +574,10 @@ a named invariant that the native runtime cannot protect yet. lane over rectified synchronized camera texture pairs, emitting GPU-resident disparity/depth/confidence resources and FieldEvidence claims. Keep `libSGM` as provenance for algorithm shape and benchmark pressure, not - as imported CUDA authority. The typed contract and smoke exist; the next cut - is the actual HLSL/D3D12 cost-volume and SGM aggregation kernel behind that - contract. + as imported CUDA authority. The typed contract, live packed-Leap disparity + path, and Leap point-cloud Mesh socket exist; the next cuts are full + HLSL/D3D12 SGM aggregation, disparity-to-point vertices, editor rendering, + and then a global residual/calibration owner. 9. Move mic alignment, room suppression, voice separation, spatialization, and stem generation into Faust/native DSP. 10. Keep the OBS bridge witness ledger as evidence before expanding receiver diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 382e6b5..0b9910f 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -448,6 +448,18 @@ publishes min disparity 0, 128 disparities, four paths, census radius 2, P1 8, and P2 96. Fensalir now has a first packed-Leap D3D12 kernel that writes R16F disparity through a crude SAD/block-match pass. This is intentionally a small kernel-shaped proof, not full libSGM-style SGM or calibrated metric depth yet. +The first point-cloud root is also code-visible now: +`MimirPointCloudConfigurations.LeapDisparityPointCloudRoot` records the +stereo-disparity projection profile and provenance, while +`MimirFensalirFieldLowering.BuildLeapPackedStereoPointCloudCandidateFrame` +declares a GPU-resident `FieldMesh` point-list resource derived from the Leap +R16F disparity SurfacePage. `MimirRuntime` merges that Mesh claim into the same +evidence frame as the Leap stereo-depth lowering. `Mimir.BufferSmoke +--leap-point-cloud-root-smoke` proves the combined depth/point-cloud contract +plans as one `SurfacePage` packet and one `Mesh` packet with zero deferred +requests. Fensalir still needs the actual D3D12 disparity-to-vertex compute +lowering and editor draw path before those points are visible in the Mimir +scene. Fensalir must not be treated as a traditional rendering pipeline. Mimir does not ask it to "draw a thing" and hope post-processing makes the result true. diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 548863b..5353adf 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -100,7 +100,14 @@ Get-Content .\state\evidence.jsonl -Tail 8 same declared texture as left/right resource identity and treats R/G as the packed observation lanes. Fensalir now has a first packed-Leap D3D12 SAD/block-match disparity writer, not full libSGM-style SGM or calibrated - metric depth. + metric depth. Mimir also now declares the first Leap point-cloud root: + `MimirPointCloudConfigurations.LeapDisparityPointCloudRoot` and + `BuildLeapPackedStereoPointCloudCandidateFrame` emit a GPU-resident + `FieldMesh` point-list resource derived from that disparity SurfacePage. Run + `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --leap-point-cloud-root-smoke` + to verify the combined `SurfacePage` plus `Mesh` packet plan. This is a + typed socket only; Fensalir still needs the D3D12 vertex-generation lowering + and editor draw path to show the point cloud. - Fensalir also owns the EVE-facing dashboard pixels through `Global\MimirFensalirProgramTexture`. `src/Mimir.EveRelay` opens that shared texture, encodes H.264 Annex-B with NVENC by default, and serves EveCanvas @@ -129,7 +136,9 @@ Get-Content .\state\evidence.jsonl -Tail 8 ## Current Pressure -- Implement the first concrete Leap direct capture driver and measure cadence. +- Implement Fensalir's Leap disparity-to-point vertex lowering and render the + resulting point cloud in the Mimir editor; then add the global + residual/calibration owner. - Cut the Phase 1 Mimir-to-Fensalir bridge DTOs for rolling windows, observations, calibration constraints, and surface intent before adding another backend-specific visual path. diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 4fa7070..5671fad 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -155,6 +155,11 @@ if (args.Any(arg => string.Equals(arg, "--leap-packed-stereo-depth-smoke", Strin return RunLeapPackedStereoDepthSmoke(); } +if (args.Any(arg => string.Equals(arg, "--leap-point-cloud-root-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunLeapPointCloudRootSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--ks-camera-driver-smoke", StringComparison.OrdinalIgnoreCase))) { return RunKsCameraDriverSmoke(args); @@ -2536,6 +2541,95 @@ static int RunLeapPackedStereoDepthSmoke() : 1; } +static int RunLeapPointCloudRootSmoke() +{ + const string sourceId = "leap-stereo-ir"; + var frameDescriptor = new MimirVideoFrameDescriptor( + Width: 640, + Height: 240, + PixelFormat: MimirVideoPixelFormat.LeapStereoIr, + StrideBytes: 1280, + DeviceTimestampNs: 1_000_000_000L, + NativeHandle: 0xCAFE, + NativeHandleKind: "fensalir-owned-d3d12-texture2d", + ResourceKey: MimirFensalirTextureLeaseClient.ResourceKeyForSource(sourceId), + ProducerFenceHandle: 0xD00D, + ProducerFenceValue: 7); + var sample = new MimirStreamSample( + sourceId, + MimirStreamKind.Video, + MimirStreamOrigin.LocalDevice, + TimestampNs: frameDescriptor.DeviceTimestampNs, + ArrivalNs: frameDescriptor.DeviceTimestampNs, + Sequence: 7, + PayloadHandle: frameDescriptor.NativeHandle, + ByteLength: 0, + VideoFrame: frameDescriptor); + var buffer = new MimirRollingStreamBuffer( + new MimirStreamDescriptor(sourceId, MimirStreamKind.Video, MimirStreamOrigin.LocalDevice), + TimeSpan.FromSeconds(5)); + buffer.Append(sample); + + var lowerer = new MimirFensalirFieldLowering(); + var cameraFrame = lowerer.BuildCameraObservationFrame([buffer]); + var windows = new List(); + MimirFensalirBridgeMapper.MapWindows([buffer], windows); + var stereoFrame = lowerer.BuildLeapPackedStereoDepthCandidateFrame(windows, cameraFrame.Resources); + var pointCloudFrame = lowerer.BuildLeapPackedStereoPointCloudCandidateFrame(windows); + var mergedResources = stereoFrame.Resources + .Concat(pointCloudFrame.Resources) + .Where(static resource => resource.HasIdentity) + .GroupBy(static resource => resource.ResourceKey, StringComparer.Ordinal) + .Select(static group => group.First()) + .ToArray(); + var merged = new AquariumFieldEvidenceFrame + { + Domains = [.. stereoFrame.Domains, .. pointCloudFrame.Domains], + Claims = [.. stereoFrame.Claims, .. pointCloudFrame.Claims], + Candidates = [.. stereoFrame.Candidates, .. pointCloudFrame.Candidates], + Resources = mergedResources, + StereoDepthLowerings = stereoFrame.StereoDepthLowerings, + AccumulationWindowSeconds = cameraFrame.AccumulationWindowSeconds, + PresentationDelaySeconds = cameraFrame.PresentationDelaySeconds + }; + var validation = AquariumFieldEvidenceValidator.Validate(merged); + var plan = AquariumFieldLoweringPlanner.Plan(merged); + var profile = MimirPointCloudConfigurations.LeapDisparityPointCloudRoot; + var pointResource = pointCloudFrame.Resources.FirstOrDefault(resource => + string.Equals(resource.ResourceKey, "mimir:resource:point-cloud:leap-stereo-ir:leap-points", StringComparison.Ordinal)); + var pointClaim = pointCloudFrame.Claims.FirstOrDefault(); + var meshPacket = plan.Packets.FirstOrDefault(packet => packet.Backend == AquariumFieldBackendKind.Mesh); + var stereoPacket = plan.Packets.FirstOrDefault(packet => packet.Backend == AquariumFieldBackendKind.SurfacePage); + + Console.WriteLine( + $"leap-point-cloud-root profile={profile.Id} provenance={profile.Provenance} liveDependency={profile.LiveDependency} stride={profile.DefaultSampleStride} baseline={profile.BaselineMeters:0.###} focal={profile.FocalLengthPixels:0.###}"); + Console.WriteLine( + $"leap-point-cloud-root-field resources={merged.Resources.Count} pointResources={pointCloudFrame.Resources.Count} claims={pointCloudFrame.Claims.Count} planned={plan.Packets.Count} deferred={plan.DeferredRequests.Count} errors={validation.HasErrors}"); + Console.WriteLine( + $"leap-point-cloud-root-output resource={pointResource.ResourceKey} kind={pointResource.Kind} topology={pointResource.Mesh.Topology} layout={pointResource.Mesh.Layout} vertices={pointResource.Mesh.Vertices.Count} indices={pointResource.Mesh.Indices.Count} source={pointResource.SourceUri} backend={meshPacket.Backend}"); + + return !validation.HasErrors && + !profile.LiveDependency && + profile.ProjectionKind == MimirPointCloudProjectionKind.StereoDisparityProjection && + profile.NegativeChecks.Any(check => check.Contains("no synthetic point cloud", StringComparison.OrdinalIgnoreCase)) && + pointCloudFrame.Resources.Count == 1 && + pointCloudFrame.Claims.Count == 1 && + pointClaim.Encoding == AquariumFieldEncoding.Mesh && + pointClaim.PayloadHandle == pointResource.ResourceKey && + pointClaim.ProducerKey == profile.Id && + pointResource.Kind == AquariumFieldResourceKind.Mesh && + pointResource.Mesh.Topology == AquariumFieldMeshTopology.PointList && + pointResource.Mesh.Layout == AquariumFieldMeshLayout.PositionNormalUvColor && + pointResource.Mesh.Vertices.Count == 320 * 120 && + pointResource.Mesh.Indices.Count == pointResource.Mesh.Vertices.Count && + pointResource.SourceUri == "derived-from:mimir:resource:stereo-depth:leap-stereo-ir:disparity-r16f" && + stereoPacket.Backend == AquariumFieldBackendKind.SurfacePage && + meshPacket.Backend == AquariumFieldBackendKind.Mesh && + plan.DeferredRequests.Count == 0 + ? 0 + : 1; +} + static int RunPerfectMachineProfileSmoke() { var profiles = MimirPerfectMachineProfiles.All; diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index bfbab0a..72cfabd 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -322,7 +322,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec var frame = fieldLowering.BuildFieldEvidenceFrame(windows, observations, constraints, intents); frame = AddSpectrumFieldEvidence(frame); - return AddLeapStereoDepthFieldEvidence(frame, windows); + return AddLeapGeometryFieldEvidence(frame, windows); } private void CommitProducerLeases(IReadOnlyList windows) @@ -369,7 +369,7 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec Purpose: MimirSurfaceIntentPurpose.Production); } - private AquariumFieldEvidenceFrame AddLeapStereoDepthFieldEvidence( + private AquariumFieldEvidenceFrame AddLeapGeometryFieldEvidence( AquariumFieldEvidenceFrame frame, IReadOnlyList windows) { @@ -379,16 +379,17 @@ public sealed class MimirRuntime : IAquariumRuntime, IAquariumRuntimeServicesRec return frame; } + var pointCloudFrame = fieldLowering.BuildLeapPackedStereoPointCloudCandidateFrame(windows); return new AquariumFieldEvidenceFrame { - Domains = [.. frame.Domains, .. stereoFrame.Domains], - Claims = [.. frame.Claims, .. stereoFrame.Claims], - Candidates = [.. frame.Candidates, .. stereoFrame.Candidates], - BackendPackets = [.. frame.BackendPackets, .. stereoFrame.BackendPackets], - Resources = MergeResources(frame.Resources, stereoFrame.Resources), - ResourceUploads = [.. frame.ResourceUploads, .. stereoFrame.ResourceUploads], - TubeSplineLowerings = [.. frame.TubeSplineLowerings, .. stereoFrame.TubeSplineLowerings], - StereoDepthLowerings = [.. frame.StereoDepthLowerings, .. stereoFrame.StereoDepthLowerings], + Domains = [.. frame.Domains, .. stereoFrame.Domains, .. pointCloudFrame.Domains], + Claims = [.. frame.Claims, .. stereoFrame.Claims, .. pointCloudFrame.Claims], + Candidates = [.. frame.Candidates, .. stereoFrame.Candidates, .. pointCloudFrame.Candidates], + BackendPackets = [.. frame.BackendPackets, .. stereoFrame.BackendPackets, .. pointCloudFrame.BackendPackets], + Resources = MergeResources(MergeResources(frame.Resources, stereoFrame.Resources), pointCloudFrame.Resources), + ResourceUploads = [.. frame.ResourceUploads, .. stereoFrame.ResourceUploads, .. pointCloudFrame.ResourceUploads], + TubeSplineLowerings = [.. frame.TubeSplineLowerings, .. stereoFrame.TubeSplineLowerings, .. pointCloudFrame.TubeSplineLowerings], + StereoDepthLowerings = [.. frame.StereoDepthLowerings, .. stereoFrame.StereoDepthLowerings, .. pointCloudFrame.StereoDepthLowerings], AccumulationWindowSeconds = frame.AccumulationWindowSeconds, PresentationDelaySeconds = frame.PresentationDelaySeconds, }; diff --git a/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs b/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs index c41736d..fcbed04 100644 --- a/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs +++ b/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs @@ -210,33 +210,59 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt var profile = MimirStereoDepthConfigurations.D3D12SgmLibSgmProvenance; var sourceSegment = SanitizeResourceSegment(leapWindow.StreamId); - var candidate = new MimirStereoDepthFieldCandidate( - CandidateKey: $"live-{sourceSegment}-{leapWindow.SequenceId}", - CalibrationId: "leapuvc-packed-stereo-ir-calibration-pending", - CameraPairId: leapWindow.StreamId, - ProducerKey: profile.Id, - ProfileId: profile.Id, - LeftObservationKey: $"{leapWindow.WindowId}:left-packed-r", - RightObservationKey: $"{leapWindow.WindowId}:right-packed-g", - LeftResourceKey: packedResourceKey, - RightResourceKey: packedResourceKey, - DisparityResourceKey: $"mimir:resource:stereo-depth:{sourceSegment}:disparity-r16f", - ConfidenceResourceKey: $"mimir:resource:stereo-depth:{sourceSegment}:confidence-r8", - Width: Math.Max(1, leapWindow.SampleDescriptor.Width), - Height: Math.Max(1, leapWindow.SampleDescriptor.Height), - MinDisparity: profile.MinDisparity, - DisparityLevels: profile.DisparityLevels, - AggregationPathCount: profile.AggregationPathCount, - CensusRadius: profile.CensusRadius, - SmoothnessPenaltySmall: profile.SmoothnessPenaltySmall, - SmoothnessPenaltyLarge: profile.SmoothnessPenaltyLarge, - MinDepthMeters: 0.20, - MaxDepthMeters: 4.0, - Confidence: 0.50, - ObservedTimeNs: leapWindow.EdgeNs); + var candidate = BuildLeapPackedStereoDepthCandidate(leapWindow, packedResourceKey, profile, sourceSegment); return BuildStereoDepthCandidateFrame([candidate], [packedInput]); } + public AquariumFieldEvidenceFrame BuildLeapPackedStereoPointCloudCandidateFrame( + IEnumerable windows) + { + var leapWindow = LatestLiveLeapPackedStereoWindow(windows); + if (string.IsNullOrWhiteSpace(leapWindow.WindowId)) + { + return AquariumFieldEvidenceFrame.Empty; + } + + var packedResourceKey = MimirFensalirBridgeMapper.ResourceKeyForPayload(leapWindow.Payload); + if (string.IsNullOrWhiteSpace(packedResourceKey)) + { + return AquariumFieldEvidenceFrame.Empty; + } + + var stereoProfile = MimirStereoDepthConfigurations.D3D12SgmLibSgmProvenance; + var sourceSegment = SanitizeResourceSegment(leapWindow.StreamId); + var depthCandidate = BuildLeapPackedStereoDepthCandidate(leapWindow, packedResourceKey, stereoProfile, sourceSegment); + var pointCloudProfile = MimirPointCloudConfigurations.LeapDisparityPointCloudRoot; + var stride = Math.Max(1, pointCloudProfile.DefaultSampleStride); + var width = Math.Max(1, depthCandidate.Width); + var height = Math.Max(1, depthCandidate.Height); + var sampledWidth = Math.Max(1, (width + stride - 1) / stride); + var sampledHeight = Math.Max(1, (height + stride - 1) / stride); + var candidate = new MimirPointCloudFieldCandidate( + CandidateKey: $"live-{sourceSegment}-point-cloud-{leapWindow.SequenceId}", + CalibrationId: depthCandidate.CalibrationId, + CameraRigId: depthCandidate.CameraPairId, + ProducerKey: pointCloudProfile.Id, + ProfileId: pointCloudProfile.Id, + SourceDisparityResourceKey: depthCandidate.DisparityResourceKey, + SourceConfidenceResourceKey: depthCandidate.ConfidenceResourceKey, + PointCloudResourceKey: $"mimir:resource:point-cloud:{sourceSegment}:leap-points", + Width: width, + Height: height, + SampleStride: stride, + MaxPointCount: sampledWidth * sampledHeight, + BaselineMeters: pointCloudProfile.BaselineMeters, + FocalLengthPixels: pointCloudProfile.FocalLengthPixels, + PrincipalPointX: Math.Clamp(pointCloudProfile.PrincipalPointX, 0.0, width), + PrincipalPointY: Math.Clamp(pointCloudProfile.PrincipalPointY, 0.0, height), + MinDepthMeters: depthCandidate.MinDepthMeters, + MaxDepthMeters: depthCandidate.MaxDepthMeters, + Confidence: Math.Min(depthCandidate.Confidence, 0.45), + ObservedTimeNs: depthCandidate.ObservedTimeNs); + + return BuildPointCloudCandidateFrame([candidate]); + } + public AquariumFieldEvidenceFrame BuildStereoDepthCandidateFrame( IEnumerable depthCandidates, IEnumerable? inputResources = null) @@ -285,6 +311,87 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt }; } + public AquariumFieldEvidenceFrame BuildPointCloudCandidateFrame( + IEnumerable pointCloudCandidates) + { + var domains = new List(); + var claims = new List(); + var candidates = new List(); + var resources = new List(); + var seenDomains = new HashSet(StringComparer.Ordinal); + var seenResources = new HashSet(StringComparer.Ordinal); + + foreach (var pointCloudCandidate in pointCloudCandidates) + { + if (string.IsNullOrWhiteSpace(pointCloudCandidate.CandidateKey) || + string.IsNullOrWhiteSpace(pointCloudCandidate.PointCloudResourceKey) || + string.IsNullOrWhiteSpace(pointCloudCandidate.SourceDisparityResourceKey)) + { + continue; + } + + var domain = DomainForPointCloudCandidate(pointCloudCandidate); + AddDomain(domains, seenDomains, domain); + AddResource(resources, seenResources, PointCloudResourceForCandidate(pointCloudCandidate)); + + var claim = ClaimForPointCloudCandidate(pointCloudCandidate, domain.DomainKey); + claims.Add(claim); + candidates.Add(CandidateForClaim(claim, AquariumFieldGuide.Valid(claim.Confidence))); + } + + return new AquariumFieldEvidenceFrame + { + Domains = domains, + Claims = claims, + Candidates = candidates, + Resources = resources, + AccumulationWindowSeconds = (float)options.AccumulationWindowSeconds, + PresentationDelaySeconds = (float)options.PresentationDelaySeconds + }; + } + + private static MimirRollingStreamWindow LatestLiveLeapPackedStereoWindow( + IEnumerable windows) => + windows + .Where(static window => + window.Status == MimirBridgeWindowStatus.Live && + window.SourceKind == MimirStreamKind.Video && + window.SampleDescriptor.PixelFormat == MimirVideoPixelFormat.LeapStereoIr && + window.Payload.HasResource && + !string.IsNullOrWhiteSpace(window.Payload.ResourceKey)) + .OrderByDescending(static window => window.EdgeNs) + .FirstOrDefault(); + + private static MimirStereoDepthFieldCandidate BuildLeapPackedStereoDepthCandidate( + MimirRollingStreamWindow leapWindow, + string packedResourceKey, + MimirStereoDepthKernelProfile profile, + string sourceSegment) => + new( + CandidateKey: $"live-{sourceSegment}-{leapWindow.SequenceId}", + CalibrationId: "leapuvc-packed-stereo-ir-calibration-pending", + CameraPairId: leapWindow.StreamId, + ProducerKey: profile.Id, + ProfileId: profile.Id, + LeftObservationKey: $"{leapWindow.WindowId}:left-packed-r", + RightObservationKey: $"{leapWindow.WindowId}:right-packed-g", + LeftResourceKey: packedResourceKey, + RightResourceKey: packedResourceKey, + DisparityResourceKey: $"mimir:resource:stereo-depth:{sourceSegment}:disparity-r16f", + ConfidenceResourceKey: $"mimir:resource:stereo-depth:{sourceSegment}:confidence-r8", + Width: Math.Max(1, leapWindow.SampleDescriptor.Width), + Height: Math.Max(1, leapWindow.SampleDescriptor.Height), + MinDisparity: profile.MinDisparity, + DisparityLevels: profile.DisparityLevels, + AggregationPathCount: profile.AggregationPathCount, + CensusRadius: profile.CensusRadius, + SmoothnessPenaltySmall: profile.SmoothnessPenaltySmall, + SmoothnessPenaltyLarge: profile.SmoothnessPenaltyLarge, + MinDepthMeters: 0.20, + MaxDepthMeters: 4.0, + Confidence: 0.50, + ObservedTimeNs: leapWindow.EdgeNs); + private static void AddDomain( ICollection domains, ISet seenDomains, @@ -477,6 +584,25 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt "Fensalir D3D12 compute"); } + private static AquariumFieldDomain DomainForPointCloudCandidate(MimirPointCloudFieldCandidate candidate) + { + var maxDepth = Math.Max(candidate.MinDepthMeters, candidate.MaxDepthMeters); + var halfWidthMeters = (float)Math.Max(0.01, maxDepth * candidate.Width / Math.Max(1.0, candidate.FocalLengthPixels) * 0.5); + var halfHeightMeters = (float)Math.Max(0.01, maxDepth * candidate.Height / Math.Max(1.0, candidate.FocalLengthPixels) * 0.5); + var minDepth = (float)Math.Max(0.0, candidate.MinDepthMeters); + var maxDepthFloat = (float)Math.Max(minDepth + 0.001, maxDepth); + return new AquariumFieldDomain( + DomainKeyForPointCloudCandidate(candidate), + DomainKeyForStereoDepthCandidate(candidate.CalibrationId, candidate.CameraRigId, candidate.SourceDisparityResourceKey), + AquariumFieldDomainKind.Object3D, + Matrix4x4.Identity, + Matrix4x4.Identity, + new Vector3(-halfWidthMeters, -halfHeightMeters, minDepth), + new Vector3(halfWidthMeters, halfHeightMeters, maxDepthFloat), + Vector3.Zero, + "Fensalir D3D12 compute"); + } + private static AquariumFieldDomain DomainForSurfaceIntent(MimirSurfaceIntent intent) => new( DomainKeyForSurfaceIntent(intent.IntentKey), @@ -611,6 +737,32 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt Confidence: confidence); } + private AquariumFieldClaim ClaimForPointCloudCandidate( + MimirPointCloudFieldCandidate pointCloudCandidate, + string domainKey) + { + var confidence = Clamp01((float)pointCloudCandidate.Confidence); + var represented = Math.Max(1, pointCloudCandidate.MaxPointCount); + var proposal = new AquariumFieldProposalPolicy( + AquariumFieldProposalKind.DeterministicStructural, + SourcePdf: 1.0f, + TargetContribution: confidence, + RepresentedCandidateCount: represented, + Seed: StableSeed(pointCloudCandidate.CandidateKey)); + + return new AquariumFieldClaim( + ClaimKey: $"point-cloud:{pointCloudCandidate.CalibrationId}:{pointCloudCandidate.CameraRigId}:{pointCloudCandidate.CandidateKey}", + DomainKey: domainKey, + ProducerKey: pointCloudCandidate.ProducerKey, + Layer: AquariumFieldLayer.Form, + Encoding: AquariumFieldEncoding.Mesh, + Support: SupportForPointCloudCandidate(pointCloudCandidate), + Proposal: proposal, + PayloadHandle: pointCloudCandidate.PointCloudResourceKey, + ObservedTimeNs: pointCloudCandidate.ObservedTimeNs, + Confidence: confidence); + } + private AquariumFieldClaim ClaimForSurfaceIntent( MimirSurfaceIntent intent, string domainKey, @@ -767,6 +919,22 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt TemporalUncertainty: 0.0f); } + private static AquariumFieldSupport SupportForPointCloudCandidate(MimirPointCloudFieldCandidate pointCloudCandidate) + { + var maxDepth = Math.Max(pointCloudCandidate.MinDepthMeters, pointCloudCandidate.MaxDepthMeters); + var halfWidthMeters = (float)Math.Max(0.01, maxDepth * pointCloudCandidate.Width / Math.Max(1.0, pointCloudCandidate.FocalLengthPixels) * 0.5); + var halfHeightMeters = (float)Math.Max(0.01, maxDepth * pointCloudCandidate.Height / Math.Max(1.0, pointCloudCandidate.FocalLengthPixels) * 0.5); + var depthRange = Math.Max(0.001f, (float)(pointCloudCandidate.MaxDepthMeters - pointCloudCandidate.MinDepthMeters)); + return new AquariumFieldSupport( + Center: new Vector3(0.0f, 0.0f, (float)(pointCloudCandidate.MinDepthMeters + pointCloudCandidate.MaxDepthMeters) * 0.5f), + Radius: new Vector3(halfWidthMeters, halfHeightMeters, depthRange * 0.5f), + LocalFrame: Matrix4x4.Identity, + ConservativeRadius: MathF.Max(MathF.Max(halfWidthMeters, halfHeightMeters), depthRange * 0.5f), + ProjectedError: 1.0f / Math.Max(1, pointCloudCandidate.SampleStride), + Curvature: 0.0f, + TemporalUncertainty: 0.0f); + } + private static AquariumFieldResourceDeclaration DisparityResourceForStereoDepthCandidate( MimirStereoDepthFieldCandidate depthCandidate) => new( @@ -810,6 +978,49 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt NativeHandleKind: "fensalir-stereo-depth-confidence"); } + private static AquariumFieldResourceDeclaration PointCloudResourceForCandidate( + MimirPointCloudFieldCandidate pointCloudCandidate) + { + var pointCount = Math.Max(1, pointCloudCandidate.MaxPointCount); + var mesh = new AquariumFieldMeshResource( + Vertices: new AquariumFieldMeshBuffer( + BufferKey: $"{pointCloudCandidate.PointCloudResourceKey}:vertices", + Count: pointCount, + StrideBytes: AquariumFieldMeshResource.PositionNormalUvColorStrideBytes, + NativeHandle: IntPtr.Zero, + NativeHandleKind: "fensalir-generated-vertex-buffer"), + Indices: new AquariumFieldMeshBuffer( + BufferKey: $"{pointCloudCandidate.PointCloudResourceKey}:indices", + Count: pointCount, + StrideBytes: sizeof(uint), + NativeHandle: IntPtr.Zero, + NativeHandleKind: "fensalir-generated-index-buffer"), + Topology: AquariumFieldMeshTopology.PointList, + IndexFormat: AquariumFieldMeshIndexFormat.UInt32, + Layout: AquariumFieldMeshLayout.PositionNormalUvColor, + BoundsMin: new Vector3(-1.0f, -1.0f, (float)Math.Max(0.0, pointCloudCandidate.MinDepthMeters)), + BoundsMax: new Vector3(1.0f, 1.0f, (float)Math.Max(pointCloudCandidate.MinDepthMeters + 0.001, pointCloudCandidate.MaxDepthMeters)), + SubmeshCount: 1); + + return new AquariumFieldResourceDeclaration( + ResourceKey: pointCloudCandidate.PointCloudResourceKey, + Kind: AquariumFieldResourceKind.Mesh, + Residency: AquariumFieldResourceResidency.GpuResident, + Access: AquariumFieldShaderAccess.ShaderResource, + Format: "FieldMesh", + Width: pointCount, + Height: 1, + DepthOrCount: pointCount, + StrideBytes: AquariumFieldMeshResource.PositionNormalUvColorStrideBytes, + ValidFromNs: pointCloudCandidate.ObservedTimeNs, + ValidUntilNs: pointCloudCandidate.ObservedTimeNs, + Version: checked((ulong)Math.Max(0L, pointCloudCandidate.ObservedTimeNs)), + NativeHandle: IntPtr.Zero, + NativeHandleKind: "fensalir-leap-point-cloud", + SourceUri: $"derived-from:{pointCloudCandidate.SourceDisparityResourceKey}", + Mesh: mesh); + } + private static AquariumFieldStereoDepthLowering StereoDepthLoweringForCandidate( MimirStereoDepthFieldCandidate depthCandidate, string claimKey) => @@ -1004,7 +1215,13 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt $"mimir:visual-marker:{candidate.CalibrationId}:{candidate.MarkerId}:{candidate.CandidateKey}"; private static string DomainKeyForStereoDepthCandidate(MimirStereoDepthFieldCandidate candidate) => - $"mimir:stereo-depth:{candidate.CalibrationId}:{candidate.CameraPairId}:{candidate.CandidateKey}"; + DomainKeyForStereoDepthCandidate(candidate.CalibrationId, candidate.CameraPairId, candidate.CandidateKey); + + private static string DomainKeyForStereoDepthCandidate(string calibrationId, string cameraPairId, string candidateKey) => + $"mimir:stereo-depth:{calibrationId}:{cameraPairId}:{candidateKey}"; + + private static string DomainKeyForPointCloudCandidate(MimirPointCloudFieldCandidate candidate) => + $"mimir:point-cloud:{candidate.CalibrationId}:{candidate.CameraRigId}:{candidate.CandidateKey}"; private static string DomainKeyForSurfaceIntent(string intentKey) => $"mimir:intent:{intentKey}"; diff --git a/src/Mimir.Runtime/Synchronization/MimirStereoDepthConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirStereoDepthConfiguration.cs index 6377b97..7f45288 100644 --- a/src/Mimir.Runtime/Synchronization/MimirStereoDepthConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirStereoDepthConfiguration.cs @@ -14,6 +14,13 @@ public enum MimirStereoDepthOutputKind ConfidenceTexture } +public enum MimirPointCloudProjectionKind +{ + StereoDisparityProjection, + VisualFeatureTriangulation, + ExternalDepthSensor +} + public sealed record MimirStereoDepthKernelProfile( string Id, string Description, @@ -59,6 +66,45 @@ public sealed record MimirStereoDepthFieldCandidate( double Confidence, long ObservedTimeNs); +public sealed record MimirPointCloudProjectionProfile( + string Id, + string Description, + MimirPointCloudProjectionKind ProjectionKind, + string Owner, + string Provenance, + string License, + bool LiveDependency, + string[] RequiredInputResources, + string[] OutputResources, + int DefaultSampleStride, + double BaselineMeters, + double FocalLengthPixels, + double PrincipalPointX, + double PrincipalPointY, + string[] NegativeChecks); + +public sealed record MimirPointCloudFieldCandidate( + string CandidateKey, + string CalibrationId, + string CameraRigId, + string ProducerKey, + string ProfileId, + string SourceDisparityResourceKey, + string SourceConfidenceResourceKey, + string PointCloudResourceKey, + int Width, + int Height, + int SampleStride, + int MaxPointCount, + double BaselineMeters, + double FocalLengthPixels, + double PrincipalPointX, + double PrincipalPointY, + double MinDepthMeters, + double MaxDepthMeters, + double Confidence, + long ObservedTimeNs); + public static class MimirStereoDepthConfigurations { public static MimirStereoDepthKernelProfile D3D12SgmLibSgmProvenance { get; } = new( @@ -134,3 +180,39 @@ public static class MimirStereoDepthConfigurations DepthAnythingSmallReference ]; } + +public static class MimirPointCloudConfigurations +{ + public static MimirPointCloudProjectionProfile LeapDisparityPointCloudRoot { get; } = new( + "leap-disparity-point-cloud-root", + "D3D12 point-cloud projection rooted in the Leap packed-stereo disparity SurfacePage.", + MimirPointCloudProjectionKind.StereoDisparityProjection, + "Fensalir D3D12 compute", + "Derived from calibrated stereo pinhole projection: z = f * baseline / disparity; disparity source follows the libSGM-provenance D3D12 lane.", + "Repo-native implementation; upstream stereo kernel provenance remains Apache-2.0 libSGM.", + LiveDependency: false, + RequiredInputResources: + [ + "R16Float disparity SurfacePage", + "R8_UNorm confidence Texture2D", + "stereo intrinsics/extrinsics calibration" + ], + OutputResources: ["FieldMesh PointList PositionNormalUvColor"], + DefaultSampleStride: 2, + BaselineMeters: 0.04, + FocalLengthPixels: 250.0, + PrincipalPointX: 320.0, + PrincipalPointY: 120.0, + NegativeChecks: + [ + "no synthetic point cloud without a live disparity resource", + "no RGB monocular depth accepted as metric geometry owner", + "no calibration optimizer update before residual ownership exists", + "no CPU point array in the live D3D12 path" + ]); + + public static IReadOnlyList BuiltIn { get; } = + [ + LeapDisparityPointCloudRoot + ]; +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index a825fc9..9dacfa0 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -242,3 +242,4 @@ {"ts": "2026-05-30T22:18:07+01:00", "type": "decision", "status": "ok", "note": "Viewer authority moved to root gamecult-site at https://gamecult.org/livestream. Mimir no longer owns a static live viewer page; Mimir.Broadcast remains the Starfire sender, and Yggdrasil serves HLS through the existing StreamPixels TLS route https://streampixels.gamecult.org/mimir/live/hls/mimir.m3u8. The deployed StreamPixels nginx route health check returned mimir-live-ok, and an active synthetic NVENC push generated playlist/segments served publicly through StreamPixels; curl fallback benchmark fetched 320 segment requests in 6756 ms over HTTPS from Yggdrasil."} {"ts": "2026-05-30T22:45:01+01:00", "type": "decision", "status": "ok", "note": "Added Raven/Eve ingest setup: ffmpeg-rawvideo runtime source decodes SRT feeds into Mimir video samples; raven-display listens on 5200, eve-camera on 5201; Raven desktop sender script uses gdigrab plus NVENC. Eve receiver lane exists but the iPad camera producer remains an explicit AVFoundation/external-app decision."} {"ts": "2026-05-30T22:46:17+01:00", "type": "verification", "status": "warn", "note": "Raven/Eve ingest host probe: Eve SSH is reachable and reports iPadOS Darwin 20.3.0, but /var/mobile/Projects/Eve only contains the receiver/control app sources and no ffmpeg/avcapture helper was found. Raven SSH with the recorded key still fails publickey authentication, so Raven screen capture is prepared as a desktop-run script until SSH authority is repaired."} +{"ts": "2026-05-30T22:53:29+01:00", "type": "decision", "status": "ok", "note": "Leap dense geometry now has a Mimir-side point-cloud root: live Leap packed stereo lowers to R16F disparity, and Mimir declares a derived GPU FieldMesh PointList resource that plans as a Mesh backend packet. This is a typed socket only; Fensalir still owns D3D12 vertex generation, editor draw, and global residual/calibration ownership."} diff --git a/state/map.yaml b/state/map.yaml index f8476a6..4a8c1ba 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -15,10 +15,10 @@ objective: microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP, and OBS output. current: > - Promote the active bioacoustic packet-song receiver into runtime state: - self-identifying song contours should pin multiple time/frequency anchors - per heard call, learn per-path response, and drive the audio actuator while - direct camera/audio capture continues feeding Mimir.Runtime buffers. + Make Leap dense geometry visible and calibratable first: live Leap packed + stereo IR feeds Fensalir D3D12 disparity, Mimir declares a derived + point-cloud Mesh socket, then Fensalir should generate/render points in the + editor and feed global residuals into online calibration. constraints: - Sender is Windows. @@ -75,6 +75,16 @@ invariants: packed-Leap D3D12 SAD/block-match kernel that writes R16F disparity through the typed SurfacePage socket; full SGM and calibrated metric depth remain future work. + - The first Leap point-cloud root is code-visible: + `MimirPointCloudConfigurations.LeapDisparityPointCloudRoot` records the + stereo-disparity projection profile and provenance, and + `MimirFensalirFieldLowering.BuildLeapPackedStereoPointCloudCandidateFrame` + declares a GPU-resident `FieldMesh` PointList resource derived from the + Leap disparity SurfacePage. `MimirRuntime` merges that Mesh claim with the + live Leap stereo-depth claim. `Mimir.BufferSmoke --leap-point-cloud-root-smoke` + proves one `SurfacePage` packet plus one `Mesh` packet and zero deferred + requests. This is a typed socket, not visible points yet; Fensalir still + owns the D3D12 disparity-to-vertex lowering and editor draw path. - Fensalir packet payload handles are not authority. Live payload references must resolve through declared typed GPU/native resources with shape, residency, shader access, validity, version, and handle metadata before @@ -624,7 +634,10 @@ accepted_design: whenever dirty. active_subgoals: - - Implement the first concrete Leap direct capture driver and measure cadence. + - Implement Fensalir's Leap disparity-to-point vertex lowering and render the + resulting point cloud in the Mimir editor. + - Add a global residual/calibration owner that can consume Leap point-cloud + errors before PS3 Eyes, Kiyos, and Eve feed wider online calibration. - Add remaining local camera drivers through `IMimirVideoCaptureDriver`. - Use the verified in-process Focusrite ASIO source for local Scarlett mic and loopback ingest; WASAPI remains a diagnostic witness. @@ -712,31 +725,13 @@ current_status: Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > - Use the new bioacoustic timeline watermark as the active acoustic sync - direction. Cut the next implementation in this order: streaming indexed - log-mel/MFCC motif decoder state with a global clock/path hypothesis fed by - `--bioacoustic-train` receipts, calibration-session command/replay proof that learns - per-output/mic motif/formant response, fractional-delay/SRO actuator, then - native camera payload handles and Fensalir lowering. Keep chirp-bin - calibration captures as reference evidence, not the runtime target. - Current app testing proves Fensalir WASAPI output wakes Scarlett speaker - loopback, live mic buffers stay populated, runtime sync analysis now runs as - a bounded rotating service without UI/telemetry re-entry, and the bioacoustic - motif timeline produces deterministic synthetic active sync. The next - invariant is self-identifying song contours producing stable intra-call - time/frequency anchors and clock fits through real loopback and microphones, - so the actuator can use decoded time instead of filtered lag guesses. - Replace the remaining - diagnostic JSON-line process bridges with direct native capture drivers that - push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports - for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when - calibration witnesses are playing. Treat PS3 Eye audio as - enumeration/runtime-fragile, not unavailable. Keep the Focusrite away from - the Leap USB neighborhood. The Scarlett Solo 4th Gen is now attached to - Starfire, and `native/probes/asio_audio_cadence` proves local Focusrite USB - ASIO access with 4 inputs / 2 outputs including Loopback 1/2, 192-frame - preferred buffers, 44.1-192 kHz support, and nonzero 4-channel Int32LSB - input callbacks at 192 kHz. Raven also has a 192 kHz loopback-capable - Scarlett for co-streamer/game timing evidence, but Starfire owns the heavy - soundfield and sensor-fusion work. Treat Kiyo Pro as a ~25 fps source until - its SuperSpeed negotiation is fixed. + Continue the Leap-first visual geometry cut. Mimir now lowers live + `LeapStereoIr` packed R/G stereo into a disparity SurfacePage and declares a + derived point-cloud Mesh socket. The next implementation belongs in + Fensalir: generate PointList vertices from disparity on D3D12, render that + resource in the Mimir editor camera, then add a global residual/calibration + owner that can consume point-cloud alignment error. Do not fan RGB cameras + into fake depth: PS3 Eyes are fast motion/feature witnesses, Kiyos and Eve + are RGB/context witnesses, and Leap owns the first dense close-range + geometry. Keep the Focusrite away from the Leap USB neighborhood. Treat Kiyo + Pro as a ~25 fps source until its SuperSpeed negotiation is fixed.