Solve LED curves into camera calibration updates
This commit is contained in:
parent
de75fa3445
commit
43e0f91f6a
7 changed files with 440 additions and 50 deletions
|
|
@ -382,10 +382,14 @@ a named invariant that the native runtime cannot protect yet.
|
|||
`BuildLedSplineCandidateFrame` lowers that curve as deterministic
|
||||
camera-sensor Feature evidence, and `--led-spline-calibration-smoke` proves
|
||||
Fensalir plans it as one `DebugOverlay` packet with no deferred request.
|
||||
`MimirVisualCalibrationResidualSolver` is the first higher-level owner shape:
|
||||
it compares coded/indexed observations across camera frustums and emits
|
||||
residual reports, but it does not write camera pose until the calibration
|
||||
state contract exists. This is calibration evidence only. Exact
|
||||
`MimirLedSplineCurveSolver` now owns the first per-sensor curve solve from
|
||||
bright LED detections into ordered image/clipspace samples. `MimirCameraRigCalibrationSolver`
|
||||
owns the next bounded pose step: given stable LED indices plus a scene-space
|
||||
LED curve anchor, it fits camera translation updates by minimizing ray-to-
|
||||
curve distance from each local frustum. The smoke synthesizes a scene curve,
|
||||
projects it through two camera frustums, solves the curves back out of the
|
||||
detections, and recovers bounded camera-position updates with near-zero ray
|
||||
residual. Rotation and intrinsics fitting remain explicit future work. Exact
|
||||
correspondence requires coded or otherwise stable LED identity; uncoded
|
||||
repeated lights remain curve constraints until the global residual solver
|
||||
resolves them.
|
||||
|
|
|
|||
|
|
@ -10,9 +10,9 @@ flowchart TD
|
|||
C["Leap stereo IR driver"] --> B
|
||||
B --> D["native reservoir handles"]
|
||||
D --> E["Fensalir GPU feature extraction"]
|
||||
E --> J["LED spline observations in local clipspace"]
|
||||
E --> J["per-sensor LED curve solve"]
|
||||
E --> F["cross-view matching + flow"]
|
||||
J --> K["global residual calibration owner"]
|
||||
J --> K["camera rig calibration solver"]
|
||||
K --> F
|
||||
F --> G["surface/material claims"]
|
||||
G --> H["brush/splat render budget"]
|
||||
|
|
@ -33,6 +33,9 @@ flowchart TD
|
|||
- Stable LED index identity requires temporal/color/address coding or another
|
||||
correspondence source; uncoded identical lights are spline constraints, not
|
||||
metric depth authority.
|
||||
- The first camera-rig solver fits bounded camera translation updates from a
|
||||
scene-anchored LED curve by minimizing ray-to-curve distance. Rotation,
|
||||
intrinsics, and distortion are future owners.
|
||||
- Once camera frustums are coherent, every detected surface candidate can be
|
||||
resampled across the synchronized view set before Fensalir resolves it into
|
||||
spatiotemporal surface/splat claims.
|
||||
|
|
|
|||
|
|
@ -612,13 +612,17 @@ deterministic camera-sensor Feature evidence; Fensalir plans it as a
|
|||
blink-coded strips can provide exact cross-camera LED identity. Plain identical
|
||||
white LEDs still provide a strong spline/epipolar constraint, but their point
|
||||
correspondence is ambiguous under occlusion, saturation, repeated spacing, and
|
||||
motion blur until temporal continuity, Leap/body anchors, or a future residual
|
||||
solver disambiguates it. Individual cameras do not write calibration state; the
|
||||
future global residual owner fits local frustums into a coherent scene and is
|
||||
the only authority allowed to update camera intrinsics/extrinsics. Once that
|
||||
frustum fit is coherent, detected surface candidates can be sampled across
|
||||
every synchronized view before Fensalir resolves them into 4D Gaussian/surface
|
||||
splat claims.
|
||||
motion blur until temporal continuity, Leap/body anchors, or the residual
|
||||
solver disambiguates it. `MimirLedSplineCurveSolver` is the first per-sensor
|
||||
curve owner: it orders bright detections and computes image/clipspace curve
|
||||
samples. `MimirCameraRigCalibrationSolver` is the first higher-level frustum
|
||||
fit owner: given stable LED indices and a scene-space LED curve anchor, it
|
||||
minimizes ray-to-curve distance and emits bounded camera translation updates.
|
||||
Individual cameras do not write calibration state. Rotation, focal length, and
|
||||
distortion solving are still future cuts, not hidden behavior. Once the frustum
|
||||
fit is coherent, detected surface candidates can be sampled across every
|
||||
synchronized view before Fensalir resolves them into 4D Gaussian/surface splat
|
||||
claims.
|
||||
|
||||
The current teardown/migration map is
|
||||
`docs/fensalir-rendering-rebuild-migration.md`, paired with Fensalir's
|
||||
|
|
|
|||
|
|
@ -798,6 +798,31 @@ static int RunFensalirCameraObservationSmoke()
|
|||
|
||||
static int RunLedSplineCalibrationSmoke()
|
||||
{
|
||||
var scenePoints = new[]
|
||||
{
|
||||
new MimirLedSplineScenePoint(0, new Vector3(-0.20f, 0.30f, 2.20f), 0.95),
|
||||
new MimirLedSplineScenePoint(1, new Vector3(-0.08f, 0.18f, 2.05f), 0.96),
|
||||
new MimirLedSplineScenePoint(2, new Vector3(0.04f, 0.02f, 1.95f), 0.97),
|
||||
new MimirLedSplineScenePoint(3, new Vector3(0.18f, -0.14f, 1.88f), 0.95),
|
||||
new MimirLedSplineScenePoint(4, new Vector3(0.35f, -0.28f, 1.82f), 0.93),
|
||||
};
|
||||
var truePs3Pose = new MimirCameraPoseEstimate("ps3-eye-0", Vector3.Zero, Quaternion.Identity, 1.0);
|
||||
var trueKiyoPose = new MimirCameraPoseEstimate("kiyo-basic", new Vector3(0.24f, 0.02f, -0.03f), Quaternion.Identity, 1.0);
|
||||
var curveSolver = new MimirLedSplineCurveSolver();
|
||||
var ps3Fit = curveSolver.SolveCameraCurve(
|
||||
truePs3Pose.SourceId,
|
||||
"ps3-eye-0:led-spline:42",
|
||||
320,
|
||||
240,
|
||||
1_000_000_000L,
|
||||
ProjectLedDetections(scenePoints, truePs3Pose, 320, 240, radiusPixels: 2.4));
|
||||
var kiyoFit = curveSolver.SolveCameraCurve(
|
||||
trueKiyoPose.SourceId,
|
||||
"kiyo-basic:led-spline:42",
|
||||
1920,
|
||||
1080,
|
||||
1_000_000_004L,
|
||||
ProjectLedDetections(scenePoints, trueKiyoPose, 1920, 1080, radiusPixels: 4.5));
|
||||
var candidate = new MimirLedSplineFieldCandidate(
|
||||
CandidateKey: "co-streamer-led-strip-frame-42",
|
||||
CalibrationId: "room-camera-rig-online-calibration",
|
||||
|
|
@ -805,41 +830,13 @@ static int RunLedSplineCalibrationSmoke()
|
|||
ProducerKey: "mimir-led-spline-detector",
|
||||
CameraObservations:
|
||||
[
|
||||
new MimirLedSplineCameraObservation(
|
||||
ObservationKey: "ps3-eye-0:led-spline:42",
|
||||
SourceId: "ps3-eye-0",
|
||||
Width: 320,
|
||||
Height: 240,
|
||||
ObservedTimeNs: 1_000_000_000L,
|
||||
Points:
|
||||
[
|
||||
new MimirLedSplineObservationPoint(0, 0.00, 110.0, 82.0, -0.3125, 0.3167, 2.5, 0.91),
|
||||
new MimirLedSplineObservationPoint(1, 0.25, 132.0, 98.0, -0.1750, 0.1833, 2.2, 0.94),
|
||||
new MimirLedSplineObservationPoint(2, 0.50, 148.0, 126.0, -0.0750, -0.0500, 2.4, 0.92),
|
||||
new MimirLedSplineObservationPoint(3, 0.75, 171.0, 152.0, 0.0688, -0.2667, 2.6, 0.90),
|
||||
new MimirLedSplineObservationPoint(4, 1.00, 204.0, 184.0, 0.2750, -0.5333, 2.7, 0.89),
|
||||
],
|
||||
Confidence: 0.91),
|
||||
new MimirLedSplineCameraObservation(
|
||||
ObservationKey: "kiyo-basic:led-spline:42",
|
||||
SourceId: "kiyo-basic",
|
||||
Width: 1920,
|
||||
Height: 1080,
|
||||
ObservedTimeNs: 1_000_000_004L,
|
||||
Points:
|
||||
[
|
||||
new MimirLedSplineObservationPoint(0, 0.00, 763.0, 356.0, -0.2052, 0.3407, 4.4, 0.88),
|
||||
new MimirLedSplineObservationPoint(1, 0.25, 840.0, 432.0, -0.1250, 0.2000, 4.1, 0.93),
|
||||
new MimirLedSplineObservationPoint(2, 0.50, 914.0, 552.0, -0.0479, -0.0222, 4.3, 0.94),
|
||||
new MimirLedSplineObservationPoint(3, 0.75, 1018.0, 673.0, 0.0604, -0.2463, 4.6, 0.90),
|
||||
new MimirLedSplineObservationPoint(4, 1.00, 1188.0, 816.0, 0.2375, -0.5111, 4.9, 0.86),
|
||||
],
|
||||
Confidence: 0.90),
|
||||
curveSolver.ToCameraObservation(ps3Fit, 320, 240, 1_000_000_000L),
|
||||
curveSolver.ToCameraObservation(kiyoFit, 1920, 1080, 1_000_000_004L),
|
||||
],
|
||||
EstimatedLedCount: 5,
|
||||
HasTemporalCode: true,
|
||||
HasStableLedIndices: true,
|
||||
CurveLengthPixels: 568.0,
|
||||
CurveLengthPixels: Math.Max(ps3Fit.CurveLengthPixels, kiyoFit.CurveLengthPixels),
|
||||
Confidence: 0.89,
|
||||
ObservedTimeNs: 1_000_000_004L);
|
||||
|
||||
|
|
@ -848,10 +845,18 @@ static int RunLedSplineCalibrationSmoke()
|
|||
var validation = AquariumFieldEvidenceValidator.Validate(frame);
|
||||
var plan = AquariumFieldLoweringPlanner.Plan(frame);
|
||||
var residual = new MimirVisualCalibrationResidualSolver().EvaluateLedSpline(candidate);
|
||||
var rigCalibration = new MimirCameraRigCalibrationSolver().FitCameraPositionsFromLedSpline(
|
||||
candidate,
|
||||
scenePoints,
|
||||
[
|
||||
truePs3Pose with { PositionMeters = new Vector3(0.02f, -0.01f, 0.01f), Confidence = 0.75 },
|
||||
trueKiyoPose with { PositionMeters = new Vector3(0.10f, 0.00f, -0.02f), Confidence = 0.70 },
|
||||
],
|
||||
maximumDeltaMeters: 0.50);
|
||||
var claim = frame.Claims.SingleOrDefault();
|
||||
|
||||
Console.WriteLine(
|
||||
$"led-spline-calibration-smoke domains={frame.Domains.Count} claims={frame.Claims.Count} planned={plan.Packets.Count} deferred={plan.DeferredRequests.Count} pairs={residual.PairResiduals.Count} clipResidual={residual.PairResiduals.FirstOrDefault()?.MeanClipspaceDistance ?? double.NaN:0.000000} points={candidate.CameraObservations.Sum(static observation => observation.Points.Count)} errors={validation.HasErrors}");
|
||||
$"led-spline-calibration-smoke domains={frame.Domains.Count} claims={frame.Claims.Count} planned={plan.Packets.Count} deferred={plan.DeferredRequests.Count} pairs={residual.PairResiduals.Count} poseUpdates={rigCalibration.PoseUpdates.Count} rayResidual={rigCalibration.MeanRayDistanceMeters:0.000000} points={candidate.CameraObservations.Sum(static observation => observation.Points.Count)} errors={validation.HasErrors}");
|
||||
|
||||
if (validation.HasErrors)
|
||||
{
|
||||
|
|
@ -874,7 +879,11 @@ static int RunLedSplineCalibrationSmoke()
|
|||
!residual.HasPoseUpdate &&
|
||||
residual.PairResiduals.Count == 1 &&
|
||||
residual.PairResiduals[0].SharedLedCount == 5 &&
|
||||
residual.PairResiduals[0].MeanClipspaceDistance < 0.12 &&
|
||||
rigCalibration.HasPoseUpdate &&
|
||||
rigCalibration.PoseUpdates.Count == 2 &&
|
||||
rigCalibration.PoseUpdates.All(static update => update.UsedPointCount == 5) &&
|
||||
rigCalibration.PoseUpdates.Any(static update => update.SourceId == "kiyo-basic" && update.DeltaMeters.Length() > 0.10f) &&
|
||||
rigCalibration.MeanRayDistanceMeters < 0.02 &&
|
||||
plan.Packets.Count == 1 &&
|
||||
plan.Packets[0].Backend == AquariumFieldBackendKind.DebugOverlay &&
|
||||
plan.DeferredRequests.Count == 0
|
||||
|
|
@ -882,6 +891,34 @@ static int RunLedSplineCalibrationSmoke()
|
|||
: 1;
|
||||
}
|
||||
|
||||
static IReadOnlyList<MimirLedPixelDetection> ProjectLedDetections(
|
||||
IReadOnlyList<MimirLedSplineScenePoint> scenePoints,
|
||||
MimirCameraPoseEstimate pose,
|
||||
int width,
|
||||
int height,
|
||||
double radiusPixels)
|
||||
{
|
||||
var output = new MimirLedPixelDetection[scenePoints.Count];
|
||||
var worldToCamera = Quaternion.Inverse(pose.CameraToWorldRotation);
|
||||
for (var index = 0; index < scenePoints.Count; index++)
|
||||
{
|
||||
var cameraSpace = Vector3.Transform(scenePoints[index].PositionMeters - pose.PositionMeters, worldToCamera);
|
||||
var z = Math.Max(1.0e-6f, cameraSpace.Z);
|
||||
var clipX = cameraSpace.X / z;
|
||||
var clipY = cameraSpace.Y / z;
|
||||
var imageX = (clipX + 1.0f) * 0.5f * width;
|
||||
var imageY = (1.0f - clipY) * 0.5f * height;
|
||||
output[index] = new MimirLedPixelDetection(
|
||||
scenePoints[index].LedIndex,
|
||||
imageX,
|
||||
imageY,
|
||||
radiusPixels,
|
||||
scenePoints[index].Confidence);
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
static int RunFensalirTextureLeaseSmoke()
|
||||
{
|
||||
var broker = new FakeFieldResourceBroker();
|
||||
|
|
|
|||
|
|
@ -1,5 +1,21 @@
|
|||
using System.Numerics;
|
||||
|
||||
namespace Mimir.Runtime.Synchronization;
|
||||
|
||||
public readonly record struct MimirLedPixelDetection(
|
||||
int LedIndex,
|
||||
double ImageX,
|
||||
double ImageY,
|
||||
double RadiusPixels,
|
||||
double Confidence);
|
||||
|
||||
public sealed record MimirLedSplineCurveFit(
|
||||
string SourceId,
|
||||
string ObservationKey,
|
||||
IReadOnlyList<MimirLedSplineObservationPoint> Points,
|
||||
double CurveLengthPixels,
|
||||
double Confidence);
|
||||
|
||||
public enum MimirVisualCalibrationResidualStatus
|
||||
{
|
||||
Empty,
|
||||
|
|
@ -35,6 +51,154 @@ public sealed record MimirVisualCalibrationResidualFrame(
|
|||
double Confidence,
|
||||
bool HasPoseUpdate);
|
||||
|
||||
public readonly record struct MimirLedSplineScenePoint(
|
||||
int LedIndex,
|
||||
Vector3 PositionMeters,
|
||||
double Confidence);
|
||||
|
||||
public sealed record MimirCameraPoseEstimate(
|
||||
string SourceId,
|
||||
Vector3 PositionMeters,
|
||||
Quaternion CameraToWorldRotation,
|
||||
double Confidence);
|
||||
|
||||
public sealed record MimirCameraPoseUpdate(
|
||||
string SourceId,
|
||||
Vector3 EstimatedPositionMeters,
|
||||
Vector3 DeltaMeters,
|
||||
int UsedPointCount,
|
||||
double MeanRayDistanceMeters,
|
||||
double Confidence);
|
||||
|
||||
public sealed record MimirCameraRigCalibrationFrame(
|
||||
string CalibrationId,
|
||||
string CandidateKey,
|
||||
string SplineId,
|
||||
IReadOnlyList<MimirCameraPoseUpdate> PoseUpdates,
|
||||
double MeanRayDistanceMeters,
|
||||
double Confidence,
|
||||
bool HasPoseUpdate);
|
||||
|
||||
public sealed class MimirLedSplineCurveSolver
|
||||
{
|
||||
public MimirLedSplineCurveFit SolveCameraCurve(
|
||||
string sourceId,
|
||||
string observationKey,
|
||||
int width,
|
||||
int height,
|
||||
long observedTimeNs,
|
||||
IEnumerable<MimirLedPixelDetection> detections)
|
||||
{
|
||||
var ordered = OrderDetections(detections).ToArray();
|
||||
if (ordered.Length == 0)
|
||||
{
|
||||
return new MimirLedSplineCurveFit(sourceId, observationKey, [], 0.0, 0.0);
|
||||
}
|
||||
|
||||
var distances = new double[ordered.Length];
|
||||
var length = 0.0;
|
||||
for (var index = 1; index < ordered.Length; index++)
|
||||
{
|
||||
var previous = ordered[index - 1];
|
||||
var current = ordered[index];
|
||||
var dx = current.ImageX - previous.ImageX;
|
||||
var dy = current.ImageY - previous.ImageY;
|
||||
length += Math.Sqrt(dx * dx + dy * dy);
|
||||
distances[index] = length;
|
||||
}
|
||||
|
||||
var points = new MimirLedSplineObservationPoint[ordered.Length];
|
||||
for (var index = 0; index < ordered.Length; index++)
|
||||
{
|
||||
var detection = ordered[index];
|
||||
var curveT = length <= 1.0e-9 ? 0.0 : distances[index] / length;
|
||||
points[index] = new MimirLedSplineObservationPoint(
|
||||
detection.LedIndex,
|
||||
curveT,
|
||||
detection.ImageX,
|
||||
detection.ImageY,
|
||||
PixelToClipX(detection.ImageX, width),
|
||||
PixelToClipY(detection.ImageY, height),
|
||||
detection.RadiusPixels,
|
||||
detection.Confidence);
|
||||
}
|
||||
|
||||
var confidence = ordered.Average(static detection => Clamp01(detection.Confidence));
|
||||
return new MimirLedSplineCurveFit(sourceId, observationKey, points, length, confidence);
|
||||
}
|
||||
|
||||
public MimirLedSplineCameraObservation ToCameraObservation(
|
||||
MimirLedSplineCurveFit fit,
|
||||
int width,
|
||||
int height,
|
||||
long observedTimeNs) =>
|
||||
new(
|
||||
fit.ObservationKey,
|
||||
fit.SourceId,
|
||||
width,
|
||||
height,
|
||||
observedTimeNs,
|
||||
fit.Points,
|
||||
fit.Confidence);
|
||||
|
||||
private static IReadOnlyList<MimirLedPixelDetection> OrderDetections(IEnumerable<MimirLedPixelDetection> detections)
|
||||
{
|
||||
var usable = detections
|
||||
.Where(static detection => detection.Confidence > 0.0)
|
||||
.ToArray();
|
||||
if (usable.Length == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
if (usable.All(static detection => detection.LedIndex >= 0))
|
||||
{
|
||||
return usable
|
||||
.OrderBy(static detection => detection.LedIndex)
|
||||
.ThenByDescending(static detection => detection.Confidence)
|
||||
.GroupBy(static detection => detection.LedIndex)
|
||||
.Select(static group => group.First())
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
var remaining = usable.ToList();
|
||||
var current = remaining
|
||||
.OrderBy(static detection => detection.ImageY)
|
||||
.ThenBy(static detection => detection.ImageX)
|
||||
.First();
|
||||
var ordered = new List<MimirLedPixelDetection>(usable.Length) { current };
|
||||
remaining.Remove(current);
|
||||
while (remaining.Count > 0)
|
||||
{
|
||||
var next = remaining
|
||||
.OrderBy(detection => SquaredDistance(current, detection))
|
||||
.ThenByDescending(static detection => detection.Confidence)
|
||||
.First();
|
||||
ordered.Add(next);
|
||||
remaining.Remove(next);
|
||||
current = next;
|
||||
}
|
||||
|
||||
return ordered;
|
||||
}
|
||||
|
||||
private static double SquaredDistance(MimirLedPixelDetection left, MimirLedPixelDetection right)
|
||||
{
|
||||
var dx = left.ImageX - right.ImageX;
|
||||
var dy = left.ImageY - right.ImageY;
|
||||
return dx * dx + dy * dy;
|
||||
}
|
||||
|
||||
private static double PixelToClipX(double imageX, int width) =>
|
||||
Math.Clamp(imageX / Math.Max(1.0, width) * 2.0 - 1.0, -1.0, 1.0);
|
||||
|
||||
private static double PixelToClipY(double imageY, int height) =>
|
||||
Math.Clamp(1.0 - imageY / Math.Max(1.0, height) * 2.0, -1.0, 1.0);
|
||||
|
||||
private static double Clamp01(double value) =>
|
||||
double.IsFinite(value) ? Math.Clamp(value, 0.0, 1.0) : 0.0;
|
||||
}
|
||||
|
||||
public sealed class MimirVisualCalibrationResidualSolver
|
||||
{
|
||||
public MimirVisualCalibrationResidualFrame EvaluateLedSpline(MimirLedSplineFieldCandidate candidate)
|
||||
|
|
@ -170,3 +334,179 @@ public sealed class MimirVisualCalibrationResidualSolver
|
|||
private static double Clamp01(double value) =>
|
||||
double.IsFinite(value) ? Math.Clamp(value, 0.0, 1.0) : 0.0;
|
||||
}
|
||||
|
||||
public sealed class MimirCameraRigCalibrationSolver
|
||||
{
|
||||
public MimirCameraRigCalibrationFrame FitCameraPositionsFromLedSpline(
|
||||
MimirLedSplineFieldCandidate candidate,
|
||||
IEnumerable<MimirLedSplineScenePoint> scenePoints,
|
||||
IEnumerable<MimirCameraPoseEstimate> currentPoses,
|
||||
double maximumDeltaMeters = 0.25)
|
||||
{
|
||||
var sceneByLed = scenePoints
|
||||
.Where(static point => point.LedIndex >= 0 && point.Confidence > 0.0)
|
||||
.GroupBy(static point => point.LedIndex)
|
||||
.ToDictionary(static group => group.Key, static group => group.OrderByDescending(point => point.Confidence).First());
|
||||
var poseBySource = currentPoses.ToDictionary(static pose => pose.SourceId, StringComparer.Ordinal);
|
||||
var updates = new List<MimirCameraPoseUpdate>();
|
||||
|
||||
foreach (var observation in candidate.CameraObservations)
|
||||
{
|
||||
if (!poseBySource.TryGetValue(observation.SourceId, out var pose) ||
|
||||
pose.Confidence <= 0.0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var rays = observation.Points
|
||||
.Where(point => sceneByLed.ContainsKey(point.LedIndex) && point.Confidence > 0.0)
|
||||
.Select(point => new RayPoint(
|
||||
Scene: sceneByLed[point.LedIndex].PositionMeters,
|
||||
Direction: RayDirection(point, pose.CameraToWorldRotation),
|
||||
Confidence: Math.Min(point.Confidence, sceneByLed[point.LedIndex].Confidence)))
|
||||
.ToArray();
|
||||
if (rays.Length < 3 || !TryEstimateCameraPosition(rays, out var estimatedPosition))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var delta = estimatedPosition - pose.PositionMeters;
|
||||
var clampedDelta = ClampLength(delta, (float)Math.Max(0.0, maximumDeltaMeters));
|
||||
var nextPosition = pose.PositionMeters + clampedDelta;
|
||||
var meanDistance = rays.Average(ray => DistancePointToRay(ray.Scene, nextPosition, ray.Direction));
|
||||
var confidence = Clamp01(candidate.Confidence) *
|
||||
Clamp01(pose.Confidence) *
|
||||
rays.Average(static ray => Clamp01(ray.Confidence)) *
|
||||
(1.0 / (1.0 + meanDistance));
|
||||
updates.Add(new MimirCameraPoseUpdate(
|
||||
observation.SourceId,
|
||||
nextPosition,
|
||||
clampedDelta,
|
||||
rays.Length,
|
||||
meanDistance,
|
||||
confidence));
|
||||
}
|
||||
|
||||
var meanRayDistance = updates.Count == 0
|
||||
? double.NaN
|
||||
: updates.Average(static update => update.MeanRayDistanceMeters);
|
||||
var frameConfidence = updates.Count == 0
|
||||
? 0.0
|
||||
: updates.Average(static update => update.Confidence);
|
||||
return new MimirCameraRigCalibrationFrame(
|
||||
candidate.CalibrationId,
|
||||
candidate.CandidateKey,
|
||||
candidate.SplineId,
|
||||
updates,
|
||||
meanRayDistance,
|
||||
frameConfidence,
|
||||
HasPoseUpdate: updates.Count > 0);
|
||||
}
|
||||
|
||||
private static Vector3 RayDirection(MimirLedSplineObservationPoint point, Quaternion cameraToWorldRotation)
|
||||
{
|
||||
var local = Vector3.Normalize(new Vector3((float)point.ClipX, (float)point.ClipY, 1.0f));
|
||||
return Vector3.Normalize(Vector3.Transform(local, cameraToWorldRotation));
|
||||
}
|
||||
|
||||
private static bool TryEstimateCameraPosition(IReadOnlyList<RayPoint> rays, out Vector3 position)
|
||||
{
|
||||
var a00 = 0.0;
|
||||
var a01 = 0.0;
|
||||
var a02 = 0.0;
|
||||
var a11 = 0.0;
|
||||
var a12 = 0.0;
|
||||
var a22 = 0.0;
|
||||
var b0 = 0.0;
|
||||
var b1 = 0.0;
|
||||
var b2 = 0.0;
|
||||
|
||||
foreach (var ray in rays)
|
||||
{
|
||||
var d = Vector3.Normalize(ray.Direction);
|
||||
var weight = Math.Max(1.0e-6, ray.Confidence);
|
||||
var m00 = 1.0 - d.X * d.X;
|
||||
var m01 = 0.0 - d.X * d.Y;
|
||||
var m02 = 0.0 - d.X * d.Z;
|
||||
var m11 = 1.0 - d.Y * d.Y;
|
||||
var m12 = 0.0 - d.Y * d.Z;
|
||||
var m22 = 1.0 - d.Z * d.Z;
|
||||
|
||||
a00 += weight * m00;
|
||||
a01 += weight * m01;
|
||||
a02 += weight * m02;
|
||||
a11 += weight * m11;
|
||||
a12 += weight * m12;
|
||||
a22 += weight * m22;
|
||||
b0 += weight * (m00 * ray.Scene.X + m01 * ray.Scene.Y + m02 * ray.Scene.Z);
|
||||
b1 += weight * (m01 * ray.Scene.X + m11 * ray.Scene.Y + m12 * ray.Scene.Z);
|
||||
b2 += weight * (m02 * ray.Scene.X + m12 * ray.Scene.Y + m22 * ray.Scene.Z);
|
||||
}
|
||||
|
||||
return TrySolveSymmetric3x3(a00, a01, a02, a11, a12, a22, b0, b1, b2, out position);
|
||||
}
|
||||
|
||||
private static bool TrySolveSymmetric3x3(
|
||||
double a00,
|
||||
double a01,
|
||||
double a02,
|
||||
double a11,
|
||||
double a12,
|
||||
double a22,
|
||||
double b0,
|
||||
double b1,
|
||||
double b2,
|
||||
out Vector3 solution)
|
||||
{
|
||||
var det =
|
||||
a00 * (a11 * a22 - a12 * a12) -
|
||||
a01 * (a01 * a22 - a12 * a02) +
|
||||
a02 * (a01 * a12 - a11 * a02);
|
||||
if (Math.Abs(det) < 1.0e-9)
|
||||
{
|
||||
solution = default;
|
||||
return false;
|
||||
}
|
||||
|
||||
var x =
|
||||
b0 * (a11 * a22 - a12 * a12) -
|
||||
a01 * (b1 * a22 - a12 * b2) +
|
||||
a02 * (b1 * a12 - a11 * b2);
|
||||
var y =
|
||||
a00 * (b1 * a22 - a12 * b2) -
|
||||
b0 * (a01 * a22 - a12 * a02) +
|
||||
a02 * (a01 * b2 - b1 * a02);
|
||||
var z =
|
||||
a00 * (a11 * b2 - b1 * a12) -
|
||||
a01 * (a01 * b2 - b1 * a02) +
|
||||
b0 * (a01 * a12 - a11 * a02);
|
||||
solution = new Vector3((float)(x / det), (float)(y / det), (float)(z / det));
|
||||
return true;
|
||||
}
|
||||
|
||||
private static double DistancePointToRay(Vector3 point, Vector3 origin, Vector3 direction)
|
||||
{
|
||||
var delta = point - origin;
|
||||
var projection = Vector3.Dot(delta, direction);
|
||||
var nearest = origin + direction * projection;
|
||||
return Vector3.Distance(point, nearest);
|
||||
}
|
||||
|
||||
private static Vector3 ClampLength(Vector3 value, float maximumLength)
|
||||
{
|
||||
if (maximumLength <= 0.0f)
|
||||
{
|
||||
return Vector3.Zero;
|
||||
}
|
||||
|
||||
var length = value.Length();
|
||||
return length <= maximumLength || length <= 1.0e-9f
|
||||
? value
|
||||
: value / length * maximumLength;
|
||||
}
|
||||
|
||||
private static double Clamp01(double value) =>
|
||||
double.IsFinite(value) ? Math.Clamp(value, 0.0, 1.0) : 0.0;
|
||||
|
||||
private readonly record struct RayPoint(Vector3 Scene, Vector3 Direction, double Confidence);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -252,3 +252,4 @@
|
|||
{"ts": "2026-05-31T01:20:43+01:00", "type": "verification", "status": "ok", "note": "Reservoir-history startup repaired for the Leap surface-claim path. Fensalir now compiles dedicated D3D12ReservoirHistoryUpdate.hlsl instead of the monolithic post shader for D3D12ReservoirHistoryUpdateCS; history-enabled live capture artifacts/runtime/mimir-leap-history-split-shader-visible.png completed without AQUARIUM_DISABLE_RESERVOIR_HISTORY, with 19,200 Leap surface splats, point-list rendering at 0, reservoir-update timing around 13 ms, and visible resolver output. Temporal/spatial reuse is intentionally not restored in the lean current-sample history shader yet."}
|
||||
{"ts": "2026-05-31T01:22:17+01:00", "type": "verification", "status": "ok", "note": "Eve dashboard broker cut: /eve/deck now routes provider-owned retained dashboard trees, /eve/deck/providers exposes switchboard/Mimir/StreamPixels manifests, and EveCanvas connects to /eve/deck with provider-card open-provider commands. Added typed CultMesh-compatible document shapes mimir.eve_dashboard_manifest.v1 and mimir.eve_dashboard_state.v1 while keeping live iPad transport WebSocket/TCP for LAN and SSH tunnel use. Smoke verified /health, provider catalog, WebSocket switchboard state, and open-provider transition to mimir.stream.layout."}
|
||||
{"ts": "2026-05-31T01:34:32+01:00", "type": "verification", "status": "ok", "note": "Eve dashboard broker gained native VoidBot provider voidbot.swarm. It reads VoidBot's existing .voidbot/status/swarm-state.json projection, exposes a provider manifest with CTB/agent-status/state-tree capabilities, renders upcoming turns as CTB cards, agent status cards, and selected Face state/detail nodes. Smoke verified /eve/deck/providers includes voidbot.swarm and open-provider switched to a VoidBot snapshot with paused swarm state, CTB cards, and agent detail."}
|
||||
{"ts": "2026-05-31T02:05:12+01:00", "type": "implementation", "status": "ok", "note": "Added the first LED-strip calibration machine: MimirLedSplineCurveSolver orders bright detections into per-sensor image/clipspace curves, and MimirCameraRigCalibrationSolver consumes stable LED indices plus a scene-space LED curve anchor to emit bounded camera translation updates by minimizing ray-to-curve distance. Rotation, focal length, and distortion remain explicit future calibration owners."}
|
||||
|
|
|
|||
|
|
@ -111,10 +111,11 @@ invariants:
|
|||
`DebugOverlay` calibration evidence. Exact cross-camera correspondence
|
||||
requires addressable/blink-coded LEDs or another identity source; uncoded
|
||||
identical LEDs remain useful curve constraints but cannot own camera pose.
|
||||
`MimirVisualCalibrationResidualSolver` now emits residual-only reports over
|
||||
coded/indexed LED observations; a future global residual/calibration owner
|
||||
must fit local frustums into coherent scene-space camera placement and be
|
||||
the only writer of camera intrinsics/extrinsics.
|
||||
`MimirLedSplineCurveSolver` now solves ordered per-sensor image/clipspace
|
||||
curves from bright detections. `MimirCameraRigCalibrationSolver` consumes
|
||||
stable LED indices plus a scene-space LED curve anchor and emits bounded
|
||||
camera translation updates by minimizing ray-to-curve distance. Rotation,
|
||||
focal length, and distortion are still future calibration cuts.
|
||||
- Fensalir packet payload handles are not authority. Live payload references
|
||||
must resolve through declared typed GPU/native resources with shape,
|
||||
residency, shader access, validity, version, and handle metadata before
|
||||
|
|
|
|||
Loading…
Reference in a new issue