Add PS3 Eye sparse tracking evidence

This commit is contained in:
Meta Vens 2026-05-31 02:48:20 +01:00
commit 3c4f66f920
9 changed files with 594 additions and 3 deletions

View file

@ -414,6 +414,18 @@ a named invariant that the native runtime cannot protect yet.
generic KS exposure/gain writes and only recovered two LED components; both
PS3 Eye feeds currently score fixed-control frames because their native
wrapper does not yet expose gain/exposure.
PS3 Eyes now have the first sparse tracking surface:
`MimirSparseFeatureTracker` detects grid-suppressed Bayer/luma corner
features, links them frame-to-frame by bounded nearest-neighbor motion, and
emits sensor-local `MimirFeatureTrackCameraObservation` records with track
ids, image/clipspace coordinates, velocity, age, and confidence.
`MimirFeatureTrackFieldCandidate` lowers those observations as camera-sensor
Feature claims through `BuildFeatureTrackCandidateFrame`. The CPU tracker is
a live diagnostic/bootstrap owner; Fensalir D3D12 feature extraction remains
the production owner. `--ps3eye-tracking-live` proved both Eyes can produce
stable tracks at 320x240@187: with sensitive wide tracking defaults, each Eye
returned 160 live tracks from each Eye in a one-second pass, with 160 stable
tracks from `ps3-eye-0` and 156 stable tracks from `ps3-eye-1`.
Phase 6 program-output receipts are also structurally complete: Fensalir
publishes the final program surface through named shared D3D12 texture/fence
resources, the OBS plugin consumes that surface with an explicit D3D12-to-

View file

@ -40,6 +40,11 @@ flowchart TD
now: it opens direct sensor feeds, applies KS/UVC manual exposure/gain where
supported, scores actual luma frames, and reports the best setting per
source. PS3 Eye and Leap-specific controls still need native control cuts.
- PS3 Eyes own high-rate sparse tracking evidence before they own depth:
`MimirSparseFeatureTracker` turns live Bayer/luma frames into stable
sensor-local feature tracks, and `MimirFeatureTrackFieldCandidate` lowers
them as camera Feature claims. This CPU path is a diagnostic bootstrap; the
production hot path belongs in Fensalir D3D12 feature extraction.
- The first camera-rig solver fits bounded camera translation updates from a
scene-anchored LED curve by minimizing ray-to-curve distance. Rotation,
intrinsics, and distortion are future owners.

View file

@ -636,9 +636,18 @@ format, and the best spline setting is reported per source. On 2026-05-31 with
`-9`, gain `8` (`kiyo-pro-rgb` score `0.666`; `kiyo-basic-rgb` score `0.650`).
LeapUVC rejected generic KS exposure/gain writes and only found two LED
components; PS3 Eye feeds are still fixed-control until the WinUSB wrapper owns
gain/exposure. Once the frustum fit is coherent, detected surface candidates
can be sampled across every synchronized view before Fensalir resolves them
into 4D Gaussian/surface splat claims.
gain/exposure. PS3 Eyes now also have a sparse tracking bootstrap:
`MimirSparseFeatureTracker` detects grid-suppressed Bayer/luma corner features,
links them frame-to-frame, and emits `MimirFeatureTrackCameraObservation`
records with sensor-local image/clipspace position, velocity, age, and
confidence. `MimirFeatureTrackFieldCandidate` lowers those observations as
camera Feature claims through Fensalir field evidence. A one-second live pass
at 320x240@187 with sensitive defaults returned 160 live tracks from each Eye:
160 stable tracks from `ps3-eye-0` and 156 stable tracks from `ps3-eye-1`.
This is a CPU diagnostic owner; the hot production path remains Fensalir D3D12
feature extraction. Once the frustum fit is coherent, detected
surface candidates can be sampled across every synchronized view before
Fensalir resolves them into 4D Gaussian/surface splat claims.
The current teardown/migration map is
`docs/fensalir-rendering-rebuild-migration.md`, paired with Fensalir's

View file

@ -160,6 +160,16 @@ if (args.Any(arg => string.Equals(arg, "--led-spline-live-score", StringComparis
return RunLedSplineLiveScore(args);
}
if (args.Any(arg => string.Equals(arg, "--ps3eye-tracking-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunPs3EyeTrackingSmoke();
}
if (args.Any(arg => string.Equals(arg, "--ps3eye-tracking-live", StringComparison.OrdinalIgnoreCase)))
{
return RunPs3EyeTrackingLive(args);
}
if (args.Any(arg => string.Equals(arg, "--fensalir-texture-lease-smoke", StringComparison.OrdinalIgnoreCase)))
{
return RunFensalirTextureLeaseSmoke();
@ -1096,6 +1106,139 @@ static int RunLedSplineLiveScore(string[] args)
return anyCaptured ? 0 : 1;
}
static int RunPs3EyeTrackingSmoke()
{
const int width = 160;
const int height = 120;
var tracker = new MimirSparseFeatureTracker(new MimirSparseFeatureTrackerOptions(
MaxFeatures: 32,
CellSizePixels: 10,
SearchRadiusPixels: 12,
MinimumTrackAgeFrames: 3,
MinimumCornerScore: 0.08,
MinimumLuma: 0.12));
MimirSparseFeatureTrackerFrame latest = default!;
for (var frameIndex = 0; frameIndex < 8; frameIndex++)
{
var luma = new byte[width * height];
for (var marker = 0; marker < 8; marker++)
{
var x = 24 + marker * 14 + frameIndex * 2;
var y = 20 + marker * 9 + frameIndex;
StampBlob(luma, width, height, x, y, radius: 2, peak: 230);
}
latest = tracker.Update(
"ps3-eye-0",
width,
height,
luma,
1_000_000_000L + frameIndex * 5_350_000L);
}
var observation = tracker.ToCameraObservation(latest);
var candidate = new MimirFeatureTrackFieldCandidate(
"ps3-eye-synthetic-tracks",
"visual-calibration",
"Mimir.BufferSmoke",
[observation],
latest.StableTrackCount,
latest.MeanTrackAgeFrames,
latest.MeanSpeedPixelsPerSecond,
latest.Confidence,
latest.ObservedTimeNs);
var evidence = new MimirFensalirFieldLowering().BuildFeatureTrackCandidateFrame([candidate]);
var validation = AquariumFieldEvidenceValidator.Validate(evidence);
Console.WriteLine(
$"ps3eye-tracking-smoke frames={latest.FrameCount} tracks={latest.Tracks.Count} stable={latest.StableTrackCount} meanAge={latest.MeanTrackAgeFrames:0.000} meanSpeed={latest.MeanSpeedPixelsPerSecond:0.000} confidence={latest.Confidence:0.000} claims={evidence.Claims.Count} planned={evidence.BackendPackets.Count} errors={validation.HasErrors}");
return latest.StableTrackCount >= 8 &&
latest.Confidence > 0.50 &&
evidence.Claims.Count == 1 &&
!validation.HasErrors
? 0
: 1;
}
static int RunPs3EyeTrackingLive(string[] args)
{
var library = ParseStringOption(args, "--native-library", Path.GetFullPath("native/camera_capture/build/Release/mimir_ps3eye_capture.dll"));
var seconds = Math.Max(0.10, ParseDoubleOption(args, "--seconds", 1.0));
var width = ParseIntOption(args, "--width", 320);
var height = ParseIntOption(args, "--height", 240);
var fps = ParseIntOption(args, "--fps", 187);
var maxFeatures = Math.Max(8, ParseIntOption(args, "--max-features", 160));
var cameraFilter = ParseIntOption(args, "--camera-index", -1);
var cameras = cameraFilter >= 0 ? [cameraFilter] : new[] { 0, 1 };
var anyCaptured = false;
foreach (var cameraIndex in cameras)
{
var sourceId = $"ps3-eye-{cameraIndex}";
try
{
using var driver = new MimirPs3EyeVideoCaptureDriver(new MimirPs3EyeVideoCaptureDriverOptions(
library,
sourceId,
cameraIndex,
width,
height,
fps));
var tracker = new MimirSparseFeatureTracker(new MimirSparseFeatureTrackerOptions(
MaxFeatures: maxFeatures,
CellSizePixels: ParseIntOption(args, "--cell-size", 12),
SearchRadiusPixels: ParseIntOption(args, "--search-radius", 24),
MinimumTrackAgeFrames: ParseIntOption(args, "--min-age", 3),
MinimumCornerScore: ParseDoubleOption(args, "--min-corner", 0.05),
MinimumLuma: ParseDoubleOption(args, "--min-luma", 0.03)));
var deadline = Stopwatch.GetTimestamp() + (long)(seconds * Stopwatch.Frequency);
MimirSparseFeatureTrackerFrame latest = default!;
var frames = 0;
while (Stopwatch.GetTimestamp() < deadline)
{
if (!driver.TryCapture(out var frame, out var data))
{
Thread.Sleep(1);
continue;
}
if (data.IsEmpty || !TryExtractLiveLuma(frame, data.Span, out var luma, out var lumaWidth, out var lumaHeight))
{
continue;
}
latest = tracker.Update(sourceId, lumaWidth, lumaHeight, luma, frame.DeviceTimestampNs);
frames++;
}
anyCaptured |= frames > 0;
var observation = frames == 0 ? null : tracker.ToCameraObservation(latest);
var candidate = observation == null
? null
: new MimirFeatureTrackFieldCandidate(
$"{sourceId}:live-tracks:{latest.FrameCount}",
"visual-calibration-live",
"Mimir.BufferSmoke",
[observation],
latest.StableTrackCount,
latest.MeanTrackAgeFrames,
latest.MeanSpeedPixelsPerSecond,
latest.Confidence,
latest.ObservedTimeNs);
var evidence = candidate == null
? AquariumFieldEvidenceFrame.Empty
: new MimirFensalirFieldLowering().BuildFeatureTrackCandidateFrame([candidate]);
Console.WriteLine(
$"ps3eye-tracking-live source={sourceId} frames={frames} tracks={latest?.Tracks.Count ?? 0} stable={latest?.StableTrackCount ?? 0} meanAge={latest?.MeanTrackAgeFrames ?? 0.0:0.000} meanSpeed={latest?.MeanSpeedPixelsPerSecond ?? 0.0:0.000} confidence={latest?.Confidence ?? 0.0:0.000} claims={evidence.Claims.Count} format={MimirVideoPixelFormat.Bayer8} size={width}x{height}");
}
catch (Exception ex)
{
Console.WriteLine($"ps3eye-tracking-live source={sourceId} status=failed error=\"{ex.Message.Replace("\"", "'", StringComparison.Ordinal)}\"");
}
}
return anyCaptured ? 0 : 1;
}
static IReadOnlyList<LiveLedSplineExposureGainSetting> LiveLedSplineExposureGainSettings() =>
[
new("exp-10_gain-0", -10, 0),

View file

@ -193,6 +193,40 @@ public sealed record MimirLedSplineFieldCandidate(
double Confidence,
long ObservedTimeNs);
public readonly record struct MimirFeatureTrackPoint(
int TrackId,
double ImageX,
double ImageY,
double ClipX,
double ClipY,
double VelocityXPerSecond,
double VelocityYPerSecond,
int AgeFrames,
double Confidence);
public sealed record MimirFeatureTrackCameraObservation(
string ObservationKey,
string SourceId,
int Width,
int Height,
long ObservedTimeNs,
IReadOnlyList<MimirFeatureTrackPoint> Tracks,
int FrameCount,
double MeanTrackAgeFrames,
double MeanSpeedPixelsPerSecond,
double Confidence);
public sealed record MimirFeatureTrackFieldCandidate(
string CandidateKey,
string CalibrationId,
string ProducerKey,
IReadOnlyList<MimirFeatureTrackCameraObservation> CameraObservations,
int StableTrackCount,
double MeanTrackAgeFrames,
double MeanSpeedPixelsPerSecond,
double Confidence,
long ObservedTimeNs);
public readonly record struct MimirSurfaceAxes(
string X,
string Y,

View file

@ -214,6 +214,40 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
};
}
public AquariumFieldEvidenceFrame BuildFeatureTrackCandidateFrame(
IEnumerable<MimirFeatureTrackFieldCandidate> trackCandidates)
{
var domains = new List<AquariumFieldDomain>();
var claims = new List<AquariumFieldClaim>();
var candidates = new List<AquariumFieldCandidate>();
var seenDomains = new HashSet<string>(StringComparer.Ordinal);
foreach (var trackCandidate in trackCandidates)
{
if (string.IsNullOrWhiteSpace(trackCandidate.CandidateKey) ||
trackCandidate.CameraObservations.Count == 0)
{
continue;
}
var domain = DomainForFeatureTrackCandidate(trackCandidate);
AddDomain(domains, seenDomains, domain);
var claim = ClaimForFeatureTrackCandidate(trackCandidate, domain.DomainKey);
claims.Add(claim);
candidates.Add(CandidateForClaim(claim, AquariumFieldGuide.Valid(claim.Confidence)));
}
return new AquariumFieldEvidenceFrame
{
Domains = domains,
Claims = claims,
Candidates = candidates,
AccumulationWindowSeconds = (float)options.AccumulationWindowSeconds,
PresentationDelaySeconds = (float)options.PresentationDelaySeconds
};
}
public AquariumFieldEvidenceFrame BuildLeapPackedStereoDepthCandidateFrame(
IEnumerable<MimirRollingStreamWindow> windows,
IEnumerable<AquariumFieldResourceDeclaration> inputResources)
@ -621,6 +655,27 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
"Mimir.Runtime");
}
private static AquariumFieldDomain DomainForFeatureTrackCandidate(MimirFeatureTrackFieldCandidate candidate)
{
var tracks = candidate.CameraObservations.SelectMany(static observation => observation.Tracks).ToArray();
var maxWidth = Math.Max(1, candidate.CameraObservations.Max(static observation => observation.Width));
var maxHeight = Math.Max(1, candidate.CameraObservations.Max(static observation => observation.Height));
var minX = tracks.Length == 0 ? 0.0f : Math.Clamp((float)tracks.Min(static track => track.ImageX), 0.0f, maxWidth);
var minY = tracks.Length == 0 ? 0.0f : Math.Clamp((float)tracks.Min(static track => track.ImageY), 0.0f, maxHeight);
var maxX = tracks.Length == 0 ? maxWidth : Math.Clamp((float)tracks.Max(static track => track.ImageX), minX, maxWidth);
var maxY = tracks.Length == 0 ? maxHeight : Math.Clamp((float)tracks.Max(static track => track.ImageY), minY, maxHeight);
return new AquariumFieldDomain(
DomainKeyForFeatureTrackCandidate(candidate),
"",
AquariumFieldDomainKind.CameraSensor,
Matrix4x4.Identity,
Matrix4x4.Identity,
new Vector3(minX, minY, 0.0f),
new Vector3(maxX, maxY, Math.Max(1, candidate.CameraObservations.Count)),
Vector3.Zero,
"Mimir.Runtime");
}
private static AquariumFieldDomain DomainForStereoDepthCandidate(MimirStereoDepthFieldCandidate candidate)
{
var width = Math.Max(1, candidate.Width);
@ -794,6 +849,32 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
Confidence: confidence);
}
private AquariumFieldClaim ClaimForFeatureTrackCandidate(
MimirFeatureTrackFieldCandidate trackCandidate,
string domainKey)
{
var confidence = Clamp01((float)trackCandidate.Confidence);
var represented = Math.Max(1, trackCandidate.CameraObservations.Sum(static observation => observation.Tracks.Count));
var proposal = new AquariumFieldProposalPolicy(
AquariumFieldProposalKind.SensorObservation,
SourcePdf: 1.0f,
TargetContribution: confidence,
RepresentedCandidateCount: represented,
Seed: StableSeed(trackCandidate.CandidateKey));
return new AquariumFieldClaim(
ClaimKey: $"feature-tracks:{trackCandidate.CalibrationId}:{trackCandidate.CandidateKey}",
DomainKey: domainKey,
ProducerKey: trackCandidate.ProducerKey,
Layer: AquariumFieldLayer.Form,
Encoding: AquariumFieldEncoding.Feature,
Support: SupportForFeatureTrackCandidate(trackCandidate),
Proposal: proposal,
PayloadHandle: $"{trackCandidate.CalibrationId}:stable={trackCandidate.StableTrackCount}",
ObservedTimeNs: trackCandidate.ObservedTimeNs,
Confidence: confidence);
}
private AquariumFieldClaim ClaimForStereoDepthCandidate(
MimirStereoDepthFieldCandidate depthCandidate,
string domainKey)
@ -1008,6 +1089,26 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
TemporalUncertainty: splineCandidate.HasTemporalCode ? 0.0f : 1.0f / 187.0f);
}
private AquariumFieldSupport SupportForFeatureTrackCandidate(MimirFeatureTrackFieldCandidate trackCandidate)
{
var tracks = trackCandidate.CameraObservations.SelectMany(static observation => observation.Tracks).ToArray();
var radiusPixels = tracks.Length == 0
? 1.0f
: Math.Max(1.0f, (float)(tracks.Average(static track => Math.Max(1, track.AgeFrames)) * 0.5));
var meanSpeed = Math.Max(0.0f, (float)trackCandidate.MeanSpeedPixelsPerSecond);
var confidence = tracks.Length == 0
? 0.0f
: (float)tracks.Average(static track => Math.Clamp(track.Confidence, 0.0, 1.0));
return new AquariumFieldSupport(
Center: new Vector3(0.0f, 0.0f, Math.Max(1, trackCandidate.CameraObservations.Count)),
Radius: new Vector3(radiusPixels, radiusPixels, Math.Max(1.0f, trackCandidate.CameraObservations.Count * 0.5f)),
LocalFrame: Matrix4x4.Identity,
ConservativeRadius: Math.Max(radiusPixels, meanSpeed / 187.0f),
ProjectedError: 1.0f - confidence,
Curvature: 0.0f,
TemporalUncertainty: 1.0f / 187.0f);
}
private AquariumFieldSupport SupportForStereoDepthCandidate(MimirStereoDepthFieldCandidate depthCandidate)
{
var width = Math.Max(1.0f, depthCandidate.Width);
@ -1321,6 +1422,9 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
private static string DomainKeyForLedSplineCandidate(MimirLedSplineFieldCandidate candidate) =>
$"mimir:led-spline:{candidate.CalibrationId}:{candidate.SplineId}:{candidate.CandidateKey}";
private static string DomainKeyForFeatureTrackCandidate(MimirFeatureTrackFieldCandidate candidate) =>
$"mimir:feature-tracks:{candidate.CalibrationId}:{candidate.CandidateKey}";
private static string DomainKeyForStereoDepthCandidate(MimirStereoDepthFieldCandidate candidate) =>
DomainKeyForStereoDepthCandidate(candidate.CalibrationId, candidate.CameraPairId, candidate.CandidateKey);

View file

@ -0,0 +1,274 @@
namespace Mimir.Runtime.Synchronization;
public sealed record MimirSparseFeatureTrackerOptions(
int MaxFeatures = 96,
int CellSizePixels = 12,
int SearchRadiusPixels = 18,
int MinimumTrackAgeFrames = 3,
double MinimumCornerScore = 0.10,
double MinimumLuma = 0.08);
public sealed record MimirSparseFeatureTrackerFrame(
string SourceId,
int Width,
int Height,
long ObservedTimeNs,
int FrameCount,
IReadOnlyList<MimirFeatureTrackPoint> Tracks,
int StableTrackCount,
double MeanTrackAgeFrames,
double MeanSpeedPixelsPerSecond,
double Confidence);
public sealed class MimirSparseFeatureTracker(MimirSparseFeatureTrackerOptions? options = null)
{
private readonly MimirSparseFeatureTrackerOptions options = options ?? new();
private readonly List<TrackedFeature> tracks = [];
private int nextTrackId = 1;
private int frameCount;
private long previousTimeNs;
public MimirSparseFeatureTrackerFrame Update(
string sourceId,
int width,
int height,
ReadOnlySpan<byte> luma,
long observedTimeNs)
{
if (width <= 2 || height <= 2 || luma.Length < width * height)
{
return Empty(sourceId, width, height, observedTimeNs);
}
frameCount++;
var dtSeconds = previousTimeNs <= 0 || observedTimeNs <= previousTimeNs
? 1.0 / 187.0
: Math.Max(1.0e-6, (observedTimeNs - previousTimeNs) / 1_000_000_000.0);
previousTimeNs = observedTimeNs;
var detections = DetectFeatures(width, height, luma);
MatchDetections(detections, dtSeconds);
PruneTracks();
var output = tracks
.Where(track => track.MissedFrames == 0)
.OrderByDescending(track => track.AgeFrames)
.ThenByDescending(track => track.Confidence)
.Take(options.MaxFeatures)
.Select(track => new MimirFeatureTrackPoint(
track.TrackId,
track.X,
track.Y,
PixelToClipX(track.X, width),
PixelToClipY(track.Y, height),
track.VelocityXPerSecond,
track.VelocityYPerSecond,
track.AgeFrames,
track.Confidence))
.ToArray();
var stable = output.Count(point => point.AgeFrames >= options.MinimumTrackAgeFrames);
var meanAge = output.Length == 0 ? 0.0 : output.Average(static point => point.AgeFrames);
var meanSpeed = output.Length == 0
? 0.0
: output.Average(static point => Math.Sqrt(
point.VelocityXPerSecond * point.VelocityXPerSecond +
point.VelocityYPerSecond * point.VelocityYPerSecond));
var stability = output.Length == 0 ? 0.0 : stable / (double)Math.Min(options.MaxFeatures, output.Length);
var density = Math.Clamp(output.Length / (double)Math.Max(1, options.MaxFeatures), 0.0, 1.0);
var confidence = Math.Clamp(0.55 * stability + 0.45 * density, 0.0, 1.0);
return new MimirSparseFeatureTrackerFrame(
sourceId,
width,
height,
observedTimeNs,
frameCount,
output,
stable,
meanAge,
meanSpeed,
confidence);
}
public MimirFeatureTrackCameraObservation ToCameraObservation(MimirSparseFeatureTrackerFrame frame) =>
new(
$"{frame.SourceId}:feature-tracks:{frame.FrameCount}",
frame.SourceId,
frame.Width,
frame.Height,
frame.ObservedTimeNs,
frame.Tracks,
frame.FrameCount,
frame.MeanTrackAgeFrames,
frame.MeanSpeedPixelsPerSecond,
frame.Confidence);
private MimirSparseFeatureTrackerFrame Empty(string sourceId, int width, int height, long observedTimeNs) =>
new(sourceId, width, height, observedTimeNs, frameCount, [], 0, 0.0, 0.0, 0.0);
private IReadOnlyList<FeatureDetection> DetectFeatures(int width, int height, ReadOnlySpan<byte> luma)
{
var cell = Math.Max(4, options.CellSizePixels);
var gridWidth = Math.Max(1, (width + cell - 1) / cell);
var gridHeight = Math.Max(1, (height + cell - 1) / cell);
var bestByCell = new FeatureDetection?[gridWidth * gridHeight];
var minimumLuma = Math.Clamp(options.MinimumLuma, 0.0, 1.0) * 255.0;
var minimumScore = Math.Clamp(options.MinimumCornerScore, 0.0, 1.0) * 255.0;
for (var y = 1; y < height - 1; y++)
{
var row = y * width;
for (var x = 1; x < width - 1; x++)
{
var center = luma[row + x];
if (center < minimumLuma)
{
continue;
}
var gx = Math.Abs(luma[row + x + 1] - luma[row + x - 1]);
var gy = Math.Abs(luma[row + width + x] - luma[row - width + x]);
var diagonalA = Math.Abs(luma[row + width + x + 1] - luma[row - width + x - 1]);
var diagonalB = Math.Abs(luma[row + width + x - 1] - luma[row - width + x + 1]);
var score = Math.Min(Math.Max(gx, diagonalA), Math.Max(gy, diagonalB));
if (score < minimumScore)
{
continue;
}
var cellX = Math.Min(gridWidth - 1, x / cell);
var cellY = Math.Min(gridHeight - 1, y / cell);
var cellIndex = cellY * gridWidth + cellX;
var candidate = new FeatureDetection(x, y, score / 255.0);
if (bestByCell[cellIndex] is not { } existing || candidate.Score > existing.Score)
{
bestByCell[cellIndex] = candidate;
}
}
}
return bestByCell
.Where(static detection => detection.HasValue)
.Select(static detection => detection!.Value)
.OrderByDescending(static detection => detection.Score)
.Take(options.MaxFeatures)
.ToArray();
}
private void MatchDetections(IReadOnlyList<FeatureDetection> detections, double dtSeconds)
{
var matchedTracks = new HashSet<int>();
var matchedDetections = new bool[detections.Count];
var radiusSquared = options.SearchRadiusPixels * options.SearchRadiusPixels;
for (var detectionIndex = 0; detectionIndex < detections.Count; detectionIndex++)
{
var detection = detections[detectionIndex];
TrackedFeature? bestTrack = null;
var bestDistance = double.MaxValue;
foreach (var track in tracks)
{
if (matchedTracks.Contains(track.TrackId))
{
continue;
}
var dx = detection.X - track.X;
var dy = detection.Y - track.Y;
var distance = dx * dx + dy * dy;
if (distance <= radiusSquared && distance < bestDistance)
{
bestDistance = distance;
bestTrack = track;
}
}
if (bestTrack == null)
{
continue;
}
var velocityX = (detection.X - bestTrack.X) / dtSeconds;
var velocityY = (detection.Y - bestTrack.Y) / dtSeconds;
bestTrack.X = detection.X;
bestTrack.Y = detection.Y;
bestTrack.VelocityXPerSecond = velocityX;
bestTrack.VelocityYPerSecond = velocityY;
bestTrack.Confidence = Math.Clamp(0.7 * bestTrack.Confidence + 0.3 * detection.Score, 0.0, 1.0);
bestTrack.AgeFrames++;
bestTrack.MissedFrames = 0;
matchedTracks.Add(bestTrack.TrackId);
matchedDetections[detectionIndex] = true;
}
foreach (var track in tracks)
{
if (!matchedTracks.Contains(track.TrackId))
{
track.MissedFrames++;
}
}
for (var index = 0; index < detections.Count; index++)
{
if (matchedDetections[index])
{
continue;
}
var detection = detections[index];
tracks.Add(new TrackedFeature(
nextTrackId++,
detection.X,
detection.Y,
0.0,
0.0,
1,
0,
detection.Score));
}
}
private void PruneTracks()
{
tracks.RemoveAll(static track => track.MissedFrames > 5);
if (tracks.Count <= options.MaxFeatures * 2)
{
return;
}
var survivors = tracks
.OrderBy(static track => track.MissedFrames)
.ThenByDescending(static track => track.AgeFrames)
.ThenByDescending(static track => track.Confidence)
.Take(options.MaxFeatures * 2)
.ToHashSet();
tracks.RemoveAll(track => !survivors.Contains(track));
}
private static double PixelToClipX(double imageX, int width) =>
Math.Clamp(imageX / Math.Max(1.0, width) * 2.0 - 1.0, -1.0, 1.0);
private static double PixelToClipY(double imageY, int height) =>
Math.Clamp(1.0 - imageY / Math.Max(1.0, height) * 2.0, -1.0, 1.0);
private readonly record struct FeatureDetection(double X, double Y, double Score);
private sealed class TrackedFeature(
int trackId,
double x,
double y,
double velocityXPerSecond,
double velocityYPerSecond,
int ageFrames,
int missedFrames,
double confidence)
{
public int TrackId { get; } = trackId;
public double X { get; set; } = x;
public double Y { get; set; } = y;
public double VelocityXPerSecond { get; set; } = velocityXPerSecond;
public double VelocityYPerSecond { get; set; } = velocityYPerSecond;
public int AgeFrames { get; set; } = ageFrames;
public int MissedFrames { get; set; } = missedFrames;
public double Confidence { get; set; } = confidence;
}
}

View file

@ -257,3 +257,4 @@
{"ts": "2026-05-31T02:16:04+01:00", "type": "implementation", "status": "ok", "note": "Added MimirLedSplineFrameAnalyzer as the frame-level bridge for physical camera sweeps: it extracts bright luma components, solves a per-sensor LED curve, and feeds MimirLedSplineQualityScorer so saturated blobs can be detected but rejected as poor calibration evidence."}
{"ts": "2026-05-31T03:05:00+01:00", "type": "verification", "status": "ok", "note": "Live LED spline sweep now runs on actual direct sensor feeds. `dotnet run --project src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj -- --led-spline-live-score --setting-seconds 0.25 --settle-ms 100 --expected-leds 38 --minimum-luma 0.55` selected exposure -9/gain 8 for `kiyo-pro-rgb` (score 0.666, 41 detections, usable) and `kiyo-basic-rgb` (score 0.650, 38 detections, usable). LeapUVC rejected generic KS exposure/gain writes and only recovered two LED components; both PS3 Eye feeds scored 0 at current fixed controls because their native wrapper does not yet expose gain/exposure."}
{"ts": "2026-05-31T02:45:05+01:00", "type": "verification", "status": "ok", "note": "Eve dashboard now renders voidbot.swarm as a native VoidBot-style cockpit with CTB avatar URLs, selected Face/status pane, state tree, and detail panel; device build/install passed and broker smoke reported 8 CTB avatar URLs."}
{"ts": "2026-05-31T03:22:00+01:00", "type": "verification", "status": "ok", "note": "PS3 Eye sparse tracking bootstrap is live. `--ps3eye-tracking-smoke` produced 13 synthetic tracks, 12 stable tracks, and one Feature claim. The final default `--ps3eye-tracking-live --seconds 1.0` captured both Eyes at 320x240@187 and returned 160 tracks / 160 stable tracks from `ps3-eye-0` and 160 tracks / 156 stable tracks from `ps3-eye-1`. This is CPU diagnostic tracking; production feature extraction remains a Fensalir D3D12 owner."}

View file

@ -129,6 +129,15 @@ invariants:
for both Kiyo Pro and Kiyo Basic. LeapUVC rejected the generic KS
exposure/gain writes and PS3 Eye capture remains fixed-control until its
native wrapper owns gain/exposure.
- PS3 Eyes now have a first sparse tracking surface instead of only cadence
proof. `MimirSparseFeatureTracker` detects Bayer/luma corner features,
links them into sensor-local tracks with clipspace position, velocity, age,
and confidence, and `MimirFeatureTrackFieldCandidate` lowers them as
camera Feature claims. `Mimir.BufferSmoke --ps3eye-tracking-live` proved a
one-second 320x240@187 run can return 160 live tracks from each Eye: 160
stable tracks from `ps3-eye-0` and 156 stable tracks from `ps3-eye-1` with
sensitive defaults. This is CPU diagnostic/bootstrap work;
Fensalir D3D12 feature extraction remains the production owner.
- Fensalir packet payload handles are not authority. Live payload references
must resolve through declared typed GPU/native resources with shape,
residency, shader access, validity, version, and handle metadata before