Mimir/docs/sensor-fusion-architecture.md
2026-05-31 02:12:17 +01:00

2.2 KiB

Sensor Fusion Architecture

Visual fusion belongs in Fensalir over current Mimir runtime buffers.

Live Flow

flowchart TD
    A["direct camera drivers"] --> B["Mimir.Runtime video buffers"]
    C["Leap stereo IR driver"] --> B
    B --> D["native reservoir handles"]
    D --> E["Fensalir GPU feature extraction"]
    E --> J["per-sensor LED curve solve"]
    E --> F["cross-view matching + flow"]
    J --> K["camera rig calibration solver"]
    K --> F
    F --> G["surface/material claims"]
    G --> H["brush/splat render budget"]
    H --> I["Spout2/program video"]

Invariants

  • Device timestamps beat arrival timestamps when available.
  • Leap is the first timing-camera candidate.
  • Process capture is a bridge edge, not the local six-camera foundation.
  • Fensalir owns GPU extraction, fusion, material fitting, and render budgeting.
  • Runtime buffers own retention and stream health, not scene reconstruction.
  • LED strips are active calibration evidence. Each camera publishes ordered bright-curve observations in its own local frustum/clipspace; only the global residual calibration owner may fit those frustums into scene-space camera intrinsics/extrinsics.
  • Stable LED index identity requires temporal/color/address coding or another correspondence source; uncoded identical lights are spline constraints, not metric depth authority.
  • Exposure/gain sweeps optimize recovered spline quality: detected LED count, curve coverage, spacing coherence, smoothness, exposure fitness, and saturation penalty. Bright clipped blobs are not good evidence.
  • The first camera-rig solver fits bounded camera translation updates from a scene-anchored LED curve by minimizing ray-to-curve distance. Rotation, intrinsics, and distortion are future owners.
  • Once camera frustums are coherent, every detected surface candidate can be resampled across the synchronized view set before Fensalir resolves it into spatiotemporal surface/splat claims.

Next Cut

Replace provisional Leap projection constants with calibrated intrinsics and add the global residual owner that consumes Leap surface errors, LED spline observations, and later PS3 Eye/Kiyo/Eve feature tracks through one calibration commit path.