Add camera quality control and debug dense stereo

This commit is contained in:
Meta Vens 2026-05-18 20:12:00 +01:00
commit bd28491b42
8 changed files with 523 additions and 37 deletions

View file

@ -13,7 +13,9 @@ The rig currently has usable capture surfaces:
- LeapUVC raw IR through OpenCV for near-field evidence.
- ChArUco tooling for intrinsics capture, but no shared fusion state yet.
That is capture. Capture is not a world model.
That is capture. Capture is not a world model. The previous live producer also
painted camera colors onto guessed body and room surfaces. That was useful
deadline scaffolding, but it is not enough for detailed reconstruction.
## Invariants
@ -25,27 +27,41 @@ That is capture. Capture is not a world model.
## Intended Change
The live pipeline now has two measurement-quality paths: sparse marker fusion
for calibrated tracked objects and GPU-owned dense RGB stereo/flow for surface
detail. A CPU dense stereo matcher exists only as a debug reference and is
opt-in through `--cpu-dense-stereo`; production dense fusion may not route
through Python loops. The provisional Kiyo geometry remains a lie detector, not
truth, until `config/sensor-fusion.json` carries measured extrinsics.
The first pipeline slice adds these boundaries:
```mermaid
flowchart TD
A["Capture adapters"] --> B["FramePacket ring"]
B --> C["Detection adapter"]
C --> D["Observation2D"]
E["SensorRig calibration"] --> F["Fusion core"]
D --> F
F --> G["TrackCache"]
G --> H["PointCloud PLY"]
G --> I["CultCache RenderFrame document"]
I --> J["Spout sink / Aquarium GPU renderer"]
J --> K["Spout2 sender texture"]
K --> L["OBS Spout2 source"]
B --> C["Camera quality controller"]
C --> D["manual exposure/gain/focus commands"]
B --> E["Detection adapter"]
E --> F["Observation2D"]
B --> G["Dense stereo matcher"]
H["SensorRig calibration"] --> I["Fusion core"]
F --> I
I --> J["TrackCache"]
G --> K["RGB surface claims"]
J --> L["PointCloud PLY"]
J --> M["CultCache RenderFrame document"]
K --> M
M --> N["Spout sink / Aquarium GPU renderer"]
N --> O["Spout2 sender texture"]
O --> P["OBS Spout2 source"]
```
## Ownership
- `localcast.sensor_fusion.core` owns calibrated camera models, 2D observations, DLT triangulation, reprojection gating, confidence scoring, point-cloud export, and track cache expiry.
- `localcast.sensor_fusion.adapters` owns driver-facing frame ingress. Its first concrete adapter is an FFmpeg raw BGR reader for the PS3 Eye DirectShow path.
- `localcast.sensor_fusion.camera_control` owns measurement-quality policy: luminance, clipping, contrast, and sharpness in; normalized exposure/gain/focus commands out. Driver adapters own translating those commands to OpenCV, DirectShow, vendor tools, or no-op mocks.
- `localcast.sensor_fusion.dense_stereo` owns a debug CPU reference for dense calibrated RGB surface claims. It is useful for tests and shader parity, but the live million-splat path belongs to Aquarium/GPU compute.
- `localcast.sensor_fusion.render_bridge` owns the in-process render-frame ABI: cached point claims plus target dimensions, Spout sender name, source timestamp range, intended visual presentation time, and ambisonic/audio alignment time.
- `localcast.sensor_fusion.cultcache_docs` owns the typed CultCache document boundary for live visual state.
- `localcast.sensor_fusion.spout_output` owns the deadline OBS publication sink: render-frame packet in, GPU texture plus Spout sender heartbeat out.
@ -80,6 +96,8 @@ The core can be tested without cameras:
- timestamp windows reject stale pairs
- track cache expiry is deterministic
- raw-frame adapter can be tested from an in-memory byte stream
- adaptive camera control responds to overexposure without live hardware
- dense stereo turns shifted textured camera pairs into calibrated RGB surface claims
Those tests matter because camera drivers are not unit-test dependencies. They are weather.
@ -127,10 +145,10 @@ That lets the final stream compositor buffer point-cloud visuals until they alig
## Next Cut
1. Add detector adapters that turn PS3 Eye frames into `Observation2D` marker detections.
2. Load real ChArUco intrinsics/extrinsics into `config/sensor-fusion.json`.
3. Record an observation run from the two PS3 Eyes into typed CultCache docs.
4. Generate a live sparse PLY stream or debug preview.
1. Load real ChArUco intrinsics/extrinsics into `config/sensor-fusion.json`; the current dense Kiyo pair geometry is provisional and should not be mistaken for truth.
2. Move dense matching from CPU block search to GPU-resident stereo/flow so the million-sample target is not murdered by Python loops.
3. Add detector adapters that turn PS3 Eye frames into `Observation2D` marker detections.
4. Record an observation run from the two PS3 Eyes into typed CultCache docs.
5. Feed `RenderFramePacket` into Aquarium Engine.
6. Add Aquarium-side GPU point/brush rendering to a D3D render target.
7. Publish the render target as a Spout2 sender and receive it in OBS.

View file

@ -58,13 +58,16 @@ CultNetDocumentPut(localcast.visual.render_frame)
For the deadline rig, both producer and renderer share the local CultCache file. The boundary is still the document contract: a future camera process, Aquarium process, or remote sensor node should publish the same document through CultNet `document_put` messages rather than inventing another transport shape.
The live visual frame may contain multiple claim families. `room-rgb:*` claims
are camera-resolved room/background surface samples, `host-rgb:*` and
`deru-rgb:*` claims are RGB body/object surface samples, and `leap-motion:*`
claims are high-rate LeapUVC packed-channel motion samples. Leap packed maps are
split into explicit green, magenta, red, and blue channels before publication so
the downstream accumulator can use Leap as the strongest timing/motion witness
without pretending the packed image is one ordinary RGB camera.
The live visual frame may contain multiple claim families. `dense-rgb:*` claims
are calibrated two-camera RGB surface samples from the debug CPU stereo
reference or, in the intended production path, Aquarium/GPU dense stereo and
flow. `room-rgb:*` claims are camera-resolved room/background surface samples;
`host-rgb:*` and `deru-rgb:*` claims are fallback RGB body/object surface
samples; and `leap-motion:*` claims are high-rate LeapUVC packed-channel motion
samples. Leap packed maps are split into explicit green, magenta, red, and blue
channels before publication so the downstream accumulator can use Leap as the
strongest timing/motion witness without pretending the packed image is one
ordinary RGB camera.
LocalCastBridge does not own the final splat budget. It publishes dense typed
claims with stable keys and timestamps; Aquarium owns temporal accumulation,