Add camera quality control and debug dense stereo
This commit is contained in:
parent
41295fe45f
commit
bd28491b42
8 changed files with 523 additions and 37 deletions
|
|
@ -13,7 +13,9 @@ The rig currently has usable capture surfaces:
|
|||
- LeapUVC raw IR through OpenCV for near-field evidence.
|
||||
- ChArUco tooling for intrinsics capture, but no shared fusion state yet.
|
||||
|
||||
That is capture. Capture is not a world model.
|
||||
That is capture. Capture is not a world model. The previous live producer also
|
||||
painted camera colors onto guessed body and room surfaces. That was useful
|
||||
deadline scaffolding, but it is not enough for detailed reconstruction.
|
||||
|
||||
## Invariants
|
||||
|
||||
|
|
@ -25,27 +27,41 @@ That is capture. Capture is not a world model.
|
|||
|
||||
## Intended Change
|
||||
|
||||
The live pipeline now has two measurement-quality paths: sparse marker fusion
|
||||
for calibrated tracked objects and GPU-owned dense RGB stereo/flow for surface
|
||||
detail. A CPU dense stereo matcher exists only as a debug reference and is
|
||||
opt-in through `--cpu-dense-stereo`; production dense fusion may not route
|
||||
through Python loops. The provisional Kiyo geometry remains a lie detector, not
|
||||
truth, until `config/sensor-fusion.json` carries measured extrinsics.
|
||||
|
||||
The first pipeline slice adds these boundaries:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Capture adapters"] --> B["FramePacket ring"]
|
||||
B --> C["Detection adapter"]
|
||||
C --> D["Observation2D"]
|
||||
E["SensorRig calibration"] --> F["Fusion core"]
|
||||
D --> F
|
||||
F --> G["TrackCache"]
|
||||
G --> H["PointCloud PLY"]
|
||||
G --> I["CultCache RenderFrame document"]
|
||||
I --> J["Spout sink / Aquarium GPU renderer"]
|
||||
J --> K["Spout2 sender texture"]
|
||||
K --> L["OBS Spout2 source"]
|
||||
B --> C["Camera quality controller"]
|
||||
C --> D["manual exposure/gain/focus commands"]
|
||||
B --> E["Detection adapter"]
|
||||
E --> F["Observation2D"]
|
||||
B --> G["Dense stereo matcher"]
|
||||
H["SensorRig calibration"] --> I["Fusion core"]
|
||||
F --> I
|
||||
I --> J["TrackCache"]
|
||||
G --> K["RGB surface claims"]
|
||||
J --> L["PointCloud PLY"]
|
||||
J --> M["CultCache RenderFrame document"]
|
||||
K --> M
|
||||
M --> N["Spout sink / Aquarium GPU renderer"]
|
||||
N --> O["Spout2 sender texture"]
|
||||
O --> P["OBS Spout2 source"]
|
||||
```
|
||||
|
||||
## Ownership
|
||||
|
||||
- `localcast.sensor_fusion.core` owns calibrated camera models, 2D observations, DLT triangulation, reprojection gating, confidence scoring, point-cloud export, and track cache expiry.
|
||||
- `localcast.sensor_fusion.adapters` owns driver-facing frame ingress. Its first concrete adapter is an FFmpeg raw BGR reader for the PS3 Eye DirectShow path.
|
||||
- `localcast.sensor_fusion.camera_control` owns measurement-quality policy: luminance, clipping, contrast, and sharpness in; normalized exposure/gain/focus commands out. Driver adapters own translating those commands to OpenCV, DirectShow, vendor tools, or no-op mocks.
|
||||
- `localcast.sensor_fusion.dense_stereo` owns a debug CPU reference for dense calibrated RGB surface claims. It is useful for tests and shader parity, but the live million-splat path belongs to Aquarium/GPU compute.
|
||||
- `localcast.sensor_fusion.render_bridge` owns the in-process render-frame ABI: cached point claims plus target dimensions, Spout sender name, source timestamp range, intended visual presentation time, and ambisonic/audio alignment time.
|
||||
- `localcast.sensor_fusion.cultcache_docs` owns the typed CultCache document boundary for live visual state.
|
||||
- `localcast.sensor_fusion.spout_output` owns the deadline OBS publication sink: render-frame packet in, GPU texture plus Spout sender heartbeat out.
|
||||
|
|
@ -80,6 +96,8 @@ The core can be tested without cameras:
|
|||
- timestamp windows reject stale pairs
|
||||
- track cache expiry is deterministic
|
||||
- raw-frame adapter can be tested from an in-memory byte stream
|
||||
- adaptive camera control responds to overexposure without live hardware
|
||||
- dense stereo turns shifted textured camera pairs into calibrated RGB surface claims
|
||||
|
||||
Those tests matter because camera drivers are not unit-test dependencies. They are weather.
|
||||
|
||||
|
|
@ -127,10 +145,10 @@ That lets the final stream compositor buffer point-cloud visuals until they alig
|
|||
|
||||
## Next Cut
|
||||
|
||||
1. Add detector adapters that turn PS3 Eye frames into `Observation2D` marker detections.
|
||||
2. Load real ChArUco intrinsics/extrinsics into `config/sensor-fusion.json`.
|
||||
3. Record an observation run from the two PS3 Eyes into typed CultCache docs.
|
||||
4. Generate a live sparse PLY stream or debug preview.
|
||||
1. Load real ChArUco intrinsics/extrinsics into `config/sensor-fusion.json`; the current dense Kiyo pair geometry is provisional and should not be mistaken for truth.
|
||||
2. Move dense matching from CPU block search to GPU-resident stereo/flow so the million-sample target is not murdered by Python loops.
|
||||
3. Add detector adapters that turn PS3 Eye frames into `Observation2D` marker detections.
|
||||
4. Record an observation run from the two PS3 Eyes into typed CultCache docs.
|
||||
5. Feed `RenderFramePacket` into Aquarium Engine.
|
||||
6. Add Aquarium-side GPU point/brush rendering to a D3D render target.
|
||||
7. Publish the render target as a Spout2 sender and receive it in OBS.
|
||||
|
|
|
|||
|
|
@ -58,13 +58,16 @@ CultNetDocumentPut(localcast.visual.render_frame)
|
|||
|
||||
For the deadline rig, both producer and renderer share the local CultCache file. The boundary is still the document contract: a future camera process, Aquarium process, or remote sensor node should publish the same document through CultNet `document_put` messages rather than inventing another transport shape.
|
||||
|
||||
The live visual frame may contain multiple claim families. `room-rgb:*` claims
|
||||
are camera-resolved room/background surface samples, `host-rgb:*` and
|
||||
`deru-rgb:*` claims are RGB body/object surface samples, and `leap-motion:*`
|
||||
claims are high-rate LeapUVC packed-channel motion samples. Leap packed maps are
|
||||
split into explicit green, magenta, red, and blue channels before publication so
|
||||
the downstream accumulator can use Leap as the strongest timing/motion witness
|
||||
without pretending the packed image is one ordinary RGB camera.
|
||||
The live visual frame may contain multiple claim families. `dense-rgb:*` claims
|
||||
are calibrated two-camera RGB surface samples from the debug CPU stereo
|
||||
reference or, in the intended production path, Aquarium/GPU dense stereo and
|
||||
flow. `room-rgb:*` claims are camera-resolved room/background surface samples;
|
||||
`host-rgb:*` and `deru-rgb:*` claims are fallback RGB body/object surface
|
||||
samples; and `leap-motion:*` claims are high-rate LeapUVC packed-channel motion
|
||||
samples. Leap packed maps are split into explicit green, magenta, red, and blue
|
||||
channels before publication so the downstream accumulator can use Leap as the
|
||||
strongest timing/motion witness without pretending the packed image is one
|
||||
ordinary RGB camera.
|
||||
|
||||
LocalCastBridge does not own the final splat budget. It publishes dense typed
|
||||
claims with stable keys and timestamps; Aquarium owns temporal accumulation,
|
||||
|
|
|
|||
Loading…
Reference in a new issue