Add OBS Spout render packet bridge

This commit is contained in:
Meta Vens 2026-05-18 16:08:14 +01:00
commit 63606980e3
13 changed files with 4021 additions and 11 deletions

View file

@ -36,13 +36,17 @@ flowchart TD
D --> F
F --> G["TrackCache"]
G --> H["PointCloud PLY"]
G --> I["Future brush/splat packets"]
G --> I["RenderFramePacket"]
I --> J["Aquarium GPU renderer"]
J --> K["Spout2 sender texture"]
K --> L["OBS Spout2 source"]
```
## Ownership
- `localcast.sensor_fusion.core` owns calibrated camera models, 2D observations, DLT triangulation, reprojection gating, confidence scoring, point-cloud export, and track cache expiry.
- `localcast.sensor_fusion.adapters` owns driver-facing frame ingress. Its first concrete adapter is an FFmpeg raw BGR reader for the PS3 Eye DirectShow path.
- `localcast.sensor_fusion.render_bridge` owns the render-frame ABI: cached point claims plus target dimensions, Spout sender name, source timestamp range, intended visual presentation time, and ambisonic/audio alignment time.
- `scripts/sensor_fusion.py` owns CLI composition for offline observation files.
- `config/sensor-fusion.example.json` owns the declarative rig shape: capture hints, intrinsics, extrinsics, latency, and fusion thresholds.
@ -57,8 +61,10 @@ sensor observation
-> calibrated world ray
-> triangulated marker claim
-> cached track
-> point cloud / brush packet / splat packet
-> renderer
-> render frame packet
-> Aquarium brush/splat packet
-> Spout sender texture
-> OBS source
```
The cache is not a bucket. `TrackCache` has a TTL and confidence update rule so the system can buffer and converge without pretending stale points are fresh. Later GPU buffers should mirror this shape: observation SoA, track SoA, brush packet SoA.
@ -74,17 +80,51 @@ The core can be tested without cameras:
Those tests matter because camera drivers are not unit-test dependencies. They are weather.
## OBS And Spout Boundary
OBS should see a named Spout sender, not a camera window, file tail, or desktop capture. Current references confirm the ordinary Windows path: the Off World Live OBS Spout2 plugin receives Spout shared textures, while the Spout2 SDK is the app-side texture sharing surface.
The intended production handoff is:
```text
LocalCastBridge TrackCache
-> RenderFramePacket JSON/binary stream
-> Aquarium Engine typed runtime state
-> GPU point/brush/splat buffer
-> D3D render target
-> Spout2 sender named "LocalCastBridge Point Cloud"
-> OBS Spout2 Capture source
```
`RenderFramePacket` is a scaffold ABI, not the final hot path. JSON is useful for inspection, tests, and handoff. Once Aquarium consumes the shape, switch the transport to a binary or shared-memory ring with the same fields. The invariant is the packet contract, not JSON.
## Latency Contract
Latency is allowed. Drift is not.
Each render frame carries:
- `source_time_min_ns` and `source_time_max_ns`: the observation window that produced the cached points.
- `present_time_ns`: when the visual frame wants to be presented after visual buffering.
- `audio_alignment_time_ns`: the corresponding ambisonic output time.
That lets the final stream compositor buffer point-cloud visuals until they align with the bounded audio-field cache. The renderer may show a slightly older world if the packet is coherent and labeled. It may not quietly mix fresh audio with stale visual geometry and call that sync.
## Next Cut
1. Add detector adapters that turn PS3 Eye frames into `Observation2D` marker detections.
2. Load real ChArUco intrinsics/extrinsics into `config/sensor-fusion.json`.
3. Record an observation JSONL/JSON run from the two PS3 Eyes.
4. Generate a live sparse PLY stream or debug preview.
5. Lower cached tracks into brush/splat packets once the point cloud is stable.
5. Feed `RenderFramePacket` into Aquarium Engine.
6. Add Aquarium-side GPU point/brush rendering to a D3D render target.
7. Publish the render target as a Spout2 sender and receive it in OBS.
8. Replace JSON packet transport with a binary/shared-memory ring once the contract is stable.
## References
- OpenCV multi-camera calibration and triangulation are the geometry spine. See `research/visual-spatial-map/summary.md`.
- Jacob and Haeb-Umbach 2015 motivates audio-visual coordinate alignment. See `research/visual-spatial-map/mirrors/krekovic-2015-audio-visual-geometry-calibration.pdf`.
- Kerbl et al. 2023 and RTG-SLAM motivate splats as render packets, not state ownership. See `research/visual-spatial-map/summary.md` and `research/gpu-fusion-splatting/summary.md`.
- Spout2 and the OBS Spout2 plugin are the Windows/OBS texture-sharing bridge. See `research/spout-obs-bridge/summary.md`.
- `E:\Projects\gamecult-site\GameCult\Blog\fractal-domains-cache-that-bites.md` supplies the cache discipline: semantic authority first, backend packet lowering second.