Add common-space calibration and surface tracking

This commit is contained in:
Meta Vens 2026-05-18 21:31:09 +01:00
commit 905fc60dbd
10 changed files with 610 additions and 16 deletions

View file

@ -22,6 +22,8 @@
- `audio_field/` contains testable core modules for source buffering, bounded latency convergence, injectable capture/alignment/encoder/sink ports, and pipeline orchestration.
- `docs/audio-field.md` maps the audio pipeline and its invariants.
- Sensor-fusion render bridge:
- `localcast.sensor_fusion.calibration_space` solves fixed-board ChArUco observations into camera poses in one common world frame.
- `localcast.sensor_fusion.surface_features` matches cross-view features and triangulates calibrated surface tracks.
- `localcast.sensor_fusion.render_bridge` emits render-frame packets with visual/audio timing metadata and Spout sender identity.
- `localcast.sensor_fusion.cultcache_docs` stores live visual state as typed CultCache MessagePack documents.
- `localcast.sensor_fusion.spout_output` renders render-frame packets into a GPU texture and publishes it as a named Spout sender for OBS.
@ -56,6 +58,7 @@
9. Create local `config/audio-field.json`, confirm local Kiyo/PS Eye/Focusrite device matches, and confirm the neighbor Focusrite shotgun capture/transport path.
10. Replace placeholder mic/speaker geometry with measured world coordinates, then build the delay/SRO alignment stage that feeds the bounded field cache and emits aligned six-channel blocks before FOA encoding.
11. Move the render-frame consumer into Aquarium Engine so dense brush/splat rendering replaces the deadline OpenGL point sink behind the same OBS Spout boundary.
12. Replace synthetic live-fusion observations with PS3 Eye detector observations.
13. Keep the neighbor loopback leg running through the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present.
14. Let the co-streamer surface delay drive the shared presentation buffer horizon for audio stems, AmbiX, and remote video.
12. Capture fixed-board ChArUco observations for every camera, solve `config/sensor-fusion.json`, and verify cross-view feature triangulation with `scripts/triangulate_surface_features.py`.
13. Replace synthetic live-fusion observations with PS3 Eye detector observations and calibrated surface-feature tracks.
14. Keep the neighbor loopback leg running through the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present.
15. Let the co-streamer surface delay drive the shared presentation buffer horizon for audio stems, AmbiX, and remote video.