Quarantine visual cache adapter
This commit is contained in:
Meta Vens
parent
91c91ac63d
commit
b3e41fa2bd
11 changed files with 57 additions and 58 deletions
|
|
@ -51,7 +51,8 @@ shape removes independent per-kind storage from the live foundation.
|
|||
- `localcast.sensor_fusion.calibration_space` solves fixed-board ChArUco observations into camera poses in one common world frame.
|
||||
- `localcast.sensor_fusion.surface_features` matches cross-view features and triangulates calibrated surface tracks.
|
||||
- `localcast.sensor_fusion.render_bridge` emits render-frame packets with visual/audio timing metadata and Spout sender identity.
|
||||
- `localcast.sensor_fusion.cultcache_docs` stores live visual state as typed CultCache MessagePack documents.
|
||||
- `localcast.diagnostics.visual_cache` stores diagnostic visual state as typed
|
||||
CultCache MessagePack documents. It is not the production live boundary.
|
||||
- `localcast.diagnostics.spout_output` renders render-frame packets into a GPU texture and publishes it as a named Spout sender for OBS. This is diagnostic/migration code, not production runtime.
|
||||
- `scripts/diagnostic_live_sensor_fusion.py` writes fused render frames into `calibration/runs/visual-state.msgpack`. This is diagnostic/migration code, not production runtime.
|
||||
- The live visual producer writes a multi-LOD scene cache with source kind and priority. Real Leap frames are promoted as the highest-priority visual timing/spatial evidence; Leap fallback frames remain lower-priority diagnostics.
|
||||
|
|
@ -66,7 +67,8 @@ shape removes independent per-kind storage from the live foundation.
|
|||
- `scripts/diagnostic_stream_spout.py` runs the deadline Spout sender loop from the typed cache with typed and JSON heartbeat status. This is diagnostic/migration code, not production runtime.
|
||||
- The deadline Spout renderer has a named `kiyo-mid-deru` virtual camera preset: eye at the midpoint between the two Kiyo-class cameras and target on the co-streamer body volume. It also applies a renderer-owned point budget with pinned calibration/cross-modal constraints, stable high-confidence anchors, and frame-varying remainder samples so downstream TAA/supersampling can accumulate detail without forcing every source claim into every OBS frame. Its JSON heartbeat reports the prefix counts that survived the render budget.
|
||||
- `docs/obs-spout-streaming.md` documents OBS setup and the Aquarium replacement boundary.
|
||||
- `docs/typed-visual-state.md` documents the CultCache/CultNet visual boundary.
|
||||
- `docs/typed-visual-state.md` documents the diagnostic CultCache file shape
|
||||
and the CultNet visual schema target.
|
||||
- OBS synchronized program surface:
|
||||
- `scripts/setup_obs_synced_program.py` derives OBS-controllable stems from an aligned program audio timeline: host voice, co-streamer voice, ambient, transients, co-streamer loopback, and local loopback.
|
||||
- `scripts/capture_co_streamer_surfaces.py` captures neighbor Focusrite and neighbor loopback with local loopback ground truth, estimates the late remote-family offset, and writes aligned co-streamer surfaces for the stem packer.
|
||||
|
|
|
|||
|
|
@ -90,7 +90,9 @@ The sync heartbeat is written to `calibration/runs/av-sync-status.json`. It incl
|
|||
|
||||
- OBS receives a named Spout texture.
|
||||
- The Spout sink owns presentation only; it does not own camera calibration, fusion, or scene truth.
|
||||
- CultCache MessagePack documents are the live local state authority. CultNet document replication is the process/network boundary.
|
||||
- CultCache MessagePack documents are diagnostic local state. Production visual
|
||||
state crosses native/CultNet typed document boundaries; the Python file
|
||||
adapter is not live authority.
|
||||
- JSON render-frame polling is compatibility scaffolding only.
|
||||
- Latency is explicit in the packet timestamps. The renderer may buffer, but it may not silently erase the visual/audio alignment fields.
|
||||
- The neighbor SRT video feed is a timed artifact. OBS must not be left to eyeball-sync it against the Spout texture and AmbiX bed.
|
||||
|
|
|
|||
|
|
@ -72,7 +72,9 @@ flowchart TD
|
|||
- `localcast.sensor_fusion.active_illumination` owns deliberate light pulses as calibration telemetry. A Tuya bulb is an optional local-network actuator, not a rendering authority; its pulse timestamps are evidence for camera exposure/depth response.
|
||||
- `localcast.sensor_fusion.clap_calibration` owns deliberate clap detection: chirplet-synced audio transients nominate oracle timestamps, camera frame windows around that oracle look for motion derivatives, calibrated camera peaks triangulate the impact point, and only audio-plus-visual agreement becomes a clap calibration event.
|
||||
- `localcast.sensor_fusion.render_bridge` owns the in-process render-frame ABI: cached point claims plus target dimensions, Spout sender name, source timestamp range, intended visual presentation time, and ambisonic/audio alignment time.
|
||||
- `localcast.sensor_fusion.cultcache_docs` owns the typed CultCache document boundary for live visual state.
|
||||
- `localcast.diagnostics.visual_cache` owns the typed CultCache document shape
|
||||
used by diagnostic visual-state files. Production visual state should cross
|
||||
the native/CultNet boundary directly, not through this Python file adapter.
|
||||
- `localcast.diagnostics.spout_output` owns the deadline OBS publication sink: render-frame packet in, GPU texture plus Spout sender heartbeat out. It is diagnostic/migration code, not the production visual authority.
|
||||
- `scripts/sensor_fusion.py` owns CLI composition for offline observation files.
|
||||
- `scripts/diagnostic_stream_spout.py` owns the diagnostic Spout sender loop for OBS.
|
||||
|
|
@ -133,7 +135,10 @@ LocalCastBridge TrackCache
|
|||
-> OBS Spout2 Capture source
|
||||
```
|
||||
|
||||
`RenderFramePacket` is the in-process shape. The live boundary is now a typed CultCache MessagePack document, with CultNet `document_put` as the intended process/network API boundary. JSON render-frame files are compatibility scaffolding only.
|
||||
`RenderFramePacket` is the diagnostic Python in-process shape. The diagnostic
|
||||
file boundary is a typed CultCache MessagePack document. The production live
|
||||
boundary is native typed state plus CultNet `document_put`; JSON render-frame
|
||||
files are compatibility scaffolding only.
|
||||
|
||||
Clap calibration events use the same typed-state rule. A clap is not every sharp
|
||||
audio transient. The detector first finds a sharp broadband audio onset in the
|
||||
|
|
|
|||
|
|
@ -1,10 +1,14 @@
|
|||
# Typed Visual State
|
||||
# Diagnostic Typed Visual State
|
||||
|
||||
## Objective
|
||||
|
||||
Keep visual fusion state as typed CultCache documents and expose the boundary as CultNet document messages.
|
||||
Keep the diagnostic visual file shape as typed CultCache documents and expose
|
||||
the production boundary as native CultNet document messages.
|
||||
|
||||
JSON is no longer the authority for live visual state. The renderer consumes `calibration/runs/visual-state.msgpack`, which stores a `localcast.visual.render_frame` CultCache document.
|
||||
JSON is no longer the authority for visual state. The diagnostic renderer
|
||||
consumes `calibration/runs/visual-state.msgpack`, which stores a
|
||||
`localcast.visual.render_frame` CultCache document. Production visual state must
|
||||
not be owned by this Python file adapter.
|
||||
|
||||
## Current Mechanism
|
||||
|
||||
|
|
@ -23,9 +27,9 @@ flowchart TD
|
|||
H --> I["OBS Spout2 Capture"]
|
||||
```
|
||||
|
||||
Live files:
|
||||
Diagnostic files:
|
||||
|
||||
- `calibration/runs/visual-state.msgpack`: typed live visual frame state.
|
||||
- `calibration/runs/visual-state.msgpack`: typed diagnostic visual frame state.
|
||||
- `calibration/runs/visual-stream-status.msgpack`: typed stream status state.
|
||||
- `calibration/runs/stream-spout-status.json`: compatibility heartbeat for quick terminal checks.
|
||||
- `calibration/runs/av-sync-status.json`: visual/audio sync heartbeat from the Spout/Aquarium publisher.
|
||||
|
|
@ -64,7 +68,11 @@ CultNetDocumentPut(localcast.visual.render_frame)
|
|||
-> renderer frame source
|
||||
```
|
||||
|
||||
For the deadline rig, both producer and renderer share the local CultCache file. The boundary is still the document contract: a future camera process, Aquarium process, or remote sensor node should publish the same document through CultNet `document_put` messages rather than inventing another transport shape.
|
||||
For the deadline rig, both producer and renderer share the local CultCache file.
|
||||
That arrangement is diagnostic scaffolding. Aquarium, native capture, and remote
|
||||
sensor nodes should publish the same typed document through CultNet
|
||||
`document_put` messages rather than treating a Python-polled file as live
|
||||
authority.
|
||||
|
||||
The live visual frame may contain multiple claim families. `dense-rgb:*` claims
|
||||
are calibrated two-camera RGB surface samples from the debug CPU stereo
|
||||
|
|
|
|||
Loading…
Reference in a new issue