GameCult/Mimir
Watch
9
0
Fork
You've already forked Mimir
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Pin cross-modal evidence through adaptive LOD

Browse source
This commit is contained in:
Meta Vens 2026-05-19 00:44:21 +01:00
commit c18bcaa646
8 changed files with 196 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/implementation-plan.md
View file

@ -31,9 +31,9 @@
- The live visual producer writes a multi-LOD scene cache with source kind and priority. Real Leap frames are promoted as the highest-priority visual timing/spatial evidence; Leap fallback frames remain lower-priority diagnostics.
- Live clap calibration is wired into the visual producer. It keeps a rolling frame window from the Kiyo pair plus Leap, reads the live spatial audio frame for transient candidates, publishes `clap-events.msgpack`, injects clap calibration markers into the render frame, and writes clap evidence into the LOD cache. Kiyo stereo owns the current rough 3D solve; Leap owns the best visual timing witness until its geometric model is calibrated.
- Clap peaks now update a per-camera clock sync model. The point-cloud builder can sample camera history buffers at `oracle_time - camera_offset` instead of using whichever frame arrived last.
- `localcast.sensor_fusion.chirp_pose` converts live phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints. `scripts/live_sensor_fusion.py` injects those constraints as `camera-chirp:*` render points and `chirp-camera-pose` LOD evidence.
- `localcast.sensor_fusion.chirp_pose` converts live phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints. It can load camera-mic and speaker geometry from the audio-field profile, with the example profile used as the live fallback until measured local geometry exists. `scripts/live_sensor_fusion.py` injects those constraints as `camera-chirp:*` render points and `chirp-camera-pose` LOD evidence.
- `scripts/stream_spout.py` runs the deadline Spout sender loop from the typed cache with typed and JSON heartbeat status.
- The deadline Spout renderer has a named `kiyo-mid-deru` virtual camera preset: eye at the midpoint between the two Kiyo-class cameras and target on the co-streamer body volume. It also applies a renderer-owned point budget with stable high-confidence anchors plus frame-varying remainder samples so downstream TAA/supersampling can accumulate detail without forcing every source claim into every OBS frame.
- The deadline Spout renderer has a named `kiyo-mid-deru` virtual camera preset: eye at the midpoint between the two Kiyo-class cameras and target on the co-streamer body volume. It also applies a renderer-owned point budget with pinned calibration/cross-modal constraints, stable high-confidence anchors, and frame-varying remainder samples so downstream TAA/supersampling can accumulate detail without forcing every source claim into every OBS frame.
- `docs/obs-spout-streaming.md` documents OBS setup and the Aquarium replacement boundary.
- `docs/typed-visual-state.md` documents the CultCache/CultNet visual boundary.
- OBS synchronized program surface:
@ -75,7 +75,7 @@
11. Move the render-frame consumer into Aquarium Engine so dense brush/splat rendering replaces the deadline OpenGL point sink behind the same OBS Spout boundary.
12. Capture fixed-board ChArUco observations for every camera, solve `config/sensor-fusion.json`, and verify cross-view feature triangulation with `scripts/triangulate_surface_features.py`.
13. Replace synthetic live-fusion observations with PS3 Eye detector observations and calibrated surface-feature tracks.
14. Replace the placeholder speaker/camera-mic geometry used by `chirp_pose` with measured rig coordinates, then let the common-space optimizer consume range residuals instead of only visualizing them.
14. Replace the example speaker/camera-mic geometry used by `chirp_pose` with measured local `config/audio-field.json` coordinates, then let the common-space optimizer consume range residuals instead of only visualizing them.
15. Calibrate Leap as a geometric camera in the shared rig so the timing authority can also contribute direct 3D rays instead of only visual timing peaks.
16. Fix the live Leap capture backend so actual Leap frames, not fallback frames, populate `leap-ground-truth` cells in the multi-LOD cache.
17. Keep the neighbor loopback leg running through the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present.

8
localcast/sensor_fusion/__init__.py
View file

@ -71,9 +71,13 @@ from .chirp_pose import (
CameraChirpPoseConstraint,
CameraMicGeometry,
SpeakerGeometry,
camera_mic_geometry_from_audio_profile,
camera_sensor_id_for_microphone,
constraints_from_phase_sources,
default_camera_mic_geometry,
default_speaker_geometry,
phase_source_id_for_microphone,
speaker_geometry_from_audio_profile,
)
from .cultcache_docs import (
CLAP_EVENTS_KEY,
@ -138,6 +142,8 @@ __all__ = [
"CameraChirpPoseConstraint",
"CameraMicGeometry",
"SpeakerGeometry",
"camera_mic_geometry_from_audio_profile",
"camera_sensor_id_for_microphone",
"ClapCalibrationEvent",
"ClapDetectorConfig",
"TimestampedFrame",
@ -178,6 +184,8 @@ __all__ = [
"constraints_from_phase_sources",
"default_camera_mic_geometry",
"default_speaker_geometry",
"phase_source_id_for_microphone",
"speaker_geometry_from_audio_profile",
"remote_video_artifact_for_present_time",
"put_live_clap_events",
"put_live_render_frame",

54
localcast/sensor_fusion/chirp_pose.py
View file

@ -1,6 +1,8 @@
from __future__ import annotations
from dataclasses import dataclass
import json
from pathlib import Path
from typing import Mapping, Sequence
import numpy as np
@ -53,6 +55,58 @@ def default_speaker_geometry() -> tuple[SpeakerGeometry, ...]:
)
def camera_mic_geometry_from_audio_profile(path: Path) -> tuple[CameraMicGeometry, ...]:
data = json.loads(Path(path).read_text(encoding="utf-8"))
rows: list[CameraMicGeometry] = []
for mic in data.get("microphones", []):
source_id = phase_source_id_for_microphone(mic)
sensor_id = camera_sensor_id_for_microphone(mic)
position = mic.get("positionMeters")
if source_id is None or sensor_id is None or position is None:
continue
rows.append(CameraMicGeometry(sensor_id, source_id, np.asarray(position, dtype=np.float64)))
return tuple(rows)
def speaker_geometry_from_audio_profile(path: Path) -> tuple[SpeakerGeometry, ...]:
data = json.loads(Path(path).read_text(encoding="utf-8"))
rows: list[SpeakerGeometry] = []
for speaker in data.get("speakers", []):
position = speaker.get("positionMeters")
if position is None:
continue
rows.append(SpeakerGeometry(str(speaker.get("id", "")), np.asarray(position, dtype=np.float64)))
return tuple(row for row in rows if row.speaker_id)
def phase_source_id_for_microphone(mic: Mapping[str, object]) -> str | None:
explicit = mic.get("phaseSourceId") or mic.get("sourceId")
if explicit:
return str(explicit)
mic_id = str(mic.get("id", ""))
known = {
"mic_kiyo_left": "kiyo-0",
"mic_kiyo_right": "kiyo-1",
"mic_ps_eye_left": "ps-eye-0",
"mic_ps_eye_right": "ps-eye-1",
}
return known.get(mic_id)
def camera_sensor_id_for_microphone(mic: Mapping[str, object]) -> str | None:
explicit = mic.get("cameraSensorId") or mic.get("sensorId")
if explicit:
return str(explicit)
mic_id = str(mic.get("id", ""))
known = {
"mic_kiyo_left": "kiyo-primary",
"mic_kiyo_right": "kiyo-secondary",
"mic_ps_eye_left": "ps3eye-left",
"mic_ps_eye_right": "ps3eye-right",
}
return known.get(mic_id)
def constraints_from_phase_sources(
sources: Sequence[Mapping[str, object]],
*,

48
localcast/sensor_fusion/spout_output.py
View file

@ -232,20 +232,33 @@ def frame_with_point_budget(
) -> RenderFramePacket:
if max_points <= 0 or len(frame.points) <= max_points:
return frame
priority_count = max(1, int(max_points * 0.55))
temporal_count = max(0, max_points - priority_count)
pinned = sorted(
(point for point in frame.points if is_pinned_render_point(point)),
key=lambda point: (-render_priority(point, camera_preset_name), point.stable_key),
)[:max_points]
pinned_keys = {point.stable_key for point in pinned}
remaining_budget = max(0, max_points - len(pinned))
if remaining_budget <= 0:
points = pinned
return _frame_with_points(frame, points)
priority_count = max(1, int(remaining_budget * 0.55))
temporal_count = max(0, remaining_budget - priority_count)
priority = sorted(
frame.points,
(point for point in frame.points if point.stable_key not in pinned_keys),
key=lambda point: (-render_priority(point, camera_preset_name), stable_hash(point.stable_key, 0)),
)[:priority_count]
selected_keys = {point.stable_key for point in priority}
selected_keys = pinned_keys | {point.stable_key for point in priority}
remainder = [point for point in frame.points if point.stable_key not in selected_keys]
temporal = sorted(
remainder,
key=lambda point: stable_hash(point.stable_key, frame.frame_id) / max(0.05, render_priority(point, camera_preset_name)),
)[:temporal_count]
points = priority + temporal
points = pinned + priority + temporal
points = sorted(points, key=lambda point: point.stable_key)
return _frame_with_points(frame, points)
def _frame_with_points(frame: RenderFramePacket, points: list[RenderPointPacket]) -> RenderFramePacket:
return RenderFramePacket(
schema=frame.schema,
frame_id=frame.frame_id,
@ -261,6 +274,19 @@ def frame_with_point_budget(
)
def is_pinned_render_point(point: RenderPointPacket) -> bool:
return point.stable_key.startswith(
(
"camera-chirp:",
"camera-sync:",
"clap-calibration:",
"clap-ray:",
"clap-timing:",
"audio-event-",
)
)
def stable_hash(key: str, salt: int) -> int:
digest = hashlib.blake2s(f"{salt}:{key}".encode("utf-8"), digest_size=8).digest()
return int.from_bytes(digest, "little")
@ -270,12 +296,18 @@ def render_priority(point: RenderPointPacket, camera_preset_name: str) -> float:
key = point.stable_key
confidence = max(0.0, min(1.0, float(point.confidence)))
semantic = 1.0
if key.startswith("dense-rgb:"):
if key.startswith("clap-calibration:"):
semantic = 3.2
elif key.startswith(("camera-sync:", "camera-chirp:")):
semantic = 2.6
elif key.startswith(("clap-ray:", "clap-timing:")):
semantic = 2.2
elif key.startswith("dense-rgb:"):
semantic = 2.0
elif key.startswith("leap-motion:"):
semantic = 1.45
elif key.startswith("room-rgb:"):
semantic = 1.15
elif key.startswith("leap-motion:"):
semantic = 1.0
elif key.startswith("stochastic:"):
semantic = 0.36
elif key.startswith("audio-event-"):

3
notes/current-system-map.md
View file

@ -106,7 +106,8 @@ Ownership:
- `scripts/live_sensor_fusion.py` currently owns the live producer and writes `localcast.visual.render_frame` into CultCache.
- The live producer now keeps a rolling RGB/Leap frame window for clap calibration. Kiyo stereo motion gives the first rough 3D clap position; Leap participates as the high-precision visual timing witness until a calibrated Leap geometric model joins the rig.
- Clap peaks update a per-camera clock sync model. Camera history sampling uses `oracle_time - camera_offset`, so the point builder can reconstruct a shared instant instead of mixing latest frames from different clock domains.
- `localcast.sensor_fusion.chirp_pose` now turns phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints. The live producer publishes them as `camera-chirp:*` render points and `chirp-camera-pose` LOD evidence. Audio range/timing evidence augments visual calibration; it does not own image geometry.
- `localcast.sensor_fusion.chirp_pose` now turns phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints. It loads camera-mic and speaker geometry from the audio-field profile when available, falling back to the example profile/defaults. The live producer publishes those constraints as `camera-chirp:*` render points and `chirp-camera-pose` LOD evidence. Audio range/timing evidence augments visual calibration; it does not own image geometry.
- The Spout point-budget policy pins calibration and cross-modal constraints before spending the remaining adaptive LOD budget on stable high-priority surface claims and frame-rotated samples.
- The live producer also writes a multi-LOD scene cache with source kind and priority. Real Leap frames are the highest-priority timing/spatial evidence; Leap fallback frames are tagged as fallback and do not become ground truth.
- `scripts/stream_spout.py` consumes typed CultCache state and publishes Spout. Its presentation camera defaults to `kiyo-mid-deru`: the virtual eye sits halfway between the two Kiyo-class cameras and aims at the co-streamer body target. Its render budget is temporal: high-confidence anchors stay stable while the remaining samples rotate by frame id for TAA/supersampling reconciliation.
- CultNet document replication is the intended API boundary for other producers/consumers.

23
scripts/live_sensor_fusion.py
View file

@ -27,6 +27,7 @@ from localcast.sensor_fusion import (
TimestampedFrame,
TimestampedFrameHistory,
ClapDetectorConfig,
camera_mic_geometry_from_audio_profile,
constraints_from_phase_sources,
detect_clap_events,
dense_stereo_points,
@ -40,6 +41,7 @@ from localcast.sensor_fusion import (
put_live_render_frame,
lower_points_to_render_frame,
stochastic_transient_matches,
speaker_geometry_from_audio_profile,
)
from localcast.sensor_fusion.camera_control import AdaptiveCameraController, OpenCvCameraSettingPort
from audio_field.cultcache_audio import frame_to_numpy, get_live_audio_phase_field, get_live_spatial_audio_frame
@ -658,8 +660,15 @@ def sync_status_position(sensor_id: str) -> np.ndarray:
class LiveChirpPoseMapper:
def __init__(self, *, phase_cache: Path) -> None:
def __init__(self, *, phase_cache: Path, audio_profile: Path | None = None) -> None:
self.phase_cache = phase_cache
self.camera_mics = None
self.speakers = None
if audio_profile is not None and audio_profile.exists():
loaded_mics = camera_mic_geometry_from_audio_profile(audio_profile)
loaded_speakers = speaker_geometry_from_audio_profile(audio_profile)
self.camera_mics = loaded_mics or None
self.speakers = loaded_speakers or None
self.last_frame_id: int | None = None
self.constraints: tuple[CameraChirpPoseConstraint, ...] = ()
@ -674,6 +683,8 @@ class LiveChirpPoseMapper:
self.constraints = constraints_from_phase_sources(
field.sources,
audio_time_ns=field.audio_time_ns,
camera_mics=self.camera_mics,
speakers=self.speakers,
)
return self._constraint_points(field.audio_time_ns)
@ -1085,6 +1096,7 @@ def main() -> None:
parser.add_argument("--lod-cache", default=str(ROOT / "calibration" / "runs" / "visual-lod-cache.json"))
parser.add_argument("--audio-cache", default=str(ROOT / "calibration" / "runs" / "audio-state.msgpack"))
parser.add_argument("--phase-cache", default=str(ROOT / "calibration" / "runs" / "audio-phase-field.msgpack"))
parser.add_argument("--audio-profile", default=str(ROOT / "config" / "audio-field.json"))
parser.add_argument("--clap-cache", default=str(ROOT / "calibration" / "runs" / "clap-events.msgpack"))
parser.add_argument("--no-clap-calibration", action="store_true")
parser.add_argument("--no-chirp-pose", action="store_true")
@ -1134,7 +1146,14 @@ def main() -> None:
camera_width=args.rgb_width,
camera_height=args.rgb_height,
)
chirp_pose_mapper = None if args.no_chirp_pose else LiveChirpPoseMapper(phase_cache=Path(args.phase_cache))
audio_profile = Path(args.audio_profile)
if not audio_profile.exists() and audio_profile.name == "audio-field.json":
example_profile = audio_profile.with_name("audio-field.example.json")
audio_profile = example_profile if example_profile.exists() else audio_profile
chirp_pose_mapper = None if args.no_chirp_pose else LiveChirpPoseMapper(
phase_cache=Path(args.phase_cache),
audio_profile=audio_profile if audio_profile.exists() else None,
)
try:
while True:
now = time.monotonic()

8
state/map.yaml
View file

@ -174,8 +174,12 @@ research:
`localcast.sensor_fusion.chirp_pose` converts live phase-field delay
meaning for Kiyo/PS Eye microphone sources into camera-body range
constraints, which the live producer emits as `camera-chirp:*` render
points and `chirp-camera-pose` LOD evidence. These constraints augment
visual calibration; they do not own image geometry.
points and `chirp-camera-pose` LOD evidence. It can load geometry from
the audio-field profile, falling back to the example/default geometry
until measured local coordinates exist. These constraints augment visual
calibration; they do not own image geometry. The Spout point-budget policy
pins calibration and cross-modal constraints before spending adaptive LOD
budget on stable high-priority surfaces and frame-rotated samples.
typed_visual_state:
summary: docs/typed-visual-state.md
key_decision: >

62
tests/test_sensor_fusion.py
View file

@ -29,6 +29,8 @@ from localcast.sensor_fusion import (
TimestampedFrameHistory,
TrackCache,
VirtualCamera,
camera_mic_geometry_from_audio_profile,
camera_sensor_id_for_microphone,
constraints_from_phase_sources,
detect_clap_events,
frame_to_vertex_array,
@ -44,7 +46,9 @@ from localcast.sensor_fusion import (
put_live_clap_events,
put_live_render_frame,
remote_video_artifact_for_present_time,
phase_source_id_for_microphone,
solve_common_space_from_fixed_board,
speaker_geometry_from_audio_profile,
SurfaceFeatureObservation,
triangulate_surface_tracks,
)
@ -318,6 +322,36 @@ class SensorFusionTests(unittest.TestCase):
self.assertIs(frame, limited)
def test_render_point_budget_keeps_cross_modal_calibration_constraints(self):
pinned = (
RenderPointPacket("camera-chirp:kiyo-primary:local-speaker-left:body", np.array([0.0, 0.0, 1.0]), 0.02, (1, 1, 0, 1), 0.15, 1),
RenderPointPacket("camera-sync:kiyo-primary", np.array([0.0, 0.0, 1.0]), 0.02, (0, 1, 1, 1), 0.15, 1),
RenderPointPacket("clap-calibration:clap:test", np.array([0.0, 0.0, 1.0]), 0.02, (0, 1, 1, 1), 0.15, 1),
)
noisy = tuple(
RenderPointPacket(f"room-rgb:noisy:{index}", np.array([0.48, 0.28, 1.2]), 0.02, (1, 1, 1, 1), 1.0, 1)
for index in range(20)
)
frame = RenderFramePacket(
schema="localcast.sensor_fusion.render_frame.v1",
frame_id=3,
created_monotonic_ns=1,
source_time_min_ns=1,
source_time_max_ns=1,
present_time_ns=1,
audio_alignment_time_ns=1,
spout_sender_name="test",
target_width=320,
target_height=180,
points=pinned + noisy,
)
limited = frame_with_point_budget(frame, 6, "kiyo-mid-deru")
keys = {point.stable_key for point in limited.points}
self.assertEqual(6, len(limited.points))
self.assertTrue(all(point.stable_key in keys for point in pinned))
def test_rasterizer_uses_gaussian_envelope_not_solid_ellipse(self):
frame = RenderFramePacket(
schema="localcast.sensor_fusion.render_frame.v1",
@ -577,6 +611,34 @@ class SensorFusionTests(unittest.TestCase):
self.assertAlmostEqual(0.12, constraint.range_residual_m)
self.assertEqual(12_345, constraint.timestamp_ns)
def test_chirp_pose_geometry_loads_from_audio_profile(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "audio-field.json"
path.write_text(
"""
{
"microphones": [
{"id": "mic_kiyo_left", "positionMeters": [-0.4, 0.2, 1.3]},
{"id": "ignored_focusrite", "positionMeters": [0.0, 0.0, 1.0]},
{"id": "custom", "phaseSourceId": "ps-eye-1", "cameraSensorId": "ps3eye-right", "positionMeters": [0.7, -0.1, 1.2]}
],
"speakers": [
{"id": "speaker_left", "positionMeters": [-0.6, 1.0, 1.1]}
]
}
""",
encoding="utf-8",
)
mics = camera_mic_geometry_from_audio_profile(path)
speakers = speaker_geometry_from_audio_profile(path)
self.assertEqual(("kiyo-0", "ps-eye-1"), tuple(mic.source_id for mic in mics))
self.assertEqual(("kiyo-primary", "ps3eye-right"), tuple(mic.sensor_id for mic in mics))
self.assertEqual("speaker_left", speakers[0].speaker_id)
self.assertEqual("kiyo-0", phase_source_id_for_microphone({"id": "mic_kiyo_left"}))
self.assertEqual("kiyo-primary", camera_sensor_id_for_microphone({"id": "mic_kiyo_left"}))
def test_audio_source_events_overlay_against_audio_alignment_time(self):
frame = RenderFramePacket(
schema="localcast.sensor_fusion.render_frame.v1",