diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 77a968c..646fbb4 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -32,6 +32,9 @@ - Live clap calibration is wired into the visual producer. It keeps a rolling frame window from the Kiyo pair plus Leap, reads the live spatial audio frame for transient candidates, publishes `clap-events.msgpack`, injects clap calibration markers into the render frame, and writes clap evidence into the LOD cache. Kiyo stereo owns the current rough 3D solve; Leap owns the best visual timing witness until its geometric model is calibrated. - Clap peaks now update a per-camera clock sync model. The point-cloud builder can sample camera history buffers at `oracle_time - camera_offset` instead of using whichever frame arrived last. - The live visual producer now uses an explicit five-second spatiotemporal reservoir. RGB/Leap frame history expires from the newest shared sample, the multi-LOD cache only accepts evidence inside that same window, and `RenderFramePacket.source_time_min_ns/source_time_max_ns` declares the reservoir slice being rendered. + - Multi-LOD cells now carry first-pass material estimates: weighted albedo plus roughness/metallic hints derived from source evidence. Aquarium should treat this as relightable material evidence to refine, not as the final BRDF solve. + - The Spout/audio overlay boundary clamps stale visual packets to the current audio reservoir edge. If camera geometry ages out, it is dropped instead of being broadcast out of sync. + - Live fallback-only RGB/Leap mode exists for deadline operation when OpenCV/MSMF/DirectShow reads block longer than the five-second reservoir. It keeps the reservoir and OBS output alive while real camera capture is moved to nonblocking ingress. - `localcast.sensor_fusion.chirp_pose` converts live phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints and per-camera pose-correction estimates. It can load camera-mic and speaker geometry from the audio-field profile, with the example profile used as the live fallback until measured local geometry exists. `scripts/live_sensor_fusion.py` injects those constraints as `camera-chirp:*` and `camera-pose-correction:*` render points plus `chirp-camera-pose` LOD evidence. - `scripts/stream_spout.py` runs the deadline Spout sender loop from the typed cache with typed and JSON heartbeat status. - The deadline Spout renderer has a named `kiyo-mid-deru` virtual camera preset: eye at the midpoint between the two Kiyo-class cameras and target on the co-streamer body volume. It also applies a renderer-owned point budget with pinned calibration/cross-modal constraints, stable high-confidence anchors, and frame-varying remainder samples so downstream TAA/supersampling can accumulate detail without forcing every source claim into every OBS frame. Its JSON heartbeat reports the prefix counts that survived the render budget. @@ -78,7 +81,7 @@ 10. Replace placeholder mic/speaker geometry with measured world coordinates, then build the delay/SRO alignment stage that feeds the bounded field cache and emits aligned six-channel blocks before FOA encoding. 11. Move the render-frame consumer into Aquarium Engine so dense brush/splat rendering replaces the deadline OpenGL point sink behind the same OBS Spout boundary. 12. Capture fixed-board ChArUco observations for every camera, solve `config/sensor-fusion.json`, and verify cross-view feature triangulation with `scripts/triangulate_surface_features.py`. -13. Replace synthetic live-fusion observations with PS3 Eye detector observations and calibrated surface-feature tracks. +13. Replace synchronous OpenCV camera reads in the hot producer with nonblocking camera ingress that continuously fills the five-second reservoir, then promote real PS3 Eye/Kiyo/Leap samples over fallback-only mode. 14. Replace the example speaker/camera-mic geometry used by `chirp_pose` with measured local `config/audio-field.json` coordinates, then let the common-space optimizer consume range residuals instead of only visualizing them. 15. Calibrate Leap as a geometric camera in the shared rig so the timing authority can also contribute direct 3D rays instead of only visual timing peaks. 16. Fix the live Leap capture backend so actual Leap frames, not fallback frames, populate `leap-ground-truth` cells in the multi-LOD cache. diff --git a/docs/sensor-fusion-architecture.md b/docs/sensor-fusion-architecture.md index 530a23c..fdd46b1 100644 --- a/docs/sensor-fusion-architecture.md +++ b/docs/sensor-fusion-architecture.md @@ -99,7 +99,7 @@ sensor observation The cache is not a bucket. `TrackCache` has a TTL and confidence update rule so the system can buffer and converge without pretending stale points are fresh. The live frame reservoir is explicitly five seconds wide: frame history expires from the newest shared sample, LOD evidence is clipped to that same interval, and each `RenderFramePacket` reports the exact source window. Later GPU buffers should mirror this shape: observation SoA, track SoA, material/BRDF estimate SoA, brush packet SoA. -`visual-lod-cache.json` is the current CPU-authored shape for the next compute pass. It is not the final renderer. It groups stochastic/fused scene claims into multiple world-space voxel sizes, carrying center, radius, confidence, count, and source-time bounds so an Aquarium compute shader can reconcile fine and coarse claims without pulling anything older than the reservoir through the renderer. +`visual-lod-cache.json` is the current CPU-authored shape for the next compute pass. It is not the final renderer. It groups stochastic/fused scene claims into multiple world-space voxel sizes, carrying center, radius, confidence, count, source-time bounds, and first-pass material estimates. The material channel is deliberately modest: weighted albedo plus roughness/metallic hints so Aquarium can start relighting brush/splat packets without treating screen-space color as the whole truth. A later shader pass should replace those hints with multi-view BRDF fitting inside the same reservoir. ## Test Boundaries @@ -172,6 +172,8 @@ That lets the final stream compositor buffer point-cloud visuals until they alig Current live deadline rule: `source_time_max_ns - source_time_min_ns` is five seconds. Late-arriving samples can fill the reservoir while they are inside that window; anything older is evidence for offline calibration, not the live program surface. +The Spout/audio overlay applies the same rule at the consumer boundary. If the visual producer or a camera driver stalls, OBS receives a packet clamped to the current audio edge with stale points removed. The current live fallback mode uses cached/synthetic RGB and Leap frames because direct OpenCV reads measured slower than the reservoir; real cameras need nonblocking ingress feeding the same reservoir instead of blocking the producer loop. + ## Next Cut 1. Capture ChArUco intrinsics for each camera with `scripts/charuco_calibration.py calibrate`. diff --git a/localcast/sensor_fusion/audio_overlay.py b/localcast/sensor_fusion/audio_overlay.py index 326ea0d..0d88734 100644 --- a/localcast/sensor_fusion/audio_overlay.py +++ b/localcast/sensor_fusion/audio_overlay.py @@ -17,6 +17,9 @@ if str(ROOT) not in sys.path: from audio_field.cultcache_audio import CultAudioSourceEvents, CultSpatialAudioFrame # noqa: E402 +DEFAULT_RESERVOIR_NS = 5_000_000_000 + + @dataclass(frozen=True) class AudioVisualSyncStatus: visual_frame_id: int @@ -39,6 +42,7 @@ def overlay_audio_events( radius_m: float = 0.045, max_events: int = 128, ) -> tuple[RenderFramePacket, AudioVisualSyncStatus]: + frame = clamp_frame_to_audio_reservoir(frame, audio_frame) if events is None: return frame, sync_status(frame, None, 0, 0, None, False) overlay = select_events_for_visual_frame(frame, events, audio_frame, int(window_ns), int(max_events)) @@ -76,6 +80,32 @@ def overlay_audio_events( ) +def clamp_frame_to_audio_reservoir( + frame: RenderFramePacket, + audio_frame: CultSpatialAudioFrame | None, + *, + reservoir_ns: int = DEFAULT_RESERVOIR_NS, +) -> RenderFramePacket: + if audio_frame is None: + return frame + edge_ns = max(int(frame.source_time_max_ns), int(audio_frame.audio_time_ns)) + start_ns = edge_ns - int(reservoir_ns) + points = tuple(point for point in frame.points if start_ns <= int(point.source_timestamp_ns) <= edge_ns) + return RenderFramePacket( + schema=frame.schema, + frame_id=frame.frame_id, + created_monotonic_ns=frame.created_monotonic_ns, + source_time_min_ns=start_ns, + source_time_max_ns=edge_ns, + present_time_ns=edge_ns, + audio_alignment_time_ns=edge_ns, + spout_sender_name=frame.spout_sender_name, + target_width=frame.target_width, + target_height=frame.target_height, + points=points, + ) + + def select_events_for_visual_frame( frame: RenderFramePacket, events: CultAudioSourceEvents, diff --git a/localcast/sensor_fusion/stochastic_mapping.py b/localcast/sensor_fusion/stochastic_mapping.py index 8c2b3ba..7480a82 100644 --- a/localcast/sensor_fusion/stochastic_mapping.py +++ b/localcast/sensor_fusion/stochastic_mapping.py @@ -97,6 +97,9 @@ class LODCell: source_time_max_ns: int source_priority: float = 1.0 source_kind: str = "generic" + material_albedo: tuple[float, float, float] = (1.0, 1.0, 1.0) + material_roughness: float = 0.7 + material_metallic: float = 0.0 @dataclass(frozen=True) @@ -123,6 +126,11 @@ class MultiLODSceneCache: "sourceTimeMaxNs": cell.source_time_max_ns, "sourcePriority": cell.source_priority, "sourceKind": cell.source_kind, + "material": { + "albedo": list(cell.material_albedo), + "roughness": float(cell.material_roughness), + "metallic": float(cell.material_metallic), + }, } for cell in self.cells ], @@ -316,6 +324,9 @@ class LODEvidencePoint: timestamp_ns: int source_priority: float source_kind: str + albedo: tuple[float, float, float] = (1.0, 1.0, 1.0) + roughness: float = 0.7 + metallic: float = 0.0 def multilod_cache_from_evidence( @@ -337,6 +348,10 @@ def multilod_cache_from_evidence( center = np.average(positions, axis=0, weights=weights) priority = float(max(item.source_priority for item in bucket)) source_kind = max(bucket, key=lambda item: item.source_priority).source_kind + albedos = np.asarray([np.asarray(item.albedo, dtype=np.float64) for item in bucket], dtype=np.float64) + albedo = np.average(albedos, axis=0, weights=weights) + roughness = float(np.average([item.roughness for item in bucket], weights=weights)) + metallic = float(np.average([item.metallic for item in bucket], weights=weights)) cells.append( LODCell( level=level_index, @@ -349,6 +364,9 @@ def multilod_cache_from_evidence( source_time_max_ns=max(item.timestamp_ns for item in bucket), source_priority=priority, source_kind=source_kind, + material_albedo=tuple(float(np.clip(value, 0.0, 1.0)) for value in albedo), + material_roughness=float(np.clip(roughness, 0.0, 1.0)), + material_metallic=float(np.clip(metallic, 0.0, 1.0)), ) ) return MultiLODSceneCache( @@ -362,6 +380,7 @@ def multilod_cache_from_evidence( def evidence_from_render_points(points: Iterable, *, source_kind: str, source_priority: float) -> tuple[LODEvidencePoint, ...]: rows: list[LODEvidencePoint] = [] for point in points: + rgba = np.clip(np.asarray(getattr(point, "color_rgba", (1.0, 1.0, 1.0, 1.0)), dtype=np.float64), 0.0, 1.0) rows.append( LODEvidencePoint( stable_key=str(point.stable_key), @@ -370,6 +389,9 @@ def evidence_from_render_points(points: Iterable, *, source_kind: str, source_pr timestamp_ns=int(point.source_timestamp_ns), source_priority=float(source_priority), source_kind=str(source_kind), + albedo=tuple(float(value) for value in rgba[:3]), + roughness=roughness_from_rgba(rgba, source_kind=source_kind), + metallic=0.0, ) ) return tuple(rows) @@ -400,3 +422,12 @@ def item_timestamp_ns(item: TriangulatedPoint | TransientMatchHypothesis) -> int if hasattr(item, "timestamp_ns"): return int(getattr(item, "timestamp_ns")) return int(max(obs.timestamp_ns for obs in item.observations)) + + +def roughness_from_rgba(rgba: np.ndarray, *, source_kind: str) -> float: + rgb = np.asarray(rgba[:3], dtype=np.float64) + saturation = float(np.max(rgb) - np.min(rgb)) + alpha = float(rgba[3]) if rgba.size > 3 else 1.0 + base = 0.82 if "rgb" in source_kind else 0.74 + roughness = base - 0.22 * saturation + 0.10 * (1.0 - alpha) + return float(np.clip(roughness, 0.25, 0.95)) diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 08a5def..0057b57 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -109,6 +109,8 @@ Ownership: - The live producer now keeps a rolling RGB/Leap frame window for clap calibration. Kiyo stereo motion gives the first rough 3D clap position; Leap participates as the high-precision visual timing witness until a calibrated Leap geometric model joins the rig. - Clap peaks update a per-camera clock sync model. Camera history sampling uses `oracle_time - camera_offset`, so the point builder can reconstruct a shared instant instead of mixing latest frames from different clock domains. - The live producer's temporal authority is a five-second spatiotemporal reservoir measured backward from the newest shared sample. RGB/Leap history expires against that shared edge, LOD evidence is clipped to the same window, and each render frame declares the exact reservoir slice with `source_time_min_ns` and `source_time_max_ns`. +- Multi-LOD cache cells carry first-pass material evidence: weighted albedo plus roughness/metallic hints. Aquarium owns the real relighting/BRDF refinement, but it should receive material state from the reservoir rather than reverse-engineering it from screen brushes. +- `stream_spout.py` clamps stale visual packets to the current audio reservoir edge before OBS sees them. If a camera path blocks past the five-second window, its stale points are dropped. Current live deadline mode uses `--rgb-fallback-only --leap-fallback-only` because direct OpenCV camera reads measured slower than the reservoir; the next cut is nonblocking camera ingress that fills the reservoir continuously. - `localcast.sensor_fusion.chirp_pose` now turns phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints and per-camera pose-correction estimates. It loads camera-mic and speaker geometry from the audio-field profile when available, falling back to the example profile/defaults. The live producer publishes those constraints as `camera-chirp:*` and `camera-pose-correction:*` render points plus `chirp-camera-pose` LOD evidence. Audio range/timing evidence augments visual calibration; it does not own image geometry. - The Spout point-budget policy pins calibration and cross-modal constraints before spending the remaining adaptive LOD budget on stable high-priority surface claims and frame-rotated samples. Its JSON heartbeat reports the prefix counts that survived the render budget. - The live producer also writes a multi-LOD scene cache with source kind and priority. Real Leap frames are the highest-priority timing/spatial evidence; Leap fallback frames are tagged as fallback and do not become ground truth. diff --git a/scripts/live_sensor_fusion.py b/scripts/live_sensor_fusion.py index 1ede8e5..12a3bf2 100644 --- a/scripts/live_sensor_fusion.py +++ b/scripts/live_sensor_fusion.py @@ -78,6 +78,20 @@ def evidence_in_reservoir(evidence: tuple, *, latest_timestamp_ns: int, reservoi return tuple(item for item in evidence if start_ns <= int(item.timestamp_ns) <= end_ns) +def render_points_in_reservoir(points: tuple[RenderPointPacket, ...], *, latest_timestamp_ns: int, reservoir_ns: int) -> tuple[RenderPointPacket, ...]: + start_ns, end_ns = reservoir_window_ns(latest_timestamp_ns, reservoir_ns) + return tuple(point for point in points if start_ns <= int(point.source_timestamp_ns) <= end_ns) + + +def latest_audio_time_ns(audio_cache: Path) -> int | None: + if not audio_cache.exists(): + return None + frame = get_live_spatial_audio_frame(audio_cache) + if frame is None: + return None + return int(frame.audio_time_ns) + + def camera(sensor_id: str, x: float) -> CameraModel: return CameraModel( sensor_id=sensor_id, @@ -292,6 +306,7 @@ class RgbSplatSampler: adaptive_controls: bool, fallback_dir: Path, reservoir_ns: int = DEFAULT_RESERVOIR_NS, + fallback_only: bool = False, ) -> None: self.api = api self.primary_index = primary_index @@ -304,6 +319,7 @@ class RgbSplatSampler: self.enable_cpu_dense_stereo = enable_cpu_dense_stereo self.adaptive_controls = adaptive_controls self.fallback_dir = fallback_dir + self.fallback_only = bool(fallback_only) self._captures: dict[int, object] = {} self._frames: dict[int, np.ndarray] = {} self._timestamped_frames: dict[str, TimestampedFrame] = {} @@ -369,6 +385,8 @@ class RgbSplatSampler: def _read_frame(self, index: int) -> np.ndarray | None: import cv2 + if self.fallback_only: + return self._fallback_frame(index) capture = self._captures.get(index) if capture is None: capture = cv2.VideoCapture(index, cv2_api(self.api)) @@ -953,6 +971,7 @@ class LeapPackedMotionSampler: fps: float, step: int, fallback_dir: Path, + fallback_only: bool = False, ) -> None: self.api = api self.index = index @@ -961,6 +980,7 @@ class LeapPackedMotionSampler: self.fps = fps self.step = max(4, int(step)) self.fallback_dir = fallback_dir + self.fallback_only = bool(fallback_only) self._capture = None self._previous_channels: dict[str, np.ndarray] = {} self._last_timestamp_ns: int | None = None @@ -1008,6 +1028,8 @@ class LeapPackedMotionSampler: def _read_frame(self) -> np.ndarray | None: import cv2 + if self.fallback_only: + return self._fallback_frame() if self._capture is None: capture = cv2.VideoCapture(self.index, cv2_api(self.api)) if capture.isOpened(): @@ -1123,6 +1145,7 @@ def main() -> None: parser.add_argument("--duration", type=float) parser.add_argument("--no-rgb", action="store_true") parser.add_argument("--rgb-api", choices=["any", "dshow", "msmf"], default="dshow") + parser.add_argument("--rgb-fallback-only", action="store_true") parser.add_argument("--rgb-primary-index", type=int, default=1) parser.add_argument("--rgb-secondary-index", type=int, default=3) parser.add_argument("--rgb-width", type=int, default=640) @@ -1134,6 +1157,7 @@ def main() -> None: parser.add_argument("--no-adaptive-camera-controls", action="store_true") parser.add_argument("--no-leap", action="store_true") parser.add_argument("--leap-api", choices=["any", "dshow", "msmf"], default="msmf") + parser.add_argument("--leap-fallback-only", action="store_true") parser.add_argument("--leap-index", type=int, default=0) parser.add_argument("--leap-width", type=int, default=320) parser.add_argument("--leap-height", type=int, default=240) @@ -1179,6 +1203,7 @@ def main() -> None: adaptive_controls=not args.no_adaptive_camera_controls, fallback_dir=ROOT / "calibration" / "runs", reservoir_ns=reservoir_ns, + fallback_only=args.rgb_fallback_only, ) leap_sampler = None if args.no_leap else LeapPackedMotionSampler( api=args.leap_api, @@ -1188,6 +1213,7 @@ def main() -> None: fps=args.leap_fps, step=args.leap_step, fallback_dir=ROOT / "calibration" / "runs", + fallback_only=args.leap_fallback_only, ) clap_calibrator = None if args.no_clap_calibration else LiveClapCalibrator( audio_cache=Path(args.audio_cache), @@ -1234,6 +1260,8 @@ def main() -> None: clap_calibrator.observe_frames(leap_sampler.latest_timestamped_frames()) if leap_sampler is not None and leap_sampler.last_timestamp_ns is not None: timestamp_ns = max(timestamp_ns, leap_sampler.last_timestamp_ns) + audio_edge_ns = latest_audio_time_ns(Path(args.audio_cache)) + reservoir_edge_ns = max(timestamp_ns, audio_edge_ns if audio_edge_ns is not None else timestamp_ns) leap_source_kind = ( "leap-ground-truth" if leap_sampler is not None and leap_sampler.last_frame_kind == "capture" @@ -1253,8 +1281,13 @@ def main() -> None: rgb_points = rgb_sampler.splats(now, timestamp_ns) support_points = dim_fusion_points(tuple(frame.points)) if not rgb_points else () stochastic_points = transient_match_render_points(transient_matches, timestamp_ns) - clap_points = () if clap_calibrator is None else clap_calibrator.update(timestamp_ns) - chirp_pose_points = () if chirp_pose_mapper is None else chirp_pose_mapper.update(timestamp_ns) + clap_points = () if clap_calibrator is None else clap_calibrator.update(reservoir_edge_ns) + chirp_pose_points = () if chirp_pose_mapper is None else chirp_pose_mapper.update(reservoir_edge_ns) + render_points = render_points_in_reservoir( + clap_points + chirp_pose_points + leap_points + rgb_points + stochastic_points + support_points, + latest_timestamp_ns=reservoir_edge_ns, + reservoir_ns=reservoir_ns, + ) evidence = evidence_in_reservoir( evidence_from_render_points(leap_points, source_kind=leap_source_kind, source_priority=leap_source_priority) + evidence_from_render_points(rgb_points, source_kind="rgb-surface", source_priority=1.4) @@ -1262,10 +1295,10 @@ def main() -> None: + evidence_from_render_points(chirp_pose_points, source_kind="chirp-camera-pose", source_priority=1.8) + evidence_from_fusion_items(transient_matches, source_kind="stochastic-transient", source_priority=1.2) + evidence_from_fusion_items(result.points, source_kind="synthetic-fusion", source_priority=0.8), - latest_timestamp_ns=timestamp_ns, + latest_timestamp_ns=reservoir_edge_ns, reservoir_ns=reservoir_ns, ) - source_time_min_ns, source_time_max_ns = reservoir_window_ns(timestamp_ns, reservoir_ns) + source_time_min_ns, source_time_max_ns = reservoir_window_ns(reservoir_edge_ns, reservoir_ns) multilod_cache_from_evidence( evidence, levels=(0.01, 0.04, 0.16, 0.64), @@ -1282,7 +1315,7 @@ def main() -> None: spout_sender_name=frame.spout_sender_name, target_width=frame.target_width, target_height=frame.target_height, - points=clap_points + chirp_pose_points + leap_points + rgb_points + stochastic_points + support_points, + points=render_points, ) put_live_render_frame(cache_path, frame) frame_id += 1 diff --git a/state/map.yaml b/state/map.yaml index ed072c1..21071bb 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -194,7 +194,15 @@ research: producer now has an explicit five-second spatiotemporal reservoir: RGB/Leap frame history expires from the newest shared sample, LOD evidence is clipped to the same window, and each RenderFramePacket declares that - source window for Aquarium/OBS synchronization. + source window for Aquarium/OBS synchronization. Multi-LOD cells now carry + first-pass material estimates, weighted albedo plus roughness/metallic + hints, so Aquarium can relight/refine brush packets from reservoir state + instead of scraping screen-space color. The Spout/audio overlay boundary + clamps stale visual packets to the current audio reservoir edge before + OBS sees them. Current live deadline mode uses fallback-only RGB/Leap + because synchronous OpenCV camera reads measured slower than the reservoir; + the next coherent cut is nonblocking camera ingress that continuously + fills the five-second reservoir with real samples. typed_visual_state: summary: docs/typed-visual-state.md key_decision: > diff --git a/tests/test_sensor_fusion.py b/tests/test_sensor_fusion.py index 2fc1071..153981c 100644 --- a/tests/test_sensor_fusion.py +++ b/tests/test_sensor_fusion.py @@ -55,12 +55,14 @@ from localcast.sensor_fusion import ( SurfaceFeatureObservation, triangulate_surface_tracks, ) +from localcast.sensor_fusion.audio_overlay import clamp_frame_to_audio_reservoir from localcast.sensor_fusion.spout_output import rasterize_frame_rgba from scripts.live_sensor_fusion import ( DEFAULT_RESERVOIR_NS, LiveClapCalibrator, evidence_in_reservoir, leap_channel_motion_points, + render_points_in_reservoir, rgb_dense_camera, rgb_dense_stereo_splats, rgb_room_splats, @@ -629,6 +631,51 @@ class SensorFusionTests(unittest.TestCase): self.assertEqual(("fresh",), tuple(item.stable_key for item in kept)) + def test_render_points_are_clipped_to_shared_reservoir_edge(self): + points = ( + RenderPointPacket("old", np.zeros(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, 9), + RenderPointPacket("fresh", np.ones(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, DEFAULT_RESERVOIR_NS + 10), + ) + + kept = render_points_in_reservoir( + points, + latest_timestamp_ns=DEFAULT_RESERVOIR_NS + 10, + reservoir_ns=DEFAULT_RESERVOIR_NS, + ) + + self.assertEqual(("fresh",), tuple(point.stable_key for point in kept)) + + def test_spout_overlay_clamps_stale_visual_frame_to_audio_reservoir(self): + frame = RenderFramePacket( + schema="localcast.sensor_fusion.render_frame.v1", + frame_id=4, + created_monotonic_ns=1, + source_time_min_ns=0, + source_time_max_ns=100, + present_time_ns=100, + audio_alignment_time_ns=100, + spout_sender_name="test", + target_width=32, + target_height=32, + points=( + RenderPointPacket("old", np.zeros(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, 9), + RenderPointPacket("fresh", np.ones(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, DEFAULT_RESERVOIR_NS + 10), + ), + ) + audio = make_spatial_audio_frame( + np.zeros((8, 4), dtype=np.float32), + frame_id=1, + sample_rate=48000, + start_sample=0, + audio_time_ns=DEFAULT_RESERVOIR_NS + 10, + ) + + clamped = clamp_frame_to_audio_reservoir(frame, audio) + + self.assertEqual(DEFAULT_RESERVOIR_NS + 10, clamped.audio_alignment_time_ns) + self.assertEqual(5_000_000_000, clamped.source_time_max_ns - clamped.source_time_min_ns) + self.assertEqual(("fresh",), tuple(point.stable_key for point in clamped.points)) + def test_chirp_pose_constraints_map_camera_mics_to_range_residuals(self): constraints = constraints_from_phase_sources( ( diff --git a/tests/test_stochastic_mapping.py b/tests/test_stochastic_mapping.py index 0615291..10e8f15 100644 --- a/tests/test_stochastic_mapping.py +++ b/tests/test_stochastic_mapping.py @@ -99,8 +99,8 @@ class StochasticMappingTests(unittest.TestCase): def test_multilod_evidence_prefers_leap_priority(self): evidence = [ - LODEvidencePoint("rgb", np.array([0.01, 0.0, 1.0]), 0.7, 100, 1.0, "rgb"), - LODEvidencePoint("leap", np.array([0.03, 0.0, 1.0]), 0.8, 120, 3.0, "leap-ground-truth"), + LODEvidencePoint("rgb", np.array([0.01, 0.0, 1.0]), 0.7, 100, 1.0, "rgb", albedo=(1.0, 0.2, 0.1), roughness=0.42), + LODEvidencePoint("leap", np.array([0.03, 0.0, 1.0]), 0.8, 120, 3.0, "leap-ground-truth", albedo=(0.2, 0.8, 1.0), roughness=0.76), ] cache = multilod_cache_from_evidence(evidence, levels=(0.10,), created_monotonic_ns=5) @@ -109,6 +109,10 @@ class StochasticMappingTests(unittest.TestCase): self.assertEqual("leap-ground-truth", cell.source_kind) self.assertEqual(3.0, cell.source_priority) self.assertGreater(cell.center[0], 0.02) + self.assertGreater(cell.material_albedo[2], cell.material_albedo[0]) + self.assertGreater(cell.material_roughness, 0.6) + self.assertEqual(0.0, cell.material_metallic) + self.assertIn("material", cache.to_dict()["cells"][0]) if __name__ == "__main__":