Clamp fusion output to shared reservoir
This commit is contained in:
parent
792d274f7d
commit
2dfa5fbb81
9 changed files with 170 additions and 10 deletions
|
|
@ -32,6 +32,9 @@
|
|||
- Live clap calibration is wired into the visual producer. It keeps a rolling frame window from the Kiyo pair plus Leap, reads the live spatial audio frame for transient candidates, publishes `clap-events.msgpack`, injects clap calibration markers into the render frame, and writes clap evidence into the LOD cache. Kiyo stereo owns the current rough 3D solve; Leap owns the best visual timing witness until its geometric model is calibrated.
|
||||
- Clap peaks now update a per-camera clock sync model. The point-cloud builder can sample camera history buffers at `oracle_time - camera_offset` instead of using whichever frame arrived last.
|
||||
- The live visual producer now uses an explicit five-second spatiotemporal reservoir. RGB/Leap frame history expires from the newest shared sample, the multi-LOD cache only accepts evidence inside that same window, and `RenderFramePacket.source_time_min_ns/source_time_max_ns` declares the reservoir slice being rendered.
|
||||
- Multi-LOD cells now carry first-pass material estimates: weighted albedo plus roughness/metallic hints derived from source evidence. Aquarium should treat this as relightable material evidence to refine, not as the final BRDF solve.
|
||||
- The Spout/audio overlay boundary clamps stale visual packets to the current audio reservoir edge. If camera geometry ages out, it is dropped instead of being broadcast out of sync.
|
||||
- Live fallback-only RGB/Leap mode exists for deadline operation when OpenCV/MSMF/DirectShow reads block longer than the five-second reservoir. It keeps the reservoir and OBS output alive while real camera capture is moved to nonblocking ingress.
|
||||
- `localcast.sensor_fusion.chirp_pose` converts live phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints and per-camera pose-correction estimates. It can load camera-mic and speaker geometry from the audio-field profile, with the example profile used as the live fallback until measured local geometry exists. `scripts/live_sensor_fusion.py` injects those constraints as `camera-chirp:*` and `camera-pose-correction:*` render points plus `chirp-camera-pose` LOD evidence.
|
||||
- `scripts/stream_spout.py` runs the deadline Spout sender loop from the typed cache with typed and JSON heartbeat status.
|
||||
- The deadline Spout renderer has a named `kiyo-mid-deru` virtual camera preset: eye at the midpoint between the two Kiyo-class cameras and target on the co-streamer body volume. It also applies a renderer-owned point budget with pinned calibration/cross-modal constraints, stable high-confidence anchors, and frame-varying remainder samples so downstream TAA/supersampling can accumulate detail without forcing every source claim into every OBS frame. Its JSON heartbeat reports the prefix counts that survived the render budget.
|
||||
|
|
@ -78,7 +81,7 @@
|
|||
10. Replace placeholder mic/speaker geometry with measured world coordinates, then build the delay/SRO alignment stage that feeds the bounded field cache and emits aligned six-channel blocks before FOA encoding.
|
||||
11. Move the render-frame consumer into Aquarium Engine so dense brush/splat rendering replaces the deadline OpenGL point sink behind the same OBS Spout boundary.
|
||||
12. Capture fixed-board ChArUco observations for every camera, solve `config/sensor-fusion.json`, and verify cross-view feature triangulation with `scripts/triangulate_surface_features.py`.
|
||||
13. Replace synthetic live-fusion observations with PS3 Eye detector observations and calibrated surface-feature tracks.
|
||||
13. Replace synchronous OpenCV camera reads in the hot producer with nonblocking camera ingress that continuously fills the five-second reservoir, then promote real PS3 Eye/Kiyo/Leap samples over fallback-only mode.
|
||||
14. Replace the example speaker/camera-mic geometry used by `chirp_pose` with measured local `config/audio-field.json` coordinates, then let the common-space optimizer consume range residuals instead of only visualizing them.
|
||||
15. Calibrate Leap as a geometric camera in the shared rig so the timing authority can also contribute direct 3D rays instead of only visual timing peaks.
|
||||
16. Fix the live Leap capture backend so actual Leap frames, not fallback frames, populate `leap-ground-truth` cells in the multi-LOD cache.
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@ sensor observation
|
|||
|
||||
The cache is not a bucket. `TrackCache` has a TTL and confidence update rule so the system can buffer and converge without pretending stale points are fresh. The live frame reservoir is explicitly five seconds wide: frame history expires from the newest shared sample, LOD evidence is clipped to that same interval, and each `RenderFramePacket` reports the exact source window. Later GPU buffers should mirror this shape: observation SoA, track SoA, material/BRDF estimate SoA, brush packet SoA.
|
||||
|
||||
`visual-lod-cache.json` is the current CPU-authored shape for the next compute pass. It is not the final renderer. It groups stochastic/fused scene claims into multiple world-space voxel sizes, carrying center, radius, confidence, count, and source-time bounds so an Aquarium compute shader can reconcile fine and coarse claims without pulling anything older than the reservoir through the renderer.
|
||||
`visual-lod-cache.json` is the current CPU-authored shape for the next compute pass. It is not the final renderer. It groups stochastic/fused scene claims into multiple world-space voxel sizes, carrying center, radius, confidence, count, source-time bounds, and first-pass material estimates. The material channel is deliberately modest: weighted albedo plus roughness/metallic hints so Aquarium can start relighting brush/splat packets without treating screen-space color as the whole truth. A later shader pass should replace those hints with multi-view BRDF fitting inside the same reservoir.
|
||||
|
||||
## Test Boundaries
|
||||
|
||||
|
|
@ -172,6 +172,8 @@ That lets the final stream compositor buffer point-cloud visuals until they alig
|
|||
|
||||
Current live deadline rule: `source_time_max_ns - source_time_min_ns` is five seconds. Late-arriving samples can fill the reservoir while they are inside that window; anything older is evidence for offline calibration, not the live program surface.
|
||||
|
||||
The Spout/audio overlay applies the same rule at the consumer boundary. If the visual producer or a camera driver stalls, OBS receives a packet clamped to the current audio edge with stale points removed. The current live fallback mode uses cached/synthetic RGB and Leap frames because direct OpenCV reads measured slower than the reservoir; real cameras need nonblocking ingress feeding the same reservoir instead of blocking the producer loop.
|
||||
|
||||
## Next Cut
|
||||
|
||||
1. Capture ChArUco intrinsics for each camera with `scripts/charuco_calibration.py calibrate`.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,9 @@ if str(ROOT) not in sys.path:
|
|||
from audio_field.cultcache_audio import CultAudioSourceEvents, CultSpatialAudioFrame # noqa: E402
|
||||
|
||||
|
||||
DEFAULT_RESERVOIR_NS = 5_000_000_000
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AudioVisualSyncStatus:
|
||||
visual_frame_id: int
|
||||
|
|
@ -39,6 +42,7 @@ def overlay_audio_events(
|
|||
radius_m: float = 0.045,
|
||||
max_events: int = 128,
|
||||
) -> tuple[RenderFramePacket, AudioVisualSyncStatus]:
|
||||
frame = clamp_frame_to_audio_reservoir(frame, audio_frame)
|
||||
if events is None:
|
||||
return frame, sync_status(frame, None, 0, 0, None, False)
|
||||
overlay = select_events_for_visual_frame(frame, events, audio_frame, int(window_ns), int(max_events))
|
||||
|
|
@ -76,6 +80,32 @@ def overlay_audio_events(
|
|||
)
|
||||
|
||||
|
||||
def clamp_frame_to_audio_reservoir(
|
||||
frame: RenderFramePacket,
|
||||
audio_frame: CultSpatialAudioFrame | None,
|
||||
*,
|
||||
reservoir_ns: int = DEFAULT_RESERVOIR_NS,
|
||||
) -> RenderFramePacket:
|
||||
if audio_frame is None:
|
||||
return frame
|
||||
edge_ns = max(int(frame.source_time_max_ns), int(audio_frame.audio_time_ns))
|
||||
start_ns = edge_ns - int(reservoir_ns)
|
||||
points = tuple(point for point in frame.points if start_ns <= int(point.source_timestamp_ns) <= edge_ns)
|
||||
return RenderFramePacket(
|
||||
schema=frame.schema,
|
||||
frame_id=frame.frame_id,
|
||||
created_monotonic_ns=frame.created_monotonic_ns,
|
||||
source_time_min_ns=start_ns,
|
||||
source_time_max_ns=edge_ns,
|
||||
present_time_ns=edge_ns,
|
||||
audio_alignment_time_ns=edge_ns,
|
||||
spout_sender_name=frame.spout_sender_name,
|
||||
target_width=frame.target_width,
|
||||
target_height=frame.target_height,
|
||||
points=points,
|
||||
)
|
||||
|
||||
|
||||
def select_events_for_visual_frame(
|
||||
frame: RenderFramePacket,
|
||||
events: CultAudioSourceEvents,
|
||||
|
|
|
|||
|
|
@ -97,6 +97,9 @@ class LODCell:
|
|||
source_time_max_ns: int
|
||||
source_priority: float = 1.0
|
||||
source_kind: str = "generic"
|
||||
material_albedo: tuple[float, float, float] = (1.0, 1.0, 1.0)
|
||||
material_roughness: float = 0.7
|
||||
material_metallic: float = 0.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -123,6 +126,11 @@ class MultiLODSceneCache:
|
|||
"sourceTimeMaxNs": cell.source_time_max_ns,
|
||||
"sourcePriority": cell.source_priority,
|
||||
"sourceKind": cell.source_kind,
|
||||
"material": {
|
||||
"albedo": list(cell.material_albedo),
|
||||
"roughness": float(cell.material_roughness),
|
||||
"metallic": float(cell.material_metallic),
|
||||
},
|
||||
}
|
||||
for cell in self.cells
|
||||
],
|
||||
|
|
@ -316,6 +324,9 @@ class LODEvidencePoint:
|
|||
timestamp_ns: int
|
||||
source_priority: float
|
||||
source_kind: str
|
||||
albedo: tuple[float, float, float] = (1.0, 1.0, 1.0)
|
||||
roughness: float = 0.7
|
||||
metallic: float = 0.0
|
||||
|
||||
|
||||
def multilod_cache_from_evidence(
|
||||
|
|
@ -337,6 +348,10 @@ def multilod_cache_from_evidence(
|
|||
center = np.average(positions, axis=0, weights=weights)
|
||||
priority = float(max(item.source_priority for item in bucket))
|
||||
source_kind = max(bucket, key=lambda item: item.source_priority).source_kind
|
||||
albedos = np.asarray([np.asarray(item.albedo, dtype=np.float64) for item in bucket], dtype=np.float64)
|
||||
albedo = np.average(albedos, axis=0, weights=weights)
|
||||
roughness = float(np.average([item.roughness for item in bucket], weights=weights))
|
||||
metallic = float(np.average([item.metallic for item in bucket], weights=weights))
|
||||
cells.append(
|
||||
LODCell(
|
||||
level=level_index,
|
||||
|
|
@ -349,6 +364,9 @@ def multilod_cache_from_evidence(
|
|||
source_time_max_ns=max(item.timestamp_ns for item in bucket),
|
||||
source_priority=priority,
|
||||
source_kind=source_kind,
|
||||
material_albedo=tuple(float(np.clip(value, 0.0, 1.0)) for value in albedo),
|
||||
material_roughness=float(np.clip(roughness, 0.0, 1.0)),
|
||||
material_metallic=float(np.clip(metallic, 0.0, 1.0)),
|
||||
)
|
||||
)
|
||||
return MultiLODSceneCache(
|
||||
|
|
@ -362,6 +380,7 @@ def multilod_cache_from_evidence(
|
|||
def evidence_from_render_points(points: Iterable, *, source_kind: str, source_priority: float) -> tuple[LODEvidencePoint, ...]:
|
||||
rows: list[LODEvidencePoint] = []
|
||||
for point in points:
|
||||
rgba = np.clip(np.asarray(getattr(point, "color_rgba", (1.0, 1.0, 1.0, 1.0)), dtype=np.float64), 0.0, 1.0)
|
||||
rows.append(
|
||||
LODEvidencePoint(
|
||||
stable_key=str(point.stable_key),
|
||||
|
|
@ -370,6 +389,9 @@ def evidence_from_render_points(points: Iterable, *, source_kind: str, source_pr
|
|||
timestamp_ns=int(point.source_timestamp_ns),
|
||||
source_priority=float(source_priority),
|
||||
source_kind=str(source_kind),
|
||||
albedo=tuple(float(value) for value in rgba[:3]),
|
||||
roughness=roughness_from_rgba(rgba, source_kind=source_kind),
|
||||
metallic=0.0,
|
||||
)
|
||||
)
|
||||
return tuple(rows)
|
||||
|
|
@ -400,3 +422,12 @@ def item_timestamp_ns(item: TriangulatedPoint | TransientMatchHypothesis) -> int
|
|||
if hasattr(item, "timestamp_ns"):
|
||||
return int(getattr(item, "timestamp_ns"))
|
||||
return int(max(obs.timestamp_ns for obs in item.observations))
|
||||
|
||||
|
||||
def roughness_from_rgba(rgba: np.ndarray, *, source_kind: str) -> float:
|
||||
rgb = np.asarray(rgba[:3], dtype=np.float64)
|
||||
saturation = float(np.max(rgb) - np.min(rgb))
|
||||
alpha = float(rgba[3]) if rgba.size > 3 else 1.0
|
||||
base = 0.82 if "rgb" in source_kind else 0.74
|
||||
roughness = base - 0.22 * saturation + 0.10 * (1.0 - alpha)
|
||||
return float(np.clip(roughness, 0.25, 0.95))
|
||||
|
|
|
|||
|
|
@ -109,6 +109,8 @@ Ownership:
|
|||
- The live producer now keeps a rolling RGB/Leap frame window for clap calibration. Kiyo stereo motion gives the first rough 3D clap position; Leap participates as the high-precision visual timing witness until a calibrated Leap geometric model joins the rig.
|
||||
- Clap peaks update a per-camera clock sync model. Camera history sampling uses `oracle_time - camera_offset`, so the point builder can reconstruct a shared instant instead of mixing latest frames from different clock domains.
|
||||
- The live producer's temporal authority is a five-second spatiotemporal reservoir measured backward from the newest shared sample. RGB/Leap history expires against that shared edge, LOD evidence is clipped to the same window, and each render frame declares the exact reservoir slice with `source_time_min_ns` and `source_time_max_ns`.
|
||||
- Multi-LOD cache cells carry first-pass material evidence: weighted albedo plus roughness/metallic hints. Aquarium owns the real relighting/BRDF refinement, but it should receive material state from the reservoir rather than reverse-engineering it from screen brushes.
|
||||
- `stream_spout.py` clamps stale visual packets to the current audio reservoir edge before OBS sees them. If a camera path blocks past the five-second window, its stale points are dropped. Current live deadline mode uses `--rgb-fallback-only --leap-fallback-only` because direct OpenCV camera reads measured slower than the reservoir; the next cut is nonblocking camera ingress that fills the reservoir continuously.
|
||||
- `localcast.sensor_fusion.chirp_pose` now turns phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints and per-camera pose-correction estimates. It loads camera-mic and speaker geometry from the audio-field profile when available, falling back to the example profile/defaults. The live producer publishes those constraints as `camera-chirp:*` and `camera-pose-correction:*` render points plus `chirp-camera-pose` LOD evidence. Audio range/timing evidence augments visual calibration; it does not own image geometry.
|
||||
- The Spout point-budget policy pins calibration and cross-modal constraints before spending the remaining adaptive LOD budget on stable high-priority surface claims and frame-rotated samples. Its JSON heartbeat reports the prefix counts that survived the render budget.
|
||||
- The live producer also writes a multi-LOD scene cache with source kind and priority. Real Leap frames are the highest-priority timing/spatial evidence; Leap fallback frames are tagged as fallback and do not become ground truth.
|
||||
|
|
|
|||
|
|
@ -78,6 +78,20 @@ def evidence_in_reservoir(evidence: tuple, *, latest_timestamp_ns: int, reservoi
|
|||
return tuple(item for item in evidence if start_ns <= int(item.timestamp_ns) <= end_ns)
|
||||
|
||||
|
||||
def render_points_in_reservoir(points: tuple[RenderPointPacket, ...], *, latest_timestamp_ns: int, reservoir_ns: int) -> tuple[RenderPointPacket, ...]:
|
||||
start_ns, end_ns = reservoir_window_ns(latest_timestamp_ns, reservoir_ns)
|
||||
return tuple(point for point in points if start_ns <= int(point.source_timestamp_ns) <= end_ns)
|
||||
|
||||
|
||||
def latest_audio_time_ns(audio_cache: Path) -> int | None:
|
||||
if not audio_cache.exists():
|
||||
return None
|
||||
frame = get_live_spatial_audio_frame(audio_cache)
|
||||
if frame is None:
|
||||
return None
|
||||
return int(frame.audio_time_ns)
|
||||
|
||||
|
||||
def camera(sensor_id: str, x: float) -> CameraModel:
|
||||
return CameraModel(
|
||||
sensor_id=sensor_id,
|
||||
|
|
@ -292,6 +306,7 @@ class RgbSplatSampler:
|
|||
adaptive_controls: bool,
|
||||
fallback_dir: Path,
|
||||
reservoir_ns: int = DEFAULT_RESERVOIR_NS,
|
||||
fallback_only: bool = False,
|
||||
) -> None:
|
||||
self.api = api
|
||||
self.primary_index = primary_index
|
||||
|
|
@ -304,6 +319,7 @@ class RgbSplatSampler:
|
|||
self.enable_cpu_dense_stereo = enable_cpu_dense_stereo
|
||||
self.adaptive_controls = adaptive_controls
|
||||
self.fallback_dir = fallback_dir
|
||||
self.fallback_only = bool(fallback_only)
|
||||
self._captures: dict[int, object] = {}
|
||||
self._frames: dict[int, np.ndarray] = {}
|
||||
self._timestamped_frames: dict[str, TimestampedFrame] = {}
|
||||
|
|
@ -369,6 +385,8 @@ class RgbSplatSampler:
|
|||
def _read_frame(self, index: int) -> np.ndarray | None:
|
||||
import cv2
|
||||
|
||||
if self.fallback_only:
|
||||
return self._fallback_frame(index)
|
||||
capture = self._captures.get(index)
|
||||
if capture is None:
|
||||
capture = cv2.VideoCapture(index, cv2_api(self.api))
|
||||
|
|
@ -953,6 +971,7 @@ class LeapPackedMotionSampler:
|
|||
fps: float,
|
||||
step: int,
|
||||
fallback_dir: Path,
|
||||
fallback_only: bool = False,
|
||||
) -> None:
|
||||
self.api = api
|
||||
self.index = index
|
||||
|
|
@ -961,6 +980,7 @@ class LeapPackedMotionSampler:
|
|||
self.fps = fps
|
||||
self.step = max(4, int(step))
|
||||
self.fallback_dir = fallback_dir
|
||||
self.fallback_only = bool(fallback_only)
|
||||
self._capture = None
|
||||
self._previous_channels: dict[str, np.ndarray] = {}
|
||||
self._last_timestamp_ns: int | None = None
|
||||
|
|
@ -1008,6 +1028,8 @@ class LeapPackedMotionSampler:
|
|||
def _read_frame(self) -> np.ndarray | None:
|
||||
import cv2
|
||||
|
||||
if self.fallback_only:
|
||||
return self._fallback_frame()
|
||||
if self._capture is None:
|
||||
capture = cv2.VideoCapture(self.index, cv2_api(self.api))
|
||||
if capture.isOpened():
|
||||
|
|
@ -1123,6 +1145,7 @@ def main() -> None:
|
|||
parser.add_argument("--duration", type=float)
|
||||
parser.add_argument("--no-rgb", action="store_true")
|
||||
parser.add_argument("--rgb-api", choices=["any", "dshow", "msmf"], default="dshow")
|
||||
parser.add_argument("--rgb-fallback-only", action="store_true")
|
||||
parser.add_argument("--rgb-primary-index", type=int, default=1)
|
||||
parser.add_argument("--rgb-secondary-index", type=int, default=3)
|
||||
parser.add_argument("--rgb-width", type=int, default=640)
|
||||
|
|
@ -1134,6 +1157,7 @@ def main() -> None:
|
|||
parser.add_argument("--no-adaptive-camera-controls", action="store_true")
|
||||
parser.add_argument("--no-leap", action="store_true")
|
||||
parser.add_argument("--leap-api", choices=["any", "dshow", "msmf"], default="msmf")
|
||||
parser.add_argument("--leap-fallback-only", action="store_true")
|
||||
parser.add_argument("--leap-index", type=int, default=0)
|
||||
parser.add_argument("--leap-width", type=int, default=320)
|
||||
parser.add_argument("--leap-height", type=int, default=240)
|
||||
|
|
@ -1179,6 +1203,7 @@ def main() -> None:
|
|||
adaptive_controls=not args.no_adaptive_camera_controls,
|
||||
fallback_dir=ROOT / "calibration" / "runs",
|
||||
reservoir_ns=reservoir_ns,
|
||||
fallback_only=args.rgb_fallback_only,
|
||||
)
|
||||
leap_sampler = None if args.no_leap else LeapPackedMotionSampler(
|
||||
api=args.leap_api,
|
||||
|
|
@ -1188,6 +1213,7 @@ def main() -> None:
|
|||
fps=args.leap_fps,
|
||||
step=args.leap_step,
|
||||
fallback_dir=ROOT / "calibration" / "runs",
|
||||
fallback_only=args.leap_fallback_only,
|
||||
)
|
||||
clap_calibrator = None if args.no_clap_calibration else LiveClapCalibrator(
|
||||
audio_cache=Path(args.audio_cache),
|
||||
|
|
@ -1234,6 +1260,8 @@ def main() -> None:
|
|||
clap_calibrator.observe_frames(leap_sampler.latest_timestamped_frames())
|
||||
if leap_sampler is not None and leap_sampler.last_timestamp_ns is not None:
|
||||
timestamp_ns = max(timestamp_ns, leap_sampler.last_timestamp_ns)
|
||||
audio_edge_ns = latest_audio_time_ns(Path(args.audio_cache))
|
||||
reservoir_edge_ns = max(timestamp_ns, audio_edge_ns if audio_edge_ns is not None else timestamp_ns)
|
||||
leap_source_kind = (
|
||||
"leap-ground-truth"
|
||||
if leap_sampler is not None and leap_sampler.last_frame_kind == "capture"
|
||||
|
|
@ -1253,8 +1281,13 @@ def main() -> None:
|
|||
rgb_points = rgb_sampler.splats(now, timestamp_ns)
|
||||
support_points = dim_fusion_points(tuple(frame.points)) if not rgb_points else ()
|
||||
stochastic_points = transient_match_render_points(transient_matches, timestamp_ns)
|
||||
clap_points = () if clap_calibrator is None else clap_calibrator.update(timestamp_ns)
|
||||
chirp_pose_points = () if chirp_pose_mapper is None else chirp_pose_mapper.update(timestamp_ns)
|
||||
clap_points = () if clap_calibrator is None else clap_calibrator.update(reservoir_edge_ns)
|
||||
chirp_pose_points = () if chirp_pose_mapper is None else chirp_pose_mapper.update(reservoir_edge_ns)
|
||||
render_points = render_points_in_reservoir(
|
||||
clap_points + chirp_pose_points + leap_points + rgb_points + stochastic_points + support_points,
|
||||
latest_timestamp_ns=reservoir_edge_ns,
|
||||
reservoir_ns=reservoir_ns,
|
||||
)
|
||||
evidence = evidence_in_reservoir(
|
||||
evidence_from_render_points(leap_points, source_kind=leap_source_kind, source_priority=leap_source_priority)
|
||||
+ evidence_from_render_points(rgb_points, source_kind="rgb-surface", source_priority=1.4)
|
||||
|
|
@ -1262,10 +1295,10 @@ def main() -> None:
|
|||
+ evidence_from_render_points(chirp_pose_points, source_kind="chirp-camera-pose", source_priority=1.8)
|
||||
+ evidence_from_fusion_items(transient_matches, source_kind="stochastic-transient", source_priority=1.2)
|
||||
+ evidence_from_fusion_items(result.points, source_kind="synthetic-fusion", source_priority=0.8),
|
||||
latest_timestamp_ns=timestamp_ns,
|
||||
latest_timestamp_ns=reservoir_edge_ns,
|
||||
reservoir_ns=reservoir_ns,
|
||||
)
|
||||
source_time_min_ns, source_time_max_ns = reservoir_window_ns(timestamp_ns, reservoir_ns)
|
||||
source_time_min_ns, source_time_max_ns = reservoir_window_ns(reservoir_edge_ns, reservoir_ns)
|
||||
multilod_cache_from_evidence(
|
||||
evidence,
|
||||
levels=(0.01, 0.04, 0.16, 0.64),
|
||||
|
|
@ -1282,7 +1315,7 @@ def main() -> None:
|
|||
spout_sender_name=frame.spout_sender_name,
|
||||
target_width=frame.target_width,
|
||||
target_height=frame.target_height,
|
||||
points=clap_points + chirp_pose_points + leap_points + rgb_points + stochastic_points + support_points,
|
||||
points=render_points,
|
||||
)
|
||||
put_live_render_frame(cache_path, frame)
|
||||
frame_id += 1
|
||||
|
|
|
|||
|
|
@ -194,7 +194,15 @@ research:
|
|||
producer now has an explicit five-second spatiotemporal reservoir:
|
||||
RGB/Leap frame history expires from the newest shared sample, LOD evidence
|
||||
is clipped to the same window, and each RenderFramePacket declares that
|
||||
source window for Aquarium/OBS synchronization.
|
||||
source window for Aquarium/OBS synchronization. Multi-LOD cells now carry
|
||||
first-pass material estimates, weighted albedo plus roughness/metallic
|
||||
hints, so Aquarium can relight/refine brush packets from reservoir state
|
||||
instead of scraping screen-space color. The Spout/audio overlay boundary
|
||||
clamps stale visual packets to the current audio reservoir edge before
|
||||
OBS sees them. Current live deadline mode uses fallback-only RGB/Leap
|
||||
because synchronous OpenCV camera reads measured slower than the reservoir;
|
||||
the next coherent cut is nonblocking camera ingress that continuously
|
||||
fills the five-second reservoir with real samples.
|
||||
typed_visual_state:
|
||||
summary: docs/typed-visual-state.md
|
||||
key_decision: >
|
||||
|
|
|
|||
|
|
@ -55,12 +55,14 @@ from localcast.sensor_fusion import (
|
|||
SurfaceFeatureObservation,
|
||||
triangulate_surface_tracks,
|
||||
)
|
||||
from localcast.sensor_fusion.audio_overlay import clamp_frame_to_audio_reservoir
|
||||
from localcast.sensor_fusion.spout_output import rasterize_frame_rgba
|
||||
from scripts.live_sensor_fusion import (
|
||||
DEFAULT_RESERVOIR_NS,
|
||||
LiveClapCalibrator,
|
||||
evidence_in_reservoir,
|
||||
leap_channel_motion_points,
|
||||
render_points_in_reservoir,
|
||||
rgb_dense_camera,
|
||||
rgb_dense_stereo_splats,
|
||||
rgb_room_splats,
|
||||
|
|
@ -629,6 +631,51 @@ class SensorFusionTests(unittest.TestCase):
|
|||
|
||||
self.assertEqual(("fresh",), tuple(item.stable_key for item in kept))
|
||||
|
||||
def test_render_points_are_clipped_to_shared_reservoir_edge(self):
|
||||
points = (
|
||||
RenderPointPacket("old", np.zeros(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, 9),
|
||||
RenderPointPacket("fresh", np.ones(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, DEFAULT_RESERVOIR_NS + 10),
|
||||
)
|
||||
|
||||
kept = render_points_in_reservoir(
|
||||
points,
|
||||
latest_timestamp_ns=DEFAULT_RESERVOIR_NS + 10,
|
||||
reservoir_ns=DEFAULT_RESERVOIR_NS,
|
||||
)
|
||||
|
||||
self.assertEqual(("fresh",), tuple(point.stable_key for point in kept))
|
||||
|
||||
def test_spout_overlay_clamps_stale_visual_frame_to_audio_reservoir(self):
|
||||
frame = RenderFramePacket(
|
||||
schema="localcast.sensor_fusion.render_frame.v1",
|
||||
frame_id=4,
|
||||
created_monotonic_ns=1,
|
||||
source_time_min_ns=0,
|
||||
source_time_max_ns=100,
|
||||
present_time_ns=100,
|
||||
audio_alignment_time_ns=100,
|
||||
spout_sender_name="test",
|
||||
target_width=32,
|
||||
target_height=32,
|
||||
points=(
|
||||
RenderPointPacket("old", np.zeros(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, 9),
|
||||
RenderPointPacket("fresh", np.ones(3), 0.01, (1.0, 1.0, 1.0, 1.0), 1.0, DEFAULT_RESERVOIR_NS + 10),
|
||||
),
|
||||
)
|
||||
audio = make_spatial_audio_frame(
|
||||
np.zeros((8, 4), dtype=np.float32),
|
||||
frame_id=1,
|
||||
sample_rate=48000,
|
||||
start_sample=0,
|
||||
audio_time_ns=DEFAULT_RESERVOIR_NS + 10,
|
||||
)
|
||||
|
||||
clamped = clamp_frame_to_audio_reservoir(frame, audio)
|
||||
|
||||
self.assertEqual(DEFAULT_RESERVOIR_NS + 10, clamped.audio_alignment_time_ns)
|
||||
self.assertEqual(5_000_000_000, clamped.source_time_max_ns - clamped.source_time_min_ns)
|
||||
self.assertEqual(("fresh",), tuple(point.stable_key for point in clamped.points))
|
||||
|
||||
def test_chirp_pose_constraints_map_camera_mics_to_range_residuals(self):
|
||||
constraints = constraints_from_phase_sources(
|
||||
(
|
||||
|
|
|
|||
|
|
@ -99,8 +99,8 @@ class StochasticMappingTests(unittest.TestCase):
|
|||
|
||||
def test_multilod_evidence_prefers_leap_priority(self):
|
||||
evidence = [
|
||||
LODEvidencePoint("rgb", np.array([0.01, 0.0, 1.0]), 0.7, 100, 1.0, "rgb"),
|
||||
LODEvidencePoint("leap", np.array([0.03, 0.0, 1.0]), 0.8, 120, 3.0, "leap-ground-truth"),
|
||||
LODEvidencePoint("rgb", np.array([0.01, 0.0, 1.0]), 0.7, 100, 1.0, "rgb", albedo=(1.0, 0.2, 0.1), roughness=0.42),
|
||||
LODEvidencePoint("leap", np.array([0.03, 0.0, 1.0]), 0.8, 120, 3.0, "leap-ground-truth", albedo=(0.2, 0.8, 1.0), roughness=0.76),
|
||||
]
|
||||
|
||||
cache = multilod_cache_from_evidence(evidence, levels=(0.10,), created_monotonic_ns=5)
|
||||
|
|
@ -109,6 +109,10 @@ class StochasticMappingTests(unittest.TestCase):
|
|||
self.assertEqual("leap-ground-truth", cell.source_kind)
|
||||
self.assertEqual(3.0, cell.source_priority)
|
||||
self.assertGreater(cell.center[0], 0.02)
|
||||
self.assertGreater(cell.material_albedo[2], cell.material_albedo[0])
|
||||
self.assertGreater(cell.material_roughness, 0.6)
|
||||
self.assertEqual(0.0, cell.material_metallic)
|
||||
self.assertIn("material", cache.to_dict()["cells"][0])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
Loading…
Reference in a new issue