Track remote video as synchronized media artifact

docs/obs-receiver-setup.md

@@ -28,6 +28,18 @@ If using a passphrase, append:
 
 OBS should be open before the sender starts if OBS is the SRT listener. If a source gets wedged, deactivate and reactivate the Media Source before changing ports; port churn is a symptom, not a configuration strategy.
 
+## Synchronization
+
+The video SRT source is a timed media artifact in the Aquarium path, not just scenery in OBS. Keep the listener URL and latency aligned with `stream_spout.py`:
+
+```powershell
+.\.venv\Scripts\python.exe .\scripts\stream_spout.py `
+  --remote-video-url "srt://0.0.0.0:5100?mode=listener&latency=120000&timeout=5000000" `
+  --remote-video-latency-ms 250
+```
+
+The live sync heartbeat is `calibration/runs/av-sync-status.json`. `remote-video-latency-ms` is presentation delay, not merely the SRT URL's `latency` parameter. Check `remote_video.delta_ns` before trusting the composite. If that number grows, fix the configured media delay or the ingest path instead of eyeballing it in OBS like a doomed little stage magician.
+
 ## Source Layout
 
 Recommended OBS scene:
@@ -39,4 +51,3 @@ Husband PC - Voice       Media Source, audio only, monitor as needed
 ```
 
 Mute the video source if it carries no audio. Keep audio filters on the separate audio sources.
-

docs/obs-spout-streaming.md

@@ -42,7 +42,11 @@ Start the Spout sender:
   --height 1080 `
   --fps 60 `
   --status .\calibration\runs\stream-spout-status.json `
-  --status-cache .\calibration\runs\visual-stream-status.msgpack
+  --status-cache .\calibration\runs\visual-stream-status.msgpack `
+  --audio-cache .\calibration\runs\audio-state.msgpack `
+  --audio-events-cache .\calibration\runs\audio-events.msgpack `
+  --remote-video-url "srt://0.0.0.0:5100?mode=listener&latency=120000&timeout=5000000" `
+  --remote-video-latency-ms 250
 ```
 
 Start the current live fusion producer:
@@ -74,6 +78,14 @@ Fields:
 
 The typed status document is written to `calibration/runs/visual-stream-status.msgpack`. The JSON file is only the blunt terminal heartbeat.
 
+The sync heartbeat is written to `calibration/runs/av-sync-status.json`. It includes the audio frame delta and the remote SRT video artifact:
+
+- `remote_video.source_name`: OBS/Aquarium-facing source name.
+- `remote_video.url`: the SRT listener URL for the neighbor feed.
+- `remote_video.expected_latency_ns`: configured presentation delay, not just the SRT socket latency.
+- `remote_video.delta_ns`: difference between the render presentation clock and the expected remote video presentation time.
+- `remote_video.synchronized`: whether the delta is inside tolerance.
+
 ## Invariants
 
 - OBS receives a named Spout texture.
@@ -81,6 +93,7 @@ The typed status document is written to `calibration/runs/visual-stream-status.m
 - CultCache MessagePack documents are the live local state authority. CultNet document replication is the process/network boundary.
 - JSON render-frame polling is compatibility scaffolding only.
 - Latency is explicit in the packet timestamps. The renderer may buffer, but it may not silently erase the visual/audio alignment fields.
+- The neighbor SRT video feed is a timed artifact. OBS must not be left to eyeball-sync it against the Spout texture and AmbiX bed.
 
 ## Aquarium Cut
 

docs/typed-visual-state.md

@@ -60,10 +60,24 @@ For the deadline rig, both producer and renderer share the local CultCache file.
 
 ## Audio Synchronization
 
-`stream_spout.py` can now read `calibration/runs/audio-state.msgpack` and `calibration/runs/audio-events.msgpack` while it renders the visual frame. It selects source events against `RenderFramePacket.audio_alignment_time_ns`, adds synchronized audio-event points to the render packet, and writes `calibration/runs/av-sync-status.json` with the active visual frame id, audio frame id, audio delta, event count, and overlay count.
+`stream_spout.py` can now read `calibration/runs/audio-state.msgpack` and `calibration/runs/audio-events.msgpack` while it renders the visual frame. It selects source events against `RenderFramePacket.audio_alignment_time_ns`, adds synchronized audio-event points to the render packet, and writes `calibration/runs/av-sync-status.json` with the active visual frame id, audio frame id, audio delta, event count, overlay count, and remote video sync status.
 
 The audio bed still travels as `localcast.audio.spatial_frame`; the visual render packet only receives renderer-visible transient geometry. That keeps the machine legible: audio owns sound, source-event analysis owns acoustic facts, Aquarium owns the final pixel/audio package for OBS.
 
+The neighbor SRT video feed is also a timed media artifact. For the deadline path, `stream_spout.py` records its URL and expected SRT latency in the sync status:
+
+```powershell
+.\.venv\Scripts\python.exe .\scripts\stream_spout.py `
+  --source cultcache `
+  --frame-cache .\calibration\runs\visual-state.msgpack `
+  --audio-cache .\calibration\runs\audio-state.msgpack `
+  --audio-events-cache .\calibration\runs\audio-events.msgpack `
+  --remote-video-url "srt://0.0.0.0:5100?mode=listener&latency=120000&timeout=5000000" `
+  --remote-video-latency-ms 250
+```
+
+That does not make the SRT feed disappear into the point renderer. It makes the dependency honest: OBS/Aquarium must delay or present the remote video according to the same presentation clock as the Spout texture and AmbiX bed. This value is presentation delay, not only the SRT socket's `latency` parameter.
+
 ## Cut Line
 
 - Keep: typed CultCache document contracts and MessagePack payloads.
@@ -71,3 +85,4 @@ The audio bed still travels as `localcast.audio.spatial_frame`; the visual rende
 - Cut next: JSON render-frame polling once no script depends on it.
 - Do not add a second scene authority. Sensor fusion owns world claims; renderer owns pixels.
 - Do not make OBS synchronize separate sources. Aquarium receives synchronized visual/audio documents and emits the OBS-facing package.
+- Do not treat the neighbor SRT video feed as untimed scenery. It is a presentation artifact and belongs in the same sync status as audio and render frames.

localcast/sensor_fusion/__init__.py

@@ -13,6 +13,7 @@ from .core import (
 )
 from .render_bridge import RenderBridgeConfig, RenderFramePacket, RenderPointPacket, lower_points_to_render_frame
 from .audio_overlay import AudioVisualSyncStatus, overlay_audio_events
+from .media_artifacts import RemoteVideoArtifact, remote_video_artifact_for_present_time
 from .cultcache_docs import (
     LIVE_RENDER_FRAME_KEY,
     CultRenderFrame,
@@ -37,6 +38,7 @@ __all__ = [
     "RenderFramePacket",
     "RenderPointPacket",
     "AudioVisualSyncStatus",
+    "RemoteVideoArtifact",
     "LIVE_RENDER_FRAME_KEY",
     "CultRenderFrame",
     "CultRenderPoint",
@@ -48,5 +50,6 @@ __all__ = [
     "lower_frame_to_screen_brushes",
     "lower_points_to_render_frame",
     "overlay_audio_events",
+    "remote_video_artifact_for_present_time",
     "put_live_render_frame",
 ]

localcast/sensor_fusion/audio_overlay.py

@@ -7,6 +7,7 @@ import sys
 import numpy as np
 
 from .render_bridge import RenderFramePacket, RenderPointPacket
+from .media_artifacts import RemoteVideoArtifact
 
 
 ROOT = Path(__file__).resolve().parents[2]
@@ -25,6 +26,7 @@ class AudioVisualSyncStatus:
     audio_delta_ns: int | None
     source_event_count: int
     overlay_event_count: int
+    remote_video: RemoteVideoArtifact | None
     synchronized: bool
 
 
@@ -38,7 +40,7 @@ def overlay_audio_events(
     max_events: int = 128,
 ) -> tuple[RenderFramePacket, AudioVisualSyncStatus]:
     if events is None:
-        return frame, sync_status(frame, None, 0, 0, False)
+        return frame, sync_status(frame, None, 0, 0, None, False)
     overlay = select_events_for_visual_frame(frame, events, audio_frame, int(window_ns), int(max_events))
     points = list(frame.points)
     for event in overlay:
@@ -70,7 +72,7 @@ def overlay_audio_events(
             target_height=frame.target_height,
             points=tuple(points),
         ),
-        sync_status(frame, audio_frame, len(events.events), len(overlay), True),
+        sync_status(frame, audio_frame, len(events.events), len(overlay), None, True),
     )
 
 
@@ -114,9 +116,11 @@ def sync_status(
     audio_frame: CultSpatialAudioFrame | None,
     source_event_count: int,
     overlay_event_count: int,
+    remote_video: RemoteVideoArtifact | None,
     synchronized: bool,
 ) -> AudioVisualSyncStatus:
     audio_delta_ns = None if audio_frame is None else frame.audio_alignment_time_ns - audio_frame.audio_time_ns
+    remote_ok = True if remote_video is None else remote_video.synchronized
     return AudioVisualSyncStatus(
         visual_frame_id=frame.frame_id,
         visual_present_time_ns=frame.present_time_ns,
@@ -125,6 +129,23 @@ def sync_status(
         audio_delta_ns=audio_delta_ns,
         source_event_count=source_event_count,
         overlay_event_count=overlay_event_count,
-        synchronized=bool(synchronized and audio_frame is not None),
+        remote_video=remote_video,
+        synchronized=bool(synchronized and audio_frame is not None and remote_ok),
     )
 
+
+def with_remote_video_status(status: AudioVisualSyncStatus | None, remote_video: RemoteVideoArtifact | None) -> AudioVisualSyncStatus | None:
+    if status is None:
+        return None
+    remote_ok = True if remote_video is None else remote_video.synchronized
+    return AudioVisualSyncStatus(
+        visual_frame_id=status.visual_frame_id,
+        visual_present_time_ns=status.visual_present_time_ns,
+        audio_frame_id=status.audio_frame_id,
+        audio_time_ns=status.audio_time_ns,
+        audio_delta_ns=status.audio_delta_ns,
+        source_event_count=status.source_event_count,
+        overlay_event_count=status.overlay_event_count,
+        remote_video=remote_video,
+        synchronized=bool(status.synchronized and remote_ok),
+    )

localcast/sensor_fusion/media_artifacts.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class RemoteVideoArtifact:
+    source_name: str
+    url: str
+    expected_latency_ns: int
+    present_time_ns: int
+    delta_ns: int
+    synchronized: bool
+
+
+def remote_video_artifact_for_present_time(
+    *,
+    source_name: str,
+    url: str,
+    present_time_ns: int,
+    observed_time_ns: int,
+    expected_latency_ns: int,
+    tolerance_ns: int = 120_000_000,
+) -> RemoteVideoArtifact:
+    media_present_time = int(observed_time_ns) + int(expected_latency_ns)
+    delta_ns = int(present_time_ns) - media_present_time
+    return RemoteVideoArtifact(
+        source_name=str(source_name),
+        url=str(url),
+        expected_latency_ns=int(expected_latency_ns),
+        present_time_ns=media_present_time,
+        delta_ns=delta_ns,
+        synchronized=abs(delta_ns) <= int(tolerance_ns),
+    )
+

localcast/sensor_fusion/spout_output.py

@@ -10,7 +10,8 @@ import time
 
 import numpy as np
 
-from .audio_overlay import AudioVisualSyncStatus, overlay_audio_events
+from .audio_overlay import AudioVisualSyncStatus, overlay_audio_events, with_remote_video_status
+from .media_artifacts import remote_video_artifact_for_present_time
 from .render_bridge import RenderFramePacket, RenderPointPacket, load_render_frame
 from .cultcache_docs import CultStreamStatus, get_live_render_frame, put_stream_status
 
@@ -62,6 +63,10 @@ class CultCacheRenderFrameSource:
         audio_cache: Path | None = None,
         audio_events_cache: Path | None = None,
         audio_event_window_ns: int = 120_000_000,
+        remote_video_name: str | None = None,
+        remote_video_url: str | None = None,
+        remote_video_latency_ns: int = 120_000_000,
+        remote_video_tolerance_ns: int = 120_000_000,
     ) -> None:
         self.path = path
         self.demo_if_missing = demo_if_missing
@@ -69,6 +74,10 @@ class CultCacheRenderFrameSource:
         self.audio_cache = audio_cache
         self.audio_events_cache = audio_events_cache
         self.audio_event_window_ns = int(audio_event_window_ns)
+        self.remote_video_name = remote_video_name
+        self.remote_video_url = remote_video_url
+        self.remote_video_latency_ns = int(remote_video_latency_ns)
+        self.remote_video_tolerance_ns = int(remote_video_tolerance_ns)
         self._mtime_ns: int | None = None
         self._frame: RenderFramePacket | None = None
         self._audio_mtime_ns: int | None = None
@@ -101,6 +110,17 @@ class CultCacheRenderFrameSource:
             audio_frame,
             window_ns=self.audio_event_window_ns,
         )
+        remote_video = None
+        if self.remote_video_name and self.remote_video_url:
+            remote_video = remote_video_artifact_for_present_time(
+                source_name=self.remote_video_name,
+                url=self.remote_video_url,
+                present_time_ns=frame.present_time_ns,
+                observed_time_ns=frame.source_time_max_ns,
+                expected_latency_ns=self.remote_video_latency_ns,
+                tolerance_ns=self.remote_video_tolerance_ns,
+            )
+        status = with_remote_video_status(status, remote_video)
         self.last_sync_status = status
         return augmented
 
@@ -285,6 +305,16 @@ def write_sync_status(path: Path, status: AudioVisualSyncStatus | None) -> None:
                 "audio_delta_ns": status.audio_delta_ns,
                 "source_event_count": status.source_event_count,
                 "overlay_event_count": status.overlay_event_count,
+                "remote_video": None
+                if status.remote_video is None
+                else {
+                    "source_name": status.remote_video.source_name,
+                    "url": status.remote_video.url,
+                    "expected_latency_ns": status.remote_video.expected_latency_ns,
+                    "present_time_ns": status.remote_video.present_time_ns,
+                    "delta_ns": status.remote_video.delta_ns,
+                    "synchronized": status.remote_video.synchronized,
+                },
                 "synchronized": status.synchronized,
                 "updated_monotonic_ns": time.monotonic_ns(),
             },

scripts/stream_spout.py

@@ -36,6 +36,11 @@ def main() -> None:
     parser.add_argument("--audio-events-cache", default=str(ROOT / "calibration" / "runs" / "audio-events.msgpack"))
     parser.add_argument("--audio-event-window-ms", type=float, default=120.0)
     parser.add_argument("--no-audio-overlay", action="store_true")
+    parser.add_argument("--remote-video-name", default="Neighbor PC - Video")
+    parser.add_argument("--remote-video-url", default="srt://0.0.0.0:5100?mode=listener&latency=120000&timeout=5000000")
+    parser.add_argument("--remote-video-latency-ms", type=float, default=250.0)
+    parser.add_argument("--remote-video-tolerance-ms", type=float, default=120.0)
+    parser.add_argument("--no-remote-video-sync", action="store_true")
     parser.add_argument("--source", choices=["cultcache", "json"], default="cultcache")
     parser.add_argument("--duration", type=float, help="Optional run duration in seconds for smoke tests.")
     args = parser.parse_args()
@@ -61,6 +66,10 @@ def main() -> None:
             audio_cache=None if args.no_audio_overlay else Path(args.audio_cache),
             audio_events_cache=None if args.no_audio_overlay else Path(args.audio_events_cache),
             audio_event_window_ns=int(float(args.audio_event_window_ms) * 1_000_000),
+            remote_video_name=None if args.no_remote_video_sync else args.remote_video_name,
+            remote_video_url=None if args.no_remote_video_sync else args.remote_video_url,
+            remote_video_latency_ns=int(float(args.remote_video_latency_ms) * 1_000_000),
+            remote_video_tolerance_ns=int(float(args.remote_video_tolerance_ms) * 1_000_000),
         )
         frame_source_name = str(frame_cache_path)
     else:

tests/test_sensor_fusion.py

@@ -21,6 +21,7 @@ from localcast.sensor_fusion import (
     lower_points_to_render_frame,
     overlay_audio_events,
     put_live_render_frame,
+    remote_video_artifact_for_present_time,
 )
 from audio_field.cultcache_audio import make_audio_source_events, make_spatial_audio_frame
 from localcast.sensor_fusion.adapters import read_raw_bgr_frames
@@ -232,6 +233,19 @@ class SensorFusionTests(unittest.TestCase):
         self.assertTrue(status.synchronized)
         self.assertEqual(1, status.overlay_event_count)
 
+    def test_remote_video_artifact_reports_sync_delta(self):
+        artifact = remote_video_artifact_for_present_time(
+            source_name="Neighbor PC - Video",
+            url="srt://0.0.0.0:5100?mode=listener&latency=120000",
+            present_time_ns=1_250_000_000,
+            observed_time_ns=1_100_000_000,
+            expected_latency_ns=120_000_000,
+            tolerance_ns=40_000_000,
+        )
+
+        self.assertEqual(30_000_000, artifact.delta_ns)
+        self.assertTrue(artifact.synchronized)
+
 
 if __name__ == "__main__":
     unittest.main()