workspace: name: LocalCastBridge status: active focus: local-obs-lan-source premise: > A second Windows PC should appear inside OBS as video plus separately mixable audio sources. Use ordinary media transport first; native OBS code only earns its keep after standard OBS/FFmpeg surfaces fail a real invariant. objective: primary: > Provide a repeatable local streaming environment where the sender captures desktop/video and audio sources, encodes video with NVENC, and delivers stable OBS Media Source URLs to the receiver over LAN. current: > Establish the script-and-doc spine for a two-PC smoke test. constraints: - Sender is Windows. - Receiver uses OBS. - Video encode must happen on the sender using NVENC when available. - Audio sources must remain separately controllable in OBS. - V1 is LAN-only. - Persistent state must clarify rehydrate, not become transcript tax. invariants: - OBS composition authority stays in OBS. - FFmpeg owns capture, encoding, and transport. - One stable port equals one OBS source endpoint. - Video endpoint uses H.264 over MPEG-TS/SRT. - Audio endpoints default to AAC over MPEG-TS/SRT for broad OBS/FFmpeg compatibility. - Local secrets remain untracked. accepted_design: - `config/localcast.example.json` defines receiver address, base port, video encode settings, and audio sources. - `scripts/sender-discover.ps1` lists FFmpeg devices, protocols, and encoders. - `scripts/sender-start.ps1` launches one FFmpeg process per enabled source. - OBS receives each source through Media Source URLs documented in `docs/obs-receiver-setup.md`. - Persistence follows the active repo pattern: map, scratch, evidence, branches, handoff, system map, implementation plan, state helpers. active_subgoals: - Validate the actual sender FFmpeg build and capture device names. - Smoke-test the video endpoint in OBS. - Smoke-test each audio endpoint in OBS and tune latency. - Decide whether an OBS source generator or plugin investigation is justified. - Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock. - Build the six-microphone/two-speaker audio-field sidecar around the actual distributed camera/Focusrite clocks, explicit geometry, speaker calibration, bounded buffering, delay/SRO alignment, testable pipeline seams, and FOA AmbiX output. - Build the visual sensor-fusion sidecar around timestamped observations, calibrated sensor models, bounded track caches, sparse point-cloud output, and later brush/splat packet lowering. research: ambisonic_aquasynth: summary: research/ambisonic-aquasynth/summary.md bibliography: research/ambisonic-aquasynth/bibliography.md key_decision: > A spatial field must be built from one clocked multichannel capture path feeding a declared ambisonic bus. The existing one-audio-source-per-SRT endpoint design remains valid for plain OBS mixing, but it is not the authority for synchronized ambisonic DSP. live_adaptive_sync: summary: research/live-adaptive-sync/summary.md bibliography: research/live-adaptive-sync/bibliography.md key_decision: > If independent mic clocks must be used, sync needs two explicit loops: slow sampling-rate-offset estimation driving variable-rate resampling, and faster delay/TDOA tracking for current alignment and localization. Prove this with a two-mic harness before connecting it to Ambisonics. playback_calibration: summary: research/playback-calibration/summary.md bibliography: research/playback-calibration/bibliography.md key_decision: > Speaker output is a layout-specific render stage, not a generic sink. The system needs a playback profile with decoder matrix/layout, output-device latency, channel mapping, per-speaker delay/gain/response correction, and a latency ledger. Keep playback calibration separate from capture-side adaptive sync. feedback_calibration: summary: research/feedback-calibration/summary.md bibliography: research/feedback-calibration/bibliography.md key_decision: > Speaker feedback is also a calibration signal when the outgoing audio is known. Model it with an online speaker-to-mic transfer matrix and expose confidence/convergence state. Keep this estimator separate from clock sync, localization, and Ambisonic decoding. visual_spatial_map: summary: research/visual-spatial-map/summary.md bibliography: research/visual-spatial-map/bibliography.md brushstroke_rendering: research/visual-spatial-map/brushstroke-fusion-rendering.md key_decision: > The visual rig is two PlayStation Eye / PS3 Eye cameras as high-rate low-resolution marker trackers, two Razer Kiyo-class RGB/mic cameras including one Kiyo Pro, and one Leap Motion/LeapUVC stereo IR sensor for near-field fusion. LeapUVC driver mode makes Leap appear as a generic UVC device and should be treated as mutually exclusive with Ultraleap software tracking unless proven otherwise. Build one canonical world coordinate system for all visual sensors, microphones, speakers, props, virtual camera pose, and ambisonic listener pose. Prove sparse calibrated tracking before adding RGB projection or Gaussian splat rendering. For a Dreams-like volumetric brushstroke look, keep the world model as truth and lower semantic scene claims into cached brush/splat packets under budget. gpu_fusion_splatting: summary: research/gpu-fusion-splatting/summary.md bibliography: research/gpu-fusion-splatting/bibliography.md key_decision: > The live visual fusion/render hot path should be GPU-resident: pinned host frame rings, async CUDA uploads, GPU marker/stereo/feature extraction, GPU-shaped observation and brush packet buffers, and CUDA/Vulkan/D3D splat rendering with minimal readback. CPU owns orchestration and semantics; kernels do not own calibration truth. spout_obs_bridge: summary: research/spout-obs-bridge/summary.md bibliography: research/spout-obs-bridge/bibliography.md key_decision: > OBS should receive a named Spout2 shared texture from Aquarium Engine, not desktop capture or file polling. LocalCastBridge emits typed render-frame packets with source timestamps, intended presentation time, audio-alignment time, target size, Spout sender name, and point/brush claims. Aquarium owns GPU rendering and Spout publication. live_spout_deadline_sink: summary: docs/obs-spout-streaming.md key_decision: > The deadline OBS surface is a real Spout sender named LocalCastBridge Point Cloud. `scripts/stream_spout.py` consumes render-frame packets, lowers them into compact anisotropic screen brush packets inspired by the Zyphos/Aquarium brush model, uploads the texture through SpoutGL, and writes a heartbeat status file. This is a sink boundary, not a new scene authority; Aquarium should replace the renderer body behind the same RenderFramePacket contract for dense brush/splat rendering. live_audio_phase_meaning: summary: docs/audio-field.md key_decision: > Phase/chirplet evidence now has a live semantic boundary: `audio_field.phase_meaning` keeps raw phase internal and publishes `localcast.audio.phase_field` with delay correction, coherence, confidence, distance-equivalent delta, reference-bleed/suppression weights, correction energy, and active-probe need. `audio_field.active_probe` closes the next loop by feeding that confidence state into ActiveProbeOptimizer and emitting bounded chirplet WAVs/manifests, with optional default-device playback. The live launcher now uses dense harmonic near-ultrasonic probe textures so aliases have a musical relationship instead of arbitrary squeal. This is learned audio-field state, not a phase display. faust_voice_separation: summary: docs/aquarium-spatial-audio.md key_decision: > Voice separation should run in Aquarium's Faust DSP, not as Python hot audio math after the mic field has already collapsed to AmbiX. LocalCastBridge now publishes `localcast.audio.mic_field` with six aligned mic channels, roles, controls, and graph id `localcast.faust.voice_separation.v1`; the matching Faust graph lives at `faust/localcast_voice_separation.dsp`. sensor_fusion_architecture: summary: docs/sensor-fusion-architecture.md key_decision: > The visual world model begins as timestamped sensor observations and calibrated camera transforms, not render packets. The first sidecar keeps capture adapters, detection, triangulation, track caching, and point-cloud export as separate testable authorities. This follows the fractal-domain cache rule: semantic claims first, cached backend lowerings second. typed_visual_state: summary: docs/typed-visual-state.md key_decision: > Live visual state now lives in typed CultCache MessagePack documents: `localcast.visual.render_frame` in `calibration/runs/visual-state.msgpack` and `localcast.visual.stream_status` in `calibration/runs/visual-stream-status.msgpack`. CultNet document_put is the intended process/network API boundary. JSON render-frame files are no longer the authority. current_status: summary: > First repo scaffold is in place and deployed to the neighbor sender at C:\Meta\LocalCastBridge. FFmpeg 8.1.1 with SRT and h264_nvenc is installed there, direct WASAPI console-session capture now owns system-output loopback, Madman's desktop has start/stop launchers, the sender script has launcher-safe FFmpeg arguments and repo-root logs, desktop launchers use absolute tool paths, and the receiver OBS scene has matching Media Sources. Local calibration harness now has first physical-device access: LeapUVC raw IR video, two Razer Kiyo-class RGB cameras, all expected microphones, and Scarlett speaker output are accessible. LeapUVC MSMF mode probing accepts a 320x240x120 request and reports about 115 fps, though measured OpenCV throughput is lower in the first rough probe. The opentrack/libusb PS3 Eye path still enumerates modes but fails frame reads across libusb-win32, libusbK, and WinUSB. The AllanCat PS3 Eye Universal Driver / PS3EyeDirectShow path now exposes both PS3 Eyes as DirectShow indices 3 and 4 after registering the source filter for two cameras. OpenCV index capture is reliable at default 640x480 near 30 fps but does not negotiate high-rate modes correctly. FFmpeg DirectShow capture by device name negotiates the filter pins properly: both Eyes can run simultaneously at 320x240@60, and dual 640x480@60 lands around 54-56 fps per Eye. Requests above 60 do not increase delivered frame rate through the installed filter. Audio-field scaffolding now exists as a separate sidecar for the actual distributed rig: two Kiyo mics, two PS Eye mics, local Focusrite ADC with a shielded cardioid near the host's face, neighbor Focusrite ADC with a shotgun mic pointed at the co-streamer, and two local speakers. The Focusrite mics are high-quality dialogue anchors; camera mics provide spatial/context evidence. The tool refuses to treat independent USB/network clocks as an Ambisonic field until they have been aligned and resampled into one six-channel reference timeline. Local PortAudio/WASAPI probe shows the Scarlett input accepts 48 kHz but rejects 96 kHz, while the Scarlett output accepts 44.1 kHz and rejects 48/96 kHz on this path; check ASIO/native Focusrite access before relying on high-rate calibration. Audio latency is now an explicit bounded-buffering policy: output may run behind live input while delay/SRO estimates settle, but cache depth must converge toward real-time. Core audio modules now expose testable boundaries for buffering, latency convergence, injectable capture/alignment/encode/sink ports, and pipeline orchestration. Calibration analysis now has a local chirplet refinement pass: coarse sweep match followed by fractional-delay and chirp-rate grid search around the detected arrival. Runtime sync is now modeled as per-block known-speaker chirplet telemetry updating delay, SRO, phase, and confidence state; low-confidence observations freeze rather than poisoning alignment. Active probe optimization owns extra chirplet emission when sync confidence drops, bounded by spacing, level, and audibility budget. The first visual sensor-fusion sidecar is in place: `localcast.sensor_fusion.core` owns calibrated camera models, timestamped 2D observations, DLT triangulation, reprojection/confidence gating, a TTL track cache, and PLY point-cloud export; adapters own FFmpeg/raw-frame ingress; `scripts/sensor_fusion.py fuse-file` can turn an observation file into a sparse PLY. The render bridge now lowers cached points into `localcast.sensor_fusion.render_frame.v1` packets carrying source time ranges, intended visual presentation time, audio-alignment time, target render size, Spout sender name, and point draw metadata for Aquarium/Spout. A deadline OBS Spout sender is now live-shaped: `scripts/live_sensor_fusion.py` now writes fused render frames into typed CultCache state at `calibration/runs/visual-state.msgpack`. `scripts/stream_spout.py` consumes that CultCache document, lowers points into compact anisotropic screen brush packets, uploads an OpenGL texture, publishes Spout sender `LocalCastBridge Point Cloud`, and writes both typed status `calibration/runs/visual-stream-status.msgpack` and compatibility JSON `calibration/runs/stream-spout-status.json`; OBS now has a Spout2 Capture source pointed at that sender, and an OBS websocket screenshot verified nonblack brush output. OBS synchronized program setup now derives six per-channel stems from the aligned program audio timeline, creates/updates OBS Media Sources for host voice, co-streamer voice, ambient, transients, co-streamer loopback, and local loopback, and can disable every scene item except synchronized LocalCastBridge program surfaces. Co-streamer surface capture now records neighbor Focusrite and neighbor loopback while recording local loopback ground truth, then writes aligned co-streamer surface WAVs for the stem packer. Live phase-field meaning now writes `localcast.audio.phase_field` to `calibration/runs/audio-phase-field.msgpack` from internal chirplet/phase evidence so downstream code sees delay, coherence, confidence, suppression, correction, and probe-control state instead of raw phases. The neighbor desktop launcher no longer uses Voicemeeter; it restores the Focusrite speakers as the primary playback device and the Focusrite analogue input as the default capture device. A direct Core Audio WASAPI loopback shim now works when launched in the neighbor's interactive console session; SSH-only launches can see the endpoint but receive no render packets. Missing synchronized neighbor feeds are explicit silent placeholders, not raw unsynchronized substitutes. The live producer currently uses synthetic calibrated observations through the real `SensorRig.fuse` path until camera detector observations replace them. The RGB path now has a debug-only CPU dense stereo reference: the two Kiyo-class frames can be block-matched as a provisional calibrated pair and triangulated into `dense-rgb:*` surface claims, but production dense reconstruction must move to Aquarium/GPU compute rather than Python loops. A separate adaptive camera-control policy measures luminance, clipping, contrast, and sharpness, then emits normalized exposure/gain/focus commands through the OpenCV capture adapter when the driver accepts them. Active illumination now has an optional Tuya local-network pulse hook for deliberately modulating room light as calibration telemetry; pulse timestamps can be aligned against camera brightness/depth response. Common-space calibration now has a real owner: fixed-board ChArUco observations plus intrinsics solve `CameraModel.world_from_sensor` transforms, and cross-view surface-feature observations can be descriptor-matched and triangulated into world-space tracks. Unit tests inject synthetic cameras and frames so the core does not depend on live camera drivers. next_action: > Capture real ChArUco intrinsics for the PS3 Eyes and Kiyos, capture fixed-board observations from every camera, solve a local `config/sensor-fusion.json`, and prove calibrated cross-view surface tracks with `scripts/triangulate_surface_features.py`. Then move the dense RGB stereo matcher to a GPU-resident stereo/flow pass so high-detail surface claims can scale toward the million-splat temporal accumulator without Python owning each point. If a Tuya light is available, run the local-key pulse hook and record its timestamps against camera responses as active illumination evidence. Then replace the synthetic live producer with a detector adapter that turns FFmpeg PS3 Eye frames into marker `Observation2D` records for the fusion core and publishes the same typed CultCache/CultNet render-frame document. In parallel, move the render-frame consumer into Aquarium Engine so Aquarium renders cached points/brushes into its GPU render target and publishes or hands off the same Spout sender `LocalCastBridge Point Cloud` for OBS, replacing the deadline OpenGL point sink without changing packet ownership. In parallel, create local `config/audio-field.json`, confirm local Kiyo/PS Eye/Focusrite device matches, confirm the neighbor Focusrite shotgun capture/transport path, replace placeholder mic/speaker geometry with measured world coordinates, then build the delay/SRO alignment stage that feeds the bounded field cache and emits aligned six-channel blocks for FOA encoding. Keep neighbor loopback on the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present; do not reintroduce a virtual mixer unless direct loopback fails a concrete invariant that the mixer actually satisfies.