workspace: name: LocalCastBridge status: active focus: perfect-machine-native-reservoir premise: > A second Windows PC should appear inside OBS as video plus separately mixable audio sources. Use ordinary media transport first; native OBS code only earns its keep after standard OBS/FFmpeg surfaces fail a real invariant. objective: primary: > Build one synchronized spatial stream machine from six cameras, six microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP, and OBS output. current: > Cut the deadline bridge down to tooling and define the native five-second reservoir contract Aquarium/Faust must implement. constraints: - Sender is Windows. - Receiver uses OBS. - Video encode must happen on the sender using NVENC when available. - Audio sources must remain separately controllable in OBS. - V1 is LAN-only. - Persistent state must clarify rehydrate, not become transcript tax. invariants: - OBS composition authority stays in OBS. - FFmpeg owns capture, encoding, and transport. - One stable port equals one OBS source endpoint. - Video endpoint uses H.264 over MPEG-TS/SRT. - Audio endpoints default to AAC over MPEG-TS/SRT for broad OBS/FFmpeg compatibility. - Local secrets remain untracked. accepted_design: - `docs/perfect-machine.md` and `config/perfect-machine.example.json` define the native target: one five-second spatiotemporal reservoir, Aquarium GPU fusion/rendering, Faust DSP, OBS output controls, and LocalCastBridge tooling. - `native/reservoir` is the first native Rust crate. It implements shared-edge reservoir rings so all sample types expire from the newest live sample rather than private source histories. - `config/localcast.example.json` defines receiver address, base port, video encode settings, and audio sources. - `scripts/sender-discover.ps1` lists FFmpeg devices, protocols, and encoders. - `scripts/sender-start.ps1` launches one FFmpeg process per enabled source. - OBS receives each source through Media Source URLs documented in `docs/obs-receiver-setup.md`. - Persistence follows the active repo pattern: map, scratch, evidence, branches, handoff, system map, implementation plan, state helpers. active_subgoals: - Implement the Perfect Machine native reservoir contract described by `docs/perfect-machine.md` and `config/perfect-machine.example.json`. - Move dense visual fusion/material fitting/brush generation into Aquarium GPU compute. - Move hot audio DSP/voice separation/room suppression/spatialization into Faust/native runtime. - Validate the actual sender FFmpeg build and capture device names. - Smoke-test the video endpoint in OBS. - Smoke-test each audio endpoint in OBS and tune latency. - Decide whether an OBS source generator or plugin investigation is justified. - Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock. - Build the six-microphone/two-speaker audio-field sidecar around the actual distributed camera/Focusrite clocks, explicit geometry, speaker calibration, bounded buffering, delay/SRO alignment, testable pipeline seams, and FOA AmbiX output. - Build the visual sensor-fusion sidecar around timestamped observations, calibrated sensor models, bounded track caches, sparse point-cloud output, and later brush/splat packet lowering. research: ambisonic_aquasynth: summary: research/ambisonic-aquasynth/summary.md bibliography: research/ambisonic-aquasynth/bibliography.md key_decision: > A spatial field must be built from one clocked multichannel capture path feeding a declared ambisonic bus. The existing one-audio-source-per-SRT endpoint design remains valid for plain OBS mixing, but it is not the authority for synchronized ambisonic DSP. live_adaptive_sync: summary: research/live-adaptive-sync/summary.md bibliography: research/live-adaptive-sync/bibliography.md key_decision: > If independent mic clocks must be used, sync needs two explicit loops: slow sampling-rate-offset estimation driving variable-rate resampling, and faster delay/TDOA tracking for current alignment and localization. Prove this with a two-mic harness before connecting it to Ambisonics. playback_calibration: summary: research/playback-calibration/summary.md bibliography: research/playback-calibration/bibliography.md key_decision: > Speaker output is a layout-specific render stage, not a generic sink. The system needs a playback profile with decoder matrix/layout, output-device latency, channel mapping, per-speaker delay/gain/response correction, and a latency ledger. Keep playback calibration separate from capture-side adaptive sync. feedback_calibration: summary: research/feedback-calibration/summary.md bibliography: research/feedback-calibration/bibliography.md key_decision: > Speaker feedback is also a calibration signal when the outgoing audio is known. Model it with an online speaker-to-mic transfer matrix and expose confidence/convergence state. Keep this estimator separate from clock sync, localization, and Ambisonic decoding. visual_spatial_map: summary: research/visual-spatial-map/summary.md bibliography: research/visual-spatial-map/bibliography.md brushstroke_rendering: research/visual-spatial-map/brushstroke-fusion-rendering.md key_decision: > The visual rig is two PlayStation Eye / PS3 Eye cameras as high-rate low-resolution marker trackers, two Razer Kiyo-class RGB/mic cameras including one Kiyo Pro, and one Leap Motion/LeapUVC stereo IR sensor for near-field fusion. LeapUVC driver mode makes Leap appear as a generic UVC device and should be treated as mutually exclusive with Ultraleap software tracking unless proven otherwise. Build one canonical world coordinate system for all visual sensors, microphones, speakers, props, virtual camera pose, and ambisonic listener pose. Prove sparse calibrated tracking before adding RGB projection or Gaussian splat rendering. For a Dreams-like volumetric brushstroke look, keep the world model as truth and lower semantic scene claims into cached brush/splat packets under budget. gpu_fusion_splatting: summary: research/gpu-fusion-splatting/summary.md bibliography: research/gpu-fusion-splatting/bibliography.md key_decision: > The live visual fusion/render hot path should be GPU-resident: pinned host frame rings, async CUDA uploads, GPU marker/stereo/feature extraction, GPU-shaped observation and brush packet buffers, and CUDA/Vulkan/D3D splat rendering with minimal readback. CPU owns orchestration and semantics; kernels do not own calibration truth. spout_obs_bridge: summary: research/spout-obs-bridge/summary.md bibliography: research/spout-obs-bridge/bibliography.md key_decision: > OBS should receive a named Spout2 shared texture from Aquarium Engine, not desktop capture or file polling. LocalCastBridge emits typed render-frame packets with source timestamps, intended presentation time, audio-alignment time, target size, Spout sender name, and point/brush claims. The deadline Spout sink now defaults to a `kiyo-mid-deru` virtual camera, with the eye halfway between the two Kiyo-class cameras and the target on the co-streamer body volume. Its point budget is temporal: stable high-confidence anchors plus a frame-varying remainder for TAA/supersampling. Aquarium owns GPU rendering and Spout publication. live_spout_deadline_sink: summary: docs/obs-spout-streaming.md key_decision: > The deadline OBS surface is a real Spout sender named LocalCastBridge Point Cloud. `scripts/stream_spout.py` consumes render-frame packets, lowers them into compact anisotropic screen brush packets inspired by the Zyphos/Aquarium brush model, uploads the texture through SpoutGL, and writes a heartbeat status file. This is a sink boundary, not a new scene authority; Aquarium should replace the renderer body behind the same RenderFramePacket contract for dense brush/splat rendering. live_audio_phase_meaning: summary: docs/audio-field.md key_decision: > Phase/chirplet evidence now has a live semantic boundary: `audio_field.phase_meaning` keeps raw phase internal and publishes `localcast.audio.phase_field` with delay correction, coherence, confidence, distance-equivalent delta, reference-bleed/suppression weights, correction energy, and active-probe need. `audio_field.active_probe` closes the next loop by feeding that confidence state into ActiveProbeOptimizer and emitting bounded chirplet WAVs/manifests, with optional playback. The live launcher now uses dense harmonic near-ultrasonic probe textures so aliases have a musical relationship instead of arbitrary squeal. Probe artifacts rotate through fixed slots and the manifest rotates at a byte cap. `scripts/stream_live_audio_field.py` is the current live local capture owner: it writes mic-field and phase-field CultCache documents from visible local mics plus Scarlett loopback, sends probes through the selected Scarlett output, resamples probe playback when that output is 44.1 kHz, and reports missing distributed channels as explicit placeholders. Active probe scheduling is restricted to source ids with live local capture devices, so placeholders do not consume chirp budget. `audio-phase-field-status.json` declares input mode and whether active probes are closed-loop; current deadline mode is `live-local-capture` with closed-loop probe capture when Scarlett loopback is available. This is learned audio-field state, not a phase display. faust_voice_separation: summary: docs/aquarium-spatial-audio.md key_decision: > Voice separation should run in Aquarium's Faust DSP, not as Python hot audio math after the mic field has already collapsed to AmbiX. LocalCastBridge now publishes `localcast.audio.mic_field` with six aligned mic channels, roles, controls, and graph id `localcast.faust.voice_separation.v1`; the matching Faust graph lives at `faust/localcast_voice_separation.dsp`. sensor_fusion_architecture: summary: docs/sensor-fusion-architecture.md key_decision: > The visual world model begins as timestamped sensor observations and calibrated camera transforms, not render packets. The first sidecar keeps capture adapters, detection, triangulation, track caching, and point-cloud export as separate testable authorities. This follows the fractal-domain cache rule: semantic claims first, cached backend lowerings second. Stochastic transient matching now lowers descriptor-consistent image observations into scene-ray hypotheses and a multi-LOD world-space cache for Aquarium compute-shader reconciliation. Leap evidence is the highest-priority temporal/spatial source when live Leap frames are present; fallback Leap frames stay tagged as fallback evidence and must not be treated as ground truth. Live clap calibration now uses the Kiyo stereo pair for the first rough 3D impact solve and includes Leap motion as the high-precision visual timing witness; Leap should become a geometric camera authority after shared calibration. Clap peaks update per-camera clock offsets, allowing history buffers to be sampled at `oracle_time - camera_offset` for coherent point-cloud construction. `localcast.sensor_fusion.chirp_pose` converts live phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints and per-camera pose-correction estimates, which the live producer emits as `camera-chirp:*` and `camera-pose-correction:*` render points plus `chirp-camera-pose` LOD evidence. It can load geometry from the audio-field profile, falling back to the example/default geometry until measured local coordinates exist. These constraints augment visual calibration; they do not own image geometry. The Spout point-budget policy pins calibration and cross-modal constraints before spending adaptive LOD budget on stable high-priority surfaces and frame-rotated samples, and its JSON heartbeat reports budgeted point prefix counts. The live fusion producer now has an explicit five-second spatiotemporal reservoir: RGB/Leap frame history expires from the newest shared sample, LOD evidence is clipped to the same window, and each RenderFramePacket declares that source window for Aquarium/OBS synchronization. Multi-LOD cells now carry first-pass material estimates, weighted albedo plus roughness/metallic hints, so Aquarium can relight/refine brush packets from reservoir state instead of scraping screen-space color. The Spout/audio overlay boundary clamps stale visual packets to the current audio reservoir edge before OBS sees them. Current live deadline mode uses fallback-only RGB/Leap because synchronous OpenCV camera reads measured slower than the reservoir; the next coherent cut is nonblocking camera ingress that continuously fills the five-second reservoir with real samples. `LatestFramePump` now moves blocking OpenCV reads off the fusion hot loop; the running deadline mode uses lower per-frame sample density so Spout can render visual points inside the five-second budget while TAA/reservoir accumulation carries detail over time. typed_visual_state: summary: docs/typed-visual-state.md key_decision: > Live visual state now lives in typed CultCache MessagePack documents: `localcast.visual.render_frame` in `calibration/runs/visual-state.msgpack` and `localcast.visual.stream_status` in `calibration/runs/visual-stream-status.msgpack`. CultNet document_put is the intended process/network API boundary. JSON render-frame files are no longer the authority. current_status: summary: > First repo scaffold is in place and deployed to the neighbor sender at C:\Meta\LocalCastBridge. FFmpeg 8.1.1 with SRT and h264_nvenc is installed there, direct WASAPI console-session capture now owns system-output loopback, Madman's desktop has start/stop launchers, the sender script has launcher-safe FFmpeg arguments and repo-root logs, desktop launchers use absolute tool paths, and the receiver OBS scene has matching Media Sources. Local calibration harness now has first physical-device access: LeapUVC raw IR video, two Razer Kiyo-class RGB cameras, all expected microphones, and Scarlett speaker output are accessible. LeapUVC MSMF mode probing accepts a 320x240x120 request and reports about 115 fps, though measured OpenCV throughput is lower in the first rough probe. The opentrack/libusb PS3 Eye path still enumerates modes but fails frame reads across libusb-win32, libusbK, and WinUSB. The AllanCat PS3 Eye Universal Driver / PS3EyeDirectShow path now exposes both PS3 Eyes as DirectShow indices 3 and 4 after registering the source filter for two cameras. OpenCV index capture is reliable at default 640x480 near 30 fps but does not negotiate high-rate modes correctly. FFmpeg DirectShow capture by device name negotiates the filter pins properly: both Eyes can run simultaneously at 320x240@60, and dual 640x480@60 lands around 54-56 fps per Eye. Requests above 60 do not increase delivered frame rate through the installed filter. Audio-field scaffolding now exists as a separate sidecar for the actual distributed rig: two Kiyo mics, two PS Eye mics, local Focusrite ADC with a shielded cardioid near the host's face, neighbor Focusrite ADC with a shotgun mic pointed at the co-streamer, and two local speakers. The Focusrite mics are high-quality dialogue anchors; camera mics provide spatial/context evidence. The tool refuses to treat independent USB/network clocks as an Ambisonic field until they have been aligned and resampled into one six-channel reference timeline. Local PortAudio/WASAPI probe shows the Scarlett input accepts 48 kHz but rejects 96 kHz, while the Scarlett output accepts 44.1 kHz and rejects 48/96 kHz on this path; check ASIO/native Focusrite access before relying on high-rate calibration. Audio latency is now an explicit bounded-buffering policy: output may run behind live input while delay/SRO estimates settle, but cache depth must converge toward real-time. Core audio modules now expose testable boundaries for buffering, latency convergence, injectable capture/alignment/encode/sink ports, and pipeline orchestration. Calibration analysis now has a local chirplet refinement pass: coarse sweep match followed by fractional-delay and chirp-rate grid search around the detected arrival. Runtime sync is now modeled as per-block known-speaker chirplet telemetry updating delay, SRO, phase, and confidence state; low-confidence observations freeze rather than poisoning alignment. Active probe optimization owns extra chirplet emission when sync confidence drops, bounded by spacing, level, and audibility budget. The first visual sensor-fusion sidecar is in place: `localcast.sensor_fusion.core` owns calibrated camera models, timestamped 2D observations, DLT triangulation, reprojection/confidence gating, a TTL track cache, and PLY point-cloud export; adapters own FFmpeg/raw-frame ingress; `scripts/sensor_fusion.py fuse-file` can turn an observation file into a sparse PLY. The render bridge now lowers cached points into `localcast.sensor_fusion.render_frame.v1` packets carrying source time ranges, intended visual presentation time, audio-alignment time, target render size, Spout sender name, and point draw metadata for Aquarium/Spout. A deadline OBS Spout sender is now live-shaped: `scripts/live_sensor_fusion.py` now writes fused render frames into typed CultCache state at `calibration/runs/visual-state.msgpack`. `scripts/stream_spout.py` consumes that CultCache document, lowers points into compact anisotropic screen brush packets, uploads an OpenGL texture, publishes Spout sender `LocalCastBridge Point Cloud`, and writes both typed status `calibration/runs/visual-stream-status.msgpack` and compatibility JSON `calibration/runs/stream-spout-status.json`; OBS now has a Spout2 Capture source pointed at that sender, and an OBS websocket screenshot verified nonblack brush output. OBS synchronized program setup now derives six per-channel stems from the aligned program audio timeline, creates/updates OBS Media Sources for host voice, co-streamer voice, ambient, transients, co-streamer loopback, and local loopback, and can disable every scene item except synchronized LocalCastBridge program surfaces. Co-streamer surface capture now records neighbor Focusrite and neighbor loopback while recording local loopback ground truth, then writes aligned co-streamer surface WAVs for the stem packer. Live phase-field meaning now writes `localcast.audio.phase_field` to `calibration/runs/audio-phase-field.msgpack` from internal chirplet/phase evidence so downstream code sees delay, coherence, confidence, suppression, correction, and probe-control state instead of raw phases. The neighbor desktop launcher no longer uses Voicemeeter; it restores the Focusrite speakers as the primary playback device and the Focusrite analogue input as the default capture device. A direct Core Audio WASAPI loopback shim now works when launched in the neighbor's interactive console session; SSH-only launches can see the endpoint but receive no render packets. Missing synchronized neighbor feeds are explicit silent placeholders, not raw unsynchronized substitutes. The live producer currently uses synthetic calibrated observations through the real `SensorRig.fuse` path until camera detector observations replace them. The RGB path now has a debug-only CPU dense stereo reference: the two Kiyo-class frames can be block-matched as a provisional calibrated pair and triangulated into `dense-rgb:*` surface claims, but production dense reconstruction must move to Aquarium/GPU compute rather than Python loops. A separate adaptive camera-control policy measures luminance, clipping, contrast, and sharpness, then emits normalized exposure/gain/focus commands through the OpenCV capture adapter when the driver accepts them. Active illumination now has an optional Tuya local-network pulse hook for deliberately modulating room light as calibration telemetry; pulse timestamps can be aligned against camera brightness/depth response. Common-space calibration now has a real owner: fixed-board ChArUco observations plus intrinsics solve `CameraModel.world_from_sensor` transforms, and cross-view surface-feature observations can be descriptor-matched and triangulated into world-space tracks. Unit tests inject synthetic cameras and frames so the core does not depend on live camera drivers. The multi-LOD visual cache now carries source kind and source priority so Aquarium can prefer real Leap evidence over RGB surfaces, stochastic transient hypotheses, and synthetic fusion support while still seeing lower-priority fallback diagnostics when the Leap capture path stalls. next_action: > Capture real ChArUco intrinsics for the PS3 Eyes and Kiyos, capture fixed-board observations from every camera, solve a local `config/sensor-fusion.json`, and prove calibrated cross-view surface tracks with `scripts/triangulate_surface_features.py`. Then move the dense RGB stereo matcher to a GPU-resident stereo/flow pass so high-detail surface claims can scale toward the million-splat temporal accumulator without Python owning each point. If a Tuya light is available, run the local-key pulse hook and record its timestamps against camera responses as active illumination evidence. Then replace the synthetic live producer with a detector adapter that turns FFmpeg PS3 Eye frames into marker `Observation2D` records for the fusion core and publishes the same typed CultCache/CultNet render-frame document. Calibrate Leap as a geometric camera model so clap timing peaks can become direct 3D rays instead of timing-only evidence. Also fix the live Leap capture backend so the cache emits `leap-ground-truth` rather than `leap-fallback`; current MSMF probing can still stall on frame reads. In parallel, move the render-frame consumer into Aquarium Engine so Aquarium renders cached points/brushes into its GPU render target and publishes or hands off the same Spout sender `LocalCastBridge Point Cloud` for OBS, replacing the deadline OpenGL point sink without changing packet ownership. In parallel, create local `config/audio-field.json`, confirm local Kiyo/PS Eye/Focusrite device matches, confirm the neighbor Focusrite shotgun capture/transport path, replace placeholder mic/speaker geometry with measured world coordinates, then build the delay/SRO alignment stage that feeds the bounded field cache and emits aligned six-channel blocks for FOA encoding. Keep neighbor loopback on the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present; do not reintroduce a virtual mixer unless direct loopback fails a concrete invariant that the mixer actually satisfies.