187 lines
11 KiB
YAML
187 lines
11 KiB
YAML
workspace:
|
|
name: LocalCastBridge
|
|
status: active
|
|
focus: local-obs-lan-source
|
|
premise: >
|
|
A second Windows PC should appear inside OBS as video plus separately
|
|
mixable audio sources. Use ordinary media transport first; native OBS code
|
|
only earns its keep after standard OBS/FFmpeg surfaces fail a real invariant.
|
|
|
|
objective:
|
|
primary: >
|
|
Provide a repeatable local streaming environment where the sender captures
|
|
desktop/video and audio sources, encodes video with NVENC, and delivers
|
|
stable OBS Media Source URLs to the receiver over LAN.
|
|
current: >
|
|
Establish the script-and-doc spine for a two-PC smoke test.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must happen on the sender using NVENC when available.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport.
|
|
- One stable port equals one OBS source endpoint.
|
|
- Video endpoint uses H.264 over MPEG-TS/SRT.
|
|
- Audio endpoints default to AAC over MPEG-TS/SRT for broad OBS/FFmpeg compatibility.
|
|
- Local secrets remain untracked.
|
|
|
|
accepted_design:
|
|
- `config/localcast.example.json` defines receiver address, base port, video encode settings, and audio sources.
|
|
- `scripts/sender-discover.ps1` lists FFmpeg devices, protocols, and encoders.
|
|
- `scripts/sender-start.ps1` launches one FFmpeg process per enabled source.
|
|
- OBS receives each source through Media Source URLs documented in `docs/obs-receiver-setup.md`.
|
|
- Persistence follows the active repo pattern: map, scratch, evidence, branches, handoff, system map, implementation plan, state helpers.
|
|
|
|
active_subgoals:
|
|
- Validate the actual sender FFmpeg build and capture device names.
|
|
- Smoke-test the video endpoint in OBS.
|
|
- Smoke-test each audio endpoint in OBS and tune latency.
|
|
- Decide whether an OBS source generator or plugin investigation is justified.
|
|
- Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock.
|
|
- Build the six-microphone/two-speaker audio-field sidecar around the actual distributed camera/Focusrite clocks, explicit geometry, speaker calibration, bounded buffering, delay/SRO alignment, testable pipeline seams, and FOA AmbiX output.
|
|
- Build the visual sensor-fusion sidecar around timestamped observations, calibrated sensor models, bounded track caches, sparse point-cloud output, and later brush/splat packet lowering.
|
|
|
|
research:
|
|
ambisonic_aquasynth:
|
|
summary: research/ambisonic-aquasynth/summary.md
|
|
bibliography: research/ambisonic-aquasynth/bibliography.md
|
|
key_decision: >
|
|
A spatial field must be built from one clocked multichannel capture path
|
|
feeding a declared ambisonic bus. The existing one-audio-source-per-SRT
|
|
endpoint design remains valid for plain OBS mixing, but it is not the
|
|
authority for synchronized ambisonic DSP.
|
|
live_adaptive_sync:
|
|
summary: research/live-adaptive-sync/summary.md
|
|
bibliography: research/live-adaptive-sync/bibliography.md
|
|
key_decision: >
|
|
If independent mic clocks must be used, sync needs two explicit loops:
|
|
slow sampling-rate-offset estimation driving variable-rate resampling,
|
|
and faster delay/TDOA tracking for current alignment and localization.
|
|
Prove this with a two-mic harness before connecting it to Ambisonics.
|
|
playback_calibration:
|
|
summary: research/playback-calibration/summary.md
|
|
bibliography: research/playback-calibration/bibliography.md
|
|
key_decision: >
|
|
Speaker output is a layout-specific render stage, not a generic sink.
|
|
The system needs a playback profile with decoder matrix/layout,
|
|
output-device latency, channel mapping, per-speaker delay/gain/response
|
|
correction, and a latency ledger. Keep playback calibration separate from
|
|
capture-side adaptive sync.
|
|
feedback_calibration:
|
|
summary: research/feedback-calibration/summary.md
|
|
bibliography: research/feedback-calibration/bibliography.md
|
|
key_decision: >
|
|
Speaker feedback is also a calibration signal when the outgoing audio is
|
|
known. Model it with an online speaker-to-mic transfer matrix and expose
|
|
confidence/convergence state. Keep this estimator separate from clock
|
|
sync, localization, and Ambisonic decoding.
|
|
visual_spatial_map:
|
|
summary: research/visual-spatial-map/summary.md
|
|
bibliography: research/visual-spatial-map/bibliography.md
|
|
brushstroke_rendering: research/visual-spatial-map/brushstroke-fusion-rendering.md
|
|
key_decision: >
|
|
The visual rig is two PlayStation Eye / PS3 Eye cameras as high-rate
|
|
low-resolution marker trackers, two Razer Kiyo-class RGB/mic cameras
|
|
including one Kiyo Pro, and one Leap Motion/LeapUVC stereo IR sensor for
|
|
near-field fusion. LeapUVC driver mode makes Leap appear as a generic UVC
|
|
device and should be treated as mutually exclusive with Ultraleap software
|
|
tracking unless proven otherwise. Build one canonical world coordinate
|
|
system for all visual sensors, microphones, speakers, props, virtual
|
|
camera pose, and ambisonic listener pose. Prove sparse calibrated tracking
|
|
before adding RGB projection or Gaussian splat rendering. For a
|
|
Dreams-like volumetric brushstroke look, keep the world model as truth and
|
|
lower semantic scene claims into cached brush/splat packets under budget.
|
|
gpu_fusion_splatting:
|
|
summary: research/gpu-fusion-splatting/summary.md
|
|
bibliography: research/gpu-fusion-splatting/bibliography.md
|
|
key_decision: >
|
|
The live visual fusion/render hot path should be GPU-resident: pinned host
|
|
frame rings, async CUDA uploads, GPU marker/stereo/feature extraction,
|
|
GPU-shaped observation and brush packet buffers, and CUDA/Vulkan/D3D
|
|
splat rendering with minimal readback. CPU owns orchestration and
|
|
semantics; kernels do not own calibration truth.
|
|
spout_obs_bridge:
|
|
summary: research/spout-obs-bridge/summary.md
|
|
bibliography: research/spout-obs-bridge/bibliography.md
|
|
key_decision: >
|
|
OBS should receive a named Spout2 shared texture from Aquarium Engine,
|
|
not desktop capture or file polling. LocalCastBridge emits typed
|
|
render-frame packets with source timestamps, intended presentation time,
|
|
audio-alignment time, target size, Spout sender name, and point/brush
|
|
claims. Aquarium owns GPU rendering and Spout publication.
|
|
sensor_fusion_architecture:
|
|
summary: docs/sensor-fusion-architecture.md
|
|
key_decision: >
|
|
The visual world model begins as timestamped sensor observations and
|
|
calibrated camera transforms, not render packets. The first sidecar keeps
|
|
capture adapters, detection, triangulation, track caching, and point-cloud
|
|
export as separate testable authorities. This follows the fractal-domain
|
|
cache rule: semantic claims first, cached backend lowerings second.
|
|
|
|
current_status:
|
|
summary: >
|
|
First repo scaffold is in place and deployed to the neighbor sender at
|
|
C:\Meta\LocalCastBridge. FFmpeg 8.1.1 with SRT and h264_nvenc is installed
|
|
there, Voicemeeter supplies system-output loopback, Madman's desktop has
|
|
start/stop launchers, the sender script has launcher-safe FFmpeg arguments
|
|
and repo-root logs, desktop launchers use absolute tool paths, and the
|
|
receiver OBS scene has matching Media Sources. Local calibration harness now
|
|
has first physical-device access: LeapUVC raw IR video, two Razer Kiyo-class
|
|
RGB cameras, all expected microphones, and Scarlett speaker output are
|
|
accessible. LeapUVC MSMF mode probing accepts a 320x240x120 request and
|
|
reports about 115 fps, though measured OpenCV throughput is lower in the
|
|
first rough probe. The opentrack/libusb PS3 Eye path still enumerates modes
|
|
but fails frame reads across libusb-win32, libusbK, and WinUSB. The
|
|
AllanCat PS3 Eye Universal Driver / PS3EyeDirectShow path now exposes both
|
|
PS3 Eyes as DirectShow indices 3 and 4 after registering the source filter
|
|
for two cameras. OpenCV index capture is reliable at default 640x480 near
|
|
30 fps but does not negotiate high-rate modes correctly. FFmpeg DirectShow
|
|
capture by device name negotiates the filter pins properly: both Eyes can
|
|
run simultaneously at 320x240@60, and dual 640x480@60 lands around 54-56
|
|
fps per Eye. Requests above 60 do not increase delivered frame rate through
|
|
the installed filter. Audio-field scaffolding now exists as a separate sidecar
|
|
for the actual distributed rig: two Kiyo mics, two PS Eye mics, local
|
|
Focusrite ADC with a shielded cardioid near the host's face, neighbor
|
|
Focusrite ADC with a shotgun mic pointed at the co-streamer, and two local
|
|
speakers. The Focusrite mics are high-quality dialogue anchors; camera mics
|
|
provide spatial/context evidence. The tool refuses to treat independent
|
|
USB/network clocks as an Ambisonic field until they have been aligned and
|
|
resampled into one six-channel reference timeline. Local PortAudio/WASAPI
|
|
probe shows the Scarlett input accepts 48 kHz but rejects 96 kHz, while the
|
|
Scarlett output accepts 44.1 kHz and rejects 48/96 kHz on this path; check
|
|
ASIO/native Focusrite access before relying on high-rate calibration. Audio
|
|
latency is now an explicit bounded-buffering policy: output may run behind
|
|
live input while delay/SRO estimates settle, but cache depth must converge
|
|
toward real-time. Core audio modules now expose testable boundaries for
|
|
buffering, latency convergence, injectable capture/alignment/encode/sink
|
|
ports, and pipeline orchestration. The first visual sensor-fusion sidecar is
|
|
in place: `localcast.sensor_fusion.core` owns calibrated camera models,
|
|
timestamped 2D observations, DLT triangulation, reprojection/confidence
|
|
gating, a TTL track cache, and PLY point-cloud export; adapters own
|
|
FFmpeg/raw-frame ingress; `scripts/sensor_fusion.py fuse-file` can turn an
|
|
observation file into a sparse PLY. The render bridge now lowers cached
|
|
points into `localcast.sensor_fusion.render_frame.v1` packets carrying
|
|
source time ranges, intended visual presentation time, audio-alignment
|
|
time, target render size, Spout sender name, and point draw metadata for
|
|
Aquarium/Spout. Unit tests inject synthetic cameras and frames so the core
|
|
does not depend on live camera drivers.
|
|
next_action: >
|
|
Capture real ChArUco intrinsics/extrinsics for the PS3 Eyes and Kiyos,
|
|
replace `config/sensor-fusion.example.json` placeholder transforms with a
|
|
local `config/sensor-fusion.json`, then add a detector adapter that turns
|
|
FFmpeg PS3 Eye frames into marker `Observation2D` records for the fusion
|
|
core. Then add an Aquarium Engine consumer for render-frame packets that
|
|
renders cached points/brushes into a GPU render target and publishes it as
|
|
Spout sender `LocalCastBridge Point Cloud` for OBS. In parallel, create local
|
|
`config/audio-field.json`, confirm local
|
|
Kiyo/PS Eye/Focusrite device matches, confirm the neighbor Focusrite
|
|
shotgun capture/transport path, replace placeholder mic/speaker geometry
|
|
with measured world coordinates, then build the delay/SRO alignment stage
|
|
that feeds the bounded field cache and emits aligned six-channel blocks for
|
|
FOA encoding.
|