135 lines
7.6 KiB
YAML
135 lines
7.6 KiB
YAML
workspace:
|
|
name: LocalCastBridge
|
|
status: active
|
|
focus: local-obs-lan-source
|
|
premise: >
|
|
A second Windows PC should appear inside OBS as video plus separately
|
|
mixable audio sources. Use ordinary media transport first; native OBS code
|
|
only earns its keep after standard OBS/FFmpeg surfaces fail a real invariant.
|
|
|
|
objective:
|
|
primary: >
|
|
Provide a repeatable local streaming environment where the sender captures
|
|
desktop/video and audio sources, encodes video with NVENC, and delivers
|
|
stable OBS Media Source URLs to the receiver over LAN.
|
|
current: >
|
|
Establish the script-and-doc spine for a two-PC smoke test.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must happen on the sender using NVENC when available.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport.
|
|
- One stable port equals one OBS source endpoint.
|
|
- Video endpoint uses H.264 over MPEG-TS/SRT.
|
|
- Audio endpoints default to AAC over MPEG-TS/SRT for broad OBS/FFmpeg compatibility.
|
|
- Local secrets remain untracked.
|
|
|
|
accepted_design:
|
|
- `config/localcast.example.json` defines receiver address, base port, video encode settings, and audio sources.
|
|
- `scripts/sender-discover.ps1` lists FFmpeg devices, protocols, and encoders.
|
|
- `scripts/sender-start.ps1` launches one FFmpeg process per enabled source.
|
|
- OBS receives each source through Media Source URLs documented in `docs/obs-receiver-setup.md`.
|
|
- Persistence follows the active repo pattern: map, scratch, evidence, branches, handoff, system map, implementation plan, state helpers.
|
|
|
|
active_subgoals:
|
|
- Validate the actual sender FFmpeg build and capture device names.
|
|
- Smoke-test the video endpoint in OBS.
|
|
- Smoke-test each audio endpoint in OBS and tune latency.
|
|
- Decide whether an OBS source generator or plugin investigation is justified.
|
|
- Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock.
|
|
- Build the six-microphone/two-speaker audio-field sidecar around one clocked capture path, explicit geometry, speaker calibration, and FOA AmbiX output.
|
|
|
|
research:
|
|
ambisonic_aquasynth:
|
|
summary: research/ambisonic-aquasynth/summary.md
|
|
bibliography: research/ambisonic-aquasynth/bibliography.md
|
|
key_decision: >
|
|
A spatial field must be built from one clocked multichannel capture path
|
|
feeding a declared ambisonic bus. The existing one-audio-source-per-SRT
|
|
endpoint design remains valid for plain OBS mixing, but it is not the
|
|
authority for synchronized ambisonic DSP.
|
|
live_adaptive_sync:
|
|
summary: research/live-adaptive-sync/summary.md
|
|
bibliography: research/live-adaptive-sync/bibliography.md
|
|
key_decision: >
|
|
If independent mic clocks must be used, sync needs two explicit loops:
|
|
slow sampling-rate-offset estimation driving variable-rate resampling,
|
|
and faster delay/TDOA tracking for current alignment and localization.
|
|
Prove this with a two-mic harness before connecting it to Ambisonics.
|
|
playback_calibration:
|
|
summary: research/playback-calibration/summary.md
|
|
bibliography: research/playback-calibration/bibliography.md
|
|
key_decision: >
|
|
Speaker output is a layout-specific render stage, not a generic sink.
|
|
The system needs a playback profile with decoder matrix/layout,
|
|
output-device latency, channel mapping, per-speaker delay/gain/response
|
|
correction, and a latency ledger. Keep playback calibration separate from
|
|
capture-side adaptive sync.
|
|
feedback_calibration:
|
|
summary: research/feedback-calibration/summary.md
|
|
bibliography: research/feedback-calibration/bibliography.md
|
|
key_decision: >
|
|
Speaker feedback is also a calibration signal when the outgoing audio is
|
|
known. Model it with an online speaker-to-mic transfer matrix and expose
|
|
confidence/convergence state. Keep this estimator separate from clock
|
|
sync, localization, and Ambisonic decoding.
|
|
visual_spatial_map:
|
|
summary: research/visual-spatial-map/summary.md
|
|
bibliography: research/visual-spatial-map/bibliography.md
|
|
brushstroke_rendering: research/visual-spatial-map/brushstroke-fusion-rendering.md
|
|
key_decision: >
|
|
The visual rig is two PlayStation Eye / PS3 Eye cameras as high-rate
|
|
low-resolution marker trackers, two Razer Kiyo-class RGB/mic cameras
|
|
including one Kiyo Pro, and one Leap Motion/LeapUVC stereo IR sensor for
|
|
near-field fusion. LeapUVC driver mode makes Leap appear as a generic UVC
|
|
device and should be treated as mutually exclusive with Ultraleap software
|
|
tracking unless proven otherwise. Build one canonical world coordinate
|
|
system for all visual sensors, microphones, speakers, props, virtual
|
|
camera pose, and ambisonic listener pose. Prove sparse calibrated tracking
|
|
before adding RGB projection or Gaussian splat rendering. For a
|
|
Dreams-like volumetric brushstroke look, keep the world model as truth and
|
|
lower semantic scene claims into cached brush/splat packets under budget.
|
|
gpu_fusion_splatting:
|
|
summary: research/gpu-fusion-splatting/summary.md
|
|
bibliography: research/gpu-fusion-splatting/bibliography.md
|
|
key_decision: >
|
|
The live visual fusion/render hot path should be GPU-resident: pinned host
|
|
frame rings, async CUDA uploads, GPU marker/stereo/feature extraction,
|
|
GPU-shaped observation and brush packet buffers, and CUDA/Vulkan/D3D
|
|
splat rendering with minimal readback. CPU owns orchestration and
|
|
semantics; kernels do not own calibration truth.
|
|
|
|
current_status:
|
|
summary: >
|
|
First repo scaffold is in place and deployed to the neighbor sender at
|
|
C:\Meta\LocalCastBridge. FFmpeg 8.1.1 with SRT and h264_nvenc is installed
|
|
there, Voicemeeter supplies system-output loopback, Madman's desktop has
|
|
start/stop launchers, the sender script has launcher-safe FFmpeg arguments
|
|
and repo-root logs, desktop launchers use absolute tool paths, and the
|
|
receiver OBS scene has matching Media Sources. Local calibration harness now
|
|
has first physical-device access: LeapUVC raw IR video, two Razer Kiyo-class
|
|
RGB cameras, all expected microphones, and Scarlett speaker output are
|
|
accessible; both PS3 Eye video interfaces are attached but missing their
|
|
video-side driver. LeapUVC MSMF mode probing accepts a 320x240x120 request
|
|
and reports about 115 fps, though measured OpenCV throughput is lower in the
|
|
first rough probe. PS3 Eye video interfaces now bind through libusb-win32
|
|
and opentrack enumerates high-rate modes, but frame reads fail while all
|
|
cameras sit on one external hub/root path; USB topology is now a live
|
|
suspect. Audio-field scaffolding now exists as a separate sidecar:
|
|
six-mic/two-speaker profile, validation, calibration sweep capture/analysis,
|
|
raw field recording, and offline FOA AmbiX encoding. Current PortAudio
|
|
enumeration does not expose a matching six-channel WASAPI input, so live
|
|
field capture requires either a shared six-input device/profile or a
|
|
separately proven adaptive sync path for independent USB microphones.
|
|
next_action: >
|
|
In parallel with the camera driver's next cut, create local
|
|
config/audio-field.json, confirm whether one device exposes six synchronized
|
|
mic inputs, replace placeholder mic/speaker geometry with measured world
|
|
coordinates, then run the two-speaker calibration sweep.
|