357 lines
23 KiB
YAML
357 lines
23 KiB
YAML
workspace:
|
|
name: LocalCastBridge
|
|
status: active
|
|
focus: perfect-machine-native-reservoir
|
|
premise: >
|
|
A second Windows PC should appear inside OBS as video plus separately
|
|
mixable audio sources. The synchronized program path now cuts away the
|
|
Python/OpenGL deadline bridge and rebuilds around one native rolling
|
|
reservoir feeding Aquarium, Faust, CultNet, and OBS.
|
|
|
|
objective:
|
|
primary: >
|
|
Build one synchronized spatial stream machine from six cameras, six
|
|
microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP,
|
|
and OBS output.
|
|
current: >
|
|
Carve out Python live producers, the OpenGL Spout sink, JSON live stores,
|
|
and duplicated reservoir windows. Rebuild the live path around one native
|
|
rolling buffer with typed indexes/views and CultNet typed documents.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must happen on the sender using NVENC when available.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
- No Python live hot path.
|
|
- No OpenGL production Spout sink.
|
|
- Edge JSON is schema/diagnostic only; live network/process data is typed
|
|
CultNet documents.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport.
|
|
- One stable port equals one OBS source endpoint.
|
|
- Video endpoint uses H.264 over MPEG-TS/SRT.
|
|
- Audio endpoints default to AAC over MPEG-TS/SRT for broad OBS/FFmpeg compatibility.
|
|
- Local secrets remain untracked.
|
|
- One native rolling reservoir owns the live edge and expiry.
|
|
- Typed views may index the rolling buffer, but no per-kind ring owns
|
|
retention.
|
|
- Aquarium owns production Spout publication; Faust/native DSP owns hot audio.
|
|
- Python and JSON may diagnose or test the machine, not run it.
|
|
|
|
accepted_design:
|
|
- `docs/native-rebuild-plan.md` is the current cut plan. It supersedes the
|
|
deadline Python/OpenGL bridge as a foundation and defines the next coherent
|
|
rebuild target.
|
|
- `docs/perfect-machine.md` and `config/perfect-machine.example.json` define the native target: one five-second spatiotemporal reservoir, Aquarium GPU fusion/rendering, Faust DSP, OBS output controls, and LocalCastBridge tooling.
|
|
- `native/reservoir` is the first native Rust crate. It now uses one
|
|
time-ordered rolling buffer with typed views, so retention has exactly one
|
|
owner while Aquarium/Faust still get kind-scoped queries.
|
|
- `config/localcast.example.json` defines receiver address, base port, video encode settings, and audio sources.
|
|
- `scripts/sender-discover.ps1` lists FFmpeg devices, protocols, and encoders.
|
|
- `scripts/sender-start.ps1` launches one FFmpeg process per enabled source.
|
|
- OBS receives each source through Media Source URLs documented in `docs/obs-receiver-setup.md`.
|
|
- Persistence follows the active repo pattern: map, scratch, evidence, branches, handoff, system map, implementation plan, state helpers.
|
|
|
|
active_subgoals:
|
|
- Bind Aquarium/Faust to the rolling-buffer `LocalcastRuntime` in the native
|
|
reservoir C ABI described by `native/reservoir/include/localcast_reservoir.h`.
|
|
- Delete or quarantine Python live producers and the OpenGL Spout sink before
|
|
adding new live features.
|
|
- Bind Aquarium/Faust to `LocalcastRuntime` in the native reservoir C ABI described by `native/reservoir/include/localcast_reservoir.h`.
|
|
- Move dense visual fusion/material fitting/brush generation into Aquarium GPU compute.
|
|
- Move hot audio DSP/voice separation/room suppression/spatialization into Faust/native runtime.
|
|
- Validate the actual sender FFmpeg build and capture device names.
|
|
- Smoke-test the video endpoint in OBS.
|
|
- Smoke-test each audio endpoint in OBS and tune latency.
|
|
- Decide whether an OBS source generator or plugin investigation is justified.
|
|
- Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock.
|
|
- Build the six-microphone/two-speaker audio-field sidecar around the actual distributed camera/Focusrite clocks, explicit geometry, speaker calibration, bounded buffering, delay/SRO alignment, testable pipeline seams, and FOA AmbiX output.
|
|
- Build the visual sensor-fusion sidecar around timestamped observations, calibrated sensor models, bounded track caches, sparse point-cloud output, and later brush/splat packet lowering.
|
|
|
|
research:
|
|
ambisonic_aquasynth:
|
|
summary: research/ambisonic-aquasynth/summary.md
|
|
bibliography: research/ambisonic-aquasynth/bibliography.md
|
|
key_decision: >
|
|
A spatial field must be built from one clocked multichannel capture path
|
|
feeding a declared ambisonic bus. The existing one-audio-source-per-SRT
|
|
endpoint design remains valid for plain OBS mixing, but it is not the
|
|
authority for synchronized ambisonic DSP.
|
|
live_adaptive_sync:
|
|
summary: research/live-adaptive-sync/summary.md
|
|
bibliography: research/live-adaptive-sync/bibliography.md
|
|
key_decision: >
|
|
If independent mic clocks must be used, sync needs two explicit loops:
|
|
slow sampling-rate-offset estimation driving variable-rate resampling,
|
|
and faster delay/TDOA tracking for current alignment and localization.
|
|
Prove this with a two-mic harness before connecting it to Ambisonics.
|
|
playback_calibration:
|
|
summary: research/playback-calibration/summary.md
|
|
bibliography: research/playback-calibration/bibliography.md
|
|
key_decision: >
|
|
Speaker output is a layout-specific render stage, not a generic sink.
|
|
The system needs a playback profile with decoder matrix/layout,
|
|
output-device latency, channel mapping, per-speaker delay/gain/response
|
|
correction, and a latency ledger. Keep playback calibration separate from
|
|
capture-side adaptive sync.
|
|
feedback_calibration:
|
|
summary: research/feedback-calibration/summary.md
|
|
bibliography: research/feedback-calibration/bibliography.md
|
|
key_decision: >
|
|
Speaker feedback is also a calibration signal when the outgoing audio is
|
|
known. Model it with an online speaker-to-mic transfer matrix and expose
|
|
confidence/convergence state. Keep this estimator separate from clock
|
|
sync, localization, and Ambisonic decoding.
|
|
visual_spatial_map:
|
|
summary: research/visual-spatial-map/summary.md
|
|
bibliography: research/visual-spatial-map/bibliography.md
|
|
brushstroke_rendering: research/visual-spatial-map/brushstroke-fusion-rendering.md
|
|
key_decision: >
|
|
The visual rig is two PlayStation Eye / PS3 Eye cameras as high-rate
|
|
low-resolution marker trackers, two Razer Kiyo-class RGB/mic cameras
|
|
including one Kiyo Pro, and one Leap Motion/LeapUVC stereo IR sensor for
|
|
near-field fusion. LeapUVC driver mode makes Leap appear as a generic UVC
|
|
device and should be treated as mutually exclusive with Ultraleap software
|
|
tracking unless proven otherwise. Build one canonical world coordinate
|
|
system for all visual sensors, microphones, speakers, props, virtual
|
|
camera pose, and ambisonic listener pose. Prove sparse calibrated tracking
|
|
before adding RGB projection or Gaussian splat rendering. For a
|
|
Dreams-like volumetric brushstroke look, keep the world model as truth and
|
|
lower semantic scene claims into cached brush/splat packets under budget.
|
|
gpu_fusion_splatting:
|
|
summary: research/gpu-fusion-splatting/summary.md
|
|
bibliography: research/gpu-fusion-splatting/bibliography.md
|
|
key_decision: >
|
|
The live visual fusion/render hot path should be GPU-resident: pinned host
|
|
frame rings, async CUDA uploads, GPU marker/stereo/feature extraction,
|
|
GPU-shaped observation and brush packet buffers, and CUDA/Vulkan/D3D
|
|
splat rendering with minimal readback. CPU owns orchestration and
|
|
semantics; kernels do not own calibration truth.
|
|
spout_obs_bridge:
|
|
summary: research/spout-obs-bridge/summary.md
|
|
bibliography: research/spout-obs-bridge/bibliography.md
|
|
key_decision: >
|
|
OBS should receive a named Spout2 shared texture from Aquarium Engine,
|
|
not desktop capture or file polling. LocalCastBridge emits typed
|
|
render-frame packets with source timestamps, intended presentation time,
|
|
audio-alignment time, target size, Spout sender name, and point/brush
|
|
claims. The deadline Spout sink now defaults to a `kiyo-mid-deru`
|
|
virtual camera, with the eye halfway between the two Kiyo-class cameras
|
|
and the target on the co-streamer body volume. Its point budget is
|
|
temporal: stable high-confidence anchors plus a frame-varying remainder
|
|
for TAA/supersampling. Aquarium owns GPU rendering and Spout publication.
|
|
live_spout_deadline_sink:
|
|
summary: docs/obs-spout-streaming.md
|
|
key_decision: >
|
|
The deadline OBS surface proved the need for a named Spout sender, but it
|
|
is not a foundation. `scripts/diagnostic_stream_spout.py`, SpoutGL/OpenGL rendering,
|
|
Python render-frame production, and JSON heartbeat stores are diagnostics
|
|
or migration fossils only. Production Spout publication belongs in
|
|
Aquarium, fed by the native rolling reservoir and typed CultNet documents.
|
|
live_audio_phase_meaning:
|
|
summary: docs/audio-field.md
|
|
key_decision: >
|
|
Phase/chirplet evidence now has a live semantic boundary:
|
|
`audio_field.phase_meaning` keeps raw phase internal and publishes
|
|
`localcast.audio.phase_field` with delay correction, coherence,
|
|
confidence, distance-equivalent delta, reference-bleed/suppression
|
|
weights, correction energy, and active-probe need. `audio_field.active_probe`
|
|
closes the next loop by feeding that confidence state into
|
|
ActiveProbeOptimizer and emitting bounded chirplet WAVs/manifests, with
|
|
optional playback. The live launcher now uses dense harmonic
|
|
near-ultrasonic probe textures so aliases have a musical relationship
|
|
instead of arbitrary squeal. Probe artifacts rotate through fixed slots
|
|
and the manifest rotates at a byte cap. `scripts/stream_live_audio_field.py`
|
|
is the current live local capture owner: it writes mic-field and
|
|
phase-field CultCache documents from visible local mics plus Scarlett
|
|
loopback, sends probes through the selected Scarlett output, resamples
|
|
probe playback when that output is 44.1 kHz, and reports missing
|
|
distributed channels as explicit placeholders. Active probe scheduling is
|
|
restricted to source ids with live local capture devices, so placeholders
|
|
do not consume chirp budget. `audio-phase-field-status.json`
|
|
declares input mode and whether active probes are closed-loop; current
|
|
deadline mode is `live-local-capture` with closed-loop probe capture when
|
|
Scarlett loopback is available.
|
|
This is learned audio-field state, not a phase display.
|
|
faust_voice_separation:
|
|
summary: docs/aquarium-spatial-audio.md
|
|
key_decision: >
|
|
Voice separation should run in Aquarium's Faust DSP, not as Python hot
|
|
audio math after the mic field has already collapsed to AmbiX.
|
|
LocalCastBridge now publishes `localcast.audio.mic_field` with six
|
|
aligned mic channels, roles, controls, and graph id
|
|
`localcast.faust.voice_separation.v1`; the matching Faust graph lives at
|
|
`faust/localcast_voice_separation.dsp`.
|
|
sensor_fusion_architecture:
|
|
summary: docs/sensor-fusion-architecture.md
|
|
key_decision: >
|
|
The visual world model begins as timestamped sensor observations and
|
|
calibrated camera transforms, not render packets. The first sidecar keeps
|
|
capture adapters, detection, triangulation, track caching, and point-cloud
|
|
export as separate testable authorities. This follows the fractal-domain
|
|
cache rule: semantic claims first, cached backend lowerings second.
|
|
Stochastic transient matching now lowers descriptor-consistent image
|
|
observations into scene-ray hypotheses and a multi-LOD world-space cache
|
|
for Aquarium compute-shader reconciliation. Leap evidence is the
|
|
highest-priority temporal/spatial source when live Leap frames are present;
|
|
fallback Leap frames stay tagged as fallback evidence and must not be
|
|
treated as ground truth. Live clap calibration now uses the Kiyo stereo
|
|
pair for the first rough 3D impact solve and includes Leap motion as the
|
|
high-precision visual timing witness; Leap should become a geometric
|
|
camera authority after shared calibration. Clap peaks update per-camera
|
|
clock offsets, allowing history buffers to be sampled at
|
|
`oracle_time - camera_offset` for coherent point-cloud construction.
|
|
`localcast.sensor_fusion.chirp_pose` converts live phase-field delay
|
|
meaning for Kiyo/PS Eye microphone sources into camera-body range
|
|
constraints and per-camera pose-correction estimates, which the live
|
|
producer emits as `camera-chirp:*` and `camera-pose-correction:*` render
|
|
points plus `chirp-camera-pose` LOD evidence. It can load geometry from
|
|
the audio-field profile, falling back to the example/default geometry
|
|
until measured local coordinates exist. These constraints augment visual
|
|
calibration; they do not own image geometry. The Spout point-budget policy
|
|
pins calibration and cross-modal constraints before spending adaptive LOD
|
|
budget on stable high-priority surfaces and frame-rotated samples, and
|
|
its JSON heartbeat reports budgeted point prefix counts. The live fusion
|
|
producer now has an explicit five-second spatiotemporal reservoir:
|
|
RGB/Leap frame history expires from the newest shared sample, LOD evidence
|
|
is clipped to the same window, and each RenderFramePacket declares that
|
|
source window for Aquarium/OBS synchronization. Multi-LOD cells now carry
|
|
first-pass material estimates, weighted albedo plus roughness/metallic
|
|
hints, so Aquarium can relight/refine brush packets from reservoir state
|
|
instead of scraping screen-space color. The Spout/audio overlay boundary
|
|
clamps stale visual packets to the current audio reservoir edge before
|
|
OBS sees them. Current live deadline mode uses fallback-only RGB/Leap
|
|
because synchronous OpenCV camera reads measured slower than the reservoir;
|
|
the next coherent cut is nonblocking camera ingress that continuously
|
|
fills the five-second reservoir with real samples. `LatestFramePump` now
|
|
moves blocking OpenCV reads off the fusion hot loop; the running deadline
|
|
mode uses lower per-frame sample density so Spout can render visual points
|
|
inside the five-second budget while TAA/reservoir accumulation carries
|
|
detail over time.
|
|
typed_visual_state:
|
|
summary: docs/typed-visual-state.md
|
|
key_decision: >
|
|
Diagnostic visual state can still be exported as typed CultCache
|
|
MessagePack documents: `localcast.visual.render_frame` in
|
|
`calibration/runs/visual-state.msgpack` and
|
|
`localcast.visual.stream_status` in
|
|
`calibration/runs/visual-stream-status.msgpack`. Production visual state
|
|
crosses native/CultNet typed document boundaries; the Python file adapter
|
|
is not live authority. Edge JSON may define schemas or export
|
|
diagnostics, but runtime data should remain typed documents because every
|
|
networked subsystem is under our control.
|
|
|
|
current_status:
|
|
summary: >
|
|
First repo scaffold is in place and deployed to the neighbor sender at
|
|
C:\Meta\LocalCastBridge. FFmpeg 8.1.1 with SRT and h264_nvenc is installed
|
|
there, direct WASAPI console-session capture now owns system-output loopback, Madman's desktop has
|
|
start/stop launchers, the sender script has launcher-safe FFmpeg arguments
|
|
and repo-root logs, desktop launchers use absolute tool paths, and the
|
|
receiver OBS scene has matching Media Sources. Local calibration harness now
|
|
has first physical-device access: LeapUVC raw IR video, two Razer Kiyo-class
|
|
RGB cameras, all expected microphones, and Scarlett speaker output are
|
|
accessible. LeapUVC MSMF mode probing accepts a 320x240x120 request and
|
|
reports about 115 fps, though measured OpenCV throughput is lower in the
|
|
first rough probe. The opentrack/libusb PS3 Eye path still enumerates modes
|
|
but fails frame reads across libusb-win32, libusbK, and WinUSB. The
|
|
AllanCat PS3 Eye Universal Driver / PS3EyeDirectShow path now exposes both
|
|
PS3 Eyes as DirectShow indices 3 and 4 after registering the source filter
|
|
for two cameras. OpenCV index capture is reliable at default 640x480 near
|
|
30 fps but does not negotiate high-rate modes correctly. FFmpeg DirectShow
|
|
capture by device name negotiates the filter pins properly: both Eyes can
|
|
run simultaneously at 320x240@60, and dual 640x480@60 lands around 54-56
|
|
fps per Eye. Requests above 60 do not increase delivered frame rate through
|
|
the installed filter. Audio-field scaffolding now exists as a separate sidecar
|
|
for the actual distributed rig: two Kiyo mics, two PS Eye mics, local
|
|
Focusrite ADC with a shielded cardioid near the host's face, neighbor
|
|
Focusrite ADC with a shotgun mic pointed at the co-streamer, and two local
|
|
speakers. The Focusrite mics are high-quality dialogue anchors; camera mics
|
|
provide spatial/context evidence. The tool refuses to treat independent
|
|
USB/network clocks as an Ambisonic field until they have been aligned and
|
|
resampled into one six-channel reference timeline. Local PortAudio/WASAPI
|
|
probe shows the Scarlett input accepts 48 kHz but rejects 96 kHz, while the
|
|
Scarlett output accepts 44.1 kHz and rejects 48/96 kHz on this path; check
|
|
ASIO/native Focusrite access before relying on high-rate calibration. Audio
|
|
latency is now an explicit bounded-buffering policy: output may run behind
|
|
live input while delay/SRO estimates settle, but cache depth must converge
|
|
toward real-time. Core audio modules now expose testable boundaries for
|
|
buffering, latency convergence, injectable capture/alignment/encode/sink
|
|
ports, and pipeline orchestration. Calibration analysis now has a local
|
|
chirplet refinement pass: coarse sweep match followed by fractional-delay
|
|
and chirp-rate grid search around the detected arrival. Runtime sync is now
|
|
modeled as per-block known-speaker chirplet telemetry updating delay, SRO,
|
|
phase, and confidence state; low-confidence observations freeze rather than
|
|
poisoning alignment. Active probe optimization owns extra chirplet emission
|
|
when sync confidence drops, bounded by spacing, level, and audibility budget.
|
|
The first visual
|
|
sensor-fusion sidecar is
|
|
in place: `localcast.sensor_fusion.core` owns calibrated camera models,
|
|
timestamped 2D observations, DLT triangulation, reprojection/confidence
|
|
gating, a TTL track cache, and PLY point-cloud export; adapters own
|
|
FFmpeg/raw-frame ingress; `scripts/sensor_fusion.py fuse-file` can turn an
|
|
observation file into a sparse PLY. The render bridge now lowers cached
|
|
points into `localcast.sensor_fusion.render_frame.v1` packets carrying
|
|
source time ranges, intended visual presentation time, audio-alignment
|
|
time, target render size, Spout sender name, and point draw metadata for
|
|
Aquarium/Spout. A deadline OBS Spout sender is now live-shaped:
|
|
`scripts/diagnostic_live_sensor_fusion.py` writes fused render frames into
|
|
diagnostic typed CultCache state at `calibration/runs/visual-state.msgpack`.
|
|
`scripts/diagnostic_stream_spout.py` consumes that diagnostic document,
|
|
lowers points into compact anisotropic
|
|
screen brush packets, uploads an OpenGL texture, publishes Spout sender
|
|
`LocalCastBridge Point Cloud`, and writes both typed status
|
|
`calibration/runs/visual-stream-status.msgpack` and compatibility JSON
|
|
`calibration/runs/stream-spout-status.json`; OBS now has a Spout2 Capture
|
|
source pointed at that sender, and an OBS websocket screenshot verified
|
|
nonblack brush output. OBS synchronized program setup now derives six
|
|
per-channel stems from the aligned program audio timeline, creates/updates
|
|
OBS Media Sources for host voice, co-streamer voice, ambient, transients,
|
|
co-streamer loopback, and local loopback, and can disable every scene item
|
|
except synchronized LocalCastBridge program surfaces. Co-streamer surface
|
|
capture now records neighbor Focusrite and neighbor loopback while recording
|
|
local loopback ground truth, then writes aligned co-streamer surface WAVs
|
|
for the stem packer. Live phase-field meaning now writes
|
|
`localcast.audio.phase_field` to `calibration/runs/audio-phase-field.msgpack`
|
|
from internal chirplet/phase evidence so downstream code sees delay,
|
|
coherence, confidence, suppression, correction, and probe-control state
|
|
instead of raw phases. The neighbor desktop launcher no longer uses
|
|
Voicemeeter; it restores the Focusrite speakers as the primary playback
|
|
device and the Focusrite analogue input as the default capture device. A
|
|
direct Core Audio WASAPI loopback shim now works when launched in the
|
|
neighbor's interactive console session; SSH-only launches can see the
|
|
endpoint but receive no render packets.
|
|
Missing synchronized neighbor feeds are explicit
|
|
silent placeholders, not raw unsynchronized substitutes. The live producer currently uses synthetic calibrated
|
|
observations through the real `SensorRig.fuse` path until camera detector
|
|
observations replace them. The RGB path now has a debug-only CPU dense
|
|
stereo reference: the two Kiyo-class frames can be block-matched as a
|
|
provisional calibrated pair and triangulated into `dense-rgb:*` surface
|
|
claims, but production dense reconstruction must move to Aquarium/GPU
|
|
compute rather than Python loops. A separate
|
|
adaptive camera-control policy measures luminance, clipping, contrast, and
|
|
sharpness, then emits normalized exposure/gain/focus commands through the
|
|
OpenCV capture adapter when the driver accepts them. Active illumination now
|
|
has an optional Tuya local-network pulse hook for deliberately modulating
|
|
room light as calibration telemetry; pulse timestamps can be aligned against
|
|
camera brightness/depth response. Common-space calibration now has a real
|
|
owner: fixed-board ChArUco observations plus intrinsics solve
|
|
`CameraModel.world_from_sensor` transforms, and cross-view surface-feature
|
|
observations can be descriptor-matched and triangulated into world-space
|
|
tracks. Unit tests inject
|
|
synthetic cameras and frames so the core does not depend on live camera
|
|
drivers. The multi-LOD visual cache now carries source kind and source
|
|
priority so Aquarium can prefer real Leap evidence over RGB surfaces,
|
|
stochastic transient hypotheses, and synthetic fusion support while still
|
|
seeing lower-priority fallback diagnostics when the Leap capture path stalls.
|
|
next_action: >
|
|
Stop live feature work on the Python/OpenGL bridge. The first native cut is
|
|
complete: `native/reservoir` is now a single time-ordered rolling buffer
|
|
with typed views and updated ABI/tests. Next quarantine Python/OpenGL code
|
|
as diagnostics, then bind Aquarium/Faust to the native runtime and resume
|
|
hardware calibration through typed CultNet/native surfaces only.
|