Mimir/notes/current-system-map.md

Current System Map

LocalCastBridge is intentionally thin. The target live machine is not the deadline bridge; it is the native reservoir described in docs/perfect-machine.md and config/perfect-machine.example.json.

Perfect Machine Target

flowchart TD
    A["6 cameras"] --> R["5s native reservoir"]
    B["6 microphones"] --> R
    C["Leap timing/IR"] --> R
    D["loopback + remote video/audio"] --> R
    R --> E["Aquarium GPU fusion + material field"]
    R --> F["Faust DSP"]
    E --> G["Spout2 texture"]
    F --> H["program stems + spatial bed"]
    G --> I["OBS"]
    H --> I

Ownership:

• LocalCastBridge owns configuration, calibration, contract tests, launch, status, and persistence.
• Aquarium owns dense visual fusion, material/brush/splat reconciliation, final render target, and Spout publication.
• Faust owns hot audio DSP and program stem generation.
• OBS owns broadcast controls.
• Python remains tooling, not the stream hot path.
• native/reservoir is the first native crate. It owns shared-edge five-second sample retention for typed rings and is the seed of the Aquarium/Faust runtime boundary.

Invariant: the five-second spatiotemporal reservoir is the live authority. Producers append, optimizers refine, Aquarium/Faust sample. No private history outlives the reservoir.

flowchart TD
    A["config/localcast.json"] --> B["sender-start.ps1"]
    C["FFmpeg on sender"] --> D["Windows desktop capture"]
    C --> E["DirectShow audio capture"]
    D --> F["h264_nvenc encode"]
    E --> G["AAC encode"]
    F --> H["SRT MPEG-TS video endpoint"]
    G --> I["SRT MPEG-TS audio endpoint(s)"]
    H --> J["OBS Media Source"]
    I --> K["OBS Media Source per audio source"]
    J --> L["OBS scene composition"]
    K --> L

Ownership

• Config owns source identity, receiver address, ports, and encoding knobs.
• FFmpeg owns capture, encoding, and network transport.
• OBS owns source activation, layout, filters, monitoring, and recording/streaming.
• This repo owns repeatability and memory.

Why Not Plugin First

An OBS plugin would be justified if OBS could not ingest stable LAN streams, or if a plugin were needed to expose independent audio controls. V1 gets independent audio controls by making each source an OBS Media Source. That is boring. Boring is allowed to win when it is correct.

Known Risks

• Windows audio source names vary by driver and localization.
• Some FFmpeg builds omit SRT or NVENC.
• Separate endpoints can drift; local latency should be tuned before adding a synchronization layer.
• OBS SRT reconnection behavior can be fussy. Use stable ports and source reactivation before treating port changes as a fix.

Audio Field Sidecar

The six-microphone Ambisonic path is separate from the OBS endpoint path.

flowchart TD
    A["config/audio-field.json"] --> B["scripts/audio_field.py"]
    C["local Focusrite shielded cardioid"] --> D["local reference timeline"]
    E["neighbor Focusrite shotgun"] --> F["remote dialogue capture"]
    G["Kiyo + PS Eye camera mics"] --> H["spatial/context captures"]
    I["2 speaker outputs"] --> J["calibration sweep"]
    J --> D
    J --> F
    J --> H
    R["confidence probe optimizer"] --> P["known speaker chirplets"]
    P --> Q["runtime delay/SRO/phase estimator"]
    D --> Q
    F --> Q
    H --> Q
    Q --> S["phase-meaning extractor"]
    S --> T["audio-phase-field.msgpack"]
    S --> K["delay + SRO alignment"]
    K --> L["bounded field cache"]
    L --> M["aligned six-channel blocks"]
    M --> N["FOA encoder"]
    M --> V["Faust mic-field publisher"]
    V --> W["localcast.audio.mic_field"]
    W --> X["Aquarium Faust voice-separation graph"]
    N --> O["AmbiX ACN/SN3D bus: W,Y,Z,X"]

Ownership:

• config/audio-field.json owns mic/speaker identity, machine/device mapping, clock domains, field channel order, geometry, gain, delay, polarity, role/quality priority, capture policy, and Ambisonic bus format.
• scripts/audio_field.py owns profile validation, local device checks, calibration stimulus generation, clock-domain planning, shared-input capture helpers, and FOA encoding of already aligned six-channel WAVs.
• audio_field/ owns unit-testable buffering, bounded-latency convergence, injectable port protocols, and pipeline orchestration.
• Runtime sync owns per-block chirplet observations from known speaker output and updates delay/SRO/phase estimates with confidence gates before alignment.
• audio_field.phase_meaning owns extracting usable meaning from internal phase/chirplet evidence. The live cache publishes delay correction, coherence, confidence, distance-equivalent delta, reference-bleed/suppression estimates, correction energy, and active-probe need; raw phase bands stay inside the estimator.
• scripts/stream_live_audio_field.py is the current live local capture owner. It publishes localcast.audio.mic_field and localcast.audio.phase_field from the visible local Kiyo/PS Eye/Scarlett inputs, uses Scarlett loopback as the known reference, and keeps missing neighbor/Kiyo-right channels as explicit placeholders until synchronized feeds exist.
• audio-phase-field-status.json declares the phase runtime mode. Current deadline mode is live-local-capture with closed-loop probe capture when Scarlett loopback is available.
• Active probe optimization owns extra chirplet emission when confidence drops, bounded by level/spacing/audibility budget. audio_field.active_probe is the runtime join that converts phase-field confidence into emitted chirplet WAVs/manifests and playback. Probe WAVs rotate through fixed slots and the manifest rotates at a byte cap; calibration may be dense, but artifacts must stay bounded.
• The camera/sensor-fusion pipeline may publish world poses later; it does not own audio clocks or channel timing.
• OBS may ingest rendered output later; it is not the authority for the Ambisonic field.
• Aquarium/Faust owns hot voice separation after LocalCastBridge publishes localcast.audio.mic_field; LocalCastBridge owns alignment, timing, mic roles, graph id, and controls.

Invariant: distributed camera/Focusrite microphones must be aligned and resampled into one reference timeline before FOA encoding. Latency is allowed as bounded buffering, but cache depth must converge toward real-time. Speaker output chirplets are live telemetry; delay/SRO/phase state must update during runtime, not only during setup. Extra chirplets may be emitted automatically when confidence drops, but only under the active probe optimizer's budget. The local shielded cardioid and neighbor shotgun are the high-quality dialogue anchors; camera mics provide spatial/context evidence.

Visual Fusion Sidecar

flowchart TD
    A["synthetic/live observations"] --> B["SensorRig.fuse"]
    B --> C["RenderFramePacket"]
    C --> D["CultCache visual-state.msgpack"]
    D --> E["stream_spout.py"]
    E --> F["ScreenBrushPacket lowering"]
    F --> G["Spout sender"]
    G --> H["OBS Spout2 Capture"]
    I["Leap motion evidence"] --> J["multi-LOD scene cache"]
    K["RGB/stochastic/synthetic evidence"] --> J
    M["live clap audio transient"] --> N["Leap-timed clap calibration"]
    O["Kiyo stereo motion window"] --> N
    N --> C
    N --> J
    J --> L["Aquarium compute reconciliation"]

Ownership:

• localcast.sensor_fusion.cultcache_docs owns typed visual state documents.
• scripts/live_sensor_fusion.py currently owns the live producer and writes localcast.visual.render_frame into CultCache.
• The live producer now keeps a rolling RGB/Leap frame window for clap calibration. Kiyo stereo motion gives the first rough 3D clap position; Leap participates as the high-precision visual timing witness until a calibrated Leap geometric model joins the rig.
• Clap peaks update a per-camera clock sync model. Camera history sampling uses oracle_time - camera_offset, so the point builder can reconstruct a shared instant instead of mixing latest frames from different clock domains.
• The live producer's temporal authority is a five-second spatiotemporal reservoir measured backward from the newest shared sample. RGB/Leap history expires against that shared edge, LOD evidence is clipped to the same window, and each render frame declares the exact reservoir slice with source_time_min_ns and source_time_max_ns.
• Multi-LOD cache cells carry first-pass material evidence: weighted albedo plus roughness/metallic hints. Aquarium owns the real relighting/BRDF refinement, but it should receive material state from the reservoir rather than reverse-engineering it from screen brushes.
• stream_spout.py clamps stale visual packets to the current audio reservoir edge before OBS sees them. If a camera path blocks past the five-second window, its stale points are dropped. Current live deadline mode uses --rgb-fallback-only --leap-fallback-only because direct OpenCV camera reads measured slower than the reservoir; the next cut is nonblocking camera ingress that fills the reservoir continuously.
• LatestFramePump now runs blocking OpenCV camera sources off the producer hot loop. Deadline live mode keeps per-frame visual density low enough for TAA/reservoir accumulation: roughly 1.6k Spout points at audio_delta_ns: 0 with --rgb-room-step 16 --leap-step 16 --points 64.
• localcast.sensor_fusion.chirp_pose now turns phase-field delay meaning for Kiyo/PS Eye microphone sources into camera-body range constraints and per-camera pose-correction estimates. It loads camera-mic and speaker geometry from the audio-field profile when available, falling back to the example profile/defaults. The live producer publishes those constraints as camera-chirp:* and camera-pose-correction:* render points plus chirp-camera-pose LOD evidence. Audio range/timing evidence augments visual calibration; it does not own image geometry.
• The Spout point-budget policy pins calibration and cross-modal constraints before spending the remaining adaptive LOD budget on stable high-priority surface claims and frame-rotated samples. Its JSON heartbeat reports the prefix counts that survived the render budget.
• The live producer also writes a multi-LOD scene cache with source kind and priority. Real Leap frames are the highest-priority timing/spatial evidence; Leap fallback frames are tagged as fallback and do not become ground truth.
• scripts/stream_spout.py consumes typed CultCache state and publishes Spout. Its presentation camera defaults to kiyo-mid-deru: the virtual eye sits halfway between the two Kiyo-class cameras and aims at the co-streamer body target. Its render budget is temporal: high-confidence anchors stay stable while the remaining samples rotate by frame id for TAA/supersampling reconciliation.
• CultNet document replication is the intended API boundary for other producers/consumers.

Invariant: JSON is not the visual-state authority. The current renderer can still write a JSON heartbeat for human inspection, but live visual state lives in typed CultCache MessagePack documents. Leap outranks camera surfaces only when the frame source is actual Leap capture; diagnostics must stay labeled as diagnostics. The reservoir horizon is five seconds; producers may fill gaps as late samples arrive inside that window, but they may not silently preserve older evidence in the live fusion cache.

OBS Program Surface

OBS should not expose raw unsynchronized ingest as broadcast controls once a synchronized audio/video program is available.

flowchart TD
    A["aligned audio program timeline"] --> B["setup_obs_synced_program.py"]
    B --> C["Host Voice stem"]
    B --> D["CoStreamer Voice stem"]
    B --> E["Ambient stem"]
    B --> F["Transients stem"]
    B --> G["CoStreamer Loopback stem"]
    B --> H["Local Loopback stem"]
    I["CultCache visual/audio timing"] --> J["Spout/Aquarium program video"]
    C --> K["OBS synchronized scene"]
    D --> K
    E --> K
    F --> K
    G --> K
    H --> K
    J --> K

Ownership:

• scripts/setup_obs_synced_program.py owns packaging aligned program audio into OBS-controllable stems and muting/disabling raw unsynchronized OBS inputs.
• scripts/capture_co_streamer_surfaces.py owns the late-arriving neighbor surface import: it captures neighbor Focusrite and neighbor loopback over SSH while recording local loopback, estimates the remote-family offset, and writes aligned co-streamer surfaces for the OBS stem packer.
• scripts/wasapi-loopback-capture.ps1 owns the direct primary-playback loopback path. It bypasses virtual mixers and asks Windows Core Audio for the default render endpoint loopback stream. It must run in the neighbor's interactive console session to receive render packets.
• OBS owns final source volume, track assignment, filters, and stream/record output.
• Aquarium/Spout owns synchronized program video, not raw remote desktop capture.

Invariant: raw SRT ingest, Desktop Audio, Mic/Aux, and other non-program scene items may exist for diagnostics, but they must be disabled or muted when broadcasting the synchronized program. Missing synchronized feeds should appear as explicit silent placeholders, not as live unsynchronized substitutes. The co-streamer Focusrite, co-streamer loopback, and remote video are the latest-arriving surface family; their measured delay should set the presentation buffer horizon instead of being patched in after the local field. Do not reintroduce Voicemeeter as the loopback authority without a concrete invariant; prefer direct WASAPI loopback or a sender FFmpeg build with native WASAPI input.