Mimir/state/map.yaml

workspace:
  name: LocalCastBridge
  status: active
  focus: perfect-machine-native-reservoir
  premise: >
    A second Windows PC should appear inside OBS as video plus separately
    mixable audio sources. Use ordinary media transport first; native OBS code
    only earns its keep after standard OBS/FFmpeg surfaces fail a real invariant.

objective:
  primary: >
    Build one synchronized spatial stream machine from six cameras, six
    microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP,
    and OBS output.
  current: >
    Cut the deadline bridge down to tooling and define the native five-second
    reservoir contract Aquarium/Faust must implement.

constraints:
  - Sender is Windows.
  - Receiver uses OBS.
  - Video encode must happen on the sender using NVENC when available.
  - Audio sources must remain separately controllable in OBS.
  - V1 is LAN-only.
  - Persistent state must clarify rehydrate, not become transcript tax.

invariants:
  - OBS composition authority stays in OBS.
  - FFmpeg owns capture, encoding, and transport.
  - One stable port equals one OBS source endpoint.
  - Video endpoint uses H.264 over MPEG-TS/SRT.
  - Audio endpoints default to AAC over MPEG-TS/SRT for broad OBS/FFmpeg compatibility.
  - Local secrets remain untracked.

accepted_design:
  - `docs/perfect-machine.md` and `config/perfect-machine.example.json` define the native target: one five-second spatiotemporal reservoir, Aquarium GPU fusion/rendering, Faust DSP, OBS output controls, and LocalCastBridge tooling.
  - `native/reservoir` is the first native Rust crate. It implements shared-edge reservoir rings so all sample types expire from the newest live sample rather than private source histories.
  - `config/localcast.example.json` defines receiver address, base port, video encode settings, and audio sources.
  - `scripts/sender-discover.ps1` lists FFmpeg devices, protocols, and encoders.
  - `scripts/sender-start.ps1` launches one FFmpeg process per enabled source.
  - OBS receives each source through Media Source URLs documented in `docs/obs-receiver-setup.md`.
  - Persistence follows the active repo pattern: map, scratch, evidence, branches, handoff, system map, implementation plan, state helpers.

active_subgoals:
  - Implement the Perfect Machine native reservoir contract described by `docs/perfect-machine.md` and `config/perfect-machine.example.json`.
  - Move dense visual fusion/material fitting/brush generation into Aquarium GPU compute.
  - Move hot audio DSP/voice separation/room suppression/spatialization into Faust/native runtime.
  - Validate the actual sender FFmpeg build and capture device names.
  - Smoke-test the video endpoint in OBS.
  - Smoke-test each audio endpoint in OBS and tune latency.
  - Decide whether an OBS source generator or plugin investigation is justified.
  - Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock.
  - Build the six-microphone/two-speaker audio-field sidecar around the actual distributed camera/Focusrite clocks, explicit geometry, speaker calibration, bounded buffering, delay/SRO alignment, testable pipeline seams, and FOA AmbiX output.
  - Build the visual sensor-fusion sidecar around timestamped observations, calibrated sensor models, bounded track caches, sparse point-cloud output, and later brush/splat packet lowering.

research:
  ambisonic_aquasynth:
    summary: research/ambisonic-aquasynth/summary.md
    bibliography: research/ambisonic-aquasynth/bibliography.md
    key_decision: >
      A spatial field must be built from one clocked multichannel capture path
      feeding a declared ambisonic bus. The existing one-audio-source-per-SRT
      endpoint design remains valid for plain OBS mixing, but it is not the
      authority for synchronized ambisonic DSP.
  live_adaptive_sync:
    summary: research/live-adaptive-sync/summary.md
    bibliography: research/live-adaptive-sync/bibliography.md
    key_decision: >
      If independent mic clocks must be used, sync needs two explicit loops:
      slow sampling-rate-offset estimation driving variable-rate resampling,
      and faster delay/TDOA tracking for current alignment and localization.
      Prove this with a two-mic harness before connecting it to Ambisonics.
  playback_calibration:
    summary: research/playback-calibration/summary.md
    bibliography: research/playback-calibration/bibliography.md
    key_decision: >
      Speaker output is a layout-specific render stage, not a generic sink.
      The system needs a playback profile with decoder matrix/layout,
      output-device latency, channel mapping, per-speaker delay/gain/response
      correction, and a latency ledger. Keep playback calibration separate from
      capture-side adaptive sync.
  feedback_calibration:
    summary: research/feedback-calibration/summary.md
    bibliography: research/feedback-calibration/bibliography.md
    key_decision: >
      Speaker feedback is also a calibration signal when the outgoing audio is
      known. Model it with an online speaker-to-mic transfer matrix and expose
      confidence/convergence state. Keep this estimator separate from clock
      sync, localization, and Ambisonic decoding.
  visual_spatial_map:
    summary: research/visual-spatial-map/summary.md
    bibliography: research/visual-spatial-map/bibliography.md
    brushstroke_rendering: research/visual-spatial-map/brushstroke-fusion-rendering.md
    key_decision: >
      The visual rig is two PlayStation Eye / PS3 Eye cameras as high-rate
      low-resolution marker trackers, two Razer Kiyo-class RGB/mic cameras
      including one Kiyo Pro, and one Leap Motion/LeapUVC stereo IR sensor for
      near-field fusion. LeapUVC driver mode makes Leap appear as a generic UVC
      device and should be treated as mutually exclusive with Ultraleap software
      tracking unless proven otherwise. Build one canonical world coordinate
      system for all visual sensors, microphones, speakers, props, virtual
      camera pose, and ambisonic listener pose. Prove sparse calibrated tracking
      before adding RGB projection or Gaussian splat rendering. For a
      Dreams-like volumetric brushstroke look, keep the world model as truth and
      lower semantic scene claims into cached brush/splat packets under budget.
  gpu_fusion_splatting:
    summary: research/gpu-fusion-splatting/summary.md
    bibliography: research/gpu-fusion-splatting/bibliography.md
    key_decision: >
      The live visual fusion/render hot path should be GPU-resident: pinned host
      frame rings, async CUDA uploads, GPU marker/stereo/feature extraction,
      GPU-shaped observation and brush packet buffers, and CUDA/Vulkan/D3D
      splat rendering with minimal readback. CPU owns orchestration and
      semantics; kernels do not own calibration truth.
  spout_obs_bridge:
    summary: research/spout-obs-bridge/summary.md
    bibliography: research/spout-obs-bridge/bibliography.md
    key_decision: >
      OBS should receive a named Spout2 shared texture from Aquarium Engine,
      not desktop capture or file polling. LocalCastBridge emits typed
      render-frame packets with source timestamps, intended presentation time,
      audio-alignment time, target size, Spout sender name, and point/brush
      claims. The deadline Spout sink now defaults to a `kiyo-mid-deru`
      virtual camera, with the eye halfway between the two Kiyo-class cameras
      and the target on the co-streamer body volume. Its point budget is
      temporal: stable high-confidence anchors plus a frame-varying remainder
      for TAA/supersampling. Aquarium owns GPU rendering and Spout publication.
  live_spout_deadline_sink:
    summary: docs/obs-spout-streaming.md
    key_decision: >
      The deadline OBS surface is a real Spout sender named LocalCastBridge
      Point Cloud. `scripts/stream_spout.py` consumes render-frame packets,
      lowers them into compact anisotropic screen brush packets inspired by the
      Zyphos/Aquarium brush model, uploads the texture through SpoutGL, and
      writes a heartbeat status file. This is a sink boundary, not a new scene
      authority; Aquarium should replace the renderer body behind the same
      RenderFramePacket contract for dense brush/splat rendering.
  live_audio_phase_meaning:
    summary: docs/audio-field.md
    key_decision: >
      Phase/chirplet evidence now has a live semantic boundary:
      `audio_field.phase_meaning` keeps raw phase internal and publishes
      `localcast.audio.phase_field` with delay correction, coherence,
      confidence, distance-equivalent delta, reference-bleed/suppression
      weights, correction energy, and active-probe need. `audio_field.active_probe`
      closes the next loop by feeding that confidence state into
      ActiveProbeOptimizer and emitting bounded chirplet WAVs/manifests, with
      optional playback. The live launcher now uses dense harmonic
      near-ultrasonic probe textures so aliases have a musical relationship
      instead of arbitrary squeal. Probe artifacts rotate through fixed slots
      and the manifest rotates at a byte cap. `scripts/stream_live_audio_field.py`
      is the current live local capture owner: it writes mic-field and
      phase-field CultCache documents from visible local mics plus Scarlett
      loopback, sends probes through the selected Scarlett output, resamples
      probe playback when that output is 44.1 kHz, and reports missing
      distributed channels as explicit placeholders. Active probe scheduling is
      restricted to source ids with live local capture devices, so placeholders
      do not consume chirp budget. `audio-phase-field-status.json`
      declares input mode and whether active probes are closed-loop; current
      deadline mode is `live-local-capture` with closed-loop probe capture when
      Scarlett loopback is available.
      This is learned audio-field state, not a phase display.
  faust_voice_separation:
    summary: docs/aquarium-spatial-audio.md
    key_decision: >
      Voice separation should run in Aquarium's Faust DSP, not as Python hot
      audio math after the mic field has already collapsed to AmbiX.
      LocalCastBridge now publishes `localcast.audio.mic_field` with six
      aligned mic channels, roles, controls, and graph id
      `localcast.faust.voice_separation.v1`; the matching Faust graph lives at
      `faust/localcast_voice_separation.dsp`.
  sensor_fusion_architecture:
    summary: docs/sensor-fusion-architecture.md
    key_decision: >
      The visual world model begins as timestamped sensor observations and
      calibrated camera transforms, not render packets. The first sidecar keeps
      capture adapters, detection, triangulation, track caching, and point-cloud
      export as separate testable authorities. This follows the fractal-domain
      cache rule: semantic claims first, cached backend lowerings second.
      Stochastic transient matching now lowers descriptor-consistent image
      observations into scene-ray hypotheses and a multi-LOD world-space cache
      for Aquarium compute-shader reconciliation. Leap evidence is the
      highest-priority temporal/spatial source when live Leap frames are present;
      fallback Leap frames stay tagged as fallback evidence and must not be
      treated as ground truth. Live clap calibration now uses the Kiyo stereo
      pair for the first rough 3D impact solve and includes Leap motion as the
      high-precision visual timing witness; Leap should become a geometric
      camera authority after shared calibration. Clap peaks update per-camera
      clock offsets, allowing history buffers to be sampled at
      `oracle_time - camera_offset` for coherent point-cloud construction.
      `localcast.sensor_fusion.chirp_pose` converts live phase-field delay
      meaning for Kiyo/PS Eye microphone sources into camera-body range
      constraints and per-camera pose-correction estimates, which the live
      producer emits as `camera-chirp:*` and `camera-pose-correction:*` render
      points plus `chirp-camera-pose` LOD evidence. It can load geometry from
      the audio-field profile, falling back to the example/default geometry
      until measured local coordinates exist. These constraints augment visual
      calibration; they do not own image geometry. The Spout point-budget policy
      pins calibration and cross-modal constraints before spending adaptive LOD
      budget on stable high-priority surfaces and frame-rotated samples, and
      its JSON heartbeat reports budgeted point prefix counts. The live fusion
      producer now has an explicit five-second spatiotemporal reservoir:
      RGB/Leap frame history expires from the newest shared sample, LOD evidence
      is clipped to the same window, and each RenderFramePacket declares that
      source window for Aquarium/OBS synchronization. Multi-LOD cells now carry
      first-pass material estimates, weighted albedo plus roughness/metallic
      hints, so Aquarium can relight/refine brush packets from reservoir state
      instead of scraping screen-space color. The Spout/audio overlay boundary
      clamps stale visual packets to the current audio reservoir edge before
      OBS sees them. Current live deadline mode uses fallback-only RGB/Leap
      because synchronous OpenCV camera reads measured slower than the reservoir;
      the next coherent cut is nonblocking camera ingress that continuously
      fills the five-second reservoir with real samples. `LatestFramePump` now
      moves blocking OpenCV reads off the fusion hot loop; the running deadline
      mode uses lower per-frame sample density so Spout can render visual points
      inside the five-second budget while TAA/reservoir accumulation carries
      detail over time.
  typed_visual_state:
    summary: docs/typed-visual-state.md
    key_decision: >
      Live visual state now lives in typed CultCache MessagePack documents:
      `localcast.visual.render_frame` in `calibration/runs/visual-state.msgpack`
      and `localcast.visual.stream_status` in
      `calibration/runs/visual-stream-status.msgpack`. CultNet document_put is
      the intended process/network API boundary. JSON render-frame files are no
      longer the authority.

current_status:
  summary: >
    First repo scaffold is in place and deployed to the neighbor sender at
    C:\Meta\LocalCastBridge. FFmpeg 8.1.1 with SRT and h264_nvenc is installed
    there, direct WASAPI console-session capture now owns system-output loopback, Madman's desktop has
    start/stop launchers, the sender script has launcher-safe FFmpeg arguments
    and repo-root logs, desktop launchers use absolute tool paths, and the
    receiver OBS scene has matching Media Sources. Local calibration harness now
    has first physical-device access: LeapUVC raw IR video, two Razer Kiyo-class
    RGB cameras, all expected microphones, and Scarlett speaker output are
    accessible. LeapUVC MSMF mode probing accepts a 320x240x120 request and
    reports about 115 fps, though measured OpenCV throughput is lower in the
    first rough probe. The opentrack/libusb PS3 Eye path still enumerates modes
    but fails frame reads across libusb-win32, libusbK, and WinUSB. The
    AllanCat PS3 Eye Universal Driver / PS3EyeDirectShow path now exposes both
    PS3 Eyes as DirectShow indices 3 and 4 after registering the source filter
    for two cameras. OpenCV index capture is reliable at default 640x480 near
    30 fps but does not negotiate high-rate modes correctly. FFmpeg DirectShow
    capture by device name negotiates the filter pins properly: both Eyes can
    run simultaneously at 320x240@60, and dual 640x480@60 lands around 54-56
    fps per Eye. Requests above 60 do not increase delivered frame rate through
    the installed filter. Audio-field scaffolding now exists as a separate sidecar
    for the actual distributed rig: two Kiyo mics, two PS Eye mics, local
    Focusrite ADC with a shielded cardioid near the host's face, neighbor
    Focusrite ADC with a shotgun mic pointed at the co-streamer, and two local
    speakers. The Focusrite mics are high-quality dialogue anchors; camera mics
    provide spatial/context evidence. The tool refuses to treat independent
    USB/network clocks as an Ambisonic field until they have been aligned and
    resampled into one six-channel reference timeline. Local PortAudio/WASAPI
    probe shows the Scarlett input accepts 48 kHz but rejects 96 kHz, while the
    Scarlett output accepts 44.1 kHz and rejects 48/96 kHz on this path; check
    ASIO/native Focusrite access before relying on high-rate calibration. Audio
    latency is now an explicit bounded-buffering policy: output may run behind
    live input while delay/SRO estimates settle, but cache depth must converge
    toward real-time. Core audio modules now expose testable boundaries for
    buffering, latency convergence, injectable capture/alignment/encode/sink
    ports, and pipeline orchestration. Calibration analysis now has a local
    chirplet refinement pass: coarse sweep match followed by fractional-delay
    and chirp-rate grid search around the detected arrival. Runtime sync is now
    modeled as per-block known-speaker chirplet telemetry updating delay, SRO,
    phase, and confidence state; low-confidence observations freeze rather than
    poisoning alignment. Active probe optimization owns extra chirplet emission
    when sync confidence drops, bounded by spacing, level, and audibility budget.
    The first visual
    sensor-fusion sidecar is
    in place: `localcast.sensor_fusion.core` owns calibrated camera models,
    timestamped 2D observations, DLT triangulation, reprojection/confidence
    gating, a TTL track cache, and PLY point-cloud export; adapters own
    FFmpeg/raw-frame ingress; `scripts/sensor_fusion.py fuse-file` can turn an
    observation file into a sparse PLY. The render bridge now lowers cached
    points into `localcast.sensor_fusion.render_frame.v1` packets carrying
    source time ranges, intended visual presentation time, audio-alignment
    time, target render size, Spout sender name, and point draw metadata for
    Aquarium/Spout. A deadline OBS Spout sender is now live-shaped:
    `scripts/live_sensor_fusion.py` now writes fused render frames into typed
    CultCache state at `calibration/runs/visual-state.msgpack`. `scripts/stream_spout.py`
    consumes that CultCache document, lowers points into compact anisotropic
    screen brush packets, uploads an OpenGL texture, publishes Spout sender
    `LocalCastBridge Point Cloud`, and writes both typed status
    `calibration/runs/visual-stream-status.msgpack` and compatibility JSON
    `calibration/runs/stream-spout-status.json`; OBS now has a Spout2 Capture
    source pointed at that sender, and an OBS websocket screenshot verified
    nonblack brush output. OBS synchronized program setup now derives six
    per-channel stems from the aligned program audio timeline, creates/updates
    OBS Media Sources for host voice, co-streamer voice, ambient, transients,
    co-streamer loopback, and local loopback, and can disable every scene item
    except synchronized LocalCastBridge program surfaces. Co-streamer surface
    capture now records neighbor Focusrite and neighbor loopback while recording
    local loopback ground truth, then writes aligned co-streamer surface WAVs
    for the stem packer. Live phase-field meaning now writes
    `localcast.audio.phase_field` to `calibration/runs/audio-phase-field.msgpack`
    from internal chirplet/phase evidence so downstream code sees delay,
    coherence, confidence, suppression, correction, and probe-control state
    instead of raw phases. The neighbor desktop launcher no longer uses
    Voicemeeter; it restores the Focusrite speakers as the primary playback
    device and the Focusrite analogue input as the default capture device. A
    direct Core Audio WASAPI loopback shim now works when launched in the
    neighbor's interactive console session; SSH-only launches can see the
    endpoint but receive no render packets.
    Missing synchronized neighbor feeds are explicit
    silent placeholders, not raw unsynchronized substitutes. The live producer currently uses synthetic calibrated
    observations through the real `SensorRig.fuse` path until camera detector
    observations replace them. The RGB path now has a debug-only CPU dense
    stereo reference: the two Kiyo-class frames can be block-matched as a
    provisional calibrated pair and triangulated into `dense-rgb:*` surface
    claims, but production dense reconstruction must move to Aquarium/GPU
    compute rather than Python loops. A separate
    adaptive camera-control policy measures luminance, clipping, contrast, and
    sharpness, then emits normalized exposure/gain/focus commands through the
    OpenCV capture adapter when the driver accepts them. Active illumination now
    has an optional Tuya local-network pulse hook for deliberately modulating
    room light as calibration telemetry; pulse timestamps can be aligned against
    camera brightness/depth response. Common-space calibration now has a real
    owner: fixed-board ChArUco observations plus intrinsics solve
    `CameraModel.world_from_sensor` transforms, and cross-view surface-feature
    observations can be descriptor-matched and triangulated into world-space
    tracks. Unit tests inject
    synthetic cameras and frames so the core does not depend on live camera
    drivers. The multi-LOD visual cache now carries source kind and source
    priority so Aquarium can prefer real Leap evidence over RGB surfaces,
    stochastic transient hypotheses, and synthetic fusion support while still
    seeing lower-priority fallback diagnostics when the Leap capture path stalls.
  next_action: >
    Capture real ChArUco intrinsics for the PS3 Eyes and Kiyos, capture
    fixed-board observations from every camera, solve a local
    `config/sensor-fusion.json`, and prove calibrated cross-view surface tracks
    with `scripts/triangulate_surface_features.py`. Then move the dense RGB stereo matcher
    to a GPU-resident stereo/flow pass so high-detail surface claims can scale
    toward the million-splat temporal accumulator without Python owning each
    point. If a Tuya light is available, run the local-key pulse hook and record
    its timestamps against camera responses as active illumination evidence.
    Then replace the synthetic live producer with a detector adapter that
    turns FFmpeg PS3 Eye frames into marker `Observation2D` records for the
    fusion core and publishes the same typed CultCache/CultNet render-frame
    document. Calibrate Leap as a geometric camera model so clap timing peaks
    can become direct 3D rays instead of timing-only evidence. Also fix the live Leap capture backend so the cache emits
    `leap-ground-truth` rather than `leap-fallback`; current MSMF probing can
    still stall on frame reads. In parallel, move the render-frame consumer into Aquarium Engine so
    Aquarium renders cached points/brushes into its GPU render target and
    publishes or hands off the same Spout sender `LocalCastBridge Point Cloud`
    for OBS, replacing the deadline OpenGL point sink without changing packet
    ownership. In parallel, create local
    `config/audio-field.json`, confirm local
    Kiyo/PS Eye/Focusrite device matches, confirm the neighbor Focusrite
    shotgun capture/transport path, replace placeholder mic/speaker geometry
    with measured world coordinates, then build the delay/SRO alignment stage
    that feeds the bounded field cache and emits aligned six-channel blocks for
    FOA encoding. Keep neighbor loopback on the scheduled interactive WASAPI
    capture path and use it as the timing witness once actual neighbor app
    audio is present; do not reintroduce a virtual mixer unless direct loopback
    fails a concrete invariant that the mixer actually satisfies.