Perfect Machine

Objective

Build one synchronized spatial stream machine from six cameras, six microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP, and OBS output.

The bridge scripts are not the product. They are scaffolding, launch surfaces, calibration tools, and compatibility probes. The live hot path belongs to native runtime code, GPU compute, and Faust.

Current Mechanism

The repo currently proves the contracts:

flowchart TD
    A["camera/audio/phase/render producers"] --> B["LocalcastRuntime C ABI"]
    B --> C["5s native reservoir"]
    C --> D["Aquarium GPU + Faust DSP"]
    D --> E["Spout/audio stems"]
    E --> F["OBS"]

The old CultCache/Python/Spout path remains a compatibility diagnostic. It is not the runtime API.

Invariants

One five-second spatiotemporal reservoir is the live authority.
The reservoir is keyed by one shared presentation clock.
Every sample has sensor id, local hardware time if available, arrival time, calibrated world transform, confidence, and provenance.
Missing data is absence, not a stale substitute.
Late data can refine the reservoir while it is inside the five-second window.
Anything older than the live reservoir is offline calibration evidence, not broadcast geometry.
Audio clocks, camera clocks, Leap timing, remote video, and OBS presentation are reconciled before rendering.
Aquarium owns dense visual fusion, temporal accumulation, material fitting, brush/splat generation, final render target, and Spout publication.
Faust owns hot audio DSP: voice separation, room suppression, Ambisonic/HOA spatialization, and loopback/source stem generation.
LocalCastBridge owns configuration, calibration, typed contracts, launch, status, persistence, and compatibility surfaces.

Native Runtime Shape

flowchart TD
    A["camera capture workers"] --> R["5s native reservoir"]
    B["mic/loopback capture workers"] --> R
    C["Leap worker"] --> R
    D["remote video/audio ingest"] --> R
    E["speaker probe scheduler"] --> R
    R --> F["clock/pose optimizer"]
    F --> R
    R --> G["Aquarium GPU fusion"]
    G --> H["multi-LOD surface/material field"]
    H --> I["brush/splat/TAA renderer"]
    R --> J["Faust DSP graph"]
    J --> K["program stems + spatial bed"]
    I --> L["Spout2 texture"]
    K --> M["OBS audio controls"]
    L --> N["OBS video source"]

Reservoir Contract

The reservoir stores one time-ordered rolling buffer of sample handles, not one giant JSON-ish object and not independent per-kind histories. Typed views over that buffer expose the kinds consumers need.

Required sample kinds / typed views:

camera_frame: raw or GPU-importable image handles plus timestamps.
camera_feature: keypoints, descriptors, optical flow, confidence, sensor id.
scene_ray: calibrated rays from feature observations.
surface_claim: triangulated or depth-estimated world samples.
material_claim: albedo, roughness, metallic/specular hints, view direction.
audio_block: aligned mic/loopback samples and per-source clock state.
phase_claim: delay/SRO/phase/coherence estimates from chirplets and program reference.
event_claim: claps, transients, gestures, active illumination pulses.
render_packet: selected brush/splat work for the current presentation frame.

The reservoir owns retention. Producers append. Optimizers refine. Renderers sample. No stage owns a private history that can outlive the reservoir.

LocalcastRuntime is the first native owner above the raw reservoir. It exposes typed producer calls for each sample kind so camera capture, audio capture, phase estimation, material fitting, event detection, and render planning all append into one shared five-second window without crossing a Python file cache.

Visual Fusion

Camera fusion is feature matching across the reservoir, not a latest-frame operation.

The visual path should:

ingest all cameras continuously into pinned/native buffers referenced by reservoir sample handles;
extract features and flow on GPU where possible;
match every plausible cross-view feature within time and epipolar bounds;
use Leap as the strongest near-field timing/spatial witness when live;
estimate surface points, normals, confidence, and material observations;
keep multi-LOD cells resident for Aquarium compute reconciliation;
emit only the frame budget needed for OBS while TAA accumulates detail.

The live target is not "points per frame." The live target is evidence per reservoir window and how much of it Aquarium can reconcile before presentation.

Audio Fusion

The audio path should:

align all six microphones into one clocked field;
use Focusrite dialogue mics as voice anchors;
use camera mics as spatial/context witnesses;
use loopback/program audio as ground truth where available;
emit active probes only for channels that can observe them;
continuously update delay, SRO, phase, frequency response, and confidence;
feed Faust with aligned mic channels and control telemetry;
produce host voice, co-streamer voice, ambient, transients, local loopback, co-streamer loopback, and spatial bed as synchronized outputs.

Raw phase is internal. Meaning crosses the boundary.

Cut Line

Cut these from the hot path:

Python dense rendering.
Python camera feature extraction in the presentation loop.
File polling as a native runtime API.
OBS-side sync between independent raw sources.
Stale geometry clamped into looking current.
Probe scheduling against placeholder channels.

Keep these as tooling:

Python calibration scripts.
Python contract tests.
Python device discovery.
Compatibility Spout sender for diagnostics.
Offline analysis and reproducible evidence ledgers.

First Native Cut

Define the native reservoir ABI from config/perfect-machine.example.json.
Build a native reservoir process or Aquarium module that owns the five-second rolling buffer.
Move camera ingest into native capture workers that write reservoir samples.
Move audio ingest/phase claims into the same presentation-clock reservoir.
Let Aquarium consume reservoir surfaces/material claims directly on GPU.
Let Faust consume reservoir audio blocks and source controls directly.
Keep the old bridge only as a comparison and fallback surface.

Native Work Started

native/reservoir is the first native crate. It implements the shared-edge five-second rolling reservoir invariant:

append samples into one rolling buffer with typed views;
advance the reservoir edge from the newest sample;
evict all sample kinds against the same edge;
query the latest valid sample per sensor inside a typed view.

It now builds as rlib, cdylib, and staticlib, with a small C ABI declared in native/reservoir/include/localcast_reservoir.h:

create/destroy an opaque LocalcastReservoir;
push timestamped sample metadata into a typed view of the rolling buffer;
advance/query the shared edge and window start;
query total length and typed-view length;
read samples by total rolling-buffer index or typed-view index;
query the latest sample for a sensor hash.
reject diagnostic/probe/fallback samples at the live reservoir boundary via sample provenance flags.
create/destroy an opaque LocalcastRuntime;
append typed camera/audio/phase/event/render handles through producer calls;
query a native runtime status struct with shared edge, window start, total count, and typed-view counts.

The ABI carries payload_handle, not payload bytes. Aquarium owns image, feature, surface, material, and GPU memory interpretation. Faust/native DSP owns audio buffer interpretation. The reservoir owns only retention, sample kind, timestamp order, and the five-second shared-edge invariant.

7.7 KiB Raw Blame History