Mimir/docs/move-controller-stream.md

Move Controller Stream

Objective

Ship the first Mimir program surface around one concrete show shape: Kiyo Pro is the RGB program view, PS3 Eyes track multiple PS Move spheres, Raven screen capture enters as a synchronized video source, and Scarlett ASIO provides the two hero mics plus Raven program-loopback timing evidence.

Authority Map

• Owner: Mimir.Runtime owns sample ingest, rolling buffers, and typed visual evidence. Fensalir owns overlay presentation and final program pixels.
• Inputs: Kiyo Pro YUY2 video, both PS3 Eye Bayer8 feeds, Scarlett ASIO channels asio-ch0..asio-ch3, and Raven SRT screen capture on port 5200.
• Outputs: kiyo-pro-rgb is the base program view; PS Move controller histories lower as Feature evidence through MimirMoveControllerTracker; raven-display stays in the raven-sync clock domain; ASIO labels identify shotgun, cardioid, and Raven loopback lanes.
• Derived state: Move 2D histories are observations, not 6DoF pose. Raven display timing is derived from Scarlett audio evidence, not SRT arrival time.
• Forbidden writers: PS3 Eye blob tracks do not own camera calibration; Raven network timestamps do not own global sync; Kiyo Pro does not own high-rate motion truth.
• Cut line: this profile is the first streamable overlay product, not the full online calibration solve.

Nightwing Eyes/Moves Split

Nightwing is the right body for the PS3 Eyes and PS Move controllers: it has the spare USB topology, BlueZ/Bluetooth, and the local HID path. That does not make it a camera-placement authority.

The split is:

• Nightwing-local authority: read PS3 Eye frames, talk to PS Move HID/Bluetooth, apply LED schedules, preserve device timestamps, and extract compact sphere, marker, and sparse Eye feature observations.
• Starfire authority: own the canonical rolling window, Scarlett timing, global residual solve, camera placement, Kiyo/Raven program sync, and OBS program publication.
• Fensalir authority: own D3D12 dense feature extraction/fusion, surface claims, splat/reservoir resolution, and program pixels.

MimirDistributedWitnessConfigurations.NightwingEyesMoves records this as a typed witness: it may emit mimir.move_controller_observation_state, mimir.camera_feature_track_state, and mimir.visual_marker_state, but it must not stream raw Eye media as the normal live contract and must not own the canonical clock. Raw frames from Nightwing are diagnostic receipts only.

The first Nightwing receipt is good enough to build on: both PS3 Eyes enumerate as stock Linux ov534/gspca V4L2 cameras on /dev/video2 and /dev/video3. Default V4L2 cadence is only 30 fps, but explicitly setting timeperframe=1/187 before streaming makes both cameras deliver about 187 fps at 320x240 with zero sequence gaps. The witness worker must set frame interval; otherwise Linux will quietly look worse than it is.

Nightwing also has a typed witness publisher staged as ~/.local/bin/nightwing_typed_witness_publisher.py. The repo source is tools/nightwing_typed_witness_publisher.py. It sends binary MessagePack mimir.eve_sensor_observation.v1 records to Mimir.EveSensorReceiver on ws://192.168.1.66:8796/eve/periwinkle; the receiver normalizes them into the Mimir observation ledger, Odin projects them as observation-stream nodes, and Nightwing's Eve TUI lowers those nodes in the CultMesh Witness Streams panel. The current status records are camera-device receipts for /dev/video* and a nightwing-leap-tracking placeholder that is intentionally 0 until real Leap tracking owns that signal. Values are [present, videoIndex, isPs3Eye] for camera-device receipts.

MimirRoomCalibrationLockSolver is the first room-lock coordinator. It consumes the LED string candidate/residual, the camera-rig pose update frame, Move controller feature-history candidates, and PS3 Eye sparse-feature candidates. It only reports Locked when every witness family is present: multiple cameras share enough LED indices, the rig solver produced enough pose updates with low mean ray residual, multiple Move identities have stable history, and multiple Eyes contribute enough stable feature tracks. Missing families are reported by name instead of being papered over. Mimir.BufferSmoke --room-calibration-lock-smoke proves a synthetic three-camera LED string, two Moves, and two Eyes lock at confidence 0.778, while removing Move history correctly drops the result to InsufficientWitnesses.

Runtime Profile

Use:

$env:MIMIR_RUNTIME_CONFIG = "E:\Projects\Mimir\config\mimir-runtime.move-stream.local.json"
dotnet run --project .\src\Mimir.App\Mimir.App.csproj

The profile declares:

• kiyo-pro-rgb: Kiyo Pro program view, 640x480 YUY2 at the current reliable 25 fps class.
• ps3-eye-0, ps3-eye-1: high-rate Move sphere visual witnesses.
• focusrite-asio: one ASIO producer that emits: asio-ch0 shotgun hero mic, asio-ch1 cardioid hero mic, asio-ch2 Raven program loopback L, and asio-ch3 Raven program loopback R.
• raven-display: Raven screen capture over SRT port 5200.

Run the synthetic overlay smoke:

dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-controller-overlay-smoke

Run the live ingest smoke once Raven is sending:

$env:MIMIR_RUNTIME_CONFIG = "E:\Projects\Mimir\config\mimir-runtime.move-stream.local.json"
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --seconds 5 --require-samples --sync-reference asio-ch2

Raven Sender

On Raven, start screen capture from the logged-in desktop session:

E:\Projects\Mimir\scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200

Route Raven program audio into the Scarlett path that appears on asio-ch2/asio-ch3. That audio is the timing witness that earns the raven-sync correction for raven-display.

Multiple Move Controllers

Do not collapse Move observations into a generic bright blob. Each controller has its own controllerId and expected RGB/time code. The PS3 Eyes can track the sphere as a high-rate brightness witness; Kiyo Pro can later confirm color identity; the commanded flash schedule ties the observed marker to the controller identity.

MimirMoveControllerTracker keeps a bounded history per controller and lowers those histories into the existing Feature evidence path. This is deliberately 2D observation history. 6DoF pose waits for calibrated frustums and the global residual owner.

Music-Keyed Calibration Words

tools/music_keyed_move_chirp_sync.py is the first planner for calibration words that are both musical and machine-decodable. It listens to Scarlett ASIO, uses the same Perlines-style adaptive FFT delta whitening as tools/nightwing_psmove_music_pulse.py, autocorrelates the whitened broadband spectral-rise function for tempo, estimates a lightweight root from recent best-fit fundamentals, then writes one shared plan:

• beat-aligned chirps quantized to the estimated minor-pentatonic scale;
• de Bruijn-coded event symbols so adjacent timing/frequency/color words remain distinguishable;
• per-Move visual words using stable controller identities and unique colors; the default is no longer a square flash, but a sampled RGB contour with attack/release envelope, symbol-dependent hue glide, tremolo articulation, and an emphasized controller lane;
• a JSON receipt shaped like AquaSynth song analysis vocabulary: tempo_bpm, beat_seconds, tempo_confidence, root_note, suggested_scale, scale_frequencies_hz, and whitened_spectral_autocorr.

The synthetic smoke is:

& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' .\tools\music_keyed_move_chirp_sync.py --mode synthetic --dry-run --events 8 --out-dir artifacts/runtime/music-keyed-move-chirp-sync-smoke

The live dry-run against Scarlett is:

& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' .\tools\music_keyed_move_chirp_sync.py --dry-run --analyze-seconds 5 --events 8 --out-dir artifacts/runtime/music-keyed-move-chirp-sync-live-dry

For double-time material:

& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' .\tools\music_keyed_move_chirp_sync.py --dry-run --tempo-grid doubletime --tempo-max-bpm 260 --analyze-seconds 5 --events 8 --out-dir artifacts/runtime/music-keyed-move-chirp-sync-dirtyphonics-dry

On the 2026-05-31 live dry-run, the planner produced a usable but not final receipt: about 92.31 BPM, beat 0.650s, tempo confidence 0.343, root A, key confidence 0.716. That is enough to prove the dataflow, not enough to claim final musical tempo authority.

For dense double-time material, use --tempo-grid doubletime; the planner then selects the double-time member of the autocorrelation tempo family instead of preferring the slower parent grid. On the Dirtyphonics-style live source this produced about 171.43 BPM, beat 0.350s, tempo confidence 0.358, root A, key confidence 0.703 after widening the search ceiling with --tempo-max-bpm 260. The receipt records tempo_family so later decoders can see the selected grid alongside half-time/double-time alternatives.

Visual calibration words are deliberately shaped like the audio words: one gesture carries more than one bit. --visual-gesture contour records a mimir.move_visual_contour_word.v1 per Move per event, with sample_rate_hz, duration, base identity color, symbol emphasis, and the exact RGB samples sent over HID. Start conservatively around --visual-gesture-hz 80; the HID writer accepted a short 80 Hz contour test on Nightwing, but the useful rate is the rate cameras can recover cleanly, not merely the rate the controller accepts. --visual-gesture square remains available as a baseline comparator.

tools/sync_sweep.py can now consume that exact plan with --music-plan. That makes the plan the single authority for audio chirps, sampled Move RGB contours, and offline video scoring. The sweep writes gesture-plan.json, event-schedule.json, copied/rendered chirp-train.wav, per-sensor captures, and summary.json. RGB camera receipts include a visual_contour block with brightness-envelope correlation and color-chroma error against the expected Move contour. The old peak-only score is still present as a timing witness; it no longer pretends square flashes are the only visual language.

When --music-plan is used, the sweep also runs typed Mimir runtime sensor receipts unless --skip-mimir-live is passed. The default receipts are:

• config/mimir-runtime.perfect-machine.local.json for local Leap stereo IR, Kiyo Pro, Kiyo, and local camera texture/depth/field-resource ingress;
• config/mimir-runtime.raven-eve.example.json for Raven display and Eve camera/mic frame-event listeners.

That receipt path is not raw media archival yet. It is the audit hook that prevents Leap or Eve from being omitted by a DirectShow/Nightwing-only sweep.

After a sweep, run tools/crunch_sync_sweep.py over the sweep directory to extract a compact calibration observation table. It merges Nightwing's typed Move observations with local Kiyo/Kiyo Pro frame analysis, estimates each local video's schedule shift against the event train, and writes calibration-observations.json plus calibration-observations.csv.

Example using an already-written musical plan:

& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' .\tools\sync_sweep.py --music-plan artifacts/runtime/music-keyed-move-chirp-sync-live/music-keyed-move-chirp-plan.json --out-dir artifacts/runtime/all-sensor-articulated-sync
& 'C:\Users\Meta\.cache\codex-runtimes\codex-primary-runtime\dependencies\python\python.exe' .\tools\crunch_sync_sweep.py artifacts/runtime/all-sensor-articulated-sync

The non-dry run targets the two wireless Nightwing Moves by default: 00:07:04:a6:be:5f on /dev/hidraw2 and 00:06:f5:23:e2:d1 on /dev/hidraw3. Nightwing now carries /etc/udev/rules.d/70-move.rules, which sets Bluetooth PS Move hidraw nodes to root:input 0660. The metacrat user is in input, so the non-root SSH writer can drive both spheres. Root SSH is available as the short local alias nwroot for maintenance only; the hot LED path should stay non-root.