759 lines
52 KiB
YAML
759 lines
52 KiB
YAML
workspace:
|
|
name: Mimir
|
|
status: active
|
|
focus: direct-driver-fensalir-stream-app
|
|
implementation_name: Mimir public surface / native runtime contracts
|
|
premise: >
|
|
Mimir is the local realtime field machine: cameras, microphones, speakers,
|
|
network feeds, and timing signals become one coherent OBS-facing volumetric
|
|
program surface. The live path is a C# Fensalir-hosted app with in-memory
|
|
rolling buffers and native capture/runtime boundaries.
|
|
|
|
objective:
|
|
primary: >
|
|
Build one synchronized spatial stream machine from six cameras, six
|
|
microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP,
|
|
and OBS output.
|
|
current: >
|
|
Make Leap dense geometry visible and calibratable first: live Leap packed
|
|
stereo IR feeds Fensalir D3D12 disparity, Mimir declares a derived
|
|
point-cloud Mesh socket, then Fensalir should generate/render points in the
|
|
editor and feed global residuals into online calibration.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must use NVENC when the bridge path is active.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 bridge is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
- Live ingest belongs to Mimir.Runtime, native drivers, Fensalir, and
|
|
Faust/native DSP.
|
|
- Edge JSON is schema/diagnostic only; live process data is typed or in
|
|
memory.
|
|
- Networked feeds are producers into the live runtime, not independent clock
|
|
authorities.
|
|
- Mimir explicitly trades latency and memory for throughput and field quality:
|
|
the default five-second buffer is compute budget for stream alignment,
|
|
GPU/audio analysis, and volumetric sound-field extraction.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport only for bridge utilities.
|
|
- One stable bridge port equals one OBS source endpoint.
|
|
- One rolling live window owns the timing edge and expiry.
|
|
- The rolling window is not accidental latency. It must preserve enough time
|
|
to line up feeds and crunch data before OBS-facing output is emitted.
|
|
- Typed views may index the rolling buffer, but no per-kind ring owns
|
|
retention.
|
|
- Fensalir owns production UI, GPU fusion, D3D12 interop, Spout2 output, and
|
|
the EVE shared D3D12 program-output texture.
|
|
- Mimir publishes physical observations, calibration constraints, and surface
|
|
intent. Fensalir owns field claims, selected lowerings, evidence reservoirs,
|
|
temporal guide lanes, and program presentation.
|
|
- Fast stereo depth belongs to Fensalir's D3D12 compute/resource authority.
|
|
`research/d3d12-stereo-depth-provenance-2026-05-29.md` records libSGM as
|
|
the permissive north-star reference for a future SGM-shaped stereo
|
|
disparity lane over synchronized rectified texture pairs and calibration
|
|
state. CUDA, TensorRT, and monocular depth demos are provenance only, not
|
|
live dependencies or geometry authority.
|
|
- The first stereo-depth contract is code-visible:
|
|
`MimirStereoDepthConfigurations` names the libSGM-provenance D3D12 SGM
|
|
profile, and `MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame`
|
|
references caller-declared shader-readable left/right Texture2D resources,
|
|
then lowers rectified stereo depth into a compute-writable R16F disparity
|
|
SurfacePage, R8 confidence Texture2D, a Height FieldEvidence claim, and an
|
|
`AquariumFieldStereoDepthLowering` sidecar tying profile, calibration,
|
|
camera pair, inputs, disparity output, SGM constants, and depth range
|
|
together. `BuildLeapPackedStereoDepthCandidateFrame` is the live Leap cut:
|
|
it recognizes a live `LeapStereoIr` rolling video window, reuses the
|
|
declared packed R8G8 texture as both left and right resource identity,
|
|
names the left/right observations as packed R/G lanes, and emits the same
|
|
depth lowering from the runtime frame. The current libSGM-provenance D3D12
|
|
SGM profile publishes min disparity 0, 128 disparities, four aggregation
|
|
paths, census radius 2, P1 8, and P2 96. Fensalir now has a first
|
|
packed-Leap D3D12 SAD/block-match kernel that writes R16F disparity through
|
|
the typed SurfacePage socket; full SGM and calibrated metric depth remain
|
|
future work.
|
|
- The first Leap point-cloud root is code-visible:
|
|
`MimirPointCloudConfigurations.LeapDisparityPointCloudRoot` records the
|
|
stereo-disparity projection profile and provenance, and
|
|
`MimirFensalirFieldLowering.BuildLeapPackedStereoPointCloudCandidateFrame`
|
|
declares a GPU-resident `FieldMesh` PointList resource derived from the
|
|
Leap disparity SurfacePage. `MimirRuntime` merges that Mesh claim with the
|
|
live Leap stereo-depth claim. `Mimir.BufferSmoke --leap-point-cloud-root-smoke`
|
|
proves one `SurfacePage` packet plus one `Mesh` packet and zero deferred
|
|
requests. Fensalir branch `codex/leap-packed-depth` now owns the matching
|
|
first native surface-claim lane: generated PointList Mesh buffers are
|
|
filled from the disparity SurfacePage on D3D12 for diagnostics, while the
|
|
same compute pass writes approximated normals, curvature, SDF, PBR, and
|
|
radiosity data into Fensalir's splat reservoir buffers. The point-list draw
|
|
path is diagnostic-only; live presentation goes through Fensalir's
|
|
splat/reservoir resolver. The live proof
|
|
`artifacts/runtime/mimir-leap-surface-claims-clean.png` showed LeapUVC
|
|
frames uploaded through a Fensalir-owned D3D12 texture, packed stereo depth
|
|
dispatch-ready/dispatched, 19,200 generated surface claims, point-list
|
|
rendering at `rendered 0`, and visible reservoir-resolved Leap splats.
|
|
Calibrated projection constants, full SGM, reservoir-history startup repair,
|
|
and global residual/calibration ownership remain open.
|
|
- Fensalir packet payload handles are not authority. Live payload references
|
|
must resolve through declared typed GPU/native resources with shape,
|
|
residency, shader access, validity, version, and handle metadata before
|
|
shader lowering can consume them.
|
|
- EVE fullscreen output must use native decode/composite over Fensalir-owned
|
|
program pixels; WebKit/DOM layout is forbidden as pixel authority.
|
|
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
|
|
stems.
|
|
- Local secrets remain untracked.
|
|
- Mimir is a CultMesh app. Codebooks, schedules, calibration receipts, path
|
|
response, and compute-offload work need typed mesh-syncable state surfaces
|
|
so phones, microcontrollers, Raven, and Starfire can cooperate without
|
|
becoming independent timing authorities.
|
|
|
|
accepted_design:
|
|
- `Mimir.slnx` owns the app surface.
|
|
- `src/Mimir.App` launches Fensalir as host/window/D3D12 library.
|
|
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
|
|
`MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`,
|
|
direct native ingest, process bridge ingest, and video capture driver seams.
|
|
- `MimirVideoFrameDescriptor` carries width, height, pixel format, stride,
|
|
device timestamp, and optional native/GPU handle.
|
|
- `MimirEveProgramOutputConfigurations.NativeSharedTexture` records the
|
|
EVE-facing program-output contract: Fensalir publishes
|
|
`Global\MimirFensalirProgramTexture` when
|
|
`FENSALIR_PROGRAM_OUTPUT_D3D12=1`; EVE receives decoded native pixels and
|
|
owns only final composite/input uplink.
|
|
- `src/Mimir.EveRelay` is the current EVE dashboard transport slice:
|
|
it opens the named D3D12 program-output texture, feeds low-latency H.264
|
|
(`h264_nvenc` by default), frames Annex-B access units over `/stream`, and
|
|
reports `/health`. EveCanvas uses `AVSampleBufferDisplayLayer` for native
|
|
decode/composite. The local launch script opens an SSH reverse tunnel to
|
|
Eve when Windows firewall policy blocks direct inbound TCP, and EveCanvas
|
|
tries localhost before the LAN host.
|
|
- `src/Mimir.Broadcast` is the first self-hosted public edge sender:
|
|
Starfire uses FFmpeg/NVENC to push one final H.264/AAC RTMP stream through
|
|
an SSH local forward to Yggdrasil. Yggdrasil owns nginx RTMP ingest and HLS
|
|
segment serving only; it does not transcode or compose. The public viewer
|
|
route is `https://gamecult.org/livestream`, and it reads the existing
|
|
Yggdrasil-routed StreamPixels URL
|
|
`https://streampixels.gamecult.org/mimir/live/hls/mimir.m3u8`. The deployed
|
|
Yggdrasil origin already validates, listens on localhost RTMP
|
|
`127.0.0.1:1935`, and produced HLS segments from an eight-second Starfire
|
|
synthetic NVENC push on 2026-05-30. Dedicated Mimir-site and
|
|
`live.mimir.gamecult.org` viewer/media hostnames are not needed for v1.
|
|
- `MimirAudioBlockDescriptor` carries sample rate, channels, sample format,
|
|
frame count, timestamp, and optional native handle.
|
|
- `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for
|
|
JSON-line video frame and audio block metadata from native probes into
|
|
runtime rolling buffers; it does not carry pixels/audio payloads or own the
|
|
final local-device hot path.
|
|
- `MimirBioacousticTimeline` is the active runtime watermark. It renders a
|
|
low-gain birdsong-like word language with 128 self-identifying word
|
|
positions, left-speaker and right-speaker variants, four formant-rich
|
|
syllables per word, rhythm variation, and direct word identity instead of a
|
|
de Bruijn grammar. The active receiver is a song-contour anchor machine:
|
|
one heard call should expose syllable timing, contour bends, formants,
|
|
payload ornaments, speaker tint, and log-mel evidence as a cluster of
|
|
time/frequency anchors. `MimirRuntime` emits separate panned vocabularies
|
|
per speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as
|
|
`evidence=bioacoustic`; the old chirp-bin path remains a
|
|
calibration/reference artifact with ASIO tooling.
|
|
- The diagnostic audio path can optionally carry base64 float sample payloads
|
|
for sync smoke tests. `MimirChirpletTimeline` owns the older structured
|
|
chirplet reference fingerprint: an order-3 de Bruijn sequence over 32
|
|
time/frequency constellation symbols, plus PCM segment rendering, matched
|
|
timing atoms, symbol/event decoding, and per-band response kernels. Symbol
|
|
identity is carried by start band, glide shape, duration, and following
|
|
inter-chirp gap. Any three consecutive correctly detected symbols identify a
|
|
timeline event within the current operating horizon. `MimirChirpletSymbolCodebook`
|
|
owns the 32 symbol definitions so every symbol has a unique chirp shape, with
|
|
rhythm as additional evidence. `MimirChirpletStreamDecoder` is the first
|
|
constrained chirplet-transform receiver: it owns a bounded PCM window, emits
|
|
transform frames with multiple phase-invariant symbol candidates and refined
|
|
per-candidate sample offsets, decodes code-valid triplet anchors through
|
|
gap/clock coherence, and fits a per-source sample clock from those anchors.
|
|
Its synthetic smoke path is `dotnet run --project
|
|
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`.
|
|
It currently decodes all 15 emitted chirps in a two-second window into 13
|
|
anchors for events 0-12 with a 47999.999990 Hz clock fit and 0.000014 sample
|
|
MAE. `MimirAudioSynchronizationAnalyzer`
|
|
now treats `loopback-scarlett-speakers` as timing authority and derives
|
|
delay only from matched decoded timeline anchors. Positive delay means
|
|
the candidate mic is late relative to loopback. Reports include rounded
|
|
delay, fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub`
|
|
caches the latest per-source reports and `MimirAudioSynchronizationStateTracker`
|
|
owns the smoothed per-source belief: latest delay, smoothed delay,
|
|
confidence, per-band response evidence, and delay-slope/SRO estimate in
|
|
ppm. `MimirRuntime` now runs this analysis online as a bounded rotating
|
|
service on a fixed cadence, while UI and telemetry read cached reports/states
|
|
instead of invoking analysis. `MIMIR_SYNC_TELEMETRY_SECONDS` prints live
|
|
buffer/report/state telemetry. `MimirAudioSynchronizationSettings.Mode`
|
|
now owns the active evidence policy: `chirp-only` emits the calibration
|
|
witness, `passive` stays silent and uses program-audio phase correlation,
|
|
and `hybrid` emits active pilot chunks only while passive confidence is
|
|
weak. `MimirPassiveAudioSynchronizationEstimator` is the first passive path:
|
|
it uses PHAT-weighted cross-spectrum correlation between loopback and mic
|
|
windows before any watermark is needed. Negative passive lags are
|
|
contradictory evidence and carry zero confidence; single-window passive
|
|
confidence is capped below certainty until repeated state earns it.
|
|
`MimirBioacousticTimeline` is now the active calibration witness for both
|
|
chirp-only and hybrid: 128 self-identifying word positions, left/right
|
|
speaker variants, four formant-rich syllables per word, rhythm variation,
|
|
and direct word identity so any correctly decoded word identifies one
|
|
canonical timeline location. The analyzer refines final active delay with constrained local
|
|
waveform correlation around the decoded anchor delay. Each detected motif
|
|
carries per-band response evidence, and the next calibration cut should use
|
|
those profiles to tune contours, formants, usable bands, gain, and rhythm
|
|
weighting. `Mimir.BufferSmoke --bioacoustic-cepstral-smoke` now proves the
|
|
non-brute-force receiver direction: degraded mel-cepstral calls are decoded
|
|
by an augmented MFCC/projection-hash word index before any expensive
|
|
confirmation. `Mimir.BufferSmoke --bioacoustic-train` now runs a
|
|
receipt-backed hypothesis panel over indexed cepstral decoder shapes and
|
|
writes typed CultCache results plus pre-warp/post-warp/reconstructed WAV
|
|
artifacts under `artifacts/bioacoustic-training/`. The first receipt
|
|
`bioacoustic-20260524-103438` showed compact indexing fastest on clean
|
|
audio and exposed timing error under domain warp. The corrected receipt
|
|
`bioacoustic-20260524-150729` adds global clock hypothesis scoring over a
|
|
longer window; compact indexing wins clean throughput, baseline indexing
|
|
remains steadier under degradation, highband indexing has warped-path value,
|
|
and anchor density needs its own metric. `MimirChirpBinTimeline` remains the controlled chirp-bin
|
|
calibration/reference artifact with ASIO tooling and
|
|
`MimirChirpBinCalibrationModel`; keep those captures as response evidence,
|
|
not the active runtime sound. Reports and sync states expose delay in
|
|
microseconds as well as fractional samples. Hybrid emits the bioacoustic
|
|
witness as low-gain half-second bursts while passive confidence is weak,
|
|
never as catch-up backlog. The analyzer no longer applies the passive
|
|
two-second window floor to active fallback; `dotnet run --project
|
|
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 48000`
|
|
recovers a 317.375-sample synthetic delay with printed 0.000 us error using
|
|
`evidence=bioacoustic`, while `--hybrid-sync-self-test --sample-rate 48000`
|
|
correctly accepts strong passive evidence at 2.772 us error. The synthetic
|
|
standalone receiver proof `--standalone-bioacoustic-self-test --sample-rate
|
|
48000 --delay-samples 1269.5` recovers delayed source time below printed
|
|
microsecond precision using only the codebook/schedule. `--bioacoustic-actuator-self-test
|
|
--sample-rate 48000 --delay-samples 317.375` now proves the first
|
|
measurement-to-sample-movement loop: estimate bioacoustic delay, apply
|
|
fractional correction, and remeasure residual below printed microsecond
|
|
precision. `faust/mimir_alignment_actuator.dsp` is the first Faust-owned
|
|
actuator surface for six channels of bounded fractional delay/gain.
|
|
`MimirAlignmentActuatorBank` now converts runtime sync states into explicit
|
|
nonnegative Faust holdback commands with stable source slots, a separate
|
|
reference holdback, and overflow reporting, then queues those commands into
|
|
`AquariumAudioDocument` as engine audio control frames. Mimir also queues
|
|
`faust/mimir_alignment_actuator.dsp` as an `AquariumStreamingDspProgram`
|
|
so Fensalir can compile/host the persistent Faust graph that consumes those
|
|
controls. The same runtime cadence queues sample-bearing
|
|
`AquariumStreamingAudioBlock` lanes from the latest actuator source buffers,
|
|
with channel indices derived from the bank-assigned Faust `sourceN` control
|
|
path. Fensalir now publishes processed DSP output lanes as
|
|
`AquariumAudioStemFrame` records on an engine-owned audio stem bus, and
|
|
Mimir declares deterministic aligned-source stem names with the actuator
|
|
program. Mimir now consumes that bus through `MimirObsStemPublicationState`,
|
|
validating ready/missing/unconfigured lanes against the alignment actuator
|
|
stem-bus profile without copying audio. `MimirObsStemSharedMemoryPublisher`
|
|
writes validated stems into the `Local\MimirObsStemBus` Windows
|
|
shared-memory ABI, and `native/obs_stem_source` is the first OBS source
|
|
plugin bundle. It registers `Mimir Audio Stem`, which reads one named stem
|
|
from that map and submits it as libobs audio, and `Mimir Program Texture`,
|
|
which opens Fensalir's named shared D3D12 program-output texture and draws
|
|
it through an OBS-readable shared D3D11 bridge texture. Fensalir also
|
|
publishes `Global\MimirFensalirProgramFence` as a program-output
|
|
publication fence; the OBS source opens that fence and observes completion
|
|
before copying. The video path is GPU-to-GPU and does not require Spout2 or
|
|
CPU readback; the remaining copy is the explicit D3D12 to libobs-D3D11
|
|
boundary. Fensalir and the OBS source can optionally agree on
|
|
`FENSALIR_PROGRAM_OUTPUT_RING_COUNT>1`; Fensalir publishes
|
|
`Global\MimirFensalirProgramTexture.0`, `.1`, and so on, and OBS chooses the
|
|
latest completed slot from the program-output fence value. OBS can publish
|
|
`Global\MimirObsProgramConsumerFence`; when Fensalir is launched with
|
|
`FENSALIR_PROGRAM_OUTPUT_CONSUMER_FENCE_NAME`, it skips publication instead
|
|
of reusing a ring slot OBS has not acknowledged.
|
|
The rendering migration's program-output receipt phase is structurally
|
|
complete: OBS consumes final Mimir/Fensalir program surfaces and stems
|
|
through the native plugin, raw feeds remain debug inputs, and the ledger
|
|
records the live texture, texture ring, producer-fence, and consumer-fence
|
|
proofs.
|
|
`MimirSynchronizedBufferPlanner` is the first aligned-buffer readout
|
|
primitive: it chooses one canonical presentation time inside the rolling
|
|
window and returns per-stream slices for local cameras, network display
|
|
feeds, and audio blocks. Timing corrections can target a single source or a
|
|
shared `clockDomainId`; the Raven path uses audio evidence routed into
|
|
Scarlett to earn a `raven-sync` correction for Raven display frames.
|
|
Network arrival timestamps remain metadata only.
|
|
`MimirFfmpegRawVideoStreamSource` is the network-pixel adapter for Raven
|
|
ingest: FFmpeg owns SRT receive/decode, while Mimir owns exact raw-video
|
|
frame reads, payload geometry, video descriptors, and rolling-buffer
|
|
samples. `config/mimir-runtime.raven-eve.example.json` listens for
|
|
`raven-display` on port 5200. EveCanvas now owns Eve camera/mic capture
|
|
directly and sends typed frame-events to `Mimir.EveSensorReceiver` on
|
|
WebSocket ports 8793 (`eve-camera`) and 8794 (`eve-mic`).
|
|
Raven can send desktop capture with `scripts/start-raven-screen-capture-sender.ps1`.
|
|
`MimirPresentationControlState` owns program-control intent for Fensalir:
|
|
video feed visibility/solo/opacity/layer order, audio mute/solo/gain, and
|
|
global LUT preset selection. The `Mimir Program` panel filters video
|
|
production surface intents and applies audio mix gain before/inside the
|
|
Faust streaming DSP path. LUT presets are typed postprocess state; exposure
|
|
and bloom map into current Fensalir `GraphicsSettings`, while actual LUT
|
|
texture sampling remains the named renderer hook.
|
|
`Mimir.EveDashboard` is the first native Eve operator dashboard server:
|
|
`/eve/dashboard` sends scene/control snapshots and accepts select, move,
|
|
scale, rotate, visibility, and reset commands. EveCanvas renders the scene
|
|
graph and source panels natively in UIKit with multitouch transform editing.
|
|
This is currently fixture dashboard state for the transport/control proof;
|
|
live binding to `MimirPresentationControlState` and `MimirSceneEditorState`
|
|
is the next authority cut.
|
|
`MimirSceneEditorState` owns the separate Mimir-window editor graph:
|
|
editor camera, scene nodes, selected node, visibility/locks, 2D-plane
|
|
transforms, reset affordances, and grab/rotate/resize gizmo mode. Rolling
|
|
video buffers derive sensor-feed panel nodes, while SDF text and model
|
|
import controls create editor graph nodes. `MimirRuntime` renders derived
|
|
outlines/handles through the Fensalir frame and exposes the `Mimir Editor`
|
|
hierarchy/transform panel. This is not OBS program output. World SDF glyph
|
|
rendering, ASSIMP-style mesh decoding/upload, and pixel-accurate gizmo
|
|
hit-testing remain Fensalir renderer cuts.
|
|
`scripts/build-obs-stem-plugin.ps1` stages the upstream OBS plugin template
|
|
SDK under `artifacts/obs-sdk/` and builds the plugin DLL locally.
|
|
The synthetic
|
|
codebook proof is `dotnet run --project
|
|
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`.
|
|
`Mimir.BufferSmoke --render-chirp-bin-f32` plus
|
|
`native/probes/asio_audio_cadence --play-f32-mono --record-f32-interleaved`
|
|
now provides a real Scarlett ASIO artifact proof for active mode. At 192 kHz,
|
|
Loopback 1 -> Loopback 2 decoded at 0.000 us with 12 anchors and 0.999
|
|
confidence. `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` now renders
|
|
or ingests a calibration capture and persists the response/confusion/delay
|
|
model under `calibration/chirp-bin/`; `--analyze-asio-f32 --calibration`
|
|
loads it back into the chirp-bin reference decoder. Physical inputs still fail accepted
|
|
pairwise timing against loopback in the stored artifact, but decoded mic
|
|
channels now leave calibration models; physical input 1 produced two reliable
|
|
symbols and a different strongest measured band set.
|
|
`native/asio_capture` and `MimirAsioStreamSource` are now the runtime
|
|
Scarlett path: Focusrite ASIO callbacks feed sample-bearing 192 kHz Float32
|
|
blocks into `Mimir.Runtime` in process. `config/mimir-runtime.asio.example.json`
|
|
proves continuous ASIO runtime ingest: a two-second BufferSmoke run at
|
|
192 kHz ingested more than 12,000 sample-bearing blocks across `asio-ch0`
|
|
through `asio-ch3` and retained 2,048 blocks per channel, all from the
|
|
Focusrite ASIO callback stream.
|
|
`Mimir.BufferSmoke --calibrate-contestant-asio-f32` now persists the active
|
|
packet-song physical response model under `calibration/bioacoustic/`,
|
|
including per-channel delay search, polarity, payload reliability,
|
|
response-normalization bands, gain, confidence, and pairwise propagation
|
|
delay. The latest canary-packet Scarlett receipt runs at 10.7x realtime
|
|
across four 192 kHz channels: loopback is 37/37 with 2.524 us MAE,
|
|
co-streamer shotgun is 37/37 payload with 58.785 us MAE, and the cardioid
|
|
is 26/37 with 90.558 us MAE. This is a serious calibration surface, not yet
|
|
true physical microsecond sync.
|
|
A fresh 192 kHz room receipt at
|
|
`calibration/bioacoustic/scarlett-canary-packet-192k-contour-fresh.json`
|
|
keeps loopback at 2.524 us MAE, improves the co-streamer shotgun to 37/37
|
|
payload with 34.083 us MAE, and improves the cardioid to 31/37 payload
|
|
with 92.996 us MAE. The next precision cut is intra-call contour anchors,
|
|
not more sequence machinery.
|
|
`calibration/bioacoustic/scarlett-canary-packet-anchor-rich-192k.json`
|
|
is the first anchor-rich receipt: timing chips, formant pivots,
|
|
harmonic-envelope notches, payload ornaments, renderer template caching, and
|
|
per-event intra-call anchor measurements. It clears 10x realtime and
|
|
improves loopback to about 2.23 us MAE, but physical mics regress/hold at
|
|
tens of microseconds, so anchor survivability and path weighting are now the
|
|
problem.
|
|
Path-level loopback truth now compares candidate mic anchors directly
|
|
against loopback anchors, then refines delay with event-local waveform
|
|
correlation against the captured loopback packet. On the stored anchor-rich
|
|
capture, Release calibration runs at 21.1x realtime and improves physical
|
|
path precision to 7.576 us for the cardioid and 6.578 us for the shotgun.
|
|
A fresh room capture keeps the shotgun near 5.916 us but worsens the
|
|
cardioid to 18.930 us. The one-microsecond target remains open; next cut is
|
|
phase/group-delay correction or a better direct-path anchor family. A naive
|
|
recursive waveform phase-lock pass was tested and reverted because it could
|
|
improve one cardioid receipt while worsening the shotgun and locking a fresh
|
|
cardioid run onto a later reflection lobe. Recursion is only allowed back as
|
|
a constrained direct-path/group-delay model, not as unconstrained waveform
|
|
resampling with prettier decimals. The sonar/DSP research note
|
|
`research/sonar-dsp-recursive-refinement-2026-05-25.md` points the next
|
|
decoder cut at complex contour matched filters, phase-slope/group-delay
|
|
fitting, acquisition/tracking loop bandwidths, and sparse multipath
|
|
residuals. Its source ledger and local mirrors live under
|
|
`research/sonar-dsp-recursive-refinement-2026-05-25/`.
|
|
`MimirComplexContourMatchedFilterBank` and `MimirDirectPathTracker` are the
|
|
first implementation slice of that shape: complex anchor responses keep
|
|
multiple lobes, the current path prediction gates direct-path ownership, and
|
|
later coherent clusters are reported as reflection taps. Synthetic 192 kHz
|
|
reflection smoke currently lands about 5.249 us from expected delay; stored
|
|
Scarlett shotgun receipts land within about 5.860 us and 7.247 us of seeded
|
|
path fits, while cardioid receipts land within about 15.817 us and 8.963 us.
|
|
Low confidence is expected until phase/group-delay surfaces are learned. The
|
|
tracker now exposes per-band delay/phase residual observations and a
|
|
`MimirDirectPathChannelModel` correction surface for multi-window
|
|
calibration; one-window self-correction is not live authority. The tracker
|
|
now consumes explicit channel models as phase/group-delay correction and
|
|
downweights bands outside the learned usable surface. `Mimir.BufferSmoke
|
|
--learn-complex-contour-channel-model` persists
|
|
`calibration/bioacoustic/complex-contour-channel-model.json` from the
|
|
replay panel, and `--evaluate-complex-contour-channel-model` persists
|
|
`calibration/bioacoustic/complex-contour-channel-model-evaluation.json`.
|
|
Current evaluation improves 3/4 absolute path-seed errors and lowers mean
|
|
direct-cluster MAE; the fresh cardioid case reaches about 0.202 us from
|
|
seed, while the stored cardioid case still worsens to about 17.476 us. The
|
|
model is explicit calibration evidence, not default runtime authority.
|
|
The complex contour receiver is now wired into runtime as an explicit live
|
|
lane: when `enableComplexContourRuntime` is true, Mimir emits the configured
|
|
`bioacousticWitnessProfileId` through `MimirBioacousticContestantRenderer`,
|
|
loads `complexContourChannelModelPath`, extracts Float32 windows from the
|
|
rolling buffers, and publishes `evidence=complex-contour` reports. The
|
|
synthetic runtime-shaped proof `--complex-contour-runtime-self-test` at
|
|
192 kHz recovers a 693.5-sample delay with about 0.219 us error.
|
|
A fresh live Scarlett proof is preserved at
|
|
`calibration/bioacoustic/complex-contour-live-20260525-063229.md`:
|
|
Focusrite ASIO playback/capture measured Loopback 1 -> Loopback 2 at
|
|
-0.014 us with corrected confidence 0.971. The apparent `asio-ch0`
|
|
near-seed/cardioid estimate is invalidated: signal inspection shows flat
|
|
low-level noise with zero-lag loopback correlation 0.000032, so the tracker
|
|
fit noise under seeded timing pressure. `asio-ch1` remains only a weak,
|
|
non-authoritative input witness. The next cut is signal-presence gating,
|
|
independent path truth, and better frequency/direct-path discrimination.
|
|
`calibration/bioacoustic/complex-contour-live-20260528-002313.md` keeps
|
|
the loopback proof healthy at -0.062 us with 0.956 confidence but finds
|
|
zero contour hits on both physical inputs, so the current physical mic
|
|
proof is blocked on input signal/gain/routing/source placement before
|
|
receiver logic changes.
|
|
After the mics were armed, `complex-contour-live-20260528-080604.md`
|
|
proved both physical inputs were alive but was routed to headphones by
|
|
mistake. The monitor-routed
|
|
`complex-contour-live-20260528-080727.md` is the first credible fresh
|
|
physical contour proof: loopback is -0.014 us at 0.971 confidence,
|
|
`asio-ch0` lands -0.410 us from its stored path seed at 0.524 confidence,
|
|
and `asio-ch1` lands -2.467 us at 0.363 confidence. Physical mapping:
|
|
`asio-ch0` is the shotgun, about 1.5 m from the right monitor and 3.0 m
|
|
from the left; `asio-ch1` is the cardioid, about 0.5 m from the left
|
|
monitor and 1.5 m from the right. Repeat captures are needed before
|
|
channel-model updates or actuator authority.
|
|
`complex-contour-monitor-split-20260528-081813.md` added ASIO
|
|
`--play-output-channel` and captured output 0/output 1 separately.
|
|
Loopback channels prove the ASIO output buffers are distinct, but
|
|
physical mic delays differ by only about 20.885 us for the shotgun and
|
|
24.789 us for the cardioid across outputs. This does not match the
|
|
declared monitor geometry; verify downstream monitor routing/isolation
|
|
before learning left/right acoustic paths.
|
|
`calibration/bioacoustic/complex-contour-replay-panel.json` persists the
|
|
current four-case replay receipt.
|
|
Acoustic robustness remains open.
|
|
The old arbitrary-chirplet matcher is a diagnostic/reference artifact only,
|
|
not the runtime hot path.
|
|
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
|
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
|
|
texture capture.
|
|
- `native/reservoir` owns the lower shared-edge rolling-buffer invariant for
|
|
Fensalir/Faust integration.
|
|
- `MimirFensalirFieldLowering` now emits `AquariumFieldResourceDeclaration`
|
|
rows for live payload views. Observation claims reference
|
|
`mimir:resource:*` keys, and Fensalir validation/planning can reject or
|
|
defer packets whose resource declaration is missing, CPU-only, duplicated,
|
|
or backend-incompatible. Audio spectrum/waveform surface intents now use
|
|
the source observation resource key and plan as `TubeField` packets when
|
|
the resource is compatible. `Mimir.BufferSmoke
|
|
--fensalir-field-evidence-smoke` proves one declared resource, three
|
|
claims, one planned resource-backed `TubeField` packet, two deferred
|
|
non-backend claims, and no validation errors.
|
|
`--fensalir-field-dsl-resource-smoke` proves Fensalir's field evidence DSL
|
|
can bind a declared resource directly and produce one planned `TubeField`
|
|
packet with no deferred requests. `MimirFensalirFieldLowering` also owns
|
|
the current camera observation bridge: latest video buffers lower into
|
|
FieldEvidence camera claims, GPU/native video handles declare shared
|
|
Texture2D resources, and metadata-only cadence frames do not create camera
|
|
surface intents or fake payload requests. The old direct
|
|
`AquariumGpuSensorFrame` builder is no longer the active proof path;
|
|
Fensalir can now resolve imported shared Texture2D resources by native D3D12
|
|
handle and accepts Mimir video format names such as Bgra8, Gray8, Rg8,
|
|
Nv12, and LeapStereoIr. The preferred live camera path is now
|
|
Fensalir-owned texture leasing: Mimir receives `AquariumRuntimeServices`,
|
|
asks the engine field resource broker for a keyed Texture2D lease, writes
|
|
decoded frames into the returned shared D3D12 texture, signals the returned
|
|
producer fence, and commits that fence value before the resource key reaches
|
|
shader sampling. `MimirVideoCaptureDriverSource` forwards that lease client
|
|
to direct camera drivers that implement `IMimirFensalirTextureLeaseReceiver`.
|
|
Camera backends must use the closest-to-device path available and report
|
|
unavoidable copies; process or managed convenience layers are diagnostics,
|
|
not production camera hot-path owners.
|
|
Raw single-plane system-memory frames use the explicit fallback upload lane:
|
|
`MimirVideoCaptureDriverSource` uploads the bytes into the Fensalir lease,
|
|
drops the managed payload from the live sample, and increments
|
|
`UnavoidableCopyCount`. NV12 CPU upload is rejected until Fensalir has a
|
|
real planar upload path; device/GPU NV12 producers should write the leased
|
|
texture directly.
|
|
`MimirKsVideoCaptureDriver` plus
|
|
`native/camera_capture/mimir_camera_capture.dll` is the first
|
|
production-shaped KS/UVC direct camera path: it opens a Kernel Streaming pin
|
|
in process, queues uncompressed UVC frames, and feeds the same driver source
|
|
and Fensalir lease upload path. MJPG/H264 remain future device/GPU decode
|
|
producer work.
|
|
`MimirPs3EyeVideoCaptureDriver` plus
|
|
`native/camera_capture/mimir_ps3eye_capture.dll` owns the equivalent raw
|
|
WinUSB/libusb PS3 Eye path and emits Bayer8 frames through the same upload
|
|
and copy-count machinery.
|
|
Fresh direct-driver smokes uploaded real frames for LeapUVC 640x240 YUY2,
|
|
PS3 Eye 0/1 320x240 Bayer8, Kiyo Pro 1920x1080 YUY2, and regular Kiyo
|
|
640x480 YUY2. Regular Kiyo 1280x720 YUY2 did not open; compressed
|
|
Kiyo Pro MJPG/H264 now have a first GPU-resident producer:
|
|
`MimirMediaFoundationGpuVideoCaptureDriver` uses
|
|
`native/camera_capture/mimir_mf_gpu_capture.dll` to run Media Foundation
|
|
SourceReader with a D3D11 device manager, decode MJPG/H264 on the GPU, copy
|
|
the decoded GPU surface into a shared BGRA D3D11 texture, and publish a
|
|
`shared-d3d11-texture` handle with zero live CPU payload bytes. The MF GPU
|
|
producer now also creates a shared D3D11 fence, signals it after the
|
|
texture copy, and carries the fence handle/value through
|
|
`MimirVideoFrameDescriptor`, `MimirBridgePayloadView`, and the Fensalir
|
|
field resource declaration so D3D12 can wait before sampling. Kiyo Pro
|
|
1920x1080 MJPG->RGB32 and H264->RGB32 smokes passed with nonzero shared
|
|
texture and producer fence handles. Direct shared NV12
|
|
texture creation failed in D3D11, so NV12 remains future plane-aware interop
|
|
or GPU bridge-copy work.
|
|
`Mimir.BufferSmoke --fensalir-camera-observation-smoke`
|
|
verifies the split, and `--fensalir-texture-lease-smoke` verifies the
|
|
engine-owned lease path. Camera image claims stay debug-view-only for now;
|
|
production visual fusion/feature extraction lowerings are deferred until
|
|
the renderer and audio work are fixed. Fensalir now owns the first in-process
|
|
D3D12 field resource registry: declared shared structured/curve buffers
|
|
import/alias GPU-resident resources by handle, while Fensalir-owned
|
|
resources allocate GPU slots only when Fensalir is the producer.
|
|
Mimir-declared rendering buffers are therefore Fensalir GPU resources in
|
|
the same runtime, not cross-process payload truth and not CPU payloads to
|
|
copy back and forth. The TubeField DSL can now express a 2D rolling float
|
|
buffer as Catmull-Rom XY tubes with modulo column addressing, amplitude
|
|
power/normalization, radius, ramp texture path, and emission scale.
|
|
Fensalir now has the first D3D12 TubeField compute/render lowering: compute
|
|
binds the declared rolling sample buffer as read-only GPU memory, expands
|
|
endpoint-clamped Catmull-Rom tube envelopes into fixed-budget vertex/index
|
|
UAVs, honors declared float sample stride with byte-address loads, records
|
|
one draw batch per lowering, then the TubeField pixel shader binds the
|
|
matching source buffer and resamples material per pixel by column+spline
|
|
coordinate with the cheap 5-tap approximation. Texture2D local assets are
|
|
now first-class Fensalir field resources: the DSL can declare a local PNG
|
|
ramp resource, TubeField `ramp=` aliases bind automatically by resource
|
|
key, and D3D12 uploads/binds the texture SRV with a linear-clamp sampler.
|
|
Missing ramps fall back to a Fensalir-owned 0-1 linear texture. The first
|
|
material is opaque field geometry with squared normalized emission, fake
|
|
Fresnel center-bright/edge-dark shaping, and geometry-budget overflow
|
|
reporting. TubeField compute now emits GPU-resident D3D12 indirect draw
|
|
argument packets, and render executes those packets after binding each
|
|
batch's source/ramp material state. SurfacePage resources now have the first Fensalir
|
|
resolver: engine users can declare GPU-resident 2D pages with dimensions,
|
|
format, version, and optional local source, and Fensalir allocates/binds
|
|
them as shader-readable page textures for future Height/SDF/material
|
|
lowerings. VolumeTexture resources can now be declared and resolved as
|
|
GPU-resident shader-readable 3D textures for future density/extinction/SDF3D
|
|
lowerings. Mesh resources now package friendly vertex/index ownership:
|
|
`Mesh.Vertices` and `Mesh.Indices` describe the GPU buffers, plus topology,
|
|
index format, bounds, submesh count, and version. The Fensalir DSL can
|
|
declare those mesh packages, and the D3D12 registry allocates/reuses the
|
|
backing vertex/index buffers under the mesh resource key. The DSL can also
|
|
create generic resource-backed claims, so mesh/page/volume resources can
|
|
reach backend packet planning. Mesh draw/material lowerings remain future
|
|
engine work. Mesh layout authority is split: imported/user-facing meshes use
|
|
the standard `PositionNormalUvColor` vertex layout, while generated meshes
|
|
can be `PipelinePrivate` and let the lowering own the minimal bytes.
|
|
TubeField now renders through the first explicit pipeline-private generated
|
|
mesh binding: compute owns the private vertex/index/indirect layout, and
|
|
render binds that generated mesh before applying TubeField material state.
|
|
TubeField writes stable field ids, real tube normals, coverage/confidence,
|
|
and domain-validity guide data into the scene metadata/control/reservoir
|
|
guide targets consumed by reservoir resolve for spatiotemporal
|
|
reconstruction. The shared field candidate inlet now keeps four fixed slots
|
|
per pixel, so producer-specific paths compete inside a resolver-owned budget
|
|
instead of choosing private winners. Reservoir history update is now a
|
|
shared Fensalir compute pass over a ping-ponged structured buffer with the
|
|
same four rows per pixel; it emits the resolved HDR field texture that
|
|
bloom and presentation consume. The presentation shader is no longer a
|
|
hidden history/TAA owner. TubeField reservoir input has moved from
|
|
generated segment packets to rolling-buffer column packets: expansion emits
|
|
one GPU-resident column record per logical history/source column, tile
|
|
binning admits those columns, and ReSTIR candidate evaluation samples the
|
|
Catmull-Rom tube SDF/material directly from the source buffer. Generated
|
|
mesh draw args are zeroed for the live path and remain only a diagnostic
|
|
reference.
|
|
The generated-mesh DrawIndexed indirect command signature is shared ABI now;
|
|
TubeField only owns the argument buffer it emits. TubeField expansion is now
|
|
gated by the selected backend packet plan, so lowering metadata cannot
|
|
bypass validation/planning authority. Validation also rejects TubeSpline
|
|
lowering metadata if its claim is not Tube-encoded or if its resource does
|
|
not match the claim payload. Mimir's typed surface-intent lowering now emits
|
|
TubeSpline lowering metadata for audio spectrum/waveform Tube claims, so the
|
|
generated mesh path is fed by real Mimir evidence rather than only the DSL
|
|
smoke. The legacy direct `AquariumSplineFrame` spectrum dashboard is silent:
|
|
`MimirRuntime` submits an empty spline frame and the old direct spectrum
|
|
spline builder has been deleted. Live spectrum samples are now normalized
|
|
into a row-major Float32 matrix, declared as
|
|
`mimir:resource:spectrum:field-upload`, uploaded into a Fensalir-owned
|
|
GPU-resident structured buffer, and consumed by the TubeField lowering.
|
|
`Mimir.BufferSmoke --mimir-spectrum-upload-smoke` verifies the live
|
|
`MimirRuntime` frame path emits that Float32 upload and planned TubeField
|
|
packet while direct spline and buffer-field dashboard inputs remain empty.
|
|
The runtime spectrum path also declares the local blackbody PNG as a
|
|
GPU-resident Texture2D field resource and binds the TubeField material by
|
|
`RampResourceKey`.
|
|
Runtime spectrum columns now represent flattened `(history age, source
|
|
lane)` pairs. Mimir emits one TubeField claim/lowering per active source.
|
|
The spectrum resource advertises fixed history and source-lane capacities
|
|
so frame content versions and source topology changes do not resize the GPU
|
|
buffer while the rolling trail fills. `MIMIR_SPECTRUM_SOURCE_LANES` sets
|
|
the lane capacity; surplus sources are truncated and reported in the
|
|
runtime UI. Lowerings use that capacity as `ColumnStride` through the
|
|
shared Float32 resource so each active lane renders a true age trail at
|
|
`z = age * 0.1`. Physical history slots are addressed as a ring with
|
|
`RollingOffset`, so logical age no longer requires moving every column in
|
|
the upload matrix before shader sampling. Mimir emits one Float32 resource
|
|
upload for the newest spectrum ring slot with an explicit element offset
|
|
instead of uploading the full fixed-capacity backing buffer every frame.
|
|
Fensalir owns rolling-slot validity and clamps TubeField dispatch so invalid
|
|
older slots are not sampled after renderer allocation, reset, or partial
|
|
update.
|
|
`MIMIR_SPECTRUM_TUBE_SUBDIVISIONS` controls the Mimir-requested Catmull-Rom
|
|
subdivisions so Fensalir's TubeField truncation report has a direct cost
|
|
lever. Mimir's active proof path no longer uses the direct
|
|
`AquariumAcousticFieldFrame` builder: sync states lower into FieldEvidence
|
|
calibration constraints. Fensalir now selects AudioPath Phase/Confidence
|
|
claims as DebugOverlay backend packets, so acoustic calibration evidence is
|
|
planned and inspectable through the FieldEvidence authority. Acoustic source
|
|
candidates and confidence volumes remain Phase 5 fusion work. The first
|
|
synthetic acoustic source candidate proof now lowers an SRP/PHAT-style
|
|
localization result with calibration/source identity and a world-space
|
|
confidence envelope, and Fensalir plans it as a FieldEvidence DebugOverlay
|
|
packet. Live room geometry still needs an explicit measured coordinate
|
|
frame before those candidates become runtime truth. Phase 5's synthetic
|
|
marker proof now lowers a deterministic multi-camera marker candidate with
|
|
calibration/marker identity and world-space support, and Fensalir plans it
|
|
as DebugOverlay evidence while ambiguous raw camera features still defer.
|
|
- `config/perfect-machine.example.json` declares live owners, sensors,
|
|
reservoir views, and runtime policy.
|
|
- `src/Mimir.Runtime/Synchronization` now includes the Perfect Machine module
|
|
library distilled from the research notes: node profiles, decoder/language
|
|
profiles, path learning, benchmark panels, actuator strategies, capture and
|
|
ingest strategies, reservoir strategies, field profiles, compute offload,
|
|
OBS publication, CultMesh contracts, Fensalir lowering, and exportable
|
|
manifest generation.
|
|
- `docs/perfect-machine-module-library.md` indexes those reusable runtime
|
|
modules and their smoke commands.
|
|
- `docs/code-algorithm-map.md` is the source-level ownership and algorithm map
|
|
for the current repo code surface.
|
|
- `docs/perfect-machine-domain-index.md` indexes Mimir's full problem domain,
|
|
current evidence taxonomy, hot paths, and implementation cuts.
|
|
- `research/perfect-machine-study-2026-05-23/` preserves the architecture
|
|
rumination, optimization ledger, research references, source/hotspot audits,
|
|
Fensalir/native boundary maps, benchmark plan, calibration-session spec,
|
|
distributed receiver spec, failure-mode ledger, data dictionary, low-level
|
|
implementation notes, and sample code sketches for streaming chirp-bin
|
|
decode, AVX2/GPU scoring, batched native ABI, fractional-delay/SRO
|
|
actuation, acoustic source localization, Mimir-to-Fensalir lowering, and
|
|
native capture rings. The corrected deep-study pass ran from
|
|
`2026-05-23T23:54:42.9041588+01:00` through at least
|
|
`2026-05-24T01:54:56.9747630+01:00`; use
|
|
`research/perfect-machine-study-2026-05-23/reading-guide.md` as the
|
|
re-entry index.
|
|
- `.voidbot` state is part of repo memory and must be committed separately
|
|
whenever dirty.
|
|
|
|
active_subgoals:
|
|
- Verify the live Leap point cloud in the Mimir editor.
|
|
- Add calibrated projection constants and a global residual/calibration owner
|
|
that can consume Leap point-cloud errors before PS3 Eyes, Kiyos, and Eve
|
|
feed wider online calibration.
|
|
- Add remaining local camera drivers through `IMimirVideoCaptureDriver`.
|
|
- Use the verified in-process Focusrite ASIO source for local Scarlett mic and
|
|
loopback ingest; WASAPI remains a diagnostic witness.
|
|
- Add network audio feed producers that enter the same runtime clock/rebuffer
|
|
machinery instead of becoming independent authorities.
|
|
- Bind Fensalir UI to stream health, buffer depth, source timestamps,
|
|
settings, and output management.
|
|
- Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
|
|
replaces them.
|
|
|
|
current_status:
|
|
summary: >
|
|
The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS
|
|
bridge scripts, and persistence surfaces. Script-era packages, tests,
|
|
helper tools, dependency manifests, and stale docs have been removed from
|
|
the working tree. Research archives remain as reference-only memory for raw
|
|
native/GPU implementation shape. A raw PS3 Eye libusb/WinUSB cadence probe
|
|
measured both PS3 Eyes simultaneously at about 183 fps each at 320x240 and
|
|
57-59 fps each at 640x480. A native Kernel Streaming LeapUVC probe can pull
|
|
frames through usbvideo.sys without Media Foundation, DirectShow, OpenCV, or
|
|
frame processing; after discarding a two-second warm-up, the useful 640x240
|
|
stereo IR mode advertises 115 fps and currently measures about 110.8 fps
|
|
with eight queued asynchronous KS reads. The half-height 640x120 mode
|
|
advertises 214 fps and currently measures about 205.9 fps. Plain Kiyo is
|
|
usable through KS when manual exposure is forced shorter: 1080p MJPG reaches
|
|
about 30 fps and 720p MJPG reaches about 51.5 fps. Kiyo Pro advertises 60
|
|
fps MJPG at 720p/1080p but currently delivers only about 25 fps through the
|
|
same KS path even with manual exposure tuning. Standard KS low-light
|
|
compensation off does not restore 60 fps, powerline 60Hz only raises Pro
|
|
cadence to about 29 fps, and standard extended Video HDR is not exposed on
|
|
this KS path. The controls-only probe now parses USB VideoControl
|
|
descriptors and found the Kiyo Pro's two vendor extension units. One
|
|
extension unit answers writable KS selectors. Moving the Pro directly to a
|
|
motherboard USB3 port changed the path to `root_hub30`, `bcdUSB=0x0320`,
|
|
and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still
|
|
reports `UsbHighSpeed` rather than `UsbSuperSpeed`, and baseline cadence is
|
|
still about 25 fps across every tested format. The first simultaneous local
|
|
camera pull only had three visible devices: LeapUVC, Kiyo Pro, and one PS3
|
|
Eye. The KS multi-stream harness plus raw PS3 Eye probe pulled all three at
|
|
once, but Leap cadence collapsed under shared load: with PS3 Eye at
|
|
640x480@60 the set delivered Leap 40.12 fps, Kiyo Pro 24.99 fps, and PS3
|
|
Eye 56.06 fps; with PS3 Eye at 320x240@187 it delivered Leap 81.10 fps,
|
|
Kiyo Pro 25.33 fps, and PS3 Eye 185.74 fps. After port swapping, five
|
|
camera devices / six optical sensors were visible and pulled at once:
|
|
LeapUVC's packed stereo IR pair, Kiyo Pro, regular Kiyo, and two PS3 Eyes.
|
|
With both Eyes at 640x480@60, the set delivered Leap stereo 57.33 fps,
|
|
Kiyo Pro 24.97 fps, Kiyo 30.30 fps, PS3 Eye 0 59.59 fps, and PS3 Eye 1
|
|
57.60 fps. With both Eyes at 320x240@187, it delivered Leap stereo 81.05
|
|
fps, Kiyo Pro 24.97 fps, Kiyo 30.28 fps, PS3 Eye 0 187.60 fps, and PS3
|
|
Eye 1 187.03 fps. Kiyo Pro still negotiates high-speed USB, and Leap
|
|
remains topology-sensitive under shared load. A follow-up moved Leap alone
|
|
onto the blue rear USB3 group/root_hub30#5 while both Kiyos shared
|
|
root_hub30#7&18eea21b and one visible PS3 Eye used root_hub30#7&124f193f;
|
|
Leap still delivered only 80.99 fps with the other visible cameras active,
|
|
so the remaining loss is not explained by same-root-hub contention alone.
|
|
Splitting Kiyo Pro and one PS3 Eye to Raven left the Mimir host with
|
|
LeapUVC, regular Kiyo, and one PS3 Eye, but local cadence still stayed low:
|
|
Leap delivered 81.34 fps with Kiyo+Eye, 82.77 fps with Kiyo only, and only
|
|
84.18 fps in a same-session Leap-only baseline. The previous ~110.8 fps
|
|
Leap result was not reproduced until the Focusrite Scarlett Solo was moved
|
|
away from the Leap USB neighborhood. After that, Leap-only baseline rose to
|
|
95.48 fps, best full-height scenario to 101.67 fps, and the local split set
|
|
delivered Leap 105.10 fps, regular Kiyo 30.17 fps, and one PS3 Eye 185.81
|
|
fps. The split-host layout is viable if the Leap's USB neighborhood stays
|
|
free of the Focusrite and other heavy devices. Raven's PS3 Eye raw interface
|
|
now has the same libwdi WinUSB binding as the local Eye and can be opened by
|
|
the copied cadence probe, but Raven Eye cadence is currently unusable:
|
|
320x240@187 delivered 132.24 fps then 78.93 fps, and 640x480@60 delivered
|
|
6.32 fps with multi-second gaps. Moving that Eye back to the Mimir host
|
|
produced the current clean local topology: LeapUVC, regular Kiyo, and two
|
|
PS3 Eyes run in parallel while Kiyo Pro stays off-host. With both Eyes at
|
|
320x240@187, Leap delivered 106.07 fps, Kiyo 30.33 fps, and the Eyes 186.57
|
|
/ 186.14 fps. With both Eyes at 640x480@60, Leap delivered 108.07 fps, Kiyo
|
|
30.34 fps, and the Eyes 57.79 / 58.08 fps. Moving Kiyo Pro back to the
|
|
Mimir host while regular Kiyo was absent kept Leap healthy: with both Eyes
|
|
at 320x240@187, Leap delivered 110.80 fps, Kiyo Pro 24.97 fps, and the Eyes
|
|
187.46 / 186.89 fps; with both Eyes at 640x480@60, Leap delivered 110.72
|
|
fps, Kiyo Pro 24.87 fps, and the Eyes 59.91 / 59.19 fps. Kiyo Pro still
|
|
negotiates UsbHighSpeed(2), but it does not poison Leap cadence in the
|
|
current layout. Moving regular Kiyo from front panel to a rear port made
|
|
all five camera devices / six optical sensors visible locally. With both
|
|
Eyes at 320x240@187, the full set delivered Leap 110.84 fps, Kiyo Pro 25.08
|
|
fps, Kiyo 30.18 fps, PS3 Eye 0 187.59 fps, and PS3 Eye 1 187.02 fps. With
|
|
both Eyes at 640x480@60, it delivered Leap 110.76 fps, Kiyo Pro 25.14 fps,
|
|
Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
|
|
the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
|
|
next_action: >
|
|
Continue the Leap-first visual geometry cut from the verified surface-claim
|
|
path. Mimir lowers live `LeapStereoIr` packed R/G stereo into a disparity
|
|
SurfacePage and declares a derived point-cloud Mesh socket. Fensalir now
|
|
generates Leap surface splats from that disparity, feeds them into the
|
|
native splat/reservoir resolver, and presents visible Leap-derived points
|
|
without the diagnostic point-list draw path. Next: replace provisional
|
|
projection display fit with calibrated Leap intrinsics/extrinsics, repair
|
|
reservoir-history startup so the temporal pass can stay enabled, then add a
|
|
global residual/calibration owner that can consume point-cloud alignment
|
|
error. Do not fan RGB cameras into fake depth: PS3 Eyes are fast
|
|
motion/feature witnesses, Kiyos and Eve are RGB/context witnesses, and Leap
|
|
owns the first dense close-range geometry. Keep the Focusrite away from the
|
|
Leap USB neighborhood. Treat Kiyo Pro as a ~25 fps source until its
|
|
SuperSpeed negotiation is fixed.
|