1195 lines
83 KiB
YAML
1195 lines
83 KiB
YAML
workspace:
|
|
name: Mimir
|
|
status: active
|
|
focus: direct-driver-fensalir-stream-app
|
|
implementation_name: Mimir public surface / native runtime contracts
|
|
premise: >
|
|
Mimir is the local realtime field machine: cameras, microphones, speakers,
|
|
network feeds, and timing signals become one coherent OBS-facing volumetric
|
|
program surface. The live path is a C# Fensalir-hosted app with in-memory
|
|
rolling buffers and native capture/runtime boundaries.
|
|
|
|
objective:
|
|
primary: >
|
|
Build one synchronized spatial stream machine from six cameras, six
|
|
microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP,
|
|
and OBS output.
|
|
current: >
|
|
Build the Mimir editor as a 2D OBS-shaped program mixer first: synced video
|
|
streams become compositable program layers, audio streams remain mixable
|
|
stems, and telemetry stays on Odin's Nightwing TUI/CultMesh surface instead
|
|
of stealing editor space. Leap/3D calibration work remains important, but
|
|
it is not the live editor authority for this pass.
|
|
|
|
constraints:
|
|
- Sender is Windows.
|
|
- Receiver uses OBS.
|
|
- Video encode must use NVENC when the bridge path is active.
|
|
- Audio sources must remain separately controllable in OBS.
|
|
- V1 bridge is LAN-only.
|
|
- Persistent state must clarify rehydrate, not become transcript tax.
|
|
- Live ingest belongs to Mimir.Runtime, native drivers, Fensalir, and
|
|
Faust/native DSP.
|
|
- Edge JSON is schema/diagnostic only; live process data is typed or in
|
|
memory.
|
|
- Networked feeds are producers into the live runtime, not independent clock
|
|
authorities.
|
|
- Mimir explicitly trades latency and memory for throughput and field quality:
|
|
the default five-second buffer is compute budget for stream alignment,
|
|
GPU/audio analysis, and volumetric sound-field extraction.
|
|
|
|
invariants:
|
|
- OBS composition authority stays in OBS.
|
|
- FFmpeg owns capture, encoding, and transport only for bridge utilities.
|
|
- One stable bridge port equals one OBS source endpoint.
|
|
- One rolling live window owns the timing edge and expiry.
|
|
- The rolling window is not accidental latency. It must preserve enough time
|
|
to line up feeds and crunch data before OBS-facing output is emitted.
|
|
- Typed views may index the rolling buffer, but no per-kind ring owns
|
|
retention.
|
|
- Fensalir owns production UI, GPU fusion, D3D12 interop, Spout2 output, and
|
|
the EVE shared D3D12 program-output texture.
|
|
- Mimir publishes physical observations, calibration constraints, and surface
|
|
intent. Fensalir owns field claims, selected lowerings, evidence reservoirs,
|
|
temporal guide lanes, and program presentation.
|
|
- The Mimir editor is currently a 2D program mixer. `MimirSceneEditorState`
|
|
owns normalized video-layer placement for synced video streams and emits
|
|
only video-source preview items; camera/SDF/model authoring has no live UI
|
|
authority, and editor camera/SDF/spline output must not contribute to the
|
|
runtime frame. The center preview requests a Fensalir-owned blit of the
|
|
current output buffer, then draws those normalized placement items on top;
|
|
the preview observes the program surface without becoming composition
|
|
authority. Placement commits keep layers inside the frame, snap near
|
|
edges/center, and preserve source aspect while resizing through sliders or
|
|
the scale gizmo. The center preview now has direct preview
|
|
body/corner-handle gestures wired to the same placement owner through an
|
|
Eve-style retained dashboard document; sliders are fallback/precision
|
|
controls, not the primary compositor surface. Explicit
|
|
source/transform controls are inspector-only and visible only while a
|
|
program source is selected. Empty-canvas clicks clear source selection;
|
|
hover/active handle state and snap guides are supplied through the shared
|
|
document state lowered by Fensalir's Direct2D/D3D12 paths; Eve is the
|
|
portable dashboard authority and CultUI is only the original C# reference
|
|
API shape. Dashboard fields now carry explicit typed CultCache/CultMesh
|
|
bindings with document schema/id, field path, value kind, access,
|
|
authority, and optional command id; local callbacks are native lowering
|
|
actuators while Verse-backed CultCache fields are the portable state
|
|
contract. Keyboard arrows nudge selected layers without
|
|
snapping, PageUp/PageDown reorder layers, and the inspector owns the
|
|
selected source contain/cover/stretch fit mode.
|
|
- The right rail `Output` panel owns OBS-like program actions below the
|
|
inspector. `MimirProgramRecorder` owns `Record` / `Stop`, status text,
|
|
timestamped receipts under `C:\Users\Meta\Videos\Mimir` by default, and
|
|
process-tree shutdown. Its current actuator delegates to
|
|
`scripts/record-stream-proof-output.ps1`, so the app records the same Raven
|
|
+ Kiyo PIP + Starfire Scarlett NVENC/AAC composite intended for
|
|
Yggdrasil/public judgment. `MIMIR_RECORD_OUTPUT_DIR` redirects receipts and
|
|
`MIMIR_RECORD_START_RAVEN=1` asks the app to launch Raven's sender too.
|
|
`MimirProgramStreamer` owns `Live` / `Stop` for the `Mimir.Broadcast` RTMP
|
|
path and requires `MIMIR_STREAM_INPUT` unless
|
|
`MIMIR_STREAM_ALLOW_TEST_PATTERN=1` explicitly authorizes a synthetic
|
|
smoke. Native program-output recording/streaming should replace those
|
|
actuators behind the same owner.
|
|
- Fast stereo depth belongs to Fensalir's D3D12 compute/resource authority.
|
|
`research/d3d12-stereo-depth-provenance-2026-05-29.md` records libSGM as
|
|
the permissive north-star reference for a future SGM-shaped stereo
|
|
disparity lane over synchronized rectified texture pairs and calibration
|
|
state. CUDA, TensorRT, and monocular depth demos are provenance only, not
|
|
live dependencies or geometry authority.
|
|
- The first stereo-depth contract is code-visible:
|
|
`MimirStereoDepthConfigurations` names the libSGM-provenance D3D12 SGM
|
|
profile, and `MimirFensalirFieldLowering.BuildStereoDepthCandidateFrame`
|
|
references caller-declared shader-readable left/right Texture2D resources,
|
|
then lowers rectified stereo depth into a compute-writable R16F disparity
|
|
SurfacePage, R8 confidence Texture2D, a Height FieldEvidence claim, and an
|
|
`AquariumFieldStereoDepthLowering` sidecar tying profile, calibration,
|
|
camera pair, inputs, disparity output, SGM constants, and depth range
|
|
together. `BuildLeapPackedStereoDepthCandidateFrame` is the live Leap cut:
|
|
it recognizes a live `LeapStereoIr` rolling video window, reuses the
|
|
declared packed R8G8 texture as both left and right resource identity,
|
|
names the left/right observations as packed R/G lanes, and emits the same
|
|
depth lowering from the runtime frame. The current libSGM-provenance D3D12
|
|
SGM profile publishes min disparity 0, 128 disparities, four aggregation
|
|
paths, census radius 2, P1 8, and P2 96. Fensalir now has a first
|
|
packed-Leap D3D12 SAD/block-match kernel that writes R16F disparity through
|
|
the typed SurfacePage socket; full SGM and calibrated metric depth remain
|
|
future work.
|
|
- The first Leap point-cloud root is code-visible:
|
|
`MimirPointCloudConfigurations.LeapDisparityPointCloudRoot` records the
|
|
stereo-disparity projection profile and provenance, and
|
|
`MimirFensalirFieldLowering.BuildLeapPackedStereoPointCloudCandidateFrame`
|
|
declares a GPU-resident `FieldMesh` PointList resource derived from the
|
|
Leap disparity SurfacePage. `MimirRuntime` merges that Mesh claim with the
|
|
live Leap stereo-depth claim. `Mimir.BufferSmoke --leap-point-cloud-root-smoke`
|
|
proves one `SurfacePage` packet plus one `Mesh` packet and zero deferred
|
|
requests. Fensalir branch `codex/leap-packed-depth` now owns the matching
|
|
first native surface-claim lane: generated PointList Mesh buffers are
|
|
filled from the disparity SurfacePage on D3D12 for diagnostics, while the
|
|
same compute pass writes approximated normals, curvature, SDF, PBR, and
|
|
radiosity data into Fensalir's splat reservoir buffers. The point-list draw
|
|
path is diagnostic-only; live presentation goes through Fensalir's
|
|
splat/reservoir resolver. The live proof
|
|
`artifacts/runtime/mimir-leap-surface-claims-clean.png` showed LeapUVC
|
|
frames uploaded through a Fensalir-owned D3D12 texture, packed stereo depth
|
|
dispatch-ready/dispatched, 19,200 generated surface claims, point-list
|
|
rendering at `rendered 0`, and visible reservoir-resolved Leap splats.
|
|
The follow-up proof
|
|
`artifacts/runtime/mimir-leap-history-split-shader-visible.png` ran without
|
|
`AQUARIUM_DISABLE_RESERVOIR_HISTORY`: Fensalir now compiles a dedicated lean
|
|
`D3D12ReservoirHistoryUpdate.hlsl` for current-sample history/resolved
|
|
output instead of compiling the monolithic post shader as a compute entry.
|
|
The 2026-05-31 all-sensor proof uses
|
|
`config/mimir-runtime.perfect-machine.local.json`: Leap packed stereo IR,
|
|
both PS3 Eyes at 320x240@187, Kiyo Pro RGB, and regular Kiyo RGB all open
|
|
through `MimirRuntime` and enter as Fensalir texture leases.
|
|
`Mimir.BufferSmoke --perfect-machine-live-smoke` now requires five live
|
|
camera Texture2D resources, one Leap stereo-depth lowering, one disparity
|
|
SurfacePage, and one derived point-cloud Mesh. The Fensalir headless run
|
|
started all five sources, declared eight field resources, resolved six
|
|
Texture2D resources, dispatched the packed-Leap stereo-depth kernel,
|
|
generated one point-cloud/surface stream, reported 19,200 visible reservoir
|
|
splats, and captured
|
|
`artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png`.
|
|
Generic camera Feature/SensorObservation claims now plan through Fensalir's
|
|
`GpuSensorFusion` backend with zero deferred requests. The first dispatch
|
|
samples LeapStereoIr plus both Bayer8 PS3 Eyes, generates 49,152 sparse
|
|
gaussians at about 0.059 ms/frame sensor-fusion GPU time, and captured
|
|
`artifacts/runtime/mimir-perfect-machine-gpu-feature-extraction-headless.png`.
|
|
The follow-up YUY2 decode proof samples LeapStereoIr, both Bayer8 PS3 Eyes,
|
|
Kiyo Pro YUY2, and regular Kiyo YUY2; it dispatches five camera textures
|
|
with zero unsupported field textures, generates 81,920 sparse gaussians at
|
|
about 0.08 ms/frame sensor-fusion GPU time, and captured
|
|
`artifacts/runtime/mimir-perfect-machine-yuy2-gpu-feature-extraction-headless.png`.
|
|
Packed video must not masquerade as ordinary RGBA: Fensalir binds
|
|
`DXGI_FORMAT_YUY2` through the documented `R8G8B8A8_UNORM` SRV view and
|
|
decodes `Y0/U/Y1/V` in compute.
|
|
Calibrated projection constants, full SGM, temporal/spatial reservoir reuse
|
|
in a compilable shader, and global residual/calibration ownership remain
|
|
open.
|
|
- LED strips on performers are active visual calibration evidence, not fake
|
|
depth. `MimirLedSplineFieldCandidate` carries per-camera ordered bright
|
|
curve observations in image pixels and local clipspace/frustum coordinates
|
|
with optional temporal code / stable LED indices, and
|
|
`MimirFensalirFieldLowering.BuildLedSplineCandidateFrame` lowers them as
|
|
deterministic camera-sensor Feature claims that Fensalir plans as
|
|
`DebugOverlay` calibration evidence. Exact cross-camera correspondence
|
|
requires addressable/blink-coded LEDs or another identity source; uncoded
|
|
identical LEDs remain useful curve constraints but cannot own camera pose.
|
|
`MimirLedSplineCurveSolver` now solves ordered per-sensor image/clipspace
|
|
curves from bright detections. `MimirCameraRigCalibrationSolver` consumes
|
|
stable LED indices plus a scene-space LED curve anchor and emits bounded
|
|
camera translation updates by minimizing ray-to-curve distance. Rotation,
|
|
focal length, and distortion are still future calibration cuts. Exposure
|
|
and gain sweeps must optimize `MimirLedSplineQualityScorer` output:
|
|
recovered LED count, curve coverage, spacing coherence, smoothness,
|
|
exposure fitness, and saturation penalty decide whether a setting is
|
|
usable for calibration. `MimirLedSplineFrameAnalyzer` now extracts bright
|
|
luma components from real frames into the same score path, so clipped blobs
|
|
can be detected while still rejected as bad spline evidence.
|
|
`Mimir.BufferSmoke --led-spline-live-score` now scores actual direct sensor
|
|
feeds. KS/UVC sources sweep manual exposure/gain before each live scoring
|
|
window and print a best setting per source; the 2026-05-31 live proof with
|
|
`--expected-leds 38 --minimum-luma 0.55` selected exposure `-9`, gain `8`
|
|
for both Kiyo Pro and Kiyo Basic. LeapUVC rejected the generic KS
|
|
exposure/gain writes and PS3 Eye capture remains fixed-control until its
|
|
native wrapper owns gain/exposure.
|
|
- PS Move is usable as an active marker/IMU witness, not as an optical depth
|
|
owner. `Mimir.PsMoveProbe` now enumerates Sony `VID_054C` / `PID_03D5`
|
|
through Windows HID without third-party packages. The connected controller
|
|
exposes three collections: a generic desktop gamepad collection with
|
|
49-byte input/output/feature reports and two vendor-defined feature-report
|
|
collections. USB report decoding should own identity/buttons/IMU and later
|
|
LED sphere control; camera observations of the sphere should enter the
|
|
existing global calibration evidence path.
|
|
`--pair-host <bdaddr>` now writes the target Bluetooth host to the
|
|
controller through feature report `0x05` on the `col02` collection. The
|
|
first live controller `00:06:f5:23:e2:d1` was written with Nightwing's
|
|
host address `5c:93:a2:9c:a8:a8`. Nightwing BlueZ is installed and
|
|
registered with the PS Move API Linux BlueZ record shape, so the
|
|
controller now connects wirelessly as `Motion Controller` and exposes
|
|
`/dev/input/js0`, `/dev/input/event14`, and `/dev/hidraw1`.
|
|
`tools/nightwing_psmove_music_pulse.py` proves the output path by reading
|
|
Starfire Scarlett ASIO loopback channels 2/3, computing a thresholdable
|
|
Perlines-style adaptive-whitened spectral-delta novelty function, fitting
|
|
a provisional fundamental, folding spectral bin energy into one octave
|
|
around that fundamental for RGB balance, and streaming LED report `0x06`
|
|
frames over SSH to Nightwing's hidraw writer. The LED idles near black.
|
|
Onset amplitude ranks the current whitened-delta broadband flux against a
|
|
recent rolling history and emits `percentile^exponent`, so exponent `2`
|
|
makes a median notable frame 25% brightness while the strongest recent
|
|
rises approach full brightness. The ASIO reader now drains the native
|
|
capture queue to the newest loopback blocks before each LED frame, because
|
|
stale queued audio was the plausible seconds-late music-pulse cause.
|
|
`tools/online_move_music_sync.py` is now the live multi-Move music witness:
|
|
Realtek loopback remains the primary program reference, optional Scarlett
|
|
ASIO channels contribute auxiliary mic/loopback onset streams with their
|
|
own autocorrelation tempo estimates, chroma scoring estimates key and chord
|
|
candidates, and a recent loudness-percentile gate owns whether the Moves
|
|
are allowed to brighten. Flux alone no longer has authority to shout.
|
|
The current score cut adds `ScoreGestureScheduler` as the only owner of
|
|
Move visual gesture openings. Raw level, hit, flux, and color balance are
|
|
inputs; they no longer emit continuous RGB by themselves. The scheduler
|
|
divides the beat into score slots, assigns slots across Moves with
|
|
de-Bruijn/polyrhythmic identity, opens short envelopes only when recent
|
|
broadband spectral rise reaches the adaptive loudness gate, and maps the
|
|
top of that gate to the configured maximum envelope. Quiet frames are
|
|
black. The 2026-06-01 live worker
|
|
`artifacts/runtime/move-music-score-20260601-220807/online-sync.jsonl`
|
|
showed `max_env=0.4`, `max_rgb=4`, and only 9 lit frames in an 824-frame
|
|
short-window poll, which is the intended conservative instrument behavior.
|
|
AquaSynth Syrinx voices are the correct Realtek audio-signal target, but
|
|
the live emitter is still the Mimir bioacoustic/Move script until the
|
|
Realtek path is explicitly routed through AquaSynth.Faust/Syrinx patch
|
|
compilation.
|
|
`tools/move_latency_probe.py` is the diagnostic PS Move latency receipt:
|
|
PS3 Eyes are the default fast visual witnesses, Scarlett cardioid is the
|
|
default chirp timing witness, and the event train uses an order-3 de Bruijn
|
|
schedule with symbol-coded chirp bands, LED colors, and small variable
|
|
gaps. The probe writes `event-schedule.json`, arms Nightwing's LED script
|
|
before playback, fits audio/video detections back to the known schedule,
|
|
and marks receipts invalid when the Move sphere is not visible enough.
|
|
Live encoded Eye/Scarlett testing recovered all eight visual events with
|
|
schedule residuals within about 4 ms; `ffplay`/DirectShow audio startup is
|
|
still too soft for final physical latency authority, so native timestamped
|
|
Scarlett playback/capture remains the production timing cut.
|
|
`tools/music_keyed_move_chirp_sync.py` now writes articulated calibration
|
|
word plans where one `mimir.music_keyed_move_chirp_plan.v1` owns beat/key
|
|
selection, symbol sequence, chirp contours, and per-Move sampled RGB visual
|
|
contours. `tools/sync_sweep.py --music-plan` consumes that same plan as the
|
|
shared capture authority: it plays/copies the chirp train, sends exact Move
|
|
contour samples over Nightwing HID, scores RGB camera receipts with a
|
|
`visual_contour` brightness/color decoder block, and runs Mimir
|
|
perfect-machine live receipts for local Leap/Kiyos and Raven/Eve configs
|
|
unless disabled. DirectShow/Nightwing media capture remains an archive and
|
|
diagnostic path; Leap/Eve presence is now explicit in typed Mimir receipts
|
|
rather than silently inferred. `tools/crunch_sync_sweep.py` turns a sweep
|
|
directory into `calibration-observations.json/csv` by merging Nightwing's
|
|
typed Move observations with local Kiyo frame-derived marker centroids.
|
|
The ASAP streaming MVP is narrower and authority-correct:
|
|
`config/mimir-runtime.mvp-leap-eyes-kiyo.local.json` keeps Starfire's live
|
|
rolling buffers to local Scarlett ASIO, local Leap stereo IR, and local
|
|
Kiyo Pro RGB only. Nightwing's PS3 Eyes do not cross as raw video; the
|
|
`start-mvp-leap-eyes-kiyo.ps1` launcher stages Nightwing tools and starts
|
|
`nightwing_typed_witness_publisher.py --track-eyes`, so Nightwing publishes
|
|
compact `mimir.move_controller_observation_state.v1` claims through the
|
|
`/eve/periwinkle` CultMesh observation receiver.
|
|
The stream-proof profile is the live demo cut:
|
|
`config/mimir-runtime.stream-proof.local.json` keeps Starfire's canonical
|
|
window on local Scarlett ASIO, local Leap stereo IR, local Kiyo Pro RGB,
|
|
and Raven display/audio from one muxed Raven NVENC/AAC MPEG-TS SRT stream
|
|
on port 5200. `scripts/start-raven-av-sender.ps1` captures Raven's desktop
|
|
through FFmpeg `ddagrab` Desktop Duplication API plus NVENC and captures
|
|
Raven's Realtek/default render
|
|
loopback through the compiled `Mimir.WasapiLoopback` helper when staged at
|
|
`tools/Mimir.WasapiLoopback/Mimir.WasapiLoopback.exe`; the older
|
|
PowerShell loopback reader is now only a diagnostic fallback. BtbN current
|
|
FFmpeg was tested on Raven and still exposes no native `wasapi` input
|
|
device, so loopback capture cannot be solved by swapping FFmpeg builds
|
|
alone. Raven's local `ddagrab + Mimir.WasapiLoopback + AAC + h264_nvenc`
|
|
proof recorded 568 frames in ten seconds at 1080p60; sparse-frame direct
|
|
TCP/SSH proof captures are therefore transport/proof-path evidence, not
|
|
Raven GPU/capture evidence. `scripts/start-raven-av-demux.ps1`
|
|
is the Starfire local demux edge, copying video to local SRT port 5210 and
|
|
decoding audio to f32 PCM on local SRT port 5212 for Mimir's current raw
|
|
ingest adapters. `MimirFfmpegPcmAudioStreamSource` owns the post-demux PCM
|
|
edge and emits real `MimirAudioBlockDescriptor` samples instead of generic
|
|
process bytes. `scripts/start-stream-proof.ps1` stages Nightwing's compact
|
|
Eye/Move publisher and keeps only one `--track-eyes` worker alive.
|
|
`tools/solve_stream_proof_frustum.py`
|
|
turns articulated sweep observations into a provisional Kiyo AR trail
|
|
receipt: Nightwing matched views define a temporary local witness space,
|
|
Kiyo observations are fit into it, and the output
|
|
`mimir.stream_proof_frustum_solve.v1` is calibration evidence for the demo,
|
|
not final metric room truth.
|
|
Raven display sync is explicit: audio-derived timing states for
|
|
`raven-realtk-loopback` now carry the `raven-sync` clock domain from the
|
|
source descriptor, and `MimirSynchronizedBufferPlanner` applies that earned
|
|
correction to every buffer in the same domain, including `raven-display`.
|
|
The LAN Raven path is now one muxed A/V transport; the local split is only
|
|
a Starfire demux boundary for Mimir's raw video/PCM source adapters.
|
|
`MimirMoveControllerTracker` is the first Mimir-native overlay owner for
|
|
multiple PS Move controllers: it tracks bright sphere observations from
|
|
PS3 Eye luma frames, keeps a bounded history per `controllerId`, preserves
|
|
the commanded expected RGB identity for each controller, and lowers those
|
|
histories into Feature evidence without claiming solved 6DoF pose.
|
|
`Mimir.BufferSmoke --move-controller-overlay-smoke` proves two distinct
|
|
controller histories produce one valid FieldEvidence claim/candidate.
|
|
`config/mimir-runtime.move-stream.local.json` is the first show profile:
|
|
Kiyo Pro program view, both PS3 Eyes as Move trackers, Scarlett ASIO
|
|
labeled as shotgun/cardioid/Raven loopback, and Raven display over SRT
|
|
port 5200.
|
|
Nightwing is now modeled as the dedicated Eyes/Moves witness body rather
|
|
than a vague remote helper: it owns PS3 Eye USB reads, PS Move
|
|
Bluetooth/HID, LED schedules, device timestamps, and compact
|
|
marker/feature extraction receipts. Starfire keeps the canonical rolling
|
|
window, Scarlett timing, global residuals, Kiyo/Raven program sync, and
|
|
OBS publication. Fensalir keeps D3D12 dense feature/fusion work, surface
|
|
claims, and splat/reservoir presentation. The
|
|
`nightwing-eyes-moves-witness` config and `nightwing-eyes-moves` Perfect
|
|
Machine profile emit typed Move observation, camera feature-track, and
|
|
visual marker state; raw Eye media from Nightwing is diagnostic only and
|
|
Nightwing does not own camera placement or the canonical clock.
|
|
`Mimir.EveSensorReceiver` now has the first subscriber lane for this
|
|
witness stream: `/eve/periwinkle` ingests typed observations and
|
|
`/eve/periwinkle/subscribe` broadcasts normalized records to live
|
|
subscribers. `Mimir.VerseRecorder` subscribes and writes a session manifest
|
|
plus JSONL observation ledger under `C:\Users\Meta\Videos\Mimir\VerseCaptures`.
|
|
`Mimir.Well` is the Starfire Well daemon: it loads the broad Well runtime
|
|
profile, starts every configured source it can, polls `MimirSynchronizationHub`
|
|
with a bounded daemon budget, updates audio sync/probe state, and publishes
|
|
`mimir.well_snapshot.v1` records containing buffer health, composite
|
|
controls, publication/stem contract, and sync status. The online supervisor
|
|
now starts relay, recorder, Nightwing Eyes/Move/builtin-camera/builtin-mic
|
|
witness publication, conservative de-Bruijn/microtonal music-synced Move
|
|
gestures with Realtek plus opportunistic ASIO music evidence, and
|
|
`Mimir.Well`. This owns typed state publication, not raw CultMesh media
|
|
payloads yet. The live timestamp-domain scar is explicit: ASIO device time
|
|
and camera driver time are not in one canonical timeline, so sync frame
|
|
publication is degraded until a camera-clock/global-time correction owner
|
|
exists.
|
|
The Eve live-stats provider now reads the latest `mimir.well_snapshot.v1`
|
|
records from the observation ledger and the latest
|
|
`mimir.online_move_music_sync_frame.v1` trace from runtime artifacts. Its
|
|
compact retained surface subdivides into overview, Move score, Well,
|
|
audio, sync, video, Leap/Eyes/Moves calibration evidence, and actuator
|
|
panes. Leap usefulness is reported as live Well buffer health and edge age,
|
|
while Nightwing Eyes/Moves remain compact typed witness observations; the
|
|
dashboard observes those numbers and does not become placement authority.
|
|
Eve follows the same distributed-witness rule for its own body:
|
|
`eve-local-sensor-witness` / `eve-local-sensors` owns Eve camera, Eve mic,
|
|
local flash/chirp decode, feature tracks, visual markers, and local frustum
|
|
hints. It may send compact typed evidence, but raw Eve media is not global
|
|
placement authority and Eve never owns the canonical clock.
|
|
`MimirCameraFrustumCalibrationSolver` is now the first Starfire-owned
|
|
frustum solve: stable LED/marker correspondences plus scene-space marker
|
|
points and seed frustums produce per-source position, rotation,
|
|
horizontal/vertical tangent half-FOV, correspondence count, clipspace
|
|
reprojection residual, and confidence. This supersedes translation-only
|
|
rig updates for global fusion setup; the older
|
|
`MimirCameraRigCalibrationSolver` remains a narrow bounded translation
|
|
residual helper. The same solver also consumes
|
|
`MimirFeatureTrackFieldCandidate` plus scene-space `TrackId` anchors, so
|
|
PS Move sphere histories, Eye feature tracks, and Eve-local marker tracks
|
|
can enter the frustum solve without becoming a separate authority.
|
|
`mimir.local_frustum_hint_state.v1` is the typed CultMesh document for
|
|
publishing those local solved hints: producer node, calibration id,
|
|
candidate key, marker set id, per-camera pose/frustum snapshot, residual,
|
|
point count, and confidence. Starfire may ingest this as evidence; it is
|
|
still not canonical placement until the global solve accepts it.
|
|
`Mimir.BufferSmoke --frustum-calibration-smoke` proves five LED-spline
|
|
frustums and five feature-track frustums for both Eyes, Kiyo Pro, Leap
|
|
stereo IR, and Eve camera, then lowers the feature-track solve into a
|
|
`mimir.local_frustum_hint_state.v1` document.
|
|
The Eyes have physically moved to Nightwing and Linux sees them as stock
|
|
`ov534`/`gspca` V4L2 cameras on `/dev/video2` and `/dev/video3`. A
|
|
repo-owned `tools/nightwing_v4l2_cadence_probe.py` copied to Nightwing
|
|
proved both devices can deliver 320x240 at about 187 fps with zero
|
|
sequence gaps when the worker explicitly sets `VIDIOC_S_PARM`
|
|
`timeperframe=1/187`; default V4L2 streaming remains a false 30 fps signal.
|
|
- PS3 Eyes now have a first sparse tracking surface instead of only cadence
|
|
proof. `MimirSparseFeatureTracker` detects Bayer/luma corner features,
|
|
links them into sensor-local tracks with clipspace position, velocity, age,
|
|
and confidence, and `MimirFeatureTrackFieldCandidate` lowers them as
|
|
camera Feature claims. `Mimir.BufferSmoke --ps3eye-tracking-live` proved a
|
|
one-second 320x240@187 run can return 160 live tracks from each Eye: 160
|
|
stable tracks from `ps3-eye-0` and 156 stable tracks from `ps3-eye-1` with
|
|
sensitive defaults. This is CPU diagnostic/bootstrap work;
|
|
Fensalir D3D12 feature extraction remains the production owner.
|
|
- Fensalir packet payload handles are not authority. Live payload references
|
|
must resolve through declared typed GPU/native resources with shape,
|
|
residency, shader access, validity, version, and handle metadata before
|
|
shader lowering can consume them.
|
|
- EVE fullscreen output must use native decode/composite over Fensalir-owned
|
|
program pixels; WebKit/DOM layout is forbidden as pixel authority.
|
|
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
|
|
stems.
|
|
- Local secrets remain untracked.
|
|
- Mimir is a CultMesh app. Codebooks, schedules, calibration receipts, path
|
|
response, and compute-offload work need typed mesh-syncable state surfaces
|
|
so phones, microcontrollers, Raven, and Starfire can cooperate without
|
|
becoming independent timing authorities.
|
|
|
|
accepted_design:
|
|
- `Mimir.slnx` owns the app surface.
|
|
- `src/Mimir.App` launches Fensalir as host/window/D3D12 library.
|
|
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
|
|
`MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`,
|
|
direct native ingest, process bridge ingest, and video capture driver seams.
|
|
- `MimirVideoFrameDescriptor` carries width, height, pixel format, stride,
|
|
device timestamp, and optional native/GPU handle.
|
|
- `MimirEveProgramOutputConfigurations.NativeSharedTexture` records the
|
|
EVE-facing program-output contract: Fensalir publishes
|
|
`Global\MimirFensalirProgramTexture` when
|
|
`FENSALIR_PROGRAM_OUTPUT_D3D12=1`; EVE receives decoded native pixels and
|
|
owns only final composite/input uplink.
|
|
- `src/Mimir.EveRelay` is the current EVE dashboard transport slice:
|
|
it opens the named D3D12 program-output texture, feeds low-latency H.264
|
|
(`h264_nvenc` by default), frames Annex-B access units over `/stream`, and
|
|
reports `/health`. EveCanvas uses `AVSampleBufferDisplayLayer` for native
|
|
decode/composite. The local launch script opens an SSH reverse tunnel to
|
|
Eve when Windows firewall policy blocks direct inbound TCP, and EveCanvas
|
|
tries localhost before the LAN host.
|
|
- `src/Mimir.Broadcast` is the first self-hosted public edge sender:
|
|
Starfire uses FFmpeg/NVENC to push one final H.264/AAC RTMP stream through
|
|
an SSH local forward to Yggdrasil. Yggdrasil owns nginx RTMP ingest and HLS
|
|
segment serving only; it does not transcode or compose. The public viewer
|
|
route is `https://gamecult.org/livestream`, and it reads the existing
|
|
Yggdrasil-routed StreamPixels URL
|
|
`https://streampixels.gamecult.org/mimir/live/hls/mimir.m3u8`. The deployed
|
|
Yggdrasil origin already validates, listens on localhost RTMP
|
|
`127.0.0.1:1935`, and produced HLS segments from an eight-second Starfire
|
|
synthetic NVENC push on 2026-05-30. Dedicated Mimir-site and
|
|
`live.mimir.gamecult.org` viewer/media hostnames are not needed for v1.
|
|
- `MimirAudioBlockDescriptor` carries sample rate, channels, sample format,
|
|
frame count, timestamp, and optional native handle.
|
|
- `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for
|
|
JSON-line video frame and audio block metadata from native probes into
|
|
runtime rolling buffers; it does not carry pixels/audio payloads or own the
|
|
final local-device hot path.
|
|
- `MimirBioacousticTimeline` is the active runtime watermark. It renders a
|
|
low-gain birdsong-like word language with 128 self-identifying word
|
|
positions, left-speaker and right-speaker variants, four formant-rich
|
|
syllables per word, rhythm variation, and direct word identity instead of a
|
|
de Bruijn grammar. The active receiver is a song-contour anchor machine:
|
|
one heard call should expose syllable timing, contour bends, formants,
|
|
payload ornaments, speaker tint, and log-mel evidence as a cluster of
|
|
time/frequency anchors. `MimirRuntime` emits separate panned vocabularies
|
|
per speaker. `MimirAudioSynchronizationAnalyzer` reports active timing as
|
|
`evidence=bioacoustic`; the old chirp-bin path remains a
|
|
calibration/reference artifact with ASIO tooling.
|
|
- The diagnostic audio path can optionally carry base64 float sample payloads
|
|
for sync smoke tests. `MimirChirpletTimeline` owns the older structured
|
|
chirplet reference fingerprint: an order-3 de Bruijn sequence over 32
|
|
time/frequency constellation symbols, plus PCM segment rendering, matched
|
|
timing atoms, symbol/event decoding, and per-band response kernels. Symbol
|
|
identity is carried by start band, glide shape, duration, and following
|
|
inter-chirp gap. Any three consecutive correctly detected symbols identify a
|
|
timeline event within the current operating horizon. `MimirChirpletSymbolCodebook`
|
|
owns the 32 symbol definitions so every symbol has a unique chirp shape, with
|
|
rhythm as additional evidence. `MimirChirpletStreamDecoder` is the first
|
|
constrained chirplet-transform receiver: it owns a bounded PCM window, emits
|
|
transform frames with multiple phase-invariant symbol candidates and refined
|
|
per-candidate sample offsets, decodes code-valid triplet anchors through
|
|
gap/clock coherence, and fits a per-source sample clock from those anchors.
|
|
Its synthetic smoke path is `dotnet run --project
|
|
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirplet-self-test`.
|
|
It currently decodes all 15 emitted chirps in a two-second window into 13
|
|
anchors for events 0-12 with a 47999.999990 Hz clock fit and 0.000014 sample
|
|
MAE. `MimirAudioSynchronizationAnalyzer`
|
|
now treats `loopback-scarlett-speakers` as timing authority and derives
|
|
delay only from matched decoded timeline anchors. Positive delay means
|
|
the candidate mic is late relative to loopback. Reports include rounded
|
|
delay, fractional delay, confidence, and per-band matched energy. `MimirSynchronizationHub`
|
|
caches the latest per-source reports and `MimirAudioSynchronizationStateTracker`
|
|
owns the smoothed per-source belief: latest delay, smoothed delay,
|
|
confidence, per-band response evidence, and delay-slope/SRO estimate in
|
|
ppm. `MimirRuntime` now runs this analysis online as a bounded rotating
|
|
service on a fixed cadence, while UI and telemetry read cached reports/states
|
|
instead of invoking analysis. `MIMIR_SYNC_TELEMETRY_SECONDS` prints live
|
|
buffer/report/state telemetry. `MimirAudioSynchronizationSettings.Mode`
|
|
now owns the active evidence policy: `chirp-only` emits the calibration
|
|
witness, `passive` stays silent and uses program-audio phase correlation,
|
|
and `hybrid` emits active pilot chunks only while passive confidence is
|
|
weak. `MimirPassiveAudioSynchronizationEstimator` is the first passive path:
|
|
it uses PHAT-weighted cross-spectrum correlation between loopback and mic
|
|
windows before any watermark is needed. Negative passive lags are
|
|
contradictory evidence and carry zero confidence; single-window passive
|
|
confidence is capped below certainty until repeated state earns it.
|
|
`MimirBioacousticTimeline` is now the active calibration witness for both
|
|
chirp-only and hybrid: 128 self-identifying word positions, left/right
|
|
speaker variants, four formant-rich syllables per word, rhythm variation,
|
|
and direct word identity so any correctly decoded word identifies one
|
|
canonical timeline location. The analyzer refines final active delay with constrained local
|
|
waveform correlation around the decoded anchor delay. Each detected motif
|
|
carries per-band response evidence, and the next calibration cut should use
|
|
those profiles to tune contours, formants, usable bands, gain, and rhythm
|
|
weighting. `Mimir.BufferSmoke --bioacoustic-cepstral-smoke` now proves the
|
|
non-brute-force receiver direction: degraded mel-cepstral calls are decoded
|
|
by an augmented MFCC/projection-hash word index before any expensive
|
|
confirmation. `Mimir.BufferSmoke --bioacoustic-train` now runs a
|
|
receipt-backed hypothesis panel over indexed cepstral decoder shapes and
|
|
writes typed CultCache results plus pre-warp/post-warp/reconstructed WAV
|
|
artifacts under `artifacts/bioacoustic-training/`. The first receipt
|
|
`bioacoustic-20260524-103438` showed compact indexing fastest on clean
|
|
audio and exposed timing error under domain warp. The corrected receipt
|
|
`bioacoustic-20260524-150729` adds global clock hypothesis scoring over a
|
|
longer window; compact indexing wins clean throughput, baseline indexing
|
|
remains steadier under degradation, highband indexing has warped-path value,
|
|
and anchor density needs its own metric. `MimirChirpBinTimeline` remains the controlled chirp-bin
|
|
calibration/reference artifact with ASIO tooling and
|
|
`MimirChirpBinCalibrationModel`; keep those captures as response evidence,
|
|
not the active runtime sound. Reports and sync states expose delay in
|
|
microseconds as well as fractional samples. Hybrid emits the bioacoustic
|
|
witness as low-gain half-second bursts while passive confidence is weak,
|
|
never as catch-up backlog. The analyzer no longer applies the passive
|
|
two-second window floor to active fallback; `dotnet run --project
|
|
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 48000`
|
|
recovers a 317.375-sample synthetic delay with printed 0.000 us error using
|
|
`evidence=bioacoustic`, while `--hybrid-sync-self-test --sample-rate 48000`
|
|
correctly accepts strong passive evidence at 2.772 us error. The synthetic
|
|
standalone receiver proof `--standalone-bioacoustic-self-test --sample-rate
|
|
48000 --delay-samples 1269.5` recovers delayed source time below printed
|
|
microsecond precision using only the codebook/schedule. `--bioacoustic-actuator-self-test
|
|
--sample-rate 48000 --delay-samples 317.375` now proves the first
|
|
measurement-to-sample-movement loop: estimate bioacoustic delay, apply
|
|
fractional correction, and remeasure residual below printed microsecond
|
|
precision. `faust/mimir_alignment_actuator.dsp` is the first Faust-owned
|
|
actuator surface for six channels of bounded fractional delay/gain.
|
|
`MimirAlignmentActuatorBank` now converts runtime sync states into explicit
|
|
nonnegative Faust holdback commands with stable source slots, a separate
|
|
reference holdback, and overflow reporting, then queues those commands into
|
|
`AquariumAudioDocument` as engine audio control frames. Mimir also queues
|
|
`faust/mimir_alignment_actuator.dsp` as an `AquariumStreamingDspProgram`
|
|
so Fensalir can compile/host the persistent Faust graph that consumes those
|
|
controls. The same runtime cadence queues sample-bearing
|
|
`AquariumStreamingAudioBlock` lanes from the latest actuator source buffers,
|
|
with channel indices derived from the bank-assigned Faust `sourceN` control
|
|
path. Fensalir now publishes processed DSP output lanes as
|
|
`AquariumAudioStemFrame` records on an engine-owned audio stem bus, and
|
|
Mimir declares deterministic aligned-source stem names with the actuator
|
|
program. Mimir now consumes that bus through `MimirObsStemPublicationState`,
|
|
validating ready/missing/unconfigured lanes against the alignment actuator
|
|
stem-bus profile without copying audio. `MimirObsStemSharedMemoryPublisher`
|
|
writes validated stems into the `Local\MimirObsStemBus` Windows
|
|
shared-memory ABI, and `native/obs_stem_source` is the first OBS source
|
|
plugin bundle. It registers `Mimir Audio Stem`, which reads one named stem
|
|
from that map and submits it as libobs audio, and `Mimir Program Texture`,
|
|
which opens Fensalir's named shared D3D12 program-output texture and draws
|
|
it through an OBS-readable shared D3D11 bridge texture. Fensalir also
|
|
publishes `Global\MimirFensalirProgramFence` as a program-output
|
|
publication fence; the OBS source opens that fence and observes completion
|
|
before copying. The video path is GPU-to-GPU and does not require Spout2 or
|
|
CPU readback; the remaining copy is the explicit D3D12 to libobs-D3D11
|
|
boundary. Fensalir and the OBS source can optionally agree on
|
|
`FENSALIR_PROGRAM_OUTPUT_RING_COUNT>1`; Fensalir publishes
|
|
`Global\MimirFensalirProgramTexture.0`, `.1`, and so on, and OBS chooses the
|
|
latest completed slot from the program-output fence value. OBS can publish
|
|
`Global\MimirObsProgramConsumerFence`; when Fensalir is launched with
|
|
`FENSALIR_PROGRAM_OUTPUT_CONSUMER_FENCE_NAME`, it skips publication instead
|
|
of reusing a ring slot OBS has not acknowledged.
|
|
The rendering migration's program-output receipt phase is structurally
|
|
complete: OBS consumes final Mimir/Fensalir program surfaces and stems
|
|
through the native plugin, raw feeds remain debug inputs, and the ledger
|
|
records the live texture, texture ring, producer-fence, and consumer-fence
|
|
proofs.
|
|
`MimirSynchronizedBufferPlanner` is the first aligned-buffer readout
|
|
primitive: it chooses one canonical presentation time inside the rolling
|
|
window and returns per-stream slices for local cameras, network display
|
|
feeds, and audio blocks. Timing corrections can target a single source or a
|
|
shared `clockDomainId`; the Raven path uses audio evidence routed into
|
|
Scarlett to earn a `raven-sync` correction for Raven display frames.
|
|
Network arrival timestamps remain metadata only.
|
|
`MimirFfmpegRawVideoStreamSource` is the network-pixel adapter for Raven
|
|
ingest: FFmpeg owns SRT receive/decode, while Mimir owns exact raw-video
|
|
frame reads, payload geometry, video descriptors, and rolling-buffer
|
|
samples. `config/mimir-runtime.raven-eve.example.json` listens for
|
|
`raven-display` on port 5200. EveCanvas now owns Eve camera/mic capture
|
|
directly and sends typed frame-events to `Mimir.EveSensorReceiver` on
|
|
WebSocket ports 8793 (`eve-camera`) and 8794 (`eve-mic`).
|
|
Raven can send desktop capture with `scripts/start-raven-screen-capture-sender.ps1`.
|
|
`MimirPresentationControlState` owns program-control intent for Fensalir:
|
|
video feed visibility/solo/opacity/layer order, audio mute/solo/gain, and
|
|
global LUT preset selection. The `Mimir Program` panel filters video
|
|
production surface intents and applies audio mix gain before/inside the
|
|
Faust streaming DSP path. LUT presets are typed postprocess state; exposure
|
|
and bloom map into current Fensalir `GraphicsSettings`, while actual LUT
|
|
texture sampling remains the named renderer hook.
|
|
`Mimir.EveDashboard` is the native Eve dashboard broker: `/eve/deck` sends
|
|
retained dashboard trees, `/eve/dashboard` remains the compatibility
|
|
endpoint, and `/eve/deck/providers` exposes provider manifests. Built-in
|
|
providers are the switchboard, `mimir.stream.layout`, `voidbot.swarm`, and
|
|
`yggdrasil.streampixels.edge`; launch-time external providers can be
|
|
registered as `id|title|ws-url` descriptors for LAN or SSH-forwarded TCP
|
|
surfaces. EveCanvas renders the scene graph and source panels natively in
|
|
UIKit with multitouch transform editing, and provider-card taps send
|
|
`open-provider`. `voidbot.swarm` reads VoidBot's existing
|
|
`.voidbot/status/swarm-state.json` projection, publishes avatar URLs on
|
|
identity nodes, and Eve renders a native VoidBot-style cockpit: CTB rail,
|
|
selected Face/status pane, state tree, and detail panel.
|
|
Dashboard manifests and states now have typed CultMesh-compatible document
|
|
shapes `mimir.eve_dashboard_manifest.v1` and
|
|
`mimir.eve_dashboard_state.v1`, while the live iPad lane stays
|
|
WebSocket/TCP because it is the proven tunnelable transport. The Mimir
|
|
stream-layout provider still uses fixture dashboard state; live binding to
|
|
`MimirPresentationControlState` and `MimirSceneEditorState` is the next
|
|
authority cut.
|
|
`MimirSceneEditorState` owns the separate Mimir-window editor graph:
|
|
editor camera, scene nodes, selected node, visibility/locks, 2D-plane
|
|
transforms, reset affordances, and grab/rotate/resize gizmo mode. Rolling
|
|
video buffers derive sensor-feed panel nodes, while SDF text and model
|
|
import controls create editor graph nodes. `MimirRuntime` renders derived
|
|
outlines/handles through the Fensalir frame and exposes the `Mimir Editor`
|
|
hierarchy/transform panel. This is not OBS program output. World SDF glyph
|
|
rendering, ASSIMP-style mesh decoding/upload, and pixel-accurate gizmo
|
|
hit-testing remain Fensalir renderer cuts.
|
|
`scripts/build-obs-stem-plugin.ps1` stages the upstream OBS plugin template
|
|
SDK under `artifacts/obs-sdk/` and builds the plugin DLL locally.
|
|
The synthetic
|
|
codebook proof is `dotnet run --project
|
|
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-self-test`.
|
|
`Mimir.BufferSmoke --render-chirp-bin-f32` plus
|
|
`native/probes/asio_audio_cadence --play-f32-mono --record-f32-interleaved`
|
|
now provides a real Scarlett ASIO artifact proof for active mode. At 192 kHz,
|
|
Loopback 1 -> Loopback 2 decoded at 0.000 us with 12 anchors and 0.999
|
|
confidence. `Mimir.BufferSmoke --calibrate-chirp-bin-asio-f32` now renders
|
|
or ingests a calibration capture and persists the response/confusion/delay
|
|
model under `calibration/chirp-bin/`; `--analyze-asio-f32 --calibration`
|
|
loads it back into the chirp-bin reference decoder. Physical inputs still fail accepted
|
|
pairwise timing against loopback in the stored artifact, but decoded mic
|
|
channels now leave calibration models; physical input 1 produced two reliable
|
|
symbols and a different strongest measured band set.
|
|
`native/asio_capture` and `MimirAsioStreamSource` are now the runtime
|
|
Scarlett path: Focusrite ASIO callbacks feed sample-bearing 192 kHz Float32
|
|
blocks into `Mimir.Runtime` in process. `config/mimir-runtime.asio.example.json`
|
|
proves continuous ASIO runtime ingest: a two-second BufferSmoke run at
|
|
192 kHz ingested more than 12,000 sample-bearing blocks across `asio-ch0`
|
|
through `asio-ch3` and retained 2,048 blocks per channel, all from the
|
|
Focusrite ASIO callback stream.
|
|
`Mimir.BufferSmoke --calibrate-contestant-asio-f32` now persists the active
|
|
packet-song physical response model under `calibration/bioacoustic/`,
|
|
including per-channel delay search, polarity, payload reliability,
|
|
response-normalization bands, gain, confidence, and pairwise propagation
|
|
delay. The latest canary-packet Scarlett receipt runs at 10.7x realtime
|
|
across four 192 kHz channels: loopback is 37/37 with 2.524 us MAE,
|
|
co-streamer shotgun is 37/37 payload with 58.785 us MAE, and the cardioid
|
|
is 26/37 with 90.558 us MAE. This is a serious calibration surface, not yet
|
|
true physical microsecond sync.
|
|
A fresh 192 kHz room receipt at
|
|
`calibration/bioacoustic/scarlett-canary-packet-192k-contour-fresh.json`
|
|
keeps loopback at 2.524 us MAE, improves the co-streamer shotgun to 37/37
|
|
payload with 34.083 us MAE, and improves the cardioid to 31/37 payload
|
|
with 92.996 us MAE. The next precision cut is intra-call contour anchors,
|
|
not more sequence machinery.
|
|
`calibration/bioacoustic/scarlett-canary-packet-anchor-rich-192k.json`
|
|
is the first anchor-rich receipt: timing chips, formant pivots,
|
|
harmonic-envelope notches, payload ornaments, renderer template caching, and
|
|
per-event intra-call anchor measurements. It clears 10x realtime and
|
|
improves loopback to about 2.23 us MAE, but physical mics regress/hold at
|
|
tens of microseconds, so anchor survivability and path weighting are now the
|
|
problem.
|
|
Path-level loopback truth now compares candidate mic anchors directly
|
|
against loopback anchors, then refines delay with event-local waveform
|
|
correlation against the captured loopback packet. On the stored anchor-rich
|
|
capture, Release calibration runs at 21.1x realtime and improves physical
|
|
path precision to 7.576 us for the cardioid and 6.578 us for the shotgun.
|
|
A fresh room capture keeps the shotgun near 5.916 us but worsens the
|
|
cardioid to 18.930 us. The one-microsecond target remains open; next cut is
|
|
phase/group-delay correction or a better direct-path anchor family. A naive
|
|
recursive waveform phase-lock pass was tested and reverted because it could
|
|
improve one cardioid receipt while worsening the shotgun and locking a fresh
|
|
cardioid run onto a later reflection lobe. Recursion is only allowed back as
|
|
a constrained direct-path/group-delay model, not as unconstrained waveform
|
|
resampling with prettier decimals. The sonar/DSP research note
|
|
`research/sonar-dsp-recursive-refinement-2026-05-25.md` points the next
|
|
decoder cut at complex contour matched filters, phase-slope/group-delay
|
|
fitting, acquisition/tracking loop bandwidths, and sparse multipath
|
|
residuals. Its source ledger and local mirrors live under
|
|
`research/sonar-dsp-recursive-refinement-2026-05-25/`.
|
|
`MimirComplexContourMatchedFilterBank` and `MimirDirectPathTracker` are the
|
|
first implementation slice of that shape: complex anchor responses keep
|
|
multiple lobes, the current path prediction gates direct-path ownership, and
|
|
later coherent clusters are reported as reflection taps. Synthetic 192 kHz
|
|
reflection smoke currently lands about 5.249 us from expected delay; stored
|
|
Scarlett shotgun receipts land within about 5.860 us and 7.247 us of seeded
|
|
path fits, while cardioid receipts land within about 15.817 us and 8.963 us.
|
|
Low confidence is expected until phase/group-delay surfaces are learned. The
|
|
tracker now exposes per-band delay/phase residual observations and a
|
|
`MimirDirectPathChannelModel` correction surface for multi-window
|
|
calibration; one-window self-correction is not live authority. The tracker
|
|
now consumes explicit channel models as phase/group-delay correction and
|
|
downweights bands outside the learned usable surface. `Mimir.BufferSmoke
|
|
--learn-complex-contour-channel-model` persists
|
|
`calibration/bioacoustic/complex-contour-channel-model.json` from the
|
|
replay panel, and `--evaluate-complex-contour-channel-model` persists
|
|
`calibration/bioacoustic/complex-contour-channel-model-evaluation.json`.
|
|
Current evaluation improves 3/4 absolute path-seed errors and lowers mean
|
|
direct-cluster MAE; the fresh cardioid case reaches about 0.202 us from
|
|
seed, while the stored cardioid case still worsens to about 17.476 us. The
|
|
model is explicit calibration evidence, not default runtime authority.
|
|
The complex contour receiver is now wired into runtime as an explicit live
|
|
lane: when `enableComplexContourRuntime` is true, Mimir emits the configured
|
|
`bioacousticWitnessProfileId` through `MimirBioacousticContestantRenderer`,
|
|
loads `complexContourChannelModelPath`, extracts Float32 windows from the
|
|
rolling buffers, and publishes `evidence=complex-contour` reports. The
|
|
synthetic runtime-shaped proof `--complex-contour-runtime-self-test` at
|
|
192 kHz recovers a 693.5-sample delay with about 0.219 us error.
|
|
`MimirCalibratedAudioCompositeBuilder` is the first diagnostic composite
|
|
owner for corrected mic streams: it consumes audio samples plus
|
|
`MimirAudioSynchronizationState` delay/confidence and measured band
|
|
responses, applies fractional alignment, sparse measured-band
|
|
equalization, calibrated-band coherence scoring, incoherent-band
|
|
suppression, and confidence/noise-weighted summing, then emits composite
|
|
samples plus flatness/weight/coherence reports and per-band
|
|
coherence/suppression reports. `Mimir.BufferSmoke
|
|
--calibrated-audio-composite-smoke` proves the contract on synthetic
|
|
delayed/noisy/colored mic paths, improving SNR from -3.171 dB uncalibrated
|
|
to 15.135 dB calibrated and response flatness from 0.548 to 1.000.
|
|
`Mimir.BufferSmoke --dual-mic-coherence-cancellation-smoke` proves the
|
|
two-witness cancellation cut: shared tones keep about 0.999 coherence, a
|
|
mic-local 3.52 kHz interferer scores about 0.001 coherence and 0.948
|
|
suppression, its measured energy falls from 0.039195 to 0.001306, and SNR
|
|
rises from 1.366 dB to 16.304 dB.
|
|
`scripts/record-kiyo-dual-mic-composite.ps1` is the first listenable
|
|
Faust-chain bridge receipt for that path. It starts Kiyo FFmpeg capture and
|
|
Scarlett ASIO raw capture together, converts ASIO channels 0/1 through
|
|
`--asio-dual-mic-faust-cleaner-wav`, then muxes the Faust dialogue WAV with
|
|
Kiyo video. The offline command compiles/runs
|
|
`faust/mimir_alignment_actuator.dsp` before
|
|
`faust/mimir_dual_mic_dialogue_cleaner.dsp`, so the receipt exercises the
|
|
real alignment-then-cleaner chain. The fresh 2026-06-01 receipt
|
|
`artifacts/runtime/kiyo-dual-mic-composite/20260601-051044` and latest MP4
|
|
`artifacts/runtime/kiyo-dual-mic-composite/kiyo-dual-mic-composite-latest.mp4`
|
|
has 10.000 s of 640x480 30 fps Kiyo video and 10.000 s of 48 kHz mono AAC
|
|
from the Faust dialogue stem; the report estimated the cardioid 359.398
|
|
samples late (7487.452 us), applied holdback 359.398/0.000 samples to
|
|
shotgun/cardioid, and wrote both dialogue and residual WAVs. The first mux
|
|
was nearly silent because `NormalizePeak` only attenuated hot samples; after
|
|
fixing it to amplify quiet receipts too, the regenerated dialogue WAV peaks
|
|
at -0.724 dBFS with RMS -14.202 dBFS.
|
|
Faust/native DSP now has the first actual cleaner socket:
|
|
`faust/mimir_dual_mic_dialogue_cleaner.dsp` compiles as a smooth two-input
|
|
shotgun/cardioid dialogue cleaner and Mimir queues it as
|
|
`dual-mic-dialogue-cleaner`, fed from the alignment actuator's published
|
|
`aligned_source_*` stem frame rather than raw mic buffers, emitting
|
|
`host_voice` plus a rejected residual witness. `Mimir.BufferSmoke
|
|
--dialogue-cleaner-routing-smoke` verifies that aligned stem frames queue
|
|
the cleaner program/block and that cleaner output, not raw alignment lanes,
|
|
feeds `NativeProgram` stem publication. The C# path is a contract, scorer,
|
|
and offline diagnostic, not a competing live mixer.
|
|
`MimirTargetedBioacousticProbePlanner` is now the residual-driven probe
|
|
planner for improving that calibration instead of blindly EQing harder. It
|
|
consumes per-source spectral residuals plus optional source pathology notes
|
|
such as high noise floor and low-heavy tilt, then emits gain-bounded probe
|
|
bands for the bioacoustic renderer. The first smoke is intentionally scoped
|
|
to the two professional 192 kHz Scarlett mics: `asio-ch0-shotgun` is
|
|
modeled as noisy and low-heavy, `asio-ch1-cardioid` as cleaner. The planner
|
|
selects about 18.6 kHz first as unmeasured high-band evidence, then about
|
|
9.9 kHz weak response, 3.6 kHz phase instability, and 910 Hz magnitude
|
|
residual. Use the two good mics first; every connected mic can become room
|
|
field evidence later, but it should not dilute the initial direct-path
|
|
calibration loop.
|
|
`MimirBioacousticProbeScheduler` now closes the first control loop above
|
|
that planner: smoothed sync confidence and derived per-band frequency
|
|
response confidence decide when active sound is worth spending. The hub
|
|
exposes `LastBioacousticProbeSchedule` / `UpdateBioacousticProbeSchedule`,
|
|
and `MimirRuntime` logs `mimir-bioacoustic-probe-schedule` plus per-source
|
|
probe confidence lines. `Mimir.BufferSmoke --bioacoustic-probe-scheduler-smoke`
|
|
proves the visible climb: simulated feedback raises sync confidence from
|
|
0.360 to 0.920 and frequency-response confidence from 0.558 to 0.955 over
|
|
four scheduled probes. The scheduler owns cadence and telemetry only; it
|
|
does not override the Realtek routing authority for physical emission.
|
|
The Realtek physical emission edge now exists in tooling:
|
|
`Mimir.WasapiLoopback` targets `net10.0-windows`, can list active render
|
|
endpoints, select by friendly-name substring, and play mono Float32 probes
|
|
through the selected WASAPI render endpoint. `Mimir.BufferSmoke
|
|
--targeted-bioacoustic-probe-realtk-live` couples that renderer with
|
|
Scarlett ASIO capture and scores the mic response. The first proof selected
|
|
`Speakers (Realtek(R) Audio)` and wrote
|
|
`artifacts/wasapi/targeted-bioacoustic-realtk-live/20260601-033851`; at
|
|
gain 0.75 the shotgun peak was 0.407887 and cardioid peak was 0.074494,
|
|
but 18.6 kHz remained near-zero confidence and the strongest printed band
|
|
was cardioid 3.6 kHz at 0.098 confidence. Realtek emission is proven; high
|
|
bands remain poor physical probe candidates for this route. The targeted
|
|
probe planner now defaults to a 12 kHz ceiling and biases measurable low/mid
|
|
bands so the scheduler asks questions the current speakers and mics can
|
|
plausibly answer.
|
|
The Realtek calibration runner is now scheduler-driven rather than a blind
|
|
full-spectrum sweep: `--bioacoustic-realtk-scheduled-calibration-live`
|
|
asks `MimirBioacousticProbeScheduler` for each probe, emits through
|
|
Realtek, captures Scarlett ASIO, updates residuals, and writes a CSV plus
|
|
balance plan. It is a live physical I/O diagnostic harness seeded from the
|
|
current Scarlett residual model; production still needs measured runtime
|
|
confidence state wired into the same scheduler/emitter path. The 2026-06-01
|
|
receipt
|
|
`artifacts/wasapi/realtk-scheduled-calibration-live/20260601-035354`
|
|
ran four scheduled steps at gain 1.15 and found strong usable response
|
|
near 910 Hz, 3.6 kHz, and 9.84-9.87 kHz on both Scarlett mics using local
|
|
peak-vs-percentile-floor scoring.
|
|
The scheduler now filters confidence/status to the same 120 Hz-12 kHz
|
|
schedulable window as the targeted planner, so old 18.5 kHz residual scars
|
|
can remain recorded without owning live cadence or weakest-band status.
|
|
The first live targeted probe ran through Focusrite ASIO at 192 kHz. Gain
|
|
1.0 artifact `artifacts/asio/targeted-bioacoustic-live/20260601-030836`
|
|
measured shotgun low/high tilt 20.199 dB and cardioid 15.082 dB. Gain 2.0
|
|
artifact `artifacts/asio/targeted-bioacoustic-live/20260601-030857`
|
|
doubled loopback energy but did not make 18.6 kHz usable in the physical
|
|
mics: shotgun stayed -51.475 dB vs loopback and cardioid -56.585 dB. 910 Hz
|
|
was the strongest physical probe band, with shotgun at -21.994 dB. Current
|
|
high-band acoustic path is therefore weak or absent until speaker/mic
|
|
placement, output device, or probe shape changes.
|
|
Louder follow-ups kept the signal honest: gain 4 artifact
|
|
`artifacts/asio/targeted-bioacoustic-live/20260601-031030` still did not
|
|
clip, and gain 8 artifact
|
|
`artifacts/asio/targeted-bioacoustic-live/20260601-031051` reached only
|
|
shotgun peak 0.140715 and cardioid peak 0.027774. Even there, 18.6 kHz
|
|
stayed unusable at shotgun -54.910 dB and cardioid -63.258 dB relative to
|
|
loopback. The current physical path should bias usable low/mid anchors
|
|
while treating ultrasonic/high-band probes as failed-path evidence.
|
|
Routing correction: Starfire's Windows system default output is now
|
|
Scarlett for monitoring, while Starfire Realtek is the intended
|
|
chirp-emission and loopback-measurement path. The 20260601 targeted
|
|
ASIO-output artifacts remain useful Scarlett-output diagnostics, but the
|
|
next active probe runner must explicitly target Realtek output/loopback
|
|
instead of assuming default output or Scarlett ASIO playback.
|
|
A fresh live Scarlett proof is preserved at
|
|
`calibration/bioacoustic/complex-contour-live-20260525-063229.md`:
|
|
Focusrite ASIO playback/capture measured Loopback 1 -> Loopback 2 at
|
|
-0.014 us with corrected confidence 0.971. The apparent `asio-ch0`
|
|
near-seed/cardioid estimate is invalidated: signal inspection shows flat
|
|
low-level noise with zero-lag loopback correlation 0.000032, so the tracker
|
|
fit noise under seeded timing pressure. `asio-ch1` remains only a weak,
|
|
non-authoritative input witness. The next cut is signal-presence gating,
|
|
independent path truth, and better frequency/direct-path discrimination.
|
|
`calibration/bioacoustic/complex-contour-live-20260528-002313.md` keeps
|
|
the loopback proof healthy at -0.062 us with 0.956 confidence but finds
|
|
zero contour hits on both physical inputs, so the current physical mic
|
|
proof is blocked on input signal/gain/routing/source placement before
|
|
receiver logic changes.
|
|
After the mics were armed, `complex-contour-live-20260528-080604.md`
|
|
proved both physical inputs were alive but was routed to headphones by
|
|
mistake. The monitor-routed
|
|
`complex-contour-live-20260528-080727.md` is the first credible fresh
|
|
physical contour proof: loopback is -0.014 us at 0.971 confidence,
|
|
`asio-ch0` lands -0.410 us from its stored path seed at 0.524 confidence,
|
|
and `asio-ch1` lands -2.467 us at 0.363 confidence. Physical mapping:
|
|
`asio-ch0` is the shotgun, about 1.5 m from the right monitor and 3.0 m
|
|
from the left; `asio-ch1` is the cardioid, about 0.5 m from the left
|
|
monitor and 1.5 m from the right. Repeat captures are needed before
|
|
channel-model updates or actuator authority.
|
|
`complex-contour-monitor-split-20260528-081813.md` added ASIO
|
|
`--play-output-channel` and captured output 0/output 1 separately.
|
|
Loopback channels prove the ASIO output buffers are distinct, but
|
|
physical mic delays differ by only about 20.885 us for the shotgun and
|
|
24.789 us for the cardioid across outputs. This does not match the
|
|
declared monitor geometry; verify downstream monitor routing/isolation
|
|
before learning left/right acoustic paths.
|
|
`calibration/bioacoustic/complex-contour-replay-panel.json` persists the
|
|
current four-case replay receipt.
|
|
Acoustic robustness remains open.
|
|
The old arbitrary-chirplet matcher is a diagnostic/reference artifact only,
|
|
not the runtime hot path.
|
|
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` are the live
|
|
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
|
|
texture capture.
|
|
- `native/reservoir` owns the lower shared-edge rolling-buffer invariant for
|
|
Fensalir/Faust integration.
|
|
- `MimirFensalirFieldLowering` now emits `AquariumFieldResourceDeclaration`
|
|
rows for live payload views. Observation claims reference
|
|
`mimir:resource:*` keys, and Fensalir validation/planning can reject or
|
|
defer packets whose resource declaration is missing, CPU-only, duplicated,
|
|
or backend-incompatible. Audio spectrum/waveform surface intents now use
|
|
the source observation resource key and plan as `TubeField` packets when
|
|
the resource is compatible. `Mimir.BufferSmoke
|
|
--fensalir-field-evidence-smoke` proves one declared resource, three
|
|
claims, one planned resource-backed `TubeField` packet, two deferred
|
|
non-backend claims, and no validation errors.
|
|
`--fensalir-field-dsl-resource-smoke` proves Fensalir's field evidence DSL
|
|
can bind a declared resource directly and produce one planned `TubeField`
|
|
packet with no deferred requests. `MimirFensalirFieldLowering` also owns
|
|
the current camera observation bridge: latest video buffers lower into
|
|
FieldEvidence camera claims, GPU/native video handles declare shared
|
|
Texture2D resources, and metadata-only cadence frames do not create camera
|
|
surface intents or fake payload requests. The old direct
|
|
`AquariumGpuSensorFrame` builder is no longer the active proof path;
|
|
Fensalir can now resolve imported shared Texture2D resources by native D3D12
|
|
handle and accepts Mimir video format names such as Bgra8, Gray8, Rg8,
|
|
Nv12, and LeapStereoIr. The preferred live camera path is now
|
|
Fensalir-owned texture leasing: Mimir receives `AquariumRuntimeServices`,
|
|
asks the engine field resource broker for a keyed Texture2D lease, writes
|
|
decoded frames into the returned shared D3D12 texture, signals the returned
|
|
producer fence, and commits that fence value before the resource key reaches
|
|
shader sampling. `MimirVideoCaptureDriverSource` forwards that lease client
|
|
to direct camera drivers that implement `IMimirFensalirTextureLeaseReceiver`.
|
|
Camera backends must use the closest-to-device path available and report
|
|
unavoidable copies; process or managed convenience layers are diagnostics,
|
|
not production camera hot-path owners.
|
|
Raw single-plane system-memory frames use the explicit fallback upload lane:
|
|
`MimirVideoCaptureDriverSource` uploads the bytes into the Fensalir lease,
|
|
drops the managed payload from the live sample, and increments
|
|
`UnavoidableCopyCount`. NV12 CPU upload is rejected until Fensalir has a
|
|
real planar upload path; device/GPU NV12 producers should write the leased
|
|
texture directly.
|
|
`MimirKsVideoCaptureDriver` plus
|
|
`native/camera_capture/mimir_camera_capture.dll` is the first
|
|
production-shaped KS/UVC direct camera path: it opens a Kernel Streaming pin
|
|
in process, queues uncompressed UVC frames, and feeds the same driver source
|
|
and Fensalir lease upload path. MJPG/H264 remain future device/GPU decode
|
|
producer work.
|
|
`MimirPs3EyeVideoCaptureDriver` plus
|
|
`native/camera_capture/mimir_ps3eye_capture.dll` owns the equivalent raw
|
|
WinUSB/libusb PS3 Eye path and emits Bayer8 frames through the same upload
|
|
and copy-count machinery.
|
|
Fresh direct-driver smokes uploaded real frames for LeapUVC 640x240 YUY2,
|
|
PS3 Eye 0/1 320x240 Bayer8, Kiyo Pro 1920x1080 YUY2, and regular Kiyo
|
|
640x480 YUY2. Regular Kiyo 1280x720 YUY2 did not open; compressed
|
|
Kiyo Pro MJPG/H264 now have a first GPU-resident producer:
|
|
`MimirMediaFoundationGpuVideoCaptureDriver` uses
|
|
`native/camera_capture/mimir_mf_gpu_capture.dll` to run Media Foundation
|
|
SourceReader with a D3D11 device manager, decode MJPG/H264 on the GPU, copy
|
|
the decoded GPU surface into a shared BGRA D3D11 texture, and publish a
|
|
`shared-d3d11-texture` handle with zero live CPU payload bytes. The MF GPU
|
|
producer now also creates a shared D3D11 fence, signals it after the
|
|
texture copy, and carries the fence handle/value through
|
|
`MimirVideoFrameDescriptor`, `MimirBridgePayloadView`, and the Fensalir
|
|
field resource declaration so D3D12 can wait before sampling. Kiyo Pro
|
|
1920x1080 MJPG->RGB32 and H264->RGB32 smokes passed with nonzero shared
|
|
texture and producer fence handles. Direct shared NV12
|
|
texture creation failed in D3D11, so NV12 remains future plane-aware interop
|
|
or GPU bridge-copy work.
|
|
`Mimir.BufferSmoke --fensalir-camera-observation-smoke`
|
|
verifies the split, and `--fensalir-texture-lease-smoke` verifies the
|
|
engine-owned lease path. `config/mimir-runtime.perfect-machine.local.json`
|
|
is the current local all-sensor runtime profile, and
|
|
`--perfect-machine-live-smoke` proves the five camera texture leases plus
|
|
Leap depth/page/mesh sockets are present in one live frame. Camera image
|
|
claims now select Fensalir's D3D12 `GpuSensorFusion` backend; Leap, both
|
|
PS3 Eyes, Kiyo Pro, and regular Kiyo dispatch today, with YUY2 decoded in
|
|
compute from the declared packed format. Fensalir now owns the first in-process
|
|
D3D12 field resource registry: declared shared structured/curve buffers
|
|
import/alias GPU-resident resources by handle, while Fensalir-owned
|
|
resources allocate GPU slots only when Fensalir is the producer.
|
|
Mimir-declared rendering buffers are therefore Fensalir GPU resources in
|
|
the same runtime, not cross-process payload truth and not CPU payloads to
|
|
copy back and forth. The TubeField DSL can now express a 2D rolling float
|
|
buffer as Catmull-Rom XY tubes with modulo column addressing, amplitude
|
|
power/normalization, radius, ramp texture path, and emission scale.
|
|
Fensalir now has the first D3D12 TubeField compute/render lowering: compute
|
|
binds the declared rolling sample buffer as read-only GPU memory, expands
|
|
endpoint-clamped Catmull-Rom tube envelopes into fixed-budget vertex/index
|
|
UAVs, honors declared float sample stride with byte-address loads, records
|
|
one draw batch per lowering, then the TubeField pixel shader binds the
|
|
matching source buffer and resamples material per pixel by column+spline
|
|
coordinate with the cheap 5-tap approximation. Texture2D local assets are
|
|
now first-class Fensalir field resources: the DSL can declare a local PNG
|
|
ramp resource, TubeField `ramp=` aliases bind automatically by resource
|
|
key, and D3D12 uploads/binds the texture SRV with a linear-clamp sampler.
|
|
Missing ramps fall back to a Fensalir-owned 0-1 linear texture. The first
|
|
material is opaque field geometry with squared normalized emission, fake
|
|
Fresnel center-bright/edge-dark shaping, and geometry-budget overflow
|
|
reporting. TubeField compute now emits GPU-resident D3D12 indirect draw
|
|
argument packets, and render executes those packets after binding each
|
|
batch's source/ramp material state. SurfacePage resources now have the first Fensalir
|
|
resolver: engine users can declare GPU-resident 2D pages with dimensions,
|
|
format, version, and optional local source, and Fensalir allocates/binds
|
|
them as shader-readable page textures for future Height/SDF/material
|
|
lowerings. VolumeTexture resources can now be declared and resolved as
|
|
GPU-resident shader-readable 3D textures for future density/extinction/SDF3D
|
|
lowerings. Mesh resources now package friendly vertex/index ownership:
|
|
`Mesh.Vertices` and `Mesh.Indices` describe the GPU buffers, plus topology,
|
|
index format, bounds, submesh count, and version. The Fensalir DSL can
|
|
declare those mesh packages, and the D3D12 registry allocates/reuses the
|
|
backing vertex/index buffers under the mesh resource key. The DSL can also
|
|
create generic resource-backed claims, so mesh/page/volume resources can
|
|
reach backend packet planning. Mesh draw/material lowerings remain future
|
|
engine work. Mesh layout authority is split: imported/user-facing meshes use
|
|
the standard `PositionNormalUvColor` vertex layout, while generated meshes
|
|
can be `PipelinePrivate` and let the lowering own the minimal bytes.
|
|
TubeField now renders through the first explicit pipeline-private generated
|
|
mesh binding: compute owns the private vertex/index/indirect layout, and
|
|
render binds that generated mesh before applying TubeField material state.
|
|
TubeField writes stable field ids, real tube normals, coverage/confidence,
|
|
and domain-validity guide data into the scene metadata/control/reservoir
|
|
guide targets consumed by reservoir resolve for spatiotemporal
|
|
reconstruction. The shared field candidate inlet now keeps four fixed slots
|
|
per pixel, so producer-specific paths compete inside a resolver-owned budget
|
|
instead of choosing private winners. Reservoir history update is now a
|
|
shared Fensalir compute pass over a ping-ponged structured buffer with the
|
|
same four rows per pixel; it emits the resolved HDR field texture that
|
|
bloom and presentation consume. The presentation shader is no longer a
|
|
hidden history/TAA owner. TubeField reservoir input has moved from
|
|
generated segment packets to rolling-buffer column packets: expansion emits
|
|
one GPU-resident column record per logical history/source column, tile
|
|
binning admits those columns, and ReSTIR candidate evaluation samples the
|
|
Catmull-Rom tube SDF/material directly from the source buffer. Generated
|
|
mesh draw args are zeroed for the live path and remain only a diagnostic
|
|
reference.
|
|
The generated-mesh DrawIndexed indirect command signature is shared ABI now;
|
|
TubeField only owns the argument buffer it emits. TubeField expansion is now
|
|
gated by the selected backend packet plan, so lowering metadata cannot
|
|
bypass validation/planning authority. Validation also rejects TubeSpline
|
|
lowering metadata if its claim is not Tube-encoded or if its resource does
|
|
not match the claim payload. Mimir's typed surface-intent lowering now emits
|
|
TubeSpline lowering metadata for audio spectrum/waveform Tube claims, so the
|
|
generated mesh path is fed by real Mimir evidence rather than only the DSL
|
|
smoke. The legacy direct `AquariumSplineFrame` spectrum dashboard is silent:
|
|
`MimirRuntime` submits an empty spline frame and the old direct spectrum
|
|
spline builder has been deleted. Live spectrum samples are now normalized
|
|
into a row-major Float32 matrix, declared as
|
|
`mimir:resource:spectrum:field-upload`, uploaded into a Fensalir-owned
|
|
GPU-resident structured buffer, and consumed by the TubeField lowering.
|
|
`Mimir.BufferSmoke --mimir-spectrum-upload-smoke` verifies the live
|
|
`MimirRuntime` frame path emits that Float32 upload and planned TubeField
|
|
packet while direct spline and buffer-field dashboard inputs remain empty.
|
|
The runtime spectrum path also declares the local blackbody PNG as a
|
|
GPU-resident Texture2D field resource and binds the TubeField material by
|
|
`RampResourceKey`.
|
|
Runtime spectrum columns now represent flattened `(history age, source
|
|
lane)` pairs. Mimir emits one TubeField claim/lowering per active source.
|
|
The spectrum resource advertises fixed history and source-lane capacities
|
|
so frame content versions and source topology changes do not resize the GPU
|
|
buffer while the rolling trail fills. `MIMIR_SPECTRUM_SOURCE_LANES` sets
|
|
the lane capacity; surplus sources are truncated and reported in the
|
|
runtime UI. Lowerings use that capacity as `ColumnStride` through the
|
|
shared Float32 resource so each active lane renders a true age trail at
|
|
`z = age * 0.1`. Physical history slots are addressed as a ring with
|
|
`RollingOffset`, so logical age no longer requires moving every column in
|
|
the upload matrix before shader sampling. Mimir emits one Float32 resource
|
|
upload for the newest spectrum ring slot with an explicit element offset
|
|
instead of uploading the full fixed-capacity backing buffer every frame.
|
|
Fensalir owns rolling-slot validity and clamps TubeField dispatch so invalid
|
|
older slots are not sampled after renderer allocation, reset, or partial
|
|
update.
|
|
`MIMIR_SPECTRUM_TUBE_SUBDIVISIONS` controls the Mimir-requested Catmull-Rom
|
|
subdivisions so Fensalir's TubeField truncation report has a direct cost
|
|
lever. Mimir's active proof path no longer uses the direct
|
|
`AquariumAcousticFieldFrame` builder: sync states lower into FieldEvidence
|
|
calibration constraints. Fensalir now selects AudioPath Phase/Confidence
|
|
claims as DebugOverlay backend packets, so acoustic calibration evidence is
|
|
planned and inspectable through the FieldEvidence authority. Acoustic source
|
|
candidates and confidence volumes remain Phase 5 fusion work. The first
|
|
synthetic acoustic source candidate proof now lowers an SRP/PHAT-style
|
|
localization result with calibration/source identity and a world-space
|
|
confidence envelope, and Fensalir plans it as a FieldEvidence DebugOverlay
|
|
packet. Live room geometry still needs an explicit measured coordinate
|
|
frame before those candidates become runtime truth. Phase 5's synthetic
|
|
marker proof now lowers a deterministic multi-camera marker candidate with
|
|
calibration/marker identity and world-space support, and Fensalir plans it
|
|
as DebugOverlay evidence while ambiguous raw camera features still defer.
|
|
- `config/perfect-machine.example.json` declares live owners, sensors,
|
|
reservoir views, and runtime policy.
|
|
- `src/Mimir.Runtime/Synchronization` now includes the Perfect Machine module
|
|
library distilled from the research notes: node profiles, decoder/language
|
|
profiles, path learning, benchmark panels, actuator strategies, capture and
|
|
ingest strategies, reservoir strategies, field profiles, compute offload,
|
|
OBS publication, CultMesh contracts, Fensalir lowering, and exportable
|
|
manifest generation.
|
|
- `docs/perfect-machine-module-library.md` indexes those reusable runtime
|
|
modules and their smoke commands.
|
|
- `docs/code-algorithm-map.md` is the source-level ownership and algorithm map
|
|
for the current repo code surface.
|
|
- `docs/perfect-machine-domain-index.md` indexes Mimir's full problem domain,
|
|
current evidence taxonomy, hot paths, and implementation cuts.
|
|
- `research/perfect-machine-study-2026-05-23/` preserves the architecture
|
|
rumination, optimization ledger, research references, source/hotspot audits,
|
|
Fensalir/native boundary maps, benchmark plan, calibration-session spec,
|
|
distributed receiver spec, failure-mode ledger, data dictionary, low-level
|
|
implementation notes, and sample code sketches for streaming chirp-bin
|
|
decode, AVX2/GPU scoring, batched native ABI, fractional-delay/SRO
|
|
actuation, acoustic source localization, Mimir-to-Fensalir lowering, and
|
|
native capture rings. The corrected deep-study pass ran from
|
|
`2026-05-23T23:54:42.9041588+01:00` through at least
|
|
`2026-05-24T01:54:56.9747630+01:00`; use
|
|
`research/perfect-machine-study-2026-05-23/reading-guide.md` as the
|
|
re-entry index.
|
|
- `.voidbot` state is part of repo memory and must be committed separately
|
|
whenever dirty.
|
|
|
|
active_subgoals:
|
|
- Make the live Mimir editor behave like an OBS-style 2D compositor for
|
|
synced video layers and audio stems.
|
|
- Add calibrated projection constants and a global residual/calibration owner
|
|
that can consume Leap point-cloud errors before PS3 Eyes, Kiyos, and Eve
|
|
feed wider online calibration.
|
|
- Add remaining local camera drivers through `IMimirVideoCaptureDriver`.
|
|
- Use the verified in-process Focusrite ASIO source for local Scarlett mic and
|
|
loopback ingest; WASAPI remains a diagnostic witness.
|
|
- Add network audio feed producers that enter the same runtime clock/rebuffer
|
|
machinery instead of becoming independent authorities.
|
|
- Bind Fensalir UI to stream health, buffer depth, source timestamps,
|
|
settings, and output management.
|
|
- Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
|
|
replaces them.
|
|
|
|
current_status:
|
|
summary: >
|
|
The repo owns a C# Mimir app/runtime scaffold, a native reservoir crate, OBS
|
|
bridge scripts, and persistence surfaces. Script-era packages, tests,
|
|
helper tools, dependency manifests, and stale docs have been removed from
|
|
the working tree. Research archives remain as reference-only memory for raw
|
|
native/GPU implementation shape. A raw PS3 Eye libusb/WinUSB cadence probe
|
|
measured both PS3 Eyes simultaneously at about 183 fps each at 320x240 and
|
|
57-59 fps each at 640x480. A native Kernel Streaming LeapUVC probe can pull
|
|
frames through usbvideo.sys without Media Foundation, DirectShow, OpenCV, or
|
|
frame processing; after discarding a two-second warm-up, the useful 640x240
|
|
stereo IR mode advertises 115 fps and currently measures about 110.8 fps
|
|
with eight queued asynchronous KS reads. The half-height 640x120 mode
|
|
advertises 214 fps and currently measures about 205.9 fps. Plain Kiyo is
|
|
usable through KS when manual exposure is forced shorter: 1080p MJPG reaches
|
|
about 30 fps and 720p MJPG reaches about 51.5 fps. Kiyo Pro advertises 60
|
|
fps MJPG at 720p/1080p but currently delivers only about 25 fps through the
|
|
same KS path even with manual exposure tuning. Standard KS low-light
|
|
compensation off does not restore 60 fps, powerline 60Hz only raises Pro
|
|
cadence to about 29 fps, and standard extended Video HDR is not exposed on
|
|
this KS path. The controls-only probe now parses USB VideoControl
|
|
descriptors and found the Kiyo Pro's two vendor extension units. One
|
|
extension unit answers writable KS selectors. Moving the Pro directly to a
|
|
motherboard USB3 port changed the path to `root_hub30`, `bcdUSB=0x0320`,
|
|
and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still
|
|
reports `UsbHighSpeed` rather than `UsbSuperSpeed`, and baseline cadence is
|
|
still about 25 fps across every tested format. The first simultaneous local
|
|
camera pull only had three visible devices: LeapUVC, Kiyo Pro, and one PS3
|
|
Eye. The KS multi-stream harness plus raw PS3 Eye probe pulled all three at
|
|
once, but Leap cadence collapsed under shared load: with PS3 Eye at
|
|
640x480@60 the set delivered Leap 40.12 fps, Kiyo Pro 24.99 fps, and PS3
|
|
Eye 56.06 fps; with PS3 Eye at 320x240@187 it delivered Leap 81.10 fps,
|
|
Kiyo Pro 25.33 fps, and PS3 Eye 185.74 fps. After port swapping, five
|
|
camera devices / six optical sensors were visible and pulled at once:
|
|
LeapUVC's packed stereo IR pair, Kiyo Pro, regular Kiyo, and two PS3 Eyes.
|
|
With both Eyes at 640x480@60, the set delivered Leap stereo 57.33 fps,
|
|
Kiyo Pro 24.97 fps, Kiyo 30.30 fps, PS3 Eye 0 59.59 fps, and PS3 Eye 1
|
|
57.60 fps. With both Eyes at 320x240@187, it delivered Leap stereo 81.05
|
|
fps, Kiyo Pro 24.97 fps, Kiyo 30.28 fps, PS3 Eye 0 187.60 fps, and PS3
|
|
Eye 1 187.03 fps. Kiyo Pro still negotiates high-speed USB, and Leap
|
|
remains topology-sensitive under shared load. A follow-up moved Leap alone
|
|
onto the blue rear USB3 group/root_hub30#5 while both Kiyos shared
|
|
root_hub30#7&18eea21b and one visible PS3 Eye used root_hub30#7&124f193f;
|
|
Leap still delivered only 80.99 fps with the other visible cameras active,
|
|
so the remaining loss is not explained by same-root-hub contention alone.
|
|
Splitting Kiyo Pro and one PS3 Eye to Raven left the Mimir host with
|
|
LeapUVC, regular Kiyo, and one PS3 Eye, but local cadence still stayed low:
|
|
Leap delivered 81.34 fps with Kiyo+Eye, 82.77 fps with Kiyo only, and only
|
|
84.18 fps in a same-session Leap-only baseline. The previous ~110.8 fps
|
|
Leap result was not reproduced until the Focusrite Scarlett Solo was moved
|
|
away from the Leap USB neighborhood. After that, Leap-only baseline rose to
|
|
95.48 fps, best full-height scenario to 101.67 fps, and the local split set
|
|
delivered Leap 105.10 fps, regular Kiyo 30.17 fps, and one PS3 Eye 185.81
|
|
fps. The split-host layout is viable if the Leap's USB neighborhood stays
|
|
free of the Focusrite and other heavy devices. Raven's PS3 Eye raw interface
|
|
now has the same libwdi WinUSB binding as the local Eye and can be opened by
|
|
the copied cadence probe, but Raven Eye cadence is currently unusable:
|
|
320x240@187 delivered 132.24 fps then 78.93 fps, and 640x480@60 delivered
|
|
6.32 fps with multi-second gaps. Moving that Eye back to the Mimir host
|
|
produced the current clean local topology: LeapUVC, regular Kiyo, and two
|
|
PS3 Eyes run in parallel while Kiyo Pro stays off-host. With both Eyes at
|
|
320x240@187, Leap delivered 106.07 fps, Kiyo 30.33 fps, and the Eyes 186.57
|
|
/ 186.14 fps. With both Eyes at 640x480@60, Leap delivered 108.07 fps, Kiyo
|
|
30.34 fps, and the Eyes 57.79 / 58.08 fps. Moving Kiyo Pro back to the
|
|
Mimir host while regular Kiyo was absent kept Leap healthy: with both Eyes
|
|
at 320x240@187, Leap delivered 110.80 fps, Kiyo Pro 24.97 fps, and the Eyes
|
|
187.46 / 186.89 fps; with both Eyes at 640x480@60, Leap delivered 110.72
|
|
fps, Kiyo Pro 24.87 fps, and the Eyes 59.91 / 59.19 fps. Kiyo Pro still
|
|
negotiates UsbHighSpeed(2), but it does not poison Leap cadence in the
|
|
current layout. Moving regular Kiyo from front panel to a rear port made
|
|
all five camera devices / six optical sensors visible locally. With both
|
|
Eyes at 320x240@187, the full set delivered Leap 110.84 fps, Kiyo Pro 25.08
|
|
fps, Kiyo 30.18 fps, PS3 Eye 0 187.59 fps, and PS3 Eye 1 187.02 fps. With
|
|
both Eyes at 640x480@60, it delivered Leap 110.76 fps, Kiyo Pro 25.14 fps,
|
|
Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
|
|
the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
|
|
next_action: >
|
|
Continue the Leap-first visual geometry cut from the verified surface-claim
|
|
path. Mimir lowers live `LeapStereoIr` packed R/G stereo into a disparity
|
|
SurfacePage and declares a derived point-cloud Mesh socket. Fensalir now
|
|
generates Leap surface splats from that disparity, feeds them into the
|
|
native splat/reservoir resolver, and presents visible Leap-derived points
|
|
without the diagnostic point-list draw path. The reservoir-history pass now
|
|
starts through a dedicated lean current-sample shader; temporal/spatial
|
|
reuse remains the next reservoir shader cut. Next: replace provisional
|
|
projection display fit with calibrated Leap intrinsics/extrinsics, restore
|
|
temporal reservoir reuse in the split shader, then add a global
|
|
residual/calibration owner that can consume point-cloud alignment error. Do
|
|
not fan RGB cameras into fake depth: PS3 Eyes are fast
|
|
motion/feature witnesses, Kiyos and Eve are RGB/context witnesses, and Leap
|
|
owns the first dense close-range geometry. Keep the Focusrite away from the
|
|
Leap USB neighborhood. Treat Kiyo Pro as a ~25 fps source until its
|
|
SuperSpeed negotiation is fixed.
|