Prove all-sensor GPU runtime path
This commit is contained in:
parent
95ba849af4
commit
1269d42520
8 changed files with 227 additions and 4 deletions
BIN
artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png
Normal file
BIN
artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
85
config/mimir-runtime.perfect-machine.local.json
Normal file
85
config/mimir-runtime.perfect-machine.local.json
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
{
|
||||
"bufferSeconds": 5.0,
|
||||
"audioSync": {
|
||||
"mode": "passive",
|
||||
"referenceSourceId": "loopback-scarlett-speakers",
|
||||
"calibrationGain": 0.0,
|
||||
"watermarkGain": 0.0
|
||||
},
|
||||
"streams": [
|
||||
{
|
||||
"sourceId": "leap-stereo-ir",
|
||||
"displayName": "LeapUVC stereo IR packed R/G",
|
||||
"kind": "Video",
|
||||
"origin": "LocalDevice",
|
||||
"enabled": true,
|
||||
"adapter": "ks-camera",
|
||||
"command": "../native/camera_capture/build/Release/mimir_camera_capture.dll",
|
||||
"pathNeedle": "vid_f182&pid_0003&mi_00",
|
||||
"width": 640,
|
||||
"height": 240,
|
||||
"inputFormat": "YUY2",
|
||||
"pixelFormat": "LeapStereoIr",
|
||||
"minimumFramesPerSecond": 100.0,
|
||||
"queueDepth": 8
|
||||
},
|
||||
{
|
||||
"sourceId": "ps3-eye-0",
|
||||
"displayName": "PS3 Eye 0 high-rate tracking",
|
||||
"kind": "Video",
|
||||
"origin": "LocalDevice",
|
||||
"enabled": true,
|
||||
"adapter": "ps3eye",
|
||||
"command": "../native/camera_capture/build/Release/mimir_ps3eye_capture.dll",
|
||||
"cameraIndex": 0,
|
||||
"width": 320,
|
||||
"height": 240,
|
||||
"framesPerSecond": 187
|
||||
},
|
||||
{
|
||||
"sourceId": "ps3-eye-1",
|
||||
"displayName": "PS3 Eye 1 high-rate tracking",
|
||||
"kind": "Video",
|
||||
"origin": "LocalDevice",
|
||||
"enabled": true,
|
||||
"adapter": "ps3eye",
|
||||
"command": "../native/camera_capture/build/Release/mimir_ps3eye_capture.dll",
|
||||
"cameraIndex": 1,
|
||||
"width": 320,
|
||||
"height": 240,
|
||||
"framesPerSecond": 187
|
||||
},
|
||||
{
|
||||
"sourceId": "kiyo-pro-rgb",
|
||||
"displayName": "Kiyo Pro RGB context",
|
||||
"kind": "Video",
|
||||
"origin": "LocalDevice",
|
||||
"enabled": true,
|
||||
"adapter": "ks-camera",
|
||||
"command": "../native/camera_capture/build/Release/mimir_camera_capture.dll",
|
||||
"pathNeedle": "vid_1532&pid_0e05&mi_00",
|
||||
"width": 640,
|
||||
"height": 480,
|
||||
"inputFormat": "YUY2",
|
||||
"pixelFormat": "YUY2",
|
||||
"minimumFramesPerSecond": 25.0,
|
||||
"queueDepth": 8
|
||||
},
|
||||
{
|
||||
"sourceId": "kiyo-basic-rgb",
|
||||
"displayName": "Kiyo RGB context",
|
||||
"kind": "Video",
|
||||
"origin": "LocalDevice",
|
||||
"enabled": true,
|
||||
"adapter": "ks-camera",
|
||||
"command": "../native/camera_capture/build/Release/mimir_camera_capture.dll",
|
||||
"pathNeedle": "vid_1532&pid_0e03&mi_00",
|
||||
"width": 640,
|
||||
"height": 480,
|
||||
"inputFormat": "YUY2",
|
||||
"pixelFormat": "YUY2",
|
||||
"minimumFramesPerSecond": 25.0,
|
||||
"queueDepth": 8
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -343,6 +343,15 @@ a named invariant that the native runtime cannot protect yet.
|
|||
Current direct-driver smokes prove real-frame upload for LeapUVC 640x240
|
||||
YUY2, both PS3 Eyes at 320x240 Bayer8, Kiyo Pro 1920x1080 YUY2, and regular
|
||||
Kiyo 640x480 YUY2. Regular Kiyo 1280x720 YUY2 did not open.
|
||||
`config/mimir-runtime.perfect-machine.local.json` is the current local
|
||||
all-sensor blast profile: Leap packed stereo IR, both PS3 Eyes at
|
||||
320x240@187, Kiyo Pro RGB, and regular Kiyo RGB. `Mimir.BufferSmoke
|
||||
--perfect-machine-live-smoke` opens all five configured video sources,
|
||||
attaches Fensalir texture leases, and passes only when at least five live
|
||||
camera textures, a Leap stereo-depth lowering, a disparity SurfacePage, and a
|
||||
derived point-cloud Mesh are present. It prints deferred camera texture
|
||||
Feature requests explicitly; those are not capture failures, they are the
|
||||
future Fensalir D3D12 feature-extraction cut.
|
||||
`MimirMediaFoundationGpuVideoCaptureDriver` is the first compressed camera
|
||||
path: `native/camera_capture/mimir_mf_gpu_capture.dll` uses Media Foundation
|
||||
SourceReader with a D3D11 device manager, selects MJPG/H264 camera modes,
|
||||
|
|
@ -426,6 +435,14 @@ a named invariant that the native runtime cannot protect yet.
|
|||
stable tracks at 320x240@187: with sensitive wide tracking defaults, each Eye
|
||||
returned 160 live tracks from each Eye in a one-second pass, with 160 stable
|
||||
tracks from `ps3-eye-0` and 156 stable tracks from `ps3-eye-1`.
|
||||
A full Fensalir headless proof using that all-sensor profile now runs the
|
||||
actual D3D12 host path. The 2026-05-31 run started all five video sources,
|
||||
declared eight field resources, resolved six Texture2D resources, dispatched
|
||||
one packed-Leap stereo-depth kernel, generated one point-cloud/surface stream,
|
||||
and reported 19,200 visible reservoir splats. The captured proof is
|
||||
`artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png`. Generic
|
||||
camera Feature claims still defer; live PS3/Kiyo feature extraction remains a
|
||||
Fensalir compute shader owner, not the CPU tracker.
|
||||
Phase 6 program-output receipts are also structurally complete: Fensalir
|
||||
publishes the final program surface through named shared D3D12 texture/fence
|
||||
resources, the OBS plugin consumes that surface with an explicit D3D12-to-
|
||||
|
|
|
|||
|
|
@ -40,6 +40,14 @@ flowchart TD
|
|||
now: it opens direct sensor feeds, applies KS/UVC manual exposure/gain where
|
||||
supported, scores actual luma frames, and reports the best setting per
|
||||
source. PS3 Eye and Leap-specific controls still need native control cuts.
|
||||
- The current all-sensor GPU proof is
|
||||
`config/mimir-runtime.perfect-machine.local.json` plus
|
||||
`Mimir.BufferSmoke --perfect-machine-live-smoke`: Leap, both PS3 Eyes, Kiyo
|
||||
Pro, and regular Kiyo all enter the runtime as Fensalir texture leases. The
|
||||
smoke requires five live camera Texture2D resources, one Leap stereo-depth
|
||||
lowering, one disparity SurfacePage, and one derived point-cloud Mesh.
|
||||
Deferred generic camera Feature requests are printed as the selected future
|
||||
shader work, not hidden as capture success.
|
||||
- PS3 Eyes own high-rate sparse tracking evidence before they own depth:
|
||||
`MimirSparseFeatureTracker` turns live Bayer/luma frames into stable
|
||||
sensor-local feature tracks, and `MimirFeatureTrackFieldCandidate` lowers
|
||||
|
|
|
|||
|
|
@ -546,6 +546,15 @@ WinUSB/libusb capture, emitting Bayer8 frames through the common upload lane.
|
|||
Driver smokes have pulled and uploaded real frames for LeapUVC 640x240 YUY2,
|
||||
both PS3 Eyes at 320x240 Bayer8, Kiyo Pro 1920x1080 YUY2, and regular Kiyo
|
||||
640x480 YUY2. Regular Kiyo 1280x720 YUY2 did not open.
|
||||
`config/mimir-runtime.perfect-machine.local.json` is the current local
|
||||
all-sensor runtime profile: Leap packed stereo IR, both PS3 Eyes at
|
||||
320x240@187, Kiyo Pro RGB, and regular Kiyo RGB. `Mimir.BufferSmoke
|
||||
--perfect-machine-live-smoke` opens all five video sources through the runtime,
|
||||
attaches Fensalir texture leases, and requires five camera Texture2D resources,
|
||||
one Leap stereo-depth lowering, one disparity SurfacePage, and one derived
|
||||
point-cloud Mesh. It prints the remaining deferred generic camera Feature
|
||||
claims so the gap stays visible: PS3/Kiyo/Leap texture feature extraction is
|
||||
still a Fensalir D3D12 compute owner, not a CPU tracker hidden in the hot path.
|
||||
`MimirMediaFoundationGpuVideoCaptureDriver` plus
|
||||
`native/camera_capture/mimir_mf_gpu_capture.dll` is the compressed camera path:
|
||||
Media Foundation SourceReader runs with a D3D11 device manager, decodes MJPG or
|
||||
|
|
@ -645,7 +654,13 @@ camera Feature claims through Fensalir field evidence. A one-second live pass
|
|||
at 320x240@187 with sensitive defaults returned 160 live tracks from each Eye:
|
||||
160 stable tracks from `ps3-eye-0` and 156 stable tracks from `ps3-eye-1`.
|
||||
This is a CPU diagnostic owner; the hot production path remains Fensalir D3D12
|
||||
feature extraction. Once the frustum fit is coherent, detected
|
||||
feature extraction. A full 2026-05-31 Fensalir headless run using the
|
||||
all-sensor profile started all five video sources, declared eight field
|
||||
resources, resolved six Texture2D resources, dispatched one packed-Leap
|
||||
stereo-depth kernel, generated one point-cloud/surface stream, reported 19,200
|
||||
visible reservoir splats, and captured
|
||||
`artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png`. Once the
|
||||
frustum fit is coherent, detected
|
||||
surface candidates can be sampled across every synchronized view before
|
||||
Fensalir resolves them into 4D Gaussian/surface splat claims.
|
||||
|
||||
|
|
|
|||
|
|
@ -120,6 +120,11 @@ if (args.Any(arg => string.Equals(arg, "--perfect-machine-profile-smoke", String
|
|||
return RunPerfectMachineProfileSmoke();
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--perfect-machine-live-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunPerfectMachineLiveSmoke(args);
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--eve-program-output-contract-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunEveProgramOutputContractSmoke();
|
||||
|
|
@ -3695,6 +3700,79 @@ static int RunPerfectMachineProfileSmoke()
|
|||
: 1;
|
||||
}
|
||||
|
||||
static int RunPerfectMachineLiveSmoke(string[] args)
|
||||
{
|
||||
var configPath = Path.GetFullPath(ParseStringOption(
|
||||
args,
|
||||
"--config",
|
||||
"config/mimir-runtime.perfect-machine.local.json"));
|
||||
var seconds = Math.Max(0.25, ParseDoubleOption(args, "--seconds", 1.5));
|
||||
var previousConfig = Environment.GetEnvironmentVariable("MIMIR_RUNTIME_CONFIG");
|
||||
Environment.SetEnvironmentVariable("MIMIR_RUNTIME_CONFIG", configPath);
|
||||
try
|
||||
{
|
||||
var configuration = MimirRuntimeConfiguration.Load();
|
||||
var broker = new FakeFieldResourceBroker();
|
||||
using var runtime = new MimirRuntime(
|
||||
new AquariumRuntimeOptions(Headless: true, CultCachePath: null),
|
||||
configuration);
|
||||
runtime.AttachServices(new AquariumRuntimeServices(broker));
|
||||
runtime.OnSceneReady();
|
||||
var input = new InputState();
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
var updates = 0;
|
||||
while (stopwatch.Elapsed.TotalSeconds < seconds)
|
||||
{
|
||||
runtime.Update(1.0f / 60.0f, input);
|
||||
updates++;
|
||||
Thread.Sleep(1);
|
||||
}
|
||||
|
||||
var field = runtime.Frame.Scene.FieldEvidenceFrame;
|
||||
var validation = AquariumFieldEvidenceValidator.Validate(field);
|
||||
var plan = AquariumFieldLoweringPlanner.Plan(field);
|
||||
var videoResources = field.Resources
|
||||
.Where(static resource =>
|
||||
resource.Kind == AquariumFieldResourceKind.Texture2D &&
|
||||
resource.Residency is AquariumFieldResourceResidency.SharedGpu or AquariumFieldResourceResidency.GpuResident)
|
||||
.ToArray();
|
||||
var cameraTextures = videoResources
|
||||
.Where(static resource => resource.ResourceKey.StartsWith("mimir:resource:camera:", StringComparison.Ordinal))
|
||||
.ToArray();
|
||||
var surfacePages = field.Resources.Count(static resource => resource.Kind == AquariumFieldResourceKind.SurfacePage);
|
||||
var meshResources = field.Resources.Count(static resource => resource.Kind == AquariumFieldResourceKind.Mesh);
|
||||
var stereoLowerings = field.StereoDepthLowerings.Count;
|
||||
var resourceClaims = field.Claims.Count(static claim =>
|
||||
claim.Encoding is AquariumFieldEncoding.Radiance or AquariumFieldEncoding.Feature or AquariumFieldEncoding.Height or AquariumFieldEncoding.Mesh);
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-live-smoke config={configPath} updates={updates} resources={field.Resources.Count} gpuTextures={videoResources.Length} cameraTextures={cameraTextures.Length} claims={field.Claims.Count} resourceClaims={resourceClaims} stereoLowerings={stereoLowerings} meshResources={meshResources} surfacePages={surfacePages} planned={plan.Packets.Count} deferred={plan.DeferredRequests.Count} uploaded={broker.UploadedResourceKey} committed={broker.CommittedResourceKey} errors={validation.HasErrors}");
|
||||
foreach (var resource in videoResources.OrderBy(static resource => resource.ResourceKey, StringComparer.Ordinal))
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-resource key={resource.ResourceKey} {resource.Width}x{resource.Height} format={resource.Format} residency={resource.Residency} handle={resource.NativeHandle} kind={resource.NativeHandleKind} version={resource.Version}");
|
||||
}
|
||||
|
||||
foreach (var deferred in plan.DeferredRequests.OrderBy(static request => request.RequestKey, StringComparer.Ordinal))
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-deferred request={deferred.RequestKey} claim={deferred.ClaimKey} domain={deferred.DomainKey} encoding={deferred.Encoding} proposal={deferred.ProposalKind} payload={deferred.PayloadHandle}");
|
||||
}
|
||||
|
||||
return cameraTextures.Length >= 5 &&
|
||||
stereoLowerings >= 1 &&
|
||||
meshResources >= 1 &&
|
||||
surfacePages >= 1 &&
|
||||
plan.Packets.Count >= 2 &&
|
||||
!validation.HasErrors
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
finally
|
||||
{
|
||||
Environment.SetEnvironmentVariable("MIMIR_RUNTIME_CONFIG", previousConfig);
|
||||
}
|
||||
}
|
||||
|
||||
static MimirAudioSynchronizationReport? EstimateBioacousticDelay(float[] reference, float[] candidate, int sampleRate)
|
||||
{
|
||||
const string referenceSourceId = "actuator-reference";
|
||||
|
|
|
|||
|
|
@ -258,3 +258,4 @@
|
|||
{"ts": "2026-05-31T03:05:00+01:00", "type": "verification", "status": "ok", "note": "Live LED spline sweep now runs on actual direct sensor feeds. `dotnet run --project src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj -- --led-spline-live-score --setting-seconds 0.25 --settle-ms 100 --expected-leds 38 --minimum-luma 0.55` selected exposure -9/gain 8 for `kiyo-pro-rgb` (score 0.666, 41 detections, usable) and `kiyo-basic-rgb` (score 0.650, 38 detections, usable). LeapUVC rejected generic KS exposure/gain writes and only recovered two LED components; both PS3 Eye feeds scored 0 at current fixed controls because their native wrapper does not yet expose gain/exposure."}
|
||||
{"ts": "2026-05-31T02:45:05+01:00", "type": "verification", "status": "ok", "note": "Eve dashboard now renders voidbot.swarm as a native VoidBot-style cockpit with CTB avatar URLs, selected Face/status pane, state tree, and detail panel; device build/install passed and broker smoke reported 8 CTB avatar URLs."}
|
||||
{"ts": "2026-05-31T03:22:00+01:00", "type": "verification", "status": "ok", "note": "PS3 Eye sparse tracking bootstrap is live. `--ps3eye-tracking-smoke` produced 13 synthetic tracks, 12 stable tracks, and one Feature claim. The final default `--ps3eye-tracking-live --seconds 1.0` captured both Eyes at 320x240@187 and returned 160 tracks / 160 stable tracks from `ps3-eye-0` and 160 tracks / 156 stable tracks from `ps3-eye-1`. This is CPU diagnostic tracking; production feature extraction remains a Fensalir D3D12 owner."}
|
||||
{"ts": "2026-05-31T03:31:08+01:00", "type": "verification", "status": "ok", "note": "2026-05-31 all-sensor GPU proof: config/mimir-runtime.perfect-machine.local.json opened Leap, both PS3 Eyes, Kiyo Pro, and regular Kiyo; BufferSmoke --perfect-machine-live-smoke reported 5 camera Texture2D leases, Leap stereo lowering, SurfacePage, Mesh, 7 planned packets, and 5 deferred generic camera Feature requests. Mimir.App headless D3D12 dispatched packed-Leap stereo depth, generated one point-cloud/surface stream, reported 19,200 visible reservoir splats, and captured artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png."}
|
||||
|
|
|
|||
|
|
@ -99,6 +99,21 @@ invariants:
|
|||
`AQUARIUM_DISABLE_RESERVOIR_HISTORY`: Fensalir now compiles a dedicated lean
|
||||
`D3D12ReservoirHistoryUpdate.hlsl` for current-sample history/resolved
|
||||
output instead of compiling the monolithic post shader as a compute entry.
|
||||
The 2026-05-31 all-sensor proof uses
|
||||
`config/mimir-runtime.perfect-machine.local.json`: Leap packed stereo IR,
|
||||
both PS3 Eyes at 320x240@187, Kiyo Pro RGB, and regular Kiyo RGB all open
|
||||
through `MimirRuntime` and enter as Fensalir texture leases.
|
||||
`Mimir.BufferSmoke --perfect-machine-live-smoke` now requires five live
|
||||
camera Texture2D resources, one Leap stereo-depth lowering, one disparity
|
||||
SurfacePage, and one derived point-cloud Mesh. The Fensalir headless run
|
||||
started all five sources, declared eight field resources, resolved six
|
||||
Texture2D resources, dispatched the packed-Leap stereo-depth kernel,
|
||||
generated one point-cloud/surface stream, reported 19,200 visible reservoir
|
||||
splats, and captured
|
||||
`artifacts/runtime/mimir-perfect-machine-all-sensors-headless.png`.
|
||||
Generic camera Feature claims still defer because PS3/Kiyo/Leap texture
|
||||
feature extraction belongs to a future Fensalir D3D12 compute lowering, not
|
||||
the CPU diagnostic tracker.
|
||||
Calibrated projection constants, full SGM, temporal/spatial reservoir reuse
|
||||
in a compilable shader, and global residual/calibration ownership remain
|
||||
open.
|
||||
|
|
@ -563,9 +578,13 @@ accepted_design:
|
|||
or GPU bridge-copy work.
|
||||
`Mimir.BufferSmoke --fensalir-camera-observation-smoke`
|
||||
verifies the split, and `--fensalir-texture-lease-smoke` verifies the
|
||||
engine-owned lease path. Camera image claims stay debug-view-only for now;
|
||||
production visual fusion/feature extraction lowerings are deferred until
|
||||
the renderer and audio work are fixed. Fensalir now owns the first in-process
|
||||
engine-owned lease path. `config/mimir-runtime.perfect-machine.local.json`
|
||||
is the current local all-sensor runtime profile, and
|
||||
`--perfect-machine-live-smoke` proves the five camera texture leases plus
|
||||
Leap depth/page/mesh sockets are present in one live frame. Camera image
|
||||
claims stay debug-view-only for now; production visual fusion/feature
|
||||
extraction lowerings are deferred until Fensalir owns the D3D12 feature
|
||||
shader. Fensalir now owns the first in-process
|
||||
D3D12 field resource registry: declared shared structured/curve buffers
|
||||
import/alias GPU-resident resources by handle, while Fensalir-owned
|
||||
resources allocate GPU slots only when Fensalir is the producer.
|
||||
|
|
|
|||
Loading…
Reference in a new issue