Bridge camera observations through field evidence
This commit is contained in:
Meta Vens
parent
3ab163da57
commit
ddad34f69b
8 changed files with 190 additions and 99 deletions
|
|
@ -24,16 +24,8 @@ drivers / ASIO / network feeds
|
|||
-> OBS
|
||||
```
|
||||
|
||||
But the current debug-spectrum path still exposes the pressure leak:
|
||||
|
||||
```text
|
||||
ASIO spectrum history
|
||||
-> AquariumSplineFrame
|
||||
-> direct Fensalir spline tube debug surface
|
||||
```
|
||||
|
||||
That path is allowed as a deterministic diagnostic. It is not yet the full
|
||||
Perfect Machine. The production shape is:
|
||||
The old direct debug-spectrum pressure leak has been cut. Current spectrum
|
||||
debug output is already in the production-shaped bridge:
|
||||
|
||||
```text
|
||||
rolling buffer slice + source identity + calibration + surface intent
|
||||
|
|
@ -93,8 +85,9 @@ Derived state:
|
|||
|
||||
- spectrum lines, debug views, screenshots, OBS bridge endpoints, FFmpeg/SRT
|
||||
helpers, and Eve dashboard feeds are views or diagnostics.
|
||||
- The old direct spline-spectrum dashboard is silent. FieldEvidence TubeField
|
||||
owns the spectrum surface path; screenshots and logs remain diagnostics.
|
||||
- The old direct spline-spectrum dashboard and old direct GPU sensor frame
|
||||
bridge are silent. FieldEvidence owns spectrum and camera observation
|
||||
publication; screenshots and logs remain diagnostics.
|
||||
|
||||
Forbidden writers:
|
||||
|
||||
|
|
@ -200,6 +193,11 @@ Current proof surface:
|
|||
- `Mimir.BufferSmoke --fensalir-field-dsl-resource-smoke` uses Fensalir's field
|
||||
evidence DSL to bind the declared resource directly and produce one planned
|
||||
`TubeField` packet with no deferred requests.
|
||||
- `Mimir.BufferSmoke --fensalir-camera-observation-smoke` maps one
|
||||
GPU-backed video frame and one metadata-only cadence witness into field
|
||||
evidence. Only the shared D3D12 video handle becomes a declared `Texture2D`
|
||||
resource and camera surface intent; the metadata-only frame remains
|
||||
observation evidence with no render payload.
|
||||
|
||||
Fensalir now owns the first D3D12 resource resolver cut below this contract:
|
||||
shared structured/curve buffers import/alias GPU-resident resources by handle,
|
||||
|
|
@ -250,6 +248,13 @@ can distinguish source-lane claims. Mimir exposes
|
|||
`MIMIR_SPECTRUM_SOURCE_LANES` as the source-lane capacity lever and
|
||||
`MIMIR_SPECTRUM_TUBE_SUBDIVISIONS` as the geometry cost lever for responding to
|
||||
Fensalir's requested/dispatched/truncated/invalid TubeField budget report.
|
||||
Camera descriptors now follow the same evidence/resource boundary:
|
||||
`MimirFensalirFieldLowering.BuildCameraObservationFrame` lowers latest video
|
||||
buffers into observation claims and declares GPU-backed frames as shared
|
||||
`Texture2D` resources. Metadata-only/process-cadence frames do not create
|
||||
surface intents, so a missing payload cannot become a fake `latest-window`
|
||||
render request. Camera image claims currently defer because Fensalir has not
|
||||
selected a visual-fusion/render lowering for camera textures yet.
|
||||
|
||||
### 3. Calibration Constraint
|
||||
|
||||
|
|
@ -374,9 +379,12 @@ Lower native camera descriptors into observation surfaces:
|
|||
|
||||
Acceptance:
|
||||
|
||||
- at least one local camera stream appears as Fensalir sensor observations;
|
||||
- missing frames are absence, not zero-valued evidence;
|
||||
- Fensalir owns any feature extraction result.
|
||||
- implemented bridge proof: GPU-backed video descriptors become declared
|
||||
shared `Texture2D` resources plus camera observation/surface claims;
|
||||
- metadata-only cadence descriptors remain observation evidence and do not
|
||||
become render payloads;
|
||||
- no selected camera-image lowering exists yet, so Fensalir defers those claims
|
||||
until visual fusion/feature extraction owns them.
|
||||
|
||||
### Phase 4: Audio Field Observation Bridge
|
||||
|
||||
|
|
|
|||
|
|
@ -174,6 +174,14 @@ a named invariant that the native runtime cannot protect yet.
|
|||
`Mimir.BufferSmoke --fensalir-field-dsl-resource-smoke` proves Fensalir's DSL
|
||||
evidence compiler can bind a declared resource and produce one planned
|
||||
`TubeField` packet with no deferred requests.
|
||||
- `MimirFensalirFieldLowering.BuildCameraObservationFrame` is the current
|
||||
camera bridge proof. It lowers latest video buffers into FieldEvidence
|
||||
observation claims, declares GPU/native video payloads as shared `Texture2D`
|
||||
resources, and creates camera surface intent only when a resource exists.
|
||||
Metadata-only cadence frames remain observations with empty payload handles;
|
||||
they do not become fake render requests. The old direct
|
||||
`AquariumGpuSensorFrame` builder has been removed from Mimir's proof path.
|
||||
`Mimir.BufferSmoke --fensalir-camera-observation-smoke` verifies the split.
|
||||
- Fensalir now owns the first in-process D3D12 field resource resolver cut:
|
||||
shared structured/curve buffer resources import/alias GPU-resident handles,
|
||||
and Fensalir-owned resources allocate GPU slots only when Fensalir is the
|
||||
|
|
@ -300,8 +308,8 @@ a named invariant that the native runtime cannot protect yet.
|
|||
proof, `--perfect-machine-manifest` to export the module manifest for
|
||||
tooling/UI/remote witness use, and `--perfect-machine-lowering-benchmark` to
|
||||
measure the Mimir-to-Fensalir lowering path. The current six-camera/two-audio
|
||||
synthetic lowering benchmark runs at roughly 2.5 us per iteration with about
|
||||
2.3 KB allocated per iteration.
|
||||
synthetic lowering benchmark now measures FieldEvidence camera/resource
|
||||
lowering rather than the retired direct GPU sensor DTO path.
|
||||
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
|
||||
timestamp, and native/GPU handle metadata.
|
||||
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
|
||||
|
|
@ -359,11 +367,11 @@ a named invariant that the native runtime cannot protect yet.
|
|||
volume textures, and mesh packages, and the evidence DSL can plan generic
|
||||
resource-backed claims against those resources. The next engine cuts are
|
||||
selected render lowerings that consume those resources as mesh geometry,
|
||||
height/SDF/material pages, and density/extinction/SDF3D volumes. The generic
|
||||
imported mesh ABI is no longer the blocker; TubeField now occupies the first
|
||||
generated-mesh lowering lane. The next renderer decisions are how to expose
|
||||
additional generated mesh producers and which page/volume/material lowering
|
||||
should follow.
|
||||
camera/feature textures, height/SDF/material pages, and
|
||||
density/extinction/SDF3D volumes. The generic imported mesh ABI is no longer
|
||||
the blocker; TubeField now occupies the first generated-mesh lowering lane.
|
||||
The next renderer decisions are how to expose additional generated mesh
|
||||
producers and which camera/page/volume/material lowering should follow.
|
||||
9. Move mic alignment, room suppression, voice separation, spatialization, and
|
||||
stem generation into Faust/native DSP.
|
||||
10. Keep the OBS bridge witness ledger as evidence before expanding receiver
|
||||
|
|
|
|||
|
|
@ -326,11 +326,19 @@ shader lowering. Mimir-declared rendering buffers are Fensalir GPU resources in
|
|||
the same runtime; native/shared handle metadata is an import edge, not a
|
||||
separate authority regime. Rendering-relevant buffers move to GPU residency as
|
||||
early as possible and stay there; CPU readback/tessellation is diagnostic only.
|
||||
Camera descriptors now enter this bridge too: latest video buffers lower into
|
||||
FieldEvidence camera observations, GPU/native handles declare shared
|
||||
`Texture2D` resources, and metadata-only cadence frames do not create camera
|
||||
surface intents or fake payload requests. The old direct `AquariumGpuSensorFrame`
|
||||
bridge proof has been removed from Mimir's active proof path. Camera image
|
||||
claims currently defer until Fensalir owns a selected visual-fusion lowering for
|
||||
camera textures.
|
||||
The current smokes prove planning, not visible packet rendering:
|
||||
`--fensalir-field-evidence-smoke` now produces one planned resource-backed
|
||||
`TubeField` packet from Mimir's spectrum intent, and
|
||||
`--fensalir-field-dsl-resource-smoke` proves the Fensalir DSL can bind the same
|
||||
kind of declared resource directly. Fensalir now has the first D3D12 resolver
|
||||
kind of declared resource directly. `--fensalir-camera-observation-smoke`
|
||||
proves the camera observation/resource split. Fensalir now has the first D3D12 resolver
|
||||
cut for structured/curve buffers: it imports shared GPU buffers by handle and
|
||||
allocates engine-owned GPU slots only for Fensalir-produced resources. The DSL
|
||||
can describe a 2D rolling float buffer as Catmull-Rom XY tubes with modulo
|
||||
|
|
@ -341,8 +349,9 @@ textures allocate shader-readable GPU textures, and mesh packages own
|
|||
`Mesh.Vertices`/`Mesh.Indices` GPU buffer shape under one resource key. The DSL
|
||||
can plan generic resource-backed claims over those declarations. The next
|
||||
blocker is no longer resolver ownership; it is selected render lowerings that
|
||||
consume mesh/page/volume resources as geometry, height/SDF/material pages, or
|
||||
density/extinction/SDF3D domains. Mesh layout authority is split by source:
|
||||
consume camera/feature textures, mesh/page/volume resources as geometry,
|
||||
height/SDF/material pages, or density/extinction/SDF3D domains. Mesh layout
|
||||
authority is split by source:
|
||||
imported/user meshes use the standard `PositionNormalUvColor` layout, and
|
||||
generated meshes can be `PipelinePrivate` so each lowering owns the bytes it
|
||||
emits and consumes. TubeField is the first concrete generated-mesh consumer:
|
||||
|
|
|
|||
|
|
@ -103,6 +103,11 @@ if (args.Any(arg => string.Equals(arg, "--fensalir-field-dsl-resource-smoke", St
|
|||
return RunFensalirFieldDslResourceSmoke();
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--fensalir-camera-observation-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunFensalirCameraObservationSmoke();
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--mimir-spectrum-upload-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunMimirSpectrumUploadSmoke();
|
||||
|
|
@ -627,6 +632,92 @@ tubespline id=spectrum-trail resource=spectrum domain=mimir:domain:spectrum conf
|
|||
: 1;
|
||||
}
|
||||
|
||||
static int RunFensalirCameraObservationSmoke()
|
||||
{
|
||||
var gpuBuffer = new MimirRollingStreamBuffer(
|
||||
new MimirStreamDescriptor("kiyo-pro-rgb", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice),
|
||||
TimeSpan.FromSeconds(5));
|
||||
gpuBuffer.Append(new MimirStreamSample(
|
||||
"kiyo-pro-rgb",
|
||||
MimirStreamKind.Video,
|
||||
MimirStreamOrigin.LocalDevice,
|
||||
TimestampNs: 1_000_000_000L,
|
||||
ArrivalNs: 1_000_000_010L,
|
||||
Sequence: 7,
|
||||
PayloadHandle: 0,
|
||||
ByteLength: 1920 * 1080 * 4,
|
||||
Data: default,
|
||||
VideoFrame: new MimirVideoFrameDescriptor(
|
||||
1920,
|
||||
1080,
|
||||
MimirVideoPixelFormat.Bgra8,
|
||||
1920 * 4,
|
||||
1_000_000_000L,
|
||||
NativeHandle: 0xCAFEul,
|
||||
NativeHandleKind: "shared-d3d12-texture")));
|
||||
|
||||
var metadataOnlyBuffer = new MimirRollingStreamBuffer(
|
||||
new MimirStreamDescriptor("json-cadence-witness", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice),
|
||||
TimeSpan.FromSeconds(5));
|
||||
metadataOnlyBuffer.Append(new MimirStreamSample(
|
||||
"json-cadence-witness",
|
||||
MimirStreamKind.Video,
|
||||
MimirStreamOrigin.LocalDevice,
|
||||
TimestampNs: 1_000_000_050L,
|
||||
ArrivalNs: 1_000_000_060L,
|
||||
Sequence: 8,
|
||||
PayloadHandle: 0,
|
||||
ByteLength: 0,
|
||||
Data: default,
|
||||
VideoFrame: new MimirVideoFrameDescriptor(
|
||||
640,
|
||||
480,
|
||||
MimirVideoPixelFormat.Gray8,
|
||||
640,
|
||||
1_000_000_050L)));
|
||||
|
||||
MimirRollingStreamBuffer[] buffers = [gpuBuffer, metadataOnlyBuffer];
|
||||
var lowerer = new MimirFensalirFieldLowering();
|
||||
var frame = lowerer.BuildCameraObservationFrame(buffers);
|
||||
var validation = AquariumFieldEvidenceValidator.Validate(frame);
|
||||
var plan = AquariumFieldLoweringPlanner.Plan(frame);
|
||||
var requests = AquariumFieldEvidenceNormalizer.BuildLoweringRequests(frame);
|
||||
var cameraResourceKey = "mimir:resource:shared-d3d12-texture:cafe";
|
||||
|
||||
Console.WriteLine(
|
||||
$"fensalir-camera-observation-smoke domains={frame.Domains.Count} resources={frame.Resources.Count} claims={frame.Claims.Count} requests={requests.Count} planned={plan.Packets.Count} deferred={plan.DeferredRequests.Count} errors={validation.HasErrors}");
|
||||
|
||||
if (validation.HasErrors)
|
||||
{
|
||||
foreach (var issue in validation.Issues)
|
||||
{
|
||||
Console.Error.WriteLine($"camera-observation-issue {issue.Severity} {issue.Key}: {issue.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
return !validation.HasErrors &&
|
||||
frame.Resources.Count == 1 &&
|
||||
frame.Resources[0].ResourceKey == cameraResourceKey &&
|
||||
frame.Resources[0].Kind == AquariumFieldResourceKind.Texture2D &&
|
||||
frame.Resources[0].Residency == AquariumFieldResourceResidency.SharedGpu &&
|
||||
frame.Resources[0].Access == AquariumFieldShaderAccess.ShaderResource &&
|
||||
frame.Resources[0].Format == nameof(MimirVideoPixelFormat.Bgra8) &&
|
||||
frame.Resources[0].Width == 1920 &&
|
||||
frame.Resources[0].Height == 1080 &&
|
||||
frame.Resources[0].NativeHandle == new IntPtr(0xCAFE) &&
|
||||
frame.Claims.Count == 3 &&
|
||||
frame.Claims.Any(claim => claim.ClaimKey.StartsWith("observation:Video:LocalDevice:json-cadence-witness", StringComparison.Ordinal) &&
|
||||
string.IsNullOrWhiteSpace(claim.PayloadHandle)) &&
|
||||
frame.Claims.Any(claim => claim.ClaimKey.StartsWith("intent:Video:LocalDevice:kiyo-pro-rgb:surface", StringComparison.Ordinal) &&
|
||||
claim.PayloadHandle == cameraResourceKey) &&
|
||||
frame.Claims.All(claim => !string.Equals(claim.PayloadHandle, "latest-window", StringComparison.Ordinal)) &&
|
||||
requests.Count == 3 &&
|
||||
plan.Packets.Count == 0 &&
|
||||
plan.DeferredRequests.Count == 3
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
||||
static int RunMimirSpectrumUploadSmoke()
|
||||
{
|
||||
var previousSyntheticPreview = Environment.GetEnvironmentVariable("MIMIR_SYNTHETIC_SPECTRUM_PREVIEW");
|
||||
|
|
@ -1604,6 +1695,7 @@ static int RunPerfectMachineProfileSmoke()
|
|||
ArrivalNs: 1_000_000_010L,
|
||||
Sequence: 1,
|
||||
PayloadHandle: 0,
|
||||
ByteLength: 1920 * 1080 * 4,
|
||||
VideoFrame: new MimirVideoFrameDescriptor(
|
||||
1920,
|
||||
1080,
|
||||
|
|
@ -1613,7 +1705,8 @@ static int RunPerfectMachineProfileSmoke()
|
|||
NativeHandle: 42,
|
||||
NativeHandleKind: "shared-d3d12-texture")));
|
||||
var lowerer = new MimirFensalirFieldLowering();
|
||||
var gpuFrame = lowerer.BuildGpuSensorFrame([videoBuffer]);
|
||||
var cameraFieldFrame = lowerer.BuildCameraObservationFrame([videoBuffer]);
|
||||
var cameraPlan = AquariumFieldLoweringPlanner.Plan(cameraFieldFrame);
|
||||
var acousticFrame = lowerer.BuildAcousticFieldFrame([
|
||||
new MimirAudioSynchronizationState(
|
||||
"loopback-scarlett-speakers",
|
||||
|
|
@ -1665,7 +1758,7 @@ static int RunPerfectMachineProfileSmoke()
|
|||
Console.WriteLine(
|
||||
$"perfect-machine-clock anchors={clock?.AnchorCount ?? 0} coverage={clock?.AnchorCoverage ?? 0:0.000} offset={clock?.SourceOffsetSamples ?? double.NaN:0.000} confidence={clock?.Confidence ?? 0:0.000}");
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-fensalir gpuCameras={gpuFrame.Cameras.Count} gpuTextures={gpuFrame.ExternalTextures.Count} acousticConstraints={acousticFrame.Constraints.Count} timingConfidence={acousticFrame.TimingConfidence:0.000}");
|
||||
$"perfect-machine-fensalir cameraResources={cameraFieldFrame.Resources.Count} cameraClaims={cameraFieldFrame.Claims.Count} cameraPlanned={cameraPlan.Packets.Count} cameraDeferred={cameraPlan.DeferredRequests.Count} acousticConstraints={acousticFrame.Constraints.Count} timingConfidence={acousticFrame.TimingConfidence:0.000}");
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-localization candidates={localizationGrid.Count} best=({localization?.PositionMeters.X ?? float.NaN:0.000},{localization?.PositionMeters.Y ?? float.NaN:0.000},{localization?.PositionMeters.Z ?? float.NaN:0.000}) score={localization?.Score ?? 0.0:0.000}");
|
||||
Console.WriteLine(
|
||||
|
|
@ -1695,7 +1788,10 @@ static int RunPerfectMachineProfileSmoke()
|
|||
codebook.Motifs.Length == MimirBioacousticTimeline.SymbolCount &&
|
||||
decoder.Configuration.TemplateAugmentations.Length > 0 &&
|
||||
actuator.FaustControls.Length >= 2 &&
|
||||
gpuFrame.HasInput &&
|
||||
cameraFieldFrame.Resources.Count == 1 &&
|
||||
cameraFieldFrame.Claims.Count == 2 &&
|
||||
cameraPlan.Packets.Count == 0 &&
|
||||
cameraPlan.DeferredRequests.Count == 2 &&
|
||||
acousticFrame.HasInput &&
|
||||
localization is { Score: > 0.90 } &&
|
||||
authority.Decision == MimirAuthorityDecision.TrustedEvidence &&
|
||||
|
|
@ -1913,6 +2009,7 @@ static int RunPerfectMachineLoweringBenchmark(int iterations)
|
|||
ArrivalNs: 1_000_000_050L,
|
||||
Sequence: 1,
|
||||
PayloadHandle: 0,
|
||||
ByteLength: Math.Max(1, profile.PreferredWidth) * Math.Max(1, profile.PreferredHeight) * 4,
|
||||
VideoFrame: new MimirVideoFrameDescriptor(
|
||||
Math.Max(1, profile.PreferredWidth),
|
||||
Math.Max(1, profile.PreferredHeight),
|
||||
|
|
@ -1954,18 +2051,20 @@ static int RunPerfectMachineLoweringBenchmark(int iterations)
|
|||
1)
|
||||
};
|
||||
var lowerer = new MimirFensalirFieldLowering();
|
||||
_ = lowerer.BuildGpuSensorFrame(buffers);
|
||||
_ = lowerer.BuildCameraObservationFrame(buffers);
|
||||
_ = lowerer.BuildAcousticFieldFrame(states);
|
||||
|
||||
var allocatedBefore = GC.GetAllocatedBytesForCurrentThread();
|
||||
var stamp = Stopwatch.GetTimestamp();
|
||||
var cameras = 0;
|
||||
var cameraResources = 0;
|
||||
var cameraClaims = 0;
|
||||
var constraints = 0;
|
||||
for (var iteration = 0; iteration < iterations; iteration++)
|
||||
{
|
||||
var gpuFrame = lowerer.BuildGpuSensorFrame(buffers);
|
||||
var cameraFrame = lowerer.BuildCameraObservationFrame(buffers);
|
||||
var acousticFrame = lowerer.BuildAcousticFieldFrame(states);
|
||||
cameras += gpuFrame.Cameras.Count;
|
||||
cameraResources += cameraFrame.Resources.Count;
|
||||
cameraClaims += cameraFrame.Claims.Count;
|
||||
constraints += acousticFrame.Constraints.Count;
|
||||
}
|
||||
|
||||
|
|
@ -1974,8 +2073,8 @@ static int RunPerfectMachineLoweringBenchmark(int iterations)
|
|||
var perIterationUs = elapsed.TotalMilliseconds * 1000.0 / iterations;
|
||||
var allocatedPerIteration = allocatedBytes / (double)iterations;
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-lowering-benchmark iterations={iterations} camerasPer={cameras / (double)iterations:0.0} constraintsPer={constraints / (double)iterations:0.0} elapsedMs={elapsed.TotalMilliseconds:0.000} perIterationUs={perIterationUs:0.000} allocatedPerIterationBytes={allocatedPerIteration:0.0}");
|
||||
return perIterationUs < 250.0 && allocatedPerIteration < 10_000.0
|
||||
$"perfect-machine-lowering-benchmark iterations={iterations} cameraResourcesPer={cameraResources / (double)iterations:0.0} cameraClaimsPer={cameraClaims / (double)iterations:0.0} constraintsPer={constraints / (double)iterations:0.0} elapsedMs={elapsed.TotalMilliseconds:0.000} perIterationUs={perIterationUs:0.000} allocatedPerIterationBytes={allocatedPerIteration:0.0}");
|
||||
return perIterationUs < 250.0 && allocatedPerIteration < 64_000.0
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -204,6 +204,11 @@ public sealed class MimirRuntime : IAquariumRuntime
|
|||
continue;
|
||||
}
|
||||
|
||||
if (!window.Payload.HasResource)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
intents.Add(BuildSurfaceIntent(window, observation));
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -87,53 +87,31 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
|
|||
};
|
||||
}
|
||||
|
||||
public AquariumGpuSensorFrame BuildGpuSensorFrame(IEnumerable<MimirRollingStreamBuffer> buffers)
|
||||
public AquariumFieldEvidenceFrame BuildCameraObservationFrame(IEnumerable<MimirRollingStreamBuffer> buffers)
|
||||
{
|
||||
var capacity = buffers.TryGetNonEnumeratedCount(out var count) ? count : 0;
|
||||
var textures = new List<AquariumExternalGpuTexture>(capacity);
|
||||
var cameras = new List<AquariumGpuSensorCamera>(capacity);
|
||||
foreach (var buffer in buffers)
|
||||
var videoBuffers = buffers
|
||||
.Where(static buffer => buffer.Descriptor.Kind == MimirStreamKind.Video)
|
||||
.ToArray();
|
||||
var windows = new List<MimirRollingStreamWindow>(videoBuffers.Length);
|
||||
var observations = new List<MimirObservation>(videoBuffers.Length);
|
||||
MimirFensalirBridgeMapper.MapWindows(videoBuffers, windows);
|
||||
MimirFensalirBridgeMapper.MapLatestObservations(videoBuffers, observations);
|
||||
|
||||
var observationsByWindow = observations.ToDictionary(static observation => observation.WindowId, StringComparer.Ordinal);
|
||||
var intents = new List<MimirSurfaceIntent>(windows.Count);
|
||||
foreach (var window in windows)
|
||||
{
|
||||
if (buffer.Descriptor.Kind != MimirStreamKind.Video || buffer.Latest?.VideoFrame is not { } frame)
|
||||
if (window.Status != MimirBridgeWindowStatus.Live ||
|
||||
!window.Payload.HasResource ||
|
||||
!observationsByWindow.TryGetValue(window.WindowId, out var observation))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var firstTexture = textures.Count;
|
||||
if (frame.NativeHandle != 0)
|
||||
{
|
||||
textures.Add(new AquariumExternalGpuTexture(
|
||||
default,
|
||||
new IntPtr(unchecked((long)frame.NativeHandle)),
|
||||
frame.Width,
|
||||
frame.Height,
|
||||
ToFensalirPixelFormat(frame.PixelFormat),
|
||||
frame.DeviceTimestampNs,
|
||||
SharedHandleName: frame.NativeHandleKind));
|
||||
}
|
||||
|
||||
cameras.Add(new AquariumGpuSensorCamera(
|
||||
buffer.Descriptor.SourceId,
|
||||
ToFensalirSensorKind(buffer.Descriptor.SourceId, frame.PixelFormat),
|
||||
Matrix4x4.Identity,
|
||||
Matrix4x4.Identity,
|
||||
Vector4.Zero,
|
||||
Vector4.Zero,
|
||||
Vector4.Zero,
|
||||
frame.Width,
|
||||
frame.Height,
|
||||
firstTexture,
|
||||
textures.Count - firstTexture,
|
||||
frame.DeviceTimestampNs));
|
||||
intents.Add(MimirFensalirBridgeMapper.MapDefaultSurfaceIntent(window, observation));
|
||||
}
|
||||
|
||||
return new AquariumGpuSensorFrame
|
||||
{
|
||||
Cameras = cameras,
|
||||
ExternalTextures = textures,
|
||||
AccumulationWindowSeconds = (float)options.AccumulationWindowSeconds,
|
||||
PresentationDelaySeconds = (float)options.PresentationDelaySeconds
|
||||
};
|
||||
return BuildFieldEvidenceFrame(windows, observations, [], intents);
|
||||
}
|
||||
|
||||
public AquariumAcousticFieldFrame BuildAcousticFieldFrame(IEnumerable<MimirAudioSynchronizationState> states)
|
||||
|
|
@ -185,31 +163,6 @@ public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? opt
|
|||
};
|
||||
}
|
||||
|
||||
private static AquariumGpuSensorKind ToFensalirSensorKind(string sourceId, MimirVideoPixelFormat pixelFormat)
|
||||
{
|
||||
if (pixelFormat == MimirVideoPixelFormat.LeapStereoIr || sourceId.Contains("leap", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return AquariumGpuSensorKind.LeapPackedMap;
|
||||
}
|
||||
|
||||
if (sourceId.Contains("eye", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return AquariumGpuSensorKind.HighRateTracker;
|
||||
}
|
||||
|
||||
return AquariumGpuSensorKind.RgbCamera;
|
||||
}
|
||||
|
||||
private static AquariumGpuSensorPixelFormat ToFensalirPixelFormat(MimirVideoPixelFormat pixelFormat) =>
|
||||
pixelFormat switch
|
||||
{
|
||||
MimirVideoPixelFormat.Gray8 or MimirVideoPixelFormat.R8 or MimirVideoPixelFormat.Bayer8 => AquariumGpuSensorPixelFormat.R8Unorm,
|
||||
MimirVideoPixelFormat.Rg8 => AquariumGpuSensorPixelFormat.Rg8Unorm,
|
||||
MimirVideoPixelFormat.Bgra8 => AquariumGpuSensorPixelFormat.Bgra8Unorm,
|
||||
MimirVideoPixelFormat.LeapStereoIr => AquariumGpuSensorPixelFormat.LeapPackedMap,
|
||||
_ => AquariumGpuSensorPixelFormat.Unknown
|
||||
};
|
||||
|
||||
private static void AddDomain(
|
||||
ICollection<AquariumFieldDomain> domains,
|
||||
ISet<string> seenDomains,
|
||||
|
|
|
|||
|
|
@ -170,3 +170,4 @@
|
|||
{"ts": "2026-05-27T12:00:41+01:00", "type": "implementation", "status": "ok", "note": "Mimir now emits partial spectrum resource uploads for the active ring slots it samples. The declared GPU buffer remains fixed at 48 bands x 8 lanes x 40 history slots, but the smoke now verifies three offset uploads totaling 1152 floats for the three active ages instead of a full 15360-float backing-store upload. Headless Fensalir rendered the offset-copy path with empty stderr."}
|
||||
{"ts": "2026-05-27T12:21:38+01:00", "type": "implementation", "status": "ok", "note": "Mimir now sends only the newest spectrum ring slot as an offset Float32 resource upload. Fensalir owns rolling structured-buffer slot validity and clamps TubeField dispatch to valid contiguous logical columns, so invalid older slots cannot reach shader sampling after renderer allocation or partial update. Smoke reports uploadFloats=384 and headless render succeeded with empty stderr."}
|
||||
{"ts": "2026-05-27T12:32:42+01:00", "type": "implementation", "status": "ok", "note": "Updated docs/fensalir-rendering-rebuild-migration.md so the rendering migration map matches the live spectrum TubeField path: newest-slot Float32 uploads, fixed source-lane capacity, RollingOffset ring addressing, Fensalir-owned valid-slot gating, stable claim-derived TubeField metadata ids, and the current budget levers."}
|
||||
{"ts": "2026-05-27T12:43:45+01:00", "type": "implementation", "status": "ok", "note": "Camera descriptors now lower through FieldEvidence instead of the retired direct GPU sensor frame bridge. GPU/native video handles declare shared Texture2D resources and camera claims; metadata-only cadence frames remain observations with no render payload, so missing pixels cannot become fake latest-window requests. --fensalir-camera-observation-smoke verifies the split and camera image claims defer until Fensalir owns a selected visual-fusion lowering."}
|
||||
|
|
|
|||
|
|
@ -299,7 +299,15 @@ accepted_design:
|
|||
non-backend claims, and no validation errors.
|
||||
`--fensalir-field-dsl-resource-smoke` proves Fensalir's field evidence DSL
|
||||
can bind a declared resource directly and produce one planned `TubeField`
|
||||
packet with no deferred requests. Fensalir now owns the first in-process
|
||||
packet with no deferred requests. `MimirFensalirFieldLowering` also owns
|
||||
the current camera observation bridge: latest video buffers lower into
|
||||
FieldEvidence camera claims, GPU/native video handles declare shared
|
||||
Texture2D resources, and metadata-only cadence frames do not create camera
|
||||
surface intents or fake payload requests. The old direct
|
||||
`AquariumGpuSensorFrame` builder is no longer the active proof path;
|
||||
`Mimir.BufferSmoke --fensalir-camera-observation-smoke` verifies the split.
|
||||
Camera image claims currently defer until Fensalir has a selected visual
|
||||
fusion/feature extraction lowering for camera textures. Fensalir now owns the first in-process
|
||||
D3D12 field resource registry: declared shared structured/curve buffers
|
||||
import/alias GPU-resident resources by handle, while Fensalir-owned
|
||||
resources allocate GPU slots only when Fensalir is the producer.
|
||||
|
|
|
|||
Loading…
Reference in a new issue