Add Move tracking observation streams
This commit is contained in:
Meta Vens
parent
28536681f6
commit
6dc18ecf18
16 changed files with 406 additions and 13 deletions
|
|
@ -10,7 +10,7 @@ For the source-level ownership map, read [[code-algorithm-map|Code Algorithm Map
|
|||
- `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12
|
||||
bridge.
|
||||
- `src/Mimir.Runtime` owns stream descriptors, source polling, direct push
|
||||
ingest, one rolling buffer per configured audio/video stream, and the
|
||||
ingest, one rolling buffer per configured audio/video/tracking stream, and the
|
||||
synchronization hub Fensalir can inspect.
|
||||
- The default five-second window is an intentional latency/memory trade: use it
|
||||
to line up streams and extract the volumetric audio/video field before Mimir
|
||||
|
|
@ -31,8 +31,14 @@ a named invariant that the native runtime cannot protect yet.
|
|||
- `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and
|
||||
`IMimirStreamSource`.
|
||||
- Configurable five-second default rolling buffers for local and network audio
|
||||
and video streams.
|
||||
and video/tracking streams.
|
||||
- `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers.
|
||||
- `MimirStreamKind.Tracking` plus `MimirTrackingObservation` carry PS Move
|
||||
pose/button/battery/confidence samples as typed
|
||||
`mimir.move_tracking_observation.v1` documents. Starfire and Nightwing both
|
||||
have USB-attached Moves; Starfire can feed local tracking streams, while
|
||||
Muninn on Nightwing publishes remote tracking streams. Odin owns discovery and
|
||||
schema projection, not pose capture or Mimir fusion.
|
||||
- `MimirProcessStreamSource` for bridge/network command edges.
|
||||
- `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata
|
||||
from native probes into the same rolling buffers Fensalir inspects. One probe
|
||||
|
|
@ -192,7 +198,10 @@ a named invariant that the native runtime cannot protect yet.
|
|||
tooling/UI/remote witness use, and `--perfect-machine-lowering-benchmark` to
|
||||
measure the Mimir-to-Fensalir lowering path. The current six-camera/two-audio
|
||||
synthetic lowering benchmark runs at roughly 2.5 us per iteration with about
|
||||
2.3 KB allocated per iteration.
|
||||
2.3 KB allocated per iteration. `--move-tracking-contract-smoke` proves
|
||||
Starfire-local and Nightwing-remote Move tracking observations enter
|
||||
Mimir's rolling buffers with Muninn producer identity and Odin discovery
|
||||
provenance.
|
||||
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
|
||||
timestamp, and native/GPU handle metadata.
|
||||
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
|
||||
|
|
|
|||
|
|
@ -13,11 +13,11 @@ job-interview answer for why it belongs in the body of the machine.
|
|||
| Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` |
|
||||
| Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` |
|
||||
| Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` |
|
||||
| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, and actuator state documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs` |
|
||||
| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, and Move tracking observation documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` |
|
||||
| Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` |
|
||||
| Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` |
|
||||
| Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` |
|
||||
| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, and Raven Scarlett. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` |
|
||||
| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, Raven Scarlett, and USB-attached Moves on Starfire/Nightwing. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` |
|
||||
| Camera ingest strategies | `Mimir.Runtime` + native workers | Diagnostic metadata, managed native wrapper, native SPSC ring, and shared-GPU-texture ingest shapes. | `src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs` |
|
||||
| Reservoir strategies | `Mimir.Runtime` + native reservoir + Fensalir | Managed rolling buffers, native shared-edge rings, and derived GPU temporal field storage. | `src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs` |
|
||||
| Distributed witnesses | `Mimir.Runtime` + CultMesh | Raven, phone, microcontroller, and remote camera rig configurations for compact timing/path state. | `src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs` |
|
||||
|
|
@ -42,6 +42,9 @@ job-interview answer for why it belongs in the body of the machine.
|
|||
stem generation.
|
||||
- CultMesh carries typed state surfaces; remote nodes may observe and decode,
|
||||
but they do not become independent clock authorities.
|
||||
- USB-attached Moves on Starfire and Nightwing publish
|
||||
`mimir.move_tracking_observation.v1` as tracking streams. Muninn emits the
|
||||
witness, Odin discovers/projects it, and Mimir owns buffering plus fusion.
|
||||
- OBS may receive compatibility program surfaces and stems. It does not own
|
||||
calibration, composition, preview/control, or publication.
|
||||
|
||||
|
|
@ -54,6 +57,7 @@ dotnet build .\Mimir.slnx
|
|||
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke
|
||||
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-lowering-benchmark
|
||||
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke
|
||||
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke
|
||||
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest
|
||||
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test
|
||||
```
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ flowchart TD
|
|||
B["mic/loopback drivers"] --> R
|
||||
C["Leap timing/IR driver"] --> R
|
||||
D["network feed producers"] --> R
|
||||
M["Starfire/Nightwing Move tracking"] --> R
|
||||
R --> N["native reservoir handles"]
|
||||
N --> E["Fensalir GPU fusion + UI"]
|
||||
N --> F["Faust/native DSP"]
|
||||
|
|
@ -44,6 +45,11 @@ Ownership:
|
|||
becoming independent clock authorities.
|
||||
- `Mimir.Runtime` owns app-level stream buffers and synchronization.
|
||||
- Native capture workers own device reads.
|
||||
- Move tracking observations are stream samples, not program controls. USB
|
||||
Moves plugged into Starfire produce local tracking streams; USB Moves plugged
|
||||
into Nightwing are published by Muninn on Nightwing as remote tracking
|
||||
streams. Odin owns discovery/schema projection for those streams and does not
|
||||
own pose capture or Mimir fusion.
|
||||
- Mimir owns program composition, source subscription policy, preview/control
|
||||
state, stats, and publication intent.
|
||||
- Fensalir owns dense visual fusion, material/brush/splat reconciliation,
|
||||
|
|
@ -77,6 +83,7 @@ program video plus separately controllable audio stems.
|
|||
- `MimirProcessStreamSource`;
|
||||
- `MimirFrameEventProcessStreamSource`;
|
||||
- `MimirVideoFrameDescriptor`;
|
||||
- `MimirTrackingObservation`;
|
||||
- `MimirAudioSynchronizationAnalyzer`;
|
||||
- `IMimirVideoCaptureDriver`;
|
||||
- `MimirVideoCaptureDriverSource`.
|
||||
|
|
@ -89,6 +96,30 @@ the rolling buffers while the direct ABI driver is being cut. It does not carry
|
|||
pixels and does not own the final six-camera hot path. Multi-camera probes are
|
||||
one process with declared accepted source ids, not one process per camera.
|
||||
|
||||
## Move Tracking
|
||||
|
||||
Starfire and Nightwing both have PS Move controllers directly attached over
|
||||
USB. Mimir represents those controller poses as `MimirStreamKind.Tracking`
|
||||
buffers carrying typed `mimir.move_tracking_observation.v1` samples. Each
|
||||
sample names the stream, device, producer, host, Odin discovery provider,
|
||||
tracking space, calibration id, source timestamp, pose, velocity, buttons,
|
||||
battery, latency, and confidence.
|
||||
|
||||
Authority:
|
||||
|
||||
- Starfire-local Move tracking is a local Mimir/Muninn tracking stream.
|
||||
- Nightwing Move tracking is emitted by Muninn on Nightwing over the typed
|
||||
observation path.
|
||||
- Odin discovers and projects the stream/schema surface for operator and agent
|
||||
access.
|
||||
- Mimir owns subscription, rolling retention, clock alignment, and later fusion
|
||||
with cameras/audio/Leap/Fensalir. OBS and dashboard summaries do not own this
|
||||
state.
|
||||
|
||||
Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj
|
||||
-- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing
|
||||
Move observation into two tracking buffers.
|
||||
|
||||
## Program Composition
|
||||
|
||||
[[docs/mimir-program-composition|Mimir Program Composition]] is the live
|
||||
|
|
|
|||
|
|
@ -91,6 +91,11 @@ if (args.Any(arg => string.Equals(arg, "--perfect-machine-contract-smoke", Strin
|
|||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--move-tracking-contract-smoke", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return RunMoveTrackingContractSmoke();
|
||||
}
|
||||
|
||||
if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
return await ImportObsProgramSceneAsync(
|
||||
|
|
@ -1353,6 +1358,8 @@ static int RunPerfectMachineProfileSmoke()
|
|||
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state"],
|
||||
StringComparer.Ordinal),
|
||||
new HashSet<string>(StringComparer.Ordinal)));
|
||||
var trackingSmoke = BuildMoveTrackingSmokeHub();
|
||||
var trackingObservations = trackingSmoke.Hub.TrackingObservations;
|
||||
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-profile-smoke profiles={profiles.Count} calibrationPlans={calibrationPlans.Count} audioFields={audioFields.Count} visualFields={visualFields.Count} computePlans={computePlans.Count} assemblyPlans={assemblyPlans.Count} captureProfiles={captureProfiles.Count} publications={publications.Count} languageProfiles={languageProfiles.Count} pathLearning={pathLearningProfiles.Count} localization={localizationProfiles.Count} benchmarkPanels={benchmarkPanels.Count} actuatorStrategies={actuatorStrategies.Count} cameraIngest={cameraIngestStrategies.Count} reservoirs={reservoirStrategies.Count} distributedWitnesses={distributedWitnesses.Count} networkTransports={networkTransports.Count} authorityPolicies={authorityPolicies.Count} modules={moduleCatalog.Count}");
|
||||
|
|
@ -1372,6 +1379,8 @@ static int RunPerfectMachineProfileSmoke()
|
|||
$"perfect-machine-authority appliesTo=raven-scarlett-witness decision={authority.Decision} rule={authority.RuleId}");
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-transport payload=TypedTimingState status={transport.Status} selected={transport.Transport?.Id ?? "none"}");
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-move-tracking consumed={trackingSmoke.Consumed} observations={trackingObservations.Count} hosts={string.Join(",", trackingObservations.Select(observation => observation.HostId).Order(StringComparer.Ordinal))}");
|
||||
|
||||
return profiles.Count >= 6 &&
|
||||
calibrationPlans.Count >= 3 &&
|
||||
|
|
@ -1400,11 +1409,106 @@ static int RunPerfectMachineProfileSmoke()
|
|||
localization is { Score: > 0.90 } &&
|
||||
authority.Decision == MimirAuthorityDecision.TrustedEvidence &&
|
||||
transport.Transport?.Id == MimirNetworkTransportConfigurations.CultMeshTimingState.Id &&
|
||||
trackingSmoke.Consumed == 2 &&
|
||||
trackingObservations.Count == 2 &&
|
||||
trackingObservations.Any(observation => observation.HostId == "starfire" && observation.ProducerId == "muninn:starfire:move") &&
|
||||
trackingObservations.Any(observation => observation.HostId == "nightwing" && observation.ProducerId == "muninn:nightwing:move") &&
|
||||
clock is { AnchorCount: >= 4, Confidence: > 0.70 }
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
||||
static int RunMoveTrackingContractSmoke()
|
||||
{
|
||||
using var trackingSmoke = BuildMoveTrackingSmokeHub();
|
||||
var observations = trackingSmoke.Hub.TrackingObservations;
|
||||
foreach (var observation in observations)
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"move-tracking-observation stream={observation.StreamId} device={observation.DeviceId} host={observation.HostId} producer={observation.ProducerId} discovery={observation.DiscoveryProviderId} confidence={observation.Confidence:0.000} pos=({observation.PositionMeters.X:0.000},{observation.PositionMeters.Y:0.000},{observation.PositionMeters.Z:0.000})");
|
||||
}
|
||||
|
||||
Console.WriteLine(
|
||||
$"move-tracking-contract-smoke consumed={trackingSmoke.Consumed} observations={observations.Count} summary=\"{trackingSmoke.Hub.Summary()}\"");
|
||||
return trackingSmoke.Consumed == 2 &&
|
||||
observations.Count == 2 &&
|
||||
observations.All(observation => observation.Kind == MimirTrackingObservationKind.PsMoveController) &&
|
||||
observations.Any(observation => observation.HostId == "starfire" && observation.DiscoveryProviderId == "odin") &&
|
||||
observations.Any(observation => observation.HostId == "nightwing" && observation.DiscoveryProviderId == "odin")
|
||||
? 0
|
||||
: 1;
|
||||
}
|
||||
|
||||
static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub()
|
||||
{
|
||||
var starfireDescriptor = new MimirStreamDescriptor(
|
||||
"starfire:move:primary",
|
||||
MimirStreamKind.Tracking,
|
||||
MimirStreamOrigin.LocalDevice,
|
||||
DisplayName: "Starfire PS Move primary");
|
||||
var nightwingDescriptor = new MimirStreamDescriptor(
|
||||
"nightwing:move:primary",
|
||||
MimirStreamKind.Tracking,
|
||||
MimirStreamOrigin.Network,
|
||||
DisplayName: "Nightwing PS Move primary");
|
||||
var settings = new MimirSynchronizationSettings
|
||||
{
|
||||
Streams = [starfireDescriptor, nightwingDescriptor],
|
||||
};
|
||||
var hub = new MimirSynchronizationHub(settings);
|
||||
var starfire = new MimirNativeIngestStreamSource(starfireDescriptor);
|
||||
var nightwing = new MimirNativeIngestStreamSource(nightwingDescriptor);
|
||||
hub.AddSource(starfire);
|
||||
hub.AddSource(nightwing);
|
||||
starfire.PushTrackingObservation(MimirTrackingObservation.PsMove(
|
||||
starfireDescriptor.SourceId,
|
||||
"psmove:starfire:00",
|
||||
sequence: 1,
|
||||
sourceTimestampNs: 1_000_000_000L,
|
||||
arrivalTimestampNs: 1_000_000_120L,
|
||||
positionMeters: new MimirVector3Snapshot(0.21, 1.12, 0.84),
|
||||
orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0),
|
||||
linearVelocityMetersPerSecond: new MimirVector3Snapshot(0.04, 0.0, -0.02),
|
||||
angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.0, 0.1, 0.0),
|
||||
confidence: 0.93,
|
||||
calibrationId: "starfire-move-usb-calibration-v1",
|
||||
trackingSpaceId: "starfire-move-space",
|
||||
producerId: "muninn:starfire:move",
|
||||
hostId: "starfire",
|
||||
latencyMilliseconds: 1.2,
|
||||
battery01: 0.8,
|
||||
buttons:
|
||||
[
|
||||
new MimirTrackingButtonSnapshot("move", Pressed: false, Value: 0.0),
|
||||
new MimirTrackingButtonSnapshot("trigger", Pressed: true, Value: 0.72)
|
||||
]));
|
||||
nightwing.PushTrackingObservation(MimirTrackingObservation.PsMove(
|
||||
nightwingDescriptor.SourceId,
|
||||
"psmove:nightwing:00",
|
||||
sequence: 7,
|
||||
sourceTimestampNs: 1_000_008_333L,
|
||||
arrivalTimestampNs: 1_000_012_900L,
|
||||
positionMeters: new MimirVector3Snapshot(-0.34, 1.05, 1.42),
|
||||
orientation: new MimirQuaternionSnapshot(0.0, 0.18, 0.0, 0.984),
|
||||
linearVelocityMetersPerSecond: new MimirVector3Snapshot(-0.02, 0.01, 0.03),
|
||||
angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.2, 0.0, 0.0),
|
||||
confidence: 0.89,
|
||||
calibrationId: "nightwing-move-usb-calibration-v1",
|
||||
trackingSpaceId: "nightwing-move-space",
|
||||
producerId: "muninn:nightwing:move",
|
||||
hostId: "nightwing",
|
||||
latencyMilliseconds: 4.6,
|
||||
battery01: 0.67,
|
||||
buttons:
|
||||
[
|
||||
new MimirTrackingButtonSnapshot("move", Pressed: true, Value: 1.0),
|
||||
new MimirTrackingButtonSnapshot("trigger", Pressed: false, Value: 0.0)
|
||||
]));
|
||||
|
||||
var consumed = hub.PollSources();
|
||||
return new MoveTrackingSmokeContext(hub, consumed);
|
||||
}
|
||||
|
||||
static MimirAudioSynchronizationReport? EstimateBioacousticDelay(float[] reference, float[] candidate, int sampleRate)
|
||||
{
|
||||
const string referenceSourceId = "actuator-reference";
|
||||
|
|
@ -1563,6 +1667,40 @@ static async Task<int> RunPerfectMachineContractSmokeAsync(string outputPath)
|
|||
scene.SceneId,
|
||||
MimirProgramPublicationConfigurations.OperatorSurfaces[0],
|
||||
createdAt);
|
||||
var starfireMove = MimirTrackingObservation.PsMove(
|
||||
"starfire:move:primary",
|
||||
"psmove:starfire:00",
|
||||
sequence: 1,
|
||||
sourceTimestampNs: 1_000_000_000L,
|
||||
arrivalTimestampNs: 1_000_000_120L,
|
||||
positionMeters: new MimirVector3Snapshot(0.21, 1.12, 0.84),
|
||||
orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0),
|
||||
linearVelocityMetersPerSecond: new MimirVector3Snapshot(0.04, 0.0, -0.02),
|
||||
angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.0, 0.1, 0.0),
|
||||
confidence: 0.93,
|
||||
calibrationId: "starfire-move-usb-calibration-v1",
|
||||
trackingSpaceId: "starfire-move-space",
|
||||
producerId: "muninn:starfire:move",
|
||||
hostId: "starfire",
|
||||
latencyMilliseconds: 1.2,
|
||||
battery01: 0.8);
|
||||
var nightwingMove = MimirTrackingObservation.PsMove(
|
||||
"nightwing:move:primary",
|
||||
"psmove:nightwing:00",
|
||||
sequence: 7,
|
||||
sourceTimestampNs: 1_000_008_333L,
|
||||
arrivalTimestampNs: 1_000_012_900L,
|
||||
positionMeters: new MimirVector3Snapshot(-0.34, 1.05, 1.42),
|
||||
orientation: new MimirQuaternionSnapshot(0.0, 0.18, 0.0, 0.984),
|
||||
linearVelocityMetersPerSecond: new MimirVector3Snapshot(-0.02, 0.01, 0.03),
|
||||
angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.2, 0.0, 0.0),
|
||||
confidence: 0.89,
|
||||
calibrationId: "nightwing-move-usb-calibration-v1",
|
||||
trackingSpaceId: "nightwing-move-space",
|
||||
producerId: "muninn:nightwing:move",
|
||||
hostId: "nightwing",
|
||||
latencyMilliseconds: 4.6,
|
||||
battery01: 0.67);
|
||||
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? ".");
|
||||
using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions
|
||||
|
|
@ -1597,10 +1735,18 @@ static async Task<int> RunPerfectMachineContractSmokeAsync(string outputPath)
|
|||
eveSurface,
|
||||
new CultRecordHandle<MimirEveOperatorSurfaceDocument>(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}")))
|
||||
.ConfigureAwait(false);
|
||||
await cache.UpsertAsync(
|
||||
starfireMove,
|
||||
new CultRecordHandle<MimirTrackingObservation>(new CultRecordKey($"mimir-move-tracking:{starfireMove.ObservationId}")))
|
||||
.ConfigureAwait(false);
|
||||
await cache.UpsertAsync(
|
||||
nightwingMove,
|
||||
new CultRecordHandle<MimirTrackingObservation>(new CultRecordKey($"mimir-move-tracking:{nightwingMove.ObservationId}")))
|
||||
.ConfigureAwait(false);
|
||||
await cache.FlushAsync().ConfigureAwait(false);
|
||||
|
||||
Console.WriteLine(
|
||||
$"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length}");
|
||||
$"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length} moveTracking=2 hosts={starfireMove.HostId},{nightwingMove.HostId}");
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -5602,6 +5748,11 @@ public sealed record ComplexContourBandCorrection(
|
|||
double Reliability,
|
||||
bool Usable);
|
||||
|
||||
sealed record MoveTrackingSmokeContext(MimirSynchronizationHub Hub, int Consumed) : IDisposable
|
||||
{
|
||||
public void Dispose() => Hub.Dispose();
|
||||
}
|
||||
|
||||
public sealed record ComplexContourReflectionCorrection(
|
||||
double RelativeDelaySamples,
|
||||
int ObservationCount,
|
||||
|
|
|
|||
|
|
@ -862,6 +862,11 @@ direct
|
|||
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {block.Channels}ch {block.SampleRate}Hz {block.SampleFormat} frames {block.FrameCount} bytes {latest.Value.ByteLength} edge {buffer.EdgeNs}";
|
||||
}
|
||||
|
||||
if (latest?.TrackingObservation is { } tracking)
|
||||
{
|
||||
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {tracking.Kind} device={tracking.DeviceId} producer={tracking.ProducerId} host={tracking.HostId} confidence={tracking.Confidence:0.00} edge {buffer.EdgeNs}";
|
||||
}
|
||||
|
||||
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} edge {buffer.EdgeNs}";
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -47,8 +47,8 @@ public static class MimirModuleLibrary
|
|||
new(
|
||||
"cultmesh-contracts",
|
||||
"Mimir.Runtime",
|
||||
"Typed state documents for codebooks, decoder state, acoustic path state, and actuator state.",
|
||||
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state"],
|
||||
"Typed state documents for codebooks, decoder state, acoustic path state, actuator state, program state, and Move tracking observations.",
|
||||
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation"],
|
||||
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]),
|
||||
new(
|
||||
"path-learning",
|
||||
|
|
|
|||
|
|
@ -132,6 +132,48 @@ public static class MimirNativeCaptureConfigurations
|
|||
Notes = "Remote co-streamer audio witness. It decodes locally and ships typed timing/path state back over CultMesh."
|
||||
};
|
||||
|
||||
public static MimirNativeCaptureDeviceProfile StarfirePsMoves { get; } = new(
|
||||
"starfire-ps-moves",
|
||||
"Starfire PS Move tracking witnesses",
|
||||
MimirStreamKind.Tracking,
|
||||
MimirNativeCaptureTransport.VendorSdk,
|
||||
MimirCaptureClockDomain.DeviceHardware,
|
||||
ChannelCount: 2,
|
||||
PreferredWidth: 0,
|
||||
PreferredHeight: 0,
|
||||
PreferredSampleRate: 0,
|
||||
PreferredFramesPerSecond: 120.0,
|
||||
DefaultRollingBufferSeconds,
|
||||
Required: false,
|
||||
OwnedOutputs: ["local-controller-pose-observations", "performance-motion-cues"],
|
||||
"USB-attached local Moves. Mimir buffers the observations; tracking code owns hardware pose estimation.");
|
||||
|
||||
public static MimirNativeCaptureDeviceProfile NightwingPsMoves { get; } = StarfirePsMoves with
|
||||
{
|
||||
Id = "nightwing-ps-moves",
|
||||
DisplayName = "Nightwing PS Move tracking witnesses",
|
||||
Transport = MimirNativeCaptureTransport.NetworkCultMesh,
|
||||
ClockDomain = MimirCaptureClockDomain.NetworkSender,
|
||||
OwnedOutputs = ["remote-controller-pose-observations", "performance-motion-cues"],
|
||||
Notes = "USB-attached Nightwing Moves. Muninn on Nightwing publishes typed pose/button observations; Odin owns discovery/schema projection; Mimir owns buffering and fusion."
|
||||
};
|
||||
|
||||
public static MimirNativeCaptureDeviceProfile OdinMoveTrackingCatalog { get; } = new(
|
||||
"odin-move-tracking-catalog",
|
||||
"Odin Move tracking discovery catalog",
|
||||
MimirStreamKind.Tracking,
|
||||
MimirNativeCaptureTransport.NetworkCultMesh,
|
||||
MimirCaptureClockDomain.CanonicalFitted,
|
||||
ChannelCount: 4,
|
||||
PreferredWidth: 0,
|
||||
PreferredHeight: 0,
|
||||
PreferredSampleRate: 0,
|
||||
PreferredFramesPerSecond: 120.0,
|
||||
DefaultRollingBufferSeconds,
|
||||
Required: false,
|
||||
OwnedOutputs: ["move-stream-discovery", "tracking-schema-rendezvous"],
|
||||
"Discovery/projection surface only. Odin names Starfire and Nightwing Move streams; it does not own pose capture or Mimir fusion.");
|
||||
|
||||
public static IReadOnlyList<MimirNativeCaptureDeviceProfile> LocalSixCameraProfiles { get; } =
|
||||
[
|
||||
LeapLeftIr,
|
||||
|
|
@ -146,6 +188,9 @@ public static class MimirNativeCaptureConfigurations
|
|||
[
|
||||
.. LocalSixCameraProfiles,
|
||||
ScarlettStarfire,
|
||||
ScarlettRaven
|
||||
ScarlettRaven,
|
||||
StarfirePsMoves,
|
||||
NightwingPsMoves,
|
||||
OdinMoveTrackingCatalog
|
||||
];
|
||||
}
|
||||
|
|
|
|||
|
|
@ -20,7 +20,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
|
|||
ulong payloadHandle,
|
||||
int byteLength = 0,
|
||||
ReadOnlyMemory<byte> data = default,
|
||||
MimirVideoFrameDescriptor? videoFrame = null)
|
||||
MimirVideoFrameDescriptor? videoFrame = null,
|
||||
MimirTrackingObservation? trackingObservation = null)
|
||||
{
|
||||
samples.Enqueue(new MimirStreamSample(
|
||||
Descriptor.SourceId,
|
||||
|
|
@ -32,7 +33,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
|
|||
payloadHandle,
|
||||
byteLength,
|
||||
data,
|
||||
videoFrame));
|
||||
videoFrame,
|
||||
TrackingObservation: trackingObservation));
|
||||
}
|
||||
|
||||
public void PushVideoFrame(
|
||||
|
|
@ -54,6 +56,25 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
|
|||
frame);
|
||||
}
|
||||
|
||||
public void PushTrackingObservation(MimirTrackingObservation observation)
|
||||
{
|
||||
if (Descriptor.Kind != MimirStreamKind.Tracking)
|
||||
{
|
||||
throw new InvalidOperationException("Tracking observations can only be pushed into a tracking stream source.");
|
||||
}
|
||||
|
||||
if (!string.Equals(observation.StreamId, Descriptor.SourceId, StringComparison.Ordinal))
|
||||
{
|
||||
throw new ArgumentException("Tracking observation stream id does not match this stream source.", nameof(observation));
|
||||
}
|
||||
|
||||
Push(
|
||||
observation.SourceTimestampNs,
|
||||
observation.ArrivalTimestampNs,
|
||||
payloadHandle: 0,
|
||||
trackingObservation: observation);
|
||||
}
|
||||
|
||||
public bool TryRead(out MimirStreamSample sample)
|
||||
{
|
||||
return samples.TryDequeue(out sample);
|
||||
|
|
|
|||
|
|
@ -155,6 +155,7 @@ public static class MimirPerfectMachineProfiles
|
|||
[
|
||||
"muninn.telemetry_surface.v1",
|
||||
"muninn.capture_stream.v1",
|
||||
"mimir.move_tracking_observation.v1",
|
||||
"mimir.cultmesh_media_frame.v1"
|
||||
]);
|
||||
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ public enum MimirStreamKind
|
|||
{
|
||||
Audio,
|
||||
Video,
|
||||
Tracking,
|
||||
}
|
||||
|
||||
public enum MimirStreamOrigin
|
||||
|
|
|
|||
|
|
@ -11,4 +11,5 @@ public readonly record struct MimirStreamSample(
|
|||
int ByteLength = 0,
|
||||
ReadOnlyMemory<byte> Data = default,
|
||||
MimirVideoFrameDescriptor? VideoFrame = null,
|
||||
MimirAudioBlockDescriptor? AudioBlock = null);
|
||||
MimirAudioBlockDescriptor? AudioBlock = null,
|
||||
MimirTrackingObservation? TrackingObservation = null);
|
||||
|
|
|
|||
|
|
@ -51,6 +51,15 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
public IReadOnlyList<MimirChirpBinCalibrationProfile> AudioChirpBinCalibrationProfiles =>
|
||||
audioSynchronization.LastCalibrationProfiles;
|
||||
|
||||
public IReadOnlyList<MimirTrackingObservation> TrackingObservations =>
|
||||
Buffers.Buffers
|
||||
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking)
|
||||
.Select(buffer => buffer.Latest?.TrackingObservation)
|
||||
.Where(observation => observation != null)
|
||||
.Cast<MimirTrackingObservation>()
|
||||
.OrderBy(observation => observation.StreamId, StringComparer.Ordinal)
|
||||
.ToArray();
|
||||
|
||||
public MimirChirpBinCodebookPlan? ChirpBinEmissionPlan =>
|
||||
chirpBinCalibrationModel?.EmissionPlan;
|
||||
|
||||
|
|
@ -87,7 +96,8 @@ public sealed class MimirSynchronizationHub : IDisposable
|
|||
{
|
||||
var audio = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio);
|
||||
var video = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Video);
|
||||
return $"{video} video / {audio} audio buffers";
|
||||
var tracking = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking);
|
||||
return $"{video} video / {audio} audio / {tracking} tracking buffers";
|
||||
}
|
||||
|
||||
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronization(
|
||||
|
|
|
|||
|
|
@ -14,8 +14,10 @@ public sealed class MimirSynchronizationSettings
|
|||
var streams = new List<MimirStreamDescriptor>();
|
||||
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.LocalDevice);
|
||||
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.LocalDevice);
|
||||
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.LocalDevice);
|
||||
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.Network);
|
||||
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.Network);
|
||||
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.Network);
|
||||
|
||||
return new MimirSynchronizationSettings
|
||||
{
|
||||
|
|
|
|||
100
src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs
Normal file
100
src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
using MessagePack;
|
||||
using GameCult.Caching;
|
||||
|
||||
namespace Mimir.Runtime.Synchronization;
|
||||
|
||||
public enum MimirTrackingObservationKind
|
||||
{
|
||||
Unknown,
|
||||
PsMoveController,
|
||||
Hand,
|
||||
Headset,
|
||||
Tool,
|
||||
}
|
||||
|
||||
[MessagePackObject]
|
||||
public readonly record struct MimirVector3Snapshot(
|
||||
[property: Key(0)] double X,
|
||||
[property: Key(1)] double Y,
|
||||
[property: Key(2)] double Z);
|
||||
|
||||
[MessagePackObject]
|
||||
public readonly record struct MimirQuaternionSnapshot(
|
||||
[property: Key(0)] double X,
|
||||
[property: Key(1)] double Y,
|
||||
[property: Key(2)] double Z,
|
||||
[property: Key(3)] double W);
|
||||
|
||||
[MessagePackObject]
|
||||
public sealed record MimirTrackingButtonSnapshot(
|
||||
[property: Key(0)] string Id,
|
||||
[property: Key(1)] bool Pressed,
|
||||
[property: Key(2)] double Value);
|
||||
|
||||
[MessagePackObject]
|
||||
[CultDocument("mimir.move_tracking_observation", "mimir.move_tracking_observation.v1")]
|
||||
public sealed record MimirTrackingObservation(
|
||||
[property: Key(0)]
|
||||
[property: CultName]
|
||||
string ObservationId,
|
||||
[property: Key(1)] string StreamId,
|
||||
[property: Key(2)] string DeviceId,
|
||||
[property: Key(3)] MimirTrackingObservationKind Kind,
|
||||
[property: Key(4)] string ProducerId,
|
||||
[property: Key(5)] string HostId,
|
||||
[property: Key(6)] string DiscoveryProviderId,
|
||||
[property: Key(7)] string TrackingSpaceId,
|
||||
[property: Key(8)] string CalibrationId,
|
||||
[property: Key(9)] long SourceTimestampNs,
|
||||
[property: Key(10)] long ArrivalTimestampNs,
|
||||
[property: Key(11)] ulong Sequence,
|
||||
[property: Key(12)] MimirVector3Snapshot PositionMeters,
|
||||
[property: Key(13)] MimirQuaternionSnapshot Orientation,
|
||||
[property: Key(14)] MimirVector3Snapshot LinearVelocityMetersPerSecond,
|
||||
[property: Key(15)] MimirVector3Snapshot AngularVelocityRadiansPerSecond,
|
||||
[property: Key(16)] double Confidence,
|
||||
[property: Key(17)] double LatencyMilliseconds,
|
||||
[property: Key(18)] double Battery01,
|
||||
[property: Key(19)] MimirTrackingButtonSnapshot[] Buttons)
|
||||
{
|
||||
public static MimirTrackingObservation PsMove(
|
||||
string streamId,
|
||||
string deviceId,
|
||||
ulong sequence,
|
||||
long sourceTimestampNs,
|
||||
long arrivalTimestampNs,
|
||||
MimirVector3Snapshot positionMeters,
|
||||
MimirQuaternionSnapshot orientation,
|
||||
MimirVector3Snapshot linearVelocityMetersPerSecond,
|
||||
MimirVector3Snapshot angularVelocityRadiansPerSecond,
|
||||
double confidence,
|
||||
string calibrationId,
|
||||
string trackingSpaceId = "nightwing-move-space",
|
||||
string producerId = "muninn:nightwing:move",
|
||||
string hostId = "nightwing",
|
||||
string discoveryProviderId = "odin",
|
||||
double latencyMilliseconds = 0.0,
|
||||
double battery01 = double.NaN,
|
||||
MimirTrackingButtonSnapshot[]? buttons = null) =>
|
||||
new(
|
||||
$"{streamId}:{sequence}",
|
||||
streamId,
|
||||
deviceId,
|
||||
MimirTrackingObservationKind.PsMoveController,
|
||||
producerId,
|
||||
hostId,
|
||||
discoveryProviderId,
|
||||
trackingSpaceId,
|
||||
calibrationId,
|
||||
sourceTimestampNs,
|
||||
arrivalTimestampNs,
|
||||
sequence,
|
||||
positionMeters,
|
||||
orientation,
|
||||
linearVelocityMetersPerSecond,
|
||||
angularVelocityRadiansPerSecond,
|
||||
Math.Clamp(confidence, 0.0, 1.0),
|
||||
Math.Max(0.0, latencyMilliseconds),
|
||||
double.IsFinite(battery01) ? Math.Clamp(battery01, 0.0, 1.0) : double.NaN,
|
||||
buttons ?? []);
|
||||
}
|
||||
|
|
@ -123,3 +123,4 @@
|
|||
{"ts": "2026-06-08T09:59:22+01:00", "type": "verification", "status": "blocked", "note": "CultMesh media bridge verification: Yggdrasil UFW was opened for 3075/udp on wg-gamecult and Starfire receiver now connects from 10.77.0.2. Synthetic sender on Yggdrasil connects and sends chunks without User Not Verified after CultLib pre-verification schema-v0 policy patch, but relay cache does not receive mimir.cultmesh_media_frame updates and Starfire receiver logs no frames. Remaining blocker is CultNet document put/apply path, not WireGuard routing or OBS endpoint."}
|
||||
{"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."}
|
||||
{"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."}
|
||||
{"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."}
|
||||
|
|
|
|||
|
|
@ -33,6 +33,9 @@ constraints:
|
|||
memory.
|
||||
- Networked feeds are producers into the live runtime, not independent clock
|
||||
authorities.
|
||||
- USB-attached Moves on Starfire and Nightwing are tracking witnesses. Odin
|
||||
may discover/project their stream schemas, but it does not own pose capture
|
||||
or Mimir fusion.
|
||||
- CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may
|
||||
consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but
|
||||
Mimir owns program selection, composition, preview/control, and publication.
|
||||
|
|
@ -63,6 +66,9 @@ invariants:
|
|||
response, and compute-offload work need typed mesh-syncable state surfaces
|
||||
so phones, microcontrollers, Raven, and Starfire can cooperate without
|
||||
becoming independent timing authorities.
|
||||
- Move tracking observations are stream samples, not program controls or
|
||||
dashboard truth. Starfire-local and Nightwing-remote Move samples must enter
|
||||
Mimir.Runtime as typed tracking buffers before Fensalir or Eve acts on them.
|
||||
|
||||
accepted_design:
|
||||
- `Mimir.slnx` owns the app surface.
|
||||
|
|
@ -74,6 +80,11 @@ accepted_design:
|
|||
device timestamp, and optional native/GPU handle.
|
||||
- `MimirAudioBlockDescriptor` carries sample rate, channels, sample format,
|
||||
frame count, timestamp, and optional native handle.
|
||||
- `MimirTrackingObservation` carries typed PS Move pose/button/battery/
|
||||
confidence samples as `mimir.move_tracking_observation.v1`; Starfire Moves
|
||||
are local tracking streams, Nightwing Moves are emitted by Muninn on
|
||||
Nightwing, Odin owns discovery/schema projection, and Mimir owns buffering
|
||||
and fusion.
|
||||
- `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for
|
||||
JSON-line video frame and audio block metadata from native probes into
|
||||
runtime rolling buffers; it does not carry pixels/audio payloads or own the
|
||||
|
|
|
|||
Loading…
Reference in a new issue