From 6dc18ecf18fd6b0baee6be7df130fb7a05a8e609 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 16:43:12 +0100 Subject: [PATCH] Add Move tracking observation streams --- docs/implementation-plan.md | 15 +- docs/perfect-machine-module-library.md | 8 +- notes/current-system-map.md | 31 ++++ src/Mimir.BufferSmoke/Program.cs | 153 +++++++++++++++++- src/Mimir.Runtime/MimirRuntime.cs | 5 + .../Synchronization/MimirModuleLibrary.cs | 4 +- .../MimirNativeCaptureConfiguration.cs | 47 +++++- .../MimirNativeIngestStreamSource.cs | 25 ++- .../MimirPerfectMachineProfiles.cs | 1 + .../Synchronization/MimirStreamKind.cs | 1 + .../Synchronization/MimirStreamSample.cs | 3 +- .../MimirSynchronizationHub.cs | 12 +- .../MimirSynchronizationSettings.cs | 2 + .../MimirTrackingObservation.cs | 100 ++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 11 ++ 16 files changed, 406 insertions(+), 13 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index a44944a..2b3abf3 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -10,7 +10,7 @@ For the source-level ownership map, read [[code-algorithm-map|Code Algorithm Map - `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12 bridge. - `src/Mimir.Runtime` owns stream descriptors, source polling, direct push - ingest, one rolling buffer per configured audio/video stream, and the + ingest, one rolling buffer per configured audio/video/tracking stream, and the synchronization hub Fensalir can inspect. - The default five-second window is an intentional latency/memory trade: use it to line up streams and extract the volumetric audio/video field before Mimir @@ -31,8 +31,14 @@ a named invariant that the native runtime cannot protect yet. - `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and `IMimirStreamSource`. - Configurable five-second default rolling buffers for local and network audio - and video streams. + and video/tracking streams. - `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers. +- `MimirStreamKind.Tracking` plus `MimirTrackingObservation` carry PS Move + pose/button/battery/confidence samples as typed + `mimir.move_tracking_observation.v1` documents. Starfire and Nightwing both + have USB-attached Moves; Starfire can feed local tracking streams, while + Muninn on Nightwing publishes remote tracking streams. Odin owns discovery and + schema projection, not pose capture or Mimir fusion. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe @@ -192,7 +198,10 @@ a named invariant that the native runtime cannot protect yet. tooling/UI/remote witness use, and `--perfect-machine-lowering-benchmark` to measure the Mimir-to-Fensalir lowering path. The current six-camera/two-audio synthetic lowering benchmark runs at roughly 2.5 us per iteration with about - 2.3 KB allocated per iteration. + 2.3 KB allocated per iteration. `--move-tracking-contract-smoke` proves + Starfire-local and Nightwing-remote Move tracking observations enter + Mimir's rolling buffers with Muninn producer identity and Odin discovery + provenance. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device timestamp, and native/GPU handle metadata. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index c5fa824..1c6a313 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -13,11 +13,11 @@ job-interview answer for why it belongs in the body of the machine. | Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` | | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | -| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, and actuator state documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs` | +| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, and Move tracking observation documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | -| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, and Raven Scarlett. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` | +| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, Raven Scarlett, and USB-attached Moves on Starfire/Nightwing. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` | | Camera ingest strategies | `Mimir.Runtime` + native workers | Diagnostic metadata, managed native wrapper, native SPSC ring, and shared-GPU-texture ingest shapes. | `src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs` | | Reservoir strategies | `Mimir.Runtime` + native reservoir + Fensalir | Managed rolling buffers, native shared-edge rings, and derived GPU temporal field storage. | `src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs` | | Distributed witnesses | `Mimir.Runtime` + CultMesh | Raven, phone, microcontroller, and remote camera rig configurations for compact timing/path state. | `src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs` | @@ -42,6 +42,9 @@ job-interview answer for why it belongs in the body of the machine. stem generation. - CultMesh carries typed state surfaces; remote nodes may observe and decode, but they do not become independent clock authorities. +- USB-attached Moves on Starfire and Nightwing publish + `mimir.move_tracking_observation.v1` as tracking streams. Muninn emits the + witness, Odin discovers/projects it, and Mimir owns buffering plus fusion. - OBS may receive compatibility program surfaces and stems. It does not own calibration, composition, preview/control, or publication. @@ -54,6 +57,7 @@ dotnet build .\Mimir.slnx dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-lowering-benchmark dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test ``` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 52b0621..936b4e4 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -22,6 +22,7 @@ flowchart TD B["mic/loopback drivers"] --> R C["Leap timing/IR driver"] --> R D["network feed producers"] --> R + M["Starfire/Nightwing Move tracking"] --> R R --> N["native reservoir handles"] N --> E["Fensalir GPU fusion + UI"] N --> F["Faust/native DSP"] @@ -44,6 +45,11 @@ Ownership: becoming independent clock authorities. - `Mimir.Runtime` owns app-level stream buffers and synchronization. - Native capture workers own device reads. +- Move tracking observations are stream samples, not program controls. USB + Moves plugged into Starfire produce local tracking streams; USB Moves plugged + into Nightwing are published by Muninn on Nightwing as remote tracking + streams. Odin owns discovery/schema projection for those streams and does not + own pose capture or Mimir fusion. - Mimir owns program composition, source subscription policy, preview/control state, stats, and publication intent. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, @@ -77,6 +83,7 @@ program video plus separately controllable audio stems. - `MimirProcessStreamSource`; - `MimirFrameEventProcessStreamSource`; - `MimirVideoFrameDescriptor`; +- `MimirTrackingObservation`; - `MimirAudioSynchronizationAnalyzer`; - `IMimirVideoCaptureDriver`; - `MimirVideoCaptureDriverSource`. @@ -89,6 +96,30 @@ the rolling buffers while the direct ABI driver is being cut. It does not carry pixels and does not own the final six-camera hot path. Multi-camera probes are one process with declared accepted source ids, not one process per camera. +## Move Tracking + +Starfire and Nightwing both have PS Move controllers directly attached over +USB. Mimir represents those controller poses as `MimirStreamKind.Tracking` +buffers carrying typed `mimir.move_tracking_observation.v1` samples. Each +sample names the stream, device, producer, host, Odin discovery provider, +tracking space, calibration id, source timestamp, pose, velocity, buttons, +battery, latency, and confidence. + +Authority: + +- Starfire-local Move tracking is a local Mimir/Muninn tracking stream. +- Nightwing Move tracking is emitted by Muninn on Nightwing over the typed + observation path. +- Odin discovers and projects the stream/schema surface for operator and agent + access. +- Mimir owns subscription, rolling retention, clock alignment, and later fusion + with cameras/audio/Leap/Fensalir. OBS and dashboard summaries do not own this + state. + +Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj +-- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing +Move observation into two tracking buffers. + ## Program Composition [[docs/mimir-program-composition|Mimir Program Composition]] is the live diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 5aa9d1e..eb631d4 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -91,6 +91,11 @@ if (args.Any(arg => string.Equals(arg, "--perfect-machine-contract-smoke", Strin .ConfigureAwait(false); } +if (args.Any(arg => string.Equals(arg, "--move-tracking-contract-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveTrackingContractSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1353,6 +1358,8 @@ static int RunPerfectMachineProfileSmoke() ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state"], StringComparer.Ordinal), new HashSet(StringComparer.Ordinal))); + var trackingSmoke = BuildMoveTrackingSmokeHub(); + var trackingObservations = trackingSmoke.Hub.TrackingObservations; Console.WriteLine( $"perfect-machine-profile-smoke profiles={profiles.Count} calibrationPlans={calibrationPlans.Count} audioFields={audioFields.Count} visualFields={visualFields.Count} computePlans={computePlans.Count} assemblyPlans={assemblyPlans.Count} captureProfiles={captureProfiles.Count} publications={publications.Count} languageProfiles={languageProfiles.Count} pathLearning={pathLearningProfiles.Count} localization={localizationProfiles.Count} benchmarkPanels={benchmarkPanels.Count} actuatorStrategies={actuatorStrategies.Count} cameraIngest={cameraIngestStrategies.Count} reservoirs={reservoirStrategies.Count} distributedWitnesses={distributedWitnesses.Count} networkTransports={networkTransports.Count} authorityPolicies={authorityPolicies.Count} modules={moduleCatalog.Count}"); @@ -1372,6 +1379,8 @@ static int RunPerfectMachineProfileSmoke() $"perfect-machine-authority appliesTo=raven-scarlett-witness decision={authority.Decision} rule={authority.RuleId}"); Console.WriteLine( $"perfect-machine-transport payload=TypedTimingState status={transport.Status} selected={transport.Transport?.Id ?? "none"}"); + Console.WriteLine( + $"perfect-machine-move-tracking consumed={trackingSmoke.Consumed} observations={trackingObservations.Count} hosts={string.Join(",", trackingObservations.Select(observation => observation.HostId).Order(StringComparer.Ordinal))}"); return profiles.Count >= 6 && calibrationPlans.Count >= 3 && @@ -1400,11 +1409,106 @@ static int RunPerfectMachineProfileSmoke() localization is { Score: > 0.90 } && authority.Decision == MimirAuthorityDecision.TrustedEvidence && transport.Transport?.Id == MimirNetworkTransportConfigurations.CultMeshTimingState.Id && + trackingSmoke.Consumed == 2 && + trackingObservations.Count == 2 && + trackingObservations.Any(observation => observation.HostId == "starfire" && observation.ProducerId == "muninn:starfire:move") && + trackingObservations.Any(observation => observation.HostId == "nightwing" && observation.ProducerId == "muninn:nightwing:move") && clock is { AnchorCount: >= 4, Confidence: > 0.70 } ? 0 : 1; } +static int RunMoveTrackingContractSmoke() +{ + using var trackingSmoke = BuildMoveTrackingSmokeHub(); + var observations = trackingSmoke.Hub.TrackingObservations; + foreach (var observation in observations) + { + Console.WriteLine( + $"move-tracking-observation stream={observation.StreamId} device={observation.DeviceId} host={observation.HostId} producer={observation.ProducerId} discovery={observation.DiscoveryProviderId} confidence={observation.Confidence:0.000} pos=({observation.PositionMeters.X:0.000},{observation.PositionMeters.Y:0.000},{observation.PositionMeters.Z:0.000})"); + } + + Console.WriteLine( + $"move-tracking-contract-smoke consumed={trackingSmoke.Consumed} observations={observations.Count} summary=\"{trackingSmoke.Hub.Summary()}\""); + return trackingSmoke.Consumed == 2 && + observations.Count == 2 && + observations.All(observation => observation.Kind == MimirTrackingObservationKind.PsMoveController) && + observations.Any(observation => observation.HostId == "starfire" && observation.DiscoveryProviderId == "odin") && + observations.Any(observation => observation.HostId == "nightwing" && observation.DiscoveryProviderId == "odin") + ? 0 + : 1; +} + +static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() +{ + var starfireDescriptor = new MimirStreamDescriptor( + "starfire:move:primary", + MimirStreamKind.Tracking, + MimirStreamOrigin.LocalDevice, + DisplayName: "Starfire PS Move primary"); + var nightwingDescriptor = new MimirStreamDescriptor( + "nightwing:move:primary", + MimirStreamKind.Tracking, + MimirStreamOrigin.Network, + DisplayName: "Nightwing PS Move primary"); + var settings = new MimirSynchronizationSettings + { + Streams = [starfireDescriptor, nightwingDescriptor], + }; + var hub = new MimirSynchronizationHub(settings); + var starfire = new MimirNativeIngestStreamSource(starfireDescriptor); + var nightwing = new MimirNativeIngestStreamSource(nightwingDescriptor); + hub.AddSource(starfire); + hub.AddSource(nightwing); + starfire.PushTrackingObservation(MimirTrackingObservation.PsMove( + starfireDescriptor.SourceId, + "psmove:starfire:00", + sequence: 1, + sourceTimestampNs: 1_000_000_000L, + arrivalTimestampNs: 1_000_000_120L, + positionMeters: new MimirVector3Snapshot(0.21, 1.12, 0.84), + orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(0.04, 0.0, -0.02), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.0, 0.1, 0.0), + confidence: 0.93, + calibrationId: "starfire-move-usb-calibration-v1", + trackingSpaceId: "starfire-move-space", + producerId: "muninn:starfire:move", + hostId: "starfire", + latencyMilliseconds: 1.2, + battery01: 0.8, + buttons: + [ + new MimirTrackingButtonSnapshot("move", Pressed: false, Value: 0.0), + new MimirTrackingButtonSnapshot("trigger", Pressed: true, Value: 0.72) + ])); + nightwing.PushTrackingObservation(MimirTrackingObservation.PsMove( + nightwingDescriptor.SourceId, + "psmove:nightwing:00", + sequence: 7, + sourceTimestampNs: 1_000_008_333L, + arrivalTimestampNs: 1_000_012_900L, + positionMeters: new MimirVector3Snapshot(-0.34, 1.05, 1.42), + orientation: new MimirQuaternionSnapshot(0.0, 0.18, 0.0, 0.984), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(-0.02, 0.01, 0.03), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.2, 0.0, 0.0), + confidence: 0.89, + calibrationId: "nightwing-move-usb-calibration-v1", + trackingSpaceId: "nightwing-move-space", + producerId: "muninn:nightwing:move", + hostId: "nightwing", + latencyMilliseconds: 4.6, + battery01: 0.67, + buttons: + [ + new MimirTrackingButtonSnapshot("move", Pressed: true, Value: 1.0), + new MimirTrackingButtonSnapshot("trigger", Pressed: false, Value: 0.0) + ])); + + var consumed = hub.PollSources(); + return new MoveTrackingSmokeContext(hub, consumed); +} + static MimirAudioSynchronizationReport? EstimateBioacousticDelay(float[] reference, float[] candidate, int sampleRate) { const string referenceSourceId = "actuator-reference"; @@ -1563,6 +1667,40 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) scene.SceneId, MimirProgramPublicationConfigurations.OperatorSurfaces[0], createdAt); + var starfireMove = MimirTrackingObservation.PsMove( + "starfire:move:primary", + "psmove:starfire:00", + sequence: 1, + sourceTimestampNs: 1_000_000_000L, + arrivalTimestampNs: 1_000_000_120L, + positionMeters: new MimirVector3Snapshot(0.21, 1.12, 0.84), + orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(0.04, 0.0, -0.02), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.0, 0.1, 0.0), + confidence: 0.93, + calibrationId: "starfire-move-usb-calibration-v1", + trackingSpaceId: "starfire-move-space", + producerId: "muninn:starfire:move", + hostId: "starfire", + latencyMilliseconds: 1.2, + battery01: 0.8); + var nightwingMove = MimirTrackingObservation.PsMove( + "nightwing:move:primary", + "psmove:nightwing:00", + sequence: 7, + sourceTimestampNs: 1_000_008_333L, + arrivalTimestampNs: 1_000_012_900L, + positionMeters: new MimirVector3Snapshot(-0.34, 1.05, 1.42), + orientation: new MimirQuaternionSnapshot(0.0, 0.18, 0.0, 0.984), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(-0.02, 0.01, 0.03), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.2, 0.0, 0.0), + confidence: 0.89, + calibrationId: "nightwing-move-usb-calibration-v1", + trackingSpaceId: "nightwing-move-space", + producerId: "muninn:nightwing:move", + hostId: "nightwing", + latencyMilliseconds: 4.6, + battery01: 0.67); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions @@ -1597,10 +1735,18 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) eveSurface, new CultRecordHandle(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}"))) .ConfigureAwait(false); + await cache.UpsertAsync( + starfireMove, + new CultRecordHandle(new CultRecordKey($"mimir-move-tracking:{starfireMove.ObservationId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + nightwingMove, + new CultRecordHandle(new CultRecordKey($"mimir-move-tracking:{nightwingMove.ObservationId}"))) + .ConfigureAwait(false); await cache.FlushAsync().ConfigureAwait(false); Console.WriteLine( - $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length}"); + $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length} moveTracking=2 hosts={starfireMove.HostId},{nightwingMove.HostId}"); return 0; } @@ -5602,6 +5748,11 @@ public sealed record ComplexContourBandCorrection( double Reliability, bool Usable); +sealed record MoveTrackingSmokeContext(MimirSynchronizationHub Hub, int Consumed) : IDisposable +{ + public void Dispose() => Hub.Dispose(); +} + public sealed record ComplexContourReflectionCorrection( double RelativeDelaySamples, int ObservationCount, diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index f50a46c..f485689 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -862,6 +862,11 @@ direct return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {block.Channels}ch {block.SampleRate}Hz {block.SampleFormat} frames {block.FrameCount} bytes {latest.Value.ByteLength} edge {buffer.EdgeNs}"; } + if (latest?.TrackingObservation is { } tracking) + { + return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {tracking.Kind} device={tracking.DeviceId} producer={tracking.ProducerId} host={tracking.HostId} confidence={tracking.Confidence:0.00} edge {buffer.EdgeNs}"; + } + return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} edge {buffer.EdgeNs}"; } diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index b09d387..1b84f06 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -47,8 +47,8 @@ public static class MimirModuleLibrary new( "cultmesh-contracts", "Mimir.Runtime", - "Typed state documents for codebooks, decoder state, acoustic path state, and actuator state.", - ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state"], + "Typed state documents for codebooks, decoder state, acoustic path state, actuator state, program state, and Move tracking observations.", + ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation"], ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]), new( "path-learning", diff --git a/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs index ba54c21..6fd188d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs @@ -132,6 +132,48 @@ public static class MimirNativeCaptureConfigurations Notes = "Remote co-streamer audio witness. It decodes locally and ships typed timing/path state back over CultMesh." }; + public static MimirNativeCaptureDeviceProfile StarfirePsMoves { get; } = new( + "starfire-ps-moves", + "Starfire PS Move tracking witnesses", + MimirStreamKind.Tracking, + MimirNativeCaptureTransport.VendorSdk, + MimirCaptureClockDomain.DeviceHardware, + ChannelCount: 2, + PreferredWidth: 0, + PreferredHeight: 0, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 120.0, + DefaultRollingBufferSeconds, + Required: false, + OwnedOutputs: ["local-controller-pose-observations", "performance-motion-cues"], + "USB-attached local Moves. Mimir buffers the observations; tracking code owns hardware pose estimation."); + + public static MimirNativeCaptureDeviceProfile NightwingPsMoves { get; } = StarfirePsMoves with + { + Id = "nightwing-ps-moves", + DisplayName = "Nightwing PS Move tracking witnesses", + Transport = MimirNativeCaptureTransport.NetworkCultMesh, + ClockDomain = MimirCaptureClockDomain.NetworkSender, + OwnedOutputs = ["remote-controller-pose-observations", "performance-motion-cues"], + Notes = "USB-attached Nightwing Moves. Muninn on Nightwing publishes typed pose/button observations; Odin owns discovery/schema projection; Mimir owns buffering and fusion." + }; + + public static MimirNativeCaptureDeviceProfile OdinMoveTrackingCatalog { get; } = new( + "odin-move-tracking-catalog", + "Odin Move tracking discovery catalog", + MimirStreamKind.Tracking, + MimirNativeCaptureTransport.NetworkCultMesh, + MimirCaptureClockDomain.CanonicalFitted, + ChannelCount: 4, + PreferredWidth: 0, + PreferredHeight: 0, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 120.0, + DefaultRollingBufferSeconds, + Required: false, + OwnedOutputs: ["move-stream-discovery", "tracking-schema-rendezvous"], + "Discovery/projection surface only. Odin names Starfire and Nightwing Move streams; it does not own pose capture or Mimir fusion."); + public static IReadOnlyList LocalSixCameraProfiles { get; } = [ LeapLeftIr, @@ -146,6 +188,9 @@ public static class MimirNativeCaptureConfigurations [ .. LocalSixCameraProfiles, ScarlettStarfire, - ScarlettRaven + ScarlettRaven, + StarfirePsMoves, + NightwingPsMoves, + OdinMoveTrackingCatalog ]; } diff --git a/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs b/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs index 04ff34e..5ea9c63 100644 --- a/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs +++ b/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs @@ -20,7 +20,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource ulong payloadHandle, int byteLength = 0, ReadOnlyMemory data = default, - MimirVideoFrameDescriptor? videoFrame = null) + MimirVideoFrameDescriptor? videoFrame = null, + MimirTrackingObservation? trackingObservation = null) { samples.Enqueue(new MimirStreamSample( Descriptor.SourceId, @@ -32,7 +33,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource payloadHandle, byteLength, data, - videoFrame)); + videoFrame, + TrackingObservation: trackingObservation)); } public void PushVideoFrame( @@ -54,6 +56,25 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource frame); } + public void PushTrackingObservation(MimirTrackingObservation observation) + { + if (Descriptor.Kind != MimirStreamKind.Tracking) + { + throw new InvalidOperationException("Tracking observations can only be pushed into a tracking stream source."); + } + + if (!string.Equals(observation.StreamId, Descriptor.SourceId, StringComparison.Ordinal)) + { + throw new ArgumentException("Tracking observation stream id does not match this stream source.", nameof(observation)); + } + + Push( + observation.SourceTimestampNs, + observation.ArrivalTimestampNs, + payloadHandle: 0, + trackingObservation: observation); + } + public bool TryRead(out MimirStreamSample sample) { return samples.TryDequeue(out sample); diff --git a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs index d872b9b..ad44300 100644 --- a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs +++ b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs @@ -155,6 +155,7 @@ public static class MimirPerfectMachineProfiles [ "muninn.telemetry_surface.v1", "muninn.capture_stream.v1", + "mimir.move_tracking_observation.v1", "mimir.cultmesh_media_frame.v1" ]); diff --git a/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs b/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs index 8fcb617..4806622 100644 --- a/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs +++ b/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs @@ -4,6 +4,7 @@ public enum MimirStreamKind { Audio, Video, + Tracking, } public enum MimirStreamOrigin diff --git a/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs b/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs index 00ab704..57cf42a 100644 --- a/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs +++ b/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs @@ -11,4 +11,5 @@ public readonly record struct MimirStreamSample( int ByteLength = 0, ReadOnlyMemory Data = default, MimirVideoFrameDescriptor? VideoFrame = null, - MimirAudioBlockDescriptor? AudioBlock = null); + MimirAudioBlockDescriptor? AudioBlock = null, + MimirTrackingObservation? TrackingObservation = null); diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs index 4a16aed..c9bfd73 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs @@ -51,6 +51,15 @@ public sealed class MimirSynchronizationHub : IDisposable public IReadOnlyList AudioChirpBinCalibrationProfiles => audioSynchronization.LastCalibrationProfiles; + public IReadOnlyList TrackingObservations => + Buffers.Buffers + .Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking) + .Select(buffer => buffer.Latest?.TrackingObservation) + .Where(observation => observation != null) + .Cast() + .OrderBy(observation => observation.StreamId, StringComparer.Ordinal) + .ToArray(); + public MimirChirpBinCodebookPlan? ChirpBinEmissionPlan => chirpBinCalibrationModel?.EmissionPlan; @@ -87,7 +96,8 @@ public sealed class MimirSynchronizationHub : IDisposable { var audio = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio); var video = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Video); - return $"{video} video / {audio} audio buffers"; + var tracking = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking); + return $"{video} video / {audio} audio / {tracking} tracking buffers"; } public IReadOnlyList AnalyzeAudioSynchronization( diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs index 8a5e054..b0b0b9d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs @@ -14,8 +14,10 @@ public sealed class MimirSynchronizationSettings var streams = new List(); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.LocalDevice); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.LocalDevice); + AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.LocalDevice); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.Network); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.Network); + AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.Network); return new MimirSynchronizationSettings { diff --git a/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs b/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs new file mode 100644 index 0000000..fc18182 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs @@ -0,0 +1,100 @@ +using MessagePack; +using GameCult.Caching; + +namespace Mimir.Runtime.Synchronization; + +public enum MimirTrackingObservationKind +{ + Unknown, + PsMoveController, + Hand, + Headset, + Tool, +} + +[MessagePackObject] +public readonly record struct MimirVector3Snapshot( + [property: Key(0)] double X, + [property: Key(1)] double Y, + [property: Key(2)] double Z); + +[MessagePackObject] +public readonly record struct MimirQuaternionSnapshot( + [property: Key(0)] double X, + [property: Key(1)] double Y, + [property: Key(2)] double Z, + [property: Key(3)] double W); + +[MessagePackObject] +public sealed record MimirTrackingButtonSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] bool Pressed, + [property: Key(2)] double Value); + +[MessagePackObject] +[CultDocument("mimir.move_tracking_observation", "mimir.move_tracking_observation.v1")] +public sealed record MimirTrackingObservation( + [property: Key(0)] + [property: CultName] + string ObservationId, + [property: Key(1)] string StreamId, + [property: Key(2)] string DeviceId, + [property: Key(3)] MimirTrackingObservationKind Kind, + [property: Key(4)] string ProducerId, + [property: Key(5)] string HostId, + [property: Key(6)] string DiscoveryProviderId, + [property: Key(7)] string TrackingSpaceId, + [property: Key(8)] string CalibrationId, + [property: Key(9)] long SourceTimestampNs, + [property: Key(10)] long ArrivalTimestampNs, + [property: Key(11)] ulong Sequence, + [property: Key(12)] MimirVector3Snapshot PositionMeters, + [property: Key(13)] MimirQuaternionSnapshot Orientation, + [property: Key(14)] MimirVector3Snapshot LinearVelocityMetersPerSecond, + [property: Key(15)] MimirVector3Snapshot AngularVelocityRadiansPerSecond, + [property: Key(16)] double Confidence, + [property: Key(17)] double LatencyMilliseconds, + [property: Key(18)] double Battery01, + [property: Key(19)] MimirTrackingButtonSnapshot[] Buttons) +{ + public static MimirTrackingObservation PsMove( + string streamId, + string deviceId, + ulong sequence, + long sourceTimestampNs, + long arrivalTimestampNs, + MimirVector3Snapshot positionMeters, + MimirQuaternionSnapshot orientation, + MimirVector3Snapshot linearVelocityMetersPerSecond, + MimirVector3Snapshot angularVelocityRadiansPerSecond, + double confidence, + string calibrationId, + string trackingSpaceId = "nightwing-move-space", + string producerId = "muninn:nightwing:move", + string hostId = "nightwing", + string discoveryProviderId = "odin", + double latencyMilliseconds = 0.0, + double battery01 = double.NaN, + MimirTrackingButtonSnapshot[]? buttons = null) => + new( + $"{streamId}:{sequence}", + streamId, + deviceId, + MimirTrackingObservationKind.PsMoveController, + producerId, + hostId, + discoveryProviderId, + trackingSpaceId, + calibrationId, + sourceTimestampNs, + arrivalTimestampNs, + sequence, + positionMeters, + orientation, + linearVelocityMetersPerSecond, + angularVelocityRadiansPerSecond, + Math.Clamp(confidence, 0.0, 1.0), + Math.Max(0.0, latencyMilliseconds), + double.IsFinite(battery01) ? Math.Clamp(battery01, 0.0, 1.0) : double.NaN, + buttons ?? []); +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 5d1df26..ab7ff83 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -123,3 +123,4 @@ {"ts": "2026-06-08T09:59:22+01:00", "type": "verification", "status": "blocked", "note": "CultMesh media bridge verification: Yggdrasil UFW was opened for 3075/udp on wg-gamecult and Starfire receiver now connects from 10.77.0.2. Synthetic sender on Yggdrasil connects and sends chunks without User Not Verified after CultLib pre-verification schema-v0 policy patch, but relay cache does not receive mimir.cultmesh_media_frame updates and Starfire receiver logs no frames. Remaining blocker is CultNet document put/apply path, not WireGuard routing or OBS endpoint."} {"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."} {"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."} +{"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."} diff --git a/state/map.yaml b/state/map.yaml index 8f70fd0..2c038ae 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -33,6 +33,9 @@ constraints: memory. - Networked feeds are producers into the live runtime, not independent clock authorities. + - USB-attached Moves on Starfire and Nightwing are tracking witnesses. Odin + may discover/project their stream schemas, but it does not own pose capture + or Mimir fusion. - CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but Mimir owns program selection, composition, preview/control, and publication. @@ -63,6 +66,9 @@ invariants: response, and compute-offload work need typed mesh-syncable state surfaces so phones, microcontrollers, Raven, and Starfire can cooperate without becoming independent timing authorities. + - Move tracking observations are stream samples, not program controls or + dashboard truth. Starfire-local and Nightwing-remote Move samples must enter + Mimir.Runtime as typed tracking buffers before Fensalir or Eve acts on them. accepted_design: - `Mimir.slnx` owns the app surface. @@ -74,6 +80,11 @@ accepted_design: device timestamp, and optional native/GPU handle. - `MimirAudioBlockDescriptor` carries sample rate, channels, sample format, frame count, timestamp, and optional native handle. + - `MimirTrackingObservation` carries typed PS Move pose/button/battery/ + confidence samples as `mimir.move_tracking_observation.v1`; Starfire Moves + are local tracking streams, Nightwing Moves are emitted by Muninn on + Nightwing, Odin owns discovery/schema projection, and Mimir owns buffering + and fusion. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the