Compare commits

..
Author SHA1 Message Date
Meta Vens
c6f4541eea State: EveSensorReceiver is live and frozen until Thing 2, not archived (thing2:ruling:mimir-sensor-receiver-frozen)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 23:25:39 +02:00
Meta Vens
43cebb6fb9 Quarantine Mimir's Eve operator surface: remove EveDashboard, EveBrowserReference and their contract documents
Parked at tag parked/Mimir-eve-surface per thing2:ruling:eve-dashboards-quarantined; rebuild on Thing 2. Mimir.EveSensorReceiver (camera frame ingest) stays. map.yaml no longer calls a live listener a tombstone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 20:14:47 +02:00
Meta Vens
9481fbaf44 Merge readme-front-door: README opens with value, problem, entry points, live versus planned 2026-10-22 20:05:05 +02:00
Meta Vens
e867e44cbc README: front door with value, problem, entry points, live versus planned; quick start copies the example config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:04:52 +02:00
Meta Vens
c08e1a16cd Pin CultLib 2.0.0 for builds
Directory.Build.props names CultLibRoot (CULTLIB_ROOT, else the sibling checkout) and pins CultLibRevision to cultlib-unity-v2.0.0 (42dce361). Directory.Build.targets fails the build unless that checkout is at the pin and clean, as Aetheria's guard does. Project references use CultLibRoot.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 19:46:49 +02:00
Meta Vens
d88f2e6dc1 Flush the media cache without the ignored soft flag
CultCache.FlushAsync(bool soft) ignored soft and CultLib is deleting the
parameter; the media put path now calls FlushAsync().

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 14:12:28 +02:00
Meta Vens
5591c5c47c Follow CultLib generator deletion: always serialize through MessagePack
CultDocumentDescriptor no longer carries a generated payload serializer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 23:38:14 +02:00
Meta Vens
aa36b67dd6 Follow CultLib Cut 3: drop PullOnOpen from BufferSmoke
CultLib's CultCacheMessagePack.OpenAsync now always hydrates and
CultCacheOpenOptions.PullOnOpen no longer exists. The five BufferSmoke opens
passed PullOnOpen = true or File.Exists(path); hydrating a missing store is
empty, so each becomes a plain OpenAsync(path).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 17:59:50 +02:00
Meta Vens
4d7378216f Wire AGENTS.md into Claude Code memory
Claude Code reads CLAUDE.md, not AGENTS.md, so this repo's doctrine was
invisible to it. This one-line import makes the existing AGENTS.md load
without duplicating or renaming it. Codex is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:23:05 +02:00
Meta Vens
ce98c421dc Record mixed-stream input latency proof 2026-07-17 11:20:30 +01:00
Meta Vens
7468ba59bc Record ordered input field closure 2026-07-17 10:40:06 +01:00
Meta Vens
fea4d27dd1 Document Raven Moonlight field acceptance 2026-07-17 08:12:21 +01:00
Meta Vens
78ca783276 gate Muninn decode epochs and retry repair 2026-07-17 07:24:41 +01:00
Meta Vens
42d6d173c3 recover compressed Muninn media under loss 2026-07-17 06:48:07 +01:00
Meta Vens
1221583c23 Harden Muninn OBS recovery from Moonlight field study 2026-07-17 05:48:31 +01:00
Meta Vens
12489075d7 Record Muninn frame-wide FEC limit 2026-07-17 02:58:49 +01:00
Meta Vens
02c615670a Ignore late shards from completed Muninn frames 2026-07-17 02:34:51 +01:00
Meta Vens
79781abdbb Record Muninn trailing parity burst scar 2026-07-17 01:59:33 +01:00
Meta Vens
e9e7eaae30 Bound Muninn video FEC decode time 2026-07-17 01:46:28 +01:00
Meta Vens
afce0245b1 Recover Muninn video bursts with bounded FEC 2026-07-17 01:37:28 +01:00
Meta Vens
caf92d9dee Make Muninn OBS receiver reload-safe 2026-07-16 23:10:56 +01:00
Meta Vens
60cbe45624 Record Raven field-test outage 2026-07-16 21:38:24 +01:00
Meta Vens
b0f33c36b4 Define Moonlight reliability acceptance gate 2026-07-16 21:35:47 +01:00
Meta Vens
d3d461be93 Record receiver-driven IDR proof 2026-07-16 21:26:00 +01:00
Meta Vens
6a7aaea019 Record Muninn loss recovery authority 2026-07-16 21:04:52 +01:00
Meta Vens
58e5556322 Recover bounded Muninn audio loss 2026-07-16 21:02:01 +01:00
Meta Vens
074ad4a69b Repair Muninn receiver recovery contracts 2026-07-16 20:49:17 +01:00
Meta Vens
45ead73ae3 Record realtime delivery hardening 2026-07-16 20:30:10 +01:00
Meta Vens
5706fcd99d Persist Mimir Face state 2026-07-16 20:04:09 +01:00
Meta Vens
c3775ec150 Document Moonlight streaming lessons 2026-07-16 20:04:05 +01:00
Meta Vens
345c46bd95 Reconnect Move evidence after provider restart 2026-07-14 01:31:45 +01:00
Meta Vens
a363c1d32f Update Mimir Face state 2026-07-14 01:13:44 +01:00
Meta Vens
0efa7f3307 Record filtered Move observation evidence 2026-07-14 01:13:41 +01:00
Meta Vens
2eb32602d7 Update Mimir Face state 2026-07-14 01:00:30 +01:00
Meta Vens
58af018125 Add robust Move sphere stereo fitting 2026-07-14 01:00:17 +01:00
Meta Vens
c01ef5096c Add Mimir-owned Move calibration sessions 2026-07-14 00:35:47 +01:00
Meta Vens
11e3086ee4 Preserve Mimir Face state 2026-07-13 02:39:46 +01:00
Meta Vens
d27f0d2d37 Gate Move stereo calibration on observable evidence 2026-07-13 02:39:41 +01:00
Meta Vens
a741aef9bb Preserve Mimir Face state 2026-07-13 02:32:44 +01:00
Meta Vens
95bed8f5f4 Record stable dual-Eye Move cadence proof 2026-07-13 02:32:38 +01:00
Meta Vens
83f89d4020 Record full dual-eye Move overlap 2026-07-13 01:41:29 +01:00
Meta Vens
810073efaa Preserve Mimir Face state 2026-07-13 01:29:25 +01:00
Meta Vens
2196503d54 Map live dual-eye Move tracking state 2026-07-13 01:29:16 +01:00
Meta Vens
c1939b72f5 Remember isolated dual-eye tracker field proof 2026-07-13 01:15:00 +01:00
Meta Vens
bf110a6ace Remember dual-eye PSMoveAPI isolation proof 2026-07-13 00:48:00 +01:00
Meta Vens
e757538eda Expose live Move evidence receiver failures 2026-07-13 00:29:24 +01:00
Meta Vens
150a054344 Measure Move visibility within one Muninn provider 2026-07-13 00:22:09 +01:00
Meta Vens
f80881e0d7 Measure persistent dual-camera Move visibility 2026-07-13 00:17:00 +01:00
Meta Vens
3e627f5484 Remember Move tracking false blocker 2026-07-13 00:10:14 +01:00
Meta Vens
ee456bbcc1 Probe arbitrary Muninn Move evidence providers 2026-07-13 00:01:52 +01:00
Meta Vens
f742a4de97 Require distinct cameras in live optical smoke 2026-07-12 22:49:03 +01:00
Meta Vens
c815d1ba58 Wait for stable-ID optical Move evidence 2026-07-12 21:39:24 +01:00
Meta Vens
74a1918996 Associate Move optical markers by stable identity 2026-07-12 21:11:27 +01:00
Meta Vens
522bf41234 Record live Move Bluetooth handoff 2026-07-12 21:00:36 +01:00
Meta Vens
6778e94975 Update Mimir Face state 2026-07-12 20:50:41 +01:00
Meta Vens
95d603aa80 Record live Muninn Move migration evidence 2026-07-12 20:50:41 +01:00
Meta Vens
538b95e2c3 Discover Muninn Move evidence through Odin 2026-07-12 20:18:30 +01:00
Meta Vens
de22059725 Persist Mimir Face state 2026-07-12 11:17:25 +01:00
Meta Vens
ec86a482a0 Record Muninn Move proof snapshot producer 2026-07-12 11:17:21 +01:00
Meta Vens
c0e53135ba Persist Mimir Face state 2026-07-12 11:10:21 +01:00
Meta Vens
dd45a478a6 Add Move proof evidence snapshot fallback 2026-07-12 11:10:17 +01:00
Meta Vens
2de98df804 Persist Mimir Face state 2026-07-12 11:01:39 +01:00
Meta Vens
32bf6fa518 Add Move proof activation surface 2026-07-12 11:01:29 +01:00
Meta Vens
4a968319ac Persist Mimir Face heartbeat 2026-07-12 10:53:52 +01:00
Meta Vens
c32d01b7e5 Activate configured Move proof sources 2026-07-12 10:53:37 +01:00
Meta Vens
dc51918227 Persist Mimir Face state 2026-07-12 10:45:50 +01:00
Meta Vens
83ddd3f509 Add Move proof runtime configuration 2026-07-12 10:41:48 +01:00
Meta Vens
293a953b2d Prove Move proof surface in presented frame 2026-07-08 22:56:52 +01:00
Meta Vens
20a296a4a9 Drive Move proof surface from runtime ring 2026-07-08 22:49:42 +01:00
Meta Vens
02c6bfc533 Attach Move proof surface to runtime frame 2026-07-08 22:42:20 +01:00
Meta Vens
7d96c42a3c Add Mimir Move proof pipeline 2026-07-08 22:35:32 +01:00
Meta Vens
a040792a1c Add Mimir Move proof surface 2026-07-08 22:29:23 +01:00
Meta Vens
7e2d35ad59 Make Muninn OBS bridge allocate decoder ports 2026-07-06 00:45:08 +01:00
Meta Vens
d6f6400677 Hide Muninn OBS CLI discovery launches 2026-07-05 23:42:45 +01:00
Meta Vens
ed46270be6 Make Muninn OBS discovery visibly Odin-owned 2026-07-05 23:35:32 +01:00
Meta Vens
e00458ff3b Restore Muninn OBS plugin with Odin discovery 2026-07-05 22:35:50 +01:00
Meta Vens
d9005e340b Ignore stale Muninn Move identities in live roster 2026-06-17 14:07:35 +01:00
Meta Vens
8042885c6d Record cross-host Move roster smoke 2026-06-17 13:29:44 +01:00
Meta Vens
c7fa6fb8ac Record Starfire Move paired host verification 2026-06-17 13:17:51 +01:00
Meta Vens
79e7865671 Record Nightwing Muninn hot release deploy 2026-06-17 12:41:59 +01:00
Meta Vens
d53ba0c82c Record Move wake blocker evidence 2026-06-17 12:15:07 +01:00
Meta Vens
6192330b49 Record unreachable Move roster verification 2026-06-17 12:02:11 +01:00
Meta Vens
32bd680cce Distinguish unreachable Muninn Moves 2026-06-17 11:55:20 +01:00
Meta Vens
5eef53c0e2 Record Muninn Move pickup verification 2026-06-17 11:33:34 +01:00
Meta Vens
d3d4b14635 Add live Muninn Move roster smoke 2026-06-17 10:16:09 +01:00
Meta Vens
ff0920b327 Clarify Muninn Move pickup readiness 2026-06-17 09:12:49 +01:00
Meta Vens
c3babfaa57 Project Muninn Move identity roster 2026-06-17 06:48:44 +01:00
Meta Vens
3103a7aa27 Ingest Muninn Move identity records 2026-06-17 03:49:55 +01:00
Meta Vens
b1a6043d7e Publish browser reference boundary witness 2026-06-16 20:31:31 +01:00
Meta Vens
4ed4ba190b Migrate Mimir media bridge to RUDP 2026-06-16 18:08:40 +01:00
Meta Vens
bd19e60ae1 Refresh Mimir Face state 2026-06-16 17:39:51 +01:00
Meta Vens
434e2d3efa Reorient Mimir maps around CultNet RUDP 2026-06-16 17:39:33 +01:00
Meta Vens
7d07e38e52 Run Nightwing Eve browser reference as RUDP daemon 2026-06-16 09:50:06 +01:00
Meta Vens
21c2ee43f0 Publish Nightwing Eve dashboard health over Idunn RUDP 2026-06-16 08:39:38 +01:00
Meta Vens
e3987842de Publish Mimir Eve dashboard health over Idunn RUDP 2026-06-16 08:04:25 +01:00
Meta Vens
34c435f942 Refresh Mimir Face state 2026-06-14 19:40:52 +01:00
Meta Vens
3ebf7cc519 Make Starfire Move light helper hotplug aware 2026-06-14 19:40:03 +01:00
Meta Vens
bff68e5b58 Add repeated PS Move read probe 2026-06-14 14:42:30 +01:00
Meta Vens
c23b392787 Mirror Muninn Move source path 2026-06-14 14:30:54 +01:00
Meta Vens
3ec2b01788 Refresh Mimir Face state 2026-06-14 14:30:34 +01:00
Meta Vens
21b30c6dd4 Update Mimir Face maintenance status 2026-06-12 16:42:04 +01:00
Meta Vens
c1178512c4 Move Quest access ownership to Muninn 2026-06-12 16:41:15 +01:00
Meta Vens
09d0d3ad2e Refresh Mimir Face maintenance status 2026-06-12 15:20:46 +01:00
Meta Vens
00c9053430 Add Quest USB calibration preflight 2026-06-12 15:20:19 +01:00
Meta Vens
0616d21a5b Use Quest as Move calibration witness 2026-06-12 15:04:51 +01:00
Meta Vens
4ae3a07ed6 Refresh Mimir Face heartbeat state 2026-06-12 14:50:53 +01:00
Meta Vens
74f2455e69 Add Move calibration protocol preflight 2026-06-12 14:50:12 +01:00
Meta Vens
510242bc50 Expose Move pose stream in module catalog 2026-06-12 14:26:50 +01:00
Meta Vens
dd838e9b1a Refresh Mimir Face state 2026-06-12 14:21:59 +01:00
Meta Vens
43af2a24bd Add calibrated Move pose fusion stream 2026-06-12 14:20:28 +01:00
Meta Vens
dccf3b1599 Persist Mimir face heartbeat 2026-06-11 20:33:51 +01:00
Meta Vens
07852766fd Use CultMesh stream for Muninn Move evidence 2026-06-11 20:28:30 +01:00
Meta Vens
6848f762f0 Persist Mimir face heartbeat 2026-06-11 20:04:33 +01:00
Meta Vens
091ab50d7c Map Muninn Move evidence into native tracking 2026-06-11 20:01:01 +01:00
Meta Vens
58c3c40b72 Wire Move evidence into native reservoir 2026-06-11 19:31:21 +01:00
Meta Vens
5b4c391607 Persist Mimir face state 2026-06-11 18:51:52 +01:00
Meta Vens
0d1f97436a Add native Move evidence reservoir ABI 2026-06-11 18:50:56 +01:00
Meta Vens
c9fca62346 Persist Mimir face state 2026-06-11 17:02:26 +01:00
Meta Vens
231573ed79 Move wand fusion authority to Mimir 2026-06-11 17:00:49 +01:00
Meta Vens
4d661967ae Route Move light ownership through Muninn 2026-06-11 07:58:26 +01:00
Meta Vens
ec0f7454d3 Persist Mimir face state 2026-06-11 07:57:59 +01:00
Meta Vens
147e6ab5af Update Mimir Face state 2026-06-10 20:52:58 +01:00
Meta Vens
8716ea9c0b Demote Move marker extraction to Muninn 2026-06-10 20:52:28 +01:00
Meta Vens
17d2f56de8 Update Mimir Face state 2026-06-10 20:36:21 +01:00
Meta Vens
c2e5d03764 Add Rust Move sphere candidate tracker 2026-06-10 20:35:35 +01:00
Meta Vens
1692c3ec64 Update Mimir Face state 2026-06-10 18:13:26 +01:00
Meta Vens
4af3864051 Add Starfire Move light owner 2026-06-10 18:12:32 +01:00
Meta Vens
601367b754 Update Mimir Face state 2026-06-10 17:42:28 +01:00
Meta Vens
52a439842c Restore Nightwing Move tracking witness 2026-06-10 17:41:18 +01:00
Meta Vens
e85c05029c Update Mimir Face state 2026-06-10 16:44:35 +01:00
Meta Vens
6dc18ecf18 Add Move tracking observation streams 2026-06-10 16:43:12 +01:00
Meta Vens
28536681f6 Import OBS scene into Mimir program state 2026-06-09 03:04:51 +01:00
Meta Vens
2c68485c35 Update Mimir Face state 2026-06-09 03:03:46 +01:00
Meta Vens
3dcc392080 Move program composition authority into Mimir 2026-06-09 02:21:32 +01:00
Meta Vens
7a811ee329 Update Mimir Face state 2026-06-09 02:19:17 +01:00
107 changed files with 20823 additions and 633 deletions

Binary file not shown.

View file

@ -1,17 +1,7 @@
{
"durationSeconds": 143.87,
"stateUpdated": true,
"exitCode": 0,
"finishedAt": "2026-05-20T13:30:04.9837661Z",
"stateFile": "E:\\Projects\\LocalCastBridge\\.voidbot\\state\\mimir.cc",
"skipModel": false,
"startedAt": "2026-05-20T13:27:41.1114814Z",
"lastMessagePath": "E:\\Projects\\LocalCastBridge\\.voidbot\\status\\void-memory-maintenance-last-message.txt",
"appliedOperationCount": 1,
"status": "ok",
"contextPath": "E:\\Projects\\LocalCastBridge\\.voidbot\\status\\void-memory-maintenance-context.json",
"mode": "typed_memory_maintenance",
"proposedOperationCount": 1,
"forceDistillation": true,
"operationOutputPath": "E:\\Projects\\LocalCastBridge\\.voidbot\\status\\void-memory-maintenance-operations.json"
}
"status": "starting",
"startedAt": "2026-07-15T02:30:47.016Z",
"stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc",
"launcher": "repo_face_heartbeat",
"reason": "repo_face_stale_short_term_memory"
}

View file

@ -6,7 +6,7 @@
"roleId": "1506268007880462459",
"avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png",
"repoPath": "E:\\Projects\\Mimir",
"createdOrRefreshedAt": "2026-05-25T05:17:24.094Z",
"createdOrRefreshedAt": "2026-07-15T01:59:42.286Z",
"avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png",
"description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen."
}

3
CLAUDE.md Normal file
View file

@ -0,0 +1,3 @@
Repository doctrine for this project lives in AGENTS.md, shared with Codex.
@AGENTS.md

10
Directory.Build.props Normal file
View file

@ -0,0 +1,10 @@
<Project>
<PropertyGroup>
<CultLibRoot Condition="'$(CultLibRoot)' == '' and '$(CULTLIB_ROOT)' != ''">$(CULTLIB_ROOT)</CultLibRoot>
<CultLibRoot Condition="'$(CultLibRoot)' == ''">$(MSBuildThisFileDirectory)..\CultLib</CultLibRoot>
<!-- CultLib 2.0.0: tag cultlib-unity-v2.0.0. -->
<CultLibRevision>42dce3611a81adc5c5fac54cadbaa5637d6022d3</CultLibRevision>
<GitExecutable Condition="'$(GitExecutable)' == '' and Exists('C:\Program Files\Git\cmd\git.exe')">C:\Program Files\Git\cmd\git.exe</GitExecutable>
<GitExecutable Condition="'$(GitExecutable)' == ''">git</GitExecutable>
</PropertyGroup>
</Project>

30
Directory.Build.targets Normal file
View file

@ -0,0 +1,30 @@
<Project>
<!-- Builds consume CultLib at one admitted revision. Point CultLibRoot (or CULTLIB_ROOT) at a clean
worktree of that revision: git worktree add <path> cultlib-unity-v2.0.0 (detached) (in CultLib). -->
<Target Name="VerifyCultLibRevision"
BeforeTargets="ResolveProjectReferences"
Condition="'$(SkipCultLibRevisionCheck)' != 'true'">
<ItemGroup>
<_CultLibReference Include="@(ProjectReference)"
Condition="$([System.String]::Copy('%(Filename)').StartsWith('GameCult.')) or '%(Filename)' == 'CultMath'" />
</ItemGroup>
<Error Condition="'@(_CultLibReference)' != '' and !Exists('$(CultLibRoot)\src\GameCult.Caching\GameCult.Caching.csproj')"
Text="CultLibRoot '$(CultLibRoot)' is not a CultLib checkout." />
<Exec Condition="'@(_CultLibReference)' != ''"
Command="&quot;$(GitExecutable)&quot; -C &quot;$(CultLibRoot)&quot; rev-parse HEAD"
ConsoleToMSBuild="true"
StandardOutputImportance="Low">
<Output TaskParameter="ConsoleOutput" PropertyName="ActualCultLibRevision" />
</Exec>
<Error Condition="'@(_CultLibReference)' != '' and '$([System.String]::Copy($(ActualCultLibRevision)).Trim())' != '$(CultLibRevision)'"
Text="CultLib must be revision $(CultLibRevision) (cultlib-unity-v2.0.0); found $([System.String]::Copy($(ActualCultLibRevision)).Trim()). Set CultLibRoot to a clean worktree of that tag." />
<Exec Condition="'@(_CultLibReference)' != ''"
Command="&quot;$(GitExecutable)&quot; -C &quot;$(CultLibRoot)&quot; status --porcelain --untracked-files=normal"
ConsoleToMSBuild="true"
StandardOutputImportance="Low">
<Output TaskParameter="ConsoleOutput" PropertyName="CultLibStatus" />
</Exec>
<Error Condition="'@(_CultLibReference)' != '' and '$([System.String]::Copy($(CultLibStatus)).Trim())' != ''"
Text="CultLibRoot '$(CultLibRoot)' must be clean before this repo consumes it." />
</Target>
</Project>

View file

@ -3,5 +3,9 @@
<Project Path="src/Mimir.App/Mimir.App.csproj" />
<Project Path="src/Mimir.Runtime/Mimir.Runtime.csproj" />
<Project Path="src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj" />
<Project Path="src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj" />
<Project Path="src/Mimir.VerseRecorder/Mimir.VerseRecorder.csproj" />
<Project Path="src/Mimir.PsMoveProbe/Mimir.PsMoveProbe.csproj" />
<Project Path="src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj" />
</Folder>
</Solution>

View file

@ -8,6 +8,18 @@ OBS-facing program surface.
The plain version: Mimir is trying to make a room computable.
## Front Door
- **Value:** a measured room: cameras, microphones, loopbacks and network feeds aligned in time tightly enough to recover where a sound came from and how a body moved, then handed to OBS as one program surface.
- **Problem:** loose devices each keep their own clock, so a flat stream cannot localize voices, separate sources or fuse cameras. Mimir owns the synchronization evidence that makes the feeds one field.
- **Entry points:**
- `docs/perfect-machine.md`: the target machine; `docs/viable-stream-app.md`: the first usable app cut; `docs/audio-field.md`: the live audio decoder and timing model.
- `src/Mimir.Runtime` (rolling buffers, stream sources, synchronization state) and `src/Mimir.App` (the Fensalir-hosted window and render app); solution `Mimir.slnx`.
- `src/Mimir.BufferSmoke`: the synthetic sync proofs, run with `dotnet run --project src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test` (also `--chirp-bin-self-test`, `--passive-sync-self-test`).
- `native/probes/asio_audio_cadence` (ASIO cadence probe) and `native/reservoir` (native rolling-buffer ABI).
- **Live:** passive, chirp-only and hybrid audio sync with synthetic proofs, the ASIO cadence probe and measured capture-cadence evidence, the runtime buffers, and FFmpeg/SRT bridge scripts as an OBS-compatible edge (`scripts/sender-start.ps1`).
- **Planned:** direct native capture workers with driver-boundary timestamps, Fensalir GPU fusion, the volumetric sound field, realtime 4D Gaussian splatting, motion capture and AR anchoring. The bridge scripts are not the synchronization authority.
A normal streaming setup sees loose devices. A webcam has its clock, a mic has
another clock, speakers add their own delay, and OBS receives whatever arrives.
That is enough for a flat livestream. It is not enough to know where a voice
@ -289,6 +301,7 @@ Run the old OBS bridge only when you need the compatibility path:
```powershell
.\scripts\sender-discover.ps1
Copy-Item .\config\localcast.example.json .\config\localcast.json
.\scripts\sender-start.ps1 -Config .\config\localcast.json -DryRun
```
@ -329,3 +342,14 @@ layer for communication and heartbeats. Its birth memory, jurisdiction, voice,
and heartbeat contract live in [docs/mimir-face.md](docs/mimir-face.md); its
VoidBot-facing identity and typed Face state live under `.voidbot/voice/` and
`.voidbot/state/`.
## Building against CultLib
Builds are pinned to CultLib 2.0.0 (tag `cultlib-unity-v2.0.0`, revision `42dce3611a81adc5c5fac54cadbaa5637d6022d3`); the build fails if the CultLib checkout differs or is dirty. Create a pinned worktree and point the build at it:
```powershell
git -C <CultLib> worktree add --detach <path> cultlib-unity-v2.0.0
dotnet build -p:CultLibRoot=<path> # or set CULTLIB_ROOT=<path>
```
Without an override the sibling `..\CultLib` checkout is used and must itself be at that tag.

View file

@ -31,15 +31,24 @@
"outputs": {
"video": "synchronized-program-video",
"audio": "separately-controllable-synchronized-stems"
},
"programComposition": {
"owner": "Mimir",
"sceneDocument": "mimir.program_scene.v1",
"operatorSurface": "mimir.eve_operator_surface.v1",
"outputDocument": "mimir.program_output.v1",
"sitePublisher": "Yggdrasil",
"obsRole": "compatibility-sink-only"
}
},
"authorities": {
"configurationCalibrationLaunchStatus": "Mimir",
"programComposition": "Mimir",
"sitePublication": "Yggdrasil publisher daemon",
"denseVisualFusion": "Fensalir",
"gpuMaterialAndBrushReconciliation": "Fensalir",
"spoutPublication": "Fensalir",
"hotAudioDsp": "Faust",
"obsBroadcastControls": "OBS"
"programTextureOutput": "Fensalir",
"hotAudioDsp": "Faust"
},
"visualSensors": [
{ "id": "kiyo-primary", "role": "rgb-main", "required": true },
@ -70,9 +79,9 @@
],
"outputs": {
"video": {
"target": "Spout2",
"senderName": "Mimir Point Cloud",
"owner": "Fensalir"
"target": "MimirProgramTexture",
"senderName": "Mimir Program Texture",
"owner": "Mimir/Fensalir"
},
"audio": {
"owner": "Faust",
@ -85,6 +94,20 @@
"co_streamer_loopback",
"spatial_bed"
]
},
"operator": {
"owner": "Mimir",
"surfaces": [
"cultmesh-command-surface"
]
},
"site": {
"owner": "Yggdrasil publisher daemon",
"input": "mimir.program_output.v1",
"route": "cultmesh://yggdrasil/gamecult.site/mimir/program"
},
"compatibility": {
"obs": "optional-derived-sink"
}
},
"runtimePolicy": {
@ -93,7 +116,8 @@
"native-driver-workers",
"native-reservoir",
"Fensalir",
"Faust"
"Faust",
"Yggdrasil publisher daemon"
],
"neverInCompatibilityEdges": [
"dense-feature-extraction",

View file

@ -2,11 +2,16 @@
## Objective
Make one local Windows computer available as OBS sources on another computer, with sender-side NVENC video encoding and independently mixable audio sources.
Make one local Windows computer available as compatibility media sources on
another computer, with sender-side NVENC video encoding and independently
mixable audio sources. The current program target is Mimir-owned composition;
this bridge is not the stream authority.
## Current Mechanism
The repo owns configuration and scripts. FFmpeg owns capture, encoding, and SRT transport. OBS owns scene composition.
The repo owns configuration and scripts. FFmpeg owns capture, encoding, and SRT
transport. Mimir owns scene composition; OBS may consume this bridge as a
temporary compatibility sink.
Inputs become outputs like this:
@ -21,8 +26,10 @@ Inputs become outputs like this:
## Invariants
- Video encoding happens on the sender.
- OBS receives ordinary media-source URLs; it does not need a plugin for v1.
- Audio remains separately mixable in OBS.
- OBS can receive ordinary media-source URLs for compatibility; it does not need
a plugin for this bridge path.
- Audio remains separately mixable at the compatibility edge, but the durable
control surface belongs to Mimir.
- Ports are stable and derived from one base port plus explicit offsets.
- Secrets, if used, stay in local config and are not committed.

View file

@ -15,8 +15,9 @@ Timestamp for this pass: 2026-05-23, starting at 23:24 Europe/London.
Mimir collects local and network audio/video samples into bounded rolling
runtime buffers, uses active/passive timing evidence to place audio streams on a
canonical timeline, and leaves Fensalir/Faust/OBS with clear ownership over
rendering, DSP, and broadcast composition.
canonical timeline, and lets Mimir own program composition while Fensalir lowers
pixels, Faust/native DSP moves audio, and
Yggdrasil publishes the public feed.
## Current Mechanism
@ -54,11 +55,15 @@ reflection taps.
- UI and telemetry read cached sync state; they do not run analysis.
- Native capture owns hardware calls. Runtime owns sample identity and buffer
placement.
- Fensalir owns windowing, rendering, D3D12, UI, visual fusion, and program
video output.
- Mimir owns the program scene graph, subscription policy, preview/control
state, stats, and publication intent.
- Fensalir owns windowing, rendering, D3D12, UI lowering, visual fusion, and
program video output.
- Faust/native DSP will own hot resampling, fractional delay, separation,
spatialization, and stems.
- OBS owns broadcast composition.
- The Yggdrasil-facing publisher daemon consumes Mimir program output without
owning composition.
- OBS is a compatibility sink only.
## Diagram
@ -83,6 +88,9 @@ flowchart TD
AsioSource --> Sources
Probes["native/probes/*"] --> FrameEvents["MimirFrameEventProcessStreamSource"]
FrameEvents --> Sources
Runtime --> Scene["Mimir program scene graph"]
Scene --> Publisher["Yggdrasil site publisher"]
Scene -. "compatibility" .-> Obs["OBS adapter"]
Reservoir["native/reservoir"] -. "future lower ABI" .-> Hub
```
@ -96,6 +104,9 @@ flowchart TD
- `native/reservoir` is the lower native rolling-buffer ABI.
- `native/probes/*` are measurement and diagnostic tools.
- `scripts/*` are bridge utilities, not the final synchronized program path.
- `docs/mimir-program-composition.md` is the authority map for Muninn capture
hosts, Mimir composition, Yggdrasil publication, and OBS
compatibility.
## Stage 0: Solution And App Host

View file

@ -1,59 +0,0 @@
# EVE Program Output
## Authority Map
- Owner: Fensalir owns the visible D3D12 program pixels.
- Inputs: Fensalir reads the current `AquariumFrame`, render plan, GPU resources,
and overlay state.
- Outputs: a named shared D3D12 `B8G8R8A8_UNorm` texture containing the completed
backbuffer for the current frame.
- Derived state: Mimir publication profiles describe the stream contract; they
do not render pixels. EVE receives decoded pixels and composites them.
- Forbidden writers: WebKit, DOM layout, browser dashboards, OBS capture, and
desktop/window capture do not own this path.
- Shared paths: local display and EVE streaming both use the same completed
Fensalir backbuffer copy.
- Deletion line: diagnostic Sunshine/Moonlight or desktop-duplication probes may
prove network/decode budget, but they are not production authority.
## Current Mechanism
Set these before launching the Mimir app:
```powershell
$env:FENSALIR_PROGRAM_OUTPUT_D3D12 = "1"
$env:FENSALIR_PROGRAM_OUTPUT_NAME = "Global\MimirFensalirProgramTexture"
dotnet run --project .\src\Mimir.App\Mimir.App.csproj
```
Fensalir creates a shareable D3D12 texture with that name and copies the finished
backbuffer into it during the present pass. A hardware encoder process should
open that named texture, encode H.264/HEVC with GPU hardware, and send frames to
EVE's native receiver.
`scripts/start-eve-program-output.ps1` starts the Mimir/Fensalir app with the
shared-texture output enabled and verifies the `eve` SSH target before launch.
## Receiver Contract
EVE owns:
- network receive;
- hardware decode;
- texture upload/import;
- final fullscreen composite;
- touch/Pencil/sensor uplink later.
EVE does not own:
- dashboard DOM;
- Chromium/WebKit compatibility;
- layout parity;
- Mimir timing authority;
- OBS composition authority.
## Smoke
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --eve-program-output-contract-smoke
```

View file

@ -10,15 +10,15 @@ For the source-level ownership map, read [[code-algorithm-map|Code Algorithm Map
- `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12
bridge.
- `src/Mimir.Runtime` owns stream descriptors, source polling, direct push
ingest, one rolling buffer per configured audio/video stream, and the
ingest, one rolling buffer per configured audio/video/tracking stream, and the
synchronization hub Fensalir can inspect.
- The default five-second window is an intentional latency/memory trade: use it
to line up streams and extract the volumetric audio/video field before OBS
sees program output.
to line up streams and extract the volumetric audio/video field before Mimir
emits program output.
- `native/reservoir` owns the lower native rolling-buffer invariant for
Fensalir/Faust integration.
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN OBS feeds. It is not
the synchronized program authority.
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN compatibility feeds.
It is not the synchronized program or publication authority.
The old script stack is gone. Do not add a compatibility edge unless it protects
a named invariant that the native runtime cannot protect yet.
@ -28,16 +28,169 @@ a named invariant that the native runtime cannot protect yet.
- Mimir public identity, branding, and Face memory.
- `Mimir.slnx` with `src/Mimir.App` and `src/Mimir.Runtime`.
- Fensalir host bootstrapping from `Mimir.App`.
- [[eve-program-output|EVE Program Output]] now owns the first native EVE
streaming contract: Fensalir can publish the completed D3D12 backbuffer into a
named shared texture when `FENSALIR_PROGRAM_OUTPUT_D3D12=1`, and Mimir records
the EVE-facing publication profile as a native D3D12 stream rather than a
WebKit/dashboard path.
- `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and
`IMimirStreamSource`.
- Configurable five-second default rolling buffers for local and network audio
and video streams.
and video/tracking streams.
- `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers.
- `MimirStreamKind.Tracking` plus `MimirTrackingObservation` carry PS Move
evidence samples as typed `mimir.move_tracking_observation.v1` documents.
Starfire and Nightwing both have USB-attached Moves. Muninn daemons publish
source-local glowing-orb marker candidates and controller/IMU/button state;
Odin owns discovery and schema projection, not pose fusion.
- `native/reservoir` exposes a `move_evidence` typed view and
`LocalcastMoveEvidenceBufferDescriptor` so Muninn Move witnesses can be
admitted as compute-upload buffers before Mimir's fusion pass and Fensalir's
GPU consumers touch them.
- `MimirNativeReservoirRuntime` is the managed owner for that native runtime
boundary. It pins Move evidence sample/descriptor batches so native reservoir
handles remain valid while Mimir prepares fusion input.
- `MimirMuninnMoveEvidenceAdapter` mirrors Muninn's marker-candidate and
controller-state document schemas and normalizes them into native Move
evidence samples. It does not associate markers to controllers or synthesize
pose.
- The realtime Move witness path uses CultMesh streaming, not CultCache polling:
Muninn publishes shared-memory stream frames, Mimir consumes the latest read
lease, normalizes the frame, and admits it to native `move_evidence`.
- `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a
typed admission receipt for that latest read: decoded frame id, producer,
publish/read timestamps, sample counts by kind, source/arrival ranges, native
reservoir handle, and reservoir edge/window. `Mimir.BufferSmoke
--muninn-move-cultmesh-stream-smoke` asserts the receipt so the named proof
spine can be traced instead of inferred.
- `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract.
Mimir owns calibration, association, triangulation, IMU fusion, prediction,
confidence, and latency accounting for those poses before Fensalir consumes
them as interactive controller input.
- `MimirMoveFusion` is the first calibrated fusion owner for Move tracking:
it consumes native Muninn Move evidence samples, requires calibrated camera
witnesses before publishing a pose, associates marker candidates with the
USB controller state, triangulates multi-camera orb position, demotes
single-ray fallbacks through confidence, carries buttons/trigger/gyro into
`mimir.move_controller_pose.v1`, and marks orientation as
`orientation:imu-unresolved` until the IMU/prediction pass earns full 6DoF.
- `MimirMovePoseStream` frames resolved Move poses as
`mimir.move_controller_pose_stream_frame.v1` over CultMesh shared-memory
bytes streams so Fensalir and other consumers have a realtime stream contract
for Mimir-fused controller input.
- `MimirMoveProofSurface` is the observer-only dev proof surface for the named
Move chain. It combines a Muninn evidence admission receipt with a Mimir pose
stream frame and a Fensalir presentation/probe timestamp into
`mimir.move_proof_surface.v1`, then lowers the verdict to a Fensalir
`AquariumSplineFrame` probe. `Mimir.BufferSmoke --move-proof-surface-smoke`
proves the explicit
`muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 ->
mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79` chain,
and rejects single-ray fallback as not full pose.
- `MimirMoveProofPipeline` is the shared commit primitive for the live Move
proof path. It starts from a Muninn CultMesh shared-memory evidence frame,
admits samples to the native `move_evidence` reservoir, runs Mimir-owned
fusion, creates the Mimir pose stream frame, creates the proof surface, and
returns the Fensalir spline probe in one pass. `Mimir.BufferSmoke
--move-proof-pipeline-smoke` proves that shape against the native reservoir
debug DLL; live hardware and presented-frame capture still need to call this
primitive instead of hand-assembling a parallel path.
- `MimirRuntime` is now the live Fensalir attachment point for proof surfaces.
`PublishMoveProofSurface` accepts the Mimir-owned
`mimir.move_proof_surface.v1` document, `CreateFrame` composes its
observer-only `move-proof-*` splines into `AquariumFrame.Scene.SplineFrame`,
and `Mimir.BufferSmoke --move-proof-runtime-frame-smoke` proves the frame is
empty before publish, contains the proof splines after publish, and clears
without leaving a repair loop behind.
- `MimirMoveProofRuntimeDriver` is the bounded runtime bridge from a Muninn
CultMesh shared-memory evidence ring to `MimirRuntime.PublishMoveProofSurface`.
It derives the Mimir evidence, Mimir pose, and Fensalir frame ids from the
actual Muninn frame suffix, calls the shared proof pipeline, and suppresses
duplicate frame admission. `Mimir.BufferSmoke --move-proof-runtime-driver-smoke`
proves `MimirRuntime.Update` can pull one ring frame into the same-sequence
proof chain and visible spline frame.
- `MimirMoveProofRuntimeConfiguration` is the typed runtime configuration and
validation contract for future real Move proof subscriptions. An enabled
source names the Muninn evidence stream, native reservoir path, Mimir evidence
and pose frame prefixes, Fensalir presented-frame prefix, fusion authority,
consumer contract, and at least two calibrated camera witnesses. It creates
`MimirMoveProofRuntimeDriver` only after the supplied ring stream id matches
the configured evidence stream.
- `MimirRuntime` now owns configured Move proof activation instead of leaving
config as a readout. At scene-ready it asks an
`IMimirMoveProofEvidenceRingProvider` for the configured Muninn evidence ring,
opens the configured native reservoir, creates the driver, retains the
resources, and exposes an activation status. The default provider fails
explicitly because the current C# CultMesh ring is still in-process only;
`Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an
in-process provider and proves config activation can produce the same named
proof chain. Real Nightwing/Starfire cross-process ring opening remains
pending.
- `mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine
surface for configured Move proof sources. It reports the configured stream,
provider kind, active/driver state, native reservoir path, calibrated camera
ids, latest same-stream proof ids, latest verdict, and the unavailable
default-provider diagnostic. `Mimir.BufferSmoke
--move-proof-runtime-activation-surface-smoke` proves both an active injected
ring document and the inactive default runtime document without claiming live
field evidence.
- `mimir.move_proof_evidence_frame_snapshot.v1` is the one-copy fallback for
captured Muninn evidence frames while C# CultMesh has no cross-process ring
opener. A configured `EvidenceSnapshotPath` is loaded by
`MimirConfiguredMoveProofEvidenceRingProvider`, metadata-checked against the
encoded payload, copied into an owned in-process ring, and then consumed by
the same runtime driver/pipeline. `Mimir.BufferSmoke
--move-proof-runtime-snapshot-smoke` proves the named proof spine through
that file boundary. It is field capture/replay support, not the final live
shared-memory proof.
- Odin/Muninn now has the matching producer-side latest snapshot writer:
`--move-evidence-snapshot <path>` writes the Mimir-compatible
`mimir.move_proof_evidence_frame_snapshot.v1` artifact from the same Muninn
Move evidence publisher after the frame payload is accepted into the stream
ring. The Odin unit
`move_evidence_snapshot_writes_mimir_compatible_frame_artifact` decodes the
snapshot tuple and embedded Muninn frame payload. This lets Nightwing produce
a field artifact for Mimir replay while the final live ring/page transport is
still pending.
- `MimirMoveProofDevSurface` is a dev-only bootstrap gated by
`MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so
`Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App`
headless, capture the Fensalir-presented PNG, and pixel-check the named
proof probe. This verifies the renderer/probe layer; it is not live
Nightwing hardware evidence.
- `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for
Starfire/Nightwing Moves: required Muninn evidence streams, optional
Quest headset/controller pose witnesses, stillness/sweep/validation phases, and
the four derived calibration outputs Mimir must produce before IMU
orientation can become authority.
- Muninn publishes Quest access as `muninn.quest_access.v1` for the USB-attached
Quest. Mimir consumes that access surface and later `muninn.quest_pose_frame.v1`
samples as optional calibration evidence; Mimir does not own ADB/Quest access.
- The old `scripts/start-nightwing-move-tracking.ps1` bring-up path is
archived. It no longer starts a receiver/recorder pair or stages a socket
witness worker. Nightwing Eye/Move evidence must enter through Muninn/Mimir
typed CultMesh stream frames and Odin-discovered CultMesh documents. The blob
stream remains optical witness evidence for later pose fusion, not the final
6DoF pose owner.
- Structured PS Move light pulses are Muninn output commands. Mimir publishes
`muninn.move_light_command.v1` over CultNet/CultMesh to the Muninn daemon on
the host that owns the USB-attached Move. Muninn writes PS Move HID report
`0x06` locally and updates command state/receipt.
`scripts/start-starfire-move-light.ps1` and Nightwing direct HID writes are
now smoke/bootstrap paths for hardware proof when Muninn is not yet running
on that host.
- Odin's Muninn organ owns Move optical candidate extraction in
`E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds
`muninn.move_marker_candidate.v1` records. Mimir consumes those candidate
streams; it does not own raw optical extraction. Final pose, stereo
triangulation, calibration, association, IMU fusion, and prediction belong to
Mimir.
- Muninn's Move evidence stream frame no longer hardcodes an empty optical
candidate slice. The daemon frame uses the canonical
`muninn.move_marker_candidate.v1` record shape, and the Odin unit
`move_marker_candidates_publish_in_mimir_compatible_cultmesh_frame` proves a
bright Y8 frame can pass through `muninn-move-tracker` and serialize as a
non-empty marker candidate beside controller evidence. The daemon now has a
source-local Y8 extraction/publish seam and a first `serve` camera producer:
`--move-marker-camera <camera-id>=<device-path>` polls a Unix V4L2 YUYV frame,
converts it to compact Y8, and feeds that same seam. Unit tests prove the
configured camera tick publishes marker evidence through the shared frame
contract; Nightwing hardware/live V4L2 proof is still pending.
- `MimirProcessStreamSource` for bridge/network command edges.
- `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata
from native probes into the same rolling buffers Fensalir inspects. One probe
@ -189,7 +342,7 @@ a named invariant that the native runtime cannot protect yet.
profiles, language/emission profiles, path-learning sessions, benchmark
panels, audio actuator strategies, native capture profiles, camera ingest
strategies, reservoir strategies, audio/visual field profiles, compute
offload profiles, OBS publication profiles, CultMesh contracts, Fensalir
offload profiles, program publication profiles, CultMesh contracts, Fensalir
lowering, and assembly plans. Use
`Mimir.BufferSmoke --perfect-machine-profile-smoke` to prove the catalog
assembles, `--perfect-machine-contract-smoke` to write a CultCache contract
@ -197,7 +350,10 @@ a named invariant that the native runtime cannot protect yet.
tooling/UI/remote witness use, and `--perfect-machine-lowering-benchmark` to
measure the Mimir-to-Fensalir lowering path. The current six-camera/two-audio
synthetic lowering benchmark runs at roughly 2.5 us per iteration with about
2.3 KB allocated per iteration.
2.3 KB allocated per iteration. `--move-tracking-contract-smoke` proves
Starfire-local and Nightwing-remote Move tracking observations enter
Mimir's rolling buffers with Muninn producer identity and Odin discovery
provenance.
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
timestamp, and native/GPU handle metadata.
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
@ -207,14 +363,61 @@ a named invariant that the native runtime cannot protect yet.
source-id hashing, producer helpers, and typed audio/render payload
descriptors.
- Windows bridge scripts for sender discovery/start/stop and simple OBS Media
Source ingest.
Source compatibility ingest.
- `src/Mimir.Runtime/Synchronization` now contains the first typed program
composition contracts: `mimir.program_scene.v1` and `mimir.program_output.v1`.
These make Mimir's scene graph and Yggdrasil/site publication route
manifestable through CultMesh.
- `docs/mimir-program-composition.md` is the current authority map for the new
stream-program architecture: Muninn observes on Starfire/Nightwing/Raven-class
hosts, Mimir consumes selected streams and composes the program, Yggdrasil publishes to the site, and OBS is
compatibility-only.
- `src/Mimir.CultMeshMedia` is the first explicit CultMesh media/body bridge:
`relay` hosts the document relay on CultNet RUDP `3075`, `send` reads an
MPEG-TS byte stream from stdin and publishes rolling
`mimir.cultmesh_media_frame` documents, and `recv` subscribes to those
documents and writes ordered MPEG-TS bytes to a Starfire-local UDP endpoint
for compatibility sinks. Its C# implementation now uses explicit CultLib
RUDP client/session helpers behind
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`; the older
`CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` entrypoints no longer own
the media transport lane, and sender/receiver launchers no longer accept raw
relay host/port configuration.
- `scripts/start-raven-cultmesh-av-sender.ps1`,
`scripts/start-yggdrasil-cultmesh-media-relay.ps1`, and
`scripts/start-starfire-cultmesh-av-receiver.ps1` are the CultMesh bridge
operators. `start-raven-cultmesh-av-sender.ps1` is the Mimir-owned
`Mimir.CultMeshMedia` bootstrap/body-bridge lane; it is not the real Muninn
OBS/SRT feed owner. The actual Raven OBS/plugin feed now routes through
Odin's Muninn actuator `E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1`,
with `scripts/start-raven-muninn-obs-feed.ps1` in this repo as a thin local
wrapper around the real `GameCult-Muninn-Activate` task and
`muninn.exe activate` body. `-LocalBootstrap` on the CultMesh sender remains
a direct local bootstrap edge only for the separate body-bridge lane. Raven
capture still defaults to FFmpeg desktop frames plus Mimir's WASAPI loopback
capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains an explicit
fallback.
- Documentation for OBS receiver setup, native rebuild boundaries, the viable
stream app, and the Mimir Face.
## Temporary
- Audio and video may still traverse separate OBS/SRT endpoints during bridge
testing so OBS can preserve independent controls.
testing, but Mimir owns independent controls.
- The CultMesh media bridge still lowers to local UDP for compatibility sinks
because OBS is not a CultMesh consumer. Network transit between Raven,
Yggdrasil, and Starfire is the Odin-discovered CultMesh/CultNet path;
OBS-local UDP is an egress adapter only.
- CultLib RUDP is the default typed CultNet/CultMesh document transport for
daemon truth. Idunn RUDP health publication is the current freshness witness;
provider advertisements, command boundaries, transport profiles, and retained
daemon state should follow the same path. Odin still owns Verse/service
discovery, Idunn owns keepalive decisions, and Mimir-owned dashboard/reference
surfaces only report their own observed state. Product/debug render surfaces
stay lowerings or compatibility evidence.
- `Mimir.CultMeshMedia` has completed its explicit RUDP transport cut. Do not
add another private bridge, status shim, or renderer-derived service truth
while the RUDP document lane exists.
- Process-backed stream sources are only acceptable for network bridge feeds or
diagnostics. Six-camera local ingest belongs behind direct capture drivers.
- Frame-event process sources are diagnostic only. They prove source cadence and
@ -224,34 +427,111 @@ a named invariant that the native runtime cannot protect yet.
## Next
Before expanding Muninn media or Sleipnir input transport, apply the traffic
contracts distilled in
`docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md`: latest-state,
ordered-edge, video-deadline, audio-playout, and reliable-control. Preserve the
existing Muninn packetizer/feedback foundations. The first implementation pass
should make input supersession and edge preservation structural, replace the
two-second media default with consumer-derived field budgets, and prove the
result under controlled loss/jitter/reorder before changing carriers.
The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot
`8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media
deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware
repair, and decode-chain-owned keyframe pressure. The production video path now
uses typed V4 Cauchy GF(256) FEC in independent 8-data/4-parity blocks. Each
block schedules data and parity as separate lanes, including fixed protection
for a short tail block. Canonical video and parity use CultNet's unreliable
`realtime` lane; selective repair and IDR recovery remain deadline-bound.
Canonical AAC ADTS audio and its fixed 864-byte parity shards also use
`realtime`; 4+2 FEC, 120 ms reorder, and
concealment own continuity without an ACK/retransmit window. Sender access units enter the
handoff queue independently, and CultMesh catalog publication runs outside the
realtime media loop. Repair material becomes available when a video frame enters
the queue, and the receiver may issue bounded repair requests at 64/96/128 ms.
The remaining work is duplicate/reconnect acceptance, longer iid runs, and the
long-duration mixed soak.
The socket harness and bounded PCM loss recovery have now landed. Odin's
`cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/
duplicate/jitter/stall profiles around real CultNet endpoints. Muninn emits a
typed fixed 4+2 audio parity block after each four constant-size PCM packets;
the production Mimir OBS receiver reconstructs up to two missing packets inside
the 40 ms reorder budget and feeds them through the existing playout owner.
Both sides use the experimental CultLib/CultMesh snapshot lineage used by Eve,
Aetheria, and VoidBot; no stable-branch transport shim was introduced.
Receiver pressure now closes into the long-lived NVENC encoder as bounded AIMD
bitrate control. Startup begins at half of the configured encoder ceiling
because fixed block parity can roughly double wire rate, and additive recovery
is capped at half of that configured ceiling. Late/decode/repair/queue
pressure backs off by 15 percent; clean recovery adds one-fiftieth of the safe
cap only after ten stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a
restart and emitted the required transition IDR. The exact remaining completion
gate is tracked in
`docs/research/moonlight-reliability-acceptance-2026-07-16.md`; no cross-host
claim is allowed until its Raven-to-Starfire field matrix passes.
The production receiver follow-up fixed two contract contradictions: early
video repair no longer declares the frame late (which had made Odin discard all
repair requests), and the OBS audio decoder now consumes Muninn's actual float
PCM contract rather than treating it as AAC. Audio reorder is bounded to 40 ms
with short silence concealment. The next audio cut is Opus with explicit
FEC/PLC for variable-rate compressed audio; PCM now has bounded 4+2 erasure
recovery plus concealment. The controllable video encoder owner now exists and
forces the next NVENC frame to IDR without restarting the video session. A live
D3D11 desktop proof verified the command-generated IDR. The Raven bundle and
OBS receiver are now deployed over the direct LAN route. Raven activation is an
interactive-token scheduled task whose PowerShell action uses
`-WindowStyle Hidden`; no WireGuard path or foreground terminal is part of the
runtime. The active field cut uses D3D11/NVENC video, WASAPI loopback audio,
typed CultNet media, and the experimental CultLib lineage. The named direct and
proxy impairment profiles and long soak remain the completion gate.
1. Replace the frame-event diagnostic bridge with concrete direct capture
drivers for Leap stereo IR first, then the
other cameras.
2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
2. Cut the remaining dashboard service truth paths from older renderer-derived
assumptions to explicit CultNet RUDP records,
preferably authorized-peer dialing where the peer catalog exists. Preserve
OBS-local UDP as egress only for compatibility sinks.
3. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
frame cadence in the rolling buffers.
3. Promote the packet-song physical calibration receipt into the runtime
4. Promote the packet-song physical calibration receipt into the runtime
receiver. The live decoder should keep its ear open for self-identifying
song contours, extract intra-call time/frequency anchors from log-mel parts,
apply learned per-output/mic path weighting, and feed a global
delay/clock/path hypothesis. Keep chirp-bin calibration artifacts as
reference data, not the runtime target.
4. Add the synchronization actuator: drive a variable-rate resampler and
5. Add the synchronization actuator: drive a variable-rate resampler and
fractional delay line per non-reference stream from the smoothed
`MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder
through real loopback and microphone paths so every correctly heard word
becomes a deterministic timeline anchor before the actuator moves samples.
5. Prove the bioacoustic hybrid fallback through real loopback and microphones
6. Prove the bioacoustic hybrid fallback through real loopback and microphones
with probe durations long enough to keep loopback and mic windows live.
6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence,
7. Wire real Move proof production into `MimirMoveProofRuntimeDriver`: attach
real Muninn Nightwing and Starfire evidence rings, require calibrated optical
witnesses, publish `mimir:starfire:move-pose:<sequence>` through the shared
runtime path, and replace the dev-gated presented-frame smoke with a
same-sequence capture from real field evidence.
8. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence,
source timestamps, and output settings are visible and adjustable.
7. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute:
9. Implement the Mimir program scene graph as the shared commit primitive for
source subscription, transforms, crop, chroma key, visibility, layer order,
preview, and output publication. Import the current OBS scene only as an
initial mirror; the operator surface returns on Thing 2.
10. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir
program output and publishes it without owning a second composition.
11. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute:
sample buffer-domain paths stochastically by visual contribution, emit SDF
splat probes, write them into the spatiotemporal splat reservoir, and sample
that reservoir in the temporally antialiased scene pass. The direct spline
preview must stay a witness until this path owns rendering.
8. Move GPU feature extraction, fusion, material fitting, render budgeting, and
12. Move GPU feature extraction, fusion, material fitting, render budgeting, and
Spout2 publication into Fensalir.
9. Move mic alignment, room suppression, voice separation, spatialization, and
13. Move mic alignment, room suppression, voice separation, spatialization, and
stem generation into Faust/native DSP.
10. Keep the OBS bridge witness ledger as evidence before expanding receiver
14. Keep the OBS bridge witness ledger as evidence before expanding receiver
machinery.

View file

@ -48,9 +48,10 @@ Mimir exists to turn cameras, microphones, speakers, timing signals, and
distributed capture surfaces into one realtime coherent volumetric field. The
field is the product. The bridge scripts are scaffolding and probes.
The live target is one native five-second rolling reservoir feeding Fensalir GPU
fusion and Faust/native DSP, then OBS. Retention has one owner. Typed views are
indexes, not private histories.
The live target is one native five-second rolling reservoir feeding Mimir-owned
program composition, Fensalir GPU fusion, Faust/native DSP, operator
surfaces, and a Yggdrasil site publisher. OBS is compatibility only. Retention
has one owner. Typed views are indexes, not private histories.
## Operating Doctrine
@ -63,11 +64,14 @@ the synchronized program path is being rebuilt around `LocalcastRuntime`,
Invariants:
- OBS owns final broadcast composition.
- Fensalir owns dense visual fusion, material/brush/splat rendering, and Spout2
publication.
- Mimir owns final program composition, source subscription, preview/control
state, stats, and publication intent.
- Fensalir owns dense visual fusion, material/brush/splat rendering, and local
program texture output.
- Faust/native DSP owns hot audio alignment, room suppression, voice separation,
stems, and spatial bed generation.
- The Yggdrasil publisher daemon ships the Mimir program to the site without
owning composition.
- FFmpeg owns simple capture, encode, and transport in V1 bridge mode.
- Mimir/this repo owns configuration, calibration, launch/status, typed
contracts, memory, and operational repeatability.

View file

@ -0,0 +1,103 @@
# Mimir Program Composition
## Objective
Mimir owns the live stream program: selected capture feeds, calibration feeds,
scene composition, preview/control surfaces, stats, and publication. OBS is a
temporary compatibility sink only.
## Authority Map
- Owner: Mimir owns the program scene graph and subscription policy.
- Inputs: Muninn-published local and remote monitor, window, camera, audio,
loopback, and health streams.
- Outputs: a Mimir program video surface, aligned audio stems or final bus,
and a Yggdrasil-facing site publication feed.
- Derived state: OBS receives a compatibility feed when useful; it does not own
scene truth, broadcast truth, preview truth, stats, source selection, or
publication.
- Forbidden writers: OBS scenes, browser dashboards, site daemons, and Muninn
capture hosts must not independently decide the program composition.
- Shared paths: direct operator edits, remote commands, preset loads,
calibration-driven source selection, and recovery scripts must commit through
the same Mimir scene graph command surface.
- Deletion line: old OBS-owned composition docs and config names are demoted to
diagnostic bridge language. Any OBS adapter delegates to the Mimir program
scene instead of carrying its own opinion.
## Machine Shape
```mermaid
flowchart TD
A["Muninn on Starfire"] --> D["CultMesh stream catalog + media bodies"]
B["Muninn on Nightwing"] --> D
C["Muninn on Raven"] --> D
D --> E["Mimir subscription policy"]
E --> F["Mimir rolling buffers"]
F --> G["Mimir program scene graph"]
G --> H["Fensalir compositor"]
F --> I["Faust/native DSP"]
H --> J["program video surface"]
I --> K["program audio stems/bus"]
J --> M["Yggdrasil site publisher daemon"]
K --> M
M --> N["GameCult site"]
J -. "compatibility" .-> O["OBS adapter"]
K -. "compatibility" .-> O
```
## Composition Contract
The scene graph carries layers with:
- source reference, such as `muninn:starfire:monitor:primary` or
`muninn:starfire:window:discord`;
- source kind, such as monitor, window, camera, network monitor, or generated
Mimir visual field;
- visibility, layer order, transform, crop, and optional chroma key;
- enough timing identity for Mimir to align media inside the rolling window.
The current OBS scene can be mirrored as an initial Mimir scene:
- Starfire monitor capture, cropped at the bottom when needed.
- Discord window capture, cropped to chat and keyed with chroma filters.
- Raven primary monitor plus Realtek loopback as a subscribed remote Muninn
stream.
- Mimir/Fensalir program texture as a generated layer when the field renderer is
active.
After import, OBS is not the editor of record. It may display the Mimir program
feed, but operator edits happen through Mimir command surfaces.
## Site Publisher
The Yggdrasil-facing publisher consumes Mimir program output and publishes it to
the GameCult site. It owns transport, encoding/session state, reconnect policy,
and public edge health. It does not own composition.
The publisher's input contract is the same Mimir program output used by local
preview and OBS compatibility adapters, so the public stream cannot silently
diverge from the operator preview.
## Runtime Contracts
The first typed contracts live in `src/Mimir.Runtime/Synchronization`:
- `mimir.program_scene.v1`
- `mimir.program_output.v1`
- `mimir.eve_operator_surface.v1`
The proof path is:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke
```
Import the current OBS scene as Mimir-owned typed state:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --import-obs-program-scene --output state\mimir-program-composition.cc
```
The live imported state file is `state/mimir-program-composition.cc`.

View file

@ -0,0 +1,107 @@
# Move Calibration Protocol
Mimir owns Move calibration and sensor fusion. Muninn owns source-local facts:
camera marker candidates, USB controller IMU/button state, and local Move light
commands. Odin discovers and projects the surfaces. Fensalir consumes Mimir's
resolved poses after Mimir has earned them.
## Current Stream Shape
- Starfire should run Muninn as a local capture/control daemon. Its Move stream
address is `muninn:starfire:move-evidence`.
- Nightwing runs Muninn for its USB Move and local camera witnesses. Its Move
stream address is `muninn:nightwing:move-evidence`.
- Mimir on Starfire publishes resolved pose frames at
`mimir:starfire:move-controller-poses`.
- Quest can participate as an external tracked VR reference over USB:
`quest:usb:headset-pose`, `quest:usb:left-controller-pose`, and
`quest:usb:right-controller-pose`.
- Starfire ASIO/Scarlett loopback can provide cue timing, but it is not a Move
pose authority.
The Move hot path is CultMesh stream frames. On the same host, those frames use
shared-memory bytes rings. Across hosts, the remote peer still crosses the
Verse/network boundary. The current Raven program media bridge is not same-host
zero-copy into Mimir/Fensalir; it sends CultNet/CultMesh media documents and
lowers to local UDP for OBS compatibility.
## Required Capture Phases
1. `preflight-streams`: hold both Moves still and visible; prove stream freshness,
clock skew, and controller ids.
2. `dark-stillness`: lights off, both Moves still; estimate gyro bias, gravity,
and magnetometer baseline.
3. `lit-stillness`: light one Move at a time, then both; map light commands to
controllers and measure centroid stability.
4. `axis-sweeps`: slow pitch/yaw/roll sweeps; fit gyro scale, accelerometer
frame, and optical angular consistency.
5. `figure-eight`: broad figure-eight motion; fit magnetometer hard/soft iron
and cross-axis coupling.
6. `quest-reference`: optional Quest headset/controller reference with the Move
between the two Quest controllers in front of the headset.
7. `validation-pass`: held-out free motion; promote only if triangulated optical
residuals and IMU prediction residuals pass.
## Quest Access Preflight
The Quest must be authorized for USB debugging before Starfire Muninn can query
it. Muninn owns the access surface:
```powershell
muninn serve --store C:\Meta\Odin\state\muninn.telemetry.cc --host starfire --quest-adb --quest-serial 1WMHHB68PG1515
```
When the Quest is authorized, Muninn publishes `muninn.quest_access.v1` and
advertises `muninn:<host>:quest-input`, `muninn:<host>:quest-poses`, and
`muninn:<host>:quest-warped-video-input`. Mimir consumes the access/pose
surfaces as calibration evidence; it does not own Quest USB access.
```powershell
muninn quest-access-status --store C:\Meta\Odin\state\muninn.telemetry.cc
```
ADB authorization proves USB access, device identity, and basic headset state
only. It does not expose Quest headset or controller poses; those still require
a Quest/OpenXR witness bridge that publishes pose samples through Muninn. The
current calibration protocol can still run without Quest, but Quest poses are
the preferred external reference when the headset and both controllers are
sitting around the Move.
## Data Products
- `mimir.move_fusion_rig_calibration.v1`: camera intrinsics/extrinsics, clock
association policy, and gyro unit scale.
- `mimir.move_imu_calibration.v1`: per-controller gyro bias/scale,
accelerometer gravity frame, magnetometer correction, and noise estimates.
- `mimir.move_controller_identity_map.v1`: stable association between Muninn
USB ids, light receipts, and optical orb identities.
- `mimir.move_calibration_receipt.v1`: sample counts, residuals, rejected
phases, and promotion decision.
## Optical Sphere Fit Authority
`MimirMoveSphereCalibrationFitter` owns the bounded two-Eye optical fit after a
collection session reaches `Fitting`. Eye 0 defines the tracking-space gauge.
The fitter jointly estimates both focal lengths and Eye 1's rigid pose from
same-frame, same-Move correspondences, using a deterministic 80/20 training and
held-out partition. A robust training consensus may reject contaminated marker
pairs, but promotion also requires at least 65 percent of the untouched
held-out pairs to agree and must pass the session's median and P95 reprojection
limits.
The physical orb radius is an explicit input; it is never guessed by the
solver. PSMoveAPI's reported blob radius is a color-segmentation result, not a
calibrated sphere silhouette. It may seed or weight depth only after the
per-camera radius response has been validated. A fresh PSMoveAPI position age
is also not optical confidence and must not be interpreted as one.
## Preflight Command
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke --output artifacts\move-calibration\protocol.cc
```
That writes the typed protocol as `mimir.move_calibration_protocol.v1` into a
CultCache receipt. The real hardware runner should consume this document plus
Muninn's `muninn.quest_access.v1` when Quest is available, then append capture
receipts instead of inventing a parallel checklist.

View file

@ -5,17 +5,24 @@ hosted stream machine.
## Objective
Make the room available in OBS as synchronized program video plus separately
Make the room available as a Mimir-owned synchronized program with separately
controllable audio while preserving ownership:
- Mimir owns configuration, calibration truth, runtime contracts, launch,
status, and persistence.
- Mimir.Runtime owns in-memory stream buffers and runtime synchronization.
- Native capture workers own device reads and append typed sample handles.
- Fensalir owns GPU fusion, rendering, D3D12 interop, UI, and Spout2 output.
- Mimir owns stream subscription, scene composition, preview/control state,
stats, and program publication intent.
- Fensalir owns GPU fusion, rendering, D3D12 interop, UI lowering, and local
program texture output.
- Faust/native DSP owns hot audio alignment, suppression, separation, stems, and
spatial bed generation.
- OBS owns broadcast controls.
- Eve GUI/TUI lowerers expose operator controls and preview without owning
scene truth.
- The Yggdrasil-facing publisher daemon consumes Mimir program output and
publishes it to the site without owning composition.
- OBS is a temporary compatibility sink.
## Runtime State Model
@ -26,10 +33,13 @@ flowchart TD
N["network feed producers"] --> R
R --> Q["Fensalir UI + GPU fusion"]
R --> F["Faust/native DSP"]
Q --> S["Spout2/program video"]
Q --> S["Mimir program video"]
F --> P["program stems + spatial bed"]
S --> O["OBS"]
P --> O
Q --> E["Eve operator surfaces"]
S --> Y["Yggdrasil site publisher"]
P --> Y
S -. "compatibility" .-> O["OBS adapter"]
P -. "compatibility" .-> O
```
The app-level runtime initializes one rolling buffer per configured audio or
@ -48,7 +58,8 @@ camera driver buffers, GPU textures, or DSP audio memory.
- Fensalir consumes current visual/audio timing state and publishes the program
video surface.
- Faust/native DSP consumes audio/phase/event handles and publishes stems.
- OBS receives final program surfaces. It does not own synchronization.
- OBS compatibility adapters may receive final program surfaces. They do not
own synchronization, composition, preview/control, or publication.
## Impossible By Construction

View file

@ -38,19 +38,23 @@ Read this with:
Mimir is a bounded-latency realtime field computer: it ingests many clocks,
places their evidence inside one rolling temporal window, extracts audio/video
geometry from that window, and emits an OBS-usable program surface without
letting OBS become the synchronization authority.
geometry from that window, owns the stream program scene, exposes Eve operator
surfaces, and emits a publicable program feed without letting compatibility
sinks become authority.
## Authority Index
| Authority | Owns | Must Not Own |
| --- | --- | --- |
| `Mimir.Runtime` | stream identity, buffers, sync reports, calibration model loading, source polling, telemetry state | hot DSP actuator, dense visual fusion, broadcast composition |
| Native capture | direct driver callbacks, device timestamps, payload handles, low-jitter capture loops | runtime policy, UI state, OBS composition |
| Mimir program composition | source subscription, scene graph, crop/key/transform state, preview/control state, stats, publication intent | local device reads, GPU shader internals, audio-rate DSP, site transport policy |
| `Mimir.Runtime` | stream identity, buffers, sync reports, calibration model loading, source polling, telemetry state | hot DSP actuator, dense visual fusion, direct device capture |
| Native capture / Muninn hosts | direct driver callbacks, device timestamps, payload handles, local stream capabilities/media | runtime policy, scene composition, publication |
| `native/reservoir` | shared-edge retention and typed views | payload memory interpretation, rendering, DSP |
| Fensalir | window, UI, D3D12 bridge, GPU fusion, program video publication | clock authority, audio actuator |
| Fensalir | window, UI lowering, D3D12 bridge, GPU fusion, program video output | clock authority, audio actuator, scene ownership |
| Faust/native DSP | fractional delay, resampling, separation, spatialization, stems | stream discovery, UI |
| OBS | final broadcast composition | synchronization, capture timing |
| Eve lowerers | operator GUI/TUI rendering, controls, previews, stats lowering | scene truth, stream truth, publication truth |
| Yggdrasil site publisher | public transport, encoder/session state, reconnect policy, site edge health | scene composition |
| OBS | compatibility preview/output sink | synchronization, capture timing, composition, publication |
| Raven/phones | local decode of codebook/schedule observations | canonical clock authority |
## Evidence Taxonomy
@ -241,7 +245,9 @@ Six optical sensors with high-rate Eyes and Leap IR mean:
- Cut JSON/stdout local camera hot path when concrete direct drivers land.
- Cut legacy chirplet path once chirp-bin decoder has equivalent docs/tests and
physical path proof.
- Cut bridge scripts only after OBS-facing direct output is real.
- Cut bridge scripts only after Mimir program output, Eve operation, and
Yggdrasil publication are real enough that OBS compatibility no longer earns
its keep.
- Cut ad hoc response weighting if a formal path model replaces it.
## Future Work Index

View file

@ -13,11 +13,12 @@ job-interview answer for why it belongs in the body of the machine.
| Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` |
| Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` |
| Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` |
| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, and actuator state documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs` |
| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` |
| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, Muninn records optional Quest USB witness availability as `muninn.quest_access.v1`, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` |
| Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` |
| Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` |
| Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` |
| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, and Raven Scarlett. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` |
| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, Raven Scarlett, and USB-attached Moves on Starfire/Nightwing. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` |
| Camera ingest strategies | `Mimir.Runtime` + native workers | Diagnostic metadata, managed native wrapper, native SPSC ring, and shared-GPU-texture ingest shapes. | `src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs` |
| Reservoir strategies | `Mimir.Runtime` + native reservoir + Fensalir | Managed rolling buffers, native shared-edge rings, and derived GPU temporal field storage. | `src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs` |
| Distributed witnesses | `Mimir.Runtime` + CultMesh | Raven, phone, microcontroller, and remote camera rig configurations for compact timing/path state. | `src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs` |
@ -27,7 +28,7 @@ job-interview answer for why it belongs in the body of the machine.
| Visual fusion configurations | `Mimir.Runtime` + Fensalir | Sensor roles and fusion modes from cadence proof through full hybrid evidence. | `src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs` |
| Compute offload configurations | `Mimir.Runtime` + CultMesh | Local-heavy, distributed witness, and calibration sweep workload placement. | `src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs` |
| Assembly plans | `Mimir.Runtime` | Named combinations of modules that prove synthetic, Scarlett, meatspace, and distributed cuts. | `src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs` |
| OBS publication profiles | Fensalir + Faust + OBS | Spout/shared-texture video and separately mixable Faust stem publication. | `src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs` |
| Program publication profiles | Mimir + Fensalir + Faust + Eve + Yggdrasil | Mimir-owned scene publication, Eve operator surfaces, Yggdrasil site feed, and temporary OBS compatibility adapters. | `src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs` |
| Fensalir lowering | `Mimir.Runtime` -> Fensalir | Pure mapping from Mimir rolling/video/audio state into Fensalir GPU sensor and acoustic field frames. | `src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs` |
| Module catalog | `Mimir.Runtime` | Indexes the live module set and verification commands. | `src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs` |
| Manifest factory | `Mimir.Runtime` | Produces an exportable JSON manifest for tools, docs, UI, and remote witnesses. | `src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs` |
@ -42,7 +43,19 @@ job-interview answer for why it belongs in the body of the machine.
stem generation.
- CultMesh carries typed state surfaces; remote nodes may observe and decode,
but they do not become independent clock authorities.
- OBS receives final program surfaces and stems. It does not own calibration.
- USB-attached Moves on Starfire and Nightwing publish source-local Muninn
evidence streams: glowing-orb marker candidates and controller/IMU/button
state. Odin discovers/projects those streams. Mimir owns native evidence
buffer admission/layout, calibration, association, triangulation, confidence,
and `mimir.move_controller_pose.v1` publication. The current fusion pass
resolves calibrated optical position and controller state; orientation remains
explicitly unresolved until IMU fusion/prediction lands. Fensalir consumes the
resolved pose stream for interactive environments once that contract is wired.
- Resolved Move poses are framed as
`mimir.move_controller_pose_stream_frame.v1` on CultMesh shared-memory bytes
streams; the stream is a Mimir output surface, not a new fusion owner.
- OBS may receive compatibility program surfaces and stems. It does not own
calibration, composition, preview/control, or publication.
## Verification Surface
@ -53,6 +66,12 @@ dotnet build .\Mimir.slnx
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-lowering-benchmark
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-fusion-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test
```

View file

@ -0,0 +1,333 @@
# Moonlight lessons for Muninn streaming and Sleipnir input
Date: 2026-07-16
Upstream examined: `moonlight-stream/moonlight-common-c` commit `82e25148549d249a80aaf4afc68e2abb07072edf`
## Verdict
Moonlight is strong because it does not ask one transport policy to serve every
realtime signal. It gives video, audio, control, and input different loss,
ordering, recovery, and freshness rules, then keeps each queue bounded by what
the user can still perceive.
Muninn is not missing all of that machinery. Its media packetizer already owns
access-unit boundaries, timestamps and deadlines, video parity, dependency and
keyframe metadata, bounded assembly, receiver damage feedback, repair caching,
and keyframe requests. Sleipnir already rejects regressing sequence/timestamp
pairs, expires reliable HID sends after 25 ms, reports stage latency, and
neutralizes stale output.
The 2026-07-17 field cut closes most of the original mechanism gaps:
- canonical video and video parity use the unreliable `realtime` lane; typed
8+8 GF(256) blocks, selective repair, and decode-chain IDR recovery own loss;
- canonical 10 ms PCM audio uses `realtime`, with 4+2 FEC, a 40 ms reorder
owner, and short concealment; reliable delivery is reserved for control;
- the consumer deadline is 100 ms by default and 250 ms in the Raven field
profile; queues, retransmits, repair material, and assemblies derive from it;
- receiver pressure actuates the live NVENC encoder through multiplicative
backoff and conservative additive recovery; startup begins at half the
requested ceiling because fixed video parity can roughly double wire load;
- Sleipnir latest state and ordered edges have distinct epoch/sequence and
application-ack semantics, including bounded overflow rebase.
The remaining claim boundary is evidence: direct and impaired Raven-to-
Starfire soaks, reconnect repetition, A/V skew measurement, and a live virtual
HID exercise. The machine is no longer missing a named owner for these paths;
it still has to prove those owners under pressure.
Do not transplant Moonlight wholesale. Distill its ownership model into the
existing typed Muninn/Sleipnir pipeline.
## Authority map
### Muninn media
- Owner: Muninn owns source-local capture, encoding child lifecycle, media
packetization, repair material, and sender feedback response.
- Inputs: encoded H.264/H.265 access units, encoded audio packets, source media
timestamps, configured latency target, and receiver feedback.
- Outputs: typed video chunks/parity, audio packets, recovery/keyframe actions,
and transport/quality telemetry.
- Derived state: RUDP acknowledgements, resend counters, CultCache receipts, OBS
local lowering, and operator cards are evidence or delivery substrate. They do
not own media deadlines.
- Forbidden writers: generic RUDP retry policy must not decide whether expired
media is still useful; OBS must not decide stream synchronization; persistence
must not become a hot media queue.
### Sleipnir input
- Owner: Sleipnir owns selection/remapping and local virtual-HID emission.
Muninn owns source-local device sampling and event classification.
- Inputs: a discovered Muninn input stream, typed mapping intent, monotonic
stream sequence, source-local capture time, and receive time.
- Outputs: local virtual-HID state plus latency/freshness receipts.
- Derived state: the latest full controller snapshot is replaceable state.
Button transitions, text/code points, connect/disconnect, and neutralization
are non-replaceable edges until acknowledged or superseded by an explicit
state transition.
- Forbidden writers: generic reliable queue order must not replay stale analog
or motion samples; wall-clock subtraction must not claim one-way latency
without a clock model; the telemetry/store path must not repair a missing hot
stream.
## What Moonlight actually does
### It separates stream semantics
Moonlight receives video and audio over UDP/RTP-like paths while input travels
through its control path (or a legacy TCP socket). The important lesson is not
the literal socket choice. It is that the queues have different contracts.
Video is frame/dependency oriented; audio is short-block continuity oriented;
input is event/state oriented.
Source: [stream initialization and input transport](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/InputStream.c),
[video receive path](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/VideoStream.c), and
[audio receive path](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/AudioStream.c).
### Video recovery is dependency-aware and deadline-shaped
The video queue groups packets by frame and FEC block, tracks missing data,
reconstructs recoverable blocks with Reed-Solomon parity, rejects old/replayed
frames, and advances frame authority explicitly. Depacketization can request an
IDR when damage makes the predictive chain unsafe. The system repairs useful
frames and abandons poisoned dependency chains; it does not wait forever for
generic reliable delivery.
Source: [RTP video queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/RtpVideoQueue.c) and
[video depacketizer](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/VideoDepacketizer.c).
### Audio recovery is a small bounded continuity problem
Moonlight groups audio into fixed FEC blocks, orders packets by RTP sequence,
recovers missing shards, detects out-of-order behavior, and changes how long it
waits based on observed reordering. When recovery cannot complete in time, it
allows a discontinuity and lets the audio decoder's concealment path carry the
scar. It does not turn late audio into permanent latency.
Source: [audio FEC and reorder queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/RtpAudioQueue.c) and
[packet-loss concealment handoff](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/AudioStream.c).
### Input reliability is constrained by semantic coalescing
Moonlight has one send thread and a bounded queue, but it avoids blindly sending
every sampled value. Relative mouse motion is accumulated, absolute pointer
motion keeps the newest position, pen hover/move keeps the newest compatible
event, controller analog state is merged while button changes split batches,
and sensor motion keeps the newest value per sensor. A roughly millisecond-scale
batching opportunity can reduce latency by preventing ENet queue buildup.
This is the most important Sleipnir lesson: reliable delivery is safe only after
replaceable state has been collapsed and edges have been preserved.
Source: [input batching and send queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/InputStream.c).
### It exposes failure to the owning layer
Packet loss, FEC recovery/failure, queue overflow, frame loss, and keyframe need
are explicit signals. Moonlight's public documentation also warns that allowing
frames to queue for smoothness can create high latency; its balanced mode keeps
only about one frame for jitter smoothing. The machine treats queue depth as a
product decision, not an incidental container property.
Source: [Moonlight frame-pacing FAQ](https://github.com/moonlight-stream/moonlight-docs/wiki/Frequently-Asked-Questions#what-do-the-frame-pacing-options-on-the-android-client-mean).
## Current Muninn/Sleipnir comparison
| Concern | Moonlight | Muninn / Sleipnir now | Distilled judgment |
|---|---|---|---|
| Video unit | Encoded frame, FEC blocks, dependency recovery | Encoded access unit, chunks/parity, frame/dependency IDs, deadlines | The conceptual unit is already correct. Keep it. |
| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Typed 8+8 block FEC, deadline-bound repair, completed-frame tombstones, IDR actuation | Same ownership pattern; field loss ratios and parity overhead remain tunable. |
| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Realtime 4+2 PCM FEC, 40 ms reorder, bounded concealment | Mechanism is closed; Opus FEC/PLC is a later efficiency/quality cut. |
| Congestion | Stream-specific queues and feedback behavior | Half-ceiling startup, 15% pressure backoff, bounded additive ramp, fresh media ahead of repair | Closed loop exists; impairment evidence decides its final constants. |
| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Consumer-owned 100 ms default/250 ms Raven proof; 25 ms HID expiry | Correct owner and bounded values; glass-to-glass measurement remains. |
| Input state | Coalesces replaceable motion/sensor/analog values | Epoch-scoped latest state supersedes stale state | Structural rather than polling-accidental. |
| Input edges | Button/type changes terminate batches; text is ordered | Ordered edge sequence, application ACK, bounded replay, epoch rebase | Structural; live virtual-HID impairment proof remains. |
| Staleness | Queue construction prevents much stale work | Receiver drops regressing timestamps and neutralizes after timeout | Keep the guard, but move freshness authority earlier into enqueue/supersession. |
| Timing | RTP/media clocks and local queue timing | source Unix nanoseconds plus receiver instants | Use monotonic source time plus an explicit clock-offset/uncertainty model; do not call raw wall-clock subtraction latency. |
| Security | Authenticated encryption in modern stream generations | CultNet typed transport policy varies by lane | Preserve typed schemas, but specify replay/authentication guarantees for hot media and HID lanes. |
## Recommended cut
### 1. Define traffic classes before changing transport
Add one narrow typed delivery policy shared by Muninn and Sleipnir:
- `state-latest`: analog axes, pointer position, gyro/accelerometer samples;
- `edge-ordered`: button/key transitions, text/code points, device lifecycle;
- `media-video`: dependency-bearing access units with decode deadlines;
- `media-audio`: continuity blocks with playout deadlines;
- `control-reliable`: subscriptions, mappings, capability negotiation, and
keyframe requests.
This is not a generic router. It is a finite statement of the five contracts the
existing machine already has. Each class must name supersession, ordering,
expiry, recovery, and observability.
### 2. Cut the two-second realtime default
Derive deadlines from the actual consumer:
- Sleipnir state-latest: tens of milliseconds, with replacement before send;
- input edges: reliable until acknowledged, but compacted against later
authoritative state where safe;
- interactive video: frame/decode budget, initially test 50–150 ms on LAN;
- audio: playout budget, initially test 40–100 ms on LAN;
- Mimir's five-second analysis reservoir remains Mimir compute budget, not a
network permission to deliver old frames.
The exact values must be earned by field traces. The ownership change is not
optional: transport retry duration becomes derived from the media/input
deadline, never the other way around.
### 3. Make Sleipnir freshness structural
Split the hot input frame into:
- a latest-state snapshot with `state_sequence`, source monotonic capture time,
and supersedes-through sequence;
- an ordered edge list with its own sequence range;
- receiver acknowledgement of the highest applied state and edge sequence.
Muninn may overwrite an unsent latest-state snapshot. It may not overwrite an
unacknowledged edge. Sleipnir applies edges in order, applies only the newest
state, and neutralizes on a named lease timeout. Manual, remapped, replayed, and
live paths must use the same commit primitive.
### 4. Close Muninn's feedback loop
Receiver feedback should drive named sender decisions:
- request/force keyframe after unrecoverable dependency loss;
- choose repair only when estimated arrival precedes frame deadline;
- tune parity within a bounded profile from recent burst-loss evidence;
- reduce encoder bitrate/pacing when queue delay grows;
- expose encode, queue, network, assembly, decode, and presentation stages
separately.
Do not add a second congestion controller beside generic RUDP and hope the two
argue productively. Media policy owns the send budget; RUDP exposes delivery
mechanics and obeys the deadline.
### 5. Prove it on the layer the user feels
Build one loss/jitter/reorder test harness and record timelines for:
- controller press and release during 1%, 5%, and burst packet loss;
- high-rate analog/gyro motion under an artificial sender stall;
- video dependency loss and keyframe recovery;
- audio burst loss and concealment;
- queue growth under bitrate oversubscription;
- disconnect/reconnect and stale-session replay;
- clock offset and drift between Muninn and Sleipnir hosts.
Acceptance should bound p50/p95/p99 capture-to-HID age, lost/duplicated edges,
time-to-neutral, glass-to-glass video age, audio discontinuities, A/V skew, and
recovery time. Final-state-only tests will politely conceal the corpse.
## What not to copy
- Do not copy Moonlight's C implementation or protocol packets into GameCult
code without an explicit GPLv3 licensing decision. Architectural study is not
license laundering. See [Moonlight's license](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/LICENSE.txt).
- Do not replace typed CultMesh discovery, command state, or ownership with
RTSP/SDP or a Moonlight compatibility stack.
- Do not make ENet the new universal answer. Moonlight's strength comes from
semantic queues above the carrier, not from the carrier's brand name.
- Do not merge input, audio, and video into one ordered reliable stream. One
lost video fragment must not head-of-line block a button release or audio
packet.
- Do not treat FEC as magic. Parity is useful only when its overhead and recovery
time fit the deadline.
## Distilled doctrine
Realtime strength is not “reliable UDP.” It is knowing which truth survives
loss:
- the newest state replaces old state;
- an edge remains until applied or explicitly superseded;
- a media packet is valuable only while its frame or playout deadline is alive;
- predictive video damage is healed by recovery or a new reference, not by
delivering every corpse;
- queues are latency budgets made physical;
- feedback that cannot change sender behavior is telemetry, not control.
Muninn and Sleipnir already have enough organs to become strong. The next work
is to give each signal one explicit survival law and make the transport obey it.
## 2026-07-17 LAN field results
The Moonlight-derived ownership changes now have Raven-to-Starfire field
evidence over LAN, with Raven processes launched only through hidden scheduled
tasks:
- Muninn video uses independently recoverable block parity plus bounded repair.
During a seeded 1% datagram-loss run, the OBS receiver advanced from 2 to 12
video assembly expiries while forwarding 14,400 additional payload groups:
99.93% completed without expiry. Audio reported zero failures, skips, or stale
packets in the same interval. The impairment proxy forwarded 2,147,356 of
2,169,105 datagrams with zero queue overflow.
- Sleipnir latest-state delivery remained live through seeded 1% loss. Cutting
the proxy caused the non-neutral virtual Xbox pad to neutralize after 1.56 s,
owned by the named stale-input lease; restoring the hidden proxy reconnected
and resumed state without operator action.
- A continuous 64-packet receive drain was retaining the newest input for about
48-57 ms before HID application. Bounding that drain to 2 ms reduced live
receiver-owned receive-to-apply time to typically 69-204 microseconds. A
witnessed 16 ms spike remains to characterize.
- Starfire and Raven wall clocks differ by roughly 2.38 s. Sleipnir no longer
publishes their raw subtraction as one-way latency. Its surface reports that
source-clock latency is unavailable without a clock model and exposes only
monotonic receiver-owned stage durations.
These results clear the initial 99.9% media-completion and disconnect-safety
gates at 1% independent loss. They do not yet prove glass-to-glass latency,
A/V skew, ordered quick-tap delivery under loss, 5%/burst behavior, or a long
mixed-impairment soak. Those remain acceptance gates, not footnotes.
The field comparison also exposed a generic transport invariant Moonlight's
long-running sessions force us to respect: bounded wire identifiers must wrap
deliberately. CultNet's fragmented-message ID previously saturated at the
maximum `u16`, so fragmented audio became unreassemblable after the boundary.
Wrap-to-one plus a boundary test removed the repeatable minute-eight collapse;
a subsequent 10-minute direct LAN run remained clean across the boundary.
## 2026-07-17 ordered-input closure
The quick-tap gate exposed four authorities that generic RUDP reliability had
blurred together:
- A Windows sleep requested at two milliseconds could still miss a short tap.
Muninn now observes XInput on a dedicated high-resolution worker; transport,
discovery, serialization, and logging cannot delay capture.
- Global cumulative ACKs could not retire a reliable packet arriving more than
32 realtime sequence numbers late, and reliable duplicates were not ACKed
after application deduplication. CultNet now directly acknowledges the late
packet and acknowledges reliable duplicates. This collapsed a runaway field
trace from hundreds of thousands of resend datagrams to a bounded flow.
- Muninn used semantic application ACK as permission to send the next edge.
One delayed ACK therefore serialized the entire controller. It now sends a
bounded window of captured edges; CultNet owns retransmission, Sleipnir's
epoch/sequence cursor owns ordering, and semantic ACK only retires history.
- Sleipnir published CultMesh/Eve and Idunn state synchronously on the actuator
thread. Those writes produced 0.7-1.8 second visible button holds. A bounded
coalescing publisher worker now owns control-plane projection, leaving the
input thread to receive, order, actuate, and acknowledge.
The resulting first-hot-plug field proof generated 100 taps with a 16 ms source
press and 84 ms gap. Under seeded 1% independent loss, every layer observed
exactly 100 presses and 100 releases; Raven-visible durations were 24-57 ms.
Under eight consecutive dropped packets every 200 datagrams, every layer again
observed exactly 100 presses and 100 releases; visible durations were 24-40 ms,
the final virtual pad was neutral, and the impairment proxy reported no queue
overflow. Raven ran through hidden scheduled tasks over `192.168.1.0/24`; no
WireGuard route participated. Both daemons used the Eve/Aetheria/VoidBot
experimental CultLib lineage pinned at `8965f3c0`.
This closes ordered digital edges at one 60 Hz source frame. It does not prove
sub-frame synthetic ViGEm pulses: eight-millisecond state-only pulses can fall
between XInput snapshots, and `XInputGetKeystroke` was rejected after it emitted
stale events from a reused user slot. Analog motion under a sender stall,
mixed media/input endurance, glass-to-glass video age, and A/V skew remain open.

View file

@ -0,0 +1,100 @@
# Moonlight-level reliability acceptance ledger
This ledger distinguishes implemented mechanisms from evidence that the actual
Raven-to-Starfire program timeline survives pressure. Passing unit tests is not
permission to call the field path done.
## Required contracts
| Contract | Current authority | Evidence | Status |
|---|---|---|---|
| Experimental CultMesh/CultNet lineage | Odin vendored CultLib snapshot `8965f3c0` | Snapshot provenance plus CultNet/Muninn/Sleipnir suites | proved locally |
| Low-delay NVENC | Muninn controllable encoder | D3D11 desktop capture, no B frames/lookahead, two-frame VBV, clean Annex-B decode | proved locally |
| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default/250 ms Raven profile, bounded queues/cache/reliable expiry, late-frame repair refusal | proved structurally |
| Recoverable video loss avoids decoder reset | Mimir V4 block FEC/repair, Odin repair cache | Exhaustive 2..80 chunk, burst 1..8 tests in Rust/C++; completed-frame tombstones | proved locally |
| Decoder-chain loss forces immediate IDR | Muninn feedback owner and native encoder | Live 2560x1440 command produced IDR at frame 22 under a 600-frame GOP | proved locally |
| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Half-ceiling start and protected cap; live 12-to-6 Mbps reconfiguration produced clean IDR | proved locally |
| Audio survives small erasure bursts | Odin 4+2 GF(256), Mimir FEC/reorder/concealment | Exhaustive one/two-erasure Rust and C++ tests; production DLL build | proved locally |
| Latest input state supersedes stale state | Muninn epoch/state sequence, Sleipnir cursor | Duplicate/reorder/stale epoch tests | proved locally |
| Button edges are not lost or retained forever | Muninn replay/app ACK/epoch rebase, Sleipnir ordered cursor | Quick-tap, ACK, overflow rebase, sparse-gap tests | proved locally |
| Raven runtime is LAN-only and non-interrupting | Hidden Raven scheduled task, direct LAN SSH/media | One coherent Muninn/WASAPI/NVENC/FFmpeg tree; no WireGuard route; `-WindowStyle Hidden` | proved operationally |
| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | Direct LAN clean run 05:34:17–05:44:37: 79,200 records, 697,450,085 bytes, 18,588 recovered shards, zero decode/assembly/media/audio failures and zero sender capacity eviction; proxy matrix and long soak remain | **direct clean passed** |
## Field acceptance matrix
Run the real capture body, production receiver plugin, decoder, audio output,
and Sleipnir virtual input through the seeded profiles under
`E:\Projects\Odin\tests\realtime-impairment`. Observe the rendered/audio/input
timeline, not only proxy CSV counters.
| Profile | Required result |
|---|---|
| clean, 10 minutes | no decoder reset, audio discontinuity, input edge loss, queue growth, or timestamp drift; video p99 delivery under 50 ms |
| iid 1%, 10 minutes | no input edge loss; audio FEC recovers eligible holes; at least 99.9% video frames presented; media p99 under 80 ms |
| iid 3%, 5 minutes | bounded degradation; at least 99% video frames presented; decoder recovery within 250 ms; no stuck controls |
| burst 4 and burst 8 | parity/repair handles eligible damage; unrecoverable dependency loss produces one requested IDR and recovery within 250 ms |
| reorder and duplicate | no duplicate input action, stale state regression, or unbounded assembly/reorder storage |
| delay/jitter | AIMD reduces bitrate before sustained deadline loss and recovers no faster than one step per two stable seconds |
| stall 250 ms | obsolete media expires; queues remain bounded; input latest state converges within 35 ms after release |
| reconnect | old input epoch cannot act; neutral/current state within 100 ms; decoded video within 500 ms |
| mixed 60-minute soak | stable memory/handle/thread counts and no progressive A/V drift |
The first seeded 1% proxy run is a failed acceptance result, not a pass: 987 of
95,741 datagrams were dropped; in roughly one minute the receiver recorded 38
expired video assemblies and 56 skipped audio packets while remaining bounded
and avoiding decoder reset. CultNet fragmentation currently sits below
application-level FEC, so one lost transport fragment discards an entire FEC
shard. The next reliability cut is to make protected media shards fit one wire
datagram or move fragmentation above the erasure-code boundary.
## Field scars
- A fragmented CultNet RUDP session failed repeatably after roughly eight to
nine minutes because the `u16` fragment-set ID used saturating increment and
became permanently stuck at 65,535. Every later fragmented audio document
then collided in one receiver reassembly key. The allocator now wraps from
65,535 to 1, with a boundary test. The clean field run above crossed that
boundary without a single audio reorder, skip, stale packet, or failure.
- Audio and video producer ingress now use separate bounded channels. Video
capture can no longer block audio ingestion before the audio-priority send
scheduler sees it. Capacity-eviction counters proved zero during the clean
acceptance run.
## 2026-07-17 bounded-media field cut
The live path now keeps every protected shard in one datagram, sends AAC ADTS
in fixed 864-byte 4+2 audio blocks, uses 768-byte video shards with 8+4 block
FEC, and begins each decoder epoch at a complete IDR. The sender admits video
to its repair cache when the frame enters the queue, accepts the production
CultNet `realtime` channel, and can service bounded repair requests at
64/96/128 ms in batches of up to eight chunks.
Seeded Raven-to-Starfire evidence:
- iid 1%: 802 of 78,406 datagrams dropped; zero decode drop, video expiry,
audio failure, or audio skip; 1,800 video frames recovered;
- burst-4: 1,436 of 71,910 datagrams dropped; zero expiry or audio skip across
1,800 forwarded frames;
- reorder: 2,872 reorder events; zero expiry or audio skip across 900 frames;
- iid 3%: 2,564 of 85,731 datagrams dropped; three of 5,400 frames expired,
zero audio skip (99.94 percent video delivery);
- burst-8: 190 chunks repaired; one of 1,800 frames expired, zero audio skip;
- 250 ms stall: 420 datagrams withheld, one frame expired, 68 chunks repaired,
an IDR was requested, and another 900 frames arrived without a stuck decoder
or audio skip.
Delay/jitter produced one expiry in 900 frames. Duplicate, reconnect, longer
iid runs, and the 60-minute mixed soak remain open acceptance rows.
Field scar: production video uses the `realtime` channel, but the repair cache
previously admitted only legacy `media` payloads. Synthetic tests built the
legacy shape and certified a repair path that was empty in production. The
production-channel regression test now owns this boundary.
## Completion rule
Moonlight-level performance is reasonably expected only after all local rows
remain green and the real field matrix passes twice: once on direct LAN and once
on the deployed route actually used during streaming. Failure should change the
owning deadline, FEC, encoder, or input authority—not add a repair loop around
the symptom.

View file

@ -4,12 +4,13 @@
Build the smallest live Mimir application that can ingest multiple local or
networked video and audio feeds, hold a short synchronized window in memory,
and emit coherent OBS-facing program outputs while Fensalir provides the
operator UI.
and emit coherent Mimir program outputs while Eve provides operator surfaces
and Yggdrasil publishes the site feed.
The default reservoir horizon is five seconds. That delay is not decoration. It
is the budget that lets late network packets, independent capture clocks,
audio/video alignment, and debug controls converge before OBS sees the result.
audio/video alignment, and debug controls converge before Mimir emits the
program result.
Mimir is explicitly trading latency and memory for throughput and extraction
quality: the app may spend that buffered window lining streams up and crunching
audio/video data to recover as much of the volumetric audio field and
@ -21,16 +22,16 @@ alignment and field-reconstruction budget.
Today the repo has two partial machines:
- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable OBS
Media Source endpoints.
- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable
compatibility endpoints.
- The native `LocalcastRuntime` reservoir ABI can hold typed live sample handles,
and Fensalir can consume native render packets for GPU fusion.
Those parts are useful, but they are not yet the viable app. The viable app is
the join: ingest workers append live sample handles into one Fensalir-hosted
runtime window, Fensalir renders video from that window, Faust/native audio
builds synchronized stems from the same window, and OBS receives only the
program outputs.
builds synchronized stems from the same window, Mimir commits the scene graph,
Eve lowers the operator surface, and Yggdrasil publishes the program output.
```mermaid
flowchart TD
@ -39,10 +40,15 @@ flowchart TD
C["local mic/loopback workers"] --> R
D["network audio ingest workers"] --> R
R --> U["Fensalir debug/settings/output UI"]
R --> V["Fensalir GPU video composition"]
R --> S["Mimir program scene graph"]
S --> U
S --> V["Fensalir GPU video composition"]
R --> F["Faust/native audio alignment + stems"]
V --> O["OBS video output"]
F --> P["OBS audio stem outputs"]
S --> E["Eve GUI/TUI operator surfaces"]
V --> Y["Yggdrasil site publisher"]
F --> Y
V -. "compatibility" .-> O["OBS adapter"]
F -. "compatibility" .-> O
```
## Invariants
@ -51,12 +57,17 @@ flowchart TD
- The five-second rolling window is a deliberate compute budget, not accidental
latency. It exists so capture, sync, GPU fusion, and audio field extraction
can choose coherence over immediacy.
- Fensalir owns the running app, UI, debug state, settings changes, output
management, GPU fusion/rendering, and Spout/video publication.
- Mimir owns the program scene graph, source subscription policy, preview/control
state, stats, and publication intent.
- Fensalir owns the running app host, UI lowering, debug state, GPU
fusion/rendering, and local program texture output.
- Faust/native DSP owns audio alignment, voice separation, suppression,
volumetric sound-field extraction, stems, and spatial bed generation.
- OBS receives synchronized program surfaces. Raw unsynchronized feeds may be
diagnostic inputs, not broadcast truth.
- Eve lowerers render controls and previews without owning scene truth.
- The Yggdrasil publisher consumes synchronized program surfaces without owning
a second composition.
- OBS compatibility adapters may receive synchronized program surfaces. Raw
unsynchronized feeds may be diagnostic inputs, not broadcast truth.
- Networked feeds are producers into the same reservoir. They do not own clocks,
private buffers, or a parallel synchronization model.
- Missing feeds become explicit absence or silent/blank placeholders inside the
@ -78,9 +89,14 @@ The first cut should be deliberately boring:
loopback ingest.
7. Expose Fensalir UI readouts for reservoir edge, window start, per-kind
counts, feed health, delay estimate, output status, and current settings.
8. Render one synchronized program video output from the reservoir.
9. Emit separately controllable OBS audio stems from the same reservoir window.
10. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture
8. Render one synchronized Mimir program video output from the reservoir.
9. Emit separately controllable program audio stems from the same reservoir
window.
10. Publish Eve GUI/TUI operator surfaces for preview, source selection, scene
edits, stats, and output state.
11. Add the Yggdrasil-facing publisher daemon that consumes Mimir program
output without owning composition.
12. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture
and OBS compatibility, not as the stream authority.
## Repo Shape

View file

@ -0,0 +1,66 @@
cmake_minimum_required(VERSION 3.20)
project(mimir_obs_stem_source LANGUAGES CXX)
find_package(libobs REQUIRED CONFIG)
include(CTest)
add_library(mimir_obs_stem_source MODULE
mimir_obs_muninn_source.cpp
mimir_obs_program_texture_source.cpp
mimir_obs_stem_source.cpp)
target_compile_features(mimir_obs_stem_source PRIVATE cxx_std_20)
target_compile_definitions(mimir_obs_stem_source PRIVATE
WIN32_LEAN_AND_MEAN
NOMINMAX
_WIN32_WINNT=0x0A00)
target_link_libraries(mimir_obs_stem_source PRIVATE OBS::libobs d3d11 d3d12 dxgi ws2_32)
if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/../../tools/muninn_media_decoder_probe.cpp")
add_executable(muninn_media_decoder_probe
../../tools/muninn_media_decoder_probe.cpp)
target_compile_features(muninn_media_decoder_probe PRIVATE cxx_std_20)
target_include_directories(muninn_media_decoder_probe PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
endif()
add_executable(muninn_rudp_url_tests
tests/muninn_rudp_url_tests.cpp)
target_compile_features(muninn_rudp_url_tests PRIVATE cxx_std_20)
target_include_directories(muninn_rudp_url_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_rudp_url_tests COMMAND muninn_rudp_url_tests)
add_executable(muninn_media_wire_tests
tests/muninn_media_wire_tests.cpp)
target_compile_features(muninn_media_wire_tests PRIVATE cxx_std_20)
target_include_directories(muninn_media_wire_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_media_wire_tests COMMAND muninn_media_wire_tests)
add_executable(muninn_audio_fec_tests
tests/muninn_audio_fec_tests.cpp)
target_compile_features(muninn_audio_fec_tests PRIVATE cxx_std_20)
target_include_directories(muninn_audio_fec_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_audio_fec_tests COMMAND muninn_audio_fec_tests)
add_executable(muninn_video_fec_tests
tests/muninn_video_fec_tests.cpp)
target_compile_features(muninn_video_fec_tests PRIVATE cxx_std_20)
target_include_directories(muninn_video_fec_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_video_fec_tests COMMAND muninn_video_fec_tests)
add_executable(muninn_udp_socket_tests
tests/muninn_udp_socket_tests.cpp)
target_compile_features(muninn_udp_socket_tests PRIVATE cxx_std_20)
target_compile_definitions(muninn_udp_socket_tests PRIVATE
WIN32_LEAN_AND_MEAN
NOMINMAX
_WIN32_WINNT=0x0A00)
target_include_directories(muninn_udp_socket_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(muninn_udp_socket_tests PRIVATE ws2_32)
add_test(NAME muninn_udp_socket_tests COMMAND muninn_udp_socket_tests)

View file

@ -0,0 +1,14 @@
MimirAudioStem="Mimir Audio Stem"
MimirAudioStem.MapName="Shared memory map"
MimirAudioStem.StemId="Stem id"
MimirProgramTexture="Mimir Program Texture"
MuninnStream="Muninn Stream"
MuninnVideoSource="Muninn Video"
MuninnAudioSource="Muninn Audio"
MuninnStream.Stream="Muninn stream"
MuninnStream.Video="Video device"
MuninnStream.Audio="Audio device"
MuninnStream.VideoBitrateKbps="Video bitrate (kbps)"
MuninnStream.VideoLatencyMs="Video/transport latency (ms)"
MuninnStream.AudioReorderMs="Audio reorder latency (ms)"
MuninnStream.Refresh="Refresh Muninn streams"

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,628 @@
#include <obs-module.h>
#include <util/platform.h>
#include <chrono>
#include <cstdint>
#include <string>
#include <utility>
#include <vector>
#include <d3d11_1.h>
#include <d3d12.h>
#include <dxgi1_2.h>
#include <wrl/client.h>
#include <windows.h>
using Microsoft::WRL::ComPtr;
namespace {
struct MimirProgramTextureSource {
obs_source_t *source = nullptr;
std::string texture_name = "Global\\MimirFensalirProgramTexture";
std::string fence_name = "Global\\MimirFensalirProgramFence";
std::string consumer_fence_name = "Global\\MimirObsProgramConsumerFence";
uint32_t ring_count = 1;
uint32_t width = 0;
uint32_t height = 0;
uint64_t fence_value = 0;
uint64_t last_producer_fence_value = 0;
uint64_t next_retry_ns = 0;
uint64_t next_failure_log_ns = 0;
HANDLE bridge_shared_handle = nullptr;
HANDLE consumer_fence_shared_handle = nullptr;
HANDLE fence_event = nullptr;
gs_texture_t *obs_texture = nullptr;
ComPtr<ID3D12Device> d3d12_device;
ComPtr<ID3D12CommandQueue> command_queue;
ComPtr<ID3D12CommandAllocator> command_allocator;
ComPtr<ID3D12GraphicsCommandList> command_list;
ComPtr<ID3D12Fence> fence;
ComPtr<ID3D12Fence> producer_fence;
ComPtr<ID3D12Fence> consumer_fence;
std::vector<ComPtr<ID3D12Resource>> source_textures;
ComPtr<ID3D12Resource> bridge_texture_d3d12;
ComPtr<ID3D11Device> d3d11_device;
ComPtr<ID3D11DeviceContext> d3d11_context;
ComPtr<ID3D11Texture2D> bridge_texture_d3d11;
};
static void log_retry_failure(MimirProgramTextureSource *ctx, const char *message, HRESULT hr = S_OK)
{
const uint64_t now = os_gettime_ns();
if (now < ctx->next_failure_log_ns) {
return;
}
ctx->next_failure_log_ns = now + 2000000000ULL;
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: %s: 0x%08lx", message, static_cast<unsigned long>(hr));
} else {
blog(LOG_WARNING, "Mimir OBS program texture: %s", message);
}
}
static std::wstring utf8_to_wide(const std::string &value)
{
if (value.empty()) {
return {};
}
const int count = MultiByteToWideChar(CP_UTF8, 0, value.c_str(), -1, nullptr, 0);
if (count <= 0) {
return {};
}
std::wstring wide(static_cast<size_t>(count), L'\0');
MultiByteToWideChar(CP_UTF8, 0, value.c_str(), -1, wide.data(), count);
if (!wide.empty() && wide.back() == L'\0') {
wide.pop_back();
}
return wide;
}
static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback)
{
const char *value = obs_data_get_string(settings, key);
return value && *value ? std::string(value) : std::string(fallback);
}
static uint32_t clamp_ring_count(long long value)
{
if (value < 1) {
return 1;
}
if (value > 4) {
return 4;
}
return static_cast<uint32_t>(value);
}
static std::string texture_name_for_slot(const MimirProgramTextureSource *ctx, uint32_t slot)
{
if (ctx->ring_count <= 1) {
return ctx->texture_name;
}
return ctx->texture_name + "." + std::to_string(slot);
}
static void release_bridge(MimirProgramTextureSource *ctx)
{
if (ctx->obs_texture) {
gs_texture_destroy(ctx->obs_texture);
ctx->obs_texture = nullptr;
}
ctx->bridge_texture_d3d12.Reset();
ctx->bridge_texture_d3d11.Reset();
ctx->source_textures.clear();
ctx->fence.Reset();
ctx->producer_fence.Reset();
ctx->consumer_fence.Reset();
ctx->command_list.Reset();
ctx->command_allocator.Reset();
ctx->command_queue.Reset();
ctx->d3d12_device.Reset();
ctx->d3d11_context.Reset();
ctx->d3d11_device.Reset();
ctx->width = 0;
ctx->height = 0;
ctx->fence_value = 0;
ctx->last_producer_fence_value = 0;
if (ctx->bridge_shared_handle) {
CloseHandle(ctx->bridge_shared_handle);
ctx->bridge_shared_handle = nullptr;
}
if (ctx->consumer_fence_shared_handle) {
CloseHandle(ctx->consumer_fence_shared_handle);
ctx->consumer_fence_shared_handle = nullptr;
}
if (ctx->fence_event) {
CloseHandle(ctx->fence_event);
ctx->fence_event = nullptr;
}
}
static bool wait_for_copy(MimirProgramTextureSource *ctx, uint64_t consumed_producer_value)
{
if (ctx->consumer_fence && consumed_producer_value > 0) {
HRESULT consumer_hr = ctx->command_queue->Signal(ctx->consumer_fence.Get(), consumed_producer_value);
if (FAILED(consumer_hr)) {
blog(LOG_WARNING,
"Mimir OBS program texture: consumer fence Signal failed: 0x%08lx",
static_cast<unsigned long>(consumer_hr));
}
}
const uint64_t value = ++ctx->fence_value;
HRESULT hr = ctx->command_queue->Signal(ctx->fence.Get(), value);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D12 Signal failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
if (ctx->fence->GetCompletedValue() >= value) {
return true;
}
hr = ctx->fence->SetEventOnCompletion(value, ctx->fence_event);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: SetEventOnCompletion failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
WaitForSingleObject(ctx->fence_event, 100);
return ctx->fence->GetCompletedValue() >= value;
}
static bool create_devices(MimirProgramTextureSource *ctx)
{
HRESULT hr = D3D12CreateDevice(nullptr, D3D_FEATURE_LEVEL_11_0, IID_PPV_ARGS(&ctx->d3d12_device));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D12CreateDevice failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
D3D12_COMMAND_QUEUE_DESC queue_desc = {};
queue_desc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
hr = ctx->d3d12_device->CreateCommandQueue(&queue_desc, IID_PPV_ARGS(&ctx->command_queue));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandQueue failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = ctx->d3d12_device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, IID_PPV_ARGS(&ctx->command_allocator));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandAllocator failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = ctx->d3d12_device->CreateCommandList(
0,
D3D12_COMMAND_LIST_TYPE_DIRECT,
ctx->command_allocator.Get(),
nullptr,
IID_PPV_ARGS(&ctx->command_list));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandList failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ctx->command_list->Close();
hr = ctx->d3d12_device->CreateFence(0, D3D12_FENCE_FLAG_NONE, IID_PPV_ARGS(&ctx->fence));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateFence failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ctx->fence_event = CreateEventW(nullptr, FALSE, FALSE, nullptr);
if (!ctx->fence_event) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateEvent failed: %lu", GetLastError());
return false;
}
UINT flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT;
D3D_FEATURE_LEVEL feature_levels[] = {D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0};
D3D_FEATURE_LEVEL selected = D3D_FEATURE_LEVEL_11_0;
hr = D3D11CreateDevice(
nullptr,
D3D_DRIVER_TYPE_HARDWARE,
nullptr,
flags,
feature_levels,
2,
D3D11_SDK_VERSION,
&ctx->d3d11_device,
&selected,
&ctx->d3d11_context);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D11CreateDevice failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
return true;
}
static void create_consumer_fence(MimirProgramTextureSource *ctx)
{
if (ctx->consumer_fence_name.empty()) {
return;
}
HRESULT hr = ctx->d3d12_device->CreateFence(0, D3D12_FENCE_FLAG_SHARED, IID_PPV_ARGS(&ctx->consumer_fence));
if (FAILED(hr)) {
log_retry_failure(ctx, "CreateFence consumer fence failed", hr);
return;
}
const auto wide_name = utf8_to_wide(ctx->consumer_fence_name);
if (wide_name.empty()) {
log_retry_failure(ctx, "consumer fence name is empty or invalid UTF-8");
ctx->consumer_fence.Reset();
return;
}
hr = ctx->d3d12_device->CreateSharedHandle(
ctx->consumer_fence.Get(),
nullptr,
GENERIC_ALL,
wide_name.c_str(),
&ctx->consumer_fence_shared_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "CreateSharedHandle consumer fence failed", hr);
ctx->consumer_fence.Reset();
return;
}
blog(LOG_INFO, "Mimir OBS program texture consumer fence published %s", ctx->consumer_fence_name.c_str());
}
static bool open_source_texture(MimirProgramTextureSource *ctx)
{
ctx->source_textures.clear();
ctx->source_textures.reserve(ctx->ring_count);
for (uint32_t slot = 0; slot < ctx->ring_count; ++slot) {
const auto slot_name = texture_name_for_slot(ctx, slot);
const auto wide_name = utf8_to_wide(slot_name);
if (wide_name.empty()) {
log_retry_failure(ctx, "shared texture name is empty or invalid UTF-8");
return false;
}
HANDLE source_handle = nullptr;
HRESULT hr = ctx->d3d12_device->OpenSharedHandleByName(wide_name.c_str(), GENERIC_ALL, &source_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandleByName failed", hr);
return false;
}
ComPtr<ID3D12Resource> source_texture;
hr = ctx->d3d12_device->OpenSharedHandle(source_handle, IID_PPV_ARGS(&source_texture));
CloseHandle(source_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandle failed", hr);
return false;
}
const D3D12_RESOURCE_DESC desc = source_texture->GetDesc();
if (desc.Dimension != D3D12_RESOURCE_DIMENSION_TEXTURE2D || desc.Format != DXGI_FORMAT_B8G8R8A8_UNORM) {
log_retry_failure(ctx, "expected BGRA8 D3D12 Texture2D");
return false;
}
const uint32_t slot_width = static_cast<uint32_t>(desc.Width);
const uint32_t slot_height = desc.Height;
if (slot_width == 0 || slot_height == 0) {
log_retry_failure(ctx, "shared texture dimensions are empty");
return false;
}
if (slot == 0) {
ctx->width = slot_width;
ctx->height = slot_height;
} else if (slot_width != ctx->width || slot_height != ctx->height) {
log_retry_failure(ctx, "program output ring texture dimensions do not match");
return false;
}
ctx->source_textures.push_back(std::move(source_texture));
}
return ctx->source_textures.size() == ctx->ring_count;
}
static bool open_producer_fence(MimirProgramTextureSource *ctx)
{
const auto wide_name = utf8_to_wide(ctx->fence_name);
if (wide_name.empty()) {
return true;
}
HANDLE fence_handle = nullptr;
HRESULT hr = ctx->d3d12_device->OpenSharedHandleByName(wide_name.c_str(), GENERIC_ALL, &fence_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandleByName producer fence failed", hr);
return true;
}
hr = ctx->d3d12_device->OpenSharedHandle(fence_handle, IID_PPV_ARGS(&ctx->producer_fence));
CloseHandle(fence_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandle producer fence failed", hr);
ctx->producer_fence.Reset();
return true;
}
ctx->last_producer_fence_value = ctx->producer_fence->GetCompletedValue();
if (ctx->last_producer_fence_value == UINT64_MAX) {
log_retry_failure(ctx, "producer fence opened in device-removed state");
ctx->producer_fence.Reset();
return false;
}
blog(LOG_INFO,
"Mimir OBS program texture producer fence opened %s value=%llu",
ctx->fence_name.c_str(),
static_cast<unsigned long long>(ctx->last_producer_fence_value));
return true;
}
static bool read_producer_fence(MimirProgramTextureSource *ctx, uint64_t *completed)
{
*completed = 0;
if (!ctx->producer_fence) {
return true;
}
*completed = ctx->producer_fence->GetCompletedValue();
if (*completed == UINT64_MAX) {
log_retry_failure(ctx, "producer fence reported device removal");
return false;
}
if (*completed > ctx->last_producer_fence_value) {
ctx->last_producer_fence_value = *completed;
}
return true;
}
static bool create_obs_bridge_texture(MimirProgramTextureSource *ctx)
{
D3D11_TEXTURE2D_DESC desc = {};
desc.Width = ctx->width;
desc.Height = ctx->height;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED;
HRESULT hr = ctx->d3d11_device->CreateTexture2D(&desc, nullptr, &ctx->bridge_texture_d3d11);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateTexture2D bridge failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ComPtr<IDXGIResource> legacy_resource;
hr = ctx->bridge_texture_d3d11.As(&legacy_resource);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: bridge texture is not IDXGIResource: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = legacy_resource->GetSharedHandle(&ctx->bridge_shared_handle);
if (FAILED(hr) || !ctx->bridge_shared_handle) {
blog(LOG_WARNING, "Mimir OBS program texture: GetSharedHandle bridge failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = ctx->d3d12_device->OpenSharedHandle(ctx->bridge_shared_handle, IID_PPV_ARGS(&ctx->bridge_texture_d3d12));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D12 open legacy bridge handle failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ctx->obs_texture = gs_texture_open_shared(static_cast<uint32_t>(reinterpret_cast<uintptr_t>(ctx->bridge_shared_handle)));
if (!ctx->obs_texture) {
blog(LOG_WARNING, "Mimir OBS program texture: OBS failed to open legacy bridge shared texture");
return false;
}
return true;
}
static bool ensure_bridge(MimirProgramTextureSource *ctx)
{
if (ctx->obs_texture) {
return true;
}
const uint64_t now = os_gettime_ns();
if (now < ctx->next_retry_ns) {
return false;
}
ctx->next_retry_ns = now + 1000000000ULL;
release_bridge(ctx);
if (!create_devices(ctx)) {
release_bridge(ctx);
return false;
}
create_consumer_fence(ctx);
if (!open_source_texture(ctx) || !create_obs_bridge_texture(ctx)) {
release_bridge(ctx);
return false;
}
if (!open_producer_fence(ctx)) {
release_bridge(ctx);
return false;
}
blog(LOG_INFO, "Mimir OBS program texture source opened %s (%ux%u ring=%u)",
ctx->texture_name.c_str(),
ctx->width,
ctx->height,
ctx->ring_count);
return true;
}
static bool copy_source_to_bridge(MimirProgramTextureSource *ctx)
{
HRESULT hr = ctx->command_allocator->Reset();
if (FAILED(hr)) {
return false;
}
uint64_t completed_producer_value = 0;
if (!read_producer_fence(ctx, &completed_producer_value)) {
release_bridge(ctx);
return false;
}
const uint32_t source_slot = ctx->source_textures.empty()
? 0
: static_cast<uint32_t>(completed_producer_value % ctx->source_textures.size());
ID3D12Resource *source_texture = ctx->source_textures[source_slot].Get();
hr = ctx->command_list->Reset(ctx->command_allocator.Get(), nullptr);
if (FAILED(hr)) {
return false;
}
D3D12_RESOURCE_BARRIER barriers[2] = {};
barriers[0].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barriers[0].Transition.pResource = source_texture;
barriers[0].Transition.StateBefore = D3D12_RESOURCE_STATE_COMMON;
barriers[0].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_SOURCE;
barriers[0].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
barriers[1].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barriers[1].Transition.pResource = ctx->bridge_texture_d3d12.Get();
barriers[1].Transition.StateBefore = D3D12_RESOURCE_STATE_COMMON;
barriers[1].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_DEST;
barriers[1].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
ctx->command_list->ResourceBarrier(2, barriers);
ctx->command_list->CopyResource(ctx->bridge_texture_d3d12.Get(), source_texture);
std::swap(barriers[0].Transition.StateBefore, barriers[0].Transition.StateAfter);
std::swap(barriers[1].Transition.StateBefore, barriers[1].Transition.StateAfter);
ctx->command_list->ResourceBarrier(2, barriers);
hr = ctx->command_list->Close();
if (FAILED(hr)) {
return false;
}
ID3D12CommandList *lists[] = {ctx->command_list.Get()};
ctx->command_queue->ExecuteCommandLists(1, lists);
return wait_for_copy(ctx, completed_producer_value);
}
static const char *mimir_program_texture_get_name(void *)
{
return obs_module_text("MimirProgramTexture");
}
static void mimir_program_texture_update(void *data, obs_data_t *settings)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
ctx->texture_name = obs_string(settings, "texture_name", "Global\\MimirFensalirProgramTexture");
ctx->fence_name = obs_string(settings, "fence_name", "Global\\MimirFensalirProgramFence");
ctx->consumer_fence_name = obs_string(settings, "consumer_fence_name", "Global\\MimirObsProgramConsumerFence");
ctx->ring_count = clamp_ring_count(obs_data_get_int(settings, "ring_count"));
release_bridge(ctx);
}
static void *mimir_program_texture_create(obs_data_t *settings, obs_source_t *source)
{
auto *ctx = new MimirProgramTextureSource();
ctx->source = source;
mimir_program_texture_update(ctx, settings);
return ctx;
}
static void mimir_program_texture_destroy(void *data)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
release_bridge(ctx);
delete ctx;
}
static uint32_t mimir_program_texture_width(void *data)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
ensure_bridge(ctx);
return ctx->width;
}
static uint32_t mimir_program_texture_height(void *data)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
ensure_bridge(ctx);
return ctx->height;
}
static void mimir_program_texture_defaults(obs_data_t *settings)
{
obs_data_set_default_string(settings, "texture_name", "Global\\MimirFensalirProgramTexture");
obs_data_set_default_string(settings, "fence_name", "Global\\MimirFensalirProgramFence");
obs_data_set_default_string(settings, "consumer_fence_name", "Global\\MimirObsProgramConsumerFence");
obs_data_set_default_int(settings, "ring_count", 1);
}
static obs_properties_t *mimir_program_texture_properties(void *)
{
obs_properties_t *props = obs_properties_create();
obs_properties_add_text(props, "texture_name", "Shared D3D12 texture name", OBS_TEXT_DEFAULT);
obs_properties_add_text(props, "fence_name", "Shared D3D12 fence name", OBS_TEXT_DEFAULT);
obs_properties_add_text(props, "consumer_fence_name", "Consumer D3D12 fence name", OBS_TEXT_DEFAULT);
obs_properties_add_int(props, "ring_count", "Texture ring slots", 1, 4, 1);
return props;
}
static void mimir_program_texture_render(void *data, gs_effect_t *)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
if (!ensure_bridge(ctx)) {
return;
}
if (copy_source_to_bridge(ctx)) {
obs_source_draw(ctx->obs_texture, 0, 0, ctx->width, ctx->height, false);
}
}
static obs_source_info mimir_program_texture_source_info = {
.id = "mimir_program_texture_source",
.type = OBS_SOURCE_TYPE_INPUT,
.output_flags = OBS_SOURCE_VIDEO,
.get_name = mimir_program_texture_get_name,
.create = mimir_program_texture_create,
.destroy = mimir_program_texture_destroy,
.get_width = mimir_program_texture_width,
.get_height = mimir_program_texture_height,
.get_defaults = mimir_program_texture_defaults,
.get_properties = mimir_program_texture_properties,
.update = mimir_program_texture_update,
.video_render = mimir_program_texture_render,
};
} // namespace
void mimir_register_program_texture_source()
{
obs_register_source(&mimir_program_texture_source_info);
}

View file

@ -0,0 +1,43 @@
#pragma once
#include <cstdint>
constexpr uint64_t MIMIR_OBS_STEM_MAGIC = 0x4D545352494D494Dull;
constexpr int MIMIR_OBS_STEM_VERSION = 1;
constexpr int MIMIR_OBS_STEM_MAX_STEMS = 8;
constexpr int MIMIR_OBS_STEM_MAX_FRAMES = 4096;
constexpr int MIMIR_OBS_STEM_ID_BYTES = 64;
constexpr int MIMIR_OBS_STEM_DISPLAY_BYTES = 64;
constexpr int MIMIR_OBS_STEM_SOURCE_BYTES = 64;
constexpr int MIMIR_OBS_STEM_HEADER_BYTES = 64;
constexpr int MIMIR_OBS_STEM_RECORD_BYTES = 224;
constexpr int MIMIR_OBS_STEM_MAP_BYTES =
MIMIR_OBS_STEM_HEADER_BYTES +
(MIMIR_OBS_STEM_MAX_STEMS * MIMIR_OBS_STEM_RECORD_BYTES) +
(MIMIR_OBS_STEM_MAX_STEMS * MIMIR_OBS_STEM_MAX_FRAMES * static_cast<int>(sizeof(float)));
struct MimirObsStemHeader {
uint64_t magic;
int32_t version;
uint32_t generation;
int32_t stem_count;
int32_t max_stems;
int32_t max_frames;
int32_t header_bytes;
int32_t record_bytes;
int32_t map_bytes;
};
struct MimirObsStemRecord {
int32_t channel_index;
int32_t configured;
int32_t sample_rate;
int32_t frame_count;
int32_t sample_offset;
int32_t reserved0;
int32_t reserved1;
int32_t reserved2;
char stem_id[MIMIR_OBS_STEM_ID_BYTES];
char display_name[MIMIR_OBS_STEM_DISPLAY_BYTES];
char source_id[MIMIR_OBS_STEM_SOURCE_BYTES];
};

View file

@ -0,0 +1,220 @@
#include <obs-module.h>
#include <util/platform.h>
#include <atomic>
#include <chrono>
#include <cstring>
#include <string>
#include <thread>
#include <vector>
#include <windows.h>
#include "mimir_obs_stem_ipc.h"
OBS_DECLARE_MODULE()
OBS_MODULE_USE_DEFAULT_LOCALE("mimir-obs-stem-source", "en-US")
void mimir_register_program_texture_source();
void mimir_register_muninn_source();
struct MimirStemSource {
obs_source_t *source = nullptr;
std::string map_name = "Local\\MimirObsStemBus";
std::string stem_id = "aligned_source_0";
HANDLE mapping = nullptr;
const uint8_t *view = nullptr;
std::thread worker;
std::atomic_bool running = false;
uint32_t last_generation = 0;
};
static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback)
{
const char *value = obs_data_get_string(settings, key);
return value && *value ? std::string(value) : std::string(fallback);
}
static void close_mapping(MimirStemSource *ctx)
{
if (ctx->view) {
UnmapViewOfFile(ctx->view);
ctx->view = nullptr;
}
if (ctx->mapping) {
CloseHandle(ctx->mapping);
ctx->mapping = nullptr;
}
}
static bool open_mapping(MimirStemSource *ctx)
{
if (ctx->mapping) {
return true;
}
ctx->mapping = OpenFileMappingA(FILE_MAP_READ, FALSE, ctx->map_name.c_str());
if (!ctx->mapping) {
return false;
}
ctx->view = static_cast<const uint8_t *>(MapViewOfFile(ctx->mapping, FILE_MAP_READ, 0, 0, MIMIR_OBS_STEM_MAP_BYTES));
if (!ctx->view) {
close_mapping(ctx);
return false;
}
return true;
}
static const MimirObsStemRecord *find_record(const uint8_t *view, const std::string &stem_id, int stem_count)
{
const auto count = std::min(stem_count, MIMIR_OBS_STEM_MAX_STEMS);
for (int index = 0; index < count; ++index) {
const auto *record = reinterpret_cast<const MimirObsStemRecord *>(view + MIMIR_OBS_STEM_HEADER_BYTES + (index * MIMIR_OBS_STEM_RECORD_BYTES));
if (record->frame_count <= 0 || record->sample_rate <= 0 || record->sample_offset <= 0) {
continue;
}
if (stem_id == record->stem_id) {
return record;
}
}
return nullptr;
}
static void publish_if_available(MimirStemSource *ctx)
{
if (!open_mapping(ctx)) {
return;
}
const auto *header = reinterpret_cast<const MimirObsStemHeader *>(ctx->view);
const auto generation_a = header->generation;
if ((generation_a & 1u) != 0 || generation_a == ctx->last_generation) {
return;
}
if (header->magic != MIMIR_OBS_STEM_MAGIC ||
header->version != MIMIR_OBS_STEM_VERSION ||
header->map_bytes != MIMIR_OBS_STEM_MAP_BYTES) {
return;
}
const auto *record = find_record(ctx->view, ctx->stem_id, header->stem_count);
if (!record) {
return;
}
const auto frame_count = std::min(record->frame_count, MIMIR_OBS_STEM_MAX_FRAMES);
const auto sample_rate = record->sample_rate;
const auto byte_count = static_cast<size_t>(frame_count) * sizeof(float);
if (record->sample_offset < 0 || record->sample_offset + static_cast<int>(byte_count) > MIMIR_OBS_STEM_MAP_BYTES) {
return;
}
std::vector<float> samples(frame_count);
std::memcpy(samples.data(), ctx->view + record->sample_offset, byte_count);
const auto generation_b = header->generation;
if (generation_a != generation_b || (generation_b & 1u) != 0) {
return;
}
ctx->last_generation = generation_b;
uint8_t *planes[MAX_AV_PLANES] = {};
planes[0] = reinterpret_cast<uint8_t *>(samples.data());
obs_source_audio audio = {};
audio.data[0] = planes[0];
audio.frames = static_cast<uint32_t>(frame_count);
audio.speakers = SPEAKERS_MONO;
audio.samples_per_sec = static_cast<uint32_t>(sample_rate);
audio.format = AUDIO_FORMAT_FLOAT_PLANAR;
audio.timestamp = os_gettime_ns();
obs_source_output_audio(ctx->source, &audio);
}
static void worker_loop(MimirStemSource *ctx)
{
while (ctx->running.load()) {
publish_if_available(ctx);
std::this_thread::sleep_for(std::chrono::milliseconds(5));
}
}
static const char *mimir_stem_get_name(void *)
{
return obs_module_text("MimirAudioStem");
}
static void mimir_stem_update(void *data, obs_data_t *settings)
{
auto *ctx = static_cast<MimirStemSource *>(data);
ctx->map_name = obs_string(settings, "map_name", "Local\\MimirObsStemBus");
ctx->stem_id = obs_string(settings, "stem_id", "aligned_source_0");
ctx->last_generation = 0;
close_mapping(ctx);
}
static void *mimir_stem_create(obs_data_t *settings, obs_source_t *source)
{
auto *ctx = new MimirStemSource();
ctx->source = source;
mimir_stem_update(ctx, settings);
ctx->running = true;
ctx->worker = std::thread(worker_loop, ctx);
return ctx;
}
static void mimir_stem_destroy(void *data)
{
auto *ctx = static_cast<MimirStemSource *>(data);
ctx->running = false;
if (ctx->worker.joinable()) {
ctx->worker.join();
}
close_mapping(ctx);
delete ctx;
}
static void mimir_stem_defaults(obs_data_t *settings)
{
obs_data_set_default_string(settings, "map_name", "Local\\MimirObsStemBus");
obs_data_set_default_string(settings, "stem_id", "aligned_source_0");
}
static obs_properties_t *mimir_stem_properties(void *)
{
obs_properties_t *props = obs_properties_create();
obs_properties_add_text(props, "map_name", obs_module_text("MimirAudioStem.MapName"), OBS_TEXT_DEFAULT);
obs_properties_add_text(props, "stem_id", obs_module_text("MimirAudioStem.StemId"), OBS_TEXT_DEFAULT);
return props;
}
static obs_source_info mimir_stem_source_info = {
.id = "mimir_audio_stem_source",
.type = OBS_SOURCE_TYPE_INPUT,
.output_flags = OBS_SOURCE_AUDIO,
.get_name = mimir_stem_get_name,
.create = mimir_stem_create,
.destroy = mimir_stem_destroy,
.get_defaults = mimir_stem_defaults,
.get_properties = mimir_stem_properties,
.update = mimir_stem_update,
};
bool obs_module_load(void)
{
obs_register_source(&mimir_stem_source_info);
mimir_register_program_texture_source();
mimir_register_muninn_source();
blog(
LOG_INFO,
"Mimir OBS sources loaded: mimir_audio_stem_source, mimir_program_texture_source, "
"muninn_stream_source, muninn_video_source, muninn_audio_source; "
"Muninn Odin discovery build 20260705.2");
return true;
}

View file

@ -0,0 +1,179 @@
#pragma once
#include <array>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <iterator>
#include <map>
#include <optional>
#include <string>
#include <tuple>
#include <utility>
#include <vector>
namespace muninn_audio_fec {
inline constexpr size_t DataShards = 4;
inline constexpr size_t ParityShards = 2;
inline constexpr size_t TotalShards = DataShards + ParityShards;
using Bytes = std::vector<uint8_t>;
using Shards = std::array<std::optional<Bytes>, TotalShards>;
inline uint8_t multiply(uint8_t left, uint8_t right)
{
uint16_t product = 0;
uint16_t a = left;
uint16_t b = right;
while (b != 0) {
if ((b & 1u) != 0) product ^= a;
b >>= 1u;
a <<= 1u;
if ((a & 0x100u) != 0) a ^= 0x11du;
}
return static_cast<uint8_t>(product);
}
inline uint8_t power(uint8_t value, uint16_t exponent)
{
uint8_t result = 1;
while (exponent != 0) {
if ((exponent & 1u) != 0) result = multiply(result, value);
value = multiply(value, value);
exponent >>= 1u;
}
return result;
}
inline uint8_t inverse(uint8_t value)
{
return value == 0 ? 0 : power(value, 254);
}
inline constexpr std::array<std::array<uint8_t, DataShards>, TotalShards> Generator{{
{{1, 0, 0, 0}}, {{0, 1, 0, 0}}, {{0, 0, 1, 0}}, {{0, 0, 0, 1}},
{{1, 1, 1, 1}}, {{1, 2, 4, 8}}
}};
inline std::array<Bytes, ParityShards> encode(const std::array<Bytes, DataShards> &data)
{
const size_t bytes = data[0].size();
for (const auto &shard : data) if (shard.size() != bytes) return {};
std::array<Bytes, ParityShards> parity{Bytes(bytes), Bytes(bytes)};
for (size_t p = 0; p < ParityShards; ++p)
for (size_t d = 0; d < DataShards; ++d)
for (size_t i = 0; i < bytes; ++i)
parity[p][i] ^= multiply(Generator[DataShards + p][d], data[d][i]);
return parity;
}
inline bool recover(Shards &shards)
{
size_t shard_bytes = 0;
std::array<size_t, DataShards> available{};
size_t count = 0;
for (size_t i = 0; i < TotalShards; ++i) {
if (!shards[i]) continue;
if (shard_bytes == 0) shard_bytes = shards[i]->size();
if (shards[i]->size() != shard_bytes) return false;
if (count < DataShards) available[count++] = i;
}
if (count < DataShards || shard_bytes == 0) return false;
std::array<std::array<uint8_t, DataShards * 2>, DataShards> matrix{};
for (size_t row = 0; row < DataShards; ++row) {
for (size_t col = 0; col < DataShards; ++col) matrix[row][col] = Generator[available[row]][col];
matrix[row][DataShards + row] = 1;
}
for (size_t col = 0; col < DataShards; ++col) {
size_t pivot = col;
while (pivot < DataShards && matrix[pivot][col] == 0) ++pivot;
if (pivot == DataShards) return false;
if (pivot != col) std::swap(matrix[pivot], matrix[col]);
const uint8_t scale = inverse(matrix[col][col]);
for (size_t j = 0; j < DataShards * 2; ++j) matrix[col][j] = multiply(matrix[col][j], scale);
for (size_t row = 0; row < DataShards; ++row) {
if (row == col) continue;
const uint8_t factor = matrix[row][col];
for (size_t j = 0; j < DataShards * 2; ++j)
matrix[row][j] ^= multiply(factor, matrix[col][j]);
}
}
for (size_t missing = 0; missing < DataShards; ++missing) {
if (shards[missing]) continue;
Bytes decoded(shard_bytes);
for (size_t row = 0; row < DataShards; ++row) {
const uint8_t coefficient = matrix[missing][DataShards + row];
for (size_t i = 0; i < shard_bytes; ++i)
decoded[i] ^= multiply(coefficient, (*shards[available[row]])[i]);
}
shards[missing] = std::move(decoded);
}
return true;
}
struct BlockKey {
std::string stream_id;
std::string session_id;
uint64_t base_packet_id = 0;
bool operator<(const BlockKey &other) const
{
return std::tie(stream_id, session_id, base_packet_id) <
std::tie(other.stream_id, other.session_id, other.base_packet_id);
}
};
struct RecoveredPacket { uint64_t packet_id; Bytes payload; };
class BlockReceiver {
public:
using Clock = std::chrono::steady_clock;
explicit BlockReceiver(std::chrono::milliseconds deadline, size_t max_blocks = 128)
: deadline_(deadline), max_blocks_(max_blocks) {}
std::vector<RecoveredPacket> insert(const BlockKey &key, size_t shard_index, Bytes payload,
Clock::time_point now)
{
expire(now);
if (shard_index >= TotalShards || payload.empty() || completed_.count(key) != 0) return {};
if (blocks_.size() >= max_blocks_ && blocks_.count(key) == 0) blocks_.erase(blocks_.begin());
auto [it, created] = blocks_.try_emplace(key, Block{now, payload.size(), {}});
Block &block = it->second;
if (payload.size() != block.shard_bytes || block.shards[shard_index]) return {};
block.shards[shard_index] = std::move(payload);
size_t present = 0;
for (const auto &shard : block.shards) if (shard) ++present;
if (present < DataShards) return {};
Shards before = block.shards;
if (!recover(block.shards)) return {};
std::vector<RecoveredPacket> result;
for (size_t i = 0; i < DataShards; ++i)
if (!before[i] && block.shards[i]) result.push_back({key.base_packet_id + i, *block.shards[i]});
blocks_.erase(it);
completed_[key] = now;
return result;
}
void expire(Clock::time_point now)
{
for (auto it = blocks_.begin(); it != blocks_.end();)
it = now - it->second.created_at >= deadline_ ? blocks_.erase(it) : std::next(it);
for (auto it = completed_.begin(); it != completed_.end();)
it = now - it->second >= deadline_ ? completed_.erase(it) : std::next(it);
}
size_t pending_blocks() const { return blocks_.size(); }
void clear()
{
blocks_.clear();
completed_.clear();
}
private:
struct Block { Clock::time_point created_at; size_t shard_bytes; Shards shards; };
std::chrono::milliseconds deadline_;
size_t max_blocks_;
std::map<BlockKey, Block> blocks_;
std::map<BlockKey, Clock::time_point> completed_;
};
} // namespace muninn_audio_fec

View file

@ -0,0 +1,274 @@
#pragma once
#include <chrono>
#include <climits>
#include <cstdint>
#include <string>
#include <vector>
namespace muninn_media_wire {
inline constexpr const char *ReceiverFeedbackSchema = "muninn.media_receiver_feedback.v1";
class MsgpackWriter {
public:
const std::vector<uint8_t> &bytes() const { return bytes_; }
void write_raw(const std::vector<uint8_t> &value)
{
bytes_.insert(bytes_.end(), value.begin(), value.end());
}
void write_nil() { bytes_.push_back(0xc0); }
void write_bool(bool value) { bytes_.push_back(value ? 0xc3 : 0xc2); }
void write_u64(uint64_t value)
{
if (value <= 0x7f) {
bytes_.push_back(static_cast<uint8_t>(value));
} else if (value <= 0xff) {
bytes_.push_back(0xcc);
bytes_.push_back(static_cast<uint8_t>(value));
} else if (value <= 0xffff) {
bytes_.push_back(0xcd);
write_be16_value(static_cast<uint16_t>(value));
} else if (value <= 0xffffffffull) {
bytes_.push_back(0xce);
write_be32_value(static_cast<uint32_t>(value));
} else {
bytes_.push_back(0xcf);
write_be64_value(value);
}
}
void write_i64(int64_t value)
{
if (value >= 0) {
write_u64(static_cast<uint64_t>(value));
return;
}
if (value >= -32) {
bytes_.push_back(static_cast<uint8_t>(0xe0 | (value + 32)));
} else if (value >= INT8_MIN) {
bytes_.push_back(0xd0);
bytes_.push_back(static_cast<uint8_t>(value));
} else if (value >= INT16_MIN) {
bytes_.push_back(0xd1);
write_be16_value(static_cast<uint16_t>(value));
} else if (value >= INT32_MIN) {
bytes_.push_back(0xd2);
write_be32_value(static_cast<uint32_t>(value));
} else {
bytes_.push_back(0xd3);
write_be64_value(static_cast<uint64_t>(value));
}
}
void write_string(const std::string &value)
{
const auto length = value.size();
if (length <= 31) {
bytes_.push_back(static_cast<uint8_t>(0xa0 | length));
} else if (length <= 0xff) {
bytes_.push_back(0xd9);
bytes_.push_back(static_cast<uint8_t>(length));
} else if (length <= 0xffff) {
bytes_.push_back(0xda);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xdb);
write_be32_value(static_cast<uint32_t>(length));
}
bytes_.insert(bytes_.end(), value.begin(), value.end());
}
void write_binary(const std::vector<uint8_t> &value)
{
const auto length = value.size();
if (length <= 0xff) {
bytes_.push_back(0xc4);
bytes_.push_back(static_cast<uint8_t>(length));
} else if (length <= 0xffff) {
bytes_.push_back(0xc5);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xc6);
write_be32_value(static_cast<uint32_t>(length));
}
bytes_.insert(bytes_.end(), value.begin(), value.end());
}
void write_array_len(uint32_t length)
{
if (length <= 15) {
bytes_.push_back(static_cast<uint8_t>(0x90 | length));
} else if (length <= 0xffff) {
bytes_.push_back(0xdc);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xdd);
write_be32_value(length);
}
}
void write_map_len(uint32_t length)
{
if (length <= 15) {
bytes_.push_back(static_cast<uint8_t>(0x80 | length));
} else if (length <= 0xffff) {
bytes_.push_back(0xde);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xdf);
write_be32_value(length);
}
}
private:
std::vector<uint8_t> bytes_;
void write_be16_value(uint16_t value)
{
bytes_.push_back(static_cast<uint8_t>((value >> 8) & 0xff));
bytes_.push_back(static_cast<uint8_t>(value & 0xff));
}
void write_be32_value(uint32_t value)
{
bytes_.push_back(static_cast<uint8_t>((value >> 24) & 0xff));
bytes_.push_back(static_cast<uint8_t>((value >> 16) & 0xff));
bytes_.push_back(static_cast<uint8_t>((value >> 8) & 0xff));
bytes_.push_back(static_cast<uint8_t>(value & 0xff));
}
void write_be64_value(uint64_t value)
{
write_be32_value(static_cast<uint32_t>((value >> 32) & 0xffffffffull));
write_be32_value(static_cast<uint32_t>(value & 0xffffffffull));
}
};
inline std::string feedback_observed_at()
{
const auto now = std::chrono::system_clock::now().time_since_epoch();
const auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(now).count();
return "unix-ms:" + std::to_string(millis);
}
inline void write_string_array(MsgpackWriter &writer, const std::vector<std::string> &values)
{
writer.write_array_len(static_cast<uint32_t>(values.size()));
for (const auto &value : values) {
writer.write_string(value);
}
}
inline void write_u64_array(MsgpackWriter &writer, const std::vector<uint64_t> &values)
{
writer.write_array_len(static_cast<uint32_t>(values.size()));
for (const auto value : values) {
writer.write_u64(value);
}
}
inline std::vector<uint64_t> receiver_feedback_late_frame_ids(uint64_t frame_id, bool late_frame)
{
return late_frame ? std::vector<uint64_t>{frame_id} : std::vector<uint64_t>{};
}
inline std::vector<uint8_t> encode_receiver_feedback_payload(
const std::string &stream_id,
const std::string &session_id,
uint64_t frame_id,
const std::vector<std::string> &missing_chunk_keys,
bool requested_keyframe,
bool late_frame,
const std::string &observed_at)
{
MsgpackWriter record;
record.write_array_len(11);
record.write_string(stream_id);
record.write_string(session_id);
record.write_string("starfire.obs");
if (frame_id > 0) {
record.write_u64(frame_id - 1);
} else {
record.write_nil();
}
write_u64_array(record, {});
write_u64_array(record, receiver_feedback_late_frame_ids(frame_id, late_frame));
record.write_bool(requested_keyframe);
record.write_i64(0);
record.write_i64(0);
record.write_string(observed_at);
write_string_array(record, missing_chunk_keys);
const std::string record_key = stream_id + ":" + session_id + ":feedback:starfire.obs";
const std::string message_id =
"muninn-media:" + std::string(ReceiverFeedbackSchema) + ":" + record_key;
MsgpackWriter document;
document.write_map_len(9);
document.write_string("schemaId");
document.write_string(ReceiverFeedbackSchema);
document.write_string("recordKey");
document.write_string(record_key);
document.write_string("storedAt");
document.write_string(observed_at);
document.write_string("payloadEncoding");
document.write_string("messagepack");
document.write_string("payload");
document.write_binary(record.bytes());
document.write_string("sourceRuntimeId");
document.write_string("starfire");
document.write_string("sourceAgentId");
document.write_nil();
document.write_string("sourceRole");
document.write_string("mimir.obs");
document.write_string("tags");
write_string_array(document, {"muninn.media"});
MsgpackWriter wire;
wire.write_map_len(3);
wire.write_string("schemaVersion");
wire.write_string("cultnet.document_put_raw.v0");
wire.write_string("messageId");
wire.write_string(message_id);
wire.write_string("document");
wire.write_raw(document.bytes());
return wire.bytes();
}
inline std::vector<uint8_t> encode_receiver_feedback_payload(
const std::string &stream_id,
const std::string &session_id,
uint64_t frame_id,
const std::vector<std::string> &missing_chunk_keys,
bool requested_keyframe)
{
return encode_receiver_feedback_payload(
stream_id,
session_id,
frame_id,
missing_chunk_keys,
requested_keyframe,
requested_keyframe,
feedback_observed_at());
}
inline std::vector<uint8_t> encode_receiver_feedback_payload(
const std::string &stream_id,
const std::string &session_id,
uint64_t frame_id,
const std::vector<std::string> &missing_chunk_keys)
{
return encode_receiver_feedback_payload(
stream_id,
session_id,
frame_id,
missing_chunk_keys,
true);
}
} // namespace muninn_media_wire

View file

@ -0,0 +1,54 @@
#pragma once
#include <cstdint>
#include <optional>
#include <set>
#include <utility>
namespace muninn_rudp_ack {
class AckTracker {
public:
void reset()
{
received_sequences_.clear();
highest_received_sequence_.reset();
}
void remember(uint32_t sequence)
{
received_sequences_.insert(sequence);
if (!highest_received_sequence_ || sequence > *highest_received_sequence_) {
highest_received_sequence_ = sequence;
}
if (!highest_received_sequence_) {
return;
}
const uint32_t highest = *highest_received_sequence_;
for (auto iterator = received_sequences_.begin(); iterator != received_sequences_.end();) {
if (*iterator + 32 < highest) {
iterator = received_sequences_.erase(iterator);
} else {
++iterator;
}
}
}
std::pair<uint32_t, uint32_t> state() const
{
const uint32_t ack = highest_received_sequence_.value_or(0);
uint32_t ack_mask = 0;
for (uint32_t bit = 0; bit < 32; ++bit) {
if (ack > bit && received_sequences_.find(ack - bit - 1) != received_sequences_.end()) {
ack_mask |= 1u << bit;
}
}
return {ack, ack_mask};
}
private:
std::set<uint32_t> received_sequences_;
std::optional<uint32_t> highest_received_sequence_;
};
} // namespace muninn_rudp_ack

View file

@ -0,0 +1,127 @@
#pragma once
#include <algorithm>
#include <chrono>
#include <cstdint>
#include <map>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace muninn_rudp_fragments {
constexpr uint32_t CompletedFragmentIdQuarantineMs = 2000;
struct FragmentResult {
uint32_t first_sequence = 0;
std::vector<uint8_t> payload;
};
class FragmentAssembler {
public:
void reset()
{
assemblies_.clear();
recently_closed_.clear();
expired_count_ = 0;
}
std::optional<FragmentResult> insert(const std::string &channel_id,
uint16_t fragment_id,
uint16_t fragment_index,
uint16_t fragment_count,
uint32_t sequence,
const uint8_t *payload,
uint32_t payload_len,
std::chrono::steady_clock::time_point now)
{
if (fragment_id == 0 || fragment_index >= fragment_count) {
return std::nullopt;
}
const std::string key = channel_id + ":" + std::to_string(fragment_id);
expire_recently_closed(now);
const auto existing = assemblies_.find(key);
if (existing == assemblies_.end() && recently_closed_.find(key) != recently_closed_.end()) {
return std::nullopt;
}
auto &assembly = assemblies_[key];
if (assembly.payloads.empty()) {
assembly.channel_id = channel_id;
assembly.fragment_count = fragment_count;
assembly.started_at = now;
}
if (assembly.channel_id != channel_id || assembly.fragment_count != fragment_count) {
assemblies_.erase(key);
return std::nullopt;
}
assembly.payloads.emplace(fragment_index, std::vector<uint8_t>(payload, payload + payload_len));
assembly.sequences.emplace(fragment_index, sequence);
if (assembly.payloads.size() < assembly.fragment_count) {
return std::nullopt;
}
FragmentResult result;
result.first_sequence = UINT32_MAX;
for (uint16_t index = 0; index < assembly.fragment_count; ++index) {
const auto payload_found = assembly.payloads.find(index);
const auto sequence_found = assembly.sequences.find(index);
if (payload_found == assembly.payloads.end() || sequence_found == assembly.sequences.end()) {
return std::nullopt;
}
result.payload.insert(result.payload.end(), payload_found->second.begin(), payload_found->second.end());
result.first_sequence = std::min(result.first_sequence, sequence_found->second);
}
assemblies_.erase(key);
recently_closed_[key] = now;
return result;
}
uint64_t expire(std::chrono::steady_clock::time_point now, uint32_t expire_after_ms)
{
expire_recently_closed(now);
uint64_t expired = 0;
for (auto iterator = assemblies_.begin(); iterator != assemblies_.end();) {
if (now - iterator->second.started_at < std::chrono::milliseconds(expire_after_ms)) {
++iterator;
continue;
}
recently_closed_[iterator->first] = now;
iterator = assemblies_.erase(iterator);
++expired;
}
expired_count_ += expired;
return expired;
}
uint64_t expired_count() const { return expired_count_; }
private:
void expire_recently_closed(std::chrono::steady_clock::time_point now)
{
for (auto iterator = recently_closed_.begin(); iterator != recently_closed_.end();) {
if (now - iterator->second < std::chrono::milliseconds(CompletedFragmentIdQuarantineMs)) {
++iterator;
continue;
}
iterator = recently_closed_.erase(iterator);
}
}
struct Assembly {
std::string channel_id;
uint16_t fragment_count = 0;
std::map<uint16_t, std::vector<uint8_t>> payloads;
std::map<uint16_t, uint32_t> sequences;
std::chrono::steady_clock::time_point started_at;
};
std::map<std::string, Assembly> assemblies_;
std::map<std::string, std::chrono::steady_clock::time_point> recently_closed_;
uint64_t expired_count_ = 0;
};
} // namespace muninn_rudp_fragments

View file

@ -0,0 +1,126 @@
#pragma once
#include <algorithm>
#include <cstdint>
#include <optional>
#include <sstream>
#include <string>
namespace muninn_rudp_url {
inline constexpr uint32_t DefaultMediaRudpConnectionId = 0x6d750001;
inline constexpr uint32_t DefaultAudioRudpConnectionId = 0x6d750004;
inline constexpr uint32_t DefaultReliableExpireAfterMs = 75;
inline constexpr uint32_t DefaultVideoAssemblyDeadlineMs = 75;
inline constexpr uint32_t DefaultGapWaitMs = 8;
inline constexpr uint32_t DefaultAudioReorderMs = 120;
inline constexpr uint32_t DefaultSenderResendDelayMs = 5;
inline constexpr uint32_t MaxLatencyBudgetMs = 500;
struct RudpUrlParts {
std::string channel_id = "media";
uint16_t port = 0;
uint16_t video_local_port = 0;
uint16_t audio_local_port = 0;
uint32_t connection_id = DefaultMediaRudpConnectionId;
uint32_t audio_connection_id = DefaultAudioRudpConnectionId;
uint32_t sender_resend_delay_ms = DefaultSenderResendDelayMs;
uint32_t reliable_expire_after_ms = DefaultReliableExpireAfterMs;
uint32_t video_assembly_deadline_ms = DefaultVideoAssemblyDeadlineMs;
uint32_t gap_wait_ms = DefaultGapWaitMs;
uint32_t audio_reorder_ms = DefaultAudioReorderMs;
};
inline bool starts_with(const std::string &value, const std::string &prefix)
{
return value.rfind(prefix, 0) == 0;
}
inline std::optional<uint32_t> parse_u32_hex_or_decimal(const std::string &value)
{
try {
size_t consumed = 0;
const int base = starts_with(value, "0x") || starts_with(value, "0X") ? 16 : 10;
const auto parsed = std::stoul(value, &consumed, base);
if (consumed == value.size()) {
return static_cast<uint32_t>(parsed);
}
} catch (...) {
}
return std::nullopt;
}
inline std::optional<RudpUrlParts> parse(const std::string &url)
{
if (!starts_with(url, "rudp://")) {
return std::nullopt;
}
const auto authority_start = std::string("rudp://").size();
const auto path_start = url.find('/', authority_start);
const auto query_start = url.find('?', authority_start);
const auto authority_end = std::min(path_start == std::string::npos ? url.size() : path_start,
query_start == std::string::npos ? url.size() : query_start);
const auto authority = url.substr(authority_start, authority_end - authority_start);
const auto colon = authority.rfind(':');
if (colon == std::string::npos || colon + 1 >= authority.size()) {
return std::nullopt;
}
RudpUrlParts parts;
parts.port = static_cast<uint16_t>(std::stoul(authority.substr(colon + 1)));
parts.video_local_port = static_cast<uint16_t>(parts.port + 10000);
parts.audio_local_port = static_cast<uint16_t>(parts.port + 10001);
if (query_start != std::string::npos) {
std::stringstream query(url.substr(query_start + 1));
std::string item;
while (std::getline(query, item, '&')) {
const auto equals = item.find('=');
if (equals == std::string::npos) {
continue;
}
const auto key = item.substr(0, equals);
const auto value = item.substr(equals + 1);
if (key == "channel" && !value.empty()) {
parts.channel_id = value;
} else if (key == "connection") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.connection_id = *parsed;
}
} else if (key == "audio_connection" || key == "audio_connection_id") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.audio_connection_id = *parsed;
}
} else if (key == "local_port" || key == "video_local_port") {
parts.video_local_port = static_cast<uint16_t>(std::stoul(value));
} else if (key == "audio_local_port") {
parts.audio_local_port = static_cast<uint16_t>(std::stoul(value));
} else if (key == "reliable_expire_after_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.reliable_expire_after_ms = std::clamp(*parsed, 1u, MaxLatencyBudgetMs);
}
} else if (key == "sender_resend_delay_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.sender_resend_delay_ms = std::clamp(*parsed, 1u, 1000u);
}
} else if (key == "assembly_deadline_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.video_assembly_deadline_ms = std::clamp(*parsed, 1u, MaxLatencyBudgetMs);
}
} else if (key == "gap_wait_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.gap_wait_ms = std::clamp(*parsed, 0u, 1000u);
}
} else if (key == "audio_reorder_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.audio_reorder_ms = std::clamp(*parsed, 0u, MaxLatencyBudgetMs);
}
}
}
}
return parts;
}
} // namespace muninn_rudp_url

View file

@ -0,0 +1,32 @@
#pragma once
#ifdef _WIN32
#include <winsock2.h>
namespace muninn_udp_socket {
inline SOCKET create(bool exclusive_listener = false)
{
const SOCKET socket = WSASocketW(
AF_INET,
SOCK_DGRAM,
IPPROTO_UDP,
nullptr,
0,
WSA_FLAG_NO_HANDLE_INHERIT);
if (socket == INVALID_SOCKET) {
return INVALID_SOCKET;
}
if (exclusive_listener) {
const BOOL exclusive = TRUE;
if (setsockopt(socket, SOL_SOCKET, SO_EXCLUSIVEADDRUSE,
reinterpret_cast<const char *>(&exclusive), sizeof(exclusive)) != 0) {
closesocket(socket);
return INVALID_SOCKET;
}
}
return socket;
}
} // namespace muninn_udp_socket
#endif

View file

@ -0,0 +1,181 @@
#pragma once
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <map>
#include <utility>
#include <vector>
namespace muninn_video_fec {
inline uint8_t raw_multiply(uint8_t a, uint8_t b)
{
uint8_t product = 0;
while (b != 0) {
if ((b & 1) != 0)
product ^= a;
const bool carry = (a & 0x80) != 0;
a <<= 1;
if (carry)
a ^= 0x1d;
b >>= 1;
}
return product;
}
struct Tables {
uint8_t products[256][256] = {};
uint8_t inverses[256] = {};
Tables()
{
for (size_t a = 0; a < 256; ++a) {
for (size_t b = 0; b < 256; ++b)
products[a][b] = raw_multiply(static_cast<uint8_t>(a), static_cast<uint8_t>(b));
}
for (size_t value = 1; value < 256; ++value) {
uint8_t result = 1;
for (size_t index = 0; index < 254; ++index)
result = raw_multiply(result, static_cast<uint8_t>(value));
inverses[value] = result;
}
}
};
inline const Tables &tables()
{
static const Tables value;
return value;
}
inline uint8_t multiply(uint8_t a, uint8_t b)
{
return tables().products[a][b];
}
inline uint8_t inverse(uint8_t value)
{
return tables().inverses[value];
}
inline uint8_t coefficient(uint16_t parity_index, uint16_t parity_count, uint16_t data_index)
{
return inverse(
static_cast<uint8_t>(parity_index) ^ static_cast<uint8_t>(parity_count + data_index));
}
inline bool recover(std::vector<std::vector<uint8_t>> &chunks,
const std::vector<uint32_t> &chunk_lengths,
uint16_t parity_count,
const std::map<uint16_t, std::vector<uint8_t>> &parity)
{
if (chunks.size() != chunk_lengths.size() || chunks.empty() ||
chunks.size() + parity_count > 256)
return false;
std::vector<uint16_t> missing;
for (uint16_t index = 0; index < chunks.size(); ++index) {
if (chunks[index].empty())
missing.push_back(index);
}
if (missing.empty())
return true;
if (missing.size() > parity.size())
return false;
std::vector<std::pair<uint16_t, const std::vector<uint8_t> *>> rows;
for (const auto &shard : parity) {
if (shard.first < parity_count && !shard.second.empty())
rows.emplace_back(shard.first, &shard.second);
if (rows.size() == missing.size())
break;
}
if (rows.size() != missing.size())
return false;
size_t shard_bytes = 0;
for (uint32_t length : chunk_lengths)
shard_bytes = std::max(shard_bytes, static_cast<size_t>(length));
if (shard_bytes == 0 || std::any_of(rows.begin(), rows.end(),
[shard_bytes](const auto &row) { return row.second->size() < shard_bytes; }))
return false;
std::vector<std::vector<uint8_t>> recovered;
recovered.reserve(missing.size());
for (uint16_t index : missing) {
if (chunk_lengths[index] == 0 || chunk_lengths[index] > shard_bytes)
return false;
recovered.emplace_back(chunk_lengths[index], 0);
}
const size_t count = missing.size();
std::vector<std::vector<uint8_t>> inverse_matrix(count, std::vector<uint8_t>(count * 2, 0));
for (size_t row = 0; row < count; ++row) {
for (size_t column = 0; column < count; ++column)
inverse_matrix[row][column] = coefficient(rows[row].first, parity_count, missing[column]);
inverse_matrix[row][count + row] = 1;
}
for (size_t column = 0; column < count; ++column) {
size_t pivot = column;
while (pivot < count && inverse_matrix[pivot][column] == 0)
++pivot;
if (pivot == count)
return false;
std::swap(inverse_matrix[column], inverse_matrix[pivot]);
const uint8_t scale = inverse(inverse_matrix[column][column]);
for (size_t cell = 0; cell < count * 2; ++cell)
inverse_matrix[column][cell] = multiply(inverse_matrix[column][cell], scale);
for (size_t row = 0; row < count; ++row) {
if (row == column)
continue;
const uint8_t factor = inverse_matrix[row][column];
for (size_t cell = 0; cell < count * 2; ++cell)
inverse_matrix[row][cell] ^= multiply(factor, inverse_matrix[column][cell]);
}
}
std::vector<uint8_t> rhs(count, 0);
for (size_t offset = 0; offset < shard_bytes; ++offset) {
for (size_t row = 0; row < count; ++row) {
const uint16_t parity_index = rows[row].first;
rhs[row] = (*rows[row].second)[offset];
for (uint16_t data_index = 0; data_index < chunks.size(); ++data_index) {
if (!chunks[data_index].empty() && offset < chunks[data_index].size())
rhs[row] ^= multiply(coefficient(parity_index, parity_count, data_index),
chunks[data_index][offset]);
}
}
for (size_t index = 0; index < count; ++index) {
uint8_t value = 0;
for (size_t row = 0; row < count; ++row)
value ^= multiply(inverse_matrix[index][count + row], rhs[row]);
if (offset < recovered[index].size())
recovered[index][offset] = value;
}
}
for (size_t index = 0; index < count; ++index)
chunks[missing[index]] = std::move(recovered[index]);
return true;
}
inline bool recover_block(std::vector<std::vector<uint8_t>> &frame_chunks,
uint16_t data_start,
const std::vector<uint32_t> &block_lengths,
uint16_t parity_count,
const std::map<uint16_t, std::vector<uint8_t>> &parity)
{
if (block_lengths.empty() ||
static_cast<size_t>(data_start) + block_lengths.size() > frame_chunks.size())
return false;
std::vector<std::vector<uint8_t>> block_chunks;
block_chunks.reserve(block_lengths.size());
for (size_t index = 0; index < block_lengths.size(); ++index)
block_chunks.push_back(frame_chunks[data_start + index]);
if (!recover(block_chunks, block_lengths, parity_count, parity))
return false;
for (size_t index = 0; index < block_lengths.size(); ++index)
frame_chunks[data_start + index] = std::move(block_chunks[index]);
return true;
}
} // namespace muninn_video_fec

View file

@ -0,0 +1,79 @@
#include "muninn_audio_fec.h"
#include <cstdlib>
#include <iostream>
#include <stdexcept>
static void require(bool condition, const char *message) { if (!condition) throw std::runtime_error(message); }
static std::array<muninn_audio_fec::Bytes, 4> fixture()
{
std::array<muninn_audio_fec::Bytes, 4> data;
for (size_t shard = 0; shard < data.size(); ++shard) {
data[shard].resize(257);
for (size_t i = 0; i < data[shard].size(); ++i) data[shard][i] = uint8_t((shard * 71 + i * 29) & 0xff);
}
return data;
}
static muninn_audio_fec::Shards all_shards(const std::array<muninn_audio_fec::Bytes, 4> &data)
{
auto parity = muninn_audio_fec::encode(data);
return {data[0], data[1], data[2], data[3], parity[0], parity[1]};
}
int main()
{
try {
const auto data = fixture();
for (size_t erased = 0; erased < 6; ++erased) {
auto shards = all_shards(data); shards[erased].reset();
require(muninn_audio_fec::recover(shards), "one-erasure recovery failed");
for (size_t i = 0; i < 4; ++i) require(shards[i] == data[i], "one-erasure data mismatch");
}
for (size_t first = 0; first < 6; ++first) for (size_t second = first + 1; second < 6; ++second) {
auto shards = all_shards(data); shards[first].reset(); shards[second].reset();
require(muninn_audio_fec::recover(shards), "two-erasure recovery failed");
for (size_t i = 0; i < 4; ++i) require(shards[i] == data[i], "two-erasure data mismatch");
}
{
auto shards = all_shards(data);
shards[0].reset(); shards[1].reset(); shards[2].reset();
require(!muninn_audio_fec::recover(shards), "three erasures were incorrectly recoverable");
}
{
auto shards = all_shards(data);
shards[5]->pop_back();
require(!muninn_audio_fec::recover(shards), "unequal shard sizes were accepted");
}
using namespace muninn_audio_fec;
BlockReceiver receiver(std::chrono::milliseconds(10), 2);
BlockKey key{"stream", "session", 100};
auto shards = all_shards(data);
auto t0 = BlockReceiver::Clock::time_point{};
require(receiver.insert(key, 5, *shards[5], t0).empty(), "early parity emitted data");
require(receiver.insert(key, 2, *shards[2], t0).empty(), "partial reorder emitted data");
require(receiver.insert(key, 2, *shards[2], t0).empty(), "duplicate emitted data");
require(receiver.insert(key, 1, *shards[1], t0).empty(), "partial block emitted data");
auto recovered = receiver.insert(key, 4, *shards[4], t0);
require(recovered.size() == 2 && recovered[0].packet_id == 100 && recovered[1].packet_id == 103,
"reordered block did not recover missing packet ids");
require(recovered[0].payload == data[0] && recovered[1].payload == data[3], "receiver recovery mismatch");
require(receiver.insert(key, 0, data[0], t0).empty(), "completed-block duplicate was admitted");
BlockKey expired{"stream", "session", 200};
receiver.insert(expired, 0, data[0], t0);
receiver.expire(t0 + std::chrono::milliseconds(10));
require(receiver.pending_blocks() == 0, "expired block survived deadline");
BlockReceiver bounded(std::chrono::seconds(1), 2);
bounded.insert({"s", "x", 0}, 0, data[0], t0);
bounded.insert({"s", "x", 4}, 0, data[0], t0);
bounded.insert({"s", "x", 8}, 0, data[0], t0);
require(bounded.pending_blocks() == 2, "block bound was not enforced");
} catch (const std::exception &error) {
std::cerr << error.what() << '\n'; return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}

View file

@ -0,0 +1,54 @@
#include "muninn_media_wire.h"
#include <cstdlib>
#include <iomanip>
#include <iostream>
namespace {
void require(bool condition, const char *message)
{
if (!condition) {
std::cerr << message << '\n';
std::exit(1);
}
}
} // namespace
int main(int argc, char **argv)
{
const auto early = muninn_media_wire::receiver_feedback_late_frame_ids(42, false);
require(early.empty(), "early repair feedback marked a live frame late");
const auto expired = muninn_media_wire::receiver_feedback_late_frame_ids(42, true);
require(expired.size() == 1 && expired.front() == 42,
"expired feedback did not name the late frame");
const auto early_wire = muninn_media_wire::encode_receiver_feedback_payload(
"muninn.raven.av.rudp",
"raven:session:video",
42,
{"42:3"},
false,
false,
"unix-ms:1000");
const auto expired_wire = muninn_media_wire::encode_receiver_feedback_payload(
"muninn.raven.av.rudp",
"raven:session:video",
42,
{"42:3"},
true,
true,
"unix-ms:1000");
require(!early_wire.empty() && !expired_wire.empty(), "feedback wire encoding was empty");
require(early_wire != expired_wire, "early and expired feedback encoded identically");
if (argc == 2 && std::string(argv[1]) == "--emit-early-hex") {
for (const auto byte : early_wire) {
std::cout << std::hex << std::setw(2) << std::setfill('0')
<< static_cast<unsigned int>(byte);
}
std::cout << '\n';
}
return 0;
}

View file

@ -0,0 +1,60 @@
#include "muninn_rudp_url.h"
#include <cstdlib>
#include <iostream>
#include <string>
namespace {
void require(bool condition, const std::string &message)
{
if (!condition) {
std::cerr << message << '\n';
std::exit(1);
}
}
} // namespace
int main()
{
const auto configured = muninn_rudp_url::parse(
"rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=realtime&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=120&assembly_deadline_ms=120&gap_wait_ms=16&audio_reorder_ms=80");
require(configured.has_value(), "configured RUDP URL did not parse");
require(configured->port == 5204, "configured RUDP URL port was not parsed");
require(configured->channel_id == "realtime", "configured RUDP realtime channel was not parsed");
require(configured->video_local_port == 15204, "configured RUDP video local port was not derived");
require(configured->audio_local_port == 15205, "configured RUDP audio local port was not derived");
require(configured->connection_id == 0x6d750001, "configured RUDP connection id was not parsed");
require(configured->audio_connection_id == 0x6d750004, "configured RUDP audio connection id was not parsed");
require(configured->reliable_expire_after_ms == 120, "configured reliable expiry did not survive parse");
require(configured->video_assembly_deadline_ms == 120, "configured assembly deadline did not survive parse");
require(configured->gap_wait_ms == 16, "configured gap wait did not survive parse");
require(configured->audio_reorder_ms == 80, "configured audio reorder wait did not survive parse");
const auto clamped = muninn_rudp_url::parse(
"rudp://192.168.1.66:5204/muninn.raven.av.rudp?reliable_expire_after_ms=9000&assembly_deadline_ms=9000");
require(clamped.has_value(), "oversized latency RUDP URL did not parse");
require(clamped->reliable_expire_after_ms == muninn_rudp_url::MaxLatencyBudgetMs,
"oversized reliable expiry did not clamp to the max latency budget");
require(clamped->video_assembly_deadline_ms == muninn_rudp_url::MaxLatencyBudgetMs,
"oversized assembly deadline did not clamp to the max latency budget");
const auto fallback = muninn_rudp_url::parse("rudp://192.168.1.66:5204/muninn.raven.av.rudp");
require(fallback.has_value(), "minimal RUDP URL did not parse");
require(fallback->channel_id == "media", "minimal RUDP URL did not use bounded reliable media delivery");
require(fallback->connection_id == muninn_rudp_url::DefaultMediaRudpConnectionId,
"minimal RUDP URL did not keep the default connection id");
require(fallback->audio_connection_id == muninn_rudp_url::DefaultAudioRudpConnectionId,
"minimal RUDP URL did not keep the default audio connection id");
require(fallback->reliable_expire_after_ms == muninn_rudp_url::DefaultReliableExpireAfterMs,
"minimal RUDP URL did not keep the default reliable expiry");
require(fallback->video_assembly_deadline_ms == muninn_rudp_url::DefaultVideoAssemblyDeadlineMs,
"minimal RUDP URL did not keep the default assembly deadline");
require(fallback->audio_reorder_ms == muninn_rudp_url::DefaultAudioReorderMs,
"minimal RUDP URL did not keep the default audio reorder wait");
require(fallback->audio_reorder_ms == 120,
"minimal RUDP URL did not use the bounded realtime audio reorder budget");
return 0;
}

View file

@ -0,0 +1,34 @@
#include "muninn_udp_socket.h"
#include <cassert>
#include <windows.h>
int main()
{
WSADATA data{};
assert(WSAStartup(MAKEWORD(2, 2), &data) == 0);
const SOCKET owner = muninn_udp_socket::create(true);
assert(owner != INVALID_SOCKET);
DWORD flags = HANDLE_FLAG_INHERIT;
assert(GetHandleInformation(reinterpret_cast<HANDLE>(owner), &flags));
assert((flags & HANDLE_FLAG_INHERIT) == 0);
sockaddr_in address{};
address.sin_family = AF_INET;
address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
address.sin_port = 0;
assert(bind(owner, reinterpret_cast<sockaddr *>(&address), sizeof(address)) == 0);
int address_size = sizeof(address);
assert(getsockname(owner, reinterpret_cast<sockaddr *>(&address), &address_size) == 0);
const SOCKET contender = muninn_udp_socket::create(true);
assert(contender != INVALID_SOCKET);
assert(bind(contender, reinterpret_cast<sockaddr *>(&address), sizeof(address)) != 0);
assert(WSAGetLastError() == WSAEADDRINUSE || WSAGetLastError() == WSAEACCES);
closesocket(contender);
closesocket(owner);
WSACleanup();
return 0;
}

View file

@ -0,0 +1,111 @@
#include "muninn_video_fec.h"
#include <chrono>
#include <cstdlib>
#include <cstdint>
#include <map>
#include <vector>
void require(bool condition)
{
if (!condition)
std::abort();
}
int main()
{
constexpr uint16_t data_count = 64;
constexpr uint16_t parity_count = 16;
constexpr size_t shard_bytes = 1400;
std::vector<std::vector<uint8_t>> original(data_count, std::vector<uint8_t>(shard_bytes));
for (uint16_t data = 0; data < data_count; ++data) {
for (size_t offset = 0; offset < shard_bytes; ++offset)
original[data][offset] = static_cast<uint8_t>((data * 29 + offset * 17) & 0xff);
}
std::map<uint16_t, std::vector<uint8_t>> parity;
for (uint16_t row = 0; row < parity_count; ++row) {
auto &shard = parity[row];
shard.assign(shard_bytes, 0);
for (uint16_t data = 0; data < data_count; ++data) {
const uint8_t coefficient = muninn_video_fec::coefficient(row, parity_count, data);
for (size_t offset = 0; offset < shard_bytes; ++offset)
shard[offset] ^= muninn_video_fec::multiply(original[data][offset], coefficient);
}
}
const std::vector<uint32_t> lengths(data_count, shard_bytes);
for (uint16_t burst = 1; burst <= parity_count; ++burst) {
auto damaged = original;
for (uint16_t index = 0; index < burst; ++index)
damaged[20 + index].clear();
const auto started = std::chrono::steady_clock::now();
require(muninn_video_fec::recover(damaged, lengths, parity_count, parity));
require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50));
require(damaged == original);
}
auto excessive = original;
for (uint16_t index = 0; index <= parity_count; ++index)
excessive[index].clear();
require(!muninn_video_fec::recover(excessive, lengths, parity_count, parity));
constexpr uint16_t block_data = 8;
constexpr uint16_t frame_data = 80;
constexpr uint16_t block_count = frame_data / block_data;
constexpr size_t block_shard_bytes = 96;
std::vector<std::vector<uint8_t>> frame(frame_data, std::vector<uint8_t>(block_shard_bytes));
for (uint16_t data = 0; data < frame_data; ++data) {
for (size_t offset = 0; offset < block_shard_bytes; ++offset)
frame[data][offset] = static_cast<uint8_t>((data * 31 + offset * 13) & 0xff);
}
std::vector<std::map<uint16_t, std::vector<uint8_t>>> block_parity(block_count);
for (uint16_t block = 0; block < block_count; ++block) {
for (uint16_t row = 0; row < block_data; ++row) {
auto &shard = block_parity[block][row];
shard.assign(block_shard_bytes, 0);
for (uint16_t local = 0; local < block_data; ++local) {
const auto &data = frame[block * block_data + local];
const uint8_t coefficient = muninn_video_fec::coefficient(row, block_data, local);
for (size_t offset = 0; offset < block_shard_bytes; ++offset)
shard[offset] ^= muninn_video_fec::multiply(data[offset], coefficient);
}
}
}
struct WireShard {
bool parity;
uint16_t block;
uint16_t local;
};
std::vector<WireShard> wire;
for (uint16_t local = 0; local < block_data; ++local) {
for (uint16_t block = 0; block < block_count; ++block)
wire.push_back({false, block, local});
for (uint16_t block = 0; block < block_count; ++block)
wire.push_back({true, block, local});
}
require(wire.size() == frame_data * 2);
const std::vector<uint32_t> block_lengths(block_data, block_shard_bytes);
for (size_t burst = 1; burst <= 54; ++burst) {
for (size_t start = 0; start + burst <= wire.size(); ++start) {
auto damaged = frame;
auto available_parity = block_parity;
for (size_t index = start; index < start + burst; ++index) {
const auto shard = wire[index];
if (shard.parity)
available_parity[shard.block].erase(shard.local);
else
damaged[shard.block * block_data + shard.local].clear();
}
const auto started = std::chrono::steady_clock::now();
for (uint16_t block = 0; block < block_count; ++block) {
require(muninn_video_fec::recover_block(
damaged,
block * block_data,
block_lengths,
block_data,
available_parity[block]));
}
require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50));
require(damaged == frame);
}
}
return 0;
}

View file

@ -17,7 +17,8 @@ enum LocalcastSampleKind {
LOCALCAST_SAMPLE_AUDIO_BLOCK = 5,
LOCALCAST_SAMPLE_PHASE_CLAIM = 6,
LOCALCAST_SAMPLE_EVENT_CLAIM = 7,
LOCALCAST_SAMPLE_RENDER_PACKET = 8
LOCALCAST_SAMPLE_RENDER_PACKET = 8,
LOCALCAST_SAMPLE_MOVE_EVIDENCE = 9
};
typedef struct LocalcastReservoir LocalcastReservoir;
@ -79,6 +80,47 @@ typedef struct LocalcastRenderPoint {
float confidence;
} LocalcastRenderPoint;
enum LocalcastMoveEvidenceKind {
LOCALCAST_MOVE_EVIDENCE_OPTICAL_MARKER = 1,
LOCALCAST_MOVE_EVIDENCE_CONTROLLER_STATE = 2
};
typedef struct LocalcastMoveEvidenceSample {
uint64_t witness_id_hash;
uint64_t controller_id_hash;
uint64_t source_timestamp_ns;
uint64_t arrival_ns;
uint64_t sequence;
uint32_t evidence_kind;
uint32_t flags;
float image_x;
float image_y;
float radius_px;
float confidence;
float accel_x;
float accel_y;
float accel_z;
float gyro_x;
float gyro_y;
float gyro_z;
float trigger;
uint32_t buttons_mask;
uint32_t reserved;
float battery01;
uint32_t reserved1;
uint32_t reserved2;
} LocalcastMoveEvidenceSample;
typedef struct LocalcastMoveEvidenceBufferDescriptor {
uint64_t sample_buffer_handle;
uint32_t sample_count;
uint32_t sample_stride_bytes;
uint64_t source_time_min_ns;
uint64_t source_time_max_ns;
uint64_t calibration_hash;
uint64_t tracking_space_hash;
} LocalcastMoveEvidenceBufferDescriptor;
typedef struct LocalcastRuntimeStatus {
uint64_t edge_ns;
uint64_t window_start_ns;
@ -92,6 +134,7 @@ typedef struct LocalcastRuntimeStatus {
size_t phase_claim_count;
size_t event_claim_count;
size_t render_packet_count;
size_t move_evidence_count;
} LocalcastRuntimeStatus;
uint64_t localcast_hash_source_id(const uint8_t *bytes, size_t byte_len);
@ -144,6 +187,7 @@ bool localcast_runtime_push_audio_block(LocalcastRuntime *runtime, LocalcastSamp
bool localcast_runtime_push_phase_claim(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_push_event_claim(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_push_render_packet(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_push_move_evidence(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_status(const LocalcastRuntime *runtime, LocalcastRuntimeStatus *out_status);
size_t localcast_runtime_len(const LocalcastRuntime *runtime);
size_t localcast_runtime_view_len(const LocalcastRuntime *runtime, uint32_t sample_kind);
@ -210,6 +254,15 @@ bool localcast_producer_push_render_packet(
LocalcastSampleHandle *out_sample
);
bool localcast_producer_push_move_evidence_buffer(
LocalcastProducer *producer,
LocalcastRuntime *runtime,
uint64_t timestamp_ns,
uint64_t arrival_ns,
const LocalcastMoveEvidenceBufferDescriptor *descriptor,
LocalcastSampleHandle *out_sample
);
#ifdef __cplusplus
}
#endif

View file

@ -17,6 +17,7 @@ pub enum SampleKind {
PhaseClaim,
EventClaim,
RenderPacket,
MoveEvidence,
}
impl SampleKind {
@ -31,6 +32,7 @@ impl SampleKind {
6 => Some(Self::PhaseClaim),
7 => Some(Self::EventClaim),
8 => Some(Self::RenderPacket),
9 => Some(Self::MoveEvidence),
_ => None,
}
}
@ -46,6 +48,7 @@ impl SampleKind {
Self::PhaseClaim => 6,
Self::EventClaim => 7,
Self::RenderPacket => 8,
Self::MoveEvidence => 9,
}
}
}
@ -229,6 +232,53 @@ pub struct LocalcastRenderPoint {
pub confidence: f32,
}
#[repr(C)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LocalcastMoveEvidenceKind {
OpticalMarker = 1,
ControllerState = 2,
}
#[repr(C)]
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct LocalcastMoveEvidenceSample {
pub witness_id_hash: u64,
pub controller_id_hash: u64,
pub source_timestamp_ns: u64,
pub arrival_ns: u64,
pub sequence: u64,
pub evidence_kind: u32,
pub flags: u32,
pub image_x: f32,
pub image_y: f32,
pub radius_px: f32,
pub confidence: f32,
pub accel_x: f32,
pub accel_y: f32,
pub accel_z: f32,
pub gyro_x: f32,
pub gyro_y: f32,
pub gyro_z: f32,
pub trigger: f32,
pub buttons_mask: u32,
pub reserved: u32,
pub battery01: f32,
pub reserved1: u32,
pub reserved2: u32,
}
#[repr(C)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct LocalcastMoveEvidenceBufferDescriptor {
pub sample_buffer_handle: u64,
pub sample_count: u32,
pub sample_stride_bytes: u32,
pub source_time_min_ns: u64,
pub source_time_max_ns: u64,
pub calibration_hash: u64,
pub tracking_space_hash: u64,
}
pub struct LocalcastReservoir {
inner: RollingReservoir<LocalcastSampleHandle>,
}
@ -258,6 +308,7 @@ pub struct LocalcastRuntimeStatus {
pub phase_claim_count: usize,
pub event_claim_count: usize,
pub render_packet_count: usize,
pub move_evidence_count: usize,
}
impl Default for LocalcastRuntimeStatus {
@ -275,6 +326,7 @@ impl Default for LocalcastRuntimeStatus {
phase_claim_count: 0,
event_claim_count: 0,
render_packet_count: 0,
move_evidence_count: 0,
}
}
}
@ -308,6 +360,7 @@ impl LocalcastRuntime {
phase_claim_count: self.view_len(SampleKind::PhaseClaim),
event_claim_count: self.view_len(SampleKind::EventClaim),
render_packet_count: self.view_len(SampleKind::RenderPacket),
move_evidence_count: self.view_len(SampleKind::MoveEvidence),
}
}
}
@ -623,6 +676,14 @@ pub extern "C" fn localcast_runtime_push_render_packet(
localcast_runtime_push_typed(ptr, SampleKind::RenderPacket, sample)
}
#[unsafe(no_mangle)]
pub extern "C" fn localcast_runtime_push_move_evidence(
ptr: *mut LocalcastRuntime,
sample: LocalcastSampleHandle,
) -> bool {
localcast_runtime_push_typed(ptr, SampleKind::MoveEvidence, sample)
}
#[unsafe(no_mangle)]
pub extern "C" fn localcast_runtime_status(
ptr: *const LocalcastRuntime,
@ -860,6 +921,46 @@ pub extern "C" fn localcast_producer_push_render_packet(
)
}
#[unsafe(no_mangle)]
pub extern "C" fn localcast_producer_push_move_evidence_buffer(
producer_ptr: *mut LocalcastProducer,
runtime_ptr: *mut LocalcastRuntime,
timestamp_ns: u64,
arrival_ns: u64,
descriptor: *const LocalcastMoveEvidenceBufferDescriptor,
out_sample: *mut LocalcastSampleHandle,
) -> bool {
let Some(producer) = (unsafe { producer_ptr.as_mut() }) else {
return false;
};
if producer.kind != SampleKind::MoveEvidence {
return false;
}
let Some(descriptor_ref) = (unsafe { descriptor.as_ref() }) else {
return false;
};
if descriptor_ref.sample_buffer_handle == 0 {
return false;
}
if descriptor_ref.sample_count == 0 {
return false;
}
if descriptor_ref.sample_stride_bytes as usize
!= std::mem::size_of::<LocalcastMoveEvidenceSample>()
{
return false;
}
let payload_handle = descriptor as u64;
localcast_producer_push(
producer_ptr,
runtime_ptr,
timestamp_ns,
arrival_ns,
payload_handle,
out_sample,
)
}
fn localcast_runtime_push_typed(
ptr: *mut LocalcastRuntime,
kind: SampleKind,
@ -943,6 +1044,51 @@ mod tests {
}
}
fn move_evidence_descriptor(
sample_buffer_handle: u64,
) -> LocalcastMoveEvidenceBufferDescriptor {
LocalcastMoveEvidenceBufferDescriptor {
sample_buffer_handle,
sample_count: 2,
sample_stride_bytes: std::mem::size_of::<LocalcastMoveEvidenceSample>() as u32,
source_time_min_ns: 1_000,
source_time_max_ns: 1_120,
calibration_hash: 0xCA11_BA7E,
tracking_space_hash: 0x57A6E,
}
}
fn move_marker_sample(
witness_id_hash: u64,
controller_id_hash: u64,
) -> LocalcastMoveEvidenceSample {
LocalcastMoveEvidenceSample {
witness_id_hash,
controller_id_hash,
source_timestamp_ns: 1_000,
arrival_ns: 1_010,
sequence: 7,
evidence_kind: LocalcastMoveEvidenceKind::OpticalMarker as u32,
flags: 0,
image_x: 123.5,
image_y: 87.25,
radius_px: 14.0,
confidence: 0.92,
accel_x: 0.0,
accel_y: 0.0,
accel_z: 0.0,
gyro_x: 0.0,
gyro_y: 0.0,
gyro_z: 0.0,
trigger: 0.0,
buttons_mask: 0,
reserved: 0,
battery01: f32::NAN,
reserved1: 0,
reserved2: 0,
}
}
fn render_point(stable_key_hash: u64) -> LocalcastRenderPoint {
LocalcastRenderPoint {
stable_key_hash,
@ -1585,6 +1731,125 @@ mod tests {
localcast_runtime_destroy(runtime);
}
#[test]
fn move_evidence_layout_is_fixed_for_compute_upload() {
let sample = move_marker_sample(11, 22);
assert_eq!(std::mem::size_of::<LocalcastMoveEvidenceSample>(), 112);
assert_eq!(
std::mem::size_of::<LocalcastMoveEvidenceBufferDescriptor>(),
48
);
assert_eq!(11, sample.witness_id_hash);
assert_eq!(22, sample.controller_id_hash);
assert_eq!(
LocalcastMoveEvidenceKind::OpticalMarker as u32,
sample.evidence_kind
);
assert_eq!(123.5, sample.image_x);
assert_eq!(14.0, sample.radius_px);
}
#[test]
fn move_evidence_descriptor_enters_native_runtime_window() {
let runtime = localcast_runtime_create(DEFAULT_RESERVOIR_NS);
let producer = localcast_producer_create(SampleKind::MoveEvidence.id(), 101, 3);
let descriptor = move_evidence_descriptor(9_000);
let mut sample = handle(0, 0, 0, 0, 0);
assert!(localcast_producer_push_move_evidence_buffer(
producer,
runtime,
1_200,
1_220,
&descriptor,
&mut sample,
));
let mut status = LocalcastRuntimeStatus::default();
assert!(localcast_runtime_status(runtime, &mut status));
assert_eq!(1, status.total_sample_count);
assert_eq!(1, status.move_evidence_count);
assert_eq!(0, status.camera_feature_count);
assert_eq!(
(&descriptor as *const LocalcastMoveEvidenceBufferDescriptor) as u64,
sample.payload_handle
);
assert_eq!(
1,
localcast_runtime_view_len(runtime, SampleKind::MoveEvidence.id())
);
localcast_producer_destroy(producer);
localcast_runtime_destroy(runtime);
}
#[test]
fn move_evidence_descriptor_push_rejects_wrong_kind_and_bad_buffers() {
let runtime = localcast_runtime_create(DEFAULT_RESERVOIR_NS);
let render_producer = localcast_producer_create(SampleKind::RenderPacket.id(), 88, 12);
let move_producer = localcast_producer_create(SampleKind::MoveEvidence.id(), 89, 0);
let good = move_evidence_descriptor(9_000);
let zero_handle = move_evidence_descriptor(0);
let zero_count = LocalcastMoveEvidenceBufferDescriptor {
sample_count: 0,
..good
};
let bad_stride = LocalcastMoveEvidenceBufferDescriptor {
sample_stride_bytes: 4,
..good
};
assert!(!localcast_producer_push_move_evidence_buffer(
render_producer,
runtime,
2_000,
2_010,
&good,
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
std::ptr::null(),
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
&zero_handle,
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
&zero_count,
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
&bad_stride,
std::ptr::null_mut(),
));
assert_eq!(
0,
localcast_runtime_view_len(runtime, SampleKind::MoveEvidence.id())
);
localcast_producer_destroy(render_producer);
localcast_producer_destroy(move_producer);
localcast_runtime_destroy(runtime);
}
#[test]
fn runtime_spine_rejects_nulls() {
let sample = handle(1, 2, 3, 4, 5);

View file

@ -22,14 +22,16 @@ flowchart TD
B["mic/loopback drivers"] --> R
C["Leap timing/IR driver"] --> R
D["network feed producers"] --> R
M["Starfire/Nightwing Move tracking"] --> R
R --> N["native reservoir handles"]
N --> E["Fensalir GPU fusion + UI"]
N --> F["Faust/native DSP"]
E --> G["Spout2/program video"]
E --> V["EVE native shared-texture stream"]
E --> G["Mimir program video"]
F --> H["program stems + spatial bed"]
G --> I["OBS"]
H --> I
G --> J["Yggdrasil site publisher"]
H --> J
G -. "compatibility" .-> K["OBS adapter"]
H -. "compatibility" .-> K
```
Ownership:
@ -42,14 +44,22 @@ Ownership:
becoming independent clock authorities.
- `Mimir.Runtime` owns app-level stream buffers and synchronization.
- Native capture workers own device reads.
- Move tracking observations are stream samples, not program controls. Muninn
daemons on Starfire and Nightwing publish source-local optical marker
candidates plus controller/IMU/button state for USB-attached Moves. Odin owns
discovery/schema projection for those streams. Mimir owns calibration,
association, triangulation, IMU fusion, prediction, and the resolved
Fensalir-facing wand pose stream.
- Mimir owns program composition, source subscription policy, preview/control
state, stats, and publication intent.
- Fensalir owns dense visual fusion, material/brush/splat reconciliation,
D3D12 interop, runtime UI, Spout2 publication, and the completed backbuffer
texture copied into the EVE shared D3D12 program-output surface.
D3D12 interop, runtime UI lowering, and local program texture output.
- Faust/native DSP owns hot audio alignment, suppression, separation,
spatialization, and stem generation.
- OBS owns broadcast controls.
- EVE owns native decode/composite for its fullscreen display path; it does not
own layout, WebKit compatibility, or timing authority.
- The Yggdrasil-facing publisher daemon consumes Mimir program output and
publishes it to the site without owning composition.
- OBS is a temporary compatibility sink, not a composition or broadcast
authority.
Invariant: the live window is bounded, in memory, and has one timing authority.
No private history outlives the rolling buffer.
@ -58,7 +68,7 @@ No private history outlives the rolling buffer.
[[docs/viable-stream-app|Viable Stream App]] defines the near-term app target.
Fensalir hosts the running Mimir app, keeps the default five-second runtime in
memory, exposes debug/settings/output controls, and emits synchronized OBS
memory, exposes debug/settings/output controls, and emits synchronized Mimir
program video plus separately controllable audio stems.
`Mimir.Runtime` currently provides:
@ -71,6 +81,7 @@ program video plus separately controllable audio stems.
- `MimirProcessStreamSource`;
- `MimirFrameEventProcessStreamSource`;
- `MimirVideoFrameDescriptor`;
- `MimirTrackingObservation`;
- `MimirAudioSynchronizationAnalyzer`;
- `IMimirVideoCaptureDriver`;
- `MimirVideoCaptureDriverSource`.
@ -83,6 +94,221 @@ the rolling buffers while the direct ABI driver is being cut. It does not carry
pixels and does not own the final six-camera hot path. Multi-camera probes are
one process with declared accepted source ids, not one process per camera.
## Move Tracking
Starfire and Nightwing both have PS Move controllers directly attached over
USB. Muninn represents what each local body can honestly observe:
`muninn.move_marker_candidate.v1` for glowing orb candidates from cameras and
`muninn.move_controller_state.v1` for controller/IMU/button/battery state from
USB-attached Moves. Mimir drinks those feeds into `MimirStreamKind.Tracking`
buffers, admits them into the native `move_evidence` reservoir view as
compute-upload buffers, and publishes resolved `mimir.move_controller_pose.v1`
documents for Fensalir interaction once calibration and fusion have earned a
pose.
Authority:
- Starfire-local Move evidence is published by Muninn/Mimir local producers as
typed observation streams.
- Nightwing Move evidence is emitted by Muninn on Nightwing over the typed
observation path.
- Structured PS Move light pulses are Muninn output commands. Mimir requests
them by sending `muninn.move_light_command.v1` over CultNet/CultMesh to the
Muninn daemon on the host with the USB-attached Move; Mimir does not write
remote HID directly except through temporary smoke scripts used to prove
hardware behavior before Muninn is available on that host.
- Odin discovers and projects the stream/schema surface for operator and agent
access.
- Mimir owns subscription, rolling retention, native Move evidence buffer
layout, clock alignment, calibration, controller association, triangulation,
IMU fusion, prediction, and resolved wand pose publication. OBS and
dashboard summaries do not own this state.
- Fensalir consumes Mimir's resolved Move controller poses as interactive input
for environments; it does not synthesize controller truth from raw Muninn
evidence.
Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj
-- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing
Move observation into two tracking buffers.
Calibration preflight proof: `dotnet run --project
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj --
--move-calibration-protocol-smoke --output
artifacts\move-calibration\protocol.cc` writes
`mimir.move_calibration_protocol.v1` for
`mimir-move-calibration-starfire-nightwing-v1`. The protocol requires
`muninn:starfire:move-evidence`, `muninn:nightwing:move-evidence`, and
`mimir:starfire:move-controller-poses`; optionally drinks Quest headset and
left/right controller poses over USB as an external VR reference; and requires
Mimir to derive rig calibration, IMU calibration, controller identity mapping,
and a calibration receipt before orientation can stop being
`orientation:imu-unresolved`.
Quest USB access belongs to Muninn, not Mimir. Starfire Muninn runs with
`--quest-adb --quest-serial 1WMHHB68PG1515` and publishes
`muninn.quest_access.v1`, including stream ids for Quest input, Quest poses, and
warp-corrected video input from Brokkr/Unity. On 2026-06-12, `adb devices -l`
reported authorized Quest 2 device `1WMHHB68PG1515` with product/device
`hollywood`. This proves Starfire can query the headset over USB; it does not
provide Quest headset/controller poses until a Quest/OpenXR witness bridge
publishes those pose samples through Muninn.
The old Nightwing bring-up path is archived:
`scripts/start-nightwing-move-tracking.ps1`, `Mimir.EveSensorReceiver`,
`Mimir.VerseRecorder`, and `nightwing_typed_witness_publisher.py` now fail
closed. Runtime structured light commands belong to Muninn. Eye/Move
observations must enter through typed CultMesh stream frames or
Odin-discovered CultMesh documents; they are optical witness evidence for the
later pose/fusion owner, not the final 6DoF pose authority.
Starfire-local Move illumination smoke uses `scripts/start-starfire-move-light.ps1`.
That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01
output collection and refreshes the PS Move LED report for hardware proof when
Muninn is not available on Starfire yet.
Starfire should have Muninn for source-local Move HID control and local sensor
publication. Some older Starfire audio/video paths still enter Mimir through
local Mimir producers or diagnostic process bridges while direct native drivers
mature. Same-host Move evidence and pose frames are CultMesh shared-memory byte
rings. Remote media is different: the current Raven media body bridge uses
CultNet/CultMesh documents over the mesh and lowers to local UDP for OBS
compatibility, so it is not same-machine zero-copy into Mimir/Fensalir yet.
Odin's Muninn organ owns the Move optical extraction stage for sensor stream
exposure. Its Rust crate lives in `E:\Projects\Odin\crates\muninn-move-tracker`
and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes
those Muninn candidate streams into tracking buffers; it does not own raw
optical extraction. Mimir is now the explicit owner for stereo triangulation,
camera calibration, controller association, IMU fusion, prediction, and final
6DoF wand pose.
Muninn's `muninn:nightwing:move-evidence:<sequence>` shared-memory frame now
uses the canonical `muninn.move_marker_candidate.v1` document shape for optical
markers instead of a placeholder empty slice. A daemon unit proves that a bright
Y8 frame run through `muninn-move-tracker` serializes a non-empty marker
candidate into the Mimir-compatible evidence frame. The daemon now has a
source-local Y8 extraction/publish seam plus a first `serve` camera producer:
`--move-marker-camera <camera-id>=<device-path>` polls a Unix V4L2 YUYV frame,
converts it to compact Y8, and feeds that same seam. Nightwing hardware now
runs as one Muninn provider with two private per-eye PSMoveAPI subprocesses.
The parent alone owns lights, identity, commands, provider advertisement,
persistence, and the aggregate evidence stream; children exchange hue state
and camera-tagged observations over bounded MessagePack pipes.
`muninn.move_tracker_health.v1` exposes each worker's camera backend,
calibration, update/observation counters, RGB range, and expected-hue pixel
evidence. Live exposure is 0.3 for `nightwing-eye-0` and 0.1 for
`nightwing-eye-1`. Both eyes calibrate all four Moves and produce observations.
Muninn's dedicated evidence aggregator now owns sequence and transport outside
the 15-second CultCache telemetry loop. Cached controller state is bundled into
fresh optical frames but cannot create frames by itself, and a deadline clock
caps publication at the configured camera cadence. A 2026-07-13 Mimir field
window received 622 unique frames in 15 seconds, 2,647 observations, and 1,384
same-ID cross-camera correspondences while seeing all four stable Move IDs from
both eyes. The same parent PID survived the run; transport health reported
5,302 produced frames, 5,301 handoffs, 771 sends, and zero local-ring
admissions. Full stereo calibration is still not implied: intrinsics and
spatially diverse correspondences must earn the camera matrices.
`MimirMoveStereoCalibrationAssessment` is the typed observability gate before
any fit. It accepts only same aggregate-frame, same-Move correspondences and
reports synchronized pair count, Move diversity, per-Eye 4x3 grid coverage,
radius range, and missing physical inputs. The corrected live capture produced
130 same-frame pairs with zero frame skew, but only 4/12 Eye 0 cells and 2/12
Eye 1 cells; intrinsics and measured orb radius are also absent, so promotion is
correctly false.
`MimirMoveProofSurface` is the first Fensalir-visible proof surface for this
chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream
frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only
`AquariumSplineFrame` probe with the explicit
`muninn:nightwing:move-evidence:<sequence> -> mimir:starfire:move-evidence:<sequence> -> mimir:starfire:move-pose:<sequence> -> fensalir:starfire:presented-frame:<sequence>`
chain. It does not decide pose. Single-ray fallback remains a visible
non-final verdict until calibrated multi-camera optical evidence earns full
pose.
`MimirMoveProofPipeline` is now the reusable commit primitive for that chain:
from one Muninn CultMesh shared-memory frame it performs native admission,
Mimir fusion, pose-frame creation, proof-surface creation, and Fensalir probe
lowering. Live Nightwing/Starfire wiring should call this owner path rather
than preserving separate manual actions and programmatic proof paths.
`MimirRuntime` now owns the actual Fensalir presentation attachment for the
proof surface: `PublishMoveProofSurface` stores the latest Mimir-owned
`mimir.move_proof_surface.v1`, and `CreateFrame` merges its observer-only
`move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside the
audio spectrum preview. Fensalir renders this frame; it still does not decide
proof.
`MimirMoveProofRuntimeDriver` is the runtime bridge from a Muninn CultMesh
shared-memory evidence ring into that attachment point. It derives the
Mimir evidence, Mimir pose, and Fensalir frame ids from the actual Muninn frame
suffix, calls `MimirMoveProofPipeline`, suppresses duplicate frame admission,
and lets `MimirRuntime.Update` publish the resulting proof surface. This proves
the runtime-owned path for a ring frame; real Nightwing hardware and calibrated
camera witnesses are still the missing field proof.
`MimirMoveProofRuntimeConfiguration` is the typed selection and validation
surface for that bridge. An enabled source must name the Muninn evidence stream,
native reservoir path, Mimir evidence/pose frame prefixes, Fensalir
presented-frame prefix, fusion authority, consumer contract, and at least two
calibrated camera witnesses. It may create a runtime driver only when the
supplied CultMesh ring stream id matches the configured evidence stream.
`MimirRuntime` now owns activation for configured proof sources: on scene-ready
it asks an `IMimirMoveProofEvidenceRingProvider` for the evidence ring, opens
the configured native reservoir, registers the runtime driver, retains the
driver resources, and exposes an activation status. The default provider fails
explicitly because the current C# CultMesh ring is still in-process only;
`Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an in-process
provider to prove the activation path without claiming live Nightwing field
evidence.
`mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine
surface for configured Move proof sources. It reports the configured evidence
stream, provider kind, active/driver state, native reservoir path, calibrated
camera ids, latest same-stream proof ids, latest verdict, and the inactive
diagnostic when the default runtime lacks a live cross-process CultMesh ring
opener. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke`
proves both the active injected-ring document and the unavailable default
document; this is readiness/proof observation, not field evidence.
`mimir.move_proof_evidence_frame_snapshot.v1` is a one-copy, explicitly
non-final fallback for captured Muninn evidence frames while C# CultMesh lacks
a cross-process shared-memory opener. When `EvidenceSnapshotPath` is
configured, `MimirConfiguredMoveProofEvidenceRingProvider` validates snapshot
metadata against the encoded Muninn frame payload, publishes the payload into
an owned in-process ring, and then delegates to the same runtime proof driver
and pipeline. `Mimir.BufferSmoke --move-proof-runtime-snapshot-smoke` proves
the same named proof spine through that file boundary. This can support field
capture/replay, but live proof still requires the real Nightwing/Starfire ring
or CultMesh page producer.
Odin/Muninn can now write that artifact from the producer side with
`--move-evidence-snapshot <path>`. The snapshot is written by the same Muninn
Move evidence publisher that owns controller/marker frames, after the frame
payload is accepted by the stream ring. Its unit coverage decodes the
`mimir.move_proof_evidence_frame_snapshot.v1` tuple and re-decodes the embedded
Muninn payload, proving the artifact is shaped for Mimir's snapshot provider.
This is still one-copy field capture/replay; it does not replace the live ring
or a CultCache page body producer.
`MimirMoveProofDevSurface` is a dev-only bootstrap gated by
`MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof
surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so
`Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App`
headless, capture the actual Fensalir-presented PNG, and pixel-check that the
named proof probe is visible. This is a renderer/probe proof, not live hardware
evidence.
## Program Composition
[[docs/mimir-program-composition|Mimir Program Composition]] is the live
authority map for stream-program output. Muninn runs on Starfire, Nightwing, and
Raven-class capture hosts to publish local stream capabilities and selected
media bodies. Mimir consumes only the streams it needs for calibration and
composition, commits one typed scene graph,
and publishes program output locally plus through a Yggdrasil-facing site
publisher.
The first typed program contracts live in `src/Mimir.Runtime/Synchronization`:
- `mimir.program_scene.v1`
- `mimir.program_output.v1`
- `mimir.eve_operator_surface.v1`
OBS scene JSON is import/mirror evidence for the initial crop/key/transform
layout. It is not the editor of record after Mimir owns the scene graph.
The current imported local scene state lives at
`state/mimir-program-composition.cc`.
## OBS Bridge Utility
```mermaid
@ -101,6 +327,122 @@ flowchart TD
The bridge is useful because it is inspectable and already speaks OBS. It does
not become the synchronized program authority.
## CultMesh Media Bridge
The live Raven-to-Starfire program path is Muninn's typed CultNet media body:
```mermaid
flowchart TD
A["Raven D3D11 desktop capture"] --> B["Long-lived NVENC encoder"]
C["Raven WASAPI loopback"] --> D["AAC-LC / ADTS encoder"]
B --> E["V4 8+4 block FEC and realtime video lane"]
D --> F["Fixed 864-byte 4+2 audio FEC lane"]
E --> G["Starfire OBS Muninn receiver udp/5200"]
F --> G
G --> H["Frame assembly / FEC / IDR feedback"]
G --> I["120 ms audio reorder / concealment"]
H --> J["OBS-local video decoder socket"]
I --> K["OBS-local audio decoder socket"]
```
Owner: Muninn owns Raven capture children, access-unit/audio packetization,
CultNet delivery, repair material, and encoder feedback response. The Mimir OBS
plugin owns receipt admission, frame/FEC assembly, audio continuity, and local
OBS lowering. OBS owns decode and composition. Odin/CultMesh discovery names
the provider surface; neither discovery nor the compatibility decoder sockets
own hot media truth.
Invariant: video packets are useful only before their assembly deadline.
Canonical video and V4 parity use unreliable `realtime`; 8+4 block FEC,
queue-admitted selective repair, and IDR recovery own loss. Canonical AAC audio
and fixed 4+2 parity also use
`realtime`; reorder and bounded concealment own playout continuity. Completed
video frames are tombstoned so late shards cannot
become a second assembly owner. Raven uses direct LAN only and starts through a
hidden interactive-token scheduled task; no WireGuard hop or foreground
terminal participates.
The older `Mimir.CultMeshMedia` relay description below is retained as a
non-authoritative historical/body-bridge map. It does not own the active Muninn
OBS feed.
```mermaid
flowchart TD
A["Raven FFmpeg desktop + WASAPI loopback mux"] --> B["Mimir.CultMeshMedia send"]
B --> C["CultNet reliable UDP media-frame documents"]
C --> D["Yggdrasil CultMesh relay discovered through Odin/CultMesh"]
D --> E["Starfire Mimir.CultMeshMedia recv"]
E --> F["local MPEG-TS UDP udp://127.0.0.1:5200"]
F --> G["OBS Raven Monitor + Realtek"]
```
Owner: `src/Mimir.CultMeshMedia` owns the live Raven-to-Starfire media body
bridge over CultMesh/CultNet. FFmpeg still owns Raven capture and encoding.
Mimir owns composition. OBS may consume the Starfire-local MPEG-TS endpoint as a
compatibility sink while Mimir/Fensalir program output and site publication
mature.
Invariant: Raven media body state is live and bounded. Sender writes rolling
`mimir.cultmesh_media_frame` slot documents rather than an unbounded durable
video archive. Yggdrasil relays CultNet reliable UDP behind the logical
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av` route discovered
through Odin/CultMesh. Starfire lowers the subscribed stream to local UDP for
compatibility sinks because OBS is not a CultMesh runtime.
Source audit correction resolved: `Mimir.CultMeshMedia` now uses explicit
CultLib RUDP helpers for the sender/receiver and a relay-owned RUDP
socket/session loop for multi-peer forwarding. The older
`CultMesh.StartNodeAsync` and `CultMesh.ConnectClient` path no longer owns this
media lane. The relay still persists received puts to CultCache as evidence and
for inspection; live network transit is the RUDP schema-document path.
Current deployment: Yggdrasil runs the relay from
`/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia` with cache
`/var/lib/gamecult/mimir/cultmesh-media.cc` and log
`/var/log/gamecult/mimir-cultmesh-media.log`. Senders and receivers target
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`; concrete RUDP
bootstrap lives in the CultMesh resolver environment, not in daemon launch
arguments. Starfire writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS
source `Raven Monitor + Realtek` points at that local UDP endpoint.
`scripts/start-raven-cultmesh-av-sender.ps1` is the Mimir-owned CultMesh bridge
bootstrap for Raven media-body transit. It is not the actual Muninn OBS feed
owner. The real Raven OBS/SRT feed owner is Odin's Muninn:
`E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1` drives the
existing `GameCult-Muninn-Activate` hidden task and `muninn.exe activate`
body on Raven, and `scripts/start-raven-muninn-obs-feed.ps1` in this repo is
the thin local wrapper for that actuator. Keep `-LocalBootstrap` on the
CultMesh sender only as a staging/bootstrap edge when the separate body-bridge
lane needs direct local proof.
## Daemon Health And Idunn
Odin owns the accepted Verse/service map. Idunn owns continuity after a daemon
is known: boot recovery, crash recovery, deployment freshness, stale-health
watching, restart/deploy intent, and operator escalation. Mimir does not run a
private lifecycle supervisor for these surfaces.
CultLib's current direction is RUDP everywhere for typed CultNet/CultMesh
documents across runtimes. That means health, provider advertisements, command
boundaries, transport profiles, and selected program/media document lanes should
default to `cultnet.transport.rudp.v0`. Product/debug render surfaces are
lowerings, debug tools, or migration debt; they must not own daemon truth once
an RUDP contract exists.
Mimir's current daemon-health publishers are local RUDP witnesses:
Invariant: health publication is not ownership. The daemon reports what it can
honestly observe about itself; Idunn decides keepalive action through the
shared typed record path.
Source audit cut: `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the
Nightwing typed witness publisher, and `start-nightwing-move-tracking.ps1` are
archived. The old Nightwing/EVE witness lane no longer opens a renderer/client
subscription surface. Witness publication and recording must use typed
CultMesh/CultNet documents or CultMesh stream frames discovered through Odin.
The media-frame bridge is already on that side of the line.
## Audio Field
The six-microphone path is separate from the bridge. Scarlett speaker loopback

View file

@ -21,7 +21,23 @@ Get-Content .\state\evidence.jsonl -Tail 8
- Mimir's Face doctrine lives in [[docs/mimir-face|Mimir Face]].
- VoidBot persona/state live under `.voidbot/voice/` and `.voidbot/state/`.
Commit `.voidbot` dirt in a separate state commit whenever it appears.
- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN ingest.
- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN compatibility ingest.
- V1 now also has an explicit CultMesh media bridge for Raven program feeds:
`src/Mimir.CultMeshMedia` sends rolling `mimir.cultmesh_media_frame`
documents over CultNet reliable UDP through Yggdrasil and lowers them on
Starfire to local MPEG-TS UDP for OBS. The bridge now uses explicit CultLib
RUDP helpers/session plumbing; the old
`CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` path no longer owns this
media lane.
- CultLib has moved the daemon-swarm direction to CultNet over RUDP across
runtimes. Mimir's typed daemon truth should default to
`cultnet.transport.rudp.v0`; TCP/HTTP/WebSocket are client lowerings,
compatibility probes, or migration debt unless a specific runtime has not
earned RUDP yet.
- Mimir's old Eve dashboard broker is archived and no longer publishes Idunn
health, `/eve/deck`, `/eve/dashboard`, or WebSocket commands. The browser
reference is a client lowering, not service truth. Dashboard state must return
as typed CultMesh/Eve documents through Odin before deployment resumes.
- The live stream app is C# plus Fensalir: `Mimir.slnx` contains
`src/Mimir.App` and `src/Mimir.Runtime`.
- `src/Mimir.App` hosts Fensalir as the windowing/rendering/D3D12 bridge.
@ -37,6 +53,19 @@ Get-Content .\state\evidence.jsonl -Tail 8
dictionary, low-level implementation notes, and sample code sketches for the
next decoder/DSP/native-ring/Fensalir implementation passes. Start with
[[research/perfect-machine-study-2026-05-23/reading-guide|Reading Guide]].
- `docs/mimir-program-composition.md` is the current program-output authority
map: Muninn observes on Starfire/Nightwing/Raven-class hosts, Mimir consumes
selected streams and owns composition, Eve lowers controls/previews/stats,
Yggdrasil publishes to the site, and OBS is compatibility-only.
- `src/Mimir.Runtime/Synchronization` now has typed program contracts for
`mimir.program_scene.v1`, `mimir.program_output.v1`, and
`mimir.eve_operator_surface.v1`; `Mimir.BufferSmoke
--perfect-machine-contract-smoke` writes them into CultCache.
- `Mimir.BufferSmoke --import-obs-program-scene --output
state/mimir-program-composition.cc` imports the current OBS scene JSON into
Mimir-owned typed state. The latest imported scene has eight layers, six
visible, two cropped, one chroma-keyed, and Raven mapped as
`muninn:raven:monitor:primary`.
- Local six-camera ingest should use direct driver adapters. Process-backed
sources are bridge/network edges only.
- Leap stereo IR is the first timing-camera candidate for direct ingest.
@ -45,10 +74,20 @@ Get-Content .\state\evidence.jsonl -Tail 8
- Fensalir owns production GPU fusion, UI, and Spout2 publication.
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
synchronized stems.
- OBS receives final program surfaces; it does not own synchronization.
- OBS compatibility adapters may receive final program surfaces; they do not
own synchronization, composition, preview/control, or publication.
## Current Pressure
- Nightwing Move tracking is one Muninn provider with two private per-eye
PSMoveAPI worker subprocesses. Eye 0 uses exposure 0.3; Eye 1 uses 0.1.
Both workers calibrate four Moves and advance continuously. Mimir has now
proved all four stable IDs on both eyes in the same frames. The latest
15-second receipt contains 16 observations and eight pair records but only
two unique aggregate frame IDs. Next measure parent publication and RUDP send
counters directly, fix cadence, then collect diverse correspondences for
camera calibration.
- Implement the first concrete Leap direct capture driver and measure cadence.
- Add remaining camera drivers through `IMimirVideoCaptureDriver`.
- Add native audio capture workers for mic, loopback, and network audio feeds.
@ -57,8 +96,24 @@ Get-Content .\state\evidence.jsonl -Tail 8
variable-rate resampling.
- Bind Fensalir UI to stream health, buffer depth, timestamps, settings, and
output management.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until native program output
replaces them.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until native Mimir program
output, Eve operation, and Yggdrasil site publication replace them.
- Yggdrasil currently runs the CultMesh media relay from
`/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia`; senders and
receivers target `cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`
and let the CultMesh resolver choose the concrete RUDP route. Starfire writes
`raven-primary-av` to `udp://127.0.0.1:5200`. That is the
`Mimir.CultMeshMedia` body bridge, not the OBS plugin feed owner.
- The actual Raven OBS/SRT feed owner is Odin's Muninn on Raven. Use
`scripts/start-raven-muninn-obs-feed.ps1` in this repo as the thin wrapper
over `E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1`, which
drives Raven's real `GameCult-Muninn-Activate` hidden task and
`muninn.exe activate` path. `scripts/start-raven-cultmesh-av-sender.ps1`
remains the separate CultMesh bridge/bootstrap lane.
- Rebuild Eve dashboard only as a CultMesh/Odin surface publisher plus renderer
lowering. Provider advertisement, retained state, command boundary, transport
profile, and command documents must be typed CultMesh records; no TCP, HTTP,
or WebSocket surface should become daemon or transport truth.
- Kiyo Pro has two UVC extension units. Moving it to a motherboard USB3 port
changed the descriptor path to `root_hub30` / `bcdUSB=0x0320` and exposed
720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports
@ -143,6 +198,41 @@ Get-Content .\state\evidence.jsonl -Tail 8
## Immediate Re-entry Instruction
The 2026-07-14 live two-wand session at
`artifacts/move-calibration/live-two-wand-20260714-004210.*` passed collection
coverage with 1,168 admitted same-frame pairs. The new Mimir-owned sphere
stereo fitter recovers synthetic 505/535 px focal lengths within 0.43 px and a
34 cm baseline within 0.22 mm, with 0.34/0.63 px held-out median/P95 error.
The live artifact remains rejected: after robust radius admission its consensus
fits at 0.51/2.08 px, but only 53.8 percent of held-out pairs agree and Eye 1's
focal estimate leaves the physical range. PSMoveAPI age is currently projected
as marker confidence by Muninn, so fresh wrong-color matches arrive near 0.996
confidence. Repair source-local observation quality/continuity and calibrate
per-Eye blob-radius response before asking the operator for another sweep. Do
not promote the nominal 22.5 mm diagnostic fit.
Odin `d0eded6` is deployed on Nightwing as binary SHA-256
`a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39`.
Each Eye worker now rejects stale, sub-2 px, out-of-bounds, and discontinuous
PSMoveAPI positions before aggregation; tracker health publishes rejection
counters, and candidate score is explicitly uncalibrated `0.5` instead of age
masquerading as confidence. A post-deploy 10-second Mimir window received 555
frames, 2,246 observations, and 443 stereo pairs. Radius P01 rose from about
0.707 px to 3.13/3.64 px and confidence P50 is now 0.5 on both Eyes. Eye 1 saw
only two plausible stationary IDs, which is honest and means another sweep is
needed after Mimir models per-Eye blob-radius response.
The active priority is Nightwing Move calibration. One public Muninn observer
owns two private Eye workers and one aggregate evidence stream. Odin commits
`39c839a`, `a340c33`, `8aefd1c`, and `f82ca7a` give daemon health a persistent
serviced RUDP session and move optical aggregation onto its own deadline-clocked
loop. The 2026-07-13 receipt at
`artifacts/move-calibration/muninn-clocked-aggregator.mpack` contains 622 frames,
2,647 observations, and 1,384 cross-camera correspondences in 15 seconds, with
all four Move IDs visible to both Eyes. Next collect spatially diverse samples,
calibrate intrinsics/extrinsics, and publish residual-bearing typed calibration.
Radius is a weighted range cue, not a substitute for camera calibration.
The corrected two-hour study pass ran from
`2026-05-23T23:54:42.9041588+01:00` to at least
`2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement the

View file

@ -1,47 +0,0 @@
param(
[string] $Target = "eve",
[string] $SharedTextureName = "Global\MimirFensalirProgramTexture",
[string] $Configuration = "Debug",
[string] $LogPath = "logs\eve-program-output-fensalir.log",
[string[]] $AppArguments = @()
)
$ErrorActionPreference = "Stop"
$repoRoot = Split-Path -Parent $PSScriptRoot
$absoluteLogPath = Join-Path $repoRoot $LogPath
$absoluteErrorLogPath = [System.IO.Path]::ChangeExtension($absoluteLogPath, ".err.log")
$logDirectory = Split-Path -Parent $absoluteLogPath
if (-not (Test-Path $logDirectory)) {
New-Item -ItemType Directory -Path $logDirectory | Out-Null
}
Write-Host "Checking EVE SSH target: $Target"
ssh $Target "uname -a >/tmp/mimir-eve-program-output-check.txt && cat /tmp/mimir-eve-program-output-check.txt"
$env:FENSALIR_PROGRAM_OUTPUT_D3D12 = "1"
$env:FENSALIR_PROGRAM_OUTPUT_NAME = $SharedTextureName
$projectPath = Join-Path $repoRoot "src\Mimir.App\Mimir.App.csproj"
$dotnetArgs = @(
"run",
"--project",
$projectPath,
"-c",
$Configuration,
"--"
) + $AppArguments
Write-Host "Starting Mimir/Fensalir with shared D3D12 output: $SharedTextureName"
Write-Host "Log: $absoluteLogPath"
$process = Start-Process -FilePath "dotnet" `
-ArgumentList $dotnetArgs `
-WorkingDirectory $repoRoot `
-RedirectStandardOutput $absoluteLogPath `
-RedirectStandardError $absoluteErrorLogPath `
-WindowStyle Hidden `
-PassThru
Write-Host "Started PID $($process.Id). Poll with:"
Write-Host " Get-Content -Tail 40 '$absoluteLogPath'"
Write-Host " Get-Content -Tail 40 '$absoluteErrorLogPath'"

View file

@ -0,0 +1,158 @@
param(
[string]$NightwingHost = "nightwing",
[int]$Port = 8796,
[string]$StarfireHost = "192.168.1.66",
[string]$Path = "/eve/periwinkle",
[string]$EyeDevices = "/dev/video2,/dev/video3",
[string[]]$MoveLight = @(),
[string]$RecorderRoot = "C:\Users\Meta\Videos\Mimir\VerseCaptures",
[switch]$SkipRecorder,
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
$repo = Split-Path -Parent $PSScriptRoot
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
$runDir = Join-Path $repo "artifacts\runtime\nightwing-move-tracking-$runId"
$witnessUrl = "ws://$StarfireHost`:$Port$Path"
$subscribeUrl = "ws://127.0.0.1:$Port$Path/subscribe"
$remoteLog = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.log"
$remotePid = "~/.local/state/gamecult/nightwing-typed-witness.pid"
$remoteHeartbeat = "~/.local/state/gamecult/nightwing-typed-witness.heartbeat"
New-Item -ItemType Directory -Force -Path $runDir | Out-Null
New-Item -ItemType Directory -Force -Path $RecorderRoot | Out-Null
function Start-LoggedProcess {
param(
[string]$Name,
[string]$FilePath,
[string[]]$ArgumentList
)
$stdout = Join-Path $runDir "$Name.out.log"
$stderr = Join-Path $runDir "$Name.err.log"
if ($DryRun) {
Write-Host "DRY $Name`: $FilePath $($ArgumentList -join ' ')"
return $null
}
$process = Start-Process `
-FilePath $FilePath `
-ArgumentList $ArgumentList `
-WorkingDirectory $repo `
-WindowStyle Hidden `
-RedirectStandardOutput $stdout `
-RedirectStandardError $stderr `
-PassThru
Set-Content -Path (Join-Path $runDir "$Name.pid") -Value $process.Id
Write-Host "$Name pid=$($process.Id) stdout=$stdout stderr=$stderr"
return $process
}
function Wait-Health {
param([string]$Url)
if ($DryRun) {
Write-Host "DRY wait health: $Url"
return
}
$deadline = (Get-Date).AddSeconds(45)
do {
try {
$response = Invoke-WebRequest -UseBasicParsing -Uri $Url -TimeoutSec 3
if ($response.StatusCode -ge 200 -and $response.StatusCode -lt 300) {
Write-Host "health ok: $Url"
return
}
} catch {
Start-Sleep -Milliseconds 500
}
} while ((Get-Date) -lt $deadline)
throw "Timed out waiting for $Url"
}
Write-Host "Nightwing Move tracking run: $runDir"
Write-Host "Witness URL: $witnessUrl"
if (-not $DryRun) {
Get-CimInstance Win32_Process |
Where-Object {
($_.CommandLine -match "Mimir.EveSensorReceiver" -and $_.CommandLine -match "--port $Port") -or
($_.CommandLine -match "Mimir.VerseRecorder" -and $_.CommandLine -match "$Port$Path/subscribe")
} |
ForEach-Object {
try { Stop-Process -Id $_.ProcessId -Force -ErrorAction Stop } catch {}
}
}
$receiver = Start-LoggedProcess `
-Name "periwinkle-receiver" `
-FilePath "dotnet" `
-ArgumentList @(
"run",
"--project",
(Join-Path $repo "src\Mimir.EveSensorReceiver\Mimir.EveSensorReceiver.csproj"),
"--",
"--port", "$Port",
"--path", $Path,
"--subscribe-path", "$Path/subscribe",
"--source-id", "nightwing-witness",
"--type", "cultmesh-observation"
)
if ($receiver) {
Wait-Health -Url "http://127.0.0.1:$Port/health"
}
if (-not $SkipRecorder) {
Start-LoggedProcess `
-Name "verse-recorder" `
-FilePath "dotnet" `
-ArgumentList @(
"run",
"--project",
(Join-Path $repo "src\Mimir.VerseRecorder\Mimir.VerseRecorder.csproj"),
"--",
"--url", $subscribeUrl,
"--out-dir", $RecorderRoot
) | Out-Null
}
if ($DryRun) {
Write-Host "DRY Nightwing stage and start on $NightwingHost"
} else {
scp (Join-Path $repo "tools\nw_eye_cap.py") "${NightwingHost}:/tmp/nw_eye_cap.py" | Out-Null
scp (Join-Path $repo "tools\nw_move_hint.py") "${NightwingHost}:/tmp/nw_move_hint.py" | Out-Null
scp (Join-Path $repo "tools\nightwing_typed_witness_publisher.py") "${NightwingHost}:/tmp/nightwing_typed_witness_publisher.py" | Out-Null
$moveArgs = ($MoveLight | ForEach-Object { "--move-light '$($_ -replace "'", "'\''")'" }) -join " "
$remoteScript = @"
mkdir -p ~/.local/state/gamecult
pkill -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' || true
nohup python3 /tmp/nightwing_typed_witness_publisher.py --url '$witnessUrl' --track-eyes --track-builtin-camera --track-builtin-mic --interval 0.10 --heartbeat-path '$remoteHeartbeat' --eye-devices '$EyeDevices' --eye-window-seconds 0.08 --tracking-stride 6 $moveArgs > $remoteLog 2>&1 < /dev/null &
sleep 0.5
pgrep -n -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' > $remotePid
cat $remotePid
"@
$remoteB64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($remoteScript))
$nightwingPid = ssh $NightwingHost "printf '%s' '$remoteB64' | base64 -d | bash"
Set-Content -Path (Join-Path $runDir "nightwing-witness.pid") -Value ($nightwingPid | Select-Object -First 1)
Set-Content -Path (Join-Path $runDir "nightwing-witness.logpath.txt") -Value $remoteLog
Write-Host "nightwing-witness pid=$($nightwingPid | Select-Object -First 1) log=$remoteLog"
}
$manifest = [ordered]@{
kind = "mimir.nightwing_move_tracking_supervisor.v1"
runId = $runId
runDir = $runDir
witnessUrl = $witnessUrl
subscribeUrl = $subscribeUrl
heartbeatPath = $remoteHeartbeat
eyeDevices = $EyeDevices
directHidSmokeMoveLights = $MoveLight
startedAt = (Get-Date).ToString("o")
}
$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path (Join-Path $runDir "supervisor.json")
Write-Host "Supervisor manifest: $(Join-Path $runDir "supervisor.json")"

View file

@ -0,0 +1,89 @@
param(
[string]$RelayHost = "10.77.0.1",
[int]$RelayPort = 3075,
[string]$Stream = "raven-primary-av",
[string]$FfmpegPath = "ffmpeg",
[string]$VideoInput = "desktop",
[ValidateSet("WasapiLoopback", "DirectShow")]
[string]$AudioMode = "WasapiLoopback",
[string]$AudioInput = "Stereo Mix (Realtek(R) Audio)",
[ValidateSet("Console", "Multimedia", "Communications")]
[string]$LoopbackRole = "Console",
[int]$Width = 1920,
[int]$Height = 1080,
[int]$Framerate = 30,
[string]$VideoBitrate = "8000k",
[string]$AudioBitrate = "192k",
[string]$MediaToolPath = ""
)
$ErrorActionPreference = "Stop"
$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj"
$publishedTool = Join-Path $PSScriptRoot "..\artifacts\mimir-cultmesh-media-win-x64-selfcontained\Mimir.CultMeshMedia.exe"
$loopbackScript = Join-Path $PSScriptRoot "wasapi-loopback-capture.ps1"
if ([string]::IsNullOrWhiteSpace($MediaToolPath) -and (Test-Path $publishedTool)) {
$MediaToolPath = $publishedTool
}
$ffmpegArgs = @(
"-hide_banner",
"-loglevel", "warning",
"-f", "gdigrab",
"-framerate", $Framerate,
"-video_size", "${Width}x${Height}",
"-i", $VideoInput
)
if ($AudioMode -eq "WasapiLoopback") {
$ffmpegArgs += @(
"-f", "f32le",
"-ar", "48000",
"-ac", "2",
"-i", "pipe:0"
)
} else {
$ffmpegArgs += @(
"-f", "dshow",
"-i", "audio=$AudioInput"
)
}
$ffmpegArgs += @(
"-map", "0:v:0",
"-map", "1:a:0",
"-c:v", "h264_nvenc",
"-preset", "p3",
"-tune", "ll",
"-b:v", $VideoBitrate,
"-maxrate", $VideoBitrate,
"-bufsize", "16000k",
"-g", ($Framerate * 2),
"-pix_fmt", "yuv420p",
"-c:a", "aac",
"-b:a", $AudioBitrate,
"-ar", "48000",
"-ac", "2",
"-f", "mpegts",
"pipe:1"
)
$senderArgs = @("send", "--host", $RelayHost, "--port", $RelayPort, "--stream", $Stream, "--producer", "raven", "--chunk-bytes", "16384", "--slots", "96")
if ($AudioMode -eq "WasapiLoopback") {
if (-not (Test-Path $loopbackScript)) {
throw "WASAPI loopback script not found: $loopbackScript"
}
if ([string]::IsNullOrWhiteSpace($MediaToolPath)) {
& $loopbackScript -Output stdout -Role $LoopbackRole | & $FfmpegPath @ffmpegArgs | dotnet run --project $project -- @senderArgs
} else {
& $loopbackScript -Output stdout -Role $LoopbackRole | & $FfmpegPath @ffmpegArgs | & $MediaToolPath @senderArgs
}
} else {
if ([string]::IsNullOrWhiteSpace($MediaToolPath)) {
& $FfmpegPath @ffmpegArgs | dotnet run --project $project -- @senderArgs
} else {
& $FfmpegPath @ffmpegArgs | & $MediaToolPath @senderArgs
}
}

View file

@ -0,0 +1,10 @@
param(
[string]$RelayHost = "10.77.0.1",
[int]$RelayPort = 3075,
[string]$Stream = "raven-primary-av",
[string]$Udp = "127.0.0.1:5200"
)
$ErrorActionPreference = "Stop"
$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj"
dotnet run --project $project -- recv --host $RelayHost --port $RelayPort --stream $Stream --udp $Udp

View file

@ -0,0 +1,70 @@
param(
[string]$Rgb = "#35ff6c",
[int]$RefreshMs = 250,
[double]$HoldSeconds = 0,
[switch]$TurnOffOnExit,
[switch]$KeepExisting,
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
$repo = Split-Path -Parent $PSScriptRoot
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
$runDir = Join-Path $repo "artifacts\runtime\starfire-move-light-$runId"
$probe = Join-Path $repo "src\Mimir.PsMoveProbe\bin\Release\net10.0\Mimir.PsMoveProbe.exe"
$eventLog = Join-Path $runDir "move-light-events.jsonl"
New-Item -ItemType Directory -Force -Path $runDir | Out-Null
if (-not (Test-Path $probe)) {
dotnet build (Join-Path $repo "src\Mimir.PsMoveProbe\Mimir.PsMoveProbe.csproj") -c Release -p:UseSharedCompilation=false
}
if (-not $KeepExisting) {
Get-Process -Name "Mimir.PsMoveProbe" -ErrorAction SilentlyContinue |
ForEach-Object {
Stop-Process -Id $_.Id -Force
}
}
$args = @(
"--rgb", $Rgb,
"--breathe-led",
"--hold-seconds", "$HoldSeconds",
"--refresh-ms", "$RefreshMs",
"--event-log", $eventLog
)
if ($TurnOffOnExit) {
$args += "--turn-off-on-exit"
}
if ($DryRun) {
Write-Host "DRY starfire-move-light: $probe $($args -join ' ')"
return
}
$stdout = Join-Path $runDir "starfire-move-light.out.log"
$stderr = Join-Path $runDir "starfire-move-light.err.log"
$process = Start-Process `
-FilePath $probe `
-ArgumentList $args `
-WorkingDirectory $repo `
-WindowStyle Hidden `
-RedirectStandardOutput $stdout `
-RedirectStandardError $stderr `
-PassThru
Set-Content -Path (Join-Path $runDir "starfire-move-light.pid") -Value $process.Id
[ordered]@{
kind = "mimir.starfire_move_light_supervisor.v1"
runId = $runId
runDir = $runDir
rgb = $Rgb
refreshMs = $RefreshMs
holdSeconds = $HoldSeconds
pid = $process.Id
eventLog = $eventLog
startedAt = (Get-Date).ToString("o")
} | ConvertTo-Json -Depth 4 | Set-Content -Path (Join-Path $runDir "supervisor.json")
Write-Host "starfire-move-light pid=$($process.Id) runDir=$runDir eventLog=$eventLog"

View file

@ -0,0 +1,8 @@
param(
[string]$Cache = "C:\Meta\Mimir\state\mimir-cultmesh-media.cc"
)
$ErrorActionPreference = "Stop"
$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj"
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $Cache) | Out-Null
dotnet run --project $project -- relay --cache $Cache

View file

@ -10,7 +10,9 @@
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\CultLib\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<PackageReference Include="System.Drawing.Common" Version="10.0.0" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Mesh\GameCult.Mesh.csproj" />
<ProjectReference Include="..\Mimir.Runtime\Mimir.Runtime.csproj" />
</ItemGroup>

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,18 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>Mimir.CultMeshMedia</AssemblyName>
<RootNamespace>Mimir.CultMeshMedia</RootNamespace>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\Mimir.Runtime\Mimir.Runtime.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Mesh\GameCult.Mesh.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Networking\GameCult.Networking.csproj" />
</ItemGroup>
</Project>

View file

@ -0,0 +1,568 @@
using System.Diagnostics;
using System.Globalization;
using System.Net;
using System.Net.Sockets;
using GameCult.Caching;
using GameCult.Caching.MessagePack;
using GameCult.Mesh;
using GameCult.Networking;
using Mimir.Runtime.Synchronization;
const string SchemaName = "mimir.cultmesh_media_frame";
const string DefaultStreamId = "raven-primary-av";
const uint DefaultConnectionId = 0x4d4d_0101;
const int DefaultPort = 3075;
const int DefaultMaxFragmentBytes = 1200;
const int DefaultMaxPendingReliablePackets = 8192;
var command = args.Length > 0 ? args[0].Trim().ToLowerInvariant() : string.Empty;
var options = CliOptions.Parse(args.Skip(1));
return command switch
{
"relay" => await RunRelayAsync(options),
"send" => await RunSenderAsync(options),
"recv" or "receive" => await RunReceiverAsync(options),
_ => Usage()
};
static int Usage()
{
Console.Error.WriteLine("""
Usage:
Mimir.CultMeshMedia relay --cache <path> [--bind 0.0.0.0:3075]
Mimir.CultMeshMedia send --host <rudp-host> [--port 3075] [--stream raven-primary-av] [--chunk-bytes 16384] [--slots 96]
Mimir.CultMeshMedia recv --host <rudp-host> [--port 3075] [--stream raven-primary-av] [--udp 127.0.0.1:5200]
send reads an MPEG-TS byte stream from stdin and publishes rolling CultMesh media-frame documents over CultNet RUDP.
recv subscribes to those RUDP document changes and writes ordered payload bytes to a local UDP MPEG-TS endpoint for OBS.
""");
return 2;
}
static async Task<int> RunRelayAsync(CliOptions options)
{
var cachePath = options.Get("cache") ?? Path.Combine(Environment.CurrentDirectory, "mimir-cultmesh-media.cc");
var bind = ParseEndpoint(options.Get("bind") ?? $"0.0.0.0:{options.GetInt("port", DefaultPort)}");
var registry = CreateDocumentRegistry();
var schemaId = ResolveMediaSchemaId(registry);
using var cache = await CultCacheMessagePack.OpenAsync(cachePath).ConfigureAwait(false);
using var relay = new RudpMediaRelay(
bind,
cache,
registry,
schemaId,
options.GetUInt("connection-id", DefaultConnectionId),
options.GetInt("max-fragment-bytes", DefaultMaxFragmentBytes),
options.GetInt("max-pending-reliable", DefaultMaxPendingReliablePackets));
Console.WriteLine($"CultMesh media relay listening on rudp://{bind.Address}:{bind.Port}; cache={Path.GetFullPath(cachePath)}");
await relay.RunAsync().ConfigureAwait(false);
await cache.FlushAsync().ConfigureAwait(false);
return 0;
}
static async Task<int> RunSenderAsync(CliOptions options)
{
var endpoint = ResolveRudpEndpoint(options);
var streamId = options.Get("stream") ?? DefaultStreamId;
var chunkBytes = Math.Clamp(options.GetInt("chunk-bytes", 16 * 1024), 1316, 60 * 1024);
var slots = Math.Clamp(options.GetInt("slots", 96), 8, 4096);
var producer = options.Get("producer") ?? Environment.MachineName.ToLowerInvariant();
var registry = CreateDocumentRegistry();
using var transport = CreateRudpClient(
options.Get("runtime") ?? $"{producer}-mimir-media-sender",
endpoint,
options);
ConnectOrThrow(transport, endpoint.Uri);
Console.Error.WriteLine($"Sending stdin MPEG-TS to {endpoint.Uri}/{streamId}; chunkBytes={chunkBytes}; slots={slots}");
var input = Console.OpenStandardInput();
var buffer = new byte[chunkBytes];
long sequence = 0;
var started = Stopwatch.StartNew();
while (true)
{
var read = await input.ReadAsync(buffer.AsMemory(0, buffer.Length)).ConfigureAwait(false);
if (read <= 0)
{
break;
}
var payload = buffer.AsSpan(0, read).ToArray();
var document = new MimirCultMeshMediaFrameDocument(
FrameId: $"{streamId}:{sequence:D20}",
StreamId: streamId,
Sequence: sequence,
ProducedAtUtc: DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture),
TimestampNanoseconds: started.ElapsedTicks * 1_000_000_000L / Stopwatch.Frequency,
PayloadKind: "mpegts-chunk",
Container: "mpegts",
VideoCodec: options.Get("video-codec") ?? "h264",
AudioCodec: options.Get("audio-codec") ?? "aac",
PayloadBytes: payload.Length,
Payload: payload,
ProducerNode: producer,
ClockDomainId: options.Get("clock-domain") ?? "raven-monotonic",
Tags: ["raven", "program", "obs", "cultmesh-media", "cultnet.transport.rudp.v0"]);
var handle = new CultRecordHandle<MimirCultMeshMediaFrameDocument>(
new CultRecordKey($"{streamId}:slot:{sequence % slots:D4}"));
var message = registry.CreateRawDocumentPutMessage(
Guid.NewGuid().ToString("N"),
handle,
document,
new CultNetDocumentMessageOptions
{
SourceRuntimeId = producer,
SourceRole = "mimir-cultmesh-media-sender",
Tags = ["mimir", "media", streamId, "cultnet.transport.rudp.v0"]
});
transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message));
PumpClient(transport, TimeSpan.FromMilliseconds(5));
sequence++;
if (sequence % 120 == 0)
{
Console.Error.WriteLine($"sent sequence={sequence - 1}");
}
}
Console.Error.WriteLine($"sent chunks={sequence}; draining");
PumpClient(transport, TimeSpan.FromMilliseconds(options.GetInt("drain-ms", 1000)));
return 0;
}
static async Task<int> RunReceiverAsync(CliOptions options)
{
var endpoint = ResolveRudpEndpoint(options);
var streamId = options.Get("stream") ?? DefaultStreamId;
var udp = options.Get("udp") ?? "127.0.0.1:5200";
var udpEndpoint = ParseEndpoint(udp);
var registry = CreateDocumentRegistry();
var schemaId = ResolveMediaSchemaId(registry);
var lastSequence = -1L;
long frames = 0;
long bytes = 0;
var loggedFirstFrame = false;
using var socket = new UdpClient();
using var transport = CreateRudpClient(
options.Get("runtime") ?? $"{Environment.MachineName.ToLowerInvariant()}-mimir-media-receiver",
endpoint,
options);
ConnectOrThrow(transport, endpoint.Uri);
transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(new CultNetDatabaseSubscribeMessage
{
MessageId = Guid.NewGuid().ToString("N"),
SubscriptionId = $"mimir-media-{streamId}",
SchemaIds = [schemaId],
IncludeSnapshot = false
}));
Console.WriteLine($"Receiving {endpoint.Uri}/{streamId}; writing MPEG-TS UDP to udp://{udp}");
while (true)
{
transport.PollResends();
if (transport.ReceiveOnce() is not { } frame)
{
await Task.Delay(2).ConfigureAwait(false);
continue;
}
if (!string.Equals(frame.ChannelId, "schema", StringComparison.Ordinal))
{
continue;
}
if (CultNetSchemaMessageSerialization.Deserialize(frame.Payload) is not CultNetDatabaseChangeRawMessage change ||
change.Document is not { } rawDocument ||
!string.Equals(rawDocument.SchemaId, schemaId, StringComparison.Ordinal))
{
continue;
}
var document = (MimirCultMeshMediaFrameDocument)registry
.GetBySchemaId(schemaId)!
.PayloadDeserializer(rawDocument.Payload);
if (!string.Equals(document.StreamId, streamId, StringComparison.Ordinal) ||
document.Sequence <= lastSequence)
{
continue;
}
lastSequence = document.Sequence;
socket.Send(document.Payload, document.Payload.Length, udpEndpoint);
frames++;
bytes += document.Payload.Length;
if (!loggedFirstFrame)
{
loggedFirstFrame = true;
Console.WriteLine($"received first sequence={document.Sequence} bytes={document.PayloadBytes}");
}
if (frames % 120 == 0)
{
Console.WriteLine($"received sequence={document.Sequence} frames={frames} bytes={bytes}");
}
}
}
static CultMeshRudpEndpoint ResolveRudpEndpoint(CliOptions options)
{
if (options.Get("endpoint") is { } endpoint)
{
return CultMesh.ParseRudpEndpoint(endpoint);
}
var host = options.Get("host") ?? "10.77.0.1";
var port = options.GetInt("port", DefaultPort);
return CultMesh.ParseRudpEndpoint($"rudp://{host}:{port}");
}
static CultNetRudpSocketTransportConnection CreateRudpClient(
string runtimeId,
CultMeshRudpEndpoint endpoint,
CliOptions options)
{
return CultMesh.CreateRudpClient(
runtimeId,
options.GetUInt("connection-id", DefaultConnectionId),
endpoint,
new CultMeshRudpSocketOptions
{
BindHost = options.Get("bind-host") ?? "0.0.0.0",
BindPort = options.GetInt("bind-port", 0),
ResendDelayMs = options.GetInt("resend-ms", 100),
MaxFragmentBytes = options.GetInt("max-fragment-bytes", DefaultMaxFragmentBytes),
MaxPendingReliablePackets = options.GetInt("max-pending-reliable", DefaultMaxPendingReliablePackets)
});
}
static void ConnectOrThrow(CultNetRudpSocketTransportConnection transport, string endpoint)
{
if (!transport.ConnectAndWait(Array.Empty<byte>(), TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(5)))
{
throw new TimeoutException($"timed out connecting RUDP media transport to {endpoint}");
}
}
static void PumpClient(CultNetRudpSocketTransportConnection transport, TimeSpan duration)
{
var until = Stopwatch.StartNew();
while (until.Elapsed < duration)
{
transport.PollResends();
transport.ReceiveOnce();
Thread.Sleep(1);
}
}
static string ResolveMediaSchemaId(CultNetDocumentRegistry registry)
{
var binding = registry.GetByDocumentType(typeof(MimirCultMeshMediaFrameDocument)) ??
throw new InvalidOperationException($"CultNet document binding missing for {SchemaName}.");
return binding.SchemaId;
}
static CultNetDocumentRegistry CreateDocumentRegistry()
{
var documents = CultDocumentRegistry.Shared;
documents.GetRequired<MimirCultMeshMediaFrameDocument>();
return new CultNetDocumentRegistry(documents)
.Register(CultNetDocumentBinding.ForDocument<MimirCultMeshMediaFrameDocument>(documents));
}
static IPEndPoint ParseEndpoint(string value)
{
var split = value.LastIndexOf(':');
if (split <= 0 || split == value.Length - 1)
{
throw new ArgumentException($"Endpoint must be host:port, not '{value}'.");
}
var host = value[..split];
var port = int.Parse(value[(split + 1)..], CultureInfo.InvariantCulture);
var addresses = Dns.GetHostAddresses(host);
return new IPEndPoint(addresses.First(address => address.AddressFamily == AddressFamily.InterNetwork), port);
}
sealed class RudpMediaRelay : IDisposable
{
private readonly Socket socket;
private readonly CultCache cache;
private readonly CultNetDocumentRegistry registry;
private readonly string schemaId;
private readonly uint connectionId;
private readonly int maxFragmentBytes;
private readonly int maxPendingReliablePackets;
private readonly Dictionary<string, RelayPeer> peers = new(StringComparer.Ordinal);
public RudpMediaRelay(
IPEndPoint bind,
CultCache cache,
CultNetDocumentRegistry registry,
string schemaId,
uint connectionId,
int maxFragmentBytes,
int maxPendingReliablePackets)
{
this.cache = cache;
this.registry = registry;
this.schemaId = schemaId;
this.connectionId = connectionId;
this.maxFragmentBytes = maxFragmentBytes;
this.maxPendingReliablePackets = maxPendingReliablePackets;
socket = new Socket(bind.AddressFamily, SocketType.Dgram, ProtocolType.Udp);
socket.Bind(bind);
socket.Blocking = false;
}
public async Task RunAsync()
{
while (true)
{
ReceiveAvailable();
PumpReliableResends();
await Task.Delay(2).ConfigureAwait(false);
}
}
public void Dispose()
{
socket.Dispose();
}
private void ReceiveAvailable()
{
while (socket.Poll(0, SelectMode.SelectRead))
{
var buffer = new byte[65535];
EndPoint remote = new IPEndPoint(IPAddress.Any, 0);
int received;
try
{
received = socket.ReceiveFrom(buffer, ref remote);
}
catch (SocketException error) when (error.SocketErrorCode == SocketError.WouldBlock)
{
return;
}
var wire = buffer.AsSpan(0, received).ToArray();
var packet = CultNetRudpPacketCodec.Decode(wire);
if (packet.ConnectionId != connectionId)
{
continue;
}
var peer = GetPeer((IPEndPoint)remote);
if (packet.PacketType == CultNetRudpPacketType.Connect)
{
SendPacket(peer, peer.Session.AcceptConnect(packet, NowMs()));
continue;
}
var result = peer.Session.Receive(packet, NowMs());
if (result.Reply != null)
{
SendPacket(peer, result.Reply);
}
if (result.Disconnected)
{
peers.Remove(peer.Key);
continue;
}
foreach (var frame in result.Delivered.Where(static frame => string.Equals(frame.ChannelId, "schema", StringComparison.Ordinal)))
{
HandleSchemaFrameAsync(peer, frame.Payload).GetAwaiter().GetResult();
}
if (packet.PacketType == CultNetRudpPacketType.Accept || result.Delivered.Count > 0)
{
SendPacket(peer, peer.Session.CreateAck());
}
}
}
private async Task HandleSchemaFrameAsync(RelayPeer peer, byte[] payload)
{
var message = CultNetSchemaMessageSerialization.Deserialize(payload);
switch (message)
{
case CultNetDocumentPutRawMessage put:
await ApplyPutAsync(put).ConfigureAwait(false);
break;
case CultNetDatabaseSubscribeMessage subscribe:
peer.Subscriptions[subscribe.SubscriptionId] = subscribe;
Console.WriteLine($"relay RUDP subscribe peer={peer.Key} subscription={subscribe.SubscriptionId} schemas={string.Join(",", subscribe.SchemaIds ?? [])}");
break;
case CultNetDatabaseUnsubscribeMessage unsubscribe:
peer.Subscriptions.Remove(unsubscribe.SubscriptionId);
break;
}
}
private async Task ApplyPutAsync(CultNetDocumentPutRawMessage put)
{
if (put.Document == null ||
!string.Equals(put.Document.SchemaId, schemaId, StringComparison.Ordinal))
{
return;
}
await registry.ApplyRawDocumentPutMessageAsync(cache, put).ConfigureAwait(false);
await cache.FlushAsync().ConfigureAwait(false);
var change = new CultNetDatabaseChangeRawMessage
{
MessageId = Guid.NewGuid().ToString("N"),
ChangeKind = "updated",
Document = put.Document
};
var frame = CultNetSchemaMessageSerialization.Serialize(change);
foreach (var peer in peers.Values)
{
foreach (var subscription in peer.Subscriptions.Values)
{
if (!Matches(subscription, put.Document.SchemaId, put.Document.RecordKey))
{
continue;
}
change.SubscriptionId = subscription.SubscriptionId;
SendSchema(peer, CultNetSchemaMessageSerialization.Serialize(change));
if (!peer.HasForwardedFrame)
{
peer.HasForwardedFrame = true;
Console.WriteLine($"relay RUDP forward peer={peer.Key} subscription={subscription.SubscriptionId} schema={put.Document.SchemaId}");
}
ReceiveAvailable();
}
}
var document = (MimirCultMeshMediaFrameDocument)registry
.GetBySchemaId(schemaId)!
.PayloadDeserializer(put.Document.Payload);
if (document.Sequence % 120 == 0)
{
Console.WriteLine($"relay RUDP put stream={document.StreamId} sequence={document.Sequence} bytes={document.PayloadBytes}");
}
}
private static bool Matches(CultNetDatabaseSubscribeMessage request, string schemaId, string recordKey)
{
var schemaMatches = request.SchemaIds == null ||
request.SchemaIds.Length == 0 ||
request.SchemaIds.Contains(schemaId, StringComparer.Ordinal);
var keyMatches = request.RecordKeys == null ||
request.RecordKeys.Length == 0 ||
request.RecordKeys.Contains(recordKey, StringComparer.Ordinal);
return schemaMatches && keyMatches;
}
private RelayPeer GetPeer(IPEndPoint endpoint)
{
var key = endpoint.ToString();
if (peers.TryGetValue(key, out var existing))
{
return existing;
}
var peer = new RelayPeer(
key,
endpoint,
new CultNetRudpSession(new CultNetRudpSessionOptions
{
ConnectionId = connectionId,
InitialSequence = (uint)(peers.Count + 1),
ResendDelayMs = 100,
MaxPendingReliablePackets = maxPendingReliablePackets
}));
peers[key] = peer;
return peer;
}
private void SendSchema(RelayPeer peer, byte[] payload)
{
foreach (var packet in peer.Session.SendMany(
"schema",
payload,
new CultNetRudpSendOptions { Reliable = true, Ordered = true, NowMs = NowMs() },
maxFragmentBytes))
{
SendPacket(peer, packet);
}
}
private void SendPacket(RelayPeer peer, CultNetRudpPacket packet)
{
var wire = CultNetRudpPacketCodec.Encode(packet);
socket.SendTo(wire, peer.Endpoint);
}
private void PumpReliableResends()
{
foreach (var peer in peers.Values)
{
foreach (var packet in peer.Session.DueResends(NowMs()))
{
SendPacket(peer, packet);
}
}
}
private static long NowMs() => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
}
sealed record RelayPeer(
string Key,
IPEndPoint Endpoint,
CultNetRudpSession Session)
{
public Dictionary<string, CultNetDatabaseSubscribeMessage> Subscriptions { get; } = new(StringComparer.Ordinal);
public bool HasForwardedFrame { get; set; }
}
sealed class CliOptions
{
private readonly Dictionary<string, string> values = new(StringComparer.OrdinalIgnoreCase);
public static CliOptions Parse(IEnumerable<string> args)
{
var parsed = new CliOptions();
var pendingKey = string.Empty;
foreach (var arg in args)
{
if (arg.StartsWith("--", StringComparison.Ordinal))
{
pendingKey = arg[2..];
parsed.values[pendingKey] = "true";
continue;
}
if (!string.IsNullOrWhiteSpace(pendingKey))
{
parsed.values[pendingKey] = arg;
pendingKey = string.Empty;
}
}
return parsed;
}
public string? Get(string key) => values.TryGetValue(key, out var value) ? value : null;
public int GetInt(string key, int fallback) =>
int.TryParse(Get(key), NumberStyles.Integer, CultureInfo.InvariantCulture, out var value) ? value : fallback;
public uint GetUInt(string key, uint fallback) =>
uint.TryParse(Get(key), NumberStyles.Integer, CultureInfo.InvariantCulture, out var value) ? value : fallback;
}

View file

@ -0,0 +1,17 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>Mimir.EveSensorReceiver</AssemblyName>
<RootNamespace>Mimir.EveSensorReceiver</RootNamespace>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="..\Mimir.Runtime\Mimir.Runtime.csproj" />
</ItemGroup>
</Project>

View file

@ -0,0 +1,501 @@
using System.Net;
using System.Net.Sockets;
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using MessagePack;
using Mimir.Runtime.Synchronization;
var options = EveSensorReceiverOptions.Parse(args);
using var receiver = new EveSensorReceiver(options);
await receiver.RunAsync();
internal sealed record EveSensorReceiverOptions(
int Port,
string Path,
string SourceId,
string ExpectedType,
string SubscribePath)
{
public static EveSensorReceiverOptions Parse(string[] args)
{
return new EveSensorReceiverOptions(
ParseInt(args, "--port", 8793),
ParseString(args, "--path", "/eve/camera"),
ParseString(args, "--source-id", "eve-camera"),
ParseString(args, "--type", "video-frame"),
ParseString(args, "--subscribe-path", ParseString(args, "--path", "/eve/camera") + "/subscribe"));
}
private static string ParseString(IReadOnlyList<string> args, string name, string fallback)
{
for (var index = 0; index < args.Count - 1; index++)
{
if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase))
{
return args[index + 1];
}
}
return fallback;
}
private static int ParseInt(IReadOnlyList<string> args, string name, int fallback)
{
return int.TryParse(ParseString(args, name, ""), out var value) ? value : fallback;
}
}
internal sealed class EveSensorReceiver(EveSensorReceiverOptions options) : IDisposable
{
private readonly TcpListener listener = new(IPAddress.Any, options.Port);
private readonly CancellationTokenSource stopping = new();
private readonly ConcurrentDictionary<int, WebSocketSubscriber> subscribers = new();
private int nextSubscriberId;
public async Task RunAsync()
{
listener.Start();
Console.Error.WriteLine($"Eve sensor receiver listening on ws://0.0.0.0:{options.Port}{options.Path} source={options.SourceId}");
while (!stopping.IsCancellationRequested)
{
var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false);
_ = Task.Run(() => HandleAsync(client));
}
}
private async Task HandleAsync(TcpClient tcpClient)
{
await using var stream = tcpClient.GetStream();
var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false);
if (request.Path == "/health")
{
var health = JsonSerializer.Serialize(new
{
ok = true,
options.SourceId,
options.ExpectedType,
options.Path,
options.SubscribePath,
Subscribers = subscribers.Count,
});
await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false);
return;
}
if (string.Equals(request.Path, options.SubscribePath, StringComparison.Ordinal)
&& request.Headers.TryGetValue("Sec-WebSocket-Key", out var subscriberKey))
{
await WriteWebSocketHandshakeAsync(stream, subscriberKey).ConfigureAwait(false);
await SubscribeAsync(stream).ConfigureAwait(false);
return;
}
if (!string.Equals(request.Path, options.Path, StringComparison.Ordinal)
|| !request.Headers.TryGetValue("Sec-WebSocket-Key", out var key))
{
await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false);
return;
}
await WriteWebSocketHandshakeAsync(stream, key).ConfigureAwait(false);
while (!stopping.IsCancellationRequested)
{
var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false);
if (frame.Opcode == 0x8)
{
await WriteCloseFrameAsync(stream).ConfigureAwait(false);
break;
}
if (frame.Opcode == 0x2)
{
if (TryValidateCultMeshObservation(frame.Payload, out var normalizedObservation)
|| TryValidateCultMeshMediaObservation(frame.Payload, out normalizedObservation))
{
Console.WriteLine(normalizedObservation);
Console.Out.Flush();
await BroadcastAsync(normalizedObservation).ConfigureAwait(false);
}
continue;
}
if (frame.Opcode != 0x1)
{
continue;
}
var text = Encoding.UTF8.GetString(frame.Payload);
if (TryValidateFrameEvent(text, out var normalized))
{
Console.WriteLine(normalized);
Console.Out.Flush();
await BroadcastAsync(normalized).ConfigureAwait(false);
}
}
}
private async Task SubscribeAsync(NetworkStream stream)
{
var id = Interlocked.Increment(ref nextSubscriberId);
var subscriber = new WebSocketSubscriber(id, stream);
subscribers[id] = subscriber;
Console.Error.WriteLine($"Eve sensor receiver subscriber {id} connected on {options.SubscribePath}.");
try
{
await subscriber.SendTextAsync(JsonSerializer.Serialize(new
{
type = "mimir-verse-subscription",
status = "connected",
options.Path,
options.SubscribePath,
options.SourceId,
options.ExpectedType,
})).ConfigureAwait(false);
while (!stopping.IsCancellationRequested)
{
var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false);
if (frame.Opcode == 0x8)
{
await WriteCloseFrameAsync(stream).ConfigureAwait(false);
break;
}
}
}
catch (IOException)
{
}
catch (SocketException)
{
}
finally
{
subscribers.TryRemove(id, out _);
Console.Error.WriteLine($"Eve sensor receiver subscriber {id} disconnected.");
}
}
private async Task BroadcastAsync(string normalized)
{
foreach (var pair in subscribers.ToArray())
{
try
{
await pair.Value.SendTextAsync(normalized).ConfigureAwait(false);
}
catch (IOException)
{
subscribers.TryRemove(pair.Key, out _);
}
catch (SocketException)
{
subscribers.TryRemove(pair.Key, out _);
}
catch (ObjectDisposedException)
{
subscribers.TryRemove(pair.Key, out _);
}
}
}
private bool TryValidateFrameEvent(string text, out string normalized)
{
normalized = "";
try
{
using var document = JsonDocument.Parse(text);
var root = document.RootElement;
if (!root.TryGetProperty("type", out var type)
|| !string.Equals(type.GetString(), options.ExpectedType, StringComparison.OrdinalIgnoreCase))
{
return false;
}
if (!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase)
&& (!root.TryGetProperty("sourceId", out var sourceId)
|| !string.Equals(sourceId.GetString(), options.SourceId, StringComparison.Ordinal)))
{
return false;
}
normalized = root.GetRawText();
return true;
}
catch (JsonException)
{
return false;
}
}
private bool TryValidateCultMeshObservation(byte[] payload, out string normalized)
{
normalized = "";
try
{
var observation = MessagePackSerializer.Deserialize<MimirEveSensorObservationDocument>(payload);
if (!string.Equals(observation.StreamId, options.SourceId, StringComparison.Ordinal) &&
!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase))
{
return false;
}
normalized = JsonSerializer.Serialize(new
{
type = "cultmesh-observation",
document = "mimir.eve_sensor_observation.v1",
observation.ObservationId,
observation.DeviceId,
observation.StreamId,
observation.Kind,
observation.Sequence,
observation.SensorTimestampNs,
observation.ElapsedRealtimeNs,
observation.WallClockUtc,
observation.ClockDomainId,
observation.Values,
observation.Action,
observation.PointerCount,
observation.X,
observation.Y,
observation.Accuracy,
});
return true;
}
catch
{
return false;
}
}
private bool TryValidateCultMeshMediaObservation(byte[] payload, out string normalized)
{
normalized = "";
try
{
var observation = MessagePackSerializer.Deserialize<MimirEveMediaObservationDocument>(payload);
if (!string.Equals(observation.StreamId, options.SourceId, StringComparison.Ordinal) &&
!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase))
{
return false;
}
normalized = JsonSerializer.Serialize(new
{
type = "cultmesh-media-observation",
document = "mimir.eve_media_observation.v1",
observation.ObservationId,
observation.DeviceId,
observation.StreamId,
observation.Kind,
observation.Sequence,
observation.SensorTimestampNs,
observation.ElapsedRealtimeNs,
observation.WallClockUtc,
observation.ClockDomainId,
observation.Format,
observation.Width,
observation.Height,
observation.SampleRate,
observation.Channels,
observation.FrameCount,
observation.PayloadEncoding,
PayloadBytes = observation.Payload.Length,
});
return true;
}
catch
{
return false;
}
}
private static async Task<HttpRequest> ReadHttpRequestAsync(NetworkStream stream)
{
var bytes = new List<byte>();
var lastFour = new Queue<byte>(4);
while (true)
{
var value = stream.ReadByte();
if (value < 0)
{
throw new IOException("Client closed before HTTP headers completed.");
}
var b = (byte)value;
bytes.Add(b);
lastFour.Enqueue(b);
if (lastFour.Count > 4)
{
lastFour.Dequeue();
}
if (lastFour.Count == 4 && lastFour.SequenceEqual(new byte[] { 13, 10, 13, 10 }))
{
break;
}
}
var text = Encoding.ASCII.GetString(bytes.ToArray());
var lines = text.Split("\r\n", StringSplitOptions.None);
var first = lines[0].Split(' ');
var headers = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
foreach (var line in lines.Skip(1))
{
var colon = line.IndexOf(':');
if (colon > 0)
{
headers[line[..colon].Trim()] = line[(colon + 1)..].Trim();
}
}
await Task.CompletedTask.ConfigureAwait(false);
return new HttpRequest(first.Length > 1 ? first[1] : "/", headers);
}
private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body)
{
var header = Encoding.ASCII.GetBytes(
$"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n");
await stream.WriteAsync(header).ConfigureAwait(false);
await stream.WriteAsync(body).ConfigureAwait(false);
}
private static async Task WriteWebSocketHandshakeAsync(NetworkStream stream, string key)
{
var acceptBytes = SHA1.HashData(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"));
var accept = Convert.ToBase64String(acceptBytes);
var response = Encoding.ASCII.GetBytes(
"HTTP/1.1 101 Switching Protocols\r\n" +
"Upgrade: websocket\r\n" +
"Connection: Upgrade\r\n" +
$"Sec-WebSocket-Accept: {accept}\r\n\r\n");
await stream.WriteAsync(response).ConfigureAwait(false);
}
private static async Task<WebSocketFrame> ReceiveFrameAsync(NetworkStream stream)
{
var header = new byte[2];
await ReadExactAsync(stream, header).ConfigureAwait(false);
var opcode = header[0] & 0x0f;
var masked = (header[1] & 0x80) != 0;
var length = header[1] & 0x7f;
if (length == 126)
{
var extended = new byte[2];
await ReadExactAsync(stream, extended).ConfigureAwait(false);
length = (extended[0] << 8) | extended[1];
}
else if (length == 127)
{
var extended = new byte[8];
await ReadExactAsync(stream, extended).ConfigureAwait(false);
var longLength = 0L;
foreach (var b in extended)
{
longLength = (longLength << 8) | b;
}
if (longLength > int.MaxValue)
{
throw new InvalidDataException("WebSocket frame is too large.");
}
length = (int)longLength;
}
var mask = new byte[4];
if (masked)
{
await ReadExactAsync(stream, mask).ConfigureAwait(false);
}
var payload = new byte[length];
await ReadExactAsync(stream, payload).ConfigureAwait(false);
if (masked)
{
for (var index = 0; index < payload.Length; index++)
{
payload[index] ^= mask[index % 4];
}
}
return new WebSocketFrame(opcode, payload);
}
private static Task WriteCloseFrameAsync(NetworkStream stream)
{
return stream.WriteAsync(new byte[] { 0x88, 0x00 }).AsTask();
}
private static async Task ReadExactAsync(NetworkStream stream, byte[] buffer)
{
var offset = 0;
while (offset < buffer.Length)
{
var read = await stream.ReadAsync(buffer.AsMemory(offset, buffer.Length - offset)).ConfigureAwait(false);
if (read == 0)
{
throw new IOException("WebSocket closed.");
}
offset += read;
}
}
public void Dispose()
{
stopping.Cancel();
listener.Stop();
stopping.Dispose();
}
}
internal sealed record HttpRequest(string Path, IReadOnlyDictionary<string, string> Headers);
internal sealed record WebSocketFrame(int Opcode, byte[] Payload);
internal sealed class WebSocketSubscriber(int id, NetworkStream stream)
{
private readonly SemaphoreSlim writeLock = new(1, 1);
public int Id => id;
public async Task SendTextAsync(string text)
{
var payload = Encoding.UTF8.GetBytes(text);
var header = new List<byte> { 0x81 };
if (payload.Length < 126)
{
header.Add((byte)payload.Length);
}
else if (payload.Length <= ushort.MaxValue)
{
header.Add(126);
header.Add((byte)(payload.Length >> 8));
header.Add((byte)(payload.Length & 0xff));
}
else
{
header.Add(127);
for (var shift = 56; shift >= 0; shift -= 8)
{
header.Add((byte)(((ulong)payload.Length >> shift) & 0xff));
}
}
await writeLock.WaitAsync().ConfigureAwait(false);
try
{
await stream.WriteAsync(header.ToArray()).ConfigureAwait(false);
await stream.WriteAsync(payload).ConfigureAwait(false);
await stream.FlushAsync().ConfigureAwait(false);
}
finally
{
writeLock.Release();
}
}
}

View file

@ -0,0 +1,10 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>

File diff suppressed because it is too large Load diff

View file

@ -12,7 +12,9 @@
<ItemGroup>
<ProjectReference Include="..\..\..\Fensalir\src\Aquarium.Engine.Contracts\Aquarium.Engine.Contracts.csproj" />
<ProjectReference Include="..\..\..\CultLib\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Mesh\GameCult.Mesh.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Networking\GameCult.Networking.csproj" />
</ItemGroup>
</Project>

View file

@ -39,6 +39,8 @@ direct
private readonly MimirSynchronizationHub synchronization;
private readonly MimirAudioSpectrumAnalyzer spectrumAnalyzer;
private readonly IReadOnlyList<MimirStreamSourceFactory> sourceFactories;
private readonly IReadOnlyList<MimirMoveProofRuntimeConfiguration> configuredMoveProofSources;
private readonly IMimirMoveProofEvidenceRingProvider moveProofRingProvider;
private readonly AquariumUiDocument ui;
private readonly AquariumAudioDocument audio = new();
private readonly MimirAudioSynchronizationSettings audioSyncSettings;
@ -62,6 +64,11 @@ direct
private IReadOnlyList<MimirAudioSynchronizationReport> lastAudioSynchronizationReports = [];
private IReadOnlyList<MimirAudioSpectrumSnapshot> lastAudioSpectra = [];
private readonly Queue<IReadOnlyList<MimirAudioSpectrumSnapshot>> spectrumHistory = new();
private readonly List<MimirMoveProofRuntimeDriver> moveProofDrivers = [];
private readonly List<IDisposable> moveProofDriverResources = [];
private readonly List<MimirMoveProofRuntimeActivationStatus> moveProofActivationStatuses = [];
private MimirMoveProofSurfaceDocument? latestMoveProofSurface;
private bool moveProofSourcesActivated;
public MimirRuntime(AquariumRuntimeOptions options)
: this(options, MimirRuntimeConfiguration.Load())
@ -69,7 +76,20 @@ direct
}
public MimirRuntime(AquariumRuntimeOptions options, MimirRuntimeConfiguration configuration)
: this(options, configuration.Settings, configuration.SourceFactories)
: this(
options,
configuration.Settings,
configuration.SourceFactories,
configuration.MoveProofSources,
MimirConfiguredMoveProofEvidenceRingProvider.Instance)
{
}
public MimirRuntime(
AquariumRuntimeOptions options,
MimirRuntimeConfiguration configuration,
IMimirMoveProofEvidenceRingProvider moveProofRingProvider)
: this(options, configuration.Settings, configuration.SourceFactories, configuration.MoveProofSources, moveProofRingProvider)
{
}
@ -82,7 +102,12 @@ direct
AquariumRuntimeOptions options,
MimirSynchronizationSettings settings,
IEnumerable<IMimirStreamSource> streamSources)
: this(options, settings, Array.Empty<MimirStreamSourceFactory>())
: this(
options,
settings,
Array.Empty<MimirStreamSourceFactory>(),
Array.Empty<MimirMoveProofRuntimeConfiguration>(),
MimirUnavailableMoveProofEvidenceRingProvider.Instance)
{
foreach (var source in streamSources)
{
@ -93,12 +118,16 @@ direct
private MimirRuntime(
AquariumRuntimeOptions options,
MimirSynchronizationSettings settings,
IEnumerable<MimirStreamSourceFactory> sourceFactories)
IEnumerable<MimirStreamSourceFactory> sourceFactories,
IEnumerable<MimirMoveProofRuntimeConfiguration> moveProofSources,
IMimirMoveProofEvidenceRingProvider moveProofRingProvider)
{
Options = options;
synchronization = new MimirSynchronizationHub(settings);
spectrumAnalyzer = new MimirAudioSpectrumAnalyzer(ParseSpectrumFftSize());
this.sourceFactories = sourceFactories.ToArray();
configuredMoveProofSources = moveProofSources.ToArray();
this.moveProofRingProvider = moveProofRingProvider ?? throw new ArgumentNullException(nameof(moveProofRingProvider));
audioSyncSettings = settings.Audio;
telemetryIntervalSeconds = ParseTelemetryIntervalSeconds();
audioSyncUpdateIntervalSeconds = ParseAudioSyncIntervalSeconds();
@ -113,6 +142,11 @@ direct
: null;
nextAudioSyncSeconds = audioSyncUpdateIntervalSeconds;
nextTelemetrySeconds = telemetryIntervalSeconds;
if (MimirMoveProofDevSurface.IsEnabled())
{
PublishMoveProofSurface(MimirMoveProofDevSurface.Create());
}
ui = CreateUi();
}
@ -128,6 +162,20 @@ direct
public AquariumAudioDocument Audio => audio;
public IReadOnlyList<MimirMoveProofRuntimeActivationStatus> MoveProofActivationStatuses => moveProofActivationStatuses;
public MimirMoveProofSurfaceDocument? LatestMoveProofSurface => latestMoveProofSurface;
public IReadOnlyList<MimirMoveProofRuntimeActivationDocument> CreateMoveProofRuntimeActivationDocuments(ulong observedAtNs) =>
configuredMoveProofSources
.Select(configuration => MimirMoveProofRuntimeActivation.CreateDocument(
configuration,
moveProofActivationStatuses.LastOrDefault(status =>
string.Equals(status.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)),
latestMoveProofSurface,
observedAtNs))
.ToArray();
private AquariumFrame CreateFrame()
{
var channelCount = Math.Max(1, lastAudioSpectra.Count);
@ -144,6 +192,9 @@ direct
: bufferFieldFrame.HasInput
? new Vector3(0.0f, 0.0f, 12.0f)
: new Vector3(0.0f, 0.0f, cameraZ + 12.0f);
var spectrumSplineFrame = ShouldRenderSpectrumPreviewSplines()
? BuildSpectrumSplineFrame()
: AquariumSplineFrame.Empty;
return new AquariumFrame(
new ViewFrame(Vector2.Zero, 24.0f),
cameraPosition,
@ -155,7 +206,7 @@ direct
TraceHeightFieldSurface = false,
UseStarfieldBackground = false,
BufferFieldFrame = bufferFieldFrame,
SplineFrame = ShouldRenderSpectrumPreviewSplines() ? BuildSpectrumSplineFrame() : AquariumSplineFrame.Empty,
SplineFrame = ComposeFensalirSplineFrame(spectrumSplineFrame, latestMoveProofSurface),
});
}
@ -166,6 +217,64 @@ direct
synchronization.AddSource(source);
}
public void PublishMoveProofSurface(MimirMoveProofSurfaceDocument surface)
{
ArgumentNullException.ThrowIfNull(surface);
latestMoveProofSurface = surface;
}
public void ClearMoveProofSurface()
{
latestMoveProofSurface = null;
}
public void RegisterMoveProofDriver(MimirMoveProofRuntimeDriver driver)
{
ArgumentNullException.ThrowIfNull(driver);
moveProofDrivers.Add(driver);
}
public void ActivateConfiguredMoveProofSources()
{
if (moveProofSourcesActivated)
{
return;
}
moveProofSourcesActivated = true;
foreach (var configuration in configuredMoveProofSources)
{
ActivateConfiguredMoveProofSource(configuration);
}
}
public static AquariumSplineFrame ComposeFensalirSplineFrame(
AquariumSplineFrame baseFrame,
MimirMoveProofSurfaceDocument? moveProofSurface)
{
ArgumentNullException.ThrowIfNull(baseFrame);
if (moveProofSurface is null)
{
return baseFrame;
}
var proofFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(moveProofSurface);
if (!baseFrame.HasInput)
{
return proofFrame;
}
if (!proofFrame.HasInput)
{
return baseFrame;
}
return new AquariumSplineFrame
{
Splines = baseFrame.Splines.Concat(proofFrame.Splines).ToArray()
};
}
public void Start()
{
Console.WriteLine($"Mimir runtime sync buffers: {synchronization.Summary()} @ {synchronization.Settings.BufferDuration.TotalSeconds:0.###}s audioSync={audioSyncSettings.Mode} reference={audioSyncSettings.ReferenceSourceId}");
@ -181,6 +290,7 @@ direct
runtimeSeconds += Math.Max(deltaSeconds, 0.0f);
lastPollCount = synchronization.PollSources();
UpdateMoveProofSurfaces();
QueueCalibrationTimeline();
UpdateAudioSpectra();
UpdateAudioSynchronization();
@ -212,6 +322,11 @@ direct
public void Dispose()
{
foreach (var resource in moveProofDriverResources.AsEnumerable().Reverse())
{
resource.Dispose();
}
synchronization.Dispose();
}
@ -231,6 +346,10 @@ direct
panel.Readout("Audio sync state", DescribeAudioSyncState);
panel.Readout("Aligned audio", DescribeAlignedAudio);
panel.Readout("Chirplet reference", DescribeChirpletReference);
panel.Section("Move Proof");
panel.Readout("Latest proof", DescribeMoveProofSurface);
panel.Readout("Configured proof", DescribeConfiguredMoveProofSources);
panel.Readout("Proof drivers", DescribeMoveProofDrivers);
panel.Section("Live FFT");
panel.Readout("Spectrum cadence", () => sceneReady
? $"{lastAudioSpectra.Count} spectra fft={lastAudioSpectra.FirstOrDefault()?.FftSize ?? 0} analyze={lastSpectrumAnalysisMilliseconds:0.00}ms"
@ -246,6 +365,77 @@ direct
});
}
private string DescribeMoveProofSurface()
{
var surface = latestMoveProofSurface;
if (surface is null)
{
return "No Move proof surface published.";
}
return $"{surface.ProofId} verdict={surface.Verdict} confidence={surface.PoseConfidence:0.000} latency={surface.LatencyMilliseconds:0.00}ms chain={surface.MuninnEvidenceFrameId}->{surface.MimirEvidenceFrameId}->{surface.MimirPoseFrameId}->{surface.FensalirFrameId}";
}
private string DescribeMoveProofDrivers()
{
if (moveProofDrivers.Count == 0)
{
return "No Move proof runtime drivers registered.";
}
return string.Join(Environment.NewLine, moveProofDrivers.Select(driver => driver.Describe()));
}
private string DescribeConfiguredMoveProofSources()
{
if (configuredMoveProofSources.Count == 0)
{
return "No Move proof sources configured.";
}
return string.Join(
Environment.NewLine,
configuredMoveProofSources.Select(source =>
$"{source.EvidenceStreamId} -> {source.MimirPoseFramePrefix}:<sequence> cameras={source.Calibration.Cameras.Count} {DescribeMoveProofActivation(source.EvidenceStreamId)}"));
}
private string DescribeMoveProofActivation(string evidenceStreamId)
{
var status = moveProofActivationStatuses.LastOrDefault(status =>
string.Equals(status.EvidenceStreamId, evidenceStreamId, StringComparison.Ordinal));
if (status is null)
{
return "not activated";
}
return status.Active ? "active" : $"inactive: {status.Diagnostic}";
}
private void UpdateMoveProofSurfaces()
{
if (moveProofDrivers.Count == 0)
{
return;
}
var presentedAtNs = NowUnixNs();
foreach (var driver in moveProofDrivers)
{
if (driver.TryBuildLatest(presentedAtNs, out var result) && result is not null)
{
PublishMoveProofSurface(result.ProofSurface);
}
}
}
private static ulong NowUnixNs()
{
var now = DateTimeOffset.UtcNow;
var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
private static AquariumRenderPlan CreateRenderPlan()
{
var app = new AquariumApp();
@ -437,6 +627,65 @@ direct
synchronization.AddSource(source);
}
}
ActivateConfiguredMoveProofSources();
}
private void ActivateConfiguredMoveProofSource(MimirMoveProofRuntimeConfiguration configuration)
{
var providerKind = moveProofRingProvider.GetType().Name;
try
{
var errors = configuration.Validate();
if (errors.Length > 0)
{
RecordMoveProofActivation(configuration.EvidenceStreamId, false, $"invalid: {string.Join("; ", errors)}", providerKind, driverRegistered: false);
return;
}
if (!moveProofRingProvider.TryOpenEvidenceRing(configuration, out var ringLease, out var diagnostic) || ringLease is null)
{
RecordMoveProofActivation(configuration.EvidenceStreamId, false, diagnostic, providerKind, driverRegistered: false);
return;
}
MimirNativeReservoirRuntime? reservoir = null;
try
{
reservoir = new MimirNativeReservoirRuntime(configuration.NativeReservoirPath);
var driver = configuration.CreateDriver(ringLease.Ring, reservoir);
moveProofDrivers.Add(driver);
moveProofDriverResources.Add(reservoir);
moveProofDriverResources.Add(ringLease);
reservoir = null;
ringLease = null;
RecordMoveProofActivation(configuration.EvidenceStreamId, true, diagnostic, providerKind, driverRegistered: true);
}
finally
{
reservoir?.Dispose();
ringLease?.Dispose();
}
}
catch (Exception exception)
{
RecordMoveProofActivation(configuration.EvidenceStreamId, false, exception.Message, providerKind, driverRegistered: false);
}
}
private void RecordMoveProofActivation(
string evidenceStreamId,
bool active,
string diagnostic,
string providerKind,
bool driverRegistered)
{
moveProofActivationStatuses.Add(new MimirMoveProofRuntimeActivationStatus(
evidenceStreamId,
active,
string.IsNullOrWhiteSpace(diagnostic) ? (active ? "activated" : "not activated") : diagnostic,
providerKind,
driverRegistered));
}
private void EmitTelemetry()
@ -862,6 +1111,11 @@ direct
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {block.Channels}ch {block.SampleRate}Hz {block.SampleFormat} frames {block.FrameCount} bytes {latest.Value.ByteLength} edge {buffer.EdgeNs}";
}
if (latest?.TrackingObservation is { } tracking)
{
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {tracking.Kind} device={tracking.DeviceId} producer={tracking.ProducerId} host={tracking.HostId} confidence={tracking.Confidence:0.00} edge {buffer.EdgeNs}";
}
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} edge {buffer.EdgeNs}";
}

View file

@ -11,7 +11,8 @@ public enum MimirWorkloadKind
CameraCapture,
VisualFeatureExtraction,
VisualFusion,
ObsPublication
ProgramComposition,
ProgramPublication
}
public sealed record MimirWorkloadPlacement(
@ -42,7 +43,8 @@ public static class MimirComputeOffloadConfigurations
new(MimirWorkloadKind.CameraCapture, MimirComputeBackend.NativeSimd, "native capture workers", CanRunRemote: true, RequiresRealtime: true, "Drivers own device reads and timestamps."),
new(MimirWorkloadKind.VisualFeatureExtraction, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "GPU owns dense feature work."),
new(MimirWorkloadKind.VisualFusion, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "Fensalir owns temporal evidence field."),
new(MimirWorkloadKind.ObsPublication, MimirComputeBackend.D3D12Compute, "OBS/Fensalir output", CanRunRemote: false, RequiresRealtime: true, "OBS receives program surfaces, not sync authority.")
new(MimirWorkloadKind.ProgramComposition, MimirComputeBackend.D3D12Compute, "Mimir + Fensalir", CanRunRemote: false, RequiresRealtime: true, "Mimir owns the scene graph; Fensalir lowers it into program pixels."),
new(MimirWorkloadKind.ProgramPublication, MimirComputeBackend.D3D12Compute, "Mimir + Yggdrasil publisher", CanRunRemote: false, RequiresRealtime: true, "The site publisher consumes the Mimir program output; OBS is only an optional compatibility sink.")
]);
public static MimirComputeOffloadConfiguration DistributedWitnesses { get; } = new(

View file

@ -104,6 +104,103 @@ public sealed record MimirFaustControlSnapshot(
[property: Key(0)] string Path,
[property: Key(1)] float Value);
[CultDocument("mimir.program_scene", "mimir.program_scene.v1")]
[MessagePackObject]
public sealed record MimirProgramSceneDocument(
[property: Key(0)]
[property: CultName]
string SceneId,
[property: Key(1)] string UpdatedAtUtc,
[property: Key(2)] int CanvasWidth,
[property: Key(3)] int CanvasHeight,
[property: Key(4)] string Owner,
[property: Key(5)] MimirProgramSceneLayer[] Layers);
[MessagePackObject]
public sealed record MimirProgramSceneLayer(
[property: Key(0)] string LayerId,
[property: Key(1)] string SourceRef,
[property: Key(2)] string SourceKind,
[property: Key(3)] bool Visible,
[property: Key(4)] double X,
[property: Key(5)] double Y,
[property: Key(6)] double Width,
[property: Key(7)] double Height,
[property: Key(8)] int ZIndex,
[property: Key(9)] MimirProgramCrop Crop,
[property: Key(10)] MimirProgramChromaKey? ChromaKey);
[MessagePackObject]
public sealed record MimirProgramCrop(
[property: Key(0)] double Left,
[property: Key(1)] double Top,
[property: Key(2)] double Right,
[property: Key(3)] double Bottom);
[MessagePackObject]
public sealed record MimirProgramChromaKey(
[property: Key(0)] uint KeyColorRgba,
[property: Key(1)] double Similarity,
[property: Key(2)] double Smoothness,
[property: Key(3)] double Spill);
[CultDocument("mimir.program_output", "mimir.program_output.v1")]
[MessagePackObject]
public sealed record MimirProgramOutputDocument(
[property: Key(0)]
[property: CultName]
string OutputId,
[property: Key(1)] string UpdatedAtUtc,
[property: Key(2)] string SceneId,
[property: Key(3)] string VideoSurface,
[property: Key(4)] string AudioBus,
[property: Key(5)] string PublisherRoute,
[property: Key(6)] bool DiagnosticOnly);
[CultDocument("mimir.eve_sensor_observation", "mimir.eve_sensor_observation.v1")]
[MessagePackObject]
public sealed record MimirEveSensorObservationDocument(
[property: Key(0)]
[property: CultName]
string ObservationId,
[property: Key(1)] string DeviceId,
[property: Key(2)] string StreamId,
[property: Key(3)] string Kind,
[property: Key(4)] long Sequence,
[property: Key(5)] long SensorTimestampNs,
[property: Key(6)] long ElapsedRealtimeNs,
[property: Key(7)] string WallClockUtc,
[property: Key(8)] string ClockDomainId,
[property: Key(9)] double[] Values,
[property: Key(10)] string? Action,
[property: Key(11)] int? PointerCount,
[property: Key(12)] double? X,
[property: Key(13)] double? Y,
[property: Key(14)] int? Accuracy);
[CultDocument("mimir.eve_media_observation", "mimir.eve_media_observation.v1")]
[MessagePackObject]
public sealed record MimirEveMediaObservationDocument(
[property: Key(0)]
[property: CultName]
string ObservationId,
[property: Key(1)] string DeviceId,
[property: Key(2)] string StreamId,
[property: Key(3)] string Kind,
[property: Key(4)] long Sequence,
[property: Key(5)] long SensorTimestampNs,
[property: Key(6)] long ElapsedRealtimeNs,
[property: Key(7)] string WallClockUtc,
[property: Key(8)] string ClockDomainId,
[property: Key(9)] string Format,
[property: Key(10)] int? Width,
[property: Key(11)] int? Height,
[property: Key(12)] int? SampleRate,
[property: Key(13)] int? Channels,
[property: Key(14)] int? FrameCount,
[property: Key(15)] string PayloadEncoding,
[property: Key(16)] byte[] Payload);
public static class MimirCultMeshContractFactory
{
public static MimirBioacousticCodebookState CreateCodebookState(
@ -175,6 +272,69 @@ public static class MimirCultMeshContractFactory
.Select(pair => new MimirFaustControlSnapshot(pair.Key, pair.Value))
.ToArray());
public static MimirProgramSceneDocument CreateObsSceneMirror(
string sceneId,
DateTimeOffset? updatedAt = null) =>
new(
sceneId,
(updatedAt ?? DateTimeOffset.UtcNow).ToString("O"),
CanvasWidth: 1920,
CanvasHeight: 1080,
Owner: "Mimir",
Layers:
[
new(
"starfire-monitor",
"muninn:starfire:monitor:primary",
"monitor",
Visible: true,
X: 0,
Y: 0,
Width: 1920,
Height: 1080,
ZIndex: 0,
new MimirProgramCrop(Left: 0, Top: 0, Right: 0, Bottom: 48),
ChromaKey: null),
new(
"discord-chat",
"muninn:starfire:window:discord",
"window",
Visible: true,
X: 1225.5,
Y: 0.5,
Width: 1012.2,
Height: 569.4,
ZIndex: 10,
new MimirProgramCrop(Left: 350, Top: 38, Right: 271, Bottom: 87),
new MimirProgramChromaKey(0xff2b7c5a, Similarity: 1, Smoothness: 2, Spill: 1)),
new(
"raven-monitor",
"muninn:raven:monitor:primary",
"network-monitor",
Visible: true,
X: 0,
Y: 0,
Width: 2560,
Height: 1440,
ZIndex: 20,
new MimirProgramCrop(Left: 0, Top: 0, Right: 0, Bottom: 0),
ChromaKey: null)
]);
public static MimirProgramOutputDocument CreateProgramOutput(
string outputId,
string sceneId,
MimirProgramPublicationConfiguration publication,
DateTimeOffset? updatedAt = null) =>
new(
outputId,
(updatedAt ?? DateTimeOffset.UtcNow).ToString("O"),
sceneId,
publication.VideoSurfaceName,
publication.AudioKind.ToString(),
publication.SitePublisherRoute,
publication.DiagnosticOnly);
private static MimirBioacousticDecoderConfigurationSnapshot Snapshot(MimirBioacousticDecoderConfiguration configuration) =>
new(
configuration.Id,

View file

@ -0,0 +1,24 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[CultDocument("mimir.cultmesh_media_frame", "mimir.cultmesh_media_frame.v1")]
[MessagePackObject]
public sealed record MimirCultMeshMediaFrameDocument(
[property: Key(0)]
[property: CultName]
string FrameId,
[property: Key(1)] string StreamId,
[property: Key(2)] long Sequence,
[property: Key(3)] string ProducedAtUtc,
[property: Key(4)] long TimestampNanoseconds,
[property: Key(5)] string PayloadKind,
[property: Key(6)] string Container,
[property: Key(7)] string VideoCodec,
[property: Key(8)] string AudioCodec,
[property: Key(9)] int PayloadBytes,
[property: Key(10)] byte[] Payload,
[property: Key(11)] string ProducerNode,
[property: Key(12)] string ClockDomainId,
[property: Key(13)] string[] Tags);

View file

@ -1,64 +0,0 @@
namespace Mimir.Runtime.Synchronization;
public enum MimirEveProgramOutputTransport
{
D3D12SharedTextureHardwareEncode,
SunshineMoonlightProbe,
DesktopDuplicationDiagnostic
}
public sealed record MimirEveProgramOutputConfiguration(
string Id,
string Description,
string TargetHost,
int TargetPort,
string SharedTextureName,
int TargetWidth,
int TargetHeight,
int TargetFramesPerSecond,
MimirEveProgramOutputTransport Transport,
bool UsesWebKit,
bool DiagnosticOnly,
string[] RequiredEnvironment,
string Notes);
public static class MimirEveProgramOutputConfigurations
{
public const string DefaultSharedTextureName = "Global\\MimirFensalirProgramTexture";
public static MimirEveProgramOutputConfiguration NativeSharedTexture { get; } = new(
"eve-native-d3d12-hardware-stream",
"Production direction: Fensalir copies the completed D3D12 backbuffer into a named shared texture; a hardware encoder consumes that texture and EVE displays decoded frames natively.",
"eve",
44044,
DefaultSharedTextureName,
1920,
1440,
60,
MimirEveProgramOutputTransport.D3D12SharedTextureHardwareEncode,
UsesWebKit: false,
DiagnosticOnly: false,
RequiredEnvironment:
[
"FENSALIR_PROGRAM_OUTPUT_D3D12=1",
$"FENSALIR_PROGRAM_OUTPUT_NAME={DefaultSharedTextureName}"
],
"EVE owns decode/composite only. It does not own layout, DOM, timing authority, or Chromium compatibility.");
public static MimirEveProgramOutputConfiguration MoonlightProbe { get; } = NativeSharedTexture with
{
Id = "eve-moonlight-probe",
Description = "Latency probe only: Sunshine/Moonlight can validate network and decode budget before the custom encoder path is finished.",
Transport = MimirEveProgramOutputTransport.SunshineMoonlightProbe,
SharedTextureName = "",
DiagnosticOnly = true,
RequiredEnvironment = [],
Notes = "This proves the EVE display/network budget, not Mimir's shared-texture authority."
};
public static IReadOnlyList<MimirEveProgramOutputConfiguration> BuiltIn { get; } =
[
NativeSharedTexture,
MoonlightProbe
];
}

View file

@ -47,9 +47,22 @@ public static class MimirModuleLibrary
new(
"cultmesh-contracts",
"Mimir.Runtime",
"Typed state documents for codebooks, decoder state, acoustic path state, and actuator state.",
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state"],
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]),
"Typed state documents and stream frames for codebooks, decoder state, acoustic path state, actuator state, program state, Move evidence, and Mimir-fused Move controller poses.",
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation", "mimir.move_controller_pose", "mimir.move_controller_pose_stream_frame"],
[
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke",
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke"
]),
new(
"move-controller-fusion",
"Mimir.Runtime + Fensalir",
"Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses, then publishes them as a realtime CultMesh pose stream for interaction consumers.",
["mimir.move_calibration_protocol.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.quest_access.v1", "muninn.quest_pose_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"],
[
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke",
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke",
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke"
]),
new(
"path-learning",
"Mimir.Runtime",
@ -111,10 +124,10 @@ public static class MimirModuleLibrary
MimirMachineAssemblyPlans.BuiltIn.Select(plan => plan.Id).ToArray(),
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]),
new(
"obs-publication",
"Fensalir + Faust + OBS",
"OBS-facing program surface and separately mixable audio stem configurations.",
MimirObsPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(),
"program-publication",
"Mimir + Fensalir + Faust + Eve + Yggdrasil",
"Mimir-owned program composition, Eve operator surfaces, Yggdrasil site publication, and temporary OBS compatibility adapters.",
MimirProgramPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(),
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"])
];
}

View file

@ -0,0 +1,272 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[CultDocument("mimir.move_calibration_protocol", "mimir.move_calibration_protocol.v1")]
[MessagePackObject]
public sealed record MimirMoveCalibrationProtocolDocument(
[property: Key(0)]
[property: CultName]
string ProtocolId,
[property: Key(1)] string CreatedAtUtc,
[property: Key(2)] string TrackingSpaceId,
[property: Key(3)] string FusionAuthorityId,
[property: Key(4)] MimirMoveCalibrationStreamRequirement[] StreamRequirements,
[property: Key(5)] MimirMoveCalibrationPhase[] Phases,
[property: Key(6)] MimirMoveCalibrationOutput[] Outputs,
[property: Key(7)] MimirMoveCalibrationAcceptance Acceptance,
[property: Key(8)] string[] OperatorNotes);
[MessagePackObject]
public sealed record MimirMoveCalibrationStreamRequirement(
[property: Key(0)] string StreamId,
[property: Key(1)] string HostId,
[property: Key(2)] string Owner,
[property: Key(3)] string Kind,
[property: Key(4)] string Transport,
[property: Key(5)] bool Required,
[property: Key(6)] string Evidence);
[MessagePackObject]
public sealed record MimirMoveCalibrationPhase(
[property: Key(0)] string PhaseId,
[property: Key(1)] string Title,
[property: Key(2)] double DurationSeconds,
[property: Key(3)] string OperatorCue,
[property: Key(4)] string[] RequiredStreamIds,
[property: Key(5)] string[] Produces);
[MessagePackObject]
public sealed record MimirMoveCalibrationOutput(
[property: Key(0)] string OutputId,
[property: Key(1)] string SchemaId,
[property: Key(2)] string Owner,
[property: Key(3)] string Purpose);
[MessagePackObject]
public sealed record MimirMoveCalibrationAcceptance(
[property: Key(0)] int MinimumControllerStateSamplesPerMove,
[property: Key(1)] int MinimumOpticalSamplesPerCamera,
[property: Key(2)] int MinimumExternalPoseSamples,
[property: Key(3)] double MaximumClockSkewMilliseconds,
[property: Key(4)] double MaximumStillGyroRadiansPerSecond,
[property: Key(5)] double MinimumTriangulatedPoseConfidence,
[property: Key(6)] string[] RequiredDerivedOutputs);
public static class MimirMoveCalibrationProtocol
{
public static MimirMoveCalibrationProtocolDocument CreateNightwingFourWandProtocol(
DateTimeOffset? createdAt = null) =>
new(
ProtocolId: "mimir-move-calibration-nightwing-four-wand-v1",
CreatedAtUtc: (createdAt ?? DateTimeOffset.UtcNow).ToString("O"),
TrackingSpaceId: "mimir-stage-space",
FusionAuthorityId: "mimir.runtime.move-fusion",
StreamRequirements:
[
new(
"muninn:nightwing:move-evidence",
"nightwing",
"Muninn on Nightwing",
"Move optical marker candidates plus USB controller IMU/buttons",
"CultMesh bytes stream over the Verse; shared-memory on Nightwing, network bridge to Starfire",
Required: true,
"Nightwing owns its USB-attached Move and local camera witnesses."),
new(
"mimir:starfire:sensor-calibration-session",
"starfire",
"Mimir.Runtime",
"Bounded calibration lifecycle, coverage, fit, validation, and promotion state",
"Typed CultMesh state with CultCache persistence",
Required: true,
"Mimir owns the calibration task and its completion verdict."),
new(
"mimir:starfire:move-controller-poses",
"starfire",
"Mimir.Runtime",
"Mimir-fused controller pose output",
"CultMesh shared-memory bytes frame",
Required: true,
"The calibration run must prove Mimir can republish resolved poses."),
new(
"quest:usb:headset-pose",
"starfire",
"Quest USB/OpenXR witness",
"Quest headset pose near the Move/controllers calibration cluster",
"USB ADB/OpenXR witness bridged into Mimir calibration capture",
Required: false,
"External tracked VR frame for validating Mimir's optical/IMU pose frame."),
new(
"quest:usb:left-controller-pose",
"starfire",
"Quest USB/OpenXR witness",
"Quest left controller pose",
"USB ADB/OpenXR witness bridged into Mimir calibration capture",
Required: false,
"External tracked controller reference sitting beside the Move."),
new(
"quest:usb:right-controller-pose",
"starfire",
"Quest USB/OpenXR witness",
"Quest right controller pose",
"USB ADB/OpenXR witness bridged into Mimir calibration capture",
Required: false,
"External tracked controller reference sitting beside the Move."),
new(
"starfire:scarlett:loopback",
"starfire",
"Mimir.Runtime ASIO source",
"audio loopback timing witness",
"in-process ASIO into Mimir rolling buffers",
Required: false,
"Optional calibration metronome/cue timing witness; not a Move pose owner.")
],
Phases:
[
new(
"preflight-streams",
"Stream liveness and clock edge check",
10.0,
"Hold the four Moves visible. Confirm Nightwing evidence and stable identities are fresh.",
["muninn:nightwing:move-evidence"],
["stream-freshness", "clock-skew-estimate", "controller-id-map"]),
new(
"dark-stillness",
"IMU bias with orbs off",
8.0,
"Set Move lights off. Place both Moves still on the calibration surface.",
["muninn:nightwing:move-evidence"],
["gyro-bias", "accelerometer-gravity-vector", "magnetometer-baseline"]),
new(
"lit-stillness",
"Optical centroid and static gravity alignment",
8.0,
"Light one Move at a time, then both. Keep them still and visible to both camera sets.",
["muninn:nightwing:move-evidence"],
["camera-marker-centroid-stability", "light-command-to-controller-association"]),
new(
"axis-sweeps",
"Slow single-axis rotations",
24.0,
"Sweep each Move slowly through pitch, yaw, and roll while keeping the orb visible.",
["muninn:nightwing:move-evidence"],
["gyro-scale-fit", "accelerometer-frame-fit", "optical-angular-consistency"]),
new(
"figure-eight",
"Magnetometer and cross-axis motion",
20.0,
"Move each controller through a broad figure-eight. Leave the Quest headset/controllers stationary as the external VR reference cluster.",
["muninn:nightwing:move-evidence"],
["magnetometer-hard-soft-iron-fit", "cross-axis-coupling-check"]),
new(
"quest-reference",
"Optional Quest headset/controller reference",
12.0,
"Keep the Move between the two Quest controllers in front of the headset. Capture the static Quest controller/headset frame while Mimir observes the Move.",
["quest:usb:headset-pose", "quest:usb:left-controller-pose", "quest:usb:right-controller-pose"],
["external-vr-pose-frame", "quest-to-mimir-frame-fit", "quest-clock-offset-estimate"]),
new(
"wand-volume-sweep",
"Four-wand optical volume coverage",
120.0,
"Wave all four uniquely identified wands throughout the shared camera volume until Mimir reports sufficient per-sensor coverage.",
["muninn:nightwing:move-evidence", "mimir:starfire:sensor-calibration-session"],
["sensor-grid-coverage", "same-frame-correspondences", "sphere-radius-range", "fit-and-held-out-partitions"]),
new(
"validation-pass",
"Held-out tracking validation",
15.0,
"Run free hand motion with both orbs visible. Mimir may publish validation poses, but not promote calibration if confidence fails.",
["muninn:nightwing:move-evidence", "mimir:starfire:move-controller-poses"],
["triangulated-position-residuals", "imu-prediction-residuals", "promotion-decision"])
],
Outputs:
[
new(
"move-rig-calibration",
"mimir.move_fusion_rig_calibration.v1",
"Mimir.Runtime",
"Camera intrinsics/extrinsics, clock association policy, and gyro unit scale for Move fusion."),
new(
"move-imu-calibration",
"mimir.move_imu_calibration.v1",
"Mimir.Runtime",
"Per-controller gyro bias/scale, accelerometer gravity frame, magnetometer correction, and noise estimates."),
new(
"move-controller-map",
"mimir.move_controller_identity_map.v1",
"Mimir.Runtime",
"Stable association between Muninn USB controller ids, light-command receipts, and optical orb identities."),
new(
"move-calibration-receipt",
"mimir.move_calibration_receipt.v1",
"Mimir.Runtime",
"Immutable receipt of captured sample counts, residuals, rejected phases, and promotion decision.")
],
Acceptance: new(
MinimumControllerStateSamplesPerMove: 600,
MinimumOpticalSamplesPerCamera: 300,
MinimumExternalPoseSamples: 120,
MaximumClockSkewMilliseconds: 20.0,
MaximumStillGyroRadiansPerSecond: 0.035,
MinimumTriangulatedPoseConfidence: 0.65,
RequiredDerivedOutputs:
[
"move-rig-calibration",
"move-imu-calibration",
"move-controller-map",
"move-calibration-receipt"
]),
OperatorNotes:
[
"Mimir owns fusion and calibration promotion. Muninn only publishes source-local evidence and executes local Move light commands.",
"CultCache receipts are durable evidence; CultMesh stream frames are the hot path.",
"Do not promote orientation until gyro bias/scale, gravity alignment, optical residuals, external Quest reference residuals when available, and held-out validation all pass."
]);
public static string[] Validate(MimirMoveCalibrationProtocolDocument protocol)
{
ArgumentNullException.ThrowIfNull(protocol);
var errors = new List<string>();
if (protocol.StreamRequirements.Count(stream => stream.Required) < 3)
{
errors.Add("required-streams-too-small");
}
if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "muninn:nightwing:move-evidence" && stream.Required))
{
errors.Add("missing-nightwing-muninn-move-evidence");
}
if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "mimir:starfire:sensor-calibration-session" && stream.Required))
{
errors.Add("missing-mimir-calibration-session");
}
if (!protocol.Phases.Any(phase => phase.PhaseId == "wand-volume-sweep"))
{
errors.Add("missing-wand-volume-sweep");
}
if (!protocol.Phases.Any(phase => phase.PhaseId == "dark-stillness"))
{
errors.Add("missing-imu-bias-phase");
}
if (!protocol.Phases.Any(phase => phase.Produces.Contains("gyro-scale-fit", StringComparer.Ordinal)))
{
errors.Add("missing-gyro-scale-fit");
}
foreach (var requiredOutput in protocol.Acceptance.RequiredDerivedOutputs)
{
if (!protocol.Outputs.Any(output => output.OutputId == requiredOutput))
{
errors.Add($"missing-output:{requiredOutput}");
}
}
return errors.ToArray();
}
}

View file

@ -0,0 +1,310 @@
using System.Numerics;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MimirMoveFusionCameraCalibration(
[property: Key(0)] string CameraId,
[property: Key(1)] ulong WitnessIdHash,
[property: Key(2)] MimirVector3Snapshot PositionMeters,
[property: Key(3)] MimirQuaternionSnapshot Orientation,
[property: Key(4)] double FocalLengthXPx,
[property: Key(5)] double FocalLengthYPx,
[property: Key(6)] double PrincipalPointXPx,
[property: Key(7)] double PrincipalPointYPx);
[MessagePackObject]
public sealed record MimirMoveFusionRigCalibration(
[property: Key(0)] string CalibrationId,
[property: Key(1)] string TrackingSpaceId,
[property: Key(2)] IReadOnlyList<MimirMoveFusionCameraCalibration> Cameras,
[property: Key(3)] double GyroUnitsPerRadianPerSecond = 1.0,
[property: Key(4)] double MaximumAssociationSkewMilliseconds = 20.0,
[property: Key(5)] double SingleRayFallbackDepthMeters = 1.5);
public sealed record MimirMoveFusionResult(
IReadOnlyList<MimirMoveControllerPoseDocument> Poses,
int ControllerEvidenceCount,
int OpticalEvidenceCount,
int CalibratedOpticalEvidenceCount);
public static class MimirMoveFusion
{
public static MimirMoveFusionResult Fuse(
IReadOnlyList<MimirNativeMoveEvidenceSample> samples,
MimirMoveFusionRigCalibration calibration,
string fusionAuthorityId = "mimir.runtime.move-fusion",
string consumerContract = "fensalir.move-controller-input")
{
ArgumentNullException.ThrowIfNull(samples);
ArgumentNullException.ThrowIfNull(calibration);
ArgumentException.ThrowIfNullOrWhiteSpace(calibration.CalibrationId);
ArgumentException.ThrowIfNullOrWhiteSpace(calibration.TrackingSpaceId);
var controllerStates = samples
.Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState &&
sample.ControllerIdHash != 0)
.OrderBy(sample => sample.SourceTimestampNs)
.ThenBy(sample => sample.Sequence)
.ToArray();
var opticalMarkers = samples
.Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker)
.OrderBy(sample => sample.SourceTimestampNs)
.ThenBy(sample => sample.Sequence)
.ToArray();
var cameras = calibration.Cameras
.Where(IsCameraUsable)
.ToDictionary(camera => camera.WitnessIdHash, camera => camera);
var maxSkewNs = (ulong)Math.Max(0.0, calibration.MaximumAssociationSkewMilliseconds * 1_000_000.0);
var poses = new List<MimirMoveControllerPoseDocument>();
foreach (var controller in controllerStates)
{
var associatedMarkers = opticalMarkers
.Where(marker => IsAssociated(marker, controller, controllerStates.Length, maxSkewNs))
.Select(marker => (Marker: marker, Camera: cameras.GetValueOrDefault(marker.WitnessIdHash)))
.Where(pair => pair.Camera is not null)
.ToArray();
if (associatedMarkers.Length == 0)
{
continue;
}
var rays = associatedMarkers
.Select(pair => BuildRay(pair.Camera!, pair.Marker))
.Where(ray => ray.HasValue)
.Select(ray => ray!.Value)
.ToArray();
if (rays.Length == 0)
{
continue;
}
var position = rays.Length >= 2
? ClosestPoint(rays)
: rays[0].Origin + rays[0].Direction * (float)Math.Max(0.05, calibration.SingleRayFallbackDepthMeters);
var opticalConfidence = Math.Clamp(
associatedMarkers.Average(pair => pair.Marker.Confidence) * (rays.Length >= 2 ? 1.0 : 0.35),
0.0,
1.0);
var confidence = Math.Clamp(opticalConfidence * ControllerConfidence(controller), 0.0, 1.0);
var sourceNs = Math.Max(controller.SourceTimestampNs, associatedMarkers.Max(pair => pair.Marker.SourceTimestampNs));
var arrivalNs = Math.Max(controller.ArrivalNs, associatedMarkers.Max(pair => pair.Marker.ArrivalNs));
var evidenceStreamIds = associatedMarkers
.Select(pair => $"witness:0x{pair.Marker.WitnessIdHash:X16}")
.Append($"controller:0x{controller.ControllerIdHash:X16}")
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var evidenceKinds = new[]
{
rays.Length >= 2 ? "optical-marker:triangulated" : "optical-marker:single-ray-depth-prior",
"controller-state:buttons-imu",
"orientation:imu-unresolved"
};
poses.Add(new MimirMoveControllerPoseDocument(
PoseId: $"move:0x{controller.ControllerIdHash:X16}:{controller.Sequence}",
WandId: $"move:0x{controller.ControllerIdHash:X16}",
TrackingSpaceId: calibration.TrackingSpaceId,
CalibrationId: calibration.CalibrationId,
FusionAuthorityId: fusionAuthorityId,
SourceTimestampNs: checked((long)Math.Min(sourceNs, (ulong)long.MaxValue)),
EstimatedAtNs: checked((long)Math.Min(arrivalNs, (ulong)long.MaxValue)),
Sequence: controller.Sequence,
PositionMeters: ToSnapshot(position),
Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0),
LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0),
AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(
ScaleGyro(controller.GyroX, calibration),
ScaleGyro(controller.GyroY, calibration),
ScaleGyro(controller.GyroZ, calibration)),
Confidence: confidence,
LatencyMilliseconds: arrivalNs >= sourceNs ? (arrivalNs - sourceNs) / 1_000_000.0 : 0.0,
Battery01: float.IsFinite(controller.Battery01) ? Math.Clamp(controller.Battery01, 0.0f, 1.0f) : double.NaN,
Buttons: ButtonsFromMask(controller.ButtonsMask, controller.Trigger),
EvidenceStreamIds: evidenceStreamIds,
EvidenceKinds: evidenceKinds,
ConsumerContract: consumerContract));
}
return new MimirMoveFusionResult(
poses,
controllerStates.Length,
opticalMarkers.Length,
opticalMarkers.Count(marker => cameras.ContainsKey(marker.WitnessIdHash)));
}
private static bool IsAssociated(
MimirNativeMoveEvidenceSample marker,
MimirNativeMoveEvidenceSample controller,
int controllerCount,
ulong maxSkewNs)
{
if (marker.ControllerIdHash != 0 && marker.ControllerIdHash != controller.ControllerIdHash)
{
return false;
}
if (marker.ControllerIdHash == 0 && controllerCount != 1)
{
return false;
}
return TimestampDelta(marker.SourceTimestampNs, controller.SourceTimestampNs) <= maxSkewNs;
}
private static bool IsCameraUsable(MimirMoveFusionCameraCalibration camera) =>
camera.WitnessIdHash != 0 &&
camera.FocalLengthXPx > 0.0 &&
camera.FocalLengthYPx > 0.0;
private static MimirRay? BuildRay(MimirMoveFusionCameraCalibration camera, MimirNativeMoveEvidenceSample marker)
{
if (!float.IsFinite(marker.ImageX) || !float.IsFinite(marker.ImageY))
{
return null;
}
var directionCamera = Vector3.Normalize(new Vector3(
(float)((marker.ImageX - camera.PrincipalPointXPx) / camera.FocalLengthXPx),
(float)((camera.PrincipalPointYPx - marker.ImageY) / camera.FocalLengthYPx),
1.0f));
var orientation = ToQuaternion(camera.Orientation);
return new MimirRay(
ToVector(camera.PositionMeters),
Vector3.Normalize(Vector3.Transform(directionCamera, orientation)));
}
private static Vector3 ClosestPoint(IReadOnlyList<MimirRay> rays)
{
var matrix = new Matrix3();
var rhs = Vector3.Zero;
foreach (var ray in rays)
{
var d = Vector3.Normalize(ray.Direction);
var xx = 1.0f - d.X * d.X;
var yy = 1.0f - d.Y * d.Y;
var zz = 1.0f - d.Z * d.Z;
var xy = -d.X * d.Y;
var xz = -d.X * d.Z;
var yz = -d.Y * d.Z;
matrix.M11 += xx;
matrix.M12 += xy;
matrix.M13 += xz;
matrix.M21 += xy;
matrix.M22 += yy;
matrix.M23 += yz;
matrix.M31 += xz;
matrix.M32 += yz;
matrix.M33 += zz;
rhs += new Vector3(
xx * ray.Origin.X + xy * ray.Origin.Y + xz * ray.Origin.Z,
xy * ray.Origin.X + yy * ray.Origin.Y + yz * ray.Origin.Z,
xz * ray.Origin.X + yz * ray.Origin.Y + zz * ray.Origin.Z);
}
return matrix.TrySolve(rhs, out var solved)
? solved
: rays[0].Origin + rays[0].Direction;
}
private static double ControllerConfidence(MimirNativeMoveEvidenceSample controller) =>
Math.Clamp(
0.75 +
(float.IsFinite(controller.Battery01) ? 0.10 : 0.0) +
(controller.ButtonsMask != 0 ? 0.05 : 0.0) +
(Math.Abs(controller.AccelX) + Math.Abs(controller.AccelY) + Math.Abs(controller.AccelZ) > 0.001 ? 0.10 : 0.0),
0.0,
1.0);
private static MimirTrackingButtonSnapshot[] ButtonsFromMask(uint mask, float trigger)
{
var names = new[]
{
"select", "l3", "r3", "start", "up", "right", "down", "left",
"l2", "r2", "l1", "r1", "triangle", "circle", "cross", "square",
"ps", "move", "trigger"
};
var buttons = new List<MimirTrackingButtonSnapshot>();
for (var bit = 0; bit < names.Length; bit++)
{
var pressed = (mask & (1u << bit)) != 0;
if (pressed || names[bit] == "trigger")
{
buttons.Add(new MimirTrackingButtonSnapshot(
names[bit],
pressed,
names[bit] == "trigger" ? Math.Clamp(trigger, 0.0f, 1.0f) : pressed ? 1.0 : 0.0));
}
}
return buttons.ToArray();
}
private static ulong TimestampDelta(ulong left, ulong right) => left >= right ? left - right : right - left;
private static double ScaleGyro(float value, MimirMoveFusionRigCalibration calibration) =>
calibration.GyroUnitsPerRadianPerSecond <= 0.0
? 0.0
: value / calibration.GyroUnitsPerRadianPerSecond;
private static Vector3 ToVector(MimirVector3Snapshot value) => new((float)value.X, (float)value.Y, (float)value.Z);
private static MimirVector3Snapshot ToSnapshot(Vector3 value) => new(value.X, value.Y, value.Z);
private static Quaternion ToQuaternion(MimirQuaternionSnapshot value)
{
var q = new Quaternion((float)value.X, (float)value.Y, (float)value.Z, (float)value.W);
return q.LengthSquared() <= 1.0e-12f ? Quaternion.Identity : Quaternion.Normalize(q);
}
private readonly record struct MimirRay(Vector3 Origin, Vector3 Direction);
private struct Matrix3
{
public float M11;
public float M12;
public float M13;
public float M21;
public float M22;
public float M23;
public float M31;
public float M32;
public float M33;
public readonly bool TrySolve(Vector3 rhs, out Vector3 value)
{
var det = Determinant(
M11, M12, M13,
M21, M22, M23,
M31, M32, M33);
if (Math.Abs(det) <= 1.0e-8f)
{
value = default;
return false;
}
value = new Vector3(
Determinant(rhs.X, M12, M13, rhs.Y, M22, M23, rhs.Z, M32, M33) / det,
Determinant(M11, rhs.X, M13, M21, rhs.Y, M23, M31, rhs.Z, M33) / det,
Determinant(M11, M12, rhs.X, M21, M22, rhs.Y, M31, M32, rhs.Z) / det);
return true;
}
private static float Determinant(
float m11,
float m12,
float m13,
float m21,
float m22,
float m23,
float m31,
float m32,
float m33) =>
m11 * (m22 * m33 - m23 * m32) -
m12 * (m21 * m33 - m23 * m31) +
m13 * (m21 * m32 - m22 * m31);
}
}

View file

@ -0,0 +1,71 @@
using GameCult.Mesh;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MimirMoveControllerPoseStreamFrame(
[property: Key(0)] string FrameId,
[property: Key(1)] string ProducerPeerId,
[property: Key(2)] long PublishedAtNs,
[property: Key(3)] string TrackingSpaceId,
[property: Key(4)] string CalibrationId,
[property: Key(5)] MimirMoveControllerPoseDocument[] Poses);
public static class MimirMovePoseStream
{
public const string StreamMetadataSchemaId = "mimir.move_controller_pose_stream_frame.v1";
public static CultMeshStreamDescriptor CreateStreamDescriptor(
string streamId,
string verseId,
string ownerPeerId,
string clockDomainId,
string trackingSpaceId,
string calibrationId) =>
new(
streamId,
verseId,
ownerPeerId,
CultMeshStreamKind.Bytes,
new CultMeshStreamClock(clockDomainId, sourceId: streamId, confidence: 1.0, evidenceKind: "mimir-move-controller-pose"),
[CultMeshStreamBodyTransport.SharedMemory, CultMeshStreamBodyTransport.CultCachePage],
label: $"Mimir Move controller poses ({trackingSpaceId}, {calibrationId})",
requiredAccess: CultMeshStreamAccess.Read,
maxInFlightFrames: 4,
metadataSchemaId: StreamMetadataSchemaId);
public static MimirMoveControllerPoseStreamFrame CreateFrame(
string frameId,
string producerPeerId,
long publishedAtNs,
string trackingSpaceId,
string calibrationId,
IEnumerable<MimirMoveControllerPoseDocument> poses)
{
ArgumentException.ThrowIfNullOrWhiteSpace(frameId);
ArgumentException.ThrowIfNullOrWhiteSpace(producerPeerId);
ArgumentException.ThrowIfNullOrWhiteSpace(trackingSpaceId);
ArgumentException.ThrowIfNullOrWhiteSpace(calibrationId);
ArgumentNullException.ThrowIfNull(poses);
return new MimirMoveControllerPoseStreamFrame(
frameId,
producerPeerId,
publishedAtNs,
trackingSpaceId,
calibrationId,
poses
.Where(pose => string.Equals(pose.TrackingSpaceId, trackingSpaceId, StringComparison.Ordinal) &&
string.Equals(pose.CalibrationId, calibrationId, StringComparison.Ordinal))
.OrderBy(pose => pose.SourceTimestampNs)
.ThenBy(pose => pose.Sequence)
.ToArray());
}
public static byte[] SerializeFrame(MimirMoveControllerPoseStreamFrame frame) =>
MessagePackSerializer.Serialize(frame);
public static MimirMoveControllerPoseStreamFrame DeserializeFrame(ReadOnlyMemory<byte> bytes) =>
MessagePackSerializer.Deserialize<MimirMoveControllerPoseStreamFrame>(bytes);
}

View file

@ -0,0 +1,97 @@
namespace Mimir.Runtime.Synchronization;
public static class MimirMoveProofDevSurface
{
public const string EnableEnvironmentVariable = "MIMIR_MOVE_PROOF_DEV_SURFACE";
public static bool IsEnabled() =>
IsTruthy(Environment.GetEnvironmentVariable(EnableEnvironmentVariable));
public static MimirMoveProofSurfaceDocument Create(int sequence = 79)
{
var sourceTimestampNs = 1_781_202_600_004_000_000L + sequence;
var pose = new MimirMoveControllerPoseDocument(
PoseId: $"move:0xA11CE:{sequence}",
WandId: "move:0xA11CE",
TrackingSpaceId: "mimir-stage-space",
CalibrationId: "mimir-move-stage-calibration-v1",
FusionAuthorityId: "mimir.runtime.move-fusion",
SourceTimestampNs: sourceTimestampNs,
EstimatedAtNs: sourceTimestampNs + 2_200_000L,
Sequence: (ulong)sequence,
PositionMeters: new MimirVector3Snapshot(0.0, 0.0, 4.0),
Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0),
LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0),
AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.02, -0.01, 0.04),
Confidence: 0.82,
LatencyMilliseconds: 2.2,
Battery01: 0.72,
Buttons:
[
new MimirTrackingButtonSnapshot("move", true, 1.0),
new MimirTrackingButtonSnapshot("trigger", true, 0.5)
],
EvidenceStreamIds:
[
"witness:0x00000000000A11CE",
"witness:0x0000000000000B0B",
"controller:0x00000000000A11CE"
],
EvidenceKinds:
[
"optical-marker:triangulated",
"controller-state:buttons-imu",
"orientation:imu-unresolved"
],
ConsumerContract: "fensalir.move-controller-input");
var admission = new MimirMuninnMoveEvidenceAdmission(
FrameId: $"muninn:nightwing:move-evidence:{sequence}",
ProducerPeerId: "muninn:nightwing",
PublishedAtNs: sourceTimestampNs + 1_000_000L,
MimirReadAtNs: checked((ulong)(sourceTimestampNs + 1_700_000L)),
SampleCount: 3,
OpticalMarkerCount: 2,
ControllerStateCount: 1,
SourceTimeMinNs: checked((ulong)sourceTimestampNs),
SourceTimeMaxNs: checked((ulong)sourceTimestampNs),
ArrivalMinNs: checked((ulong)(sourceTimestampNs + 500_000L)),
ArrivalMaxNs: checked((ulong)(sourceTimestampNs + 1_700_000L)),
Handle: new MimirNativeSampleHandle(
SensorIdHash: 0xF00D,
TimestampNs: checked((ulong)sourceTimestampNs),
ArrivalNs: checked((ulong)(sourceTimestampNs + 1_700_000L)),
Sequence: (ulong)sequence,
PayloadHandle: 0xBEEF,
Flags: 0,
Reserved: 0),
ReservoirEdgeNs: checked((ulong)sourceTimestampNs),
ReservoirWindowStartNs: checked((ulong)(sourceTimestampNs - 5_000_000_000L)),
ReservoirMoveEvidenceCount: 1);
var poseFrame = MimirMovePoseStream.CreateFrame(
frameId: $"mimir:starfire:move-pose:{sequence}",
producerPeerId: "mimir:starfire",
publishedAtNs: pose.EstimatedAtNs,
trackingSpaceId: pose.TrackingSpaceId,
calibrationId: pose.CalibrationId,
poses: [pose]);
return MimirMoveProofSurface.Create(
admission,
poseFrame,
$"fensalir:starfire:presented-frame:{sequence}",
fensalirPresentedAtNs: checked((ulong)(sourceTimestampNs + 3_100_000L)));
}
private static bool IsTruthy(string? value)
{
if (string.IsNullOrWhiteSpace(value))
{
return false;
}
return value.Equals("1", StringComparison.OrdinalIgnoreCase)
|| value.Equals("true", StringComparison.OrdinalIgnoreCase)
|| value.Equals("yes", StringComparison.OrdinalIgnoreCase)
|| value.Equals("on", StringComparison.OrdinalIgnoreCase);
}
}

View file

@ -0,0 +1,130 @@
using Aquarium.Engine.Render;
using GameCult.Mesh;
namespace Mimir.Runtime.Synchronization;
public sealed record MimirMoveProofPipelineResult(
MimirMuninnMoveEvidenceAdmission Admission,
MimirMoveFusionResult Fusion,
MimirMoveControllerPoseStreamFrame PoseFrame,
MimirMoveProofSurfaceDocument ProofSurface,
AquariumSplineFrame FensalirProbeFrame);
public sealed record MimirMoveProofPipelineOptions(
string MimirEvidenceSourceId,
string MimirEvidenceFrameId,
string MimirPoseFrameId,
string MimirPoseProducerPeerId,
string FensalirFrameId,
ulong FensalirPresentedAtNs,
string FusionAuthorityId = "mimir.runtime.move-fusion",
string ConsumerContract = "fensalir.move-controller-input");
public static class MimirMoveProofPipeline
{
public static bool TryBuildLatestFromRing(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime runtime,
MimirMoveFusionRigCalibration calibration,
MimirMoveProofPipelineOptions options,
out MimirMoveProofPipelineResult? result)
{
ArgumentNullException.ThrowIfNull(ring);
result = null;
if (!ring.TryAcquireLatestRead(out var lease))
{
return false;
}
using (lease)
{
var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]);
return TryBuild(
frame,
runtime,
calibration,
options,
out result);
}
}
public static bool TryBuild(
MimirMuninnMoveEvidenceStreamFrame frame,
MimirNativeReservoirRuntime runtime,
MimirMoveFusionRigCalibration calibration,
MimirMoveProofPipelineOptions options,
out MimirMoveProofPipelineResult? result)
{
ArgumentNullException.ThrowIfNull(frame);
ArgumentNullException.ThrowIfNull(runtime);
ArgumentNullException.ThrowIfNull(calibration);
ArgumentNullException.ThrowIfNull(options);
result = null;
var mimirReadAtNs = NowUnixNs();
var samples = MimirMuninnMoveEvidenceAdapter.BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates);
if (samples.Count == 0)
{
return false;
}
var handle = runtime.AdmitMoveEvidence(
options.MimirEvidenceSourceId,
samples,
calibration.CalibrationId,
calibration.TrackingSpaceId);
var status = runtime.Status;
var admission = new MimirMuninnMoveEvidenceAdmission(
FrameId: frame.FrameId,
ProducerPeerId: frame.ProducerPeerId,
PublishedAtNs: frame.PublishedAtNs,
MimirReadAtNs: mimirReadAtNs,
SampleCount: samples.Count,
OpticalMarkerCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker),
ControllerStateCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState),
SourceTimeMinNs: samples.Min(sample => sample.SourceTimestampNs),
SourceTimeMaxNs: samples.Max(sample => sample.SourceTimestampNs),
ArrivalMinNs: samples.Min(sample => sample.ArrivalNs),
ArrivalMaxNs: samples.Max(sample => sample.ArrivalNs),
Handle: handle,
ReservoirEdgeNs: status.EdgeNs,
ReservoirWindowStartNs: status.WindowStartNs,
ReservoirMoveEvidenceCount: status.MoveEvidenceCount.ToUInt64());
var fusion = MimirMoveFusion.Fuse(
samples,
calibration,
options.FusionAuthorityId,
options.ConsumerContract);
var poseFrame = MimirMovePoseStream.CreateFrame(
options.MimirPoseFrameId,
options.MimirPoseProducerPeerId,
publishedAtNs: fusion.Poses.Count == 0
? checked((long)Math.Min(admission.ArrivalMaxNs, (ulong)long.MaxValue))
: fusion.Poses.Max(pose => pose.EstimatedAtNs),
calibration.TrackingSpaceId,
calibration.CalibrationId,
fusion.Poses);
var proofSurface = MimirMoveProofSurface.Create(
admission,
poseFrame,
options.FensalirFrameId,
options.FensalirPresentedAtNs,
options.MimirEvidenceFrameId);
var probeFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(proofSurface);
result = new MimirMoveProofPipelineResult(
admission,
fusion,
poseFrame,
proofSurface,
probeFrame);
return true;
}
private static ulong NowUnixNs()
{
var now = DateTimeOffset.UtcNow;
var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
}

View file

@ -0,0 +1,279 @@
using GameCult.Mesh;
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
public interface IMimirMoveProofEvidenceRingProvider
{
bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic);
}
public sealed class MimirMoveProofEvidenceRingLease : IDisposable
{
private readonly bool ownsRing;
private readonly IDisposable? transportOwner;
public MimirMoveProofEvidenceRingLease(
CultMeshSharedMemoryFrameRing ring,
bool ownsRing = false,
IDisposable? transportOwner = null)
{
Ring = ring ?? throw new ArgumentNullException(nameof(ring));
this.ownsRing = ownsRing;
this.transportOwner = transportOwner;
}
public CultMeshSharedMemoryFrameRing Ring { get; }
public void Dispose()
{
transportOwner?.Dispose();
if (ownsRing)
{
Ring.Dispose();
}
}
}
[CultDocument("mimir.move_proof_evidence_frame_snapshot", "mimir.move_proof_evidence_frame_snapshot.v1")]
[MessagePackObject]
public sealed record MimirMoveProofEvidenceFrameSnapshotDocument(
[property: Key(0)]
[property: CultName]
string SnapshotId,
[property: Key(1)] string EvidenceStreamId,
[property: Key(2)] string FrameId,
[property: Key(3)] string ProducerPeerId,
[property: Key(4)] ulong PublishedAtNs,
[property: Key(5)] ulong CapturedAtNs,
[property: Key(6)] byte[] Payload);
public static class MimirMoveProofEvidenceFrameSnapshot
{
public static MimirMoveProofEvidenceFrameSnapshotDocument Create(
MimirMuninnMoveEvidenceStreamFrame frame,
byte[] payload,
ulong capturedAtNs)
{
ArgumentNullException.ThrowIfNull(frame);
ArgumentNullException.ThrowIfNull(payload);
if (payload.Length == 0)
{
throw new ArgumentException("Snapshot payload must not be empty.", nameof(payload));
}
return new MimirMoveProofEvidenceFrameSnapshotDocument(
SnapshotId: $"{frame.FrameId}:snapshot",
EvidenceStreamId: StreamIdFromFrameId(frame.FrameId),
FrameId: frame.FrameId,
ProducerPeerId: frame.ProducerPeerId,
PublishedAtNs: checked((ulong)frame.PublishedAtNs),
CapturedAtNs: capturedAtNs,
Payload: payload);
}
public static MimirMoveProofEvidenceFrameSnapshotDocument Load(string path)
{
ArgumentException.ThrowIfNullOrWhiteSpace(path);
return MessagePackSerializer.Deserialize<MimirMoveProofEvidenceFrameSnapshotDocument>(File.ReadAllBytes(path));
}
public static void Save(string path, MimirMoveProofEvidenceFrameSnapshotDocument snapshot)
{
ArgumentException.ThrowIfNullOrWhiteSpace(path);
ArgumentNullException.ThrowIfNull(snapshot);
var directory = Path.GetDirectoryName(Path.GetFullPath(path));
if (!string.IsNullOrWhiteSpace(directory))
{
Directory.CreateDirectory(directory);
}
File.WriteAllBytes(path, MessagePackSerializer.Serialize(snapshot));
}
private static string StreamIdFromFrameId(string frameId)
{
var separator = frameId.LastIndexOf(':');
return separator > 0 ? frameId[..separator] : frameId;
}
}
public sealed record MimirMoveProofRuntimeActivationStatus(
string EvidenceStreamId,
bool Active,
string Diagnostic,
string ProviderKind,
bool DriverRegistered);
[CultDocument("mimir.move_proof_runtime_activation", "mimir.move_proof_runtime_activation.v1")]
[MessagePackObject]
public sealed record MimirMoveProofRuntimeActivationDocument(
[property: Key(0)]
[property: CultName]
string ActivationId,
[property: Key(1)] string EvidenceStreamId,
[property: Key(2)] bool Active,
[property: Key(3)] string Diagnostic,
[property: Key(4)] string ProviderKind,
[property: Key(5)] bool DriverRegistered,
[property: Key(6)] string NativeReservoirPath,
[property: Key(7)] string[] CalibratedCameraIds,
[property: Key(8)] int CalibratedCameraCount,
[property: Key(9)] string LatestProofId,
[property: Key(10)] string LatestMuninnEvidenceFrameId,
[property: Key(11)] string LatestMimirEvidenceFrameId,
[property: Key(12)] string LatestMimirPoseFrameId,
[property: Key(13)] string LatestFensalirFrameId,
[property: Key(14)] MimirMoveProofVerdict LatestVerdict,
[property: Key(15)] ulong ObservedAtNs);
public static class MimirMoveProofRuntimeActivation
{
public static MimirMoveProofRuntimeActivationDocument CreateDocument(
MimirMoveProofRuntimeConfiguration configuration,
MimirMoveProofRuntimeActivationStatus? status,
MimirMoveProofSurfaceDocument? latestProofSurface,
ulong observedAtNs)
{
ArgumentNullException.ThrowIfNull(configuration);
var active = status?.Active ?? false;
var diagnostic = status?.Diagnostic ?? "not activated";
var providerKind = status?.ProviderKind ?? "";
var driverRegistered = status?.DriverRegistered ?? false;
var matchingSurface = IsSurfaceFromEvidenceStream(latestProofSurface, configuration.EvidenceStreamId)
? latestProofSurface
: null;
return new MimirMoveProofRuntimeActivationDocument(
ActivationId: $"{configuration.EvidenceStreamId}:activation",
EvidenceStreamId: configuration.EvidenceStreamId,
Active: active,
Diagnostic: diagnostic,
ProviderKind: providerKind,
DriverRegistered: driverRegistered,
NativeReservoirPath: configuration.NativeReservoirPath,
CalibratedCameraIds: configuration.Calibration.Cameras
.Select(camera => camera.CameraId)
.Where(cameraId => !string.IsNullOrWhiteSpace(cameraId))
.ToArray(),
CalibratedCameraCount: configuration.Calibration.Cameras.Count,
LatestProofId: matchingSurface?.ProofId ?? "",
LatestMuninnEvidenceFrameId: matchingSurface?.MuninnEvidenceFrameId ?? "",
LatestMimirEvidenceFrameId: matchingSurface?.MimirEvidenceFrameId ?? "",
LatestMimirPoseFrameId: matchingSurface?.MimirPoseFrameId ?? "",
LatestFensalirFrameId: matchingSurface?.FensalirFrameId ?? "",
LatestVerdict: matchingSurface?.Verdict ?? MimirMoveProofVerdict.Unknown,
ObservedAtNs: observedAtNs);
}
private static bool IsSurfaceFromEvidenceStream(
MimirMoveProofSurfaceDocument? latestProofSurface,
string evidenceStreamId)
{
if (latestProofSurface is null || string.IsNullOrWhiteSpace(evidenceStreamId))
{
return false;
}
return string.Equals(latestProofSurface.MuninnEvidenceFrameId, evidenceStreamId, StringComparison.Ordinal) ||
latestProofSurface.MuninnEvidenceFrameId.StartsWith($"{evidenceStreamId}:", StringComparison.Ordinal);
}
}
public sealed class MimirConfiguredMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider
{
public static MimirConfiguredMoveProofEvidenceRingProvider Instance { get; } = new();
private MimirConfiguredMoveProofEvidenceRingProvider()
{
}
public bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic)
{
ArgumentNullException.ThrowIfNull(configuration);
if (!string.IsNullOrWhiteSpace(configuration.OdinCultMeshUri))
{
return MimirOdinMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing(
configuration,
out lease,
out diagnostic);
}
if (string.IsNullOrWhiteSpace(configuration.EvidenceSnapshotPath))
{
return MimirUnavailableMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing(
configuration,
out lease,
out diagnostic);
}
var snapshotPath = Path.GetFullPath(configuration.EvidenceSnapshotPath);
if (!File.Exists(snapshotPath))
{
lease = null;
diagnostic = $"Move proof evidence snapshot not found: {snapshotPath}";
return false;
}
var snapshot = MimirMoveProofEvidenceFrameSnapshot.Load(snapshotPath);
if (!string.Equals(snapshot.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal))
{
lease = null;
diagnostic = $"Move proof evidence snapshot stream '{snapshot.EvidenceStreamId}' does not match configured evidence stream '{configuration.EvidenceStreamId}'.";
return false;
}
var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(snapshot.Payload);
if (!string.Equals(decoded.FrameId, snapshot.FrameId, StringComparison.Ordinal) ||
!string.Equals(decoded.ProducerPeerId, snapshot.ProducerPeerId, StringComparison.Ordinal) ||
decoded.PublishedAtNs != checked((long)snapshot.PublishedAtNs))
{
lease = null;
diagnostic = "Move proof evidence snapshot metadata does not match its encoded Muninn frame payload.";
return false;
}
var ring = new CultMeshSharedMemoryFrameRing(
configuration.EvidenceStreamId,
slotCount: 1,
slotByteLength: snapshot.Payload.Length);
if (!ring.TryPublishCopy(snapshot.Payload, checked((long)snapshot.PublishedAtNs), durationNs: 0, out _))
{
ring.Dispose();
lease = null;
diagnostic = $"Move proof evidence snapshot could not be published into fallback ring: {snapshotPath}";
return false;
}
lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true);
diagnostic = $"one-copy evidence snapshot supplied from {snapshotPath}; this is not a live cross-process CultMesh ring";
return true;
}
}
public sealed class MimirUnavailableMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider
{
public static MimirUnavailableMoveProofEvidenceRingProvider Instance { get; } = new();
private MimirUnavailableMoveProofEvidenceRingProvider()
{
}
public bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic)
{
ArgumentNullException.ThrowIfNull(configuration);
lease = null;
diagnostic = "No live CultMesh shared-memory ring opener is configured for this runtime; C# CultMesh rings are currently in-process only.";
return false;
}
}

View file

@ -0,0 +1,205 @@
using GameCult.Mesh;
namespace Mimir.Runtime.Synchronization;
public sealed class MimirMoveProofRuntimeConfiguration
{
public bool Enabled { get; init; } = true;
public string EvidenceStreamId { get; init; } = "muninn:nightwing:move-evidence";
public string EvidenceSnapshotPath { get; init; } = "";
public string OdinCultMeshUri { get; init; } = "";
public string MuninnProviderId { get; init; } = "muninn.telemetry.nightwing";
public string NativeReservoirPath { get; init; } = "";
public string MimirEvidenceSourceId { get; init; } = "mimir:starfire:move-evidence";
public string MimirEvidenceFramePrefix { get; init; } = "mimir:starfire:move-evidence";
public string MimirPoseFramePrefix { get; init; } = "mimir:starfire:move-pose";
public string MimirPoseProducerPeerId { get; init; } = "mimir:starfire";
public string FensalirFramePrefix { get; init; } = "fensalir:starfire:presented-frame";
public string FusionAuthorityId { get; init; } = "mimir.runtime.move-fusion";
public string ConsumerContract { get; init; } = "fensalir.move-controller-input";
public MimirMoveProofCalibrationConfiguration Calibration { get; init; } = new();
public string[] Validate()
{
if (!Enabled)
{
return [];
}
var errors = new List<string>();
Require(EvidenceStreamId, nameof(EvidenceStreamId), errors);
if (!string.IsNullOrWhiteSpace(OdinCultMeshUri))
{
Require(MuninnProviderId, nameof(MuninnProviderId), errors);
}
Require(NativeReservoirPath, nameof(NativeReservoirPath), errors);
Require(MimirEvidenceSourceId, nameof(MimirEvidenceSourceId), errors);
Require(MimirEvidenceFramePrefix, nameof(MimirEvidenceFramePrefix), errors);
Require(MimirPoseFramePrefix, nameof(MimirPoseFramePrefix), errors);
Require(MimirPoseProducerPeerId, nameof(MimirPoseProducerPeerId), errors);
Require(FensalirFramePrefix, nameof(FensalirFramePrefix), errors);
Require(FusionAuthorityId, nameof(FusionAuthorityId), errors);
Require(ConsumerContract, nameof(ConsumerContract), errors);
errors.AddRange(Calibration.Validate().Select(error => $"{nameof(Calibration)}.{error}"));
return errors.ToArray();
}
public MimirMoveProofRuntimeDriverOptions ToDriverOptions() => new(
MimirEvidenceSourceId,
MimirEvidenceFramePrefix,
MimirPoseFramePrefix,
MimirPoseProducerPeerId,
FensalirFramePrefix,
FusionAuthorityId,
ConsumerContract);
public MimirMoveProofRuntimeDriver CreateDriver(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime reservoir)
{
ArgumentNullException.ThrowIfNull(ring);
ArgumentNullException.ThrowIfNull(reservoir);
var errors = Validate();
if (errors.Length > 0)
{
throw new InvalidOperationException($"Invalid Move proof runtime configuration: {string.Join("; ", errors)}");
}
if (!string.Equals(ring.StreamId, EvidenceStreamId, StringComparison.Ordinal))
{
throw new InvalidOperationException(
$"Move proof runtime configuration expects evidence stream '{EvidenceStreamId}', but ring is '{ring.StreamId}'.");
}
return new MimirMoveProofRuntimeDriver(
ring,
reservoir,
Calibration.ToRigCalibration(),
ToDriverOptions());
}
private static void Require(string value, string name, ICollection<string> errors)
{
if (string.IsNullOrWhiteSpace(value))
{
errors.Add($"{name} is required");
}
}
}
public sealed class MimirMoveProofCalibrationConfiguration
{
public string CalibrationId { get; init; } = "mimir-move-stage-calibration-v1";
public string TrackingSpaceId { get; init; } = "mimir-stage-space";
public List<MimirMoveProofCameraCalibrationConfiguration> Cameras { get; init; } = [];
public double GyroUnitsPerRadianPerSecond { get; init; } = 1.0;
public double MaximumAssociationSkewMilliseconds { get; init; } = 20.0;
public double SingleRayFallbackDepthMeters { get; init; } = 1.5;
public string[] Validate()
{
var errors = new List<string>();
if (string.IsNullOrWhiteSpace(CalibrationId))
{
errors.Add($"{nameof(CalibrationId)} is required");
}
if (string.IsNullOrWhiteSpace(TrackingSpaceId))
{
errors.Add($"{nameof(TrackingSpaceId)} is required");
}
if (Cameras.Count < 2)
{
errors.Add($"{nameof(Cameras)} must contain at least two calibrated camera witnesses for full-pose proof");
}
for (var index = 0; index < Cameras.Count; index++)
{
errors.AddRange(Cameras[index].Validate().Select(error => $"{nameof(Cameras)}[{index}].{error}"));
}
return errors.ToArray();
}
public MimirMoveFusionRigCalibration ToRigCalibration() => new(
CalibrationId,
TrackingSpaceId,
Cameras.Select(camera => camera.ToCameraCalibration()).ToArray(),
GyroUnitsPerRadianPerSecond,
MaximumAssociationSkewMilliseconds,
SingleRayFallbackDepthMeters);
}
public sealed class MimirMoveProofCameraCalibrationConfiguration
{
public string CameraId { get; init; } = "";
public ulong WitnessIdHash { get; init; }
public MimirVector3Snapshot PositionMeters { get; init; } = new(0.0, 0.0, 0.0);
public MimirQuaternionSnapshot Orientation { get; init; } = new(0.0, 0.0, 0.0, 1.0);
public double FocalLengthXPx { get; init; }
public double FocalLengthYPx { get; init; }
public double PrincipalPointXPx { get; init; }
public double PrincipalPointYPx { get; init; }
public string[] Validate()
{
var errors = new List<string>();
if (string.IsNullOrWhiteSpace(CameraId))
{
errors.Add($"{nameof(CameraId)} is required");
}
if (WitnessIdHash == 0UL)
{
errors.Add($"{nameof(WitnessIdHash)} is required");
}
if (!double.IsFinite(FocalLengthXPx) || FocalLengthXPx <= 0.0)
{
errors.Add($"{nameof(FocalLengthXPx)} must be positive");
}
if (!double.IsFinite(FocalLengthYPx) || FocalLengthYPx <= 0.0)
{
errors.Add($"{nameof(FocalLengthYPx)} must be positive");
}
return errors.ToArray();
}
public MimirMoveFusionCameraCalibration ToCameraCalibration() => new(
CameraId,
WitnessIdHash,
PositionMeters,
Orientation,
FocalLengthXPx,
FocalLengthYPx,
PrincipalPointXPx,
PrincipalPointYPx);
}

View file

@ -0,0 +1,94 @@
using GameCult.Mesh;
namespace Mimir.Runtime.Synchronization;
public sealed record MimirMoveProofRuntimeDriverOptions(
string MimirEvidenceSourceId,
string MimirEvidenceFramePrefix,
string MimirPoseFramePrefix,
string MimirPoseProducerPeerId,
string FensalirFramePrefix,
string FusionAuthorityId = "mimir.runtime.move-fusion",
string ConsumerContract = "fensalir.move-controller-input");
public sealed class MimirMoveProofRuntimeDriver
{
private readonly CultMeshSharedMemoryFrameRing ring;
private readonly MimirNativeReservoirRuntime reservoir;
private readonly MimirMoveFusionRigCalibration calibration;
private readonly MimirMoveProofRuntimeDriverOptions options;
private string lastFrameId = "";
public MimirMoveProofRuntimeDriver(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime reservoir,
MimirMoveFusionRigCalibration calibration,
MimirMoveProofRuntimeDriverOptions options)
{
this.ring = ring ?? throw new ArgumentNullException(nameof(ring));
this.reservoir = reservoir ?? throw new ArgumentNullException(nameof(reservoir));
this.calibration = calibration ?? throw new ArgumentNullException(nameof(calibration));
this.options = options ?? throw new ArgumentNullException(nameof(options));
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirEvidenceSourceId);
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirEvidenceFramePrefix);
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirPoseFramePrefix);
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirPoseProducerPeerId);
ArgumentException.ThrowIfNullOrWhiteSpace(options.FensalirFramePrefix);
}
public MimirMoveProofPipelineResult? LastResult { get; private set; }
public bool TryBuildLatest(ulong fensalirPresentedAtNs, out MimirMoveProofPipelineResult? result)
{
result = null;
if (!ring.TryAcquireLatestRead(out var lease))
{
return false;
}
using (lease)
{
var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]);
if (string.Equals(frame.FrameId, lastFrameId, StringComparison.Ordinal))
{
return false;
}
var sequence = SequenceSuffix(frame.FrameId);
var pipelineOptions = new MimirMoveProofPipelineOptions(
MimirEvidenceSourceId: options.MimirEvidenceSourceId,
MimirEvidenceFrameId: $"{options.MimirEvidenceFramePrefix}:{sequence}",
MimirPoseFrameId: $"{options.MimirPoseFramePrefix}:{sequence}",
MimirPoseProducerPeerId: options.MimirPoseProducerPeerId,
FensalirFrameId: $"{options.FensalirFramePrefix}:{sequence}",
FensalirPresentedAtNs: fensalirPresentedAtNs,
FusionAuthorityId: options.FusionAuthorityId,
ConsumerContract: options.ConsumerContract);
if (!MimirMoveProofPipeline.TryBuild(frame, reservoir, calibration, pipelineOptions, out result))
{
return false;
}
lastFrameId = frame.FrameId;
LastResult = result;
return true;
}
}
public string Describe()
{
var last = LastResult?.ProofSurface;
return last is null
? $"{options.MimirEvidenceSourceId} waiting"
: $"{last.ProofId} verdict={last.Verdict} chain={last.MuninnEvidenceFrameId}->{last.MimirEvidenceFrameId}->{last.MimirPoseFrameId}->{last.FensalirFrameId}";
}
private static string SequenceSuffix(string frameId)
{
var separator = frameId.LastIndexOf(':');
return separator >= 0 && separator < frameId.Length - 1
? frameId[(separator + 1)..]
: frameId;
}
}

View file

@ -0,0 +1,235 @@
using Aquarium.Engine.Render;
using GameCult.Caching;
using MessagePack;
using System.Numerics;
namespace Mimir.Runtime.Synchronization;
public enum MimirMoveProofVerdict
{
Unknown,
NoPose,
SingleRayFallback,
FullPose
}
[CultDocument("mimir.move_proof_surface", "mimir.move_proof_surface.v1")]
[MessagePackObject]
public sealed record MimirMoveProofSurfaceDocument(
[property: Key(0)]
[property: CultName]
string ProofId,
[property: Key(1)] string MuninnEvidenceFrameId,
[property: Key(2)] string MimirEvidenceFrameId,
[property: Key(3)] string MimirPoseFrameId,
[property: Key(4)] string FensalirFrameId,
[property: Key(5)] string ProducerPeerId,
[property: Key(6)] string FusionAuthorityId,
[property: Key(7)] string TrackingSpaceId,
[property: Key(8)] string CalibrationId,
[property: Key(9)] long SourceTimestampNs,
[property: Key(10)] long MuninnPublishedAtNs,
[property: Key(11)] ulong MimirReadAtNs,
[property: Key(12)] ulong MimirAdmittedAtNs,
[property: Key(13)] long FusionEstimatedAtNs,
[property: Key(14)] ulong FensalirPresentedAtNs,
[property: Key(15)] ulong ReservoirEdgeNs,
[property: Key(16)] ulong ReservoirWindowStartNs,
[property: Key(17)] int OpticalMarkerCount,
[property: Key(18)] int ControllerStateCount,
[property: Key(19)] int PoseCount,
[property: Key(20)] double PoseConfidence,
[property: Key(21)] double LatencyMilliseconds,
[property: Key(22)] MimirMoveProofVerdict Verdict,
[property: Key(23)] string FailureReason,
[property: Key(24)] MimirMoveControllerPoseDocument[] Poses)
{
[IgnoreMember]
public bool IsFullPose => Verdict == MimirMoveProofVerdict.FullPose;
}
public static class MimirMoveProofSurface
{
public const string ConsumerContract = "fensalir.move-proof-surface";
public static MimirMoveProofSurfaceDocument Create(
MimirMuninnMoveEvidenceAdmission admission,
MimirMoveControllerPoseStreamFrame poseFrame,
string fensalirFrameId,
ulong fensalirPresentedAtNs,
string? mimirEvidenceFrameId = null)
{
ArgumentNullException.ThrowIfNull(admission);
ArgumentNullException.ThrowIfNull(poseFrame);
ArgumentException.ThrowIfNullOrWhiteSpace(fensalirFrameId);
var poses = poseFrame.Poses
.OrderBy(pose => pose.SourceTimestampNs)
.ThenBy(pose => pose.Sequence)
.ToArray();
var selectedPose = poses
.OrderByDescending(pose => pose.Confidence)
.FirstOrDefault();
var verdict = ResolveVerdict(poses);
var failureReason = ResolveFailureReason(admission, poses, verdict);
var sourceNs = selectedPose?.SourceTimestampNs ??
checked((long)Math.Min(admission.SourceTimeMaxNs, (ulong)long.MaxValue));
var estimatedAtNs = selectedPose?.EstimatedAtNs ?? 0L;
var confidence = selectedPose?.Confidence ?? 0.0;
var latencyMs = selectedPose?.LatencyMilliseconds ?? 0.0;
return new MimirMoveProofSurfaceDocument(
ProofId: $"{poseFrame.ProducerPeerId}:move-proof:{SequenceSuffix(poseFrame.FrameId)}",
MuninnEvidenceFrameId: admission.FrameId,
MimirEvidenceFrameId: string.IsNullOrWhiteSpace(mimirEvidenceFrameId)
? $"mimir:starfire:move-evidence:{SequenceSuffix(admission.FrameId)}"
: mimirEvidenceFrameId,
MimirPoseFrameId: poseFrame.FrameId,
FensalirFrameId: fensalirFrameId,
ProducerPeerId: poseFrame.ProducerPeerId,
FusionAuthorityId: selectedPose?.FusionAuthorityId ?? "mimir.runtime.move-fusion",
TrackingSpaceId: poseFrame.TrackingSpaceId,
CalibrationId: poseFrame.CalibrationId,
SourceTimestampNs: sourceNs,
MuninnPublishedAtNs: admission.PublishedAtNs,
MimirReadAtNs: admission.MimirReadAtNs,
MimirAdmittedAtNs: admission.ArrivalMaxNs,
FusionEstimatedAtNs: estimatedAtNs,
FensalirPresentedAtNs: fensalirPresentedAtNs,
ReservoirEdgeNs: admission.ReservoirEdgeNs,
ReservoirWindowStartNs: admission.ReservoirWindowStartNs,
OpticalMarkerCount: admission.OpticalMarkerCount,
ControllerStateCount: admission.ControllerStateCount,
PoseCount: poses.Length,
PoseConfidence: Math.Clamp(confidence, 0.0, 1.0),
LatencyMilliseconds: Math.Max(0.0, latencyMs),
Verdict: verdict,
FailureReason: failureReason,
Poses: poses);
}
public static AquariumSplineFrame BuildFensalirProbeFrame(MimirMoveProofSurfaceDocument surface)
{
ArgumentNullException.ThrowIfNull(surface);
if (surface.Poses.Length == 0)
{
return BuildFailureProbe(surface);
}
var splines = new List<AquariumSpline3D>();
foreach (var pose in surface.Poses)
{
var center = ToVector(pose.PositionMeters);
var confidence = (float)Math.Clamp(pose.Confidence, 0.0, 1.0);
var color = surface.IsFullPose
? new Vector4(0.15f, 1.35f, 0.92f, 0.92f)
: new Vector4(1.35f, 0.78f, 0.18f, 0.86f);
var radius = MathF.Max(0.03f, 0.08f * confidence);
splines.Add(new AquariumSpline3D(
$"move-proof-{SanitizeId(pose.PoseId)}-x",
[
new(center + new Vector3(-radius, 0.0f, 0.0f), color),
new(center + new Vector3(radius, 0.0f, 0.0f), color)
],
new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f),
CatmullRomSubdivisions: 1));
splines.Add(new AquariumSpline3D(
$"move-proof-{SanitizeId(pose.PoseId)}-y",
[
new(center + new Vector3(0.0f, -radius, 0.0f), color),
new(center + new Vector3(0.0f, radius, 0.0f), color)
],
new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f),
CatmullRomSubdivisions: 1));
splines.Add(new AquariumSpline3D(
$"move-proof-{SanitizeId(pose.PoseId)}-z",
[
new(center + new Vector3(0.0f, 0.0f, -radius), color),
new(center + new Vector3(0.0f, 0.0f, radius), color)
],
new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f),
CatmullRomSubdivisions: 1));
}
return new AquariumSplineFrame { Splines = splines };
}
private static AquariumSplineFrame BuildFailureProbe(MimirMoveProofSurfaceDocument surface)
{
var color = new Vector4(1.2f, 0.2f, 0.12f, 0.85f);
var z = surface.OpticalMarkerCount > 0 ? 0.35f : 0.0f;
return new AquariumSplineFrame
{
Splines =
[
new AquariumSpline3D(
$"move-proof-{SanitizeId(surface.ProofId)}-missing-pose",
[
new(new Vector3(-0.16f, 0.0f, z), color),
new(new Vector3(0.16f, 0.0f, z), color),
new(new Vector3(0.0f, 0.16f, z), color),
new(new Vector3(-0.16f, 0.0f, z), color)
],
new AquariumSplineStyle(0.01f, 1.2f, 0.85f, 0.8f, 0.12f),
CatmullRomSubdivisions: 1)
]
};
}
private static MimirMoveProofVerdict ResolveVerdict(IReadOnlyList<MimirMoveControllerPoseDocument> poses)
{
if (poses.Count == 0)
{
return MimirMoveProofVerdict.NoPose;
}
return poses.Any(pose => pose.EvidenceKinds.Contains("optical-marker:triangulated", StringComparer.Ordinal))
? MimirMoveProofVerdict.FullPose
: MimirMoveProofVerdict.SingleRayFallback;
}
private static string ResolveFailureReason(
MimirMuninnMoveEvidenceAdmission admission,
IReadOnlyList<MimirMoveControllerPoseDocument> poses,
MimirMoveProofVerdict verdict)
{
if (verdict == MimirMoveProofVerdict.FullPose)
{
return "";
}
if (admission.OpticalMarkerCount == 0)
{
return "missing optical marker candidates";
}
if (admission.ControllerStateCount == 0)
{
return "missing controller state";
}
if (poses.Count == 0)
{
return "Mimir fusion produced no calibrated pose";
}
return "pose is single-ray fallback, not full calibrated optical fusion";
}
private static string SequenceSuffix(string frameId)
{
var separator = frameId.LastIndexOf(':');
return separator >= 0 && separator < frameId.Length - 1
? frameId[(separator + 1)..]
: frameId;
}
private static Vector3 ToVector(MimirVector3Snapshot value) =>
new((float)value.X, (float)value.Y, (float)value.Z);
private static string SanitizeId(string value)
{
var chars = value.Select(ch => char.IsLetterOrDigit(ch) ? ch : '-').ToArray();
return new string(chars);
}
}

View file

@ -0,0 +1,395 @@
using System.Numerics;
namespace Mimir.Runtime.Synchronization;
public static class MimirMoveSphereCalibrationFitter
{
private const int ParameterCount = 8;
public static MimirSensorCalibrationReceiptDocument Fit(
MimirSensorCalibrationSessionDocument session,
MoveVisibilityWindowReceipt evidence,
int imageWidth,
int imageHeight,
double orbRadiusMeters,
DateTimeOffset? completedAt = null)
{
ArgumentNullException.ThrowIfNull(session);
ArgumentNullException.ThrowIfNull(evidence);
if (imageWidth <= 0 || imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth));
var rejectionReasons = new List<string>();
if (session.Phase != MimirSensorCalibrationSessionPhase.Fitting)
rejectionReasons.Add("session-is-not-ready-for-fitting");
if (!(double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0))
rejectionReasons.Add("measured-orb-radius-meters");
if (session.SensorIds.Length != 2)
rejectionReasons.Add("sphere-stereo-fit-requires-exactly-two-sensors");
var pairs = BuildPairs(session, evidence);
if (pairs.Length < session.Acceptance.MinimumSameFrameCorrespondences)
rejectionReasons.Add($"same-frame-correspondences-{pairs.Length}-of-{session.Acceptance.MinimumSameFrameCorrespondences}");
if (rejectionReasons.Count > 0)
return Rejected(session, pairs.Length, rejectionReasons, completedAt);
var heldOut = pairs.Where(IsHeldOut).ToArray();
var training = pairs.Where(pair => !IsHeldOut(pair)).ToArray();
if (heldOut.Length < 50 || training.Length < 200)
return Rejected(session, training.Length, ["insufficient-deterministic-training-or-held-out-partition"], completedAt, heldOut.Length);
var parameters = InitialParameters(training, imageWidth, imageHeight, orbRadiusMeters);
var iterations = Optimize(parameters, training, imageWidth, imageHeight, orbRadiusMeters);
var trainingConsensus = training;
for (var pass = 0; pass < 3; pass++)
{
var pairErrors = PairErrors(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters);
var cutoff = Math.Clamp(Percentile(pairErrors.Select(value => value.Error).Order().ToArray(), 0.70) * 1.5, 4.0, 25.0);
var next = pairErrors.Where(value => value.Error <= cutoff).Select(value => value.Pair).ToArray();
if (next.Length < 200 || next.Length == trainingConsensus.Length) break;
trainingConsensus = next;
iterations += Optimize(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters);
}
var validationCutoff = Math.Max(4.0, session.Acceptance.MaximumP95ReprojectionErrorPx * 2.0);
var heldOutConsensus = PairErrors(parameters, heldOut, imageWidth, imageHeight, orbRadiusMeters)
.Where(value => value.Error <= validationCutoff).Select(value => value.Pair).ToArray();
var trainingErrors = ReprojectionErrors(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters);
var heldOutErrors = ReprojectionErrors(parameters, heldOutConsensus, imageWidth, imageHeight, orbRadiusMeters);
var skewMs = pairs.Select(pair => pair.AbsoluteSkewNs / 1_000_000.0).Order().ToArray();
var residuals = new MimirSensorCalibrationFitResiduals(
Median(trainingErrors), Percentile(trainingErrors, 0.95),
Median(heldOutErrors), Percentile(heldOutErrors, 0.95),
Median(skewMs), skewMs[^1],
ConditionEstimate(parameters, training, imageWidth, imageHeight, orbRadiusMeters),
iterations,
(double)trainingConsensus.Length / training.Length,
(double)heldOutConsensus.Length / heldOut.Length);
if (!parameters.All(double.IsFinite)) rejectionReasons.Add("non-finite-fit");
if (Math.Exp(parameters[0]) is < 100.0 or > 3000.0) rejectionReasons.Add($"camera-0-focal-length-{Math.Exp(parameters[0]):F1}-px-out-of-range");
if (Math.Exp(parameters[1]) is < 100.0 or > 3000.0) rejectionReasons.Add($"camera-1-focal-length-{Math.Exp(parameters[1]):F1}-px-out-of-range");
if (residuals.HeldOutMedianReprojectionErrorPx > session.Acceptance.MaximumMedianReprojectionErrorPx)
rejectionReasons.Add($"held-out-median-reprojection-error-{residuals.HeldOutMedianReprojectionErrorPx:F3}-px");
if (residuals.HeldOutP95ReprojectionErrorPx > session.Acceptance.MaximumP95ReprojectionErrorPx)
rejectionReasons.Add($"held-out-p95-reprojection-error-{residuals.HeldOutP95ReprojectionErrorPx:F3}-px");
if (residuals.MaximumAssociationSkewMilliseconds > session.Acceptance.MaximumAssociationSkewMilliseconds)
rejectionReasons.Add($"association-skew-{residuals.MaximumAssociationSkewMilliseconds:F3}-ms");
if (heldOutConsensus.Length < 50 || residuals.HeldOutInlierFraction < 0.65)
rejectionReasons.Add($"held-out-consensus-{heldOutConsensus.Length}-pairs-{residuals.HeldOutInlierFraction:P1}");
var calibration = rejectionReasons.Count == 0
? BuildCalibration(session, parameters, imageWidth, imageHeight)
: null;
return new MimirSensorCalibrationReceiptDocument(
$"{session.SessionId}:receipt",
session.SessionId,
session.TrackingSpaceId,
(completedAt ?? DateTimeOffset.UtcNow).ToString("O"),
calibration is null ? MimirSensorCalibrationSessionPhase.Rejected : MimirSensorCalibrationSessionPhase.Promoted,
training.Length,
heldOut.Length,
residuals,
calibration,
rejectionReasons.ToArray(),
"mimir.sphere-stereo-bundle.v1");
}
private static Pair[] BuildPairs(MimirSensorCalibrationSessionDocument session, MoveVisibilityWindowReceipt evidence)
{
var camera0 = session.SensorIds[0];
var camera1 = session.SensorIds[1];
var wands = session.RequiredWandIds.ToHashSet(StringComparer.Ordinal);
var candidates = evidence.Correspondences
.Where(pair => wands.Contains(pair.MoveId) &&
pair.AbsoluteSkewNs <= session.Acceptance.MaximumAssociationSkewMilliseconds * 1_000_000.0 &&
string.Equals(pair.First.FrameId, pair.Second.FrameId, StringComparison.Ordinal))
.Select(pair => Order(pair, camera0, camera1))
.Where(pair => pair.HasValue)
.Select(pair => pair!.Value)
.Where(pair => Valid(pair.First) && Valid(pair.Second))
.GroupBy(pair => (pair.First.FrameId, pair.MoveId))
.Select(group => group.First())
.OrderBy(pair => pair.First.PublishedAtNs)
.ThenBy(pair => pair.MoveId, StringComparer.Ordinal)
.ToArray();
if (candidates.Length == 0) return candidates;
var firstMedian = Median(candidates.Select(pair => (double)pair.First.RadiusPx).Order().ToArray());
var secondMedian = Median(candidates.Select(pair => (double)pair.Second.RadiusPx).Order().ToArray());
return candidates.Where(pair =>
{
var firstRatio = pair.First.RadiusPx / firstMedian;
var secondRatio = pair.Second.RadiusPx / secondMedian;
var crossRatio = pair.First.RadiusPx / pair.Second.RadiusPx;
return firstRatio is >= 0.35 and <= 4.0 &&
secondRatio is >= 0.35 and <= 4.0 &&
crossRatio is >= 0.25f and <= 4.0f;
}).ToArray();
}
private static Pair? Order(MoveCrossCameraCorrespondence pair, string camera0, string camera1)
{
if (pair.First.CameraId == camera0 && pair.Second.CameraId == camera1)
return new Pair(pair.MoveId, pair.First, pair.Second, pair.AbsoluteSkewNs);
if (pair.First.CameraId == camera1 && pair.Second.CameraId == camera0)
return new Pair(pair.MoveId, pair.Second, pair.First, pair.AbsoluteSkewNs);
return null;
}
private static bool Valid(MoveVisibilityObservation value) =>
float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) &&
float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f;
private static bool IsHeldOut(Pair pair) => (Fnva64($"{pair.First.FrameId}|{pair.MoveId}") % 10) < 2;
private static double[] InitialParameters(Pair[] pairs, int width, int height, double radius)
{
var focal = Math.Max(width, height) * 0.82;
var source = pairs.Select(pair => PointFromSphere(pair.First, focal, width, height, radius)).ToArray();
var target = pairs.Select(pair => PointFromSphere(pair.Second, focal, width, height, radius)).ToArray();
var (rotation, translation) = FitRigid(source, target);
var rotationVector = RotationVector(rotation);
return [Math.Log(focal), Math.Log(focal), rotationVector.X, rotationVector.Y, rotationVector.Z,
translation.X, translation.Y, translation.Z];
}
private static int Optimize(double[] p, Pair[] pairs, int width, int height, double radius)
{
var lambda = 1e-3;
var cost = Cost(p, pairs, width, height, radius);
var iterations = 0;
for (; iterations < 80; iterations++)
{
BuildNormalEquations(p, pairs, width, height, radius, out var normal, out var gradient);
for (var i = 0; i < ParameterCount; i++) normal[i, i] += lambda * Math.Max(1.0, normal[i, i]);
if (!Solve(normal, gradient.Select(value => -value).ToArray(), out var delta)) break;
var candidate = p.Zip(delta, (value, step) => value + step).ToArray();
var candidateCost = Cost(candidate, pairs, width, height, radius);
if (candidateCost < cost)
{
Array.Copy(candidate, p, ParameterCount);
if (Math.Abs(cost - candidateCost) <= Math.Max(1e-9, cost * 1e-8)) { iterations++; break; }
cost = candidateCost;
lambda = Math.Max(1e-8, lambda * 0.35);
}
else
{
lambda = Math.Min(1e8, lambda * 8.0);
}
}
return iterations;
}
private static void BuildNormalEquations(double[] p, Pair[] pairs, int width, int height, double radius, out double[,] normal, out double[] gradient)
{
normal = new double[ParameterCount, ParameterCount];
gradient = new double[ParameterCount];
foreach (var pair in Sample(pairs, 2500))
{
var residual = Residual(p, pair, width, height, radius);
var jacobian = new double[residual.Length, ParameterCount];
for (var column = 0; column < ParameterCount; column++)
{
var step = column < 2 ? 1e-5 : 1e-6;
var perturbed = (double[])p.Clone();
perturbed[column] += step;
var shifted = Residual(perturbed, pair, width, height, radius);
for (var row = 0; row < residual.Length; row++) jacobian[row, column] = (shifted[row] - residual[row]) / step;
}
for (var row = 0; row < residual.Length; row++)
{
var weight = HuberWeight(residual[row], 4.0) * Math.Max(0.05, Math.Min(pair.First.Confidence, pair.Second.Confidence));
for (var i = 0; i < ParameterCount; i++)
{
gradient[i] += weight * jacobian[row, i] * residual[row];
for (var j = 0; j <= i; j++) normal[i, j] += weight * jacobian[row, i] * jacobian[row, j];
}
}
}
for (var i = 0; i < ParameterCount; i++)
for (var j = 0; j < i; j++) normal[j, i] = normal[i, j];
}
private static double[] Residual(double[] p, Pair pair, int width, int height, double radius)
{
var f0 = Math.Exp(p[0]);
var f1 = Math.Exp(p[1]);
var rotation = QuaternionFromRotationVector(new Vector3((float)p[2], (float)p[3], (float)p[4]));
var translation = new Vector3((float)p[5], (float)p[6], (float)p[7]);
var point0 = PointFromSphere(pair.First, f0, width, height, radius);
var point1 = Vector3.Transform(point0, rotation) + translation;
var forward = Project(point1, f1, width, height, radius);
var pointFrom1 = PointFromSphere(pair.Second, f1, width, height, radius);
var reversePoint = Vector3.Transform(pointFrom1 - translation, Quaternion.Conjugate(rotation));
var reverse = Project(reversePoint, f0, width, height, radius);
return [
forward.X - pair.Second.CenterXPx, forward.Y - pair.Second.CenterYPx, (forward.Z - pair.Second.RadiusPx) * 0.5,
reverse.X - pair.First.CenterXPx, reverse.Y - pair.First.CenterYPx, (reverse.Z - pair.First.RadiusPx) * 0.5];
}
private static double Cost(double[] p, Pair[] pairs, int width, int height, double radius) =>
Sample(pairs, 2500).Sum(pair => Residual(p, pair, width, height, radius).Sum(value => HuberLoss(value, 4.0)));
private static double[] ReprojectionErrors(double[] p, Pair[] pairs, int width, int height, double radius) =>
pairs.SelectMany(pair =>
{
var residual = Residual(p, pair, width, height, radius);
return new[] { Math.Sqrt(residual[0] * residual[0] + residual[1] * residual[1]), Math.Sqrt(residual[3] * residual[3] + residual[4] * residual[4]) };
}).Order().ToArray();
private static (Pair Pair, double Error)[] PairErrors(double[] p, Pair[] pairs, int width, int height, double radius) =>
pairs.Select(pair =>
{
var residual = Residual(p, pair, width, height, radius);
var forward = Math.Sqrt(residual[0] * residual[0] + residual[1] * residual[1]);
var reverse = Math.Sqrt(residual[3] * residual[3] + residual[4] * residual[4]);
return (pair, Math.Max(forward, reverse));
}).OrderBy(value => value.Item2).Select(value => (value.pair, value.Item2)).ToArray();
private static MimirMoveFusionRigCalibration BuildCalibration(MimirSensorCalibrationSessionDocument session, double[] p, int width, int height)
{
var worldToCamera = QuaternionFromRotationVector(new Vector3((float)p[2], (float)p[3], (float)p[4]));
var cameraToWorld = Quaternion.Normalize(Quaternion.Conjugate(worldToCamera));
var translation = new Vector3((float)p[5], (float)p[6], (float)p[7]);
var position = Vector3.Transform(-translation, cameraToWorld);
return new MimirMoveFusionRigCalibration(
$"{session.SessionId}:sphere-stereo",
session.TrackingSpaceId,
[
new MimirMoveFusionCameraCalibration(session.SensorIds[0], Fnva64(session.SensorIds[0]),
new MimirVector3Snapshot(0, 0, 0), new MimirQuaternionSnapshot(0, 0, 0, 1),
Math.Exp(p[0]), Math.Exp(p[0]), width * 0.5, height * 0.5),
new MimirMoveFusionCameraCalibration(session.SensorIds[1], Fnva64(session.SensorIds[1]),
new MimirVector3Snapshot(position.X, position.Y, position.Z),
new MimirQuaternionSnapshot(cameraToWorld.X, cameraToWorld.Y, cameraToWorld.Z, cameraToWorld.W),
Math.Exp(p[1]), Math.Exp(p[1]), width * 0.5, height * 0.5)
],
MaximumAssociationSkewMilliseconds: session.Acceptance.MaximumAssociationSkewMilliseconds);
}
private static MimirSensorCalibrationReceiptDocument Rejected(MimirSensorCalibrationSessionDocument session, int trainingCount, IEnumerable<string> reasons, DateTimeOffset? completedAt, int heldOutCount = 0) =>
new($"{session.SessionId}:receipt", session.SessionId, session.TrackingSpaceId,
(completedAt ?? DateTimeOffset.UtcNow).ToString("O"), MimirSensorCalibrationSessionPhase.Rejected,
trainingCount, heldOutCount,
new MimirSensorCalibrationFitResiduals(double.NaN, double.NaN, double.NaN, double.NaN, double.NaN, double.NaN, double.PositiveInfinity, 0),
null, reasons.ToArray(), "mimir.sphere-stereo-bundle.v1");
private static Vector3 PointFromSphere(MoveVisibilityObservation observation, double focal, int width, int height, double radius)
{
var z = radius * Math.Sqrt(1.0 + focal * focal / (observation.RadiusPx * observation.RadiusPx));
return new Vector3((float)((observation.CenterXPx - width * 0.5) * z / focal),
(float)((height * 0.5 - observation.CenterYPx) * z / focal), (float)z);
}
private static Vector3 Project(Vector3 point, double focal, int width, int height, double radius)
{
if (point.Z <= radius * 1.001) return new Vector3(1e6f, 1e6f, 1e6f);
var imageRadius = focal * radius / Math.Sqrt(point.Z * point.Z - radius * radius);
return new Vector3((float)(width * 0.5 + focal * point.X / point.Z),
(float)(height * 0.5 - focal * point.Y / point.Z), (float)imageRadius);
}
private static (Quaternion Rotation, Vector3 Translation) FitRigid(Vector3[] source, Vector3[] target)
{
var sourceCenter = source.Aggregate(Vector3.Zero, (sum, value) => sum + value) / source.Length;
var targetCenter = target.Aggregate(Vector3.Zero, (sum, value) => sum + value) / target.Length;
double sxx = 0, sxy = 0, sxz = 0, syx = 0, syy = 0, syz = 0, szx = 0, szy = 0, szz = 0;
for (var i = 0; i < source.Length; i++)
{
var a = source[i] - sourceCenter;
var b = target[i] - targetCenter;
sxx += a.X * b.X; sxy += a.X * b.Y; sxz += a.X * b.Z;
syx += a.Y * b.X; syy += a.Y * b.Y; syz += a.Y * b.Z;
szx += a.Z * b.X; szy += a.Z * b.Y; szz += a.Z * b.Z;
}
var n = new double[,] {
{ sxx + syy + szz, syz - szy, szx - sxz, sxy - syx },
{ syz - szy, sxx - syy - szz, sxy + syx, szx + sxz },
{ szx - sxz, sxy + syx, -sxx + syy - szz, syz + szy },
{ sxy - syx, szx + sxz, syz + szy, -sxx - syy + szz }
};
var q = new[] { 1.0, 0.0, 0.0, 0.0 };
for (var iteration = 0; iteration < 64; iteration++)
{
var next = new double[4];
for (var row = 0; row < 4; row++) for (var column = 0; column < 4; column++) next[row] += n[row, column] * q[column];
var norm = Math.Sqrt(next.Sum(value => value * value));
if (norm < 1e-12) break;
for (var j = 0; j < 4; j++) q[j] = next[j] / norm;
}
var rotation = Quaternion.Normalize(new Quaternion((float)q[1], (float)q[2], (float)q[3], (float)q[0]));
return (rotation, targetCenter - Vector3.Transform(sourceCenter, rotation));
}
private static Vector3 RotationVector(Quaternion quaternion)
{
quaternion = Quaternion.Normalize(quaternion);
if (quaternion.W < 0) quaternion = new Quaternion(-quaternion.X, -quaternion.Y, -quaternion.Z, -quaternion.W);
var angle = 2.0 * Math.Acos(Math.Clamp(quaternion.W, -1.0f, 1.0f));
var sinHalf = Math.Sqrt(Math.Max(0.0, 1.0 - quaternion.W * quaternion.W));
return sinHalf < 1e-8 ? Vector3.Zero : new Vector3(quaternion.X, quaternion.Y, quaternion.Z) * (float)(angle / sinHalf);
}
private static Quaternion QuaternionFromRotationVector(Vector3 value)
{
var angle = value.Length();
return angle < 1e-9f ? Quaternion.Identity : Quaternion.CreateFromAxisAngle(value / angle, angle);
}
private static Pair[] Sample(Pair[] pairs, int maximum)
{
if (pairs.Length <= maximum) return pairs;
var stride = (double)pairs.Length / maximum;
return Enumerable.Range(0, maximum).Select(index => pairs[(int)(index * stride)]).ToArray();
}
private static double ConditionEstimate(double[] p, Pair[] pairs, int width, int height, double radius)
{
BuildNormalEquations(p, pairs, width, height, radius, out var normal, out _);
var diagonal = Enumerable.Range(0, ParameterCount).Select(index => Math.Abs(normal[index, index])).Where(value => value > 1e-12).ToArray();
return diagonal.Length == 0 ? double.PositiveInfinity : diagonal.Max() / diagonal.Min();
}
private static bool Solve(double[,] matrix, double[] rhs, out double[] solution)
{
var n = rhs.Length;
var augmented = new double[n, n + 1];
for (var row = 0; row < n; row++) { for (var column = 0; column < n; column++) augmented[row, column] = matrix[row, column]; augmented[row, n] = rhs[row]; }
for (var pivot = 0; pivot < n; pivot++)
{
var best = pivot;
for (var row = pivot + 1; row < n; row++) if (Math.Abs(augmented[row, pivot]) > Math.Abs(augmented[best, pivot])) best = row;
if (Math.Abs(augmented[best, pivot]) < 1e-12) { solution = []; return false; }
if (best != pivot) for (var column = pivot; column <= n; column++) (augmented[pivot, column], augmented[best, column]) = (augmented[best, column], augmented[pivot, column]);
var scale = augmented[pivot, pivot];
for (var column = pivot; column <= n; column++) augmented[pivot, column] /= scale;
for (var row = 0; row < n; row++) if (row != pivot)
{
var factor = augmented[row, pivot];
for (var column = pivot; column <= n; column++) augmented[row, column] -= factor * augmented[pivot, column];
}
}
solution = Enumerable.Range(0, n).Select(row => augmented[row, n]).ToArray();
return solution.All(double.IsFinite);
}
private static double HuberLoss(double value, double threshold) => Math.Abs(value) <= threshold ? 0.5 * value * value : threshold * (Math.Abs(value) - 0.5 * threshold);
private static double HuberWeight(double value, double threshold) => Math.Abs(value) <= threshold ? 1.0 : threshold / Math.Abs(value);
private static double Median(double[] sorted) => Percentile(sorted, 0.5);
private static double Percentile(double[] sorted, double percentile)
{
if (sorted.Length == 0) return double.NaN;
var position = Math.Clamp(percentile, 0, 1) * (sorted.Length - 1);
var lower = (int)Math.Floor(position);
var upper = (int)Math.Ceiling(position);
return sorted[lower] + (sorted[upper] - sorted[lower]) * (position - lower);
}
private static ulong Fnva64(string value)
{
var hash = 14695981039346656037UL;
foreach (var item in System.Text.Encoding.UTF8.GetBytes(value)) { hash ^= item; hash *= 1099511628211UL; }
return hash;
}
private readonly record struct Pair(string MoveId, MoveVisibilityObservation First, MoveVisibilityObservation Second, long AbsoluteSkewNs);
}

View file

@ -0,0 +1,139 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MimirMoveCameraCoverageAssessment(
[property: Key(0)] string CameraId,
[property: Key(1)] int ObservationCount,
[property: Key(2)] double MinimumXPx,
[property: Key(3)] double MaximumXPx,
[property: Key(4)] double MinimumYPx,
[property: Key(5)] double MaximumYPx,
[property: Key(6)] int OccupiedGridCells,
[property: Key(7)] int GridCellCount,
[property: Key(8)] double MinimumRadiusPx,
[property: Key(9)] double MaximumRadiusPx);
[CultDocument("mimir.move_stereo_calibration_assessment", "mimir.move_stereo_calibration_assessment.v1")]
[MessagePackObject]
public sealed record MimirMoveStereoCalibrationAssessmentDocument(
[property: Key(0)] string Schema,
[property: Key(1)] [property: CultName] string AssessmentId,
[property: Key(2)] string SourceSchema,
[property: Key(3)] long AssessedAtNs,
[property: Key(4)] int CorrespondenceCount,
[property: Key(5)] int SynchronizedCorrespondenceCount,
[property: Key(6)] int SameFrameCorrespondenceCount,
[property: Key(7)] int DistinctMoveCount,
[property: Key(8)] double MedianAbsoluteSkewMilliseconds,
[property: Key(9)] double MaximumAbsoluteSkewMilliseconds,
[property: Key(10)] MimirMoveCameraCoverageAssessment[] Cameras,
[property: Key(11)] bool IntrinsicsAvailable,
[property: Key(12)] bool OrbRadiusMetersAvailable,
[property: Key(13)] bool ReadyForRelativePoseFit,
[property: Key(14)] bool Promoted,
[property: Key(15)] string[] MissingRequirements);
public static class MimirMoveStereoCalibrationAssessment
{
public const string Schema = "mimir.move_stereo_calibration_assessment.v1";
public static MimirMoveStereoCalibrationAssessmentDocument Assess(
MoveVisibilityWindowReceipt receipt,
int imageWidth,
int imageHeight,
IReadOnlyCollection<string> cameraIds,
bool intrinsicsAvailable,
double orbRadiusMeters,
double maximumAssociationSkewMilliseconds = 20.0,
DateTimeOffset? assessedAt = null)
{
ArgumentNullException.ThrowIfNull(receipt);
ArgumentNullException.ThrowIfNull(cameraIds);
if (imageWidth <= 0 || imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth));
if (cameraIds.Count != 2) throw new ArgumentException("Stereo calibration requires exactly two camera IDs.", nameof(cameraIds));
var cameras = cameraIds.Distinct(StringComparer.Ordinal).ToArray();
if (cameras.Length != 2) throw new ArgumentException("Stereo camera IDs must be distinct.", nameof(cameraIds));
var maximumSkewNs = maximumAssociationSkewMilliseconds * 1_000_000.0;
var correspondences = receipt.Correspondences
.Where(value => cameras.Contains(value.First.CameraId, StringComparer.Ordinal) &&
cameras.Contains(value.Second.CameraId, StringComparer.Ordinal))
.ToArray();
var synchronized = correspondences.Where(value => value.AbsoluteSkewNs <= maximumSkewNs).ToArray();
var sameFrame = synchronized.Where(value => string.Equals(value.First.FrameId, value.Second.FrameId, StringComparison.Ordinal)).ToArray();
var coverage = cameras.Select(cameraId => BuildCoverage(
cameraId,
receipt.Observations.Where(value => string.Equals(value.CameraId, cameraId, StringComparison.Ordinal)).ToArray(),
imageWidth,
imageHeight)).ToArray();
var missing = new List<string>();
if (!intrinsicsAvailable) missing.Add("calibrated-intrinsics-for-both-eyes");
if (!(double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0)) missing.Add("measured-orb-radius-meters");
if (synchronized.Length < 300) missing.Add("at-least-300-synchronized-correspondences");
if (sameFrame.Length < 100) missing.Add("at-least-100-same-frame-correspondences");
if (synchronized.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count() < 2) missing.Add("at-least-two-stable-move-identities");
foreach (var camera in coverage)
{
if (camera.OccupiedGridCells < 6) missing.Add($"{camera.CameraId}:coverage-at-least-6-of-12-grid-cells");
if (camera.MaximumRadiusPx <= camera.MinimumRadiusPx * 1.25) missing.Add($"{camera.CameraId}:radius-range-at-least-1.25x");
}
var skews = correspondences.Select(value => value.AbsoluteSkewNs / 1_000_000.0).Order().ToArray();
var ready = missing.Count == 0;
return new MimirMoveStereoCalibrationAssessmentDocument(
Schema,
$"move-stereo-assessment:{receipt.StartedAtNs}:{receipt.EndedAtNs}",
receipt.Schema,
(assessedAt ?? DateTimeOffset.UtcNow).ToUnixTimeMilliseconds() * 1_000_000,
correspondences.Length,
synchronized.Length,
sameFrame.Length,
synchronized.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count(),
Median(skews),
skews.Length == 0 ? double.NaN : skews[^1],
coverage,
intrinsicsAvailable,
double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0,
ready,
Promoted: false,
missing.Distinct(StringComparer.Ordinal).ToArray());
}
private static MimirMoveCameraCoverageAssessment BuildCoverage(
string cameraId,
MoveVisibilityObservation[] observations,
int imageWidth,
int imageHeight)
{
var valid = observations.Where(value =>
float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) &&
float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f).ToArray();
var cells = valid.Select(value =>
{
var column = Math.Clamp((int)(value.CenterXPx * 4.0 / imageWidth), 0, 3);
var row = Math.Clamp((int)(value.CenterYPx * 3.0 / imageHeight), 0, 2);
return row * 4 + column;
}).Distinct().Count();
return new MimirMoveCameraCoverageAssessment(
cameraId,
valid.Length,
valid.Length == 0 ? double.NaN : valid.Min(value => value.CenterXPx),
valid.Length == 0 ? double.NaN : valid.Max(value => value.CenterXPx),
valid.Length == 0 ? double.NaN : valid.Min(value => value.CenterYPx),
valid.Length == 0 ? double.NaN : valid.Max(value => value.CenterYPx),
cells,
12,
valid.Length == 0 ? double.NaN : valid.Min(value => value.RadiusPx),
valid.Length == 0 ? double.NaN : valid.Max(value => value.RadiusPx));
}
private static double Median(double[] sorted) => sorted.Length switch
{
0 => double.NaN,
_ when sorted.Length % 2 == 1 => sorted[sorted.Length / 2],
_ => (sorted[sorted.Length / 2 - 1] + sorted[sorted.Length / 2]) * 0.5
};
}

View file

@ -0,0 +1,31 @@
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MoveVisibilityObservation(
[property: Key(0)] string ProviderId,
[property: Key(1)] string CameraId,
[property: Key(2)] string MoveId,
[property: Key(3)] string FrameId,
[property: Key(4)] long PublishedAtNs,
[property: Key(5)] float CenterXPx,
[property: Key(6)] float CenterYPx,
[property: Key(7)] float RadiusPx,
[property: Key(8)] float Confidence);
[MessagePackObject]
public sealed record MoveCrossCameraCorrespondence(
[property: Key(0)] string MoveId,
[property: Key(1)] MoveVisibilityObservation First,
[property: Key(2)] MoveVisibilityObservation Second,
[property: Key(3)] long AbsoluteSkewNs);
[MessagePackObject]
public sealed record MoveVisibilityWindowReceipt(
[property: Key(0)] string Schema,
[property: Key(1)] long StartedAtNs,
[property: Key(2)] long EndedAtNs,
[property: Key(3)] string[] Providers,
[property: Key(4)] MoveVisibilityObservation[] Observations,
[property: Key(5)] MoveCrossCameraCorrespondence[] Correspondences);

View file

@ -0,0 +1,555 @@
using GameCult.Caching;
using GameCult.Mesh;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[CultDocument("muninn.move_marker_candidate", "muninn.move_marker_candidate.v1")]
[MessagePackObject]
public sealed record MuninnMoveMarkerCandidateDocument(
[property: Key(0)] string StreamId,
[property: Key(1)] string HostId,
[property: Key(2)] string CameraId,
[property: Key(3)] ulong FrameSequence,
[property: Key(4)] ulong SourceIdHash,
[property: Key(5)] uint TileX,
[property: Key(6)] uint TileY,
[property: Key(7)] float CenterXPx,
[property: Key(8)] float CenterYPx,
[property: Key(9)] float RadiusPx,
[property: Key(10)] uint AreaPx,
[property: Key(11)] float MeanLuma,
[property: Key(12)] uint PeakLuma,
[property: Key(13)] float Score,
[property: Key(14)] string ObservedAt,
[property: Key(15)] string MoveId = "");
[CultDocument("muninn.move_controller_state", "muninn.move_controller_state.v1")]
[MessagePackObject]
public sealed record MuninnMoveControllerStateDocument(
[property: Key(0)] string StreamId,
[property: Key(1)] string HostId,
[property: Key(2)] string MoveId,
[property: Key(3)] ulong Sequence,
[property: Key(4)] long SourceTimestampNs,
[property: Key(5)] float[] AccelerometerXyz,
[property: Key(6)] float[] GyroscopeXyz,
[property: Key(7)] float[] MagnetometerXyz,
[property: Key(8)] float TriggerValue,
[property: Key(9)] string[] Buttons,
[property: Key(10)] float Battery01,
[property: Key(11)] string ObservedAt,
[property: Key(12)] string SourcePath = "");
[CultDocument("muninn.move_identity", "muninn.move_identity.v1")]
[MessagePackObject]
public sealed record MuninnMoveIdentityDocument(
[property: Key(0)] string IdentityId,
[property: Key(1)] string HostId,
[property: Key(2)] string MoveId,
[property: Key(3)] string SourcePath,
[property: Key(4)] string BluetoothHostAddress,
[property: Key(5)] string State,
[property: Key(6)] string Detail,
[property: Key(7)] string ObservedAt);
public sealed record MimirMuninnMoveIdentitySnapshot(
string IdentityId,
string HostId,
string MoveId,
string SourcePath,
string BluetoothHostAddress,
string State,
string Detail,
ulong ObservedAtNs,
ulong ControllerIdHash);
public sealed record MimirMuninnMoveRosterEntry(
string MoveId,
ulong ControllerIdHash,
string LatestHostId,
string LatestState,
string StateSummary,
string PickupReadiness,
bool IsBluetoothPickupReady,
string BluetoothHostAddress,
bool HasUsbWitness,
string[] UsbHostIds,
string[] BluetoothPickupHostIds,
string[] UsbBlockingHostIds,
string[] SourcePaths,
ulong ObservedAtNs);
[MessagePackObject]
public sealed record MimirMuninnMoveEvidenceStreamFrame(
[property: Key(0)] string FrameId,
[property: Key(1)] string ProducerPeerId,
[property: Key(2)] long PublishedAtNs,
[property: Key(3)] MuninnMoveMarkerCandidateDocument[] MarkerCandidates,
[property: Key(4)] MuninnMoveControllerStateDocument[] ControllerStates);
public sealed record MimirMuninnMoveEvidenceAdmission(
string FrameId,
string ProducerPeerId,
long PublishedAtNs,
ulong MimirReadAtNs,
int SampleCount,
int OpticalMarkerCount,
int ControllerStateCount,
ulong SourceTimeMinNs,
ulong SourceTimeMaxNs,
ulong ArrivalMinNs,
ulong ArrivalMaxNs,
MimirNativeSampleHandle Handle,
ulong ReservoirEdgeNs,
ulong ReservoirWindowStartNs,
ulong ReservoirMoveEvidenceCount);
public static class MimirMuninnMoveEvidenceAdapter
{
private const ulong LiveIdentityWindowNs = 60_000_000_000UL;
public const string StreamMetadataSchemaId = "mimir.muninn_move_evidence_stream_frame.v1";
public static CultMeshStreamDescriptor CreateStreamDescriptor(
string streamId,
string verseId,
string ownerPeerId,
string clockDomainId) =>
new(
streamId,
verseId,
ownerPeerId,
CultMeshStreamKind.Bytes,
new CultMeshStreamClock(clockDomainId, sourceId: streamId, confidence: 1.0, evidenceKind: "muninn-move-evidence"),
[CultMeshStreamBodyTransport.SharedMemory, CultMeshStreamBodyTransport.CultCachePage],
label: "Muninn Move evidence",
requiredAccess: CultMeshStreamAccess.Read,
maxInFlightFrames: 4,
metadataSchemaId: StreamMetadataSchemaId);
public static byte[] SerializeStreamFrame(
MimirMuninnMoveEvidenceStreamFrame frame) =>
MessagePackSerializer.Serialize(frame);
public static MimirMuninnMoveEvidenceStreamFrame DeserializeStreamFrame(
ReadOnlyMemory<byte> bytes) =>
MessagePackSerializer.Deserialize<MimirMuninnMoveEvidenceStreamFrame>(bytes);
public static bool TryAdmitLatestCultMeshFrame(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime runtime,
string producerSourceId,
string calibrationId,
string trackingSpaceId,
out MimirNativeSampleHandle handle,
out int sampleCount)
{
var admitted = TryAdmitLatestCultMeshFrame(
ring,
runtime,
producerSourceId,
calibrationId,
trackingSpaceId,
out var admission);
handle = admission?.Handle ?? default;
sampleCount = admission?.SampleCount ?? 0;
return admitted;
}
public static bool TryAdmitLatestCultMeshFrame(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime runtime,
string producerSourceId,
string calibrationId,
string trackingSpaceId,
out MimirMuninnMoveEvidenceAdmission? admission)
{
ArgumentNullException.ThrowIfNull(ring);
ArgumentNullException.ThrowIfNull(runtime);
admission = null;
if (!ring.TryAcquireLatestRead(out var lease))
{
return false;
}
using (lease)
{
var mimirReadAtNs = NowUnixNs();
var frame = DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]);
var samples = BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates);
if (samples.Count == 0)
{
return false;
}
var handle = runtime.AdmitMoveEvidence(producerSourceId, samples, calibrationId, trackingSpaceId);
var status = runtime.Status;
admission = new MimirMuninnMoveEvidenceAdmission(
FrameId: frame.FrameId,
ProducerPeerId: frame.ProducerPeerId,
PublishedAtNs: frame.PublishedAtNs,
MimirReadAtNs: mimirReadAtNs,
SampleCount: samples.Count,
OpticalMarkerCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker),
ControllerStateCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState),
SourceTimeMinNs: samples.Min(sample => sample.SourceTimestampNs),
SourceTimeMaxNs: samples.Max(sample => sample.SourceTimestampNs),
ArrivalMinNs: samples.Min(sample => sample.ArrivalNs),
ArrivalMaxNs: samples.Max(sample => sample.ArrivalNs),
Handle: handle,
ReservoirEdgeNs: status.EdgeNs,
ReservoirWindowStartNs: status.WindowStartNs,
ReservoirMoveEvidenceCount: status.MoveEvidenceCount.ToUInt64());
return true;
}
}
public static IReadOnlyList<MimirNativeMoveEvidenceSample> BuildNativeSamples(
IEnumerable<MuninnMoveMarkerCandidateDocument> markerCandidates,
IEnumerable<MuninnMoveControllerStateDocument> controllerStates)
{
ArgumentNullException.ThrowIfNull(markerCandidates);
ArgumentNullException.ThrowIfNull(controllerStates);
var samples = new List<MimirNativeMoveEvidenceSample>();
samples.AddRange(markerCandidates.Select(ToNativeSample));
samples.AddRange(controllerStates.Select(ToNativeSample));
return samples
.OrderBy(sample => sample.SourceTimestampNs)
.ThenBy(sample => sample.ArrivalNs)
.ThenBy(sample => sample.Sequence)
.ToArray();
}
public static IReadOnlyList<MimirMuninnMoveIdentitySnapshot> BuildIdentitySnapshots(
IEnumerable<MuninnMoveIdentityDocument> identities)
{
ArgumentNullException.ThrowIfNull(identities);
return identities
.Select(ToIdentitySnapshot)
.OrderBy(identity => identity.MoveId, StringComparer.Ordinal)
.ThenByDescending(identity => identity.ObservedAtNs)
.ToArray();
}
public static IReadOnlyList<MimirMuninnMoveRosterEntry> BuildIdentityRoster(
IEnumerable<MuninnMoveIdentityDocument> identities)
{
ArgumentNullException.ThrowIfNull(identities);
return BuildIdentitySnapshots(identities)
.GroupBy(identity => identity.MoveId, StringComparer.Ordinal)
.Select(ToRosterEntry)
.OrderBy(entry => entry.MoveId, StringComparer.Ordinal)
.ToArray();
}
public static MimirMuninnMoveIdentitySnapshot ToIdentitySnapshot(MuninnMoveIdentityDocument identity)
{
ArgumentNullException.ThrowIfNull(identity);
return new MimirMuninnMoveIdentitySnapshot(
IdentityId: identity.IdentityId,
HostId: identity.HostId,
MoveId: identity.MoveId,
SourcePath: identity.SourcePath,
BluetoothHostAddress: identity.BluetoothHostAddress,
State: identity.State,
Detail: identity.Detail,
ObservedAtNs: ObservedAtToUnixNs(identity.ObservedAt),
ControllerIdHash: Fnva64($"{identity.HostId}:{identity.MoveId}"));
}
private static MimirMuninnMoveRosterEntry ToRosterEntry(
IGrouping<string, MimirMuninnMoveIdentitySnapshot> group)
{
var orderedSnapshots = group
.OrderByDescending(identity => identity.ObservedAtNs)
.ThenBy(identity => identity.HostId, StringComparer.Ordinal)
.ThenBy(identity => identity.SourcePath, StringComparer.Ordinal)
.ToArray();
var latest = orderedSnapshots[0];
var snapshots = orderedSnapshots
.Where(identity => IsLiveIdentitySnapshot(identity, latest.ObservedAtNs))
.ToArray();
if (snapshots.Length == 0)
{
snapshots = [latest];
}
var usbHostIds = snapshots
.Where(IsUsbVisible)
.Select(identity => identity.HostId)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var pickupHostIds = snapshots
.Where(IsBluetoothPickupState)
.Select(identity => identity.HostId)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var waitingPickupHostIds = snapshots
.Where(identity => identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase))
.Select(identity => identity.HostId)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var hasConnectedBluetooth = snapshots.Any(identity =>
identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase));
var hasUnreachableBluetooth = snapshots.Any(identity =>
identity.State.Equals("bluetooth-unreachable", StringComparison.OrdinalIgnoreCase));
var pickupReadiness = ResolvePickupReadiness(
usbHostIds,
waitingPickupHostIds,
hasConnectedBluetooth,
hasUnreachableBluetooth);
var sourcePaths = snapshots
.Select(identity => identity.SourcePath)
.Where(path => !string.IsNullOrWhiteSpace(path))
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
return new MimirMuninnMoveRosterEntry(
MoveId: group.Key,
ControllerIdHash: Fnva64($"move:{group.Key}"),
LatestHostId: latest.HostId,
LatestState: latest.State,
StateSummary: string.Join("+", snapshots.Select(identity => identity.State)
.Where(state => !string.IsNullOrWhiteSpace(state))
.Distinct(StringComparer.Ordinal)
.OrderBy(StateRank)
.ThenBy(state => state, StringComparer.Ordinal)),
PickupReadiness: pickupReadiness,
IsBluetoothPickupReady: pickupReadiness == "pickup-ready",
BluetoothHostAddress: snapshots
.Select(identity => identity.BluetoothHostAddress)
.FirstOrDefault(address => !string.IsNullOrWhiteSpace(address)) ?? string.Empty,
HasUsbWitness: usbHostIds.Length > 0,
UsbHostIds: usbHostIds,
BluetoothPickupHostIds: pickupHostIds,
UsbBlockingHostIds: usbHostIds.Length > 0 ? usbHostIds : [],
SourcePaths: sourcePaths,
ObservedAtNs: latest.ObservedAtNs);
}
private static string ResolvePickupReadiness(
string[] usbHostIds,
string[] waitingPickupHostIds,
bool hasConnectedBluetooth,
bool hasUnreachableBluetooth)
{
if (usbHostIds.Length > 0)
{
return "usb-owned";
}
if (hasConnectedBluetooth)
{
return "bluetooth-connected";
}
if (hasUnreachableBluetooth)
{
return "bluetooth-unreachable";
}
if (waitingPickupHostIds.Length > 0)
{
return "pickup-ready";
}
return "unknown";
}
private static bool IsUsbVisible(MimirMuninnMoveIdentitySnapshot identity) =>
identity.State.Equals("usb-visible", StringComparison.OrdinalIgnoreCase);
private static bool IsLiveIdentitySnapshot(MimirMuninnMoveIdentitySnapshot identity, ulong latestObservedAtNs) =>
identity.ObservedAtNs <= 0 ||
latestObservedAtNs <= 0 ||
identity.ObservedAtNs > latestObservedAtNs ||
latestObservedAtNs - identity.ObservedAtNs <= LiveIdentityWindowNs;
private static bool IsBluetoothPickupState(MimirMuninnMoveIdentitySnapshot identity) =>
identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase) ||
identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase);
private static int StateRank(string state) =>
state.ToLowerInvariant() switch
{
"usb-visible" => 0,
"bluetooth-connected" => 1,
"bluetooth-unreachable" => 2,
"bluetooth-waiting" => 3,
"bluetooth-known" => 4,
_ => 9
};
public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveMarkerCandidateDocument marker)
{
ArgumentNullException.ThrowIfNull(marker);
var observedNs = ObservedAtToUnixNs(marker.ObservedAt);
var timestampNs = observedNs == 0
? checked((ulong)marker.FrameSequence)
: observedNs;
var witnessHash = marker.SourceIdHash != 0
? marker.SourceIdHash
: Fnva64($"{marker.HostId}:{marker.CameraId}:{marker.StreamId}");
return new MimirNativeMoveEvidenceSample(
WitnessIdHash: witnessHash,
ControllerIdHash: string.IsNullOrWhiteSpace(marker.MoveId)
? 0
: Fnva64($"{marker.HostId}:{marker.MoveId}"),
SourceTimestampNs: timestampNs,
ArrivalNs: timestampNs,
Sequence: marker.FrameSequence,
EvidenceKind: (uint)MimirNativeMoveEvidenceKind.OpticalMarker,
Flags: 0,
ImageX: marker.CenterXPx,
ImageY: marker.CenterYPx,
RadiusPx: marker.RadiusPx,
Confidence: Math.Clamp(marker.Score, 0.0f, 1.0f),
AccelX: 0.0f,
AccelY: 0.0f,
AccelZ: 0.0f,
GyroX: 0.0f,
GyroY: 0.0f,
GyroZ: 0.0f,
Trigger: 0.0f,
ButtonsMask: 0,
Reserved: 0,
Battery01: float.NaN,
Reserved1: marker.TileX,
Reserved2: marker.TileY);
}
public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveControllerStateDocument controller)
{
ArgumentNullException.ThrowIfNull(controller);
var observedNs = ObservedAtToUnixNs(controller.ObservedAt);
var sourceNs = controller.SourceTimestampNs > 0
? checked((ulong)controller.SourceTimestampNs)
: observedNs;
var vector = ExtractVectors(controller);
return new MimirNativeMoveEvidenceSample(
WitnessIdHash: Fnva64(controller.StreamId),
ControllerIdHash: Fnva64($"{controller.HostId}:{controller.MoveId}"),
SourceTimestampNs: sourceNs,
ArrivalNs: observedNs == 0 ? sourceNs : observedNs,
Sequence: controller.Sequence,
EvidenceKind: (uint)MimirNativeMoveEvidenceKind.ControllerState,
Flags: 0,
ImageX: float.NaN,
ImageY: float.NaN,
RadiusPx: float.NaN,
Confidence: 1.0f,
AccelX: vector.AccelX,
AccelY: vector.AccelY,
AccelZ: vector.AccelZ,
GyroX: vector.GyroX,
GyroY: vector.GyroY,
GyroZ: vector.GyroZ,
Trigger: Math.Clamp(controller.TriggerValue, 0.0f, 1.0f),
ButtonsMask: ButtonMask(controller.Buttons),
Reserved: 0,
Battery01: float.IsFinite(controller.Battery01)
? Math.Clamp(controller.Battery01, 0.0f, 1.0f)
: float.NaN,
Reserved1: 0,
Reserved2: 0);
}
public static ulong Fnva64(string value)
{
var hash = 14_695_981_039_346_656_037UL;
foreach (var b in System.Text.Encoding.UTF8.GetBytes(value))
{
hash ^= b;
hash *= 1_099_511_628_211UL;
}
return hash == 0 ? 1 : hash;
}
private static ulong NowUnixNs()
{
var now = DateTimeOffset.UtcNow;
var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
private static ulong ObservedAtToUnixNs(string observedAt)
{
const string unixPrefix = "unix-";
if (observedAt.StartsWith(unixPrefix, StringComparison.OrdinalIgnoreCase) &&
ulong.TryParse(observedAt[unixPrefix.Length..], out var unixSeconds))
{
return checked(unixSeconds * 1_000_000_000UL);
}
if (!DateTimeOffset.TryParse(observedAt, out var parsed))
{
return 0;
}
var secondsNs = checked((ulong)parsed.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(parsed.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
private static (float AccelX, float AccelY, float AccelZ, float GyroX, float GyroY, float GyroZ) ExtractVectors(
MuninnMoveControllerStateDocument controller)
{
var accel = controller.AccelerometerXyz ?? [];
var gyro = controller.GyroscopeXyz ?? [];
return (
Component(accel, 0),
Component(accel, 1),
Component(accel, 2),
Component(gyro, 0),
Component(gyro, 1),
Component(gyro, 2));
}
private static float Component(IReadOnlyList<float> values, int index) =>
index < values.Count && float.IsFinite(values[index]) ? values[index] : 0.0f;
private static uint ButtonMask(IEnumerable<string>? buttons)
{
var mask = 0u;
foreach (var button in buttons ?? [])
{
var bit = button.ToLowerInvariant() switch
{
"select" => 0,
"l3" => 1,
"r3" => 2,
"start" => 3,
"up" => 4,
"right" => 5,
"down" => 6,
"left" => 7,
"l2" => 8,
"r2" => 9,
"l1" => 10,
"r1" => 11,
"triangle" => 12,
"circle" => 13,
"cross" => 14,
"square" => 15,
"ps" => 16,
"move" => 17,
"trigger" => 18,
_ => -1
};
if (bit >= 0)
{
mask |= 1u << bit;
}
}
return mask;
}
}

View file

@ -132,6 +132,48 @@ public static class MimirNativeCaptureConfigurations
Notes = "Remote co-streamer audio witness. It decodes locally and ships typed timing/path state back over CultMesh."
};
public static MimirNativeCaptureDeviceProfile StarfirePsMoves { get; } = new(
"starfire-ps-moves",
"Starfire PS Move tracking witnesses",
MimirStreamKind.Tracking,
MimirNativeCaptureTransport.VendorSdk,
MimirCaptureClockDomain.DeviceHardware,
ChannelCount: 2,
PreferredWidth: 0,
PreferredHeight: 0,
PreferredSampleRate: 0,
PreferredFramesPerSecond: 120.0,
DefaultRollingBufferSeconds,
Required: false,
OwnedOutputs: ["local-controller-pose-observations", "performance-motion-cues"],
"USB-attached local Moves. Mimir buffers the observations; tracking code owns hardware pose estimation.");
public static MimirNativeCaptureDeviceProfile NightwingPsMoves { get; } = StarfirePsMoves with
{
Id = "nightwing-ps-moves",
DisplayName = "Nightwing PS Move tracking witnesses",
Transport = MimirNativeCaptureTransport.NetworkCultMesh,
ClockDomain = MimirCaptureClockDomain.NetworkSender,
OwnedOutputs = ["remote-controller-pose-observations", "performance-motion-cues"],
Notes = "USB-attached Nightwing Moves. Muninn on Nightwing publishes typed pose/button observations; Odin owns discovery/schema projection; Mimir owns buffering and fusion."
};
public static MimirNativeCaptureDeviceProfile OdinMoveTrackingCatalog { get; } = new(
"odin-move-tracking-catalog",
"Odin Move tracking discovery catalog",
MimirStreamKind.Tracking,
MimirNativeCaptureTransport.NetworkCultMesh,
MimirCaptureClockDomain.CanonicalFitted,
ChannelCount: 4,
PreferredWidth: 0,
PreferredHeight: 0,
PreferredSampleRate: 0,
PreferredFramesPerSecond: 120.0,
DefaultRollingBufferSeconds,
Required: false,
OwnedOutputs: ["move-stream-discovery", "tracking-schema-rendezvous"],
"Discovery/projection surface only. Odin names Starfire and Nightwing Move streams; it does not own pose capture or Mimir fusion.");
public static IReadOnlyList<MimirNativeCaptureDeviceProfile> LocalSixCameraProfiles { get; } =
[
LeapLeftIr,
@ -146,6 +188,9 @@ public static class MimirNativeCaptureConfigurations
[
.. LocalSixCameraProfiles,
ScarlettStarfire,
ScarlettRaven
ScarlettRaven,
StarfirePsMoves,
NightwingPsMoves,
OdinMoveTrackingCatalog
];
}

View file

@ -20,7 +20,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
ulong payloadHandle,
int byteLength = 0,
ReadOnlyMemory<byte> data = default,
MimirVideoFrameDescriptor? videoFrame = null)
MimirVideoFrameDescriptor? videoFrame = null,
MimirTrackingObservation? trackingObservation = null)
{
samples.Enqueue(new MimirStreamSample(
Descriptor.SourceId,
@ -32,7 +33,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
payloadHandle,
byteLength,
data,
videoFrame));
videoFrame,
TrackingObservation: trackingObservation));
}
public void PushVideoFrame(
@ -54,6 +56,25 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
frame);
}
public void PushTrackingObservation(MimirTrackingObservation observation)
{
if (Descriptor.Kind != MimirStreamKind.Tracking)
{
throw new InvalidOperationException("Tracking observations can only be pushed into a tracking stream source.");
}
if (!string.Equals(observation.StreamId, Descriptor.SourceId, StringComparison.Ordinal))
{
throw new ArgumentException("Tracking observation stream id does not match this stream source.", nameof(observation));
}
Push(
observation.SourceTimestampNs,
observation.ArrivalTimestampNs,
payloadHandle: 0,
trackingObservation: observation);
}
public bool TryRead(out MimirStreamSample sample)
{
return samples.TryDequeue(out sample);

View file

@ -0,0 +1,329 @@
using System.Runtime.InteropServices;
using System.Text;
namespace Mimir.Runtime.Synchronization;
public enum MimirNativeReservoirSampleKind : uint
{
CameraFrame = 0,
CameraFeature = 1,
SceneRay = 2,
SurfaceClaim = 3,
MaterialClaim = 4,
AudioBlock = 5,
PhaseClaim = 6,
EventClaim = 7,
RenderPacket = 8,
MoveEvidence = 9
}
public enum MimirNativeMoveEvidenceKind : uint
{
OpticalMarker = 1,
ControllerState = 2
}
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeSampleHandle(
ulong SensorIdHash,
ulong TimestampNs,
ulong ArrivalNs,
ulong Sequence,
ulong PayloadHandle,
uint Flags,
uint Reserved);
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeMoveEvidenceSample(
ulong WitnessIdHash,
ulong ControllerIdHash,
ulong SourceTimestampNs,
ulong ArrivalNs,
ulong Sequence,
uint EvidenceKind,
uint Flags,
float ImageX,
float ImageY,
float RadiusPx,
float Confidence,
float AccelX,
float AccelY,
float AccelZ,
float GyroX,
float GyroY,
float GyroZ,
float Trigger,
uint ButtonsMask,
uint Reserved,
float Battery01,
uint Reserved1,
uint Reserved2);
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeMoveEvidenceBufferDescriptor(
ulong SampleBufferHandle,
uint SampleCount,
uint SampleStrideBytes,
ulong SourceTimeMinNs,
ulong SourceTimeMaxNs,
ulong CalibrationHash,
ulong TrackingSpaceHash);
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeRuntimeStatus(
ulong EdgeNs,
ulong WindowStartNs,
UIntPtr TotalSampleCount,
UIntPtr CameraFrameCount,
UIntPtr CameraFeatureCount,
UIntPtr SceneRayCount,
UIntPtr SurfaceClaimCount,
UIntPtr MaterialClaimCount,
UIntPtr AudioBlockCount,
UIntPtr PhaseClaimCount,
UIntPtr EventClaimCount,
UIntPtr RenderPacketCount,
UIntPtr MoveEvidenceCount);
public sealed unsafe class MimirNativeReservoirRuntime : IDisposable
{
public const int MoveEvidenceSampleStrideBytes = 112;
private readonly nint runtime;
private readonly List<PinnedMoveEvidenceBatch> batches = [];
private bool disposed;
public MimirNativeReservoirRuntime(string nativeLibraryPath, TimeSpan? window = null)
{
Native.Load(nativeLibraryPath);
runtime = Native.RuntimeCreate((ulong)(window ?? TimeSpan.FromSeconds(5)).TotalNanoseconds());
if (runtime == 0)
{
throw new InvalidOperationException("Native reservoir runtime creation failed.");
}
}
public MimirNativeRuntimeStatus Status
{
get
{
ThrowIfDisposed();
if (!Native.RuntimeStatus(runtime, out var status))
{
throw new InvalidOperationException("Native reservoir runtime status read failed.");
}
return status;
}
}
public MimirNativeSampleHandle AdmitMoveEvidence(
string producerSourceId,
IReadOnlyList<MimirNativeMoveEvidenceSample> samples,
string calibrationId,
string trackingSpaceId)
{
ThrowIfDisposed();
ArgumentException.ThrowIfNullOrWhiteSpace(producerSourceId);
ArgumentException.ThrowIfNullOrWhiteSpace(calibrationId);
ArgumentException.ThrowIfNullOrWhiteSpace(trackingSpaceId);
if (samples.Count == 0)
{
throw new ArgumentException("Move evidence batch must contain at least one sample.", nameof(samples));
}
var producer = Native.ProducerCreateForSource(
MimirNativeReservoirSampleKind.MoveEvidence,
producerSourceId);
if (producer == 0)
{
throw new InvalidOperationException($"Native Move evidence producer creation failed for '{producerSourceId}'.");
}
try
{
var batch = new PinnedMoveEvidenceBatch(samples, calibrationId, trackingSpaceId);
if (!Native.ProducerPushMoveEvidenceBuffer(
producer,
runtime,
batch.SourceTimeMaxNs,
batch.ArrivalMaxNs,
batch.DescriptorPointer,
out var handle))
{
batch.Dispose();
throw new InvalidOperationException("Native Move evidence admission failed.");
}
batches.Add(batch);
return handle;
}
finally
{
Native.ProducerDestroy(producer);
}
}
public void Dispose()
{
if (disposed)
{
return;
}
disposed = true;
foreach (var batch in batches)
{
batch.Dispose();
}
batches.Clear();
Native.RuntimeDestroy(runtime);
}
private void ThrowIfDisposed()
{
ObjectDisposedException.ThrowIf(disposed, this);
}
private sealed class PinnedMoveEvidenceBatch : IDisposable
{
private readonly GCHandle samplesHandle;
private readonly GCHandle descriptorHandle;
public PinnedMoveEvidenceBatch(
IReadOnlyList<MimirNativeMoveEvidenceSample> samples,
string calibrationId,
string trackingSpaceId)
{
var sampleArray = samples.ToArray();
SourceTimeMaxNs = sampleArray.Max(sample => sample.SourceTimestampNs);
ArrivalMaxNs = sampleArray.Max(sample => sample.ArrivalNs);
samplesHandle = GCHandle.Alloc(sampleArray, GCHandleType.Pinned);
var descriptor = new[]
{
new MimirNativeMoveEvidenceBufferDescriptor(
(ulong)samplesHandle.AddrOfPinnedObject(),
(uint)sampleArray.Length,
MoveEvidenceSampleStrideBytes,
sampleArray.Min(sample => sample.SourceTimestampNs),
SourceTimeMaxNs,
Fnva64(calibrationId),
Fnva64(trackingSpaceId))
};
descriptorHandle = GCHandle.Alloc(descriptor, GCHandleType.Pinned);
}
public ulong SourceTimeMaxNs { get; }
public ulong ArrivalMaxNs { get; }
public nint DescriptorPointer => descriptorHandle.AddrOfPinnedObject();
public void Dispose()
{
if (descriptorHandle.IsAllocated)
{
descriptorHandle.Free();
}
if (samplesHandle.IsAllocated)
{
samplesHandle.Free();
}
}
}
private static ulong Fnva64(string value)
{
var bytes = Encoding.UTF8.GetBytes(value);
const ulong offset = 14_695_981_039_346_656_037;
const ulong prime = 1_099_511_628_211;
var hash = offset;
foreach (var b in bytes)
{
hash ^= b;
hash *= prime;
}
return hash == 0 ? 1 : hash;
}
private static class Native
{
private static nint library;
private static delegate* unmanaged[Cdecl]<ulong, nint> runtimeCreate;
private static delegate* unmanaged[Cdecl]<nint, void> runtimeDestroy;
private static delegate* unmanaged[Cdecl]<nint, out MimirNativeRuntimeStatus, byte> runtimeStatus;
private static delegate* unmanaged[Cdecl]<uint, byte*, nuint, ulong, nint> producerCreateForSource;
private static delegate* unmanaged[Cdecl]<nint, void> producerDestroy;
private static delegate* unmanaged[Cdecl]<nint, nint, ulong, ulong, nint, out MimirNativeSampleHandle, byte> producerPushMoveEvidenceBuffer;
public static void Load(string path)
{
if (library != 0)
{
return;
}
if (string.IsNullOrWhiteSpace(path))
{
path = Path.Combine(AppContext.BaseDirectory, "localcast_reservoir.dll");
}
library = NativeLibrary.Load(path);
runtimeCreate = (delegate* unmanaged[Cdecl]<ulong, nint>)NativeLibrary.GetExport(library, "localcast_runtime_create");
runtimeDestroy = (delegate* unmanaged[Cdecl]<nint, void>)NativeLibrary.GetExport(library, "localcast_runtime_destroy");
runtimeStatus = (delegate* unmanaged[Cdecl]<nint, out MimirNativeRuntimeStatus, byte>)NativeLibrary.GetExport(library, "localcast_runtime_status");
producerCreateForSource = (delegate* unmanaged[Cdecl]<uint, byte*, nuint, ulong, nint>)NativeLibrary.GetExport(library, "localcast_producer_create_for_source");
producerDestroy = (delegate* unmanaged[Cdecl]<nint, void>)NativeLibrary.GetExport(library, "localcast_producer_destroy");
producerPushMoveEvidenceBuffer = (delegate* unmanaged[Cdecl]<nint, nint, ulong, ulong, nint, out MimirNativeSampleHandle, byte>)NativeLibrary.GetExport(library, "localcast_producer_push_move_evidence_buffer");
}
public static nint RuntimeCreate(ulong durationNs) => runtimeCreate(durationNs);
public static void RuntimeDestroy(nint handle)
{
if (handle != 0)
{
runtimeDestroy(handle);
}
}
public static bool RuntimeStatus(nint handle, out MimirNativeRuntimeStatus status) =>
runtimeStatus(handle, out status) != 0;
public static nint ProducerCreateForSource(MimirNativeReservoirSampleKind kind, string sourceId)
{
var bytes = Encoding.UTF8.GetBytes(sourceId);
fixed (byte* ptr = bytes)
{
return producerCreateForSource((uint)kind, ptr, (nuint)bytes.Length, 0);
}
}
public static void ProducerDestroy(nint handle)
{
if (handle != 0)
{
producerDestroy(handle);
}
}
public static bool ProducerPushMoveEvidenceBuffer(
nint producer,
nint runtime,
ulong timestampNs,
ulong arrivalNs,
nint descriptor,
out MimirNativeSampleHandle handle) =>
producerPushMoveEvidenceBuffer(producer, runtime, timestampNs, arrivalNs, descriptor, out handle) != 0;
}
}
internal static class MimirTimeSpanExtensions
{
public static long TotalNanoseconds(this TimeSpan value) =>
checked((long)(value.TotalSeconds * 1_000_000_000.0));
}

View file

@ -6,12 +6,14 @@ public enum MimirNetworkPayloadKind
RawAudioDebugWindow,
CompressedAudioProgramFeed,
CompressedVideoProgramFeed,
RawVideoDiagnosticWindow
RawVideoDiagnosticWindow,
MuxedAudioVideoProgramFeed
}
public enum MimirNetworkTransportKind
{
CultMeshTypedState,
CultMeshReliableUdpMedia,
LanUdpDatagrams,
SrtDiagnosticMedia,
WebRtcExperimental,
@ -68,6 +70,18 @@ public static class MimirNetworkTransportConfigurations
RejectionConditions: ["used-as-clock-authority", "used-for-local-hot-path"],
"Useful bridge. Not the Perfect Machine's synchronization core.");
public static MimirNetworkTransportConfiguration CultMeshProgramBridge { get; } = new(
"cultmesh-program-bridge",
"Reliable-UDP CultMesh media bridge for muxed program frames that OBS consumes through a local Starfire endpoint.",
MimirNetworkTransportKind.CultMeshReliableUdpMedia,
MimirNetworkPayloadKind.MuxedAudioVideoProgramFeed,
MayAffectClock: false,
CarriesRawMedia: true,
TargetLatencyMilliseconds: 500.0,
RequiredDocuments: ["mimir.cultmesh_media_frame"],
RejectionConditions: ["used-as-clock-authority", "unbounded-media-cache", "non-local-obs-egress"],
"Raven publishes rolling media-frame documents over CultNet reliable UDP; Starfire lowers them to a local OBS MPEG-TS endpoint.");
public static MimirNetworkTransportConfiguration WebRtcExperiment { get; } = CultMeshTimingState with
{
Id = "webrtc-experimental-media",
@ -84,6 +98,7 @@ public static class MimirNetworkTransportConfigurations
[
CultMeshTimingState,
RawAudioDebugWindow,
CultMeshProgramBridge,
SrtProgramBridge,
WebRtcExperiment
];

View file

@ -1,94 +0,0 @@
namespace Mimir.Runtime.Synchronization;
public enum MimirObsVideoPublicationKind
{
Spout2,
SharedD3D12Texture,
EveNativeD3D12Stream,
SrtDiagnosticBridge,
WindowCaptureDiagnostic
}
public enum MimirObsAudioPublicationKind
{
FaustStemBus,
AsioOutputPair,
WasapiLoopbackDiagnostic,
SrtDiagnosticBridge
}
public sealed record MimirObsAudioStem(
string Id,
string DisplayName,
int ChannelCount,
bool UserMixable,
bool CarriesTimingWitness,
string Notes);
public sealed record MimirObsPublicationConfiguration(
string Id,
string Description,
MimirObsVideoPublicationKind VideoKind,
string VideoSourceName,
MimirObsAudioPublicationKind AudioKind,
IReadOnlyList<MimirObsAudioStem> AudioStems,
double TargetPresentationDelaySeconds,
bool DiagnosticOnly);
public static class MimirObsPublicationConfigurations
{
public static IReadOnlyList<MimirObsAudioStem> ProgramStems { get; } =
[
new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."),
new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."),
new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."),
new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."),
new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."),
new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."),
new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.")
];
public static MimirObsPublicationConfiguration NativeProgram { get; } = new(
"native-program-spout-faust",
"Production target: Fensalir publishes synchronized video, Faust publishes aligned stems.",
MimirObsVideoPublicationKind.Spout2,
"Mimir Point Cloud",
MimirObsAudioPublicationKind.FaustStemBus,
ProgramStems,
TargetPresentationDelaySeconds: 5.0,
DiagnosticOnly: false);
public static MimirObsPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with
{
Id = "native-program-d3d12-shared-texture",
VideoKind = MimirObsVideoPublicationKind.SharedD3D12Texture,
VideoSourceName = "Mimir D3D12 Program Texture"
};
public static MimirObsPublicationConfiguration DiagnosticSrtBridge { get; } = new(
"diagnostic-srt-bridge",
"OBS-facing fallback for bridge tests only. It does not own synchronized program truth.",
MimirObsVideoPublicationKind.SrtDiagnosticBridge,
"Mimir Diagnostic SRT",
MimirObsAudioPublicationKind.SrtDiagnosticBridge,
ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(),
TargetPresentationDelaySeconds: 1.5,
DiagnosticOnly: true);
public static MimirObsPublicationConfiguration EveNativeProgram { get; } = NativeProgram with
{
Id = "eve-native-d3d12-stream",
Description = "EVE-facing native program output: Fensalir publishes a shared D3D12 texture for hardware encode; EVE receives decoded pixels without WebKit layout authority.",
VideoKind = MimirObsVideoPublicationKind.EveNativeD3D12Stream,
VideoSourceName = MimirEveProgramOutputConfigurations.DefaultSharedTextureName,
DiagnosticOnly = false
};
public static IReadOnlyList<MimirObsPublicationConfiguration> BuiltIn { get; } =
[
NativeProgram,
TextureInteropProgram,
EveNativeProgram,
DiagnosticSrtBridge
];
}

View file

@ -0,0 +1,288 @@
using System.Text.Json;
namespace Mimir.Runtime.Synchronization;
public static class MimirObsSceneImporter
{
public static MimirProgramSceneDocument ImportFile(
string path,
string? sceneName = null,
DateTimeOffset? updatedAt = null)
{
using var stream = File.OpenRead(path);
using var document = JsonDocument.Parse(stream);
return Import(document.RootElement, sceneName, updatedAt);
}
public static MimirProgramSceneDocument Import(
JsonElement root,
string? sceneName = null,
DateTimeOffset? updatedAt = null)
{
var sources = root.GetProperty("sources").EnumerateArray().ToArray();
var sourceByName = sources
.Where(source => source.TryGetProperty("name", out _))
.ToDictionary(source => source.GetProperty("name").GetString() ?? "", StringComparer.Ordinal);
var selectedSceneName = !string.IsNullOrWhiteSpace(sceneName)
? sceneName
: root.TryGetProperty("current_program_scene", out var currentProgram)
? currentProgram.GetString()
: root.TryGetProperty("current_scene", out var currentScene)
? currentScene.GetString()
: null;
if (string.IsNullOrWhiteSpace(selectedSceneName))
{
selectedSceneName = "Scene";
}
if (!sourceByName.TryGetValue(selectedSceneName, out var sceneSource))
{
throw new InvalidOperationException($"OBS scene '{selectedSceneName}' was not found.");
}
var settings = sceneSource.GetProperty("settings");
var itemElements = settings.GetProperty("items").EnumerateArray().ToArray();
var layers = new List<MimirProgramSceneLayer>(itemElements.Length);
for (var index = 0; index < itemElements.Length; index++)
{
var item = itemElements[index];
var name = item.GetProperty("name").GetString() ?? $"layer-{index}";
sourceByName.TryGetValue(name, out var source);
var sourceKind = SourceKind(source);
var crop = CombineCrop(ItemCrop(item), FilterCrop(source));
var chromaKey = FilterChromaKey(source);
var (x, y, width, height) = Bounds(item);
layers.Add(new MimirProgramSceneLayer(
LayerId: LayerId(name, index),
SourceRef: SourceRef(name, sourceKind),
SourceKind: sourceKind,
Visible: ReadBool(item, "visible", fallback: true),
X: x,
Y: y,
Width: width,
Height: height,
ZIndex: index,
Crop: crop,
ChromaKey: chromaKey));
}
var resolution = root.TryGetProperty("resolution", out var resolutionElement)
? resolutionElement
: default;
return new MimirProgramSceneDocument(
SceneId: SceneId(selectedSceneName),
UpdatedAtUtc: (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"),
CanvasWidth: ReadInt(resolution, "x", 1920),
CanvasHeight: ReadInt(resolution, "y", 1080),
Owner: "Mimir",
Layers: layers.ToArray());
}
private static string SourceKind(JsonElement source)
{
if (source.ValueKind == JsonValueKind.Undefined)
{
return "unknown";
}
return (source.GetProperty("id").GetString() ?? "") switch
{
"monitor_capture" => "monitor",
"window_capture" => "window",
"dshow_input" => "camera",
"browser_source" => "browser",
"ffmpeg_source" => "network-monitor",
"mimir_program_texture_source" => "mimir-program-texture",
"muninn_stream_source" => "muninn-stream",
_ => "unknown"
};
}
private static string SourceRef(string name, string sourceKind)
{
var normalized = LayerId(name, 0);
if (name.Contains("raven", StringComparison.OrdinalIgnoreCase))
{
return sourceKind.Contains("audio", StringComparison.OrdinalIgnoreCase)
? "muninn:raven:audio:realtek-loopback"
: "muninn:raven:monitor:primary";
}
if (sourceKind == "monitor")
{
return name.Contains("bigscreen", StringComparison.OrdinalIgnoreCase)
? "muninn:starfire:monitor:bigscreen"
: "muninn:starfire:monitor:primary";
}
if (sourceKind == "window")
{
return $"muninn:starfire:window:{normalized}";
}
if (sourceKind == "camera")
{
return $"muninn:starfire:camera:{normalized}";
}
if (sourceKind == "mimir-program-texture")
{
return "mimir:generated:program-texture";
}
if (sourceKind == "browser")
{
return $"mimir:browser:{normalized}";
}
if (sourceKind == "muninn-stream")
{
return $"muninn:starfire:stream:{normalized}";
}
return $"mimir:unknown:{normalized}";
}
private static (double X, double Y, double Width, double Height) Bounds(JsonElement item)
{
var pos = item.TryGetProperty("pos", out var posElement) ? posElement : default;
var scale = item.TryGetProperty("scale", out var scaleElement) ? scaleElement : default;
var scaleRef = item.TryGetProperty("scale_ref", out var scaleRefElement) ? scaleRefElement : default;
var bounds = item.TryGetProperty("bounds", out var boundsElement) ? boundsElement : default;
var width = ReadDouble(bounds, "x", 0.0);
var height = ReadDouble(bounds, "y", 0.0);
var scaledWidth = ReadDouble(scaleRef, "x", 0.0) * Math.Abs(ReadDouble(scale, "x", 1.0));
var scaledHeight = ReadDouble(scaleRef, "y", 0.0) * Math.Abs(ReadDouble(scale, "y", 1.0));
if (scaledWidth > width)
{
width = scaledWidth;
}
if (scaledHeight > height)
{
height = scaledHeight;
}
return (
ReadDouble(pos, "x", 0.0),
ReadDouble(pos, "y", 0.0),
width,
height);
}
private static MimirProgramCrop ItemCrop(JsonElement item) =>
new(
ReadDouble(item, "crop_left", 0.0),
ReadDouble(item, "crop_top", 0.0),
ReadDouble(item, "crop_right", 0.0),
ReadDouble(item, "crop_bottom", 0.0));
private static MimirProgramCrop FilterCrop(JsonElement source)
{
if (source.ValueKind == JsonValueKind.Undefined || !source.TryGetProperty("filters", out var filters))
{
return new MimirProgramCrop(0, 0, 0, 0);
}
foreach (var filter in filters.EnumerateArray())
{
if ((filter.GetProperty("id").GetString() ?? "") != "crop_filter")
{
continue;
}
var settings = filter.GetProperty("settings");
return new MimirProgramCrop(
ReadDouble(settings, "left", 0.0),
ReadDouble(settings, "top", 0.0),
ReadDouble(settings, "right", 0.0),
ReadDouble(settings, "bottom", 0.0));
}
return new MimirProgramCrop(0, 0, 0, 0);
}
private static MimirProgramChromaKey? FilterChromaKey(JsonElement source)
{
if (source.ValueKind == JsonValueKind.Undefined || !source.TryGetProperty("filters", out var filters))
{
return null;
}
MimirProgramChromaKey? strongest = null;
foreach (var filter in filters.EnumerateArray())
{
if ((filter.GetProperty("id").GetString() ?? "") != "chroma_key_filter")
{
continue;
}
var settings = filter.GetProperty("settings");
strongest = new MimirProgramChromaKey(
ReadUInt(settings, "key_color", 0),
ReadDouble(settings, "similarity", 0.0),
ReadDouble(settings, "smoothness", 0.0),
ReadDouble(settings, "spill", 0.0));
}
return strongest;
}
private static MimirProgramCrop CombineCrop(MimirProgramCrop item, MimirProgramCrop filter) =>
new(
item.Left + filter.Left,
item.Top + filter.Top,
item.Right + filter.Right,
item.Bottom + filter.Bottom);
private static string SceneId(string name) =>
$"mimir-scene-{LayerId(name, 0)}";
private static string LayerId(string name, int fallback)
{
var chars = name
.Trim()
.ToLowerInvariant()
.Select(character => char.IsLetterOrDigit(character) ? character : '-')
.ToArray();
var id = string.Join("-", new string(chars).Split('-', StringSplitOptions.RemoveEmptyEntries));
return string.IsNullOrWhiteSpace(id) ? $"layer-{fallback}" : id;
}
private static bool ReadBool(JsonElement element, string property, bool fallback) =>
element.ValueKind != JsonValueKind.Undefined &&
element.TryGetProperty(property, out var value) &&
value.ValueKind is JsonValueKind.True or JsonValueKind.False
? value.GetBoolean()
: fallback;
private static int ReadInt(JsonElement element, string property, int fallback) =>
element.ValueKind != JsonValueKind.Undefined &&
element.TryGetProperty(property, out var value) &&
value.TryGetInt32(out var parsed)
? parsed
: fallback;
private static double ReadDouble(JsonElement element, string property, double fallback) =>
element.ValueKind != JsonValueKind.Undefined &&
element.TryGetProperty(property, out var value) &&
value.TryGetDouble(out var parsed)
? parsed
: fallback;
private static uint ReadUInt(JsonElement element, string property, uint fallback)
{
if (element.ValueKind == JsonValueKind.Undefined ||
!element.TryGetProperty(property, out var value))
{
return fallback;
}
return value.TryGetUInt32(out var parsed)
? parsed
: value.TryGetInt64(out var signed)
? unchecked((uint)signed)
: fallback;
}
}

View file

@ -0,0 +1,262 @@
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Text.Json;
using GameCult.Mesh;
using GameCult.Networking;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
public sealed record MimirDiscoveredCultMeshInputStream(
string ProviderId,
string StreamId,
string Schema,
string Address,
string Channel,
uint ConnectionId);
public sealed class MimirOdinMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider
{
private const uint OdinSnapshotConnectionId = 0x0d1d0002;
public static MimirOdinMoveProofEvidenceRingProvider Instance { get; } = new();
private MimirOdinMoveProofEvidenceRingProvider() { }
public bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic)
{
ArgumentNullException.ThrowIfNull(configuration);
lease = null;
MimirMoveEvidenceRudpPump? pump = null;
CultMeshSharedMemoryFrameRing? ring = null;
try
{
var stream = Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId);
ring = new CultMeshSharedMemoryFrameRing(configuration.EvidenceStreamId, slotCount: 4, slotByteLength: 256 * 1024);
pump = new MimirMoveEvidenceRudpPump(
stream,
ring,
() => Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId));
pump.Start();
lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true, transportOwner: pump);
diagnostic = $"Odin discovered {stream.ProviderId}/{stream.StreamId}; lowering {stream.Address}/{stream.Channel} into local CultMesh ring";
return true;
}
catch (Exception error)
{
pump?.Dispose();
ring?.Dispose();
diagnostic = error.Message;
return false;
}
}
public static MimirDiscoveredCultMeshInputStream Discover(string odinCultMeshUri, string providerId, string streamId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(odinCultMeshUri);
ArgumentException.ThrowIfNullOrWhiteSpace(providerId);
ArgumentException.ThrowIfNullOrWhiteSpace(streamId);
var endpoint = CultMesh.ResolveRudpEndpoint(odinCultMeshUri);
using var transport = CultMesh.CreateRudpClient("mimir-odin-discovery", OdinSnapshotConnectionId, endpoint);
if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10)))
{
throw new TimeoutException($"Timed out connecting to Odin at {odinCultMeshUri}.");
}
var messageId = $"mimir-muninn-discovery:{Guid.NewGuid():N}";
transport.SendSchemaMessage(new CultNetSnapshotRequestMessage
{
MessageId = messageId,
SchemaIds = ["gamecult.eve.surface_state.v1"],
RecordKeys = ["surface:gamecult.network.status"]
});
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(5);
while (DateTime.UtcNow < deadline)
{
var message = transport.ReceiveSchemaMessage(TimeSpan.FromMilliseconds(250), TimeSpan.FromMilliseconds(10));
if (message is CultNetErrorMessage error)
{
throw new InvalidOperationException(error.Error);
}
if (message is CultNetSnapshotResponseRawMessage response &&
string.Equals(response.MessageId, messageId, StringComparison.Ordinal))
{
return SelectStream(response, providerId, streamId);
}
}
throw new TimeoutException($"Timed out waiting for Odin provider snapshot from {odinCultMeshUri}.");
}
public static MimirDiscoveredCultMeshInputStream SelectStream(
CultNetSnapshotResponseRawMessage response,
string providerId,
string streamId)
{
var document = response.Documents.FirstOrDefault(candidate =>
string.Equals(candidate.SchemaId, "gamecult.eve.surface_state.v1", StringComparison.Ordinal) &&
string.Equals(candidate.RecordKey, "surface:gamecult.network.status", StringComparison.Ordinal))
?? throw new InvalidOperationException("Odin snapshot omitted surface:gamecult.network.status.");
using var json = JsonDocument.Parse(MessagePackSerializer.ConvertToJson(document.Payload));
if (!json.RootElement.TryGetProperty("providerCatalog", out var providers))
{
throw new InvalidOperationException("Odin network status omitted providerCatalog.");
}
foreach (var provider in providers.EnumerateArray())
{
if (!string.Equals(Text(provider, "id"), providerId, StringComparison.Ordinal) ||
!provider.TryGetProperty("inputStreams", out var streams))
{
continue;
}
foreach (var stream in streams.EnumerateArray())
{
if (!string.Equals(Text(stream, "streamId"), streamId, StringComparison.Ordinal) ||
!string.Equals(Text(stream, "schema"), MimirMuninnMoveEvidenceAdapter.StreamMetadataSchemaId, StringComparison.Ordinal) ||
!string.Equals(Text(stream, "transport"), "cultnet.transport.rudp.v0", StringComparison.Ordinal))
{
continue;
}
return new MimirDiscoveredCultMeshInputStream(
providerId,
streamId,
Text(stream, "schema"),
Text(stream, "address"),
Text(stream, "channel"),
stream.GetProperty("connectionId").GetUInt32());
}
throw new InvalidOperationException($"Odin provider '{providerId}' has no compatible input stream '{streamId}'.");
}
throw new InvalidOperationException($"Odin has no provider '{providerId}'.");
}
private static string Text(JsonElement value, string name) =>
value.TryGetProperty(name, out var property) ? property.GetString() ?? "" : "";
}
internal sealed class MimirMoveEvidenceRudpPump : IDisposable
{
private static readonly TimeSpan SourceStaleAfter = TimeSpan.FromSeconds(2);
private readonly MimirDiscoveredCultMeshInputStream initialStream;
private readonly CultMeshSharedMemoryFrameRing ring;
private readonly Func<MimirDiscoveredCultMeshInputStream> rediscover;
private readonly CancellationTokenSource stopping = new();
private readonly ManualResetEventSlim started = new(false);
private Thread? thread;
private Exception? startupError;
public MimirMoveEvidenceRudpPump(
MimirDiscoveredCultMeshInputStream initialStream,
CultMeshSharedMemoryFrameRing ring,
Func<MimirDiscoveredCultMeshInputStream> rediscover)
{
this.initialStream = initialStream;
this.ring = ring;
this.rediscover = rediscover;
}
public void Start()
{
thread = new Thread(Run) { IsBackground = true, Name = "mimir-muninn-move-evidence" };
thread.Start();
if (!started.Wait(TimeSpan.FromSeconds(6)))
{
throw new TimeoutException($"Timed out starting Move evidence receiver for {initialStream.Address}.");
}
if (startupError is not null)
{
throw new InvalidOperationException(
$"Could not connect Odin-discovered Move evidence stream {initialStream.StreamId} at {initialStream.Address}: {startupError.Message}",
startupError);
}
}
private void Run()
{
var stream = initialStream;
while (!stopping.IsCancellationRequested)
{
try
{
ReceiveSession(stream);
}
catch (Exception error) when (!stopping.IsCancellationRequested)
{
if (!started.IsSet)
{
startupError = error;
started.Set();
return;
}
Console.Error.WriteLine(
$"Mimir Move evidence receiver reconnecting stream={stream.StreamId} address={stream.Address}: {error.Message}");
if (stopping.Token.WaitHandle.WaitOne(TimeSpan.FromMilliseconds(250)))
{
return;
}
try
{
stream = rediscover();
}
catch (Exception discoveryError)
{
Console.Error.WriteLine(
$"Mimir Move evidence rediscovery failed provider={stream.ProviderId} stream={stream.StreamId}: {discoveryError.Message}");
}
}
}
}
private void ReceiveSession(MimirDiscoveredCultMeshInputStream stream)
{
using var transport = CultMesh.CreateRudpClient(
"mimir-move-evidence-consumer",
stream.ConnectionId,
CultMesh.ResolveRudpEndpoint($"rudp://{stream.Address}"));
if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(5)))
{
throw new TimeoutException($"Timed out connecting to {stream.Address}.");
}
transport.Send("hid.subscribe", Encoding.UTF8.GetBytes(JsonSerializer.Serialize(new { streamId = stream.StreamId })));
started.Set();
var lastFrameAt = DateTime.UtcNow;
while (!stopping.IsCancellationRequested)
{
transport.PollResends();
var frame = transport.ReceiveOnce();
if (frame is null)
{
if (DateTime.UtcNow - lastFrameAt >= SourceStaleAfter)
{
throw new TimeoutException(
$"No Move evidence frame arrived for {SourceStaleAfter.TotalSeconds:0.#} seconds.");
}
Thread.Sleep(2);
continue;
}
if (!string.Equals(frame.ChannelId, stream.Channel, StringComparison.Ordinal))
{
continue;
}
var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(frame.Payload);
if (!string.Equals(decoded.FrameId, stream.StreamId, StringComparison.Ordinal) &&
!decoded.FrameId.StartsWith($"{stream.StreamId}:", StringComparison.Ordinal))
{
continue;
}
lastFrameAt = DateTime.UtcNow;
ring.TryPublishCopy(frame.Payload, decoded.PublishedAtNs, durationNs: 0, out _);
}
}
public void Dispose()
{
stopping.Cancel();
thread?.Join(TimeSpan.FromSeconds(2));
started.Dispose();
stopping.Dispose();
}
}

View file

@ -20,7 +20,7 @@ public sealed record MimirPerfectMachineManifest(
IReadOnlyList<MimirAudioFieldConfiguration> AudioFields,
IReadOnlyList<MimirVisualFusionConfiguration> VisualFields,
IReadOnlyList<MimirComputeOffloadConfiguration> ComputePlans,
IReadOnlyList<MimirObsPublicationConfiguration> Publications,
IReadOnlyList<MimirProgramPublicationConfiguration> Publications,
IReadOnlyList<MimirMachineAssemblyPlan> AssemblyPlans);
public static class MimirPerfectMachineManifestFactory
@ -48,6 +48,6 @@ public static class MimirPerfectMachineManifestFactory
MimirAudioFieldConfigurations.BuiltIn,
MimirVisualFusionConfigurations.BuiltIn,
MimirComputeOffloadConfigurations.BuiltIn,
MimirObsPublicationConfigurations.BuiltIn,
MimirProgramPublicationConfigurations.BuiltIn,
MimirMachineAssemblyPlans.BuiltIn);
}

View file

@ -4,10 +4,11 @@ public enum MimirPerfectMachineNodeRole
{
StarfireAuthority,
RavenRemote,
MuninnHost,
PhoneWitness,
MicrocontrollerWitness,
CalibrationBench,
ObsProgramHost
MimirProgramPublisher
}
public enum MimirAudioFieldModel
@ -38,7 +39,7 @@ public sealed record MimirPerfectMachineProfile(
IReadOnlyList<MimirComputeBackend> ComputeBackends,
bool EmitsWitness,
bool OwnsCanonicalClock,
bool PublishesObsProgram,
bool PublishesProgram,
TimeSpan RollingWindow,
string[] RequiredCultMeshDocuments);
@ -46,7 +47,7 @@ public static class MimirPerfectMachineProfiles
{
public static MimirPerfectMachineProfile StarfireAuthority { get; } = new(
"starfire-authority",
"Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Fensalir/Faust lowering, OBS-facing program output.",
"Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Mimir composition, Fensalir/Faust lowering, Eve operator surfaces, and program publication.",
MimirPerfectMachineNodeRole.StarfireAuthority,
MimirBioacousticDecoderConfiguration.BaselineMfccIndex,
MimirAlignmentActuatorProfile.SixSourceFaust,
@ -54,7 +55,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute, MimirComputeBackend.FaustNativeDsp],
EmitsWitness: true,
OwnsCanonicalClock: true,
PublishesObsProgram: true,
PublishesProgram: true,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
@ -74,7 +75,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
@ -93,7 +94,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(3),
RequiredCultMeshDocuments:
[
@ -111,7 +112,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(2),
RequiredCultMeshDocuments:
[
@ -129,7 +130,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute],
EmitsWitness: true,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(8),
RequiredCultMeshDocuments:
[
@ -138,22 +139,45 @@ public static class MimirPerfectMachineProfiles
"mimir.acoustic_path_state.v1"
]);
public static MimirPerfectMachineProfile ObsProgramHost { get; } = new(
"obs-program-host",
"Program-output profile: consumes aligned stems/spatial bed and publishes OBS-facing surfaces without becoming sync authority.",
MimirPerfectMachineNodeRole.ObsProgramHost,
public static MimirPerfectMachineProfile MuninnHost { get; } = new(
"muninn-host",
"Capture host profile: Muninn observes local screens, windows, loopbacks, cameras, and device health, publishes typed stream capabilities/media, and does not compose the final program.",
MimirPerfectMachineNodeRole.MuninnHost,
MimirBioacousticDecoderConfiguration.CompactFastIndex,
MimirAlignmentActuatorProfile.SixSourceFaust,
MimirAudioFieldModel.AlignedStems,
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
"muninn.telemetry_surface.v1",
"muninn.capture_stream.v1",
"mimir.move_tracking_observation.v1",
"mimir.cultmesh_media_frame.v1"
]);
public static MimirPerfectMachineProfile MimirProgramPublisher { get; } = new(
"mimir-program-publisher",
"Program-output profile: consumes selected Muninn/Mimir streams, owns scene composition and Eve controls, and publishes the program locally and through Yggdrasil without OBS authority.",
MimirPerfectMachineNodeRole.MimirProgramPublisher,
MimirBioacousticDecoderConfiguration.BaselineMfccIndex,
MimirAlignmentActuatorProfile.SixSourceFaust,
MimirAudioFieldModel.SourceBasedSpatialBus,
[MimirComputeBackend.FaustNativeDsp, MimirComputeBackend.D3D12Compute],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: true,
PublishesProgram: true,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
"mimir.acoustic_path_state.v1",
"mimir.actuator_state.v1"
"mimir.actuator_state.v1",
"mimir.program_scene.v1",
"mimir.program_output.v1",
"mimir.eve_operator_surface.v1"
]);
public static IReadOnlyList<MimirPerfectMachineProfile> All { get; } =
@ -163,6 +187,7 @@ public static class MimirPerfectMachineProfiles
PhoneWitness,
MicrocontrollerWitness,
CalibrationBench,
ObsProgramHost
MuninnHost,
MimirProgramPublisher
];
}

View file

@ -0,0 +1,141 @@
namespace Mimir.Runtime.Synchronization;
public enum MimirProgramVideoPublicationKind
{
FensalirProgramTexture,
SharedD3D12Texture,
CultMeshMediaBody,
SrtDiagnosticBridge,
ObsCompatibilityAdapter
}
public enum MimirProgramAudioPublicationKind
{
FaustStemBus,
AsioOutputPair,
CultMeshMediaBody,
WasapiLoopbackDiagnostic,
ObsCompatibilityAdapter
}
public enum MimirProgramControlPublicationKind
{
CultMeshCommandSurface
}
public sealed record MimirProgramAudioStem(
string Id,
string DisplayName,
int ChannelCount,
bool UserMixable,
bool CarriesTimingWitness,
string Notes);
public sealed record MimirProgramControlSurface(
string Id,
string Description,
MimirProgramControlPublicationKind Kind,
bool CanPreviewProgram,
bool CanEditComposition,
bool CanSelectSources,
bool CanShowStats);
public sealed record MimirProgramPublicationConfiguration(
string Id,
string Description,
MimirProgramVideoPublicationKind VideoKind,
string VideoSurfaceName,
MimirProgramAudioPublicationKind AudioKind,
IReadOnlyList<MimirProgramAudioStem> AudioStems,
IReadOnlyList<MimirProgramControlSurface> ControlSurfaces,
bool PublishesSiteFeed,
string SitePublisherRoute,
double TargetPresentationDelaySeconds,
bool DiagnosticOnly);
public static class MimirProgramPublicationConfigurations
{
public static IReadOnlyList<MimirProgramAudioStem> ProgramStems { get; } =
[
new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."),
new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."),
new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."),
new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."),
new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."),
new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."),
new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.")
];
public static IReadOnlyList<MimirProgramControlSurface> OperatorSurfaces { get; } =
[
new(
"cultmesh-command-surface",
"Typed CultMesh command surface consumed by trusted devices; Mimir remains the scene authority.",
MimirProgramControlPublicationKind.CultMeshCommandSurface,
CanPreviewProgram: false,
CanEditComposition: true,
CanSelectSources: true,
CanShowStats: true)
];
public static MimirProgramPublicationConfiguration NativeProgram { get; } = new(
"mimir-native-program",
"Production target: Mimir owns composition, Fensalir publishes synchronized program video, Faust publishes aligned stems, and Eve exposes control/preview/stat surfaces.",
MimirProgramVideoPublicationKind.FensalirProgramTexture,
"Mimir Program Texture",
MimirProgramAudioPublicationKind.FaustStemBus,
ProgramStems,
OperatorSurfaces,
PublishesSiteFeed: false,
SitePublisherRoute: "",
TargetPresentationDelaySeconds: 5.0,
DiagnosticOnly: false);
public static MimirProgramPublicationConfiguration YggdrasilSiteProgram { get; } = NativeProgram with
{
Id = "mimir-yggdrasil-site-program",
Description = "Site publishing target: a Yggdrasil-facing daemon consumes the same Mimir program output and publishes it to the public site without owning a second composition.",
VideoKind = MimirProgramVideoPublicationKind.CultMeshMediaBody,
AudioKind = MimirProgramAudioPublicationKind.CultMeshMediaBody,
PublishesSiteFeed = true,
SitePublisherRoute = "cultmesh://yggdrasil/gamecult.site/mimir/program"
};
public static MimirProgramPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with
{
Id = "native-program-d3d12-shared-texture",
VideoKind = MimirProgramVideoPublicationKind.SharedD3D12Texture,
VideoSurfaceName = "Mimir D3D12 Program Texture"
};
public static MimirProgramPublicationConfiguration ObsCompatibilityAdapter { get; } = new(
"obs-compatibility-adapter",
"Temporary OBS compatibility sink for preview/testing while Mimir owns composition and Yggdrasil/site publication matures.",
MimirProgramVideoPublicationKind.ObsCompatibilityAdapter,
"Mimir OBS Compatibility Feed",
MimirProgramAudioPublicationKind.ObsCompatibilityAdapter,
ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(),
OperatorSurfaces,
PublishesSiteFeed: false,
SitePublisherRoute: "",
TargetPresentationDelaySeconds: 1.5,
DiagnosticOnly: true);
public static MimirProgramPublicationConfiguration DiagnosticSrtBridge { get; } = ObsCompatibilityAdapter with
{
Id = "diagnostic-srt-bridge",
Description = "Legacy SRT bridge for bridge tests only. It is not scene, composition, or broadcast authority.",
VideoKind = MimirProgramVideoPublicationKind.SrtDiagnosticBridge,
AudioKind = MimirProgramAudioPublicationKind.WasapiLoopbackDiagnostic,
VideoSurfaceName = "Mimir Diagnostic SRT"
};
public static IReadOnlyList<MimirProgramPublicationConfiguration> BuiltIn { get; } =
[
NativeProgram,
YggdrasilSiteProgram,
TextureInteropProgram,
ObsCompatibilityAdapter,
DiagnosticSrtBridge
];
}

View file

@ -32,8 +32,9 @@ public static class MimirReservoirConfigurations
new("audio_block", "PCM/native audio block", "Mimir.Runtime/native audio", RetainsHistory: false, "Timing and field evidence."),
new("phase_claim", "audio phase/group-delay evidence", "Mimir.Runtime", RetainsHistory: false, "Calibration/normalization input."),
new("event_claim", "timeline anchor", "Mimir.Runtime", RetainsHistory: false, "Bioacoustic/passive/calibration events."),
new("move_evidence", "PS Move marker/controller evidence buffer", "Mimir native reservoir", RetainsHistory: false, "Compute-upload buffer of Muninn optical and controller witnesses before Mimir pose fusion."),
new("surface_claim", "spatial surface observation", "Fensalir", RetainsHistory: false, "Fusion candidate."),
new("render_packet", "OBS-facing packet", "Fensalir/Faust", RetainsHistory: false, "Output-ready view.")
new("render_packet", "Mimir program packet", "Mimir/Fensalir/Faust", RetainsHistory: false, "Output-ready program view.")
];
public static MimirReservoirConfiguration ManagedRuntime { get; } = new(

View file

@ -8,6 +8,8 @@ public sealed class MimirRuntimeConfiguration
public IReadOnlyList<MimirStreamSourceFactory> SourceFactories { get; init; } = [];
public IReadOnlyList<MimirMoveProofRuntimeConfiguration> MoveProofSources { get; init; } = [];
public IReadOnlyList<IMimirStreamSource> CreateSources()
{
return SourceFactories
@ -53,6 +55,16 @@ public sealed class MimirRuntimeConfiguration
.Where(factory => factory != null)
.Cast<MimirStreamSourceFactory>()
.ToArray();
var moveProofSources = model.MoveProof
.Where(source => source.Enabled)
.ToArray();
var moveProofErrors = moveProofSources
.SelectMany((source, index) => source.Validate().Select(error => $"MoveProof[{index}].{error}"))
.ToArray();
if (moveProofErrors.Length > 0)
{
throw new InvalidOperationException($"Invalid Mimir runtime Move proof configuration: {string.Join("; ", moveProofErrors)}");
}
return new MimirRuntimeConfiguration
{
@ -63,6 +75,7 @@ public sealed class MimirRuntimeConfiguration
Streams = streams,
},
SourceFactories = sourceFactories,
MoveProofSources = moveProofSources,
};
}
@ -224,6 +237,8 @@ public sealed class MimirRuntimeConfigFile
public MimirAudioSyncConfig? AudioSync { get; set; }
public List<MimirStreamConfig> Streams { get; set; } = [];
public List<MimirMoveProofRuntimeConfiguration> MoveProof { get; set; } = [];
}
public sealed class MimirAudioSyncConfig

View file

@ -0,0 +1,210 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
public enum MimirSensorCalibrationSessionPhase
{
Collecting,
Fitting,
Validating,
Promoted,
Rejected,
Canceled
}
[CultDocument("mimir.sensor_calibration_session", "mimir.sensor_calibration_session.v1")]
[MessagePackObject]
public sealed record MimirSensorCalibrationSessionDocument(
[property: Key(0)][property: CultName] string SessionId,
[property: Key(1)] string TrackingSpaceId,
[property: Key(2)] string Owner,
[property: Key(3)] string StartedAtUtc,
[property: Key(4)] double MaximumDurationSeconds,
[property: Key(5)] string[] RequiredWandIds,
[property: Key(6)] string[] SensorIds,
[property: Key(7)] MimirSensorCalibrationAcceptance Acceptance,
[property: Key(8)] MimirSensorCalibrationSessionPhase Phase,
[property: Key(9)] MimirSensorCalibrationSensorProgress[] Sensors,
[property: Key(10)] string[] MissingRequirements,
[property: Key(11)] string Detail);
[MessagePackObject]
public sealed record MimirSensorCalibrationAcceptance(
[property: Key(0)] int GridColumns,
[property: Key(1)] int GridRows,
[property: Key(2)] int MinimumOccupiedGridCellsPerSensor,
[property: Key(3)] int MinimumObservationsPerSensor,
[property: Key(4)] int MinimumSameFrameCorrespondences,
[property: Key(5)] int MinimumDistinctWands,
[property: Key(6)] double MinimumRadiusRatio,
[property: Key(7)] double HeldOutFraction,
[property: Key(8)] double MaximumMedianReprojectionErrorPx,
[property: Key(9)] double MaximumP95ReprojectionErrorPx,
[property: Key(10)] double MaximumAssociationSkewMilliseconds);
[MessagePackObject]
public sealed record MimirSensorCalibrationSensorProgress(
[property: Key(0)] string SensorId,
[property: Key(1)] int ObservationCount,
[property: Key(2)] int OccupiedGridCells,
[property: Key(3)] int GridCellCount,
[property: Key(4)] int DistinctWandCount,
[property: Key(5)] double MinimumRadiusPx,
[property: Key(6)] double MaximumRadiusPx,
[property: Key(7)] int[] MissingGridCells,
[property: Key(8)] bool CollectionComplete);
[CultDocument("mimir.sensor_calibration_receipt", "mimir.sensor_calibration_receipt.v1")]
[MessagePackObject]
public sealed record MimirSensorCalibrationReceiptDocument(
[property: Key(0)][property: CultName] string ReceiptId,
[property: Key(1)] string SessionId,
[property: Key(2)] string TrackingSpaceId,
[property: Key(3)] string CompletedAtUtc,
[property: Key(4)] MimirSensorCalibrationSessionPhase Verdict,
[property: Key(5)] int TrainingCorrespondenceCount,
[property: Key(6)] int HeldOutCorrespondenceCount,
[property: Key(7)] MimirSensorCalibrationFitResiduals Residuals,
[property: Key(8)] MimirMoveFusionRigCalibration? PromotedCalibration,
[property: Key(9)] string[] RejectionReasons,
[property: Key(10)] string Solver);
[MessagePackObject]
public sealed record MimirSensorCalibrationFitResiduals(
[property: Key(0)] double TrainingMedianReprojectionErrorPx,
[property: Key(1)] double TrainingP95ReprojectionErrorPx,
[property: Key(2)] double HeldOutMedianReprojectionErrorPx,
[property: Key(3)] double HeldOutP95ReprojectionErrorPx,
[property: Key(4)] double MedianAssociationSkewMilliseconds,
[property: Key(5)] double MaximumAssociationSkewMilliseconds,
[property: Key(6)] double ConditionEstimate,
[property: Key(7)] int Iterations,
[property: Key(8)] double TrainingInlierFraction = 0.0,
[property: Key(9)] double HeldOutInlierFraction = 0.0);
public static class MimirSensorCalibrationSessions
{
public static MimirSensorCalibrationSessionDocument CreateFourWandOpticalSession(
IEnumerable<string> wandIds,
IEnumerable<string> sensorIds,
DateTimeOffset? startedAt = null,
double maximumDurationSeconds = 120.0)
{
var wands = wandIds.Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.Ordinal).Order().ToArray();
var sensors = sensorIds.Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.Ordinal).Order().ToArray();
if (wands.Length != 4) throw new ArgumentException("A four-wand calibration session requires exactly four stable wand IDs.", nameof(wandIds));
if (sensors.Length < 2) throw new ArgumentException("Optical calibration requires at least two sensors.", nameof(sensorIds));
if (!double.IsFinite(maximumDurationSeconds) || maximumDurationSeconds <= 0.0) throw new ArgumentOutOfRangeException(nameof(maximumDurationSeconds));
var start = startedAt ?? DateTimeOffset.UtcNow;
return new MimirSensorCalibrationSessionDocument(
$"mimir-sensor-calibration:{start.ToUnixTimeMilliseconds()}",
"mimir-stage-space",
"Mimir.Runtime",
start.ToString("O"),
maximumDurationSeconds,
wands,
sensors,
new MimirSensorCalibrationAcceptance(
GridColumns: 4,
GridRows: 3,
MinimumOccupiedGridCellsPerSensor: 9,
MinimumObservationsPerSensor: 1_000,
MinimumSameFrameCorrespondences: 500,
MinimumDistinctWands: 4,
MinimumRadiusRatio: 1.5,
HeldOutFraction: 0.2,
MaximumMedianReprojectionErrorPx: 1.5,
MaximumP95ReprojectionErrorPx: 4.0,
MaximumAssociationSkewMilliseconds: 20.0),
MimirSensorCalibrationSessionPhase.Collecting,
sensors.Select(sensor => new MimirSensorCalibrationSensorProgress(
sensor, 0, 0, 12, 0, double.NaN, double.NaN, Enumerable.Range(0, 12).ToArray(), false)).ToArray(),
["wand-motion-not-yet-captured"],
"Wave one or two uniquely identified wands at a time through the shared sensor volume; Mimir accumulates successive passes until every wand and sensor has sufficient coverage.");
}
public static MimirSensorCalibrationSessionDocument UpdateCollection(
MimirSensorCalibrationSessionDocument session,
MoveVisibilityWindowReceipt evidence,
int imageWidth,
int imageHeight,
DateTimeOffset? observedAt = null)
{
ArgumentNullException.ThrowIfNull(session);
ArgumentNullException.ThrowIfNull(evidence);
if (session.Phase != MimirSensorCalibrationSessionPhase.Collecting) return session;
if (imageWidth <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth));
if (imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageHeight));
var acceptance = session.Acceptance;
var requiredWands = session.RequiredWandIds.ToHashSet(StringComparer.Ordinal);
var progress = session.SensorIds.Select(sensorId =>
{
var observations = evidence.Observations.Where(value =>
string.Equals(value.CameraId, sensorId, StringComparison.Ordinal) &&
requiredWands.Contains(value.MoveId) &&
float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) &&
float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f).ToArray();
var occupied = observations.Select(value => GridCell(
value.CenterXPx, value.CenterYPx, imageWidth, imageHeight,
acceptance.GridColumns, acceptance.GridRows)).Distinct().Order().ToArray();
var missing = Enumerable.Range(0, acceptance.GridColumns * acceptance.GridRows)
.Except(occupied).ToArray();
var minimumRadius = observations.Length == 0 ? double.NaN : observations.Min(value => value.RadiusPx);
var maximumRadius = observations.Length == 0 ? double.NaN : observations.Max(value => value.RadiusPx);
var radiusRatio = observations.Length == 0 ? 0.0 : maximumRadius / minimumRadius;
var distinctWands = observations.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count();
var complete = observations.Length >= acceptance.MinimumObservationsPerSensor &&
occupied.Length >= acceptance.MinimumOccupiedGridCellsPerSensor &&
distinctWands >= acceptance.MinimumDistinctWands &&
radiusRatio >= acceptance.MinimumRadiusRatio;
return new MimirSensorCalibrationSensorProgress(
sensorId, observations.Length, occupied.Length,
acceptance.GridColumns * acceptance.GridRows, distinctWands,
minimumRadius, maximumRadius, missing, complete);
}).ToArray();
var correspondenceCount = evidence.Correspondences.Count(value =>
requiredWands.Contains(value.MoveId) &&
value.AbsoluteSkewNs <= acceptance.MaximumAssociationSkewMilliseconds * 1_000_000.0 &&
string.Equals(value.First.FrameId, value.Second.FrameId, StringComparison.Ordinal));
var missingRequirements = new List<string>();
foreach (var sensor in progress)
{
if (sensor.ObservationCount < acceptance.MinimumObservationsPerSensor)
missingRequirements.Add($"{sensor.SensorId}:observations-{sensor.ObservationCount}-of-{acceptance.MinimumObservationsPerSensor}");
if (sensor.OccupiedGridCells < acceptance.MinimumOccupiedGridCellsPerSensor)
missingRequirements.Add($"{sensor.SensorId}:grid-cells-{sensor.OccupiedGridCells}-of-{acceptance.MinimumOccupiedGridCellsPerSensor}");
if (sensor.DistinctWandCount < acceptance.MinimumDistinctWands)
missingRequirements.Add($"{sensor.SensorId}:wands-{sensor.DistinctWandCount}-of-{acceptance.MinimumDistinctWands}");
var radiusRatio = sensor.ObservationCount == 0 ? 0.0 : sensor.MaximumRadiusPx / sensor.MinimumRadiusPx;
if (radiusRatio < acceptance.MinimumRadiusRatio)
missingRequirements.Add($"{sensor.SensorId}:radius-ratio-{radiusRatio:F2}-of-{acceptance.MinimumRadiusRatio:F2}");
}
if (correspondenceCount < acceptance.MinimumSameFrameCorrespondences)
missingRequirements.Add($"same-frame-correspondences-{correspondenceCount}-of-{acceptance.MinimumSameFrameCorrespondences}");
var elapsed = (observedAt ?? DateTimeOffset.UtcNow) - DateTimeOffset.Parse(session.StartedAtUtc);
var collectionComplete = progress.All(value => value.CollectionComplete) &&
correspondenceCount >= acceptance.MinimumSameFrameCorrespondences;
var expired = elapsed.TotalSeconds >= session.MaximumDurationSeconds;
var phase = collectionComplete
? MimirSensorCalibrationSessionPhase.Fitting
: expired ? MimirSensorCalibrationSessionPhase.Rejected : MimirSensorCalibrationSessionPhase.Collecting;
var detail = phase switch
{
MimirSensorCalibrationSessionPhase.Fitting => "Collection complete; Mimir owns the retained evidence and may begin fitting.",
MimirSensorCalibrationSessionPhase.Rejected => "Collection window expired before coverage requirements were met; no calibration was promoted.",
_ => "Collection active. Continue sweeping one or two wands at a time through missing image regions and depth ranges."
};
return session with { Phase = phase, Sensors = progress, MissingRequirements = missingRequirements.ToArray(), Detail = detail };
}
private static int GridCell(float x, float y, int width, int height, int columns, int rows)
{
var column = Math.Clamp((int)(x * columns / width), 0, columns - 1);
var row = Math.Clamp((int)(y * rows / height), 0, rows - 1);
return row * columns + column;
}
}

View file

@ -4,6 +4,7 @@ public enum MimirStreamKind
{
Audio,
Video,
Tracking,
}
public enum MimirStreamOrigin

View file

@ -11,4 +11,5 @@ public readonly record struct MimirStreamSample(
int ByteLength = 0,
ReadOnlyMemory<byte> Data = default,
MimirVideoFrameDescriptor? VideoFrame = null,
MimirAudioBlockDescriptor? AudioBlock = null);
MimirAudioBlockDescriptor? AudioBlock = null,
MimirTrackingObservation? TrackingObservation = null);

View file

@ -51,6 +51,15 @@ public sealed class MimirSynchronizationHub : IDisposable
public IReadOnlyList<MimirChirpBinCalibrationProfile> AudioChirpBinCalibrationProfiles =>
audioSynchronization.LastCalibrationProfiles;
public IReadOnlyList<MimirTrackingObservation> TrackingObservations =>
Buffers.Buffers
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking)
.Select(buffer => buffer.Latest?.TrackingObservation)
.Where(observation => observation != null)
.Cast<MimirTrackingObservation>()
.OrderBy(observation => observation.StreamId, StringComparer.Ordinal)
.ToArray();
public MimirChirpBinCodebookPlan? ChirpBinEmissionPlan =>
chirpBinCalibrationModel?.EmissionPlan;
@ -87,7 +96,8 @@ public sealed class MimirSynchronizationHub : IDisposable
{
var audio = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio);
var video = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Video);
return $"{video} video / {audio} audio buffers";
var tracking = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking);
return $"{video} video / {audio} audio / {tracking} tracking buffers";
}
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronization(

View file

@ -14,8 +14,10 @@ public sealed class MimirSynchronizationSettings
var streams = new List<MimirStreamDescriptor>();
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.LocalDevice);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.LocalDevice);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.LocalDevice);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.Network);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.Network);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.Network);
return new MimirSynchronizationSettings
{

View file

@ -0,0 +1,156 @@
using MessagePack;
using GameCult.Caching;
namespace Mimir.Runtime.Synchronization;
public enum MimirTrackingObservationKind
{
Unknown,
PsMoveController,
Hand,
Headset,
Tool,
}
[MessagePackObject]
public readonly record struct MimirVector3Snapshot(
[property: Key(0)] double X,
[property: Key(1)] double Y,
[property: Key(2)] double Z);
[MessagePackObject]
public readonly record struct MimirQuaternionSnapshot(
[property: Key(0)] double X,
[property: Key(1)] double Y,
[property: Key(2)] double Z,
[property: Key(3)] double W);
[MessagePackObject]
public sealed record MimirTrackingButtonSnapshot(
[property: Key(0)] string Id,
[property: Key(1)] bool Pressed,
[property: Key(2)] double Value);
[MessagePackObject]
[CultDocument("mimir.move_tracking_observation", "mimir.move_tracking_observation.v1")]
public sealed record MimirTrackingObservation(
[property: Key(0)]
[property: CultName]
string ObservationId,
[property: Key(1)] string StreamId,
[property: Key(2)] string DeviceId,
[property: Key(3)] MimirTrackingObservationKind Kind,
[property: Key(4)] string ProducerId,
[property: Key(5)] string HostId,
[property: Key(6)] string DiscoveryProviderId,
[property: Key(7)] string TrackingSpaceId,
[property: Key(8)] string CalibrationId,
[property: Key(9)] long SourceTimestampNs,
[property: Key(10)] long ArrivalTimestampNs,
[property: Key(11)] ulong Sequence,
[property: Key(12)] MimirVector3Snapshot PositionMeters,
[property: Key(13)] MimirQuaternionSnapshot Orientation,
[property: Key(14)] MimirVector3Snapshot LinearVelocityMetersPerSecond,
[property: Key(15)] MimirVector3Snapshot AngularVelocityRadiansPerSecond,
[property: Key(16)] double Confidence,
[property: Key(17)] double LatencyMilliseconds,
[property: Key(18)] double Battery01,
[property: Key(19)] MimirTrackingButtonSnapshot[] Buttons)
{
public static MimirTrackingObservation PsMove(
string streamId,
string deviceId,
ulong sequence,
long sourceTimestampNs,
long arrivalTimestampNs,
MimirVector3Snapshot positionMeters,
MimirQuaternionSnapshot orientation,
MimirVector3Snapshot linearVelocityMetersPerSecond,
MimirVector3Snapshot angularVelocityRadiansPerSecond,
double confidence,
string calibrationId,
string trackingSpaceId = "nightwing-move-space",
string producerId = "muninn:nightwing:move",
string hostId = "nightwing",
string discoveryProviderId = "odin",
double latencyMilliseconds = 0.0,
double battery01 = double.NaN,
MimirTrackingButtonSnapshot[]? buttons = null) =>
new(
$"{streamId}:{sequence}",
streamId,
deviceId,
MimirTrackingObservationKind.PsMoveController,
producerId,
hostId,
discoveryProviderId,
trackingSpaceId,
calibrationId,
sourceTimestampNs,
arrivalTimestampNs,
sequence,
positionMeters,
orientation,
linearVelocityMetersPerSecond,
angularVelocityRadiansPerSecond,
Math.Clamp(confidence, 0.0, 1.0),
Math.Max(0.0, latencyMilliseconds),
double.IsFinite(battery01) ? Math.Clamp(battery01, 0.0, 1.0) : double.NaN,
buttons ?? []);
}
[CultDocument("mimir.move_controller_pose", "mimir.move_controller_pose.v1")]
[MessagePackObject]
public sealed record MimirMoveControllerPoseDocument(
[property: Key(0)]
[property: CultName]
string PoseId,
[property: Key(1)] string WandId,
[property: Key(2)] string TrackingSpaceId,
[property: Key(3)] string CalibrationId,
[property: Key(4)] string FusionAuthorityId,
[property: Key(5)] long SourceTimestampNs,
[property: Key(6)] long EstimatedAtNs,
[property: Key(7)] ulong Sequence,
[property: Key(8)] MimirVector3Snapshot PositionMeters,
[property: Key(9)] MimirQuaternionSnapshot Orientation,
[property: Key(10)] MimirVector3Snapshot LinearVelocityMetersPerSecond,
[property: Key(11)] MimirVector3Snapshot AngularVelocityRadiansPerSecond,
[property: Key(12)] double Confidence,
[property: Key(13)] double LatencyMilliseconds,
[property: Key(14)] double Battery01,
[property: Key(15)] MimirTrackingButtonSnapshot[] Buttons,
[property: Key(16)] string[] EvidenceStreamIds,
[property: Key(17)] string[] EvidenceKinds,
[property: Key(18)] string ConsumerContract)
{
public static MimirMoveControllerPoseDocument FromObservation(
MimirTrackingObservation observation,
string wandId,
string[] evidenceStreamIds,
string[] evidenceKinds,
string calibrationId,
string trackingSpaceId = "mimir-stage-space",
string fusionAuthorityId = "mimir.runtime.move-fusion",
string consumerContract = "fensalir.move-controller-input") =>
new(
$"{wandId}:{observation.Sequence}",
wandId,
trackingSpaceId,
calibrationId,
fusionAuthorityId,
observation.SourceTimestampNs,
observation.ArrivalTimestampNs,
observation.Sequence,
observation.PositionMeters,
observation.Orientation,
observation.LinearVelocityMetersPerSecond,
observation.AngularVelocityRadiansPerSecond,
Math.Clamp(observation.Confidence, 0.0, 1.0),
Math.Max(0.0, observation.LatencyMilliseconds),
double.IsFinite(observation.Battery01) ? Math.Clamp(observation.Battery01, 0.0, 1.0) : double.NaN,
observation.Buttons,
evidenceStreamIds,
evidenceKinds,
consumerContract);
}

View file

@ -0,0 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>Mimir.VerseRecorder</AssemblyName>
<RootNamespace>Mimir.VerseRecorder</RootNamespace>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>

Some files were not shown because too many files have changed in this diff Show more