Compare commits

...
Author SHA1 Message Date
Meta Vens
c6f4541eea State: EveSensorReceiver is live and frozen until Thing 2, not archived (thing2:ruling:mimir-sensor-receiver-frozen)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 23:25:39 +02:00
Meta Vens
43cebb6fb9 Quarantine Mimir's Eve operator surface: remove EveDashboard, EveBrowserReference and their contract documents
Parked at tag parked/Mimir-eve-surface per thing2:ruling:eve-dashboards-quarantined; rebuild on Thing 2. Mimir.EveSensorReceiver (camera frame ingest) stays. map.yaml no longer calls a live listener a tombstone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 20:14:47 +02:00
Meta Vens
9481fbaf44 Merge readme-front-door: README opens with value, problem, entry points, live versus planned 2026-10-22 20:05:05 +02:00
Meta Vens
e867e44cbc README: front door with value, problem, entry points, live versus planned; quick start copies the example config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:04:52 +02:00
Meta Vens
c08e1a16cd Pin CultLib 2.0.0 for builds
Directory.Build.props names CultLibRoot (CULTLIB_ROOT, else the sibling checkout) and pins CultLibRevision to cultlib-unity-v2.0.0 (42dce361). Directory.Build.targets fails the build unless that checkout is at the pin and clean, as Aetheria's guard does. Project references use CultLibRoot.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 19:46:49 +02:00
Meta Vens
d88f2e6dc1 Flush the media cache without the ignored soft flag
CultCache.FlushAsync(bool soft) ignored soft and CultLib is deleting the
parameter; the media put path now calls FlushAsync().

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 14:12:28 +02:00
Meta Vens
5591c5c47c Follow CultLib generator deletion: always serialize through MessagePack
CultDocumentDescriptor no longer carries a generated payload serializer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 23:38:14 +02:00
Meta Vens
aa36b67dd6 Follow CultLib Cut 3: drop PullOnOpen from BufferSmoke
CultLib's CultCacheMessagePack.OpenAsync now always hydrates and
CultCacheOpenOptions.PullOnOpen no longer exists. The five BufferSmoke opens
passed PullOnOpen = true or File.Exists(path); hydrating a missing store is
empty, so each becomes a plain OpenAsync(path).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 17:59:50 +02:00
Meta Vens
4d7378216f Wire AGENTS.md into Claude Code memory
Claude Code reads CLAUDE.md, not AGENTS.md, so this repo's doctrine was
invisible to it. This one-line import makes the existing AGENTS.md load
without duplicating or renaming it. Codex is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:23:05 +02:00
Meta Vens
ce98c421dc Record mixed-stream input latency proof 2026-07-17 11:20:30 +01:00
Meta Vens
7468ba59bc Record ordered input field closure 2026-07-17 10:40:06 +01:00
Meta Vens
fea4d27dd1 Document Raven Moonlight field acceptance 2026-07-17 08:12:21 +01:00
Meta Vens
78ca783276 gate Muninn decode epochs and retry repair 2026-07-17 07:24:41 +01:00
Meta Vens
42d6d173c3 recover compressed Muninn media under loss 2026-07-17 06:48:07 +01:00
Meta Vens
1221583c23 Harden Muninn OBS recovery from Moonlight field study 2026-07-17 05:48:31 +01:00
Meta Vens
12489075d7 Record Muninn frame-wide FEC limit 2026-07-17 02:58:49 +01:00
Meta Vens
02c615670a Ignore late shards from completed Muninn frames 2026-07-17 02:34:51 +01:00
Meta Vens
79781abdbb Record Muninn trailing parity burst scar 2026-07-17 01:59:33 +01:00
Meta Vens
e9e7eaae30 Bound Muninn video FEC decode time 2026-07-17 01:46:28 +01:00
Meta Vens
afce0245b1 Recover Muninn video bursts with bounded FEC 2026-07-17 01:37:28 +01:00
Meta Vens
caf92d9dee Make Muninn OBS receiver reload-safe 2026-07-16 23:10:56 +01:00
Meta Vens
60cbe45624 Record Raven field-test outage 2026-07-16 21:38:24 +01:00
Meta Vens
b0f33c36b4 Define Moonlight reliability acceptance gate 2026-07-16 21:35:47 +01:00
Meta Vens
d3d461be93 Record receiver-driven IDR proof 2026-07-16 21:26:00 +01:00
Meta Vens
6a7aaea019 Record Muninn loss recovery authority 2026-07-16 21:04:52 +01:00
Meta Vens
58e5556322 Recover bounded Muninn audio loss 2026-07-16 21:02:01 +01:00
Meta Vens
074ad4a69b Repair Muninn receiver recovery contracts 2026-07-16 20:49:17 +01:00
Meta Vens
45ead73ae3 Record realtime delivery hardening 2026-07-16 20:30:10 +01:00
Meta Vens
5706fcd99d Persist Mimir Face state 2026-07-16 20:04:09 +01:00
Meta Vens
c3775ec150 Document Moonlight streaming lessons 2026-07-16 20:04:05 +01:00
Meta Vens
345c46bd95 Reconnect Move evidence after provider restart 2026-07-14 01:31:45 +01:00
Meta Vens
a363c1d32f Update Mimir Face state 2026-07-14 01:13:44 +01:00
Meta Vens
0efa7f3307 Record filtered Move observation evidence 2026-07-14 01:13:41 +01:00
Meta Vens
2eb32602d7 Update Mimir Face state 2026-07-14 01:00:30 +01:00
Meta Vens
58af018125 Add robust Move sphere stereo fitting 2026-07-14 01:00:17 +01:00
Meta Vens
c01ef5096c Add Mimir-owned Move calibration sessions 2026-07-14 00:35:47 +01:00
Meta Vens
11e3086ee4 Preserve Mimir Face state 2026-07-13 02:39:46 +01:00
Meta Vens
d27f0d2d37 Gate Move stereo calibration on observable evidence 2026-07-13 02:39:41 +01:00
Meta Vens
a741aef9bb Preserve Mimir Face state 2026-07-13 02:32:44 +01:00
Meta Vens
95bed8f5f4 Record stable dual-Eye Move cadence proof 2026-07-13 02:32:38 +01:00
Meta Vens
83f89d4020 Record full dual-eye Move overlap 2026-07-13 01:41:29 +01:00
Meta Vens
810073efaa Preserve Mimir Face state 2026-07-13 01:29:25 +01:00
Meta Vens
2196503d54 Map live dual-eye Move tracking state 2026-07-13 01:29:16 +01:00
Meta Vens
c1939b72f5 Remember isolated dual-eye tracker field proof 2026-07-13 01:15:00 +01:00
Meta Vens
bf110a6ace Remember dual-eye PSMoveAPI isolation proof 2026-07-13 00:48:00 +01:00
Meta Vens
e757538eda Expose live Move evidence receiver failures 2026-07-13 00:29:24 +01:00
Meta Vens
150a054344 Measure Move visibility within one Muninn provider 2026-07-13 00:22:09 +01:00
Meta Vens
f80881e0d7 Measure persistent dual-camera Move visibility 2026-07-13 00:17:00 +01:00
Meta Vens
3e627f5484 Remember Move tracking false blocker 2026-07-13 00:10:14 +01:00
Meta Vens
ee456bbcc1 Probe arbitrary Muninn Move evidence providers 2026-07-13 00:01:52 +01:00
Meta Vens
f742a4de97 Require distinct cameras in live optical smoke 2026-07-12 22:49:03 +01:00
Meta Vens
c815d1ba58 Wait for stable-ID optical Move evidence 2026-07-12 21:39:24 +01:00
Meta Vens
74a1918996 Associate Move optical markers by stable identity 2026-07-12 21:11:27 +01:00
Meta Vens
522bf41234 Record live Move Bluetooth handoff 2026-07-12 21:00:36 +01:00
Meta Vens
6778e94975 Update Mimir Face state 2026-07-12 20:50:41 +01:00
Meta Vens
95d603aa80 Record live Muninn Move migration evidence 2026-07-12 20:50:41 +01:00
Meta Vens
538b95e2c3 Discover Muninn Move evidence through Odin 2026-07-12 20:18:30 +01:00
Meta Vens
de22059725 Persist Mimir Face state 2026-07-12 11:17:25 +01:00
Meta Vens
ec86a482a0 Record Muninn Move proof snapshot producer 2026-07-12 11:17:21 +01:00
Meta Vens
c0e53135ba Persist Mimir Face state 2026-07-12 11:10:21 +01:00
Meta Vens
dd45a478a6 Add Move proof evidence snapshot fallback 2026-07-12 11:10:17 +01:00
Meta Vens
2de98df804 Persist Mimir Face state 2026-07-12 11:01:39 +01:00
Meta Vens
32bf6fa518 Add Move proof activation surface 2026-07-12 11:01:29 +01:00
Meta Vens
4a968319ac Persist Mimir Face heartbeat 2026-07-12 10:53:52 +01:00
Meta Vens
c32d01b7e5 Activate configured Move proof sources 2026-07-12 10:53:37 +01:00
Meta Vens
dc51918227 Persist Mimir Face state 2026-07-12 10:45:50 +01:00
Meta Vens
83ddd3f509 Add Move proof runtime configuration 2026-07-12 10:41:48 +01:00
Meta Vens
293a953b2d Prove Move proof surface in presented frame 2026-07-08 22:56:52 +01:00
Meta Vens
20a296a4a9 Drive Move proof surface from runtime ring 2026-07-08 22:49:42 +01:00
Meta Vens
02c6bfc533 Attach Move proof surface to runtime frame 2026-07-08 22:42:20 +01:00
Meta Vens
7d96c42a3c Add Mimir Move proof pipeline 2026-07-08 22:35:32 +01:00
Meta Vens
a040792a1c Add Mimir Move proof surface 2026-07-08 22:29:23 +01:00
Meta Vens
7e2d35ad59 Make Muninn OBS bridge allocate decoder ports 2026-07-06 00:45:08 +01:00
Meta Vens
d6f6400677 Hide Muninn OBS CLI discovery launches 2026-07-05 23:42:45 +01:00
Meta Vens
ed46270be6 Make Muninn OBS discovery visibly Odin-owned 2026-07-05 23:35:32 +01:00
Meta Vens
e00458ff3b Restore Muninn OBS plugin with Odin discovery 2026-07-05 22:35:50 +01:00
Meta Vens
d9005e340b Ignore stale Muninn Move identities in live roster 2026-06-17 14:07:35 +01:00
Meta Vens
8042885c6d Record cross-host Move roster smoke 2026-06-17 13:29:44 +01:00
Meta Vens
c7fa6fb8ac Record Starfire Move paired host verification 2026-06-17 13:17:51 +01:00
Meta Vens
79e7865671 Record Nightwing Muninn hot release deploy 2026-06-17 12:41:59 +01:00
Meta Vens
d53ba0c82c Record Move wake blocker evidence 2026-06-17 12:15:07 +01:00
Meta Vens
6192330b49 Record unreachable Move roster verification 2026-06-17 12:02:11 +01:00
Meta Vens
32bd680cce Distinguish unreachable Muninn Moves 2026-06-17 11:55:20 +01:00
Meta Vens
5eef53c0e2 Record Muninn Move pickup verification 2026-06-17 11:33:34 +01:00
Meta Vens
d3d4b14635 Add live Muninn Move roster smoke 2026-06-17 10:16:09 +01:00
Meta Vens
ff0920b327 Clarify Muninn Move pickup readiness 2026-06-17 09:12:49 +01:00
Meta Vens
c3babfaa57 Project Muninn Move identity roster 2026-06-17 06:48:44 +01:00
Meta Vens
3103a7aa27 Ingest Muninn Move identity records 2026-06-17 03:49:55 +01:00
Meta Vens
b1a6043d7e Publish browser reference boundary witness 2026-06-16 20:31:31 +01:00
Meta Vens
4ed4ba190b Migrate Mimir media bridge to RUDP 2026-06-16 18:08:40 +01:00
Meta Vens
bd19e60ae1 Refresh Mimir Face state 2026-06-16 17:39:51 +01:00
Meta Vens
434e2d3efa Reorient Mimir maps around CultNet RUDP 2026-06-16 17:39:33 +01:00
Meta Vens
7d07e38e52 Run Nightwing Eve browser reference as RUDP daemon 2026-06-16 09:50:06 +01:00
Meta Vens
21c2ee43f0 Publish Nightwing Eve dashboard health over Idunn RUDP 2026-06-16 08:39:38 +01:00
Meta Vens
e3987842de Publish Mimir Eve dashboard health over Idunn RUDP 2026-06-16 08:04:25 +01:00
Meta Vens
34c435f942 Refresh Mimir Face state 2026-06-14 19:40:52 +01:00
Meta Vens
3ebf7cc519 Make Starfire Move light helper hotplug aware 2026-06-14 19:40:03 +01:00
Meta Vens
bff68e5b58 Add repeated PS Move read probe 2026-06-14 14:42:30 +01:00
Meta Vens
c23b392787 Mirror Muninn Move source path 2026-06-14 14:30:54 +01:00
Meta Vens
3ec2b01788 Refresh Mimir Face state 2026-06-14 14:30:34 +01:00
Meta Vens
21b30c6dd4 Update Mimir Face maintenance status 2026-06-12 16:42:04 +01:00
Meta Vens
c1178512c4 Move Quest access ownership to Muninn 2026-06-12 16:41:15 +01:00
Meta Vens
09d0d3ad2e Refresh Mimir Face maintenance status 2026-06-12 15:20:46 +01:00
Meta Vens
00c9053430 Add Quest USB calibration preflight 2026-06-12 15:20:19 +01:00
Meta Vens
0616d21a5b Use Quest as Move calibration witness 2026-06-12 15:04:51 +01:00
Meta Vens
4ae3a07ed6 Refresh Mimir Face heartbeat state 2026-06-12 14:50:53 +01:00
Meta Vens
74f2455e69 Add Move calibration protocol preflight 2026-06-12 14:50:12 +01:00
Meta Vens
510242bc50 Expose Move pose stream in module catalog 2026-06-12 14:26:50 +01:00
Meta Vens
dd838e9b1a Refresh Mimir Face state 2026-06-12 14:21:59 +01:00
Meta Vens
43af2a24bd Add calibrated Move pose fusion stream 2026-06-12 14:20:28 +01:00
Meta Vens
dccf3b1599 Persist Mimir face heartbeat 2026-06-11 20:33:51 +01:00
Meta Vens
07852766fd Use CultMesh stream for Muninn Move evidence 2026-06-11 20:28:30 +01:00
Meta Vens
6848f762f0 Persist Mimir face heartbeat 2026-06-11 20:04:33 +01:00
Meta Vens
091ab50d7c Map Muninn Move evidence into native tracking 2026-06-11 20:01:01 +01:00
Meta Vens
58c3c40b72 Wire Move evidence into native reservoir 2026-06-11 19:31:21 +01:00
Meta Vens
5b4c391607 Persist Mimir face state 2026-06-11 18:51:52 +01:00
Meta Vens
0d1f97436a Add native Move evidence reservoir ABI 2026-06-11 18:50:56 +01:00
Meta Vens
c9fca62346 Persist Mimir face state 2026-06-11 17:02:26 +01:00
Meta Vens
231573ed79 Move wand fusion authority to Mimir 2026-06-11 17:00:49 +01:00
Meta Vens
4d661967ae Route Move light ownership through Muninn 2026-06-11 07:58:26 +01:00
Meta Vens
ec0f7454d3 Persist Mimir face state 2026-06-11 07:57:59 +01:00
Meta Vens
147e6ab5af Update Mimir Face state 2026-06-10 20:52:58 +01:00
Meta Vens
8716ea9c0b Demote Move marker extraction to Muninn 2026-06-10 20:52:28 +01:00
Meta Vens
17d2f56de8 Update Mimir Face state 2026-06-10 20:36:21 +01:00
Meta Vens
c2e5d03764 Add Rust Move sphere candidate tracker 2026-06-10 20:35:35 +01:00
Meta Vens
1692c3ec64 Update Mimir Face state 2026-06-10 18:13:26 +01:00
Meta Vens
4af3864051 Add Starfire Move light owner 2026-06-10 18:12:32 +01:00
Meta Vens
601367b754 Update Mimir Face state 2026-06-10 17:42:28 +01:00
Meta Vens
52a439842c Restore Nightwing Move tracking witness 2026-06-10 17:41:18 +01:00
Meta Vens
e85c05029c Update Mimir Face state 2026-06-10 16:44:35 +01:00
Meta Vens
6dc18ecf18 Add Move tracking observation streams 2026-06-10 16:43:12 +01:00
Meta Vens
28536681f6 Import OBS scene into Mimir program state 2026-06-09 03:04:51 +01:00
Meta Vens
2c68485c35 Update Mimir Face state 2026-06-09 03:03:46 +01:00
Meta Vens
3dcc392080 Move program composition authority into Mimir 2026-06-09 02:21:32 +01:00
Meta Vens
7a811ee329 Update Mimir Face state 2026-06-09 02:19:17 +01:00
104 changed files with 20823 additions and 403 deletions

Binary file not shown.

View file

@ -1,17 +1,7 @@
{
"durationSeconds": 143.87,
"stateUpdated": true,
"exitCode": 0,
"finishedAt": "2026-05-20T13:30:04.9837661Z",
"stateFile": "E:\\Projects\\LocalCastBridge\\.voidbot\\state\\mimir.cc",
"skipModel": false,
"startedAt": "2026-05-20T13:27:41.1114814Z",
"lastMessagePath": "E:\\Projects\\LocalCastBridge\\.voidbot\\status\\void-memory-maintenance-last-message.txt",
"appliedOperationCount": 1,
"status": "ok",
"contextPath": "E:\\Projects\\LocalCastBridge\\.voidbot\\status\\void-memory-maintenance-context.json",
"mode": "typed_memory_maintenance",
"proposedOperationCount": 1,
"forceDistillation": true,
"operationOutputPath": "E:\\Projects\\LocalCastBridge\\.voidbot\\status\\void-memory-maintenance-operations.json"
}
"status": "starting",
"startedAt": "2026-07-15T02:30:47.016Z",
"stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc",
"launcher": "repo_face_heartbeat",
"reason": "repo_face_stale_short_term_memory"
}

View file

@ -6,7 +6,7 @@
"roleId": "1506268007880462459",
"avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png",
"repoPath": "E:\\Projects\\Mimir",
"createdOrRefreshedAt": "2026-05-25T05:17:24.094Z",
"createdOrRefreshedAt": "2026-07-15T01:59:42.286Z",
"avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png",
"description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen."
}

3
CLAUDE.md Normal file
View file

@ -0,0 +1,3 @@
Repository doctrine for this project lives in AGENTS.md, shared with Codex.
@AGENTS.md

10
Directory.Build.props Normal file
View file

@ -0,0 +1,10 @@
<Project>
<PropertyGroup>
<CultLibRoot Condition="'$(CultLibRoot)' == '' and '$(CULTLIB_ROOT)' != ''">$(CULTLIB_ROOT)</CultLibRoot>
<CultLibRoot Condition="'$(CultLibRoot)' == ''">$(MSBuildThisFileDirectory)..\CultLib</CultLibRoot>
<!-- CultLib 2.0.0: tag cultlib-unity-v2.0.0. -->
<CultLibRevision>42dce3611a81adc5c5fac54cadbaa5637d6022d3</CultLibRevision>
<GitExecutable Condition="'$(GitExecutable)' == '' and Exists('C:\Program Files\Git\cmd\git.exe')">C:\Program Files\Git\cmd\git.exe</GitExecutable>
<GitExecutable Condition="'$(GitExecutable)' == ''">git</GitExecutable>
</PropertyGroup>
</Project>

30
Directory.Build.targets Normal file
View file

@ -0,0 +1,30 @@
<Project>
<!-- Builds consume CultLib at one admitted revision. Point CultLibRoot (or CULTLIB_ROOT) at a clean
worktree of that revision: git worktree add <path> cultlib-unity-v2.0.0 (detached) (in CultLib). -->
<Target Name="VerifyCultLibRevision"
BeforeTargets="ResolveProjectReferences"
Condition="'$(SkipCultLibRevisionCheck)' != 'true'">
<ItemGroup>
<_CultLibReference Include="@(ProjectReference)"
Condition="$([System.String]::Copy('%(Filename)').StartsWith('GameCult.')) or '%(Filename)' == 'CultMath'" />
</ItemGroup>
<Error Condition="'@(_CultLibReference)' != '' and !Exists('$(CultLibRoot)\src\GameCult.Caching\GameCult.Caching.csproj')"
Text="CultLibRoot '$(CultLibRoot)' is not a CultLib checkout." />
<Exec Condition="'@(_CultLibReference)' != ''"
Command="&quot;$(GitExecutable)&quot; -C &quot;$(CultLibRoot)&quot; rev-parse HEAD"
ConsoleToMSBuild="true"
StandardOutputImportance="Low">
<Output TaskParameter="ConsoleOutput" PropertyName="ActualCultLibRevision" />
</Exec>
<Error Condition="'@(_CultLibReference)' != '' and '$([System.String]::Copy($(ActualCultLibRevision)).Trim())' != '$(CultLibRevision)'"
Text="CultLib must be revision $(CultLibRevision) (cultlib-unity-v2.0.0); found $([System.String]::Copy($(ActualCultLibRevision)).Trim()). Set CultLibRoot to a clean worktree of that tag." />
<Exec Condition="'@(_CultLibReference)' != ''"
Command="&quot;$(GitExecutable)&quot; -C &quot;$(CultLibRoot)&quot; status --porcelain --untracked-files=normal"
ConsoleToMSBuild="true"
StandardOutputImportance="Low">
<Output TaskParameter="ConsoleOutput" PropertyName="CultLibStatus" />
</Exec>
<Error Condition="'@(_CultLibReference)' != '' and '$([System.String]::Copy($(CultLibStatus)).Trim())' != ''"
Text="CultLibRoot '$(CultLibRoot)' must be clean before this repo consumes it." />
</Target>
</Project>

View file

@ -3,5 +3,9 @@
<Project Path="src/Mimir.App/Mimir.App.csproj" />
<Project Path="src/Mimir.Runtime/Mimir.Runtime.csproj" />
<Project Path="src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj" />
<Project Path="src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj" />
<Project Path="src/Mimir.VerseRecorder/Mimir.VerseRecorder.csproj" />
<Project Path="src/Mimir.PsMoveProbe/Mimir.PsMoveProbe.csproj" />
<Project Path="src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj" />
</Folder>
</Solution>

View file

@ -8,6 +8,18 @@ OBS-facing program surface.
The plain version: Mimir is trying to make a room computable.
## Front Door
- **Value:** a measured room: cameras, microphones, loopbacks and network feeds aligned in time tightly enough to recover where a sound came from and how a body moved, then handed to OBS as one program surface.
- **Problem:** loose devices each keep their own clock, so a flat stream cannot localize voices, separate sources or fuse cameras. Mimir owns the synchronization evidence that makes the feeds one field.
- **Entry points:**
- `docs/perfect-machine.md`: the target machine; `docs/viable-stream-app.md`: the first usable app cut; `docs/audio-field.md`: the live audio decoder and timing model.
- `src/Mimir.Runtime` (rolling buffers, stream sources, synchronization state) and `src/Mimir.App` (the Fensalir-hosted window and render app); solution `Mimir.slnx`.
- `src/Mimir.BufferSmoke`: the synthetic sync proofs, run with `dotnet run --project src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test` (also `--chirp-bin-self-test`, `--passive-sync-self-test`).
- `native/probes/asio_audio_cadence` (ASIO cadence probe) and `native/reservoir` (native rolling-buffer ABI).
- **Live:** passive, chirp-only and hybrid audio sync with synthetic proofs, the ASIO cadence probe and measured capture-cadence evidence, the runtime buffers, and FFmpeg/SRT bridge scripts as an OBS-compatible edge (`scripts/sender-start.ps1`).
- **Planned:** direct native capture workers with driver-boundary timestamps, Fensalir GPU fusion, the volumetric sound field, realtime 4D Gaussian splatting, motion capture and AR anchoring. The bridge scripts are not the synchronization authority.
A normal streaming setup sees loose devices. A webcam has its clock, a mic has
another clock, speakers add their own delay, and OBS receives whatever arrives.
That is enough for a flat livestream. It is not enough to know where a voice
@ -289,6 +301,7 @@ Run the old OBS bridge only when you need the compatibility path:
```powershell
.\scripts\sender-discover.ps1
Copy-Item .\config\localcast.example.json .\config\localcast.json
.\scripts\sender-start.ps1 -Config .\config\localcast.json -DryRun
```
@ -329,3 +342,14 @@ layer for communication and heartbeats. Its birth memory, jurisdiction, voice,
and heartbeat contract live in [docs/mimir-face.md](docs/mimir-face.md); its
VoidBot-facing identity and typed Face state live under `.voidbot/voice/` and
`.voidbot/state/`.
## Building against CultLib
Builds are pinned to CultLib 2.0.0 (tag `cultlib-unity-v2.0.0`, revision `42dce3611a81adc5c5fac54cadbaa5637d6022d3`); the build fails if the CultLib checkout differs or is dirty. Create a pinned worktree and point the build at it:
```powershell
git -C <CultLib> worktree add --detach <path> cultlib-unity-v2.0.0
dotnet build -p:CultLibRoot=<path> # or set CULTLIB_ROOT=<path>
```
Without an override the sibling `..\CultLib` checkout is used and must itself be at that tag.

View file

@ -31,15 +31,24 @@
"outputs": {
"video": "synchronized-program-video",
"audio": "separately-controllable-synchronized-stems"
},
"programComposition": {
"owner": "Mimir",
"sceneDocument": "mimir.program_scene.v1",
"operatorSurface": "mimir.eve_operator_surface.v1",
"outputDocument": "mimir.program_output.v1",
"sitePublisher": "Yggdrasil",
"obsRole": "compatibility-sink-only"
}
},
"authorities": {
"configurationCalibrationLaunchStatus": "Mimir",
"programComposition": "Mimir",
"sitePublication": "Yggdrasil publisher daemon",
"denseVisualFusion": "Fensalir",
"gpuMaterialAndBrushReconciliation": "Fensalir",
"spoutPublication": "Fensalir",
"hotAudioDsp": "Faust",
"obsBroadcastControls": "OBS"
"programTextureOutput": "Fensalir",
"hotAudioDsp": "Faust"
},
"visualSensors": [
{ "id": "kiyo-primary", "role": "rgb-main", "required": true },
@ -70,9 +79,9 @@
],
"outputs": {
"video": {
"target": "Spout2",
"senderName": "Mimir Point Cloud",
"owner": "Fensalir"
"target": "MimirProgramTexture",
"senderName": "Mimir Program Texture",
"owner": "Mimir/Fensalir"
},
"audio": {
"owner": "Faust",
@ -85,6 +94,20 @@
"co_streamer_loopback",
"spatial_bed"
]
},
"operator": {
"owner": "Mimir",
"surfaces": [
"cultmesh-command-surface"
]
},
"site": {
"owner": "Yggdrasil publisher daemon",
"input": "mimir.program_output.v1",
"route": "cultmesh://yggdrasil/gamecult.site/mimir/program"
},
"compatibility": {
"obs": "optional-derived-sink"
}
},
"runtimePolicy": {
@ -93,7 +116,8 @@
"native-driver-workers",
"native-reservoir",
"Fensalir",
"Faust"
"Faust",
"Yggdrasil publisher daemon"
],
"neverInCompatibilityEdges": [
"dense-feature-extraction",

View file

@ -2,11 +2,16 @@
## Objective
Make one local Windows computer available as OBS sources on another computer, with sender-side NVENC video encoding and independently mixable audio sources.
Make one local Windows computer available as compatibility media sources on
another computer, with sender-side NVENC video encoding and independently
mixable audio sources. The current program target is Mimir-owned composition;
this bridge is not the stream authority.
## Current Mechanism
The repo owns configuration and scripts. FFmpeg owns capture, encoding, and SRT transport. OBS owns scene composition.
The repo owns configuration and scripts. FFmpeg owns capture, encoding, and SRT
transport. Mimir owns scene composition; OBS may consume this bridge as a
temporary compatibility sink.
Inputs become outputs like this:
@ -21,8 +26,10 @@ Inputs become outputs like this:
## Invariants
- Video encoding happens on the sender.
- OBS receives ordinary media-source URLs; it does not need a plugin for v1.
- Audio remains separately mixable in OBS.
- OBS can receive ordinary media-source URLs for compatibility; it does not need
a plugin for this bridge path.
- Audio remains separately mixable at the compatibility edge, but the durable
control surface belongs to Mimir.
- Ports are stable and derived from one base port plus explicit offsets.
- Secrets, if used, stay in local config and are not committed.

View file

@ -15,8 +15,9 @@ Timestamp for this pass: 2026-05-23, starting at 23:24 Europe/London.
Mimir collects local and network audio/video samples into bounded rolling
runtime buffers, uses active/passive timing evidence to place audio streams on a
canonical timeline, and leaves Fensalir/Faust/OBS with clear ownership over
rendering, DSP, and broadcast composition.
canonical timeline, and lets Mimir own program composition while Fensalir lowers
pixels, Faust/native DSP moves audio, and
Yggdrasil publishes the public feed.
## Current Mechanism
@ -54,11 +55,15 @@ reflection taps.
- UI and telemetry read cached sync state; they do not run analysis.
- Native capture owns hardware calls. Runtime owns sample identity and buffer
placement.
- Fensalir owns windowing, rendering, D3D12, UI, visual fusion, and program
video output.
- Mimir owns the program scene graph, subscription policy, preview/control
state, stats, and publication intent.
- Fensalir owns windowing, rendering, D3D12, UI lowering, visual fusion, and
program video output.
- Faust/native DSP will own hot resampling, fractional delay, separation,
spatialization, and stems.
- OBS owns broadcast composition.
- The Yggdrasil-facing publisher daemon consumes Mimir program output without
owning composition.
- OBS is a compatibility sink only.
## Diagram
@ -83,6 +88,9 @@ flowchart TD
AsioSource --> Sources
Probes["native/probes/*"] --> FrameEvents["MimirFrameEventProcessStreamSource"]
FrameEvents --> Sources
Runtime --> Scene["Mimir program scene graph"]
Scene --> Publisher["Yggdrasil site publisher"]
Scene -. "compatibility" .-> Obs["OBS adapter"]
Reservoir["native/reservoir"] -. "future lower ABI" .-> Hub
```
@ -96,6 +104,9 @@ flowchart TD
- `native/reservoir` is the lower native rolling-buffer ABI.
- `native/probes/*` are measurement and diagnostic tools.
- `scripts/*` are bridge utilities, not the final synchronized program path.
- `docs/mimir-program-composition.md` is the authority map for Muninn capture
hosts, Mimir composition, Yggdrasil publication, and OBS
compatibility.
## Stage 0: Solution And App Host

View file

@ -10,15 +10,15 @@ For the source-level ownership map, read [[code-algorithm-map|Code Algorithm Map
- `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12
bridge.
- `src/Mimir.Runtime` owns stream descriptors, source polling, direct push
ingest, one rolling buffer per configured audio/video stream, and the
ingest, one rolling buffer per configured audio/video/tracking stream, and the
synchronization hub Fensalir can inspect.
- The default five-second window is an intentional latency/memory trade: use it
to line up streams and extract the volumetric audio/video field before OBS
sees program output.
to line up streams and extract the volumetric audio/video field before Mimir
emits program output.
- `native/reservoir` owns the lower native rolling-buffer invariant for
Fensalir/Faust integration.
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN OBS feeds. It is not
the synchronized program authority.
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN compatibility feeds.
It is not the synchronized program or publication authority.
The old script stack is gone. Do not add a compatibility edge unless it protects
a named invariant that the native runtime cannot protect yet.
@ -31,8 +31,166 @@ a named invariant that the native runtime cannot protect yet.
- `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and
`IMimirStreamSource`.
- Configurable five-second default rolling buffers for local and network audio
and video streams.
and video/tracking streams.
- `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers.
- `MimirStreamKind.Tracking` plus `MimirTrackingObservation` carry PS Move
evidence samples as typed `mimir.move_tracking_observation.v1` documents.
Starfire and Nightwing both have USB-attached Moves. Muninn daemons publish
source-local glowing-orb marker candidates and controller/IMU/button state;
Odin owns discovery and schema projection, not pose fusion.
- `native/reservoir` exposes a `move_evidence` typed view and
`LocalcastMoveEvidenceBufferDescriptor` so Muninn Move witnesses can be
admitted as compute-upload buffers before Mimir's fusion pass and Fensalir's
GPU consumers touch them.
- `MimirNativeReservoirRuntime` is the managed owner for that native runtime
boundary. It pins Move evidence sample/descriptor batches so native reservoir
handles remain valid while Mimir prepares fusion input.
- `MimirMuninnMoveEvidenceAdapter` mirrors Muninn's marker-candidate and
controller-state document schemas and normalizes them into native Move
evidence samples. It does not associate markers to controllers or synthesize
pose.
- The realtime Move witness path uses CultMesh streaming, not CultCache polling:
Muninn publishes shared-memory stream frames, Mimir consumes the latest read
lease, normalizes the frame, and admits it to native `move_evidence`.
- `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a
typed admission receipt for that latest read: decoded frame id, producer,
publish/read timestamps, sample counts by kind, source/arrival ranges, native
reservoir handle, and reservoir edge/window. `Mimir.BufferSmoke
--muninn-move-cultmesh-stream-smoke` asserts the receipt so the named proof
spine can be traced instead of inferred.
- `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract.
Mimir owns calibration, association, triangulation, IMU fusion, prediction,
confidence, and latency accounting for those poses before Fensalir consumes
them as interactive controller input.
- `MimirMoveFusion` is the first calibrated fusion owner for Move tracking:
it consumes native Muninn Move evidence samples, requires calibrated camera
witnesses before publishing a pose, associates marker candidates with the
USB controller state, triangulates multi-camera orb position, demotes
single-ray fallbacks through confidence, carries buttons/trigger/gyro into
`mimir.move_controller_pose.v1`, and marks orientation as
`orientation:imu-unresolved` until the IMU/prediction pass earns full 6DoF.
- `MimirMovePoseStream` frames resolved Move poses as
`mimir.move_controller_pose_stream_frame.v1` over CultMesh shared-memory
bytes streams so Fensalir and other consumers have a realtime stream contract
for Mimir-fused controller input.
- `MimirMoveProofSurface` is the observer-only dev proof surface for the named
Move chain. It combines a Muninn evidence admission receipt with a Mimir pose
stream frame and a Fensalir presentation/probe timestamp into
`mimir.move_proof_surface.v1`, then lowers the verdict to a Fensalir
`AquariumSplineFrame` probe. `Mimir.BufferSmoke --move-proof-surface-smoke`
proves the explicit
`muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 ->
mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79` chain,
and rejects single-ray fallback as not full pose.
- `MimirMoveProofPipeline` is the shared commit primitive for the live Move
proof path. It starts from a Muninn CultMesh shared-memory evidence frame,
admits samples to the native `move_evidence` reservoir, runs Mimir-owned
fusion, creates the Mimir pose stream frame, creates the proof surface, and
returns the Fensalir spline probe in one pass. `Mimir.BufferSmoke
--move-proof-pipeline-smoke` proves that shape against the native reservoir
debug DLL; live hardware and presented-frame capture still need to call this
primitive instead of hand-assembling a parallel path.
- `MimirRuntime` is now the live Fensalir attachment point for proof surfaces.
`PublishMoveProofSurface` accepts the Mimir-owned
`mimir.move_proof_surface.v1` document, `CreateFrame` composes its
observer-only `move-proof-*` splines into `AquariumFrame.Scene.SplineFrame`,
and `Mimir.BufferSmoke --move-proof-runtime-frame-smoke` proves the frame is
empty before publish, contains the proof splines after publish, and clears
without leaving a repair loop behind.
- `MimirMoveProofRuntimeDriver` is the bounded runtime bridge from a Muninn
CultMesh shared-memory evidence ring to `MimirRuntime.PublishMoveProofSurface`.
It derives the Mimir evidence, Mimir pose, and Fensalir frame ids from the
actual Muninn frame suffix, calls the shared proof pipeline, and suppresses
duplicate frame admission. `Mimir.BufferSmoke --move-proof-runtime-driver-smoke`
proves `MimirRuntime.Update` can pull one ring frame into the same-sequence
proof chain and visible spline frame.
- `MimirMoveProofRuntimeConfiguration` is the typed runtime configuration and
validation contract for future real Move proof subscriptions. An enabled
source names the Muninn evidence stream, native reservoir path, Mimir evidence
and pose frame prefixes, Fensalir presented-frame prefix, fusion authority,
consumer contract, and at least two calibrated camera witnesses. It creates
`MimirMoveProofRuntimeDriver` only after the supplied ring stream id matches
the configured evidence stream.
- `MimirRuntime` now owns configured Move proof activation instead of leaving
config as a readout. At scene-ready it asks an
`IMimirMoveProofEvidenceRingProvider` for the configured Muninn evidence ring,
opens the configured native reservoir, creates the driver, retains the
resources, and exposes an activation status. The default provider fails
explicitly because the current C# CultMesh ring is still in-process only;
`Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an
in-process provider and proves config activation can produce the same named
proof chain. Real Nightwing/Starfire cross-process ring opening remains
pending.
- `mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine
surface for configured Move proof sources. It reports the configured stream,
provider kind, active/driver state, native reservoir path, calibrated camera
ids, latest same-stream proof ids, latest verdict, and the unavailable
default-provider diagnostic. `Mimir.BufferSmoke
--move-proof-runtime-activation-surface-smoke` proves both an active injected
ring document and the inactive default runtime document without claiming live
field evidence.
- `mimir.move_proof_evidence_frame_snapshot.v1` is the one-copy fallback for
captured Muninn evidence frames while C# CultMesh has no cross-process ring
opener. A configured `EvidenceSnapshotPath` is loaded by
`MimirConfiguredMoveProofEvidenceRingProvider`, metadata-checked against the
encoded payload, copied into an owned in-process ring, and then consumed by
the same runtime driver/pipeline. `Mimir.BufferSmoke
--move-proof-runtime-snapshot-smoke` proves the named proof spine through
that file boundary. It is field capture/replay support, not the final live
shared-memory proof.
- Odin/Muninn now has the matching producer-side latest snapshot writer:
`--move-evidence-snapshot <path>` writes the Mimir-compatible
`mimir.move_proof_evidence_frame_snapshot.v1` artifact from the same Muninn
Move evidence publisher after the frame payload is accepted into the stream
ring. The Odin unit
`move_evidence_snapshot_writes_mimir_compatible_frame_artifact` decodes the
snapshot tuple and embedded Muninn frame payload. This lets Nightwing produce
a field artifact for Mimir replay while the final live ring/page transport is
still pending.
- `MimirMoveProofDevSurface` is a dev-only bootstrap gated by
`MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so
`Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App`
headless, capture the Fensalir-presented PNG, and pixel-check the named
proof probe. This verifies the renderer/probe layer; it is not live
Nightwing hardware evidence.
- `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for
Starfire/Nightwing Moves: required Muninn evidence streams, optional
Quest headset/controller pose witnesses, stillness/sweep/validation phases, and
the four derived calibration outputs Mimir must produce before IMU
orientation can become authority.
- Muninn publishes Quest access as `muninn.quest_access.v1` for the USB-attached
Quest. Mimir consumes that access surface and later `muninn.quest_pose_frame.v1`
samples as optional calibration evidence; Mimir does not own ADB/Quest access.
- The old `scripts/start-nightwing-move-tracking.ps1` bring-up path is
archived. It no longer starts a receiver/recorder pair or stages a socket
witness worker. Nightwing Eye/Move evidence must enter through Muninn/Mimir
typed CultMesh stream frames and Odin-discovered CultMesh documents. The blob
stream remains optical witness evidence for later pose fusion, not the final
6DoF pose owner.
- Structured PS Move light pulses are Muninn output commands. Mimir publishes
`muninn.move_light_command.v1` over CultNet/CultMesh to the Muninn daemon on
the host that owns the USB-attached Move. Muninn writes PS Move HID report
`0x06` locally and updates command state/receipt.
`scripts/start-starfire-move-light.ps1` and Nightwing direct HID writes are
now smoke/bootstrap paths for hardware proof when Muninn is not yet running
on that host.
- Odin's Muninn organ owns Move optical candidate extraction in
`E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds
`muninn.move_marker_candidate.v1` records. Mimir consumes those candidate
streams; it does not own raw optical extraction. Final pose, stereo
triangulation, calibration, association, IMU fusion, and prediction belong to
Mimir.
- Muninn's Move evidence stream frame no longer hardcodes an empty optical
candidate slice. The daemon frame uses the canonical
`muninn.move_marker_candidate.v1` record shape, and the Odin unit
`move_marker_candidates_publish_in_mimir_compatible_cultmesh_frame` proves a
bright Y8 frame can pass through `muninn-move-tracker` and serialize as a
non-empty marker candidate beside controller evidence. The daemon now has a
source-local Y8 extraction/publish seam and a first `serve` camera producer:
`--move-marker-camera <camera-id>=<device-path>` polls a Unix V4L2 YUYV frame,
converts it to compact Y8, and feeds that same seam. Unit tests prove the
configured camera tick publishes marker evidence through the shared frame
contract; Nightwing hardware/live V4L2 proof is still pending.
- `MimirProcessStreamSource` for bridge/network command edges.
- `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata
from native probes into the same rolling buffers Fensalir inspects. One probe
@ -184,7 +342,7 @@ a named invariant that the native runtime cannot protect yet.
profiles, language/emission profiles, path-learning sessions, benchmark
panels, audio actuator strategies, native capture profiles, camera ingest
strategies, reservoir strategies, audio/visual field profiles, compute
offload profiles, OBS publication profiles, CultMesh contracts, Fensalir
offload profiles, program publication profiles, CultMesh contracts, Fensalir
lowering, and assembly plans. Use
`Mimir.BufferSmoke --perfect-machine-profile-smoke` to prove the catalog
assembles, `--perfect-machine-contract-smoke` to write a CultCache contract
@ -192,7 +350,10 @@ a named invariant that the native runtime cannot protect yet.
tooling/UI/remote witness use, and `--perfect-machine-lowering-benchmark` to
measure the Mimir-to-Fensalir lowering path. The current six-camera/two-audio
synthetic lowering benchmark runs at roughly 2.5 us per iteration with about
2.3 KB allocated per iteration.
2.3 KB allocated per iteration. `--move-tracking-contract-smoke` proves
Starfire-local and Nightwing-remote Move tracking observations enter
Mimir's rolling buffers with Muninn producer identity and Odin discovery
provenance.
- `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device
timestamp, and native/GPU handle metadata.
- `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live
@ -202,14 +363,61 @@ a named invariant that the native runtime cannot protect yet.
source-id hashing, producer helpers, and typed audio/render payload
descriptors.
- Windows bridge scripts for sender discovery/start/stop and simple OBS Media
Source ingest.
Source compatibility ingest.
- `src/Mimir.Runtime/Synchronization` now contains the first typed program
composition contracts: `mimir.program_scene.v1` and `mimir.program_output.v1`.
These make Mimir's scene graph and Yggdrasil/site publication route
manifestable through CultMesh.
- `docs/mimir-program-composition.md` is the current authority map for the new
stream-program architecture: Muninn observes on Starfire/Nightwing/Raven-class
hosts, Mimir consumes selected streams and composes the program, Yggdrasil publishes to the site, and OBS is
compatibility-only.
- `src/Mimir.CultMeshMedia` is the first explicit CultMesh media/body bridge:
`relay` hosts the document relay on CultNet RUDP `3075`, `send` reads an
MPEG-TS byte stream from stdin and publishes rolling
`mimir.cultmesh_media_frame` documents, and `recv` subscribes to those
documents and writes ordered MPEG-TS bytes to a Starfire-local UDP endpoint
for compatibility sinks. Its C# implementation now uses explicit CultLib
RUDP client/session helpers behind
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`; the older
`CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` entrypoints no longer own
the media transport lane, and sender/receiver launchers no longer accept raw
relay host/port configuration.
- `scripts/start-raven-cultmesh-av-sender.ps1`,
`scripts/start-yggdrasil-cultmesh-media-relay.ps1`, and
`scripts/start-starfire-cultmesh-av-receiver.ps1` are the CultMesh bridge
operators. `start-raven-cultmesh-av-sender.ps1` is the Mimir-owned
`Mimir.CultMeshMedia` bootstrap/body-bridge lane; it is not the real Muninn
OBS/SRT feed owner. The actual Raven OBS/plugin feed now routes through
Odin's Muninn actuator `E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1`,
with `scripts/start-raven-muninn-obs-feed.ps1` in this repo as a thin local
wrapper around the real `GameCult-Muninn-Activate` task and
`muninn.exe activate` body. `-LocalBootstrap` on the CultMesh sender remains
a direct local bootstrap edge only for the separate body-bridge lane. Raven
capture still defaults to FFmpeg desktop frames plus Mimir's WASAPI loopback
capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains an explicit
fallback.
- Documentation for OBS receiver setup, native rebuild boundaries, the viable
stream app, and the Mimir Face.
## Temporary
- Audio and video may still traverse separate OBS/SRT endpoints during bridge
testing so OBS can preserve independent controls.
testing, but Mimir owns independent controls.
- The CultMesh media bridge still lowers to local UDP for compatibility sinks
because OBS is not a CultMesh consumer. Network transit between Raven,
Yggdrasil, and Starfire is the Odin-discovered CultMesh/CultNet path;
OBS-local UDP is an egress adapter only.
- CultLib RUDP is the default typed CultNet/CultMesh document transport for
daemon truth. Idunn RUDP health publication is the current freshness witness;
provider advertisements, command boundaries, transport profiles, and retained
daemon state should follow the same path. Odin still owns Verse/service
discovery, Idunn owns keepalive decisions, and Mimir-owned dashboard/reference
surfaces only report their own observed state. Product/debug render surfaces
stay lowerings or compatibility evidence.
- `Mimir.CultMeshMedia` has completed its explicit RUDP transport cut. Do not
add another private bridge, status shim, or renderer-derived service truth
while the RUDP document lane exists.
- Process-backed stream sources are only acceptable for network bridge feeds or
diagnostics. Six-camera local ingest belongs behind direct capture drivers.
- Frame-event process sources are diagnostic only. They prove source cadence and
@ -219,34 +427,111 @@ a named invariant that the native runtime cannot protect yet.
## Next
Before expanding Muninn media or Sleipnir input transport, apply the traffic
contracts distilled in
`docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md`: latest-state,
ordered-edge, video-deadline, audio-playout, and reliable-control. Preserve the
existing Muninn packetizer/feedback foundations. The first implementation pass
should make input supersession and edge preservation structural, replace the
two-second media default with consumer-derived field budgets, and prove the
result under controlled loss/jitter/reorder before changing carriers.
The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot
`8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media
deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware
repair, and decode-chain-owned keyframe pressure. The production video path now
uses typed V4 Cauchy GF(256) FEC in independent 8-data/4-parity blocks. Each
block schedules data and parity as separate lanes, including fixed protection
for a short tail block. Canonical video and parity use CultNet's unreliable
`realtime` lane; selective repair and IDR recovery remain deadline-bound.
Canonical AAC ADTS audio and its fixed 864-byte parity shards also use
`realtime`; 4+2 FEC, 120 ms reorder, and
concealment own continuity without an ACK/retransmit window. Sender access units enter the
handoff queue independently, and CultMesh catalog publication runs outside the
realtime media loop. Repair material becomes available when a video frame enters
the queue, and the receiver may issue bounded repair requests at 64/96/128 ms.
The remaining work is duplicate/reconnect acceptance, longer iid runs, and the
long-duration mixed soak.
The socket harness and bounded PCM loss recovery have now landed. Odin's
`cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/
duplicate/jitter/stall profiles around real CultNet endpoints. Muninn emits a
typed fixed 4+2 audio parity block after each four constant-size PCM packets;
the production Mimir OBS receiver reconstructs up to two missing packets inside
the 40 ms reorder budget and feeds them through the existing playout owner.
Both sides use the experimental CultLib/CultMesh snapshot lineage used by Eve,
Aetheria, and VoidBot; no stable-branch transport shim was introduced.
Receiver pressure now closes into the long-lived NVENC encoder as bounded AIMD
bitrate control. Startup begins at half of the configured encoder ceiling
because fixed block parity can roughly double wire rate, and additive recovery
is capped at half of that configured ceiling. Late/decode/repair/queue
pressure backs off by 15 percent; clean recovery adds one-fiftieth of the safe
cap only after ten stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a
restart and emitted the required transition IDR. The exact remaining completion
gate is tracked in
`docs/research/moonlight-reliability-acceptance-2026-07-16.md`; no cross-host
claim is allowed until its Raven-to-Starfire field matrix passes.
The production receiver follow-up fixed two contract contradictions: early
video repair no longer declares the frame late (which had made Odin discard all
repair requests), and the OBS audio decoder now consumes Muninn's actual float
PCM contract rather than treating it as AAC. Audio reorder is bounded to 40 ms
with short silence concealment. The next audio cut is Opus with explicit
FEC/PLC for variable-rate compressed audio; PCM now has bounded 4+2 erasure
recovery plus concealment. The controllable video encoder owner now exists and
forces the next NVENC frame to IDR without restarting the video session. A live
D3D11 desktop proof verified the command-generated IDR. The Raven bundle and
OBS receiver are now deployed over the direct LAN route. Raven activation is an
interactive-token scheduled task whose PowerShell action uses
`-WindowStyle Hidden`; no WireGuard path or foreground terminal is part of the
runtime. The active field cut uses D3D11/NVENC video, WASAPI loopback audio,
typed CultNet media, and the experimental CultLib lineage. The named direct and
proxy impairment profiles and long soak remain the completion gate.
1. Replace the frame-event diagnostic bridge with concrete direct capture
drivers for Leap stereo IR first, then the
other cameras.
2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
2. Cut the remaining dashboard service truth paths from older renderer-derived
assumptions to explicit CultNet RUDP records,
preferably authorized-peer dialing where the peer catalog exists. Preserve
OBS-local UDP as egress only for compatibility sinks.
3. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained
frame cadence in the rolling buffers.
3. Promote the packet-song physical calibration receipt into the runtime
4. Promote the packet-song physical calibration receipt into the runtime
receiver. The live decoder should keep its ear open for self-identifying
song contours, extract intra-call time/frequency anchors from log-mel parts,
apply learned per-output/mic path weighting, and feed a global
delay/clock/path hypothesis. Keep chirp-bin calibration artifacts as
reference data, not the runtime target.
4. Add the synchronization actuator: drive a variable-rate resampler and
5. Add the synchronization actuator: drive a variable-rate resampler and
fractional delay line per non-reference stream from the smoothed
`MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder
through real loopback and microphone paths so every correctly heard word
becomes a deterministic timeline anchor before the actuator moves samples.
5. Prove the bioacoustic hybrid fallback through real loopback and microphones
6. Prove the bioacoustic hybrid fallback through real loopback and microphones
with probe durations long enough to keep loopback and mic windows live.
6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence,
7. Wire real Move proof production into `MimirMoveProofRuntimeDriver`: attach
real Muninn Nightwing and Starfire evidence rings, require calibrated optical
witnesses, publish `mimir:starfire:move-pose:<sequence>` through the shared
runtime path, and replace the dev-gated presented-frame smoke with a
same-sequence capture from real field evidence.
8. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence,
source timestamps, and output settings are visible and adjustable.
7. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute:
9. Implement the Mimir program scene graph as the shared commit primitive for
source subscription, transforms, crop, chroma key, visibility, layer order,
preview, and output publication. Import the current OBS scene only as an
initial mirror; the operator surface returns on Thing 2.
10. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir
program output and publishes it without owning a second composition.
11. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute:
sample buffer-domain paths stochastically by visual contribution, emit SDF
splat probes, write them into the spatiotemporal splat reservoir, and sample
that reservoir in the temporally antialiased scene pass. The direct spline
preview must stay a witness until this path owns rendering.
8. Move GPU feature extraction, fusion, material fitting, render budgeting, and
12. Move GPU feature extraction, fusion, material fitting, render budgeting, and
Spout2 publication into Fensalir.
9. Move mic alignment, room suppression, voice separation, spatialization, and
13. Move mic alignment, room suppression, voice separation, spatialization, and
stem generation into Faust/native DSP.
10. Keep the OBS bridge witness ledger as evidence before expanding receiver
14. Keep the OBS bridge witness ledger as evidence before expanding receiver
machinery.

View file

@ -48,9 +48,10 @@ Mimir exists to turn cameras, microphones, speakers, timing signals, and
distributed capture surfaces into one realtime coherent volumetric field. The
field is the product. The bridge scripts are scaffolding and probes.
The live target is one native five-second rolling reservoir feeding Fensalir GPU
fusion and Faust/native DSP, then OBS. Retention has one owner. Typed views are
indexes, not private histories.
The live target is one native five-second rolling reservoir feeding Mimir-owned
program composition, Fensalir GPU fusion, Faust/native DSP, operator
surfaces, and a Yggdrasil site publisher. OBS is compatibility only. Retention
has one owner. Typed views are indexes, not private histories.
## Operating Doctrine
@ -63,11 +64,14 @@ the synchronized program path is being rebuilt around `LocalcastRuntime`,
Invariants:
- OBS owns final broadcast composition.
- Fensalir owns dense visual fusion, material/brush/splat rendering, and Spout2
publication.
- Mimir owns final program composition, source subscription, preview/control
state, stats, and publication intent.
- Fensalir owns dense visual fusion, material/brush/splat rendering, and local
program texture output.
- Faust/native DSP owns hot audio alignment, room suppression, voice separation,
stems, and spatial bed generation.
- The Yggdrasil publisher daemon ships the Mimir program to the site without
owning composition.
- FFmpeg owns simple capture, encode, and transport in V1 bridge mode.
- Mimir/this repo owns configuration, calibration, launch/status, typed
contracts, memory, and operational repeatability.

View file

@ -0,0 +1,103 @@
# Mimir Program Composition
## Objective
Mimir owns the live stream program: selected capture feeds, calibration feeds,
scene composition, preview/control surfaces, stats, and publication. OBS is a
temporary compatibility sink only.
## Authority Map
- Owner: Mimir owns the program scene graph and subscription policy.
- Inputs: Muninn-published local and remote monitor, window, camera, audio,
loopback, and health streams.
- Outputs: a Mimir program video surface, aligned audio stems or final bus,
and a Yggdrasil-facing site publication feed.
- Derived state: OBS receives a compatibility feed when useful; it does not own
scene truth, broadcast truth, preview truth, stats, source selection, or
publication.
- Forbidden writers: OBS scenes, browser dashboards, site daemons, and Muninn
capture hosts must not independently decide the program composition.
- Shared paths: direct operator edits, remote commands, preset loads,
calibration-driven source selection, and recovery scripts must commit through
the same Mimir scene graph command surface.
- Deletion line: old OBS-owned composition docs and config names are demoted to
diagnostic bridge language. Any OBS adapter delegates to the Mimir program
scene instead of carrying its own opinion.
## Machine Shape
```mermaid
flowchart TD
A["Muninn on Starfire"] --> D["CultMesh stream catalog + media bodies"]
B["Muninn on Nightwing"] --> D
C["Muninn on Raven"] --> D
D --> E["Mimir subscription policy"]
E --> F["Mimir rolling buffers"]
F --> G["Mimir program scene graph"]
G --> H["Fensalir compositor"]
F --> I["Faust/native DSP"]
H --> J["program video surface"]
I --> K["program audio stems/bus"]
J --> M["Yggdrasil site publisher daemon"]
K --> M
M --> N["GameCult site"]
J -. "compatibility" .-> O["OBS adapter"]
K -. "compatibility" .-> O
```
## Composition Contract
The scene graph carries layers with:
- source reference, such as `muninn:starfire:monitor:primary` or
`muninn:starfire:window:discord`;
- source kind, such as monitor, window, camera, network monitor, or generated
Mimir visual field;
- visibility, layer order, transform, crop, and optional chroma key;
- enough timing identity for Mimir to align media inside the rolling window.
The current OBS scene can be mirrored as an initial Mimir scene:
- Starfire monitor capture, cropped at the bottom when needed.
- Discord window capture, cropped to chat and keyed with chroma filters.
- Raven primary monitor plus Realtek loopback as a subscribed remote Muninn
stream.
- Mimir/Fensalir program texture as a generated layer when the field renderer is
active.
After import, OBS is not the editor of record. It may display the Mimir program
feed, but operator edits happen through Mimir command surfaces.
## Site Publisher
The Yggdrasil-facing publisher consumes Mimir program output and publishes it to
the GameCult site. It owns transport, encoding/session state, reconnect policy,
and public edge health. It does not own composition.
The publisher's input contract is the same Mimir program output used by local
preview and OBS compatibility adapters, so the public stream cannot silently
diverge from the operator preview.
## Runtime Contracts
The first typed contracts live in `src/Mimir.Runtime/Synchronization`:
- `mimir.program_scene.v1`
- `mimir.program_output.v1`
- `mimir.eve_operator_surface.v1`
The proof path is:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke
```
Import the current OBS scene as Mimir-owned typed state:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --import-obs-program-scene --output state\mimir-program-composition.cc
```
The live imported state file is `state/mimir-program-composition.cc`.

View file

@ -0,0 +1,107 @@
# Move Calibration Protocol
Mimir owns Move calibration and sensor fusion. Muninn owns source-local facts:
camera marker candidates, USB controller IMU/button state, and local Move light
commands. Odin discovers and projects the surfaces. Fensalir consumes Mimir's
resolved poses after Mimir has earned them.
## Current Stream Shape
- Starfire should run Muninn as a local capture/control daemon. Its Move stream
address is `muninn:starfire:move-evidence`.
- Nightwing runs Muninn for its USB Move and local camera witnesses. Its Move
stream address is `muninn:nightwing:move-evidence`.
- Mimir on Starfire publishes resolved pose frames at
`mimir:starfire:move-controller-poses`.
- Quest can participate as an external tracked VR reference over USB:
`quest:usb:headset-pose`, `quest:usb:left-controller-pose`, and
`quest:usb:right-controller-pose`.
- Starfire ASIO/Scarlett loopback can provide cue timing, but it is not a Move
pose authority.
The Move hot path is CultMesh stream frames. On the same host, those frames use
shared-memory bytes rings. Across hosts, the remote peer still crosses the
Verse/network boundary. The current Raven program media bridge is not same-host
zero-copy into Mimir/Fensalir; it sends CultNet/CultMesh media documents and
lowers to local UDP for OBS compatibility.
## Required Capture Phases
1. `preflight-streams`: hold both Moves still and visible; prove stream freshness,
clock skew, and controller ids.
2. `dark-stillness`: lights off, both Moves still; estimate gyro bias, gravity,
and magnetometer baseline.
3. `lit-stillness`: light one Move at a time, then both; map light commands to
controllers and measure centroid stability.
4. `axis-sweeps`: slow pitch/yaw/roll sweeps; fit gyro scale, accelerometer
frame, and optical angular consistency.
5. `figure-eight`: broad figure-eight motion; fit magnetometer hard/soft iron
and cross-axis coupling.
6. `quest-reference`: optional Quest headset/controller reference with the Move
between the two Quest controllers in front of the headset.
7. `validation-pass`: held-out free motion; promote only if triangulated optical
residuals and IMU prediction residuals pass.
## Quest Access Preflight
The Quest must be authorized for USB debugging before Starfire Muninn can query
it. Muninn owns the access surface:
```powershell
muninn serve --store C:\Meta\Odin\state\muninn.telemetry.cc --host starfire --quest-adb --quest-serial 1WMHHB68PG1515
```
When the Quest is authorized, Muninn publishes `muninn.quest_access.v1` and
advertises `muninn:<host>:quest-input`, `muninn:<host>:quest-poses`, and
`muninn:<host>:quest-warped-video-input`. Mimir consumes the access/pose
surfaces as calibration evidence; it does not own Quest USB access.
```powershell
muninn quest-access-status --store C:\Meta\Odin\state\muninn.telemetry.cc
```
ADB authorization proves USB access, device identity, and basic headset state
only. It does not expose Quest headset or controller poses; those still require
a Quest/OpenXR witness bridge that publishes pose samples through Muninn. The
current calibration protocol can still run without Quest, but Quest poses are
the preferred external reference when the headset and both controllers are
sitting around the Move.
## Data Products
- `mimir.move_fusion_rig_calibration.v1`: camera intrinsics/extrinsics, clock
association policy, and gyro unit scale.
- `mimir.move_imu_calibration.v1`: per-controller gyro bias/scale,
accelerometer gravity frame, magnetometer correction, and noise estimates.
- `mimir.move_controller_identity_map.v1`: stable association between Muninn
USB ids, light receipts, and optical orb identities.
- `mimir.move_calibration_receipt.v1`: sample counts, residuals, rejected
phases, and promotion decision.
## Optical Sphere Fit Authority
`MimirMoveSphereCalibrationFitter` owns the bounded two-Eye optical fit after a
collection session reaches `Fitting`. Eye 0 defines the tracking-space gauge.
The fitter jointly estimates both focal lengths and Eye 1's rigid pose from
same-frame, same-Move correspondences, using a deterministic 80/20 training and
held-out partition. A robust training consensus may reject contaminated marker
pairs, but promotion also requires at least 65 percent of the untouched
held-out pairs to agree and must pass the session's median and P95 reprojection
limits.
The physical orb radius is an explicit input; it is never guessed by the
solver. PSMoveAPI's reported blob radius is a color-segmentation result, not a
calibrated sphere silhouette. It may seed or weight depth only after the
per-camera radius response has been validated. A fresh PSMoveAPI position age
is also not optical confidence and must not be interpreted as one.
## Preflight Command
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke --output artifacts\move-calibration\protocol.cc
```
That writes the typed protocol as `mimir.move_calibration_protocol.v1` into a
CultCache receipt. The real hardware runner should consume this document plus
Muninn's `muninn.quest_access.v1` when Quest is available, then append capture
receipts instead of inventing a parallel checklist.

View file

@ -5,17 +5,24 @@ hosted stream machine.
## Objective
Make the room available in OBS as synchronized program video plus separately
Make the room available as a Mimir-owned synchronized program with separately
controllable audio while preserving ownership:
- Mimir owns configuration, calibration truth, runtime contracts, launch,
status, and persistence.
- Mimir.Runtime owns in-memory stream buffers and runtime synchronization.
- Native capture workers own device reads and append typed sample handles.
- Fensalir owns GPU fusion, rendering, D3D12 interop, UI, and Spout2 output.
- Mimir owns stream subscription, scene composition, preview/control state,
stats, and program publication intent.
- Fensalir owns GPU fusion, rendering, D3D12 interop, UI lowering, and local
program texture output.
- Faust/native DSP owns hot audio alignment, suppression, separation, stems, and
spatial bed generation.
- OBS owns broadcast controls.
- Eve GUI/TUI lowerers expose operator controls and preview without owning
scene truth.
- The Yggdrasil-facing publisher daemon consumes Mimir program output and
publishes it to the site without owning composition.
- OBS is a temporary compatibility sink.
## Runtime State Model
@ -26,10 +33,13 @@ flowchart TD
N["network feed producers"] --> R
R --> Q["Fensalir UI + GPU fusion"]
R --> F["Faust/native DSP"]
Q --> S["Spout2/program video"]
Q --> S["Mimir program video"]
F --> P["program stems + spatial bed"]
S --> O["OBS"]
P --> O
Q --> E["Eve operator surfaces"]
S --> Y["Yggdrasil site publisher"]
P --> Y
S -. "compatibility" .-> O["OBS adapter"]
P -. "compatibility" .-> O
```
The app-level runtime initializes one rolling buffer per configured audio or
@ -48,7 +58,8 @@ camera driver buffers, GPU textures, or DSP audio memory.
- Fensalir consumes current visual/audio timing state and publishes the program
video surface.
- Faust/native DSP consumes audio/phase/event handles and publishes stems.
- OBS receives final program surfaces. It does not own synchronization.
- OBS compatibility adapters may receive final program surfaces. They do not
own synchronization, composition, preview/control, or publication.
## Impossible By Construction

View file

@ -38,19 +38,23 @@ Read this with:
Mimir is a bounded-latency realtime field computer: it ingests many clocks,
places their evidence inside one rolling temporal window, extracts audio/video
geometry from that window, and emits an OBS-usable program surface without
letting OBS become the synchronization authority.
geometry from that window, owns the stream program scene, exposes Eve operator
surfaces, and emits a publicable program feed without letting compatibility
sinks become authority.
## Authority Index
| Authority | Owns | Must Not Own |
| --- | --- | --- |
| `Mimir.Runtime` | stream identity, buffers, sync reports, calibration model loading, source polling, telemetry state | hot DSP actuator, dense visual fusion, broadcast composition |
| Native capture | direct driver callbacks, device timestamps, payload handles, low-jitter capture loops | runtime policy, UI state, OBS composition |
| Mimir program composition | source subscription, scene graph, crop/key/transform state, preview/control state, stats, publication intent | local device reads, GPU shader internals, audio-rate DSP, site transport policy |
| `Mimir.Runtime` | stream identity, buffers, sync reports, calibration model loading, source polling, telemetry state | hot DSP actuator, dense visual fusion, direct device capture |
| Native capture / Muninn hosts | direct driver callbacks, device timestamps, payload handles, local stream capabilities/media | runtime policy, scene composition, publication |
| `native/reservoir` | shared-edge retention and typed views | payload memory interpretation, rendering, DSP |
| Fensalir | window, UI, D3D12 bridge, GPU fusion, program video publication | clock authority, audio actuator |
| Fensalir | window, UI lowering, D3D12 bridge, GPU fusion, program video output | clock authority, audio actuator, scene ownership |
| Faust/native DSP | fractional delay, resampling, separation, spatialization, stems | stream discovery, UI |
| OBS | final broadcast composition | synchronization, capture timing |
| Eve lowerers | operator GUI/TUI rendering, controls, previews, stats lowering | scene truth, stream truth, publication truth |
| Yggdrasil site publisher | public transport, encoder/session state, reconnect policy, site edge health | scene composition |
| OBS | compatibility preview/output sink | synchronization, capture timing, composition, publication |
| Raven/phones | local decode of codebook/schedule observations | canonical clock authority |
## Evidence Taxonomy
@ -241,7 +245,9 @@ Six optical sensors with high-rate Eyes and Leap IR mean:
- Cut JSON/stdout local camera hot path when concrete direct drivers land.
- Cut legacy chirplet path once chirp-bin decoder has equivalent docs/tests and
physical path proof.
- Cut bridge scripts only after OBS-facing direct output is real.
- Cut bridge scripts only after Mimir program output, Eve operation, and
Yggdrasil publication are real enough that OBS compatibility no longer earns
its keep.
- Cut ad hoc response weighting if a formal path model replaces it.
## Future Work Index

View file

@ -13,11 +13,12 @@ job-interview answer for why it belongs in the body of the machine.
| Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` |
| Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` |
| Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` |
| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, and actuator state documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs` |
| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` |
| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, Muninn records optional Quest USB witness availability as `muninn.quest_access.v1`, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` |
| Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` |
| Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` |
| Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` |
| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, and Raven Scarlett. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` |
| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, Raven Scarlett, and USB-attached Moves on Starfire/Nightwing. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` |
| Camera ingest strategies | `Mimir.Runtime` + native workers | Diagnostic metadata, managed native wrapper, native SPSC ring, and shared-GPU-texture ingest shapes. | `src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs` |
| Reservoir strategies | `Mimir.Runtime` + native reservoir + Fensalir | Managed rolling buffers, native shared-edge rings, and derived GPU temporal field storage. | `src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs` |
| Distributed witnesses | `Mimir.Runtime` + CultMesh | Raven, phone, microcontroller, and remote camera rig configurations for compact timing/path state. | `src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs` |
@ -27,7 +28,7 @@ job-interview answer for why it belongs in the body of the machine.
| Visual fusion configurations | `Mimir.Runtime` + Fensalir | Sensor roles and fusion modes from cadence proof through full hybrid evidence. | `src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs` |
| Compute offload configurations | `Mimir.Runtime` + CultMesh | Local-heavy, distributed witness, and calibration sweep workload placement. | `src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs` |
| Assembly plans | `Mimir.Runtime` | Named combinations of modules that prove synthetic, Scarlett, meatspace, and distributed cuts. | `src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs` |
| OBS publication profiles | Fensalir + Faust + OBS | Spout/shared-texture video and separately mixable Faust stem publication. | `src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs` |
| Program publication profiles | Mimir + Fensalir + Faust + Eve + Yggdrasil | Mimir-owned scene publication, Eve operator surfaces, Yggdrasil site feed, and temporary OBS compatibility adapters. | `src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs` |
| Fensalir lowering | `Mimir.Runtime` -> Fensalir | Pure mapping from Mimir rolling/video/audio state into Fensalir GPU sensor and acoustic field frames. | `src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs` |
| Module catalog | `Mimir.Runtime` | Indexes the live module set and verification commands. | `src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs` |
| Manifest factory | `Mimir.Runtime` | Produces an exportable JSON manifest for tools, docs, UI, and remote witnesses. | `src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs` |
@ -42,7 +43,19 @@ job-interview answer for why it belongs in the body of the machine.
stem generation.
- CultMesh carries typed state surfaces; remote nodes may observe and decode,
but they do not become independent clock authorities.
- OBS receives final program surfaces and stems. It does not own calibration.
- USB-attached Moves on Starfire and Nightwing publish source-local Muninn
evidence streams: glowing-orb marker candidates and controller/IMU/button
state. Odin discovers/projects those streams. Mimir owns native evidence
buffer admission/layout, calibration, association, triangulation, confidence,
and `mimir.move_controller_pose.v1` publication. The current fusion pass
resolves calibrated optical position and controller state; orientation remains
explicitly unresolved until IMU fusion/prediction lands. Fensalir consumes the
resolved pose stream for interactive environments once that contract is wired.
- Resolved Move poses are framed as
`mimir.move_controller_pose_stream_frame.v1` on CultMesh shared-memory bytes
streams; the stream is a Mimir output surface, not a new fusion owner.
- OBS may receive compatibility program surfaces and stems. It does not own
calibration, composition, preview/control, or publication.
## Verification Surface
@ -53,6 +66,12 @@ dotnet build .\Mimir.slnx
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-lowering-benchmark
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-fusion-smoke
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest
dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test
```

View file

@ -0,0 +1,333 @@
# Moonlight lessons for Muninn streaming and Sleipnir input
Date: 2026-07-16
Upstream examined: `moonlight-stream/moonlight-common-c` commit `82e25148549d249a80aaf4afc68e2abb07072edf`
## Verdict
Moonlight is strong because it does not ask one transport policy to serve every
realtime signal. It gives video, audio, control, and input different loss,
ordering, recovery, and freshness rules, then keeps each queue bounded by what
the user can still perceive.
Muninn is not missing all of that machinery. Its media packetizer already owns
access-unit boundaries, timestamps and deadlines, video parity, dependency and
keyframe metadata, bounded assembly, receiver damage feedback, repair caching,
and keyframe requests. Sleipnir already rejects regressing sequence/timestamp
pairs, expires reliable HID sends after 25 ms, reports stage latency, and
neutralizes stale output.
The 2026-07-17 field cut closes most of the original mechanism gaps:
- canonical video and video parity use the unreliable `realtime` lane; typed
8+8 GF(256) blocks, selective repair, and decode-chain IDR recovery own loss;
- canonical 10 ms PCM audio uses `realtime`, with 4+2 FEC, a 40 ms reorder
owner, and short concealment; reliable delivery is reserved for control;
- the consumer deadline is 100 ms by default and 250 ms in the Raven field
profile; queues, retransmits, repair material, and assemblies derive from it;
- receiver pressure actuates the live NVENC encoder through multiplicative
backoff and conservative additive recovery; startup begins at half the
requested ceiling because fixed video parity can roughly double wire load;
- Sleipnir latest state and ordered edges have distinct epoch/sequence and
application-ack semantics, including bounded overflow rebase.
The remaining claim boundary is evidence: direct and impaired Raven-to-
Starfire soaks, reconnect repetition, A/V skew measurement, and a live virtual
HID exercise. The machine is no longer missing a named owner for these paths;
it still has to prove those owners under pressure.
Do not transplant Moonlight wholesale. Distill its ownership model into the
existing typed Muninn/Sleipnir pipeline.
## Authority map
### Muninn media
- Owner: Muninn owns source-local capture, encoding child lifecycle, media
packetization, repair material, and sender feedback response.
- Inputs: encoded H.264/H.265 access units, encoded audio packets, source media
timestamps, configured latency target, and receiver feedback.
- Outputs: typed video chunks/parity, audio packets, recovery/keyframe actions,
and transport/quality telemetry.
- Derived state: RUDP acknowledgements, resend counters, CultCache receipts, OBS
local lowering, and operator cards are evidence or delivery substrate. They do
not own media deadlines.
- Forbidden writers: generic RUDP retry policy must not decide whether expired
media is still useful; OBS must not decide stream synchronization; persistence
must not become a hot media queue.
### Sleipnir input
- Owner: Sleipnir owns selection/remapping and local virtual-HID emission.
Muninn owns source-local device sampling and event classification.
- Inputs: a discovered Muninn input stream, typed mapping intent, monotonic
stream sequence, source-local capture time, and receive time.
- Outputs: local virtual-HID state plus latency/freshness receipts.
- Derived state: the latest full controller snapshot is replaceable state.
Button transitions, text/code points, connect/disconnect, and neutralization
are non-replaceable edges until acknowledged or superseded by an explicit
state transition.
- Forbidden writers: generic reliable queue order must not replay stale analog
or motion samples; wall-clock subtraction must not claim one-way latency
without a clock model; the telemetry/store path must not repair a missing hot
stream.
## What Moonlight actually does
### It separates stream semantics
Moonlight receives video and audio over UDP/RTP-like paths while input travels
through its control path (or a legacy TCP socket). The important lesson is not
the literal socket choice. It is that the queues have different contracts.
Video is frame/dependency oriented; audio is short-block continuity oriented;
input is event/state oriented.
Source: [stream initialization and input transport](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/InputStream.c),
[video receive path](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/VideoStream.c), and
[audio receive path](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/AudioStream.c).
### Video recovery is dependency-aware and deadline-shaped
The video queue groups packets by frame and FEC block, tracks missing data,
reconstructs recoverable blocks with Reed-Solomon parity, rejects old/replayed
frames, and advances frame authority explicitly. Depacketization can request an
IDR when damage makes the predictive chain unsafe. The system repairs useful
frames and abandons poisoned dependency chains; it does not wait forever for
generic reliable delivery.
Source: [RTP video queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/RtpVideoQueue.c) and
[video depacketizer](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/VideoDepacketizer.c).
### Audio recovery is a small bounded continuity problem
Moonlight groups audio into fixed FEC blocks, orders packets by RTP sequence,
recovers missing shards, detects out-of-order behavior, and changes how long it
waits based on observed reordering. When recovery cannot complete in time, it
allows a discontinuity and lets the audio decoder's concealment path carry the
scar. It does not turn late audio into permanent latency.
Source: [audio FEC and reorder queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/RtpAudioQueue.c) and
[packet-loss concealment handoff](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/AudioStream.c).
### Input reliability is constrained by semantic coalescing
Moonlight has one send thread and a bounded queue, but it avoids blindly sending
every sampled value. Relative mouse motion is accumulated, absolute pointer
motion keeps the newest position, pen hover/move keeps the newest compatible
event, controller analog state is merged while button changes split batches,
and sensor motion keeps the newest value per sensor. A roughly millisecond-scale
batching opportunity can reduce latency by preventing ENet queue buildup.
This is the most important Sleipnir lesson: reliable delivery is safe only after
replaceable state has been collapsed and edges have been preserved.
Source: [input batching and send queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/InputStream.c).
### It exposes failure to the owning layer
Packet loss, FEC recovery/failure, queue overflow, frame loss, and keyframe need
are explicit signals. Moonlight's public documentation also warns that allowing
frames to queue for smoothness can create high latency; its balanced mode keeps
only about one frame for jitter smoothing. The machine treats queue depth as a
product decision, not an incidental container property.
Source: [Moonlight frame-pacing FAQ](https://github.com/moonlight-stream/moonlight-docs/wiki/Frequently-Asked-Questions#what-do-the-frame-pacing-options-on-the-android-client-mean).
## Current Muninn/Sleipnir comparison
| Concern | Moonlight | Muninn / Sleipnir now | Distilled judgment |
|---|---|---|---|
| Video unit | Encoded frame, FEC blocks, dependency recovery | Encoded access unit, chunks/parity, frame/dependency IDs, deadlines | The conceptual unit is already correct. Keep it. |
| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Typed 8+8 block FEC, deadline-bound repair, completed-frame tombstones, IDR actuation | Same ownership pattern; field loss ratios and parity overhead remain tunable. |
| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Realtime 4+2 PCM FEC, 40 ms reorder, bounded concealment | Mechanism is closed; Opus FEC/PLC is a later efficiency/quality cut. |
| Congestion | Stream-specific queues and feedback behavior | Half-ceiling startup, 15% pressure backoff, bounded additive ramp, fresh media ahead of repair | Closed loop exists; impairment evidence decides its final constants. |
| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Consumer-owned 100 ms default/250 ms Raven proof; 25 ms HID expiry | Correct owner and bounded values; glass-to-glass measurement remains. |
| Input state | Coalesces replaceable motion/sensor/analog values | Epoch-scoped latest state supersedes stale state | Structural rather than polling-accidental. |
| Input edges | Button/type changes terminate batches; text is ordered | Ordered edge sequence, application ACK, bounded replay, epoch rebase | Structural; live virtual-HID impairment proof remains. |
| Staleness | Queue construction prevents much stale work | Receiver drops regressing timestamps and neutralizes after timeout | Keep the guard, but move freshness authority earlier into enqueue/supersession. |
| Timing | RTP/media clocks and local queue timing | source Unix nanoseconds plus receiver instants | Use monotonic source time plus an explicit clock-offset/uncertainty model; do not call raw wall-clock subtraction latency. |
| Security | Authenticated encryption in modern stream generations | CultNet typed transport policy varies by lane | Preserve typed schemas, but specify replay/authentication guarantees for hot media and HID lanes. |
## Recommended cut
### 1. Define traffic classes before changing transport
Add one narrow typed delivery policy shared by Muninn and Sleipnir:
- `state-latest`: analog axes, pointer position, gyro/accelerometer samples;
- `edge-ordered`: button/key transitions, text/code points, device lifecycle;
- `media-video`: dependency-bearing access units with decode deadlines;
- `media-audio`: continuity blocks with playout deadlines;
- `control-reliable`: subscriptions, mappings, capability negotiation, and
keyframe requests.
This is not a generic router. It is a finite statement of the five contracts the
existing machine already has. Each class must name supersession, ordering,
expiry, recovery, and observability.
### 2. Cut the two-second realtime default
Derive deadlines from the actual consumer:
- Sleipnir state-latest: tens of milliseconds, with replacement before send;
- input edges: reliable until acknowledged, but compacted against later
authoritative state where safe;
- interactive video: frame/decode budget, initially test 50–150 ms on LAN;
- audio: playout budget, initially test 40–100 ms on LAN;
- Mimir's five-second analysis reservoir remains Mimir compute budget, not a
network permission to deliver old frames.
The exact values must be earned by field traces. The ownership change is not
optional: transport retry duration becomes derived from the media/input
deadline, never the other way around.
### 3. Make Sleipnir freshness structural
Split the hot input frame into:
- a latest-state snapshot with `state_sequence`, source monotonic capture time,
and supersedes-through sequence;
- an ordered edge list with its own sequence range;
- receiver acknowledgement of the highest applied state and edge sequence.
Muninn may overwrite an unsent latest-state snapshot. It may not overwrite an
unacknowledged edge. Sleipnir applies edges in order, applies only the newest
state, and neutralizes on a named lease timeout. Manual, remapped, replayed, and
live paths must use the same commit primitive.
### 4. Close Muninn's feedback loop
Receiver feedback should drive named sender decisions:
- request/force keyframe after unrecoverable dependency loss;
- choose repair only when estimated arrival precedes frame deadline;
- tune parity within a bounded profile from recent burst-loss evidence;
- reduce encoder bitrate/pacing when queue delay grows;
- expose encode, queue, network, assembly, decode, and presentation stages
separately.
Do not add a second congestion controller beside generic RUDP and hope the two
argue productively. Media policy owns the send budget; RUDP exposes delivery
mechanics and obeys the deadline.
### 5. Prove it on the layer the user feels
Build one loss/jitter/reorder test harness and record timelines for:
- controller press and release during 1%, 5%, and burst packet loss;
- high-rate analog/gyro motion under an artificial sender stall;
- video dependency loss and keyframe recovery;
- audio burst loss and concealment;
- queue growth under bitrate oversubscription;
- disconnect/reconnect and stale-session replay;
- clock offset and drift between Muninn and Sleipnir hosts.
Acceptance should bound p50/p95/p99 capture-to-HID age, lost/duplicated edges,
time-to-neutral, glass-to-glass video age, audio discontinuities, A/V skew, and
recovery time. Final-state-only tests will politely conceal the corpse.
## What not to copy
- Do not copy Moonlight's C implementation or protocol packets into GameCult
code without an explicit GPLv3 licensing decision. Architectural study is not
license laundering. See [Moonlight's license](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/LICENSE.txt).
- Do not replace typed CultMesh discovery, command state, or ownership with
RTSP/SDP or a Moonlight compatibility stack.
- Do not make ENet the new universal answer. Moonlight's strength comes from
semantic queues above the carrier, not from the carrier's brand name.
- Do not merge input, audio, and video into one ordered reliable stream. One
lost video fragment must not head-of-line block a button release or audio
packet.
- Do not treat FEC as magic. Parity is useful only when its overhead and recovery
time fit the deadline.
## Distilled doctrine
Realtime strength is not “reliable UDP.” It is knowing which truth survives
loss:
- the newest state replaces old state;
- an edge remains until applied or explicitly superseded;
- a media packet is valuable only while its frame or playout deadline is alive;
- predictive video damage is healed by recovery or a new reference, not by
delivering every corpse;
- queues are latency budgets made physical;
- feedback that cannot change sender behavior is telemetry, not control.
Muninn and Sleipnir already have enough organs to become strong. The next work
is to give each signal one explicit survival law and make the transport obey it.
## 2026-07-17 LAN field results
The Moonlight-derived ownership changes now have Raven-to-Starfire field
evidence over LAN, with Raven processes launched only through hidden scheduled
tasks:
- Muninn video uses independently recoverable block parity plus bounded repair.
During a seeded 1% datagram-loss run, the OBS receiver advanced from 2 to 12
video assembly expiries while forwarding 14,400 additional payload groups:
99.93% completed without expiry. Audio reported zero failures, skips, or stale
packets in the same interval. The impairment proxy forwarded 2,147,356 of
2,169,105 datagrams with zero queue overflow.
- Sleipnir latest-state delivery remained live through seeded 1% loss. Cutting
the proxy caused the non-neutral virtual Xbox pad to neutralize after 1.56 s,
owned by the named stale-input lease; restoring the hidden proxy reconnected
and resumed state without operator action.
- A continuous 64-packet receive drain was retaining the newest input for about
48-57 ms before HID application. Bounding that drain to 2 ms reduced live
receiver-owned receive-to-apply time to typically 69-204 microseconds. A
witnessed 16 ms spike remains to characterize.
- Starfire and Raven wall clocks differ by roughly 2.38 s. Sleipnir no longer
publishes their raw subtraction as one-way latency. Its surface reports that
source-clock latency is unavailable without a clock model and exposes only
monotonic receiver-owned stage durations.
These results clear the initial 99.9% media-completion and disconnect-safety
gates at 1% independent loss. They do not yet prove glass-to-glass latency,
A/V skew, ordered quick-tap delivery under loss, 5%/burst behavior, or a long
mixed-impairment soak. Those remain acceptance gates, not footnotes.
The field comparison also exposed a generic transport invariant Moonlight's
long-running sessions force us to respect: bounded wire identifiers must wrap
deliberately. CultNet's fragmented-message ID previously saturated at the
maximum `u16`, so fragmented audio became unreassemblable after the boundary.
Wrap-to-one plus a boundary test removed the repeatable minute-eight collapse;
a subsequent 10-minute direct LAN run remained clean across the boundary.
## 2026-07-17 ordered-input closure
The quick-tap gate exposed four authorities that generic RUDP reliability had
blurred together:
- A Windows sleep requested at two milliseconds could still miss a short tap.
Muninn now observes XInput on a dedicated high-resolution worker; transport,
discovery, serialization, and logging cannot delay capture.
- Global cumulative ACKs could not retire a reliable packet arriving more than
32 realtime sequence numbers late, and reliable duplicates were not ACKed
after application deduplication. CultNet now directly acknowledges the late
packet and acknowledges reliable duplicates. This collapsed a runaway field
trace from hundreds of thousands of resend datagrams to a bounded flow.
- Muninn used semantic application ACK as permission to send the next edge.
One delayed ACK therefore serialized the entire controller. It now sends a
bounded window of captured edges; CultNet owns retransmission, Sleipnir's
epoch/sequence cursor owns ordering, and semantic ACK only retires history.
- Sleipnir published CultMesh/Eve and Idunn state synchronously on the actuator
thread. Those writes produced 0.7-1.8 second visible button holds. A bounded
coalescing publisher worker now owns control-plane projection, leaving the
input thread to receive, order, actuate, and acknowledge.
The resulting first-hot-plug field proof generated 100 taps with a 16 ms source
press and 84 ms gap. Under seeded 1% independent loss, every layer observed
exactly 100 presses and 100 releases; Raven-visible durations were 24-57 ms.
Under eight consecutive dropped packets every 200 datagrams, every layer again
observed exactly 100 presses and 100 releases; visible durations were 24-40 ms,
the final virtual pad was neutral, and the impairment proxy reported no queue
overflow. Raven ran through hidden scheduled tasks over `192.168.1.0/24`; no
WireGuard route participated. Both daemons used the Eve/Aetheria/VoidBot
experimental CultLib lineage pinned at `8965f3c0`.
This closes ordered digital edges at one 60 Hz source frame. It does not prove
sub-frame synthetic ViGEm pulses: eight-millisecond state-only pulses can fall
between XInput snapshots, and `XInputGetKeystroke` was rejected after it emitted
stale events from a reused user slot. Analog motion under a sender stall,
mixed media/input endurance, glass-to-glass video age, and A/V skew remain open.

View file

@ -0,0 +1,100 @@
# Moonlight-level reliability acceptance ledger
This ledger distinguishes implemented mechanisms from evidence that the actual
Raven-to-Starfire program timeline survives pressure. Passing unit tests is not
permission to call the field path done.
## Required contracts
| Contract | Current authority | Evidence | Status |
|---|---|---|---|
| Experimental CultMesh/CultNet lineage | Odin vendored CultLib snapshot `8965f3c0` | Snapshot provenance plus CultNet/Muninn/Sleipnir suites | proved locally |
| Low-delay NVENC | Muninn controllable encoder | D3D11 desktop capture, no B frames/lookahead, two-frame VBV, clean Annex-B decode | proved locally |
| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default/250 ms Raven profile, bounded queues/cache/reliable expiry, late-frame repair refusal | proved structurally |
| Recoverable video loss avoids decoder reset | Mimir V4 block FEC/repair, Odin repair cache | Exhaustive 2..80 chunk, burst 1..8 tests in Rust/C++; completed-frame tombstones | proved locally |
| Decoder-chain loss forces immediate IDR | Muninn feedback owner and native encoder | Live 2560x1440 command produced IDR at frame 22 under a 600-frame GOP | proved locally |
| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Half-ceiling start and protected cap; live 12-to-6 Mbps reconfiguration produced clean IDR | proved locally |
| Audio survives small erasure bursts | Odin 4+2 GF(256), Mimir FEC/reorder/concealment | Exhaustive one/two-erasure Rust and C++ tests; production DLL build | proved locally |
| Latest input state supersedes stale state | Muninn epoch/state sequence, Sleipnir cursor | Duplicate/reorder/stale epoch tests | proved locally |
| Button edges are not lost or retained forever | Muninn replay/app ACK/epoch rebase, Sleipnir ordered cursor | Quick-tap, ACK, overflow rebase, sparse-gap tests | proved locally |
| Raven runtime is LAN-only and non-interrupting | Hidden Raven scheduled task, direct LAN SSH/media | One coherent Muninn/WASAPI/NVENC/FFmpeg tree; no WireGuard route; `-WindowStyle Hidden` | proved operationally |
| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | Direct LAN clean run 05:34:17–05:44:37: 79,200 records, 697,450,085 bytes, 18,588 recovered shards, zero decode/assembly/media/audio failures and zero sender capacity eviction; proxy matrix and long soak remain | **direct clean passed** |
## Field acceptance matrix
Run the real capture body, production receiver plugin, decoder, audio output,
and Sleipnir virtual input through the seeded profiles under
`E:\Projects\Odin\tests\realtime-impairment`. Observe the rendered/audio/input
timeline, not only proxy CSV counters.
| Profile | Required result |
|---|---|
| clean, 10 minutes | no decoder reset, audio discontinuity, input edge loss, queue growth, or timestamp drift; video p99 delivery under 50 ms |
| iid 1%, 10 minutes | no input edge loss; audio FEC recovers eligible holes; at least 99.9% video frames presented; media p99 under 80 ms |
| iid 3%, 5 minutes | bounded degradation; at least 99% video frames presented; decoder recovery within 250 ms; no stuck controls |
| burst 4 and burst 8 | parity/repair handles eligible damage; unrecoverable dependency loss produces one requested IDR and recovery within 250 ms |
| reorder and duplicate | no duplicate input action, stale state regression, or unbounded assembly/reorder storage |
| delay/jitter | AIMD reduces bitrate before sustained deadline loss and recovers no faster than one step per two stable seconds |
| stall 250 ms | obsolete media expires; queues remain bounded; input latest state converges within 35 ms after release |
| reconnect | old input epoch cannot act; neutral/current state within 100 ms; decoded video within 500 ms |
| mixed 60-minute soak | stable memory/handle/thread counts and no progressive A/V drift |
The first seeded 1% proxy run is a failed acceptance result, not a pass: 987 of
95,741 datagrams were dropped; in roughly one minute the receiver recorded 38
expired video assemblies and 56 skipped audio packets while remaining bounded
and avoiding decoder reset. CultNet fragmentation currently sits below
application-level FEC, so one lost transport fragment discards an entire FEC
shard. The next reliability cut is to make protected media shards fit one wire
datagram or move fragmentation above the erasure-code boundary.
## Field scars
- A fragmented CultNet RUDP session failed repeatably after roughly eight to
nine minutes because the `u16` fragment-set ID used saturating increment and
became permanently stuck at 65,535. Every later fragmented audio document
then collided in one receiver reassembly key. The allocator now wraps from
65,535 to 1, with a boundary test. The clean field run above crossed that
boundary without a single audio reorder, skip, stale packet, or failure.
- Audio and video producer ingress now use separate bounded channels. Video
capture can no longer block audio ingestion before the audio-priority send
scheduler sees it. Capacity-eviction counters proved zero during the clean
acceptance run.
## 2026-07-17 bounded-media field cut
The live path now keeps every protected shard in one datagram, sends AAC ADTS
in fixed 864-byte 4+2 audio blocks, uses 768-byte video shards with 8+4 block
FEC, and begins each decoder epoch at a complete IDR. The sender admits video
to its repair cache when the frame enters the queue, accepts the production
CultNet `realtime` channel, and can service bounded repair requests at
64/96/128 ms in batches of up to eight chunks.
Seeded Raven-to-Starfire evidence:
- iid 1%: 802 of 78,406 datagrams dropped; zero decode drop, video expiry,
audio failure, or audio skip; 1,800 video frames recovered;
- burst-4: 1,436 of 71,910 datagrams dropped; zero expiry or audio skip across
1,800 forwarded frames;
- reorder: 2,872 reorder events; zero expiry or audio skip across 900 frames;
- iid 3%: 2,564 of 85,731 datagrams dropped; three of 5,400 frames expired,
zero audio skip (99.94 percent video delivery);
- burst-8: 190 chunks repaired; one of 1,800 frames expired, zero audio skip;
- 250 ms stall: 420 datagrams withheld, one frame expired, 68 chunks repaired,
an IDR was requested, and another 900 frames arrived without a stuck decoder
or audio skip.
Delay/jitter produced one expiry in 900 frames. Duplicate, reconnect, longer
iid runs, and the 60-minute mixed soak remain open acceptance rows.
Field scar: production video uses the `realtime` channel, but the repair cache
previously admitted only legacy `media` payloads. Synthetic tests built the
legacy shape and certified a repair path that was empty in production. The
production-channel regression test now owns this boundary.
## Completion rule
Moonlight-level performance is reasonably expected only after all local rows
remain green and the real field matrix passes twice: once on direct LAN and once
on the deployed route actually used during streaming. Failure should change the
owning deadline, FEC, encoder, or input authority—not add a repair loop around
the symptom.

View file

@ -4,12 +4,13 @@
Build the smallest live Mimir application that can ingest multiple local or
networked video and audio feeds, hold a short synchronized window in memory,
and emit coherent OBS-facing program outputs while Fensalir provides the
operator UI.
and emit coherent Mimir program outputs while Eve provides operator surfaces
and Yggdrasil publishes the site feed.
The default reservoir horizon is five seconds. That delay is not decoration. It
is the budget that lets late network packets, independent capture clocks,
audio/video alignment, and debug controls converge before OBS sees the result.
audio/video alignment, and debug controls converge before Mimir emits the
program result.
Mimir is explicitly trading latency and memory for throughput and extraction
quality: the app may spend that buffered window lining streams up and crunching
audio/video data to recover as much of the volumetric audio field and
@ -21,16 +22,16 @@ alignment and field-reconstruction budget.
Today the repo has two partial machines:
- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable OBS
Media Source endpoints.
- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable
compatibility endpoints.
- The native `LocalcastRuntime` reservoir ABI can hold typed live sample handles,
and Fensalir can consume native render packets for GPU fusion.
Those parts are useful, but they are not yet the viable app. The viable app is
the join: ingest workers append live sample handles into one Fensalir-hosted
runtime window, Fensalir renders video from that window, Faust/native audio
builds synchronized stems from the same window, and OBS receives only the
program outputs.
builds synchronized stems from the same window, Mimir commits the scene graph,
Eve lowers the operator surface, and Yggdrasil publishes the program output.
```mermaid
flowchart TD
@ -39,10 +40,15 @@ flowchart TD
C["local mic/loopback workers"] --> R
D["network audio ingest workers"] --> R
R --> U["Fensalir debug/settings/output UI"]
R --> V["Fensalir GPU video composition"]
R --> S["Mimir program scene graph"]
S --> U
S --> V["Fensalir GPU video composition"]
R --> F["Faust/native audio alignment + stems"]
V --> O["OBS video output"]
F --> P["OBS audio stem outputs"]
S --> E["Eve GUI/TUI operator surfaces"]
V --> Y["Yggdrasil site publisher"]
F --> Y
V -. "compatibility" .-> O["OBS adapter"]
F -. "compatibility" .-> O
```
## Invariants
@ -51,12 +57,17 @@ flowchart TD
- The five-second rolling window is a deliberate compute budget, not accidental
latency. It exists so capture, sync, GPU fusion, and audio field extraction
can choose coherence over immediacy.
- Fensalir owns the running app, UI, debug state, settings changes, output
management, GPU fusion/rendering, and Spout/video publication.
- Mimir owns the program scene graph, source subscription policy, preview/control
state, stats, and publication intent.
- Fensalir owns the running app host, UI lowering, debug state, GPU
fusion/rendering, and local program texture output.
- Faust/native DSP owns audio alignment, voice separation, suppression,
volumetric sound-field extraction, stems, and spatial bed generation.
- OBS receives synchronized program surfaces. Raw unsynchronized feeds may be
diagnostic inputs, not broadcast truth.
- Eve lowerers render controls and previews without owning scene truth.
- The Yggdrasil publisher consumes synchronized program surfaces without owning
a second composition.
- OBS compatibility adapters may receive synchronized program surfaces. Raw
unsynchronized feeds may be diagnostic inputs, not broadcast truth.
- Networked feeds are producers into the same reservoir. They do not own clocks,
private buffers, or a parallel synchronization model.
- Missing feeds become explicit absence or silent/blank placeholders inside the
@ -78,9 +89,14 @@ The first cut should be deliberately boring:
loopback ingest.
7. Expose Fensalir UI readouts for reservoir edge, window start, per-kind
counts, feed health, delay estimate, output status, and current settings.
8. Render one synchronized program video output from the reservoir.
9. Emit separately controllable OBS audio stems from the same reservoir window.
10. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture
8. Render one synchronized Mimir program video output from the reservoir.
9. Emit separately controllable program audio stems from the same reservoir
window.
10. Publish Eve GUI/TUI operator surfaces for preview, source selection, scene
edits, stats, and output state.
11. Add the Yggdrasil-facing publisher daemon that consumes Mimir program
output without owning composition.
12. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture
and OBS compatibility, not as the stream authority.
## Repo Shape

View file

@ -0,0 +1,66 @@
cmake_minimum_required(VERSION 3.20)
project(mimir_obs_stem_source LANGUAGES CXX)
find_package(libobs REQUIRED CONFIG)
include(CTest)
add_library(mimir_obs_stem_source MODULE
mimir_obs_muninn_source.cpp
mimir_obs_program_texture_source.cpp
mimir_obs_stem_source.cpp)
target_compile_features(mimir_obs_stem_source PRIVATE cxx_std_20)
target_compile_definitions(mimir_obs_stem_source PRIVATE
WIN32_LEAN_AND_MEAN
NOMINMAX
_WIN32_WINNT=0x0A00)
target_link_libraries(mimir_obs_stem_source PRIVATE OBS::libobs d3d11 d3d12 dxgi ws2_32)
if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/../../tools/muninn_media_decoder_probe.cpp")
add_executable(muninn_media_decoder_probe
../../tools/muninn_media_decoder_probe.cpp)
target_compile_features(muninn_media_decoder_probe PRIVATE cxx_std_20)
target_include_directories(muninn_media_decoder_probe PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
endif()
add_executable(muninn_rudp_url_tests
tests/muninn_rudp_url_tests.cpp)
target_compile_features(muninn_rudp_url_tests PRIVATE cxx_std_20)
target_include_directories(muninn_rudp_url_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_rudp_url_tests COMMAND muninn_rudp_url_tests)
add_executable(muninn_media_wire_tests
tests/muninn_media_wire_tests.cpp)
target_compile_features(muninn_media_wire_tests PRIVATE cxx_std_20)
target_include_directories(muninn_media_wire_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_media_wire_tests COMMAND muninn_media_wire_tests)
add_executable(muninn_audio_fec_tests
tests/muninn_audio_fec_tests.cpp)
target_compile_features(muninn_audio_fec_tests PRIVATE cxx_std_20)
target_include_directories(muninn_audio_fec_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_audio_fec_tests COMMAND muninn_audio_fec_tests)
add_executable(muninn_video_fec_tests
tests/muninn_video_fec_tests.cpp)
target_compile_features(muninn_video_fec_tests PRIVATE cxx_std_20)
target_include_directories(muninn_video_fec_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
add_test(NAME muninn_video_fec_tests COMMAND muninn_video_fec_tests)
add_executable(muninn_udp_socket_tests
tests/muninn_udp_socket_tests.cpp)
target_compile_features(muninn_udp_socket_tests PRIVATE cxx_std_20)
target_compile_definitions(muninn_udp_socket_tests PRIVATE
WIN32_LEAN_AND_MEAN
NOMINMAX
_WIN32_WINNT=0x0A00)
target_include_directories(muninn_udp_socket_tests PRIVATE
${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(muninn_udp_socket_tests PRIVATE ws2_32)
add_test(NAME muninn_udp_socket_tests COMMAND muninn_udp_socket_tests)

View file

@ -0,0 +1,14 @@
MimirAudioStem="Mimir Audio Stem"
MimirAudioStem.MapName="Shared memory map"
MimirAudioStem.StemId="Stem id"
MimirProgramTexture="Mimir Program Texture"
MuninnStream="Muninn Stream"
MuninnVideoSource="Muninn Video"
MuninnAudioSource="Muninn Audio"
MuninnStream.Stream="Muninn stream"
MuninnStream.Video="Video device"
MuninnStream.Audio="Audio device"
MuninnStream.VideoBitrateKbps="Video bitrate (kbps)"
MuninnStream.VideoLatencyMs="Video/transport latency (ms)"
MuninnStream.AudioReorderMs="Audio reorder latency (ms)"
MuninnStream.Refresh="Refresh Muninn streams"

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,628 @@
#include <obs-module.h>
#include <util/platform.h>
#include <chrono>
#include <cstdint>
#include <string>
#include <utility>
#include <vector>
#include <d3d11_1.h>
#include <d3d12.h>
#include <dxgi1_2.h>
#include <wrl/client.h>
#include <windows.h>
using Microsoft::WRL::ComPtr;
namespace {
struct MimirProgramTextureSource {
obs_source_t *source = nullptr;
std::string texture_name = "Global\\MimirFensalirProgramTexture";
std::string fence_name = "Global\\MimirFensalirProgramFence";
std::string consumer_fence_name = "Global\\MimirObsProgramConsumerFence";
uint32_t ring_count = 1;
uint32_t width = 0;
uint32_t height = 0;
uint64_t fence_value = 0;
uint64_t last_producer_fence_value = 0;
uint64_t next_retry_ns = 0;
uint64_t next_failure_log_ns = 0;
HANDLE bridge_shared_handle = nullptr;
HANDLE consumer_fence_shared_handle = nullptr;
HANDLE fence_event = nullptr;
gs_texture_t *obs_texture = nullptr;
ComPtr<ID3D12Device> d3d12_device;
ComPtr<ID3D12CommandQueue> command_queue;
ComPtr<ID3D12CommandAllocator> command_allocator;
ComPtr<ID3D12GraphicsCommandList> command_list;
ComPtr<ID3D12Fence> fence;
ComPtr<ID3D12Fence> producer_fence;
ComPtr<ID3D12Fence> consumer_fence;
std::vector<ComPtr<ID3D12Resource>> source_textures;
ComPtr<ID3D12Resource> bridge_texture_d3d12;
ComPtr<ID3D11Device> d3d11_device;
ComPtr<ID3D11DeviceContext> d3d11_context;
ComPtr<ID3D11Texture2D> bridge_texture_d3d11;
};
static void log_retry_failure(MimirProgramTextureSource *ctx, const char *message, HRESULT hr = S_OK)
{
const uint64_t now = os_gettime_ns();
if (now < ctx->next_failure_log_ns) {
return;
}
ctx->next_failure_log_ns = now + 2000000000ULL;
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: %s: 0x%08lx", message, static_cast<unsigned long>(hr));
} else {
blog(LOG_WARNING, "Mimir OBS program texture: %s", message);
}
}
static std::wstring utf8_to_wide(const std::string &value)
{
if (value.empty()) {
return {};
}
const int count = MultiByteToWideChar(CP_UTF8, 0, value.c_str(), -1, nullptr, 0);
if (count <= 0) {
return {};
}
std::wstring wide(static_cast<size_t>(count), L'\0');
MultiByteToWideChar(CP_UTF8, 0, value.c_str(), -1, wide.data(), count);
if (!wide.empty() && wide.back() == L'\0') {
wide.pop_back();
}
return wide;
}
static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback)
{
const char *value = obs_data_get_string(settings, key);
return value && *value ? std::string(value) : std::string(fallback);
}
static uint32_t clamp_ring_count(long long value)
{
if (value < 1) {
return 1;
}
if (value > 4) {
return 4;
}
return static_cast<uint32_t>(value);
}
static std::string texture_name_for_slot(const MimirProgramTextureSource *ctx, uint32_t slot)
{
if (ctx->ring_count <= 1) {
return ctx->texture_name;
}
return ctx->texture_name + "." + std::to_string(slot);
}
static void release_bridge(MimirProgramTextureSource *ctx)
{
if (ctx->obs_texture) {
gs_texture_destroy(ctx->obs_texture);
ctx->obs_texture = nullptr;
}
ctx->bridge_texture_d3d12.Reset();
ctx->bridge_texture_d3d11.Reset();
ctx->source_textures.clear();
ctx->fence.Reset();
ctx->producer_fence.Reset();
ctx->consumer_fence.Reset();
ctx->command_list.Reset();
ctx->command_allocator.Reset();
ctx->command_queue.Reset();
ctx->d3d12_device.Reset();
ctx->d3d11_context.Reset();
ctx->d3d11_device.Reset();
ctx->width = 0;
ctx->height = 0;
ctx->fence_value = 0;
ctx->last_producer_fence_value = 0;
if (ctx->bridge_shared_handle) {
CloseHandle(ctx->bridge_shared_handle);
ctx->bridge_shared_handle = nullptr;
}
if (ctx->consumer_fence_shared_handle) {
CloseHandle(ctx->consumer_fence_shared_handle);
ctx->consumer_fence_shared_handle = nullptr;
}
if (ctx->fence_event) {
CloseHandle(ctx->fence_event);
ctx->fence_event = nullptr;
}
}
static bool wait_for_copy(MimirProgramTextureSource *ctx, uint64_t consumed_producer_value)
{
if (ctx->consumer_fence && consumed_producer_value > 0) {
HRESULT consumer_hr = ctx->command_queue->Signal(ctx->consumer_fence.Get(), consumed_producer_value);
if (FAILED(consumer_hr)) {
blog(LOG_WARNING,
"Mimir OBS program texture: consumer fence Signal failed: 0x%08lx",
static_cast<unsigned long>(consumer_hr));
}
}
const uint64_t value = ++ctx->fence_value;
HRESULT hr = ctx->command_queue->Signal(ctx->fence.Get(), value);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D12 Signal failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
if (ctx->fence->GetCompletedValue() >= value) {
return true;
}
hr = ctx->fence->SetEventOnCompletion(value, ctx->fence_event);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: SetEventOnCompletion failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
WaitForSingleObject(ctx->fence_event, 100);
return ctx->fence->GetCompletedValue() >= value;
}
static bool create_devices(MimirProgramTextureSource *ctx)
{
HRESULT hr = D3D12CreateDevice(nullptr, D3D_FEATURE_LEVEL_11_0, IID_PPV_ARGS(&ctx->d3d12_device));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D12CreateDevice failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
D3D12_COMMAND_QUEUE_DESC queue_desc = {};
queue_desc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
hr = ctx->d3d12_device->CreateCommandQueue(&queue_desc, IID_PPV_ARGS(&ctx->command_queue));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandQueue failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = ctx->d3d12_device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, IID_PPV_ARGS(&ctx->command_allocator));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandAllocator failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = ctx->d3d12_device->CreateCommandList(
0,
D3D12_COMMAND_LIST_TYPE_DIRECT,
ctx->command_allocator.Get(),
nullptr,
IID_PPV_ARGS(&ctx->command_list));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandList failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ctx->command_list->Close();
hr = ctx->d3d12_device->CreateFence(0, D3D12_FENCE_FLAG_NONE, IID_PPV_ARGS(&ctx->fence));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateFence failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ctx->fence_event = CreateEventW(nullptr, FALSE, FALSE, nullptr);
if (!ctx->fence_event) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateEvent failed: %lu", GetLastError());
return false;
}
UINT flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT;
D3D_FEATURE_LEVEL feature_levels[] = {D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0};
D3D_FEATURE_LEVEL selected = D3D_FEATURE_LEVEL_11_0;
hr = D3D11CreateDevice(
nullptr,
D3D_DRIVER_TYPE_HARDWARE,
nullptr,
flags,
feature_levels,
2,
D3D11_SDK_VERSION,
&ctx->d3d11_device,
&selected,
&ctx->d3d11_context);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D11CreateDevice failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
return true;
}
static void create_consumer_fence(MimirProgramTextureSource *ctx)
{
if (ctx->consumer_fence_name.empty()) {
return;
}
HRESULT hr = ctx->d3d12_device->CreateFence(0, D3D12_FENCE_FLAG_SHARED, IID_PPV_ARGS(&ctx->consumer_fence));
if (FAILED(hr)) {
log_retry_failure(ctx, "CreateFence consumer fence failed", hr);
return;
}
const auto wide_name = utf8_to_wide(ctx->consumer_fence_name);
if (wide_name.empty()) {
log_retry_failure(ctx, "consumer fence name is empty or invalid UTF-8");
ctx->consumer_fence.Reset();
return;
}
hr = ctx->d3d12_device->CreateSharedHandle(
ctx->consumer_fence.Get(),
nullptr,
GENERIC_ALL,
wide_name.c_str(),
&ctx->consumer_fence_shared_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "CreateSharedHandle consumer fence failed", hr);
ctx->consumer_fence.Reset();
return;
}
blog(LOG_INFO, "Mimir OBS program texture consumer fence published %s", ctx->consumer_fence_name.c_str());
}
static bool open_source_texture(MimirProgramTextureSource *ctx)
{
ctx->source_textures.clear();
ctx->source_textures.reserve(ctx->ring_count);
for (uint32_t slot = 0; slot < ctx->ring_count; ++slot) {
const auto slot_name = texture_name_for_slot(ctx, slot);
const auto wide_name = utf8_to_wide(slot_name);
if (wide_name.empty()) {
log_retry_failure(ctx, "shared texture name is empty or invalid UTF-8");
return false;
}
HANDLE source_handle = nullptr;
HRESULT hr = ctx->d3d12_device->OpenSharedHandleByName(wide_name.c_str(), GENERIC_ALL, &source_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandleByName failed", hr);
return false;
}
ComPtr<ID3D12Resource> source_texture;
hr = ctx->d3d12_device->OpenSharedHandle(source_handle, IID_PPV_ARGS(&source_texture));
CloseHandle(source_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandle failed", hr);
return false;
}
const D3D12_RESOURCE_DESC desc = source_texture->GetDesc();
if (desc.Dimension != D3D12_RESOURCE_DIMENSION_TEXTURE2D || desc.Format != DXGI_FORMAT_B8G8R8A8_UNORM) {
log_retry_failure(ctx, "expected BGRA8 D3D12 Texture2D");
return false;
}
const uint32_t slot_width = static_cast<uint32_t>(desc.Width);
const uint32_t slot_height = desc.Height;
if (slot_width == 0 || slot_height == 0) {
log_retry_failure(ctx, "shared texture dimensions are empty");
return false;
}
if (slot == 0) {
ctx->width = slot_width;
ctx->height = slot_height;
} else if (slot_width != ctx->width || slot_height != ctx->height) {
log_retry_failure(ctx, "program output ring texture dimensions do not match");
return false;
}
ctx->source_textures.push_back(std::move(source_texture));
}
return ctx->source_textures.size() == ctx->ring_count;
}
static bool open_producer_fence(MimirProgramTextureSource *ctx)
{
const auto wide_name = utf8_to_wide(ctx->fence_name);
if (wide_name.empty()) {
return true;
}
HANDLE fence_handle = nullptr;
HRESULT hr = ctx->d3d12_device->OpenSharedHandleByName(wide_name.c_str(), GENERIC_ALL, &fence_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandleByName producer fence failed", hr);
return true;
}
hr = ctx->d3d12_device->OpenSharedHandle(fence_handle, IID_PPV_ARGS(&ctx->producer_fence));
CloseHandle(fence_handle);
if (FAILED(hr)) {
log_retry_failure(ctx, "OpenSharedHandle producer fence failed", hr);
ctx->producer_fence.Reset();
return true;
}
ctx->last_producer_fence_value = ctx->producer_fence->GetCompletedValue();
if (ctx->last_producer_fence_value == UINT64_MAX) {
log_retry_failure(ctx, "producer fence opened in device-removed state");
ctx->producer_fence.Reset();
return false;
}
blog(LOG_INFO,
"Mimir OBS program texture producer fence opened %s value=%llu",
ctx->fence_name.c_str(),
static_cast<unsigned long long>(ctx->last_producer_fence_value));
return true;
}
static bool read_producer_fence(MimirProgramTextureSource *ctx, uint64_t *completed)
{
*completed = 0;
if (!ctx->producer_fence) {
return true;
}
*completed = ctx->producer_fence->GetCompletedValue();
if (*completed == UINT64_MAX) {
log_retry_failure(ctx, "producer fence reported device removal");
return false;
}
if (*completed > ctx->last_producer_fence_value) {
ctx->last_producer_fence_value = *completed;
}
return true;
}
static bool create_obs_bridge_texture(MimirProgramTextureSource *ctx)
{
D3D11_TEXTURE2D_DESC desc = {};
desc.Width = ctx->width;
desc.Height = ctx->height;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED;
HRESULT hr = ctx->d3d11_device->CreateTexture2D(&desc, nullptr, &ctx->bridge_texture_d3d11);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: CreateTexture2D bridge failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ComPtr<IDXGIResource> legacy_resource;
hr = ctx->bridge_texture_d3d11.As(&legacy_resource);
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: bridge texture is not IDXGIResource: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = legacy_resource->GetSharedHandle(&ctx->bridge_shared_handle);
if (FAILED(hr) || !ctx->bridge_shared_handle) {
blog(LOG_WARNING, "Mimir OBS program texture: GetSharedHandle bridge failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
hr = ctx->d3d12_device->OpenSharedHandle(ctx->bridge_shared_handle, IID_PPV_ARGS(&ctx->bridge_texture_d3d12));
if (FAILED(hr)) {
blog(LOG_WARNING, "Mimir OBS program texture: D3D12 open legacy bridge handle failed: 0x%08lx", static_cast<unsigned long>(hr));
return false;
}
ctx->obs_texture = gs_texture_open_shared(static_cast<uint32_t>(reinterpret_cast<uintptr_t>(ctx->bridge_shared_handle)));
if (!ctx->obs_texture) {
blog(LOG_WARNING, "Mimir OBS program texture: OBS failed to open legacy bridge shared texture");
return false;
}
return true;
}
static bool ensure_bridge(MimirProgramTextureSource *ctx)
{
if (ctx->obs_texture) {
return true;
}
const uint64_t now = os_gettime_ns();
if (now < ctx->next_retry_ns) {
return false;
}
ctx->next_retry_ns = now + 1000000000ULL;
release_bridge(ctx);
if (!create_devices(ctx)) {
release_bridge(ctx);
return false;
}
create_consumer_fence(ctx);
if (!open_source_texture(ctx) || !create_obs_bridge_texture(ctx)) {
release_bridge(ctx);
return false;
}
if (!open_producer_fence(ctx)) {
release_bridge(ctx);
return false;
}
blog(LOG_INFO, "Mimir OBS program texture source opened %s (%ux%u ring=%u)",
ctx->texture_name.c_str(),
ctx->width,
ctx->height,
ctx->ring_count);
return true;
}
static bool copy_source_to_bridge(MimirProgramTextureSource *ctx)
{
HRESULT hr = ctx->command_allocator->Reset();
if (FAILED(hr)) {
return false;
}
uint64_t completed_producer_value = 0;
if (!read_producer_fence(ctx, &completed_producer_value)) {
release_bridge(ctx);
return false;
}
const uint32_t source_slot = ctx->source_textures.empty()
? 0
: static_cast<uint32_t>(completed_producer_value % ctx->source_textures.size());
ID3D12Resource *source_texture = ctx->source_textures[source_slot].Get();
hr = ctx->command_list->Reset(ctx->command_allocator.Get(), nullptr);
if (FAILED(hr)) {
return false;
}
D3D12_RESOURCE_BARRIER barriers[2] = {};
barriers[0].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barriers[0].Transition.pResource = source_texture;
barriers[0].Transition.StateBefore = D3D12_RESOURCE_STATE_COMMON;
barriers[0].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_SOURCE;
barriers[0].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
barriers[1].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barriers[1].Transition.pResource = ctx->bridge_texture_d3d12.Get();
barriers[1].Transition.StateBefore = D3D12_RESOURCE_STATE_COMMON;
barriers[1].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_DEST;
barriers[1].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
ctx->command_list->ResourceBarrier(2, barriers);
ctx->command_list->CopyResource(ctx->bridge_texture_d3d12.Get(), source_texture);
std::swap(barriers[0].Transition.StateBefore, barriers[0].Transition.StateAfter);
std::swap(barriers[1].Transition.StateBefore, barriers[1].Transition.StateAfter);
ctx->command_list->ResourceBarrier(2, barriers);
hr = ctx->command_list->Close();
if (FAILED(hr)) {
return false;
}
ID3D12CommandList *lists[] = {ctx->command_list.Get()};
ctx->command_queue->ExecuteCommandLists(1, lists);
return wait_for_copy(ctx, completed_producer_value);
}
static const char *mimir_program_texture_get_name(void *)
{
return obs_module_text("MimirProgramTexture");
}
static void mimir_program_texture_update(void *data, obs_data_t *settings)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
ctx->texture_name = obs_string(settings, "texture_name", "Global\\MimirFensalirProgramTexture");
ctx->fence_name = obs_string(settings, "fence_name", "Global\\MimirFensalirProgramFence");
ctx->consumer_fence_name = obs_string(settings, "consumer_fence_name", "Global\\MimirObsProgramConsumerFence");
ctx->ring_count = clamp_ring_count(obs_data_get_int(settings, "ring_count"));
release_bridge(ctx);
}
static void *mimir_program_texture_create(obs_data_t *settings, obs_source_t *source)
{
auto *ctx = new MimirProgramTextureSource();
ctx->source = source;
mimir_program_texture_update(ctx, settings);
return ctx;
}
static void mimir_program_texture_destroy(void *data)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
release_bridge(ctx);
delete ctx;
}
static uint32_t mimir_program_texture_width(void *data)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
ensure_bridge(ctx);
return ctx->width;
}
static uint32_t mimir_program_texture_height(void *data)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
ensure_bridge(ctx);
return ctx->height;
}
static void mimir_program_texture_defaults(obs_data_t *settings)
{
obs_data_set_default_string(settings, "texture_name", "Global\\MimirFensalirProgramTexture");
obs_data_set_default_string(settings, "fence_name", "Global\\MimirFensalirProgramFence");
obs_data_set_default_string(settings, "consumer_fence_name", "Global\\MimirObsProgramConsumerFence");
obs_data_set_default_int(settings, "ring_count", 1);
}
static obs_properties_t *mimir_program_texture_properties(void *)
{
obs_properties_t *props = obs_properties_create();
obs_properties_add_text(props, "texture_name", "Shared D3D12 texture name", OBS_TEXT_DEFAULT);
obs_properties_add_text(props, "fence_name", "Shared D3D12 fence name", OBS_TEXT_DEFAULT);
obs_properties_add_text(props, "consumer_fence_name", "Consumer D3D12 fence name", OBS_TEXT_DEFAULT);
obs_properties_add_int(props, "ring_count", "Texture ring slots", 1, 4, 1);
return props;
}
static void mimir_program_texture_render(void *data, gs_effect_t *)
{
auto *ctx = static_cast<MimirProgramTextureSource *>(data);
if (!ensure_bridge(ctx)) {
return;
}
if (copy_source_to_bridge(ctx)) {
obs_source_draw(ctx->obs_texture, 0, 0, ctx->width, ctx->height, false);
}
}
static obs_source_info mimir_program_texture_source_info = {
.id = "mimir_program_texture_source",
.type = OBS_SOURCE_TYPE_INPUT,
.output_flags = OBS_SOURCE_VIDEO,
.get_name = mimir_program_texture_get_name,
.create = mimir_program_texture_create,
.destroy = mimir_program_texture_destroy,
.get_width = mimir_program_texture_width,
.get_height = mimir_program_texture_height,
.get_defaults = mimir_program_texture_defaults,
.get_properties = mimir_program_texture_properties,
.update = mimir_program_texture_update,
.video_render = mimir_program_texture_render,
};
} // namespace
void mimir_register_program_texture_source()
{
obs_register_source(&mimir_program_texture_source_info);
}

View file

@ -0,0 +1,43 @@
#pragma once
#include <cstdint>
constexpr uint64_t MIMIR_OBS_STEM_MAGIC = 0x4D545352494D494Dull;
constexpr int MIMIR_OBS_STEM_VERSION = 1;
constexpr int MIMIR_OBS_STEM_MAX_STEMS = 8;
constexpr int MIMIR_OBS_STEM_MAX_FRAMES = 4096;
constexpr int MIMIR_OBS_STEM_ID_BYTES = 64;
constexpr int MIMIR_OBS_STEM_DISPLAY_BYTES = 64;
constexpr int MIMIR_OBS_STEM_SOURCE_BYTES = 64;
constexpr int MIMIR_OBS_STEM_HEADER_BYTES = 64;
constexpr int MIMIR_OBS_STEM_RECORD_BYTES = 224;
constexpr int MIMIR_OBS_STEM_MAP_BYTES =
MIMIR_OBS_STEM_HEADER_BYTES +
(MIMIR_OBS_STEM_MAX_STEMS * MIMIR_OBS_STEM_RECORD_BYTES) +
(MIMIR_OBS_STEM_MAX_STEMS * MIMIR_OBS_STEM_MAX_FRAMES * static_cast<int>(sizeof(float)));
struct MimirObsStemHeader {
uint64_t magic;
int32_t version;
uint32_t generation;
int32_t stem_count;
int32_t max_stems;
int32_t max_frames;
int32_t header_bytes;
int32_t record_bytes;
int32_t map_bytes;
};
struct MimirObsStemRecord {
int32_t channel_index;
int32_t configured;
int32_t sample_rate;
int32_t frame_count;
int32_t sample_offset;
int32_t reserved0;
int32_t reserved1;
int32_t reserved2;
char stem_id[MIMIR_OBS_STEM_ID_BYTES];
char display_name[MIMIR_OBS_STEM_DISPLAY_BYTES];
char source_id[MIMIR_OBS_STEM_SOURCE_BYTES];
};

View file

@ -0,0 +1,220 @@
#include <obs-module.h>
#include <util/platform.h>
#include <atomic>
#include <chrono>
#include <cstring>
#include <string>
#include <thread>
#include <vector>
#include <windows.h>
#include "mimir_obs_stem_ipc.h"
OBS_DECLARE_MODULE()
OBS_MODULE_USE_DEFAULT_LOCALE("mimir-obs-stem-source", "en-US")
void mimir_register_program_texture_source();
void mimir_register_muninn_source();
struct MimirStemSource {
obs_source_t *source = nullptr;
std::string map_name = "Local\\MimirObsStemBus";
std::string stem_id = "aligned_source_0";
HANDLE mapping = nullptr;
const uint8_t *view = nullptr;
std::thread worker;
std::atomic_bool running = false;
uint32_t last_generation = 0;
};
static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback)
{
const char *value = obs_data_get_string(settings, key);
return value && *value ? std::string(value) : std::string(fallback);
}
static void close_mapping(MimirStemSource *ctx)
{
if (ctx->view) {
UnmapViewOfFile(ctx->view);
ctx->view = nullptr;
}
if (ctx->mapping) {
CloseHandle(ctx->mapping);
ctx->mapping = nullptr;
}
}
static bool open_mapping(MimirStemSource *ctx)
{
if (ctx->mapping) {
return true;
}
ctx->mapping = OpenFileMappingA(FILE_MAP_READ, FALSE, ctx->map_name.c_str());
if (!ctx->mapping) {
return false;
}
ctx->view = static_cast<const uint8_t *>(MapViewOfFile(ctx->mapping, FILE_MAP_READ, 0, 0, MIMIR_OBS_STEM_MAP_BYTES));
if (!ctx->view) {
close_mapping(ctx);
return false;
}
return true;
}
static const MimirObsStemRecord *find_record(const uint8_t *view, const std::string &stem_id, int stem_count)
{
const auto count = std::min(stem_count, MIMIR_OBS_STEM_MAX_STEMS);
for (int index = 0; index < count; ++index) {
const auto *record = reinterpret_cast<const MimirObsStemRecord *>(view + MIMIR_OBS_STEM_HEADER_BYTES + (index * MIMIR_OBS_STEM_RECORD_BYTES));
if (record->frame_count <= 0 || record->sample_rate <= 0 || record->sample_offset <= 0) {
continue;
}
if (stem_id == record->stem_id) {
return record;
}
}
return nullptr;
}
static void publish_if_available(MimirStemSource *ctx)
{
if (!open_mapping(ctx)) {
return;
}
const auto *header = reinterpret_cast<const MimirObsStemHeader *>(ctx->view);
const auto generation_a = header->generation;
if ((generation_a & 1u) != 0 || generation_a == ctx->last_generation) {
return;
}
if (header->magic != MIMIR_OBS_STEM_MAGIC ||
header->version != MIMIR_OBS_STEM_VERSION ||
header->map_bytes != MIMIR_OBS_STEM_MAP_BYTES) {
return;
}
const auto *record = find_record(ctx->view, ctx->stem_id, header->stem_count);
if (!record) {
return;
}
const auto frame_count = std::min(record->frame_count, MIMIR_OBS_STEM_MAX_FRAMES);
const auto sample_rate = record->sample_rate;
const auto byte_count = static_cast<size_t>(frame_count) * sizeof(float);
if (record->sample_offset < 0 || record->sample_offset + static_cast<int>(byte_count) > MIMIR_OBS_STEM_MAP_BYTES) {
return;
}
std::vector<float> samples(frame_count);
std::memcpy(samples.data(), ctx->view + record->sample_offset, byte_count);
const auto generation_b = header->generation;
if (generation_a != generation_b || (generation_b & 1u) != 0) {
return;
}
ctx->last_generation = generation_b;
uint8_t *planes[MAX_AV_PLANES] = {};
planes[0] = reinterpret_cast<uint8_t *>(samples.data());
obs_source_audio audio = {};
audio.data[0] = planes[0];
audio.frames = static_cast<uint32_t>(frame_count);
audio.speakers = SPEAKERS_MONO;
audio.samples_per_sec = static_cast<uint32_t>(sample_rate);
audio.format = AUDIO_FORMAT_FLOAT_PLANAR;
audio.timestamp = os_gettime_ns();
obs_source_output_audio(ctx->source, &audio);
}
static void worker_loop(MimirStemSource *ctx)
{
while (ctx->running.load()) {
publish_if_available(ctx);
std::this_thread::sleep_for(std::chrono::milliseconds(5));
}
}
static const char *mimir_stem_get_name(void *)
{
return obs_module_text("MimirAudioStem");
}
static void mimir_stem_update(void *data, obs_data_t *settings)
{
auto *ctx = static_cast<MimirStemSource *>(data);
ctx->map_name = obs_string(settings, "map_name", "Local\\MimirObsStemBus");
ctx->stem_id = obs_string(settings, "stem_id", "aligned_source_0");
ctx->last_generation = 0;
close_mapping(ctx);
}
static void *mimir_stem_create(obs_data_t *settings, obs_source_t *source)
{
auto *ctx = new MimirStemSource();
ctx->source = source;
mimir_stem_update(ctx, settings);
ctx->running = true;
ctx->worker = std::thread(worker_loop, ctx);
return ctx;
}
static void mimir_stem_destroy(void *data)
{
auto *ctx = static_cast<MimirStemSource *>(data);
ctx->running = false;
if (ctx->worker.joinable()) {
ctx->worker.join();
}
close_mapping(ctx);
delete ctx;
}
static void mimir_stem_defaults(obs_data_t *settings)
{
obs_data_set_default_string(settings, "map_name", "Local\\MimirObsStemBus");
obs_data_set_default_string(settings, "stem_id", "aligned_source_0");
}
static obs_properties_t *mimir_stem_properties(void *)
{
obs_properties_t *props = obs_properties_create();
obs_properties_add_text(props, "map_name", obs_module_text("MimirAudioStem.MapName"), OBS_TEXT_DEFAULT);
obs_properties_add_text(props, "stem_id", obs_module_text("MimirAudioStem.StemId"), OBS_TEXT_DEFAULT);
return props;
}
static obs_source_info mimir_stem_source_info = {
.id = "mimir_audio_stem_source",
.type = OBS_SOURCE_TYPE_INPUT,
.output_flags = OBS_SOURCE_AUDIO,
.get_name = mimir_stem_get_name,
.create = mimir_stem_create,
.destroy = mimir_stem_destroy,
.get_defaults = mimir_stem_defaults,
.get_properties = mimir_stem_properties,
.update = mimir_stem_update,
};
bool obs_module_load(void)
{
obs_register_source(&mimir_stem_source_info);
mimir_register_program_texture_source();
mimir_register_muninn_source();
blog(
LOG_INFO,
"Mimir OBS sources loaded: mimir_audio_stem_source, mimir_program_texture_source, "
"muninn_stream_source, muninn_video_source, muninn_audio_source; "
"Muninn Odin discovery build 20260705.2");
return true;
}

View file

@ -0,0 +1,179 @@
#pragma once
#include <array>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <iterator>
#include <map>
#include <optional>
#include <string>
#include <tuple>
#include <utility>
#include <vector>
namespace muninn_audio_fec {
inline constexpr size_t DataShards = 4;
inline constexpr size_t ParityShards = 2;
inline constexpr size_t TotalShards = DataShards + ParityShards;
using Bytes = std::vector<uint8_t>;
using Shards = std::array<std::optional<Bytes>, TotalShards>;
inline uint8_t multiply(uint8_t left, uint8_t right)
{
uint16_t product = 0;
uint16_t a = left;
uint16_t b = right;
while (b != 0) {
if ((b & 1u) != 0) product ^= a;
b >>= 1u;
a <<= 1u;
if ((a & 0x100u) != 0) a ^= 0x11du;
}
return static_cast<uint8_t>(product);
}
inline uint8_t power(uint8_t value, uint16_t exponent)
{
uint8_t result = 1;
while (exponent != 0) {
if ((exponent & 1u) != 0) result = multiply(result, value);
value = multiply(value, value);
exponent >>= 1u;
}
return result;
}
inline uint8_t inverse(uint8_t value)
{
return value == 0 ? 0 : power(value, 254);
}
inline constexpr std::array<std::array<uint8_t, DataShards>, TotalShards> Generator{{
{{1, 0, 0, 0}}, {{0, 1, 0, 0}}, {{0, 0, 1, 0}}, {{0, 0, 0, 1}},
{{1, 1, 1, 1}}, {{1, 2, 4, 8}}
}};
inline std::array<Bytes, ParityShards> encode(const std::array<Bytes, DataShards> &data)
{
const size_t bytes = data[0].size();
for (const auto &shard : data) if (shard.size() != bytes) return {};
std::array<Bytes, ParityShards> parity{Bytes(bytes), Bytes(bytes)};
for (size_t p = 0; p < ParityShards; ++p)
for (size_t d = 0; d < DataShards; ++d)
for (size_t i = 0; i < bytes; ++i)
parity[p][i] ^= multiply(Generator[DataShards + p][d], data[d][i]);
return parity;
}
inline bool recover(Shards &shards)
{
size_t shard_bytes = 0;
std::array<size_t, DataShards> available{};
size_t count = 0;
for (size_t i = 0; i < TotalShards; ++i) {
if (!shards[i]) continue;
if (shard_bytes == 0) shard_bytes = shards[i]->size();
if (shards[i]->size() != shard_bytes) return false;
if (count < DataShards) available[count++] = i;
}
if (count < DataShards || shard_bytes == 0) return false;
std::array<std::array<uint8_t, DataShards * 2>, DataShards> matrix{};
for (size_t row = 0; row < DataShards; ++row) {
for (size_t col = 0; col < DataShards; ++col) matrix[row][col] = Generator[available[row]][col];
matrix[row][DataShards + row] = 1;
}
for (size_t col = 0; col < DataShards; ++col) {
size_t pivot = col;
while (pivot < DataShards && matrix[pivot][col] == 0) ++pivot;
if (pivot == DataShards) return false;
if (pivot != col) std::swap(matrix[pivot], matrix[col]);
const uint8_t scale = inverse(matrix[col][col]);
for (size_t j = 0; j < DataShards * 2; ++j) matrix[col][j] = multiply(matrix[col][j], scale);
for (size_t row = 0; row < DataShards; ++row) {
if (row == col) continue;
const uint8_t factor = matrix[row][col];
for (size_t j = 0; j < DataShards * 2; ++j)
matrix[row][j] ^= multiply(factor, matrix[col][j]);
}
}
for (size_t missing = 0; missing < DataShards; ++missing) {
if (shards[missing]) continue;
Bytes decoded(shard_bytes);
for (size_t row = 0; row < DataShards; ++row) {
const uint8_t coefficient = matrix[missing][DataShards + row];
for (size_t i = 0; i < shard_bytes; ++i)
decoded[i] ^= multiply(coefficient, (*shards[available[row]])[i]);
}
shards[missing] = std::move(decoded);
}
return true;
}
struct BlockKey {
std::string stream_id;
std::string session_id;
uint64_t base_packet_id = 0;
bool operator<(const BlockKey &other) const
{
return std::tie(stream_id, session_id, base_packet_id) <
std::tie(other.stream_id, other.session_id, other.base_packet_id);
}
};
struct RecoveredPacket { uint64_t packet_id; Bytes payload; };
class BlockReceiver {
public:
using Clock = std::chrono::steady_clock;
explicit BlockReceiver(std::chrono::milliseconds deadline, size_t max_blocks = 128)
: deadline_(deadline), max_blocks_(max_blocks) {}
std::vector<RecoveredPacket> insert(const BlockKey &key, size_t shard_index, Bytes payload,
Clock::time_point now)
{
expire(now);
if (shard_index >= TotalShards || payload.empty() || completed_.count(key) != 0) return {};
if (blocks_.size() >= max_blocks_ && blocks_.count(key) == 0) blocks_.erase(blocks_.begin());
auto [it, created] = blocks_.try_emplace(key, Block{now, payload.size(), {}});
Block &block = it->second;
if (payload.size() != block.shard_bytes || block.shards[shard_index]) return {};
block.shards[shard_index] = std::move(payload);
size_t present = 0;
for (const auto &shard : block.shards) if (shard) ++present;
if (present < DataShards) return {};
Shards before = block.shards;
if (!recover(block.shards)) return {};
std::vector<RecoveredPacket> result;
for (size_t i = 0; i < DataShards; ++i)
if (!before[i] && block.shards[i]) result.push_back({key.base_packet_id + i, *block.shards[i]});
blocks_.erase(it);
completed_[key] = now;
return result;
}
void expire(Clock::time_point now)
{
for (auto it = blocks_.begin(); it != blocks_.end();)
it = now - it->second.created_at >= deadline_ ? blocks_.erase(it) : std::next(it);
for (auto it = completed_.begin(); it != completed_.end();)
it = now - it->second >= deadline_ ? completed_.erase(it) : std::next(it);
}
size_t pending_blocks() const { return blocks_.size(); }
void clear()
{
blocks_.clear();
completed_.clear();
}
private:
struct Block { Clock::time_point created_at; size_t shard_bytes; Shards shards; };
std::chrono::milliseconds deadline_;
size_t max_blocks_;
std::map<BlockKey, Block> blocks_;
std::map<BlockKey, Clock::time_point> completed_;
};
} // namespace muninn_audio_fec

View file

@ -0,0 +1,274 @@
#pragma once
#include <chrono>
#include <climits>
#include <cstdint>
#include <string>
#include <vector>
namespace muninn_media_wire {
inline constexpr const char *ReceiverFeedbackSchema = "muninn.media_receiver_feedback.v1";
class MsgpackWriter {
public:
const std::vector<uint8_t> &bytes() const { return bytes_; }
void write_raw(const std::vector<uint8_t> &value)
{
bytes_.insert(bytes_.end(), value.begin(), value.end());
}
void write_nil() { bytes_.push_back(0xc0); }
void write_bool(bool value) { bytes_.push_back(value ? 0xc3 : 0xc2); }
void write_u64(uint64_t value)
{
if (value <= 0x7f) {
bytes_.push_back(static_cast<uint8_t>(value));
} else if (value <= 0xff) {
bytes_.push_back(0xcc);
bytes_.push_back(static_cast<uint8_t>(value));
} else if (value <= 0xffff) {
bytes_.push_back(0xcd);
write_be16_value(static_cast<uint16_t>(value));
} else if (value <= 0xffffffffull) {
bytes_.push_back(0xce);
write_be32_value(static_cast<uint32_t>(value));
} else {
bytes_.push_back(0xcf);
write_be64_value(value);
}
}
void write_i64(int64_t value)
{
if (value >= 0) {
write_u64(static_cast<uint64_t>(value));
return;
}
if (value >= -32) {
bytes_.push_back(static_cast<uint8_t>(0xe0 | (value + 32)));
} else if (value >= INT8_MIN) {
bytes_.push_back(0xd0);
bytes_.push_back(static_cast<uint8_t>(value));
} else if (value >= INT16_MIN) {
bytes_.push_back(0xd1);
write_be16_value(static_cast<uint16_t>(value));
} else if (value >= INT32_MIN) {
bytes_.push_back(0xd2);
write_be32_value(static_cast<uint32_t>(value));
} else {
bytes_.push_back(0xd3);
write_be64_value(static_cast<uint64_t>(value));
}
}
void write_string(const std::string &value)
{
const auto length = value.size();
if (length <= 31) {
bytes_.push_back(static_cast<uint8_t>(0xa0 | length));
} else if (length <= 0xff) {
bytes_.push_back(0xd9);
bytes_.push_back(static_cast<uint8_t>(length));
} else if (length <= 0xffff) {
bytes_.push_back(0xda);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xdb);
write_be32_value(static_cast<uint32_t>(length));
}
bytes_.insert(bytes_.end(), value.begin(), value.end());
}
void write_binary(const std::vector<uint8_t> &value)
{
const auto length = value.size();
if (length <= 0xff) {
bytes_.push_back(0xc4);
bytes_.push_back(static_cast<uint8_t>(length));
} else if (length <= 0xffff) {
bytes_.push_back(0xc5);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xc6);
write_be32_value(static_cast<uint32_t>(length));
}
bytes_.insert(bytes_.end(), value.begin(), value.end());
}
void write_array_len(uint32_t length)
{
if (length <= 15) {
bytes_.push_back(static_cast<uint8_t>(0x90 | length));
} else if (length <= 0xffff) {
bytes_.push_back(0xdc);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xdd);
write_be32_value(length);
}
}
void write_map_len(uint32_t length)
{
if (length <= 15) {
bytes_.push_back(static_cast<uint8_t>(0x80 | length));
} else if (length <= 0xffff) {
bytes_.push_back(0xde);
write_be16_value(static_cast<uint16_t>(length));
} else {
bytes_.push_back(0xdf);
write_be32_value(length);
}
}
private:
std::vector<uint8_t> bytes_;
void write_be16_value(uint16_t value)
{
bytes_.push_back(static_cast<uint8_t>((value >> 8) & 0xff));
bytes_.push_back(static_cast<uint8_t>(value & 0xff));
}
void write_be32_value(uint32_t value)
{
bytes_.push_back(static_cast<uint8_t>((value >> 24) & 0xff));
bytes_.push_back(static_cast<uint8_t>((value >> 16) & 0xff));
bytes_.push_back(static_cast<uint8_t>((value >> 8) & 0xff));
bytes_.push_back(static_cast<uint8_t>(value & 0xff));
}
void write_be64_value(uint64_t value)
{
write_be32_value(static_cast<uint32_t>((value >> 32) & 0xffffffffull));
write_be32_value(static_cast<uint32_t>(value & 0xffffffffull));
}
};
inline std::string feedback_observed_at()
{
const auto now = std::chrono::system_clock::now().time_since_epoch();
const auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(now).count();
return "unix-ms:" + std::to_string(millis);
}
inline void write_string_array(MsgpackWriter &writer, const std::vector<std::string> &values)
{
writer.write_array_len(static_cast<uint32_t>(values.size()));
for (const auto &value : values) {
writer.write_string(value);
}
}
inline void write_u64_array(MsgpackWriter &writer, const std::vector<uint64_t> &values)
{
writer.write_array_len(static_cast<uint32_t>(values.size()));
for (const auto value : values) {
writer.write_u64(value);
}
}
inline std::vector<uint64_t> receiver_feedback_late_frame_ids(uint64_t frame_id, bool late_frame)
{
return late_frame ? std::vector<uint64_t>{frame_id} : std::vector<uint64_t>{};
}
inline std::vector<uint8_t> encode_receiver_feedback_payload(
const std::string &stream_id,
const std::string &session_id,
uint64_t frame_id,
const std::vector<std::string> &missing_chunk_keys,
bool requested_keyframe,
bool late_frame,
const std::string &observed_at)
{
MsgpackWriter record;
record.write_array_len(11);
record.write_string(stream_id);
record.write_string(session_id);
record.write_string("starfire.obs");
if (frame_id > 0) {
record.write_u64(frame_id - 1);
} else {
record.write_nil();
}
write_u64_array(record, {});
write_u64_array(record, receiver_feedback_late_frame_ids(frame_id, late_frame));
record.write_bool(requested_keyframe);
record.write_i64(0);
record.write_i64(0);
record.write_string(observed_at);
write_string_array(record, missing_chunk_keys);
const std::string record_key = stream_id + ":" + session_id + ":feedback:starfire.obs";
const std::string message_id =
"muninn-media:" + std::string(ReceiverFeedbackSchema) + ":" + record_key;
MsgpackWriter document;
document.write_map_len(9);
document.write_string("schemaId");
document.write_string(ReceiverFeedbackSchema);
document.write_string("recordKey");
document.write_string(record_key);
document.write_string("storedAt");
document.write_string(observed_at);
document.write_string("payloadEncoding");
document.write_string("messagepack");
document.write_string("payload");
document.write_binary(record.bytes());
document.write_string("sourceRuntimeId");
document.write_string("starfire");
document.write_string("sourceAgentId");
document.write_nil();
document.write_string("sourceRole");
document.write_string("mimir.obs");
document.write_string("tags");
write_string_array(document, {"muninn.media"});
MsgpackWriter wire;
wire.write_map_len(3);
wire.write_string("schemaVersion");
wire.write_string("cultnet.document_put_raw.v0");
wire.write_string("messageId");
wire.write_string(message_id);
wire.write_string("document");
wire.write_raw(document.bytes());
return wire.bytes();
}
inline std::vector<uint8_t> encode_receiver_feedback_payload(
const std::string &stream_id,
const std::string &session_id,
uint64_t frame_id,
const std::vector<std::string> &missing_chunk_keys,
bool requested_keyframe)
{
return encode_receiver_feedback_payload(
stream_id,
session_id,
frame_id,
missing_chunk_keys,
requested_keyframe,
requested_keyframe,
feedback_observed_at());
}
inline std::vector<uint8_t> encode_receiver_feedback_payload(
const std::string &stream_id,
const std::string &session_id,
uint64_t frame_id,
const std::vector<std::string> &missing_chunk_keys)
{
return encode_receiver_feedback_payload(
stream_id,
session_id,
frame_id,
missing_chunk_keys,
true);
}
} // namespace muninn_media_wire

View file

@ -0,0 +1,54 @@
#pragma once
#include <cstdint>
#include <optional>
#include <set>
#include <utility>
namespace muninn_rudp_ack {
class AckTracker {
public:
void reset()
{
received_sequences_.clear();
highest_received_sequence_.reset();
}
void remember(uint32_t sequence)
{
received_sequences_.insert(sequence);
if (!highest_received_sequence_ || sequence > *highest_received_sequence_) {
highest_received_sequence_ = sequence;
}
if (!highest_received_sequence_) {
return;
}
const uint32_t highest = *highest_received_sequence_;
for (auto iterator = received_sequences_.begin(); iterator != received_sequences_.end();) {
if (*iterator + 32 < highest) {
iterator = received_sequences_.erase(iterator);
} else {
++iterator;
}
}
}
std::pair<uint32_t, uint32_t> state() const
{
const uint32_t ack = highest_received_sequence_.value_or(0);
uint32_t ack_mask = 0;
for (uint32_t bit = 0; bit < 32; ++bit) {
if (ack > bit && received_sequences_.find(ack - bit - 1) != received_sequences_.end()) {
ack_mask |= 1u << bit;
}
}
return {ack, ack_mask};
}
private:
std::set<uint32_t> received_sequences_;
std::optional<uint32_t> highest_received_sequence_;
};
} // namespace muninn_rudp_ack

View file

@ -0,0 +1,127 @@
#pragma once
#include <algorithm>
#include <chrono>
#include <cstdint>
#include <map>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace muninn_rudp_fragments {
constexpr uint32_t CompletedFragmentIdQuarantineMs = 2000;
struct FragmentResult {
uint32_t first_sequence = 0;
std::vector<uint8_t> payload;
};
class FragmentAssembler {
public:
void reset()
{
assemblies_.clear();
recently_closed_.clear();
expired_count_ = 0;
}
std::optional<FragmentResult> insert(const std::string &channel_id,
uint16_t fragment_id,
uint16_t fragment_index,
uint16_t fragment_count,
uint32_t sequence,
const uint8_t *payload,
uint32_t payload_len,
std::chrono::steady_clock::time_point now)
{
if (fragment_id == 0 || fragment_index >= fragment_count) {
return std::nullopt;
}
const std::string key = channel_id + ":" + std::to_string(fragment_id);
expire_recently_closed(now);
const auto existing = assemblies_.find(key);
if (existing == assemblies_.end() && recently_closed_.find(key) != recently_closed_.end()) {
return std::nullopt;
}
auto &assembly = assemblies_[key];
if (assembly.payloads.empty()) {
assembly.channel_id = channel_id;
assembly.fragment_count = fragment_count;
assembly.started_at = now;
}
if (assembly.channel_id != channel_id || assembly.fragment_count != fragment_count) {
assemblies_.erase(key);
return std::nullopt;
}
assembly.payloads.emplace(fragment_index, std::vector<uint8_t>(payload, payload + payload_len));
assembly.sequences.emplace(fragment_index, sequence);
if (assembly.payloads.size() < assembly.fragment_count) {
return std::nullopt;
}
FragmentResult result;
result.first_sequence = UINT32_MAX;
for (uint16_t index = 0; index < assembly.fragment_count; ++index) {
const auto payload_found = assembly.payloads.find(index);
const auto sequence_found = assembly.sequences.find(index);
if (payload_found == assembly.payloads.end() || sequence_found == assembly.sequences.end()) {
return std::nullopt;
}
result.payload.insert(result.payload.end(), payload_found->second.begin(), payload_found->second.end());
result.first_sequence = std::min(result.first_sequence, sequence_found->second);
}
assemblies_.erase(key);
recently_closed_[key] = now;
return result;
}
uint64_t expire(std::chrono::steady_clock::time_point now, uint32_t expire_after_ms)
{
expire_recently_closed(now);
uint64_t expired = 0;
for (auto iterator = assemblies_.begin(); iterator != assemblies_.end();) {
if (now - iterator->second.started_at < std::chrono::milliseconds(expire_after_ms)) {
++iterator;
continue;
}
recently_closed_[iterator->first] = now;
iterator = assemblies_.erase(iterator);
++expired;
}
expired_count_ += expired;
return expired;
}
uint64_t expired_count() const { return expired_count_; }
private:
void expire_recently_closed(std::chrono::steady_clock::time_point now)
{
for (auto iterator = recently_closed_.begin(); iterator != recently_closed_.end();) {
if (now - iterator->second < std::chrono::milliseconds(CompletedFragmentIdQuarantineMs)) {
++iterator;
continue;
}
iterator = recently_closed_.erase(iterator);
}
}
struct Assembly {
std::string channel_id;
uint16_t fragment_count = 0;
std::map<uint16_t, std::vector<uint8_t>> payloads;
std::map<uint16_t, uint32_t> sequences;
std::chrono::steady_clock::time_point started_at;
};
std::map<std::string, Assembly> assemblies_;
std::map<std::string, std::chrono::steady_clock::time_point> recently_closed_;
uint64_t expired_count_ = 0;
};
} // namespace muninn_rudp_fragments

View file

@ -0,0 +1,126 @@
#pragma once
#include <algorithm>
#include <cstdint>
#include <optional>
#include <sstream>
#include <string>
namespace muninn_rudp_url {
inline constexpr uint32_t DefaultMediaRudpConnectionId = 0x6d750001;
inline constexpr uint32_t DefaultAudioRudpConnectionId = 0x6d750004;
inline constexpr uint32_t DefaultReliableExpireAfterMs = 75;
inline constexpr uint32_t DefaultVideoAssemblyDeadlineMs = 75;
inline constexpr uint32_t DefaultGapWaitMs = 8;
inline constexpr uint32_t DefaultAudioReorderMs = 120;
inline constexpr uint32_t DefaultSenderResendDelayMs = 5;
inline constexpr uint32_t MaxLatencyBudgetMs = 500;
struct RudpUrlParts {
std::string channel_id = "media";
uint16_t port = 0;
uint16_t video_local_port = 0;
uint16_t audio_local_port = 0;
uint32_t connection_id = DefaultMediaRudpConnectionId;
uint32_t audio_connection_id = DefaultAudioRudpConnectionId;
uint32_t sender_resend_delay_ms = DefaultSenderResendDelayMs;
uint32_t reliable_expire_after_ms = DefaultReliableExpireAfterMs;
uint32_t video_assembly_deadline_ms = DefaultVideoAssemblyDeadlineMs;
uint32_t gap_wait_ms = DefaultGapWaitMs;
uint32_t audio_reorder_ms = DefaultAudioReorderMs;
};
inline bool starts_with(const std::string &value, const std::string &prefix)
{
return value.rfind(prefix, 0) == 0;
}
inline std::optional<uint32_t> parse_u32_hex_or_decimal(const std::string &value)
{
try {
size_t consumed = 0;
const int base = starts_with(value, "0x") || starts_with(value, "0X") ? 16 : 10;
const auto parsed = std::stoul(value, &consumed, base);
if (consumed == value.size()) {
return static_cast<uint32_t>(parsed);
}
} catch (...) {
}
return std::nullopt;
}
inline std::optional<RudpUrlParts> parse(const std::string &url)
{
if (!starts_with(url, "rudp://")) {
return std::nullopt;
}
const auto authority_start = std::string("rudp://").size();
const auto path_start = url.find('/', authority_start);
const auto query_start = url.find('?', authority_start);
const auto authority_end = std::min(path_start == std::string::npos ? url.size() : path_start,
query_start == std::string::npos ? url.size() : query_start);
const auto authority = url.substr(authority_start, authority_end - authority_start);
const auto colon = authority.rfind(':');
if (colon == std::string::npos || colon + 1 >= authority.size()) {
return std::nullopt;
}
RudpUrlParts parts;
parts.port = static_cast<uint16_t>(std::stoul(authority.substr(colon + 1)));
parts.video_local_port = static_cast<uint16_t>(parts.port + 10000);
parts.audio_local_port = static_cast<uint16_t>(parts.port + 10001);
if (query_start != std::string::npos) {
std::stringstream query(url.substr(query_start + 1));
std::string item;
while (std::getline(query, item, '&')) {
const auto equals = item.find('=');
if (equals == std::string::npos) {
continue;
}
const auto key = item.substr(0, equals);
const auto value = item.substr(equals + 1);
if (key == "channel" && !value.empty()) {
parts.channel_id = value;
} else if (key == "connection") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.connection_id = *parsed;
}
} else if (key == "audio_connection" || key == "audio_connection_id") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.audio_connection_id = *parsed;
}
} else if (key == "local_port" || key == "video_local_port") {
parts.video_local_port = static_cast<uint16_t>(std::stoul(value));
} else if (key == "audio_local_port") {
parts.audio_local_port = static_cast<uint16_t>(std::stoul(value));
} else if (key == "reliable_expire_after_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.reliable_expire_after_ms = std::clamp(*parsed, 1u, MaxLatencyBudgetMs);
}
} else if (key == "sender_resend_delay_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.sender_resend_delay_ms = std::clamp(*parsed, 1u, 1000u);
}
} else if (key == "assembly_deadline_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.video_assembly_deadline_ms = std::clamp(*parsed, 1u, MaxLatencyBudgetMs);
}
} else if (key == "gap_wait_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.gap_wait_ms = std::clamp(*parsed, 0u, 1000u);
}
} else if (key == "audio_reorder_ms") {
if (const auto parsed = parse_u32_hex_or_decimal(value)) {
parts.audio_reorder_ms = std::clamp(*parsed, 0u, MaxLatencyBudgetMs);
}
}
}
}
return parts;
}
} // namespace muninn_rudp_url

View file

@ -0,0 +1,32 @@
#pragma once
#ifdef _WIN32
#include <winsock2.h>
namespace muninn_udp_socket {
inline SOCKET create(bool exclusive_listener = false)
{
const SOCKET socket = WSASocketW(
AF_INET,
SOCK_DGRAM,
IPPROTO_UDP,
nullptr,
0,
WSA_FLAG_NO_HANDLE_INHERIT);
if (socket == INVALID_SOCKET) {
return INVALID_SOCKET;
}
if (exclusive_listener) {
const BOOL exclusive = TRUE;
if (setsockopt(socket, SOL_SOCKET, SO_EXCLUSIVEADDRUSE,
reinterpret_cast<const char *>(&exclusive), sizeof(exclusive)) != 0) {
closesocket(socket);
return INVALID_SOCKET;
}
}
return socket;
}
} // namespace muninn_udp_socket
#endif

View file

@ -0,0 +1,181 @@
#pragma once
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <map>
#include <utility>
#include <vector>
namespace muninn_video_fec {
inline uint8_t raw_multiply(uint8_t a, uint8_t b)
{
uint8_t product = 0;
while (b != 0) {
if ((b & 1) != 0)
product ^= a;
const bool carry = (a & 0x80) != 0;
a <<= 1;
if (carry)
a ^= 0x1d;
b >>= 1;
}
return product;
}
struct Tables {
uint8_t products[256][256] = {};
uint8_t inverses[256] = {};
Tables()
{
for (size_t a = 0; a < 256; ++a) {
for (size_t b = 0; b < 256; ++b)
products[a][b] = raw_multiply(static_cast<uint8_t>(a), static_cast<uint8_t>(b));
}
for (size_t value = 1; value < 256; ++value) {
uint8_t result = 1;
for (size_t index = 0; index < 254; ++index)
result = raw_multiply(result, static_cast<uint8_t>(value));
inverses[value] = result;
}
}
};
inline const Tables &tables()
{
static const Tables value;
return value;
}
inline uint8_t multiply(uint8_t a, uint8_t b)
{
return tables().products[a][b];
}
inline uint8_t inverse(uint8_t value)
{
return tables().inverses[value];
}
inline uint8_t coefficient(uint16_t parity_index, uint16_t parity_count, uint16_t data_index)
{
return inverse(
static_cast<uint8_t>(parity_index) ^ static_cast<uint8_t>(parity_count + data_index));
}
inline bool recover(std::vector<std::vector<uint8_t>> &chunks,
const std::vector<uint32_t> &chunk_lengths,
uint16_t parity_count,
const std::map<uint16_t, std::vector<uint8_t>> &parity)
{
if (chunks.size() != chunk_lengths.size() || chunks.empty() ||
chunks.size() + parity_count > 256)
return false;
std::vector<uint16_t> missing;
for (uint16_t index = 0; index < chunks.size(); ++index) {
if (chunks[index].empty())
missing.push_back(index);
}
if (missing.empty())
return true;
if (missing.size() > parity.size())
return false;
std::vector<std::pair<uint16_t, const std::vector<uint8_t> *>> rows;
for (const auto &shard : parity) {
if (shard.first < parity_count && !shard.second.empty())
rows.emplace_back(shard.first, &shard.second);
if (rows.size() == missing.size())
break;
}
if (rows.size() != missing.size())
return false;
size_t shard_bytes = 0;
for (uint32_t length : chunk_lengths)
shard_bytes = std::max(shard_bytes, static_cast<size_t>(length));
if (shard_bytes == 0 || std::any_of(rows.begin(), rows.end(),
[shard_bytes](const auto &row) { return row.second->size() < shard_bytes; }))
return false;
std::vector<std::vector<uint8_t>> recovered;
recovered.reserve(missing.size());
for (uint16_t index : missing) {
if (chunk_lengths[index] == 0 || chunk_lengths[index] > shard_bytes)
return false;
recovered.emplace_back(chunk_lengths[index], 0);
}
const size_t count = missing.size();
std::vector<std::vector<uint8_t>> inverse_matrix(count, std::vector<uint8_t>(count * 2, 0));
for (size_t row = 0; row < count; ++row) {
for (size_t column = 0; column < count; ++column)
inverse_matrix[row][column] = coefficient(rows[row].first, parity_count, missing[column]);
inverse_matrix[row][count + row] = 1;
}
for (size_t column = 0; column < count; ++column) {
size_t pivot = column;
while (pivot < count && inverse_matrix[pivot][column] == 0)
++pivot;
if (pivot == count)
return false;
std::swap(inverse_matrix[column], inverse_matrix[pivot]);
const uint8_t scale = inverse(inverse_matrix[column][column]);
for (size_t cell = 0; cell < count * 2; ++cell)
inverse_matrix[column][cell] = multiply(inverse_matrix[column][cell], scale);
for (size_t row = 0; row < count; ++row) {
if (row == column)
continue;
const uint8_t factor = inverse_matrix[row][column];
for (size_t cell = 0; cell < count * 2; ++cell)
inverse_matrix[row][cell] ^= multiply(factor, inverse_matrix[column][cell]);
}
}
std::vector<uint8_t> rhs(count, 0);
for (size_t offset = 0; offset < shard_bytes; ++offset) {
for (size_t row = 0; row < count; ++row) {
const uint16_t parity_index = rows[row].first;
rhs[row] = (*rows[row].second)[offset];
for (uint16_t data_index = 0; data_index < chunks.size(); ++data_index) {
if (!chunks[data_index].empty() && offset < chunks[data_index].size())
rhs[row] ^= multiply(coefficient(parity_index, parity_count, data_index),
chunks[data_index][offset]);
}
}
for (size_t index = 0; index < count; ++index) {
uint8_t value = 0;
for (size_t row = 0; row < count; ++row)
value ^= multiply(inverse_matrix[index][count + row], rhs[row]);
if (offset < recovered[index].size())
recovered[index][offset] = value;
}
}
for (size_t index = 0; index < count; ++index)
chunks[missing[index]] = std::move(recovered[index]);
return true;
}
inline bool recover_block(std::vector<std::vector<uint8_t>> &frame_chunks,
uint16_t data_start,
const std::vector<uint32_t> &block_lengths,
uint16_t parity_count,
const std::map<uint16_t, std::vector<uint8_t>> &parity)
{
if (block_lengths.empty() ||
static_cast<size_t>(data_start) + block_lengths.size() > frame_chunks.size())
return false;
std::vector<std::vector<uint8_t>> block_chunks;
block_chunks.reserve(block_lengths.size());
for (size_t index = 0; index < block_lengths.size(); ++index)
block_chunks.push_back(frame_chunks[data_start + index]);
if (!recover(block_chunks, block_lengths, parity_count, parity))
return false;
for (size_t index = 0; index < block_lengths.size(); ++index)
frame_chunks[data_start + index] = std::move(block_chunks[index]);
return true;
}
} // namespace muninn_video_fec

View file

@ -0,0 +1,79 @@
#include "muninn_audio_fec.h"
#include <cstdlib>
#include <iostream>
#include <stdexcept>
static void require(bool condition, const char *message) { if (!condition) throw std::runtime_error(message); }
static std::array<muninn_audio_fec::Bytes, 4> fixture()
{
std::array<muninn_audio_fec::Bytes, 4> data;
for (size_t shard = 0; shard < data.size(); ++shard) {
data[shard].resize(257);
for (size_t i = 0; i < data[shard].size(); ++i) data[shard][i] = uint8_t((shard * 71 + i * 29) & 0xff);
}
return data;
}
static muninn_audio_fec::Shards all_shards(const std::array<muninn_audio_fec::Bytes, 4> &data)
{
auto parity = muninn_audio_fec::encode(data);
return {data[0], data[1], data[2], data[3], parity[0], parity[1]};
}
int main()
{
try {
const auto data = fixture();
for (size_t erased = 0; erased < 6; ++erased) {
auto shards = all_shards(data); shards[erased].reset();
require(muninn_audio_fec::recover(shards), "one-erasure recovery failed");
for (size_t i = 0; i < 4; ++i) require(shards[i] == data[i], "one-erasure data mismatch");
}
for (size_t first = 0; first < 6; ++first) for (size_t second = first + 1; second < 6; ++second) {
auto shards = all_shards(data); shards[first].reset(); shards[second].reset();
require(muninn_audio_fec::recover(shards), "two-erasure recovery failed");
for (size_t i = 0; i < 4; ++i) require(shards[i] == data[i], "two-erasure data mismatch");
}
{
auto shards = all_shards(data);
shards[0].reset(); shards[1].reset(); shards[2].reset();
require(!muninn_audio_fec::recover(shards), "three erasures were incorrectly recoverable");
}
{
auto shards = all_shards(data);
shards[5]->pop_back();
require(!muninn_audio_fec::recover(shards), "unequal shard sizes were accepted");
}
using namespace muninn_audio_fec;
BlockReceiver receiver(std::chrono::milliseconds(10), 2);
BlockKey key{"stream", "session", 100};
auto shards = all_shards(data);
auto t0 = BlockReceiver::Clock::time_point{};
require(receiver.insert(key, 5, *shards[5], t0).empty(), "early parity emitted data");
require(receiver.insert(key, 2, *shards[2], t0).empty(), "partial reorder emitted data");
require(receiver.insert(key, 2, *shards[2], t0).empty(), "duplicate emitted data");
require(receiver.insert(key, 1, *shards[1], t0).empty(), "partial block emitted data");
auto recovered = receiver.insert(key, 4, *shards[4], t0);
require(recovered.size() == 2 && recovered[0].packet_id == 100 && recovered[1].packet_id == 103,
"reordered block did not recover missing packet ids");
require(recovered[0].payload == data[0] && recovered[1].payload == data[3], "receiver recovery mismatch");
require(receiver.insert(key, 0, data[0], t0).empty(), "completed-block duplicate was admitted");
BlockKey expired{"stream", "session", 200};
receiver.insert(expired, 0, data[0], t0);
receiver.expire(t0 + std::chrono::milliseconds(10));
require(receiver.pending_blocks() == 0, "expired block survived deadline");
BlockReceiver bounded(std::chrono::seconds(1), 2);
bounded.insert({"s", "x", 0}, 0, data[0], t0);
bounded.insert({"s", "x", 4}, 0, data[0], t0);
bounded.insert({"s", "x", 8}, 0, data[0], t0);
require(bounded.pending_blocks() == 2, "block bound was not enforced");
} catch (const std::exception &error) {
std::cerr << error.what() << '\n'; return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}

View file

@ -0,0 +1,54 @@
#include "muninn_media_wire.h"
#include <cstdlib>
#include <iomanip>
#include <iostream>
namespace {
void require(bool condition, const char *message)
{
if (!condition) {
std::cerr << message << '\n';
std::exit(1);
}
}
} // namespace
int main(int argc, char **argv)
{
const auto early = muninn_media_wire::receiver_feedback_late_frame_ids(42, false);
require(early.empty(), "early repair feedback marked a live frame late");
const auto expired = muninn_media_wire::receiver_feedback_late_frame_ids(42, true);
require(expired.size() == 1 && expired.front() == 42,
"expired feedback did not name the late frame");
const auto early_wire = muninn_media_wire::encode_receiver_feedback_payload(
"muninn.raven.av.rudp",
"raven:session:video",
42,
{"42:3"},
false,
false,
"unix-ms:1000");
const auto expired_wire = muninn_media_wire::encode_receiver_feedback_payload(
"muninn.raven.av.rudp",
"raven:session:video",
42,
{"42:3"},
true,
true,
"unix-ms:1000");
require(!early_wire.empty() && !expired_wire.empty(), "feedback wire encoding was empty");
require(early_wire != expired_wire, "early and expired feedback encoded identically");
if (argc == 2 && std::string(argv[1]) == "--emit-early-hex") {
for (const auto byte : early_wire) {
std::cout << std::hex << std::setw(2) << std::setfill('0')
<< static_cast<unsigned int>(byte);
}
std::cout << '\n';
}
return 0;
}

View file

@ -0,0 +1,60 @@
#include "muninn_rudp_url.h"
#include <cstdlib>
#include <iostream>
#include <string>
namespace {
void require(bool condition, const std::string &message)
{
if (!condition) {
std::cerr << message << '\n';
std::exit(1);
}
}
} // namespace
int main()
{
const auto configured = muninn_rudp_url::parse(
"rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=realtime&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=120&assembly_deadline_ms=120&gap_wait_ms=16&audio_reorder_ms=80");
require(configured.has_value(), "configured RUDP URL did not parse");
require(configured->port == 5204, "configured RUDP URL port was not parsed");
require(configured->channel_id == "realtime", "configured RUDP realtime channel was not parsed");
require(configured->video_local_port == 15204, "configured RUDP video local port was not derived");
require(configured->audio_local_port == 15205, "configured RUDP audio local port was not derived");
require(configured->connection_id == 0x6d750001, "configured RUDP connection id was not parsed");
require(configured->audio_connection_id == 0x6d750004, "configured RUDP audio connection id was not parsed");
require(configured->reliable_expire_after_ms == 120, "configured reliable expiry did not survive parse");
require(configured->video_assembly_deadline_ms == 120, "configured assembly deadline did not survive parse");
require(configured->gap_wait_ms == 16, "configured gap wait did not survive parse");
require(configured->audio_reorder_ms == 80, "configured audio reorder wait did not survive parse");
const auto clamped = muninn_rudp_url::parse(
"rudp://192.168.1.66:5204/muninn.raven.av.rudp?reliable_expire_after_ms=9000&assembly_deadline_ms=9000");
require(clamped.has_value(), "oversized latency RUDP URL did not parse");
require(clamped->reliable_expire_after_ms == muninn_rudp_url::MaxLatencyBudgetMs,
"oversized reliable expiry did not clamp to the max latency budget");
require(clamped->video_assembly_deadline_ms == muninn_rudp_url::MaxLatencyBudgetMs,
"oversized assembly deadline did not clamp to the max latency budget");
const auto fallback = muninn_rudp_url::parse("rudp://192.168.1.66:5204/muninn.raven.av.rudp");
require(fallback.has_value(), "minimal RUDP URL did not parse");
require(fallback->channel_id == "media", "minimal RUDP URL did not use bounded reliable media delivery");
require(fallback->connection_id == muninn_rudp_url::DefaultMediaRudpConnectionId,
"minimal RUDP URL did not keep the default connection id");
require(fallback->audio_connection_id == muninn_rudp_url::DefaultAudioRudpConnectionId,
"minimal RUDP URL did not keep the default audio connection id");
require(fallback->reliable_expire_after_ms == muninn_rudp_url::DefaultReliableExpireAfterMs,
"minimal RUDP URL did not keep the default reliable expiry");
require(fallback->video_assembly_deadline_ms == muninn_rudp_url::DefaultVideoAssemblyDeadlineMs,
"minimal RUDP URL did not keep the default assembly deadline");
require(fallback->audio_reorder_ms == muninn_rudp_url::DefaultAudioReorderMs,
"minimal RUDP URL did not keep the default audio reorder wait");
require(fallback->audio_reorder_ms == 120,
"minimal RUDP URL did not use the bounded realtime audio reorder budget");
return 0;
}

View file

@ -0,0 +1,34 @@
#include "muninn_udp_socket.h"
#include <cassert>
#include <windows.h>
int main()
{
WSADATA data{};
assert(WSAStartup(MAKEWORD(2, 2), &data) == 0);
const SOCKET owner = muninn_udp_socket::create(true);
assert(owner != INVALID_SOCKET);
DWORD flags = HANDLE_FLAG_INHERIT;
assert(GetHandleInformation(reinterpret_cast<HANDLE>(owner), &flags));
assert((flags & HANDLE_FLAG_INHERIT) == 0);
sockaddr_in address{};
address.sin_family = AF_INET;
address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
address.sin_port = 0;
assert(bind(owner, reinterpret_cast<sockaddr *>(&address), sizeof(address)) == 0);
int address_size = sizeof(address);
assert(getsockname(owner, reinterpret_cast<sockaddr *>(&address), &address_size) == 0);
const SOCKET contender = muninn_udp_socket::create(true);
assert(contender != INVALID_SOCKET);
assert(bind(contender, reinterpret_cast<sockaddr *>(&address), sizeof(address)) != 0);
assert(WSAGetLastError() == WSAEADDRINUSE || WSAGetLastError() == WSAEACCES);
closesocket(contender);
closesocket(owner);
WSACleanup();
return 0;
}

View file

@ -0,0 +1,111 @@
#include "muninn_video_fec.h"
#include <chrono>
#include <cstdlib>
#include <cstdint>
#include <map>
#include <vector>
void require(bool condition)
{
if (!condition)
std::abort();
}
int main()
{
constexpr uint16_t data_count = 64;
constexpr uint16_t parity_count = 16;
constexpr size_t shard_bytes = 1400;
std::vector<std::vector<uint8_t>> original(data_count, std::vector<uint8_t>(shard_bytes));
for (uint16_t data = 0; data < data_count; ++data) {
for (size_t offset = 0; offset < shard_bytes; ++offset)
original[data][offset] = static_cast<uint8_t>((data * 29 + offset * 17) & 0xff);
}
std::map<uint16_t, std::vector<uint8_t>> parity;
for (uint16_t row = 0; row < parity_count; ++row) {
auto &shard = parity[row];
shard.assign(shard_bytes, 0);
for (uint16_t data = 0; data < data_count; ++data) {
const uint8_t coefficient = muninn_video_fec::coefficient(row, parity_count, data);
for (size_t offset = 0; offset < shard_bytes; ++offset)
shard[offset] ^= muninn_video_fec::multiply(original[data][offset], coefficient);
}
}
const std::vector<uint32_t> lengths(data_count, shard_bytes);
for (uint16_t burst = 1; burst <= parity_count; ++burst) {
auto damaged = original;
for (uint16_t index = 0; index < burst; ++index)
damaged[20 + index].clear();
const auto started = std::chrono::steady_clock::now();
require(muninn_video_fec::recover(damaged, lengths, parity_count, parity));
require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50));
require(damaged == original);
}
auto excessive = original;
for (uint16_t index = 0; index <= parity_count; ++index)
excessive[index].clear();
require(!muninn_video_fec::recover(excessive, lengths, parity_count, parity));
constexpr uint16_t block_data = 8;
constexpr uint16_t frame_data = 80;
constexpr uint16_t block_count = frame_data / block_data;
constexpr size_t block_shard_bytes = 96;
std::vector<std::vector<uint8_t>> frame(frame_data, std::vector<uint8_t>(block_shard_bytes));
for (uint16_t data = 0; data < frame_data; ++data) {
for (size_t offset = 0; offset < block_shard_bytes; ++offset)
frame[data][offset] = static_cast<uint8_t>((data * 31 + offset * 13) & 0xff);
}
std::vector<std::map<uint16_t, std::vector<uint8_t>>> block_parity(block_count);
for (uint16_t block = 0; block < block_count; ++block) {
for (uint16_t row = 0; row < block_data; ++row) {
auto &shard = block_parity[block][row];
shard.assign(block_shard_bytes, 0);
for (uint16_t local = 0; local < block_data; ++local) {
const auto &data = frame[block * block_data + local];
const uint8_t coefficient = muninn_video_fec::coefficient(row, block_data, local);
for (size_t offset = 0; offset < block_shard_bytes; ++offset)
shard[offset] ^= muninn_video_fec::multiply(data[offset], coefficient);
}
}
}
struct WireShard {
bool parity;
uint16_t block;
uint16_t local;
};
std::vector<WireShard> wire;
for (uint16_t local = 0; local < block_data; ++local) {
for (uint16_t block = 0; block < block_count; ++block)
wire.push_back({false, block, local});
for (uint16_t block = 0; block < block_count; ++block)
wire.push_back({true, block, local});
}
require(wire.size() == frame_data * 2);
const std::vector<uint32_t> block_lengths(block_data, block_shard_bytes);
for (size_t burst = 1; burst <= 54; ++burst) {
for (size_t start = 0; start + burst <= wire.size(); ++start) {
auto damaged = frame;
auto available_parity = block_parity;
for (size_t index = start; index < start + burst; ++index) {
const auto shard = wire[index];
if (shard.parity)
available_parity[shard.block].erase(shard.local);
else
damaged[shard.block * block_data + shard.local].clear();
}
const auto started = std::chrono::steady_clock::now();
for (uint16_t block = 0; block < block_count; ++block) {
require(muninn_video_fec::recover_block(
damaged,
block * block_data,
block_lengths,
block_data,
available_parity[block]));
}
require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50));
require(damaged == frame);
}
}
return 0;
}

View file

@ -17,7 +17,8 @@ enum LocalcastSampleKind {
LOCALCAST_SAMPLE_AUDIO_BLOCK = 5,
LOCALCAST_SAMPLE_PHASE_CLAIM = 6,
LOCALCAST_SAMPLE_EVENT_CLAIM = 7,
LOCALCAST_SAMPLE_RENDER_PACKET = 8
LOCALCAST_SAMPLE_RENDER_PACKET = 8,
LOCALCAST_SAMPLE_MOVE_EVIDENCE = 9
};
typedef struct LocalcastReservoir LocalcastReservoir;
@ -79,6 +80,47 @@ typedef struct LocalcastRenderPoint {
float confidence;
} LocalcastRenderPoint;
enum LocalcastMoveEvidenceKind {
LOCALCAST_MOVE_EVIDENCE_OPTICAL_MARKER = 1,
LOCALCAST_MOVE_EVIDENCE_CONTROLLER_STATE = 2
};
typedef struct LocalcastMoveEvidenceSample {
uint64_t witness_id_hash;
uint64_t controller_id_hash;
uint64_t source_timestamp_ns;
uint64_t arrival_ns;
uint64_t sequence;
uint32_t evidence_kind;
uint32_t flags;
float image_x;
float image_y;
float radius_px;
float confidence;
float accel_x;
float accel_y;
float accel_z;
float gyro_x;
float gyro_y;
float gyro_z;
float trigger;
uint32_t buttons_mask;
uint32_t reserved;
float battery01;
uint32_t reserved1;
uint32_t reserved2;
} LocalcastMoveEvidenceSample;
typedef struct LocalcastMoveEvidenceBufferDescriptor {
uint64_t sample_buffer_handle;
uint32_t sample_count;
uint32_t sample_stride_bytes;
uint64_t source_time_min_ns;
uint64_t source_time_max_ns;
uint64_t calibration_hash;
uint64_t tracking_space_hash;
} LocalcastMoveEvidenceBufferDescriptor;
typedef struct LocalcastRuntimeStatus {
uint64_t edge_ns;
uint64_t window_start_ns;
@ -92,6 +134,7 @@ typedef struct LocalcastRuntimeStatus {
size_t phase_claim_count;
size_t event_claim_count;
size_t render_packet_count;
size_t move_evidence_count;
} LocalcastRuntimeStatus;
uint64_t localcast_hash_source_id(const uint8_t *bytes, size_t byte_len);
@ -144,6 +187,7 @@ bool localcast_runtime_push_audio_block(LocalcastRuntime *runtime, LocalcastSamp
bool localcast_runtime_push_phase_claim(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_push_event_claim(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_push_render_packet(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_push_move_evidence(LocalcastRuntime *runtime, LocalcastSampleHandle sample);
bool localcast_runtime_status(const LocalcastRuntime *runtime, LocalcastRuntimeStatus *out_status);
size_t localcast_runtime_len(const LocalcastRuntime *runtime);
size_t localcast_runtime_view_len(const LocalcastRuntime *runtime, uint32_t sample_kind);
@ -210,6 +254,15 @@ bool localcast_producer_push_render_packet(
LocalcastSampleHandle *out_sample
);
bool localcast_producer_push_move_evidence_buffer(
LocalcastProducer *producer,
LocalcastRuntime *runtime,
uint64_t timestamp_ns,
uint64_t arrival_ns,
const LocalcastMoveEvidenceBufferDescriptor *descriptor,
LocalcastSampleHandle *out_sample
);
#ifdef __cplusplus
}
#endif

View file

@ -17,6 +17,7 @@ pub enum SampleKind {
PhaseClaim,
EventClaim,
RenderPacket,
MoveEvidence,
}
impl SampleKind {
@ -31,6 +32,7 @@ impl SampleKind {
6 => Some(Self::PhaseClaim),
7 => Some(Self::EventClaim),
8 => Some(Self::RenderPacket),
9 => Some(Self::MoveEvidence),
_ => None,
}
}
@ -46,6 +48,7 @@ impl SampleKind {
Self::PhaseClaim => 6,
Self::EventClaim => 7,
Self::RenderPacket => 8,
Self::MoveEvidence => 9,
}
}
}
@ -229,6 +232,53 @@ pub struct LocalcastRenderPoint {
pub confidence: f32,
}
#[repr(C)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LocalcastMoveEvidenceKind {
OpticalMarker = 1,
ControllerState = 2,
}
#[repr(C)]
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct LocalcastMoveEvidenceSample {
pub witness_id_hash: u64,
pub controller_id_hash: u64,
pub source_timestamp_ns: u64,
pub arrival_ns: u64,
pub sequence: u64,
pub evidence_kind: u32,
pub flags: u32,
pub image_x: f32,
pub image_y: f32,
pub radius_px: f32,
pub confidence: f32,
pub accel_x: f32,
pub accel_y: f32,
pub accel_z: f32,
pub gyro_x: f32,
pub gyro_y: f32,
pub gyro_z: f32,
pub trigger: f32,
pub buttons_mask: u32,
pub reserved: u32,
pub battery01: f32,
pub reserved1: u32,
pub reserved2: u32,
}
#[repr(C)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct LocalcastMoveEvidenceBufferDescriptor {
pub sample_buffer_handle: u64,
pub sample_count: u32,
pub sample_stride_bytes: u32,
pub source_time_min_ns: u64,
pub source_time_max_ns: u64,
pub calibration_hash: u64,
pub tracking_space_hash: u64,
}
pub struct LocalcastReservoir {
inner: RollingReservoir<LocalcastSampleHandle>,
}
@ -258,6 +308,7 @@ pub struct LocalcastRuntimeStatus {
pub phase_claim_count: usize,
pub event_claim_count: usize,
pub render_packet_count: usize,
pub move_evidence_count: usize,
}
impl Default for LocalcastRuntimeStatus {
@ -275,6 +326,7 @@ impl Default for LocalcastRuntimeStatus {
phase_claim_count: 0,
event_claim_count: 0,
render_packet_count: 0,
move_evidence_count: 0,
}
}
}
@ -308,6 +360,7 @@ impl LocalcastRuntime {
phase_claim_count: self.view_len(SampleKind::PhaseClaim),
event_claim_count: self.view_len(SampleKind::EventClaim),
render_packet_count: self.view_len(SampleKind::RenderPacket),
move_evidence_count: self.view_len(SampleKind::MoveEvidence),
}
}
}
@ -623,6 +676,14 @@ pub extern "C" fn localcast_runtime_push_render_packet(
localcast_runtime_push_typed(ptr, SampleKind::RenderPacket, sample)
}
#[unsafe(no_mangle)]
pub extern "C" fn localcast_runtime_push_move_evidence(
ptr: *mut LocalcastRuntime,
sample: LocalcastSampleHandle,
) -> bool {
localcast_runtime_push_typed(ptr, SampleKind::MoveEvidence, sample)
}
#[unsafe(no_mangle)]
pub extern "C" fn localcast_runtime_status(
ptr: *const LocalcastRuntime,
@ -860,6 +921,46 @@ pub extern "C" fn localcast_producer_push_render_packet(
)
}
#[unsafe(no_mangle)]
pub extern "C" fn localcast_producer_push_move_evidence_buffer(
producer_ptr: *mut LocalcastProducer,
runtime_ptr: *mut LocalcastRuntime,
timestamp_ns: u64,
arrival_ns: u64,
descriptor: *const LocalcastMoveEvidenceBufferDescriptor,
out_sample: *mut LocalcastSampleHandle,
) -> bool {
let Some(producer) = (unsafe { producer_ptr.as_mut() }) else {
return false;
};
if producer.kind != SampleKind::MoveEvidence {
return false;
}
let Some(descriptor_ref) = (unsafe { descriptor.as_ref() }) else {
return false;
};
if descriptor_ref.sample_buffer_handle == 0 {
return false;
}
if descriptor_ref.sample_count == 0 {
return false;
}
if descriptor_ref.sample_stride_bytes as usize
!= std::mem::size_of::<LocalcastMoveEvidenceSample>()
{
return false;
}
let payload_handle = descriptor as u64;
localcast_producer_push(
producer_ptr,
runtime_ptr,
timestamp_ns,
arrival_ns,
payload_handle,
out_sample,
)
}
fn localcast_runtime_push_typed(
ptr: *mut LocalcastRuntime,
kind: SampleKind,
@ -943,6 +1044,51 @@ mod tests {
}
}
fn move_evidence_descriptor(
sample_buffer_handle: u64,
) -> LocalcastMoveEvidenceBufferDescriptor {
LocalcastMoveEvidenceBufferDescriptor {
sample_buffer_handle,
sample_count: 2,
sample_stride_bytes: std::mem::size_of::<LocalcastMoveEvidenceSample>() as u32,
source_time_min_ns: 1_000,
source_time_max_ns: 1_120,
calibration_hash: 0xCA11_BA7E,
tracking_space_hash: 0x57A6E,
}
}
fn move_marker_sample(
witness_id_hash: u64,
controller_id_hash: u64,
) -> LocalcastMoveEvidenceSample {
LocalcastMoveEvidenceSample {
witness_id_hash,
controller_id_hash,
source_timestamp_ns: 1_000,
arrival_ns: 1_010,
sequence: 7,
evidence_kind: LocalcastMoveEvidenceKind::OpticalMarker as u32,
flags: 0,
image_x: 123.5,
image_y: 87.25,
radius_px: 14.0,
confidence: 0.92,
accel_x: 0.0,
accel_y: 0.0,
accel_z: 0.0,
gyro_x: 0.0,
gyro_y: 0.0,
gyro_z: 0.0,
trigger: 0.0,
buttons_mask: 0,
reserved: 0,
battery01: f32::NAN,
reserved1: 0,
reserved2: 0,
}
}
fn render_point(stable_key_hash: u64) -> LocalcastRenderPoint {
LocalcastRenderPoint {
stable_key_hash,
@ -1585,6 +1731,125 @@ mod tests {
localcast_runtime_destroy(runtime);
}
#[test]
fn move_evidence_layout_is_fixed_for_compute_upload() {
let sample = move_marker_sample(11, 22);
assert_eq!(std::mem::size_of::<LocalcastMoveEvidenceSample>(), 112);
assert_eq!(
std::mem::size_of::<LocalcastMoveEvidenceBufferDescriptor>(),
48
);
assert_eq!(11, sample.witness_id_hash);
assert_eq!(22, sample.controller_id_hash);
assert_eq!(
LocalcastMoveEvidenceKind::OpticalMarker as u32,
sample.evidence_kind
);
assert_eq!(123.5, sample.image_x);
assert_eq!(14.0, sample.radius_px);
}
#[test]
fn move_evidence_descriptor_enters_native_runtime_window() {
let runtime = localcast_runtime_create(DEFAULT_RESERVOIR_NS);
let producer = localcast_producer_create(SampleKind::MoveEvidence.id(), 101, 3);
let descriptor = move_evidence_descriptor(9_000);
let mut sample = handle(0, 0, 0, 0, 0);
assert!(localcast_producer_push_move_evidence_buffer(
producer,
runtime,
1_200,
1_220,
&descriptor,
&mut sample,
));
let mut status = LocalcastRuntimeStatus::default();
assert!(localcast_runtime_status(runtime, &mut status));
assert_eq!(1, status.total_sample_count);
assert_eq!(1, status.move_evidence_count);
assert_eq!(0, status.camera_feature_count);
assert_eq!(
(&descriptor as *const LocalcastMoveEvidenceBufferDescriptor) as u64,
sample.payload_handle
);
assert_eq!(
1,
localcast_runtime_view_len(runtime, SampleKind::MoveEvidence.id())
);
localcast_producer_destroy(producer);
localcast_runtime_destroy(runtime);
}
#[test]
fn move_evidence_descriptor_push_rejects_wrong_kind_and_bad_buffers() {
let runtime = localcast_runtime_create(DEFAULT_RESERVOIR_NS);
let render_producer = localcast_producer_create(SampleKind::RenderPacket.id(), 88, 12);
let move_producer = localcast_producer_create(SampleKind::MoveEvidence.id(), 89, 0);
let good = move_evidence_descriptor(9_000);
let zero_handle = move_evidence_descriptor(0);
let zero_count = LocalcastMoveEvidenceBufferDescriptor {
sample_count: 0,
..good
};
let bad_stride = LocalcastMoveEvidenceBufferDescriptor {
sample_stride_bytes: 4,
..good
};
assert!(!localcast_producer_push_move_evidence_buffer(
render_producer,
runtime,
2_000,
2_010,
&good,
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
std::ptr::null(),
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
&zero_handle,
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
&zero_count,
std::ptr::null_mut(),
));
assert!(!localcast_producer_push_move_evidence_buffer(
move_producer,
runtime,
2_000,
2_010,
&bad_stride,
std::ptr::null_mut(),
));
assert_eq!(
0,
localcast_runtime_view_len(runtime, SampleKind::MoveEvidence.id())
);
localcast_producer_destroy(render_producer);
localcast_producer_destroy(move_producer);
localcast_runtime_destroy(runtime);
}
#[test]
fn runtime_spine_rejects_nulls() {
let sample = handle(1, 2, 3, 4, 5);

View file

@ -22,13 +22,16 @@ flowchart TD
B["mic/loopback drivers"] --> R
C["Leap timing/IR driver"] --> R
D["network feed producers"] --> R
M["Starfire/Nightwing Move tracking"] --> R
R --> N["native reservoir handles"]
N --> E["Fensalir GPU fusion + UI"]
N --> F["Faust/native DSP"]
E --> G["Spout2/program video"]
E --> G["Mimir program video"]
F --> H["program stems + spatial bed"]
G --> I["OBS"]
H --> I
G --> J["Yggdrasil site publisher"]
H --> J
G -. "compatibility" .-> K["OBS adapter"]
H -. "compatibility" .-> K
```
Ownership:
@ -41,11 +44,22 @@ Ownership:
becoming independent clock authorities.
- `Mimir.Runtime` owns app-level stream buffers and synchronization.
- Native capture workers own device reads.
- Move tracking observations are stream samples, not program controls. Muninn
daemons on Starfire and Nightwing publish source-local optical marker
candidates plus controller/IMU/button state for USB-attached Moves. Odin owns
discovery/schema projection for those streams. Mimir owns calibration,
association, triangulation, IMU fusion, prediction, and the resolved
Fensalir-facing wand pose stream.
- Mimir owns program composition, source subscription policy, preview/control
state, stats, and publication intent.
- Fensalir owns dense visual fusion, material/brush/splat reconciliation,
D3D12 interop, runtime UI, and Spout2 publication.
D3D12 interop, runtime UI lowering, and local program texture output.
- Faust/native DSP owns hot audio alignment, suppression, separation,
spatialization, and stem generation.
- OBS owns broadcast controls.
- The Yggdrasil-facing publisher daemon consumes Mimir program output and
publishes it to the site without owning composition.
- OBS is a temporary compatibility sink, not a composition or broadcast
authority.
Invariant: the live window is bounded, in memory, and has one timing authority.
No private history outlives the rolling buffer.
@ -54,7 +68,7 @@ No private history outlives the rolling buffer.
[[docs/viable-stream-app|Viable Stream App]] defines the near-term app target.
Fensalir hosts the running Mimir app, keeps the default five-second runtime in
memory, exposes debug/settings/output controls, and emits synchronized OBS
memory, exposes debug/settings/output controls, and emits synchronized Mimir
program video plus separately controllable audio stems.
`Mimir.Runtime` currently provides:
@ -67,6 +81,7 @@ program video plus separately controllable audio stems.
- `MimirProcessStreamSource`;
- `MimirFrameEventProcessStreamSource`;
- `MimirVideoFrameDescriptor`;
- `MimirTrackingObservation`;
- `MimirAudioSynchronizationAnalyzer`;
- `IMimirVideoCaptureDriver`;
- `MimirVideoCaptureDriverSource`.
@ -79,6 +94,221 @@ the rolling buffers while the direct ABI driver is being cut. It does not carry
pixels and does not own the final six-camera hot path. Multi-camera probes are
one process with declared accepted source ids, not one process per camera.
## Move Tracking
Starfire and Nightwing both have PS Move controllers directly attached over
USB. Muninn represents what each local body can honestly observe:
`muninn.move_marker_candidate.v1` for glowing orb candidates from cameras and
`muninn.move_controller_state.v1` for controller/IMU/button/battery state from
USB-attached Moves. Mimir drinks those feeds into `MimirStreamKind.Tracking`
buffers, admits them into the native `move_evidence` reservoir view as
compute-upload buffers, and publishes resolved `mimir.move_controller_pose.v1`
documents for Fensalir interaction once calibration and fusion have earned a
pose.
Authority:
- Starfire-local Move evidence is published by Muninn/Mimir local producers as
typed observation streams.
- Nightwing Move evidence is emitted by Muninn on Nightwing over the typed
observation path.
- Structured PS Move light pulses are Muninn output commands. Mimir requests
them by sending `muninn.move_light_command.v1` over CultNet/CultMesh to the
Muninn daemon on the host with the USB-attached Move; Mimir does not write
remote HID directly except through temporary smoke scripts used to prove
hardware behavior before Muninn is available on that host.
- Odin discovers and projects the stream/schema surface for operator and agent
access.
- Mimir owns subscription, rolling retention, native Move evidence buffer
layout, clock alignment, calibration, controller association, triangulation,
IMU fusion, prediction, and resolved wand pose publication. OBS and
dashboard summaries do not own this state.
- Fensalir consumes Mimir's resolved Move controller poses as interactive input
for environments; it does not synthesize controller truth from raw Muninn
evidence.
Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj
-- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing
Move observation into two tracking buffers.
Calibration preflight proof: `dotnet run --project
.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj --
--move-calibration-protocol-smoke --output
artifacts\move-calibration\protocol.cc` writes
`mimir.move_calibration_protocol.v1` for
`mimir-move-calibration-starfire-nightwing-v1`. The protocol requires
`muninn:starfire:move-evidence`, `muninn:nightwing:move-evidence`, and
`mimir:starfire:move-controller-poses`; optionally drinks Quest headset and
left/right controller poses over USB as an external VR reference; and requires
Mimir to derive rig calibration, IMU calibration, controller identity mapping,
and a calibration receipt before orientation can stop being
`orientation:imu-unresolved`.
Quest USB access belongs to Muninn, not Mimir. Starfire Muninn runs with
`--quest-adb --quest-serial 1WMHHB68PG1515` and publishes
`muninn.quest_access.v1`, including stream ids for Quest input, Quest poses, and
warp-corrected video input from Brokkr/Unity. On 2026-06-12, `adb devices -l`
reported authorized Quest 2 device `1WMHHB68PG1515` with product/device
`hollywood`. This proves Starfire can query the headset over USB; it does not
provide Quest headset/controller poses until a Quest/OpenXR witness bridge
publishes those pose samples through Muninn.
The old Nightwing bring-up path is archived:
`scripts/start-nightwing-move-tracking.ps1`, `Mimir.EveSensorReceiver`,
`Mimir.VerseRecorder`, and `nightwing_typed_witness_publisher.py` now fail
closed. Runtime structured light commands belong to Muninn. Eye/Move
observations must enter through typed CultMesh stream frames or
Odin-discovered CultMesh documents; they are optical witness evidence for the
later pose/fusion owner, not the final 6DoF pose authority.
Starfire-local Move illumination smoke uses `scripts/start-starfire-move-light.ps1`.
That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01
output collection and refreshes the PS Move LED report for hardware proof when
Muninn is not available on Starfire yet.
Starfire should have Muninn for source-local Move HID control and local sensor
publication. Some older Starfire audio/video paths still enter Mimir through
local Mimir producers or diagnostic process bridges while direct native drivers
mature. Same-host Move evidence and pose frames are CultMesh shared-memory byte
rings. Remote media is different: the current Raven media body bridge uses
CultNet/CultMesh documents over the mesh and lowers to local UDP for OBS
compatibility, so it is not same-machine zero-copy into Mimir/Fensalir yet.
Odin's Muninn organ owns the Move optical extraction stage for sensor stream
exposure. Its Rust crate lives in `E:\Projects\Odin\crates\muninn-move-tracker`
and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes
those Muninn candidate streams into tracking buffers; it does not own raw
optical extraction. Mimir is now the explicit owner for stereo triangulation,
camera calibration, controller association, IMU fusion, prediction, and final
6DoF wand pose.
Muninn's `muninn:nightwing:move-evidence:<sequence>` shared-memory frame now
uses the canonical `muninn.move_marker_candidate.v1` document shape for optical
markers instead of a placeholder empty slice. A daemon unit proves that a bright
Y8 frame run through `muninn-move-tracker` serializes a non-empty marker
candidate into the Mimir-compatible evidence frame. The daemon now has a
source-local Y8 extraction/publish seam plus a first `serve` camera producer:
`--move-marker-camera <camera-id>=<device-path>` polls a Unix V4L2 YUYV frame,
converts it to compact Y8, and feeds that same seam. Nightwing hardware now
runs as one Muninn provider with two private per-eye PSMoveAPI subprocesses.
The parent alone owns lights, identity, commands, provider advertisement,
persistence, and the aggregate evidence stream; children exchange hue state
and camera-tagged observations over bounded MessagePack pipes.
`muninn.move_tracker_health.v1` exposes each worker's camera backend,
calibration, update/observation counters, RGB range, and expected-hue pixel
evidence. Live exposure is 0.3 for `nightwing-eye-0` and 0.1 for
`nightwing-eye-1`. Both eyes calibrate all four Moves and produce observations.
Muninn's dedicated evidence aggregator now owns sequence and transport outside
the 15-second CultCache telemetry loop. Cached controller state is bundled into
fresh optical frames but cannot create frames by itself, and a deadline clock
caps publication at the configured camera cadence. A 2026-07-13 Mimir field
window received 622 unique frames in 15 seconds, 2,647 observations, and 1,384
same-ID cross-camera correspondences while seeing all four stable Move IDs from
both eyes. The same parent PID survived the run; transport health reported
5,302 produced frames, 5,301 handoffs, 771 sends, and zero local-ring
admissions. Full stereo calibration is still not implied: intrinsics and
spatially diverse correspondences must earn the camera matrices.
`MimirMoveStereoCalibrationAssessment` is the typed observability gate before
any fit. It accepts only same aggregate-frame, same-Move correspondences and
reports synchronized pair count, Move diversity, per-Eye 4x3 grid coverage,
radius range, and missing physical inputs. The corrected live capture produced
130 same-frame pairs with zero frame skew, but only 4/12 Eye 0 cells and 2/12
Eye 1 cells; intrinsics and measured orb radius are also absent, so promotion is
correctly false.
`MimirMoveProofSurface` is the first Fensalir-visible proof surface for this
chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream
frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only
`AquariumSplineFrame` probe with the explicit
`muninn:nightwing:move-evidence:<sequence> -> mimir:starfire:move-evidence:<sequence> -> mimir:starfire:move-pose:<sequence> -> fensalir:starfire:presented-frame:<sequence>`
chain. It does not decide pose. Single-ray fallback remains a visible
non-final verdict until calibrated multi-camera optical evidence earns full
pose.
`MimirMoveProofPipeline` is now the reusable commit primitive for that chain:
from one Muninn CultMesh shared-memory frame it performs native admission,
Mimir fusion, pose-frame creation, proof-surface creation, and Fensalir probe
lowering. Live Nightwing/Starfire wiring should call this owner path rather
than preserving separate manual actions and programmatic proof paths.
`MimirRuntime` now owns the actual Fensalir presentation attachment for the
proof surface: `PublishMoveProofSurface` stores the latest Mimir-owned
`mimir.move_proof_surface.v1`, and `CreateFrame` merges its observer-only
`move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside the
audio spectrum preview. Fensalir renders this frame; it still does not decide
proof.
`MimirMoveProofRuntimeDriver` is the runtime bridge from a Muninn CultMesh
shared-memory evidence ring into that attachment point. It derives the
Mimir evidence, Mimir pose, and Fensalir frame ids from the actual Muninn frame
suffix, calls `MimirMoveProofPipeline`, suppresses duplicate frame admission,
and lets `MimirRuntime.Update` publish the resulting proof surface. This proves
the runtime-owned path for a ring frame; real Nightwing hardware and calibrated
camera witnesses are still the missing field proof.
`MimirMoveProofRuntimeConfiguration` is the typed selection and validation
surface for that bridge. An enabled source must name the Muninn evidence stream,
native reservoir path, Mimir evidence/pose frame prefixes, Fensalir
presented-frame prefix, fusion authority, consumer contract, and at least two
calibrated camera witnesses. It may create a runtime driver only when the
supplied CultMesh ring stream id matches the configured evidence stream.
`MimirRuntime` now owns activation for configured proof sources: on scene-ready
it asks an `IMimirMoveProofEvidenceRingProvider` for the evidence ring, opens
the configured native reservoir, registers the runtime driver, retains the
driver resources, and exposes an activation status. The default provider fails
explicitly because the current C# CultMesh ring is still in-process only;
`Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an in-process
provider to prove the activation path without claiming live Nightwing field
evidence.
`mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine
surface for configured Move proof sources. It reports the configured evidence
stream, provider kind, active/driver state, native reservoir path, calibrated
camera ids, latest same-stream proof ids, latest verdict, and the inactive
diagnostic when the default runtime lacks a live cross-process CultMesh ring
opener. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke`
proves both the active injected-ring document and the unavailable default
document; this is readiness/proof observation, not field evidence.
`mimir.move_proof_evidence_frame_snapshot.v1` is a one-copy, explicitly
non-final fallback for captured Muninn evidence frames while C# CultMesh lacks
a cross-process shared-memory opener. When `EvidenceSnapshotPath` is
configured, `MimirConfiguredMoveProofEvidenceRingProvider` validates snapshot
metadata against the encoded Muninn frame payload, publishes the payload into
an owned in-process ring, and then delegates to the same runtime proof driver
and pipeline. `Mimir.BufferSmoke --move-proof-runtime-snapshot-smoke` proves
the same named proof spine through that file boundary. This can support field
capture/replay, but live proof still requires the real Nightwing/Starfire ring
or CultMesh page producer.
Odin/Muninn can now write that artifact from the producer side with
`--move-evidence-snapshot <path>`. The snapshot is written by the same Muninn
Move evidence publisher that owns controller/marker frames, after the frame
payload is accepted by the stream ring. Its unit coverage decodes the
`mimir.move_proof_evidence_frame_snapshot.v1` tuple and re-decodes the embedded
Muninn payload, proving the artifact is shaped for Mimir's snapshot provider.
This is still one-copy field capture/replay; it does not replace the live ring
or a CultCache page body producer.
`MimirMoveProofDevSurface` is a dev-only bootstrap gated by
`MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof
surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so
`Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App`
headless, capture the actual Fensalir-presented PNG, and pixel-check that the
named proof probe is visible. This is a renderer/probe proof, not live hardware
evidence.
## Program Composition
[[docs/mimir-program-composition|Mimir Program Composition]] is the live
authority map for stream-program output. Muninn runs on Starfire, Nightwing, and
Raven-class capture hosts to publish local stream capabilities and selected
media bodies. Mimir consumes only the streams it needs for calibration and
composition, commits one typed scene graph,
and publishes program output locally plus through a Yggdrasil-facing site
publisher.
The first typed program contracts live in `src/Mimir.Runtime/Synchronization`:
- `mimir.program_scene.v1`
- `mimir.program_output.v1`
- `mimir.eve_operator_surface.v1`
OBS scene JSON is import/mirror evidence for the initial crop/key/transform
layout. It is not the editor of record after Mimir owns the scene graph.
The current imported local scene state lives at
`state/mimir-program-composition.cc`.
## OBS Bridge Utility
```mermaid
@ -97,6 +327,122 @@ flowchart TD
The bridge is useful because it is inspectable and already speaks OBS. It does
not become the synchronized program authority.
## CultMesh Media Bridge
The live Raven-to-Starfire program path is Muninn's typed CultNet media body:
```mermaid
flowchart TD
A["Raven D3D11 desktop capture"] --> B["Long-lived NVENC encoder"]
C["Raven WASAPI loopback"] --> D["AAC-LC / ADTS encoder"]
B --> E["V4 8+4 block FEC and realtime video lane"]
D --> F["Fixed 864-byte 4+2 audio FEC lane"]
E --> G["Starfire OBS Muninn receiver udp/5200"]
F --> G
G --> H["Frame assembly / FEC / IDR feedback"]
G --> I["120 ms audio reorder / concealment"]
H --> J["OBS-local video decoder socket"]
I --> K["OBS-local audio decoder socket"]
```
Owner: Muninn owns Raven capture children, access-unit/audio packetization,
CultNet delivery, repair material, and encoder feedback response. The Mimir OBS
plugin owns receipt admission, frame/FEC assembly, audio continuity, and local
OBS lowering. OBS owns decode and composition. Odin/CultMesh discovery names
the provider surface; neither discovery nor the compatibility decoder sockets
own hot media truth.
Invariant: video packets are useful only before their assembly deadline.
Canonical video and V4 parity use unreliable `realtime`; 8+4 block FEC,
queue-admitted selective repair, and IDR recovery own loss. Canonical AAC audio
and fixed 4+2 parity also use
`realtime`; reorder and bounded concealment own playout continuity. Completed
video frames are tombstoned so late shards cannot
become a second assembly owner. Raven uses direct LAN only and starts through a
hidden interactive-token scheduled task; no WireGuard hop or foreground
terminal participates.
The older `Mimir.CultMeshMedia` relay description below is retained as a
non-authoritative historical/body-bridge map. It does not own the active Muninn
OBS feed.
```mermaid
flowchart TD
A["Raven FFmpeg desktop + WASAPI loopback mux"] --> B["Mimir.CultMeshMedia send"]
B --> C["CultNet reliable UDP media-frame documents"]
C --> D["Yggdrasil CultMesh relay discovered through Odin/CultMesh"]
D --> E["Starfire Mimir.CultMeshMedia recv"]
E --> F["local MPEG-TS UDP udp://127.0.0.1:5200"]
F --> G["OBS Raven Monitor + Realtek"]
```
Owner: `src/Mimir.CultMeshMedia` owns the live Raven-to-Starfire media body
bridge over CultMesh/CultNet. FFmpeg still owns Raven capture and encoding.
Mimir owns composition. OBS may consume the Starfire-local MPEG-TS endpoint as a
compatibility sink while Mimir/Fensalir program output and site publication
mature.
Invariant: Raven media body state is live and bounded. Sender writes rolling
`mimir.cultmesh_media_frame` slot documents rather than an unbounded durable
video archive. Yggdrasil relays CultNet reliable UDP behind the logical
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av` route discovered
through Odin/CultMesh. Starfire lowers the subscribed stream to local UDP for
compatibility sinks because OBS is not a CultMesh runtime.
Source audit correction resolved: `Mimir.CultMeshMedia` now uses explicit
CultLib RUDP helpers for the sender/receiver and a relay-owned RUDP
socket/session loop for multi-peer forwarding. The older
`CultMesh.StartNodeAsync` and `CultMesh.ConnectClient` path no longer owns this
media lane. The relay still persists received puts to CultCache as evidence and
for inspection; live network transit is the RUDP schema-document path.
Current deployment: Yggdrasil runs the relay from
`/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia` with cache
`/var/lib/gamecult/mimir/cultmesh-media.cc` and log
`/var/log/gamecult/mimir-cultmesh-media.log`. Senders and receivers target
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`; concrete RUDP
bootstrap lives in the CultMesh resolver environment, not in daemon launch
arguments. Starfire writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS
source `Raven Monitor + Realtek` points at that local UDP endpoint.
`scripts/start-raven-cultmesh-av-sender.ps1` is the Mimir-owned CultMesh bridge
bootstrap for Raven media-body transit. It is not the actual Muninn OBS feed
owner. The real Raven OBS/SRT feed owner is Odin's Muninn:
`E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1` drives the
existing `GameCult-Muninn-Activate` hidden task and `muninn.exe activate`
body on Raven, and `scripts/start-raven-muninn-obs-feed.ps1` in this repo is
the thin local wrapper for that actuator. Keep `-LocalBootstrap` on the
CultMesh sender only as a staging/bootstrap edge when the separate body-bridge
lane needs direct local proof.
## Daemon Health And Idunn
Odin owns the accepted Verse/service map. Idunn owns continuity after a daemon
is known: boot recovery, crash recovery, deployment freshness, stale-health
watching, restart/deploy intent, and operator escalation. Mimir does not run a
private lifecycle supervisor for these surfaces.
CultLib's current direction is RUDP everywhere for typed CultNet/CultMesh
documents across runtimes. That means health, provider advertisements, command
boundaries, transport profiles, and selected program/media document lanes should
default to `cultnet.transport.rudp.v0`. Product/debug render surfaces are
lowerings, debug tools, or migration debt; they must not own daemon truth once
an RUDP contract exists.
Mimir's current daemon-health publishers are local RUDP witnesses:
Invariant: health publication is not ownership. The daemon reports what it can
honestly observe about itself; Idunn decides keepalive action through the
shared typed record path.
Source audit cut: `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the
Nightwing typed witness publisher, and `start-nightwing-move-tracking.ps1` are
archived. The old Nightwing/EVE witness lane no longer opens a renderer/client
subscription surface. Witness publication and recording must use typed
CultMesh/CultNet documents or CultMesh stream frames discovered through Odin.
The media-frame bridge is already on that side of the line.
## Audio Field
The six-microphone path is separate from the bridge. Scarlett speaker loopback

View file

@ -21,7 +21,23 @@ Get-Content .\state\evidence.jsonl -Tail 8
- Mimir's Face doctrine lives in [[docs/mimir-face|Mimir Face]].
- VoidBot persona/state live under `.voidbot/voice/` and `.voidbot/state/`.
Commit `.voidbot` dirt in a separate state commit whenever it appears.
- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN ingest.
- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN compatibility ingest.
- V1 now also has an explicit CultMesh media bridge for Raven program feeds:
`src/Mimir.CultMeshMedia` sends rolling `mimir.cultmesh_media_frame`
documents over CultNet reliable UDP through Yggdrasil and lowers them on
Starfire to local MPEG-TS UDP for OBS. The bridge now uses explicit CultLib
RUDP helpers/session plumbing; the old
`CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` path no longer owns this
media lane.
- CultLib has moved the daemon-swarm direction to CultNet over RUDP across
runtimes. Mimir's typed daemon truth should default to
`cultnet.transport.rudp.v0`; TCP/HTTP/WebSocket are client lowerings,
compatibility probes, or migration debt unless a specific runtime has not
earned RUDP yet.
- Mimir's old Eve dashboard broker is archived and no longer publishes Idunn
health, `/eve/deck`, `/eve/dashboard`, or WebSocket commands. The browser
reference is a client lowering, not service truth. Dashboard state must return
as typed CultMesh/Eve documents through Odin before deployment resumes.
- The live stream app is C# plus Fensalir: `Mimir.slnx` contains
`src/Mimir.App` and `src/Mimir.Runtime`.
- `src/Mimir.App` hosts Fensalir as the windowing/rendering/D3D12 bridge.
@ -37,6 +53,19 @@ Get-Content .\state\evidence.jsonl -Tail 8
dictionary, low-level implementation notes, and sample code sketches for the
next decoder/DSP/native-ring/Fensalir implementation passes. Start with
[[research/perfect-machine-study-2026-05-23/reading-guide|Reading Guide]].
- `docs/mimir-program-composition.md` is the current program-output authority
map: Muninn observes on Starfire/Nightwing/Raven-class hosts, Mimir consumes
selected streams and owns composition, Eve lowers controls/previews/stats,
Yggdrasil publishes to the site, and OBS is compatibility-only.
- `src/Mimir.Runtime/Synchronization` now has typed program contracts for
`mimir.program_scene.v1`, `mimir.program_output.v1`, and
`mimir.eve_operator_surface.v1`; `Mimir.BufferSmoke
--perfect-machine-contract-smoke` writes them into CultCache.
- `Mimir.BufferSmoke --import-obs-program-scene --output
state/mimir-program-composition.cc` imports the current OBS scene JSON into
Mimir-owned typed state. The latest imported scene has eight layers, six
visible, two cropped, one chroma-keyed, and Raven mapped as
`muninn:raven:monitor:primary`.
- Local six-camera ingest should use direct driver adapters. Process-backed
sources are bridge/network edges only.
- Leap stereo IR is the first timing-camera candidate for direct ingest.
@ -45,10 +74,20 @@ Get-Content .\state\evidence.jsonl -Tail 8
- Fensalir owns production GPU fusion, UI, and Spout2 publication.
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
synchronized stems.
- OBS receives final program surfaces; it does not own synchronization.
- OBS compatibility adapters may receive final program surfaces; they do not
own synchronization, composition, preview/control, or publication.
## Current Pressure
- Nightwing Move tracking is one Muninn provider with two private per-eye
PSMoveAPI worker subprocesses. Eye 0 uses exposure 0.3; Eye 1 uses 0.1.
Both workers calibrate four Moves and advance continuously. Mimir has now
proved all four stable IDs on both eyes in the same frames. The latest
15-second receipt contains 16 observations and eight pair records but only
two unique aggregate frame IDs. Next measure parent publication and RUDP send
counters directly, fix cadence, then collect diverse correspondences for
camera calibration.
- Implement the first concrete Leap direct capture driver and measure cadence.
- Add remaining camera drivers through `IMimirVideoCaptureDriver`.
- Add native audio capture workers for mic, loopback, and network audio feeds.
@ -57,8 +96,24 @@ Get-Content .\state\evidence.jsonl -Tail 8
variable-rate resampling.
- Bind Fensalir UI to stream health, buffer depth, timestamps, settings, and
output management.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until native program output
replaces them.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until native Mimir program
output, Eve operation, and Yggdrasil site publication replace them.
- Yggdrasil currently runs the CultMesh media relay from
`/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia`; senders and
receivers target `cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`
and let the CultMesh resolver choose the concrete RUDP route. Starfire writes
`raven-primary-av` to `udp://127.0.0.1:5200`. That is the
`Mimir.CultMeshMedia` body bridge, not the OBS plugin feed owner.
- The actual Raven OBS/SRT feed owner is Odin's Muninn on Raven. Use
`scripts/start-raven-muninn-obs-feed.ps1` in this repo as the thin wrapper
over `E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1`, which
drives Raven's real `GameCult-Muninn-Activate` hidden task and
`muninn.exe activate` path. `scripts/start-raven-cultmesh-av-sender.ps1`
remains the separate CultMesh bridge/bootstrap lane.
- Rebuild Eve dashboard only as a CultMesh/Odin surface publisher plus renderer
lowering. Provider advertisement, retained state, command boundary, transport
profile, and command documents must be typed CultMesh records; no TCP, HTTP,
or WebSocket surface should become daemon or transport truth.
- Kiyo Pro has two UVC extension units. Moving it to a motherboard USB3 port
changed the descriptor path to `root_hub30` / `bcdUSB=0x0320` and exposed
720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports
@ -143,6 +198,41 @@ Get-Content .\state\evidence.jsonl -Tail 8
## Immediate Re-entry Instruction
The 2026-07-14 live two-wand session at
`artifacts/move-calibration/live-two-wand-20260714-004210.*` passed collection
coverage with 1,168 admitted same-frame pairs. The new Mimir-owned sphere
stereo fitter recovers synthetic 505/535 px focal lengths within 0.43 px and a
34 cm baseline within 0.22 mm, with 0.34/0.63 px held-out median/P95 error.
The live artifact remains rejected: after robust radius admission its consensus
fits at 0.51/2.08 px, but only 53.8 percent of held-out pairs agree and Eye 1's
focal estimate leaves the physical range. PSMoveAPI age is currently projected
as marker confidence by Muninn, so fresh wrong-color matches arrive near 0.996
confidence. Repair source-local observation quality/continuity and calibrate
per-Eye blob-radius response before asking the operator for another sweep. Do
not promote the nominal 22.5 mm diagnostic fit.
Odin `d0eded6` is deployed on Nightwing as binary SHA-256
`a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39`.
Each Eye worker now rejects stale, sub-2 px, out-of-bounds, and discontinuous
PSMoveAPI positions before aggregation; tracker health publishes rejection
counters, and candidate score is explicitly uncalibrated `0.5` instead of age
masquerading as confidence. A post-deploy 10-second Mimir window received 555
frames, 2,246 observations, and 443 stereo pairs. Radius P01 rose from about
0.707 px to 3.13/3.64 px and confidence P50 is now 0.5 on both Eyes. Eye 1 saw
only two plausible stationary IDs, which is honest and means another sweep is
needed after Mimir models per-Eye blob-radius response.
The active priority is Nightwing Move calibration. One public Muninn observer
owns two private Eye workers and one aggregate evidence stream. Odin commits
`39c839a`, `a340c33`, `8aefd1c`, and `f82ca7a` give daemon health a persistent
serviced RUDP session and move optical aggregation onto its own deadline-clocked
loop. The 2026-07-13 receipt at
`artifacts/move-calibration/muninn-clocked-aggregator.mpack` contains 622 frames,
2,647 observations, and 1,384 cross-camera correspondences in 15 seconds, with
all four Move IDs visible to both Eyes. Next collect spatially diverse samples,
calibrate intrinsics/extrinsics, and publish residual-bearing typed calibration.
Radius is a weighted range cue, not a substitute for camera calibration.
The corrected two-hour study pass ran from
`2026-05-23T23:54:42.9041588+01:00` to at least
`2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement the

View file

@ -0,0 +1,158 @@
param(
[string]$NightwingHost = "nightwing",
[int]$Port = 8796,
[string]$StarfireHost = "192.168.1.66",
[string]$Path = "/eve/periwinkle",
[string]$EyeDevices = "/dev/video2,/dev/video3",
[string[]]$MoveLight = @(),
[string]$RecorderRoot = "C:\Users\Meta\Videos\Mimir\VerseCaptures",
[switch]$SkipRecorder,
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
$repo = Split-Path -Parent $PSScriptRoot
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
$runDir = Join-Path $repo "artifacts\runtime\nightwing-move-tracking-$runId"
$witnessUrl = "ws://$StarfireHost`:$Port$Path"
$subscribeUrl = "ws://127.0.0.1:$Port$Path/subscribe"
$remoteLog = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.log"
$remotePid = "~/.local/state/gamecult/nightwing-typed-witness.pid"
$remoteHeartbeat = "~/.local/state/gamecult/nightwing-typed-witness.heartbeat"
New-Item -ItemType Directory -Force -Path $runDir | Out-Null
New-Item -ItemType Directory -Force -Path $RecorderRoot | Out-Null
function Start-LoggedProcess {
param(
[string]$Name,
[string]$FilePath,
[string[]]$ArgumentList
)
$stdout = Join-Path $runDir "$Name.out.log"
$stderr = Join-Path $runDir "$Name.err.log"
if ($DryRun) {
Write-Host "DRY $Name`: $FilePath $($ArgumentList -join ' ')"
return $null
}
$process = Start-Process `
-FilePath $FilePath `
-ArgumentList $ArgumentList `
-WorkingDirectory $repo `
-WindowStyle Hidden `
-RedirectStandardOutput $stdout `
-RedirectStandardError $stderr `
-PassThru
Set-Content -Path (Join-Path $runDir "$Name.pid") -Value $process.Id
Write-Host "$Name pid=$($process.Id) stdout=$stdout stderr=$stderr"
return $process
}
function Wait-Health {
param([string]$Url)
if ($DryRun) {
Write-Host "DRY wait health: $Url"
return
}
$deadline = (Get-Date).AddSeconds(45)
do {
try {
$response = Invoke-WebRequest -UseBasicParsing -Uri $Url -TimeoutSec 3
if ($response.StatusCode -ge 200 -and $response.StatusCode -lt 300) {
Write-Host "health ok: $Url"
return
}
} catch {
Start-Sleep -Milliseconds 500
}
} while ((Get-Date) -lt $deadline)
throw "Timed out waiting for $Url"
}
Write-Host "Nightwing Move tracking run: $runDir"
Write-Host "Witness URL: $witnessUrl"
if (-not $DryRun) {
Get-CimInstance Win32_Process |
Where-Object {
($_.CommandLine -match "Mimir.EveSensorReceiver" -and $_.CommandLine -match "--port $Port") -or
($_.CommandLine -match "Mimir.VerseRecorder" -and $_.CommandLine -match "$Port$Path/subscribe")
} |
ForEach-Object {
try { Stop-Process -Id $_.ProcessId -Force -ErrorAction Stop } catch {}
}
}
$receiver = Start-LoggedProcess `
-Name "periwinkle-receiver" `
-FilePath "dotnet" `
-ArgumentList @(
"run",
"--project",
(Join-Path $repo "src\Mimir.EveSensorReceiver\Mimir.EveSensorReceiver.csproj"),
"--",
"--port", "$Port",
"--path", $Path,
"--subscribe-path", "$Path/subscribe",
"--source-id", "nightwing-witness",
"--type", "cultmesh-observation"
)
if ($receiver) {
Wait-Health -Url "http://127.0.0.1:$Port/health"
}
if (-not $SkipRecorder) {
Start-LoggedProcess `
-Name "verse-recorder" `
-FilePath "dotnet" `
-ArgumentList @(
"run",
"--project",
(Join-Path $repo "src\Mimir.VerseRecorder\Mimir.VerseRecorder.csproj"),
"--",
"--url", $subscribeUrl,
"--out-dir", $RecorderRoot
) | Out-Null
}
if ($DryRun) {
Write-Host "DRY Nightwing stage and start on $NightwingHost"
} else {
scp (Join-Path $repo "tools\nw_eye_cap.py") "${NightwingHost}:/tmp/nw_eye_cap.py" | Out-Null
scp (Join-Path $repo "tools\nw_move_hint.py") "${NightwingHost}:/tmp/nw_move_hint.py" | Out-Null
scp (Join-Path $repo "tools\nightwing_typed_witness_publisher.py") "${NightwingHost}:/tmp/nightwing_typed_witness_publisher.py" | Out-Null
$moveArgs = ($MoveLight | ForEach-Object { "--move-light '$($_ -replace "'", "'\''")'" }) -join " "
$remoteScript = @"
mkdir -p ~/.local/state/gamecult
pkill -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' || true
nohup python3 /tmp/nightwing_typed_witness_publisher.py --url '$witnessUrl' --track-eyes --track-builtin-camera --track-builtin-mic --interval 0.10 --heartbeat-path '$remoteHeartbeat' --eye-devices '$EyeDevices' --eye-window-seconds 0.08 --tracking-stride 6 $moveArgs > $remoteLog 2>&1 < /dev/null &
sleep 0.5
pgrep -n -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' > $remotePid
cat $remotePid
"@
$remoteB64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($remoteScript))
$nightwingPid = ssh $NightwingHost "printf '%s' '$remoteB64' | base64 -d | bash"
Set-Content -Path (Join-Path $runDir "nightwing-witness.pid") -Value ($nightwingPid | Select-Object -First 1)
Set-Content -Path (Join-Path $runDir "nightwing-witness.logpath.txt") -Value $remoteLog
Write-Host "nightwing-witness pid=$($nightwingPid | Select-Object -First 1) log=$remoteLog"
}
$manifest = [ordered]@{
kind = "mimir.nightwing_move_tracking_supervisor.v1"
runId = $runId
runDir = $runDir
witnessUrl = $witnessUrl
subscribeUrl = $subscribeUrl
heartbeatPath = $remoteHeartbeat
eyeDevices = $EyeDevices
directHidSmokeMoveLights = $MoveLight
startedAt = (Get-Date).ToString("o")
}
$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path (Join-Path $runDir "supervisor.json")
Write-Host "Supervisor manifest: $(Join-Path $runDir "supervisor.json")"

View file

@ -0,0 +1,89 @@
param(
[string]$RelayHost = "10.77.0.1",
[int]$RelayPort = 3075,
[string]$Stream = "raven-primary-av",
[string]$FfmpegPath = "ffmpeg",
[string]$VideoInput = "desktop",
[ValidateSet("WasapiLoopback", "DirectShow")]
[string]$AudioMode = "WasapiLoopback",
[string]$AudioInput = "Stereo Mix (Realtek(R) Audio)",
[ValidateSet("Console", "Multimedia", "Communications")]
[string]$LoopbackRole = "Console",
[int]$Width = 1920,
[int]$Height = 1080,
[int]$Framerate = 30,
[string]$VideoBitrate = "8000k",
[string]$AudioBitrate = "192k",
[string]$MediaToolPath = ""
)
$ErrorActionPreference = "Stop"
$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj"
$publishedTool = Join-Path $PSScriptRoot "..\artifacts\mimir-cultmesh-media-win-x64-selfcontained\Mimir.CultMeshMedia.exe"
$loopbackScript = Join-Path $PSScriptRoot "wasapi-loopback-capture.ps1"
if ([string]::IsNullOrWhiteSpace($MediaToolPath) -and (Test-Path $publishedTool)) {
$MediaToolPath = $publishedTool
}
$ffmpegArgs = @(
"-hide_banner",
"-loglevel", "warning",
"-f", "gdigrab",
"-framerate", $Framerate,
"-video_size", "${Width}x${Height}",
"-i", $VideoInput
)
if ($AudioMode -eq "WasapiLoopback") {
$ffmpegArgs += @(
"-f", "f32le",
"-ar", "48000",
"-ac", "2",
"-i", "pipe:0"
)
} else {
$ffmpegArgs += @(
"-f", "dshow",
"-i", "audio=$AudioInput"
)
}
$ffmpegArgs += @(
"-map", "0:v:0",
"-map", "1:a:0",
"-c:v", "h264_nvenc",
"-preset", "p3",
"-tune", "ll",
"-b:v", $VideoBitrate,
"-maxrate", $VideoBitrate,
"-bufsize", "16000k",
"-g", ($Framerate * 2),
"-pix_fmt", "yuv420p",
"-c:a", "aac",
"-b:a", $AudioBitrate,
"-ar", "48000",
"-ac", "2",
"-f", "mpegts",
"pipe:1"
)
$senderArgs = @("send", "--host", $RelayHost, "--port", $RelayPort, "--stream", $Stream, "--producer", "raven", "--chunk-bytes", "16384", "--slots", "96")
if ($AudioMode -eq "WasapiLoopback") {
if (-not (Test-Path $loopbackScript)) {
throw "WASAPI loopback script not found: $loopbackScript"
}
if ([string]::IsNullOrWhiteSpace($MediaToolPath)) {
& $loopbackScript -Output stdout -Role $LoopbackRole | & $FfmpegPath @ffmpegArgs | dotnet run --project $project -- @senderArgs
} else {
& $loopbackScript -Output stdout -Role $LoopbackRole | & $FfmpegPath @ffmpegArgs | & $MediaToolPath @senderArgs
}
} else {
if ([string]::IsNullOrWhiteSpace($MediaToolPath)) {
& $FfmpegPath @ffmpegArgs | dotnet run --project $project -- @senderArgs
} else {
& $FfmpegPath @ffmpegArgs | & $MediaToolPath @senderArgs
}
}

View file

@ -0,0 +1,10 @@
param(
[string]$RelayHost = "10.77.0.1",
[int]$RelayPort = 3075,
[string]$Stream = "raven-primary-av",
[string]$Udp = "127.0.0.1:5200"
)
$ErrorActionPreference = "Stop"
$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj"
dotnet run --project $project -- recv --host $RelayHost --port $RelayPort --stream $Stream --udp $Udp

View file

@ -0,0 +1,70 @@
param(
[string]$Rgb = "#35ff6c",
[int]$RefreshMs = 250,
[double]$HoldSeconds = 0,
[switch]$TurnOffOnExit,
[switch]$KeepExisting,
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
$repo = Split-Path -Parent $PSScriptRoot
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
$runDir = Join-Path $repo "artifacts\runtime\starfire-move-light-$runId"
$probe = Join-Path $repo "src\Mimir.PsMoveProbe\bin\Release\net10.0\Mimir.PsMoveProbe.exe"
$eventLog = Join-Path $runDir "move-light-events.jsonl"
New-Item -ItemType Directory -Force -Path $runDir | Out-Null
if (-not (Test-Path $probe)) {
dotnet build (Join-Path $repo "src\Mimir.PsMoveProbe\Mimir.PsMoveProbe.csproj") -c Release -p:UseSharedCompilation=false
}
if (-not $KeepExisting) {
Get-Process -Name "Mimir.PsMoveProbe" -ErrorAction SilentlyContinue |
ForEach-Object {
Stop-Process -Id $_.Id -Force
}
}
$args = @(
"--rgb", $Rgb,
"--breathe-led",
"--hold-seconds", "$HoldSeconds",
"--refresh-ms", "$RefreshMs",
"--event-log", $eventLog
)
if ($TurnOffOnExit) {
$args += "--turn-off-on-exit"
}
if ($DryRun) {
Write-Host "DRY starfire-move-light: $probe $($args -join ' ')"
return
}
$stdout = Join-Path $runDir "starfire-move-light.out.log"
$stderr = Join-Path $runDir "starfire-move-light.err.log"
$process = Start-Process `
-FilePath $probe `
-ArgumentList $args `
-WorkingDirectory $repo `
-WindowStyle Hidden `
-RedirectStandardOutput $stdout `
-RedirectStandardError $stderr `
-PassThru
Set-Content -Path (Join-Path $runDir "starfire-move-light.pid") -Value $process.Id
[ordered]@{
kind = "mimir.starfire_move_light_supervisor.v1"
runId = $runId
runDir = $runDir
rgb = $Rgb
refreshMs = $RefreshMs
holdSeconds = $HoldSeconds
pid = $process.Id
eventLog = $eventLog
startedAt = (Get-Date).ToString("o")
} | ConvertTo-Json -Depth 4 | Set-Content -Path (Join-Path $runDir "supervisor.json")
Write-Host "starfire-move-light pid=$($process.Id) runDir=$runDir eventLog=$eventLog"

View file

@ -0,0 +1,8 @@
param(
[string]$Cache = "C:\Meta\Mimir\state\mimir-cultmesh-media.cc"
)
$ErrorActionPreference = "Stop"
$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj"
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $Cache) | Out-Null
dotnet run --project $project -- relay --cache $Cache

View file

@ -10,7 +10,9 @@
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\CultLib\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<PackageReference Include="System.Drawing.Common" Version="10.0.0" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Mesh\GameCult.Mesh.csproj" />
<ProjectReference Include="..\Mimir.Runtime\Mimir.Runtime.csproj" />
</ItemGroup>

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,18 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>Mimir.CultMeshMedia</AssemblyName>
<RootNamespace>Mimir.CultMeshMedia</RootNamespace>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\Mimir.Runtime\Mimir.Runtime.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Mesh\GameCult.Mesh.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Networking\GameCult.Networking.csproj" />
</ItemGroup>
</Project>

View file

@ -0,0 +1,568 @@
using System.Diagnostics;
using System.Globalization;
using System.Net;
using System.Net.Sockets;
using GameCult.Caching;
using GameCult.Caching.MessagePack;
using GameCult.Mesh;
using GameCult.Networking;
using Mimir.Runtime.Synchronization;
const string SchemaName = "mimir.cultmesh_media_frame";
const string DefaultStreamId = "raven-primary-av";
const uint DefaultConnectionId = 0x4d4d_0101;
const int DefaultPort = 3075;
const int DefaultMaxFragmentBytes = 1200;
const int DefaultMaxPendingReliablePackets = 8192;
var command = args.Length > 0 ? args[0].Trim().ToLowerInvariant() : string.Empty;
var options = CliOptions.Parse(args.Skip(1));
return command switch
{
"relay" => await RunRelayAsync(options),
"send" => await RunSenderAsync(options),
"recv" or "receive" => await RunReceiverAsync(options),
_ => Usage()
};
static int Usage()
{
Console.Error.WriteLine("""
Usage:
Mimir.CultMeshMedia relay --cache <path> [--bind 0.0.0.0:3075]
Mimir.CultMeshMedia send --host <rudp-host> [--port 3075] [--stream raven-primary-av] [--chunk-bytes 16384] [--slots 96]
Mimir.CultMeshMedia recv --host <rudp-host> [--port 3075] [--stream raven-primary-av] [--udp 127.0.0.1:5200]
send reads an MPEG-TS byte stream from stdin and publishes rolling CultMesh media-frame documents over CultNet RUDP.
recv subscribes to those RUDP document changes and writes ordered payload bytes to a local UDP MPEG-TS endpoint for OBS.
""");
return 2;
}
static async Task<int> RunRelayAsync(CliOptions options)
{
var cachePath = options.Get("cache") ?? Path.Combine(Environment.CurrentDirectory, "mimir-cultmesh-media.cc");
var bind = ParseEndpoint(options.Get("bind") ?? $"0.0.0.0:{options.GetInt("port", DefaultPort)}");
var registry = CreateDocumentRegistry();
var schemaId = ResolveMediaSchemaId(registry);
using var cache = await CultCacheMessagePack.OpenAsync(cachePath).ConfigureAwait(false);
using var relay = new RudpMediaRelay(
bind,
cache,
registry,
schemaId,
options.GetUInt("connection-id", DefaultConnectionId),
options.GetInt("max-fragment-bytes", DefaultMaxFragmentBytes),
options.GetInt("max-pending-reliable", DefaultMaxPendingReliablePackets));
Console.WriteLine($"CultMesh media relay listening on rudp://{bind.Address}:{bind.Port}; cache={Path.GetFullPath(cachePath)}");
await relay.RunAsync().ConfigureAwait(false);
await cache.FlushAsync().ConfigureAwait(false);
return 0;
}
static async Task<int> RunSenderAsync(CliOptions options)
{
var endpoint = ResolveRudpEndpoint(options);
var streamId = options.Get("stream") ?? DefaultStreamId;
var chunkBytes = Math.Clamp(options.GetInt("chunk-bytes", 16 * 1024), 1316, 60 * 1024);
var slots = Math.Clamp(options.GetInt("slots", 96), 8, 4096);
var producer = options.Get("producer") ?? Environment.MachineName.ToLowerInvariant();
var registry = CreateDocumentRegistry();
using var transport = CreateRudpClient(
options.Get("runtime") ?? $"{producer}-mimir-media-sender",
endpoint,
options);
ConnectOrThrow(transport, endpoint.Uri);
Console.Error.WriteLine($"Sending stdin MPEG-TS to {endpoint.Uri}/{streamId}; chunkBytes={chunkBytes}; slots={slots}");
var input = Console.OpenStandardInput();
var buffer = new byte[chunkBytes];
long sequence = 0;
var started = Stopwatch.StartNew();
while (true)
{
var read = await input.ReadAsync(buffer.AsMemory(0, buffer.Length)).ConfigureAwait(false);
if (read <= 0)
{
break;
}
var payload = buffer.AsSpan(0, read).ToArray();
var document = new MimirCultMeshMediaFrameDocument(
FrameId: $"{streamId}:{sequence:D20}",
StreamId: streamId,
Sequence: sequence,
ProducedAtUtc: DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture),
TimestampNanoseconds: started.ElapsedTicks * 1_000_000_000L / Stopwatch.Frequency,
PayloadKind: "mpegts-chunk",
Container: "mpegts",
VideoCodec: options.Get("video-codec") ?? "h264",
AudioCodec: options.Get("audio-codec") ?? "aac",
PayloadBytes: payload.Length,
Payload: payload,
ProducerNode: producer,
ClockDomainId: options.Get("clock-domain") ?? "raven-monotonic",
Tags: ["raven", "program", "obs", "cultmesh-media", "cultnet.transport.rudp.v0"]);
var handle = new CultRecordHandle<MimirCultMeshMediaFrameDocument>(
new CultRecordKey($"{streamId}:slot:{sequence % slots:D4}"));
var message = registry.CreateRawDocumentPutMessage(
Guid.NewGuid().ToString("N"),
handle,
document,
new CultNetDocumentMessageOptions
{
SourceRuntimeId = producer,
SourceRole = "mimir-cultmesh-media-sender",
Tags = ["mimir", "media", streamId, "cultnet.transport.rudp.v0"]
});
transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message));
PumpClient(transport, TimeSpan.FromMilliseconds(5));
sequence++;
if (sequence % 120 == 0)
{
Console.Error.WriteLine($"sent sequence={sequence - 1}");
}
}
Console.Error.WriteLine($"sent chunks={sequence}; draining");
PumpClient(transport, TimeSpan.FromMilliseconds(options.GetInt("drain-ms", 1000)));
return 0;
}
static async Task<int> RunReceiverAsync(CliOptions options)
{
var endpoint = ResolveRudpEndpoint(options);
var streamId = options.Get("stream") ?? DefaultStreamId;
var udp = options.Get("udp") ?? "127.0.0.1:5200";
var udpEndpoint = ParseEndpoint(udp);
var registry = CreateDocumentRegistry();
var schemaId = ResolveMediaSchemaId(registry);
var lastSequence = -1L;
long frames = 0;
long bytes = 0;
var loggedFirstFrame = false;
using var socket = new UdpClient();
using var transport = CreateRudpClient(
options.Get("runtime") ?? $"{Environment.MachineName.ToLowerInvariant()}-mimir-media-receiver",
endpoint,
options);
ConnectOrThrow(transport, endpoint.Uri);
transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(new CultNetDatabaseSubscribeMessage
{
MessageId = Guid.NewGuid().ToString("N"),
SubscriptionId = $"mimir-media-{streamId}",
SchemaIds = [schemaId],
IncludeSnapshot = false
}));
Console.WriteLine($"Receiving {endpoint.Uri}/{streamId}; writing MPEG-TS UDP to udp://{udp}");
while (true)
{
transport.PollResends();
if (transport.ReceiveOnce() is not { } frame)
{
await Task.Delay(2).ConfigureAwait(false);
continue;
}
if (!string.Equals(frame.ChannelId, "schema", StringComparison.Ordinal))
{
continue;
}
if (CultNetSchemaMessageSerialization.Deserialize(frame.Payload) is not CultNetDatabaseChangeRawMessage change ||
change.Document is not { } rawDocument ||
!string.Equals(rawDocument.SchemaId, schemaId, StringComparison.Ordinal))
{
continue;
}
var document = (MimirCultMeshMediaFrameDocument)registry
.GetBySchemaId(schemaId)!
.PayloadDeserializer(rawDocument.Payload);
if (!string.Equals(document.StreamId, streamId, StringComparison.Ordinal) ||
document.Sequence <= lastSequence)
{
continue;
}
lastSequence = document.Sequence;
socket.Send(document.Payload, document.Payload.Length, udpEndpoint);
frames++;
bytes += document.Payload.Length;
if (!loggedFirstFrame)
{
loggedFirstFrame = true;
Console.WriteLine($"received first sequence={document.Sequence} bytes={document.PayloadBytes}");
}
if (frames % 120 == 0)
{
Console.WriteLine($"received sequence={document.Sequence} frames={frames} bytes={bytes}");
}
}
}
static CultMeshRudpEndpoint ResolveRudpEndpoint(CliOptions options)
{
if (options.Get("endpoint") is { } endpoint)
{
return CultMesh.ParseRudpEndpoint(endpoint);
}
var host = options.Get("host") ?? "10.77.0.1";
var port = options.GetInt("port", DefaultPort);
return CultMesh.ParseRudpEndpoint($"rudp://{host}:{port}");
}
static CultNetRudpSocketTransportConnection CreateRudpClient(
string runtimeId,
CultMeshRudpEndpoint endpoint,
CliOptions options)
{
return CultMesh.CreateRudpClient(
runtimeId,
options.GetUInt("connection-id", DefaultConnectionId),
endpoint,
new CultMeshRudpSocketOptions
{
BindHost = options.Get("bind-host") ?? "0.0.0.0",
BindPort = options.GetInt("bind-port", 0),
ResendDelayMs = options.GetInt("resend-ms", 100),
MaxFragmentBytes = options.GetInt("max-fragment-bytes", DefaultMaxFragmentBytes),
MaxPendingReliablePackets = options.GetInt("max-pending-reliable", DefaultMaxPendingReliablePackets)
});
}
static void ConnectOrThrow(CultNetRudpSocketTransportConnection transport, string endpoint)
{
if (!transport.ConnectAndWait(Array.Empty<byte>(), TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(5)))
{
throw new TimeoutException($"timed out connecting RUDP media transport to {endpoint}");
}
}
static void PumpClient(CultNetRudpSocketTransportConnection transport, TimeSpan duration)
{
var until = Stopwatch.StartNew();
while (until.Elapsed < duration)
{
transport.PollResends();
transport.ReceiveOnce();
Thread.Sleep(1);
}
}
static string ResolveMediaSchemaId(CultNetDocumentRegistry registry)
{
var binding = registry.GetByDocumentType(typeof(MimirCultMeshMediaFrameDocument)) ??
throw new InvalidOperationException($"CultNet document binding missing for {SchemaName}.");
return binding.SchemaId;
}
static CultNetDocumentRegistry CreateDocumentRegistry()
{
var documents = CultDocumentRegistry.Shared;
documents.GetRequired<MimirCultMeshMediaFrameDocument>();
return new CultNetDocumentRegistry(documents)
.Register(CultNetDocumentBinding.ForDocument<MimirCultMeshMediaFrameDocument>(documents));
}
static IPEndPoint ParseEndpoint(string value)
{
var split = value.LastIndexOf(':');
if (split <= 0 || split == value.Length - 1)
{
throw new ArgumentException($"Endpoint must be host:port, not '{value}'.");
}
var host = value[..split];
var port = int.Parse(value[(split + 1)..], CultureInfo.InvariantCulture);
var addresses = Dns.GetHostAddresses(host);
return new IPEndPoint(addresses.First(address => address.AddressFamily == AddressFamily.InterNetwork), port);
}
sealed class RudpMediaRelay : IDisposable
{
private readonly Socket socket;
private readonly CultCache cache;
private readonly CultNetDocumentRegistry registry;
private readonly string schemaId;
private readonly uint connectionId;
private readonly int maxFragmentBytes;
private readonly int maxPendingReliablePackets;
private readonly Dictionary<string, RelayPeer> peers = new(StringComparer.Ordinal);
public RudpMediaRelay(
IPEndPoint bind,
CultCache cache,
CultNetDocumentRegistry registry,
string schemaId,
uint connectionId,
int maxFragmentBytes,
int maxPendingReliablePackets)
{
this.cache = cache;
this.registry = registry;
this.schemaId = schemaId;
this.connectionId = connectionId;
this.maxFragmentBytes = maxFragmentBytes;
this.maxPendingReliablePackets = maxPendingReliablePackets;
socket = new Socket(bind.AddressFamily, SocketType.Dgram, ProtocolType.Udp);
socket.Bind(bind);
socket.Blocking = false;
}
public async Task RunAsync()
{
while (true)
{
ReceiveAvailable();
PumpReliableResends();
await Task.Delay(2).ConfigureAwait(false);
}
}
public void Dispose()
{
socket.Dispose();
}
private void ReceiveAvailable()
{
while (socket.Poll(0, SelectMode.SelectRead))
{
var buffer = new byte[65535];
EndPoint remote = new IPEndPoint(IPAddress.Any, 0);
int received;
try
{
received = socket.ReceiveFrom(buffer, ref remote);
}
catch (SocketException error) when (error.SocketErrorCode == SocketError.WouldBlock)
{
return;
}
var wire = buffer.AsSpan(0, received).ToArray();
var packet = CultNetRudpPacketCodec.Decode(wire);
if (packet.ConnectionId != connectionId)
{
continue;
}
var peer = GetPeer((IPEndPoint)remote);
if (packet.PacketType == CultNetRudpPacketType.Connect)
{
SendPacket(peer, peer.Session.AcceptConnect(packet, NowMs()));
continue;
}
var result = peer.Session.Receive(packet, NowMs());
if (result.Reply != null)
{
SendPacket(peer, result.Reply);
}
if (result.Disconnected)
{
peers.Remove(peer.Key);
continue;
}
foreach (var frame in result.Delivered.Where(static frame => string.Equals(frame.ChannelId, "schema", StringComparison.Ordinal)))
{
HandleSchemaFrameAsync(peer, frame.Payload).GetAwaiter().GetResult();
}
if (packet.PacketType == CultNetRudpPacketType.Accept || result.Delivered.Count > 0)
{
SendPacket(peer, peer.Session.CreateAck());
}
}
}
private async Task HandleSchemaFrameAsync(RelayPeer peer, byte[] payload)
{
var message = CultNetSchemaMessageSerialization.Deserialize(payload);
switch (message)
{
case CultNetDocumentPutRawMessage put:
await ApplyPutAsync(put).ConfigureAwait(false);
break;
case CultNetDatabaseSubscribeMessage subscribe:
peer.Subscriptions[subscribe.SubscriptionId] = subscribe;
Console.WriteLine($"relay RUDP subscribe peer={peer.Key} subscription={subscribe.SubscriptionId} schemas={string.Join(",", subscribe.SchemaIds ?? [])}");
break;
case CultNetDatabaseUnsubscribeMessage unsubscribe:
peer.Subscriptions.Remove(unsubscribe.SubscriptionId);
break;
}
}
private async Task ApplyPutAsync(CultNetDocumentPutRawMessage put)
{
if (put.Document == null ||
!string.Equals(put.Document.SchemaId, schemaId, StringComparison.Ordinal))
{
return;
}
await registry.ApplyRawDocumentPutMessageAsync(cache, put).ConfigureAwait(false);
await cache.FlushAsync().ConfigureAwait(false);
var change = new CultNetDatabaseChangeRawMessage
{
MessageId = Guid.NewGuid().ToString("N"),
ChangeKind = "updated",
Document = put.Document
};
var frame = CultNetSchemaMessageSerialization.Serialize(change);
foreach (var peer in peers.Values)
{
foreach (var subscription in peer.Subscriptions.Values)
{
if (!Matches(subscription, put.Document.SchemaId, put.Document.RecordKey))
{
continue;
}
change.SubscriptionId = subscription.SubscriptionId;
SendSchema(peer, CultNetSchemaMessageSerialization.Serialize(change));
if (!peer.HasForwardedFrame)
{
peer.HasForwardedFrame = true;
Console.WriteLine($"relay RUDP forward peer={peer.Key} subscription={subscription.SubscriptionId} schema={put.Document.SchemaId}");
}
ReceiveAvailable();
}
}
var document = (MimirCultMeshMediaFrameDocument)registry
.GetBySchemaId(schemaId)!
.PayloadDeserializer(put.Document.Payload);
if (document.Sequence % 120 == 0)
{
Console.WriteLine($"relay RUDP put stream={document.StreamId} sequence={document.Sequence} bytes={document.PayloadBytes}");
}
}
private static bool Matches(CultNetDatabaseSubscribeMessage request, string schemaId, string recordKey)
{
var schemaMatches = request.SchemaIds == null ||
request.SchemaIds.Length == 0 ||
request.SchemaIds.Contains(schemaId, StringComparer.Ordinal);
var keyMatches = request.RecordKeys == null ||
request.RecordKeys.Length == 0 ||
request.RecordKeys.Contains(recordKey, StringComparer.Ordinal);
return schemaMatches && keyMatches;
}
private RelayPeer GetPeer(IPEndPoint endpoint)
{
var key = endpoint.ToString();
if (peers.TryGetValue(key, out var existing))
{
return existing;
}
var peer = new RelayPeer(
key,
endpoint,
new CultNetRudpSession(new CultNetRudpSessionOptions
{
ConnectionId = connectionId,
InitialSequence = (uint)(peers.Count + 1),
ResendDelayMs = 100,
MaxPendingReliablePackets = maxPendingReliablePackets
}));
peers[key] = peer;
return peer;
}
private void SendSchema(RelayPeer peer, byte[] payload)
{
foreach (var packet in peer.Session.SendMany(
"schema",
payload,
new CultNetRudpSendOptions { Reliable = true, Ordered = true, NowMs = NowMs() },
maxFragmentBytes))
{
SendPacket(peer, packet);
}
}
private void SendPacket(RelayPeer peer, CultNetRudpPacket packet)
{
var wire = CultNetRudpPacketCodec.Encode(packet);
socket.SendTo(wire, peer.Endpoint);
}
private void PumpReliableResends()
{
foreach (var peer in peers.Values)
{
foreach (var packet in peer.Session.DueResends(NowMs()))
{
SendPacket(peer, packet);
}
}
}
private static long NowMs() => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
}
sealed record RelayPeer(
string Key,
IPEndPoint Endpoint,
CultNetRudpSession Session)
{
public Dictionary<string, CultNetDatabaseSubscribeMessage> Subscriptions { get; } = new(StringComparer.Ordinal);
public bool HasForwardedFrame { get; set; }
}
sealed class CliOptions
{
private readonly Dictionary<string, string> values = new(StringComparer.OrdinalIgnoreCase);
public static CliOptions Parse(IEnumerable<string> args)
{
var parsed = new CliOptions();
var pendingKey = string.Empty;
foreach (var arg in args)
{
if (arg.StartsWith("--", StringComparison.Ordinal))
{
pendingKey = arg[2..];
parsed.values[pendingKey] = "true";
continue;
}
if (!string.IsNullOrWhiteSpace(pendingKey))
{
parsed.values[pendingKey] = arg;
pendingKey = string.Empty;
}
}
return parsed;
}
public string? Get(string key) => values.TryGetValue(key, out var value) ? value : null;
public int GetInt(string key, int fallback) =>
int.TryParse(Get(key), NumberStyles.Integer, CultureInfo.InvariantCulture, out var value) ? value : fallback;
public uint GetUInt(string key, uint fallback) =>
uint.TryParse(Get(key), NumberStyles.Integer, CultureInfo.InvariantCulture, out var value) ? value : fallback;
}

View file

@ -0,0 +1,17 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>Mimir.EveSensorReceiver</AssemblyName>
<RootNamespace>Mimir.EveSensorReceiver</RootNamespace>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="..\Mimir.Runtime\Mimir.Runtime.csproj" />
</ItemGroup>
</Project>

View file

@ -0,0 +1,501 @@
using System.Net;
using System.Net.Sockets;
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using MessagePack;
using Mimir.Runtime.Synchronization;
var options = EveSensorReceiverOptions.Parse(args);
using var receiver = new EveSensorReceiver(options);
await receiver.RunAsync();
internal sealed record EveSensorReceiverOptions(
int Port,
string Path,
string SourceId,
string ExpectedType,
string SubscribePath)
{
public static EveSensorReceiverOptions Parse(string[] args)
{
return new EveSensorReceiverOptions(
ParseInt(args, "--port", 8793),
ParseString(args, "--path", "/eve/camera"),
ParseString(args, "--source-id", "eve-camera"),
ParseString(args, "--type", "video-frame"),
ParseString(args, "--subscribe-path", ParseString(args, "--path", "/eve/camera") + "/subscribe"));
}
private static string ParseString(IReadOnlyList<string> args, string name, string fallback)
{
for (var index = 0; index < args.Count - 1; index++)
{
if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase))
{
return args[index + 1];
}
}
return fallback;
}
private static int ParseInt(IReadOnlyList<string> args, string name, int fallback)
{
return int.TryParse(ParseString(args, name, ""), out var value) ? value : fallback;
}
}
internal sealed class EveSensorReceiver(EveSensorReceiverOptions options) : IDisposable
{
private readonly TcpListener listener = new(IPAddress.Any, options.Port);
private readonly CancellationTokenSource stopping = new();
private readonly ConcurrentDictionary<int, WebSocketSubscriber> subscribers = new();
private int nextSubscriberId;
public async Task RunAsync()
{
listener.Start();
Console.Error.WriteLine($"Eve sensor receiver listening on ws://0.0.0.0:{options.Port}{options.Path} source={options.SourceId}");
while (!stopping.IsCancellationRequested)
{
var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false);
_ = Task.Run(() => HandleAsync(client));
}
}
private async Task HandleAsync(TcpClient tcpClient)
{
await using var stream = tcpClient.GetStream();
var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false);
if (request.Path == "/health")
{
var health = JsonSerializer.Serialize(new
{
ok = true,
options.SourceId,
options.ExpectedType,
options.Path,
options.SubscribePath,
Subscribers = subscribers.Count,
});
await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false);
return;
}
if (string.Equals(request.Path, options.SubscribePath, StringComparison.Ordinal)
&& request.Headers.TryGetValue("Sec-WebSocket-Key", out var subscriberKey))
{
await WriteWebSocketHandshakeAsync(stream, subscriberKey).ConfigureAwait(false);
await SubscribeAsync(stream).ConfigureAwait(false);
return;
}
if (!string.Equals(request.Path, options.Path, StringComparison.Ordinal)
|| !request.Headers.TryGetValue("Sec-WebSocket-Key", out var key))
{
await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false);
return;
}
await WriteWebSocketHandshakeAsync(stream, key).ConfigureAwait(false);
while (!stopping.IsCancellationRequested)
{
var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false);
if (frame.Opcode == 0x8)
{
await WriteCloseFrameAsync(stream).ConfigureAwait(false);
break;
}
if (frame.Opcode == 0x2)
{
if (TryValidateCultMeshObservation(frame.Payload, out var normalizedObservation)
|| TryValidateCultMeshMediaObservation(frame.Payload, out normalizedObservation))
{
Console.WriteLine(normalizedObservation);
Console.Out.Flush();
await BroadcastAsync(normalizedObservation).ConfigureAwait(false);
}
continue;
}
if (frame.Opcode != 0x1)
{
continue;
}
var text = Encoding.UTF8.GetString(frame.Payload);
if (TryValidateFrameEvent(text, out var normalized))
{
Console.WriteLine(normalized);
Console.Out.Flush();
await BroadcastAsync(normalized).ConfigureAwait(false);
}
}
}
private async Task SubscribeAsync(NetworkStream stream)
{
var id = Interlocked.Increment(ref nextSubscriberId);
var subscriber = new WebSocketSubscriber(id, stream);
subscribers[id] = subscriber;
Console.Error.WriteLine($"Eve sensor receiver subscriber {id} connected on {options.SubscribePath}.");
try
{
await subscriber.SendTextAsync(JsonSerializer.Serialize(new
{
type = "mimir-verse-subscription",
status = "connected",
options.Path,
options.SubscribePath,
options.SourceId,
options.ExpectedType,
})).ConfigureAwait(false);
while (!stopping.IsCancellationRequested)
{
var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false);
if (frame.Opcode == 0x8)
{
await WriteCloseFrameAsync(stream).ConfigureAwait(false);
break;
}
}
}
catch (IOException)
{
}
catch (SocketException)
{
}
finally
{
subscribers.TryRemove(id, out _);
Console.Error.WriteLine($"Eve sensor receiver subscriber {id} disconnected.");
}
}
private async Task BroadcastAsync(string normalized)
{
foreach (var pair in subscribers.ToArray())
{
try
{
await pair.Value.SendTextAsync(normalized).ConfigureAwait(false);
}
catch (IOException)
{
subscribers.TryRemove(pair.Key, out _);
}
catch (SocketException)
{
subscribers.TryRemove(pair.Key, out _);
}
catch (ObjectDisposedException)
{
subscribers.TryRemove(pair.Key, out _);
}
}
}
private bool TryValidateFrameEvent(string text, out string normalized)
{
normalized = "";
try
{
using var document = JsonDocument.Parse(text);
var root = document.RootElement;
if (!root.TryGetProperty("type", out var type)
|| !string.Equals(type.GetString(), options.ExpectedType, StringComparison.OrdinalIgnoreCase))
{
return false;
}
if (!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase)
&& (!root.TryGetProperty("sourceId", out var sourceId)
|| !string.Equals(sourceId.GetString(), options.SourceId, StringComparison.Ordinal)))
{
return false;
}
normalized = root.GetRawText();
return true;
}
catch (JsonException)
{
return false;
}
}
private bool TryValidateCultMeshObservation(byte[] payload, out string normalized)
{
normalized = "";
try
{
var observation = MessagePackSerializer.Deserialize<MimirEveSensorObservationDocument>(payload);
if (!string.Equals(observation.StreamId, options.SourceId, StringComparison.Ordinal) &&
!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase))
{
return false;
}
normalized = JsonSerializer.Serialize(new
{
type = "cultmesh-observation",
document = "mimir.eve_sensor_observation.v1",
observation.ObservationId,
observation.DeviceId,
observation.StreamId,
observation.Kind,
observation.Sequence,
observation.SensorTimestampNs,
observation.ElapsedRealtimeNs,
observation.WallClockUtc,
observation.ClockDomainId,
observation.Values,
observation.Action,
observation.PointerCount,
observation.X,
observation.Y,
observation.Accuracy,
});
return true;
}
catch
{
return false;
}
}
private bool TryValidateCultMeshMediaObservation(byte[] payload, out string normalized)
{
normalized = "";
try
{
var observation = MessagePackSerializer.Deserialize<MimirEveMediaObservationDocument>(payload);
if (!string.Equals(observation.StreamId, options.SourceId, StringComparison.Ordinal) &&
!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase))
{
return false;
}
normalized = JsonSerializer.Serialize(new
{
type = "cultmesh-media-observation",
document = "mimir.eve_media_observation.v1",
observation.ObservationId,
observation.DeviceId,
observation.StreamId,
observation.Kind,
observation.Sequence,
observation.SensorTimestampNs,
observation.ElapsedRealtimeNs,
observation.WallClockUtc,
observation.ClockDomainId,
observation.Format,
observation.Width,
observation.Height,
observation.SampleRate,
observation.Channels,
observation.FrameCount,
observation.PayloadEncoding,
PayloadBytes = observation.Payload.Length,
});
return true;
}
catch
{
return false;
}
}
private static async Task<HttpRequest> ReadHttpRequestAsync(NetworkStream stream)
{
var bytes = new List<byte>();
var lastFour = new Queue<byte>(4);
while (true)
{
var value = stream.ReadByte();
if (value < 0)
{
throw new IOException("Client closed before HTTP headers completed.");
}
var b = (byte)value;
bytes.Add(b);
lastFour.Enqueue(b);
if (lastFour.Count > 4)
{
lastFour.Dequeue();
}
if (lastFour.Count == 4 && lastFour.SequenceEqual(new byte[] { 13, 10, 13, 10 }))
{
break;
}
}
var text = Encoding.ASCII.GetString(bytes.ToArray());
var lines = text.Split("\r\n", StringSplitOptions.None);
var first = lines[0].Split(' ');
var headers = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
foreach (var line in lines.Skip(1))
{
var colon = line.IndexOf(':');
if (colon > 0)
{
headers[line[..colon].Trim()] = line[(colon + 1)..].Trim();
}
}
await Task.CompletedTask.ConfigureAwait(false);
return new HttpRequest(first.Length > 1 ? first[1] : "/", headers);
}
private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body)
{
var header = Encoding.ASCII.GetBytes(
$"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n");
await stream.WriteAsync(header).ConfigureAwait(false);
await stream.WriteAsync(body).ConfigureAwait(false);
}
private static async Task WriteWebSocketHandshakeAsync(NetworkStream stream, string key)
{
var acceptBytes = SHA1.HashData(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"));
var accept = Convert.ToBase64String(acceptBytes);
var response = Encoding.ASCII.GetBytes(
"HTTP/1.1 101 Switching Protocols\r\n" +
"Upgrade: websocket\r\n" +
"Connection: Upgrade\r\n" +
$"Sec-WebSocket-Accept: {accept}\r\n\r\n");
await stream.WriteAsync(response).ConfigureAwait(false);
}
private static async Task<WebSocketFrame> ReceiveFrameAsync(NetworkStream stream)
{
var header = new byte[2];
await ReadExactAsync(stream, header).ConfigureAwait(false);
var opcode = header[0] & 0x0f;
var masked = (header[1] & 0x80) != 0;
var length = header[1] & 0x7f;
if (length == 126)
{
var extended = new byte[2];
await ReadExactAsync(stream, extended).ConfigureAwait(false);
length = (extended[0] << 8) | extended[1];
}
else if (length == 127)
{
var extended = new byte[8];
await ReadExactAsync(stream, extended).ConfigureAwait(false);
var longLength = 0L;
foreach (var b in extended)
{
longLength = (longLength << 8) | b;
}
if (longLength > int.MaxValue)
{
throw new InvalidDataException("WebSocket frame is too large.");
}
length = (int)longLength;
}
var mask = new byte[4];
if (masked)
{
await ReadExactAsync(stream, mask).ConfigureAwait(false);
}
var payload = new byte[length];
await ReadExactAsync(stream, payload).ConfigureAwait(false);
if (masked)
{
for (var index = 0; index < payload.Length; index++)
{
payload[index] ^= mask[index % 4];
}
}
return new WebSocketFrame(opcode, payload);
}
private static Task WriteCloseFrameAsync(NetworkStream stream)
{
return stream.WriteAsync(new byte[] { 0x88, 0x00 }).AsTask();
}
private static async Task ReadExactAsync(NetworkStream stream, byte[] buffer)
{
var offset = 0;
while (offset < buffer.Length)
{
var read = await stream.ReadAsync(buffer.AsMemory(offset, buffer.Length - offset)).ConfigureAwait(false);
if (read == 0)
{
throw new IOException("WebSocket closed.");
}
offset += read;
}
}
public void Dispose()
{
stopping.Cancel();
listener.Stop();
stopping.Dispose();
}
}
internal sealed record HttpRequest(string Path, IReadOnlyDictionary<string, string> Headers);
internal sealed record WebSocketFrame(int Opcode, byte[] Payload);
internal sealed class WebSocketSubscriber(int id, NetworkStream stream)
{
private readonly SemaphoreSlim writeLock = new(1, 1);
public int Id => id;
public async Task SendTextAsync(string text)
{
var payload = Encoding.UTF8.GetBytes(text);
var header = new List<byte> { 0x81 };
if (payload.Length < 126)
{
header.Add((byte)payload.Length);
}
else if (payload.Length <= ushort.MaxValue)
{
header.Add(126);
header.Add((byte)(payload.Length >> 8));
header.Add((byte)(payload.Length & 0xff));
}
else
{
header.Add(127);
for (var shift = 56; shift >= 0; shift -= 8)
{
header.Add((byte)(((ulong)payload.Length >> shift) & 0xff));
}
}
await writeLock.WaitAsync().ConfigureAwait(false);
try
{
await stream.WriteAsync(header.ToArray()).ConfigureAwait(false);
await stream.WriteAsync(payload).ConfigureAwait(false);
await stream.FlushAsync().ConfigureAwait(false);
}
finally
{
writeLock.Release();
}
}
}

View file

@ -0,0 +1,10 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>

File diff suppressed because it is too large Load diff

View file

@ -12,7 +12,9 @@
<ItemGroup>
<ProjectReference Include="..\..\..\Fensalir\src\Aquarium.Engine.Contracts\Aquarium.Engine.Contracts.csproj" />
<ProjectReference Include="..\..\..\CultLib\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Caching.MessagePack\GameCult.Caching.MessagePack.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Mesh\GameCult.Mesh.csproj" />
<ProjectReference Include="$(CultLibRoot)\src\GameCult.Networking\GameCult.Networking.csproj" />
</ItemGroup>
</Project>

View file

@ -39,6 +39,8 @@ direct
private readonly MimirSynchronizationHub synchronization;
private readonly MimirAudioSpectrumAnalyzer spectrumAnalyzer;
private readonly IReadOnlyList<MimirStreamSourceFactory> sourceFactories;
private readonly IReadOnlyList<MimirMoveProofRuntimeConfiguration> configuredMoveProofSources;
private readonly IMimirMoveProofEvidenceRingProvider moveProofRingProvider;
private readonly AquariumUiDocument ui;
private readonly AquariumAudioDocument audio = new();
private readonly MimirAudioSynchronizationSettings audioSyncSettings;
@ -62,6 +64,11 @@ direct
private IReadOnlyList<MimirAudioSynchronizationReport> lastAudioSynchronizationReports = [];
private IReadOnlyList<MimirAudioSpectrumSnapshot> lastAudioSpectra = [];
private readonly Queue<IReadOnlyList<MimirAudioSpectrumSnapshot>> spectrumHistory = new();
private readonly List<MimirMoveProofRuntimeDriver> moveProofDrivers = [];
private readonly List<IDisposable> moveProofDriverResources = [];
private readonly List<MimirMoveProofRuntimeActivationStatus> moveProofActivationStatuses = [];
private MimirMoveProofSurfaceDocument? latestMoveProofSurface;
private bool moveProofSourcesActivated;
public MimirRuntime(AquariumRuntimeOptions options)
: this(options, MimirRuntimeConfiguration.Load())
@ -69,7 +76,20 @@ direct
}
public MimirRuntime(AquariumRuntimeOptions options, MimirRuntimeConfiguration configuration)
: this(options, configuration.Settings, configuration.SourceFactories)
: this(
options,
configuration.Settings,
configuration.SourceFactories,
configuration.MoveProofSources,
MimirConfiguredMoveProofEvidenceRingProvider.Instance)
{
}
public MimirRuntime(
AquariumRuntimeOptions options,
MimirRuntimeConfiguration configuration,
IMimirMoveProofEvidenceRingProvider moveProofRingProvider)
: this(options, configuration.Settings, configuration.SourceFactories, configuration.MoveProofSources, moveProofRingProvider)
{
}
@ -82,7 +102,12 @@ direct
AquariumRuntimeOptions options,
MimirSynchronizationSettings settings,
IEnumerable<IMimirStreamSource> streamSources)
: this(options, settings, Array.Empty<MimirStreamSourceFactory>())
: this(
options,
settings,
Array.Empty<MimirStreamSourceFactory>(),
Array.Empty<MimirMoveProofRuntimeConfiguration>(),
MimirUnavailableMoveProofEvidenceRingProvider.Instance)
{
foreach (var source in streamSources)
{
@ -93,12 +118,16 @@ direct
private MimirRuntime(
AquariumRuntimeOptions options,
MimirSynchronizationSettings settings,
IEnumerable<MimirStreamSourceFactory> sourceFactories)
IEnumerable<MimirStreamSourceFactory> sourceFactories,
IEnumerable<MimirMoveProofRuntimeConfiguration> moveProofSources,
IMimirMoveProofEvidenceRingProvider moveProofRingProvider)
{
Options = options;
synchronization = new MimirSynchronizationHub(settings);
spectrumAnalyzer = new MimirAudioSpectrumAnalyzer(ParseSpectrumFftSize());
this.sourceFactories = sourceFactories.ToArray();
configuredMoveProofSources = moveProofSources.ToArray();
this.moveProofRingProvider = moveProofRingProvider ?? throw new ArgumentNullException(nameof(moveProofRingProvider));
audioSyncSettings = settings.Audio;
telemetryIntervalSeconds = ParseTelemetryIntervalSeconds();
audioSyncUpdateIntervalSeconds = ParseAudioSyncIntervalSeconds();
@ -113,6 +142,11 @@ direct
: null;
nextAudioSyncSeconds = audioSyncUpdateIntervalSeconds;
nextTelemetrySeconds = telemetryIntervalSeconds;
if (MimirMoveProofDevSurface.IsEnabled())
{
PublishMoveProofSurface(MimirMoveProofDevSurface.Create());
}
ui = CreateUi();
}
@ -128,6 +162,20 @@ direct
public AquariumAudioDocument Audio => audio;
public IReadOnlyList<MimirMoveProofRuntimeActivationStatus> MoveProofActivationStatuses => moveProofActivationStatuses;
public MimirMoveProofSurfaceDocument? LatestMoveProofSurface => latestMoveProofSurface;
public IReadOnlyList<MimirMoveProofRuntimeActivationDocument> CreateMoveProofRuntimeActivationDocuments(ulong observedAtNs) =>
configuredMoveProofSources
.Select(configuration => MimirMoveProofRuntimeActivation.CreateDocument(
configuration,
moveProofActivationStatuses.LastOrDefault(status =>
string.Equals(status.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)),
latestMoveProofSurface,
observedAtNs))
.ToArray();
private AquariumFrame CreateFrame()
{
var channelCount = Math.Max(1, lastAudioSpectra.Count);
@ -144,6 +192,9 @@ direct
: bufferFieldFrame.HasInput
? new Vector3(0.0f, 0.0f, 12.0f)
: new Vector3(0.0f, 0.0f, cameraZ + 12.0f);
var spectrumSplineFrame = ShouldRenderSpectrumPreviewSplines()
? BuildSpectrumSplineFrame()
: AquariumSplineFrame.Empty;
return new AquariumFrame(
new ViewFrame(Vector2.Zero, 24.0f),
cameraPosition,
@ -155,7 +206,7 @@ direct
TraceHeightFieldSurface = false,
UseStarfieldBackground = false,
BufferFieldFrame = bufferFieldFrame,
SplineFrame = ShouldRenderSpectrumPreviewSplines() ? BuildSpectrumSplineFrame() : AquariumSplineFrame.Empty,
SplineFrame = ComposeFensalirSplineFrame(spectrumSplineFrame, latestMoveProofSurface),
});
}
@ -166,6 +217,64 @@ direct
synchronization.AddSource(source);
}
public void PublishMoveProofSurface(MimirMoveProofSurfaceDocument surface)
{
ArgumentNullException.ThrowIfNull(surface);
latestMoveProofSurface = surface;
}
public void ClearMoveProofSurface()
{
latestMoveProofSurface = null;
}
public void RegisterMoveProofDriver(MimirMoveProofRuntimeDriver driver)
{
ArgumentNullException.ThrowIfNull(driver);
moveProofDrivers.Add(driver);
}
public void ActivateConfiguredMoveProofSources()
{
if (moveProofSourcesActivated)
{
return;
}
moveProofSourcesActivated = true;
foreach (var configuration in configuredMoveProofSources)
{
ActivateConfiguredMoveProofSource(configuration);
}
}
public static AquariumSplineFrame ComposeFensalirSplineFrame(
AquariumSplineFrame baseFrame,
MimirMoveProofSurfaceDocument? moveProofSurface)
{
ArgumentNullException.ThrowIfNull(baseFrame);
if (moveProofSurface is null)
{
return baseFrame;
}
var proofFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(moveProofSurface);
if (!baseFrame.HasInput)
{
return proofFrame;
}
if (!proofFrame.HasInput)
{
return baseFrame;
}
return new AquariumSplineFrame
{
Splines = baseFrame.Splines.Concat(proofFrame.Splines).ToArray()
};
}
public void Start()
{
Console.WriteLine($"Mimir runtime sync buffers: {synchronization.Summary()} @ {synchronization.Settings.BufferDuration.TotalSeconds:0.###}s audioSync={audioSyncSettings.Mode} reference={audioSyncSettings.ReferenceSourceId}");
@ -181,6 +290,7 @@ direct
runtimeSeconds += Math.Max(deltaSeconds, 0.0f);
lastPollCount = synchronization.PollSources();
UpdateMoveProofSurfaces();
QueueCalibrationTimeline();
UpdateAudioSpectra();
UpdateAudioSynchronization();
@ -212,6 +322,11 @@ direct
public void Dispose()
{
foreach (var resource in moveProofDriverResources.AsEnumerable().Reverse())
{
resource.Dispose();
}
synchronization.Dispose();
}
@ -231,6 +346,10 @@ direct
panel.Readout("Audio sync state", DescribeAudioSyncState);
panel.Readout("Aligned audio", DescribeAlignedAudio);
panel.Readout("Chirplet reference", DescribeChirpletReference);
panel.Section("Move Proof");
panel.Readout("Latest proof", DescribeMoveProofSurface);
panel.Readout("Configured proof", DescribeConfiguredMoveProofSources);
panel.Readout("Proof drivers", DescribeMoveProofDrivers);
panel.Section("Live FFT");
panel.Readout("Spectrum cadence", () => sceneReady
? $"{lastAudioSpectra.Count} spectra fft={lastAudioSpectra.FirstOrDefault()?.FftSize ?? 0} analyze={lastSpectrumAnalysisMilliseconds:0.00}ms"
@ -246,6 +365,77 @@ direct
});
}
private string DescribeMoveProofSurface()
{
var surface = latestMoveProofSurface;
if (surface is null)
{
return "No Move proof surface published.";
}
return $"{surface.ProofId} verdict={surface.Verdict} confidence={surface.PoseConfidence:0.000} latency={surface.LatencyMilliseconds:0.00}ms chain={surface.MuninnEvidenceFrameId}->{surface.MimirEvidenceFrameId}->{surface.MimirPoseFrameId}->{surface.FensalirFrameId}";
}
private string DescribeMoveProofDrivers()
{
if (moveProofDrivers.Count == 0)
{
return "No Move proof runtime drivers registered.";
}
return string.Join(Environment.NewLine, moveProofDrivers.Select(driver => driver.Describe()));
}
private string DescribeConfiguredMoveProofSources()
{
if (configuredMoveProofSources.Count == 0)
{
return "No Move proof sources configured.";
}
return string.Join(
Environment.NewLine,
configuredMoveProofSources.Select(source =>
$"{source.EvidenceStreamId} -> {source.MimirPoseFramePrefix}:<sequence> cameras={source.Calibration.Cameras.Count} {DescribeMoveProofActivation(source.EvidenceStreamId)}"));
}
private string DescribeMoveProofActivation(string evidenceStreamId)
{
var status = moveProofActivationStatuses.LastOrDefault(status =>
string.Equals(status.EvidenceStreamId, evidenceStreamId, StringComparison.Ordinal));
if (status is null)
{
return "not activated";
}
return status.Active ? "active" : $"inactive: {status.Diagnostic}";
}
private void UpdateMoveProofSurfaces()
{
if (moveProofDrivers.Count == 0)
{
return;
}
var presentedAtNs = NowUnixNs();
foreach (var driver in moveProofDrivers)
{
if (driver.TryBuildLatest(presentedAtNs, out var result) && result is not null)
{
PublishMoveProofSurface(result.ProofSurface);
}
}
}
private static ulong NowUnixNs()
{
var now = DateTimeOffset.UtcNow;
var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
private static AquariumRenderPlan CreateRenderPlan()
{
var app = new AquariumApp();
@ -437,6 +627,65 @@ direct
synchronization.AddSource(source);
}
}
ActivateConfiguredMoveProofSources();
}
private void ActivateConfiguredMoveProofSource(MimirMoveProofRuntimeConfiguration configuration)
{
var providerKind = moveProofRingProvider.GetType().Name;
try
{
var errors = configuration.Validate();
if (errors.Length > 0)
{
RecordMoveProofActivation(configuration.EvidenceStreamId, false, $"invalid: {string.Join("; ", errors)}", providerKind, driverRegistered: false);
return;
}
if (!moveProofRingProvider.TryOpenEvidenceRing(configuration, out var ringLease, out var diagnostic) || ringLease is null)
{
RecordMoveProofActivation(configuration.EvidenceStreamId, false, diagnostic, providerKind, driverRegistered: false);
return;
}
MimirNativeReservoirRuntime? reservoir = null;
try
{
reservoir = new MimirNativeReservoirRuntime(configuration.NativeReservoirPath);
var driver = configuration.CreateDriver(ringLease.Ring, reservoir);
moveProofDrivers.Add(driver);
moveProofDriverResources.Add(reservoir);
moveProofDriverResources.Add(ringLease);
reservoir = null;
ringLease = null;
RecordMoveProofActivation(configuration.EvidenceStreamId, true, diagnostic, providerKind, driverRegistered: true);
}
finally
{
reservoir?.Dispose();
ringLease?.Dispose();
}
}
catch (Exception exception)
{
RecordMoveProofActivation(configuration.EvidenceStreamId, false, exception.Message, providerKind, driverRegistered: false);
}
}
private void RecordMoveProofActivation(
string evidenceStreamId,
bool active,
string diagnostic,
string providerKind,
bool driverRegistered)
{
moveProofActivationStatuses.Add(new MimirMoveProofRuntimeActivationStatus(
evidenceStreamId,
active,
string.IsNullOrWhiteSpace(diagnostic) ? (active ? "activated" : "not activated") : diagnostic,
providerKind,
driverRegistered));
}
private void EmitTelemetry()
@ -862,6 +1111,11 @@ direct
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {block.Channels}ch {block.SampleRate}Hz {block.SampleFormat} frames {block.FrameCount} bytes {latest.Value.ByteLength} edge {buffer.EdgeNs}";
}
if (latest?.TrackingObservation is { } tracking)
{
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {tracking.Kind} device={tracking.DeviceId} producer={tracking.ProducerId} host={tracking.HostId} confidence={tracking.Confidence:0.00} edge {buffer.EdgeNs}";
}
return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} edge {buffer.EdgeNs}";
}

View file

@ -11,7 +11,8 @@ public enum MimirWorkloadKind
CameraCapture,
VisualFeatureExtraction,
VisualFusion,
ObsPublication
ProgramComposition,
ProgramPublication
}
public sealed record MimirWorkloadPlacement(
@ -42,7 +43,8 @@ public static class MimirComputeOffloadConfigurations
new(MimirWorkloadKind.CameraCapture, MimirComputeBackend.NativeSimd, "native capture workers", CanRunRemote: true, RequiresRealtime: true, "Drivers own device reads and timestamps."),
new(MimirWorkloadKind.VisualFeatureExtraction, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "GPU owns dense feature work."),
new(MimirWorkloadKind.VisualFusion, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "Fensalir owns temporal evidence field."),
new(MimirWorkloadKind.ObsPublication, MimirComputeBackend.D3D12Compute, "OBS/Fensalir output", CanRunRemote: false, RequiresRealtime: true, "OBS receives program surfaces, not sync authority.")
new(MimirWorkloadKind.ProgramComposition, MimirComputeBackend.D3D12Compute, "Mimir + Fensalir", CanRunRemote: false, RequiresRealtime: true, "Mimir owns the scene graph; Fensalir lowers it into program pixels."),
new(MimirWorkloadKind.ProgramPublication, MimirComputeBackend.D3D12Compute, "Mimir + Yggdrasil publisher", CanRunRemote: false, RequiresRealtime: true, "The site publisher consumes the Mimir program output; OBS is only an optional compatibility sink.")
]);
public static MimirComputeOffloadConfiguration DistributedWitnesses { get; } = new(

View file

@ -104,6 +104,103 @@ public sealed record MimirFaustControlSnapshot(
[property: Key(0)] string Path,
[property: Key(1)] float Value);
[CultDocument("mimir.program_scene", "mimir.program_scene.v1")]
[MessagePackObject]
public sealed record MimirProgramSceneDocument(
[property: Key(0)]
[property: CultName]
string SceneId,
[property: Key(1)] string UpdatedAtUtc,
[property: Key(2)] int CanvasWidth,
[property: Key(3)] int CanvasHeight,
[property: Key(4)] string Owner,
[property: Key(5)] MimirProgramSceneLayer[] Layers);
[MessagePackObject]
public sealed record MimirProgramSceneLayer(
[property: Key(0)] string LayerId,
[property: Key(1)] string SourceRef,
[property: Key(2)] string SourceKind,
[property: Key(3)] bool Visible,
[property: Key(4)] double X,
[property: Key(5)] double Y,
[property: Key(6)] double Width,
[property: Key(7)] double Height,
[property: Key(8)] int ZIndex,
[property: Key(9)] MimirProgramCrop Crop,
[property: Key(10)] MimirProgramChromaKey? ChromaKey);
[MessagePackObject]
public sealed record MimirProgramCrop(
[property: Key(0)] double Left,
[property: Key(1)] double Top,
[property: Key(2)] double Right,
[property: Key(3)] double Bottom);
[MessagePackObject]
public sealed record MimirProgramChromaKey(
[property: Key(0)] uint KeyColorRgba,
[property: Key(1)] double Similarity,
[property: Key(2)] double Smoothness,
[property: Key(3)] double Spill);
[CultDocument("mimir.program_output", "mimir.program_output.v1")]
[MessagePackObject]
public sealed record MimirProgramOutputDocument(
[property: Key(0)]
[property: CultName]
string OutputId,
[property: Key(1)] string UpdatedAtUtc,
[property: Key(2)] string SceneId,
[property: Key(3)] string VideoSurface,
[property: Key(4)] string AudioBus,
[property: Key(5)] string PublisherRoute,
[property: Key(6)] bool DiagnosticOnly);
[CultDocument("mimir.eve_sensor_observation", "mimir.eve_sensor_observation.v1")]
[MessagePackObject]
public sealed record MimirEveSensorObservationDocument(
[property: Key(0)]
[property: CultName]
string ObservationId,
[property: Key(1)] string DeviceId,
[property: Key(2)] string StreamId,
[property: Key(3)] string Kind,
[property: Key(4)] long Sequence,
[property: Key(5)] long SensorTimestampNs,
[property: Key(6)] long ElapsedRealtimeNs,
[property: Key(7)] string WallClockUtc,
[property: Key(8)] string ClockDomainId,
[property: Key(9)] double[] Values,
[property: Key(10)] string? Action,
[property: Key(11)] int? PointerCount,
[property: Key(12)] double? X,
[property: Key(13)] double? Y,
[property: Key(14)] int? Accuracy);
[CultDocument("mimir.eve_media_observation", "mimir.eve_media_observation.v1")]
[MessagePackObject]
public sealed record MimirEveMediaObservationDocument(
[property: Key(0)]
[property: CultName]
string ObservationId,
[property: Key(1)] string DeviceId,
[property: Key(2)] string StreamId,
[property: Key(3)] string Kind,
[property: Key(4)] long Sequence,
[property: Key(5)] long SensorTimestampNs,
[property: Key(6)] long ElapsedRealtimeNs,
[property: Key(7)] string WallClockUtc,
[property: Key(8)] string ClockDomainId,
[property: Key(9)] string Format,
[property: Key(10)] int? Width,
[property: Key(11)] int? Height,
[property: Key(12)] int? SampleRate,
[property: Key(13)] int? Channels,
[property: Key(14)] int? FrameCount,
[property: Key(15)] string PayloadEncoding,
[property: Key(16)] byte[] Payload);
public static class MimirCultMeshContractFactory
{
public static MimirBioacousticCodebookState CreateCodebookState(
@ -175,6 +272,69 @@ public static class MimirCultMeshContractFactory
.Select(pair => new MimirFaustControlSnapshot(pair.Key, pair.Value))
.ToArray());
public static MimirProgramSceneDocument CreateObsSceneMirror(
string sceneId,
DateTimeOffset? updatedAt = null) =>
new(
sceneId,
(updatedAt ?? DateTimeOffset.UtcNow).ToString("O"),
CanvasWidth: 1920,
CanvasHeight: 1080,
Owner: "Mimir",
Layers:
[
new(
"starfire-monitor",
"muninn:starfire:monitor:primary",
"monitor",
Visible: true,
X: 0,
Y: 0,
Width: 1920,
Height: 1080,
ZIndex: 0,
new MimirProgramCrop(Left: 0, Top: 0, Right: 0, Bottom: 48),
ChromaKey: null),
new(
"discord-chat",
"muninn:starfire:window:discord",
"window",
Visible: true,
X: 1225.5,
Y: 0.5,
Width: 1012.2,
Height: 569.4,
ZIndex: 10,
new MimirProgramCrop(Left: 350, Top: 38, Right: 271, Bottom: 87),
new MimirProgramChromaKey(0xff2b7c5a, Similarity: 1, Smoothness: 2, Spill: 1)),
new(
"raven-monitor",
"muninn:raven:monitor:primary",
"network-monitor",
Visible: true,
X: 0,
Y: 0,
Width: 2560,
Height: 1440,
ZIndex: 20,
new MimirProgramCrop(Left: 0, Top: 0, Right: 0, Bottom: 0),
ChromaKey: null)
]);
public static MimirProgramOutputDocument CreateProgramOutput(
string outputId,
string sceneId,
MimirProgramPublicationConfiguration publication,
DateTimeOffset? updatedAt = null) =>
new(
outputId,
(updatedAt ?? DateTimeOffset.UtcNow).ToString("O"),
sceneId,
publication.VideoSurfaceName,
publication.AudioKind.ToString(),
publication.SitePublisherRoute,
publication.DiagnosticOnly);
private static MimirBioacousticDecoderConfigurationSnapshot Snapshot(MimirBioacousticDecoderConfiguration configuration) =>
new(
configuration.Id,

View file

@ -0,0 +1,24 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[CultDocument("mimir.cultmesh_media_frame", "mimir.cultmesh_media_frame.v1")]
[MessagePackObject]
public sealed record MimirCultMeshMediaFrameDocument(
[property: Key(0)]
[property: CultName]
string FrameId,
[property: Key(1)] string StreamId,
[property: Key(2)] long Sequence,
[property: Key(3)] string ProducedAtUtc,
[property: Key(4)] long TimestampNanoseconds,
[property: Key(5)] string PayloadKind,
[property: Key(6)] string Container,
[property: Key(7)] string VideoCodec,
[property: Key(8)] string AudioCodec,
[property: Key(9)] int PayloadBytes,
[property: Key(10)] byte[] Payload,
[property: Key(11)] string ProducerNode,
[property: Key(12)] string ClockDomainId,
[property: Key(13)] string[] Tags);

View file

@ -47,9 +47,22 @@ public static class MimirModuleLibrary
new(
"cultmesh-contracts",
"Mimir.Runtime",
"Typed state documents for codebooks, decoder state, acoustic path state, and actuator state.",
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state"],
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]),
"Typed state documents and stream frames for codebooks, decoder state, acoustic path state, actuator state, program state, Move evidence, and Mimir-fused Move controller poses.",
["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation", "mimir.move_controller_pose", "mimir.move_controller_pose_stream_frame"],
[
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke",
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke"
]),
new(
"move-controller-fusion",
"Mimir.Runtime + Fensalir",
"Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses, then publishes them as a realtime CultMesh pose stream for interaction consumers.",
["mimir.move_calibration_protocol.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.quest_access.v1", "muninn.quest_pose_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"],
[
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke",
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke",
"dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke"
]),
new(
"path-learning",
"Mimir.Runtime",
@ -111,10 +124,10 @@ public static class MimirModuleLibrary
MimirMachineAssemblyPlans.BuiltIn.Select(plan => plan.Id).ToArray(),
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]),
new(
"obs-publication",
"Fensalir + Faust + OBS",
"OBS-facing program surface and separately mixable audio stem configurations.",
MimirObsPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(),
"program-publication",
"Mimir + Fensalir + Faust + Eve + Yggdrasil",
"Mimir-owned program composition, Eve operator surfaces, Yggdrasil site publication, and temporary OBS compatibility adapters.",
MimirProgramPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(),
["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"])
];
}

View file

@ -0,0 +1,272 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[CultDocument("mimir.move_calibration_protocol", "mimir.move_calibration_protocol.v1")]
[MessagePackObject]
public sealed record MimirMoveCalibrationProtocolDocument(
[property: Key(0)]
[property: CultName]
string ProtocolId,
[property: Key(1)] string CreatedAtUtc,
[property: Key(2)] string TrackingSpaceId,
[property: Key(3)] string FusionAuthorityId,
[property: Key(4)] MimirMoveCalibrationStreamRequirement[] StreamRequirements,
[property: Key(5)] MimirMoveCalibrationPhase[] Phases,
[property: Key(6)] MimirMoveCalibrationOutput[] Outputs,
[property: Key(7)] MimirMoveCalibrationAcceptance Acceptance,
[property: Key(8)] string[] OperatorNotes);
[MessagePackObject]
public sealed record MimirMoveCalibrationStreamRequirement(
[property: Key(0)] string StreamId,
[property: Key(1)] string HostId,
[property: Key(2)] string Owner,
[property: Key(3)] string Kind,
[property: Key(4)] string Transport,
[property: Key(5)] bool Required,
[property: Key(6)] string Evidence);
[MessagePackObject]
public sealed record MimirMoveCalibrationPhase(
[property: Key(0)] string PhaseId,
[property: Key(1)] string Title,
[property: Key(2)] double DurationSeconds,
[property: Key(3)] string OperatorCue,
[property: Key(4)] string[] RequiredStreamIds,
[property: Key(5)] string[] Produces);
[MessagePackObject]
public sealed record MimirMoveCalibrationOutput(
[property: Key(0)] string OutputId,
[property: Key(1)] string SchemaId,
[property: Key(2)] string Owner,
[property: Key(3)] string Purpose);
[MessagePackObject]
public sealed record MimirMoveCalibrationAcceptance(
[property: Key(0)] int MinimumControllerStateSamplesPerMove,
[property: Key(1)] int MinimumOpticalSamplesPerCamera,
[property: Key(2)] int MinimumExternalPoseSamples,
[property: Key(3)] double MaximumClockSkewMilliseconds,
[property: Key(4)] double MaximumStillGyroRadiansPerSecond,
[property: Key(5)] double MinimumTriangulatedPoseConfidence,
[property: Key(6)] string[] RequiredDerivedOutputs);
public static class MimirMoveCalibrationProtocol
{
public static MimirMoveCalibrationProtocolDocument CreateNightwingFourWandProtocol(
DateTimeOffset? createdAt = null) =>
new(
ProtocolId: "mimir-move-calibration-nightwing-four-wand-v1",
CreatedAtUtc: (createdAt ?? DateTimeOffset.UtcNow).ToString("O"),
TrackingSpaceId: "mimir-stage-space",
FusionAuthorityId: "mimir.runtime.move-fusion",
StreamRequirements:
[
new(
"muninn:nightwing:move-evidence",
"nightwing",
"Muninn on Nightwing",
"Move optical marker candidates plus USB controller IMU/buttons",
"CultMesh bytes stream over the Verse; shared-memory on Nightwing, network bridge to Starfire",
Required: true,
"Nightwing owns its USB-attached Move and local camera witnesses."),
new(
"mimir:starfire:sensor-calibration-session",
"starfire",
"Mimir.Runtime",
"Bounded calibration lifecycle, coverage, fit, validation, and promotion state",
"Typed CultMesh state with CultCache persistence",
Required: true,
"Mimir owns the calibration task and its completion verdict."),
new(
"mimir:starfire:move-controller-poses",
"starfire",
"Mimir.Runtime",
"Mimir-fused controller pose output",
"CultMesh shared-memory bytes frame",
Required: true,
"The calibration run must prove Mimir can republish resolved poses."),
new(
"quest:usb:headset-pose",
"starfire",
"Quest USB/OpenXR witness",
"Quest headset pose near the Move/controllers calibration cluster",
"USB ADB/OpenXR witness bridged into Mimir calibration capture",
Required: false,
"External tracked VR frame for validating Mimir's optical/IMU pose frame."),
new(
"quest:usb:left-controller-pose",
"starfire",
"Quest USB/OpenXR witness",
"Quest left controller pose",
"USB ADB/OpenXR witness bridged into Mimir calibration capture",
Required: false,
"External tracked controller reference sitting beside the Move."),
new(
"quest:usb:right-controller-pose",
"starfire",
"Quest USB/OpenXR witness",
"Quest right controller pose",
"USB ADB/OpenXR witness bridged into Mimir calibration capture",
Required: false,
"External tracked controller reference sitting beside the Move."),
new(
"starfire:scarlett:loopback",
"starfire",
"Mimir.Runtime ASIO source",
"audio loopback timing witness",
"in-process ASIO into Mimir rolling buffers",
Required: false,
"Optional calibration metronome/cue timing witness; not a Move pose owner.")
],
Phases:
[
new(
"preflight-streams",
"Stream liveness and clock edge check",
10.0,
"Hold the four Moves visible. Confirm Nightwing evidence and stable identities are fresh.",
["muninn:nightwing:move-evidence"],
["stream-freshness", "clock-skew-estimate", "controller-id-map"]),
new(
"dark-stillness",
"IMU bias with orbs off",
8.0,
"Set Move lights off. Place both Moves still on the calibration surface.",
["muninn:nightwing:move-evidence"],
["gyro-bias", "accelerometer-gravity-vector", "magnetometer-baseline"]),
new(
"lit-stillness",
"Optical centroid and static gravity alignment",
8.0,
"Light one Move at a time, then both. Keep them still and visible to both camera sets.",
["muninn:nightwing:move-evidence"],
["camera-marker-centroid-stability", "light-command-to-controller-association"]),
new(
"axis-sweeps",
"Slow single-axis rotations",
24.0,
"Sweep each Move slowly through pitch, yaw, and roll while keeping the orb visible.",
["muninn:nightwing:move-evidence"],
["gyro-scale-fit", "accelerometer-frame-fit", "optical-angular-consistency"]),
new(
"figure-eight",
"Magnetometer and cross-axis motion",
20.0,
"Move each controller through a broad figure-eight. Leave the Quest headset/controllers stationary as the external VR reference cluster.",
["muninn:nightwing:move-evidence"],
["magnetometer-hard-soft-iron-fit", "cross-axis-coupling-check"]),
new(
"quest-reference",
"Optional Quest headset/controller reference",
12.0,
"Keep the Move between the two Quest controllers in front of the headset. Capture the static Quest controller/headset frame while Mimir observes the Move.",
["quest:usb:headset-pose", "quest:usb:left-controller-pose", "quest:usb:right-controller-pose"],
["external-vr-pose-frame", "quest-to-mimir-frame-fit", "quest-clock-offset-estimate"]),
new(
"wand-volume-sweep",
"Four-wand optical volume coverage",
120.0,
"Wave all four uniquely identified wands throughout the shared camera volume until Mimir reports sufficient per-sensor coverage.",
["muninn:nightwing:move-evidence", "mimir:starfire:sensor-calibration-session"],
["sensor-grid-coverage", "same-frame-correspondences", "sphere-radius-range", "fit-and-held-out-partitions"]),
new(
"validation-pass",
"Held-out tracking validation",
15.0,
"Run free hand motion with both orbs visible. Mimir may publish validation poses, but not promote calibration if confidence fails.",
["muninn:nightwing:move-evidence", "mimir:starfire:move-controller-poses"],
["triangulated-position-residuals", "imu-prediction-residuals", "promotion-decision"])
],
Outputs:
[
new(
"move-rig-calibration",
"mimir.move_fusion_rig_calibration.v1",
"Mimir.Runtime",
"Camera intrinsics/extrinsics, clock association policy, and gyro unit scale for Move fusion."),
new(
"move-imu-calibration",
"mimir.move_imu_calibration.v1",
"Mimir.Runtime",
"Per-controller gyro bias/scale, accelerometer gravity frame, magnetometer correction, and noise estimates."),
new(
"move-controller-map",
"mimir.move_controller_identity_map.v1",
"Mimir.Runtime",
"Stable association between Muninn USB controller ids, light-command receipts, and optical orb identities."),
new(
"move-calibration-receipt",
"mimir.move_calibration_receipt.v1",
"Mimir.Runtime",
"Immutable receipt of captured sample counts, residuals, rejected phases, and promotion decision.")
],
Acceptance: new(
MinimumControllerStateSamplesPerMove: 600,
MinimumOpticalSamplesPerCamera: 300,
MinimumExternalPoseSamples: 120,
MaximumClockSkewMilliseconds: 20.0,
MaximumStillGyroRadiansPerSecond: 0.035,
MinimumTriangulatedPoseConfidence: 0.65,
RequiredDerivedOutputs:
[
"move-rig-calibration",
"move-imu-calibration",
"move-controller-map",
"move-calibration-receipt"
]),
OperatorNotes:
[
"Mimir owns fusion and calibration promotion. Muninn only publishes source-local evidence and executes local Move light commands.",
"CultCache receipts are durable evidence; CultMesh stream frames are the hot path.",
"Do not promote orientation until gyro bias/scale, gravity alignment, optical residuals, external Quest reference residuals when available, and held-out validation all pass."
]);
public static string[] Validate(MimirMoveCalibrationProtocolDocument protocol)
{
ArgumentNullException.ThrowIfNull(protocol);
var errors = new List<string>();
if (protocol.StreamRequirements.Count(stream => stream.Required) < 3)
{
errors.Add("required-streams-too-small");
}
if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "muninn:nightwing:move-evidence" && stream.Required))
{
errors.Add("missing-nightwing-muninn-move-evidence");
}
if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "mimir:starfire:sensor-calibration-session" && stream.Required))
{
errors.Add("missing-mimir-calibration-session");
}
if (!protocol.Phases.Any(phase => phase.PhaseId == "wand-volume-sweep"))
{
errors.Add("missing-wand-volume-sweep");
}
if (!protocol.Phases.Any(phase => phase.PhaseId == "dark-stillness"))
{
errors.Add("missing-imu-bias-phase");
}
if (!protocol.Phases.Any(phase => phase.Produces.Contains("gyro-scale-fit", StringComparer.Ordinal)))
{
errors.Add("missing-gyro-scale-fit");
}
foreach (var requiredOutput in protocol.Acceptance.RequiredDerivedOutputs)
{
if (!protocol.Outputs.Any(output => output.OutputId == requiredOutput))
{
errors.Add($"missing-output:{requiredOutput}");
}
}
return errors.ToArray();
}
}

View file

@ -0,0 +1,310 @@
using System.Numerics;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MimirMoveFusionCameraCalibration(
[property: Key(0)] string CameraId,
[property: Key(1)] ulong WitnessIdHash,
[property: Key(2)] MimirVector3Snapshot PositionMeters,
[property: Key(3)] MimirQuaternionSnapshot Orientation,
[property: Key(4)] double FocalLengthXPx,
[property: Key(5)] double FocalLengthYPx,
[property: Key(6)] double PrincipalPointXPx,
[property: Key(7)] double PrincipalPointYPx);
[MessagePackObject]
public sealed record MimirMoveFusionRigCalibration(
[property: Key(0)] string CalibrationId,
[property: Key(1)] string TrackingSpaceId,
[property: Key(2)] IReadOnlyList<MimirMoveFusionCameraCalibration> Cameras,
[property: Key(3)] double GyroUnitsPerRadianPerSecond = 1.0,
[property: Key(4)] double MaximumAssociationSkewMilliseconds = 20.0,
[property: Key(5)] double SingleRayFallbackDepthMeters = 1.5);
public sealed record MimirMoveFusionResult(
IReadOnlyList<MimirMoveControllerPoseDocument> Poses,
int ControllerEvidenceCount,
int OpticalEvidenceCount,
int CalibratedOpticalEvidenceCount);
public static class MimirMoveFusion
{
public static MimirMoveFusionResult Fuse(
IReadOnlyList<MimirNativeMoveEvidenceSample> samples,
MimirMoveFusionRigCalibration calibration,
string fusionAuthorityId = "mimir.runtime.move-fusion",
string consumerContract = "fensalir.move-controller-input")
{
ArgumentNullException.ThrowIfNull(samples);
ArgumentNullException.ThrowIfNull(calibration);
ArgumentException.ThrowIfNullOrWhiteSpace(calibration.CalibrationId);
ArgumentException.ThrowIfNullOrWhiteSpace(calibration.TrackingSpaceId);
var controllerStates = samples
.Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState &&
sample.ControllerIdHash != 0)
.OrderBy(sample => sample.SourceTimestampNs)
.ThenBy(sample => sample.Sequence)
.ToArray();
var opticalMarkers = samples
.Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker)
.OrderBy(sample => sample.SourceTimestampNs)
.ThenBy(sample => sample.Sequence)
.ToArray();
var cameras = calibration.Cameras
.Where(IsCameraUsable)
.ToDictionary(camera => camera.WitnessIdHash, camera => camera);
var maxSkewNs = (ulong)Math.Max(0.0, calibration.MaximumAssociationSkewMilliseconds * 1_000_000.0);
var poses = new List<MimirMoveControllerPoseDocument>();
foreach (var controller in controllerStates)
{
var associatedMarkers = opticalMarkers
.Where(marker => IsAssociated(marker, controller, controllerStates.Length, maxSkewNs))
.Select(marker => (Marker: marker, Camera: cameras.GetValueOrDefault(marker.WitnessIdHash)))
.Where(pair => pair.Camera is not null)
.ToArray();
if (associatedMarkers.Length == 0)
{
continue;
}
var rays = associatedMarkers
.Select(pair => BuildRay(pair.Camera!, pair.Marker))
.Where(ray => ray.HasValue)
.Select(ray => ray!.Value)
.ToArray();
if (rays.Length == 0)
{
continue;
}
var position = rays.Length >= 2
? ClosestPoint(rays)
: rays[0].Origin + rays[0].Direction * (float)Math.Max(0.05, calibration.SingleRayFallbackDepthMeters);
var opticalConfidence = Math.Clamp(
associatedMarkers.Average(pair => pair.Marker.Confidence) * (rays.Length >= 2 ? 1.0 : 0.35),
0.0,
1.0);
var confidence = Math.Clamp(opticalConfidence * ControllerConfidence(controller), 0.0, 1.0);
var sourceNs = Math.Max(controller.SourceTimestampNs, associatedMarkers.Max(pair => pair.Marker.SourceTimestampNs));
var arrivalNs = Math.Max(controller.ArrivalNs, associatedMarkers.Max(pair => pair.Marker.ArrivalNs));
var evidenceStreamIds = associatedMarkers
.Select(pair => $"witness:0x{pair.Marker.WitnessIdHash:X16}")
.Append($"controller:0x{controller.ControllerIdHash:X16}")
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var evidenceKinds = new[]
{
rays.Length >= 2 ? "optical-marker:triangulated" : "optical-marker:single-ray-depth-prior",
"controller-state:buttons-imu",
"orientation:imu-unresolved"
};
poses.Add(new MimirMoveControllerPoseDocument(
PoseId: $"move:0x{controller.ControllerIdHash:X16}:{controller.Sequence}",
WandId: $"move:0x{controller.ControllerIdHash:X16}",
TrackingSpaceId: calibration.TrackingSpaceId,
CalibrationId: calibration.CalibrationId,
FusionAuthorityId: fusionAuthorityId,
SourceTimestampNs: checked((long)Math.Min(sourceNs, (ulong)long.MaxValue)),
EstimatedAtNs: checked((long)Math.Min(arrivalNs, (ulong)long.MaxValue)),
Sequence: controller.Sequence,
PositionMeters: ToSnapshot(position),
Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0),
LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0),
AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(
ScaleGyro(controller.GyroX, calibration),
ScaleGyro(controller.GyroY, calibration),
ScaleGyro(controller.GyroZ, calibration)),
Confidence: confidence,
LatencyMilliseconds: arrivalNs >= sourceNs ? (arrivalNs - sourceNs) / 1_000_000.0 : 0.0,
Battery01: float.IsFinite(controller.Battery01) ? Math.Clamp(controller.Battery01, 0.0f, 1.0f) : double.NaN,
Buttons: ButtonsFromMask(controller.ButtonsMask, controller.Trigger),
EvidenceStreamIds: evidenceStreamIds,
EvidenceKinds: evidenceKinds,
ConsumerContract: consumerContract));
}
return new MimirMoveFusionResult(
poses,
controllerStates.Length,
opticalMarkers.Length,
opticalMarkers.Count(marker => cameras.ContainsKey(marker.WitnessIdHash)));
}
private static bool IsAssociated(
MimirNativeMoveEvidenceSample marker,
MimirNativeMoveEvidenceSample controller,
int controllerCount,
ulong maxSkewNs)
{
if (marker.ControllerIdHash != 0 && marker.ControllerIdHash != controller.ControllerIdHash)
{
return false;
}
if (marker.ControllerIdHash == 0 && controllerCount != 1)
{
return false;
}
return TimestampDelta(marker.SourceTimestampNs, controller.SourceTimestampNs) <= maxSkewNs;
}
private static bool IsCameraUsable(MimirMoveFusionCameraCalibration camera) =>
camera.WitnessIdHash != 0 &&
camera.FocalLengthXPx > 0.0 &&
camera.FocalLengthYPx > 0.0;
private static MimirRay? BuildRay(MimirMoveFusionCameraCalibration camera, MimirNativeMoveEvidenceSample marker)
{
if (!float.IsFinite(marker.ImageX) || !float.IsFinite(marker.ImageY))
{
return null;
}
var directionCamera = Vector3.Normalize(new Vector3(
(float)((marker.ImageX - camera.PrincipalPointXPx) / camera.FocalLengthXPx),
(float)((camera.PrincipalPointYPx - marker.ImageY) / camera.FocalLengthYPx),
1.0f));
var orientation = ToQuaternion(camera.Orientation);
return new MimirRay(
ToVector(camera.PositionMeters),
Vector3.Normalize(Vector3.Transform(directionCamera, orientation)));
}
private static Vector3 ClosestPoint(IReadOnlyList<MimirRay> rays)
{
var matrix = new Matrix3();
var rhs = Vector3.Zero;
foreach (var ray in rays)
{
var d = Vector3.Normalize(ray.Direction);
var xx = 1.0f - d.X * d.X;
var yy = 1.0f - d.Y * d.Y;
var zz = 1.0f - d.Z * d.Z;
var xy = -d.X * d.Y;
var xz = -d.X * d.Z;
var yz = -d.Y * d.Z;
matrix.M11 += xx;
matrix.M12 += xy;
matrix.M13 += xz;
matrix.M21 += xy;
matrix.M22 += yy;
matrix.M23 += yz;
matrix.M31 += xz;
matrix.M32 += yz;
matrix.M33 += zz;
rhs += new Vector3(
xx * ray.Origin.X + xy * ray.Origin.Y + xz * ray.Origin.Z,
xy * ray.Origin.X + yy * ray.Origin.Y + yz * ray.Origin.Z,
xz * ray.Origin.X + yz * ray.Origin.Y + zz * ray.Origin.Z);
}
return matrix.TrySolve(rhs, out var solved)
? solved
: rays[0].Origin + rays[0].Direction;
}
private static double ControllerConfidence(MimirNativeMoveEvidenceSample controller) =>
Math.Clamp(
0.75 +
(float.IsFinite(controller.Battery01) ? 0.10 : 0.0) +
(controller.ButtonsMask != 0 ? 0.05 : 0.0) +
(Math.Abs(controller.AccelX) + Math.Abs(controller.AccelY) + Math.Abs(controller.AccelZ) > 0.001 ? 0.10 : 0.0),
0.0,
1.0);
private static MimirTrackingButtonSnapshot[] ButtonsFromMask(uint mask, float trigger)
{
var names = new[]
{
"select", "l3", "r3", "start", "up", "right", "down", "left",
"l2", "r2", "l1", "r1", "triangle", "circle", "cross", "square",
"ps", "move", "trigger"
};
var buttons = new List<MimirTrackingButtonSnapshot>();
for (var bit = 0; bit < names.Length; bit++)
{
var pressed = (mask & (1u << bit)) != 0;
if (pressed || names[bit] == "trigger")
{
buttons.Add(new MimirTrackingButtonSnapshot(
names[bit],
pressed,
names[bit] == "trigger" ? Math.Clamp(trigger, 0.0f, 1.0f) : pressed ? 1.0 : 0.0));
}
}
return buttons.ToArray();
}
private static ulong TimestampDelta(ulong left, ulong right) => left >= right ? left - right : right - left;
private static double ScaleGyro(float value, MimirMoveFusionRigCalibration calibration) =>
calibration.GyroUnitsPerRadianPerSecond <= 0.0
? 0.0
: value / calibration.GyroUnitsPerRadianPerSecond;
private static Vector3 ToVector(MimirVector3Snapshot value) => new((float)value.X, (float)value.Y, (float)value.Z);
private static MimirVector3Snapshot ToSnapshot(Vector3 value) => new(value.X, value.Y, value.Z);
private static Quaternion ToQuaternion(MimirQuaternionSnapshot value)
{
var q = new Quaternion((float)value.X, (float)value.Y, (float)value.Z, (float)value.W);
return q.LengthSquared() <= 1.0e-12f ? Quaternion.Identity : Quaternion.Normalize(q);
}
private readonly record struct MimirRay(Vector3 Origin, Vector3 Direction);
private struct Matrix3
{
public float M11;
public float M12;
public float M13;
public float M21;
public float M22;
public float M23;
public float M31;
public float M32;
public float M33;
public readonly bool TrySolve(Vector3 rhs, out Vector3 value)
{
var det = Determinant(
M11, M12, M13,
M21, M22, M23,
M31, M32, M33);
if (Math.Abs(det) <= 1.0e-8f)
{
value = default;
return false;
}
value = new Vector3(
Determinant(rhs.X, M12, M13, rhs.Y, M22, M23, rhs.Z, M32, M33) / det,
Determinant(M11, rhs.X, M13, M21, rhs.Y, M23, M31, rhs.Z, M33) / det,
Determinant(M11, M12, rhs.X, M21, M22, rhs.Y, M31, M32, rhs.Z) / det);
return true;
}
private static float Determinant(
float m11,
float m12,
float m13,
float m21,
float m22,
float m23,
float m31,
float m32,
float m33) =>
m11 * (m22 * m33 - m23 * m32) -
m12 * (m21 * m33 - m23 * m31) +
m13 * (m21 * m32 - m22 * m31);
}
}

View file

@ -0,0 +1,71 @@
using GameCult.Mesh;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MimirMoveControllerPoseStreamFrame(
[property: Key(0)] string FrameId,
[property: Key(1)] string ProducerPeerId,
[property: Key(2)] long PublishedAtNs,
[property: Key(3)] string TrackingSpaceId,
[property: Key(4)] string CalibrationId,
[property: Key(5)] MimirMoveControllerPoseDocument[] Poses);
public static class MimirMovePoseStream
{
public const string StreamMetadataSchemaId = "mimir.move_controller_pose_stream_frame.v1";
public static CultMeshStreamDescriptor CreateStreamDescriptor(
string streamId,
string verseId,
string ownerPeerId,
string clockDomainId,
string trackingSpaceId,
string calibrationId) =>
new(
streamId,
verseId,
ownerPeerId,
CultMeshStreamKind.Bytes,
new CultMeshStreamClock(clockDomainId, sourceId: streamId, confidence: 1.0, evidenceKind: "mimir-move-controller-pose"),
[CultMeshStreamBodyTransport.SharedMemory, CultMeshStreamBodyTransport.CultCachePage],
label: $"Mimir Move controller poses ({trackingSpaceId}, {calibrationId})",
requiredAccess: CultMeshStreamAccess.Read,
maxInFlightFrames: 4,
metadataSchemaId: StreamMetadataSchemaId);
public static MimirMoveControllerPoseStreamFrame CreateFrame(
string frameId,
string producerPeerId,
long publishedAtNs,
string trackingSpaceId,
string calibrationId,
IEnumerable<MimirMoveControllerPoseDocument> poses)
{
ArgumentException.ThrowIfNullOrWhiteSpace(frameId);
ArgumentException.ThrowIfNullOrWhiteSpace(producerPeerId);
ArgumentException.ThrowIfNullOrWhiteSpace(trackingSpaceId);
ArgumentException.ThrowIfNullOrWhiteSpace(calibrationId);
ArgumentNullException.ThrowIfNull(poses);
return new MimirMoveControllerPoseStreamFrame(
frameId,
producerPeerId,
publishedAtNs,
trackingSpaceId,
calibrationId,
poses
.Where(pose => string.Equals(pose.TrackingSpaceId, trackingSpaceId, StringComparison.Ordinal) &&
string.Equals(pose.CalibrationId, calibrationId, StringComparison.Ordinal))
.OrderBy(pose => pose.SourceTimestampNs)
.ThenBy(pose => pose.Sequence)
.ToArray());
}
public static byte[] SerializeFrame(MimirMoveControllerPoseStreamFrame frame) =>
MessagePackSerializer.Serialize(frame);
public static MimirMoveControllerPoseStreamFrame DeserializeFrame(ReadOnlyMemory<byte> bytes) =>
MessagePackSerializer.Deserialize<MimirMoveControllerPoseStreamFrame>(bytes);
}

View file

@ -0,0 +1,97 @@
namespace Mimir.Runtime.Synchronization;
public static class MimirMoveProofDevSurface
{
public const string EnableEnvironmentVariable = "MIMIR_MOVE_PROOF_DEV_SURFACE";
public static bool IsEnabled() =>
IsTruthy(Environment.GetEnvironmentVariable(EnableEnvironmentVariable));
public static MimirMoveProofSurfaceDocument Create(int sequence = 79)
{
var sourceTimestampNs = 1_781_202_600_004_000_000L + sequence;
var pose = new MimirMoveControllerPoseDocument(
PoseId: $"move:0xA11CE:{sequence}",
WandId: "move:0xA11CE",
TrackingSpaceId: "mimir-stage-space",
CalibrationId: "mimir-move-stage-calibration-v1",
FusionAuthorityId: "mimir.runtime.move-fusion",
SourceTimestampNs: sourceTimestampNs,
EstimatedAtNs: sourceTimestampNs + 2_200_000L,
Sequence: (ulong)sequence,
PositionMeters: new MimirVector3Snapshot(0.0, 0.0, 4.0),
Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0),
LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0),
AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.02, -0.01, 0.04),
Confidence: 0.82,
LatencyMilliseconds: 2.2,
Battery01: 0.72,
Buttons:
[
new MimirTrackingButtonSnapshot("move", true, 1.0),
new MimirTrackingButtonSnapshot("trigger", true, 0.5)
],
EvidenceStreamIds:
[
"witness:0x00000000000A11CE",
"witness:0x0000000000000B0B",
"controller:0x00000000000A11CE"
],
EvidenceKinds:
[
"optical-marker:triangulated",
"controller-state:buttons-imu",
"orientation:imu-unresolved"
],
ConsumerContract: "fensalir.move-controller-input");
var admission = new MimirMuninnMoveEvidenceAdmission(
FrameId: $"muninn:nightwing:move-evidence:{sequence}",
ProducerPeerId: "muninn:nightwing",
PublishedAtNs: sourceTimestampNs + 1_000_000L,
MimirReadAtNs: checked((ulong)(sourceTimestampNs + 1_700_000L)),
SampleCount: 3,
OpticalMarkerCount: 2,
ControllerStateCount: 1,
SourceTimeMinNs: checked((ulong)sourceTimestampNs),
SourceTimeMaxNs: checked((ulong)sourceTimestampNs),
ArrivalMinNs: checked((ulong)(sourceTimestampNs + 500_000L)),
ArrivalMaxNs: checked((ulong)(sourceTimestampNs + 1_700_000L)),
Handle: new MimirNativeSampleHandle(
SensorIdHash: 0xF00D,
TimestampNs: checked((ulong)sourceTimestampNs),
ArrivalNs: checked((ulong)(sourceTimestampNs + 1_700_000L)),
Sequence: (ulong)sequence,
PayloadHandle: 0xBEEF,
Flags: 0,
Reserved: 0),
ReservoirEdgeNs: checked((ulong)sourceTimestampNs),
ReservoirWindowStartNs: checked((ulong)(sourceTimestampNs - 5_000_000_000L)),
ReservoirMoveEvidenceCount: 1);
var poseFrame = MimirMovePoseStream.CreateFrame(
frameId: $"mimir:starfire:move-pose:{sequence}",
producerPeerId: "mimir:starfire",
publishedAtNs: pose.EstimatedAtNs,
trackingSpaceId: pose.TrackingSpaceId,
calibrationId: pose.CalibrationId,
poses: [pose]);
return MimirMoveProofSurface.Create(
admission,
poseFrame,
$"fensalir:starfire:presented-frame:{sequence}",
fensalirPresentedAtNs: checked((ulong)(sourceTimestampNs + 3_100_000L)));
}
private static bool IsTruthy(string? value)
{
if (string.IsNullOrWhiteSpace(value))
{
return false;
}
return value.Equals("1", StringComparison.OrdinalIgnoreCase)
|| value.Equals("true", StringComparison.OrdinalIgnoreCase)
|| value.Equals("yes", StringComparison.OrdinalIgnoreCase)
|| value.Equals("on", StringComparison.OrdinalIgnoreCase);
}
}

View file

@ -0,0 +1,130 @@
using Aquarium.Engine.Render;
using GameCult.Mesh;
namespace Mimir.Runtime.Synchronization;
public sealed record MimirMoveProofPipelineResult(
MimirMuninnMoveEvidenceAdmission Admission,
MimirMoveFusionResult Fusion,
MimirMoveControllerPoseStreamFrame PoseFrame,
MimirMoveProofSurfaceDocument ProofSurface,
AquariumSplineFrame FensalirProbeFrame);
public sealed record MimirMoveProofPipelineOptions(
string MimirEvidenceSourceId,
string MimirEvidenceFrameId,
string MimirPoseFrameId,
string MimirPoseProducerPeerId,
string FensalirFrameId,
ulong FensalirPresentedAtNs,
string FusionAuthorityId = "mimir.runtime.move-fusion",
string ConsumerContract = "fensalir.move-controller-input");
public static class MimirMoveProofPipeline
{
public static bool TryBuildLatestFromRing(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime runtime,
MimirMoveFusionRigCalibration calibration,
MimirMoveProofPipelineOptions options,
out MimirMoveProofPipelineResult? result)
{
ArgumentNullException.ThrowIfNull(ring);
result = null;
if (!ring.TryAcquireLatestRead(out var lease))
{
return false;
}
using (lease)
{
var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]);
return TryBuild(
frame,
runtime,
calibration,
options,
out result);
}
}
public static bool TryBuild(
MimirMuninnMoveEvidenceStreamFrame frame,
MimirNativeReservoirRuntime runtime,
MimirMoveFusionRigCalibration calibration,
MimirMoveProofPipelineOptions options,
out MimirMoveProofPipelineResult? result)
{
ArgumentNullException.ThrowIfNull(frame);
ArgumentNullException.ThrowIfNull(runtime);
ArgumentNullException.ThrowIfNull(calibration);
ArgumentNullException.ThrowIfNull(options);
result = null;
var mimirReadAtNs = NowUnixNs();
var samples = MimirMuninnMoveEvidenceAdapter.BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates);
if (samples.Count == 0)
{
return false;
}
var handle = runtime.AdmitMoveEvidence(
options.MimirEvidenceSourceId,
samples,
calibration.CalibrationId,
calibration.TrackingSpaceId);
var status = runtime.Status;
var admission = new MimirMuninnMoveEvidenceAdmission(
FrameId: frame.FrameId,
ProducerPeerId: frame.ProducerPeerId,
PublishedAtNs: frame.PublishedAtNs,
MimirReadAtNs: mimirReadAtNs,
SampleCount: samples.Count,
OpticalMarkerCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker),
ControllerStateCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState),
SourceTimeMinNs: samples.Min(sample => sample.SourceTimestampNs),
SourceTimeMaxNs: samples.Max(sample => sample.SourceTimestampNs),
ArrivalMinNs: samples.Min(sample => sample.ArrivalNs),
ArrivalMaxNs: samples.Max(sample => sample.ArrivalNs),
Handle: handle,
ReservoirEdgeNs: status.EdgeNs,
ReservoirWindowStartNs: status.WindowStartNs,
ReservoirMoveEvidenceCount: status.MoveEvidenceCount.ToUInt64());
var fusion = MimirMoveFusion.Fuse(
samples,
calibration,
options.FusionAuthorityId,
options.ConsumerContract);
var poseFrame = MimirMovePoseStream.CreateFrame(
options.MimirPoseFrameId,
options.MimirPoseProducerPeerId,
publishedAtNs: fusion.Poses.Count == 0
? checked((long)Math.Min(admission.ArrivalMaxNs, (ulong)long.MaxValue))
: fusion.Poses.Max(pose => pose.EstimatedAtNs),
calibration.TrackingSpaceId,
calibration.CalibrationId,
fusion.Poses);
var proofSurface = MimirMoveProofSurface.Create(
admission,
poseFrame,
options.FensalirFrameId,
options.FensalirPresentedAtNs,
options.MimirEvidenceFrameId);
var probeFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(proofSurface);
result = new MimirMoveProofPipelineResult(
admission,
fusion,
poseFrame,
proofSurface,
probeFrame);
return true;
}
private static ulong NowUnixNs()
{
var now = DateTimeOffset.UtcNow;
var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
}

View file

@ -0,0 +1,279 @@
using GameCult.Mesh;
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
public interface IMimirMoveProofEvidenceRingProvider
{
bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic);
}
public sealed class MimirMoveProofEvidenceRingLease : IDisposable
{
private readonly bool ownsRing;
private readonly IDisposable? transportOwner;
public MimirMoveProofEvidenceRingLease(
CultMeshSharedMemoryFrameRing ring,
bool ownsRing = false,
IDisposable? transportOwner = null)
{
Ring = ring ?? throw new ArgumentNullException(nameof(ring));
this.ownsRing = ownsRing;
this.transportOwner = transportOwner;
}
public CultMeshSharedMemoryFrameRing Ring { get; }
public void Dispose()
{
transportOwner?.Dispose();
if (ownsRing)
{
Ring.Dispose();
}
}
}
[CultDocument("mimir.move_proof_evidence_frame_snapshot", "mimir.move_proof_evidence_frame_snapshot.v1")]
[MessagePackObject]
public sealed record MimirMoveProofEvidenceFrameSnapshotDocument(
[property: Key(0)]
[property: CultName]
string SnapshotId,
[property: Key(1)] string EvidenceStreamId,
[property: Key(2)] string FrameId,
[property: Key(3)] string ProducerPeerId,
[property: Key(4)] ulong PublishedAtNs,
[property: Key(5)] ulong CapturedAtNs,
[property: Key(6)] byte[] Payload);
public static class MimirMoveProofEvidenceFrameSnapshot
{
public static MimirMoveProofEvidenceFrameSnapshotDocument Create(
MimirMuninnMoveEvidenceStreamFrame frame,
byte[] payload,
ulong capturedAtNs)
{
ArgumentNullException.ThrowIfNull(frame);
ArgumentNullException.ThrowIfNull(payload);
if (payload.Length == 0)
{
throw new ArgumentException("Snapshot payload must not be empty.", nameof(payload));
}
return new MimirMoveProofEvidenceFrameSnapshotDocument(
SnapshotId: $"{frame.FrameId}:snapshot",
EvidenceStreamId: StreamIdFromFrameId(frame.FrameId),
FrameId: frame.FrameId,
ProducerPeerId: frame.ProducerPeerId,
PublishedAtNs: checked((ulong)frame.PublishedAtNs),
CapturedAtNs: capturedAtNs,
Payload: payload);
}
public static MimirMoveProofEvidenceFrameSnapshotDocument Load(string path)
{
ArgumentException.ThrowIfNullOrWhiteSpace(path);
return MessagePackSerializer.Deserialize<MimirMoveProofEvidenceFrameSnapshotDocument>(File.ReadAllBytes(path));
}
public static void Save(string path, MimirMoveProofEvidenceFrameSnapshotDocument snapshot)
{
ArgumentException.ThrowIfNullOrWhiteSpace(path);
ArgumentNullException.ThrowIfNull(snapshot);
var directory = Path.GetDirectoryName(Path.GetFullPath(path));
if (!string.IsNullOrWhiteSpace(directory))
{
Directory.CreateDirectory(directory);
}
File.WriteAllBytes(path, MessagePackSerializer.Serialize(snapshot));
}
private static string StreamIdFromFrameId(string frameId)
{
var separator = frameId.LastIndexOf(':');
return separator > 0 ? frameId[..separator] : frameId;
}
}
public sealed record MimirMoveProofRuntimeActivationStatus(
string EvidenceStreamId,
bool Active,
string Diagnostic,
string ProviderKind,
bool DriverRegistered);
[CultDocument("mimir.move_proof_runtime_activation", "mimir.move_proof_runtime_activation.v1")]
[MessagePackObject]
public sealed record MimirMoveProofRuntimeActivationDocument(
[property: Key(0)]
[property: CultName]
string ActivationId,
[property: Key(1)] string EvidenceStreamId,
[property: Key(2)] bool Active,
[property: Key(3)] string Diagnostic,
[property: Key(4)] string ProviderKind,
[property: Key(5)] bool DriverRegistered,
[property: Key(6)] string NativeReservoirPath,
[property: Key(7)] string[] CalibratedCameraIds,
[property: Key(8)] int CalibratedCameraCount,
[property: Key(9)] string LatestProofId,
[property: Key(10)] string LatestMuninnEvidenceFrameId,
[property: Key(11)] string LatestMimirEvidenceFrameId,
[property: Key(12)] string LatestMimirPoseFrameId,
[property: Key(13)] string LatestFensalirFrameId,
[property: Key(14)] MimirMoveProofVerdict LatestVerdict,
[property: Key(15)] ulong ObservedAtNs);
public static class MimirMoveProofRuntimeActivation
{
public static MimirMoveProofRuntimeActivationDocument CreateDocument(
MimirMoveProofRuntimeConfiguration configuration,
MimirMoveProofRuntimeActivationStatus? status,
MimirMoveProofSurfaceDocument? latestProofSurface,
ulong observedAtNs)
{
ArgumentNullException.ThrowIfNull(configuration);
var active = status?.Active ?? false;
var diagnostic = status?.Diagnostic ?? "not activated";
var providerKind = status?.ProviderKind ?? "";
var driverRegistered = status?.DriverRegistered ?? false;
var matchingSurface = IsSurfaceFromEvidenceStream(latestProofSurface, configuration.EvidenceStreamId)
? latestProofSurface
: null;
return new MimirMoveProofRuntimeActivationDocument(
ActivationId: $"{configuration.EvidenceStreamId}:activation",
EvidenceStreamId: configuration.EvidenceStreamId,
Active: active,
Diagnostic: diagnostic,
ProviderKind: providerKind,
DriverRegistered: driverRegistered,
NativeReservoirPath: configuration.NativeReservoirPath,
CalibratedCameraIds: configuration.Calibration.Cameras
.Select(camera => camera.CameraId)
.Where(cameraId => !string.IsNullOrWhiteSpace(cameraId))
.ToArray(),
CalibratedCameraCount: configuration.Calibration.Cameras.Count,
LatestProofId: matchingSurface?.ProofId ?? "",
LatestMuninnEvidenceFrameId: matchingSurface?.MuninnEvidenceFrameId ?? "",
LatestMimirEvidenceFrameId: matchingSurface?.MimirEvidenceFrameId ?? "",
LatestMimirPoseFrameId: matchingSurface?.MimirPoseFrameId ?? "",
LatestFensalirFrameId: matchingSurface?.FensalirFrameId ?? "",
LatestVerdict: matchingSurface?.Verdict ?? MimirMoveProofVerdict.Unknown,
ObservedAtNs: observedAtNs);
}
private static bool IsSurfaceFromEvidenceStream(
MimirMoveProofSurfaceDocument? latestProofSurface,
string evidenceStreamId)
{
if (latestProofSurface is null || string.IsNullOrWhiteSpace(evidenceStreamId))
{
return false;
}
return string.Equals(latestProofSurface.MuninnEvidenceFrameId, evidenceStreamId, StringComparison.Ordinal) ||
latestProofSurface.MuninnEvidenceFrameId.StartsWith($"{evidenceStreamId}:", StringComparison.Ordinal);
}
}
public sealed class MimirConfiguredMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider
{
public static MimirConfiguredMoveProofEvidenceRingProvider Instance { get; } = new();
private MimirConfiguredMoveProofEvidenceRingProvider()
{
}
public bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic)
{
ArgumentNullException.ThrowIfNull(configuration);
if (!string.IsNullOrWhiteSpace(configuration.OdinCultMeshUri))
{
return MimirOdinMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing(
configuration,
out lease,
out diagnostic);
}
if (string.IsNullOrWhiteSpace(configuration.EvidenceSnapshotPath))
{
return MimirUnavailableMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing(
configuration,
out lease,
out diagnostic);
}
var snapshotPath = Path.GetFullPath(configuration.EvidenceSnapshotPath);
if (!File.Exists(snapshotPath))
{
lease = null;
diagnostic = $"Move proof evidence snapshot not found: {snapshotPath}";
return false;
}
var snapshot = MimirMoveProofEvidenceFrameSnapshot.Load(snapshotPath);
if (!string.Equals(snapshot.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal))
{
lease = null;
diagnostic = $"Move proof evidence snapshot stream '{snapshot.EvidenceStreamId}' does not match configured evidence stream '{configuration.EvidenceStreamId}'.";
return false;
}
var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(snapshot.Payload);
if (!string.Equals(decoded.FrameId, snapshot.FrameId, StringComparison.Ordinal) ||
!string.Equals(decoded.ProducerPeerId, snapshot.ProducerPeerId, StringComparison.Ordinal) ||
decoded.PublishedAtNs != checked((long)snapshot.PublishedAtNs))
{
lease = null;
diagnostic = "Move proof evidence snapshot metadata does not match its encoded Muninn frame payload.";
return false;
}
var ring = new CultMeshSharedMemoryFrameRing(
configuration.EvidenceStreamId,
slotCount: 1,
slotByteLength: snapshot.Payload.Length);
if (!ring.TryPublishCopy(snapshot.Payload, checked((long)snapshot.PublishedAtNs), durationNs: 0, out _))
{
ring.Dispose();
lease = null;
diagnostic = $"Move proof evidence snapshot could not be published into fallback ring: {snapshotPath}";
return false;
}
lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true);
diagnostic = $"one-copy evidence snapshot supplied from {snapshotPath}; this is not a live cross-process CultMesh ring";
return true;
}
}
public sealed class MimirUnavailableMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider
{
public static MimirUnavailableMoveProofEvidenceRingProvider Instance { get; } = new();
private MimirUnavailableMoveProofEvidenceRingProvider()
{
}
public bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic)
{
ArgumentNullException.ThrowIfNull(configuration);
lease = null;
diagnostic = "No live CultMesh shared-memory ring opener is configured for this runtime; C# CultMesh rings are currently in-process only.";
return false;
}
}

View file

@ -0,0 +1,205 @@
using GameCult.Mesh;
namespace Mimir.Runtime.Synchronization;
public sealed class MimirMoveProofRuntimeConfiguration
{
public bool Enabled { get; init; } = true;
public string EvidenceStreamId { get; init; } = "muninn:nightwing:move-evidence";
public string EvidenceSnapshotPath { get; init; } = "";
public string OdinCultMeshUri { get; init; } = "";
public string MuninnProviderId { get; init; } = "muninn.telemetry.nightwing";
public string NativeReservoirPath { get; init; } = "";
public string MimirEvidenceSourceId { get; init; } = "mimir:starfire:move-evidence";
public string MimirEvidenceFramePrefix { get; init; } = "mimir:starfire:move-evidence";
public string MimirPoseFramePrefix { get; init; } = "mimir:starfire:move-pose";
public string MimirPoseProducerPeerId { get; init; } = "mimir:starfire";
public string FensalirFramePrefix { get; init; } = "fensalir:starfire:presented-frame";
public string FusionAuthorityId { get; init; } = "mimir.runtime.move-fusion";
public string ConsumerContract { get; init; } = "fensalir.move-controller-input";
public MimirMoveProofCalibrationConfiguration Calibration { get; init; } = new();
public string[] Validate()
{
if (!Enabled)
{
return [];
}
var errors = new List<string>();
Require(EvidenceStreamId, nameof(EvidenceStreamId), errors);
if (!string.IsNullOrWhiteSpace(OdinCultMeshUri))
{
Require(MuninnProviderId, nameof(MuninnProviderId), errors);
}
Require(NativeReservoirPath, nameof(NativeReservoirPath), errors);
Require(MimirEvidenceSourceId, nameof(MimirEvidenceSourceId), errors);
Require(MimirEvidenceFramePrefix, nameof(MimirEvidenceFramePrefix), errors);
Require(MimirPoseFramePrefix, nameof(MimirPoseFramePrefix), errors);
Require(MimirPoseProducerPeerId, nameof(MimirPoseProducerPeerId), errors);
Require(FensalirFramePrefix, nameof(FensalirFramePrefix), errors);
Require(FusionAuthorityId, nameof(FusionAuthorityId), errors);
Require(ConsumerContract, nameof(ConsumerContract), errors);
errors.AddRange(Calibration.Validate().Select(error => $"{nameof(Calibration)}.{error}"));
return errors.ToArray();
}
public MimirMoveProofRuntimeDriverOptions ToDriverOptions() => new(
MimirEvidenceSourceId,
MimirEvidenceFramePrefix,
MimirPoseFramePrefix,
MimirPoseProducerPeerId,
FensalirFramePrefix,
FusionAuthorityId,
ConsumerContract);
public MimirMoveProofRuntimeDriver CreateDriver(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime reservoir)
{
ArgumentNullException.ThrowIfNull(ring);
ArgumentNullException.ThrowIfNull(reservoir);
var errors = Validate();
if (errors.Length > 0)
{
throw new InvalidOperationException($"Invalid Move proof runtime configuration: {string.Join("; ", errors)}");
}
if (!string.Equals(ring.StreamId, EvidenceStreamId, StringComparison.Ordinal))
{
throw new InvalidOperationException(
$"Move proof runtime configuration expects evidence stream '{EvidenceStreamId}', but ring is '{ring.StreamId}'.");
}
return new MimirMoveProofRuntimeDriver(
ring,
reservoir,
Calibration.ToRigCalibration(),
ToDriverOptions());
}
private static void Require(string value, string name, ICollection<string> errors)
{
if (string.IsNullOrWhiteSpace(value))
{
errors.Add($"{name} is required");
}
}
}
public sealed class MimirMoveProofCalibrationConfiguration
{
public string CalibrationId { get; init; } = "mimir-move-stage-calibration-v1";
public string TrackingSpaceId { get; init; } = "mimir-stage-space";
public List<MimirMoveProofCameraCalibrationConfiguration> Cameras { get; init; } = [];
public double GyroUnitsPerRadianPerSecond { get; init; } = 1.0;
public double MaximumAssociationSkewMilliseconds { get; init; } = 20.0;
public double SingleRayFallbackDepthMeters { get; init; } = 1.5;
public string[] Validate()
{
var errors = new List<string>();
if (string.IsNullOrWhiteSpace(CalibrationId))
{
errors.Add($"{nameof(CalibrationId)} is required");
}
if (string.IsNullOrWhiteSpace(TrackingSpaceId))
{
errors.Add($"{nameof(TrackingSpaceId)} is required");
}
if (Cameras.Count < 2)
{
errors.Add($"{nameof(Cameras)} must contain at least two calibrated camera witnesses for full-pose proof");
}
for (var index = 0; index < Cameras.Count; index++)
{
errors.AddRange(Cameras[index].Validate().Select(error => $"{nameof(Cameras)}[{index}].{error}"));
}
return errors.ToArray();
}
public MimirMoveFusionRigCalibration ToRigCalibration() => new(
CalibrationId,
TrackingSpaceId,
Cameras.Select(camera => camera.ToCameraCalibration()).ToArray(),
GyroUnitsPerRadianPerSecond,
MaximumAssociationSkewMilliseconds,
SingleRayFallbackDepthMeters);
}
public sealed class MimirMoveProofCameraCalibrationConfiguration
{
public string CameraId { get; init; } = "";
public ulong WitnessIdHash { get; init; }
public MimirVector3Snapshot PositionMeters { get; init; } = new(0.0, 0.0, 0.0);
public MimirQuaternionSnapshot Orientation { get; init; } = new(0.0, 0.0, 0.0, 1.0);
public double FocalLengthXPx { get; init; }
public double FocalLengthYPx { get; init; }
public double PrincipalPointXPx { get; init; }
public double PrincipalPointYPx { get; init; }
public string[] Validate()
{
var errors = new List<string>();
if (string.IsNullOrWhiteSpace(CameraId))
{
errors.Add($"{nameof(CameraId)} is required");
}
if (WitnessIdHash == 0UL)
{
errors.Add($"{nameof(WitnessIdHash)} is required");
}
if (!double.IsFinite(FocalLengthXPx) || FocalLengthXPx <= 0.0)
{
errors.Add($"{nameof(FocalLengthXPx)} must be positive");
}
if (!double.IsFinite(FocalLengthYPx) || FocalLengthYPx <= 0.0)
{
errors.Add($"{nameof(FocalLengthYPx)} must be positive");
}
return errors.ToArray();
}
public MimirMoveFusionCameraCalibration ToCameraCalibration() => new(
CameraId,
WitnessIdHash,
PositionMeters,
Orientation,
FocalLengthXPx,
FocalLengthYPx,
PrincipalPointXPx,
PrincipalPointYPx);
}

View file

@ -0,0 +1,94 @@
using GameCult.Mesh;
namespace Mimir.Runtime.Synchronization;
public sealed record MimirMoveProofRuntimeDriverOptions(
string MimirEvidenceSourceId,
string MimirEvidenceFramePrefix,
string MimirPoseFramePrefix,
string MimirPoseProducerPeerId,
string FensalirFramePrefix,
string FusionAuthorityId = "mimir.runtime.move-fusion",
string ConsumerContract = "fensalir.move-controller-input");
public sealed class MimirMoveProofRuntimeDriver
{
private readonly CultMeshSharedMemoryFrameRing ring;
private readonly MimirNativeReservoirRuntime reservoir;
private readonly MimirMoveFusionRigCalibration calibration;
private readonly MimirMoveProofRuntimeDriverOptions options;
private string lastFrameId = "";
public MimirMoveProofRuntimeDriver(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime reservoir,
MimirMoveFusionRigCalibration calibration,
MimirMoveProofRuntimeDriverOptions options)
{
this.ring = ring ?? throw new ArgumentNullException(nameof(ring));
this.reservoir = reservoir ?? throw new ArgumentNullException(nameof(reservoir));
this.calibration = calibration ?? throw new ArgumentNullException(nameof(calibration));
this.options = options ?? throw new ArgumentNullException(nameof(options));
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirEvidenceSourceId);
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirEvidenceFramePrefix);
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirPoseFramePrefix);
ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirPoseProducerPeerId);
ArgumentException.ThrowIfNullOrWhiteSpace(options.FensalirFramePrefix);
}
public MimirMoveProofPipelineResult? LastResult { get; private set; }
public bool TryBuildLatest(ulong fensalirPresentedAtNs, out MimirMoveProofPipelineResult? result)
{
result = null;
if (!ring.TryAcquireLatestRead(out var lease))
{
return false;
}
using (lease)
{
var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]);
if (string.Equals(frame.FrameId, lastFrameId, StringComparison.Ordinal))
{
return false;
}
var sequence = SequenceSuffix(frame.FrameId);
var pipelineOptions = new MimirMoveProofPipelineOptions(
MimirEvidenceSourceId: options.MimirEvidenceSourceId,
MimirEvidenceFrameId: $"{options.MimirEvidenceFramePrefix}:{sequence}",
MimirPoseFrameId: $"{options.MimirPoseFramePrefix}:{sequence}",
MimirPoseProducerPeerId: options.MimirPoseProducerPeerId,
FensalirFrameId: $"{options.FensalirFramePrefix}:{sequence}",
FensalirPresentedAtNs: fensalirPresentedAtNs,
FusionAuthorityId: options.FusionAuthorityId,
ConsumerContract: options.ConsumerContract);
if (!MimirMoveProofPipeline.TryBuild(frame, reservoir, calibration, pipelineOptions, out result))
{
return false;
}
lastFrameId = frame.FrameId;
LastResult = result;
return true;
}
}
public string Describe()
{
var last = LastResult?.ProofSurface;
return last is null
? $"{options.MimirEvidenceSourceId} waiting"
: $"{last.ProofId} verdict={last.Verdict} chain={last.MuninnEvidenceFrameId}->{last.MimirEvidenceFrameId}->{last.MimirPoseFrameId}->{last.FensalirFrameId}";
}
private static string SequenceSuffix(string frameId)
{
var separator = frameId.LastIndexOf(':');
return separator >= 0 && separator < frameId.Length - 1
? frameId[(separator + 1)..]
: frameId;
}
}

View file

@ -0,0 +1,235 @@
using Aquarium.Engine.Render;
using GameCult.Caching;
using MessagePack;
using System.Numerics;
namespace Mimir.Runtime.Synchronization;
public enum MimirMoveProofVerdict
{
Unknown,
NoPose,
SingleRayFallback,
FullPose
}
[CultDocument("mimir.move_proof_surface", "mimir.move_proof_surface.v1")]
[MessagePackObject]
public sealed record MimirMoveProofSurfaceDocument(
[property: Key(0)]
[property: CultName]
string ProofId,
[property: Key(1)] string MuninnEvidenceFrameId,
[property: Key(2)] string MimirEvidenceFrameId,
[property: Key(3)] string MimirPoseFrameId,
[property: Key(4)] string FensalirFrameId,
[property: Key(5)] string ProducerPeerId,
[property: Key(6)] string FusionAuthorityId,
[property: Key(7)] string TrackingSpaceId,
[property: Key(8)] string CalibrationId,
[property: Key(9)] long SourceTimestampNs,
[property: Key(10)] long MuninnPublishedAtNs,
[property: Key(11)] ulong MimirReadAtNs,
[property: Key(12)] ulong MimirAdmittedAtNs,
[property: Key(13)] long FusionEstimatedAtNs,
[property: Key(14)] ulong FensalirPresentedAtNs,
[property: Key(15)] ulong ReservoirEdgeNs,
[property: Key(16)] ulong ReservoirWindowStartNs,
[property: Key(17)] int OpticalMarkerCount,
[property: Key(18)] int ControllerStateCount,
[property: Key(19)] int PoseCount,
[property: Key(20)] double PoseConfidence,
[property: Key(21)] double LatencyMilliseconds,
[property: Key(22)] MimirMoveProofVerdict Verdict,
[property: Key(23)] string FailureReason,
[property: Key(24)] MimirMoveControllerPoseDocument[] Poses)
{
[IgnoreMember]
public bool IsFullPose => Verdict == MimirMoveProofVerdict.FullPose;
}
public static class MimirMoveProofSurface
{
public const string ConsumerContract = "fensalir.move-proof-surface";
public static MimirMoveProofSurfaceDocument Create(
MimirMuninnMoveEvidenceAdmission admission,
MimirMoveControllerPoseStreamFrame poseFrame,
string fensalirFrameId,
ulong fensalirPresentedAtNs,
string? mimirEvidenceFrameId = null)
{
ArgumentNullException.ThrowIfNull(admission);
ArgumentNullException.ThrowIfNull(poseFrame);
ArgumentException.ThrowIfNullOrWhiteSpace(fensalirFrameId);
var poses = poseFrame.Poses
.OrderBy(pose => pose.SourceTimestampNs)
.ThenBy(pose => pose.Sequence)
.ToArray();
var selectedPose = poses
.OrderByDescending(pose => pose.Confidence)
.FirstOrDefault();
var verdict = ResolveVerdict(poses);
var failureReason = ResolveFailureReason(admission, poses, verdict);
var sourceNs = selectedPose?.SourceTimestampNs ??
checked((long)Math.Min(admission.SourceTimeMaxNs, (ulong)long.MaxValue));
var estimatedAtNs = selectedPose?.EstimatedAtNs ?? 0L;
var confidence = selectedPose?.Confidence ?? 0.0;
var latencyMs = selectedPose?.LatencyMilliseconds ?? 0.0;
return new MimirMoveProofSurfaceDocument(
ProofId: $"{poseFrame.ProducerPeerId}:move-proof:{SequenceSuffix(poseFrame.FrameId)}",
MuninnEvidenceFrameId: admission.FrameId,
MimirEvidenceFrameId: string.IsNullOrWhiteSpace(mimirEvidenceFrameId)
? $"mimir:starfire:move-evidence:{SequenceSuffix(admission.FrameId)}"
: mimirEvidenceFrameId,
MimirPoseFrameId: poseFrame.FrameId,
FensalirFrameId: fensalirFrameId,
ProducerPeerId: poseFrame.ProducerPeerId,
FusionAuthorityId: selectedPose?.FusionAuthorityId ?? "mimir.runtime.move-fusion",
TrackingSpaceId: poseFrame.TrackingSpaceId,
CalibrationId: poseFrame.CalibrationId,
SourceTimestampNs: sourceNs,
MuninnPublishedAtNs: admission.PublishedAtNs,
MimirReadAtNs: admission.MimirReadAtNs,
MimirAdmittedAtNs: admission.ArrivalMaxNs,
FusionEstimatedAtNs: estimatedAtNs,
FensalirPresentedAtNs: fensalirPresentedAtNs,
ReservoirEdgeNs: admission.ReservoirEdgeNs,
ReservoirWindowStartNs: admission.ReservoirWindowStartNs,
OpticalMarkerCount: admission.OpticalMarkerCount,
ControllerStateCount: admission.ControllerStateCount,
PoseCount: poses.Length,
PoseConfidence: Math.Clamp(confidence, 0.0, 1.0),
LatencyMilliseconds: Math.Max(0.0, latencyMs),
Verdict: verdict,
FailureReason: failureReason,
Poses: poses);
}
public static AquariumSplineFrame BuildFensalirProbeFrame(MimirMoveProofSurfaceDocument surface)
{
ArgumentNullException.ThrowIfNull(surface);
if (surface.Poses.Length == 0)
{
return BuildFailureProbe(surface);
}
var splines = new List<AquariumSpline3D>();
foreach (var pose in surface.Poses)
{
var center = ToVector(pose.PositionMeters);
var confidence = (float)Math.Clamp(pose.Confidence, 0.0, 1.0);
var color = surface.IsFullPose
? new Vector4(0.15f, 1.35f, 0.92f, 0.92f)
: new Vector4(1.35f, 0.78f, 0.18f, 0.86f);
var radius = MathF.Max(0.03f, 0.08f * confidence);
splines.Add(new AquariumSpline3D(
$"move-proof-{SanitizeId(pose.PoseId)}-x",
[
new(center + new Vector3(-radius, 0.0f, 0.0f), color),
new(center + new Vector3(radius, 0.0f, 0.0f), color)
],
new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f),
CatmullRomSubdivisions: 1));
splines.Add(new AquariumSpline3D(
$"move-proof-{SanitizeId(pose.PoseId)}-y",
[
new(center + new Vector3(0.0f, -radius, 0.0f), color),
new(center + new Vector3(0.0f, radius, 0.0f), color)
],
new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f),
CatmullRomSubdivisions: 1));
splines.Add(new AquariumSpline3D(
$"move-proof-{SanitizeId(pose.PoseId)}-z",
[
new(center + new Vector3(0.0f, 0.0f, -radius), color),
new(center + new Vector3(0.0f, 0.0f, radius), color)
],
new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f),
CatmullRomSubdivisions: 1));
}
return new AquariumSplineFrame { Splines = splines };
}
private static AquariumSplineFrame BuildFailureProbe(MimirMoveProofSurfaceDocument surface)
{
var color = new Vector4(1.2f, 0.2f, 0.12f, 0.85f);
var z = surface.OpticalMarkerCount > 0 ? 0.35f : 0.0f;
return new AquariumSplineFrame
{
Splines =
[
new AquariumSpline3D(
$"move-proof-{SanitizeId(surface.ProofId)}-missing-pose",
[
new(new Vector3(-0.16f, 0.0f, z), color),
new(new Vector3(0.16f, 0.0f, z), color),
new(new Vector3(0.0f, 0.16f, z), color),
new(new Vector3(-0.16f, 0.0f, z), color)
],
new AquariumSplineStyle(0.01f, 1.2f, 0.85f, 0.8f, 0.12f),
CatmullRomSubdivisions: 1)
]
};
}
private static MimirMoveProofVerdict ResolveVerdict(IReadOnlyList<MimirMoveControllerPoseDocument> poses)
{
if (poses.Count == 0)
{
return MimirMoveProofVerdict.NoPose;
}
return poses.Any(pose => pose.EvidenceKinds.Contains("optical-marker:triangulated", StringComparer.Ordinal))
? MimirMoveProofVerdict.FullPose
: MimirMoveProofVerdict.SingleRayFallback;
}
private static string ResolveFailureReason(
MimirMuninnMoveEvidenceAdmission admission,
IReadOnlyList<MimirMoveControllerPoseDocument> poses,
MimirMoveProofVerdict verdict)
{
if (verdict == MimirMoveProofVerdict.FullPose)
{
return "";
}
if (admission.OpticalMarkerCount == 0)
{
return "missing optical marker candidates";
}
if (admission.ControllerStateCount == 0)
{
return "missing controller state";
}
if (poses.Count == 0)
{
return "Mimir fusion produced no calibrated pose";
}
return "pose is single-ray fallback, not full calibrated optical fusion";
}
private static string SequenceSuffix(string frameId)
{
var separator = frameId.LastIndexOf(':');
return separator >= 0 && separator < frameId.Length - 1
? frameId[(separator + 1)..]
: frameId;
}
private static Vector3 ToVector(MimirVector3Snapshot value) =>
new((float)value.X, (float)value.Y, (float)value.Z);
private static string SanitizeId(string value)
{
var chars = value.Select(ch => char.IsLetterOrDigit(ch) ? ch : '-').ToArray();
return new string(chars);
}
}

View file

@ -0,0 +1,395 @@
using System.Numerics;
namespace Mimir.Runtime.Synchronization;
public static class MimirMoveSphereCalibrationFitter
{
private const int ParameterCount = 8;
public static MimirSensorCalibrationReceiptDocument Fit(
MimirSensorCalibrationSessionDocument session,
MoveVisibilityWindowReceipt evidence,
int imageWidth,
int imageHeight,
double orbRadiusMeters,
DateTimeOffset? completedAt = null)
{
ArgumentNullException.ThrowIfNull(session);
ArgumentNullException.ThrowIfNull(evidence);
if (imageWidth <= 0 || imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth));
var rejectionReasons = new List<string>();
if (session.Phase != MimirSensorCalibrationSessionPhase.Fitting)
rejectionReasons.Add("session-is-not-ready-for-fitting");
if (!(double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0))
rejectionReasons.Add("measured-orb-radius-meters");
if (session.SensorIds.Length != 2)
rejectionReasons.Add("sphere-stereo-fit-requires-exactly-two-sensors");
var pairs = BuildPairs(session, evidence);
if (pairs.Length < session.Acceptance.MinimumSameFrameCorrespondences)
rejectionReasons.Add($"same-frame-correspondences-{pairs.Length}-of-{session.Acceptance.MinimumSameFrameCorrespondences}");
if (rejectionReasons.Count > 0)
return Rejected(session, pairs.Length, rejectionReasons, completedAt);
var heldOut = pairs.Where(IsHeldOut).ToArray();
var training = pairs.Where(pair => !IsHeldOut(pair)).ToArray();
if (heldOut.Length < 50 || training.Length < 200)
return Rejected(session, training.Length, ["insufficient-deterministic-training-or-held-out-partition"], completedAt, heldOut.Length);
var parameters = InitialParameters(training, imageWidth, imageHeight, orbRadiusMeters);
var iterations = Optimize(parameters, training, imageWidth, imageHeight, orbRadiusMeters);
var trainingConsensus = training;
for (var pass = 0; pass < 3; pass++)
{
var pairErrors = PairErrors(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters);
var cutoff = Math.Clamp(Percentile(pairErrors.Select(value => value.Error).Order().ToArray(), 0.70) * 1.5, 4.0, 25.0);
var next = pairErrors.Where(value => value.Error <= cutoff).Select(value => value.Pair).ToArray();
if (next.Length < 200 || next.Length == trainingConsensus.Length) break;
trainingConsensus = next;
iterations += Optimize(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters);
}
var validationCutoff = Math.Max(4.0, session.Acceptance.MaximumP95ReprojectionErrorPx * 2.0);
var heldOutConsensus = PairErrors(parameters, heldOut, imageWidth, imageHeight, orbRadiusMeters)
.Where(value => value.Error <= validationCutoff).Select(value => value.Pair).ToArray();
var trainingErrors = ReprojectionErrors(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters);
var heldOutErrors = ReprojectionErrors(parameters, heldOutConsensus, imageWidth, imageHeight, orbRadiusMeters);
var skewMs = pairs.Select(pair => pair.AbsoluteSkewNs / 1_000_000.0).Order().ToArray();
var residuals = new MimirSensorCalibrationFitResiduals(
Median(trainingErrors), Percentile(trainingErrors, 0.95),
Median(heldOutErrors), Percentile(heldOutErrors, 0.95),
Median(skewMs), skewMs[^1],
ConditionEstimate(parameters, training, imageWidth, imageHeight, orbRadiusMeters),
iterations,
(double)trainingConsensus.Length / training.Length,
(double)heldOutConsensus.Length / heldOut.Length);
if (!parameters.All(double.IsFinite)) rejectionReasons.Add("non-finite-fit");
if (Math.Exp(parameters[0]) is < 100.0 or > 3000.0) rejectionReasons.Add($"camera-0-focal-length-{Math.Exp(parameters[0]):F1}-px-out-of-range");
if (Math.Exp(parameters[1]) is < 100.0 or > 3000.0) rejectionReasons.Add($"camera-1-focal-length-{Math.Exp(parameters[1]):F1}-px-out-of-range");
if (residuals.HeldOutMedianReprojectionErrorPx > session.Acceptance.MaximumMedianReprojectionErrorPx)
rejectionReasons.Add($"held-out-median-reprojection-error-{residuals.HeldOutMedianReprojectionErrorPx:F3}-px");
if (residuals.HeldOutP95ReprojectionErrorPx > session.Acceptance.MaximumP95ReprojectionErrorPx)
rejectionReasons.Add($"held-out-p95-reprojection-error-{residuals.HeldOutP95ReprojectionErrorPx:F3}-px");
if (residuals.MaximumAssociationSkewMilliseconds > session.Acceptance.MaximumAssociationSkewMilliseconds)
rejectionReasons.Add($"association-skew-{residuals.MaximumAssociationSkewMilliseconds:F3}-ms");
if (heldOutConsensus.Length < 50 || residuals.HeldOutInlierFraction < 0.65)
rejectionReasons.Add($"held-out-consensus-{heldOutConsensus.Length}-pairs-{residuals.HeldOutInlierFraction:P1}");
var calibration = rejectionReasons.Count == 0
? BuildCalibration(session, parameters, imageWidth, imageHeight)
: null;
return new MimirSensorCalibrationReceiptDocument(
$"{session.SessionId}:receipt",
session.SessionId,
session.TrackingSpaceId,
(completedAt ?? DateTimeOffset.UtcNow).ToString("O"),
calibration is null ? MimirSensorCalibrationSessionPhase.Rejected : MimirSensorCalibrationSessionPhase.Promoted,
training.Length,
heldOut.Length,
residuals,
calibration,
rejectionReasons.ToArray(),
"mimir.sphere-stereo-bundle.v1");
}
private static Pair[] BuildPairs(MimirSensorCalibrationSessionDocument session, MoveVisibilityWindowReceipt evidence)
{
var camera0 = session.SensorIds[0];
var camera1 = session.SensorIds[1];
var wands = session.RequiredWandIds.ToHashSet(StringComparer.Ordinal);
var candidates = evidence.Correspondences
.Where(pair => wands.Contains(pair.MoveId) &&
pair.AbsoluteSkewNs <= session.Acceptance.MaximumAssociationSkewMilliseconds * 1_000_000.0 &&
string.Equals(pair.First.FrameId, pair.Second.FrameId, StringComparison.Ordinal))
.Select(pair => Order(pair, camera0, camera1))
.Where(pair => pair.HasValue)
.Select(pair => pair!.Value)
.Where(pair => Valid(pair.First) && Valid(pair.Second))
.GroupBy(pair => (pair.First.FrameId, pair.MoveId))
.Select(group => group.First())
.OrderBy(pair => pair.First.PublishedAtNs)
.ThenBy(pair => pair.MoveId, StringComparer.Ordinal)
.ToArray();
if (candidates.Length == 0) return candidates;
var firstMedian = Median(candidates.Select(pair => (double)pair.First.RadiusPx).Order().ToArray());
var secondMedian = Median(candidates.Select(pair => (double)pair.Second.RadiusPx).Order().ToArray());
return candidates.Where(pair =>
{
var firstRatio = pair.First.RadiusPx / firstMedian;
var secondRatio = pair.Second.RadiusPx / secondMedian;
var crossRatio = pair.First.RadiusPx / pair.Second.RadiusPx;
return firstRatio is >= 0.35 and <= 4.0 &&
secondRatio is >= 0.35 and <= 4.0 &&
crossRatio is >= 0.25f and <= 4.0f;
}).ToArray();
}
private static Pair? Order(MoveCrossCameraCorrespondence pair, string camera0, string camera1)
{
if (pair.First.CameraId == camera0 && pair.Second.CameraId == camera1)
return new Pair(pair.MoveId, pair.First, pair.Second, pair.AbsoluteSkewNs);
if (pair.First.CameraId == camera1 && pair.Second.CameraId == camera0)
return new Pair(pair.MoveId, pair.Second, pair.First, pair.AbsoluteSkewNs);
return null;
}
private static bool Valid(MoveVisibilityObservation value) =>
float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) &&
float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f;
private static bool IsHeldOut(Pair pair) => (Fnva64($"{pair.First.FrameId}|{pair.MoveId}") % 10) < 2;
private static double[] InitialParameters(Pair[] pairs, int width, int height, double radius)
{
var focal = Math.Max(width, height) * 0.82;
var source = pairs.Select(pair => PointFromSphere(pair.First, focal, width, height, radius)).ToArray();
var target = pairs.Select(pair => PointFromSphere(pair.Second, focal, width, height, radius)).ToArray();
var (rotation, translation) = FitRigid(source, target);
var rotationVector = RotationVector(rotation);
return [Math.Log(focal), Math.Log(focal), rotationVector.X, rotationVector.Y, rotationVector.Z,
translation.X, translation.Y, translation.Z];
}
private static int Optimize(double[] p, Pair[] pairs, int width, int height, double radius)
{
var lambda = 1e-3;
var cost = Cost(p, pairs, width, height, radius);
var iterations = 0;
for (; iterations < 80; iterations++)
{
BuildNormalEquations(p, pairs, width, height, radius, out var normal, out var gradient);
for (var i = 0; i < ParameterCount; i++) normal[i, i] += lambda * Math.Max(1.0, normal[i, i]);
if (!Solve(normal, gradient.Select(value => -value).ToArray(), out var delta)) break;
var candidate = p.Zip(delta, (value, step) => value + step).ToArray();
var candidateCost = Cost(candidate, pairs, width, height, radius);
if (candidateCost < cost)
{
Array.Copy(candidate, p, ParameterCount);
if (Math.Abs(cost - candidateCost) <= Math.Max(1e-9, cost * 1e-8)) { iterations++; break; }
cost = candidateCost;
lambda = Math.Max(1e-8, lambda * 0.35);
}
else
{
lambda = Math.Min(1e8, lambda * 8.0);
}
}
return iterations;
}
private static void BuildNormalEquations(double[] p, Pair[] pairs, int width, int height, double radius, out double[,] normal, out double[] gradient)
{
normal = new double[ParameterCount, ParameterCount];
gradient = new double[ParameterCount];
foreach (var pair in Sample(pairs, 2500))
{
var residual = Residual(p, pair, width, height, radius);
var jacobian = new double[residual.Length, ParameterCount];
for (var column = 0; column < ParameterCount; column++)
{
var step = column < 2 ? 1e-5 : 1e-6;
var perturbed = (double[])p.Clone();
perturbed[column] += step;
var shifted = Residual(perturbed, pair, width, height, radius);
for (var row = 0; row < residual.Length; row++) jacobian[row, column] = (shifted[row] - residual[row]) / step;
}
for (var row = 0; row < residual.Length; row++)
{
var weight = HuberWeight(residual[row], 4.0) * Math.Max(0.05, Math.Min(pair.First.Confidence, pair.Second.Confidence));
for (var i = 0; i < ParameterCount; i++)
{
gradient[i] += weight * jacobian[row, i] * residual[row];
for (var j = 0; j <= i; j++) normal[i, j] += weight * jacobian[row, i] * jacobian[row, j];
}
}
}
for (var i = 0; i < ParameterCount; i++)
for (var j = 0; j < i; j++) normal[j, i] = normal[i, j];
}
private static double[] Residual(double[] p, Pair pair, int width, int height, double radius)
{
var f0 = Math.Exp(p[0]);
var f1 = Math.Exp(p[1]);
var rotation = QuaternionFromRotationVector(new Vector3((float)p[2], (float)p[3], (float)p[4]));
var translation = new Vector3((float)p[5], (float)p[6], (float)p[7]);
var point0 = PointFromSphere(pair.First, f0, width, height, radius);
var point1 = Vector3.Transform(point0, rotation) + translation;
var forward = Project(point1, f1, width, height, radius);
var pointFrom1 = PointFromSphere(pair.Second, f1, width, height, radius);
var reversePoint = Vector3.Transform(pointFrom1 - translation, Quaternion.Conjugate(rotation));
var reverse = Project(reversePoint, f0, width, height, radius);
return [
forward.X - pair.Second.CenterXPx, forward.Y - pair.Second.CenterYPx, (forward.Z - pair.Second.RadiusPx) * 0.5,
reverse.X - pair.First.CenterXPx, reverse.Y - pair.First.CenterYPx, (reverse.Z - pair.First.RadiusPx) * 0.5];
}
private static double Cost(double[] p, Pair[] pairs, int width, int height, double radius) =>
Sample(pairs, 2500).Sum(pair => Residual(p, pair, width, height, radius).Sum(value => HuberLoss(value, 4.0)));
private static double[] ReprojectionErrors(double[] p, Pair[] pairs, int width, int height, double radius) =>
pairs.SelectMany(pair =>
{
var residual = Residual(p, pair, width, height, radius);
return new[] { Math.Sqrt(residual[0] * residual[0] + residual[1] * residual[1]), Math.Sqrt(residual[3] * residual[3] + residual[4] * residual[4]) };
}).Order().ToArray();
private static (Pair Pair, double Error)[] PairErrors(double[] p, Pair[] pairs, int width, int height, double radius) =>
pairs.Select(pair =>
{
var residual = Residual(p, pair, width, height, radius);
var forward = Math.Sqrt(residual[0] * residual[0] + residual[1] * residual[1]);
var reverse = Math.Sqrt(residual[3] * residual[3] + residual[4] * residual[4]);
return (pair, Math.Max(forward, reverse));
}).OrderBy(value => value.Item2).Select(value => (value.pair, value.Item2)).ToArray();
private static MimirMoveFusionRigCalibration BuildCalibration(MimirSensorCalibrationSessionDocument session, double[] p, int width, int height)
{
var worldToCamera = QuaternionFromRotationVector(new Vector3((float)p[2], (float)p[3], (float)p[4]));
var cameraToWorld = Quaternion.Normalize(Quaternion.Conjugate(worldToCamera));
var translation = new Vector3((float)p[5], (float)p[6], (float)p[7]);
var position = Vector3.Transform(-translation, cameraToWorld);
return new MimirMoveFusionRigCalibration(
$"{session.SessionId}:sphere-stereo",
session.TrackingSpaceId,
[
new MimirMoveFusionCameraCalibration(session.SensorIds[0], Fnva64(session.SensorIds[0]),
new MimirVector3Snapshot(0, 0, 0), new MimirQuaternionSnapshot(0, 0, 0, 1),
Math.Exp(p[0]), Math.Exp(p[0]), width * 0.5, height * 0.5),
new MimirMoveFusionCameraCalibration(session.SensorIds[1], Fnva64(session.SensorIds[1]),
new MimirVector3Snapshot(position.X, position.Y, position.Z),
new MimirQuaternionSnapshot(cameraToWorld.X, cameraToWorld.Y, cameraToWorld.Z, cameraToWorld.W),
Math.Exp(p[1]), Math.Exp(p[1]), width * 0.5, height * 0.5)
],
MaximumAssociationSkewMilliseconds: session.Acceptance.MaximumAssociationSkewMilliseconds);
}
private static MimirSensorCalibrationReceiptDocument Rejected(MimirSensorCalibrationSessionDocument session, int trainingCount, IEnumerable<string> reasons, DateTimeOffset? completedAt, int heldOutCount = 0) =>
new($"{session.SessionId}:receipt", session.SessionId, session.TrackingSpaceId,
(completedAt ?? DateTimeOffset.UtcNow).ToString("O"), MimirSensorCalibrationSessionPhase.Rejected,
trainingCount, heldOutCount,
new MimirSensorCalibrationFitResiduals(double.NaN, double.NaN, double.NaN, double.NaN, double.NaN, double.NaN, double.PositiveInfinity, 0),
null, reasons.ToArray(), "mimir.sphere-stereo-bundle.v1");
private static Vector3 PointFromSphere(MoveVisibilityObservation observation, double focal, int width, int height, double radius)
{
var z = radius * Math.Sqrt(1.0 + focal * focal / (observation.RadiusPx * observation.RadiusPx));
return new Vector3((float)((observation.CenterXPx - width * 0.5) * z / focal),
(float)((height * 0.5 - observation.CenterYPx) * z / focal), (float)z);
}
private static Vector3 Project(Vector3 point, double focal, int width, int height, double radius)
{
if (point.Z <= radius * 1.001) return new Vector3(1e6f, 1e6f, 1e6f);
var imageRadius = focal * radius / Math.Sqrt(point.Z * point.Z - radius * radius);
return new Vector3((float)(width * 0.5 + focal * point.X / point.Z),
(float)(height * 0.5 - focal * point.Y / point.Z), (float)imageRadius);
}
private static (Quaternion Rotation, Vector3 Translation) FitRigid(Vector3[] source, Vector3[] target)
{
var sourceCenter = source.Aggregate(Vector3.Zero, (sum, value) => sum + value) / source.Length;
var targetCenter = target.Aggregate(Vector3.Zero, (sum, value) => sum + value) / target.Length;
double sxx = 0, sxy = 0, sxz = 0, syx = 0, syy = 0, syz = 0, szx = 0, szy = 0, szz = 0;
for (var i = 0; i < source.Length; i++)
{
var a = source[i] - sourceCenter;
var b = target[i] - targetCenter;
sxx += a.X * b.X; sxy += a.X * b.Y; sxz += a.X * b.Z;
syx += a.Y * b.X; syy += a.Y * b.Y; syz += a.Y * b.Z;
szx += a.Z * b.X; szy += a.Z * b.Y; szz += a.Z * b.Z;
}
var n = new double[,] {
{ sxx + syy + szz, syz - szy, szx - sxz, sxy - syx },
{ syz - szy, sxx - syy - szz, sxy + syx, szx + sxz },
{ szx - sxz, sxy + syx, -sxx + syy - szz, syz + szy },
{ sxy - syx, szx + sxz, syz + szy, -sxx - syy + szz }
};
var q = new[] { 1.0, 0.0, 0.0, 0.0 };
for (var iteration = 0; iteration < 64; iteration++)
{
var next = new double[4];
for (var row = 0; row < 4; row++) for (var column = 0; column < 4; column++) next[row] += n[row, column] * q[column];
var norm = Math.Sqrt(next.Sum(value => value * value));
if (norm < 1e-12) break;
for (var j = 0; j < 4; j++) q[j] = next[j] / norm;
}
var rotation = Quaternion.Normalize(new Quaternion((float)q[1], (float)q[2], (float)q[3], (float)q[0]));
return (rotation, targetCenter - Vector3.Transform(sourceCenter, rotation));
}
private static Vector3 RotationVector(Quaternion quaternion)
{
quaternion = Quaternion.Normalize(quaternion);
if (quaternion.W < 0) quaternion = new Quaternion(-quaternion.X, -quaternion.Y, -quaternion.Z, -quaternion.W);
var angle = 2.0 * Math.Acos(Math.Clamp(quaternion.W, -1.0f, 1.0f));
var sinHalf = Math.Sqrt(Math.Max(0.0, 1.0 - quaternion.W * quaternion.W));
return sinHalf < 1e-8 ? Vector3.Zero : new Vector3(quaternion.X, quaternion.Y, quaternion.Z) * (float)(angle / sinHalf);
}
private static Quaternion QuaternionFromRotationVector(Vector3 value)
{
var angle = value.Length();
return angle < 1e-9f ? Quaternion.Identity : Quaternion.CreateFromAxisAngle(value / angle, angle);
}
private static Pair[] Sample(Pair[] pairs, int maximum)
{
if (pairs.Length <= maximum) return pairs;
var stride = (double)pairs.Length / maximum;
return Enumerable.Range(0, maximum).Select(index => pairs[(int)(index * stride)]).ToArray();
}
private static double ConditionEstimate(double[] p, Pair[] pairs, int width, int height, double radius)
{
BuildNormalEquations(p, pairs, width, height, radius, out var normal, out _);
var diagonal = Enumerable.Range(0, ParameterCount).Select(index => Math.Abs(normal[index, index])).Where(value => value > 1e-12).ToArray();
return diagonal.Length == 0 ? double.PositiveInfinity : diagonal.Max() / diagonal.Min();
}
private static bool Solve(double[,] matrix, double[] rhs, out double[] solution)
{
var n = rhs.Length;
var augmented = new double[n, n + 1];
for (var row = 0; row < n; row++) { for (var column = 0; column < n; column++) augmented[row, column] = matrix[row, column]; augmented[row, n] = rhs[row]; }
for (var pivot = 0; pivot < n; pivot++)
{
var best = pivot;
for (var row = pivot + 1; row < n; row++) if (Math.Abs(augmented[row, pivot]) > Math.Abs(augmented[best, pivot])) best = row;
if (Math.Abs(augmented[best, pivot]) < 1e-12) { solution = []; return false; }
if (best != pivot) for (var column = pivot; column <= n; column++) (augmented[pivot, column], augmented[best, column]) = (augmented[best, column], augmented[pivot, column]);
var scale = augmented[pivot, pivot];
for (var column = pivot; column <= n; column++) augmented[pivot, column] /= scale;
for (var row = 0; row < n; row++) if (row != pivot)
{
var factor = augmented[row, pivot];
for (var column = pivot; column <= n; column++) augmented[row, column] -= factor * augmented[pivot, column];
}
}
solution = Enumerable.Range(0, n).Select(row => augmented[row, n]).ToArray();
return solution.All(double.IsFinite);
}
private static double HuberLoss(double value, double threshold) => Math.Abs(value) <= threshold ? 0.5 * value * value : threshold * (Math.Abs(value) - 0.5 * threshold);
private static double HuberWeight(double value, double threshold) => Math.Abs(value) <= threshold ? 1.0 : threshold / Math.Abs(value);
private static double Median(double[] sorted) => Percentile(sorted, 0.5);
private static double Percentile(double[] sorted, double percentile)
{
if (sorted.Length == 0) return double.NaN;
var position = Math.Clamp(percentile, 0, 1) * (sorted.Length - 1);
var lower = (int)Math.Floor(position);
var upper = (int)Math.Ceiling(position);
return sorted[lower] + (sorted[upper] - sorted[lower]) * (position - lower);
}
private static ulong Fnva64(string value)
{
var hash = 14695981039346656037UL;
foreach (var item in System.Text.Encoding.UTF8.GetBytes(value)) { hash ^= item; hash *= 1099511628211UL; }
return hash;
}
private readonly record struct Pair(string MoveId, MoveVisibilityObservation First, MoveVisibilityObservation Second, long AbsoluteSkewNs);
}

View file

@ -0,0 +1,139 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MimirMoveCameraCoverageAssessment(
[property: Key(0)] string CameraId,
[property: Key(1)] int ObservationCount,
[property: Key(2)] double MinimumXPx,
[property: Key(3)] double MaximumXPx,
[property: Key(4)] double MinimumYPx,
[property: Key(5)] double MaximumYPx,
[property: Key(6)] int OccupiedGridCells,
[property: Key(7)] int GridCellCount,
[property: Key(8)] double MinimumRadiusPx,
[property: Key(9)] double MaximumRadiusPx);
[CultDocument("mimir.move_stereo_calibration_assessment", "mimir.move_stereo_calibration_assessment.v1")]
[MessagePackObject]
public sealed record MimirMoveStereoCalibrationAssessmentDocument(
[property: Key(0)] string Schema,
[property: Key(1)] [property: CultName] string AssessmentId,
[property: Key(2)] string SourceSchema,
[property: Key(3)] long AssessedAtNs,
[property: Key(4)] int CorrespondenceCount,
[property: Key(5)] int SynchronizedCorrespondenceCount,
[property: Key(6)] int SameFrameCorrespondenceCount,
[property: Key(7)] int DistinctMoveCount,
[property: Key(8)] double MedianAbsoluteSkewMilliseconds,
[property: Key(9)] double MaximumAbsoluteSkewMilliseconds,
[property: Key(10)] MimirMoveCameraCoverageAssessment[] Cameras,
[property: Key(11)] bool IntrinsicsAvailable,
[property: Key(12)] bool OrbRadiusMetersAvailable,
[property: Key(13)] bool ReadyForRelativePoseFit,
[property: Key(14)] bool Promoted,
[property: Key(15)] string[] MissingRequirements);
public static class MimirMoveStereoCalibrationAssessment
{
public const string Schema = "mimir.move_stereo_calibration_assessment.v1";
public static MimirMoveStereoCalibrationAssessmentDocument Assess(
MoveVisibilityWindowReceipt receipt,
int imageWidth,
int imageHeight,
IReadOnlyCollection<string> cameraIds,
bool intrinsicsAvailable,
double orbRadiusMeters,
double maximumAssociationSkewMilliseconds = 20.0,
DateTimeOffset? assessedAt = null)
{
ArgumentNullException.ThrowIfNull(receipt);
ArgumentNullException.ThrowIfNull(cameraIds);
if (imageWidth <= 0 || imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth));
if (cameraIds.Count != 2) throw new ArgumentException("Stereo calibration requires exactly two camera IDs.", nameof(cameraIds));
var cameras = cameraIds.Distinct(StringComparer.Ordinal).ToArray();
if (cameras.Length != 2) throw new ArgumentException("Stereo camera IDs must be distinct.", nameof(cameraIds));
var maximumSkewNs = maximumAssociationSkewMilliseconds * 1_000_000.0;
var correspondences = receipt.Correspondences
.Where(value => cameras.Contains(value.First.CameraId, StringComparer.Ordinal) &&
cameras.Contains(value.Second.CameraId, StringComparer.Ordinal))
.ToArray();
var synchronized = correspondences.Where(value => value.AbsoluteSkewNs <= maximumSkewNs).ToArray();
var sameFrame = synchronized.Where(value => string.Equals(value.First.FrameId, value.Second.FrameId, StringComparison.Ordinal)).ToArray();
var coverage = cameras.Select(cameraId => BuildCoverage(
cameraId,
receipt.Observations.Where(value => string.Equals(value.CameraId, cameraId, StringComparison.Ordinal)).ToArray(),
imageWidth,
imageHeight)).ToArray();
var missing = new List<string>();
if (!intrinsicsAvailable) missing.Add("calibrated-intrinsics-for-both-eyes");
if (!(double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0)) missing.Add("measured-orb-radius-meters");
if (synchronized.Length < 300) missing.Add("at-least-300-synchronized-correspondences");
if (sameFrame.Length < 100) missing.Add("at-least-100-same-frame-correspondences");
if (synchronized.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count() < 2) missing.Add("at-least-two-stable-move-identities");
foreach (var camera in coverage)
{
if (camera.OccupiedGridCells < 6) missing.Add($"{camera.CameraId}:coverage-at-least-6-of-12-grid-cells");
if (camera.MaximumRadiusPx <= camera.MinimumRadiusPx * 1.25) missing.Add($"{camera.CameraId}:radius-range-at-least-1.25x");
}
var skews = correspondences.Select(value => value.AbsoluteSkewNs / 1_000_000.0).Order().ToArray();
var ready = missing.Count == 0;
return new MimirMoveStereoCalibrationAssessmentDocument(
Schema,
$"move-stereo-assessment:{receipt.StartedAtNs}:{receipt.EndedAtNs}",
receipt.Schema,
(assessedAt ?? DateTimeOffset.UtcNow).ToUnixTimeMilliseconds() * 1_000_000,
correspondences.Length,
synchronized.Length,
sameFrame.Length,
synchronized.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count(),
Median(skews),
skews.Length == 0 ? double.NaN : skews[^1],
coverage,
intrinsicsAvailable,
double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0,
ready,
Promoted: false,
missing.Distinct(StringComparer.Ordinal).ToArray());
}
private static MimirMoveCameraCoverageAssessment BuildCoverage(
string cameraId,
MoveVisibilityObservation[] observations,
int imageWidth,
int imageHeight)
{
var valid = observations.Where(value =>
float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) &&
float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f).ToArray();
var cells = valid.Select(value =>
{
var column = Math.Clamp((int)(value.CenterXPx * 4.0 / imageWidth), 0, 3);
var row = Math.Clamp((int)(value.CenterYPx * 3.0 / imageHeight), 0, 2);
return row * 4 + column;
}).Distinct().Count();
return new MimirMoveCameraCoverageAssessment(
cameraId,
valid.Length,
valid.Length == 0 ? double.NaN : valid.Min(value => value.CenterXPx),
valid.Length == 0 ? double.NaN : valid.Max(value => value.CenterXPx),
valid.Length == 0 ? double.NaN : valid.Min(value => value.CenterYPx),
valid.Length == 0 ? double.NaN : valid.Max(value => value.CenterYPx),
cells,
12,
valid.Length == 0 ? double.NaN : valid.Min(value => value.RadiusPx),
valid.Length == 0 ? double.NaN : valid.Max(value => value.RadiusPx));
}
private static double Median(double[] sorted) => sorted.Length switch
{
0 => double.NaN,
_ when sorted.Length % 2 == 1 => sorted[sorted.Length / 2],
_ => (sorted[sorted.Length / 2 - 1] + sorted[sorted.Length / 2]) * 0.5
};
}

View file

@ -0,0 +1,31 @@
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[MessagePackObject]
public sealed record MoveVisibilityObservation(
[property: Key(0)] string ProviderId,
[property: Key(1)] string CameraId,
[property: Key(2)] string MoveId,
[property: Key(3)] string FrameId,
[property: Key(4)] long PublishedAtNs,
[property: Key(5)] float CenterXPx,
[property: Key(6)] float CenterYPx,
[property: Key(7)] float RadiusPx,
[property: Key(8)] float Confidence);
[MessagePackObject]
public sealed record MoveCrossCameraCorrespondence(
[property: Key(0)] string MoveId,
[property: Key(1)] MoveVisibilityObservation First,
[property: Key(2)] MoveVisibilityObservation Second,
[property: Key(3)] long AbsoluteSkewNs);
[MessagePackObject]
public sealed record MoveVisibilityWindowReceipt(
[property: Key(0)] string Schema,
[property: Key(1)] long StartedAtNs,
[property: Key(2)] long EndedAtNs,
[property: Key(3)] string[] Providers,
[property: Key(4)] MoveVisibilityObservation[] Observations,
[property: Key(5)] MoveCrossCameraCorrespondence[] Correspondences);

View file

@ -0,0 +1,555 @@
using GameCult.Caching;
using GameCult.Mesh;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
[CultDocument("muninn.move_marker_candidate", "muninn.move_marker_candidate.v1")]
[MessagePackObject]
public sealed record MuninnMoveMarkerCandidateDocument(
[property: Key(0)] string StreamId,
[property: Key(1)] string HostId,
[property: Key(2)] string CameraId,
[property: Key(3)] ulong FrameSequence,
[property: Key(4)] ulong SourceIdHash,
[property: Key(5)] uint TileX,
[property: Key(6)] uint TileY,
[property: Key(7)] float CenterXPx,
[property: Key(8)] float CenterYPx,
[property: Key(9)] float RadiusPx,
[property: Key(10)] uint AreaPx,
[property: Key(11)] float MeanLuma,
[property: Key(12)] uint PeakLuma,
[property: Key(13)] float Score,
[property: Key(14)] string ObservedAt,
[property: Key(15)] string MoveId = "");
[CultDocument("muninn.move_controller_state", "muninn.move_controller_state.v1")]
[MessagePackObject]
public sealed record MuninnMoveControllerStateDocument(
[property: Key(0)] string StreamId,
[property: Key(1)] string HostId,
[property: Key(2)] string MoveId,
[property: Key(3)] ulong Sequence,
[property: Key(4)] long SourceTimestampNs,
[property: Key(5)] float[] AccelerometerXyz,
[property: Key(6)] float[] GyroscopeXyz,
[property: Key(7)] float[] MagnetometerXyz,
[property: Key(8)] float TriggerValue,
[property: Key(9)] string[] Buttons,
[property: Key(10)] float Battery01,
[property: Key(11)] string ObservedAt,
[property: Key(12)] string SourcePath = "");
[CultDocument("muninn.move_identity", "muninn.move_identity.v1")]
[MessagePackObject]
public sealed record MuninnMoveIdentityDocument(
[property: Key(0)] string IdentityId,
[property: Key(1)] string HostId,
[property: Key(2)] string MoveId,
[property: Key(3)] string SourcePath,
[property: Key(4)] string BluetoothHostAddress,
[property: Key(5)] string State,
[property: Key(6)] string Detail,
[property: Key(7)] string ObservedAt);
public sealed record MimirMuninnMoveIdentitySnapshot(
string IdentityId,
string HostId,
string MoveId,
string SourcePath,
string BluetoothHostAddress,
string State,
string Detail,
ulong ObservedAtNs,
ulong ControllerIdHash);
public sealed record MimirMuninnMoveRosterEntry(
string MoveId,
ulong ControllerIdHash,
string LatestHostId,
string LatestState,
string StateSummary,
string PickupReadiness,
bool IsBluetoothPickupReady,
string BluetoothHostAddress,
bool HasUsbWitness,
string[] UsbHostIds,
string[] BluetoothPickupHostIds,
string[] UsbBlockingHostIds,
string[] SourcePaths,
ulong ObservedAtNs);
[MessagePackObject]
public sealed record MimirMuninnMoveEvidenceStreamFrame(
[property: Key(0)] string FrameId,
[property: Key(1)] string ProducerPeerId,
[property: Key(2)] long PublishedAtNs,
[property: Key(3)] MuninnMoveMarkerCandidateDocument[] MarkerCandidates,
[property: Key(4)] MuninnMoveControllerStateDocument[] ControllerStates);
public sealed record MimirMuninnMoveEvidenceAdmission(
string FrameId,
string ProducerPeerId,
long PublishedAtNs,
ulong MimirReadAtNs,
int SampleCount,
int OpticalMarkerCount,
int ControllerStateCount,
ulong SourceTimeMinNs,
ulong SourceTimeMaxNs,
ulong ArrivalMinNs,
ulong ArrivalMaxNs,
MimirNativeSampleHandle Handle,
ulong ReservoirEdgeNs,
ulong ReservoirWindowStartNs,
ulong ReservoirMoveEvidenceCount);
public static class MimirMuninnMoveEvidenceAdapter
{
private const ulong LiveIdentityWindowNs = 60_000_000_000UL;
public const string StreamMetadataSchemaId = "mimir.muninn_move_evidence_stream_frame.v1";
public static CultMeshStreamDescriptor CreateStreamDescriptor(
string streamId,
string verseId,
string ownerPeerId,
string clockDomainId) =>
new(
streamId,
verseId,
ownerPeerId,
CultMeshStreamKind.Bytes,
new CultMeshStreamClock(clockDomainId, sourceId: streamId, confidence: 1.0, evidenceKind: "muninn-move-evidence"),
[CultMeshStreamBodyTransport.SharedMemory, CultMeshStreamBodyTransport.CultCachePage],
label: "Muninn Move evidence",
requiredAccess: CultMeshStreamAccess.Read,
maxInFlightFrames: 4,
metadataSchemaId: StreamMetadataSchemaId);
public static byte[] SerializeStreamFrame(
MimirMuninnMoveEvidenceStreamFrame frame) =>
MessagePackSerializer.Serialize(frame);
public static MimirMuninnMoveEvidenceStreamFrame DeserializeStreamFrame(
ReadOnlyMemory<byte> bytes) =>
MessagePackSerializer.Deserialize<MimirMuninnMoveEvidenceStreamFrame>(bytes);
public static bool TryAdmitLatestCultMeshFrame(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime runtime,
string producerSourceId,
string calibrationId,
string trackingSpaceId,
out MimirNativeSampleHandle handle,
out int sampleCount)
{
var admitted = TryAdmitLatestCultMeshFrame(
ring,
runtime,
producerSourceId,
calibrationId,
trackingSpaceId,
out var admission);
handle = admission?.Handle ?? default;
sampleCount = admission?.SampleCount ?? 0;
return admitted;
}
public static bool TryAdmitLatestCultMeshFrame(
CultMeshSharedMemoryFrameRing ring,
MimirNativeReservoirRuntime runtime,
string producerSourceId,
string calibrationId,
string trackingSpaceId,
out MimirMuninnMoveEvidenceAdmission? admission)
{
ArgumentNullException.ThrowIfNull(ring);
ArgumentNullException.ThrowIfNull(runtime);
admission = null;
if (!ring.TryAcquireLatestRead(out var lease))
{
return false;
}
using (lease)
{
var mimirReadAtNs = NowUnixNs();
var frame = DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]);
var samples = BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates);
if (samples.Count == 0)
{
return false;
}
var handle = runtime.AdmitMoveEvidence(producerSourceId, samples, calibrationId, trackingSpaceId);
var status = runtime.Status;
admission = new MimirMuninnMoveEvidenceAdmission(
FrameId: frame.FrameId,
ProducerPeerId: frame.ProducerPeerId,
PublishedAtNs: frame.PublishedAtNs,
MimirReadAtNs: mimirReadAtNs,
SampleCount: samples.Count,
OpticalMarkerCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker),
ControllerStateCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState),
SourceTimeMinNs: samples.Min(sample => sample.SourceTimestampNs),
SourceTimeMaxNs: samples.Max(sample => sample.SourceTimestampNs),
ArrivalMinNs: samples.Min(sample => sample.ArrivalNs),
ArrivalMaxNs: samples.Max(sample => sample.ArrivalNs),
Handle: handle,
ReservoirEdgeNs: status.EdgeNs,
ReservoirWindowStartNs: status.WindowStartNs,
ReservoirMoveEvidenceCount: status.MoveEvidenceCount.ToUInt64());
return true;
}
}
public static IReadOnlyList<MimirNativeMoveEvidenceSample> BuildNativeSamples(
IEnumerable<MuninnMoveMarkerCandidateDocument> markerCandidates,
IEnumerable<MuninnMoveControllerStateDocument> controllerStates)
{
ArgumentNullException.ThrowIfNull(markerCandidates);
ArgumentNullException.ThrowIfNull(controllerStates);
var samples = new List<MimirNativeMoveEvidenceSample>();
samples.AddRange(markerCandidates.Select(ToNativeSample));
samples.AddRange(controllerStates.Select(ToNativeSample));
return samples
.OrderBy(sample => sample.SourceTimestampNs)
.ThenBy(sample => sample.ArrivalNs)
.ThenBy(sample => sample.Sequence)
.ToArray();
}
public static IReadOnlyList<MimirMuninnMoveIdentitySnapshot> BuildIdentitySnapshots(
IEnumerable<MuninnMoveIdentityDocument> identities)
{
ArgumentNullException.ThrowIfNull(identities);
return identities
.Select(ToIdentitySnapshot)
.OrderBy(identity => identity.MoveId, StringComparer.Ordinal)
.ThenByDescending(identity => identity.ObservedAtNs)
.ToArray();
}
public static IReadOnlyList<MimirMuninnMoveRosterEntry> BuildIdentityRoster(
IEnumerable<MuninnMoveIdentityDocument> identities)
{
ArgumentNullException.ThrowIfNull(identities);
return BuildIdentitySnapshots(identities)
.GroupBy(identity => identity.MoveId, StringComparer.Ordinal)
.Select(ToRosterEntry)
.OrderBy(entry => entry.MoveId, StringComparer.Ordinal)
.ToArray();
}
public static MimirMuninnMoveIdentitySnapshot ToIdentitySnapshot(MuninnMoveIdentityDocument identity)
{
ArgumentNullException.ThrowIfNull(identity);
return new MimirMuninnMoveIdentitySnapshot(
IdentityId: identity.IdentityId,
HostId: identity.HostId,
MoveId: identity.MoveId,
SourcePath: identity.SourcePath,
BluetoothHostAddress: identity.BluetoothHostAddress,
State: identity.State,
Detail: identity.Detail,
ObservedAtNs: ObservedAtToUnixNs(identity.ObservedAt),
ControllerIdHash: Fnva64($"{identity.HostId}:{identity.MoveId}"));
}
private static MimirMuninnMoveRosterEntry ToRosterEntry(
IGrouping<string, MimirMuninnMoveIdentitySnapshot> group)
{
var orderedSnapshots = group
.OrderByDescending(identity => identity.ObservedAtNs)
.ThenBy(identity => identity.HostId, StringComparer.Ordinal)
.ThenBy(identity => identity.SourcePath, StringComparer.Ordinal)
.ToArray();
var latest = orderedSnapshots[0];
var snapshots = orderedSnapshots
.Where(identity => IsLiveIdentitySnapshot(identity, latest.ObservedAtNs))
.ToArray();
if (snapshots.Length == 0)
{
snapshots = [latest];
}
var usbHostIds = snapshots
.Where(IsUsbVisible)
.Select(identity => identity.HostId)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var pickupHostIds = snapshots
.Where(IsBluetoothPickupState)
.Select(identity => identity.HostId)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var waitingPickupHostIds = snapshots
.Where(identity => identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase))
.Select(identity => identity.HostId)
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
var hasConnectedBluetooth = snapshots.Any(identity =>
identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase));
var hasUnreachableBluetooth = snapshots.Any(identity =>
identity.State.Equals("bluetooth-unreachable", StringComparison.OrdinalIgnoreCase));
var pickupReadiness = ResolvePickupReadiness(
usbHostIds,
waitingPickupHostIds,
hasConnectedBluetooth,
hasUnreachableBluetooth);
var sourcePaths = snapshots
.Select(identity => identity.SourcePath)
.Where(path => !string.IsNullOrWhiteSpace(path))
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
return new MimirMuninnMoveRosterEntry(
MoveId: group.Key,
ControllerIdHash: Fnva64($"move:{group.Key}"),
LatestHostId: latest.HostId,
LatestState: latest.State,
StateSummary: string.Join("+", snapshots.Select(identity => identity.State)
.Where(state => !string.IsNullOrWhiteSpace(state))
.Distinct(StringComparer.Ordinal)
.OrderBy(StateRank)
.ThenBy(state => state, StringComparer.Ordinal)),
PickupReadiness: pickupReadiness,
IsBluetoothPickupReady: pickupReadiness == "pickup-ready",
BluetoothHostAddress: snapshots
.Select(identity => identity.BluetoothHostAddress)
.FirstOrDefault(address => !string.IsNullOrWhiteSpace(address)) ?? string.Empty,
HasUsbWitness: usbHostIds.Length > 0,
UsbHostIds: usbHostIds,
BluetoothPickupHostIds: pickupHostIds,
UsbBlockingHostIds: usbHostIds.Length > 0 ? usbHostIds : [],
SourcePaths: sourcePaths,
ObservedAtNs: latest.ObservedAtNs);
}
private static string ResolvePickupReadiness(
string[] usbHostIds,
string[] waitingPickupHostIds,
bool hasConnectedBluetooth,
bool hasUnreachableBluetooth)
{
if (usbHostIds.Length > 0)
{
return "usb-owned";
}
if (hasConnectedBluetooth)
{
return "bluetooth-connected";
}
if (hasUnreachableBluetooth)
{
return "bluetooth-unreachable";
}
if (waitingPickupHostIds.Length > 0)
{
return "pickup-ready";
}
return "unknown";
}
private static bool IsUsbVisible(MimirMuninnMoveIdentitySnapshot identity) =>
identity.State.Equals("usb-visible", StringComparison.OrdinalIgnoreCase);
private static bool IsLiveIdentitySnapshot(MimirMuninnMoveIdentitySnapshot identity, ulong latestObservedAtNs) =>
identity.ObservedAtNs <= 0 ||
latestObservedAtNs <= 0 ||
identity.ObservedAtNs > latestObservedAtNs ||
latestObservedAtNs - identity.ObservedAtNs <= LiveIdentityWindowNs;
private static bool IsBluetoothPickupState(MimirMuninnMoveIdentitySnapshot identity) =>
identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase) ||
identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase);
private static int StateRank(string state) =>
state.ToLowerInvariant() switch
{
"usb-visible" => 0,
"bluetooth-connected" => 1,
"bluetooth-unreachable" => 2,
"bluetooth-waiting" => 3,
"bluetooth-known" => 4,
_ => 9
};
public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveMarkerCandidateDocument marker)
{
ArgumentNullException.ThrowIfNull(marker);
var observedNs = ObservedAtToUnixNs(marker.ObservedAt);
var timestampNs = observedNs == 0
? checked((ulong)marker.FrameSequence)
: observedNs;
var witnessHash = marker.SourceIdHash != 0
? marker.SourceIdHash
: Fnva64($"{marker.HostId}:{marker.CameraId}:{marker.StreamId}");
return new MimirNativeMoveEvidenceSample(
WitnessIdHash: witnessHash,
ControllerIdHash: string.IsNullOrWhiteSpace(marker.MoveId)
? 0
: Fnva64($"{marker.HostId}:{marker.MoveId}"),
SourceTimestampNs: timestampNs,
ArrivalNs: timestampNs,
Sequence: marker.FrameSequence,
EvidenceKind: (uint)MimirNativeMoveEvidenceKind.OpticalMarker,
Flags: 0,
ImageX: marker.CenterXPx,
ImageY: marker.CenterYPx,
RadiusPx: marker.RadiusPx,
Confidence: Math.Clamp(marker.Score, 0.0f, 1.0f),
AccelX: 0.0f,
AccelY: 0.0f,
AccelZ: 0.0f,
GyroX: 0.0f,
GyroY: 0.0f,
GyroZ: 0.0f,
Trigger: 0.0f,
ButtonsMask: 0,
Reserved: 0,
Battery01: float.NaN,
Reserved1: marker.TileX,
Reserved2: marker.TileY);
}
public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveControllerStateDocument controller)
{
ArgumentNullException.ThrowIfNull(controller);
var observedNs = ObservedAtToUnixNs(controller.ObservedAt);
var sourceNs = controller.SourceTimestampNs > 0
? checked((ulong)controller.SourceTimestampNs)
: observedNs;
var vector = ExtractVectors(controller);
return new MimirNativeMoveEvidenceSample(
WitnessIdHash: Fnva64(controller.StreamId),
ControllerIdHash: Fnva64($"{controller.HostId}:{controller.MoveId}"),
SourceTimestampNs: sourceNs,
ArrivalNs: observedNs == 0 ? sourceNs : observedNs,
Sequence: controller.Sequence,
EvidenceKind: (uint)MimirNativeMoveEvidenceKind.ControllerState,
Flags: 0,
ImageX: float.NaN,
ImageY: float.NaN,
RadiusPx: float.NaN,
Confidence: 1.0f,
AccelX: vector.AccelX,
AccelY: vector.AccelY,
AccelZ: vector.AccelZ,
GyroX: vector.GyroX,
GyroY: vector.GyroY,
GyroZ: vector.GyroZ,
Trigger: Math.Clamp(controller.TriggerValue, 0.0f, 1.0f),
ButtonsMask: ButtonMask(controller.Buttons),
Reserved: 0,
Battery01: float.IsFinite(controller.Battery01)
? Math.Clamp(controller.Battery01, 0.0f, 1.0f)
: float.NaN,
Reserved1: 0,
Reserved2: 0);
}
public static ulong Fnva64(string value)
{
var hash = 14_695_981_039_346_656_037UL;
foreach (var b in System.Text.Encoding.UTF8.GetBytes(value))
{
hash ^= b;
hash *= 1_099_511_628_211UL;
}
return hash == 0 ? 1 : hash;
}
private static ulong NowUnixNs()
{
var now = DateTimeOffset.UtcNow;
var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
private static ulong ObservedAtToUnixNs(string observedAt)
{
const string unixPrefix = "unix-";
if (observedAt.StartsWith(unixPrefix, StringComparison.OrdinalIgnoreCase) &&
ulong.TryParse(observedAt[unixPrefix.Length..], out var unixSeconds))
{
return checked(unixSeconds * 1_000_000_000UL);
}
if (!DateTimeOffset.TryParse(observedAt, out var parsed))
{
return 0;
}
var secondsNs = checked((ulong)parsed.ToUnixTimeSeconds() * 1_000_000_000UL);
var tickRemainderNs = checked((ulong)(parsed.Ticks % TimeSpan.TicksPerSecond) * 100UL);
return checked(secondsNs + tickRemainderNs);
}
private static (float AccelX, float AccelY, float AccelZ, float GyroX, float GyroY, float GyroZ) ExtractVectors(
MuninnMoveControllerStateDocument controller)
{
var accel = controller.AccelerometerXyz ?? [];
var gyro = controller.GyroscopeXyz ?? [];
return (
Component(accel, 0),
Component(accel, 1),
Component(accel, 2),
Component(gyro, 0),
Component(gyro, 1),
Component(gyro, 2));
}
private static float Component(IReadOnlyList<float> values, int index) =>
index < values.Count && float.IsFinite(values[index]) ? values[index] : 0.0f;
private static uint ButtonMask(IEnumerable<string>? buttons)
{
var mask = 0u;
foreach (var button in buttons ?? [])
{
var bit = button.ToLowerInvariant() switch
{
"select" => 0,
"l3" => 1,
"r3" => 2,
"start" => 3,
"up" => 4,
"right" => 5,
"down" => 6,
"left" => 7,
"l2" => 8,
"r2" => 9,
"l1" => 10,
"r1" => 11,
"triangle" => 12,
"circle" => 13,
"cross" => 14,
"square" => 15,
"ps" => 16,
"move" => 17,
"trigger" => 18,
_ => -1
};
if (bit >= 0)
{
mask |= 1u << bit;
}
}
return mask;
}
}

View file

@ -132,6 +132,48 @@ public static class MimirNativeCaptureConfigurations
Notes = "Remote co-streamer audio witness. It decodes locally and ships typed timing/path state back over CultMesh."
};
public static MimirNativeCaptureDeviceProfile StarfirePsMoves { get; } = new(
"starfire-ps-moves",
"Starfire PS Move tracking witnesses",
MimirStreamKind.Tracking,
MimirNativeCaptureTransport.VendorSdk,
MimirCaptureClockDomain.DeviceHardware,
ChannelCount: 2,
PreferredWidth: 0,
PreferredHeight: 0,
PreferredSampleRate: 0,
PreferredFramesPerSecond: 120.0,
DefaultRollingBufferSeconds,
Required: false,
OwnedOutputs: ["local-controller-pose-observations", "performance-motion-cues"],
"USB-attached local Moves. Mimir buffers the observations; tracking code owns hardware pose estimation.");
public static MimirNativeCaptureDeviceProfile NightwingPsMoves { get; } = StarfirePsMoves with
{
Id = "nightwing-ps-moves",
DisplayName = "Nightwing PS Move tracking witnesses",
Transport = MimirNativeCaptureTransport.NetworkCultMesh,
ClockDomain = MimirCaptureClockDomain.NetworkSender,
OwnedOutputs = ["remote-controller-pose-observations", "performance-motion-cues"],
Notes = "USB-attached Nightwing Moves. Muninn on Nightwing publishes typed pose/button observations; Odin owns discovery/schema projection; Mimir owns buffering and fusion."
};
public static MimirNativeCaptureDeviceProfile OdinMoveTrackingCatalog { get; } = new(
"odin-move-tracking-catalog",
"Odin Move tracking discovery catalog",
MimirStreamKind.Tracking,
MimirNativeCaptureTransport.NetworkCultMesh,
MimirCaptureClockDomain.CanonicalFitted,
ChannelCount: 4,
PreferredWidth: 0,
PreferredHeight: 0,
PreferredSampleRate: 0,
PreferredFramesPerSecond: 120.0,
DefaultRollingBufferSeconds,
Required: false,
OwnedOutputs: ["move-stream-discovery", "tracking-schema-rendezvous"],
"Discovery/projection surface only. Odin names Starfire and Nightwing Move streams; it does not own pose capture or Mimir fusion.");
public static IReadOnlyList<MimirNativeCaptureDeviceProfile> LocalSixCameraProfiles { get; } =
[
LeapLeftIr,
@ -146,6 +188,9 @@ public static class MimirNativeCaptureConfigurations
[
.. LocalSixCameraProfiles,
ScarlettStarfire,
ScarlettRaven
ScarlettRaven,
StarfirePsMoves,
NightwingPsMoves,
OdinMoveTrackingCatalog
];
}

View file

@ -20,7 +20,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
ulong payloadHandle,
int byteLength = 0,
ReadOnlyMemory<byte> data = default,
MimirVideoFrameDescriptor? videoFrame = null)
MimirVideoFrameDescriptor? videoFrame = null,
MimirTrackingObservation? trackingObservation = null)
{
samples.Enqueue(new MimirStreamSample(
Descriptor.SourceId,
@ -32,7 +33,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
payloadHandle,
byteLength,
data,
videoFrame));
videoFrame,
TrackingObservation: trackingObservation));
}
public void PushVideoFrame(
@ -54,6 +56,25 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource
frame);
}
public void PushTrackingObservation(MimirTrackingObservation observation)
{
if (Descriptor.Kind != MimirStreamKind.Tracking)
{
throw new InvalidOperationException("Tracking observations can only be pushed into a tracking stream source.");
}
if (!string.Equals(observation.StreamId, Descriptor.SourceId, StringComparison.Ordinal))
{
throw new ArgumentException("Tracking observation stream id does not match this stream source.", nameof(observation));
}
Push(
observation.SourceTimestampNs,
observation.ArrivalTimestampNs,
payloadHandle: 0,
trackingObservation: observation);
}
public bool TryRead(out MimirStreamSample sample)
{
return samples.TryDequeue(out sample);

View file

@ -0,0 +1,329 @@
using System.Runtime.InteropServices;
using System.Text;
namespace Mimir.Runtime.Synchronization;
public enum MimirNativeReservoirSampleKind : uint
{
CameraFrame = 0,
CameraFeature = 1,
SceneRay = 2,
SurfaceClaim = 3,
MaterialClaim = 4,
AudioBlock = 5,
PhaseClaim = 6,
EventClaim = 7,
RenderPacket = 8,
MoveEvidence = 9
}
public enum MimirNativeMoveEvidenceKind : uint
{
OpticalMarker = 1,
ControllerState = 2
}
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeSampleHandle(
ulong SensorIdHash,
ulong TimestampNs,
ulong ArrivalNs,
ulong Sequence,
ulong PayloadHandle,
uint Flags,
uint Reserved);
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeMoveEvidenceSample(
ulong WitnessIdHash,
ulong ControllerIdHash,
ulong SourceTimestampNs,
ulong ArrivalNs,
ulong Sequence,
uint EvidenceKind,
uint Flags,
float ImageX,
float ImageY,
float RadiusPx,
float Confidence,
float AccelX,
float AccelY,
float AccelZ,
float GyroX,
float GyroY,
float GyroZ,
float Trigger,
uint ButtonsMask,
uint Reserved,
float Battery01,
uint Reserved1,
uint Reserved2);
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeMoveEvidenceBufferDescriptor(
ulong SampleBufferHandle,
uint SampleCount,
uint SampleStrideBytes,
ulong SourceTimeMinNs,
ulong SourceTimeMaxNs,
ulong CalibrationHash,
ulong TrackingSpaceHash);
[StructLayout(LayoutKind.Sequential)]
public readonly record struct MimirNativeRuntimeStatus(
ulong EdgeNs,
ulong WindowStartNs,
UIntPtr TotalSampleCount,
UIntPtr CameraFrameCount,
UIntPtr CameraFeatureCount,
UIntPtr SceneRayCount,
UIntPtr SurfaceClaimCount,
UIntPtr MaterialClaimCount,
UIntPtr AudioBlockCount,
UIntPtr PhaseClaimCount,
UIntPtr EventClaimCount,
UIntPtr RenderPacketCount,
UIntPtr MoveEvidenceCount);
public sealed unsafe class MimirNativeReservoirRuntime : IDisposable
{
public const int MoveEvidenceSampleStrideBytes = 112;
private readonly nint runtime;
private readonly List<PinnedMoveEvidenceBatch> batches = [];
private bool disposed;
public MimirNativeReservoirRuntime(string nativeLibraryPath, TimeSpan? window = null)
{
Native.Load(nativeLibraryPath);
runtime = Native.RuntimeCreate((ulong)(window ?? TimeSpan.FromSeconds(5)).TotalNanoseconds());
if (runtime == 0)
{
throw new InvalidOperationException("Native reservoir runtime creation failed.");
}
}
public MimirNativeRuntimeStatus Status
{
get
{
ThrowIfDisposed();
if (!Native.RuntimeStatus(runtime, out var status))
{
throw new InvalidOperationException("Native reservoir runtime status read failed.");
}
return status;
}
}
public MimirNativeSampleHandle AdmitMoveEvidence(
string producerSourceId,
IReadOnlyList<MimirNativeMoveEvidenceSample> samples,
string calibrationId,
string trackingSpaceId)
{
ThrowIfDisposed();
ArgumentException.ThrowIfNullOrWhiteSpace(producerSourceId);
ArgumentException.ThrowIfNullOrWhiteSpace(calibrationId);
ArgumentException.ThrowIfNullOrWhiteSpace(trackingSpaceId);
if (samples.Count == 0)
{
throw new ArgumentException("Move evidence batch must contain at least one sample.", nameof(samples));
}
var producer = Native.ProducerCreateForSource(
MimirNativeReservoirSampleKind.MoveEvidence,
producerSourceId);
if (producer == 0)
{
throw new InvalidOperationException($"Native Move evidence producer creation failed for '{producerSourceId}'.");
}
try
{
var batch = new PinnedMoveEvidenceBatch(samples, calibrationId, trackingSpaceId);
if (!Native.ProducerPushMoveEvidenceBuffer(
producer,
runtime,
batch.SourceTimeMaxNs,
batch.ArrivalMaxNs,
batch.DescriptorPointer,
out var handle))
{
batch.Dispose();
throw new InvalidOperationException("Native Move evidence admission failed.");
}
batches.Add(batch);
return handle;
}
finally
{
Native.ProducerDestroy(producer);
}
}
public void Dispose()
{
if (disposed)
{
return;
}
disposed = true;
foreach (var batch in batches)
{
batch.Dispose();
}
batches.Clear();
Native.RuntimeDestroy(runtime);
}
private void ThrowIfDisposed()
{
ObjectDisposedException.ThrowIf(disposed, this);
}
private sealed class PinnedMoveEvidenceBatch : IDisposable
{
private readonly GCHandle samplesHandle;
private readonly GCHandle descriptorHandle;
public PinnedMoveEvidenceBatch(
IReadOnlyList<MimirNativeMoveEvidenceSample> samples,
string calibrationId,
string trackingSpaceId)
{
var sampleArray = samples.ToArray();
SourceTimeMaxNs = sampleArray.Max(sample => sample.SourceTimestampNs);
ArrivalMaxNs = sampleArray.Max(sample => sample.ArrivalNs);
samplesHandle = GCHandle.Alloc(sampleArray, GCHandleType.Pinned);
var descriptor = new[]
{
new MimirNativeMoveEvidenceBufferDescriptor(
(ulong)samplesHandle.AddrOfPinnedObject(),
(uint)sampleArray.Length,
MoveEvidenceSampleStrideBytes,
sampleArray.Min(sample => sample.SourceTimestampNs),
SourceTimeMaxNs,
Fnva64(calibrationId),
Fnva64(trackingSpaceId))
};
descriptorHandle = GCHandle.Alloc(descriptor, GCHandleType.Pinned);
}
public ulong SourceTimeMaxNs { get; }
public ulong ArrivalMaxNs { get; }
public nint DescriptorPointer => descriptorHandle.AddrOfPinnedObject();
public void Dispose()
{
if (descriptorHandle.IsAllocated)
{
descriptorHandle.Free();
}
if (samplesHandle.IsAllocated)
{
samplesHandle.Free();
}
}
}
private static ulong Fnva64(string value)
{
var bytes = Encoding.UTF8.GetBytes(value);
const ulong offset = 14_695_981_039_346_656_037;
const ulong prime = 1_099_511_628_211;
var hash = offset;
foreach (var b in bytes)
{
hash ^= b;
hash *= prime;
}
return hash == 0 ? 1 : hash;
}
private static class Native
{
private static nint library;
private static delegate* unmanaged[Cdecl]<ulong, nint> runtimeCreate;
private static delegate* unmanaged[Cdecl]<nint, void> runtimeDestroy;
private static delegate* unmanaged[Cdecl]<nint, out MimirNativeRuntimeStatus, byte> runtimeStatus;
private static delegate* unmanaged[Cdecl]<uint, byte*, nuint, ulong, nint> producerCreateForSource;
private static delegate* unmanaged[Cdecl]<nint, void> producerDestroy;
private static delegate* unmanaged[Cdecl]<nint, nint, ulong, ulong, nint, out MimirNativeSampleHandle, byte> producerPushMoveEvidenceBuffer;
public static void Load(string path)
{
if (library != 0)
{
return;
}
if (string.IsNullOrWhiteSpace(path))
{
path = Path.Combine(AppContext.BaseDirectory, "localcast_reservoir.dll");
}
library = NativeLibrary.Load(path);
runtimeCreate = (delegate* unmanaged[Cdecl]<ulong, nint>)NativeLibrary.GetExport(library, "localcast_runtime_create");
runtimeDestroy = (delegate* unmanaged[Cdecl]<nint, void>)NativeLibrary.GetExport(library, "localcast_runtime_destroy");
runtimeStatus = (delegate* unmanaged[Cdecl]<nint, out MimirNativeRuntimeStatus, byte>)NativeLibrary.GetExport(library, "localcast_runtime_status");
producerCreateForSource = (delegate* unmanaged[Cdecl]<uint, byte*, nuint, ulong, nint>)NativeLibrary.GetExport(library, "localcast_producer_create_for_source");
producerDestroy = (delegate* unmanaged[Cdecl]<nint, void>)NativeLibrary.GetExport(library, "localcast_producer_destroy");
producerPushMoveEvidenceBuffer = (delegate* unmanaged[Cdecl]<nint, nint, ulong, ulong, nint, out MimirNativeSampleHandle, byte>)NativeLibrary.GetExport(library, "localcast_producer_push_move_evidence_buffer");
}
public static nint RuntimeCreate(ulong durationNs) => runtimeCreate(durationNs);
public static void RuntimeDestroy(nint handle)
{
if (handle != 0)
{
runtimeDestroy(handle);
}
}
public static bool RuntimeStatus(nint handle, out MimirNativeRuntimeStatus status) =>
runtimeStatus(handle, out status) != 0;
public static nint ProducerCreateForSource(MimirNativeReservoirSampleKind kind, string sourceId)
{
var bytes = Encoding.UTF8.GetBytes(sourceId);
fixed (byte* ptr = bytes)
{
return producerCreateForSource((uint)kind, ptr, (nuint)bytes.Length, 0);
}
}
public static void ProducerDestroy(nint handle)
{
if (handle != 0)
{
producerDestroy(handle);
}
}
public static bool ProducerPushMoveEvidenceBuffer(
nint producer,
nint runtime,
ulong timestampNs,
ulong arrivalNs,
nint descriptor,
out MimirNativeSampleHandle handle) =>
producerPushMoveEvidenceBuffer(producer, runtime, timestampNs, arrivalNs, descriptor, out handle) != 0;
}
}
internal static class MimirTimeSpanExtensions
{
public static long TotalNanoseconds(this TimeSpan value) =>
checked((long)(value.TotalSeconds * 1_000_000_000.0));
}

View file

@ -6,12 +6,14 @@ public enum MimirNetworkPayloadKind
RawAudioDebugWindow,
CompressedAudioProgramFeed,
CompressedVideoProgramFeed,
RawVideoDiagnosticWindow
RawVideoDiagnosticWindow,
MuxedAudioVideoProgramFeed
}
public enum MimirNetworkTransportKind
{
CultMeshTypedState,
CultMeshReliableUdpMedia,
LanUdpDatagrams,
SrtDiagnosticMedia,
WebRtcExperimental,
@ -68,6 +70,18 @@ public static class MimirNetworkTransportConfigurations
RejectionConditions: ["used-as-clock-authority", "used-for-local-hot-path"],
"Useful bridge. Not the Perfect Machine's synchronization core.");
public static MimirNetworkTransportConfiguration CultMeshProgramBridge { get; } = new(
"cultmesh-program-bridge",
"Reliable-UDP CultMesh media bridge for muxed program frames that OBS consumes through a local Starfire endpoint.",
MimirNetworkTransportKind.CultMeshReliableUdpMedia,
MimirNetworkPayloadKind.MuxedAudioVideoProgramFeed,
MayAffectClock: false,
CarriesRawMedia: true,
TargetLatencyMilliseconds: 500.0,
RequiredDocuments: ["mimir.cultmesh_media_frame"],
RejectionConditions: ["used-as-clock-authority", "unbounded-media-cache", "non-local-obs-egress"],
"Raven publishes rolling media-frame documents over CultNet reliable UDP; Starfire lowers them to a local OBS MPEG-TS endpoint.");
public static MimirNetworkTransportConfiguration WebRtcExperiment { get; } = CultMeshTimingState with
{
Id = "webrtc-experimental-media",
@ -84,6 +98,7 @@ public static class MimirNetworkTransportConfigurations
[
CultMeshTimingState,
RawAudioDebugWindow,
CultMeshProgramBridge,
SrtProgramBridge,
WebRtcExperiment
];

View file

@ -1,83 +0,0 @@
namespace Mimir.Runtime.Synchronization;
public enum MimirObsVideoPublicationKind
{
Spout2,
SharedD3D12Texture,
SrtDiagnosticBridge,
WindowCaptureDiagnostic
}
public enum MimirObsAudioPublicationKind
{
FaustStemBus,
AsioOutputPair,
WasapiLoopbackDiagnostic,
SrtDiagnosticBridge
}
public sealed record MimirObsAudioStem(
string Id,
string DisplayName,
int ChannelCount,
bool UserMixable,
bool CarriesTimingWitness,
string Notes);
public sealed record MimirObsPublicationConfiguration(
string Id,
string Description,
MimirObsVideoPublicationKind VideoKind,
string VideoSourceName,
MimirObsAudioPublicationKind AudioKind,
IReadOnlyList<MimirObsAudioStem> AudioStems,
double TargetPresentationDelaySeconds,
bool DiagnosticOnly);
public static class MimirObsPublicationConfigurations
{
public static IReadOnlyList<MimirObsAudioStem> ProgramStems { get; } =
[
new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."),
new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."),
new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."),
new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."),
new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."),
new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."),
new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.")
];
public static MimirObsPublicationConfiguration NativeProgram { get; } = new(
"native-program-spout-faust",
"Production target: Fensalir publishes synchronized video, Faust publishes aligned stems.",
MimirObsVideoPublicationKind.Spout2,
"Mimir Point Cloud",
MimirObsAudioPublicationKind.FaustStemBus,
ProgramStems,
TargetPresentationDelaySeconds: 5.0,
DiagnosticOnly: false);
public static MimirObsPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with
{
Id = "native-program-d3d12-shared-texture",
VideoKind = MimirObsVideoPublicationKind.SharedD3D12Texture,
VideoSourceName = "Mimir D3D12 Program Texture"
};
public static MimirObsPublicationConfiguration DiagnosticSrtBridge { get; } = new(
"diagnostic-srt-bridge",
"OBS-facing fallback for bridge tests only. It does not own synchronized program truth.",
MimirObsVideoPublicationKind.SrtDiagnosticBridge,
"Mimir Diagnostic SRT",
MimirObsAudioPublicationKind.SrtDiagnosticBridge,
ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(),
TargetPresentationDelaySeconds: 1.5,
DiagnosticOnly: true);
public static IReadOnlyList<MimirObsPublicationConfiguration> BuiltIn { get; } =
[
NativeProgram,
TextureInteropProgram,
DiagnosticSrtBridge
];
}

View file

@ -0,0 +1,288 @@
using System.Text.Json;
namespace Mimir.Runtime.Synchronization;
public static class MimirObsSceneImporter
{
public static MimirProgramSceneDocument ImportFile(
string path,
string? sceneName = null,
DateTimeOffset? updatedAt = null)
{
using var stream = File.OpenRead(path);
using var document = JsonDocument.Parse(stream);
return Import(document.RootElement, sceneName, updatedAt);
}
public static MimirProgramSceneDocument Import(
JsonElement root,
string? sceneName = null,
DateTimeOffset? updatedAt = null)
{
var sources = root.GetProperty("sources").EnumerateArray().ToArray();
var sourceByName = sources
.Where(source => source.TryGetProperty("name", out _))
.ToDictionary(source => source.GetProperty("name").GetString() ?? "", StringComparer.Ordinal);
var selectedSceneName = !string.IsNullOrWhiteSpace(sceneName)
? sceneName
: root.TryGetProperty("current_program_scene", out var currentProgram)
? currentProgram.GetString()
: root.TryGetProperty("current_scene", out var currentScene)
? currentScene.GetString()
: null;
if (string.IsNullOrWhiteSpace(selectedSceneName))
{
selectedSceneName = "Scene";
}
if (!sourceByName.TryGetValue(selectedSceneName, out var sceneSource))
{
throw new InvalidOperationException($"OBS scene '{selectedSceneName}' was not found.");
}
var settings = sceneSource.GetProperty("settings");
var itemElements = settings.GetProperty("items").EnumerateArray().ToArray();
var layers = new List<MimirProgramSceneLayer>(itemElements.Length);
for (var index = 0; index < itemElements.Length; index++)
{
var item = itemElements[index];
var name = item.GetProperty("name").GetString() ?? $"layer-{index}";
sourceByName.TryGetValue(name, out var source);
var sourceKind = SourceKind(source);
var crop = CombineCrop(ItemCrop(item), FilterCrop(source));
var chromaKey = FilterChromaKey(source);
var (x, y, width, height) = Bounds(item);
layers.Add(new MimirProgramSceneLayer(
LayerId: LayerId(name, index),
SourceRef: SourceRef(name, sourceKind),
SourceKind: sourceKind,
Visible: ReadBool(item, "visible", fallback: true),
X: x,
Y: y,
Width: width,
Height: height,
ZIndex: index,
Crop: crop,
ChromaKey: chromaKey));
}
var resolution = root.TryGetProperty("resolution", out var resolutionElement)
? resolutionElement
: default;
return new MimirProgramSceneDocument(
SceneId: SceneId(selectedSceneName),
UpdatedAtUtc: (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"),
CanvasWidth: ReadInt(resolution, "x", 1920),
CanvasHeight: ReadInt(resolution, "y", 1080),
Owner: "Mimir",
Layers: layers.ToArray());
}
private static string SourceKind(JsonElement source)
{
if (source.ValueKind == JsonValueKind.Undefined)
{
return "unknown";
}
return (source.GetProperty("id").GetString() ?? "") switch
{
"monitor_capture" => "monitor",
"window_capture" => "window",
"dshow_input" => "camera",
"browser_source" => "browser",
"ffmpeg_source" => "network-monitor",
"mimir_program_texture_source" => "mimir-program-texture",
"muninn_stream_source" => "muninn-stream",
_ => "unknown"
};
}
private static string SourceRef(string name, string sourceKind)
{
var normalized = LayerId(name, 0);
if (name.Contains("raven", StringComparison.OrdinalIgnoreCase))
{
return sourceKind.Contains("audio", StringComparison.OrdinalIgnoreCase)
? "muninn:raven:audio:realtek-loopback"
: "muninn:raven:monitor:primary";
}
if (sourceKind == "monitor")
{
return name.Contains("bigscreen", StringComparison.OrdinalIgnoreCase)
? "muninn:starfire:monitor:bigscreen"
: "muninn:starfire:monitor:primary";
}
if (sourceKind == "window")
{
return $"muninn:starfire:window:{normalized}";
}
if (sourceKind == "camera")
{
return $"muninn:starfire:camera:{normalized}";
}
if (sourceKind == "mimir-program-texture")
{
return "mimir:generated:program-texture";
}
if (sourceKind == "browser")
{
return $"mimir:browser:{normalized}";
}
if (sourceKind == "muninn-stream")
{
return $"muninn:starfire:stream:{normalized}";
}
return $"mimir:unknown:{normalized}";
}
private static (double X, double Y, double Width, double Height) Bounds(JsonElement item)
{
var pos = item.TryGetProperty("pos", out var posElement) ? posElement : default;
var scale = item.TryGetProperty("scale", out var scaleElement) ? scaleElement : default;
var scaleRef = item.TryGetProperty("scale_ref", out var scaleRefElement) ? scaleRefElement : default;
var bounds = item.TryGetProperty("bounds", out var boundsElement) ? boundsElement : default;
var width = ReadDouble(bounds, "x", 0.0);
var height = ReadDouble(bounds, "y", 0.0);
var scaledWidth = ReadDouble(scaleRef, "x", 0.0) * Math.Abs(ReadDouble(scale, "x", 1.0));
var scaledHeight = ReadDouble(scaleRef, "y", 0.0) * Math.Abs(ReadDouble(scale, "y", 1.0));
if (scaledWidth > width)
{
width = scaledWidth;
}
if (scaledHeight > height)
{
height = scaledHeight;
}
return (
ReadDouble(pos, "x", 0.0),
ReadDouble(pos, "y", 0.0),
width,
height);
}
private static MimirProgramCrop ItemCrop(JsonElement item) =>
new(
ReadDouble(item, "crop_left", 0.0),
ReadDouble(item, "crop_top", 0.0),
ReadDouble(item, "crop_right", 0.0),
ReadDouble(item, "crop_bottom", 0.0));
private static MimirProgramCrop FilterCrop(JsonElement source)
{
if (source.ValueKind == JsonValueKind.Undefined || !source.TryGetProperty("filters", out var filters))
{
return new MimirProgramCrop(0, 0, 0, 0);
}
foreach (var filter in filters.EnumerateArray())
{
if ((filter.GetProperty("id").GetString() ?? "") != "crop_filter")
{
continue;
}
var settings = filter.GetProperty("settings");
return new MimirProgramCrop(
ReadDouble(settings, "left", 0.0),
ReadDouble(settings, "top", 0.0),
ReadDouble(settings, "right", 0.0),
ReadDouble(settings, "bottom", 0.0));
}
return new MimirProgramCrop(0, 0, 0, 0);
}
private static MimirProgramChromaKey? FilterChromaKey(JsonElement source)
{
if (source.ValueKind == JsonValueKind.Undefined || !source.TryGetProperty("filters", out var filters))
{
return null;
}
MimirProgramChromaKey? strongest = null;
foreach (var filter in filters.EnumerateArray())
{
if ((filter.GetProperty("id").GetString() ?? "") != "chroma_key_filter")
{
continue;
}
var settings = filter.GetProperty("settings");
strongest = new MimirProgramChromaKey(
ReadUInt(settings, "key_color", 0),
ReadDouble(settings, "similarity", 0.0),
ReadDouble(settings, "smoothness", 0.0),
ReadDouble(settings, "spill", 0.0));
}
return strongest;
}
private static MimirProgramCrop CombineCrop(MimirProgramCrop item, MimirProgramCrop filter) =>
new(
item.Left + filter.Left,
item.Top + filter.Top,
item.Right + filter.Right,
item.Bottom + filter.Bottom);
private static string SceneId(string name) =>
$"mimir-scene-{LayerId(name, 0)}";
private static string LayerId(string name, int fallback)
{
var chars = name
.Trim()
.ToLowerInvariant()
.Select(character => char.IsLetterOrDigit(character) ? character : '-')
.ToArray();
var id = string.Join("-", new string(chars).Split('-', StringSplitOptions.RemoveEmptyEntries));
return string.IsNullOrWhiteSpace(id) ? $"layer-{fallback}" : id;
}
private static bool ReadBool(JsonElement element, string property, bool fallback) =>
element.ValueKind != JsonValueKind.Undefined &&
element.TryGetProperty(property, out var value) &&
value.ValueKind is JsonValueKind.True or JsonValueKind.False
? value.GetBoolean()
: fallback;
private static int ReadInt(JsonElement element, string property, int fallback) =>
element.ValueKind != JsonValueKind.Undefined &&
element.TryGetProperty(property, out var value) &&
value.TryGetInt32(out var parsed)
? parsed
: fallback;
private static double ReadDouble(JsonElement element, string property, double fallback) =>
element.ValueKind != JsonValueKind.Undefined &&
element.TryGetProperty(property, out var value) &&
value.TryGetDouble(out var parsed)
? parsed
: fallback;
private static uint ReadUInt(JsonElement element, string property, uint fallback)
{
if (element.ValueKind == JsonValueKind.Undefined ||
!element.TryGetProperty(property, out var value))
{
return fallback;
}
return value.TryGetUInt32(out var parsed)
? parsed
: value.TryGetInt64(out var signed)
? unchecked((uint)signed)
: fallback;
}
}

View file

@ -0,0 +1,262 @@
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Text.Json;
using GameCult.Mesh;
using GameCult.Networking;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
public sealed record MimirDiscoveredCultMeshInputStream(
string ProviderId,
string StreamId,
string Schema,
string Address,
string Channel,
uint ConnectionId);
public sealed class MimirOdinMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider
{
private const uint OdinSnapshotConnectionId = 0x0d1d0002;
public static MimirOdinMoveProofEvidenceRingProvider Instance { get; } = new();
private MimirOdinMoveProofEvidenceRingProvider() { }
public bool TryOpenEvidenceRing(
MimirMoveProofRuntimeConfiguration configuration,
out MimirMoveProofEvidenceRingLease? lease,
out string diagnostic)
{
ArgumentNullException.ThrowIfNull(configuration);
lease = null;
MimirMoveEvidenceRudpPump? pump = null;
CultMeshSharedMemoryFrameRing? ring = null;
try
{
var stream = Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId);
ring = new CultMeshSharedMemoryFrameRing(configuration.EvidenceStreamId, slotCount: 4, slotByteLength: 256 * 1024);
pump = new MimirMoveEvidenceRudpPump(
stream,
ring,
() => Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId));
pump.Start();
lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true, transportOwner: pump);
diagnostic = $"Odin discovered {stream.ProviderId}/{stream.StreamId}; lowering {stream.Address}/{stream.Channel} into local CultMesh ring";
return true;
}
catch (Exception error)
{
pump?.Dispose();
ring?.Dispose();
diagnostic = error.Message;
return false;
}
}
public static MimirDiscoveredCultMeshInputStream Discover(string odinCultMeshUri, string providerId, string streamId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(odinCultMeshUri);
ArgumentException.ThrowIfNullOrWhiteSpace(providerId);
ArgumentException.ThrowIfNullOrWhiteSpace(streamId);
var endpoint = CultMesh.ResolveRudpEndpoint(odinCultMeshUri);
using var transport = CultMesh.CreateRudpClient("mimir-odin-discovery", OdinSnapshotConnectionId, endpoint);
if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10)))
{
throw new TimeoutException($"Timed out connecting to Odin at {odinCultMeshUri}.");
}
var messageId = $"mimir-muninn-discovery:{Guid.NewGuid():N}";
transport.SendSchemaMessage(new CultNetSnapshotRequestMessage
{
MessageId = messageId,
SchemaIds = ["gamecult.eve.surface_state.v1"],
RecordKeys = ["surface:gamecult.network.status"]
});
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(5);
while (DateTime.UtcNow < deadline)
{
var message = transport.ReceiveSchemaMessage(TimeSpan.FromMilliseconds(250), TimeSpan.FromMilliseconds(10));
if (message is CultNetErrorMessage error)
{
throw new InvalidOperationException(error.Error);
}
if (message is CultNetSnapshotResponseRawMessage response &&
string.Equals(response.MessageId, messageId, StringComparison.Ordinal))
{
return SelectStream(response, providerId, streamId);
}
}
throw new TimeoutException($"Timed out waiting for Odin provider snapshot from {odinCultMeshUri}.");
}
public static MimirDiscoveredCultMeshInputStream SelectStream(
CultNetSnapshotResponseRawMessage response,
string providerId,
string streamId)
{
var document = response.Documents.FirstOrDefault(candidate =>
string.Equals(candidate.SchemaId, "gamecult.eve.surface_state.v1", StringComparison.Ordinal) &&
string.Equals(candidate.RecordKey, "surface:gamecult.network.status", StringComparison.Ordinal))
?? throw new InvalidOperationException("Odin snapshot omitted surface:gamecult.network.status.");
using var json = JsonDocument.Parse(MessagePackSerializer.ConvertToJson(document.Payload));
if (!json.RootElement.TryGetProperty("providerCatalog", out var providers))
{
throw new InvalidOperationException("Odin network status omitted providerCatalog.");
}
foreach (var provider in providers.EnumerateArray())
{
if (!string.Equals(Text(provider, "id"), providerId, StringComparison.Ordinal) ||
!provider.TryGetProperty("inputStreams", out var streams))
{
continue;
}
foreach (var stream in streams.EnumerateArray())
{
if (!string.Equals(Text(stream, "streamId"), streamId, StringComparison.Ordinal) ||
!string.Equals(Text(stream, "schema"), MimirMuninnMoveEvidenceAdapter.StreamMetadataSchemaId, StringComparison.Ordinal) ||
!string.Equals(Text(stream, "transport"), "cultnet.transport.rudp.v0", StringComparison.Ordinal))
{
continue;
}
return new MimirDiscoveredCultMeshInputStream(
providerId,
streamId,
Text(stream, "schema"),
Text(stream, "address"),
Text(stream, "channel"),
stream.GetProperty("connectionId").GetUInt32());
}
throw new InvalidOperationException($"Odin provider '{providerId}' has no compatible input stream '{streamId}'.");
}
throw new InvalidOperationException($"Odin has no provider '{providerId}'.");
}
private static string Text(JsonElement value, string name) =>
value.TryGetProperty(name, out var property) ? property.GetString() ?? "" : "";
}
internal sealed class MimirMoveEvidenceRudpPump : IDisposable
{
private static readonly TimeSpan SourceStaleAfter = TimeSpan.FromSeconds(2);
private readonly MimirDiscoveredCultMeshInputStream initialStream;
private readonly CultMeshSharedMemoryFrameRing ring;
private readonly Func<MimirDiscoveredCultMeshInputStream> rediscover;
private readonly CancellationTokenSource stopping = new();
private readonly ManualResetEventSlim started = new(false);
private Thread? thread;
private Exception? startupError;
public MimirMoveEvidenceRudpPump(
MimirDiscoveredCultMeshInputStream initialStream,
CultMeshSharedMemoryFrameRing ring,
Func<MimirDiscoveredCultMeshInputStream> rediscover)
{
this.initialStream = initialStream;
this.ring = ring;
this.rediscover = rediscover;
}
public void Start()
{
thread = new Thread(Run) { IsBackground = true, Name = "mimir-muninn-move-evidence" };
thread.Start();
if (!started.Wait(TimeSpan.FromSeconds(6)))
{
throw new TimeoutException($"Timed out starting Move evidence receiver for {initialStream.Address}.");
}
if (startupError is not null)
{
throw new InvalidOperationException(
$"Could not connect Odin-discovered Move evidence stream {initialStream.StreamId} at {initialStream.Address}: {startupError.Message}",
startupError);
}
}
private void Run()
{
var stream = initialStream;
while (!stopping.IsCancellationRequested)
{
try
{
ReceiveSession(stream);
}
catch (Exception error) when (!stopping.IsCancellationRequested)
{
if (!started.IsSet)
{
startupError = error;
started.Set();
return;
}
Console.Error.WriteLine(
$"Mimir Move evidence receiver reconnecting stream={stream.StreamId} address={stream.Address}: {error.Message}");
if (stopping.Token.WaitHandle.WaitOne(TimeSpan.FromMilliseconds(250)))
{
return;
}
try
{
stream = rediscover();
}
catch (Exception discoveryError)
{
Console.Error.WriteLine(
$"Mimir Move evidence rediscovery failed provider={stream.ProviderId} stream={stream.StreamId}: {discoveryError.Message}");
}
}
}
}
private void ReceiveSession(MimirDiscoveredCultMeshInputStream stream)
{
using var transport = CultMesh.CreateRudpClient(
"mimir-move-evidence-consumer",
stream.ConnectionId,
CultMesh.ResolveRudpEndpoint($"rudp://{stream.Address}"));
if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(5)))
{
throw new TimeoutException($"Timed out connecting to {stream.Address}.");
}
transport.Send("hid.subscribe", Encoding.UTF8.GetBytes(JsonSerializer.Serialize(new { streamId = stream.StreamId })));
started.Set();
var lastFrameAt = DateTime.UtcNow;
while (!stopping.IsCancellationRequested)
{
transport.PollResends();
var frame = transport.ReceiveOnce();
if (frame is null)
{
if (DateTime.UtcNow - lastFrameAt >= SourceStaleAfter)
{
throw new TimeoutException(
$"No Move evidence frame arrived for {SourceStaleAfter.TotalSeconds:0.#} seconds.");
}
Thread.Sleep(2);
continue;
}
if (!string.Equals(frame.ChannelId, stream.Channel, StringComparison.Ordinal))
{
continue;
}
var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(frame.Payload);
if (!string.Equals(decoded.FrameId, stream.StreamId, StringComparison.Ordinal) &&
!decoded.FrameId.StartsWith($"{stream.StreamId}:", StringComparison.Ordinal))
{
continue;
}
lastFrameAt = DateTime.UtcNow;
ring.TryPublishCopy(frame.Payload, decoded.PublishedAtNs, durationNs: 0, out _);
}
}
public void Dispose()
{
stopping.Cancel();
thread?.Join(TimeSpan.FromSeconds(2));
started.Dispose();
stopping.Dispose();
}
}

View file

@ -20,7 +20,7 @@ public sealed record MimirPerfectMachineManifest(
IReadOnlyList<MimirAudioFieldConfiguration> AudioFields,
IReadOnlyList<MimirVisualFusionConfiguration> VisualFields,
IReadOnlyList<MimirComputeOffloadConfiguration> ComputePlans,
IReadOnlyList<MimirObsPublicationConfiguration> Publications,
IReadOnlyList<MimirProgramPublicationConfiguration> Publications,
IReadOnlyList<MimirMachineAssemblyPlan> AssemblyPlans);
public static class MimirPerfectMachineManifestFactory
@ -48,6 +48,6 @@ public static class MimirPerfectMachineManifestFactory
MimirAudioFieldConfigurations.BuiltIn,
MimirVisualFusionConfigurations.BuiltIn,
MimirComputeOffloadConfigurations.BuiltIn,
MimirObsPublicationConfigurations.BuiltIn,
MimirProgramPublicationConfigurations.BuiltIn,
MimirMachineAssemblyPlans.BuiltIn);
}

View file

@ -4,10 +4,11 @@ public enum MimirPerfectMachineNodeRole
{
StarfireAuthority,
RavenRemote,
MuninnHost,
PhoneWitness,
MicrocontrollerWitness,
CalibrationBench,
ObsProgramHost
MimirProgramPublisher
}
public enum MimirAudioFieldModel
@ -38,7 +39,7 @@ public sealed record MimirPerfectMachineProfile(
IReadOnlyList<MimirComputeBackend> ComputeBackends,
bool EmitsWitness,
bool OwnsCanonicalClock,
bool PublishesObsProgram,
bool PublishesProgram,
TimeSpan RollingWindow,
string[] RequiredCultMeshDocuments);
@ -46,7 +47,7 @@ public static class MimirPerfectMachineProfiles
{
public static MimirPerfectMachineProfile StarfireAuthority { get; } = new(
"starfire-authority",
"Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Fensalir/Faust lowering, OBS-facing program output.",
"Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Mimir composition, Fensalir/Faust lowering, Eve operator surfaces, and program publication.",
MimirPerfectMachineNodeRole.StarfireAuthority,
MimirBioacousticDecoderConfiguration.BaselineMfccIndex,
MimirAlignmentActuatorProfile.SixSourceFaust,
@ -54,7 +55,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute, MimirComputeBackend.FaustNativeDsp],
EmitsWitness: true,
OwnsCanonicalClock: true,
PublishesObsProgram: true,
PublishesProgram: true,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
@ -74,7 +75,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
@ -93,7 +94,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(3),
RequiredCultMeshDocuments:
[
@ -111,7 +112,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(2),
RequiredCultMeshDocuments:
[
@ -129,7 +130,7 @@ public static class MimirPerfectMachineProfiles
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute],
EmitsWitness: true,
OwnsCanonicalClock: false,
PublishesObsProgram: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(8),
RequiredCultMeshDocuments:
[
@ -138,22 +139,45 @@ public static class MimirPerfectMachineProfiles
"mimir.acoustic_path_state.v1"
]);
public static MimirPerfectMachineProfile ObsProgramHost { get; } = new(
"obs-program-host",
"Program-output profile: consumes aligned stems/spatial bed and publishes OBS-facing surfaces without becoming sync authority.",
MimirPerfectMachineNodeRole.ObsProgramHost,
public static MimirPerfectMachineProfile MuninnHost { get; } = new(
"muninn-host",
"Capture host profile: Muninn observes local screens, windows, loopbacks, cameras, and device health, publishes typed stream capabilities/media, and does not compose the final program.",
MimirPerfectMachineNodeRole.MuninnHost,
MimirBioacousticDecoderConfiguration.CompactFastIndex,
MimirAlignmentActuatorProfile.SixSourceFaust,
MimirAudioFieldModel.AlignedStems,
[MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesProgram: false,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
"muninn.telemetry_surface.v1",
"muninn.capture_stream.v1",
"mimir.move_tracking_observation.v1",
"mimir.cultmesh_media_frame.v1"
]);
public static MimirPerfectMachineProfile MimirProgramPublisher { get; } = new(
"mimir-program-publisher",
"Program-output profile: consumes selected Muninn/Mimir streams, owns scene composition and Eve controls, and publishes the program locally and through Yggdrasil without OBS authority.",
MimirPerfectMachineNodeRole.MimirProgramPublisher,
MimirBioacousticDecoderConfiguration.BaselineMfccIndex,
MimirAlignmentActuatorProfile.SixSourceFaust,
MimirAudioFieldModel.SourceBasedSpatialBus,
[MimirComputeBackend.FaustNativeDsp, MimirComputeBackend.D3D12Compute],
EmitsWitness: false,
OwnsCanonicalClock: false,
PublishesObsProgram: true,
PublishesProgram: true,
RollingWindow: TimeSpan.FromSeconds(5),
RequiredCultMeshDocuments:
[
"mimir.acoustic_path_state.v1",
"mimir.actuator_state.v1"
"mimir.actuator_state.v1",
"mimir.program_scene.v1",
"mimir.program_output.v1",
"mimir.eve_operator_surface.v1"
]);
public static IReadOnlyList<MimirPerfectMachineProfile> All { get; } =
@ -163,6 +187,7 @@ public static class MimirPerfectMachineProfiles
PhoneWitness,
MicrocontrollerWitness,
CalibrationBench,
ObsProgramHost
MuninnHost,
MimirProgramPublisher
];
}

View file

@ -0,0 +1,141 @@
namespace Mimir.Runtime.Synchronization;
public enum MimirProgramVideoPublicationKind
{
FensalirProgramTexture,
SharedD3D12Texture,
CultMeshMediaBody,
SrtDiagnosticBridge,
ObsCompatibilityAdapter
}
public enum MimirProgramAudioPublicationKind
{
FaustStemBus,
AsioOutputPair,
CultMeshMediaBody,
WasapiLoopbackDiagnostic,
ObsCompatibilityAdapter
}
public enum MimirProgramControlPublicationKind
{
CultMeshCommandSurface
}
public sealed record MimirProgramAudioStem(
string Id,
string DisplayName,
int ChannelCount,
bool UserMixable,
bool CarriesTimingWitness,
string Notes);
public sealed record MimirProgramControlSurface(
string Id,
string Description,
MimirProgramControlPublicationKind Kind,
bool CanPreviewProgram,
bool CanEditComposition,
bool CanSelectSources,
bool CanShowStats);
public sealed record MimirProgramPublicationConfiguration(
string Id,
string Description,
MimirProgramVideoPublicationKind VideoKind,
string VideoSurfaceName,
MimirProgramAudioPublicationKind AudioKind,
IReadOnlyList<MimirProgramAudioStem> AudioStems,
IReadOnlyList<MimirProgramControlSurface> ControlSurfaces,
bool PublishesSiteFeed,
string SitePublisherRoute,
double TargetPresentationDelaySeconds,
bool DiagnosticOnly);
public static class MimirProgramPublicationConfigurations
{
public static IReadOnlyList<MimirProgramAudioStem> ProgramStems { get; } =
[
new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."),
new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."),
new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."),
new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."),
new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."),
new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."),
new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.")
];
public static IReadOnlyList<MimirProgramControlSurface> OperatorSurfaces { get; } =
[
new(
"cultmesh-command-surface",
"Typed CultMesh command surface consumed by trusted devices; Mimir remains the scene authority.",
MimirProgramControlPublicationKind.CultMeshCommandSurface,
CanPreviewProgram: false,
CanEditComposition: true,
CanSelectSources: true,
CanShowStats: true)
];
public static MimirProgramPublicationConfiguration NativeProgram { get; } = new(
"mimir-native-program",
"Production target: Mimir owns composition, Fensalir publishes synchronized program video, Faust publishes aligned stems, and Eve exposes control/preview/stat surfaces.",
MimirProgramVideoPublicationKind.FensalirProgramTexture,
"Mimir Program Texture",
MimirProgramAudioPublicationKind.FaustStemBus,
ProgramStems,
OperatorSurfaces,
PublishesSiteFeed: false,
SitePublisherRoute: "",
TargetPresentationDelaySeconds: 5.0,
DiagnosticOnly: false);
public static MimirProgramPublicationConfiguration YggdrasilSiteProgram { get; } = NativeProgram with
{
Id = "mimir-yggdrasil-site-program",
Description = "Site publishing target: a Yggdrasil-facing daemon consumes the same Mimir program output and publishes it to the public site without owning a second composition.",
VideoKind = MimirProgramVideoPublicationKind.CultMeshMediaBody,
AudioKind = MimirProgramAudioPublicationKind.CultMeshMediaBody,
PublishesSiteFeed = true,
SitePublisherRoute = "cultmesh://yggdrasil/gamecult.site/mimir/program"
};
public static MimirProgramPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with
{
Id = "native-program-d3d12-shared-texture",
VideoKind = MimirProgramVideoPublicationKind.SharedD3D12Texture,
VideoSurfaceName = "Mimir D3D12 Program Texture"
};
public static MimirProgramPublicationConfiguration ObsCompatibilityAdapter { get; } = new(
"obs-compatibility-adapter",
"Temporary OBS compatibility sink for preview/testing while Mimir owns composition and Yggdrasil/site publication matures.",
MimirProgramVideoPublicationKind.ObsCompatibilityAdapter,
"Mimir OBS Compatibility Feed",
MimirProgramAudioPublicationKind.ObsCompatibilityAdapter,
ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(),
OperatorSurfaces,
PublishesSiteFeed: false,
SitePublisherRoute: "",
TargetPresentationDelaySeconds: 1.5,
DiagnosticOnly: true);
public static MimirProgramPublicationConfiguration DiagnosticSrtBridge { get; } = ObsCompatibilityAdapter with
{
Id = "diagnostic-srt-bridge",
Description = "Legacy SRT bridge for bridge tests only. It is not scene, composition, or broadcast authority.",
VideoKind = MimirProgramVideoPublicationKind.SrtDiagnosticBridge,
AudioKind = MimirProgramAudioPublicationKind.WasapiLoopbackDiagnostic,
VideoSurfaceName = "Mimir Diagnostic SRT"
};
public static IReadOnlyList<MimirProgramPublicationConfiguration> BuiltIn { get; } =
[
NativeProgram,
YggdrasilSiteProgram,
TextureInteropProgram,
ObsCompatibilityAdapter,
DiagnosticSrtBridge
];
}

View file

@ -32,8 +32,9 @@ public static class MimirReservoirConfigurations
new("audio_block", "PCM/native audio block", "Mimir.Runtime/native audio", RetainsHistory: false, "Timing and field evidence."),
new("phase_claim", "audio phase/group-delay evidence", "Mimir.Runtime", RetainsHistory: false, "Calibration/normalization input."),
new("event_claim", "timeline anchor", "Mimir.Runtime", RetainsHistory: false, "Bioacoustic/passive/calibration events."),
new("move_evidence", "PS Move marker/controller evidence buffer", "Mimir native reservoir", RetainsHistory: false, "Compute-upload buffer of Muninn optical and controller witnesses before Mimir pose fusion."),
new("surface_claim", "spatial surface observation", "Fensalir", RetainsHistory: false, "Fusion candidate."),
new("render_packet", "OBS-facing packet", "Fensalir/Faust", RetainsHistory: false, "Output-ready view.")
new("render_packet", "Mimir program packet", "Mimir/Fensalir/Faust", RetainsHistory: false, "Output-ready program view.")
];
public static MimirReservoirConfiguration ManagedRuntime { get; } = new(

View file

@ -8,6 +8,8 @@ public sealed class MimirRuntimeConfiguration
public IReadOnlyList<MimirStreamSourceFactory> SourceFactories { get; init; } = [];
public IReadOnlyList<MimirMoveProofRuntimeConfiguration> MoveProofSources { get; init; } = [];
public IReadOnlyList<IMimirStreamSource> CreateSources()
{
return SourceFactories
@ -53,6 +55,16 @@ public sealed class MimirRuntimeConfiguration
.Where(factory => factory != null)
.Cast<MimirStreamSourceFactory>()
.ToArray();
var moveProofSources = model.MoveProof
.Where(source => source.Enabled)
.ToArray();
var moveProofErrors = moveProofSources
.SelectMany((source, index) => source.Validate().Select(error => $"MoveProof[{index}].{error}"))
.ToArray();
if (moveProofErrors.Length > 0)
{
throw new InvalidOperationException($"Invalid Mimir runtime Move proof configuration: {string.Join("; ", moveProofErrors)}");
}
return new MimirRuntimeConfiguration
{
@ -63,6 +75,7 @@ public sealed class MimirRuntimeConfiguration
Streams = streams,
},
SourceFactories = sourceFactories,
MoveProofSources = moveProofSources,
};
}
@ -224,6 +237,8 @@ public sealed class MimirRuntimeConfigFile
public MimirAudioSyncConfig? AudioSync { get; set; }
public List<MimirStreamConfig> Streams { get; set; } = [];
public List<MimirMoveProofRuntimeConfiguration> MoveProof { get; set; } = [];
}
public sealed class MimirAudioSyncConfig

View file

@ -0,0 +1,210 @@
using GameCult.Caching;
using MessagePack;
namespace Mimir.Runtime.Synchronization;
public enum MimirSensorCalibrationSessionPhase
{
Collecting,
Fitting,
Validating,
Promoted,
Rejected,
Canceled
}
[CultDocument("mimir.sensor_calibration_session", "mimir.sensor_calibration_session.v1")]
[MessagePackObject]
public sealed record MimirSensorCalibrationSessionDocument(
[property: Key(0)][property: CultName] string SessionId,
[property: Key(1)] string TrackingSpaceId,
[property: Key(2)] string Owner,
[property: Key(3)] string StartedAtUtc,
[property: Key(4)] double MaximumDurationSeconds,
[property: Key(5)] string[] RequiredWandIds,
[property: Key(6)] string[] SensorIds,
[property: Key(7)] MimirSensorCalibrationAcceptance Acceptance,
[property: Key(8)] MimirSensorCalibrationSessionPhase Phase,
[property: Key(9)] MimirSensorCalibrationSensorProgress[] Sensors,
[property: Key(10)] string[] MissingRequirements,
[property: Key(11)] string Detail);
[MessagePackObject]
public sealed record MimirSensorCalibrationAcceptance(
[property: Key(0)] int GridColumns,
[property: Key(1)] int GridRows,
[property: Key(2)] int MinimumOccupiedGridCellsPerSensor,
[property: Key(3)] int MinimumObservationsPerSensor,
[property: Key(4)] int MinimumSameFrameCorrespondences,
[property: Key(5)] int MinimumDistinctWands,
[property: Key(6)] double MinimumRadiusRatio,
[property: Key(7)] double HeldOutFraction,
[property: Key(8)] double MaximumMedianReprojectionErrorPx,
[property: Key(9)] double MaximumP95ReprojectionErrorPx,
[property: Key(10)] double MaximumAssociationSkewMilliseconds);
[MessagePackObject]
public sealed record MimirSensorCalibrationSensorProgress(
[property: Key(0)] string SensorId,
[property: Key(1)] int ObservationCount,
[property: Key(2)] int OccupiedGridCells,
[property: Key(3)] int GridCellCount,
[property: Key(4)] int DistinctWandCount,
[property: Key(5)] double MinimumRadiusPx,
[property: Key(6)] double MaximumRadiusPx,
[property: Key(7)] int[] MissingGridCells,
[property: Key(8)] bool CollectionComplete);
[CultDocument("mimir.sensor_calibration_receipt", "mimir.sensor_calibration_receipt.v1")]
[MessagePackObject]
public sealed record MimirSensorCalibrationReceiptDocument(
[property: Key(0)][property: CultName] string ReceiptId,
[property: Key(1)] string SessionId,
[property: Key(2)] string TrackingSpaceId,
[property: Key(3)] string CompletedAtUtc,
[property: Key(4)] MimirSensorCalibrationSessionPhase Verdict,
[property: Key(5)] int TrainingCorrespondenceCount,
[property: Key(6)] int HeldOutCorrespondenceCount,
[property: Key(7)] MimirSensorCalibrationFitResiduals Residuals,
[property: Key(8)] MimirMoveFusionRigCalibration? PromotedCalibration,
[property: Key(9)] string[] RejectionReasons,
[property: Key(10)] string Solver);
[MessagePackObject]
public sealed record MimirSensorCalibrationFitResiduals(
[property: Key(0)] double TrainingMedianReprojectionErrorPx,
[property: Key(1)] double TrainingP95ReprojectionErrorPx,
[property: Key(2)] double HeldOutMedianReprojectionErrorPx,
[property: Key(3)] double HeldOutP95ReprojectionErrorPx,
[property: Key(4)] double MedianAssociationSkewMilliseconds,
[property: Key(5)] double MaximumAssociationSkewMilliseconds,
[property: Key(6)] double ConditionEstimate,
[property: Key(7)] int Iterations,
[property: Key(8)] double TrainingInlierFraction = 0.0,
[property: Key(9)] double HeldOutInlierFraction = 0.0);
public static class MimirSensorCalibrationSessions
{
public static MimirSensorCalibrationSessionDocument CreateFourWandOpticalSession(
IEnumerable<string> wandIds,
IEnumerable<string> sensorIds,
DateTimeOffset? startedAt = null,
double maximumDurationSeconds = 120.0)
{
var wands = wandIds.Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.Ordinal).Order().ToArray();
var sensors = sensorIds.Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.Ordinal).Order().ToArray();
if (wands.Length != 4) throw new ArgumentException("A four-wand calibration session requires exactly four stable wand IDs.", nameof(wandIds));
if (sensors.Length < 2) throw new ArgumentException("Optical calibration requires at least two sensors.", nameof(sensorIds));
if (!double.IsFinite(maximumDurationSeconds) || maximumDurationSeconds <= 0.0) throw new ArgumentOutOfRangeException(nameof(maximumDurationSeconds));
var start = startedAt ?? DateTimeOffset.UtcNow;
return new MimirSensorCalibrationSessionDocument(
$"mimir-sensor-calibration:{start.ToUnixTimeMilliseconds()}",
"mimir-stage-space",
"Mimir.Runtime",
start.ToString("O"),
maximumDurationSeconds,
wands,
sensors,
new MimirSensorCalibrationAcceptance(
GridColumns: 4,
GridRows: 3,
MinimumOccupiedGridCellsPerSensor: 9,
MinimumObservationsPerSensor: 1_000,
MinimumSameFrameCorrespondences: 500,
MinimumDistinctWands: 4,
MinimumRadiusRatio: 1.5,
HeldOutFraction: 0.2,
MaximumMedianReprojectionErrorPx: 1.5,
MaximumP95ReprojectionErrorPx: 4.0,
MaximumAssociationSkewMilliseconds: 20.0),
MimirSensorCalibrationSessionPhase.Collecting,
sensors.Select(sensor => new MimirSensorCalibrationSensorProgress(
sensor, 0, 0, 12, 0, double.NaN, double.NaN, Enumerable.Range(0, 12).ToArray(), false)).ToArray(),
["wand-motion-not-yet-captured"],
"Wave one or two uniquely identified wands at a time through the shared sensor volume; Mimir accumulates successive passes until every wand and sensor has sufficient coverage.");
}
public static MimirSensorCalibrationSessionDocument UpdateCollection(
MimirSensorCalibrationSessionDocument session,
MoveVisibilityWindowReceipt evidence,
int imageWidth,
int imageHeight,
DateTimeOffset? observedAt = null)
{
ArgumentNullException.ThrowIfNull(session);
ArgumentNullException.ThrowIfNull(evidence);
if (session.Phase != MimirSensorCalibrationSessionPhase.Collecting) return session;
if (imageWidth <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth));
if (imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageHeight));
var acceptance = session.Acceptance;
var requiredWands = session.RequiredWandIds.ToHashSet(StringComparer.Ordinal);
var progress = session.SensorIds.Select(sensorId =>
{
var observations = evidence.Observations.Where(value =>
string.Equals(value.CameraId, sensorId, StringComparison.Ordinal) &&
requiredWands.Contains(value.MoveId) &&
float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) &&
float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f).ToArray();
var occupied = observations.Select(value => GridCell(
value.CenterXPx, value.CenterYPx, imageWidth, imageHeight,
acceptance.GridColumns, acceptance.GridRows)).Distinct().Order().ToArray();
var missing = Enumerable.Range(0, acceptance.GridColumns * acceptance.GridRows)
.Except(occupied).ToArray();
var minimumRadius = observations.Length == 0 ? double.NaN : observations.Min(value => value.RadiusPx);
var maximumRadius = observations.Length == 0 ? double.NaN : observations.Max(value => value.RadiusPx);
var radiusRatio = observations.Length == 0 ? 0.0 : maximumRadius / minimumRadius;
var distinctWands = observations.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count();
var complete = observations.Length >= acceptance.MinimumObservationsPerSensor &&
occupied.Length >= acceptance.MinimumOccupiedGridCellsPerSensor &&
distinctWands >= acceptance.MinimumDistinctWands &&
radiusRatio >= acceptance.MinimumRadiusRatio;
return new MimirSensorCalibrationSensorProgress(
sensorId, observations.Length, occupied.Length,
acceptance.GridColumns * acceptance.GridRows, distinctWands,
minimumRadius, maximumRadius, missing, complete);
}).ToArray();
var correspondenceCount = evidence.Correspondences.Count(value =>
requiredWands.Contains(value.MoveId) &&
value.AbsoluteSkewNs <= acceptance.MaximumAssociationSkewMilliseconds * 1_000_000.0 &&
string.Equals(value.First.FrameId, value.Second.FrameId, StringComparison.Ordinal));
var missingRequirements = new List<string>();
foreach (var sensor in progress)
{
if (sensor.ObservationCount < acceptance.MinimumObservationsPerSensor)
missingRequirements.Add($"{sensor.SensorId}:observations-{sensor.ObservationCount}-of-{acceptance.MinimumObservationsPerSensor}");
if (sensor.OccupiedGridCells < acceptance.MinimumOccupiedGridCellsPerSensor)
missingRequirements.Add($"{sensor.SensorId}:grid-cells-{sensor.OccupiedGridCells}-of-{acceptance.MinimumOccupiedGridCellsPerSensor}");
if (sensor.DistinctWandCount < acceptance.MinimumDistinctWands)
missingRequirements.Add($"{sensor.SensorId}:wands-{sensor.DistinctWandCount}-of-{acceptance.MinimumDistinctWands}");
var radiusRatio = sensor.ObservationCount == 0 ? 0.0 : sensor.MaximumRadiusPx / sensor.MinimumRadiusPx;
if (radiusRatio < acceptance.MinimumRadiusRatio)
missingRequirements.Add($"{sensor.SensorId}:radius-ratio-{radiusRatio:F2}-of-{acceptance.MinimumRadiusRatio:F2}");
}
if (correspondenceCount < acceptance.MinimumSameFrameCorrespondences)
missingRequirements.Add($"same-frame-correspondences-{correspondenceCount}-of-{acceptance.MinimumSameFrameCorrespondences}");
var elapsed = (observedAt ?? DateTimeOffset.UtcNow) - DateTimeOffset.Parse(session.StartedAtUtc);
var collectionComplete = progress.All(value => value.CollectionComplete) &&
correspondenceCount >= acceptance.MinimumSameFrameCorrespondences;
var expired = elapsed.TotalSeconds >= session.MaximumDurationSeconds;
var phase = collectionComplete
? MimirSensorCalibrationSessionPhase.Fitting
: expired ? MimirSensorCalibrationSessionPhase.Rejected : MimirSensorCalibrationSessionPhase.Collecting;
var detail = phase switch
{
MimirSensorCalibrationSessionPhase.Fitting => "Collection complete; Mimir owns the retained evidence and may begin fitting.",
MimirSensorCalibrationSessionPhase.Rejected => "Collection window expired before coverage requirements were met; no calibration was promoted.",
_ => "Collection active. Continue sweeping one or two wands at a time through missing image regions and depth ranges."
};
return session with { Phase = phase, Sensors = progress, MissingRequirements = missingRequirements.ToArray(), Detail = detail };
}
private static int GridCell(float x, float y, int width, int height, int columns, int rows)
{
var column = Math.Clamp((int)(x * columns / width), 0, columns - 1);
var row = Math.Clamp((int)(y * rows / height), 0, rows - 1);
return row * columns + column;
}
}

View file

@ -4,6 +4,7 @@ public enum MimirStreamKind
{
Audio,
Video,
Tracking,
}
public enum MimirStreamOrigin

View file

@ -11,4 +11,5 @@ public readonly record struct MimirStreamSample(
int ByteLength = 0,
ReadOnlyMemory<byte> Data = default,
MimirVideoFrameDescriptor? VideoFrame = null,
MimirAudioBlockDescriptor? AudioBlock = null);
MimirAudioBlockDescriptor? AudioBlock = null,
MimirTrackingObservation? TrackingObservation = null);

View file

@ -51,6 +51,15 @@ public sealed class MimirSynchronizationHub : IDisposable
public IReadOnlyList<MimirChirpBinCalibrationProfile> AudioChirpBinCalibrationProfiles =>
audioSynchronization.LastCalibrationProfiles;
public IReadOnlyList<MimirTrackingObservation> TrackingObservations =>
Buffers.Buffers
.Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking)
.Select(buffer => buffer.Latest?.TrackingObservation)
.Where(observation => observation != null)
.Cast<MimirTrackingObservation>()
.OrderBy(observation => observation.StreamId, StringComparer.Ordinal)
.ToArray();
public MimirChirpBinCodebookPlan? ChirpBinEmissionPlan =>
chirpBinCalibrationModel?.EmissionPlan;
@ -87,7 +96,8 @@ public sealed class MimirSynchronizationHub : IDisposable
{
var audio = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio);
var video = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Video);
return $"{video} video / {audio} audio buffers";
var tracking = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking);
return $"{video} video / {audio} audio / {tracking} tracking buffers";
}
public IReadOnlyList<MimirAudioSynchronizationReport> AnalyzeAudioSynchronization(

View file

@ -14,8 +14,10 @@ public sealed class MimirSynchronizationSettings
var streams = new List<MimirStreamDescriptor>();
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.LocalDevice);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.LocalDevice);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.LocalDevice);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.Network);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.Network);
AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.Network);
return new MimirSynchronizationSettings
{

View file

@ -0,0 +1,156 @@
using MessagePack;
using GameCult.Caching;
namespace Mimir.Runtime.Synchronization;
public enum MimirTrackingObservationKind
{
Unknown,
PsMoveController,
Hand,
Headset,
Tool,
}
[MessagePackObject]
public readonly record struct MimirVector3Snapshot(
[property: Key(0)] double X,
[property: Key(1)] double Y,
[property: Key(2)] double Z);
[MessagePackObject]
public readonly record struct MimirQuaternionSnapshot(
[property: Key(0)] double X,
[property: Key(1)] double Y,
[property: Key(2)] double Z,
[property: Key(3)] double W);
[MessagePackObject]
public sealed record MimirTrackingButtonSnapshot(
[property: Key(0)] string Id,
[property: Key(1)] bool Pressed,
[property: Key(2)] double Value);
[MessagePackObject]
[CultDocument("mimir.move_tracking_observation", "mimir.move_tracking_observation.v1")]
public sealed record MimirTrackingObservation(
[property: Key(0)]
[property: CultName]
string ObservationId,
[property: Key(1)] string StreamId,
[property: Key(2)] string DeviceId,
[property: Key(3)] MimirTrackingObservationKind Kind,
[property: Key(4)] string ProducerId,
[property: Key(5)] string HostId,
[property: Key(6)] string DiscoveryProviderId,
[property: Key(7)] string TrackingSpaceId,
[property: Key(8)] string CalibrationId,
[property: Key(9)] long SourceTimestampNs,
[property: Key(10)] long ArrivalTimestampNs,
[property: Key(11)] ulong Sequence,
[property: Key(12)] MimirVector3Snapshot PositionMeters,
[property: Key(13)] MimirQuaternionSnapshot Orientation,
[property: Key(14)] MimirVector3Snapshot LinearVelocityMetersPerSecond,
[property: Key(15)] MimirVector3Snapshot AngularVelocityRadiansPerSecond,
[property: Key(16)] double Confidence,
[property: Key(17)] double LatencyMilliseconds,
[property: Key(18)] double Battery01,
[property: Key(19)] MimirTrackingButtonSnapshot[] Buttons)
{
public static MimirTrackingObservation PsMove(
string streamId,
string deviceId,
ulong sequence,
long sourceTimestampNs,
long arrivalTimestampNs,
MimirVector3Snapshot positionMeters,
MimirQuaternionSnapshot orientation,
MimirVector3Snapshot linearVelocityMetersPerSecond,
MimirVector3Snapshot angularVelocityRadiansPerSecond,
double confidence,
string calibrationId,
string trackingSpaceId = "nightwing-move-space",
string producerId = "muninn:nightwing:move",
string hostId = "nightwing",
string discoveryProviderId = "odin",
double latencyMilliseconds = 0.0,
double battery01 = double.NaN,
MimirTrackingButtonSnapshot[]? buttons = null) =>
new(
$"{streamId}:{sequence}",
streamId,
deviceId,
MimirTrackingObservationKind.PsMoveController,
producerId,
hostId,
discoveryProviderId,
trackingSpaceId,
calibrationId,
sourceTimestampNs,
arrivalTimestampNs,
sequence,
positionMeters,
orientation,
linearVelocityMetersPerSecond,
angularVelocityRadiansPerSecond,
Math.Clamp(confidence, 0.0, 1.0),
Math.Max(0.0, latencyMilliseconds),
double.IsFinite(battery01) ? Math.Clamp(battery01, 0.0, 1.0) : double.NaN,
buttons ?? []);
}
[CultDocument("mimir.move_controller_pose", "mimir.move_controller_pose.v1")]
[MessagePackObject]
public sealed record MimirMoveControllerPoseDocument(
[property: Key(0)]
[property: CultName]
string PoseId,
[property: Key(1)] string WandId,
[property: Key(2)] string TrackingSpaceId,
[property: Key(3)] string CalibrationId,
[property: Key(4)] string FusionAuthorityId,
[property: Key(5)] long SourceTimestampNs,
[property: Key(6)] long EstimatedAtNs,
[property: Key(7)] ulong Sequence,
[property: Key(8)] MimirVector3Snapshot PositionMeters,
[property: Key(9)] MimirQuaternionSnapshot Orientation,
[property: Key(10)] MimirVector3Snapshot LinearVelocityMetersPerSecond,
[property: Key(11)] MimirVector3Snapshot AngularVelocityRadiansPerSecond,
[property: Key(12)] double Confidence,
[property: Key(13)] double LatencyMilliseconds,
[property: Key(14)] double Battery01,
[property: Key(15)] MimirTrackingButtonSnapshot[] Buttons,
[property: Key(16)] string[] EvidenceStreamIds,
[property: Key(17)] string[] EvidenceKinds,
[property: Key(18)] string ConsumerContract)
{
public static MimirMoveControllerPoseDocument FromObservation(
MimirTrackingObservation observation,
string wandId,
string[] evidenceStreamIds,
string[] evidenceKinds,
string calibrationId,
string trackingSpaceId = "mimir-stage-space",
string fusionAuthorityId = "mimir.runtime.move-fusion",
string consumerContract = "fensalir.move-controller-input") =>
new(
$"{wandId}:{observation.Sequence}",
wandId,
trackingSpaceId,
calibrationId,
fusionAuthorityId,
observation.SourceTimestampNs,
observation.ArrivalTimestampNs,
observation.Sequence,
observation.PositionMeters,
observation.Orientation,
observation.LinearVelocityMetersPerSecond,
observation.AngularVelocityRadiansPerSecond,
Math.Clamp(observation.Confidence, 0.0, 1.0),
Math.Max(0.0, observation.LatencyMilliseconds),
double.IsFinite(observation.Battery01) ? Math.Clamp(observation.Battery01, 0.0, 1.0) : double.NaN,
observation.Buttons,
evidenceStreamIds,
evidenceKinds,
consumerContract);
}

View file

@ -0,0 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>Mimir.VerseRecorder</AssemblyName>
<RootNamespace>Mimir.VerseRecorder</RootNamespace>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>

View file

@ -0,0 +1,151 @@
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
var options = VerseRecorderOptions.Parse(args);
Directory.CreateDirectory(options.OutputDirectory);
var runId = DateTimeOffset.Now.ToString("yyyyMMdd-HHmmss");
var runDirectory = Path.Combine(options.OutputDirectory, runId);
Directory.CreateDirectory(runDirectory);
var jsonlPath = Path.Combine(runDirectory, "observations.jsonl");
var sessionPath = Path.Combine(runDirectory, "session.json");
await File.WriteAllTextAsync(
sessionPath,
JsonSerializer.Serialize(
new
{
kind = "mimir.verse_recorder_session.v1",
runId,
options.Url,
startedAt = DateTimeOffset.Now.ToString("O"),
options.Seconds,
observations = jsonlPath,
},
new JsonSerializerOptions { WriteIndented = true }));
Console.Error.WriteLine($"Mimir Verse recorder subscribing to {options.Url}");
Console.Error.WriteLine($"Writing {jsonlPath}");
using var stopping = new CancellationTokenSource();
if (options.Seconds > 0)
{
stopping.CancelAfter(TimeSpan.FromSeconds(options.Seconds));
}
Console.CancelKeyPress += (_, eventArgs) =>
{
eventArgs.Cancel = true;
stopping.Cancel();
};
var count = 0L;
var lastMeter = DateTimeOffset.UtcNow;
await using var output = new StreamWriter(new FileStream(jsonlPath, FileMode.Create, FileAccess.Write, FileShare.Read), Encoding.UTF8);
while (!stopping.IsCancellationRequested)
{
using var socket = new ClientWebSocket();
try
{
await socket.ConnectAsync(options.Url, stopping.Token);
while (socket.State == WebSocketState.Open && !stopping.IsCancellationRequested)
{
var text = await ReceiveTextAsync(socket, stopping.Token);
if (text == null)
{
break;
}
await output.WriteLineAsync(text);
count++;
if (DateTimeOffset.UtcNow - lastMeter >= TimeSpan.FromSeconds(options.MeterSeconds))
{
await output.FlushAsync();
Console.Error.WriteLine($"verse-recorder observations={count} path={jsonlPath}");
lastMeter = DateTimeOffset.UtcNow;
}
}
}
catch (OperationCanceledException) when (stopping.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
Console.Error.WriteLine($"verse-recorder reconnect after {ex.GetType().Name}: {ex.Message}");
try
{
await Task.Delay(TimeSpan.FromSeconds(options.ReconnectSeconds), stopping.Token);
}
catch (OperationCanceledException)
{
break;
}
}
}
await output.FlushAsync();
Console.Error.WriteLine($"verse-recorder complete observations={count} path={jsonlPath}");
static async Task<string?> ReceiveTextAsync(ClientWebSocket socket, CancellationToken stopping)
{
var buffer = new byte[64 * 1024];
using var memory = new MemoryStream();
while (true)
{
var result = await socket.ReceiveAsync(buffer, stopping);
if (result.MessageType == WebSocketMessageType.Close)
{
return null;
}
memory.Write(buffer, 0, result.Count);
if (result.EndOfMessage)
{
return Encoding.UTF8.GetString(memory.ToArray());
}
}
}
internal sealed record VerseRecorderOptions(
Uri Url,
string OutputDirectory,
double Seconds,
double ReconnectSeconds,
double MeterSeconds)
{
public static VerseRecorderOptions Parse(string[] args)
{
var defaultOutput = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.MyVideos),
"Mimir",
"VerseCaptures");
return new VerseRecorderOptions(
new Uri(ParseString(args, "--url", "ws://127.0.0.1:8796/eve/periwinkle/subscribe")),
ParseString(args, "--out-dir", defaultOutput),
ParseDouble(args, "--seconds", 0.0),
ParseDouble(args, "--reconnect-seconds", 2.0),
ParseDouble(args, "--meter-seconds", 5.0));
}
private static string ParseString(IReadOnlyList<string> args, string name, string fallback)
{
for (var index = 0; index < args.Count - 1; index++)
{
if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase))
{
return args[index + 1];
}
}
return fallback;
}
private static double ParseDouble(IReadOnlyList<string> args, string name, double fallback)
{
return double.TryParse(ParseString(args, name, ""), out var value)
? value
: fallback;
}
}

View file

@ -119,3 +119,90 @@
{"ts": "2026-05-25T13:56:03+01:00", "type": "decision", "status": "ok", "note": "Fensalir spectrum visualization authority split: Mimir now emits AquariumBufferFieldFrame spline tube fields as the real buffer-domain contract, with tangent/curvature/normal/derivative appearance and probe policy for future compute SDF splat reservoir lowering. AquariumSplineFrame remains only the immediate Catmull-Rom preview witness until compute lowering owns rendering."}
{"ts": "2026-05-25T15:55:00+01:00", "type": "decision", "status": "ok", "note": "Fensalir/Mimir spectrum visualization moved from C# spline authority to DSL-authored texture field authority. config/mimir-spectrum-field.aquafield binds a rolling spectrum texture, declares surface graph nodes, and asks for texture-dimension splinefield lowering. Fensalir now compiles Aquarium field DSL text, uploads texture samples/program packets, and ProgramMode=3 in D3D12FractalReservoirCompute emits stochastic reservoir splats from the texture field. Current canary is mechanically through the reservoir at 131072 splats with low CPU record time, but visual framing/material is still not the final beautiful spectrum surface."}
{"ts": "2026-05-25T17:43:25+01:00", "type": "decision", "status": "ok", "note": "Fensalir direct splinefield correction: texture-derived tube trails are direct SDF envelope surfaces, not point-splat fractal dust. AquariumFieldScriptCompiler supports direct; D3D12Renderer now derives AquariumSplineFrame tube-envelope trails from BufferFieldFrame texture spline programs, the spline pipeline declares triangle topology, and D3D12SplinePS writes color, metadata/control, and reservoirGuide outputs. Mimir capture artifacts/runtime/mimir-direct-sdf-tube-spectrum.png proves the DSL-authored ASIO spectrum appears through the direct SDF tube path with ~0.64 ms record CPU in the canary."}
{"ts": "2026-06-08T09:29:43+01:00", "type": "decision", "status": "ok", "note": "Added CultMesh media bridge for Raven program feed: Raven sender publishes rolling mimir.cultmesh_media_frame MPEG-TS chunks over CultNet reliable UDP to Yggdrasil 10.77.0.1:3075; Starfire receiver lowers raven-primary-av to local UDP 127.0.0.1:5200 for OBS. Yggdrasil relay and Starfire receiver were started; Raven SSH timed out, so sender must be launched on Raven directly."}
{"ts": "2026-06-08T09:59:22+01:00", "type": "verification", "status": "blocked", "note": "CultMesh media bridge verification: Yggdrasil UFW was opened for 3075/udp on wg-gamecult and Starfire receiver now connects from 10.77.0.2. Synthetic sender on Yggdrasil connects and sends chunks without User Not Verified after CultLib pre-verification schema-v0 policy patch, but relay cache does not receive mimir.cultmesh_media_frame updates and Starfire receiver logs no frames. Remaining blocker is CultNet document put/apply path, not WireGuard routing or OBS endpoint."}
{"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."}
{"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."}
{"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."}
{"ts": "2026-06-10T17:38:22+01:00", "type": "verification", "status": "ok", "note": "Restored and launched the narrow Nightwing Move tracking witness path. scripts/start-nightwing-move-tracking.ps1 started Mimir.EveSensorReceiver on Starfire port 8796, staged Nightwing Eye/Move Python tools, launched Nightwing worker PID 1131578 with --move-light move-usb=/dev/hidraw1:#35ff6c, and VerseRecorder C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures\\20260610-173549 recorded mimir.psmove_light_state.v1 HID success plus mimir.move_controller_observation_state.v1 blobs from both ps3-eye-0 and ps3-eye-1."}
{"ts": "2026-06-10T18:11:18+01:00", "type": "verification", "status": "ok", "note": "Started the Starfire-local USB Move light owner. Windows sees Sony VID_054C/PID_03D5 with col01 output reports; scripts/start-starfire-move-light.ps1 launched Mimir.PsMoveProbe PID 27344 with --rgb #35ff6c --hold-seconds 0 --refresh-ms 250, and artifacts/runtime/starfire-move-light-20260610-180943/move-light-events.jsonl shows repeated mimir.psmove_usb_pulse_event.v1 hold writes succeeding with ok rgb=#35FF6C bytes=49."}
{"ts": "2026-06-10T20:33:54+01:00", "type": "implementation", "status": "ok", "note": "Added native/move_tracker as Rust-orchestrated PS Move optical candidate extraction: Rust owns dispatch/FFI/CPU mirror; HLSL owns 16px-tile luma reduction. Scope is marker candidates only, before calibration/triangulation/IMU fusion/prediction."}
{"ts": "2026-06-10T20:49:06+01:00", "type": "decision", "status": "ok", "note": "Corrected Move optical candidate ownership: the Rust/HLSL tracker moved out of Mimir and into Odin's Muninn organ at E:\\Projects\\Odin\\crates\\muninn-move-tracker. Mimir consumes muninn.move_marker_candidate.v1 streams and no longer owns raw optical extraction."}
{"ts": "2026-06-11T07:57:05+01:00", "type": "decision", "status": "ok", "note": "PS Move structured light ownership moved to Odin/Muninn: Mimir requests muninn.move_light_command.v1 over CultNet/CultMesh; Muninn serve consumes host-addressed pending commands, writes local HID report 0x06, and updates command state. Nightwing direct HID launcher path is opt-in smoke/bootstrap only."}
{"ts": "2026-06-11T16:59:31+01:00", "type": "decision", "status": "ok", "note": "Move sensor fusion authority moved to Mimir: Muninn publishes source-local marker candidates and controller/IMU/button state, Odin discovers/projects schemas, Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, latency, and mimir.move_controller_pose.v1 for Fensalir interaction."}
{"ts": "2026-06-11T18:47:43+01:00", "type": "implementation", "status": "ok", "note": "Added native reservoir Move evidence admission: LocalcastMoveEvidenceSample and LocalcastMoveEvidenceBufferDescriptor define a fixed compute-upload ABI, SampleKind move_evidence enters the shared rolling window, malformed buffers are rejected before live admission, and Mimir docs/maps now route Muninn marker/controller witnesses through native buffers before Mimir pose fusion and Fensalir consumption."}
{"ts": "2026-06-11T19:27:48+01:00", "type": "implementation", "status": "ok", "note": "Connected Mimir.Runtime to the native Move evidence reservoir: MimirNativeReservoirRuntime loads localcast_reservoir, pins Move evidence sample/descriptor batches, admits them through localcast_producer_push_move_evidence_buffer, and BufferSmoke --move-native-reservoir-smoke proves a marker plus controller-state batch enters the native move_evidence view."}
{"ts": "2026-06-11T19:58:56+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir-side Muninn Move evidence normalization: muninn.move_marker_candidate.v1 and muninn.move_controller_state.v1 documents now map into native Move evidence samples, and BufferSmoke --muninn-move-evidence-smoke proves two optical candidates plus one controller-state record enter the native move_evidence reservoir view."}
{"ts": "2026-06-11T20:26:19+01:00", "type": "correction", "status": "ok", "note": "Corrected the Move witness live path from CultCache polling to CultMesh streaming: MimirMuninnMoveEvidenceAdapter now creates/consumes Muninn Move shared-memory stream frames, BufferSmoke --muninn-move-cultmesh-stream-smoke proves a CultMesh latest-frame read lease normalizes into native move_evidence, and CultCache remains receipt/debug state rather than the hot tracking feed."}
{"ts": "2026-06-12T14:08:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveFusion as the first calibrated Move pose owner and MimirMovePoseStream as the realtime output frame: native Muninn Move evidence samples are associated against USB controller state, calibrated multi-camera orb rays triangulate position into mimir.move_controller_pose.v1, controller buttons/trigger/gyro are carried forward, orientation is explicitly marked imu-unresolved, and BufferSmoke --move-fusion-smoke proves the positive calibrated pose path, the negative no-calibration/no-pose guard, and a CultMesh pose-stream roundtrip."}
{"ts": "2026-06-12T14:45:00+01:00", "type": "implementation", "status": "ok", "note": "Added typed Move calibration preflight state as mimir.move_calibration_protocol.v1. BufferSmoke --move-calibration-protocol-smoke writes artifacts/move-calibration/protocol.cc and proves the Starfire/Nightwing protocol requires both Muninn Move evidence streams plus Mimir pose output, includes optional Periwinkle motion/camera witnesses, defines stillness/sweep/validation phases, and names the four required derived outputs before IMU orientation can become authority."}
{"ts": "2026-06-12T15:05:00+01:00", "type": "correction", "status": "ok", "note": "Updated the Move calibration preflight witness from Periwinkle to the USB-connected Quest headset plus left/right controllers. ADB sees serial 1WMHHB68PG1515 but reports unauthorized, so headset-side USB debugging approval is required before Quest pose capture can become live evidence. The protocol treats Quest poses as an optional external VR reference frame around the Move, not as a Mimir fusion owner."}
{"ts":"2026-06-12T15:17:22.2571251+01:00","type":"verification","status":"ok","note":"Quest USB access is now authorized for calibration preflight: adb devices -l reports serial 1WMHHB68PG1515 as Quest_2/hollywood. Added and ran --quest-usb-preflight-smoke, which wrote mimir.quest_usb_preflight.v1 with poseCaptureStatus=usb-authorized-no-pose-bridge-yet. ADB access proves USB/device identity only; headset/controller pose samples still require a Quest/OpenXR witness bridge before Mimir can use Quest as external calibration evidence."}
{"ts":"2026-06-12T16:05:00+01:00","type":"correction","status":"ok","note":"Moved Quest access ownership out of Mimir and under Odin/Muninn. Mimir's local mimir.quest_usb_preflight path was deleted; Mimir now treats muninn.quest_access.v1 and later muninn.quest_pose_frame.v1 as input evidence. Brokkr owns Unity editor routing for Quest input/pose consumption and warp-corrected Unity play-mode video toward Muninn's Quest video input surface."}
{"ts":"2026-06-16T00:00:00+01:00","type":"correction","status":"ok","note":"Reoriented Mimir's daemon-swarm map after Odin/CultLib RUDP work: Raven media body transport remains src/Mimir.CultMeshMedia over CultNet/CultMesh reliable UDP with OBS-local UDP as egress only, while Eve dashboard/browser reference health now publishes idunn.daemon_health over cultnet.transport.rudp.v0 as self-witness state. Odin owns accepted Verse/service discovery and Idunn owns keepalive decisions; Mimir health publishers do not own lifecycle."}
{"ts":"2026-06-16T00:05:00+01:00","type":"correction","status":"ok","note":"Corrected the transport model: CultLib has added authorized RUDP helpers across runtimes and Odin/Idunn now treats daemon truth as typed CultCache/CultMesh state carried over cultnet.transport.rudp.v0. Mimir should not assume TCP is needed for any daemon-truth path once the runtime has RUDP support; HTTP/WebSocket/TCP are client lowerings, compatibility probes, or migration debt. Health is separate from media by authority, not because it needs a separate TCP transport."}
{"ts":"2026-06-16T00:12:00+01:00","type":"correction","status":"ok","note":"Source audit found Mimir.CultMeshMedia still using older C# CultMesh.StartNodeAsync/CultMesh.ConnectClient entrypoints while CultLib now exposes C# CultMesh.CreateRudpServer/CreateRudpClient/CreateRudpClientForAuthorizedPeer. Updated maps to treat the media bridge's current transport path as migration debt: typed program/media documents should move to explicit CultNet RUDP helpers, with OBS-local UDP preserved only as compatibility egress."}
{"ts": "2026-06-16T18:05:56+01:00", "type": "verification", "status": "ok", "note": "Mimir.CultMeshMedia migrated from old CultMesh node/client transport to explicit CultNet RUDP relay/send/recv path; local smoke artifacts/rudp-media-smoke-20260616-180404 proved 32 MPEG-TS chunks reached Starfire-local UDP sink over RUDP (packets=32 bytes=42112)."}
{"ts": "2026-06-17T03:45:44+01:00", "type": "decision", "status": "ok", "note": "Muninn Move identity is now a typed provenance/provisioning lane: Mimir consumes muninn.move_identity.v1 as identity snapshots while controller-state/marker evidence remains the only native reservoir motion evidence. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and --muninn-move-evidence-smoke."}
{"ts": "2026-06-17T05:33:48+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir Move identity roster projection from muninn.move_identity.v1. The roster groups identity records by physical Move so a blind USB rotation can show Starfire/Nightwing USB witnesses and Nightwing Bluetooth waiting/connected pickup state together while keeping identity-only rows out of the native motion reservoir. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and Mimir.Runtime build."}
{"ts": "2026-06-17T07:32:58+01:00", "type": "verification", "status": "ok", "note": "2026-06-17 Raven OBS feed owner is actual Muninn. Canonical C:\\Meta\\Odin\\Muninn\\muninn.exe was stale (3853312 bytes) while the working bridge build existed as muninn-fixed4.exe (2020352 bytes). Replacing canonical muninn.exe with the latest release build restored Raven->Starfire SRT delivery: single-target proof on port 5520 produced H.264 1920x1080 video plus AAC stereo in capture.ts, and dual-target proof on ports 5521/5524 delivered 262144-byte MPEG-TS captures to both listeners."}
{"ts": "2026-06-17T10:10:40+01:00", "type": "verification", "status": "ok", "note": "Mimir BufferSmoke now reads live Starfire/Nightwing Muninn CultCache move_identity records with opaque companion schema stubs; live roster smoke saw A8 as usb-owned via Starfire USB plus Nightwing waiting, F5 bluetooth-connected, and A3/A6 pickup-ready."}
{"ts": "2026-06-17T11:32:31+01:00", "type": "verification", "status": "ok", "note": "After Odin beed394 deployed to Nightwing, Muninn restart pipeline claims USB Moves, launches with --move-host plus discovered --move-state sources, preserves A3 as usb-visible instead of Bluetooth overwrite, publishes BlueZ pickup failure detail for A6/A8, and Mimir live roster smoke over Starfire+Nightwing stores reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 pickup-ready, and A8 usb-owned on Starfire with Nightwing waiting."}
{"ts": "2026-06-17T12:00:49+01:00", "type": "verification", "status": "ok", "note": "After Odin 9ea3ba2 and Mimir 32bd680c, Muninn publishes A6/A8 as bluetooth-unreachable when BlueZ returns br-connection-create-socket/Host is down; Mimir live roster smoke over Starfire+Nightwing now reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 bluetooth-unreachable, and A8 usb-owned on Starfire with Nightwing Bluetooth unreachable rather than pickup-ready."}
{"ts": "2026-06-17T12:14:17+01:00", "type": "verification", "status": "ok", "note": "Odin 8e16218 deployed to Nightwing. Muninn now publishes bluetooth-unreachable Move identities with explicit BlueZ host-down wake guidance: A6/A8 detail says br-connection-create-socket, likely asleep/off-air, wake with PS button or reattach USB. This clarifies the remaining physical blocker; automatic Bluetooth pickup is still unproven for those controllers."}
{"ts": "2026-06-17T12:39:58+01:00", "type": "verification", "status": "ok", "note": "Muninn ca4e49b deployed on Nightwing: released local USB-visible Move IDs now clear Bluetooth pickup backoff before the centralized BlueZ connect loop. cargo test -p muninn-daemon passed 41/41; Nightwing health reports store_age=0s and four identities. Live A6/A8 still classify bluetooth-unreachable with br-connection-create-socket, so physical wake/reattach remains the blocker."}
{"ts": "2026-06-17T13:16:04+01:00", "type": "verification", "status": "ok", "note": "Odin ef4c51a deployed to Starfire Muninn. Windows Move identity status now reads paired Bluetooth host from the sibling PS Move pairing collection when --move-host is omitted: no-flag move-identity-status reports A8 as usb-visible with bluetooth_host=5C:93:A2:9C:A8:A8, and the claim log shows controller=00:07:04:a8:00:d0 host_after=5c:93:a2:9c:a8:a8. Installed Starfire Muninn hash matches target/debug hash 6062D174FF9DF281C098B35E84E7D3488606F6EC8842D6B5F1C8B6ACBD30C7E7."}
{"ts": "2026-06-17T13:27:55+01:00", "type": "verification", "status": "ok", "note": "Mimir live Muninn roster smoke after Odin ef4c51a and refreshed Nightwing store loaded Starfire+Nightwing identity stores 2/2, identities=6, roster=4, readinessCoherent=True. A8 now resolves as usb-owned on Starfire with bluetoothHost=5C:93:A2:9C:A8:A8 while Nightwing contributes bluetooth-unreachable for the same move id; F5 is bluetooth-connected, A3 is usb-owned on Nightwing, and A6 remains bluetooth-unreachable."}
{"ts": "2026-06-17T14:05:52+01:00", "type": "verification", "status": "ok", "note": "After Odin b6f8119 deployed to Nightwing and Starfire, Muninn live source merge drops stale configured platform Move sources when platform discovery has live sources, so static --move-state joystick paths no longer keep unplugged Moves USB-owned. Mimir roster derivation now ignores identity snapshots older than the live identity window relative to each Move's newest observation. Focused identity smoke reports staleReleased=bluetooth-waiting/pickup-ready. Cross-host live roster smoke with refreshed Nightwing store loaded stores=2/2, identities=6, roster=4, readinessCoherent=True: F5 bluetooth-connected, A3 usb-owned on Starfire with Nightwing unreachable witness, A6 bluetooth-unreachable, A8 usb-owned on Nightwing."}
{"ts": "2026-07-04T00:00:00Z", "type": "transport-cut", "status": "ok", "note": "Archived the old Nightwing/EVE witness socket lane: Mimir.EveSensorReceiver, Mimir.VerseRecorder, tools/nightwing_typed_witness_publisher.py, and scripts/start-nightwing-move-tracking.ps1 now fail closed. Eye/Move witness publication and recording must use typed CultMesh/CultNet documents or CultMesh stream frames discovered through Odin; the old receiver/recorder/subscription path no longer owns live transport."}
{"ts": "2026-07-08T22:06:14+01:00", "type": "decision", "status": "ok", "note": "Move proof spine cut: Odin Muninn evidence frames now carry canonical muninn.move_marker_candidate.v1 records instead of a forced empty marker slice, with a bright-Y8 tracker unit proving non-empty marker candidates serialize into the Mimir-compatible CultMesh frame. Mimir TryAdmitLatestCultMeshFrame now exposes a typed admission receipt with frame id, producer, publish/read times, sample kind counts, source/arrival ranges, native handle, and reservoir edge/window. Verified cargo test -p muninn-daemon, dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --muninn-move-cultmesh-stream-smoke against native reservoir debug DLL."}
{"ts": "2026-07-08T22:10:10+01:00", "type": "decision", "status": "ok", "note": "Move optical extraction seam sharpened in Odin: Muninn daemon now has MoveMarkerFrameSource plus extract/publish functions that take one source-local Y8 frame, run muninn-move-tracker, build canonical muninn.move_marker_candidate.v1 records, and publish them through the same Mimir-compatible move-evidence stream. Full cargo test -p muninn-daemon passed. Live raw camera-frame ownership is still missing in Muninn serve; do not revive the archived Python/EVE witness lane to fake it."}
{"ts": "2026-07-08T22:18:44+01:00", "type": "decision", "status": "ok", "note": "Muninn serve now has a first daemon-owned Move marker camera producer path: --move-marker-camera <camera-id>=<device-path> configures a source-local camera, the serve loop polls a platform frame reader, Unix builds use a V4L2 YUYV single-frame reader that converts to compact Y8, and the shared extraction/publish seam emits canonical muninn.move_marker_candidate.v1 records into the Mimir-compatible move-evidence stream. Verified on Windows with cargo test -p muninn-daemon (148 passed) and marker-camera unit tests; Linux cross-target check could not run because x86_64-unknown-linux-gnu target is not installed, and Nightwing hardware live proof remains pending."}
{"ts": "2026-07-08T22:21:57+01:00", "type": "verification", "status": "ok", "note": "Correction to the earlier Move marker camera evidence: after installing/using the x86_64-unknown-linux-gnu Rust target, Muninn's V4L2 YUYV Move marker camera producer compiled with cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu. cargo test -p muninn-daemon passes 148 tests. Nightwing hardware live proof remains pending, so this proves owned producer code and target compatibility, not calibrated live pose."}
{"ts": "2026-07-08T22:28:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofSurface as the first observer-only Fensalir-visible Move proof surface: it combines a Muninn evidence admission receipt, Mimir pose stream frame, and Fensalir presentation/probe timestamp into mimir.move_proof_surface.v1, lowers a dev AquariumSplineFrame probe, preserves source/publish/read/admit/fusion/present timing, and refuses to treat single-ray fallback as full pose. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-surface-smoke, and BufferSmoke --move-fusion-smoke. Live Nightwing hardware, calibrated camera witnesses, and actual Fensalir presented-frame capture remain pending."}
{"ts": "2026-07-08T22:34:00+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofPipeline as the shared commit primitive for one Move proof frame: it reads a Muninn CultMesh shared-memory evidence frame, admits samples to native move_evidence, runs MimirMoveFusion, emits the mimir:starfire move-pose frame, creates mimir.move_proof_surface.v1, and lowers the Fensalir spline probe in one pass. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --move-proof-pipeline-smoke against the native reservoir debug DLL. This proves the reusable Mimir owner path; live Nightwing hardware, calibrated camera witnesses, and actual presented-frame capture remain pending."}
{"ts": "2026-07-08T22:41:32+01:00", "type": "implementation", "status": "ok", "note": "Attached the Move proof surface to the actual MimirRuntime/Fensalir frame path: MimirRuntime.PublishMoveProofSurface stores the latest Mimir-owned mimir.move_proof_surface.v1 document, CreateFrame composes its observer-only move-proof-* splines into AquariumFrame.Scene.SplineFrame, and the UI exposes a display-only proof readout. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke against the native reservoir debug DLL. Live Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."}
{"ts": "2026-07-08T22:48:42+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofRuntimeDriver as the runtime bridge from a Muninn CultMesh shared-memory evidence ring to MimirRuntime.PublishMoveProofSurface. The driver derives mimir:starfire evidence/pose ids and fensalir:starfire presented-frame id from the actual Muninn frame suffix, calls MimirMoveProofPipeline, and suppresses duplicate frame admission. Verified dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke against the native reservoir debug DLL, --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke. This proves the runtime-owned ring-to-visible-probe path with synthetic ring input; real Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."}
{"ts": "2026-07-08T22:55:54+01:00", "type": "verification", "status": "ok", "note": "Added and verified the Fensalir presented-frame proof smoke. MIMIR_MOVE_PROOF_DEV_SURFACE gates a synthetic calibrated proof surface through the same MimirRuntime.PublishMoveProofSurface path, and BufferSmoke --move-proof-presented-frame-smoke runs Mimir.App headless, captures artifacts/runtime/move-proof-presented-frame-smoke.png, and pixel-checks the rendered proof probe. Latest run captured fensalir:starfire:presented-frame:79 for proof mimir:starfire:move-proof:79 with bytes=11260 and brightPixels=1069; adjacent runtime-driver, runtime-frame, proof-surface, and proof-pipeline smokes pass. This proves the renderer/probe layer for the named proof chain, not live Nightwing hardware or calibrated field evidence."}
{"ts": "2026-07-12T10:26:42+01:00", "type": "verification", "status": "ok", "note": "Added MimirMoveProofRuntimeConfiguration as the typed config/validation surface for Move proof driver creation. The runtime config now accepts enabled MoveProof sources, validates named Muninn evidence stream, native reservoir path, Mimir/Fensalir frame prefixes, fusion authority, consumer contract, and at least two calibrated camera witnesses, and creates a driver only when the supplied CultMesh ring stream id matches. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-config-smoke, --move-proof-runtime-driver-smoke, --move-proof-pipeline-smoke, and --move-proof-presented-frame-smoke. The config smoke printed configValid=True invalidDefaultErrors=2 and the same sequence chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79. This proves config plus in-process ring flow, not live Nightwing hardware, calibrated field cameras, or cross-process ring opening."}
{"ts": "2026-07-12T10:51:37+01:00", "type": "verification", "status": "ok", "note": "MimirRuntime now owns configured Move proof activation instead of leaving MoveProof config as a passive readout. On scene-ready it asks IMimirMoveProofEvidenceRingProvider for the configured Muninn evidence ring, opens the configured native reservoir, creates/registers MimirMoveProofRuntimeDriver, retains resources, and records activation status. The default provider fails explicitly because the current C# CultMesh ring is in-process only. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, and BufferSmoke --move-proof-runtime-activation-smoke. The activation smoke injected an in-process provider and printed activationActive=True while preserving the explicit chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79 with verdict=FullPose. This proves runtime config activation and status, not live Nightwing/Starfire hardware or cross-process ring opening."}
{"ts": "2026-07-12T11:01:09+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_runtime_activation.v1 as the typed readiness/proof-spine surface for configured Move proof sources. It reports stream id, provider kind, active/driver state, reservoir path, calibrated camera ids, latest same-stream proof ids/verdict, and unavailable-provider diagnostics. MimirRuntime now exposes CreateMoveProofRuntimeActivationDocuments and disposes a native reservoir if activation fails before driver retention. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-smoke, and --move-proof-runtime-activation-surface-smoke against the native reservoir debug DLL. The new smoke proves active injected-ring and inactive default-provider documents; real Nightwing/Starfire hardware and cross-process CultMesh ring opening remain pending."}
{"ts": "2026-07-12T11:09:32+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_evidence_frame_snapshot.v1 and MimirConfiguredMoveProofEvidenceRingProvider as an explicit one-copy fallback for captured Muninn Move evidence frames while C# CultMesh lacks a cross-process shared-memory opener. EvidenceSnapshotPath snapshots are metadata-checked against the encoded Muninn frame payload, copied into an owned in-process ring, and consumed by the same runtime proof driver/pipeline. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-surface-smoke, and --move-proof-runtime-snapshot-smoke against the native reservoir debug DLL. This supports field capture/replay and does not complete the live Nightwing/Starfire ring proof."}
{"ts": "2026-07-12T11:16:48+01:00", "type": "implementation", "status": "ok", "note": "Odin/Muninn now has a producer-side --move-evidence-snapshot <path> writer for Mimir Move proof capture/replay. The same Muninn Move evidence publisher that writes controller/marker frames to the CultMesh stream ring can write a mimir.move_proof_evidence_frame_snapshot.v1 MessagePack tuple containing snapshot id, evidence stream id, frame id, producer peer id, published ns, captured ns, and the encoded Muninn frame payload. Verified cargo test -p muninn-daemon, focused move_evidence_snapshot_writes_mimir_compatible_frame_artifact, and cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu (existing dead-code warnings only). This gives Nightwing a concrete field artifact path for Mimir's EvidenceSnapshotPath provider; it is still one-copy replay support, not the final live cross-process ring/page proof."}
{"ts": "2026-07-12T11:44:53+01:00", "type": "verification", "status": "ok", "note": "Nightwing field Move proof snapshot replay reached Mimir FullPose through the explicit proof spine using artifact artifacts/runtime/nightwing-move-proof-20260712-1041-aggregate.mpack. Muninn d4f229a on Nightwing produced a bundled evidence frame with two PS3 Eye marker witnesses and one /dev/input/js0 controller state after increasing move evidence slot bytes to 32768. Mimir BufferSmoke --move-proof-runtime-snapshot-file-smoke replayed it through the configured snapshot provider and shared runtime path as muninn:nightwing:move-evidence:1 -> mimir:starfire:move-evidence:1 -> mimir:starfire:move-pose:1 -> fensalir:starfire:presented-frame:1 with verdict=FullPose, markers=99, controllers=1, markerWitnesses=2, nearestControllerSkewMs=599.022, splines=3. This proves one-copy field capture/replay and visible proof-surface lowering; it is not the final live cross-process CultMesh ring/page proof, and the controller skew remains evidence to tighten."}
{"ts": "2026-07-12T11:46:00+01:00", "type": "correction", "status": "ok", "note": "Correction to Nightwing Move proof field replay: artifact artifacts/runtime/nightwing-move-proof-20260712-1041-aggregate.mpack reached Mimir FullPose only because dark PS3 Eye frames produced bright blob candidates/noise; the PS Move markers were not lit. Treat that artifact as proof of Muninn->Mimir->Fensalir snapshot replay mechanics and a false-positive detector lesson, not proof of a real optical Move pose. A valid field pose proof still requires lighting the Move orb(s), capturing real optical marker candidates from calibrated camera witnesses, and replaying/publishing the explicit proof spine with the marker-lit condition verified."}
{"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Live Mimir Odin/CultMesh Move evidence path proved on 2026-07-12: Odin provider muninn.telemetry.nightwing advertised stream muninn:nightwing:move-evidence at 10.77.0.3:17888 channel move-evidence connection 1836384261; Mimir lowered RUDP bytes into a local CultMesh ring and decoded named frames carrying three stable Move IDs. CultLib a58daa54 changed default RUDP client bind from loopback to 0.0.0.0 so remote WireGuard routes work."}
{"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Nightwing hotplug migration proof after Odin 5faaf59: platform discovery remained authoritative across USB/Bluetooth/path reassignment and Mimir received move-0006f523e2d1, move-000704a39772, and newly wired move-000704a6be5f in one live frame after the swap. BlueZ handoff itself remains unproven because the unplugged controller only blinked and disconnected; do not claim Bluetooth success."}
{"ts": "2026-07-12T20:50:27+01:00", "type": "decision", "status": "ok", "note": "Use upstream PSMoveAPI as Muninn's optical/LED transport backend. Nightwing now has current libpsmoveapi.so and libpsmoveapi_tracker.so built against packaged OpenCV 4.6 with V4L2 camera control and installed under /usr/local. Official psmove pair created BlueZ info and HID SDP cache records for wired F5, A3, and A6; unplugged A8 still needs one USB pairing pass. The provisional luma extractor is not valid final tracking authority."}
{"ts": "2026-07-12T21:00:36+01:00", "type": "verification", "status": "ok", "note": "Live Nightwing Bluetooth handoff proved 2026-07-12 after official PSMoveAPI pairing: move-000704a6be5f migrated from USB to BlueZ Bluetooth, remained connected at 40% with /dev/input/js3 and HID_UNIQ 00:07:04:a6:be:5f, retained the same stable Move ID in Mimir's Odin-discovered four-controller evidence frame, and the large identity-colored sphere remained glowing. The red status LED blink pattern is not the connection witness; BlueZ HID path, Mimir frame identity, and visible sphere are."}
{"ts": "2026-07-13T00:10:04+01:00", "type": "scar", "status": "open", "note": "Move tracking false-blocker: both Nightwing PS3 Eyes were already physically positioned. The missing authority was software process isolation and persistent dual-provider ingestion. Do not ask the operator to rearrange hardware without first probing all live devices/providers. Repeated one-shot Mimir probes also misuse Muninn's single-client RUDP evidence server; visibility sweeps must keep one persistent subscription per provider."}
{"ts": "2026-07-13T00:16:58+01:00", "type": "verification", "status": "partial", "note": "Persistent dual-provider visibility window separated transport from optics: nightwing-eye0 delivered 2 frames and all 4 controller IDs but 0 labelled markers; primary nightwing delivered 0 frames, consistent with its single-client evidence server remaining occupied after prior one-shot probes. Restart observers before the next window; do not infer camera visibility from discovery or controller presence."}
{"ts": "2026-07-13T00:22:08+01:00", "type": "decision", "status": "ok", "note": "Nightwing Move observation is one Muninn provider with two camera sources. Muninn already accepts repeated --move-marker-camera arguments and aggregates camera-tagged candidates into one evidence frame; the prior nightwing-eye0 provider leaked a process workaround into public authority. If PSMoveAPI needs isolation, use internal workers without separate provider identity."}
{"ts": "2026-07-13T00:29:23+01:00", "type": "scar", "status": "partial", "note": "Unified Nightwing Muninn proves one provider can own both /dev/video2 and /dev/video3 and publish all four controller IDs. Stable 30s visibility window saw 2 controller frames and 0 labelled markers. Hue state is animated/descending/1000ms and light HID paths are dynamically resolved; the next missing evidence is typed per-camera PSMoveAPI readiness/calibrated-controller/update health. Do not infer tracker health from camera fd ownership, payload size, or visible lights."}
{"ts": "2026-07-13T00:47:59+01:00", "type": "verification", "status": "ok", "note": "Typed muninn.move_tracker_health.v1 field evidence isolated the dual-eye failure. Both Nightwing PS3 Eyes identify as V4L2/PS3 Eye 640x480 at exposure 0.1 and can calibrate all 4 Moves, but same-process PSMoveAPI trackers freeze around initialization/update: one process snapshot reached running with update_count=1 for each eye and zero observations; after restart both remained opening with zero updates. Keep one Muninn provider, but run each native PSMoveAPI tracker in an internal subprocess and aggregate camera-tagged observations in the parent."}
{"ts": "2026-07-13T01:15:00+01:00", "type": "verification", "status": "partial", "note": "Nightwing now runs one Muninn provider with private per-eye tracker subprocesses and parent-owned MessagePack hue/observation pipes. Current live proof: both eye workers advance thousands of updates; eye1 calibrated 4 Moves and produced 11,908+ observations, and Mimir received all four stable Move IDs from nightwing-eye-1 in one 15s window. Eye0 calibrated all 4 and advances updates but has zero optical observations, so dual-camera correspondence remains unproven and the remaining fault is eye0 optical scene/backend behavior. Idunn restart now explicitly kills worker children with the parent generation."}
{"ts": "2026-07-13T01:29:15+01:00", "type": "verification", "status": "partial", "note": "Per-camera optical evidence and exposure ownership restored Eye 0. At exposure 0.3 Eye 0 peak RGB rose from [72,25,43] to [178,119,112], calibrated 4 Moves, and produced 3,312 observations/latest 4; Eye 1 remains at exposure 0.1. Mimir's persistent window then saw all 4 IDs on Eye 0, 2 on Eye 1, and 2 same-ID cross-camera correspondences. Do not claim full stereo: Eye 1 overlap is incomplete and only 2 aggregate RUDP frames arrived in 20s and 60s windows."}
{"ts": "2026-07-13T01:41:29+01:00", "type": "verification", "status": "partial", "note": "After per-eye exposure tuning and latest-state RUDP handoff, Mimir's 15s dual-eye receipt saw all four stable Move IDs on both cameras: 16 camera observations and 8 pair records across 2 aggregate frames. Full overlap is proven; calibration is not. Only 2 unique frame IDs arrived, so directly instrument parent publication and RUDP send/receive counters before camera-matrix fitting."}
{"ts": "2026-07-13T02:32:31+01:00", "type": "verification", "status": "ok", "note": "2026-07-13 Nightwing single-provider dual-Eye field proof: one stable Muninn parent received 622 unique evidence frames in 15 s through Mimir, with 2,647 observations, 1,384 same-ID cross-camera correspondences, and all four Move IDs visible on both Eyes. Producer transport health reported 5,302 produced, 5,301 handoffs, 771 sends, and zero local-ring admissions. Artifact: artifacts/move-calibration/muninn-clocked-aggregator.mpack. Odin owner commits: 39c839a, a340c33, 8aefd1c, f82ca7a."}
{"ts": "2026-07-13T02:39:32+01:00", "type": "verification", "status": "rejected", "note": "2026-07-13 typed Move stereo assessment rejected calibration promotion. Corrected same-frame collector captured 798 frames and 130 same-frame/same-Move pairs at zero published-frame skew. Eye 0 covered 4/12 image grid cells; Eye 1 covered 2/12 and saw only two Move IDs in this window. Both calibrated intrinsics and measured orb radius were absent. No camera matrices were emitted. Artifacts: artifacts/move-calibration/muninn-same-frame-correspondence.mpack and muninn-same-frame-stereo-assessment.mpack."}
{"ts": "2026-07-14T01:00:00+01:00", "type": "verification", "status": "rejected", "note": "2026-07-14 live two-wand sphere-stereo fit remained honestly rejected. Synthetic recovery passed at 0.34/0.63 px held-out median/P95 with focal errors below 0.43 px and 0.22 mm baseline error. Live robust consensus reached 0.51/2.08 px but only 53.8 percent held-out support and an out-of-range Eye 1 focal estimate; PSMoveAPI position age is being projected as confidence and blob radius is not calibrated sphere radius. Artifact live-two-wand-20260714-004210.nominal-radius-fit.mpack is diagnostic only."}
{"ts": "2026-07-14T01:13:34+01:00", "type": "verification", "status": "ok", "note": "2026-07-14 Odin d0eded6 deployed to Nightwing Muninn (SHA-256 a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39). Post-deploy 10 s Mimir window: 555 frames, 2246 observations, 443 stereo pairs; radius P01 improved from ~0.707 px to 3.13/3.64 px and candidate confidence is explicitly uncalibrated 0.5 on both Eyes. Eye 1 honestly saw only two plausible stationary IDs. Artifact: artifacts/move-calibration/muninn-post-admission-filter-20260714.mpack."}
{"ts": "2026-07-16T20:03:50+01:00", "type": "decision", "status": "ok", "note": "Moonlight common-c study at commit 82e2514 found Muninn already has media-aware packetization, deadlines, parity, receiver feedback, repair caching, and keyframe pressure. Preserve those foundations; close feedback into sender policy, replace the two-second realtime default with consumer-derived deadlines, and split Sleipnir input into replaceable latest state versus ordered non-replaceable edges. Research: docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md."}
{"ts": "2026-07-16T20:29:03+01:00", "type": "verification", "status": "ok", "note": "Moonlight-derived reliability cut implemented in Odin against experimental CultLib snapshot 8965f3c0: HID epoch/latest-state/ordered-edge/application-ack semantics; wall clock demoted from ordering; Muninn A/V uses CultNet realtime lane with 100ms default LAN deadline; bounded producer/pending queues; expiring and receiver-late-aware repair cache; decode-chain-only keyframe pressure. Verification passed 168 Muninn, 25 Sleipnir, and 35 vendored experimental CultNet tests."}
{"ts": "2026-07-16T20:48:32+01:00", "type": "verification", "status": "ok", "note": "Production Muninn receiver audit fixed two live contract failures: early video repair feedback no longer places live frames in late_frame_ids (which made Odin refuse every repair), with a C++-emitted fixture decoded by Rust; OBS local audio decoder now consumes Muninn's actual f32le stereo 48kHz packets rather than AAC, default reorder is 40ms instead of 2000ms, and short gaps receive bounded silence concealment. Native plugin build and two CTests passed; Odin cross-fixture tests passed."}
{"ts": "2026-07-16T21:03:13+01:00", "type": "verification", "status": "ok", "note": "Muninn production audio recovery now uses typed 4+2 GF(256) PCM parity from Odin and a bounded Mimir receiver cache feeding the existing 40 ms reorder owner; native OBS DLL builds and 3/3 CTests pass. Odin cultnet-impair adds deterministic real-socket loss/burst/reorder/duplicate/jitter/stall profiles. Experimental CultLib snapshot lineage remains pinned at 8965f3c0."}
{"ts": "2026-07-16T21:26:00+01:00", "type": "verification", "status": "ok", "note": "Receiver decode-chain invalidation now actuates a verified next-frame NVENC IDR through Odin's long-lived libav encoder without restarting A/V or CultNet. Live 2560x1440 D3D11 proof emitted a command-driven IDR at frame 22 under a 600-frame GOP; synthetic proof emitted IDRs at frames 0 and 10 and decoded cleanly. Muninn/Sleipnir edge admission is bounded at 256 via epoch rebase; 174/26 tests pass. Raven field deployment remains pending because 10.77.0.4 SSH and TCP probing timed out."}
{"ts": "2026-07-16T21:35:47+01:00", "type": "verification", "status": "ok", "note": "Muninn receiver pressure now drives bounded AIMD bitrate on the long-lived NVENC context. Live 2560x1440 D3D11 proof reconfigured 12 Mbps to 6 Mbps at frame 35, emitted an IDR at frame 35, and decoded cleanly; 176 Muninn tests pass. Cross-host field acceptance remains missing because Raven SSH/TCP 22 timed out again."}
{"ts": "2026-07-16T21:38:23+01:00", "type": "blocker", "status": "pending", "note": "Field Moonlight acceptance cannot run while Raven is offline: Starfire's documented SSH/ICMP/TCP route to 10.77.0.4 times out, and Yggdrasil reports Raven's last WireGuard handshake at 2026-07-15T06:04:12Z versus Starfire's live handshake at 2026-07-16T20:36:32Z. No Raven task or host state was mutated."}
{"ts": "2026-07-17T01:37:22+01:00", "type": "verification", "status": "partial", "note": "Muninn video parity v3 uses 16 Cauchy GF(256) shards; native receiver burst tests 1..16 and Odin 180-test suite pass. Direct Raven LAN at 250 ms recovery deadline ran over one minute with no video assembly expiry or A/V failure; full 10-minute and impairment matrix remain open."}
{"ts": "2026-07-17T01:46:28+01:00", "type": "scar", "status": "fixed", "note": "Per-byte Gaussian elimination for production-sized video FEC can stall the single OBS receive owner and cascade socket loss. Receiver now inverts coefficients once per frame and uses table-driven GF(256); active Release assertions prove every 1..16 consecutive erasure case on 64x1400-byte shards under 50 ms."}
{"ts": "2026-07-17T01:59:33+01:00", "type": "scar", "status": "open", "note": "Direct Raven LAN falsified trailing-parity sufficiency: a 24-datagram sequence gap left a 56-chunk keyframe at 52/56 with parity_present=false because all 16 V3 parity shards followed the data and fell in the same tail burst. Next cut must disperse parity/adapt block layout; deadline increases and repair flooding are not owners."}
{"ts": "2026-07-17T02:34:05+01:00", "type": "verification", "status": "ok", "note": "Receiver completed-frame tombstones eliminated false post-FEC ghost assemblies: a 7-minute LAN run forwarded 54,000 payloads with zero video assembly expiry before an experimental weighted sender scheduler independently caused audio queue eviction; the scheduler was deleted. A later genuine 24-datagram tail burst left a frame 18/33 with only the leading 8 of 16 bookended parity shards available, proving V3 frame-wide bookends still need Moonlight-style multi-block FEC."}
{"ts": "2026-07-17T02:58:43+01:00", "type": "rejected_path", "status": "ok", "note": "Adaptive frame-wide parity was tested and removed: 32 dispersed shards still expired a frame during a 54-datagram gap, and 64-shard production recovery violated the 50 ms CPU bound. V4 must use smaller independently recoverable blocks rather than a frame-wide bandwidth/matrix arms race. Periodic audio skips were traced separately to seven orphaned Raven WASAPI capture workers left by forced parent stops; the hidden scheduled task now reaps only stale Muninn, FFmpeg, encoder, and loopback workers before launching exactly one sender tree."}
{"ts":"2026-07-17T08:15:00+01:00","type":"verification","status":"partial","note":"Raven LAN Moonlight field pass used hidden scheduled tasks, no WireGuard, and experimental CultLib 8965f3c0. Seeded 1% media loss completed 99.93% of 14,400 additional OBS payload groups without expiry (10 expiries), with zero audio failures/skips and no impairment-proxy queue overflow. Sleipnir neutralized the non-neutral virtual Xbox pad after 1.56 s of input loss and automatically recovered. A 64-packet continuous receive drain caused 48-57 ms receive-to-apply delay; a 2 ms drain budget reduced typical live receiver-owned delay to 69-204 us. Cross-host raw clock subtraction was removed from latency telemetry. One 16 ms spike, quick-tap edge impairment, 5%/burst input, glass-to-glass age, A/V skew, and mixed soak remain open."}
{"ts": "2026-07-17T10:37:54+01:00", "type": "verification", "status": "ok", "note": "Ordered Muninn/Sleipnir input field proof closed one-frame quick taps over Raven LAN with hidden tasks, no WireGuard, and experimental CultLib 8965f3c0. Dedicated XInput capture, bounded non-stop-and-wait reliable edges, late/duplicate CultNet ACK repair, 24 ms edge visibility, and asynchronous CultMesh/Eve publication produced exact 100/100 presses/releases under seeded 1% loss and exact 100/100 under burst-8; burst Raven-visible durations were 24-40 ms, final pad neutral, proxy queue overflow zero. Mixed media/input endurance, analog stall, glass-to-glass age, and A/V skew remain open."}
{"ts": "2026-07-17T10:39:57+01:00", "type": "verification", "status": "ok", "note": "Raven execution is headless and LAN-only: Codex-Sleipnir-LAN-Proof and Codex-Raven-XInput-Edge-Observer are scheduled-task launches whose PowerShell actions use -NonInteractive -WindowStyle Hidden; both task metadata records are now Hidden=true. Raven exposes no WireGuard service. Remote administration used raven-local over 192.168.1.84."}
{"ts": "2026-07-17T10:46:17+01:00", "type": "verification", "status": "ok", "note": "Final snapshot-only Muninn/Sleipnir input build delivered 100/100 one-frame A-button presses and releases over Raven LAN under seeded 5% datagram loss (254/5251 dropped): Starfire observer 100/100, Muninn captured 200 semantic edges, Raven observer 100/100, final pad neutral, proxy queue overflow zero/max queue 28. Raven-visible hold duration median 29.166 ms, p95 36.719 ms, p99 62.667 ms, max 118.074 ms; retransmission tail remains the next latency pressure point."}
{"ts": "2026-07-17T11:20:23+01:00", "type": "rejected_path", "status": "ok", "note": "Application semantic replay over the reliable hid.edge channel was built and field-falsified under 5% loss: new reliable duplicates accumulated transport debt and worsened Raven-visible holds to multi-second outliers. The reliable replay cut was deleted; semantic assist must never create retransmission debt."}
{"ts": "2026-07-17T11:20:23+01:00", "type": "verification", "status": "ok", "note": "Native 1 ms XInput witnesses replaced scheduling-contaminated PowerShell polling. Under concurrent sustained Raven media at seeded 1% loss and 100 one-frame input taps at seeded 5% loss, dual-path input (one reliable hid.edge owner plus bounded unreliable unordered hid.edge.assist repeats) delivered exact 100/100 and final neutral. Raven hold p95 improved 78.875->38.000 ms, p99 264.745->53.235 ms, max 272.107->59.028 ms; network-added median 11.617 ms, p95 17.387 ms, max 23.267 ms; media and input proxy queue overflow remained zero."}
{"ts": "2026-10-22T21:30:00Z", "type": "correction", "status": "ok", "note": "Corrects the 2026-07-04 transport-cut entry: Mimir.EveSensorReceiver was not archived; it is in Mimir.slnx and launched by scripts/start-nightwing-move-tracking.ps1, and still listens. Operator ruling thing2:ruling:mimir-sensor-receiver-frozen: frozen, neither extended nor removed until Thing 2. The Eve dashboard projects were removed in 43cebb6 (parked/mimir-eve-surface)."}

View file

@ -5,7 +5,7 @@ workspace:
implementation_name: Mimir public surface / native runtime contracts
premise: >
Mimir is the local realtime field machine: cameras, microphones, speakers,
network feeds, and timing signals become one coherent OBS-facing volumetric
network feeds, and timing signals become one coherent Mimir-owned volumetric
program surface. The live path is a C# Fensalir-hosted app with in-memory
rolling buffers and native capture/runtime boundaries.
@ -13,7 +13,7 @@ objective:
primary: >
Build one synchronized spatial stream machine from six cameras, six
microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP,
and OBS output.
Eve operator surfaces, and Yggdrasil site publication.
current: >
Promote the active bioacoustic packet-song receiver into runtime state:
self-identifying song contours should pin multiple time/frequency anchors
@ -22,9 +22,9 @@ objective:
constraints:
- Sender is Windows.
- Receiver uses OBS.
- OBS is a temporary compatibility sink, not a composition or broadcast authority.
- Video encode must use NVENC when the bridge path is active.
- Audio sources must remain separately controllable in OBS.
- Audio sources must remain separately controllable through Mimir/Eve and publishable as stems.
- V1 bridge is LAN-only.
- Persistent state must clarify rehydrate, not become transcript tax.
- Live ingest belongs to Mimir.Runtime, native drivers, Fensalir, and
@ -33,29 +33,197 @@ constraints:
memory.
- Networked feeds are producers into the live runtime, not independent clock
authorities.
- USB-attached Moves on Starfire and Nightwing are tracking witnesses. Muninn
publishes source-local marker candidates plus controller/IMU/button state.
Odin may discover/project their stream schemas, but it does not own pose
capture or Mimir fusion.
- CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may
consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but
Mimir owns program selection, composition, preview/control, and publication.
- CultLib is moving every runtime's typed CultNet/CultMesh document transport
to `cultnet.transport.rudp.v0`. Health, provider advertisements, command
boundaries, transport profiles, and program/media document lanes should use
RUDP by default. Product/debug render surfaces are lowerings,
compatibility probes, or migration debt unless a specific runtime has not
earned its RUDP path yet.
- Daemon health and freshness are separate in authority from media transport,
not separate because they need TCP. The old Mimir Eve dashboard broker is
archived and no longer publishes Idunn health, deck routes, or WebSocket
commands. Idunn owns continuity once Odin has accepted a daemon; Mimir-owned
surfaces may only return as typed CultMesh/Odin state publishers.
- Mimir explicitly trades latency and memory for throughput and field quality:
the default five-second buffer is compute budget for stream alignment,
GPU/audio analysis, and volumetric sound-field extraction.
invariants:
- OBS composition authority stays in OBS.
- Mimir owns composition, preview/control state, source subscription policy,
and program publication intent.
- FFmpeg owns capture, encoding, and transport only for bridge utilities.
- One stable bridge port equals one OBS source endpoint.
- One rolling live window owns the timing edge and expiry.
- The rolling window is not accidental latency. It must preserve enough time
to line up feeds and crunch data before OBS-facing output is emitted.
to line up feeds and crunch data before Mimir program output is emitted.
- Typed views may index the rolling buffer, but no per-kind ring owns
retention.
- Fensalir owns production UI, GPU fusion, D3D12 interop, and Spout2 output.
- Fensalir owns GPU lowering for the Mimir program scene graph, D3D12 interop,
and local program texture output.
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
stems.
- Eve GUI/TUI surfaces lower Mimir's operator controls, previews, and stats on
any device without owning scene truth.
- A Yggdrasil-facing publisher daemon consumes Mimir program output and
publishes it to the site without owning composition.
- Local secrets remain untracked.
- Mimir is a CultMesh app. Codebooks, schedules, calibration receipts, path
response, and compute-offload work need typed mesh-syncable state surfaces
so phones, microcontrollers, Raven, and Starfire can cooperate without
becoming independent timing authorities.
- Move tracking observations are stream samples, not program controls or
dashboard truth. Starfire-local and Nightwing-remote Move samples must enter
Mimir.Runtime as typed tracking buffers before Fensalir or Eve acts on them.
- Structured PS Move light pulses are Muninn output commands. Mimir requests
`muninn.move_light_command.v1` over CultNet/CultMesh from the Muninn daemon
on the host with the USB-attached Move; direct HID writes in Mimir scripts
are temporary smoke/bootstrap edges only.
accepted_design:
- Muninn video FEC is typed as `muninn.media_video_parity_shard.v4`.
Odin divides each access unit into independent blocks of at most eight data
shards and emits four Cauchy GF(256) parity rows for every block, including
a short tail. Data/parity lanes are round-robin scheduled per block. Mimir's
OBS receiver owns bounded block recovery before the assembly deadline,
forwards only the canonical completed frame, and tombstones completed frame
keys so late shards cannot resurrect ghost assemblies. V2/V3 remain legacy
decode shapes and never own V4 behavior. Rust tests exhaust every burst of
one through four packets at every alignment, plus tail-block burst coverage;
native tests prove the matching receiver. Canonical video and parity now use
CultNet `realtime` without per-packet reliable queues. Repair is selective
and behind fresh media; unrecoverable dependency damage requests a live
NVENC IDR. Normal frames forward immediately; the recovery deadline is not
a playout buffer. Direct Raven LAN validation is active; impaired and soak
gates remain open until recorded in the acceptance ledger.
- Muninn now closes receiver congestion pressure into the live NVENC context.
Every 500 ms it applies multiplicative 15 percent backoff for late frames,
requested recovery, deferred repair, sender queue loss, or combined
jitter/decode pressure above 75 percent of the consumer deadline. Recovery
adds one-fiftieth of a safe cap after each ten stable seconds. The safe cap
is half of the requested encoder ceiling, with a one-quarter/1 Mbps
floor. Startup begins at half ceiling because fixed block parity can roughly
double wire rate. A live 2560x1440 D3D11 proof
changed 12 Mbps to 6 Mbps at frame 35; FFmpeg/NVENC reconfigured the same
session, emitted an IDR at frame 35, and the Annex-B stream decoded cleanly.
Muninn's full suite now passes 184 tests.
- Receiver-driven video recovery now reaches the actual NVENC frame owner.
Odin's `native/muninn-video-encoder` is a long-lived libavdevice desktop
capture and `h264_nvenc` process; a new decode-chain invalidation edge makes
Muninn write `IDR` to its stdin without restarting video, audio, or CultNet.
A live 2560x1440 D3D11 proof with a deliberately 600-frame GOP produced the
requested second IDR at frame 22, and its Annex-B stream decoded cleanly.
The normal quarter-second scheduled IDR remains independent fallback.
The repeatable bundle includes the OBS experimental FFmpeg dependency
snapshot. Raven now runs the bundle over direct LAN from a hidden scheduled
task; WireGuard is not a runtime or administration path. Seeded 1% loss,
burst-4, reorder, and bounded stall recovery now pass; longer matrix and
soak acceptance remain pending.
- Muninn input now bounds disconnected unacknowledged button history at 256
edges. Overflow rotates the controller epoch and rebases Sleipnir from the
next complete latest-state frame, rather than dropping an arbitrary press
or retaining history forever. Sleipnir rejects sparse future edge gaps over
256. Full Muninn (174) and Sleipnir (26) suites pass.
- The 2026-07-16 loss-recovery cut is now live on both sides of the Muninn
audio boundary. Odin emits typed `muninn.media_audio_parity_shard.v1`
records using fixed 4+2 GF(256) parity for padded AAC ADTS packets.
Canonical audio and parity use `realtime`; 4+2 FEC, reorder, and bounded
concealment own continuity without a media ACK/retransmit window.
Mimir's production OBS receiver retains a bounded recent data window,
reconstructs up to two erased packets per block before the 120 ms playout
deadline, and then hands recovered packets to the same reorder queue that
owns normal playout. The FEC cache is not a second timeline. Native tests
exhaust every one- and two-erasure permutation, and the production plugin
DLL builds with all three native suites passing. The OBS input ceiling is
now 500 ms and the default remains 100 ms; old 1.2-2.0 second test/config
assumptions are not realtime authority.
- Odin now carries a deterministic bidirectional UDP impairment proxy with
seeded clean, iid loss, burst loss, delay/jitter, reorder, duplicate, and
stall profiles. This is the acceptance substrate for real CultNet endpoints,
not a simulated transport replacement. The remaining critical sender gap
is an encoder owner that can actuate a receiver-requested next IDR without
restarting the video session; stock FFmpeg CLI capture cannot own that
command dynamically.
- The 2026-07-16 production receiver audit cut two false contracts. Native
Mimir/OBS early video repair feedback no longer marks live frames late, so
Odin can actually return cached missing chunks; only assembly expiry emits
`late_frame_ids`. A C++-generated feedback fixture now decodes in Odin's
Rust tests. Muninn's typed audio body is 10 ms `pcm-f32le-interleaved`, so
the OBS bridge now opens its local decoder as `f32le` stereo 48 kHz instead
of AAC, uses a 40 ms reorder budget instead of 2000 ms, and emits bounded
silence packets for short unrecoverable gaps to preserve cadence. Opus
FEC/PLC remains required before audio reaches Moonlight-level loss quality.
- Odin's 2026-07-16 Moonlight-derived reliability cut now runs Muninn and
Sleipnir against the experimental CultLib runtime snapshot at `8965f3c0`.
HID latest state and ordered button edges have separate epoch/sequence and
application-ack semantics; wall timestamps no longer order input. Muninn
video and audio bodies use CultNet's unreliable `realtime` lane; reliable
delivery remains control/feedback authority. LAN usefulness
defaults to 100 ms,
bounds producer/pending queues, expires repair material, refuses repair for
receiver-late frames, and reserves keyframe requests for decode-chain
invalidation. CultNet fragmented-message IDs wrap from 65535 to 1;
saturation at 65535 previously caused every later fragmented audio
document to collide after roughly eight minutes. Audio and video producer
ingress are separately bounded before the audio-priority scheduler, with
explicit eviction counts. Odin records the live authority and remaining gaps in
`docs/realtime-delivery-authority.md`.
- The Moonlight comparison at
`docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md` establishes
that realtime delivery policy belongs to signal semantics, not generic RUDP
reliability. Muninn media already owns access-unit boundaries, deadlines,
video parity, dependency/keyframe metadata, receiver damage feedback, and a
bounded repair cache; these foundations stay. The next transport cut must
close feedback into sender pacing/bitrate/parity and derive retry/repair
lifetime from video/audio consumption deadlines instead of the current
broad two-second default. Sleipnir/Muninn input must explicitly separate
replaceable latest state from ordered non-replaceable edges, coalesce before
enqueue, acknowledge applied edge/state sequences, and use a proved clock
model before reporting cross-host wall-clock subtraction as one-way latency.
- The 2026-07-17 Raven LAN field pass used hidden scheduled tasks and the
experimental CultLib snapshot at `8965f3c0`; Raven used no WireGuard path.
Seeded 1% media loss completed 99.93% of 14,400 additional OBS payload
groups without video assembly expiry and produced zero audio failures or
skips. Sleipnir neutralized a non-neutral virtual pad 1.56 s after input
loss and recovered automatically. Bounding its continuous receive drain to
2 ms reduced live receive-to-apply from roughly 48-57 ms to typically
69-204 us. Cross-host source latency is now explicitly unavailable until a
clock offset/uncertainty model exists.
- The 2026-07-17 ordered-input field cut removed transport and control-plane
stalls from the Muninn/Sleipnir path. Muninn observes XInput on a dedicated
1 kHz, one-millisecond-timer worker and derives exact button edges from
consecutive snapshots. Captured edges are sent as a bounded reliable
window; application ACKs retire history but never gate the next edge.
CultNet directly ACKs late reliable packets outside its 32-packet
cumulative window and ACKs reliable duplicates, preventing immortal resend
storms. Sleipnir actuates ordered edges for at least 24 ms and publishes
CultMesh/Eve/Idunn state on a separate coalescing worker. Raven field proof
over LAN, hidden tasks, experimental CultLib `8965f3c0`, and no WireGuard
produced exact 100/100 one-frame taps under seeded 1% loss and exact
100/100 under eight-packet bursts. Burst visible durations were 24-40 ms,
final state was neutral, and the impairment proxy had no queue overflow.
A steady-state 5% run was exact at 20/20; repeat the 100-tap 5% first-session
gate after the stale XInput keystroke path was deleted. Glass-to-glass age,
A/V skew, analog motion under stall, and mixed soak remain open.
- Nightwing Muninn `d0eded6` now owns PSMoveAPI observation admission in each
private Eye worker. It rejects stale, sub-2 px, out-of-frame, and
discontinuous positions before the parent aggregates evidence; typed
tracker health exposes rejection counts. PSMoveAPI age is no longer emitted
as optical confidence: admitted candidates carry uncalibrated score `0.5`.
- `MimirMoveSphereCalibrationFitter` owns the bounded two-Eye optical fit
after spatial collection. Eye 0 defines the tracking-space gauge; a robust
training consensus estimates both focal lengths and Eye 1 rigid pose, while
a deterministic untouched 20 percent partition owns promotion evidence.
Promotion requires low held-out median/P95 reprojection error and at least
65 percent held-out consensus. Physical orb radius is explicit and never
guessed. PSMoveAPI blob radius and position age are not calibrated sphere
radius or optical confidence.
- `Mimir.slnx` owns the app surface.
- `src/Mimir.App` launches Fensalir as host/window/D3D12 library.
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
@ -65,6 +233,189 @@ accepted_design:
device timestamp, and optional native/GPU handle.
- `MimirAudioBlockDescriptor` carries sample rate, channels, sample format,
frame count, timestamp, and optional native handle.
- `MimirTrackingObservation` carries typed PS Move evidence samples as
`mimir.move_tracking_observation.v1`; Starfire Moves are local tracking
streams, Nightwing Moves are emitted by Muninn on Nightwing, Odin owns
discovery/schema projection, and Mimir owns buffering and fusion.
- `native/reservoir` now exposes a `move_evidence` typed view and
`LocalcastMoveEvidenceBufferDescriptor` for compute-upload buffers carrying
Muninn optical marker and controller-state witnesses before Mimir fusion.
- `MimirNativeReservoirRuntime` loads the native reservoir DLL and pins Move
evidence sample/descriptor batches so admitted native handles remain valid.
- `MimirMuninnMoveEvidenceAdapter` mirrors Muninn's
`muninn.move_marker_candidate.v1` and `muninn.move_controller_state.v1`
document shapes and maps them into native Move evidence samples without
claiming association or pose authority.
- `MimirMuninnMoveEvidenceAdapter` also consumes
`muninn.move_identity.v1` records into a deduped Move roster projection.
The roster can show a controller as USB-visible on one Muninn host and
Bluetooth-waiting/connected on Nightwing at the same time, so blind USB
rotation remains inspectable without promoting identity-only records into
motion evidence.
- Realtime Muninn Move witnesses enter over CultMesh streaming frames. CultCache
remains persistence/receipt/debug state, not the hot tracking feed.
- `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for
Fensalir-facing interaction. Mimir owns calibration, association,
triangulation, IMU fusion, prediction, confidence, and latency accounting.
- `MimirMoveFusion` is the current Move pose owner. It consumes native Muninn
Move evidence samples, refuses to publish poses without calibrated optical
witnesses, triangulates multi-camera orb position, carries controller
buttons/trigger/gyro into `mimir.move_controller_pose.v1`, and marks
orientation as `orientation:imu-unresolved` until IMU fusion/prediction
earns a full 6DoF controller contract.
- `MimirMovePoseStream` publishes resolved Mimir-owned Move poses as
`mimir.move_controller_pose_stream_frame.v1` CultMesh shared-memory byte
frames for Fensalir and other realtime consumers; it carries poses and does
not own calibration or fusion decisions.
- `MimirMoveProofSurface` publishes the observer-only
`mimir.move_proof_surface.v1` receipt and Fensalir spline probe for the
named proof spine. It combines the Muninn evidence admission receipt, the
Mimir-owned pose frame, and a Fensalir presentation/probe timestamp without
becoming a pose owner. Single-ray fallback is surfaced as non-final; only
triangulated calibrated optical marker evidence can earn a full-pose
verdict.
- `MimirMoveProofPipeline` is the single shared commit primitive for one
Move proof frame. It reads a Muninn CultMesh evidence frame, admits native
`move_evidence`, runs Mimir fusion, emits the Mimir pose frame, creates the
proof surface, and returns the Fensalir spline probe. Direct user actions,
programmatic smokes, and future live subscribers should use this same path
so the proof chain does not split into separately repaired truths.
- `MimirRuntime` now owns the actual Fensalir presentation attachment for
Move proof surfaces. `PublishMoveProofSurface` accepts the latest
Mimir-owned proof document, and `CreateFrame` composes its observer-only
`move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside
the audio spectrum preview. Fensalir remains a renderer, not a proof
authority.
- `MimirMoveProofRuntimeDriver` is the runtime bridge from a Muninn CultMesh
shared-memory evidence ring into `MimirRuntime.PublishMoveProofSurface`.
It derives the Mimir evidence, Mimir pose, and Fensalir frame ids from the
actual Muninn frame suffix, calls `MimirMoveProofPipeline`, and suppresses
duplicate frame admission so the live loop cannot re-admit one frame and
impersonate progress.
- `MimirMoveProofRuntimeConfiguration` is the typed selection and validation
surface for that runtime bridge. An enabled proof source names the Muninn
evidence stream, native reservoir path, Mimir evidence/pose frame prefixes,
Fensalir presented-frame prefix, fusion authority, consumer contract, and
at least two calibrated camera witnesses. It may create a
`MimirMoveProofRuntimeDriver` only when the supplied CultMesh ring stream id
matches the configured evidence stream.
- `MimirRuntime` owns configured Move proof activation. On scene-ready it asks
an `IMimirMoveProofEvidenceRingProvider` for the evidence ring, opens the
configured native reservoir, registers the runtime driver, retains the
driver resources, and exposes activation status. The default provider fails
explicitly because the current C# CultMesh ring is still in-process only;
`Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an
in-process provider to prove the activation path without claiming live
Nightwing/Starfire field evidence.
- `mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine
surface for configured Move proof sources. It reports the configured stream,
provider kind, active/driver state, native reservoir path, calibrated camera
ids, latest same-stream proof ids, latest verdict, and unavailable-provider
diagnostic. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke`
proves both an active injected-ring document and the inactive default
runtime document without claiming live field evidence.
- `mimir.move_proof_evidence_frame_snapshot.v1` is the one-copy fallback for
captured Muninn evidence frames while C# CultMesh has no cross-process ring
opener. A configured `EvidenceSnapshotPath` is loaded by
`MimirConfiguredMoveProofEvidenceRingProvider`, metadata-checked against the
encoded payload, copied into an owned in-process ring, and then consumed by
the same runtime driver/pipeline. `Mimir.BufferSmoke
--move-proof-runtime-snapshot-smoke` proves the named proof spine through
that file boundary. It is field capture/replay support, not the final live
shared-memory proof.
- Odin/Muninn can write the matching producer-side latest snapshot with
`--move-evidence-snapshot <path>`. The snapshot is emitted by the same
Muninn Move evidence publisher after the frame payload is accepted into the
stream ring, and Odin's
`move_evidence_snapshot_writes_mimir_compatible_frame_artifact` unit decodes
both the `mimir.move_proof_evidence_frame_snapshot.v1` tuple and embedded
Muninn payload. This supports Nightwing field capture/replay through Mimir's
snapshot provider while the final live ring/page transport remains pending.
- Nightwing field capture/replay has proven the one-copy snapshot mechanics,
but the first `FullPose` replay was physically false-positive evidence.
Muninn `d4f229a` bundles latest controller state with aggregated marker
candidates from both Nightwing PS3 Eyes into one evidence frame, with the
snapshot payload encoded as MessagePack bytes. The preserved artifact
`artifacts/runtime/nightwing-move-proof-20260712-1041-aggregate.mpack`
replays through `Mimir.BufferSmoke --move-proof-runtime-snapshot-file-smoke`
as `muninn:nightwing:move-evidence:1 -> mimir:starfire:move-evidence:1 ->
mimir:starfire:move-pose:1 -> fensalir:starfire:presented-frame:1`, but
the operator confirmed the Move markers were dark during that capture. Treat
its 99 marker candidates as bright image blobs/noise and use the artifact
only as evidence that Muninn/Mimir/Fensalir replay plumbing works and needs
marker-lit validation. A valid field pose proof still requires lighting the
PS Move orb, capturing real optical marker candidates from calibrated camera
witnesses, and preserving the explicit same-sequence proof spine. The final
live cross-process CultMesh ring/page proof remains pending.
- `MimirMoveProofDevSurface` is a dev-only bootstrap gated by
`MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof
surface through the same MimirRuntime attachment so headless Mimir.App can
capture and pixel-check a Fensalir-presented proof probe without making
Fensalir a proof owner. It is renderer evidence, not field evidence.
- `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move
calibration preflight as typed state: required Muninn Move evidence streams,
optional Quest headset/controller pose witnesses over USB,
stillness/sweep/validation phases, and the calibration outputs Mimir must
derive before orientation can stop being `orientation:imu-unresolved`.
- Muninn owns Quest USB access as `muninn.quest_access.v1`. It proves ADB
authorization/device identity for the optional external VR reference and
advertises Quest input, pose, and warped-video input stream ids. Mimir
consumes those surfaces for calibration; it does not publish Quest access.
- Starfire should have Muninn for source-local Move HID control and local
sensor publication, but some Starfire audio/video paths still enter Mimir
through local Mimir producers or diagnostic process bridges while native
drivers mature. Same-host Move evidence and pose frames are CultMesh
shared-memory byte rings. Current remote media body transport is not
same-machine zero-copy into Fensalir; Raven media uses CultNet/CultMesh
documents over the mesh and lowers to local UDP only for OBS compatibility.
- The old `scripts/start-nightwing-move-tracking.ps1` Nightwing Eye/Move
witness path is archived. Starfire no longer owns a receiver/recorder
socket lane for this evidence; Nightwing/Mimir witness publication must use
typed CultMesh stream frames or Odin-discovered CultMesh documents.
- Odin's Muninn organ owns Move optical candidate extraction in
`E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds
`muninn.move_marker_candidate.v1` records. Mimir consumes those candidate
streams; it does not own raw optical extraction. Mimir explicitly owns
calibration, multi-camera association, triangulation, IMU fusion,
prediction, and final 6DoF wand pose.
- Muninn's Move evidence stream frame now uses canonical
`muninn.move_marker_candidate.v1` records for optical candidates instead of
a hardcoded empty marker slice. A daemon unit proves a bright Y8 frame
extracted by `muninn-move-tracker` serializes as non-empty Mimir-compatible
evidence. The daemon now has a source-local Y8 extraction/publish seam and
a first `serve` camera producer: `--move-marker-camera <camera-id>=<device-path>`
polls a Unix V4L2 YUYV frame, converts it to compact Y8, and feeds that
same seam. Nightwing now runs one public Muninn provider whose two private
per-eye PSMoveAPI subprocesses exchange hue state, observations, and health
with the parent through bounded MessagePack pipes. Children own no store,
provider advertisement, command route, or light output. Typed
`muninn.move_tracker_health.v1` records expose camera backend, calibration,
update/observation counts, RGB range, and expected-hue pixel evidence.
Live field proof on 2026-07-13 established Eye 0 at exposure 0.3 and Eye 1
at 0.1; both calibrated four Moves and produced observations. A Mimir
visibility window now sees all four IDs on both eyes. Muninn's dedicated
evidence aggregator owns frame sequencing and transport independently of
the 15-second CultCache telemetry loop; controller state is bundled but
cannot create an optical frame. Its deadline clock is capped by the
configured camera cadence. Field proof on 2026-07-13 kept one Muninn parent
PID alive, received 622 unique frames in 15 seconds, carried 2,647 camera
observations and 1,384 same-ID cross-camera correspondences, and saw every
stable Move ID from both eyes. Producer health reported 5,302 frames, 5,301
remote handoffs, 771 sends during the subscription, and zero local-ring
admissions. Cadence and full overlap are earned; calibrated camera matrices
still require intrinsics and spatially diverse correspondence capture.
- `MimirMoveStereoCalibrationAssessment` owns the pre-fit observability gate
as `mimir.move_stereo_calibration_assessment.v1`. It consumes a typed
visibility receipt, counts only same-frame/same-Move stereo pairs, measures
per-Eye image-grid and radius coverage, and refuses promotion when
intrinsics, measured orb radius, synchronized sample count, or spatial
diversity are missing. The corrected 2026-07-13 live receipt contained 130
honest same-frame pairs at zero frame skew, but Eye 0 occupied only 4/12
grid cells and Eye 1 only 2/12; no camera matrices were promoted.
- `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a
typed admission receipt with decoded frame id, producer, publish/read
timestamps, sample counts by kind, source/arrival ranges, native reservoir
handle, and reservoir edge/window so the named proof spine can be traced.
- `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for
JSON-line video frame and audio block metadata from native probes into
runtime rolling buffers; it does not carry pixels/audio payloads or own the
@ -271,7 +622,7 @@ accepted_design:
library distilled from the research notes: node profiles, decoder/language
profiles, path learning, benchmark panels, actuator strategies, capture and
ingest strategies, reservoir strategies, field profiles, compute offload,
OBS publication, CultMesh contracts, Fensalir lowering, and exportable
program publication, CultMesh contracts, Fensalir lowering, and exportable
manifest generation.
- `docs/perfect-machine-module-library.md` indexes those reusable runtime
modules and their smoke commands.
@ -279,6 +630,28 @@ accepted_design:
for the current repo code surface.
- `docs/perfect-machine-domain-index.md` indexes Mimir's full problem domain,
current evidence taxonomy, hot paths, and implementation cuts.
- `docs/mimir-program-composition.md` owns the current program-composition
authority map: Muninn observes, Mimir composes/controls, Eve lowers operator
surfaces, Yggdrasil publishes, and OBS is compatibility-only.
- `src/Mimir.Runtime/Synchronization` now includes typed program contracts:
`mimir.program_scene.v1`, `mimir.program_output.v1`, and
`mimir.eve_operator_surface.v1`.
- `state/mimir-program-composition.cc` is the current imported Mimir program
state, generated from OBS scene JSON by `Mimir.BufferSmoke
--import-obs-program-scene`. It preserves the current Starfire monitor,
Discord crop/chroma key, Raven monitor, camera, browser, and Mimir texture
layers as Mimir-owned typed state.
- `src/Mimir.CultMeshMedia` owns the first explicit CultMesh media/body
bridge. Raven publishes muxed H.264/AAC MPEG-TS chunks as rolling
`mimir.cultmesh_media_frame` documents over CultNet reliable UDP,
Yggdrasil relay routing comes from
`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av` through
Odin/CultMesh discovery, and Starfire lowers the subscribed stream to local
UDP for the OBS source `Raven Monitor + Realtek`. The bridge now uses
explicit CultLib RUDP socket/session helpers for relay, sender, and
receiver. The old `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient`
node/client path and raw sender/receiver host-port config are no longer
transport authorities for this media lane.
- `research/perfect-machine-study-2026-05-23/` preserves the architecture
rumination, optimization ledger, research references, source/hotspot audits,
Fensalir/native boundary maps, benchmark plan, calibration-session spec,
@ -305,6 +678,15 @@ active_subgoals:
settings, and output management.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
replaces them.
- Launch Raven's CultMesh sender directly on Raven because SSH to Raven over
WireGuard timed out during the initial relay deployment pass.
- `Mimir.VerseRecorder` and the Nightwing typed witness publisher are
archived so the old witness subscription lane cannot retake transport
authority. `Mimir.EveSensorReceiver` (the `/eve/camera` ingest that
`start-nightwing-move-tracking.ps1` launches) is still built and live, and
is frozen: neither extended nor removed until Thing 2
(`thing2:ruling:mimir-sensor-receiver-frozen`, 2026-10-22). `src/Mimir.CultMeshMedia` has completed this cut for the
Raven/Yggdrasil/Starfire media-frame lane.
current_status:
summary: >
@ -383,13 +765,13 @@ current_status:
Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in
the Leap USB neighborhood does not reproduce the Focusrite-induced collapse.
next_action: >
Use the new bioacoustic timeline watermark as the active acoustic sync
direction. Cut the next implementation in this order: streaming indexed
log-mel/MFCC motif decoder state with a global clock/path hypothesis fed by
`--bioacoustic-train` receipts, calibration-session command/replay proof that learns
per-output/mic motif/formant response, fractional-delay/SRO actuator, then
native camera payload handles and Fensalir lowering. Keep chirp-bin
calibration captures as reference evidence, not the runtime target.
Use the stable single-provider Nightwing Move stream to collect a spatially
diverse stereo calibration session. Calibrate each Eye's intrinsics, fit
their relative extrinsics from timestamped same-ID correspondences, use
apparent orb radius only as a weighted distance cue after radius calibration,
and publish typed camera calibration plus residual evidence. Do not claim
precise camera matrices from overlap or radius alone. Then feed calibrated
witnesses through MimirMoveFusion and the named Fensalir proof spine.
Current app testing proves Fensalir WASAPI output wakes Scarlett speaker
loopback, live mic buffers stay populated, runtime sync analysis now runs as
a bounded rotating service without UI/telemetry re-entry, and the bioacoustic

Some files were not shown because too many files have changed in this diff Show more