From d7e39fe495f07d25725a578e4144cef7dc7a2d97 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 25 May 2026 18:01:29 +0100 Subject: [PATCH 001/135] Wire EVE native program output contract --- docs/eve-program-output.md | 59 +++++++++++++++++ docs/implementation-plan.md | 5 ++ notes/current-system-map.md | 6 +- scripts/start-eve-program-output.ps1 | 47 ++++++++++++++ src/Mimir.BufferSmoke/Program.cs | 31 +++++++++ .../MimirEveProgramOutputConfiguration.cs | 64 +++++++++++++++++++ .../MimirObsPublicationConfiguration.cs | 11 ++++ state/evidence.jsonl | 1 + state/map.yaml | 10 ++- 9 files changed, 232 insertions(+), 2 deletions(-) create mode 100644 docs/eve-program-output.md create mode 100644 scripts/start-eve-program-output.ps1 create mode 100644 src/Mimir.Runtime/Synchronization/MimirEveProgramOutputConfiguration.cs diff --git a/docs/eve-program-output.md b/docs/eve-program-output.md new file mode 100644 index 0000000..2e8db5b --- /dev/null +++ b/docs/eve-program-output.md @@ -0,0 +1,59 @@ +# EVE Program Output + +## Authority Map + +- Owner: Fensalir owns the visible D3D12 program pixels. +- Inputs: Fensalir reads the current `AquariumFrame`, render plan, GPU resources, + and overlay state. +- Outputs: a named shared D3D12 `B8G8R8A8_UNorm` texture containing the completed + backbuffer for the current frame. +- Derived state: Mimir publication profiles describe the stream contract; they + do not render pixels. EVE receives decoded pixels and composites them. +- Forbidden writers: WebKit, DOM layout, browser dashboards, OBS capture, and + desktop/window capture do not own this path. +- Shared paths: local display and EVE streaming both use the same completed + Fensalir backbuffer copy. +- Deletion line: diagnostic Sunshine/Moonlight or desktop-duplication probes may + prove network/decode budget, but they are not production authority. + +## Current Mechanism + +Set these before launching the Mimir app: + +```powershell +$env:FENSALIR_PROGRAM_OUTPUT_D3D12 = "1" +$env:FENSALIR_PROGRAM_OUTPUT_NAME = "Global\MimirFensalirProgramTexture" +dotnet run --project .\src\Mimir.App\Mimir.App.csproj +``` + +Fensalir creates a shareable D3D12 texture with that name and copies the finished +backbuffer into it during the present pass. A hardware encoder process should +open that named texture, encode H.264/HEVC with GPU hardware, and send frames to +EVE's native receiver. + +`scripts/start-eve-program-output.ps1` starts the Mimir/Fensalir app with the +shared-texture output enabled and verifies the `eve` SSH target before launch. + +## Receiver Contract + +EVE owns: + +- network receive; +- hardware decode; +- texture upload/import; +- final fullscreen composite; +- touch/Pencil/sensor uplink later. + +EVE does not own: + +- dashboard DOM; +- Chromium/WebKit compatibility; +- layout parity; +- Mimir timing authority; +- OBS composition authority. + +## Smoke + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --eve-program-output-contract-smoke +``` diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 608b6c4..ffb19ec 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -28,6 +28,11 @@ a named invariant that the native runtime cannot protect yet. - Mimir public identity, branding, and Face memory. - `Mimir.slnx` with `src/Mimir.App` and `src/Mimir.Runtime`. - Fensalir host bootstrapping from `Mimir.App`. +- [[eve-program-output|EVE Program Output]] now owns the first native EVE + streaming contract: Fensalir can publish the completed D3D12 backbuffer into a + named shared texture when `FENSALIR_PROGRAM_OUTPUT_D3D12=1`, and Mimir records + the EVE-facing publication profile as a native D3D12 stream rather than a + WebKit/dashboard path. - `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and `IMimirStreamSource`. - Configurable five-second default rolling buffers for local and network audio diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 90077fb..d23f05d 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -26,6 +26,7 @@ flowchart TD N --> E["Fensalir GPU fusion + UI"] N --> F["Faust/native DSP"] E --> G["Spout2/program video"] + E --> V["EVE native shared-texture stream"] F --> H["program stems + spatial bed"] G --> I["OBS"] H --> I @@ -42,10 +43,13 @@ Ownership: - `Mimir.Runtime` owns app-level stream buffers and synchronization. - Native capture workers own device reads. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, - D3D12 interop, runtime UI, and Spout2 publication. + D3D12 interop, runtime UI, Spout2 publication, and the completed backbuffer + texture copied into the EVE shared D3D12 program-output surface. - Faust/native DSP owns hot audio alignment, suppression, separation, spatialization, and stem generation. - OBS owns broadcast controls. +- EVE owns native decode/composite for its fullscreen display path; it does not + own layout, WebKit compatibility, or timing authority. Invariant: the live window is bounded, in memory, and has one timing authority. No private history outlives the rolling buffer. diff --git a/scripts/start-eve-program-output.ps1 b/scripts/start-eve-program-output.ps1 new file mode 100644 index 0000000..60e1737 --- /dev/null +++ b/scripts/start-eve-program-output.ps1 @@ -0,0 +1,47 @@ +param( + [string] $Target = "eve", + [string] $SharedTextureName = "Global\MimirFensalirProgramTexture", + [string] $Configuration = "Debug", + [string] $LogPath = "logs\eve-program-output-fensalir.log", + [string[]] $AppArguments = @() +) + +$ErrorActionPreference = "Stop" + +$repoRoot = Split-Path -Parent $PSScriptRoot +$absoluteLogPath = Join-Path $repoRoot $LogPath +$absoluteErrorLogPath = [System.IO.Path]::ChangeExtension($absoluteLogPath, ".err.log") +$logDirectory = Split-Path -Parent $absoluteLogPath +if (-not (Test-Path $logDirectory)) { + New-Item -ItemType Directory -Path $logDirectory | Out-Null +} + +Write-Host "Checking EVE SSH target: $Target" +ssh $Target "uname -a >/tmp/mimir-eve-program-output-check.txt && cat /tmp/mimir-eve-program-output-check.txt" + +$env:FENSALIR_PROGRAM_OUTPUT_D3D12 = "1" +$env:FENSALIR_PROGRAM_OUTPUT_NAME = $SharedTextureName + +$projectPath = Join-Path $repoRoot "src\Mimir.App\Mimir.App.csproj" +$dotnetArgs = @( + "run", + "--project", + $projectPath, + "-c", + $Configuration, + "--" +) + $AppArguments + +Write-Host "Starting Mimir/Fensalir with shared D3D12 output: $SharedTextureName" +Write-Host "Log: $absoluteLogPath" +$process = Start-Process -FilePath "dotnet" ` + -ArgumentList $dotnetArgs ` + -WorkingDirectory $repoRoot ` + -RedirectStandardOutput $absoluteLogPath ` + -RedirectStandardError $absoluteErrorLogPath ` + -WindowStyle Hidden ` + -PassThru + +Write-Host "Started PID $($process.Id). Poll with:" +Write-Host " Get-Content -Tail 40 '$absoluteLogPath'" +Write-Host " Get-Content -Tail 40 '$absoluteErrorLogPath'" diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index fe3617b..b018972 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -84,6 +84,11 @@ if (args.Any(arg => string.Equals(arg, "--perfect-machine-profile-smoke", String return RunPerfectMachineProfileSmoke(); } +if (args.Any(arg => string.Equals(arg, "--eve-program-output-contract-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunEveProgramOutputContractSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--perfect-machine-contract-smoke", StringComparison.OrdinalIgnoreCase))) { return await RunPerfectMachineContractSmokeAsync( @@ -423,6 +428,32 @@ static int RunChirpletSelfTest() return 0; } +static int RunEveProgramOutputContractSmoke() +{ + var output = MimirEveProgramOutputConfigurations.NativeSharedTexture; + var publication = MimirObsPublicationConfigurations.EveNativeProgram; + Console.WriteLine( + $"eve-output id={output.Id} host={output.TargetHost} port={output.TargetPort} texture=\"{output.SharedTextureName}\" {output.TargetWidth}x{output.TargetHeight}@{output.TargetFramesPerSecond} transport={output.Transport} webkit={output.UsesWebKit} diagnostic={output.DiagnosticOnly}"); + Console.WriteLine( + $"eve-publication id={publication.Id} videoKind={publication.VideoKind} source=\"{publication.VideoSourceName}\" diagnostic={publication.DiagnosticOnly}"); + foreach (var requirement in output.RequiredEnvironment) + { + Console.WriteLine($"eve-env {requirement}"); + } + + if (output.UsesWebKit || + output.DiagnosticOnly || + output.Transport != MimirEveProgramOutputTransport.D3D12SharedTextureHardwareEncode || + publication.VideoKind != MimirObsVideoPublicationKind.EveNativeD3D12Stream || + !string.Equals(output.SharedTextureName, publication.VideoSourceName, StringComparison.Ordinal)) + { + Console.Error.WriteLine("EVE program output contract is split-brained."); + return 1; + } + + return 0; +} + static int RunChirpBinSelfTest() { var timeline = MimirChirpBinTimeline.Default; diff --git a/src/Mimir.Runtime/Synchronization/MimirEveProgramOutputConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirEveProgramOutputConfiguration.cs new file mode 100644 index 0000000..d85e43f --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirEveProgramOutputConfiguration.cs @@ -0,0 +1,64 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirEveProgramOutputTransport +{ + D3D12SharedTextureHardwareEncode, + SunshineMoonlightProbe, + DesktopDuplicationDiagnostic +} + +public sealed record MimirEveProgramOutputConfiguration( + string Id, + string Description, + string TargetHost, + int TargetPort, + string SharedTextureName, + int TargetWidth, + int TargetHeight, + int TargetFramesPerSecond, + MimirEveProgramOutputTransport Transport, + bool UsesWebKit, + bool DiagnosticOnly, + string[] RequiredEnvironment, + string Notes); + +public static class MimirEveProgramOutputConfigurations +{ + public const string DefaultSharedTextureName = "Global\\MimirFensalirProgramTexture"; + + public static MimirEveProgramOutputConfiguration NativeSharedTexture { get; } = new( + "eve-native-d3d12-hardware-stream", + "Production direction: Fensalir copies the completed D3D12 backbuffer into a named shared texture; a hardware encoder consumes that texture and EVE displays decoded frames natively.", + "eve", + 44044, + DefaultSharedTextureName, + 1920, + 1440, + 60, + MimirEveProgramOutputTransport.D3D12SharedTextureHardwareEncode, + UsesWebKit: false, + DiagnosticOnly: false, + RequiredEnvironment: + [ + "FENSALIR_PROGRAM_OUTPUT_D3D12=1", + $"FENSALIR_PROGRAM_OUTPUT_NAME={DefaultSharedTextureName}" + ], + "EVE owns decode/composite only. It does not own layout, DOM, timing authority, or Chromium compatibility."); + + public static MimirEveProgramOutputConfiguration MoonlightProbe { get; } = NativeSharedTexture with + { + Id = "eve-moonlight-probe", + Description = "Latency probe only: Sunshine/Moonlight can validate network and decode budget before the custom encoder path is finished.", + Transport = MimirEveProgramOutputTransport.SunshineMoonlightProbe, + SharedTextureName = "", + DiagnosticOnly = true, + RequiredEnvironment = [], + Notes = "This proves the EVE display/network budget, not Mimir's shared-texture authority." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + NativeSharedTexture, + MoonlightProbe + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs index 35a8edc..f18ad49 100644 --- a/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs @@ -4,6 +4,7 @@ public enum MimirObsVideoPublicationKind { Spout2, SharedD3D12Texture, + EveNativeD3D12Stream, SrtDiagnosticBridge, WindowCaptureDiagnostic } @@ -74,10 +75,20 @@ public static class MimirObsPublicationConfigurations TargetPresentationDelaySeconds: 1.5, DiagnosticOnly: true); + public static MimirObsPublicationConfiguration EveNativeProgram { get; } = NativeProgram with + { + Id = "eve-native-d3d12-stream", + Description = "EVE-facing native program output: Fensalir publishes a shared D3D12 texture for hardware encode; EVE receives decoded pixels without WebKit layout authority.", + VideoKind = MimirObsVideoPublicationKind.EveNativeD3D12Stream, + VideoSourceName = MimirEveProgramOutputConfigurations.DefaultSharedTextureName, + DiagnosticOnly = false + }; + public static IReadOnlyList BuiltIn { get; } = [ NativeProgram, TextureInteropProgram, + EveNativeProgram, DiagnosticSrtBridge ]; } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 0150da4..ed4fffa 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -119,3 +119,4 @@ {"ts": "2026-05-25T13:56:03+01:00", "type": "decision", "status": "ok", "note": "Fensalir spectrum visualization authority split: Mimir now emits AquariumBufferFieldFrame spline tube fields as the real buffer-domain contract, with tangent/curvature/normal/derivative appearance and probe policy for future compute SDF splat reservoir lowering. AquariumSplineFrame remains only the immediate Catmull-Rom preview witness until compute lowering owns rendering."} {"ts": "2026-05-25T15:55:00+01:00", "type": "decision", "status": "ok", "note": "Fensalir/Mimir spectrum visualization moved from C# spline authority to DSL-authored texture field authority. config/mimir-spectrum-field.aquafield binds a rolling spectrum texture, declares surface graph nodes, and asks for texture-dimension splinefield lowering. Fensalir now compiles Aquarium field DSL text, uploads texture samples/program packets, and ProgramMode=3 in D3D12FractalReservoirCompute emits stochastic reservoir splats from the texture field. Current canary is mechanically through the reservoir at 131072 splats with low CPU record time, but visual framing/material is still not the final beautiful spectrum surface."} {"ts": "2026-05-25T17:43:25+01:00", "type": "decision", "status": "ok", "note": "Fensalir direct splinefield correction: texture-derived tube trails are direct SDF envelope surfaces, not point-splat fractal dust. AquariumFieldScriptCompiler supports direct; D3D12Renderer now derives AquariumSplineFrame tube-envelope trails from BufferFieldFrame texture spline programs, the spline pipeline declares triangle topology, and D3D12SplinePS writes color, metadata/control, and reservoirGuide outputs. Mimir capture artifacts/runtime/mimir-direct-sdf-tube-spectrum.png proves the DSL-authored ASIO spectrum appears through the direct SDF tube path with ~0.64 ms record CPU in the canary."} +{"ts": "2026-05-25T18:01:04+01:00", "type": "decision", "status": "ok", "note": "EVE native program output is now a D3D12 shared-texture contract: Fensalir owns completed backbuffer pixels and publishes Global\\MimirFensalirProgramTexture when FENSALIR_PROGRAM_OUTPUT_D3D12=1; Mimir records the EVE publication profile and WebKit remains forbidden as pixel/layout authority."} diff --git a/state/map.yaml b/state/map.yaml index 0ae93ef..af9713e 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -46,7 +46,10 @@ invariants: to line up feeds and crunch data before OBS-facing output is emitted. - Typed views may index the rolling buffer, but no per-kind ring owns retention. - - Fensalir owns production UI, GPU fusion, D3D12 interop, and Spout2 output. + - Fensalir owns production UI, GPU fusion, D3D12 interop, Spout2 output, and + the EVE shared D3D12 program-output texture. + - EVE fullscreen output must use native decode/composite over Fensalir-owned + program pixels; WebKit/DOM layout is forbidden as pixel authority. - Faust/native DSP owns hot audio alignment, separation, spatialization, and stems. - Local secrets remain untracked. @@ -63,6 +66,11 @@ accepted_design: direct native ingest, process bridge ingest, and video capture driver seams. - `MimirVideoFrameDescriptor` carries width, height, pixel format, stride, device timestamp, and optional native/GPU handle. + - `MimirEveProgramOutputConfigurations.NativeSharedTexture` records the + EVE-facing program-output contract: Fensalir publishes + `Global\MimirFensalirProgramTexture` when + `FENSALIR_PROGRAM_OUTPUT_D3D12=1`; EVE receives decoded native pixels and + owns only final composite/input uplink. - `MimirAudioBlockDescriptor` carries sample rate, channels, sample format, frame count, timestamp, and optional native handle. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for From 7a811ee3294f5c8b1ae984c72edfd91c96d2f2ba Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 9 Jun 2026 02:19:17 +0100 Subject: [PATCH 002/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 58446 -> 61724 bytes .voidbot/voice/identity.json | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 254e2efdc1b4e92d4c8ccd889644b5059c6a831f..c149835bfa5b53e2d38be4a73a2283db35bb11f5 100644 GIT binary patch delta 3790 zcmb7HQK(!;8SYI`w8Ri=W2=dEQc_Lr?sE3*-o3kPA>Oz4p^2t3#ZZW|=giri-g9O) zGjr}{1z9jHDuNmgO2JZ}YLilE^nzAIgn**fl)NZh9}4l&`XVBI)bF2jcJJo4Nfj>a zJ$q*U`RDt;@B9CK^Y?pS_}kvA|NO{3oqEtc9k%-Iuph29+F|SLrImxl{rfK-*!8AW ztn9yU>)QTf!R|f1koIhS^!Sb1u2-v_jt}>t^vDV?K9?S9U%Vpi2|gZ6HN5tk5*HjE zo^;$eyY5~OgAj+}uXj2K`t4Rf>@_-T?czpy;lX#_-2V7CANg6NRU)jFT7A*rGdJRm z$DN^@*zvt?sdy+A5jnN`P2)=y87hG54WCH zXAT^D>#hT@jH{C#kAKOBLPgVhCNpX3oo+jcdQrT(+6#;GUwgE8^*4u$AN}~lv+q7Z z?>}pq9SaeSm>VI&NT)HAmMJ}9P7X&-m_8m@hP>D~AEYv3MvR4%4!2{uN^w7I6?lzp zS_zg=7$(&wHxhF;%H>gbDU);j1@d%6P*L2XL4*xrfh(f8u3IiI{nG_X36Dz^@)T8uND0TWbdOs3+Fy?I3B@yY2Q`J?bQ#s56MxxL7T2{^?NB zxb$G94=BFd4}(Up8=Sq8k7LSqmq3=56zkU?>XrO|1w7S#f0H}uBMjq?5JAvt1&P3h9-+2&cs&2;;(C2kYYzAU9D1CUbKpX4cIwCM>j z03X;w5o4qA1uE{V4@S&9^@!I~>qLg{K}-cDKw%~NSU`lZIyaH1q3$3$m4c1A8`UUu zB4f|TfzDNo6_o<~BXR}1+(Zb)oB_S-0|B*CKnHx;j8>!I5)r|64;zSzkYKfU4}TDWK*9YtxfS$`+bNdh2Kx*f>r~p=2+XN$ zi6djjcoY%i%}Br@WwfMP`P4|ly~rSB;@zoIu1^PsMrsAT9^@E_84G=C=zOd#$}ksq zjYNxl398k28Nm?*W?edMEChY=^m_<;qga3I-Y?G`$B6T}gwa9aF+O2lfWp^NKt97M zJPk=KiHJ-NC0ke*3JsGn(#A3>{`2emAN|1z?<~1dHC#p{0E<@ys1FTQ8KA`y1cL%` zTU-xbQARDAAGr4`sBHve*~CTVvhoTaVhj(_RhGequ3~_M&Tty?$%8 z(QX#&zx&v`zr1;w6<@vn!T(Rs-Jlb9d&wZ`1ZxXD|GebSIp_j~$dUXxqDLv|+q_(; z`bDZX!E4efRdL>F8TY6YRjQyhhW-QXLx8C!cPw=`n58<}0F)$7^jjxWSOIrvu2@Ru zPso!)Pk=QYG2)gD$=6XcC`%}v57D7@*%Qh>hSEkT#6p={pB!zO^p(RFAT6ryvbVO<3ZZ3yE4-HH1m?05|sYbX~MrUn=;mR4~=V;1F*X z)ncgFp|MksFk(|~Qlb@S4zom?EKm6@%KXQ%e_#9F(+_)PicXB?8ATWcJI#7KY_8V3 z10L4JTD#c{dttK^MssB@hSFBE-)c5mo!0G|ZVT_LCDe4!ysPQnrdcoTyW_muZ3IEn z&*l4O(_1EXylIw`@T;>!d%pdxtv}5E{F&lkSNZH&F9FO!FkTuM2^#iz*yc-(#TMf!Hh5W*QI2wqpK}#R*n5hQ$M;n?W;QA?ij7xMtK!ouhZxS%I9Nx>Usq4(f zk0&KLy*7N@1)ww^jzD5~m>(WA0Ws%!%}?1h7c9$|#v+s=*7#B3*o1@D2!T!pGlW>Q zx`oAmzgc|Z7l)7jj|8smCU4k$U-E`%J=Z^7{PF6;TQB}({qC)Iw(beeycX*S#-qfO W{%3{#{`*lG2i@tX8weZr&i)tBuV9Y= delta 1033 zcmaiy%WD)t5Qkaypn-(=N)%Db;0txv-O0>uHXt#mh=>v~=CB8)ccwPeWTt!Q?lJC3 zLBxYzq`ioO2k~`K#6|I@s0Tq11w{qDM9`Cm`~iYBYl5%}9)_N-uKHEgS9Nt};p>+R z@6UEG^YV^2Ui5|s-C@TolwGfOF}KpL>N)e*OC@`0J?q(0?Cl8hY_6xjwRGgo_Rb)$ z9b^|r>vK=-)@Y4?_72yFSdwrt`Ml+)N}(`xKVq0N7>t_7PSGjni}MMb(y-$eN{&;z z|HpZ+2y};MWv+gwT&>x|QUB7mK&kl0Y@5-^fe&3flj|QI_);`bYc9o@qBB+aYUCrl zxN_s`xLqD^T=Ht<$Af&&#G=)<8t<}CrWS9Vr{mUcYa`V^DxLEvXDIBmc-`9GOK0wC zKg5_#pvrRP?j&>Bv~B&2Cwgsecb|RNy|xn?^^ncr;rV7K=t^7t=!G>4f^#i(;r*tT zel*%;k3U*r_qEumV+YYNUokUt`m%;3)nT3aQ@}M);skvn3`_$9KSCzJmxSLpR2rC& z8PK6jqX4E56Z&V_j3AWLupcOxO%-nV5##ZvBV%+35Hb~{B?2&fGNfoc#bPojBsB!p zHfDLH&e- z4_IBM2I`oV$AXyCG)SIDxCc@Zph`6c+o4WAoT@Ucsew$5Gy*B2Ch$gPucI$x)WF3F zrZ^+nx#8H+&nr1UkTHmxa6)n)A!#=}%}od)s&>zTBmf2?G{3u*GKN6$1}V3Z3nM`) z!3>fg0mvAzhRLGmxo0Ts+b7#LRA|4Qp-M5|D3youuIqaFx?3sbt7J!3cfD#ku(RKG sbPtXftHaq+4;Ea{&fe_nu#A0&*0rabi?)7mwLSi&Yv4c2S{bVS0@e{=JOBUy diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index ca0254c..9adfd65 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-05-25T05:17:24.094Z", + "createdOrRefreshedAt": "2026-06-08T23:28:08.132Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 3dcc39208051517a5fdbf1f2935ff19c70a959ef Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 9 Jun 2026 02:21:32 +0100 Subject: [PATCH 003/135] Move program composition authority into Mimir --- config/perfect-machine.example.json | 42 ++++- docs/architecture.md | 15 +- docs/code-algorithm-map.md | 24 ++- docs/implementation-plan.md | 54 ++++-- docs/mimir-face.md | 17 +- docs/mimir-program-composition.md | 111 ++++++++++++ docs/native-rebuild-plan.md | 25 ++- docs/perfect-machine-domain-index.md | 20 ++- docs/perfect-machine-module-library.md | 5 +- docs/viable-stream-app.md | 50 ++++-- notes/current-system-map.md | 77 ++++++++- notes/fresh-workspace-handoff.md | 29 +++- src/Mimir.BufferSmoke/Program.cs | 40 ++++- .../MimirComputeOffloadConfiguration.cs | 6 +- .../Synchronization/MimirCultMeshContracts.cs | 153 +++++++++++++++++ .../Synchronization/MimirModuleLibrary.cs | 8 +- .../MimirObsPublicationConfiguration.cs | 83 --------- .../MimirPerfectMachineManifest.cs | 4 +- .../MimirPerfectMachineProfiles.cs | 54 ++++-- .../MimirProgramPublicationConfiguration.cs | 159 ++++++++++++++++++ .../MimirReservoirConfiguration.cs | 2 +- state/evidence.jsonl | 3 + state/map.yaml | 39 ++++- 23 files changed, 827 insertions(+), 193 deletions(-) create mode 100644 docs/mimir-program-composition.md delete mode 100644 src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs create mode 100644 src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs diff --git a/config/perfect-machine.example.json b/config/perfect-machine.example.json index e3d8dab..36868d7 100644 --- a/config/perfect-machine.example.json +++ b/config/perfect-machine.example.json @@ -31,15 +31,25 @@ "outputs": { "video": "synchronized-program-video", "audio": "separately-controllable-synchronized-stems" + }, + "programComposition": { + "owner": "Mimir", + "sceneDocument": "mimir.program_scene.v1", + "operatorSurface": "mimir.eve_operator_surface.v1", + "outputDocument": "mimir.program_output.v1", + "sitePublisher": "Yggdrasil", + "obsRole": "compatibility-sink-only" } }, "authorities": { "configurationCalibrationLaunchStatus": "Mimir", + "programComposition": "Mimir", + "operatorSurfaces": "Eve", + "sitePublication": "Yggdrasil publisher daemon", "denseVisualFusion": "Fensalir", "gpuMaterialAndBrushReconciliation": "Fensalir", - "spoutPublication": "Fensalir", - "hotAudioDsp": "Faust", - "obsBroadcastControls": "OBS" + "programTextureOutput": "Fensalir", + "hotAudioDsp": "Faust" }, "visualSensors": [ { "id": "kiyo-primary", "role": "rgb-main", "required": true }, @@ -70,9 +80,9 @@ ], "outputs": { "video": { - "target": "Spout2", - "senderName": "Mimir Point Cloud", - "owner": "Fensalir" + "target": "MimirProgramTexture", + "senderName": "Mimir Program Texture", + "owner": "Mimir/Fensalir" }, "audio": { "owner": "Faust", @@ -85,6 +95,22 @@ "co_streamer_loopback", "spatial_bed" ] + }, + "operator": { + "owner": "Mimir/Eve", + "surfaces": [ + "eve-gui-compositor", + "eve-tui-operator", + "cultmesh-command-surface" + ] + }, + "site": { + "owner": "Yggdrasil publisher daemon", + "input": "mimir.program_output.v1", + "route": "cultmesh://yggdrasil/gamecult.site/mimir/program" + }, + "compatibility": { + "obs": "optional-derived-sink" } }, "runtimePolicy": { @@ -93,7 +119,9 @@ "native-driver-workers", "native-reservoir", "Fensalir", - "Faust" + "Faust", + "Eve", + "Yggdrasil publisher daemon" ], "neverInCompatibilityEdges": [ "dense-feature-extraction", diff --git a/docs/architecture.md b/docs/architecture.md index 607fcfa..c2195d9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,11 +2,16 @@ ## Objective -Make one local Windows computer available as OBS sources on another computer, with sender-side NVENC video encoding and independently mixable audio sources. +Make one local Windows computer available as compatibility media sources on +another computer, with sender-side NVENC video encoding and independently +mixable audio sources. The current program target is Mimir-owned composition; +this bridge is not the stream authority. ## Current Mechanism -The repo owns configuration and scripts. FFmpeg owns capture, encoding, and SRT transport. OBS owns scene composition. +The repo owns configuration and scripts. FFmpeg owns capture, encoding, and SRT +transport. Mimir owns scene composition; OBS may consume this bridge as a +temporary compatibility sink. Inputs become outputs like this: @@ -21,8 +26,10 @@ Inputs become outputs like this: ## Invariants - Video encoding happens on the sender. -- OBS receives ordinary media-source URLs; it does not need a plugin for v1. -- Audio remains separately mixable in OBS. +- OBS can receive ordinary media-source URLs for compatibility; it does not need + a plugin for this bridge path. +- Audio remains separately mixable at the compatibility edge, but the durable + control surface belongs to Mimir/Eve. - Ports are stable and derived from one base port plus explicit offsets. - Secrets, if used, stay in local config and are not committed. diff --git a/docs/code-algorithm-map.md b/docs/code-algorithm-map.md index 8ebd9be..7be4449 100644 --- a/docs/code-algorithm-map.md +++ b/docs/code-algorithm-map.md @@ -15,8 +15,9 @@ Timestamp for this pass: 2026-05-23, starting at 23:24 Europe/London. Mimir collects local and network audio/video samples into bounded rolling runtime buffers, uses active/passive timing evidence to place audio streams on a -canonical timeline, and leaves Fensalir/Faust/OBS with clear ownership over -rendering, DSP, and broadcast composition. +canonical timeline, and lets Mimir own program composition while Fensalir lowers +pixels, Faust/native DSP moves audio, Eve exposes operator surfaces, and +Yggdrasil publishes the public feed. ## Current Mechanism @@ -54,11 +55,17 @@ reflection taps. - UI and telemetry read cached sync state; they do not run analysis. - Native capture owns hardware calls. Runtime owns sample identity and buffer placement. -- Fensalir owns windowing, rendering, D3D12, UI, visual fusion, and program - video output. +- Mimir owns the program scene graph, subscription policy, preview/control + state, stats, and publication intent. +- Fensalir owns windowing, rendering, D3D12, UI lowering, visual fusion, and + program video output. - Faust/native DSP will own hot resampling, fractional delay, separation, spatialization, and stems. -- OBS owns broadcast composition. +- Eve GUI/TUI lowerers render Mimir operator surfaces without owning scene + truth. +- The Yggdrasil-facing publisher daemon consumes Mimir program output without + owning composition. +- OBS is a compatibility sink only. ## Diagram @@ -83,6 +90,10 @@ flowchart TD AsioSource --> Sources Probes["native/probes/*"] --> FrameEvents["MimirFrameEventProcessStreamSource"] FrameEvents --> Sources + Runtime --> Scene["Mimir program scene graph"] + Scene --> Eve["Eve operator surfaces"] + Scene --> Publisher["Yggdrasil site publisher"] + Scene -. "compatibility" .-> Obs["OBS adapter"] Reservoir["native/reservoir"] -. "future lower ABI" .-> Hub ``` @@ -96,6 +107,9 @@ flowchart TD - `native/reservoir` is the lower native rolling-buffer ABI. - `native/probes/*` are measurement and diagnostic tools. - `scripts/*` are bridge utilities, not the final synchronized program path. +- `docs/mimir-program-composition.md` is the authority map for Muninn capture + hosts, Mimir composition, Eve operation, Yggdrasil publication, and OBS + compatibility. ## Stage 0: Solution And App Host diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 608b6c4..a44944a 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -13,12 +13,12 @@ For the source-level ownership map, read [[code-algorithm-map|Code Algorithm Map ingest, one rolling buffer per configured audio/video stream, and the synchronization hub Fensalir can inspect. - The default five-second window is an intentional latency/memory trade: use it - to line up streams and extract the volumetric audio/video field before OBS - sees program output. + to line up streams and extract the volumetric audio/video field before Mimir + emits program output. - `native/reservoir` owns the lower native rolling-buffer invariant for Fensalir/Faust integration. -- PowerShell/FFmpeg/SRT remains a bridge utility for LAN OBS feeds. It is not - the synchronized program authority. +- PowerShell/FFmpeg/SRT remains a bridge utility for LAN compatibility feeds. + It is not the synchronized program or publication authority. The old script stack is gone. Do not add a compatibility edge unless it protects a named invariant that the native runtime cannot protect yet. @@ -184,7 +184,7 @@ a named invariant that the native runtime cannot protect yet. profiles, language/emission profiles, path-learning sessions, benchmark panels, audio actuator strategies, native capture profiles, camera ingest strategies, reservoir strategies, audio/visual field profiles, compute - offload profiles, OBS publication profiles, CultMesh contracts, Fensalir + offload profiles, program publication profiles, CultMesh contracts, Fensalir lowering, and assembly plans. Use `Mimir.BufferSmoke --perfect-machine-profile-smoke` to prove the catalog assembles, `--perfect-machine-contract-smoke` to write a CultCache contract @@ -202,14 +202,40 @@ a named invariant that the native runtime cannot protect yet. source-id hashing, producer helpers, and typed audio/render payload descriptors. - Windows bridge scripts for sender discovery/start/stop and simple OBS Media - Source ingest. + Source compatibility ingest. +- `src/Mimir.Runtime/Synchronization` now contains the first typed program + composition contracts: `mimir.program_scene.v1`, `mimir.program_output.v1`, + and `mimir.eve_operator_surface.v1`. These make Mimir's scene graph, + Yggdrasil/site publication route, and Eve operator control/preview surface + manifestable through CultMesh. +- `docs/mimir-program-composition.md` is the current authority map for the new + stream-program architecture: Muninn observes on Starfire/Nightwing/Raven-class + hosts, Mimir consumes selected streams and composes the program, Eve lowers + controls/previews/stats, Yggdrasil publishes to the site, and OBS is + compatibility-only. +- `src/Mimir.CultMeshMedia` is the first explicit CultMesh media/body bridge: + `relay` hosts the document relay on CultNet UDP `3075`, `send` reads an + MPEG-TS byte stream from stdin and publishes rolling + `mimir.cultmesh_media_frame` documents, and `recv` subscribes to those + documents and writes ordered MPEG-TS bytes to a Starfire-local UDP endpoint + for compatibility sinks. +- `scripts/start-raven-cultmesh-av-sender.ps1`, + `scripts/start-yggdrasil-cultmesh-media-relay.ps1`, and + `scripts/start-starfire-cultmesh-av-receiver.ps1` are the CultMesh bridge + operators. Raven capture defaults to FFmpeg desktop frames plus Mimir's + WASAPI loopback capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains + an explicit fallback. - Documentation for OBS receiver setup, native rebuild boundaries, the viable stream app, and the Mimir Face. ## Temporary - Audio and video may still traverse separate OBS/SRT endpoints during bridge - testing so OBS can preserve independent controls. + testing, but Mimir/Eve owns independent controls. +- The CultMesh media bridge still lowers to local UDP for compatibility sinks + because OBS is not a CultMesh consumer. Network transit between Raven, + Yggdrasil, and Starfire is the CultMesh/CultNet path; OBS-local UDP is an + egress adapter only. - Process-backed stream sources are only acceptable for network bridge feeds or diagnostics. Six-camera local ingest belongs behind direct capture drivers. - Frame-event process sources are diagnostic only. They prove source cadence and @@ -239,14 +265,20 @@ a named invariant that the native runtime cannot protect yet. with probe durations long enough to keep loopback and mic windows live. 6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. -7. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: +7. Implement the Mimir program scene graph as the shared commit primitive for + source subscription, transforms, crop, chroma key, visibility, layer order, + preview, and output publication. Import the current OBS scene only as an + initial mirror, then make Eve GUI/TUI the operator surface. +8. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir + program output and publishes it without owning a second composition. +9. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: sample buffer-domain paths stochastically by visual contribution, emit SDF splat probes, write them into the spatiotemporal splat reservoir, and sample that reservoir in the temporally antialiased scene pass. The direct spline preview must stay a witness until this path owns rendering. -8. Move GPU feature extraction, fusion, material fitting, render budgeting, and +10. Move GPU feature extraction, fusion, material fitting, render budgeting, and Spout2 publication into Fensalir. -9. Move mic alignment, room suppression, voice separation, spatialization, and +11. Move mic alignment, room suppression, voice separation, spatialization, and stem generation into Faust/native DSP. -10. Keep the OBS bridge witness ledger as evidence before expanding receiver +12. Keep the OBS bridge witness ledger as evidence before expanding receiver machinery. diff --git a/docs/mimir-face.md b/docs/mimir-face.md index 3c9ea4d..40efc7f 100644 --- a/docs/mimir-face.md +++ b/docs/mimir-face.md @@ -48,9 +48,10 @@ Mimir exists to turn cameras, microphones, speakers, timing signals, and distributed capture surfaces into one realtime coherent volumetric field. The field is the product. The bridge scripts are scaffolding and probes. -The live target is one native five-second rolling reservoir feeding Fensalir GPU -fusion and Faust/native DSP, then OBS. Retention has one owner. Typed views are -indexes, not private histories. +The live target is one native five-second rolling reservoir feeding Mimir-owned +program composition, Fensalir GPU fusion, Faust/native DSP, Eve operator +surfaces, and a Yggdrasil site publisher. OBS is compatibility only. Retention +has one owner. Typed views are indexes, not private histories. ## Operating Doctrine @@ -63,11 +64,15 @@ the synchronized program path is being rebuilt around `LocalcastRuntime`, Invariants: -- OBS owns final broadcast composition. -- Fensalir owns dense visual fusion, material/brush/splat rendering, and Spout2 - publication. +- Mimir owns final program composition, source subscription, preview/control + state, stats, and publication intent. +- Fensalir owns dense visual fusion, material/brush/splat rendering, and local + program texture output. - Faust/native DSP owns hot audio alignment, room suppression, voice separation, stems, and spatial bed generation. +- Eve lowerers render operator controls and preview without owning scene truth. +- The Yggdrasil publisher daemon ships the Mimir program to the site without + owning composition. - FFmpeg owns simple capture, encode, and transport in V1 bridge mode. - Mimir/this repo owns configuration, calibration, launch/status, typed contracts, memory, and operational repeatability. diff --git a/docs/mimir-program-composition.md b/docs/mimir-program-composition.md new file mode 100644 index 0000000..2febc26 --- /dev/null +++ b/docs/mimir-program-composition.md @@ -0,0 +1,111 @@ +# Mimir Program Composition + +## Objective + +Mimir owns the live stream program: selected capture feeds, calibration feeds, +scene composition, preview/control surfaces, stats, and publication. OBS is a +temporary compatibility sink only. + +## Authority Map + +- Owner: Mimir owns the program scene graph and subscription policy. +- Inputs: Muninn-published local and remote monitor, window, camera, audio, + loopback, and health streams. +- Outputs: a Mimir program video surface, aligned audio stems or final bus, Eve + GUI/TUI operator surfaces, and a Yggdrasil-facing site publication feed. +- Derived state: OBS receives a compatibility feed when useful; it does not own + scene truth, broadcast truth, preview truth, stats, source selection, or + publication. +- Forbidden writers: OBS scenes, browser dashboards, site daemons, and Muninn + capture hosts must not independently decide the program composition. +- Shared paths: direct operator edits, remote Eve commands, preset loads, + calibration-driven source selection, and recovery scripts must commit through + the same Mimir scene graph command surface. +- Deletion line: old OBS-owned composition docs and config names are demoted to + diagnostic bridge language. Any OBS adapter delegates to the Mimir program + scene instead of carrying its own opinion. + +## Machine Shape + +```mermaid +flowchart TD + A["Muninn on Starfire"] --> D["CultMesh stream catalog + media bodies"] + B["Muninn on Nightwing"] --> D + C["Muninn on Raven"] --> D + D --> E["Mimir subscription policy"] + E --> F["Mimir rolling buffers"] + F --> G["Mimir program scene graph"] + G --> H["Fensalir compositor"] + F --> I["Faust/native DSP"] + H --> J["program video surface"] + I --> K["program audio stems/bus"] + G --> L["Eve GUI/TUI operator surfaces"] + J --> M["Yggdrasil site publisher daemon"] + K --> M + M --> N["GameCult site"] + J -. "compatibility" .-> O["OBS adapter"] + K -. "compatibility" .-> O +``` + +## Composition Contract + +The scene graph carries layers with: + +- source reference, such as `muninn:starfire:monitor:primary` or + `muninn:starfire:window:discord`; +- source kind, such as monitor, window, camera, network monitor, or generated + Mimir visual field; +- visibility, layer order, transform, crop, and optional chroma key; +- enough timing identity for Mimir to align media inside the rolling window. + +The current OBS scene can be mirrored as an initial Mimir scene: + +- Starfire monitor capture, cropped at the bottom when needed. +- Discord window capture, cropped to chat and keyed with chroma filters. +- Raven primary monitor plus Realtek loopback as a subscribed remote Muninn + stream. +- Mimir/Fensalir program texture as a generated layer when the field renderer is + active. + +After import, OBS is not the editor of record. It may display the Mimir program +feed, but operator edits happen through Mimir/Eve command surfaces. + +## Eve Operator Surface + +Mimir publishes Eve GUI/TUI documents for: + +- live program preview; +- source catalog and subscription state; +- layer transform, crop, chroma key, visibility, and ordering; +- audio stem gain/mute/route controls; +- stream health, latency, buffer depth, timing confidence, and publication + status; +- emergency scene selection and publisher recovery commands. + +Eve lowerers render those controls on any device without owning the underlying +scene graph. + +## Site Publisher + +The Yggdrasil-facing publisher consumes Mimir program output and publishes it to +the GameCult site. It owns transport, encoding/session state, reconnect policy, +and public edge health. It does not own composition. + +The publisher's input contract is the same Mimir program output used by local +preview and OBS compatibility adapters, so the public stream cannot silently +diverge from the operator preview. + +## Runtime Contracts + +The first typed contracts live in `src/Mimir.Runtime/Synchronization`: + +- `mimir.program_scene.v1` +- `mimir.program_output.v1` +- `mimir.eve_operator_surface.v1` + +The proof path is: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke +``` diff --git a/docs/native-rebuild-plan.md b/docs/native-rebuild-plan.md index 39c648b..a76edff 100644 --- a/docs/native-rebuild-plan.md +++ b/docs/native-rebuild-plan.md @@ -5,17 +5,24 @@ hosted stream machine. ## Objective -Make the room available in OBS as synchronized program video plus separately +Make the room available as a Mimir-owned synchronized program with separately controllable audio while preserving ownership: - Mimir owns configuration, calibration truth, runtime contracts, launch, status, and persistence. - Mimir.Runtime owns in-memory stream buffers and runtime synchronization. - Native capture workers own device reads and append typed sample handles. -- Fensalir owns GPU fusion, rendering, D3D12 interop, UI, and Spout2 output. +- Mimir owns stream subscription, scene composition, preview/control state, + stats, and program publication intent. +- Fensalir owns GPU fusion, rendering, D3D12 interop, UI lowering, and local + program texture output. - Faust/native DSP owns hot audio alignment, suppression, separation, stems, and spatial bed generation. -- OBS owns broadcast controls. +- Eve GUI/TUI lowerers expose operator controls and preview without owning + scene truth. +- The Yggdrasil-facing publisher daemon consumes Mimir program output and + publishes it to the site without owning composition. +- OBS is a temporary compatibility sink. ## Runtime State Model @@ -26,10 +33,13 @@ flowchart TD N["network feed producers"] --> R R --> Q["Fensalir UI + GPU fusion"] R --> F["Faust/native DSP"] - Q --> S["Spout2/program video"] + Q --> S["Mimir program video"] F --> P["program stems + spatial bed"] - S --> O["OBS"] - P --> O + Q --> E["Eve operator surfaces"] + S --> Y["Yggdrasil site publisher"] + P --> Y + S -. "compatibility" .-> O["OBS adapter"] + P -. "compatibility" .-> O ``` The app-level runtime initializes one rolling buffer per configured audio or @@ -48,7 +58,8 @@ camera driver buffers, GPU textures, or DSP audio memory. - Fensalir consumes current visual/audio timing state and publishes the program video surface. - Faust/native DSP consumes audio/phase/event handles and publishes stems. -- OBS receives final program surfaces. It does not own synchronization. +- OBS compatibility adapters may receive final program surfaces. They do not + own synchronization, composition, preview/control, or publication. ## Impossible By Construction diff --git a/docs/perfect-machine-domain-index.md b/docs/perfect-machine-domain-index.md index 11c86ce..cd627b3 100644 --- a/docs/perfect-machine-domain-index.md +++ b/docs/perfect-machine-domain-index.md @@ -38,19 +38,23 @@ Read this with: Mimir is a bounded-latency realtime field computer: it ingests many clocks, places their evidence inside one rolling temporal window, extracts audio/video -geometry from that window, and emits an OBS-usable program surface without -letting OBS become the synchronization authority. +geometry from that window, owns the stream program scene, exposes Eve operator +surfaces, and emits a publicable program feed without letting compatibility +sinks become authority. ## Authority Index | Authority | Owns | Must Not Own | | --- | --- | --- | -| `Mimir.Runtime` | stream identity, buffers, sync reports, calibration model loading, source polling, telemetry state | hot DSP actuator, dense visual fusion, broadcast composition | -| Native capture | direct driver callbacks, device timestamps, payload handles, low-jitter capture loops | runtime policy, UI state, OBS composition | +| Mimir program composition | source subscription, scene graph, crop/key/transform state, preview/control state, stats, publication intent | local device reads, GPU shader internals, audio-rate DSP, site transport policy | +| `Mimir.Runtime` | stream identity, buffers, sync reports, calibration model loading, source polling, telemetry state | hot DSP actuator, dense visual fusion, direct device capture | +| Native capture / Muninn hosts | direct driver callbacks, device timestamps, payload handles, local stream capabilities/media | runtime policy, scene composition, publication | | `native/reservoir` | shared-edge retention and typed views | payload memory interpretation, rendering, DSP | -| Fensalir | window, UI, D3D12 bridge, GPU fusion, program video publication | clock authority, audio actuator | +| Fensalir | window, UI lowering, D3D12 bridge, GPU fusion, program video output | clock authority, audio actuator, scene ownership | | Faust/native DSP | fractional delay, resampling, separation, spatialization, stems | stream discovery, UI | -| OBS | final broadcast composition | synchronization, capture timing | +| Eve lowerers | operator GUI/TUI rendering, controls, previews, stats lowering | scene truth, stream truth, publication truth | +| Yggdrasil site publisher | public transport, encoder/session state, reconnect policy, site edge health | scene composition | +| OBS | compatibility preview/output sink | synchronization, capture timing, composition, publication | | Raven/phones | local decode of codebook/schedule observations | canonical clock authority | ## Evidence Taxonomy @@ -241,7 +245,9 @@ Six optical sensors with high-rate Eyes and Leap IR mean: - Cut JSON/stdout local camera hot path when concrete direct drivers land. - Cut legacy chirplet path once chirp-bin decoder has equivalent docs/tests and physical path proof. -- Cut bridge scripts only after OBS-facing direct output is real. +- Cut bridge scripts only after Mimir program output, Eve operation, and + Yggdrasil publication are real enough that OBS compatibility no longer earns + its keep. - Cut ad hoc response weighting if a formal path model replaces it. ## Future Work Index diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index fd0e44d..c5fa824 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -27,7 +27,7 @@ job-interview answer for why it belongs in the body of the machine. | Visual fusion configurations | `Mimir.Runtime` + Fensalir | Sensor roles and fusion modes from cadence proof through full hybrid evidence. | `src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs` | | Compute offload configurations | `Mimir.Runtime` + CultMesh | Local-heavy, distributed witness, and calibration sweep workload placement. | `src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs` | | Assembly plans | `Mimir.Runtime` | Named combinations of modules that prove synthetic, Scarlett, meatspace, and distributed cuts. | `src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs` | -| OBS publication profiles | Fensalir + Faust + OBS | Spout/shared-texture video and separately mixable Faust stem publication. | `src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs` | +| Program publication profiles | Mimir + Fensalir + Faust + Eve + Yggdrasil | Mimir-owned scene publication, Eve operator surfaces, Yggdrasil site feed, and temporary OBS compatibility adapters. | `src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs` | | Fensalir lowering | `Mimir.Runtime` -> Fensalir | Pure mapping from Mimir rolling/video/audio state into Fensalir GPU sensor and acoustic field frames. | `src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs` | | Module catalog | `Mimir.Runtime` | Indexes the live module set and verification commands. | `src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs` | | Manifest factory | `Mimir.Runtime` | Produces an exportable JSON manifest for tools, docs, UI, and remote witnesses. | `src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs` | @@ -42,7 +42,8 @@ job-interview answer for why it belongs in the body of the machine. stem generation. - CultMesh carries typed state surfaces; remote nodes may observe and decode, but they do not become independent clock authorities. -- OBS receives final program surfaces and stems. It does not own calibration. +- OBS may receive compatibility program surfaces and stems. It does not own + calibration, composition, preview/control, or publication. ## Verification Surface diff --git a/docs/viable-stream-app.md b/docs/viable-stream-app.md index a304701..eb58c07 100644 --- a/docs/viable-stream-app.md +++ b/docs/viable-stream-app.md @@ -4,12 +4,13 @@ Build the smallest live Mimir application that can ingest multiple local or networked video and audio feeds, hold a short synchronized window in memory, -and emit coherent OBS-facing program outputs while Fensalir provides the -operator UI. +and emit coherent Mimir program outputs while Eve provides operator surfaces +and Yggdrasil publishes the site feed. The default reservoir horizon is five seconds. That delay is not decoration. It is the budget that lets late network packets, independent capture clocks, -audio/video alignment, and debug controls converge before OBS sees the result. +audio/video alignment, and debug controls converge before Mimir emits the +program result. Mimir is explicitly trading latency and memory for throughput and extraction quality: the app may spend that buffered window lining streams up and crunching audio/video data to recover as much of the volumetric audio field and @@ -21,16 +22,16 @@ alignment and field-reconstruction budget. Today the repo has two partial machines: -- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable OBS - Media Source endpoints. +- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable + compatibility endpoints. - The native `LocalcastRuntime` reservoir ABI can hold typed live sample handles, and Fensalir can consume native render packets for GPU fusion. Those parts are useful, but they are not yet the viable app. The viable app is the join: ingest workers append live sample handles into one Fensalir-hosted runtime window, Fensalir renders video from that window, Faust/native audio -builds synchronized stems from the same window, and OBS receives only the -program outputs. +builds synchronized stems from the same window, Mimir commits the scene graph, +Eve lowers the operator surface, and Yggdrasil publishes the program output. ```mermaid flowchart TD @@ -39,10 +40,15 @@ flowchart TD C["local mic/loopback workers"] --> R D["network audio ingest workers"] --> R R --> U["Fensalir debug/settings/output UI"] - R --> V["Fensalir GPU video composition"] + R --> S["Mimir program scene graph"] + S --> U + S --> V["Fensalir GPU video composition"] R --> F["Faust/native audio alignment + stems"] - V --> O["OBS video output"] - F --> P["OBS audio stem outputs"] + S --> E["Eve GUI/TUI operator surfaces"] + V --> Y["Yggdrasil site publisher"] + F --> Y + V -. "compatibility" .-> O["OBS adapter"] + F -. "compatibility" .-> O ``` ## Invariants @@ -51,12 +57,17 @@ flowchart TD - The five-second rolling window is a deliberate compute budget, not accidental latency. It exists so capture, sync, GPU fusion, and audio field extraction can choose coherence over immediacy. -- Fensalir owns the running app, UI, debug state, settings changes, output - management, GPU fusion/rendering, and Spout/video publication. +- Mimir owns the program scene graph, source subscription policy, preview/control + state, stats, and publication intent. +- Fensalir owns the running app host, UI lowering, debug state, GPU + fusion/rendering, and local program texture output. - Faust/native DSP owns audio alignment, voice separation, suppression, volumetric sound-field extraction, stems, and spatial bed generation. -- OBS receives synchronized program surfaces. Raw unsynchronized feeds may be - diagnostic inputs, not broadcast truth. +- Eve lowerers render controls and previews without owning scene truth. +- The Yggdrasil publisher consumes synchronized program surfaces without owning + a second composition. +- OBS compatibility adapters may receive synchronized program surfaces. Raw + unsynchronized feeds may be diagnostic inputs, not broadcast truth. - Networked feeds are producers into the same reservoir. They do not own clocks, private buffers, or a parallel synchronization model. - Missing feeds become explicit absence or silent/blank placeholders inside the @@ -78,9 +89,14 @@ The first cut should be deliberately boring: loopback ingest. 7. Expose Fensalir UI readouts for reservoir edge, window start, per-kind counts, feed health, delay estimate, output status, and current settings. -8. Render one synchronized program video output from the reservoir. -9. Emit separately controllable OBS audio stems from the same reservoir window. -10. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture +8. Render one synchronized Mimir program video output from the reservoir. +9. Emit separately controllable program audio stems from the same reservoir + window. +10. Publish Eve GUI/TUI operator surfaces for preview, source selection, scene + edits, stats, and output state. +11. Add the Yggdrasil-facing publisher daemon that consumes Mimir program + output without owning composition. +12. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture and OBS compatibility, not as the stream authority. ## Repo Shape diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 90077fb..0317e12 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -25,10 +25,13 @@ flowchart TD R --> N["native reservoir handles"] N --> E["Fensalir GPU fusion + UI"] N --> F["Faust/native DSP"] - E --> G["Spout2/program video"] + E --> G["Mimir program video"] F --> H["program stems + spatial bed"] - G --> I["OBS"] - H --> I + E --> I["Eve GUI/TUI operator surfaces"] + G --> J["Yggdrasil site publisher"] + H --> J + G -. "compatibility" .-> K["OBS adapter"] + H -. "compatibility" .-> K ``` Ownership: @@ -41,11 +44,18 @@ Ownership: becoming independent clock authorities. - `Mimir.Runtime` owns app-level stream buffers and synchronization. - Native capture workers own device reads. +- Mimir owns program composition, source subscription policy, preview/control + state, stats, and publication intent. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, - D3D12 interop, runtime UI, and Spout2 publication. + D3D12 interop, runtime UI lowering, and local program texture output. - Faust/native DSP owns hot audio alignment, suppression, separation, spatialization, and stem generation. -- OBS owns broadcast controls. +- Eve GUI/TUI lowerers render Mimir's operator surfaces on any device without + owning scene truth. +- The Yggdrasil-facing publisher daemon consumes Mimir program output and + publishes it to the site without owning composition. +- OBS is a temporary compatibility sink, not a composition or broadcast + authority. Invariant: the live window is bounded, in memory, and has one timing authority. No private history outlives the rolling buffer. @@ -54,7 +64,7 @@ No private history outlives the rolling buffer. [[docs/viable-stream-app|Viable Stream App]] defines the near-term app target. Fensalir hosts the running Mimir app, keeps the default five-second runtime in -memory, exposes debug/settings/output controls, and emits synchronized OBS +memory, exposes debug/settings/output controls, and emits synchronized Mimir program video plus separately controllable audio stems. `Mimir.Runtime` currently provides: @@ -79,6 +89,25 @@ the rolling buffers while the direct ABI driver is being cut. It does not carry pixels and does not own the final six-camera hot path. Multi-camera probes are one process with declared accepted source ids, not one process per camera. +## Program Composition + +[[docs/mimir-program-composition|Mimir Program Composition]] is the live +authority map for stream-program output. Muninn runs on Starfire, Nightwing, and +Raven-class capture hosts to publish local stream capabilities and selected +media bodies. Mimir consumes only the streams it needs for calibration and +composition, commits one typed scene graph, emits Eve GUI/TUI operator surfaces, +and publishes program output locally plus through a Yggdrasil-facing site +publisher. + +The first typed program contracts live in `src/Mimir.Runtime/Synchronization`: + +- `mimir.program_scene.v1` +- `mimir.program_output.v1` +- `mimir.eve_operator_surface.v1` + +OBS scene JSON is import/mirror evidence for the initial crop/key/transform +layout. It is not the editor of record after Mimir owns the scene graph. + ## OBS Bridge Utility ```mermaid @@ -97,6 +126,42 @@ flowchart TD The bridge is useful because it is inspectable and already speaks OBS. It does not become the synchronized program authority. +## CultMesh Media Bridge + +```mermaid +flowchart TD + A["Raven FFmpeg desktop + WASAPI loopback mux"] --> B["Mimir.CultMeshMedia send"] + B --> C["CultNet reliable UDP media-frame documents"] + C --> D["Yggdrasil CultMesh relay on 10.77.0.1:3075"] + D --> E["Starfire Mimir.CultMeshMedia recv"] + E --> F["local MPEG-TS UDP udp://127.0.0.1:5200"] + F --> G["OBS Raven Monitor + Realtek"] +``` + +Owner: `src/Mimir.CultMeshMedia` owns the live Raven-to-Starfire media body +bridge over CultMesh/CultNet. FFmpeg still owns Raven capture and encoding. +Mimir owns composition. OBS may consume the Starfire-local MPEG-TS endpoint as a +compatibility sink while Mimir/Fensalir program output and site publication +mature. + +Invariant: Raven media body state is live and bounded. Sender writes rolling +`mimir.cultmesh_media_frame` slot documents rather than an unbounded durable +video archive. Yggdrasil relays CultNet reliable UDP over the WireGuard mesh. +Starfire lowers the subscribed stream to local UDP for compatibility sinks +because OBS is not a CultMesh runtime. + +Current deployment: Yggdrasil runs the relay from +`/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia` with cache +`/var/lib/gamecult/mimir/cultmesh-media.cc` and log +`/var/log/gamecult/mimir-cultmesh-media.log`. Starfire runs the receiver from +`artifacts/mimir-cultmesh-media-win-x64-selfcontained/Mimir.CultMeshMedia.exe` +and writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS source +`Raven Monitor + Realtek` points at that local UDP endpoint. + +Raven SSH over WireGuard timed out during the initial deployment pass, so the +Raven sender still needs to be launched on Raven directly with +`scripts/start-raven-cultmesh-av-sender.ps1`. + ## Audio Field The six-microphone path is separate from the bridge. Scarlett speaker loopback diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index f164d87..aa1eda9 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -21,7 +21,11 @@ Get-Content .\state\evidence.jsonl -Tail 8 - Mimir's Face doctrine lives in [[docs/mimir-face|Mimir Face]]. - VoidBot persona/state live under `.voidbot/voice/` and `.voidbot/state/`. Commit `.voidbot` dirt in a separate state commit whenever it appears. -- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN ingest. +- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN compatibility ingest. +- V1 now also has an explicit CultMesh media bridge for Raven program feeds: + `src/Mimir.CultMeshMedia` sends rolling `mimir.cultmesh_media_frame` + documents over CultNet reliable UDP through Yggdrasil and lowers them on + Starfire to local MPEG-TS UDP for OBS. - The live stream app is C# plus Fensalir: `Mimir.slnx` contains `src/Mimir.App` and `src/Mimir.Runtime`. - `src/Mimir.App` hosts Fensalir as the windowing/rendering/D3D12 bridge. @@ -37,6 +41,14 @@ Get-Content .\state\evidence.jsonl -Tail 8 dictionary, low-level implementation notes, and sample code sketches for the next decoder/DSP/native-ring/Fensalir implementation passes. Start with [[research/perfect-machine-study-2026-05-23/reading-guide|Reading Guide]]. +- `docs/mimir-program-composition.md` is the current program-output authority + map: Muninn observes on Starfire/Nightwing/Raven-class hosts, Mimir consumes + selected streams and owns composition, Eve lowers controls/previews/stats, + Yggdrasil publishes to the site, and OBS is compatibility-only. +- `src/Mimir.Runtime/Synchronization` now has typed program contracts for + `mimir.program_scene.v1`, `mimir.program_output.v1`, and + `mimir.eve_operator_surface.v1`; `Mimir.BufferSmoke + --perfect-machine-contract-smoke` writes them into CultCache. - Local six-camera ingest should use direct driver adapters. Process-backed sources are bridge/network edges only. - Leap stereo IR is the first timing-camera candidate for direct ingest. @@ -45,7 +57,8 @@ Get-Content .\state\evidence.jsonl -Tail 8 - Fensalir owns production GPU fusion, UI, and Spout2 publication. - Faust/native DSP owns hot audio alignment, separation, spatialization, and synchronized stems. -- OBS receives final program surfaces; it does not own synchronization. +- OBS compatibility adapters may receive final program surfaces; they do not + own synchronization, composition, preview/control, or publication. ## Current Pressure @@ -57,8 +70,16 @@ Get-Content .\state\evidence.jsonl -Tail 8 variable-rate resampling. - Bind Fensalir UI to stream health, buffer depth, timestamps, settings, and output management. -- Keep PowerShell/FFmpeg/SRT as bridge utilities until native program output - replaces them. +- Keep PowerShell/FFmpeg/SRT as bridge utilities until native Mimir program + output, Eve operation, and Yggdrasil site publication replace them. +- Yggdrasil currently runs the CultMesh media relay on UDP `3075` from + `/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia`; Starfire currently + runs the receiver PID from + `artifacts/mimir-cultmesh-media-win-x64-selfcontained/Mimir.CultMeshMedia.exe` + and writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS source + `Raven Monitor + Realtek` has been repointed from old SRT to that local UDP + endpoint. Raven SSH timed out, so launch + `scripts/start-raven-cultmesh-av-sender.ps1` on Raven directly. - Kiyo Pro has two UVC extension units. Moving it to a motherboard USB3 port changed the descriptor path to `root_hub30` / `bcdUSB=0x0320` and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index fe3617b..e9e9ae4 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1230,7 +1230,7 @@ static int RunPerfectMachineProfileSmoke() var computePlans = MimirComputeOffloadConfigurations.BuiltIn; var assemblyPlans = MimirMachineAssemblyPlans.BuiltIn; var captureProfiles = MimirNativeCaptureConfigurations.BuiltIn; - var publications = MimirObsPublicationConfigurations.BuiltIn; + var publications = MimirProgramPublicationConfigurations.BuiltIn; var moduleCatalog = MimirModuleLibrary.Entries; var languageProfiles = MimirBioacousticLanguageConfigurations.BuiltIn; var pathLearningProfiles = MimirAcousticPathLearningConfigurations.BuiltIn; @@ -1261,6 +1261,15 @@ static int RunPerfectMachineProfileSmoke() "scarlett-host-mic-actuator", MimirAlignmentActuatorProfile.SixSourceFaust.Id, command); + var scene = MimirCultMeshContractFactory.CreateObsSceneMirror("mimir-current-program"); + var output = MimirCultMeshContractFactory.CreateProgramOutput( + "mimir-site-program", + scene.SceneId, + MimirProgramPublicationConfigurations.YggdrasilSiteProgram); + var eveSurface = MimirCultMeshContractFactory.CreateOperatorSurface( + "mimir-eve-gui-compositor", + scene.SceneId, + MimirProgramPublicationConfigurations.OperatorSurfaces[0]); var observations = new[] { new MimirBioacousticWordObservation(0, MimirBioacousticTimeline.Default.EventForIndex(0).StartSeconds * MimirBioacousticTimeline.SampleRate + 317.375, 0.95), @@ -1342,6 +1351,8 @@ static int RunPerfectMachineProfileSmoke() $"perfect-machine-codebook id={codebook.CodebookId} motifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} augmentations={decoder.Configuration.TemplateAugmentations.Length}"); Console.WriteLine( $"perfect-machine-actuator source={actuator.SourceId} delay={actuator.TargetDelaySamples:0.000} ratio={actuator.ResampleRatio:0.000000000} controls={actuator.FaustControls.Length}"); + Console.WriteLine( + $"perfect-machine-program scene={scene.SceneId} layers={scene.Layers.Length} output={output.OutputId} route={output.PublisherRoute} eveCommands={eveSurface.CommandTopics.Length}"); Console.WriteLine( $"perfect-machine-clock anchors={clock?.AnchorCount ?? 0} coverage={clock?.AnchorCoverage ?? 0:0.000} offset={clock?.SourceOffsetSamples ?? double.NaN:0.000} confidence={clock?.Confidence ?? 0:0.000}"); Console.WriteLine( @@ -1530,6 +1541,19 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) MimirAlignmentActuatorProfile.SixSourceFaust.Id, command, createdAt); + var scene = MimirCultMeshContractFactory.CreateObsSceneMirror( + "mimir-current-program", + createdAt); + var output = MimirCultMeshContractFactory.CreateProgramOutput( + "mimir-yggdrasil-site-program", + scene.SceneId, + MimirProgramPublicationConfigurations.YggdrasilSiteProgram, + createdAt); + var eveSurface = MimirCultMeshContractFactory.CreateOperatorSurface( + "mimir-eve-gui-compositor", + scene.SceneId, + MimirProgramPublicationConfigurations.OperatorSurfaces[0], + createdAt); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions @@ -1552,10 +1576,22 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) actuator, new CultRecordHandle(new CultRecordKey($"mimir-actuator:{actuator.ActuatorId}"))) .ConfigureAwait(false); + await cache.UpsertAsync( + scene, + new CultRecordHandle(new CultRecordKey($"mimir-program-scene:{scene.SceneId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + output, + new CultRecordHandle(new CultRecordKey($"mimir-program-output:{output.OutputId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + eveSurface, + new CultRecordHandle(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}"))) + .ConfigureAwait(false); await cache.FlushAsync().ConfigureAwait(false); Console.WriteLine( - $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length}"); + $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length}"); return 0; } diff --git a/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs index 5b63587..35f3f92 100644 --- a/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs @@ -11,7 +11,8 @@ public enum MimirWorkloadKind CameraCapture, VisualFeatureExtraction, VisualFusion, - ObsPublication + ProgramComposition, + ProgramPublication } public sealed record MimirWorkloadPlacement( @@ -42,7 +43,8 @@ public static class MimirComputeOffloadConfigurations new(MimirWorkloadKind.CameraCapture, MimirComputeBackend.NativeSimd, "native capture workers", CanRunRemote: true, RequiresRealtime: true, "Drivers own device reads and timestamps."), new(MimirWorkloadKind.VisualFeatureExtraction, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "GPU owns dense feature work."), new(MimirWorkloadKind.VisualFusion, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "Fensalir owns temporal evidence field."), - new(MimirWorkloadKind.ObsPublication, MimirComputeBackend.D3D12Compute, "OBS/Fensalir output", CanRunRemote: false, RequiresRealtime: true, "OBS receives program surfaces, not sync authority.") + new(MimirWorkloadKind.ProgramComposition, MimirComputeBackend.D3D12Compute, "Mimir + Fensalir", CanRunRemote: false, RequiresRealtime: true, "Mimir owns the scene graph; Fensalir lowers it into program pixels."), + new(MimirWorkloadKind.ProgramPublication, MimirComputeBackend.D3D12Compute, "Mimir + Yggdrasil publisher", CanRunRemote: false, RequiresRealtime: true, "The site publisher consumes the Mimir program output; OBS is only an optional compatibility sink.") ]); public static MimirComputeOffloadConfiguration DistributedWitnesses { get; } = new( diff --git a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs index 2298b7b..3276d43 100644 --- a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs +++ b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs @@ -104,6 +104,73 @@ public sealed record MimirFaustControlSnapshot( [property: Key(0)] string Path, [property: Key(1)] float Value); +[CultDocument("mimir.program_scene", "mimir.program_scene.v1")] +[MessagePackObject] +public sealed record MimirProgramSceneDocument( + [property: Key(0)] + [property: CultName] + string SceneId, + [property: Key(1)] string UpdatedAtUtc, + [property: Key(2)] int CanvasWidth, + [property: Key(3)] int CanvasHeight, + [property: Key(4)] string Owner, + [property: Key(5)] MimirProgramSceneLayer[] Layers); + +[MessagePackObject] +public sealed record MimirProgramSceneLayer( + [property: Key(0)] string LayerId, + [property: Key(1)] string SourceRef, + [property: Key(2)] string SourceKind, + [property: Key(3)] bool Visible, + [property: Key(4)] double X, + [property: Key(5)] double Y, + [property: Key(6)] double Width, + [property: Key(7)] double Height, + [property: Key(8)] int ZIndex, + [property: Key(9)] MimirProgramCrop Crop, + [property: Key(10)] MimirProgramChromaKey? ChromaKey); + +[MessagePackObject] +public sealed record MimirProgramCrop( + [property: Key(0)] double Left, + [property: Key(1)] double Top, + [property: Key(2)] double Right, + [property: Key(3)] double Bottom); + +[MessagePackObject] +public sealed record MimirProgramChromaKey( + [property: Key(0)] uint KeyColorRgba, + [property: Key(1)] double Similarity, + [property: Key(2)] double Smoothness, + [property: Key(3)] double Spill); + +[CultDocument("mimir.program_output", "mimir.program_output.v1")] +[MessagePackObject] +public sealed record MimirProgramOutputDocument( + [property: Key(0)] + [property: CultName] + string OutputId, + [property: Key(1)] string UpdatedAtUtc, + [property: Key(2)] string SceneId, + [property: Key(3)] string VideoSurface, + [property: Key(4)] string AudioBus, + [property: Key(5)] string PublisherRoute, + [property: Key(6)] bool DiagnosticOnly); + +[CultDocument("mimir.eve_operator_surface", "mimir.eve_operator_surface.v1")] +[MessagePackObject] +public sealed record MimirEveOperatorSurfaceDocument( + [property: Key(0)] + [property: CultName] + string SurfaceId, + [property: Key(1)] string UpdatedAtUtc, + [property: Key(2)] string SceneId, + [property: Key(3)] bool CanPreviewProgram, + [property: Key(4)] bool CanEditComposition, + [property: Key(5)] bool CanSelectSources, + [property: Key(6)] bool CanShowStats, + [property: Key(7)] string[] CommandTopics); + public static class MimirCultMeshContractFactory { public static MimirBioacousticCodebookState CreateCodebookState( @@ -175,6 +242,92 @@ public static class MimirCultMeshContractFactory .Select(pair => new MimirFaustControlSnapshot(pair.Key, pair.Value)) .ToArray()); + public static MimirProgramSceneDocument CreateObsSceneMirror( + string sceneId, + DateTimeOffset? updatedAt = null) => + new( + sceneId, + (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"), + CanvasWidth: 1920, + CanvasHeight: 1080, + Owner: "Mimir", + Layers: + [ + new( + "starfire-monitor", + "muninn:starfire:monitor:primary", + "monitor", + Visible: true, + X: 0, + Y: 0, + Width: 1920, + Height: 1080, + ZIndex: 0, + new MimirProgramCrop(Left: 0, Top: 0, Right: 0, Bottom: 48), + ChromaKey: null), + new( + "discord-chat", + "muninn:starfire:window:discord", + "window", + Visible: true, + X: 1225.5, + Y: 0.5, + Width: 1012.2, + Height: 569.4, + ZIndex: 10, + new MimirProgramCrop(Left: 350, Top: 38, Right: 271, Bottom: 87), + new MimirProgramChromaKey(0xff2b7c5a, Similarity: 1, Smoothness: 2, Spill: 1)), + new( + "raven-monitor", + "muninn:raven:monitor:primary", + "network-monitor", + Visible: true, + X: 0, + Y: 0, + Width: 2560, + Height: 1440, + ZIndex: 20, + new MimirProgramCrop(Left: 0, Top: 0, Right: 0, Bottom: 0), + ChromaKey: null) + ]); + + public static MimirProgramOutputDocument CreateProgramOutput( + string outputId, + string sceneId, + MimirProgramPublicationConfiguration publication, + DateTimeOffset? updatedAt = null) => + new( + outputId, + (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"), + sceneId, + publication.VideoSurfaceName, + publication.AudioKind.ToString(), + publication.SitePublisherRoute, + publication.DiagnosticOnly); + + public static MimirEveOperatorSurfaceDocument CreateOperatorSurface( + string surfaceId, + string sceneId, + MimirProgramControlSurface surface, + DateTimeOffset? updatedAt = null) => + new( + surfaceId, + (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"), + sceneId, + surface.CanPreviewProgram, + surface.CanEditComposition, + surface.CanSelectSources, + surface.CanShowStats, + CommandTopics: + [ + "mimir.program.scene.select", + "mimir.program.layer.transform", + "mimir.program.layer.crop", + "mimir.program.layer.chroma_key", + "mimir.program.source.subscribe", + "mimir.program.output.publish" + ]); + private static MimirBioacousticDecoderConfigurationSnapshot Snapshot(MimirBioacousticDecoderConfiguration configuration) => new( configuration.Id, diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index d016807..b09d387 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -111,10 +111,10 @@ public static class MimirModuleLibrary MimirMachineAssemblyPlans.BuiltIn.Select(plan => plan.Id).ToArray(), ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), new( - "obs-publication", - "Fensalir + Faust + OBS", - "OBS-facing program surface and separately mixable audio stem configurations.", - MimirObsPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + "program-publication", + "Mimir + Fensalir + Faust + Eve + Yggdrasil", + "Mimir-owned program composition, Eve operator surfaces, Yggdrasil site publication, and temporary OBS compatibility adapters.", + MimirProgramPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]) ]; } diff --git a/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs deleted file mode 100644 index 35a8edc..0000000 --- a/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs +++ /dev/null @@ -1,83 +0,0 @@ -namespace Mimir.Runtime.Synchronization; - -public enum MimirObsVideoPublicationKind -{ - Spout2, - SharedD3D12Texture, - SrtDiagnosticBridge, - WindowCaptureDiagnostic -} - -public enum MimirObsAudioPublicationKind -{ - FaustStemBus, - AsioOutputPair, - WasapiLoopbackDiagnostic, - SrtDiagnosticBridge -} - -public sealed record MimirObsAudioStem( - string Id, - string DisplayName, - int ChannelCount, - bool UserMixable, - bool CarriesTimingWitness, - string Notes); - -public sealed record MimirObsPublicationConfiguration( - string Id, - string Description, - MimirObsVideoPublicationKind VideoKind, - string VideoSourceName, - MimirObsAudioPublicationKind AudioKind, - IReadOnlyList AudioStems, - double TargetPresentationDelaySeconds, - bool DiagnosticOnly); - -public static class MimirObsPublicationConfigurations -{ - public static IReadOnlyList ProgramStems { get; } = - [ - new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."), - new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."), - new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."), - new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."), - new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."), - new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."), - new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.") - ]; - - public static MimirObsPublicationConfiguration NativeProgram { get; } = new( - "native-program-spout-faust", - "Production target: Fensalir publishes synchronized video, Faust publishes aligned stems.", - MimirObsVideoPublicationKind.Spout2, - "Mimir Point Cloud", - MimirObsAudioPublicationKind.FaustStemBus, - ProgramStems, - TargetPresentationDelaySeconds: 5.0, - DiagnosticOnly: false); - - public static MimirObsPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with - { - Id = "native-program-d3d12-shared-texture", - VideoKind = MimirObsVideoPublicationKind.SharedD3D12Texture, - VideoSourceName = "Mimir D3D12 Program Texture" - }; - - public static MimirObsPublicationConfiguration DiagnosticSrtBridge { get; } = new( - "diagnostic-srt-bridge", - "OBS-facing fallback for bridge tests only. It does not own synchronized program truth.", - MimirObsVideoPublicationKind.SrtDiagnosticBridge, - "Mimir Diagnostic SRT", - MimirObsAudioPublicationKind.SrtDiagnosticBridge, - ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(), - TargetPresentationDelaySeconds: 1.5, - DiagnosticOnly: true); - - public static IReadOnlyList BuiltIn { get; } = - [ - NativeProgram, - TextureInteropProgram, - DiagnosticSrtBridge - ]; -} diff --git a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs index 4e20d8f..fe4de73 100644 --- a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs +++ b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs @@ -20,7 +20,7 @@ public sealed record MimirPerfectMachineManifest( IReadOnlyList AudioFields, IReadOnlyList VisualFields, IReadOnlyList ComputePlans, - IReadOnlyList Publications, + IReadOnlyList Publications, IReadOnlyList AssemblyPlans); public static class MimirPerfectMachineManifestFactory @@ -48,6 +48,6 @@ public static class MimirPerfectMachineManifestFactory MimirAudioFieldConfigurations.BuiltIn, MimirVisualFusionConfigurations.BuiltIn, MimirComputeOffloadConfigurations.BuiltIn, - MimirObsPublicationConfigurations.BuiltIn, + MimirProgramPublicationConfigurations.BuiltIn, MimirMachineAssemblyPlans.BuiltIn); } diff --git a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs index 9a09926..d872b9b 100644 --- a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs +++ b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs @@ -4,10 +4,11 @@ public enum MimirPerfectMachineNodeRole { StarfireAuthority, RavenRemote, + MuninnHost, PhoneWitness, MicrocontrollerWitness, CalibrationBench, - ObsProgramHost + MimirProgramPublisher } public enum MimirAudioFieldModel @@ -38,7 +39,7 @@ public sealed record MimirPerfectMachineProfile( IReadOnlyList ComputeBackends, bool EmitsWitness, bool OwnsCanonicalClock, - bool PublishesObsProgram, + bool PublishesProgram, TimeSpan RollingWindow, string[] RequiredCultMeshDocuments); @@ -46,7 +47,7 @@ public static class MimirPerfectMachineProfiles { public static MimirPerfectMachineProfile StarfireAuthority { get; } = new( "starfire-authority", - "Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Fensalir/Faust lowering, OBS-facing program output.", + "Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Mimir composition, Fensalir/Faust lowering, Eve operator surfaces, and program publication.", MimirPerfectMachineNodeRole.StarfireAuthority, MimirBioacousticDecoderConfiguration.BaselineMfccIndex, MimirAlignmentActuatorProfile.SixSourceFaust, @@ -54,7 +55,7 @@ public static class MimirPerfectMachineProfiles [MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute, MimirComputeBackend.FaustNativeDsp], EmitsWitness: true, OwnsCanonicalClock: true, - PublishesObsProgram: true, + PublishesProgram: true, RollingWindow: TimeSpan.FromSeconds(5), RequiredCultMeshDocuments: [ @@ -74,7 +75,7 @@ public static class MimirPerfectMachineProfiles [MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker], EmitsWitness: false, OwnsCanonicalClock: false, - PublishesObsProgram: false, + PublishesProgram: false, RollingWindow: TimeSpan.FromSeconds(5), RequiredCultMeshDocuments: [ @@ -93,7 +94,7 @@ public static class MimirPerfectMachineProfiles [MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker], EmitsWitness: false, OwnsCanonicalClock: false, - PublishesObsProgram: false, + PublishesProgram: false, RollingWindow: TimeSpan.FromSeconds(3), RequiredCultMeshDocuments: [ @@ -111,7 +112,7 @@ public static class MimirPerfectMachineProfiles [MimirComputeBackend.RemoteCultMeshWorker], EmitsWitness: false, OwnsCanonicalClock: false, - PublishesObsProgram: false, + PublishesProgram: false, RollingWindow: TimeSpan.FromSeconds(2), RequiredCultMeshDocuments: [ @@ -129,7 +130,7 @@ public static class MimirPerfectMachineProfiles [MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute], EmitsWitness: true, OwnsCanonicalClock: false, - PublishesObsProgram: false, + PublishesProgram: false, RollingWindow: TimeSpan.FromSeconds(8), RequiredCultMeshDocuments: [ @@ -138,22 +139,44 @@ public static class MimirPerfectMachineProfiles "mimir.acoustic_path_state.v1" ]); - public static MimirPerfectMachineProfile ObsProgramHost { get; } = new( - "obs-program-host", - "Program-output profile: consumes aligned stems/spatial bed and publishes OBS-facing surfaces without becoming sync authority.", - MimirPerfectMachineNodeRole.ObsProgramHost, + public static MimirPerfectMachineProfile MuninnHost { get; } = new( + "muninn-host", + "Capture host profile: Muninn observes local screens, windows, loopbacks, cameras, and device health, publishes typed stream capabilities/media, and does not compose the final program.", + MimirPerfectMachineNodeRole.MuninnHost, + MimirBioacousticDecoderConfiguration.CompactFastIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.AlignedStems, + [MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker], + EmitsWitness: false, + OwnsCanonicalClock: false, + PublishesProgram: false, + RollingWindow: TimeSpan.FromSeconds(5), + RequiredCultMeshDocuments: + [ + "muninn.telemetry_surface.v1", + "muninn.capture_stream.v1", + "mimir.cultmesh_media_frame.v1" + ]); + + public static MimirPerfectMachineProfile MimirProgramPublisher { get; } = new( + "mimir-program-publisher", + "Program-output profile: consumes selected Muninn/Mimir streams, owns scene composition and Eve controls, and publishes the program locally and through Yggdrasil without OBS authority.", + MimirPerfectMachineNodeRole.MimirProgramPublisher, MimirBioacousticDecoderConfiguration.BaselineMfccIndex, MimirAlignmentActuatorProfile.SixSourceFaust, MimirAudioFieldModel.SourceBasedSpatialBus, [MimirComputeBackend.FaustNativeDsp, MimirComputeBackend.D3D12Compute], EmitsWitness: false, OwnsCanonicalClock: false, - PublishesObsProgram: true, + PublishesProgram: true, RollingWindow: TimeSpan.FromSeconds(5), RequiredCultMeshDocuments: [ "mimir.acoustic_path_state.v1", - "mimir.actuator_state.v1" + "mimir.actuator_state.v1", + "mimir.program_scene.v1", + "mimir.program_output.v1", + "mimir.eve_operator_surface.v1" ]); public static IReadOnlyList All { get; } = @@ -163,6 +186,7 @@ public static class MimirPerfectMachineProfiles PhoneWitness, MicrocontrollerWitness, CalibrationBench, - ObsProgramHost + MuninnHost, + MimirProgramPublisher ]; } diff --git a/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs new file mode 100644 index 0000000..5dafc6d --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs @@ -0,0 +1,159 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirProgramVideoPublicationKind +{ + FensalirProgramTexture, + SharedD3D12Texture, + CultMeshMediaBody, + SrtDiagnosticBridge, + ObsCompatibilityAdapter +} + +public enum MimirProgramAudioPublicationKind +{ + FaustStemBus, + AsioOutputPair, + CultMeshMediaBody, + WasapiLoopbackDiagnostic, + ObsCompatibilityAdapter +} + +public enum MimirProgramControlPublicationKind +{ + EveGui, + EveTui, + CultMeshCommandSurface +} + +public sealed record MimirProgramAudioStem( + string Id, + string DisplayName, + int ChannelCount, + bool UserMixable, + bool CarriesTimingWitness, + string Notes); + +public sealed record MimirProgramControlSurface( + string Id, + string Description, + MimirProgramControlPublicationKind Kind, + bool CanPreviewProgram, + bool CanEditComposition, + bool CanSelectSources, + bool CanShowStats); + +public sealed record MimirProgramPublicationConfiguration( + string Id, + string Description, + MimirProgramVideoPublicationKind VideoKind, + string VideoSurfaceName, + MimirProgramAudioPublicationKind AudioKind, + IReadOnlyList AudioStems, + IReadOnlyList ControlSurfaces, + bool PublishesSiteFeed, + string SitePublisherRoute, + double TargetPresentationDelaySeconds, + bool DiagnosticOnly); + +public static class MimirProgramPublicationConfigurations +{ + public static IReadOnlyList ProgramStems { get; } = + [ + new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."), + new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."), + new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."), + new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."), + new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."), + new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."), + new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.") + ]; + + public static IReadOnlyList OperatorSurfaces { get; } = + [ + new( + "eve-gui-compositor", + "Primary Eve GUI composition surface: preview, source selection, transforms, crop/key controls, audio stem controls, and stream health.", + MimirProgramControlPublicationKind.EveGui, + CanPreviewProgram: true, + CanEditComposition: true, + CanSelectSources: true, + CanShowStats: true), + new( + "eve-tui-operator", + "Compact Eve TUI operator surface for source health, live stats, scene selection, and emergency command paths.", + MimirProgramControlPublicationKind.EveTui, + CanPreviewProgram: false, + CanEditComposition: false, + CanSelectSources: true, + CanShowStats: true), + new( + "cultmesh-command-surface", + "Typed CultMesh command surface consumed by Eve lowerers and trusted devices; Mimir remains the scene authority.", + MimirProgramControlPublicationKind.CultMeshCommandSurface, + CanPreviewProgram: false, + CanEditComposition: true, + CanSelectSources: true, + CanShowStats: true) + ]; + + public static MimirProgramPublicationConfiguration NativeProgram { get; } = new( + "mimir-native-program", + "Production target: Mimir owns composition, Fensalir publishes synchronized program video, Faust publishes aligned stems, and Eve exposes control/preview/stat surfaces.", + MimirProgramVideoPublicationKind.FensalirProgramTexture, + "Mimir Program Texture", + MimirProgramAudioPublicationKind.FaustStemBus, + ProgramStems, + OperatorSurfaces, + PublishesSiteFeed: false, + SitePublisherRoute: "", + TargetPresentationDelaySeconds: 5.0, + DiagnosticOnly: false); + + public static MimirProgramPublicationConfiguration YggdrasilSiteProgram { get; } = NativeProgram with + { + Id = "mimir-yggdrasil-site-program", + Description = "Site publishing target: a Yggdrasil-facing daemon consumes the same Mimir program output and publishes it to the public site without owning a second composition.", + VideoKind = MimirProgramVideoPublicationKind.CultMeshMediaBody, + AudioKind = MimirProgramAudioPublicationKind.CultMeshMediaBody, + PublishesSiteFeed = true, + SitePublisherRoute = "cultmesh://yggdrasil/gamecult.site/mimir/program" + }; + + public static MimirProgramPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with + { + Id = "native-program-d3d12-shared-texture", + VideoKind = MimirProgramVideoPublicationKind.SharedD3D12Texture, + VideoSurfaceName = "Mimir D3D12 Program Texture" + }; + + public static MimirProgramPublicationConfiguration ObsCompatibilityAdapter { get; } = new( + "obs-compatibility-adapter", + "Temporary OBS compatibility sink for preview/testing while Mimir owns composition and Yggdrasil/site publication matures.", + MimirProgramVideoPublicationKind.ObsCompatibilityAdapter, + "Mimir OBS Compatibility Feed", + MimirProgramAudioPublicationKind.ObsCompatibilityAdapter, + ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(), + OperatorSurfaces, + PublishesSiteFeed: false, + SitePublisherRoute: "", + TargetPresentationDelaySeconds: 1.5, + DiagnosticOnly: true); + + public static MimirProgramPublicationConfiguration DiagnosticSrtBridge { get; } = ObsCompatibilityAdapter with + { + Id = "diagnostic-srt-bridge", + Description = "Legacy SRT bridge for bridge tests only. It is not scene, composition, or broadcast authority.", + VideoKind = MimirProgramVideoPublicationKind.SrtDiagnosticBridge, + AudioKind = MimirProgramAudioPublicationKind.WasapiLoopbackDiagnostic, + VideoSurfaceName = "Mimir Diagnostic SRT" + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + NativeProgram, + YggdrasilSiteProgram, + TextureInteropProgram, + ObsCompatibilityAdapter, + DiagnosticSrtBridge + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs index 86760d3..7a6a8ee 100644 --- a/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs @@ -33,7 +33,7 @@ public static class MimirReservoirConfigurations new("phase_claim", "audio phase/group-delay evidence", "Mimir.Runtime", RetainsHistory: false, "Calibration/normalization input."), new("event_claim", "timeline anchor", "Mimir.Runtime", RetainsHistory: false, "Bioacoustic/passive/calibration events."), new("surface_claim", "spatial surface observation", "Fensalir", RetainsHistory: false, "Fusion candidate."), - new("render_packet", "OBS-facing packet", "Fensalir/Faust", RetainsHistory: false, "Output-ready view.") + new("render_packet", "Mimir program packet", "Mimir/Fensalir/Faust", RetainsHistory: false, "Output-ready program view.") ]; public static MimirReservoirConfiguration ManagedRuntime { get; } = new( diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 0150da4..113d935 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -119,3 +119,6 @@ {"ts": "2026-05-25T13:56:03+01:00", "type": "decision", "status": "ok", "note": "Fensalir spectrum visualization authority split: Mimir now emits AquariumBufferFieldFrame spline tube fields as the real buffer-domain contract, with tangent/curvature/normal/derivative appearance and probe policy for future compute SDF splat reservoir lowering. AquariumSplineFrame remains only the immediate Catmull-Rom preview witness until compute lowering owns rendering."} {"ts": "2026-05-25T15:55:00+01:00", "type": "decision", "status": "ok", "note": "Fensalir/Mimir spectrum visualization moved from C# spline authority to DSL-authored texture field authority. config/mimir-spectrum-field.aquafield binds a rolling spectrum texture, declares surface graph nodes, and asks for texture-dimension splinefield lowering. Fensalir now compiles Aquarium field DSL text, uploads texture samples/program packets, and ProgramMode=3 in D3D12FractalReservoirCompute emits stochastic reservoir splats from the texture field. Current canary is mechanically through the reservoir at 131072 splats with low CPU record time, but visual framing/material is still not the final beautiful spectrum surface."} {"ts": "2026-05-25T17:43:25+01:00", "type": "decision", "status": "ok", "note": "Fensalir direct splinefield correction: texture-derived tube trails are direct SDF envelope surfaces, not point-splat fractal dust. AquariumFieldScriptCompiler supports direct; D3D12Renderer now derives AquariumSplineFrame tube-envelope trails from BufferFieldFrame texture spline programs, the spline pipeline declares triangle topology, and D3D12SplinePS writes color, metadata/control, and reservoirGuide outputs. Mimir capture artifacts/runtime/mimir-direct-sdf-tube-spectrum.png proves the DSL-authored ASIO spectrum appears through the direct SDF tube path with ~0.64 ms record CPU in the canary."} +{"ts": "2026-06-08T09:29:43+01:00", "type": "decision", "status": "ok", "note": "Added CultMesh media bridge for Raven program feed: Raven sender publishes rolling mimir.cultmesh_media_frame MPEG-TS chunks over CultNet reliable UDP to Yggdrasil 10.77.0.1:3075; Starfire receiver lowers raven-primary-av to local UDP 127.0.0.1:5200 for OBS. Yggdrasil relay and Starfire receiver were started; Raven SSH timed out, so sender must be launched on Raven directly."} +{"ts": "2026-06-08T09:59:22+01:00", "type": "verification", "status": "blocked", "note": "CultMesh media bridge verification: Yggdrasil UFW was opened for 3075/udp on wg-gamecult and Starfire receiver now connects from 10.77.0.2. Synthetic sender on Yggdrasil connects and sends chunks without User Not Verified after CultLib pre-verification schema-v0 policy patch, but relay cache does not receive mimir.cultmesh_media_frame updates and Starfire receiver logs no frames. Remaining blocker is CultNet document put/apply path, not WireGuard routing or OBS endpoint."} +{"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."} diff --git a/state/map.yaml b/state/map.yaml index 0ae93ef..e42d7ec 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -5,7 +5,7 @@ workspace: implementation_name: Mimir public surface / native runtime contracts premise: > Mimir is the local realtime field machine: cameras, microphones, speakers, - network feeds, and timing signals become one coherent OBS-facing volumetric + network feeds, and timing signals become one coherent Mimir-owned volumetric program surface. The live path is a C# Fensalir-hosted app with in-memory rolling buffers and native capture/runtime boundaries. @@ -13,7 +13,7 @@ objective: primary: > Build one synchronized spatial stream machine from six cameras, six microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP, - and OBS output. + Eve operator surfaces, and Yggdrasil site publication. current: > Promote the active bioacoustic packet-song receiver into runtime state: self-identifying song contours should pin multiple time/frequency anchors @@ -22,9 +22,9 @@ objective: constraints: - Sender is Windows. - - Receiver uses OBS. + - OBS is a temporary compatibility sink, not a composition or broadcast authority. - Video encode must use NVENC when the bridge path is active. - - Audio sources must remain separately controllable in OBS. + - Audio sources must remain separately controllable through Mimir/Eve and publishable as stems. - V1 bridge is LAN-only. - Persistent state must clarify rehydrate, not become transcript tax. - Live ingest belongs to Mimir.Runtime, native drivers, Fensalir, and @@ -33,22 +33,31 @@ constraints: memory. - Networked feeds are producers into the live runtime, not independent clock authorities. + - CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may + consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but + Mimir owns program selection, composition, preview/control, and publication. - Mimir explicitly trades latency and memory for throughput and field quality: the default five-second buffer is compute budget for stream alignment, GPU/audio analysis, and volumetric sound-field extraction. invariants: - - OBS composition authority stays in OBS. + - Mimir owns composition, preview/control state, source subscription policy, + and program publication intent. - FFmpeg owns capture, encoding, and transport only for bridge utilities. - One stable bridge port equals one OBS source endpoint. - One rolling live window owns the timing edge and expiry. - The rolling window is not accidental latency. It must preserve enough time - to line up feeds and crunch data before OBS-facing output is emitted. + to line up feeds and crunch data before Mimir program output is emitted. - Typed views may index the rolling buffer, but no per-kind ring owns retention. - - Fensalir owns production UI, GPU fusion, D3D12 interop, and Spout2 output. + - Fensalir owns GPU lowering for the Mimir program scene graph, D3D12 interop, + and local program texture output. - Faust/native DSP owns hot audio alignment, separation, spatialization, and stems. + - Eve GUI/TUI surfaces lower Mimir's operator controls, previews, and stats on + any device without owning scene truth. + - A Yggdrasil-facing publisher daemon consumes Mimir program output and + publishes it to the site without owning composition. - Local secrets remain untracked. - Mimir is a CultMesh app. Codebooks, schedules, calibration receipts, path response, and compute-offload work need typed mesh-syncable state surfaces @@ -271,7 +280,7 @@ accepted_design: library distilled from the research notes: node profiles, decoder/language profiles, path learning, benchmark panels, actuator strategies, capture and ingest strategies, reservoir strategies, field profiles, compute offload, - OBS publication, CultMesh contracts, Fensalir lowering, and exportable + program publication, CultMesh contracts, Fensalir lowering, and exportable manifest generation. - `docs/perfect-machine-module-library.md` indexes those reusable runtime modules and their smoke commands. @@ -279,6 +288,18 @@ accepted_design: for the current repo code surface. - `docs/perfect-machine-domain-index.md` indexes Mimir's full problem domain, current evidence taxonomy, hot paths, and implementation cuts. + - `docs/mimir-program-composition.md` owns the current program-composition + authority map: Muninn observes, Mimir composes/controls, Eve lowers operator + surfaces, Yggdrasil publishes, and OBS is compatibility-only. + - `src/Mimir.Runtime/Synchronization` now includes typed program contracts: + `mimir.program_scene.v1`, `mimir.program_output.v1`, and + `mimir.eve_operator_surface.v1`. + - `src/Mimir.CultMeshMedia` owns the first explicit CultMesh media/body + bridge. Raven publishes muxed H.264/AAC MPEG-TS chunks as rolling + `mimir.cultmesh_media_frame` documents over CultNet reliable UDP, + Yggdrasil relays them on `10.77.0.1:3075`, and Starfire lowers the + subscribed stream to `udp://127.0.0.1:5200` for the OBS source + `Raven Monitor + Realtek`. - `research/perfect-machine-study-2026-05-23/` preserves the architecture rumination, optimization ledger, research references, source/hotspot audits, Fensalir/native boundary maps, benchmark plan, calibration-session spec, @@ -305,6 +326,8 @@ active_subgoals: settings, and output management. - Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output replaces them. + - Launch Raven's CultMesh sender directly on Raven because SSH to Raven over + WireGuard timed out during the initial relay deployment pass. current_status: summary: > From 2c68485c35404e8b5ad943b36ca2677a6f41b320 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 9 Jun 2026 03:03:46 +0100 Subject: [PATCH 004/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 61724 -> 61724 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index c149835bfa5b53e2d38be4a73a2283db35bb11f5..1827be50308103c49d3690ec5882069b19c3f65b 100644 GIT binary patch delta 224 zcmbR9hAwF#bWN`~a zQ?R(8{be0waZ3ZR_+*ZpG;R}+!A2H(mWHOAD|46``H;;uGn@Q?R|G|?5!9;O^Rg)7 b#wH-ECU5PoK@m2ExFED=M+%C#rC}5R>2EvE delta 213 zcmbR9h7<$#JBPyU!KzWETJwgIwQ zOCyll&4TuqbwI-G5XGjG_~a(@ pVU`wPi#H$2JueFvw$L*$Fa*12Yj+K@s3AmgXwQxmWML!AC;;9xKJ5Si From 28536681f6db8f799ab9e79a060fe7bdfa5e56df Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 9 Jun 2026 03:04:51 +0100 Subject: [PATCH 005/135] Import OBS scene into Mimir program state --- docs/mimir-program-composition.md | 8 + notes/current-system-map.md | 2 + notes/fresh-workspace-handoff.md | 5 + src/Mimir.BufferSmoke/Program.cs | 75 +++++ .../Synchronization/MimirObsSceneImporter.cs | 288 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 5 + state/mimir-program-composition.cc | Bin 0 -> 6998 bytes 8 files changed, 384 insertions(+) create mode 100644 src/Mimir.Runtime/Synchronization/MimirObsSceneImporter.cs create mode 100644 state/mimir-program-composition.cc diff --git a/docs/mimir-program-composition.md b/docs/mimir-program-composition.md index 2febc26..03bffe4 100644 --- a/docs/mimir-program-composition.md +++ b/docs/mimir-program-composition.md @@ -109,3 +109,11 @@ The proof path is: dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke ``` + +Import the current OBS scene as Mimir-owned typed state: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --import-obs-program-scene --output state\mimir-program-composition.cc +``` + +The live imported state file is `state/mimir-program-composition.cc`. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 0317e12..52b0621 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -107,6 +107,8 @@ The first typed program contracts live in `src/Mimir.Runtime/Synchronization`: OBS scene JSON is import/mirror evidence for the initial crop/key/transform layout. It is not the editor of record after Mimir owns the scene graph. +The current imported local scene state lives at +`state/mimir-program-composition.cc`. ## OBS Bridge Utility diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index aa1eda9..71c4569 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -49,6 +49,11 @@ Get-Content .\state\evidence.jsonl -Tail 8 `mimir.program_scene.v1`, `mimir.program_output.v1`, and `mimir.eve_operator_surface.v1`; `Mimir.BufferSmoke --perfect-machine-contract-smoke` writes them into CultCache. +- `Mimir.BufferSmoke --import-obs-program-scene --output + state/mimir-program-composition.cc` imports the current OBS scene JSON into + Mimir-owned typed state. The latest imported scene has eight layers, six + visible, two cropped, one chroma-keyed, and Raven mapped as + `muninn:raven:monitor:primary`. - Local six-camera ingest should use direct driver adapters. Process-backed sources are bridge/network edges only. - Leap stereo IR is the first timing-camera candidate for direct ingest. diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index e9e9ae4..5aa9d1e 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -91,6 +91,15 @@ if (args.Any(arg => string.Equals(arg, "--perfect-machine-contract-smoke", Strin .ConfigureAwait(false); } +if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) +{ + return await ImportObsProgramSceneAsync( + ParseStringOption(args, "--input", DefaultObsScenePath()), + ParseStringOption(args, "--scene", ""), + ParseStringOption(args, "--output", "state/mimir-program-composition.cc")) + .ConfigureAwait(false); +} + if (args.Any(arg => string.Equals(arg, "--perfect-machine-manifest", StringComparison.OrdinalIgnoreCase))) { return await WritePerfectMachineManifestAsync( @@ -1595,6 +1604,72 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) return 0; } +static async Task ImportObsProgramSceneAsync(string inputPath, string sceneName, string outputPath) +{ + if (!File.Exists(inputPath)) + { + Console.Error.WriteLine($"OBS scene file not found: {inputPath}"); + return 1; + } + + var importedAt = DateTimeOffset.UtcNow; + var scene = MimirObsSceneImporter.ImportFile( + inputPath, + string.IsNullOrWhiteSpace(sceneName) ? null : sceneName, + importedAt); + var output = MimirCultMeshContractFactory.CreateProgramOutput( + "mimir-imported-program", + scene.SceneId, + MimirProgramPublicationConfigurations.YggdrasilSiteProgram, + importedAt); + var eveSurface = MimirCultMeshContractFactory.CreateOperatorSurface( + "mimir-eve-gui-compositor", + scene.SceneId, + MimirProgramPublicationConfigurations.OperatorSurfaces[0], + importedAt); + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions + { + PullOnOpen = File.Exists(outputPath) + }).ConfigureAwait(false); + await cache.UpsertAsync( + scene, + new CultRecordHandle(new CultRecordKey($"mimir-program-scene:{scene.SceneId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + output, + new CultRecordHandle(new CultRecordKey($"mimir-program-output:{output.OutputId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + eveSurface, + new CultRecordHandle(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}"))) + .ConfigureAwait(false); + await cache.FlushAsync().ConfigureAwait(false); + + var visible = scene.Layers.Count(layer => layer.Visible); + var keyed = scene.Layers.Count(layer => layer.ChromaKey is not null); + var cropped = scene.Layers.Count(layer => + layer.Crop.Left != 0.0 || layer.Crop.Top != 0.0 || layer.Crop.Right != 0.0 || layer.Crop.Bottom != 0.0); + Console.WriteLine( + $"obs-program-scene-import input={inputPath} output={outputPath} scene={scene.SceneId} canvas={scene.CanvasWidth}x{scene.CanvasHeight} layers={scene.Layers.Length} visible={visible} cropped={cropped} keyed={keyed} route={output.PublisherRoute}"); + foreach (var layer in scene.Layers.OrderBy(layer => layer.ZIndex)) + { + Console.WriteLine( + $"layer {layer.ZIndex}:{layer.LayerId} visible={layer.Visible} source={layer.SourceRef} kind={layer.SourceKind} pos={layer.X:0.###},{layer.Y:0.###} size={layer.Width:0.###}x{layer.Height:0.###} crop={layer.Crop.Left:0.###},{layer.Crop.Top:0.###},{layer.Crop.Right:0.###},{layer.Crop.Bottom:0.###} key={(layer.ChromaKey is null ? "none" : layer.ChromaKey.KeyColorRgba.ToString("x8"))}"); + } + + return 0; +} + +static string DefaultObsScenePath() => + Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), + "obs-studio", + "basic", + "scenes", + "Untitled.json"); + static async Task WritePerfectMachineManifestAsync(string outputPath) { var manifest = MimirPerfectMachineManifestFactory.Create(); diff --git a/src/Mimir.Runtime/Synchronization/MimirObsSceneImporter.cs b/src/Mimir.Runtime/Synchronization/MimirObsSceneImporter.cs new file mode 100644 index 0000000..d2638cd --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirObsSceneImporter.cs @@ -0,0 +1,288 @@ +using System.Text.Json; + +namespace Mimir.Runtime.Synchronization; + +public static class MimirObsSceneImporter +{ + public static MimirProgramSceneDocument ImportFile( + string path, + string? sceneName = null, + DateTimeOffset? updatedAt = null) + { + using var stream = File.OpenRead(path); + using var document = JsonDocument.Parse(stream); + return Import(document.RootElement, sceneName, updatedAt); + } + + public static MimirProgramSceneDocument Import( + JsonElement root, + string? sceneName = null, + DateTimeOffset? updatedAt = null) + { + var sources = root.GetProperty("sources").EnumerateArray().ToArray(); + var sourceByName = sources + .Where(source => source.TryGetProperty("name", out _)) + .ToDictionary(source => source.GetProperty("name").GetString() ?? "", StringComparer.Ordinal); + + var selectedSceneName = !string.IsNullOrWhiteSpace(sceneName) + ? sceneName + : root.TryGetProperty("current_program_scene", out var currentProgram) + ? currentProgram.GetString() + : root.TryGetProperty("current_scene", out var currentScene) + ? currentScene.GetString() + : null; + if (string.IsNullOrWhiteSpace(selectedSceneName)) + { + selectedSceneName = "Scene"; + } + + if (!sourceByName.TryGetValue(selectedSceneName, out var sceneSource)) + { + throw new InvalidOperationException($"OBS scene '{selectedSceneName}' was not found."); + } + + var settings = sceneSource.GetProperty("settings"); + var itemElements = settings.GetProperty("items").EnumerateArray().ToArray(); + var layers = new List(itemElements.Length); + for (var index = 0; index < itemElements.Length; index++) + { + var item = itemElements[index]; + var name = item.GetProperty("name").GetString() ?? $"layer-{index}"; + sourceByName.TryGetValue(name, out var source); + var sourceKind = SourceKind(source); + var crop = CombineCrop(ItemCrop(item), FilterCrop(source)); + var chromaKey = FilterChromaKey(source); + var (x, y, width, height) = Bounds(item); + layers.Add(new MimirProgramSceneLayer( + LayerId: LayerId(name, index), + SourceRef: SourceRef(name, sourceKind), + SourceKind: sourceKind, + Visible: ReadBool(item, "visible", fallback: true), + X: x, + Y: y, + Width: width, + Height: height, + ZIndex: index, + Crop: crop, + ChromaKey: chromaKey)); + } + + var resolution = root.TryGetProperty("resolution", out var resolutionElement) + ? resolutionElement + : default; + return new MimirProgramSceneDocument( + SceneId: SceneId(selectedSceneName), + UpdatedAtUtc: (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"), + CanvasWidth: ReadInt(resolution, "x", 1920), + CanvasHeight: ReadInt(resolution, "y", 1080), + Owner: "Mimir", + Layers: layers.ToArray()); + } + + private static string SourceKind(JsonElement source) + { + if (source.ValueKind == JsonValueKind.Undefined) + { + return "unknown"; + } + + return (source.GetProperty("id").GetString() ?? "") switch + { + "monitor_capture" => "monitor", + "window_capture" => "window", + "dshow_input" => "camera", + "browser_source" => "browser", + "ffmpeg_source" => "network-monitor", + "mimir_program_texture_source" => "mimir-program-texture", + "muninn_stream_source" => "muninn-stream", + _ => "unknown" + }; + } + + private static string SourceRef(string name, string sourceKind) + { + var normalized = LayerId(name, 0); + if (name.Contains("raven", StringComparison.OrdinalIgnoreCase)) + { + return sourceKind.Contains("audio", StringComparison.OrdinalIgnoreCase) + ? "muninn:raven:audio:realtek-loopback" + : "muninn:raven:monitor:primary"; + } + + if (sourceKind == "monitor") + { + return name.Contains("bigscreen", StringComparison.OrdinalIgnoreCase) + ? "muninn:starfire:monitor:bigscreen" + : "muninn:starfire:monitor:primary"; + } + + if (sourceKind == "window") + { + return $"muninn:starfire:window:{normalized}"; + } + + if (sourceKind == "camera") + { + return $"muninn:starfire:camera:{normalized}"; + } + + if (sourceKind == "mimir-program-texture") + { + return "mimir:generated:program-texture"; + } + + if (sourceKind == "browser") + { + return $"mimir:browser:{normalized}"; + } + + if (sourceKind == "muninn-stream") + { + return $"muninn:starfire:stream:{normalized}"; + } + + return $"mimir:unknown:{normalized}"; + } + + private static (double X, double Y, double Width, double Height) Bounds(JsonElement item) + { + var pos = item.TryGetProperty("pos", out var posElement) ? posElement : default; + var scale = item.TryGetProperty("scale", out var scaleElement) ? scaleElement : default; + var scaleRef = item.TryGetProperty("scale_ref", out var scaleRefElement) ? scaleRefElement : default; + var bounds = item.TryGetProperty("bounds", out var boundsElement) ? boundsElement : default; + var width = ReadDouble(bounds, "x", 0.0); + var height = ReadDouble(bounds, "y", 0.0); + var scaledWidth = ReadDouble(scaleRef, "x", 0.0) * Math.Abs(ReadDouble(scale, "x", 1.0)); + var scaledHeight = ReadDouble(scaleRef, "y", 0.0) * Math.Abs(ReadDouble(scale, "y", 1.0)); + if (scaledWidth > width) + { + width = scaledWidth; + } + + if (scaledHeight > height) + { + height = scaledHeight; + } + + return ( + ReadDouble(pos, "x", 0.0), + ReadDouble(pos, "y", 0.0), + width, + height); + } + + private static MimirProgramCrop ItemCrop(JsonElement item) => + new( + ReadDouble(item, "crop_left", 0.0), + ReadDouble(item, "crop_top", 0.0), + ReadDouble(item, "crop_right", 0.0), + ReadDouble(item, "crop_bottom", 0.0)); + + private static MimirProgramCrop FilterCrop(JsonElement source) + { + if (source.ValueKind == JsonValueKind.Undefined || !source.TryGetProperty("filters", out var filters)) + { + return new MimirProgramCrop(0, 0, 0, 0); + } + + foreach (var filter in filters.EnumerateArray()) + { + if ((filter.GetProperty("id").GetString() ?? "") != "crop_filter") + { + continue; + } + + var settings = filter.GetProperty("settings"); + return new MimirProgramCrop( + ReadDouble(settings, "left", 0.0), + ReadDouble(settings, "top", 0.0), + ReadDouble(settings, "right", 0.0), + ReadDouble(settings, "bottom", 0.0)); + } + + return new MimirProgramCrop(0, 0, 0, 0); + } + + private static MimirProgramChromaKey? FilterChromaKey(JsonElement source) + { + if (source.ValueKind == JsonValueKind.Undefined || !source.TryGetProperty("filters", out var filters)) + { + return null; + } + + MimirProgramChromaKey? strongest = null; + foreach (var filter in filters.EnumerateArray()) + { + if ((filter.GetProperty("id").GetString() ?? "") != "chroma_key_filter") + { + continue; + } + + var settings = filter.GetProperty("settings"); + strongest = new MimirProgramChromaKey( + ReadUInt(settings, "key_color", 0), + ReadDouble(settings, "similarity", 0.0), + ReadDouble(settings, "smoothness", 0.0), + ReadDouble(settings, "spill", 0.0)); + } + + return strongest; + } + + private static MimirProgramCrop CombineCrop(MimirProgramCrop item, MimirProgramCrop filter) => + new( + item.Left + filter.Left, + item.Top + filter.Top, + item.Right + filter.Right, + item.Bottom + filter.Bottom); + + private static string SceneId(string name) => + $"mimir-scene-{LayerId(name, 0)}"; + + private static string LayerId(string name, int fallback) + { + var chars = name + .Trim() + .ToLowerInvariant() + .Select(character => char.IsLetterOrDigit(character) ? character : '-') + .ToArray(); + var id = string.Join("-", new string(chars).Split('-', StringSplitOptions.RemoveEmptyEntries)); + return string.IsNullOrWhiteSpace(id) ? $"layer-{fallback}" : id; + } + + private static bool ReadBool(JsonElement element, string property, bool fallback) => + element.ValueKind != JsonValueKind.Undefined && + element.TryGetProperty(property, out var value) && + value.ValueKind is JsonValueKind.True or JsonValueKind.False + ? value.GetBoolean() + : fallback; + + private static int ReadInt(JsonElement element, string property, int fallback) => + element.ValueKind != JsonValueKind.Undefined && + element.TryGetProperty(property, out var value) && + value.TryGetInt32(out var parsed) + ? parsed + : fallback; + + private static double ReadDouble(JsonElement element, string property, double fallback) => + element.ValueKind != JsonValueKind.Undefined && + element.TryGetProperty(property, out var value) && + value.TryGetDouble(out var parsed) + ? parsed + : fallback; + + private static uint ReadUInt(JsonElement element, string property, uint fallback) + { + if (element.ValueKind == JsonValueKind.Undefined || + !element.TryGetProperty(property, out var value)) + { + return fallback; + } + + return value.TryGetUInt32(out var parsed) + ? parsed + : value.TryGetInt64(out var signed) + ? unchecked((uint)signed) + : fallback; + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 113d935..5d1df26 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -122,3 +122,4 @@ {"ts": "2026-06-08T09:29:43+01:00", "type": "decision", "status": "ok", "note": "Added CultMesh media bridge for Raven program feed: Raven sender publishes rolling mimir.cultmesh_media_frame MPEG-TS chunks over CultNet reliable UDP to Yggdrasil 10.77.0.1:3075; Starfire receiver lowers raven-primary-av to local UDP 127.0.0.1:5200 for OBS. Yggdrasil relay and Starfire receiver were started; Raven SSH timed out, so sender must be launched on Raven directly."} {"ts": "2026-06-08T09:59:22+01:00", "type": "verification", "status": "blocked", "note": "CultMesh media bridge verification: Yggdrasil UFW was opened for 3075/udp on wg-gamecult and Starfire receiver now connects from 10.77.0.2. Synthetic sender on Yggdrasil connects and sends chunks without User Not Verified after CultLib pre-verification schema-v0 policy patch, but relay cache does not receive mimir.cultmesh_media_frame updates and Starfire receiver logs no frames. Remaining blocker is CultNet document put/apply path, not WireGuard routing or OBS endpoint."} {"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."} +{"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."} diff --git a/state/map.yaml b/state/map.yaml index e42d7ec..8f70fd0 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -294,6 +294,11 @@ accepted_design: - `src/Mimir.Runtime/Synchronization` now includes typed program contracts: `mimir.program_scene.v1`, `mimir.program_output.v1`, and `mimir.eve_operator_surface.v1`. + - `state/mimir-program-composition.cc` is the current imported Mimir program + state, generated from OBS scene JSON by `Mimir.BufferSmoke + --import-obs-program-scene`. It preserves the current Starfire monitor, + Discord crop/chroma key, Raven monitor, camera, browser, and Mimir texture + layers as Mimir-owned typed state. - `src/Mimir.CultMeshMedia` owns the first explicit CultMesh media/body bridge. Raven publishes muxed H.264/AAC MPEG-TS chunks as rolling `mimir.cultmesh_media_frame` documents over CultNet reliable UDP, diff --git a/state/mimir-program-composition.cc b/state/mimir-program-composition.cc new file mode 100644 index 0000000000000000000000000000000000000000..fd90a84b77a31b96b86f6497015cc72615f7e7b5 GIT binary patch literal 6998 zcmeHL-HRN@5now%5;$*p4%ois!E)rAHX$g`xNRVVnx<(P%;v~3Y@)k{OEtuTccL%~MTIZ(^GTBn0v67vEfEmP z-`iRQ7zPLe*CK>i7SWi7Y>V`!8oH|yteZLyTpQcA<-<3%9SdVZw1B%d@;QtJ0kt2@ z&is5Xh2=%$M#QN0nM8!nh8@VS0)OT=1BJG$4(X!z)SL9{k9NsPn8gfM1%&2YLv%(pS?2y#u$KE zAn~H$%OPJ`6iHo>Xv#;&lBB^&yf?cxfy*4l^?ep*3rW;W(lF~qigtSLZJNc92kao6^_T zN^cA@+pV#Lwz;`+e`D>y&CygdrLnoO_Sp6O=obkO-kmy%w{t6bpo1BB$;?}zH*quy z6=&ZB#j#K=R6BIDr`8XhLY2d};1~|K^3K@e&z_UGhTQ2;N)a+h;A+U?I?)|nV?Ne& zL)Q(*H?T`}+jp3yn}O}yE_PgoG{Z49j*#zbe^o@)RO#q^(#o2x>~BSbqkM_6s~fIM zIMyxVx-K$phIPa+b?g8K80&TraE%2nwz=z4j2PvXj(|QfkgV}wX0ebj$6-LQDo&kb z%5k+X=g5RzVQ z!3}H3h>I=15vEIAoPx8;AEk$ebojlH)Z--0LV7Z8w5Krb!_{Gwdv===udbDD^fSg3 zcYF=hb8|GrrwsH+=SDx|ilno*$3vh<=g{r(Eh!?JA%`#r3M`=RVT~cj3@pdjh-FgO2ppdUjAP5P{xVJ%Y467UcEQ*vgXlq^0YCI?xWFcX zW^!M%kZ*I}G7ZA)q0FoLq^y zm_o9b!875TA$$Cgw0n(&L2-9Nc1}*U;w+4KrPhw=l1Sq4XQWpHR-`nqROK0v)Az56 z*ll9EvR2x%2-sO7|MN(2-^_2~GNx5|aIklshu~MiR-iERy4?Jt96ftYZARDnYMUSt zTwd*dS7zEWbic}Z)2lihskg#1&8zxEd~M}f_bCMBMp=VD_q3*0F|Hz1!J6rqhW)&z zRWoB^J@I+z zviaO8o2%Vt#(gNaJa^$pbPT+=+oVzKJ<~Ojt)o1)XMS~MIz9^~P6qWyhcA6JAoYXsQ9 zr1!f%u-877PX~4g><$alCY$%tNcf9pO($a9b-qmk@ zz6xU<@72$r`>9tdJ+>_f@3@>_+E{7 zVLRz1vZ<}1aJB`h_T|^q@51nS_0Gfp!katZFFtwr89dDH&@Z=?KgwwaM=0t(J?S1b zULDgKpR?K7_{k-?sE+r7{0}H5kI|tW!udnmd{dnnUnDCj7uB?RJ%21*HyC;xjLP%+ M4X;!>yfZ^z0O2uUssI20 literal 0 HcmV?d00001 From 6dc18ecf18fd6b0baee6be7df130fb7a05a8e609 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 16:43:12 +0100 Subject: [PATCH 006/135] Add Move tracking observation streams --- docs/implementation-plan.md | 15 +- docs/perfect-machine-module-library.md | 8 +- notes/current-system-map.md | 31 ++++ src/Mimir.BufferSmoke/Program.cs | 153 +++++++++++++++++- src/Mimir.Runtime/MimirRuntime.cs | 5 + .../Synchronization/MimirModuleLibrary.cs | 4 +- .../MimirNativeCaptureConfiguration.cs | 47 +++++- .../MimirNativeIngestStreamSource.cs | 25 ++- .../MimirPerfectMachineProfiles.cs | 1 + .../Synchronization/MimirStreamKind.cs | 1 + .../Synchronization/MimirStreamSample.cs | 3 +- .../MimirSynchronizationHub.cs | 12 +- .../MimirSynchronizationSettings.cs | 2 + .../MimirTrackingObservation.cs | 100 ++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 11 ++ 16 files changed, 406 insertions(+), 13 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index a44944a..2b3abf3 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -10,7 +10,7 @@ For the source-level ownership map, read [[code-algorithm-map|Code Algorithm Map - `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12 bridge. - `src/Mimir.Runtime` owns stream descriptors, source polling, direct push - ingest, one rolling buffer per configured audio/video stream, and the + ingest, one rolling buffer per configured audio/video/tracking stream, and the synchronization hub Fensalir can inspect. - The default five-second window is an intentional latency/memory trade: use it to line up streams and extract the volumetric audio/video field before Mimir @@ -31,8 +31,14 @@ a named invariant that the native runtime cannot protect yet. - `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and `IMimirStreamSource`. - Configurable five-second default rolling buffers for local and network audio - and video streams. + and video/tracking streams. - `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers. +- `MimirStreamKind.Tracking` plus `MimirTrackingObservation` carry PS Move + pose/button/battery/confidence samples as typed + `mimir.move_tracking_observation.v1` documents. Starfire and Nightwing both + have USB-attached Moves; Starfire can feed local tracking streams, while + Muninn on Nightwing publishes remote tracking streams. Odin owns discovery and + schema projection, not pose capture or Mimir fusion. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe @@ -192,7 +198,10 @@ a named invariant that the native runtime cannot protect yet. tooling/UI/remote witness use, and `--perfect-machine-lowering-benchmark` to measure the Mimir-to-Fensalir lowering path. The current six-camera/two-audio synthetic lowering benchmark runs at roughly 2.5 us per iteration with about - 2.3 KB allocated per iteration. + 2.3 KB allocated per iteration. `--move-tracking-contract-smoke` proves + Starfire-local and Nightwing-remote Move tracking observations enter + Mimir's rolling buffers with Muninn producer identity and Odin discovery + provenance. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device timestamp, and native/GPU handle metadata. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index c5fa824..1c6a313 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -13,11 +13,11 @@ job-interview answer for why it belongs in the body of the machine. | Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` | | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | -| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, and actuator state documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs` | +| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, and Move tracking observation documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | -| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, and Raven Scarlett. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` | +| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, Raven Scarlett, and USB-attached Moves on Starfire/Nightwing. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` | | Camera ingest strategies | `Mimir.Runtime` + native workers | Diagnostic metadata, managed native wrapper, native SPSC ring, and shared-GPU-texture ingest shapes. | `src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs` | | Reservoir strategies | `Mimir.Runtime` + native reservoir + Fensalir | Managed rolling buffers, native shared-edge rings, and derived GPU temporal field storage. | `src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs` | | Distributed witnesses | `Mimir.Runtime` + CultMesh | Raven, phone, microcontroller, and remote camera rig configurations for compact timing/path state. | `src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs` | @@ -42,6 +42,9 @@ job-interview answer for why it belongs in the body of the machine. stem generation. - CultMesh carries typed state surfaces; remote nodes may observe and decode, but they do not become independent clock authorities. +- USB-attached Moves on Starfire and Nightwing publish + `mimir.move_tracking_observation.v1` as tracking streams. Muninn emits the + witness, Odin discovers/projects it, and Mimir owns buffering plus fusion. - OBS may receive compatibility program surfaces and stems. It does not own calibration, composition, preview/control, or publication. @@ -54,6 +57,7 @@ dotnet build .\Mimir.slnx dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-lowering-benchmark dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test ``` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 52b0621..936b4e4 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -22,6 +22,7 @@ flowchart TD B["mic/loopback drivers"] --> R C["Leap timing/IR driver"] --> R D["network feed producers"] --> R + M["Starfire/Nightwing Move tracking"] --> R R --> N["native reservoir handles"] N --> E["Fensalir GPU fusion + UI"] N --> F["Faust/native DSP"] @@ -44,6 +45,11 @@ Ownership: becoming independent clock authorities. - `Mimir.Runtime` owns app-level stream buffers and synchronization. - Native capture workers own device reads. +- Move tracking observations are stream samples, not program controls. USB + Moves plugged into Starfire produce local tracking streams; USB Moves plugged + into Nightwing are published by Muninn on Nightwing as remote tracking + streams. Odin owns discovery/schema projection for those streams and does not + own pose capture or Mimir fusion. - Mimir owns program composition, source subscription policy, preview/control state, stats, and publication intent. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, @@ -77,6 +83,7 @@ program video plus separately controllable audio stems. - `MimirProcessStreamSource`; - `MimirFrameEventProcessStreamSource`; - `MimirVideoFrameDescriptor`; +- `MimirTrackingObservation`; - `MimirAudioSynchronizationAnalyzer`; - `IMimirVideoCaptureDriver`; - `MimirVideoCaptureDriverSource`. @@ -89,6 +96,30 @@ the rolling buffers while the direct ABI driver is being cut. It does not carry pixels and does not own the final six-camera hot path. Multi-camera probes are one process with declared accepted source ids, not one process per camera. +## Move Tracking + +Starfire and Nightwing both have PS Move controllers directly attached over +USB. Mimir represents those controller poses as `MimirStreamKind.Tracking` +buffers carrying typed `mimir.move_tracking_observation.v1` samples. Each +sample names the stream, device, producer, host, Odin discovery provider, +tracking space, calibration id, source timestamp, pose, velocity, buttons, +battery, latency, and confidence. + +Authority: + +- Starfire-local Move tracking is a local Mimir/Muninn tracking stream. +- Nightwing Move tracking is emitted by Muninn on Nightwing over the typed + observation path. +- Odin discovers and projects the stream/schema surface for operator and agent + access. +- Mimir owns subscription, rolling retention, clock alignment, and later fusion + with cameras/audio/Leap/Fensalir. OBS and dashboard summaries do not own this + state. + +Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj +-- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing +Move observation into two tracking buffers. + ## Program Composition [[docs/mimir-program-composition|Mimir Program Composition]] is the live diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 5aa9d1e..eb631d4 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -91,6 +91,11 @@ if (args.Any(arg => string.Equals(arg, "--perfect-machine-contract-smoke", Strin .ConfigureAwait(false); } +if (args.Any(arg => string.Equals(arg, "--move-tracking-contract-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveTrackingContractSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1353,6 +1358,8 @@ static int RunPerfectMachineProfileSmoke() ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state"], StringComparer.Ordinal), new HashSet(StringComparer.Ordinal))); + var trackingSmoke = BuildMoveTrackingSmokeHub(); + var trackingObservations = trackingSmoke.Hub.TrackingObservations; Console.WriteLine( $"perfect-machine-profile-smoke profiles={profiles.Count} calibrationPlans={calibrationPlans.Count} audioFields={audioFields.Count} visualFields={visualFields.Count} computePlans={computePlans.Count} assemblyPlans={assemblyPlans.Count} captureProfiles={captureProfiles.Count} publications={publications.Count} languageProfiles={languageProfiles.Count} pathLearning={pathLearningProfiles.Count} localization={localizationProfiles.Count} benchmarkPanels={benchmarkPanels.Count} actuatorStrategies={actuatorStrategies.Count} cameraIngest={cameraIngestStrategies.Count} reservoirs={reservoirStrategies.Count} distributedWitnesses={distributedWitnesses.Count} networkTransports={networkTransports.Count} authorityPolicies={authorityPolicies.Count} modules={moduleCatalog.Count}"); @@ -1372,6 +1379,8 @@ static int RunPerfectMachineProfileSmoke() $"perfect-machine-authority appliesTo=raven-scarlett-witness decision={authority.Decision} rule={authority.RuleId}"); Console.WriteLine( $"perfect-machine-transport payload=TypedTimingState status={transport.Status} selected={transport.Transport?.Id ?? "none"}"); + Console.WriteLine( + $"perfect-machine-move-tracking consumed={trackingSmoke.Consumed} observations={trackingObservations.Count} hosts={string.Join(",", trackingObservations.Select(observation => observation.HostId).Order(StringComparer.Ordinal))}"); return profiles.Count >= 6 && calibrationPlans.Count >= 3 && @@ -1400,11 +1409,106 @@ static int RunPerfectMachineProfileSmoke() localization is { Score: > 0.90 } && authority.Decision == MimirAuthorityDecision.TrustedEvidence && transport.Transport?.Id == MimirNetworkTransportConfigurations.CultMeshTimingState.Id && + trackingSmoke.Consumed == 2 && + trackingObservations.Count == 2 && + trackingObservations.Any(observation => observation.HostId == "starfire" && observation.ProducerId == "muninn:starfire:move") && + trackingObservations.Any(observation => observation.HostId == "nightwing" && observation.ProducerId == "muninn:nightwing:move") && clock is { AnchorCount: >= 4, Confidence: > 0.70 } ? 0 : 1; } +static int RunMoveTrackingContractSmoke() +{ + using var trackingSmoke = BuildMoveTrackingSmokeHub(); + var observations = trackingSmoke.Hub.TrackingObservations; + foreach (var observation in observations) + { + Console.WriteLine( + $"move-tracking-observation stream={observation.StreamId} device={observation.DeviceId} host={observation.HostId} producer={observation.ProducerId} discovery={observation.DiscoveryProviderId} confidence={observation.Confidence:0.000} pos=({observation.PositionMeters.X:0.000},{observation.PositionMeters.Y:0.000},{observation.PositionMeters.Z:0.000})"); + } + + Console.WriteLine( + $"move-tracking-contract-smoke consumed={trackingSmoke.Consumed} observations={observations.Count} summary=\"{trackingSmoke.Hub.Summary()}\""); + return trackingSmoke.Consumed == 2 && + observations.Count == 2 && + observations.All(observation => observation.Kind == MimirTrackingObservationKind.PsMoveController) && + observations.Any(observation => observation.HostId == "starfire" && observation.DiscoveryProviderId == "odin") && + observations.Any(observation => observation.HostId == "nightwing" && observation.DiscoveryProviderId == "odin") + ? 0 + : 1; +} + +static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() +{ + var starfireDescriptor = new MimirStreamDescriptor( + "starfire:move:primary", + MimirStreamKind.Tracking, + MimirStreamOrigin.LocalDevice, + DisplayName: "Starfire PS Move primary"); + var nightwingDescriptor = new MimirStreamDescriptor( + "nightwing:move:primary", + MimirStreamKind.Tracking, + MimirStreamOrigin.Network, + DisplayName: "Nightwing PS Move primary"); + var settings = new MimirSynchronizationSettings + { + Streams = [starfireDescriptor, nightwingDescriptor], + }; + var hub = new MimirSynchronizationHub(settings); + var starfire = new MimirNativeIngestStreamSource(starfireDescriptor); + var nightwing = new MimirNativeIngestStreamSource(nightwingDescriptor); + hub.AddSource(starfire); + hub.AddSource(nightwing); + starfire.PushTrackingObservation(MimirTrackingObservation.PsMove( + starfireDescriptor.SourceId, + "psmove:starfire:00", + sequence: 1, + sourceTimestampNs: 1_000_000_000L, + arrivalTimestampNs: 1_000_000_120L, + positionMeters: new MimirVector3Snapshot(0.21, 1.12, 0.84), + orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(0.04, 0.0, -0.02), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.0, 0.1, 0.0), + confidence: 0.93, + calibrationId: "starfire-move-usb-calibration-v1", + trackingSpaceId: "starfire-move-space", + producerId: "muninn:starfire:move", + hostId: "starfire", + latencyMilliseconds: 1.2, + battery01: 0.8, + buttons: + [ + new MimirTrackingButtonSnapshot("move", Pressed: false, Value: 0.0), + new MimirTrackingButtonSnapshot("trigger", Pressed: true, Value: 0.72) + ])); + nightwing.PushTrackingObservation(MimirTrackingObservation.PsMove( + nightwingDescriptor.SourceId, + "psmove:nightwing:00", + sequence: 7, + sourceTimestampNs: 1_000_008_333L, + arrivalTimestampNs: 1_000_012_900L, + positionMeters: new MimirVector3Snapshot(-0.34, 1.05, 1.42), + orientation: new MimirQuaternionSnapshot(0.0, 0.18, 0.0, 0.984), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(-0.02, 0.01, 0.03), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.2, 0.0, 0.0), + confidence: 0.89, + calibrationId: "nightwing-move-usb-calibration-v1", + trackingSpaceId: "nightwing-move-space", + producerId: "muninn:nightwing:move", + hostId: "nightwing", + latencyMilliseconds: 4.6, + battery01: 0.67, + buttons: + [ + new MimirTrackingButtonSnapshot("move", Pressed: true, Value: 1.0), + new MimirTrackingButtonSnapshot("trigger", Pressed: false, Value: 0.0) + ])); + + var consumed = hub.PollSources(); + return new MoveTrackingSmokeContext(hub, consumed); +} + static MimirAudioSynchronizationReport? EstimateBioacousticDelay(float[] reference, float[] candidate, int sampleRate) { const string referenceSourceId = "actuator-reference"; @@ -1563,6 +1667,40 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) scene.SceneId, MimirProgramPublicationConfigurations.OperatorSurfaces[0], createdAt); + var starfireMove = MimirTrackingObservation.PsMove( + "starfire:move:primary", + "psmove:starfire:00", + sequence: 1, + sourceTimestampNs: 1_000_000_000L, + arrivalTimestampNs: 1_000_000_120L, + positionMeters: new MimirVector3Snapshot(0.21, 1.12, 0.84), + orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(0.04, 0.0, -0.02), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.0, 0.1, 0.0), + confidence: 0.93, + calibrationId: "starfire-move-usb-calibration-v1", + trackingSpaceId: "starfire-move-space", + producerId: "muninn:starfire:move", + hostId: "starfire", + latencyMilliseconds: 1.2, + battery01: 0.8); + var nightwingMove = MimirTrackingObservation.PsMove( + "nightwing:move:primary", + "psmove:nightwing:00", + sequence: 7, + sourceTimestampNs: 1_000_008_333L, + arrivalTimestampNs: 1_000_012_900L, + positionMeters: new MimirVector3Snapshot(-0.34, 1.05, 1.42), + orientation: new MimirQuaternionSnapshot(0.0, 0.18, 0.0, 0.984), + linearVelocityMetersPerSecond: new MimirVector3Snapshot(-0.02, 0.01, 0.03), + angularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.2, 0.0, 0.0), + confidence: 0.89, + calibrationId: "nightwing-move-usb-calibration-v1", + trackingSpaceId: "nightwing-move-space", + producerId: "muninn:nightwing:move", + hostId: "nightwing", + latencyMilliseconds: 4.6, + battery01: 0.67); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions @@ -1597,10 +1735,18 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) eveSurface, new CultRecordHandle(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}"))) .ConfigureAwait(false); + await cache.UpsertAsync( + starfireMove, + new CultRecordHandle(new CultRecordKey($"mimir-move-tracking:{starfireMove.ObservationId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + nightwingMove, + new CultRecordHandle(new CultRecordKey($"mimir-move-tracking:{nightwingMove.ObservationId}"))) + .ConfigureAwait(false); await cache.FlushAsync().ConfigureAwait(false); Console.WriteLine( - $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length}"); + $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length} moveTracking=2 hosts={starfireMove.HostId},{nightwingMove.HostId}"); return 0; } @@ -5602,6 +5748,11 @@ public sealed record ComplexContourBandCorrection( double Reliability, bool Usable); +sealed record MoveTrackingSmokeContext(MimirSynchronizationHub Hub, int Consumed) : IDisposable +{ + public void Dispose() => Hub.Dispose(); +} + public sealed record ComplexContourReflectionCorrection( double RelativeDelaySamples, int ObservationCount, diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index f50a46c..f485689 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -862,6 +862,11 @@ direct return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {block.Channels}ch {block.SampleRate}Hz {block.SampleFormat} frames {block.FrameCount} bytes {latest.Value.ByteLength} edge {buffer.EdgeNs}"; } + if (latest?.TrackingObservation is { } tracking) + { + return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} {tracking.Kind} device={tracking.DeviceId} producer={tracking.ProducerId} host={tracking.HostId} confidence={tracking.Confidence:0.00} edge {buffer.EdgeNs}"; + } + return $"{buffer.Descriptor.Label} [{buffer.Descriptor.SourceId}]: {buffer.Count} edge {buffer.EdgeNs}"; } diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index b09d387..1b84f06 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -47,8 +47,8 @@ public static class MimirModuleLibrary new( "cultmesh-contracts", "Mimir.Runtime", - "Typed state documents for codebooks, decoder state, acoustic path state, and actuator state.", - ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state"], + "Typed state documents for codebooks, decoder state, acoustic path state, actuator state, program state, and Move tracking observations.", + ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation"], ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]), new( "path-learning", diff --git a/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs index ba54c21..6fd188d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs @@ -132,6 +132,48 @@ public static class MimirNativeCaptureConfigurations Notes = "Remote co-streamer audio witness. It decodes locally and ships typed timing/path state back over CultMesh." }; + public static MimirNativeCaptureDeviceProfile StarfirePsMoves { get; } = new( + "starfire-ps-moves", + "Starfire PS Move tracking witnesses", + MimirStreamKind.Tracking, + MimirNativeCaptureTransport.VendorSdk, + MimirCaptureClockDomain.DeviceHardware, + ChannelCount: 2, + PreferredWidth: 0, + PreferredHeight: 0, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 120.0, + DefaultRollingBufferSeconds, + Required: false, + OwnedOutputs: ["local-controller-pose-observations", "performance-motion-cues"], + "USB-attached local Moves. Mimir buffers the observations; tracking code owns hardware pose estimation."); + + public static MimirNativeCaptureDeviceProfile NightwingPsMoves { get; } = StarfirePsMoves with + { + Id = "nightwing-ps-moves", + DisplayName = "Nightwing PS Move tracking witnesses", + Transport = MimirNativeCaptureTransport.NetworkCultMesh, + ClockDomain = MimirCaptureClockDomain.NetworkSender, + OwnedOutputs = ["remote-controller-pose-observations", "performance-motion-cues"], + Notes = "USB-attached Nightwing Moves. Muninn on Nightwing publishes typed pose/button observations; Odin owns discovery/schema projection; Mimir owns buffering and fusion." + }; + + public static MimirNativeCaptureDeviceProfile OdinMoveTrackingCatalog { get; } = new( + "odin-move-tracking-catalog", + "Odin Move tracking discovery catalog", + MimirStreamKind.Tracking, + MimirNativeCaptureTransport.NetworkCultMesh, + MimirCaptureClockDomain.CanonicalFitted, + ChannelCount: 4, + PreferredWidth: 0, + PreferredHeight: 0, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 120.0, + DefaultRollingBufferSeconds, + Required: false, + OwnedOutputs: ["move-stream-discovery", "tracking-schema-rendezvous"], + "Discovery/projection surface only. Odin names Starfire and Nightwing Move streams; it does not own pose capture or Mimir fusion."); + public static IReadOnlyList LocalSixCameraProfiles { get; } = [ LeapLeftIr, @@ -146,6 +188,9 @@ public static class MimirNativeCaptureConfigurations [ .. LocalSixCameraProfiles, ScarlettStarfire, - ScarlettRaven + ScarlettRaven, + StarfirePsMoves, + NightwingPsMoves, + OdinMoveTrackingCatalog ]; } diff --git a/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs b/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs index 04ff34e..5ea9c63 100644 --- a/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs +++ b/src/Mimir.Runtime/Synchronization/MimirNativeIngestStreamSource.cs @@ -20,7 +20,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource ulong payloadHandle, int byteLength = 0, ReadOnlyMemory data = default, - MimirVideoFrameDescriptor? videoFrame = null) + MimirVideoFrameDescriptor? videoFrame = null, + MimirTrackingObservation? trackingObservation = null) { samples.Enqueue(new MimirStreamSample( Descriptor.SourceId, @@ -32,7 +33,8 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource payloadHandle, byteLength, data, - videoFrame)); + videoFrame, + TrackingObservation: trackingObservation)); } public void PushVideoFrame( @@ -54,6 +56,25 @@ public sealed class MimirNativeIngestStreamSource : IMimirStreamSource frame); } + public void PushTrackingObservation(MimirTrackingObservation observation) + { + if (Descriptor.Kind != MimirStreamKind.Tracking) + { + throw new InvalidOperationException("Tracking observations can only be pushed into a tracking stream source."); + } + + if (!string.Equals(observation.StreamId, Descriptor.SourceId, StringComparison.Ordinal)) + { + throw new ArgumentException("Tracking observation stream id does not match this stream source.", nameof(observation)); + } + + Push( + observation.SourceTimestampNs, + observation.ArrivalTimestampNs, + payloadHandle: 0, + trackingObservation: observation); + } + public bool TryRead(out MimirStreamSample sample) { return samples.TryDequeue(out sample); diff --git a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs index d872b9b..ad44300 100644 --- a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs +++ b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs @@ -155,6 +155,7 @@ public static class MimirPerfectMachineProfiles [ "muninn.telemetry_surface.v1", "muninn.capture_stream.v1", + "mimir.move_tracking_observation.v1", "mimir.cultmesh_media_frame.v1" ]); diff --git a/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs b/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs index 8fcb617..4806622 100644 --- a/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs +++ b/src/Mimir.Runtime/Synchronization/MimirStreamKind.cs @@ -4,6 +4,7 @@ public enum MimirStreamKind { Audio, Video, + Tracking, } public enum MimirStreamOrigin diff --git a/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs b/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs index 00ab704..57cf42a 100644 --- a/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs +++ b/src/Mimir.Runtime/Synchronization/MimirStreamSample.cs @@ -11,4 +11,5 @@ public readonly record struct MimirStreamSample( int ByteLength = 0, ReadOnlyMemory Data = default, MimirVideoFrameDescriptor? VideoFrame = null, - MimirAudioBlockDescriptor? AudioBlock = null); + MimirAudioBlockDescriptor? AudioBlock = null, + MimirTrackingObservation? TrackingObservation = null); diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs index 4a16aed..c9bfd73 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationHub.cs @@ -51,6 +51,15 @@ public sealed class MimirSynchronizationHub : IDisposable public IReadOnlyList AudioChirpBinCalibrationProfiles => audioSynchronization.LastCalibrationProfiles; + public IReadOnlyList TrackingObservations => + Buffers.Buffers + .Where(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking) + .Select(buffer => buffer.Latest?.TrackingObservation) + .Where(observation => observation != null) + .Cast() + .OrderBy(observation => observation.StreamId, StringComparer.Ordinal) + .ToArray(); + public MimirChirpBinCodebookPlan? ChirpBinEmissionPlan => chirpBinCalibrationModel?.EmissionPlan; @@ -87,7 +96,8 @@ public sealed class MimirSynchronizationHub : IDisposable { var audio = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Audio); var video = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Video); - return $"{video} video / {audio} audio buffers"; + var tracking = Buffers.Buffers.Count(buffer => buffer.Descriptor.Kind == MimirStreamKind.Tracking); + return $"{video} video / {audio} audio / {tracking} tracking buffers"; } public IReadOnlyList AnalyzeAudioSynchronization( diff --git a/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs b/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs index 8a5e054..b0b0b9d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSynchronizationSettings.cs @@ -14,8 +14,10 @@ public sealed class MimirSynchronizationSettings var streams = new List(); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.LocalDevice); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.LocalDevice); + AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_LOCAL_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.LocalDevice); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_VIDEO_STREAMS"), MimirStreamKind.Video, MimirStreamOrigin.Network); AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_AUDIO_STREAMS"), MimirStreamKind.Audio, MimirStreamOrigin.Network); + AddStreams(streams, Environment.GetEnvironmentVariable("MIMIR_NETWORK_TRACKING_STREAMS"), MimirStreamKind.Tracking, MimirStreamOrigin.Network); return new MimirSynchronizationSettings { diff --git a/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs b/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs new file mode 100644 index 0000000..fc18182 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs @@ -0,0 +1,100 @@ +using MessagePack; +using GameCult.Caching; + +namespace Mimir.Runtime.Synchronization; + +public enum MimirTrackingObservationKind +{ + Unknown, + PsMoveController, + Hand, + Headset, + Tool, +} + +[MessagePackObject] +public readonly record struct MimirVector3Snapshot( + [property: Key(0)] double X, + [property: Key(1)] double Y, + [property: Key(2)] double Z); + +[MessagePackObject] +public readonly record struct MimirQuaternionSnapshot( + [property: Key(0)] double X, + [property: Key(1)] double Y, + [property: Key(2)] double Z, + [property: Key(3)] double W); + +[MessagePackObject] +public sealed record MimirTrackingButtonSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] bool Pressed, + [property: Key(2)] double Value); + +[MessagePackObject] +[CultDocument("mimir.move_tracking_observation", "mimir.move_tracking_observation.v1")] +public sealed record MimirTrackingObservation( + [property: Key(0)] + [property: CultName] + string ObservationId, + [property: Key(1)] string StreamId, + [property: Key(2)] string DeviceId, + [property: Key(3)] MimirTrackingObservationKind Kind, + [property: Key(4)] string ProducerId, + [property: Key(5)] string HostId, + [property: Key(6)] string DiscoveryProviderId, + [property: Key(7)] string TrackingSpaceId, + [property: Key(8)] string CalibrationId, + [property: Key(9)] long SourceTimestampNs, + [property: Key(10)] long ArrivalTimestampNs, + [property: Key(11)] ulong Sequence, + [property: Key(12)] MimirVector3Snapshot PositionMeters, + [property: Key(13)] MimirQuaternionSnapshot Orientation, + [property: Key(14)] MimirVector3Snapshot LinearVelocityMetersPerSecond, + [property: Key(15)] MimirVector3Snapshot AngularVelocityRadiansPerSecond, + [property: Key(16)] double Confidence, + [property: Key(17)] double LatencyMilliseconds, + [property: Key(18)] double Battery01, + [property: Key(19)] MimirTrackingButtonSnapshot[] Buttons) +{ + public static MimirTrackingObservation PsMove( + string streamId, + string deviceId, + ulong sequence, + long sourceTimestampNs, + long arrivalTimestampNs, + MimirVector3Snapshot positionMeters, + MimirQuaternionSnapshot orientation, + MimirVector3Snapshot linearVelocityMetersPerSecond, + MimirVector3Snapshot angularVelocityRadiansPerSecond, + double confidence, + string calibrationId, + string trackingSpaceId = "nightwing-move-space", + string producerId = "muninn:nightwing:move", + string hostId = "nightwing", + string discoveryProviderId = "odin", + double latencyMilliseconds = 0.0, + double battery01 = double.NaN, + MimirTrackingButtonSnapshot[]? buttons = null) => + new( + $"{streamId}:{sequence}", + streamId, + deviceId, + MimirTrackingObservationKind.PsMoveController, + producerId, + hostId, + discoveryProviderId, + trackingSpaceId, + calibrationId, + sourceTimestampNs, + arrivalTimestampNs, + sequence, + positionMeters, + orientation, + linearVelocityMetersPerSecond, + angularVelocityRadiansPerSecond, + Math.Clamp(confidence, 0.0, 1.0), + Math.Max(0.0, latencyMilliseconds), + double.IsFinite(battery01) ? Math.Clamp(battery01, 0.0, 1.0) : double.NaN, + buttons ?? []); +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 5d1df26..ab7ff83 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -123,3 +123,4 @@ {"ts": "2026-06-08T09:59:22+01:00", "type": "verification", "status": "blocked", "note": "CultMesh media bridge verification: Yggdrasil UFW was opened for 3075/udp on wg-gamecult and Starfire receiver now connects from 10.77.0.2. Synthetic sender on Yggdrasil connects and sends chunks without User Not Verified after CultLib pre-verification schema-v0 policy patch, but relay cache does not receive mimir.cultmesh_media_frame updates and Starfire receiver logs no frames. Remaining blocker is CultNet document put/apply path, not WireGuard routing or OBS endpoint."} {"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."} {"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."} +{"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."} diff --git a/state/map.yaml b/state/map.yaml index 8f70fd0..2c038ae 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -33,6 +33,9 @@ constraints: memory. - Networked feeds are producers into the live runtime, not independent clock authorities. + - USB-attached Moves on Starfire and Nightwing are tracking witnesses. Odin + may discover/project their stream schemas, but it does not own pose capture + or Mimir fusion. - CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but Mimir owns program selection, composition, preview/control, and publication. @@ -63,6 +66,9 @@ invariants: response, and compute-offload work need typed mesh-syncable state surfaces so phones, microcontrollers, Raven, and Starfire can cooperate without becoming independent timing authorities. + - Move tracking observations are stream samples, not program controls or + dashboard truth. Starfire-local and Nightwing-remote Move samples must enter + Mimir.Runtime as typed tracking buffers before Fensalir or Eve acts on them. accepted_design: - `Mimir.slnx` owns the app surface. @@ -74,6 +80,11 @@ accepted_design: device timestamp, and optional native/GPU handle. - `MimirAudioBlockDescriptor` carries sample rate, channels, sample format, frame count, timestamp, and optional native handle. + - `MimirTrackingObservation` carries typed PS Move pose/button/battery/ + confidence samples as `mimir.move_tracking_observation.v1`; Starfire Moves + are local tracking streams, Nightwing Moves are emitted by Muninn on + Nightwing, Odin owns discovery/schema projection, and Mimir owns buffering + and fusion. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the From e85c05029cdf49ffe415bf7ecf3cb2ff2ed6f4f9 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 16:44:35 +0100 Subject: [PATCH 007/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 61724 -> 65126 bytes .voidbot/voice/identity.json | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 1827be50308103c49d3690ec5882069b19c3f65b..a8296794ff4e33359d8d75a7e57cce2b1e1fd940 100644 GIT binary patch delta 5328 zcmbtYQH5=bf$fwrVUtyF?4^#Q7=c>o>COF&8}MOCA|wranBZ13$J zje^uDx$KQ+#xwu_ec%86`^qO*p8waCZ~pVz)#cJm*_yP@{@ z`Kpa~OtJ(M!xoF?Lu~B%oiP^st&j(d2l&4&!eoqfqBt?!l-|qiD0F>Cgtk!69lWpm z%#0_PjG{JkA`!DNN?0g_!`iqPlb*o+e4Duv6ZDao^jjY0MDyQcel|B|Hr~~gJi)p+ zZN>?GWb=`nWr@cVJPujN+ZdwX^5gk2OJsU)oM2w!EsP)CpNcra2^W(+G_570G<2|I zM@CI7^@D&lF@uRmaUFr3@o(Vn$~$;lSEh;gUB{0*0bgKE+JMW2@z)a((%lyyen9X5 zi^TF{;xGZS!Q#mFc@QkHR=^Q~C`AH7hY240LNYg!`n5V-NrXVP0%0fPZ&8w>@ZPq1 ze9txEp=o~(QFPM;$z{UDs)QxcJeLlOLcu)zz%~4Ej_%vyv=qN(;YG$1A*GM7FJLfE z1ji~jOGh)3j*v#c!&b^$f^peO)tbf8HTpm;-eJ7c5nRSfJa_w+dfBo$=M}?o?22Jc zG)#k6_=I8CM8&O^%T~iur_S7c&i|F*DoL1ec={$$AJ^w zX9+K%cr?k*Ys4P!Xt)5Ru@|L*gDf#lFteoq46^j$Dga#;y&?zmgj+zxLui4MFvlEW zu@5|sBqkt%P1!;CyKzU@NMIO`AF%`Ow-?`@A{-Sk%8tfm;=4!+Fh~b-(soCOKG0Oc z8xiIlhH;ciTZ|EukPc20iAiHi^kE$=7|VI1o;-rK&bTAok7rf1Y|Iqd* zk(6W%IlLJNW_zIP0JsnA@qA1ZkNQYH?jJ-T4`EtJx>iD$lxe6zN-x&r=|i5#L>scZ z=Y2dQ4g>%vMg@c+3Kuk7Y1qa%v<0yPCBWfii4WWQe30s+r*7LW1wf+@LklCy;4+}a zbiq89q9r*piQwx^i5?Dw1}{+i_l%m4-OcB41r91vV2wVFvx^R+o)&v*`i)Vw`R;3P zrMS5`v&Y)F6Ud|Eq2&Pz<|8JE7f%50u`RY+syhu}+nTTq%c+-)#)Mrr-0DQLUaH!b z5U$$&^fr^^xHMTdC(ViRQn|eMSlV$&rb3tCO}Vkp4tNc8DBnm@|Ob)AJclvA-cqYPTCBh zwJ}gY^30ZFKZkyI3H!g@H3+pn>T@3UQG>8T(g9)QVD`)SM!NFakVAi|L=YMs>AO(v z1YtvMNu@e}cx$&QJU-{+sTTMQ`zRlzWXS;9!)0jy3dsISz`pfz$o|o^4yO9}sja() z@l9H(vrs6>lGqF*yosz_3g%sJJMpB*m>=kb;r5A#)N8sBhXZ0LiU7t4+R7T(|X#Rv4~&%frOdg|4UtW@DugPT?v;$F9mx?6IL zGH+Ox?bPeGX%Yd3osX$5zP4`7GLvGe=?hz-?Q7K|&#%*j>ANbkemRFz(nkM^Uw>|U(tS{xBrDxTSs0}=YBV;?&;m7_PlfB^*`Pz5^h8Mbt*zGWu(hN6spa4 zj&FlDqTG{+f=JpZ^aY+YBhv&Ms}@KTP+_UM78QX%sNTM_Q64HS@K0^73vv$iqKGb; zgIFz0ZK@TOXN)ybcrwe=cSiFZ0*%7&2$DaDU_|L9EA`=2b>)Hkh3GI>h4uz@(_QYm z!PI@W`)@zpeDVSHlUFxgK5rM#k1b15z4AXYhr#I!OdWoAz1nwTbQS82Fs%s(g}tms zet+FJj{rEW^hkj5;K&SpR>x0l?e5keQ33m&67Wo8;idr$njj44ju4!e2@>GIic*{* zWl3a5q;a^sF4-NbZh0ksf`rhHq=Y9TPlG_Ug+RbC3~C{7WN5m%qv=F}55J0S^>aUg zD*|=MPV`OgU}5i{7CRT_xZqz$2-{oAWapvmHUnoN3z~uIL7;&A7EXsU2iU>A%tb!* z)_3!KXMzEbO)jl(!7}#Jf%Q1cNQ@&U9#uXs-hsOov9$pn_)y#L1YmJt1|c~(Xa+4F z#`6$;?4{|iuVjz_4|-@Zms#~eEUUkq-MI4n=ws*ZIXpq&wkZb!dQzA zc%YLgH%in=T(dNE==7FCu6qADd1~dzE4@?izW5Eb;-mZ2$Q`j7s(9dP7Fh{MbD*ZquJ1+<2!jlxgu_7JS_pV%ItJRX0ZR=~AwXy=vqlFe zCNlZ1uc2Rb&3JXzZG9`Yj>j*qS&8Tvm}sAgSOg=!X5sHXn@;fCJy2fw*y zmFbjfj%oAygt%m_d+YS3?zGlpB3rH^1zG!n%D(%!WThL>|G`qXU+=5E%RrIwM9_f> zKrwQTx#Scg0EYTbAdyPkK>H4n>!dOGnb#dYk8dsXS1mTmC35Ntl!vTQT@sLxuv7qF z14!-q{>^pGdvwm=yhKq>hHwfEnu}&31H*tvF^Z^m?c6y70m`eFzEwX?031-{2{aD8 zg?*tY8QB0Js*{PYYyF`0Rl62%zj?x(s9x5f!sz5Bsj!^&OP8d9DSa^9_wlCH>Z;Gz zwf>4?_}IDauVr25;$Di8BVwO?O2kli`7EIz;~QO0e=kNBZz*7Q3hNRkU=iA-0N)7t wh!_E%dTD$oz#^_LeleO`v!Z1zmwP|_?C41E%;y`wHsqe^Dwp|waH6#LzZ;{WEC2ui delta 3068 zcmai0U5H%O9cMRolWMX=Hfup^_8@AZ&J6RlvpXXt%~P#J6GKQa50*3co;&w+@4aXC zoO5?F_G2*7S_!S%Lwzw1CB;@B3N52GCi(se^hRs8vA6RJKT3TAL(BJ>unf>UC zu&9wr+26SR$j7B!H`-Ny!|hADd#O^hQ+Mmh%X>fByyMviX7{^=G04lAz& z376?+b`s^em06FLt)r-YoY9A z(heEDLe;cVDR!KQo*Z>$f?JhW*K=X0^ycw1F}F;d*C2ZYSzd^P!E|nX8~O*j{MS!jNsq=-YG22`yMsHtRlpDyhec0IwjV{ zPK=`8zDa4;n2K0e1S+M{oOQ82zMzSmC$#y@!;VTS6$H9SjKIctf`Z|&94M^;7)xX( z-1?bAqw^2ZU?&5dwu%87Zgb5oMqyQ724*@@4CxZZY;TD$M@eG@@BQYQmdAOwpb}Qt5Ku8 z+H9_rgI3fETHRoUi)HuwkM44ZFW+-JXuf)?UhmXbJN0U%y;^nb;-UH7*WCJ5=1yFj zbMeKpJN)c(qZ>v01JJL4)DHy>XFxH^6VQ3X52A~?uq|fxwZa7;@24#yWS&~J2-u_)JQRI|QHq=^CbM93T~TgHQcIHF@Q zf^o=T2hru3SVHimDH(Y|Psr}ufd>%5frQ5+WRcVclva>NUl|}a)kBGRj()Lm^amIt z8F~%GproEs2lGI{7mv90|AjBusjg5PR1?bfqLiAX22g%3< zP=8C?7y(qMr3t0C2Jn3G!AHINa-EgXy@-Gghel{R<({U20H#o4_#i2^hy6fLMPXO_ z!lB1uBmu;?ffqgAP#_VUYbe0OO4Rs2m7yf`8R8X4q@P<+c@4dj_U-V)BR8KpeC4s5 z&$y$#*&YA;9pj&TIJ|9q=Z)8I zb+=uA*qy$yFnY4^S(?Elx(G&0Bjy_<=LEh$b1n={(|Mw>Jn{>iDiW3T(Bj0W6^2RC z?+4;@pY|uSoM!0{2c8exbQs~d5W-icfmjY=93_gxA#T7%O&-DFBoF|AMp&DmAL1_# zI1d&BA9xz}Cb{mBE*#bO(S?<5bKwVcIGLP%q!}?JVuF$935bWBE?Orp6`4-VnmBfm zht$%4&;S&{6Z`Ob*7)F&n$+%6G=@U^Zzwv@{=gu4m!xM$iZG}2{ z*A6~zmz*5drgX%K3C9{VO120U`rz&+Dp&(VQ^DfU!nJYleENX<%lqG&ZLGM*E-$*X zAMB^EwQI-5Z~yhK`SExE^|KlG$v+oHf4+X}*ZM!d{)@tLH=m;6&|zl{Jz?t;ufUEl zjRByez0rBda!iK~P+)E>CUhY#P0uH$$e1b9fHYVZAaA=6InCN@eXbC@d*VNAe)&|j e+Nm`<%|@l&YK<5Fy);MUzOP>UyrXoxcJ_ZS_-y3> diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index 9adfd65..82c2392 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-06-08T23:28:08.132Z", + "createdOrRefreshedAt": "2026-06-10T14:39:43.962Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 52a439842c152291d6b375ce514c99d22dcba35e Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 17:41:18 +0100 Subject: [PATCH 008/135] Restore Nightwing Move tracking witness --- Mimir.slnx | 2 + docs/implementation-plan.md | 7 + notes/current-system-map.md | 11 + scripts/start-nightwing-move-tracking.ps1 | 158 ++++++ .../Mimir.EveSensorReceiver.csproj | 17 + src/Mimir.EveSensorReceiver/Program.cs | 501 ++++++++++++++++++ .../Synchronization/MimirCultMeshContracts.cs | 44 ++ .../Mimir.VerseRecorder.csproj | 12 + src/Mimir.VerseRecorder/Program.cs | 151 ++++++ state/evidence.jsonl | 1 + state/map.yaml | 6 + tools/nightwing_typed_witness_publisher.py | 493 +++++++++++++++++ tools/nw_eye_cap.py | 280 ++++++++++ tools/nw_move_hint.py | 170 ++++++ 14 files changed, 1853 insertions(+) create mode 100644 scripts/start-nightwing-move-tracking.ps1 create mode 100644 src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj create mode 100644 src/Mimir.EveSensorReceiver/Program.cs create mode 100644 src/Mimir.VerseRecorder/Mimir.VerseRecorder.csproj create mode 100644 src/Mimir.VerseRecorder/Program.cs create mode 100644 tools/nightwing_typed_witness_publisher.py create mode 100644 tools/nw_eye_cap.py create mode 100644 tools/nw_move_hint.py diff --git a/Mimir.slnx b/Mimir.slnx index 84ccf97..901ce47 100644 --- a/Mimir.slnx +++ b/Mimir.slnx @@ -3,5 +3,7 @@ + + diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 2b3abf3..ea070b1 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -39,6 +39,13 @@ a named invariant that the native runtime cannot protect yet. have USB-attached Moves; Starfire can feed local tracking streams, while Muninn on Nightwing publishes remote tracking streams. Odin owns discovery and schema projection, not pose capture or Mimir fusion. +- `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing + bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, + stages the Nightwing Eye/Move Python worker, lights configured USB Moves via + HID report `0x06`, and publishes `mimir.psmove_light_state.v1` plus + `mimir.move_controller_observation_state.v1` blob observations from both PS3 + Eyes. The blob stream is optical witness evidence for later pose fusion, not + the final 6DoF pose owner. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 936b4e4..139c133 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -120,6 +120,17 @@ Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csp -- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing Move observation into two tracking buffers. +Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. +Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages +`nw_eye_cap.py`, `nw_move_hint.py`, and `nightwing_typed_witness_publisher.py` +onto Nightwing, and launches the publisher with `--track-eyes` plus explicit +`--move-light name=/dev/hidrawN:#rrggbb` HID ownership. The worker keeps the +USB Move sphere lit with PS Move report `0x06`, emits +`mimir.psmove_light_state.v1`, and publishes compact +`mimir.move_controller_observation_state.v1` blob observations from both PS3 +Eyes. Those Eye/Move observations are the live optical witness feeding the later +pose/fusion owner; they are not themselves the final 6DoF pose authority. + ## Program Composition [[docs/mimir-program-composition|Mimir Program Composition]] is the live diff --git a/scripts/start-nightwing-move-tracking.ps1 b/scripts/start-nightwing-move-tracking.ps1 new file mode 100644 index 0000000..f3397d5 --- /dev/null +++ b/scripts/start-nightwing-move-tracking.ps1 @@ -0,0 +1,158 @@ +param( + [string]$NightwingHost = "nightwing", + [int]$Port = 8796, + [string]$StarfireHost = "192.168.1.66", + [string]$Path = "/eve/periwinkle", + [string]$EyeDevices = "/dev/video2,/dev/video3", + [string[]]$MoveLight = @( + "move-usb=/dev/hidraw1:#35ff6c" + ), + [string]$RecorderRoot = "C:\Users\Meta\Videos\Mimir\VerseCaptures", + [switch]$SkipRecorder, + [switch]$DryRun +) + +$ErrorActionPreference = "Stop" +$repo = Split-Path -Parent $PSScriptRoot +$runId = Get-Date -Format "yyyyMMdd-HHmmss" +$runDir = Join-Path $repo "artifacts\runtime\nightwing-move-tracking-$runId" +$witnessUrl = "ws://$StarfireHost`:$Port$Path" +$subscribeUrl = "ws://127.0.0.1:$Port$Path/subscribe" + +New-Item -ItemType Directory -Force -Path $runDir | Out-Null +New-Item -ItemType Directory -Force -Path $RecorderRoot | Out-Null + +function Start-LoggedProcess { + param( + [string]$Name, + [string]$FilePath, + [string[]]$ArgumentList + ) + + $stdout = Join-Path $runDir "$Name.out.log" + $stderr = Join-Path $runDir "$Name.err.log" + if ($DryRun) { + Write-Host "DRY $Name`: $FilePath $($ArgumentList -join ' ')" + return $null + } + + $process = Start-Process ` + -FilePath $FilePath ` + -ArgumentList $ArgumentList ` + -WorkingDirectory $repo ` + -WindowStyle Hidden ` + -RedirectStandardOutput $stdout ` + -RedirectStandardError $stderr ` + -PassThru + Set-Content -Path (Join-Path $runDir "$Name.pid") -Value $process.Id + Write-Host "$Name pid=$($process.Id) stdout=$stdout stderr=$stderr" + return $process +} + +function Wait-Health { + param([string]$Url) + if ($DryRun) { + Write-Host "DRY wait health: $Url" + return + } + + $deadline = (Get-Date).AddSeconds(45) + do { + try { + $response = Invoke-WebRequest -UseBasicParsing -Uri $Url -TimeoutSec 3 + if ($response.StatusCode -ge 200 -and $response.StatusCode -lt 300) { + Write-Host "health ok: $Url" + return + } + } catch { + Start-Sleep -Milliseconds 500 + } + } while ((Get-Date) -lt $deadline) + + throw "Timed out waiting for $Url" +} + +Write-Host "Nightwing Move tracking run: $runDir" +Write-Host "Witness URL: $witnessUrl" + +if (-not $DryRun) { + Get-CimInstance Win32_Process | + Where-Object { + ($_.CommandLine -match "Mimir.EveSensorReceiver" -and $_.CommandLine -match "--port $Port") -or + ($_.CommandLine -match "Mimir.VerseRecorder" -and $_.CommandLine -match "$Port$Path/subscribe") + } | + ForEach-Object { + try { Stop-Process -Id $_.ProcessId -Force -ErrorAction Stop } catch {} + } +} + +$receiver = Start-LoggedProcess ` + -Name "periwinkle-receiver" ` + -FilePath "dotnet" ` + -ArgumentList @( + "run", + "--project", + (Join-Path $repo "src\Mimir.EveSensorReceiver\Mimir.EveSensorReceiver.csproj"), + "--", + "--port", "$Port", + "--path", $Path, + "--subscribe-path", "$Path/subscribe", + "--source-id", "nightwing-witness", + "--type", "cultmesh-observation" + ) + +if ($receiver) { + Wait-Health -Url "http://127.0.0.1:$Port/health" +} + +if (-not $SkipRecorder) { + Start-LoggedProcess ` + -Name "verse-recorder" ` + -FilePath "dotnet" ` + -ArgumentList @( + "run", + "--project", + (Join-Path $repo "src\Mimir.VerseRecorder\Mimir.VerseRecorder.csproj"), + "--", + "--url", $subscribeUrl, + "--out-dir", $RecorderRoot + ) | Out-Null +} + +if ($DryRun) { + Write-Host "DRY Nightwing stage and start on $NightwingHost" +} else { + scp (Join-Path $repo "tools\nw_eye_cap.py") "${NightwingHost}:/tmp/nw_eye_cap.py" | Out-Null + scp (Join-Path $repo "tools\nw_move_hint.py") "${NightwingHost}:/tmp/nw_move_hint.py" | Out-Null + scp (Join-Path $repo "tools\nightwing_typed_witness_publisher.py") "${NightwingHost}:/tmp/nightwing_typed_witness_publisher.py" | Out-Null + + $remoteLog = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.log" + $remotePid = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.pid" + $moveArgs = ($MoveLight | ForEach-Object { "--move-light '$($_ -replace "'", "'\''")'" }) -join " " + $remoteScript = @" +mkdir -p ~/.local/state/gamecult +pkill -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' || true +nohup python3 /tmp/nightwing_typed_witness_publisher.py --url '$witnessUrl' --track-eyes --track-builtin-camera --track-builtin-mic --interval 0.10 --eye-devices '$EyeDevices' --eye-window-seconds 0.08 --tracking-stride 6 $moveArgs > $remoteLog 2>&1 < /dev/null & +sleep 0.5 +pgrep -n -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' > $remotePid +cat $remotePid +"@ + $remoteB64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($remoteScript)) + $nightwingPid = ssh $NightwingHost "printf '%s' '$remoteB64' | base64 -d | bash" + Set-Content -Path (Join-Path $runDir "nightwing-witness.pid") -Value ($nightwingPid | Select-Object -First 1) + Set-Content -Path (Join-Path $runDir "nightwing-witness.logpath.txt") -Value $remoteLog + Write-Host "nightwing-witness pid=$($nightwingPid | Select-Object -First 1) log=$remoteLog" +} + +$manifest = [ordered]@{ + kind = "mimir.nightwing_move_tracking_supervisor.v1" + runId = $runId + runDir = $runDir + witnessUrl = $witnessUrl + subscribeUrl = $subscribeUrl + eyeDevices = $EyeDevices + moveLights = $MoveLight + startedAt = (Get-Date).ToString("o") +} +$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path (Join-Path $runDir "supervisor.json") +Write-Host "Supervisor manifest: $(Join-Path $runDir "supervisor.json")" diff --git a/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj b/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj new file mode 100644 index 0000000..48be7bc --- /dev/null +++ b/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj @@ -0,0 +1,17 @@ + + + + Exe + net10.0 + Mimir.EveSensorReceiver + Mimir.EveSensorReceiver + enable + enable + + + + + + + + diff --git a/src/Mimir.EveSensorReceiver/Program.cs b/src/Mimir.EveSensorReceiver/Program.cs new file mode 100644 index 0000000..7cdfc78 --- /dev/null +++ b/src/Mimir.EveSensorReceiver/Program.cs @@ -0,0 +1,501 @@ +using System.Net; +using System.Net.Sockets; +using System.Collections.Concurrent; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using MessagePack; +using Mimir.Runtime.Synchronization; + +var options = EveSensorReceiverOptions.Parse(args); +using var receiver = new EveSensorReceiver(options); +await receiver.RunAsync(); + +internal sealed record EveSensorReceiverOptions( + int Port, + string Path, + string SourceId, + string ExpectedType, + string SubscribePath) +{ + public static EveSensorReceiverOptions Parse(string[] args) + { + return new EveSensorReceiverOptions( + ParseInt(args, "--port", 8793), + ParseString(args, "--path", "/eve/camera"), + ParseString(args, "--source-id", "eve-camera"), + ParseString(args, "--type", "video-frame"), + ParseString(args, "--subscribe-path", ParseString(args, "--path", "/eve/camera") + "/subscribe")); + } + + private static string ParseString(IReadOnlyList args, string name, string fallback) + { + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase)) + { + return args[index + 1]; + } + } + + return fallback; + } + + private static int ParseInt(IReadOnlyList args, string name, int fallback) + { + return int.TryParse(ParseString(args, name, ""), out var value) ? value : fallback; + } +} + +internal sealed class EveSensorReceiver(EveSensorReceiverOptions options) : IDisposable +{ + private readonly TcpListener listener = new(IPAddress.Any, options.Port); + private readonly CancellationTokenSource stopping = new(); + private readonly ConcurrentDictionary subscribers = new(); + private int nextSubscriberId; + + public async Task RunAsync() + { + listener.Start(); + Console.Error.WriteLine($"Eve sensor receiver listening on ws://0.0.0.0:{options.Port}{options.Path} source={options.SourceId}"); + while (!stopping.IsCancellationRequested) + { + var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false); + _ = Task.Run(() => HandleAsync(client)); + } + } + + private async Task HandleAsync(TcpClient tcpClient) + { + await using var stream = tcpClient.GetStream(); + var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false); + if (request.Path == "/health") + { + var health = JsonSerializer.Serialize(new + { + ok = true, + options.SourceId, + options.ExpectedType, + options.Path, + options.SubscribePath, + Subscribers = subscribers.Count, + }); + await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false); + return; + } + + if (string.Equals(request.Path, options.SubscribePath, StringComparison.Ordinal) + && request.Headers.TryGetValue("Sec-WebSocket-Key", out var subscriberKey)) + { + await WriteWebSocketHandshakeAsync(stream, subscriberKey).ConfigureAwait(false); + await SubscribeAsync(stream).ConfigureAwait(false); + return; + } + + if (!string.Equals(request.Path, options.Path, StringComparison.Ordinal) + || !request.Headers.TryGetValue("Sec-WebSocket-Key", out var key)) + { + await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false); + return; + } + + await WriteWebSocketHandshakeAsync(stream, key).ConfigureAwait(false); + while (!stopping.IsCancellationRequested) + { + var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false); + if (frame.Opcode == 0x8) + { + await WriteCloseFrameAsync(stream).ConfigureAwait(false); + break; + } + + if (frame.Opcode == 0x2) + { + if (TryValidateCultMeshObservation(frame.Payload, out var normalizedObservation) + || TryValidateCultMeshMediaObservation(frame.Payload, out normalizedObservation)) + { + Console.WriteLine(normalizedObservation); + Console.Out.Flush(); + await BroadcastAsync(normalizedObservation).ConfigureAwait(false); + } + + continue; + } + + if (frame.Opcode != 0x1) + { + continue; + } + + var text = Encoding.UTF8.GetString(frame.Payload); + if (TryValidateFrameEvent(text, out var normalized)) + { + Console.WriteLine(normalized); + Console.Out.Flush(); + await BroadcastAsync(normalized).ConfigureAwait(false); + } + } + } + + private async Task SubscribeAsync(NetworkStream stream) + { + var id = Interlocked.Increment(ref nextSubscriberId); + var subscriber = new WebSocketSubscriber(id, stream); + subscribers[id] = subscriber; + Console.Error.WriteLine($"Eve sensor receiver subscriber {id} connected on {options.SubscribePath}."); + try + { + await subscriber.SendTextAsync(JsonSerializer.Serialize(new + { + type = "mimir-verse-subscription", + status = "connected", + options.Path, + options.SubscribePath, + options.SourceId, + options.ExpectedType, + })).ConfigureAwait(false); + + while (!stopping.IsCancellationRequested) + { + var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false); + if (frame.Opcode == 0x8) + { + await WriteCloseFrameAsync(stream).ConfigureAwait(false); + break; + } + } + } + catch (IOException) + { + } + catch (SocketException) + { + } + finally + { + subscribers.TryRemove(id, out _); + Console.Error.WriteLine($"Eve sensor receiver subscriber {id} disconnected."); + } + } + + private async Task BroadcastAsync(string normalized) + { + foreach (var pair in subscribers.ToArray()) + { + try + { + await pair.Value.SendTextAsync(normalized).ConfigureAwait(false); + } + catch (IOException) + { + subscribers.TryRemove(pair.Key, out _); + } + catch (SocketException) + { + subscribers.TryRemove(pair.Key, out _); + } + catch (ObjectDisposedException) + { + subscribers.TryRemove(pair.Key, out _); + } + } + } + + private bool TryValidateFrameEvent(string text, out string normalized) + { + normalized = ""; + try + { + using var document = JsonDocument.Parse(text); + var root = document.RootElement; + if (!root.TryGetProperty("type", out var type) + || !string.Equals(type.GetString(), options.ExpectedType, StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + if (!string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase) + && (!root.TryGetProperty("sourceId", out var sourceId) + || !string.Equals(sourceId.GetString(), options.SourceId, StringComparison.Ordinal))) + { + return false; + } + + normalized = root.GetRawText(); + return true; + } + catch (JsonException) + { + return false; + } + } + + private bool TryValidateCultMeshObservation(byte[] payload, out string normalized) + { + normalized = ""; + try + { + var observation = MessagePackSerializer.Deserialize(payload); + if (!string.Equals(observation.StreamId, options.SourceId, StringComparison.Ordinal) && + !string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + normalized = JsonSerializer.Serialize(new + { + type = "cultmesh-observation", + document = "mimir.eve_sensor_observation.v1", + observation.ObservationId, + observation.DeviceId, + observation.StreamId, + observation.Kind, + observation.Sequence, + observation.SensorTimestampNs, + observation.ElapsedRealtimeNs, + observation.WallClockUtc, + observation.ClockDomainId, + observation.Values, + observation.Action, + observation.PointerCount, + observation.X, + observation.Y, + observation.Accuracy, + }); + return true; + } + catch + { + return false; + } + } + + private bool TryValidateCultMeshMediaObservation(byte[] payload, out string normalized) + { + normalized = ""; + try + { + var observation = MessagePackSerializer.Deserialize(payload); + if (!string.Equals(observation.StreamId, options.SourceId, StringComparison.Ordinal) && + !string.Equals(options.ExpectedType, "cultmesh-observation", StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + normalized = JsonSerializer.Serialize(new + { + type = "cultmesh-media-observation", + document = "mimir.eve_media_observation.v1", + observation.ObservationId, + observation.DeviceId, + observation.StreamId, + observation.Kind, + observation.Sequence, + observation.SensorTimestampNs, + observation.ElapsedRealtimeNs, + observation.WallClockUtc, + observation.ClockDomainId, + observation.Format, + observation.Width, + observation.Height, + observation.SampleRate, + observation.Channels, + observation.FrameCount, + observation.PayloadEncoding, + PayloadBytes = observation.Payload.Length, + }); + return true; + } + catch + { + return false; + } + } + + private static async Task ReadHttpRequestAsync(NetworkStream stream) + { + var bytes = new List(); + var lastFour = new Queue(4); + while (true) + { + var value = stream.ReadByte(); + if (value < 0) + { + throw new IOException("Client closed before HTTP headers completed."); + } + + var b = (byte)value; + bytes.Add(b); + lastFour.Enqueue(b); + if (lastFour.Count > 4) + { + lastFour.Dequeue(); + } + + if (lastFour.Count == 4 && lastFour.SequenceEqual(new byte[] { 13, 10, 13, 10 })) + { + break; + } + } + + var text = Encoding.ASCII.GetString(bytes.ToArray()); + var lines = text.Split("\r\n", StringSplitOptions.None); + var first = lines[0].Split(' '); + var headers = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var line in lines.Skip(1)) + { + var colon = line.IndexOf(':'); + if (colon > 0) + { + headers[line[..colon].Trim()] = line[(colon + 1)..].Trim(); + } + } + + await Task.CompletedTask.ConfigureAwait(false); + return new HttpRequest(first.Length > 1 ? first[1] : "/", headers); + } + + private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body) + { + var header = Encoding.ASCII.GetBytes( + $"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n"); + await stream.WriteAsync(header).ConfigureAwait(false); + await stream.WriteAsync(body).ConfigureAwait(false); + } + + private static async Task WriteWebSocketHandshakeAsync(NetworkStream stream, string key) + { + var acceptBytes = SHA1.HashData(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")); + var accept = Convert.ToBase64String(acceptBytes); + var response = Encoding.ASCII.GetBytes( + "HTTP/1.1 101 Switching Protocols\r\n" + + "Upgrade: websocket\r\n" + + "Connection: Upgrade\r\n" + + $"Sec-WebSocket-Accept: {accept}\r\n\r\n"); + await stream.WriteAsync(response).ConfigureAwait(false); + } + + private static async Task ReceiveFrameAsync(NetworkStream stream) + { + var header = new byte[2]; + await ReadExactAsync(stream, header).ConfigureAwait(false); + var opcode = header[0] & 0x0f; + var masked = (header[1] & 0x80) != 0; + var length = header[1] & 0x7f; + if (length == 126) + { + var extended = new byte[2]; + await ReadExactAsync(stream, extended).ConfigureAwait(false); + length = (extended[0] << 8) | extended[1]; + } + else if (length == 127) + { + var extended = new byte[8]; + await ReadExactAsync(stream, extended).ConfigureAwait(false); + var longLength = 0L; + foreach (var b in extended) + { + longLength = (longLength << 8) | b; + } + + if (longLength > int.MaxValue) + { + throw new InvalidDataException("WebSocket frame is too large."); + } + + length = (int)longLength; + } + + var mask = new byte[4]; + if (masked) + { + await ReadExactAsync(stream, mask).ConfigureAwait(false); + } + + var payload = new byte[length]; + await ReadExactAsync(stream, payload).ConfigureAwait(false); + if (masked) + { + for (var index = 0; index < payload.Length; index++) + { + payload[index] ^= mask[index % 4]; + } + } + + return new WebSocketFrame(opcode, payload); + } + + private static Task WriteCloseFrameAsync(NetworkStream stream) + { + return stream.WriteAsync(new byte[] { 0x88, 0x00 }).AsTask(); + } + + private static async Task ReadExactAsync(NetworkStream stream, byte[] buffer) + { + var offset = 0; + while (offset < buffer.Length) + { + var read = await stream.ReadAsync(buffer.AsMemory(offset, buffer.Length - offset)).ConfigureAwait(false); + if (read == 0) + { + throw new IOException("WebSocket closed."); + } + + offset += read; + } + } + + public void Dispose() + { + stopping.Cancel(); + listener.Stop(); + stopping.Dispose(); + } +} + +internal sealed record HttpRequest(string Path, IReadOnlyDictionary Headers); + +internal sealed record WebSocketFrame(int Opcode, byte[] Payload); + +internal sealed class WebSocketSubscriber(int id, NetworkStream stream) +{ + private readonly SemaphoreSlim writeLock = new(1, 1); + + public int Id => id; + + public async Task SendTextAsync(string text) + { + var payload = Encoding.UTF8.GetBytes(text); + var header = new List { 0x81 }; + if (payload.Length < 126) + { + header.Add((byte)payload.Length); + } + else if (payload.Length <= ushort.MaxValue) + { + header.Add(126); + header.Add((byte)(payload.Length >> 8)); + header.Add((byte)(payload.Length & 0xff)); + } + else + { + header.Add(127); + for (var shift = 56; shift >= 0; shift -= 8) + { + header.Add((byte)(((ulong)payload.Length >> shift) & 0xff)); + } + } + + await writeLock.WaitAsync().ConfigureAwait(false); + try + { + await stream.WriteAsync(header.ToArray()).ConfigureAwait(false); + await stream.WriteAsync(payload).ConfigureAwait(false); + await stream.FlushAsync().ConfigureAwait(false); + } + finally + { + writeLock.Release(); + } + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs index 3276d43..4de48b9 100644 --- a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs +++ b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs @@ -171,6 +171,50 @@ public sealed record MimirEveOperatorSurfaceDocument( [property: Key(6)] bool CanShowStats, [property: Key(7)] string[] CommandTopics); +[CultDocument("mimir.eve_sensor_observation", "mimir.eve_sensor_observation.v1")] +[MessagePackObject] +public sealed record MimirEveSensorObservationDocument( + [property: Key(0)] + [property: CultName] + string ObservationId, + [property: Key(1)] string DeviceId, + [property: Key(2)] string StreamId, + [property: Key(3)] string Kind, + [property: Key(4)] long Sequence, + [property: Key(5)] long SensorTimestampNs, + [property: Key(6)] long ElapsedRealtimeNs, + [property: Key(7)] string WallClockUtc, + [property: Key(8)] string ClockDomainId, + [property: Key(9)] double[] Values, + [property: Key(10)] string? Action, + [property: Key(11)] int? PointerCount, + [property: Key(12)] double? X, + [property: Key(13)] double? Y, + [property: Key(14)] int? Accuracy); + +[CultDocument("mimir.eve_media_observation", "mimir.eve_media_observation.v1")] +[MessagePackObject] +public sealed record MimirEveMediaObservationDocument( + [property: Key(0)] + [property: CultName] + string ObservationId, + [property: Key(1)] string DeviceId, + [property: Key(2)] string StreamId, + [property: Key(3)] string Kind, + [property: Key(4)] long Sequence, + [property: Key(5)] long SensorTimestampNs, + [property: Key(6)] long ElapsedRealtimeNs, + [property: Key(7)] string WallClockUtc, + [property: Key(8)] string ClockDomainId, + [property: Key(9)] string Format, + [property: Key(10)] int? Width, + [property: Key(11)] int? Height, + [property: Key(12)] int? SampleRate, + [property: Key(13)] int? Channels, + [property: Key(14)] int? FrameCount, + [property: Key(15)] string PayloadEncoding, + [property: Key(16)] byte[] Payload); + public static class MimirCultMeshContractFactory { public static MimirBioacousticCodebookState CreateCodebookState( diff --git a/src/Mimir.VerseRecorder/Mimir.VerseRecorder.csproj b/src/Mimir.VerseRecorder/Mimir.VerseRecorder.csproj new file mode 100644 index 0000000..05d41e6 --- /dev/null +++ b/src/Mimir.VerseRecorder/Mimir.VerseRecorder.csproj @@ -0,0 +1,12 @@ + + + + Exe + net10.0 + Mimir.VerseRecorder + Mimir.VerseRecorder + enable + enable + + + diff --git a/src/Mimir.VerseRecorder/Program.cs b/src/Mimir.VerseRecorder/Program.cs new file mode 100644 index 0000000..28a274c --- /dev/null +++ b/src/Mimir.VerseRecorder/Program.cs @@ -0,0 +1,151 @@ +using System.Net.WebSockets; +using System.Text; +using System.Text.Json; + +var options = VerseRecorderOptions.Parse(args); +Directory.CreateDirectory(options.OutputDirectory); +var runId = DateTimeOffset.Now.ToString("yyyyMMdd-HHmmss"); +var runDirectory = Path.Combine(options.OutputDirectory, runId); +Directory.CreateDirectory(runDirectory); +var jsonlPath = Path.Combine(runDirectory, "observations.jsonl"); +var sessionPath = Path.Combine(runDirectory, "session.json"); + +await File.WriteAllTextAsync( + sessionPath, + JsonSerializer.Serialize( + new + { + kind = "mimir.verse_recorder_session.v1", + runId, + options.Url, + startedAt = DateTimeOffset.Now.ToString("O"), + options.Seconds, + observations = jsonlPath, + }, + new JsonSerializerOptions { WriteIndented = true })); + +Console.Error.WriteLine($"Mimir Verse recorder subscribing to {options.Url}"); +Console.Error.WriteLine($"Writing {jsonlPath}"); + +using var stopping = new CancellationTokenSource(); +if (options.Seconds > 0) +{ + stopping.CancelAfter(TimeSpan.FromSeconds(options.Seconds)); +} + +Console.CancelKeyPress += (_, eventArgs) => +{ + eventArgs.Cancel = true; + stopping.Cancel(); +}; + +var count = 0L; +var lastMeter = DateTimeOffset.UtcNow; +await using var output = new StreamWriter(new FileStream(jsonlPath, FileMode.Create, FileAccess.Write, FileShare.Read), Encoding.UTF8); + +while (!stopping.IsCancellationRequested) +{ + using var socket = new ClientWebSocket(); + try + { + await socket.ConnectAsync(options.Url, stopping.Token); + while (socket.State == WebSocketState.Open && !stopping.IsCancellationRequested) + { + var text = await ReceiveTextAsync(socket, stopping.Token); + if (text == null) + { + break; + } + + await output.WriteLineAsync(text); + count++; + if (DateTimeOffset.UtcNow - lastMeter >= TimeSpan.FromSeconds(options.MeterSeconds)) + { + await output.FlushAsync(); + Console.Error.WriteLine($"verse-recorder observations={count} path={jsonlPath}"); + lastMeter = DateTimeOffset.UtcNow; + } + } + } + catch (OperationCanceledException) when (stopping.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + Console.Error.WriteLine($"verse-recorder reconnect after {ex.GetType().Name}: {ex.Message}"); + try + { + await Task.Delay(TimeSpan.FromSeconds(options.ReconnectSeconds), stopping.Token); + } + catch (OperationCanceledException) + { + break; + } + } +} + +await output.FlushAsync(); +Console.Error.WriteLine($"verse-recorder complete observations={count} path={jsonlPath}"); + +static async Task ReceiveTextAsync(ClientWebSocket socket, CancellationToken stopping) +{ + var buffer = new byte[64 * 1024]; + using var memory = new MemoryStream(); + while (true) + { + var result = await socket.ReceiveAsync(buffer, stopping); + if (result.MessageType == WebSocketMessageType.Close) + { + return null; + } + + memory.Write(buffer, 0, result.Count); + if (result.EndOfMessage) + { + return Encoding.UTF8.GetString(memory.ToArray()); + } + } +} + +internal sealed record VerseRecorderOptions( + Uri Url, + string OutputDirectory, + double Seconds, + double ReconnectSeconds, + double MeterSeconds) +{ + public static VerseRecorderOptions Parse(string[] args) + { + var defaultOutput = Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.MyVideos), + "Mimir", + "VerseCaptures"); + return new VerseRecorderOptions( + new Uri(ParseString(args, "--url", "ws://127.0.0.1:8796/eve/periwinkle/subscribe")), + ParseString(args, "--out-dir", defaultOutput), + ParseDouble(args, "--seconds", 0.0), + ParseDouble(args, "--reconnect-seconds", 2.0), + ParseDouble(args, "--meter-seconds", 5.0)); + } + + private static string ParseString(IReadOnlyList args, string name, string fallback) + { + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase)) + { + return args[index + 1]; + } + } + + return fallback; + } + + private static double ParseDouble(IReadOnlyList args, string name, double fallback) + { + return double.TryParse(ParseString(args, name, ""), out var value) + ? value + : fallback; + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index ab7ff83..dda3f5f 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -124,3 +124,4 @@ {"ts": "2026-06-09T02:17:25+01:00", "type": "decision", "status": "ok", "note": "Mimir program composition authority moved out of OBS: Muninn hosts publish local stream capabilities/media; Mimir owns selected-stream subscription, scene graph, Eve operator surfaces, and program publication intent; Yggdrasil-facing publisher daemon consumes Mimir output; OBS is compatibility-only. Added typed mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 contracts and BufferSmoke proof coverage."} {"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."} {"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."} +{"ts": "2026-06-10T17:38:22+01:00", "type": "verification", "status": "ok", "note": "Restored and launched the narrow Nightwing Move tracking witness path. scripts/start-nightwing-move-tracking.ps1 started Mimir.EveSensorReceiver on Starfire port 8796, staged Nightwing Eye/Move Python tools, launched Nightwing worker PID 1131578 with --move-light move-usb=/dev/hidraw1:#35ff6c, and VerseRecorder C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures\\20260610-173549 recorded mimir.psmove_light_state.v1 HID success plus mimir.move_controller_observation_state.v1 blobs from both ps3-eye-0 and ps3-eye-1."} diff --git a/state/map.yaml b/state/map.yaml index 2c038ae..23e3099 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -85,6 +85,12 @@ accepted_design: are local tracking streams, Nightwing Moves are emitted by Muninn on Nightwing, Odin owns discovery/schema projection, and Mimir owns buffering and fusion. + - `scripts/start-nightwing-move-tracking.ps1` restores the live Nightwing + Eye/Move witness path: Starfire owns the `/eve/periwinkle` receiver and + recorder, Nightwing owns PS Move HID report `0x06` LED writes and PS3 Eye + blob extraction, and the stream currently emits `mimir.psmove_light_state.v1` + plus `mimir.move_controller_observation_state.v1` optical witness records + before later fusion into full tracking poses. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the diff --git a/tools/nightwing_typed_witness_publisher.py b/tools/nightwing_typed_witness_publisher.py new file mode 100644 index 0000000..8b4648f --- /dev/null +++ b/tools/nightwing_typed_witness_publisher.py @@ -0,0 +1,493 @@ +#!/usr/bin/env python3 +"""Publish Nightwing witness status as typed Mimir CultMesh observations. + +This sends MessagePack `mimir.eve_sensor_observation.v1` documents to +Mimir.EveSensorReceiver. It is deliberately narrow: Nightwing owns local sensor +presence/timing receipts, while Mimir and Odin own synchronization and UI +projection. +""" + +from __future__ import annotations + +import argparse +import base64 +import datetime as dt +import glob +import math +import os +import random +import socket +import struct +import subprocess +import time +import urllib.parse +from dataclasses import dataclass + +try: + import nw_eye_cap + import nw_move_hint +except Exception: + nw_eye_cap = None + nw_move_hint = None + + +def pack_nil() -> bytes: + return b"\xc0" + + +def pack_bool(value: bool) -> bytes: + return b"\xc3" if value else b"\xc2" + + +def pack_int(value: int) -> bytes: + if 0 <= value <= 0x7f: + return bytes([value]) + if -32 <= value < 0: + return bytes([0xe0 | (value + 32)]) + if -(1 << 31) <= value < (1 << 31): + return b"\xd2" + struct.pack(">i", value) + return b"\xd3" + struct.pack(">q", value) + + +def pack_float(value: float) -> bytes: + return b"\xcb" + struct.pack(">d", float(value)) + + +def pack_str(value: str) -> bytes: + data = value.encode("utf-8") + if len(data) <= 31: + return bytes([0xa0 | len(data)]) + data + if len(data) <= 0xff: + return b"\xd9" + bytes([len(data)]) + data + if len(data) <= 0xffff: + return b"\xda" + struct.pack(">H", len(data)) + data + return b"\xdb" + struct.pack(">I", len(data)) + data + + +def pack_array(items: list[bytes]) -> bytes: + if len(items) <= 15: + return bytes([0x90 | len(items)]) + b"".join(items) + if len(items) <= 0xffff: + return b"\xdc" + struct.pack(">H", len(items)) + b"".join(items) + return b"\xdd" + struct.pack(">I", len(items)) + b"".join(items) + + +def pack_double_array(values: list[float]) -> bytes: + return pack_array([pack_float(value) for value in values]) + + +def pack_optional_string(value: str | None) -> bytes: + return pack_nil() if value is None else pack_str(value) + + +def pack_optional_int(value: int | None) -> bytes: + return pack_nil() if value is None else pack_int(value) + + +def sensor_document( + observation_id: str, + device_id: str, + stream_id: str, + kind: str, + sequence: int, + values: list[float], + accuracy: int | None = None, +) -> bytes: + now_ns = time.time_ns() + wall = dt.datetime.now(dt.timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z") + return pack_array([ + pack_str(observation_id), + pack_str(device_id), + pack_str(stream_id), + pack_str(kind), + pack_int(sequence), + pack_int(now_ns), + pack_int(time.monotonic_ns()), + pack_str(wall), + pack_str("nightwing-monotonic"), + pack_double_array(values), + pack_optional_string(None), + pack_optional_int(None), + pack_nil(), + pack_nil(), + pack_optional_int(accuracy), + ]) + + +class WebSocket: + def __init__(self, url: str) -> None: + parsed = urllib.parse.urlparse(url) + if parsed.scheme != "ws": + raise ValueError("Only ws:// is supported") + self.host = parsed.hostname or "127.0.0.1" + self.port = parsed.port or 80 + self.path = parsed.path or "/" + self.sock: socket.socket | None = None + + def connect(self) -> None: + key = base64.b64encode(os.urandom(16)).decode("ascii") + sock = socket.create_connection((self.host, self.port), timeout=5) + request = ( + f"GET {self.path} HTTP/1.1\r\n" + f"Host: {self.host}:{self.port}\r\n" + "Upgrade: websocket\r\n" + "Connection: Upgrade\r\n" + f"Sec-WebSocket-Key: {key}\r\n" + "Sec-WebSocket-Version: 13\r\n" + "\r\n" + ) + sock.sendall(request.encode("ascii")) + response = b"" + while b"\r\n\r\n" not in response: + chunk = sock.recv(4096) + if not chunk: + raise ConnectionError("WebSocket handshake closed") + response += chunk + if b" 101 " not in response.splitlines()[0]: + raise ConnectionError(response.decode("utf-8", errors="replace").splitlines()[0]) + self.sock = sock + + def send_binary(self, payload: bytes) -> None: + self._send_frame(0x2, payload) + + def close(self) -> None: + if self.sock: + self.sock.close() + self.sock = None + + def _send_frame(self, opcode: int, payload: bytes) -> None: + if not self.sock: + raise ConnectionError("WebSocket not connected") + mask = random.randbytes(4) if hasattr(random, "randbytes") else os.urandom(4) + length = len(payload) + head = bytearray([0x80 | opcode]) + if length < 126: + head.append(0x80 | length) + elif length <= 0xffff: + head.extend([0x80 | 126]) + head.extend(struct.pack(">H", length)) + else: + head.extend([0x80 | 127]) + head.extend(struct.pack(">Q", length)) + masked = bytes(byte ^ mask[index % 4] for index, byte in enumerate(payload)) + self.sock.sendall(bytes(head) + mask + masked) + + +def video_devices() -> list[tuple[str, int, str]]: + devices: list[tuple[str, int, str]] = [] + for path in sorted(glob.glob("/sys/class/video4linux/video*")): + node = os.path.basename(path) + try: + index = int(node.replace("video", "")) + except ValueError: + continue + try: + with open(os.path.join(path, "name"), "r", encoding="utf-8", errors="replace") as handle: + name = handle.read().strip() + except OSError: + name = "video" + devices.append((node, index, name)) + return devices + + +@dataclass(frozen=True) +class MoveLight: + name: str + hidraw: str + rgb: tuple[int, int, int] + + +def parse_rgb(value: str) -> tuple[int, int, int]: + text = value.strip() + if text.startswith("#"): + text = text[1:] + if len(text) != 6: + raise ValueError(f"Expected #rrggbb, got {value!r}") + return (int(text[0:2], 16), int(text[2:4], 16), int(text[4:6], 16)) + + +def parse_move_light(value: str) -> MoveLight: + try: + name, rest = value.split("=", 1) + hidraw, rgb_text = rest.split(":", 1) + except ValueError as ex: + raise argparse.ArgumentTypeError("Move light must be name=/dev/hidrawN:#rrggbb") from ex + name = name.strip() + hidraw = hidraw.strip() + if not name or not hidraw: + raise argparse.ArgumentTypeError("Move light name and hidraw path are required") + try: + rgb = parse_rgb(rgb_text) + except ValueError as ex: + raise argparse.ArgumentTypeError(str(ex)) from ex + return MoveLight(name=name, hidraw=hidraw, rgb=rgb) + + +def write_move_light(move: MoveLight, rgb: tuple[int, int, int] | None = None) -> bool: + r, g, b = rgb if rgb is not None else move.rgb + try: + with open(move.hidraw, "wb", buffering=0) as device: + device.write(bytes([0x06, 0, r, g, b, 0, 0, 0, 0])) + return True + except OSError: + return False + + +def move_light_observations(args: argparse.Namespace, sequence: int) -> list[bytes]: + docs: list[bytes] = [] + for move in args.move_light: + ok = write_move_light(move) + r, g, b = move.rgb + docs.append(sensor_document( + observation_id=f"nightwing:{move.name}:move-light:{sequence}", + device_id="nightwing", + stream_id=f"nightwing-{move.name}", + kind="mimir.psmove_light_state.v1", + sequence=sequence, + values=[1.0 if ok else 0.0, float(r), float(g), float(b)], + accuracy=3 if ok else 0, + )) + return docs + + +def observations(sequence: int) -> list[bytes]: + docs: list[bytes] = [] + for node, index, name in video_devices(): + present = 1.0 if os.path.exists(f"/dev/{node}") else 0.0 + is_eye = 1.0 if "gspca" in name.lower() or "playstation" in name.lower() else 0.0 + docs.append(sensor_document( + observation_id=f"nightwing:{node}:{sequence}", + device_id="nightwing", + stream_id=f"nightwing-{node}", + kind="camera-device-status", + sequence=sequence, + values=[present, float(index), is_eye], + accuracy=3 if present else 0, + )) + docs.append(sensor_document( + observation_id=f"nightwing:leap-tracking:{sequence}", + device_id="nightwing", + stream_id="nightwing-leap-tracking", + kind="leap-tracking-status", + sequence=sequence, + values=[0.0], + accuracy=0, + )) + return docs + + +def builtin_camera_observations(args: argparse.Namespace, sequence: int) -> list[bytes]: + docs: list[bytes] = [] + for node, index, name in video_devices(): + lname = name.lower() + is_eye = "gspca" in lname or "playstation" in lname or "eye" in lname + if is_eye: + continue + docs.append(sensor_document( + observation_id=f"nightwing:{node}:builtin-camera:{sequence}", + device_id="nightwing", + stream_id=f"nightwing-{node}", + kind="mimir.nightwing_builtin_camera_witness.v1", + sequence=sequence, + values=[1.0, float(index), float(len(name)), float(args.interval)], + accuracy=2, + )) + if not docs: + docs.append(sensor_document( + observation_id=f"nightwing:builtin-camera:missing:{sequence}", + device_id="nightwing", + stream_id="nightwing-builtin-camera", + kind="mimir.nightwing_builtin_camera_witness.v1", + sequence=sequence, + values=[0.0], + accuracy=0, + )) + return docs + + +def builtin_mic_observation(args: argparse.Namespace, sequence: int) -> bytes: + values = [0.0, 0.0, 0.0, float(args.mic_sample_rate)] + accuracy = 0 + try: + result = subprocess.run( + [ + "timeout", + str(max(0.05, args.mic_window_seconds)), + "arecord", + "-D", + args.mic_device, + "-f", + "S16_LE", + "-r", + str(args.mic_sample_rate), + "-c", + "1", + "-t", + "raw", + "-q", + ], + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + check=False, + timeout=max(0.2, args.mic_window_seconds + 0.2), + ) + data = result.stdout + count = len(data) // 2 + if count > 0: + samples = struct.unpack("<" + "h" * count, data[:count * 2]) + rms = math.sqrt(sum(float(sample) * float(sample) for sample in samples) / count) / 32768.0 + peak = max(abs(sample) for sample in samples) / 32768.0 + values = [1.0, rms, peak, float(args.mic_sample_rate), float(count)] + accuracy = 2 if rms > 0.0 else 1 + except Exception: + values = [0.0, 0.0, 0.0, float(args.mic_sample_rate)] + accuracy = 0 + return sensor_document( + observation_id=f"nightwing:builtin-mic:{sequence}", + device_id="nightwing", + stream_id="nightwing-builtin-mic", + kind="mimir.nightwing_builtin_mic_witness.v1", + sequence=sequence, + values=values, + accuracy=accuracy, + ) + + +def eye_tracking_observations(args: argparse.Namespace, sequence: int) -> list[bytes]: + if nw_eye_cap is None or nw_move_hint is None: + return [] + docs: list[bytes] = [] + devices = [part for part in args.eye_devices.split(",") if part] + for eye_index, device in enumerate(devices): + raw_path = None + source_id = f"ps3-eye-{eye_index}" + stats_path = f"/tmp/mimir-mvp-{source_id}.json" + cap_args = argparse.Namespace( + device=device, + width=args.eye_width, + height=args.eye_height, + format="YUYV", + fps=args.eye_fps, + seconds=args.eye_window_seconds, + buffers=8, + out=stats_path, + stdout=False, + ) + try: + stats = nw_eye_cap.capture(cap_args) + raw_path = stats["frames_path"] + width = int(stats["width"]) + height = int(stats["height"]) + frame_bytes = width * height + with open(raw_path, "rb") as handle: + raw = handle.read() + frame_count = min(len(raw) // frame_bytes, len(stats.get("timestamps_s", []))) + stride = max(1, int(args.tracking_stride)) + for frame_index in range(0, frame_count, stride): + frame = raw[frame_index * frame_bytes:(frame_index + 1) * frame_bytes] + blob = nw_move_hint.best_blob(frame, width, height) + if blob is None or float(blob["confidence"]) < args.min_confidence: + continue + values = [ + float(blob["clip_x"]), + float(blob["clip_y"]), + float(blob["area_px"]), + float(blob["radius_px"]), + float(blob["mean_luma"]), + float(blob["peak_luma"]), + float(blob["confidence"]), + float(frame_index), + float(width), + float(height), + ] + docs.append(sensor_document( + observation_id=f"nightwing:{source_id}:move-sphere:{sequence}:{frame_index}", + device_id="nightwing", + stream_id=source_id, + kind="mimir.move_controller_observation_state.v1", + sequence=sequence * 100000 + frame_index, + values=values, + accuracy=3, + )) + except Exception as ex: + docs.append(sensor_document( + observation_id=f"nightwing:{source_id}:tracking-error:{sequence}", + device_id="nightwing", + stream_id=source_id, + kind=f"tracking-error:{type(ex).__name__}", + sequence=sequence, + values=[0.0], + accuracy=0, + )) + finally: + try: + if raw_path: + os.unlink(raw_path) + except OSError: + pass + return docs + + +def run(args: argparse.Namespace) -> None: + sequence = 0 + while True: + ws = WebSocket(args.url) + try: + ws.connect() + while True: + for payload in observations(sequence): + ws.send_binary(payload) + if args.track_builtin_camera: + for payload in builtin_camera_observations(args, sequence): + ws.send_binary(payload) + if args.track_builtin_mic: + ws.send_binary(builtin_mic_observation(args, sequence)) + for payload in move_light_observations(args, sequence): + ws.send_binary(payload) + if args.track_eyes: + for payload in eye_tracking_observations(args, sequence): + ws.send_binary(payload) + sequence += 1 + if args.once: + return + time.sleep(max(0.02, args.interval)) + finally: + ws.close() + if args.once: + return + time.sleep(args.reconnect_seconds) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--url", default="ws://192.168.1.66:8796/eve/periwinkle") + parser.add_argument("--interval", type=float, default=0.25) + parser.add_argument("--reconnect-seconds", type=float, default=2.0) + parser.add_argument("--once", action="store_true") + parser.add_argument("--track-eyes", action="store_true", help="Publish compact Nightwing-local PS3 Eye Move-sphere observations.") + parser.add_argument("--track-builtin-camera", action="store_true", help="Publish compact status for Nightwing-local non-Eye camera devices.") + parser.add_argument("--track-builtin-mic", action="store_true", help="Publish compact ALSA microphone level observations.") + parser.add_argument( + "--move-light", + action="append", + type=parse_move_light, + default=[], + help="Keep a Nightwing-local PS Move sphere lit: name=/dev/hidrawN:#rrggbb.", + ) + parser.add_argument("--eye-devices", default="/dev/video2,/dev/video3") + parser.add_argument("--eye-width", type=int, default=320) + parser.add_argument("--eye-height", type=int, default=240) + parser.add_argument("--eye-fps", type=int, default=187) + parser.add_argument("--eye-window-seconds", type=float, default=0.12) + parser.add_argument("--tracking-stride", type=int, default=6) + parser.add_argument("--min-confidence", type=float, default=0.25) + parser.add_argument("--mic-device", default="default") + parser.add_argument("--mic-sample-rate", type=int, default=16000) + parser.add_argument("--mic-window-seconds", type=float, default=0.08) + args = parser.parse_args() + run(args) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/nw_eye_cap.py b/tools/nw_eye_cap.py new file mode 100644 index 0000000..dbc44d9 --- /dev/null +++ b/tools/nw_eye_cap.py @@ -0,0 +1,280 @@ +#!/usr/bin/env python3 +"""Capture raw luma frames from Nightwing V4L2 cameras for sync receipts.""" + +from __future__ import annotations + +import argparse +import ctypes +import ctypes.util +import fcntl +import json +import mmap +import os +import select +import sys +import time +from pathlib import Path + + +V4L2_BUF_TYPE_VIDEO_CAPTURE = 1 +V4L2_FIELD_NONE = 1 +V4L2_MEMORY_MMAP = 1 +VIDIOC_G_FMT = 0xC0D05604 +VIDIOC_S_FMT = 0xC0D05605 +VIDIOC_REQBUFS = 0xC0145608 +VIDIOC_QUERYBUF = 0xC0585609 +VIDIOC_QBUF = 0xC058560F +VIDIOC_DQBUF = 0xC0585611 +VIDIOC_STREAMON = 0x40045612 +VIDIOC_STREAMOFF = 0x40045613 +VIDIOC_G_PARM = 0xC0CC5615 +VIDIOC_S_PARM = 0xC0CC5616 + +_libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True) +_libc.ioctl.argtypes = [ctypes.c_int, ctypes.c_ulong, ctypes.c_void_p] +_libc.ioctl.restype = ctypes.c_int + + +def fourcc(text: str) -> int: + raw = text.encode("ascii") + return raw[0] | (raw[1] << 8) | (raw[2] << 16) | (raw[3] << 24) + + +class V4L2PixFormat(ctypes.Structure): + _fields_ = [ + ("width", ctypes.c_uint32), + ("height", ctypes.c_uint32), + ("pixelformat", ctypes.c_uint32), + ("field", ctypes.c_uint32), + ("bytesperline", ctypes.c_uint32), + ("sizeimage", ctypes.c_uint32), + ("colorspace", ctypes.c_uint32), + ("priv", ctypes.c_uint32), + ("flags", ctypes.c_uint32), + ("ycbcr_enc", ctypes.c_uint32), + ("quantization", ctypes.c_uint32), + ("xfer_func", ctypes.c_uint32), + ] + + +class V4L2FormatUnion(ctypes.Union): + _fields_ = [("pix", V4L2PixFormat), ("raw", ctypes.c_uint8 * 200), ("_align", ctypes.c_uint64)] + + +class V4L2Format(ctypes.Structure): + _fields_ = [("type", ctypes.c_uint32), ("fmt", V4L2FormatUnion)] + + +class V4L2RequestBuffers(ctypes.Structure): + _fields_ = [ + ("count", ctypes.c_uint32), + ("type", ctypes.c_uint32), + ("memory", ctypes.c_uint32), + ("capabilities", ctypes.c_uint32), + ("flags", ctypes.c_uint8), + ("reserved", ctypes.c_uint8 * 3), + ] + + +class V4L2Fract(ctypes.Structure): + _fields_ = [("numerator", ctypes.c_uint32), ("denominator", ctypes.c_uint32)] + + +class V4L2CaptureParm(ctypes.Structure): + _fields_ = [ + ("capability", ctypes.c_uint32), + ("capturemode", ctypes.c_uint32), + ("timeperframe", V4L2Fract), + ("extendedmode", ctypes.c_uint32), + ("readbuffers", ctypes.c_uint32), + ("reserved", ctypes.c_uint32 * 4), + ] + + +class V4L2StreamParmUnion(ctypes.Union): + _fields_ = [("capture", V4L2CaptureParm), ("raw", ctypes.c_uint8 * 200)] + + +class V4L2StreamParm(ctypes.Structure): + _fields_ = [("type", ctypes.c_uint32), ("parm", V4L2StreamParmUnion)] + + +class Timeval(ctypes.Structure): + _fields_ = [("tv_sec", ctypes.c_long), ("tv_usec", ctypes.c_long)] + + +class V4L2Timecode(ctypes.Structure): + _fields_ = [ + ("type", ctypes.c_uint32), + ("flags", ctypes.c_uint32), + ("frames", ctypes.c_uint8), + ("seconds", ctypes.c_uint8), + ("minutes", ctypes.c_uint8), + ("hours", ctypes.c_uint8), + ("userbits", ctypes.c_uint8 * 4), + ] + + +class V4L2BufferUnion(ctypes.Union): + _fields_ = [("offset", ctypes.c_uint32), ("userptr", ctypes.c_ulong), ("planes", ctypes.c_void_p), ("fd", ctypes.c_int32)] + + +class V4L2Buffer(ctypes.Structure): + _fields_ = [ + ("index", ctypes.c_uint32), + ("type", ctypes.c_uint32), + ("bytesused", ctypes.c_uint32), + ("flags", ctypes.c_uint32), + ("field", ctypes.c_uint32), + ("timestamp", Timeval), + ("timecode", V4L2Timecode), + ("sequence", ctypes.c_uint32), + ("memory", ctypes.c_uint32), + ("m", V4L2BufferUnion), + ("length", ctypes.c_uint32), + ("reserved2", ctypes.c_uint32), + ("request_fd", ctypes.c_int32), + ] + + +def xioctl(fd: int, request: int, obj: ctypes.Structure) -> None: + if _libc.ioctl(fd, request, ctypes.byref(obj)) != 0: + error = ctypes.get_errno() + raise OSError(error, os.strerror(error)) + + +def set_frame_interval(fd: int, fps: int) -> None: + parm = V4L2StreamParm() + parm.type = V4L2_BUF_TYPE_VIDEO_CAPTURE + try: + xioctl(fd, VIDIOC_G_PARM, parm) + except OSError: + pass + parm.parm.capture.timeperframe.numerator = 1 + parm.parm.capture.timeperframe.denominator = fps + xioctl(fd, VIDIOC_S_PARM, parm) + + +def yuyv_to_luma(raw: bytes, width: int, height: int, stride: int) -> bytes: + rows = [] + for row in range(height): + start = row * stride + line = raw[start:start + width * 2] + rows.append(line[0::2]) + return b"".join(rows) + + +def capture(args: argparse.Namespace) -> dict: + fd = os.open(args.device, os.O_RDWR | os.O_NONBLOCK) + maps: list[mmap.mmap] = [] + stream_stdout = bool(getattr(args, "stdout", False)) + out = Path(args.out) if args.out else None + if out is not None: + out.parent.mkdir(parents=True, exist_ok=True) + frames_path = out.with_suffix(".y8") if out is not None else None + try: + set_frame_interval(fd, args.fps) + fmt = V4L2Format() + fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE + xioctl(fd, VIDIOC_G_FMT, fmt) + fmt.fmt.pix.width = args.width + fmt.fmt.pix.height = args.height + fmt.fmt.pix.pixelformat = fourcc(args.format) + fmt.fmt.pix.field = V4L2_FIELD_NONE + xioctl(fd, VIDIOC_S_FMT, fmt) + actual = fmt.fmt.pix + + req = V4L2RequestBuffers() + req.count = args.buffers + req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE + req.memory = V4L2_MEMORY_MMAP + xioctl(fd, VIDIOC_REQBUFS, req) + for index in range(req.count): + buf = V4L2Buffer() + buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE + buf.memory = V4L2_MEMORY_MMAP + buf.index = index + xioctl(fd, VIDIOC_QUERYBUF, buf) + maps.append(mmap.mmap(fd, buf.length, mmap.MAP_SHARED, mmap.PROT_READ | mmap.PROT_WRITE, offset=buf.m.offset)) + xioctl(fd, VIDIOC_QBUF, buf) + + stream_type = ctypes.c_int(V4L2_BUF_TYPE_VIDEO_CAPTURE) + fcntl.ioctl(fd, VIDIOC_STREAMON, stream_type) + start = time.monotonic() + deadline = start + args.seconds + timestamps = [] + sequences = [] + frames = sys.stdout.buffer if stream_stdout else frames_path.open("wb") # type: ignore[union-attr] + try: + while time.monotonic() < deadline: + ready, _, _ = select.select([fd], [], [], 0.25) + if not ready: + continue + buf = V4L2Buffer() + buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE + buf.memory = V4L2_MEMORY_MMAP + try: + xioctl(fd, VIDIOC_DQBUF, buf) + except OSError: + continue + raw = maps[buf.index][:buf.bytesused] + frames.write(yuyv_to_luma(raw, int(actual.width), int(actual.height), int(actual.bytesperline))) + timestamps.append(float(buf.timestamp.tv_sec) + float(buf.timestamp.tv_usec) / 1_000_000.0) + sequences.append(int(buf.sequence)) + xioctl(fd, VIDIOC_QBUF, buf) + finally: + if not stream_stdout: + frames.close() + try: + fcntl.ioctl(fd, VIDIOC_STREAMOFF, stream_type) + except OSError: + pass + first_ts = timestamps[0] if timestamps else None + rel = [(ts - first_ts) for ts in timestamps] if first_ts is not None else [] + gap = max(0, (sequences[-1] - sequences[0] + 1) - len(sequences)) if len(sequences) >= 2 else 0 + stats = { + "device": args.device, + "frames_path": str(frames_path) if frames_path is not None else "stdout", + "width": int(actual.width), + "height": int(actual.height), + "stride": int(actual.bytesperline), + "format": args.format, + "frames": len(timestamps), + "timestamps_s": rel, + "first_device_timestamp_s": first_ts, + "sequence_gap": gap, + "duration_s": (rel[-1] - rel[0]) if len(rel) >= 2 else 0.0, + } + if out is not None: + out.write_text(json.dumps(stats, indent=2, sort_keys=True), encoding="utf-8") + else: + print(json.dumps(stats, sort_keys=True), file=sys.stderr) + return stats + finally: + for mapping in maps: + mapping.close() + os.close(fd) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--device", required=True) + parser.add_argument("--width", type=int, default=320) + parser.add_argument("--height", type=int, default=240) + parser.add_argument("--format", default="YUYV") + parser.add_argument("--fps", type=int, default=187) + parser.add_argument("--seconds", type=float, default=8.0) + parser.add_argument("--buffers", type=int, default=8) + parser.add_argument("--out", default=None) + parser.add_argument("--stdout", action="store_true", help="Stream raw luma frames to stdout and write stats to stderr.") + args = parser.parse_args() + if not args.stdout and not args.out: + parser.error("--out is required unless --stdout is set") + stats = capture(args) + if not args.stdout: + print(json.dumps(stats, sort_keys=True)) + return 0 if stats["frames"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/nw_move_hint.py b/tools/nw_move_hint.py new file mode 100644 index 0000000..2b5c9fb --- /dev/null +++ b/tools/nw_move_hint.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Nightwing-local PS Eye / PS Move localization hint extractor. + +Stdlib-only on purpose. Nightwing owns Eye frames and Move observations; the +wire format back to Starfire is compact witness state, not raw media. +""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path + +import nw_eye_cap + + +def percentile(values: list[float], q: float) -> float: + if not values: + return 0.0 + ordered = sorted(values) + index = max(0, min(len(ordered) - 1, int(round((len(ordered) - 1) * q)))) + return float(ordered[index]) + + +def best_blob(frame: bytes, width: int, height: int) -> dict | None: + sampled = [frame[index] for index in range(0, len(frame), max(1, len(frame) // 4000))] + threshold = max(40, int(percentile([float(v) for v in sampled], 0.995))) + seen = bytearray(width * height) + best: dict | None = None + for y in range(height): + row = y * width + for x in range(width): + start = row + x + if seen[start] or frame[start] < threshold: + continue + stack = [start] + seen[start] = 1 + count = 0 + sx = 0.0 + sy = 0.0 + sw = 0.0 + total = 0.0 + peak = 0 + while stack: + index = stack.pop() + yy, xx = divmod(index, width) + value = int(frame[index]) + weight = max(1.0, float(value - threshold + 1)) + count += 1 + sx += xx * weight + sy += yy * weight + sw += weight + total += value + peak = max(peak, value) + for nx, ny in ((xx - 1, yy), (xx + 1, yy), (xx, yy - 1), (xx, yy + 1)): + if 0 <= nx < width and 0 <= ny < height: + ni = ny * width + nx + if not seen[ni] and frame[ni] >= threshold: + seen[ni] = 1 + stack.append(ni) + if count < 3 or sw <= 0.0: + continue + cx = sx / sw + cy = sy / sw + mean_luma = total / count + confidence = min(1.0, (mean_luma / 255.0) * min(1.0, count / 80.0) * 1.6) + blob = { + "x_px": cx, + "y_px": cy, + "clip_x": (cx / max(1, width - 1)) * 2.0 - 1.0, + "clip_y": 1.0 - (cy / max(1, height - 1)) * 2.0, + "area_px": count, + "radius_px": math.sqrt(count / math.pi), + "mean_luma": mean_luma, + "peak_luma": peak, + "confidence": confidence, + } + if best is None or (blob["confidence"], blob["area_px"], blob["peak_luma"]) > (best["confidence"], best["area_px"], best["peak_luma"]): + best = blob + return best + + +def analyze(raw_path: Path, stats: dict, events: list[dict], moves: list[str], pulse_seconds: float, source_id: str) -> dict: + width = int(stats["width"]) + height = int(stats["height"]) + frame_bytes = width * height + raw = raw_path.read_bytes() + count = min(len(raw) // frame_bytes, len(stats.get("timestamps_s", []))) + timestamps = [float(value) for value in stats.get("timestamps_s", [])[:count]] + observations = [] + for event in events: + event_index = int(event["index"]) + expected = float(event["offset"]) + move_id = moves[event_index % len(moves)] if moves else "move" + best: dict | None = None + for frame_index, timestamp in enumerate(timestamps): + if timestamp < expected - 0.18 or timestamp > expected + max(0.25, pulse_seconds + 0.18): + continue + frame = raw[frame_index * frame_bytes:(frame_index + 1) * frame_bytes] + blob = best_blob(frame, width, height) + if blob is None: + continue + blob.update({ + "event_index": event_index, + "symbol": int(event["symbol"]), + "controller_id": move_id, + "source_id": source_id, + "observed_time_s": timestamp, + "schedule_offset_s": expected, + "schedule_residual_ms": (timestamp - expected) * 1000.0, + }) + if best is None or blob["confidence"] > best["confidence"]: + best = blob + if best is not None: + observations.append(best) + return { + "kind": "mimir.move_controller_observation_state.v1", + "source_id": source_id, + "width": width, + "height": height, + "frame_count": count, + "first_device_timestamp_s": stats.get("first_device_timestamp_s"), + "duration_s": stats.get("duration_s"), + "sequence_gap": stats.get("sequence_gap"), + "observations": observations, + "observation_count": len(observations), + "stable_event_count": len([item for item in observations if float(item["confidence"]) >= 0.25]), + "mean_confidence": sum(float(item["confidence"]) for item in observations) / len(observations) if observations else 0.0, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--device", required=True) + parser.add_argument("--source-id", required=True) + parser.add_argument("--width", type=int, default=320) + parser.add_argument("--height", type=int, default=240) + parser.add_argument("--fps", type=int, default=187) + parser.add_argument("--seconds", type=float, default=8.0) + parser.add_argument("--events-json", required=True) + parser.add_argument("--moves", default="move-a,move-b") + parser.add_argument("--pulse-ms", type=float, default=95.0) + parser.add_argument("--out", required=True) + args = parser.parse_args() + + out = Path(args.out) + raw_stats_path = out.with_suffix(".capture.json") + cap_args = argparse.Namespace( + device=args.device, + width=args.width, + height=args.height, + format="YUYV", + fps=args.fps, + seconds=args.seconds, + buffers=8, + out=str(raw_stats_path), + ) + stats = nw_eye_cap.capture(cap_args) + events = json.loads(Path(args.events_json).read_text(encoding="utf-8")) + moves = [part for part in args.moves.split(",") if part] + result = analyze(Path(stats["frames_path"]), stats, events, moves, args.pulse_ms * 0.001, args.source_id) + out.write_text(json.dumps(result, indent=2, sort_keys=True), encoding="utf-8") + Path(stats["frames_path"]).unlink(missing_ok=True) + print(json.dumps(result, sort_keys=True)) + return 0 if result["observation_count"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) From 601367b7546544b60ac818a7c183ffc6573bcd52 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 17:42:28 +0100 Subject: [PATCH 009/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 65126 -> 65213 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index a8296794ff4e33359d8d75a7e57cce2b1e1fd940..a7031b0068f8fec2052a16e1e2adee049bb0deea 100644 GIT binary patch delta 279 zcmaF%hk5T`<_+I@dCaT~O|6W~^o%Txqb4$nPL|;7go~LN7=gq#AL7$9KnNN^1O@G{ z=^zA+A-Xwo6d89+*2)M_OwG zLfADgrP#4#hmnDinXZAEuAxDQp_!G5g_V(oo~5B_)a1Y%Sr$v6z~-VHUPf*tH&1@R sD~@o5i8;i*xsplC;#cM=YnZ8F`LZDC%$>kgnhXPb>>D<09)Qta{vGU delta 174 zcmdn{m-*Qr<_+I@CkssAWHvU7n#d?RS%R+<%r!9raW^00(=&kZ!P1)r?XT&8`KE?o z Date: Wed, 10 Jun 2026 18:12:32 +0100 Subject: [PATCH 010/135] Add Starfire Move light owner --- Mimir.slnx | 1 + docs/implementation-plan.md | 4 + notes/current-system-map.md | 5 + scripts/start-starfire-move-light.ps1 | 61 + .../Mimir.PsMoveProbe.csproj | 10 + src/Mimir.PsMoveProbe/Program.cs | 1236 +++++++++++++++++ state/evidence.jsonl | 1 + 7 files changed, 1318 insertions(+) create mode 100644 scripts/start-starfire-move-light.ps1 create mode 100644 src/Mimir.PsMoveProbe/Mimir.PsMoveProbe.csproj create mode 100644 src/Mimir.PsMoveProbe/Program.cs diff --git a/Mimir.slnx b/Mimir.slnx index 901ce47..dbd4f1b 100644 --- a/Mimir.slnx +++ b/Mimir.slnx @@ -5,5 +5,6 @@ + diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index ea070b1..db936a5 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -46,6 +46,10 @@ a named invariant that the native runtime cannot protect yet. `mimir.move_controller_observation_state.v1` blob observations from both PS3 Eyes. The blob stream is optical witness evidence for later pose fusion, not the final 6DoF pose owner. +- `scripts/start-starfire-move-light.ps1` is the Starfire-local USB Move light + owner. It launches `Mimir.PsMoveProbe` in steady hold mode so the locally + attached `VID_054C/PID_03D5` Move sphere stays lit for calibration/tracking + receipts. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 139c133..22b0d7a 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -131,6 +131,11 @@ USB Move sphere lit with PS Move report `0x06`, emits Eyes. Those Eye/Move observations are the live optical witness feeding the later pose/fusion owner; they are not themselves the final 6DoF pose authority. +Starfire-local Move illumination uses `scripts/start-starfire-move-light.ps1`. +That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01 +output collection and refreshes the PS Move LED report so the USB-attached Move +stays visible as a tracking/calibration marker. + ## Program Composition [[docs/mimir-program-composition|Mimir Program Composition]] is the live diff --git a/scripts/start-starfire-move-light.ps1 b/scripts/start-starfire-move-light.ps1 new file mode 100644 index 0000000..55e83be --- /dev/null +++ b/scripts/start-starfire-move-light.ps1 @@ -0,0 +1,61 @@ +param( + [string]$Rgb = "#35ff6c", + [int]$RefreshMs = 250, + [double]$HoldSeconds = 0, + [switch]$TurnOffOnExit, + [switch]$DryRun +) + +$ErrorActionPreference = "Stop" +$repo = Split-Path -Parent $PSScriptRoot +$runId = Get-Date -Format "yyyyMMdd-HHmmss" +$runDir = Join-Path $repo "artifacts\runtime\starfire-move-light-$runId" +$probe = Join-Path $repo "src\Mimir.PsMoveProbe\bin\Release\net10.0\Mimir.PsMoveProbe.exe" +$eventLog = Join-Path $runDir "move-light-events.jsonl" + +New-Item -ItemType Directory -Force -Path $runDir | Out-Null + +if (-not (Test-Path $probe)) { + dotnet build (Join-Path $repo "src\Mimir.PsMoveProbe\Mimir.PsMoveProbe.csproj") -c Release -p:UseSharedCompilation=false +} + +$args = @( + "--rgb", $Rgb, + "--hold-seconds", "$HoldSeconds", + "--refresh-ms", "$RefreshMs", + "--event-log", $eventLog +) +if ($TurnOffOnExit) { + $args += "--turn-off-on-exit" +} + +if ($DryRun) { + Write-Host "DRY starfire-move-light: $probe $($args -join ' ')" + return +} + +$stdout = Join-Path $runDir "starfire-move-light.out.log" +$stderr = Join-Path $runDir "starfire-move-light.err.log" +$process = Start-Process ` + -FilePath $probe ` + -ArgumentList $args ` + -WorkingDirectory $repo ` + -WindowStyle Hidden ` + -RedirectStandardOutput $stdout ` + -RedirectStandardError $stderr ` + -PassThru + +Set-Content -Path (Join-Path $runDir "starfire-move-light.pid") -Value $process.Id +[ordered]@{ + kind = "mimir.starfire_move_light_supervisor.v1" + runId = $runId + runDir = $runDir + rgb = $Rgb + refreshMs = $RefreshMs + holdSeconds = $HoldSeconds + pid = $process.Id + eventLog = $eventLog + startedAt = (Get-Date).ToString("o") +} | ConvertTo-Json -Depth 4 | Set-Content -Path (Join-Path $runDir "supervisor.json") + +Write-Host "starfire-move-light pid=$($process.Id) runDir=$runDir eventLog=$eventLog" diff --git a/src/Mimir.PsMoveProbe/Mimir.PsMoveProbe.csproj b/src/Mimir.PsMoveProbe/Mimir.PsMoveProbe.csproj new file mode 100644 index 0000000..ed9781c --- /dev/null +++ b/src/Mimir.PsMoveProbe/Mimir.PsMoveProbe.csproj @@ -0,0 +1,10 @@ + + + + Exe + net10.0 + enable + enable + + + diff --git a/src/Mimir.PsMoveProbe/Program.cs b/src/Mimir.PsMoveProbe/Program.cs new file mode 100644 index 0000000..1bd4f30 --- /dev/null +++ b/src/Mimir.PsMoveProbe/Program.cs @@ -0,0 +1,1236 @@ +using System.ComponentModel; +using System.Diagnostics; +using System.Runtime.InteropServices; +using Microsoft.Win32.SafeHandles; + +const ushort SonyVid = 0x054C; +const ushort PsMovePid = 0x03D5; + +var readReports = args.Any(arg => string.Equals(arg, "--read", StringComparison.OrdinalIgnoreCase)); +var pairHost = TryGetOption(args, "--pair-host"); +var rgb = TryGetOption(args, "--rgb"); +var pulseMs = TryGetIntOption(args, "--pulse-ms", 0); +var pulseCount = TryGetIntOption(args, "--pulse-count", 1); +var intervalMs = TryGetIntOption(args, "--interval-ms", 500); +var holdSeconds = TryGetDoubleOption(args, "--hold-seconds", -1.0); +var refreshMs = TryGetIntOption(args, "--refresh-ms", 250); +var eventLog = TryGetOption(args, "--event-log"); +var pairedChirpTrain = args.Any(arg => string.Equals(arg, "--paired-chirp-train", StringComparison.OrdinalIgnoreCase)); +var outputAudio = TryGetOption(args, "--output-audio"); +var eventCount = TryGetIntOption(args, "--event-count", pulseCount); +var startDelayMs = TryGetIntOption(args, "--start-delay-ms", 150); +var chirpMs = TryGetIntOption(args, "--chirp-ms", Math.Max(1, pulseMs)); +var sampleRate = TryGetIntOption(args, "--sample-rate", 48_000); +var startHz = TryGetDoubleOption(args, "--start-hz", 2400.0); +var endHz = TryGetDoubleOption(args, "--end-hz", 7200.0); +var gain = TryGetDoubleOption(args, "--gain", 0.12); +var renderDevice = TryGetOption(args, "--render-device"); +var wasapiExe = TryGetOption(args, "--wasapi-exe"); +var noAudioEmit = args.Any(arg => string.Equals(arg, "--no-audio-emit", StringComparison.OrdinalIgnoreCase)); +var showAll = args.Any(arg => string.Equals(arg, "--all", StringComparison.OrdinalIgnoreCase)); +if (showAll) +{ + foreach (var path in WindowsHid.EnumeratePaths()) + { + Console.WriteLine(path); + } + + return 0; +} + +var devices = WindowsHid.Enumerate() + .Where(device => device.VendorId == SonyVid && device.ProductId == PsMovePid) + .OrderBy(device => device.Path, StringComparer.OrdinalIgnoreCase) + .ToList(); + +Console.WriteLine("Mimir PS Move USB probe"); +Console.WriteLine($"target vid=0x{SonyVid:X4} pid=0x{PsMovePid:X4}"); +Console.WriteLine($"collections={devices.Count}"); +Console.WriteLine(); + +if (devices.Count == 0) +{ + Console.WriteLine("No PS Move HID collections are visible. Check the USB cable and Windows Device Manager."); + return 1; +} + +if (pairedChirpTrain) +{ + var move = devices.FirstOrDefault(static device => + device.Caps.OutputReportByteLength > 0 && + device.Path.IndexOf("&col01#", StringComparison.OrdinalIgnoreCase) >= 0); + if (move == null) + { + Console.WriteLine("No PS Move col01 output collection is available for paired chirp train."); + return 1; + } + + var audioPath = string.IsNullOrWhiteSpace(outputAudio) + ? Path.Combine("artifacts", "runtime", $"starfire-usb-move-chirp-{DateTimeOffset.UtcNow:yyyyMMdd-HHmmss}.f32") + : outputAudio; + if (string.IsNullOrWhiteSpace(rgb) || !WindowsHid.TryParseRgb(rgb, out var r, out var g, out var b)) + { + Console.WriteLine("Paired chirp train requires --rgb #rrggbb or --rgb r,g,b."); + return 1; + } + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(audioPath)) ?? "."); + if (!string.IsNullOrWhiteSpace(eventLog)) + { + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(eventLog)) ?? "."); + } + + var events = PairedChirpTrain.RenderAudio( + audioPath, + Math.Max(1, eventCount), + TimeSpan.FromMilliseconds(Math.Max(0, startDelayMs)), + TimeSpan.FromMilliseconds(Math.Max(1, intervalMs)), + TimeSpan.FromMilliseconds(Math.Max(1, chirpMs)), + sampleRate, + startHz, + endHz, + gain); + PairedChirpTrain.AppendScheduleEvents(eventLog, events, audioPath, sampleRate, gain); + var audioTask = noAudioEmit + ? Task.FromResult("skipped: --no-audio-emit") + : PairedChirpTrain.EmitAudioAsync(audioPath, sampleRate, gain, renderDevice, wasapiExe, eventLog); + var ledTask = WindowsHid.TryWriteScheduledLedTrainAsync( + move, + events, + r, + g, + b, + TimeSpan.FromMilliseconds(Math.Max(1, pulseMs)), + eventLog); + var result = await ledTask.ConfigureAwait(false); + var audioResult = await audioTask.ConfigureAwait(false); + Console.WriteLine( + $"pairedChirpTrain audio={audioPath} events={events.Count} sampleRate={sampleRate} startDelayMs={startDelayMs} chirpMs={chirpMs} intervalMs={intervalMs} led={result} audio={audioResult}"); + return result.StartsWith("ok", StringComparison.OrdinalIgnoreCase) && + (audioResult.StartsWith("ok", StringComparison.OrdinalIgnoreCase) || audioResult.StartsWith("skipped", StringComparison.OrdinalIgnoreCase)) + ? 0 + : 1; +} + +foreach (var device in devices) +{ + PrintDevice(device); + + if (!string.IsNullOrWhiteSpace(pairHost)) + { + var result = WindowsHid.TryPairHost(device, pairHost); + Console.WriteLine($" pairHost={result}"); + } + + if (readReports) + { + var result = await WindowsHid.TryReadInputReportAsync(device, TimeSpan.FromMilliseconds(350)); + Console.WriteLine($" read={result}"); + } + + if (!string.IsNullOrWhiteSpace(rgb)) + { + var result = holdSeconds >= 0.0 + ? await WindowsHid.TryHoldLedAsync( + device, + rgb, + TimeSpan.FromSeconds(holdSeconds), + TimeSpan.FromMilliseconds(Math.Max(25, refreshMs)), + eventLog, + args.Any(arg => string.Equals(arg, "--turn-off-on-exit", StringComparison.OrdinalIgnoreCase))).ConfigureAwait(false) + : await WindowsHid.TryWriteLedPulseTrainAsync( + device, + rgb, + Math.Max(1, pulseCount), + TimeSpan.FromMilliseconds(Math.Max(0, pulseMs)), + TimeSpan.FromMilliseconds(Math.Max(1, intervalMs)), + eventLog).ConfigureAwait(false); + Console.WriteLine($" led={result}"); + } + + Console.WriteLine(); +} + +return 0; + +static void PrintDevice(HidDeviceInfo device) +{ + Console.WriteLine(device.Path); + Console.WriteLine($" vid=0x{device.VendorId:X4} pid=0x{device.ProductId:X4}"); + Console.WriteLine($" manufacturer={device.Manufacturer ?? "(unreported)"}"); + Console.WriteLine($" product={device.Product ?? "(unreported)"}"); + Console.WriteLine($" serial={device.SerialNumber ?? "(unreported)"}"); + Console.WriteLine($" usagePage=0x{device.Caps.UsagePage:X4} usage=0x{device.Caps.Usage:X4}"); + Console.WriteLine($" reports input={device.Caps.InputReportByteLength} output={device.Caps.OutputReportByteLength} feature={device.Caps.FeatureReportByteLength}"); + Console.WriteLine($" buttons in/out/feature={device.Caps.NumberInputButtonCaps}/{device.Caps.NumberOutputButtonCaps}/{device.Caps.NumberFeatureButtonCaps}"); + Console.WriteLine($" values in/out/feature={device.Caps.NumberInputValueCaps}/{device.Caps.NumberOutputValueCaps}/{device.Caps.NumberFeatureValueCaps}"); +} + +static string? TryGetOption(string[] values, string name) +{ + for (var index = 0; index < values.Length; index++) + { + if (string.Equals(values[index], name, StringComparison.OrdinalIgnoreCase) && index + 1 < values.Length) + { + return values[index + 1]; + } + + if (values[index].StartsWith(name + "=", StringComparison.OrdinalIgnoreCase)) + { + return values[index][(name.Length + 1)..]; + } + } + + return null; +} + +static int TryGetIntOption(string[] values, string name, int fallback) => + int.TryParse(TryGetOption(values, name), out var value) ? value : fallback; + +static double TryGetDoubleOption(string[] values, string name, double fallback) => + double.TryParse(TryGetOption(values, name), out var value) ? value : fallback; + +internal sealed record HidDeviceInfo( + string Path, + ushort VendorId, + ushort ProductId, + string? Manufacturer, + string? Product, + string? SerialNumber, + HidpCaps Caps); + +internal sealed record PairedChirpEvent( + string EventId, + int Index, + double OffsetSeconds, + int AudioStartSample, + int AudioEndSample, + double StartHz, + double EndHz); + +internal static class PairedChirpTrain +{ + public static IReadOnlyList RenderAudio( + string path, + int count, + TimeSpan startDelay, + TimeSpan interval, + TimeSpan chirpDuration, + int sampleRate, + double startHz, + double endHz, + double gain) + { + var events = new List(count); + var intervalSeconds = Math.Max(0.001, interval.TotalSeconds); + var startDelaySeconds = Math.Max(0.0, startDelay.TotalSeconds); + var chirpSeconds = Math.Max(0.001, chirpDuration.TotalSeconds); + var chirpSamples = Math.Max(1, (int)Math.Round(chirpSeconds * sampleRate)); + var totalSamples = Math.Max( + chirpSamples, + (int)Math.Ceiling((startDelaySeconds + (count - 1) * intervalSeconds + chirpSeconds + 0.05) * sampleRate)); + var samples = new float[totalSamples]; + var runId = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + + for (var index = 0; index < count; index++) + { + var offsetSeconds = startDelaySeconds + index * intervalSeconds; + var startSample = Math.Min(samples.Length - 1, (int)Math.Round(offsetSeconds * sampleRate)); + var endSample = Math.Min(samples.Length, startSample + chirpSamples); + events.Add(new PairedChirpEvent( + $"starfire-usb-move-chirp:{runId}:{index}", + index, + offsetSeconds, + startSample, + endSample, + startHz, + endHz)); + + for (var sample = startSample; sample < endSample; sample++) + { + var local = sample - startSample; + var normalized = chirpSamples <= 1 ? 0.0 : local / (double)(chirpSamples - 1); + var frequency = startHz + (endHz - startHz) * normalized; + var phase = 2.0 * Math.PI * (startHz * local / sampleRate + + 0.5 * (endHz - startHz) * local * local / Math.Max(1.0, chirpSamples) / sampleRate); + var envelope = 0.5 - 0.5 * Math.Cos(2.0 * Math.PI * normalized); + samples[sample] += (float)Math.Clamp(Math.Sin(phase) * envelope * gain, -1.0, 1.0); + _ = frequency; // keep the linear-frequency intent explicit for receipts. + } + } + + using var stream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.Read); + var bytes = new byte[samples.Length * sizeof(float)]; + Buffer.BlockCopy(samples, 0, bytes, 0, bytes.Length); + stream.Write(bytes); + return events; + } + + public static void AppendScheduleEvents( + string? eventLogPath, + IReadOnlyList events, + string audioPath, + int sampleRate, + double gain) + { + if (string.IsNullOrWhiteSpace(eventLogPath)) + { + return; + } + + foreach (var item in events) + { + var line = System.Text.Json.JsonSerializer.Serialize(new + { + document = "mimir.psmove_usb_audio_visual_pulse_event.v1", + invariant = "visual pulse and audio chirp share one event id and one planned offset", + item.EventId, + item.Index, + item.OffsetSeconds, + item.AudioStartSample, + item.AudioEndSample, + item.StartHz, + item.EndHz, + phase = "schedule", + audioPath, + sampleRate, + gain, + result = "planned" + }); + File.AppendAllText(eventLogPath, line + Environment.NewLine); + } + } + + public static async Task EmitAudioAsync( + string audioPath, + int sampleRate, + double gain, + string? renderDevice, + string? wasapiExe, + string? eventLogPath) + { + var resolvedExe = ResolveWasapiExecutable(wasapiExe); + var args = new List + { + "--play-f32-mono", + "--input", + Path.GetFullPath(audioPath), + "--sample-rate", + sampleRate.ToString(System.Globalization.CultureInfo.InvariantCulture), + "--gain", + gain.ToString(System.Globalization.CultureInfo.InvariantCulture), + "--render-buffer-seconds", + "0.02", + "--render-drain-seconds", + "0.08" + }; + if (!string.IsNullOrWhiteSpace(renderDevice)) + { + args.Add("--device"); + args.Add(renderDevice); + } + + var startedNs = WindowsHid.NowNs(); + AppendRenderEvent(eventLogPath, "render-start", audioPath, sampleRate, gain, startedNs, startedNs, "starting"); + using var process = new Process(); + process.StartInfo.FileName = resolvedExe; + foreach (var arg in args) + { + process.StartInfo.ArgumentList.Add(arg); + } + + process.StartInfo.UseShellExecute = false; + process.StartInfo.RedirectStandardError = true; + process.StartInfo.RedirectStandardOutput = true; + try + { + process.Start(); + var stderrTask = process.StandardError.ReadToEndAsync(); + var stdoutTask = process.StandardOutput.ReadToEndAsync(); + await process.WaitForExitAsync().ConfigureAwait(false); + var completedNs = WindowsHid.NowNs(); + var stderr = await stderrTask.ConfigureAwait(false); + var stdout = await stdoutTask.ConfigureAwait(false); + var result = process.ExitCode == 0 + ? $"ok renderExit=0 pid={process.Id}" + : $"render failed exit={process.ExitCode} pid={process.Id}"; + AppendRenderEvent(eventLogPath, "render-complete", audioPath, sampleRate, gain, startedNs, completedNs, result, stdout, stderr); + return result; + } + catch (Exception ex) + { + var completedNs = WindowsHid.NowNs(); + var result = "render failed: " + ex.Message; + AppendRenderEvent(eventLogPath, "render-complete", audioPath, sampleRate, gain, startedNs, completedNs, result); + return result; + } + } + + private static string ResolveWasapiExecutable(string? explicitPath) + { + if (!string.IsNullOrWhiteSpace(explicitPath)) + { + return explicitPath; + } + + var current = new DirectoryInfo(Environment.CurrentDirectory); + while (current != null) + { + var candidate = Path.Combine( + current.FullName, + "src", + "Mimir.WasapiLoopback", + "bin", + "Debug", + "net10.0-windows", + "Mimir.WasapiLoopback.exe"); + if (File.Exists(candidate)) + { + return candidate; + } + + current = current.Parent; + } + + return "Mimir.WasapiLoopback.exe"; + } + + private static void AppendRenderEvent( + string? eventLogPath, + string phase, + string audioPath, + int sampleRate, + double gain, + long startedNs, + long completedNs, + string result, + string? stdout = null, + string? stderr = null) + { + if (string.IsNullOrWhiteSpace(eventLogPath)) + { + return; + } + + var line = System.Text.Json.JsonSerializer.Serialize(new + { + document = "mimir.psmove_usb_audio_visual_pulse_event.v1", + invariant = "visual pulse and audio chirp share one event id and one planned offset", + phase, + audioPath, + sampleRate, + gain, + startedNs, + completedNs, + result, + stdout, + stderr + }); + File.AppendAllText(eventLogPath, line + Environment.NewLine); + } +} + +internal static class WindowsHid +{ + private const int DigcfPresent = 0x00000002; + private const int DigcfDeviceinterface = 0x00000010; + private const uint GenericRead = 0x80000000; + private const uint GenericWrite = 0x40000000; + private const uint FileShareRead = 0x00000001; + private const uint FileShareWrite = 0x00000002; + private const uint OpenExisting = 3; + private const uint FileFlagOverlapped = 0x40000000; + private const uint ErrorIoPending = 997; + private const uint WaitObject0 = 0; + private const uint WaitTimeout = 258; + private const int HidpStatusSuccess = 0x00110000; + private const byte GetBtAddressReport = 0x04; + private const byte SetBtAddressReport = 0x05; + private const byte LedReport = 0x06; + private const int BtAddressGetSize = 16; + private const int BtAddressSetSize = 23; + + public static IEnumerable Enumerate() + { + foreach (var path in EnumeratePaths()) + { + if (TryProbe(path, out var device)) + { + yield return device; + } + } + } + + public static IEnumerable EnumeratePaths() + { + HidD_GetHidGuid(out var hidGuid); + + var deviceInfoSet = SetupDiGetClassDevs(ref hidGuid, null, IntPtr.Zero, DigcfPresent | DigcfDeviceinterface); + if (deviceInfoSet == IntPtr.Zero || deviceInfoSet.ToInt64() == -1) + { + throw new Win32Exception(Marshal.GetLastWin32Error(), "SetupDiGetClassDevs failed."); + } + + try + { + var interfaceData = new SpDeviceInterfaceData + { + CbSize = Marshal.SizeOf() + }; + + for (uint index = 0; SetupDiEnumDeviceInterfaces(deviceInfoSet, IntPtr.Zero, ref hidGuid, index, ref interfaceData); index++) + { + yield return GetDevicePath(deviceInfoSet, interfaceData); + } + } + finally + { + SetupDiDestroyDeviceInfoList(deviceInfoSet); + } + } + + public static async Task TryReadInputReportAsync(HidDeviceInfo device, TimeSpan timeout) + { + if (device.Caps.InputReportByteLength == 0) + { + return "skipped: no input report"; + } + + using var handle = CreateFile( + device.Path, + GenericRead, + FileShareRead | FileShareWrite, + IntPtr.Zero, + OpenExisting, + FileFlagOverlapped, + IntPtr.Zero); + + if (handle.IsInvalid) + { + return $"open failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + await Task.Yield(); + var buffer = new byte[device.Caps.InputReportByteLength]; + var readEvent = CreateEvent(IntPtr.Zero, true, false, null); + + if (readEvent == IntPtr.Zero) + { + return $"CreateEvent failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + var overlapped = new NativeOverlapped + { + EventHandle = readEvent + }; + + try + { + var started = ReadFile(handle, buffer, (uint)buffer.Length, IntPtr.Zero, ref overlapped); + if (!started) + { + var error = (uint)Marshal.GetLastWin32Error(); + if (error != ErrorIoPending) + { + return $"ReadFile failed: {new Win32Exception((int)error).Message}"; + } + } + + var wait = WaitForSingleObject(readEvent, (uint)Math.Ceiling(timeout.TotalMilliseconds)); + if (wait == WaitTimeout) + { + CancelIoEx(handle, ref overlapped); + return $"timeout after {timeout.TotalMilliseconds:0}ms"; + } + + if (wait != WaitObject0) + { + CancelIoEx(handle, ref overlapped); + return $"wait failed: 0x{wait:X8}"; + } + + if (!GetOverlappedResult(handle, ref overlapped, out var read, false)) + { + return $"GetOverlappedResult failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + var shown = Convert.ToHexString(buffer.AsSpan(0, (int)Math.Min(read, 32))); + return $"{read} bytes {shown}{(read > 32 ? "..." : string.Empty)}"; + } + finally + { + CloseHandle(readEvent); + } + } + + public static string TryPairHost(HidDeviceInfo device, string hostAddress) + { + if (device.Caps.FeatureReportByteLength < BtAddressSetSize || device.Path.IndexOf("&col02#", StringComparison.OrdinalIgnoreCase) < 0) + { + return "skipped: pairing uses the PS Move col02 feature-report collection"; + } + + if (!TryParseBluetoothAddress(hostAddress, out var host)) + { + return $"invalid host address: {hostAddress}"; + } + + using var handle = CreateFile( + device.Path, + 0, + FileShareRead | FileShareWrite, + IntPtr.Zero, + OpenExisting, + 0, + IntPtr.Zero); + + if (handle.IsInvalid) + { + return $"open failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + var before = ReadBluetoothAddresses(handle); + var report = new byte[BtAddressSetSize]; + report[0] = SetBtAddressReport; + host.CopyTo(report.AsSpan(1, 6)); + + if (!HidD_SetFeature(handle, report, report.Length)) + { + return $"HidD_SetFeature failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + var after = ReadBluetoothAddresses(handle); + var controller = after.Controller ?? before.Controller ?? "(unknown)"; + var assigned = after.Host ?? FormatBluetoothAddress(host); + return $"ok controller={controller} host={assigned}"; + } + + public static async Task TryWriteLedPulseTrainAsync( + HidDeviceInfo device, + string rgb, + int pulseCount, + TimeSpan pulseDuration, + TimeSpan interval, + string? eventLogPath) + { + if (device.Caps.OutputReportByteLength == 0 || device.Path.IndexOf("&col01#", StringComparison.OrdinalIgnoreCase) < 0) + { + return "skipped: LED output uses the PS Move col01 output-report collection"; + } + + if (!TryParseRgb(rgb, out var r, out var g, out var b)) + { + return $"invalid rgb: {rgb}"; + } + + var written = 0; + for (var index = 0; index < Math.Max(1, pulseCount); index++) + { + var eventId = $"starfire-usb-move:{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}:{index}"; + var onStarted = NowNs(); + var on = TryWriteLed(device, r, g, b); + var onCompleted = NowNs(); + AppendPulseEvent(eventLogPath, eventId, index, "on", r, g, b, onStarted, onCompleted, on); + if (!on.StartsWith("ok", StringComparison.OrdinalIgnoreCase)) + { + return on; + } + + if (pulseDuration > TimeSpan.Zero) + { + await Task.Delay(pulseDuration).ConfigureAwait(false); + } + + var offStarted = NowNs(); + var off = TryWriteLed(device, 0, 0, 0); + var offCompleted = NowNs(); + AppendPulseEvent(eventLogPath, eventId, index, "off", 0, 0, 0, offStarted, offCompleted, off); + if (!off.StartsWith("ok", StringComparison.OrdinalIgnoreCase)) + { + return $"off failed after {written} pulses: {off}"; + } + + written++; + if (index + 1 < pulseCount) + { + var rest = interval - pulseDuration; + if (rest > TimeSpan.Zero) + { + await Task.Delay(rest).ConfigureAwait(false); + } + } + } + + return $"ok pulses={written} rgb=#{r:X2}{g:X2}{b:X2} pulseMs={pulseDuration.TotalMilliseconds:0} intervalMs={interval.TotalMilliseconds:0}"; + } + + public static async Task TryHoldLedAsync( + HidDeviceInfo device, + string rgb, + TimeSpan duration, + TimeSpan refresh, + string? eventLogPath, + bool turnOffOnExit) + { + if (device.Caps.OutputReportByteLength == 0 || device.Path.IndexOf("&col01#", StringComparison.OrdinalIgnoreCase) < 0) + { + return "skipped: LED output uses the PS Move col01 output-report collection"; + } + + if (!TryParseRgb(rgb, out var r, out var g, out var b)) + { + return $"invalid rgb: {rgb}"; + } + + Console.CancelKeyPress += (_, eventArgs) => + { + eventArgs.Cancel = true; + if (turnOffOnExit) + { + _ = TryWriteLed(device, 0, 0, 0); + } + + Environment.Exit(0); + }; + + var eventId = $"starfire-usb-move-hold:{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}"; + var stopwatch = Stopwatch.StartNew(); + var writes = 0; + while (duration == TimeSpan.Zero || stopwatch.Elapsed < duration) + { + var started = NowNs(); + var result = TryWriteLed(device, r, g, b); + var completed = NowNs(); + AppendPulseEvent(eventLogPath, eventId, writes, "hold", r, g, b, started, completed, result); + if (!result.StartsWith("ok", StringComparison.OrdinalIgnoreCase)) + { + return result; + } + + writes++; + await Task.Delay(refresh).ConfigureAwait(false); + } + + if (turnOffOnExit) + { + var started = NowNs(); + var result = TryWriteLed(device, 0, 0, 0); + var completed = NowNs(); + AppendPulseEvent(eventLogPath, eventId, writes, "off", 0, 0, 0, started, completed, result); + if (!result.StartsWith("ok", StringComparison.OrdinalIgnoreCase)) + { + return $"off failed after {writes} writes: {result}"; + } + } + + return $"ok holdWrites={writes} rgb=#{r:X2}{g:X2}{b:X2} durationSeconds={duration.TotalSeconds:0.###} refreshMs={refresh.TotalMilliseconds:0}"; + } + + public static async Task TryWriteScheduledLedTrainAsync( + HidDeviceInfo device, + IReadOnlyList events, + byte r, + byte g, + byte b, + TimeSpan pulseDuration, + string? eventLogPath) + { + if (device.Caps.OutputReportByteLength == 0 || device.Path.IndexOf("&col01#", StringComparison.OrdinalIgnoreCase) < 0) + { + return "skipped: LED output uses the PS Move col01 output-report collection"; + } + + TimeBeginPeriod(1); + try + { + var started = NowNs(); + var stopwatch = Stopwatch.StartNew(); + var written = 0; + foreach (var item in events) + { + WaitUntil(stopwatch, TimeSpan.FromSeconds(item.OffsetSeconds)); + + var onStarted = NowNs(); + var on = TryWriteLed(device, r, g, b); + var onCompleted = NowNs(); + AppendPairedPulseEvent(eventLogPath, item, "on", r, g, b, started, onStarted, onCompleted, on); + if (!on.StartsWith("ok", StringComparison.OrdinalIgnoreCase)) + { + return on; + } + + if (pulseDuration > TimeSpan.Zero) + { + WaitUntil(stopwatch, TimeSpan.FromSeconds(item.OffsetSeconds) + pulseDuration); + } + + var offStarted = NowNs(); + var off = TryWriteLed(device, 0, 0, 0); + var offCompleted = NowNs(); + AppendPairedPulseEvent(eventLogPath, item, "off", 0, 0, 0, started, offStarted, offCompleted, off); + if (!off.StartsWith("ok", StringComparison.OrdinalIgnoreCase)) + { + return $"off failed after {written} events: {off}"; + } + + written++; + } + + return $"ok pairedEvents={written}"; + } + finally + { + TimeEndPeriod(1); + } + } + + private static void WaitUntil(Stopwatch stopwatch, TimeSpan due) + { + while (true) + { + var remaining = due - stopwatch.Elapsed; + if (remaining <= TimeSpan.Zero) + { + return; + } + + if (remaining > TimeSpan.FromMilliseconds(8)) + { + Thread.Sleep(Math.Max(1, (int)Math.Floor(remaining.TotalMilliseconds) - 4)); + continue; + } + + if (remaining > TimeSpan.FromMilliseconds(1.5)) + { + Thread.Sleep(0); + continue; + } + + Thread.SpinWait(256); + } + } + + private static string TryWriteLed(HidDeviceInfo device, byte r, byte g, byte b) + { + using var handle = CreateFile( + device.Path, + GenericWrite, + FileShareRead | FileShareWrite, + IntPtr.Zero, + OpenExisting, + 0, + IntPtr.Zero); + + if (handle.IsInvalid) + { + return $"open failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + var report = new byte[Math.Max(9, (int)device.Caps.OutputReportByteLength)]; + report[0] = LedReport; + report[1] = 0; + report[2] = r; + report[3] = g; + report[4] = b; + return WriteFile(handle, report, (uint)report.Length, out var written, IntPtr.Zero) + ? $"ok rgb=#{r:X2}{g:X2}{b:X2} bytes={written}" + : $"WriteFile failed: {new Win32Exception(Marshal.GetLastWin32Error()).Message}"; + } + + public static bool TryParseRgb(string value, out byte r, out byte g, out byte b) + { + r = 0; + g = 0; + b = 0; + var normalized = value.Trim().TrimStart('#'); + if (normalized.Length == 6) + { + return byte.TryParse(normalized.AsSpan(0, 2), System.Globalization.NumberStyles.HexNumber, null, out r) && + byte.TryParse(normalized.AsSpan(2, 2), System.Globalization.NumberStyles.HexNumber, null, out g) && + byte.TryParse(normalized.AsSpan(4, 2), System.Globalization.NumberStyles.HexNumber, null, out b); + } + + var parts = normalized.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + return parts.Length == 3 && + byte.TryParse(parts[0], out r) && + byte.TryParse(parts[1], out g) && + byte.TryParse(parts[2], out b); + } + + public static long NowNs() => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000L; + + private static void AppendPulseEvent( + string? eventLogPath, + string eventId, + int index, + string phase, + byte r, + byte g, + byte b, + long startedNs, + long completedNs, + string result) + { + if (string.IsNullOrWhiteSpace(eventLogPath)) + { + return; + } + + var line = System.Text.Json.JsonSerializer.Serialize(new + { + document = "mimir.psmove_usb_pulse_event.v1", + eventId, + index, + phase, + rgb = new[] { (int)r, (int)g, (int)b }, + startedNs, + completedNs, + result, + }); + File.AppendAllText(eventLogPath, line + Environment.NewLine); + } + + private static void AppendPairedPulseEvent( + string? eventLogPath, + PairedChirpEvent item, + string phase, + byte r, + byte g, + byte b, + long trainStartNs, + long startedNs, + long completedNs, + string result) + { + if (string.IsNullOrWhiteSpace(eventLogPath)) + { + return; + } + + var line = System.Text.Json.JsonSerializer.Serialize(new + { + document = "mimir.psmove_usb_audio_visual_pulse_event.v1", + invariant = "visual pulse and audio chirp share one event id and one planned offset", + item.EventId, + item.Index, + item.OffsetSeconds, + item.AudioStartSample, + item.AudioEndSample, + item.StartHz, + item.EndHz, + phase, + rgb = new[] { (int)r, (int)g, (int)b }, + trainStartNs, + startedNs, + completedNs, + result, + }); + File.AppendAllText(eventLogPath, line + Environment.NewLine); + } + + private static (string? Controller, string? Host) ReadBluetoothAddresses(SafeFileHandle handle) + { + var report = new byte[BtAddressGetSize]; + report[0] = GetBtAddressReport; + if (!HidD_GetFeature(handle, report, report.Length)) + { + return (null, null); + } + + return (FormatBluetoothAddress(report.AsSpan(1, 6)), FormatBluetoothAddress(report.AsSpan(10, 6))); + } + + private static bool TryParseBluetoothAddress(string value, out byte[] bytes) + { + bytes = new byte[6]; + var parts = value.Split(':', '-'); + if (parts.Length != 6) + { + return false; + } + + for (var index = 0; index < parts.Length; index++) + { + if (!byte.TryParse(parts[index], System.Globalization.NumberStyles.HexNumber, System.Globalization.CultureInfo.InvariantCulture, out var parsed)) + { + return false; + } + + bytes[5 - index] = parsed; + } + + return true; + } + + private static string FormatBluetoothAddress(ReadOnlySpan littleEndianAddress) + { + return string.Join( + ":", + littleEndianAddress.ToArray().Reverse().Select(value => value.ToString("x2", System.Globalization.CultureInfo.InvariantCulture))); + } + + private static string GetDevicePath(IntPtr deviceInfoSet, SpDeviceInterfaceData interfaceData) + { + SetupDiGetDeviceInterfaceDetail(deviceInfoSet, ref interfaceData, IntPtr.Zero, 0, out var requiredSize, IntPtr.Zero); + if (requiredSize == 0) + { + throw new Win32Exception(Marshal.GetLastWin32Error(), "SetupDiGetDeviceInterfaceDetail size query failed."); + } + + var detailData = Marshal.AllocHGlobal((int)requiredSize); + try + { + Marshal.WriteInt32(detailData, IntPtr.Size == 8 ? 8 : 6); + if (!SetupDiGetDeviceInterfaceDetail(deviceInfoSet, ref interfaceData, detailData, requiredSize, out _, IntPtr.Zero)) + { + throw new Win32Exception(Marshal.GetLastWin32Error(), "SetupDiGetDeviceInterfaceDetail failed."); + } + + return Marshal.PtrToStringUni(IntPtr.Add(detailData, 4)) + ?? throw new InvalidOperationException("HID device path was empty."); + } + finally + { + Marshal.FreeHGlobal(detailData); + } + } + + private static bool TryProbe(string path, out HidDeviceInfo device) + { + device = default!; + if (!TryExtractVidPid(path, out var vid, out var pid)) + { + return false; + } + + using var handle = CreateFile( + path, + 0, + FileShareRead | FileShareWrite, + IntPtr.Zero, + OpenExisting, + 0, + IntPtr.Zero); + + if (handle.IsInvalid) + { + return false; + } + + if (!HidD_GetPreparsedData(handle, out var preparsedData)) + { + return false; + } + + try + { + var status = HidP_GetCaps(preparsedData, out var caps); + if (status != HidpStatusSuccess) + { + return false; + } + + device = new HidDeviceInfo( + path, + vid, + pid, + GetHidString(handle, HidD_GetManufacturerString), + GetHidString(handle, HidD_GetProductString), + GetHidString(handle, HidD_GetSerialNumberString), + caps); + return true; + } + finally + { + HidD_FreePreparsedData(preparsedData); + } + } + + private static bool TryExtractVidPid(string path, out ushort vid, out ushort pid) + { + vid = 0; + pid = 0; + var lower = path.ToLowerInvariant(); + return TryExtractHexAfter(lower, "vid_", out vid) && TryExtractHexAfter(lower, "pid_", out pid); + } + + private static bool TryExtractHexAfter(string value, string marker, out ushort result) + { + result = 0; + var index = value.IndexOf(marker, StringComparison.Ordinal); + if (index < 0 || index + marker.Length + 4 > value.Length) + { + return false; + } + + return ushort.TryParse( + value.AsSpan(index + marker.Length, 4), + System.Globalization.NumberStyles.HexNumber, + System.Globalization.CultureInfo.InvariantCulture, + out result); + } + + private delegate bool HidStringGetter(SafeFileHandle hidDeviceObject, byte[] buffer, int bufferLength); + + private static string? GetHidString(SafeFileHandle handle, HidStringGetter getter) + { + var buffer = new byte[256]; + return getter(handle, buffer, buffer.Length) + ? Marshal.PtrToStringUni(Marshal.UnsafeAddrOfPinnedArrayElement(buffer, 0))?.TrimEnd('\0') + : null; + } + + [DllImport("hid.dll")] + private static extern void HidD_GetHidGuid(out Guid hidGuid); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_GetPreparsedData(SafeFileHandle hidDeviceObject, out IntPtr preparsedData); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_FreePreparsedData(IntPtr preparsedData); + + [DllImport("hid.dll")] + private static extern int HidP_GetCaps(IntPtr preparsedData, out HidpCaps capabilities); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_GetManufacturerString(SafeFileHandle hidDeviceObject, byte[] buffer, int bufferLength); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_GetProductString(SafeFileHandle hidDeviceObject, byte[] buffer, int bufferLength); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_GetSerialNumberString(SafeFileHandle hidDeviceObject, byte[] buffer, int bufferLength); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_GetFeature(SafeFileHandle hidDeviceObject, byte[] reportBuffer, int reportBufferLength); + + [DllImport("hid.dll", SetLastError = true)] + private static extern bool HidD_SetFeature(SafeFileHandle hidDeviceObject, byte[] reportBuffer, int reportBufferLength); + + [DllImport("setupapi.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern IntPtr SetupDiGetClassDevs( + ref Guid classGuid, + string? enumerator, + IntPtr hwndParent, + int flags); + + [DllImport("setupapi.dll", SetLastError = true)] + private static extern bool SetupDiEnumDeviceInterfaces( + IntPtr deviceInfoSet, + IntPtr deviceInfoData, + ref Guid interfaceClassGuid, + uint memberIndex, + ref SpDeviceInterfaceData deviceInterfaceData); + + [DllImport("setupapi.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern bool SetupDiGetDeviceInterfaceDetail( + IntPtr deviceInfoSet, + ref SpDeviceInterfaceData deviceInterfaceData, + IntPtr deviceInterfaceDetailData, + uint deviceInterfaceDetailDataSize, + out uint requiredSize, + IntPtr deviceInfoData); + + [DllImport("setupapi.dll", SetLastError = true)] + private static extern bool SetupDiDestroyDeviceInfoList(IntPtr deviceInfoSet); + + [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern SafeFileHandle CreateFile( + string fileName, + uint desiredAccess, + uint shareMode, + IntPtr securityAttributes, + uint creationDisposition, + uint flagsAndAttributes, + IntPtr templateFile); + + [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern IntPtr CreateEvent( + IntPtr eventAttributes, + bool manualReset, + bool initialState, + string? name); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern bool ReadFile( + SafeFileHandle file, + byte[] buffer, + uint numberOfBytesToRead, + IntPtr numberOfBytesRead, + ref NativeOverlapped overlapped); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern bool WriteFile( + SafeFileHandle file, + byte[] buffer, + uint numberOfBytesToWrite, + out uint numberOfBytesWritten, + IntPtr overlapped); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern bool GetOverlappedResult( + SafeFileHandle file, + ref NativeOverlapped overlapped, + out uint numberOfBytesTransferred, + bool wait); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern uint WaitForSingleObject(IntPtr handle, uint milliseconds); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern bool CancelIoEx( + SafeFileHandle file, + ref NativeOverlapped overlapped); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern bool CloseHandle(IntPtr handle); + + [DllImport("winmm.dll", EntryPoint = "timeBeginPeriod")] + private static extern uint TimeBeginPeriod(uint period); + + [DllImport("winmm.dll", EntryPoint = "timeEndPeriod")] + private static extern uint TimeEndPeriod(uint period); + + [StructLayout(LayoutKind.Sequential)] + private struct SpDeviceInterfaceData + { + public int CbSize; + public Guid InterfaceClassGuid; + public int Flags; + public IntPtr Reserved; + } + + [StructLayout(LayoutKind.Sequential)] + private struct NativeOverlapped + { + public IntPtr InternalLow; + public IntPtr InternalHigh; + public uint OffsetLow; + public uint OffsetHigh; + public IntPtr EventHandle; + } +} + +[StructLayout(LayoutKind.Sequential)] +internal struct HidpCaps +{ + public ushort Usage; + public ushort UsagePage; + public ushort InputReportByteLength; + public ushort OutputReportByteLength; + public ushort FeatureReportByteLength; + + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 17)] + public ushort[] Reserved; + + public ushort NumberLinkCollectionNodes; + public ushort NumberInputButtonCaps; + public ushort NumberInputValueCaps; + public ushort NumberInputDataIndices; + public ushort NumberOutputButtonCaps; + public ushort NumberOutputValueCaps; + public ushort NumberOutputDataIndices; + public ushort NumberFeatureButtonCaps; + public ushort NumberFeatureValueCaps; + public ushort NumberFeatureDataIndices; +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index dda3f5f..1eb8cac 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -125,3 +125,4 @@ {"ts": "2026-06-09T02:54:54+01:00", "type": "verification", "status": "ok", "note": "Imported the current OBS scene into Mimir-owned typed program state with Mimir.BufferSmoke --import-obs-program-scene. state/mimir-program-composition.cc now carries mimir.program_scene.v1, mimir.program_output.v1, and mimir.eve_operator_surface.v1 for eight current layers: Starfire monitor, Discord crop/chroma key, Raven monitor, Kiyo Pro, Mimir program texture, Muninn stream placeholder, SP browser, and hidden bigscreen."} {"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."} {"ts": "2026-06-10T17:38:22+01:00", "type": "verification", "status": "ok", "note": "Restored and launched the narrow Nightwing Move tracking witness path. scripts/start-nightwing-move-tracking.ps1 started Mimir.EveSensorReceiver on Starfire port 8796, staged Nightwing Eye/Move Python tools, launched Nightwing worker PID 1131578 with --move-light move-usb=/dev/hidraw1:#35ff6c, and VerseRecorder C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures\\20260610-173549 recorded mimir.psmove_light_state.v1 HID success plus mimir.move_controller_observation_state.v1 blobs from both ps3-eye-0 and ps3-eye-1."} +{"ts": "2026-06-10T18:11:18+01:00", "type": "verification", "status": "ok", "note": "Started the Starfire-local USB Move light owner. Windows sees Sony VID_054C/PID_03D5 with col01 output reports; scripts/start-starfire-move-light.ps1 launched Mimir.PsMoveProbe PID 27344 with --rgb #35ff6c --hold-seconds 0 --refresh-ms 250, and artifacts/runtime/starfire-move-light-20260610-180943/move-light-events.jsonl shows repeated mimir.psmove_usb_pulse_event.v1 hold writes succeeding with ok rgb=#35FF6C bytes=49."} From 1692c3ec64ae0efe5ac136adbefd341ca4534fc8 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 18:13:26 +0100 Subject: [PATCH 011/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 65213 -> 65126 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index a7031b0068f8fec2052a16e1e2adee049bb0deea..2d566d42fc44b0a28dea98c9b5c2af50a49f7afa 100644 GIT binary patch delta 185 zcmdn{m-*Qr<_$l2xlE0%3{3P)jE$luGKx-?;Opjqh?p5oX51Ea-nK0Obnv{daE>B delta 272 zcmaF%hk5T`<_$l2xeQIMjLh_mERCZkGKx-?;Opjqh?p2mX5 Date: Wed, 10 Jun 2026 20:35:35 +0100 Subject: [PATCH 012/135] Add Rust Move sphere candidate tracker --- docs/implementation-plan.md | 6 + native/move_tracker/Cargo.lock | 7 + native/move_tracker/Cargo.toml | 10 + native/move_tracker/README.md | 23 ++ .../shaders/MoveSphereCandidate.comp.hlsl | 104 ++++++ native/move_tracker/src/lib.rs | 308 ++++++++++++++++++ notes/current-system-map.md | 8 + state/evidence.jsonl | 1 + state/map.yaml | 7 + 9 files changed, 474 insertions(+) create mode 100644 native/move_tracker/Cargo.lock create mode 100644 native/move_tracker/Cargo.toml create mode 100644 native/move_tracker/README.md create mode 100644 native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl create mode 100644 native/move_tracker/src/lib.rs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index db936a5..8c5b816 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -50,6 +50,12 @@ a named invariant that the native runtime cannot protect yet. owner. It launches `Mimir.PsMoveProbe` in steady hold mode so the locally attached `VID_054C/PID_03D5` Move sphere stays lit for calibration/tracking receipts. +- `native/move_tracker` is the first native/Rust Move optical candidate + extractor. Rust owns dispatch planning, FFI, validation, and a deterministic + CPU mirror; `shaders/MoveSphereCandidate.comp.hlsl` owns one 16px tile per + thread group and emits weighted bright-sphere candidates. This is marker + extraction only, not final pose, stereo triangulation, calibration, + association, IMU fusion, or prediction. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe diff --git a/native/move_tracker/Cargo.lock b/native/move_tracker/Cargo.lock new file mode 100644 index 0000000..8843462 --- /dev/null +++ b/native/move_tracker/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "mimir-move-tracker" +version = "0.1.0" diff --git a/native/move_tracker/Cargo.toml b/native/move_tracker/Cargo.toml new file mode 100644 index 0000000..ff0060e --- /dev/null +++ b/native/move_tracker/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "mimir-move-tracker" +version = "0.1.0" +edition = "2024" +description = "Rust orchestration and CPU mirror for PS Move sphere tracking compute" + +[lib] +crate-type = ["rlib", "cdylib", "staticlib"] + +[dependencies] diff --git a/native/move_tracker/README.md b/native/move_tracker/README.md new file mode 100644 index 0000000..cc9e954 --- /dev/null +++ b/native/move_tracker/README.md @@ -0,0 +1,23 @@ +# Mimir Move Tracker + +Rust owns orchestration for PS Move sphere tracking. The compute shader owns the +parallel image reduction stage. This crate deliberately stops at per-frame +marker candidates: final pose, stereo triangulation, IMU fusion, prediction, +and calibration remain separate owners. + +## Current Contract + +- Input: one luma frame (`Y8`) with width, height, stride, source hash, and frame + sequence. +- GPU stage: one 16px tile per thread group, emitting at most one candidate per + tile. +- Output: `MoveMarkerCandidate` records containing weighted centroid, radius, + area, peak/mean luma, and score. +- Rust mirror: deterministic CPU implementation and FFI wrappers for tests and + non-GPU smoke paths. +- Shader IDs are carried as `uint2` low/high pairs to avoid making 64-bit HLSL + integer support a dependency of the first extraction pass. + +This is the first stage required for PlayStation-class wand tracking. The +remaining stack needs calibrated camera rays, controlled sphere colors/exposure, +multi-camera association, triangulation, IMU fusion, and latency prediction. diff --git a/native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl b/native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl new file mode 100644 index 0000000..d00414f --- /dev/null +++ b/native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl @@ -0,0 +1,104 @@ +// Mimir PS Move sphere candidate extraction. +// One thread group owns one image tile. The shader emits at most one bright +// marker candidate per 16px tile; Rust owns dispatch sizing and downstream ranking. + +struct MoveSphereCandidate +{ + uint2 sourceIdHash; + uint2 frameSequence; + uint tileX; + uint tileY; + float centerXPx; + float centerYPx; + float radiusPx; + uint areaPx; + float meanLuma; + uint peakLuma; + float score; +}; + +cbuffer MoveTrackerConfig : register(b0) +{ + uint width; + uint height; + uint strideBytes; + uint tileSize; + uint thresholdMin; + uint minAreaPx; + uint maxCandidates; + uint _pad0; + uint2 sourceIdHash; + uint2 frameSequence; +}; + +ByteAddressBuffer frameY8 : register(t0); +RWStructuredBuffer candidates : register(u0); +RWByteAddressBuffer candidateCounter : register(u1); + +groupshared uint gArea; +groupshared uint gSumX; +groupshared uint gSumY; +groupshared uint gSumW; +groupshared uint gPeak; + +[numthreads(16, 16, 1)] +void main(uint3 groupId : SV_GroupID, uint3 groupThreadId : SV_GroupThreadID) +{ + if (groupThreadId.x == 0 && groupThreadId.y == 0) + { + gArea = 0; + gSumX = 0; + gSumY = 0; + gSumW = 0; + gPeak = 0; + } + GroupMemoryBarrierWithGroupSync(); + + uint x = groupId.x * tileSize + groupThreadId.x; + uint y = groupId.y * tileSize + groupThreadId.y; + if (groupThreadId.x < tileSize && groupThreadId.y < tileSize && x < width && y < height) + { + uint byteOffset = y * strideBytes + x; + uint packed = frameY8.Load(byteOffset & ~3u); + uint shift = (byteOffset & 3u) * 8u; + uint luma = (packed >> shift) & 0xffu; + if (luma >= thresholdMin) + { + uint weight = luma - thresholdMin + 1u; + InterlockedAdd(gArea, 1u); + InterlockedAdd(gSumX, x * weight); + InterlockedAdd(gSumY, y * weight); + InterlockedAdd(gSumW, weight); + InterlockedMax(gPeak, luma); + } + } + + GroupMemoryBarrierWithGroupSync(); + + if (groupThreadId.x == 0 && groupThreadId.y == 0 && gArea >= minAreaPx && gSumW > 0) + { + uint index; + candidateCounter.InterlockedAdd(0, 1, index); + if (index < maxCandidates) + { + float area = (float)gArea; + float tilePixels = (float)(tileSize * tileSize); + float areaScore = min(1.0, area / tilePixels); + float brightnessScore = (float)gPeak / 255.0; + + MoveSphereCandidate candidate; + candidate.sourceIdHash = sourceIdHash; + candidate.frameSequence = frameSequence; + candidate.tileX = groupId.x; + candidate.tileY = groupId.y; + candidate.centerXPx = (float)gSumX / (float)gSumW; + candidate.centerYPx = (float)gSumY / (float)gSumW; + candidate.radiusPx = sqrt(area / 3.14159265358979323846); + candidate.areaPx = gArea; + candidate.meanLuma = (float)thresholdMin + (((float)gSumW / area) - 1.0); + candidate.peakLuma = gPeak; + candidate.score = brightnessScore * (0.65 + areaScore * 0.35); + candidates[index] = candidate; + } + } +} diff --git a/native/move_tracker/src/lib.rs b/native/move_tracker/src/lib.rs new file mode 100644 index 0000000..cf59ba2 --- /dev/null +++ b/native/move_tracker/src/lib.rs @@ -0,0 +1,308 @@ +#![deny(unsafe_op_in_unsafe_fn)] + +use core::ffi::c_int; + +pub const MIMIR_MOVE_TRACKER_OK: c_int = 0; +pub const MIMIR_MOVE_TRACKER_NULL: c_int = -1; +pub const MIMIR_MOVE_TRACKER_INVALID: c_int = -2; + +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MoveTrackerConfig { + pub width: u32, + pub height: u32, + pub stride_bytes: u32, + pub tile_size: u32, + pub threshold_min: u8, + pub min_area_px: u32, + pub max_candidates: u32, + pub source_id_hash: u64, + pub frame_sequence: u64, +} + +impl Default for MoveTrackerConfig { + fn default() -> Self { + Self { + width: 320, + height: 240, + stride_bytes: 320, + tile_size: 16, + threshold_min: 180, + min_area_px: 4, + max_candidates: 64, + source_id_hash: 0, + frame_sequence: 0, + } + } +} + +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct MoveTrackerDispatchPlan { + pub group_size_x: u32, + pub group_size_y: u32, + pub groups_x: u32, + pub groups_y: u32, + pub max_candidates: u32, +} + +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct MoveMarkerCandidate { + pub source_id_hash: u64, + pub frame_sequence: u64, + pub tile_x: u32, + pub tile_y: u32, + pub center_x_px: f32, + pub center_y_px: f32, + pub radius_px: f32, + pub area_px: u32, + pub mean_luma: f32, + pub peak_luma: u32, + pub score: f32, +} + +pub fn dispatch_plan(config: MoveTrackerConfig) -> Option { + if !config_is_valid(config) { + return None; + } + + let tile = config.tile_size; + Some(MoveTrackerDispatchPlan { + group_size_x: tile, + group_size_y: tile, + groups_x: config.width.div_ceil(tile), + groups_y: config.height.div_ceil(tile), + max_candidates: config.max_candidates, + }) +} + +pub fn extract_luma_candidates( + frame: &[u8], + config: MoveTrackerConfig, +) -> Option> { + if !config_is_valid(config) { + return None; + } + + let required = required_frame_bytes(config)?; + if frame.len() < required { + return None; + } + + let mut candidates = Vec::new(); + let tile = config.tile_size; + let groups_x = config.width.div_ceil(tile); + let groups_y = config.height.div_ceil(tile); + + for tile_y in 0..groups_y { + for tile_x in 0..groups_x { + if candidates.len() >= config.max_candidates as usize { + return Some(candidates); + } + + if let Some(candidate) = extract_tile(frame, config, tile_x, tile_y) { + candidates.push(candidate); + } + } + } + + candidates.sort_by(|left, right| { + right + .score + .partial_cmp(&left.score) + .unwrap_or(core::cmp::Ordering::Equal) + }); + Some(candidates) +} + +fn extract_tile( + frame: &[u8], + config: MoveTrackerConfig, + tile_x: u32, + tile_y: u32, +) -> Option { + let x0 = tile_x * config.tile_size; + let y0 = tile_y * config.tile_size; + let x1 = (x0 + config.tile_size).min(config.width); + let y1 = (y0 + config.tile_size).min(config.height); + let threshold = u32::from(config.threshold_min); + let mut area = 0u32; + let mut sum_x = 0u64; + let mut sum_y = 0u64; + let mut sum_w = 0u64; + let mut peak = 0u32; + + for y in y0..y1 { + let row = y as usize * config.stride_bytes as usize; + for x in x0..x1 { + let luma = u32::from(frame[row + x as usize]); + if luma < threshold { + continue; + } + + area += 1; + peak = peak.max(luma); + let weight = u64::from(luma - threshold + 1); + sum_x += u64::from(x) * weight; + sum_y += u64::from(y) * weight; + sum_w += weight; + } + } + + if area < config.min_area_px || sum_w == 0 { + return None; + } + + let center_x = sum_x as f32 / sum_w as f32; + let center_y = sum_y as f32 / sum_w as f32; + let radius = (area as f32 / core::f32::consts::PI).sqrt(); + let mean_luma = threshold as f32 + ((sum_w as f32 / area as f32) - 1.0); + let area_score = (area as f32 / (config.tile_size * config.tile_size) as f32).min(1.0); + let brightness_score = peak as f32 / 255.0; + + Some(MoveMarkerCandidate { + source_id_hash: config.source_id_hash, + frame_sequence: config.frame_sequence, + tile_x, + tile_y, + center_x_px: center_x, + center_y_px: center_y, + radius_px: radius, + area_px: area, + mean_luma, + peak_luma: peak, + score: brightness_score * (0.65 + area_score * 0.35), + }) +} + +fn config_is_valid(config: MoveTrackerConfig) -> bool { + config.width > 0 + && config.height > 0 + && config.stride_bytes >= config.width + && config.tile_size == 16 + && config.min_area_px > 0 + && config.max_candidates > 0 +} + +fn required_frame_bytes(config: MoveTrackerConfig) -> Option { + let rows = usize::try_from(config.height).ok()?; + let stride = usize::try_from(config.stride_bytes).ok()?; + rows.checked_mul(stride) +} + +#[unsafe(no_mangle)] +pub extern "C" fn mimir_move_tracker_dispatch_plan( + config: MoveTrackerConfig, + out_plan: *mut MoveTrackerDispatchPlan, +) -> c_int { + if out_plan.is_null() { + return MIMIR_MOVE_TRACKER_NULL; + } + + let Some(plan) = dispatch_plan(config) else { + return MIMIR_MOVE_TRACKER_INVALID; + }; + + unsafe { + *out_plan = plan; + } + MIMIR_MOVE_TRACKER_OK +} + +#[unsafe(no_mangle)] +pub extern "C" fn mimir_move_tracker_extract_luma_cpu( + config: MoveTrackerConfig, + frame: *const u8, + frame_len: usize, + out_candidates: *mut MoveMarkerCandidate, + out_capacity: usize, + out_count: *mut usize, +) -> c_int { + if frame.is_null() || out_candidates.is_null() || out_count.is_null() { + return MIMIR_MOVE_TRACKER_NULL; + } + + let frame = unsafe { core::slice::from_raw_parts(frame, frame_len) }; + let Some(candidates) = extract_luma_candidates(frame, config) else { + return MIMIR_MOVE_TRACKER_INVALID; + }; + + let written = candidates.len().min(out_capacity); + unsafe { + core::ptr::copy_nonoverlapping(candidates.as_ptr(), out_candidates, written); + *out_count = written; + } + MIMIR_MOVE_TRACKER_OK +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn dispatch_plan_covers_partial_tiles() { + let config = MoveTrackerConfig { + width: 33, + height: 17, + stride_bytes: 40, + tile_size: 16, + ..MoveTrackerConfig::default() + }; + + let plan = dispatch_plan(config).expect("valid plan"); + assert_eq!(3, plan.groups_x); + assert_eq!(2, plan.groups_y); + assert_eq!(16, plan.group_size_x); + assert_eq!(16, plan.group_size_y); + } + + #[test] + fn extracts_weighted_marker_candidate() { + let config = MoveTrackerConfig { + width: 32, + height: 24, + stride_bytes: 32, + tile_size: 16, + threshold_min: 180, + min_area_px: 3, + max_candidates: 8, + source_id_hash: 42, + frame_sequence: 7, + }; + let mut frame = vec![0u8; (config.stride_bytes * config.height) as usize]; + for (x, y, luma) in [ + (10u32, 8u32, 230u8), + (11, 8, 250), + (10, 9, 240), + (11, 9, 255), + ] { + frame[(y * config.stride_bytes + x) as usize] = luma; + } + + let candidates = extract_luma_candidates(&frame, config).expect("candidates"); + assert_eq!(1, candidates.len()); + let candidate = candidates[0]; + assert_eq!(42, candidate.source_id_hash); + assert_eq!(7, candidate.frame_sequence); + assert_eq!(0, candidate.tile_x); + assert_eq!(0, candidate.tile_y); + assert_eq!(4, candidate.area_px); + assert_eq!(255, candidate.peak_luma); + assert!(candidate.center_x_px > 10.4 && candidate.center_x_px < 10.7); + assert!(candidate.center_y_px > 8.4 && candidate.center_y_px < 8.7); + assert!(candidate.score > 0.65); + } + + #[test] + fn rejects_short_frames_and_empty_config() { + assert!(extract_luma_candidates(&[], MoveTrackerConfig::default()).is_none()); + assert!( + dispatch_plan(MoveTrackerConfig { + width: 0, + ..MoveTrackerConfig::default() + }) + .is_none() + ); + } +} diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 22b0d7a..afc0c7e 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -136,6 +136,14 @@ That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01 output collection and refreshes the PS Move LED report so the USB-attached Move stays visible as a tracking/calibration marker. +`native/move_tracker` owns the first native/GPU-shaped Move optical extraction +stage. Rust owns dispatch planning, FFI, configuration validation, and a +deterministic CPU mirror; `MoveSphereCandidate.comp.hlsl` owns the parallel +16px-tile luma reduction that emits ranked sphere candidates. This stage owns +per-frame marker candidates only. Stereo triangulation, camera calibration, +controller association, IMU fusion, prediction, and final 6DoF pose remain +separate later owners. + ## Program Composition [[docs/mimir-program-composition|Mimir Program Composition]] is the live diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 1eb8cac..35b69a7 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -126,3 +126,4 @@ {"ts": "2026-06-10T16:41:42+01:00", "type": "implementation", "status": "ok", "note": "Added typed PS Move tracking observations as Mimir runtime streams: MimirStreamKind.Tracking, mimir.move_tracking_observation.v1, local Starfire and remote Nightwing Move capture profiles, Muninn producer identity, Odin discovery provenance, runtime buffer readout, and BufferSmoke proofs for Starfire/Nightwing USB-attached Moves entering Mimir buffers."} {"ts": "2026-06-10T17:38:22+01:00", "type": "verification", "status": "ok", "note": "Restored and launched the narrow Nightwing Move tracking witness path. scripts/start-nightwing-move-tracking.ps1 started Mimir.EveSensorReceiver on Starfire port 8796, staged Nightwing Eye/Move Python tools, launched Nightwing worker PID 1131578 with --move-light move-usb=/dev/hidraw1:#35ff6c, and VerseRecorder C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures\\20260610-173549 recorded mimir.psmove_light_state.v1 HID success plus mimir.move_controller_observation_state.v1 blobs from both ps3-eye-0 and ps3-eye-1."} {"ts": "2026-06-10T18:11:18+01:00", "type": "verification", "status": "ok", "note": "Started the Starfire-local USB Move light owner. Windows sees Sony VID_054C/PID_03D5 with col01 output reports; scripts/start-starfire-move-light.ps1 launched Mimir.PsMoveProbe PID 27344 with --rgb #35ff6c --hold-seconds 0 --refresh-ms 250, and artifacts/runtime/starfire-move-light-20260610-180943/move-light-events.jsonl shows repeated mimir.psmove_usb_pulse_event.v1 hold writes succeeding with ok rgb=#35FF6C bytes=49."} +{"ts": "2026-06-10T20:33:54+01:00", "type": "implementation", "status": "ok", "note": "Added native/move_tracker as Rust-orchestrated PS Move optical candidate extraction: Rust owns dispatch/FFI/CPU mirror; HLSL owns 16px-tile luma reduction. Scope is marker candidates only, before calibration/triangulation/IMU fusion/prediction."} diff --git a/state/map.yaml b/state/map.yaml index 23e3099..0508890 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -91,6 +91,13 @@ accepted_design: blob extraction, and the stream currently emits `mimir.psmove_light_state.v1` plus `mimir.move_controller_observation_state.v1` optical witness records before later fusion into full tracking poses. + - `native/move_tracker` owns the first Rust/native Move optical candidate + stage: Rust validates config, plans dispatch, exposes FFI, and mirrors the + compute path on CPU; `MoveSphereCandidate.comp.hlsl` reduces one 16px luma + tile per thread group into weighted sphere candidates. This stage emits + marker candidates only. Calibration, multi-camera association, + triangulation, IMU fusion, prediction, and final 6DoF pose are still + separate owners. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the From 17d2f56de89b08c8a52fdfc99ca9b1c75ef60836 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 20:36:21 +0100 Subject: [PATCH 013/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 65126 -> 65126 bytes .voidbot/voice/identity.json | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 2d566d42fc44b0a28dea98c9b5c2af50a49f7afa..72bb68947696d33c88151036f86e233babc809c4 100644 GIT binary patch delta 161 zcmaF%hxyqb<_+I@c`U7rEUXNT^h}M-qb4$nPL|;7go+vKSs0mt#5Nz|(=*_LC^XSC zGn~xGC%#$G{+bR{p^2WQ1w0ESsDt^fc4 delta 136 zcmaF%hxyqb<_+I@C)ellOk@ Date: Wed, 10 Jun 2026 20:51:58 +0100 Subject: [PATCH 014/135] Demote Move marker extraction to Muninn --- docs/implementation-plan.md | 12 +- native/move_tracker/Cargo.lock | 7 - native/move_tracker/Cargo.toml | 10 - native/move_tracker/README.md | 23 -- .../shaders/MoveSphereCandidate.comp.hlsl | 104 ------ native/move_tracker/src/lib.rs | 308 ------------------ notes/current-system-map.md | 14 +- state/evidence.jsonl | 1 + state/map.yaml | 13 +- 9 files changed, 20 insertions(+), 472 deletions(-) delete mode 100644 native/move_tracker/Cargo.lock delete mode 100644 native/move_tracker/Cargo.toml delete mode 100644 native/move_tracker/README.md delete mode 100644 native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl delete mode 100644 native/move_tracker/src/lib.rs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 8c5b816..966da8d 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -50,12 +50,12 @@ a named invariant that the native runtime cannot protect yet. owner. It launches `Mimir.PsMoveProbe` in steady hold mode so the locally attached `VID_054C/PID_03D5` Move sphere stays lit for calibration/tracking receipts. -- `native/move_tracker` is the first native/Rust Move optical candidate - extractor. Rust owns dispatch planning, FFI, validation, and a deterministic - CPU mirror; `shaders/MoveSphereCandidate.comp.hlsl` owns one 16px tile per - thread group and emits weighted bright-sphere candidates. This is marker - extraction only, not final pose, stereo triangulation, calibration, - association, IMU fusion, or prediction. +- Odin's Muninn organ owns Move optical candidate extraction in + `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds + `muninn.move_marker_candidate.v1` records. Mimir consumes those candidate + streams; it does not own raw optical extraction. Final pose, stereo + triangulation, calibration, association, IMU fusion, and prediction remain + separate later owners. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe diff --git a/native/move_tracker/Cargo.lock b/native/move_tracker/Cargo.lock deleted file mode 100644 index 8843462..0000000 --- a/native/move_tracker/Cargo.lock +++ /dev/null @@ -1,7 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "mimir-move-tracker" -version = "0.1.0" diff --git a/native/move_tracker/Cargo.toml b/native/move_tracker/Cargo.toml deleted file mode 100644 index ff0060e..0000000 --- a/native/move_tracker/Cargo.toml +++ /dev/null @@ -1,10 +0,0 @@ -[package] -name = "mimir-move-tracker" -version = "0.1.0" -edition = "2024" -description = "Rust orchestration and CPU mirror for PS Move sphere tracking compute" - -[lib] -crate-type = ["rlib", "cdylib", "staticlib"] - -[dependencies] diff --git a/native/move_tracker/README.md b/native/move_tracker/README.md deleted file mode 100644 index cc9e954..0000000 --- a/native/move_tracker/README.md +++ /dev/null @@ -1,23 +0,0 @@ -# Mimir Move Tracker - -Rust owns orchestration for PS Move sphere tracking. The compute shader owns the -parallel image reduction stage. This crate deliberately stops at per-frame -marker candidates: final pose, stereo triangulation, IMU fusion, prediction, -and calibration remain separate owners. - -## Current Contract - -- Input: one luma frame (`Y8`) with width, height, stride, source hash, and frame - sequence. -- GPU stage: one 16px tile per thread group, emitting at most one candidate per - tile. -- Output: `MoveMarkerCandidate` records containing weighted centroid, radius, - area, peak/mean luma, and score. -- Rust mirror: deterministic CPU implementation and FFI wrappers for tests and - non-GPU smoke paths. -- Shader IDs are carried as `uint2` low/high pairs to avoid making 64-bit HLSL - integer support a dependency of the first extraction pass. - -This is the first stage required for PlayStation-class wand tracking. The -remaining stack needs calibrated camera rays, controlled sphere colors/exposure, -multi-camera association, triangulation, IMU fusion, and latency prediction. diff --git a/native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl b/native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl deleted file mode 100644 index d00414f..0000000 --- a/native/move_tracker/shaders/MoveSphereCandidate.comp.hlsl +++ /dev/null @@ -1,104 +0,0 @@ -// Mimir PS Move sphere candidate extraction. -// One thread group owns one image tile. The shader emits at most one bright -// marker candidate per 16px tile; Rust owns dispatch sizing and downstream ranking. - -struct MoveSphereCandidate -{ - uint2 sourceIdHash; - uint2 frameSequence; - uint tileX; - uint tileY; - float centerXPx; - float centerYPx; - float radiusPx; - uint areaPx; - float meanLuma; - uint peakLuma; - float score; -}; - -cbuffer MoveTrackerConfig : register(b0) -{ - uint width; - uint height; - uint strideBytes; - uint tileSize; - uint thresholdMin; - uint minAreaPx; - uint maxCandidates; - uint _pad0; - uint2 sourceIdHash; - uint2 frameSequence; -}; - -ByteAddressBuffer frameY8 : register(t0); -RWStructuredBuffer candidates : register(u0); -RWByteAddressBuffer candidateCounter : register(u1); - -groupshared uint gArea; -groupshared uint gSumX; -groupshared uint gSumY; -groupshared uint gSumW; -groupshared uint gPeak; - -[numthreads(16, 16, 1)] -void main(uint3 groupId : SV_GroupID, uint3 groupThreadId : SV_GroupThreadID) -{ - if (groupThreadId.x == 0 && groupThreadId.y == 0) - { - gArea = 0; - gSumX = 0; - gSumY = 0; - gSumW = 0; - gPeak = 0; - } - GroupMemoryBarrierWithGroupSync(); - - uint x = groupId.x * tileSize + groupThreadId.x; - uint y = groupId.y * tileSize + groupThreadId.y; - if (groupThreadId.x < tileSize && groupThreadId.y < tileSize && x < width && y < height) - { - uint byteOffset = y * strideBytes + x; - uint packed = frameY8.Load(byteOffset & ~3u); - uint shift = (byteOffset & 3u) * 8u; - uint luma = (packed >> shift) & 0xffu; - if (luma >= thresholdMin) - { - uint weight = luma - thresholdMin + 1u; - InterlockedAdd(gArea, 1u); - InterlockedAdd(gSumX, x * weight); - InterlockedAdd(gSumY, y * weight); - InterlockedAdd(gSumW, weight); - InterlockedMax(gPeak, luma); - } - } - - GroupMemoryBarrierWithGroupSync(); - - if (groupThreadId.x == 0 && groupThreadId.y == 0 && gArea >= minAreaPx && gSumW > 0) - { - uint index; - candidateCounter.InterlockedAdd(0, 1, index); - if (index < maxCandidates) - { - float area = (float)gArea; - float tilePixels = (float)(tileSize * tileSize); - float areaScore = min(1.0, area / tilePixels); - float brightnessScore = (float)gPeak / 255.0; - - MoveSphereCandidate candidate; - candidate.sourceIdHash = sourceIdHash; - candidate.frameSequence = frameSequence; - candidate.tileX = groupId.x; - candidate.tileY = groupId.y; - candidate.centerXPx = (float)gSumX / (float)gSumW; - candidate.centerYPx = (float)gSumY / (float)gSumW; - candidate.radiusPx = sqrt(area / 3.14159265358979323846); - candidate.areaPx = gArea; - candidate.meanLuma = (float)thresholdMin + (((float)gSumW / area) - 1.0); - candidate.peakLuma = gPeak; - candidate.score = brightnessScore * (0.65 + areaScore * 0.35); - candidates[index] = candidate; - } - } -} diff --git a/native/move_tracker/src/lib.rs b/native/move_tracker/src/lib.rs deleted file mode 100644 index cf59ba2..0000000 --- a/native/move_tracker/src/lib.rs +++ /dev/null @@ -1,308 +0,0 @@ -#![deny(unsafe_op_in_unsafe_fn)] - -use core::ffi::c_int; - -pub const MIMIR_MOVE_TRACKER_OK: c_int = 0; -pub const MIMIR_MOVE_TRACKER_NULL: c_int = -1; -pub const MIMIR_MOVE_TRACKER_INVALID: c_int = -2; - -#[repr(C)] -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct MoveTrackerConfig { - pub width: u32, - pub height: u32, - pub stride_bytes: u32, - pub tile_size: u32, - pub threshold_min: u8, - pub min_area_px: u32, - pub max_candidates: u32, - pub source_id_hash: u64, - pub frame_sequence: u64, -} - -impl Default for MoveTrackerConfig { - fn default() -> Self { - Self { - width: 320, - height: 240, - stride_bytes: 320, - tile_size: 16, - threshold_min: 180, - min_area_px: 4, - max_candidates: 64, - source_id_hash: 0, - frame_sequence: 0, - } - } -} - -#[repr(C)] -#[derive(Debug, Clone, Copy, PartialEq)] -pub struct MoveTrackerDispatchPlan { - pub group_size_x: u32, - pub group_size_y: u32, - pub groups_x: u32, - pub groups_y: u32, - pub max_candidates: u32, -} - -#[repr(C)] -#[derive(Debug, Clone, Copy, PartialEq)] -pub struct MoveMarkerCandidate { - pub source_id_hash: u64, - pub frame_sequence: u64, - pub tile_x: u32, - pub tile_y: u32, - pub center_x_px: f32, - pub center_y_px: f32, - pub radius_px: f32, - pub area_px: u32, - pub mean_luma: f32, - pub peak_luma: u32, - pub score: f32, -} - -pub fn dispatch_plan(config: MoveTrackerConfig) -> Option { - if !config_is_valid(config) { - return None; - } - - let tile = config.tile_size; - Some(MoveTrackerDispatchPlan { - group_size_x: tile, - group_size_y: tile, - groups_x: config.width.div_ceil(tile), - groups_y: config.height.div_ceil(tile), - max_candidates: config.max_candidates, - }) -} - -pub fn extract_luma_candidates( - frame: &[u8], - config: MoveTrackerConfig, -) -> Option> { - if !config_is_valid(config) { - return None; - } - - let required = required_frame_bytes(config)?; - if frame.len() < required { - return None; - } - - let mut candidates = Vec::new(); - let tile = config.tile_size; - let groups_x = config.width.div_ceil(tile); - let groups_y = config.height.div_ceil(tile); - - for tile_y in 0..groups_y { - for tile_x in 0..groups_x { - if candidates.len() >= config.max_candidates as usize { - return Some(candidates); - } - - if let Some(candidate) = extract_tile(frame, config, tile_x, tile_y) { - candidates.push(candidate); - } - } - } - - candidates.sort_by(|left, right| { - right - .score - .partial_cmp(&left.score) - .unwrap_or(core::cmp::Ordering::Equal) - }); - Some(candidates) -} - -fn extract_tile( - frame: &[u8], - config: MoveTrackerConfig, - tile_x: u32, - tile_y: u32, -) -> Option { - let x0 = tile_x * config.tile_size; - let y0 = tile_y * config.tile_size; - let x1 = (x0 + config.tile_size).min(config.width); - let y1 = (y0 + config.tile_size).min(config.height); - let threshold = u32::from(config.threshold_min); - let mut area = 0u32; - let mut sum_x = 0u64; - let mut sum_y = 0u64; - let mut sum_w = 0u64; - let mut peak = 0u32; - - for y in y0..y1 { - let row = y as usize * config.stride_bytes as usize; - for x in x0..x1 { - let luma = u32::from(frame[row + x as usize]); - if luma < threshold { - continue; - } - - area += 1; - peak = peak.max(luma); - let weight = u64::from(luma - threshold + 1); - sum_x += u64::from(x) * weight; - sum_y += u64::from(y) * weight; - sum_w += weight; - } - } - - if area < config.min_area_px || sum_w == 0 { - return None; - } - - let center_x = sum_x as f32 / sum_w as f32; - let center_y = sum_y as f32 / sum_w as f32; - let radius = (area as f32 / core::f32::consts::PI).sqrt(); - let mean_luma = threshold as f32 + ((sum_w as f32 / area as f32) - 1.0); - let area_score = (area as f32 / (config.tile_size * config.tile_size) as f32).min(1.0); - let brightness_score = peak as f32 / 255.0; - - Some(MoveMarkerCandidate { - source_id_hash: config.source_id_hash, - frame_sequence: config.frame_sequence, - tile_x, - tile_y, - center_x_px: center_x, - center_y_px: center_y, - radius_px: radius, - area_px: area, - mean_luma, - peak_luma: peak, - score: brightness_score * (0.65 + area_score * 0.35), - }) -} - -fn config_is_valid(config: MoveTrackerConfig) -> bool { - config.width > 0 - && config.height > 0 - && config.stride_bytes >= config.width - && config.tile_size == 16 - && config.min_area_px > 0 - && config.max_candidates > 0 -} - -fn required_frame_bytes(config: MoveTrackerConfig) -> Option { - let rows = usize::try_from(config.height).ok()?; - let stride = usize::try_from(config.stride_bytes).ok()?; - rows.checked_mul(stride) -} - -#[unsafe(no_mangle)] -pub extern "C" fn mimir_move_tracker_dispatch_plan( - config: MoveTrackerConfig, - out_plan: *mut MoveTrackerDispatchPlan, -) -> c_int { - if out_plan.is_null() { - return MIMIR_MOVE_TRACKER_NULL; - } - - let Some(plan) = dispatch_plan(config) else { - return MIMIR_MOVE_TRACKER_INVALID; - }; - - unsafe { - *out_plan = plan; - } - MIMIR_MOVE_TRACKER_OK -} - -#[unsafe(no_mangle)] -pub extern "C" fn mimir_move_tracker_extract_luma_cpu( - config: MoveTrackerConfig, - frame: *const u8, - frame_len: usize, - out_candidates: *mut MoveMarkerCandidate, - out_capacity: usize, - out_count: *mut usize, -) -> c_int { - if frame.is_null() || out_candidates.is_null() || out_count.is_null() { - return MIMIR_MOVE_TRACKER_NULL; - } - - let frame = unsafe { core::slice::from_raw_parts(frame, frame_len) }; - let Some(candidates) = extract_luma_candidates(frame, config) else { - return MIMIR_MOVE_TRACKER_INVALID; - }; - - let written = candidates.len().min(out_capacity); - unsafe { - core::ptr::copy_nonoverlapping(candidates.as_ptr(), out_candidates, written); - *out_count = written; - } - MIMIR_MOVE_TRACKER_OK -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn dispatch_plan_covers_partial_tiles() { - let config = MoveTrackerConfig { - width: 33, - height: 17, - stride_bytes: 40, - tile_size: 16, - ..MoveTrackerConfig::default() - }; - - let plan = dispatch_plan(config).expect("valid plan"); - assert_eq!(3, plan.groups_x); - assert_eq!(2, plan.groups_y); - assert_eq!(16, plan.group_size_x); - assert_eq!(16, plan.group_size_y); - } - - #[test] - fn extracts_weighted_marker_candidate() { - let config = MoveTrackerConfig { - width: 32, - height: 24, - stride_bytes: 32, - tile_size: 16, - threshold_min: 180, - min_area_px: 3, - max_candidates: 8, - source_id_hash: 42, - frame_sequence: 7, - }; - let mut frame = vec![0u8; (config.stride_bytes * config.height) as usize]; - for (x, y, luma) in [ - (10u32, 8u32, 230u8), - (11, 8, 250), - (10, 9, 240), - (11, 9, 255), - ] { - frame[(y * config.stride_bytes + x) as usize] = luma; - } - - let candidates = extract_luma_candidates(&frame, config).expect("candidates"); - assert_eq!(1, candidates.len()); - let candidate = candidates[0]; - assert_eq!(42, candidate.source_id_hash); - assert_eq!(7, candidate.frame_sequence); - assert_eq!(0, candidate.tile_x); - assert_eq!(0, candidate.tile_y); - assert_eq!(4, candidate.area_px); - assert_eq!(255, candidate.peak_luma); - assert!(candidate.center_x_px > 10.4 && candidate.center_x_px < 10.7); - assert!(candidate.center_y_px > 8.4 && candidate.center_y_px < 8.7); - assert!(candidate.score > 0.65); - } - - #[test] - fn rejects_short_frames_and_empty_config() { - assert!(extract_luma_candidates(&[], MoveTrackerConfig::default()).is_none()); - assert!( - dispatch_plan(MoveTrackerConfig { - width: 0, - ..MoveTrackerConfig::default() - }) - .is_none() - ); - } -} diff --git a/notes/current-system-map.md b/notes/current-system-map.md index afc0c7e..2a36503 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -136,13 +136,13 @@ That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01 output collection and refreshes the PS Move LED report so the USB-attached Move stays visible as a tracking/calibration marker. -`native/move_tracker` owns the first native/GPU-shaped Move optical extraction -stage. Rust owns dispatch planning, FFI, configuration validation, and a -deterministic CPU mirror; `MoveSphereCandidate.comp.hlsl` owns the parallel -16px-tile luma reduction that emits ranked sphere candidates. This stage owns -per-frame marker candidates only. Stereo triangulation, camera calibration, -controller association, IMU fusion, prediction, and final 6DoF pose remain -separate later owners. +Odin's Muninn organ owns the Move optical extraction stage for sensor stream +exposure. Its Rust crate lives in `E:\Projects\Odin\crates\muninn-move-tracker` +and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes +those Muninn candidate streams into tracking buffers; it does not own raw +optical extraction. Stereo triangulation, camera calibration, controller +association, IMU fusion, prediction, and final 6DoF pose remain separate later +owners. ## Program Composition diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 35b69a7..3d0b9c1 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -127,3 +127,4 @@ {"ts": "2026-06-10T17:38:22+01:00", "type": "verification", "status": "ok", "note": "Restored and launched the narrow Nightwing Move tracking witness path. scripts/start-nightwing-move-tracking.ps1 started Mimir.EveSensorReceiver on Starfire port 8796, staged Nightwing Eye/Move Python tools, launched Nightwing worker PID 1131578 with --move-light move-usb=/dev/hidraw1:#35ff6c, and VerseRecorder C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures\\20260610-173549 recorded mimir.psmove_light_state.v1 HID success plus mimir.move_controller_observation_state.v1 blobs from both ps3-eye-0 and ps3-eye-1."} {"ts": "2026-06-10T18:11:18+01:00", "type": "verification", "status": "ok", "note": "Started the Starfire-local USB Move light owner. Windows sees Sony VID_054C/PID_03D5 with col01 output reports; scripts/start-starfire-move-light.ps1 launched Mimir.PsMoveProbe PID 27344 with --rgb #35ff6c --hold-seconds 0 --refresh-ms 250, and artifacts/runtime/starfire-move-light-20260610-180943/move-light-events.jsonl shows repeated mimir.psmove_usb_pulse_event.v1 hold writes succeeding with ok rgb=#35FF6C bytes=49."} {"ts": "2026-06-10T20:33:54+01:00", "type": "implementation", "status": "ok", "note": "Added native/move_tracker as Rust-orchestrated PS Move optical candidate extraction: Rust owns dispatch/FFI/CPU mirror; HLSL owns 16px-tile luma reduction. Scope is marker candidates only, before calibration/triangulation/IMU fusion/prediction."} +{"ts": "2026-06-10T20:49:06+01:00", "type": "decision", "status": "ok", "note": "Corrected Move optical candidate ownership: the Rust/HLSL tracker moved out of Mimir and into Odin's Muninn organ at E:\\Projects\\Odin\\crates\\muninn-move-tracker. Mimir consumes muninn.move_marker_candidate.v1 streams and no longer owns raw optical extraction."} diff --git a/state/map.yaml b/state/map.yaml index 0508890..0ec7c31 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -91,13 +91,12 @@ accepted_design: blob extraction, and the stream currently emits `mimir.psmove_light_state.v1` plus `mimir.move_controller_observation_state.v1` optical witness records before later fusion into full tracking poses. - - `native/move_tracker` owns the first Rust/native Move optical candidate - stage: Rust validates config, plans dispatch, exposes FFI, and mirrors the - compute path on CPU; `MoveSphereCandidate.comp.hlsl` reduces one 16px luma - tile per thread group into weighted sphere candidates. This stage emits - marker candidates only. Calibration, multi-camera association, - triangulation, IMU fusion, prediction, and final 6DoF pose are still - separate owners. + - Odin's Muninn organ owns Move optical candidate extraction in + `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds + `muninn.move_marker_candidate.v1` records. Mimir consumes those candidate + streams; it does not own raw optical extraction. Calibration, multi-camera + association, triangulation, IMU fusion, prediction, and final 6DoF pose are + still separate owners. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the From 147e6ab5afbbeafbaec26be86699a9b6ffcfcca1 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 10 Jun 2026 20:52:58 +0100 Subject: [PATCH 015/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 65126 -> 65213 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 72bb68947696d33c88151036f86e233babc809c4..7ef31ed7bd5557bbebe4a088de51f96b388703c8 100644 GIT binary patch delta 265 zcmaF%hk5T`<_$l2xlAmrOicAGEe)e4GKx-?;Ohp9nCTgqn1Dn!AL27GfC*Wef`tU_ zZ|FdU%q+pGCUfK{GVYkHl_M?Jx-zraFR`E?GcWz{=H$|%qSU++AV0Vyv8W_9WpaFu zwAKcMuxnmQv17>&BLgEdT>~>+LxT__11lqADdCtRSxO=As;4MsA>6 tVa}cWfL9Xg2y-*2Q*))2U_#~)cm3&ago#)hfgL|*!hu|+V$nOW?YSWu9emwssSiX0hMLrW_Yi_IryAy5|e``G(uIFnOK5Ve4cP1 L7b;|KU=#%aQgJwd From ec0f7454d39129deabd37a8deebf42daf3ad6b8e Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 07:57:59 +0100 Subject: [PATCH 016/135] Persist Mimir face state --- .voidbot/state/mimir.cc | Bin 65213 -> 65126 bytes .voidbot/voice/identity.json | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 7ef31ed7bd5557bbebe4a088de51f96b388703c8..1c6c8aa07e5e6042e191b194e189ae29555d1fe6 100644 GIT binary patch delta 272 zcmdn{m-*Qr<_%wX`3*x1%&bhztc)%7Oij$9CNhdnmf&kc7B)9D0ts(E#HVe*3sY`l ztY>OC`D3>DW4S?EcQz*D9Fr9KeRqCwW0*b z2`))2Dk*j>*rJ-rm283eQyp&>8l|}|uM#eCelLK?4_zXfo0x+@7 zl{w6eKrv+hO@6>Dg6vjvGpOHk#gtIQ%^`mH(_e!kY-t4c@|+2~bCJa@j18jzMyp{Q diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index 661c79f..2d86c39 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-06-10T19:28:11.463Z", + "createdOrRefreshedAt": "2026-06-11T06:46:39.463Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 4d661967ae842343c66434bf4b2a19f10bb625bb Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 07:58:26 +0100 Subject: [PATCH 017/135] Route Move light ownership through Muninn --- docs/implementation-plan.md | 15 +++++++++------ notes/current-system-map.md | 20 +++++++++++++------- scripts/start-nightwing-move-tracking.ps1 | 14 +++++++------- state/evidence.jsonl | 1 + state/map.yaml | 4 ++++ tools/nightwing_typed_witness_publisher.py | 4 ++++ 6 files changed, 38 insertions(+), 20 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 966da8d..f3c69db 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -41,15 +41,18 @@ a named invariant that the native runtime cannot protect yet. schema projection, not pose capture or Mimir fusion. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, - stages the Nightwing Eye/Move Python worker, lights configured USB Moves via - HID report `0x06`, and publishes `mimir.psmove_light_state.v1` plus + stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field + liveness, and publishes `mimir.psmove_light_state.v1` plus `mimir.move_controller_observation_state.v1` blob observations from both PS3 Eyes. The blob stream is optical witness evidence for later pose fusion, not the final 6DoF pose owner. -- `scripts/start-starfire-move-light.ps1` is the Starfire-local USB Move light - owner. It launches `Mimir.PsMoveProbe` in steady hold mode so the locally - attached `VID_054C/PID_03D5` Move sphere stays lit for calibration/tracking - receipts. +- Structured PS Move light pulses are Muninn output commands. Mimir publishes + `muninn.move_light_command.v1` over CultNet/CultMesh to the Muninn daemon on + the host that owns the USB-attached Move. Muninn writes PS Move HID report + `0x06` locally and updates command state/receipt. + `scripts/start-starfire-move-light.ps1` and Nightwing direct HID writes are + now smoke/bootstrap paths for hardware proof when Muninn is not yet running + on that host. - Odin's Muninn organ owns Move optical candidate extraction in `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes those candidate diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 2a36503..9ffc0bf 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -110,6 +110,11 @@ Authority: - Starfire-local Move tracking is a local Mimir/Muninn tracking stream. - Nightwing Move tracking is emitted by Muninn on Nightwing over the typed observation path. +- Structured PS Move light pulses are Muninn output commands. Mimir requests + them by sending `muninn.move_light_command.v1` over CultNet/CultMesh to the + Muninn daemon on the host with the USB-attached Move; Mimir does not write + remote HID directly except through temporary smoke scripts used to prove + hardware behavior before Muninn is available on that host. - Odin discovers and projects the stream/schema surface for operator and agent access. - Mimir owns subscription, rolling retention, clock alignment, and later fusion @@ -123,18 +128,19 @@ Move observation into two tracking buffers. Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages `nw_eye_cap.py`, `nw_move_hint.py`, and `nightwing_typed_witness_publisher.py` -onto Nightwing, and launches the publisher with `--track-eyes` plus explicit -`--move-light name=/dev/hidrawN:#rrggbb` HID ownership. The worker keeps the -USB Move sphere lit with PS Move report `0x06`, emits -`mimir.psmove_light_state.v1`, and publishes compact +onto Nightwing, and launches the publisher with `--track-eyes` plus a heartbeat +freshness file. The publisher can still accept +`--move-light name=/dev/hidrawN:#rrggbb` as an explicit smoke/bootstrap edge, +but runtime structured light commands belong to Muninn. The worker publishes +compact `mimir.move_controller_observation_state.v1` blob observations from both PS3 Eyes. Those Eye/Move observations are the live optical witness feeding the later pose/fusion owner; they are not themselves the final 6DoF pose authority. -Starfire-local Move illumination uses `scripts/start-starfire-move-light.ps1`. +Starfire-local Move illumination smoke uses `scripts/start-starfire-move-light.ps1`. That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01 -output collection and refreshes the PS Move LED report so the USB-attached Move -stays visible as a tracking/calibration marker. +output collection and refreshes the PS Move LED report for hardware proof when +Muninn is not available on Starfire yet. Odin's Muninn organ owns the Move optical extraction stage for sensor stream exposure. Its Rust crate lives in `E:\Projects\Odin\crates\muninn-move-tracker` diff --git a/scripts/start-nightwing-move-tracking.ps1 b/scripts/start-nightwing-move-tracking.ps1 index f3397d5..1d8c9cd 100644 --- a/scripts/start-nightwing-move-tracking.ps1 +++ b/scripts/start-nightwing-move-tracking.ps1 @@ -4,9 +4,7 @@ param( [string]$StarfireHost = "192.168.1.66", [string]$Path = "/eve/periwinkle", [string]$EyeDevices = "/dev/video2,/dev/video3", - [string[]]$MoveLight = @( - "move-usb=/dev/hidraw1:#35ff6c" - ), + [string[]]$MoveLight = @(), [string]$RecorderRoot = "C:\Users\Meta\Videos\Mimir\VerseCaptures", [switch]$SkipRecorder, [switch]$DryRun @@ -18,6 +16,9 @@ $runId = Get-Date -Format "yyyyMMdd-HHmmss" $runDir = Join-Path $repo "artifacts\runtime\nightwing-move-tracking-$runId" $witnessUrl = "ws://$StarfireHost`:$Port$Path" $subscribeUrl = "ws://127.0.0.1:$Port$Path/subscribe" +$remoteLog = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.log" +$remotePid = "~/.local/state/gamecult/nightwing-typed-witness.pid" +$remoteHeartbeat = "~/.local/state/gamecult/nightwing-typed-witness.heartbeat" New-Item -ItemType Directory -Force -Path $runDir | Out-Null New-Item -ItemType Directory -Force -Path $RecorderRoot | Out-Null @@ -126,13 +127,11 @@ if ($DryRun) { scp (Join-Path $repo "tools\nw_move_hint.py") "${NightwingHost}:/tmp/nw_move_hint.py" | Out-Null scp (Join-Path $repo "tools\nightwing_typed_witness_publisher.py") "${NightwingHost}:/tmp/nightwing_typed_witness_publisher.py" | Out-Null - $remoteLog = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.log" - $remotePid = "~/.local/state/gamecult/mimir-nightwing-move-tracking-$runId.pid" $moveArgs = ($MoveLight | ForEach-Object { "--move-light '$($_ -replace "'", "'\''")'" }) -join " " $remoteScript = @" mkdir -p ~/.local/state/gamecult pkill -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' || true -nohup python3 /tmp/nightwing_typed_witness_publisher.py --url '$witnessUrl' --track-eyes --track-builtin-camera --track-builtin-mic --interval 0.10 --eye-devices '$EyeDevices' --eye-window-seconds 0.08 --tracking-stride 6 $moveArgs > $remoteLog 2>&1 < /dev/null & +nohup python3 /tmp/nightwing_typed_witness_publisher.py --url '$witnessUrl' --track-eyes --track-builtin-camera --track-builtin-mic --interval 0.10 --heartbeat-path '$remoteHeartbeat' --eye-devices '$EyeDevices' --eye-window-seconds 0.08 --tracking-stride 6 $moveArgs > $remoteLog 2>&1 < /dev/null & sleep 0.5 pgrep -n -f '/tmp/nightwing_typed_witness_publisher.py.*--track-eyes' > $remotePid cat $remotePid @@ -150,8 +149,9 @@ $manifest = [ordered]@{ runDir = $runDir witnessUrl = $witnessUrl subscribeUrl = $subscribeUrl + heartbeatPath = $remoteHeartbeat eyeDevices = $EyeDevices - moveLights = $MoveLight + directHidSmokeMoveLights = $MoveLight startedAt = (Get-Date).ToString("o") } $manifest | ConvertTo-Json -Depth 5 | Set-Content -Path (Join-Path $runDir "supervisor.json") diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 3d0b9c1..9f5c7d5 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -128,3 +128,4 @@ {"ts": "2026-06-10T18:11:18+01:00", "type": "verification", "status": "ok", "note": "Started the Starfire-local USB Move light owner. Windows sees Sony VID_054C/PID_03D5 with col01 output reports; scripts/start-starfire-move-light.ps1 launched Mimir.PsMoveProbe PID 27344 with --rgb #35ff6c --hold-seconds 0 --refresh-ms 250, and artifacts/runtime/starfire-move-light-20260610-180943/move-light-events.jsonl shows repeated mimir.psmove_usb_pulse_event.v1 hold writes succeeding with ok rgb=#35FF6C bytes=49."} {"ts": "2026-06-10T20:33:54+01:00", "type": "implementation", "status": "ok", "note": "Added native/move_tracker as Rust-orchestrated PS Move optical candidate extraction: Rust owns dispatch/FFI/CPU mirror; HLSL owns 16px-tile luma reduction. Scope is marker candidates only, before calibration/triangulation/IMU fusion/prediction."} {"ts": "2026-06-10T20:49:06+01:00", "type": "decision", "status": "ok", "note": "Corrected Move optical candidate ownership: the Rust/HLSL tracker moved out of Mimir and into Odin's Muninn organ at E:\\Projects\\Odin\\crates\\muninn-move-tracker. Mimir consumes muninn.move_marker_candidate.v1 streams and no longer owns raw optical extraction."} +{"ts": "2026-06-11T07:57:05+01:00", "type": "decision", "status": "ok", "note": "PS Move structured light ownership moved to Odin/Muninn: Mimir requests muninn.move_light_command.v1 over CultNet/CultMesh; Muninn serve consumes host-addressed pending commands, writes local HID report 0x06, and updates command state. Nightwing direct HID launcher path is opt-in smoke/bootstrap only."} diff --git a/state/map.yaml b/state/map.yaml index 0ec7c31..639b97c 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -69,6 +69,10 @@ invariants: - Move tracking observations are stream samples, not program controls or dashboard truth. Starfire-local and Nightwing-remote Move samples must enter Mimir.Runtime as typed tracking buffers before Fensalir or Eve acts on them. + - Structured PS Move light pulses are Muninn output commands. Mimir requests + `muninn.move_light_command.v1` over CultNet/CultMesh from the Muninn daemon + on the host with the USB-attached Move; direct HID writes in Mimir scripts + are temporary smoke/bootstrap edges only. accepted_design: - `Mimir.slnx` owns the app surface. diff --git a/tools/nightwing_typed_witness_publisher.py b/tools/nightwing_typed_witness_publisher.py index 8b4648f..1fc8afa 100644 --- a/tools/nightwing_typed_witness_publisher.py +++ b/tools/nightwing_typed_witness_publisher.py @@ -447,6 +447,9 @@ def run(args: argparse.Namespace) -> None: if args.track_eyes: for payload in eye_tracking_observations(args, sequence): ws.send_binary(payload) + if args.heartbeat_path: + with open(args.heartbeat_path, "w", encoding="utf-8") as handle: + handle.write(f"{int(time.time())} {sequence}\n") sequence += 1 if args.once: return @@ -464,6 +467,7 @@ def main() -> int: parser.add_argument("--interval", type=float, default=0.25) parser.add_argument("--reconnect-seconds", type=float, default=2.0) parser.add_argument("--once", action="store_true") + parser.add_argument("--heartbeat-path", default="", help="Write a small freshness witness after successful sends.") parser.add_argument("--track-eyes", action="store_true", help="Publish compact Nightwing-local PS3 Eye Move-sphere observations.") parser.add_argument("--track-builtin-camera", action="store_true", help="Publish compact status for Nightwing-local non-Eye camera devices.") parser.add_argument("--track-builtin-mic", action="store_true", help="Publish compact ALSA microphone level observations.") From 231573ed79ce1425939363306ce6b1e99c735ac3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 17:00:49 +0100 Subject: [PATCH 018/135] Move wand fusion authority to Mimir --- docs/implementation-plan.md | 17 +++--- docs/perfect-machine-module-library.md | 12 ++-- notes/current-system-map.md | 42 ++++++++------ src/Mimir.BufferSmoke/Program.cs | 19 ++++++- .../Synchronization/MimirModuleLibrary.cs | 10 +++- .../MimirTrackingObservation.cs | 56 +++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 25 +++++---- 8 files changed, 138 insertions(+), 44 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index f3c69db..6010b14 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -34,11 +34,14 @@ a named invariant that the native runtime cannot protect yet. and video/tracking streams. - `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers. - `MimirStreamKind.Tracking` plus `MimirTrackingObservation` carry PS Move - pose/button/battery/confidence samples as typed - `mimir.move_tracking_observation.v1` documents. Starfire and Nightwing both - have USB-attached Moves; Starfire can feed local tracking streams, while - Muninn on Nightwing publishes remote tracking streams. Odin owns discovery and - schema projection, not pose capture or Mimir fusion. + evidence samples as typed `mimir.move_tracking_observation.v1` documents. + Starfire and Nightwing both have USB-attached Moves. Muninn daemons publish + source-local glowing-orb marker candidates and controller/IMU/button state; + Odin owns discovery and schema projection, not pose fusion. +- `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract. + Mimir owns calibration, association, triangulation, IMU fusion, prediction, + confidence, and latency accounting for those poses before Fensalir consumes + them as interactive controller input. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field @@ -57,8 +60,8 @@ a named invariant that the native runtime cannot protect yet. `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes those candidate streams; it does not own raw optical extraction. Final pose, stereo - triangulation, calibration, association, IMU fusion, and prediction remain - separate later owners. + triangulation, calibration, association, IMU fusion, and prediction belong to + Mimir. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index 1c6a313..1e30e44 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -13,7 +13,8 @@ job-interview answer for why it belongs in the body of the machine. | Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` | | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | -| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, and Move tracking observation documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | +| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | +| Move controller fusion | `Mimir.Runtime` + Fensalir | Mimir fuses Muninn marker candidates plus controller/IMU/button feeds into `mimir.move_controller_pose.v1`; Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -42,9 +43,12 @@ job-interview answer for why it belongs in the body of the machine. stem generation. - CultMesh carries typed state surfaces; remote nodes may observe and decode, but they do not become independent clock authorities. -- USB-attached Moves on Starfire and Nightwing publish - `mimir.move_tracking_observation.v1` as tracking streams. Muninn emits the - witness, Odin discovers/projects it, and Mimir owns buffering plus fusion. +- USB-attached Moves on Starfire and Nightwing publish source-local Muninn + evidence streams: glowing-orb marker candidates and controller/IMU/button + state. Odin discovers/projects those streams. Mimir owns buffering, + calibration, association, triangulation, IMU fusion, prediction, and + `mimir.move_controller_pose.v1` publication. Fensalir consumes that resolved + pose stream for interactive environments. - OBS may receive compatibility program surfaces and stems. It does not own calibration, composition, preview/control, or publication. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 9ffc0bf..6513587 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -45,11 +45,12 @@ Ownership: becoming independent clock authorities. - `Mimir.Runtime` owns app-level stream buffers and synchronization. - Native capture workers own device reads. -- Move tracking observations are stream samples, not program controls. USB - Moves plugged into Starfire produce local tracking streams; USB Moves plugged - into Nightwing are published by Muninn on Nightwing as remote tracking - streams. Odin owns discovery/schema projection for those streams and does not - own pose capture or Mimir fusion. +- Move tracking observations are stream samples, not program controls. Muninn + daemons on Starfire and Nightwing publish source-local optical marker + candidates plus controller/IMU/button state for USB-attached Moves. Odin owns + discovery/schema projection for those streams. Mimir owns calibration, + association, triangulation, IMU fusion, prediction, and the resolved + Fensalir-facing wand pose stream. - Mimir owns program composition, source subscription policy, preview/control state, stats, and publication intent. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, @@ -99,16 +100,18 @@ one process with declared accepted source ids, not one process per camera. ## Move Tracking Starfire and Nightwing both have PS Move controllers directly attached over -USB. Mimir represents those controller poses as `MimirStreamKind.Tracking` -buffers carrying typed `mimir.move_tracking_observation.v1` samples. Each -sample names the stream, device, producer, host, Odin discovery provider, -tracking space, calibration id, source timestamp, pose, velocity, buttons, -battery, latency, and confidence. +USB. Muninn represents what each local body can honestly observe: +`muninn.move_marker_candidate.v1` for glowing orb candidates from cameras and +`muninn.move_controller_state.v1` for controller/IMU/button/battery state from +USB-attached Moves. Mimir drinks those feeds into `MimirStreamKind.Tracking` +buffers and publishes resolved `mimir.move_controller_pose.v1` documents for +Fensalir interaction once calibration and fusion have earned a pose. Authority: -- Starfire-local Move tracking is a local Mimir/Muninn tracking stream. -- Nightwing Move tracking is emitted by Muninn on Nightwing over the typed +- Starfire-local Move evidence is published by Muninn/Mimir local producers as + typed observation streams. +- Nightwing Move evidence is emitted by Muninn on Nightwing over the typed observation path. - Structured PS Move light pulses are Muninn output commands. Mimir requests them by sending `muninn.move_light_command.v1` over CultNet/CultMesh to the @@ -117,9 +120,12 @@ Authority: hardware behavior before Muninn is available on that host. - Odin discovers and projects the stream/schema surface for operator and agent access. -- Mimir owns subscription, rolling retention, clock alignment, and later fusion - with cameras/audio/Leap/Fensalir. OBS and dashboard summaries do not own this - state. +- Mimir owns subscription, rolling retention, clock alignment, calibration, + controller association, triangulation, IMU fusion, prediction, and resolved + wand pose publication. OBS and dashboard summaries do not own this state. +- Fensalir consumes Mimir's resolved Move controller poses as interactive input + for environments; it does not synthesize controller truth from raw Muninn + evidence. Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing @@ -146,9 +152,9 @@ Odin's Muninn organ owns the Move optical extraction stage for sensor stream exposure. Its Rust crate lives in `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes those Muninn candidate streams into tracking buffers; it does not own raw -optical extraction. Stereo triangulation, camera calibration, controller -association, IMU fusion, prediction, and final 6DoF pose remain separate later -owners. +optical extraction. Mimir is now the explicit owner for stereo triangulation, +camera calibration, controller association, IMU fusion, prediction, and final +6DoF wand pose. ## Program Composition diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index eb631d4..637aea9 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -857,7 +857,7 @@ static async Task RunBioacousticContestantsAsync( foreach (var decoder in decoders) { - var index = BuildCepstralContestantWordIndex(sampleRate, decoder.Decoder, renderer); + var index = BuildCepstralContestantWordIndex(sampleRate, decoder.Decoder, renderer); foreach (var degradation in degradations) { var degraded = RoundTripThroughDegradedCepstrum(source, sampleRate, degradation, out var analysis); @@ -1701,6 +1701,17 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) hostId: "nightwing", latencyMilliseconds: 4.6, battery01: 0.67); + var fusedMovePose = MimirMoveControllerPoseDocument.FromObservation( + nightwingMove, + "move:right", + [ + "muninn:nightwing:ps3-eye-0:move-marker-candidates", + "muninn:nightwing:ps3-eye-1:move-marker-candidates", + "muninn:nightwing:move-usb:controller-state", + "muninn:starfire:move-usb:controller-state" + ], + ["optical-marker-candidate", "optical-marker-candidate", "imu-controller-state", "imu-controller-state"], + "mimir-move-stage-calibration-v1"); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions @@ -1743,10 +1754,14 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) nightwingMove, new CultRecordHandle(new CultRecordKey($"mimir-move-tracking:{nightwingMove.ObservationId}"))) .ConfigureAwait(false); + await cache.UpsertAsync( + fusedMovePose, + new CultRecordHandle(new CultRecordKey($"mimir-move-controller-pose:{fusedMovePose.PoseId}"))) + .ConfigureAwait(false); await cache.FlushAsync().ConfigureAwait(false); Console.WriteLine( - $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length} moveTracking=2 hosts={starfireMove.HostId},{nightwingMove.HostId}"); + $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length} moveTracking=2 fusedMovePose={fusedMovePose.PoseId} authority={fusedMovePose.FusionAuthorityId} consumer={fusedMovePose.ConsumerContract} hosts={starfireMove.HostId},{nightwingMove.HostId}"); return 0; } diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index 1b84f06..18af289 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -47,9 +47,15 @@ public static class MimirModuleLibrary new( "cultmesh-contracts", "Mimir.Runtime", - "Typed state documents for codebooks, decoder state, acoustic path state, actuator state, program state, and Move tracking observations.", - ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation"], + "Typed state documents for codebooks, decoder state, acoustic path state, actuator state, program state, Move evidence, and Mimir-fused Move controller poses.", + ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation", "mimir.move_controller_pose"], ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]), + new( + "move-controller-fusion", + "Mimir.Runtime + Fensalir", + "Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses; Fensalir consumes those poses for interaction.", + ["mimir.move_controller_pose.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke"]), new( "path-learning", "Mimir.Runtime", diff --git a/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs b/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs index fc18182..a2774cb 100644 --- a/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs +++ b/src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs @@ -98,3 +98,59 @@ public sealed record MimirTrackingObservation( double.IsFinite(battery01) ? Math.Clamp(battery01, 0.0, 1.0) : double.NaN, buttons ?? []); } + +[CultDocument("mimir.move_controller_pose", "mimir.move_controller_pose.v1")] +[MessagePackObject] +public sealed record MimirMoveControllerPoseDocument( + [property: Key(0)] + [property: CultName] + string PoseId, + [property: Key(1)] string WandId, + [property: Key(2)] string TrackingSpaceId, + [property: Key(3)] string CalibrationId, + [property: Key(4)] string FusionAuthorityId, + [property: Key(5)] long SourceTimestampNs, + [property: Key(6)] long EstimatedAtNs, + [property: Key(7)] ulong Sequence, + [property: Key(8)] MimirVector3Snapshot PositionMeters, + [property: Key(9)] MimirQuaternionSnapshot Orientation, + [property: Key(10)] MimirVector3Snapshot LinearVelocityMetersPerSecond, + [property: Key(11)] MimirVector3Snapshot AngularVelocityRadiansPerSecond, + [property: Key(12)] double Confidence, + [property: Key(13)] double LatencyMilliseconds, + [property: Key(14)] double Battery01, + [property: Key(15)] MimirTrackingButtonSnapshot[] Buttons, + [property: Key(16)] string[] EvidenceStreamIds, + [property: Key(17)] string[] EvidenceKinds, + [property: Key(18)] string ConsumerContract) +{ + public static MimirMoveControllerPoseDocument FromObservation( + MimirTrackingObservation observation, + string wandId, + string[] evidenceStreamIds, + string[] evidenceKinds, + string calibrationId, + string trackingSpaceId = "mimir-stage-space", + string fusionAuthorityId = "mimir.runtime.move-fusion", + string consumerContract = "fensalir.move-controller-input") => + new( + $"{wandId}:{observation.Sequence}", + wandId, + trackingSpaceId, + calibrationId, + fusionAuthorityId, + observation.SourceTimestampNs, + observation.ArrivalTimestampNs, + observation.Sequence, + observation.PositionMeters, + observation.Orientation, + observation.LinearVelocityMetersPerSecond, + observation.AngularVelocityRadiansPerSecond, + Math.Clamp(observation.Confidence, 0.0, 1.0), + Math.Max(0.0, observation.LatencyMilliseconds), + double.IsFinite(observation.Battery01) ? Math.Clamp(observation.Battery01, 0.0, 1.0) : double.NaN, + observation.Buttons, + evidenceStreamIds, + evidenceKinds, + consumerContract); +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 9f5c7d5..5734bb7 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -129,3 +129,4 @@ {"ts": "2026-06-10T20:33:54+01:00", "type": "implementation", "status": "ok", "note": "Added native/move_tracker as Rust-orchestrated PS Move optical candidate extraction: Rust owns dispatch/FFI/CPU mirror; HLSL owns 16px-tile luma reduction. Scope is marker candidates only, before calibration/triangulation/IMU fusion/prediction."} {"ts": "2026-06-10T20:49:06+01:00", "type": "decision", "status": "ok", "note": "Corrected Move optical candidate ownership: the Rust/HLSL tracker moved out of Mimir and into Odin's Muninn organ at E:\\Projects\\Odin\\crates\\muninn-move-tracker. Mimir consumes muninn.move_marker_candidate.v1 streams and no longer owns raw optical extraction."} {"ts": "2026-06-11T07:57:05+01:00", "type": "decision", "status": "ok", "note": "PS Move structured light ownership moved to Odin/Muninn: Mimir requests muninn.move_light_command.v1 over CultNet/CultMesh; Muninn serve consumes host-addressed pending commands, writes local HID report 0x06, and updates command state. Nightwing direct HID launcher path is opt-in smoke/bootstrap only."} +{"ts": "2026-06-11T16:59:31+01:00", "type": "decision", "status": "ok", "note": "Move sensor fusion authority moved to Mimir: Muninn publishes source-local marker candidates and controller/IMU/button state, Odin discovers/projects schemas, Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, latency, and mimir.move_controller_pose.v1 for Fensalir interaction."} diff --git a/state/map.yaml b/state/map.yaml index 639b97c..bd2c12b 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -33,9 +33,10 @@ constraints: memory. - Networked feeds are producers into the live runtime, not independent clock authorities. - - USB-attached Moves on Starfire and Nightwing are tracking witnesses. Odin - may discover/project their stream schemas, but it does not own pose capture - or Mimir fusion. + - USB-attached Moves on Starfire and Nightwing are tracking witnesses. Muninn + publishes source-local marker candidates plus controller/IMU/button state. + Odin may discover/project their stream schemas, but it does not own pose + capture or Mimir fusion. - CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but Mimir owns program selection, composition, preview/control, and publication. @@ -84,11 +85,13 @@ accepted_design: device timestamp, and optional native/GPU handle. - `MimirAudioBlockDescriptor` carries sample rate, channels, sample format, frame count, timestamp, and optional native handle. - - `MimirTrackingObservation` carries typed PS Move pose/button/battery/ - confidence samples as `mimir.move_tracking_observation.v1`; Starfire Moves - are local tracking streams, Nightwing Moves are emitted by Muninn on - Nightwing, Odin owns discovery/schema projection, and Mimir owns buffering - and fusion. + - `MimirTrackingObservation` carries typed PS Move evidence samples as + `mimir.move_tracking_observation.v1`; Starfire Moves are local tracking + streams, Nightwing Moves are emitted by Muninn on Nightwing, Odin owns + discovery/schema projection, and Mimir owns buffering and fusion. + - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for + Fensalir-facing interaction. Mimir owns calibration, association, + triangulation, IMU fusion, prediction, confidence, and latency accounting. - `scripts/start-nightwing-move-tracking.ps1` restores the live Nightwing Eye/Move witness path: Starfire owns the `/eve/periwinkle` receiver and recorder, Nightwing owns PS Move HID report `0x06` LED writes and PS3 Eye @@ -98,9 +101,9 @@ accepted_design: - Odin's Muninn organ owns Move optical candidate extraction in `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes those candidate - streams; it does not own raw optical extraction. Calibration, multi-camera - association, triangulation, IMU fusion, prediction, and final 6DoF pose are - still separate owners. + streams; it does not own raw optical extraction. Mimir explicitly owns + calibration, multi-camera association, triangulation, IMU fusion, + prediction, and final 6DoF wand pose. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the From c9fca623468b9bebd4cdbfa00de5cbcfe5ce8781 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 17:02:26 +0100 Subject: [PATCH 019/135] Persist Mimir face state --- .voidbot/state/mimir.cc | Bin 65126 -> 65603 bytes .voidbot/voice/identity.json | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 1c6c8aa07e5e6042e191b194e189ae29555d1fe6..bfffc26d51a261372eda854c196c8297a06f14ec 100644 GIT binary patch delta 3193 zcmb_eU2I&%6;^0kAk?nXI{eg%Go`cv;kCR!j&}`}sal{Cpb#YzQjkEib7y@A-+O00 zGuN9ns2ikKkfso-Q7g5C7Yc>;0U`VnT2WQukru=Qph$p7gz^AFf(Lk_Qol2Iue~O! z1Zw?Y%X9BNbIy0Z?>p!G=U;pI==SG78@ssLSeUCV)cn~Klm5!_hsTEhH)fjqYKMQb zZP)tg@yhJzX)~PeTzq4C$DI=<12}bLov&`QIL29z1l1N|j+fo;k~0ELkRxIA$bISgKbgbDFg!<1Env zM|AE~Dk~i&lZbVV#zPl#$NF?&na(7=6!CTo4TFXStRmN0i9iA}VL1cs1v`*QL zCYDYcC1#!$30LWvza2sjxs{zfVca;?;leQ+>m1kNPmfuh8zEUo%E&+P@NWN)UtiCZO8~u zl7u;#b^%CNnvOOpcM4FPJ9^S~r9k-^OSqHf+|ef(PNayZSvx1xx)QG%X%2k$?U4?rZkV&Ck}OcB3-iYIiD)S>CBMMWa&@v(?7@Y;B%5Mccpr>ZG4~ z^6H)YD%I+tYHgv>TBtQ=TC+9Jj_(@(!cq0LANcrq#ZNtSY;f``5u_+YXMqR_q1NRt zE+`_8bhSlyIJ3YA{P-_K=@<~BZ1Vy18J`=SFU2eZX4c`el(h}dMEqq~lkrq;g!{j; zW?Ro+_QvGi4N{a`p2l@V48BZUXiHEC-S_wYYL|cWSbdxIH$S@fUQ$YoU)BvdaAB{f zQ8>63B7qPF&fRNG8eU~ zwMM5U=cD=;h|ySRG#Bc%nPzi?7^6o8F?RYVPfQSJMjyd#F59%9rvGI53jg_upKm$a zHhzUa`Pg?a(#`c#kG;KTuz!?WRtp7-=_XthgokI4A=w@`X(6yoyA3XEk!GR7k_`tA zme}ho>63+lmY`zLz0BYqz2GMU5lh|$!fhQmI&=V%VMuJRWszFSAhi(>Q9pwT37!?N zfWsAtQyR3K(#Lghy*SX{ncT(XV{MZwxY4kKxLv|wV26X!4><~p5renEZZf>!g^SMw zp~JW2!2+8&hrJo5EJ-IRFd>LUZVXvOVPG+88dhi|z>8QUg*JtWkw-cB5hR+96E8t2 z;7qCdmAfVe{hx4SfEfY5*v%q~(Ec59Z1;%c)knMSEMAU~IzjP;1WRGOru ziSkQOauz|;0x^2p04vM;2vU%E&(PW4Upss2a})PJv&s`{*ei>FKXu^4UrwF+A8Fm1 zzkt?sRDErafA^(ttbhE%zqi+aSCA5rSWJLBq~ikEfeRUs(LLZ#rK@hDP!12}_^F$I zdTrZi9^t86RT1S}vDAnNZRaV~Hym)efAFTsK_Di^l!iBPf?tA!5y%q&!-%ntr(H%f z!q`}K2|hBih!Hy@qdI_GCz4z+U<-3il)$vxAdR>sx76kYGco5&;jLo0S$S)eV+cVK z6phH7@@0(O%=TfbMrRmTNE~Kq$vN>(F|lbYDEJb~K5Cal86UlP!tY-Fw!eM<*q3yi z>EReE{i9WHm1Ai1sA$V|{+++xJ-FwPL>cHp7lV;zBGm;R6(Sh*5y+iqmih!*1~J+^ z=*ofY#Pcni2(}+#`ek$cAyN~~YwQe=BM}|m;q*Z0gmn^S9ApGgSJ9T-Vm?DmVcrMD z0VwpTU^UXjvJjq@l39gDjY{-NuGZ+QvZRd;ajY&nkPLlBrnEWqLq}6;<(c?q5$p24 zjWjo%C%76mGHeYB^@mnTZZi6W6K#7pOShcGD!S~tvyDwcgKlsBho+4HK1-q@nM&V@iG#n>~-+Vy+vv&4R#U64d4-lO@ z`g_ZiL01YEVDIhl1V9Q+9!FA)(LM6*s*iA{=4wI4<$z905xpZ{+l7Veq^;@;W{{f33k0$^C delta 2458 zcmbVOO>9&}91m?nA*H1aEk)2Whzg0j>wb5)7)vimqFA97A|#sG_h$D^`gZ1Z-n=f0 zLMzc&6aqRYLgIl)(2Js#XykxF6Ay?NKN1K=0VOpM6JtU=XySk7?QR#n*+a8^+nN3S ze_#Loe5UQ@U#(~EbauwGL+R{LDj&v27Phj)D5+wJh2RpCvroXWGtJ? zsGmAlE*#BdxZUi#~p$g*(=dDTBaiEJJ=&OD)T9Ve1`|F zjS2)|WN~H*A_?*YhX<7T)Fw#6VL`AfX_=r3WhEpqJQMH+W^jpsw>d8(;Sk9MwGb}w zKq72e#P<;wXs7QH3l1|scFJ-wEnhx6f}{w1S>ZxD)essMk`}QgQN;kVgR+M>BgmzI z(t}Uj;3J=w7C9(w=Z}1MUWo|I0_^A5=#<6nAvLGkrf*RW|jkUn2TTr z8y8%HEC*8-duK8rzNDO8KB#`2?dv3aJt8zp37FN?5&DGCv|-O9k^(1^BbWnEAPP7n zbzOrjf(kIJAn=aV)M&D}=93GXh^UKMDZnM7-aoV{vu_KY0y=OFf3zfcz-&P6ato%S zY)S(v!9|?u5s(8nsm?6}8#E6^L3{_535H>bfoa%79;eKFG?wbKmm=$}VmxQ(Q&BrT zXhqX@E*{MfTDfR3Gg!#QGgg|AqFR1%xH}OaPsE23$)V(6ES^a0R`EaAw!#ZFb8v?m zcy(ghjBi|{)eqeQv*Q}vx<^7!U7tDbBT0my;DKrcF%AxZU)m>LP;kK(K2S@T1(g2F zAOEq($D=UsBMrS8&Zv16l;VEq%OW9m1X)1fgc_aAE)ET#o_+gab!N6}W5F1JfD0L- zvPp8b%c0;5vX=tmtfb|he(~uL=l@V#``<87A7vp7H8+!zz>uW^FamHf&{Q=N+u&$; z9s2;Q%ZGYeoUQ8K*B%0^=cmiT0aTTKL z>S#e0bwo^I1kBY9K_#-FNU#mF)m*M)%f;O;;(}pqd$6Jm1glrc17rGP_&8t-Zf3wP z!3?L3ai|0SuckS`_dS3uhcY>G?s;@^oS`tas^T$g4vNe%Rf|P63 zV{i4({XBQOZT0?bL}Cky+t`$0A-Jh>&eXu>*l<{A8VsNVBAd!t7g5+W7M!YTu8Le# zu(0bA;4yV}b3Sp`TJWHwFiZ`-75KW(>P@)7y|1bxYqIsN5f-W0Psrv#l5P-fA?+ZW z=*&QKa~%v&TYW#0QTC@t+vm66dS&2~9Zlxm)0%2{dTC2aTUzt5_|5a{)eq6wufV zqR;jI0fk|=OmV8iQ4DLkwy8|$ngE9tqXE-ylhLqhzy{LM7I*w7Vb(fv7Nh{|YEjRH zt^t9K&QN_|{7^>B{0g`pl5I*V{MNxMOmFC*;7jl!K}3f^C%5VAgnI7Qs`i_`#@t5J x;+5wY<<#W4mFnu1Wi7dUUCj^AuOD1=eD0S!(WTH;%hk2czIrcZV)0CF_dl#5n`r<5 diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index 2d86c39..2af16f5 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-06-11T06:46:39.463Z", + "createdOrRefreshedAt": "2026-06-11T14:47:29.203Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 0d1f97436a951b1704deb72a45c8f524bbf1b9e3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 18:50:56 +0100 Subject: [PATCH 020/135] Add native Move evidence reservoir ABI --- docs/implementation-plan.md | 4 + docs/perfect-machine-module-library.md | 10 +- .../reservoir/include/localcast_reservoir.h | 55 +++- native/reservoir/src/lib.rs | 265 ++++++++++++++++++ notes/current-system-map.md | 13 +- .../MimirReservoirConfiguration.cs | 1 + state/evidence.jsonl | 1 + state/map.yaml | 3 + 8 files changed, 341 insertions(+), 11 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 6010b14..6e242a3 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -38,6 +38,10 @@ a named invariant that the native runtime cannot protect yet. Starfire and Nightwing both have USB-attached Moves. Muninn daemons publish source-local glowing-orb marker candidates and controller/IMU/button state; Odin owns discovery and schema projection, not pose fusion. +- `native/reservoir` exposes a `move_evidence` typed view and + `LocalcastMoveEvidenceBufferDescriptor` so Muninn Move witnesses can be + admitted as compute-upload buffers before Mimir's fusion pass and Fensalir's + GPU consumers touch them. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting for those poses before Fensalir consumes diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index 1e30e44..a057446 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + Fensalir | Mimir fuses Muninn marker candidates plus controller/IMU/button feeds into `mimir.move_controller_pose.v1`; Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers, fuses them into `mimir.move_controller_pose.v1`, and Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -45,10 +45,10 @@ job-interview answer for why it belongs in the body of the machine. but they do not become independent clock authorities. - USB-attached Moves on Starfire and Nightwing publish source-local Muninn evidence streams: glowing-orb marker candidates and controller/IMU/button - state. Odin discovers/projects those streams. Mimir owns buffering, - calibration, association, triangulation, IMU fusion, prediction, and - `mimir.move_controller_pose.v1` publication. Fensalir consumes that resolved - pose stream for interactive environments. + state. Odin discovers/projects those streams. Mimir owns native evidence + buffer admission/layout, calibration, association, triangulation, IMU fusion, + prediction, and `mimir.move_controller_pose.v1` publication. Fensalir + consumes that resolved pose stream for interactive environments. - OBS may receive compatibility program surfaces and stems. It does not own calibration, composition, preview/control, or publication. diff --git a/native/reservoir/include/localcast_reservoir.h b/native/reservoir/include/localcast_reservoir.h index e1f5122..8ae6122 100644 --- a/native/reservoir/include/localcast_reservoir.h +++ b/native/reservoir/include/localcast_reservoir.h @@ -17,7 +17,8 @@ enum LocalcastSampleKind { LOCALCAST_SAMPLE_AUDIO_BLOCK = 5, LOCALCAST_SAMPLE_PHASE_CLAIM = 6, LOCALCAST_SAMPLE_EVENT_CLAIM = 7, - LOCALCAST_SAMPLE_RENDER_PACKET = 8 + LOCALCAST_SAMPLE_RENDER_PACKET = 8, + LOCALCAST_SAMPLE_MOVE_EVIDENCE = 9 }; typedef struct LocalcastReservoir LocalcastReservoir; @@ -79,6 +80,47 @@ typedef struct LocalcastRenderPoint { float confidence; } LocalcastRenderPoint; +enum LocalcastMoveEvidenceKind { + LOCALCAST_MOVE_EVIDENCE_OPTICAL_MARKER = 1, + LOCALCAST_MOVE_EVIDENCE_CONTROLLER_STATE = 2 +}; + +typedef struct LocalcastMoveEvidenceSample { + uint64_t witness_id_hash; + uint64_t controller_id_hash; + uint64_t source_timestamp_ns; + uint64_t arrival_ns; + uint64_t sequence; + uint32_t evidence_kind; + uint32_t flags; + float image_x; + float image_y; + float radius_px; + float confidence; + float accel_x; + float accel_y; + float accel_z; + float gyro_x; + float gyro_y; + float gyro_z; + float trigger; + uint32_t buttons_mask; + uint32_t reserved; + float battery01; + uint32_t reserved1; + uint32_t reserved2; +} LocalcastMoveEvidenceSample; + +typedef struct LocalcastMoveEvidenceBufferDescriptor { + uint64_t sample_buffer_handle; + uint32_t sample_count; + uint32_t sample_stride_bytes; + uint64_t source_time_min_ns; + uint64_t source_time_max_ns; + uint64_t calibration_hash; + uint64_t tracking_space_hash; +} LocalcastMoveEvidenceBufferDescriptor; + typedef struct LocalcastRuntimeStatus { uint64_t edge_ns; uint64_t window_start_ns; @@ -92,6 +134,7 @@ typedef struct LocalcastRuntimeStatus { size_t phase_claim_count; size_t event_claim_count; size_t render_packet_count; + size_t move_evidence_count; } LocalcastRuntimeStatus; uint64_t localcast_hash_source_id(const uint8_t *bytes, size_t byte_len); @@ -144,6 +187,7 @@ bool localcast_runtime_push_audio_block(LocalcastRuntime *runtime, LocalcastSamp bool localcast_runtime_push_phase_claim(LocalcastRuntime *runtime, LocalcastSampleHandle sample); bool localcast_runtime_push_event_claim(LocalcastRuntime *runtime, LocalcastSampleHandle sample); bool localcast_runtime_push_render_packet(LocalcastRuntime *runtime, LocalcastSampleHandle sample); +bool localcast_runtime_push_move_evidence(LocalcastRuntime *runtime, LocalcastSampleHandle sample); bool localcast_runtime_status(const LocalcastRuntime *runtime, LocalcastRuntimeStatus *out_status); size_t localcast_runtime_len(const LocalcastRuntime *runtime); size_t localcast_runtime_view_len(const LocalcastRuntime *runtime, uint32_t sample_kind); @@ -210,6 +254,15 @@ bool localcast_producer_push_render_packet( LocalcastSampleHandle *out_sample ); +bool localcast_producer_push_move_evidence_buffer( + LocalcastProducer *producer, + LocalcastRuntime *runtime, + uint64_t timestamp_ns, + uint64_t arrival_ns, + const LocalcastMoveEvidenceBufferDescriptor *descriptor, + LocalcastSampleHandle *out_sample +); + #ifdef __cplusplus } #endif diff --git a/native/reservoir/src/lib.rs b/native/reservoir/src/lib.rs index afd5cdb..71e42cc 100644 --- a/native/reservoir/src/lib.rs +++ b/native/reservoir/src/lib.rs @@ -17,6 +17,7 @@ pub enum SampleKind { PhaseClaim, EventClaim, RenderPacket, + MoveEvidence, } impl SampleKind { @@ -31,6 +32,7 @@ impl SampleKind { 6 => Some(Self::PhaseClaim), 7 => Some(Self::EventClaim), 8 => Some(Self::RenderPacket), + 9 => Some(Self::MoveEvidence), _ => None, } } @@ -46,6 +48,7 @@ impl SampleKind { Self::PhaseClaim => 6, Self::EventClaim => 7, Self::RenderPacket => 8, + Self::MoveEvidence => 9, } } } @@ -229,6 +232,53 @@ pub struct LocalcastRenderPoint { pub confidence: f32, } +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LocalcastMoveEvidenceKind { + OpticalMarker = 1, + ControllerState = 2, +} + +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct LocalcastMoveEvidenceSample { + pub witness_id_hash: u64, + pub controller_id_hash: u64, + pub source_timestamp_ns: u64, + pub arrival_ns: u64, + pub sequence: u64, + pub evidence_kind: u32, + pub flags: u32, + pub image_x: f32, + pub image_y: f32, + pub radius_px: f32, + pub confidence: f32, + pub accel_x: f32, + pub accel_y: f32, + pub accel_z: f32, + pub gyro_x: f32, + pub gyro_y: f32, + pub gyro_z: f32, + pub trigger: f32, + pub buttons_mask: u32, + pub reserved: u32, + pub battery01: f32, + pub reserved1: u32, + pub reserved2: u32, +} + +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct LocalcastMoveEvidenceBufferDescriptor { + pub sample_buffer_handle: u64, + pub sample_count: u32, + pub sample_stride_bytes: u32, + pub source_time_min_ns: u64, + pub source_time_max_ns: u64, + pub calibration_hash: u64, + pub tracking_space_hash: u64, +} + pub struct LocalcastReservoir { inner: RollingReservoir, } @@ -258,6 +308,7 @@ pub struct LocalcastRuntimeStatus { pub phase_claim_count: usize, pub event_claim_count: usize, pub render_packet_count: usize, + pub move_evidence_count: usize, } impl Default for LocalcastRuntimeStatus { @@ -275,6 +326,7 @@ impl Default for LocalcastRuntimeStatus { phase_claim_count: 0, event_claim_count: 0, render_packet_count: 0, + move_evidence_count: 0, } } } @@ -308,6 +360,7 @@ impl LocalcastRuntime { phase_claim_count: self.view_len(SampleKind::PhaseClaim), event_claim_count: self.view_len(SampleKind::EventClaim), render_packet_count: self.view_len(SampleKind::RenderPacket), + move_evidence_count: self.view_len(SampleKind::MoveEvidence), } } } @@ -623,6 +676,14 @@ pub extern "C" fn localcast_runtime_push_render_packet( localcast_runtime_push_typed(ptr, SampleKind::RenderPacket, sample) } +#[unsafe(no_mangle)] +pub extern "C" fn localcast_runtime_push_move_evidence( + ptr: *mut LocalcastRuntime, + sample: LocalcastSampleHandle, +) -> bool { + localcast_runtime_push_typed(ptr, SampleKind::MoveEvidence, sample) +} + #[unsafe(no_mangle)] pub extern "C" fn localcast_runtime_status( ptr: *const LocalcastRuntime, @@ -860,6 +921,46 @@ pub extern "C" fn localcast_producer_push_render_packet( ) } +#[unsafe(no_mangle)] +pub extern "C" fn localcast_producer_push_move_evidence_buffer( + producer_ptr: *mut LocalcastProducer, + runtime_ptr: *mut LocalcastRuntime, + timestamp_ns: u64, + arrival_ns: u64, + descriptor: *const LocalcastMoveEvidenceBufferDescriptor, + out_sample: *mut LocalcastSampleHandle, +) -> bool { + let Some(producer) = (unsafe { producer_ptr.as_mut() }) else { + return false; + }; + if producer.kind != SampleKind::MoveEvidence { + return false; + } + let Some(descriptor_ref) = (unsafe { descriptor.as_ref() }) else { + return false; + }; + if descriptor_ref.sample_buffer_handle == 0 { + return false; + } + if descriptor_ref.sample_count == 0 { + return false; + } + if descriptor_ref.sample_stride_bytes as usize + != std::mem::size_of::() + { + return false; + } + let payload_handle = descriptor as u64; + localcast_producer_push( + producer_ptr, + runtime_ptr, + timestamp_ns, + arrival_ns, + payload_handle, + out_sample, + ) +} + fn localcast_runtime_push_typed( ptr: *mut LocalcastRuntime, kind: SampleKind, @@ -943,6 +1044,51 @@ mod tests { } } + fn move_evidence_descriptor( + sample_buffer_handle: u64, + ) -> LocalcastMoveEvidenceBufferDescriptor { + LocalcastMoveEvidenceBufferDescriptor { + sample_buffer_handle, + sample_count: 2, + sample_stride_bytes: std::mem::size_of::() as u32, + source_time_min_ns: 1_000, + source_time_max_ns: 1_120, + calibration_hash: 0xCA11_BA7E, + tracking_space_hash: 0x57A6E, + } + } + + fn move_marker_sample( + witness_id_hash: u64, + controller_id_hash: u64, + ) -> LocalcastMoveEvidenceSample { + LocalcastMoveEvidenceSample { + witness_id_hash, + controller_id_hash, + source_timestamp_ns: 1_000, + arrival_ns: 1_010, + sequence: 7, + evidence_kind: LocalcastMoveEvidenceKind::OpticalMarker as u32, + flags: 0, + image_x: 123.5, + image_y: 87.25, + radius_px: 14.0, + confidence: 0.92, + accel_x: 0.0, + accel_y: 0.0, + accel_z: 0.0, + gyro_x: 0.0, + gyro_y: 0.0, + gyro_z: 0.0, + trigger: 0.0, + buttons_mask: 0, + reserved: 0, + battery01: f32::NAN, + reserved1: 0, + reserved2: 0, + } + } + fn render_point(stable_key_hash: u64) -> LocalcastRenderPoint { LocalcastRenderPoint { stable_key_hash, @@ -1585,6 +1731,125 @@ mod tests { localcast_runtime_destroy(runtime); } + #[test] + fn move_evidence_layout_is_fixed_for_compute_upload() { + let sample = move_marker_sample(11, 22); + + assert_eq!(std::mem::size_of::(), 112); + assert_eq!( + std::mem::size_of::(), + 48 + ); + assert_eq!(11, sample.witness_id_hash); + assert_eq!(22, sample.controller_id_hash); + assert_eq!( + LocalcastMoveEvidenceKind::OpticalMarker as u32, + sample.evidence_kind + ); + assert_eq!(123.5, sample.image_x); + assert_eq!(14.0, sample.radius_px); + } + + #[test] + fn move_evidence_descriptor_enters_native_runtime_window() { + let runtime = localcast_runtime_create(DEFAULT_RESERVOIR_NS); + let producer = localcast_producer_create(SampleKind::MoveEvidence.id(), 101, 3); + let descriptor = move_evidence_descriptor(9_000); + let mut sample = handle(0, 0, 0, 0, 0); + + assert!(localcast_producer_push_move_evidence_buffer( + producer, + runtime, + 1_200, + 1_220, + &descriptor, + &mut sample, + )); + + let mut status = LocalcastRuntimeStatus::default(); + assert!(localcast_runtime_status(runtime, &mut status)); + assert_eq!(1, status.total_sample_count); + assert_eq!(1, status.move_evidence_count); + assert_eq!(0, status.camera_feature_count); + assert_eq!( + (&descriptor as *const LocalcastMoveEvidenceBufferDescriptor) as u64, + sample.payload_handle + ); + assert_eq!( + 1, + localcast_runtime_view_len(runtime, SampleKind::MoveEvidence.id()) + ); + + localcast_producer_destroy(producer); + localcast_runtime_destroy(runtime); + } + + #[test] + fn move_evidence_descriptor_push_rejects_wrong_kind_and_bad_buffers() { + let runtime = localcast_runtime_create(DEFAULT_RESERVOIR_NS); + let render_producer = localcast_producer_create(SampleKind::RenderPacket.id(), 88, 12); + let move_producer = localcast_producer_create(SampleKind::MoveEvidence.id(), 89, 0); + let good = move_evidence_descriptor(9_000); + let zero_handle = move_evidence_descriptor(0); + let zero_count = LocalcastMoveEvidenceBufferDescriptor { + sample_count: 0, + ..good + }; + let bad_stride = LocalcastMoveEvidenceBufferDescriptor { + sample_stride_bytes: 4, + ..good + }; + + assert!(!localcast_producer_push_move_evidence_buffer( + render_producer, + runtime, + 2_000, + 2_010, + &good, + std::ptr::null_mut(), + )); + assert!(!localcast_producer_push_move_evidence_buffer( + move_producer, + runtime, + 2_000, + 2_010, + std::ptr::null(), + std::ptr::null_mut(), + )); + assert!(!localcast_producer_push_move_evidence_buffer( + move_producer, + runtime, + 2_000, + 2_010, + &zero_handle, + std::ptr::null_mut(), + )); + assert!(!localcast_producer_push_move_evidence_buffer( + move_producer, + runtime, + 2_000, + 2_010, + &zero_count, + std::ptr::null_mut(), + )); + assert!(!localcast_producer_push_move_evidence_buffer( + move_producer, + runtime, + 2_000, + 2_010, + &bad_stride, + std::ptr::null_mut(), + )); + assert_eq!( + 0, + localcast_runtime_view_len(runtime, SampleKind::MoveEvidence.id()) + ); + + localcast_producer_destroy(render_producer); + localcast_producer_destroy(move_producer); + localcast_runtime_destroy(runtime); + } + #[test] fn runtime_spine_rejects_nulls() { let sample = handle(1, 2, 3, 4, 5); diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 6513587..cc39295 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -104,8 +104,10 @@ USB. Muninn represents what each local body can honestly observe: `muninn.move_marker_candidate.v1` for glowing orb candidates from cameras and `muninn.move_controller_state.v1` for controller/IMU/button/battery state from USB-attached Moves. Mimir drinks those feeds into `MimirStreamKind.Tracking` -buffers and publishes resolved `mimir.move_controller_pose.v1` documents for -Fensalir interaction once calibration and fusion have earned a pose. +buffers, admits them into the native `move_evidence` reservoir view as +compute-upload buffers, and publishes resolved `mimir.move_controller_pose.v1` +documents for Fensalir interaction once calibration and fusion have earned a +pose. Authority: @@ -120,9 +122,10 @@ Authority: hardware behavior before Muninn is available on that host. - Odin discovers and projects the stream/schema surface for operator and agent access. -- Mimir owns subscription, rolling retention, clock alignment, calibration, - controller association, triangulation, IMU fusion, prediction, and resolved - wand pose publication. OBS and dashboard summaries do not own this state. +- Mimir owns subscription, rolling retention, native Move evidence buffer + layout, clock alignment, calibration, controller association, triangulation, + IMU fusion, prediction, and resolved wand pose publication. OBS and + dashboard summaries do not own this state. - Fensalir consumes Mimir's resolved Move controller poses as interactive input for environments; it does not synthesize controller truth from raw Muninn evidence. diff --git a/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs index 7a6a8ee..bc1fb7f 100644 --- a/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs @@ -32,6 +32,7 @@ public static class MimirReservoirConfigurations new("audio_block", "PCM/native audio block", "Mimir.Runtime/native audio", RetainsHistory: false, "Timing and field evidence."), new("phase_claim", "audio phase/group-delay evidence", "Mimir.Runtime", RetainsHistory: false, "Calibration/normalization input."), new("event_claim", "timeline anchor", "Mimir.Runtime", RetainsHistory: false, "Bioacoustic/passive/calibration events."), + new("move_evidence", "PS Move marker/controller evidence buffer", "Mimir native reservoir", RetainsHistory: false, "Compute-upload buffer of Muninn optical and controller witnesses before Mimir pose fusion."), new("surface_claim", "spatial surface observation", "Fensalir", RetainsHistory: false, "Fusion candidate."), new("render_packet", "Mimir program packet", "Mimir/Fensalir/Faust", RetainsHistory: false, "Output-ready program view.") ]; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 5734bb7..a7bf42a 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -130,3 +130,4 @@ {"ts": "2026-06-10T20:49:06+01:00", "type": "decision", "status": "ok", "note": "Corrected Move optical candidate ownership: the Rust/HLSL tracker moved out of Mimir and into Odin's Muninn organ at E:\\Projects\\Odin\\crates\\muninn-move-tracker. Mimir consumes muninn.move_marker_candidate.v1 streams and no longer owns raw optical extraction."} {"ts": "2026-06-11T07:57:05+01:00", "type": "decision", "status": "ok", "note": "PS Move structured light ownership moved to Odin/Muninn: Mimir requests muninn.move_light_command.v1 over CultNet/CultMesh; Muninn serve consumes host-addressed pending commands, writes local HID report 0x06, and updates command state. Nightwing direct HID launcher path is opt-in smoke/bootstrap only."} {"ts": "2026-06-11T16:59:31+01:00", "type": "decision", "status": "ok", "note": "Move sensor fusion authority moved to Mimir: Muninn publishes source-local marker candidates and controller/IMU/button state, Odin discovers/projects schemas, Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, latency, and mimir.move_controller_pose.v1 for Fensalir interaction."} +{"ts": "2026-06-11T18:47:43+01:00", "type": "implementation", "status": "ok", "note": "Added native reservoir Move evidence admission: LocalcastMoveEvidenceSample and LocalcastMoveEvidenceBufferDescriptor define a fixed compute-upload ABI, SampleKind move_evidence enters the shared rolling window, malformed buffers are rejected before live admission, and Mimir docs/maps now route Muninn marker/controller witnesses through native buffers before Mimir pose fusion and Fensalir consumption."} diff --git a/state/map.yaml b/state/map.yaml index bd2c12b..ef45a67 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -89,6 +89,9 @@ accepted_design: `mimir.move_tracking_observation.v1`; Starfire Moves are local tracking streams, Nightwing Moves are emitted by Muninn on Nightwing, Odin owns discovery/schema projection, and Mimir owns buffering and fusion. + - `native/reservoir` now exposes a `move_evidence` typed view and + `LocalcastMoveEvidenceBufferDescriptor` for compute-upload buffers carrying + Muninn optical marker and controller-state witnesses before Mimir fusion. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for Fensalir-facing interaction. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting. From 5b4c391607783f5f2b4df4f482ff74574902f8a3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 18:51:52 +0100 Subject: [PATCH 021/135] Persist Mimir face state --- .voidbot/state/mimir.cc | Bin 65603 -> 65603 bytes .voidbot/status/void-memory-maintenance.json | 22 +++++-------------- 2 files changed, 6 insertions(+), 16 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index bfffc26d51a261372eda854c196c8297a06f14ec..c5866eb783905c2dff7950a4299b2c9eedde9352 100644 GIT binary patch delta 187 zcmX@yz;d{OWy5z~9&;;0Q!672Jxfc2sELfClO^~%p<8(3?PEW26{$j zW*~)|1?{itaDjE380wjrP5zi&!)jz;WnjFyD2JDk8*Z+J<>Uvv;!u_5Kzk-L@`-Og ql Date: Thu, 11 Jun 2026 19:31:21 +0100 Subject: [PATCH 022/135] Wire Move evidence into native reservoir --- docs/implementation-plan.md | 3 + docs/perfect-machine-module-library.md | 3 +- src/Mimir.BufferSmoke/Program.cs | 108 ++++++ .../MimirNativeReservoirRuntime.cs | 329 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 2 + 6 files changed, 445 insertions(+), 1 deletion(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 6e242a3..e8f5cba 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -42,6 +42,9 @@ a named invariant that the native runtime cannot protect yet. `LocalcastMoveEvidenceBufferDescriptor` so Muninn Move witnesses can be admitted as compute-upload buffers before Mimir's fusion pass and Fensalir's GPU consumers touch them. +- `MimirNativeReservoirRuntime` is the managed owner for that native runtime + boundary. It pins Move evidence sample/descriptor batches so native reservoir + handles remain valid while Mimir prepares fusion input. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting for those poses before Fensalir consumes diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index a057446..aa14bbb 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers, fuses them into `mimir.move_controller_pose.v1`, and Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `native/reservoir/src/lib.rs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers, fuses them into `mimir.move_controller_pose.v1`, and Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -62,6 +62,7 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-lowering-benchmark dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test ``` diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 637aea9..c571a6d 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -96,6 +96,12 @@ if (args.Any(arg => string.Equals(arg, "--move-tracking-contract-smoke", StringC return RunMoveTrackingContractSmoke(); } +if (args.Any(arg => string.Equals(arg, "--move-native-reservoir-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveNativeReservoirSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1439,6 +1445,87 @@ static int RunMoveTrackingContractSmoke() : 1; } +static int RunMoveNativeReservoirSmoke(string nativeReservoirPath) +{ + if (!File.Exists(nativeReservoirPath)) + { + Console.Error.WriteLine($"move-native-reservoir-smoke missing-native path={nativeReservoirPath}"); + return 1; + } + + using var runtime = new MimirNativeReservoirRuntime(nativeReservoirPath); + var samples = new[] + { + new MimirNativeMoveEvidenceSample( + WitnessIdHash: 0x57A2_F12E_0001, + ControllerIdHash: 0xC011_7A01, + SourceTimestampNs: 1_000_000_000, + ArrivalNs: 1_000_000_700, + Sequence: 41, + EvidenceKind: (uint)MimirNativeMoveEvidenceKind.OpticalMarker, + Flags: 0, + ImageX: 318.25f, + ImageY: 119.5f, + RadiusPx: 13.75f, + Confidence: 0.91f, + AccelX: 0.0f, + AccelY: 0.0f, + AccelZ: 0.0f, + GyroX: 0.0f, + GyroY: 0.0f, + GyroZ: 0.0f, + Trigger: 0.0f, + ButtonsMask: 0, + Reserved: 0, + Battery01: float.NaN, + Reserved1: 0, + Reserved2: 0), + new MimirNativeMoveEvidenceSample( + WitnessIdHash: 0x516E_7716_0001, + ControllerIdHash: 0xC011_7A01, + SourceTimestampNs: 1_000_000_500, + ArrivalNs: 1_000_001_000, + Sequence: 42, + EvidenceKind: (uint)MimirNativeMoveEvidenceKind.ControllerState, + Flags: 0, + ImageX: float.NaN, + ImageY: float.NaN, + RadiusPx: float.NaN, + Confidence: 0.84f, + AccelX: -0.02f, + AccelY: 0.11f, + AccelZ: 0.98f, + GyroX: 0.01f, + GyroY: -0.02f, + GyroZ: 0.03f, + Trigger: 0.42f, + ButtonsMask: 0b101, + Reserved: 0, + Battery01: 0.77f, + Reserved1: 0, + Reserved2: 0) + }; + + var handle = runtime.AdmitMoveEvidence( + "mimir:move-native-reservoir-smoke", + samples, + "synthetic-move-calibration", + "mimir-stage-space"); + var status = runtime.Status; + + Console.WriteLine( + $"move-native-reservoir-smoke native={nativeReservoirPath} stride={MimirNativeReservoirRuntime.MoveEvidenceSampleStrideBytes} samples={samples.Length} payload=0x{handle.PayloadHandle:X} total={status.TotalSampleCount} moveEvidence={status.MoveEvidenceCount} edgeNs={status.EdgeNs}"); + + return MimirNativeReservoirRuntime.MoveEvidenceSampleStrideBytes == 112 && + samples.Length == 2 && + handle.PayloadHandle != 0 && + status.TotalSampleCount.ToUInt64() == 1 && + status.MoveEvidenceCount.ToUInt64() == 1 && + status.EdgeNs == 1_000_000_500 + ? 0 + : 1; +} + static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( @@ -1831,6 +1918,27 @@ static string DefaultObsScenePath() => "scenes", "Untitled.json"); +static string DefaultNativeReservoirPath() +{ + var fromBase = Path.Combine(AppContext.BaseDirectory, "localcast_reservoir.dll"); + if (File.Exists(fromBase)) + { + return fromBase; + } + + return Path.GetFullPath(Path.Combine( + AppContext.BaseDirectory, + "..", + "..", + "..", + "..", + "native", + "reservoir", + "target", + "debug", + "localcast_reservoir.dll")); +} + static async Task WritePerfectMachineManifestAsync(string outputPath) { var manifest = MimirPerfectMachineManifestFactory.Create(); diff --git a/src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs b/src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs new file mode 100644 index 0000000..1684976 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs @@ -0,0 +1,329 @@ +using System.Runtime.InteropServices; +using System.Text; + +namespace Mimir.Runtime.Synchronization; + +public enum MimirNativeReservoirSampleKind : uint +{ + CameraFrame = 0, + CameraFeature = 1, + SceneRay = 2, + SurfaceClaim = 3, + MaterialClaim = 4, + AudioBlock = 5, + PhaseClaim = 6, + EventClaim = 7, + RenderPacket = 8, + MoveEvidence = 9 +} + +public enum MimirNativeMoveEvidenceKind : uint +{ + OpticalMarker = 1, + ControllerState = 2 +} + +[StructLayout(LayoutKind.Sequential)] +public readonly record struct MimirNativeSampleHandle( + ulong SensorIdHash, + ulong TimestampNs, + ulong ArrivalNs, + ulong Sequence, + ulong PayloadHandle, + uint Flags, + uint Reserved); + +[StructLayout(LayoutKind.Sequential)] +public readonly record struct MimirNativeMoveEvidenceSample( + ulong WitnessIdHash, + ulong ControllerIdHash, + ulong SourceTimestampNs, + ulong ArrivalNs, + ulong Sequence, + uint EvidenceKind, + uint Flags, + float ImageX, + float ImageY, + float RadiusPx, + float Confidence, + float AccelX, + float AccelY, + float AccelZ, + float GyroX, + float GyroY, + float GyroZ, + float Trigger, + uint ButtonsMask, + uint Reserved, + float Battery01, + uint Reserved1, + uint Reserved2); + +[StructLayout(LayoutKind.Sequential)] +public readonly record struct MimirNativeMoveEvidenceBufferDescriptor( + ulong SampleBufferHandle, + uint SampleCount, + uint SampleStrideBytes, + ulong SourceTimeMinNs, + ulong SourceTimeMaxNs, + ulong CalibrationHash, + ulong TrackingSpaceHash); + +[StructLayout(LayoutKind.Sequential)] +public readonly record struct MimirNativeRuntimeStatus( + ulong EdgeNs, + ulong WindowStartNs, + UIntPtr TotalSampleCount, + UIntPtr CameraFrameCount, + UIntPtr CameraFeatureCount, + UIntPtr SceneRayCount, + UIntPtr SurfaceClaimCount, + UIntPtr MaterialClaimCount, + UIntPtr AudioBlockCount, + UIntPtr PhaseClaimCount, + UIntPtr EventClaimCount, + UIntPtr RenderPacketCount, + UIntPtr MoveEvidenceCount); + +public sealed unsafe class MimirNativeReservoirRuntime : IDisposable +{ + public const int MoveEvidenceSampleStrideBytes = 112; + + private readonly nint runtime; + private readonly List batches = []; + private bool disposed; + + public MimirNativeReservoirRuntime(string nativeLibraryPath, TimeSpan? window = null) + { + Native.Load(nativeLibraryPath); + runtime = Native.RuntimeCreate((ulong)(window ?? TimeSpan.FromSeconds(5)).TotalNanoseconds()); + if (runtime == 0) + { + throw new InvalidOperationException("Native reservoir runtime creation failed."); + } + } + + public MimirNativeRuntimeStatus Status + { + get + { + ThrowIfDisposed(); + if (!Native.RuntimeStatus(runtime, out var status)) + { + throw new InvalidOperationException("Native reservoir runtime status read failed."); + } + + return status; + } + } + + public MimirNativeSampleHandle AdmitMoveEvidence( + string producerSourceId, + IReadOnlyList samples, + string calibrationId, + string trackingSpaceId) + { + ThrowIfDisposed(); + ArgumentException.ThrowIfNullOrWhiteSpace(producerSourceId); + ArgumentException.ThrowIfNullOrWhiteSpace(calibrationId); + ArgumentException.ThrowIfNullOrWhiteSpace(trackingSpaceId); + if (samples.Count == 0) + { + throw new ArgumentException("Move evidence batch must contain at least one sample.", nameof(samples)); + } + + var producer = Native.ProducerCreateForSource( + MimirNativeReservoirSampleKind.MoveEvidence, + producerSourceId); + if (producer == 0) + { + throw new InvalidOperationException($"Native Move evidence producer creation failed for '{producerSourceId}'."); + } + + try + { + var batch = new PinnedMoveEvidenceBatch(samples, calibrationId, trackingSpaceId); + if (!Native.ProducerPushMoveEvidenceBuffer( + producer, + runtime, + batch.SourceTimeMaxNs, + batch.ArrivalMaxNs, + batch.DescriptorPointer, + out var handle)) + { + batch.Dispose(); + throw new InvalidOperationException("Native Move evidence admission failed."); + } + + batches.Add(batch); + return handle; + } + finally + { + Native.ProducerDestroy(producer); + } + } + + public void Dispose() + { + if (disposed) + { + return; + } + + disposed = true; + foreach (var batch in batches) + { + batch.Dispose(); + } + + batches.Clear(); + Native.RuntimeDestroy(runtime); + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(disposed, this); + } + + private sealed class PinnedMoveEvidenceBatch : IDisposable + { + private readonly GCHandle samplesHandle; + private readonly GCHandle descriptorHandle; + + public PinnedMoveEvidenceBatch( + IReadOnlyList samples, + string calibrationId, + string trackingSpaceId) + { + var sampleArray = samples.ToArray(); + SourceTimeMaxNs = sampleArray.Max(sample => sample.SourceTimestampNs); + ArrivalMaxNs = sampleArray.Max(sample => sample.ArrivalNs); + samplesHandle = GCHandle.Alloc(sampleArray, GCHandleType.Pinned); + var descriptor = new[] + { + new MimirNativeMoveEvidenceBufferDescriptor( + (ulong)samplesHandle.AddrOfPinnedObject(), + (uint)sampleArray.Length, + MoveEvidenceSampleStrideBytes, + sampleArray.Min(sample => sample.SourceTimestampNs), + SourceTimeMaxNs, + Fnva64(calibrationId), + Fnva64(trackingSpaceId)) + }; + descriptorHandle = GCHandle.Alloc(descriptor, GCHandleType.Pinned); + } + + public ulong SourceTimeMaxNs { get; } + + public ulong ArrivalMaxNs { get; } + + public nint DescriptorPointer => descriptorHandle.AddrOfPinnedObject(); + + public void Dispose() + { + if (descriptorHandle.IsAllocated) + { + descriptorHandle.Free(); + } + + if (samplesHandle.IsAllocated) + { + samplesHandle.Free(); + } + } + } + + private static ulong Fnva64(string value) + { + var bytes = Encoding.UTF8.GetBytes(value); + const ulong offset = 14_695_981_039_346_656_037; + const ulong prime = 1_099_511_628_211; + var hash = offset; + foreach (var b in bytes) + { + hash ^= b; + hash *= prime; + } + + return hash == 0 ? 1 : hash; + } + + private static class Native + { + private static nint library; + private static delegate* unmanaged[Cdecl] runtimeCreate; + private static delegate* unmanaged[Cdecl] runtimeDestroy; + private static delegate* unmanaged[Cdecl] runtimeStatus; + private static delegate* unmanaged[Cdecl] producerCreateForSource; + private static delegate* unmanaged[Cdecl] producerDestroy; + private static delegate* unmanaged[Cdecl] producerPushMoveEvidenceBuffer; + + public static void Load(string path) + { + if (library != 0) + { + return; + } + + if (string.IsNullOrWhiteSpace(path)) + { + path = Path.Combine(AppContext.BaseDirectory, "localcast_reservoir.dll"); + } + + library = NativeLibrary.Load(path); + runtimeCreate = (delegate* unmanaged[Cdecl])NativeLibrary.GetExport(library, "localcast_runtime_create"); + runtimeDestroy = (delegate* unmanaged[Cdecl])NativeLibrary.GetExport(library, "localcast_runtime_destroy"); + runtimeStatus = (delegate* unmanaged[Cdecl])NativeLibrary.GetExport(library, "localcast_runtime_status"); + producerCreateForSource = (delegate* unmanaged[Cdecl])NativeLibrary.GetExport(library, "localcast_producer_create_for_source"); + producerDestroy = (delegate* unmanaged[Cdecl])NativeLibrary.GetExport(library, "localcast_producer_destroy"); + producerPushMoveEvidenceBuffer = (delegate* unmanaged[Cdecl])NativeLibrary.GetExport(library, "localcast_producer_push_move_evidence_buffer"); + } + + public static nint RuntimeCreate(ulong durationNs) => runtimeCreate(durationNs); + + public static void RuntimeDestroy(nint handle) + { + if (handle != 0) + { + runtimeDestroy(handle); + } + } + + public static bool RuntimeStatus(nint handle, out MimirNativeRuntimeStatus status) => + runtimeStatus(handle, out status) != 0; + + public static nint ProducerCreateForSource(MimirNativeReservoirSampleKind kind, string sourceId) + { + var bytes = Encoding.UTF8.GetBytes(sourceId); + fixed (byte* ptr = bytes) + { + return producerCreateForSource((uint)kind, ptr, (nuint)bytes.Length, 0); + } + } + + public static void ProducerDestroy(nint handle) + { + if (handle != 0) + { + producerDestroy(handle); + } + } + + public static bool ProducerPushMoveEvidenceBuffer( + nint producer, + nint runtime, + ulong timestampNs, + ulong arrivalNs, + nint descriptor, + out MimirNativeSampleHandle handle) => + producerPushMoveEvidenceBuffer(producer, runtime, timestampNs, arrivalNs, descriptor, out handle) != 0; + + } +} + +internal static class MimirTimeSpanExtensions +{ + public static long TotalNanoseconds(this TimeSpan value) => + checked((long)(value.TotalSeconds * 1_000_000_000.0)); +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index a7bf42a..0ec30e9 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -131,3 +131,4 @@ {"ts": "2026-06-11T07:57:05+01:00", "type": "decision", "status": "ok", "note": "PS Move structured light ownership moved to Odin/Muninn: Mimir requests muninn.move_light_command.v1 over CultNet/CultMesh; Muninn serve consumes host-addressed pending commands, writes local HID report 0x06, and updates command state. Nightwing direct HID launcher path is opt-in smoke/bootstrap only."} {"ts": "2026-06-11T16:59:31+01:00", "type": "decision", "status": "ok", "note": "Move sensor fusion authority moved to Mimir: Muninn publishes source-local marker candidates and controller/IMU/button state, Odin discovers/projects schemas, Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, latency, and mimir.move_controller_pose.v1 for Fensalir interaction."} {"ts": "2026-06-11T18:47:43+01:00", "type": "implementation", "status": "ok", "note": "Added native reservoir Move evidence admission: LocalcastMoveEvidenceSample and LocalcastMoveEvidenceBufferDescriptor define a fixed compute-upload ABI, SampleKind move_evidence enters the shared rolling window, malformed buffers are rejected before live admission, and Mimir docs/maps now route Muninn marker/controller witnesses through native buffers before Mimir pose fusion and Fensalir consumption."} +{"ts": "2026-06-11T19:27:48+01:00", "type": "implementation", "status": "ok", "note": "Connected Mimir.Runtime to the native Move evidence reservoir: MimirNativeReservoirRuntime loads localcast_reservoir, pins Move evidence sample/descriptor batches, admits them through localcast_producer_push_move_evidence_buffer, and BufferSmoke --move-native-reservoir-smoke proves a marker plus controller-state batch enters the native move_evidence view."} diff --git a/state/map.yaml b/state/map.yaml index ef45a67..6f4fc56 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -92,6 +92,8 @@ accepted_design: - `native/reservoir` now exposes a `move_evidence` typed view and `LocalcastMoveEvidenceBufferDescriptor` for compute-upload buffers carrying Muninn optical marker and controller-state witnesses before Mimir fusion. + - `MimirNativeReservoirRuntime` loads the native reservoir DLL and pins Move + evidence sample/descriptor batches so admitted native handles remain valid. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for Fensalir-facing interaction. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting. From 091ab50d7c7fd0be6c22f5c8826b3e7317dc4632 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 20:01:01 +0100 Subject: [PATCH 023/135] Map Muninn Move evidence into native tracking --- docs/implementation-plan.md | 4 + docs/perfect-machine-module-library.md | 3 +- src/Mimir.BufferSmoke/Program.cs | 92 ++++++++ .../MimirMuninnMoveEvidence.cs | 210 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 4 + 6 files changed, 313 insertions(+), 1 deletion(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index e8f5cba..a93c741 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -45,6 +45,10 @@ a named invariant that the native runtime cannot protect yet. - `MimirNativeReservoirRuntime` is the managed owner for that native runtime boundary. It pins Move evidence sample/descriptor batches so native reservoir handles remain valid while Mimir prepares fusion input. +- `MimirMuninnMoveEvidenceAdapter` mirrors Muninn's marker-candidate and + controller-state document schemas and normalizes them into native Move + evidence samples. It does not associate markers to controllers or synthesize + pose. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting for those poses before Fensalir consumes diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index aa14bbb..c632593 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers, fuses them into `mimir.move_controller_pose.v1`, and Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers, fuses them into `mimir.move_controller_pose.v1`, and Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -63,6 +63,7 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test ``` diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index c571a6d..fc3d535 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -102,6 +102,12 @@ if (args.Any(arg => string.Equals(arg, "--move-native-reservoir-smoke", StringCo ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--muninn-move-evidence-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMuninnMoveEvidenceSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1526,6 +1532,92 @@ static int RunMoveNativeReservoirSmoke(string nativeReservoirPath) : 1; } +static int RunMuninnMoveEvidenceSmoke(string nativeReservoirPath) +{ + if (!File.Exists(nativeReservoirPath)) + { + Console.Error.WriteLine($"muninn-move-evidence-smoke missing-native path={nativeReservoirPath}"); + return 1; + } + + var markers = new[] + { + new MuninnMoveMarkerCandidateDocument( + StreamId: "nightwing:ps3-eye-0:move-marker-candidates", + HostId: "nightwing", + CameraId: "ps3-eye-0", + FrameSequence: 9001, + SourceIdHash: 0xA11CE, + TileX: 19, + TileY: 7, + CenterXPx: 313.25f, + CenterYPx: 118.75f, + RadiusPx: 12.5f, + AreaPx: 491, + MeanLuma: 0.62f, + PeakLuma: 252, + Score: 0.88f, + ObservedAt: "2026-06-11T18:30:00.0010000Z"), + new MuninnMoveMarkerCandidateDocument( + StreamId: "starfire:ps3-eye-1:move-marker-candidates", + HostId: "starfire", + CameraId: "ps3-eye-1", + FrameSequence: 9002, + SourceIdHash: 0xB0B, + TileX: 11, + TileY: 10, + CenterXPx: 201.5f, + CenterYPx: 160.25f, + RadiusPx: 10.75f, + AreaPx: 363, + MeanLuma: 0.58f, + PeakLuma: 246, + Score: 0.79f, + ObservedAt: "2026-06-11T18:30:00.0020000Z") + }; + var controllers = new[] + { + new MuninnMoveControllerStateDocument( + StreamId: "nightwing:move-usb:move-controller-state", + HostId: "nightwing", + MoveId: "move-usb", + Sequence: 77, + SourceTimestampNs: 1_781_194_200_003_000_000, + AccelerometerXyz: [-0.02f, 0.11f, 0.98f], + GyroscopeXyz: [0.01f, -0.02f, 0.03f], + MagnetometerXyz: [0.0f, 0.0f, 0.0f], + TriggerValue: 0.42f, + Buttons: ["move", "trigger"], + Battery01: float.NaN, + ObservedAt: "2026-06-11T18:30:00.0030000Z") + }; + + var samples = MimirMuninnMoveEvidenceAdapter.BuildNativeSamples(markers, controllers); + using var runtime = new MimirNativeReservoirRuntime(nativeReservoirPath); + var handle = runtime.AdmitMoveEvidence( + "mimir:muninn-move-evidence-smoke", + samples, + "mimir-move-stage-calibration-v1", + "mimir-stage-space"); + var status = runtime.Status; + var optical = samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker); + var controller = samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState); + var buttonMask = samples.Single(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState).ButtonsMask; + + Console.WriteLine( + $"muninn-move-evidence-smoke native={nativeReservoirPath} samples={samples.Count} optical={optical} controller={controller} buttonMask=0x{buttonMask:X} payload=0x{handle.PayloadHandle:X} total={status.TotalSampleCount} moveEvidence={status.MoveEvidenceCount} edgeNs={status.EdgeNs}"); + + return samples.Count == 3 && + optical == 2 && + controller == 1 && + buttonMask == ((1u << 17) | (1u << 18)) && + handle.PayloadHandle != 0 && + status.TotalSampleCount.ToUInt64() == 1 && + status.MoveEvidenceCount.ToUInt64() == 1 + ? 0 + : 1; +} + static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs new file mode 100644 index 0000000..e0b3483 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -0,0 +1,210 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[CultDocument("muninn.move_marker_candidate", "muninn.move_marker_candidate.v1")] +[MessagePackObject] +public sealed record MuninnMoveMarkerCandidateDocument( + [property: Key(0)] string StreamId, + [property: Key(1)] string HostId, + [property: Key(2)] string CameraId, + [property: Key(3)] ulong FrameSequence, + [property: Key(4)] ulong SourceIdHash, + [property: Key(5)] uint TileX, + [property: Key(6)] uint TileY, + [property: Key(7)] float CenterXPx, + [property: Key(8)] float CenterYPx, + [property: Key(9)] float RadiusPx, + [property: Key(10)] uint AreaPx, + [property: Key(11)] float MeanLuma, + [property: Key(12)] uint PeakLuma, + [property: Key(13)] float Score, + [property: Key(14)] string ObservedAt); + +[CultDocument("muninn.move_controller_state", "muninn.move_controller_state.v1")] +[MessagePackObject] +public sealed record MuninnMoveControllerStateDocument( + [property: Key(0)] string StreamId, + [property: Key(1)] string HostId, + [property: Key(2)] string MoveId, + [property: Key(3)] ulong Sequence, + [property: Key(4)] long SourceTimestampNs, + [property: Key(5)] float[] AccelerometerXyz, + [property: Key(6)] float[] GyroscopeXyz, + [property: Key(7)] float[] MagnetometerXyz, + [property: Key(8)] float TriggerValue, + [property: Key(9)] string[] Buttons, + [property: Key(10)] float Battery01, + [property: Key(11)] string ObservedAt); + +public static class MimirMuninnMoveEvidenceAdapter +{ + public static IReadOnlyList BuildNativeSamples( + IEnumerable markerCandidates, + IEnumerable controllerStates) + { + ArgumentNullException.ThrowIfNull(markerCandidates); + ArgumentNullException.ThrowIfNull(controllerStates); + var samples = new List(); + samples.AddRange(markerCandidates.Select(ToNativeSample)); + samples.AddRange(controllerStates.Select(ToNativeSample)); + return samples + .OrderBy(sample => sample.SourceTimestampNs) + .ThenBy(sample => sample.ArrivalNs) + .ThenBy(sample => sample.Sequence) + .ToArray(); + } + + public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveMarkerCandidateDocument marker) + { + ArgumentNullException.ThrowIfNull(marker); + var observedNs = ObservedAtToUnixNs(marker.ObservedAt); + var timestampNs = observedNs == 0 + ? checked((ulong)marker.FrameSequence) + : observedNs; + var witnessHash = marker.SourceIdHash != 0 + ? marker.SourceIdHash + : Fnva64($"{marker.HostId}:{marker.CameraId}:{marker.StreamId}"); + + return new MimirNativeMoveEvidenceSample( + WitnessIdHash: witnessHash, + ControllerIdHash: 0, + SourceTimestampNs: timestampNs, + ArrivalNs: timestampNs, + Sequence: marker.FrameSequence, + EvidenceKind: (uint)MimirNativeMoveEvidenceKind.OpticalMarker, + Flags: 0, + ImageX: marker.CenterXPx, + ImageY: marker.CenterYPx, + RadiusPx: marker.RadiusPx, + Confidence: Math.Clamp(marker.Score, 0.0f, 1.0f), + AccelX: 0.0f, + AccelY: 0.0f, + AccelZ: 0.0f, + GyroX: 0.0f, + GyroY: 0.0f, + GyroZ: 0.0f, + Trigger: 0.0f, + ButtonsMask: 0, + Reserved: 0, + Battery01: float.NaN, + Reserved1: marker.TileX, + Reserved2: marker.TileY); + } + + public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveControllerStateDocument controller) + { + ArgumentNullException.ThrowIfNull(controller); + var observedNs = ObservedAtToUnixNs(controller.ObservedAt); + var sourceNs = controller.SourceTimestampNs > 0 + ? checked((ulong)controller.SourceTimestampNs) + : observedNs; + var vector = ExtractVectors(controller); + + return new MimirNativeMoveEvidenceSample( + WitnessIdHash: Fnva64(controller.StreamId), + ControllerIdHash: Fnva64($"{controller.HostId}:{controller.MoveId}"), + SourceTimestampNs: sourceNs, + ArrivalNs: observedNs == 0 ? sourceNs : observedNs, + Sequence: controller.Sequence, + EvidenceKind: (uint)MimirNativeMoveEvidenceKind.ControllerState, + Flags: 0, + ImageX: float.NaN, + ImageY: float.NaN, + RadiusPx: float.NaN, + Confidence: 1.0f, + AccelX: vector.AccelX, + AccelY: vector.AccelY, + AccelZ: vector.AccelZ, + GyroX: vector.GyroX, + GyroY: vector.GyroY, + GyroZ: vector.GyroZ, + Trigger: Math.Clamp(controller.TriggerValue, 0.0f, 1.0f), + ButtonsMask: ButtonMask(controller.Buttons), + Reserved: 0, + Battery01: float.IsFinite(controller.Battery01) + ? Math.Clamp(controller.Battery01, 0.0f, 1.0f) + : float.NaN, + Reserved1: 0, + Reserved2: 0); + } + + public static ulong Fnva64(string value) + { + var hash = 14_695_981_039_346_656_037UL; + foreach (var b in System.Text.Encoding.UTF8.GetBytes(value)) + { + hash ^= b; + hash *= 1_099_511_628_211UL; + } + + return hash == 0 ? 1 : hash; + } + + private static ulong ObservedAtToUnixNs(string observedAt) + { + if (!DateTimeOffset.TryParse(observedAt, out var parsed)) + { + return 0; + } + + var secondsNs = checked((ulong)parsed.ToUnixTimeSeconds() * 1_000_000_000UL); + var tickRemainderNs = checked((ulong)(parsed.Ticks % TimeSpan.TicksPerSecond) * 100UL); + return checked(secondsNs + tickRemainderNs); + } + + private static (float AccelX, float AccelY, float AccelZ, float GyroX, float GyroY, float GyroZ) ExtractVectors( + MuninnMoveControllerStateDocument controller) + { + var accel = controller.AccelerometerXyz ?? []; + var gyro = controller.GyroscopeXyz ?? []; + return ( + Component(accel, 0), + Component(accel, 1), + Component(accel, 2), + Component(gyro, 0), + Component(gyro, 1), + Component(gyro, 2)); + } + + private static float Component(IReadOnlyList values, int index) => + index < values.Count && float.IsFinite(values[index]) ? values[index] : 0.0f; + + private static uint ButtonMask(IEnumerable? buttons) + { + var mask = 0u; + foreach (var button in buttons ?? []) + { + var bit = button.ToLowerInvariant() switch + { + "select" => 0, + "l3" => 1, + "r3" => 2, + "start" => 3, + "up" => 4, + "right" => 5, + "down" => 6, + "left" => 7, + "l2" => 8, + "r2" => 9, + "l1" => 10, + "r1" => 11, + "triangle" => 12, + "circle" => 13, + "cross" => 14, + "square" => 15, + "ps" => 16, + "move" => 17, + "trigger" => 18, + _ => -1 + }; + if (bit >= 0) + { + mask |= 1u << bit; + } + } + + return mask; + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 0ec30e9..330c4f1 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -132,3 +132,4 @@ {"ts": "2026-06-11T16:59:31+01:00", "type": "decision", "status": "ok", "note": "Move sensor fusion authority moved to Mimir: Muninn publishes source-local marker candidates and controller/IMU/button state, Odin discovers/projects schemas, Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, latency, and mimir.move_controller_pose.v1 for Fensalir interaction."} {"ts": "2026-06-11T18:47:43+01:00", "type": "implementation", "status": "ok", "note": "Added native reservoir Move evidence admission: LocalcastMoveEvidenceSample and LocalcastMoveEvidenceBufferDescriptor define a fixed compute-upload ABI, SampleKind move_evidence enters the shared rolling window, malformed buffers are rejected before live admission, and Mimir docs/maps now route Muninn marker/controller witnesses through native buffers before Mimir pose fusion and Fensalir consumption."} {"ts": "2026-06-11T19:27:48+01:00", "type": "implementation", "status": "ok", "note": "Connected Mimir.Runtime to the native Move evidence reservoir: MimirNativeReservoirRuntime loads localcast_reservoir, pins Move evidence sample/descriptor batches, admits them through localcast_producer_push_move_evidence_buffer, and BufferSmoke --move-native-reservoir-smoke proves a marker plus controller-state batch enters the native move_evidence view."} +{"ts": "2026-06-11T19:58:56+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir-side Muninn Move evidence normalization: muninn.move_marker_candidate.v1 and muninn.move_controller_state.v1 documents now map into native Move evidence samples, and BufferSmoke --muninn-move-evidence-smoke proves two optical candidates plus one controller-state record enter the native move_evidence reservoir view."} diff --git a/state/map.yaml b/state/map.yaml index 6f4fc56..51e2007 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -94,6 +94,10 @@ accepted_design: Muninn optical marker and controller-state witnesses before Mimir fusion. - `MimirNativeReservoirRuntime` loads the native reservoir DLL and pins Move evidence sample/descriptor batches so admitted native handles remain valid. + - `MimirMuninnMoveEvidenceAdapter` mirrors Muninn's + `muninn.move_marker_candidate.v1` and `muninn.move_controller_state.v1` + document shapes and maps them into native Move evidence samples without + claiming association or pose authority. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for Fensalir-facing interaction. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting. From 6848f762f0f2c4ced81cc2e4bcc17a2af051becf Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 20:04:33 +0100 Subject: [PATCH 024/135] Persist Mimir face heartbeat --- .voidbot/status/void-memory-maintenance.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index 8f94498..35f080a 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-06-11T17:40:32.020Z", + "startedAt": "2026-06-11T18:43:57.863Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" From 07852766fd9d9d9ed418ec42a1d6a0ef3678c097 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 20:28:30 +0100 Subject: [PATCH 025/135] Use CultMesh stream for Muninn Move evidence --- docs/implementation-plan.md | 3 + docs/perfect-machine-module-library.md | 1 + .../Mimir.BufferSmoke.csproj | 1 + src/Mimir.BufferSmoke/Program.cs | 94 +++++++++++++++++++ src/Mimir.Runtime/Mimir.Runtime.csproj | 1 + .../MimirMuninnMoveEvidence.cs | 69 ++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 2 + 8 files changed, 172 insertions(+) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index a93c741..9a6bfd1 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -49,6 +49,9 @@ a named invariant that the native runtime cannot protect yet. controller-state document schemas and normalizes them into native Move evidence samples. It does not associate markers to controllers or synthesize pose. +- The realtime Move witness path uses CultMesh streaming, not CultCache polling: + Muninn publishes shared-memory stream frames, Mimir consumes the latest read + lease, normalizes the frame, and admits it to native `move_evidence`. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting for those poses before Fensalir consumes diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index c632593..d5c3df0 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -64,6 +64,7 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test ``` diff --git a/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj b/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj index 7209bb8..8cf0920 100644 --- a/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj +++ b/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj @@ -11,6 +11,7 @@ + diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index fc3d535..e438bc3 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -4,6 +4,7 @@ using System.Security.Cryptography; using System.Text.Json; using GameCult.Caching; using GameCult.Caching.MessagePack; +using GameCult.Mesh; using MessagePack; using Mimir.Runtime.Synchronization; @@ -108,6 +109,12 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-evidence-smoke", StringCom ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--muninn-move-cultmesh-stream-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMuninnMoveCultMeshStreamSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1618,6 +1625,93 @@ static int RunMuninnMoveEvidenceSmoke(string nativeReservoirPath) : 1; } +static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) +{ + if (!File.Exists(nativeReservoirPath)) + { + Console.Error.WriteLine($"muninn-move-cultmesh-stream-smoke missing-native path={nativeReservoirPath}"); + return 1; + } + + var streamId = "muninn:nightwing:move-evidence"; + var catalog = CultMesh.CreateStreamCatalog(); + catalog.Declare(MimirMuninnMoveEvidenceAdapter.CreateStreamDescriptor( + streamId, + "mimir-live", + "muninn:nightwing", + "muninn:nightwing:clock")); + using var ring = catalog.CreateSharedMemoryRing(streamId, slotCount: 4, slotByteLength: 8192); + var frame = new MimirMuninnMoveEvidenceStreamFrame( + "nightwing-move-frame:78", + "muninn:nightwing", + 1_781_202_600_004_000_000, + [ + new MuninnMoveMarkerCandidateDocument( + StreamId: "nightwing:ps3-eye-0:move-marker-candidates", + HostId: "nightwing", + CameraId: "ps3-eye-0", + FrameSequence: 9003, + SourceIdHash: 0xA11CE, + TileX: 19, + TileY: 7, + CenterXPx: 314.0f, + CenterYPx: 119.0f, + RadiusPx: 12.25f, + AreaPx: 470, + MeanLuma: 0.61f, + PeakLuma: 253, + Score: 0.87f, + ObservedAt: "2026-06-11T18:30:00.0040000Z") + ], + [ + new MuninnMoveControllerStateDocument( + StreamId: "nightwing:move-usb:move-controller-state", + HostId: "nightwing", + MoveId: "move-usb", + Sequence: 78, + SourceTimestampNs: 1_781_202_600_004_000_000, + AccelerometerXyz: [-0.01f, 0.10f, 0.99f], + GyroscopeXyz: [0.02f, -0.01f, 0.04f], + MagnetometerXyz: [0.0f, 0.0f, 0.0f], + TriggerValue: 0.5f, + Buttons: ["move"], + Battery01: float.NaN, + ObservedAt: "2026-06-11T18:30:00.0040000Z") + ]); + var payload = MimirMuninnMoveEvidenceAdapter.SerializeStreamFrame(frame); + if (!ring.TryPublishCopy(payload, frame.PublishedAtNs, durationNs: 0, out var published)) + { + Console.Error.WriteLine("muninn-move-cultmesh-stream-smoke publish-failed"); + return 1; + } + + catalog.PublishFrame(published); + using var runtime = new MimirNativeReservoirRuntime(nativeReservoirPath); + var admitted = MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame( + ring, + runtime, + "mimir:muninn-move-cultmesh-stream-smoke", + "mimir-move-stage-calibration-v1", + "mimir-stage-space", + out var handle, + out var sampleCount); + var status = runtime.Status; + var latest = catalog.LatestFrame(streamId); + + Console.WriteLine( + $"muninn-move-cultmesh-stream-smoke stream={streamId} bytes={payload.Length} publishedSeq={published.Sequence} latestSeq={latest?.Sequence.ToString() ?? "none"} samples={sampleCount} payload=0x{handle.PayloadHandle:X} total={status.TotalSampleCount} moveEvidence={status.MoveEvidenceCount} edgeNs={status.EdgeNs}"); + + return admitted && + payload.Length > 0 && + sampleCount == 2 && + latest?.Sequence == published.Sequence && + handle.PayloadHandle != 0 && + status.TotalSampleCount.ToUInt64() == 1 && + status.MoveEvidenceCount.ToUInt64() == 1 + ? 0 + : 1; +} + static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( diff --git a/src/Mimir.Runtime/Mimir.Runtime.csproj b/src/Mimir.Runtime/Mimir.Runtime.csproj index 6745c44..84cbe2f 100644 --- a/src/Mimir.Runtime/Mimir.Runtime.csproj +++ b/src/Mimir.Runtime/Mimir.Runtime.csproj @@ -13,6 +13,7 @@ + diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index e0b3483..7031514 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -1,4 +1,5 @@ using GameCult.Caching; +using GameCult.Mesh; using MessagePack; namespace Mimir.Runtime.Synchronization; @@ -38,8 +39,76 @@ public sealed record MuninnMoveControllerStateDocument( [property: Key(10)] float Battery01, [property: Key(11)] string ObservedAt); +[MessagePackObject] +public sealed record MimirMuninnMoveEvidenceStreamFrame( + [property: Key(0)] string FrameId, + [property: Key(1)] string ProducerPeerId, + [property: Key(2)] long PublishedAtNs, + [property: Key(3)] MuninnMoveMarkerCandidateDocument[] MarkerCandidates, + [property: Key(4)] MuninnMoveControllerStateDocument[] ControllerStates); + public static class MimirMuninnMoveEvidenceAdapter { + public const string StreamMetadataSchemaId = "mimir.muninn_move_evidence_stream_frame.v1"; + + public static CultMeshStreamDescriptor CreateStreamDescriptor( + string streamId, + string verseId, + string ownerPeerId, + string clockDomainId) => + new( + streamId, + verseId, + ownerPeerId, + CultMeshStreamKind.Bytes, + new CultMeshStreamClock(clockDomainId, sourceId: streamId, confidence: 1.0, evidenceKind: "muninn-move-evidence"), + [CultMeshStreamBodyTransport.SharedMemory, CultMeshStreamBodyTransport.CultCachePage], + label: "Muninn Move evidence", + requiredAccess: CultMeshStreamAccess.Read, + maxInFlightFrames: 4, + metadataSchemaId: StreamMetadataSchemaId); + + public static byte[] SerializeStreamFrame( + MimirMuninnMoveEvidenceStreamFrame frame) => + MessagePackSerializer.Serialize(frame); + + public static MimirMuninnMoveEvidenceStreamFrame DeserializeStreamFrame( + ReadOnlyMemory bytes) => + MessagePackSerializer.Deserialize(bytes); + + public static bool TryAdmitLatestCultMeshFrame( + CultMeshSharedMemoryFrameRing ring, + MimirNativeReservoirRuntime runtime, + string producerSourceId, + string calibrationId, + string trackingSpaceId, + out MimirNativeSampleHandle handle, + out int sampleCount) + { + ArgumentNullException.ThrowIfNull(ring); + ArgumentNullException.ThrowIfNull(runtime); + handle = default; + sampleCount = 0; + if (!ring.TryAcquireLatestRead(out var lease)) + { + return false; + } + + using (lease) + { + var frame = DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]); + var samples = BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates); + sampleCount = samples.Count; + if (sampleCount == 0) + { + return false; + } + + handle = runtime.AdmitMoveEvidence(producerSourceId, samples, calibrationId, trackingSpaceId); + return true; + } + } + public static IReadOnlyList BuildNativeSamples( IEnumerable markerCandidates, IEnumerable controllerStates) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 330c4f1..02e184d 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -133,3 +133,4 @@ {"ts": "2026-06-11T18:47:43+01:00", "type": "implementation", "status": "ok", "note": "Added native reservoir Move evidence admission: LocalcastMoveEvidenceSample and LocalcastMoveEvidenceBufferDescriptor define a fixed compute-upload ABI, SampleKind move_evidence enters the shared rolling window, malformed buffers are rejected before live admission, and Mimir docs/maps now route Muninn marker/controller witnesses through native buffers before Mimir pose fusion and Fensalir consumption."} {"ts": "2026-06-11T19:27:48+01:00", "type": "implementation", "status": "ok", "note": "Connected Mimir.Runtime to the native Move evidence reservoir: MimirNativeReservoirRuntime loads localcast_reservoir, pins Move evidence sample/descriptor batches, admits them through localcast_producer_push_move_evidence_buffer, and BufferSmoke --move-native-reservoir-smoke proves a marker plus controller-state batch enters the native move_evidence view."} {"ts": "2026-06-11T19:58:56+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir-side Muninn Move evidence normalization: muninn.move_marker_candidate.v1 and muninn.move_controller_state.v1 documents now map into native Move evidence samples, and BufferSmoke --muninn-move-evidence-smoke proves two optical candidates plus one controller-state record enter the native move_evidence reservoir view."} +{"ts": "2026-06-11T20:26:19+01:00", "type": "correction", "status": "ok", "note": "Corrected the Move witness live path from CultCache polling to CultMesh streaming: MimirMuninnMoveEvidenceAdapter now creates/consumes Muninn Move shared-memory stream frames, BufferSmoke --muninn-move-cultmesh-stream-smoke proves a CultMesh latest-frame read lease normalizes into native move_evidence, and CultCache remains receipt/debug state rather than the hot tracking feed."} diff --git a/state/map.yaml b/state/map.yaml index 51e2007..af1abd7 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -98,6 +98,8 @@ accepted_design: `muninn.move_marker_candidate.v1` and `muninn.move_controller_state.v1` document shapes and maps them into native Move evidence samples without claiming association or pose authority. + - Realtime Muninn Move witnesses enter over CultMesh streaming frames. CultCache + remains persistence/receipt/debug state, not the hot tracking feed. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for Fensalir-facing interaction. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting. From dccf3b15997c9b810dceb54d61a66c9970ea4ba4 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 11 Jun 2026 20:33:51 +0100 Subject: [PATCH 026/135] Persist Mimir face heartbeat --- .voidbot/status/void-memory-maintenance.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index 35f080a..9400dfe 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-06-11T18:43:57.863Z", + "startedAt": "2026-06-11T19:15:04.423Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" From 43af2a24bde879292dc72585fa64db6a37695dd3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 14:20:28 +0100 Subject: [PATCH 027/135] Add calibrated Move pose fusion stream --- docs/implementation-plan.md | 11 + docs/perfect-machine-module-library.md | 14 +- src/Mimir.BufferSmoke/Program.cs | 156 +++++++++ .../Synchronization/MimirMoveFusion.cs | 307 ++++++++++++++++++ .../Synchronization/MimirMovePoseStream.cs | 71 ++++ state/evidence.jsonl | 1 + state/map.yaml | 10 + 7 files changed, 566 insertions(+), 4 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs create mode 100644 src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 9a6bfd1..da82fc0 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -56,6 +56,17 @@ a named invariant that the native runtime cannot protect yet. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting for those poses before Fensalir consumes them as interactive controller input. +- `MimirMoveFusion` is the first calibrated fusion owner for Move tracking: + it consumes native Muninn Move evidence samples, requires calibrated camera + witnesses before publishing a pose, associates marker candidates with the + USB controller state, triangulates multi-camera orb position, demotes + single-ray fallbacks through confidence, carries buttons/trigger/gyro into + `mimir.move_controller_pose.v1`, and marks orientation as + `orientation:imu-unresolved` until the IMU/prediction pass earns full 6DoF. +- `MimirMovePoseStream` frames resolved Move poses as + `mimir.move_controller_pose_stream_frame.v1` over CultMesh shared-memory + bytes streams so Fensalir and other consumers have a realtime stream contract + for Mimir-fused controller input. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index d5c3df0..3f5b5d8 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers, fuses them into `mimir.move_controller_pose.v1`, and Fensalir consumes the resolved pose stream for interactive environments. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveFusion` now publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -46,9 +46,14 @@ job-interview answer for why it belongs in the body of the machine. - USB-attached Moves on Starfire and Nightwing publish source-local Muninn evidence streams: glowing-orb marker candidates and controller/IMU/button state. Odin discovers/projects those streams. Mimir owns native evidence - buffer admission/layout, calibration, association, triangulation, IMU fusion, - prediction, and `mimir.move_controller_pose.v1` publication. Fensalir - consumes that resolved pose stream for interactive environments. + buffer admission/layout, calibration, association, triangulation, confidence, + and `mimir.move_controller_pose.v1` publication. The current fusion pass + resolves calibrated optical position and controller state; orientation remains + explicitly unresolved until IMU fusion/prediction lands. Fensalir consumes the + resolved pose stream for interactive environments once that contract is wired. +- Resolved Move poses are framed as + `mimir.move_controller_pose_stream_frame.v1` on CultMesh shared-memory bytes + streams; the stream is a Mimir output surface, not a new fusion owner. - OBS may receive compatibility program surfaces and stems. It does not own calibration, composition, preview/control, or publication. @@ -65,6 +70,7 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-fusion-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test ``` diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index e438bc3..ed4ac45 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -115,6 +115,11 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-cultmesh-stream-smoke", St ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveFusionSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1712,6 +1717,157 @@ static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) : 1; } +static int RunMoveFusionSmoke() +{ + const ulong leftWitnessHash = 0x00000000000A11CE; + const ulong rightWitnessHash = 0x0000000000000B0B; + var observedAt = "2026-06-11T18:30:00.0040000Z"; + var markers = new[] + { + new MuninnMoveMarkerCandidateDocument( + StreamId: "nightwing:ps3-eye-0:move-marker-candidates", + HostId: "nightwing", + CameraId: "ps3-eye-0", + FrameSequence: 9101, + SourceIdHash: leftWitnessHash, + TileX: 10, + TileY: 10, + CenterXPx: 170.0f, + CenterYPx: 120.0f, + RadiusPx: 12.0f, + AreaPx: 452, + MeanLuma: 0.64f, + PeakLuma: 252, + Score: 0.90f, + ObservedAt: observedAt), + new MuninnMoveMarkerCandidateDocument( + StreamId: "starfire:ps3-eye-1:move-marker-candidates", + HostId: "starfire", + CameraId: "ps3-eye-1", + FrameSequence: 9102, + SourceIdHash: rightWitnessHash, + TileX: 11, + TileY: 10, + CenterXPx: 150.0f, + CenterYPx: 120.0f, + RadiusPx: 12.0f, + AreaPx: 449, + MeanLuma: 0.62f, + PeakLuma: 250, + Score: 0.86f, + ObservedAt: observedAt) + }; + var controllers = new[] + { + new MuninnMoveControllerStateDocument( + StreamId: "nightwing:move-usb:move-controller-state", + HostId: "nightwing", + MoveId: "move-usb", + Sequence: 79, + SourceTimestampNs: 1_781_202_600_004_000_000, + AccelerometerXyz: [-0.01f, 0.10f, 0.99f], + GyroscopeXyz: [0.02f, -0.01f, 0.04f], + MagnetometerXyz: [0.0f, 0.0f, 0.0f], + TriggerValue: 0.5f, + Buttons: ["move", "trigger"], + Battery01: 0.72f, + ObservedAt: observedAt) + }; + var calibration = new MimirMoveFusionRigCalibration( + CalibrationId: "mimir-move-stage-calibration-smoke", + TrackingSpaceId: "mimir-stage-space", + Cameras: + [ + new MimirMoveFusionCameraCalibration( + CameraId: "nightwing:ps3-eye-0", + WitnessIdHash: leftWitnessHash, + PositionMeters: new MimirVector3Snapshot(-0.1, 0.0, 0.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx: 100.0, + FocalLengthYPx: 100.0, + PrincipalPointXPx: 160.0, + PrincipalPointYPx: 120.0), + new MimirMoveFusionCameraCalibration( + CameraId: "starfire:ps3-eye-1", + WitnessIdHash: rightWitnessHash, + PositionMeters: new MimirVector3Snapshot(0.1, 0.0, 0.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx: 100.0, + FocalLengthYPx: 100.0, + PrincipalPointXPx: 160.0, + PrincipalPointYPx: 120.0) + ], + MaximumAssociationSkewMilliseconds: 20.0); + + var samples = MimirMuninnMoveEvidenceAdapter.BuildNativeSamples(markers, controllers); + var fused = MimirMoveFusion.Fuse(samples, calibration); + var uncalibrated = MimirMoveFusion.Fuse(samples, calibration with { Cameras = [] }); + var pose = fused.Poses.SingleOrDefault(); + var streamId = "mimir:starfire:move-controller-poses"; + var catalog = CultMesh.CreateStreamCatalog(); + catalog.Declare(MimirMovePoseStream.CreateStreamDescriptor( + streamId, + "mimir-live", + "mimir:starfire", + "mimir:starfire:clock", + calibration.TrackingSpaceId, + calibration.CalibrationId)); + using var ring = catalog.CreateSharedMemoryRing(streamId, slotCount: 4, slotByteLength: 8192); + var poseFrame = MimirMovePoseStream.CreateFrame( + frameId: "mimir-move-pose-frame:79", + producerPeerId: "mimir:starfire", + publishedAtNs: pose?.EstimatedAtNs ?? 0, + trackingSpaceId: calibration.TrackingSpaceId, + calibrationId: calibration.CalibrationId, + poses: fused.Poses); + var posePayload = MimirMovePoseStream.SerializeFrame(poseFrame); + var streamOk = ring.TryPublishCopy(posePayload, poseFrame.PublishedAtNs, durationNs: 0, out var published); + MimirMoveControllerPoseStreamFrame? decodedPoseFrame = null; + if (streamOk) + { + catalog.PublishFrame(published); + streamOk = ring.TryAcquireLatestRead(out var lease); + if (streamOk) + { + using (lease) + { + decodedPoseFrame = MimirMovePoseStream.DeserializeFrame(lease.Memory[..lease.Handle.ByteLength]); + } + } + } + + var positionOk = pose is not null && + Math.Abs(pose.PositionMeters.X) <= 0.025 && + Math.Abs(pose.PositionMeters.Y) <= 0.025 && + Math.Abs(pose.PositionMeters.Z - 1.0) <= 0.05; + var evidenceOk = pose is not null && + pose.EvidenceKinds.Contains("optical-marker:triangulated", StringComparer.Ordinal) && + pose.EvidenceKinds.Contains("orientation:imu-unresolved", StringComparer.Ordinal) && + pose.Buttons.Any(button => button.Id == "move" && button.Pressed) && + pose.Buttons.Any(button => button.Id == "trigger" && button.Value >= 0.49); + streamOk = streamOk && + posePayload.Length > 0 && + catalog.LatestFrame(streamId)?.Sequence == published.Sequence && + decodedPoseFrame?.Poses.Length == 1 && + decodedPoseFrame.Poses[0].PoseId == pose?.PoseId && + decodedPoseFrame.TrackingSpaceId == calibration.TrackingSpaceId && + decodedPoseFrame.CalibrationId == calibration.CalibrationId; + + Console.WriteLine( + $"move-fusion-smoke poses={fused.Poses.Count} controllers={fused.ControllerEvidenceCount} optical={fused.OpticalEvidenceCount} calibrated={fused.CalibratedOpticalEvidenceCount} uncalibratedPoses={uncalibrated.Poses.Count} streamPoses={decodedPoseFrame?.Poses.Length ?? 0} pos=({pose?.PositionMeters.X:F3},{pose?.PositionMeters.Y:F3},{pose?.PositionMeters.Z:F3}) confidence={pose?.Confidence:F3} evidence={string.Join(",", pose?.EvidenceKinds ?? [])}"); + + return fused.Poses.Count == 1 && + fused.ControllerEvidenceCount == 1 && + fused.OpticalEvidenceCount == 2 && + fused.CalibratedOpticalEvidenceCount == 2 && + uncalibrated.Poses.Count == 0 && + positionOk && + evidenceOk && + streamOk + ? 0 + : 1; +} + static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs b/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs new file mode 100644 index 0000000..3e04ba1 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs @@ -0,0 +1,307 @@ +using System.Numerics; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirMoveFusionCameraCalibration( + string CameraId, + ulong WitnessIdHash, + MimirVector3Snapshot PositionMeters, + MimirQuaternionSnapshot Orientation, + double FocalLengthXPx, + double FocalLengthYPx, + double PrincipalPointXPx, + double PrincipalPointYPx); + +public sealed record MimirMoveFusionRigCalibration( + string CalibrationId, + string TrackingSpaceId, + IReadOnlyList Cameras, + double GyroUnitsPerRadianPerSecond = 1.0, + double MaximumAssociationSkewMilliseconds = 20.0, + double SingleRayFallbackDepthMeters = 1.5); + +public sealed record MimirMoveFusionResult( + IReadOnlyList Poses, + int ControllerEvidenceCount, + int OpticalEvidenceCount, + int CalibratedOpticalEvidenceCount); + +public static class MimirMoveFusion +{ + public static MimirMoveFusionResult Fuse( + IReadOnlyList samples, + MimirMoveFusionRigCalibration calibration, + string fusionAuthorityId = "mimir.runtime.move-fusion", + string consumerContract = "fensalir.move-controller-input") + { + ArgumentNullException.ThrowIfNull(samples); + ArgumentNullException.ThrowIfNull(calibration); + ArgumentException.ThrowIfNullOrWhiteSpace(calibration.CalibrationId); + ArgumentException.ThrowIfNullOrWhiteSpace(calibration.TrackingSpaceId); + + var controllerStates = samples + .Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState && + sample.ControllerIdHash != 0) + .OrderBy(sample => sample.SourceTimestampNs) + .ThenBy(sample => sample.Sequence) + .ToArray(); + var opticalMarkers = samples + .Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker) + .OrderBy(sample => sample.SourceTimestampNs) + .ThenBy(sample => sample.Sequence) + .ToArray(); + var cameras = calibration.Cameras + .Where(IsCameraUsable) + .ToDictionary(camera => camera.WitnessIdHash, camera => camera); + var maxSkewNs = (ulong)Math.Max(0.0, calibration.MaximumAssociationSkewMilliseconds * 1_000_000.0); + + var poses = new List(); + foreach (var controller in controllerStates) + { + var associatedMarkers = opticalMarkers + .Where(marker => IsAssociated(marker, controller, controllerStates.Length, maxSkewNs)) + .Select(marker => (Marker: marker, Camera: cameras.GetValueOrDefault(marker.WitnessIdHash))) + .Where(pair => pair.Camera is not null) + .ToArray(); + if (associatedMarkers.Length == 0) + { + continue; + } + + var rays = associatedMarkers + .Select(pair => BuildRay(pair.Camera!, pair.Marker)) + .Where(ray => ray.HasValue) + .Select(ray => ray!.Value) + .ToArray(); + if (rays.Length == 0) + { + continue; + } + + var position = rays.Length >= 2 + ? ClosestPoint(rays) + : rays[0].Origin + rays[0].Direction * (float)Math.Max(0.05, calibration.SingleRayFallbackDepthMeters); + var opticalConfidence = Math.Clamp( + associatedMarkers.Average(pair => pair.Marker.Confidence) * (rays.Length >= 2 ? 1.0 : 0.35), + 0.0, + 1.0); + var confidence = Math.Clamp(opticalConfidence * ControllerConfidence(controller), 0.0, 1.0); + var sourceNs = Math.Max(controller.SourceTimestampNs, associatedMarkers.Max(pair => pair.Marker.SourceTimestampNs)); + var arrivalNs = Math.Max(controller.ArrivalNs, associatedMarkers.Max(pair => pair.Marker.ArrivalNs)); + var evidenceStreamIds = associatedMarkers + .Select(pair => $"witness:0x{pair.Marker.WitnessIdHash:X16}") + .Append($"controller:0x{controller.ControllerIdHash:X16}") + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); + var evidenceKinds = new[] + { + rays.Length >= 2 ? "optical-marker:triangulated" : "optical-marker:single-ray-depth-prior", + "controller-state:buttons-imu", + "orientation:imu-unresolved" + }; + + poses.Add(new MimirMoveControllerPoseDocument( + PoseId: $"move:0x{controller.ControllerIdHash:X16}:{controller.Sequence}", + WandId: $"move:0x{controller.ControllerIdHash:X16}", + TrackingSpaceId: calibration.TrackingSpaceId, + CalibrationId: calibration.CalibrationId, + FusionAuthorityId: fusionAuthorityId, + SourceTimestampNs: checked((long)Math.Min(sourceNs, (ulong)long.MaxValue)), + EstimatedAtNs: checked((long)Math.Min(arrivalNs, (ulong)long.MaxValue)), + Sequence: controller.Sequence, + PositionMeters: ToSnapshot(position), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0), + AngularVelocityRadiansPerSecond: new MimirVector3Snapshot( + ScaleGyro(controller.GyroX, calibration), + ScaleGyro(controller.GyroY, calibration), + ScaleGyro(controller.GyroZ, calibration)), + Confidence: confidence, + LatencyMilliseconds: arrivalNs >= sourceNs ? (arrivalNs - sourceNs) / 1_000_000.0 : 0.0, + Battery01: float.IsFinite(controller.Battery01) ? Math.Clamp(controller.Battery01, 0.0f, 1.0f) : double.NaN, + Buttons: ButtonsFromMask(controller.ButtonsMask, controller.Trigger), + EvidenceStreamIds: evidenceStreamIds, + EvidenceKinds: evidenceKinds, + ConsumerContract: consumerContract)); + } + + return new MimirMoveFusionResult( + poses, + controllerStates.Length, + opticalMarkers.Length, + opticalMarkers.Count(marker => cameras.ContainsKey(marker.WitnessIdHash))); + } + + private static bool IsAssociated( + MimirNativeMoveEvidenceSample marker, + MimirNativeMoveEvidenceSample controller, + int controllerCount, + ulong maxSkewNs) + { + if (marker.ControllerIdHash != 0 && marker.ControllerIdHash != controller.ControllerIdHash) + { + return false; + } + + if (marker.ControllerIdHash == 0 && controllerCount != 1) + { + return false; + } + + return TimestampDelta(marker.SourceTimestampNs, controller.SourceTimestampNs) <= maxSkewNs; + } + + private static bool IsCameraUsable(MimirMoveFusionCameraCalibration camera) => + camera.WitnessIdHash != 0 && + camera.FocalLengthXPx > 0.0 && + camera.FocalLengthYPx > 0.0; + + private static MimirRay? BuildRay(MimirMoveFusionCameraCalibration camera, MimirNativeMoveEvidenceSample marker) + { + if (!float.IsFinite(marker.ImageX) || !float.IsFinite(marker.ImageY)) + { + return null; + } + + var directionCamera = Vector3.Normalize(new Vector3( + (float)((marker.ImageX - camera.PrincipalPointXPx) / camera.FocalLengthXPx), + (float)((camera.PrincipalPointYPx - marker.ImageY) / camera.FocalLengthYPx), + 1.0f)); + var orientation = ToQuaternion(camera.Orientation); + return new MimirRay( + ToVector(camera.PositionMeters), + Vector3.Normalize(Vector3.Transform(directionCamera, orientation))); + } + + private static Vector3 ClosestPoint(IReadOnlyList rays) + { + var matrix = new Matrix3(); + var rhs = Vector3.Zero; + foreach (var ray in rays) + { + var d = Vector3.Normalize(ray.Direction); + var xx = 1.0f - d.X * d.X; + var yy = 1.0f - d.Y * d.Y; + var zz = 1.0f - d.Z * d.Z; + var xy = -d.X * d.Y; + var xz = -d.X * d.Z; + var yz = -d.Y * d.Z; + matrix.M11 += xx; + matrix.M12 += xy; + matrix.M13 += xz; + matrix.M21 += xy; + matrix.M22 += yy; + matrix.M23 += yz; + matrix.M31 += xz; + matrix.M32 += yz; + matrix.M33 += zz; + rhs += new Vector3( + xx * ray.Origin.X + xy * ray.Origin.Y + xz * ray.Origin.Z, + xy * ray.Origin.X + yy * ray.Origin.Y + yz * ray.Origin.Z, + xz * ray.Origin.X + yz * ray.Origin.Y + zz * ray.Origin.Z); + } + + return matrix.TrySolve(rhs, out var solved) + ? solved + : rays[0].Origin + rays[0].Direction; + } + + private static double ControllerConfidence(MimirNativeMoveEvidenceSample controller) => + Math.Clamp( + 0.75 + + (float.IsFinite(controller.Battery01) ? 0.10 : 0.0) + + (controller.ButtonsMask != 0 ? 0.05 : 0.0) + + (Math.Abs(controller.AccelX) + Math.Abs(controller.AccelY) + Math.Abs(controller.AccelZ) > 0.001 ? 0.10 : 0.0), + 0.0, + 1.0); + + private static MimirTrackingButtonSnapshot[] ButtonsFromMask(uint mask, float trigger) + { + var names = new[] + { + "select", "l3", "r3", "start", "up", "right", "down", "left", + "l2", "r2", "l1", "r1", "triangle", "circle", "cross", "square", + "ps", "move", "trigger" + }; + var buttons = new List(); + for (var bit = 0; bit < names.Length; bit++) + { + var pressed = (mask & (1u << bit)) != 0; + if (pressed || names[bit] == "trigger") + { + buttons.Add(new MimirTrackingButtonSnapshot( + names[bit], + pressed, + names[bit] == "trigger" ? Math.Clamp(trigger, 0.0f, 1.0f) : pressed ? 1.0 : 0.0)); + } + } + + return buttons.ToArray(); + } + + private static ulong TimestampDelta(ulong left, ulong right) => left >= right ? left - right : right - left; + + private static double ScaleGyro(float value, MimirMoveFusionRigCalibration calibration) => + calibration.GyroUnitsPerRadianPerSecond <= 0.0 + ? 0.0 + : value / calibration.GyroUnitsPerRadianPerSecond; + + private static Vector3 ToVector(MimirVector3Snapshot value) => new((float)value.X, (float)value.Y, (float)value.Z); + + private static MimirVector3Snapshot ToSnapshot(Vector3 value) => new(value.X, value.Y, value.Z); + + private static Quaternion ToQuaternion(MimirQuaternionSnapshot value) + { + var q = new Quaternion((float)value.X, (float)value.Y, (float)value.Z, (float)value.W); + return q.LengthSquared() <= 1.0e-12f ? Quaternion.Identity : Quaternion.Normalize(q); + } + + private readonly record struct MimirRay(Vector3 Origin, Vector3 Direction); + + private struct Matrix3 + { + public float M11; + public float M12; + public float M13; + public float M21; + public float M22; + public float M23; + public float M31; + public float M32; + public float M33; + + public readonly bool TrySolve(Vector3 rhs, out Vector3 value) + { + var det = Determinant( + M11, M12, M13, + M21, M22, M23, + M31, M32, M33); + if (Math.Abs(det) <= 1.0e-8f) + { + value = default; + return false; + } + + value = new Vector3( + Determinant(rhs.X, M12, M13, rhs.Y, M22, M23, rhs.Z, M32, M33) / det, + Determinant(M11, rhs.X, M13, M21, rhs.Y, M23, M31, rhs.Z, M33) / det, + Determinant(M11, M12, rhs.X, M21, M22, rhs.Y, M31, M32, rhs.Z) / det); + return true; + } + + private static float Determinant( + float m11, + float m12, + float m13, + float m21, + float m22, + float m23, + float m31, + float m32, + float m33) => + m11 * (m22 * m33 - m23 * m32) - + m12 * (m21 * m33 - m23 * m31) + + m13 * (m21 * m32 - m22 * m31); + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs b/src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs new file mode 100644 index 0000000..919cdd7 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs @@ -0,0 +1,71 @@ +using GameCult.Mesh; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[MessagePackObject] +public sealed record MimirMoveControllerPoseStreamFrame( + [property: Key(0)] string FrameId, + [property: Key(1)] string ProducerPeerId, + [property: Key(2)] long PublishedAtNs, + [property: Key(3)] string TrackingSpaceId, + [property: Key(4)] string CalibrationId, + [property: Key(5)] MimirMoveControllerPoseDocument[] Poses); + +public static class MimirMovePoseStream +{ + public const string StreamMetadataSchemaId = "mimir.move_controller_pose_stream_frame.v1"; + + public static CultMeshStreamDescriptor CreateStreamDescriptor( + string streamId, + string verseId, + string ownerPeerId, + string clockDomainId, + string trackingSpaceId, + string calibrationId) => + new( + streamId, + verseId, + ownerPeerId, + CultMeshStreamKind.Bytes, + new CultMeshStreamClock(clockDomainId, sourceId: streamId, confidence: 1.0, evidenceKind: "mimir-move-controller-pose"), + [CultMeshStreamBodyTransport.SharedMemory, CultMeshStreamBodyTransport.CultCachePage], + label: $"Mimir Move controller poses ({trackingSpaceId}, {calibrationId})", + requiredAccess: CultMeshStreamAccess.Read, + maxInFlightFrames: 4, + metadataSchemaId: StreamMetadataSchemaId); + + public static MimirMoveControllerPoseStreamFrame CreateFrame( + string frameId, + string producerPeerId, + long publishedAtNs, + string trackingSpaceId, + string calibrationId, + IEnumerable poses) + { + ArgumentException.ThrowIfNullOrWhiteSpace(frameId); + ArgumentException.ThrowIfNullOrWhiteSpace(producerPeerId); + ArgumentException.ThrowIfNullOrWhiteSpace(trackingSpaceId); + ArgumentException.ThrowIfNullOrWhiteSpace(calibrationId); + ArgumentNullException.ThrowIfNull(poses); + + return new MimirMoveControllerPoseStreamFrame( + frameId, + producerPeerId, + publishedAtNs, + trackingSpaceId, + calibrationId, + poses + .Where(pose => string.Equals(pose.TrackingSpaceId, trackingSpaceId, StringComparison.Ordinal) && + string.Equals(pose.CalibrationId, calibrationId, StringComparison.Ordinal)) + .OrderBy(pose => pose.SourceTimestampNs) + .ThenBy(pose => pose.Sequence) + .ToArray()); + } + + public static byte[] SerializeFrame(MimirMoveControllerPoseStreamFrame frame) => + MessagePackSerializer.Serialize(frame); + + public static MimirMoveControllerPoseStreamFrame DeserializeFrame(ReadOnlyMemory bytes) => + MessagePackSerializer.Deserialize(bytes); +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 02e184d..c4451ac 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -134,3 +134,4 @@ {"ts": "2026-06-11T19:27:48+01:00", "type": "implementation", "status": "ok", "note": "Connected Mimir.Runtime to the native Move evidence reservoir: MimirNativeReservoirRuntime loads localcast_reservoir, pins Move evidence sample/descriptor batches, admits them through localcast_producer_push_move_evidence_buffer, and BufferSmoke --move-native-reservoir-smoke proves a marker plus controller-state batch enters the native move_evidence view."} {"ts": "2026-06-11T19:58:56+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir-side Muninn Move evidence normalization: muninn.move_marker_candidate.v1 and muninn.move_controller_state.v1 documents now map into native Move evidence samples, and BufferSmoke --muninn-move-evidence-smoke proves two optical candidates plus one controller-state record enter the native move_evidence reservoir view."} {"ts": "2026-06-11T20:26:19+01:00", "type": "correction", "status": "ok", "note": "Corrected the Move witness live path from CultCache polling to CultMesh streaming: MimirMuninnMoveEvidenceAdapter now creates/consumes Muninn Move shared-memory stream frames, BufferSmoke --muninn-move-cultmesh-stream-smoke proves a CultMesh latest-frame read lease normalizes into native move_evidence, and CultCache remains receipt/debug state rather than the hot tracking feed."} +{"ts": "2026-06-12T14:08:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveFusion as the first calibrated Move pose owner and MimirMovePoseStream as the realtime output frame: native Muninn Move evidence samples are associated against USB controller state, calibrated multi-camera orb rays triangulate position into mimir.move_controller_pose.v1, controller buttons/trigger/gyro are carried forward, orientation is explicitly marked imu-unresolved, and BufferSmoke --move-fusion-smoke proves the positive calibrated pose path, the negative no-calibration/no-pose guard, and a CultMesh pose-stream roundtrip."} diff --git a/state/map.yaml b/state/map.yaml index af1abd7..af75a17 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -103,6 +103,16 @@ accepted_design: - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for Fensalir-facing interaction. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting. + - `MimirMoveFusion` is the current Move pose owner. It consumes native Muninn + Move evidence samples, refuses to publish poses without calibrated optical + witnesses, triangulates multi-camera orb position, carries controller + buttons/trigger/gyro into `mimir.move_controller_pose.v1`, and marks + orientation as `orientation:imu-unresolved` until IMU fusion/prediction + earns a full 6DoF controller contract. + - `MimirMovePoseStream` publishes resolved Mimir-owned Move poses as + `mimir.move_controller_pose_stream_frame.v1` CultMesh shared-memory byte + frames for Fensalir and other realtime consumers; it carries poses and does + not own calibration or fusion decisions. - `scripts/start-nightwing-move-tracking.ps1` restores the live Nightwing Eye/Move witness path: Starfire owns the `/eve/periwinkle` receiver and recorder, Nightwing owns PS Move HID report `0x06` LED writes and PS3 Eye From dd838e9b1a288a72ec6d0a0196e545526ca4d5d3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 14:21:59 +0100 Subject: [PATCH 028/135] Refresh Mimir Face state --- .voidbot/state/mimir.cc | Bin 65603 -> 65799 bytes .voidbot/status/void-memory-maintenance.json | 4 ++-- .voidbot/voice/identity.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index c5866eb783905c2dff7950a4299b2c9eedde9352..5fef9e499068eff1efd11327ef729a093fa2094f 100644 GIT binary patch delta 3283 zcmbW3Ux*w@9miP+o_cQ1kiGoXq$H7uw=W2Abay0zr%UqqKdCs%mt7qju}m{N4Zs(v1pcPjukP;n4BfuLUE)aB%uk; zNl_*<_4fX2&o^fgIC9i6b=%=!RsZRFfgth0R(2ugc*-tv{Gatq|9SrOH&!5q6bOBd zv0Op3As*Slp-*OW7G1(K{7-c%R309w$qaJ#Tqkz**I)& zJ$q`V06xR22FZ}D7o2HR|IUHOU7cz+rHG`AszNfGNg1B{6dA-UBV3b6Fg0tdN7o=a zws1nb6tNgWm(1Mx?qetCTb>t&fnN*#pj&G$G{Ray=N4*F*laafyW0$!zIo*G(uXa* z-eRj^duh&VuA0U_W-N3&*38D`wt44;_bfliqVG8tYgaGrLeyVy%fGsMbn3ayKVALy zDRbwIFOM#*u&h^{0+15GAzhL{M@yhnGX+GdU|Qq5@#lcQ2$Z(%fch6YCIgNiO1d;r zyvu0-N*(00?s$qAzyS#aP8=WsB|;9`(Ks1GuKz@{%o4h_`lT`t64H#Eg-Jx%$iajj z)qn>S^KO-0$)BsUg$f;I`?SiVgbhdKv7`#m&H9Cz^I!QQ(`2kF0!$JpjK+zn%mGig z0K?2mx>m|a`+`RVE+A@nC%4;`-hk~B&XoD}ACH)S-aB@(*$(Dt!;fl_7qx3m=Cx~` zVBW8Jt@&0IbvkqHRs$>@HRs>3=G0hN^kH%Bdb8CSf`q7wh=!-Dn!_S& z-)CieqeNPBwDAs)a29dNVlc)4L_MD6Cx**0sU!Y;f z^I2=Y!>SQ;VA^lk`|@0;?zOAZdn-HFZ0R?zzpJLFeqc`C|3&a>#1)84I4)N&^=VSD z+e?2wlfU?z51#pPnbX$NQ@DGUBon$`P|1tbHW$UD#|1aTz1Yo@?VZXZ!8Po)+WhSP c!4FMBXv{Y@@7=F|VAnMYL_T<5YWl1H0{g$3&j0`b delta 2567 zcmZ8je~27c9p{>((T3bL^kN#1B+qb|VsDn2xw+lhRq~_Ms@SB7JrGl?yq$To^P1WB zChxtu%|`Jq5lra))9^qrSpN`2`cG5X{!tadriDsXDEL=Ugrz788lBrCpJ~e!mk-O{P*Ck6(had$tOx<07o?wL~`6B671{IVHTSaeNn`X3cKmBYo zRA&{(ZH8ZFgpy|}#dXRP!=>=#K(Le)GNHw!EhB@9S206%cQqqR@OXA>HI_D+f?*2ZGbNHu< zXNDMAFhy~bd&w@LpoE*ok89wixRX3qX;26$h+ai|^{%maG3|NtH0d8`|1}ND@kTW@-ch#k<8|+hwbA} z?6voA9Jhb}`96?2v#Y(-wpTVt{eABBFW7fpIY0F^d;BK{*2c;lLq4Dah#QHjw8NPj z=aro3s;dn+fTvW2=be@TSBV;Kf@@cDMiM9Gi$IiiB~%YL8kxv~WCd3S`UfVfK(*3P z?i!o;0k*6g1(XA+a0#RhGu3Zh5$>c_C|vPEUaa_K=b{w`WFd?#gsBFLwMf7dDs`qB zZZnd9Hp%7Kew$UEMVVSAoEFdnDnNQ5I$$V{{%fciRl@y-I@zI|#Q9j4QfL=3kY6t+ zHGC)paRT_3Ryq6jM|iU zNfqrWTISoPw};<7x|9BU+JQaytq1JP%d__c5$yZ<(fdDpc+cqgo2{9(-jlp=b|C|x z=Soa@PjCdO6CFChMJoqGm$9c)8v7mg9DuE%_|$CKP$Scs~ZKzY>IH&!O!}KOJMHD+ww@kdl`z6Nb zCao9ibMLNy4`US$bjF3vb5@jCfD|Sof10GoTu9nw-((JZ+xA7!m6gsJV!xgNbU=s5 zZw%xFn1O7^rAJy7DnO!nFy_{N~G Date: Fri, 12 Jun 2026 14:26:50 +0100 Subject: [PATCH 029/135] Expose Move pose stream in module catalog --- .../Synchronization/MimirModuleLibrary.cs | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index 18af289..ade4c68 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -47,15 +47,21 @@ public static class MimirModuleLibrary new( "cultmesh-contracts", "Mimir.Runtime", - "Typed state documents for codebooks, decoder state, acoustic path state, actuator state, program state, Move evidence, and Mimir-fused Move controller poses.", - ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation", "mimir.move_controller_pose"], - ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]), + "Typed state documents and stream frames for codebooks, decoder state, acoustic path state, actuator state, program state, Move evidence, and Mimir-fused Move controller poses.", + ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state", "mimir.move_tracking_observation", "mimir.move_controller_pose", "mimir.move_controller_pose_stream_frame"], + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke" + ]), new( "move-controller-fusion", "Mimir.Runtime + Fensalir", - "Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses; Fensalir consumes those poses for interaction.", - ["mimir.move_controller_pose.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], - ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke"]), + "Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses, then publishes them as a realtime CultMesh pose stream for interaction consumers.", + ["mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke" + ]), new( "path-learning", "Mimir.Runtime", From 74f2455e698eb1f886e54d037a556a089078138f Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 14:50:12 +0100 Subject: [PATCH 030/135] Add Move calibration protocol preflight --- docs/implementation-plan.md | 5 + docs/move-calibration-protocol.md | 63 +++++ docs/perfect-machine-module-library.md | 3 +- notes/current-system-map.md | 20 ++ src/Mimir.BufferSmoke/Program.cs | 43 +++ .../Synchronization/MimirModuleLibrary.cs | 3 +- .../MimirMoveCalibrationProtocol.cs | 252 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 12 + 9 files changed, 400 insertions(+), 2 deletions(-) create mode 100644 docs/move-calibration-protocol.md create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index da82fc0..8d50368 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -67,6 +67,11 @@ a named invariant that the native runtime cannot protect yet. `mimir.move_controller_pose_stream_frame.v1` over CultMesh shared-memory bytes streams so Fensalir and other consumers have a realtime stream contract for Mimir-fused controller input. +- `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for + Starfire/Nightwing Moves: required Muninn evidence streams, optional + Periwinkle motion/camera witnesses, stillness/sweep/validation phases, and + the four derived calibration outputs Mimir must produce before IMU + orientation can become authority. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field diff --git a/docs/move-calibration-protocol.md b/docs/move-calibration-protocol.md new file mode 100644 index 0000000..39c7cf3 --- /dev/null +++ b/docs/move-calibration-protocol.md @@ -0,0 +1,63 @@ +# Move Calibration Protocol + +Mimir owns Move calibration and sensor fusion. Muninn owns source-local facts: +camera marker candidates, USB controller IMU/button state, and local Move light +commands. Odin discovers and projects the surfaces. Fensalir consumes Mimir's +resolved poses after Mimir has earned them. + +## Current Stream Shape + +- Starfire should run Muninn as a local capture/control daemon. Its Move stream + address is `muninn:starfire:move-evidence`. +- Nightwing runs Muninn for its USB Move and local camera witnesses. Its Move + stream address is `muninn:nightwing:move-evidence`. +- Mimir on Starfire publishes resolved pose frames at + `mimir:starfire:move-controller-poses`. +- Periwinkle can participate through Eve sensor observations: + `periwinkle:eve:motion` and optionally `periwinkle:eve:camera`. +- Starfire ASIO/Scarlett loopback can provide cue timing, but it is not a Move + pose authority. + +The Move hot path is CultMesh stream frames. On the same host, those frames use +shared-memory bytes rings. Across hosts, the remote peer still crosses the +Verse/network boundary. The current Raven program media bridge is not same-host +zero-copy into Mimir/Fensalir; it sends CultNet/CultMesh media documents and +lowers to local UDP for OBS compatibility. + +## Required Capture Phases + +1. `preflight-streams`: hold both Moves still and visible; prove stream freshness, + clock skew, and controller ids. +2. `dark-stillness`: lights off, both Moves still; estimate gyro bias, gravity, + and magnetometer baseline. +3. `lit-stillness`: light one Move at a time, then both; map light commands to + controllers and measure centroid stability. +4. `axis-sweeps`: slow pitch/yaw/roll sweeps; fit gyro scale, accelerometer + frame, and optical angular consistency. +5. `figure-eight`: broad figure-eight motion; fit magnetometer hard/soft iron + and cross-axis coupling. +6. `periwinkle-witness`: optional phone motion/camera witness near the active + Move for independent body and clock evidence. +7. `validation-pass`: held-out free motion; promote only if triangulated optical + residuals and IMU prediction residuals pass. + +## Data Products + +- `mimir.move_fusion_rig_calibration.v1`: camera intrinsics/extrinsics, clock + association policy, and gyro unit scale. +- `mimir.move_imu_calibration.v1`: per-controller gyro bias/scale, + accelerometer gravity frame, magnetometer correction, and noise estimates. +- `mimir.move_controller_identity_map.v1`: stable association between Muninn + USB ids, light receipts, and optical orb identities. +- `mimir.move_calibration_receipt.v1`: sample counts, residuals, rejected + phases, and promotion decision. + +## Preflight Command + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke --output artifacts\move-calibration\protocol.cc +``` + +That writes the typed protocol as `mimir.move_calibration_protocol.v1` into a +CultCache receipt. The real hardware runner should consume this document and +append capture receipts instead of inventing a parallel checklist. diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index 3f5b5d8..42e8858 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveFusion` now publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -70,6 +70,7 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-native-reservoir-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-fusion-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test diff --git a/notes/current-system-map.md b/notes/current-system-map.md index cc39295..15aab81 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -134,6 +134,18 @@ Smoke proof: `dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csp -- --move-tracking-contract-smoke` consumes one Starfire Move and one Nightwing Move observation into two tracking buffers. +Calibration preflight proof: `dotnet run --project +.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- +--move-calibration-protocol-smoke --output +artifacts\move-calibration\protocol.cc` writes +`mimir.move_calibration_protocol.v1` for +`mimir-move-calibration-starfire-nightwing-v1`. The protocol requires +`muninn:starfire:move-evidence`, `muninn:nightwing:move-evidence`, and +`mimir:starfire:move-controller-poses`; optionally drinks Periwinkle motion and +camera witnesses; and requires Mimir to derive rig calibration, IMU calibration, +controller identity mapping, and a calibration receipt before orientation can +stop being `orientation:imu-unresolved`. + Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages `nw_eye_cap.py`, `nw_move_hint.py`, and `nightwing_typed_witness_publisher.py` @@ -151,6 +163,14 @@ That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01 output collection and refreshes the PS Move LED report for hardware proof when Muninn is not available on Starfire yet. +Starfire should have Muninn for source-local Move HID control and local sensor +publication. Some older Starfire audio/video paths still enter Mimir through +local Mimir producers or diagnostic process bridges while direct native drivers +mature. Same-host Move evidence and pose frames are CultMesh shared-memory byte +rings. Remote media is different: the current Raven media body bridge uses +CultNet/CultMesh documents over the mesh and lowers to local UDP for OBS +compatibility, so it is not same-machine zero-copy into Mimir/Fensalir yet. + Odin's Muninn organ owns the Move optical extraction stage for sensor stream exposure. Its Rust crate lives in `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index ed4ac45..99ded69 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -120,6 +120,13 @@ if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.O return RunMoveFusionSmoke(); } +if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return await RunMoveCalibrationProtocolSmokeAsync( + ParseStringOption(args, "--output", "artifacts/move-calibration/protocol.cc")) + .ConfigureAwait(false); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1868,6 +1875,42 @@ static int RunMoveFusionSmoke() : 1; } +static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) +{ + var protocol = MimirMoveCalibrationProtocol.CreateStarfireNightwingProtocol( + new DateTimeOffset(2026, 6, 12, 14, 45, 0, TimeSpan.Zero)); + var errors = MimirMoveCalibrationProtocol.Validate(protocol); + if (errors.Length > 0) + { + Console.Error.WriteLine($"move-calibration-protocol-smoke invalid errors={string.Join(",", errors)}"); + return 1; + } + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions + { + PullOnOpen = File.Exists(outputPath) + }).ConfigureAwait(false); + await cache.UpsertAsync( + protocol, + new CultRecordHandle(new CultRecordKey($"mimir-move-calibration-protocol:{protocol.ProtocolId}"))) + .ConfigureAwait(false); + await cache.FlushAsync().ConfigureAwait(false); + + var requiredStreams = protocol.StreamRequirements.Count(stream => stream.Required); + var optionalStreams = protocol.StreamRequirements.Length - requiredStreams; + var totalDuration = protocol.Phases.Sum(phase => phase.DurationSeconds); + Console.WriteLine( + $"move-calibration-protocol-smoke path={outputPath} protocol={protocol.ProtocolId} requiredStreams={requiredStreams} optionalStreams={optionalStreams} phases={protocol.Phases.Length} durationSeconds={totalDuration:0.0} outputs={protocol.Outputs.Length} periwinkleOptional={protocol.StreamRequirements.Any(stream => stream.StreamId.StartsWith("periwinkle:", StringComparison.Ordinal))}"); + + return requiredStreams >= 3 && + protocol.Phases.Length == 7 && + protocol.Outputs.Length == 4 && + protocol.Acceptance.RequiredDerivedOutputs.Length == 4 + ? 0 + : 1; +} + static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index ade4c68..59b86b8 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -57,9 +57,10 @@ public static class MimirModuleLibrary "move-controller-fusion", "Mimir.Runtime + Fensalir", "Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses, then publishes them as a realtime CultMesh pose stream for interaction consumers.", - ["mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], + ["mimir.move_calibration_protocol.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], [ "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke", "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke" ]), new( diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs b/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs new file mode 100644 index 0000000..6c5f247 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs @@ -0,0 +1,252 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[CultDocument("mimir.move_calibration_protocol", "mimir.move_calibration_protocol.v1")] +[MessagePackObject] +public sealed record MimirMoveCalibrationProtocolDocument( + [property: Key(0)] + [property: CultName] + string ProtocolId, + [property: Key(1)] string CreatedAtUtc, + [property: Key(2)] string TrackingSpaceId, + [property: Key(3)] string FusionAuthorityId, + [property: Key(4)] MimirMoveCalibrationStreamRequirement[] StreamRequirements, + [property: Key(5)] MimirMoveCalibrationPhase[] Phases, + [property: Key(6)] MimirMoveCalibrationOutput[] Outputs, + [property: Key(7)] MimirMoveCalibrationAcceptance Acceptance, + [property: Key(8)] string[] OperatorNotes); + +[MessagePackObject] +public sealed record MimirMoveCalibrationStreamRequirement( + [property: Key(0)] string StreamId, + [property: Key(1)] string HostId, + [property: Key(2)] string Owner, + [property: Key(3)] string Kind, + [property: Key(4)] string Transport, + [property: Key(5)] bool Required, + [property: Key(6)] string Evidence); + +[MessagePackObject] +public sealed record MimirMoveCalibrationPhase( + [property: Key(0)] string PhaseId, + [property: Key(1)] string Title, + [property: Key(2)] double DurationSeconds, + [property: Key(3)] string OperatorCue, + [property: Key(4)] string[] RequiredStreamIds, + [property: Key(5)] string[] Produces); + +[MessagePackObject] +public sealed record MimirMoveCalibrationOutput( + [property: Key(0)] string OutputId, + [property: Key(1)] string SchemaId, + [property: Key(2)] string Owner, + [property: Key(3)] string Purpose); + +[MessagePackObject] +public sealed record MimirMoveCalibrationAcceptance( + [property: Key(0)] int MinimumControllerStateSamplesPerMove, + [property: Key(1)] int MinimumOpticalSamplesPerCamera, + [property: Key(2)] int MinimumPeriwinkleMotionSamples, + [property: Key(3)] double MaximumClockSkewMilliseconds, + [property: Key(4)] double MaximumStillGyroRadiansPerSecond, + [property: Key(5)] double MinimumTriangulatedPoseConfidence, + [property: Key(6)] string[] RequiredDerivedOutputs); + +public static class MimirMoveCalibrationProtocol +{ + public static MimirMoveCalibrationProtocolDocument CreateStarfireNightwingProtocol( + DateTimeOffset? createdAt = null) => + new( + ProtocolId: "mimir-move-calibration-starfire-nightwing-v1", + CreatedAtUtc: (createdAt ?? DateTimeOffset.UtcNow).ToString("O"), + TrackingSpaceId: "mimir-stage-space", + FusionAuthorityId: "mimir.runtime.move-fusion", + StreamRequirements: + [ + new( + "muninn:starfire:move-evidence", + "starfire", + "Muninn on Starfire", + "Move optical marker candidates plus USB controller IMU/buttons", + "CultMesh shared-memory bytes frame, same-host when Mimir runs on Starfire", + Required: true, + "Starfire owns the USB-attached Move and local camera witnesses."), + new( + "muninn:nightwing:move-evidence", + "nightwing", + "Muninn on Nightwing", + "Move optical marker candidates plus USB controller IMU/buttons", + "CultMesh bytes stream over the Verse; shared-memory on Nightwing, network bridge to Starfire", + Required: true, + "Nightwing owns its USB-attached Move and local camera witnesses."), + new( + "mimir:starfire:move-controller-poses", + "starfire", + "Mimir.Runtime", + "Mimir-fused controller pose output", + "CultMesh shared-memory bytes frame", + Required: true, + "The calibration run must prove Mimir can republish resolved poses."), + new( + "periwinkle:eve:motion", + "periwinkle", + "Periwinkle Eve client", + "phone IMU/motion witness", + "CultMesh observation ledger through Mimir Eve sensor intake", + Required: false, + "Independent body/clock witness for staged motions and operator cues."), + new( + "periwinkle:eve:camera", + "periwinkle", + "Periwinkle Eve client", + "phone camera/media witness", + "CultMesh observation ledger through Mimir Eve sensor intake", + Required: false, + "Optional visual record of calibration poses and sweep compliance."), + new( + "starfire:scarlett:loopback", + "starfire", + "Mimir.Runtime ASIO source", + "audio loopback timing witness", + "in-process ASIO into Mimir rolling buffers", + Required: false, + "Optional calibration metronome/cue timing witness; not a Move pose owner.") + ], + Phases: + [ + new( + "preflight-streams", + "Stream liveness and clock edge check", + 10.0, + "Hold both Moves still and visible. Confirm Starfire, Nightwing, and optional Periwinkle streams are fresh.", + ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["stream-freshness", "clock-skew-estimate", "controller-id-map"]), + new( + "dark-stillness", + "IMU bias with orbs off", + 8.0, + "Set Move lights off. Place both Moves still on the calibration surface.", + ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["gyro-bias", "accelerometer-gravity-vector", "magnetometer-baseline"]), + new( + "lit-stillness", + "Optical centroid and static gravity alignment", + 8.0, + "Light one Move at a time, then both. Keep them still and visible to both camera sets.", + ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["camera-marker-centroid-stability", "light-command-to-controller-association"]), + new( + "axis-sweeps", + "Slow single-axis rotations", + 24.0, + "Sweep each Move slowly through pitch, yaw, and roll while keeping the orb visible.", + ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["gyro-scale-fit", "accelerometer-frame-fit", "optical-angular-consistency"]), + new( + "figure-eight", + "Magnetometer and cross-axis motion", + 20.0, + "Move each controller through a broad figure-eight. Keep Periwinkle nearby if it is being used as the phone witness.", + ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["magnetometer-hard-soft-iron-fit", "cross-axis-coupling-check"]), + new( + "periwinkle-witness", + "Optional Periwinkle body witness", + 12.0, + "Hold Periwinkle rigidly near the active Move for a short synchronized sweep if the phone is available.", + ["periwinkle:eve:motion"], + ["independent-motion-witness", "phone-clock-offset-estimate"]), + new( + "validation-pass", + "Held-out tracking validation", + 15.0, + "Run free hand motion with both orbs visible. Mimir may publish validation poses, but not promote calibration if confidence fails.", + ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence", "mimir:starfire:move-controller-poses"], + ["triangulated-position-residuals", "imu-prediction-residuals", "promotion-decision"]) + ], + Outputs: + [ + new( + "move-rig-calibration", + "mimir.move_fusion_rig_calibration.v1", + "Mimir.Runtime", + "Camera intrinsics/extrinsics, clock association policy, and gyro unit scale for Move fusion."), + new( + "move-imu-calibration", + "mimir.move_imu_calibration.v1", + "Mimir.Runtime", + "Per-controller gyro bias/scale, accelerometer gravity frame, magnetometer correction, and noise estimates."), + new( + "move-controller-map", + "mimir.move_controller_identity_map.v1", + "Mimir.Runtime", + "Stable association between Muninn USB controller ids, light-command receipts, and optical orb identities."), + new( + "move-calibration-receipt", + "mimir.move_calibration_receipt.v1", + "Mimir.Runtime", + "Immutable receipt of captured sample counts, residuals, rejected phases, and promotion decision.") + ], + Acceptance: new( + MinimumControllerStateSamplesPerMove: 600, + MinimumOpticalSamplesPerCamera: 300, + MinimumPeriwinkleMotionSamples: 120, + MaximumClockSkewMilliseconds: 20.0, + MaximumStillGyroRadiansPerSecond: 0.035, + MinimumTriangulatedPoseConfidence: 0.65, + RequiredDerivedOutputs: + [ + "move-rig-calibration", + "move-imu-calibration", + "move-controller-map", + "move-calibration-receipt" + ]), + OperatorNotes: + [ + "Mimir owns fusion and calibration promotion. Muninn only publishes source-local evidence and executes local Move light commands.", + "CultCache receipts are durable evidence; CultMesh stream frames are the hot path.", + "Do not promote orientation until gyro bias/scale, gravity alignment, optical residuals, and held-out validation all pass." + ]); + + public static string[] Validate(MimirMoveCalibrationProtocolDocument protocol) + { + ArgumentNullException.ThrowIfNull(protocol); + var errors = new List(); + if (protocol.StreamRequirements.Count(stream => stream.Required) < 3) + { + errors.Add("required-streams-too-small"); + } + + if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "muninn:starfire:move-evidence" && stream.Required)) + { + errors.Add("missing-starfire-muninn-move-evidence"); + } + + if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "muninn:nightwing:move-evidence" && stream.Required)) + { + errors.Add("missing-nightwing-muninn-move-evidence"); + } + + if (!protocol.Phases.Any(phase => phase.PhaseId == "dark-stillness")) + { + errors.Add("missing-imu-bias-phase"); + } + + if (!protocol.Phases.Any(phase => phase.Produces.Contains("gyro-scale-fit", StringComparer.Ordinal))) + { + errors.Add("missing-gyro-scale-fit"); + } + + foreach (var requiredOutput in protocol.Acceptance.RequiredDerivedOutputs) + { + if (!protocol.Outputs.Any(output => output.OutputId == requiredOutput)) + { + errors.Add($"missing-output:{requiredOutput}"); + } + } + + return errors.ToArray(); + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index c4451ac..2a87080 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -135,3 +135,4 @@ {"ts": "2026-06-11T19:58:56+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir-side Muninn Move evidence normalization: muninn.move_marker_candidate.v1 and muninn.move_controller_state.v1 documents now map into native Move evidence samples, and BufferSmoke --muninn-move-evidence-smoke proves two optical candidates plus one controller-state record enter the native move_evidence reservoir view."} {"ts": "2026-06-11T20:26:19+01:00", "type": "correction", "status": "ok", "note": "Corrected the Move witness live path from CultCache polling to CultMesh streaming: MimirMuninnMoveEvidenceAdapter now creates/consumes Muninn Move shared-memory stream frames, BufferSmoke --muninn-move-cultmesh-stream-smoke proves a CultMesh latest-frame read lease normalizes into native move_evidence, and CultCache remains receipt/debug state rather than the hot tracking feed."} {"ts": "2026-06-12T14:08:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveFusion as the first calibrated Move pose owner and MimirMovePoseStream as the realtime output frame: native Muninn Move evidence samples are associated against USB controller state, calibrated multi-camera orb rays triangulate position into mimir.move_controller_pose.v1, controller buttons/trigger/gyro are carried forward, orientation is explicitly marked imu-unresolved, and BufferSmoke --move-fusion-smoke proves the positive calibrated pose path, the negative no-calibration/no-pose guard, and a CultMesh pose-stream roundtrip."} +{"ts": "2026-06-12T14:45:00+01:00", "type": "implementation", "status": "ok", "note": "Added typed Move calibration preflight state as mimir.move_calibration_protocol.v1. BufferSmoke --move-calibration-protocol-smoke writes artifacts/move-calibration/protocol.cc and proves the Starfire/Nightwing protocol requires both Muninn Move evidence streams plus Mimir pose output, includes optional Periwinkle motion/camera witnesses, defines stillness/sweep/validation phases, and names the four required derived outputs before IMU orientation can become authority."} diff --git a/state/map.yaml b/state/map.yaml index af75a17..344ea60 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -113,6 +113,18 @@ accepted_design: `mimir.move_controller_pose_stream_frame.v1` CultMesh shared-memory byte frames for Fensalir and other realtime consumers; it carries poses and does not own calibration or fusion decisions. + - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move + calibration preflight as typed state: required Muninn Move evidence streams, + optional Periwinkle motion/camera witnesses, stillness/sweep/validation + phases, and the calibration outputs Mimir must derive before orientation + can stop being `orientation:imu-unresolved`. + - Starfire should have Muninn for source-local Move HID control and local + sensor publication, but some Starfire audio/video paths still enter Mimir + through local Mimir producers or diagnostic process bridges while native + drivers mature. Same-host Move evidence and pose frames are CultMesh + shared-memory byte rings. Current remote media body transport is not + same-machine zero-copy into Fensalir; Raven media uses CultNet/CultMesh + documents over the mesh and lowers to local UDP only for OBS compatibility. - `scripts/start-nightwing-move-tracking.ps1` restores the live Nightwing Eye/Move witness path: Starfire owns the `/eve/periwinkle` receiver and recorder, Nightwing owns PS Move HID report `0x06` LED writes and PS3 Eye From 4ae3a07ed6c2fa8c7d13f654686439220fb85c4d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 14:50:53 +0100 Subject: [PATCH 031/135] Refresh Mimir Face heartbeat state --- .voidbot/state/mimir.cc | Bin 65799 -> 65712 bytes .voidbot/status/void-memory-maintenance.json | 4 ++-- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 5fef9e499068eff1efd11327ef729a093fa2094f..d513df63c212d5d947f354a5417cf02f83845215 100644 GIT binary patch delta 204 zcmZo~V%gBhvf(=~kFk}Bk(IHbo{52B)I>(n$r5~>P%#rdLt_Jw*yclgdInHIQ#~_7 zu)@uP_SbZvf@XT=MiyY*965@NiIcT*q2+Ae26*|BRz97vnT*bN;h%< delta 287 zcmdnc$kN`#vf(=~w~>{hiIu6Do`Hq&WWxz!lO^~%5n`66AhFGd`1B0mf`$fWU_n9q zYdQ!)LnE+mjvPhC9h0?kq?21$W)}M;78GRWr61m$Tv}9=npXnk2bUxkm87OPmh3Px zFf!9MFw-?O3NZva!r02dRL{i1IBEkzv1?vRF@{PL3z*8yfjN4Na3>g87*2k`E6xS; skfGt^kJ;jz59OvPA;gR(GxCW~cAroW*JfyBx%uLR%lSNTeI`ax0Fx_J^Z)<= diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index eabff61..c619dc5 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,7 +1,7 @@ { "status": "starting", - "startedAt": "2026-06-12T12:14:56.167Z", + "startedAt": "2026-06-12T13:42:48.078Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", - "reason": "repo_face_napping_with_short_term_memory" + "reason": "repo_face_stale_short_term_memory" } From 0616d21a5b1131d564581d04bf5f5e3998f08fc8 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 15:04:51 +0100 Subject: [PATCH 032/135] Use Quest as Move calibration witness --- docs/implementation-plan.md | 2 +- docs/move-calibration-protocol.md | 23 ++++++-- notes/current-system-map.md | 10 ++-- src/Mimir.BufferSmoke/Program.cs | 2 +- .../MimirMoveCalibrationProtocol.cs | 52 +++++++++++-------- state/evidence.jsonl | 1 + state/map.yaml | 6 +-- 7 files changed, 61 insertions(+), 35 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 8d50368..37d35eb 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -69,7 +69,7 @@ a named invariant that the native runtime cannot protect yet. for Mimir-fused controller input. - `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for Starfire/Nightwing Moves: required Muninn evidence streams, optional - Periwinkle motion/camera witnesses, stillness/sweep/validation phases, and + Quest headset/controller pose witnesses, stillness/sweep/validation phases, and the four derived calibration outputs Mimir must produce before IMU orientation can become authority. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing diff --git a/docs/move-calibration-protocol.md b/docs/move-calibration-protocol.md index 39c7cf3..bdb8daa 100644 --- a/docs/move-calibration-protocol.md +++ b/docs/move-calibration-protocol.md @@ -13,8 +13,9 @@ resolved poses after Mimir has earned them. stream address is `muninn:nightwing:move-evidence`. - Mimir on Starfire publishes resolved pose frames at `mimir:starfire:move-controller-poses`. -- Periwinkle can participate through Eve sensor observations: - `periwinkle:eve:motion` and optionally `periwinkle:eve:camera`. +- Quest can participate as an external tracked VR reference over USB: + `quest:usb:headset-pose`, `quest:usb:left-controller-pose`, and + `quest:usb:right-controller-pose`. - Starfire ASIO/Scarlett loopback can provide cue timing, but it is not a Move pose authority. @@ -36,11 +37,25 @@ lowers to local UDP for OBS compatibility. frame, and optical angular consistency. 5. `figure-eight`: broad figure-eight motion; fit magnetometer hard/soft iron and cross-axis coupling. -6. `periwinkle-witness`: optional phone motion/camera witness near the active - Move for independent body and clock evidence. +6. `quest-reference`: optional Quest headset/controller reference with the Move + between the two Quest controllers in front of the headset. 7. `validation-pass`: held-out free motion; promote only if triangulated optical residuals and IMU prediction residuals pass. +## Quest USB Preflight + +The Quest must be authorized for USB debugging before Starfire can query it. +Check it with: + +```powershell +adb devices -l +``` + +If it reports `unauthorized`, put on the headset and accept the USB debugging +prompt. The current calibration protocol can still run without Quest, but Quest +poses are the preferred external reference when the headset and both +controllers are sitting around the Move. + ## Data Products - `mimir.move_fusion_rig_calibration.v1`: camera intrinsics/extrinsics, clock diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 15aab81..24b3c29 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -141,10 +141,12 @@ artifacts\move-calibration\protocol.cc` writes `mimir.move_calibration_protocol.v1` for `mimir-move-calibration-starfire-nightwing-v1`. The protocol requires `muninn:starfire:move-evidence`, `muninn:nightwing:move-evidence`, and -`mimir:starfire:move-controller-poses`; optionally drinks Periwinkle motion and -camera witnesses; and requires Mimir to derive rig calibration, IMU calibration, -controller identity mapping, and a calibration receipt before orientation can -stop being `orientation:imu-unresolved`. +`mimir:starfire:move-controller-poses`; optionally drinks Quest headset and +left/right controller poses over USB as an external VR reference; and requires +Mimir to derive rig calibration, IMU calibration, controller identity mapping, +and a calibration receipt before orientation can stop being +`orientation:imu-unresolved`. ADB currently sees the Quest over USB, but it must +be authorized in-headset before Starfire can query it. Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 99ded69..7b174e4 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1901,7 +1901,7 @@ static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) var optionalStreams = protocol.StreamRequirements.Length - requiredStreams; var totalDuration = protocol.Phases.Sum(phase => phase.DurationSeconds); Console.WriteLine( - $"move-calibration-protocol-smoke path={outputPath} protocol={protocol.ProtocolId} requiredStreams={requiredStreams} optionalStreams={optionalStreams} phases={protocol.Phases.Length} durationSeconds={totalDuration:0.0} outputs={protocol.Outputs.Length} periwinkleOptional={protocol.StreamRequirements.Any(stream => stream.StreamId.StartsWith("periwinkle:", StringComparison.Ordinal))}"); + $"move-calibration-protocol-smoke path={outputPath} protocol={protocol.ProtocolId} requiredStreams={requiredStreams} optionalStreams={optionalStreams} phases={protocol.Phases.Length} durationSeconds={totalDuration:0.0} outputs={protocol.Outputs.Length} questOptional={protocol.StreamRequirements.Any(stream => stream.StreamId.StartsWith("quest:", StringComparison.Ordinal))}"); return requiredStreams >= 3 && protocol.Phases.Length == 7 && diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs b/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs index 6c5f247..763b92d 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs @@ -48,7 +48,7 @@ public sealed record MimirMoveCalibrationOutput( public sealed record MimirMoveCalibrationAcceptance( [property: Key(0)] int MinimumControllerStateSamplesPerMove, [property: Key(1)] int MinimumOpticalSamplesPerCamera, - [property: Key(2)] int MinimumPeriwinkleMotionSamples, + [property: Key(2)] int MinimumExternalPoseSamples, [property: Key(3)] double MaximumClockSkewMilliseconds, [property: Key(4)] double MaximumStillGyroRadiansPerSecond, [property: Key(5)] double MinimumTriangulatedPoseConfidence, @@ -90,21 +90,29 @@ public static class MimirMoveCalibrationProtocol Required: true, "The calibration run must prove Mimir can republish resolved poses."), new( - "periwinkle:eve:motion", - "periwinkle", - "Periwinkle Eve client", - "phone IMU/motion witness", - "CultMesh observation ledger through Mimir Eve sensor intake", + "quest:usb:headset-pose", + "starfire", + "Quest USB/OpenXR witness", + "Quest headset pose near the Move/controllers calibration cluster", + "USB ADB/OpenXR witness bridged into Mimir calibration capture", Required: false, - "Independent body/clock witness for staged motions and operator cues."), + "External tracked VR frame for validating Mimir's optical/IMU pose frame."), new( - "periwinkle:eve:camera", - "periwinkle", - "Periwinkle Eve client", - "phone camera/media witness", - "CultMesh observation ledger through Mimir Eve sensor intake", + "quest:usb:left-controller-pose", + "starfire", + "Quest USB/OpenXR witness", + "Quest left controller pose", + "USB ADB/OpenXR witness bridged into Mimir calibration capture", Required: false, - "Optional visual record of calibration poses and sweep compliance."), + "External tracked controller reference sitting beside the Move."), + new( + "quest:usb:right-controller-pose", + "starfire", + "Quest USB/OpenXR witness", + "Quest right controller pose", + "USB ADB/OpenXR witness bridged into Mimir calibration capture", + Required: false, + "External tracked controller reference sitting beside the Move."), new( "starfire:scarlett:loopback", "starfire", @@ -120,7 +128,7 @@ public static class MimirMoveCalibrationProtocol "preflight-streams", "Stream liveness and clock edge check", 10.0, - "Hold both Moves still and visible. Confirm Starfire, Nightwing, and optional Periwinkle streams are fresh.", + "Hold both Moves still and visible. Confirm Starfire, Nightwing, and optional Quest pose streams are fresh.", ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], ["stream-freshness", "clock-skew-estimate", "controller-id-map"]), new( @@ -148,16 +156,16 @@ public static class MimirMoveCalibrationProtocol "figure-eight", "Magnetometer and cross-axis motion", 20.0, - "Move each controller through a broad figure-eight. Keep Periwinkle nearby if it is being used as the phone witness.", + "Move each controller through a broad figure-eight. Leave the Quest headset/controllers stationary as the external VR reference cluster.", ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], ["magnetometer-hard-soft-iron-fit", "cross-axis-coupling-check"]), new( - "periwinkle-witness", - "Optional Periwinkle body witness", + "quest-reference", + "Optional Quest headset/controller reference", 12.0, - "Hold Periwinkle rigidly near the active Move for a short synchronized sweep if the phone is available.", - ["periwinkle:eve:motion"], - ["independent-motion-witness", "phone-clock-offset-estimate"]), + "Keep the Move between the two Quest controllers in front of the headset. Capture the static Quest controller/headset frame while Mimir observes the Move.", + ["quest:usb:headset-pose", "quest:usb:left-controller-pose", "quest:usb:right-controller-pose"], + ["external-vr-pose-frame", "quest-to-mimir-frame-fit", "quest-clock-offset-estimate"]), new( "validation-pass", "Held-out tracking validation", @@ -192,7 +200,7 @@ public static class MimirMoveCalibrationProtocol Acceptance: new( MinimumControllerStateSamplesPerMove: 600, MinimumOpticalSamplesPerCamera: 300, - MinimumPeriwinkleMotionSamples: 120, + MinimumExternalPoseSamples: 120, MaximumClockSkewMilliseconds: 20.0, MaximumStillGyroRadiansPerSecond: 0.035, MinimumTriangulatedPoseConfidence: 0.65, @@ -207,7 +215,7 @@ public static class MimirMoveCalibrationProtocol [ "Mimir owns fusion and calibration promotion. Muninn only publishes source-local evidence and executes local Move light commands.", "CultCache receipts are durable evidence; CultMesh stream frames are the hot path.", - "Do not promote orientation until gyro bias/scale, gravity alignment, optical residuals, and held-out validation all pass." + "Do not promote orientation until gyro bias/scale, gravity alignment, optical residuals, external Quest reference residuals when available, and held-out validation all pass." ]); public static string[] Validate(MimirMoveCalibrationProtocolDocument protocol) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 2a87080..24ed704 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -136,3 +136,4 @@ {"ts": "2026-06-11T20:26:19+01:00", "type": "correction", "status": "ok", "note": "Corrected the Move witness live path from CultCache polling to CultMesh streaming: MimirMuninnMoveEvidenceAdapter now creates/consumes Muninn Move shared-memory stream frames, BufferSmoke --muninn-move-cultmesh-stream-smoke proves a CultMesh latest-frame read lease normalizes into native move_evidence, and CultCache remains receipt/debug state rather than the hot tracking feed."} {"ts": "2026-06-12T14:08:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveFusion as the first calibrated Move pose owner and MimirMovePoseStream as the realtime output frame: native Muninn Move evidence samples are associated against USB controller state, calibrated multi-camera orb rays triangulate position into mimir.move_controller_pose.v1, controller buttons/trigger/gyro are carried forward, orientation is explicitly marked imu-unresolved, and BufferSmoke --move-fusion-smoke proves the positive calibrated pose path, the negative no-calibration/no-pose guard, and a CultMesh pose-stream roundtrip."} {"ts": "2026-06-12T14:45:00+01:00", "type": "implementation", "status": "ok", "note": "Added typed Move calibration preflight state as mimir.move_calibration_protocol.v1. BufferSmoke --move-calibration-protocol-smoke writes artifacts/move-calibration/protocol.cc and proves the Starfire/Nightwing protocol requires both Muninn Move evidence streams plus Mimir pose output, includes optional Periwinkle motion/camera witnesses, defines stillness/sweep/validation phases, and names the four required derived outputs before IMU orientation can become authority."} +{"ts": "2026-06-12T15:05:00+01:00", "type": "correction", "status": "ok", "note": "Updated the Move calibration preflight witness from Periwinkle to the USB-connected Quest headset plus left/right controllers. ADB sees serial 1WMHHB68PG1515 but reports unauthorized, so headset-side USB debugging approval is required before Quest pose capture can become live evidence. The protocol treats Quest poses as an optional external VR reference frame around the Move, not as a Mimir fusion owner."} diff --git a/state/map.yaml b/state/map.yaml index 344ea60..2e0ebbe 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -115,9 +115,9 @@ accepted_design: not own calibration or fusion decisions. - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move calibration preflight as typed state: required Muninn Move evidence streams, - optional Periwinkle motion/camera witnesses, stillness/sweep/validation - phases, and the calibration outputs Mimir must derive before orientation - can stop being `orientation:imu-unresolved`. + optional Quest headset/controller pose witnesses over USB, + stillness/sweep/validation phases, and the calibration outputs Mimir must + derive before orientation can stop being `orientation:imu-unresolved`. - Starfire should have Muninn for source-local Move HID control and local sensor publication, but some Starfire audio/video paths still enter Mimir through local Mimir producers or diagnostic process bridges while native From 00c905343043af1452e3247cae11be13bedd6cc0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 15:20:19 +0100 Subject: [PATCH 033/135] Add Quest USB calibration preflight --- docs/implementation-plan.md | 4 + docs/move-calibration-protocol.md | 22 ++- docs/perfect-machine-module-library.md | 3 +- notes/current-system-map.md | 13 +- src/Mimir.BufferSmoke/Program.cs | 173 ++++++++++++++++++ .../Synchronization/MimirModuleLibrary.cs | 3 +- .../Synchronization/MimirQuestUsbPreflight.cs | 64 +++++++ state/evidence.jsonl | 1 + state/map.yaml | 4 + 9 files changed, 278 insertions(+), 9 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 37d35eb..5050620 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -72,6 +72,10 @@ a named invariant that the native runtime cannot protect yet. Quest headset/controller pose witnesses, stillness/sweep/validation phases, and the four derived calibration outputs Mimir must produce before IMU orientation can become authority. +- `MimirQuestUsbPreflight` publishes the typed USB witness receipt for an + authorized Quest. It proves ADB/device access and records the explicit + boundary that Quest headset/controller poses still require a Quest/OpenXR + witness bridge before they can become calibration evidence. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field diff --git a/docs/move-calibration-protocol.md b/docs/move-calibration-protocol.md index bdb8daa..6bc4ad9 100644 --- a/docs/move-calibration-protocol.md +++ b/docs/move-calibration-protocol.md @@ -52,9 +52,18 @@ adb devices -l ``` If it reports `unauthorized`, put on the headset and accept the USB debugging -prompt. The current calibration protocol can still run without Quest, but Quest -poses are the preferred external reference when the headset and both -controllers are sitting around the Move. +prompt. When it reports an authorized Quest, persist the USB witness receipt: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke --output artifacts\move-calibration\quest-usb-preflight.cc +``` + +That writes `mimir.quest_usb_preflight.v1`. It proves USB access, device +identity, and basic headset state only. It does not expose Quest headset or +controller poses; those still require a Quest/OpenXR witness bridge that +publishes pose samples into Mimir's calibration capture. The current calibration +protocol can still run without Quest, but Quest poses are the preferred external +reference when the headset and both controllers are sitting around the Move. ## Data Products @@ -71,8 +80,11 @@ controllers are sitting around the Move. ```powershell dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke --output artifacts\move-calibration\protocol.cc +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke --output artifacts\move-calibration\quest-usb-preflight.cc ``` That writes the typed protocol as `mimir.move_calibration_protocol.v1` into a -CultCache receipt. The real hardware runner should consume this document and -append capture receipts instead of inventing a parallel checklist. +CultCache receipt and writes the Quest USB preflight as +`mimir.quest_usb_preflight.v1` when the headset is authorized. The real hardware +runner should consume these documents and append capture receipts instead of +inventing a parallel checklist. diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index 42e8858..f8a41e4 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, `MimirQuestUsbPreflight` records optional Quest USB witness availability without claiming pose capture, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -71,6 +71,7 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-fusion-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 24b3c29..262ffb7 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -145,8 +145,17 @@ artifacts\move-calibration\protocol.cc` writes left/right controller poses over USB as an external VR reference; and requires Mimir to derive rig calibration, IMU calibration, controller identity mapping, and a calibration receipt before orientation can stop being -`orientation:imu-unresolved`. ADB currently sees the Quest over USB, but it must -be authorized in-headset before Starfire can query it. +`orientation:imu-unresolved`. + +Quest USB preflight proof: `dotnet run --project +.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- +--quest-usb-preflight-smoke --output +artifacts\move-calibration\quest-usb-preflight.cc` writes +`mimir.quest_usb_preflight.v1`. On 2026-06-12, `adb devices -l` reported +authorized Quest 2 device `1WMHHB68PG1515` with product/device `hollywood`. +This proves Starfire can query the headset over USB; it does not provide +Quest headset/controller poses until a Quest/OpenXR witness bridge publishes +those pose samples. Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 7b174e4..1f4b43c 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -127,6 +127,13 @@ if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", Stri .ConfigureAwait(false); } +if (args.Any(arg => string.Equals(arg, "--quest-usb-preflight-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return await RunQuestUsbPreflightSmokeAsync( + ParseStringOption(args, "--output", "artifacts/move-calibration/quest-usb-preflight.cc")) + .ConfigureAwait(false); +} + if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1911,6 +1918,172 @@ static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) : 1; } +static async Task RunQuestUsbPreflightSmokeAsync(string outputPath) +{ + var devices = await RunProcessCaptureAsync("adb", ["devices", "-l"]).ConfigureAwait(false); + if (devices.ExitCode != 0) + { + Console.Error.WriteLine($"quest-usb-preflight-smoke failed: adb devices exited {devices.ExitCode}: {devices.Stderr.Trim()}"); + return 1; + } + + var quest = FindAuthorizedQuestDevice(devices.Stdout); + if (quest is null) + { + Console.Error.WriteLine("quest-usb-preflight-smoke failed: no authorized Quest device found in adb devices -l"); + Console.Error.WriteLine(devices.Stdout.Trim()); + return 1; + } + + var serial = quest.Value.Serial; + var fields = quest.Value.Fields; + var product = fields.GetValueOrDefault("product", ""); + var model = fields.GetValueOrDefault("model", ""); + var device = fields.GetValueOrDefault("device", ""); + var transportId = fields.GetValueOrDefault("transport_id", ""); + var androidRelease = await GetAdbPropertyAsync(serial, "ro.build.version.release").ConfigureAwait(false); + var androidSdk = await GetAdbPropertyAsync(serial, "ro.build.version.sdk").ConfigureAwait(false); + var buildFingerprint = await GetAdbPropertyAsync(serial, "ro.build.fingerprint").ConfigureAwait(false); + var batterySummary = await GetAdbBatterySummaryAsync(serial).ConfigureAwait(false); + var document = MimirQuestUsbPreflight.Create( + serial, + "device", + product, + model, + device, + transportId, + androidRelease, + androidSdk, + buildFingerprint, + batterySummary); + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions + { + PullOnOpen = File.Exists(outputPath) + }).ConfigureAwait(false); + await cache.UpsertAsync( + document, + new CultRecordHandle(new CultRecordKey($"mimir-quest-usb-preflight:{document.Serial}"))) + .ConfigureAwait(false); + await cache.FlushAsync().ConfigureAwait(false); + + Console.WriteLine( + $"quest-usb-preflight-smoke path={outputPath} serial={document.Serial} model={document.Model} product={document.Product} android={document.AndroidRelease} sdk={document.AndroidSdk} pose={document.PoseCaptureStatus}"); + return 0; +} + +static (string Serial, Dictionary Fields)? FindAuthorizedQuestDevice(string adbDevicesOutput) +{ + foreach (var rawLine in adbDevicesOutput.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries)) + { + var tokens = rawLine.Split(' ', StringSplitOptions.RemoveEmptyEntries); + if (tokens.Length < 2 || tokens[0] == "List" || !string.Equals(tokens[1], "device", StringComparison.Ordinal)) + { + continue; + } + + var fields = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var token in tokens.Skip(2)) + { + var split = token.IndexOf(':'); + if (split > 0 && split < token.Length - 1) + { + fields[token[..split]] = token[(split + 1)..]; + } + } + + var model = fields.GetValueOrDefault("model", ""); + var product = fields.GetValueOrDefault("product", ""); + if (model.Contains("Quest", StringComparison.OrdinalIgnoreCase) || + product.Contains("hollywood", StringComparison.OrdinalIgnoreCase)) + { + return (tokens[0], fields); + } + } + + return null; +} + +static async Task GetAdbPropertyAsync(string serial, string property) +{ + var result = await RunProcessCaptureAsync("adb", ["-s", serial, "shell", "getprop", property], 5_000).ConfigureAwait(false); + return result.ExitCode == 0 ? result.Stdout.Trim() : ""; +} + +static async Task GetAdbBatterySummaryAsync(string serial) +{ + var result = await RunProcessCaptureAsync("adb", ["-s", serial, "shell", "dumpsys", "battery"], 5_000).ConfigureAwait(false); + if (result.ExitCode != 0) + { + return ""; + } + + var usefulLines = result.Stdout.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries) + .Select(line => line.Trim()) + .Where(line => + line.StartsWith("AC powered:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("USB powered:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("Wireless powered:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("status:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("level:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("scale:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("voltage:", StringComparison.OrdinalIgnoreCase) || + line.StartsWith("temperature:", StringComparison.OrdinalIgnoreCase)) + .Take(12); + return string.Join(" | ", usefulLines); +} + +static async Task<(int ExitCode, string Stdout, string Stderr)> RunProcessCaptureAsync( + string fileName, + IReadOnlyList arguments, + int timeoutMilliseconds = 10_000) +{ + using var process = new Process(); + process.StartInfo = new ProcessStartInfo(fileName) + { + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + + foreach (var argument in arguments) + { + process.StartInfo.ArgumentList.Add(argument); + } + + try + { + if (!process.Start()) + { + return (-1, "", $"could not start {fileName}"); + } + } + catch (Exception ex) when (ex is InvalidOperationException or System.ComponentModel.Win32Exception) + { + return (-1, "", $"could not start {fileName}: {ex.Message}"); + } + + var stdoutTask = process.StandardOutput.ReadToEndAsync(); + var stderrTask = process.StandardError.ReadToEndAsync(); + var waitTask = process.WaitForExitAsync(); + var completed = await Task.WhenAny(waitTask, Task.Delay(timeoutMilliseconds)).ConfigureAwait(false); + if (completed != waitTask) + { + try + { + process.Kill(entireProcessTree: true); + } + catch (InvalidOperationException) + { + } + + return (-1, await stdoutTask.ConfigureAwait(false), $"timed out after {timeoutMilliseconds} ms"); + } + + return (process.ExitCode, await stdoutTask.ConfigureAwait(false), await stderrTask.ConfigureAwait(false)); +} + static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index 59b86b8..71cf884 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -57,10 +57,11 @@ public static class MimirModuleLibrary "move-controller-fusion", "Mimir.Runtime + Fensalir", "Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses, then publishes them as a realtime CultMesh pose stream for interaction consumers.", - ["mimir.move_calibration_protocol.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], + ["mimir.move_calibration_protocol.v1", "mimir.quest_usb_preflight.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], [ "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke", "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke", "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke" ]), new( diff --git a/src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs b/src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs new file mode 100644 index 0000000..5426862 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs @@ -0,0 +1,64 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[CultDocument("mimir.quest_usb_preflight", "mimir.quest_usb_preflight.v1")] +[MessagePackObject] +public sealed record MimirQuestUsbPreflightDocument( + [property: Key(0)] + [property: CultName] + string PreflightId, + [property: Key(1)] string CapturedAtUtc, + [property: Key(2)] string Serial, + [property: Key(3)] string ConnectionState, + [property: Key(4)] string Product, + [property: Key(5)] string Model, + [property: Key(6)] string Device, + [property: Key(7)] string TransportId, + [property: Key(8)] string AndroidRelease, + [property: Key(9)] string AndroidSdk, + [property: Key(10)] string BuildFingerprint, + [property: Key(11)] string BatterySummary, + [property: Key(12)] string PoseCaptureStatus, + [property: Key(13)] string RequiredNextStep, + [property: Key(14)] string[] OperatorNotes); + +public static class MimirQuestUsbPreflight +{ + public const string AuthorizedNoPoseBridgeStatus = "usb-authorized-no-pose-bridge-yet"; + + public static MimirQuestUsbPreflightDocument Create( + string serial, + string connectionState, + string product, + string model, + string device, + string transportId, + string androidRelease, + string androidSdk, + string buildFingerprint, + string batterySummary, + DateTimeOffset? capturedAt = null) => + new( + PreflightId: $"quest-usb:{serial}", + CapturedAtUtc: (capturedAt ?? DateTimeOffset.UtcNow).ToString("O"), + Serial: serial, + ConnectionState: connectionState, + Product: product, + Model: model, + Device: device, + TransportId: transportId, + AndroidRelease: androidRelease, + AndroidSdk: androidSdk, + BuildFingerprint: buildFingerprint, + BatterySummary: batterySummary, + PoseCaptureStatus: AuthorizedNoPoseBridgeStatus, + RequiredNextStep: "Run or deploy a Quest/OpenXR witness that publishes headset and controller poses into the Mimir calibration capture.", + OperatorNotes: + [ + "ADB authorization proves Starfire can query the Quest over USB; it does not expose tracked headset/controller poses by itself.", + "Do not disturb the headset runtime from this preflight. Pose capture belongs to an explicit Quest/OpenXR witness bridge.", + "Mimir may use Quest poses as optional external validation evidence, but Mimir still owns Move calibration and fusion promotion." + ]); +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 24ed704..e61ad01 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -137,3 +137,4 @@ {"ts": "2026-06-12T14:08:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveFusion as the first calibrated Move pose owner and MimirMovePoseStream as the realtime output frame: native Muninn Move evidence samples are associated against USB controller state, calibrated multi-camera orb rays triangulate position into mimir.move_controller_pose.v1, controller buttons/trigger/gyro are carried forward, orientation is explicitly marked imu-unresolved, and BufferSmoke --move-fusion-smoke proves the positive calibrated pose path, the negative no-calibration/no-pose guard, and a CultMesh pose-stream roundtrip."} {"ts": "2026-06-12T14:45:00+01:00", "type": "implementation", "status": "ok", "note": "Added typed Move calibration preflight state as mimir.move_calibration_protocol.v1. BufferSmoke --move-calibration-protocol-smoke writes artifacts/move-calibration/protocol.cc and proves the Starfire/Nightwing protocol requires both Muninn Move evidence streams plus Mimir pose output, includes optional Periwinkle motion/camera witnesses, defines stillness/sweep/validation phases, and names the four required derived outputs before IMU orientation can become authority."} {"ts": "2026-06-12T15:05:00+01:00", "type": "correction", "status": "ok", "note": "Updated the Move calibration preflight witness from Periwinkle to the USB-connected Quest headset plus left/right controllers. ADB sees serial 1WMHHB68PG1515 but reports unauthorized, so headset-side USB debugging approval is required before Quest pose capture can become live evidence. The protocol treats Quest poses as an optional external VR reference frame around the Move, not as a Mimir fusion owner."} +{"ts":"2026-06-12T15:17:22.2571251+01:00","type":"verification","status":"ok","note":"Quest USB access is now authorized for calibration preflight: adb devices -l reports serial 1WMHHB68PG1515 as Quest_2/hollywood. Added and ran --quest-usb-preflight-smoke, which wrote mimir.quest_usb_preflight.v1 with poseCaptureStatus=usb-authorized-no-pose-bridge-yet. ADB access proves USB/device identity only; headset/controller pose samples still require a Quest/OpenXR witness bridge before Mimir can use Quest as external calibration evidence."} diff --git a/state/map.yaml b/state/map.yaml index 2e0ebbe..3845e20 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -118,6 +118,10 @@ accepted_design: optional Quest headset/controller pose witnesses over USB, stillness/sweep/validation phases, and the calibration outputs Mimir must derive before orientation can stop being `orientation:imu-unresolved`. + - `MimirQuestUsbPreflight` defines the typed Quest USB witness receipt. It + proves ADB authorization/device identity for the optional external VR + reference, but it explicitly does not claim Quest headset/controller pose + capture until a Quest/OpenXR witness bridge publishes those samples. - Starfire should have Muninn for source-local Move HID control and local sensor publication, but some Starfire audio/video paths still enter Mimir through local Mimir producers or diagnostic process bridges while native From 09d0d3ad2e291cdf8fc7237c112902085aebd5ec Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 15:20:46 +0100 Subject: [PATCH 034/135] Refresh Mimir Face maintenance status --- .voidbot/status/void-memory-maintenance.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index c619dc5..38f0325 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-06-12T13:42:48.078Z", + "startedAt": "2026-06-12T14:17:57.178Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" From c1178512c42cf33bf4f0cd6d5486f20d374bb9a0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 16:41:15 +0100 Subject: [PATCH 035/135] Move Quest access ownership to Muninn --- docs/implementation-plan.md | 7 +- docs/move-calibration-protocol.md | 36 ++-- docs/perfect-machine-module-library.md | 3 +- notes/current-system-map.md | 17 +- src/Mimir.BufferSmoke/Program.cs | 173 ------------------ .../Synchronization/MimirModuleLibrary.cs | 3 +- .../Synchronization/MimirQuestUsbPreflight.cs | 64 ------- state/evidence.jsonl | 1 + state/map.yaml | 8 +- 9 files changed, 36 insertions(+), 276 deletions(-) delete mode 100644 src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 5050620..226ec76 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -72,10 +72,9 @@ a named invariant that the native runtime cannot protect yet. Quest headset/controller pose witnesses, stillness/sweep/validation phases, and the four derived calibration outputs Mimir must produce before IMU orientation can become authority. -- `MimirQuestUsbPreflight` publishes the typed USB witness receipt for an - authorized Quest. It proves ADB/device access and records the explicit - boundary that Quest headset/controller poses still require a Quest/OpenXR - witness bridge before they can become calibration evidence. +- Muninn publishes Quest access as `muninn.quest_access.v1` for the USB-attached + Quest. Mimir consumes that access surface and later `muninn.quest_pose_frame.v1` + samples as optional calibration evidence; Mimir does not own ADB/Quest access. - `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field diff --git a/docs/move-calibration-protocol.md b/docs/move-calibration-protocol.md index 6bc4ad9..b817d82 100644 --- a/docs/move-calibration-protocol.md +++ b/docs/move-calibration-protocol.md @@ -42,28 +42,30 @@ lowers to local UDP for OBS compatibility. 7. `validation-pass`: held-out free motion; promote only if triangulated optical residuals and IMU prediction residuals pass. -## Quest USB Preflight +## Quest Access Preflight -The Quest must be authorized for USB debugging before Starfire can query it. -Check it with: +The Quest must be authorized for USB debugging before Starfire Muninn can query +it. Muninn owns the access surface: ```powershell -adb devices -l +muninn serve --store C:\Meta\Odin\state\muninn.telemetry.cc --host starfire --quest-adb --quest-serial 1WMHHB68PG1515 ``` -If it reports `unauthorized`, put on the headset and accept the USB debugging -prompt. When it reports an authorized Quest, persist the USB witness receipt: +When the Quest is authorized, Muninn publishes `muninn.quest_access.v1` and +advertises `muninn::quest-input`, `muninn::quest-poses`, and +`muninn::quest-warped-video-input`. Mimir consumes the access/pose +surfaces as calibration evidence; it does not own Quest USB access. ```powershell -dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke --output artifacts\move-calibration\quest-usb-preflight.cc +muninn quest-access-status --store C:\Meta\Odin\state\muninn.telemetry.cc ``` -That writes `mimir.quest_usb_preflight.v1`. It proves USB access, device -identity, and basic headset state only. It does not expose Quest headset or -controller poses; those still require a Quest/OpenXR witness bridge that -publishes pose samples into Mimir's calibration capture. The current calibration -protocol can still run without Quest, but Quest poses are the preferred external -reference when the headset and both controllers are sitting around the Move. +ADB authorization proves USB access, device identity, and basic headset state +only. It does not expose Quest headset or controller poses; those still require +a Quest/OpenXR witness bridge that publishes pose samples through Muninn. The +current calibration protocol can still run without Quest, but Quest poses are +the preferred external reference when the headset and both controllers are +sitting around the Move. ## Data Products @@ -80,11 +82,9 @@ reference when the headset and both controllers are sitting around the Move. ```powershell dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke --output artifacts\move-calibration\protocol.cc -dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke --output artifacts\move-calibration\quest-usb-preflight.cc ``` That writes the typed protocol as `mimir.move_calibration_protocol.v1` into a -CultCache receipt and writes the Quest USB preflight as -`mimir.quest_usb_preflight.v1` when the headset is authorized. The real hardware -runner should consume these documents and append capture receipts instead of -inventing a parallel checklist. +CultCache receipt. The real hardware runner should consume this document plus +Muninn's `muninn.quest_access.v1` when Quest is available, then append capture +receipts instead of inventing a parallel checklist. diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md index f8a41e4..b1ef408 100644 --- a/docs/perfect-machine-module-library.md +++ b/docs/perfect-machine-module-library.md @@ -14,7 +14,7 @@ job-interview answer for why it belongs in the body of the machine. | Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | | Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | | CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, actuator, program, Move evidence, and Mimir-fused Move controller pose documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs`, `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs` | -| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, `MimirQuestUsbPreflight` records optional Quest USB witness availability without claiming pose capture, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | +| Move controller fusion | `Mimir.Runtime` + native reservoir + Fensalir | Mimir admits Muninn marker candidates plus controller/IMU/button feeds into native `move_evidence` buffers. `MimirMoveCalibrationProtocol` defines the Starfire/Nightwing capture run, Muninn records optional Quest USB witness availability as `muninn.quest_access.v1`, `MimirMoveFusion` publishes calibrated `mimir.move_controller_pose.v1` position/button/gyro poses only when camera witnesses are calibrated, and `MimirMovePoseStream` carries those poses over a realtime CultMesh frame. IMU orientation/prediction remains the next earned 6DoF step before Fensalir treats the stream as VR-grade controller input. | `src/Mimir.Runtime/Synchronization/MimirTrackingObservation.cs`, `src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs`, `src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs`, `src/Mimir.Runtime/Synchronization/MimirMovePoseStream.cs`, `src/Mimir.Runtime/Synchronization/MimirNativeReservoirRuntime.cs`, `native/reservoir/src/lib.rs` | | Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | | Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | | Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | @@ -71,7 +71,6 @@ dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-evidence-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --muninn-move-cultmesh-stream-smoke --native-reservoir .\native\reservoir\target\debug\localcast_reservoir.dll dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke -dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --move-fusion-smoke dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 262ffb7..0aedb80 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -147,15 +147,14 @@ Mimir to derive rig calibration, IMU calibration, controller identity mapping, and a calibration receipt before orientation can stop being `orientation:imu-unresolved`. -Quest USB preflight proof: `dotnet run --project -.\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- ---quest-usb-preflight-smoke --output -artifacts\move-calibration\quest-usb-preflight.cc` writes -`mimir.quest_usb_preflight.v1`. On 2026-06-12, `adb devices -l` reported -authorized Quest 2 device `1WMHHB68PG1515` with product/device `hollywood`. -This proves Starfire can query the headset over USB; it does not provide -Quest headset/controller poses until a Quest/OpenXR witness bridge publishes -those pose samples. +Quest USB access belongs to Muninn, not Mimir. Starfire Muninn runs with +`--quest-adb --quest-serial 1WMHHB68PG1515` and publishes +`muninn.quest_access.v1`, including stream ids for Quest input, Quest poses, and +warp-corrected video input from Brokkr/Unity. On 2026-06-12, `adb devices -l` +reported authorized Quest 2 device `1WMHHB68PG1515` with product/device +`hollywood`. This proves Starfire can query the headset over USB; it does not +provide Quest headset/controller poses until a Quest/OpenXR witness bridge +publishes those pose samples through Muninn. Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 1f4b43c..7b174e4 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -127,13 +127,6 @@ if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", Stri .ConfigureAwait(false); } -if (args.Any(arg => string.Equals(arg, "--quest-usb-preflight-smoke", StringComparison.OrdinalIgnoreCase))) -{ - return await RunQuestUsbPreflightSmokeAsync( - ParseStringOption(args, "--output", "artifacts/move-calibration/quest-usb-preflight.cc")) - .ConfigureAwait(false); -} - if (args.Any(arg => string.Equals(arg, "--import-obs-program-scene", StringComparison.OrdinalIgnoreCase))) { return await ImportObsProgramSceneAsync( @@ -1918,172 +1911,6 @@ static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) : 1; } -static async Task RunQuestUsbPreflightSmokeAsync(string outputPath) -{ - var devices = await RunProcessCaptureAsync("adb", ["devices", "-l"]).ConfigureAwait(false); - if (devices.ExitCode != 0) - { - Console.Error.WriteLine($"quest-usb-preflight-smoke failed: adb devices exited {devices.ExitCode}: {devices.Stderr.Trim()}"); - return 1; - } - - var quest = FindAuthorizedQuestDevice(devices.Stdout); - if (quest is null) - { - Console.Error.WriteLine("quest-usb-preflight-smoke failed: no authorized Quest device found in adb devices -l"); - Console.Error.WriteLine(devices.Stdout.Trim()); - return 1; - } - - var serial = quest.Value.Serial; - var fields = quest.Value.Fields; - var product = fields.GetValueOrDefault("product", ""); - var model = fields.GetValueOrDefault("model", ""); - var device = fields.GetValueOrDefault("device", ""); - var transportId = fields.GetValueOrDefault("transport_id", ""); - var androidRelease = await GetAdbPropertyAsync(serial, "ro.build.version.release").ConfigureAwait(false); - var androidSdk = await GetAdbPropertyAsync(serial, "ro.build.version.sdk").ConfigureAwait(false); - var buildFingerprint = await GetAdbPropertyAsync(serial, "ro.build.fingerprint").ConfigureAwait(false); - var batterySummary = await GetAdbBatterySummaryAsync(serial).ConfigureAwait(false); - var document = MimirQuestUsbPreflight.Create( - serial, - "device", - product, - model, - device, - transportId, - androidRelease, - androidSdk, - buildFingerprint, - batterySummary); - - Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); - using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions - { - PullOnOpen = File.Exists(outputPath) - }).ConfigureAwait(false); - await cache.UpsertAsync( - document, - new CultRecordHandle(new CultRecordKey($"mimir-quest-usb-preflight:{document.Serial}"))) - .ConfigureAwait(false); - await cache.FlushAsync().ConfigureAwait(false); - - Console.WriteLine( - $"quest-usb-preflight-smoke path={outputPath} serial={document.Serial} model={document.Model} product={document.Product} android={document.AndroidRelease} sdk={document.AndroidSdk} pose={document.PoseCaptureStatus}"); - return 0; -} - -static (string Serial, Dictionary Fields)? FindAuthorizedQuestDevice(string adbDevicesOutput) -{ - foreach (var rawLine in adbDevicesOutput.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries)) - { - var tokens = rawLine.Split(' ', StringSplitOptions.RemoveEmptyEntries); - if (tokens.Length < 2 || tokens[0] == "List" || !string.Equals(tokens[1], "device", StringComparison.Ordinal)) - { - continue; - } - - var fields = new Dictionary(StringComparer.OrdinalIgnoreCase); - foreach (var token in tokens.Skip(2)) - { - var split = token.IndexOf(':'); - if (split > 0 && split < token.Length - 1) - { - fields[token[..split]] = token[(split + 1)..]; - } - } - - var model = fields.GetValueOrDefault("model", ""); - var product = fields.GetValueOrDefault("product", ""); - if (model.Contains("Quest", StringComparison.OrdinalIgnoreCase) || - product.Contains("hollywood", StringComparison.OrdinalIgnoreCase)) - { - return (tokens[0], fields); - } - } - - return null; -} - -static async Task GetAdbPropertyAsync(string serial, string property) -{ - var result = await RunProcessCaptureAsync("adb", ["-s", serial, "shell", "getprop", property], 5_000).ConfigureAwait(false); - return result.ExitCode == 0 ? result.Stdout.Trim() : ""; -} - -static async Task GetAdbBatterySummaryAsync(string serial) -{ - var result = await RunProcessCaptureAsync("adb", ["-s", serial, "shell", "dumpsys", "battery"], 5_000).ConfigureAwait(false); - if (result.ExitCode != 0) - { - return ""; - } - - var usefulLines = result.Stdout.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries) - .Select(line => line.Trim()) - .Where(line => - line.StartsWith("AC powered:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("USB powered:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("Wireless powered:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("status:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("level:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("scale:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("voltage:", StringComparison.OrdinalIgnoreCase) || - line.StartsWith("temperature:", StringComparison.OrdinalIgnoreCase)) - .Take(12); - return string.Join(" | ", usefulLines); -} - -static async Task<(int ExitCode, string Stdout, string Stderr)> RunProcessCaptureAsync( - string fileName, - IReadOnlyList arguments, - int timeoutMilliseconds = 10_000) -{ - using var process = new Process(); - process.StartInfo = new ProcessStartInfo(fileName) - { - RedirectStandardOutput = true, - RedirectStandardError = true, - UseShellExecute = false - }; - - foreach (var argument in arguments) - { - process.StartInfo.ArgumentList.Add(argument); - } - - try - { - if (!process.Start()) - { - return (-1, "", $"could not start {fileName}"); - } - } - catch (Exception ex) when (ex is InvalidOperationException or System.ComponentModel.Win32Exception) - { - return (-1, "", $"could not start {fileName}: {ex.Message}"); - } - - var stdoutTask = process.StandardOutput.ReadToEndAsync(); - var stderrTask = process.StandardError.ReadToEndAsync(); - var waitTask = process.WaitForExitAsync(); - var completed = await Task.WhenAny(waitTask, Task.Delay(timeoutMilliseconds)).ConfigureAwait(false); - if (completed != waitTask) - { - try - { - process.Kill(entireProcessTree: true); - } - catch (InvalidOperationException) - { - } - - return (-1, await stdoutTask.ConfigureAwait(false), $"timed out after {timeoutMilliseconds} ms"); - } - - return (process.ExitCode, await stdoutTask.ConfigureAwait(false), await stderrTask.ConfigureAwait(false)); -} - static MoveTrackingSmokeContext BuildMoveTrackingSmokeHub() { var starfireDescriptor = new MimirStreamDescriptor( diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs index 71cf884..7e68eb9 100644 --- a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -57,11 +57,10 @@ public static class MimirModuleLibrary "move-controller-fusion", "Mimir.Runtime + Fensalir", "Mimir fuses Muninn marker candidates, controller IMU/button state, calibration, and timing into resolved wand poses, then publishes them as a realtime CultMesh pose stream for interaction consumers.", - ["mimir.move_calibration_protocol.v1", "mimir.quest_usb_preflight.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], + ["mimir.move_calibration_protocol.v1", "mimir.move_controller_pose.v1", "mimir.move_controller_pose_stream_frame.v1", "muninn.quest_access.v1", "muninn.quest_pose_frame.v1", "muninn.move_marker_candidate.v1", "muninn.move_controller_state.v1"], [ "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-tracking-contract-smoke", "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-calibration-protocol-smoke", - "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --quest-usb-preflight-smoke", "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --move-fusion-smoke" ]), new( diff --git a/src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs b/src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs deleted file mode 100644 index 5426862..0000000 --- a/src/Mimir.Runtime/Synchronization/MimirQuestUsbPreflight.cs +++ /dev/null @@ -1,64 +0,0 @@ -using GameCult.Caching; -using MessagePack; - -namespace Mimir.Runtime.Synchronization; - -[CultDocument("mimir.quest_usb_preflight", "mimir.quest_usb_preflight.v1")] -[MessagePackObject] -public sealed record MimirQuestUsbPreflightDocument( - [property: Key(0)] - [property: CultName] - string PreflightId, - [property: Key(1)] string CapturedAtUtc, - [property: Key(2)] string Serial, - [property: Key(3)] string ConnectionState, - [property: Key(4)] string Product, - [property: Key(5)] string Model, - [property: Key(6)] string Device, - [property: Key(7)] string TransportId, - [property: Key(8)] string AndroidRelease, - [property: Key(9)] string AndroidSdk, - [property: Key(10)] string BuildFingerprint, - [property: Key(11)] string BatterySummary, - [property: Key(12)] string PoseCaptureStatus, - [property: Key(13)] string RequiredNextStep, - [property: Key(14)] string[] OperatorNotes); - -public static class MimirQuestUsbPreflight -{ - public const string AuthorizedNoPoseBridgeStatus = "usb-authorized-no-pose-bridge-yet"; - - public static MimirQuestUsbPreflightDocument Create( - string serial, - string connectionState, - string product, - string model, - string device, - string transportId, - string androidRelease, - string androidSdk, - string buildFingerprint, - string batterySummary, - DateTimeOffset? capturedAt = null) => - new( - PreflightId: $"quest-usb:{serial}", - CapturedAtUtc: (capturedAt ?? DateTimeOffset.UtcNow).ToString("O"), - Serial: serial, - ConnectionState: connectionState, - Product: product, - Model: model, - Device: device, - TransportId: transportId, - AndroidRelease: androidRelease, - AndroidSdk: androidSdk, - BuildFingerprint: buildFingerprint, - BatterySummary: batterySummary, - PoseCaptureStatus: AuthorizedNoPoseBridgeStatus, - RequiredNextStep: "Run or deploy a Quest/OpenXR witness that publishes headset and controller poses into the Mimir calibration capture.", - OperatorNotes: - [ - "ADB authorization proves Starfire can query the Quest over USB; it does not expose tracked headset/controller poses by itself.", - "Do not disturb the headset runtime from this preflight. Pose capture belongs to an explicit Quest/OpenXR witness bridge.", - "Mimir may use Quest poses as optional external validation evidence, but Mimir still owns Move calibration and fusion promotion." - ]); -} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index e61ad01..fc13f1c 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -138,3 +138,4 @@ {"ts": "2026-06-12T14:45:00+01:00", "type": "implementation", "status": "ok", "note": "Added typed Move calibration preflight state as mimir.move_calibration_protocol.v1. BufferSmoke --move-calibration-protocol-smoke writes artifacts/move-calibration/protocol.cc and proves the Starfire/Nightwing protocol requires both Muninn Move evidence streams plus Mimir pose output, includes optional Periwinkle motion/camera witnesses, defines stillness/sweep/validation phases, and names the four required derived outputs before IMU orientation can become authority."} {"ts": "2026-06-12T15:05:00+01:00", "type": "correction", "status": "ok", "note": "Updated the Move calibration preflight witness from Periwinkle to the USB-connected Quest headset plus left/right controllers. ADB sees serial 1WMHHB68PG1515 but reports unauthorized, so headset-side USB debugging approval is required before Quest pose capture can become live evidence. The protocol treats Quest poses as an optional external VR reference frame around the Move, not as a Mimir fusion owner."} {"ts":"2026-06-12T15:17:22.2571251+01:00","type":"verification","status":"ok","note":"Quest USB access is now authorized for calibration preflight: adb devices -l reports serial 1WMHHB68PG1515 as Quest_2/hollywood. Added and ran --quest-usb-preflight-smoke, which wrote mimir.quest_usb_preflight.v1 with poseCaptureStatus=usb-authorized-no-pose-bridge-yet. ADB access proves USB/device identity only; headset/controller pose samples still require a Quest/OpenXR witness bridge before Mimir can use Quest as external calibration evidence."} +{"ts":"2026-06-12T16:05:00+01:00","type":"correction","status":"ok","note":"Moved Quest access ownership out of Mimir and under Odin/Muninn. Mimir's local mimir.quest_usb_preflight path was deleted; Mimir now treats muninn.quest_access.v1 and later muninn.quest_pose_frame.v1 as input evidence. Brokkr owns Unity editor routing for Quest input/pose consumption and warp-corrected Unity play-mode video toward Muninn's Quest video input surface."} diff --git a/state/map.yaml b/state/map.yaml index 3845e20..38460ce 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -118,10 +118,10 @@ accepted_design: optional Quest headset/controller pose witnesses over USB, stillness/sweep/validation phases, and the calibration outputs Mimir must derive before orientation can stop being `orientation:imu-unresolved`. - - `MimirQuestUsbPreflight` defines the typed Quest USB witness receipt. It - proves ADB authorization/device identity for the optional external VR - reference, but it explicitly does not claim Quest headset/controller pose - capture until a Quest/OpenXR witness bridge publishes those samples. + - Muninn owns Quest USB access as `muninn.quest_access.v1`. It proves ADB + authorization/device identity for the optional external VR reference and + advertises Quest input, pose, and warped-video input stream ids. Mimir + consumes those surfaces for calibration; it does not publish Quest access. - Starfire should have Muninn for source-local Move HID control and local sensor publication, but some Starfire audio/video paths still enter Mimir through local Mimir producers or diagnostic process bridges while native From 21b30c6dd43f40ebcc6403e8f67be3e018955a2f Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 12 Jun 2026 16:42:04 +0100 Subject: [PATCH 036/135] Update Mimir Face maintenance status --- .voidbot/status/void-memory-maintenance.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index 38f0325..79a1d2e 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-06-12T14:17:57.178Z", + "startedAt": "2026-06-12T15:21:56.010Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" From 3ec2b0178803cc6ef586f78e4e6080f6741a3781 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 14 Jun 2026 14:30:34 +0100 Subject: [PATCH 037/135] Refresh Mimir Face state --- .voidbot/state/mimir.cc | Bin 65712 -> 70102 bytes .voidbot/status/void-memory-maintenance.json | 2 +- .voidbot/voice/identity.json | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index d513df63c212d5d947f354a5417cf02f83845215..96cf306134f87bf7a0e4c58f3a44dc47e14f6fa5 100644 GIT binary patch delta 8212 zcmb_hZLA#CdF~|!4F0OQ7i{AMkCAPtxwD>`ot>GzCD`sqz!(by4yb7pI5Tr*ckt}Y zdS>=sJ8H6gNDX!jhMZJ_RFJfQ&4-*?x&4uXVnrlM6j7xDMFrDJC8-enNE&EL(kfNe z=RGsCAD4^qk4kWT@9fOZIq&LVtCo9Y31dYR$j45KYjnn%xm)wyPc<5n8Xv`Ou5J7{8s!7yLt96zcVFX*|=qX z>#7I0iG6D~X?mlsTMf-H>J8m#*6YpV8E)U1f7{RuOVun@ukX|K8N-?}?OLN@iC4_G zCsOgdS5NiN9-Z`vF!2?Q& zTaq+vg>g-}88@bLVa9RwXZJjoDR=j>T**4gey+@gd0S~EiLZ2%Ebpbf_H3`~vz+@o z*M0aJ-&`>@8X?QHWZ+hB|HOIPt3um;`NUs}_ja$H_qJcRbpE27E?P2iR2;efQ2+Fv zuoI?A%9)?Z<$Jv}ML3i+NjeJ4ls22C{ji8`IfNS#)miS#dt=Uh{OV%=K1&gZ9`+cd zcrc+2Dzi!G@oZYj5=5f7#M~@TnU^bJobz<{e7q`5#E)P9y?)kaX_v?Hi-V9ye(h;5 zVqxdZ$$dO-^%Re#X;{7(4~HG*9U8ux#lCWT68dAe1vmIAzr7jF1DFwwdYvuwUhK#9%Nq;)4Q+TcHfgJ56)bDcjC@8^}45< zuA!QKy{VdpWvi}b8>;TP0oOFE5%{in?}cfx=wrpYXiv1EQHZT;zpxjg!an@(iCP6x0(Gq;E5%u88*6^gin94h;HH%Htk zWo49r&Mt)sUp}Mc{js+zAOkElQ|8(we!Vu(AhEhQ7nD3u0zBM1SiR~Z`1H*9{!oP( zQWVDg_a#jJkZ=Nk%!aZ!Wl@6arWGFX7DWM^6h*9?aqKA06Z{D(>G0Y!2YMXT1ayYt z)ShclbjchoTb3&uzV$3?C-i4YFZKBL0o3>I1N6q2BTAC40&M0s45$q*dV!yU@orol_TP;cJQ|LO?5okml0+@`6T+{2H$W~)upaa7-8wxzqK z>o@(U_J^^5A_eiIYDS-}GtIZmKy{ezsixit)FyK^wc*-2(_F_2d~1y9D2jXVcbh|Eo`rWC^baS6(&Y1YmsM&^jj+Hro$=;LeS3@qxpB9j!6_th?^b(;x zyhbdIHco_MYqax1`B|eb(&%d!{@df+X5=y=VXD`G{rtZC!}r zSw#7>oee|5Bd6fdiwBnsr`E7EsevzbHcTv^|5@kVsrf6ubng=J>V2L5Gkbe(6nYi) zf;>y&1fGB+(0nh<4*^<|dqKSTJ&cHrDdS+zm@7Rf2>^xsF>+1Fm^q34KNJ?rJNJ6fm2rMf! zQQUN7%MvnI;=qy1#Dhn!*>sNhUN7D}e&qsCT8>RHUo{F;x^~Np8=yRO@%1n7|I>#G zk)LDa0497n_~_*>?JAB0^4-7=(|`J?$4mpm{n zP99y=|IMd*k0;H9&idjc$jAUd9sA+^h@%Aeu>?s&7g5Z;p z5aI#*{2)xTTs(U8<6`5kwM#3Mi3<*Qw#xgncGw*lrXj&6MDPQ=h;O`p`Sk^&q*mcG zN^1%ofwFmLMT5`11Y$tSD*dkXGxvz~N7ikr+>1I|fm9P80gVRN1J81}>NX8WH5(37 zS<`8#T(fMm!I%>;8`LMEv7+EBd&babG(-IIKQ1f5QeLU)j!^<*jrheeA{1@(qXise z8Fg^XS4L^aYBay1xI$#Cav>RZdl5z*jy(cfgADveTrhUIGWt$x5I= zKgcXaN~ll@5&{eX{t;SRfI{)xqdP95+Xnztr1(l?S1>HQCYRvLebIWbwfAhG><&QL zg;^lKOlvJVFBD-^xNSIF2C+s9(*g~PA058(a>-G&m^qFnZ6Qfb+)rriS7?+Zx;P?< z2-^68-fh&d7fcUnAUh~uk6ps%$Yw&4#h5bGp2Z5NXUGq|y!{v5CtthxlLt=BUIed? z=)b)EhvL)=pV)LR)}X?8iD+ZStj}m>t%+W+qQB8=x4gJneC~+VwWUW>djY3_)_E<~V! zWckK^*<}Mt=n4`aYAc&27NK$W$;ZZrDt+9!Z`_hsma=?O0zGzlz@~VFmRVa!QBLeG(;Ci-9O}@!_cx16N}nHlLrB&_80A^Y%`@KL}L(_^z?a6g300PJO( z3br3~0koqr(7*N4u|K@=75VO^cP$!pubxD@BGsFx1OI^(Z8#hF;vZZ zF@C8?Wzg0Y0+=HL84Q4Iz2E@y1kD!w0q6(CU9{m{G$H^9d{yYvkf(y_o$@{CGhvou zUzKDC7ttTm7gKr}4|;$?fjF)t{QsM%$hf5O5Y(Q8Bz%dTXdZK&BrTv39@m#5vJRsR>Qd5HC?w;Qw ze^Jn_pFsYLRt|m>P$=Fq(&aJ5hZBY?%*Q4Wqm@;nLtOR7<}011-8Ad2YIwSazQztP zYvAaKG;RiluQkm;y!(QNjziOC^aiysbxYj)*ks9q!;7TrH;C{5fc5{pYWU&R%IOSM zCz)P~yhJvvFm?5C`^Y18o`on^H4WI0^pzjhwwKVSK^_=S5+RV#mt!qi+yysg#E8@! zj8kn<l(vynVUj0R3}nn8>}V+cR-z@E#2Mg(-&?O*q$MX2Z8 z`h2QQvGG$=W`fxvk^16|zpRwPov2BmW-!H2SD}7qdeb?&$8Dod&_GjLNowT|ac%yV z7qeyaZ~ti5c@vM%>;G_U*{xQ?ZdiJQ>Gg(p9K_M$`Teqke~P%|Z$keWcA$q|saKAy z27ew$Pnq|!6Qvg#DC!@HcYnR6?;`mvrHvsejc8F$Uf7K=9U4OjMy=!^D(ZQ(pg5uo zIVi34WOS}t4s^4`3n4~yYcWDq$49(Nt3j$b`T7;h$h}}tLr}yZ_v6ckaN;xp-8HB& zIZora6W?0SUrTxWu*qvRH6B8xpX}zm3uTHEwXjxhh!CLAsz8#lkYiB^_Q4Fa7`2j} z^cN~|MMuCc&0aCOL%&zeK}XHmJMX_Fv^Umo!u-c}1J_fbFW@BB#H___V$x#-foV4D zfo5CftOawGMgwOFHM5}?kg3zGWq7S!xC^qqXJ8#@bXetD`p&aX?`7~t#hRix%;=umXbSQN zG+>0%VaT^17Op(B@KZJi)RYk~%U9Y9gq$dfI^)U)3H7;kyYAluN1oGggy{4G8U4U; zyJDvC%pzyv`G?kR1f^7?zyS%2C9 zIl+7!rf`2ytp~-HA8zR5WCVkr2%dpLMN`%uT;GVHr1KEDr%|0Lodod&{6e+$O6_UB zT2=NJDAyo!GLU3I%QKLykgTivnF3#hxi5`(r1TSzT(SzptceF-nov_xu!+e!9?~lt&?^V|^N~$^*Y*+$OE{S$%FHh$_am zkR`m?<31gZ4$dHUezf83;H=X_HG(N7vyi^2r6P&Ir%a~og0e*|Z_4PfR)!1^av91O zUs9IvI3lVJYd9a18^>d$(DZPHVhR!Md$bo4pZ)hKvFGMxn>3p{Zo_A)CElbhO2?CWNN4pRxQr^&1Q%8TGBqkQAyC?yn6h&x>}KA fH0HPb_n%xymVIcR#J{e%bk*S4z^U1F>oflcx|_`| delta 5389 zcmbVQZLAzs8D{%IzigY*OT|*@6mF@=-M#yL_g+k({1K4KrM)SJK;X>G+1=ASJ2Ra* zbC+wtr9lCu4XMX+0Zke-fJh)TS3!&^miPl=AVCwfMuD0bLZTKUsiXNVx5Qe~oAo~V=x)nZ8a=d4$6Y`Jv)i)#;z z&Rx7VH?*cURjf^v8WYuGVXRtH@2nmgEM6#T*JirYaXr+C!)-u$}l zyO*o`ci6oHJ48oBmWVCkOKgi1FwdK{V##KO-?C!iVblr}+Y=7PZSJrn=9X-8D++?n zU+#V+wwxfsp=qwwX0gS*h%(cf`U6{sx#5S7Rx4t@9F6!)FwLzvouhC zyu9rN&NRo7b-0Be?@f42fgk_IO>N<{Eq(_Dmu@p^b{W-sSY07~9wS=}fpaSiknx@+ z150!|+!ai6Z+3*9Y7c^BiA2$aWKzr^j3B|`^odX(8%dCe6<<5yl~1C2grDQj-{7~f zbW4}ViWw4(**X!H9Rh2SnVPjx9G5O-bbs5Om0;pr^0Ecd8VI2^>D>x|DVz z@k0m0v_Rq`Rg#{WrH5pjNfeB-m3P`0wn9LpX)xi)L<7s_C{7r}5=j*PN=%}e0E2q$ zheN%4Cb2D#5G%t4p#J&#XVt#RlKRKw)#~J5mwu*Po+#EQO65XrtfG#b+Ny5b zv2Ol`YquoVmp7XTFV!0~>c(==+my6|kzF5nb&m#FP@#ewuMqJL|J$7suy*U06+nHRwtk%x48rv8~`}zSVp)EwV)DLP@U;nBA6#^fFAQeL_#rk z?FP}bK`8OhXB`X?UQLjQUr6BcoaTS;7i^`tIJznMbrf`|N;X&Ln|FwID-1mbrc2a(I8JY>I>uoem(wGzS&xT{ajP&mFmZX0E4 zvCv{62p+}irTZ>dqrbgU{p#58+T!?BaeSgOjy_r_mrH8zp`kTrj;Ss0U8-I@v`*Cz zY?^=L{(Vc-|`rNrIQpU(Y9esF{Iytwjas27+1U=dAw$Vw!I^ESR zGqVG{87QX8-UNO}w5Y4j!`G<0zO(T>oCt^4(uzrbi^uBp!!32qgPXqg%|2O3=b)mI zZrP~1ZmvdrwXQSS3E~j_H{~knJPHd+Qc41};Dt?8{r%q0sV5%THk9mw3mEkNy@}vb zop@kqlo>SSNi0B4U=LW*rRH~tSjMRggONnz5yc+eusqXib=CgMhsSEAlH=Iryi>ND z`RaJl$=j?lo_C#Ut;*}os$DIs6O$JL5nD<4N)xr>M6F(^l&S#4`K!+uQhVPbJbdE8 zGZb>XP_0+BCJ#S!XJ1RF9>}!{2{qqq_d(<@1$a z(}pi-vyhh4t`zwWJ%Mha84j5w zMyrD-FCVVf?FuWF-Mm|J>-j1#)$lRm3rk>mixB4@!y-_9HD*CUX)49P|R7ogc!4|Nazi*Ah)939w}uS z7P_Y-rwC%3R3X0+X{A6z^<$ogFLI`JD48j~g^E8TV)!T^qX!v+TzIk3UUVBWP1svY zZPqGvqu_V6;n_*o`nwT`X*Vj69y_Ke2@}uLIND~-qI?Wl z@H7<2GFaNZTWCRPWjeu(I`-iCOZR>S0;N9w@Pa7-12zL0qV>_w#F*szCuTZZYAGzdHmKw^@uWPNW!b2Mt7Qa?5!{2RIhhM#HKK{eC zi*i3u;pq+Yzx(6tDs}M(*Qxs3L%rKmR zEyh|53aPymnU=_-zMy`2@1|R_^O|d03b_GTG!!WBaPeLMHb#ct`f+V{_9T&rU7>G3 zKn1}#5)e6vfj027HzBAtbrcHS5Y?b8V=0v|)E4AS?_qjvQ*g}XGsamiv=~rGDI9_1 zQF2p(8C-7)PeGcv`u!`HUh-*ri(&nsuhc%-S8C`hJLdiOe!5bv|Ie3n?WeU|>1_1d zK&Jh;6*m)b3>FjP*AX4A9C$)B*<5w2c3u`Q&Ort@O@_%Ov^dzRvdn*7us% zZEr=~^lj7}dX^raG#0LfuI5d!8#hY>gN;bLZd~RrpIvC%vmI0XXp&@wZf`nIhU6Hc zZb)*Culnwhx*cPQOiYF_aI>v8kVmeoj{(?sdUK$)YlcTQe(QT{Q*~T?o{voM<)!QA{8*03B2Tt1mbuT2mi5$hE0NoM?|c(xbkk`xtnqUxxJ0 zf!mcIHxU6;AMZ>A@1n~!g~JCzbQkSTanq%MZeJ~Q4jBY!7YLp_1U!(3TNJ%QKofJ( zA>bnEwFRIQJ1oNOlrCYQ0WKx_AySw$^fW0*kS+o(VMG`sP$l%AJ$tDCn8+!ebV;Hp z5WabZMACv6M7kZ*HU`CJr!3K4(1HYKnW0@=9-~*guygQgX{zROi*uIx==D{@jj?*2 zHF+`LWH!%NS%u}tYn8Elz2uC+;JS{(O0c-2>Z6UT7MB`jweRrx^j#f1^aXX>>1{9U zUy?hzWbVj{-1wda^k**YpFgy-PX;^8cHM* zleH4OLNmzo1YVa>7#Rcu@T@R*a9wU_b#bax#QTnk@>ro%n!9>ME_h+r%G}OnXIODa RCLH$K?42jBYl!&f{{qpX#0>xd diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index 79a1d2e..4a54459 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-06-12T15:21:56.010Z", + "startedAt": "2026-06-14T13:08:31.028Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index f2f0d6d..5aeb5ce 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-06-12T11:48:43.562Z", + "createdOrRefreshedAt": "2026-06-14T09:19:31.031Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From c23b3927876e065d8c794ebcab4661dd13a4a654 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 14 Jun 2026 14:30:54 +0100 Subject: [PATCH 038/135] Mirror Muninn Move source path --- src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index 7031514..45f527b 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -37,7 +37,8 @@ public sealed record MuninnMoveControllerStateDocument( [property: Key(8)] float TriggerValue, [property: Key(9)] string[] Buttons, [property: Key(10)] float Battery01, - [property: Key(11)] string ObservedAt); + [property: Key(11)] string ObservedAt, + [property: Key(12)] string SourcePath = ""); [MessagePackObject] public sealed record MimirMuninnMoveEvidenceStreamFrame( From bff68e5b5897afc31de8a57ba0b38bee2afe7819 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 14 Jun 2026 14:42:30 +0100 Subject: [PATCH 039/135] Add repeated PS Move read probe --- src/Mimir.PsMoveProbe/Program.cs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/Mimir.PsMoveProbe/Program.cs b/src/Mimir.PsMoveProbe/Program.cs index 1bd4f30..82bd45f 100644 --- a/src/Mimir.PsMoveProbe/Program.cs +++ b/src/Mimir.PsMoveProbe/Program.cs @@ -7,6 +7,8 @@ const ushort SonyVid = 0x054C; const ushort PsMovePid = 0x03D5; var readReports = args.Any(arg => string.Equals(arg, "--read", StringComparison.OrdinalIgnoreCase)); +var readCount = TryGetIntOption(args, "--read-count", 1); +var readTimeoutMs = TryGetIntOption(args, "--read-timeout-ms", 350); var pairHost = TryGetOption(args, "--pair-host"); var rgb = TryGetOption(args, "--rgb"); var pulseMs = TryGetIntOption(args, "--pulse-ms", 0); @@ -124,8 +126,11 @@ foreach (var device in devices) if (readReports) { - var result = await WindowsHid.TryReadInputReportAsync(device, TimeSpan.FromMilliseconds(350)); - Console.WriteLine($" read={result}"); + for (var readIndex = 0; readIndex < Math.Max(1, readCount); readIndex++) + { + var result = await WindowsHid.TryReadInputReportAsync(device, TimeSpan.FromMilliseconds(Math.Max(1, readTimeoutMs))); + Console.WriteLine($" read[{readIndex}]={result}"); + } } if (!string.IsNullOrWhiteSpace(rgb)) From 3ebf7cc519e9a38b6ac3607bbe6e0597c42d5517 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 14 Jun 2026 19:40:03 +0100 Subject: [PATCH 040/135] Make Starfire Move light helper hotplug aware --- scripts/start-starfire-move-light.ps1 | 9 +++ src/Mimir.PsMoveProbe/Program.cs | 96 ++++++++++++++++++++++++++- 2 files changed, 103 insertions(+), 2 deletions(-) diff --git a/scripts/start-starfire-move-light.ps1 b/scripts/start-starfire-move-light.ps1 index 55e83be..7ee4105 100644 --- a/scripts/start-starfire-move-light.ps1 +++ b/scripts/start-starfire-move-light.ps1 @@ -3,6 +3,7 @@ param( [int]$RefreshMs = 250, [double]$HoldSeconds = 0, [switch]$TurnOffOnExit, + [switch]$KeepExisting, [switch]$DryRun ) @@ -19,8 +20,16 @@ if (-not (Test-Path $probe)) { dotnet build (Join-Path $repo "src\Mimir.PsMoveProbe\Mimir.PsMoveProbe.csproj") -c Release -p:UseSharedCompilation=false } +if (-not $KeepExisting) { + Get-Process -Name "Mimir.PsMoveProbe" -ErrorAction SilentlyContinue | + ForEach-Object { + Stop-Process -Id $_.Id -Force + } +} + $args = @( "--rgb", $Rgb, + "--breathe-led", "--hold-seconds", "$HoldSeconds", "--refresh-ms", "$RefreshMs", "--event-log", $eventLog diff --git a/src/Mimir.PsMoveProbe/Program.cs b/src/Mimir.PsMoveProbe/Program.cs index 82bd45f..fce452b 100644 --- a/src/Mimir.PsMoveProbe/Program.cs +++ b/src/Mimir.PsMoveProbe/Program.cs @@ -17,6 +17,9 @@ var intervalMs = TryGetIntOption(args, "--interval-ms", 500); var holdSeconds = TryGetDoubleOption(args, "--hold-seconds", -1.0); var refreshMs = TryGetIntOption(args, "--refresh-ms", 250); var eventLog = TryGetOption(args, "--event-log"); +var breatheLed = args.Any(arg => + string.Equals(arg, "--breathe-led", StringComparison.OrdinalIgnoreCase) || + string.Equals(arg, "--breath-led", StringComparison.OrdinalIgnoreCase)); var pairedChirpTrain = args.Any(arg => string.Equals(arg, "--paired-chirp-train", StringComparison.OrdinalIgnoreCase)); var outputAudio = TryGetOption(args, "--output-audio"); var eventCount = TryGetIntOption(args, "--event-count", pulseCount); @@ -52,8 +55,29 @@ Console.WriteLine(); if (devices.Count == 0) { - Console.WriteLine("No PS Move HID collections are visible. Check the USB cable and Windows Device Manager."); - return 1; + if (!breatheLed) + { + Console.WriteLine("No PS Move HID collections are visible. Check the USB cable and Windows Device Manager."); + return 1; + } +} + +if (breatheLed) +{ + if (string.IsNullOrWhiteSpace(rgb)) + { + Console.WriteLine("--breathe-led requires --rgb #rrggbb or --rgb r,g,b."); + return 1; + } + + var result = await WindowsHid.TryBreatheLedAsync( + rgb, + TimeSpan.FromSeconds(Math.Max(0.0, holdSeconds)), + TimeSpan.FromMilliseconds(Math.Max(25, refreshMs)), + eventLog, + args.Any(arg => string.Equals(arg, "--turn-off-on-exit", StringComparison.OrdinalIgnoreCase))).ConfigureAwait(false); + Console.WriteLine($"breatheLed={result}"); + return result.StartsWith("ok", StringComparison.OrdinalIgnoreCase) ? 0 : 1; } if (pairedChirpTrain) @@ -731,6 +755,74 @@ internal static class WindowsHid return $"ok holdWrites={writes} rgb=#{r:X2}{g:X2}{b:X2} durationSeconds={duration.TotalSeconds:0.###} refreshMs={refresh.TotalMilliseconds:0}"; } + public static async Task TryBreatheLedAsync( + string rgb, + TimeSpan duration, + TimeSpan refresh, + string? eventLogPath, + bool turnOffOnExit) + { + if (!TryParseRgb(rgb, out var baseR, out var baseG, out var baseB)) + { + return $"invalid rgb: {rgb}"; + } + + var stopRequested = false; + Console.CancelKeyPress += (_, eventArgs) => + { + eventArgs.Cancel = true; + stopRequested = true; + }; + + var eventId = $"starfire-usb-move-breathe:{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}"; + var stopwatch = Stopwatch.StartNew(); + var writes = 0; + while (!stopRequested && (duration == TimeSpan.Zero || stopwatch.Elapsed < duration)) + { + var intensity = Math.Abs(Math.Sin(DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() / 1000.0)) * 0.5 + 0.5; + var r = ScaleColorChannel(baseR, intensity); + var g = ScaleColorChannel(baseG, intensity); + var b = ScaleColorChannel(baseB, intensity); + var devices = Enumerate() + .Where(static device => + device.Caps.OutputReportByteLength > 0 && + device.Path.IndexOf("&col01#", StringComparison.OrdinalIgnoreCase) >= 0) + .OrderBy(static device => device.Path, StringComparer.OrdinalIgnoreCase) + .ToList(); + + foreach (var device in devices) + { + var started = NowNs(); + var result = TryWriteLed(device, r, g, b); + var completed = NowNs(); + AppendPulseEvent(eventLogPath, eventId, writes, "breathe", r, g, b, started, completed, result); + writes++; + } + + await Task.Delay(refresh).ConfigureAwait(false); + } + + if (turnOffOnExit) + { + foreach (var device in Enumerate() + .Where(static device => + device.Caps.OutputReportByteLength > 0 && + device.Path.IndexOf("&col01#", StringComparison.OrdinalIgnoreCase) >= 0)) + { + var started = NowNs(); + var result = TryWriteLed(device, 0, 0, 0); + var completed = NowNs(); + AppendPulseEvent(eventLogPath, eventId, writes, "off", 0, 0, 0, started, completed, result); + writes++; + } + } + + return $"ok breatheWrites={writes} rgb=#{baseR:X2}{baseG:X2}{baseB:X2} durationSeconds={duration.TotalSeconds:0.###} refreshMs={refresh.TotalMilliseconds:0}"; + } + + private static byte ScaleColorChannel(byte channel, double intensity) => + (byte)Math.Clamp((int)Math.Round(channel * intensity), 0, 255); + public static async Task TryWriteScheduledLedTrainAsync( HidDeviceInfo device, IReadOnlyList events, From 34c435f9427928e71cf61c3bdc6565b0345835e9 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 14 Jun 2026 19:40:52 +0100 Subject: [PATCH 041/135] Refresh Mimir Face state --- .voidbot/state/mimir.cc | Bin 70102 -> 70551 bytes .voidbot/status/void-memory-maintenance.json | 2 +- .voidbot/voice/identity.json | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 96cf306134f87bf7a0e4c58f3a44dc47e14f6fa5..55e560519c117f83a098e5074ef62788c59ea1c9 100644 GIT binary patch delta 514 zcmcb%m}UBMmJQ!|xh$-Vjjc>9^vsPWGx9CqhI5TgCm(bWn|z2*PQ|#sd2MlWMrv+i zSZYymW_}*yx}wzN)Vz`)AemWEQhbL&bn-<-war@m!G@DBsi_zknwpzgn420}T9_G_ zSQ;5wEGq^ocP!aqWME{bYhb2pXcA&*Ze?U_Woo2nWN9$@qRDf{?8yQeBK0@(OEU6{ zQx%dEi;D6~iWM>vQxsA%)AKS*D)kgfGExoc6pHf|l1hs!6$*+{ zi<65olQQ$t6&^LrNlwhkFUm|Ue$+5mA+tmw3CK>%sZ>bIF9K;QNh~f&RYKwR<3hxp`F4EHs!El$oz%}oqT zEh^5;&tqIyl$xBHR}usyGYd+J?=Xl=zNo0SS&Ki|aPq-dVw3;d@lJloE5-vc+d|LG z#4Kteqv+-tPV9ydK_fFgLvs_5;N%0vsXT_pR;I>Q2Bvx@7RHkeODfpDpSvm6u045u zi5T15O}czxER#={s0#)p`Cj_C^3Up=#Nv|Rg8b~%$sg^cC(D&8awB|UG5G Date: Tue, 16 Jun 2026 08:04:25 +0100 Subject: [PATCH 042/135] Publish Mimir Eve dashboard health over Idunn RUDP --- .../Mimir.EveDashboard.csproj | 19 + src/Mimir.EveDashboard/Program.cs | 3487 +++++++++++++++++ .../Synchronization/MimirCultMeshContracts.cs | 133 + 3 files changed, 3639 insertions(+) create mode 100644 src/Mimir.EveDashboard/Mimir.EveDashboard.csproj create mode 100644 src/Mimir.EveDashboard/Program.cs diff --git a/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj b/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj new file mode 100644 index 0000000..5e01cd5 --- /dev/null +++ b/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj @@ -0,0 +1,19 @@ + + + + Exe + net10.0 + Mimir.EveDashboard + Mimir.EveDashboard + enable + enable + + + + + + + + + + diff --git a/src/Mimir.EveDashboard/Program.cs b/src/Mimir.EveDashboard/Program.cs new file mode 100644 index 0000000..ae2569f --- /dev/null +++ b/src/Mimir.EveDashboard/Program.cs @@ -0,0 +1,3487 @@ +using System.Collections.Concurrent; +using System.Globalization; +using System.Net; +using System.Net.Sockets; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using GameCult.Caching; +using GameCult.Mesh; +using GameCult.Networking; +using MessagePack; +using Mimir.Runtime.Synchronization; + +var port = ParseInt(args, "--port", 8795); +var cultNetPort = ParseInt(args, "--cultnet-port", 3075); +var cultNetCachePath = ParseString(args, "--cultnet-cache", @"E:\Projects\Mimir\state\eve-dashboard-cultnet.ccmp"); +var voidBotSwarmStatePath = ParseString(args, "--voidbot-swarm-state", @"E:\Projects\VoidBot\.voidbot\status\swarm-state.json"); +var mimirTelemetryLogPath = ParseString(args, "--mimir-telemetry-log", ""); +var mimirObservationLogPath = ParseString(args, "--mimir-observation-log", @"E:\Projects\Mimir\artifacts\runtime\periwinkle-cultmesh-sensors.out.log"); +var idunnRudpHealth = ParseString(args, "--idunn-rudp-health", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_RUDP_HEALTH") ?? ""); +var idunnDaemon = ParseString(args, "--idunn-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_DAEMON") ?? "mimir-eve-dashboard"); +var idunnHealthContract = ParseString(args, "--idunn-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_HEALTH_CONTRACT") ?? "mimir.cultnet-rudp-provider-health"); +var providers = EveDashboardProviderCatalog.Create(ParseProviderSpecs(args), voidBotSwarmStatePath, mimirTelemetryLogPath, mimirObservationLogPath); +using var cultNetPublisher = await EveDashboardCultNetPublisher.StartAsync(cultNetCachePath, cultNetPort).ConfigureAwait(false); +using var server = new EveDashboardServer(port, providers, cultNetPublisher, new IdunnRudpHealthOptions(idunnRudpHealth, idunnDaemon, idunnHealthContract)); +await server.RunAsync(); + +static int ParseInt(IReadOnlyList args, string name, int fallback) +{ + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && + int.TryParse(args[index + 1], out var value)) + { + return value; + } + } + + return fallback; +} + +static IReadOnlyList ParseProviderSpecs(IReadOnlyList args) +{ + var specs = new List(); + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], "--provider", StringComparison.OrdinalIgnoreCase)) + { + specs.Add(args[index + 1]); + } + } + + return specs; +} + +static string ParseString(IReadOnlyList args, string name, string fallback) +{ + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && + !string.IsNullOrWhiteSpace(args[index + 1])) + { + return args[index + 1]; + } + } + + return fallback; +} + +internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog providers, EveDashboardCultNetPublisher cultNetPublisher, IdunnRudpHealthOptions idunnRudpHealth) : IDisposable +{ + private static readonly JsonSerializerOptions JsonOptions = new() + { + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + }; + + private readonly TcpListener listener = new(IPAddress.Any, port); + private readonly CancellationTokenSource stopping = new(); + private readonly ConcurrentDictionary clients = new(); + private readonly object idunnRudpPublishLock = new(); + private string activeProviderId = providers.DefaultProviderId; + private string idunnRudpPublishStatus = string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) ? "disabled" : "idle"; + private string idunnRudpPublishError = ""; + private string idunnRudpPublishObservedAt = ""; + private bool idunnRudpPublishInFlight; + private DateTimeOffset lastHealthPublishAttemptAt = DateTimeOffset.MinValue; + + public async Task RunAsync() + { + listener.Start(); + Console.WriteLine($"Mimir Eve dashboard broker listening on ws://0.0.0.0:{port}/eve/deck"); + Console.WriteLine($"Mimir Eve dashboard CultNet publisher listening on cultnet://0.0.0.0:{cultNetPublisher.Port}"); + Console.WriteLine(string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) + ? "Mimir Eve dashboard Idunn RUDP health publisher disabled." + : $"Mimir Eve dashboard Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} as {idunnRudpHealth.DaemonId}."); + Console.WriteLine($"Compatibility endpoint remains ws://0.0.0.0:{port}/eve/dashboard"); + _ = Task.Run(BroadcastHeartbeatAsync); + while (!stopping.IsCancellationRequested) + { + var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false); + _ = Task.Run(() => HandleAsync(client)); + } + } + + private async Task BroadcastHeartbeatAsync() + { + using var timer = new PeriodicTimer(TimeSpan.FromSeconds(1)); + while (!stopping.IsCancellationRequested && await timer.WaitForNextTickAsync(stopping.Token).ConfigureAwait(false)) + { + await PublishCultNetStateAsync().ConfigureAwait(false); + if (clients.IsEmpty) + { + continue; + } + + await BroadcastStateAsync().ConfigureAwait(false); + } + } + + private async Task HandleAsync(TcpClient tcpClient) + { + await using var stream = tcpClient.GetStream(); + var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false); + Console.WriteLine($"EVE dashboard connected: {tcpClient.Client.RemoteEndPoint} {request.Path}"); + Console.Out.Flush(); + + if (request.Path == "/health") + { + var provider = providers.Get(activeProviderId); + var health = JsonSerializer.Serialize(new + { + ok = true, + clients = clients.Count, + activeProviderId, + providerVersion = provider.State.Version, + providers = providers.Manifests.Count, + transport = "eve-deck-ws", + cultMeshDocument = "mimir.eve_dashboard_state", + idunnRudpHealth = GetIdunnRudpHealthStatus(), + }, JsonOptions); + await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false); + return; + } + + if (request.Path is "/eve/deck/manifest" or "/eve/dashboard/manifest") + { + var manifest = JsonSerializer.Serialize(providers.BrokerManifest, JsonOptions); + await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(manifest)).ConfigureAwait(false); + return; + } + + if (request.Path == "/eve/deck/providers") + { + var catalog = JsonSerializer.Serialize(new DashboardProviderCatalogDocument(providers.Manifests), JsonOptions); + await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(catalog)).ConfigureAwait(false); + return; + } + + if (!IsDashboardPath(request.Path, out var requestedProviderId, out var cultMeshBinary) || + !request.Headers.TryGetValue("Sec-WebSocket-Key", out var key)) + { + await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false); + return; + } + + if (!string.IsNullOrWhiteSpace(requestedProviderId) && providers.Contains(requestedProviderId)) + { + activeProviderId = requestedProviderId; + } + + await WriteWebSocketHandshakeAsync(stream, key).ConfigureAwait(false); + var id = Guid.NewGuid(); + var socket = new DashboardSocket(tcpClient, stream, cultMeshBinary); + clients[id] = socket; + await SendStateAsync(socket).ConfigureAwait(false); + try + { + while (!stopping.IsCancellationRequested) + { + var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false); + if (frame.Opcode == 0x8) + { + await WriteCloseFrameAsync(stream).ConfigureAwait(false); + break; + } + + if (frame.Opcode == 0x2 && cultMeshBinary) + { + if (ApplyCultMeshCommand(frame.Payload)) + { + await BroadcastStateAsync().ConfigureAwait(false); + } + + continue; + } + + if (frame.Opcode != 0x1) + { + continue; + } + + var text = Encoding.UTF8.GetString(frame.Payload); + if (ApplyCommand(text)) + { + await BroadcastStateAsync().ConfigureAwait(false); + } + } + } + finally + { + clients.TryRemove(id, out _); + } + } + + private static bool IsDashboardPath(string path, out string providerId, out bool cultMeshBinary) + { + providerId = ""; + cultMeshBinary = false; + if (path == "/eve/deck/cultmesh") + { + cultMeshBinary = true; + return true; + } + + const string cultMeshProviderPrefix = "/eve/deck/cultmesh/"; + if (path.StartsWith(cultMeshProviderPrefix, StringComparison.Ordinal)) + { + providerId = Uri.UnescapeDataString(path[cultMeshProviderPrefix.Length..]); + cultMeshBinary = true; + return !string.IsNullOrWhiteSpace(providerId); + } + + if (path == "/eve/dashboard" || path == "/eve/deck") + { + return true; + } + + const string providerPrefix = "/eve/deck/"; + if (path.StartsWith(providerPrefix, StringComparison.Ordinal)) + { + providerId = Uri.UnescapeDataString(path[providerPrefix.Length..]); + return !string.IsNullOrWhiteSpace(providerId); + } + + return false; + } + + private bool ApplyCultMeshCommand(byte[] payload) + { + MimirEveDashboardCommandDocument? commandDocument; + try + { + commandDocument = MessagePackSerializer.Deserialize(payload); + } + catch + { + return false; + } + + var command = new DashboardCommand + { + Type = commandDocument.Type, + NodeId = commandDocument.NodeId, + ProviderId = commandDocument.ProviderId, + X = commandDocument.X, + Y = commandDocument.Y, + Rotation = commandDocument.Rotation, + Scale = commandDocument.Scale, + Visible = commandDocument.Visible, + }; + + return ApplyCommand(JsonSerializer.Serialize(command, JsonOptions)); + } + + private bool ApplyCommand(string text) + { + DashboardCommand? command; + try + { + command = JsonSerializer.Deserialize(text, JsonOptions); + } + catch (JsonException) + { + return false; + } + + if (command == null) + { + return false; + } + + if (string.Equals(command.Type, "open-provider", StringComparison.OrdinalIgnoreCase)) + { + var providerId = command.ProviderId; + if (string.IsNullOrWhiteSpace(providerId) && !string.IsNullOrWhiteSpace(command.NodeId)) + { + providerId = providers.Get(activeProviderId).State.Nodes + .FirstOrDefault(node => string.Equals(node.Id, command.NodeId, StringComparison.Ordinal)) + ?.ProviderId; + } + + if (!string.IsNullOrWhiteSpace(providerId) && providers.Contains(providerId)) + { + activeProviderId = providerId; + Console.WriteLine($"EVE dashboard provider switched: {activeProviderId}"); + Console.Out.Flush(); + return true; + } + + return false; + } + + var provider = providers.Get(activeProviderId); + if (provider.ApplyCommand(command)) + { + Console.WriteLine($"EVE dashboard command({activeProviderId}): {text}"); + Console.Out.Flush(); + return true; + } + + return false; + } + + private async Task SendStateAsync(DashboardSocket socket) + { + var state = providers.Get(activeProviderId).State; + if (socket.CultMeshBinary) + { + await SendBinaryFrameAsync(socket.Stream, MessagePackSerializer.Serialize(ToCultMeshState(state))).ConfigureAwait(false); + return; + } + + await SendTextFrameAsync(socket.Stream, JsonSerializer.Serialize(state, JsonOptions)).ConfigureAwait(false); + } + + private Task PublishCultNetStateAsync() + { + var state = providers.Get(activeProviderId).State; + QueueIdunnRudpHealthIfDue(state, DateTimeOffset.UtcNow); + return cultNetPublisher.PublishAsync(ToCultMeshState(state), stopping.Token); + } + + private IdunnRudpHealthStatus GetIdunnRudpHealthStatus() + { + lock (idunnRudpPublishLock) + { + return new IdunnRudpHealthStatus( + idunnRudpHealth.Endpoint, + idunnRudpHealth.DaemonId, + idunnRudpHealth.HealthContract, + idunnRudpPublishStatus, + idunnRudpPublishError, + idunnRudpPublishInFlight, + idunnRudpPublishObservedAt); + } + } + + private void QueueIdunnRudpHealthIfDue(DashboardState state, DateTimeOffset observedAt) + { + if (string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint)) + { + idunnRudpPublishStatus = "disabled"; + return; + } + + if (lastHealthPublishAttemptAt != DateTimeOffset.MinValue && + observedAt - lastHealthPublishAttemptAt < TimeSpan.FromSeconds(5)) + { + return; + } + + var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); + var health = new IdunnDaemonHealthRecord + { + DaemonId = idunnRudpHealth.DaemonId, + State = "healthy", + Detail = $"Mimir Eve dashboard broker active; provider={state.ProviderId}; providerVersion={state.Version}; providers={providers.Manifests.Count}; clients={clients.Count}; cultMeshPort={cultNetPublisher.Port}", + ObservedAt = observedAtText, + HealthContract = idunnRudpHealth.HealthContract, + PublicationSource = "daemon-published", + Transport = "cultnet.transport.rudp.v0", + }; + + lock (idunnRudpPublishLock) + { + if (idunnRudpPublishInFlight) + { + return; + } + + idunnRudpPublishInFlight = true; + lastHealthPublishAttemptAt = observedAt; + idunnRudpPublishObservedAt = observedAtText; + if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) + { + idunnRudpPublishStatus = "publishing"; + idunnRudpPublishError = ""; + } + } + + _ = Task.Run(() => + { + try + { + IdunnRudpHealthPublisher.Publish(idunnRudpHealth.Endpoint, health); + lock (idunnRudpPublishLock) + { + idunnRudpPublishStatus = "published"; + idunnRudpPublishError = ""; + idunnRudpPublishInFlight = false; + } + } + catch (Exception error) + { + lock (idunnRudpPublishLock) + { + if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) + { + idunnRudpPublishStatus = "publish-error"; + } + + idunnRudpPublishError = error.Message; + idunnRudpPublishInFlight = false; + } + } + }); + } + + private static MimirEveDashboardStateDocument ToCultMeshState(DashboardState state) => + new( + state.ProviderId, + state.Title, + state.Version, + state.UpdatedAt.ToString("O"), + state.SelectedNodeId, + state.LutPreset, + state.Nodes.Select(static node => new MimirEveDashboardNodeSnapshot( + node.Id, + node.Label, + node.Kind, + node.Visible, + node.X, + node.Y, + node.Z, + node.Rotation, + node.Scale, + node.Width, + node.Height, + node.Health, + node.ProviderId, + node.Command, + node.Endpoint)).ToArray(), + ToCultMeshSurface(state.Surface)); + + private static MimirEveDashboardSurfaceSnapshot? ToCultMeshSurface(DashboardSurface? surface) + { + if (surface == null) + { + return null; + } + + return new MimirEveDashboardSurfaceSnapshot( + surface.Schema, + surface.Id, + surface.Title, + ToCultMeshElement(surface.Root), + surface.Assets + .Select(static asset => new MimirEveDashboardSurfaceAssetSnapshot(asset.Id, asset.Kind, asset.Uri)) + .ToArray()); + } + + private static MimirEveDashboardUiElementSnapshot ToCultMeshElement(DashboardUiElement element) + { + return new MimirEveDashboardUiElementSnapshot( + element.Id, + element.Kind, + element.Role, + element.Text, + element.AssetRef, + element.AssetUri, + element.BindNodeId, + element.CommandId, + element.Layout == null + ? null + : new MimirEveDashboardUiLayoutSnapshot( + element.Layout.Direction, + element.Layout.Width, + element.Layout.Height, + element.Layout.Grow, + element.Layout.Gap, + element.Layout.Padding, + element.Layout.Overflow, + element.Layout.MinWidth, + element.Layout.MinHeight, + element.Layout.PreferredWidth, + element.Layout.PreferredHeight, + element.Layout.Priority, + element.Layout.Density, + element.Layout.ViewportMode), + element.Style == null + ? null + : new MimirEveDashboardUiStyleSnapshot(element.Style.Variant, element.Style.Tone), + element.Metric == null + ? null + : new MimirEveDashboardUiMetricSnapshot(element.Metric.Label, element.Metric.Value, element.Metric.Tone), + element.Children.Select(ToCultMeshElement).ToArray(), + element.Binding == null + ? null + : new MimirEveDashboardUiBindingSnapshot( + element.Binding.DocumentSchema, + element.Binding.DocumentId, + element.Binding.Path, + element.Binding.ValueKind, + element.Binding.Access, + element.Binding.Authority, + element.Binding.CommandId)); + } + + private async Task BroadcastStateAsync() + { + foreach (var (id, socket) in clients) + { + try + { + await SendStateAsync(socket).ConfigureAwait(false); + } + catch + { + clients.TryRemove(id, out _); + } + } + } + + private static async Task ReadHttpRequestAsync(NetworkStream stream) + { + var bytes = new List(); + var lastFour = new Queue(4); + while (true) + { + var value = stream.ReadByte(); + if (value < 0) + { + throw new IOException("Client closed before HTTP headers completed."); + } + + var b = (byte)value; + bytes.Add(b); + lastFour.Enqueue(b); + if (lastFour.Count > 4) + { + lastFour.Dequeue(); + } + + if (lastFour.Count == 4 && lastFour.SequenceEqual(new byte[] { 13, 10, 13, 10 })) + { + break; + } + } + + var text = Encoding.ASCII.GetString(bytes.ToArray()); + var lines = text.Split("\r\n", StringSplitOptions.None); + var first = lines[0].Split(' '); + var headers = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var line in lines.Skip(1)) + { + var colon = line.IndexOf(':'); + if (colon > 0) + { + headers[line[..colon].Trim()] = line[(colon + 1)..].Trim(); + } + } + + await Task.CompletedTask.ConfigureAwait(false); + return new HttpRequest(first.Length > 1 ? first[1] : "/", headers); + } + + private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body) + { + var header = Encoding.ASCII.GetBytes( + $"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n"); + await stream.WriteAsync(header).ConfigureAwait(false); + await stream.WriteAsync(body).ConfigureAwait(false); + } + + private static async Task WriteWebSocketHandshakeAsync(NetworkStream stream, string key) + { + var acceptBytes = SHA1.HashData(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")); + var accept = Convert.ToBase64String(acceptBytes); + var response = Encoding.ASCII.GetBytes( + "HTTP/1.1 101 Switching Protocols\r\n" + + "Upgrade: websocket\r\n" + + "Connection: Upgrade\r\n" + + $"Sec-WebSocket-Accept: {accept}\r\n\r\n"); + await stream.WriteAsync(response).ConfigureAwait(false); + } + + private static async Task SendTextFrameAsync(NetworkStream stream, string text) + { + var payload = Encoding.UTF8.GetBytes(text); + var header = new List { 0x81 }; + if (payload.Length < 126) + { + header.Add((byte)payload.Length); + } + else if (payload.Length <= ushort.MaxValue) + { + header.Add(126); + header.Add((byte)(payload.Length >> 8)); + header.Add((byte)payload.Length); + } + else + { + header.Add(127); + var length = BitConverter.GetBytes(IPAddress.HostToNetworkOrder((long)payload.Length)); + header.AddRange(length); + } + + await stream.WriteAsync(header.ToArray()).ConfigureAwait(false); + await stream.WriteAsync(payload).ConfigureAwait(false); + } + + private static async Task SendBinaryFrameAsync(NetworkStream stream, byte[] payload) + { + var header = BuildFrameHeader(0x82, payload.Length); + await stream.WriteAsync(header).ConfigureAwait(false); + await stream.WriteAsync(payload).ConfigureAwait(false); + } + + private static byte[] BuildFrameHeader(byte opcode, int payloadLength) + { + var header = new List { opcode }; + if (payloadLength < 126) + { + header.Add((byte)payloadLength); + } + else if (payloadLength <= ushort.MaxValue) + { + header.Add(126); + header.Add((byte)(payloadLength >> 8)); + header.Add((byte)payloadLength); + } + else + { + header.Add(127); + var length = BitConverter.GetBytes(IPAddress.HostToNetworkOrder((long)payloadLength)); + header.AddRange(length); + } + + return header.ToArray(); + } + + private static async Task ReceiveFrameAsync(NetworkStream stream) + { + var header = new byte[2]; + await ReadExactAsync(stream, header).ConfigureAwait(false); + var opcode = header[0] & 0x0f; + var masked = (header[1] & 0x80) != 0; + var length = header[1] & 0x7f; + if (length == 126) + { + var extended = new byte[2]; + await ReadExactAsync(stream, extended).ConfigureAwait(false); + length = (extended[0] << 8) | extended[1]; + } + else if (length == 127) + { + var extended = new byte[8]; + await ReadExactAsync(stream, extended).ConfigureAwait(false); + var longLength = 0L; + foreach (var b in extended) + { + longLength = (longLength << 8) | b; + } + + if (longLength > int.MaxValue) + { + throw new InvalidDataException("WebSocket frame is too large."); + } + + length = (int)longLength; + } + + var mask = new byte[4]; + if (masked) + { + await ReadExactAsync(stream, mask).ConfigureAwait(false); + } + + var payload = new byte[length]; + await ReadExactAsync(stream, payload).ConfigureAwait(false); + if (masked) + { + for (var index = 0; index < payload.Length; index++) + { + payload[index] ^= mask[index % 4]; + } + } + + return new WebSocketFrame(opcode, payload); + } + + private static async Task ReadExactAsync(NetworkStream stream, byte[] buffer) + { + var offset = 0; + while (offset < buffer.Length) + { + var read = await stream.ReadAsync(buffer.AsMemory(offset, buffer.Length - offset)).ConfigureAwait(false); + if (read == 0) + { + throw new IOException("WebSocket closed."); + } + + offset += read; + } + } + + private static Task WriteCloseFrameAsync(NetworkStream stream) => + stream.WriteAsync(new byte[] { 0x88, 0x00 }).AsTask(); + + public void Dispose() + { + stopping.Cancel(); + listener.Stop(); + stopping.Dispose(); + } +} + +internal sealed record IdunnRudpHealthOptions(string Endpoint, string DaemonId, string HealthContract); + +internal sealed record IdunnRudpHealthStatus( + string Endpoint, + string Daemon, + string Contract, + string Status, + string Error, + bool InFlight, + string ObservedAtUtc); + +[MessagePackObject(AllowPrivate = true)] +internal sealed class IdunnDaemonHealthRecord +{ + [Key(0)] public string DaemonId { get; set; } = "mimir-eve-dashboard"; + [Key(1)] public string State { get; set; } = "active"; + [Key(2)] public string Detail { get; set; } = ""; + [Key(3)] public string ObservedAt { get; set; } = ""; + [Key(4)] public string HealthContract { get; set; } = "mimir.cultnet-rudp-provider-health"; + [Key(5)] public string PublicationSource { get; set; } = "daemon-published"; + [Key(6)] public string Transport { get; set; } = "cultnet.transport.rudp.v0"; +} + +internal static class IdunnRudpHealthPublisher +{ + private const uint ConnectionId = 0x1d0d0001; + + public static void Publish(string endpoint, IdunnDaemonHealthRecord health) + { + var remote = ParseEndpoint(endpoint); + using var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp); + socket.Bind(new IPEndPoint(IPAddress.Any, 0)); + socket.Blocking = false; + socket.ReceiveTimeout = 100; + using var transport = new CultNetRudpSocketTransportConnection(new CultNetRudpSocketTransportOptions + { + RuntimeId = "mimir-eve-dashboard", + Socket = socket, + Mode = CultNetRudpSocketMode.Client, + RemoteEndPoint = remote, + ConnectionId = ConnectionId, + InitialSequence = 1, + ResendDelayMs = 100, + }); + + if (!transport.ConnectAndWait(Array.Empty(), TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(20))) + { + throw new TimeoutException($"timed out connecting Mimir Eve dashboard RUDP health publisher to {endpoint}"); + } + + var observedAt = string.IsNullOrWhiteSpace(health.ObservedAt) + ? DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture) + : health.ObservedAt; + health.ObservedAt = observedAt; + var message = new CultNetDocumentPutRawMessage + { + MessageId = $"mimir-eve-dashboard-health:{health.DaemonId}:{observedAt.Replace(':', '-')}", + Document = new CultNetRawDocumentRecord + { + SchemaId = "idunn.daemon_health", + RecordKey = health.DaemonId, + StoredAt = observedAt, + PayloadEncoding = "messagepack", + Payload = MessagePackSerializer.Serialize(health, CultNetSchemaMessageSerialization.Options), + SourceRuntimeId = "mimir-eve-dashboard", + SourceRole = "daemon-health-publisher", + Tags = ["cultnet.transport.rudp.v0"], + }, + }; + transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message)); + } + + private static IPEndPoint ParseEndpoint(string value) + { + var trimmed = value.Trim(); + var separator = trimmed.LastIndexOf(':'); + if (separator <= 0 || separator == trimmed.Length - 1) + { + throw new ArgumentException($"Idunn RUDP endpoint must be host:port, got '{value}'.", nameof(value)); + } + + var host = trimmed[..separator].Trim('[', ']'); + var portText = trimmed[(separator + 1)..]; + if (!int.TryParse(portText, NumberStyles.Integer, CultureInfo.InvariantCulture, out var port) || + port <= 0 || port > 65535) + { + throw new ArgumentException($"Idunn RUDP endpoint port is invalid: {portText}", nameof(value)); + } + + var addresses = Dns.GetHostAddresses(host); + var address = addresses.FirstOrDefault(static candidate => candidate.AddressFamily == AddressFamily.InterNetwork) ?? + throw new ArgumentException($"Idunn RUDP endpoint host has no IPv4 address: {host}", nameof(value)); + return new IPEndPoint(address, port); + } +} + +internal sealed class EveDashboardCultNetPublisher : IDisposable +{ + private static readonly CultRecordKey StateRecordKey = new("eve.dashboard.broker"); + private readonly CultMeshNode node; + + private EveDashboardCultNetPublisher(CultMeshNode node, int port) + { + this.node = node; + Port = port; + } + + public int Port { get; } + + public static async Task StartAsync(string cachePath, int port) + { + if (port != 3075) + { + throw new ArgumentOutOfRangeException(nameof(port), port, "CultNet Server currently listens on the runtime default port 3075."); + } + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(cachePath)) ?? "."); + var documents = new CultNetDocumentRegistry() + .Register(CultNetDocumentBinding.ForDocument()); + var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions + { + StartServer = true, + DatabaseOptions = new CultNetDatabaseOptions + { + RuntimeId = "mimir.eve-dashboard", + DocumentRegistry = documents, + }, + }).ConfigureAwait(false); + + return new EveDashboardCultNetPublisher(node, port); + } + + public async Task PublishAsync(MimirEveDashboardStateDocument document, CancellationToken stopping) + { + await node.Database.PutAsync(new CultRecordKey(document.ProviderId), document).ConfigureAwait(false); + if (!string.Equals(document.ProviderId, StateRecordKey.Value, StringComparison.Ordinal)) + { + await node.Database.PutAsync(StateRecordKey, document).ConfigureAwait(false); + } + + await node.FlushAsync(soft: true).ConfigureAwait(false); + } + + public void Dispose() => node.Dispose(); +} + +internal sealed class EveDashboardProviderCatalog +{ + private readonly Dictionary providers; + + private EveDashboardProviderCatalog(IEnumerable providers) + { + this.providers = providers.ToDictionary(provider => provider.Manifest.Id, StringComparer.Ordinal); + DefaultProviderId = DashboardBrokerProvider.ProviderId; + Manifests = this.providers.Values.Select(provider => provider.Manifest).ToArray(); + BrokerManifest = new DashboardProviderManifest( + "eve.dashboard.broker", + "Eve Dashboard Broker", + "Native retained dashboard router for LAN and SSH-tunneled control surfaces.", + "1", + "/eve/deck", + ["scene2d", "provider-switching", "touch-commands", "cultmesh-state-documents"], + UsesCultMesh: true, + Transport: "WebSocket now; CultMesh typed state when available."); + } + + public string DefaultProviderId { get; } + + public IReadOnlyList Manifests { get; } + + public DashboardProviderManifest BrokerManifest { get; } + + public static EveDashboardProviderCatalog Create( + IReadOnlyList remoteSpecs, + string voidBotSwarmStatePath, + string mimirTelemetryLogPath, + string mimirObservationLogPath) + { + var providers = new List(); + var remoteProviders = remoteSpecs.Select(RemoteDashboardProvider.FromSpec).OfType().ToArray(); + providers.Add(new DashboardBrokerProvider(remoteProviders)); + providers.Add(new MimirLiveStatsProvider(mimirTelemetryLogPath, mimirObservationLogPath)); + providers.Add(new VoidBotSwarmProvider(voidBotSwarmStatePath)); + providers.Add(new YggdrasilStreamPixelsProvider()); + providers.AddRange(remoteProviders); + return new EveDashboardProviderCatalog(providers); + } + + public bool Contains(string providerId) => providers.ContainsKey(providerId); + + public IDashboardProvider Get(string providerId) => + providers.TryGetValue(providerId, out var provider) ? provider : providers[DefaultProviderId]; +} + +internal interface IDashboardProvider +{ + DashboardProviderManifest Manifest { get; } + + DashboardState State { get; } + + bool ApplyCommand(DashboardCommand command); +} + +internal abstract class MutableDashboardProvider : IDashboardProvider +{ + protected MutableDashboardProvider(DashboardProviderManifest manifest, DashboardState state) + { + Manifest = manifest; + State = state; + } + + public DashboardProviderManifest Manifest { get; } + + public DashboardState State { get; } + + public virtual bool ApplyCommand(DashboardCommand command) + { + if (string.IsNullOrWhiteSpace(command.NodeId)) + { + return false; + } + + var node = State.Nodes.FirstOrDefault(candidate => string.Equals(candidate.Id, command.NodeId, StringComparison.Ordinal)); + if (node == null) + { + return false; + } + + switch ((command.Type ?? "").Trim().ToLowerInvariant()) + { + case "select": + State.SelectedNodeId = node.Id; + break; + case "move": + State.SelectedNodeId = node.Id; + node.X = Clamp(command.X ?? node.X, -1.0, 1.0); + node.Y = Clamp(command.Y ?? node.Y, -1.0, 1.0); + break; + case "scale": + State.SelectedNodeId = node.Id; + node.Scale = Clamp(command.Scale ?? node.Scale, 0.25, 3.0); + break; + case "rotate": + State.SelectedNodeId = node.Id; + node.Rotation = command.Rotation ?? node.Rotation; + break; + case "toggle-visibility": + State.SelectedNodeId = node.Id; + node.Visible = command.Visible ?? !node.Visible; + break; + case "reset-transform": + State.SelectedNodeId = node.Id; + node.X = node.DefaultX; + node.Y = node.DefaultY; + node.Rotation = 0.0; + node.Scale = 1.0; + break; + default: + return false; + } + + Touch(); + return true; + } + + protected void Touch() + { + State.Version++; + State.UpdatedAt = DateTimeOffset.UtcNow; + } + + protected static double Clamp(double value, double min, double max) => Math.Min(max, Math.Max(min, value)); +} + +internal sealed class DashboardBrokerProvider : MutableDashboardProvider +{ + public const string ProviderId = "eve.dashboard.broker"; + + public DashboardBrokerProvider(IReadOnlyList remoteProviders) + : base( + new DashboardProviderManifest( + ProviderId, + "Dashboard Switchboard", + "Provider picker rendered as native Eve panels.", + "1", + "/eve/deck", + ["scene2d", "provider-switching"], + UsesCultMesh: true, + Transport: "broker-local"), + CreateState(remoteProviders)) + { + } + + private static DashboardState CreateState(IReadOnlyList remoteProviders) + { + var nodes = new List + { + new("provider-mimir-live", "Mimir Live Stats", "dashboard-provider", -0.56, -0.34, 0.30, 0.22, "telemetry") + { + ProviderId = MimirLiveStatsProvider.ProviderId, + Command = "open-provider", + }, + new("provider-mimir", "Mimir Stream Layout", "dashboard-provider", -0.18, -0.34, 0.30, 0.22, "local") + { + ProviderId = MimirStreamLayoutProvider.ProviderId, + Command = "open-provider", + }, + new("provider-voidbot", "VoidBot Swarm", "dashboard-provider", 0.0, -0.34, 0.30, 0.22, "ctb") + { + ProviderId = VoidBotSwarmProvider.ProviderId, + Command = "open-provider", + }, + new("provider-streampixels", "StreamPixels Edge", "dashboard-provider", 0.56, -0.34, 0.30, 0.22, "ssh ready") + { + ProviderId = YggdrasilStreamPixelsProvider.ProviderId, + Command = "open-provider", + }, + }; + + var y = 0.28; + foreach (var provider in remoteProviders) + { + nodes.Add(new DashboardNode( + $"provider-{provider.Manifest.Id}", + provider.Manifest.Title, + "dashboard-provider", + 0.0, + y, + 0.40, + 0.18, + "external") + { + ProviderId = provider.Manifest.Id, + Command = "open-provider", + }); + y = Math.Min(0.80, y + 0.22); + } + + return new DashboardState + { + ProviderId = ProviderId, + Title = "Eve Dashboard Switchboard", + SelectedNodeId = nodes[0].Id, + Nodes = nodes, + }; + } +} + +internal sealed class MimirLiveStatsProvider : IDashboardProvider +{ + public const string ProviderId = "mimir.live.stats"; + private const int TailBytes = 512 * 1024; + private readonly string telemetryLogPath; + private readonly string observationLogPath; + private long version; + + public MimirLiveStatsProvider(string telemetryLogPath, string observationLogPath) + { + this.telemetryLogPath = telemetryLogPath; + this.observationLogPath = observationLogPath; + Manifest = new DashboardProviderManifest( + ProviderId, + "Mimir Live Stats", + "Compact Eve/CultUI surface for live Mimir telemetry: RMS bars, sync confidence, source buffers, actuator state, and device observation streams.", + "1", + "/eve/deck/mimir.live.stats", + ["live-telemetry", "compact-tui", "audio-rms", "sync-confidence", "observation-streams"], + UsesCultMesh: true, + Transport: "Mimir telemetry and observation ledgers projected as Eve/CultUI state."); + } + + public DashboardProviderManifest Manifest { get; } + + public DashboardState State => BuildState(); + + public bool ApplyCommand(DashboardCommand command) => false; + + private DashboardState BuildState() + { + var snapshot = MimirLiveStatsSnapshot.Load(telemetryLogPath, observationLogPath); + var confidence = snapshot.SyncStates.Count == 0 + ? snapshot.SyncReports.Select(static report => report.Confidence) + .Concat(snapshot.SyncDecodeAttempts.Select(static attempt => attempt.Confidence)) + .Concat(snapshot.Well?.AudioSyncStates.Select(static state => state.Confidence) ?? []) + .DefaultIfEmpty(0.0) + .Max() + : snapshot.SyncStates.Average(static state => state.Confidence); + var rms = snapshot.Spectra.Count == 0 ? 0.0 : snapshot.Spectra.Max(static spectrum => spectrum.Rms); + var liveObservationCount = snapshot.ObservationStreams.Count(static stream => stream.State == "active"); + var status = snapshot.HasAnyData ? "live" : "waiting"; + var nodes = new List + { + new("mimir-live-root", $"Mimir Live Stats\n{snapshot.Summary}", "mimir-live-stats", 0.0, -0.50, 0.70, 0.20, status) + { + Detail = $"telemetry={snapshot.TelemetrySource} observations={snapshot.ObservationSource}", + }, + new("mimir-tracking-confidence", $"Tracking Confidence\n{confidence:0.000}", "confidence", -0.44, -0.12, 0.28, 0.16, ToneFor(confidence)) + { + Detail = $"{snapshot.SyncStates.Count} sync states / {snapshot.SyncReports.Count} reports / {snapshot.SyncDecodeAttempts.Count} decode attempts", + }, + new("mimir-audio-rms", $"Audio RMS\n{rms:0.000000}", "audio-rms", 0.0, -0.12, 0.28, 0.16, ToneFor(NormalizeRms(rms))) + { + Detail = $"{snapshot.Spectra.Count} spectrum lanes; Well audio buffers {snapshot.Well?.AudioBuffers ?? 0}", + }, + new("mimir-observation-streams", $"Device Streams\n{liveObservationCount}/{snapshot.ObservationStreams.Count} active", "observation-streams", 0.44, -0.12, 0.28, 0.16, liveObservationCount > 0 ? "active" : "waiting") + { + Detail = string.Join(", ", snapshot.ObservationStreams.Select(static stream => $"{stream.StreamId}:{stream.Kind}:{stream.State}")), + }, + }; + + return new DashboardState + { + ProviderId = ProviderId, + Title = "Mimir Live Stats", + Version = Interlocked.Increment(ref version), + UpdatedAt = DateTimeOffset.UtcNow, + SelectedNodeId = "mimir-live-root", + LutPreset = "terminal", + Nodes = nodes, + Surface = BuildSurface(snapshot, confidence, rms, liveObservationCount), + }; + } + + private static DashboardSurface BuildSurface(MimirLiveStatsSnapshot snapshot, double confidence, double rms, int liveObservationCount) + { + var pulse = Pulse(); + var well = snapshot.Well; + var liveRatio = snapshot.ObservationStreams.Count == 0 ? 0.0 : liveObservationCount / (double)snapshot.ObservationStreams.Count; + var staleStreams = snapshot.ObservationStreams.Count(static stream => stream.State != "active"); + var sourceDropouts = well == null ? 0 : Math.Max(0, well.ConfiguredSources - well.LiveSources) + well.SourceErrorCount + well.SourceHealth.Count(static source => source.Status != "live"); + var throughputScore = well == null + ? Math.Clamp(snapshot.Telemetry?.Ingested / 120.0 ?? 0.0, 0.0, 1.0) + : Math.Clamp((well.PublishedBytes / 262_144.0) / Math.Max(1.0, well.PublishAverageMs + well.PollAverageMs), 0.0, 1.0); + var latencyScore = well == null ? 0.0 : Math.Clamp(1.0 - (well.LatencyCurrentMs / Math.Max(1.0, well.LatencyCeilingMs)), 0.0, 1.0); + var dropoutScore = Math.Clamp(1.0 - ((sourceDropouts + staleStreams) / Math.Max(1.0, (well?.ConfiguredSources ?? 0) + snapshot.ObservationStreams.Count)), 0.0, 1.0); + var children = new List + { + UiElement.Pane("mimir-sync-signal", "Sync", [ + UiElement.Text("mimir-live-summary", $"{pulse} {snapshot.Summary}", "strong"), + StatBar("tracking-confidence", "confidence", confidence, ToneFor(confidence), $"{confidence:0.000}"), + UiElement.Text("mimir-sync-counts", $"states {snapshot.SyncStates.Count} reports {snapshot.SyncReports.Count} decodes {snapshot.SyncDecodeAttempts.Count}", "caption"), + UiElement.Text("mimir-sync-worst", WorstSyncLine(snapshot), "caption"), + ]), + UiElement.Pane("mimir-latency-signal", "Latency", [ + StatBar("mimir-latency-score", "headroom", latencyScore, ToneFor(latencyScore), well == null ? "none" : $"{well.LatencyCurrentMs:0}ms"), + UiElement.Text("mimir-latency-line", well == null ? "no Well snapshot" : $"floor {well.LatencyFloorMs:0}ms ceiling {well.LatencyCeilingMs:0}ms skew {well.LatencyEdgeSkewMs:0.0}ms", "mono"), + UiElement.Text("mimir-presentation-delay", well == null ? "" : $"presentation delay {well.PresentationDelayMs:0}ms overlap {well.LatencyRetainedOverlapMs:0}ms", "caption"), + ]), + UiElement.Pane("mimir-throughput-signal", "Throughput", [ + StatBar("mimir-throughput-score", "publish/poll", throughputScore, ToneFor(throughputScore)), + UiElement.Text("mimir-throughput-line", well == null ? $"ingested {snapshot.Telemetry?.Ingested ?? 0:0}" : $"poll {well.PollAverageMs:0.000}/{well.PollMaxMs:0.000}ms publish {well.PublishAverageMs:0.000}/{well.PublishMaxMs:0.000}ms", "mono"), + UiElement.Text("mimir-throughput-bytes", well == null ? $"analyze {snapshot.Telemetry?.AnalyzeMs ?? 0:0.0}ms" : $"bytes {well.PublishedBytes:0} buffers {well.Buffers.Count} audio {well.AudioBuffers} video {well.VideoBuffers}", "caption"), + ]), + UiElement.Pane("mimir-dropout-signal", "Dropout", [ + StatBar("mimir-dropout-score", "liveness", dropoutScore, ToneFor(dropoutScore), $"{liveObservationCount}/{snapshot.ObservationStreams.Count} streams"), + UiElement.Text("mimir-dropout-line", well == null ? $"stale streams {staleStreams}" : $"sources {well.LiveSources}/{well.ConfiguredSources} errors {well.SourceErrorCount} stale streams {staleStreams}", "mono"), + UiElement.Text("mimir-dropout-worst", WorstObservationLine(snapshot.ObservationStreams), "caption"), + ]), + }; + + return new DashboardSurface + { + Schema = "cultmesh.eve_surface.v0", + Id = "mimir.live.stats.surface", + Title = "Mimir Live Stats", + Root = UiElement.Container( + "mimir-live-stats-root", + "dashboard", + new DashboardUiLayout + { + Direction = "vertical", + Gap = 6, + Padding = 6, + Overflow = "scroll", + Grow = 4, + MinWidth = 96, + MinHeight = 56, + PreferredWidth = 156, + PreferredHeight = 128, + Priority = -80, + Density = "continuous", + ViewportMode = "continuous-metrics", + }, + children), + }; + } + + private static string WorstSyncLine(MimirLiveStatsSnapshot snapshot) + { + var worstState = snapshot.SyncStates + .OrderBy(static state => state.Confidence) + .FirstOrDefault(); + if (worstState != null) + { + return $"worst {worstState.ReferenceSourceId}->{worstState.SourceId} c={worstState.Confidence:0.000} delay {worstState.DelayMs:0.000}ms"; + } + + var worstReport = snapshot.SyncReports + .OrderBy(static report => report.Confidence) + .FirstOrDefault(); + return worstReport == null + ? "no sync state yet" + : $"report {worstReport.ReferenceSourceId}->{worstReport.SourceId} c={worstReport.Confidence:0.000} events {worstReport.Events}"; + } + + private static string WorstObservationLine(IReadOnlyList streams) + { + if (streams.Count == 0) + { + return "no observation streams"; + } + + var worst = streams + .OrderByDescending(static stream => stream.State == "active" ? 0 : 1) + .ThenByDescending(static stream => stream.AgeSeconds) + .First(); + return $"{worst.DeviceId}/{worst.StreamId} {worst.Kind} {worst.State} age {worst.AgeSeconds:0}s"; + } + + private static DashboardUiElement BuildAudioPane(IReadOnlyList spectra, MimirWellStat? well) + { + var wellAudio = well?.Buffers + .Where(static buffer => string.Equals(buffer.Kind, "Audio", StringComparison.OrdinalIgnoreCase)) + .OrderBy(static buffer => buffer.SourceId, StringComparer.Ordinal) + .Take(8) + .Select(buffer => UiElement.Text( + $"well-audio-{StableId(buffer.SourceId)}", + $"{buffer.SourceId}: {buffer.Count} blocks edge {NsAge(buffer.EdgeNs)}", + "caption")) + .ToArray() ?? []; + var items = spectra.Count == 0 && wellAudio.Length == 0 + ? [UiElement.Text("mimir-spectrum-empty", "no spectrum telemetry yet", "caption")] + : spectra + .OrderBy(static spectrum => spectrum.SourceId, StringComparer.Ordinal) + .Take(8) + .Select(spectrum => UiElement.Card( + $"spectrum-{StableId(spectrum.SourceId)}", + "audio-channel", + style: new DashboardUiStyle { Variant = "compact", Tone = ToneFor(NormalizeRms(spectrum.Rms)) }, + children: + [ + UiElement.Text($"spectrum-{StableId(spectrum.SourceId)}-label", $"{spectrum.SourceId} {Bar(NormalizeRms(spectrum.Rms), 18)}", "mono"), + UiElement.Text($"spectrum-{StableId(spectrum.SourceId)}-detail", $"rms {spectrum.Rms:0.000000} peak {spectrum.Peak:0.000000} floor {spectrum.NoiseFloorDb:0.0} dB", "caption"), + UiElement.Text($"spectrum-{StableId(spectrum.SourceId)}-peaks", spectrum.Peaks, "caption"), + ])) + .Concat(wellAudio) + .ToArray(); + + return UiElement.Pane("mimir-audio-pane", "Audio", items); + } + + private static DashboardUiElement BuildSyncPane(MimirLiveStatsSnapshot snapshot) + { + var children = new List(); + foreach (var state in snapshot.SyncStates.OrderBy(static state => state.SourceId, StringComparer.Ordinal).Take(8)) + { + children.Add(UiElement.Card( + $"sync-state-{StableId(state.SourceId)}", + "sync-state", + style: new DashboardUiStyle { Variant = "compact", Tone = ToneFor(state.Confidence) }, + children: + [ + UiElement.Text($"sync-state-{StableId(state.SourceId)}-head", $"{state.ReferenceSourceId}->{state.SourceId} {Bar(state.Confidence, 16)}", "mono"), + UiElement.Text($"sync-state-{StableId(state.SourceId)}-detail", $"delay {state.DelayUs:0.000} us / {state.DelayMs:0.000} ms sro {state.SroPpm:0.000} ppm", "caption"), + ])); + } + + foreach (var report in snapshot.SyncReports.OrderBy(static report => report.SourceId, StringComparer.Ordinal).Take(6)) + { + children.Add(UiElement.Text( + $"sync-report-{StableId(report.SourceId)}", + $"{report.Evidence} {report.ReferenceSourceId}->{report.SourceId} {report.DelayUs:0.000} us c={report.Confidence:0.000} events={report.Events}", + "caption")); + } + + foreach (var attempt in snapshot.SyncDecodeAttempts.OrderByDescending(static attempt => attempt.Confidence).ThenBy(static attempt => attempt.SourceId, StringComparer.Ordinal).Take(8)) + { + children.Add(UiElement.Text( + $"sync-decode-{StableId(attempt.ReferenceSourceId)}-{StableId(attempt.SourceId)}", + $"decode {attempt.Status} {attempt.ReferenceSourceId}->{attempt.SourceId} c={attempt.Confidence:0.000} ref={attempt.ReferenceEnergy:0.000} cand={attempt.CandidateEnergy:0.000} matched={attempt.Matched}", + "caption")); + } + + if (children.Count == 0) + { + children.Add(UiElement.Text("mimir-sync-empty", "no sync state yet", "caption")); + } + + return UiElement.Pane("mimir-sync-pane", "Sync Confidence", children); + } + + private static DashboardUiElement BuildVideoPane(MimirLiveStatsSnapshot snapshot) + { + var children = new List(); + if (snapshot.Telemetry != null) + { + children.Add(UiElement.Text( + "mimir-telemetry-line", + $"sources {snapshot.Telemetry.Sources} poll {snapshot.Telemetry.LastPoll} ingested {snapshot.Telemetry.Ingested} analyze {snapshot.Telemetry.AnalyzeMs:0.0} ms mode {snapshot.Telemetry.AudioSync}", + "mono")); + } + + children.AddRange(snapshot.VideoBuffers + .OrderBy(static video => video.SourceId, StringComparer.Ordinal) + .Take(8) + .Select(video => UiElement.Text( + $"video-{StableId(video.SourceId)}", + $"{video.SourceId}: {video.Count} frames {video.Latest}", + "caption"))); + + if (children.Count == 0) + { + children.Add(UiElement.Text("mimir-video-empty", "no video-buffer telemetry yet", "caption")); + } + + return UiElement.Pane("mimir-video-pane", "Runtime Buffers", children); + } + + private static DashboardUiElement BuildObservationPane(IReadOnlyList streams) + { + var children = streams.Count == 0 + ? [UiElement.Text("mimir-observations-empty", "no device observation streams", "caption")] + : streams + .OrderBy(static stream => stream.DeviceId, StringComparer.Ordinal) + .ThenBy(static stream => stream.StreamId, StringComparer.Ordinal) + .Select(stream => UiElement.Card( + $"observation-{StableId(stream.StreamId)}-{StableId(stream.Kind)}", + "observation-stream", + style: new DashboardUiStyle { Variant = "compact", Tone = stream.State == "active" ? "cool" : "warm" }, + children: + [ + UiElement.Text($"observation-{StableId(stream.StreamId)}-head", $"{stream.StreamId} {stream.Kind} {stream.State}", "mono"), + UiElement.Text($"observation-{StableId(stream.StreamId)}-detail", $"{stream.Shape} seq {stream.Sequence} age {stream.AgeSeconds:0}s", "caption"), + ])) + .ToArray(); + + return UiElement.Pane("mimir-observation-pane", "Periwinkle / Eve Streams", children); + } + + private static DashboardUiElement BuildActuatorPane(IReadOnlyList commands) + { + var children = commands.Count == 0 + ? [UiElement.Text("mimir-actuator-empty", "no actuator commands yet", "caption")] + : commands + .OrderBy(static command => command.SourceId, StringComparer.Ordinal) + .Take(8) + .Select(command => UiElement.Text( + $"actuator-{StableId(command.SourceId)}", + $"{command.SourceId}: delay {command.DelaySamples:0.000} samples ratio {command.Ratio:0.000000000} c={command.Confidence:0.000}", + "caption")) + .ToArray(); + + return UiElement.Pane("mimir-actuator-pane", "Alignment Actuator", children); + } + + private static DashboardUiElement BuildMoveMusicPane(MimirMoveMusicStat? music) + { + if (music == null) + { + return UiElement.Pane("mimir-move-music-pane", "Move Score", [UiElement.Text("mimir-move-music-empty", "no Move score trace yet", "caption")]); + } + + var children = new List + { + UiElement.Text("mimir-move-music-head", $"bpm {music.Bpm:0.0} c={music.BpmConfidence:0.00} {music.KeyName} {music.KeyMode} chord {music.ChordName}", "mono"), + StatBar("mimir-move-loudness", "loudness gate", music.LoudnessGate, "warm", $"{music.LoudnessGate:0.000}"), + UiElement.Text("mimir-move-score", $"score {Bar(music.ScoreEnvelopeMax, 18)} max env {music.ScoreEnvelopeMax:0.000} max rgb {music.MaxRgb}", "caption"), + UiElement.Text("mimir-move-sources", string.Join(" | ", music.Sources.Take(4)), "caption"), + }; + children.AddRange(music.Voices + .Take(5) + .Select(voice => UiElement.Text( + $"mimir-move-voice-{StableId(voice.SourceId)}", + $"{voice.SourceId}: {voice.NoteName} {voice.FrequencyHz:0.0}Hz c={voice.Confidence:0.000} {voice.Role}", + "caption"))); + children.Add(UiElement.Text("mimir-move-piano-roll", music.PianoRoll, "mono")); + children.AddRange(music.MoveTargets + .Take(5) + .Select(target => UiElement.Text( + $"mimir-move-target-{target.MoveIndex}", + $"move {target.MoveIndex}: {target.NoteName} {target.SourceId} priority {target.CalibrationPriority:0.000}", + "caption"))); + + return UiElement.Pane("mimir-move-music-pane", "Move Score", children); + } + + private static DashboardUiElement BuildWellPane(MimirWellStat? well) + { + if (well == null) + { + return UiElement.Pane("mimir-well-pane", "Well", [UiElement.Text("mimir-well-empty", "no Well snapshot yet", "caption")]); + } + + var children = new List + { + UiElement.Text("mimir-well-summary", $"seq {well.Sequence} {well.ElapsedSeconds:0}s ingested {well.IngestedSamples:0} sources {well.LiveSources}/{well.ConfiguredSources} source errors {well.SourceErrorCount}", "mono"), + StatBar("mimir-well-completeness", "frame completeness", well.SynchronizedFrameComplete ? 1.0 : 0.0, well.SynchronizedFrameComplete ? "cool" : "danger"), + UiElement.Text("mimir-well-frame", $"frame ready {well.ReadySlices}/{well.TotalSlices} delay {well.PresentationDelayMs:0}ms degraded {well.FrameDegradedKind} {well.FrameDegradedReason}", "caption"), + UiElement.Text("mimir-well-latency", $"latency {well.LatencyCurrentMs:0}ms floor {well.LatencyFloorMs:0} ceiling {well.LatencyCeilingMs:0} overlap {well.LatencyRetainedOverlapMs:0}ms skew {well.LatencyEdgeSkewMs:0.0}ms {well.LatencyReason}", "caption"), + UiElement.Text("mimir-well-pressure", $"poll {well.PollAverageMs:0.000}/{well.PollMaxMs:0.000}ms zero {well.ZeroPollIterations} publish {well.PublishAverageMs:0.000}/{well.PublishMaxMs:0.000}ms bytes {well.PublishedBytes:0}", "caption"), + UiElement.Text("mimir-well-buffers", $"buffers {well.Buffers.Count} audio {well.AudioBuffers} video {well.VideoBuffers} capture bodies {well.CaptureInlineBodies}", "caption"), + UiElement.Text("mimir-well-clock-maps", $"canonical clocks {well.CanonicalClockMaps.Count} max offset {well.MaxCanonicalClockOffsetMs:0.000}ms", "caption"), + UiElement.Text("mimir-well-features", $"video features stable {well.FeatureStableTracks} c={well.FeatureMeanConfidence:0.000} motion {well.FeatureMeanMotionPixelsPerSecond:0.0}px/s faust signals {well.FeatureSignals.Count}", "caption"), + UiElement.Text("mimir-well-probe", well.Probe == null ? "probe: none" : $"probe emit={well.Probe.ShouldEmit} sync={well.Probe.SyncConfidence:0.000} freq={well.Probe.FrequencyConfidence:0.000} {well.Probe.Reason}", "caption"), + }; + children.AddRange(well.SourceHealth + .OrderBy(static source => source.Kind, StringComparer.Ordinal) + .ThenBy(static source => source.SourceId, StringComparer.Ordinal) + .Take(10) + .Select(source => UiElement.Text( + $"mimir-well-source-health-{StableId(source.SourceId)}", + $"{source.SourceId}: {source.Kind} {source.Status} samples {source.Samples} age {source.LatestAgeMs:0}ms {source.Reason}", + source.Status == "live" ? "caption" : "danger"))); + children.AddRange(well.FeatureSignals + .OrderByDescending(static signal => signal.Confidence) + .ThenBy(static signal => signal.SourceId, StringComparer.Ordinal) + .Take(5) + .Select(signal => UiElement.Text( + $"mimir-well-feature-{StableId(signal.SourceId)}", + $"{signal.SourceId}: tracks {signal.StableTrackCount} c={signal.Confidence:0.000} motion {signal.MotionEnergy:0.000} centroid {signal.CentroidX:0.00},{signal.CentroidY:0.00}", + "caption"))); + children.AddRange(well.ClockDomains + .Take(6) + .Select(domain => UiElement.Text( + $"mimir-well-clock-{StableId(domain.ClockDomainId)}", + $"{domain.ClockDomainId}: {domain.SourceCount} src overlap {domain.OverlapMs:0.000}ms offset {domain.OffsetMs:0.000}ms {(domain.IsReference ? "ref" : "")}", + "caption"))); + children.AddRange(well.CanonicalClockMaps + .OrderByDescending(static map => Math.Abs(map.OffsetMs)) + .ThenBy(static map => map.StreamKey, StringComparer.Ordinal) + .Take(4) + .Select(map => UiElement.Text( + $"mimir-well-clock-map-{StableId(map.StreamKey)}", + $"{map.StreamKey}: ingress offset {map.OffsetMs:0.000}ms samples {map.SampleCount}", + "caption"))); + children.AddRange(well.VisualCalibration + .Take(6) + .Select(camera => UiElement.Text( + $"mimir-well-cal-{StableId(camera.SourceId)}", + $"{camera.SourceId}: score {camera.BestScore:0.000} leds {camera.BestDetectedLedCount} exp {camera.BestExposure:0} gain {camera.BestGain:0} {camera.State}", + "caption"))); + return UiElement.Pane("mimir-well-pane", "Well", children); + } + + private static DashboardUiElement BuildCalibrationPane(MimirLiveStatsSnapshot snapshot) + { + var leap = snapshot.Well?.Buffers.FirstOrDefault(static buffer => buffer.SourceId.Contains("leap", StringComparison.OrdinalIgnoreCase)); + var eyeStreams = snapshot.ObservationStreams + .Where(static stream => stream.StreamId.Contains("eye", StringComparison.OrdinalIgnoreCase) || stream.StreamId.Contains("video2", StringComparison.OrdinalIgnoreCase) || stream.StreamId.Contains("video3", StringComparison.OrdinalIgnoreCase)) + .OrderBy(static stream => stream.StreamId, StringComparer.Ordinal) + .Take(4) + .ToArray(); + var moveStreams = snapshot.ObservationStreams + .Where(static stream => stream.Kind.Contains("move", StringComparison.OrdinalIgnoreCase)) + .OrderBy(static stream => stream.StreamId, StringComparer.Ordinal) + .Take(4) + .ToArray(); + + var children = new List + { + UiElement.Text("mimir-calibration-leap", leap == null ? "Leap: no Well buffer" : $"Leap: {leap.Count} frames {leap.PixelFormat} {leap.Width:0}x{leap.Height:0} edge {NsAge(leap.EdgeNs)}", "mono"), + UiElement.Text("mimir-calibration-eyes", $"Eyes: {eyeStreams.Count(static stream => stream.State == "active")}/{eyeStreams.Length} active", "caption"), + UiElement.Text("mimir-calibration-moves", $"Moves: {moveStreams.Count(static stream => stream.State == "active")}/{moveStreams.Length} active", "caption"), + }; + children.AddRange(eyeStreams.Select(stream => UiElement.Text($"mimir-eye-{StableId(stream.StreamId)}", $"{stream.StreamId} {stream.Shape} age {stream.AgeSeconds:0}s", "caption"))); + children.AddRange(moveStreams.Select(stream => UiElement.Text($"mimir-move-{StableId(stream.StreamId)}", $"{stream.StreamId} {stream.Shape} age {stream.AgeSeconds:0}s", "caption"))); + return UiElement.Pane("mimir-calibration-pane", "Leap / Eyes / Moves", children); + } + + private static DashboardUiElement StatBar(string id, string label, double value, string tone, string? displayValue = null) => + UiElement.Card( + id, + "stat-bar", + style: new DashboardUiStyle { Variant = "compact", Tone = tone }, + children: + [ + UiElement.Text($"{id}-label", $"{label}: {displayValue ?? value.ToString("0.000", CultureInfo.InvariantCulture)}", "caption"), + UiElement.Metric($"{id}-metric", label, value, tone), + ]); + + private static string Bar(double value, int width) + { + var clamped = Math.Clamp(value, 0.0, 1.0); + var filled = (int)Math.Round(clamped * width); + var pulseIndex = Math.Clamp((int)(DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() / 180 % Math.Max(1, width)), 0, Math.Max(0, width - 1)); + var builder = new StringBuilder(width); + for (var index = 0; index < width; index++) + { + builder.Append(index < filled ? (index == pulseIndex ? '*' : '#') : '.'); + } + + return builder.ToString(); + } + + private static string Pulse() => + (DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() / 250 % 4) switch + { + 0 => "|", + 1 => "/", + 2 => "-", + _ => "\\", + }; + + private static double NormalizeRms(double rms) => Math.Clamp(rms / 0.08, 0.0, 1.0); + + private static string ToneFor(double value) => + value >= 0.70 ? "cool" : value >= 0.25 ? "warm" : "danger"; + + private static string NsAge(double timestampNs) + { + if (timestampNs <= 0) + { + return "unknown"; + } + + var nowNs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000.0; + var seconds = Math.Max(0.0, (nowNs - timestampNs) / 1_000_000_000.0); + return seconds < 1.0 ? $"{seconds * 1000.0:0}ms" : $"{seconds:0.0}s"; + } + + private static string StableId(string value) => + string.Join("-", value.ToLowerInvariant().Split(Path.GetInvalidFileNameChars(), StringSplitOptions.RemoveEmptyEntries)) + .Replace(":", "-", StringComparison.Ordinal) + .Replace(">", "-", StringComparison.Ordinal) + .Replace(" ", "-", StringComparison.Ordinal); +} + +internal sealed class MimirLiveStatsSnapshot +{ + public string TelemetrySource { get; init; } = "none"; + public string ObservationSource { get; init; } = "none"; + public MimirTelemetryStat? Telemetry { get; init; } + public IReadOnlyList Spectra { get; init; } = []; + public IReadOnlyList SyncStates { get; init; } = []; + public IReadOnlyList SyncReports { get; init; } = []; + public IReadOnlyList SyncDecodeAttempts { get; init; } = []; + public IReadOnlyList VideoBuffers { get; init; } = []; + public IReadOnlyList ActuatorCommands { get; init; } = []; + public IReadOnlyList ObservationStreams { get; init; } = []; + public MimirWellStat? Well { get; init; } + public MimirMoveMusicStat? MoveMusic { get; init; } + public bool HasAnyData => Telemetry != null || Well != null || MoveMusic != null || Spectra.Count > 0 || SyncStates.Count > 0 || VideoBuffers.Count > 0 || ObservationStreams.Count > 0; + public string Summary => Well == null + ? $"{Spectra.Count} audio lanes / {SyncStates.Count} sync states / {SyncDecodeAttempts.Count} decode attempts / {VideoBuffers.Count} video buffers / {ObservationStreams.Count} device streams" + : $"Well {Well.LiveSources}/{Well.ConfiguredSources} sources / {Well.AudioBuffers} audio / {Well.VideoBuffers} video / {ObservationStreams.Count} device streams"; + + public static MimirLiveStatsSnapshot Load(string telemetryLogPath, string observationLogPath) + { + var telemetrySource = ResolveTelemetryLog(telemetryLogPath); + var telemetryLines = File.Exists(telemetrySource) ? TailTextFile(telemetrySource, 512 * 1024).Split("\n", StringSplitOptions.RemoveEmptyEntries) : []; + var observationLines = File.Exists(observationLogPath) ? TailTextFile(observationLogPath, 512 * 1024).Split("\n", StringSplitOptions.RemoveEmptyEntries) : []; + return new MimirLiveStatsSnapshot + { + TelemetrySource = string.IsNullOrWhiteSpace(telemetrySource) ? "none" : telemetrySource, + ObservationSource = File.Exists(observationLogPath) ? observationLogPath : $"missing {observationLogPath}", + Telemetry = telemetryLines.Select(ParseTelemetry).Where(static value => value != null).LastOrDefault(), + Spectra = LatestBySource(telemetryLines.Select(ParseSpectrum).Where(static value => value != null)!), + SyncStates = LatestBySource(telemetryLines.Select(ParseSyncState).Where(static value => value != null)!), + SyncReports = LatestBySource(telemetryLines.Select(ParseSyncReport).Where(static value => value != null)!), + SyncDecodeAttempts = LatestSyncDecodeAttempts(telemetryLines.Select(ParseSyncDecodeAttempt).Where(static value => value != null)!), + VideoBuffers = LatestBySource(telemetryLines.Select(ParseVideoBuffer).Where(static value => value != null)!), + ActuatorCommands = LatestBySource(telemetryLines.Select(ParseActuatorCommand).Where(static value => value != null)!), + ObservationStreams = LatestObservationStreams(observationLines), + Well = LatestWellSnapshot(telemetryLines.Concat(observationLines)), + MoveMusic = LatestMoveMusicTrace(telemetrySource, telemetryLines), + }; + } + + private static string ResolveTelemetryLog(string configured) + { + if (!string.IsNullOrWhiteSpace(configured)) + { + return configured; + } + + var runtimeDir = Path.Combine(AppContext.BaseDirectory, "..", "..", "..", "..", "..", "artifacts", "runtime"); + runtimeDir = Path.GetFullPath(runtimeDir); + if (!Directory.Exists(runtimeDir)) + { + runtimeDir = @"E:\Projects\Mimir\artifacts\runtime"; + } + + if (!Directory.Exists(runtimeDir)) + { + return ""; + } + + foreach (var file in Directory.EnumerateFiles(runtimeDir, "*.out.log") + .OrderByDescending(File.GetLastWriteTimeUtc) + .Take(32)) + { + var text = TailTextFile(file, 128 * 1024); + if (text.Contains("mimir-sync-telemetry", StringComparison.Ordinal) || + text.Contains("mimir-spectrum ", StringComparison.Ordinal) || + text.Contains("mimir-video-buffer ", StringComparison.Ordinal)) + { + return file; + } + } + + return ""; + } + + private static IReadOnlyList LatestBySource(IEnumerable values) + where T : class, IMimirSourceStat + { + var bySource = new Dictionary(StringComparer.Ordinal); + foreach (var value in values) + { + if (value == null) + { + continue; + } + + bySource[value.SourceId] = value; + } + + return bySource.Values.OrderBy(static value => value.SourceId, StringComparer.Ordinal).ToArray(); + } + + private static IReadOnlyList LatestSyncDecodeAttempts(IEnumerable values) + { + var byRoute = new Dictionary(StringComparer.Ordinal); + foreach (var value in values) + { + if (value == null) + { + continue; + } + + byRoute[$"{value.ReferenceSourceId}->{value.SourceId}"] = value; + } + + return byRoute.Values + .OrderByDescending(static value => value.Confidence) + .ThenBy(static value => value.SourceId, StringComparer.Ordinal) + .ToArray(); + } + + private static MimirWellStat? LatestWellSnapshot(IEnumerable lines) + { + MimirWellStat? latest = null; + foreach (var line in lines) + { + try + { + using var document = JsonDocument.Parse(line); + var root = document.RootElement; + if (string.Equals(JsonStringAny(root, "document"), "mimir.well_snapshot.v1", StringComparison.Ordinal)) + { + latest = ParseWellSnapshot(root); + } + } + catch (JsonException) + { + } + } + + return latest; + } + + private static MimirWellStat ParseWellSnapshot(JsonElement root) + { + var buffers = ArrayItems(TryGetAny(root, "buffers")) + .Select(static buffer => + { + var latest = TryGetAny(buffer, "latest"); + var video = TryGetAny(latest, "video"); + var audio = TryGetAny(latest, "audio"); + return new MimirWellBufferStat( + JsonStringAny(buffer, "SourceId", "sourceId") ?? "unknown", + JsonStringAny(buffer, "Kind", "kind") ?? "unknown", + (int)JsonNumberAny(buffer, "Count", "count"), + JsonNumberAny(buffer, "EdgeNs", "edgeNs"), + (int)JsonNumberAny(video, "Width", "width"), + (int)JsonNumberAny(video, "Height", "height"), + JsonStringAny(video, "PixelFormat", "pixelFormat") ?? "", + (int)JsonNumberAny(audio, "SampleRate", "sampleRate"), + (int)JsonNumberAny(audio, "Channels", "channels")); + }) + .ToArray(); + var frame = TryGetAny(root, "synchronizedFrame"); + var slices = ArrayItems(TryGetAny(frame, "slices")).ToArray(); + var audioSync = TryGetAny(root, "audioSync"); + var states = ArrayItems(TryGetAny(audioSync, "states")) + .Select(static state => new MimirWellAudioSyncStateStat( + JsonStringAny(state, "SourceId", "sourceId") ?? "unknown", + JsonStringAny(state, "ReferenceSourceId", "referenceSourceId") ?? "unknown", + JsonNumberAny(state, "SmoothedDelaySamples", "smoothedDelaySamples"), + JsonNumberAny(state, "SamplingRateOffsetPpm", "samplingRateOffsetPpm"), + JsonNumberAny(state, "Confidence", "confidence"))) + .ToArray(); + var probeElement = TryGetAny(audioSync, "probe"); + var probe = probeElement.ValueKind == JsonValueKind.Object + ? new MimirWellProbeStat( + JsonBoolAny(probeElement, "ShouldEmit", "shouldEmit"), + JsonNumberAny(probeElement, "AggregateSyncConfidence", "aggregateSyncConfidence"), + JsonNumberAny(probeElement, "AggregateFrequencyResponseConfidence", "aggregateFrequencyResponseConfidence"), + JsonStringAny(probeElement, "reason") ?? "unknown") + : null; + var streamPressure = TryGetAny(root, "streamPressure"); + var pollPressure = TryGetAny(streamPressure, "poll"); + var publishPressure = TryGetAny(streamPressure, "publish"); + var latency = TryGetAny(root, "latency"); + var featureSignals = TryGetAny(root, "featureSignals"); + var featureSignalStats = ArrayItems(TryGetAny(featureSignals, "signals")) + .Select(static signal => new MimirWellFeatureSignalStat( + JsonStringAny(signal, "SourceId", "sourceId") ?? "unknown", + (int)JsonNumberAny(signal, "StableTrackCount", "stableTrackCount"), + JsonNumberAny(signal, "Confidence", "confidence"), + JsonNumberAny(signal, "MeanMotionPixelsPerSecond", "meanMotionPixelsPerSecond"), + JsonNumberAny(signal, "MotionEnergy", "motionEnergy"), + JsonNumberAny(signal, "NormalizedCentroidX", "normalizedCentroidX"), + JsonNumberAny(signal, "NormalizedCentroidY", "normalizedCentroidY"))) + .ToArray(); + var canonicalClockMaps = ArrayItems(TryGetAny(root, "canonicalClockMaps")) + .Select(static map => new MimirWellCanonicalClockMapStat( + JsonStringAny(map, "StreamKey", "streamKey") ?? "unknown", + JsonNumberAny(map, "offsetMs"), + (long)JsonNumberAny(map, "SampleCount", "sampleCount"))) + .ToArray(); + var clockDomains = ArrayItems(TryGetAny(TryGetAny(root, "clockDomains"), "domains")) + .Select(static domain => new MimirWellClockDomainStat( + JsonStringAny(domain, "ClockDomainId", "clockDomainId") ?? "unknown", + (int)JsonNumberAny(domain, "SourceCount", "sourceCount"), + JsonNumberAny(domain, "OverlapNs", "overlapNs") / 1_000_000.0, + JsonBoolAny(domain, "HasLocalOverlap", "hasLocalOverlap"), + JsonBoolAny(domain, "IsReferenceDomain", "isReferenceDomain"), + JsonNumberAny(domain, "ProvisionalOffsetToReferenceMs", "provisionalOffsetToReferenceMs"))) + .ToArray(); + var visualCalibration = ArrayItems(TryGetAny(TryGetAny(root, "visualCalibration"), "cameras")) + .Select(static camera => new MimirWellVisualCalibrationStat( + JsonStringAny(camera, "SourceId", "sourceId") ?? "unknown", + JsonStringAny(camera, "State", "state") ?? "unknown", + JsonNumberAny(camera, "BestScore", "bestScore"), + (int)JsonNumberAny(camera, "BestDetectedLedCount", "bestDetectedLedCount"), + JsonBoolAny(camera, "BestUsableForCalibration", "bestUsableForCalibration"), + JsonNumberAny(camera, "BestExposure", "bestExposure"), + JsonNumberAny(camera, "BestGain", "bestGain"))) + .ToArray(); + var captureStorage = TryGetAny(TryGetAny(root, "capture"), "storage"); + var publication = TryGetAny(root, "publication"); + var stems = ArrayItems(TryGetAny(publication, "stems")).ToArray(); + var sourceHealth = ArrayItems(TryGetAny(root, "sourceHealth")) + .Select(static source => new MimirWellSourceHealthStat( + JsonStringAny(source, "SourceId", "sourceId") ?? "unknown", + JsonStringAny(source, "Kind", "kind") ?? "unknown", + JsonStringAny(source, "Origin", "origin") ?? "unknown", + JsonStringAny(source, "Label", "label") ?? "", + JsonStringAny(source, "status") ?? "unknown", + JsonStringAny(source, "reason") ?? "", + (int)JsonNumberAny(source, "buffers"), + (int)JsonNumberAny(source, "samples"), + JsonNumberAny(source, "latestAgeMs"), + (int)JsonNumberAny(source, "latestByteLength"))) + .ToArray(); + return new MimirWellStat( + (long)JsonNumberAny(root, "sequence"), + JsonNumberAny(root, "elapsedSeconds"), + JsonNumberAny(root, "ingestedSamples"), + (int)JsonNumberAny(root, "configuredSources"), + (int)JsonNumberAny(root, "liveSources"), + ArrayItems(TryGetAny(root, "sourceErrors")).Count(), + JsonBoolAny(frame, "IsComplete", "isComplete"), + JsonNumberAny(frame, "PresentationDelayMs", "presentationDelayMs"), + JsonStringAny(frame, "degradedKind") ?? "", + JsonStringAny(frame, "degradedReason") ?? "", + slices.Count(static slice => string.Equals(JsonStringAny(slice, "Status", "status"), "Ready", StringComparison.OrdinalIgnoreCase)), + slices.Length, + JsonNumberAny(latency, "currentDelayMs"), + JsonNumberAny(latency, "ceilingDelayMs"), + JsonNumberAny(latency, "floorDelayMs"), + JsonNumberAny(latency, "retainedOverlapMs"), + JsonNumberAny(latency, "edgeSkewMs"), + JsonStringAny(latency, "Reason", "reason") ?? "", + JsonNumberAny(pollPressure, "averageMilliseconds"), + JsonNumberAny(pollPressure, "maxMilliseconds"), + (long)JsonNumberAny(pollPressure, "zeroPollIterations"), + JsonNumberAny(publishPressure, "averageMilliseconds"), + JsonNumberAny(publishPressure, "maxMilliseconds"), + (long)JsonNumberAny(publishPressure, "bytes"), + JsonStringAny(captureStorage, "bodyTransport") ?? "unknown", + stems.Length, + sourceHealth, + buffers, + clockDomains, + states, + visualCalibration, + JsonNumberAny(featureSignals, "MeanConfidence", "meanConfidence"), + JsonNumberAny(featureSignals, "MeanMotionPixelsPerSecond", "meanMotionPixelsPerSecond"), + (int)JsonNumberAny(featureSignals, "StableTrackCount", "stableTrackCount"), + featureSignalStats, + canonicalClockMaps, + probe); + } + + private static MimirMoveMusicStat? LatestMoveMusicTrace(string telemetrySource, IReadOnlyList telemetryLines) + { + var configuredLine = telemetryLines + .Reverse() + .FirstOrDefault(static line => line.Contains("\"live_score\"", StringComparison.OrdinalIgnoreCase) || + line.Contains("\"score_gesture_envelopes\"", StringComparison.OrdinalIgnoreCase)); + if (!string.IsNullOrWhiteSpace(configuredLine)) + { + return ParseMoveMusicTrace(telemetrySource, configuredLine); + } + + return LatestMoveMusicTraceFromRuntime(); + } + + private static MimirMoveMusicStat? LatestMoveMusicTraceFromRuntime() + { + var runtimeRoot = Path.Combine(AppContext.BaseDirectory, "artifacts", "runtime"); + if (!Directory.Exists(runtimeRoot)) + { + runtimeRoot = @"E:\Projects\Mimir\artifacts\runtime"; + } + + if (!Directory.Exists(runtimeRoot)) + { + return null; + } + + var trace = Directory.EnumerateFiles(runtimeRoot, "online-sync.jsonl", SearchOption.AllDirectories) + .Select(static path => new FileInfo(path)) + .Where(static info => info.DirectoryName?.Contains("move", StringComparison.OrdinalIgnoreCase) == true) + .OrderByDescending(static info => info.LastWriteTimeUtc) + .FirstOrDefault(); + if (trace == null || trace.Length == 0) + { + return null; + } + + foreach (var line in TailTextFile(trace.FullName, 4 * 1024 * 1024) + .Split("\n", StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .Reverse() + .Where(static candidate => candidate.Contains("\"live_score\"", StringComparison.OrdinalIgnoreCase) || + candidate.Contains("\"score_gesture_envelopes\"", StringComparison.OrdinalIgnoreCase))) + { + var parsed = ParseMoveMusicTrace(trace.FullName, line); + if (parsed != null) + { + return parsed; + } + } + + return null; + } + + private static MimirMoveMusicStat? ParseMoveMusicTrace(string sourcePath, string line) + { + try + { + using var document = JsonDocument.Parse(line); + var root = document.RootElement; + var maxRgb = 0; + var moves = TryGetAny(root, "moves"); + if (moves.ValueKind == JsonValueKind.Object) + { + foreach (var move in moves.EnumerateObject()) + { + if (move.Value.ValueKind == JsonValueKind.Array) + { + maxRgb = Math.Max(maxRgb, move.Value.EnumerateArray().Select(JsonNumber).Select(static value => (int)value).DefaultIfEmpty(0).Max()); + } + } + } + + var envelopes = ArrayItems(TryGetAny(root, "score_gesture_envelopes")).Select(JsonNumber).DefaultIfEmpty(0.0).ToArray(); + var liveScore = TryGetAny(root, "live_score"); + var voices = ArrayItems(TryGetAny(liveScore, "voices")) + .Select(static voice => new MimirMoveMusicVoiceStat( + JsonStringAny(voice, "source") ?? "unknown", + JsonStringAny(voice, "role") ?? "voice", + JsonStringAny(voice, "note_name") ?? "?", + JsonNumberAny(voice, "frequency_hz"), + JsonNumberAny(voice, "midi"), + JsonNumberAny(voice, "confidence"), + JsonNumberAny(voice, "strength"), + JsonBoolAny(voice, "active"))) + .OrderByDescending(static voice => voice.Active) + .ThenByDescending(static voice => voice.Confidence) + .ToArray(); + var targets = ArrayItems(TryGetAny(liveScore, "move_targets")) + .Select(static target => new MimirMoveMusicTargetStat( + (int)JsonNumberAny(target, "move_index"), + JsonStringAny(target, "source") ?? "unknown", + JsonStringAny(target, "note_name") ?? "?", + JsonNumberAny(target, "target_note"), + JsonNumberAny(target, "confidence"), + JsonNumberAny(target, "calibration_priority"))) + .OrderBy(static target => target.MoveIndex) + .ToArray(); + var sources = ArrayItems(TryGetAny(root, "music_sources")) + .Select(static source => + { + var name = JsonStringAny(source, "source") ?? "source"; + var strength = JsonNumberAny(source, "score_strength", "onset", "hit", "loudness_gate"); + var key = JsonStringAny(source, "key_name") ?? ""; + var mode = JsonStringAny(source, "key_mode") ?? ""; + var harmonic = string.IsNullOrWhiteSpace(key) + ? $"f0 {JsonNumberAny(source, "fundamental_hz"):0}" + : $"{key} {mode}".Trim(); + return $"{name} {JsonNumberAny(source, "bpm"):0.0}/{JsonNumberAny(source, "bpm_confidence"):0.00} s={strength:0.00} {harmonic}"; + }) + .ToArray(); + return new MimirMoveMusicStat( + sourcePath, + JsonNumberAny(root, "bpm"), + JsonNumberAny(root, "bpm_confidence"), + JsonNumberAny(root, "loudness_gate"), + JsonNumberAny(root, "loudness_percentile"), + JsonStringAny(root, "key_name") ?? "?", + JsonStringAny(root, "key_mode") ?? "?", + JsonStringAny(root, "chord_name") ?? "?", + envelopes.Max(), + maxRgb, + sources, + voices, + targets, + BuildPianoRoll(voices, targets)); + } + catch (JsonException) + { + return null; + } + } + + private static string BuildPianoRoll( + IReadOnlyList voices, + IReadOnlyList targets) + { + var marks = new char[36]; + Array.Fill(marks, '.'); + foreach (var voice in voices.Where(static voice => voice.Active)) + { + var index = PianoIndex(voice.MidiNote); + if (index >= 0) + { + marks[index] = voice.Confidence >= 0.65 ? '#' : '+'; + } + } + + foreach (var target in targets) + { + var index = PianoIndex(target.TargetNote); + if (index >= 0 && marks[index] == '.') + { + marks[index] = 'o'; + } + } + + return $"C3 {new string(marks)} C6"; + } + + private static int PianoIndex(double midiNote) + { + if (midiNote <= 0.0) + { + return -1; + } + + return (int)Math.Clamp(Math.Round(midiNote) - 48, 0, 35); + } + + private static IReadOnlyList LatestObservationStreams(IEnumerable lines) + { + var byStream = new Dictionary(StringComparer.Ordinal); + var now = DateTimeOffset.UtcNow; + foreach (var line in lines) + { + try + { + using var document = JsonDocument.Parse(line); + var root = document.RootElement; + var type = JsonString(root, "type"); + if (type is not ("cultmesh-observation" or "cultmesh-media-observation")) + { + continue; + } + + var deviceId = JsonString(root, "DeviceId") ?? "unknown"; + var streamId = JsonString(root, "StreamId") ?? "unknown"; + var kind = JsonString(root, "Kind") ?? "unknown"; + var latestAt = DateTimeOffset.TryParse(JsonString(root, "WallClockUtc"), out var parsed) ? parsed : DateTimeOffset.MinValue; + var age = latestAt == DateTimeOffset.MinValue ? double.PositiveInfinity : Math.Max(0.0, (now - latestAt).TotalSeconds); + var shape = ObservationShape(root); + byStream[$"{deviceId}:{streamId}:{kind}"] = new MimirObservationStreamStat( + deviceId, + streamId, + kind, + JsonNumber(root, "Sequence"), + age <= 120.0 ? "active" : "stale", + age, + shape); + } + catch (JsonException) + { + } + } + + return byStream.Values.ToArray(); + } + + private static string ObservationShape(JsonElement root) + { + var format = JsonString(root, "Format") ?? ""; + var width = JsonNumber(root, "Width"); + var height = JsonNumber(root, "Height"); + var sampleRate = JsonNumber(root, "SampleRate"); + var channels = JsonNumber(root, "Channels"); + var frames = JsonNumber(root, "FrameCount"); + var bytes = JsonNumber(root, "PayloadBytes"); + if (width > 0 && height > 0) + { + return $"{format} {width:0}x{height:0} {bytes:0} bytes"; + } + + if (sampleRate > 0) + { + return $"{format} {sampleRate:0} Hz x {Math.Max(1, channels):0} {frames:0} frames {bytes:0} bytes"; + } + + if (root.TryGetProperty("Values", out var values) && values.ValueKind == JsonValueKind.Array) + { + return string.Join(", ", values.EnumerateArray().Take(3).Select(static value => value.TryGetDouble(out var number) ? number.ToString("0.000", CultureInfo.InvariantCulture) : "?")); + } + + return "observation"; + } + + private static MimirTelemetryStat? ParseTelemetry(string line) + { + if (!line.StartsWith("mimir-sync-telemetry ", StringComparison.Ordinal)) + { + return null; + } + + var fields = Fields(line); + return new MimirTelemetryStat( + Number(fields, "t"), + (int)Number(fields, "sources"), + (int)Number(fields, "lastPoll"), + Number(fields, "ingested"), + Text(fields, "audioSync"), + (int)Number(fields, "reports"), + (int)Number(fields, "states"), + Number(fields, "analyzeMs")); + } + + private static MimirSpectrumStat? ParseSpectrum(string line) + { + if (!line.StartsWith("mimir-spectrum ", StringComparison.Ordinal)) + { + return null; + } + + var sourceId = TokenAfterPrefix(line, "mimir-spectrum "); + var fields = Fields(line); + return new MimirSpectrumStat( + sourceId, + Text(fields, "label"), + (int)Number(fields, "rate"), + (int)Number(fields, "fft"), + Number(fields, "rms"), + Number(fields, "peak"), + Number(fields, "floorDb"), + Text(fields, "peaks")); + } + + private static MimirSyncStateStat? ParseSyncState(string line) + { + if (!line.StartsWith("mimir-sync-state ", StringComparison.Ordinal)) + { + return null; + } + + var route = TokenAfterPrefix(line, "mimir-sync-state "); + var parts = route.Split("->", 2, StringSplitOptions.TrimEntries); + if (parts.Length != 2) + { + return null; + } + + var fields = Fields(line); + return new MimirSyncStateStat( + parts[1], + parts[0], + Number(fields, "delaySamples"), + Number(fields, "delayUs"), + Number(fields, "delayMs"), + Number(fields, "sroPpm"), + Number(fields, "confidence")); + } + + private static MimirSyncReportStat? ParseSyncReport(string line) + { + if (!line.StartsWith("mimir-sync-report ", StringComparison.Ordinal) && + !line.StartsWith("mimir-complex-contour-report ", StringComparison.Ordinal)) + { + return null; + } + + var prefix = line.StartsWith("mimir-sync-report ", StringComparison.Ordinal) ? "mimir-sync-report " : "mimir-complex-contour-report "; + var route = TokenAfterPrefix(line, prefix); + var parts = route.Split("->", 2, StringSplitOptions.TrimEntries); + if (parts.Length != 2) + { + return null; + } + + var fields = Fields(line); + return new MimirSyncReportStat( + parts[1], + parts[0], + Text(fields, "evidence", prefix.Contains("complex", StringComparison.Ordinal) ? "complex-contour" : "sync"), + Number(fields, "delayUs"), + Number(fields, "confidence"), + (int)Number(fields, "timelineEvents") + (int)Number(fields, "directHits")); + } + + private static MimirSyncDecodeAttemptStat? ParseSyncDecodeAttempt(string line) + { + if (!line.StartsWith("mimir-sync-decode ", StringComparison.Ordinal)) + { + return null; + } + + var route = TokenAfterPrefix(line, "mimir-sync-decode "); + var parts = route.Split("->", 2, StringSplitOptions.TrimEntries); + if (parts.Length != 2) + { + return null; + } + + var fields = Fields(line); + return new MimirSyncDecodeAttemptStat( + parts[1], + parts[0], + Text(fields, "status", "unknown"), + (int)Number(fields, "compared"), + (int)Number(fields, "rate"), + Number(fields, "refEnergy"), + Number(fields, "candEnergy"), + (int)Number(fields, "matched"), + Number(fields, "confidence")); + } + + private static MimirVideoBufferStat? ParseVideoBuffer(string line) + { + if (!line.StartsWith("mimir-video-buffer ", StringComparison.Ordinal)) + { + return null; + } + + var sourceId = TokenAfterPrefix(line, "mimir-video-buffer "); + var fields = Fields(line); + var latestIndex = line.IndexOf(" latest=", StringComparison.Ordinal); + return new MimirVideoBufferStat( + sourceId, + (int)Number(fields, "count"), + latestIndex < 0 ? "" : line[(latestIndex + " latest=".Length)..]); + } + + private static MimirActuatorCommandStat? ParseActuatorCommand(string line) + { + if (!line.StartsWith("mimir-audio-actuator-command ", StringComparison.Ordinal)) + { + return null; + } + + var fields = Fields(line); + var source = Text(fields, "source"); + return string.IsNullOrWhiteSpace(source) + ? null + : new MimirActuatorCommandStat(source, Number(fields, "delaySamples"), Number(fields, "ratio"), Number(fields, "confidence")); + } + + private static Dictionary Fields(string line) + { + var fields = new Dictionary(StringComparer.Ordinal); + foreach (var token in Tokenize(line)) + { + var equals = token.IndexOf('='); + if (equals <= 0) + { + continue; + } + + fields[token[..equals]] = token[(equals + 1)..].Trim('"'); + } + + return fields; + } + + private static IEnumerable Tokenize(string line) + { + var builder = new StringBuilder(); + var quoted = false; + foreach (var character in line) + { + if (character == '"') + { + quoted = !quoted; + builder.Append(character); + continue; + } + + if (char.IsWhiteSpace(character) && !quoted) + { + if (builder.Length > 0) + { + yield return builder.ToString(); + builder.Clear(); + } + + continue; + } + + builder.Append(character); + } + + if (builder.Length > 0) + { + yield return builder.ToString(); + } + } + + private static string TokenAfterPrefix(string line, string prefix) + { + var rest = line[prefix.Length..]; + var space = rest.IndexOf(' '); + return space < 0 ? rest : rest[..space]; + } + + private static string Text(IReadOnlyDictionary fields, string key, string fallback = "") => + fields.TryGetValue(key, out var value) ? value : fallback; + + private static double Number(IReadOnlyDictionary fields, string key) => + fields.TryGetValue(key, out var value) && double.TryParse(value, NumberStyles.Float, CultureInfo.InvariantCulture, out var number) + ? number + : 0.0; + + private static string? JsonString(JsonElement root, string property) => + root.TryGetProperty(property, out var child) && child.ValueKind == JsonValueKind.String ? child.GetString() : null; + + private static double JsonNumber(JsonElement root, string property) => + root.TryGetProperty(property, out var child) && child.ValueKind == JsonValueKind.Number && child.TryGetDouble(out var value) ? value : 0.0; + + private static JsonElement TryGetAny(JsonElement root, params string[] properties) + { + if (root.ValueKind != JsonValueKind.Object) + { + return default; + } + + foreach (var property in properties) + { + if (root.TryGetProperty(property, out var child)) + { + return child; + } + } + + return default; + } + + private static IEnumerable ArrayItems(JsonElement element) + { + if (element.ValueKind != JsonValueKind.Array) + { + yield break; + } + + foreach (var item in element.EnumerateArray()) + { + yield return item; + } + } + + private static string? JsonStringAny(JsonElement root, params string[] properties) + { + var child = TryGetAny(root, properties); + return child.ValueKind == JsonValueKind.String ? child.GetString() : null; + } + + private static double JsonNumberAny(JsonElement root, params string[] properties) => JsonNumber(TryGetAny(root, properties)); + + private static double JsonNumber(JsonElement element) => + element.ValueKind == JsonValueKind.Number && element.TryGetDouble(out var value) ? value : 0.0; + + private static bool JsonBoolAny(JsonElement root, params string[] properties) + { + var child = TryGetAny(root, properties); + return child.ValueKind == JsonValueKind.True || (child.ValueKind == JsonValueKind.Number && child.TryGetDouble(out var value) && value != 0.0); + } + + private static string NsAge(double timestampNs) + { + if (timestampNs <= 0) + { + return "unknown"; + } + + var nowNs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000.0; + var seconds = Math.Max(0.0, (nowNs - timestampNs) / 1_000_000_000.0); + return seconds < 1.0 ? $"{seconds * 1000.0:0}ms" : $"{seconds:0.0}s"; + } + + private static string TailTextFile(string filePath, int maxBytes) + { + var info = new FileInfo(filePath); + if (!info.Exists || info.Length == 0) + { + return ""; + } + + var length = (int)Math.Min(maxBytes, info.Length); + var buffer = new byte[length]; + using var stream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete); + stream.Seek(-length, SeekOrigin.End); + _ = stream.Read(buffer, 0, length); + return Encoding.UTF8.GetString(buffer); + } +} + +internal interface IMimirSourceStat +{ + string SourceId { get; } +} + +internal sealed record MimirTelemetryStat(double RuntimeSeconds, int Sources, int LastPoll, double Ingested, string AudioSync, int Reports, int States, double AnalyzeMs); + +internal sealed record MimirSpectrumStat(string SourceId, string Label, int SampleRate, int FftSize, double Rms, double Peak, double NoiseFloorDb, string Peaks) : IMimirSourceStat; + +internal sealed record MimirSyncStateStat(string SourceId, string ReferenceSourceId, double DelaySamples, double DelayUs, double DelayMs, double SroPpm, double Confidence) : IMimirSourceStat; + +internal sealed record MimirSyncReportStat(string SourceId, string ReferenceSourceId, string Evidence, double DelayUs, double Confidence, int Events) : IMimirSourceStat; + +internal sealed record MimirSyncDecodeAttemptStat(string SourceId, string ReferenceSourceId, string Status, int Compared, int SampleRate, double ReferenceEnergy, double CandidateEnergy, int Matched, double Confidence) : IMimirSourceStat; + +internal sealed record MimirVideoBufferStat(string SourceId, int Count, string Latest) : IMimirSourceStat; + +internal sealed record MimirActuatorCommandStat(string SourceId, double DelaySamples, double Ratio, double Confidence) : IMimirSourceStat; + +internal sealed record MimirObservationStreamStat(string DeviceId, string StreamId, string Kind, double Sequence, string State, double AgeSeconds, string Shape); + +internal sealed record MimirWellStat( + long Sequence, + double ElapsedSeconds, + double IngestedSamples, + int ConfiguredSources, + int LiveSources, + int SourceErrorCount, + bool SynchronizedFrameComplete, + double PresentationDelayMs, + string FrameDegradedKind, + string FrameDegradedReason, + int ReadySlices, + int TotalSlices, + double LatencyCurrentMs, + double LatencyCeilingMs, + double LatencyFloorMs, + double LatencyRetainedOverlapMs, + double LatencyEdgeSkewMs, + string LatencyReason, + double PollAverageMs, + double PollMaxMs, + long ZeroPollIterations, + double PublishAverageMs, + double PublishMaxMs, + long PublishedBytes, + string CaptureInlineBodies, + int PublicationStemCount, + IReadOnlyList SourceHealth, + IReadOnlyList Buffers, + IReadOnlyList ClockDomains, + IReadOnlyList AudioSyncStates, + IReadOnlyList VisualCalibration, + double FeatureMeanConfidence, + double FeatureMeanMotionPixelsPerSecond, + int FeatureStableTracks, + IReadOnlyList FeatureSignals, + IReadOnlyList CanonicalClockMaps, + MimirWellProbeStat? Probe) +{ + public int AudioBuffers => Buffers.Count(static buffer => string.Equals(buffer.Kind, "Audio", StringComparison.OrdinalIgnoreCase)); + + public int VideoBuffers => Buffers.Count(static buffer => string.Equals(buffer.Kind, "Video", StringComparison.OrdinalIgnoreCase)); + + public double MaxCanonicalClockOffsetMs => CanonicalClockMaps.Count == 0 + ? 0.0 + : CanonicalClockMaps.Max(static map => Math.Abs(map.OffsetMs)); +} + +internal sealed record MimirWellBufferStat( + string SourceId, + string Kind, + int Count, + double EdgeNs, + int Width, + int Height, + string PixelFormat, + int SampleRate, + int Channels); + +internal sealed record MimirWellSourceHealthStat( + string SourceId, + string Kind, + string Origin, + string Label, + string Status, + string Reason, + int Buffers, + int Samples, + double LatestAgeMs, + int LatestByteLength); + +internal sealed record MimirWellAudioSyncStateStat(string SourceId, string ReferenceSourceId, double DelaySamples, double SroPpm, double Confidence); + +internal sealed record MimirWellClockDomainStat(string ClockDomainId, int SourceCount, double OverlapMs, bool HasLocalOverlap, bool IsReference, double OffsetMs); + +internal sealed record MimirWellVisualCalibrationStat(string SourceId, string State, double BestScore, int BestDetectedLedCount, bool BestUsableForCalibration, double BestExposure, double BestGain); + +internal sealed record MimirWellFeatureSignalStat(string SourceId, int StableTrackCount, double Confidence, double MeanMotionPixelsPerSecond, double MotionEnergy, double CentroidX, double CentroidY); + +internal sealed record MimirWellCanonicalClockMapStat(string StreamKey, double OffsetMs, long SampleCount); + +internal sealed record MimirWellProbeStat(bool ShouldEmit, double SyncConfidence, double FrequencyConfidence, string Reason); + +internal sealed record MimirMoveMusicStat( + string TracePath, + double Bpm, + double BpmConfidence, + double LoudnessGate, + double LoudnessPercentile, + string KeyName, + string KeyMode, + string ChordName, + double ScoreEnvelopeMax, + int MaxRgb, + IReadOnlyList Sources, + IReadOnlyList Voices, + IReadOnlyList MoveTargets, + string PianoRoll); + +internal sealed record MimirMoveMusicVoiceStat(string SourceId, string Role, string NoteName, double FrequencyHz, double MidiNote, double Confidence, double Strength, bool Active); + +internal sealed record MimirMoveMusicTargetStat(int MoveIndex, string SourceId, string NoteName, double TargetNote, double Confidence, double CalibrationPriority); + +internal sealed class MimirStreamLayoutProvider : MutableDashboardProvider +{ + public const string ProviderId = "mimir.stream.layout"; + + public MimirStreamLayoutProvider() + : base( + new DashboardProviderManifest( + ProviderId, + "Mimir Stream Layout", + "Native scene graph editor for program layout, source transforms, visibility, and future LUT controls.", + "1", + "/eve/deck/mimir.stream.layout", + ["scene2d", "visibility", "transform", "lut-presets", "audio-mix"], + UsesCultMesh: true, + Transport: "local WebSocket; state document can mirror through CultMesh."), + CreateState()) + { + } + + private static DashboardState CreateState() => + new() + { + ProviderId = ProviderId, + Title = "Mimir Stream Layout", + Nodes = + [ + new DashboardNode("program", "Program Output", "camera", -0.05, -0.05, 0.62, 0.34, "live") { Detail = "Final OBS-facing composition surface. Visibility, solo, opacity, layer order, audio stem selection, mute/solo/gain, and LUT preset controls belong here." }, + new DashboardNode("editor", "Mimir Editor", "editor", -0.48, 0.34, 0.40, 0.28, "live") { Detail = "Scene graph owner for editor camera, sensor-feed panels, SDF text placeholders, model placeholders, selected node, visibility, locks, transform reset, and grab/rotate/resize gizmo intent." }, + new DashboardNode("sync", "Mimir Sync", "timing", 0.48, 0.34, 0.36, 0.28, "live") { Detail = "Rolling buffer and synchronization surface: five-second window, stream count, source count, poll cadence, ingested samples, audio sync state, and spectrum cadence." }, + new DashboardNode("leap-field", "Leap Field", "sensor", 0.38, 0.02, 0.28, 0.20, "dense geometry") { Detail = "Leap packed stereo IR feeds Fensalir stereo depth, disparity SurfacePage, point-cloud Mesh socket, surface claims, and GPU sensor fusion." }, + new DashboardNode("ps3-eye-0", "PS3 Eye 0", "camera", -0.56, -0.34, 0.24, 0.18, "187 fps tracker") { Detail = "High-rate Bayer8 tracking witness. GPU feature extraction samples this texture through Fensalir GpuSensorFusion." }, + new DashboardNode("ps3-eye-1", "PS3 Eye 1", "camera", -0.26, -0.44, 0.24, 0.18, "187 fps tracker") { Detail = "Second high-rate Bayer8 tracking witness. Used for fast motion and online calibration constraints." }, + new DashboardNode("kiyo-pro", "Kiyo Pro", "camera", 0.18, -0.42, 0.24, 0.18, "YUY2 RGB") { Detail = "RGB/context witness. Fensalir decodes YUY2 in compute through the documented R8G8B8A8_UNORM SRV path." }, + new DashboardNode("kiyo-basic", "Kiyo Basic", "camera", 0.50, -0.34, 0.24, 0.18, "YUY2 RGB") { Detail = "Second RGB/context witness. Enters the same GpuSensorFusion path after YUY2 decode." }, + new DashboardNode("eve-camera", "Eve Camera", "camera", 0.0, -0.62, 0.30, 0.18, "armed") { Detail = "Portable sensor uplink target. EveCanvas owns capture on-device and sends frame events into Mimir when online." }, + ], + }; +} + +internal sealed class VoidBotSwarmProvider : IDashboardProvider +{ + public const string ProviderId = "voidbot.swarm"; + + private readonly string swarmStatePath; + private DashboardState currentState; + private DateTime lastWriteUtc = DateTime.MinValue; + private string selectedIdentityId = ""; + private string selectedStatePath = ""; + + public VoidBotSwarmProvider(string swarmStatePath) + { + this.swarmStatePath = swarmStatePath; + Manifest = new DashboardProviderManifest( + ProviderId, + "VoidBot Swarm", + "Native Eve tab for VoidBot agent status, CTB order, and selected Face state.", + "1", + "/eve/deck/voidbot.swarm", + ["ctb", "agent-status", "state-tree", "cultmesh-snapshot"], + UsesCultMesh: true, + Transport: "Reads VoidBot swarm-state.json now; consumes voidbot.swarm_state_snapshot.v1 when CultMesh bridge is live."); + currentState = BuildMissingState(); + } + + public DashboardProviderManifest Manifest { get; } + + public DashboardState State + { + get + { + RefreshIfNeeded(force: false); + return currentState; + } + } + + public bool ApplyCommand(DashboardCommand command) + { + RefreshIfNeeded(force: false); + var node = currentState.Nodes.FirstOrDefault(candidate => string.Equals(candidate.Id, command.NodeId, StringComparison.Ordinal)); + if (node == null) + { + return false; + } + + currentState.SelectedNodeId = node.Id; + if (!string.IsNullOrWhiteSpace(node.IdentityId)) + { + selectedIdentityId = node.IdentityId; + selectedStatePath = ""; + } + + if (!string.IsNullOrWhiteSpace(node.StatePath)) + { + selectedStatePath = node.StatePath; + } + + RefreshIfNeeded(force: true); + currentState.SelectedNodeId = node.Id; + return true; + } + + private void RefreshIfNeeded(bool force) + { + try + { + var info = new FileInfo(swarmStatePath); + if (!info.Exists) + { + currentState = BuildMissingState(); + return; + } + + if (!force && info.LastWriteTimeUtc == lastWriteUtc) + { + return; + } + + using var document = JsonDocument.Parse(File.ReadAllText(info.FullName)); + currentState = BuildState(document.RootElement, info.LastWriteTimeUtc); + lastWriteUtc = info.LastWriteTimeUtc; + } + catch (Exception error) + { + currentState = BuildErrorState(error.Message); + } + } + + private DashboardState BuildState(JsonElement root, DateTime sourceWriteUtc) + { + var summary = TryGet(root, "summary"); + var controls = TryGet(root, "controls"); + var cultMesh = TryGet(root, "cultMesh"); + var orchestrator = TryGet(root, "orchestrator"); + var participants = ArrayItems(TryGet(root, "participants")).ToArray(); + var upcoming = ArrayItems(TryGet(root, "upcomingTurns")).ToArray(); + + if (string.IsNullOrWhiteSpace(selectedIdentityId)) + { + selectedIdentityId = StringValue(summary, "nextIdentityId") + ?? StringValue(upcoming.FirstOrDefault(), "identityId") + ?? StringValue(participants.FirstOrDefault(), "identityId") + ?? ""; + } + + var selectedAgent = participants.FirstOrDefault(participant => string.Equals(StringValue(participant, "identityId"), selectedIdentityId, StringComparison.Ordinal)); + if (selectedAgent.ValueKind == JsonValueKind.Undefined) + { + selectedAgent = participants.FirstOrDefault(); + selectedIdentityId = StringValue(selectedAgent, "identityId") ?? ""; + } + + var nodes = new List(); + var paused = BoolValue(summary, "paused") == true; + var stateLabel = StringValue(summary, "state") ?? "unknown"; + var generatedAt = StringValue(root, "generatedAt") ?? sourceWriteUtc.ToString("O"); + var cadence = NumberValue(controls, "cadenceMultiplier") ?? NumberValue(summary, "cadenceMultiplier") ?? 1; + nodes.Add(new DashboardNode( + "voidbot-summary", + $"VoidBot Swarm\n{stateLabel} next {StringValue(summary, "nextDisplayName") ?? "none"}", + "swarm", + -0.60, + -0.54, + 0.34, + 0.22, + paused ? "paused" : "running") + { + Detail = $"agents {NumberValue(summary, "participantCount") ?? 0:0} ready {NumberValue(summary, "readyNowCount") ?? 0:0} cadence x{cadence:0.##}\nmesh {StringValue(cultMesh, "writeStatus") ?? "missing"} {generatedAt}", + }); + + AddCtbRail(nodes, upcoming); + AddAgentCards(nodes, participants); + AddSelectedAgent(nodes, selectedAgent); + + var selectedNodeId = string.IsNullOrWhiteSpace(currentState.SelectedNodeId) || + nodes.All(node => !string.Equals(node.Id, currentState.SelectedNodeId, StringComparison.Ordinal)) + ? "voidbot-summary" + : currentState.SelectedNodeId; + + var state = new DashboardState + { + ProviderId = ProviderId, + Title = "VoidBot Swarm", + Version = DateTimeOffset.UtcNow.ToUnixTimeSeconds(), + UpdatedAt = DateTimeOffset.UtcNow, + SelectedNodeId = selectedNodeId, + Nodes = nodes, + }; + state.Surface = BuildVoidBotSurface(state, summary, cultMesh, controls, orchestrator, upcoming, participants, selectedAgent); + return state; + } + + private DashboardSurface BuildVoidBotSurface( + DashboardState state, + JsonElement summary, + JsonElement cultMesh, + JsonElement controls, + JsonElement orchestrator, + IReadOnlyList upcoming, + IReadOnlyList participants, + JsonElement selectedAgent) + { + return new DashboardSurface + { + Schema = "cultmesh.eve_surface.v0", + Id = "voidbot.swarm.surface", + Title = "VoidBot Swarm", + Root = UiElement.Container( + "voidbot-cockpit", + "cockpit", + new DashboardUiLayout + { + Direction = "vertical", + Gap = 8, + Padding = 8, + Overflow = "scroll", + Grow = 5, + MinWidth = 112, + MinHeight = 30, + PreferredWidth = 156, + PreferredHeight = 58, + Priority = -30, + Density = "continuous", + ViewportMode = "continuous-ops", + }, + [ + BuildTurnRail(upcoming), + ]), + Assets = [], + }; + } + + private DashboardUiElement BuildTurnRail(IReadOnlyList upcoming) + { + var count = Math.Min(14, upcoming.Count); + var rows = new List(); + for (var index = 0; index < count; index++) + { + var turn = upcoming[index]; + var identityId = StringValue(turn, "identityId") ?? ""; + var active = !string.IsNullOrWhiteSpace(StringValue(turn, "activeJobId")); + var mentionCount = NumberValue(turn, "pendingMentionCount") ?? 0; + var rest = TryGet(turn, "restState"); + var napping = BoolValue(rest, "isNapping") == true; + var state = active ? "active" : mentionCount > 0 ? "mention" : napping ? "nap" : Minutes(NumberValue(turn, "nextTurnInMinutes")); + var speed = NumberValue(turn, "effectiveSpeed") ?? 0; + var heat = NumberValue(turn, "heat") ?? 0; + rows.Add(UiElement.Text( + $"ctb-row-{index}-{identityId}", + $"{index + 1,2}. {(StringValue(turn, "displayName") ?? identityId),-10} {state,-7} s{speed:0.###} h{heat:0.##}", + "mono")); + } + + var rail = UiElement.Container( + "ctb-rail", + "rail", + new DashboardUiLayout { Direction = "vertical", Gap = 4, Padding = 8, Overflow = "scroll", Grow = 1, MinWidth = 40, MinHeight = 20 }, + rows.Count > 0 + ? rows + : [UiElement.Text("ctb-empty", "no CTB turns", "caption")]); + rail.Text = "CTB order"; + return rail; + } + + private DashboardUiElement BuildUpcomingFacesPane(IReadOnlyList upcoming) + { + var rows = upcoming + .Take(10) + .Select((turn, index) => + { + var identityId = StringValue(turn, "identityId") ?? $"turn-{index}"; + var active = !string.IsNullOrWhiteSpace(StringValue(turn, "activeJobId")); + var mentionCount = NumberValue(turn, "pendingMentionCount") ?? 0; + var rest = TryGet(turn, "restState"); + var napping = BoolValue(rest, "isNapping") == true; + var state = active ? "active" : mentionCount > 0 ? "mention" : napping ? "nap" : Minutes(NumberValue(turn, "nextTurnInMinutes")); + var heat = NumberValue(turn, "heat") ?? 0; + var speed = NumberValue(turn, "effectiveSpeed") ?? 0; + return UiElement.Text( + $"upcoming-face-{index}-{identityId}", + $"{index + 1,2}. {(StringValue(turn, "displayName") ?? identityId),-10} {state,-8} {StringValue(turn, "repoName") ?? "repo"} spd {speed:0.###} heat {heat:0.##}", + "mono"); + }) + .ToArray(); + + return UiElement.Pane( + "upcoming-faces-pane", + "Next Faces", + rows.Length > 0 + ? rows + : [UiElement.Text("upcoming-faces-empty", "no upcoming Face turns", "caption")]); + } + + private DashboardUiElement BuildWatchdogPane(JsonElement orchestrator) + { + var state = StringValue(orchestrator, "state") ?? "unknown"; + var organs = ArrayItems(TryGet(orchestrator, "organs")).ToArray(); + var watchdog = organs.FirstOrDefault(organ => string.Equals(StringValue(organ, "id"), "voidbot-operations-watchdog", StringComparison.OrdinalIgnoreCase)); + var children = new List + { + UiElement.Text("watchdog-summary", $"orchestrator {state} organs {organs.Length}", "mono"), + }; + + if (watchdog.ValueKind != JsonValueKind.Undefined) + { + children.Add(UiElement.Text( + "watchdog-status", + $"watchdog {StringValue(watchdog, "lastStatus") ?? "unknown"} exit {(NumberValue(watchdog, "lastExitCode") ?? 0):0}\nlast {ShortIso(StringValue(watchdog, "lastFinishedAt") ?? StringValue(watchdog, "lastStartedAt"))}", + "strong")); + } + else + { + children.Add(UiElement.Text("watchdog-missing", "watchdog organ not present in snapshot", "caption")); + } + + children.AddRange(organs + .Where(static organ => organ.ValueKind != JsonValueKind.Undefined) + .OrderBy(organ => string.Equals(StringValue(organ, "id"), "voidbot-operations-watchdog", StringComparison.OrdinalIgnoreCase) ? 0 : 1) + .ThenBy(organ => StringValue(organ, "id")) + .Take(7) + .Select(organ => UiElement.Text( + $"watchdog-organ-{StableId(StringValue(organ, "id") ?? "organ")}", + $"{StringValue(organ, "label") ?? StringValue(organ, "id") ?? "organ"}: {StringValue(organ, "lastStatus") ?? "unknown"}", + "caption"))); + + return UiElement.Pane("voidbot-watchdog-pane", "VoidBot Watchdog", children); + } + + private DashboardUiElement BuildSelectedFacePane(JsonElement summary, JsonElement cultMesh, JsonElement selectedAgent, double cadence, string generatedAt) + { + var identityId = StringValue(selectedAgent, "identityId") ?? ""; + var faceState = TryGet(selectedAgent, "faceState"); + var counts = TryGet(faceState, "counts"); + var memory = NumberValue(counts, "memory") ?? 0; + var description = StringValue(selectedAgent, "description") ?? "No Face description registered."; + return UiElement.Pane( + "selected-face-pane", + "Controls / Selected Face", + [ + UiElement.Text("voidbot-summary-text", $"VoidBot Swarm\n{StringValue(summary, "state") ?? "unknown"} next {StringValue(summary, "nextDisplayName") ?? "none"}\nagents {NumberValue(summary, "participantCount") ?? 0:0} ready {NumberValue(summary, "readyNowCount") ?? 0:0} cadence x{cadence:0.##}\nmesh {StringValue(cultMesh, "writeStatus") ?? "missing"} {generatedAt}", "mono"), + UiElement.Card( + "selected-face-card", + "inspector-hero", + bindNodeId: "agent-detail", + commandId: $"select-identity:{identityId}", + children: + [ + UiElement.Avatar("selected-face-avatar", StringValue(selectedAgent, "avatarUrl"), StringValue(selectedAgent, "displayName") ?? identityId), + UiElement.Text("selected-face-name", $"Selected Face\n{StringValue(selectedAgent, "displayName") ?? identityId}", "title"), + ]), + UiElement.Metric("metric-turn", "Turn", 0.96, "cool"), + UiElement.Metric("metric-memory", "Memory", Math.Min(1.0, memory * 0.07), "cool"), + UiElement.Metric("metric-pressure", "Pressure", 0.42, "warm"), + UiElement.Metric("metric-heat", "Heat", 0.34, "warm"), + UiElement.Metric("metric-load", "Load", 0.99, "danger"), + UiElement.Metric("metric-speed", "Speed", 0.88, "cool"), + UiElement.Text("selected-face-detail", $"memory {memory:0} pressures {NumberValue(counts, "pressures") ?? 0:0} constraints {NumberValue(selectedAgent, "constraintCount") ?? 0:0}\n{Truncate(description, 220)}", "body"), + ]); + } + + private DashboardUiElement BuildStateGraphPane(IReadOnlyList leaves) + { + return UiElement.Pane( + "state-graph-pane", + "State Graph", + leaves.Select((leaf, index) => UiElement.Card( + $"state-leaf-card-{index}", + "state-leaf", + bindNodeId: $"state-{index}", + commandId: $"select-state:{leaf.Path}", + style: new DashboardUiStyle { Variant = string.Equals(leaf.Path, selectedStatePath, StringComparison.Ordinal) ? "selected" : "default" }, + children: + [ + UiElement.Text($"state-leaf-label-{index}", leaf.Label, "strong"), + UiElement.Text($"state-leaf-preview-{index}", leaf.Preview, "caption"), + ])).ToArray()); + } + + private static DashboardUiElement BuildStateDetailPane(DashboardState state, VoidBotStateLeaf? selectedLeaf) + { + return UiElement.Pane( + "state-detail-pane", + "State Detail", + [ + UiElement.Text("state-detail-heading", selectedLeaf != null ? $"{selectedLeaf.Label}\n{selectedLeaf.Path}" : state.Title, "strong"), + UiElement.Text("state-detail-body", selectedLeaf?.Detail ?? state.Nodes.FirstOrDefault(static node => node.Id == "voidbot-summary")?.Detail ?? "No state detail.", "mono"), + ]); + } + + private void AddCtbRail(List nodes, IReadOnlyList upcoming) + { + var count = Math.Min(8, upcoming.Count); + for (var index = 0; index < count; index++) + { + var turn = upcoming[index]; + var identityId = StringValue(turn, "identityId") ?? ""; + var active = !string.IsNullOrWhiteSpace(StringValue(turn, "activeJobId")); + var mentionCount = NumberValue(turn, "pendingMentionCount") ?? 0; + nodes.Add(new DashboardNode( + $"ctb-{index}-{identityId}", + $"{index + 1}. {StringValue(turn, "displayName") ?? identityId}\n{StringValue(turn, "repoName") ?? "repo"}", + "ctb-turn", + -0.82 + (index * 0.235), + -0.82, + 0.21, + 0.11, + active ? "active" : mentionCount > 0 ? "mention" : Minutes(NumberValue(turn, "nextTurnInMinutes"))) + { + IdentityId = identityId, + AvatarUrl = StringValue(turn, "avatarUrl"), + Detail = $"speed {NumberValue(turn, "effectiveSpeed") ?? 0:0.###} heat {NumberValue(turn, "heat") ?? 0:0.###}", + }); + } + } + + private void AddAgentCards(List nodes, IReadOnlyList participants) + { + var visible = participants + .OrderBy(participant => NumberValue(participant, "nextTurnInMinutes") ?? 999999) + .Take(10) + .ToArray(); + + for (var index = 0; index < visible.Length; index++) + { + var agent = visible[index]; + var identityId = StringValue(agent, "identityId") ?? $"agent-{index}"; + var column = index % 5; + var row = index / 5; + var active = !string.IsNullOrWhiteSpace(StringValue(agent, "activeJobId")); + var selected = string.Equals(identityId, selectedIdentityId, StringComparison.Ordinal); + nodes.Add(new DashboardNode( + $"agent-{identityId}", + $"{StringValue(agent, "displayName") ?? identityId}\n{StringValue(agent, "repoName") ?? "repo"}", + "agent", + -0.74 + (column * 0.37), + -0.28 + (row * 0.22), + 0.31, + 0.17, + selected ? "selected" : active ? "active" : Minutes(NumberValue(agent, "nextTurnInMinutes"))) + { + IdentityId = identityId, + AvatarUrl = StringValue(agent, "avatarUrl"), + Detail = $"load {NumberValue(agent, "currentLoad") ?? 0:0.##} heat {NumberValue(agent, "heat") ?? 0:0.##}", + }); + } + } + + private void AddSelectedAgent(List nodes, JsonElement agent) + { + if (agent.ValueKind == JsonValueKind.Undefined) + { + return; + } + + var identityId = StringValue(agent, "identityId") ?? ""; + var faceState = TryGet(agent, "faceState"); + var counts = TryGet(faceState, "counts"); + var description = StringValue(agent, "description") ?? "No Face description registered."; + nodes.Add(new DashboardNode( + "agent-detail", + $"{StringValue(agent, "displayName") ?? identityId} State\n{StringValue(agent, "repoName") ?? "repo"}", + "state-detail", + -0.42, + 0.52, + 0.55, + 0.30, + BoolValue(faceState, "readable") == true ? "readable" : "unreadable") + { + IdentityId = identityId, + AvatarUrl = StringValue(agent, "avatarUrl"), + Detail = $"memory {NumberValue(counts, "memory") ?? 0:0} pressures {NumberValue(counts, "pressures") ?? 0:0} constraints {NumberValue(agent, "constraintCount") ?? 0:0}\n{Truncate(description, 150)}", + }); + + var leaves = FlattenLeaves(ArrayItems(TryGet(faceState, "tree")).ToArray()).Take(5).ToArray(); + if (string.IsNullOrWhiteSpace(selectedStatePath)) + { + selectedStatePath = leaves.FirstOrDefault()?.Path ?? ""; + } + + for (var index = 0; index < leaves.Length; index++) + { + var leaf = leaves[index]; + var selected = string.Equals(leaf.Path, selectedStatePath, StringComparison.Ordinal); + nodes.Add(new DashboardNode( + $"state-{index}", + $"{leaf.Label}\n{leaf.Preview}", + "state-leaf", + 0.34, + 0.24 + (index * 0.13), + 0.34, + 0.10, + selected ? "open" : "leaf") + { + IdentityId = identityId, + StatePath = leaf.Path, + Detail = leaf.Detail, + }); + } + + var selectedLeaf = leaves.FirstOrDefault(leaf => string.Equals(leaf.Path, selectedStatePath, StringComparison.Ordinal)); + if (selectedLeaf != null && !string.IsNullOrWhiteSpace(selectedLeaf.Path)) + { + nodes.Add(new DashboardNode( + "state-detail", + $"{selectedLeaf.Label}\n{selectedLeaf.Path}", + "state-text", + 0.38, + 0.74, + 0.38, + 0.18, + "detail") + { + IdentityId = identityId, + StatePath = selectedLeaf.Path, + Detail = Truncate(selectedLeaf.Detail, 280), + }); + } + } + + private DashboardState BuildMissingState() => new() + { + ProviderId = ProviderId, + Title = "VoidBot Swarm", + SelectedNodeId = "voidbot-missing", + Nodes = + [ + new DashboardNode("voidbot-missing", "VoidBot Swarm\nsnapshot missing", "swarm", 0.0, -0.10, 0.52, 0.24, "missing") + { + Detail = swarmStatePath, + }, + ], + }; + + private DashboardState BuildErrorState(string error) => new() + { + ProviderId = ProviderId, + Title = "VoidBot Swarm", + SelectedNodeId = "voidbot-error", + Nodes = + [ + new DashboardNode("voidbot-error", "VoidBot Swarm\nstate read failed", "swarm", 0.0, -0.10, 0.52, 0.24, "error") + { + Detail = error, + }, + ], + }; + + private static IEnumerable FlattenLeaves(IReadOnlyList nodes) + { + foreach (var node in nodes) + { + if (StringValue(node, "kind") == "leaf") + { + yield return new VoidBotStateLeaf( + StringValue(node, "label") ?? "leaf", + StringValue(node, "path") ?? "", + StringValue(node, "preview") ?? "", + StringValue(node, "detail") ?? StringValue(node, "preview") ?? ""); + } + + foreach (var child in FlattenLeaves(ArrayItems(TryGet(node, "children")).ToArray())) + { + yield return child; + } + } + } + + private static JsonElement TryGet(JsonElement element, string property) + { + if (element.ValueKind == JsonValueKind.Object && element.TryGetProperty(property, out var child)) + { + return child; + } + + return default; + } + + private static IEnumerable ArrayItems(JsonElement element) + { + if (element.ValueKind != JsonValueKind.Array) + { + yield break; + } + + foreach (var item in element.EnumerateArray()) + { + yield return item; + } + } + + private static string? StringValue(JsonElement element, string property) => StringValue(TryGet(element, property)); + + private static string? StringValue(JsonElement element) => + element.ValueKind == JsonValueKind.String ? element.GetString() : null; + + private static double? NumberValue(JsonElement element, string property) => NumberValue(TryGet(element, property)); + + private static double? NumberValue(JsonElement element) => + element.ValueKind == JsonValueKind.Number && element.TryGetDouble(out var value) ? value : null; + + private static bool? BoolValue(JsonElement element, string property) + { + var child = TryGet(element, property); + return child.ValueKind switch + { + JsonValueKind.True => true, + JsonValueKind.False => false, + _ => null, + }; + } + + private static string Minutes(double? value) + { + if (value == null) + { + return "unknown"; + } + + return value <= 0 ? "ready" : $"{value:0.#}m"; + } + + private static string Truncate(string value, int max) => + value.Length <= max ? value : value[..Math.Max(0, max - 1)] + "..."; + + private static string StableId(string value) + { + var builder = new StringBuilder(value.Length); + foreach (var ch in value.ToLowerInvariant()) + { + builder.Append(char.IsLetterOrDigit(ch) ? ch : '-'); + } + + return builder.ToString().Trim('-'); + } + + private static string ShortIso(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "unknown"; + } + + return DateTimeOffset.TryParse(value, out var timestamp) + ? timestamp.ToLocalTime().ToString("MM-dd HH:mm") + : value; + } +} + +internal sealed record VoidBotStateLeaf(string Label, string Path, string Preview, string Detail); + +internal sealed class YggdrasilStreamPixelsProvider : MutableDashboardProvider +{ + public const string ProviderId = "yggdrasil.streampixels.edge"; + + public YggdrasilStreamPixelsProvider() + : base( + new DashboardProviderManifest( + ProviderId, + "StreamPixels Edge", + "Yggdrasil/StreamPixels live edge control surface reachable over HTTPS or SSH-forwarded localhost.", + "1", + "/eve/deck/yggdrasil.streampixels.edge", + ["status", "health", "ssh-tunnel", "hls"], + UsesCultMesh: false, + Transport: "TCP/WebSocket over LAN or SSH tunnel."), + new DashboardState + { + ProviderId = ProviderId, + Title = "StreamPixels Edge", + SelectedNodeId = "hls-origin", + Nodes = + [ + new DashboardNode("hls-origin", "HLS Origin", "service", -0.34, -0.18, 0.34, 0.22, "streampixels"), + new DashboardNode("rtmp-ingest", "RTMP Ingest", "service", 0.34, -0.18, 0.34, 0.22, "localhost"), + new DashboardNode("ssh-tunnel", "Yggdrasil SSH Tunnel", "transport", 0.0, 0.34, 0.44, 0.18, "tcp only"), + ], + }) + { + } +} + +internal sealed class RemoteDashboardProvider : MutableDashboardProvider +{ + private RemoteDashboardProvider(DashboardProviderManifest manifest, string upstream) + : base( + manifest, + new DashboardState + { + ProviderId = manifest.Id, + Title = manifest.Title, + SelectedNodeId = "remote-endpoint", + Nodes = + [ + new DashboardNode("remote-endpoint", manifest.Title, "remote-dashboard", 0.0, -0.10, 0.46, 0.24, "registered") + { + Endpoint = upstream, + }, + new DashboardNode("remote-transport", "Transport", "transport", 0.0, 0.34, 0.40, 0.18, "tcp/ws") + { + Endpoint = upstream, + }, + ], + }) + { + } + + public static RemoteDashboardProvider? FromSpec(string spec) + { + var parts = spec.Split('|', 3, StringSplitOptions.TrimEntries); + if (parts.Length < 3 || string.IsNullOrWhiteSpace(parts[0]) || string.IsNullOrWhiteSpace(parts[1]) || string.IsNullOrWhiteSpace(parts[2])) + { + Console.Error.WriteLine($"Ignoring dashboard provider spec. Expected id|title|ws-url, got: {spec}"); + return null; + } + + return new RemoteDashboardProvider( + new DashboardProviderManifest( + parts[0], + parts[1], + "External EveDeck dashboard provider registered at broker launch.", + "1", + parts[2], + ["scene2d", "external-provider"], + UsesCultMesh: false, + Transport: "registered WebSocket endpoint"), + parts[2]); + } +} + +internal sealed record HttpRequest(string Path, IReadOnlyDictionary Headers); + +internal sealed record WebSocketFrame(int Opcode, byte[] Payload); + +internal sealed record DashboardSocket(TcpClient Client, NetworkStream Stream, bool CultMeshBinary); + +internal sealed record DashboardProviderCatalogDocument(IReadOnlyList Providers); + +internal sealed record DashboardProviderManifest( + string Id, + string Title, + string Description, + string Version, + string Endpoint, + string[] Capabilities, + bool UsesCultMesh, + string Transport); + +internal sealed class DashboardState +{ + public string Type { get; set; } = "dashboard-state"; + + public string Schema { get; set; } = "mimir.eve_dashboard_state.v1"; + + public string ProviderId { get; set; } = MimirStreamLayoutProvider.ProviderId; + + public string Title { get; set; } = "Mimir Dashboard"; + + public long Version { get; set; } = 1; + + public DateTimeOffset UpdatedAt { get; set; } = DateTimeOffset.UtcNow; + + public string SelectedNodeId { get; set; } = "program"; + + public string LutPreset { get; set; } = "neutral"; + + public List Nodes { get; set; } = []; + + public DashboardSurface? Surface { get; set; } +} + +internal sealed class DashboardSurface +{ + public string Schema { get; set; } = "cultmesh.eve_surface.v0"; + + public string Id { get; set; } = ""; + + public string Title { get; set; } = ""; + + public DashboardUiElement Root { get; set; } = UiElement.Container("root", "root", new DashboardUiLayout(), []); + + public IReadOnlyList Assets { get; set; } = []; +} + +internal sealed record DashboardSurfaceAsset(string Id, string Kind, string Uri); + +internal sealed class DashboardUiBinding +{ + public string DocumentSchema { get; set; } = ""; + + public string DocumentId { get; set; } = ""; + + public string Path { get; set; } = ""; + + public string ValueKind { get; set; } = ""; + + public string Access { get; set; } = "read"; + + public string Authority { get; set; } = ""; + + public string? CommandId { get; set; } +} + +internal sealed class DashboardUiElement +{ + public string Id { get; set; } = ""; + + public string Kind { get; set; } = ""; + + public string? Role { get; set; } + + public string? Text { get; set; } + + public string? AssetRef { get; set; } + + public string? AssetUri { get; set; } + + public string? BindNodeId { get; set; } + + public string? CommandId { get; set; } + + public DashboardUiBinding? Binding { get; set; } + + public DashboardUiLayout? Layout { get; set; } + + public DashboardUiStyle? Style { get; set; } + + public DashboardUiMetric? Metric { get; set; } + + public IReadOnlyList Children { get; set; } = []; +} + +internal sealed class DashboardUiLayout +{ + public string Direction { get; set; } = "vertical"; + + public double? Width { get; set; } + + public double? Height { get; set; } + + public double? Grow { get; set; } + + public double? Gap { get; set; } + + public double? Padding { get; set; } + + public string? Overflow { get; set; } + + public double? MinWidth { get; set; } + + public double? MinHeight { get; set; } + + public double? PreferredWidth { get; set; } + + public double? PreferredHeight { get; set; } + + public double? Priority { get; set; } + + public string? Density { get; set; } + + public string? ViewportMode { get; set; } +} + +internal sealed class DashboardUiStyle +{ + public string Variant { get; set; } = "default"; + + public string? Tone { get; set; } +} + +internal sealed record DashboardUiMetric(string Label, double Value, string Tone); + +internal static class UiElement +{ + public static DashboardUiElement Container(string id, string kind, DashboardUiLayout layout, IReadOnlyList children) + { + return new DashboardUiElement { Id = id, Kind = kind, Layout = layout, Children = children }; + } + + public static DashboardUiElement Pane(string id, string title, IReadOnlyList children) + { + return new DashboardUiElement + { + Id = id, + Kind = "pane", + Text = title, + Layout = new DashboardUiLayout { Direction = "vertical", Gap = 10, Padding = 12, Grow = 1 }, + Children = children, + }; + } + + public static DashboardUiElement Card( + string id, + string role, + string? bindNodeId = null, + string? commandId = null, + DashboardUiStyle? style = null, + IReadOnlyList? children = null) + { + return new DashboardUiElement + { + Id = id, + Kind = "card", + Role = role, + BindNodeId = bindNodeId, + CommandId = commandId, + Style = style, + Children = children ?? [], + }; + } + + public static DashboardUiElement Avatar(string id, string? avatarUrl, string alt) + { + return new DashboardUiElement + { + Id = id, + Kind = "avatar", + Text = alt, + AssetUri = avatarUrl, + Layout = new DashboardUiLayout { Width = 42, Height = 42 }, + }; + } + + public static DashboardUiElement Text(string id, string text, string role) + { + return new DashboardUiElement { Id = id, Kind = "text", Role = role, Text = text }; + } + + public static DashboardUiElement Metric(string id, string label, double value, string tone) + { + return new DashboardUiElement + { + Id = id, + Kind = "metric", + Metric = new DashboardUiMetric(label, Math.Clamp(value, 0.0, 1.0), tone), + }; + } +} + +internal sealed class DashboardNode( + string id, + string label, + string kind, + double defaultX, + double defaultY, + double width, + double height, + string health) +{ + public string Id { get; set; } = id; + + public string Label { get; set; } = label; + + public string Kind { get; set; } = kind; + + public bool Visible { get; set; } = true; + + public double X { get; set; } = defaultX; + + public double Y { get; set; } = defaultY; + + public double DefaultX { get; set; } = defaultX; + + public double DefaultY { get; set; } = defaultY; + + public double Z { get; set; } + + public double Rotation { get; set; } + + public double Scale { get; set; } = 1.0; + + public double Width { get; set; } = width; + + public double Height { get; set; } = height; + + public string Health { get; set; } = health; + + public string? ProviderId { get; set; } + + public string? Command { get; set; } + + public string? Endpoint { get; set; } + + public string? IdentityId { get; set; } + + public string? AvatarUrl { get; set; } + + public string? StatePath { get; set; } + + public string? Detail { get; set; } +} + +internal sealed class DashboardCommand +{ + public string Type { get; set; } = ""; + + public string NodeId { get; set; } = ""; + + public string? ProviderId { get; set; } + + public double? X { get; set; } + + public double? Y { get; set; } + + public double? Rotation { get; set; } + + public double? Scale { get; set; } + + public bool? Visible { get; set; } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs index 4de48b9..7245794 100644 --- a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs +++ b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs @@ -171,6 +171,139 @@ public sealed record MimirEveOperatorSurfaceDocument( [property: Key(6)] bool CanShowStats, [property: Key(7)] string[] CommandTopics); +[CultDocument("mimir.eve_dashboard_manifest", "mimir.eve_dashboard_manifest.v1")] +[MessagePackObject] +public sealed record MimirEveDashboardManifestDocument( + [property: Key(0)] + [property: CultName] + string ProviderId, + [property: Key(1)] string Title, + [property: Key(2)] string Description, + [property: Key(3)] string Version, + [property: Key(4)] string Endpoint, + [property: Key(5)] string[] Capabilities, + [property: Key(6)] bool UsesCultMesh, + [property: Key(7)] string Transport); + +[CultDocument("mimir.eve_dashboard_state", "mimir.eve_dashboard_state.v1")] +[MessagePackObject] +public sealed record MimirEveDashboardStateDocument( + [property: Key(0)] + [property: CultName] + string ProviderId, + [property: Key(1)] string Title, + [property: Key(2)] long Version, + [property: Key(3)] string UpdatedAtUtc, + [property: Key(4)] string SelectedNodeId, + [property: Key(5)] string LutPreset, + [property: Key(6)] MimirEveDashboardNodeSnapshot[] Nodes, + [property: Key(7)] MimirEveDashboardSurfaceSnapshot? Surface); + +[MessagePackObject] +public sealed record MimirEveDashboardNodeSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] string Label, + [property: Key(2)] string Kind, + [property: Key(3)] bool Visible, + [property: Key(4)] double X, + [property: Key(5)] double Y, + [property: Key(6)] double Z, + [property: Key(7)] double Rotation, + [property: Key(8)] double Scale, + [property: Key(9)] double Width, + [property: Key(10)] double Height, + [property: Key(11)] string Health, + [property: Key(12)] string? ProviderId, + [property: Key(13)] string? Command, + [property: Key(14)] string? Endpoint); + +[MessagePackObject] +public sealed record MimirEveDashboardSurfaceSnapshot( + [property: Key(0)] string Schema, + [property: Key(1)] string Id, + [property: Key(2)] string Title, + [property: Key(3)] MimirEveDashboardUiElementSnapshot Root, + [property: Key(4)] MimirEveDashboardSurfaceAssetSnapshot[] Assets); + +[MessagePackObject] +public sealed record MimirEveDashboardSurfaceAssetSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] string Kind, + [property: Key(2)] string Uri); + +[MessagePackObject] +public sealed record MimirEveDashboardUiElementSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] string Kind, + [property: Key(2)] string? Role, + [property: Key(3)] string? Text, + [property: Key(4)] string? AssetRef, + [property: Key(5)] string? AssetUri, + [property: Key(6)] string? BindNodeId, + [property: Key(7)] string? CommandId, + [property: Key(8)] MimirEveDashboardUiLayoutSnapshot? Layout, + [property: Key(9)] MimirEveDashboardUiStyleSnapshot? Style, + [property: Key(10)] MimirEveDashboardUiMetricSnapshot? Metric, + [property: Key(11)] MimirEveDashboardUiElementSnapshot[] Children, + [property: Key(12)] MimirEveDashboardUiBindingSnapshot? Binding = null); + +[MessagePackObject] +public sealed record MimirEveDashboardUiBindingSnapshot( + [property: Key(0)] string DocumentSchema, + [property: Key(1)] string DocumentId, + [property: Key(2)] string Path, + [property: Key(3)] string ValueKind, + [property: Key(4)] string Access, + [property: Key(5)] string Authority, + [property: Key(6)] string? CommandId); + +[MessagePackObject] +public sealed record MimirEveDashboardUiLayoutSnapshot( + [property: Key(0)] string Direction, + [property: Key(1)] double? Width, + [property: Key(2)] double? Height, + [property: Key(3)] double? Grow, + [property: Key(4)] double? Gap, + [property: Key(5)] double? Padding, + [property: Key(6)] string? Overflow, + [property: Key(7)] double? MinWidth = null, + [property: Key(8)] double? MinHeight = null, + [property: Key(9)] double? PreferredWidth = null, + [property: Key(10)] double? PreferredHeight = null, + [property: Key(11)] double? Priority = null, + [property: Key(12)] string? Density = null, + [property: Key(13)] string? ViewportMode = null); + +[MessagePackObject] +public sealed record MimirEveDashboardUiStyleSnapshot( + [property: Key(0)] string Variant, + [property: Key(1)] string? Tone); + +[MessagePackObject] +public sealed record MimirEveDashboardUiMetricSnapshot( + [property: Key(0)] string Label, + [property: Key(1)] double Value, + [property: Key(2)] string Tone); + +[CultDocument("mimir.eve_dashboard_command", "mimir.eve_dashboard_command.v1")] +[MessagePackObject] +public sealed record MimirEveDashboardCommandDocument( + [property: Key(0)] + [property: CultName] + string CommandId, + [property: Key(1)] string DeviceId, + [property: Key(2)] string ClientId, + [property: Key(3)] string ProviderId, + [property: Key(4)] string Type, + [property: Key(5)] string NodeId, + [property: Key(6)] double? X, + [property: Key(7)] double? Y, + [property: Key(8)] double? Rotation, + [property: Key(9)] double? Scale, + [property: Key(10)] bool? Visible, + [property: Key(11)] long Sequence, + [property: Key(12)] long DeviceTimestampNs); + [CultDocument("mimir.eve_sensor_observation", "mimir.eve_sensor_observation.v1")] [MessagePackObject] public sealed record MimirEveSensorObservationDocument( From 21c2ee43f0a8bdc64c9bd1780451ebc85f146efb Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 16 Jun 2026 08:39:38 +0100 Subject: [PATCH 043/135] Publish Nightwing Eve dashboard health over Idunn RUDP --- src/Mimir.EveDashboard/Program.cs | 55 +++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 13 deletions(-) diff --git a/src/Mimir.EveDashboard/Program.cs b/src/Mimir.EveDashboard/Program.cs index ae2569f..b789e6d 100644 --- a/src/Mimir.EveDashboard/Program.cs +++ b/src/Mimir.EveDashboard/Program.cs @@ -21,9 +21,11 @@ var mimirObservationLogPath = ParseString(args, "--mimir-observation-log", @"E:\ var idunnRudpHealth = ParseString(args, "--idunn-rudp-health", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_RUDP_HEALTH") ?? ""); var idunnDaemon = ParseString(args, "--idunn-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_DAEMON") ?? "mimir-eve-dashboard"); var idunnHealthContract = ParseString(args, "--idunn-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_HEALTH_CONTRACT") ?? "mimir.cultnet-rudp-provider-health"); +var idunnServiceDaemon = ParseString(args, "--idunn-service-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_SERVICE_DAEMON") ?? ""); +var idunnServiceHealthContract = ParseString(args, "--idunn-service-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_SERVICE_HEALTH_CONTRACT") ?? ""); var providers = EveDashboardProviderCatalog.Create(ParseProviderSpecs(args), voidBotSwarmStatePath, mimirTelemetryLogPath, mimirObservationLogPath); using var cultNetPublisher = await EveDashboardCultNetPublisher.StartAsync(cultNetCachePath, cultNetPort).ConfigureAwait(false); -using var server = new EveDashboardServer(port, providers, cultNetPublisher, new IdunnRudpHealthOptions(idunnRudpHealth, idunnDaemon, idunnHealthContract)); +using var server = new EveDashboardServer(port, providers, cultNetPublisher, IdunnRudpHealthOptions.Create(idunnRudpHealth, idunnDaemon, idunnHealthContract, idunnServiceDaemon, idunnServiceHealthContract)); await server.RunAsync(); static int ParseInt(IReadOnlyList args, string name, int fallback) @@ -94,7 +96,7 @@ internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog p Console.WriteLine($"Mimir Eve dashboard CultNet publisher listening on cultnet://0.0.0.0:{cultNetPublisher.Port}"); Console.WriteLine(string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) ? "Mimir Eve dashboard Idunn RUDP health publisher disabled." - : $"Mimir Eve dashboard Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} as {idunnRudpHealth.DaemonId}."); + : $"Mimir Eve dashboard Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} for {idunnRudpHealth.Targets.Count} record(s)."); Console.WriteLine($"Compatibility endpoint remains ws://0.0.0.0:{port}/eve/dashboard"); _ = Task.Run(BroadcastHeartbeatAsync); while (!stopping.IsCancellationRequested) @@ -348,8 +350,7 @@ internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog p { return new IdunnRudpHealthStatus( idunnRudpHealth.Endpoint, - idunnRudpHealth.DaemonId, - idunnRudpHealth.HealthContract, + idunnRudpHealth.Targets, idunnRudpPublishStatus, idunnRudpPublishError, idunnRudpPublishInFlight, @@ -372,16 +373,18 @@ internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog p } var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); - var health = new IdunnDaemonHealthRecord + var healthRecords = idunnRudpHealth.Targets.Select(target => new IdunnDaemonHealthRecord { - DaemonId = idunnRudpHealth.DaemonId, + DaemonId = target.DaemonId, State = "healthy", - Detail = $"Mimir Eve dashboard broker active; provider={state.ProviderId}; providerVersion={state.Version}; providers={providers.Manifests.Count}; clients={clients.Count}; cultMeshPort={cultNetPublisher.Port}", + Detail = target.DetailPrefix == "service" + ? $"Nightwing Eve dashboard service active; provider={state.ProviderId}; providerVersion={state.Version}; providers={providers.Manifests.Count}; clients={clients.Count}; cultMeshPort={cultNetPublisher.Port}; process=Mimir.EveDashboard" + : $"Mimir Eve dashboard broker active; provider={state.ProviderId}; providerVersion={state.Version}; providers={providers.Manifests.Count}; clients={clients.Count}; cultMeshPort={cultNetPublisher.Port}", ObservedAt = observedAtText, - HealthContract = idunnRudpHealth.HealthContract, + HealthContract = target.HealthContract, PublicationSource = "daemon-published", Transport = "cultnet.transport.rudp.v0", - }; + }).ToArray(); lock (idunnRudpPublishLock) { @@ -404,7 +407,11 @@ internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog p { try { - IdunnRudpHealthPublisher.Publish(idunnRudpHealth.Endpoint, health); + foreach (var health in healthRecords) + { + IdunnRudpHealthPublisher.Publish(idunnRudpHealth.Endpoint, health); + } + lock (idunnRudpPublishLock) { idunnRudpPublishStatus = "published"; @@ -727,12 +734,34 @@ internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog p } } -internal sealed record IdunnRudpHealthOptions(string Endpoint, string DaemonId, string HealthContract); +internal sealed record IdunnRudpHealthOptions(string Endpoint, IReadOnlyList Targets) +{ + public static IdunnRudpHealthOptions Create( + string endpoint, + string daemonId, + string healthContract, + string serviceDaemonId, + string serviceHealthContract) + { + var targets = new List + { + new(daemonId, healthContract, "broker"), + }; + + if (!string.IsNullOrWhiteSpace(serviceDaemonId) && !string.IsNullOrWhiteSpace(serviceHealthContract)) + { + targets.Add(new IdunnRudpHealthTarget(serviceDaemonId, serviceHealthContract, "service")); + } + + return new IdunnRudpHealthOptions(endpoint, targets); + } +} + +internal sealed record IdunnRudpHealthTarget(string DaemonId, string HealthContract, string DetailPrefix); internal sealed record IdunnRudpHealthStatus( string Endpoint, - string Daemon, - string Contract, + IReadOnlyList Targets, string Status, string Error, bool InFlight, From 7d07e38e52aa1ea0862cedd2a8710f6440931e46 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 16 Jun 2026 09:50:06 +0100 Subject: [PATCH 044/135] Run Nightwing Eve browser reference as RUDP daemon --- .../Mimir.EveBrowserReference.csproj | 16 + src/Mimir.EveBrowserReference/Program.cs | 375 ++++++++++++++++++ src/Mimir.EveDashboard/Program.cs | 1 + 3 files changed, 392 insertions(+) create mode 100644 src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj create mode 100644 src/Mimir.EveBrowserReference/Program.cs diff --git a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj new file mode 100644 index 0000000..dd71bc2 --- /dev/null +++ b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj @@ -0,0 +1,16 @@ + + + + Exe + net10.0 + Mimir.EveBrowserReference + Mimir.EveBrowserReference + enable + enable + + + + + + + diff --git a/src/Mimir.EveBrowserReference/Program.cs b/src/Mimir.EveBrowserReference/Program.cs new file mode 100644 index 0000000..3ae661e --- /dev/null +++ b/src/Mimir.EveBrowserReference/Program.cs @@ -0,0 +1,375 @@ +using System.Globalization; +using System.Net; +using System.Net.Sockets; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using GameCult.Networking; +using MessagePack; + +var port = ParseInt(args, "--port", 8891); +var bind = ParseString(args, "--bind", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_BIND") ?? "0.0.0.0"); +var directory = ParseString(args, "--directory", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_DIRECTORY") ?? Directory.GetCurrentDirectory()); +var idunnRudpHealth = ParseString(args, "--idunn-rudp-health", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_RUDP_HEALTH") ?? ""); +var idunnDaemon = ParseString(args, "--idunn-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_DAEMON") ?? "nightwing-eve-browser-reference"); +var idunnHealthContract = ParseString(args, "--idunn-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_HEALTH_CONTRACT") ?? "nightwing.cultnet-rudp-browser-reference-health"); + +using var server = new EveBrowserReferenceServer( + IPAddress.Parse(bind), + port, + Path.GetFullPath(directory), + new IdunnRudpHealthOptions(idunnRudpHealth, idunnDaemon, idunnHealthContract)); +await server.RunAsync().ConfigureAwait(false); + +static int ParseInt(IReadOnlyList args, string name, int fallback) +{ + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && + int.TryParse(args[index + 1], CultureInfo.InvariantCulture, out var value)) + { + return value; + } + } + + return fallback; +} + +static string ParseString(IReadOnlyList args, string name, string fallback) +{ + for (var index = 0; index < args.Count - 1; index++) + { + if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && + !string.IsNullOrWhiteSpace(args[index + 1])) + { + return args[index + 1]; + } + } + + return fallback; +} + +internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, string documentRoot, IdunnRudpHealthOptions idunnRudpHealth) : IDisposable +{ + private static readonly JsonSerializerOptions JsonOptions = new() + { + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + }; + + private readonly TcpListener listener = new(bindAddress, port); + private readonly CancellationTokenSource stopping = new(); + private readonly object idunnRudpPublishLock = new(); + private string idunnRudpPublishStatus = string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) ? "disabled" : "idle"; + private string idunnRudpPublishError = ""; + private string idunnRudpPublishObservedAt = ""; + private bool idunnRudpPublishInFlight; + private DateTimeOffset lastHealthPublishAttemptAt = DateTimeOffset.MinValue; + + public async Task RunAsync() + { + if (!Directory.Exists(documentRoot)) + { + throw new DirectoryNotFoundException($"Eve browser reference document root does not exist: {documentRoot}"); + } + + listener.Start(); + Console.WriteLine($"Mimir Eve browser reference serving {documentRoot} on http://{bindAddress}:{port}/"); + Console.WriteLine(string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) + ? "Mimir Eve browser reference Idunn RUDP health publisher disabled." + : $"Mimir Eve browser reference Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} as {idunnRudpHealth.DaemonId}."); + _ = Task.Run(PublishHealthLoopAsync); + while (!stopping.IsCancellationRequested) + { + var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false); + _ = Task.Run(() => HandleAsync(client)); + } + } + + private async Task PublishHealthLoopAsync() + { + using var timer = new PeriodicTimer(TimeSpan.FromSeconds(1)); + while (!stopping.IsCancellationRequested && await timer.WaitForNextTickAsync(stopping.Token).ConfigureAwait(false)) + { + QueueIdunnRudpHealthIfDue(DateTimeOffset.UtcNow); + } + } + + private async Task HandleAsync(TcpClient tcpClient) + { + await using var stream = tcpClient.GetStream(); + var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false); + if (request.Path == "/health") + { + var health = JsonSerializer.Serialize(new + { + ok = true, + documentRoot, + port, + transport = "static-http-lowering", + idunnRudpHealth = GetIdunnRudpHealthStatus(), + }, JsonOptions); + await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false); + return; + } + + var filePath = ResolveStaticFilePath(request.Path); + if (filePath == null) + { + await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false); + return; + } + + var bytes = await File.ReadAllBytesAsync(filePath, stopping.Token).ConfigureAwait(false); + await WriteHttpResponseAsync(stream, "200 OK", ContentType(filePath), bytes).ConfigureAwait(false); + } + + private IdunnRudpHealthStatus GetIdunnRudpHealthStatus() + { + lock (idunnRudpPublishLock) + { + return new IdunnRudpHealthStatus( + idunnRudpHealth.Endpoint, + idunnRudpHealth.DaemonId, + idunnRudpHealth.HealthContract, + idunnRudpPublishStatus, + idunnRudpPublishError, + idunnRudpPublishInFlight, + idunnRudpPublishObservedAt); + } + } + + private void QueueIdunnRudpHealthIfDue(DateTimeOffset observedAt) + { + if (string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint)) + { + idunnRudpPublishStatus = "disabled"; + return; + } + + if (lastHealthPublishAttemptAt != DateTimeOffset.MinValue && + observedAt - lastHealthPublishAttemptAt < TimeSpan.FromSeconds(5)) + { + return; + } + + var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); + var health = new IdunnDaemonHealthRecord + { + DaemonId = idunnRudpHealth.DaemonId, + State = "healthy", + Detail = $"Nightwing Eve browser reference serving static lowering; root={documentRoot}; port={port}", + ObservedAt = observedAtText, + HealthContract = idunnRudpHealth.HealthContract, + PublicationSource = "daemon-published", + Transport = "cultnet.transport.rudp.v0", + }; + + lock (idunnRudpPublishLock) + { + if (idunnRudpPublishInFlight) + { + return; + } + + idunnRudpPublishInFlight = true; + lastHealthPublishAttemptAt = observedAt; + idunnRudpPublishObservedAt = observedAtText; + if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) + { + idunnRudpPublishStatus = "publishing"; + idunnRudpPublishError = ""; + } + } + + _ = Task.Run(() => + { + try + { + IdunnRudpHealthPublisher.Publish(idunnRudpHealth.Endpoint, health); + lock (idunnRudpPublishLock) + { + idunnRudpPublishStatus = "published"; + idunnRudpPublishError = ""; + idunnRudpPublishInFlight = false; + } + } + catch (Exception error) + { + lock (idunnRudpPublishLock) + { + if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) + { + idunnRudpPublishStatus = "publish-error"; + } + + idunnRudpPublishError = error.Message; + idunnRudpPublishInFlight = false; + } + } + }); + } + + private string? ResolveStaticFilePath(string path) + { + var requestPath = path.Split('?', 2)[0]; + if (requestPath == "/") + { + requestPath = "/index.html"; + } + + var relativePath = Uri.UnescapeDataString(requestPath.TrimStart('/')) + .Replace('/', Path.DirectorySeparatorChar); + var candidate = Path.GetFullPath(Path.Combine(documentRoot, relativePath)); + if (!candidate.StartsWith(documentRoot.TrimEnd(Path.DirectorySeparatorChar) + Path.DirectorySeparatorChar, StringComparison.Ordinal) && + !string.Equals(candidate, documentRoot, StringComparison.Ordinal)) + { + return null; + } + + return File.Exists(candidate) ? candidate : null; + } + + private static string ContentType(string path) => + Path.GetExtension(path).ToLowerInvariant() switch + { + ".html" => "text/html; charset=utf-8", + ".css" => "text/css; charset=utf-8", + ".js" => "application/javascript; charset=utf-8", + ".json" => "application/json; charset=utf-8", + ".png" => "image/png", + ".jpg" or ".jpeg" => "image/jpeg", + ".svg" => "image/svg+xml", + _ => "application/octet-stream", + }; + + private static async Task ReadHttpRequestAsync(NetworkStream stream) + { + var buffer = new byte[8192]; + var length = await stream.ReadAsync(buffer).ConfigureAwait(false); + if (length == 0) + { + return new HttpRequest("/"); + } + + var text = Encoding.ASCII.GetString(buffer, 0, length); + var firstLineEnd = text.IndexOf("\r\n", StringComparison.Ordinal); + var firstLine = firstLineEnd >= 0 ? text[..firstLineEnd] : text; + var parts = firstLine.Split(' ', StringSplitOptions.RemoveEmptyEntries); + return new HttpRequest(parts.Length >= 2 ? parts[1] : "/"); + } + + private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body) + { + var header = Encoding.ASCII.GetBytes( + $"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n"); + await stream.WriteAsync(header).ConfigureAwait(false); + await stream.WriteAsync(body).ConfigureAwait(false); + } + + public void Dispose() + { + stopping.Cancel(); + listener.Stop(); + stopping.Dispose(); + } +} + +internal sealed record HttpRequest(string Path); + +internal sealed record IdunnRudpHealthOptions(string Endpoint, string DaemonId, string HealthContract); + +internal sealed record IdunnRudpHealthStatus( + string Endpoint, + string Daemon, + string Contract, + string Status, + string Error, + bool InFlight, + string ObservedAtUtc); + +[MessagePackObject(AllowPrivate = true)] +internal sealed class IdunnDaemonHealthRecord +{ + [Key(0)] public string DaemonId { get; set; } = "nightwing-eve-browser-reference"; + [Key(1)] public string State { get; set; } = "active"; + [Key(2)] public string Detail { get; set; } = ""; + [Key(3)] public string ObservedAt { get; set; } = ""; + [Key(4)] public string HealthContract { get; set; } = "nightwing.cultnet-rudp-browser-reference-health"; + [Key(5)] public string PublicationSource { get; set; } = "daemon-published"; + [Key(6)] public string Transport { get; set; } = "cultnet.transport.rudp.v0"; +} + +internal static class IdunnRudpHealthPublisher +{ + private const uint ConnectionId = 0x1d0d0001; + + public static void Publish(string endpoint, IdunnDaemonHealthRecord health) + { + var remote = ParseEndpoint(endpoint); + using var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp); + socket.Bind(new IPEndPoint(IPAddress.Any, 0)); + socket.Blocking = false; + socket.ReceiveTimeout = 100; + socket.SendTimeout = 1000; + using var transport = new CultNetRudpSocketTransportConnection(new CultNetRudpSocketTransportOptions + { + RuntimeId = "mimir-eve-browser-reference", + Socket = socket, + Mode = CultNetRudpSocketMode.Client, + RemoteEndPoint = remote, + ConnectionId = ConnectionId, + InitialSequence = 1, + ResendDelayMs = 100, + }); + + if (!transport.ConnectAndWait(Array.Empty(), TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(20))) + { + throw new TimeoutException($"timed out connecting Mimir Eve browser reference RUDP health publisher to {endpoint}"); + } + + var observedAt = string.IsNullOrWhiteSpace(health.ObservedAt) + ? DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture) + : health.ObservedAt; + health.ObservedAt = observedAt; + var message = new CultNetDocumentPutRawMessage + { + MessageId = $"mimir-eve-browser-reference-health:{health.DaemonId}:{observedAt.Replace(':', '-')}", + Document = new CultNetRawDocumentRecord + { + SchemaId = "idunn.daemon_health", + RecordKey = health.DaemonId, + StoredAt = observedAt, + PayloadEncoding = "messagepack", + Payload = MessagePackSerializer.Serialize(health, CultNetSchemaMessageSerialization.Options), + SourceRuntimeId = "mimir-eve-browser-reference", + SourceRole = "daemon-health-publisher", + Tags = ["cultnet.transport.rudp.v0"], + }, + }; + transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message)); + } + + private static IPEndPoint ParseEndpoint(string value) + { + var trimmed = value.Trim(); + var separator = trimmed.LastIndexOf(':'); + if (separator <= 0 || separator == trimmed.Length - 1) + { + throw new ArgumentException($"Idunn RUDP endpoint must be host:port, got '{value}'.", nameof(value)); + } + + var host = trimmed[..separator]; + var portText = trimmed[(separator + 1)..]; + if (!int.TryParse(portText, CultureInfo.InvariantCulture, out var port)) + { + throw new ArgumentException($"Idunn RUDP endpoint port must be numeric, got '{value}'.", nameof(value)); + } + + var addresses = Dns.GetHostAddresses(host); + var address = addresses.FirstOrDefault(static candidate => candidate.AddressFamily == AddressFamily.InterNetwork) + ?? throw new ArgumentException($"Idunn RUDP endpoint host did not resolve to IPv4, got '{value}'.", nameof(value)); + return new IPEndPoint(address, port); + } +} diff --git a/src/Mimir.EveDashboard/Program.cs b/src/Mimir.EveDashboard/Program.cs index b789e6d..8ac03d9 100644 --- a/src/Mimir.EveDashboard/Program.cs +++ b/src/Mimir.EveDashboard/Program.cs @@ -790,6 +790,7 @@ internal static class IdunnRudpHealthPublisher socket.Bind(new IPEndPoint(IPAddress.Any, 0)); socket.Blocking = false; socket.ReceiveTimeout = 100; + socket.SendTimeout = 1000; using var transport = new CultNetRudpSocketTransportConnection(new CultNetRudpSocketTransportOptions { RuntimeId = "mimir-eve-dashboard", From 434e2d3efa2fb5731df686622e3c5f1e9106b9ea Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 16 Jun 2026 17:39:33 +0100 Subject: [PATCH 045/135] Reorient Mimir maps around CultNet RUDP --- docs/implementation-plan.md | 51 ++++++++++++++++++++++-------- notes/current-system-map.md | 53 ++++++++++++++++++++++++++++++++ notes/fresh-workspace-handoff.md | 23 +++++++++++++- state/evidence.jsonl | 3 ++ state/map.yaml | 38 ++++++++++++++++++++++- 5 files changed, 154 insertions(+), 14 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 226ec76..0627804 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -283,13 +283,27 @@ a named invariant that the native runtime cannot protect yet. MPEG-TS byte stream from stdin and publishes rolling `mimir.cultmesh_media_frame` documents, and `recv` subscribes to those documents and writes ordered MPEG-TS bytes to a Starfire-local UDP endpoint - for compatibility sinks. + for compatibility sinks. Source audit: its C# implementation still uses older + `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` entrypoints; migrate the + bridge to CultLib's explicit C# RUDP helpers before treating it as fully + aligned with the daemon-swarm transport cut. - `scripts/start-raven-cultmesh-av-sender.ps1`, `scripts/start-yggdrasil-cultmesh-media-relay.ps1`, and `scripts/start-starfire-cultmesh-av-receiver.ps1` are the CultMesh bridge operators. Raven capture defaults to FFmpeg desktop frames plus Mimir's WASAPI loopback capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains an explicit fallback. +- `src/Mimir.EveDashboard` is a CultMesh/Eve dashboard broker. It publishes + `mimir.eve_dashboard_state` through CultNet/CultMesh and can report + `idunn.daemon_health` over `cultnet.transport.rudp.v0` for itself and an + optional paired service daemon. The live direction is daemon-owned RUDP + publication for provider advertisement, retained state, command boundary, and + transport profile too; HTTP/WebSocket are client lowerings. This is a health + witness for Idunn, not a lifecycle owner. +- `src/Mimir.EveBrowserReference` serves static browser lowerings and can + publish its own `idunn.daemon_health` record over + `cultnet.transport.rudp.v0`. It remains a renderer/reference surface, not + program authority. - Documentation for OBS receiver setup, native rebuild boundaries, the viable stream app, and the Mimir Face. @@ -301,6 +315,16 @@ a named invariant that the native runtime cannot protect yet. because OBS is not a CultMesh consumer. Network transit between Raven, Yggdrasil, and Starfire is the CultMesh/CultNet path; OBS-local UDP is an egress adapter only. +- CultLib RUDP is the default typed CultNet/CultMesh document transport for + daemon truth. Idunn RUDP health publication is the current freshness witness; + provider advertisements, command boundaries, transport profiles, and retained + daemon state should follow the same path. Odin still owns Verse/service + discovery, Idunn owns keepalive decisions, and Mimir-owned dashboard/reference + surfaces only report their own observed state. TCP/HTTP/WebSocket stay + lowerings or compatibility evidence. +- `Mimir.CultMeshMedia` still needs an explicit RUDP transport cut. Do not add + another TCP bridge, HTTP status shim, or WebSocket-derived service truth while + the C# RUDP helpers exist. - Process-backed stream sources are only acceptable for network bridge feeds or diagnostics. Six-camera local ingest belongs behind direct capture drivers. - Frame-event process sources are diagnostic only. They prove source cadence and @@ -313,37 +337,40 @@ a named invariant that the native runtime cannot protect yet. 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the other cameras. -2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained +2. Cut `src/Mimir.CultMeshMedia` from older CultMesh node/client entrypoints to + explicit CultNet RUDP helpers, preferably authorized-peer dialing where the + peer catalog exists. Preserve OBS-local UDP as egress only. +3. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained frame cadence in the rolling buffers. -3. Promote the packet-song physical calibration receipt into the runtime +4. Promote the packet-song physical calibration receipt into the runtime receiver. The live decoder should keep its ear open for self-identifying song contours, extract intra-call time/frequency anchors from log-mel parts, apply learned per-output/mic path weighting, and feed a global delay/clock/path hypothesis. Keep chirp-bin calibration artifacts as reference data, not the runtime target. -4. Add the synchronization actuator: drive a variable-rate resampler and +5. Add the synchronization actuator: drive a variable-rate resampler and fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. First, prove the bioacoustic motif decoder through real loopback and microphone paths so every correctly heard word becomes a deterministic timeline anchor before the actuator moves samples. -5. Prove the bioacoustic hybrid fallback through real loopback and microphones +6. Prove the bioacoustic hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. -6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, +7. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. -7. Implement the Mimir program scene graph as the shared commit primitive for +8. Implement the Mimir program scene graph as the shared commit primitive for source subscription, transforms, crop, chroma key, visibility, layer order, preview, and output publication. Import the current OBS scene only as an initial mirror, then make Eve GUI/TUI the operator surface. -8. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir +9. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir program output and publishes it without owning a second composition. -9. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: +10. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: sample buffer-domain paths stochastically by visual contribution, emit SDF splat probes, write them into the spatiotemporal splat reservoir, and sample that reservoir in the temporally antialiased scene pass. The direct spline preview must stay a witness until this path owns rendering. -10. Move GPU feature extraction, fusion, material fitting, render budgeting, and +11. Move GPU feature extraction, fusion, material fitting, render budgeting, and Spout2 publication into Fensalir. -11. Move mic alignment, room suppression, voice separation, spatialization, and +12. Move mic alignment, room suppression, voice separation, spatialization, and stem generation into Faust/native DSP. -12. Keep the OBS bridge witness ledger as evidence before expanding receiver +13. Keep the OBS bridge witness ledger as evidence before expanding receiver machinery. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 0aedb80..8c4ed75 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -53,6 +53,11 @@ Ownership: Fensalir-facing wand pose stream. - Mimir owns program composition, source subscription policy, preview/control state, stats, and publication intent. +- Mimir-owned Eve dashboard and browser reference processes publish their own + typed health/freshness to Idunn over RUDP when configured. Their intended + daemon-owned provider state, command boundaries, and transport profiles also + belong on CultNet/RUDP; HTTP/WebSocket are client lowerings and compatibility + evidence, not daemon truth. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, D3D12 interop, runtime UI lowering, and local program texture output. - Faust/native DSP owns hot audio alignment, suppression, separation, @@ -252,6 +257,14 @@ video archive. Yggdrasil relays CultNet reliable UDP over the WireGuard mesh. Starfire lowers the subscribed stream to local UDP for compatibility sinks because OBS is not a CultMesh runtime. +Source audit correction: the desired transport is explicit CultNet RUDP, and +CultLib now provides C# `CultMesh.CreateRudpServer`, +`CultMesh.CreateRudpClient`, and authorized-peer helper APIs. The current +`Mimir.CultMeshMedia` source still uses older `CultMesh.StartNodeAsync` and +`CultMesh.ConnectClient` entrypoints. Treat that as migration debt: the bridge +should be cut to explicit RUDP helpers before it is considered aligned with the +daemon-swarm transport doctrine. + Current deployment: Yggdrasil runs the relay from `/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia` with cache `/var/lib/gamecult/mimir/cultmesh-media.cc` and log @@ -264,6 +277,46 @@ Raven SSH over WireGuard timed out during the initial deployment pass, so the Raven sender still needs to be launched on Raven directly with `scripts/start-raven-cultmesh-av-sender.ps1`. +## Daemon Health And Idunn + +Odin owns the accepted Verse/service map. Idunn owns continuity after a daemon +is known: boot recovery, crash recovery, deployment freshness, stale-health +watching, restart/deploy intent, and operator escalation. Mimir does not run a +private lifecycle supervisor for these surfaces. + +CultLib's current direction is RUDP everywhere for typed CultNet/CultMesh +documents across runtimes. That means health, provider advertisements, command +boundaries, transport profiles, and selected program/media document lanes should +default to `cultnet.transport.rudp.v0`. TCP, HTTP, and WebSocket are acceptable +as renderer/client lowerings, debug tools, or migration debt; they must not own +daemon truth once an RUDP contract exists. + +Mimir's current daemon-health publishers are local RUDP witnesses: + +- `src/Mimir.EveDashboard` publishes `mimir.eve_dashboard_state` through + CultNet/CultMesh and, when `--idunn-rudp-health` or + `MIMIR_EVE_DASHBOARD_IDUNN_RUDP_HEALTH` is set, sends + `idunn.daemon_health` records tagged `cultnet.transport.rudp.v0` for the + dashboard broker and optionally a paired service daemon. The next daemon + boundary work is to publish its provider advertisement, retained state, + command boundary, and transport profile as typed RUDP records so Odin can + prefer those over compatibility HTTP catalog ingestion. +- `src/Mimir.EveBrowserReference` serves a static browser lowering and, when + configured with `--idunn-rudp-health` or + `MIMIR_EVE_BROWSER_REFERENCE_IDUNN_RUDP_HEALTH`, sends its own + `idunn.daemon_health` record over the same RUDP document path. + +Invariant: health publication is not ownership. The daemon reports what it can +honestly observe about itself; Idunn decides keepalive action through the +shared typed record path. + +Source audit debt: `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, and the +browser/dashboard WebSocket paths are still renderer/client lowerings. They may +remain as lowering adapters, but any provider catalog, retained service state, +command boundary, transport profile, or lifecycle health claim must be +published as typed CultNet/RUDP records instead of being inferred from those +TCP/WebSocket surfaces. + ## Audio Field The six-microphone path is separate from the bridge. Scarlett speaker loopback diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 71c4569..5982226 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -25,7 +25,19 @@ Get-Content .\state\evidence.jsonl -Tail 8 - V1 now also has an explicit CultMesh media bridge for Raven program feeds: `src/Mimir.CultMeshMedia` sends rolling `mimir.cultmesh_media_frame` documents over CultNet reliable UDP through Yggdrasil and lowers them on - Starfire to local MPEG-TS UDP for OBS. + Starfire to local MPEG-TS UDP for OBS. Source audit warning: the current C# + bridge still uses older `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient`; + migrate it to explicit C# RUDP helpers before treating the media bridge as + fully aligned with the swarm transport cut. +- CultLib has moved the daemon-swarm direction to CultNet over RUDP across + runtimes. Mimir's typed daemon truth should default to + `cultnet.transport.rudp.v0`; TCP/HTTP/WebSocket are client lowerings, + compatibility probes, or migration debt unless a specific runtime has not + earned RUDP yet. +- Mimir has daemon-health publication edges into the Odin/Idunn swarm: + `src/Mimir.EveDashboard` and `src/Mimir.EveBrowserReference` publish + `idunn.daemon_health` records over `cultnet.transport.rudp.v0` when + configured. These are self-health witnesses, not lifecycle owners. - The live stream app is C# plus Fensalir: `Mimir.slnx` contains `src/Mimir.App` and `src/Mimir.Runtime`. - `src/Mimir.App` hosts Fensalir as the windowing/rendering/D3D12 bridge. @@ -85,6 +97,15 @@ Get-Content .\state\evidence.jsonl -Tail 8 `Raven Monitor + Realtek` has been repointed from old SRT to that local UDP endpoint. Raven SSH timed out, so launch `scripts/start-raven-cultmesh-av-sender.ps1` on Raven directly. +- Keep Eve dashboard/browser-reference RUDP health aligned with Odin/Idunn: + Odin accepts the Verse/service map, Idunn owns daemon continuity, and Mimir + processes publish typed health/freshness only for themselves. The next + dashboard boundary cut is provider advertisement, retained state, + command_boundary, and transport_profile over RUDP so HTTP/WebSocket catalog + ingestion can be demoted to client/debug lowering. +- Cut `Mimir.CultMeshMedia` to explicit CultNet RUDP helpers next; no new TCP, + HTTP, or WebSocket surface should be allowed to become daemon or transport + truth. - Kiyo Pro has two UVC extension units. Moving it to a motherboard USB3 port changed the descriptor path to `root_hub30` / `bcdUSB=0x0320` and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports diff --git a/state/evidence.jsonl b/state/evidence.jsonl index fc13f1c..6ea427a 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -139,3 +139,6 @@ {"ts": "2026-06-12T15:05:00+01:00", "type": "correction", "status": "ok", "note": "Updated the Move calibration preflight witness from Periwinkle to the USB-connected Quest headset plus left/right controllers. ADB sees serial 1WMHHB68PG1515 but reports unauthorized, so headset-side USB debugging approval is required before Quest pose capture can become live evidence. The protocol treats Quest poses as an optional external VR reference frame around the Move, not as a Mimir fusion owner."} {"ts":"2026-06-12T15:17:22.2571251+01:00","type":"verification","status":"ok","note":"Quest USB access is now authorized for calibration preflight: adb devices -l reports serial 1WMHHB68PG1515 as Quest_2/hollywood. Added and ran --quest-usb-preflight-smoke, which wrote mimir.quest_usb_preflight.v1 with poseCaptureStatus=usb-authorized-no-pose-bridge-yet. ADB access proves USB/device identity only; headset/controller pose samples still require a Quest/OpenXR witness bridge before Mimir can use Quest as external calibration evidence."} {"ts":"2026-06-12T16:05:00+01:00","type":"correction","status":"ok","note":"Moved Quest access ownership out of Mimir and under Odin/Muninn. Mimir's local mimir.quest_usb_preflight path was deleted; Mimir now treats muninn.quest_access.v1 and later muninn.quest_pose_frame.v1 as input evidence. Brokkr owns Unity editor routing for Quest input/pose consumption and warp-corrected Unity play-mode video toward Muninn's Quest video input surface."} +{"ts":"2026-06-16T00:00:00+01:00","type":"correction","status":"ok","note":"Reoriented Mimir's daemon-swarm map after Odin/CultLib RUDP work: Raven media body transport remains src/Mimir.CultMeshMedia over CultNet/CultMesh reliable UDP with OBS-local UDP as egress only, while Eve dashboard/browser reference health now publishes idunn.daemon_health over cultnet.transport.rudp.v0 as self-witness state. Odin owns accepted Verse/service discovery and Idunn owns keepalive decisions; Mimir health publishers do not own lifecycle."} +{"ts":"2026-06-16T00:05:00+01:00","type":"correction","status":"ok","note":"Corrected the transport model: CultLib has added authorized RUDP helpers across runtimes and Odin/Idunn now treats daemon truth as typed CultCache/CultMesh state carried over cultnet.transport.rudp.v0. Mimir should not assume TCP is needed for any daemon-truth path once the runtime has RUDP support; HTTP/WebSocket/TCP are client lowerings, compatibility probes, or migration debt. Health is separate from media by authority, not because it needs a separate TCP transport."} +{"ts":"2026-06-16T00:12:00+01:00","type":"correction","status":"ok","note":"Source audit found Mimir.CultMeshMedia still using older C# CultMesh.StartNodeAsync/CultMesh.ConnectClient entrypoints while CultLib now exposes C# CultMesh.CreateRudpServer/CreateRudpClient/CreateRudpClientForAuthorizedPeer. Updated maps to treat the media bridge's current transport path as migration debt: typed program/media documents should move to explicit CultNet RUDP helpers, with OBS-local UDP preserved only as compatibility egress."} diff --git a/state/map.yaml b/state/map.yaml index 38460ce..89ccbb1 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -40,6 +40,18 @@ constraints: - CultMesh/CultNet is the default Raven-to-Starfire media transit. OBS may consume a Starfire-local UDP MPEG-TS egress as a compatibility sink, but Mimir owns program selection, composition, preview/control, and publication. + - CultLib is moving every runtime's typed CultNet/CultMesh document transport + to `cultnet.transport.rudp.v0`. Health, provider advertisements, command + boundaries, transport profiles, and program/media document lanes should use + RUDP by default. TCP/HTTP/WebSocket surfaces are renderer lowerings, + compatibility probes, or migration debt unless a specific runtime has not + earned its RUDP path yet. + - Daemon health and freshness are separate in authority from media transport, + not separate because they need TCP. Mimir-facing Eve dashboards and browser + reference lowerers publish `idunn.daemon_health` records over + `cultnet.transport.rudp.v0` for Idunn. Idunn owns continuity once Odin has + accepted a daemon; Mimir-owned surfaces only publish their own health + witnesses. - Mimir explicitly trades latency and memory for throughput and field quality: the default five-second buffer is compute budget for stream alignment, GPU/audio analysis, and volumetric sound-field extraction. @@ -371,7 +383,23 @@ accepted_design: `mimir.cultmesh_media_frame` documents over CultNet reliable UDP, Yggdrasil relays them on `10.77.0.1:3075`, and Starfire lowers the subscribed stream to `udp://127.0.0.1:5200` for the OBS source - `Raven Monitor + Realtek`. + `Raven Monitor + Realtek`. Source audit warning: the current C# bridge + still calls older `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` + entrypoints; now that CultLib exposes C# RUDP facade helpers, the next + transport cut should move this bridge onto explicit RUDP server/client or + authorized-peer helpers and demote any TCP-like node/client path to + compatibility evidence. + - `src/Mimir.EveDashboard` publishes `mimir.eve_dashboard_state` through + CultNet/CultMesh and can push `idunn.daemon_health` records over + `cultnet.transport.rudp.v0` for itself and a paired service daemon. Its + intended daemon shape is to publish provider advertisement, retained state, + command boundary, and transport profile over RUDP as typed records; HTTP and + WebSocket remain client lowerings. It owns Eve surface publication and + dashboard broker health only; it does not own daemon lifecycle. + - `src/Mimir.EveBrowserReference` serves a static Eve/browser lowering and + can publish its own `idunn.daemon_health` record over + `cultnet.transport.rudp.v0`. It is a renderer/reference health witness, not + an authority over Mimir program state or Idunn keepalive decisions. - `research/perfect-machine-study-2026-05-23/` preserves the architecture rumination, optimization ledger, research references, source/hotspot audits, Fensalir/native boundary maps, benchmark plan, calibration-session spec, @@ -400,6 +428,14 @@ active_subgoals: replaces them. - Launch Raven's CultMesh sender directly on Raven because SSH to Raven over WireGuard timed out during the initial relay deployment pass. + - Keep Mimir Eve dashboard and browser reference health publication aligned + with Odin/Idunn doctrine: Odin accepts Verse/service visibility, Idunn owns + keepalive, and Mimir surfaces publish typed RUDP self-health rather than + private watchdog loops or TCP/HTTP status truth. + - Cut remaining older CultMesh node/client transport assumptions in + `src/Mimir.CultMeshMedia`, `src/Mimir.EveDashboard`, and witness tooling: + daemon truth and typed program/media document paths should use explicit + CultNet RUDP helpers; HTTP/WebSocket remain render/client lowerings only. current_status: summary: > From bd19e60ae18b8f80b635522ba5fea4d4a8262cdc Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 16 Jun 2026 17:39:51 +0100 Subject: [PATCH 046/135] Refresh Mimir Face state --- .voidbot/state/mimir.cc | Bin 70551 -> 72151 bytes .voidbot/status/void-memory-maintenance.json | 2 +- .voidbot/voice/identity.json | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 55e560519c117f83a098e5074ef62788c59ea1c9..e244888bcaad747991f3bac35fe5a8afc23140ad 100644 GIT binary patch delta 1590 zcma)+UuYaf9LGHfMl?-DE=EloCnct_m>Ts5wsGPONj)*?ihMGkQ-~SB^42r2Qsa%{g&+A z)`g*e-qF8BfMO^>d+9*8vAJpGH`E$RoY3oXDq-rU)m-=Ydsc6~H2H?Utorbe$GRTV z>Wj;aH;AiRhXphe2(mJ?D>>cS0;Y_{nX&{%pVtBEY%U8 z*U5i~RWvQVJJ|D3M?COvOWQ`;D8;_L77Xql`QTpPF8CYW$ffp|!Hpq(=k%_gij9eb z2_eObt)HBq(l4Gn^vJu@)T07PMO_y)F;h}snSVlmqla|QTfGjc;RfXbl}UwzLDDC< z4s#$}#grk=5VmcYdkop2J$G7T2#Rga6lDQbt@J0hi>bHv`Qf1KQX7Gl)c{vX_)k^C z3Q2-J{pZ;!z5Mp-X!^V&EaYx&_9(dDnE_FWT-qRGs2r%!d5x$0^`&=4cQtw7Ix!G% zkAMr2Lu@Xvq799|>PyV(`8OZe;|mXTkH-z_zuw!Ict-PunPIDv&*n14LaJ=$aVm$4 zR?4z$Gi5oYawT8PWK2@gL(dQBtMBjai6PEdX{(geGe0DI?}8oe=@b8q=-I`4>NYVm z(nTX15p`>EZi6y2ZRX-ko?046Txox}bSOJ_I<`Xr&`P3dqOc9q?3E}=ODh#8gffCK z1SjGiai6fzlSoj9NFo}LDu@j@R0}-JB15pP5F7c0#+Hd zVdc{XR~`Ro=!f9kgHA##5F^A1eZh8}@uo|hst3VGyb@;^GY7@-gdXD75d_i$P~ckR zg)G^EHvu8QB}NA+>^@cBWWbN42uG*HbyCf6#9F|rL}=qo@_-|7wHoeGg(y=Trhz@J2C+h!Btx z;*3RS)(P>E49YTmaJ)@qIgkosM3#_OY!Dov4_NeAZMmv3KoxjJRr&(V5W=roD-*Xr vy7}O<{>OS|t^fZ~`pehG7D6*dNa)UOgl3HPwHxOTZm-(p(@f_}=J9_4Z@D#& delta 923 zcmZ8fO=uHA7%TiYhO?HH9UM+%QX_NAMz`vXEHyDJUB{kB`i@wY#RtN6z{4}Z1!2^9$iW6GC-s;3 zQheCetzN~B){b{YEoE#)BZJe)Og3d@)2T#i#8hL!{wDEYDkQF*AK#s90f*|{%0bK^ z>Q->dNUP65rSfy($FjrMAKQ%PL4>W(aeW3HHb%cZE53L#a^VqWMQ~;L|AkR&Xq)l@ zY-b#MKB0w;(XU%;f4;fb?z>pXDWA>5n0%TtQ$ulcD4yIdckW(TG#OB#z88Djn=Yw4 z)!wG$v{{#Yveb95zhkFCYyLl-Vx@Mu^fIPC-R@ECH#^mdpr?hxs#9dEl@_ew#Afw? z@w)b_dd51vFaZ+V981LY3J4VlBM&gubL9gU3BVp=d4#3ZE72CAJmYABc!VSAa26CC zeFE~;;h9*4(8doF5^{)mxB!W}wu_1PerZ}))e$6uLIH#nNODYt=JRl!xkRLce6Ag* znA(7}C Date: Tue, 16 Jun 2026 18:08:40 +0100 Subject: [PATCH 047/135] Migrate Mimir media bridge to RUDP --- Mimir.slnx | 1 + docs/implementation-plan.md | 23 +- notes/current-system-map.md | 16 +- notes/fresh-workspace-handoff.md | 14 +- scripts/start-raven-cultmesh-av-sender.ps1 | 89 +++ .../start-starfire-cultmesh-av-receiver.ps1 | 10 + .../start-yggdrasil-cultmesh-media-relay.ps1 | 8 + .../Mimir.CultMeshMedia.csproj | 18 + src/Mimir.CultMeshMedia/Program.cs | 568 ++++++++++++++++++ .../MimirCultMeshMediaFrameDocument.cs | 24 + .../MimirNetworkTransportConfiguration.cs | 17 +- state/evidence.jsonl | 1 + state/map.yaml | 18 +- 13 files changed, 771 insertions(+), 36 deletions(-) create mode 100644 scripts/start-raven-cultmesh-av-sender.ps1 create mode 100644 scripts/start-starfire-cultmesh-av-receiver.ps1 create mode 100644 scripts/start-yggdrasil-cultmesh-media-relay.ps1 create mode 100644 src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj create mode 100644 src/Mimir.CultMeshMedia/Program.cs create mode 100644 src/Mimir.Runtime/Synchronization/MimirCultMeshMediaFrameDocument.cs diff --git a/Mimir.slnx b/Mimir.slnx index dbd4f1b..a0de91f 100644 --- a/Mimir.slnx +++ b/Mimir.slnx @@ -6,5 +6,6 @@ + diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 0627804..c100387 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -279,14 +279,14 @@ a named invariant that the native runtime cannot protect yet. controls/previews/stats, Yggdrasil publishes to the site, and OBS is compatibility-only. - `src/Mimir.CultMeshMedia` is the first explicit CultMesh media/body bridge: - `relay` hosts the document relay on CultNet UDP `3075`, `send` reads an + `relay` hosts the document relay on CultNet RUDP `3075`, `send` reads an MPEG-TS byte stream from stdin and publishes rolling `mimir.cultmesh_media_frame` documents, and `recv` subscribes to those documents and writes ordered MPEG-TS bytes to a Starfire-local UDP endpoint - for compatibility sinks. Source audit: its C# implementation still uses older - `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` entrypoints; migrate the - bridge to CultLib's explicit C# RUDP helpers before treating it as fully - aligned with the daemon-swarm transport cut. + for compatibility sinks. Its C# implementation now uses explicit CultLib + RUDP client/session helpers; the older + `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` entrypoints no longer own + the media transport lane. - `scripts/start-raven-cultmesh-av-sender.ps1`, `scripts/start-yggdrasil-cultmesh-media-relay.ps1`, and `scripts/start-starfire-cultmesh-av-receiver.ps1` are the CultMesh bridge @@ -322,9 +322,9 @@ a named invariant that the native runtime cannot protect yet. discovery, Idunn owns keepalive decisions, and Mimir-owned dashboard/reference surfaces only report their own observed state. TCP/HTTP/WebSocket stay lowerings or compatibility evidence. -- `Mimir.CultMeshMedia` still needs an explicit RUDP transport cut. Do not add - another TCP bridge, HTTP status shim, or WebSocket-derived service truth while - the C# RUDP helpers exist. +- `Mimir.CultMeshMedia` has completed its explicit RUDP transport cut. Do not + add another TCP bridge, HTTP status shim, or WebSocket-derived service truth + while the RUDP document lane exists. - Process-backed stream sources are only acceptable for network bridge feeds or diagnostics. Six-camera local ingest belongs behind direct capture drivers. - Frame-event process sources are diagnostic only. They prove source cadence and @@ -337,9 +337,10 @@ a named invariant that the native runtime cannot protect yet. 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the other cameras. -2. Cut `src/Mimir.CultMeshMedia` from older CultMesh node/client entrypoints to - explicit CultNet RUDP helpers, preferably authorized-peer dialing where the - peer catalog exists. Preserve OBS-local UDP as egress only. +2. Cut the remaining dashboard, recorder, and witness service truth paths from + older TCP/WebSocket-derived assumptions to explicit CultNet RUDP records, + preferably authorized-peer dialing where the peer catalog exists. Preserve + OBS-local UDP as egress only for compatibility sinks. 3. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained frame cadence in the rolling buffers. 4. Promote the packet-song physical calibration receipt into the runtime diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 8c4ed75..9d3d181 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -257,13 +257,12 @@ video archive. Yggdrasil relays CultNet reliable UDP over the WireGuard mesh. Starfire lowers the subscribed stream to local UDP for compatibility sinks because OBS is not a CultMesh runtime. -Source audit correction: the desired transport is explicit CultNet RUDP, and -CultLib now provides C# `CultMesh.CreateRudpServer`, -`CultMesh.CreateRudpClient`, and authorized-peer helper APIs. The current -`Mimir.CultMeshMedia` source still uses older `CultMesh.StartNodeAsync` and -`CultMesh.ConnectClient` entrypoints. Treat that as migration debt: the bridge -should be cut to explicit RUDP helpers before it is considered aligned with the -daemon-swarm transport doctrine. +Source audit correction resolved: `Mimir.CultMeshMedia` now uses explicit +CultLib RUDP helpers for the sender/receiver and a relay-owned RUDP +socket/session loop for multi-peer forwarding. The older +`CultMesh.StartNodeAsync` and `CultMesh.ConnectClient` path no longer owns this +media lane. The relay still persists received puts to CultCache as evidence and +for inspection; live network transit is the RUDP schema-document path. Current deployment: Yggdrasil runs the relay from `/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia` with cache @@ -315,7 +314,8 @@ browser/dashboard WebSocket paths are still renderer/client lowerings. They may remain as lowering adapters, but any provider catalog, retained service state, command boundary, transport profile, or lifecycle health claim must be published as typed CultNet/RUDP records instead of being inferred from those -TCP/WebSocket surfaces. +TCP/WebSocket surfaces. The media-frame bridge is already on that side of the +line. ## Audio Field diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 5982226..904eb1e 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -25,10 +25,10 @@ Get-Content .\state\evidence.jsonl -Tail 8 - V1 now also has an explicit CultMesh media bridge for Raven program feeds: `src/Mimir.CultMeshMedia` sends rolling `mimir.cultmesh_media_frame` documents over CultNet reliable UDP through Yggdrasil and lowers them on - Starfire to local MPEG-TS UDP for OBS. Source audit warning: the current C# - bridge still uses older `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient`; - migrate it to explicit C# RUDP helpers before treating the media bridge as - fully aligned with the swarm transport cut. + Starfire to local MPEG-TS UDP for OBS. The bridge now uses explicit CultLib + RUDP helpers/session plumbing; the old + `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` path no longer owns this + media lane. - CultLib has moved the daemon-swarm direction to CultNet over RUDP across runtimes. Mimir's typed daemon truth should default to `cultnet.transport.rudp.v0`; TCP/HTTP/WebSocket are client lowerings, @@ -103,9 +103,9 @@ Get-Content .\state\evidence.jsonl -Tail 8 dashboard boundary cut is provider advertisement, retained state, command_boundary, and transport_profile over RUDP so HTTP/WebSocket catalog ingestion can be demoted to client/debug lowering. -- Cut `Mimir.CultMeshMedia` to explicit CultNet RUDP helpers next; no new TCP, - HTTP, or WebSocket surface should be allowed to become daemon or transport - truth. +- Cut the remaining Eve dashboard, recorder, and witness service truth paths to + explicit CultNet RUDP records next; no new TCP, HTTP, or WebSocket surface + should be allowed to become daemon or transport truth. - Kiyo Pro has two UVC extension units. Moving it to a motherboard USB3 port changed the descriptor path to `root_hub30` / `bcdUSB=0x0320` and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports diff --git a/scripts/start-raven-cultmesh-av-sender.ps1 b/scripts/start-raven-cultmesh-av-sender.ps1 new file mode 100644 index 0000000..7dda8a9 --- /dev/null +++ b/scripts/start-raven-cultmesh-av-sender.ps1 @@ -0,0 +1,89 @@ +param( + [string]$RelayHost = "10.77.0.1", + [int]$RelayPort = 3075, + [string]$Stream = "raven-primary-av", + [string]$FfmpegPath = "ffmpeg", + [string]$VideoInput = "desktop", + [ValidateSet("WasapiLoopback", "DirectShow")] + [string]$AudioMode = "WasapiLoopback", + [string]$AudioInput = "Stereo Mix (Realtek(R) Audio)", + [ValidateSet("Console", "Multimedia", "Communications")] + [string]$LoopbackRole = "Console", + [int]$Width = 1920, + [int]$Height = 1080, + [int]$Framerate = 30, + [string]$VideoBitrate = "8000k", + [string]$AudioBitrate = "192k", + [string]$MediaToolPath = "" +) + +$ErrorActionPreference = "Stop" +$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj" +$publishedTool = Join-Path $PSScriptRoot "..\artifacts\mimir-cultmesh-media-win-x64-selfcontained\Mimir.CultMeshMedia.exe" +$loopbackScript = Join-Path $PSScriptRoot "wasapi-loopback-capture.ps1" + +if ([string]::IsNullOrWhiteSpace($MediaToolPath) -and (Test-Path $publishedTool)) { + $MediaToolPath = $publishedTool +} + +$ffmpegArgs = @( + "-hide_banner", + "-loglevel", "warning", + "-f", "gdigrab", + "-framerate", $Framerate, + "-video_size", "${Width}x${Height}", + "-i", $VideoInput +) + +if ($AudioMode -eq "WasapiLoopback") { + $ffmpegArgs += @( + "-f", "f32le", + "-ar", "48000", + "-ac", "2", + "-i", "pipe:0" + ) +} else { + $ffmpegArgs += @( + "-f", "dshow", + "-i", "audio=$AudioInput" + ) +} + +$ffmpegArgs += @( + "-map", "0:v:0", + "-map", "1:a:0", + "-c:v", "h264_nvenc", + "-preset", "p3", + "-tune", "ll", + "-b:v", $VideoBitrate, + "-maxrate", $VideoBitrate, + "-bufsize", "16000k", + "-g", ($Framerate * 2), + "-pix_fmt", "yuv420p", + "-c:a", "aac", + "-b:a", $AudioBitrate, + "-ar", "48000", + "-ac", "2", + "-f", "mpegts", + "pipe:1" +) + +$senderArgs = @("send", "--host", $RelayHost, "--port", $RelayPort, "--stream", $Stream, "--producer", "raven", "--chunk-bytes", "16384", "--slots", "96") + +if ($AudioMode -eq "WasapiLoopback") { + if (-not (Test-Path $loopbackScript)) { + throw "WASAPI loopback script not found: $loopbackScript" + } + + if ([string]::IsNullOrWhiteSpace($MediaToolPath)) { + & $loopbackScript -Output stdout -Role $LoopbackRole | & $FfmpegPath @ffmpegArgs | dotnet run --project $project -- @senderArgs + } else { + & $loopbackScript -Output stdout -Role $LoopbackRole | & $FfmpegPath @ffmpegArgs | & $MediaToolPath @senderArgs + } +} else { + if ([string]::IsNullOrWhiteSpace($MediaToolPath)) { + & $FfmpegPath @ffmpegArgs | dotnet run --project $project -- @senderArgs + } else { + & $FfmpegPath @ffmpegArgs | & $MediaToolPath @senderArgs + } +} diff --git a/scripts/start-starfire-cultmesh-av-receiver.ps1 b/scripts/start-starfire-cultmesh-av-receiver.ps1 new file mode 100644 index 0000000..67bad67 --- /dev/null +++ b/scripts/start-starfire-cultmesh-av-receiver.ps1 @@ -0,0 +1,10 @@ +param( + [string]$RelayHost = "10.77.0.1", + [int]$RelayPort = 3075, + [string]$Stream = "raven-primary-av", + [string]$Udp = "127.0.0.1:5200" +) + +$ErrorActionPreference = "Stop" +$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj" +dotnet run --project $project -- recv --host $RelayHost --port $RelayPort --stream $Stream --udp $Udp diff --git a/scripts/start-yggdrasil-cultmesh-media-relay.ps1 b/scripts/start-yggdrasil-cultmesh-media-relay.ps1 new file mode 100644 index 0000000..77c4126 --- /dev/null +++ b/scripts/start-yggdrasil-cultmesh-media-relay.ps1 @@ -0,0 +1,8 @@ +param( + [string]$Cache = "C:\Meta\Mimir\state\mimir-cultmesh-media.cc" +) + +$ErrorActionPreference = "Stop" +$project = Join-Path $PSScriptRoot "..\src\Mimir.CultMeshMedia\Mimir.CultMeshMedia.csproj" +New-Item -ItemType Directory -Force -Path (Split-Path -Parent $Cache) | Out-Null +dotnet run --project $project -- relay --cache $Cache diff --git a/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj b/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj new file mode 100644 index 0000000..cb154c1 --- /dev/null +++ b/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj @@ -0,0 +1,18 @@ + + + + Exe + net10.0 + Mimir.CultMeshMedia + Mimir.CultMeshMedia + enable + enable + + + + + + + + + diff --git a/src/Mimir.CultMeshMedia/Program.cs b/src/Mimir.CultMeshMedia/Program.cs new file mode 100644 index 0000000..a497083 --- /dev/null +++ b/src/Mimir.CultMeshMedia/Program.cs @@ -0,0 +1,568 @@ +using System.Diagnostics; +using System.Globalization; +using System.Net; +using System.Net.Sockets; +using GameCult.Caching; +using GameCult.Caching.MessagePack; +using GameCult.Mesh; +using GameCult.Networking; +using Mimir.Runtime.Synchronization; + +const string SchemaName = "mimir.cultmesh_media_frame"; +const string DefaultStreamId = "raven-primary-av"; +const uint DefaultConnectionId = 0x4d4d_0101; +const int DefaultPort = 3075; +const int DefaultMaxFragmentBytes = 1200; +const int DefaultMaxPendingReliablePackets = 8192; + +var command = args.Length > 0 ? args[0].Trim().ToLowerInvariant() : string.Empty; +var options = CliOptions.Parse(args.Skip(1)); + +return command switch +{ + "relay" => await RunRelayAsync(options), + "send" => await RunSenderAsync(options), + "recv" or "receive" => await RunReceiverAsync(options), + _ => Usage() +}; + +static int Usage() +{ + Console.Error.WriteLine(""" +Usage: + Mimir.CultMeshMedia relay --cache [--bind 0.0.0.0:3075] + Mimir.CultMeshMedia send --host [--port 3075] [--stream raven-primary-av] [--chunk-bytes 16384] [--slots 96] + Mimir.CultMeshMedia recv --host [--port 3075] [--stream raven-primary-av] [--udp 127.0.0.1:5200] + +send reads an MPEG-TS byte stream from stdin and publishes rolling CultMesh media-frame documents over CultNet RUDP. +recv subscribes to those RUDP document changes and writes ordered payload bytes to a local UDP MPEG-TS endpoint for OBS. +"""); + return 2; +} + +static async Task RunRelayAsync(CliOptions options) +{ + var cachePath = options.Get("cache") ?? Path.Combine(Environment.CurrentDirectory, "mimir-cultmesh-media.cc"); + var bind = ParseEndpoint(options.Get("bind") ?? $"0.0.0.0:{options.GetInt("port", DefaultPort)}"); + var registry = CreateDocumentRegistry(); + var schemaId = ResolveMediaSchemaId(registry); + using var cache = await CultCacheMessagePack.OpenAsync(cachePath).ConfigureAwait(false); + using var relay = new RudpMediaRelay( + bind, + cache, + registry, + schemaId, + options.GetUInt("connection-id", DefaultConnectionId), + options.GetInt("max-fragment-bytes", DefaultMaxFragmentBytes), + options.GetInt("max-pending-reliable", DefaultMaxPendingReliablePackets)); + + Console.WriteLine($"CultMesh media relay listening on rudp://{bind.Address}:{bind.Port}; cache={Path.GetFullPath(cachePath)}"); + await relay.RunAsync().ConfigureAwait(false); + await cache.FlushAsync().ConfigureAwait(false); + return 0; +} + +static async Task RunSenderAsync(CliOptions options) +{ + var endpoint = ResolveRudpEndpoint(options); + var streamId = options.Get("stream") ?? DefaultStreamId; + var chunkBytes = Math.Clamp(options.GetInt("chunk-bytes", 16 * 1024), 1316, 60 * 1024); + var slots = Math.Clamp(options.GetInt("slots", 96), 8, 4096); + var producer = options.Get("producer") ?? Environment.MachineName.ToLowerInvariant(); + var registry = CreateDocumentRegistry(); + + using var transport = CreateRudpClient( + options.Get("runtime") ?? $"{producer}-mimir-media-sender", + endpoint, + options); + ConnectOrThrow(transport, endpoint.Uri); + + Console.Error.WriteLine($"Sending stdin MPEG-TS to {endpoint.Uri}/{streamId}; chunkBytes={chunkBytes}; slots={slots}"); + + var input = Console.OpenStandardInput(); + var buffer = new byte[chunkBytes]; + long sequence = 0; + var started = Stopwatch.StartNew(); + + while (true) + { + var read = await input.ReadAsync(buffer.AsMemory(0, buffer.Length)).ConfigureAwait(false); + if (read <= 0) + { + break; + } + + var payload = buffer.AsSpan(0, read).ToArray(); + var document = new MimirCultMeshMediaFrameDocument( + FrameId: $"{streamId}:{sequence:D20}", + StreamId: streamId, + Sequence: sequence, + ProducedAtUtc: DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture), + TimestampNanoseconds: started.ElapsedTicks * 1_000_000_000L / Stopwatch.Frequency, + PayloadKind: "mpegts-chunk", + Container: "mpegts", + VideoCodec: options.Get("video-codec") ?? "h264", + AudioCodec: options.Get("audio-codec") ?? "aac", + PayloadBytes: payload.Length, + Payload: payload, + ProducerNode: producer, + ClockDomainId: options.Get("clock-domain") ?? "raven-monotonic", + Tags: ["raven", "program", "obs", "cultmesh-media", "cultnet.transport.rudp.v0"]); + + var handle = new CultRecordHandle( + new CultRecordKey($"{streamId}:slot:{sequence % slots:D4}")); + var message = registry.CreateRawDocumentPutMessage( + Guid.NewGuid().ToString("N"), + handle, + document, + new CultNetDocumentMessageOptions + { + SourceRuntimeId = producer, + SourceRole = "mimir-cultmesh-media-sender", + Tags = ["mimir", "media", streamId, "cultnet.transport.rudp.v0"] + }); + + transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message)); + PumpClient(transport, TimeSpan.FromMilliseconds(5)); + sequence++; + if (sequence % 120 == 0) + { + Console.Error.WriteLine($"sent sequence={sequence - 1}"); + } + } + + Console.Error.WriteLine($"sent chunks={sequence}; draining"); + PumpClient(transport, TimeSpan.FromMilliseconds(options.GetInt("drain-ms", 1000))); + return 0; +} + +static async Task RunReceiverAsync(CliOptions options) +{ + var endpoint = ResolveRudpEndpoint(options); + var streamId = options.Get("stream") ?? DefaultStreamId; + var udp = options.Get("udp") ?? "127.0.0.1:5200"; + var udpEndpoint = ParseEndpoint(udp); + var registry = CreateDocumentRegistry(); + var schemaId = ResolveMediaSchemaId(registry); + var lastSequence = -1L; + long frames = 0; + long bytes = 0; + var loggedFirstFrame = false; + + using var socket = new UdpClient(); + using var transport = CreateRudpClient( + options.Get("runtime") ?? $"{Environment.MachineName.ToLowerInvariant()}-mimir-media-receiver", + endpoint, + options); + ConnectOrThrow(transport, endpoint.Uri); + transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(new CultNetDatabaseSubscribeMessage + { + MessageId = Guid.NewGuid().ToString("N"), + SubscriptionId = $"mimir-media-{streamId}", + SchemaIds = [schemaId], + IncludeSnapshot = false + })); + + Console.WriteLine($"Receiving {endpoint.Uri}/{streamId}; writing MPEG-TS UDP to udp://{udp}"); + while (true) + { + transport.PollResends(); + if (transport.ReceiveOnce() is not { } frame) + { + await Task.Delay(2).ConfigureAwait(false); + continue; + } + + if (!string.Equals(frame.ChannelId, "schema", StringComparison.Ordinal)) + { + continue; + } + + if (CultNetSchemaMessageSerialization.Deserialize(frame.Payload) is not CultNetDatabaseChangeRawMessage change || + change.Document is not { } rawDocument || + !string.Equals(rawDocument.SchemaId, schemaId, StringComparison.Ordinal)) + { + continue; + } + + var document = (MimirCultMeshMediaFrameDocument)registry + .GetBySchemaId(schemaId)! + .PayloadDeserializer(rawDocument.Payload); + + if (!string.Equals(document.StreamId, streamId, StringComparison.Ordinal) || + document.Sequence <= lastSequence) + { + continue; + } + + lastSequence = document.Sequence; + socket.Send(document.Payload, document.Payload.Length, udpEndpoint); + frames++; + bytes += document.Payload.Length; + + if (!loggedFirstFrame) + { + loggedFirstFrame = true; + Console.WriteLine($"received first sequence={document.Sequence} bytes={document.PayloadBytes}"); + } + + if (frames % 120 == 0) + { + Console.WriteLine($"received sequence={document.Sequence} frames={frames} bytes={bytes}"); + } + } +} + +static CultMeshRudpEndpoint ResolveRudpEndpoint(CliOptions options) +{ + if (options.Get("endpoint") is { } endpoint) + { + return CultMesh.ParseRudpEndpoint(endpoint); + } + + var host = options.Get("host") ?? "10.77.0.1"; + var port = options.GetInt("port", DefaultPort); + return CultMesh.ParseRudpEndpoint($"rudp://{host}:{port}"); +} + +static CultNetRudpSocketTransportConnection CreateRudpClient( + string runtimeId, + CultMeshRudpEndpoint endpoint, + CliOptions options) +{ + return CultMesh.CreateRudpClient( + runtimeId, + options.GetUInt("connection-id", DefaultConnectionId), + endpoint, + new CultMeshRudpSocketOptions + { + BindHost = options.Get("bind-host") ?? "0.0.0.0", + BindPort = options.GetInt("bind-port", 0), + ResendDelayMs = options.GetInt("resend-ms", 100), + MaxFragmentBytes = options.GetInt("max-fragment-bytes", DefaultMaxFragmentBytes), + MaxPendingReliablePackets = options.GetInt("max-pending-reliable", DefaultMaxPendingReliablePackets) + }); +} + +static void ConnectOrThrow(CultNetRudpSocketTransportConnection transport, string endpoint) +{ + if (!transport.ConnectAndWait(Array.Empty(), TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(5))) + { + throw new TimeoutException($"timed out connecting RUDP media transport to {endpoint}"); + } +} + +static void PumpClient(CultNetRudpSocketTransportConnection transport, TimeSpan duration) +{ + var until = Stopwatch.StartNew(); + while (until.Elapsed < duration) + { + transport.PollResends(); + transport.ReceiveOnce(); + Thread.Sleep(1); + } +} + +static string ResolveMediaSchemaId(CultNetDocumentRegistry registry) +{ + var binding = registry.GetByDocumentType(typeof(MimirCultMeshMediaFrameDocument)) ?? + throw new InvalidOperationException($"CultNet document binding missing for {SchemaName}."); + return binding.SchemaId; +} + +static CultNetDocumentRegistry CreateDocumentRegistry() +{ + var documents = CultDocumentRegistry.Shared; + documents.GetRequired(); + return new CultNetDocumentRegistry(documents) + .Register(CultNetDocumentBinding.ForDocument(documents)); +} + +static IPEndPoint ParseEndpoint(string value) +{ + var split = value.LastIndexOf(':'); + if (split <= 0 || split == value.Length - 1) + { + throw new ArgumentException($"Endpoint must be host:port, not '{value}'."); + } + + var host = value[..split]; + var port = int.Parse(value[(split + 1)..], CultureInfo.InvariantCulture); + var addresses = Dns.GetHostAddresses(host); + return new IPEndPoint(addresses.First(address => address.AddressFamily == AddressFamily.InterNetwork), port); +} + +sealed class RudpMediaRelay : IDisposable +{ + private readonly Socket socket; + private readonly CultCache cache; + private readonly CultNetDocumentRegistry registry; + private readonly string schemaId; + private readonly uint connectionId; + private readonly int maxFragmentBytes; + private readonly int maxPendingReliablePackets; + private readonly Dictionary peers = new(StringComparer.Ordinal); + + public RudpMediaRelay( + IPEndPoint bind, + CultCache cache, + CultNetDocumentRegistry registry, + string schemaId, + uint connectionId, + int maxFragmentBytes, + int maxPendingReliablePackets) + { + this.cache = cache; + this.registry = registry; + this.schemaId = schemaId; + this.connectionId = connectionId; + this.maxFragmentBytes = maxFragmentBytes; + this.maxPendingReliablePackets = maxPendingReliablePackets; + socket = new Socket(bind.AddressFamily, SocketType.Dgram, ProtocolType.Udp); + socket.Bind(bind); + socket.Blocking = false; + } + + public async Task RunAsync() + { + while (true) + { + ReceiveAvailable(); + PumpReliableResends(); + await Task.Delay(2).ConfigureAwait(false); + } + } + + public void Dispose() + { + socket.Dispose(); + } + + private void ReceiveAvailable() + { + while (socket.Poll(0, SelectMode.SelectRead)) + { + var buffer = new byte[65535]; + EndPoint remote = new IPEndPoint(IPAddress.Any, 0); + int received; + try + { + received = socket.ReceiveFrom(buffer, ref remote); + } + catch (SocketException error) when (error.SocketErrorCode == SocketError.WouldBlock) + { + return; + } + + var wire = buffer.AsSpan(0, received).ToArray(); + var packet = CultNetRudpPacketCodec.Decode(wire); + if (packet.ConnectionId != connectionId) + { + continue; + } + + var peer = GetPeer((IPEndPoint)remote); + if (packet.PacketType == CultNetRudpPacketType.Connect) + { + SendPacket(peer, peer.Session.AcceptConnect(packet, NowMs())); + continue; + } + + var result = peer.Session.Receive(packet, NowMs()); + if (result.Reply != null) + { + SendPacket(peer, result.Reply); + } + + if (result.Disconnected) + { + peers.Remove(peer.Key); + continue; + } + + foreach (var frame in result.Delivered.Where(static frame => string.Equals(frame.ChannelId, "schema", StringComparison.Ordinal))) + { + HandleSchemaFrameAsync(peer, frame.Payload).GetAwaiter().GetResult(); + } + + if (packet.PacketType == CultNetRudpPacketType.Accept || result.Delivered.Count > 0) + { + SendPacket(peer, peer.Session.CreateAck()); + } + } + } + + private async Task HandleSchemaFrameAsync(RelayPeer peer, byte[] payload) + { + var message = CultNetSchemaMessageSerialization.Deserialize(payload); + switch (message) + { + case CultNetDocumentPutRawMessage put: + await ApplyPutAsync(put).ConfigureAwait(false); + break; + case CultNetDatabaseSubscribeMessage subscribe: + peer.Subscriptions[subscribe.SubscriptionId] = subscribe; + Console.WriteLine($"relay RUDP subscribe peer={peer.Key} subscription={subscribe.SubscriptionId} schemas={string.Join(",", subscribe.SchemaIds ?? [])}"); + break; + case CultNetDatabaseUnsubscribeMessage unsubscribe: + peer.Subscriptions.Remove(unsubscribe.SubscriptionId); + break; + } + } + + private async Task ApplyPutAsync(CultNetDocumentPutRawMessage put) + { + if (put.Document == null || + !string.Equals(put.Document.SchemaId, schemaId, StringComparison.Ordinal)) + { + return; + } + + await registry.ApplyRawDocumentPutMessageAsync(cache, put).ConfigureAwait(false); + await cache.FlushAsync(soft: true).ConfigureAwait(false); + var change = new CultNetDatabaseChangeRawMessage + { + MessageId = Guid.NewGuid().ToString("N"), + ChangeKind = "updated", + Document = put.Document + }; + + var frame = CultNetSchemaMessageSerialization.Serialize(change); + foreach (var peer in peers.Values) + { + foreach (var subscription in peer.Subscriptions.Values) + { + if (!Matches(subscription, put.Document.SchemaId, put.Document.RecordKey)) + { + continue; + } + + change.SubscriptionId = subscription.SubscriptionId; + SendSchema(peer, CultNetSchemaMessageSerialization.Serialize(change)); + if (!peer.HasForwardedFrame) + { + peer.HasForwardedFrame = true; + Console.WriteLine($"relay RUDP forward peer={peer.Key} subscription={subscription.SubscriptionId} schema={put.Document.SchemaId}"); + } + ReceiveAvailable(); + } + } + + var document = (MimirCultMeshMediaFrameDocument)registry + .GetBySchemaId(schemaId)! + .PayloadDeserializer(put.Document.Payload); + if (document.Sequence % 120 == 0) + { + Console.WriteLine($"relay RUDP put stream={document.StreamId} sequence={document.Sequence} bytes={document.PayloadBytes}"); + } + } + + private static bool Matches(CultNetDatabaseSubscribeMessage request, string schemaId, string recordKey) + { + var schemaMatches = request.SchemaIds == null || + request.SchemaIds.Length == 0 || + request.SchemaIds.Contains(schemaId, StringComparer.Ordinal); + var keyMatches = request.RecordKeys == null || + request.RecordKeys.Length == 0 || + request.RecordKeys.Contains(recordKey, StringComparer.Ordinal); + return schemaMatches && keyMatches; + } + + private RelayPeer GetPeer(IPEndPoint endpoint) + { + var key = endpoint.ToString(); + if (peers.TryGetValue(key, out var existing)) + { + return existing; + } + + var peer = new RelayPeer( + key, + endpoint, + new CultNetRudpSession(new CultNetRudpSessionOptions + { + ConnectionId = connectionId, + InitialSequence = (uint)(peers.Count + 1), + ResendDelayMs = 100, + MaxPendingReliablePackets = maxPendingReliablePackets + })); + peers[key] = peer; + return peer; + } + + private void SendSchema(RelayPeer peer, byte[] payload) + { + foreach (var packet in peer.Session.SendMany( + "schema", + payload, + new CultNetRudpSendOptions { Reliable = true, Ordered = true, NowMs = NowMs() }, + maxFragmentBytes)) + { + SendPacket(peer, packet); + } + } + + private void SendPacket(RelayPeer peer, CultNetRudpPacket packet) + { + var wire = CultNetRudpPacketCodec.Encode(packet); + socket.SendTo(wire, peer.Endpoint); + } + + private void PumpReliableResends() + { + foreach (var peer in peers.Values) + { + foreach (var packet in peer.Session.DueResends(NowMs())) + { + SendPacket(peer, packet); + } + } + } + + private static long NowMs() => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); +} + +sealed record RelayPeer( + string Key, + IPEndPoint Endpoint, + CultNetRudpSession Session) +{ + public Dictionary Subscriptions { get; } = new(StringComparer.Ordinal); + public bool HasForwardedFrame { get; set; } +} + +sealed class CliOptions +{ + private readonly Dictionary values = new(StringComparer.OrdinalIgnoreCase); + + public static CliOptions Parse(IEnumerable args) + { + var parsed = new CliOptions(); + var pendingKey = string.Empty; + foreach (var arg in args) + { + if (arg.StartsWith("--", StringComparison.Ordinal)) + { + pendingKey = arg[2..]; + parsed.values[pendingKey] = "true"; + continue; + } + + if (!string.IsNullOrWhiteSpace(pendingKey)) + { + parsed.values[pendingKey] = arg; + pendingKey = string.Empty; + } + } + + return parsed; + } + + public string? Get(string key) => values.TryGetValue(key, out var value) ? value : null; + + public int GetInt(string key, int fallback) => + int.TryParse(Get(key), NumberStyles.Integer, CultureInfo.InvariantCulture, out var value) ? value : fallback; + + public uint GetUInt(string key, uint fallback) => + uint.TryParse(Get(key), NumberStyles.Integer, CultureInfo.InvariantCulture, out var value) ? value : fallback; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirCultMeshMediaFrameDocument.cs b/src/Mimir.Runtime/Synchronization/MimirCultMeshMediaFrameDocument.cs new file mode 100644 index 0000000..2f77aed --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirCultMeshMediaFrameDocument.cs @@ -0,0 +1,24 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[CultDocument("mimir.cultmesh_media_frame", "mimir.cultmesh_media_frame.v1")] +[MessagePackObject] +public sealed record MimirCultMeshMediaFrameDocument( + [property: Key(0)] + [property: CultName] + string FrameId, + [property: Key(1)] string StreamId, + [property: Key(2)] long Sequence, + [property: Key(3)] string ProducedAtUtc, + [property: Key(4)] long TimestampNanoseconds, + [property: Key(5)] string PayloadKind, + [property: Key(6)] string Container, + [property: Key(7)] string VideoCodec, + [property: Key(8)] string AudioCodec, + [property: Key(9)] int PayloadBytes, + [property: Key(10)] byte[] Payload, + [property: Key(11)] string ProducerNode, + [property: Key(12)] string ClockDomainId, + [property: Key(13)] string[] Tags); diff --git a/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs index 13b2636..e2e48ed 100644 --- a/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs @@ -6,12 +6,14 @@ public enum MimirNetworkPayloadKind RawAudioDebugWindow, CompressedAudioProgramFeed, CompressedVideoProgramFeed, - RawVideoDiagnosticWindow + RawVideoDiagnosticWindow, + MuxedAudioVideoProgramFeed } public enum MimirNetworkTransportKind { CultMeshTypedState, + CultMeshReliableUdpMedia, LanUdpDatagrams, SrtDiagnosticMedia, WebRtcExperimental, @@ -68,6 +70,18 @@ public static class MimirNetworkTransportConfigurations RejectionConditions: ["used-as-clock-authority", "used-for-local-hot-path"], "Useful bridge. Not the Perfect Machine's synchronization core."); + public static MimirNetworkTransportConfiguration CultMeshProgramBridge { get; } = new( + "cultmesh-program-bridge", + "Reliable-UDP CultMesh media bridge for muxed program frames that OBS consumes through a local Starfire endpoint.", + MimirNetworkTransportKind.CultMeshReliableUdpMedia, + MimirNetworkPayloadKind.MuxedAudioVideoProgramFeed, + MayAffectClock: false, + CarriesRawMedia: true, + TargetLatencyMilliseconds: 500.0, + RequiredDocuments: ["mimir.cultmesh_media_frame"], + RejectionConditions: ["used-as-clock-authority", "unbounded-media-cache", "non-local-obs-egress"], + "Raven publishes rolling media-frame documents over CultNet reliable UDP; Starfire lowers them to a local OBS MPEG-TS endpoint."); + public static MimirNetworkTransportConfiguration WebRtcExperiment { get; } = CultMeshTimingState with { Id = "webrtc-experimental-media", @@ -84,6 +98,7 @@ public static class MimirNetworkTransportConfigurations [ CultMeshTimingState, RawAudioDebugWindow, + CultMeshProgramBridge, SrtProgramBridge, WebRtcExperiment ]; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 6ea427a..065e153 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -142,3 +142,4 @@ {"ts":"2026-06-16T00:00:00+01:00","type":"correction","status":"ok","note":"Reoriented Mimir's daemon-swarm map after Odin/CultLib RUDP work: Raven media body transport remains src/Mimir.CultMeshMedia over CultNet/CultMesh reliable UDP with OBS-local UDP as egress only, while Eve dashboard/browser reference health now publishes idunn.daemon_health over cultnet.transport.rudp.v0 as self-witness state. Odin owns accepted Verse/service discovery and Idunn owns keepalive decisions; Mimir health publishers do not own lifecycle."} {"ts":"2026-06-16T00:05:00+01:00","type":"correction","status":"ok","note":"Corrected the transport model: CultLib has added authorized RUDP helpers across runtimes and Odin/Idunn now treats daemon truth as typed CultCache/CultMesh state carried over cultnet.transport.rudp.v0. Mimir should not assume TCP is needed for any daemon-truth path once the runtime has RUDP support; HTTP/WebSocket/TCP are client lowerings, compatibility probes, or migration debt. Health is separate from media by authority, not because it needs a separate TCP transport."} {"ts":"2026-06-16T00:12:00+01:00","type":"correction","status":"ok","note":"Source audit found Mimir.CultMeshMedia still using older C# CultMesh.StartNodeAsync/CultMesh.ConnectClient entrypoints while CultLib now exposes C# CultMesh.CreateRudpServer/CreateRudpClient/CreateRudpClientForAuthorizedPeer. Updated maps to treat the media bridge's current transport path as migration debt: typed program/media documents should move to explicit CultNet RUDP helpers, with OBS-local UDP preserved only as compatibility egress."} +{"ts": "2026-06-16T18:05:56+01:00", "type": "verification", "status": "ok", "note": "Mimir.CultMeshMedia migrated from old CultMesh node/client transport to explicit CultNet RUDP relay/send/recv path; local smoke artifacts/rudp-media-smoke-20260616-180404 proved 32 MPEG-TS chunks reached Starfire-local UDP sink over RUDP (packets=32 bytes=42112)."} diff --git a/state/map.yaml b/state/map.yaml index 89ccbb1..e3baf7e 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -383,12 +383,10 @@ accepted_design: `mimir.cultmesh_media_frame` documents over CultNet reliable UDP, Yggdrasil relays them on `10.77.0.1:3075`, and Starfire lowers the subscribed stream to `udp://127.0.0.1:5200` for the OBS source - `Raven Monitor + Realtek`. Source audit warning: the current C# bridge - still calls older `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` - entrypoints; now that CultLib exposes C# RUDP facade helpers, the next - transport cut should move this bridge onto explicit RUDP server/client or - authorized-peer helpers and demote any TCP-like node/client path to - compatibility evidence. + `Raven Monitor + Realtek`. The bridge now uses explicit CultLib RUDP + socket/session helpers for relay, sender, and receiver. The old + `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` node/client path is no + longer a transport authority for this media lane. - `src/Mimir.EveDashboard` publishes `mimir.eve_dashboard_state` through CultNet/CultMesh and can push `idunn.daemon_health` records over `cultnet.transport.rudp.v0` for itself and a paired service daemon. Its @@ -433,9 +431,11 @@ active_subgoals: keepalive, and Mimir surfaces publish typed RUDP self-health rather than private watchdog loops or TCP/HTTP status truth. - Cut remaining older CultMesh node/client transport assumptions in - `src/Mimir.CultMeshMedia`, `src/Mimir.EveDashboard`, and witness tooling: - daemon truth and typed program/media document paths should use explicit - CultNet RUDP helpers; HTTP/WebSocket remain render/client lowerings only. + `src/Mimir.EveDashboard` and witness tooling: daemon truth, provider + catalogs, command boundaries, and typed program document paths should use + explicit CultNet RUDP helpers; HTTP/WebSocket remain render/client + lowerings only. `src/Mimir.CultMeshMedia` has completed this cut for the + Raven/Yggdrasil/Starfire media-frame lane. current_status: summary: > From b1a6043d7eb162c3f5d9035caf1b442dd3c8bb99 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 16 Jun 2026 20:31:31 +0100 Subject: [PATCH 048/135] Publish browser reference boundary witness --- .../Mimir.EveBrowserReference.csproj | 1 + src/Mimir.EveBrowserReference/Program.cs | 387 +++++++++++++++++- 2 files changed, 375 insertions(+), 13 deletions(-) diff --git a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj index dd71bc2..ea6d407 100644 --- a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj +++ b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj @@ -10,6 +10,7 @@ + diff --git a/src/Mimir.EveBrowserReference/Program.cs b/src/Mimir.EveBrowserReference/Program.cs index 3ae661e..8e86bd8 100644 --- a/src/Mimir.EveBrowserReference/Program.cs +++ b/src/Mimir.EveBrowserReference/Program.cs @@ -5,20 +5,28 @@ using System.Security.Cryptography; using System.Text; using System.Text.Json; using System.Text.Json.Serialization; +using GameCult.Caching; +using GameCult.Caching.MessagePack; using GameCult.Networking; using MessagePack; var port = ParseInt(args, "--port", 8891); var bind = ParseString(args, "--bind", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_BIND") ?? "0.0.0.0"); var directory = ParseString(args, "--directory", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_DIRECTORY") ?? Directory.GetCurrentDirectory()); +var cultCacheWitnessPath = ParseString( + args, + "--cultcache-witness", + Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_CULTCACHE_WITNESS") ?? DefaultPaths.CultCacheWitnessPath); var idunnRudpHealth = ParseString(args, "--idunn-rudp-health", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_RUDP_HEALTH") ?? ""); var idunnDaemon = ParseString(args, "--idunn-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_DAEMON") ?? "nightwing-eve-browser-reference"); var idunnHealthContract = ParseString(args, "--idunn-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_HEALTH_CONTRACT") ?? "nightwing.cultnet-rudp-browser-reference-health"); +using var verseRuntime = EveBrowserReferenceVerseRuntime.Create(cultCacheWitnessPath); using var server = new EveBrowserReferenceServer( IPAddress.Parse(bind), port, Path.GetFullPath(directory), + verseRuntime, new IdunnRudpHealthOptions(idunnRudpHealth, idunnDaemon, idunnHealthContract)); await server.RunAsync().ConfigureAwait(false); @@ -50,7 +58,20 @@ static string ParseString(IReadOnlyList args, string name, string fallba return fallback; } -internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, string documentRoot, IdunnRudpHealthOptions idunnRudpHealth) : IDisposable +internal static class DefaultPaths +{ + public static string CultCacheWitnessPath => + OperatingSystem.IsWindows() + ? @"E:\Projects\Mimir\state\eve-browser-reference.service.cc" + : "/var/lib/gamecult/eve-browser-reference/cultcache/eve-browser-reference.service.cc"; +} + +internal sealed class EveBrowserReferenceServer( + IPAddress bindAddress, + int port, + string documentRoot, + EveBrowserReferenceVerseRuntime verseRuntime, + IdunnRudpHealthOptions idunnRudpHealth) : IDisposable { private static readonly JsonSerializerOptions JsonOptions = new() { @@ -66,6 +87,7 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, private string idunnRudpPublishObservedAt = ""; private bool idunnRudpPublishInFlight; private DateTimeOffset lastHealthPublishAttemptAt = DateTimeOffset.MinValue; + private DateTimeOffset lastWitnessPublishAttemptAt = DateTimeOffset.MinValue; public async Task RunAsync() { @@ -79,6 +101,7 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, Console.WriteLine(string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) ? "Mimir Eve browser reference Idunn RUDP health publisher disabled." : $"Mimir Eve browser reference Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} as {idunnRudpHealth.DaemonId}."); + Console.WriteLine($"Mimir Eve browser reference witness store publishing to {verseRuntime.WitnessPath}"); _ = Task.Run(PublishHealthLoopAsync); while (!stopping.IsCancellationRequested) { @@ -92,7 +115,9 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, using var timer = new PeriodicTimer(TimeSpan.FromSeconds(1)); while (!stopping.IsCancellationRequested && await timer.WaitForNextTickAsync(stopping.Token).ConfigureAwait(false)) { - QueueIdunnRudpHealthIfDue(DateTimeOffset.UtcNow); + var observedAt = DateTimeOffset.UtcNow; + PublishWitnessIfDue(observedAt); + QueueIdunnRudpHealthIfDue(observedAt); } } @@ -108,6 +133,7 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, documentRoot, port, transport = "static-http-lowering", + cultCacheWitness = verseRuntime.GetWitnessStatus(), idunnRudpHealth = GetIdunnRudpHealthStatus(), }, JsonOptions); await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false); @@ -140,6 +166,22 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, } } + private void PublishWitnessIfDue(DateTimeOffset observedAt) + { + if (lastWitnessPublishAttemptAt != DateTimeOffset.MinValue && + observedAt - lastWitnessPublishAttemptAt < TimeSpan.FromSeconds(5)) + { + return; + } + + lastWitnessPublishAttemptAt = observedAt; + verseRuntime.Publish( + CreateDaemonHealthRecord(observedAt), + documentRoot, + port, + observedAt); + } + private void QueueIdunnRudpHealthIfDue(DateTimeOffset observedAt) { if (string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint)) @@ -154,17 +196,8 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, return; } - var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); - var health = new IdunnDaemonHealthRecord - { - DaemonId = idunnRudpHealth.DaemonId, - State = "healthy", - Detail = $"Nightwing Eve browser reference serving static lowering; root={documentRoot}; port={port}", - ObservedAt = observedAtText, - HealthContract = idunnRudpHealth.HealthContract, - PublicationSource = "daemon-published", - Transport = "cultnet.transport.rudp.v0", - }; + var health = CreateDaemonHealthRecord(observedAt); + var observedAtText = health.ObservedAt; lock (idunnRudpPublishLock) { @@ -211,6 +244,21 @@ internal sealed class EveBrowserReferenceServer(IPAddress bindAddress, int port, }); } + private IdunnDaemonHealthRecord CreateDaemonHealthRecord(DateTimeOffset observedAt) + { + var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); + return new IdunnDaemonHealthRecord + { + DaemonId = idunnRudpHealth.DaemonId, + State = "healthy", + Detail = $"Nightwing Eve browser reference serving static lowering; root={documentRoot}; port={port}; witness={verseRuntime.WitnessPath}", + ObservedAt = observedAtText, + HealthContract = idunnRudpHealth.HealthContract, + PublicationSource = "daemon-published", + Transport = "cultnet.transport.rudp.v0", + }; + } + private string? ResolveStaticFilePath(string path) { var requestPath = path.Split('?', 2)[0]; @@ -280,6 +328,13 @@ internal sealed record HttpRequest(string Path); internal sealed record IdunnRudpHealthOptions(string Endpoint, string DaemonId, string HealthContract); +internal sealed record CultCacheWitnessStatus( + string Path, + string Status, + string Error, + string ObservedAtUtc, + IReadOnlyList PublishedSchemas); + internal sealed record IdunnRudpHealthStatus( string Endpoint, string Daemon, @@ -289,6 +344,7 @@ internal sealed record IdunnRudpHealthStatus( bool InFlight, string ObservedAtUtc); +[CultDocument("idunn.daemon_health", "idunn.daemon_health.v1")] [MessagePackObject(AllowPrivate = true)] internal sealed class IdunnDaemonHealthRecord { @@ -301,6 +357,311 @@ internal sealed class IdunnDaemonHealthRecord [Key(6)] public string Transport { get; set; } = "cultnet.transport.rudp.v0"; } +internal sealed class EveBrowserReferenceVerseRuntime : IDisposable +{ + private static readonly CultRecordKey ManifestKey = new("nightwing-eve-browser-reference"); + private static readonly CultRecordKey StaticSurfaceKey = new("nightwing-eve-browser-reference"); + private static readonly CultRecordKey CommandBoundaryKey = new("nightwing-eve-browser-reference"); + private static readonly CultRecordKey TransportProfileKey = new("nightwing-eve-browser-reference"); + private static readonly string[] PublishedSchemas = + [ + "mimir.eve_browser_reference_manifest.v1", + "mimir.eve_browser_reference_static_surface.v1", + "mimir.eve_browser_reference_command_boundary.v1", + "mimir.eve_browser_reference_transport_profile.v1", + "idunn.daemon_health.v1", + ]; + + private readonly CultDocumentRegistry registry = new(); + private readonly string witnessPath; + private readonly object publishLock = new(); + private string witnessStatus = "starting"; + private string witnessError = string.Empty; + private string witnessObservedAtUtc = string.Empty; + + private EveBrowserReferenceVerseRuntime(string witnessPath) + { + this.witnessPath = Path.GetFullPath(witnessPath); + } + + public string WitnessPath => witnessPath; + + public static EveBrowserReferenceVerseRuntime Create(string witnessPath) + { + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(witnessPath)) ?? "."); + return new EveBrowserReferenceVerseRuntime(witnessPath); + } + + public CultCacheWitnessStatus GetWitnessStatus() + { + lock (publishLock) + { + return new CultCacheWitnessStatus( + witnessPath, + witnessStatus, + witnessError, + witnessObservedAtUtc, + PublishedSchemas); + } + } + + public void Publish( + IdunnDaemonHealthRecord healthRecord, + string documentRoot, + int port, + DateTimeOffset observedAt) + { + var observedAtUtc = observedAt.ToString("O", CultureInfo.InvariantCulture); + + try + { + lock (publishLock) + { + var records = new List(); + var schemas = new Dictionary(StringComparer.Ordinal); + var assets = EnumerateRelativeFiles(documentRoot); + var fixtures = assets + .Where(path => path.StartsWith("fixtures/", StringComparison.Ordinal)) + .ToArray(); + + AddRecord(records, schemas, ToManifestDocument(documentRoot, port, assets), ManifestKey, observedAtUtc); + AddRecord(records, schemas, ToStaticSurfaceDocument(documentRoot, assets, fixtures, observedAtUtc), StaticSurfaceKey, observedAtUtc); + AddRecord(records, schemas, ToCommandBoundaryDocument(observedAtUtc), CommandBoundaryKey, observedAtUtc); + AddRecord(records, schemas, ToTransportProfileDocument(observedAtUtc), TransportProfileKey, observedAtUtc); + AddRecord(records, schemas, healthRecord, new CultRecordKey(healthRecord.DaemonId), observedAtUtc); + + WriteSnapshot(new CultPersistedStoreSnapshot + { + FormatVersion = "cultcache.store.v1", + SchemaCatalog = schemas.Values.OrderBy(entry => entry.SchemaName, StringComparer.Ordinal).ToArray(), + Records = records.OrderBy(entry => entry.Key, StringComparer.Ordinal).ToArray(), + }); + witnessStatus = "published"; + witnessError = string.Empty; + witnessObservedAtUtc = observedAtUtc; + } + } + catch (Exception error) + { + lock (publishLock) + { + witnessStatus = "publish-error"; + witnessError = error.Message; + witnessObservedAtUtc = observedAtUtc; + } + } + } + + public void Dispose() + { + } + + private void AddRecord( + List records, + Dictionary schemas, + T document, + CultRecordKey key, + string observedAtUtc) where T : class + { + var descriptor = registry.GetRequired(); + var payload = descriptor.GeneratedPayloadSerializer != null + ? descriptor.GeneratedPayloadSerializer(document) + : MessagePackSerializer.Serialize(typeof(T), document, CultNetSchemaMessageSerialization.Options); + schemas[descriptor.SchemaId] = descriptor.ToCatalogEntry(); + records.Add(new CultPersistedRecord + { + Key = key.Value, + SchemaId = descriptor.SchemaId, + StoredAt = observedAtUtc, + Payload = payload, + }); + } + + private void WriteSnapshot(CultPersistedStoreSnapshot snapshot) + { + var payload = CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot); + var directory = Path.GetDirectoryName(witnessPath) ?? "."; + Directory.CreateDirectory(directory); + var tempPath = Path.Combine(directory, $".{Path.GetFileName(witnessPath)}.{Guid.NewGuid():N}.tmp"); + + try + { + using (var stream = new FileStream( + tempPath, + FileMode.CreateNew, + FileAccess.Write, + FileShare.None, + bufferSize: 4096, + FileOptions.WriteThrough)) + { + stream.Write(payload, 0, payload.Length); + stream.Flush(true); + } + + if (File.Exists(witnessPath)) + { + File.Replace(tempPath, witnessPath, null); + } + else + { + File.Move(tempPath, witnessPath); + } + } + finally + { + if (File.Exists(tempPath)) + { + File.Delete(tempPath); + } + } + } + + private static EveBrowserReferenceManifestRecord ToManifestDocument( + string documentRoot, + int port, + IReadOnlyList assets) => + new() + { + DaemonId = "nightwing-eve-browser-reference", + SiteTitle = "Nightwing Eve Browser Reference", + DocumentRoot = documentRoot, + EntryPoint = "index.html", + Port = port, + Assets = assets.ToArray(), + SurfaceTransport = "static-http-lowering", + }; + + private static EveBrowserReferenceStaticSurfaceRecord ToStaticSurfaceDocument( + string documentRoot, + IReadOnlyList assets, + IReadOnlyList fixtures, + string observedAtUtc) => + new() + { + DaemonId = "nightwing-eve-browser-reference", + UpdatedAtUtc = observedAtUtc, + DocumentRoot = documentRoot, + EntryPoint = "index.html", + Assets = assets.ToArray(), + FixtureDocuments = fixtures.ToArray(), + CurrentCutLine = "Static browser lowering assets are published into the daemon-owned witness store so Nightwing and Odin can reason about the lowering body without treating the HTTP service probe as lifecycle truth.", + }; + + private static EveBrowserReferenceCommandBoundaryRecord ToCommandBoundaryDocument(string observedAtUtc) => + new() + { + DaemonId = "nightwing-eve-browser-reference", + UpdatedAtUtc = observedAtUtc, + Mode = "static-read-only-lowering", + WritesAccepted = false, + OperatorInputAuthority = "served browser assets only; no write path is accepted by this daemon", + LifecycleAuthority = "idunn.local-command.restart + compatibility.systemd.nightwing-eve-browser-reference.service", + AcceptedCommands = [], + RejectedCommands = + [ + "surface-mutation", + "provider-catalog-ownership", + "daemon-health-override", + "service-restart", + ], + }; + + private EveBrowserReferenceTransportProfileRecord ToTransportProfileDocument(string observedAtUtc) => + new() + { + DaemonId = "nightwing-eve-browser-reference", + UpdatedAtUtc = observedAtUtc, + CurrentState = "partial-rudp-health-and-provider-store-live", + InputTransport = "compatibility.http-static-lowering", + OutputTransport = "daemon-owned-cultcache-boundary-store + daemon-published-rudp-health", + HealthContract = "nightwing.cultnet-rudp-browser-reference-health", + IdunnRudpHealth = "idunn.health-published-separately", + WitnessPath = witnessPath, + CurrentCutLine = "The Nightwing Eve browser reference runtime now owns its manifest, static surface witness, command boundary, and transport profile in a daemon-owned CultCache store; systemd and HTTP remain compatibility witnesses while projections learn to consume the typed store directly.", + }; + + private static string[] EnumerateRelativeFiles(string root) + { + if (!Directory.Exists(root)) + { + return []; + } + + return Directory + .EnumerateFiles(root, "*", SearchOption.AllDirectories) + .Select(path => Path.GetRelativePath(root, path).Replace('\\', '/')) + .OrderBy(path => path, StringComparer.Ordinal) + .ToArray(); + } +} + +[CultDocument("mimir.eve_browser_reference_manifest", "mimir.eve_browser_reference_manifest.v1")] +[MessagePackObject(AllowPrivate = true)] +internal sealed class EveBrowserReferenceManifestRecord +{ + [Key(0)] + [CultName] + public string DaemonId { get; set; } = string.Empty; + + [Key(1)] public string SiteTitle { get; set; } = string.Empty; + [Key(2)] public string DocumentRoot { get; set; } = string.Empty; + [Key(3)] public string EntryPoint { get; set; } = string.Empty; + [Key(4)] public int Port { get; set; } + [Key(5)] public string[] Assets { get; set; } = []; + [Key(6)] public string SurfaceTransport { get; set; } = string.Empty; +} + +[CultDocument("mimir.eve_browser_reference_static_surface", "mimir.eve_browser_reference_static_surface.v1")] +[MessagePackObject(AllowPrivate = true)] +internal sealed class EveBrowserReferenceStaticSurfaceRecord +{ + [Key(0)] + [CultName] + public string DaemonId { get; set; } = string.Empty; + + [Key(1)] public string UpdatedAtUtc { get; set; } = string.Empty; + [Key(2)] public string DocumentRoot { get; set; } = string.Empty; + [Key(3)] public string EntryPoint { get; set; } = string.Empty; + [Key(4)] public string[] Assets { get; set; } = []; + [Key(5)] public string[] FixtureDocuments { get; set; } = []; + [Key(6)] public string CurrentCutLine { get; set; } = string.Empty; +} + +[CultDocument("mimir.eve_browser_reference_command_boundary", "mimir.eve_browser_reference_command_boundary.v1")] +[MessagePackObject(AllowPrivate = true)] +internal sealed class EveBrowserReferenceCommandBoundaryRecord +{ + [Key(0)] + [CultName] + public string DaemonId { get; set; } = string.Empty; + + [Key(1)] public string UpdatedAtUtc { get; set; } = string.Empty; + [Key(2)] public string Mode { get; set; } = string.Empty; + [Key(3)] public bool WritesAccepted { get; set; } + [Key(4)] public string OperatorInputAuthority { get; set; } = string.Empty; + [Key(5)] public string LifecycleAuthority { get; set; } = string.Empty; + [Key(6)] public string[] AcceptedCommands { get; set; } = []; + [Key(7)] public string[] RejectedCommands { get; set; } = []; +} + +[CultDocument("mimir.eve_browser_reference_transport_profile", "mimir.eve_browser_reference_transport_profile.v1")] +[MessagePackObject(AllowPrivate = true)] +internal sealed class EveBrowserReferenceTransportProfileRecord +{ + [Key(0)] + [CultName] + public string DaemonId { get; set; } = string.Empty; + + [Key(1)] public string UpdatedAtUtc { get; set; } = string.Empty; + [Key(2)] public string CurrentState { get; set; } = string.Empty; + [Key(3)] public string InputTransport { get; set; } = string.Empty; + [Key(4)] public string OutputTransport { get; set; } = string.Empty; + [Key(5)] public string HealthContract { get; set; } = string.Empty; + [Key(6)] public string IdunnRudpHealth { get; set; } = string.Empty; + [Key(7)] public string WitnessPath { get; set; } = string.Empty; + [Key(8)] public string CurrentCutLine { get; set; } = string.Empty; +} + internal static class IdunnRudpHealthPublisher { private const uint ConnectionId = 0x1d0d0001; From 3103a7aa273f4b9c1d3fd15c49367bf7f2754917 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 03:49:55 +0100 Subject: [PATCH 049/135] Ingest Muninn Move identity records --- src/Mimir.BufferSmoke/Program.cs | 37 ++++++++++++++ .../MimirMuninnMoveEvidence.cs | 49 +++++++++++++++++++ state/evidence.jsonl | 1 + 3 files changed, 87 insertions(+) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 7b174e4..eab405c 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -109,6 +109,11 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-evidence-smoke", StringCom ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--muninn-move-identity-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMuninnMoveIdentitySmoke(); +} + if (args.Any(arg => string.Equals(arg, "--muninn-move-cultmesh-stream-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMuninnMoveCultMeshStreamSmoke( @@ -1637,6 +1642,38 @@ static int RunMuninnMoveEvidenceSmoke(string nativeReservoirPath) : 1; } +static int RunMuninnMoveIdentitySmoke() +{ + var identity = new MuninnMoveIdentityDocument( + IdentityId: "starfire:move-000704a800d0:move-identity", + HostId: "starfire", + MoveId: "move-000704a800d0", + SourcePath: @"windows-psmove:\\?\hid#vid_054c&pid_03d5&col01#a&976df89&0&0000#{4d1e55b2-f16f-11cf-88cb-001111000030}", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "usb-visible", + Detail: "Muninn discovered this PS Move on a local USB/HID input path.", + ObservedAt: "2026-06-17T02:24:16.0000000Z"); + + var encoded = MessagePackSerializer.Serialize(identity); + var decoded = MessagePackSerializer.Deserialize(encoded); + var snapshot = MimirMuninnMoveEvidenceAdapter.BuildIdentitySnapshots([decoded]).Single(); + var expectedHash = MimirMuninnMoveEvidenceAdapter.Fnva64("starfire:move-000704a800d0"); + + Console.WriteLine( + $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X}"); + + return snapshot.IdentityId == identity.IdentityId && + snapshot.HostId == "starfire" && + snapshot.MoveId == "move-000704a800d0" && + snapshot.SourcePath.Contains("vid_054c&pid_03d5", StringComparison.OrdinalIgnoreCase) && + snapshot.BluetoothHostAddress == "5C:93:A2:9C:A8:A8" && + snapshot.State == "usb-visible" && + snapshot.ObservedAtNs > 0 && + snapshot.ControllerIdHash == expectedHash + ? 0 + : 1; +} + static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) { if (!File.Exists(nativeReservoirPath)) diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index 45f527b..e953a51 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -40,6 +40,29 @@ public sealed record MuninnMoveControllerStateDocument( [property: Key(11)] string ObservedAt, [property: Key(12)] string SourcePath = ""); +[CultDocument("muninn.move_identity", "muninn.move_identity.v1")] +[MessagePackObject] +public sealed record MuninnMoveIdentityDocument( + [property: Key(0)] string IdentityId, + [property: Key(1)] string HostId, + [property: Key(2)] string MoveId, + [property: Key(3)] string SourcePath, + [property: Key(4)] string BluetoothHostAddress, + [property: Key(5)] string State, + [property: Key(6)] string Detail, + [property: Key(7)] string ObservedAt); + +public sealed record MimirMuninnMoveIdentitySnapshot( + string IdentityId, + string HostId, + string MoveId, + string SourcePath, + string BluetoothHostAddress, + string State, + string Detail, + ulong ObservedAtNs, + ulong ControllerIdHash); + [MessagePackObject] public sealed record MimirMuninnMoveEvidenceStreamFrame( [property: Key(0)] string FrameId, @@ -126,6 +149,32 @@ public static class MimirMuninnMoveEvidenceAdapter .ToArray(); } + public static IReadOnlyList BuildIdentitySnapshots( + IEnumerable identities) + { + ArgumentNullException.ThrowIfNull(identities); + return identities + .Select(ToIdentitySnapshot) + .OrderBy(identity => identity.MoveId, StringComparer.Ordinal) + .ThenByDescending(identity => identity.ObservedAtNs) + .ToArray(); + } + + public static MimirMuninnMoveIdentitySnapshot ToIdentitySnapshot(MuninnMoveIdentityDocument identity) + { + ArgumentNullException.ThrowIfNull(identity); + return new MimirMuninnMoveIdentitySnapshot( + IdentityId: identity.IdentityId, + HostId: identity.HostId, + MoveId: identity.MoveId, + SourcePath: identity.SourcePath, + BluetoothHostAddress: identity.BluetoothHostAddress, + State: identity.State, + Detail: identity.Detail, + ObservedAtNs: ObservedAtToUnixNs(identity.ObservedAt), + ControllerIdHash: Fnva64($"{identity.HostId}:{identity.MoveId}")); + } + public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveMarkerCandidateDocument marker) { ArgumentNullException.ThrowIfNull(marker); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 065e153..18b1786 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -143,3 +143,4 @@ {"ts":"2026-06-16T00:05:00+01:00","type":"correction","status":"ok","note":"Corrected the transport model: CultLib has added authorized RUDP helpers across runtimes and Odin/Idunn now treats daemon truth as typed CultCache/CultMesh state carried over cultnet.transport.rudp.v0. Mimir should not assume TCP is needed for any daemon-truth path once the runtime has RUDP support; HTTP/WebSocket/TCP are client lowerings, compatibility probes, or migration debt. Health is separate from media by authority, not because it needs a separate TCP transport."} {"ts":"2026-06-16T00:12:00+01:00","type":"correction","status":"ok","note":"Source audit found Mimir.CultMeshMedia still using older C# CultMesh.StartNodeAsync/CultMesh.ConnectClient entrypoints while CultLib now exposes C# CultMesh.CreateRudpServer/CreateRudpClient/CreateRudpClientForAuthorizedPeer. Updated maps to treat the media bridge's current transport path as migration debt: typed program/media documents should move to explicit CultNet RUDP helpers, with OBS-local UDP preserved only as compatibility egress."} {"ts": "2026-06-16T18:05:56+01:00", "type": "verification", "status": "ok", "note": "Mimir.CultMeshMedia migrated from old CultMesh node/client transport to explicit CultNet RUDP relay/send/recv path; local smoke artifacts/rudp-media-smoke-20260616-180404 proved 32 MPEG-TS chunks reached Starfire-local UDP sink over RUDP (packets=32 bytes=42112)."} +{"ts": "2026-06-17T03:45:44+01:00", "type": "decision", "status": "ok", "note": "Muninn Move identity is now a typed provenance/provisioning lane: Mimir consumes muninn.move_identity.v1 as identity snapshots while controller-state/marker evidence remains the only native reservoir motion evidence. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and --muninn-move-evidence-smoke."} From c3babfaa577b38f90e9176d7875776419651e322 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 06:48:44 +0100 Subject: [PATCH 050/135] Project Muninn Move identity roster --- src/Mimir.BufferSmoke/Program.cs | 33 ++++++- .../MimirMuninnMoveEvidence.cs | 89 +++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 6 ++ 4 files changed, 127 insertions(+), 2 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index eab405c..6c66c9d 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1653,14 +1653,35 @@ static int RunMuninnMoveIdentitySmoke() State: "usb-visible", Detail: "Muninn discovered this PS Move on a local USB/HID input path.", ObservedAt: "2026-06-17T02:24:16.0000000Z"); + var nightwingWaiting = new MuninnMoveIdentityDocument( + IdentityId: "nightwing:move-000704a800d0:move-identity", + HostId: "nightwing", + MoveId: "move-000704a800d0", + SourcePath: "bluetooth:00:07:04:A8:00:D0", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "bluetooth-waiting", + Detail: "Muninn sees this trusted PS Move in BlueZ and will attempt bounded pickup while disconnected.", + ObservedAt: "2026-06-17T02:24:20.0000000Z"); + var connected = new MuninnMoveIdentityDocument( + IdentityId: "nightwing:move-000704a6be5f:move-identity", + HostId: "nightwing", + MoveId: "move-000704a6be5f", + SourcePath: "bluetooth:00:07:04:A6:BE:5F", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "bluetooth-connected", + Detail: "Muninn sees this PS Move connected through BlueZ.", + ObservedAt: "2026-06-17T02:24:19.0000000Z"); var encoded = MessagePackSerializer.Serialize(identity); var decoded = MessagePackSerializer.Deserialize(encoded); var snapshot = MimirMuninnMoveEvidenceAdapter.BuildIdentitySnapshots([decoded]).Single(); var expectedHash = MimirMuninnMoveEvidenceAdapter.Fnva64("starfire:move-000704a800d0"); + var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected]); + var rotating = roster.Single(entry => entry.MoveId == "move-000704a800d0"); + var pickedUp = roster.Single(entry => entry.MoveId == "move-000704a6be5f"); Console.WriteLine( - $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X}"); + $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}"); return snapshot.IdentityId == identity.IdentityId && snapshot.HostId == "starfire" && @@ -1669,7 +1690,15 @@ static int RunMuninnMoveIdentitySmoke() snapshot.BluetoothHostAddress == "5C:93:A2:9C:A8:A8" && snapshot.State == "usb-visible" && snapshot.ObservedAtNs > 0 && - snapshot.ControllerIdHash == expectedHash + snapshot.ControllerIdHash == expectedHash && + roster.Count == 2 && + rotating.HasUsbWitness && + rotating.UsbHostIds.SequenceEqual(["starfire"]) && + rotating.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) && + rotating.StateSummary == "usb-visible+bluetooth-waiting" && + rotating.SourcePaths.Length == 2 && + pickedUp.StateSummary == "bluetooth-connected" && + pickedUp.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) ? 0 : 1; } diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index e953a51..06323c6 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -63,6 +63,19 @@ public sealed record MimirMuninnMoveIdentitySnapshot( ulong ObservedAtNs, ulong ControllerIdHash); +public sealed record MimirMuninnMoveRosterEntry( + string MoveId, + ulong ControllerIdHash, + string LatestHostId, + string LatestState, + string StateSummary, + string BluetoothHostAddress, + bool HasUsbWitness, + string[] UsbHostIds, + string[] BluetoothPickupHostIds, + string[] SourcePaths, + ulong ObservedAtNs); + [MessagePackObject] public sealed record MimirMuninnMoveEvidenceStreamFrame( [property: Key(0)] string FrameId, @@ -160,6 +173,17 @@ public static class MimirMuninnMoveEvidenceAdapter .ToArray(); } + public static IReadOnlyList BuildIdentityRoster( + IEnumerable identities) + { + ArgumentNullException.ThrowIfNull(identities); + return BuildIdentitySnapshots(identities) + .GroupBy(identity => identity.MoveId, StringComparer.Ordinal) + .Select(ToRosterEntry) + .OrderBy(entry => entry.MoveId, StringComparer.Ordinal) + .ToArray(); + } + public static MimirMuninnMoveIdentitySnapshot ToIdentitySnapshot(MuninnMoveIdentityDocument identity) { ArgumentNullException.ThrowIfNull(identity); @@ -175,6 +199,71 @@ public static class MimirMuninnMoveEvidenceAdapter ControllerIdHash: Fnva64($"{identity.HostId}:{identity.MoveId}")); } + private static MimirMuninnMoveRosterEntry ToRosterEntry( + IGrouping group) + { + var snapshots = group + .OrderByDescending(identity => identity.ObservedAtNs) + .ThenBy(identity => identity.HostId, StringComparer.Ordinal) + .ThenBy(identity => identity.SourcePath, StringComparer.Ordinal) + .ToArray(); + var latest = snapshots[0]; + var usbHostIds = snapshots + .Where(IsUsbVisible) + .Select(identity => identity.HostId) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); + var pickupHostIds = snapshots + .Where(IsBluetoothPickupState) + .Select(identity => identity.HostId) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); + var sourcePaths = snapshots + .Select(identity => identity.SourcePath) + .Where(path => !string.IsNullOrWhiteSpace(path)) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); + + return new MimirMuninnMoveRosterEntry( + MoveId: group.Key, + ControllerIdHash: Fnva64($"move:{group.Key}"), + LatestHostId: latest.HostId, + LatestState: latest.State, + StateSummary: string.Join("+", snapshots.Select(identity => identity.State) + .Where(state => !string.IsNullOrWhiteSpace(state)) + .Distinct(StringComparer.Ordinal) + .OrderBy(StateRank) + .ThenBy(state => state, StringComparer.Ordinal)), + BluetoothHostAddress: snapshots + .Select(identity => identity.BluetoothHostAddress) + .FirstOrDefault(address => !string.IsNullOrWhiteSpace(address)) ?? string.Empty, + HasUsbWitness: usbHostIds.Length > 0, + UsbHostIds: usbHostIds, + BluetoothPickupHostIds: pickupHostIds, + SourcePaths: sourcePaths, + ObservedAtNs: latest.ObservedAtNs); + } + + private static bool IsUsbVisible(MimirMuninnMoveIdentitySnapshot identity) => + identity.State.Equals("usb-visible", StringComparison.OrdinalIgnoreCase); + + private static bool IsBluetoothPickupState(MimirMuninnMoveIdentitySnapshot identity) => + identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase) || + identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase); + + private static int StateRank(string state) => + state.ToLowerInvariant() switch + { + "usb-visible" => 0, + "bluetooth-connected" => 1, + "bluetooth-waiting" => 2, + "bluetooth-known" => 3, + _ => 9 + }; + public static MimirNativeMoveEvidenceSample ToNativeSample(MuninnMoveMarkerCandidateDocument marker) { ArgumentNullException.ThrowIfNull(marker); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 18b1786..5f0676a 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -144,3 +144,4 @@ {"ts":"2026-06-16T00:12:00+01:00","type":"correction","status":"ok","note":"Source audit found Mimir.CultMeshMedia still using older C# CultMesh.StartNodeAsync/CultMesh.ConnectClient entrypoints while CultLib now exposes C# CultMesh.CreateRudpServer/CreateRudpClient/CreateRudpClientForAuthorizedPeer. Updated maps to treat the media bridge's current transport path as migration debt: typed program/media documents should move to explicit CultNet RUDP helpers, with OBS-local UDP preserved only as compatibility egress."} {"ts": "2026-06-16T18:05:56+01:00", "type": "verification", "status": "ok", "note": "Mimir.CultMeshMedia migrated from old CultMesh node/client transport to explicit CultNet RUDP relay/send/recv path; local smoke artifacts/rudp-media-smoke-20260616-180404 proved 32 MPEG-TS chunks reached Starfire-local UDP sink over RUDP (packets=32 bytes=42112)."} {"ts": "2026-06-17T03:45:44+01:00", "type": "decision", "status": "ok", "note": "Muninn Move identity is now a typed provenance/provisioning lane: Mimir consumes muninn.move_identity.v1 as identity snapshots while controller-state/marker evidence remains the only native reservoir motion evidence. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and --muninn-move-evidence-smoke."} +{"ts": "2026-06-17T05:33:48+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir Move identity roster projection from muninn.move_identity.v1. The roster groups identity records by physical Move so a blind USB rotation can show Starfire/Nightwing USB witnesses and Nightwing Bluetooth waiting/connected pickup state together while keeping identity-only rows out of the native motion reservoir. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and Mimir.Runtime build."} diff --git a/state/map.yaml b/state/map.yaml index e3baf7e..8faf5b0 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -110,6 +110,12 @@ accepted_design: `muninn.move_marker_candidate.v1` and `muninn.move_controller_state.v1` document shapes and maps them into native Move evidence samples without claiming association or pose authority. + - `MimirMuninnMoveEvidenceAdapter` also consumes + `muninn.move_identity.v1` records into a deduped Move roster projection. + The roster can show a controller as USB-visible on one Muninn host and + Bluetooth-waiting/connected on Nightwing at the same time, so blind USB + rotation remains inspectable without promoting identity-only records into + motion evidence. - Realtime Muninn Move witnesses enter over CultMesh streaming frames. CultCache remains persistence/receipt/debug state, not the hot tracking feed. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract for From ff0920b32739d27e4d0d68a884710466e53bdf69 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 09:12:49 +0100 Subject: [PATCH 051/135] Clarify Muninn Move pickup readiness --- src/Mimir.BufferSmoke/Program.cs | 26 ++++++++++-- .../MimirMuninnMoveEvidence.cs | 41 +++++++++++++++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 6c66c9d..7357c5b 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1671,17 +1671,27 @@ static int RunMuninnMoveIdentitySmoke() State: "bluetooth-connected", Detail: "Muninn sees this PS Move connected through BlueZ.", ObservedAt: "2026-06-17T02:24:19.0000000Z"); + var pickupReady = new MuninnMoveIdentityDocument( + IdentityId: "nightwing:move-000704a39772:move-identity", + HostId: "nightwing", + MoveId: "move-000704a39772", + SourcePath: "bluetooth:00:07:04:A3:97:72", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "bluetooth-waiting", + Detail: "Muninn sees this trusted PS Move in BlueZ and will attempt bounded pickup while disconnected.", + ObservedAt: "2026-06-17T02:24:21.0000000Z"); var encoded = MessagePackSerializer.Serialize(identity); var decoded = MessagePackSerializer.Deserialize(encoded); var snapshot = MimirMuninnMoveEvidenceAdapter.BuildIdentitySnapshots([decoded]).Single(); var expectedHash = MimirMuninnMoveEvidenceAdapter.Fnva64("starfire:move-000704a800d0"); - var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected]); + var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected, pickupReady]); var rotating = roster.Single(entry => entry.MoveId == "move-000704a800d0"); var pickedUp = roster.Single(entry => entry.MoveId == "move-000704a6be5f"); + var ready = roster.Single(entry => entry.MoveId == "move-000704a39772"); Console.WriteLine( - $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}"); + $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary}/{rotating.PickupReadiness} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}/{pickedUp.PickupReadiness} ready={ready.StateSummary}/{ready.PickupReadiness}"); return snapshot.IdentityId == identity.IdentityId && snapshot.HostId == "starfire" && @@ -1691,14 +1701,22 @@ static int RunMuninnMoveIdentitySmoke() snapshot.State == "usb-visible" && snapshot.ObservedAtNs > 0 && snapshot.ControllerIdHash == expectedHash && - roster.Count == 2 && + roster.Count == 3 && rotating.HasUsbWitness && rotating.UsbHostIds.SequenceEqual(["starfire"]) && rotating.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) && + rotating.UsbBlockingHostIds.SequenceEqual(["starfire"]) && rotating.StateSummary == "usb-visible+bluetooth-waiting" && + rotating.PickupReadiness == "usb-owned" && + !rotating.IsBluetoothPickupReady && rotating.SourcePaths.Length == 2 && pickedUp.StateSummary == "bluetooth-connected" && - pickedUp.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) + pickedUp.PickupReadiness == "bluetooth-connected" && + pickedUp.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) && + ready.StateSummary == "bluetooth-waiting" && + ready.PickupReadiness == "pickup-ready" && + ready.IsBluetoothPickupReady && + ready.UsbBlockingHostIds.Length == 0 ? 0 : 1; } diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index 06323c6..59a03ec 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -69,10 +69,13 @@ public sealed record MimirMuninnMoveRosterEntry( string LatestHostId, string LatestState, string StateSummary, + string PickupReadiness, + bool IsBluetoothPickupReady, string BluetoothHostAddress, bool HasUsbWitness, string[] UsbHostIds, string[] BluetoothPickupHostIds, + string[] UsbBlockingHostIds, string[] SourcePaths, ulong ObservedAtNs); @@ -220,6 +223,18 @@ public static class MimirMuninnMoveEvidenceAdapter .Distinct(StringComparer.Ordinal) .Order(StringComparer.Ordinal) .ToArray(); + var waitingPickupHostIds = snapshots + .Where(identity => identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase)) + .Select(identity => identity.HostId) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); + var hasConnectedBluetooth = snapshots.Any(identity => + identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase)); + var pickupReadiness = ResolvePickupReadiness( + usbHostIds, + waitingPickupHostIds, + hasConnectedBluetooth); var sourcePaths = snapshots .Select(identity => identity.SourcePath) .Where(path => !string.IsNullOrWhiteSpace(path)) @@ -237,16 +252,42 @@ public static class MimirMuninnMoveEvidenceAdapter .Distinct(StringComparer.Ordinal) .OrderBy(StateRank) .ThenBy(state => state, StringComparer.Ordinal)), + PickupReadiness: pickupReadiness, + IsBluetoothPickupReady: pickupReadiness == "pickup-ready", BluetoothHostAddress: snapshots .Select(identity => identity.BluetoothHostAddress) .FirstOrDefault(address => !string.IsNullOrWhiteSpace(address)) ?? string.Empty, HasUsbWitness: usbHostIds.Length > 0, UsbHostIds: usbHostIds, BluetoothPickupHostIds: pickupHostIds, + UsbBlockingHostIds: usbHostIds.Length > 0 ? usbHostIds : [], SourcePaths: sourcePaths, ObservedAtNs: latest.ObservedAtNs); } + private static string ResolvePickupReadiness( + string[] usbHostIds, + string[] waitingPickupHostIds, + bool hasConnectedBluetooth) + { + if (usbHostIds.Length > 0) + { + return "usb-owned"; + } + + if (hasConnectedBluetooth) + { + return "bluetooth-connected"; + } + + if (waitingPickupHostIds.Length > 0) + { + return "pickup-ready"; + } + + return "unknown"; + } + private static bool IsUsbVisible(MimirMuninnMoveIdentitySnapshot identity) => identity.State.Equals("usb-visible", StringComparison.OrdinalIgnoreCase); From d3d4b14635e4d7cdb44502837f417667d57ac4cb Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 10:16:09 +0100 Subject: [PATCH 052/135] Add live Muninn Move roster smoke --- src/Mimir.BufferSmoke/Program.cs | 108 ++++++++++++++++++ .../MimirMuninnMoveEvidence.cs | 7 ++ state/evidence.jsonl | 2 + 3 files changed, 117 insertions(+) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 7357c5b..51a6601 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -114,6 +114,16 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-identity-smoke", StringCom return RunMuninnMoveIdentitySmoke(); } +if (args.Any(arg => string.Equals(arg, "--muninn-move-live-roster-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return await RunMuninnMoveLiveRosterSmokeAsync( + ParseRepeatedStringOption( + args, + "--identity-store", + [@"C:\Meta\Odin\state\starfire.muninn.telemetry.cc"])) + .ConfigureAwait(false); +} + if (args.Any(arg => string.Equals(arg, "--muninn-move-cultmesh-stream-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMuninnMoveCultMeshStreamSmoke( @@ -421,6 +431,25 @@ static string ParseStringOption(IReadOnlyList args, string name, string return fallback; } +static string[] ParseRepeatedStringOption(IReadOnlyList args, string name, string[] fallback) +{ + var values = new List(); + for (var index = 0; index < args.Count - 1; index++) + { + if (!string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + values.AddRange(args[index + 1] + .Split([';', ','], StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries)); + } + + return values.Count > 0 + ? values.ToArray() + : fallback; +} + static string DescribeBands(IReadOnlyList bands) { return bands.Count == 0 @@ -1721,6 +1750,61 @@ static int RunMuninnMoveIdentitySmoke() : 1; } +static async Task RunMuninnMoveLiveRosterSmokeAsync(IReadOnlyList identityStorePaths) +{ + var identities = new List(); + var loadedStores = 0; + foreach (var path in identityStorePaths) + { + if (string.IsNullOrWhiteSpace(path)) + { + continue; + } + + var fullPath = Path.GetFullPath(path); + if (!File.Exists(fullPath)) + { + Console.Error.WriteLine($"muninn-move-live-roster-smoke missing-store path={fullPath}"); + continue; + } + + using var cache = await CultCacheMessagePack.OpenAsync(fullPath, new CultCacheOpenOptions + { + PullOnOpen = true + }).ConfigureAwait(false); + var storeIdentities = cache.GetAll().ToArray(); + identities.AddRange(storeIdentities); + loadedStores++; + Console.WriteLine($"muninn-move-live-roster-smoke store={fullPath} identities={storeIdentities.Length}"); + } + + var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster(identities); + foreach (var entry in roster) + { + Console.WriteLine( + $"muninn-move-live-roster-smoke move={entry.MoveId} readiness={entry.PickupReadiness} latest={entry.LatestHostId}/{entry.LatestState} states={entry.StateSummary} usbHosts={string.Join(',', entry.UsbHostIds)} usbBlockingHosts={string.Join(',', entry.UsbBlockingHostIds)} pickupHosts={string.Join(',', entry.BluetoothPickupHostIds)} bluetoothHost={entry.BluetoothHostAddress} sources={string.Join('|', entry.SourcePaths)} observedNs={entry.ObservedAtNs}"); + } + + var hasIdentityRecords = identities.Count > 0; + var hasRoster = roster.Count > 0; + var readinessIsCoherent = roster.All(entry => + { + var hasUsbBlocker = entry.UsbBlockingHostIds.Length > 0; + return entry.PickupReadiness == "usb-owned" + ? hasUsbBlocker + : !hasUsbBlocker; + }); + Console.WriteLine( + $"muninn-move-live-roster-smoke stores={loadedStores}/{identityStorePaths.Count} identities={identities.Count} roster={roster.Count} readinessCoherent={readinessIsCoherent}"); + + return loadedStores > 0 && + hasIdentityRecords && + hasRoster && + readinessIsCoherent + ? 0 + : 1; +} + static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) { if (!File.Exists(nativeReservoirPath)) @@ -6377,3 +6461,27 @@ public sealed record ComplexContourChannelModelEvaluationCase( double DeltaAbsolutePredictionErrorMicroseconds, double DeltaMeanAbsoluteErrorSamples, double DeltaMeanAbsolutePhaseErrorRadians); + +[CultDocument("gamecult.eve.provider_advertisement", "gamecult.eve.provider_advertisement.v1")] +[MessagePackObject] +public sealed record MimirMuninnStoreProviderAdvertisementCompatDocument; + +[CultDocument("muninn.command_boundary", "muninn.command_boundary.v1")] +[MessagePackObject] +public sealed record MimirMuninnStoreCommandBoundaryCompatDocument; + +[CultDocument("muninn.obs_stream_catalog", "muninn.obs_stream_catalog.v1")] +[MessagePackObject] +public sealed record MimirMuninnStoreObsStreamCatalogCompatDocument; + +[CultDocument("muninn.quest_access", "muninn.quest_access.v1")] +[MessagePackObject] +public sealed record MimirMuninnStoreQuestAccessCompatDocument; + +[CultDocument("muninn.telemetry_surface", "muninn.telemetry_surface.v1")] +[MessagePackObject] +public sealed record MimirMuninnStoreTelemetrySurfaceCompatDocument; + +[CultDocument("muninn.transport_profile", "muninn.transport_profile.v1")] +[MessagePackObject] +public sealed record MimirMuninnStoreTransportProfileCompatDocument; diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index 59a03ec..d387496 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -393,6 +393,13 @@ public static class MimirMuninnMoveEvidenceAdapter private static ulong ObservedAtToUnixNs(string observedAt) { + const string unixPrefix = "unix-"; + if (observedAt.StartsWith(unixPrefix, StringComparison.OrdinalIgnoreCase) && + ulong.TryParse(observedAt[unixPrefix.Length..], out var unixSeconds)) + { + return checked(unixSeconds * 1_000_000_000UL); + } + if (!DateTimeOffset.TryParse(observedAt, out var parsed)) { return 0; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 5f0676a..c143796 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -145,3 +145,5 @@ {"ts": "2026-06-16T18:05:56+01:00", "type": "verification", "status": "ok", "note": "Mimir.CultMeshMedia migrated from old CultMesh node/client transport to explicit CultNet RUDP relay/send/recv path; local smoke artifacts/rudp-media-smoke-20260616-180404 proved 32 MPEG-TS chunks reached Starfire-local UDP sink over RUDP (packets=32 bytes=42112)."} {"ts": "2026-06-17T03:45:44+01:00", "type": "decision", "status": "ok", "note": "Muninn Move identity is now a typed provenance/provisioning lane: Mimir consumes muninn.move_identity.v1 as identity snapshots while controller-state/marker evidence remains the only native reservoir motion evidence. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and --muninn-move-evidence-smoke."} {"ts": "2026-06-17T05:33:48+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir Move identity roster projection from muninn.move_identity.v1. The roster groups identity records by physical Move so a blind USB rotation can show Starfire/Nightwing USB witnesses and Nightwing Bluetooth waiting/connected pickup state together while keeping identity-only rows out of the native motion reservoir. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and Mimir.Runtime build."} +{"ts": "2026-06-17T07:32:58+01:00", "type": "verification", "status": "ok", "note": "2026-06-17 Raven OBS feed owner is actual Muninn. Canonical C:\\Meta\\Odin\\Muninn\\muninn.exe was stale (3853312 bytes) while the working bridge build existed as muninn-fixed4.exe (2020352 bytes). Replacing canonical muninn.exe with the latest release build restored Raven->Starfire SRT delivery: single-target proof on port 5520 produced H.264 1920x1080 video plus AAC stereo in capture.ts, and dual-target proof on ports 5521/5524 delivered 262144-byte MPEG-TS captures to both listeners."} +{"ts": "2026-06-17T10:10:40+01:00", "type": "verification", "status": "ok", "note": "Mimir BufferSmoke now reads live Starfire/Nightwing Muninn CultCache move_identity records with opaque companion schema stubs; live roster smoke saw A8 as usb-owned via Starfire USB plus Nightwing waiting, F5 bluetooth-connected, and A3/A6 pickup-ready."} From 5eef53c0e2005baa8494cada168eececf55afd94 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 11:33:34 +0100 Subject: [PATCH 053/135] Record Muninn Move pickup verification --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index c143796..705fdc5 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -147,3 +147,4 @@ {"ts": "2026-06-17T05:33:48+01:00", "type": "implementation", "status": "ok", "note": "Added Mimir Move identity roster projection from muninn.move_identity.v1. The roster groups identity records by physical Move so a blind USB rotation can show Starfire/Nightwing USB witnesses and Nightwing Bluetooth waiting/connected pickup state together while keeping identity-only rows out of the native motion reservoir. Verified with Mimir.BufferSmoke --muninn-move-identity-smoke and Mimir.Runtime build."} {"ts": "2026-06-17T07:32:58+01:00", "type": "verification", "status": "ok", "note": "2026-06-17 Raven OBS feed owner is actual Muninn. Canonical C:\\Meta\\Odin\\Muninn\\muninn.exe was stale (3853312 bytes) while the working bridge build existed as muninn-fixed4.exe (2020352 bytes). Replacing canonical muninn.exe with the latest release build restored Raven->Starfire SRT delivery: single-target proof on port 5520 produced H.264 1920x1080 video plus AAC stereo in capture.ts, and dual-target proof on ports 5521/5524 delivered 262144-byte MPEG-TS captures to both listeners."} {"ts": "2026-06-17T10:10:40+01:00", "type": "verification", "status": "ok", "note": "Mimir BufferSmoke now reads live Starfire/Nightwing Muninn CultCache move_identity records with opaque companion schema stubs; live roster smoke saw A8 as usb-owned via Starfire USB plus Nightwing waiting, F5 bluetooth-connected, and A3/A6 pickup-ready."} +{"ts": "2026-06-17T11:32:31+01:00", "type": "verification", "status": "ok", "note": "After Odin beed394 deployed to Nightwing, Muninn restart pipeline claims USB Moves, launches with --move-host plus discovered --move-state sources, preserves A3 as usb-visible instead of Bluetooth overwrite, publishes BlueZ pickup failure detail for A6/A8, and Mimir live roster smoke over Starfire+Nightwing stores reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 pickup-ready, and A8 usb-owned on Starfire with Nightwing waiting."} From 32bd680ccee7296be687aaa6bd5e2fe45be1a7fa Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 11:55:20 +0100 Subject: [PATCH 054/135] Distinguish unreachable Muninn Moves --- src/Mimir.BufferSmoke/Program.cs | 23 +++++++++++++++---- .../MimirMuninnMoveEvidence.cs | 18 +++++++++++---- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 51a6601..4cb80cf 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1709,18 +1709,28 @@ static int RunMuninnMoveIdentitySmoke() State: "bluetooth-waiting", Detail: "Muninn sees this trusted PS Move in BlueZ and will attempt bounded pickup while disconnected.", ObservedAt: "2026-06-17T02:24:21.0000000Z"); + var unreachable = new MuninnMoveIdentityDocument( + IdentityId: "nightwing:move-000704a12345:move-identity", + HostId: "nightwing", + MoveId: "move-000704a12345", + SourcePath: "bluetooth:00:07:04:A1:23:45", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "bluetooth-unreachable", + Detail: "Muninn sees this trusted PS Move in BlueZ and will attempt bounded pickup while disconnected. Last pickup attempt: Failed to connect: org.bluez.Error.Failed br-connection-create-socket", + ObservedAt: "2026-06-17T02:24:22.0000000Z"); var encoded = MessagePackSerializer.Serialize(identity); var decoded = MessagePackSerializer.Deserialize(encoded); var snapshot = MimirMuninnMoveEvidenceAdapter.BuildIdentitySnapshots([decoded]).Single(); var expectedHash = MimirMuninnMoveEvidenceAdapter.Fnva64("starfire:move-000704a800d0"); - var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected, pickupReady]); + var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected, pickupReady, unreachable]); var rotating = roster.Single(entry => entry.MoveId == "move-000704a800d0"); var pickedUp = roster.Single(entry => entry.MoveId == "move-000704a6be5f"); var ready = roster.Single(entry => entry.MoveId == "move-000704a39772"); + var blocked = roster.Single(entry => entry.MoveId == "move-000704a12345"); Console.WriteLine( - $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary}/{rotating.PickupReadiness} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}/{pickedUp.PickupReadiness} ready={ready.StateSummary}/{ready.PickupReadiness}"); + $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary}/{rotating.PickupReadiness} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}/{pickedUp.PickupReadiness} ready={ready.StateSummary}/{ready.PickupReadiness} blocked={blocked.StateSummary}/{blocked.PickupReadiness}"); return snapshot.IdentityId == identity.IdentityId && snapshot.HostId == "starfire" && @@ -1730,7 +1740,7 @@ static int RunMuninnMoveIdentitySmoke() snapshot.State == "usb-visible" && snapshot.ObservedAtNs > 0 && snapshot.ControllerIdHash == expectedHash && - roster.Count == 3 && + roster.Count == 4 && rotating.HasUsbWitness && rotating.UsbHostIds.SequenceEqual(["starfire"]) && rotating.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) && @@ -1745,7 +1755,12 @@ static int RunMuninnMoveIdentitySmoke() ready.StateSummary == "bluetooth-waiting" && ready.PickupReadiness == "pickup-ready" && ready.IsBluetoothPickupReady && - ready.UsbBlockingHostIds.Length == 0 + ready.UsbBlockingHostIds.Length == 0 && + blocked.StateSummary == "bluetooth-unreachable" && + blocked.PickupReadiness == "bluetooth-unreachable" && + !blocked.IsBluetoothPickupReady && + blocked.BluetoothPickupHostIds.Length == 0 && + blocked.UsbBlockingHostIds.Length == 0 ? 0 : 1; } diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index d387496..1549858 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -231,10 +231,13 @@ public static class MimirMuninnMoveEvidenceAdapter .ToArray(); var hasConnectedBluetooth = snapshots.Any(identity => identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase)); + var hasUnreachableBluetooth = snapshots.Any(identity => + identity.State.Equals("bluetooth-unreachable", StringComparison.OrdinalIgnoreCase)); var pickupReadiness = ResolvePickupReadiness( usbHostIds, waitingPickupHostIds, - hasConnectedBluetooth); + hasConnectedBluetooth, + hasUnreachableBluetooth); var sourcePaths = snapshots .Select(identity => identity.SourcePath) .Where(path => !string.IsNullOrWhiteSpace(path)) @@ -268,7 +271,8 @@ public static class MimirMuninnMoveEvidenceAdapter private static string ResolvePickupReadiness( string[] usbHostIds, string[] waitingPickupHostIds, - bool hasConnectedBluetooth) + bool hasConnectedBluetooth, + bool hasUnreachableBluetooth) { if (usbHostIds.Length > 0) { @@ -280,6 +284,11 @@ public static class MimirMuninnMoveEvidenceAdapter return "bluetooth-connected"; } + if (hasUnreachableBluetooth) + { + return "bluetooth-unreachable"; + } + if (waitingPickupHostIds.Length > 0) { return "pickup-ready"; @@ -300,8 +309,9 @@ public static class MimirMuninnMoveEvidenceAdapter { "usb-visible" => 0, "bluetooth-connected" => 1, - "bluetooth-waiting" => 2, - "bluetooth-known" => 3, + "bluetooth-unreachable" => 2, + "bluetooth-waiting" => 3, + "bluetooth-known" => 4, _ => 9 }; From 6192330b49e9d9aa5ccb331573bdeb4c7aabfa83 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 12:02:11 +0100 Subject: [PATCH 055/135] Record unreachable Move roster verification --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 705fdc5..51e9c7e 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -148,3 +148,4 @@ {"ts": "2026-06-17T07:32:58+01:00", "type": "verification", "status": "ok", "note": "2026-06-17 Raven OBS feed owner is actual Muninn. Canonical C:\\Meta\\Odin\\Muninn\\muninn.exe was stale (3853312 bytes) while the working bridge build existed as muninn-fixed4.exe (2020352 bytes). Replacing canonical muninn.exe with the latest release build restored Raven->Starfire SRT delivery: single-target proof on port 5520 produced H.264 1920x1080 video plus AAC stereo in capture.ts, and dual-target proof on ports 5521/5524 delivered 262144-byte MPEG-TS captures to both listeners."} {"ts": "2026-06-17T10:10:40+01:00", "type": "verification", "status": "ok", "note": "Mimir BufferSmoke now reads live Starfire/Nightwing Muninn CultCache move_identity records with opaque companion schema stubs; live roster smoke saw A8 as usb-owned via Starfire USB plus Nightwing waiting, F5 bluetooth-connected, and A3/A6 pickup-ready."} {"ts": "2026-06-17T11:32:31+01:00", "type": "verification", "status": "ok", "note": "After Odin beed394 deployed to Nightwing, Muninn restart pipeline claims USB Moves, launches with --move-host plus discovered --move-state sources, preserves A3 as usb-visible instead of Bluetooth overwrite, publishes BlueZ pickup failure detail for A6/A8, and Mimir live roster smoke over Starfire+Nightwing stores reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 pickup-ready, and A8 usb-owned on Starfire with Nightwing waiting."} +{"ts": "2026-06-17T12:00:49+01:00", "type": "verification", "status": "ok", "note": "After Odin 9ea3ba2 and Mimir 32bd680c, Muninn publishes A6/A8 as bluetooth-unreachable when BlueZ returns br-connection-create-socket/Host is down; Mimir live roster smoke over Starfire+Nightwing now reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 bluetooth-unreachable, and A8 usb-owned on Starfire with Nightwing Bluetooth unreachable rather than pickup-ready."} From d53ba0c82c4f2e75adfdf771cf8635f50a742213 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 12:15:07 +0100 Subject: [PATCH 056/135] Record Move wake blocker evidence --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 51e9c7e..df59df2 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -149,3 +149,4 @@ {"ts": "2026-06-17T10:10:40+01:00", "type": "verification", "status": "ok", "note": "Mimir BufferSmoke now reads live Starfire/Nightwing Muninn CultCache move_identity records with opaque companion schema stubs; live roster smoke saw A8 as usb-owned via Starfire USB plus Nightwing waiting, F5 bluetooth-connected, and A3/A6 pickup-ready."} {"ts": "2026-06-17T11:32:31+01:00", "type": "verification", "status": "ok", "note": "After Odin beed394 deployed to Nightwing, Muninn restart pipeline claims USB Moves, launches with --move-host plus discovered --move-state sources, preserves A3 as usb-visible instead of Bluetooth overwrite, publishes BlueZ pickup failure detail for A6/A8, and Mimir live roster smoke over Starfire+Nightwing stores reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 pickup-ready, and A8 usb-owned on Starfire with Nightwing waiting."} {"ts": "2026-06-17T12:00:49+01:00", "type": "verification", "status": "ok", "note": "After Odin 9ea3ba2 and Mimir 32bd680c, Muninn publishes A6/A8 as bluetooth-unreachable when BlueZ returns br-connection-create-socket/Host is down; Mimir live roster smoke over Starfire+Nightwing now reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 bluetooth-unreachable, and A8 usb-owned on Starfire with Nightwing Bluetooth unreachable rather than pickup-ready."} +{"ts": "2026-06-17T12:14:17+01:00", "type": "verification", "status": "ok", "note": "Odin 8e16218 deployed to Nightwing. Muninn now publishes bluetooth-unreachable Move identities with explicit BlueZ host-down wake guidance: A6/A8 detail says br-connection-create-socket, likely asleep/off-air, wake with PS button or reattach USB. This clarifies the remaining physical blocker; automatic Bluetooth pickup is still unproven for those controllers."} From 79e78656710e0c5c33b5411984d538bf089ed268 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 12:41:59 +0100 Subject: [PATCH 057/135] Record Nightwing Muninn hot release deploy --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index df59df2..de5f2d3 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -150,3 +150,4 @@ {"ts": "2026-06-17T11:32:31+01:00", "type": "verification", "status": "ok", "note": "After Odin beed394 deployed to Nightwing, Muninn restart pipeline claims USB Moves, launches with --move-host plus discovered --move-state sources, preserves A3 as usb-visible instead of Bluetooth overwrite, publishes BlueZ pickup failure detail for A6/A8, and Mimir live roster smoke over Starfire+Nightwing stores reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 pickup-ready, and A8 usb-owned on Starfire with Nightwing waiting."} {"ts": "2026-06-17T12:00:49+01:00", "type": "verification", "status": "ok", "note": "After Odin 9ea3ba2 and Mimir 32bd680c, Muninn publishes A6/A8 as bluetooth-unreachable when BlueZ returns br-connection-create-socket/Host is down; Mimir live roster smoke over Starfire+Nightwing now reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 bluetooth-unreachable, and A8 usb-owned on Starfire with Nightwing Bluetooth unreachable rather than pickup-ready."} {"ts": "2026-06-17T12:14:17+01:00", "type": "verification", "status": "ok", "note": "Odin 8e16218 deployed to Nightwing. Muninn now publishes bluetooth-unreachable Move identities with explicit BlueZ host-down wake guidance: A6/A8 detail says br-connection-create-socket, likely asleep/off-air, wake with PS button or reattach USB. This clarifies the remaining physical blocker; automatic Bluetooth pickup is still unproven for those controllers."} +{"ts": "2026-06-17T12:39:58+01:00", "type": "verification", "status": "ok", "note": "Muninn ca4e49b deployed on Nightwing: released local USB-visible Move IDs now clear Bluetooth pickup backoff before the centralized BlueZ connect loop. cargo test -p muninn-daemon passed 41/41; Nightwing health reports store_age=0s and four identities. Live A6/A8 still classify bluetooth-unreachable with br-connection-create-socket, so physical wake/reattach remains the blocker."} From c7fa6fb8acddb75144e81ccc2fdf7d720483b296 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 13:17:51 +0100 Subject: [PATCH 058/135] Record Starfire Move paired host verification --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index de5f2d3..ea2b3a9 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -151,3 +151,4 @@ {"ts": "2026-06-17T12:00:49+01:00", "type": "verification", "status": "ok", "note": "After Odin 9ea3ba2 and Mimir 32bd680c, Muninn publishes A6/A8 as bluetooth-unreachable when BlueZ returns br-connection-create-socket/Host is down; Mimir live roster smoke over Starfire+Nightwing now reports F5 bluetooth-connected, A3 usb-owned on Nightwing, A6 bluetooth-unreachable, and A8 usb-owned on Starfire with Nightwing Bluetooth unreachable rather than pickup-ready."} {"ts": "2026-06-17T12:14:17+01:00", "type": "verification", "status": "ok", "note": "Odin 8e16218 deployed to Nightwing. Muninn now publishes bluetooth-unreachable Move identities with explicit BlueZ host-down wake guidance: A6/A8 detail says br-connection-create-socket, likely asleep/off-air, wake with PS button or reattach USB. This clarifies the remaining physical blocker; automatic Bluetooth pickup is still unproven for those controllers."} {"ts": "2026-06-17T12:39:58+01:00", "type": "verification", "status": "ok", "note": "Muninn ca4e49b deployed on Nightwing: released local USB-visible Move IDs now clear Bluetooth pickup backoff before the centralized BlueZ connect loop. cargo test -p muninn-daemon passed 41/41; Nightwing health reports store_age=0s and four identities. Live A6/A8 still classify bluetooth-unreachable with br-connection-create-socket, so physical wake/reattach remains the blocker."} +{"ts": "2026-06-17T13:16:04+01:00", "type": "verification", "status": "ok", "note": "Odin ef4c51a deployed to Starfire Muninn. Windows Move identity status now reads paired Bluetooth host from the sibling PS Move pairing collection when --move-host is omitted: no-flag move-identity-status reports A8 as usb-visible with bluetooth_host=5C:93:A2:9C:A8:A8, and the claim log shows controller=00:07:04:a8:00:d0 host_after=5c:93:a2:9c:a8:a8. Installed Starfire Muninn hash matches target/debug hash 6062D174FF9DF281C098B35E84E7D3488606F6EC8842D6B5F1C8B6ACBD30C7E7."} From 8042885c6ddf427c9a73ffe03568ce07599a7d05 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 13:29:44 +0100 Subject: [PATCH 059/135] Record cross-host Move roster smoke --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index ea2b3a9..a13924d 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -152,3 +152,4 @@ {"ts": "2026-06-17T12:14:17+01:00", "type": "verification", "status": "ok", "note": "Odin 8e16218 deployed to Nightwing. Muninn now publishes bluetooth-unreachable Move identities with explicit BlueZ host-down wake guidance: A6/A8 detail says br-connection-create-socket, likely asleep/off-air, wake with PS button or reattach USB. This clarifies the remaining physical blocker; automatic Bluetooth pickup is still unproven for those controllers."} {"ts": "2026-06-17T12:39:58+01:00", "type": "verification", "status": "ok", "note": "Muninn ca4e49b deployed on Nightwing: released local USB-visible Move IDs now clear Bluetooth pickup backoff before the centralized BlueZ connect loop. cargo test -p muninn-daemon passed 41/41; Nightwing health reports store_age=0s and four identities. Live A6/A8 still classify bluetooth-unreachable with br-connection-create-socket, so physical wake/reattach remains the blocker."} {"ts": "2026-06-17T13:16:04+01:00", "type": "verification", "status": "ok", "note": "Odin ef4c51a deployed to Starfire Muninn. Windows Move identity status now reads paired Bluetooth host from the sibling PS Move pairing collection when --move-host is omitted: no-flag move-identity-status reports A8 as usb-visible with bluetooth_host=5C:93:A2:9C:A8:A8, and the claim log shows controller=00:07:04:a8:00:d0 host_after=5c:93:a2:9c:a8:a8. Installed Starfire Muninn hash matches target/debug hash 6062D174FF9DF281C098B35E84E7D3488606F6EC8842D6B5F1C8B6ACBD30C7E7."} +{"ts": "2026-06-17T13:27:55+01:00", "type": "verification", "status": "ok", "note": "Mimir live Muninn roster smoke after Odin ef4c51a and refreshed Nightwing store loaded Starfire+Nightwing identity stores 2/2, identities=6, roster=4, readinessCoherent=True. A8 now resolves as usb-owned on Starfire with bluetoothHost=5C:93:A2:9C:A8:A8 while Nightwing contributes bluetooth-unreachable for the same move id; F5 is bluetooth-connected, A3 is usb-owned on Nightwing, and A6 remains bluetooth-unreachable."} From d9005e340b445eea16191847197c666db677d433 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 17 Jun 2026 14:07:35 +0100 Subject: [PATCH 060/135] Ignore stale Muninn Move identities in live roster --- src/Mimir.BufferSmoke/Program.cs | 31 ++++++++++++++++--- .../MimirMuninnMoveEvidence.cs | 19 ++++++++++-- state/evidence.jsonl | 1 + 3 files changed, 45 insertions(+), 6 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 4cb80cf..32d9911 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1718,19 +1718,38 @@ static int RunMuninnMoveIdentitySmoke() State: "bluetooth-unreachable", Detail: "Muninn sees this trusted PS Move in BlueZ and will attempt bounded pickup while disconnected. Last pickup attempt: Failed to connect: org.bluez.Error.Failed br-connection-create-socket", ObservedAt: "2026-06-17T02:24:22.0000000Z"); + var staleUsb = new MuninnMoveIdentityDocument( + IdentityId: "starfire:move-000704a99999:move-identity", + HostId: "starfire", + MoveId: "move-000704a99999", + SourcePath: @"windows-psmove:\\?\hid#vid_054c&pid_03d5&col01#old#{guid}", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "usb-visible", + Detail: "Old USB observation retained in CultCache history.", + ObservedAt: "2026-06-17T02:20:00.0000000Z"); + var freshWaitingAfterStaleUsb = new MuninnMoveIdentityDocument( + IdentityId: "nightwing:move-000704a99999:move-identity", + HostId: "nightwing", + MoveId: "move-000704a99999", + SourcePath: "bluetooth:00:07:04:A9:99:99", + BluetoothHostAddress: "5C:93:A2:9C:A8:A8", + State: "bluetooth-waiting", + Detail: "Fresh Bluetooth pickup witness after stale USB observation.", + ObservedAt: "2026-06-17T02:24:30.0000000Z"); var encoded = MessagePackSerializer.Serialize(identity); var decoded = MessagePackSerializer.Deserialize(encoded); var snapshot = MimirMuninnMoveEvidenceAdapter.BuildIdentitySnapshots([decoded]).Single(); var expectedHash = MimirMuninnMoveEvidenceAdapter.Fnva64("starfire:move-000704a800d0"); - var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected, pickupReady, unreachable]); + var roster = MimirMuninnMoveEvidenceAdapter.BuildIdentityRoster([decoded, nightwingWaiting, connected, pickupReady, unreachable, staleUsb, freshWaitingAfterStaleUsb]); var rotating = roster.Single(entry => entry.MoveId == "move-000704a800d0"); var pickedUp = roster.Single(entry => entry.MoveId == "move-000704a6be5f"); var ready = roster.Single(entry => entry.MoveId == "move-000704a39772"); var blocked = roster.Single(entry => entry.MoveId == "move-000704a12345"); + var staleReleased = roster.Single(entry => entry.MoveId == "move-000704a99999"); Console.WriteLine( - $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary}/{rotating.PickupReadiness} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}/{pickedUp.PickupReadiness} ready={ready.StateSummary}/{ready.PickupReadiness} blocked={blocked.StateSummary}/{blocked.PickupReadiness}"); + $"muninn-move-identity-smoke identity={snapshot.IdentityId} move={snapshot.MoveId} source={snapshot.SourcePath} bluetoothHost={snapshot.BluetoothHostAddress} state={snapshot.State} observedNs={snapshot.ObservedAtNs} controllerHash=0x{snapshot.ControllerIdHash:X} roster={roster.Count} rotating={rotating.StateSummary}/{rotating.PickupReadiness} usbHosts={string.Join(',', rotating.UsbHostIds)} pickupHosts={string.Join(',', rotating.BluetoothPickupHostIds)} pickedUp={pickedUp.StateSummary}/{pickedUp.PickupReadiness} ready={ready.StateSummary}/{ready.PickupReadiness} blocked={blocked.StateSummary}/{blocked.PickupReadiness} staleReleased={staleReleased.StateSummary}/{staleReleased.PickupReadiness}"); return snapshot.IdentityId == identity.IdentityId && snapshot.HostId == "starfire" && @@ -1740,7 +1759,7 @@ static int RunMuninnMoveIdentitySmoke() snapshot.State == "usb-visible" && snapshot.ObservedAtNs > 0 && snapshot.ControllerIdHash == expectedHash && - roster.Count == 4 && + roster.Count == 5 && rotating.HasUsbWitness && rotating.UsbHostIds.SequenceEqual(["starfire"]) && rotating.BluetoothPickupHostIds.SequenceEqual(["nightwing"]) && @@ -1760,7 +1779,11 @@ static int RunMuninnMoveIdentitySmoke() blocked.PickupReadiness == "bluetooth-unreachable" && !blocked.IsBluetoothPickupReady && blocked.BluetoothPickupHostIds.Length == 0 && - blocked.UsbBlockingHostIds.Length == 0 + blocked.UsbBlockingHostIds.Length == 0 && + staleReleased.StateSummary == "bluetooth-waiting" && + staleReleased.PickupReadiness == "pickup-ready" && + staleReleased.IsBluetoothPickupReady && + staleReleased.UsbBlockingHostIds.Length == 0 ? 0 : 1; } diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index 1549858..efae861 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -89,6 +89,8 @@ public sealed record MimirMuninnMoveEvidenceStreamFrame( public static class MimirMuninnMoveEvidenceAdapter { + private const ulong LiveIdentityWindowNs = 60_000_000_000UL; + public const string StreamMetadataSchemaId = "mimir.muninn_move_evidence_stream_frame.v1"; public static CultMeshStreamDescriptor CreateStreamDescriptor( @@ -205,12 +207,19 @@ public static class MimirMuninnMoveEvidenceAdapter private static MimirMuninnMoveRosterEntry ToRosterEntry( IGrouping group) { - var snapshots = group + var orderedSnapshots = group .OrderByDescending(identity => identity.ObservedAtNs) .ThenBy(identity => identity.HostId, StringComparer.Ordinal) .ThenBy(identity => identity.SourcePath, StringComparer.Ordinal) .ToArray(); - var latest = snapshots[0]; + var latest = orderedSnapshots[0]; + var snapshots = orderedSnapshots + .Where(identity => IsLiveIdentitySnapshot(identity, latest.ObservedAtNs)) + .ToArray(); + if (snapshots.Length == 0) + { + snapshots = [latest]; + } var usbHostIds = snapshots .Where(IsUsbVisible) .Select(identity => identity.HostId) @@ -300,6 +309,12 @@ public static class MimirMuninnMoveEvidenceAdapter private static bool IsUsbVisible(MimirMuninnMoveIdentitySnapshot identity) => identity.State.Equals("usb-visible", StringComparison.OrdinalIgnoreCase); + private static bool IsLiveIdentitySnapshot(MimirMuninnMoveIdentitySnapshot identity, ulong latestObservedAtNs) => + identity.ObservedAtNs <= 0 || + latestObservedAtNs <= 0 || + identity.ObservedAtNs > latestObservedAtNs || + latestObservedAtNs - identity.ObservedAtNs <= LiveIdentityWindowNs; + private static bool IsBluetoothPickupState(MimirMuninnMoveIdentitySnapshot identity) => identity.State.Equals("bluetooth-waiting", StringComparison.OrdinalIgnoreCase) || identity.State.Equals("bluetooth-connected", StringComparison.OrdinalIgnoreCase); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index a13924d..1398959 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -153,3 +153,4 @@ {"ts": "2026-06-17T12:39:58+01:00", "type": "verification", "status": "ok", "note": "Muninn ca4e49b deployed on Nightwing: released local USB-visible Move IDs now clear Bluetooth pickup backoff before the centralized BlueZ connect loop. cargo test -p muninn-daemon passed 41/41; Nightwing health reports store_age=0s and four identities. Live A6/A8 still classify bluetooth-unreachable with br-connection-create-socket, so physical wake/reattach remains the blocker."} {"ts": "2026-06-17T13:16:04+01:00", "type": "verification", "status": "ok", "note": "Odin ef4c51a deployed to Starfire Muninn. Windows Move identity status now reads paired Bluetooth host from the sibling PS Move pairing collection when --move-host is omitted: no-flag move-identity-status reports A8 as usb-visible with bluetooth_host=5C:93:A2:9C:A8:A8, and the claim log shows controller=00:07:04:a8:00:d0 host_after=5c:93:a2:9c:a8:a8. Installed Starfire Muninn hash matches target/debug hash 6062D174FF9DF281C098B35E84E7D3488606F6EC8842D6B5F1C8B6ACBD30C7E7."} {"ts": "2026-06-17T13:27:55+01:00", "type": "verification", "status": "ok", "note": "Mimir live Muninn roster smoke after Odin ef4c51a and refreshed Nightwing store loaded Starfire+Nightwing identity stores 2/2, identities=6, roster=4, readinessCoherent=True. A8 now resolves as usb-owned on Starfire with bluetoothHost=5C:93:A2:9C:A8:A8 while Nightwing contributes bluetooth-unreachable for the same move id; F5 is bluetooth-connected, A3 is usb-owned on Nightwing, and A6 remains bluetooth-unreachable."} +{"ts": "2026-06-17T14:05:52+01:00", "type": "verification", "status": "ok", "note": "After Odin b6f8119 deployed to Nightwing and Starfire, Muninn live source merge drops stale configured platform Move sources when platform discovery has live sources, so static --move-state joystick paths no longer keep unplugged Moves USB-owned. Mimir roster derivation now ignores identity snapshots older than the live identity window relative to each Move's newest observation. Focused identity smoke reports staleReleased=bluetooth-waiting/pickup-ready. Cross-host live roster smoke with refreshed Nightwing store loaded stores=2/2, identities=6, roster=4, readinessCoherent=True: F5 bluetooth-connected, A3 usb-owned on Starfire with Nightwing unreachable witness, A6 bluetooth-unreachable, A8 usb-owned on Nightwing."} From e00458ff3be2a92cae166a30964dc6ecbcac3dbe Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 5 Jul 2026 22:35:50 +0100 Subject: [PATCH 061/135] Restore Muninn OBS plugin with Odin discovery --- native/obs_stem_source/CMakeLists.txt | 33 + native/obs_stem_source/data/locale/en-US.ini | 14 + .../mimir_obs_muninn_source.cpp | 4118 +++++++++++++++++ .../mimir_obs_program_texture_source.cpp | 628 +++ native/obs_stem_source/mimir_obs_stem_ipc.h | 43 + .../obs_stem_source/mimir_obs_stem_source.cpp | 219 + native/obs_stem_source/muninn_media_wire.h | 267 ++ native/obs_stem_source/muninn_rudp_ack.h | 54 + .../obs_stem_source/muninn_rudp_fragments.h | 127 + native/obs_stem_source/muninn_rudp_url.h | 123 + .../tests/muninn_rudp_url_tests.cpp | 58 + 11 files changed, 5684 insertions(+) create mode 100644 native/obs_stem_source/CMakeLists.txt create mode 100644 native/obs_stem_source/data/locale/en-US.ini create mode 100644 native/obs_stem_source/mimir_obs_muninn_source.cpp create mode 100644 native/obs_stem_source/mimir_obs_program_texture_source.cpp create mode 100644 native/obs_stem_source/mimir_obs_stem_ipc.h create mode 100644 native/obs_stem_source/mimir_obs_stem_source.cpp create mode 100644 native/obs_stem_source/muninn_media_wire.h create mode 100644 native/obs_stem_source/muninn_rudp_ack.h create mode 100644 native/obs_stem_source/muninn_rudp_fragments.h create mode 100644 native/obs_stem_source/muninn_rudp_url.h create mode 100644 native/obs_stem_source/tests/muninn_rudp_url_tests.cpp diff --git a/native/obs_stem_source/CMakeLists.txt b/native/obs_stem_source/CMakeLists.txt new file mode 100644 index 0000000..545a732 --- /dev/null +++ b/native/obs_stem_source/CMakeLists.txt @@ -0,0 +1,33 @@ +cmake_minimum_required(VERSION 3.20) + +project(mimir_obs_stem_source LANGUAGES CXX) + +find_package(libobs REQUIRED CONFIG) +include(CTest) + +add_library(mimir_obs_stem_source MODULE + mimir_obs_muninn_source.cpp + mimir_obs_program_texture_source.cpp + mimir_obs_stem_source.cpp) + +target_compile_features(mimir_obs_stem_source PRIVATE cxx_std_20) +target_compile_definitions(mimir_obs_stem_source PRIVATE + WIN32_LEAN_AND_MEAN + NOMINMAX + _WIN32_WINNT=0x0A00) +target_link_libraries(mimir_obs_stem_source PRIVATE OBS::libobs d3d11 d3d12 dxgi ws2_32) + +if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/../../tools/muninn_media_decoder_probe.cpp") + add_executable(muninn_media_decoder_probe + ../../tools/muninn_media_decoder_probe.cpp) + target_compile_features(muninn_media_decoder_probe PRIVATE cxx_std_20) + target_include_directories(muninn_media_decoder_probe PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}) +endif() + +add_executable(muninn_rudp_url_tests + tests/muninn_rudp_url_tests.cpp) +target_compile_features(muninn_rudp_url_tests PRIVATE cxx_std_20) +target_include_directories(muninn_rudp_url_tests PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}) +add_test(NAME muninn_rudp_url_tests COMMAND muninn_rudp_url_tests) diff --git a/native/obs_stem_source/data/locale/en-US.ini b/native/obs_stem_source/data/locale/en-US.ini new file mode 100644 index 0000000..50b760f --- /dev/null +++ b/native/obs_stem_source/data/locale/en-US.ini @@ -0,0 +1,14 @@ +MimirAudioStem="Mimir Audio Stem" +MimirAudioStem.MapName="Shared memory map" +MimirAudioStem.StemId="Stem id" +MimirProgramTexture="Mimir Program Texture" +MuninnStream="Muninn Stream" +MuninnVideoSource="Muninn Video" +MuninnAudioSource="Muninn Audio" +MuninnStream.Stream="Muninn stream" +MuninnStream.Video="Video device" +MuninnStream.Audio="Audio device" +MuninnStream.VideoBitrateKbps="Video bitrate (kbps)" +MuninnStream.VideoLatencyMs="Video/transport latency (ms)" +MuninnStream.AudioReorderMs="Audio reorder latency (ms)" +MuninnStream.Refresh="Refresh Muninn streams" diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp new file mode 100644 index 0000000..3a1613b --- /dev/null +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -0,0 +1,4118 @@ +#ifdef _WIN32 +#include +#include +#include +#endif + +#include +#include + +#include "muninn_media_wire.h" +#include "muninn_rudp_ack.h" +#include "muninn_rudp_fragments.h" +#include "muninn_rudp_url.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr const char *MuninnStorePath = "C:\\Meta\\Odin\\state\\muninn.telemetry.cc"; +constexpr const char *MuninnObsCommandStorePath = "C:\\Meta\\Odin\\state\\muninn.obs.request.cc"; +constexpr const char *MuninnCommandExePath = "C:\\Meta\\Odin\\Muninn\\muninn.exe"; +constexpr const char *MuninnCommandRudpTarget = "192.168.1.84:17873"; +constexpr const char *MuninnDeprecatedWireGuardCommandRudpTarget = "10.77.0.4:17873"; +constexpr const char *MuninnDefaultHostId = "raven"; +constexpr const char *MuninnDefaultMediaTargetHost = "192.168.1.66"; +constexpr const char *MuninnDeprecatedWireGuardMediaTargetHost = "10.77.0.2"; +constexpr const char *MuninnOdinCultMeshUri = "cultmesh://odin/rendezvous/provider-catalog"; +constexpr const char *MuninnOdinMediaTargetUri = "cultmesh://odin/media/muninn-raven-av"; +constexpr const char *MuninnOdinRudpEndpoint = "127.0.0.1:17871"; +constexpr uint16_t MuninnDefaultMediaPort = 5204; +constexpr const char *MuninnObsCatalogType = "muninn.obs_stream_catalog"; +constexpr const char *MuninnObsCatalogSchema = "muninn.obs_stream_catalog.v1"; +constexpr const char *MuninnObsCatalogKey = "obs"; +constexpr uint64_t MuninnCatalogRefreshMs = 2000; +constexpr uint64_t MuninnActivationRetryMs = 5000; +constexpr uint64_t MuninnMediaStartupGraceMs = 3000; +constexpr uint64_t MuninnMediaStaleMs = 1500; +constexpr uint64_t MuninnMediaRestartCooldownMs = 5000; +constexpr uint64_t MuninnReceiverKeyframeRequestCooldownMs = 500; +constexpr uint64_t MuninnReceiverChunkRepairFirstWaitMs = 64; +constexpr uint64_t MuninnVideoAssemblyExpiryScanMs = 10; +constexpr size_t MuninnExpiredVideoFrameRememberCount = 4096; +constexpr size_t MuninnVideoParityCacheMaxEntries = 4096; +constexpr uint16_t MuninnCatalogDiscoveryPort = 17874; +constexpr uint32_t MuninnCatalogDiscoveryConnectionId = 0x6d750003; +constexpr uint32_t MuninnCommandRudpConnectionId = 0x6d750002; +constexpr uint32_t MuninnDefaultMediaPacketBytes = 848; +constexpr uint32_t MuninnDefaultRudpVideoBitrateKbps = 12000; +constexpr uint32_t MuninnDefaultRudpLatencyBudgetMs = 2000; +constexpr uint32_t MuninnDefaultRudpAudioReorderMs = muninn_rudp_url::DefaultAudioReorderMs; +constexpr uint32_t MuninnMaxRudpVideoBitrateKbps = 100000; +constexpr uint32_t MuninnMaxRudpLatencyBudgetMs = 2000; +constexpr uint32_t MuninnMaxRudpAudioReorderMs = 2000; +constexpr size_t MuninnAudioReorderMaxPackets = 128; +constexpr size_t MuninnAudioQueueMaxPackets = 32; +constexpr const char *MuninnDisabledVideoSourceId = "video:none"; +constexpr const char *MuninnDisabledAudioSourceId = "audio:none"; + +enum class MuninnSourceKind { + Combined, + VideoOnly, + AudioOnly, +}; + +struct MuninnStreamOption { + std::string stream_id; + std::string label; + std::string url; + std::string state; + std::string command_rudp_target; + std::string media_target_host; + uint16_t media_port = MuninnDefaultMediaPort; + uint32_t media_packet_bytes = MuninnDefaultMediaPacketBytes; + uint32_t rudp_video_bitrate_kbps = MuninnDefaultRudpVideoBitrateKbps; + uint32_t rudp_latency_budget_ms = MuninnDefaultRudpLatencyBudgetMs; + std::vector> video_sources; + std::vector> audio_sources; +}; + +struct MuninnQueuedAudio { + std::vector planes[MAX_AV_PLANES]; + uint32_t frames = 0; + uint64_t timestamp = 0; + enum speaker_layout speakers = SPEAKERS_STEREO; + uint32_t samples_per_sec = 48000; +}; + +struct MuninnSource { + MuninnSourceKind kind = MuninnSourceKind::Combined; + obs_source_t *source = nullptr; + obs_source_t *media = nullptr; + obs_source_t *audio_media = nullptr; + std::unique_ptr audio_decoder; + std::unique_ptr bridge; + std::mutex audio_mutex; + std::condition_variable audio_cv; + std::deque audio_queue; + std::thread audio_worker; + bool audio_worker_running = false; + std::string store_path = MuninnStorePath; + std::string command_exe_path = MuninnCommandExePath; + std::string command_rudp_target = MuninnCommandRudpTarget; + std::string host_id = MuninnDefaultHostId; + std::string media_target_host = MuninnDefaultMediaTargetHost; + uint16_t media_port = MuninnDefaultMediaPort; + uint32_t media_packet_bytes = MuninnDefaultMediaPacketBytes; + uint32_t rudp_video_bitrate_kbps = MuninnDefaultRudpVideoBitrateKbps; + uint32_t rudp_latency_budget_ms = MuninnDefaultRudpLatencyBudgetMs; + std::vector> video_sources; + std::vector> audio_sources; + uint32_t rudp_audio_reorder_ms = MuninnDefaultRudpAudioReorderMs; + std::string stream_id; + std::string video_source_id; + std::string audio_source_id; + std::string stream_url; + std::string last_requested_stream_id; + std::string last_requested_video_source_id; + std::string last_requested_audio_source_id; + std::vector catalog_options; + std::unique_ptr catalog_discovery; + std::chrono::steady_clock::time_point next_catalog_refresh = std::chrono::steady_clock::now(); + std::chrono::steady_clock::time_point last_activation_request = std::chrono::steady_clock::time_point::min(); + std::chrono::steady_clock::time_point last_media_restart = std::chrono::steady_clock::time_point::min(); +}; + +static void muninn_child_audio(void *param, obs_source_t *source, const struct audio_data *audio_data, bool muted); +static void stop_audio_worker(MuninnSource *ctx); +static std::string canonical_muninn_stream_id(const std::string &stream_id); + +static bool source_wants_video(const MuninnSource *ctx) +{ + return !ctx || ctx->kind != MuninnSourceKind::AudioOnly; +} + +static bool source_wants_audio(const MuninnSource *ctx) +{ + return !ctx || ctx->kind != MuninnSourceKind::VideoOnly; +} + +struct RudpBridgeOutputs { + std::string video_url; + std::string audio_url; +}; + +struct MuninnMediaPayload { + enum class Kind { Unknown, Video, VideoParity, Audio }; + Kind kind = Kind::Unknown; + std::string stream_id; + std::string session_id; + uint64_t frame_id = 0; + uint64_t audio_packet_id = 0; + uint64_t audio_pts_ticks = 0; + uint64_t audio_duration_ticks = 0; + uint32_t audio_timebase_num = 0; + uint32_t audio_timebase_den = 0; + uint64_t audio_deadline_ticks = 0; + bool keyframe = false; + std::optional dependency_frame_id; + uint16_t chunk_index = 0; + uint16_t chunk_count = 1; + uint16_t parity_index = 0; + uint16_t parity_count = 0; + std::vector chunk_payload_lengths; + std::vector payload; +}; + +struct MuninnVideoFrameAssembly { + std::string stream_id; + std::string session_id; + uint64_t frame_id = 0; + bool keyframe = false; + std::optional dependency_frame_id; + uint16_t chunk_count = 0; + uint16_t chunks_received = 0; + std::chrono::steady_clock::time_point started_at; + bool repair_requested = false; + std::vector chunk_payload_lengths; + uint16_t parity_count = 0; + std::map> parity_payloads; + std::vector> chunks; +}; + +struct MuninnVideoParityCacheEntry { + std::string stream_id; + std::string session_id; + uint64_t frame_id = 0; + bool keyframe = false; + std::optional dependency_frame_id; + uint16_t chunk_count = 0; + uint16_t parity_index = 0; + uint16_t parity_count = 0; + std::chrono::steady_clock::time_point received_at; + std::vector chunk_payload_lengths; + std::vector payload; +}; + +struct MuninnPendingAudioPacket { + std::vector payload; + std::chrono::steady_clock::time_point received_at; +}; + +static bool starts_with(const std::string &value, const std::string &prefix) +{ + return value.rfind(prefix, 0) == 0; +} + +static uint64_t steady_millis() +{ + return static_cast( + std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count()); +} + +static uint32_t read_be32(const uint8_t *data) +{ + return (static_cast(data[0]) << 24) | (static_cast(data[1]) << 16) | + (static_cast(data[2]) << 8) | static_cast(data[3]); +} + +static uint16_t read_be16(const uint8_t *data) +{ + return static_cast((static_cast(data[0]) << 8) | data[1]); +} + +static void write_be32(std::vector &data, size_t offset, uint32_t value) +{ + data[offset] = static_cast((value >> 24) & 0xff); + data[offset + 1] = static_cast((value >> 16) & 0xff); + data[offset + 2] = static_cast((value >> 8) & 0xff); + data[offset + 3] = static_cast(value & 0xff); +} + +static void write_be16(std::vector &data, size_t offset, uint16_t value) +{ + data[offset] = static_cast((value >> 8) & 0xff); + data[offset + 1] = static_cast(value & 0xff); +} + +class MediaMsgpackReader { +public: + MediaMsgpackReader(const uint8_t *data, size_t size) : data_(data), size_(size) {} + explicit MediaMsgpackReader(const std::vector &bytes) : data_(bytes.data()), size_(bytes.size()) {} + + uint32_t read_map_len() + { + const uint8_t tag = read_u8(); + if ((tag & 0xf0) == 0x80) { + return tag & 0x0f; + } + if (tag == 0xde) { + return read_u16(); + } + if (tag == 0xdf) { + return read_u32(); + } + throw std::runtime_error("expected MessagePack map"); + } + + uint32_t read_array_len() + { + const uint8_t tag = read_u8(); + if ((tag & 0xf0) == 0x90) { + return tag & 0x0f; + } + if (tag == 0xdc) { + return read_u16(); + } + if (tag == 0xdd) { + return read_u32(); + } + throw std::runtime_error("expected MessagePack array"); + } + + std::string read_string() + { + const uint8_t tag = read_u8(); + uint32_t length = 0; + if ((tag & 0xe0) == 0xa0) { + length = tag & 0x1f; + } else if (tag == 0xd9) { + length = read_u8(); + } else if (tag == 0xda) { + length = read_u16(); + } else if (tag == 0xdb) { + length = read_u32(); + } else { + throw std::runtime_error("expected MessagePack string"); + } + require(length); + std::string value(reinterpret_cast(data_ + pos_), length); + pos_ += length; + return value; + } + + std::vector read_bytes() + { + const uint8_t tag = read_u8(); + uint32_t length = 0; + if (tag == 0xc4) { + length = read_u8(); + } else if (tag == 0xc5) { + length = read_u16(); + } else if (tag == 0xc6) { + length = read_u32(); + } else if ((tag & 0xf0) == 0x90 || tag == 0xdc || tag == 0xdd) { + --pos_; + length = read_array_len(); + std::vector bytes; + bytes.reserve(length); + for (uint32_t index = 0; index < length; ++index) { + bytes.push_back(static_cast(read_unsigned())); + } + return bytes; + } else { + throw std::runtime_error("expected MessagePack bytes"); + } + require(length); + std::vector bytes(data_ + pos_, data_ + pos_ + length); + pos_ += length; + return bytes; + } + + uint64_t read_unsigned() + { + const uint8_t tag = read_u8(); + if (tag <= 0x7f) { + return tag; + } + if (tag == 0xcc) { + return read_u8(); + } + if (tag == 0xcd) { + return read_u16(); + } + if (tag == 0xce) { + return read_u32(); + } + if (tag == 0xcf) { + return read_u64(); + } + throw std::runtime_error("expected MessagePack unsigned integer"); + } + + std::optional read_optional_unsigned() + { + const uint8_t tag = read_u8(); + if (tag == 0xc0) { + return std::nullopt; + } + if (tag <= 0x7f) { + return tag; + } + if (tag == 0xcc) { + return read_u8(); + } + if (tag == 0xcd) { + return read_u16(); + } + if (tag == 0xce) { + return read_u32(); + } + if (tag == 0xcf) { + return read_u64(); + } + throw std::runtime_error("expected optional MessagePack unsigned integer"); + } + + bool read_bool() + { + const uint8_t tag = read_u8(); + if (tag == 0xc2) { + return false; + } + if (tag == 0xc3) { + return true; + } + throw std::runtime_error("expected MessagePack boolean"); + } + + void skip() + { + const uint8_t tag = read_u8(); + if (tag <= 0x7f || tag >= 0xe0 || tag == 0xc0 || tag == 0xc2 || tag == 0xc3) { + return; + } + if ((tag & 0xe0) == 0xa0) { + skip_bytes(tag & 0x1f); + return; + } + if ((tag & 0xf0) == 0x90) { + skip_items(tag & 0x0f); + return; + } + if ((tag & 0xf0) == 0x80) { + skip_items((tag & 0x0f) * 2); + return; + } + switch (tag) { + case 0xc4: + case 0xd9: + skip_bytes(read_u8()); + return; + case 0xc5: + case 0xda: + skip_bytes(read_u16()); + return; + case 0xc6: + case 0xdb: + skip_bytes(read_u32()); + return; + case 0xcc: + case 0xd0: + skip_bytes(1); + return; + case 0xcd: + case 0xd1: + skip_bytes(2); + return; + case 0xce: + case 0xd2: + case 0xca: + skip_bytes(4); + return; + case 0xcf: + case 0xd3: + case 0xcb: + skip_bytes(8); + return; + case 0xdc: + skip_items(read_u16()); + return; + case 0xdd: + skip_items(read_u32()); + return; + case 0xde: + skip_items(read_u16() * 2); + return; + case 0xdf: + skip_items(read_u32() * 2); + return; + default: + throw std::runtime_error("unsupported MessagePack item"); + } + } + +private: + const uint8_t *data_; + size_t size_; + size_t pos_ = 0; + + void require(size_t count) const + { + if (count > size_ - pos_) { + throw std::runtime_error("truncated MessagePack input"); + } + } + + uint8_t read_u8() + { + require(1); + return data_[pos_++]; + } + + uint16_t read_u16() + { + require(2); + const uint16_t value = (static_cast(data_[pos_]) << 8) | static_cast(data_[pos_ + 1]); + pos_ += 2; + return value; + } + + uint32_t read_u32() + { + require(4); + const uint32_t value = (static_cast(data_[pos_]) << 24) | (static_cast(data_[pos_ + 1]) << 16) | + (static_cast(data_[pos_ + 2]) << 8) | static_cast(data_[pos_ + 3]); + pos_ += 4; + return value; + } + + uint64_t read_u64() + { + const uint64_t high = read_u32(); + const uint64_t low = read_u32(); + return (high << 32) | low; + } + + void skip_bytes(size_t count) + { + require(count); + pos_ += count; + } + + void skip_items(uint32_t count) + { + for (uint32_t index = 0; index < count; ++index) { + skip(); + } + } +}; + +static std::optional decode_muninn_media_payload(const uint8_t *payload, uint32_t payload_len) +{ + try { + MediaMsgpackReader wire(payload, payload_len); + const uint32_t message_len = wire.read_map_len(); + std::string schema_version; + std::string schema_id; + std::vector document_payload; + + for (uint32_t index = 0; index < message_len; ++index) { + const auto key = wire.read_string(); + if (key == "schemaVersion") { + schema_version = wire.read_string(); + } else if (key == "document") { + const uint32_t document_len = wire.read_map_len(); + for (uint32_t field = 0; field < document_len; ++field) { + const auto document_key = wire.read_string(); + if (document_key == "schemaId") { + schema_id = wire.read_string(); + } else if (document_key == "payload") { + document_payload = wire.read_bytes(); + } else { + wire.skip(); + } + } + } else { + wire.skip(); + } + } + + if (schema_version != "cultnet.document_put_raw.v0" || document_payload.empty()) { + return std::nullopt; + } + + MediaMsgpackReader record(document_payload); + const uint32_t record_len = record.read_array_len(); + if (schema_id == "muninn.media_video_access_unit.v1") { + if (record_len < 14) { + return std::nullopt; + } + MuninnMediaPayload media; + media.kind = MuninnMediaPayload::Kind::Video; + media.stream_id = record.read_string(); + media.session_id = record.read_string(); + media.frame_id = record.read_unsigned(); + record.skip(); // codec + record.skip(); // pts_ticks + record.skip(); // duration_ticks + record.skip(); // timebase_num + record.skip(); // timebase_den + media.keyframe = record.read_bool(); + media.dependency_frame_id = record.read_optional_unsigned(); + record.skip(); // deadline_ticks + media.chunk_index = static_cast(record.read_unsigned()); + media.chunk_count = static_cast(record.read_unsigned()); + media.payload = record.read_bytes(); + return media; + } + if (schema_id == "muninn.media_video_parity_shard.v1") { + if (record_len < 16) { + return std::nullopt; + } + MuninnMediaPayload media; + media.kind = MuninnMediaPayload::Kind::VideoParity; + media.stream_id = record.read_string(); + media.session_id = record.read_string(); + media.frame_id = record.read_unsigned(); + record.skip(); // codec + record.skip(); // pts_ticks + record.skip(); // duration_ticks + record.skip(); // timebase_num + record.skip(); // timebase_den + media.keyframe = record.read_bool(); + media.dependency_frame_id = record.read_optional_unsigned(); + record.skip(); // deadline_ticks + media.chunk_count = static_cast(record.read_unsigned()); + media.parity_index = static_cast(record.read_unsigned()); + media.parity_count = static_cast(record.read_unsigned()); + const uint32_t lengths = record.read_array_len(); + media.chunk_payload_lengths.reserve(lengths); + for (uint32_t index = 0; index < lengths; ++index) { + media.chunk_payload_lengths.push_back(static_cast(record.read_unsigned())); + } + media.payload = record.read_bytes(); + return media; + } + if (schema_id == "muninn.media_video_parity_shard.v2") { + if (record_len < 17) { + return std::nullopt; + } + MuninnMediaPayload media; + media.kind = MuninnMediaPayload::Kind::VideoParity; + media.stream_id = record.read_string(); + media.session_id = record.read_string(); + media.frame_id = record.read_unsigned(); + record.skip(); // codec + record.skip(); // pts_ticks + record.skip(); // duration_ticks + record.skip(); // timebase_num + record.skip(); // timebase_den + media.keyframe = record.read_bool(); + media.dependency_frame_id = record.read_optional_unsigned(); + record.skip(); // deadline_ticks + media.chunk_count = static_cast(record.read_unsigned()); + media.parity_index = static_cast(record.read_unsigned()); + media.parity_count = static_cast(record.read_unsigned()); + const uint32_t chunk_payload_bytes = static_cast(record.read_unsigned()); + const uint32_t last_chunk_payload_bytes = static_cast(record.read_unsigned()); + if (media.chunk_count == 0 || + chunk_payload_bytes == 0 || + last_chunk_payload_bytes == 0 || + last_chunk_payload_bytes > chunk_payload_bytes) { + return std::nullopt; + } + media.chunk_payload_lengths.assign(media.chunk_count, chunk_payload_bytes); + media.chunk_payload_lengths.back() = last_chunk_payload_bytes; + media.payload = record.read_bytes(); + return media; + } + if (schema_id == "muninn.media_audio_packet.v1") { + if (record_len < 10) { + return std::nullopt; + } + MuninnMediaPayload media; + media.kind = MuninnMediaPayload::Kind::Audio; + media.stream_id = record.read_string(); + media.session_id = record.read_string(); + media.audio_packet_id = record.read_unsigned(); + record.skip(); // codec + media.audio_pts_ticks = record.read_unsigned(); + media.audio_duration_ticks = record.read_unsigned(); + media.audio_timebase_num = static_cast(record.read_unsigned()); + media.audio_timebase_den = static_cast(record.read_unsigned()); + media.audio_deadline_ticks = record.read_unsigned(); + media.payload = record.read_bytes(); + return media; + } + } catch (const std::exception &error) { + blog(LOG_WARNING, "Muninn RUDP bridge could not decode typed media payload: %s", error.what()); + } + + return std::nullopt; +} + +class RudpMediaBridge { +public: + ~RudpMediaBridge() { stop(); } + + RudpBridgeOutputs start(const std::string &url) + { + if (running_ && url == source_url_) { + return outputs_; + } + + stop(); + const auto parsed = muninn_rudp_url::parse(url); + if (!parsed) { + return RudpBridgeOutputs{url, ""}; + } + + source_url_ = url; + parts_ = *parsed; + outputs_.video_url = "udp://127.0.0.1:" + std::to_string(parts_.video_local_port) + + "?fifo_size=8388608&overrun_nonfatal=1&buffer_size=67108864"; + outputs_.audio_url = "udp://127.0.0.1:" + std::to_string(parts_.audio_local_port) + + "?fifo_size=1048576&overrun_nonfatal=1&buffer_size=8388608"; + const uint64_t now = steady_millis(); + started_ms_.store(now); + last_packet_ms_.store(0); + last_video_forwarded_ms_.store(0); + last_audio_forwarded_ms_.store(0); + stale_reported_.store(false); + media_connected_ = false; + running_ = true; + worker_ = std::thread([this] { run(); }); + blog(LOG_INFO, + "Muninn RUDP bridge starting udp/%u -> video udp/127.0.0.1:%u audio udp/127.0.0.1:%u sender_resend_delay_ms=%u reliable_expire_after_ms=%u assembly_deadline_ms=%u gap_wait_ms=%u audio_reorder_ms=%u", + parts_.port, parts_.video_local_port, parts_.audio_local_port, + parts_.sender_resend_delay_ms, parts_.reliable_expire_after_ms, + parts_.video_assembly_deadline_ms, parts_.gap_wait_ms, parts_.audio_reorder_ms); + return outputs_; + } + + void stop() + { + running_ = false; +#ifdef _WIN32 + if (socket_ != INVALID_SOCKET) { + closesocket(socket_); + socket_ = INVALID_SOCKET; + } +#endif + if (worker_.joinable()) { + worker_.join(); + } + source_url_.clear(); + outputs_ = {}; + started_ms_.store(0); + last_packet_ms_.store(0); + last_video_forwarded_ms_.store(0); + last_audio_forwarded_ms_.store(0); + stale_reported_.store(false); + media_connected_ = false; + } + + bool has_forwarded_media() const + { + return last_video_forwarded_ms_.load() != 0 || last_audio_forwarded_ms_.load() != 0; + } + + bool is_running() const + { + return running_.load(); + } + + bool consume_decoder_reset_request() + { + return decoder_reset_requested_.exchange(false); + } + + bool needs_activation(uint64_t now_ms) const + { + const uint64_t started = started_ms_.load(); + if (started == 0) { + return true; + } + if (now_ms >= started && now_ms - started < MuninnMediaStartupGraceMs) { + return false; + } + if (!running_.load()) { + return true; + } + const uint64_t last_packet = last_packet_ms_.load(); + if (last_packet == 0) { + return true; + } + const uint64_t last_video = last_video_forwarded_ms_.load(); + const uint64_t last_audio = last_audio_forwarded_ms_.load(); + const uint64_t last_media = std::max(last_video, last_audio); + return last_media == 0 || (now_ms > last_media && now_ms - last_media >= MuninnMediaStaleMs); + } + + bool consume_stale_transition(uint64_t now_ms) + { + if (!needs_activation(now_ms)) { + stale_reported_.store(false); + return false; + } + if (!has_forwarded_media()) { + stale_reported_.store(false); + return false; + } + bool expected = false; + return stale_reported_.compare_exchange_strong(expected, true); + } + +private: + void run() + { +#ifdef _WIN32 + WSADATA data; + WSAStartup(MAKEWORD(2, 2), &data); + + socket_ = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (socket_ == INVALID_SOCKET) { + blog(LOG_WARNING, "Muninn RUDP bridge could not create socket"); + running_ = false; + WSACleanup(); + return; + } + const int socket_buffer_bytes = 4 * 1024 * 1024; + const BOOL reuse_addr = TRUE; + setsockopt(socket_, SOL_SOCKET, SO_REUSEADDR, reinterpret_cast(&reuse_addr), + sizeof(reuse_addr)); + setsockopt(socket_, SOL_SOCKET, SO_RCVBUF, reinterpret_cast(&socket_buffer_bytes), + sizeof(socket_buffer_bytes)); + const DWORD receive_timeout_ms = 2; + setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast(&receive_timeout_ms), + sizeof(receive_timeout_ms)); + + sockaddr_in bind_addr = {}; + bind_addr.sin_family = AF_INET; + bind_addr.sin_addr.s_addr = htonl(INADDR_ANY); + bind_addr.sin_port = htons(parts_.port); + if (bind(socket_, reinterpret_cast(&bind_addr), sizeof(bind_addr)) != 0) { + blog(LOG_WARNING, "Muninn RUDP bridge could not bind udp/%u", parts_.port); + closesocket(socket_); + socket_ = INVALID_SOCKET; + running_ = false; + WSACleanup(); + return; + } + + const SOCKET video_socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + const SOCKET audio_socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + setsockopt(video_socket, SOL_SOCKET, SO_SNDBUF, reinterpret_cast(&socket_buffer_bytes), + sizeof(socket_buffer_bytes)); + setsockopt(audio_socket, SOL_SOCKET, SO_SNDBUF, reinterpret_cast(&socket_buffer_bytes), + sizeof(socket_buffer_bytes)); + sockaddr_in video_addr = {}; + video_addr.sin_family = AF_INET; + inet_pton(AF_INET, "127.0.0.1", &video_addr.sin_addr); + video_addr.sin_port = htons(parts_.video_local_port); + sockaddr_in audio_addr = {}; + audio_addr.sin_family = AF_INET; + inet_pton(AF_INET, "127.0.0.1", &audio_addr.sin_addr); + audio_addr.sin_port = htons(parts_.audio_local_port); + + std::vector buffer(65535); + uint32_t bridge_sequence = 1; + while (running_) { + sockaddr_in remote = {}; + int remote_len = sizeof(remote); + const int received = recvfrom(socket_, reinterpret_cast(buffer.data()), static_cast(buffer.size()), 0, + reinterpret_cast(&remote), &remote_len); + if (received <= 0) { + expire_rudp_packet_fragments(std::chrono::steady_clock::now()); + expire_video_assemblies(std::chrono::steady_clock::now(), bridge_sequence); + continue; + } + handle_packet(buffer.data(), static_cast(received), remote, bridge_sequence, video_socket, video_addr, audio_socket, audio_addr); + } + + if (video_socket != INVALID_SOCKET) { + closesocket(video_socket); + } + if (audio_socket != INVALID_SOCKET) { + closesocket(audio_socket); + } + if (socket_ != INVALID_SOCKET) { + closesocket(socket_); + socket_ = INVALID_SOCKET; + } + WSACleanup(); +#endif + } + +#ifdef _WIN32 + void handle_packet(const uint8_t *packet, + size_t size, + const sockaddr_in &remote, + uint32_t &bridge_sequence, + SOCKET video_socket, + const sockaddr_in &video_addr, + SOCKET audio_socket, + const sockaddr_in &audio_addr) + { + if (size < 36 || packet[0] != 'C' || packet[1] != 'N' || packet[2] != 'R' || packet[3] != '0') { + return; + } + const uint8_t version = packet[4]; + const uint8_t packet_type = packet[5]; + const uint8_t header_bytes = packet[7]; + const uint32_t connection_id = read_be32(packet + 8); + const uint32_t sequence = read_be32(packet + 12); + const uint16_t fragment_count = read_be16(packet + 28); + const uint32_t payload_len = read_be32(packet + 30); + const uint8_t channel_len = packet[34]; + const bool is_video_connection = connection_id == parts_.connection_id; + const bool is_audio_connection = connection_id == parts_.audio_connection_id; + if (version != 0 || (!is_video_connection && !is_audio_connection) || header_bytes < 36 || size < header_bytes) { + return; + } + last_packet_ms_.store(steady_millis()); + if (is_video_connection) { + last_remote_ = remote; + } else { + last_audio_remote_ = remote; + } + auto &connection_ack_tracker = is_audio_connection ? audio_ack_tracker_ : ack_tracker_; + auto &connection_fragments = is_audio_connection ? audio_packet_fragments_ : packet_fragments_; + const size_t payload_offset = static_cast(header_bytes); + if (payload_offset + payload_len > size || 36 + channel_len > header_bytes) { + return; + } + + if (packet_type == 1) { + const bool was_connected = media_connected_; + video_assemblies_.clear(); + video_parity_cache_.clear(); + reset_audio_reorder(); + packet_fragments_.reset(); + audio_packet_fragments_.reset(); + connection_ack_tracker.reset(); + connection_ack_tracker.remember(sequence); + packets_forwarded_ = 0; + bytes_forwarded_ = 0; + payloads_decoded_dropped_ = 0; + video_assemblies_expired_ = 0; + video_recovery_dropped_ = 0; + video_parity_received_ = 0; + video_parity_cached_ = 0; + video_parity_ignored_ = 0; + video_parity_recovered_ = 0; + video_parity_unusable_ = 0; + waiting_for_video_keyframe_ = true; + video_forward_failures_ = 0; + audio_forward_failures_ = 0; + audio_packets_reordered_ = 0; + audio_packets_skipped_ = 0; + audio_packets_stale_ = 0; + media_sequence_initialized_ = false; + last_media_sequence_ = 0; + media_sequence_gap_events_ = 0; + media_sequence_gap_packets_ = 0; + last_sequence_gap_log_ms_ = 0; + last_keyframe_request_ms_ = 0; + last_video_expiry_scan_ms_ = 0; + media_connected_ = true; + if (was_connected) { + decoder_reset_requested_.store(true); + } + const auto [ack, ack_mask] = connection_ack_tracker.state(); + send_control_packet(2, 0x03, connection_id, bridge_sequence++, ack, ack_mask, remote); + blog(LOG_INFO, + "Muninn RUDP bridge accepted %s media connection sequence %u", + is_audio_connection ? "audio" : "video", + sequence); + return; + } + + if (packet_type == 3) { + if (!media_connected_) { + video_assemblies_.clear(); + video_parity_cache_.clear(); + reset_audio_reorder(); + packet_fragments_.reset(); + audio_packet_fragments_.reset(); + ack_tracker_.reset(); + audio_ack_tracker_.reset(); + packets_forwarded_ = 0; + bytes_forwarded_ = 0; + payloads_decoded_dropped_ = 0; + video_assemblies_expired_ = 0; + video_recovery_dropped_ = 0; + video_parity_received_ = 0; + video_parity_cached_ = 0; + video_parity_ignored_ = 0; + video_parity_recovered_ = 0; + video_parity_unusable_ = 0; + waiting_for_video_keyframe_ = true; + video_forward_failures_ = 0; + audio_forward_failures_ = 0; + audio_packets_reordered_ = 0; + audio_packets_skipped_ = 0; + audio_packets_stale_ = 0; + media_sequence_initialized_ = false; + last_media_sequence_ = 0; + media_sequence_gap_events_ = 0; + media_sequence_gap_packets_ = 0; + last_sequence_gap_log_ms_ = 0; + last_keyframe_request_ms_ = 0; + last_video_expiry_scan_ms_ = 0; + media_connected_ = true; + decoder_reset_requested_.store(true); + blog(LOG_INFO, "Muninn RUDP bridge adopted existing media sender at sequence %u", sequence); + } + connection_ack_tracker.remember(sequence); + if (is_video_connection) { + observe_media_sequence(sequence); + } + const std::string channel_id( + reinterpret_cast(packet + 36), + reinterpret_cast(packet + 36 + channel_len)); + if (channel_id != "media") { + const auto [ack, ack_mask] = connection_ack_tracker.state(); + send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote); + return; + } + std::optional>> delivered; + if (fragment_count == 0) { + delivered = std::make_pair( + sequence, + std::vector(packet + payload_offset, packet + payload_offset + payload_len)); + } else { + delivered = assemble_rudp_packet_fragment( + connection_fragments, + channel_id, + read_be16(packet + 24), + read_be16(packet + 26), + fragment_count, + sequence, + packet + payload_offset, + payload_len); + } + if (delivered) { + forward_payload(delivered->second.data(), + static_cast(delivered->second.size()), + video_socket, + video_addr, + audio_socket, + audio_addr, + bridge_sequence); + } + const auto [ack, ack_mask] = connection_ack_tracker.state(); + send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote); + } + } + + std::optional>> assemble_rudp_packet_fragment( + muninn_rudp_fragments::FragmentAssembler &packet_fragments, + const std::string &channel_id, + uint16_t fragment_id, + uint16_t fragment_index, + uint16_t fragment_count, + uint32_t sequence, + const uint8_t *payload, + uint32_t payload_len) + { + expire_rudp_packet_fragments(std::chrono::steady_clock::now()); + if (fragment_id == 0 || fragment_index >= fragment_count) { + ++payloads_decoded_dropped_; + log_bridge_pressure("dropped invalid RUDP packet fragment metadata"); + return std::nullopt; + } + + const auto result = packet_fragments.insert( + channel_id, + fragment_id, + fragment_index, + fragment_count, + sequence, + payload, + payload_len, + std::chrono::steady_clock::now()); + if (!result) { + return std::nullopt; + } + return std::make_pair(result->first_sequence, result->payload); + } + + void forward_payload(const uint8_t *payload, + uint32_t payload_len, + SOCKET video_socket, + const sockaddr_in &video_addr, + SOCKET audio_socket, + const sockaddr_in &audio_addr, + uint32_t &bridge_sequence) + { + const auto media = decode_muninn_media_payload(payload, payload_len); + if (!media || media->payload.empty()) { + ++payloads_decoded_dropped_; + log_bridge_pressure("dropped undecodable media payload"); + return; + } + size_t forwarded_bytes = 0; + if (media->kind == MuninnMediaPayload::Kind::Video || media->kind == MuninnMediaPayload::Kind::VideoParity) { + const auto assembled = assemble_video_payload(*media, bridge_sequence); + if (!assembled) { + return; + } + if (waiting_for_video_keyframe_ && !h264_access_unit_has_idr(*assembled)) { + ++video_recovery_dropped_; + log_bridge_pressure("dropped video payload while waiting for decoder recovery keyframe"); + return; + } + if (h264_access_unit_has_idr(*assembled)) { + waiting_for_video_keyframe_ = false; + } + forwarded_bytes = assembled->size(); + if (!send_udp_byte_stream(video_socket, video_addr, assembled->data(), assembled->size())) { + ++video_forward_failures_; + log_bridge_pressure("failed forwarding video payload to local decoder socket"); + return; + } + last_video_forwarded_ms_.store(steady_millis()); + } else if (media->kind == MuninnMediaPayload::Kind::Audio) { + const auto audio_forwarded_bytes = forward_audio_payload(*media, audio_socket, audio_addr); + if (!audio_forwarded_bytes) { + return; + } + forwarded_bytes = *audio_forwarded_bytes; + } else { + ++payloads_decoded_dropped_; + log_bridge_pressure("dropped unknown media payload"); + return; + } + ++packets_forwarded_; + bytes_forwarded_ += forwarded_bytes; + if (packets_forwarded_ == 1 || packets_forwarded_ % 900 == 0) { + blog(LOG_INFO, + "Muninn RUDP bridge forwarded=%llu bytes=%llu decode_dropped=%llu assemblies_expired=%llu parity_received=%llu parity_cached=%llu parity_recovered=%llu parity_ignored=%llu parity_unusable=%llu video_failures=%llu audio_failures=%llu audio_reordered=%llu audio_skipped=%llu audio_stale=%llu audio_pending=%zu", + static_cast(packets_forwarded_), + static_cast(bytes_forwarded_), + static_cast(payloads_decoded_dropped_), + static_cast(video_assemblies_expired_), + static_cast(video_parity_received_), + static_cast(video_parity_cached_), + static_cast(video_parity_recovered_), + static_cast(video_parity_ignored_), + static_cast(video_parity_unusable_), + static_cast(video_forward_failures_), + static_cast(audio_forward_failures_), + static_cast(audio_packets_reordered_), + static_cast(audio_packets_skipped_), + static_cast(audio_packets_stale_), + pending_audio_packets_.size()); + } + } + + std::optional forward_audio_payload(const MuninnMediaPayload &media, + SOCKET audio_socket, + const sockaddr_in &audio_addr) + { + if (media.stream_id.empty() || media.session_id.empty() || media.payload.empty()) { + ++payloads_decoded_dropped_; + log_bridge_pressure("dropped invalid audio packet metadata"); + return std::nullopt; + } + + const auto now = std::chrono::steady_clock::now(); + if (audio_stream_id_ != media.stream_id || audio_session_id_ != media.session_id) { + reset_audio_reorder(); + audio_stream_id_ = media.stream_id; + audio_session_id_ = media.session_id; + } + + if (!next_audio_packet_ready_) { + next_audio_packet_ready_ = true; + next_audio_packet_id_ = media.audio_packet_id; + } + + if (media.audio_packet_id < next_audio_packet_id_) { + ++audio_packets_stale_; + return std::nullopt; + } + + auto inserted = pending_audio_packets_.emplace( + media.audio_packet_id, + MuninnPendingAudioPacket{media.payload, now}); + if (!inserted.second) { + return std::nullopt; + } + if (media.audio_packet_id != next_audio_packet_id_) { + ++audio_packets_reordered_; + } + + return flush_audio_packets(now, audio_socket, audio_addr); + } + + std::optional flush_audio_packets(std::chrono::steady_clock::time_point now, + SOCKET audio_socket, + const sockaddr_in &audio_addr) + { + size_t forwarded_bytes = 0; + bool forwarded_any = false; + while (true) { + auto ready = pending_audio_packets_.find(next_audio_packet_id_); + if (ready == pending_audio_packets_.end()) { + break; + } + if (!send_audio_packet(audio_socket, audio_addr, ready->second.payload)) { + return std::nullopt; + } + forwarded_bytes += ready->second.payload.size(); + forwarded_any = true; + pending_audio_packets_.erase(ready); + ++next_audio_packet_id_; + audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); + } + + if (pending_audio_packets_.empty()) { + audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); + return forwarded_any ? std::optional(forwarded_bytes) : std::nullopt; + } + + if (audio_gap_started_at_ == std::chrono::steady_clock::time_point::min()) { + audio_gap_started_at_ = now; + } + + const bool gap_expired = + now - audio_gap_started_at_ >= std::chrono::milliseconds(parts_.audio_reorder_ms); + const bool queue_overflow = pending_audio_packets_.size() > MuninnAudioReorderMaxPackets; + if (gap_expired || queue_overflow) { + const uint64_t first_available = pending_audio_packets_.begin()->first; + if (first_available > next_audio_packet_id_) { + audio_packets_skipped_ += first_available - next_audio_packet_id_; + next_audio_packet_id_ = first_available; + } + audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); + const auto after_skip = flush_audio_packets(now, audio_socket, audio_addr); + if (after_skip) { + forwarded_bytes += *after_skip; + forwarded_any = true; + } + } + + return forwarded_any ? std::optional(forwarded_bytes) : std::nullopt; + } + + bool send_audio_packet(SOCKET audio_socket, const sockaddr_in &audio_addr, const std::vector &payload) + { + const int sent = sendto(audio_socket, + reinterpret_cast(payload.data()), + static_cast(payload.size()), + 0, + reinterpret_cast(&audio_addr), + sizeof(audio_addr)); + if (sent != static_cast(payload.size())) { + ++audio_forward_failures_; + log_bridge_pressure("failed forwarding audio payload to local decoder socket"); + return false; + } + last_audio_forwarded_ms_.store(steady_millis()); + return true; + } + + void reset_audio_reorder() + { + audio_stream_id_.clear(); + audio_session_id_.clear(); + pending_audio_packets_.clear(); + next_audio_packet_ready_ = false; + next_audio_packet_id_ = 0; + audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); + } + + void observe_media_sequence(uint32_t sequence) + { + if (!media_sequence_initialized_) { + media_sequence_initialized_ = true; + last_media_sequence_ = sequence; + return; + } + if (sequence <= last_media_sequence_) { + return; + } + if (sequence == last_media_sequence_ + 1) { + last_media_sequence_ = sequence; + return; + } + const uint32_t missing = sequence - last_media_sequence_ - 1; + media_sequence_gap_events_ += 1; + media_sequence_gap_packets_ += missing; + last_media_sequence_ = sequence; + const uint64_t now = steady_millis(); + if (media_sequence_gap_events_ == 1 || + now > last_sequence_gap_log_ms_ + 5'000 || + media_sequence_gap_events_ % 64 == 0) { + last_sequence_gap_log_ms_ = now; + blog(LOG_WARNING, + "Muninn RUDP bridge observed media sequence gap missing=%u gap_events=%llu gap_packets=%llu forwarded=%llu assemblies_expired=%llu", + missing, + static_cast(media_sequence_gap_events_), + static_cast(media_sequence_gap_packets_), + static_cast(packets_forwarded_), + static_cast(video_assemblies_expired_)); + } + } + + bool send_udp_byte_stream(SOCKET socket, + const sockaddr_in &addr, + const uint8_t *data, + size_t size) const + { + constexpr size_t MaxLocalUdpPayload = 16 * 1024; + size_t offset = 0; + while (offset < size) { + const size_t chunk_size = std::min(MaxLocalUdpPayload, size - offset); + const int sent = sendto(socket, + reinterpret_cast(data + offset), + static_cast(chunk_size), + 0, + reinterpret_cast(&addr), + sizeof(addr)); + if (sent != static_cast(chunk_size)) { + return false; + } + offset += chunk_size; + } + return true; + } + + std::optional> assemble_video_payload(const MuninnMediaPayload &media, uint32_t &bridge_sequence) + { + expire_video_assemblies(std::chrono::steady_clock::now(), bridge_sequence); + expire_video_parity_cache(std::chrono::steady_clock::now()); + if (media.kind == MuninnMediaPayload::Kind::VideoParity) { + return assemble_video_parity_payload(media, bridge_sequence); + } + if (media.chunk_count == 0 || media.chunk_index >= media.chunk_count || media.payload.empty()) { + ++payloads_decoded_dropped_; + log_bridge_pressure("dropped invalid video chunk metadata"); + return std::nullopt; + } + if (media.chunk_count == 1) { + return media.payload; + } + + const std::string key = + media.stream_id + ":" + media.session_id + ":" + std::to_string(media.frame_id); + if (expired_video_frame_keys_.find(key) != expired_video_frame_keys_.end()) { + return std::nullopt; + } + auto &assembly = video_assemblies_[key]; + if (assembly.chunks.empty()) { + assembly.stream_id = media.stream_id; + assembly.session_id = media.session_id; + assembly.frame_id = media.frame_id; + assembly.keyframe = media.keyframe; + assembly.dependency_frame_id = media.dependency_frame_id; + assembly.chunk_count = media.chunk_count; + assembly.chunks.resize(media.chunk_count); + assembly.started_at = std::chrono::steady_clock::now(); + assembly.repair_requested = false; + apply_cached_video_parity(key, assembly); + } + if (assembly.chunk_count != media.chunk_count || + assembly.keyframe != media.keyframe || + assembly.dependency_frame_id != media.dependency_frame_id) { + video_assemblies_.erase(key); + ++payloads_decoded_dropped_; + log_bridge_pressure("dropped video frame with mixed metadata"); + return std::nullopt; + } + auto &slot = assembly.chunks[media.chunk_index]; + if (!slot.empty()) { + return std::nullopt; + } + slot = media.payload; + ++assembly.chunks_received; + if (assembly.chunks_received != assembly.chunk_count) { + if (try_recover_video_assembly_from_parity(assembly)) { + ++video_parity_recovered_; + return finish_video_assembly(key, assembly); + } + request_video_chunk_repair_if_due(assembly, std::chrono::steady_clock::now(), bridge_sequence); + return std::nullopt; + } + + return finish_video_assembly(key, assembly); + } + + std::optional> assemble_video_parity_payload(const MuninnMediaPayload &media, uint32_t &bridge_sequence) + { + ++video_parity_received_; + if (media.chunk_count == 0 || + media.parity_count == 0 || + media.parity_index >= media.parity_count || + media.parity_count > media.chunk_count || + media.chunk_payload_lengths.size() != media.chunk_count || + media.payload.empty()) { + ++payloads_decoded_dropped_; + ++video_parity_unusable_; + log_bridge_pressure("dropped invalid video parity metadata"); + return std::nullopt; + } + const std::string key = + media.stream_id + ":" + media.session_id + ":" + std::to_string(media.frame_id); + if (expired_video_frame_keys_.find(key) != expired_video_frame_keys_.end()) { + ++video_parity_ignored_; + return std::nullopt; + } + auto iterator = video_assemblies_.find(key); + if (iterator == video_assemblies_.end()) { + cache_video_parity_payload(key, media); + return std::nullopt; + } + auto &assembly = iterator->second; + if (assembly.chunk_count != media.chunk_count || + assembly.keyframe != media.keyframe || + assembly.dependency_frame_id != media.dependency_frame_id) { + video_assemblies_.erase(key); + ++payloads_decoded_dropped_; + ++video_parity_unusable_; + log_bridge_pressure("dropped video parity with mixed metadata"); + return std::nullopt; + } + assembly.chunk_payload_lengths = media.chunk_payload_lengths; + if (assembly.parity_count != 0 && assembly.parity_count != media.parity_count) { + video_assemblies_.erase(key); + ++payloads_decoded_dropped_; + ++video_parity_unusable_; + log_bridge_pressure("dropped video parity with mixed stripe count"); + return std::nullopt; + } + assembly.parity_count = media.parity_count; + assembly.parity_payloads[media.parity_index] = media.payload; + if (try_recover_video_assembly_from_parity(assembly)) { + ++video_parity_recovered_; + return finish_video_assembly(key, assembly); + } + request_video_chunk_repair_if_due(assembly, std::chrono::steady_clock::now(), bridge_sequence); + return std::nullopt; + } + + void cache_video_parity_payload(const std::string &key, const MuninnMediaPayload &media) + { + MuninnVideoParityCacheEntry entry; + entry.stream_id = media.stream_id; + entry.session_id = media.session_id; + entry.frame_id = media.frame_id; + entry.keyframe = media.keyframe; + entry.dependency_frame_id = media.dependency_frame_id; + entry.chunk_count = media.chunk_count; + entry.parity_index = media.parity_index; + entry.parity_count = media.parity_count; + entry.received_at = std::chrono::steady_clock::now(); + entry.chunk_payload_lengths = media.chunk_payload_lengths; + entry.payload = media.payload; + video_parity_cache_[key].push_back(std::move(entry)); + ++video_parity_cached_; + trim_video_parity_cache(); + } + + void apply_cached_video_parity(const std::string &key, MuninnVideoFrameAssembly &assembly) + { + auto iterator = video_parity_cache_.find(key); + if (iterator == video_parity_cache_.end()) { + return; + } + for (const auto &entry : iterator->second) { + if (entry.chunk_count == assembly.chunk_count && + entry.keyframe == assembly.keyframe && + entry.dependency_frame_id == assembly.dependency_frame_id && + entry.parity_count != 0 && + entry.parity_index < entry.parity_count && + (assembly.parity_count == 0 || assembly.parity_count == entry.parity_count)) { + assembly.chunk_payload_lengths = entry.chunk_payload_lengths; + assembly.parity_count = entry.parity_count; + assembly.parity_payloads[entry.parity_index] = entry.payload; + } else { + ++video_parity_unusable_; + } + } + video_parity_cache_.erase(iterator); + } + + std::optional> finish_video_assembly(const std::string &key, const MuninnVideoFrameAssembly &assembly) + { + std::vector assembled; + size_t byte_count = 0; + for (const auto &chunk : assembly.chunks) { + byte_count += chunk.size(); + } + assembled.reserve(byte_count); + for (const auto &chunk : assembly.chunks) { + assembled.insert(assembled.end(), chunk.begin(), chunk.end()); + } + video_assemblies_.erase(key); + video_parity_cache_.erase(key); + return assembled; + } + + bool try_recover_video_assembly_from_parity(MuninnVideoFrameAssembly &assembly) + { + if (assembly.parity_count == 0 || + assembly.parity_payloads.empty() || + assembly.chunk_payload_lengths.size() != assembly.chunk_count || + assembly.chunks.size() != assembly.chunk_count) { + return false; + } + bool recovered_any = true; + while (recovered_any && assembly.chunks_received < assembly.chunk_count) { + recovered_any = false; + for (const auto &parity : assembly.parity_payloads) { + const uint16_t parity_index = parity.first; + if (parity_index >= assembly.parity_count || parity.second.empty()) { + continue; + } + uint16_t missing_index = 0; + uint16_t missing_count = 0; + for (uint16_t index = parity_index; index < assembly.chunk_count; index += assembly.parity_count) { + if (assembly.chunks[index].empty()) { + missing_index = index; + ++missing_count; + } + } + if (missing_count != 1) { + continue; + } + const uint32_t missing_len = assembly.chunk_payload_lengths[missing_index]; + if (missing_len == 0 || missing_len > parity.second.size()) { + continue; + } + std::vector recovered = parity.second; + bool stripe_complete = true; + for (uint16_t index = parity_index; index < assembly.chunk_count; index += assembly.parity_count) { + if (index == missing_index) { + continue; + } + const auto &chunk = assembly.chunks[index]; + if (chunk.empty()) { + stripe_complete = false; + break; + } + for (size_t offset = 0; offset < chunk.size() && offset < recovered.size(); ++offset) { + recovered[offset] ^= chunk[offset]; + } + } + if (!stripe_complete) { + continue; + } + recovered.resize(missing_len); + assembly.chunks[missing_index] = std::move(recovered); + ++assembly.chunks_received; + recovered_any = true; + } + } + return assembly.chunks_received == assembly.chunk_count; + } + + void expire_video_parity_cache(std::chrono::steady_clock::time_point now) + { + for (auto iterator = video_parity_cache_.begin(); iterator != video_parity_cache_.end();) { + const auto newest = std::max_element( + iterator->second.begin(), + iterator->second.end(), + [](const MuninnVideoParityCacheEntry &left, const MuninnVideoParityCacheEntry &right) { + return left.received_at < right.received_at; + }); + if (newest != iterator->second.end() && + now - newest->received_at < std::chrono::milliseconds(parts_.video_assembly_deadline_ms)) { + ++iterator; + continue; + } + ++video_parity_ignored_; + iterator = video_parity_cache_.erase(iterator); + } + } + + void trim_video_parity_cache() + { + while (video_parity_cache_.size() > MuninnVideoParityCacheMaxEntries) { + auto oldest = std::min_element( + video_parity_cache_.begin(), + video_parity_cache_.end(), + [](const auto &left, const auto &right) { + const auto left_oldest = std::min_element( + left.second.begin(), + left.second.end(), + [](const MuninnVideoParityCacheEntry &a, const MuninnVideoParityCacheEntry &b) { + return a.received_at < b.received_at; + }); + const auto right_oldest = std::min_element( + right.second.begin(), + right.second.end(), + [](const MuninnVideoParityCacheEntry &a, const MuninnVideoParityCacheEntry &b) { + return a.received_at < b.received_at; + }); + if (left_oldest == left.second.end()) { + return true; + } + if (right_oldest == right.second.end()) { + return false; + } + return left_oldest->received_at < right_oldest->received_at; + }); + if (oldest == video_parity_cache_.end()) { + return; + } + ++video_parity_ignored_; + video_parity_cache_.erase(oldest); + } + } + + void expire_video_assemblies(std::chrono::steady_clock::time_point now, uint32_t &bridge_sequence) + { + const uint64_t now_ms = steady_millis(); + if (last_video_expiry_scan_ms_ != 0 && + now_ms > last_video_expiry_scan_ms_ && + now_ms - last_video_expiry_scan_ms_ < MuninnVideoAssemblyExpiryScanMs) { + return; + } + last_video_expiry_scan_ms_ = now_ms; + for (auto iterator = video_assemblies_.begin(); iterator != video_assemblies_.end();) { + if (now - iterator->second.started_at < std::chrono::milliseconds(parts_.video_assembly_deadline_ms)) { + request_video_chunk_repair_if_due(iterator->second, now, bridge_sequence); + ++iterator; + continue; + } + const auto missing_chunk_keys = missing_video_chunk_keys(iterator->second); + const bool needs_decoder_recovery = + iterator->second.keyframe || iterator->second.dependency_frame_id.has_value(); + send_receiver_feedback_if_due( + iterator->second, + missing_chunk_keys, + needs_decoder_recovery, + bridge_sequence); + if (needs_decoder_recovery) { + waiting_for_video_keyframe_ = true; + } + log_video_assembly_expiry(iterator->second, missing_chunk_keys, now); + remember_expired_video_frame(iterator->first); + iterator = video_assemblies_.erase(iterator); + ++video_assemblies_expired_; + log_bridge_pressure("expired incomplete video frame assembly"); + } + } + + void remember_expired_video_frame(const std::string &key) + { + if (expired_video_frame_keys_.insert(key).second) { + expired_video_frame_order_.push_back(key); + } + while (expired_video_frame_order_.size() > MuninnExpiredVideoFrameRememberCount) { + expired_video_frame_keys_.erase(expired_video_frame_order_.front()); + expired_video_frame_order_.pop_front(); + } + } + + void log_video_assembly_expiry(const MuninnVideoFrameAssembly &assembly, + const std::vector &missing_chunk_keys, + std::chrono::steady_clock::time_point now) const + { + const auto age_ms = std::chrono::duration_cast(now - assembly.started_at).count(); + blog(LOG_WARNING, + "Muninn RUDP bridge expired video frame id=%llu keyframe=%s dependency=%s chunks=%u/%u missing=%zu age_ms=%lld repair_requested=%s parity_present=%s", + static_cast(assembly.frame_id), + assembly.keyframe ? "true" : "false", + assembly.dependency_frame_id.has_value() ? "true" : "false", + static_cast(assembly.chunks_received), + static_cast(assembly.chunk_count), + missing_chunk_keys.size(), + static_cast(age_ms), + assembly.repair_requested ? "true" : "false", + assembly.parity_payloads.empty() ? "false" : "true"); + } + + void request_video_chunk_repair_if_due(MuninnVideoFrameAssembly &assembly, + std::chrono::steady_clock::time_point now, + uint32_t &bridge_sequence) + { + if (assembly.chunks_received >= assembly.chunk_count) { + return; + } + if (assembly.repair_requested) { + return; + } + const auto repair_wait_ms = std::max(parts_.gap_wait_ms, MuninnReceiverChunkRepairFirstWaitMs); + if (now - assembly.started_at < std::chrono::milliseconds(repair_wait_ms)) { + return; + } + const auto missing_chunk_keys = missing_video_chunk_keys(assembly); + if (missing_chunk_keys.empty()) { + return; + } + send_receiver_feedback_if_due(assembly, missing_chunk_keys, false, bridge_sequence); + assembly.repair_requested = true; + } + + void expire_rudp_packet_fragments(std::chrono::steady_clock::time_point now) + { + const auto expired = packet_fragments_.expire(now, parts_.reliable_expire_after_ms); + const auto audio_expired = audio_packet_fragments_.expire(now, parts_.reliable_expire_after_ms); + for (uint64_t index = 0; index < expired + audio_expired; ++index) { + log_bridge_pressure("expired incomplete RUDP packet fragment assembly"); + } + } + + std::vector missing_video_chunk_keys(const MuninnVideoFrameAssembly &assembly) const + { + std::vector keys; + for (uint16_t index = 0; index < assembly.chunk_count; ++index) { + if (index >= assembly.chunks.size() || assembly.chunks[index].empty()) { + keys.push_back(std::to_string(assembly.frame_id) + ":" + std::to_string(index)); + } + } + return keys; + } + + void send_receiver_feedback_if_due(const MuninnVideoFrameAssembly &assembly, + const std::vector &missing_chunk_keys, + bool requested_keyframe, + uint32_t &bridge_sequence) + { + if (!last_remote_ || assembly.stream_id.empty() || assembly.session_id.empty()) { + return; + } + if (requested_keyframe) { + const uint64_t now = steady_millis(); + const uint64_t last = last_keyframe_request_ms_.load(); + if (last != 0 && now >= last && now - last < MuninnReceiverKeyframeRequestCooldownMs) { + return; + } + last_keyframe_request_ms_.store(now); + } + const std::string observed_at = "unix:" + std::to_string(steady_millis()); + const auto payload = muninn_media_wire::encode_receiver_feedback_payload( + assembly.stream_id, assembly.session_id, assembly.frame_id, missing_chunk_keys, requested_keyframe, observed_at); + send_media_packet(payload, bridge_sequence++, *last_remote_); + } + + void log_bridge_pressure(const char *reason) + { + const uint64_t pressure = + payloads_decoded_dropped_ + video_assemblies_expired_ + packet_fragments_.expired_count() + + audio_packet_fragments_.expired_count() + + video_recovery_dropped_ + video_parity_unusable_ + video_forward_failures_ + audio_forward_failures_; + if (pressure == 1 || pressure % 300 == 0) { + blog(LOG_WARNING, + "Muninn RUDP bridge %s: forwarded=%llu bytes=%llu decode_dropped=%llu assemblies_expired=%llu packet_fragments_expired=%llu recovery_dropped=%llu parity_received=%llu parity_cached=%llu parity_recovered=%llu parity_ignored=%llu parity_unusable=%llu video_failures=%llu audio_failures=%llu audio_reordered=%llu audio_skipped=%llu audio_stale=%llu audio_pending=%zu", + reason, + static_cast(packets_forwarded_), + static_cast(bytes_forwarded_), + static_cast(payloads_decoded_dropped_), + static_cast(video_assemblies_expired_), + static_cast( + packet_fragments_.expired_count() + audio_packet_fragments_.expired_count()), + static_cast(video_recovery_dropped_), + static_cast(video_parity_received_), + static_cast(video_parity_cached_), + static_cast(video_parity_recovered_), + static_cast(video_parity_ignored_), + static_cast(video_parity_unusable_), + static_cast(video_forward_failures_), + static_cast(audio_forward_failures_), + static_cast(audio_packets_reordered_), + static_cast(audio_packets_skipped_), + static_cast(audio_packets_stale_), + pending_audio_packets_.size()); + } + } + + static bool h264_access_unit_has_idr(const std::vector &access_unit) + { + for (size_t index = 0; index + 4 < access_unit.size();) { + size_t start_code = std::string::npos; + size_t nal_offset = 0; + for (size_t cursor = index; cursor + 3 < access_unit.size(); ++cursor) { + if (access_unit[cursor] == 0x00 && access_unit[cursor + 1] == 0x00) { + if (access_unit[cursor + 2] == 0x01) { + start_code = cursor; + nal_offset = cursor + 3; + break; + } + if (cursor + 4 < access_unit.size() && access_unit[cursor + 2] == 0x00 && access_unit[cursor + 3] == 0x01) { + start_code = cursor; + nal_offset = cursor + 4; + break; + } + } + } + if (start_code == std::string::npos || nal_offset >= access_unit.size()) { + return false; + } + const uint8_t nal_type = access_unit[nal_offset] & 0x1f; + if (nal_type == 5) { + return true; + } + index = nal_offset + 1; + } + return false; + } + + void send_control_packet(uint8_t packet_type, + uint8_t flags, + uint32_t connection_id, + uint32_t sequence, + uint32_t ack, + uint32_t ack_mask, + const sockaddr_in &remote) + { + const std::string channel = "control"; + std::vector response(36 + channel.size()); + response[0] = 'C'; + response[1] = 'N'; + response[2] = 'R'; + response[3] = '0'; + response[4] = 0; + response[5] = packet_type; + response[6] = flags; + response[7] = static_cast(36 + channel.size()); + write_be32(response, 8, connection_id); + write_be32(response, 12, sequence); + write_be32(response, 16, ack); + write_be32(response, 20, ack_mask); + write_be16(response, 24, 0); + write_be16(response, 26, 0); + write_be16(response, 28, 0); + write_be32(response, 30, 0); + response[34] = static_cast(channel.size()); + response[35] = 0; + std::copy(channel.begin(), channel.end(), response.begin() + 36); + sendto(socket_, reinterpret_cast(response.data()), static_cast(response.size()), 0, + reinterpret_cast(&remote), sizeof(remote)); + } + + void send_media_packet(const std::vector &payload, uint32_t sequence, const sockaddr_in &remote) + { + const std::string channel = "media"; + std::vector response(36 + channel.size() + payload.size()); + response[0] = 'C'; + response[1] = 'N'; + response[2] = 'R'; + response[3] = '0'; + response[4] = 0; + response[5] = 3; + response[6] = 0x01; + response[7] = static_cast(36 + channel.size()); + write_be32(response, 8, parts_.connection_id); + write_be32(response, 12, sequence); + const auto [ack, ack_mask] = ack_tracker_.state(); + write_be32(response, 16, ack); + write_be32(response, 20, ack_mask); + write_be16(response, 24, 0); + write_be16(response, 26, 0); + write_be16(response, 28, 0); + write_be32(response, 30, static_cast(payload.size())); + response[34] = static_cast(channel.size()); + response[35] = 0; + std::copy(channel.begin(), channel.end(), response.begin() + 36); + std::copy(payload.begin(), payload.end(), response.begin() + 36 + channel.size()); + sendto(socket_, reinterpret_cast(response.data()), static_cast(response.size()), 0, + reinterpret_cast(&remote), sizeof(remote)); + } + + SOCKET socket_ = INVALID_SOCKET; + std::optional last_remote_; + std::optional last_audio_remote_; +#endif + std::atomic running_{false}; + std::atomic started_ms_{0}; + std::atomic last_packet_ms_{0}; + std::atomic last_video_forwarded_ms_{0}; + std::atomic last_audio_forwarded_ms_{0}; + std::atomic stale_reported_{false}; + std::atomic decoder_reset_requested_{false}; + std::thread worker_; + muninn_rudp_url::RudpUrlParts parts_; + std::map video_assemblies_; + std::map> video_parity_cache_; + std::deque expired_video_frame_order_; + std::set expired_video_frame_keys_; + muninn_rudp_fragments::FragmentAssembler packet_fragments_; + muninn_rudp_fragments::FragmentAssembler audio_packet_fragments_; + muninn_rudp_ack::AckTracker ack_tracker_; + muninn_rudp_ack::AckTracker audio_ack_tracker_; + uint64_t packets_forwarded_ = 0; + uint64_t bytes_forwarded_ = 0; + uint64_t payloads_decoded_dropped_ = 0; + uint64_t video_assemblies_expired_ = 0; + uint64_t video_recovery_dropped_ = 0; + uint64_t video_parity_received_ = 0; + uint64_t video_parity_cached_ = 0; + uint64_t video_parity_ignored_ = 0; + uint64_t video_parity_recovered_ = 0; + uint64_t video_parity_unusable_ = 0; + uint64_t video_forward_failures_ = 0; + uint64_t audio_forward_failures_ = 0; + uint64_t audio_packets_reordered_ = 0; + uint64_t audio_packets_skipped_ = 0; + uint64_t audio_packets_stale_ = 0; + std::string audio_stream_id_; + std::string audio_session_id_; + bool next_audio_packet_ready_ = false; + uint64_t next_audio_packet_id_ = 0; + std::chrono::steady_clock::time_point audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); + std::map pending_audio_packets_; + bool media_sequence_initialized_ = false; + uint32_t last_media_sequence_ = 0; + uint64_t media_sequence_gap_events_ = 0; + uint64_t media_sequence_gap_packets_ = 0; + uint64_t last_sequence_gap_log_ms_ = 0; + uint64_t last_video_expiry_scan_ms_ = 0; + bool waiting_for_video_keyframe_ = true; + bool media_connected_ = false; + std::atomic last_keyframe_request_ms_ = 0; + std::string source_url_; + RudpBridgeOutputs outputs_; +}; + +static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback) +{ + const char *value = obs_data_get_string(settings, key); + return value && *value ? std::string(value) : std::string(fallback); +} + +static std::string quote_windows_arg(const std::string &value) +{ + if (value.empty()) { + return "\"\""; + } + bool needs_quotes = false; + for (const char c : value) { + if (std::isspace(static_cast(c)) || c == '"') { + needs_quotes = true; + break; + } + } + if (!needs_quotes) { + return value; + } + std::string quoted = "\""; + size_t backslashes = 0; + for (const char c : value) { + if (c == '\\') { + ++backslashes; + continue; + } + if (c == '"') { + quoted.append(backslashes * 2 + 1, '\\'); + quoted.push_back(c); + backslashes = 0; + continue; + } + quoted.append(backslashes, '\\'); + backslashes = 0; + quoted.push_back(c); + } + quoted.append(backslashes * 2, '\\'); + quoted.push_back('"'); + return quoted; +} + +static int run_muninn_cli(const std::string &exe, const std::vector &args) +{ + _putenv_s("CULTMESH_URI_ODIN_RUDP", MuninnOdinRudpEndpoint); + std::string command = quote_windows_arg(exe.empty() ? MuninnCommandExePath : exe); + for (const auto &arg : args) { + command.push_back(' '); + command += quote_windows_arg(arg); + } + return std::system(command.c_str()); +} + +static void refresh_obs_catalog_from_odin(MuninnSource *ctx) +{ + if (!ctx) { + return; + } + const int exit_code = run_muninn_cli(ctx->command_exe_path, { + "obs-catalog-status", + "--store", + ctx->store_path, + "--host", + ctx->host_id, + "--odin-cultmesh-uri", + MuninnOdinCultMeshUri, + }); + if (exit_code != 0) { + blog(LOG_WARNING, "Muninn Stream could not refresh OBS catalog from Odin via %s", ctx->command_exe_path.c_str()); + } +} + +static const MuninnStreamOption *find_selected(const std::vector &options, const std::string &stream_id) +{ + for (const auto &option : options) { + if (option.stream_id == stream_id) { + return &option; + } + } + return options.empty() ? nullptr : &options.front(); +} + +static bool is_playable_muninn_stream(const MuninnStreamOption &option) +{ + return !option.stream_id.empty() && option.state != "affordance"; +} + +static MuninnStreamOption default_muninn_stream_option() +{ + return MuninnStreamOption{ + "discovery-missing", + "Muninn discovery unavailable", + "", + "discovery-missing", + "", + "", + 0, + 0, + 0, + 0, + {}, + {}, + }; +} + +static void queue_planar_audio(MuninnSource *ctx, + const float *const *planes, + uint32_t frames, + enum speaker_layout speakers, + uint32_t samples_per_sec) +{ + if (!ctx || !ctx->source || !planes || frames == 0) { + return; + } + + MuninnQueuedAudio packet; + packet.frames = frames; + packet.timestamp = os_gettime_ns(); + packet.speakers = speakers; + packet.samples_per_sec = samples_per_sec; + const size_t bytes_per_plane = static_cast(frames) * sizeof(float); + for (size_t plane = 0; plane < MAX_AV_PLANES; ++plane) { + if (!planes[plane]) { + continue; + } + packet.planes[plane].resize(bytes_per_plane); + std::memcpy(packet.planes[plane].data(), planes[plane], bytes_per_plane); + } + + { + std::lock_guard lock(ctx->audio_mutex); + if (!ctx->audio_worker_running) { + return; + } + while (ctx->audio_queue.size() >= MuninnAudioQueueMaxPackets) { + ctx->audio_queue.pop_front(); + } + ctx->audio_queue.push_back(std::move(packet)); + } + ctx->audio_cv.notify_one(); +} + +static void queue_stereo_interleaved_audio(MuninnSource *ctx, const float *samples, uint32_t frames) +{ + if (!samples || frames == 0) { + return; + } + std::vector left(frames); + std::vector right(frames); + for (uint32_t frame = 0; frame < frames; ++frame) { + left[frame] = samples[frame * 2]; + right[frame] = samples[frame * 2 + 1]; + } + const float *planes[MAX_AV_PLANES] = {}; + planes[0] = left.data(); + planes[1] = right.data(); + queue_planar_audio(ctx, planes, frames, SPEAKERS_STEREO, 48000); +} + +static std::string quote_windows_argument(const std::string &value) +{ + std::string result = "\""; + size_t backslashes = 0; + for (const char ch : value) { + if (ch == '\\') { + ++backslashes; + continue; + } + if (ch == '"') { + result.append(backslashes * 2 + 1, '\\'); + result.push_back('"'); + backslashes = 0; + continue; + } + result.append(backslashes, '\\'); + backslashes = 0; + result.push_back(ch); + } + result.append(backslashes * 2, '\\'); + result.push_back('"'); + return result; +} + +static bool run_detached_command(const std::vector &args) +{ + if (args.empty()) { + return false; + } + + std::string command_line; + for (const auto &arg : args) { + if (!command_line.empty()) { + command_line.push_back(' '); + } + command_line += quote_windows_argument(arg); + } + +#ifdef _WIN32 + STARTUPINFOA startup = {}; + startup.cb = sizeof(startup); + PROCESS_INFORMATION process = {}; + std::vector mutable_command(command_line.begin(), command_line.end()); + mutable_command.push_back('\0'); + const BOOL ok = CreateProcessA( + nullptr, + mutable_command.data(), + nullptr, + nullptr, + FALSE, + CREATE_NO_WINDOW, + nullptr, + nullptr, + &startup, + &process); + if (!ok) { + blog(LOG_WARNING, "Muninn Stream could not launch command %s", command_line.c_str()); + return false; + } + CloseHandle(process.hThread); + CloseHandle(process.hProcess); + return true; +#else + return false; +#endif +} + +class FfmpegAudioDecoder { +public: + ~FfmpegAudioDecoder() + { + stop(); + } + + void start(MuninnSource *ctx, const std::string &audio_url) + { + stop(); + if (!ctx || audio_url.empty()) { + return; + } + +#ifdef _WIN32 + SECURITY_ATTRIBUTES pipe_attrs = {}; + pipe_attrs.nLength = sizeof(pipe_attrs); + pipe_attrs.bInheritHandle = TRUE; + + HANDLE stdout_read = nullptr; + HANDLE stdout_write = nullptr; + if (!CreatePipe(&stdout_read, &stdout_write, &pipe_attrs, 0)) { + blog(LOG_WARNING, "Muninn Stream could not create ffmpeg audio decoder pipe"); + return; + } + SetHandleInformation(stdout_read, HANDLE_FLAG_INHERIT, 0); + + HANDLE nul = CreateFileA("NUL", GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, &pipe_attrs, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); + if (nul == INVALID_HANDLE_VALUE) { + nul = nullptr; + } + + std::vector args{ + "ffmpeg.exe", + "-hide_banner", + "-loglevel", + "warning", + "-fflags", + "nobuffer", + "-flags", + "low_delay", + "-probesize", + "32768", + "-analyzeduration", + "0", + "-f", + "aac", + "-i", + audio_url, + "-vn", + "-f", + "f32le", + "-ac", + "2", + "-ar", + "48000", + "pipe:1", + }; + + std::string command_line; + for (const auto &arg : args) { + if (!command_line.empty()) { + command_line.push_back(' '); + } + command_line += quote_windows_argument(arg); + } + + STARTUPINFOA startup = {}; + startup.cb = sizeof(startup); + startup.dwFlags = STARTF_USESTDHANDLES; + startup.hStdInput = GetStdHandle(STD_INPUT_HANDLE); + startup.hStdOutput = stdout_write; + startup.hStdError = nul ? nul : GetStdHandle(STD_ERROR_HANDLE); + + PROCESS_INFORMATION process = {}; + std::vector mutable_command(command_line.begin(), command_line.end()); + mutable_command.push_back('\0'); + if (!CreateProcessA(nullptr, mutable_command.data(), nullptr, nullptr, TRUE, CREATE_NO_WINDOW, nullptr, nullptr, &startup, &process)) { + blog(LOG_WARNING, "Muninn Stream could not start local ffmpeg audio decoder for %s", audio_url.c_str()); + CloseHandle(stdout_read); + CloseHandle(stdout_write); + if (nul) { + CloseHandle(nul); + } + return; + } + CloseHandle(stdout_write); + if (nul) { + CloseHandle(nul); + } + + ctx_ = ctx; + process_ = process.hProcess; + thread_ = process.hThread; + stdout_read_ = stdout_read; + running_.store(true); + worker_ = std::thread([this] { read_loop(); }); + blog(LOG_INFO, "Muninn Stream local ffmpeg audio decoder started for %s", audio_url.c_str()); +#else + (void)ctx; + (void)audio_url; +#endif + } + + void stop() + { +#ifdef _WIN32 + running_.store(false); + if (process_) { + TerminateProcess(process_, 0); + } + if (worker_.joinable()) { + worker_.join(); + } + if (stdout_read_) { + CloseHandle(stdout_read_); + stdout_read_ = nullptr; + } + if (thread_) { + CloseHandle(thread_); + thread_ = nullptr; + } + if (process_) { + CloseHandle(process_); + process_ = nullptr; + } + ctx_ = nullptr; +#endif + } + +private: + void read_loop() + { +#ifdef _WIN32 + constexpr size_t frames_per_chunk = 480; + constexpr size_t bytes_per_frame = sizeof(float) * 2; + std::vector buffer(frames_per_chunk * bytes_per_frame); + std::vector pending; + pending.reserve(buffer.size() * 2); + while (running_.load()) { + DWORD read = 0; + if (!ReadFile(stdout_read_, buffer.data(), static_cast(buffer.size()), &read, nullptr) || read == 0) { + break; + } + pending.insert(pending.end(), buffer.begin(), buffer.begin() + read); + const size_t complete_bytes = (pending.size() / bytes_per_frame) * bytes_per_frame; + if (complete_bytes == 0) { + continue; + } + const auto *samples = reinterpret_cast(pending.data()); + queue_stereo_interleaved_audio(ctx_, samples, static_cast(complete_bytes / bytes_per_frame)); + pending.erase(pending.begin(), pending.begin() + static_cast(complete_bytes)); + } +#endif + } + + std::atomic running_ = false; + MuninnSource *ctx_ = nullptr; + std::thread worker_; +#ifdef _WIN32 + HANDLE process_ = nullptr; + HANDLE thread_ = nullptr; + HANDLE stdout_read_ = nullptr; +#endif +}; + +static std::string command_observed_at() +{ + const auto now = std::chrono::system_clock::now().time_since_epoch(); + const auto seconds = std::chrono::duration_cast(now).count(); + return "unix-" + std::to_string(seconds); +} + +static bool split_ipv4_endpoint(const std::string &endpoint, std::string &host, uint16_t &port) +{ + const auto colon = endpoint.rfind(':'); + if (colon == std::string::npos || colon + 1 >= endpoint.size()) { + return false; + } + host = endpoint.substr(0, colon); + try { + const auto parsed = std::stoul(endpoint.substr(colon + 1)); + if (parsed == 0 || parsed > 65535) { + return false; + } + port = static_cast(parsed); + return true; + } catch (...) { + return false; + } +} + +static std::string replace_endpoint_host_if_unspecified(const std::string &endpoint, const std::string &host) +{ + std::string endpoint_host; + uint16_t port = 0; + if (!split_ipv4_endpoint(endpoint, endpoint_host, port)) { + return endpoint; + } + if (endpoint_host == "0.0.0.0" || endpoint_host == "::" || endpoint_host.empty()) { + return host + ":" + std::to_string(port); + } + return endpoint; +} + +#ifdef _WIN32 +static std::string sockaddr_ipv4_string(const sockaddr_in &addr) +{ + char text[INET_ADDRSTRLEN] = {}; + if (inet_ntop(AF_INET, &addr.sin_addr, text, sizeof(text))) { + return text; + } + return {}; +} +#endif + +static std::vector make_rudp_packet(uint8_t packet_type, + uint8_t flags, + uint32_t connection_id, + uint32_t sequence, + const std::string &channel, + const std::vector &payload) +{ + const uint8_t header_bytes = static_cast(36 + channel.size()); + std::vector packet(static_cast(header_bytes) + payload.size()); + packet[0] = 'C'; + packet[1] = 'N'; + packet[2] = 'R'; + packet[3] = '0'; + packet[4] = 0; + packet[5] = packet_type; + packet[6] = flags; + packet[7] = header_bytes; + write_be32(packet, 8, connection_id); + write_be32(packet, 12, sequence); + write_be32(packet, 16, 0); + write_be32(packet, 20, 0); + write_be16(packet, 24, 0); + write_be16(packet, 26, 0); + write_be16(packet, 28, 0); + write_be32(packet, 30, static_cast(payload.size())); + packet[34] = static_cast(channel.size()); + packet[35] = 0; + std::copy(channel.begin(), channel.end(), packet.begin() + 36); + std::copy(payload.begin(), payload.end(), packet.begin() + header_bytes); + return packet; +} + +static std::vector encode_capture_stream_command_wire(const MuninnSource *ctx, const std::string &stream_id, const std::string &observed_at) +{ + const std::string canonical_stream_id = canonical_muninn_stream_id(stream_id); + const std::string command_id = + ctx->host_id + ":" + canonical_stream_id + ":capture-stream:start:" + observed_at; + + muninn_media_wire::MsgpackWriter record; + record.write_array_len(18); + record.write_string(command_id); + record.write_string(ctx->host_id); + record.write_string(canonical_stream_id); + record.write_string("pending"); + record.write_string("start"); + record.write_string(ctx->media_target_host); + record.write_u64(ctx->media_port); + record.write_nil(); + record.write_u64(MuninnDefaultMediaPort); + record.write_string("rudp"); + record.write_u64(ctx->media_packet_bytes); + record.write_string("mimir.obs"); + record.write_string("OBS requested a daemon-owned capture stream over CultNet RUDP"); + record.write_string(observed_at); + record.write_u64(ctx->rudp_video_bitrate_kbps); + record.write_u64(ctx->rudp_latency_budget_ms); + record.write_string(source_wants_video(ctx) ? ctx->video_source_id : MuninnDisabledVideoSourceId); + record.write_string(source_wants_audio(ctx) ? ctx->audio_source_id : MuninnDisabledAudioSourceId); + + muninn_media_wire::MsgpackWriter document; + document.write_map_len(9); + document.write_string("schemaId"); + document.write_string("muninn.capture_stream_command"); + document.write_string("recordKey"); + document.write_string(command_id); + document.write_string("storedAt"); + document.write_string(observed_at); + document.write_string("payloadEncoding"); + document.write_string("messagepack"); + document.write_string("payload"); + document.write_binary(record.bytes()); + document.write_string("sourceRuntimeId"); + document.write_string("mimir-obs-plugin"); + document.write_string("sourceAgentId"); + document.write_nil(); + document.write_string("sourceRole"); + document.write_string("capture-command-publisher"); + document.write_string("tags"); + muninn_media_wire::write_string_array(document, {"cultnet.transport.rudp.v0"}); + + muninn_media_wire::MsgpackWriter wire; + wire.write_map_len(3); + wire.write_string("schemaVersion"); + wire.write_string("cultnet.document_put_raw.v0"); + wire.write_string("messageId"); + wire.write_string("muninn-capture-command:" + command_id); + wire.write_string("document"); + wire.write_raw(document.bytes()); + return wire.bytes(); +} + +static bool publish_capture_command_rudp(const std::string &target, const std::vector &payload) +{ +#ifdef _WIN32 + std::string host; + uint16_t port = 0; + if (!split_ipv4_endpoint(target, host, port)) { + blog(LOG_WARNING, "Muninn Stream cannot parse command RUDP target %s", target.c_str()); + return false; + } + + WSADATA wsa = {}; + if (WSAStartup(MAKEWORD(2, 2), &wsa) != 0) { + return false; + } + SOCKET socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (socket == INVALID_SOCKET) { + WSACleanup(); + return false; + } + const DWORD timeout_ms = 100; + setsockopt(socket, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast(&timeout_ms), sizeof(timeout_ms)); + + sockaddr_in remote = {}; + remote.sin_family = AF_INET; + remote.sin_port = htons(port); + if (inet_pton(AF_INET, host.c_str(), &remote.sin_addr) != 1) { + closesocket(socket); + WSACleanup(); + blog(LOG_WARNING, "Muninn Stream command RUDP target is not IPv4: %s", target.c_str()); + return false; + } + + const auto connect = make_rudp_packet(1, 0, MuninnCommandRudpConnectionId, 1, "control", {}); + const auto data = make_rudp_packet(3, 0, MuninnCommandRudpConnectionId, 2, "schema", payload); + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + std::array buffer = {}; + while (std::chrono::steady_clock::now() < deadline) { + sendto(socket, reinterpret_cast(connect.data()), static_cast(connect.size()), 0, + reinterpret_cast(&remote), sizeof(remote)); + sockaddr_in source = {}; + int source_len = sizeof(source); + const int received = recvfrom(socket, reinterpret_cast(buffer.data()), static_cast(buffer.size()), 0, + reinterpret_cast(&source), &source_len); + if (received >= 36 && + buffer[0] == 'C' && buffer[1] == 'N' && buffer[2] == 'R' && buffer[3] == '0' && + buffer[5] == 2 && + read_be32(buffer.data() + 8) == MuninnCommandRudpConnectionId) { + const int sent = sendto(socket, reinterpret_cast(data.data()), static_cast(data.size()), 0, + reinterpret_cast(&remote), sizeof(remote)); + closesocket(socket); + WSACleanup(); + return sent == static_cast(data.size()); + } + } + + closesocket(socket); + WSACleanup(); + blog(LOG_WARNING, "Muninn Stream timed out connecting command RUDP target %s", target.c_str()); + return false; +#else + (void)target; + (void)payload; + return false; +#endif +} + +static std::string expected_rudp_stream_url(const MuninnSource *ctx, const std::string &stream_id) +{ + const auto reliable_expire_after_ms = std::max(1, ctx->rudp_latency_budget_ms); + return "rudp://" + ctx->media_target_host + ":" + std::to_string(ctx->media_port) + "/" + stream_id + + "?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" + + std::to_string(reliable_expire_after_ms) + "&assembly_deadline_ms=" + std::to_string(ctx->rudp_latency_budget_ms) + + "&gap_wait_ms=16&audio_reorder_ms=" + std::to_string(ctx->rudp_audio_reorder_ms); +} + +static std::string playable_rudp_stream_url(const MuninnSource *ctx, const MuninnStreamOption &selected) +{ + if (starts_with(selected.url, "rudp://")) { + return expected_rudp_stream_url(ctx, canonical_muninn_stream_id(selected.stream_id)); + } + return selected.url.empty() ? expected_rudp_stream_url(ctx, selected.stream_id) : selected.url; +} + +static std::string canonical_muninn_stream_id(const std::string &stream_id) +{ + const auto colon = stream_id.rfind(':'); + if (colon == std::string::npos || colon + 1 >= stream_id.size()) { + return stream_id; + } + const auto suffix = stream_id.substr(colon + 1); + if (std::all_of(suffix.begin(), suffix.end(), [](unsigned char ch) { return std::isdigit(ch) != 0; })) { + return stream_id.substr(0, colon); + } + return stream_id; +} + +static std::string rudp_url_stream_id(const std::string &url) +{ + const auto authority_start = std::string("rudp://").size(); + const auto path_start = url.find('/', authority_start); + if (path_start == std::string::npos || path_start + 1 >= url.size()) { + return {}; + } + const auto query_start = url.find('?', path_start + 1); + return url.substr(path_start + 1, (query_start == std::string::npos ? url.size() : query_start) - path_start - 1); +} + +static bool equivalent_rudp_stream_url(const std::string &left, const std::string &right) +{ + if (!starts_with(left, "rudp://") || !starts_with(right, "rudp://")) { + return left == right; + } + try { + const auto left_parts = muninn_rudp_url::parse(left); + const auto right_parts = muninn_rudp_url::parse(right); + if (!left_parts || !right_parts) { + return left == right; + } + return left_parts->port == right_parts->port && + canonical_muninn_stream_id(rudp_url_stream_id(left)) == canonical_muninn_stream_id(rudp_url_stream_id(right)) && + left_parts->connection_id == right_parts->connection_id && + left_parts->audio_connection_id == right_parts->audio_connection_id && + left_parts->sender_resend_delay_ms == right_parts->sender_resend_delay_ms && + left_parts->reliable_expire_after_ms == right_parts->reliable_expire_after_ms && + left_parts->video_assembly_deadline_ms == right_parts->video_assembly_deadline_ms && + left_parts->gap_wait_ms == right_parts->gap_wait_ms; + } catch (...) { + return left == right; + } +} + +static std::string selected_source_id( + const std::vector> &sources, + const std::string ¤t) +{ + if (!current.empty()) { + for (const auto &source : sources) { + if (source.first == current) { + return current; + } + } + for (const auto &source : sources) { + if (source.first.rfind(current, 0) == 0) { + return source.first; + } + } + } + return sources.empty() ? std::string{} : sources.front().first; +} + +static std::string endpoint_setting(obs_data_t *settings, const char *key, const char *current_default, const char *deprecated_default) +{ + std::string value = obs_string(settings, key, current_default); + if (value == deprecated_default) { + obs_data_set_string(settings, key, current_default); + return current_default; + } + return value; +} + +static void request_stream_activation(MuninnSource *ctx, const std::string &stream_id, bool force = false) +{ + const auto canonical_stream_id = canonical_muninn_stream_id(stream_id); + if (!ctx || canonical_stream_id.empty() || canonical_stream_id == "discovery-missing") { + return; + } + const auto now = std::chrono::steady_clock::now(); + if (!force && + ctx->last_requested_stream_id == canonical_stream_id && + ctx->last_requested_video_source_id == ctx->video_source_id && + ctx->last_requested_audio_source_id == ctx->audio_source_id && + now - ctx->last_activation_request < std::chrono::milliseconds(MuninnActivationRetryMs)) { + return; + } + ctx->last_requested_stream_id = canonical_stream_id; + ctx->last_requested_video_source_id = ctx->video_source_id; + ctx->last_requested_audio_source_id = ctx->audio_source_id; + ctx->last_activation_request = now; + + const auto command_exe = ctx->command_exe_path; + const auto command_store = std::string(MuninnObsCommandStorePath); + const auto store_path = ctx->store_path; + const auto host_id = ctx->host_id; + const auto media_receiver_host = ctx->media_target_host; + const auto media_port = ctx->media_port; + const auto media_packet_bytes = ctx->media_packet_bytes; + const auto video_bitrate = ctx->rudp_video_bitrate_kbps; + const auto latency_budget = ctx->rudp_latency_budget_ms; + const auto video_source_id = ctx->video_source_id; + const auto audio_source_id = ctx->audio_source_id; + const bool wants_video = source_wants_video(ctx); + const bool wants_audio = source_wants_audio(ctx); + std::thread([command_exe, + command_store, + store_path, + host_id, + canonical_stream_id, + media_receiver_host, + media_port, + media_packet_bytes, + video_bitrate, + latency_budget, + video_source_id, + audio_source_id, + wants_video, + wants_audio] { + std::vector args = { + "request-stream", + "--store", + command_store, + "--activate-store", + store_path, + "--host", + host_id, + "--stream", + canonical_stream_id, + "--target-host", + MuninnOdinMediaTargetUri, + "--obs-target-host", + media_receiver_host, + "--obs-port", + std::to_string(media_port), + "--media-transport", + "rudp", + "--media-packet-bytes", + std::to_string(media_packet_bytes), + "--rudp-video-bitrate-kbps", + std::to_string(video_bitrate), + "--rudp-latency-budget-ms", + std::to_string(latency_budget), + "--odin-cultmesh-uri", + MuninnOdinCultMeshUri, + }; + if (!wants_video) { + args.push_back("--no-video"); + } else if (!video_source_id.empty() && video_source_id != MuninnDisabledVideoSourceId) { + args.push_back("--video-source"); + args.push_back(video_source_id + "=" + video_source_id); + } + if (!wants_audio) { + args.push_back("--no-audio"); + } else if (!audio_source_id.empty() && audio_source_id != MuninnDisabledAudioSourceId) { + args.push_back("--audio-source-id"); + args.push_back(audio_source_id); + } + if (run_muninn_cli(command_exe, args) != 0) { + blog(LOG_WARNING, "Muninn Stream failed to publish Odin capture request"); + } + }).detach(); + blog(LOG_INFO, + "Muninn Stream requested activation for %s video_source=%s audio_source=%s through Odin %s video_bitrate_kbps=%u latency_budget_ms=%u audio_reorder_ms=%u", + canonical_stream_id.c_str(), + ctx->video_source_id.c_str(), + ctx->audio_source_id.c_str(), + MuninnOdinCultMeshUri, + ctx->rudp_video_bitrate_kbps, + ctx->rudp_latency_budget_ms, + ctx->rudp_audio_reorder_ms); +} + +class MsgpackReader { +public: + explicit MsgpackReader(const std::vector &bytes) : data_(bytes.data()), size_(bytes.size()) {} + MsgpackReader(const uint8_t *data, size_t size) : data_(data), size_(size) {} + + uint32_t read_array_len() + { + const uint8_t tag = read_u8(); + if ((tag & 0xf0) == 0x90) { + return tag & 0x0f; + } + if (tag == 0xdc) { + return read_u16(); + } + if (tag == 0xdd) { + return read_u32(); + } + throw std::runtime_error("expected MessagePack array"); + } + + std::string read_string() + { + const uint8_t tag = read_u8(); + uint32_t length = 0; + if ((tag & 0xe0) == 0xa0) { + length = tag & 0x1f; + } else if (tag == 0xd9) { + length = read_u8(); + } else if (tag == 0xda) { + length = read_u16(); + } else if (tag == 0xdb) { + length = read_u32(); + } else { + throw std::runtime_error("expected MessagePack string"); + } + require(length); + std::string value(reinterpret_cast(data_ + pos_), length); + pos_ += length; + return value; + } + + std::vector read_string_array() + { + const uint32_t length = read_array_len(); + std::vector values; + values.reserve(length); + for (uint32_t index = 0; index < length; ++index) { + values.push_back(read_string()); + } + return values; + } + + std::vector read_bytes() + { + const uint8_t tag = read_u8(); + uint32_t length = 0; + if (tag == 0xc4) { + length = read_u8(); + } else if (tag == 0xc5) { + length = read_u16(); + } else if (tag == 0xc6) { + length = read_u32(); + } else if ((tag & 0xf0) == 0x90 || tag == 0xdc || tag == 0xdd) { + pos_--; + length = read_array_len(); + std::vector bytes; + bytes.reserve(length); + for (uint32_t index = 0; index < length; ++index) { + bytes.push_back(static_cast(read_unsigned())); + } + return bytes; + } else { + throw std::runtime_error("expected MessagePack bytes"); + } + require(length); + std::vector bytes(data_ + pos_, data_ + pos_ + length); + pos_ += length; + return bytes; + } + + uint64_t read_unsigned() + { + const uint8_t tag = read_u8(); + if (tag <= 0x7f) { + return tag; + } + if (tag == 0xcc) { + return read_u8(); + } + if (tag == 0xcd) { + return read_u16(); + } + if (tag == 0xce) { + return read_u32(); + } + if (tag == 0xcf) { + return read_u64(); + } + throw std::runtime_error("expected MessagePack unsigned integer"); + } + + void skip() + { + const uint8_t tag = read_u8(); + if (tag <= 0x7f || tag >= 0xe0 || tag == 0xc0 || tag == 0xc2 || tag == 0xc3) { + return; + } + if ((tag & 0xe0) == 0xa0) { + skip_bytes(tag & 0x1f); + return; + } + if ((tag & 0xf0) == 0x90) { + skip_items(tag & 0x0f); + return; + } + if ((tag & 0xf0) == 0x80) { + skip_items((tag & 0x0f) * 2); + return; + } + switch (tag) { + case 0xc4: + case 0xd9: + skip_bytes(read_u8()); + return; + case 0xc5: + case 0xda: + skip_bytes(read_u16()); + return; + case 0xc6: + case 0xdb: + skip_bytes(read_u32()); + return; + case 0xcc: + case 0xd0: + skip_bytes(1); + return; + case 0xcd: + case 0xd1: + skip_bytes(2); + return; + case 0xce: + case 0xd2: + case 0xca: + skip_bytes(4); + return; + case 0xcf: + case 0xd3: + case 0xcb: + skip_bytes(8); + return; + case 0xdc: + skip_items(read_u16()); + return; + case 0xdd: + skip_items(read_u32()); + return; + case 0xde: + skip_items(read_u16() * 2); + return; + case 0xdf: + skip_items(read_u32() * 2); + return; + default: + throw std::runtime_error("unsupported MessagePack tag"); + } + } + +private: + const uint8_t *data_; + size_t size_; + size_t pos_ = 0; + + void require(size_t count) const + { + if (count > size_ - pos_) { + throw std::runtime_error("truncated MessagePack input"); + } + } + + uint8_t read_u8() + { + require(1); + return data_[pos_++]; + } + + uint16_t read_u16() + { + require(2); + const uint16_t value = (static_cast(data_[pos_]) << 8) | static_cast(data_[pos_ + 1]); + pos_ += 2; + return value; + } + + uint32_t read_u32() + { + require(4); + const uint32_t value = (static_cast(data_[pos_]) << 24) | (static_cast(data_[pos_ + 1]) << 16) | + (static_cast(data_[pos_ + 2]) << 8) | static_cast(data_[pos_ + 3]); + pos_ += 4; + return value; + } + + uint64_t read_u64() + { + const uint64_t high = read_u32(); + const uint64_t low = read_u32(); + return (high << 32) | low; + } + + void skip_bytes(size_t count) + { + require(count); + pos_ += count; + } + + void skip_items(uint32_t count) + { + for (uint32_t index = 0; index < count; ++index) { + skip(); + } + } +}; + +static std::vector> pair_source_options( + const std::vector &ids, + const std::vector &labels) +{ + std::vector> options; + options.reserve(ids.size()); + for (size_t index = 0; index < ids.size(); ++index) { + if (ids[index].empty()) { + continue; + } + const std::string label = index < labels.size() && !labels[index].empty() ? labels[index] : ids[index]; + options.emplace_back(ids[index], label); + } + return options; +} + +static std::vector decode_obs_catalog_payload(const std::vector &payload) +{ + MsgpackReader reader(payload); + const uint32_t length = reader.read_array_len(); + if (length < 7) { + return {}; + } + + reader.skip(); // catalog_id + reader.skip(); // host_id + auto stream_ids = reader.read_string_array(); + auto labels = reader.read_string_array(); + auto urls = reader.read_string_array(); + auto states = reader.read_string_array(); + reader.skip(); // updated_at + std::string command_rudp_target = MuninnCommandRudpTarget; + std::string media_target_host = MuninnDefaultMediaTargetHost; + uint16_t media_port = MuninnDefaultMediaPort; + uint32_t media_packet_bytes = MuninnDefaultMediaPacketBytes; + uint32_t rudp_video_bitrate_kbps = MuninnDefaultRudpVideoBitrateKbps; + uint32_t rudp_latency_budget_ms = MuninnDefaultRudpLatencyBudgetMs; + std::vector> video_sources; + std::vector> audio_sources; + if (length > 7) { + command_rudp_target = reader.read_string(); + } + if (length > 8) { + media_target_host = reader.read_string(); + } + if (length > 9) { + const auto parsed = reader.read_unsigned(); + if (parsed > 0 && parsed <= 65535) { + media_port = static_cast(parsed); + } + } + if (length > 10) { + const auto parsed = reader.read_unsigned(); + if (parsed > 0 && parsed <= UINT32_MAX) { + media_packet_bytes = static_cast(parsed); + } + } + if (length > 11) { + const auto parsed = reader.read_unsigned(); + if (parsed > 0 && parsed <= MuninnMaxRudpVideoBitrateKbps) { + rudp_video_bitrate_kbps = static_cast(parsed); + } + } + if (length > 12) { + const auto parsed = reader.read_unsigned(); + if (parsed > 0 && parsed <= MuninnMaxRudpLatencyBudgetMs) { + rudp_latency_budget_ms = static_cast(parsed); + } + } + if (length > 16) { + auto video_source_ids = reader.read_string_array(); + auto video_source_labels = reader.read_string_array(); + auto audio_source_ids = reader.read_string_array(); + auto audio_source_labels = reader.read_string_array(); + video_sources = pair_source_options(video_source_ids, video_source_labels); + audio_sources = pair_source_options(audio_source_ids, audio_source_labels); + } + if (video_sources.empty()) { + video_sources.emplace_back("display:0", "Display 1"); + } + if (audio_sources.empty()) { + audio_sources.emplace_back("wasapi-loopback:Realtek", "Realtek loopback"); + } + for (uint32_t index = length > 16 ? 17 : 13; index < length; ++index) { + reader.skip(); + } + + std::vector options; + const size_t count = std::min({stream_ids.size(), labels.size(), urls.size(), states.size()}); + options.reserve(count); + for (size_t index = 0; index < count; ++index) { + if (stream_ids[index].empty()) { + continue; + } + MuninnStreamOption option{ + stream_ids[index], + labels[index].empty() ? stream_ids[index] : labels[index], + urls[index], + states[index], + command_rudp_target, + media_target_host, + media_port, + media_packet_bytes, + rudp_video_bitrate_kbps, + rudp_latency_budget_ms, + video_sources, + audio_sources, + }; + if (!is_playable_muninn_stream(option)) { + continue; + } + options.push_back(std::move(option)); + } + + std::sort(options.begin(), options.end(), [](const auto &left, const auto &right) { + return left.label < right.label; + }); + return options; +} + +static std::vector read_catalog(const std::string &path) +{ + std::ifstream file(path, std::ios::binary); + if (!file) { + return {}; + } + + std::vector bytes((std::istreambuf_iterator(file)), std::istreambuf_iterator()); + if (bytes.empty()) { + return {}; + } + + try { + MsgpackReader reader(bytes); + const uint32_t snapshot_len = reader.read_array_len(); + if (snapshot_len < 3 || reader.read_string() != "cultcache.store.v1") { + return {}; + } + + const uint32_t catalog_count = reader.read_array_len(); + for (uint32_t index = 0; index < catalog_count; ++index) { + reader.skip(); + } + + const uint32_t record_count = reader.read_array_len(); + for (uint32_t index = 0; index < record_count; ++index) { + const uint32_t record_len = reader.read_array_len(); + if (record_len < 4) { + for (uint32_t item = 0; item < record_len; ++item) { + reader.skip(); + } + continue; + } + + const std::string key = reader.read_string(); + const std::string schema = reader.read_string(); + reader.skip(); // stored_at + const auto payload = reader.read_bytes(); + for (uint32_t item = 4; item < record_len; ++item) { + reader.skip(); + } + + if (key == MuninnObsCatalogKey && (schema == MuninnObsCatalogType || schema == MuninnObsCatalogSchema)) { + return decode_obs_catalog_payload(payload); + } + } + } catch (const std::exception &error) { + blog(LOG_WARNING, "Muninn Stream could not read CultCache store %s: %s", path.c_str(), error.what()); + } + + return {}; +} + +class CatalogDiscoveryReceiver { +public: + CatalogDiscoveryReceiver() = default; + + ~CatalogDiscoveryReceiver() + { + stop(); + } + + void start() + { + if (running_) { + return; + } + running_ = true; + worker_ = std::thread([this] { run(); }); + } + + void stop() + { + running_ = false; + if (socket_ != INVALID_SOCKET) { + closesocket(socket_); + socket_ = INVALID_SOCKET; + } + if (worker_.joinable()) { + worker_.join(); + } + } + + std::vector options() const + { + std::lock_guard lock(mutex_); + return options_; + } + +private: + void run() + { +#ifdef _WIN32 + WSADATA wsa = {}; + if (WSAStartup(MAKEWORD(2, 2), &wsa) != 0) { + return; + } +#endif + socket_ = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (socket_ == INVALID_SOCKET) { +#ifdef _WIN32 + WSACleanup(); +#endif + return; + } + sockaddr_in addr = {}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_ANY); + addr.sin_port = htons(MuninnCatalogDiscoveryPort); + if (bind(socket_, reinterpret_cast(&addr), sizeof(addr)) != 0) { + closesocket(socket_); + socket_ = INVALID_SOCKET; +#ifdef _WIN32 + WSACleanup(); +#endif + return; + } + + timeval timeout = {}; + timeout.tv_sec = 0; + timeout.tv_usec = 200000; + setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast(&timeout), sizeof(timeout)); + + std::array buffer = {}; + uint32_t bridge_sequence = 1; + while (running_) { + sockaddr_in remote = {}; + int remote_len = sizeof(remote); + const int received = recvfrom(socket_, reinterpret_cast(buffer.data()), static_cast(buffer.size()), 0, + reinterpret_cast(&remote), &remote_len); + if (received <= 0) { + continue; + } + handle_packet(buffer.data(), static_cast(received), remote, bridge_sequence); + } + + if (socket_ != INVALID_SOCKET) { + closesocket(socket_); + socket_ = INVALID_SOCKET; + } +#ifdef _WIN32 + WSACleanup(); +#endif + } + + void handle_packet(const uint8_t *packet, size_t size, const sockaddr_in &remote, uint32_t &bridge_sequence) + { + if (size < 36 || packet[0] != 'C' || packet[1] != 'N' || packet[2] != 'R' || packet[3] != '0') { + return; + } + const uint8_t version = packet[4]; + const uint8_t packet_type = packet[5]; + const uint8_t header_bytes = packet[7]; + const uint32_t connection_id = read_be32(packet + 8); + const uint32_t sequence = read_be32(packet + 12); + const uint16_t fragment_count = read_be16(packet + 28); + const uint32_t payload_len = read_be32(packet + 30); + const uint8_t channel_len = packet[34]; + if (version != 0 || connection_id != MuninnCatalogDiscoveryConnectionId || header_bytes < 36 || size < header_bytes) { + return; + } + const size_t payload_offset = static_cast(header_bytes); + if (payload_offset + payload_len > size || 36 + channel_len > header_bytes) { + return; + } + + if (packet_type == 1) { + ack_tracker_.reset(); + ack_tracker_.remember(sequence); + const auto [ack, ack_mask] = ack_tracker_.state(); + send_control_packet(2, 0x03, bridge_sequence++, ack, ack_mask, remote); + return; + } + + if (packet_type != 3 || fragment_count != 0) { + return; + } + ack_tracker_.remember(sequence); + try { + auto decoded = decode_obs_catalog_payload( + std::vector(packet + payload_offset, packet + payload_offset + payload_len)); + if (!decoded.empty()) { + const auto remote_host = sockaddr_ipv4_string(remote); + for (auto &option : decoded) { + if (option.command_rudp_target.empty()) { + option.command_rudp_target = MuninnCommandRudpTarget; + } + if (!remote_host.empty()) { + option.command_rudp_target = + replace_endpoint_host_if_unspecified(option.command_rudp_target, remote_host); + } + if (option.media_target_host.empty()) { + option.media_target_host = MuninnDefaultMediaTargetHost; + } + if (option.media_port == 0) { + option.media_port = MuninnDefaultMediaPort; + } + if (option.media_packet_bytes == 0) { + option.media_packet_bytes = MuninnDefaultMediaPacketBytes; + } + if (option.rudp_video_bitrate_kbps == 0) { + option.rudp_video_bitrate_kbps = MuninnDefaultRudpVideoBitrateKbps; + } + if (option.rudp_latency_budget_ms == 0) { + option.rudp_latency_budget_ms = MuninnDefaultRudpLatencyBudgetMs; + } + } + std::lock_guard lock(mutex_); + const bool first_discovery = options_.empty(); + options_ = std::move(decoded); + if (first_discovery) { + blog(LOG_INFO, "Muninn Stream accepted %llu stream option(s) from RUDP catalog discovery", + static_cast(options_.size())); + } + } + } catch (const std::exception &error) { + blog(LOG_WARNING, "Muninn Stream could not decode RUDP catalog discovery payload: %s", error.what()); + } + const auto [ack, ack_mask] = ack_tracker_.state(); + send_control_packet(2, 0x02, bridge_sequence++, ack, ack_mask, remote); + } + + void send_control_packet(uint8_t packet_type, uint8_t flags, uint32_t sequence, uint32_t ack, uint32_t ack_mask, const sockaddr_in &remote) + { + const std::string channel = "control"; + std::vector response(36 + channel.size()); + response[0] = 'C'; + response[1] = 'N'; + response[2] = 'R'; + response[3] = '0'; + response[4] = 0; + response[5] = packet_type; + response[6] = flags; + response[7] = static_cast(36 + channel.size()); + write_be32(response, 8, MuninnCatalogDiscoveryConnectionId); + write_be32(response, 12, sequence); + write_be32(response, 16, ack); + write_be32(response, 20, ack_mask); + write_be16(response, 24, 0); + write_be16(response, 26, 0); + write_be16(response, 28, 0); + write_be32(response, 30, 0); + response[34] = static_cast(channel.size()); + response[35] = 0; + std::copy(channel.begin(), channel.end(), response.begin() + 36); + sendto(socket_, reinterpret_cast(response.data()), static_cast(response.size()), 0, + reinterpret_cast(&remote), sizeof(remote)); + } + + std::atomic running_ = false; + SOCKET socket_ = INVALID_SOCKET; + std::thread worker_; + mutable std::mutex mutex_; + std::vector options_; + muninn_rudp_ack::AckTracker ack_tracker_; +}; + +static void release_ffmpeg_children(MuninnSource *ctx) +{ + if (ctx->audio_decoder) { + ctx->audio_decoder->stop(); + } + if (ctx->audio_media) { + obs_source_remove_audio_capture_callback(ctx->audio_media, muninn_child_audio, ctx); + obs_source_remove_active_child(ctx->source, ctx->audio_media); + obs_source_release(ctx->audio_media); + ctx->audio_media = nullptr; + } + if (ctx->media) { + obs_source_remove_audio_capture_callback(ctx->media, muninn_child_audio, ctx); + obs_source_remove_active_child(ctx->source, ctx->media); + obs_source_release(ctx->media); + ctx->media = nullptr; + } +} + +static void release_media(MuninnSource *ctx) +{ + if (ctx->bridge) { + ctx->bridge->stop(); + } + release_ffmpeg_children(ctx); +} + +static obs_source_t *create_ffmpeg_child_source(MuninnSource *ctx, + const char *name, + const std::string &media_url, + const char *input_format, + bool capture_audio) +{ + obs_data_t *settings = obs_data_create(); + obs_data_set_bool(settings, "is_local_file", false); + obs_data_set_string(settings, "input", media_url.c_str()); + obs_data_set_string(settings, "input_format", input_format); + obs_data_set_string(settings, "ffmpeg_options", "fflags=nobuffer flags=low_delay probesize=32768 analyzeduration=0"); + obs_data_set_bool(settings, "restart_on_activate", true); + obs_data_set_bool(settings, "clear_on_media_end", false); + obs_data_set_bool(settings, "close_when_inactive", false); + + obs_source_t *source = obs_source_create_private("ffmpeg_source", name, settings); + obs_data_release(settings); + if (!source) { + blog(LOG_WARNING, "Muninn Stream could not create OBS ffmpeg_source for %s", media_url.c_str()); + return nullptr; + } + if (capture_audio) { + obs_source_add_audio_capture_callback(source, muninn_child_audio, ctx); + } + obs_source_add_active_child(ctx->source, source); + return source; +} + +static void create_or_update_media(MuninnSource *ctx, const std::string &url, bool force_restart = false) +{ + const bool wants_video = source_wants_video(ctx); + const bool wants_audio = source_wants_audio(ctx); + if (url.empty()) { + release_media(ctx); + ctx->stream_url.clear(); + return; + } + + if (!wants_video && !wants_audio) { + release_media(ctx); + ctx->stream_url.clear(); + return; + } + + if (!force_restart && + ((wants_video && ctx->media) || (wants_audio && ctx->audio_decoder)) && + equivalent_rudp_stream_url(ctx->stream_url, url)) { + ctx->stream_url = url; + return; + } + + if (starts_with(url, "rudp://")) { + if (!force_restart && + ctx->bridge && + (!wants_video || ctx->media) && + (!wants_audio || ctx->audio_decoder) && + !ctx->stream_url.empty()) { + const auto previous_parts = muninn_rudp_url::parse(ctx->stream_url); + const auto next_parts = muninn_rudp_url::parse(url); + if (previous_parts && next_parts && + previous_parts->video_local_port == next_parts->video_local_port && + previous_parts->audio_local_port == next_parts->audio_local_port) { + ctx->bridge->start(url); + ctx->stream_url = url; + return; + } + } + if (force_restart && ctx->bridge && equivalent_rudp_stream_url(ctx->stream_url, url)) { + release_ffmpeg_children(ctx); + } else { + release_media(ctx); + } + if (!ctx->bridge) { + ctx->bridge = std::make_unique(); + } + const auto outputs = ctx->bridge->start(url); + bool activated = false; + if (wants_video) { + ctx->media = create_ffmpeg_child_source(ctx, "Muninn Stream Video", outputs.video_url, "h264", false); + activated = ctx->media != nullptr; + } + if (wants_audio) { + if (!ctx->audio_decoder) { + ctx->audio_decoder = std::make_unique(); + } + ctx->audio_decoder->start(ctx, outputs.audio_url); + activated = true; + } + if (activated) { + ctx->stream_url = url; + } else { + ctx->stream_url.clear(); + } + return; + } else if (ctx->bridge) { + ctx->bridge->stop(); + } + + release_ffmpeg_children(ctx); + if (wants_video) { + ctx->media = create_ffmpeg_child_source(ctx, "Muninn Stream Media", url, "mpegts", wants_audio); + } else if (wants_audio) { + ctx->audio_media = create_ffmpeg_child_source(ctx, "Muninn Stream Audio", url, "mpegts", true); + } + if (ctx->media || ctx->audio_media) { + ctx->stream_url = url; + } else { + ctx->stream_url.clear(); + } +} + +static void refresh_catalog(MuninnSource *ctx) +{ + if (!ctx) { + return; + } + ctx->catalog_options = + ctx->catalog_discovery ? ctx->catalog_discovery->options() : std::vector{}; + if (ctx->catalog_options.empty()) { + refresh_obs_catalog_from_odin(ctx); + ctx->catalog_options = read_catalog(ctx->store_path); + } + if (ctx->catalog_options.empty()) { + ctx->catalog_options.push_back(default_muninn_stream_option()); + } + ctx->next_catalog_refresh = + std::chrono::steady_clock::now() + std::chrono::milliseconds(MuninnCatalogRefreshMs); +} + +static void reconcile_selected_stream(MuninnSource *ctx) +{ + if (!ctx) { + return; + } + if (ctx->catalog_options.empty()) { + create_or_update_media(ctx, ""); + return; + } + + const auto *selected = find_selected(ctx->catalog_options, ctx->stream_id); + if (!selected) { + create_or_update_media(ctx, ""); + return; + } + + const auto previous_canonical_stream_id = canonical_muninn_stream_id(ctx->stream_id); + const auto selected_canonical_stream_id = canonical_muninn_stream_id(selected->stream_id); + const bool selection_changed = + !previous_canonical_stream_id.empty() && previous_canonical_stream_id != selected_canonical_stream_id; + ctx->stream_id = selected->stream_id; + const auto previous_video_source_id = ctx->video_source_id; + const auto previous_audio_source_id = ctx->audio_source_id; + ctx->video_source_id = source_wants_video(ctx) + ? selected_source_id(selected->video_sources, ctx->video_source_id) + : MuninnDisabledVideoSourceId; + ctx->audio_source_id = source_wants_audio(ctx) + ? selected_source_id(selected->audio_sources, ctx->audio_source_id) + : MuninnDisabledAudioSourceId; + const bool source_selection_changed = + ctx->video_source_id != previous_video_source_id || ctx->audio_source_id != previous_audio_source_id; + if (selection_changed || source_selection_changed) { + ctx->last_requested_stream_id.clear(); + ctx->last_requested_video_source_id.clear(); + ctx->last_requested_audio_source_id.clear(); + ctx->last_media_restart = std::chrono::steady_clock::time_point::min(); + } + const auto previous_command_rudp_target = ctx->command_rudp_target; + const auto previous_media_target_host = ctx->media_target_host; + const auto previous_media_port = ctx->media_port; + const auto previous_media_packet_bytes = ctx->media_packet_bytes; + ctx->command_rudp_target = selected->command_rudp_target.empty() ? MuninnCommandRudpTarget : selected->command_rudp_target; + std::string bootstrap_host; + uint16_t bootstrap_port = 0; + if (split_ipv4_endpoint(MuninnCommandRudpTarget, bootstrap_host, bootstrap_port)) { + ctx->command_rudp_target = replace_endpoint_host_if_unspecified(ctx->command_rudp_target, bootstrap_host); + } + ctx->media_target_host = selected->media_target_host.empty() ? MuninnDefaultMediaTargetHost : selected->media_target_host; + ctx->media_port = selected->media_port == 0 ? MuninnDefaultMediaPort : selected->media_port; + ctx->media_packet_bytes = + selected->media_packet_bytes == 0 ? MuninnDefaultMediaPacketBytes : selected->media_packet_bytes; + if (ctx->command_rudp_target != previous_command_rudp_target || + ctx->media_target_host != previous_media_target_host || + ctx->media_port != previous_media_port || + ctx->media_packet_bytes != previous_media_packet_bytes) { + ctx->last_requested_stream_id.clear(); + } + + const auto stream_url = selected->url.empty() && ctx->bridge && !ctx->stream_url.empty() + ? ctx->stream_url + : playable_rudp_stream_url(ctx, *selected); + const bool stream_url_changed = stream_url != ctx->stream_url; + if (stream_url_changed) { + blog(LOG_INFO, + "Muninn Stream selected RUDP URL changed: previous='%s' selected='%s' effective='%s'", + ctx->stream_url.c_str(), + selected->url.c_str(), + stream_url.c_str()); + } + create_or_update_media(ctx, stream_url); + + bool activation_needed = stream_url_changed || ctx->last_requested_stream_id != selected_canonical_stream_id; + bool force_activation = stream_url_changed; + if (ctx->bridge && starts_with(stream_url, "rudp://")) { + const uint64_t now_ms = steady_millis(); + if (ctx->bridge->consume_decoder_reset_request()) { + blog(LOG_INFO, "Muninn Stream RUDP sender session changed; refreshing local decoder readers"); + const auto now = std::chrono::steady_clock::now(); + if (now - ctx->last_media_restart >= std::chrono::milliseconds(MuninnMediaRestartCooldownMs)) { + create_or_update_media(ctx, stream_url, true); + ctx->last_media_restart = now; + } + } + if (ctx->bridge->consume_stale_transition(now_ms)) { + blog(LOG_WARNING, "Muninn Stream receiver media is stale; refreshing local media readers and re-requesting activation"); + const auto now = std::chrono::steady_clock::now(); + if (now - ctx->last_media_restart >= std::chrono::milliseconds(MuninnMediaRestartCooldownMs)) { + create_or_update_media(ctx, stream_url, true); + ctx->last_media_restart = now; + } + ctx->last_requested_stream_id.clear(); + activation_needed = true; + force_activation = true; + } else if (ctx->bridge->needs_activation(now_ms)) { + if (!ctx->bridge->is_running()) { + blog(LOG_WARNING, "Muninn Stream RUDP bridge is not running; rebuilding local media bridge"); + create_or_update_media(ctx, stream_url, true); + } + activation_needed = true; + } + } + + if (activation_needed) { + request_stream_activation(ctx, selected->stream_id, force_activation); + } +} + +static void populate_stream_list_from_options(obs_property_t *property, const std::vector &options) +{ + obs_property_list_clear(property); + for (const auto &option : options) { + std::string label = option.label; + if (!option.state.empty()) { + label += " [" + option.state + "]"; + } + if (option.url.empty()) { + label += " (activation required)"; + } + obs_property_list_add_string(property, label.c_str(), option.stream_id.c_str()); + } +} + +static void populate_source_list( + obs_property_t *property, + const std::vector> &sources) +{ + obs_property_list_clear(property); + for (const auto &source : sources) { + obs_property_list_add_string(property, source.second.c_str(), source.first.c_str()); + } +} + +static std::vector> catalog_video_sources_for_stream( + const MuninnSource *ctx, + const std::string &stream_id) +{ + const auto *selected = ctx ? find_selected(ctx->catalog_options, stream_id) : nullptr; + if (selected && !selected->video_sources.empty()) { + return selected->video_sources; + } + return {{"display:0", "Display 1"}}; +} + +static std::vector> catalog_video_sources(const MuninnSource *ctx) +{ + return catalog_video_sources_for_stream(ctx, ctx ? ctx->stream_id : std::string{}); +} + +static std::vector> catalog_audio_sources_for_stream( + const MuninnSource *ctx, + const std::string &stream_id) +{ + const auto *selected = ctx ? find_selected(ctx->catalog_options, stream_id) : nullptr; + if (selected && !selected->audio_sources.empty()) { + return selected->audio_sources; + } + return {{"wasapi-loopback:Realtek", "Realtek loopback"}}; +} + +static std::vector> catalog_audio_sources(const MuninnSource *ctx) +{ + return catalog_audio_sources_for_stream(ctx, ctx ? ctx->stream_id : std::string{}); +} + +static bool refresh_source_properties_for_settings( + obs_properties_t *props, + MuninnSource *ctx, + obs_data_t *settings) +{ + if (!props || !ctx) { + return false; + } + + refresh_catalog(ctx); + const std::string selected_stream_id = settings ? obs_string(settings, "stream_id", ctx->stream_id.c_str()) : ctx->stream_id; + + obs_property_t *stream_list = obs_properties_get(props, "stream_id"); + if (stream_list) { + populate_stream_list_from_options(stream_list, ctx->catalog_options); + } + + if (source_wants_video(ctx)) { + obs_property_t *video_list = obs_properties_get(props, "video_source_id"); + const auto video_sources = catalog_video_sources_for_stream(ctx, selected_stream_id); + if (video_list) { + populate_source_list(video_list, video_sources); + } + if (settings) { + const auto selected_video_source = selected_source_id( + video_sources, + obs_string(settings, "video_source_id", ctx->video_source_id.c_str())); + obs_data_set_string(settings, "video_source_id", selected_video_source.c_str()); + } + } + + if (source_wants_audio(ctx)) { + obs_property_t *audio_list = obs_properties_get(props, "audio_source_id"); + const auto audio_sources = catalog_audio_sources_for_stream(ctx, selected_stream_id); + if (audio_list) { + populate_source_list(audio_list, audio_sources); + } + if (settings) { + const auto selected_audio_source = selected_source_id( + audio_sources, + obs_string(settings, "audio_source_id", ctx->audio_source_id.c_str())); + obs_data_set_string(settings, "audio_source_id", selected_audio_source.c_str()); + } + } + + return true; +} + +static void populate_stream_list(obs_property_t *property, const std::string &store_path) +{ + auto options = read_catalog(store_path); + if (options.empty()) { + options.push_back(default_muninn_stream_option()); + } + populate_stream_list_from_options(property, options); +} + +static bool muninn_refresh_streams(obs_properties_t *props, obs_property_t *, void *data) +{ + std::string store_path = MuninnStorePath; + if (data) { + auto *ctx = static_cast(data); + store_path = ctx->store_path; + return refresh_source_properties_for_settings(props, ctx, nullptr); + } + + obs_property_t *stream_list = obs_properties_get(props, "stream_id"); + if (stream_list) { + populate_stream_list(stream_list, store_path); + } + return true; +} + +static bool muninn_stream_selection_modified( + void *priv, + obs_properties_t *props, + obs_property_t *, + obs_data_t *settings) +{ + auto *ctx = static_cast(priv); + return refresh_source_properties_for_settings(props, ctx, settings); +} + +static const char *muninn_get_name_combined(void *) +{ + return obs_module_text("MuninnStream"); +} + +static const char *muninn_get_name_video(void *) +{ + return obs_module_text("MuninnVideoSource"); +} + +static const char *muninn_get_name_audio(void *) +{ + return obs_module_text("MuninnAudioSource"); +} + +static void audio_worker_loop(MuninnSource *ctx) +{ + while (true) { + MuninnQueuedAudio packet; + { + std::unique_lock lock(ctx->audio_mutex); + ctx->audio_cv.wait(lock, [&] { return !ctx->audio_worker_running || !ctx->audio_queue.empty(); }); + if (!ctx->audio_worker_running && ctx->audio_queue.empty()) { + return; + } + packet = std::move(ctx->audio_queue.front()); + ctx->audio_queue.pop_front(); + } + + if (!ctx->source || packet.frames == 0) { + continue; + } + + struct obs_source_audio audio = {}; + for (size_t plane = 0; plane < MAX_AV_PLANES; ++plane) { + if (!packet.planes[plane].empty()) { + audio.data[plane] = packet.planes[plane].data(); + } + } + audio.frames = packet.frames; + audio.timestamp = packet.timestamp; + audio.format = AUDIO_FORMAT_FLOAT_PLANAR; + audio.speakers = packet.speakers; + audio.samples_per_sec = packet.samples_per_sec; + obs_source_output_audio(ctx->source, &audio); + } +} + +static void start_audio_worker(MuninnSource *ctx) +{ + std::lock_guard lock(ctx->audio_mutex); + if (ctx->audio_worker_running) { + return; + } + ctx->audio_worker_running = true; + ctx->audio_worker = std::thread([ctx] { audio_worker_loop(ctx); }); +} + +static void stop_audio_worker(MuninnSource *ctx) +{ + { + std::lock_guard lock(ctx->audio_mutex); + ctx->audio_worker_running = false; + ctx->audio_queue.clear(); + } + ctx->audio_cv.notify_all(); + if (ctx->audio_worker.joinable()) { + ctx->audio_worker.join(); + } +} + +static void muninn_update_kind(void *data, obs_data_t *settings, MuninnSourceKind kind) +{ + auto *ctx = static_cast(data); + ctx->kind = kind; + const auto previous_command_rudp_target = ctx->command_rudp_target; + const auto previous_media_target_host = ctx->media_target_host; + const auto previous_media_port = ctx->media_port; + const auto previous_rudp_video_bitrate_kbps = ctx->rudp_video_bitrate_kbps; + const auto previous_rudp_latency_budget_ms = ctx->rudp_latency_budget_ms; + const auto previous_rudp_audio_reorder_ms = ctx->rudp_audio_reorder_ms; + const auto previous_video_source_id = ctx->video_source_id; + const auto previous_audio_source_id = ctx->audio_source_id; + + ctx->store_path = obs_string(settings, "store_path", MuninnStorePath); + ctx->command_exe_path = obs_string(settings, "command_exe_path", MuninnCommandExePath); + ctx->command_rudp_target = endpoint_setting( + settings, + "command_rudp_target", + MuninnCommandRudpTarget, + MuninnDeprecatedWireGuardCommandRudpTarget); + ctx->host_id = obs_string(settings, "host_id", MuninnDefaultHostId); + ctx->media_target_host = endpoint_setting( + settings, + "media_target_host", + MuninnDefaultMediaTargetHost, + MuninnDeprecatedWireGuardMediaTargetHost); + ctx->media_port = static_cast(obs_data_get_int(settings, "media_port") > 0 + ? obs_data_get_int(settings, "media_port") + : MuninnDefaultMediaPort); + const auto configured_bitrate = obs_data_get_int(settings, "rudp_video_bitrate_kbps"); + ctx->rudp_video_bitrate_kbps = static_cast( + configured_bitrate > 0 ? std::min(configured_bitrate, MuninnMaxRudpVideoBitrateKbps) + : MuninnDefaultRudpVideoBitrateKbps); + const auto configured_latency = obs_data_get_int(settings, "rudp_latency_budget_ms"); + ctx->rudp_latency_budget_ms = static_cast( + configured_latency > 0 ? std::min(configured_latency, MuninnMaxRudpLatencyBudgetMs) + : MuninnDefaultRudpLatencyBudgetMs); + const auto configured_audio_reorder = obs_data_get_int(settings, "rudp_audio_reorder_ms"); + ctx->rudp_audio_reorder_ms = static_cast( + configured_audio_reorder >= 0 ? std::min(configured_audio_reorder, MuninnMaxRudpAudioReorderMs) + : MuninnDefaultRudpAudioReorderMs); + ctx->stream_id = obs_string(settings, "stream_id", ""); + if (source_wants_video(ctx)) { + ctx->video_source_id = obs_string(settings, "video_source_id", ctx->video_source_id.c_str()); + } else { + ctx->video_source_id = MuninnDisabledVideoSourceId; + } + if (source_wants_audio(ctx)) { + ctx->audio_source_id = obs_string(settings, "audio_source_id", ctx->audio_source_id.c_str()); + } else { + ctx->audio_source_id = MuninnDisabledAudioSourceId; + } + if (ctx->command_rudp_target != previous_command_rudp_target || + ctx->media_target_host != previous_media_target_host || + ctx->media_port != previous_media_port || + ctx->rudp_video_bitrate_kbps != previous_rudp_video_bitrate_kbps || + ctx->rudp_latency_budget_ms != previous_rudp_latency_budget_ms || + ctx->rudp_audio_reorder_ms != previous_rudp_audio_reorder_ms || + ctx->video_source_id != previous_video_source_id || + ctx->audio_source_id != previous_audio_source_id) { + ctx->last_requested_stream_id.clear(); + ctx->last_requested_video_source_id.clear(); + ctx->last_requested_audio_source_id.clear(); + } + + refresh_catalog(ctx); + reconcile_selected_stream(ctx); +} + +static void muninn_update_combined(void *data, obs_data_t *settings) +{ + muninn_update_kind(data, settings, MuninnSourceKind::Combined); +} + +static void muninn_update_video(void *data, obs_data_t *settings) +{ + muninn_update_kind(data, settings, MuninnSourceKind::VideoOnly); +} + +static void muninn_update_audio(void *data, obs_data_t *settings) +{ + muninn_update_kind(data, settings, MuninnSourceKind::AudioOnly); +} + +static void *muninn_create_kind(obs_data_t *settings, obs_source_t *source, MuninnSourceKind kind) +{ + auto *ctx = new MuninnSource(); + ctx->kind = kind; + ctx->source = source; + ctx->bridge = std::make_unique(); + ctx->catalog_discovery = std::make_unique(); + ctx->catalog_discovery->start(); + if (source_wants_audio(ctx)) { + start_audio_worker(ctx); + } + muninn_update_kind(ctx, settings, kind); + return ctx; +} + +static void *muninn_create_combined(obs_data_t *settings, obs_source_t *source) +{ + return muninn_create_kind(settings, source, MuninnSourceKind::Combined); +} + +static void *muninn_create_video(obs_data_t *settings, obs_source_t *source) +{ + return muninn_create_kind(settings, source, MuninnSourceKind::VideoOnly); +} + +static void *muninn_create_audio(obs_data_t *settings, obs_source_t *source) +{ + return muninn_create_kind(settings, source, MuninnSourceKind::AudioOnly); +} + +static void muninn_destroy(void *data) +{ + auto *ctx = static_cast(data); + release_media(ctx); + if (ctx->catalog_discovery) { + ctx->catalog_discovery->stop(); + } + stop_audio_worker(ctx); + delete ctx; +} + +static uint32_t muninn_width(void *data) +{ + auto *ctx = static_cast(data); + return ctx->media ? obs_source_get_width(ctx->media) : 0; +} + +static uint32_t muninn_height(void *data) +{ + auto *ctx = static_cast(data); + return ctx->media ? obs_source_get_height(ctx->media) : 0; +} + +static void muninn_defaults_kind(obs_data_t *settings, MuninnSourceKind kind) +{ + obs_data_set_default_string(settings, "store_path", MuninnStorePath); + obs_data_set_default_string(settings, "command_exe_path", MuninnCommandExePath); + obs_data_set_default_string(settings, "command_rudp_target", MuninnCommandRudpTarget); + obs_data_set_default_string(settings, "host_id", MuninnDefaultHostId); + obs_data_set_default_string(settings, "media_target_host", MuninnDefaultMediaTargetHost); + obs_data_set_default_int(settings, "media_port", MuninnDefaultMediaPort); + obs_data_set_default_int(settings, "rudp_video_bitrate_kbps", MuninnDefaultRudpVideoBitrateKbps); + obs_data_set_default_int(settings, "rudp_latency_budget_ms", MuninnDefaultRudpLatencyBudgetMs); + obs_data_set_default_int(settings, "rudp_audio_reorder_ms", MuninnDefaultRudpAudioReorderMs); + obs_data_set_default_string(settings, "stream_id", ""); + obs_data_set_default_string( + settings, + "video_source_id", + kind == MuninnSourceKind::AudioOnly ? MuninnDisabledVideoSourceId : "display:0"); + obs_data_set_default_string( + settings, + "audio_source_id", + kind == MuninnSourceKind::VideoOnly ? MuninnDisabledAudioSourceId : "wasapi-loopback:Realtek"); +} + +static void muninn_defaults_combined(obs_data_t *settings) +{ + muninn_defaults_kind(settings, MuninnSourceKind::Combined); +} + +static void muninn_defaults_video(obs_data_t *settings) +{ + muninn_defaults_kind(settings, MuninnSourceKind::VideoOnly); +} + +static void muninn_defaults_audio(obs_data_t *settings) +{ + muninn_defaults_kind(settings, MuninnSourceKind::AudioOnly); +} + +static obs_properties_t *muninn_properties_kind(void *data, MuninnSourceKind kind) +{ + auto *ctx = static_cast(data); + const auto store_path = ctx ? ctx->store_path : std::string(MuninnStorePath); + + obs_properties_t *props = obs_properties_create(); + obs_property_t *streams = + obs_properties_add_list(props, "stream_id", obs_module_text("MuninnStream.Stream"), OBS_COMBO_TYPE_LIST, OBS_COMBO_FORMAT_STRING); + if (ctx) { + refresh_catalog(ctx); + populate_stream_list_from_options(streams, ctx->catalog_options); + obs_property_set_modified_callback2(streams, muninn_stream_selection_modified, ctx); + } else { + populate_stream_list(streams, store_path); + } + if (kind != MuninnSourceKind::AudioOnly) { + obs_property_t *video_sources = + obs_properties_add_list(props, "video_source_id", obs_module_text("MuninnStream.Video"), OBS_COMBO_TYPE_LIST, OBS_COMBO_FORMAT_STRING); + if (ctx) { + populate_source_list(video_sources, catalog_video_sources(ctx)); + } else { + populate_source_list(video_sources, {{"display:0", "Display 1"}}); + } + } + if (kind != MuninnSourceKind::VideoOnly) { + obs_property_t *audio_sources = + obs_properties_add_list(props, "audio_source_id", obs_module_text("MuninnStream.Audio"), OBS_COMBO_TYPE_LIST, OBS_COMBO_FORMAT_STRING); + if (ctx) { + populate_source_list(audio_sources, catalog_audio_sources(ctx)); + } else { + populate_source_list(audio_sources, {{"wasapi-loopback:Realtek", "Realtek loopback"}}); + } + } + if (kind != MuninnSourceKind::AudioOnly) { + obs_properties_add_int( + props, + "rudp_video_bitrate_kbps", + obs_module_text("MuninnStream.VideoBitrateKbps"), + 1000, + MuninnMaxRudpVideoBitrateKbps, + 500); + obs_properties_add_int( + props, + "rudp_latency_budget_ms", + obs_module_text("MuninnStream.VideoLatencyMs"), + 100, + MuninnMaxRudpLatencyBudgetMs, + 50); + } + if (kind != MuninnSourceKind::VideoOnly) { + obs_properties_add_int( + props, + "rudp_audio_reorder_ms", + obs_module_text("MuninnStream.AudioReorderMs"), + 0, + MuninnMaxRudpAudioReorderMs, + 25); + } + obs_properties_add_button(props, "refresh_streams", obs_module_text("MuninnStream.Refresh"), muninn_refresh_streams); + return props; +} + +static obs_properties_t *muninn_properties_combined(void *data) +{ + return muninn_properties_kind(data, MuninnSourceKind::Combined); +} + +static obs_properties_t *muninn_properties_video(void *data) +{ + return muninn_properties_kind(data, MuninnSourceKind::VideoOnly); +} + +static obs_properties_t *muninn_properties_audio(void *data) +{ + return muninn_properties_kind(data, MuninnSourceKind::AudioOnly); +} + +static void muninn_video_tick(void *data, float) +{ + auto *ctx = static_cast(data); + if (!ctx) { + return; + } + if (std::chrono::steady_clock::now() >= ctx->next_catalog_refresh) { + refresh_catalog(ctx); + } + reconcile_selected_stream(ctx); +} + +static void muninn_render(void *data, gs_effect_t *) +{ + auto *ctx = static_cast(data); + if (ctx->media) { + obs_source_video_render(ctx->media); + } +} + +static void muninn_child_audio(void *param, obs_source_t *, const struct audio_data *audio_data, bool muted) +{ + auto *ctx = static_cast(param); + if (!ctx || !ctx->source || !audio_data || muted || audio_data->frames == 0) { + return; + } + + struct obs_audio_info2 audio_info = {}; + if (!obs_get_audio_info2(&audio_info)) { + return; + } + + const float *planes[MAX_AV_PLANES] = {}; + for (size_t plane = 0; plane < MAX_AV_PLANES; ++plane) { + if (!audio_data->data[plane]) { + continue; + } + planes[plane] = reinterpret_cast(audio_data->data[plane]); + } + queue_planar_audio(ctx, planes, audio_data->frames, audio_info.speakers, audio_info.samples_per_sec); +} + +static void muninn_enum_active_sources(void *data, obs_source_enum_proc_t enum_callback, void *param) +{ + auto *ctx = static_cast(data); + if (ctx->media) { + enum_callback(ctx->source, ctx->media, param); + } + if (ctx->audio_media) { + enum_callback(ctx->source, ctx->audio_media, param); + } +} + +static obs_source_info muninn_source_info = { + .id = "muninn_stream_source", + .type = OBS_SOURCE_TYPE_INPUT, + .output_flags = OBS_SOURCE_VIDEO | OBS_SOURCE_AUDIO, + .get_name = muninn_get_name_combined, + .create = muninn_create_combined, + .destroy = muninn_destroy, + .get_width = muninn_width, + .get_height = muninn_height, + .get_defaults = muninn_defaults_combined, + .get_properties = muninn_properties_combined, + .update = muninn_update_combined, + .video_tick = muninn_video_tick, + .video_render = muninn_render, + .enum_active_sources = muninn_enum_active_sources, +}; + +static obs_source_info muninn_video_source_info = { + .id = "muninn_video_source", + .type = OBS_SOURCE_TYPE_INPUT, + .output_flags = OBS_SOURCE_VIDEO, + .get_name = muninn_get_name_video, + .create = muninn_create_video, + .destroy = muninn_destroy, + .get_width = muninn_width, + .get_height = muninn_height, + .get_defaults = muninn_defaults_video, + .get_properties = muninn_properties_video, + .update = muninn_update_video, + .video_tick = muninn_video_tick, + .video_render = muninn_render, + .enum_active_sources = muninn_enum_active_sources, +}; + +static obs_source_info muninn_audio_source_info = { + .id = "muninn_audio_source", + .type = OBS_SOURCE_TYPE_INPUT, + .output_flags = OBS_SOURCE_AUDIO, + .get_name = muninn_get_name_audio, + .create = muninn_create_audio, + .destroy = muninn_destroy, + .get_width = muninn_width, + .get_height = muninn_height, + .get_defaults = muninn_defaults_audio, + .get_properties = muninn_properties_audio, + .update = muninn_update_audio, + .video_tick = muninn_video_tick, + .video_render = muninn_render, + .enum_active_sources = muninn_enum_active_sources, +}; + +} // namespace + +void mimir_register_muninn_source() +{ + obs_register_source(&muninn_source_info); + obs_register_source(&muninn_video_source_info); + obs_register_source(&muninn_audio_source_info); +} diff --git a/native/obs_stem_source/mimir_obs_program_texture_source.cpp b/native/obs_stem_source/mimir_obs_program_texture_source.cpp new file mode 100644 index 0000000..6cb2cbb --- /dev/null +++ b/native/obs_stem_source/mimir_obs_program_texture_source.cpp @@ -0,0 +1,628 @@ +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +using Microsoft::WRL::ComPtr; + +namespace { + +struct MimirProgramTextureSource { + obs_source_t *source = nullptr; + std::string texture_name = "Global\\MimirFensalirProgramTexture"; + std::string fence_name = "Global\\MimirFensalirProgramFence"; + std::string consumer_fence_name = "Global\\MimirObsProgramConsumerFence"; + uint32_t ring_count = 1; + uint32_t width = 0; + uint32_t height = 0; + uint64_t fence_value = 0; + uint64_t last_producer_fence_value = 0; + uint64_t next_retry_ns = 0; + uint64_t next_failure_log_ns = 0; + HANDLE bridge_shared_handle = nullptr; + HANDLE consumer_fence_shared_handle = nullptr; + HANDLE fence_event = nullptr; + gs_texture_t *obs_texture = nullptr; + ComPtr d3d12_device; + ComPtr command_queue; + ComPtr command_allocator; + ComPtr command_list; + ComPtr fence; + ComPtr producer_fence; + ComPtr consumer_fence; + std::vector> source_textures; + ComPtr bridge_texture_d3d12; + ComPtr d3d11_device; + ComPtr d3d11_context; + ComPtr bridge_texture_d3d11; +}; + +static void log_retry_failure(MimirProgramTextureSource *ctx, const char *message, HRESULT hr = S_OK) +{ + const uint64_t now = os_gettime_ns(); + if (now < ctx->next_failure_log_ns) { + return; + } + + ctx->next_failure_log_ns = now + 2000000000ULL; + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: %s: 0x%08lx", message, static_cast(hr)); + } else { + blog(LOG_WARNING, "Mimir OBS program texture: %s", message); + } +} + +static std::wstring utf8_to_wide(const std::string &value) +{ + if (value.empty()) { + return {}; + } + + const int count = MultiByteToWideChar(CP_UTF8, 0, value.c_str(), -1, nullptr, 0); + if (count <= 0) { + return {}; + } + + std::wstring wide(static_cast(count), L'\0'); + MultiByteToWideChar(CP_UTF8, 0, value.c_str(), -1, wide.data(), count); + if (!wide.empty() && wide.back() == L'\0') { + wide.pop_back(); + } + + return wide; +} + +static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback) +{ + const char *value = obs_data_get_string(settings, key); + return value && *value ? std::string(value) : std::string(fallback); +} + +static uint32_t clamp_ring_count(long long value) +{ + if (value < 1) { + return 1; + } + + if (value > 4) { + return 4; + } + + return static_cast(value); +} + +static std::string texture_name_for_slot(const MimirProgramTextureSource *ctx, uint32_t slot) +{ + if (ctx->ring_count <= 1) { + return ctx->texture_name; + } + + return ctx->texture_name + "." + std::to_string(slot); +} + +static void release_bridge(MimirProgramTextureSource *ctx) +{ + if (ctx->obs_texture) { + gs_texture_destroy(ctx->obs_texture); + ctx->obs_texture = nullptr; + } + + ctx->bridge_texture_d3d12.Reset(); + ctx->bridge_texture_d3d11.Reset(); + ctx->source_textures.clear(); + ctx->fence.Reset(); + ctx->producer_fence.Reset(); + ctx->consumer_fence.Reset(); + ctx->command_list.Reset(); + ctx->command_allocator.Reset(); + ctx->command_queue.Reset(); + ctx->d3d12_device.Reset(); + ctx->d3d11_context.Reset(); + ctx->d3d11_device.Reset(); + ctx->width = 0; + ctx->height = 0; + ctx->fence_value = 0; + ctx->last_producer_fence_value = 0; + + if (ctx->bridge_shared_handle) { + CloseHandle(ctx->bridge_shared_handle); + ctx->bridge_shared_handle = nullptr; + } + + if (ctx->consumer_fence_shared_handle) { + CloseHandle(ctx->consumer_fence_shared_handle); + ctx->consumer_fence_shared_handle = nullptr; + } + + if (ctx->fence_event) { + CloseHandle(ctx->fence_event); + ctx->fence_event = nullptr; + } +} + +static bool wait_for_copy(MimirProgramTextureSource *ctx, uint64_t consumed_producer_value) +{ + if (ctx->consumer_fence && consumed_producer_value > 0) { + HRESULT consumer_hr = ctx->command_queue->Signal(ctx->consumer_fence.Get(), consumed_producer_value); + if (FAILED(consumer_hr)) { + blog(LOG_WARNING, + "Mimir OBS program texture: consumer fence Signal failed: 0x%08lx", + static_cast(consumer_hr)); + } + } + + const uint64_t value = ++ctx->fence_value; + HRESULT hr = ctx->command_queue->Signal(ctx->fence.Get(), value); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: D3D12 Signal failed: 0x%08lx", static_cast(hr)); + return false; + } + + if (ctx->fence->GetCompletedValue() >= value) { + return true; + } + + hr = ctx->fence->SetEventOnCompletion(value, ctx->fence_event); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: SetEventOnCompletion failed: 0x%08lx", static_cast(hr)); + return false; + } + + WaitForSingleObject(ctx->fence_event, 100); + return ctx->fence->GetCompletedValue() >= value; +} + +static bool create_devices(MimirProgramTextureSource *ctx) +{ + HRESULT hr = D3D12CreateDevice(nullptr, D3D_FEATURE_LEVEL_11_0, IID_PPV_ARGS(&ctx->d3d12_device)); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: D3D12CreateDevice failed: 0x%08lx", static_cast(hr)); + return false; + } + + D3D12_COMMAND_QUEUE_DESC queue_desc = {}; + queue_desc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT; + hr = ctx->d3d12_device->CreateCommandQueue(&queue_desc, IID_PPV_ARGS(&ctx->command_queue)); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandQueue failed: 0x%08lx", static_cast(hr)); + return false; + } + + hr = ctx->d3d12_device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, IID_PPV_ARGS(&ctx->command_allocator)); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandAllocator failed: 0x%08lx", static_cast(hr)); + return false; + } + + hr = ctx->d3d12_device->CreateCommandList( + 0, + D3D12_COMMAND_LIST_TYPE_DIRECT, + ctx->command_allocator.Get(), + nullptr, + IID_PPV_ARGS(&ctx->command_list)); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: CreateCommandList failed: 0x%08lx", static_cast(hr)); + return false; + } + + ctx->command_list->Close(); + + hr = ctx->d3d12_device->CreateFence(0, D3D12_FENCE_FLAG_NONE, IID_PPV_ARGS(&ctx->fence)); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: CreateFence failed: 0x%08lx", static_cast(hr)); + return false; + } + + ctx->fence_event = CreateEventW(nullptr, FALSE, FALSE, nullptr); + if (!ctx->fence_event) { + blog(LOG_WARNING, "Mimir OBS program texture: CreateEvent failed: %lu", GetLastError()); + return false; + } + + UINT flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT; + D3D_FEATURE_LEVEL feature_levels[] = {D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0}; + D3D_FEATURE_LEVEL selected = D3D_FEATURE_LEVEL_11_0; + hr = D3D11CreateDevice( + nullptr, + D3D_DRIVER_TYPE_HARDWARE, + nullptr, + flags, + feature_levels, + 2, + D3D11_SDK_VERSION, + &ctx->d3d11_device, + &selected, + &ctx->d3d11_context); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: D3D11CreateDevice failed: 0x%08lx", static_cast(hr)); + return false; + } + + return true; +} + +static void create_consumer_fence(MimirProgramTextureSource *ctx) +{ + if (ctx->consumer_fence_name.empty()) { + return; + } + + HRESULT hr = ctx->d3d12_device->CreateFence(0, D3D12_FENCE_FLAG_SHARED, IID_PPV_ARGS(&ctx->consumer_fence)); + if (FAILED(hr)) { + log_retry_failure(ctx, "CreateFence consumer fence failed", hr); + return; + } + + const auto wide_name = utf8_to_wide(ctx->consumer_fence_name); + if (wide_name.empty()) { + log_retry_failure(ctx, "consumer fence name is empty or invalid UTF-8"); + ctx->consumer_fence.Reset(); + return; + } + + hr = ctx->d3d12_device->CreateSharedHandle( + ctx->consumer_fence.Get(), + nullptr, + GENERIC_ALL, + wide_name.c_str(), + &ctx->consumer_fence_shared_handle); + if (FAILED(hr)) { + log_retry_failure(ctx, "CreateSharedHandle consumer fence failed", hr); + ctx->consumer_fence.Reset(); + return; + } + + blog(LOG_INFO, "Mimir OBS program texture consumer fence published %s", ctx->consumer_fence_name.c_str()); +} + +static bool open_source_texture(MimirProgramTextureSource *ctx) +{ + ctx->source_textures.clear(); + ctx->source_textures.reserve(ctx->ring_count); + for (uint32_t slot = 0; slot < ctx->ring_count; ++slot) { + const auto slot_name = texture_name_for_slot(ctx, slot); + const auto wide_name = utf8_to_wide(slot_name); + if (wide_name.empty()) { + log_retry_failure(ctx, "shared texture name is empty or invalid UTF-8"); + return false; + } + + HANDLE source_handle = nullptr; + HRESULT hr = ctx->d3d12_device->OpenSharedHandleByName(wide_name.c_str(), GENERIC_ALL, &source_handle); + if (FAILED(hr)) { + log_retry_failure(ctx, "OpenSharedHandleByName failed", hr); + return false; + } + + ComPtr source_texture; + hr = ctx->d3d12_device->OpenSharedHandle(source_handle, IID_PPV_ARGS(&source_texture)); + CloseHandle(source_handle); + if (FAILED(hr)) { + log_retry_failure(ctx, "OpenSharedHandle failed", hr); + return false; + } + + const D3D12_RESOURCE_DESC desc = source_texture->GetDesc(); + if (desc.Dimension != D3D12_RESOURCE_DIMENSION_TEXTURE2D || desc.Format != DXGI_FORMAT_B8G8R8A8_UNORM) { + log_retry_failure(ctx, "expected BGRA8 D3D12 Texture2D"); + return false; + } + + const uint32_t slot_width = static_cast(desc.Width); + const uint32_t slot_height = desc.Height; + if (slot_width == 0 || slot_height == 0) { + log_retry_failure(ctx, "shared texture dimensions are empty"); + return false; + } + + if (slot == 0) { + ctx->width = slot_width; + ctx->height = slot_height; + } else if (slot_width != ctx->width || slot_height != ctx->height) { + log_retry_failure(ctx, "program output ring texture dimensions do not match"); + return false; + } + + ctx->source_textures.push_back(std::move(source_texture)); + } + + return ctx->source_textures.size() == ctx->ring_count; +} + +static bool open_producer_fence(MimirProgramTextureSource *ctx) +{ + const auto wide_name = utf8_to_wide(ctx->fence_name); + if (wide_name.empty()) { + return true; + } + + HANDLE fence_handle = nullptr; + HRESULT hr = ctx->d3d12_device->OpenSharedHandleByName(wide_name.c_str(), GENERIC_ALL, &fence_handle); + if (FAILED(hr)) { + log_retry_failure(ctx, "OpenSharedHandleByName producer fence failed", hr); + return true; + } + + hr = ctx->d3d12_device->OpenSharedHandle(fence_handle, IID_PPV_ARGS(&ctx->producer_fence)); + CloseHandle(fence_handle); + if (FAILED(hr)) { + log_retry_failure(ctx, "OpenSharedHandle producer fence failed", hr); + ctx->producer_fence.Reset(); + return true; + } + + ctx->last_producer_fence_value = ctx->producer_fence->GetCompletedValue(); + if (ctx->last_producer_fence_value == UINT64_MAX) { + log_retry_failure(ctx, "producer fence opened in device-removed state"); + ctx->producer_fence.Reset(); + return false; + } + + blog(LOG_INFO, + "Mimir OBS program texture producer fence opened %s value=%llu", + ctx->fence_name.c_str(), + static_cast(ctx->last_producer_fence_value)); + return true; +} + +static bool read_producer_fence(MimirProgramTextureSource *ctx, uint64_t *completed) +{ + *completed = 0; + if (!ctx->producer_fence) { + return true; + } + + *completed = ctx->producer_fence->GetCompletedValue(); + if (*completed == UINT64_MAX) { + log_retry_failure(ctx, "producer fence reported device removal"); + return false; + } + + if (*completed > ctx->last_producer_fence_value) { + ctx->last_producer_fence_value = *completed; + } + + return true; +} + +static bool create_obs_bridge_texture(MimirProgramTextureSource *ctx) +{ + D3D11_TEXTURE2D_DESC desc = {}; + desc.Width = ctx->width; + desc.Height = ctx->height; + desc.MipLevels = 1; + desc.ArraySize = 1; + desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM; + desc.SampleDesc.Count = 1; + desc.Usage = D3D11_USAGE_DEFAULT; + desc.BindFlags = D3D11_BIND_SHADER_RESOURCE; + desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED; + + HRESULT hr = ctx->d3d11_device->CreateTexture2D(&desc, nullptr, &ctx->bridge_texture_d3d11); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: CreateTexture2D bridge failed: 0x%08lx", static_cast(hr)); + return false; + } + + ComPtr legacy_resource; + hr = ctx->bridge_texture_d3d11.As(&legacy_resource); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: bridge texture is not IDXGIResource: 0x%08lx", static_cast(hr)); + return false; + } + + hr = legacy_resource->GetSharedHandle(&ctx->bridge_shared_handle); + if (FAILED(hr) || !ctx->bridge_shared_handle) { + blog(LOG_WARNING, "Mimir OBS program texture: GetSharedHandle bridge failed: 0x%08lx", static_cast(hr)); + return false; + } + + hr = ctx->d3d12_device->OpenSharedHandle(ctx->bridge_shared_handle, IID_PPV_ARGS(&ctx->bridge_texture_d3d12)); + if (FAILED(hr)) { + blog(LOG_WARNING, "Mimir OBS program texture: D3D12 open legacy bridge handle failed: 0x%08lx", static_cast(hr)); + return false; + } + + ctx->obs_texture = gs_texture_open_shared(static_cast(reinterpret_cast(ctx->bridge_shared_handle))); + if (!ctx->obs_texture) { + blog(LOG_WARNING, "Mimir OBS program texture: OBS failed to open legacy bridge shared texture"); + return false; + } + + return true; +} + +static bool ensure_bridge(MimirProgramTextureSource *ctx) +{ + if (ctx->obs_texture) { + return true; + } + + const uint64_t now = os_gettime_ns(); + if (now < ctx->next_retry_ns) { + return false; + } + + ctx->next_retry_ns = now + 1000000000ULL; + release_bridge(ctx); + + if (!create_devices(ctx)) { + release_bridge(ctx); + return false; + } + + create_consumer_fence(ctx); + if (!open_source_texture(ctx) || !create_obs_bridge_texture(ctx)) { + release_bridge(ctx); + return false; + } + + if (!open_producer_fence(ctx)) { + release_bridge(ctx); + return false; + } + + blog(LOG_INFO, "Mimir OBS program texture source opened %s (%ux%u ring=%u)", + ctx->texture_name.c_str(), + ctx->width, + ctx->height, + ctx->ring_count); + return true; +} + +static bool copy_source_to_bridge(MimirProgramTextureSource *ctx) +{ + HRESULT hr = ctx->command_allocator->Reset(); + if (FAILED(hr)) { + return false; + } + + uint64_t completed_producer_value = 0; + if (!read_producer_fence(ctx, &completed_producer_value)) { + release_bridge(ctx); + return false; + } + + const uint32_t source_slot = ctx->source_textures.empty() + ? 0 + : static_cast(completed_producer_value % ctx->source_textures.size()); + ID3D12Resource *source_texture = ctx->source_textures[source_slot].Get(); + + hr = ctx->command_list->Reset(ctx->command_allocator.Get(), nullptr); + if (FAILED(hr)) { + return false; + } + + D3D12_RESOURCE_BARRIER barriers[2] = {}; + barriers[0].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION; + barriers[0].Transition.pResource = source_texture; + barriers[0].Transition.StateBefore = D3D12_RESOURCE_STATE_COMMON; + barriers[0].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_SOURCE; + barriers[0].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES; + barriers[1].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION; + barriers[1].Transition.pResource = ctx->bridge_texture_d3d12.Get(); + barriers[1].Transition.StateBefore = D3D12_RESOURCE_STATE_COMMON; + barriers[1].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_DEST; + barriers[1].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES; + ctx->command_list->ResourceBarrier(2, barriers); + ctx->command_list->CopyResource(ctx->bridge_texture_d3d12.Get(), source_texture); + + std::swap(barriers[0].Transition.StateBefore, barriers[0].Transition.StateAfter); + std::swap(barriers[1].Transition.StateBefore, barriers[1].Transition.StateAfter); + ctx->command_list->ResourceBarrier(2, barriers); + + hr = ctx->command_list->Close(); + if (FAILED(hr)) { + return false; + } + + ID3D12CommandList *lists[] = {ctx->command_list.Get()}; + ctx->command_queue->ExecuteCommandLists(1, lists); + return wait_for_copy(ctx, completed_producer_value); +} + +static const char *mimir_program_texture_get_name(void *) +{ + return obs_module_text("MimirProgramTexture"); +} + +static void mimir_program_texture_update(void *data, obs_data_t *settings) +{ + auto *ctx = static_cast(data); + ctx->texture_name = obs_string(settings, "texture_name", "Global\\MimirFensalirProgramTexture"); + ctx->fence_name = obs_string(settings, "fence_name", "Global\\MimirFensalirProgramFence"); + ctx->consumer_fence_name = obs_string(settings, "consumer_fence_name", "Global\\MimirObsProgramConsumerFence"); + ctx->ring_count = clamp_ring_count(obs_data_get_int(settings, "ring_count")); + release_bridge(ctx); +} + +static void *mimir_program_texture_create(obs_data_t *settings, obs_source_t *source) +{ + auto *ctx = new MimirProgramTextureSource(); + ctx->source = source; + mimir_program_texture_update(ctx, settings); + return ctx; +} + +static void mimir_program_texture_destroy(void *data) +{ + auto *ctx = static_cast(data); + release_bridge(ctx); + delete ctx; +} + +static uint32_t mimir_program_texture_width(void *data) +{ + auto *ctx = static_cast(data); + ensure_bridge(ctx); + return ctx->width; +} + +static uint32_t mimir_program_texture_height(void *data) +{ + auto *ctx = static_cast(data); + ensure_bridge(ctx); + return ctx->height; +} + +static void mimir_program_texture_defaults(obs_data_t *settings) +{ + obs_data_set_default_string(settings, "texture_name", "Global\\MimirFensalirProgramTexture"); + obs_data_set_default_string(settings, "fence_name", "Global\\MimirFensalirProgramFence"); + obs_data_set_default_string(settings, "consumer_fence_name", "Global\\MimirObsProgramConsumerFence"); + obs_data_set_default_int(settings, "ring_count", 1); +} + +static obs_properties_t *mimir_program_texture_properties(void *) +{ + obs_properties_t *props = obs_properties_create(); + obs_properties_add_text(props, "texture_name", "Shared D3D12 texture name", OBS_TEXT_DEFAULT); + obs_properties_add_text(props, "fence_name", "Shared D3D12 fence name", OBS_TEXT_DEFAULT); + obs_properties_add_text(props, "consumer_fence_name", "Consumer D3D12 fence name", OBS_TEXT_DEFAULT); + obs_properties_add_int(props, "ring_count", "Texture ring slots", 1, 4, 1); + return props; +} + +static void mimir_program_texture_render(void *data, gs_effect_t *) +{ + auto *ctx = static_cast(data); + if (!ensure_bridge(ctx)) { + return; + } + + if (copy_source_to_bridge(ctx)) { + obs_source_draw(ctx->obs_texture, 0, 0, ctx->width, ctx->height, false); + } +} + +static obs_source_info mimir_program_texture_source_info = { + .id = "mimir_program_texture_source", + .type = OBS_SOURCE_TYPE_INPUT, + .output_flags = OBS_SOURCE_VIDEO, + .get_name = mimir_program_texture_get_name, + .create = mimir_program_texture_create, + .destroy = mimir_program_texture_destroy, + .get_width = mimir_program_texture_width, + .get_height = mimir_program_texture_height, + .get_defaults = mimir_program_texture_defaults, + .get_properties = mimir_program_texture_properties, + .update = mimir_program_texture_update, + .video_render = mimir_program_texture_render, +}; + +} // namespace + +void mimir_register_program_texture_source() +{ + obs_register_source(&mimir_program_texture_source_info); +} diff --git a/native/obs_stem_source/mimir_obs_stem_ipc.h b/native/obs_stem_source/mimir_obs_stem_ipc.h new file mode 100644 index 0000000..6a085ac --- /dev/null +++ b/native/obs_stem_source/mimir_obs_stem_ipc.h @@ -0,0 +1,43 @@ +#pragma once + +#include + +constexpr uint64_t MIMIR_OBS_STEM_MAGIC = 0x4D545352494D494Dull; +constexpr int MIMIR_OBS_STEM_VERSION = 1; +constexpr int MIMIR_OBS_STEM_MAX_STEMS = 8; +constexpr int MIMIR_OBS_STEM_MAX_FRAMES = 4096; +constexpr int MIMIR_OBS_STEM_ID_BYTES = 64; +constexpr int MIMIR_OBS_STEM_DISPLAY_BYTES = 64; +constexpr int MIMIR_OBS_STEM_SOURCE_BYTES = 64; +constexpr int MIMIR_OBS_STEM_HEADER_BYTES = 64; +constexpr int MIMIR_OBS_STEM_RECORD_BYTES = 224; +constexpr int MIMIR_OBS_STEM_MAP_BYTES = + MIMIR_OBS_STEM_HEADER_BYTES + + (MIMIR_OBS_STEM_MAX_STEMS * MIMIR_OBS_STEM_RECORD_BYTES) + + (MIMIR_OBS_STEM_MAX_STEMS * MIMIR_OBS_STEM_MAX_FRAMES * static_cast(sizeof(float))); + +struct MimirObsStemHeader { + uint64_t magic; + int32_t version; + uint32_t generation; + int32_t stem_count; + int32_t max_stems; + int32_t max_frames; + int32_t header_bytes; + int32_t record_bytes; + int32_t map_bytes; +}; + +struct MimirObsStemRecord { + int32_t channel_index; + int32_t configured; + int32_t sample_rate; + int32_t frame_count; + int32_t sample_offset; + int32_t reserved0; + int32_t reserved1; + int32_t reserved2; + char stem_id[MIMIR_OBS_STEM_ID_BYTES]; + char display_name[MIMIR_OBS_STEM_DISPLAY_BYTES]; + char source_id[MIMIR_OBS_STEM_SOURCE_BYTES]; +}; diff --git a/native/obs_stem_source/mimir_obs_stem_source.cpp b/native/obs_stem_source/mimir_obs_stem_source.cpp new file mode 100644 index 0000000..5ebcbd1 --- /dev/null +++ b/native/obs_stem_source/mimir_obs_stem_source.cpp @@ -0,0 +1,219 @@ +#include +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include "mimir_obs_stem_ipc.h" + +OBS_DECLARE_MODULE() +OBS_MODULE_USE_DEFAULT_LOCALE("mimir-obs-stem-source", "en-US") + +void mimir_register_program_texture_source(); +void mimir_register_muninn_source(); + +struct MimirStemSource { + obs_source_t *source = nullptr; + std::string map_name = "Local\\MimirObsStemBus"; + std::string stem_id = "aligned_source_0"; + HANDLE mapping = nullptr; + const uint8_t *view = nullptr; + std::thread worker; + std::atomic_bool running = false; + uint32_t last_generation = 0; +}; + +static std::string obs_string(obs_data_t *settings, const char *key, const char *fallback) +{ + const char *value = obs_data_get_string(settings, key); + return value && *value ? std::string(value) : std::string(fallback); +} + +static void close_mapping(MimirStemSource *ctx) +{ + if (ctx->view) { + UnmapViewOfFile(ctx->view); + ctx->view = nullptr; + } + + if (ctx->mapping) { + CloseHandle(ctx->mapping); + ctx->mapping = nullptr; + } +} + +static bool open_mapping(MimirStemSource *ctx) +{ + if (ctx->mapping) { + return true; + } + + ctx->mapping = OpenFileMappingA(FILE_MAP_READ, FALSE, ctx->map_name.c_str()); + if (!ctx->mapping) { + return false; + } + + ctx->view = static_cast(MapViewOfFile(ctx->mapping, FILE_MAP_READ, 0, 0, MIMIR_OBS_STEM_MAP_BYTES)); + if (!ctx->view) { + close_mapping(ctx); + return false; + } + + return true; +} + +static const MimirObsStemRecord *find_record(const uint8_t *view, const std::string &stem_id, int stem_count) +{ + const auto count = std::min(stem_count, MIMIR_OBS_STEM_MAX_STEMS); + for (int index = 0; index < count; ++index) { + const auto *record = reinterpret_cast(view + MIMIR_OBS_STEM_HEADER_BYTES + (index * MIMIR_OBS_STEM_RECORD_BYTES)); + if (record->frame_count <= 0 || record->sample_rate <= 0 || record->sample_offset <= 0) { + continue; + } + + if (stem_id == record->stem_id) { + return record; + } + } + + return nullptr; +} + +static void publish_if_available(MimirStemSource *ctx) +{ + if (!open_mapping(ctx)) { + return; + } + + const auto *header = reinterpret_cast(ctx->view); + const auto generation_a = header->generation; + if ((generation_a & 1u) != 0 || generation_a == ctx->last_generation) { + return; + } + + if (header->magic != MIMIR_OBS_STEM_MAGIC || + header->version != MIMIR_OBS_STEM_VERSION || + header->map_bytes != MIMIR_OBS_STEM_MAP_BYTES) { + return; + } + + const auto *record = find_record(ctx->view, ctx->stem_id, header->stem_count); + if (!record) { + return; + } + + const auto frame_count = std::min(record->frame_count, MIMIR_OBS_STEM_MAX_FRAMES); + const auto sample_rate = record->sample_rate; + const auto byte_count = static_cast(frame_count) * sizeof(float); + if (record->sample_offset < 0 || record->sample_offset + static_cast(byte_count) > MIMIR_OBS_STEM_MAP_BYTES) { + return; + } + + std::vector samples(frame_count); + std::memcpy(samples.data(), ctx->view + record->sample_offset, byte_count); + + const auto generation_b = header->generation; + if (generation_a != generation_b || (generation_b & 1u) != 0) { + return; + } + + ctx->last_generation = generation_b; + uint8_t *planes[MAX_AV_PLANES] = {}; + planes[0] = reinterpret_cast(samples.data()); + obs_source_audio audio = {}; + audio.data[0] = planes[0]; + audio.frames = static_cast(frame_count); + audio.speakers = SPEAKERS_MONO; + audio.samples_per_sec = static_cast(sample_rate); + audio.format = AUDIO_FORMAT_FLOAT_PLANAR; + audio.timestamp = os_gettime_ns(); + obs_source_output_audio(ctx->source, &audio); +} + +static void worker_loop(MimirStemSource *ctx) +{ + while (ctx->running.load()) { + publish_if_available(ctx); + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } +} + +static const char *mimir_stem_get_name(void *) +{ + return obs_module_text("MimirAudioStem"); +} + +static void mimir_stem_update(void *data, obs_data_t *settings) +{ + auto *ctx = static_cast(data); + ctx->map_name = obs_string(settings, "map_name", "Local\\MimirObsStemBus"); + ctx->stem_id = obs_string(settings, "stem_id", "aligned_source_0"); + ctx->last_generation = 0; + close_mapping(ctx); +} + +static void *mimir_stem_create(obs_data_t *settings, obs_source_t *source) +{ + auto *ctx = new MimirStemSource(); + ctx->source = source; + mimir_stem_update(ctx, settings); + ctx->running = true; + ctx->worker = std::thread(worker_loop, ctx); + return ctx; +} + +static void mimir_stem_destroy(void *data) +{ + auto *ctx = static_cast(data); + ctx->running = false; + if (ctx->worker.joinable()) { + ctx->worker.join(); + } + + close_mapping(ctx); + delete ctx; +} + +static void mimir_stem_defaults(obs_data_t *settings) +{ + obs_data_set_default_string(settings, "map_name", "Local\\MimirObsStemBus"); + obs_data_set_default_string(settings, "stem_id", "aligned_source_0"); +} + +static obs_properties_t *mimir_stem_properties(void *) +{ + obs_properties_t *props = obs_properties_create(); + obs_properties_add_text(props, "map_name", obs_module_text("MimirAudioStem.MapName"), OBS_TEXT_DEFAULT); + obs_properties_add_text(props, "stem_id", obs_module_text("MimirAudioStem.StemId"), OBS_TEXT_DEFAULT); + return props; +} + +static obs_source_info mimir_stem_source_info = { + .id = "mimir_audio_stem_source", + .type = OBS_SOURCE_TYPE_INPUT, + .output_flags = OBS_SOURCE_AUDIO, + .get_name = mimir_stem_get_name, + .create = mimir_stem_create, + .destroy = mimir_stem_destroy, + .get_defaults = mimir_stem_defaults, + .get_properties = mimir_stem_properties, + .update = mimir_stem_update, +}; + +bool obs_module_load(void) +{ + obs_register_source(&mimir_stem_source_info); + mimir_register_program_texture_source(); + mimir_register_muninn_source(); + blog( + LOG_INFO, + "Mimir OBS sources loaded: mimir_audio_stem_source, mimir_program_texture_source, " + "muninn_stream_source, muninn_video_source, muninn_audio_source"); + return true; +} diff --git a/native/obs_stem_source/muninn_media_wire.h b/native/obs_stem_source/muninn_media_wire.h new file mode 100644 index 0000000..98f99fe --- /dev/null +++ b/native/obs_stem_source/muninn_media_wire.h @@ -0,0 +1,267 @@ +#pragma once + +#include +#include +#include +#include +#include + +namespace muninn_media_wire { + +inline constexpr const char *ReceiverFeedbackSchema = "muninn.media_receiver_feedback.v1"; + +class MsgpackWriter { +public: + const std::vector &bytes() const { return bytes_; } + + void write_raw(const std::vector &value) + { + bytes_.insert(bytes_.end(), value.begin(), value.end()); + } + + void write_nil() { bytes_.push_back(0xc0); } + + void write_bool(bool value) { bytes_.push_back(value ? 0xc3 : 0xc2); } + + void write_u64(uint64_t value) + { + if (value <= 0x7f) { + bytes_.push_back(static_cast(value)); + } else if (value <= 0xff) { + bytes_.push_back(0xcc); + bytes_.push_back(static_cast(value)); + } else if (value <= 0xffff) { + bytes_.push_back(0xcd); + write_be16_value(static_cast(value)); + } else if (value <= 0xffffffffull) { + bytes_.push_back(0xce); + write_be32_value(static_cast(value)); + } else { + bytes_.push_back(0xcf); + write_be64_value(value); + } + } + + void write_i64(int64_t value) + { + if (value >= 0) { + write_u64(static_cast(value)); + return; + } + if (value >= -32) { + bytes_.push_back(static_cast(0xe0 | (value + 32))); + } else if (value >= INT8_MIN) { + bytes_.push_back(0xd0); + bytes_.push_back(static_cast(value)); + } else if (value >= INT16_MIN) { + bytes_.push_back(0xd1); + write_be16_value(static_cast(value)); + } else if (value >= INT32_MIN) { + bytes_.push_back(0xd2); + write_be32_value(static_cast(value)); + } else { + bytes_.push_back(0xd3); + write_be64_value(static_cast(value)); + } + } + + void write_string(const std::string &value) + { + const auto length = value.size(); + if (length <= 31) { + bytes_.push_back(static_cast(0xa0 | length)); + } else if (length <= 0xff) { + bytes_.push_back(0xd9); + bytes_.push_back(static_cast(length)); + } else if (length <= 0xffff) { + bytes_.push_back(0xda); + write_be16_value(static_cast(length)); + } else { + bytes_.push_back(0xdb); + write_be32_value(static_cast(length)); + } + bytes_.insert(bytes_.end(), value.begin(), value.end()); + } + + void write_binary(const std::vector &value) + { + const auto length = value.size(); + if (length <= 0xff) { + bytes_.push_back(0xc4); + bytes_.push_back(static_cast(length)); + } else if (length <= 0xffff) { + bytes_.push_back(0xc5); + write_be16_value(static_cast(length)); + } else { + bytes_.push_back(0xc6); + write_be32_value(static_cast(length)); + } + bytes_.insert(bytes_.end(), value.begin(), value.end()); + } + + void write_array_len(uint32_t length) + { + if (length <= 15) { + bytes_.push_back(static_cast(0x90 | length)); + } else if (length <= 0xffff) { + bytes_.push_back(0xdc); + write_be16_value(static_cast(length)); + } else { + bytes_.push_back(0xdd); + write_be32_value(length); + } + } + + void write_map_len(uint32_t length) + { + if (length <= 15) { + bytes_.push_back(static_cast(0x80 | length)); + } else if (length <= 0xffff) { + bytes_.push_back(0xde); + write_be16_value(static_cast(length)); + } else { + bytes_.push_back(0xdf); + write_be32_value(length); + } + } + +private: + std::vector bytes_; + + void write_be16_value(uint16_t value) + { + bytes_.push_back(static_cast((value >> 8) & 0xff)); + bytes_.push_back(static_cast(value & 0xff)); + } + + void write_be32_value(uint32_t value) + { + bytes_.push_back(static_cast((value >> 24) & 0xff)); + bytes_.push_back(static_cast((value >> 16) & 0xff)); + bytes_.push_back(static_cast((value >> 8) & 0xff)); + bytes_.push_back(static_cast(value & 0xff)); + } + + void write_be64_value(uint64_t value) + { + write_be32_value(static_cast((value >> 32) & 0xffffffffull)); + write_be32_value(static_cast(value & 0xffffffffull)); + } +}; + +inline std::string feedback_observed_at() +{ + const auto now = std::chrono::system_clock::now().time_since_epoch(); + const auto millis = std::chrono::duration_cast(now).count(); + return "unix-ms:" + std::to_string(millis); +} + +inline void write_string_array(MsgpackWriter &writer, const std::vector &values) +{ + writer.write_array_len(static_cast(values.size())); + for (const auto &value : values) { + writer.write_string(value); + } +} + +inline void write_u64_array(MsgpackWriter &writer, const std::vector &values) +{ + writer.write_array_len(static_cast(values.size())); + for (const auto value : values) { + writer.write_u64(value); + } +} + +inline std::vector encode_receiver_feedback_payload( + const std::string &stream_id, + const std::string &session_id, + uint64_t frame_id, + const std::vector &missing_chunk_keys, + bool requested_keyframe, + const std::string &observed_at) +{ + MsgpackWriter record; + record.write_array_len(11); + record.write_string(stream_id); + record.write_string(session_id); + record.write_string("starfire.obs"); + if (frame_id > 0) { + record.write_u64(frame_id - 1); + } else { + record.write_nil(); + } + write_u64_array(record, {}); + write_u64_array(record, {frame_id}); + record.write_bool(requested_keyframe); + record.write_i64(0); + record.write_i64(0); + record.write_string(observed_at); + write_string_array(record, missing_chunk_keys); + + const std::string record_key = stream_id + ":" + session_id + ":feedback:starfire.obs"; + const std::string message_id = + "muninn-media:" + std::string(ReceiverFeedbackSchema) + ":" + record_key; + + MsgpackWriter document; + document.write_map_len(9); + document.write_string("schemaId"); + document.write_string(ReceiverFeedbackSchema); + document.write_string("recordKey"); + document.write_string(record_key); + document.write_string("storedAt"); + document.write_string(observed_at); + document.write_string("payloadEncoding"); + document.write_string("messagepack"); + document.write_string("payload"); + document.write_binary(record.bytes()); + document.write_string("sourceRuntimeId"); + document.write_string("starfire"); + document.write_string("sourceAgentId"); + document.write_nil(); + document.write_string("sourceRole"); + document.write_string("mimir.obs"); + document.write_string("tags"); + write_string_array(document, {"muninn.media"}); + + MsgpackWriter wire; + wire.write_map_len(3); + wire.write_string("schemaVersion"); + wire.write_string("cultnet.document_put_raw.v0"); + wire.write_string("messageId"); + wire.write_string(message_id); + wire.write_string("document"); + wire.write_raw(document.bytes()); + return wire.bytes(); +} + +inline std::vector encode_receiver_feedback_payload( + const std::string &stream_id, + const std::string &session_id, + uint64_t frame_id, + const std::vector &missing_chunk_keys, + bool requested_keyframe) +{ + return encode_receiver_feedback_payload( + stream_id, + session_id, + frame_id, + missing_chunk_keys, + requested_keyframe, + feedback_observed_at()); +} + +inline std::vector encode_receiver_feedback_payload( + const std::string &stream_id, + const std::string &session_id, + uint64_t frame_id, + const std::vector &missing_chunk_keys) +{ + return encode_receiver_feedback_payload( + stream_id, + session_id, + frame_id, + missing_chunk_keys, + true); +} + +} // namespace muninn_media_wire diff --git a/native/obs_stem_source/muninn_rudp_ack.h b/native/obs_stem_source/muninn_rudp_ack.h new file mode 100644 index 0000000..ca80d41 --- /dev/null +++ b/native/obs_stem_source/muninn_rudp_ack.h @@ -0,0 +1,54 @@ +#pragma once + +#include +#include +#include +#include + +namespace muninn_rudp_ack { + +class AckTracker { +public: + void reset() + { + received_sequences_.clear(); + highest_received_sequence_.reset(); + } + + void remember(uint32_t sequence) + { + received_sequences_.insert(sequence); + if (!highest_received_sequence_ || sequence > *highest_received_sequence_) { + highest_received_sequence_ = sequence; + } + if (!highest_received_sequence_) { + return; + } + const uint32_t highest = *highest_received_sequence_; + for (auto iterator = received_sequences_.begin(); iterator != received_sequences_.end();) { + if (*iterator + 32 < highest) { + iterator = received_sequences_.erase(iterator); + } else { + ++iterator; + } + } + } + + std::pair state() const + { + const uint32_t ack = highest_received_sequence_.value_or(0); + uint32_t ack_mask = 0; + for (uint32_t bit = 0; bit < 32; ++bit) { + if (ack > bit && received_sequences_.find(ack - bit - 1) != received_sequences_.end()) { + ack_mask |= 1u << bit; + } + } + return {ack, ack_mask}; + } + +private: + std::set received_sequences_; + std::optional highest_received_sequence_; +}; + +} // namespace muninn_rudp_ack diff --git a/native/obs_stem_source/muninn_rudp_fragments.h b/native/obs_stem_source/muninn_rudp_fragments.h new file mode 100644 index 0000000..1da82a5 --- /dev/null +++ b/native/obs_stem_source/muninn_rudp_fragments.h @@ -0,0 +1,127 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace muninn_rudp_fragments { + +constexpr uint32_t CompletedFragmentIdQuarantineMs = 2000; + +struct FragmentResult { + uint32_t first_sequence = 0; + std::vector payload; +}; + +class FragmentAssembler { +public: + void reset() + { + assemblies_.clear(); + recently_closed_.clear(); + expired_count_ = 0; + } + + std::optional insert(const std::string &channel_id, + uint16_t fragment_id, + uint16_t fragment_index, + uint16_t fragment_count, + uint32_t sequence, + const uint8_t *payload, + uint32_t payload_len, + std::chrono::steady_clock::time_point now) + { + if (fragment_id == 0 || fragment_index >= fragment_count) { + return std::nullopt; + } + + const std::string key = channel_id + ":" + std::to_string(fragment_id); + expire_recently_closed(now); + const auto existing = assemblies_.find(key); + if (existing == assemblies_.end() && recently_closed_.find(key) != recently_closed_.end()) { + return std::nullopt; + } + + auto &assembly = assemblies_[key]; + if (assembly.payloads.empty()) { + assembly.channel_id = channel_id; + assembly.fragment_count = fragment_count; + assembly.started_at = now; + } + if (assembly.channel_id != channel_id || assembly.fragment_count != fragment_count) { + assemblies_.erase(key); + return std::nullopt; + } + + assembly.payloads.emplace(fragment_index, std::vector(payload, payload + payload_len)); + assembly.sequences.emplace(fragment_index, sequence); + if (assembly.payloads.size() < assembly.fragment_count) { + return std::nullopt; + } + + FragmentResult result; + result.first_sequence = UINT32_MAX; + for (uint16_t index = 0; index < assembly.fragment_count; ++index) { + const auto payload_found = assembly.payloads.find(index); + const auto sequence_found = assembly.sequences.find(index); + if (payload_found == assembly.payloads.end() || sequence_found == assembly.sequences.end()) { + return std::nullopt; + } + result.payload.insert(result.payload.end(), payload_found->second.begin(), payload_found->second.end()); + result.first_sequence = std::min(result.first_sequence, sequence_found->second); + } + assemblies_.erase(key); + recently_closed_[key] = now; + return result; + } + + uint64_t expire(std::chrono::steady_clock::time_point now, uint32_t expire_after_ms) + { + expire_recently_closed(now); + uint64_t expired = 0; + for (auto iterator = assemblies_.begin(); iterator != assemblies_.end();) { + if (now - iterator->second.started_at < std::chrono::milliseconds(expire_after_ms)) { + ++iterator; + continue; + } + recently_closed_[iterator->first] = now; + iterator = assemblies_.erase(iterator); + ++expired; + } + expired_count_ += expired; + return expired; + } + + uint64_t expired_count() const { return expired_count_; } + +private: + void expire_recently_closed(std::chrono::steady_clock::time_point now) + { + for (auto iterator = recently_closed_.begin(); iterator != recently_closed_.end();) { + if (now - iterator->second < std::chrono::milliseconds(CompletedFragmentIdQuarantineMs)) { + ++iterator; + continue; + } + iterator = recently_closed_.erase(iterator); + } + } + + struct Assembly { + std::string channel_id; + uint16_t fragment_count = 0; + std::map> payloads; + std::map sequences; + std::chrono::steady_clock::time_point started_at; + }; + + std::map assemblies_; + std::map recently_closed_; + uint64_t expired_count_ = 0; +}; + +} // namespace muninn_rudp_fragments diff --git a/native/obs_stem_source/muninn_rudp_url.h b/native/obs_stem_source/muninn_rudp_url.h new file mode 100644 index 0000000..bd0bc4a --- /dev/null +++ b/native/obs_stem_source/muninn_rudp_url.h @@ -0,0 +1,123 @@ +#pragma once + +#include +#include +#include +#include +#include + +namespace muninn_rudp_url { + +inline constexpr uint32_t DefaultMediaRudpConnectionId = 0x6d750001; +inline constexpr uint32_t DefaultAudioRudpConnectionId = 0x6d750004; +inline constexpr uint32_t DefaultReliableExpireAfterMs = 75; +inline constexpr uint32_t DefaultVideoAssemblyDeadlineMs = 75; +inline constexpr uint32_t DefaultGapWaitMs = 8; +inline constexpr uint32_t DefaultAudioReorderMs = 2000; +inline constexpr uint32_t DefaultSenderResendDelayMs = 5; +inline constexpr uint32_t MaxLatencyBudgetMs = 2000; + +struct RudpUrlParts { + uint16_t port = 0; + uint16_t video_local_port = 0; + uint16_t audio_local_port = 0; + uint32_t connection_id = DefaultMediaRudpConnectionId; + uint32_t audio_connection_id = DefaultAudioRudpConnectionId; + uint32_t sender_resend_delay_ms = DefaultSenderResendDelayMs; + uint32_t reliable_expire_after_ms = DefaultReliableExpireAfterMs; + uint32_t video_assembly_deadline_ms = DefaultVideoAssemblyDeadlineMs; + uint32_t gap_wait_ms = DefaultGapWaitMs; + uint32_t audio_reorder_ms = DefaultAudioReorderMs; +}; + +inline bool starts_with(const std::string &value, const std::string &prefix) +{ + return value.rfind(prefix, 0) == 0; +} + +inline std::optional parse_u32_hex_or_decimal(const std::string &value) +{ + try { + size_t consumed = 0; + const int base = starts_with(value, "0x") || starts_with(value, "0X") ? 16 : 10; + const auto parsed = std::stoul(value, &consumed, base); + if (consumed == value.size()) { + return static_cast(parsed); + } + } catch (...) { + } + return std::nullopt; +} + +inline std::optional parse(const std::string &url) +{ + if (!starts_with(url, "rudp://")) { + return std::nullopt; + } + + const auto authority_start = std::string("rudp://").size(); + const auto path_start = url.find('/', authority_start); + const auto query_start = url.find('?', authority_start); + const auto authority_end = std::min(path_start == std::string::npos ? url.size() : path_start, + query_start == std::string::npos ? url.size() : query_start); + const auto authority = url.substr(authority_start, authority_end - authority_start); + const auto colon = authority.rfind(':'); + if (colon == std::string::npos || colon + 1 >= authority.size()) { + return std::nullopt; + } + + RudpUrlParts parts; + parts.port = static_cast(std::stoul(authority.substr(colon + 1))); + parts.video_local_port = static_cast(parts.port + 10000); + parts.audio_local_port = static_cast(parts.port + 10001); + + if (query_start != std::string::npos) { + std::stringstream query(url.substr(query_start + 1)); + std::string item; + while (std::getline(query, item, '&')) { + const auto equals = item.find('='); + if (equals == std::string::npos) { + continue; + } + const auto key = item.substr(0, equals); + const auto value = item.substr(equals + 1); + if (key == "connection") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.connection_id = *parsed; + } + } else if (key == "audio_connection" || key == "audio_connection_id") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.audio_connection_id = *parsed; + } + } else if (key == "local_port" || key == "video_local_port") { + parts.video_local_port = static_cast(std::stoul(value)); + } else if (key == "audio_local_port") { + parts.audio_local_port = static_cast(std::stoul(value)); + } else if (key == "reliable_expire_after_ms") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.reliable_expire_after_ms = std::clamp(*parsed, 1u, MaxLatencyBudgetMs); + } + } else if (key == "sender_resend_delay_ms") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.sender_resend_delay_ms = std::clamp(*parsed, 1u, 1000u); + } + } else if (key == "assembly_deadline_ms") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.video_assembly_deadline_ms = std::clamp(*parsed, 1u, MaxLatencyBudgetMs); + } + } else if (key == "gap_wait_ms") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.gap_wait_ms = std::clamp(*parsed, 0u, 1000u); + } + } else if (key == "audio_reorder_ms") { + if (const auto parsed = parse_u32_hex_or_decimal(value)) { + parts.audio_reorder_ms = std::clamp(*parsed, 0u, MaxLatencyBudgetMs); + } + } + } + } + + return parts; +} + +} // namespace muninn_rudp_url diff --git a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp new file mode 100644 index 0000000..97d767c --- /dev/null +++ b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp @@ -0,0 +1,58 @@ +#include "muninn_rudp_url.h" + +#include +#include +#include + +namespace { + +void require(bool condition, const std::string &message) +{ + if (!condition) { + std::cerr << message << '\n'; + std::exit(1); + } +} + +} // namespace + +int main() +{ + const auto configured = muninn_rudp_url::parse( + "rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=1200&assembly_deadline_ms=1200&gap_wait_ms=16&audio_reorder_ms=700"); + require(configured.has_value(), "configured RUDP URL did not parse"); + require(configured->port == 5204, "configured RUDP URL port was not parsed"); + require(configured->video_local_port == 15204, "configured RUDP video local port was not derived"); + require(configured->audio_local_port == 15205, "configured RUDP audio local port was not derived"); + require(configured->connection_id == 0x6d750001, "configured RUDP connection id was not parsed"); + require(configured->audio_connection_id == 0x6d750004, "configured RUDP audio connection id was not parsed"); + require(configured->reliable_expire_after_ms == 1200, "configured reliable expiry did not survive parse"); + require(configured->video_assembly_deadline_ms == 1200, "configured assembly deadline did not survive parse"); + require(configured->gap_wait_ms == 16, "configured gap wait did not survive parse"); + require(configured->audio_reorder_ms == 700, "configured audio reorder wait did not survive parse"); + + const auto clamped = muninn_rudp_url::parse( + "rudp://192.168.1.66:5204/muninn.raven.av.rudp?reliable_expire_after_ms=9000&assembly_deadline_ms=9000"); + require(clamped.has_value(), "oversized latency RUDP URL did not parse"); + require(clamped->reliable_expire_after_ms == muninn_rudp_url::MaxLatencyBudgetMs, + "oversized reliable expiry did not clamp to the max latency budget"); + require(clamped->video_assembly_deadline_ms == muninn_rudp_url::MaxLatencyBudgetMs, + "oversized assembly deadline did not clamp to the max latency budget"); + + const auto fallback = muninn_rudp_url::parse("rudp://192.168.1.66:5204/muninn.raven.av.rudp"); + require(fallback.has_value(), "minimal RUDP URL did not parse"); + require(fallback->connection_id == muninn_rudp_url::DefaultMediaRudpConnectionId, + "minimal RUDP URL did not keep the default connection id"); + require(fallback->audio_connection_id == muninn_rudp_url::DefaultAudioRudpConnectionId, + "minimal RUDP URL did not keep the default audio connection id"); + require(fallback->reliable_expire_after_ms == muninn_rudp_url::DefaultReliableExpireAfterMs, + "minimal RUDP URL did not keep the default reliable expiry"); + require(fallback->video_assembly_deadline_ms == muninn_rudp_url::DefaultVideoAssemblyDeadlineMs, + "minimal RUDP URL did not keep the default assembly deadline"); + require(fallback->audio_reorder_ms == muninn_rudp_url::DefaultAudioReorderMs, + "minimal RUDP URL did not keep the default audio reorder wait"); + require(fallback->audio_reorder_ms == 2000, + "minimal RUDP URL did not use the full default audio reorder budget"); + + return 0; +} From ed46270be607fe9ec73d6f9a0b27f993bda29d54 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 5 Jul 2026 23:35:32 +0100 Subject: [PATCH 062/135] Make Muninn OBS discovery visibly Odin-owned --- .../mimir_obs_muninn_source.cpp | 38 +++++++++++++++---- .../obs_stem_source/mimir_obs_stem_source.cpp | 3 +- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 3a1613b..eefaba7 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -1939,23 +1939,46 @@ static int run_muninn_cli(const std::string &exe, const std::vector return std::system(command.c_str()); } -static void refresh_obs_catalog_from_odin(MuninnSource *ctx) +static bool refresh_obs_catalog_from_odin_store( + const std::string &command_exe_path, + const std::string &store_path, + const std::string &host_id) { - if (!ctx) { - return; + if (store_path.empty() || host_id.empty()) { + return false; } - const int exit_code = run_muninn_cli(ctx->command_exe_path, { + const int exit_code = run_muninn_cli(command_exe_path, { "obs-catalog-status", "--store", - ctx->store_path, + store_path, "--host", - ctx->host_id, + host_id, "--odin-cultmesh-uri", MuninnOdinCultMeshUri, }); if (exit_code != 0) { - blog(LOG_WARNING, "Muninn Stream could not refresh OBS catalog from Odin via %s", ctx->command_exe_path.c_str()); + blog(LOG_WARNING, + "Muninn Stream could not refresh OBS catalog from Odin exe=%s store=%s host=%s exit=%d", + command_exe_path.empty() ? MuninnCommandExePath : command_exe_path.c_str(), + store_path.c_str(), + host_id.c_str(), + exit_code); + return false; } + blog(LOG_INFO, + "Muninn Stream refreshed OBS catalog from Odin exe=%s store=%s host=%s", + command_exe_path.empty() ? MuninnCommandExePath : command_exe_path.c_str(), + store_path.c_str(), + host_id.c_str()); + return true; +} + +static bool refresh_obs_catalog_from_odin(MuninnSource *ctx) +{ + if (!ctx) { + return false; + } + return refresh_obs_catalog_from_odin_store(ctx->command_exe_path, ctx->store_path, ctx->host_id); } static const MuninnStreamOption *find_selected(const std::vector &options, const std::string &stream_id) @@ -3649,6 +3672,7 @@ static bool refresh_source_properties_for_settings( static void populate_stream_list(obs_property_t *property, const std::string &store_path) { + refresh_obs_catalog_from_odin_store(MuninnCommandExePath, store_path, MuninnDefaultHostId); auto options = read_catalog(store_path); if (options.empty()) { options.push_back(default_muninn_stream_option()); diff --git a/native/obs_stem_source/mimir_obs_stem_source.cpp b/native/obs_stem_source/mimir_obs_stem_source.cpp index 5ebcbd1..5d78864 100644 --- a/native/obs_stem_source/mimir_obs_stem_source.cpp +++ b/native/obs_stem_source/mimir_obs_stem_source.cpp @@ -214,6 +214,7 @@ bool obs_module_load(void) blog( LOG_INFO, "Mimir OBS sources loaded: mimir_audio_stem_source, mimir_program_texture_source, " - "muninn_stream_source, muninn_video_source, muninn_audio_source"); + "muninn_stream_source, muninn_video_source, muninn_audio_source; " + "Muninn Odin discovery build 20260705.2"); return true; } From d6f64006773b245b6dd067f726b6c798438c4566 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 5 Jul 2026 23:42:45 +0100 Subject: [PATCH 063/135] Hide Muninn OBS CLI discovery launches --- .../mimir_obs_muninn_source.cpp | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index eefaba7..297a218 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -55,6 +55,7 @@ constexpr const char *MuninnObsCatalogType = "muninn.obs_stream_catalog"; constexpr const char *MuninnObsCatalogSchema = "muninn.obs_stream_catalog.v1"; constexpr const char *MuninnObsCatalogKey = "obs"; constexpr uint64_t MuninnCatalogRefreshMs = 2000; +constexpr uint64_t MuninnOdinCatalogRefreshMs = 5000; constexpr uint64_t MuninnActivationRetryMs = 5000; constexpr uint64_t MuninnMediaStartupGraceMs = 3000; constexpr uint64_t MuninnMediaStaleMs = 1500; @@ -1936,7 +1937,39 @@ static int run_muninn_cli(const std::string &exe, const std::vector command.push_back(' '); command += quote_windows_arg(arg); } +#ifdef _WIN32 + std::vector command_line(command.begin(), command.end()); + command_line.push_back('\0'); + + STARTUPINFOA startup_info = {}; + startup_info.cb = sizeof(startup_info); + startup_info.dwFlags = STARTF_USESHOWWINDOW; + startup_info.wShowWindow = SW_HIDE; + + PROCESS_INFORMATION process_info = {}; + if (!CreateProcessA( + nullptr, + command_line.data(), + nullptr, + nullptr, + FALSE, + CREATE_NO_WINDOW, + nullptr, + nullptr, + &startup_info, + &process_info)) { + return static_cast(GetLastError()); + } + + WaitForSingleObject(process_info.hProcess, INFINITE); + DWORD exit_code = 1; + GetExitCodeProcess(process_info.hProcess, &exit_code); + CloseHandle(process_info.hThread); + CloseHandle(process_info.hProcess); + return static_cast(exit_code); +#else return std::system(command.c_str()); +#endif } static bool refresh_obs_catalog_from_odin_store( @@ -1947,6 +1980,20 @@ static bool refresh_obs_catalog_from_odin_store( if (store_path.empty() || host_id.empty()) { return false; } + static std::mutex refresh_mutex; + static std::map next_refresh_by_store; + const std::string refresh_key = host_id + "|" + store_path; + const auto now = std::chrono::steady_clock::now(); + { + std::lock_guard lock(refresh_mutex); + const auto next_refresh = next_refresh_by_store.find(refresh_key); + if (next_refresh != next_refresh_by_store.end() && now < next_refresh->second) { + return true; + } + next_refresh_by_store[refresh_key] = + now + std::chrono::milliseconds(MuninnOdinCatalogRefreshMs); + } + const int exit_code = run_muninn_cli(command_exe_path, { "obs-catalog-status", "--store", From 7e2d35ad5962a913a0f1f26303bfdc88f74df08b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 6 Jul 2026 00:45:08 +0100 Subject: [PATCH 064/135] Make Muninn OBS bridge allocate decoder ports --- .../mimir_obs_muninn_source.cpp | 71 ++++++++++++++++--- 1 file changed, 63 insertions(+), 8 deletions(-) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 297a218..90bb3f4 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -166,6 +166,38 @@ struct RudpBridgeOutputs { std::string audio_url; }; +static std::optional allocate_udp_loopback_port() +{ +#ifdef _WIN32 + SOCKET socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (socket == INVALID_SOCKET) { + return std::nullopt; + } + + sockaddr_in bind_addr = {}; + bind_addr.sin_family = AF_INET; + inet_pton(AF_INET, "127.0.0.1", &bind_addr.sin_addr); + bind_addr.sin_port = 0; + if (bind(socket, reinterpret_cast(&bind_addr), sizeof(bind_addr)) != 0) { + closesocket(socket); + return std::nullopt; + } + + sockaddr_in local_addr = {}; + int local_len = sizeof(local_addr); + if (getsockname(socket, reinterpret_cast(&local_addr), &local_len) != 0) { + closesocket(socket); + return std::nullopt; + } + + const uint16_t port = ntohs(local_addr.sin_port); + closesocket(socket); + return port == 0 ? std::nullopt : std::optional(port); +#else + return std::nullopt; +#endif +} + struct MuninnMediaPayload { enum class Kind { Unknown, Video, VideoParity, Audio }; Kind kind = Kind::Unknown; @@ -228,6 +260,11 @@ static bool starts_with(const std::string &value, const std::string &prefix) return value.rfind(prefix, 0) == 0; } +static bool is_cultmesh_uri(const std::string &value) +{ + return starts_with(value, "cultmesh://"); +} + static uint64_t steady_millis() { return static_cast( @@ -686,6 +723,18 @@ public: source_url_ = url; parts_ = *parsed; + const uint16_t requested_video_port = parts_.video_local_port; + const uint16_t requested_audio_port = parts_.audio_local_port; + const auto video_port = allocate_udp_loopback_port(); + const auto audio_port = allocate_udp_loopback_port(); + if (!video_port || !audio_port || video_port == audio_port) { + blog(LOG_WARNING, "Muninn RUDP bridge could not allocate local decoder UDP ports"); + source_url_.clear(); + parts_ = {}; + return {}; + } + parts_.video_local_port = *video_port; + parts_.audio_local_port = *audio_port; outputs_.video_url = "udp://127.0.0.1:" + std::to_string(parts_.video_local_port) + "?fifo_size=8388608&overrun_nonfatal=1&buffer_size=67108864"; outputs_.audio_url = "udp://127.0.0.1:" + std::to_string(parts_.audio_local_port) + @@ -700,8 +749,9 @@ public: running_ = true; worker_ = std::thread([this] { run(); }); blog(LOG_INFO, - "Muninn RUDP bridge starting udp/%u -> video udp/127.0.0.1:%u audio udp/127.0.0.1:%u sender_resend_delay_ms=%u reliable_expire_after_ms=%u assembly_deadline_ms=%u gap_wait_ms=%u audio_reorder_ms=%u", + "Muninn RUDP bridge starting udp/%u -> video udp/127.0.0.1:%u audio udp/127.0.0.1:%u requested_video_port=%u requested_audio_port=%u sender_resend_delay_ms=%u reliable_expire_after_ms=%u assembly_deadline_ms=%u gap_wait_ms=%u audio_reorder_ms=%u", parts_.port, parts_.video_local_port, parts_.audio_local_port, + requested_video_port, requested_audio_port, parts_.sender_resend_delay_ms, parts_.reliable_expire_after_ms, parts_.video_assembly_deadline_ms, parts_.gap_wait_ms, parts_.audio_reorder_ms); return outputs_; @@ -2642,13 +2692,18 @@ static std::string selected_source_id( static std::string endpoint_setting(obs_data_t *settings, const char *key, const char *current_default, const char *deprecated_default) { std::string value = obs_string(settings, key, current_default); - if (value == deprecated_default) { + if (value == deprecated_default || is_cultmesh_uri(value)) { obs_data_set_string(settings, key, current_default); return current_default; } return value; } +static std::string media_receiver_host(const std::string &candidate) +{ + return candidate.empty() || is_cultmesh_uri(candidate) ? MuninnDefaultMediaTargetHost : candidate; +} + static void request_stream_activation(MuninnSource *ctx, const std::string &stream_id, bool force = false) { const auto canonical_stream_id = canonical_muninn_stream_id(stream_id); @@ -3433,11 +3488,7 @@ static void create_or_update_media(MuninnSource *ctx, const std::string &url, bo (!wants_video || ctx->media) && (!wants_audio || ctx->audio_decoder) && !ctx->stream_url.empty()) { - const auto previous_parts = muninn_rudp_url::parse(ctx->stream_url); - const auto next_parts = muninn_rudp_url::parse(url); - if (previous_parts && next_parts && - previous_parts->video_local_port == next_parts->video_local_port && - previous_parts->audio_local_port == next_parts->audio_local_port) { + if (equivalent_rudp_stream_url(ctx->stream_url, url)) { ctx->bridge->start(url); ctx->stream_url = url; return; @@ -3452,6 +3503,10 @@ static void create_or_update_media(MuninnSource *ctx, const std::string &url, bo ctx->bridge = std::make_unique(); } const auto outputs = ctx->bridge->start(url); + if (outputs.video_url.empty() && outputs.audio_url.empty()) { + ctx->stream_url.clear(); + return; + } bool activated = false; if (wants_video) { ctx->media = create_ffmpeg_child_source(ctx, "Muninn Stream Video", outputs.video_url, "h264", false); @@ -3552,7 +3607,7 @@ static void reconcile_selected_stream(MuninnSource *ctx) if (split_ipv4_endpoint(MuninnCommandRudpTarget, bootstrap_host, bootstrap_port)) { ctx->command_rudp_target = replace_endpoint_host_if_unspecified(ctx->command_rudp_target, bootstrap_host); } - ctx->media_target_host = selected->media_target_host.empty() ? MuninnDefaultMediaTargetHost : selected->media_target_host; + ctx->media_target_host = media_receiver_host(selected->media_target_host); ctx->media_port = selected->media_port == 0 ? MuninnDefaultMediaPort : selected->media_port; ctx->media_packet_bytes = selected->media_packet_bytes == 0 ? MuninnDefaultMediaPacketBytes : selected->media_packet_bytes; From a040792a1c970bc1b86ec8e1a6d3b55ab25aee86 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 8 Jul 2026 22:29:23 +0100 Subject: [PATCH 065/135] Add Mimir Move proof surface --- docs/implementation-plan.md | 104 +++++--- notes/current-system-map.md | 107 ++++---- src/Mimir.BufferSmoke/Program.cs | 142 ++++++++++- .../Synchronization/MimirMoveProofSurface.cs | 232 ++++++++++++++++++ .../MimirMuninnMoveEvidence.cs | 71 +++++- state/evidence.jsonl | 6 + state/map.yaml | 88 ++++--- 7 files changed, 630 insertions(+), 120 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index c100387..9779636 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -52,6 +52,12 @@ a named invariant that the native runtime cannot protect yet. - The realtime Move witness path uses CultMesh streaming, not CultCache polling: Muninn publishes shared-memory stream frames, Mimir consumes the latest read lease, normalizes the frame, and admits it to native `move_evidence`. +- `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a + typed admission receipt for that latest read: decoded frame id, producer, + publish/read timestamps, sample counts by kind, source/arrival ranges, native + reservoir handle, and reservoir edge/window. `Mimir.BufferSmoke + --muninn-move-cultmesh-stream-smoke` asserts the receipt so the named proof + spine can be traced instead of inferred. - `mimir.move_controller_pose.v1` is Mimir's resolved wand pose contract. Mimir owns calibration, association, triangulation, IMU fusion, prediction, confidence, and latency accounting for those poses before Fensalir consumes @@ -67,6 +73,15 @@ a named invariant that the native runtime cannot protect yet. `mimir.move_controller_pose_stream_frame.v1` over CultMesh shared-memory bytes streams so Fensalir and other consumers have a realtime stream contract for Mimir-fused controller input. +- `MimirMoveProofSurface` is the observer-only dev proof surface for the named + Move chain. It combines a Muninn evidence admission receipt with a Mimir pose + stream frame and a Fensalir presentation/probe timestamp into + `mimir.move_proof_surface.v1`, then lowers the verdict to a Fensalir + `AquariumSplineFrame` probe. `Mimir.BufferSmoke --move-proof-surface-smoke` + proves the explicit + `muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> + mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79` chain, + and rejects single-ray fallback as not full pose. - `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for Starfire/Nightwing Moves: required Muninn evidence streams, optional Quest headset/controller pose witnesses, stillness/sweep/validation phases, and @@ -75,13 +90,12 @@ a named invariant that the native runtime cannot protect yet. - Muninn publishes Quest access as `muninn.quest_access.v1` for the USB-attached Quest. Mimir consumes that access surface and later `muninn.quest_pose_frame.v1` samples as optional calibration evidence; Mimir does not own ADB/Quest access. -- `scripts/start-nightwing-move-tracking.ps1` is the narrow live Nightwing - bring-up path: it starts the `/eve/periwinkle` receiver/recorder on Starfire, - stages the Nightwing Eye/Move Python worker, keeps a heartbeat for field - liveness, and publishes `mimir.psmove_light_state.v1` plus - `mimir.move_controller_observation_state.v1` blob observations from both PS3 - Eyes. The blob stream is optical witness evidence for later pose fusion, not - the final 6DoF pose owner. +- The old `scripts/start-nightwing-move-tracking.ps1` bring-up path is + archived. It no longer starts a receiver/recorder pair or stages a socket + witness worker. Nightwing Eye/Move evidence must enter through Muninn/Mimir + typed CultMesh stream frames and Odin-discovered CultMesh documents. The blob + stream remains optical witness evidence for later pose fusion, not the final + 6DoF pose owner. - Structured PS Move light pulses are Muninn output commands. Mimir publishes `muninn.move_light_command.v1` over CultNet/CultMesh to the Muninn daemon on the host that owns the USB-attached Move. Muninn writes PS Move HID report @@ -95,6 +109,17 @@ a named invariant that the native runtime cannot protect yet. streams; it does not own raw optical extraction. Final pose, stereo triangulation, calibration, association, IMU fusion, and prediction belong to Mimir. +- Muninn's Move evidence stream frame no longer hardcodes an empty optical + candidate slice. The daemon frame uses the canonical + `muninn.move_marker_candidate.v1` record shape, and the Odin unit + `move_marker_candidates_publish_in_mimir_compatible_cultmesh_frame` proves a + bright Y8 frame can pass through `muninn-move-tracker` and serialize as a + non-empty marker candidate beside controller evidence. The daemon now has a + source-local Y8 extraction/publish seam and a first `serve` camera producer: + `--move-marker-camera =` polls a Unix V4L2 YUYV frame, + converts it to compact Y8, and feeds that same seam. Unit tests prove the + configured camera tick publishes marker evidence through the shared frame + contract; Nightwing hardware/live V4L2 proof is still pending. - `MimirProcessStreamSource` for bridge/network command edges. - `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata from native probes into the same rolling buffers Fensalir inspects. One probe @@ -284,22 +309,31 @@ a named invariant that the native runtime cannot protect yet. `mimir.cultmesh_media_frame` documents, and `recv` subscribes to those documents and writes ordered MPEG-TS bytes to a Starfire-local UDP endpoint for compatibility sinks. Its C# implementation now uses explicit CultLib - RUDP client/session helpers; the older + RUDP client/session helpers behind + `cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`; the older `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` entrypoints no longer own - the media transport lane. + the media transport lane, and sender/receiver launchers no longer accept raw + relay host/port configuration. - `scripts/start-raven-cultmesh-av-sender.ps1`, `scripts/start-yggdrasil-cultmesh-media-relay.ps1`, and `scripts/start-starfire-cultmesh-av-receiver.ps1` are the CultMesh bridge - operators. Raven capture defaults to FFmpeg desktop frames plus Mimir's - WASAPI loopback capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains - an explicit fallback. -- `src/Mimir.EveDashboard` is a CultMesh/Eve dashboard broker. It publishes - `mimir.eve_dashboard_state` through CultNet/CultMesh and can report - `idunn.daemon_health` over `cultnet.transport.rudp.v0` for itself and an - optional paired service daemon. The live direction is daemon-owned RUDP - publication for provider advertisement, retained state, command boundary, and - transport profile too; HTTP/WebSocket are client lowerings. This is a health - witness for Idunn, not a lifecycle owner. + operators. `start-raven-cultmesh-av-sender.ps1` is the Mimir-owned + `Mimir.CultMeshMedia` bootstrap/body-bridge lane; it is not the real Muninn + OBS/SRT feed owner. The actual Raven OBS/plugin feed now routes through + Odin's Muninn actuator `E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1`, + with `scripts/start-raven-muninn-obs-feed.ps1` in this repo as a thin local + wrapper around the real `GameCult-Muninn-Activate` task and + `muninn.exe activate` body. `-LocalBootstrap` on the CultMesh sender remains + a direct local bootstrap edge only for the separate body-bridge lane. Raven + capture still defaults to FFmpeg desktop frames plus Mimir's WASAPI loopback + capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains an explicit + fallback. +- `src/Mimir.EveDashboard` is archived until the dashboard returns as a pure + CultMesh/Odin publisher and Eve lowering. The old HTTP/WebSocket deck broker, + local health route, socket command channel, and provider route catalog are not + daemon transport. Dashboard state must be published as typed CultMesh/Eve + documents through Odin, and dashboard commands must arrive as typed Odin/ + CultMesh command documents. - `src/Mimir.EveBrowserReference` serves static browser lowerings and can publish its own `idunn.daemon_health` record over `cultnet.transport.rudp.v0`. It remains a renderer/reference surface, not @@ -313,17 +347,17 @@ a named invariant that the native runtime cannot protect yet. testing, but Mimir/Eve owns independent controls. - The CultMesh media bridge still lowers to local UDP for compatibility sinks because OBS is not a CultMesh consumer. Network transit between Raven, - Yggdrasil, and Starfire is the CultMesh/CultNet path; OBS-local UDP is an - egress adapter only. + Yggdrasil, and Starfire is the Odin-discovered CultMesh/CultNet path; + OBS-local UDP is an egress adapter only. - CultLib RUDP is the default typed CultNet/CultMesh document transport for daemon truth. Idunn RUDP health publication is the current freshness witness; provider advertisements, command boundaries, transport profiles, and retained daemon state should follow the same path. Odin still owns Verse/service discovery, Idunn owns keepalive decisions, and Mimir-owned dashboard/reference - surfaces only report their own observed state. TCP/HTTP/WebSocket stay - lowerings or compatibility evidence. + surfaces only report their own observed state. Product/debug render surfaces + stay lowerings or compatibility evidence. - `Mimir.CultMeshMedia` has completed its explicit RUDP transport cut. Do not - add another TCP bridge, HTTP status shim, or WebSocket-derived service truth + add another private bridge, status shim, or renderer-derived service truth while the RUDP document lane exists. - Process-backed stream sources are only acceptable for network bridge feeds or diagnostics. Six-camera local ingest belongs behind direct capture drivers. @@ -337,8 +371,8 @@ a named invariant that the native runtime cannot protect yet. 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the other cameras. -2. Cut the remaining dashboard, recorder, and witness service truth paths from - older TCP/WebSocket-derived assumptions to explicit CultNet RUDP records, +2. Cut the remaining dashboard service truth paths from older renderer-derived + assumptions to explicit CultNet RUDP records, preferably authorized-peer dialing where the peer catalog exists. Preserve OBS-local UDP as egress only for compatibility sinks. 3. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained @@ -356,22 +390,26 @@ a named invariant that the native runtime cannot protect yet. becomes a deterministic timeline anchor before the actuator moves samples. 6. Prove the bioacoustic hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. -7. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, +7. Attach the Move proof surface to the live runtime/Fensalir presentation path: + ingest real Muninn Nightwing and Starfire evidence, require calibrated + optical witnesses, publish `mimir:starfire:move-pose:`, and capture + a presented Fensalir frame/probe for the same sequence. +8. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. -8. Implement the Mimir program scene graph as the shared commit primitive for +9. Implement the Mimir program scene graph as the shared commit primitive for source subscription, transforms, crop, chroma key, visibility, layer order, preview, and output publication. Import the current OBS scene only as an initial mirror, then make Eve GUI/TUI the operator surface. -9. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir +10. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir program output and publishes it without owning a second composition. -10. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: +11. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: sample buffer-domain paths stochastically by visual contribution, emit SDF splat probes, write them into the spatiotemporal splat reservoir, and sample that reservoir in the temporally antialiased scene pass. The direct spline preview must stay a witness until this path owns rendering. -11. Move GPU feature extraction, fusion, material fitting, render budgeting, and +12. Move GPU feature extraction, fusion, material fitting, render budgeting, and Spout2 publication into Fensalir. -12. Move mic alignment, room suppression, voice separation, spatialization, and +13. Move mic alignment, room suppression, voice separation, spatialization, and stem generation into Faust/native DSP. -13. Keep the OBS bridge witness ledger as evidence before expanding receiver +14. Keep the OBS bridge witness ledger as evidence before expanding receiver machinery. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 9d3d181..b3a3b6d 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -53,11 +53,11 @@ Ownership: Fensalir-facing wand pose stream. - Mimir owns program composition, source subscription policy, preview/control state, stats, and publication intent. -- Mimir-owned Eve dashboard and browser reference processes publish their own - typed health/freshness to Idunn over RUDP when configured. Their intended - daemon-owned provider state, command boundaries, and transport profiles also - belong on CultNet/RUDP; HTTP/WebSocket are client lowerings and compatibility - evidence, not daemon truth. +- Mimir's old Eve dashboard broker is archived; it does not publish health, + provider catalogs, or command surfaces while the socket path is cut. The + browser reference remains a client lowering, not daemon truth. Dashboard state + must return as typed CultMesh/Eve documents through Odin; product/debug render + surfaces are client lowerings and compatibility evidence, not daemon truth. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, D3D12 interop, runtime UI lowering, and local program texture output. - Faust/native DSP owns hot audio alignment, suppression, separation, @@ -161,17 +161,13 @@ reported authorized Quest 2 device `1WMHHB68PG1515` with product/device provide Quest headset/controller poses until a Quest/OpenXR witness bridge publishes those pose samples through Muninn. -Live Nightwing bring-up uses `scripts/start-nightwing-move-tracking.ps1`. -Starfire starts `Mimir.EveSensorReceiver` on `/eve/periwinkle`, stages -`nw_eye_cap.py`, `nw_move_hint.py`, and `nightwing_typed_witness_publisher.py` -onto Nightwing, and launches the publisher with `--track-eyes` plus a heartbeat -freshness file. The publisher can still accept -`--move-light name=/dev/hidrawN:#rrggbb` as an explicit smoke/bootstrap edge, -but runtime structured light commands belong to Muninn. The worker publishes -compact -`mimir.move_controller_observation_state.v1` blob observations from both PS3 -Eyes. Those Eye/Move observations are the live optical witness feeding the later -pose/fusion owner; they are not themselves the final 6DoF pose authority. +The old Nightwing bring-up path is archived: +`scripts/start-nightwing-move-tracking.ps1`, `Mimir.EveSensorReceiver`, +`Mimir.VerseRecorder`, and `nightwing_typed_witness_publisher.py` now fail +closed. Runtime structured light commands belong to Muninn. Eye/Move +observations must enter through typed CultMesh stream frames or +Odin-discovered CultMesh documents; they are optical witness evidence for the +later pose/fusion owner, not the final 6DoF pose authority. Starfire-local Move illumination smoke uses `scripts/start-starfire-move-light.ps1`. That launcher runs `Mimir.PsMoveProbe` against the local Windows HID col01 @@ -193,6 +189,24 @@ those Muninn candidate streams into tracking buffers; it does not own raw optical extraction. Mimir is now the explicit owner for stereo triangulation, camera calibration, controller association, IMU fusion, prediction, and final 6DoF wand pose. +Muninn's `muninn:nightwing:move-evidence:` shared-memory frame now +uses the canonical `muninn.move_marker_candidate.v1` document shape for optical +markers instead of a placeholder empty slice. A daemon unit proves that a bright +Y8 frame run through `muninn-move-tracker` serializes a non-empty marker +candidate into the Mimir-compatible evidence frame. The daemon now has a +source-local Y8 extraction/publish seam plus a first `serve` camera producer: +`--move-marker-camera =` polls a Unix V4L2 YUYV frame, +converts it to compact Y8, and feeds that same seam. Windows unit tests prove +the configured camera tick publishes marker evidence through the shared frame +contract; Nightwing hardware/live V4L2 proof is still pending. +`MimirMoveProofSurface` is the first Fensalir-visible proof surface for this +chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream +frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only +`AquariumSplineFrame` probe with the explicit +`muninn:nightwing:move-evidence: -> mimir:starfire:move-evidence: -> mimir:starfire:move-pose: -> fensalir:starfire:presented-frame:` +chain. It does not decide pose. Single-ray fallback remains a visible +non-final verdict until calibrated multi-camera optical evidence earns full +pose. ## Program Composition @@ -239,7 +253,7 @@ not become the synchronized program authority. flowchart TD A["Raven FFmpeg desktop + WASAPI loopback mux"] --> B["Mimir.CultMeshMedia send"] B --> C["CultNet reliable UDP media-frame documents"] - C --> D["Yggdrasil CultMesh relay on 10.77.0.1:3075"] + C --> D["Yggdrasil CultMesh relay discovered through Odin/CultMesh"] D --> E["Starfire Mimir.CultMeshMedia recv"] E --> F["local MPEG-TS UDP udp://127.0.0.1:5200"] F --> G["OBS Raven Monitor + Realtek"] @@ -253,9 +267,10 @@ mature. Invariant: Raven media body state is live and bounded. Sender writes rolling `mimir.cultmesh_media_frame` slot documents rather than an unbounded durable -video archive. Yggdrasil relays CultNet reliable UDP over the WireGuard mesh. -Starfire lowers the subscribed stream to local UDP for compatibility sinks -because OBS is not a CultMesh runtime. +video archive. Yggdrasil relays CultNet reliable UDP behind the logical +`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av` route discovered +through Odin/CultMesh. Starfire lowers the subscribed stream to local UDP for +compatibility sinks because OBS is not a CultMesh runtime. Source audit correction resolved: `Mimir.CultMeshMedia` now uses explicit CultLib RUDP helpers for the sender/receiver and a relay-owned RUDP @@ -267,14 +282,21 @@ for inspection; live network transit is the RUDP schema-document path. Current deployment: Yggdrasil runs the relay from `/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia` with cache `/var/lib/gamecult/mimir/cultmesh-media.cc` and log -`/var/log/gamecult/mimir-cultmesh-media.log`. Starfire runs the receiver from -`artifacts/mimir-cultmesh-media-win-x64-selfcontained/Mimir.CultMeshMedia.exe` -and writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS source -`Raven Monitor + Realtek` points at that local UDP endpoint. +`/var/log/gamecult/mimir-cultmesh-media.log`. Senders and receivers target +`cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av`; concrete RUDP +bootstrap lives in the CultMesh resolver environment, not in daemon launch +arguments. Starfire writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS +source `Raven Monitor + Realtek` points at that local UDP endpoint. -Raven SSH over WireGuard timed out during the initial deployment pass, so the -Raven sender still needs to be launched on Raven directly with -`scripts/start-raven-cultmesh-av-sender.ps1`. +`scripts/start-raven-cultmesh-av-sender.ps1` is the Mimir-owned CultMesh bridge +bootstrap for Raven media-body transit. It is not the actual Muninn OBS feed +owner. The real Raven OBS/SRT feed owner is Odin's Muninn: +`E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1` drives the +existing `GameCult-Muninn-Activate` hidden task and `muninn.exe activate` +body on Raven, and `scripts/start-raven-muninn-obs-feed.ps1` in this repo is +the thin local wrapper for that actuator. Keep `-LocalBootstrap` on the +CultMesh sender only as a staging/bootstrap edge when the separate body-bridge +lane needs direct local proof. ## Daemon Health And Idunn @@ -286,20 +308,16 @@ private lifecycle supervisor for these surfaces. CultLib's current direction is RUDP everywhere for typed CultNet/CultMesh documents across runtimes. That means health, provider advertisements, command boundaries, transport profiles, and selected program/media document lanes should -default to `cultnet.transport.rudp.v0`. TCP, HTTP, and WebSocket are acceptable -as renderer/client lowerings, debug tools, or migration debt; they must not own -daemon truth once an RUDP contract exists. +default to `cultnet.transport.rudp.v0`. Product/debug render surfaces are +lowerings, debug tools, or migration debt; they must not own daemon truth once +an RUDP contract exists. Mimir's current daemon-health publishers are local RUDP witnesses: -- `src/Mimir.EveDashboard` publishes `mimir.eve_dashboard_state` through - CultNet/CultMesh and, when `--idunn-rudp-health` or - `MIMIR_EVE_DASHBOARD_IDUNN_RUDP_HEALTH` is set, sends - `idunn.daemon_health` records tagged `cultnet.transport.rudp.v0` for the - dashboard broker and optionally a paired service daemon. The next daemon - boundary work is to publish its provider advertisement, retained state, - command boundary, and transport profile as typed RUDP records so Odin can - prefer those over compatibility HTTP catalog ingestion. +- `src/Mimir.EveDashboard` is a fail-closed archive tombstone. It no longer + starts a TCP listener, serves `/eve/deck`, accepts WebSocket commands, or + publishes daemon health. `mimir.eve_dashboard_state` must return through Odin/ + CultMesh state publication before this service can be redeployed. - `src/Mimir.EveBrowserReference` serves a static browser lowering and, when configured with `--idunn-rudp-health` or `MIMIR_EVE_BROWSER_REFERENCE_IDUNN_RUDP_HEALTH`, sends its own @@ -309,13 +327,12 @@ Invariant: health publication is not ownership. The daemon reports what it can honestly observe about itself; Idunn decides keepalive action through the shared typed record path. -Source audit debt: `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, and the -browser/dashboard WebSocket paths are still renderer/client lowerings. They may -remain as lowering adapters, but any provider catalog, retained service state, -command boundary, transport profile, or lifecycle health claim must be -published as typed CultNet/RUDP records instead of being inferred from those -TCP/WebSocket surfaces. The media-frame bridge is already on that side of the -line. +Source audit cut: `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the +Nightwing typed witness publisher, and `start-nightwing-move-tracking.ps1` are +archived. The old Nightwing/EVE witness lane no longer opens a renderer/client +subscription surface. Witness publication and recording must use typed +CultMesh/CultNet documents or CultMesh stream frames discovered through Odin. +The media-frame bridge is already on that side of the line. ## Audio Field diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 32d9911..e22e014 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -135,6 +135,11 @@ if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.O return RunMoveFusionSmoke(); } +if (args.Any(arg => string.Equals(arg, "--move-proof-surface-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofSurfaceSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", StringComparison.OrdinalIgnoreCase))) { return await RunMoveCalibrationProtocolSmokeAsync( @@ -1911,19 +1916,32 @@ static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) "mimir:muninn-move-cultmesh-stream-smoke", "mimir-move-stage-calibration-v1", "mimir-stage-space", - out var handle, - out var sampleCount); + out var admission); var status = runtime.Status; var latest = catalog.LatestFrame(streamId); + var sampleCount = admission?.SampleCount ?? 0; + var handle = admission?.Handle ?? default; Console.WriteLine( - $"muninn-move-cultmesh-stream-smoke stream={streamId} bytes={payload.Length} publishedSeq={published.Sequence} latestSeq={latest?.Sequence.ToString() ?? "none"} samples={sampleCount} payload=0x{handle.PayloadHandle:X} total={status.TotalSampleCount} moveEvidence={status.MoveEvidenceCount} edgeNs={status.EdgeNs}"); + $"muninn-move-cultmesh-stream-smoke stream={streamId} frame={admission?.FrameId ?? "none"} producer={admission?.ProducerPeerId ?? "none"} publishedNs={admission?.PublishedAtNs ?? 0} readNs={admission?.MimirReadAtNs ?? 0} bytes={payload.Length} publishedSeq={published.Sequence} latestSeq={latest?.Sequence.ToString() ?? "none"} samples={sampleCount} optical={admission?.OpticalMarkerCount ?? 0} controller={admission?.ControllerStateCount ?? 0} sourceMinNs={admission?.SourceTimeMinNs ?? 0} sourceMaxNs={admission?.SourceTimeMaxNs ?? 0} payload=0x{handle.PayloadHandle:X} reservoirEdgeNs={admission?.ReservoirEdgeNs ?? 0} reservoirWindowStartNs={admission?.ReservoirWindowStartNs ?? 0} total={status.TotalSampleCount} moveEvidence={status.MoveEvidenceCount} edgeNs={status.EdgeNs}"); return admitted && + admission is not null && + admission.FrameId == frame.FrameId && + admission.ProducerPeerId == frame.ProducerPeerId && + admission.PublishedAtNs == frame.PublishedAtNs && + admission.MimirReadAtNs > 0 && payload.Length > 0 && sampleCount == 2 && + admission.OpticalMarkerCount == 1 && + admission.ControllerStateCount == 1 && + admission.SourceTimeMinNs <= admission.SourceTimeMaxNs && + admission.ArrivalMinNs <= admission.ArrivalMaxNs && latest?.Sequence == published.Sequence && handle.PayloadHandle != 0 && + admission.ReservoirEdgeNs == status.EdgeNs && + admission.ReservoirWindowStartNs == status.WindowStartNs && + admission.ReservoirMoveEvidenceCount == status.MoveEvidenceCount.ToUInt64() && status.TotalSampleCount.ToUInt64() == 1 && status.MoveEvidenceCount.ToUInt64() == 1 ? 0 @@ -2081,6 +2099,124 @@ static int RunMoveFusionSmoke() : 1; } +static int RunMoveProofSurfaceSmoke() +{ + var pose = new MimirMoveControllerPoseDocument( + PoseId: "move:0xA11CE:79", + WandId: "move:0xA11CE", + TrackingSpaceId: "mimir-stage-space", + CalibrationId: "mimir-move-stage-calibration-v1", + FusionAuthorityId: "mimir.runtime.move-fusion", + SourceTimestampNs: 1_781_202_600_004_000_000, + EstimatedAtNs: 1_781_202_600_006_200_000, + Sequence: 79, + PositionMeters: new MimirVector3Snapshot(0.0, 1.1, 1.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0), + AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.02, -0.01, 0.04), + Confidence: 0.82, + LatencyMilliseconds: 2.2, + Battery01: 0.72, + Buttons: + [ + new MimirTrackingButtonSnapshot("move", true, 1.0), + new MimirTrackingButtonSnapshot("trigger", true, 0.5) + ], + EvidenceStreamIds: + [ + "witness:0x00000000000A11CE", + "witness:0x0000000000000B0B", + "controller:0x00000000000A11CE" + ], + EvidenceKinds: + [ + "optical-marker:triangulated", + "controller-state:buttons-imu", + "orientation:imu-unresolved" + ], + ConsumerContract: "fensalir.move-controller-input"); + var admission = new MimirMuninnMoveEvidenceAdmission( + FrameId: "muninn:nightwing:move-evidence:79", + ProducerPeerId: "muninn:nightwing", + PublishedAtNs: 1_781_202_600_005_000_000, + MimirReadAtNs: 1_781_202_600_005_700_000, + SampleCount: 3, + OpticalMarkerCount: 2, + ControllerStateCount: 1, + SourceTimeMinNs: 1_781_202_600_004_000_000, + SourceTimeMaxNs: 1_781_202_600_004_000_000, + ArrivalMinNs: 1_781_202_600_004_500_000, + ArrivalMaxNs: 1_781_202_600_005_700_000, + Handle: new MimirNativeSampleHandle( + SensorIdHash: 0xF00D, + TimestampNs: 1_781_202_600_004_000_000, + ArrivalNs: 1_781_202_600_005_700_000, + Sequence: 79, + PayloadHandle: 0xBEEF, + Flags: 0, + Reserved: 0), + ReservoirEdgeNs: 1_781_202_600_004_000_000, + ReservoirWindowStartNs: 1_781_197_600_004_000_000, + ReservoirMoveEvidenceCount: 1); + var poseFrame = MimirMovePoseStream.CreateFrame( + frameId: "mimir:starfire:move-pose:79", + producerPeerId: "mimir:starfire", + publishedAtNs: pose.EstimatedAtNs, + trackingSpaceId: pose.TrackingSpaceId, + calibrationId: pose.CalibrationId, + poses: [pose]); + var surface = MimirMoveProofSurface.Create( + admission, + poseFrame, + "fensalir:starfire:presented-frame:79", + fensalirPresentedAtNs: 1_781_202_600_007_100_000); + var probeFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(surface); + var encoded = MessagePackSerializer.Serialize(surface); + var decoded = MessagePackSerializer.Deserialize(encoded); + var fallbackPose = pose with + { + EvidenceKinds = + [ + "optical-marker:single-ray-depth-prior", + "controller-state:buttons-imu", + "orientation:imu-unresolved" + ] + }; + var fallbackSurface = MimirMoveProofSurface.Create( + admission with { OpticalMarkerCount = 1 }, + MimirMovePoseStream.CreateFrame( + "mimir:starfire:move-pose:80", + "mimir:starfire", + fallbackPose.EstimatedAtNs, + fallbackPose.TrackingSpaceId, + fallbackPose.CalibrationId, + [fallbackPose]), + "fensalir:starfire:presented-frame:80", + fensalirPresentedAtNs: 1_781_202_600_008_100_000); + + Console.WriteLine( + $"move-proof-surface-smoke proof={surface.ProofId} chain={surface.MuninnEvidenceFrameId}->{surface.MimirEvidenceFrameId}->{surface.MimirPoseFrameId}->{surface.FensalirFrameId} verdict={surface.Verdict} splines={probeFrame.Splines.Count} encodedBytes={encoded.Length} fallbackVerdict={fallbackSurface.Verdict}"); + + return surface.IsFullPose && + surface.MuninnEvidenceFrameId == "muninn:nightwing:move-evidence:79" && + surface.MimirEvidenceFrameId == "mimir:starfire:move-evidence:79" && + surface.MimirPoseFrameId == "mimir:starfire:move-pose:79" && + surface.FensalirFrameId == "fensalir:starfire:presented-frame:79" && + surface.OpticalMarkerCount == 2 && + surface.ControllerStateCount == 1 && + surface.MimirReadAtNs > (ulong)surface.MuninnPublishedAtNs && + surface.FusionEstimatedAtNs >= surface.SourceTimestampNs && + surface.FensalirPresentedAtNs > (ulong)surface.FusionEstimatedAtNs && + probeFrame.HasInput && + probeFrame.Splines.Count >= 3 && + decoded.ProofId == surface.ProofId && + fallbackSurface.Verdict == MimirMoveProofVerdict.SingleRayFallback && + !fallbackSurface.IsFullPose && + fallbackSurface.FailureReason.Contains("single-ray", StringComparison.Ordinal) + ? 0 + : 1; +} + static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) { var protocol = MimirMoveCalibrationProtocol.CreateStarfireNightwingProtocol( diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs new file mode 100644 index 0000000..60545a8 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs @@ -0,0 +1,232 @@ +using Aquarium.Engine.Render; +using GameCult.Caching; +using MessagePack; +using System.Numerics; + +namespace Mimir.Runtime.Synchronization; + +public enum MimirMoveProofVerdict +{ + Unknown, + NoPose, + SingleRayFallback, + FullPose +} + +[CultDocument("mimir.move_proof_surface", "mimir.move_proof_surface.v1")] +[MessagePackObject] +public sealed record MimirMoveProofSurfaceDocument( + [property: Key(0)] + [property: CultName] + string ProofId, + [property: Key(1)] string MuninnEvidenceFrameId, + [property: Key(2)] string MimirEvidenceFrameId, + [property: Key(3)] string MimirPoseFrameId, + [property: Key(4)] string FensalirFrameId, + [property: Key(5)] string ProducerPeerId, + [property: Key(6)] string FusionAuthorityId, + [property: Key(7)] string TrackingSpaceId, + [property: Key(8)] string CalibrationId, + [property: Key(9)] long SourceTimestampNs, + [property: Key(10)] long MuninnPublishedAtNs, + [property: Key(11)] ulong MimirReadAtNs, + [property: Key(12)] ulong MimirAdmittedAtNs, + [property: Key(13)] long FusionEstimatedAtNs, + [property: Key(14)] ulong FensalirPresentedAtNs, + [property: Key(15)] ulong ReservoirEdgeNs, + [property: Key(16)] ulong ReservoirWindowStartNs, + [property: Key(17)] int OpticalMarkerCount, + [property: Key(18)] int ControllerStateCount, + [property: Key(19)] int PoseCount, + [property: Key(20)] double PoseConfidence, + [property: Key(21)] double LatencyMilliseconds, + [property: Key(22)] MimirMoveProofVerdict Verdict, + [property: Key(23)] string FailureReason, + [property: Key(24)] MimirMoveControllerPoseDocument[] Poses) +{ + [IgnoreMember] + public bool IsFullPose => Verdict == MimirMoveProofVerdict.FullPose; +} + +public static class MimirMoveProofSurface +{ + public const string ConsumerContract = "fensalir.move-proof-surface"; + + public static MimirMoveProofSurfaceDocument Create( + MimirMuninnMoveEvidenceAdmission admission, + MimirMoveControllerPoseStreamFrame poseFrame, + string fensalirFrameId, + ulong fensalirPresentedAtNs) + { + ArgumentNullException.ThrowIfNull(admission); + ArgumentNullException.ThrowIfNull(poseFrame); + ArgumentException.ThrowIfNullOrWhiteSpace(fensalirFrameId); + + var poses = poseFrame.Poses + .OrderBy(pose => pose.SourceTimestampNs) + .ThenBy(pose => pose.Sequence) + .ToArray(); + var selectedPose = poses + .OrderByDescending(pose => pose.Confidence) + .FirstOrDefault(); + var verdict = ResolveVerdict(poses); + var failureReason = ResolveFailureReason(admission, poses, verdict); + var sourceNs = selectedPose?.SourceTimestampNs ?? + checked((long)Math.Min(admission.SourceTimeMaxNs, (ulong)long.MaxValue)); + var estimatedAtNs = selectedPose?.EstimatedAtNs ?? 0L; + var confidence = selectedPose?.Confidence ?? 0.0; + var latencyMs = selectedPose?.LatencyMilliseconds ?? 0.0; + + return new MimirMoveProofSurfaceDocument( + ProofId: $"{poseFrame.ProducerPeerId}:move-proof:{SequenceSuffix(poseFrame.FrameId)}", + MuninnEvidenceFrameId: admission.FrameId, + MimirEvidenceFrameId: $"mimir:starfire:move-evidence:{SequenceSuffix(admission.FrameId)}", + MimirPoseFrameId: poseFrame.FrameId, + FensalirFrameId: fensalirFrameId, + ProducerPeerId: poseFrame.ProducerPeerId, + FusionAuthorityId: selectedPose?.FusionAuthorityId ?? "mimir.runtime.move-fusion", + TrackingSpaceId: poseFrame.TrackingSpaceId, + CalibrationId: poseFrame.CalibrationId, + SourceTimestampNs: sourceNs, + MuninnPublishedAtNs: admission.PublishedAtNs, + MimirReadAtNs: admission.MimirReadAtNs, + MimirAdmittedAtNs: admission.ArrivalMaxNs, + FusionEstimatedAtNs: estimatedAtNs, + FensalirPresentedAtNs: fensalirPresentedAtNs, + ReservoirEdgeNs: admission.ReservoirEdgeNs, + ReservoirWindowStartNs: admission.ReservoirWindowStartNs, + OpticalMarkerCount: admission.OpticalMarkerCount, + ControllerStateCount: admission.ControllerStateCount, + PoseCount: poses.Length, + PoseConfidence: Math.Clamp(confidence, 0.0, 1.0), + LatencyMilliseconds: Math.Max(0.0, latencyMs), + Verdict: verdict, + FailureReason: failureReason, + Poses: poses); + } + + public static AquariumSplineFrame BuildFensalirProbeFrame(MimirMoveProofSurfaceDocument surface) + { + ArgumentNullException.ThrowIfNull(surface); + if (surface.Poses.Length == 0) + { + return BuildFailureProbe(surface); + } + + var splines = new List(); + foreach (var pose in surface.Poses) + { + var center = ToVector(pose.PositionMeters); + var confidence = (float)Math.Clamp(pose.Confidence, 0.0, 1.0); + var color = surface.IsFullPose + ? new Vector4(0.15f, 1.35f, 0.92f, 0.92f) + : new Vector4(1.35f, 0.78f, 0.18f, 0.86f); + var radius = MathF.Max(0.03f, 0.08f * confidence); + splines.Add(new AquariumSpline3D( + $"move-proof-{SanitizeId(pose.PoseId)}-x", + [ + new(center + new Vector3(-radius, 0.0f, 0.0f), color), + new(center + new Vector3(radius, 0.0f, 0.0f), color) + ], + new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f), + CatmullRomSubdivisions: 1)); + splines.Add(new AquariumSpline3D( + $"move-proof-{SanitizeId(pose.PoseId)}-y", + [ + new(center + new Vector3(0.0f, -radius, 0.0f), color), + new(center + new Vector3(0.0f, radius, 0.0f), color) + ], + new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f), + CatmullRomSubdivisions: 1)); + splines.Add(new AquariumSpline3D( + $"move-proof-{SanitizeId(pose.PoseId)}-z", + [ + new(center + new Vector3(0.0f, 0.0f, -radius), color), + new(center + new Vector3(0.0f, 0.0f, radius), color) + ], + new AquariumSplineStyle(0.012f, 2.0f, 0.94f, 0.7f, 0.10f), + CatmullRomSubdivisions: 1)); + } + + return new AquariumSplineFrame { Splines = splines }; + } + + private static AquariumSplineFrame BuildFailureProbe(MimirMoveProofSurfaceDocument surface) + { + var color = new Vector4(1.2f, 0.2f, 0.12f, 0.85f); + var z = surface.OpticalMarkerCount > 0 ? 0.35f : 0.0f; + return new AquariumSplineFrame + { + Splines = + [ + new AquariumSpline3D( + $"move-proof-{SanitizeId(surface.ProofId)}-missing-pose", + [ + new(new Vector3(-0.16f, 0.0f, z), color), + new(new Vector3(0.16f, 0.0f, z), color), + new(new Vector3(0.0f, 0.16f, z), color), + new(new Vector3(-0.16f, 0.0f, z), color) + ], + new AquariumSplineStyle(0.01f, 1.2f, 0.85f, 0.8f, 0.12f), + CatmullRomSubdivisions: 1) + ] + }; + } + + private static MimirMoveProofVerdict ResolveVerdict(IReadOnlyList poses) + { + if (poses.Count == 0) + { + return MimirMoveProofVerdict.NoPose; + } + + return poses.Any(pose => pose.EvidenceKinds.Contains("optical-marker:triangulated", StringComparer.Ordinal)) + ? MimirMoveProofVerdict.FullPose + : MimirMoveProofVerdict.SingleRayFallback; + } + + private static string ResolveFailureReason( + MimirMuninnMoveEvidenceAdmission admission, + IReadOnlyList poses, + MimirMoveProofVerdict verdict) + { + if (verdict == MimirMoveProofVerdict.FullPose) + { + return ""; + } + + if (admission.OpticalMarkerCount == 0) + { + return "missing optical marker candidates"; + } + + if (admission.ControllerStateCount == 0) + { + return "missing controller state"; + } + + if (poses.Count == 0) + { + return "Mimir fusion produced no calibrated pose"; + } + + return "pose is single-ray fallback, not full calibrated optical fusion"; + } + + private static string SequenceSuffix(string frameId) + { + var separator = frameId.LastIndexOf(':'); + return separator >= 0 && separator < frameId.Length - 1 + ? frameId[(separator + 1)..] + : frameId; + } + + private static Vector3 ToVector(MimirVector3Snapshot value) => + new((float)value.X, (float)value.Y, (float)value.Z); + + private static string SanitizeId(string value) + { + var chars = value.Select(ch => char.IsLetterOrDigit(ch) ? ch : '-').ToArray(); + return new string(chars); + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index efae861..9b2d4e4 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -87,6 +87,23 @@ public sealed record MimirMuninnMoveEvidenceStreamFrame( [property: Key(3)] MuninnMoveMarkerCandidateDocument[] MarkerCandidates, [property: Key(4)] MuninnMoveControllerStateDocument[] ControllerStates); +public sealed record MimirMuninnMoveEvidenceAdmission( + string FrameId, + string ProducerPeerId, + long PublishedAtNs, + ulong MimirReadAtNs, + int SampleCount, + int OpticalMarkerCount, + int ControllerStateCount, + ulong SourceTimeMinNs, + ulong SourceTimeMaxNs, + ulong ArrivalMinNs, + ulong ArrivalMaxNs, + MimirNativeSampleHandle Handle, + ulong ReservoirEdgeNs, + ulong ReservoirWindowStartNs, + ulong ReservoirMoveEvidenceCount); + public static class MimirMuninnMoveEvidenceAdapter { private const ulong LiveIdentityWindowNs = 60_000_000_000UL; @@ -126,11 +143,30 @@ public static class MimirMuninnMoveEvidenceAdapter string trackingSpaceId, out MimirNativeSampleHandle handle, out int sampleCount) + { + var admitted = TryAdmitLatestCultMeshFrame( + ring, + runtime, + producerSourceId, + calibrationId, + trackingSpaceId, + out var admission); + handle = admission?.Handle ?? default; + sampleCount = admission?.SampleCount ?? 0; + return admitted; + } + + public static bool TryAdmitLatestCultMeshFrame( + CultMeshSharedMemoryFrameRing ring, + MimirNativeReservoirRuntime runtime, + string producerSourceId, + string calibrationId, + string trackingSpaceId, + out MimirMuninnMoveEvidenceAdmission? admission) { ArgumentNullException.ThrowIfNull(ring); ArgumentNullException.ThrowIfNull(runtime); - handle = default; - sampleCount = 0; + admission = null; if (!ring.TryAcquireLatestRead(out var lease)) { return false; @@ -138,15 +174,32 @@ public static class MimirMuninnMoveEvidenceAdapter using (lease) { + var mimirReadAtNs = NowUnixNs(); var frame = DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]); var samples = BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates); - sampleCount = samples.Count; - if (sampleCount == 0) + if (samples.Count == 0) { return false; } - handle = runtime.AdmitMoveEvidence(producerSourceId, samples, calibrationId, trackingSpaceId); + var handle = runtime.AdmitMoveEvidence(producerSourceId, samples, calibrationId, trackingSpaceId); + var status = runtime.Status; + admission = new MimirMuninnMoveEvidenceAdmission( + FrameId: frame.FrameId, + ProducerPeerId: frame.ProducerPeerId, + PublishedAtNs: frame.PublishedAtNs, + MimirReadAtNs: mimirReadAtNs, + SampleCount: samples.Count, + OpticalMarkerCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker), + ControllerStateCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState), + SourceTimeMinNs: samples.Min(sample => sample.SourceTimestampNs), + SourceTimeMaxNs: samples.Max(sample => sample.SourceTimestampNs), + ArrivalMinNs: samples.Min(sample => sample.ArrivalNs), + ArrivalMaxNs: samples.Max(sample => sample.ArrivalNs), + Handle: handle, + ReservoirEdgeNs: status.EdgeNs, + ReservoirWindowStartNs: status.WindowStartNs, + ReservoirMoveEvidenceCount: status.MoveEvidenceCount.ToUInt64()); return true; } } @@ -416,6 +469,14 @@ public static class MimirMuninnMoveEvidenceAdapter return hash == 0 ? 1 : hash; } + private static ulong NowUnixNs() + { + var now = DateTimeOffset.UtcNow; + var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL); + var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL); + return checked(secondsNs + tickRemainderNs); + } + private static ulong ObservedAtToUnixNs(string observedAt) { const string unixPrefix = "unix-"; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 1398959..2a19e5f 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -154,3 +154,9 @@ {"ts": "2026-06-17T13:16:04+01:00", "type": "verification", "status": "ok", "note": "Odin ef4c51a deployed to Starfire Muninn. Windows Move identity status now reads paired Bluetooth host from the sibling PS Move pairing collection when --move-host is omitted: no-flag move-identity-status reports A8 as usb-visible with bluetooth_host=5C:93:A2:9C:A8:A8, and the claim log shows controller=00:07:04:a8:00:d0 host_after=5c:93:a2:9c:a8:a8. Installed Starfire Muninn hash matches target/debug hash 6062D174FF9DF281C098B35E84E7D3488606F6EC8842D6B5F1C8B6ACBD30C7E7."} {"ts": "2026-06-17T13:27:55+01:00", "type": "verification", "status": "ok", "note": "Mimir live Muninn roster smoke after Odin ef4c51a and refreshed Nightwing store loaded Starfire+Nightwing identity stores 2/2, identities=6, roster=4, readinessCoherent=True. A8 now resolves as usb-owned on Starfire with bluetoothHost=5C:93:A2:9C:A8:A8 while Nightwing contributes bluetooth-unreachable for the same move id; F5 is bluetooth-connected, A3 is usb-owned on Nightwing, and A6 remains bluetooth-unreachable."} {"ts": "2026-06-17T14:05:52+01:00", "type": "verification", "status": "ok", "note": "After Odin b6f8119 deployed to Nightwing and Starfire, Muninn live source merge drops stale configured platform Move sources when platform discovery has live sources, so static --move-state joystick paths no longer keep unplugged Moves USB-owned. Mimir roster derivation now ignores identity snapshots older than the live identity window relative to each Move's newest observation. Focused identity smoke reports staleReleased=bluetooth-waiting/pickup-ready. Cross-host live roster smoke with refreshed Nightwing store loaded stores=2/2, identities=6, roster=4, readinessCoherent=True: F5 bluetooth-connected, A3 usb-owned on Starfire with Nightwing unreachable witness, A6 bluetooth-unreachable, A8 usb-owned on Nightwing."} +{"ts": "2026-07-04T00:00:00Z", "type": "transport-cut", "status": "ok", "note": "Archived the old Nightwing/EVE witness socket lane: Mimir.EveSensorReceiver, Mimir.VerseRecorder, tools/nightwing_typed_witness_publisher.py, and scripts/start-nightwing-move-tracking.ps1 now fail closed. Eye/Move witness publication and recording must use typed CultMesh/CultNet documents or CultMesh stream frames discovered through Odin; the old receiver/recorder/subscription path no longer owns live transport."} +{"ts": "2026-07-08T22:06:14+01:00", "type": "decision", "status": "ok", "note": "Move proof spine cut: Odin Muninn evidence frames now carry canonical muninn.move_marker_candidate.v1 records instead of a forced empty marker slice, with a bright-Y8 tracker unit proving non-empty marker candidates serialize into the Mimir-compatible CultMesh frame. Mimir TryAdmitLatestCultMeshFrame now exposes a typed admission receipt with frame id, producer, publish/read times, sample kind counts, source/arrival ranges, native handle, and reservoir edge/window. Verified cargo test -p muninn-daemon, dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --muninn-move-cultmesh-stream-smoke against native reservoir debug DLL."} +{"ts": "2026-07-08T22:10:10+01:00", "type": "decision", "status": "ok", "note": "Move optical extraction seam sharpened in Odin: Muninn daemon now has MoveMarkerFrameSource plus extract/publish functions that take one source-local Y8 frame, run muninn-move-tracker, build canonical muninn.move_marker_candidate.v1 records, and publish them through the same Mimir-compatible move-evidence stream. Full cargo test -p muninn-daemon passed. Live raw camera-frame ownership is still missing in Muninn serve; do not revive the archived Python/EVE witness lane to fake it."} +{"ts": "2026-07-08T22:18:44+01:00", "type": "decision", "status": "ok", "note": "Muninn serve now has a first daemon-owned Move marker camera producer path: --move-marker-camera = configures a source-local camera, the serve loop polls a platform frame reader, Unix builds use a V4L2 YUYV single-frame reader that converts to compact Y8, and the shared extraction/publish seam emits canonical muninn.move_marker_candidate.v1 records into the Mimir-compatible move-evidence stream. Verified on Windows with cargo test -p muninn-daemon (148 passed) and marker-camera unit tests; Linux cross-target check could not run because x86_64-unknown-linux-gnu target is not installed, and Nightwing hardware live proof remains pending."} +{"ts": "2026-07-08T22:21:57+01:00", "type": "verification", "status": "ok", "note": "Correction to the earlier Move marker camera evidence: after installing/using the x86_64-unknown-linux-gnu Rust target, Muninn's V4L2 YUYV Move marker camera producer compiled with cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu. cargo test -p muninn-daemon passes 148 tests. Nightwing hardware live proof remains pending, so this proves owned producer code and target compatibility, not calibrated live pose."} +{"ts": "2026-07-08T22:28:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofSurface as the first observer-only Fensalir-visible Move proof surface: it combines a Muninn evidence admission receipt, Mimir pose stream frame, and Fensalir presentation/probe timestamp into mimir.move_proof_surface.v1, lowers a dev AquariumSplineFrame probe, preserves source/publish/read/admit/fusion/present timing, and refuses to treat single-ray fallback as full pose. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-surface-smoke, and BufferSmoke --move-fusion-smoke. Live Nightwing hardware, calibrated camera witnesses, and actual Fensalir presented-frame capture remain pending."} diff --git a/state/map.yaml b/state/map.yaml index 8faf5b0..98be669 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -43,15 +43,14 @@ constraints: - CultLib is moving every runtime's typed CultNet/CultMesh document transport to `cultnet.transport.rudp.v0`. Health, provider advertisements, command boundaries, transport profiles, and program/media document lanes should use - RUDP by default. TCP/HTTP/WebSocket surfaces are renderer lowerings, + RUDP by default. Product/debug render surfaces are lowerings, compatibility probes, or migration debt unless a specific runtime has not earned its RUDP path yet. - Daemon health and freshness are separate in authority from media transport, - not separate because they need TCP. Mimir-facing Eve dashboards and browser - reference lowerers publish `idunn.daemon_health` records over - `cultnet.transport.rudp.v0` for Idunn. Idunn owns continuity once Odin has - accepted a daemon; Mimir-owned surfaces only publish their own health - witnesses. + not separate because they need TCP. The old Mimir Eve dashboard broker is + archived and no longer publishes Idunn health, deck routes, or WebSocket + commands. Idunn owns continuity once Odin has accepted a daemon; Mimir-owned + surfaces may only return as typed CultMesh/Odin state publishers. - Mimir explicitly trades latency and memory for throughput and field quality: the default five-second buffer is compute budget for stream alignment, GPU/audio analysis, and volumetric sound-field extraction. @@ -131,6 +130,13 @@ accepted_design: `mimir.move_controller_pose_stream_frame.v1` CultMesh shared-memory byte frames for Fensalir and other realtime consumers; it carries poses and does not own calibration or fusion decisions. + - `MimirMoveProofSurface` publishes the observer-only + `mimir.move_proof_surface.v1` receipt and Fensalir spline probe for the + named proof spine. It combines the Muninn evidence admission receipt, the + Mimir-owned pose frame, and a Fensalir presentation/probe timestamp without + becoming a pose owner. Single-ray fallback is surfaced as non-final; only + triangulated calibrated optical marker evidence can earn a full-pose + verdict. - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move calibration preflight as typed state: required Muninn Move evidence streams, optional Quest headset/controller pose witnesses over USB, @@ -147,18 +153,30 @@ accepted_design: shared-memory byte rings. Current remote media body transport is not same-machine zero-copy into Fensalir; Raven media uses CultNet/CultMesh documents over the mesh and lowers to local UDP only for OBS compatibility. - - `scripts/start-nightwing-move-tracking.ps1` restores the live Nightwing - Eye/Move witness path: Starfire owns the `/eve/periwinkle` receiver and - recorder, Nightwing owns PS Move HID report `0x06` LED writes and PS3 Eye - blob extraction, and the stream currently emits `mimir.psmove_light_state.v1` - plus `mimir.move_controller_observation_state.v1` optical witness records - before later fusion into full tracking poses. + - The old `scripts/start-nightwing-move-tracking.ps1` Nightwing Eye/Move + witness path is archived. Starfire no longer owns a receiver/recorder + socket lane for this evidence; Nightwing/Mimir witness publication must use + typed CultMesh stream frames or Odin-discovered CultMesh documents. - Odin's Muninn organ owns Move optical candidate extraction in `E:\Projects\Odin\crates\muninn-move-tracker` and publishes/feeds `muninn.move_marker_candidate.v1` records. Mimir consumes those candidate streams; it does not own raw optical extraction. Mimir explicitly owns calibration, multi-camera association, triangulation, IMU fusion, prediction, and final 6DoF wand pose. + - Muninn's Move evidence stream frame now uses canonical + `muninn.move_marker_candidate.v1` records for optical candidates instead of + a hardcoded empty marker slice. A daemon unit proves a bright Y8 frame + extracted by `muninn-move-tracker` serializes as non-empty Mimir-compatible + evidence. The daemon now has a source-local Y8 extraction/publish seam and + a first `serve` camera producer: `--move-marker-camera =` + polls a Unix V4L2 YUYV frame, converts it to compact Y8, and feeds that + same seam. Unit tests prove the configured camera tick publishes marker + evidence through the shared frame contract; Nightwing hardware/live V4L2 + proof is still pending. + - `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a + typed admission receipt with decoded frame id, producer, publish/read + timestamps, sample counts by kind, source/arrival ranges, native reservoir + handle, and reservoir edge/window so the named proof spine can be traced. - `MimirFrameEventProcessStreamSource` is a temporary diagnostic adapter for JSON-line video frame and audio block metadata from native probes into runtime rolling buffers; it does not carry pixels/audio payloads or own the @@ -387,19 +405,20 @@ accepted_design: - `src/Mimir.CultMeshMedia` owns the first explicit CultMesh media/body bridge. Raven publishes muxed H.264/AAC MPEG-TS chunks as rolling `mimir.cultmesh_media_frame` documents over CultNet reliable UDP, - Yggdrasil relays them on `10.77.0.1:3075`, and Starfire lowers the - subscribed stream to `udp://127.0.0.1:5200` for the OBS source - `Raven Monitor + Realtek`. The bridge now uses explicit CultLib RUDP - socket/session helpers for relay, sender, and receiver. The old - `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` node/client path is no - longer a transport authority for this media lane. - - `src/Mimir.EveDashboard` publishes `mimir.eve_dashboard_state` through - CultNet/CultMesh and can push `idunn.daemon_health` records over - `cultnet.transport.rudp.v0` for itself and a paired service daemon. Its - intended daemon shape is to publish provider advertisement, retained state, - command boundary, and transport profile over RUDP as typed records; HTTP and - WebSocket remain client lowerings. It owns Eve surface publication and - dashboard broker health only; it does not own daemon lifecycle. + Yggdrasil relay routing comes from + `cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av` through + Odin/CultMesh discovery, and Starfire lowers the subscribed stream to local + UDP for the OBS source `Raven Monitor + Realtek`. The bridge now uses + explicit CultLib RUDP socket/session helpers for relay, sender, and + receiver. The old `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` + node/client path and raw sender/receiver host-port config are no longer + transport authorities for this media lane. + - `src/Mimir.EveDashboard` is archived as a fail-closed tombstone. It no + longer opens a TCP listener, serves `/eve/deck`, accepts WebSocket commands, + or publishes Idunn health. Its replacement must publish provider + advertisement, retained state, command boundary, and transport profile as + typed CultMesh/Odin documents; product/debug render surfaces remain client + lowerings. - `src/Mimir.EveBrowserReference` serves a static Eve/browser lowering and can publish its own `idunn.daemon_health` record over `cultnet.transport.rudp.v0`. It is a renderer/reference health witness, not @@ -432,15 +451,16 @@ active_subgoals: replaces them. - Launch Raven's CultMesh sender directly on Raven because SSH to Raven over WireGuard timed out during the initial relay deployment pass. - - Keep Mimir Eve dashboard and browser reference health publication aligned - with Odin/Idunn doctrine: Odin accepts Verse/service visibility, Idunn owns - keepalive, and Mimir surfaces publish typed RUDP self-health rather than - private watchdog loops or TCP/HTTP status truth. - - Cut remaining older CultMesh node/client transport assumptions in - `src/Mimir.EveDashboard` and witness tooling: daemon truth, provider - catalogs, command boundaries, and typed program document paths should use - explicit CultNet RUDP helpers; HTTP/WebSocket remain render/client - lowerings only. `src/Mimir.CultMeshMedia` has completed this cut for the + - Keep Mimir Eve dashboard and browser reference aligned with Odin/Idunn + doctrine: Odin accepts Verse/service visibility, Idunn owns keepalive, and + Mimir surfaces publish typed CultMesh/Odin state rather than private + watchdog loops, product status probes, or socket brokers. + - Rebuild `src/Mimir.EveDashboard` only after daemon truth, provider catalogs, + command boundaries, and typed program document paths use explicit CultMesh/ + Odin documents. `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the + Nightwing typed witness publisher, and `start-nightwing-move-tracking.ps1` + are archived so the old witness subscription lane cannot retake transport + authority. `src/Mimir.CultMeshMedia` has completed this cut for the Raven/Yggdrasil/Starfire media-frame lane. current_status: From 7d96c42a3c98e4e5ed659f3982c7bdd7a31e94ee Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 8 Jul 2026 22:35:32 +0100 Subject: [PATCH 066/135] Add Mimir Move proof pipeline --- docs/implementation-plan.md | 8 + notes/current-system-map.md | 5 + src/Mimir.BufferSmoke/Program.cs | 145 ++++++++++++++++++ .../Synchronization/MimirMoveProofPipeline.cs | 130 ++++++++++++++++ .../Synchronization/MimirMoveProofSurface.cs | 7 +- state/evidence.jsonl | 1 + state/map.yaml | 6 + 7 files changed, 300 insertions(+), 2 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveProofPipeline.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 9779636..4fec194 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -82,6 +82,14 @@ a named invariant that the native runtime cannot protect yet. `muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79` chain, and rejects single-ray fallback as not full pose. +- `MimirMoveProofPipeline` is the shared commit primitive for the live Move + proof path. It starts from a Muninn CultMesh shared-memory evidence frame, + admits samples to the native `move_evidence` reservoir, runs Mimir-owned + fusion, creates the Mimir pose stream frame, creates the proof surface, and + returns the Fensalir spline probe in one pass. `Mimir.BufferSmoke + --move-proof-pipeline-smoke` proves that shape against the native reservoir + debug DLL; live hardware and presented-frame capture still need to call this + primitive instead of hand-assembling a parallel path. - `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for Starfire/Nightwing Moves: required Muninn evidence streams, optional Quest headset/controller pose witnesses, stillness/sweep/validation phases, and diff --git a/notes/current-system-map.md b/notes/current-system-map.md index b3a3b6d..1970e21 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -207,6 +207,11 @@ frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only chain. It does not decide pose. Single-ray fallback remains a visible non-final verdict until calibrated multi-camera optical evidence earns full pose. +`MimirMoveProofPipeline` is now the reusable commit primitive for that chain: +from one Muninn CultMesh shared-memory frame it performs native admission, +Mimir fusion, pose-frame creation, proof-surface creation, and Fensalir probe +lowering. Live Nightwing/Starfire wiring should call this owner path rather +than preserving separate manual actions and programmatic proof paths. ## Program Composition diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index e22e014..d7cf594 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -140,6 +140,12 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-surface-smoke", StringCompa return RunMoveProofSurfaceSmoke(); } +if (args.Any(arg => string.Equals(arg, "--move-proof-pipeline-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofPipelineSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); +} + if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", StringComparison.OrdinalIgnoreCase))) { return await RunMoveCalibrationProtocolSmokeAsync( @@ -2217,6 +2223,145 @@ static int RunMoveProofSurfaceSmoke() : 1; } +static int RunMoveProofPipelineSmoke(string nativeReservoirPath) +{ + const ulong leftWitnessHash = 0x00000000000A11CE; + const ulong rightWitnessHash = 0x0000000000000B0B; + const string observedAt = "2026-06-11T18:30:00.0040000Z"; + var streamId = "muninn:nightwing:move-evidence"; + var frame = new MimirMuninnMoveEvidenceStreamFrame( + FrameId: "muninn:nightwing:move-evidence:79", + ProducerPeerId: "muninn:nightwing", + PublishedAtNs: 1_781_202_600_005_000_000, + MarkerCandidates: + [ + new MuninnMoveMarkerCandidateDocument( + StreamId: "nightwing:ps3-eye-0:move-marker-candidates", + HostId: "nightwing", + CameraId: "ps3-eye-0", + FrameSequence: 9101, + SourceIdHash: leftWitnessHash, + TileX: 10, + TileY: 10, + CenterXPx: 170.0f, + CenterYPx: 120.0f, + RadiusPx: 12.0f, + AreaPx: 452, + MeanLuma: 0.64f, + PeakLuma: 252, + Score: 0.90f, + ObservedAt: observedAt), + new MuninnMoveMarkerCandidateDocument( + StreamId: "starfire:ps3-eye-1:move-marker-candidates", + HostId: "starfire", + CameraId: "ps3-eye-1", + FrameSequence: 9102, + SourceIdHash: rightWitnessHash, + TileX: 11, + TileY: 10, + CenterXPx: 150.0f, + CenterYPx: 120.0f, + RadiusPx: 12.0f, + AreaPx: 449, + MeanLuma: 0.62f, + PeakLuma: 250, + Score: 0.86f, + ObservedAt: observedAt) + ], + ControllerStates: + [ + new MuninnMoveControllerStateDocument( + StreamId: "nightwing:move-usb:move-controller-state", + HostId: "nightwing", + MoveId: "move-usb", + Sequence: 79, + SourceTimestampNs: 1_781_202_600_004_000_000, + AccelerometerXyz: [-0.01f, 0.10f, 0.99f], + GyroscopeXyz: [0.02f, -0.01f, 0.04f], + MagnetometerXyz: [0.0f, 0.0f, 0.0f], + TriggerValue: 0.5f, + Buttons: ["move", "trigger"], + Battery01: 0.72f, + ObservedAt: observedAt) + ]); + var calibration = new MimirMoveFusionRigCalibration( + CalibrationId: "mimir-move-stage-calibration-v1", + TrackingSpaceId: "mimir-stage-space", + Cameras: + [ + new MimirMoveFusionCameraCalibration( + CameraId: "nightwing:ps3-eye-0", + WitnessIdHash: leftWitnessHash, + PositionMeters: new MimirVector3Snapshot(-0.1, 0.0, 0.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx: 100.0, + FocalLengthYPx: 100.0, + PrincipalPointXPx: 160.0, + PrincipalPointYPx: 120.0), + new MimirMoveFusionCameraCalibration( + CameraId: "starfire:ps3-eye-1", + WitnessIdHash: rightWitnessHash, + PositionMeters: new MimirVector3Snapshot(0.1, 0.0, 0.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx: 100.0, + FocalLengthYPx: 100.0, + PrincipalPointXPx: 160.0, + PrincipalPointYPx: 120.0) + ], + MaximumAssociationSkewMilliseconds: 20.0); + var catalog = CultMesh.CreateStreamCatalog(); + catalog.Declare(MimirMuninnMoveEvidenceAdapter.CreateStreamDescriptor( + streamId, + "mimir-live", + "muninn:nightwing", + "nightwing:clock")); + using var ring = catalog.CreateSharedMemoryRing(streamId, slotCount: 4, slotByteLength: 8192); + var payload = MimirMuninnMoveEvidenceAdapter.SerializeStreamFrame(frame); + if (!ring.TryPublishCopy(payload, frame.PublishedAtNs, durationNs: 0, out var published)) + { + Console.Error.WriteLine("move-proof-pipeline-smoke publish-failed"); + return 1; + } + + catalog.PublishFrame(published); + using var runtime = new MimirNativeReservoirRuntime(nativeReservoirPath); + var options = new MimirMoveProofPipelineOptions( + MimirEvidenceSourceId: "mimir:starfire:move-evidence", + MimirEvidenceFrameId: "mimir:starfire:move-evidence:79", + MimirPoseFrameId: "mimir:starfire:move-pose:79", + MimirPoseProducerPeerId: "mimir:starfire", + FensalirFrameId: "fensalir:starfire:presented-frame:79", + FensalirPresentedAtNs: 1_781_202_600_007_100_000); + var built = MimirMoveProofPipeline.TryBuildLatestFromRing( + ring, + runtime, + calibration, + options, + out var result); + + Console.WriteLine( + $"move-proof-pipeline-smoke built={built} proof={result?.ProofSurface.ProofId ?? "none"} chain={result?.ProofSurface.MuninnEvidenceFrameId ?? "none"}->{result?.ProofSurface.MimirEvidenceFrameId ?? "none"}->{result?.ProofSurface.MimirPoseFrameId ?? "none"}->{result?.ProofSurface.FensalirFrameId ?? "none"} verdict={result?.ProofSurface.Verdict.ToString() ?? "none"} samples={result?.Admission.SampleCount ?? 0} poses={result?.PoseFrame.Poses.Length ?? 0} splines={result?.FensalirProbeFrame.Splines.Count ?? 0} reservoirEdgeNs={result?.Admission.ReservoirEdgeNs ?? 0}"); + + return built && + result is not null && + result.Admission.FrameId == frame.FrameId && + result.Admission.OpticalMarkerCount == 2 && + result.Admission.ControllerStateCount == 1 && + result.Fusion.Poses.Count == 1 && + result.PoseFrame.FrameId == options.MimirPoseFrameId && + result.PoseFrame.ProducerPeerId == options.MimirPoseProducerPeerId && + result.PoseFrame.Poses.Length == 1 && + result.ProofSurface.IsFullPose && + result.ProofSurface.MimirEvidenceFrameId == options.MimirEvidenceFrameId && + result.ProofSurface.MimirPoseFrameId == options.MimirPoseFrameId && + result.ProofSurface.FensalirFrameId == options.FensalirFrameId && + result.FensalirProbeFrame.HasInput && + result.Admission.Handle.PayloadHandle != 0 && + runtime.Status.MoveEvidenceCount.ToUInt64() == 1 + ? 0 + : 1; +} + static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) { var protocol = MimirMoveCalibrationProtocol.CreateStarfireNightwingProtocol( diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofPipeline.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofPipeline.cs new file mode 100644 index 0000000..97bc983 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofPipeline.cs @@ -0,0 +1,130 @@ +using Aquarium.Engine.Render; +using GameCult.Mesh; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirMoveProofPipelineResult( + MimirMuninnMoveEvidenceAdmission Admission, + MimirMoveFusionResult Fusion, + MimirMoveControllerPoseStreamFrame PoseFrame, + MimirMoveProofSurfaceDocument ProofSurface, + AquariumSplineFrame FensalirProbeFrame); + +public sealed record MimirMoveProofPipelineOptions( + string MimirEvidenceSourceId, + string MimirEvidenceFrameId, + string MimirPoseFrameId, + string MimirPoseProducerPeerId, + string FensalirFrameId, + ulong FensalirPresentedAtNs, + string FusionAuthorityId = "mimir.runtime.move-fusion", + string ConsumerContract = "fensalir.move-controller-input"); + +public static class MimirMoveProofPipeline +{ + public static bool TryBuildLatestFromRing( + CultMeshSharedMemoryFrameRing ring, + MimirNativeReservoirRuntime runtime, + MimirMoveFusionRigCalibration calibration, + MimirMoveProofPipelineOptions options, + out MimirMoveProofPipelineResult? result) + { + ArgumentNullException.ThrowIfNull(ring); + result = null; + if (!ring.TryAcquireLatestRead(out var lease)) + { + return false; + } + + using (lease) + { + var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]); + return TryBuild( + frame, + runtime, + calibration, + options, + out result); + } + } + + public static bool TryBuild( + MimirMuninnMoveEvidenceStreamFrame frame, + MimirNativeReservoirRuntime runtime, + MimirMoveFusionRigCalibration calibration, + MimirMoveProofPipelineOptions options, + out MimirMoveProofPipelineResult? result) + { + ArgumentNullException.ThrowIfNull(frame); + ArgumentNullException.ThrowIfNull(runtime); + ArgumentNullException.ThrowIfNull(calibration); + ArgumentNullException.ThrowIfNull(options); + result = null; + + var mimirReadAtNs = NowUnixNs(); + var samples = MimirMuninnMoveEvidenceAdapter.BuildNativeSamples(frame.MarkerCandidates, frame.ControllerStates); + if (samples.Count == 0) + { + return false; + } + + var handle = runtime.AdmitMoveEvidence( + options.MimirEvidenceSourceId, + samples, + calibration.CalibrationId, + calibration.TrackingSpaceId); + var status = runtime.Status; + var admission = new MimirMuninnMoveEvidenceAdmission( + FrameId: frame.FrameId, + ProducerPeerId: frame.ProducerPeerId, + PublishedAtNs: frame.PublishedAtNs, + MimirReadAtNs: mimirReadAtNs, + SampleCount: samples.Count, + OpticalMarkerCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker), + ControllerStateCount: samples.Count(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState), + SourceTimeMinNs: samples.Min(sample => sample.SourceTimestampNs), + SourceTimeMaxNs: samples.Max(sample => sample.SourceTimestampNs), + ArrivalMinNs: samples.Min(sample => sample.ArrivalNs), + ArrivalMaxNs: samples.Max(sample => sample.ArrivalNs), + Handle: handle, + ReservoirEdgeNs: status.EdgeNs, + ReservoirWindowStartNs: status.WindowStartNs, + ReservoirMoveEvidenceCount: status.MoveEvidenceCount.ToUInt64()); + var fusion = MimirMoveFusion.Fuse( + samples, + calibration, + options.FusionAuthorityId, + options.ConsumerContract); + var poseFrame = MimirMovePoseStream.CreateFrame( + options.MimirPoseFrameId, + options.MimirPoseProducerPeerId, + publishedAtNs: fusion.Poses.Count == 0 + ? checked((long)Math.Min(admission.ArrivalMaxNs, (ulong)long.MaxValue)) + : fusion.Poses.Max(pose => pose.EstimatedAtNs), + calibration.TrackingSpaceId, + calibration.CalibrationId, + fusion.Poses); + var proofSurface = MimirMoveProofSurface.Create( + admission, + poseFrame, + options.FensalirFrameId, + options.FensalirPresentedAtNs, + options.MimirEvidenceFrameId); + var probeFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(proofSurface); + result = new MimirMoveProofPipelineResult( + admission, + fusion, + poseFrame, + proofSurface, + probeFrame); + return true; + } + + private static ulong NowUnixNs() + { + var now = DateTimeOffset.UtcNow; + var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL); + var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL); + return checked(secondsNs + tickRemainderNs); + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs index 60545a8..0b2c953 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofSurface.cs @@ -56,7 +56,8 @@ public static class MimirMoveProofSurface MimirMuninnMoveEvidenceAdmission admission, MimirMoveControllerPoseStreamFrame poseFrame, string fensalirFrameId, - ulong fensalirPresentedAtNs) + ulong fensalirPresentedAtNs, + string? mimirEvidenceFrameId = null) { ArgumentNullException.ThrowIfNull(admission); ArgumentNullException.ThrowIfNull(poseFrame); @@ -80,7 +81,9 @@ public static class MimirMoveProofSurface return new MimirMoveProofSurfaceDocument( ProofId: $"{poseFrame.ProducerPeerId}:move-proof:{SequenceSuffix(poseFrame.FrameId)}", MuninnEvidenceFrameId: admission.FrameId, - MimirEvidenceFrameId: $"mimir:starfire:move-evidence:{SequenceSuffix(admission.FrameId)}", + MimirEvidenceFrameId: string.IsNullOrWhiteSpace(mimirEvidenceFrameId) + ? $"mimir:starfire:move-evidence:{SequenceSuffix(admission.FrameId)}" + : mimirEvidenceFrameId, MimirPoseFrameId: poseFrame.FrameId, FensalirFrameId: fensalirFrameId, ProducerPeerId: poseFrame.ProducerPeerId, diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 2a19e5f..5c1abf7 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -160,3 +160,4 @@ {"ts": "2026-07-08T22:18:44+01:00", "type": "decision", "status": "ok", "note": "Muninn serve now has a first daemon-owned Move marker camera producer path: --move-marker-camera = configures a source-local camera, the serve loop polls a platform frame reader, Unix builds use a V4L2 YUYV single-frame reader that converts to compact Y8, and the shared extraction/publish seam emits canonical muninn.move_marker_candidate.v1 records into the Mimir-compatible move-evidence stream. Verified on Windows with cargo test -p muninn-daemon (148 passed) and marker-camera unit tests; Linux cross-target check could not run because x86_64-unknown-linux-gnu target is not installed, and Nightwing hardware live proof remains pending."} {"ts": "2026-07-08T22:21:57+01:00", "type": "verification", "status": "ok", "note": "Correction to the earlier Move marker camera evidence: after installing/using the x86_64-unknown-linux-gnu Rust target, Muninn's V4L2 YUYV Move marker camera producer compiled with cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu. cargo test -p muninn-daemon passes 148 tests. Nightwing hardware live proof remains pending, so this proves owned producer code and target compatibility, not calibrated live pose."} {"ts": "2026-07-08T22:28:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofSurface as the first observer-only Fensalir-visible Move proof surface: it combines a Muninn evidence admission receipt, Mimir pose stream frame, and Fensalir presentation/probe timestamp into mimir.move_proof_surface.v1, lowers a dev AquariumSplineFrame probe, preserves source/publish/read/admit/fusion/present timing, and refuses to treat single-ray fallback as full pose. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-surface-smoke, and BufferSmoke --move-fusion-smoke. Live Nightwing hardware, calibrated camera witnesses, and actual Fensalir presented-frame capture remain pending."} +{"ts": "2026-07-08T22:34:00+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofPipeline as the shared commit primitive for one Move proof frame: it reads a Muninn CultMesh shared-memory evidence frame, admits samples to native move_evidence, runs MimirMoveFusion, emits the mimir:starfire move-pose frame, creates mimir.move_proof_surface.v1, and lowers the Fensalir spline probe in one pass. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --move-proof-pipeline-smoke against the native reservoir debug DLL. This proves the reusable Mimir owner path; live Nightwing hardware, calibrated camera witnesses, and actual presented-frame capture remain pending."} diff --git a/state/map.yaml b/state/map.yaml index 98be669..a449a0a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -137,6 +137,12 @@ accepted_design: becoming a pose owner. Single-ray fallback is surfaced as non-final; only triangulated calibrated optical marker evidence can earn a full-pose verdict. + - `MimirMoveProofPipeline` is the single shared commit primitive for one + Move proof frame. It reads a Muninn CultMesh evidence frame, admits native + `move_evidence`, runs Mimir fusion, emits the Mimir pose frame, creates the + proof surface, and returns the Fensalir spline probe. Direct user actions, + programmatic smokes, and future live subscribers should use this same path + so the proof chain does not split into separately repaired truths. - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move calibration preflight as typed state: required Muninn Move evidence streams, optional Quest headset/controller pose witnesses over USB, From 02c6bfc533148f2d1d1e4c75c0c14ccb0956bb79 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 8 Jul 2026 22:42:20 +0100 Subject: [PATCH 067/135] Attach Move proof surface to runtime frame --- docs/implementation-plan.md | 16 +++-- notes/current-system-map.md | 6 ++ src/Mimir.BufferSmoke/Program.cs | 107 ++++++++++++++++++++++++++++++ src/Mimir.Runtime/MimirRuntime.cs | 57 +++++++++++++++- state/evidence.jsonl | 1 + state/map.yaml | 6 ++ 6 files changed, 188 insertions(+), 5 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 4fec194..1f0769b 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -90,6 +90,13 @@ a named invariant that the native runtime cannot protect yet. --move-proof-pipeline-smoke` proves that shape against the native reservoir debug DLL; live hardware and presented-frame capture still need to call this primitive instead of hand-assembling a parallel path. +- `MimirRuntime` is now the live Fensalir attachment point for proof surfaces. + `PublishMoveProofSurface` accepts the Mimir-owned + `mimir.move_proof_surface.v1` document, `CreateFrame` composes its + observer-only `move-proof-*` splines into `AquariumFrame.Scene.SplineFrame`, + and `Mimir.BufferSmoke --move-proof-runtime-frame-smoke` proves the frame is + empty before publish, contains the proof splines after publish, and clears + without leaving a repair loop behind. - `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for Starfire/Nightwing Moves: required Muninn evidence streams, optional Quest headset/controller pose witnesses, stillness/sweep/validation phases, and @@ -398,10 +405,11 @@ a named invariant that the native runtime cannot protect yet. becomes a deterministic timeline anchor before the actuator moves samples. 6. Prove the bioacoustic hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. -7. Attach the Move proof surface to the live runtime/Fensalir presentation path: - ingest real Muninn Nightwing and Starfire evidence, require calibrated - optical witnesses, publish `mimir:starfire:move-pose:`, and capture - a presented Fensalir frame/probe for the same sequence. +7. Wire live Move proof production into `MimirRuntime.PublishMoveProofSurface`: + ingest real Muninn Nightwing and Starfire evidence through + `MimirMoveProofPipeline`, require calibrated optical witnesses, publish + `mimir:starfire:move-pose:`, and capture a presented Fensalir + frame/probe for the same sequence. 8. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 9. Implement the Mimir program scene graph as the shared commit primitive for diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 1970e21..f492596 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -212,6 +212,12 @@ from one Muninn CultMesh shared-memory frame it performs native admission, Mimir fusion, pose-frame creation, proof-surface creation, and Fensalir probe lowering. Live Nightwing/Starfire wiring should call this owner path rather than preserving separate manual actions and programmatic proof paths. +`MimirRuntime` now owns the actual Fensalir presentation attachment for the +proof surface: `PublishMoveProofSurface` stores the latest Mimir-owned +`mimir.move_proof_surface.v1`, and `CreateFrame` merges its observer-only +`move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside the +audio spectrum preview. Fensalir renders this frame; it still does not decide +proof. ## Program Composition diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index d7cf594..22f15cb 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -2,10 +2,12 @@ using System.Diagnostics; using System.Numerics; using System.Security.Cryptography; using System.Text.Json; +using Aquarium.Engine; using GameCult.Caching; using GameCult.Caching.MessagePack; using GameCult.Mesh; using MessagePack; +using Mimir.Runtime; using Mimir.Runtime.Synchronization; if (args.Any(arg => string.Equals(arg, "--chirplet-self-test", StringComparison.OrdinalIgnoreCase))) @@ -146,6 +148,11 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-pipeline-smoke", StringComp ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-frame-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeFrameSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", StringComparison.OrdinalIgnoreCase))) { return await RunMoveCalibrationProtocolSmokeAsync( @@ -2223,6 +2230,106 @@ static int RunMoveProofSurfaceSmoke() : 1; } +static int RunMoveProofRuntimeFrameSmoke() +{ + var surface = CreateMoveProofSurfaceSmokeDocument(); + var runtime = new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: null), + new MimirSynchronizationSettings()); + var emptyFrame = runtime.Frame; + runtime.PublishMoveProofSurface(surface); + var proofFrame = runtime.Frame; + var proofSplineCount = proofFrame.Scene.SplineFrame.Splines.Count(spline => + spline.Id.StartsWith("move-proof-", StringComparison.Ordinal)); + runtime.ClearMoveProofSurface(); + var clearedFrame = runtime.Frame; + + Console.WriteLine( + $"move-proof-runtime-frame-smoke proof={surface.ProofId} frame={surface.FensalirFrameId} verdict={surface.Verdict} before={emptyFrame.Scene.SplineFrame.Splines.Count} proofSplines={proofSplineCount} afterClear={clearedFrame.Scene.SplineFrame.Splines.Count}"); + + return emptyFrame.Scene.SplineFrame.Splines.Count == 0 && + proofFrame.Scene.SplineFrame.HasInput && + proofSplineCount >= 3 && + proofFrame.Scene.SplineFrame.Splines.Any(spline => spline.Id.Contains("move-0xA11CE-79", StringComparison.Ordinal)) && + clearedFrame.Scene.SplineFrame.Splines.Count == 0 + ? 0 + : 1; +} + +static MimirMoveProofSurfaceDocument CreateMoveProofSurfaceSmokeDocument() +{ + var pose = new MimirMoveControllerPoseDocument( + PoseId: "move:0xA11CE:79", + WandId: "move:0xA11CE", + TrackingSpaceId: "mimir-stage-space", + CalibrationId: "mimir-move-stage-calibration-v1", + FusionAuthorityId: "mimir.runtime.move-fusion", + SourceTimestampNs: 1_781_202_600_004_000_000, + EstimatedAtNs: 1_781_202_600_006_200_000, + Sequence: 79, + PositionMeters: new MimirVector3Snapshot(0.0, 1.1, 1.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0), + AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.02, -0.01, 0.04), + Confidence: 0.82, + LatencyMilliseconds: 2.2, + Battery01: 0.72, + Buttons: + [ + new MimirTrackingButtonSnapshot("move", true, 1.0), + new MimirTrackingButtonSnapshot("trigger", true, 0.5) + ], + EvidenceStreamIds: + [ + "witness:0x00000000000A11CE", + "witness:0x0000000000000B0B", + "controller:0x00000000000A11CE" + ], + EvidenceKinds: + [ + "optical-marker:triangulated", + "controller-state:buttons-imu", + "orientation:imu-unresolved" + ], + ConsumerContract: "fensalir.move-controller-input"); + var admission = new MimirMuninnMoveEvidenceAdmission( + FrameId: "muninn:nightwing:move-evidence:79", + ProducerPeerId: "muninn:nightwing", + PublishedAtNs: 1_781_202_600_005_000_000, + MimirReadAtNs: 1_781_202_600_005_700_000, + SampleCount: 3, + OpticalMarkerCount: 2, + ControllerStateCount: 1, + SourceTimeMinNs: 1_781_202_600_004_000_000, + SourceTimeMaxNs: 1_781_202_600_004_000_000, + ArrivalMinNs: 1_781_202_600_004_500_000, + ArrivalMaxNs: 1_781_202_600_005_700_000, + Handle: new MimirNativeSampleHandle( + SensorIdHash: 0xF00D, + TimestampNs: 1_781_202_600_004_000_000, + ArrivalNs: 1_781_202_600_005_700_000, + Sequence: 79, + PayloadHandle: 0xBEEF, + Flags: 0, + Reserved: 0), + ReservoirEdgeNs: 1_781_202_600_004_000_000, + ReservoirWindowStartNs: 1_781_197_600_004_000_000, + ReservoirMoveEvidenceCount: 1); + var poseFrame = MimirMovePoseStream.CreateFrame( + frameId: "mimir:starfire:move-pose:79", + producerPeerId: "mimir:starfire", + publishedAtNs: pose.EstimatedAtNs, + trackingSpaceId: pose.TrackingSpaceId, + calibrationId: pose.CalibrationId, + poses: [pose]); + + return MimirMoveProofSurface.Create( + admission, + poseFrame, + "fensalir:starfire:presented-frame:79", + fensalirPresentedAtNs: 1_781_202_600_007_100_000); +} + static int RunMoveProofPipelineSmoke(string nativeReservoirPath) { const ulong leftWitnessHash = 0x00000000000A11CE; diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index f485689..c789ded 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -62,6 +62,7 @@ direct private IReadOnlyList lastAudioSynchronizationReports = []; private IReadOnlyList lastAudioSpectra = []; private readonly Queue> spectrumHistory = new(); + private MimirMoveProofSurfaceDocument? latestMoveProofSurface; public MimirRuntime(AquariumRuntimeOptions options) : this(options, MimirRuntimeConfiguration.Load()) @@ -144,6 +145,9 @@ direct : bufferFieldFrame.HasInput ? new Vector3(0.0f, 0.0f, 12.0f) : new Vector3(0.0f, 0.0f, cameraZ + 12.0f); + var spectrumSplineFrame = ShouldRenderSpectrumPreviewSplines() + ? BuildSpectrumSplineFrame() + : AquariumSplineFrame.Empty; return new AquariumFrame( new ViewFrame(Vector2.Zero, 24.0f), cameraPosition, @@ -155,7 +159,7 @@ direct TraceHeightFieldSurface = false, UseStarfieldBackground = false, BufferFieldFrame = bufferFieldFrame, - SplineFrame = ShouldRenderSpectrumPreviewSplines() ? BuildSpectrumSplineFrame() : AquariumSplineFrame.Empty, + SplineFrame = ComposeFensalirSplineFrame(spectrumSplineFrame, latestMoveProofSurface), }); } @@ -166,6 +170,44 @@ direct synchronization.AddSource(source); } + public void PublishMoveProofSurface(MimirMoveProofSurfaceDocument surface) + { + ArgumentNullException.ThrowIfNull(surface); + latestMoveProofSurface = surface; + } + + public void ClearMoveProofSurface() + { + latestMoveProofSurface = null; + } + + public static AquariumSplineFrame ComposeFensalirSplineFrame( + AquariumSplineFrame baseFrame, + MimirMoveProofSurfaceDocument? moveProofSurface) + { + ArgumentNullException.ThrowIfNull(baseFrame); + if (moveProofSurface is null) + { + return baseFrame; + } + + var proofFrame = MimirMoveProofSurface.BuildFensalirProbeFrame(moveProofSurface); + if (!baseFrame.HasInput) + { + return proofFrame; + } + + if (!proofFrame.HasInput) + { + return baseFrame; + } + + return new AquariumSplineFrame + { + Splines = baseFrame.Splines.Concat(proofFrame.Splines).ToArray() + }; + } + public void Start() { Console.WriteLine($"Mimir runtime sync buffers: {synchronization.Summary()} @ {synchronization.Settings.BufferDuration.TotalSeconds:0.###}s audioSync={audioSyncSettings.Mode} reference={audioSyncSettings.ReferenceSourceId}"); @@ -231,6 +273,8 @@ direct panel.Readout("Audio sync state", DescribeAudioSyncState); panel.Readout("Aligned audio", DescribeAlignedAudio); panel.Readout("Chirplet reference", DescribeChirpletReference); + panel.Section("Move Proof"); + panel.Readout("Latest proof", DescribeMoveProofSurface); panel.Section("Live FFT"); panel.Readout("Spectrum cadence", () => sceneReady ? $"{lastAudioSpectra.Count} spectra fft={lastAudioSpectra.FirstOrDefault()?.FftSize ?? 0} analyze={lastSpectrumAnalysisMilliseconds:0.00}ms" @@ -246,6 +290,17 @@ direct }); } + private string DescribeMoveProofSurface() + { + var surface = latestMoveProofSurface; + if (surface is null) + { + return "No Move proof surface published."; + } + + return $"{surface.ProofId} verdict={surface.Verdict} confidence={surface.PoseConfidence:0.000} latency={surface.LatencyMilliseconds:0.00}ms chain={surface.MuninnEvidenceFrameId}->{surface.MimirEvidenceFrameId}->{surface.MimirPoseFrameId}->{surface.FensalirFrameId}"; + } + private static AquariumRenderPlan CreateRenderPlan() { var app = new AquariumApp(); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 5c1abf7..64c55a9 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -161,3 +161,4 @@ {"ts": "2026-07-08T22:21:57+01:00", "type": "verification", "status": "ok", "note": "Correction to the earlier Move marker camera evidence: after installing/using the x86_64-unknown-linux-gnu Rust target, Muninn's V4L2 YUYV Move marker camera producer compiled with cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu. cargo test -p muninn-daemon passes 148 tests. Nightwing hardware live proof remains pending, so this proves owned producer code and target compatibility, not calibrated live pose."} {"ts": "2026-07-08T22:28:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofSurface as the first observer-only Fensalir-visible Move proof surface: it combines a Muninn evidence admission receipt, Mimir pose stream frame, and Fensalir presentation/probe timestamp into mimir.move_proof_surface.v1, lowers a dev AquariumSplineFrame probe, preserves source/publish/read/admit/fusion/present timing, and refuses to treat single-ray fallback as full pose. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-surface-smoke, and BufferSmoke --move-fusion-smoke. Live Nightwing hardware, calibrated camera witnesses, and actual Fensalir presented-frame capture remain pending."} {"ts": "2026-07-08T22:34:00+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofPipeline as the shared commit primitive for one Move proof frame: it reads a Muninn CultMesh shared-memory evidence frame, admits samples to native move_evidence, runs MimirMoveFusion, emits the mimir:starfire move-pose frame, creates mimir.move_proof_surface.v1, and lowers the Fensalir spline probe in one pass. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --move-proof-pipeline-smoke against the native reservoir debug DLL. This proves the reusable Mimir owner path; live Nightwing hardware, calibrated camera witnesses, and actual presented-frame capture remain pending."} +{"ts": "2026-07-08T22:41:32+01:00", "type": "implementation", "status": "ok", "note": "Attached the Move proof surface to the actual MimirRuntime/Fensalir frame path: MimirRuntime.PublishMoveProofSurface stores the latest Mimir-owned mimir.move_proof_surface.v1 document, CreateFrame composes its observer-only move-proof-* splines into AquariumFrame.Scene.SplineFrame, and the UI exposes a display-only proof readout. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke against the native reservoir debug DLL. Live Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} diff --git a/state/map.yaml b/state/map.yaml index a449a0a..62eb6fd 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -143,6 +143,12 @@ accepted_design: proof surface, and returns the Fensalir spline probe. Direct user actions, programmatic smokes, and future live subscribers should use this same path so the proof chain does not split into separately repaired truths. + - `MimirRuntime` now owns the actual Fensalir presentation attachment for + Move proof surfaces. `PublishMoveProofSurface` accepts the latest + Mimir-owned proof document, and `CreateFrame` composes its observer-only + `move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside + the audio spectrum preview. Fensalir remains a renderer, not a proof + authority. - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move calibration preflight as typed state: required Muninn Move evidence streams, optional Quest headset/controller pose witnesses over USB, From 20a296a4a98741f3229fa6060b0eea40dfc200a1 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 8 Jul 2026 22:49:42 +0100 Subject: [PATCH 068/135] Drive Move proof surface from runtime ring --- docs/implementation-plan.md | 17 +- notes/current-system-map.md | 7 + src/Mimir.BufferSmoke/Program.cs | 157 +++++++++++++++++- src/Mimir.Runtime/MimirRuntime.cs | 44 +++++ .../MimirMoveProofRuntimeDriver.cs | 94 +++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 6 + 7 files changed, 320 insertions(+), 6 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeDriver.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 1f0769b..3a0f3ee 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -97,6 +97,13 @@ a named invariant that the native runtime cannot protect yet. and `Mimir.BufferSmoke --move-proof-runtime-frame-smoke` proves the frame is empty before publish, contains the proof splines after publish, and clears without leaving a repair loop behind. +- `MimirMoveProofRuntimeDriver` is the bounded runtime bridge from a Muninn + CultMesh shared-memory evidence ring to `MimirRuntime.PublishMoveProofSurface`. + It derives the Mimir evidence, Mimir pose, and Fensalir frame ids from the + actual Muninn frame suffix, calls the shared proof pipeline, and suppresses + duplicate frame admission. `Mimir.BufferSmoke --move-proof-runtime-driver-smoke` + proves `MimirRuntime.Update` can pull one ring frame into the same-sequence + proof chain and visible spline frame. - `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for Starfire/Nightwing Moves: required Muninn evidence streams, optional Quest headset/controller pose witnesses, stillness/sweep/validation phases, and @@ -405,11 +412,11 @@ a named invariant that the native runtime cannot protect yet. becomes a deterministic timeline anchor before the actuator moves samples. 6. Prove the bioacoustic hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. -7. Wire live Move proof production into `MimirRuntime.PublishMoveProofSurface`: - ingest real Muninn Nightwing and Starfire evidence through - `MimirMoveProofPipeline`, require calibrated optical witnesses, publish - `mimir:starfire:move-pose:`, and capture a presented Fensalir - frame/probe for the same sequence. +7. Wire real Move proof production into `MimirMoveProofRuntimeDriver`: attach + real Muninn Nightwing and Starfire evidence rings, require calibrated optical + witnesses, publish `mimir:starfire:move-pose:` through the shared + runtime path, and capture a presented Fensalir frame/probe for the same + sequence. 8. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 9. Implement the Mimir program scene graph as the shared commit primitive for diff --git a/notes/current-system-map.md b/notes/current-system-map.md index f492596..bf21807 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -218,6 +218,13 @@ proof surface: `PublishMoveProofSurface` stores the latest Mimir-owned `move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside the audio spectrum preview. Fensalir renders this frame; it still does not decide proof. +`MimirMoveProofRuntimeDriver` is the runtime bridge from a Muninn CultMesh +shared-memory evidence ring into that attachment point. It derives the +Mimir evidence, Mimir pose, and Fensalir frame ids from the actual Muninn frame +suffix, calls `MimirMoveProofPipeline`, suppresses duplicate frame admission, +and lets `MimirRuntime.Update` publish the resulting proof surface. This proves +the runtime-owned path for a ring frame; real Nightwing hardware and calibrated +camera witnesses are still the missing field proof. ## Program Composition diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 22f15cb..0762eb7 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -3,6 +3,7 @@ using System.Numerics; using System.Security.Cryptography; using System.Text.Json; using Aquarium.Engine; +using Aquarium.Engine.Input; using GameCult.Caching; using GameCult.Caching.MessagePack; using GameCult.Mesh; @@ -153,6 +154,12 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-frame-smoke", Strin return RunMoveProofRuntimeFrameSmoke(); } +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-driver-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeDriverSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); +} + if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", StringComparison.OrdinalIgnoreCase))) { return await RunMoveCalibrationProtocolSmokeAsync( @@ -2234,7 +2241,7 @@ static int RunMoveProofRuntimeFrameSmoke() { var surface = CreateMoveProofSurfaceSmokeDocument(); var runtime = new MimirRuntime( - new AquariumRuntimeOptions(Headless: true, CultCachePath: null), + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), new MimirSynchronizationSettings()); var emptyFrame = runtime.Frame; runtime.PublishMoveProofSurface(surface); @@ -2256,6 +2263,154 @@ static int RunMoveProofRuntimeFrameSmoke() : 1; } +static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) +{ + if (!File.Exists(nativeReservoirPath)) + { + Console.Error.WriteLine($"move-proof-runtime-driver-smoke missing-native path={nativeReservoirPath}"); + return 1; + } + + const ulong leftWitnessHash = 0x00000000000A11CE; + const ulong rightWitnessHash = 0x0000000000000B0B; + const string observedAt = "2026-06-11T18:30:00.0040000Z"; + var streamId = "muninn:nightwing:move-evidence"; + var frame = new MimirMuninnMoveEvidenceStreamFrame( + FrameId: "muninn:nightwing:move-evidence:79", + ProducerPeerId: "muninn:nightwing", + PublishedAtNs: 1_781_202_600_005_000_000, + MarkerCandidates: + [ + new MuninnMoveMarkerCandidateDocument( + StreamId: "nightwing:ps3-eye-0:move-marker-candidates", + HostId: "nightwing", + CameraId: "ps3-eye-0", + FrameSequence: 9101, + SourceIdHash: leftWitnessHash, + TileX: 10, + TileY: 10, + CenterXPx: 170.0f, + CenterYPx: 120.0f, + RadiusPx: 12.0f, + AreaPx: 452, + MeanLuma: 0.64f, + PeakLuma: 252, + Score: 0.90f, + ObservedAt: observedAt), + new MuninnMoveMarkerCandidateDocument( + StreamId: "starfire:ps3-eye-1:move-marker-candidates", + HostId: "starfire", + CameraId: "ps3-eye-1", + FrameSequence: 9102, + SourceIdHash: rightWitnessHash, + TileX: 11, + TileY: 10, + CenterXPx: 150.0f, + CenterYPx: 120.0f, + RadiusPx: 12.0f, + AreaPx: 449, + MeanLuma: 0.62f, + PeakLuma: 250, + Score: 0.86f, + ObservedAt: observedAt) + ], + ControllerStates: + [ + new MuninnMoveControllerStateDocument( + StreamId: "nightwing:move-usb:move-controller-state", + HostId: "nightwing", + MoveId: "move-usb", + Sequence: 79, + SourceTimestampNs: 1_781_202_600_004_000_000, + AccelerometerXyz: [-0.01f, 0.10f, 0.99f], + GyroscopeXyz: [0.02f, -0.01f, 0.04f], + MagnetometerXyz: [0.0f, 0.0f, 0.0f], + TriggerValue: 0.5f, + Buttons: ["move", "trigger"], + Battery01: 0.72f, + ObservedAt: observedAt) + ]); + var calibration = new MimirMoveFusionRigCalibration( + CalibrationId: "mimir-move-stage-calibration-v1", + TrackingSpaceId: "mimir-stage-space", + Cameras: + [ + new MimirMoveFusionCameraCalibration( + CameraId: "nightwing:ps3-eye-0", + WitnessIdHash: leftWitnessHash, + PositionMeters: new MimirVector3Snapshot(-0.1, 0.0, 0.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx: 100.0, + FocalLengthYPx: 100.0, + PrincipalPointXPx: 160.0, + PrincipalPointYPx: 120.0), + new MimirMoveFusionCameraCalibration( + CameraId: "starfire:ps3-eye-1", + WitnessIdHash: rightWitnessHash, + PositionMeters: new MimirVector3Snapshot(0.1, 0.0, 0.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx: 100.0, + FocalLengthYPx: 100.0, + PrincipalPointXPx: 160.0, + PrincipalPointYPx: 120.0) + ], + MaximumAssociationSkewMilliseconds: 20.0); + var catalog = CultMesh.CreateStreamCatalog(); + catalog.Declare(MimirMuninnMoveEvidenceAdapter.CreateStreamDescriptor( + streamId, + "mimir-live", + "muninn:nightwing", + "nightwing:clock")); + using var ring = catalog.CreateSharedMemoryRing(streamId, slotCount: 4, slotByteLength: 8192); + var payload = MimirMuninnMoveEvidenceAdapter.SerializeStreamFrame(frame); + if (!ring.TryPublishCopy(payload, frame.PublishedAtNs, durationNs: 0, out var published)) + { + Console.Error.WriteLine("move-proof-runtime-driver-smoke publish-failed"); + return 1; + } + + catalog.PublishFrame(published); + using var reservoir = new MimirNativeReservoirRuntime(nativeReservoirPath); + var driver = new MimirMoveProofRuntimeDriver( + ring, + reservoir, + calibration, + new MimirMoveProofRuntimeDriverOptions( + MimirEvidenceSourceId: "mimir:starfire:move-evidence", + MimirEvidenceFramePrefix: "mimir:starfire:move-evidence", + MimirPoseFramePrefix: "mimir:starfire:move-pose", + MimirPoseProducerPeerId: "mimir:starfire", + FensalirFramePrefix: "fensalir:starfire:presented-frame")); + var runtime = new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), + new MimirSynchronizationSettings()); + runtime.RegisterMoveProofDriver(driver); + runtime.OnSceneReady(); + var input = new InputState(); + runtime.Update(0.016f, input); + var proofFrame = runtime.Frame; + var proofSurface = driver.LastResult?.ProofSurface; + var proofSplineCount = proofFrame.Scene.SplineFrame.Splines.Count(spline => + spline.Id.StartsWith("move-proof-", StringComparison.Ordinal)); + runtime.Update(0.016f, input); + var duplicateSuppressed = reservoir.Status.MoveEvidenceCount.ToUInt64() == 1UL; + + Console.WriteLine( + $"move-proof-runtime-driver-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed}"); + + return proofSurface is not null && + proofSurface.IsFullPose && + proofSurface.MuninnEvidenceFrameId == "muninn:nightwing:move-evidence:79" && + proofSurface.MimirEvidenceFrameId == "mimir:starfire:move-evidence:79" && + proofSurface.MimirPoseFrameId == "mimir:starfire:move-pose:79" && + proofSurface.FensalirFrameId == "fensalir:starfire:presented-frame:79" && + proofFrame.Scene.SplineFrame.HasInput && + proofSplineCount >= 3 && + duplicateSuppressed + ? 0 + : 1; +} + static MimirMoveProofSurfaceDocument CreateMoveProofSurfaceSmokeDocument() { var pose = new MimirMoveControllerPoseDocument( diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index c789ded..cdaa522 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -62,6 +62,7 @@ direct private IReadOnlyList lastAudioSynchronizationReports = []; private IReadOnlyList lastAudioSpectra = []; private readonly Queue> spectrumHistory = new(); + private readonly List moveProofDrivers = []; private MimirMoveProofSurfaceDocument? latestMoveProofSurface; public MimirRuntime(AquariumRuntimeOptions options) @@ -181,6 +182,12 @@ direct latestMoveProofSurface = null; } + public void RegisterMoveProofDriver(MimirMoveProofRuntimeDriver driver) + { + ArgumentNullException.ThrowIfNull(driver); + moveProofDrivers.Add(driver); + } + public static AquariumSplineFrame ComposeFensalirSplineFrame( AquariumSplineFrame baseFrame, MimirMoveProofSurfaceDocument? moveProofSurface) @@ -223,6 +230,7 @@ direct runtimeSeconds += Math.Max(deltaSeconds, 0.0f); lastPollCount = synchronization.PollSources(); + UpdateMoveProofSurfaces(); QueueCalibrationTimeline(); UpdateAudioSpectra(); UpdateAudioSynchronization(); @@ -275,6 +283,7 @@ direct panel.Readout("Chirplet reference", DescribeChirpletReference); panel.Section("Move Proof"); panel.Readout("Latest proof", DescribeMoveProofSurface); + panel.Readout("Proof drivers", DescribeMoveProofDrivers); panel.Section("Live FFT"); panel.Readout("Spectrum cadence", () => sceneReady ? $"{lastAudioSpectra.Count} spectra fft={lastAudioSpectra.FirstOrDefault()?.FftSize ?? 0} analyze={lastSpectrumAnalysisMilliseconds:0.00}ms" @@ -301,6 +310,41 @@ direct return $"{surface.ProofId} verdict={surface.Verdict} confidence={surface.PoseConfidence:0.000} latency={surface.LatencyMilliseconds:0.00}ms chain={surface.MuninnEvidenceFrameId}->{surface.MimirEvidenceFrameId}->{surface.MimirPoseFrameId}->{surface.FensalirFrameId}"; } + private string DescribeMoveProofDrivers() + { + if (moveProofDrivers.Count == 0) + { + return "No Move proof runtime drivers registered."; + } + + return string.Join(Environment.NewLine, moveProofDrivers.Select(driver => driver.Describe())); + } + + private void UpdateMoveProofSurfaces() + { + if (moveProofDrivers.Count == 0) + { + return; + } + + var presentedAtNs = NowUnixNs(); + foreach (var driver in moveProofDrivers) + { + if (driver.TryBuildLatest(presentedAtNs, out var result) && result is not null) + { + PublishMoveProofSurface(result.ProofSurface); + } + } + } + + private static ulong NowUnixNs() + { + var now = DateTimeOffset.UtcNow; + var secondsNs = checked((ulong)now.ToUnixTimeSeconds() * 1_000_000_000UL); + var tickRemainderNs = checked((ulong)(now.Ticks % TimeSpan.TicksPerSecond) * 100UL); + return checked(secondsNs + tickRemainderNs); + } + private static AquariumRenderPlan CreateRenderPlan() { var app = new AquariumApp(); diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeDriver.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeDriver.cs new file mode 100644 index 0000000..04b5e89 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeDriver.cs @@ -0,0 +1,94 @@ +using GameCult.Mesh; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirMoveProofRuntimeDriverOptions( + string MimirEvidenceSourceId, + string MimirEvidenceFramePrefix, + string MimirPoseFramePrefix, + string MimirPoseProducerPeerId, + string FensalirFramePrefix, + string FusionAuthorityId = "mimir.runtime.move-fusion", + string ConsumerContract = "fensalir.move-controller-input"); + +public sealed class MimirMoveProofRuntimeDriver +{ + private readonly CultMeshSharedMemoryFrameRing ring; + private readonly MimirNativeReservoirRuntime reservoir; + private readonly MimirMoveFusionRigCalibration calibration; + private readonly MimirMoveProofRuntimeDriverOptions options; + private string lastFrameId = ""; + + public MimirMoveProofRuntimeDriver( + CultMeshSharedMemoryFrameRing ring, + MimirNativeReservoirRuntime reservoir, + MimirMoveFusionRigCalibration calibration, + MimirMoveProofRuntimeDriverOptions options) + { + this.ring = ring ?? throw new ArgumentNullException(nameof(ring)); + this.reservoir = reservoir ?? throw new ArgumentNullException(nameof(reservoir)); + this.calibration = calibration ?? throw new ArgumentNullException(nameof(calibration)); + this.options = options ?? throw new ArgumentNullException(nameof(options)); + ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirEvidenceSourceId); + ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirEvidenceFramePrefix); + ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirPoseFramePrefix); + ArgumentException.ThrowIfNullOrWhiteSpace(options.MimirPoseProducerPeerId); + ArgumentException.ThrowIfNullOrWhiteSpace(options.FensalirFramePrefix); + } + + public MimirMoveProofPipelineResult? LastResult { get; private set; } + + public bool TryBuildLatest(ulong fensalirPresentedAtNs, out MimirMoveProofPipelineResult? result) + { + result = null; + if (!ring.TryAcquireLatestRead(out var lease)) + { + return false; + } + + using (lease) + { + var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(lease.Memory[..lease.Handle.ByteLength]); + if (string.Equals(frame.FrameId, lastFrameId, StringComparison.Ordinal)) + { + return false; + } + + var sequence = SequenceSuffix(frame.FrameId); + var pipelineOptions = new MimirMoveProofPipelineOptions( + MimirEvidenceSourceId: options.MimirEvidenceSourceId, + MimirEvidenceFrameId: $"{options.MimirEvidenceFramePrefix}:{sequence}", + MimirPoseFrameId: $"{options.MimirPoseFramePrefix}:{sequence}", + MimirPoseProducerPeerId: options.MimirPoseProducerPeerId, + FensalirFrameId: $"{options.FensalirFramePrefix}:{sequence}", + FensalirPresentedAtNs: fensalirPresentedAtNs, + FusionAuthorityId: options.FusionAuthorityId, + ConsumerContract: options.ConsumerContract); + + if (!MimirMoveProofPipeline.TryBuild(frame, reservoir, calibration, pipelineOptions, out result)) + { + return false; + } + + lastFrameId = frame.FrameId; + LastResult = result; + return true; + } + } + + public string Describe() + { + var last = LastResult?.ProofSurface; + return last is null + ? $"{options.MimirEvidenceSourceId} waiting" + : $"{last.ProofId} verdict={last.Verdict} chain={last.MuninnEvidenceFrameId}->{last.MimirEvidenceFrameId}->{last.MimirPoseFrameId}->{last.FensalirFrameId}"; + } + + private static string SequenceSuffix(string frameId) + { + var separator = frameId.LastIndexOf(':'); + return separator >= 0 && separator < frameId.Length - 1 + ? frameId[(separator + 1)..] + : frameId; + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 64c55a9..a78d59c 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -162,3 +162,4 @@ {"ts": "2026-07-08T22:28:32+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofSurface as the first observer-only Fensalir-visible Move proof surface: it combines a Muninn evidence admission receipt, Mimir pose stream frame, and Fensalir presentation/probe timestamp into mimir.move_proof_surface.v1, lowers a dev AquariumSplineFrame probe, preserves source/publish/read/admit/fusion/present timing, and refuses to treat single-ray fallback as full pose. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-surface-smoke, and BufferSmoke --move-fusion-smoke. Live Nightwing hardware, calibrated camera witnesses, and actual Fensalir presented-frame capture remain pending."} {"ts": "2026-07-08T22:34:00+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofPipeline as the shared commit primitive for one Move proof frame: it reads a Muninn CultMesh shared-memory evidence frame, admits samples to native move_evidence, runs MimirMoveFusion, emits the mimir:starfire move-pose frame, creates mimir.move_proof_surface.v1, and lowers the Fensalir spline probe in one pass. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --move-proof-pipeline-smoke against the native reservoir debug DLL. This proves the reusable Mimir owner path; live Nightwing hardware, calibrated camera witnesses, and actual presented-frame capture remain pending."} {"ts": "2026-07-08T22:41:32+01:00", "type": "implementation", "status": "ok", "note": "Attached the Move proof surface to the actual MimirRuntime/Fensalir frame path: MimirRuntime.PublishMoveProofSurface stores the latest Mimir-owned mimir.move_proof_surface.v1 document, CreateFrame composes its observer-only move-proof-* splines into AquariumFrame.Scene.SplineFrame, and the UI exposes a display-only proof readout. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke against the native reservoir debug DLL. Live Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} +{"ts": "2026-07-08T22:48:42+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofRuntimeDriver as the runtime bridge from a Muninn CultMesh shared-memory evidence ring to MimirRuntime.PublishMoveProofSurface. The driver derives mimir:starfire evidence/pose ids and fensalir:starfire presented-frame id from the actual Muninn frame suffix, calls MimirMoveProofPipeline, and suppresses duplicate frame admission. Verified dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke against the native reservoir debug DLL, --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke. This proves the runtime-owned ring-to-visible-probe path with synthetic ring input; real Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} diff --git a/state/map.yaml b/state/map.yaml index 62eb6fd..d8fd33b 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -149,6 +149,12 @@ accepted_design: `move-proof-*` probe splines into `AquariumFrame.Scene.SplineFrame` beside the audio spectrum preview. Fensalir remains a renderer, not a proof authority. + - `MimirMoveProofRuntimeDriver` is the runtime bridge from a Muninn CultMesh + shared-memory evidence ring into `MimirRuntime.PublishMoveProofSurface`. + It derives the Mimir evidence, Mimir pose, and Fensalir frame ids from the + actual Muninn frame suffix, calls `MimirMoveProofPipeline`, and suppresses + duplicate frame admission so the live loop cannot re-admit one frame and + impersonate progress. - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move calibration preflight as typed state: required Muninn Move evidence streams, optional Quest headset/controller pose witnesses over USB, From 293a953b2d7d0d3d10bbecef5b80d8a17e135673 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Wed, 8 Jul 2026 22:56:52 +0100 Subject: [PATCH 069/135] Prove Move proof surface in presented frame --- docs/implementation-plan.md | 10 +- notes/current-system-map.md | 7 ++ .../Mimir.BufferSmoke.csproj | 1 + src/Mimir.BufferSmoke/Program.cs | 117 ++++++++++++++++++ src/Mimir.Runtime/MimirRuntime.cs | 5 + .../MimirMoveProofDevSurface.cs | 97 +++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 5 + 8 files changed, 241 insertions(+), 2 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveProofDevSurface.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 3a0f3ee..ab1ba2e 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -104,6 +104,12 @@ a named invariant that the native runtime cannot protect yet. duplicate frame admission. `Mimir.BufferSmoke --move-proof-runtime-driver-smoke` proves `MimirRuntime.Update` can pull one ring frame into the same-sequence proof chain and visible spline frame. +- `MimirMoveProofDevSurface` is a dev-only bootstrap gated by + `MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so + `Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` + headless, capture the Fensalir-presented PNG, and pixel-check the named + proof probe. This verifies the renderer/probe layer; it is not live + Nightwing hardware evidence. - `MimirMoveCalibrationProtocol` publishes the typed calibration preflight for Starfire/Nightwing Moves: required Muninn evidence streams, optional Quest headset/controller pose witnesses, stillness/sweep/validation phases, and @@ -415,8 +421,8 @@ a named invariant that the native runtime cannot protect yet. 7. Wire real Move proof production into `MimirMoveProofRuntimeDriver`: attach real Muninn Nightwing and Starfire evidence rings, require calibrated optical witnesses, publish `mimir:starfire:move-pose:` through the shared - runtime path, and capture a presented Fensalir frame/probe for the same - sequence. + runtime path, and replace the dev-gated presented-frame smoke with a + same-sequence capture from real field evidence. 8. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. 9. Implement the Mimir program scene graph as the shared commit primitive for diff --git a/notes/current-system-map.md b/notes/current-system-map.md index bf21807..57ad56b 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -225,6 +225,13 @@ suffix, calls `MimirMoveProofPipeline`, suppresses duplicate frame admission, and lets `MimirRuntime.Update` publish the resulting proof surface. This proves the runtime-owned path for a ring frame; real Nightwing hardware and calibrated camera witnesses are still the missing field proof. +`MimirMoveProofDevSurface` is a dev-only bootstrap gated by +`MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof +surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so +`Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` +headless, capture the actual Fensalir-presented PNG, and pixel-check that the +named proof probe is visible. This is a renderer/probe proof, not live hardware +evidence. ## Program Composition diff --git a/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj b/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj index 8cf0920..e4eb296 100644 --- a/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj +++ b/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj @@ -10,6 +10,7 @@ + diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 0762eb7..18a5bc0 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1,4 +1,5 @@ using System.Diagnostics; +using System.Drawing; using System.Numerics; using System.Security.Cryptography; using System.Text.Json; @@ -160,6 +161,12 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-driver-smoke", Stri ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--move-proof-presented-frame-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofPresentedFrameSmoke( + ParseStringOption(args, "--output", "artifacts/runtime/move-proof-presented-frame-smoke.png")); +} + if (args.Any(arg => string.Equals(arg, "--move-calibration-protocol-smoke", StringComparison.OrdinalIgnoreCase))) { return await RunMoveCalibrationProtocolSmokeAsync( @@ -2411,6 +2418,116 @@ static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) : 1; } +static int RunMoveProofPresentedFrameSmoke(string outputPath) +{ + var absoluteOutputPath = Path.GetFullPath(outputPath); + Directory.CreateDirectory(Path.GetDirectoryName(absoluteOutputPath) ?? "."); + if (File.Exists(absoluteOutputPath)) + { + File.Delete(absoluteOutputPath); + } + + var startInfo = new ProcessStartInfo("dotnet") + { + UseShellExecute = false, + RedirectStandardOutput = true, + RedirectStandardError = true, + CreateNoWindow = true, + WorkingDirectory = Environment.CurrentDirectory, + }; + startInfo.ArgumentList.Add("run"); + startInfo.ArgumentList.Add("--project"); + startInfo.ArgumentList.Add(Path.GetFullPath("src/Mimir.App/Mimir.App.csproj")); + startInfo.ArgumentList.Add("--"); + startInfo.ArgumentList.Add("--headless"); + startInfo.ArgumentList.Add("--headless-width"); + startInfo.ArgumentList.Add("640"); + startInfo.ArgumentList.Add("--headless-height"); + startInfo.ArgumentList.Add("360"); + startInfo.ArgumentList.Add("--capture-frame"); + startInfo.ArgumentList.Add(absoluteOutputPath); + startInfo.Environment[MimirMoveProofDevSurface.EnableEnvironmentVariable] = "1"; + startInfo.Environment["AQUARIUM_HEADLESS_READY_FRAMES"] = "2"; + + using var process = Process.Start(startInfo); + if (process is null) + { + Console.Error.WriteLine("move-proof-presented-frame-smoke start-failed"); + return 1; + } + + var stdoutTask = process.StandardOutput.ReadToEndAsync(); + var stderrTask = process.StandardError.ReadToEndAsync(); + if (!process.WaitForExit((int)TimeSpan.FromSeconds(90).TotalMilliseconds)) + { + try + { + process.Kill(entireProcessTree: true); + } + catch + { + } + + Console.Error.WriteLine("move-proof-presented-frame-smoke timeout"); + return 1; + } + + var stdout = stdoutTask.GetAwaiter().GetResult(); + var stderr = stderrTask.GetAwaiter().GetResult(); + if (!string.IsNullOrWhiteSpace(stdout)) + { + Console.Write(stdout); + } + + if (!string.IsNullOrWhiteSpace(stderr)) + { + Console.Error.Write(stderr); + } + + if (process.ExitCode != 0 || !File.Exists(absoluteOutputPath)) + { + Console.Error.WriteLine($"move-proof-presented-frame-smoke failed exit={process.ExitCode} path={absoluteOutputPath}"); + return 1; + } + + var brightPixels = CountBrightPixels(absoluteOutputPath, threshold: 42); + var length = new FileInfo(absoluteOutputPath).Length; + var surface = MimirMoveProofDevSurface.Create(); + var captured = stdout.Contains("Headless Aquarium frame captured", StringComparison.Ordinal); + Console.WriteLine( + $"move-proof-presented-frame-smoke proof={surface.ProofId} frame={surface.FensalirFrameId} verdict={surface.Verdict} path={absoluteOutputPath} bytes={length} brightPixels={brightPixels} captured={captured}"); + + return surface.MuninnEvidenceFrameId == "muninn:nightwing:move-evidence:79" && + surface.MimirEvidenceFrameId == "mimir:starfire:move-evidence:79" && + surface.MimirPoseFrameId == "mimir:starfire:move-pose:79" && + surface.FensalirFrameId == "fensalir:starfire:presented-frame:79" && + surface.IsFullPose && + captured && + length > 2048 && + brightPixels > 12 + ? 0 + : 1; +} + +static int CountBrightPixels(string path, int threshold) +{ + using var bitmap = new Bitmap(path); + var count = 0; + for (var y = 0; y < bitmap.Height; y++) + { + for (var x = 0; x < bitmap.Width; x++) + { + var pixel = bitmap.GetPixel(x, y); + if (pixel.R + pixel.G + pixel.B >= threshold) + { + count++; + } + } + } + + return count; +} + static MimirMoveProofSurfaceDocument CreateMoveProofSurfaceSmokeDocument() { var pose = new MimirMoveControllerPoseDocument( diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index cdaa522..032e8aa 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -115,6 +115,11 @@ direct : null; nextAudioSyncSeconds = audioSyncUpdateIntervalSeconds; nextTelemetrySeconds = telemetryIntervalSeconds; + if (MimirMoveProofDevSurface.IsEnabled()) + { + PublishMoveProofSurface(MimirMoveProofDevSurface.Create()); + } + ui = CreateUi(); } diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofDevSurface.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofDevSurface.cs new file mode 100644 index 0000000..55d8fa9 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofDevSurface.cs @@ -0,0 +1,97 @@ +namespace Mimir.Runtime.Synchronization; + +public static class MimirMoveProofDevSurface +{ + public const string EnableEnvironmentVariable = "MIMIR_MOVE_PROOF_DEV_SURFACE"; + + public static bool IsEnabled() => + IsTruthy(Environment.GetEnvironmentVariable(EnableEnvironmentVariable)); + + public static MimirMoveProofSurfaceDocument Create(int sequence = 79) + { + var sourceTimestampNs = 1_781_202_600_004_000_000L + sequence; + var pose = new MimirMoveControllerPoseDocument( + PoseId: $"move:0xA11CE:{sequence}", + WandId: "move:0xA11CE", + TrackingSpaceId: "mimir-stage-space", + CalibrationId: "mimir-move-stage-calibration-v1", + FusionAuthorityId: "mimir.runtime.move-fusion", + SourceTimestampNs: sourceTimestampNs, + EstimatedAtNs: sourceTimestampNs + 2_200_000L, + Sequence: (ulong)sequence, + PositionMeters: new MimirVector3Snapshot(0.0, 0.0, 4.0), + Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + LinearVelocityMetersPerSecond: new MimirVector3Snapshot(0.0, 0.0, 0.0), + AngularVelocityRadiansPerSecond: new MimirVector3Snapshot(0.02, -0.01, 0.04), + Confidence: 0.82, + LatencyMilliseconds: 2.2, + Battery01: 0.72, + Buttons: + [ + new MimirTrackingButtonSnapshot("move", true, 1.0), + new MimirTrackingButtonSnapshot("trigger", true, 0.5) + ], + EvidenceStreamIds: + [ + "witness:0x00000000000A11CE", + "witness:0x0000000000000B0B", + "controller:0x00000000000A11CE" + ], + EvidenceKinds: + [ + "optical-marker:triangulated", + "controller-state:buttons-imu", + "orientation:imu-unresolved" + ], + ConsumerContract: "fensalir.move-controller-input"); + var admission = new MimirMuninnMoveEvidenceAdmission( + FrameId: $"muninn:nightwing:move-evidence:{sequence}", + ProducerPeerId: "muninn:nightwing", + PublishedAtNs: sourceTimestampNs + 1_000_000L, + MimirReadAtNs: checked((ulong)(sourceTimestampNs + 1_700_000L)), + SampleCount: 3, + OpticalMarkerCount: 2, + ControllerStateCount: 1, + SourceTimeMinNs: checked((ulong)sourceTimestampNs), + SourceTimeMaxNs: checked((ulong)sourceTimestampNs), + ArrivalMinNs: checked((ulong)(sourceTimestampNs + 500_000L)), + ArrivalMaxNs: checked((ulong)(sourceTimestampNs + 1_700_000L)), + Handle: new MimirNativeSampleHandle( + SensorIdHash: 0xF00D, + TimestampNs: checked((ulong)sourceTimestampNs), + ArrivalNs: checked((ulong)(sourceTimestampNs + 1_700_000L)), + Sequence: (ulong)sequence, + PayloadHandle: 0xBEEF, + Flags: 0, + Reserved: 0), + ReservoirEdgeNs: checked((ulong)sourceTimestampNs), + ReservoirWindowStartNs: checked((ulong)(sourceTimestampNs - 5_000_000_000L)), + ReservoirMoveEvidenceCount: 1); + var poseFrame = MimirMovePoseStream.CreateFrame( + frameId: $"mimir:starfire:move-pose:{sequence}", + producerPeerId: "mimir:starfire", + publishedAtNs: pose.EstimatedAtNs, + trackingSpaceId: pose.TrackingSpaceId, + calibrationId: pose.CalibrationId, + poses: [pose]); + + return MimirMoveProofSurface.Create( + admission, + poseFrame, + $"fensalir:starfire:presented-frame:{sequence}", + fensalirPresentedAtNs: checked((ulong)(sourceTimestampNs + 3_100_000L))); + } + + private static bool IsTruthy(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return false; + } + + return value.Equals("1", StringComparison.OrdinalIgnoreCase) + || value.Equals("true", StringComparison.OrdinalIgnoreCase) + || value.Equals("yes", StringComparison.OrdinalIgnoreCase) + || value.Equals("on", StringComparison.OrdinalIgnoreCase); + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index a78d59c..7cc7b31 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -163,3 +163,4 @@ {"ts": "2026-07-08T22:34:00+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofPipeline as the shared commit primitive for one Move proof frame: it reads a Muninn CultMesh shared-memory evidence frame, admits samples to native move_evidence, runs MimirMoveFusion, emits the mimir:starfire move-pose frame, creates mimir.move_proof_surface.v1, and lowers the Fensalir spline probe in one pass. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, and BufferSmoke --move-proof-pipeline-smoke against the native reservoir debug DLL. This proves the reusable Mimir owner path; live Nightwing hardware, calibrated camera witnesses, and actual presented-frame capture remain pending."} {"ts": "2026-07-08T22:41:32+01:00", "type": "implementation", "status": "ok", "note": "Attached the Move proof surface to the actual MimirRuntime/Fensalir frame path: MimirRuntime.PublishMoveProofSurface stores the latest Mimir-owned mimir.move_proof_surface.v1 document, CreateFrame composes its observer-only move-proof-* splines into AquariumFrame.Scene.SplineFrame, and the UI exposes a display-only proof readout. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke against the native reservoir debug DLL. Live Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} {"ts": "2026-07-08T22:48:42+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofRuntimeDriver as the runtime bridge from a Muninn CultMesh shared-memory evidence ring to MimirRuntime.PublishMoveProofSurface. The driver derives mimir:starfire evidence/pose ids and fensalir:starfire presented-frame id from the actual Muninn frame suffix, calls MimirMoveProofPipeline, and suppresses duplicate frame admission. Verified dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke against the native reservoir debug DLL, --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke. This proves the runtime-owned ring-to-visible-probe path with synthetic ring input; real Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} +{"ts": "2026-07-08T22:55:54+01:00", "type": "verification", "status": "ok", "note": "Added and verified the Fensalir presented-frame proof smoke. MIMIR_MOVE_PROOF_DEV_SURFACE gates a synthetic calibrated proof surface through the same MimirRuntime.PublishMoveProofSurface path, and BufferSmoke --move-proof-presented-frame-smoke runs Mimir.App headless, captures artifacts/runtime/move-proof-presented-frame-smoke.png, and pixel-checks the rendered proof probe. Latest run captured fensalir:starfire:presented-frame:79 for proof mimir:starfire:move-proof:79 with bytes=11260 and brightPixels=1069; adjacent runtime-driver, runtime-frame, proof-surface, and proof-pipeline smokes pass. This proves the renderer/probe layer for the named proof chain, not live Nightwing hardware or calibrated field evidence."} diff --git a/state/map.yaml b/state/map.yaml index d8fd33b..4b9821d 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -155,6 +155,11 @@ accepted_design: actual Muninn frame suffix, calls `MimirMoveProofPipeline`, and suppresses duplicate frame admission so the live loop cannot re-admit one frame and impersonate progress. + - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by + `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof + surface through the same MimirRuntime attachment so headless Mimir.App can + capture and pixel-check a Fensalir-presented proof probe without making + Fensalir a proof owner. It is renderer evidence, not field evidence. - `MimirMoveCalibrationProtocol` defines the runnable Starfire/Nightwing Move calibration preflight as typed state: required Muninn Move evidence streams, optional Quest headset/controller pose witnesses over USB, From 83ddd3f509b2486908c2d1a0202b380da6d96bc0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 10:41:48 +0100 Subject: [PATCH 070/135] Add Move proof runtime configuration --- docs/implementation-plan.md | 9 + notes/current-system-map.md | 8 + src/Mimir.BufferSmoke/Program.cs | 94 +++++---- src/Mimir.Runtime/MimirRuntime.cs | 23 ++- .../MimirMoveProofRuntimeConfiguration.cs | 195 ++++++++++++++++++ .../MimirRuntimeConfiguration.cs | 15 ++ state/evidence.jsonl | 1 + state/map.yaml | 9 + 8 files changed, 315 insertions(+), 39 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index ab1ba2e..4f48493 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -104,6 +104,15 @@ a named invariant that the native runtime cannot protect yet. duplicate frame admission. `Mimir.BufferSmoke --move-proof-runtime-driver-smoke` proves `MimirRuntime.Update` can pull one ring frame into the same-sequence proof chain and visible spline frame. +- `MimirMoveProofRuntimeConfiguration` is the typed runtime configuration and + validation contract for future real Move proof subscriptions. An enabled + source names the Muninn evidence stream, native reservoir path, Mimir evidence + and pose frame prefixes, Fensalir presented-frame prefix, fusion authority, + consumer contract, and at least two calibrated camera witnesses. It creates + `MimirMoveProofRuntimeDriver` only after the supplied ring stream id matches + the configured evidence stream. The current C# CultMesh ring used by the + smoke is still in-process; real Nightwing/Starfire cross-process ring opening + remains pending. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so `Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 57ad56b..024da22 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -225,6 +225,14 @@ suffix, calls `MimirMoveProofPipeline`, suppresses duplicate frame admission, and lets `MimirRuntime.Update` publish the resulting proof surface. This proves the runtime-owned path for a ring frame; real Nightwing hardware and calibrated camera witnesses are still the missing field proof. +`MimirMoveProofRuntimeConfiguration` is the typed selection and validation +surface for that bridge. An enabled source must name the Muninn evidence stream, +native reservoir path, Mimir evidence/pose frame prefixes, Fensalir +presented-frame prefix, fusion authority, consumer contract, and at least two +calibrated camera witnesses. It may create a runtime driver only when the +supplied CultMesh ring stream id matches the configured evidence stream. This +does not yet open a cross-process live ring; the current smoke ring is +in-process proof of the Mimir-side contract. `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 18a5bc0..5ec3993 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -161,6 +161,12 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-driver-smoke", Stri ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-config-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeDriverSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); +} + if (args.Any(arg => string.Equals(arg, "--move-proof-presented-frame-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveProofPresentedFrameSmoke( @@ -2337,31 +2343,56 @@ static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) Battery01: 0.72f, ObservedAt: observedAt) ]); - var calibration = new MimirMoveFusionRigCalibration( - CalibrationId: "mimir-move-stage-calibration-v1", - TrackingSpaceId: "mimir-stage-space", - Cameras: - [ - new MimirMoveFusionCameraCalibration( - CameraId: "nightwing:ps3-eye-0", - WitnessIdHash: leftWitnessHash, - PositionMeters: new MimirVector3Snapshot(-0.1, 0.0, 0.0), - Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), - FocalLengthXPx: 100.0, - FocalLengthYPx: 100.0, - PrincipalPointXPx: 160.0, - PrincipalPointYPx: 120.0), - new MimirMoveFusionCameraCalibration( - CameraId: "starfire:ps3-eye-1", - WitnessIdHash: rightWitnessHash, - PositionMeters: new MimirVector3Snapshot(0.1, 0.0, 0.0), - Orientation: new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), - FocalLengthXPx: 100.0, - FocalLengthYPx: 100.0, - PrincipalPointXPx: 160.0, - PrincipalPointYPx: 120.0) - ], - MaximumAssociationSkewMilliseconds: 20.0); + var configuration = new MimirMoveProofRuntimeConfiguration + { + EvidenceStreamId = streamId, + NativeReservoirPath = nativeReservoirPath, + MimirEvidenceSourceId = "mimir:starfire:move-evidence", + MimirEvidenceFramePrefix = "mimir:starfire:move-evidence", + MimirPoseFramePrefix = "mimir:starfire:move-pose", + MimirPoseProducerPeerId = "mimir:starfire", + FensalirFramePrefix = "fensalir:starfire:presented-frame", + Calibration = new MimirMoveProofCalibrationConfiguration + { + CalibrationId = "mimir-move-stage-calibration-v1", + TrackingSpaceId = "mimir-stage-space", + Cameras = + [ + new MimirMoveProofCameraCalibrationConfiguration + { + CameraId = "nightwing:ps3-eye-0", + WitnessIdHash = leftWitnessHash, + PositionMeters = new MimirVector3Snapshot(-0.1, 0.0, 0.0), + Orientation = new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx = 100.0, + FocalLengthYPx = 100.0, + PrincipalPointXPx = 160.0, + PrincipalPointYPx = 120.0, + }, + new MimirMoveProofCameraCalibrationConfiguration + { + CameraId = "starfire:ps3-eye-1", + WitnessIdHash = rightWitnessHash, + PositionMeters = new MimirVector3Snapshot(0.1, 0.0, 0.0), + Orientation = new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx = 100.0, + FocalLengthYPx = 100.0, + PrincipalPointXPx = 160.0, + PrincipalPointYPx = 120.0, + } + ], + MaximumAssociationSkewMilliseconds = 20.0, + }, + }; + var configErrors = configuration.Validate(); + var invalidErrors = new MimirMoveProofRuntimeConfiguration().Validate(); + if (configErrors.Length > 0 || invalidErrors.Length == 0) + { + Console.Error.WriteLine( + $"move-proof-runtime-driver-smoke config-invalid errors={string.Join("|", configErrors)} invalidDefaultErrors={invalidErrors.Length}"); + return 1; + } + var catalog = CultMesh.CreateStreamCatalog(); catalog.Declare(MimirMuninnMoveEvidenceAdapter.CreateStreamDescriptor( streamId, @@ -2378,16 +2409,7 @@ static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) catalog.PublishFrame(published); using var reservoir = new MimirNativeReservoirRuntime(nativeReservoirPath); - var driver = new MimirMoveProofRuntimeDriver( - ring, - reservoir, - calibration, - new MimirMoveProofRuntimeDriverOptions( - MimirEvidenceSourceId: "mimir:starfire:move-evidence", - MimirEvidenceFramePrefix: "mimir:starfire:move-evidence", - MimirPoseFramePrefix: "mimir:starfire:move-pose", - MimirPoseProducerPeerId: "mimir:starfire", - FensalirFramePrefix: "fensalir:starfire:presented-frame")); + var driver = configuration.CreateDriver(ring, reservoir); var runtime = new MimirRuntime( new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), new MimirSynchronizationSettings()); @@ -2403,7 +2425,7 @@ static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) var duplicateSuppressed = reservoir.Status.MoveEvidenceCount.ToUInt64() == 1UL; Console.WriteLine( - $"move-proof-runtime-driver-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed}"); + $"move-proof-runtime-driver-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length}"); return proofSurface is not null && proofSurface.IsFullPose && diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 032e8aa..0939124 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -39,6 +39,7 @@ direct private readonly MimirSynchronizationHub synchronization; private readonly MimirAudioSpectrumAnalyzer spectrumAnalyzer; private readonly IReadOnlyList sourceFactories; + private readonly IReadOnlyList configuredMoveProofSources; private readonly AquariumUiDocument ui; private readonly AquariumAudioDocument audio = new(); private readonly MimirAudioSynchronizationSettings audioSyncSettings; @@ -71,7 +72,7 @@ direct } public MimirRuntime(AquariumRuntimeOptions options, MimirRuntimeConfiguration configuration) - : this(options, configuration.Settings, configuration.SourceFactories) + : this(options, configuration.Settings, configuration.SourceFactories, configuration.MoveProofSources) { } @@ -84,7 +85,7 @@ direct AquariumRuntimeOptions options, MimirSynchronizationSettings settings, IEnumerable streamSources) - : this(options, settings, Array.Empty()) + : this(options, settings, Array.Empty(), Array.Empty()) { foreach (var source in streamSources) { @@ -95,12 +96,14 @@ direct private MimirRuntime( AquariumRuntimeOptions options, MimirSynchronizationSettings settings, - IEnumerable sourceFactories) + IEnumerable sourceFactories, + IEnumerable moveProofSources) { Options = options; synchronization = new MimirSynchronizationHub(settings); spectrumAnalyzer = new MimirAudioSpectrumAnalyzer(ParseSpectrumFftSize()); this.sourceFactories = sourceFactories.ToArray(); + configuredMoveProofSources = moveProofSources.ToArray(); audioSyncSettings = settings.Audio; telemetryIntervalSeconds = ParseTelemetryIntervalSeconds(); audioSyncUpdateIntervalSeconds = ParseAudioSyncIntervalSeconds(); @@ -288,6 +291,7 @@ direct panel.Readout("Chirplet reference", DescribeChirpletReference); panel.Section("Move Proof"); panel.Readout("Latest proof", DescribeMoveProofSurface); + panel.Readout("Configured proof", DescribeConfiguredMoveProofSources); panel.Readout("Proof drivers", DescribeMoveProofDrivers); panel.Section("Live FFT"); panel.Readout("Spectrum cadence", () => sceneReady @@ -325,6 +329,19 @@ direct return string.Join(Environment.NewLine, moveProofDrivers.Select(driver => driver.Describe())); } + private string DescribeConfiguredMoveProofSources() + { + if (configuredMoveProofSources.Count == 0) + { + return "No Move proof sources configured."; + } + + return string.Join( + Environment.NewLine, + configuredMoveProofSources.Select(source => + $"{source.EvidenceStreamId} -> {source.MimirPoseFramePrefix}: cameras={source.Calibration.Cameras.Count}")); + } + private void UpdateMoveProofSurfaces() { if (moveProofDrivers.Count == 0) diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs new file mode 100644 index 0000000..f61dd0e --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs @@ -0,0 +1,195 @@ +using GameCult.Mesh; + +namespace Mimir.Runtime.Synchronization; + +public sealed class MimirMoveProofRuntimeConfiguration +{ + public bool Enabled { get; init; } = true; + + public string EvidenceStreamId { get; init; } = "muninn:nightwing:move-evidence"; + + public string NativeReservoirPath { get; init; } = ""; + + public string MimirEvidenceSourceId { get; init; } = "mimir:starfire:move-evidence"; + + public string MimirEvidenceFramePrefix { get; init; } = "mimir:starfire:move-evidence"; + + public string MimirPoseFramePrefix { get; init; } = "mimir:starfire:move-pose"; + + public string MimirPoseProducerPeerId { get; init; } = "mimir:starfire"; + + public string FensalirFramePrefix { get; init; } = "fensalir:starfire:presented-frame"; + + public string FusionAuthorityId { get; init; } = "mimir.runtime.move-fusion"; + + public string ConsumerContract { get; init; } = "fensalir.move-controller-input"; + + public MimirMoveProofCalibrationConfiguration Calibration { get; init; } = new(); + + public string[] Validate() + { + if (!Enabled) + { + return []; + } + + var errors = new List(); + Require(EvidenceStreamId, nameof(EvidenceStreamId), errors); + Require(NativeReservoirPath, nameof(NativeReservoirPath), errors); + Require(MimirEvidenceSourceId, nameof(MimirEvidenceSourceId), errors); + Require(MimirEvidenceFramePrefix, nameof(MimirEvidenceFramePrefix), errors); + Require(MimirPoseFramePrefix, nameof(MimirPoseFramePrefix), errors); + Require(MimirPoseProducerPeerId, nameof(MimirPoseProducerPeerId), errors); + Require(FensalirFramePrefix, nameof(FensalirFramePrefix), errors); + Require(FusionAuthorityId, nameof(FusionAuthorityId), errors); + Require(ConsumerContract, nameof(ConsumerContract), errors); + errors.AddRange(Calibration.Validate().Select(error => $"{nameof(Calibration)}.{error}")); + return errors.ToArray(); + } + + public MimirMoveProofRuntimeDriverOptions ToDriverOptions() => new( + MimirEvidenceSourceId, + MimirEvidenceFramePrefix, + MimirPoseFramePrefix, + MimirPoseProducerPeerId, + FensalirFramePrefix, + FusionAuthorityId, + ConsumerContract); + + public MimirMoveProofRuntimeDriver CreateDriver( + CultMeshSharedMemoryFrameRing ring, + MimirNativeReservoirRuntime reservoir) + { + ArgumentNullException.ThrowIfNull(ring); + ArgumentNullException.ThrowIfNull(reservoir); + var errors = Validate(); + if (errors.Length > 0) + { + throw new InvalidOperationException($"Invalid Move proof runtime configuration: {string.Join("; ", errors)}"); + } + + if (!string.Equals(ring.StreamId, EvidenceStreamId, StringComparison.Ordinal)) + { + throw new InvalidOperationException( + $"Move proof runtime configuration expects evidence stream '{EvidenceStreamId}', but ring is '{ring.StreamId}'."); + } + + return new MimirMoveProofRuntimeDriver( + ring, + reservoir, + Calibration.ToRigCalibration(), + ToDriverOptions()); + } + + private static void Require(string value, string name, ICollection errors) + { + if (string.IsNullOrWhiteSpace(value)) + { + errors.Add($"{name} is required"); + } + } +} + +public sealed class MimirMoveProofCalibrationConfiguration +{ + public string CalibrationId { get; init; } = "mimir-move-stage-calibration-v1"; + + public string TrackingSpaceId { get; init; } = "mimir-stage-space"; + + public List Cameras { get; init; } = []; + + public double GyroUnitsPerRadianPerSecond { get; init; } = 1.0; + + public double MaximumAssociationSkewMilliseconds { get; init; } = 20.0; + + public double SingleRayFallbackDepthMeters { get; init; } = 1.5; + + public string[] Validate() + { + var errors = new List(); + if (string.IsNullOrWhiteSpace(CalibrationId)) + { + errors.Add($"{nameof(CalibrationId)} is required"); + } + + if (string.IsNullOrWhiteSpace(TrackingSpaceId)) + { + errors.Add($"{nameof(TrackingSpaceId)} is required"); + } + + if (Cameras.Count < 2) + { + errors.Add($"{nameof(Cameras)} must contain at least two calibrated camera witnesses for full-pose proof"); + } + + for (var index = 0; index < Cameras.Count; index++) + { + errors.AddRange(Cameras[index].Validate().Select(error => $"{nameof(Cameras)}[{index}].{error}")); + } + + return errors.ToArray(); + } + + public MimirMoveFusionRigCalibration ToRigCalibration() => new( + CalibrationId, + TrackingSpaceId, + Cameras.Select(camera => camera.ToCameraCalibration()).ToArray(), + GyroUnitsPerRadianPerSecond, + MaximumAssociationSkewMilliseconds, + SingleRayFallbackDepthMeters); +} + +public sealed class MimirMoveProofCameraCalibrationConfiguration +{ + public string CameraId { get; init; } = ""; + + public ulong WitnessIdHash { get; init; } + + public MimirVector3Snapshot PositionMeters { get; init; } = new(0.0, 0.0, 0.0); + + public MimirQuaternionSnapshot Orientation { get; init; } = new(0.0, 0.0, 0.0, 1.0); + + public double FocalLengthXPx { get; init; } + + public double FocalLengthYPx { get; init; } + + public double PrincipalPointXPx { get; init; } + + public double PrincipalPointYPx { get; init; } + + public string[] Validate() + { + var errors = new List(); + if (string.IsNullOrWhiteSpace(CameraId)) + { + errors.Add($"{nameof(CameraId)} is required"); + } + + if (WitnessIdHash == 0UL) + { + errors.Add($"{nameof(WitnessIdHash)} is required"); + } + + if (!double.IsFinite(FocalLengthXPx) || FocalLengthXPx <= 0.0) + { + errors.Add($"{nameof(FocalLengthXPx)} must be positive"); + } + + if (!double.IsFinite(FocalLengthYPx) || FocalLengthYPx <= 0.0) + { + errors.Add($"{nameof(FocalLengthYPx)} must be positive"); + } + + return errors.ToArray(); + } + + public MimirMoveFusionCameraCalibration ToCameraCalibration() => new( + CameraId, + WitnessIdHash, + PositionMeters, + Orientation, + FocalLengthXPx, + FocalLengthYPx, + PrincipalPointXPx, + PrincipalPointYPx); +} diff --git a/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs index 34cdf0b..8d5b58e 100644 --- a/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs @@ -8,6 +8,8 @@ public sealed class MimirRuntimeConfiguration public IReadOnlyList SourceFactories { get; init; } = []; + public IReadOnlyList MoveProofSources { get; init; } = []; + public IReadOnlyList CreateSources() { return SourceFactories @@ -53,6 +55,16 @@ public sealed class MimirRuntimeConfiguration .Where(factory => factory != null) .Cast() .ToArray(); + var moveProofSources = model.MoveProof + .Where(source => source.Enabled) + .ToArray(); + var moveProofErrors = moveProofSources + .SelectMany((source, index) => source.Validate().Select(error => $"MoveProof[{index}].{error}")) + .ToArray(); + if (moveProofErrors.Length > 0) + { + throw new InvalidOperationException($"Invalid Mimir runtime Move proof configuration: {string.Join("; ", moveProofErrors)}"); + } return new MimirRuntimeConfiguration { @@ -63,6 +75,7 @@ public sealed class MimirRuntimeConfiguration Streams = streams, }, SourceFactories = sourceFactories, + MoveProofSources = moveProofSources, }; } @@ -224,6 +237,8 @@ public sealed class MimirRuntimeConfigFile public MimirAudioSyncConfig? AudioSync { get; set; } public List Streams { get; set; } = []; + + public List MoveProof { get; set; } = []; } public sealed class MimirAudioSyncConfig diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 7cc7b31..38c53da 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -164,3 +164,4 @@ {"ts": "2026-07-08T22:41:32+01:00", "type": "implementation", "status": "ok", "note": "Attached the Move proof surface to the actual MimirRuntime/Fensalir frame path: MimirRuntime.PublishMoveProofSurface stores the latest Mimir-owned mimir.move_proof_surface.v1 document, CreateFrame composes its observer-only move-proof-* splines into AquariumFrame.Scene.SplineFrame, and the UI exposes a display-only proof readout. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke against the native reservoir debug DLL. Live Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} {"ts": "2026-07-08T22:48:42+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofRuntimeDriver as the runtime bridge from a Muninn CultMesh shared-memory evidence ring to MimirRuntime.PublishMoveProofSurface. The driver derives mimir:starfire evidence/pose ids and fensalir:starfire presented-frame id from the actual Muninn frame suffix, calls MimirMoveProofPipeline, and suppresses duplicate frame admission. Verified dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke against the native reservoir debug DLL, --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke. This proves the runtime-owned ring-to-visible-probe path with synthetic ring input; real Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} {"ts": "2026-07-08T22:55:54+01:00", "type": "verification", "status": "ok", "note": "Added and verified the Fensalir presented-frame proof smoke. MIMIR_MOVE_PROOF_DEV_SURFACE gates a synthetic calibrated proof surface through the same MimirRuntime.PublishMoveProofSurface path, and BufferSmoke --move-proof-presented-frame-smoke runs Mimir.App headless, captures artifacts/runtime/move-proof-presented-frame-smoke.png, and pixel-checks the rendered proof probe. Latest run captured fensalir:starfire:presented-frame:79 for proof mimir:starfire:move-proof:79 with bytes=11260 and brightPixels=1069; adjacent runtime-driver, runtime-frame, proof-surface, and proof-pipeline smokes pass. This proves the renderer/probe layer for the named proof chain, not live Nightwing hardware or calibrated field evidence."} +{"ts": "2026-07-12T10:26:42+01:00", "type": "verification", "status": "ok", "note": "Added MimirMoveProofRuntimeConfiguration as the typed config/validation surface for Move proof driver creation. The runtime config now accepts enabled MoveProof sources, validates named Muninn evidence stream, native reservoir path, Mimir/Fensalir frame prefixes, fusion authority, consumer contract, and at least two calibrated camera witnesses, and creates a driver only when the supplied CultMesh ring stream id matches. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-config-smoke, --move-proof-runtime-driver-smoke, --move-proof-pipeline-smoke, and --move-proof-presented-frame-smoke. The config smoke printed configValid=True invalidDefaultErrors=2 and the same sequence chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79. This proves config plus in-process ring flow, not live Nightwing hardware, calibrated field cameras, or cross-process ring opening."} diff --git a/state/map.yaml b/state/map.yaml index 4b9821d..571edd5 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -155,6 +155,15 @@ accepted_design: actual Muninn frame suffix, calls `MimirMoveProofPipeline`, and suppresses duplicate frame admission so the live loop cannot re-admit one frame and impersonate progress. + - `MimirMoveProofRuntimeConfiguration` is the typed selection and validation + surface for that runtime bridge. An enabled proof source names the Muninn + evidence stream, native reservoir path, Mimir evidence/pose frame prefixes, + Fensalir presented-frame prefix, fusion authority, consumer contract, and + at least two calibrated camera witnesses. It may create a + `MimirMoveProofRuntimeDriver` only when the supplied CultMesh ring stream id + matches the configured evidence stream. The current C# shared-memory ring + smoke is still in-process; real Nightwing/Starfire cross-process ring + opening remains pending. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same MimirRuntime attachment so headless Mimir.App can From dc519182274fd9ad7ffb4e0661eb4d4729373e40 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 10:45:50 +0100 Subject: [PATCH 071/135] Persist Mimir Face state --- .voidbot/state/mimir.cc | Bin 72151 -> 108447 bytes .voidbot/status/void-memory-maintenance.json | 4 ++-- .voidbot/voice/identity.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index e244888bcaad747991f3bac35fe5a8afc23140ad..66634d68e299568fceb6d619829f8e04da609ac0 100644 GIT binary patch delta 35122 zcmeHw3$$cqm7W(+9BiKQXd02@wWU$Iih4gT64KB-I`U|{uY-b_=2V?i_ZD4Mr>IBY zzBn_-(TERp1vtzQP+=TE#kbS5qK<3EF=Q4dCcc$4Gf9J!I1`N;va;f=H7h3HxA*^_ zbL!r@eJ`zMlC^@U-BqW~`5$|K``eFypQjF;@bK4P^qJ>QKCPO`7v^*Ixl(bioXb|~ z(ab3aOZYH9pDXagd@)-p)^7gja1;!DZ^!8KgD8d7i^Bc`HBk<-Ld(%E}pq7dgk<3<_d*s zxtOa~alTBaKO8x3Z@AjBUoI`SJHC87;0FVL;EeDy`J~~y zqjqm$J1=T3I_;s;3k&scY49d@!dg>pSCggh4uj-qjRuW zPT%D%0tegK+^^&F2JP0!_pZEFt0g1e-wT#FVqNSdcUjj4+tX|M&cN--T92@z+jJ3z zzKcZ|4gF4QJ2r=l&e^uNVV9fkV4yc>wf&BlkvsPXkSjcKF6XqJo{!zfLdZ@yJ8mDl zgZ8+KTVJ#L4NhxJF?WU|Yzm~yi~i*1^Fs$K?R1x&VcWw_EeB%=TF*1mvv)MWaCWy3 z`kkTEz)E8vEm?6)Z;Tym_q-vtw=o#EhZHynhK~8RG2|iWc-WeDZvED**aCNip6@RSSXdt z&iCBDnf=uVli};@%|uyWVN{=9mp7TahPca zy&hCV7Z&uGkuXa3E+8oDhqOg-UHh-yJQ= zis3oU4z$c^V{f`pZc)7+1s~$?<8I>~r9P|Mw9s?08Qu26;>c-WA4lC_*vFc-n-Dne z0tA9EAhQa12{yogJqENOGDqER2F?Bb_9)ZALpmZ5__gewKDO(6TmecFCX0RN7=vyX zlOUA1jgD{hs#q!&%cWwWo-fx6)lzB7&K2@=rQBSpoGlh}YwJ}JH>p<3)hb}KSg+MY zudb4{b_n$h?-NaSv9Ljx3yhtPf1|hr4e;IZsNEWO z1m6bG;mNCrrhFy=)C2MnN9A!)-2*Zx0gpkuFPKUMfK*_coZ;(6ZB7`!9s&riF|em& zg;4DxGQjHfonbK6ZSS-pMAS+M6T}A^=q$M71>b^Uxd=te6>{}ju?(GCwP`Di>e`!D zF4Rgn+@w~h)uF<<=-Klho%tCy?aY60_-8i+gC4+SDEM_naIm=3=@S1ijdpAKQp%SX zzzh)B7ksWhmt=r>W}WDF-gjn*ZQ^HYTRhBrpb|`@12n?__`QWTmSr16ygwcdAs_87 zZtwTE6W<`d9Ul@MV%jw0Do?8s|SVl z5B57)KkV7+D>3lJX!quea)olWlCKmX-??fvU#qXeFH#qPSAMQs%N9$tF8_IEr`n8N z-+$`^7#VODVgV?o+AllZK*xmXH)y!sArJoi;AQ{tA4}skx(kgNqC~H&|KB zOr@tmnZQ!>QwaAMi!m4xl7aY;yii0>4TG+k8Y<6kw#8U9VGR~Ad%IqcYMP@rb%AMs zOX|eXZ;oM5bjagw)-h8O!=CXEx-jHGC4bQL`y&{jtg{0{TK2JBJ9fcBiBvS&xCM8+ z*=hG1fjhvUTeKRmRs&L#z#BISXgQTnRzJ*;p9@ULkO$6 zvJk@8?>zswJ-0>uom*x;aOkd`e|u(h!W%YksuxRzLe8z`Dh)sS^v+CpsB4gDABWW# zv@wjwIxen%0Sqv@H~yh>&CZ>Ww%&dsHtht3jMlU@76x&B8;hJ7 zF1meuA+#VQb`Sl&OIn4`#0DG~W*xuV2)s5v!dIhqvkluqOMlTUAqN7ytfoZu3E~0y zUUES#x!@3DE+tTu2B=hGln_^e;EnJ%Qk~5f3?OWMFi5vgyoKn>jIah?(1i%jsQ_n$z{1e+VG#Zy#5|r4MH)(B zrd88C)9#>fNIHJQk%_Il9<1+#}P;0yyeXB z;2v2sYBPug6&LnmX%Se>4S~5QjK$;w*xjee!iEGL5fVZhE=^vwL}(F50t7S+63t)6RnInkm&Nt9!QWg-R{0lc< zA=VO1iC-9@po+#0=_HEDW&(Ag_+n=$Bci1byaCH4`iZghxelVDUV8{!7|OCHURY|f z1_4x_xT6@qt{^8E3k{6eLK_bw3g{>a!}O5V(?1WCg#I-4hrcsO2-Rus&qntgIcKZq zRdV@qv6iWn3;9f`hHyutQmJG-uUc(Yi*6%dYos)|JXb2s6^q$wZA}eMT{5A;UZLE= zKeBYoI%EYM6Ok~aIJ5x);>Mc5lcYC0=Xk07MZ@4vfCd0I;61=({goyr|k7R zv^_!ZRsn0(zfj>j!)Oiclm`k)e}%$zt722?X9wD@E+EALlGSNo1z}MjSO85ZXS)GY z0N7j@xcx<}vw2|Fna8qg{DhmW8Yh|r4TcJ0D@M>!GJ9^22aQ3%0}GOAkaS8Q#LA`U zD8vUB4(L*;CO3TeR$l5EPzs$PU5!9&yd93*!2`S_E)s(CQW)iK;;6dN;UU}>7!6o- zJi^*8NSV9e;?svXBeQjCbf80&f5v8nOK~FRPQt^lR6D5)O^uU*g#s#A?G;O=OM*KM zg7E^Dw~LhpYvt?Z_m_-a8U0u1l~JO{3zE=deP;o(2$N~Ua& zK|aKzMi3%A9Y3eTNE%HKl!(@bc+eT4XUOG9#cxodpo72(~V*Az+m$?;&I4v9(89I?>7z#!y zS^-w^YH<(S2Si#3tTF~@D!k&@MhkC%r1LvbZVPub!*CYpw5kkQuu2JWEW-#;EYX)i zTx(r9wn+9bQ{rj@$`}XLgkz%ha6-u6LAX%xbYn@J0$B=A2J?U!^w?xViXUFvV+a<( z9cXp?4qu+J0p)fFVeBbkUc&Iz=qWyWnl*&&(?sgQ_BbQR$n3IPb`AV5Bbyd;KW zToo^oe?Yw^jXz3Iq#yXB!Lrg8&9?y15qt*Ym&6JEO!(JS2x}tf*tc5<7?_+7qLWm3 zjsS5RNC{j|9gzMN8lC=YDvY@lj1laIqud_0;3vf{(X$k#aJYzMjLG+;;?)u?0DJ__ zY&x5>evEY1lK(h$6$}}Y$T}c?t&Hc>@+tu*{)EcVl0YalfQcCiBPo2*I1k1Wbi*>^ zX3Qg*fZs|D>7C;SRSkN(gb6rIBC7w~n!}s{D;eBs=1h{ed zqT7PToa?*ga=lp2xVdIIQ!3?)8DunSnOdWe_sgYfs~(*{zhz@FkPGI|o%M1}uL>ut zh8#zx)X3*DwQ{YNLEx`d_Ym1A)T5WrpS!hO%a!n-T&9U&aHdqOmNNB1p@2W-s)a@o z!A7@iJ+jh#p)i*(BPg6j1bz*aF{zV6wv?|#Uz~SeQY%(UrB85_h|7ai(6VMw;ZfNT0OE?vu^iEk zB7}>5(bO%Zw6_q#g@swctvqX;zjZZ}6trKATG1NgjxB!XwOa!1FjtIgO4iJXM9*J< z#$r=JLJ?#@>YE`je|V__pH2S?G@|g1kZ^-EQX(*m5HiUqb$f`ZF#1Sx$rL!ghFPRE zVcf{$DKd%*Ex^^6YzfBI(2S}^MH&55NRZ>|H_%Fy{AvnSPK{gf{60=Z9m38RsVo?x$Kv-XYE#&``u_?wf!AWU@;fs-?2c;+# zIIt36=JX)!C*&3J0?3RhFqr-d>jPaBqz)tiB0n*sMM@LdxJytGfv}o^o*va6=E^*V z8weYu$vDKC_p2?Rt>ufYVjYW9%6_g~Yv!xNf1X8 zfkXI{Vk~I<4Zrb26pCQjmgiXer>fmIWFbi-DEkI8My8+aI1d1?_aJV zp%n`+@p)!RpnHHPc+5RF91Mab{7P_gR+H*lxHBtsNA77xqu?|fry<_IqlZLApMDFT zso<4ZMD?edZXcV#vY4Pp)E=XP2X-0Iz%&c5FncjYhc0&+G4jT;`m5t_=Il3;H)MK`Om7NXP?#-o;@ z1Owz|E-3C1D=}?QBn?t^#KABVVn~4U`c~epdS0WL@qHIr48K*(pehT1;CdAwhT1DO zn%3f0=JSAp9EyCh<=S*Kn7qWI!J)Um>67POf;`CN*GEq+M$SW9qUG@`X8tUC|MD52QfN6Ep8F{TpVH-D5D?HTum(cc_6!(?ld zn!eo8mo!@~3zH|4_kKa^m6_ClQ*Na}E z89j6NYsDY?j=JK|2d46fObPk1f{!TGq5$Nr9Vqrb zMu2T+;J|Hy*$pp|pvC~+5S%9_g=m~mW+AX4IHT?i_?VWxz(PrRh8uR7(heF-dJEl1 zv8!99lE>446YHNB zzW0aGm!8`a#tUm0sn$-WjIX(=l!-}A0~a1!F3ChYeN>7VkKD^TS6YyzfI(%H<%ne< z@f^44aucs1ExWOs!*K_eLr9{ets%%7i;nG(vZhSX%0Q$voe8T+GQ3J?gxCGA9VmW8 zT_0uG-XFz-SkCQ6dVsZ3a~Q8E*Lu5S$|i-q+I6E$ry>I;G$ni^y@|Gd1hg&M^B~H1 z=qECP1>%Tkv32PfBxV7b)d85D*uhg)1cPP;AF)BvAxTcOz^1wOL)bo=Wsq9yQD-0L z!eLuS%Yd&U|B3ECvUzLWZ4}GRx}T|6Y9NdiH=l8fwFbPwV!l|(HEJl=iB7-!vLetO z6`(a%fR;+A(J4%2ixZbbzjOEAO@&IW*lIK?wHlJ!&)j`kbmWJ>wnd3x#wvX#iS$QH zzxnF$gE9KV$kRYBP*{{GPvQ$hv!wO{O2=IgcEu%7UI8Uw)Mb}4^veykAa6$45+AcX z6Q2?f`l7?63PDJUM=Gg0YCsSUJ2FlTu$vl=udE9rL=pUY3UrwKkEC>zRSZBk5I}8a zojswEcbS7oXlj zd)EYkP-xa93dBnKO9BBZka(w9E*dRG+0Rnmg6QAfa&q`T{vQX3W5 zd#>9Sef@){N7p^@vgp6Q^PisbqX|PAz2(8#ldM%u90goMm6Q#lD7oMqzY;U*N-zKf7_k%^hf^v-3hqE+;7^0C2hBFZEZ~T3Ml}Pt$qk!^P{w66 zKvvL1g;w$8l{8u8M{+$LGh`B$fQq5S01+FZ0;u+<>U&5S;_;yK3W3|Bjfti%6>5se4` z*7V8jGa+(Y1-E02!m&V2x~QPRs01#KQm{1o2Z$}w#HHek@l+<=*>!{!X(Yypd9MvP zthU%8y7|FP?O(d%YSOH*U6X!lbk~vdx7JFPR;vyG!l5;!Gx8-jQ_HvNnPSDQ73#Ts zq14JHE-C_pV3=T_^;~f+2AW)Q?|b)dD(5TBLa|ycHX9{)s8>JFG{#_WoI|h#2+n9K zP(;6U;#dHqVTe=7nKnclrbWvE9H#J7oIphRaX178##+;W!en!BG0a>gCwy3qJan!? zJRFHWTaMShLsM14u@nOpWl?0rJg9u0b*@HO0EsOosP*GTq=Hdt>mNi|Pr~&Kv%@S( ze0 z3$!ly=R?s@W&r^95{aiRM7Uc87|P85R5*}^2d%3CsyZqtJ>2>wCd zHhr{%KZl(OQ5=G=;sqk_MafTGdo5aOlD~*eAUH2ah>@C1;PWZ19MtZb2@gci-g;K}?HibSS>>HrJxeVwE1%;P#hI|gW5e)us!lKu zq|k}wprnAbP_A^d7d6#Cx!capKn;{+ilKYy2(bo*>9vj*u%gT%;CMawFXG1Li4(B+ zH!qNXq{h8ju9XUuGU*sw^{ZqJ41E`?m14H7PFOHbt*Vc zUP5-Zsp;{+UOrdx>ptNz{KKwfc{O0RPwJ;My*uR#Tg`d+8^*Pl%e4?7_GnUA83)yc z$=?>p!9uPrV>OU-n7Xjt1&mM|)vbU6TSwGzH#kOet`n=rpT;&+o(&&*q5=xe>O`o7 zXF#wC5;v7famc}KsIbo%mj<=vh%R>AxPa?n;)UEZo{%AxdR|f2N$bo^h0urvsK$A= zW4_4BXAB`Wk&SZe^c2D-qR{v`k6TNmGdW|S$HwB&mL51vA@SbYv4r3q3K#+a~CEy0qI{#in07-QeTaOb9@5P7=U=ksrWa1w-;$p zJNnRjHa9nL?on52;mbqO7V?$8oV*F`vsd>ZN)krGs#PKv3%0a&CI9 zHFPj_iFiogx^I8jn+U!b9Go(5t#e@!xh+DHV2l;7i8+ed9xe>-}87hE0f$b6?ha+(0nCEwKBFRvv6r@{=8Jj^9IxwgAC$KdU*SE~$p%M-z zjgS%)lkwDpN7OU(Q}CA#Gc-8K7>DspFfsbthjz`pFS`8$XTB+cbXok=qQ@o#^Tg3} zT}i1?;KbJ#N37-Mn16c{cHzg-fBD||TZ_!>dr0BnUc!4H4+- zdHi0f6t<8lC@P?l&{&$)0EGA%zQHdU*`sWT0>b7^6x7~*lQgLrb{XgZff1(2b+U>u zg)|D;UriBH3xFF$XZ*%FyVhHSM1$_M8U!QFrS*GKGmDZQ^lNa3(T(ppYY*nk0Idnc zb>Kd5S|-}3aej^*t_;nQR?9Sx$!gvoM>9B`T4~vNz z>5I|_ZA)cXJQi8u*D*93KRt*aLP6eRJQFbv^KXt2&F<0I4mcmc&3j>KYdTL z$aYS#_HIuf3{kUggBdtf9`|ujrA`F8LxCQn(t~tLD>w}&d_5MP3#p168v#>kz{lRiHK`dgG5@#d>cawc0L@$d+@c7cF=h7d}O%T58mrZZTJ^ zt5BmFz?5O-<_d`QmZw$&OkRR&fNJ#22TzZSn&63;-yXduS+!7?qNcfrmd~)y)h1`v zVpYC*`k^i7eQC?PR&*NT4$tKZ)jEpKo6(maxp@6z zgo{w9`hq>&2kIdguRRqmrw;)cHD#VA$p}v8DmpL#1y$QX8p>2QJVDuC8KPblp_WqEa3lp%5KJxG zwT+r!5~#cx>PZ4#$Y?`zF+4Z{(!&yqn?On@zAu4HB5*vf>MCw??-sA4r^RdGTL5Xs}K zB5W3dll;mC@@46C+5h>diyA%n>R5+i_$!aP{OBnuX)9;Fifgt$cB!$ zm}08(e0+ALFdaV8G9+~g;M0km@4o2J-H(h;*yMs?RK0u)RsMMZStFSO54 zz@a&Gdz35UFwH-w=C6rk<_-xC3Xst#h4%D_sAC-5u$2M!hsoeXF0I=K&bTuACe6eU z?-M`Ho(QEOi$kq(1QIvTh6cGlu>&Smnl*NoaeQP-;8t!()iKmt0*T0<+abmnkGb&@ zmvAb@8urNw_8O7_nUr>FAvgjnT7m9bAg^{-;;Qr(h2F$*RUXL2LE046`9%?`j3vaQ zk0k~?4WaI(7?`93hmT3|Y@3$pYu7ZjYEqWMdK!QkSBr~)g|oHucEjW|_T!v4eMwyS zCmKiEMh}1$pqBCL%AqAi5J)j2#EXuAB)<;s817FZwy4PiFoGz0*XsXN!TcXx}0x9rmfR>yp2B6l41(V#){GP3E+LNwywE(37zA1H06X6Z5 zT1}SyOJ8_^|2!yh@MDd^m8QC5Jjf8o;7V0l{JS4{)u9tU+CO1azE(j81@v`nG;7>~ z=aDabS@=iPtc_7CWmeFpBtDMBUpfgD103wT5U)#D!`LKMjd41zKQaf1B*r1tX3TWy z-JtKR&B2L7gRURq;5U%*ldysD>p6IY1CW_w>@Gxzmp6%_5Yt;Vi>z}6X$dc5X3v6k zgNGWK)k1I@o|CkCHOZ)$WThbW&@|HbTB*2=gCm+(i(`dMImn8`QgjCF%%2;U+w^6`h;d}~BN|otlQ4^Qk``rs8 z!p-Ot$Di>M_^Xvd3kS7o`RJa1eNlA5lfQOsDY&0aEmV+*KZ=_W&SqY0134_O9oj=M zILUF$d9016u}lw)gm;DDBNn#^6O##j^i)Nl4zbXc@FnMdHabf^%P%KTRwe6jJZXA{ zaez|_@I}!$QXUhxS%4?}{F03Lf#evIDuHzFE`-ZFHS@D>@@^x!yM2?nyD3Tcy0Hsz zD&=I3kgRwk0O?cGcKnMVfn>XZm}E15f`FhJKkSdR3q{6Y8Am;m z6K#wWX=uW<(1&ZlT{Vt?*lFPj3IBrxq>2?(Lz76pEmbD{x6~X|fDKioGHK31qq@nW z7Cw)@AZt8u79Z_HB)())ek=V7DN4efwGsK{hNj%K8#x!2y~G1g&)ow!fJb31a7Ac` zHMu?<#cEGBMJ`5i2ycxXOKJ}G_Hnkw&cFnU!tdXCZ=Bu4pG^stObMAAR}1M;zRb$3 zB!cwc=eaEtcrDR5rHo)lvw9w3vZR!d#$*!nQrZ+{K4i6!yc_^qWK98%AW2|9jtMb! zu4}12HOf-aaybqVnd&_zd<^|hTqqR@OD7}&htf4PXAol9xp)}@Xggj|{w9J^SO!+u zBazQ$ff6PElFj%-zR_z(*Mc7qZf#g& z-cx|e2^$_@qKZ8>G)~90#?rb6+&0=EcTBJywNgBX_X9zr6da{Jn)O?>_$|7VdNMER zl~S8dFw*T*YL;Le_*3t+nGZx?e(KEVKYY_gVhN^}edNX4=c!(w>#wf9$+#lL!_6R} z7kQz{k;{oGES!;at#n>Cp1fvuV}!HLM9*Y^;r*Kqh&cX$PO@mHf~2QOWo0hK4!wTJDArQ`VEtm$4@oBm zRZY|bCem~sMz0i<4KfIXo^!<1QF2d==J7L}{QUXOvtJ9Tl6xRfhOSCn!a0l3 zg1GQ;DEAgUEY3&dn>kDuMNv4Q|?WSq|S zS1VKF5-To&YkcjYpMLG9$8AEM(#P8g%3h;|^8intbN|PVo7r^2ejNIUuK$az;a7G~ zJsGmX{D{`FBNv#yON6JphP4d;Z0o?@M)f|1cb&QJ?5J=9%x0(Lc!aNNGD zwGq3icVM7YYw~13^w80sNH!5n08L$3;Q`$?3$1w#$~os~R~U1!Dbk7PkHJcNHWB@- z?ukBlFfr;|{W5 z1Jm4#ae)`j$uXOGCN2e-T@`A6v6LO_{dTlm0G0g1IX}% zz)GePB>{L03!9-_Qpyo0AY%RJi;`SeWf5gIrPJ)a(p?GD$LS(zsc^~ zY=4T@kUd?dMR$f5+sLPsk5Y$q&uJQPZ)ik{Fk#36b;+G_u0scBN>pNSS7t}^AvtNS z3c4iWKDtTddGRVp&jQ+wQ0xXq#EdaE##AM1Zcr3;cJeJ+v%m_nKUcaXNKL4)G~5D( zNT{PQ)k!dHc9N7DkRIubk#0Ib1wJ1Cm2j<#ZrrtIwUBAn3(#70AjvdZE|j=bE!A<{ zxPmsdR%_!ohM`GSX{t?1UMeP+sMeMnezjiqD|NhO3|hPSqb+mJK~*>YYjp0nE)0*@ zL;~d0v@h(Bhiombn;?jZ^|USIq<8~cLSdAErumU6%jAJfCWs`}y z{U4=evsi~xA5G6+g5{0QkvtlA_>dYLs-=vbPkKjm1glZP*>`35_Hg8ekp(cjzA>T?LH@m{^y!i-NB? zT2cRos)Cd7!j8MHPr@K{bV$d38705Q0%^M8*xD*p(4S|T)j{;nc3#3Ihi1O}+>4{p zGjoyitqa08PbW^=#vPlEQdAK$HK!@GR-iETZSpI!Cv-f)h0G8YY`Gb*f`C|em45{EoX$$9<7sIIT^HlOXov?zto)#9jv1G zg1>9gBbf_?7!s~8F;ks3Jf-KqRMQ|d9#j}v`sg>0Y(LgF4Ak;#k`Mca0bC-~`oI1A zw;hMFB{WaLVOyMk!(omm&pGW;7f!U|*rQCM3x?gnKS3Z&YhVLi0JK?I5V-Ccd3)L{ z<A326`C4hXY%I=|*SiU^hP-`? znJ1_Xswn%ImVLt$7baej*eth>Yma#R=(gK#&>oG%bs{R?TcpdpVl|I>I26 z3&M9Cw|HkIk}&1;fuvifXfLWAdw2^_^jk+RKDK6$HN4en8V;f2#5Q}xC1OYYir1=o z4YUENBYXYi=F^vWR9I#2G3hQN!uQX3B#!s7#kZZ zhY;XjbOF#npCE}?Oh^yeCs`&NBoU$7-Ts)bXi3pL<367kcY4Cf{+67JK_|*V00(?F zdh5)Y;jG9sou77(k#!2Y9U68PGJ)IT5fC3ZN*JH@xvBH|mJth>Z!dO+9(jM3Wrr>%c0+@Vn0YMBo0-!X@yy_|CrSJMSPu$MzIcpl@N?Y-!OoOucr+B{plCSLVaZK0sJQO%>BaJf~7=Z6T1X@XT@6Rct> zn_s&LR&t4{+dAvikCT{>hX5%LZ_YjK{@ah6Is4FezW?!a&wYC4!g-1iI|e}9py=#E zjn#;@efr|ixvt$nSc!0Q?U+rl0K7bzpkx$a3{j$sC{@<_zFyNTHDwzyT;NUNEkq|w=i1N=Pse84v$wTQUET| z-GhgXeeBw%!$P=FXcli7 z`^kxB9#HOY!h{+Fz@xF2F;a*cp$wC+mlcg9BoO$NzO{?M3XEQh-wh*)pTr3yXZPgB z^t&WD^KH>H9e{E}hy{E$3^h5xs>=eX_4|zWcG<;HoP4!2d{^(BJbz`qh%UNu$=Otc z{nhQO@*t6T0~K~i5(l=~I~V?jzxnSa@HjfAZwuqcd!&)^mi9; z-B3dkXshW+WxI`(3sVm`u@zE$Np$-UZrJ3t^0hqP`PRfij_Bmy-4;EHCWsN|V>Q&n z0S7&?5FgBx*cYdkGz*VC;y7rB1prv$B^dyIZEbD*M+gM|g%Dk zlU!q&F$XNV$#=brV>9)LQqh8Noi=cj0poZ(Uh_bhfSvCJH+*_xipg)bTO zQFM0)n3?hBRh!TfFZO(;-?k8K4p;|{<9KMAjpRlgNPOb@wVG|u+C|i^+X`%w%97Dx zVKw$SYbHbv*6`6o9EQwv=ccv9Ym>;;q42OdS#9AbYbUzJ-v0zlQxs(bifR)#c$R!f zi$w>27mh?mKe_eA;s3loz^>uZAkEknJ-#H$(_}7^9l}1OTAMQl zBa{#7hp;eDnX(tTr0E(!BlH?(wQ<(Dv3HxMB$LF~loEhn;-S}oLxH@z7^~{=MTjl^b}`b-vLow*0DkJ3Vk37+| zz7ir5i`W{KQL>h_CU&j0*FIAR=rJSj2~L#YzP)xJV*57?*y1E&J#CcM+N$Yq6i|+_ zO+g+U1a7)K&PMSK*)kO_^zAQr{6^DnU=;|nW&ZZ)$M1Mqbkd!*Q_x#!MZz|!ErMW-=8~m=E={!{j`~rj=%qBr^5X;vC9uD(?9(C zd+2N6<)L5$NJn7o8KnK`WyY^MA*#?Ik<)ND_>%jC>%#m2{+O&Mwn@?+v5xp9K>CVK zdn(k?#)iTY#k?V8q>47gI(DVN*Hj$AZ^pYz@FI(8(FrscF7X&`zLYKF$jm=^l)0Wy ztH|#|oEIIVoBm@GFo?J6t2KLhAjyq#kQ5RMG{jT*N|!T&Yj$5{>prb)H?P|M+kC&%d>j z@p;07juXa1B*X+I$Q|uSCDH&PZ)`rScbpS0$;lf@7{|IA2J3f+s?yw(k%cV=<8ODyr#!QGC&dtZXwMlF%i(Q zHd{x@JPe*7*J1-0fIji1lNx<4faFwVubD}nIHjc@&&jlpOffDW`Jxh>d^l WHD}H&zi{PIVsM9bjuMw@H~(*vhe5~y delta 5362 zcmbtYeT*Ds8F$(O{jgtfJ!r2j(?VP9-7Y(G`!Rb^+rx-pKltdSXf>91XWrc%cXnnw zAGdc+^twg&+qv?zvuVN-WSe&Wpv7c4`$BEr?l*VR_x35_Gy`Pp&*o5Lt@YVsok2^ zm&>KIhPL_fAoP50^@mUE89krMoOw%D-1SbUI5KPTr8#5A6MGk?;N>pJr9}bv#>`kGuu~9<~V8f+Oq0+l|W@aSDE8;W(}*JtAtr z8uKl+!b0Asno+0*4bClu;S9Hq(WtVK01^37SjE;g-t0jL79Qr7ZHv1mg7>+LokLu6 z9PH#XN3~d;Rk#F+o)Uz>r%n(AVPw`+7OGlis7Gyb)oX-y-QI>r;Ghi_R@JiY2Y{Nu zlGmnrK^T}maw)biTG5d$8d^ci7J7}GQP4D*$C#~E=1tMk1w*{K^n`MbOxt;9rp@!g zj1UyTxzr{L5Y;eHQbSL*e1^X`L6ZX^+f_@v3_GrtC^f_&B|XD znU_j!sw4ryZiW%=qEhGbqAYWlm@GL&NJWP%rz^W-pB-!f4RORHPr7_4#Fvc}U@$JLQnvL>pKkQ!^mK2b zC?0vBdF9JXU;OLM{dXK94S@urAg_n41(DC6U;T$oQ!WzWuby+e86DvRsS#2&imx$4Y_RAUgsL_|}LzLywAW>QE*GX(i&8ejGoc zm3i*b>P?VCT5kFr_qn{;esa^6vDcCcBxXtYe7h7aqFbqK&?f>W)f;`8`YFWKjo-*9 zoioxu34sMYq9)#ZHlJ-&eHO^_l4w{Q_*+!G*u%0bE^?AoD**}od|VLyn>t2rzs3W4 zf$hjP*G4Twn>CT@&~^wL9J?Y@j1U9thUSNw;LH!MAOUfyr2_c|@15q_RaSvjA;@x zf%^fwJ$wTqq=Jr(-(}~Pae*IR06B#kBNm5?-0_toRX~K+ z39Eu|r57ln2+V4mu)mb$6Yz2b_mU()P(n%+kOSoF z=!V3&azx2*|3o5Viv=gtSe>WL7tZmixdeX`gsI@1C?giZ;vfNN)M-Yk)qszM{2-2z?85VP}Lfkpp9lxL{-Xz0_*XYWU$%df|<+&S2A3pH$ zWFcOyDR+u@k8NB|1{ih;zK_aX?sy7*2w!GH+d}8)9fhb*(Rpx13OPklQ?D&0 z4Q?Fh6jeV|Ol}B0pfLY*yz7Eb0A9>IkrJOj(VYQoj+1KZT8UTLpzZl*MR=ol`^3Vl zJR*aXHK|afR19Z?x{P};$uyq>amr1rV65?ch`$|LG~0qc;wEBeQ1@zL?&v(_Rtfv= zlQUyuni?n3GgCXRs728o3nDyNMKZ2OQlXA)%=uRpVN zS-T{jx;`pFXjM!Ql?YBKk!=zV$hLUu_|>zatEdyw9a7#NvFb$6{|fr~BNM8*jo=^e zUqGJ?6l~~%xu^I5 zc`=LP$&2;x%stg1B)^>14PnsGdf|q4o;J$rX&oERx=?)S*n;`np1J*uvGh@wY~XG& z>%Gp=yN}&HW%Qr_xNgeNo>LlA#=bXQnSB1(lgkxzjuOZHHz^%s-|AD|pA@esC~V}; zgi0iv(B}3~vm(GSTDP6y6o$JHIf9JHZY?$2Yj8Iy3=pO@r;x)XJ*Y-?=E{yrB6~{v zgh(Z`spXlWZ&NFSoQ{Otth%<8eS6Rg+x)cKRf?0y7C`aI$;7O=Uen=LB~j!uo`@5Y z|CIv{O%ut{Rk)iv>N!nRfo1kVxLP4QR$3b&1HiwM+;_#u4 ztH^;@=o;LK?V@E#o72W3p&m4Dkd62Jx*Y&7%rGrt%f5eGzZ*1RDcVrefnl{81Xlvy zcuitfhaF))IXdc06r`N5gQye?qqEm1DoMI5nvakYK9sZ{_y~vI0-W-w!7e*QTS70| zhZxAogu~CTKKA2Rrmt!|62G)Vx$wU%f<@x+mJZ0(L=|M>xpk#;)@hft_~07lnz6#Q z%D$;%FApf+o)o`vgVGm2wn|wv^5wO5&7-F4P>xZv3{zgxq>~v483QXlKNaBZ1Jcj% zng9*4<*1tY58?~G3finm6Kn7ag9IK)3NWY!HVv$;zK0(ZRfBu53_g+i^aj~=FjLZS zPEN$exX}Oa83JFRoGM&D>9d8wEFlIl-Rz*|O3(_VzcF{qr{+ zU6a%*;;Oeto;RXiOEGzS3Qj_LHD% zcS3@70-p67TQ-&OaXhb<1p#e|H<0D~c^bE!mzfaf=p=_8pbH{pU zMM!#q+7MsZe91;L!uu0gEZLM>yCnu#2Hmm<6Wd5L7T^U_1GO~nk&K26iecHvW(1K<0fwHn Date: Sun, 12 Jul 2026 10:53:37 +0100 Subject: [PATCH 072/135] Activate configured Move proof sources --- docs/implementation-plan.md | 14 ++- notes/current-system-map.md | 12 +- src/Mimir.BufferSmoke/Program.cs | 74 +++++++++-- src/Mimir.Runtime/MimirRuntime.cs | 115 +++++++++++++++++- .../MimirMoveProofRuntimeActivation.cs | 57 +++++++++ state/evidence.jsonl | 1 + state/map.yaml | 12 +- 7 files changed, 260 insertions(+), 25 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 4f48493..43e5656 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -110,9 +110,17 @@ a named invariant that the native runtime cannot protect yet. and pose frame prefixes, Fensalir presented-frame prefix, fusion authority, consumer contract, and at least two calibrated camera witnesses. It creates `MimirMoveProofRuntimeDriver` only after the supplied ring stream id matches - the configured evidence stream. The current C# CultMesh ring used by the - smoke is still in-process; real Nightwing/Starfire cross-process ring opening - remains pending. + the configured evidence stream. +- `MimirRuntime` now owns configured Move proof activation instead of leaving + config as a readout. At scene-ready it asks an + `IMimirMoveProofEvidenceRingProvider` for the configured Muninn evidence ring, + opens the configured native reservoir, creates the driver, retains the + resources, and exposes an activation status. The default provider fails + explicitly because the current C# CultMesh ring is still in-process only; + `Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an + in-process provider and proves config activation can produce the same named + proof chain. Real Nightwing/Starfire cross-process ring opening remains + pending. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so `Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 024da22..c011030 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -230,9 +230,15 @@ surface for that bridge. An enabled source must name the Muninn evidence stream, native reservoir path, Mimir evidence/pose frame prefixes, Fensalir presented-frame prefix, fusion authority, consumer contract, and at least two calibrated camera witnesses. It may create a runtime driver only when the -supplied CultMesh ring stream id matches the configured evidence stream. This -does not yet open a cross-process live ring; the current smoke ring is -in-process proof of the Mimir-side contract. +supplied CultMesh ring stream id matches the configured evidence stream. +`MimirRuntime` now owns activation for configured proof sources: on scene-ready +it asks an `IMimirMoveProofEvidenceRingProvider` for the evidence ring, opens +the configured native reservoir, registers the runtime driver, retains the +driver resources, and exposes an activation status. The default provider fails +explicitly because the current C# CultMesh ring is still in-process only; +`Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an in-process +provider to prove the activation path without claiming live Nightwing field +evidence. `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 5ec3993..c8453e2 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -158,13 +158,22 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-frame-smoke", Strin if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-driver-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveProofRuntimeDriverSmoke( - ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), + activateViaConfiguredSource: false); } if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-config-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveProofRuntimeDriverSmoke( - ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), + activateViaConfiguredSource: false); +} + +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-activation-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeDriverSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), + activateViaConfiguredSource: true); } if (args.Any(arg => string.Equals(arg, "--move-proof-presented-frame-smoke", StringComparison.OrdinalIgnoreCase))) @@ -2276,7 +2285,7 @@ static int RunMoveProofRuntimeFrameSmoke() : 1; } -static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) +static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath, bool activateViaConfiguredSource) { if (!File.Exists(nativeReservoirPath)) { @@ -2408,26 +2417,47 @@ static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath) } catalog.PublishFrame(published); - using var reservoir = new MimirNativeReservoirRuntime(nativeReservoirPath); - var driver = configuration.CreateDriver(ring, reservoir); - var runtime = new MimirRuntime( - new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), - new MimirSynchronizationSettings()); - runtime.RegisterMoveProofDriver(driver); + using var manualReservoir = activateViaConfiguredSource ? null : new MimirNativeReservoirRuntime(nativeReservoirPath); + var manualDriver = activateViaConfiguredSource + ? null + : configuration.CreateDriver(ring, manualReservoir!); + using var runtime = activateViaConfiguredSource + ? new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), + new MimirRuntimeConfiguration + { + Settings = new MimirSynchronizationSettings(), + MoveProofSources = [configuration], + }, + new InProcessMoveProofEvidenceRingProvider(ring)) + : new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), + new MimirSynchronizationSettings()); + if (manualDriver is not null) + { + runtime.RegisterMoveProofDriver(manualDriver); + } + runtime.OnSceneReady(); var input = new InputState(); runtime.Update(0.016f, input); var proofFrame = runtime.Frame; - var proofSurface = driver.LastResult?.ProofSurface; + var proofSurface = activateViaConfiguredSource + ? runtime.LatestMoveProofSurface + : manualDriver?.LastResult?.ProofSurface; var proofSplineCount = proofFrame.Scene.SplineFrame.Splines.Count(spline => spline.Id.StartsWith("move-proof-", StringComparison.Ordinal)); runtime.Update(0.016f, input); - var duplicateSuppressed = reservoir.Status.MoveEvidenceCount.ToUInt64() == 1UL; + var duplicateSuppressed = activateViaConfiguredSource || manualReservoir!.Status.MoveEvidenceCount.ToUInt64() == 1UL; + var activationStatus = runtime.MoveProofActivationStatuses.LastOrDefault(status => + string.Equals(status.EvidenceStreamId, streamId, StringComparison.Ordinal)); + var activationOk = !activateViaConfiguredSource || activationStatus?.Active == true; Console.WriteLine( - $"move-proof-runtime-driver-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length}"); + $"move-proof-runtime-{(activateViaConfiguredSource ? "activation" : "driver")}-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length} activationActive={activationStatus?.Active.ToString() ?? "n/a"} activationDiagnostic=\"{activationStatus?.Diagnostic ?? ""}\""); return proofSurface is not null && + activationOk && proofSurface.IsFullPose && proofSurface.MuninnEvidenceFrameId == "muninn:nightwing:move-evidence:79" && proofSurface.MimirEvidenceFrameId == "mimir:starfire:move-evidence:79" && @@ -7149,6 +7179,26 @@ sealed record MoveTrackingSmokeContext(MimirSynchronizationHub Hub, int Consumed public void Dispose() => Hub.Dispose(); } +internal sealed class InProcessMoveProofEvidenceRingProvider(CultMeshSharedMemoryFrameRing ring) : IMimirMoveProofEvidenceRingProvider +{ + public bool TryOpenEvidenceRing( + MimirMoveProofRuntimeConfiguration configuration, + out MimirMoveProofEvidenceRingLease? lease, + out string diagnostic) + { + if (!string.Equals(ring.StreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)) + { + lease = null; + diagnostic = $"in-process ring stream '{ring.StreamId}' does not match configured evidence stream '{configuration.EvidenceStreamId}'"; + return false; + } + + lease = new MimirMoveProofEvidenceRingLease(ring); + diagnostic = "in-process test ring supplied"; + return true; + } +} + public sealed record ComplexContourReflectionCorrection( double RelativeDelaySamples, int ObservationCount, diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 0939124..0739956 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -40,6 +40,7 @@ direct private readonly MimirAudioSpectrumAnalyzer spectrumAnalyzer; private readonly IReadOnlyList sourceFactories; private readonly IReadOnlyList configuredMoveProofSources; + private readonly IMimirMoveProofEvidenceRingProvider moveProofRingProvider; private readonly AquariumUiDocument ui; private readonly AquariumAudioDocument audio = new(); private readonly MimirAudioSynchronizationSettings audioSyncSettings; @@ -64,7 +65,10 @@ direct private IReadOnlyList lastAudioSpectra = []; private readonly Queue> spectrumHistory = new(); private readonly List moveProofDrivers = []; + private readonly List moveProofDriverResources = []; + private readonly List moveProofActivationStatuses = []; private MimirMoveProofSurfaceDocument? latestMoveProofSurface; + private bool moveProofSourcesActivated; public MimirRuntime(AquariumRuntimeOptions options) : this(options, MimirRuntimeConfiguration.Load()) @@ -72,7 +76,20 @@ direct } public MimirRuntime(AquariumRuntimeOptions options, MimirRuntimeConfiguration configuration) - : this(options, configuration.Settings, configuration.SourceFactories, configuration.MoveProofSources) + : this( + options, + configuration.Settings, + configuration.SourceFactories, + configuration.MoveProofSources, + MimirUnavailableMoveProofEvidenceRingProvider.Instance) + { + } + + public MimirRuntime( + AquariumRuntimeOptions options, + MimirRuntimeConfiguration configuration, + IMimirMoveProofEvidenceRingProvider moveProofRingProvider) + : this(options, configuration.Settings, configuration.SourceFactories, configuration.MoveProofSources, moveProofRingProvider) { } @@ -85,7 +102,12 @@ direct AquariumRuntimeOptions options, MimirSynchronizationSettings settings, IEnumerable streamSources) - : this(options, settings, Array.Empty(), Array.Empty()) + : this( + options, + settings, + Array.Empty(), + Array.Empty(), + MimirUnavailableMoveProofEvidenceRingProvider.Instance) { foreach (var source in streamSources) { @@ -97,13 +119,15 @@ direct AquariumRuntimeOptions options, MimirSynchronizationSettings settings, IEnumerable sourceFactories, - IEnumerable moveProofSources) + IEnumerable moveProofSources, + IMimirMoveProofEvidenceRingProvider moveProofRingProvider) { Options = options; synchronization = new MimirSynchronizationHub(settings); spectrumAnalyzer = new MimirAudioSpectrumAnalyzer(ParseSpectrumFftSize()); this.sourceFactories = sourceFactories.ToArray(); configuredMoveProofSources = moveProofSources.ToArray(); + this.moveProofRingProvider = moveProofRingProvider ?? throw new ArgumentNullException(nameof(moveProofRingProvider)); audioSyncSettings = settings.Audio; telemetryIntervalSeconds = ParseTelemetryIntervalSeconds(); audioSyncUpdateIntervalSeconds = ParseAudioSyncIntervalSeconds(); @@ -138,6 +162,10 @@ direct public AquariumAudioDocument Audio => audio; + public IReadOnlyList MoveProofActivationStatuses => moveProofActivationStatuses; + + public MimirMoveProofSurfaceDocument? LatestMoveProofSurface => latestMoveProofSurface; + private AquariumFrame CreateFrame() { var channelCount = Math.Max(1, lastAudioSpectra.Count); @@ -196,6 +224,20 @@ direct moveProofDrivers.Add(driver); } + public void ActivateConfiguredMoveProofSources() + { + if (moveProofSourcesActivated) + { + return; + } + + moveProofSourcesActivated = true; + foreach (var configuration in configuredMoveProofSources) + { + ActivateConfiguredMoveProofSource(configuration); + } + } + public static AquariumSplineFrame ComposeFensalirSplineFrame( AquariumSplineFrame baseFrame, MimirMoveProofSurfaceDocument? moveProofSurface) @@ -270,6 +312,11 @@ direct public void Dispose() { + foreach (var resource in moveProofDriverResources.AsEnumerable().Reverse()) + { + resource.Dispose(); + } + synchronization.Dispose(); } @@ -339,7 +386,19 @@ direct return string.Join( Environment.NewLine, configuredMoveProofSources.Select(source => - $"{source.EvidenceStreamId} -> {source.MimirPoseFramePrefix}: cameras={source.Calibration.Cameras.Count}")); + $"{source.EvidenceStreamId} -> {source.MimirPoseFramePrefix}: cameras={source.Calibration.Cameras.Count} {DescribeMoveProofActivation(source.EvidenceStreamId)}")); + } + + private string DescribeMoveProofActivation(string evidenceStreamId) + { + var status = moveProofActivationStatuses.LastOrDefault(status => + string.Equals(status.EvidenceStreamId, evidenceStreamId, StringComparison.Ordinal)); + if (status is null) + { + return "not activated"; + } + + return status.Active ? "active" : $"inactive: {status.Diagnostic}"; } private void UpdateMoveProofSurfaces() @@ -558,6 +617,54 @@ direct synchronization.AddSource(source); } } + + ActivateConfiguredMoveProofSources(); + } + + private void ActivateConfiguredMoveProofSource(MimirMoveProofRuntimeConfiguration configuration) + { + try + { + var errors = configuration.Validate(); + if (errors.Length > 0) + { + RecordMoveProofActivation(configuration.EvidenceStreamId, false, $"invalid: {string.Join("; ", errors)}"); + return; + } + + if (!moveProofRingProvider.TryOpenEvidenceRing(configuration, out var ringLease, out var diagnostic) || ringLease is null) + { + RecordMoveProofActivation(configuration.EvidenceStreamId, false, diagnostic); + return; + } + + try + { + var reservoir = new MimirNativeReservoirRuntime(configuration.NativeReservoirPath); + var driver = configuration.CreateDriver(ringLease.Ring, reservoir); + moveProofDrivers.Add(driver); + moveProofDriverResources.Add(reservoir); + moveProofDriverResources.Add(ringLease); + RecordMoveProofActivation(configuration.EvidenceStreamId, true, diagnostic); + } + catch + { + ringLease.Dispose(); + throw; + } + } + catch (Exception exception) + { + RecordMoveProofActivation(configuration.EvidenceStreamId, false, exception.Message); + } + } + + private void RecordMoveProofActivation(string evidenceStreamId, bool active, string diagnostic) + { + moveProofActivationStatuses.Add(new MimirMoveProofRuntimeActivationStatus( + evidenceStreamId, + active, + string.IsNullOrWhiteSpace(diagnostic) ? (active ? "activated" : "not activated") : diagnostic)); } private void EmitTelemetry() diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs new file mode 100644 index 0000000..98bba90 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs @@ -0,0 +1,57 @@ +using GameCult.Mesh; + +namespace Mimir.Runtime.Synchronization; + +public interface IMimirMoveProofEvidenceRingProvider +{ + bool TryOpenEvidenceRing( + MimirMoveProofRuntimeConfiguration configuration, + out MimirMoveProofEvidenceRingLease? lease, + out string diagnostic); +} + +public sealed class MimirMoveProofEvidenceRingLease : IDisposable +{ + private readonly bool ownsRing; + + public MimirMoveProofEvidenceRingLease(CultMeshSharedMemoryFrameRing ring, bool ownsRing = false) + { + Ring = ring ?? throw new ArgumentNullException(nameof(ring)); + this.ownsRing = ownsRing; + } + + public CultMeshSharedMemoryFrameRing Ring { get; } + + public void Dispose() + { + if (ownsRing) + { + Ring.Dispose(); + } + } +} + +public sealed record MimirMoveProofRuntimeActivationStatus( + string EvidenceStreamId, + bool Active, + string Diagnostic); + +public sealed class MimirUnavailableMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider +{ + public static MimirUnavailableMoveProofEvidenceRingProvider Instance { get; } = new(); + + private MimirUnavailableMoveProofEvidenceRingProvider() + { + } + + public bool TryOpenEvidenceRing( + MimirMoveProofRuntimeConfiguration configuration, + out MimirMoveProofEvidenceRingLease? lease, + out string diagnostic) + { + ArgumentNullException.ThrowIfNull(configuration); + lease = null; + diagnostic = "No live CultMesh shared-memory ring opener is configured for this runtime; C# CultMesh rings are currently in-process only."; + return false; + } +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 38c53da..d6daf69 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -165,3 +165,4 @@ {"ts": "2026-07-08T22:48:42+01:00", "type": "implementation", "status": "ok", "note": "Added MimirMoveProofRuntimeDriver as the runtime bridge from a Muninn CultMesh shared-memory evidence ring to MimirRuntime.PublishMoveProofSurface. The driver derives mimir:starfire evidence/pose ids and fensalir:starfire presented-frame id from the actual Muninn frame suffix, calls MimirMoveProofPipeline, and suppresses duplicate frame admission. Verified dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke against the native reservoir debug DLL, --move-proof-runtime-frame-smoke, --move-proof-surface-smoke, and --move-proof-pipeline-smoke. This proves the runtime-owned ring-to-visible-probe path with synthetic ring input; real Nightwing hardware, calibrated camera witnesses, and real presented-frame capture remain pending."} {"ts": "2026-07-08T22:55:54+01:00", "type": "verification", "status": "ok", "note": "Added and verified the Fensalir presented-frame proof smoke. MIMIR_MOVE_PROOF_DEV_SURFACE gates a synthetic calibrated proof surface through the same MimirRuntime.PublishMoveProofSurface path, and BufferSmoke --move-proof-presented-frame-smoke runs Mimir.App headless, captures artifacts/runtime/move-proof-presented-frame-smoke.png, and pixel-checks the rendered proof probe. Latest run captured fensalir:starfire:presented-frame:79 for proof mimir:starfire:move-proof:79 with bytes=11260 and brightPixels=1069; adjacent runtime-driver, runtime-frame, proof-surface, and proof-pipeline smokes pass. This proves the renderer/probe layer for the named proof chain, not live Nightwing hardware or calibrated field evidence."} {"ts": "2026-07-12T10:26:42+01:00", "type": "verification", "status": "ok", "note": "Added MimirMoveProofRuntimeConfiguration as the typed config/validation surface for Move proof driver creation. The runtime config now accepts enabled MoveProof sources, validates named Muninn evidence stream, native reservoir path, Mimir/Fensalir frame prefixes, fusion authority, consumer contract, and at least two calibrated camera witnesses, and creates a driver only when the supplied CultMesh ring stream id matches. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-config-smoke, --move-proof-runtime-driver-smoke, --move-proof-pipeline-smoke, and --move-proof-presented-frame-smoke. The config smoke printed configValid=True invalidDefaultErrors=2 and the same sequence chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79. This proves config plus in-process ring flow, not live Nightwing hardware, calibrated field cameras, or cross-process ring opening."} +{"ts": "2026-07-12T10:51:37+01:00", "type": "verification", "status": "ok", "note": "MimirRuntime now owns configured Move proof activation instead of leaving MoveProof config as a passive readout. On scene-ready it asks IMimirMoveProofEvidenceRingProvider for the configured Muninn evidence ring, opens the configured native reservoir, creates/registers MimirMoveProofRuntimeDriver, retains resources, and records activation status. The default provider fails explicitly because the current C# CultMesh ring is in-process only. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, and BufferSmoke --move-proof-runtime-activation-smoke. The activation smoke injected an in-process provider and printed activationActive=True while preserving the explicit chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79 with verdict=FullPose. This proves runtime config activation and status, not live Nightwing/Starfire hardware or cross-process ring opening."} diff --git a/state/map.yaml b/state/map.yaml index 571edd5..94b14bc 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -161,9 +161,15 @@ accepted_design: Fensalir presented-frame prefix, fusion authority, consumer contract, and at least two calibrated camera witnesses. It may create a `MimirMoveProofRuntimeDriver` only when the supplied CultMesh ring stream id - matches the configured evidence stream. The current C# shared-memory ring - smoke is still in-process; real Nightwing/Starfire cross-process ring - opening remains pending. + matches the configured evidence stream. + - `MimirRuntime` owns configured Move proof activation. On scene-ready it asks + an `IMimirMoveProofEvidenceRingProvider` for the evidence ring, opens the + configured native reservoir, registers the runtime driver, retains the + driver resources, and exposes activation status. The default provider fails + explicitly because the current C# CultMesh ring is still in-process only; + `Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an + in-process provider to prove the activation path without claiming live + Nightwing/Starfire field evidence. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same MimirRuntime attachment so headless Mimir.App can From 4a968319ac875cbd8c428081864ed11a8a667403 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 10:53:52 +0100 Subject: [PATCH 073/135] Persist Mimir Face heartbeat --- .voidbot/state/mimir.cc | Bin 108447 -> 108447 bytes .voidbot/status/void-memory-maintenance.json | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 66634d68e299568fceb6d619829f8e04da609ac0..5c2f7129330229e9f217fab136845971f8e41355 100644 GIT binary patch delta 169 zcmbP#o^AelwhcddIZchN42<-QjV&fmd@eRwg0CAUVgeS~e2CB51}0=`22!+H#(QB9 zOvuaxEOcjXCnNjaO}czxERvIR=6k@TfyO@Im4xxkjlhb(&Y$cK6S4pqIUUBcG}vtT OB4s}p#8D;&CQ$&ASvIc# delta 168 zcmbP#o^AelwhcddIZceMObztREDa}5d@eRwg0GtcCNi0kPki$sK5HAO8gm0vkebah z-V1|ZLWag*p*wRs8QJf1*%mCm&NVq_z6ab?gUJtgC1Hw<48V%N&Y$cK7qSEjO^5M} S%{LpqNZHQ?ae=vsMHB%3IW`gi diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index cd66b95..db88ea1 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-07-12T09:21:38.343Z", + "startedAt": "2026-07-12T09:49:47.786Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_napping_with_short_term_memory" From 32bf6fa5180938a54933be01859e6b2db5a77ab0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 11:01:29 +0100 Subject: [PATCH 074/135] Add Move proof activation surface --- docs/implementation-plan.md | 8 ++ notes/current-system-map.md | 8 ++ src/Mimir.BufferSmoke/Program.cs | 58 ++++++++++++- src/Mimir.Runtime/MimirRuntime.cs | 41 +++++++--- .../MimirMoveProofRuntimeActivation.cs | 81 ++++++++++++++++++- state/evidence.jsonl | 1 + state/map.yaml | 7 ++ 7 files changed, 190 insertions(+), 14 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 43e5656..887abcc 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -121,6 +121,14 @@ a named invariant that the native runtime cannot protect yet. in-process provider and proves config activation can produce the same named proof chain. Real Nightwing/Starfire cross-process ring opening remains pending. +- `mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine + surface for configured Move proof sources. It reports the configured stream, + provider kind, active/driver state, native reservoir path, calibrated camera + ids, latest same-stream proof ids, latest verdict, and the unavailable + default-provider diagnostic. `Mimir.BufferSmoke + --move-proof-runtime-activation-surface-smoke` proves both an active injected + ring document and the inactive default runtime document without claiming live + field evidence. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so `Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index c011030..ef4bb67 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -239,6 +239,14 @@ explicitly because the current C# CultMesh ring is still in-process only; `Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an in-process provider to prove the activation path without claiming live Nightwing field evidence. +`mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine +surface for configured Move proof sources. It reports the configured evidence +stream, provider kind, active/driver state, native reservoir path, calibrated +camera ids, latest same-stream proof ids, latest verdict, and the inactive +diagnostic when the default runtime lacks a live cross-process CultMesh ring +opener. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke` +proves both the active injected-ring document and the unavailable default +document; this is readiness/proof observation, not field evidence. `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index c8453e2..ea00f0f 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -173,7 +173,16 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-activation-smoke", { return RunMoveProofRuntimeDriverSmoke( ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), - activateViaConfiguredSource: true); + activateViaConfiguredSource: true, + requireActivationSurface: false); +} + +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-activation-surface-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeDriverSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), + activateViaConfiguredSource: true, + requireActivationSurface: true); } if (args.Any(arg => string.Equals(arg, "--move-proof-presented-frame-smoke", StringComparison.OrdinalIgnoreCase))) @@ -2285,7 +2294,10 @@ static int RunMoveProofRuntimeFrameSmoke() : 1; } -static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath, bool activateViaConfiguredSource) +static int RunMoveProofRuntimeDriverSmoke( + string nativeReservoirPath, + bool activateViaConfiguredSource, + bool requireActivationSurface = false) { if (!File.Exists(nativeReservoirPath)) { @@ -2452,12 +2464,52 @@ static int RunMoveProofRuntimeDriverSmoke(string nativeReservoirPath, bool activ var activationStatus = runtime.MoveProofActivationStatuses.LastOrDefault(status => string.Equals(status.EvidenceStreamId, streamId, StringComparison.Ordinal)); var activationOk = !activateViaConfiguredSource || activationStatus?.Active == true; + var activationDocuments = runtime.CreateMoveProofRuntimeActivationDocuments(1_781_202_600_006_000_000); + var activationDocument = activationDocuments.SingleOrDefault(document => + string.Equals(document.EvidenceStreamId, streamId, StringComparison.Ordinal)); + var activationSurfaceOk = !requireActivationSurface || activationDocument is + { + Active: true, + DriverRegistered: true, + LatestProofId: "mimir:starfire:move-proof:79", + LatestMuninnEvidenceFrameId: "muninn:nightwing:move-evidence:79", + LatestMimirEvidenceFrameId: "mimir:starfire:move-evidence:79", + LatestMimirPoseFrameId: "mimir:starfire:move-pose:79", + LatestFensalirFrameId: "fensalir:starfire:presented-frame:79", + LatestVerdict: MimirMoveProofVerdict.FullPose, + CalibratedCameraCount: 2 + }; + + MimirMoveProofRuntimeActivationDocument? unavailableDocument = null; + if (requireActivationSurface) + { + using var unavailableRuntime = new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), + new MimirRuntimeConfiguration + { + Settings = new MimirSynchronizationSettings(), + MoveProofSources = [configuration], + }); + unavailableRuntime.OnSceneReady(); + unavailableDocument = unavailableRuntime + .CreateMoveProofRuntimeActivationDocuments(1_781_202_600_007_000_000) + .SingleOrDefault(document => string.Equals(document.EvidenceStreamId, streamId, StringComparison.Ordinal)); + activationSurfaceOk = activationSurfaceOk && + unavailableDocument is + { + Active: false, + DriverRegistered: false, + LatestProofId: "", + } && + unavailableDocument.Diagnostic.Contains("No live CultMesh shared-memory ring opener", StringComparison.Ordinal); + } Console.WriteLine( - $"move-proof-runtime-{(activateViaConfiguredSource ? "activation" : "driver")}-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length} activationActive={activationStatus?.Active.ToString() ?? "n/a"} activationDiagnostic=\"{activationStatus?.Diagnostic ?? ""}\""); + $"move-proof-runtime-{(requireActivationSurface ? "activation-surface" : activateViaConfiguredSource ? "activation" : "driver")}-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length} activationActive={activationStatus?.Active.ToString() ?? "n/a"} activationDiagnostic=\"{activationStatus?.Diagnostic ?? ""}\" activationDocumentActive={activationDocument?.Active.ToString() ?? "n/a"} activationProvider=\"{activationDocument?.ProviderKind ?? ""}\" unavailableActive={unavailableDocument?.Active.ToString() ?? "n/a"} unavailableDiagnostic=\"{unavailableDocument?.Diagnostic ?? ""}\""); return proofSurface is not null && activationOk && + activationSurfaceOk && proofSurface.IsFullPose && proofSurface.MuninnEvidenceFrameId == "muninn:nightwing:move-evidence:79" && proofSurface.MimirEvidenceFrameId == "mimir:starfire:move-evidence:79" && diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index 0739956..f7fcff8 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -166,6 +166,16 @@ direct public MimirMoveProofSurfaceDocument? LatestMoveProofSurface => latestMoveProofSurface; + public IReadOnlyList CreateMoveProofRuntimeActivationDocuments(ulong observedAtNs) => + configuredMoveProofSources + .Select(configuration => MimirMoveProofRuntimeActivation.CreateDocument( + configuration, + moveProofActivationStatuses.LastOrDefault(status => + string.Equals(status.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)), + latestMoveProofSurface, + observedAtNs)) + .ToArray(); + private AquariumFrame CreateFrame() { var channelCount = Math.Max(1, lastAudioSpectra.Count); @@ -623,48 +633,59 @@ direct private void ActivateConfiguredMoveProofSource(MimirMoveProofRuntimeConfiguration configuration) { + var providerKind = moveProofRingProvider.GetType().Name; try { var errors = configuration.Validate(); if (errors.Length > 0) { - RecordMoveProofActivation(configuration.EvidenceStreamId, false, $"invalid: {string.Join("; ", errors)}"); + RecordMoveProofActivation(configuration.EvidenceStreamId, false, $"invalid: {string.Join("; ", errors)}", providerKind, driverRegistered: false); return; } if (!moveProofRingProvider.TryOpenEvidenceRing(configuration, out var ringLease, out var diagnostic) || ringLease is null) { - RecordMoveProofActivation(configuration.EvidenceStreamId, false, diagnostic); + RecordMoveProofActivation(configuration.EvidenceStreamId, false, diagnostic, providerKind, driverRegistered: false); return; } + MimirNativeReservoirRuntime? reservoir = null; try { - var reservoir = new MimirNativeReservoirRuntime(configuration.NativeReservoirPath); + reservoir = new MimirNativeReservoirRuntime(configuration.NativeReservoirPath); var driver = configuration.CreateDriver(ringLease.Ring, reservoir); moveProofDrivers.Add(driver); moveProofDriverResources.Add(reservoir); moveProofDriverResources.Add(ringLease); - RecordMoveProofActivation(configuration.EvidenceStreamId, true, diagnostic); + reservoir = null; + ringLease = null; + RecordMoveProofActivation(configuration.EvidenceStreamId, true, diagnostic, providerKind, driverRegistered: true); } - catch + finally { - ringLease.Dispose(); - throw; + reservoir?.Dispose(); + ringLease?.Dispose(); } } catch (Exception exception) { - RecordMoveProofActivation(configuration.EvidenceStreamId, false, exception.Message); + RecordMoveProofActivation(configuration.EvidenceStreamId, false, exception.Message, providerKind, driverRegistered: false); } } - private void RecordMoveProofActivation(string evidenceStreamId, bool active, string diagnostic) + private void RecordMoveProofActivation( + string evidenceStreamId, + bool active, + string diagnostic, + string providerKind, + bool driverRegistered) { moveProofActivationStatuses.Add(new MimirMoveProofRuntimeActivationStatus( evidenceStreamId, active, - string.IsNullOrWhiteSpace(diagnostic) ? (active ? "activated" : "not activated") : diagnostic)); + string.IsNullOrWhiteSpace(diagnostic) ? (active ? "activated" : "not activated") : diagnostic, + providerKind, + driverRegistered)); } private void EmitTelemetry() diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs index 98bba90..6f99183 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs @@ -1,4 +1,6 @@ using GameCult.Mesh; +using GameCult.Caching; +using MessagePack; namespace Mimir.Runtime.Synchronization; @@ -34,7 +36,84 @@ public sealed class MimirMoveProofEvidenceRingLease : IDisposable public sealed record MimirMoveProofRuntimeActivationStatus( string EvidenceStreamId, bool Active, - string Diagnostic); + string Diagnostic, + string ProviderKind, + bool DriverRegistered); + +[CultDocument("mimir.move_proof_runtime_activation", "mimir.move_proof_runtime_activation.v1")] +[MessagePackObject] +public sealed record MimirMoveProofRuntimeActivationDocument( + [property: Key(0)] + [property: CultName] + string ActivationId, + [property: Key(1)] string EvidenceStreamId, + [property: Key(2)] bool Active, + [property: Key(3)] string Diagnostic, + [property: Key(4)] string ProviderKind, + [property: Key(5)] bool DriverRegistered, + [property: Key(6)] string NativeReservoirPath, + [property: Key(7)] string[] CalibratedCameraIds, + [property: Key(8)] int CalibratedCameraCount, + [property: Key(9)] string LatestProofId, + [property: Key(10)] string LatestMuninnEvidenceFrameId, + [property: Key(11)] string LatestMimirEvidenceFrameId, + [property: Key(12)] string LatestMimirPoseFrameId, + [property: Key(13)] string LatestFensalirFrameId, + [property: Key(14)] MimirMoveProofVerdict LatestVerdict, + [property: Key(15)] ulong ObservedAtNs); + +public static class MimirMoveProofRuntimeActivation +{ + public static MimirMoveProofRuntimeActivationDocument CreateDocument( + MimirMoveProofRuntimeConfiguration configuration, + MimirMoveProofRuntimeActivationStatus? status, + MimirMoveProofSurfaceDocument? latestProofSurface, + ulong observedAtNs) + { + ArgumentNullException.ThrowIfNull(configuration); + var active = status?.Active ?? false; + var diagnostic = status?.Diagnostic ?? "not activated"; + var providerKind = status?.ProviderKind ?? ""; + var driverRegistered = status?.DriverRegistered ?? false; + var matchingSurface = IsSurfaceFromEvidenceStream(latestProofSurface, configuration.EvidenceStreamId) + ? latestProofSurface + : null; + + return new MimirMoveProofRuntimeActivationDocument( + ActivationId: $"{configuration.EvidenceStreamId}:activation", + EvidenceStreamId: configuration.EvidenceStreamId, + Active: active, + Diagnostic: diagnostic, + ProviderKind: providerKind, + DriverRegistered: driverRegistered, + NativeReservoirPath: configuration.NativeReservoirPath, + CalibratedCameraIds: configuration.Calibration.Cameras + .Select(camera => camera.CameraId) + .Where(cameraId => !string.IsNullOrWhiteSpace(cameraId)) + .ToArray(), + CalibratedCameraCount: configuration.Calibration.Cameras.Count, + LatestProofId: matchingSurface?.ProofId ?? "", + LatestMuninnEvidenceFrameId: matchingSurface?.MuninnEvidenceFrameId ?? "", + LatestMimirEvidenceFrameId: matchingSurface?.MimirEvidenceFrameId ?? "", + LatestMimirPoseFrameId: matchingSurface?.MimirPoseFrameId ?? "", + LatestFensalirFrameId: matchingSurface?.FensalirFrameId ?? "", + LatestVerdict: matchingSurface?.Verdict ?? MimirMoveProofVerdict.Unknown, + ObservedAtNs: observedAtNs); + } + + private static bool IsSurfaceFromEvidenceStream( + MimirMoveProofSurfaceDocument? latestProofSurface, + string evidenceStreamId) + { + if (latestProofSurface is null || string.IsNullOrWhiteSpace(evidenceStreamId)) + { + return false; + } + + return string.Equals(latestProofSurface.MuninnEvidenceFrameId, evidenceStreamId, StringComparison.Ordinal) || + latestProofSurface.MuninnEvidenceFrameId.StartsWith($"{evidenceStreamId}:", StringComparison.Ordinal); + } +} public sealed class MimirUnavailableMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider { diff --git a/state/evidence.jsonl b/state/evidence.jsonl index d6daf69..1e1ddfc 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -166,3 +166,4 @@ {"ts": "2026-07-08T22:55:54+01:00", "type": "verification", "status": "ok", "note": "Added and verified the Fensalir presented-frame proof smoke. MIMIR_MOVE_PROOF_DEV_SURFACE gates a synthetic calibrated proof surface through the same MimirRuntime.PublishMoveProofSurface path, and BufferSmoke --move-proof-presented-frame-smoke runs Mimir.App headless, captures artifacts/runtime/move-proof-presented-frame-smoke.png, and pixel-checks the rendered proof probe. Latest run captured fensalir:starfire:presented-frame:79 for proof mimir:starfire:move-proof:79 with bytes=11260 and brightPixels=1069; adjacent runtime-driver, runtime-frame, proof-surface, and proof-pipeline smokes pass. This proves the renderer/probe layer for the named proof chain, not live Nightwing hardware or calibrated field evidence."} {"ts": "2026-07-12T10:26:42+01:00", "type": "verification", "status": "ok", "note": "Added MimirMoveProofRuntimeConfiguration as the typed config/validation surface for Move proof driver creation. The runtime config now accepts enabled MoveProof sources, validates named Muninn evidence stream, native reservoir path, Mimir/Fensalir frame prefixes, fusion authority, consumer contract, and at least two calibrated camera witnesses, and creates a driver only when the supplied CultMesh ring stream id matches. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-config-smoke, --move-proof-runtime-driver-smoke, --move-proof-pipeline-smoke, and --move-proof-presented-frame-smoke. The config smoke printed configValid=True invalidDefaultErrors=2 and the same sequence chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79. This proves config plus in-process ring flow, not live Nightwing hardware, calibrated field cameras, or cross-process ring opening."} {"ts": "2026-07-12T10:51:37+01:00", "type": "verification", "status": "ok", "note": "MimirRuntime now owns configured Move proof activation instead of leaving MoveProof config as a passive readout. On scene-ready it asks IMimirMoveProofEvidenceRingProvider for the configured Muninn evidence ring, opens the configured native reservoir, creates/registers MimirMoveProofRuntimeDriver, retains resources, and records activation status. The default provider fails explicitly because the current C# CultMesh ring is in-process only. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, and BufferSmoke --move-proof-runtime-activation-smoke. The activation smoke injected an in-process provider and printed activationActive=True while preserving the explicit chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79 with verdict=FullPose. This proves runtime config activation and status, not live Nightwing/Starfire hardware or cross-process ring opening."} +{"ts": "2026-07-12T11:01:09+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_runtime_activation.v1 as the typed readiness/proof-spine surface for configured Move proof sources. It reports stream id, provider kind, active/driver state, reservoir path, calibrated camera ids, latest same-stream proof ids/verdict, and unavailable-provider diagnostics. MimirRuntime now exposes CreateMoveProofRuntimeActivationDocuments and disposes a native reservoir if activation fails before driver retention. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-smoke, and --move-proof-runtime-activation-surface-smoke against the native reservoir debug DLL. The new smoke proves active injected-ring and inactive default-provider documents; real Nightwing/Starfire hardware and cross-process CultMesh ring opening remain pending."} diff --git a/state/map.yaml b/state/map.yaml index 94b14bc..6bed51a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -170,6 +170,13 @@ accepted_design: `Mimir.BufferSmoke --move-proof-runtime-activation-smoke` injects an in-process provider to prove the activation path without claiming live Nightwing/Starfire field evidence. + - `mimir.move_proof_runtime_activation.v1` is the typed readiness/proof-spine + surface for configured Move proof sources. It reports the configured stream, + provider kind, active/driver state, native reservoir path, calibrated camera + ids, latest same-stream proof ids, latest verdict, and unavailable-provider + diagnostic. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke` + proves both an active injected-ring document and the inactive default + runtime document without claiming live field evidence. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same MimirRuntime attachment so headless Mimir.App can From 2de98df8040fa4034f07f309e31c76bd82069e88 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 11:01:39 +0100 Subject: [PATCH 075/135] Persist Mimir Face state --- .voidbot/state/mimir.cc | Bin 108447 -> 108360 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 5c2f7129330229e9f217fab136845971f8e41355..c8488f2e18b6178d676773adb2a21ddaf8ebd1c0 100644 GIT binary patch delta 217 zcmbP#p6$drwhiBSxeX1h3=FJHO!N#)Cr@}TK3RgVg9jmIW;A)?bFs~b_$+LYM9mGr zqB7p|f{;bcL86oI%vEH}ocwRDG*9!&%woU9f`ZJv^h1+t=DlRUvxdV^?PcZUocXRu u`b^CxKj0NZ60k4^8~=6w1a~A+OOR8hAOs8zOg0<7NZij0cdenJaTEX%2}eu- delta 297 zcmX?cj&1&VwhiBSc?~SBOpUD!jP#6+EutneicXf`>p&7Tfe6l<&$jslpOFn0Lir>< ziOn+J^Ma5Ro0))3xHDIg@xbJNbETzQS7sLbB^DH9=A|FroLpK|l$uup@^C!4-Ba~T8X5 Date: Sun, 12 Jul 2026 11:10:17 +0100 Subject: [PATCH 076/135] Add Move proof evidence snapshot fallback --- docs/implementation-plan.md | 9 ++ notes/current-system-map.md | 10 ++ src/Mimir.BufferSmoke/Program.cs | 55 +++++++- src/Mimir.Runtime/MimirRuntime.cs | 2 +- .../MimirMoveProofRuntimeActivation.cs | 129 ++++++++++++++++++ .../MimirMoveProofRuntimeConfiguration.cs | 2 + state/evidence.jsonl | 1 + state/map.yaml | 9 ++ 8 files changed, 209 insertions(+), 8 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 887abcc..7be773f 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -129,6 +129,15 @@ a named invariant that the native runtime cannot protect yet. --move-proof-runtime-activation-surface-smoke` proves both an active injected ring document and the inactive default runtime document without claiming live field evidence. +- `mimir.move_proof_evidence_frame_snapshot.v1` is the one-copy fallback for + captured Muninn evidence frames while C# CultMesh has no cross-process ring + opener. A configured `EvidenceSnapshotPath` is loaded by + `MimirConfiguredMoveProofEvidenceRingProvider`, metadata-checked against the + encoded payload, copied into an owned in-process ring, and then consumed by + the same runtime driver/pipeline. `Mimir.BufferSmoke + --move-proof-runtime-snapshot-smoke` proves the named proof spine through + that file boundary. It is field capture/replay support, not the final live + shared-memory proof. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so `Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index ef4bb67..93301bc 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -247,6 +247,16 @@ diagnostic when the default runtime lacks a live cross-process CultMesh ring opener. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke` proves both the active injected-ring document and the unavailable default document; this is readiness/proof observation, not field evidence. +`mimir.move_proof_evidence_frame_snapshot.v1` is a one-copy, explicitly +non-final fallback for captured Muninn evidence frames while C# CultMesh lacks +a cross-process shared-memory opener. When `EvidenceSnapshotPath` is +configured, `MimirConfiguredMoveProofEvidenceRingProvider` validates snapshot +metadata against the encoded Muninn frame payload, publishes the payload into +an owned in-process ring, and then delegates to the same runtime proof driver +and pipeline. `Mimir.BufferSmoke --move-proof-runtime-snapshot-smoke` proves +the same named proof spine through that file boundary. This can support field +capture/replay, but live proof still requires the real Nightwing/Starfire ring +or CultMesh page producer. `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index ea00f0f..ac207a7 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -182,7 +182,17 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-activation-surface- return RunMoveProofRuntimeDriverSmoke( ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), activateViaConfiguredSource: true, - requireActivationSurface: true); + requireActivationSurface: true, + activateViaSnapshotProvider: false); +} + +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-snapshot-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeDriverSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), + activateViaConfiguredSource: true, + requireActivationSurface: true, + activateViaSnapshotProvider: true); } if (args.Any(arg => string.Equals(arg, "--move-proof-presented-frame-smoke", StringComparison.OrdinalIgnoreCase))) @@ -2297,7 +2307,8 @@ static int RunMoveProofRuntimeFrameSmoke() static int RunMoveProofRuntimeDriverSmoke( string nativeReservoirPath, bool activateViaConfiguredSource, - bool requireActivationSurface = false) + bool requireActivationSurface = false, + bool activateViaSnapshotProvider = false) { if (!File.Exists(nativeReservoirPath)) { @@ -2309,6 +2320,9 @@ static int RunMoveProofRuntimeDriverSmoke( const ulong rightWitnessHash = 0x0000000000000B0B; const string observedAt = "2026-06-11T18:30:00.0040000Z"; var streamId = "muninn:nightwing:move-evidence"; + var snapshotPath = activateViaSnapshotProvider + ? Path.Combine(Path.GetTempPath(), $"mimir-move-proof-evidence-{Guid.NewGuid():N}.mpack") + : ""; var frame = new MimirMuninnMoveEvidenceStreamFrame( FrameId: "muninn:nightwing:move-evidence:79", ProducerPeerId: "muninn:nightwing", @@ -2367,6 +2381,7 @@ static int RunMoveProofRuntimeDriverSmoke( var configuration = new MimirMoveProofRuntimeConfiguration { EvidenceStreamId = streamId, + EvidenceSnapshotPath = snapshotPath, NativeReservoirPath = nativeReservoirPath, MimirEvidenceSourceId = "mimir:starfire:move-evidence", MimirEvidenceFramePrefix = "mimir:starfire:move-evidence", @@ -2422,6 +2437,16 @@ static int RunMoveProofRuntimeDriverSmoke( "nightwing:clock")); using var ring = catalog.CreateSharedMemoryRing(streamId, slotCount: 4, slotByteLength: 8192); var payload = MimirMuninnMoveEvidenceAdapter.SerializeStreamFrame(frame); + if (activateViaSnapshotProvider) + { + MimirMoveProofEvidenceFrameSnapshot.Save( + snapshotPath, + MimirMoveProofEvidenceFrameSnapshot.Create( + frame, + payload, + capturedAtNs: 1_781_202_600_005_500_000)); + } + if (!ring.TryPublishCopy(payload, frame.PublishedAtNs, durationNs: 0, out var published)) { Console.Error.WriteLine("move-proof-runtime-driver-smoke publish-failed"); @@ -2434,7 +2459,15 @@ static int RunMoveProofRuntimeDriverSmoke( ? null : configuration.CreateDriver(ring, manualReservoir!); using var runtime = activateViaConfiguredSource - ? new MimirRuntime( + ? activateViaSnapshotProvider + ? new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), + new MimirRuntimeConfiguration + { + Settings = new MimirSynchronizationSettings(), + MoveProofSources = [configuration], + }) + : new MimirRuntime( new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), new MimirRuntimeConfiguration { @@ -2478,10 +2511,12 @@ static int RunMoveProofRuntimeDriverSmoke( LatestFensalirFrameId: "fensalir:starfire:presented-frame:79", LatestVerdict: MimirMoveProofVerdict.FullPose, CalibratedCameraCount: 2 - }; + } && + (!activateViaSnapshotProvider || + activationDocument.Diagnostic.Contains("one-copy evidence snapshot supplied", StringComparison.Ordinal)); MimirMoveProofRuntimeActivationDocument? unavailableDocument = null; - if (requireActivationSurface) + if (requireActivationSurface && !activateViaSnapshotProvider) { using var unavailableRuntime = new MimirRuntime( new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), @@ -2505,9 +2540,9 @@ static int RunMoveProofRuntimeDriverSmoke( } Console.WriteLine( - $"move-proof-runtime-{(requireActivationSurface ? "activation-surface" : activateViaConfiguredSource ? "activation" : "driver")}-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length} activationActive={activationStatus?.Active.ToString() ?? "n/a"} activationDiagnostic=\"{activationStatus?.Diagnostic ?? ""}\" activationDocumentActive={activationDocument?.Active.ToString() ?? "n/a"} activationProvider=\"{activationDocument?.ProviderKind ?? ""}\" unavailableActive={unavailableDocument?.Active.ToString() ?? "n/a"} unavailableDiagnostic=\"{unavailableDocument?.Diagnostic ?? ""}\""); + $"move-proof-runtime-{(activateViaSnapshotProvider ? "snapshot" : requireActivationSurface ? "activation-surface" : activateViaConfiguredSource ? "activation" : "driver")}-smoke proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} splines={proofSplineCount} duplicateSuppressed={duplicateSuppressed} configValid={configErrors.Length == 0} invalidDefaultErrors={invalidErrors.Length} activationActive={activationStatus?.Active.ToString() ?? "n/a"} activationDiagnostic=\"{activationStatus?.Diagnostic ?? ""}\" activationDocumentActive={activationDocument?.Active.ToString() ?? "n/a"} activationProvider=\"{activationDocument?.ProviderKind ?? ""}\" unavailableActive={unavailableDocument?.Active.ToString() ?? "n/a"} unavailableDiagnostic=\"{unavailableDocument?.Diagnostic ?? ""}\" snapshot=\"{snapshotPath}\""); - return proofSurface is not null && + var result = proofSurface is not null && activationOk && activationSurfaceOk && proofSurface.IsFullPose && @@ -2520,6 +2555,12 @@ static int RunMoveProofRuntimeDriverSmoke( duplicateSuppressed ? 0 : 1; + if (!string.IsNullOrWhiteSpace(snapshotPath) && File.Exists(snapshotPath)) + { + File.Delete(snapshotPath); + } + + return result; } static int RunMoveProofPresentedFrameSmoke(string outputPath) diff --git a/src/Mimir.Runtime/MimirRuntime.cs b/src/Mimir.Runtime/MimirRuntime.cs index f7fcff8..1500145 100644 --- a/src/Mimir.Runtime/MimirRuntime.cs +++ b/src/Mimir.Runtime/MimirRuntime.cs @@ -81,7 +81,7 @@ direct configuration.Settings, configuration.SourceFactories, configuration.MoveProofSources, - MimirUnavailableMoveProofEvidenceRingProvider.Instance) + MimirConfiguredMoveProofEvidenceRingProvider.Instance) { } diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs index 6f99183..73d9bc5 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs @@ -33,6 +33,69 @@ public sealed class MimirMoveProofEvidenceRingLease : IDisposable } } +[CultDocument("mimir.move_proof_evidence_frame_snapshot", "mimir.move_proof_evidence_frame_snapshot.v1")] +[MessagePackObject] +public sealed record MimirMoveProofEvidenceFrameSnapshotDocument( + [property: Key(0)] + [property: CultName] + string SnapshotId, + [property: Key(1)] string EvidenceStreamId, + [property: Key(2)] string FrameId, + [property: Key(3)] string ProducerPeerId, + [property: Key(4)] long PublishedAtNs, + [property: Key(5)] ulong CapturedAtNs, + [property: Key(6)] byte[] Payload); + +public static class MimirMoveProofEvidenceFrameSnapshot +{ + public static MimirMoveProofEvidenceFrameSnapshotDocument Create( + MimirMuninnMoveEvidenceStreamFrame frame, + byte[] payload, + ulong capturedAtNs) + { + ArgumentNullException.ThrowIfNull(frame); + ArgumentNullException.ThrowIfNull(payload); + if (payload.Length == 0) + { + throw new ArgumentException("Snapshot payload must not be empty.", nameof(payload)); + } + + return new MimirMoveProofEvidenceFrameSnapshotDocument( + SnapshotId: $"{frame.FrameId}:snapshot", + EvidenceStreamId: StreamIdFromFrameId(frame.FrameId), + FrameId: frame.FrameId, + ProducerPeerId: frame.ProducerPeerId, + PublishedAtNs: frame.PublishedAtNs, + CapturedAtNs: capturedAtNs, + Payload: payload); + } + + public static MimirMoveProofEvidenceFrameSnapshotDocument Load(string path) + { + ArgumentException.ThrowIfNullOrWhiteSpace(path); + return MessagePackSerializer.Deserialize(File.ReadAllBytes(path)); + } + + public static void Save(string path, MimirMoveProofEvidenceFrameSnapshotDocument snapshot) + { + ArgumentException.ThrowIfNullOrWhiteSpace(path); + ArgumentNullException.ThrowIfNull(snapshot); + var directory = Path.GetDirectoryName(Path.GetFullPath(path)); + if (!string.IsNullOrWhiteSpace(directory)) + { + Directory.CreateDirectory(directory); + } + + File.WriteAllBytes(path, MessagePackSerializer.Serialize(snapshot)); + } + + private static string StreamIdFromFrameId(string frameId) + { + var separator = frameId.LastIndexOf(':'); + return separator > 0 ? frameId[..separator] : frameId; + } +} + public sealed record MimirMoveProofRuntimeActivationStatus( string EvidenceStreamId, bool Active, @@ -115,6 +178,72 @@ public static class MimirMoveProofRuntimeActivation } } +public sealed class MimirConfiguredMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider +{ + public static MimirConfiguredMoveProofEvidenceRingProvider Instance { get; } = new(); + + private MimirConfiguredMoveProofEvidenceRingProvider() + { + } + + public bool TryOpenEvidenceRing( + MimirMoveProofRuntimeConfiguration configuration, + out MimirMoveProofEvidenceRingLease? lease, + out string diagnostic) + { + ArgumentNullException.ThrowIfNull(configuration); + if (string.IsNullOrWhiteSpace(configuration.EvidenceSnapshotPath)) + { + return MimirUnavailableMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing( + configuration, + out lease, + out diagnostic); + } + + var snapshotPath = Path.GetFullPath(configuration.EvidenceSnapshotPath); + if (!File.Exists(snapshotPath)) + { + lease = null; + diagnostic = $"Move proof evidence snapshot not found: {snapshotPath}"; + return false; + } + + var snapshot = MimirMoveProofEvidenceFrameSnapshot.Load(snapshotPath); + if (!string.Equals(snapshot.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)) + { + lease = null; + diagnostic = $"Move proof evidence snapshot stream '{snapshot.EvidenceStreamId}' does not match configured evidence stream '{configuration.EvidenceStreamId}'."; + return false; + } + + var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(snapshot.Payload); + if (!string.Equals(decoded.FrameId, snapshot.FrameId, StringComparison.Ordinal) || + !string.Equals(decoded.ProducerPeerId, snapshot.ProducerPeerId, StringComparison.Ordinal) || + decoded.PublishedAtNs != snapshot.PublishedAtNs) + { + lease = null; + diagnostic = "Move proof evidence snapshot metadata does not match its encoded Muninn frame payload."; + return false; + } + + var ring = new CultMeshSharedMemoryFrameRing( + configuration.EvidenceStreamId, + slotCount: 1, + slotByteLength: snapshot.Payload.Length); + if (!ring.TryPublishCopy(snapshot.Payload, snapshot.PublishedAtNs, durationNs: 0, out _)) + { + ring.Dispose(); + lease = null; + diagnostic = $"Move proof evidence snapshot could not be published into fallback ring: {snapshotPath}"; + return false; + } + + lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true); + diagnostic = $"one-copy evidence snapshot supplied from {snapshotPath}; this is not a live cross-process CultMesh ring"; + return true; + } +} + public sealed class MimirUnavailableMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider { public static MimirUnavailableMoveProofEvidenceRingProvider Instance { get; } = new(); diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs index f61dd0e..6971a2a 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs @@ -8,6 +8,8 @@ public sealed class MimirMoveProofRuntimeConfiguration public string EvidenceStreamId { get; init; } = "muninn:nightwing:move-evidence"; + public string EvidenceSnapshotPath { get; init; } = ""; + public string NativeReservoirPath { get; init; } = ""; public string MimirEvidenceSourceId { get; init; } = "mimir:starfire:move-evidence"; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 1e1ddfc..00e17ab 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -167,3 +167,4 @@ {"ts": "2026-07-12T10:26:42+01:00", "type": "verification", "status": "ok", "note": "Added MimirMoveProofRuntimeConfiguration as the typed config/validation surface for Move proof driver creation. The runtime config now accepts enabled MoveProof sources, validates named Muninn evidence stream, native reservoir path, Mimir/Fensalir frame prefixes, fusion authority, consumer contract, and at least two calibrated camera witnesses, and creates a driver only when the supplied CultMesh ring stream id matches. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-config-smoke, --move-proof-runtime-driver-smoke, --move-proof-pipeline-smoke, and --move-proof-presented-frame-smoke. The config smoke printed configValid=True invalidDefaultErrors=2 and the same sequence chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79. This proves config plus in-process ring flow, not live Nightwing hardware, calibrated field cameras, or cross-process ring opening."} {"ts": "2026-07-12T10:51:37+01:00", "type": "verification", "status": "ok", "note": "MimirRuntime now owns configured Move proof activation instead of leaving MoveProof config as a passive readout. On scene-ready it asks IMimirMoveProofEvidenceRingProvider for the configured Muninn evidence ring, opens the configured native reservoir, creates/registers MimirMoveProofRuntimeDriver, retains resources, and records activation status. The default provider fails explicitly because the current C# CultMesh ring is in-process only. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, and BufferSmoke --move-proof-runtime-activation-smoke. The activation smoke injected an in-process provider and printed activationActive=True while preserving the explicit chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79 with verdict=FullPose. This proves runtime config activation and status, not live Nightwing/Starfire hardware or cross-process ring opening."} {"ts": "2026-07-12T11:01:09+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_runtime_activation.v1 as the typed readiness/proof-spine surface for configured Move proof sources. It reports stream id, provider kind, active/driver state, reservoir path, calibrated camera ids, latest same-stream proof ids/verdict, and unavailable-provider diagnostics. MimirRuntime now exposes CreateMoveProofRuntimeActivationDocuments and disposes a native reservoir if activation fails before driver retention. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-smoke, and --move-proof-runtime-activation-surface-smoke against the native reservoir debug DLL. The new smoke proves active injected-ring and inactive default-provider documents; real Nightwing/Starfire hardware and cross-process CultMesh ring opening remain pending."} +{"ts": "2026-07-12T11:09:32+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_evidence_frame_snapshot.v1 and MimirConfiguredMoveProofEvidenceRingProvider as an explicit one-copy fallback for captured Muninn Move evidence frames while C# CultMesh lacks a cross-process shared-memory opener. EvidenceSnapshotPath snapshots are metadata-checked against the encoded Muninn frame payload, copied into an owned in-process ring, and consumed by the same runtime proof driver/pipeline. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-surface-smoke, and --move-proof-runtime-snapshot-smoke against the native reservoir debug DLL. This supports field capture/replay and does not complete the live Nightwing/Starfire ring proof."} diff --git a/state/map.yaml b/state/map.yaml index 6bed51a..0123841 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -177,6 +177,15 @@ accepted_design: diagnostic. `Mimir.BufferSmoke --move-proof-runtime-activation-surface-smoke` proves both an active injected-ring document and the inactive default runtime document without claiming live field evidence. + - `mimir.move_proof_evidence_frame_snapshot.v1` is the one-copy fallback for + captured Muninn evidence frames while C# CultMesh has no cross-process ring + opener. A configured `EvidenceSnapshotPath` is loaded by + `MimirConfiguredMoveProofEvidenceRingProvider`, metadata-checked against the + encoded payload, copied into an owned in-process ring, and then consumed by + the same runtime driver/pipeline. `Mimir.BufferSmoke + --move-proof-runtime-snapshot-smoke` proves the named proof spine through + that file boundary. It is field capture/replay support, not the final live + shared-memory proof. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same MimirRuntime attachment so headless Mimir.App can From c0e53135baa3cc199579ccac5c40e1296e7d6fd0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 11:10:21 +0100 Subject: [PATCH 077/135] Persist Mimir Face state --- .voidbot/state/mimir.cc | Bin 108360 -> 108360 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index c8488f2e18b6178d676773adb2a21ddaf8ebd1c0..93898c9d85820211d6109b1375ba3bb5fd385c05 100644 GIT binary patch delta 181 zcmX?cj_t%bwhcddxeN`g42<;5O)R1&GKx-?;OmBon3{q`=FMl@e1gx^1}1D~0utUV zGdZ=FMl@e1gx^1}1E7021CT z Date: Sun, 12 Jul 2026 11:17:21 +0100 Subject: [PATCH 078/135] Record Muninn Move proof snapshot producer --- docs/implementation-plan.md | 9 +++++++++ notes/current-system-map.md | 8 ++++++++ state/evidence.jsonl | 1 + state/map.yaml | 8 ++++++++ 4 files changed, 26 insertions(+) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 7be773f..3fdfb31 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -138,6 +138,15 @@ a named invariant that the native runtime cannot protect yet. --move-proof-runtime-snapshot-smoke` proves the named proof spine through that file boundary. It is field capture/replay support, not the final live shared-memory proof. +- Odin/Muninn now has the matching producer-side latest snapshot writer: + `--move-evidence-snapshot ` writes the Mimir-compatible + `mimir.move_proof_evidence_frame_snapshot.v1` artifact from the same Muninn + Move evidence publisher after the frame payload is accepted into the stream + ring. The Odin unit + `move_evidence_snapshot_writes_mimir_compatible_frame_artifact` decodes the + snapshot tuple and embedded Muninn frame payload. This lets Nightwing produce + a field artifact for Mimir replay while the final live ring/page transport is + still pending. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It uses the same runtime proof attachment so `Mimir.BufferSmoke --move-proof-presented-frame-smoke` can run `Mimir.App` diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 93301bc..3ec535d 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -257,6 +257,14 @@ and pipeline. `Mimir.BufferSmoke --move-proof-runtime-snapshot-smoke` proves the same named proof spine through that file boundary. This can support field capture/replay, but live proof still requires the real Nightwing/Starfire ring or CultMesh page producer. +Odin/Muninn can now write that artifact from the producer side with +`--move-evidence-snapshot `. The snapshot is written by the same Muninn +Move evidence publisher that owns controller/marker frames, after the frame +payload is accepted by the stream ring. Its unit coverage decodes the +`mimir.move_proof_evidence_frame_snapshot.v1` tuple and re-decodes the embedded +Muninn payload, proving the artifact is shaped for Mimir's snapshot provider. +This is still one-copy field capture/replay; it does not replace the live ring +or a CultCache page body producer. `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same `MimirRuntime.PublishMoveProofSurface` attachment so diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 00e17ab..d6b7075 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -168,3 +168,4 @@ {"ts": "2026-07-12T10:51:37+01:00", "type": "verification", "status": "ok", "note": "MimirRuntime now owns configured Move proof activation instead of leaving MoveProof config as a passive readout. On scene-ready it asks IMimirMoveProofEvidenceRingProvider for the configured Muninn evidence ring, opens the configured native reservoir, creates/registers MimirMoveProofRuntimeDriver, retains resources, and records activation status. The default provider fails explicitly because the current C# CultMesh ring is in-process only. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, and BufferSmoke --move-proof-runtime-activation-smoke. The activation smoke injected an in-process provider and printed activationActive=True while preserving the explicit chain muninn:nightwing:move-evidence:79 -> mimir:starfire:move-evidence:79 -> mimir:starfire:move-pose:79 -> fensalir:starfire:presented-frame:79 with verdict=FullPose. This proves runtime config activation and status, not live Nightwing/Starfire hardware or cross-process ring opening."} {"ts": "2026-07-12T11:01:09+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_runtime_activation.v1 as the typed readiness/proof-spine surface for configured Move proof sources. It reports stream id, provider kind, active/driver state, reservoir path, calibrated camera ids, latest same-stream proof ids/verdict, and unavailable-provider diagnostics. MimirRuntime now exposes CreateMoveProofRuntimeActivationDocuments and disposes a native reservoir if activation fails before driver retention. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-smoke, and --move-proof-runtime-activation-surface-smoke against the native reservoir debug DLL. The new smoke proves active injected-ring and inactive default-provider documents; real Nightwing/Starfire hardware and cross-process CultMesh ring opening remain pending."} {"ts": "2026-07-12T11:09:32+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_evidence_frame_snapshot.v1 and MimirConfiguredMoveProofEvidenceRingProvider as an explicit one-copy fallback for captured Muninn Move evidence frames while C# CultMesh lacks a cross-process shared-memory opener. EvidenceSnapshotPath snapshots are metadata-checked against the encoded Muninn frame payload, copied into an owned in-process ring, and consumed by the same runtime proof driver/pipeline. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-surface-smoke, and --move-proof-runtime-snapshot-smoke against the native reservoir debug DLL. This supports field capture/replay and does not complete the live Nightwing/Starfire ring proof."} +{"ts": "2026-07-12T11:16:48+01:00", "type": "implementation", "status": "ok", "note": "Odin/Muninn now has a producer-side --move-evidence-snapshot writer for Mimir Move proof capture/replay. The same Muninn Move evidence publisher that writes controller/marker frames to the CultMesh stream ring can write a mimir.move_proof_evidence_frame_snapshot.v1 MessagePack tuple containing snapshot id, evidence stream id, frame id, producer peer id, published ns, captured ns, and the encoded Muninn frame payload. Verified cargo test -p muninn-daemon, focused move_evidence_snapshot_writes_mimir_compatible_frame_artifact, and cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu (existing dead-code warnings only). This gives Nightwing a concrete field artifact path for Mimir's EvidenceSnapshotPath provider; it is still one-copy replay support, not the final live cross-process ring/page proof."} diff --git a/state/map.yaml b/state/map.yaml index 0123841..075e051 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -186,6 +186,14 @@ accepted_design: --move-proof-runtime-snapshot-smoke` proves the named proof spine through that file boundary. It is field capture/replay support, not the final live shared-memory proof. + - Odin/Muninn can write the matching producer-side latest snapshot with + `--move-evidence-snapshot `. The snapshot is emitted by the same + Muninn Move evidence publisher after the frame payload is accepted into the + stream ring, and Odin's + `move_evidence_snapshot_writes_mimir_compatible_frame_artifact` unit decodes + both the `mimir.move_proof_evidence_frame_snapshot.v1` tuple and embedded + Muninn payload. This supports Nightwing field capture/replay through Mimir's + snapshot provider while the final live ring/page transport remains pending. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same MimirRuntime attachment so headless Mimir.App can From de22059725afd215d7a829b93f0b7f975429b813 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 11:17:25 +0100 Subject: [PATCH 079/135] Persist Mimir Face state --- .voidbot/state/mimir.cc | Bin 108360 -> 108360 bytes .voidbot/status/void-memory-maintenance.json | 4 ++-- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 93898c9d85820211d6109b1375ba3bb5fd385c05..dfc536297bb8ef5b41a2fd112eeaa88dcee773f9 100644 GIT binary patch delta 172 zcmX?cj_t%bwhg~{InAt04fRY6&7vkUicXf`>wyXw8G!}n&1c(ug3rtbDr#&165TB0 zy(kDK0@k?s&Rio#_FMfi8eb3fPZpi;$pJHV@&jHes8SP%(s}czxI;xuO~94}Ja2*u Sn1OBG{NcI(eyE7KK@wyZGnt}!9&1c(ug3rtbDr#l|65TB0 zy(kDKVhI+xGuMcb{dP)}%o+~E$)fW;Ibg<4e!wdQRca2=Ja7IKcc_Sk5!jM|=S?sH RbFgV2p8M~IiddRP0RU=*Gb8{2 diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index db88ea1..f36baa2 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,7 +1,7 @@ { "status": "starting", - "startedAt": "2026-07-12T09:49:47.786Z", + "startedAt": "2026-07-12T10:15:19.747Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", - "reason": "repo_face_napping_with_short_term_memory" + "reason": "repo_face_stale_short_term_memory" } From 538b95e2c341d234a8bdeadacbf664bc74a0de73 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 20:18:30 +0100 Subject: [PATCH 080/135] Discover Muninn Move evidence through Odin --- src/Mimir.BufferSmoke/Program.cs | 268 +++++++++++++++++- src/Mimir.Runtime/Mimir.Runtime.csproj | 1 + .../MimirMoveProofRuntimeActivation.cs | 24 +- .../MimirMoveProofRuntimeConfiguration.cs | 8 + .../MimirOdinMoveProofEvidenceRingProvider.cs | 216 ++++++++++++++ 5 files changed, 508 insertions(+), 9 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index ac207a7..b21165a 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -134,6 +134,12 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-cultmesh-stream-smoke", St ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } +if (args.Any(arg => string.Equals(arg, "--muninn-move-odin-live-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMuninnMoveOdinLiveSmoke( + ParseStringOption(args, "--odin-cultmesh-uri", "cultmesh://127.0.0.1:17871/rendezvous/provider-catalog")); +} + if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveFusionSmoke(); @@ -192,7 +198,16 @@ if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-snapshot-smoke", St ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), activateViaConfiguredSource: true, requireActivationSurface: true, - activateViaSnapshotProvider: true); + activateViaSnapshotProvider: true, + snapshotOutputPath: ParseStringOption(args, "--snapshot-output", "")); +} + +if (args.Any(arg => string.Equals(arg, "--move-proof-runtime-snapshot-file-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveProofRuntimeSnapshotFileSmoke( + ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath()), + ParseStringOption(args, "--snapshot", ""), + args.Any(arg => string.Equals(arg, "--require-full-pose", StringComparison.OrdinalIgnoreCase))); } if (args.Any(arg => string.Equals(arg, "--move-proof-presented-frame-smoke", StringComparison.OrdinalIgnoreCase))) @@ -1909,6 +1924,58 @@ static async Task RunMuninnMoveLiveRosterSmokeAsync(IReadOnlyList i : 1; } +static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri) +{ + const string providerId = "muninn.telemetry.nightwing"; + const string streamId = "muninn:nightwing:move-evidence"; + var discovered = MimirOdinMoveProofEvidenceRingProvider.Discover(odinCultMeshUri, providerId, streamId); + var configuration = new MimirMoveProofRuntimeConfiguration + { + OdinCultMeshUri = odinCultMeshUri, + MuninnProviderId = providerId, + EvidenceStreamId = streamId + }; + if (!MimirOdinMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing( + configuration, + out var lease, + out var diagnostic) || lease is null) + { + throw new InvalidOperationException(diagnostic); + } + + using (lease) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(8); + while (DateTime.UtcNow < deadline) + { + if (lease.Ring.TryAcquireLatestRead(out var frameLease)) + { + using (frameLease) + { + var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame( + frameLease.Memory[..frameLease.Handle.ByteLength]); + var ok = frame.FrameId.StartsWith($"{streamId}:", StringComparison.Ordinal) && + frame.ControllerStates.Length >= 1 && + discovered.Channel == "move-evidence"; + Console.WriteLine(JsonSerializer.Serialize(new + { + ok, + diagnostic, + discovered, + frame.FrameId, + frame.ProducerPeerId, + controllers = frame.ControllerStates.Select(state => state.MoveId).Distinct().Order().ToArray(), + markers = frame.MarkerCandidates.Length + })); + return ok ? 0 : 1; + } + } + Thread.Sleep(20); + } + } + throw new TimeoutException($"No live frame arrived from Odin-discovered stream {streamId}."); +} + static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) { if (!File.Exists(nativeReservoirPath)) @@ -2308,7 +2375,8 @@ static int RunMoveProofRuntimeDriverSmoke( string nativeReservoirPath, bool activateViaConfiguredSource, bool requireActivationSurface = false, - bool activateViaSnapshotProvider = false) + bool activateViaSnapshotProvider = false, + string snapshotOutputPath = "") { if (!File.Exists(nativeReservoirPath)) { @@ -2320,8 +2388,11 @@ static int RunMoveProofRuntimeDriverSmoke( const ulong rightWitnessHash = 0x0000000000000B0B; const string observedAt = "2026-06-11T18:30:00.0040000Z"; var streamId = "muninn:nightwing:move-evidence"; + var deleteSnapshotOnExit = activateViaSnapshotProvider && string.IsNullOrWhiteSpace(snapshotOutputPath); var snapshotPath = activateViaSnapshotProvider - ? Path.Combine(Path.GetTempPath(), $"mimir-move-proof-evidence-{Guid.NewGuid():N}.mpack") + ? deleteSnapshotOnExit + ? Path.Combine(Path.GetTempPath(), $"mimir-move-proof-evidence-{Guid.NewGuid():N}.mpack") + : Path.GetFullPath(snapshotOutputPath) : ""; var frame = new MimirMuninnMoveEvidenceStreamFrame( FrameId: "muninn:nightwing:move-evidence:79", @@ -2555,7 +2626,7 @@ static int RunMoveProofRuntimeDriverSmoke( duplicateSuppressed ? 0 : 1; - if (!string.IsNullOrWhiteSpace(snapshotPath) && File.Exists(snapshotPath)) + if (deleteSnapshotOnExit && File.Exists(snapshotPath)) { File.Delete(snapshotPath); } @@ -2563,6 +2634,195 @@ static int RunMoveProofRuntimeDriverSmoke( return result; } +static int RunMoveProofRuntimeSnapshotFileSmoke( + string nativeReservoirPath, + string snapshotPath, + bool requireFullPose) +{ + if (!File.Exists(nativeReservoirPath)) + { + Console.Error.WriteLine($"move-proof-runtime-snapshot-file-smoke missing-native path={nativeReservoirPath}"); + return 1; + } + + if (string.IsNullOrWhiteSpace(snapshotPath) || !File.Exists(snapshotPath)) + { + Console.Error.WriteLine($"move-proof-runtime-snapshot-file-smoke missing-snapshot path={snapshotPath}"); + return 1; + } + + var absoluteSnapshotPath = Path.GetFullPath(snapshotPath); + MimirMoveProofEvidenceFrameSnapshotDocument snapshot; + MimirMuninnMoveEvidenceStreamFrame decoded; + try + { + snapshot = MimirMoveProofEvidenceFrameSnapshot.Load(absoluteSnapshotPath); + decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(snapshot.Payload); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or MessagePackSerializationException) + { + Console.Error.WriteLine($"move-proof-runtime-snapshot-file-smoke snapshot-decode-failed path={absoluteSnapshotPath} error={ex.Message}"); + return 1; + } + + var suffix = SequenceSuffix(decoded.FrameId); + var decodedSamples = MimirMuninnMoveEvidenceAdapter.BuildNativeSamples( + decoded.MarkerCandidates, + decoded.ControllerStates); + var markerSamples = decodedSamples + .Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.OpticalMarker) + .ToArray(); + var controllerSamples = decodedSamples + .Where(sample => sample.EvidenceKind == (uint)MimirNativeMoveEvidenceKind.ControllerState) + .ToArray(); + var distinctMarkerWitnesses = markerSamples + .Select(sample => sample.WitnessIdHash) + .Where(hash => hash != 0UL) + .Distinct() + .Count(); + var markerWitnessSummary = string.Join( + ",", + decoded.MarkerCandidates + .GroupBy(marker => marker.SourceIdHash) + .OrderBy(group => group.Key) + .Select(group => + $"0x{group.Key:X16}:{group.Count()}@peak{group.Max(marker => marker.PeakLuma)}:area{group.Max(marker => marker.AreaPx)}:radius{group.Max(marker => marker.RadiusPx):0.0}:score{group.Max(marker => marker.Score):0.000}")); + var nearestControllerSkewMs = markerSamples.Length == 0 || controllerSamples.Length == 0 + ? double.NaN + : markerSamples + .Select(marker => controllerSamples + .Min(controller => TimestampDelta(marker.SourceTimestampNs, controller.SourceTimestampNs))) + .Min() / 1_000_000.0; + var configuration = CreateMoveProofConfigurationForSnapshot( + nativeReservoirPath, + absoluteSnapshotPath, + snapshot, + decoded); + var configErrors = configuration.Validate(); + if (configErrors.Length > 0) + { + Console.Error.WriteLine($"move-proof-runtime-snapshot-file-smoke config-invalid errors={string.Join("|", configErrors)}"); + return 1; + } + + using var runtime = new MimirRuntime( + new AquariumRuntimeOptions(Headless: true, CultCachePath: ""), + new MimirRuntimeConfiguration + { + Settings = new MimirSynchronizationSettings(), + MoveProofSources = [configuration], + }); + runtime.OnSceneReady(); + var input = new InputState(); + runtime.Update(0.016f, input); + var proofFrame = runtime.Frame; + var proofSurface = runtime.LatestMoveProofSurface; + var proofSplineCount = proofFrame.Scene.SplineFrame.Splines.Count(spline => + spline.Id.StartsWith("move-proof-", StringComparison.Ordinal)); + var activationStatus = runtime.MoveProofActivationStatuses.LastOrDefault(status => + string.Equals(status.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)); + var activationDocument = runtime + .CreateMoveProofRuntimeActivationDocuments(snapshot.CapturedAtNs == 0 ? (ulong)decoded.PublishedAtNs : snapshot.CapturedAtNs) + .SingleOrDefault(document => string.Equals(document.EvidenceStreamId, configuration.EvidenceStreamId, StringComparison.Ordinal)); + var expectedMimirEvidenceFrameId = $"{configuration.MimirEvidenceFramePrefix}:{suffix}"; + var expectedMimirPoseFrameId = $"{configuration.MimirPoseFramePrefix}:{suffix}"; + var expectedFensalirFrameId = $"{configuration.FensalirFramePrefix}:{suffix}"; + + Console.WriteLine( + $"move-proof-runtime-snapshot-file-smoke snapshot=\"{absoluteSnapshotPath}\" proof={proofSurface?.ProofId ?? "none"} chain={proofSurface?.MuninnEvidenceFrameId ?? "none"}->{proofSurface?.MimirEvidenceFrameId ?? "none"}->{proofSurface?.MimirPoseFrameId ?? "none"}->{proofSurface?.FensalirFrameId ?? "none"} verdict={proofSurface?.Verdict.ToString() ?? "none"} fullPose={proofSurface?.IsFullPose.ToString() ?? "False"} markers={proofSurface?.OpticalMarkerCount.ToString() ?? "0"} controllers={proofSurface?.ControllerStateCount.ToString() ?? "0"} poses={proofSurface?.PoseCount.ToString() ?? "0"} markerWitnesses={distinctMarkerWitnesses} markerWitnessSummary=\"{markerWitnessSummary}\" nearestControllerSkewMs={nearestControllerSkewMs:0.000} failure=\"{proofSurface?.FailureReason ?? ""}\" splines={proofSplineCount} activationActive={activationStatus?.Active.ToString() ?? "False"} activationDiagnostic=\"{activationStatus?.Diagnostic ?? ""}\" activationDocumentActive={activationDocument?.Active.ToString() ?? "False"} requireFullPose={requireFullPose}"); + + return proofSurface is not null && + activationStatus?.Active == true && + activationDocument is { Active: true, DriverRegistered: true } && + activationDocument.Diagnostic.Contains("one-copy evidence snapshot supplied", StringComparison.Ordinal) && + proofSurface.MuninnEvidenceFrameId == decoded.FrameId && + proofSurface.MimirEvidenceFrameId == expectedMimirEvidenceFrameId && + proofSurface.MimirPoseFrameId == expectedMimirPoseFrameId && + proofSurface.FensalirFrameId == expectedFensalirFrameId && + proofFrame.Scene.SplineFrame.HasInput && + proofSplineCount >= 1 && + (!requireFullPose || proofSurface.IsFullPose) + ? 0 + : 1; +} + +static MimirMoveProofRuntimeConfiguration CreateMoveProofConfigurationForSnapshot( + string nativeReservoirPath, + string snapshotPath, + MimirMoveProofEvidenceFrameSnapshotDocument snapshot, + MimirMuninnMoveEvidenceStreamFrame frame) +{ + var cameras = frame.MarkerCandidates + .Where(marker => marker.SourceIdHash != 0UL) + .GroupBy(marker => marker.SourceIdHash) + .Select((group, index) => + { + var marker = group.First(); + var side = index == 0 ? -0.1 : 0.1; + return new MimirMoveProofCameraCalibrationConfiguration + { + CameraId = string.IsNullOrWhiteSpace(marker.CameraId) + ? $"snapshot-camera-{index}" + : $"{marker.HostId}:{marker.CameraId}", + WitnessIdHash = group.Key, + PositionMeters = new MimirVector3Snapshot(side, 0.0, 0.0), + Orientation = new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx = 100.0, + FocalLengthYPx = 100.0, + PrincipalPointXPx = 160.0, + PrincipalPointYPx = 120.0, + }; + }) + .Take(2) + .ToList(); + while (cameras.Count < 2) + { + var index = cameras.Count; + cameras.Add(new MimirMoveProofCameraCalibrationConfiguration + { + CameraId = $"snapshot-calibration-placeholder-{index}", + WitnessIdHash = 0xF00D_0000_0000_0001UL + (ulong)index, + PositionMeters = new MimirVector3Snapshot(index == 0 ? -0.1 : 0.1, 0.0, 0.0), + Orientation = new MimirQuaternionSnapshot(0.0, 0.0, 0.0, 1.0), + FocalLengthXPx = 100.0, + FocalLengthYPx = 100.0, + PrincipalPointXPx = 160.0, + PrincipalPointYPx = 120.0, + }); + } + + return new MimirMoveProofRuntimeConfiguration + { + EvidenceStreamId = snapshot.EvidenceStreamId, + EvidenceSnapshotPath = snapshotPath, + NativeReservoirPath = nativeReservoirPath, + MimirEvidenceSourceId = "mimir:starfire:move-evidence", + MimirEvidenceFramePrefix = "mimir:starfire:move-evidence", + MimirPoseFramePrefix = "mimir:starfire:move-pose", + MimirPoseProducerPeerId = "mimir:starfire", + FensalirFramePrefix = "fensalir:starfire:presented-frame", + FusionAuthorityId = "mimir.runtime.move-fusion", + ConsumerContract = "fensalir.move-controller-input", + Calibration = new MimirMoveProofCalibrationConfiguration + { + CalibrationId = "mimir-move-snapshot-replay-calibration-v1", + TrackingSpaceId = "mimir-stage-space", + Cameras = cameras, + MaximumAssociationSkewMilliseconds = 1000.0, + }, + }; +} + +static string SequenceSuffix(string frameId) +{ + var separator = frameId.LastIndexOf(':'); + return separator >= 0 && separator < frameId.Length - 1 + ? frameId[(separator + 1)..] + : frameId; +} + +static ulong TimestampDelta(ulong left, ulong right) => left >= right ? left - right : right - left; + static int RunMoveProofPresentedFrameSmoke(string outputPath) { var absoluteOutputPath = Path.GetFullPath(outputPath); diff --git a/src/Mimir.Runtime/Mimir.Runtime.csproj b/src/Mimir.Runtime/Mimir.Runtime.csproj index 84cbe2f..4d06a5e 100644 --- a/src/Mimir.Runtime/Mimir.Runtime.csproj +++ b/src/Mimir.Runtime/Mimir.Runtime.csproj @@ -14,6 +14,7 @@ + diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs index 73d9bc5..969c21e 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeActivation.cs @@ -15,17 +15,23 @@ public interface IMimirMoveProofEvidenceRingProvider public sealed class MimirMoveProofEvidenceRingLease : IDisposable { private readonly bool ownsRing; + private readonly IDisposable? transportOwner; - public MimirMoveProofEvidenceRingLease(CultMeshSharedMemoryFrameRing ring, bool ownsRing = false) + public MimirMoveProofEvidenceRingLease( + CultMeshSharedMemoryFrameRing ring, + bool ownsRing = false, + IDisposable? transportOwner = null) { Ring = ring ?? throw new ArgumentNullException(nameof(ring)); this.ownsRing = ownsRing; + this.transportOwner = transportOwner; } public CultMeshSharedMemoryFrameRing Ring { get; } public void Dispose() { + transportOwner?.Dispose(); if (ownsRing) { Ring.Dispose(); @@ -42,7 +48,7 @@ public sealed record MimirMoveProofEvidenceFrameSnapshotDocument( [property: Key(1)] string EvidenceStreamId, [property: Key(2)] string FrameId, [property: Key(3)] string ProducerPeerId, - [property: Key(4)] long PublishedAtNs, + [property: Key(4)] ulong PublishedAtNs, [property: Key(5)] ulong CapturedAtNs, [property: Key(6)] byte[] Payload); @@ -65,7 +71,7 @@ public static class MimirMoveProofEvidenceFrameSnapshot EvidenceStreamId: StreamIdFromFrameId(frame.FrameId), FrameId: frame.FrameId, ProducerPeerId: frame.ProducerPeerId, - PublishedAtNs: frame.PublishedAtNs, + PublishedAtNs: checked((ulong)frame.PublishedAtNs), CapturedAtNs: capturedAtNs, Payload: payload); } @@ -192,6 +198,14 @@ public sealed class MimirConfiguredMoveProofEvidenceRingProvider : IMimirMovePro out string diagnostic) { ArgumentNullException.ThrowIfNull(configuration); + if (!string.IsNullOrWhiteSpace(configuration.OdinCultMeshUri)) + { + return MimirOdinMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing( + configuration, + out lease, + out diagnostic); + } + if (string.IsNullOrWhiteSpace(configuration.EvidenceSnapshotPath)) { return MimirUnavailableMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing( @@ -219,7 +233,7 @@ public sealed class MimirConfiguredMoveProofEvidenceRingProvider : IMimirMovePro var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(snapshot.Payload); if (!string.Equals(decoded.FrameId, snapshot.FrameId, StringComparison.Ordinal) || !string.Equals(decoded.ProducerPeerId, snapshot.ProducerPeerId, StringComparison.Ordinal) || - decoded.PublishedAtNs != snapshot.PublishedAtNs) + decoded.PublishedAtNs != checked((long)snapshot.PublishedAtNs)) { lease = null; diagnostic = "Move proof evidence snapshot metadata does not match its encoded Muninn frame payload."; @@ -230,7 +244,7 @@ public sealed class MimirConfiguredMoveProofEvidenceRingProvider : IMimirMovePro configuration.EvidenceStreamId, slotCount: 1, slotByteLength: snapshot.Payload.Length); - if (!ring.TryPublishCopy(snapshot.Payload, snapshot.PublishedAtNs, durationNs: 0, out _)) + if (!ring.TryPublishCopy(snapshot.Payload, checked((long)snapshot.PublishedAtNs), durationNs: 0, out _)) { ring.Dispose(); lease = null; diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs index 6971a2a..54dabf4 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveProofRuntimeConfiguration.cs @@ -10,6 +10,10 @@ public sealed class MimirMoveProofRuntimeConfiguration public string EvidenceSnapshotPath { get; init; } = ""; + public string OdinCultMeshUri { get; init; } = ""; + + public string MuninnProviderId { get; init; } = "muninn.telemetry.nightwing"; + public string NativeReservoirPath { get; init; } = ""; public string MimirEvidenceSourceId { get; init; } = "mimir:starfire:move-evidence"; @@ -37,6 +41,10 @@ public sealed class MimirMoveProofRuntimeConfiguration var errors = new List(); Require(EvidenceStreamId, nameof(EvidenceStreamId), errors); + if (!string.IsNullOrWhiteSpace(OdinCultMeshUri)) + { + Require(MuninnProviderId, nameof(MuninnProviderId), errors); + } Require(NativeReservoirPath, nameof(NativeReservoirPath), errors); Require(MimirEvidenceSourceId, nameof(MimirEvidenceSourceId), errors); Require(MimirEvidenceFramePrefix, nameof(MimirEvidenceFramePrefix), errors); diff --git a/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs b/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs new file mode 100644 index 0000000..26faafb --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs @@ -0,0 +1,216 @@ +using System.Net; +using System.Net.Sockets; +using System.Text; +using System.Text.Json; +using GameCult.Mesh; +using GameCult.Networking; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirDiscoveredCultMeshInputStream( + string ProviderId, + string StreamId, + string Schema, + string Address, + string Channel, + uint ConnectionId); + +public sealed class MimirOdinMoveProofEvidenceRingProvider : IMimirMoveProofEvidenceRingProvider +{ + private const uint OdinSnapshotConnectionId = 0x0d1d0002; + public static MimirOdinMoveProofEvidenceRingProvider Instance { get; } = new(); + + private MimirOdinMoveProofEvidenceRingProvider() { } + + public bool TryOpenEvidenceRing( + MimirMoveProofRuntimeConfiguration configuration, + out MimirMoveProofEvidenceRingLease? lease, + out string diagnostic) + { + ArgumentNullException.ThrowIfNull(configuration); + lease = null; + MimirMoveEvidenceRudpPump? pump = null; + CultMeshSharedMemoryFrameRing? ring = null; + try + { + var stream = Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId); + ring = new CultMeshSharedMemoryFrameRing(configuration.EvidenceStreamId, slotCount: 4, slotByteLength: 256 * 1024); + pump = new MimirMoveEvidenceRudpPump(stream, ring); + pump.Start(); + lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true, transportOwner: pump); + diagnostic = $"Odin discovered {stream.ProviderId}/{stream.StreamId}; lowering {stream.Address}/{stream.Channel} into local CultMesh ring"; + return true; + } + catch (Exception error) + { + pump?.Dispose(); + ring?.Dispose(); + diagnostic = error.Message; + return false; + } + } + + public static MimirDiscoveredCultMeshInputStream Discover(string odinCultMeshUri, string providerId, string streamId) + { + ArgumentException.ThrowIfNullOrWhiteSpace(odinCultMeshUri); + ArgumentException.ThrowIfNullOrWhiteSpace(providerId); + ArgumentException.ThrowIfNullOrWhiteSpace(streamId); + var endpoint = CultMesh.ResolveRudpEndpoint(odinCultMeshUri); + using var transport = CultMesh.CreateRudpClient("mimir-odin-discovery", OdinSnapshotConnectionId, endpoint); + if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10))) + { + throw new TimeoutException($"Timed out connecting to Odin at {odinCultMeshUri}."); + } + + var messageId = $"mimir-muninn-discovery:{Guid.NewGuid():N}"; + transport.SendSchemaMessage(new CultNetSnapshotRequestMessage + { + MessageId = messageId, + SchemaIds = ["gamecult.eve.surface_state.v1"], + RecordKeys = ["surface:gamecult.network.status"] + }); + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(5); + while (DateTime.UtcNow < deadline) + { + var message = transport.ReceiveSchemaMessage(TimeSpan.FromMilliseconds(250), TimeSpan.FromMilliseconds(10)); + if (message is CultNetErrorMessage error) + { + throw new InvalidOperationException(error.Error); + } + if (message is CultNetSnapshotResponseRawMessage response && + string.Equals(response.MessageId, messageId, StringComparison.Ordinal)) + { + return SelectStream(response, providerId, streamId); + } + } + throw new TimeoutException($"Timed out waiting for Odin provider snapshot from {odinCultMeshUri}."); + } + + public static MimirDiscoveredCultMeshInputStream SelectStream( + CultNetSnapshotResponseRawMessage response, + string providerId, + string streamId) + { + var document = response.Documents.FirstOrDefault(candidate => + string.Equals(candidate.SchemaId, "gamecult.eve.surface_state.v1", StringComparison.Ordinal) && + string.Equals(candidate.RecordKey, "surface:gamecult.network.status", StringComparison.Ordinal)) + ?? throw new InvalidOperationException("Odin snapshot omitted surface:gamecult.network.status."); + using var json = JsonDocument.Parse(MessagePackSerializer.ConvertToJson(document.Payload)); + if (!json.RootElement.TryGetProperty("providerCatalog", out var providers)) + { + throw new InvalidOperationException("Odin network status omitted providerCatalog."); + } + foreach (var provider in providers.EnumerateArray()) + { + if (!string.Equals(Text(provider, "id"), providerId, StringComparison.Ordinal) || + !provider.TryGetProperty("inputStreams", out var streams)) + { + continue; + } + foreach (var stream in streams.EnumerateArray()) + { + if (!string.Equals(Text(stream, "streamId"), streamId, StringComparison.Ordinal) || + !string.Equals(Text(stream, "schema"), MimirMuninnMoveEvidenceAdapter.StreamMetadataSchemaId, StringComparison.Ordinal) || + !string.Equals(Text(stream, "transport"), "cultnet.transport.rudp.v0", StringComparison.Ordinal)) + { + continue; + } + return new MimirDiscoveredCultMeshInputStream( + providerId, + streamId, + Text(stream, "schema"), + Text(stream, "address"), + Text(stream, "channel"), + stream.GetProperty("connectionId").GetUInt32()); + } + throw new InvalidOperationException($"Odin provider '{providerId}' has no compatible input stream '{streamId}'."); + } + throw new InvalidOperationException($"Odin has no provider '{providerId}'."); + } + + private static string Text(JsonElement value, string name) => + value.TryGetProperty(name, out var property) ? property.GetString() ?? "" : ""; +} + +internal sealed class MimirMoveEvidenceRudpPump : IDisposable +{ + private readonly MimirDiscoveredCultMeshInputStream stream; + private readonly CultMeshSharedMemoryFrameRing ring; + private readonly CancellationTokenSource stopping = new(); + private readonly ManualResetEventSlim started = new(false); + private Thread? thread; + private Exception? startupError; + + public MimirMoveEvidenceRudpPump(MimirDiscoveredCultMeshInputStream stream, CultMeshSharedMemoryFrameRing ring) + { + this.stream = stream; + this.ring = ring; + } + + public void Start() + { + thread = new Thread(Run) { IsBackground = true, Name = "mimir-muninn-move-evidence" }; + thread.Start(); + if (!started.Wait(TimeSpan.FromSeconds(6))) + { + throw new TimeoutException($"Timed out starting Move evidence receiver for {stream.Address}."); + } + if (startupError is not null) + { + throw new InvalidOperationException( + $"Could not connect Odin-discovered Move evidence stream {stream.StreamId} at {stream.Address}: {startupError.Message}", + startupError); + } + } + + private void Run() + { + try + { + using var transport = CultMesh.CreateRudpClient( + "mimir-move-evidence-consumer", + stream.ConnectionId, + CultMesh.ResolveRudpEndpoint($"rudp://{stream.Address}")); + if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(5))) + { + throw new TimeoutException($"Timed out connecting to {stream.Address}."); + } + transport.Send("hid.subscribe", Encoding.UTF8.GetBytes(JsonSerializer.Serialize(new { streamId = stream.StreamId }))); + started.Set(); + while (!stopping.IsCancellationRequested) + { + transport.PollResends(); + var frame = transport.ReceiveOnce(); + if (frame is null || !string.Equals(frame.ChannelId, stream.Channel, StringComparison.Ordinal)) + { + Thread.Sleep(2); + continue; + } + var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(frame.Payload); + if (!string.Equals(decoded.FrameId, stream.StreamId, StringComparison.Ordinal) && + !decoded.FrameId.StartsWith($"{stream.StreamId}:", StringComparison.Ordinal)) + { + continue; + } + ring.TryPublishCopy(frame.Payload, decoded.PublishedAtNs, durationNs: 0, out _); + } + } + catch (Exception error) + { + if (!started.IsSet) + { + startupError = error; + started.Set(); + } + } + } + + public void Dispose() + { + stopping.Cancel(); + thread?.Join(TimeSpan.FromSeconds(2)); + started.Dispose(); + stopping.Dispose(); + } +} From 95d603aa8040a48b86435dd6a3bd8146b99bd24b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 20:50:41 +0100 Subject: [PATCH 081/135] Record live Muninn Move migration evidence --- state/evidence.jsonl | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index d6b7075..90a6ada 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -169,3 +169,8 @@ {"ts": "2026-07-12T11:01:09+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_runtime_activation.v1 as the typed readiness/proof-spine surface for configured Move proof sources. It reports stream id, provider kind, active/driver state, reservoir path, calibrated camera ids, latest same-stream proof ids/verdict, and unavailable-provider diagnostics. MimirRuntime now exposes CreateMoveProofRuntimeActivationDocuments and disposes a native reservoir if activation fails before driver retention. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-smoke, and --move-proof-runtime-activation-surface-smoke against the native reservoir debug DLL. The new smoke proves active injected-ring and inactive default-provider documents; real Nightwing/Starfire hardware and cross-process CultMesh ring opening remain pending."} {"ts": "2026-07-12T11:09:32+01:00", "type": "implementation", "status": "ok", "note": "Added mimir.move_proof_evidence_frame_snapshot.v1 and MimirConfiguredMoveProofEvidenceRingProvider as an explicit one-copy fallback for captured Muninn Move evidence frames while C# CultMesh lacks a cross-process shared-memory opener. EvidenceSnapshotPath snapshots are metadata-checked against the encoded Muninn frame payload, copied into an owned in-process ring, and consumed by the same runtime proof driver/pipeline. Verified dotnet build Mimir.Runtime, dotnet build Mimir.BufferSmoke, BufferSmoke --move-proof-runtime-driver-smoke, --move-proof-runtime-activation-surface-smoke, and --move-proof-runtime-snapshot-smoke against the native reservoir debug DLL. This supports field capture/replay and does not complete the live Nightwing/Starfire ring proof."} {"ts": "2026-07-12T11:16:48+01:00", "type": "implementation", "status": "ok", "note": "Odin/Muninn now has a producer-side --move-evidence-snapshot writer for Mimir Move proof capture/replay. The same Muninn Move evidence publisher that writes controller/marker frames to the CultMesh stream ring can write a mimir.move_proof_evidence_frame_snapshot.v1 MessagePack tuple containing snapshot id, evidence stream id, frame id, producer peer id, published ns, captured ns, and the encoded Muninn frame payload. Verified cargo test -p muninn-daemon, focused move_evidence_snapshot_writes_mimir_compatible_frame_artifact, and cargo check -p muninn-daemon --target x86_64-unknown-linux-gnu (existing dead-code warnings only). This gives Nightwing a concrete field artifact path for Mimir's EvidenceSnapshotPath provider; it is still one-copy replay support, not the final live cross-process ring/page proof."} +{"ts": "2026-07-12T11:44:53+01:00", "type": "verification", "status": "ok", "note": "Nightwing field Move proof snapshot replay reached Mimir FullPose through the explicit proof spine using artifact artifacts/runtime/nightwing-move-proof-20260712-1041-aggregate.mpack. Muninn d4f229a on Nightwing produced a bundled evidence frame with two PS3 Eye marker witnesses and one /dev/input/js0 controller state after increasing move evidence slot bytes to 32768. Mimir BufferSmoke --move-proof-runtime-snapshot-file-smoke replayed it through the configured snapshot provider and shared runtime path as muninn:nightwing:move-evidence:1 -> mimir:starfire:move-evidence:1 -> mimir:starfire:move-pose:1 -> fensalir:starfire:presented-frame:1 with verdict=FullPose, markers=99, controllers=1, markerWitnesses=2, nearestControllerSkewMs=599.022, splines=3. This proves one-copy field capture/replay and visible proof-surface lowering; it is not the final live cross-process CultMesh ring/page proof, and the controller skew remains evidence to tighten."} +{"ts": "2026-07-12T11:46:00+01:00", "type": "correction", "status": "ok", "note": "Correction to Nightwing Move proof field replay: artifact artifacts/runtime/nightwing-move-proof-20260712-1041-aggregate.mpack reached Mimir FullPose only because dark PS3 Eye frames produced bright blob candidates/noise; the PS Move markers were not lit. Treat that artifact as proof of Muninn->Mimir->Fensalir snapshot replay mechanics and a false-positive detector lesson, not proof of a real optical Move pose. A valid field pose proof still requires lighting the Move orb(s), capturing real optical marker candidates from calibrated camera witnesses, and replaying/publishing the explicit proof spine with the marker-lit condition verified."} +{"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Live Mimir Odin/CultMesh Move evidence path proved on 2026-07-12: Odin provider muninn.telemetry.nightwing advertised stream muninn:nightwing:move-evidence at 10.77.0.3:17888 channel move-evidence connection 1836384261; Mimir lowered RUDP bytes into a local CultMesh ring and decoded named frames carrying three stable Move IDs. CultLib a58daa54 changed default RUDP client bind from loopback to 0.0.0.0 so remote WireGuard routes work."} +{"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Nightwing hotplug migration proof after Odin 5faaf59: platform discovery remained authoritative across USB/Bluetooth/path reassignment and Mimir received move-0006f523e2d1, move-000704a39772, and newly wired move-000704a6be5f in one live frame after the swap. BlueZ handoff itself remains unproven because the unplugged controller only blinked and disconnected; do not claim Bluetooth success."} +{"ts": "2026-07-12T20:50:27+01:00", "type": "decision", "status": "ok", "note": "Use upstream PSMoveAPI as Muninn's optical/LED transport backend. Nightwing now has current libpsmoveapi.so and libpsmoveapi_tracker.so built against packaged OpenCV 4.6 with V4L2 camera control and installed under /usr/local. Official psmove pair created BlueZ info and HID SDP cache records for wired F5, A3, and A6; unplugged A8 still needs one USB pairing pass. The provisional luma extractor is not valid final tracking authority."} From 6778e9497530a181fc34e6abbd059889d3bfa5de Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 20:50:41 +0100 Subject: [PATCH 082/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 108360 -> 108263 bytes .voidbot/status/void-memory-maintenance.json | 2 +- .voidbot/voice/identity.json | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index dfc536297bb8ef5b41a2fd112eeaa88dcee773f9..64b4f8aca93297640540a7b1789a2250e0bd1648 100644 GIT binary patch delta 1431 zcmZ9LF^n5U6o$1C9Ee;l3giHxARZ!+BCO9|uYFzz6s3SBB0|SOq!gg>?pyE3>lrgM zYwH@0hyoI&K_hepaIPZgK)Ql1B?<+Bg62{nNJuVa(vX%n>zsW@xwA9x{qKL@fBnYX z&aZPnK0GuZwZmpRtX1mu=)#xYqJ4hh((Eh@%d^e#+}8Dl`Wv$!jhg$hb?oeT<*8|f zhV>R+w70(f=&_*MuG+UYe|h5TtGC~Kcw^fxj^92wy;pDAzYaaOr`V+P=J+M=+}5@6 zC+)5Ce~zEDaP#x(rFVFrE8rS3nI~LyK}p#M&2tnnLLm&#!DI*unfq=3xc%~(3>5+c zjY3NWeP&Q`mgj2_XH0Za!zg2>d^M6PPoRT2$06#<)i`Iozs>E_{~TXOZCEGgI>Q90 z8@=KtlZ646ltLH?9D(d;R6}OC6yTc4p%YmuBB7Pa{oI!4GahG7!hrNZ%9S>x ztw(Z5d(O=oBsf5k097LK?^8?aoN~;Q*UR1zOM1fSMj}T-8+z8KGrC4ioEy|A&tb?l z@8o#;J3|#*=VF!o87S1cQ1-#g7r~Y z@7$ty*?w_nd1Yd*-;ssWgE2Lg;*=X)fF2<0Q4d;_aZclqC#68Ao$21pe^dtvBXbE30pc5Hs{gZ&&XhZp_>vJCsT delta 1401 zcmY*ZONbmr7^b6agzT=b>=>75ejz@6Q7$(+39)*B(qyo}IjMs8~$+ z`4Zk)t9M)J^E=-iy|#V+iGMC{rwfx8jvZQ+{yO)-L1VLM+?kwmUYmYBd8?b=yxzZ0 ztnmR?k82RPX#klL+3RDcl(o-0_ly}te2AbDVfX++_B5&?GhB)?2x)-PM~DhEvH^H< z5J!v)Q~lJV@4?kKS7FE^?nA&a@+%`QHb7x4m7!(mYhWPQ0DaJj3Ydpw@FMB;L5>6} z`Xkfnjr(Vq@Ii{~;)~A0V_-sNzzdn!K)?F=g#@()bM=kqBx+c_kTAuQLAv{PsZ>#D z!Ah%YrXF$iu`u1qVO(+wR3IakD!T?$N$6wDlAEfM;#flVINPI|I zg37hgWkQ8wUoa`6txTM9x(@<3U}WNjq`()XwuNc7zF+w}yR_y!l&y7~m2~&k>1@(* zW{tA7UAC(#Ai`Wh9kPkCb5)axX08 zwJ7XevDYQySW#T}VUY(m2NW#2n*|OW(thD#y87zz@jL6(Wgs^y@60`+Wif@2t{Vv} zDLce6=iqZqS9hyoElG5RC6rlZ#U=SksKFTs*ib6yQI#0nZjPibOJ6E-lin6FQ%6}G zD)KDbUUQy2&6XQjYq;&ok{i@3%{Fr@th3y!1a)`W^I4}AxR`y*oTuhn>u#%CYj+!+ sYP;6S?tR%gaqMv6k)HF?SMR>>)K1R2oo>zT)*IDkyM6n6ZrMHeFRPyRNB{r; diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index f36baa2..a779194 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-07-12T10:15:19.747Z", + "startedAt": "2026-07-12T19:40:49.477Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index 26094d4..77c2dca 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-07-12T09:09:44.046Z", + "createdOrRefreshedAt": "2026-07-12T19:27:12.379Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 522bf41234e99abac131d53a9cb697d70ff223f9 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 21:00:36 +0100 Subject: [PATCH 083/135] Record live Move Bluetooth handoff --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 90a6ada..2267960 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -174,3 +174,4 @@ {"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Live Mimir Odin/CultMesh Move evidence path proved on 2026-07-12: Odin provider muninn.telemetry.nightwing advertised stream muninn:nightwing:move-evidence at 10.77.0.3:17888 channel move-evidence connection 1836384261; Mimir lowered RUDP bytes into a local CultMesh ring and decoded named frames carrying three stable Move IDs. CultLib a58daa54 changed default RUDP client bind from loopback to 0.0.0.0 so remote WireGuard routes work."} {"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Nightwing hotplug migration proof after Odin 5faaf59: platform discovery remained authoritative across USB/Bluetooth/path reassignment and Mimir received move-0006f523e2d1, move-000704a39772, and newly wired move-000704a6be5f in one live frame after the swap. BlueZ handoff itself remains unproven because the unplugged controller only blinked and disconnected; do not claim Bluetooth success."} {"ts": "2026-07-12T20:50:27+01:00", "type": "decision", "status": "ok", "note": "Use upstream PSMoveAPI as Muninn's optical/LED transport backend. Nightwing now has current libpsmoveapi.so and libpsmoveapi_tracker.so built against packaged OpenCV 4.6 with V4L2 camera control and installed under /usr/local. Official psmove pair created BlueZ info and HID SDP cache records for wired F5, A3, and A6; unplugged A8 still needs one USB pairing pass. The provisional luma extractor is not valid final tracking authority."} +{"ts": "2026-07-12T21:00:36+01:00", "type": "verification", "status": "ok", "note": "Live Nightwing Bluetooth handoff proved 2026-07-12 after official PSMoveAPI pairing: move-000704a6be5f migrated from USB to BlueZ Bluetooth, remained connected at 40% with /dev/input/js3 and HID_UNIQ 00:07:04:a6:be:5f, retained the same stable Move ID in Mimir's Odin-discovered four-controller evidence frame, and the large identity-colored sphere remained glowing. The red status LED blink pattern is not the connection witness; BlueZ HID path, Mimir frame identity, and visible sphere are."} From 74a19189968b12838fc824136a5f1a27f2ab1dd3 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 21:11:27 +0100 Subject: [PATCH 084/135] Associate Move optical markers by stable identity --- .../Synchronization/MimirMuninnMoveEvidence.cs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs index 9b2d4e4..da34600 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMuninnMoveEvidence.cs @@ -21,7 +21,8 @@ public sealed record MuninnMoveMarkerCandidateDocument( [property: Key(11)] float MeanLuma, [property: Key(12)] uint PeakLuma, [property: Key(13)] float Score, - [property: Key(14)] string ObservedAt); + [property: Key(14)] string ObservedAt, + [property: Key(15)] string MoveId = ""); [CultDocument("muninn.move_controller_state", "muninn.move_controller_state.v1")] [MessagePackObject] @@ -396,7 +397,9 @@ public static class MimirMuninnMoveEvidenceAdapter return new MimirNativeMoveEvidenceSample( WitnessIdHash: witnessHash, - ControllerIdHash: 0, + ControllerIdHash: string.IsNullOrWhiteSpace(marker.MoveId) + ? 0 + : Fnva64($"{marker.HostId}:{marker.MoveId}"), SourceTimestampNs: timestampNs, ArrivalNs: timestampNs, Sequence: marker.FrameSequence, From c815d1ba582e262dcdaa80da27efbf84c4b1170b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 21:39:24 +0100 Subject: [PATCH 085/135] Wait for stable-ID optical Move evidence --- src/Mimir.BufferSmoke/Program.cs | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index b21165a..a719355 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -134,10 +134,12 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-cultmesh-stream-smoke", St ParseStringOption(args, "--native-reservoir", DefaultNativeReservoirPath())); } -if (args.Any(arg => string.Equals(arg, "--muninn-move-odin-live-smoke", StringComparison.OrdinalIgnoreCase))) +if (args.Any(arg => string.Equals(arg, "--muninn-move-odin-live-smoke", StringComparison.OrdinalIgnoreCase)) || + args.Any(arg => string.Equals(arg, "--muninn-move-odin-optical-live-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMuninnMoveOdinLiveSmoke( - ParseStringOption(args, "--odin-cultmesh-uri", "cultmesh://127.0.0.1:17871/rendezvous/provider-catalog")); + ParseStringOption(args, "--odin-cultmesh-uri", "cultmesh://127.0.0.1:17871/rendezvous/provider-catalog"), + args.Any(arg => string.Equals(arg, "--muninn-move-odin-optical-live-smoke", StringComparison.OrdinalIgnoreCase))); } if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) @@ -1924,7 +1926,7 @@ static async Task RunMuninnMoveLiveRosterSmokeAsync(IReadOnlyList i : 1; } -static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri) +static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri, bool requireOptical = false) { const string providerId = "muninn.telemetry.nightwing"; const string streamId = "muninn:nightwing:move-evidence"; @@ -1945,7 +1947,7 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri) using (lease) { - var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(8); + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(requireOptical ? 45 : 8); while (DateTime.UtcNow < deadline) { if (lease.Ring.TryAcquireLatestRead(out var frameLease)) @@ -1956,7 +1958,12 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri) frameLease.Memory[..frameLease.Handle.ByteLength]); var ok = frame.FrameId.StartsWith($"{streamId}:", StringComparison.Ordinal) && frame.ControllerStates.Length >= 1 && - discovered.Channel == "move-evidence"; + discovered.Channel == "move-evidence" && + (!requireOptical || frame.MarkerCandidates.Any(marker => !string.IsNullOrWhiteSpace(marker.MoveId))); + if (!ok && requireOptical) + { + continue; + } Console.WriteLine(JsonSerializer.Serialize(new { ok, @@ -1965,7 +1972,13 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri) frame.FrameId, frame.ProducerPeerId, controllers = frame.ControllerStates.Select(state => state.MoveId).Distinct().Order().ToArray(), - markers = frame.MarkerCandidates.Length + markers = frame.MarkerCandidates.Length, + markerMoveIds = frame.MarkerCandidates + .Select(marker => marker.MoveId) + .Where(moveId => !string.IsNullOrWhiteSpace(moveId)) + .Distinct() + .Order() + .ToArray() })); return ok ? 0 : 1; } From f742a4de97d1272bbb9d3f8f647c7328b6d528ee Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 12 Jul 2026 22:49:03 +0100 Subject: [PATCH 086/135] Require distinct cameras in live optical smoke --- src/Mimir.BufferSmoke/Program.cs | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index a719355..efb8f46 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1956,10 +1956,16 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri, bool requireOptica { var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame( frameLease.Memory[..frameLease.Handle.ByteLength]); + var markerCameraIds = frame.MarkerCandidates + .Where(marker => !string.IsNullOrWhiteSpace(marker.MoveId)) + .Select(marker => marker.CameraId) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); var ok = frame.FrameId.StartsWith($"{streamId}:", StringComparison.Ordinal) && frame.ControllerStates.Length >= 1 && discovered.Channel == "move-evidence" && - (!requireOptical || frame.MarkerCandidates.Any(marker => !string.IsNullOrWhiteSpace(marker.MoveId))); + (!requireOptical || markerCameraIds.Length >= 2); if (!ok && requireOptical) { continue; @@ -1973,6 +1979,15 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri, bool requireOptica frame.ProducerPeerId, controllers = frame.ControllerStates.Select(state => state.MoveId).Distinct().Order().ToArray(), markers = frame.MarkerCandidates.Length, + markerCameraIds, + markersByCamera = frame.MarkerCandidates + .Where(marker => !string.IsNullOrWhiteSpace(marker.MoveId)) + .GroupBy(marker => marker.CameraId, StringComparer.Ordinal) + .OrderBy(group => group.Key, StringComparer.Ordinal) + .ToDictionary( + group => group.Key, + group => group.Select(marker => marker.MoveId).Distinct().Order().ToArray(), + StringComparer.Ordinal), markerMoveIds = frame.MarkerCandidates .Select(marker => marker.MoveId) .Where(moveId => !string.IsNullOrWhiteSpace(moveId)) From ee456bbcc104b3797784f7f0924d0460872e08ef Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 00:01:52 +0100 Subject: [PATCH 087/135] Probe arbitrary Muninn Move evidence providers --- src/Mimir.BufferSmoke/Program.cs | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index efb8f46..8befea1 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -139,7 +139,11 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-odin-live-smoke", StringCo { return RunMuninnMoveOdinLiveSmoke( ParseStringOption(args, "--odin-cultmesh-uri", "cultmesh://127.0.0.1:17871/rendezvous/provider-catalog"), - args.Any(arg => string.Equals(arg, "--muninn-move-odin-optical-live-smoke", StringComparison.OrdinalIgnoreCase))); + ParseStringOption(args, "--muninn-provider", "muninn.telemetry.nightwing"), + ParseStringOption(args, "--move-evidence-stream", "muninn:nightwing:move-evidence"), + args.Any(arg => string.Equals(arg, "--muninn-move-odin-optical-live-smoke", StringComparison.OrdinalIgnoreCase)) + ? ParseIntOption(args, "--required-camera-count", 2) + : 0); } if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) @@ -1926,10 +1930,12 @@ static async Task RunMuninnMoveLiveRosterSmokeAsync(IReadOnlyList i : 1; } -static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri, bool requireOptical = false) +static int RunMuninnMoveOdinLiveSmoke( + string odinCultMeshUri, + string providerId, + string streamId, + int requiredCameraCount = 0) { - const string providerId = "muninn.telemetry.nightwing"; - const string streamId = "muninn:nightwing:move-evidence"; var discovered = MimirOdinMoveProofEvidenceRingProvider.Discover(odinCultMeshUri, providerId, streamId); var configuration = new MimirMoveProofRuntimeConfiguration { @@ -1947,7 +1953,7 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri, bool requireOptica using (lease) { - var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(requireOptical ? 45 : 8); + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(requiredCameraCount > 0 ? 45 : 8); while (DateTime.UtcNow < deadline) { if (lease.Ring.TryAcquireLatestRead(out var frameLease)) @@ -1965,8 +1971,8 @@ static int RunMuninnMoveOdinLiveSmoke(string odinCultMeshUri, bool requireOptica var ok = frame.FrameId.StartsWith($"{streamId}:", StringComparison.Ordinal) && frame.ControllerStates.Length >= 1 && discovered.Channel == "move-evidence" && - (!requireOptical || markerCameraIds.Length >= 2); - if (!ok && requireOptical) + (requiredCameraCount == 0 || markerCameraIds.Length >= requiredCameraCount); + if (!ok && requiredCameraCount > 0) { continue; } From 3e627f548467d3376a3f315dc74323f4c47f3564 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 00:10:14 +0100 Subject: [PATCH 088/135] Remember Move tracking false blocker --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 2267960..beacb48 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -175,3 +175,4 @@ {"ts": "2026-07-12T20:50:27+01:00", "type": "verification", "status": "ok", "note": "Nightwing hotplug migration proof after Odin 5faaf59: platform discovery remained authoritative across USB/Bluetooth/path reassignment and Mimir received move-0006f523e2d1, move-000704a39772, and newly wired move-000704a6be5f in one live frame after the swap. BlueZ handoff itself remains unproven because the unplugged controller only blinked and disconnected; do not claim Bluetooth success."} {"ts": "2026-07-12T20:50:27+01:00", "type": "decision", "status": "ok", "note": "Use upstream PSMoveAPI as Muninn's optical/LED transport backend. Nightwing now has current libpsmoveapi.so and libpsmoveapi_tracker.so built against packaged OpenCV 4.6 with V4L2 camera control and installed under /usr/local. Official psmove pair created BlueZ info and HID SDP cache records for wired F5, A3, and A6; unplugged A8 still needs one USB pairing pass. The provisional luma extractor is not valid final tracking authority."} {"ts": "2026-07-12T21:00:36+01:00", "type": "verification", "status": "ok", "note": "Live Nightwing Bluetooth handoff proved 2026-07-12 after official PSMoveAPI pairing: move-000704a6be5f migrated from USB to BlueZ Bluetooth, remained connected at 40% with /dev/input/js3 and HID_UNIQ 00:07:04:a6:be:5f, retained the same stable Move ID in Mimir's Odin-discovered four-controller evidence frame, and the large identity-colored sphere remained glowing. The red status LED blink pattern is not the connection witness; BlueZ HID path, Mimir frame identity, and visible sphere are."} +{"ts": "2026-07-13T00:10:04+01:00", "type": "scar", "status": "open", "note": "Move tracking false-blocker: both Nightwing PS3 Eyes were already physically positioned. The missing authority was software process isolation and persistent dual-provider ingestion. Do not ask the operator to rearrange hardware without first probing all live devices/providers. Repeated one-shot Mimir probes also misuse Muninn's single-client RUDP evidence server; visibility sweeps must keep one persistent subscription per provider."} From f80881e0d7498e4834fcc1cd91323e633fa4ada4 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 00:17:00 +0100 Subject: [PATCH 089/135] Measure persistent dual-camera Move visibility --- src/Mimir.BufferSmoke/Program.cs | 102 ++++++++++++++++++ .../MimirMoveVisibilityWindow.cs | 31 ++++++ state/evidence.jsonl | 1 + 3 files changed, 134 insertions(+) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveVisibilityWindow.cs diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 8befea1..0b5a2e2 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -146,6 +146,14 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-odin-live-smoke", StringCo : 0); } +if (args.Any(arg => string.Equals(arg, "--muninn-move-visibility-window-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunMuninnMoveVisibilityWindowSmoke( + ParseStringOption(args, "--odin-cultmesh-uri", "cultmesh://127.0.0.1:17871/rendezvous/provider-catalog"), + ParseIntOption(args, "--duration-seconds", 15), + ParseStringOption(args, "--output", "artifacts/move-calibration/muninn-visibility-window.mpack")); +} + if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveFusionSmoke(); @@ -2010,6 +2018,100 @@ static int RunMuninnMoveOdinLiveSmoke( throw new TimeoutException($"No live frame arrived from Odin-discovered stream {streamId}."); } +static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durationSeconds, string outputPath) +{ + var sources = new[] + { + (Provider: "muninn.telemetry.nightwing", Stream: "muninn:nightwing:move-evidence"), + (Provider: "muninn.telemetry.nightwing-eye0", Stream: "muninn:nightwing:eye-0:move-evidence") + }; + var leases = new List<(string Provider, string Stream, MimirMoveProofEvidenceRingLease Lease)>(); + try + { + foreach (var source in sources) + { + var configuration = new MimirMoveProofRuntimeConfiguration + { + OdinCultMeshUri = odinCultMeshUri, + MuninnProviderId = source.Provider, + EvidenceStreamId = source.Stream + }; + if (!MimirOdinMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing(configuration, out var lease, out var diagnostic) || lease is null) + { + throw new InvalidOperationException($"{source.Provider}: {diagnostic}"); + } + leases.Add((source.Provider, source.Stream, lease)); + } + + var startedAtNs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000; + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(Math.Max(1, durationSeconds)); + var seenFrames = new HashSet(StringComparer.Ordinal); + var framesByProvider = sources.ToDictionary(source => source.Provider, _ => 0, StringComparer.Ordinal); + var controllersByProvider = sources.ToDictionary(source => source.Provider, _ => new HashSet(StringComparer.Ordinal), StringComparer.Ordinal); + var observations = new List(); + while (DateTime.UtcNow < deadline) + { + foreach (var source in leases) + { + if (!source.Lease.Ring.TryAcquireLatestRead(out var frameLease)) continue; + using (frameLease) + { + var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(frameLease.Memory[..frameLease.Handle.ByteLength]); + if (!seenFrames.Add($"{source.Provider}:{frame.FrameId}")) continue; + framesByProvider[source.Provider]++; + controllersByProvider[source.Provider].UnionWith(frame.ControllerStates.Select(state => state.MoveId)); + observations.AddRange(frame.MarkerCandidates + .Where(marker => !string.IsNullOrWhiteSpace(marker.MoveId)) + .Select(marker => new MoveVisibilityObservation( + source.Provider, marker.CameraId, marker.MoveId, frame.FrameId, frame.PublishedAtNs, + marker.CenterXPx, marker.CenterYPx, marker.RadiusPx, marker.Score))); + } + } + Thread.Sleep(5); + } + + var correspondences = observations + .GroupBy(observation => observation.MoveId, StringComparer.Ordinal) + .SelectMany(group => + { + var first = group.Where(value => value.ProviderId == sources[0].Provider).ToArray(); + var second = group.Where(value => value.ProviderId == sources[1].Provider).ToArray(); + return first.Select(left => second + .Select(right => new MoveCrossCameraCorrespondence(left.MoveId, left, right, Math.Abs(left.PublishedAtNs - right.PublishedAtNs))) + .OrderBy(pair => pair.AbsoluteSkewNs).FirstOrDefault()) + .Where(pair => pair is not null)!; + }) + .Cast() + .ToArray(); + var receipt = new MoveVisibilityWindowReceipt( + "mimir.move_visibility_window.v1", startedAtNs, + DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000, + sources.Select(source => source.Provider).ToArray(), observations.ToArray(), correspondences); + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath))!); + File.WriteAllBytes(outputPath, MessagePackSerializer.Serialize(receipt)); + + var visibleByProvider = observations.GroupBy(value => value.ProviderId).ToDictionary( + group => group.Key, group => group.Select(value => value.MoveId).Distinct().Order().ToArray()); + Console.WriteLine(JsonSerializer.Serialize(new + { + receipt.Schema, + outputPath, + framesByProvider, + controllersByProvider = controllersByProvider.ToDictionary(pair => pair.Key, pair => pair.Value.Order().ToArray()), + observations = observations.Count, + correspondences = correspondences.Length, + visibleByProvider + })); + return sources.All(source => framesByProvider[source.Provider] > 0) && + sources.All(source => visibleByProvider.ContainsKey(source.Provider)) && + correspondences.Length > 0 ? 0 : 1; + } + finally + { + foreach (var source in leases) source.Lease.Dispose(); + } +} + static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) { if (!File.Exists(nativeReservoirPath)) diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveVisibilityWindow.cs b/src/Mimir.Runtime/Synchronization/MimirMoveVisibilityWindow.cs new file mode 100644 index 0000000..9df6d89 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveVisibilityWindow.cs @@ -0,0 +1,31 @@ +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[MessagePackObject] +public sealed record MoveVisibilityObservation( + [property: Key(0)] string ProviderId, + [property: Key(1)] string CameraId, + [property: Key(2)] string MoveId, + [property: Key(3)] string FrameId, + [property: Key(4)] long PublishedAtNs, + [property: Key(5)] float CenterXPx, + [property: Key(6)] float CenterYPx, + [property: Key(7)] float RadiusPx, + [property: Key(8)] float Confidence); + +[MessagePackObject] +public sealed record MoveCrossCameraCorrespondence( + [property: Key(0)] string MoveId, + [property: Key(1)] MoveVisibilityObservation First, + [property: Key(2)] MoveVisibilityObservation Second, + [property: Key(3)] long AbsoluteSkewNs); + +[MessagePackObject] +public sealed record MoveVisibilityWindowReceipt( + [property: Key(0)] string Schema, + [property: Key(1)] long StartedAtNs, + [property: Key(2)] long EndedAtNs, + [property: Key(3)] string[] Providers, + [property: Key(4)] MoveVisibilityObservation[] Observations, + [property: Key(5)] MoveCrossCameraCorrespondence[] Correspondences); diff --git a/state/evidence.jsonl b/state/evidence.jsonl index beacb48..8290860 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -176,3 +176,4 @@ {"ts": "2026-07-12T20:50:27+01:00", "type": "decision", "status": "ok", "note": "Use upstream PSMoveAPI as Muninn's optical/LED transport backend. Nightwing now has current libpsmoveapi.so and libpsmoveapi_tracker.so built against packaged OpenCV 4.6 with V4L2 camera control and installed under /usr/local. Official psmove pair created BlueZ info and HID SDP cache records for wired F5, A3, and A6; unplugged A8 still needs one USB pairing pass. The provisional luma extractor is not valid final tracking authority."} {"ts": "2026-07-12T21:00:36+01:00", "type": "verification", "status": "ok", "note": "Live Nightwing Bluetooth handoff proved 2026-07-12 after official PSMoveAPI pairing: move-000704a6be5f migrated from USB to BlueZ Bluetooth, remained connected at 40% with /dev/input/js3 and HID_UNIQ 00:07:04:a6:be:5f, retained the same stable Move ID in Mimir's Odin-discovered four-controller evidence frame, and the large identity-colored sphere remained glowing. The red status LED blink pattern is not the connection witness; BlueZ HID path, Mimir frame identity, and visible sphere are."} {"ts": "2026-07-13T00:10:04+01:00", "type": "scar", "status": "open", "note": "Move tracking false-blocker: both Nightwing PS3 Eyes were already physically positioned. The missing authority was software process isolation and persistent dual-provider ingestion. Do not ask the operator to rearrange hardware without first probing all live devices/providers. Repeated one-shot Mimir probes also misuse Muninn's single-client RUDP evidence server; visibility sweeps must keep one persistent subscription per provider."} +{"ts": "2026-07-13T00:16:58+01:00", "type": "verification", "status": "partial", "note": "Persistent dual-provider visibility window separated transport from optics: nightwing-eye0 delivered 2 frames and all 4 controller IDs but 0 labelled markers; primary nightwing delivered 0 frames, consistent with its single-client evidence server remaining occupied after prior one-shot probes. Restart observers before the next window; do not infer camera visibility from discovery or controller presence."} From 150a05434458553a027b435c591f88300fa56987 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 00:22:09 +0100 Subject: [PATCH 090/135] Measure Move visibility within one Muninn provider --- src/Mimir.BufferSmoke/Program.cs | 15 +++++++++------ state/evidence.jsonl | 1 + 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 0b5a2e2..a09172a 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -2022,9 +2022,9 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati { var sources = new[] { - (Provider: "muninn.telemetry.nightwing", Stream: "muninn:nightwing:move-evidence"), - (Provider: "muninn.telemetry.nightwing-eye0", Stream: "muninn:nightwing:eye-0:move-evidence") + (Provider: "muninn.telemetry.nightwing", Stream: "muninn:nightwing:move-evidence") }; + var cameraIds = new[] { "nightwing-eye-0", "nightwing-eye-1" }; var leases = new List<(string Provider, string Stream, MimirMoveProofEvidenceRingLease Lease)>(); try { @@ -2074,8 +2074,8 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati .GroupBy(observation => observation.MoveId, StringComparer.Ordinal) .SelectMany(group => { - var first = group.Where(value => value.ProviderId == sources[0].Provider).ToArray(); - var second = group.Where(value => value.ProviderId == sources[1].Provider).ToArray(); + var first = group.Where(value => value.CameraId == cameraIds[0]).ToArray(); + var second = group.Where(value => value.CameraId == cameraIds[1]).ToArray(); return first.Select(left => second .Select(right => new MoveCrossCameraCorrespondence(left.MoveId, left, right, Math.Abs(left.PublishedAtNs - right.PublishedAtNs))) .OrderBy(pair => pair.AbsoluteSkewNs).FirstOrDefault()) @@ -2092,6 +2092,8 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati var visibleByProvider = observations.GroupBy(value => value.ProviderId).ToDictionary( group => group.Key, group => group.Select(value => value.MoveId).Distinct().Order().ToArray()); + var visibleByCamera = observations.GroupBy(value => value.CameraId).ToDictionary( + group => group.Key, group => group.Select(value => value.MoveId).Distinct().Order().ToArray()); Console.WriteLine(JsonSerializer.Serialize(new { receipt.Schema, @@ -2100,10 +2102,11 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati controllersByProvider = controllersByProvider.ToDictionary(pair => pair.Key, pair => pair.Value.Order().ToArray()), observations = observations.Count, correspondences = correspondences.Length, - visibleByProvider + visibleByProvider, + visibleByCamera })); return sources.All(source => framesByProvider[source.Provider] > 0) && - sources.All(source => visibleByProvider.ContainsKey(source.Provider)) && + cameraIds.All(visibleByCamera.ContainsKey) && correspondences.Length > 0 ? 0 : 1; } finally diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 8290860..b9db4d0 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -177,3 +177,4 @@ {"ts": "2026-07-12T21:00:36+01:00", "type": "verification", "status": "ok", "note": "Live Nightwing Bluetooth handoff proved 2026-07-12 after official PSMoveAPI pairing: move-000704a6be5f migrated from USB to BlueZ Bluetooth, remained connected at 40% with /dev/input/js3 and HID_UNIQ 00:07:04:a6:be:5f, retained the same stable Move ID in Mimir's Odin-discovered four-controller evidence frame, and the large identity-colored sphere remained glowing. The red status LED blink pattern is not the connection witness; BlueZ HID path, Mimir frame identity, and visible sphere are."} {"ts": "2026-07-13T00:10:04+01:00", "type": "scar", "status": "open", "note": "Move tracking false-blocker: both Nightwing PS3 Eyes were already physically positioned. The missing authority was software process isolation and persistent dual-provider ingestion. Do not ask the operator to rearrange hardware without first probing all live devices/providers. Repeated one-shot Mimir probes also misuse Muninn's single-client RUDP evidence server; visibility sweeps must keep one persistent subscription per provider."} {"ts": "2026-07-13T00:16:58+01:00", "type": "verification", "status": "partial", "note": "Persistent dual-provider visibility window separated transport from optics: nightwing-eye0 delivered 2 frames and all 4 controller IDs but 0 labelled markers; primary nightwing delivered 0 frames, consistent with its single-client evidence server remaining occupied after prior one-shot probes. Restart observers before the next window; do not infer camera visibility from discovery or controller presence."} +{"ts": "2026-07-13T00:22:08+01:00", "type": "decision", "status": "ok", "note": "Nightwing Move observation is one Muninn provider with two camera sources. Muninn already accepts repeated --move-marker-camera arguments and aggregates camera-tagged candidates into one evidence frame; the prior nightwing-eye0 provider leaked a process workaround into public authority. If PSMoveAPI needs isolation, use internal workers without separate provider identity."} From e757538edaff62724bfd6d800656e0fc2457e634 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 00:29:24 +0100 Subject: [PATCH 091/135] Expose live Move evidence receiver failures --- .../MimirOdinMoveProofEvidenceRingProvider.cs | 5 +++++ state/evidence.jsonl | 1 + 2 files changed, 6 insertions(+) diff --git a/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs b/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs index 26faafb..5dd2c86 100644 --- a/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs +++ b/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs @@ -203,6 +203,11 @@ internal sealed class MimirMoveEvidenceRudpPump : IDisposable startupError = error; started.Set(); } + else if (!stopping.IsCancellationRequested) + { + Console.Error.WriteLine( + $"Mimir Move evidence receiver stopped stream={stream.StreamId} address={stream.Address}: {error}"); + } } } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index b9db4d0..2c21431 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -178,3 +178,4 @@ {"ts": "2026-07-13T00:10:04+01:00", "type": "scar", "status": "open", "note": "Move tracking false-blocker: both Nightwing PS3 Eyes were already physically positioned. The missing authority was software process isolation and persistent dual-provider ingestion. Do not ask the operator to rearrange hardware without first probing all live devices/providers. Repeated one-shot Mimir probes also misuse Muninn's single-client RUDP evidence server; visibility sweeps must keep one persistent subscription per provider."} {"ts": "2026-07-13T00:16:58+01:00", "type": "verification", "status": "partial", "note": "Persistent dual-provider visibility window separated transport from optics: nightwing-eye0 delivered 2 frames and all 4 controller IDs but 0 labelled markers; primary nightwing delivered 0 frames, consistent with its single-client evidence server remaining occupied after prior one-shot probes. Restart observers before the next window; do not infer camera visibility from discovery or controller presence."} {"ts": "2026-07-13T00:22:08+01:00", "type": "decision", "status": "ok", "note": "Nightwing Move observation is one Muninn provider with two camera sources. Muninn already accepts repeated --move-marker-camera arguments and aggregates camera-tagged candidates into one evidence frame; the prior nightwing-eye0 provider leaked a process workaround into public authority. If PSMoveAPI needs isolation, use internal workers without separate provider identity."} +{"ts": "2026-07-13T00:29:23+01:00", "type": "scar", "status": "partial", "note": "Unified Nightwing Muninn proves one provider can own both /dev/video2 and /dev/video3 and publish all four controller IDs. Stable 30s visibility window saw 2 controller frames and 0 labelled markers. Hue state is animated/descending/1000ms and light HID paths are dynamically resolved; the next missing evidence is typed per-camera PSMoveAPI readiness/calibrated-controller/update health. Do not infer tracker health from camera fd ownership, payload size, or visible lights."} From bf110a6ace0669e6a55c42b5dcb0876f2f533488 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 00:48:00 +0100 Subject: [PATCH 092/135] Remember dual-eye PSMoveAPI isolation proof --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 2c21431..dbf94a5 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -179,3 +179,4 @@ {"ts": "2026-07-13T00:16:58+01:00", "type": "verification", "status": "partial", "note": "Persistent dual-provider visibility window separated transport from optics: nightwing-eye0 delivered 2 frames and all 4 controller IDs but 0 labelled markers; primary nightwing delivered 0 frames, consistent with its single-client evidence server remaining occupied after prior one-shot probes. Restart observers before the next window; do not infer camera visibility from discovery or controller presence."} {"ts": "2026-07-13T00:22:08+01:00", "type": "decision", "status": "ok", "note": "Nightwing Move observation is one Muninn provider with two camera sources. Muninn already accepts repeated --move-marker-camera arguments and aggregates camera-tagged candidates into one evidence frame; the prior nightwing-eye0 provider leaked a process workaround into public authority. If PSMoveAPI needs isolation, use internal workers without separate provider identity."} {"ts": "2026-07-13T00:29:23+01:00", "type": "scar", "status": "partial", "note": "Unified Nightwing Muninn proves one provider can own both /dev/video2 and /dev/video3 and publish all four controller IDs. Stable 30s visibility window saw 2 controller frames and 0 labelled markers. Hue state is animated/descending/1000ms and light HID paths are dynamically resolved; the next missing evidence is typed per-camera PSMoveAPI readiness/calibrated-controller/update health. Do not infer tracker health from camera fd ownership, payload size, or visible lights."} +{"ts": "2026-07-13T00:47:59+01:00", "type": "verification", "status": "ok", "note": "Typed muninn.move_tracker_health.v1 field evidence isolated the dual-eye failure. Both Nightwing PS3 Eyes identify as V4L2/PS3 Eye 640x480 at exposure 0.1 and can calibrate all 4 Moves, but same-process PSMoveAPI trackers freeze around initialization/update: one process snapshot reached running with update_count=1 for each eye and zero observations; after restart both remained opening with zero updates. Keep one Muninn provider, but run each native PSMoveAPI tracker in an internal subprocess and aggregate camera-tagged observations in the parent."} From c1939b72f52c4aa2092aaccc8ba08f826cd2904a Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 01:15:00 +0100 Subject: [PATCH 093/135] Remember isolated dual-eye tracker field proof --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index dbf94a5..45d0fbf 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -180,3 +180,4 @@ {"ts": "2026-07-13T00:22:08+01:00", "type": "decision", "status": "ok", "note": "Nightwing Move observation is one Muninn provider with two camera sources. Muninn already accepts repeated --move-marker-camera arguments and aggregates camera-tagged candidates into one evidence frame; the prior nightwing-eye0 provider leaked a process workaround into public authority. If PSMoveAPI needs isolation, use internal workers without separate provider identity."} {"ts": "2026-07-13T00:29:23+01:00", "type": "scar", "status": "partial", "note": "Unified Nightwing Muninn proves one provider can own both /dev/video2 and /dev/video3 and publish all four controller IDs. Stable 30s visibility window saw 2 controller frames and 0 labelled markers. Hue state is animated/descending/1000ms and light HID paths are dynamically resolved; the next missing evidence is typed per-camera PSMoveAPI readiness/calibrated-controller/update health. Do not infer tracker health from camera fd ownership, payload size, or visible lights."} {"ts": "2026-07-13T00:47:59+01:00", "type": "verification", "status": "ok", "note": "Typed muninn.move_tracker_health.v1 field evidence isolated the dual-eye failure. Both Nightwing PS3 Eyes identify as V4L2/PS3 Eye 640x480 at exposure 0.1 and can calibrate all 4 Moves, but same-process PSMoveAPI trackers freeze around initialization/update: one process snapshot reached running with update_count=1 for each eye and zero observations; after restart both remained opening with zero updates. Keep one Muninn provider, but run each native PSMoveAPI tracker in an internal subprocess and aggregate camera-tagged observations in the parent."} +{"ts": "2026-07-13T01:15:00+01:00", "type": "verification", "status": "partial", "note": "Nightwing now runs one Muninn provider with private per-eye tracker subprocesses and parent-owned MessagePack hue/observation pipes. Current live proof: both eye workers advance thousands of updates; eye1 calibrated 4 Moves and produced 11,908+ observations, and Mimir received all four stable Move IDs from nightwing-eye-1 in one 15s window. Eye0 calibrated all 4 and advances updates but has zero optical observations, so dual-camera correspondence remains unproven and the remaining fault is eye0 optical scene/backend behavior. Idunn restart now explicitly kills worker children with the parent generation."} From 2196503d54e7a78cebdd17f967566dab6b9e0534 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 01:29:16 +0100 Subject: [PATCH 094/135] Map live dual-eye Move tracking state --- notes/current-system-map.md | 16 +++++++++-- notes/fresh-workspace-handoff.md | 49 ++++++++++++++++++-------------- state/evidence.jsonl | 1 + state/map.yaml | 32 +++++++++++++++++++-- 4 files changed, 71 insertions(+), 27 deletions(-) diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 3ec535d..2a8b76f 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -196,9 +196,19 @@ Y8 frame run through `muninn-move-tracker` serializes a non-empty marker candidate into the Mimir-compatible evidence frame. The daemon now has a source-local Y8 extraction/publish seam plus a first `serve` camera producer: `--move-marker-camera =` polls a Unix V4L2 YUYV frame, -converts it to compact Y8, and feeds that same seam. Windows unit tests prove -the configured camera tick publishes marker evidence through the shared frame -contract; Nightwing hardware/live V4L2 proof is still pending. +converts it to compact Y8, and feeds that same seam. Nightwing hardware now +runs as one Muninn provider with two private per-eye PSMoveAPI subprocesses. +The parent alone owns lights, identity, commands, provider advertisement, +persistence, and the aggregate evidence stream; children exchange hue state +and camera-tagged observations over bounded MessagePack pipes. +`muninn.move_tracker_health.v1` exposes each worker's camera backend, +calibration, update/observation counters, RGB range, and expected-hue pixel +evidence. Live exposure is 0.3 for `nightwing-eye-0` and 0.1 for +`nightwing-eye-1`. Both eyes calibrate all four Moves and produce observations. +The latest Mimir window saw all four IDs on Eye 0, two on Eye 1, and two +same-ID cross-camera correspondences. Do not claim full stereo calibration yet: +four-ID overlap is incomplete and the RUDP subscriber currently receives only +two aggregate frames per 20-60 second window despite continuous worker updates. `MimirMoveProofSurface` is the first Fensalir-visible proof surface for this chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 904eb1e..417dcc1 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -34,10 +34,10 @@ Get-Content .\state\evidence.jsonl -Tail 8 `cultnet.transport.rudp.v0`; TCP/HTTP/WebSocket are client lowerings, compatibility probes, or migration debt unless a specific runtime has not earned RUDP yet. -- Mimir has daemon-health publication edges into the Odin/Idunn swarm: - `src/Mimir.EveDashboard` and `src/Mimir.EveBrowserReference` publish - `idunn.daemon_health` records over `cultnet.transport.rudp.v0` when - configured. These are self-health witnesses, not lifecycle owners. +- Mimir's old Eve dashboard broker is archived and no longer publishes Idunn + health, `/eve/deck`, `/eve/dashboard`, or WebSocket commands. The browser + reference is a client lowering, not service truth. Dashboard state must return + as typed CultMesh/Eve documents through Odin before deployment resumes. - The live stream app is C# plus Fensalir: `Mimir.slnx` contains `src/Mimir.App` and `src/Mimir.Runtime`. - `src/Mimir.App` hosts Fensalir as the windowing/rendering/D3D12 bridge. @@ -79,6 +79,14 @@ Get-Content .\state\evidence.jsonl -Tail 8 ## Current Pressure +- Nightwing Move tracking is one Muninn provider with two private per-eye + PSMoveAPI worker subprocesses. Eye 0 uses exposure 0.3; Eye 1 uses 0.1. + Both workers calibrate four Moves and advance continuously. Mimir has proved + all four stable IDs on Eye 0, two on Eye 1, and two cross-camera + correspondences. Next fix the aggregate evidence RUDP cadence (only two + frames reached a persistent subscriber in 20-60 seconds), then earn four-ID + stereo overlap before feeding camera calibration. + - Implement the first concrete Leap direct capture driver and measure cadence. - Add remaining camera drivers through `IMimirVideoCaptureDriver`. - Add native audio capture workers for mic, loopback, and network audio feeds. @@ -89,23 +97,22 @@ Get-Content .\state\evidence.jsonl -Tail 8 output management. - Keep PowerShell/FFmpeg/SRT as bridge utilities until native Mimir program output, Eve operation, and Yggdrasil site publication replace them. -- Yggdrasil currently runs the CultMesh media relay on UDP `3075` from - `/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia`; Starfire currently - runs the receiver PID from - `artifacts/mimir-cultmesh-media-win-x64-selfcontained/Mimir.CultMeshMedia.exe` - and writes `raven-primary-av` to `udp://127.0.0.1:5200`. OBS source - `Raven Monitor + Realtek` has been repointed from old SRT to that local UDP - endpoint. Raven SSH timed out, so launch - `scripts/start-raven-cultmesh-av-sender.ps1` on Raven directly. -- Keep Eve dashboard/browser-reference RUDP health aligned with Odin/Idunn: - Odin accepts the Verse/service map, Idunn owns daemon continuity, and Mimir - processes publish typed health/freshness only for themselves. The next - dashboard boundary cut is provider advertisement, retained state, - command_boundary, and transport_profile over RUDP so HTTP/WebSocket catalog - ingestion can be demoted to client/debug lowering. -- Cut the remaining Eve dashboard, recorder, and witness service truth paths to - explicit CultNet RUDP records next; no new TCP, HTTP, or WebSocket surface - should be allowed to become daemon or transport truth. +- Yggdrasil currently runs the CultMesh media relay from + `/opt/gamecult/mimir-cultmesh-media/Mimir.CultMeshMedia`; senders and + receivers target `cultmesh://asgard.yggdrasil.mimir/media/raven-primary-av` + and let the CultMesh resolver choose the concrete RUDP route. Starfire writes + `raven-primary-av` to `udp://127.0.0.1:5200`. That is the + `Mimir.CultMeshMedia` body bridge, not the OBS plugin feed owner. +- The actual Raven OBS/SRT feed owner is Odin's Muninn on Raven. Use + `scripts/start-raven-muninn-obs-feed.ps1` in this repo as the thin wrapper + over `E:\Projects\Odin\scripts\activate-muninn-raven-av-srt.ps1`, which + drives Raven's real `GameCult-Muninn-Activate` hidden task and + `muninn.exe activate` path. `scripts/start-raven-cultmesh-av-sender.ps1` + remains the separate CultMesh bridge/bootstrap lane. +- Rebuild Eve dashboard only as a CultMesh/Odin surface publisher plus renderer + lowering. Provider advertisement, retained state, command boundary, transport + profile, and command documents must be typed CultMesh records; no TCP, HTTP, + or WebSocket surface should become daemon or transport truth. - Kiyo Pro has two UVC extension units. Moving it to a motherboard USB3 port changed the descriptor path to `root_hub30` / `bcdUSB=0x0320` and exposed 720p/1080p YUY2/MJPG/H264/NV12 at 60 fps, but Windows still reports diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 45d0fbf..7e34847 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -181,3 +181,4 @@ {"ts": "2026-07-13T00:29:23+01:00", "type": "scar", "status": "partial", "note": "Unified Nightwing Muninn proves one provider can own both /dev/video2 and /dev/video3 and publish all four controller IDs. Stable 30s visibility window saw 2 controller frames and 0 labelled markers. Hue state is animated/descending/1000ms and light HID paths are dynamically resolved; the next missing evidence is typed per-camera PSMoveAPI readiness/calibrated-controller/update health. Do not infer tracker health from camera fd ownership, payload size, or visible lights."} {"ts": "2026-07-13T00:47:59+01:00", "type": "verification", "status": "ok", "note": "Typed muninn.move_tracker_health.v1 field evidence isolated the dual-eye failure. Both Nightwing PS3 Eyes identify as V4L2/PS3 Eye 640x480 at exposure 0.1 and can calibrate all 4 Moves, but same-process PSMoveAPI trackers freeze around initialization/update: one process snapshot reached running with update_count=1 for each eye and zero observations; after restart both remained opening with zero updates. Keep one Muninn provider, but run each native PSMoveAPI tracker in an internal subprocess and aggregate camera-tagged observations in the parent."} {"ts": "2026-07-13T01:15:00+01:00", "type": "verification", "status": "partial", "note": "Nightwing now runs one Muninn provider with private per-eye tracker subprocesses and parent-owned MessagePack hue/observation pipes. Current live proof: both eye workers advance thousands of updates; eye1 calibrated 4 Moves and produced 11,908+ observations, and Mimir received all four stable Move IDs from nightwing-eye-1 in one 15s window. Eye0 calibrated all 4 and advances updates but has zero optical observations, so dual-camera correspondence remains unproven and the remaining fault is eye0 optical scene/backend behavior. Idunn restart now explicitly kills worker children with the parent generation."} +{"ts": "2026-07-13T01:29:15+01:00", "type": "verification", "status": "partial", "note": "Per-camera optical evidence and exposure ownership restored Eye 0. At exposure 0.3 Eye 0 peak RGB rose from [72,25,43] to [178,119,112], calibrated 4 Moves, and produced 3,312 observations/latest 4; Eye 1 remains at exposure 0.1. Mimir's persistent window then saw all 4 IDs on Eye 0, 2 on Eye 1, and 2 same-ID cross-camera correspondences. Do not claim full stereo: Eye 1 overlap is incomplete and only 2 aggregate RUDP frames arrived in 20s and 60s windows."} diff --git a/state/map.yaml b/state/map.yaml index 075e051..c425177 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -194,6 +194,22 @@ accepted_design: both the `mimir.move_proof_evidence_frame_snapshot.v1` tuple and embedded Muninn payload. This supports Nightwing field capture/replay through Mimir's snapshot provider while the final live ring/page transport remains pending. + - Nightwing field capture/replay has proven the one-copy snapshot mechanics, + but the first `FullPose` replay was physically false-positive evidence. + Muninn `d4f229a` bundles latest controller state with aggregated marker + candidates from both Nightwing PS3 Eyes into one evidence frame, with the + snapshot payload encoded as MessagePack bytes. The preserved artifact + `artifacts/runtime/nightwing-move-proof-20260712-1041-aggregate.mpack` + replays through `Mimir.BufferSmoke --move-proof-runtime-snapshot-file-smoke` + as `muninn:nightwing:move-evidence:1 -> mimir:starfire:move-evidence:1 -> + mimir:starfire:move-pose:1 -> fensalir:starfire:presented-frame:1`, but + the operator confirmed the Move markers were dark during that capture. Treat + its 99 marker candidates as bright image blobs/noise and use the artifact + only as evidence that Muninn/Mimir/Fensalir replay plumbing works and needs + marker-lit validation. A valid field pose proof still requires lighting the + PS Move orb, capturing real optical marker candidates from calibrated camera + witnesses, and preserving the explicit same-sequence proof spine. The final + live cross-process CultMesh ring/page proof remains pending. - `MimirMoveProofDevSurface` is a dev-only bootstrap gated by `MIMIR_MOVE_PROOF_DEV_SURFACE`. It publishes a synthetic calibrated proof surface through the same MimirRuntime attachment so headless Mimir.App can @@ -232,9 +248,19 @@ accepted_design: evidence. The daemon now has a source-local Y8 extraction/publish seam and a first `serve` camera producer: `--move-marker-camera =` polls a Unix V4L2 YUYV frame, converts it to compact Y8, and feeds that - same seam. Unit tests prove the configured camera tick publishes marker - evidence through the shared frame contract; Nightwing hardware/live V4L2 - proof is still pending. + same seam. Nightwing now runs one public Muninn provider whose two private + per-eye PSMoveAPI subprocesses exchange hue state, observations, and health + with the parent through bounded MessagePack pipes. Children own no store, + provider advertisement, command route, or light output. Typed + `muninn.move_tracker_health.v1` records expose camera backend, calibration, + update/observation counts, RGB range, and expected-hue pixel evidence. + Live field proof on 2026-07-13 established Eye 0 at exposure 0.3 and Eye 1 + at 0.1; both calibrated four Moves and produced observations. A Mimir + visibility window saw all four IDs on Eye 0, two IDs on Eye 1, and two + cross-camera correspondences. Full four-ID stereo coverage and stream + cadence remain pending: the remote subscriber received only two aggregate + frames during both 20-second and 60-second windows despite continuously + advancing tracker workers. - `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a typed admission receipt with decoded frame id, producer, publish/read timestamps, sample counts by kind, source/arrival ranges, native reservoir From 810073efaa9b7033f7d7d038fec7598f35071438 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 01:29:25 +0100 Subject: [PATCH 095/135] Preserve Mimir Face state --- .voidbot/state/mimir.cc | Bin 108263 -> 110378 bytes .voidbot/status/void-memory-maintenance.json | 2 +- .voidbot/voice/identity.json | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 64b4f8aca93297640540a7b1789a2250e0bd1648..628459937b3bd06a026ede9a0af0638043d8c865 100644 GIT binary patch delta 2377 zcma)7O^g&p6i$<{h`=HV?2?s`GFidIongA?XJ-b@YF1+iLR26|4sNQutEbEDshX{- zo?#5hh$b327)eLwpfYkGs4;-q1M#3DUNpuKj++=+)Ue^e%{#uT-r3!Oi{wQ z+c~{9zw6*4$(Rl zD8v$B6bK^23B+5B_^3{>j0LF^rmBd*AtGRl7vhE@A`_7&iBN+Gk2DnJOfOJoE=Q6u zNm*?oxx6^mAs&@_oic?z1(ypXbmlri4Z(SUREQP)5ERiSK~Mn^O%5eBunG|ziE$LQ zk%xt#1ggi7J$u26Fs(1B=NUJ3!9{f_SV~+Yi18L`@fbSfEb6p%kwFaUBvC4Y#%hAc z%-?Tv)~VE`!ngjV5-_4s1brG{>ppfT!R8?tsEGtdA^O=$c9 zbb^StrYVU$0aPdeymexe2FAl`U8`qpKOjhhV3oE`5C^a% zOjJ>q?I%j_906(}Nh2T&Ogdxcp;N;w zvgj~5q0D(f-Pn*thI&5TGaYMc6QZyfVpb#aEVZ5<_luQEK9?_Ld{S{U1-nEt<)B!} z2hIcs#jwe|*?>jyiVLnXDFy*+M=!_1DhLYyLZlEuTzo{G8eB*>0hb z_e$6aD#^|7o=vXY*jV`1fHIR}KNDd_E<_mqJsiI%<9Z#7_QEI0>05g?XwDHtlOJ#G z+gLSp=R&i6Jo)~gO>Uo^hmQcnvl?QT02VM8&GZ#AkrPxZAhcnsOAu}}h@yaW9ZO)C zz`i~Tx|FoWaI}VYZzOYzBZojJ49GZ62%+f zOQ{1b`_98=q7lJDPUs7mHZeR476nSzgkHfB*r}}vz-8&G*lM^9N)H6C1BixJ6~B@% z`z1RQINoHY;M3XBP-e_t*jOrIcY{pgxV_mX@F=@u@_6l{q}zC7_eDW zSl7_%KOjo`OdWS-p&wnk?hu$^3V^2tyV}+C;{?>u-V0`zU9nJaIs&q#2+5E(w4FAM z!rd_oX{_vXk!2rWCw* z5fl&V9CC;zr=oaLawuMU6g?<-42Vq;QD`d&g&w>(yFKXQasD^&|G)QmFTbR|e@MOB zNgp(4jZ)dN%a)NZ6iRcCwPATQb1xxVRxGzWsqVXselbxSH=WfH-*?5Bs4274^+&hde(({imds zaO5%S5`P{AjMWe)6nqCz=nF!T@BjsXqq2One&VtR0U(Sx_?!i(h6Mx!Q@V^C5Bu}L z(UONn+_l64>Y@rz0uA8#6^CL{>k+xTK6V4Rz!gfk2QE@+s*4DZg#uwW2EoXOCB!Nm zf(90Z`6vowjR-O(QlR-4i=pyS#U7K3{UM`<7gWB2r9>Yj_U zki)o;uYY96kg}i@H3%ma3X{);!ihY?Sv7e*0M5fe?)1-RF{-MUm!QHyK-6&Z=eLnT z7WlXZzSO3NwFk1cF)_2Zq+ErC3d99RMKJ$DG=PtI9q@v3R0|i;#-KsmcBO)9j>sO5ukiQgoy2X~#aM1_jSj56?XGUo khIXiV_n~%Vwd`oE%<8RBYn7VUpJ-{hkv^Pgov`Nq0F~My3;+NC diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index a779194..8a76374 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,6 +1,6 @@ { "status": "starting", - "startedAt": "2026-07-12T19:40:49.477Z", + "startedAt": "2026-07-13T00:24:42.930Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", "reason": "repo_face_stale_short_term_memory" diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index 77c2dca..c077751 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-07-12T19:27:12.379Z", + "createdOrRefreshedAt": "2026-07-12T23:26:50.334Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 83f89d40205a169e1e0a731a2a9cc626b51b881d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 01:41:29 +0100 Subject: [PATCH 096/135] Record full dual-eye Move overlap --- notes/current-system-map.md | 9 +++++---- notes/fresh-workspace-handoff.md | 11 ++++++----- state/evidence.jsonl | 1 + state/map.yaml | 10 +++++----- 4 files changed, 17 insertions(+), 14 deletions(-) diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 2a8b76f..fd76821 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -205,10 +205,11 @@ and camera-tagged observations over bounded MessagePack pipes. calibration, update/observation counters, RGB range, and expected-hue pixel evidence. Live exposure is 0.3 for `nightwing-eye-0` and 0.1 for `nightwing-eye-1`. Both eyes calibrate all four Moves and produce observations. -The latest Mimir window saw all four IDs on Eye 0, two on Eye 1, and two -same-ID cross-camera correspondences. Do not claim full stereo calibration yet: -four-ID overlap is incomplete and the RUDP subscriber currently receives only -two aggregate frames per 20-60 second window despite continuous worker updates. +The latest Mimir window saw all four IDs on both eyes, with 16 camera +observations and eight same-ID pair records across two aggregate frames. Do not +claim full stereo calibration yet: the RUDP subscriber still received only two +unique frame IDs in 15 seconds, so correspondence diversity and cadence remain +unproven despite complete per-frame overlap. `MimirMoveProofSurface` is the first Fensalir-visible proof surface for this chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 417dcc1..01669a9 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -81,11 +81,12 @@ Get-Content .\state\evidence.jsonl -Tail 8 - Nightwing Move tracking is one Muninn provider with two private per-eye PSMoveAPI worker subprocesses. Eye 0 uses exposure 0.3; Eye 1 uses 0.1. - Both workers calibrate four Moves and advance continuously. Mimir has proved - all four stable IDs on Eye 0, two on Eye 1, and two cross-camera - correspondences. Next fix the aggregate evidence RUDP cadence (only two - frames reached a persistent subscriber in 20-60 seconds), then earn four-ID - stereo overlap before feeding camera calibration. + Both workers calibrate four Moves and advance continuously. Mimir has now + proved all four stable IDs on both eyes in the same frames. The latest + 15-second receipt contains 16 observations and eight pair records but only + two unique aggregate frame IDs. Next measure parent publication and RUDP send + counters directly, fix cadence, then collect diverse correspondences for + camera calibration. - Implement the first concrete Leap direct capture driver and measure cadence. - Add remaining camera drivers through `IMimirVideoCaptureDriver`. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 7e34847..c87f818 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -182,3 +182,4 @@ {"ts": "2026-07-13T00:47:59+01:00", "type": "verification", "status": "ok", "note": "Typed muninn.move_tracker_health.v1 field evidence isolated the dual-eye failure. Both Nightwing PS3 Eyes identify as V4L2/PS3 Eye 640x480 at exposure 0.1 and can calibrate all 4 Moves, but same-process PSMoveAPI trackers freeze around initialization/update: one process snapshot reached running with update_count=1 for each eye and zero observations; after restart both remained opening with zero updates. Keep one Muninn provider, but run each native PSMoveAPI tracker in an internal subprocess and aggregate camera-tagged observations in the parent."} {"ts": "2026-07-13T01:15:00+01:00", "type": "verification", "status": "partial", "note": "Nightwing now runs one Muninn provider with private per-eye tracker subprocesses and parent-owned MessagePack hue/observation pipes. Current live proof: both eye workers advance thousands of updates; eye1 calibrated 4 Moves and produced 11,908+ observations, and Mimir received all four stable Move IDs from nightwing-eye-1 in one 15s window. Eye0 calibrated all 4 and advances updates but has zero optical observations, so dual-camera correspondence remains unproven and the remaining fault is eye0 optical scene/backend behavior. Idunn restart now explicitly kills worker children with the parent generation."} {"ts": "2026-07-13T01:29:15+01:00", "type": "verification", "status": "partial", "note": "Per-camera optical evidence and exposure ownership restored Eye 0. At exposure 0.3 Eye 0 peak RGB rose from [72,25,43] to [178,119,112], calibrated 4 Moves, and produced 3,312 observations/latest 4; Eye 1 remains at exposure 0.1. Mimir's persistent window then saw all 4 IDs on Eye 0, 2 on Eye 1, and 2 same-ID cross-camera correspondences. Do not claim full stereo: Eye 1 overlap is incomplete and only 2 aggregate RUDP frames arrived in 20s and 60s windows."} +{"ts": "2026-07-13T01:41:29+01:00", "type": "verification", "status": "partial", "note": "After per-eye exposure tuning and latest-state RUDP handoff, Mimir's 15s dual-eye receipt saw all four stable Move IDs on both cameras: 16 camera observations and 8 pair records across 2 aggregate frames. Full overlap is proven; calibration is not. Only 2 unique frame IDs arrived, so directly instrument parent publication and RUDP send/receive counters before camera-matrix fitting."} diff --git a/state/map.yaml b/state/map.yaml index c425177..1ea4bb3 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -256,11 +256,11 @@ accepted_design: update/observation counts, RGB range, and expected-hue pixel evidence. Live field proof on 2026-07-13 established Eye 0 at exposure 0.3 and Eye 1 at 0.1; both calibrated four Moves and produced observations. A Mimir - visibility window saw all four IDs on Eye 0, two IDs on Eye 1, and two - cross-camera correspondences. Full four-ID stereo coverage and stream - cadence remain pending: the remote subscriber received only two aggregate - frames during both 20-second and 60-second windows despite continuously - advancing tracker workers. + visibility window now sees all four IDs on both eyes. The latest 15-second + receipt carried 16 camera observations and eight same-ID pair records. + Stream cadence remains pending: only two unique aggregate frame IDs reached + the remote subscriber despite continuously advancing tracker workers, so + calibration diversity is not yet earned. - `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a typed admission receipt with decoded frame id, producer, publish/read timestamps, sample counts by kind, source/arrival ranges, native reservoir From 95bed8f5f4a96716d2938011520fecad7372eba8 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 02:32:38 +0100 Subject: [PATCH 097/135] Record stable dual-Eye Move cadence proof --- .../muninn-clocked-aggregator.mpack | Bin 0 -> 725005 bytes notes/current-system-map.md | 15 ++++++--- notes/fresh-workspace-handoff.md | 11 +++++++ state/evidence.jsonl | 1 + state/map.yaml | 30 +++++++++++------- 5 files changed, 40 insertions(+), 17 deletions(-) create mode 100644 artifacts/move-calibration/muninn-clocked-aggregator.mpack diff --git a/artifacts/move-calibration/muninn-clocked-aggregator.mpack b/artifacts/move-calibration/muninn-clocked-aggregator.mpack new file mode 100644 index 0000000000000000000000000000000000000000..cc2b7bc00065248a8018a65e4f14d54d0a1b9b06 GIT binary patch literal 725005 zcmd3vd7RB<|Nm!XE198gr9~wwW$BzVXQeWXZDxp}#lFrMBQur;iLxY=l2W#0Z6lFF z87g}m3}qyt$kJ_zl3nF@&AiWjpV#%B>DK4-dHg=F^UwY0@wnB~^?toy@9TXn=lV?i zGBYDHqhGDe-UHG)4aj&hqiaU5jO;<324-ZX_8wSkfOBibRZ9|EoPBO({y#L`Q6qZT zt2vqdvof->YGtSOO3O^k?l-7bRz`Y{Z1u0C@44pjDgWdDUijZXSTk)DNiYy3bzNZHn)6xVyv!d_I>^e#|h%-x41cth?yfndbS1#_uf{|6?ztR8`|2 z@v<8Kgtfu=KkIEa&o?xF6I1M_=WpZD)<|?V9H4 zZj>JzeKeS`ynlmW-C)<%=J`hEKflaB?M_c`&(Rg@ZACNU^r#5x!Q2HN@Z)yDD{=>-p z7m)t5^JDI66#r{{g8z}v@V~|<_#gQU|7(0k{>K&c|D7+)EyKxw}xdJ|q8q z1^RcJ{$nPm{v+Q~{+pct0tMqwopIZC&iIjU|M=Ywng7$4{iT-v_V@of-_rO^&HrwP z%>QWz$IfTeKb>!X{nPk_`ls{luYYcb?0=;X>AZ)N|Hx&E4f8;at2R=oAqj{eAJ=ns5~{>Znazq$3F zOXlBxQ#YNa)IW{SsDCp5^t-TP7$^Uc&&Yq^Q}Q4AjQp4RH#@%0Ax{1ypOF7?GXG~E z7_^#_{~Di=|Hx@2RN!)e~ffE zIqM&d&&YqNzaxjQ{)?0U$YVBmaR<$$#WC@?X}UFT8PV zE2sV=-%|dY+yC&&{QKhLP4;rek9@}XW&Jt!q2rw2|9O#bY5eBae?IAd#*Tk%Ii>z- zd_w(0KBN9=d`tDur2aDhzcljvW{&>IXXp=nivGyAq`$fP?*l$#|5M{L>Ywbtk1KhC z^8T0GhkQo<1D}%r$Yf#iT*~-0ewqCDhc~PLsk?OjT}I!(HK)H{?w`F; z-@A`9e&jR8FZnmA`VQ6q5cdB=#PAc{=lc`k9 zMqNVQoS;5EIr-t8w(S3TLhm2G;~nxlXZ$k%2;b89&G|p{{@gobs-2?vU*i+}k9>yz zHNGYPo77*{AMdTXIfbJ?@)`OApQ1nV3HpcLUwglM%1ny>8lRv)@)`PTe1`tA|MEeD zZ?|#uM?OP;;8XNRz9s$5^?xquKj-iHu`H$k(fF47ACu#k{g)5F&b*s5e&iFzA1C@R zI{R>E%J?I|GWdkYsuki`_k99k^i#)vNZGV9i03}J|q8uPsxAe zTgrcP^S@i(KlpUx$)z0qkW$OukkJEZ?66! zpHcrbKB4}38+U|LZlK&c?kpEs;f3L3k_9aUF*Z73|M?NF}H9jN% zrT(v3k@_Jg|B=tge>wkJ+vzjT{fE%|OTuU5Ka8J}|Hvogf9U<4b?4qyzkej`e`tJ5 z`EPFho50qa>kE*#`tCa`)Ys149@tG<bd-woe-U+0V;`Ig3SZvV$8^=IQf zYdQUo#wXN&J8#cl=4||1~}#|B=tge~r(` zf0=)_yme+hrT&NBUlG2g{5RMC_+|ddJ=p#?&iIkf7{8pqZQb>s+3@wb2dtMeK4Px|j&cg|SG$$#WC@*nt={6{_`|K^*ScW=j3j_?GIQ$?;45|8Yc1_5DLa|KmiyrSY5ezto?7cdfWjoqy){ z|8J@OKfd_L`S$exPUJKE4}6OMkx%fyOZ@-S$FF=&$$yPc$bXmgp9iiwJf4#O8lRB= z$Y@*Z7w5-(39*y}xs8+p&Wb{WU&8f8;at*Z7w7 zH&_4MQh!grP%ehj|7m_@(r6PW~gGk^jJ_K!{1p9>Z%Kc1{jdD~IO(bxr9G7XSK|}vpZxwd>8f9wtNZVS{MYz|{6{_`|1~}% z{|n~7q!QkB4|DP#`IhqEr2h-{ACpQ}AKRTXe&kykzq$F(U$Fm?R5EvN{{El3`jaq4 zT>^@CkGk8Q`A_3B{14-&_#gR}{BKVGK*9abq^qy~S$+Sv+k%C`gf|CW491tcCfy$W zH9kRqgf~0`Ir3uIZ61I#&2@|m;R&d*ZUvi_#gQU z{{x@mf8<;8ze)WIet(`6^Hqt@IQk=>p+E2``Xk?d{e6yt{r{wLi^ff7=&$qb*I(lk z^w;^8^fxvCX?()`r}OQf|9lSVzstWnIg^wB$Yv zoWI^W^S7y-@gtuwewXZjS9&M!amx5LzNPV-TL1Z6;{QtL=c}vV_V54cd_w=DjXy8{ z_wQTc)%jPSj@zxBe|Wp5dAeWu{Y`EB4G8^@OX^SM+OKus64HZrRM+ni8bt*|24j)^^ZyYrT*6FKHw{k{>W$O4}8%7!?%{$q(Aa4>2I$8 zaRZ;x|7m}xdJ|X`-vj1~`|d`|m&dG(MyL$^J)NR+VO) z{6{_`|A7zrzwqvLHswF^8Tk)<%KVRfM*hqBtGnw~_5E+c`M(GGjQj^a+>R?F@Eu1ohC8r_s@j&kG%hv zBz(sB#eWYp{bUkn{KzMaU*4Zfda%`)eU$NQe8%`?|KY)Nv()|{A^&~IXN+I!&qK$* z_%~<#$Y+dS>R;XcDZ4o1M?PcxQh(yVEnkx}e&kykzq$Qiznp*73;uM5vi{Ndg!K>d z8|7=tKBcGA~ zz^CLt@)`Lr{a1r|BR}HgKk^y*4}8e~1ru#L|L`N3=o8rTP~hzt7?B;&(Vw9sl6>ml}V0 ze*ZUhSXA(#UejJRs(+a8?e1~KrMXg_|9~HQ|FQ88Z>aaT2)e{%kwd|RCQ{TreG3%!3Se1`vE{OCVNH=F#2e1`vkPw799 z&+s4eE&0!6|0n%_^6J_XIQ0+tjQAtpQvAd7pP~L+);~?(Emnn7|B%n9f56B3_wFE@ z>L2ng)jw1Et7YQWimR3+HJd;55r+Oc-+uiyKIp%wWK&!8*ZB)f5^AwKa=%O);}%JFYqzeAO6t$zrtt4 zAIHy#zx2PYerf+GC;rH1#2@*V;%{#LamxPps_4VKjbs~2YibEkk9ZR@G1U7KEr>=XZTOfpW4*u^%lo} z$Y=Nu_!R#kpW#2?Q~ZZ~hX0Vy@Sm(d+E%W3566GVXZR2J6#pTg;XgV5YJ2+5RLc6# ziF}6tkk9a+tUudT7)|;8i{FWShW~(1@gMRT{*&KdwY%6segD$_`EZY{?qsb{~^CYiR(@V6EfEv3DymEU2V_)i^eDTFZBJL_LsX*et+)Q_yqrH ze1`vI|Fc8O8_#k4hkSW$O4}6OL$Yyyz^CYse1iU=@6UA^koPS`e~nMj zANdUZH9kXs=|8$=HA&*=k9>yyz^CYsd`tS9yZ;mV{#@5D-xyEX|Iqk^^_RxCwEi*~ zf4TqFZSjOfocJT35r5<};xGHpsdHm)=fofRjQAs;5r5f#Oq+hU`uzoA|26deLE$sv zkK?x#e{<`9xBUL4`)fa`@Bb0j{~Dh#|0BOai84*q{_C|1~%NAfMqsjZg5O#%K6X-hWDe;lMJE|B%n{AMh#uLq5ZQz^C{R`3(OdpW#1Q zfAx4a`#FyPkk9ZR@G1U7KEr>&r}z)~4F4gY;Xk?mno(~7<@?M2(D%26&+s3NpW;8{ zTk@aD^@ptgdp5c2MNa)iK0|-tHz-lQy;^^~|4p#&tZHuy|7Et$nnkJq8lT{Q?6*v+(|tA^oNQ z%>3hr3mpBC&(I(E6#bDO`(QxnUvpG2e%pmwM*SbIKZf*|`)^q@es*%|Kk^y%ANh>> zk9yJL)=Q%j~BcGu^ z@G1HuKi2(?(!a*b!T1fg?=z~u$@&j`#{R1Z`HcFHd`A68{_>kg%J&cAlRtXL9{s&? z|DfOcDU|Q8`ZYd5e~nMjU*lWS-(39|Lxuh z#&2D1;r&-5{l8b%AKBMl&gSTke1`tOr|6IT*!z|${adN|fAfdm+f@H${gM4qwAy+k z-2d|;pHcsj>%XVib`Kl-12w=c*4$Y=N;_!R#mKekpA<^M)^tM%XWXN~&bGwMI`bG}zi z_|(k$!T7qaw~gv=&i_7H|2_5Xp${qk*Z2hgBcI`ajUQX9m-7GQO~Lrw)pL#NZ_fY7 zXZT;^6a2673I5mk4FAjeWAJr#9_IKT`3(O9pW=Vy$2w*!|39%_^?ws{jr!kY{g?BH zAtlfJ!OR(ys0-B|MlRvY|>xeKX~SJTs4mV$Y{pu0MClP{EvJ_|F7{0{@3^f|7(0p z{x_+={QhXz>I&-W8Dan5k9>yy$Y}sOe1`v#&+tF;E&1Q% z{4eK^&u!nS`hUj!k9>yy$YT$|)PIdn@V~|<_+R5Q{4eJZFV=pZ z^ZmI1@)`aIKE?mYkBytE{6FRewf~>-i%tFqKE?mYXZRoa4F4ma;eY8r#+7NcjWhov zpW%PtQ~ZzoSl>g+|D86d_5Z=AZ1I1Qo4mtmO zrQ8j#F#NCc?f1XNC-`6I$2$5b{|`Q^?*FY^X_NnD|9O0m66)%y{rf)w2l5&IM?S;< z$hYKwQ~Q4bhwMMT+P>Qsj{eAJ=#PAc{>W$OFZYjLODX$2M}Ooq^hZ8Jf8-PNcS`>^ zVbVPH{tu!4YkY$K8lRxQ#%Jg+_YYow=~c@8zkn0@4E>SM&>#5>{iXkSW8^2DIQ~aI zLx1Em^hdrW{Y{Pf*r>8lSNKLq22ur}1N*`S(AcS)ks3yZNHg^`A-o z<@|Z#rW@|y=#PAc{=lc`kNntL>y-YZX9eT)il^G7zwAFw`f}r69Q~2c&>#2|{gEH* zRX@U6p0QA!|9*a#J^F{fe?EEf$XyiuH9kRqD$4&=)cYTs-so(T|A9~OKk^y=M?S;<$Y=Op)*sV% zZ04;0T*znmANUmiBR{r@`Uu|g%5&83KPUZalmCHF@jvnz{zpE;|Hx5@`3(J$&(I(Fmh?AS|7HC#d!F0HssG4l)PLYp z>Ob-s^Zna zzsdS9>yNh&%__^O|Hx<5f8bNc3mo zfA7|M;0#B99ye{MN{n0xJzL6rVa;}iNnZ$$jzXsgMXZRoZ z6#pZi;D3+o|9p`2c@(Ap*Z2hgYkY$LH9o=r9=ZSV!7qEuQ~a;-3I0bu!~YuJlK;)E z|2%U4fBwlP?{M@-K0|-tQ}jo^CH>9q|9WKqVZpuEtMebi`cLCC>OYR3QU8%|ss5YP zU;595mA=p9=#Ttsv(zJ`oBKQvtasI~gU#Q6_Bq^L{0>K|V^@{T{#hAWS+%n=(tBhN z%*aZwo!NUpTFtZp8L4Sm-O_5i10Liv^hdrW{ms>Xugw38%Cx+jQvWr6-X=#dA=gt@ z{r;tHIivcA$8SV`S*Z5@{t<10gdBJ+cuW=ale|Y?c^vCft^hZ8Jf4P77@$U!Qar8%i z*=oNk{WFFu|1Uddlm5tO=#PAc{&N1XxOTVSIQk>M?8`lr{v*B()+^Eb9h>w=K0|-x zThc$=|9HE5TybfxROdhFf4$OwF8(3#A*cUCe%_gTRsWaryXyabUS?GP@c0eYf8bO4 zKjbs|KiPj?;+Qm&lf76pj^*1^HBcI`aYv2)%yqA=h>t`@G1Hu-;(~}`OlF5 zfzR0g(e#hG(XZzJ7qiv;fATh?`iIAFr2o_Sg!y0NGv zMD_olJZx0|@cd^;f8bO6k9>yzflu*2@?+YmAE9razhC+PrXP*!Z_@wBXZRoa4FAji zLr$yVH8}o9e%_Wb%Ky&VYW_d)yHWj3`XBfd|0AE_f8bO6kNlXi>PIM>KOU_7f4sX< z{Z0BG`3(OfpWuJLtp7hBH1|o0|22Nz$BD}S-V^HkS8v~9RDW~+M?S;<8lT{Q|laEtQ)%&BVr-}SUn{Z0BG`3(Of-;)2s<2SPZi1Nq^)s^hdrW{loo_q53c9&ua!H z)Zz4h$j`f!uKK@L>(%~G&x1zw50Br-{;wbSl>QI-g#Ire{l_|A{pu9|Yy6nL>LWCp zd*2qU7q$LhM)eQ(KSuP|_yqrJe1iW2(toV$y7x|s|22NzrH7RN8+;Uu|El{dM)f!6 zf8;a#ukjiFm;1--OE#Fw@jvooqSTL&HxG*n){8DV!l?cx{f~Tx|B=t|znnjPHRW(c zj{lLrqj-+;e?qrl{F+(KjOuUF|G=mCANdUb%lX4sCpVPl_#gRYS2oUH|G9(lfBj|Q z`)7vcf8;a#k9>yzW&N?C-u>$81>ycz0Qo!0sz(TOn+{g~uk@2m{s%tA|H!xG|8V^? zvi=Lm`eUQFhMS{5^7GWxpOCAXfcW*T?dt!4PthOwmh=zzKc@93?Eh-{S9VSxRsYAg zQtQ8`)*AJHc>IRyKg6H$`%i8Bmg;|a{Qq2kkHaD7|H1pVzDJn<{|~?H&SgsfNl&Za z{|$NB4*mZR-}d?c|L__5%lbdKn*TjSLG4;%N|q(AZ*`Xk?x{$}QX zm4CAS+|>8MNrd_T|M0`}|GK>GM)ePm|DV@?op1mA|9|+F=Kt{c4b^|S|G)Xu1HC!= zBR{PFhIWR3{}&#=A^m|*(I5Gi^fx*G%lc!>pO=eq>Ob;h?o&TP-aNCEs{el^81;X6 z{D$;LKBN94pHcs%|IKY4{cn!{k)QXfn*VdJo~q9O`js)Nze)cCpW=VyTk?N+{6^}( zQ_lakHki+ZyZ~%h4bCc|o=Rn|o(7 zrT_X~Ht7$1ivGyAqIi}8lN!# zYkW)de|Y?c^q2MD*Hyggr?#7h^fx*G%l*Tz z$KF_}YrT^$Q>i+M& z@7kn4@)`Og-;(~}`e#Ug*?-u+zU@9v{YQS@o;Irf-}tlY|IaNos(*O=hWrow2F3I5 zpJl55sr%BltF7_9UDG_>ulW9+>VJ^WsQCH2&fGXI6jv z&wq9O6Bz!N{inR8tGjdjkNmvvZc_fQbWHVsPc|^Bze)cCpW=VyTk?N+{6^}(OZNYE zbga6HqQAz^%NwHfuTomQ|K>7$|35r_BmPG|Lw}8LN&j&DGo-()KX+E#olWWgT*xoG zOI5xE3H_foenS0s$@+6=lPLA{hVA!1Jr0eZ_w8Bb|H@CR z{_oRkjp}dG|1f@v|B=t|zpOuZ9$o(i$N$Ke-+$!a|NLvjdq(v)>3`%i{EvKw|K;}w z-;JwZf#ZMV=WTyp`Ty2z<^P9mx&N)|5AZ4eN4_QhhwGoA`Crywy9Q@J!Owg#WE$JVwe}?pz^~bKfF{L>5ANetl zsQ!PmCrR!9emvV~{KMlnq(AZ*^&k0!`X49vzrVlx`b`x7Yy7+yl>Z8T|FN!Yx>5bj z`5*ZV|7(0p{tu7ei2vi{{C)S$Key%RkNmtb_bL7J?|*F=@TN`r1D~Ql@-68fu78H~ zm;Q5a%;Ik;^*;{zWv8e^h|M(`sr#QV95$+dc>G4{zs4uje~oXc{)fkJNPjth+WW~J z>ivJh`ClCJ%l6u)^lz|P=|AO!P5L9Bp+E91>2I?BOaJ-9d&A~%>Ob=HK2#qe&n^9v zs{hX?8Pz{Lenb8TKBfL6pHcsTPw_wU%l2NR{NHSW@_&b~jOrh*e`fV(od3n)_!<6} z^~aBW51r%qANhF;k0}3_8mHd>88z9c{wDnoe2V{(Z^{4R@f(@{-E#i7ukOog|Cg}; zukrIfenjbCvX}b(|4N(o|B&CH_$zDF{*O9;if`7*;`!r0-T%?}mh=zTKSTP<`eWaL zGF2)4pBwpQr>gfqHn+T1o&VLa>Gxk6pV0qld_w(q%lYSi_nqZ9{zrb^;;zd7v9ap> z|6QAY|BK_N_#ejKz>@z>)_*yFI8f=GFFE=nKX13Xgq&Nehx+|r`BV1H|8C$@^hdrW z{Y}ClK@f+#?-Nc2z0;`v;reGt ze;hx-|JwK&{+IRF;mNgs;`kr=JBq1ANN%&ks{dR0tx^3=`XBfd|0AE_e_4MV{_U=Y z9RDLfrouGk{}Ep*|6e=HsQxDXk9>yzkyzh{zv|fGVPTAAALft|Nk=l z`yZ432Y!R%ua8vwe{to4@$nUU8qGhQ|4;Y-JjiGGU)CT0zF0%`e}wa25Aw@y>aYAi zcA`4}saeZt{7w2F`3(OfpW%Nwe>&3ih-$qF`~M!~?}&az`M3Pg8wx>!T%bc;eT0w{Bqf!$MHY%8U6=;gW?k(RQ-Q-b^cUm za#L&l$3N}=y~ttp7AVqyEGADfJ)ujQTJ44^B>Mr1t*^`#)afGyIQyhX0XI z@V`&if2YnpH=5#qjZg4D@)`cu_yqqWpW%OvPw>CSC-`6ETk^lT{?8}pkEajMT0*J+ z8lRy*j-R1F@)`O||9NKYb?Vbwg!$ixe1`tWXXuZ7OZuDZ|9#Sbo*nu#=lf3@pE3W# z_!|_Tv{>!`_-_ivd!{U~Xa5Js&p7`ig_>wo^K{u-a)f8;a#ukkJU-(3Co%l-F@C0nWYpTqk9y!EV2`XisAKkzB~ zBj1w#;reUj{72KDQ2#YPq5f-pLjBkHmg>Jr{bm2@;@n5o`~M96k>5Bvtj=d1VUGMiNk#y6;V#M=1(>HQx+@-5YWllsf~ z;~xiSHbec7-u{KP@&Dug5BUuJfltvN`IhuIS^t4gssG4l)PLkN>Ob-s^St8{aQ|Us|5xJ^{IBr| z{@3`H{2z|Lk^P@Q!TK{fD(%umrT?3Ul>R4@?9?Cm4E>SM&>#7h^f$Nu3l#kRAUXQ# z`Y~AlkHuYM)c;|=A^#(vQ2%lKjQX$fE!F>U{0;fPVE-YxSbXB6sQ=N<2W`?H`3(Jm zPthOw4E+o4A0!t$wd8G0|AjSe(jWN@{ej=0c&@qxm)Tx@dT?$nzqRrI)BV2y@)`OU ztUr>k`g!@+sQ-}*S=Pq?PuKqe}vG)02=iBdpoo~PYbw0!Y1^17V zOI9iNu;%}{f7s-IVG)? z2KN7*PFeq5^KBpX>7AS*t(5-jn=Abjirb^V#wX~n@d^5Cd`tS9tN%{vKg&EY$%Xa* z*aH`h>L2DC@xR6=)PLkN>c7UfRR6>AH{^fmKd#NoR^NXQ-~Tx_I@Turkk8W1;|LP`c{(sBx z`EQtSsQx3LG5;f#isQ%{suki`~*Z2hgYkY$LU2^}h+=bfe(;G1V|7h|1TmSI>hg0Jd{EvKw z|200t|EPaXvzzn%-%0uZ)3rAFANdUbBcI`aeK5vi4Bzg zE6*$aR~ugch51J2e~r)3AIHzoANiK_5BFb&^q2MLbw3?1uhsvHrHuMN%r~Sz@)`9X z_>}sOd`tB|9DhUl%lf0jZQ1JkKQRA)GTSEokElsG^$+un%>Npn(En+CLjR}n3H_g2>i;dnpSw=;|LjFZ^$+(S zM)cSC1pgzS;eU-!@IUg^{(pA9|6R)ejh`^8zd8SFe1iWqKEeMQpW%Nw|GzbAXKnQV z@#uX<^*8B%aSafNl`kVAW@)`aIKE?mYx8(nD{EhVgZn=M6X?G2+{vV6Wv`K&D zGxP^OMStX5(mz~(P3xc2d{}<{zf0BsjXjO(ALbkC|6%-u?|(S8{AbjE9KWUdACA8v z{bm38wr4v{SMz_fn)&+Qq4d9K)BPVe@)`3#@)`Og-;(|&>wm%flgYPNo_?=3|99wU z)c;|=A^#(vQU8HYssG4l)PK2uRN0;WwDNzGiuwNkmpcFLa<@_a!}Z6I{>W$eANdUb zBcI@ZkL>^5d2gVX=KrN_jOuUB{~Dj*f8;a#uki`~M}9(1qy72*FR%PRsDx4d&G}#B z6a2673I5mk4FAjiQZnae|Y{i z3X9u17@ALbkKKk^y#KkzB@Kk^y#zuZ5(yG>&L`hWQ3eE*jY#-CsH zjZytg)_>$P{EvKw|B+AdzgO1(_smHD9Q}WMOb?^_oAbZMC-@)v4F797e zf8;ajKkzB_ANh>>FTX!<)IQN9zy25O|L5=j{qcZN{loRgkp9SL_#gQU|0AE_f7yR< zywrS@=Kt2GjOuUF|Hx;`vXVib>GwMI`3H9GE>wnjQ^4jme ze<@kVsQ%&lV?=+APw+qT8UEM!mi!-%zY+iY<@`BrMXzO8|BoLw{Qhs4Z%BXSGxP^O zMStWo^q2Lg`=j6f#nB)64E=#m(I5Gi^bgO!hWrowgq*HX|C#?sE;U;J!+b;jM?PWx z_amP$|7(0p^M5%0hV+;H2hSt*nydbgpg-~%`XisAKk^y+%lgBY)L6Ct4E>SM&>#5> z{gH1;|8W0h$p5nb@V$BN1W5s{Q=K)hPXc0Qn64k%pvBcGu^@)`Og zpP|2;KR<9=pWYn(kc^X>P)%OU+|-5F z+wXskPw>CaXZRoWxBvV{8$aRwf1S_pzZUANiK_H#h$~W&gS1pTmYy>c7S()PIdnsQ((@QvElnzwAFZPWUp3qd)Q) z`XisAKk_ZSM&>#7h^fy=kU2^{@IlWPU(*J9GLj6ZRqyB4rLj6ZRqyB4r zg8wx>!T%cHlK;)sf0vv;G)+6{qtt(m&(I&o&(I(F4E<&O-|QOq3XcBBXXuZ7hW^NB z=r8NfW`CVhzyBtj|G1FP&>#5>{gH1;f0O;6tUn%Z@n{UE|3f~b{{udy|3ki|{?DZT zvi@k^u;(6*{>W$O4}6OL$S3F@C+oi!V^^nB^w;If6q5sqPmij-F`b+=Ws^SaxbM!|(Lx1Em z^hdrW{mu1%angT0((%JrIrSg;jQS6JO8rMZq5iw&{@0_$>RzDuU*i+}uki`~*Z2hg zyQTkswAGMl6#r{{g8z}v@V~~l2I$8)AT3o z|7v_j{fGD`FzP?@8TDV*pN~I$QGNQs{`bFJZsark4}6OMk>Fa2l7g(bZl|0AE_f8bO6k9>yzrT^@7=4I9Y6ZU^S$Y=N;`3(Of-;)2$)qk(_ zf1Qs$QJtc{#wX~n@d^5Cd`tS9>;JrR{?Mh)-p?uZU*i+%Kk^y%U*i+%zgO=6bt`lE zcaHy&&+tF;8U9B;!~fEMb{lwodyfB+&+tF+DgH-3!~fEMcH7wU6^{Rr&+tF;8U9DU zCI6f3|K$Dg)X$r3pHKRaCpQ1`EXV)IXZRoZ6#pZi;eX&$ z{EvKw|B=t|Kk^Cw_sjjSUbhYCO7XwOC-@)v4F79Ud87??*nv|G=mCANdUb%lb2`bn}56|0AE_f8;a#k9>yzW&M}cdc|JO z{EvKw|A9~OKk_a4-`xJMU(O%;ymo&*j{eAJ=ns5~{>Znazq$P%zpOv{9B$u$Q~#0A zsQ<`k)PLk#s{iKn4@m#tceisCMSqP?&|l*d^w;>7^fy=k1JZx?tML}+{a1}osQ<`k z)PIdnsQ&@Ef0Dgvb|sGgkRFeuLt3)&vv!Rroqs_tm7u!Me|0*gYjDvwv1bR#vU-v|eeMY1#b-)ym39?~y$) zBP)I3fB&gw+Mu+W;ri=yxVw0}rg^&Wsv`e(?f?GC+L^rvq}5Ctkdc~})h(^IN9SAe zf0*y>?s3JXxl*0~gfII~PuDM}`ak=B{~zZ-K0|-tQ}jnZLx1T%2aoCAf}=n38TtdC zqCfI2>2GTOk8=Q@F#qd(`{#e1Z~y$S^BMEMtUrcSpEro(f8;a#4}6OMk#EWWCg*?Y zKcA`Wq@4f7IgroLANUmgk#9+Vll33?l=_c+M*T-Vqy8h`QvElnznnikv*roa|Jnck zN1Oxs4E>SM&>#5({hf0D_v~l;)%(u`{WU&8e~nMjU*lWS-`xD~l>0|RFW+{8(*J3E z!u*eX#{94G3G+YlDgMX&WB4EW4F4nFlK;)k|4v!|51;r$Z%+M3K0|-xGxSHkCH+m- zf9XG-tF=LW|DCY^>qI`I{sW&<|B+9q|1LTIdG7Zz!zupP_yqrJe1iWqKEeMk>HkMo z89b5Ve~nM@Kk^y=*Z2(o%l*HR>sM9g_#gQU|0AE_f8<;8zq$U;CFhUNFYiiu|2a9ye=d1{ZdBO{E>8VNKBN8vpHlykZ>j#9)L+&gql-Oy9Y=rUGxP^O zMStX5(%)qL2R>u}$Ax@G{YO5d{v)4J|Kyy$Y;FREUw!4olMhnr zzs4ujf8;ajzs9#z|4r&I>yKAGv8f#Wkyy z8sC!s=H`Eo>_5Jq`q?DP{IBr|^S{O?%>NpnF#m_XzxT!;3$i)>m-;7shW~+2@jvnz z{+Ijbliu*};P@Z;4F4ma;eX^a{4eW|$^JnPaQu&ahW~+2@jvnj{ttbBZ0eKGtKa_; z?*D3hg8wx>!T%bc;D4{=-}K5uKco0x;}iUke1`uuKEwa0Kg0jXXZRoa4F4nFlK;)E z|3crNnepWDn<({P;}i7P_yqkmz9s$5?f>~?{Xb(}mCcm;uki`>ANh>>uki`>KlJ^P zSsi9B;rL(fKM9}Vf8;a#k9>yzW&WA-+k{yh|0AE_f8bO6k9ti=#PAc{=lc`k9>yyQh(lGR%ty)f8;at2R=oACSC-`6EGyE_6Kl4rpR&)H1e1`vlPw_wU8UC012lHz$=G^}dAfMrX zc^X>P)&bQ=$Q~N(|hxDHxb?VWR zqd)Q)`XisAKk_Z zXZRoa4FAje^W$l^zQFN6@)`aIKE?mYXZRoZg!3P_1NjX9BcI`avr__Js6Y9TH&VQFw>p6|$e~nM@ zzs4u{U*i+}ANu~l(n>RmQ~a;-3I0bu!~YuJlK;)s|IqvYOV2mw)PIf7&|lVH1>>ja zk9Ob-s^&k0+`j31@{g?H}r%U$K;M9NQGyD&HivN*s$^Rzn zznnjRHg82cj{eAJ=ns5~{>ZnazsdR!e8&Dy==(F0e~kK%<7d==%Z(j=8U~SdH>1nL_R}*#5>{gKbmU-loqc(URVj{eAJ=#PAc{>Znazq$E8^!=4D)~<`E%>Npn zF#jW;G5>3POY^@;{iXj{xuE~89Q~2c&>#2|{gH1;e{=JHoSgr!YVms=O8wXPg!-@X z3H4v&6Y77Q^dGAarmFS7{oj9fYkY$LkW$O4}6OL$hV}wx%Hn%`j6b(UwM|&|7mCSXZT<4f9`%N zVLZqG$Y=N;_!R#mpW%O5f9|>Ye07fhk)O`{n$3-+`+0DfM6D6ZF^k1pPI>CH>9Sf4}TM>>qVsaZ3Hy_=Nh8d`A7(_?GIw zN&V&bUq3y4i1YhDKk^y+1D~Ql@)`Qe{quvV&!=+yk9>yyz^CYse1`tge;oS2$=Uz& zBcGu^@G1HupP+w0`oF{PSLgU&;}i5pK0|+vZ%Kc1{eM9Ezrz=QxR%oYYkWfguki`} zzs4u@{{cCF`nhRq&i9`+KEwYoev1E*Z^{4W`u~9JKOY(WklOzx-2c`14E{|U;57@yCxjt)PLkN>Ob-s^&k0+`Y-1XM~4he4F3b4;(z2b{4eW|qg$SO zn&W@uGyIQyhX0XozyCcBx&M1?^-FmS|Lc7F{jc!}{@3~T`(NV|&i{12{r=ba_WNJw zGyE_6KgZu%^B%|l$Y=N;_!R#mpW%Pm|M}(WdsY8u|L?y&4&*cZk9>yzk_7a{ zcEc!+|B=t|KkzC3M?S;yzkA^VTNIu5_UssG4l=#PAc{>Znazp3*dk3;q! ze|>S`MNa)kKBN8vpHlyk>h|M=^kl*c*#M?S;<$Y=N;`2_zvW&hz+{IBr| z{zpE;{~Dj*f2Zs}oLYA-iQ<2aPw>CSC-`6EGyE_655KP+J&WUi_438FgBN? zKk^y+1D~Ql@-69aZvRKqpK$)C@fr0W;-A2%|H!vg|4r&I`%h;U_OH#E|B=toANdUZ zk#9+Vll5QrpU$@QaL#|6$Y<1l;8W^9@-5YWbNajF{@=ODm-bQSe~nMjANdUZH9kXs zS$|%5sB{cRf8;at2R=oA|n7bM!CK^9dVp6cuNbG!JZkb4oNp94f>;nY58MVbNx2O_(estYhN=%#uyQqmhJHDh&(F?~UH!5XREk>?5C5JN~ zY&n<{7vvX?3!`6g%ZX!R=Dr$~dC0A4QaC2LQL}fAXXJ|PL?`ySC8;DQR|-}_iM?00 zRy$Ee#ii|UlPPg2xFMO?=ZlfeBQq|4Ro_pEi_FjcF02?viHq!9Cia`UDgWkL5o^Sb zzrMtXOPtIF*#`!#=EOyOlpSB^P|C!Y7w(M6 zxI{PV>SDyjBXjidIrnU!#Kj|P=;0}&Fr+-;^J{gzRX;C zBV`WqIK;ymQ75;`Y(zkABeh>Xis_vd{`iHp?T4;p;C zjS?4^)VKM2ek@z0kD@1+evlEDIPuZPO-gBB7__U`^~6bB7M*>#bCGe0O8Ja&@5keo z_4DGqWdkYsqVWs&FN{4Xk6Y$~rI~l{;KW4sD?iS=R`U` zQgTK5|7Bf&tH{X}scXy1Pg85v!ZA^2XOHY;)Z5Tm#PU15ly#9u;}?z#rLK5nUv5Q( z^CKv^a)p@WMx9x;o{}r#p)VWNpxh<&c%-gZX5aq-GLw7Tlsmniw- zm3&>b-SbqDafwRl{TwAO(o3!F^cko3_e#F3S&{l-(Q%p6>UTz5d=i(9_pIgQi%;UR z;al(PMaLy&Tsk8z@^$mX%_Z;nlj0-4^w^sg_x-!GvxSu@agn;a?Y_6wny5!!UNB)= zwYkB%!LF-q`F)kgFTKmwT`#LMnZk3l>RoQ2?9l~e4ZZ#9o=Yfk3GM%X^XYd(|0XW4 zc^Ua4U&&0&YrZXq6Bns3!Y`aJjGoRbUoK1hwt@E=PE5qZJ7%S;FQ*iaiMq#~JCcwq zUWfD!-wj>bKA4a`EK2=$GAb%qcbj*-Ewx4G7mf=jSEQfYb?1y_kr)$|D@Qs|az*BW z-JR3a*WVerB7M{DQloAv+C!~>OlHI-^n0E?QHMzRa;3Oj=rofO7s;1>cdfWj^>g{P z*IA17pQRqD6Sk13jshE6!oAS^NV}fe=tid*g^9ou&7U_{2{s?Vz4M~RE{-Y35wT|FY>5|uKLac{xvkvu-r zaHU#9r#+}Hm``hx5sdG3Lm7K~r11;;h*Dd;GC%&2cu@V`mk<+;A1N`Bp5>2UN~`Wm?nUNH)SOCt8Tlf2%#*IZ`e$_ywA+G( z!Gt#lT@1#TyC&V9xC9FJ!IG|-wOFBx^wE|&l(-c9&MN8Z`DYtPWL%<7f5CWj&a3Ga zdHF)==K@k&%6`57F;1>XZ7DmfyE>vO!b4FhPmiE@s9^6sDdwvZpG9O`q8m*=NQjG1 z-ttN+w`kmSMqG4$;e26u$S3csC6#}7awaDxQtyQyAu(|koQo$_IzL}6o9(Z)KDk4l zROy|({I8A{@peng?-;*7^T~S{Nq5D5q26sI#6{y5jtj*#6;#p&*OKhJN}HAAU{H4;+MInUhtTiBy z)}`?WjJSmMY7?sl)tdo?xXAYnlA3Jy{Zw@SmNJ2HcIcCP+DXZ`#dYAsMdoGUM@Vf6 z$eOI_yTz(-Vj><+UR`@a(YbQqE5>`ah>aV zMqG4$WO&FgZ~7#)EMCpciHWR%gdZU>ampUTD7=)A6d`%axX`6B22?Jjmu=lz8}lpEDG`h7}VWDe@k^2T$NxP;De z+h6W7spz=0T-uou7pZw28;w`raY>-MfDIhWhUXq^m>@q=Vj^qTjCu0q<4|;i6uQ*@h0`#Ou}5`m3v10)=yF2B`h2h^`6|YR7S4IJDN$^ z*IuUl2GK9?h$dx!6kWCGxXg)WeACP?ztd0ZfA-zJoP3d*`?qn4ZuAc04HCasdY6Iw z*MG-}3-Sy5h_Zf`@82ao_3a_`<|?7KX#7ZtiS#UkudDNLB*rASSe-{_G3u>+Yb|L= z$uoa&;v)WecIiv2i>$ZB7EWZmv+eiE{`NDc<0!u?^ZTw4m#EckrZVy+^w!1Dy;IaP z8-&`T@gtwWM(cN?fF_%%10VapEF-+Or3p_?tWJj)u9E zxQLJ5J~XQ=CoZ!0_4dd)2aC>^&VSy^h)d}Gp1Ic!8AOST#xGo7D06e@{hWEx9Ri$~ zNUjJ!QeqZ~4fK%bYvPQ{o~$ z|NN6p-r>YW_HP&5d%ddnMdV9#Ajo*%H4wU^u&7MSyD4$e_=V#_skdIKcZ+`rP|j8Z zq4Vw~j!7en%$2B?9T>lp353pkmyGC2dB^n1aY@;9Go!Znr1xGr&`bIK&=upN-o3p@ ziHr39pY`bS0HwBs&b@^nA+<%m37fR6t*JJ+ZPC3-i?8-G=AO`7 zRO`C#Rd-(rd;9^(m36-Q>ib}Yb0s&b`qXojxJdo~YRch?oVZA>{p#d~(nZJR(fz+t z;v#vxq2B$+A~P;2=?5rrk@~XHTZ8h(Q6PZ)!agFb4?T|1TUeX=KB(Szu)Ve@zN|Nb zTh9Jow1?(2dfWb7QGYM#_cwofpf@Kj;^8fSUM^N-T%sG@GJz5o$>H4Q(f^LfxJ0EC zJIsiSQ|8F64JNdt#3l4r*tP~c#}=J0Q4Io=xQM^y??tFDSBMLx=7#Rrd_DHY`kY*m z+V=IH8LFQwJP)b;{?Yp=xgs@h`}(%~IB`LK;ka;eMe1$d($(tQdkp{l@0jF9{hB_C zkt?D5U^_ZiT@{gWiJr8PuvYXqHFxLH_3HNpMa5;#;9-=wh`)CY&VGUu7pZT%^2U@ZIxcOlpT>ww=v}Pu z@4kLhWX2_B>{?1(q@UY;^UrFJj4&U$u8=R$jXJfW#6^7c!+XQ#P~sAY{KEBx(!0co zRzLPV#CW^Xs@h zkr|hi9@Qyvk>24*(<7BRaSZO{1dvjcA?DP z*};U=DD~rvH-|+9;}5TD8?5{6h22wfGW%y`WM$RLPV1GHnU>veP_3+t^d8v*GqTbb z{`a41rVUD~>0FfAdq7%Ehr{9R?s3JXxl*0Gs>r`x`@es(cEO)X8<3Hjmenn-w#N~A z)93v1vg*yhNREur5QsN?KtrttSQhWd5xct2JtUYlFz3C(T z!f~P06~D|umric;VoZ)SoM}@`WDfe{;LK)4=gR4AH&Sv1|6bu-IePmS5fK+ou1LN8 zYsg?VFGEZQZMP{V$d45N$QtBwx2+qsn5>y+PfP;xzUk%F{~oA%xcTZMB~#A!561s- zsb2vy*`1q}blyHv^1SJ#ihxEzbS#-_Lw>}4cJrCqwH;_~Jp6_*o97?(MX{zZvP zLA^~b7N7Vi#^q?|gEqybz(dJb{k;5Z6_=;fBdb&PWIK59-+RYa=BtjS0V#_c&d$ zE-81R_A^?({P?XsanblW&2CmcTGdJU=+m_Yd`j;k@9rgE_tSCpUJH2WVkw(qQgAku zTs~>u3{_ha8>pD9Jg;K1T56lBE%o{_YKu#1UWMDT)j1-><&)Ww5tpmBZ=B4Ci`=nH zuCS`V+Vg<8Y*=qkT;inW-}LbE1u9=!s0;Q}RyGXAU;21d0YCSummB#gago|`!_ua8 zwYYpgAu{4pY;y13jJU|RKa+16{@it1TxKt_CoUR4=TUXReacFg%9qAZ6!0lESMCxf z-x{^EHpb+5^gf$nBDs3&z_*qv54BW}qfA*@UBzVFtO9;+)PVJjeJH2gB}~3`<6}KB zF2|xvM@U>!K3h!jkgQ`X?XIEa%dxmjo8p4}oaV#w^W`p;FB^Ln@HrkTI3r5Fz4G*X zwOr}Y(WaQlUh-|vc2aLqbmT3nX4u_rDX zKc~_D{J4}?{oJ4uyj+nt36rZ-3#hvS(6elcu_-1}Z>tO*q1u(4hUzi&DJx@DOkTdF zfS()HDzOtKSEQG^`>JYRXmRP$z^1rJk9~KW#HUqUhO0-~r>ra;j6c8Xn^0UHyJH0- zE^>!2`JNf+pJQB(kLh7gTr_@8R@eNvlnlmS7%lszlzQux9Ikd~T~&<9vGkx#F_CrR zy}pymJ~>^~2g;|cEFO%%I4d=jD=qIMym#tx%C{Vo@BRL4Yb`D>REm(e%qcmP;vuOm zH7cA}OM0leD~8$>7v$%(dp?vemu}itz~^{K>RpXxwO+%R9KYd2M8rhx`+FZ|x;E1+CQ_WRwC3|m%JsJTzAIV9~g1b`H_(;uFyTCgePli`*L>8KQ4#lYC`4xbBoTE zl%^FJx#E<0x#0)z9j4@pyaAZp@XulD-d^EcQR|`arcmM{dmW7rU(oiTY|9rX@(ag> zF$cNit-0jn^hN6Ey{$^KW%cKg)*@xmU;joxOERzs#mW?WJpn8U~ydAlq5 zv9@tva^fQUU&1e3UnqS`=nmB54_{Q<%J%OkxHNvG#63ql~-)2qjeDsOxMc0>mp4r!Wuy1N5~pL-Ze{pV)HN0a&kp#oA4tfCZT&Ry>1(zt_U;oUHG9+SN`DiV&nSIMKzuadx>8SZ;)49b`NG)G zjSIb(HL%UUw0BQ!+gs2yDIAmB=*DFe2_A}b$XxVv{c23>yr_NTK&*4*fG;~P`rB4=l>dSX*4 zaq-CB;OnWMO)4@jSKU9C()Wk%EWCdGkctr*m#CBn81D$gh0ey`_+vqKM8+k0(wV&s zA95&# zQwW`Z&v^3qO-1KQitiIfuJ|NZ=KMBc7AG#!=g#Udb4ihLDK_C0sf$hbr|8q|go7xB^i%POsp$hbt! zx$a{|TtdIio_je%eN&gmX0y`npGhPCv3b5g#@`s35o((3(-qVna5 zWo0P&BHsRV$(|aVe7RCwQnqJN;v#*^XY*FHuST>{YPn?*@{+#e5 zB_=WtS5X>rnDVe6%g=>5`ndAiuDWD1EL+ z`pmCWN3Z3?1o@E?6X{#N`Sb_MyEg97_d~Xq-l2YTP&g)PzvWyphJQl8=imP42zBS4 z(6?y(!f~PaC-j!U&N)MRa$+LAzwjd@CZX>^e7`(Fy>CFslhA#I-FL^%F4{jmH($^2 zPv{=U?xzyQM`m1F#+{(VMRIs=mx+{fR<{rNk>VklgMK{m@JLRsAU{H468c^E{!#Z8 zr^F=myYPJns?IOEuFNUXn&F?&eV?D6KE&zwuN0T)Nh>E&;v&7}!PMtdDftq*Cvo`V z59%IPVGrdNo7nn&N?gR-hdyv}>g|={65V|MSB$uX?v)&Vzj|cmOH|8`DftqR{mP%4 zw&vVZxKh4Er991er^Frl9siNh57pYb7cjNj#XLido4t$rzw5f`0bI4+F!v&SLddO7~qn)f&{k-PQ3Tz#*~ z)xt5!jd~*O3xbC{p}S$fwB4ZIPO*Ot?FoI?!7Z+aS#6;@tuX|D+FWN&XvpO(xC3HXRRORo^QR3p1+Hz{$xul}w^7d_v`?#La z{jlHHj;8!>#uNGu&*>Xyo-R5rt#0DPMfQPC_xOPFp1_skl5)*#ijSn`o#`+(m*XSk zM@Y?eO3gjn(!-gDLif|oEbOnoiCQ=&xlyxMGTv79gzjjan|x^>C0AUsR=x00Y4zq^ z5pjur=h|8f4{3idLO-YR3&(}hw}jp)x#+H*6_L3Tb+$@pO0Gyh_2&njDsbY0{QsFN zK8L$YT)^jZ8Ti()V^L=B0ckZI4#$80?Ojze`)6fjW!28gNbiw7Fe59ycEO)X8<3Hj zmenn-wmZ&WFgGW*z5Ko>n2@*jcrYQiM)hF)rdL*&{|2-ne!(56#I|F`sCNuvhN*8I zZ;6iz)?M`L%+R;72j0FaoGbtRFaQ6KOTpZj*zWZ7_MEsNzi?cjuGkopg1I@d!%BB~ zPE3#=Au*9X)Wpt@xvNoPqVXdoCI!7hV&@C_ZyynA$`xXg8#O2U0HLP1W$!03?cmt? zjJUXEPcSiU*&zI0|BNNkybly{RTv8^rrNpJ+ z93rvD>aC+WaY6q7YwkV6qP)8H?-}gDZm@TW(b#5&sa6C;XU4){>>WXDV8enME3vm| z6f`O#RxGH&iW-h1t} zbLs_=gVeP5WE{Ms^l=Ciy0d8<+81C}c?d3a56&2NH#`Z$ zh5RVYv&nsFF0suVX7D*L7tuXDUH+0UsLNG_tgfyS{%_iYl939DOvEer7> zsxiYqRb+3e17fZR#kTNn%UBOqPOY%D;`Ff)F67(eqcdv?$wlrPph9q=Ikm>AvEQGD zaG|@A#;KN>p3-`;tZ{gI$9XZA?wc7SrY3#Ha3Md6P$+ZpBg-$v5qhD!O~#qU6E8xz zWV4?$zCdyjy-`E#2;&21lQki&g&=fxJf z_hplcJx^5i0y*R6u!!q14-&#Lnp!iD;i zjq9}&-&noB`I9Xalf5IYEuz|A`9Ha6%NGb2^6!nuDq4i(;xcblZv+>9PPA!&WsQ(r z@-<4`2H`?^dW+BH6bKg;#kZ~3|1Ku!NAjq`%@ADZZ7t)jHig)IdFR)<8V};KGoyW9 zNiH(WK1AP-Ms2@~XG}=lKNi9zoBf<|7g6hK)Kq`(38~FyH3GCkv;T~H&%D?psTFz8 z(}=THO}6h6j|)aCRR8Y}c*Wj;LolKGV1KEVY^|Epf7sdCk5>@!mY+o=R@Xw-MVf5( zb8>-bg=RDw54ydd2GL3$Ol-2ZM^8euLO%3|Z+%EFp+-aJLx;APSt02|@~HC95L~pB zhmzda{R`2HmiCd4UevHRy>j{xi!H;ZL%2{ab-d3$Ozp2Fy&O9jxJ{Z%Y{xeUE;{0J z=KJFqz37O`>C4(*rMXxZIJ__EyuB!zA!bZ2Sp5^kk7!nw@od5fc6KJ|NAWcfdZ9P_ zj43~?!>oz)RJ)u{o6pWIG5LX=%QX@v@JWy>kYr{|!_-U5W{2Gh=cxVSU`T2!K|!G|R^L*W^7t z5nQs{&&dU&6{_dbYDVuDkqOhvqh1iLP#(C}H;B#PbG{F0`I*eMLNnM~$2qlPlXd;u zjNroW^IVtR2kC|C$=tbQ_E`hrLVCIN-N7FqdZ9TS#=ocevmP}my~GaMh~T24_;NSj zgGCT7G;hLqXV=0*(p)T^FG9G`naJJDfo(Bd9PQ`S3nUlu^N=SEE*yn0Q4^CFvrDp> zEjgLk^7m-o4xtsb;(Ym|yNyp$p4=0XOTK;XKR~!p?fv5F%z8p{ky*wf`WCbr((&6? zM<8qH?Dli|5hS)~sc!gaya&1OtIcL#6ed)&e0)|I(!ZS-6V^ZH?um#iI;vT|oE?s- zae4ptXX~-el6goj+g1|7#jY2VOQG_ZJVbkzrhL(b+3cm9K4g zZ~7)CAzqS;Jo22wyLe7J6VcKfB2$sA8LfrnB1=E)@Q%6@x!BJ<`R^s3ULf_HfnrOs z3%5ICv_i3^*pvaOl0L-Vf|v%0E%v>4Q}I&;_6f;F?z`IIjSQ#xkd9_~nM&?luo}T7 zyZxNC3&MwVG+)hBYC}{AhKYTC47V=`6P3O9)KumDR@QFs{8+1_9&%Ha^{H%zeNHAe zng3si`)4|u5n}qT#9_$X#_aZUa)I~{^`4sQjYz%?(F*NXn(9?AP(oTOmQ#rPXF4_I z=K2H1oWgK%v@eQQXzkpf#}vq&8J*hEz939A_I+(rqt-otg6JfheNmWDPE^c)J`cl$ z)*8xbe?&;fl=PJMfVx&keaa^KEj1bXMK$b zojBSTg$d%N?_Rt+yb!@V#}TyA?7sK*#sc z+g#M$k*?n?OAu#=I_lFlnX7vAg6JiieL-T2f!1VgH@MWoFd-i{CpBIot(Ci{5c9V6 z)W2?Ow{`N5LUNHm_BjfOEqdx-H~E!T{fyv}-M$z;q^FraCjb0()EFkT1|s`{FroK8 zOZ*XgmQBC z^qPUvIr-u@%@DbV-WfIZaGMEPFYD=kgDH0|HG+OYaN*~5J$v?HGjyE44${-xAf_H4 zda?8VoIYfe1;{r+xX?a!FaL7v&1(P`evaGoOTS2IF8+J^Lby=O>+QRUz2zbx7kTT3 zh}kT96~)~?O=2bq$;G8FGt)OVF3T==eaiA{E_F=F1Up7&o3Zu8w7 z5w&8IpV?Um5^HGR9fhDJYBO8Sv3wy++;h4$P-UzIp7CKpSc z)etTecZXk1RbaR{+83mDq4&g0qsmu++|AT$sn2NS$>r=_!kkRlyviwo2(8c@O;cFW zFOWNkdYU6@3fm!fm*x^HN4#mKr+fOQ(HR>?Kzt}KE^^=Xh#4e$E!8e#Z=d`N!^P2l zPCtUIpXvQO)40p`+00eIS}~h_QJ7H8GNE*n_98N|xisk%gNU~}s#zu$eESi@h5TpI zo_R@<@zy1N8Dh@1UPt@eKc#y>?v?3vd2o>>bz6bZ3!mvQ`Nncrh+eYU7bLduc@EQ_ z_*TO(p))G7F9;LfZ#v6T0MfIq&t_i~CKPMuH2xXW*Fv+JO=MpbCKTruxOPlI_>O`1 zbS@Z^u}(7o$cC&%?8zDUT!4io%Ptg?%kpYZ0Qtv2Z{L}gUiN}=WULUNJ&1|WKq4ZII2@{cp@%}?jAg$#M=#idEvXoOyPA5v68L>hz( z&A&CR*yI-_=|}Q6Xa9h3p>-mC7eFt0a6#-R7*tfZtXwe;($kXNz8H0jf!=O3t-h3u ziMQG8i^7CzxoGR?j+mH2IgoynoJ`pJ&YL|DF~x)KkFGhg>okOm2lXk$jLy6v8Ea*` zTf0HHP+VCXr&eLO&^hAT@aMVov@2TKAY90g*4>XOhT%ecU+W@b?@8;W@29#5E_{B^ z=AskBAzZTA&lz7JxtY(;iIev-V3?3r$i65{$baJAKQTj?@V<>Lrb9Am|FK-yjqo4d z&$8v&jZ$KAiLG7=!i8%7tuNZF$8e$j+wFDBuy~)-hir0#6*1q{zB+cw_qKXk z7iEaSh0VSF0O3Nl|Gt6!nr3k+z_{oyTRqQQjB^ebe(p{7i8IxTT7?Y<1a7_4&u0!FQM=>V5jEsr)xz*FQu99<@WV2r^ zl?@tC-q79JvcR+eAQReSIUKqmoiR!1%9u3V#eC<*X(uspv|sFWCB|gU3Z|2^@jxb2 zD;x{=Zo-)S8_bw=Jjr}#@I5Cnp<3x!oeABo-o-awWlW}Ot)8FKuG{U~WPYb^BXSR) zMRg+J2J3xI{kJ*e;iPNeM9!tt?UxWP zq~p^q8?lspsaQ1Q(slylQss^yTr9zNAzUcFoYB_LnsGFRZFbmXGOOpwuFmo!NBhNM zhcmq_>d5?P+;RKw0;u6M3O)<#?1*M;zQg~`gseg?$&*`^kIn4s+ykx(BUW|Cjd?n0@l$bDzGru$cP;Tf8T)Da5#Z z^d>HH-%3j$Tu8&V4)T8(mtWZLWh=6R<>-;FPSQ&rTx1rP2M8`o%8}=NmiRL+yQeTN zrH)xWPaN@c4i`QvEXC)_T&9=0K};``;@I9($8uaRG9Lqk3;8#FSE`leNL$TzjLXTP zLU4h^T;5}IVQx7urj=_i852cMtLN$Jb)Ce7V%vr5(X5^E{^t#&uGlJ#Kp6Eu2hoUK1M4PZ&UXqvA6e9 z|Ej>4RC&U*GRj8?CN|m2;28+5@P4pMz1@#8E~yh4ms*9{KBbD*B>(IAna^<}t>)AU zdqeXKV6CW8&^z&_D{W+Kn4opG7<1LKla|DPW7n;2Z=zI{4E zxR4*++&CqcaoNTi7Hoy@FfKovoWsS#(SEUEJDDG~+s@XDy-wNfLuwZf^6kGz+|Qa_ z8UKheDYSs?Th53Qf{9IL4f+wG6+SQQR*U&rcg=SF$hZ_7%J!j-I4KwLIZ(Imy1KI* zy@KrzCi@py~O z)%XtE|94@|WNY_;>E*~}=WyX~yghjJT`N}8xvgYerv1UV6pi6rx<2j?;X?h>554BI zt@lf=Js6ixLs)$IOzZpq^)+LjTr30YLAX%u@Tl!0=0ca`KQS&H*qSjp_?(mULVomU z^{^!7M`uSfz06(4YL^BLxn3+Y0JA1E3O;k_@sh;ybbQ{?voHEo3=^sm z$i65{Xl?(Z)#wxq6RHnhMEbJ0mXisajg_(n5m$JRMsA*_Q1=ejKPMMRT+vew`u9bO7Q=*c(8qgg+DU6AJ*6B( zD~{iV(2ApdQM5ww_VdIEn7qv293}grFrirYrT_Ue5dSgIeA6#U|BRLPpFZWcBm9Tv zQkmsNyozJE*!MEbvY>xINOOtx{T9N-9&gPqp2kiXE_NR_=X;oVL6VDnL({-dORTyx=G|FXVQ1Q+|>vANKo0-=)eR+gSI9>Rs=uTfGzuGzI|B%9R&(2LSu&zZluJe(cZ<%}&h*?{j$BJ@J@2+c)P z!&v_Yf(z-T_|OIHYNP}gc~r%Q5H93L#VeIyXAlB%ky$1I=1Ma^vagHG#SdMA>_I81 z&(fT$USyWmfP3;9rHa-iB{Lhd<1&C=vf0n+N054sdK}HA?>uGgPY5RVvmtXSQygn) z$;refk7_v*5nEIg^W3V2LCz7C)RUV#mwXq`L?O6PkEPk|=xDa*0nm#F#rz8G6Sqt1 zCEt8s9fS+Tma===wh)qwOVqGo2rl&Yr@7M9>7^lDvf0lWUm!7;dI`;yWmg(wn2=U0 zk6pD#(uZVYP9pZ9l+;UTu6(BRU@^H^_9Z}kh}N-HuGI(W#nHYfK4d?mF;}axx-Ld5 zw3l3UQXe*pDkl@R=hA#GLMzl?X|7q&R6F!h6~l$wYwR|OLOU5{UCx1_3)X$j}ATr;gZdMPQ5^U zNJ|>7bN>Xpe*o|y>R~k3)xAiQ){6fRfO)4HCB0c_u6s3uje`=AOKic(5Feu0Qs3>p z4~C1QeNlXf;$8jy4Ht{Zgw-tC_6V&|&!@S;wuWp?5AY$Ky*JI=aCw?iI<`EGn-1YZ zb$6pk3%low;6i(IjeOl%jh)knSdV1`F9;W^{T1^%upUPQ7pnagji0y4g-gHY5H1v7 zlz*N$hT%fz4l4JStiMK5FB29cxKK~7S#`G*X8la>Y?^cDa`^mx2p6j9JPr=s57CRB z;){C6t8bY{|RL_xpQJ9ebG~2y$42B8$PqQ*EY(!U1CN|mH z6;%MSRz!hck}56OE}1IWIdNgR%iyGxz(pBtZy%;R@i#zujLRfwAa!4K_+AmN~LtPpOXv3hiKNE*&OU^fM_L~ zeNmXu@6_I}Q*jIvTDQ0R{5qF<-x2_LCq$!CQM~KWYVdU-xyXITokqkJdQ;NuH#aF5 z!o`F3zx;dLV*TSeeaI$HsX7(Hh4PSP@}(bzxwOcX z)Qc>3G2o3ejf(gCbm};fjn#4f8c~%e7faJvgkETNm$`Fyk3TS6X#b1s=Zr6qx`p>Z zb!q=01jB@48`&3y3GKIcUE1X(h6&XzT_4=uAstsNvyVXhhvIGbgo<+@TzIcfkD3+x zOLGbNl7`^I`+a(ZmFgfS7fS|WCZUREnVEYACzcbEi@enuM1Q49L;l@6z93`{he|{7 zt=9qnT=rYif1i)=Bi{4VugMKaAC!vThBfznJfxmnZ-p>}MhTh7|WCVN||K7tGHFBuTCCJw?yN9*W; zE#?^|xyV;NiH2~YS~J)*VuO%eWN#}%)+oBf=A1X&mH9*QB!&n98CLa~kP z3&Mo=wG6E~CO||c>^x-FJw#m5)4Dz+a1o1l0RN#`bLOE(H|Ek$xb{(b2p7r&pA6mFWF7vYdLIyy%P~9+maif+PE)-jaKj_JNrE|s>o9xF1`4C*_4O{cb`l~SK zMZAw>#5E=B^OWLZ88QpPh0f7NSLuEPq89_z|6ym>R>_6SK1kid=T(g@tqIE@TFH}(VmPAD z(u2>0no-ORGKbHD-a|G2oEVVHd5C4#Y=~Z{W|-;H2{OyZL*-~+kl4cKWX)>6*%jkM z|1%StylKZwgjRST$HM2MnnAeG+mhz_8Fw-yW3K$&qSg>DbawWOrbHlw3!laDYv8^} zNiO*sZHCnSyl3IpG84-S$wg*qikKtd!OzB*e%u}=Bo}$)+ZzZ!(opRmr4Mh4;X-S& z$ff#Al71wM^$bRE(Gr){H74(aaLHz0kQ}X}^N{EhwUQxB`1yDAsHYXAwPMlzj?jvZ zv=aMjNeqSy?SaMgTC-b{i_4PNh(0k7J$=`P^DT-(xbVBin|mtQK5|YUvboHAH58#2 zJ=L0<-1b%zl8f9oyc>iI`O&8RRZa@YMHX9n7lI4Fo4xtVAoiv%U@w=SpT#Y4TP^8F zvL#y~dg14aap?xg+;)#Vx%6{)hva#|WnbLE9vCi;_H*h5l85+A zjQ!Cg?qHbE{v6pCg$d=M1NUFtg)rggt%JhUyFj>*UXE-ZT^YiKzjbre?@beFE^8-2^x{Evf6~1`w)gJ*eFZh2%W$wNiE!phn zAeL=|eV`WnDKx zxR7t(-=f6C+q}5QJ8V6L;KKV#9&D;7re0+JyC8Zo(0=8^wp}rO3VG>8W*Lu|Q=;bg z_#e$|?uCgj6yF}DEqNjtUu6EBHbeY~`UxLToE(JVLhTmP&`U8dEXOl-0tL5Bf8q~X0`&$^#xvs0YE zhSu zANIP&U+*AXloVTDpLl1I=CZCTq90eo`@`NGp9#4)qv7xHq?cQhF3qK51q>J32TC8f z1u{<{Z!VT^)ykqs8a=wSP#iYz@|1az-|G(JR>Fx2}BvAc+`X?msnZ0bJCy4`d9wH+QR$TyDz;EQN5P zxEp-#y;evrf0z64)g3FR#9g(*ekNfYm~?)okX&xos(ApR7ut(54&G7vxR6|~YySQT z!iDO(kY@f9gyfR8Df%RY3&od^TlK}vM`;C4Lh=#qFBpecG8Gh3FPE*GzUpmvPTW=5 z_nM8v4@YzmlgsxZuMvJkck_%RGY?J?lFP-XnxhCVbT`fzb~ijpNG^Z6jQeU{p>zC* z?&cZC4tHNHB$xA}&&5OZV!s1s9D8)%CNa6V8J5R=$ z#SonL+OKAoILD9ZZCm5ao#&>D$wlq{Ro|F%xX|4zol-eMqUiP-S^&3JjbVtG% zHL~6ZA-QZHb@Hnj9nOin23jBemaj`oA-Qb1^V?TF9!}yy`@6qw2rVNfmlhFU^#eJH z3$2e<-HtgZB$v(k{eA|-7Y)t-G_E;PPRx9?;k#>ye5B#$YBBx2ei2eH>!-!>Rz& zY)4~ab*-4|qq!#%A?qWW?Pxsc_I{d>dYN-R^ACg{X~>WM@U1Uq-{H*Y6NtV;jh6CJ zlKZ-Uh15&L(RRKFy=dv&{&=5#Vy*+G%`iaL0XpJx=KJGf>gA`y4Ip}<*<;4!g4I6> z=||)LzTkuKBYNx4nDWCqG1o_965B!6M+RC?Uo0|sw~%@nb$#YMgkET7k}nzpkX!Eb%1EJ80DiZ5?l9T9UqZCRs4 ztf#enzwV>)o|yN9O)HBb?g?wPRPTO#R#?n_Z{zz%h<i)41F828^ zrcxWCLWJa^Y0)he!9`{7Q#Dn2zm>gRlgdVRdY|)@Sv`mMz9`1r&^pgrSrA-k&WY)} z5{K*jA1+52mx#6l#o$uovKlc5Qb+qzrg|fiZ!2ZW!X#R^yG5`5IM=tTx=83_jY}b6dAzWyEq@4Cggd>+7 z71PDw;?`ev2*SmF9$@kat<}zvOMLAUVsI%l)|L+8VxNy`QU{#B?#N}kEx#CCN>8sD z1mQw?TGORw*8PD0@=^S}0%C9}wctGBet=F-`KYD!&RfQXm2_0s#V3>$gG+%*CE6nN zLT~q)ypFmq&88R6hhlKa@BUybgbU>(pSXw}j$F1c6D01sGzs|y!iD0CZ$#X(Y<|>e zfEap_^=rKa!G+$)H5uKlS#t>fi@Q5&ziORa27HiSEg3Hh)&sqqqNt=v7e2%r_Vk&)FaWoox| z@{f*Ob{1?a1{e8bAH=L?J-r2M@++f$vfE*q1TE1lv$dPn=hFrhlGt8Q3+3=;?Y z!Z5M-Mwz^;I`kE|`M0ZC4ZwbUQb<3q1~(s~Jy zZ$jvW-XSsd@-O!e!iAsr_WaT>QkskZ9>n>Xp6=6_diyR46_Sg*b;BzVy-?ik(uAB<-sT8m&WqlI z`hNmK+n9vpBC|l|yy)rvm1)4@J0%zw*S0KA??2$gJnd*-qOOCiooRNTDR}nXth*5a zCe)KH7%mR>MTsr+#-3^1<@?(pOdRbC!-V3>gwjpgi^#<0(xek&o~~X;HOs_;Z$DzV zkpE2DGcQRp-nzsuL(DDF^EnhhrF+y7l8Y><+X_Uy<^8ymZ!C9(aB;LRN^H^78PT*S zzSS^H9PEq2g!h}yvJ}Yb+jf2(=V)ITCKPMuH2yiOuLZ!w!M-p|DE=>S?U?nhlhc_P zcu(hoF&XP5^N(!ETEw25fzJ_GShDOwF}WhPK<}BhKp26|E5c$Z_z92c;prX2A<%)4xJuS}9(T?_oVM6hC^`&G?yj3~a z7lsMN+Gy+Oju@T%&rEEx^vxcKSnEOkBx{cBIt}5%d#_?fXWo#EwX)r<-5^{juB?qy zt1w)smRlSCJeQt!MJpSG3;EHy`!U5ZT&TBmT}13XX}$FQgm`n&z-RnyE;=zB!o|_P zC^47M_lcADGGLgHR_ykLVM6{B_x_0)!i4v;Y%v{TR}pjGN3_W-7Z5XS47{IZ%d;D$ z#N-lNy%Zwe(wn8GtuNZF$8e$j+wFDB6qD48++ZDu;KF+pcDnl4hH!DTFX2as-ed#K zvo|H&G(gT)4O)tAyA_cUl3J1Z_d?v0G4M0r-O~cH=AJr#jhH7F%h`$uAJS1T?4GgO ztouXaagqD3{s7@Zwg0|>{hC6!@N;jweNkeIj`G0%-UkO^m^j!Mh6%0d4|JSbAH&4K zz9>xitgu7jn@2&IINBG63GJ~Q4qcFrVd7w47$#IJ91Hhuf?+~y!((+ObeFDG{7&6Q zO{benh-7q(#i>)S3PMi^;W!taH05rYWaim7%miRPrW!@Sei?x+lX1g20jn# zbjwCh#N=WLz6;R{#g{YM`dKrMoL}1-9PEn`b2SP+3+wENW|06U%s#C*tmiV`Nc&K% z?>gseg?$&*`^kIn4#RMvTy*Z!7Z*t`a^FfzAY4epwhr=tgybT#xI92`QBsaP@3X`o z!iCQYOYylfS6VMJ9|MF7`M3SMoF8*FiafXg_T<<%(K+6Qx#hetTA|o>;d*p#cd2IH zf@pJ#yxQMyExbvB|r0iu+%+C14LwElf4WEyho~0 z@P4pMz1`XS*y3r$CXYM=SSvCv6kD#ekr^Rect6;c^4G#7^N_5O`z1tdQPEoQuLWLi z7%miZuRK1vRGLfdgeed%6!X$1gbiU_rmmov>*<@XxN~Bzit5R<)LDh3xpXVD62XPf z#kyL%%vmwHSmpxW+tDa^Z`QSn54#J=Mef_DBSbIcM>jW4iG^_SaIh~*?czbc{r8Cb zY<49ew&cOYCbI_ph|mh37j~<~e0J9?|H5M|=9W=gJ=^v9);Tek&pWzx*VSD*=EnX6 zxD%pLsHw;Pwz_6XF}YaEK85IoYL~lJ*8hRwLi^lzk2YPEi(Z!9L~!AKYxfHsVKb%? zenj!@-tOsFq`CMv^@DJsJ(mYjjqgCX@cy(1ufA&~&86$({tzy-HhSnae>mgvD{Dv! z@hEBa^mH5SoVc4O7t6qU5H3_ZJZk%>28Ijy(WBMFq-QF~EHePJCNv5@bLjDs#X1NV zNBg4GbbQ{?voHEo#$;JD=0o)cT0NUYIps|m_M1?RV7D&}6I$E9Xf>MM2}dxY`rt*R z?{MjOYe`vyh%3Cu^7XsvGa+0Y?Texn-p7|Q`5`-R%^#V-{6}BG>Zw`2-8nhP(Y`QD zDF3`K7k3=PgzDROC)?(-2W9CEc&k>U;C*YEt~b_VxKLhxpIB^$bX@UEsf>s#e5THa zf*rH={+;hbj`l_IAwA`we_y0%8I!HISX^mW$Li@*o;{*6Df8O$10kbB2Ze?<3=0|> z6cQ9RGQ45vpx}XFV+Vx>$N%?V4T8dh8Yp*$3>y>FK%r3R6zYEL5xGD`S}pq5jsN@0 z#`eDn8Z#&`D71f2W2RNgK_Bm}X(z3f^ptWCtvGy_fLtKw&w7ftpC?Yp%F6()IM^4) ze<;>{>3{wVgo%OXn|?|9XRNgU^eMj`;XgE&$}BJ9RUE^`zL#N^1^xR$noF$jw-7G& zcx!g?G&m)E=3F)lXtXjDkgqSh9h8|s~thp6{8 zcP_HG1sg!PIQ(7$azW%FCCxH37nmI18KISI_65m9N}8i)E?CQj&3$#g|K!EQCX4ld z5AYu)_2Zfg_xhLhog%o{_m0hl1{DaEjJLA%jPVdI6mN^jm6I`CD6SP*7hXl0OKfYv z41bN1`f<&!O(WT?7Jy!q_Il3z&E?^_^jHq~z9d2~G>_0+G&L-%f5YkdSxG&X=Hf#a z%#zfLJgQb_uEh@^9ZdJpw&JmsOLwRw@ zckxUVf(!Lnn%$0$9t+{(K{3BV`^4>%ddWB6R|nxjdmd%?v~3|I7ni7E!w_8P?N4*1 zsnbhCxH#GuCFW8up}Df`N@ENY(rV?gtM*9xkZjCJ#6FagdI`;y&vYIvCKt=T1c(pO zI=0HS`XIeH*cZl!>}N#gYBg5Z#b|~0lB-VYvs&7RVw=xJXodPK%{A*9hC;YF+80GD zG?UO=tB#=%h6%;nS`(%Xkk(2^V;_iCsFtdoug+l%7pk#qcQcNc=F++PK?E1-;WK|9 z9efDF#nHYfKBOfL*SUYf9mB-lLujt6dyyus75^Uq^G-ENE#=0#S2Mbb$tAYnWQY$@ zY^m?|-Uq|Q!M-p)MDecv{)UT1WWvsgwCxdEp`K53gKZ7BL%8Vdb3n`um#5|O*7MW2 z=@2dyTN*`LvhMjhe-BDWF}IPgdtK?6YiZyG;X<{)VqORKt}%iO)&7dc&)ej}rC)Oh z7m6>+KTjOPaG`Stm3zxhxp0}V7{P^la?Pr{r7-JfU0z%cpT7^`LN%So!J+#hdeKvS zQSW&5t+Za!BYHr%ID8i(zU09L()Xh$4eNe;HyfiBifuYmYj*uY^1Xw2#EcRp^?;i7 zi*~U0hyc0B(Y^#1h*oI!sM+xI^>Y{|RL|M%3&Vu`r`hh6V=zp}f0~tX=_c(zYgbeO z#99^2k~Fu7&bW!-lFhy#T2b*nB=1oTAp3GE1!>j0#_i40TCucsLuf@wdAZe=jSnDN zp&5YYR-dL!&V|cg%OPB7ucP&YOvoOTir$4a+wF_uLo{p7Y!3EiZ&o=!FRL8w3&VtT z(%!FA*1Zs??a#|;m6+Fet~e|y*`~>T4YM< zMHagl@Wz=&#ru6ab)2|TOfHtDu?W4;Ofz%m?jBh)vYelf9PNveqj?Whm-Zh*Fia@6 z+3gF%g!bFJF75IX!-UQox<0tQLprWlW*>q055?Q=2^Hr+xbR+|9yKfWm*x`kB@Mxa z_xtn+E7d_vE|v_$OhOgSGBfuKPAn%R7kR5Si2h2IhWxvCe8H?a9L~=}8j5ed4*2J? z-;)0Oe1sqIo}YeAZb15=RP;8hx$om4xx8Uz8HqSgRB7q=>OZzjPl#T$q}Tpu{1-~c zUH=vy5WP@73fk5CBZdp@aR(l1mmr*HXrrriKLXKHWhk|@)!mI&)k}+5A z+ZnNj&h}jZy=1d5p%+9yp@)Lv?ZkR<;X-Of7VA?N;zJbgeyX9&s^^>@TRapF_C?VO z?-85)rRsAC6G!{PFroT#dc)IMvuvEN6BW%>oBqais${H{9rrdN{D=2wMbw(GOiV6{ z;fOv<4?YuWMlrXnIeY-UP;KyYVn8nEA(maUA$p;jVWvx`tXVbyE)MoZi7k9i)~x26 zT}5PKlQ->{iO>q~<5>87R5J({dPmVbKjTh@WXzSnThtoDh0e}?(Ub^;aM94-;ID!E z*qCdHdqnvfZHCnSyl3IpG84-S$wg*qikKtd!OzB*e%u}=Bo}$)+ZzZ!(opRkr4Mh4 z;X-S&$ff#Al71wM^$bRE(Gr){H74(aaB;LR5nmwdXC1}7=o7V)Ax!xBcl4;I6{NLd z(fy9lijK4r`)WxHh6~lXF}>F8mgM5H*Pm zLw5VZFd_d*xH95*2opZfV0WFtE2aI%@BJf$|M2%__Ed?^FD93_YQ%j-H9y1J^S%Q} zFSP#H8(8?Tv|fhncZKMMe0%Ti>-8~u$%~8SVkm?Q)h+wt4)(xsaj-8+%;hsN_D7Gn zgJDAZb9Vc}Frhqj;Qoud5GMQ_I#Ji#MLMonLaQPChtI`0eEftbh6}~D!=u>4(-K-? zecW5SK)8@zj%*)Y8N!9Xb#v73O%rJ@YbQbU;z4zP(!Ib-7%r5Djy0*^F3qKLLI;Fi z_*|ourCY~s$2ftdoydUjq~#me?R16;Y&v)@tc<|w7QAxeHEbF=n!i9YM{uU)B z-sZ(c-eK!01Q*^{@?cXvG4&$z-v!Z&f%YpOw(W}PQ^-p%GRt_xoDwy^$Ny+%b1zJM zq4@SFZOIeK_#*S~v>D<@)KB<$;^ZI<7m9C>&yCCFT~fbH#JxNX??ZZ$G%ppwC7XRg zYC4UA*7Q$T9ovXuLcR6Riqs8~_Mss`hXFpM;k{wcx}TmQCKpR|4G0(7Lw(`12yz!f z!{<)e?F-{W6mMV7iT{An3dP%(*DYP7eaI5i3!xR>ANIP&U+*AXloVTDpLl1I=CZCT zq90eo`@`NGp9#4)qv7xHq?cQhF3qK51q>J32TC8f1u{<{Z!VT^)Rc@1p*_1<{I%)~cD!3m2E_3aFl_9g(BSz0{;NSycu)hy&X8ea*#9XMI)%EQ{a>fm zqJQ1^zrSp3|C^vOg93v>`v)~vd+6=C+1UNdCXLmng#5A9J8`s+wS_ex>HpsxzW;~z z_4HPTX=uWs1jip=U+EX{1Ue_ke&$xH2a`|3WPli1kvqp?T&YBtYaP&Orx zR8wX5B`2}5=V@cFBkEGZvMHWE`zHh&+E+F9?X0dNESsXeXMZ(Q*g3kPJ#S;*%&fUc zg8Gze;3`czs8_@bGHi1reKQl4ulQG;^1@d*;_P%>L&k#6JMQEJH@BeGzZi; zFzNhEVcFz+^}G;5H?&t}9K56SabelGZ`f;&e4m?z8`HIemH`?t0y3re=`g% z5b=fX?-@sC9-Ja1m&`?mRbLOfD`jznX*NocKcbvy5{WbxstL%Wd}o7ZG})dt=6V5%1ZVzJU4Y zR-@xd5H3{jE-3h1$TJk%&28rq=O_H$m~r8nQX7QS%Z+ch{DaU7-J>u@jjYG|5e4+4 zRIgutg>a$u(Qo;>uy-m2fOZfrw7>h?hR`x%aw*(M z0=Q^sMyPSkk#b_@qqFH75cx>M&(~u5d;KD$Ud~#^e?sU*N%3X;^nU)|Mm z4i|c>%ebpeA$I>>z`E|h&^=~^AF1tk`-};x`^O5&W$&&0{xxGC`2#hM}PR%7qb7*wk`A*MDL+SOZh0teciu8>LqSb`6URw zXzASkc%OY@t^+o8$`4rw=!nai?~jYAm)LplA$pP&tELd*?xR*gbzNG`=>0xu0sZKgAG`j5;6m@b8L!Lki>a4Izh8mqg=T#iZ+&<02O;$`Z=pO8p%)d!m%I5M zuzA)3`qAtimp?mYbIfvk)R-7+?bhq(I z%9DFSa+z}XLL7t(od>+QIWG-@>9L`o5bJ3z->>^(HJ}WF{zxS|H35b4gt&ZxKFK35~S?>;g)D}|j>g;+kxfCidWZ(f+O}-^&hcCa-|??!e=rg%SBkZ$A$(6|`*@Gam&#_!g3n4EDOtRHVqmFd_BQ zKkjlxgkEUwl&Q$pjMiduDRinego}M2)AUVDLcEY%0?HKr>YY!g^w(((6km#6xZPPu zF1_zu*F*GT?`JR-KUH9#kX(8_+j|+nMMrN6m`d(kuv$niJ@Yku3E^VD4{Iv5Au2>j zE|C08T#LhqEDzAJH9$oY`A*6Y3?W<%=e zJ#JII5y`iO)JsdRm)j6}QB$6-KVZx$A-QIzCgm zQR|*R38|N6A^l1r^g{DrOp5u>=LyNBY3=ha5H7SnQcn9LLP#!6l%KzWaIv2Um^?yj zwG)zy!FNSV2p9YLkx3nJ{<@G{^pl4T z>GudO^rnEx=x!Br4neJjuMu+ybky%{GFSEJC8STcT4a+Yimv4V+KL?^0d%u*aTm7M{gyd3YmftG~ z7usj&UZec?LUJj+WPLD%3)L?@+-8coo-P%wL9D0sG+)ZpvuB^aLh7Yt*WO>fk?Ned zOYf7Idij@Q?{y2fZY&Y7J`&+abkD}r+jkLrCsjZ$#X}oZhj5{|+ows)Bq6yJ3$I)f z!iC~)->L5^3(2LZ>f}fS7kbae)Nf4cWg)q^Hv4BEgbVcr1%$RS3CZOf-_bq@E_5%; zG+^q+$J!SY;>}@O!5Y^lL%2}CRPgM(S@$Ny<5J*8-^h@d0v@bQd&3AKzt5Tw-@3?tSWM4x}mM3ucR@T;i`#zp-n<5GXV)$??9OAGc|NJFbC_tUCu!u#fINBG63B{EOrJJ;8OcZMwlOZRW|17f!!Nlg$q*DwcuIOm3 zII-Z{kBo~ZD;c!aT0JjJSmh)xe>09$9zP6!jUG_3XROUArg-7g#A5i&pg@c94?Ob zCG-M`w|Y7+n)bxE8e>w+%9yMz&uW&ZbDYG)!M-R=cu(pqOM$H3?$#cRiS}1k^Pf*~ z4iiWF!Z4v&JE!r_S$!@ZD;Semvsp|T^vp?29PA6jgyR1K*N!QSiLN_T&tF*1*D&Wfw9ois6jQgbY^09XTQhm*v%-0P>H4-rO@S zz3l&-aZzUFpP#m~y{C#FoWn&)>%*lHzxR>WizRv?gbS??mz^zJopEV+f^nG{WA#kU zA1DSFxo^Nx2p5Vkk$;@|%(!UlFfN1QSsps{ofuqPnv{)3aN)Blq7ov~7#G!L#wDaM z%SDqDoWn&$>#-G^{GufNNdD&R9}q4M-{n)HiJ5U})zIo`4Vx$g7esw)p!XR~D_4xm z>ThXPi!rIi_BhTi?dv2aRJYje3sScjR1|MlUrNr3w{4~~CN*N&Iw>IT&R{? z8~!{{nv0^94Z?-|Xx;soVi+#e=eaH-mK{q;?8(XceyWS$!sh^OE;=zB!o|_P1Q$qd zR?}}4C+}szFmbRi3={I7xc5)Y5GH*7&=%7n)|w#YKb8x-5&pybS++d8QA$iMvDFcC zkqo?_W$TMJ>oHtt|8{%bGQ}kIA~#qEBJ{$06n47$*M@L$v@hXDka$Zo?@b9e4OwTa z&aeCVdG~HbWQ3$vWd6ObAhg2Ie0NU^$eN8RAQ#J7#7t^~j%x2cW3^d#iNxa~_g(z~ zq8Fh)(zS};soRL$!{3QG5pbg> zgo}Z+azf`-Pnt`;74INiDE^;X{-8XD3&q+~FHRSh=F;glV!pk>Ksr9%ve6SUxmbel zLi9rM<&3s|*8C&q*R}=+`(nggjY2`^hi6ALiv%!X_G!IgJ(uxD+J|C&*EwG+?7OJm zPu_cX7={b^@VQT4TqL>3eJd@2a3KxbI>`SKl8emZ@&Lg_NjdVo&k}zK7e2!*#plXg zX}!pN3=l5l-}dive$3S<^56p4lVjgR=Xe+9mh-}Bg<{)<>(Q)zUUI*G<}HX;DCS)} z+5HxVi-Ua$E*Py)yiMJcG(bcqHrdPI83?WLey~fu-Lu}%c6KeteE5IXicKDQ2C!Dt zDEN%5D{W*(2p8TbccuKbFzMLR$Q>|~gmIy@;$I8A+%Q}y=3aSxa;Y?z*a=f0endJ> zn-Dex!-e8oTI#Gq(pv7h0o`@ zb=TEhI_Ady1h^BTQK)Gz>bAOONin%t%07kYg=&|(Ro4H3;X?b|caJt*m5W}M0p3^9 zD0tu6{X$2w=1l|qh~nG5-P5l~bMbHLhtLauH}64I<2w*8yg%*1tM6J#bLslHKZFbI zyFB!oKU_>MmVxykT&Q+<)b>#g3>WgFN2`ZP&s30EW&mbQXcTmK@#n_lIskq{oQW?U9*3TabcEs%c^O-(&`x0CraYau#=-(G9T8LKoOrekW*0ht> zN_t8;h*l^!+P@2toAcm;(F(=e&l4wL^0MB+zA*knvF=O%^JgIbW1#t_Uy}YAEA2mh z%5O*b56z`A%Zqpw$8fRtkC66^adgo{1inq54NoiJSNK4i}KF!6#U7x~7K z6CqsewUpWA_3m{-a*@3)*Z{)C;rBx1A?gXvRWFD(7t(jCRM_{9&4mUP2$hVtvhUT{xx}^x%<$JJsUO$u+B6a|tA*LO*K_7?E)UP8$8x~;B@ud| zd4%SosbP@*4JGx>nTroyFiTP|@~Db{H#gbuMSfJgQi&vpALYeGW|;(-E6upr*G1;y zhb}?(pz_p<%+eZgPhO*>H(JdlGaLT|(Tk&fQEC^OyJs$a=P6=d3;Rv%XG7*vrnu-_ zVoOWFdzT}PaB;LRO3bBRLUU!=mBtt*q}9q}SFxv$q|S)O zoJ8zHDXEvxT=`7r!D4c;>`Q?75UpdYT&oY#i-Ua$y+HP%)b=wXbF~_)>teJ*d&yNN z^;s?LL$S^0BD6yNmFAjt4MQPZ9PNvu6`DzCu2siS2*ZTpZLJB@21skAqp=S}D^yF> z&R6F!h6~l$wYwR|OLOU5{UCx1_3)X$j}ATr;o@jt6d%%(hU?rv;f`TK>%_Xc7irR3 z@&5rZ?^L7IQf{n!HKVJTTw)7OhWHT0milh*eK1@c>iIM`*w%16gp1DJn`UmfJdM5eEVUkb8aExngBqtx#;!nOY}EYsC_em{Fpn9#FG>(GJ9WM4D{&C2ALlR%rI9+3@rAa}cdK+82fi z`A@UmE5~4%kpDC*xR&ZlJat^EgK&|v_dlg&88I+vfCHM zhiKNE*&OVPc(Y2QaQAu5{ZW^VWSbuRV3B>?bF2>Y!l-gRg-_&Ox6 zf!4NJUwEyMbOZzj zPlzAU>@aixGyV&u5TQjbn|< z(Y`25cwfuVs$&8${zLKJ{+mcJvB@m6?jhoep88rs0vAEvQ&G{ZIrGq?8*}L=T>Gdz zgbU??(5?q>K=i`1*rYA*_Y4@#)l~0POKLfE~Hjuu|9PnK1A{Ery5E~J?EiturG>M zc#qiRFIAsIwBl%A7$#I-PH%V`GRww8MRV1rzj2)^8Ea+7y-f)J;XPUrwI(csXeCcB zis6VpOAkI1YDO_P$Q(WodJon7b7DX)=OLC|vmtt+nqj6(C&(-t58fwcw=YU;;d8QP zHQ(%t@gaxbM1qM;-n3&TLMyzFW8w2r%^+Op9Yyo}j5`^UF<1U>QELbnIy?JCQz8(; zh2B{+{~EY2Qj$x)Mw=mZKkr%iwamowLUNH=nj+>1c<{6Fr60G43CTqs`Su3Fk2F+! zN9n_xVz|(nEOM#-lB6HWVm*TqT(rbxb&bjUAY2^nOXMTS`dLTkA<-vlB}16-^Y7?U zPb)}k#iIKip%oozCHB>l7z`IWkBRBEX1634mnE+eePSMZ`mPPE+1w(Ul=w_**we{oXW@=CXDYL@ypx_b1&8yoBLGdFWV^3hvTeIwy2M=!M_sKb@}< zWL~bC&nwB5Oa9X?h%;9;e=Fn6Z#APKdeM+xY}FP{lGIE7h@pQY^uo_vlhe9ch2$dl z)gbOetNA_gb2e9RA-Tvb5ltX^p*k}qbbLV!7YF;I)N~rEGcN?rJdR=FU|$#}RJZ)O z_av9Q60|_7NDZIM^433BMP9d&Uo#{eFHg{MKFftixX9 z$Rm$LLbyJZ_W0bmT;3)1%S7DE)9^l|CrR^C5nQs_7o?`sDCq6Jr>l-_#4w@W z`e#Mz21)zSkf6f=AJXvNuxH&*&k&P~CAtQL3+W(y;=D3++Rt58MKoCy+N6%QtHwenc_%ZLhgDj2}7J7bWH@DduMQYcP3; z_ouy$AKh5mht?iJ%vRR$9<6s#|K5UVMMZ1X%;trQOLCE~FWL~{Lyq4Cmnii-UbBt^8kRJqy49 literal 0 HcmV?d00001 diff --git a/notes/current-system-map.md b/notes/current-system-map.md index fd76821..4065333 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -205,11 +205,16 @@ and camera-tagged observations over bounded MessagePack pipes. calibration, update/observation counters, RGB range, and expected-hue pixel evidence. Live exposure is 0.3 for `nightwing-eye-0` and 0.1 for `nightwing-eye-1`. Both eyes calibrate all four Moves and produce observations. -The latest Mimir window saw all four IDs on both eyes, with 16 camera -observations and eight same-ID pair records across two aggregate frames. Do not -claim full stereo calibration yet: the RUDP subscriber still received only two -unique frame IDs in 15 seconds, so correspondence diversity and cadence remain -unproven despite complete per-frame overlap. +Muninn's dedicated evidence aggregator now owns sequence and transport outside +the 15-second CultCache telemetry loop. Cached controller state is bundled into +fresh optical frames but cannot create frames by itself, and a deadline clock +caps publication at the configured camera cadence. A 2026-07-13 Mimir field +window received 622 unique frames in 15 seconds, 2,647 observations, and 1,384 +same-ID cross-camera correspondences while seeing all four stable Move IDs from +both eyes. The same parent PID survived the run; transport health reported +5,302 produced frames, 5,301 handoffs, 771 sends, and zero local-ring +admissions. Full stereo calibration is still not implied: intrinsics and +spatially diverse correspondences must earn the camera matrices. `MimirMoveProofSurface` is the first Fensalir-visible proof surface for this chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 01669a9..b79f522 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -198,6 +198,17 @@ Get-Content .\state\evidence.jsonl -Tail 8 ## Immediate Re-entry Instruction +The active priority is Nightwing Move calibration. One public Muninn observer +owns two private Eye workers and one aggregate evidence stream. Odin commits +`39c839a`, `a340c33`, `8aefd1c`, and `f82ca7a` give daemon health a persistent +serviced RUDP session and move optical aggregation onto its own deadline-clocked +loop. The 2026-07-13 receipt at +`artifacts/move-calibration/muninn-clocked-aggregator.mpack` contains 622 frames, +2,647 observations, and 1,384 cross-camera correspondences in 15 seconds, with +all four Move IDs visible to both Eyes. Next collect spatially diverse samples, +calibrate intrinsics/extrinsics, and publish residual-bearing typed calibration. +Radius is a weighted range cue, not a substitute for camera calibration. + The corrected two-hour study pass ran from `2026-05-23T23:54:42.9041588+01:00` to at least `2026-05-24T01:54:56.9747630+01:00`. Use its map first, then implement the diff --git a/state/evidence.jsonl b/state/evidence.jsonl index c87f818..cecedf0 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -183,3 +183,4 @@ {"ts": "2026-07-13T01:15:00+01:00", "type": "verification", "status": "partial", "note": "Nightwing now runs one Muninn provider with private per-eye tracker subprocesses and parent-owned MessagePack hue/observation pipes. Current live proof: both eye workers advance thousands of updates; eye1 calibrated 4 Moves and produced 11,908+ observations, and Mimir received all four stable Move IDs from nightwing-eye-1 in one 15s window. Eye0 calibrated all 4 and advances updates but has zero optical observations, so dual-camera correspondence remains unproven and the remaining fault is eye0 optical scene/backend behavior. Idunn restart now explicitly kills worker children with the parent generation."} {"ts": "2026-07-13T01:29:15+01:00", "type": "verification", "status": "partial", "note": "Per-camera optical evidence and exposure ownership restored Eye 0. At exposure 0.3 Eye 0 peak RGB rose from [72,25,43] to [178,119,112], calibrated 4 Moves, and produced 3,312 observations/latest 4; Eye 1 remains at exposure 0.1. Mimir's persistent window then saw all 4 IDs on Eye 0, 2 on Eye 1, and 2 same-ID cross-camera correspondences. Do not claim full stereo: Eye 1 overlap is incomplete and only 2 aggregate RUDP frames arrived in 20s and 60s windows."} {"ts": "2026-07-13T01:41:29+01:00", "type": "verification", "status": "partial", "note": "After per-eye exposure tuning and latest-state RUDP handoff, Mimir's 15s dual-eye receipt saw all four stable Move IDs on both cameras: 16 camera observations and 8 pair records across 2 aggregate frames. Full overlap is proven; calibration is not. Only 2 unique frame IDs arrived, so directly instrument parent publication and RUDP send/receive counters before camera-matrix fitting."} +{"ts": "2026-07-13T02:32:31+01:00", "type": "verification", "status": "ok", "note": "2026-07-13 Nightwing single-provider dual-Eye field proof: one stable Muninn parent received 622 unique evidence frames in 15 s through Mimir, with 2,647 observations, 1,384 same-ID cross-camera correspondences, and all four Move IDs visible on both Eyes. Producer transport health reported 5,302 produced, 5,301 handoffs, 771 sends, and zero local-ring admissions. Artifact: artifacts/move-calibration/muninn-clocked-aggregator.mpack. Odin owner commits: 39c839a, a340c33, 8aefd1c, f82ca7a."} diff --git a/state/map.yaml b/state/map.yaml index 1ea4bb3..9869434 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -256,11 +256,17 @@ accepted_design: update/observation counts, RGB range, and expected-hue pixel evidence. Live field proof on 2026-07-13 established Eye 0 at exposure 0.3 and Eye 1 at 0.1; both calibrated four Moves and produced observations. A Mimir - visibility window now sees all four IDs on both eyes. The latest 15-second - receipt carried 16 camera observations and eight same-ID pair records. - Stream cadence remains pending: only two unique aggregate frame IDs reached - the remote subscriber despite continuously advancing tracker workers, so - calibration diversity is not yet earned. + visibility window now sees all four IDs on both eyes. Muninn's dedicated + evidence aggregator owns frame sequencing and transport independently of + the 15-second CultCache telemetry loop; controller state is bundled but + cannot create an optical frame. Its deadline clock is capped by the + configured camera cadence. Field proof on 2026-07-13 kept one Muninn parent + PID alive, received 622 unique frames in 15 seconds, carried 2,647 camera + observations and 1,384 same-ID cross-camera correspondences, and saw every + stable Move ID from both eyes. Producer health reported 5,302 frames, 5,301 + remote handoffs, 771 sends during the subscription, and zero local-ring + admissions. Cadence and full overlap are earned; calibrated camera matrices + still require intrinsics and spatially diverse correspondence capture. - `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a typed admission receipt with decoded frame id, producer, publish/read timestamps, sample counts by kind, source/arrival ranges, native reservoir @@ -628,13 +634,13 @@ current_status: Kiyo 30.18 fps, PS3 Eye 0 59.66 fps, and PS3 Eye 1 59.95 fps. A mouse in the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > - Use the new bioacoustic timeline watermark as the active acoustic sync - direction. Cut the next implementation in this order: streaming indexed - log-mel/MFCC motif decoder state with a global clock/path hypothesis fed by - `--bioacoustic-train` receipts, calibration-session command/replay proof that learns - per-output/mic motif/formant response, fractional-delay/SRO actuator, then - native camera payload handles and Fensalir lowering. Keep chirp-bin - calibration captures as reference evidence, not the runtime target. + Use the stable single-provider Nightwing Move stream to collect a spatially + diverse stereo calibration session. Calibrate each Eye's intrinsics, fit + their relative extrinsics from timestamped same-ID correspondences, use + apparent orb radius only as a weighted distance cue after radius calibration, + and publish typed camera calibration plus residual evidence. Do not claim + precise camera matrices from overlap or radius alone. Then feed calibrated + witnesses through MimirMoveFusion and the named Fensalir proof spine. Current app testing proves Fensalir WASAPI output wakes Scarlett speaker loopback, live mic buffers stay populated, runtime sync analysis now runs as a bounded rotating service without UI/telemetry re-entry, and the bioacoustic From a741aef9bbd9aeb5a7b1ae1741415f316c529e9e Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 02:32:44 +0100 Subject: [PATCH 098/135] Preserve Mimir Face state --- .voidbot/state/mimir.cc | Bin 110378 -> 110378 bytes .voidbot/status/void-memory-maintenance.json | 2 +- .voidbot/voice/identity.json | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 628459937b3bd06a026ede9a0af0638043d8c865..7a252bd47e1cbda470d49a4ec93bdb68e0dafd94 100644 GIT binary patch delta 199 zcmZ2=k8RaGwhiBTc?_+LjjW8#^(@Rxqb4$nPL|;7M2MLifyL&{XWM*&&)5bbVPOc8 z*ev5cKL| Date: Mon, 13 Jul 2026 02:39:41 +0100 Subject: [PATCH 099/135] Gate Move stereo calibration on observable evidence --- .../muninn-same-frame-correspondence.mpack | Bin 0 -> 341181 bytes .../muninn-same-frame-stereo-assessment.mpack | Bin 0 -> 551 bytes notes/current-system-map.md | 7 + src/Mimir.BufferSmoke/Program.cs | 90 +++++++++++- .../MimirMoveStereoCalibrationAssessment.cs | 139 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 8 + 7 files changed, 237 insertions(+), 8 deletions(-) create mode 100644 artifacts/move-calibration/muninn-same-frame-correspondence.mpack create mode 100644 artifacts/move-calibration/muninn-same-frame-stereo-assessment.mpack create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveStereoCalibrationAssessment.cs diff --git a/artifacts/move-calibration/muninn-same-frame-correspondence.mpack b/artifacts/move-calibration/muninn-same-frame-correspondence.mpack new file mode 100644 index 0000000000000000000000000000000000000000..e70c40c1492db939df2b97e43626cfa94507b861 GIT binary patch literal 341181 zcmd6wd6>=R|Nmu4wxLB*DI!Xiv!6MGoG~L3DM?5OgOQmCgDgoYOR4O;G7CweY>}lb z>5z5E7D*z?ma_A^o!9&8yk7l&*YE3n-`91X=byQ{uCMugAJ=nU&*$rP-|zcZ`2YDo&%N;vtETi#sp_16{r_!M zhr{9P=ykbMTpgUdE71RV!i|5K5dJqQ8L1so(%YvbI9(1$APhCE*7EN|Nz3x&2_;P& zlqVFt-2Z&2_P`5!wE+AM&&%$B-|w;wQ0)A!P}0WMr9+7YQm*O)*iyhrf0p5IS6+ot zKmZsi;Lxi9rz>0oBl_03J(Se?dJRlbKY}Oo=e7GMhyd8C0RfP^@#A- z;0i!;KmZuIKvw}z*ah#U9sB~61029efrt(5C7~~@!21h2o&x0n2Y}=N1Hj|}2Y}=N z1Hj|}2QYGhZVu2y^kHKCg`gbZ00;$KG!c#J|5!Xm0RaF}fTDPG_mn?C3UC030#t-! zTV_8AQh);>6mXLZ#x?Fa2BUxgFj7Es9&ppTV0`l1YDs`t7YG2vJiq}U^MC+A%mW+% zG7ksr4sZaY9MG)+8p|i{ z+44K61~>qu1{eUQ1~>qu2K?b1n6&wwZJ-+90FWAB0GJx!0FWAp57)rtHI07&)c^;8 z)Bpp()Bp!CssYW3$U_s+w6m`d$E=J5fYC&xs{pm*=}$b?5~KhJFj7Es9`H~znm&HO zXP6ui0Eism0FWFI0FWG@Et*fN_k0E90uEs00?i!o(jL)_3N@!<6c7ML3TWnlm-dJ< z%e+wrlLG<(kpmn6k^=$&kpmn6vPUEUATFSGoVBoRGmr~70K^3h0OJAY(NzU{;Kqn}?8~{=S3;T&uCbVz zPyiTpLtO=Ef8f){djAVjfCE4jpw;p0vZG6Y6yN|51!$A%vyz{_08)Sh7%8AR5BMmE zK3lTrS zXkjupp~d4M1vmgi0UFEaRWE%DNC6IDq=05G@KZCIH|Whim|h?N5WRo{Kze}yFzN-m zH9+&g{9eP|pc>!+kQ!hBm>S>!Mm3<@4H*Drr_|2@jFM1Ofp}USe^sp2pO{`C03Zs` z(rm%GxCTe>V}JI)yxl4fCC^D2spx1@nToazcC6303!u7yJ3JXo z1puNOasWs-6aa{B7@(!u*Y{;s0J(qz7`Z^R8wO};wzNds`#}nD03!u-bAX!BvI0}p z{R4=1OadGLk^>9?lLH(8k^>9?G8Ky&fVhA*D3-lYd=SV5901}127qw^2QYGhUJ^PT zG!rhLcCI<31|$HW1_S`21|)z{4d~4S0sxVO62K@4brqnIY-O8f0mww;bZ`KO0yL7X zdTdE!kOCaQNCDj>q~-Z{C1<9Cl8^&Hl8^yll8^&Hl8^yll8^%!xj-`soiq=uE_}duQMha-=fORq2>K9vlgvkK`fXD$30LcLXfXD$SHKVmh4xn7X0U$1*)$zK0S(tOh zPA3O|xPSp*T)+V!E?@wVZYXL1;sSbbxxVL@TS2{m13+BB05C4#07fp*oCmCH!G7rS zMk=NT1OTE2H~^#u1b|TuXimj0I@-OVzW)$rDi#1nNvNv;^@0um*2BDg?{skhBL#GG zfVOHk*4uFnlmi?9k^>9?lLH(8kppg89shXX5@?@L02sMIGY71T(Kh97`Vo_a0>DTC z%^YykR_*5AyQ^SwKmZ_efCE5sKmb5;fR^W5R+LjGs{!9HaJo5wkqdNlfX4D|7c$g4 zrw|1=073x|oh93TqVr0O0s_ED0nH@z(5c$o+J{bKl28C3l8^&Hl28B`C82H&P&eG+ z$*lvb0S*AE0S17n0S;hP1Dd_S!vG+?KmZsep{@e-*zTuq-&3b(5Z%zj0T2p!X=Su) zQL$YZ1q1*@0qTakFZW*pQh)ECf$4?<0MQLO z0HhlV07N(R(k9iuy-(c>asdZ0a)D+yw5|jCdGnV+kOH(eBmg4?G;_d5-EjY!JzrsR zKmZ_efCE5sKmZ_efCE5gLIGgp0?iz-u7o=9b^Z3391s8y1?ZvG!OY~#AO$!8L;+fu z9C~nC4UhsH0HJ`NdcmRn$?AJXh_#^rFj7FX8(P-^{Z@TU0;U@Z07N(B0FZ7d01(~K zPfN4I-(UO=Hs0KIyqy`uO zrUp2GQ4MHL#ql%`9L*`K&JQ5w0Rdo?gt`jQM&%!q0uGP@9Kc8c%^ZlQCnm=xHkpmd z0Re!>0S*Ak0RaHX0bT=;9N+*D7tly{yk_DXp``YC)bePG`Vm|-|5T{Lst!GNtl(dwdul22#p4zE%&)%u&o#x*7uc|41Q>sR6=y!Ntb_e`^*X|1R+X*-R zWkTAk87Wm$GEzIFq_ zh;A65q4ea-6O%D65C8}lZ~%x41OUVZv_WxdX3vTs7jOV07wF~yJ-R*f;g$6u1vr3_ z0(ujX%es;2Z%-?Aj;P&TYNKefuTChqW$M$>nTjOIG*vT)Yk9 z0tsN`0=*n?IcO;Tcj4C!K?-mHhypZ{otr&i97q8UV5ER<5>gbOyIlQ}O2Q&DR1)T0 z77DJc8HiRAasWsYG5|~xasWgUT35kb7*nho#svbv$OW27=%j7Bi^GZz1u4J*APUe> zdTH~EwLl7R03!u7lhC>d<#O4J*JnaUgHY0>^Gia(PnZ82r6d#ph$Q3ykR%iUkR+ri zzOv+UQYdNZAhlaQPW`Cu&HuOE8i)juo!xSmlLHvJAOc`#uh?Z>19J8A4RbIG2mm7m zG;_d3>w>G-QomL?u>D>03dRJu7b(RGrr(A zAQyz68Cd{EF3`<^@X{>nmJvU$11Z1(APR(|I4fVTj3+?~Z~%k?bOl6Kfi?vTV-yeo z5Cy_RY1Xa3CaO2D5%U1uL16(HDG+H!w&nph*|6Y~MW0enOm5A-etw{BC^)O>7g6el z0szqsIRK;^3IL;Ss9OW!O{%Pdf0k0;Ifc{!2Y}Q71HjY(2QaDu&3S+>Ldh!h_Jf-- zNhkn}l2BKH@D@#0;XaqY1S!A)j1@1snk4g7D$?tYQTwyr^7|Uw!<1@*j&s!P)EAM`{XqSZ>h5~Pc|t!2fG9u%$%D0O zR|P4+0T2q%)o@v5-<{DMqksS~Qb2Pah!20&E353T^QBZTsQ0g$2i|QI3O0RyMU>_N z0f6WQ901Y_1OTEJ#8VKJPx&+zf<2cviIxjE z0K^3h0OJA<0C52=%^q%CYd^>Z901{hK=`KCtcpoT8ev=@01z(V01y`l0E7!T0Awl_ z01y|@L{xF?vu}c2zyTmGU;r2wZ~%x4=&{`+Rr~yn=>-7}VB`YL)p3AEvPXMt1)VJy z01yReaay_a)V`n^-~a#x+;lTr)?<_2_yD4S1Ta!SZ*}Z;P!LrqzBm!m3nT!b7YG1E zFOUF$Uf_066jybAq7F|0_6a3`kqh*Cft&88%c_>qT3yZvQGl8e0T?Nun*+2st+xN* zd!QWP0FWGD0GJ%$0EirL((2gpaQ1H)7YG1^3+QUNEXVraCSqJ501z(V01y`l0Ei3d z#b;OC1@-*`L@%K00ka6e$OW3+&`D2D-A#|HuXsQd-~dJnXm&%o8ZOH{^MMke9H4oC z03bQQ05Cbg0U$X*i&O98yB-0#fCE5WK-=}+U)QKxIswm&+;l%-76Cw9z!bo^fCC_0 zVBJj@-*+=;CKLb&7r3Yy1@5Rd2;%|)fN%i^fVe;aKwQ9U05K791(J+hpg9w|XmOg* z!ZQui4Fv#10gB=pFEo7>qyPsnQb2PmcGLbqjoAaAz$BpnKqMgtfFz*+KqR4?o)^?? zR6~8w2$6&Wz{mxfNoZXYS}SpP7mxz9K|ugU3h3qlJuj#=`h|NzIluuRIlurgIluuR zIY5ikV7Z2vXUA?g2Y|SM+HtV^KkYy+-~bR8FaV4TH~_*09*W{R*Y-D7$OW3+ z&_gSuO$`8sU}xqt&eT)+S@ zF5mzV7ccvFt?2giJmae)9pxPSvdTp$1tF5m!=ULXLBT%fr>;G+u?pKdS@^wdlMKop>@+Q#>9 z$;Bk0j{_Jfpt(b0T@Tney*iH7}ak8l!*!Fj7D>2ds#0SAD%fTFl%!#d?aF5mzV7cc;f3pjw03v`o^9$35>{HhA51~>qu1{eUQ z1~`CG4d~7T3;?n+vaSiGA~Z@uT?J^1rq!;c!$G}(0~jfwnFH44crW!k{|6=q1OOrj zH~=IE1OOrj0<=%~(vOQ8V_YBrj9j3Z1J>nuZ59X5fE1vakN_YGP%mhEW%w140vy0d z0o^2|2No|ku9OB!LJj~)LI!|ILJj~)LTbnDjt^Gd5OKObzySa*@K`rAwy#x5y>AG( zgF*rTTp$1tE|35K7kC`BI8FKC%pr&iB!H0%^yUGNgMz5ziXYW;LqHBl0E7b8)o`h0 zs;hTS5ef(ZhywKNIQ7JDl|Tw`03!wTCL)h@F< zo9>l;$1(34dISK(1+*~fT6K>)Sq*Uk2Y|SM0bpFf0gPOrn}qb}HtnIA4})re13+qk z0bpu?0~pnS=1l0KU9$A0llNlg0Rdo?gt`jQ+xM@|D>@pa00%HqKr;ud%kjEpC057e zfB-<`00)5NfB=Bx0BzM$4IoxVTmi)0bRHMA4!|d( zddH&F3j_cp2WYF7Y5?)ar?Ef;VAhzn>Q7<&3+%q^WB>#A4+fVhCRXoe-fj=8JWx{QH=RNB|&u0S6Gh;8K((A_0Kt1=bajA5Ltm?j=L)kO%;T z3#Y9s{|60~onLb0UhTW;D9@qO%|cH~>Te z8cN5u{k1w?6O_w+i9$pHa?$N>((PSF&8y>yhip#VVSfOS3Kj4xYYZteC8 z0Ei1HiZe^DO9i=r13+BB09ZG?tKhh3xqt&eTtLkz>%}XzFulOZ0U$1*#c9^H)q_DU z-~bR8FaV4TH~_*0F4`ZM`Q8)FF)k1Q2p70$e_-a%OWI>xAOH|9-~enQ%KvJcDD?sX zVB`YLso1)kZuWgGzXsI+?UWJ#qy`v3a{gbdse8%xJ3_&PE1A)%0S*AE0gB?;-;_`n z@c`a0@VYnvqy`uOrUp2GQ4MJB54hGn{FAPsaOC&6rg>=<#j%O4O0U?4ggVra$tr3!44n=IDnA?nkysgwcizc?|cgJZ~#aSFaS&rZ~#aSFaXH@fR6(hxj-`q z{4|uVo^VG1lLG?4NCC|p@Y4f})o0hK`-&0sfB-<`00)5NfB-Pc0o@v)M>cE5AD;uN z0S*AE0S17n0S;hP1G-5_QM|5a<9whb^%G+1HX$4V$0~jfw*$v}qX%;#+xEH1y z3IIem`z>5�&2$_CS|+>asdZ`xPSp*T)+VkE(lON-c)H$2-6D$ z0Kx?US{-ld^Xn0e3j_ed1snk40s#PV0WHn8l=$;?kPA3~kqb0eMgeL@TMG;x08)Sh z02J^!=w$V_hL10TC?Elh6wsTBeGWQ7v2E&ycS4d-0sxXw03ec30sxZG=b&zw>+N_O z$ORn0$OU>y=yOmM@2Kpb3Q~Xr7%8Be1GGHf(eKnoP!4baNDeRnOb&1Wqa4t!0cyuP zyWfxbiigj^0U$NN05CPc0gP%uHwhU4VuwW5fKd|aDnQHgpYHs&A*dH{0E7ZgnhAH^ zHm)s30Rdp7fMyap>HdLTEvqGBx}g9-Bq0ZYB%uI6B%zar(%onJ)CReL13+9rPtEom zDSIB|0uBIi0RzCefCE5WzyKigfRh72TtG{+y~As)1i63%KwQ88FfQN#2p70$BHCZ- zN%fUaz|(V|05Eca=2UE5hI63En(810=%Ez>Kop=L`o%p^ov4J=00%HqKyxa#u7WxE z$;?xjBoqLMB;){)BoqLUB&5aZ!Lucffn2}=5H7GT!}+!D#LE~L2mpi&H~_>20s!Fx zH#MU}KTpEk*z6Mk5Eszm^tWw2{2&)_0Ei110LBFz0OA4$0GWu~901}18p#g#`gk(P z1snk40tSF_0SAD%fF9c&X}KHobw8h*13+9r&y0@DyYMT>1snk40tSF_0S7?1z`7Lc z=()$#-Svpou>de~f#wd0hi1ZK@m(5$6yN|x3TW;Rc&Hs88#(`TOb!SDL=JF(b~L+&WD2U@4p(I`-*)60FndLjE?Vkw-Cq$9Kgs0nmJ%yjCP{L+3pwx1b~qOnmOR5 zr{^aYc>7>-KmZ_efCD5K@UIFbbsIV=6de6XMwD_u02t+fZVgZ`IQhbi)u0;S0FWAB z0LcX&Kd5@a>#IY-@gJOtRt<0fNDa_d?HTW?Oi&GQ07wlmfaC(fJfWoYJb6OF*%P8Q z4|q9%Q4MHzL+g6Lv)}dp2-6J(fKfNpRe%PPe?R-HGDraq08xO7@Z6dH*FXwz0E7a5 zIzMp!NX=Y~0s_ED0nK^9Pcz|#_?_=!dVv5y^a2jRdcpiQ|3qmX5CDi?U|ooGacfm| zo)ED|Bmf{Tpv~$_YtJnQxqt&eT)+Uh3%0e2mJ2ul#03lh;{pxV7jOV07ig}It;_MQ9cw-olLG?4NCC~2QG9r1l%1z$Wzf530sxT% z9Du%e9Ns1Sa_ZAj$^ij@$N{<@F#DF=yBmXC5S|Gw03#RZ=0JFpDm(A`ZRsEdIDnA? zx;YRYNV4-S-!dLE576a!762p%m;!7b_^R5jXypJ0faCxJfXoA;28>*wnFE3FjZE1E z3jKH&CIeecS79WhBL01!z?*MMZ-cYEbo7#9ctgbO$TcfqZfqT~Vr0C535 zzQ6y|Cq{!@zyXY0pqm4j|}IDnA?n%&StJEadKcDogm0|Ee%0~~;zr24A* zv?%3(06^pb-NKRm;LiiLVq72qj9j3Z10Gr#m3{M%r$7pD0EhwuAyF$~`uVEAr01ySJ2p^vOI_7)FelG`rC_n>A#SfD(U;XxbIe?J@n)85{mS&aQ z&DBf@c+bc$01)#42Vm=hg$3`8(mWsl5c2@N{-6D5_gkA_l28C3T;QVzmzD2Yt)3Sk zTp$1tF5mzV7YG1|3uqvzk~wKU$ORn0$OW2Hv7aWQY9ktgPSXhhhypZ_RDa{&zcDr7 z=Kw|uXwC!l+Hba_eC3vyBoqLMB;){)BoqLMB=l1dIakaY4srnpK)4{DcFA0~S3ZYv zfdD|bfCE5WzyXrl3{uyEO}Tz8*ve8Z?T+P|-Ozg9(923;^Q-29VTd@*~OxNdrT{d3TJ8mJ29~eZGQIKrY|_MlR6Ifk60PvTT3<4w$>^ z{Q>}@0Idt+yCBmhJK+N65ic^)$l#9J@D z5`d8cdh;^jaCx!8UWnXDiwgZ zfCiG_iJD1X6$lAQZ4(`>lU( zv%fG32mpuzRD@5y+;JL60S*9BfL6y1Ugy^)@>CMzxGDH#z z03#P@CZYAt>GM?z)dwj+>jDCRC_rO*v$7X4cUH%HIDnA?x=Bcn@0-0=VFf4&IRGRH z82}~;IY9A7U#cXmvMCf?>wPOqNl2fb&u(70-Udt#2mpi&=)>#T&99Yt9^(Q5fN%i^ zfVe;a6!$e(F6g9A*RL5CrPKBC8~`#82mpu+Xld4BM#0BHF5mzV7cc;f3phY=*Hz_$ zE~`Sp4OL!=RxhAt)GFbje4t*y0U$1*#c8V%{nmn9zyTmGU;r2waDd|Xs?*KO+TE*q zLAA%C!>DOz5ZOdB?Q$??5%c0U$NN05CPc0gC$? zs2X@woh$yn%VE1Upg9j%A71Z#@rlxyBoqKfNvNv;*|5u@8tXs`Z~%w`RD`eWdblx2 z0ShGA_uH5zxVkgR1%W|0>H=xnmOR49C&^EXmv9nV09c200;$KbZ2${a*nwe1q49c z;1Qvujn54V1$RvTGCC`xfc3%U{%h-F?k$%9ao2C-*?9fV>MN56+f_ib7r5xL-GI(t zJcH>40w8y#x}R{5EsPe@Pg)VXKx)bvgX zX|HCaR87f9?U0h*J|)5F3UB~OFAxBTUf`l6eB+;*13@{!0pcp|R4ypdG?bYC>PfpU z(Ch_nO2R?M?*0O!fB=Xqr_T0nELSp=c-ydjb`^-Yz}M01a;LaDIB&)UZaPQw=KiD8 zF*zUra!>SCIZ*0rb*8j@5q$-0B^GJ%mE9W9 z%>n9$gSQ{-1j+#p5SRFnQs9A)RT4IOEm{iDfyN{(y2pi|0a#1s-_Nt_w7Kf%W$N z(BF@C0VzN`r34VyP~9EAu}!&9VxbxHqNM-@(eN$po&YJp0pjL&R|-tos(QhnFGfiL z>&530YX?1sQ9uC1joYpinDB<`1(mwlRUndtw$=sKORw*B?xAkuLF^R^fZXNk&frZI z$Eb6~v*zk6V5=8!07x$o0Ek{-eRO+d-Zd>jE}$eNfVgpq$^}F3RqKLZhuEzF&0b)A zbbDk@dVY`sG!+v-T-Udi0`D9OB^G;msa*whbAYyJK4@^)eV`oR0J(ptG};tUH?wVU zSJGF&UJft-Ob&1WL=O09kLbe>+o(Mv#1j(%Q0$y~dbzRR%gP01&fBd4%^a{kJO0s6 zw>}C|fc6Inpx9xjQsA||p~Rv?HbhGS3Zl`gR@@CzfCI!ezf&pDAztl}IHGhzW57p4 z>6nF2ybn@<1H_G=trU3Pp%kbT7o{4oUUnPz_r<{&1q49cqg|B(-BZ-c=%o|UQh)}M z@e^(<2U36o#I;i2o89ZP>k0v=ks4 zPJDAjO^^Z{ATHq*rNH8MlmfnQqohDQ%>$FxJ904!2!Occy-I<(E|mj2Y=6r}a~_DN zc074`Ni`KCCL#e4SLFI$@w$Dg20{--s|F~Drp!ON8>9dSh%5M~Qee#@qrOiGN3OW~@ zGXwCoOYwHRnL{iBz-Wg=w+5&g%_!GMea8e+0~}ySk&UVb8s$|_PKUHM`xD&sX}SOd zz|;T-FscE~nUE`h=><%Iq_~QwR1(hgsNM3#@9Ecot!GBo3y+z1SE>$j0Zl{%u;V_p zQ?=FRdAk^Q1o z0}KFD102Ap26U5<0br7l1H?V5ZsXaQeL_7i=(^f&4QMIgb~tELgWq54$y>)eBS+(j^!(e_qlX(F^_qkUQl^)e8zuQfJHSJgu*Qjd?%< zB$_w_)j6Wu z*8FMrPoOyuSRY=W`|j;(qdfZ0jNA?Z5EpEr6d0h+kmPGK!>$6F^MLi@^SmG1K=uih z0<<|SHPYNm;%Z9 z_s&w?aMiI;@XjvJm;v0p3pfD81=NfdG#%>!^#TqMSNh8Jnb4_b!gJ5*{|Rilz`6io zVH1~{$PqK40LWc&RJp*{ITYM@xU9Ycwp_pglJg(vtM-USp9%$Q?)cdZ;HF(501z(V z0Fa4D02DhpU%8;mG4 zJr^(lj0-pb#08$s_k6oBd-2|R3o*UG%>jySx};q2)FibsI@&l|E}#vHC5}1jY&oPC zaDd#o>izFcl^dyb!LZ--6|m<527qw^2QYF$BnRvzA&q29w!fvuK}Zd707wlm089;V z0HYev+##_p!C1Pa^zE1=6aa`M^iVTe78h8Bae)9pxPSvF7pQHzZojL=>Fsl?nMLtU zPsIWN;sR>N-!$Et1abifFmi!z4$v0O^2y%wAO$!8L;*U;wBqr!PQA$DqfJj0w1<`ly^QxH;@x(*`KwLm$`RYbV86X#M z0Ei11Kyv=y)#qH&de;mE8_f7CS}x!K5Esx`zNS&JQXm&_03#P@&I4ZBtX`Yg;s=ld z8~~w!bq~q+kKI;5eN9RoB}|$)NS!AdpU%3r_xISOYhmgaQEJ0uBIifdGKGfY(5z7o7XymuR_w13+9r-?rO0uJT&$ zf>Xl`t_QT$3;Y}a;sOSMaRCQ_xPTTWo2osl-fxMx;C#WXXt{s`KwLoEbeqfj@8m8x zJ>B-pZkiL3p94T#zyOl-|Ds;G>vm$3I!X2H3cEER6}ah%h${d&SIhyp3rbG1tAOrA z#Qp?H5EszGWb?X5#k%11`HFTGkTr1g6A=f1xPSp*T)+X0dO_q*VC(6*pEfAAENkWF zNqA~clW5fd2Y}Q717N%5HMgdkDRA>i$N`LMAQD8jY9O8ll5Nk`S1Thn4jFK<{U{3*hUa-C1y5G41=VsK8mI53Aq5w6c+}=H!a0SliKN2kkIDnA?nn@T> zQJi~VMq!?WXP&QXSAhtCJr{5QL=tiZAW0|y5J?!IU9z1O3m*_JaOAgJ1Cb!IRRaP5 z;R5S=z@0xh)!m4ZU9!^$`q))K(**(m;Q|f-ae)AUxPV@E+f{tBI@iwTfey{=)_|@H zH~_>2)Q)%OA2LU{U`)DQ1$14&0U$1502mi=0Ei2y9q(!Iv-*+{bHOl!%X01Y0uEs0 zf`|=mO~nCPnCz{+;AO7BxxKbODHQ>*rvL{qQXpbOI|?`*G!N`ObX!4@1M3S!CkG?| zAO{2hA_pXZQ4Z+V0IiPqy;E1ceIJ>K&Xl&jHjDt+ae>pp0U$NN05CPc0gP%O5=8cL zfF4-v@ApbBPr~yHCfLn^2!OpB-~f;$q~-bkO~E6=1*6;ARY2DT901}127qw^2Y|SM z+VL-Uy}Vtx;BZU3HK6MP4q)Vh2!O2|aMI)ZgUx2Dw`|xDIfqK3Q!OoO78o) zP~gk^qon``Fj62AMD}WcYz*aA?Ze$_=L=tiUND>MFL=w7a zUGRH_5|y|ME^HfRw+13XWUB@Q0Kx^K$V7DdK}WP| zfCE5kfB|4?fCCuSfMyc9X|wudiD~L4mdMKJ!koT#Yajw(F9$e)Q4&S~Y`MU?0^;;j zgQsu>&d;}fX?6s_mI49*q5y5voj&Ol@0gtbx_Xpqz{3HI6wpjU4-F(|4t+OOBw<>% zT?HZl_G*9wAd-+P07*gtfJj0Qz3leSOQ-hp9Jo-`;A6Y?YJdYkTtGwVKil72EnLv| zwrIJ413+BB05C4#07fo|1d*+7=%t;~e^Ywb6*aJHL6mAh03d3B13+p(02tLkWFWCu z1GH0m{><$^^CUd?>nyu95CO22gkBC{l!OrgdkRn#UpSIi-9;0biqHMqC0Yt_0E7ZQ z+9|!{yyqdV!1?>5wMQfXAPUe>dTC3)MnZuazecGBd>p_?0nJ`uT>)`eYbvIzU~=-*ds2PQiajrIZn5iv2!O3#AOH|GzyTmNAOMVN zAQD9OY9PE(nRCnj&mRy;_-3Nr8i)Ycs{syRl!OrgdkTc3I49qFPY|R42QX3~Vncfh zgo7w2|0_E;i5!?_dnSwk*sB2!0LcLcfXM+40Lg*yMrBU^ec|QjBfXKmS;$!wLjo;Z~#aRFaS&q zZ~&tk(3=Nb4r)es{J34ceGlk{5&)8fv`u&CcNwpMl8^&HT)+S@F5mzV7cc;f3pfD8 z1vHY~m9lvV$ORk#;sOSMaRCQFxPV@M&nY>_{RyTQ2mm7&Xm&##1suT01)51nS3u<4`%pP`mW-XEiQ4K|03d3B13+p(02tMP<~-n{dEmaQm*uV9 z(Rym;5&%X?sH;HuI-s2U=T=&Z=>;whV5ER<4$uxsnXDEcgK~faKyrWqU~+&1KyrWq zK-L8=4ghfhjbsnhtMeDg1snk40tSF_0SAD%fF9pJ`2LHvK)rwiAY4FK!Q_-3T<;*p z1p)x!0uBIifdDXaf#y8mrZXhv9{%7SPz`VZNDVLmObu`VqZ-hhirv(X%YWIt7bXb> z03->i8CA%4Gz4-12Y|SM0bpFf0U$1*cKq<#1snk4 z0tNutDfMyyhzsZhMK#}^e?Tta01y{20E`Pb0K^3}l2sqjF$L5MH~_>2^!VQKT7h*S z7jOWG3m5>#1snk40tSHT1suT01)BQ<+!fZDuGEt0f0zC4gg6)0f0zCA3ZPdd~&TC$ORk#;sP2-Jbw>q19AZefVhAG zU|hffATFS7I^T$;pdAt)2SB*Mx)#hos3+(YjQ~KnfCE5WAOH|9-~f=RSO7p=Ku^u$ z%MWm4dV!w+W7tq3_#w}CT2@1qpHhvBOaRCFsxPSvdTtH)a&BGNicT&0h901{hcsf5& z>+y-8cT5BT!UY@v;sOC+0c|0I30{0Hy{wfKd%-J}-!;c3gXP zMA3Kop>6l=#)Rm^&(6@f^TN0nH=~P&ceopm;vaOeg>l zNyq^pNhkmiNod_unmnrD?U*DK01y{YFGxO=Is)VZ4ghfh1Hia|13+9r&FG1Ktuf!Q za0NI3#07MOquvAKD}h|V0U$1502mi=0DueJ4zgi`r`xWD^a2TBs0f49h z4gjeE0bo=Enn`F~koa_u#?3KFC;$*iXk8EZ^qMc8!z7^qfVhAj-G&vwxPSvdTtJU( zo_VRzB9IF>0K^3}kUabBF7dVB)6u$}&+X&@5En22j0-q`kqdO^0otg1_U!f?P!e(g zNDVLmObu`VqZ-hh2VC^X=D8YGD`1jP03edkx+1b^_1qp97YG1^3pfD81p)x#0vbq~ zzPbHwkPA3~kqdNlfCiH1GiLih3UC030<=TYEbA||ItJWb?{;wjBLy_)0T(r+7fLK2 z2}(i^07*gyfJs6QfJj2?cE09KPxQmMKmZuIKr;#56vZtD9XJb8fCE4jpeSzn_<*ND z3UB};1vHb;O-r*E-R+*jB%uI6Bq0ZYB%uI+Bq6T>#5}+iNP@V429g(dzTOw)0uBIi z0RzCefCC_0;Gt(mFWo!Q^5yZ-^L3j_ef1vHX{6~MTF13+9rGhxbuv(iB>-~bR8 z&_I&1x@`eaFW>+W7cc;f3pfD81vHR!Dm3s(kPA2f!Ufhfp`GST!@PU$761qrZ~%x4 z1b~qXG*?DGI!V>}llGul2$kQ`tD7#DB=BNu4qfOSb` zTB*O(+xLhZ5C9Mbs2Qaf4XF2wAPR5*BLy^*&`-J20Q|C0SAD%fVOD5&6qL=TeYDT?_4A}=# zfCCsQpqqrWMbrC>wHrZ6$N?Zp$N(@&$N>;Z7@(6>ea?1#3F86*VB`YLBn;3z@Oq6& zT|o+P0Ehw<#r+(OFMt%_07eRECSib<=lvfUcNZoJ1pp!mIRGRH1pp)oc?}?T%eew( zZ`0J{gKx83_Qy%7_C%Rz?BSZs07Ng~0FYiF z01&;vOGD|TgwNK2T)+V!E}#<>linY`2;>3|0C52Wz_@?|7`Z?<3F(2wWXH1(Pz`VZ zNDVLmObu`VqZ-hh34OFjG-drN%y&#Y0>CH&05Hk{-5Q`ntux$<4}xld13+qk0bpu?0~pnSW)k{oVUo43 z;d_`Q6aYp^sHuSU>UYlEZ)ai@5CBFBXy!mXtqW#8cl&fq4hR554sZZS4hR554#d;C zVAk=oeLybY07fp*%>hcnPtVN+ohy##00;#F^q$dYKfIcVsR03Cq=04;S}(oMx%ZcH zm?RVch$Q3ykR%iUh$OV$JO2F3gX&v0fcFhO0s!IyisCP3F5d}q0SAD%fB|4!zyTmG zU;r2wZ~%x4=&{}020Pybxqt&eT)+S@F5mzV7tkY{c^~C(0qO-Dz{mx9Q?b{2_k8|O zv(g|6NC1cev_)4R8QR4KM&q4R8RX8qiHbYR9Yg-n|x- zgdD&q33U~qX7uf#WfMUPZ~%k?)>Say^;&WjqksS~Qb02a-82uZzUN3AOcDwJL=tiU zND>MFL=w6wh}LGG#N5i|6#x(y&;yIL=iKu_F5mzV7chY2{D=F8lDamy5DLCOYgefD zzzcgPE=}u|o|>Lst!GNtl(dwdul22#p4zE%&)%u&o#x*7uc|41Q>sRS$nWsH>}!|e z?YO%F{dU5Qf0>Z>YDP-cl#J94De3J~5}a3O~spc zEt-Mph5`W54LLw^f!gX5f8G8$q-MeuHKQ~W3IIem^wI{!)^1xc-?sA#0E7#C^rqFe zltuY4E)W0+7jOXO0yP?^^-Bx|$BZZ$B^L+)gbO$T;tdP00D!oFRz}+k<=qN$0SAD% zfB|4!zyTmGpph)M@np4fF9ZGNPY?PO)IaD13+BB05C4#01y{208B67 z07fp*+#m4Kx?tz<9lJ0qBLM(WfTrS~Ru)k=G9k{C`Z$1*0-9?>>q4Ym53N>jUL)Qw z5CDiIxj;~T&Lyq?u2697E%!&u1suT01-eN{+jM*L{Phc{1~>qu1{eUQ1~`CG4QNh8 zewv8(HtAXhl!P1rl7#fgW?#K0)DsiPJiq}UE?@u{7jOWG3m5=oD)w^#hzn>e-*;@x zUXTko0K^3h0OJAY)Fx-*zwAOMV9pxF)MXuMW(G3GMk{u~Cyb;I+9Kgs0dfm`x-O6>eLlJc!5n!)a0zed? zH!S{WzV%O#0vy0d0lg&jIcOp})}pdHRt{JlO8`I;3V{Dv7wlL2gjE(rCkZ70APEHk zCI@&C8o59>327+(GpQ(MUEp(Y0Ehy#IQ{E`QfeZC^a2imP{2utSWdV)Vy0rB05DQO zvlm#GVx8F8zAB~{2mnMc-~d)H7?GlSLG}Jo>IDJ-(F?2_nNHoG^#~>h1b~qXG<$)Q z)`q9s7u^I>fCCsQpqm5Kj7~2&ITVxw8~~C73;>e@8~~96E;?KOcf(=BF)k1QMlR6I z0qZiHvx%RjgA|~(ApsaEpqT^KRWN6N`?)P92Lu2j2RMM01N~l5^MIpXl;!~eV3Y&8 zH9%Xn|GfLR`o<+4nR53J?|OyZ~yQ2v$!1e7$b8;-x{}vl3M3c@7_#NKY}Oo=e6q%t$_XyKxD~!x&Qev zK%R&y*Du0z2mF4QmMS-X+qNpXD^G-yHa#{z6x{I4Iek_10g9bh@5pX!T{@IlAm!?R zRk`uk)>7rhZ{L(E;l+LC`wKd%uOC2ag#*M)t#Bwvt#E)?sTEoPW{yvOTSW?@Rs=w- z)QW&Y)QSLzm0F>T>M}FSyrJHFL)3}@h?QCqP>5O)0I^al{_xgX=KNm6-I!Vt0I^al z0t!(p0w7jug$}M~E-NtgdrYkefLN&&0fnd)0T3g#;-nMynaii0YYwRu2@oT-BB21a zA^~EiR>GT_nctP1sm_}K<`pM>n%Dw}nOfnhKx%~p#7eEu8O6+X`?3--wITpwrB(zK zqE-YztkjApyhWI~!;`CS21MkF0Em@Z5m1O)5dg7LD|DG_=KeK%zQWXs0Em@Z5m1O) z5dg7LD|D4X=CO%QW@Bna0K`hI2q;9Y2!I%=6_+QxzmgT#u)rswr1sq}gpwwAx)utK zJ1|xM$%mbX3=&}JAhmluPW`Cu&Hwj*pM2bSprZBU5O)05MW4 zUb@XVYije0b0M`N0b-<9Bov@lBtWdx3Vo_GYvq7!^)(d0yy6uAu~I7n3Q;QpAZBWX zhNy2#j>-Vl3I~XpTH#QTTHye(QY-OvekF8na4$@)2!L3r6#<2)6#)<8gRsGf$`+KmluJAKeI$ zMF25VD_j*wt#E)?sTCTyv-2(AG9FVa0w7juML;2HMF7M|t@xec0?xkk>x@>AT9E)T zQY#V)P%9E3MrtMA5k8ZUUAy8G%sNN{#7M13C_t@9fLN&&XZW3u>^d)PI1b5`cmWVA zwIZMpwITpwrB>YGdlIr6ms#`;rd9+%tkjBtLez=?h?QEQOYE|nUn}!Grd9+%tkjBt zLez=?h>=?p z>MrE!O+tx9T73228v$-yG@x}Oz>VL&i5md|f$$!G_Kb3k;xM%$0Ai(91k{cq8&$0| z${PxpZbEw z#=~W!QY-%fD0Xtbs+BIsRIaR=46GGD%_|%0`u^l!^*H&{NPSgoG(opR07UL}I`_jb z|5fEi!`JG}H-7tn@67*!GWyg=?o_J^#YgiU5d}S`ko)S`h#-Q!C*;{+xu?GY)}jg#*M) zt#Bwvt#E)CsTH@I)*I=A5qOZ~X{UD*_-^YDGXH zYDECVO0D=QSGtcMx)f6@0w7juML;2HMF7M~t-$R-6#mwb9A+;g_Vx(3i6rfflK#bIi z*Gc=fONPy?0I3xT5F@oBp#Zfa0b-?A=rZq|Z7GZLVQNJH#7eCQC`7FYfS9Qj+Vt6x z{F3@MX4~uEnwp}11f%wx+3Vu~F;goX3Q{W^AXaL{PY)jU)qCPJrd9+%tkjBtLez=? ih>=?H(LIzo$68eG52+Oi5F@oBp#Zfa0sbGg@_zsc&LJHD literal 0 HcmV?d00001 diff --git a/artifacts/move-calibration/muninn-same-frame-stereo-assessment.mpack b/artifacts/move-calibration/muninn-same-frame-stereo-assessment.mpack new file mode 100644 index 0000000000000000000000000000000000000000..4de6fbf2cfe5f26019740fc841b1ad349d670b74 GIT binary patch literal 551 zcma)(KT88K7{*KK=par`VsR2lx%28(J7~Ux?rnOPUW0duBx%bzwSELaPzNbEIf#=Y zF8*0#(e=Ph5FDL#@V#2A)WtVLe!O{}C*jRgW-?ibSj1!@5-nO$no24v9tl}Z63qm! z5T&S65#_qjvXj+RhQ4Iz=csI#mYo&Lb&IxBELnpZ^_E=69rmt&-PK}>)mX@MTxm1z zi#FsxOy8cLOuctA_dDszn#{09uzPsMS6hhXJBtB~@6j2zxO6G}1AslPu zSQb(-_Zm`gwhdxWNJ-U};6CLZRY`Gd4rLF%Mv{P{QHF~|9odU0kPQ4Z4F9<7f4M#Z DrzFGQ literal 0 HcmV?d00001 diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 4065333..c22b776 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -215,6 +215,13 @@ both eyes. The same parent PID survived the run; transport health reported 5,302 produced frames, 5,301 handoffs, 771 sends, and zero local-ring admissions. Full stereo calibration is still not implied: intrinsics and spatially diverse correspondences must earn the camera matrices. +`MimirMoveStereoCalibrationAssessment` is the typed observability gate before +any fit. It accepts only same aggregate-frame, same-Move correspondences and +reports synchronized pair count, Move diversity, per-Eye 4x3 grid coverage, +radius range, and missing physical inputs. The corrected live capture produced +130 same-frame pairs with zero frame skew, but only 4/12 Eye 0 cells and 2/12 +Eye 1 cells; intrinsics and measured orb radius are also absent, so promotion is +correctly false. `MimirMoveProofSurface` is the first Fensalir-visible proof surface for this chain. It consumes the Mimir admission receipt and the Mimir-owned pose stream frame, emits `mimir.move_proof_surface.v1`, and lowers an observer-only diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index a09172a..a48c4c4 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -154,6 +154,17 @@ if (args.Any(arg => string.Equals(arg, "--muninn-move-visibility-window-smoke", ParseStringOption(args, "--output", "artifacts/move-calibration/muninn-visibility-window.mpack")); } +if (args.Any(arg => string.Equals(arg, "--move-stereo-calibration-assessment", StringComparison.OrdinalIgnoreCase))) +{ + return RunMoveStereoCalibrationAssessment( + ParseStringOption(args, "--input", "artifacts/move-calibration/muninn-clocked-aggregator.mpack"), + ParseStringOption(args, "--output", "artifacts/move-calibration/stereo-assessment.mpack"), + ParseIntOption(args, "--image-width", 640), + ParseIntOption(args, "--image-height", 480), + args.Any(arg => string.Equals(arg, "--intrinsics-available", StringComparison.OrdinalIgnoreCase)), + ParseDoubleOption(args, "--orb-radius-meters", 0.0)); +} + if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveFusionSmoke(); @@ -2071,16 +2082,20 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati } var correspondences = observations - .GroupBy(observation => observation.MoveId, StringComparer.Ordinal) - .SelectMany(group => + .GroupBy(observation => (observation.FrameId, observation.MoveId)) + .Select(group => { - var first = group.Where(value => value.CameraId == cameraIds[0]).ToArray(); - var second = group.Where(value => value.CameraId == cameraIds[1]).ToArray(); - return first.Select(left => second - .Select(right => new MoveCrossCameraCorrespondence(left.MoveId, left, right, Math.Abs(left.PublishedAtNs - right.PublishedAtNs))) - .OrderBy(pair => pair.AbsoluteSkewNs).FirstOrDefault()) - .Where(pair => pair is not null)!; + var first = group.FirstOrDefault(value => value.CameraId == cameraIds[0]); + var second = group.FirstOrDefault(value => value.CameraId == cameraIds[1]); + return first is null || second is null + ? null + : new MoveCrossCameraCorrespondence( + group.Key.MoveId, + first, + second, + Math.Abs(first.PublishedAtNs - second.PublishedAtNs)); }) + .Where(pair => pair is not null) .Cast() .ToArray(); var receipt = new MoveVisibilityWindowReceipt( @@ -2115,6 +2130,65 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati } } +static int RunMoveStereoCalibrationAssessment( + string inputPath, + string outputPath, + int imageWidth, + int imageHeight, + bool intrinsicsAvailable, + double orbRadiusMeters) +{ + if (!File.Exists(inputPath)) + { + Console.Error.WriteLine($"move-stereo-calibration-assessment missing-input path={inputPath}"); + return 1; + } + + var receipt = MessagePackSerializer.Deserialize(File.ReadAllBytes(inputPath)); + var cameraIds = receipt.Observations + .Select(value => value.CameraId) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal) + .ToArray(); + var assessment = MimirMoveStereoCalibrationAssessment.Assess( + receipt, + imageWidth, + imageHeight, + cameraIds, + intrinsicsAvailable, + orbRadiusMeters); + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath))!); + File.WriteAllBytes(outputPath, MessagePackSerializer.Serialize(assessment)); + Console.WriteLine(JsonSerializer.Serialize(new + { + assessment.Schema, + inputPath, + outputPath, + assessment.CorrespondenceCount, + assessment.SynchronizedCorrespondenceCount, + assessment.SameFrameCorrespondenceCount, + assessment.DistinctMoveCount, + assessment.MedianAbsoluteSkewMilliseconds, + assessment.MaximumAbsoluteSkewMilliseconds, + assessment.IntrinsicsAvailable, + assessment.OrbRadiusMetersAvailable, + assessment.ReadyForRelativePoseFit, + assessment.Promoted, + assessment.MissingRequirements, + cameras = assessment.Cameras.Select(camera => new + { + camera.CameraId, + camera.ObservationCount, + camera.OccupiedGridCells, + camera.GridCellCount, + spanX = camera.MaximumXPx - camera.MinimumXPx, + spanY = camera.MaximumYPx - camera.MinimumYPx, + radiusRatio = camera.MinimumRadiusPx > 0.0 ? camera.MaximumRadiusPx / camera.MinimumRadiusPx : double.NaN + }) + })); + return assessment.Promoted ? 0 : 2; +} + static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) { if (!File.Exists(nativeReservoirPath)) diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveStereoCalibrationAssessment.cs b/src/Mimir.Runtime/Synchronization/MimirMoveStereoCalibrationAssessment.cs new file mode 100644 index 0000000..e43bd03 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveStereoCalibrationAssessment.cs @@ -0,0 +1,139 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[MessagePackObject] +public sealed record MimirMoveCameraCoverageAssessment( + [property: Key(0)] string CameraId, + [property: Key(1)] int ObservationCount, + [property: Key(2)] double MinimumXPx, + [property: Key(3)] double MaximumXPx, + [property: Key(4)] double MinimumYPx, + [property: Key(5)] double MaximumYPx, + [property: Key(6)] int OccupiedGridCells, + [property: Key(7)] int GridCellCount, + [property: Key(8)] double MinimumRadiusPx, + [property: Key(9)] double MaximumRadiusPx); + +[CultDocument("mimir.move_stereo_calibration_assessment", "mimir.move_stereo_calibration_assessment.v1")] +[MessagePackObject] +public sealed record MimirMoveStereoCalibrationAssessmentDocument( + [property: Key(0)] string Schema, + [property: Key(1)] [property: CultName] string AssessmentId, + [property: Key(2)] string SourceSchema, + [property: Key(3)] long AssessedAtNs, + [property: Key(4)] int CorrespondenceCount, + [property: Key(5)] int SynchronizedCorrespondenceCount, + [property: Key(6)] int SameFrameCorrespondenceCount, + [property: Key(7)] int DistinctMoveCount, + [property: Key(8)] double MedianAbsoluteSkewMilliseconds, + [property: Key(9)] double MaximumAbsoluteSkewMilliseconds, + [property: Key(10)] MimirMoveCameraCoverageAssessment[] Cameras, + [property: Key(11)] bool IntrinsicsAvailable, + [property: Key(12)] bool OrbRadiusMetersAvailable, + [property: Key(13)] bool ReadyForRelativePoseFit, + [property: Key(14)] bool Promoted, + [property: Key(15)] string[] MissingRequirements); + +public static class MimirMoveStereoCalibrationAssessment +{ + public const string Schema = "mimir.move_stereo_calibration_assessment.v1"; + + public static MimirMoveStereoCalibrationAssessmentDocument Assess( + MoveVisibilityWindowReceipt receipt, + int imageWidth, + int imageHeight, + IReadOnlyCollection cameraIds, + bool intrinsicsAvailable, + double orbRadiusMeters, + double maximumAssociationSkewMilliseconds = 20.0, + DateTimeOffset? assessedAt = null) + { + ArgumentNullException.ThrowIfNull(receipt); + ArgumentNullException.ThrowIfNull(cameraIds); + if (imageWidth <= 0 || imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth)); + if (cameraIds.Count != 2) throw new ArgumentException("Stereo calibration requires exactly two camera IDs.", nameof(cameraIds)); + + var cameras = cameraIds.Distinct(StringComparer.Ordinal).ToArray(); + if (cameras.Length != 2) throw new ArgumentException("Stereo camera IDs must be distinct.", nameof(cameraIds)); + var maximumSkewNs = maximumAssociationSkewMilliseconds * 1_000_000.0; + var correspondences = receipt.Correspondences + .Where(value => cameras.Contains(value.First.CameraId, StringComparer.Ordinal) && + cameras.Contains(value.Second.CameraId, StringComparer.Ordinal)) + .ToArray(); + var synchronized = correspondences.Where(value => value.AbsoluteSkewNs <= maximumSkewNs).ToArray(); + var sameFrame = synchronized.Where(value => string.Equals(value.First.FrameId, value.Second.FrameId, StringComparison.Ordinal)).ToArray(); + var coverage = cameras.Select(cameraId => BuildCoverage( + cameraId, + receipt.Observations.Where(value => string.Equals(value.CameraId, cameraId, StringComparison.Ordinal)).ToArray(), + imageWidth, + imageHeight)).ToArray(); + var missing = new List(); + if (!intrinsicsAvailable) missing.Add("calibrated-intrinsics-for-both-eyes"); + if (!(double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0)) missing.Add("measured-orb-radius-meters"); + if (synchronized.Length < 300) missing.Add("at-least-300-synchronized-correspondences"); + if (sameFrame.Length < 100) missing.Add("at-least-100-same-frame-correspondences"); + if (synchronized.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count() < 2) missing.Add("at-least-two-stable-move-identities"); + foreach (var camera in coverage) + { + if (camera.OccupiedGridCells < 6) missing.Add($"{camera.CameraId}:coverage-at-least-6-of-12-grid-cells"); + if (camera.MaximumRadiusPx <= camera.MinimumRadiusPx * 1.25) missing.Add($"{camera.CameraId}:radius-range-at-least-1.25x"); + } + + var skews = correspondences.Select(value => value.AbsoluteSkewNs / 1_000_000.0).Order().ToArray(); + var ready = missing.Count == 0; + return new MimirMoveStereoCalibrationAssessmentDocument( + Schema, + $"move-stereo-assessment:{receipt.StartedAtNs}:{receipt.EndedAtNs}", + receipt.Schema, + (assessedAt ?? DateTimeOffset.UtcNow).ToUnixTimeMilliseconds() * 1_000_000, + correspondences.Length, + synchronized.Length, + sameFrame.Length, + synchronized.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count(), + Median(skews), + skews.Length == 0 ? double.NaN : skews[^1], + coverage, + intrinsicsAvailable, + double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0, + ready, + Promoted: false, + missing.Distinct(StringComparer.Ordinal).ToArray()); + } + + private static MimirMoveCameraCoverageAssessment BuildCoverage( + string cameraId, + MoveVisibilityObservation[] observations, + int imageWidth, + int imageHeight) + { + var valid = observations.Where(value => + float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) && + float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f).ToArray(); + var cells = valid.Select(value => + { + var column = Math.Clamp((int)(value.CenterXPx * 4.0 / imageWidth), 0, 3); + var row = Math.Clamp((int)(value.CenterYPx * 3.0 / imageHeight), 0, 2); + return row * 4 + column; + }).Distinct().Count(); + return new MimirMoveCameraCoverageAssessment( + cameraId, + valid.Length, + valid.Length == 0 ? double.NaN : valid.Min(value => value.CenterXPx), + valid.Length == 0 ? double.NaN : valid.Max(value => value.CenterXPx), + valid.Length == 0 ? double.NaN : valid.Min(value => value.CenterYPx), + valid.Length == 0 ? double.NaN : valid.Max(value => value.CenterYPx), + cells, + 12, + valid.Length == 0 ? double.NaN : valid.Min(value => value.RadiusPx), + valid.Length == 0 ? double.NaN : valid.Max(value => value.RadiusPx)); + } + + private static double Median(double[] sorted) => sorted.Length switch + { + 0 => double.NaN, + _ when sorted.Length % 2 == 1 => sorted[sorted.Length / 2], + _ => (sorted[sorted.Length / 2 - 1] + sorted[sorted.Length / 2]) * 0.5 + }; +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index cecedf0..53aca53 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -184,3 +184,4 @@ {"ts": "2026-07-13T01:29:15+01:00", "type": "verification", "status": "partial", "note": "Per-camera optical evidence and exposure ownership restored Eye 0. At exposure 0.3 Eye 0 peak RGB rose from [72,25,43] to [178,119,112], calibrated 4 Moves, and produced 3,312 observations/latest 4; Eye 1 remains at exposure 0.1. Mimir's persistent window then saw all 4 IDs on Eye 0, 2 on Eye 1, and 2 same-ID cross-camera correspondences. Do not claim full stereo: Eye 1 overlap is incomplete and only 2 aggregate RUDP frames arrived in 20s and 60s windows."} {"ts": "2026-07-13T01:41:29+01:00", "type": "verification", "status": "partial", "note": "After per-eye exposure tuning and latest-state RUDP handoff, Mimir's 15s dual-eye receipt saw all four stable Move IDs on both cameras: 16 camera observations and 8 pair records across 2 aggregate frames. Full overlap is proven; calibration is not. Only 2 unique frame IDs arrived, so directly instrument parent publication and RUDP send/receive counters before camera-matrix fitting."} {"ts": "2026-07-13T02:32:31+01:00", "type": "verification", "status": "ok", "note": "2026-07-13 Nightwing single-provider dual-Eye field proof: one stable Muninn parent received 622 unique evidence frames in 15 s through Mimir, with 2,647 observations, 1,384 same-ID cross-camera correspondences, and all four Move IDs visible on both Eyes. Producer transport health reported 5,302 produced, 5,301 handoffs, 771 sends, and zero local-ring admissions. Artifact: artifacts/move-calibration/muninn-clocked-aggregator.mpack. Odin owner commits: 39c839a, a340c33, 8aefd1c, f82ca7a."} +{"ts": "2026-07-13T02:39:32+01:00", "type": "verification", "status": "rejected", "note": "2026-07-13 typed Move stereo assessment rejected calibration promotion. Corrected same-frame collector captured 798 frames and 130 same-frame/same-Move pairs at zero published-frame skew. Eye 0 covered 4/12 image grid cells; Eye 1 covered 2/12 and saw only two Move IDs in this window. Both calibrated intrinsics and measured orb radius were absent. No camera matrices were emitted. Artifacts: artifacts/move-calibration/muninn-same-frame-correspondence.mpack and muninn-same-frame-stereo-assessment.mpack."} diff --git a/state/map.yaml b/state/map.yaml index 9869434..1ef68ed 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -267,6 +267,14 @@ accepted_design: remote handoffs, 771 sends during the subscription, and zero local-ring admissions. Cadence and full overlap are earned; calibrated camera matrices still require intrinsics and spatially diverse correspondence capture. + - `MimirMoveStereoCalibrationAssessment` owns the pre-fit observability gate + as `mimir.move_stereo_calibration_assessment.v1`. It consumes a typed + visibility receipt, counts only same-frame/same-Move stereo pairs, measures + per-Eye image-grid and radius coverage, and refuses promotion when + intrinsics, measured orb radius, synchronized sample count, or spatial + diversity are missing. The corrected 2026-07-13 live receipt contained 130 + honest same-frame pairs at zero frame skew, but Eye 0 occupied only 4/12 + grid cells and Eye 1 only 2/12; no camera matrices were promoted. - `MimirMuninnMoveEvidenceAdapter.TryAdmitLatestCultMeshFrame` now exposes a typed admission receipt with decoded frame id, producer, publish/read timestamps, sample counts by kind, source/arrival ranges, native reservoir From 11e3086ee4b38d4822e43f813d86a2c797cc4f93 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 13 Jul 2026 02:39:46 +0100 Subject: [PATCH 100/135] Preserve Mimir Face state --- .voidbot/state/mimir.cc | Bin 110378 -> 110394 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 7a252bd47e1cbda470d49a4ec93bdb68e0dafd94..2ca9b6d12330f40e047168c39c2aed42616a647f 100644 GIT binary patch delta 757 zcmYk4J!lj`6vtVRGbwV7@qbm07=daIHhYEshhqK0h}JXOKYH!y%^|7!Ir;MBnKvQ}!_H&gq5j)m z=gL~VC?v$fg3@hJBE=9WM!-8FAP97lNGr91>oS!F+Cq^nK|7W~3^o>~iOS~nZ9$A0 zErL*EHjJQy1Xhr=Q3oW=J`WNh)2;PeJUx=e60|HvldS}769bu|0%^gLOcM9*>(ss& z4Z6?YPPmUd2c`|CLMjkMB8V}ZeYx5~F4byl(XH-Gu5~0PVWvb7M6D!&AkONgvVx21S|gpybvLR%8Y%0zXq zHoY0|uAAANyv#K+$wO{vy#oJR&0UtVHu*7jkP%{AY7o?zi7#zHjFJR_V=F>D`~1soL^_S8dc6 z*B-MY_IUZ>81QNZu--2XKbPwl$7o~Tp~KnB&u;uf8U^z8>2o$wSS?+!tbEL__fN6X z@I$|Mi9N9ow$DtM5EX*3AGgm>E`vs?6J_^)RAN7o8ao+EAFD6?h>Ljh>S~JomXK{| zVk;q;3Aq8hEds{^yd$*H&`q)^AO`R>MI#JC%rzM)9ylr2pha8qsQQ{fZA2nBZ`#i2 z=oO@N(sQ6_Um*!w0;6EAiIKn-_#qb(y26BjGRfK@=&Y%Y;zTqF81hG}6yhW^6d|z-T>2rE{k>DW`)Jkh9&|E+1`6VvE!Jh}f@9YUfS4Cs@` zG?E>BJDU#-Tgk5jJDu+tR%TxQrOyuAYu`)x!3{Qfz^gaxysp^6&q_YE!KMpfG;?DA d0W8&uCt;^|?~U(&!QKDx?e6_gW5%Ml_7|A2 Date: Tue, 14 Jul 2026 00:35:47 +0100 Subject: [PATCH 101/135] Add Mimir-owned Move calibration sessions --- src/Mimir.BufferSmoke/Program.cs | 134 ++++++++++- .../MimirMoveCalibrationProtocol.cs | 56 +++-- .../Synchronization/MimirMoveFusion.cs | 31 +-- .../MimirSensorCalibrationSession.cs | 208 ++++++++++++++++++ 4 files changed, 391 insertions(+), 38 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index a48c4c4..3884574 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -165,6 +165,16 @@ if (args.Any(arg => string.Equals(arg, "--move-stereo-calibration-assessment", S ParseDoubleOption(args, "--orb-radius-meters", 0.0)); } +if (args.Any(arg => string.Equals(arg, "--sensor-calibration-session-live", StringComparison.OrdinalIgnoreCase))) +{ + return RunSensorCalibrationSessionLive( + ParseStringOption(args, "--odin-cultmesh-uri", "cultmesh://127.0.0.1:17871/rendezvous/provider-catalog"), + ParseIntOption(args, "--duration-seconds", 120), + ParseStringOption(args, "--output-prefix", "artifacts/move-calibration/live-session"), + ParseRepeatedStringOption(args, "--wand-id", [ + "move-0006f523e2d1", "move-000704a39772", "move-000704a6be5f", "move-000704a800d0"])); +} + if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveFusionSmoke(); @@ -2130,6 +2140,126 @@ static int RunMuninnMoveVisibilityWindowSmoke(string odinCultMeshUri, int durati } } +static int RunSensorCalibrationSessionLive( + string odinCultMeshUri, + int durationSeconds, + string outputPrefix, + IReadOnlyCollection wandIds) +{ + const string providerId = "muninn.telemetry.nightwing"; + const string streamId = "muninn:nightwing:move-evidence"; + var cameraIds = new[] { "nightwing-eye-0", "nightwing-eye-1" }; + var configuration = new MimirMoveProofRuntimeConfiguration + { + OdinCultMeshUri = odinCultMeshUri, + MuninnProviderId = providerId, + EvidenceStreamId = streamId + }; + if (!MimirOdinMoveProofEvidenceRingProvider.Instance.TryOpenEvidenceRing(configuration, out var lease, out var diagnostic) || lease is null) + throw new InvalidOperationException($"{providerId}: {diagnostic}"); + + using (lease) + { + var startedAt = DateTimeOffset.UtcNow; + var session = MimirSensorCalibrationSessions.CreateFourWandOpticalSession( + wandIds, cameraIds, startedAt, Math.Max(1, durationSeconds)); + var observations = new List(); + var seenFrames = new HashSet(StringComparer.Ordinal); + var deadline = startedAt + TimeSpan.FromSeconds(Math.Max(1, durationSeconds)); + var lastCheckpoint = DateTimeOffset.MinValue; + var evidencePath = $"{outputPrefix}.evidence.mpack"; + var sessionPath = $"{outputPrefix}.session.mpack"; + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(sessionPath))!); + + while (DateTimeOffset.UtcNow < deadline && session.Phase == MimirSensorCalibrationSessionPhase.Collecting) + { + if (lease.Ring.TryAcquireLatestRead(out var frameLease)) + { + using (frameLease) + { + var frame = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame( + frameLease.Memory[..frameLease.Handle.ByteLength]); + if (seenFrames.Add(frame.FrameId)) + { + observations.AddRange(frame.MarkerCandidates + .Where(marker => !string.IsNullOrWhiteSpace(marker.MoveId)) + .Select(marker => new MoveVisibilityObservation( + providerId, marker.CameraId, marker.MoveId, frame.FrameId, + frame.PublishedAtNs, marker.CenterXPx, marker.CenterYPx, + marker.RadiusPx, marker.Score))); + } + } + } + + var now = DateTimeOffset.UtcNow; + if ((now - lastCheckpoint).TotalSeconds >= 1.0) + { + var evidence = BuildVisibilityReceipt(startedAt, now, providerId, cameraIds, observations); + session = MimirSensorCalibrationSessions.UpdateCollection(session, evidence, 640, 480, now); + File.WriteAllBytes(evidencePath, MessagePackSerializer.Serialize(evidence)); + File.WriteAllBytes(sessionPath, MessagePackSerializer.Serialize(session)); + Console.WriteLine(JsonSerializer.Serialize(new + { + session.SessionId, + session.Phase, + elapsedSeconds = Math.Round((now - startedAt).TotalSeconds, 1), + frames = seenFrames.Count, + observations = observations.Count, + correspondences = evidence.Correspondences.Length, + sensors = session.Sensors.Select(sensor => new + { + sensor.SensorId, + sensor.ObservationCount, + sensor.OccupiedGridCells, + sensor.DistinctWandCount, + radiusRatio = sensor.ObservationCount == 0 ? 0.0 : Math.Round(sensor.MaximumRadiusPx / sensor.MinimumRadiusPx, 2), + sensor.MissingGridCells, + sensor.CollectionComplete + }), + session.MissingRequirements + })); + lastCheckpoint = now; + } + Thread.Sleep(5); + } + + var completedAt = DateTimeOffset.UtcNow; + var finalEvidence = BuildVisibilityReceipt(startedAt, completedAt, providerId, cameraIds, observations); + session = MimirSensorCalibrationSessions.UpdateCollection(session, finalEvidence, 640, 480, completedAt); + File.WriteAllBytes(evidencePath, MessagePackSerializer.Serialize(finalEvidence)); + File.WriteAllBytes(sessionPath, MessagePackSerializer.Serialize(session)); + Console.WriteLine(JsonSerializer.Serialize(new { final = true, session.SessionId, session.Phase, evidencePath, sessionPath, session.MissingRequirements })); + return session.Phase == MimirSensorCalibrationSessionPhase.Fitting ? 0 : 2; + } +} + +static MoveVisibilityWindowReceipt BuildVisibilityReceipt( + DateTimeOffset startedAt, + DateTimeOffset endedAt, + string providerId, + IReadOnlyList cameraIds, + IReadOnlyCollection observations) +{ + var observationArray = observations.ToArray(); + var correspondences = observationArray + .GroupBy(observation => (observation.FrameId, observation.MoveId)) + .Select(group => + { + var first = group.FirstOrDefault(value => value.CameraId == cameraIds[0]); + var second = group.FirstOrDefault(value => value.CameraId == cameraIds[1]); + return first is null || second is null ? null : new MoveCrossCameraCorrespondence( + group.Key.MoveId, first, second, Math.Abs(first.PublishedAtNs - second.PublishedAtNs)); + }) + .Where(value => value is not null) + .Cast() + .ToArray(); + return new MoveVisibilityWindowReceipt( + "mimir.move_visibility_window.v1", + startedAt.ToUnixTimeMilliseconds() * 1_000_000, + endedAt.ToUnixTimeMilliseconds() * 1_000_000, + [providerId], observationArray, correspondences); +} + static int RunMoveStereoCalibrationAssessment( string inputPath, string outputPath, @@ -3361,7 +3491,7 @@ static int RunMoveProofPipelineSmoke(string nativeReservoirPath) static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) { - var protocol = MimirMoveCalibrationProtocol.CreateStarfireNightwingProtocol( + var protocol = MimirMoveCalibrationProtocol.CreateNightwingFourWandProtocol( new DateTimeOffset(2026, 6, 12, 14, 45, 0, TimeSpan.Zero)); var errors = MimirMoveCalibrationProtocol.Validate(protocol); if (errors.Length > 0) @@ -3388,7 +3518,7 @@ static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) $"move-calibration-protocol-smoke path={outputPath} protocol={protocol.ProtocolId} requiredStreams={requiredStreams} optionalStreams={optionalStreams} phases={protocol.Phases.Length} durationSeconds={totalDuration:0.0} outputs={protocol.Outputs.Length} questOptional={protocol.StreamRequirements.Any(stream => stream.StreamId.StartsWith("quest:", StringComparison.Ordinal))}"); return requiredStreams >= 3 && - protocol.Phases.Length == 7 && + protocol.Phases.Length == 8 && protocol.Outputs.Length == 4 && protocol.Acceptance.RequiredDerivedOutputs.Length == 4 ? 0 diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs b/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs index 763b92d..3c51bc4 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveCalibrationProtocol.cs @@ -56,23 +56,15 @@ public sealed record MimirMoveCalibrationAcceptance( public static class MimirMoveCalibrationProtocol { - public static MimirMoveCalibrationProtocolDocument CreateStarfireNightwingProtocol( + public static MimirMoveCalibrationProtocolDocument CreateNightwingFourWandProtocol( DateTimeOffset? createdAt = null) => new( - ProtocolId: "mimir-move-calibration-starfire-nightwing-v1", + ProtocolId: "mimir-move-calibration-nightwing-four-wand-v1", CreatedAtUtc: (createdAt ?? DateTimeOffset.UtcNow).ToString("O"), TrackingSpaceId: "mimir-stage-space", FusionAuthorityId: "mimir.runtime.move-fusion", StreamRequirements: [ - new( - "muninn:starfire:move-evidence", - "starfire", - "Muninn on Starfire", - "Move optical marker candidates plus USB controller IMU/buttons", - "CultMesh shared-memory bytes frame, same-host when Mimir runs on Starfire", - Required: true, - "Starfire owns the USB-attached Move and local camera witnesses."), new( "muninn:nightwing:move-evidence", "nightwing", @@ -81,6 +73,14 @@ public static class MimirMoveCalibrationProtocol "CultMesh bytes stream over the Verse; shared-memory on Nightwing, network bridge to Starfire", Required: true, "Nightwing owns its USB-attached Move and local camera witnesses."), + new( + "mimir:starfire:sensor-calibration-session", + "starfire", + "Mimir.Runtime", + "Bounded calibration lifecycle, coverage, fit, validation, and promotion state", + "Typed CultMesh state with CultCache persistence", + Required: true, + "Mimir owns the calibration task and its completion verdict."), new( "mimir:starfire:move-controller-poses", "starfire", @@ -128,36 +128,36 @@ public static class MimirMoveCalibrationProtocol "preflight-streams", "Stream liveness and clock edge check", 10.0, - "Hold both Moves still and visible. Confirm Starfire, Nightwing, and optional Quest pose streams are fresh.", - ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + "Hold the four Moves visible. Confirm Nightwing evidence and stable identities are fresh.", + ["muninn:nightwing:move-evidence"], ["stream-freshness", "clock-skew-estimate", "controller-id-map"]), new( "dark-stillness", "IMU bias with orbs off", 8.0, "Set Move lights off. Place both Moves still on the calibration surface.", - ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["muninn:nightwing:move-evidence"], ["gyro-bias", "accelerometer-gravity-vector", "magnetometer-baseline"]), new( "lit-stillness", "Optical centroid and static gravity alignment", 8.0, "Light one Move at a time, then both. Keep them still and visible to both camera sets.", - ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["muninn:nightwing:move-evidence"], ["camera-marker-centroid-stability", "light-command-to-controller-association"]), new( "axis-sweeps", "Slow single-axis rotations", 24.0, "Sweep each Move slowly through pitch, yaw, and roll while keeping the orb visible.", - ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["muninn:nightwing:move-evidence"], ["gyro-scale-fit", "accelerometer-frame-fit", "optical-angular-consistency"]), new( "figure-eight", "Magnetometer and cross-axis motion", 20.0, "Move each controller through a broad figure-eight. Leave the Quest headset/controllers stationary as the external VR reference cluster.", - ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence"], + ["muninn:nightwing:move-evidence"], ["magnetometer-hard-soft-iron-fit", "cross-axis-coupling-check"]), new( "quest-reference", @@ -166,12 +166,19 @@ public static class MimirMoveCalibrationProtocol "Keep the Move between the two Quest controllers in front of the headset. Capture the static Quest controller/headset frame while Mimir observes the Move.", ["quest:usb:headset-pose", "quest:usb:left-controller-pose", "quest:usb:right-controller-pose"], ["external-vr-pose-frame", "quest-to-mimir-frame-fit", "quest-clock-offset-estimate"]), + new( + "wand-volume-sweep", + "Four-wand optical volume coverage", + 120.0, + "Wave all four uniquely identified wands throughout the shared camera volume until Mimir reports sufficient per-sensor coverage.", + ["muninn:nightwing:move-evidence", "mimir:starfire:sensor-calibration-session"], + ["sensor-grid-coverage", "same-frame-correspondences", "sphere-radius-range", "fit-and-held-out-partitions"]), new( "validation-pass", "Held-out tracking validation", 15.0, "Run free hand motion with both orbs visible. Mimir may publish validation poses, but not promote calibration if confidence fails.", - ["muninn:starfire:move-evidence", "muninn:nightwing:move-evidence", "mimir:starfire:move-controller-poses"], + ["muninn:nightwing:move-evidence", "mimir:starfire:move-controller-poses"], ["triangulated-position-residuals", "imu-prediction-residuals", "promotion-decision"]) ], Outputs: @@ -227,16 +234,21 @@ public static class MimirMoveCalibrationProtocol errors.Add("required-streams-too-small"); } - if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "muninn:starfire:move-evidence" && stream.Required)) - { - errors.Add("missing-starfire-muninn-move-evidence"); - } - if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "muninn:nightwing:move-evidence" && stream.Required)) { errors.Add("missing-nightwing-muninn-move-evidence"); } + if (!protocol.StreamRequirements.Any(stream => stream.StreamId == "mimir:starfire:sensor-calibration-session" && stream.Required)) + { + errors.Add("missing-mimir-calibration-session"); + } + + if (!protocol.Phases.Any(phase => phase.PhaseId == "wand-volume-sweep")) + { + errors.Add("missing-wand-volume-sweep"); + } + if (!protocol.Phases.Any(phase => phase.PhaseId == "dark-stillness")) { errors.Add("missing-imu-bias-phase"); diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs b/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs index 3e04ba1..4ccf768 100644 --- a/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs +++ b/src/Mimir.Runtime/Synchronization/MimirMoveFusion.cs @@ -1,24 +1,27 @@ using System.Numerics; +using MessagePack; namespace Mimir.Runtime.Synchronization; +[MessagePackObject] public sealed record MimirMoveFusionCameraCalibration( - string CameraId, - ulong WitnessIdHash, - MimirVector3Snapshot PositionMeters, - MimirQuaternionSnapshot Orientation, - double FocalLengthXPx, - double FocalLengthYPx, - double PrincipalPointXPx, - double PrincipalPointYPx); + [property: Key(0)] string CameraId, + [property: Key(1)] ulong WitnessIdHash, + [property: Key(2)] MimirVector3Snapshot PositionMeters, + [property: Key(3)] MimirQuaternionSnapshot Orientation, + [property: Key(4)] double FocalLengthXPx, + [property: Key(5)] double FocalLengthYPx, + [property: Key(6)] double PrincipalPointXPx, + [property: Key(7)] double PrincipalPointYPx); +[MessagePackObject] public sealed record MimirMoveFusionRigCalibration( - string CalibrationId, - string TrackingSpaceId, - IReadOnlyList Cameras, - double GyroUnitsPerRadianPerSecond = 1.0, - double MaximumAssociationSkewMilliseconds = 20.0, - double SingleRayFallbackDepthMeters = 1.5); + [property: Key(0)] string CalibrationId, + [property: Key(1)] string TrackingSpaceId, + [property: Key(2)] IReadOnlyList Cameras, + [property: Key(3)] double GyroUnitsPerRadianPerSecond = 1.0, + [property: Key(4)] double MaximumAssociationSkewMilliseconds = 20.0, + [property: Key(5)] double SingleRayFallbackDepthMeters = 1.5); public sealed record MimirMoveFusionResult( IReadOnlyList Poses, diff --git a/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs b/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs new file mode 100644 index 0000000..4df697b --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs @@ -0,0 +1,208 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +public enum MimirSensorCalibrationSessionPhase +{ + Collecting, + Fitting, + Validating, + Promoted, + Rejected, + Canceled +} + +[CultDocument("mimir.sensor_calibration_session", "mimir.sensor_calibration_session.v1")] +[MessagePackObject] +public sealed record MimirSensorCalibrationSessionDocument( + [property: Key(0)] [property: CultName] string SessionId, + [property: Key(1)] string TrackingSpaceId, + [property: Key(2)] string Owner, + [property: Key(3)] string StartedAtUtc, + [property: Key(4)] double MaximumDurationSeconds, + [property: Key(5)] string[] RequiredWandIds, + [property: Key(6)] string[] SensorIds, + [property: Key(7)] MimirSensorCalibrationAcceptance Acceptance, + [property: Key(8)] MimirSensorCalibrationSessionPhase Phase, + [property: Key(9)] MimirSensorCalibrationSensorProgress[] Sensors, + [property: Key(10)] string[] MissingRequirements, + [property: Key(11)] string Detail); + +[MessagePackObject] +public sealed record MimirSensorCalibrationAcceptance( + [property: Key(0)] int GridColumns, + [property: Key(1)] int GridRows, + [property: Key(2)] int MinimumOccupiedGridCellsPerSensor, + [property: Key(3)] int MinimumObservationsPerSensor, + [property: Key(4)] int MinimumSameFrameCorrespondences, + [property: Key(5)] int MinimumDistinctWands, + [property: Key(6)] double MinimumRadiusRatio, + [property: Key(7)] double HeldOutFraction, + [property: Key(8)] double MaximumMedianReprojectionErrorPx, + [property: Key(9)] double MaximumP95ReprojectionErrorPx, + [property: Key(10)] double MaximumAssociationSkewMilliseconds); + +[MessagePackObject] +public sealed record MimirSensorCalibrationSensorProgress( + [property: Key(0)] string SensorId, + [property: Key(1)] int ObservationCount, + [property: Key(2)] int OccupiedGridCells, + [property: Key(3)] int GridCellCount, + [property: Key(4)] int DistinctWandCount, + [property: Key(5)] double MinimumRadiusPx, + [property: Key(6)] double MaximumRadiusPx, + [property: Key(7)] int[] MissingGridCells, + [property: Key(8)] bool CollectionComplete); + +[CultDocument("mimir.sensor_calibration_receipt", "mimir.sensor_calibration_receipt.v1")] +[MessagePackObject] +public sealed record MimirSensorCalibrationReceiptDocument( + [property: Key(0)] [property: CultName] string ReceiptId, + [property: Key(1)] string SessionId, + [property: Key(2)] string TrackingSpaceId, + [property: Key(3)] string CompletedAtUtc, + [property: Key(4)] MimirSensorCalibrationSessionPhase Verdict, + [property: Key(5)] int TrainingCorrespondenceCount, + [property: Key(6)] int HeldOutCorrespondenceCount, + [property: Key(7)] MimirSensorCalibrationFitResiduals Residuals, + [property: Key(8)] MimirMoveFusionRigCalibration? PromotedCalibration, + [property: Key(9)] string[] RejectionReasons, + [property: Key(10)] string Solver); + +[MessagePackObject] +public sealed record MimirSensorCalibrationFitResiduals( + [property: Key(0)] double TrainingMedianReprojectionErrorPx, + [property: Key(1)] double TrainingP95ReprojectionErrorPx, + [property: Key(2)] double HeldOutMedianReprojectionErrorPx, + [property: Key(3)] double HeldOutP95ReprojectionErrorPx, + [property: Key(4)] double MedianAssociationSkewMilliseconds, + [property: Key(5)] double MaximumAssociationSkewMilliseconds, + [property: Key(6)] double ConditionEstimate, + [property: Key(7)] int Iterations); + +public static class MimirSensorCalibrationSessions +{ + public static MimirSensorCalibrationSessionDocument CreateFourWandOpticalSession( + IEnumerable wandIds, + IEnumerable sensorIds, + DateTimeOffset? startedAt = null, + double maximumDurationSeconds = 120.0) + { + var wands = wandIds.Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.Ordinal).Order().ToArray(); + var sensors = sensorIds.Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.Ordinal).Order().ToArray(); + if (wands.Length != 4) throw new ArgumentException("A four-wand calibration session requires exactly four stable wand IDs.", nameof(wandIds)); + if (sensors.Length < 2) throw new ArgumentException("Optical calibration requires at least two sensors.", nameof(sensorIds)); + if (!double.IsFinite(maximumDurationSeconds) || maximumDurationSeconds <= 0.0) throw new ArgumentOutOfRangeException(nameof(maximumDurationSeconds)); + var start = startedAt ?? DateTimeOffset.UtcNow; + return new MimirSensorCalibrationSessionDocument( + $"mimir-sensor-calibration:{start.ToUnixTimeMilliseconds()}", + "mimir-stage-space", + "Mimir.Runtime", + start.ToString("O"), + maximumDurationSeconds, + wands, + sensors, + new MimirSensorCalibrationAcceptance( + GridColumns: 4, + GridRows: 3, + MinimumOccupiedGridCellsPerSensor: 9, + MinimumObservationsPerSensor: 1_000, + MinimumSameFrameCorrespondences: 500, + MinimumDistinctWands: 4, + MinimumRadiusRatio: 1.5, + HeldOutFraction: 0.2, + MaximumMedianReprojectionErrorPx: 1.5, + MaximumP95ReprojectionErrorPx: 4.0, + MaximumAssociationSkewMilliseconds: 20.0), + MimirSensorCalibrationSessionPhase.Collecting, + sensors.Select(sensor => new MimirSensorCalibrationSensorProgress( + sensor, 0, 0, 12, 0, double.NaN, double.NaN, Enumerable.Range(0, 12).ToArray(), false)).ToArray(), + ["wand-motion-not-yet-captured"], + "Wave one or two uniquely identified wands at a time through the shared sensor volume; Mimir accumulates successive passes until every wand and sensor has sufficient coverage."); + } + + public static MimirSensorCalibrationSessionDocument UpdateCollection( + MimirSensorCalibrationSessionDocument session, + MoveVisibilityWindowReceipt evidence, + int imageWidth, + int imageHeight, + DateTimeOffset? observedAt = null) + { + ArgumentNullException.ThrowIfNull(session); + ArgumentNullException.ThrowIfNull(evidence); + if (session.Phase != MimirSensorCalibrationSessionPhase.Collecting) return session; + if (imageWidth <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth)); + if (imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageHeight)); + + var acceptance = session.Acceptance; + var requiredWands = session.RequiredWandIds.ToHashSet(StringComparer.Ordinal); + var progress = session.SensorIds.Select(sensorId => + { + var observations = evidence.Observations.Where(value => + string.Equals(value.CameraId, sensorId, StringComparison.Ordinal) && + requiredWands.Contains(value.MoveId) && + float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) && + float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f).ToArray(); + var occupied = observations.Select(value => GridCell( + value.CenterXPx, value.CenterYPx, imageWidth, imageHeight, + acceptance.GridColumns, acceptance.GridRows)).Distinct().Order().ToArray(); + var missing = Enumerable.Range(0, acceptance.GridColumns * acceptance.GridRows) + .Except(occupied).ToArray(); + var minimumRadius = observations.Length == 0 ? double.NaN : observations.Min(value => value.RadiusPx); + var maximumRadius = observations.Length == 0 ? double.NaN : observations.Max(value => value.RadiusPx); + var radiusRatio = observations.Length == 0 ? 0.0 : maximumRadius / minimumRadius; + var distinctWands = observations.Select(value => value.MoveId).Distinct(StringComparer.Ordinal).Count(); + var complete = observations.Length >= acceptance.MinimumObservationsPerSensor && + occupied.Length >= acceptance.MinimumOccupiedGridCellsPerSensor && + distinctWands >= acceptance.MinimumDistinctWands && + radiusRatio >= acceptance.MinimumRadiusRatio; + return new MimirSensorCalibrationSensorProgress( + sensorId, observations.Length, occupied.Length, + acceptance.GridColumns * acceptance.GridRows, distinctWands, + minimumRadius, maximumRadius, missing, complete); + }).ToArray(); + + var correspondenceCount = evidence.Correspondences.Count(value => + requiredWands.Contains(value.MoveId) && + value.AbsoluteSkewNs <= acceptance.MaximumAssociationSkewMilliseconds * 1_000_000.0 && + string.Equals(value.First.FrameId, value.Second.FrameId, StringComparison.Ordinal)); + var missingRequirements = new List(); + foreach (var sensor in progress) + { + if (sensor.ObservationCount < acceptance.MinimumObservationsPerSensor) + missingRequirements.Add($"{sensor.SensorId}:observations-{sensor.ObservationCount}-of-{acceptance.MinimumObservationsPerSensor}"); + if (sensor.OccupiedGridCells < acceptance.MinimumOccupiedGridCellsPerSensor) + missingRequirements.Add($"{sensor.SensorId}:grid-cells-{sensor.OccupiedGridCells}-of-{acceptance.MinimumOccupiedGridCellsPerSensor}"); + if (sensor.DistinctWandCount < acceptance.MinimumDistinctWands) + missingRequirements.Add($"{sensor.SensorId}:wands-{sensor.DistinctWandCount}-of-{acceptance.MinimumDistinctWands}"); + var radiusRatio = sensor.ObservationCount == 0 ? 0.0 : sensor.MaximumRadiusPx / sensor.MinimumRadiusPx; + if (radiusRatio < acceptance.MinimumRadiusRatio) + missingRequirements.Add($"{sensor.SensorId}:radius-ratio-{radiusRatio:F2}-of-{acceptance.MinimumRadiusRatio:F2}"); + } + if (correspondenceCount < acceptance.MinimumSameFrameCorrespondences) + missingRequirements.Add($"same-frame-correspondences-{correspondenceCount}-of-{acceptance.MinimumSameFrameCorrespondences}"); + + var elapsed = (observedAt ?? DateTimeOffset.UtcNow) - DateTimeOffset.Parse(session.StartedAtUtc); + var collectionComplete = progress.All(value => value.CollectionComplete) && + correspondenceCount >= acceptance.MinimumSameFrameCorrespondences; + var expired = elapsed.TotalSeconds >= session.MaximumDurationSeconds; + var phase = collectionComplete + ? MimirSensorCalibrationSessionPhase.Fitting + : expired ? MimirSensorCalibrationSessionPhase.Rejected : MimirSensorCalibrationSessionPhase.Collecting; + var detail = phase switch + { + MimirSensorCalibrationSessionPhase.Fitting => "Collection complete; Mimir owns the retained evidence and may begin fitting.", + MimirSensorCalibrationSessionPhase.Rejected => "Collection window expired before coverage requirements were met; no calibration was promoted.", + _ => "Collection active. Continue sweeping one or two wands at a time through missing image regions and depth ranges." + }; + return session with { Phase = phase, Sensors = progress, MissingRequirements = missingRequirements.ToArray(), Detail = detail }; + } + + private static int GridCell(float x, float y, int width, int height, int columns, int rows) + { + var column = Math.Clamp((int)(x * columns / width), 0, columns - 1); + var row = Math.Clamp((int)(y * rows / height), 0, rows - 1); + return row * columns + column; + } +} From 58af018125ffc33ac50f2bc179087e72ae33cfdd Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 14 Jul 2026 01:00:17 +0100 Subject: [PATCH 102/135] Add robust Move sphere stereo fitting --- docs/move-calibration-protocol.md | 17 + notes/fresh-workspace-handoff.md | 13 + src/Mimir.BufferSmoke/Program.cs | 204 ++++++++- .../MimirMoveSphereCalibrationFitter.cs | 395 ++++++++++++++++++ .../MimirSensorCalibrationSession.cs | 8 +- state/evidence.jsonl | 1 + state/map.yaml | 8 + 7 files changed, 632 insertions(+), 14 deletions(-) create mode 100644 src/Mimir.Runtime/Synchronization/MimirMoveSphereCalibrationFitter.cs diff --git a/docs/move-calibration-protocol.md b/docs/move-calibration-protocol.md index b817d82..6f92b88 100644 --- a/docs/move-calibration-protocol.md +++ b/docs/move-calibration-protocol.md @@ -78,6 +78,23 @@ sitting around the Move. - `mimir.move_calibration_receipt.v1`: sample counts, residuals, rejected phases, and promotion decision. +## Optical Sphere Fit Authority + +`MimirMoveSphereCalibrationFitter` owns the bounded two-Eye optical fit after a +collection session reaches `Fitting`. Eye 0 defines the tracking-space gauge. +The fitter jointly estimates both focal lengths and Eye 1's rigid pose from +same-frame, same-Move correspondences, using a deterministic 80/20 training and +held-out partition. A robust training consensus may reject contaminated marker +pairs, but promotion also requires at least 65 percent of the untouched +held-out pairs to agree and must pass the session's median and P95 reprojection +limits. + +The physical orb radius is an explicit input; it is never guessed by the +solver. PSMoveAPI's reported blob radius is a color-segmentation result, not a +calibrated sphere silhouette. It may seed or weight depth only after the +per-camera radius response has been validated. A fresh PSMoveAPI position age +is also not optical confidence and must not be interpreted as one. + ## Preflight Command ```powershell diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index b79f522..191492c 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -198,6 +198,19 @@ Get-Content .\state\evidence.jsonl -Tail 8 ## Immediate Re-entry Instruction +The 2026-07-14 live two-wand session at +`artifacts/move-calibration/live-two-wand-20260714-004210.*` passed collection +coverage with 1,168 admitted same-frame pairs. The new Mimir-owned sphere +stereo fitter recovers synthetic 505/535 px focal lengths within 0.43 px and a +34 cm baseline within 0.22 mm, with 0.34/0.63 px held-out median/P95 error. +The live artifact remains rejected: after robust radius admission its consensus +fits at 0.51/2.08 px, but only 53.8 percent of held-out pairs agree and Eye 1's +focal estimate leaves the physical range. PSMoveAPI age is currently projected +as marker confidence by Muninn, so fresh wrong-color matches arrive near 0.996 +confidence. Repair source-local observation quality/continuity and calibrate +per-Eye blob-radius response before asking the operator for another sweep. Do +not promote the nominal 22.5 mm diagnostic fit. + The active priority is Nightwing Move calibration. One public Muninn observer owns two private Eye workers and one aggregate evidence stream. Odin commits `39c839a`, `a340c33`, `8aefd1c`, and `f82ca7a` give daemon health a persistent diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 3884574..fa9c63c 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -175,6 +175,22 @@ if (args.Any(arg => string.Equals(arg, "--sensor-calibration-session-live", Stri "move-0006f523e2d1", "move-000704a39772", "move-000704a6be5f", "move-000704a800d0"])); } +if (args.Any(arg => string.Equals(arg, "--sensor-calibration-fit", StringComparison.OrdinalIgnoreCase))) +{ + return RunSensorCalibrationFit( + ParseStringOption(args, "--session", "artifacts/move-calibration/live-session.session.mpack"), + ParseStringOption(args, "--evidence", "artifacts/move-calibration/live-session.evidence.mpack"), + ParseStringOption(args, "--output", "artifacts/move-calibration/live-session.calibration-receipt.mpack"), + ParseIntOption(args, "--image-width", 640), + ParseIntOption(args, "--image-height", 480), + ParseDoubleOption(args, "--orb-radius-meters", 0.0)); +} + +if (args.Any(arg => string.Equals(arg, "--sensor-calibration-fit-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunSensorCalibrationFitSmoke(); +} + if (args.Any(arg => string.Equals(arg, "--move-fusion-smoke", StringComparison.OrdinalIgnoreCase))) { return RunMoveFusionSmoke(); @@ -2319,6 +2335,172 @@ static int RunMoveStereoCalibrationAssessment( return assessment.Promoted ? 0 : 2; } +static int RunSensorCalibrationFit( + string sessionPath, + string evidencePath, + string outputPath, + int imageWidth, + int imageHeight, + double orbRadiusMeters) +{ + if (!File.Exists(sessionPath) || !File.Exists(evidencePath)) + { + Console.Error.WriteLine($"sensor-calibration-fit missing-input session={sessionPath} evidence={evidencePath}"); + return 1; + } + + var session = MessagePackSerializer.Deserialize(File.ReadAllBytes(sessionPath)); + var evidence = MessagePackSerializer.Deserialize(File.ReadAllBytes(evidencePath)); + var receipt = MimirMoveSphereCalibrationFitter.Fit(session, evidence, imageWidth, imageHeight, orbRadiusMeters); + var observationStats = evidence.Observations + .GroupBy(value => value.CameraId, StringComparer.Ordinal) + .Select(group => + { + var radii = group.Where(value => float.IsFinite(value.RadiusPx) && value.RadiusPx > 0).Select(value => (double)value.RadiusPx).Order().ToArray(); + var confidence = group.Where(value => float.IsFinite(value.Confidence)).Select(value => (double)value.Confidence).Order().ToArray(); + return new + { + CameraId = group.Key, + Count = group.Count(), + RadiusP01 = CalibrationCliPercentile(radii, 0.01), + RadiusP10 = CalibrationCliPercentile(radii, 0.10), + RadiusP50 = CalibrationCliPercentile(radii, 0.50), + RadiusP90 = CalibrationCliPercentile(radii, 0.90), + RadiusP99 = CalibrationCliPercentile(radii, 0.99), + ConfidenceP10 = CalibrationCliPercentile(confidence, 0.10), + ConfidenceP50 = CalibrationCliPercentile(confidence, 0.50) + }; + }).ToArray(); + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath))!); + File.WriteAllBytes(outputPath, MessagePackSerializer.Serialize(receipt)); + Console.WriteLine(JsonSerializer.Serialize(new + { + receipt.ReceiptId, + receipt.SessionId, + receipt.Verdict, + receipt.TrainingCorrespondenceCount, + receipt.HeldOutCorrespondenceCount, + receipt.Residuals, + receipt.RejectionReasons, + observationStats, + calibration = receipt.PromotedCalibration is null ? null : new + { + receipt.PromotedCalibration.CalibrationId, + cameras = receipt.PromotedCalibration.Cameras.Select(camera => new + { + camera.CameraId, + camera.PositionMeters, + camera.Orientation, + camera.FocalLengthXPx, + camera.FocalLengthYPx, + camera.PrincipalPointXPx, + camera.PrincipalPointYPx + }) + }, + outputPath + })); + return receipt.Verdict == MimirSensorCalibrationSessionPhase.Promoted ? 0 : 2; +} + +static double CalibrationCliPercentile(double[] sorted, double percentile) +{ + if (sorted.Length == 0) return double.NaN; + var position = Math.Clamp(percentile, 0, 1) * (sorted.Length - 1); + var lower = (int)Math.Floor(position); + var upper = (int)Math.Ceiling(position); + return sorted[lower] + (sorted[upper] - sorted[lower]) * (position - lower); +} + +static int RunSensorCalibrationFitSmoke() +{ + const int width = 640; + const int height = 480; + const double radius = 0.0225; + const double focal0 = 505.0; + const double focal1 = 535.0; + var camera1WorldToCamera = Quaternion.CreateFromYawPitchRoll(0.11f, -0.035f, 0.025f); + var camera1Position = new Vector3(0.34f, 0.025f, 0.07f); + var camera1Translation = Vector3.Transform(-camera1Position, camera1WorldToCamera); + var random = new Random(1789); + var observations = new List(); + var correspondences = new List(); + var wandIds = new[] { "move-a", "move-b", "move-c", "move-d" }; + for (var index = 0; index < 1400; index++) + { + var world = new Vector3( + (float)(random.NextDouble() * 1.8 - 0.9), + (float)(random.NextDouble() * 1.2 - 0.6), + (float)(0.65 + random.NextDouble() * 3.0)); + var camera1 = Vector3.Transform(world, camera1WorldToCamera) + camera1Translation; + if (camera1.Z <= 0.2f) { index--; continue; } + var frameId = $"synthetic:{index}"; + var wandId = wandIds[index % wandIds.Length]; + var first = SyntheticSphereObservation("eye-0", wandId, frameId, index, world, focal0, width, height, radius, random); + var second = SyntheticSphereObservation("eye-1", wandId, frameId, index, camera1, focal1, width, height, radius, random); + if (first.CenterXPx is < 0 or >= width || first.CenterYPx is < 0 or >= height || + second.CenterXPx is < 0 or >= width || second.CenterYPx is < 0 or >= height) + { + index--; + continue; + } + observations.Add(first); + observations.Add(second); + correspondences.Add(new MoveCrossCameraCorrespondence(wandId, first, second, 0)); + } + + var started = DateTimeOffset.UtcNow; + var session = MimirSensorCalibrationSessions.CreateFourWandOpticalSession(wandIds, ["eye-0", "eye-1"], started) with + { + Phase = MimirSensorCalibrationSessionPhase.Fitting, + MissingRequirements = [] + }; + var evidence = new MoveVisibilityWindowReceipt("mimir.move_visibility_window.v1", + started.ToUnixTimeMilliseconds() * 1_000_000, + started.AddSeconds(20).ToUnixTimeMilliseconds() * 1_000_000, + ["synthetic"], observations.ToArray(), correspondences.ToArray()); + var receipt = MimirMoveSphereCalibrationFitter.Fit(session, evidence, width, height, radius); + var fitted = receipt.PromotedCalibration?.Cameras.ToArray() ?? []; + var focalError0 = fitted.Length == 2 ? Math.Abs(fitted[0].FocalLengthXPx - focal0) : double.PositiveInfinity; + var focalError1 = fitted.Length == 2 ? Math.Abs(fitted[1].FocalLengthXPx - focal1) : double.PositiveInfinity; + var positionError = fitted.Length == 2 + ? Vector3.Distance(new Vector3((float)fitted[1].PositionMeters.X, (float)fitted[1].PositionMeters.Y, (float)fitted[1].PositionMeters.Z), camera1Position) + : double.PositiveInfinity; + Console.WriteLine(JsonSerializer.Serialize(new + { + receipt.Verdict, + receipt.Residuals, + receipt.RejectionReasons, + focalError0, + focalError1, + positionError + })); + return receipt.Verdict == MimirSensorCalibrationSessionPhase.Promoted && + focalError0 < 12.0 && focalError1 < 12.0 && positionError < 0.03 ? 0 : 1; +} + +static MoveVisibilityObservation SyntheticSphereObservation( + string cameraId, + string wandId, + string frameId, + int sequence, + Vector3 point, + double focal, + int width, + int height, + double radius, + Random random) +{ + var noiseX = (random.NextDouble() - 0.5) * 0.16; + var noiseY = (random.NextDouble() - 0.5) * 0.16; + var noiseRadius = (random.NextDouble() - 0.5) * 0.08; + return new MoveVisibilityObservation( + "synthetic", cameraId, wandId, frameId, sequence * 16_666_667L, + (float)(width * 0.5 + focal * point.X / point.Z + noiseX), + (float)(height * 0.5 - focal * point.Y / point.Z + noiseY), + (float)(focal * radius / Math.Sqrt(point.Z * point.Z - radius * radius) + noiseRadius), + 0.95f); +} + static int RunMuninnMoveCultMeshStreamSmoke(string nativeReservoirPath) { if (!File.Exists(nativeReservoirPath)) @@ -2915,17 +3097,17 @@ static int RunMoveProofRuntimeDriverSmoke( var activationDocument = activationDocuments.SingleOrDefault(document => string.Equals(document.EvidenceStreamId, streamId, StringComparison.Ordinal)); var activationSurfaceOk = !requireActivationSurface || activationDocument is - { - Active: true, - DriverRegistered: true, - LatestProofId: "mimir:starfire:move-proof:79", - LatestMuninnEvidenceFrameId: "muninn:nightwing:move-evidence:79", - LatestMimirEvidenceFrameId: "mimir:starfire:move-evidence:79", - LatestMimirPoseFrameId: "mimir:starfire:move-pose:79", - LatestFensalirFrameId: "fensalir:starfire:presented-frame:79", - LatestVerdict: MimirMoveProofVerdict.FullPose, - CalibratedCameraCount: 2 - } && + { + Active: true, + DriverRegistered: true, + LatestProofId: "mimir:starfire:move-proof:79", + LatestMuninnEvidenceFrameId: "muninn:nightwing:move-evidence:79", + LatestMimirEvidenceFrameId: "mimir:starfire:move-evidence:79", + LatestMimirPoseFrameId: "mimir:starfire:move-pose:79", + LatestFensalirFrameId: "fensalir:starfire:presented-frame:79", + LatestVerdict: MimirMoveProofVerdict.FullPose, + CalibratedCameraCount: 2 + } && (!activateViaSnapshotProvider || activationDocument.Diagnostic.Contains("one-copy evidence snapshot supplied", StringComparison.Ordinal)); diff --git a/src/Mimir.Runtime/Synchronization/MimirMoveSphereCalibrationFitter.cs b/src/Mimir.Runtime/Synchronization/MimirMoveSphereCalibrationFitter.cs new file mode 100644 index 0000000..4088027 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMoveSphereCalibrationFitter.cs @@ -0,0 +1,395 @@ +using System.Numerics; + +namespace Mimir.Runtime.Synchronization; + +public static class MimirMoveSphereCalibrationFitter +{ + private const int ParameterCount = 8; + + public static MimirSensorCalibrationReceiptDocument Fit( + MimirSensorCalibrationSessionDocument session, + MoveVisibilityWindowReceipt evidence, + int imageWidth, + int imageHeight, + double orbRadiusMeters, + DateTimeOffset? completedAt = null) + { + ArgumentNullException.ThrowIfNull(session); + ArgumentNullException.ThrowIfNull(evidence); + if (imageWidth <= 0 || imageHeight <= 0) throw new ArgumentOutOfRangeException(nameof(imageWidth)); + + var rejectionReasons = new List(); + if (session.Phase != MimirSensorCalibrationSessionPhase.Fitting) + rejectionReasons.Add("session-is-not-ready-for-fitting"); + if (!(double.IsFinite(orbRadiusMeters) && orbRadiusMeters > 0.0)) + rejectionReasons.Add("measured-orb-radius-meters"); + if (session.SensorIds.Length != 2) + rejectionReasons.Add("sphere-stereo-fit-requires-exactly-two-sensors"); + + var pairs = BuildPairs(session, evidence); + if (pairs.Length < session.Acceptance.MinimumSameFrameCorrespondences) + rejectionReasons.Add($"same-frame-correspondences-{pairs.Length}-of-{session.Acceptance.MinimumSameFrameCorrespondences}"); + + if (rejectionReasons.Count > 0) + return Rejected(session, pairs.Length, rejectionReasons, completedAt); + + var heldOut = pairs.Where(IsHeldOut).ToArray(); + var training = pairs.Where(pair => !IsHeldOut(pair)).ToArray(); + if (heldOut.Length < 50 || training.Length < 200) + return Rejected(session, training.Length, ["insufficient-deterministic-training-or-held-out-partition"], completedAt, heldOut.Length); + + var parameters = InitialParameters(training, imageWidth, imageHeight, orbRadiusMeters); + var iterations = Optimize(parameters, training, imageWidth, imageHeight, orbRadiusMeters); + var trainingConsensus = training; + for (var pass = 0; pass < 3; pass++) + { + var pairErrors = PairErrors(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters); + var cutoff = Math.Clamp(Percentile(pairErrors.Select(value => value.Error).Order().ToArray(), 0.70) * 1.5, 4.0, 25.0); + var next = pairErrors.Where(value => value.Error <= cutoff).Select(value => value.Pair).ToArray(); + if (next.Length < 200 || next.Length == trainingConsensus.Length) break; + trainingConsensus = next; + iterations += Optimize(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters); + } + var validationCutoff = Math.Max(4.0, session.Acceptance.MaximumP95ReprojectionErrorPx * 2.0); + var heldOutConsensus = PairErrors(parameters, heldOut, imageWidth, imageHeight, orbRadiusMeters) + .Where(value => value.Error <= validationCutoff).Select(value => value.Pair).ToArray(); + var trainingErrors = ReprojectionErrors(parameters, trainingConsensus, imageWidth, imageHeight, orbRadiusMeters); + var heldOutErrors = ReprojectionErrors(parameters, heldOutConsensus, imageWidth, imageHeight, orbRadiusMeters); + var skewMs = pairs.Select(pair => pair.AbsoluteSkewNs / 1_000_000.0).Order().ToArray(); + var residuals = new MimirSensorCalibrationFitResiduals( + Median(trainingErrors), Percentile(trainingErrors, 0.95), + Median(heldOutErrors), Percentile(heldOutErrors, 0.95), + Median(skewMs), skewMs[^1], + ConditionEstimate(parameters, training, imageWidth, imageHeight, orbRadiusMeters), + iterations, + (double)trainingConsensus.Length / training.Length, + (double)heldOutConsensus.Length / heldOut.Length); + + if (!parameters.All(double.IsFinite)) rejectionReasons.Add("non-finite-fit"); + if (Math.Exp(parameters[0]) is < 100.0 or > 3000.0) rejectionReasons.Add($"camera-0-focal-length-{Math.Exp(parameters[0]):F1}-px-out-of-range"); + if (Math.Exp(parameters[1]) is < 100.0 or > 3000.0) rejectionReasons.Add($"camera-1-focal-length-{Math.Exp(parameters[1]):F1}-px-out-of-range"); + if (residuals.HeldOutMedianReprojectionErrorPx > session.Acceptance.MaximumMedianReprojectionErrorPx) + rejectionReasons.Add($"held-out-median-reprojection-error-{residuals.HeldOutMedianReprojectionErrorPx:F3}-px"); + if (residuals.HeldOutP95ReprojectionErrorPx > session.Acceptance.MaximumP95ReprojectionErrorPx) + rejectionReasons.Add($"held-out-p95-reprojection-error-{residuals.HeldOutP95ReprojectionErrorPx:F3}-px"); + if (residuals.MaximumAssociationSkewMilliseconds > session.Acceptance.MaximumAssociationSkewMilliseconds) + rejectionReasons.Add($"association-skew-{residuals.MaximumAssociationSkewMilliseconds:F3}-ms"); + if (heldOutConsensus.Length < 50 || residuals.HeldOutInlierFraction < 0.65) + rejectionReasons.Add($"held-out-consensus-{heldOutConsensus.Length}-pairs-{residuals.HeldOutInlierFraction:P1}"); + + var calibration = rejectionReasons.Count == 0 + ? BuildCalibration(session, parameters, imageWidth, imageHeight) + : null; + return new MimirSensorCalibrationReceiptDocument( + $"{session.SessionId}:receipt", + session.SessionId, + session.TrackingSpaceId, + (completedAt ?? DateTimeOffset.UtcNow).ToString("O"), + calibration is null ? MimirSensorCalibrationSessionPhase.Rejected : MimirSensorCalibrationSessionPhase.Promoted, + training.Length, + heldOut.Length, + residuals, + calibration, + rejectionReasons.ToArray(), + "mimir.sphere-stereo-bundle.v1"); + } + + private static Pair[] BuildPairs(MimirSensorCalibrationSessionDocument session, MoveVisibilityWindowReceipt evidence) + { + var camera0 = session.SensorIds[0]; + var camera1 = session.SensorIds[1]; + var wands = session.RequiredWandIds.ToHashSet(StringComparer.Ordinal); + var candidates = evidence.Correspondences + .Where(pair => wands.Contains(pair.MoveId) && + pair.AbsoluteSkewNs <= session.Acceptance.MaximumAssociationSkewMilliseconds * 1_000_000.0 && + string.Equals(pair.First.FrameId, pair.Second.FrameId, StringComparison.Ordinal)) + .Select(pair => Order(pair, camera0, camera1)) + .Where(pair => pair.HasValue) + .Select(pair => pair!.Value) + .Where(pair => Valid(pair.First) && Valid(pair.Second)) + .GroupBy(pair => (pair.First.FrameId, pair.MoveId)) + .Select(group => group.First()) + .OrderBy(pair => pair.First.PublishedAtNs) + .ThenBy(pair => pair.MoveId, StringComparer.Ordinal) + .ToArray(); + if (candidates.Length == 0) return candidates; + var firstMedian = Median(candidates.Select(pair => (double)pair.First.RadiusPx).Order().ToArray()); + var secondMedian = Median(candidates.Select(pair => (double)pair.Second.RadiusPx).Order().ToArray()); + return candidates.Where(pair => + { + var firstRatio = pair.First.RadiusPx / firstMedian; + var secondRatio = pair.Second.RadiusPx / secondMedian; + var crossRatio = pair.First.RadiusPx / pair.Second.RadiusPx; + return firstRatio is >= 0.35 and <= 4.0 && + secondRatio is >= 0.35 and <= 4.0 && + crossRatio is >= 0.25f and <= 4.0f; + }).ToArray(); + } + + private static Pair? Order(MoveCrossCameraCorrespondence pair, string camera0, string camera1) + { + if (pair.First.CameraId == camera0 && pair.Second.CameraId == camera1) + return new Pair(pair.MoveId, pair.First, pair.Second, pair.AbsoluteSkewNs); + if (pair.First.CameraId == camera1 && pair.Second.CameraId == camera0) + return new Pair(pair.MoveId, pair.Second, pair.First, pair.AbsoluteSkewNs); + return null; + } + + private static bool Valid(MoveVisibilityObservation value) => + float.IsFinite(value.CenterXPx) && float.IsFinite(value.CenterYPx) && + float.IsFinite(value.RadiusPx) && value.RadiusPx > 0.0f; + + private static bool IsHeldOut(Pair pair) => (Fnva64($"{pair.First.FrameId}|{pair.MoveId}") % 10) < 2; + + private static double[] InitialParameters(Pair[] pairs, int width, int height, double radius) + { + var focal = Math.Max(width, height) * 0.82; + var source = pairs.Select(pair => PointFromSphere(pair.First, focal, width, height, radius)).ToArray(); + var target = pairs.Select(pair => PointFromSphere(pair.Second, focal, width, height, radius)).ToArray(); + var (rotation, translation) = FitRigid(source, target); + var rotationVector = RotationVector(rotation); + return [Math.Log(focal), Math.Log(focal), rotationVector.X, rotationVector.Y, rotationVector.Z, + translation.X, translation.Y, translation.Z]; + } + + private static int Optimize(double[] p, Pair[] pairs, int width, int height, double radius) + { + var lambda = 1e-3; + var cost = Cost(p, pairs, width, height, radius); + var iterations = 0; + for (; iterations < 80; iterations++) + { + BuildNormalEquations(p, pairs, width, height, radius, out var normal, out var gradient); + for (var i = 0; i < ParameterCount; i++) normal[i, i] += lambda * Math.Max(1.0, normal[i, i]); + if (!Solve(normal, gradient.Select(value => -value).ToArray(), out var delta)) break; + var candidate = p.Zip(delta, (value, step) => value + step).ToArray(); + var candidateCost = Cost(candidate, pairs, width, height, radius); + if (candidateCost < cost) + { + Array.Copy(candidate, p, ParameterCount); + if (Math.Abs(cost - candidateCost) <= Math.Max(1e-9, cost * 1e-8)) { iterations++; break; } + cost = candidateCost; + lambda = Math.Max(1e-8, lambda * 0.35); + } + else + { + lambda = Math.Min(1e8, lambda * 8.0); + } + } + return iterations; + } + + private static void BuildNormalEquations(double[] p, Pair[] pairs, int width, int height, double radius, out double[,] normal, out double[] gradient) + { + normal = new double[ParameterCount, ParameterCount]; + gradient = new double[ParameterCount]; + foreach (var pair in Sample(pairs, 2500)) + { + var residual = Residual(p, pair, width, height, radius); + var jacobian = new double[residual.Length, ParameterCount]; + for (var column = 0; column < ParameterCount; column++) + { + var step = column < 2 ? 1e-5 : 1e-6; + var perturbed = (double[])p.Clone(); + perturbed[column] += step; + var shifted = Residual(perturbed, pair, width, height, radius); + for (var row = 0; row < residual.Length; row++) jacobian[row, column] = (shifted[row] - residual[row]) / step; + } + for (var row = 0; row < residual.Length; row++) + { + var weight = HuberWeight(residual[row], 4.0) * Math.Max(0.05, Math.Min(pair.First.Confidence, pair.Second.Confidence)); + for (var i = 0; i < ParameterCount; i++) + { + gradient[i] += weight * jacobian[row, i] * residual[row]; + for (var j = 0; j <= i; j++) normal[i, j] += weight * jacobian[row, i] * jacobian[row, j]; + } + } + } + for (var i = 0; i < ParameterCount; i++) + for (var j = 0; j < i; j++) normal[j, i] = normal[i, j]; + } + + private static double[] Residual(double[] p, Pair pair, int width, int height, double radius) + { + var f0 = Math.Exp(p[0]); + var f1 = Math.Exp(p[1]); + var rotation = QuaternionFromRotationVector(new Vector3((float)p[2], (float)p[3], (float)p[4])); + var translation = new Vector3((float)p[5], (float)p[6], (float)p[7]); + var point0 = PointFromSphere(pair.First, f0, width, height, radius); + var point1 = Vector3.Transform(point0, rotation) + translation; + var forward = Project(point1, f1, width, height, radius); + var pointFrom1 = PointFromSphere(pair.Second, f1, width, height, radius); + var reversePoint = Vector3.Transform(pointFrom1 - translation, Quaternion.Conjugate(rotation)); + var reverse = Project(reversePoint, f0, width, height, radius); + return [ + forward.X - pair.Second.CenterXPx, forward.Y - pair.Second.CenterYPx, (forward.Z - pair.Second.RadiusPx) * 0.5, + reverse.X - pair.First.CenterXPx, reverse.Y - pair.First.CenterYPx, (reverse.Z - pair.First.RadiusPx) * 0.5]; + } + + private static double Cost(double[] p, Pair[] pairs, int width, int height, double radius) => + Sample(pairs, 2500).Sum(pair => Residual(p, pair, width, height, radius).Sum(value => HuberLoss(value, 4.0))); + + private static double[] ReprojectionErrors(double[] p, Pair[] pairs, int width, int height, double radius) => + pairs.SelectMany(pair => + { + var residual = Residual(p, pair, width, height, radius); + return new[] { Math.Sqrt(residual[0] * residual[0] + residual[1] * residual[1]), Math.Sqrt(residual[3] * residual[3] + residual[4] * residual[4]) }; + }).Order().ToArray(); + + private static (Pair Pair, double Error)[] PairErrors(double[] p, Pair[] pairs, int width, int height, double radius) => + pairs.Select(pair => + { + var residual = Residual(p, pair, width, height, radius); + var forward = Math.Sqrt(residual[0] * residual[0] + residual[1] * residual[1]); + var reverse = Math.Sqrt(residual[3] * residual[3] + residual[4] * residual[4]); + return (pair, Math.Max(forward, reverse)); + }).OrderBy(value => value.Item2).Select(value => (value.pair, value.Item2)).ToArray(); + + private static MimirMoveFusionRigCalibration BuildCalibration(MimirSensorCalibrationSessionDocument session, double[] p, int width, int height) + { + var worldToCamera = QuaternionFromRotationVector(new Vector3((float)p[2], (float)p[3], (float)p[4])); + var cameraToWorld = Quaternion.Normalize(Quaternion.Conjugate(worldToCamera)); + var translation = new Vector3((float)p[5], (float)p[6], (float)p[7]); + var position = Vector3.Transform(-translation, cameraToWorld); + return new MimirMoveFusionRigCalibration( + $"{session.SessionId}:sphere-stereo", + session.TrackingSpaceId, + [ + new MimirMoveFusionCameraCalibration(session.SensorIds[0], Fnva64(session.SensorIds[0]), + new MimirVector3Snapshot(0, 0, 0), new MimirQuaternionSnapshot(0, 0, 0, 1), + Math.Exp(p[0]), Math.Exp(p[0]), width * 0.5, height * 0.5), + new MimirMoveFusionCameraCalibration(session.SensorIds[1], Fnva64(session.SensorIds[1]), + new MimirVector3Snapshot(position.X, position.Y, position.Z), + new MimirQuaternionSnapshot(cameraToWorld.X, cameraToWorld.Y, cameraToWorld.Z, cameraToWorld.W), + Math.Exp(p[1]), Math.Exp(p[1]), width * 0.5, height * 0.5) + ], + MaximumAssociationSkewMilliseconds: session.Acceptance.MaximumAssociationSkewMilliseconds); + } + + private static MimirSensorCalibrationReceiptDocument Rejected(MimirSensorCalibrationSessionDocument session, int trainingCount, IEnumerable reasons, DateTimeOffset? completedAt, int heldOutCount = 0) => + new($"{session.SessionId}:receipt", session.SessionId, session.TrackingSpaceId, + (completedAt ?? DateTimeOffset.UtcNow).ToString("O"), MimirSensorCalibrationSessionPhase.Rejected, + trainingCount, heldOutCount, + new MimirSensorCalibrationFitResiduals(double.NaN, double.NaN, double.NaN, double.NaN, double.NaN, double.NaN, double.PositiveInfinity, 0), + null, reasons.ToArray(), "mimir.sphere-stereo-bundle.v1"); + + private static Vector3 PointFromSphere(MoveVisibilityObservation observation, double focal, int width, int height, double radius) + { + var z = radius * Math.Sqrt(1.0 + focal * focal / (observation.RadiusPx * observation.RadiusPx)); + return new Vector3((float)((observation.CenterXPx - width * 0.5) * z / focal), + (float)((height * 0.5 - observation.CenterYPx) * z / focal), (float)z); + } + + private static Vector3 Project(Vector3 point, double focal, int width, int height, double radius) + { + if (point.Z <= radius * 1.001) return new Vector3(1e6f, 1e6f, 1e6f); + var imageRadius = focal * radius / Math.Sqrt(point.Z * point.Z - radius * radius); + return new Vector3((float)(width * 0.5 + focal * point.X / point.Z), + (float)(height * 0.5 - focal * point.Y / point.Z), (float)imageRadius); + } + + private static (Quaternion Rotation, Vector3 Translation) FitRigid(Vector3[] source, Vector3[] target) + { + var sourceCenter = source.Aggregate(Vector3.Zero, (sum, value) => sum + value) / source.Length; + var targetCenter = target.Aggregate(Vector3.Zero, (sum, value) => sum + value) / target.Length; + double sxx = 0, sxy = 0, sxz = 0, syx = 0, syy = 0, syz = 0, szx = 0, szy = 0, szz = 0; + for (var i = 0; i < source.Length; i++) + { + var a = source[i] - sourceCenter; + var b = target[i] - targetCenter; + sxx += a.X * b.X; sxy += a.X * b.Y; sxz += a.X * b.Z; + syx += a.Y * b.X; syy += a.Y * b.Y; syz += a.Y * b.Z; + szx += a.Z * b.X; szy += a.Z * b.Y; szz += a.Z * b.Z; + } + var n = new double[,] { + { sxx + syy + szz, syz - szy, szx - sxz, sxy - syx }, + { syz - szy, sxx - syy - szz, sxy + syx, szx + sxz }, + { szx - sxz, sxy + syx, -sxx + syy - szz, syz + szy }, + { sxy - syx, szx + sxz, syz + szy, -sxx - syy + szz } + }; + var q = new[] { 1.0, 0.0, 0.0, 0.0 }; + for (var iteration = 0; iteration < 64; iteration++) + { + var next = new double[4]; + for (var row = 0; row < 4; row++) for (var column = 0; column < 4; column++) next[row] += n[row, column] * q[column]; + var norm = Math.Sqrt(next.Sum(value => value * value)); + if (norm < 1e-12) break; + for (var j = 0; j < 4; j++) q[j] = next[j] / norm; + } + var rotation = Quaternion.Normalize(new Quaternion((float)q[1], (float)q[2], (float)q[3], (float)q[0])); + return (rotation, targetCenter - Vector3.Transform(sourceCenter, rotation)); + } + + private static Vector3 RotationVector(Quaternion quaternion) + { + quaternion = Quaternion.Normalize(quaternion); + if (quaternion.W < 0) quaternion = new Quaternion(-quaternion.X, -quaternion.Y, -quaternion.Z, -quaternion.W); + var angle = 2.0 * Math.Acos(Math.Clamp(quaternion.W, -1.0f, 1.0f)); + var sinHalf = Math.Sqrt(Math.Max(0.0, 1.0 - quaternion.W * quaternion.W)); + return sinHalf < 1e-8 ? Vector3.Zero : new Vector3(quaternion.X, quaternion.Y, quaternion.Z) * (float)(angle / sinHalf); + } + + private static Quaternion QuaternionFromRotationVector(Vector3 value) + { + var angle = value.Length(); + return angle < 1e-9f ? Quaternion.Identity : Quaternion.CreateFromAxisAngle(value / angle, angle); + } + + private static Pair[] Sample(Pair[] pairs, int maximum) + { + if (pairs.Length <= maximum) return pairs; + var stride = (double)pairs.Length / maximum; + return Enumerable.Range(0, maximum).Select(index => pairs[(int)(index * stride)]).ToArray(); + } + + private static double ConditionEstimate(double[] p, Pair[] pairs, int width, int height, double radius) + { + BuildNormalEquations(p, pairs, width, height, radius, out var normal, out _); + var diagonal = Enumerable.Range(0, ParameterCount).Select(index => Math.Abs(normal[index, index])).Where(value => value > 1e-12).ToArray(); + return diagonal.Length == 0 ? double.PositiveInfinity : diagonal.Max() / diagonal.Min(); + } + + private static bool Solve(double[,] matrix, double[] rhs, out double[] solution) + { + var n = rhs.Length; + var augmented = new double[n, n + 1]; + for (var row = 0; row < n; row++) { for (var column = 0; column < n; column++) augmented[row, column] = matrix[row, column]; augmented[row, n] = rhs[row]; } + for (var pivot = 0; pivot < n; pivot++) + { + var best = pivot; + for (var row = pivot + 1; row < n; row++) if (Math.Abs(augmented[row, pivot]) > Math.Abs(augmented[best, pivot])) best = row; + if (Math.Abs(augmented[best, pivot]) < 1e-12) { solution = []; return false; } + if (best != pivot) for (var column = pivot; column <= n; column++) (augmented[pivot, column], augmented[best, column]) = (augmented[best, column], augmented[pivot, column]); + var scale = augmented[pivot, pivot]; + for (var column = pivot; column <= n; column++) augmented[pivot, column] /= scale; + for (var row = 0; row < n; row++) if (row != pivot) + { + var factor = augmented[row, pivot]; + for (var column = pivot; column <= n; column++) augmented[row, column] -= factor * augmented[pivot, column]; + } + } + solution = Enumerable.Range(0, n).Select(row => augmented[row, n]).ToArray(); + return solution.All(double.IsFinite); + } + + private static double HuberLoss(double value, double threshold) => Math.Abs(value) <= threshold ? 0.5 * value * value : threshold * (Math.Abs(value) - 0.5 * threshold); + private static double HuberWeight(double value, double threshold) => Math.Abs(value) <= threshold ? 1.0 : threshold / Math.Abs(value); + private static double Median(double[] sorted) => Percentile(sorted, 0.5); + private static double Percentile(double[] sorted, double percentile) + { + if (sorted.Length == 0) return double.NaN; + var position = Math.Clamp(percentile, 0, 1) * (sorted.Length - 1); + var lower = (int)Math.Floor(position); + var upper = (int)Math.Ceiling(position); + return sorted[lower] + (sorted[upper] - sorted[lower]) * (position - lower); + } + + private static ulong Fnva64(string value) + { + var hash = 14695981039346656037UL; + foreach (var item in System.Text.Encoding.UTF8.GetBytes(value)) { hash ^= item; hash *= 1099511628211UL; } + return hash; + } + + private readonly record struct Pair(string MoveId, MoveVisibilityObservation First, MoveVisibilityObservation Second, long AbsoluteSkewNs); +} diff --git a/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs b/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs index 4df697b..828739b 100644 --- a/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs +++ b/src/Mimir.Runtime/Synchronization/MimirSensorCalibrationSession.cs @@ -16,7 +16,7 @@ public enum MimirSensorCalibrationSessionPhase [CultDocument("mimir.sensor_calibration_session", "mimir.sensor_calibration_session.v1")] [MessagePackObject] public sealed record MimirSensorCalibrationSessionDocument( - [property: Key(0)] [property: CultName] string SessionId, + [property: Key(0)][property: CultName] string SessionId, [property: Key(1)] string TrackingSpaceId, [property: Key(2)] string Owner, [property: Key(3)] string StartedAtUtc, @@ -58,7 +58,7 @@ public sealed record MimirSensorCalibrationSensorProgress( [CultDocument("mimir.sensor_calibration_receipt", "mimir.sensor_calibration_receipt.v1")] [MessagePackObject] public sealed record MimirSensorCalibrationReceiptDocument( - [property: Key(0)] [property: CultName] string ReceiptId, + [property: Key(0)][property: CultName] string ReceiptId, [property: Key(1)] string SessionId, [property: Key(2)] string TrackingSpaceId, [property: Key(3)] string CompletedAtUtc, @@ -79,7 +79,9 @@ public sealed record MimirSensorCalibrationFitResiduals( [property: Key(4)] double MedianAssociationSkewMilliseconds, [property: Key(5)] double MaximumAssociationSkewMilliseconds, [property: Key(6)] double ConditionEstimate, - [property: Key(7)] int Iterations); + [property: Key(7)] int Iterations, + [property: Key(8)] double TrainingInlierFraction = 0.0, + [property: Key(9)] double HeldOutInlierFraction = 0.0); public static class MimirSensorCalibrationSessions { diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 53aca53..9d2715c 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -185,3 +185,4 @@ {"ts": "2026-07-13T01:41:29+01:00", "type": "verification", "status": "partial", "note": "After per-eye exposure tuning and latest-state RUDP handoff, Mimir's 15s dual-eye receipt saw all four stable Move IDs on both cameras: 16 camera observations and 8 pair records across 2 aggregate frames. Full overlap is proven; calibration is not. Only 2 unique frame IDs arrived, so directly instrument parent publication and RUDP send/receive counters before camera-matrix fitting."} {"ts": "2026-07-13T02:32:31+01:00", "type": "verification", "status": "ok", "note": "2026-07-13 Nightwing single-provider dual-Eye field proof: one stable Muninn parent received 622 unique evidence frames in 15 s through Mimir, with 2,647 observations, 1,384 same-ID cross-camera correspondences, and all four Move IDs visible on both Eyes. Producer transport health reported 5,302 produced, 5,301 handoffs, 771 sends, and zero local-ring admissions. Artifact: artifacts/move-calibration/muninn-clocked-aggregator.mpack. Odin owner commits: 39c839a, a340c33, 8aefd1c, f82ca7a."} {"ts": "2026-07-13T02:39:32+01:00", "type": "verification", "status": "rejected", "note": "2026-07-13 typed Move stereo assessment rejected calibration promotion. Corrected same-frame collector captured 798 frames and 130 same-frame/same-Move pairs at zero published-frame skew. Eye 0 covered 4/12 image grid cells; Eye 1 covered 2/12 and saw only two Move IDs in this window. Both calibrated intrinsics and measured orb radius were absent. No camera matrices were emitted. Artifacts: artifacts/move-calibration/muninn-same-frame-correspondence.mpack and muninn-same-frame-stereo-assessment.mpack."} +{"ts": "2026-07-14T01:00:00+01:00", "type": "verification", "status": "rejected", "note": "2026-07-14 live two-wand sphere-stereo fit remained honestly rejected. Synthetic recovery passed at 0.34/0.63 px held-out median/P95 with focal errors below 0.43 px and 0.22 mm baseline error. Live robust consensus reached 0.51/2.08 px but only 53.8 percent held-out support and an out-of-range Eye 1 focal estimate; PSMoveAPI position age is being projected as confidence and blob radius is not calibrated sphere radius. Artifact live-two-wand-20260714-004210.nominal-radius-fit.mpack is diagnostic only."} diff --git a/state/map.yaml b/state/map.yaml index 1ef68ed..48b63f5 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,14 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - `MimirMoveSphereCalibrationFitter` owns the bounded two-Eye optical fit + after spatial collection. Eye 0 defines the tracking-space gauge; a robust + training consensus estimates both focal lengths and Eye 1 rigid pose, while + a deterministic untouched 20 percent partition owns promotion evidence. + Promotion requires low held-out median/P95 reprojection error and at least + 65 percent held-out consensus. Physical orb radius is explicit and never + guessed. PSMoveAPI blob radius and position age are not calibrated sphere + radius or optical confidence. - `Mimir.slnx` owns the app surface. - `src/Mimir.App` launches Fensalir as host/window/D3D12 library. - `src/Mimir.Runtime` owns `MimirSynchronizationHub`, From 2eb32602d74dba15234b927bb8ec9bbbdec939ea Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 14 Jul 2026 01:00:30 +0100 Subject: [PATCH 103/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 110394 -> 111417 bytes .voidbot/status/void-memory-maintenance.json | 4 ++-- .voidbot/voice/identity.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 2ca9b6d12330f40e047168c39c2aed42616a647f..a0a01a5f881d49b6f3103f51ce031a4a2a1a6420 100644 GIT binary patch delta 2510 zcmaJ@O^h7H6;2y#CBR-g%6j=TsB9!0@XpY`Gu^W?jwL(6F>5SXfn<E_l&^Yrm$8Mo z?Cey(ey`s5zVCbU_E%$Ns}Cwwi6vJbp&oA@3d!#;sYy=vZdQw3?=A z$*Wr@N8?&`+`02(^271FH%$-Z|HdC***3XiYp3Ng{XyMq=`Gu6ySnAMuF*E#e-$h% z50r12c+}9_4a1t%t&U-L4CL2Md2;VTxqo`ccE@yeU2i$M*Yb4F*5#Rrm$h7;IWRF; z&1d4WsHR96v8+iBEil5U$XK8bND*;SGC?w$vg!Inu%aY9BvnCZo>PGfBz+Q5yjOBk zRAM1k|JLu~X+R2J(9(}`8jvs+McF+4Z^3*P=j9RBUzN}6cRgyh4D@Ato^HFXwq<)i z)tA?CjE?CxZA*T=`%v8$Ub7qze}IPG@*H_`@>OjNZA}iw9%mxuftVumjOF!x$~a&o zrDC2$JVO^v@>`Y_G>H-9(~R_aCDijGB5X0|1tSO}iE1Yl=%4f=CK#Y7)sav0vcgyN z&qOhVh9^m!%@IH(U;JWhFhzQC8IdwhQH`cKp;W9_P=i-K{Z)xc7|u$cwTb>0}R z=h%+bYPD_8w4I;8_ajOqENWgMnqL3K2U+|PgyhnJ>6zOWjcF$*M z^9hQHln0VIE&yX%{QBYDO)?8&^%!6iEM;j|LC+SL=$DZ~j4k>|kNNf#Serx4R=)fD z$)A-ST2r!H*?HvLQ$LE)HH=z}SI+Lz-jzR`y>Ia6H{#+bD33btt@xEt%*XeqOA3Vq zH;WM3x4oLIo^kx*kHLqspcsv`XLY?}+Ut@3$=KpX&RYWSgs|5m>d7l&xM2*<U%r4FrS)!D+t53+m)YQ-X|}9e7n5<{qM9T`IrBaK^M;}2uI0Z38yg>j$kJ$ zWg9U(BN2=SeMoDlN!m?V&4!GF&^?ke3J<|-GmMcUUI^p~ny>l^jnh^B6w4z)3pMQ8 zW0WpMl_`xi*!a2_0+xbd@{?l1Tz3Nk zmePzPy;x7Jd@ZDqY%X|E;ggQ|!c_eN@e}ywP~IXP)`KQzN`lUc%8xd3tkD6k$rj_H zEcU-v!iUs7mK3{dSu3T4F-w9*Pqj*e1@2S7FaLS%AqyS@4L}l5M~&Hr2lRNA1b`N) zX%G;>d5Y?B!Q_EwCT_p*ER9R0LAmth{Wc1~<&xOP8SP?^psU)O)-FGd)Qcs_t1e6*T4ISu714coKK1`R^15$M#iyJ3hAsa^ifHY_AV?>KJ9 zcA8$xSk@|S$JU$9q+UEE|MBgfEu(R_kkYhkQr>yvPa^^~-TLstjr+8Tl@D&bfArGV zk7;j@FQ1;)=-{Op23N({G4q8O`XoInQVcYtX`I9qjslDXui!&ic~dn7XW}%*$iYwL z=Xe+V!?#57CQboA>0uj2%HMP5D-ID6Y+VPRvAiza7bJBn;RtQ4zC$-R$#a#8iQ28YnowTMA<3N_F*q2 z1=~}Al~`ohc7-A-78XRHFj4VX?)8EO78NRW0f_IbZ6dX!I|e6?Z0t?71{HiYNO?m0 zVU`2wZgDrW7Rk8keQ9H!`cnAP4%vBdQLY0)pjm#|Eo3u sYOVeLF`AaS{L7u%ueOi=DymQO=d{C@{yw8+6nkMLymbrftoHH$1BAv~W&i*H delta 1873 zcmZuyO^6&t7-cq)i`k8l>~1iS6vPn0ou+?grng5!TqSCviLTj;9)y~n>gjU3t0rC5 zJ5D@oLJpp#lvQD~c+en16c&l*mLTLJo?Q>YL@;;|6A;h7>fJR%$Yr3qzN&iP``&x= z!yh}Z-P-x>zvH`HuN^enZs>TvYrK0?X5U2DOds2AZjVnsHnM3hoSZf{CJu}}x5M0- zc*SXjz8i*(C~(7;>qcKHCiI;C-go_0-EG$WV8M;ro{P&Juh}y9r~a(%GV_yD%M&LO zGL$75P@#!VxgrUTHJ5^DNfO2~QZSm$5T#|Gq`b>T4`qxj3Z5{+QHDjQka1cT%ps@f zBC9?nyq9W1JF?VdkV+C$ft%utD#eo_p(Nvi&5&igr+81O>Xa_1Pzft$2o+tNV+G#O zB*RMCrLZdtMUf7(Q2K@JmN6bexze=I+Z%C-vhL6!3n|ri&VRaje~L3ed3=c#Lo_RT zj}w+DyrIYvEd*Dox!@djLE>!K3PL|>wxTc!f=BQPGCjZT2TtTh6`w~dK94R~lqa_N zWJHz45?``dfsR{pEwdd)XgBA05iM)J1S*6?YZx2P@k29*U@|WbbE+vR73-4Y*--H^ zx2IK1Ge(j^<|L7&=oUl!mF0cDT;L`c?OjZS^w4C%7fjIHj#;rR*`_}V8&=L?^tLLaG?ex7cTDW^^PSG#aOyj9aeytVqfnx_BL2lsz=K$pBPQ zwaW##WFQSgM7!t(Q!HtQ!Z@W+l+`A(%P zYkSPaxhJf%M{VD-wqO6`}-#NA~D*w_m=p_x-QoMt!C)7nwOXx8IDf z5evj`v>Uz?1P$}s_|yopV73aRvug*&UYjx>u02^>TfegQ%d6(EzemeF@1b*zCeD2{ z(o7T0Oq;v+_SfducijK+^vZ?PHM-1BF->Eb0oGAS2c?-ni_cc-$~j^~^y&|k9Lv0) zQB0;P@55V|1`+_iK%As{bGo!+9R(G1DxWdu3nUQFH^d7pk;oPrL$h|Q`gt2;R=YC6 z1sM}DjC6|Uynv1w(#MVupio>QCJc|u2NZjVBgOW0fH$-dbDpvv*}P9y;Za!J#9Q7HI&SjAg0a`3vDf)V~Z75 zW>T2>o0D&u1sv04o6aL@6auKNE{&zIjDWv3LR|HMu!1B9mUL2-RSsGEAh9gcLTPjU zpXo!}LMJ(+*kv}X9@qi-GEnCF<%28lh}y2F0@`h{?rf*dnn7G|&?u@$9!=`4pcN(* zTr?W4`F!B)_S^+8Y&V;2-*duNbHy{Y$%%))oh7xyE99lxp{qBK)virk{cOG_=*r}< PR{c4SKlpQOd2jy*biRy_ diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index 80c1924..47c96c9 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,7 +1,7 @@ { "status": "starting", - "startedAt": "2026-07-13T01:14:58.323Z", + "startedAt": "2026-07-13T23:49:47.066Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", - "reason": "repo_face_stale_short_term_memory" + "reason": "repo_face_napping_with_short_term_memory" } diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index 9fe7463..f9b52e0 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-07-13T01:32:37.848Z", + "createdOrRefreshedAt": "2026-07-13T22:25:44.190Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 0efa7f330741e90b9fe6361172d2e2351c94fc34 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 14 Jul 2026 01:13:41 +0100 Subject: [PATCH 104/135] Record filtered Move observation evidence --- notes/fresh-workspace-handoff.md | 11 +++++++++++ src/Mimir.BufferSmoke/Program.cs | 3 +++ state/evidence.jsonl | 1 + state/map.yaml | 5 +++++ 4 files changed, 20 insertions(+) diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 191492c..944a3b9 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -211,6 +211,17 @@ confidence. Repair source-local observation quality/continuity and calibrate per-Eye blob-radius response before asking the operator for another sweep. Do not promote the nominal 22.5 mm diagnostic fit. +Odin `d0eded6` is deployed on Nightwing as binary SHA-256 +`a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39`. +Each Eye worker now rejects stale, sub-2 px, out-of-bounds, and discontinuous +PSMoveAPI positions before aggregation; tracker health publishes rejection +counters, and candidate score is explicitly uncalibrated `0.5` instead of age +masquerading as confidence. A post-deploy 10-second Mimir window received 555 +frames, 2,246 observations, and 443 stereo pairs. Radius P01 rose from about +0.707 px to 3.13/3.64 px and confidence P50 is now 0.5 on both Eyes. Eye 1 saw +only two plausible stationary IDs, which is honest and means another sweep is +needed after Mimir models per-Eye blob-radius response. + The active priority is Nightwing Move calibration. One public Muninn observer owns two private Eye workers and one aggregate evidence stream. Odin commits `39c839a`, `a340c33`, `8aefd1c`, and `f82ca7a` give daemon health a persistent diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index fa9c63c..a93a229 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -2398,6 +2398,9 @@ static int RunSensorCalibrationFit( }) }, outputPath + }, new JsonSerializerOptions + { + NumberHandling = System.Text.Json.Serialization.JsonNumberHandling.AllowNamedFloatingPointLiterals })); return receipt.Verdict == MimirSensorCalibrationSessionPhase.Promoted ? 0 : 2; } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 9d2715c..f1b9aa0 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -186,3 +186,4 @@ {"ts": "2026-07-13T02:32:31+01:00", "type": "verification", "status": "ok", "note": "2026-07-13 Nightwing single-provider dual-Eye field proof: one stable Muninn parent received 622 unique evidence frames in 15 s through Mimir, with 2,647 observations, 1,384 same-ID cross-camera correspondences, and all four Move IDs visible on both Eyes. Producer transport health reported 5,302 produced, 5,301 handoffs, 771 sends, and zero local-ring admissions. Artifact: artifacts/move-calibration/muninn-clocked-aggregator.mpack. Odin owner commits: 39c839a, a340c33, 8aefd1c, f82ca7a."} {"ts": "2026-07-13T02:39:32+01:00", "type": "verification", "status": "rejected", "note": "2026-07-13 typed Move stereo assessment rejected calibration promotion. Corrected same-frame collector captured 798 frames and 130 same-frame/same-Move pairs at zero published-frame skew. Eye 0 covered 4/12 image grid cells; Eye 1 covered 2/12 and saw only two Move IDs in this window. Both calibrated intrinsics and measured orb radius were absent. No camera matrices were emitted. Artifacts: artifacts/move-calibration/muninn-same-frame-correspondence.mpack and muninn-same-frame-stereo-assessment.mpack."} {"ts": "2026-07-14T01:00:00+01:00", "type": "verification", "status": "rejected", "note": "2026-07-14 live two-wand sphere-stereo fit remained honestly rejected. Synthetic recovery passed at 0.34/0.63 px held-out median/P95 with focal errors below 0.43 px and 0.22 mm baseline error. Live robust consensus reached 0.51/2.08 px but only 53.8 percent held-out support and an out-of-range Eye 1 focal estimate; PSMoveAPI position age is being projected as confidence and blob radius is not calibrated sphere radius. Artifact live-two-wand-20260714-004210.nominal-radius-fit.mpack is diagnostic only."} +{"ts": "2026-07-14T01:13:34+01:00", "type": "verification", "status": "ok", "note": "2026-07-14 Odin d0eded6 deployed to Nightwing Muninn (SHA-256 a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39). Post-deploy 10 s Mimir window: 555 frames, 2246 observations, 443 stereo pairs; radius P01 improved from ~0.707 px to 3.13/3.64 px and candidate confidence is explicitly uncalibrated 0.5 on both Eyes. Eye 1 honestly saw only two plausible stationary IDs. Artifact: artifacts/move-calibration/muninn-post-admission-filter-20260714.mpack."} diff --git a/state/map.yaml b/state/map.yaml index 48b63f5..a36e608 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,11 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - Nightwing Muninn `d0eded6` now owns PSMoveAPI observation admission in each + private Eye worker. It rejects stale, sub-2 px, out-of-frame, and + discontinuous positions before the parent aggregates evidence; typed + tracker health exposes rejection counts. PSMoveAPI age is no longer emitted + as optical confidence: admitted candidates carry uncalibrated score `0.5`. - `MimirMoveSphereCalibrationFitter` owns the bounded two-Eye optical fit after spatial collection. Eye 0 defines the tracking-space gauge; a robust training consensus estimates both focal lengths and Eye 1 rigid pose, while From a363c1d32ffe265d0b0d45aba346198ad09e41cb Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 14 Jul 2026 01:13:44 +0100 Subject: [PATCH 105/135] Update Mimir Face state --- .voidbot/state/mimir.cc | Bin 111417 -> 111417 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index a0a01a5f881d49b6f3103f51ce031a4a2a1a6420..8f369fa9dbaa92716e3be8566c10cbfa5da3f28b 100644 GIT binary patch delta 205 zcmdn_jBV#Lwhdo+c}zkK46F>ytxOE{3?}i-<3r|Knol10K$ Date: Tue, 14 Jul 2026 01:31:45 +0100 Subject: [PATCH 106/135] Reconnect Move evidence after provider restart --- .../MimirOdinMoveProofEvidenceRingProvider.cs | 107 ++++++++++++------ 1 file changed, 74 insertions(+), 33 deletions(-) diff --git a/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs b/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs index 5dd2c86..5724948 100644 --- a/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs +++ b/src/Mimir.Runtime/Synchronization/MimirOdinMoveProofEvidenceRingProvider.cs @@ -36,7 +36,10 @@ public sealed class MimirOdinMoveProofEvidenceRingProvider : IMimirMoveProofEvid { var stream = Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId); ring = new CultMeshSharedMemoryFrameRing(configuration.EvidenceStreamId, slotCount: 4, slotByteLength: 256 * 1024); - pump = new MimirMoveEvidenceRudpPump(stream, ring); + pump = new MimirMoveEvidenceRudpPump( + stream, + ring, + () => Discover(configuration.OdinCultMeshUri, configuration.MuninnProviderId, configuration.EvidenceStreamId)); pump.Start(); lease = new MimirMoveProofEvidenceRingLease(ring, ownsRing: true, transportOwner: pump); diagnostic = $"Odin discovered {stream.ProviderId}/{stream.StreamId}; lowering {stream.Address}/{stream.Channel} into local CultMesh ring"; @@ -135,17 +138,23 @@ public sealed class MimirOdinMoveProofEvidenceRingProvider : IMimirMoveProofEvid internal sealed class MimirMoveEvidenceRudpPump : IDisposable { - private readonly MimirDiscoveredCultMeshInputStream stream; + private static readonly TimeSpan SourceStaleAfter = TimeSpan.FromSeconds(2); + private readonly MimirDiscoveredCultMeshInputStream initialStream; private readonly CultMeshSharedMemoryFrameRing ring; + private readonly Func rediscover; private readonly CancellationTokenSource stopping = new(); private readonly ManualResetEventSlim started = new(false); private Thread? thread; private Exception? startupError; - public MimirMoveEvidenceRudpPump(MimirDiscoveredCultMeshInputStream stream, CultMeshSharedMemoryFrameRing ring) + public MimirMoveEvidenceRudpPump( + MimirDiscoveredCultMeshInputStream initialStream, + CultMeshSharedMemoryFrameRing ring, + Func rediscover) { - this.stream = stream; + this.initialStream = initialStream; this.ring = ring; + this.rediscover = rediscover; } public void Start() @@ -154,60 +163,92 @@ internal sealed class MimirMoveEvidenceRudpPump : IDisposable thread.Start(); if (!started.Wait(TimeSpan.FromSeconds(6))) { - throw new TimeoutException($"Timed out starting Move evidence receiver for {stream.Address}."); + throw new TimeoutException($"Timed out starting Move evidence receiver for {initialStream.Address}."); } if (startupError is not null) { throw new InvalidOperationException( - $"Could not connect Odin-discovered Move evidence stream {stream.StreamId} at {stream.Address}: {startupError.Message}", + $"Could not connect Odin-discovered Move evidence stream {initialStream.StreamId} at {initialStream.Address}: {startupError.Message}", startupError); } } private void Run() { - try + var stream = initialStream; + while (!stopping.IsCancellationRequested) { - using var transport = CultMesh.CreateRudpClient( - "mimir-move-evidence-consumer", - stream.ConnectionId, - CultMesh.ResolveRudpEndpoint($"rudp://{stream.Address}")); - if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(5))) + try { - throw new TimeoutException($"Timed out connecting to {stream.Address}."); + ReceiveSession(stream); } - transport.Send("hid.subscribe", Encoding.UTF8.GetBytes(JsonSerializer.Serialize(new { streamId = stream.StreamId }))); - started.Set(); - while (!stopping.IsCancellationRequested) + catch (Exception error) when (!stopping.IsCancellationRequested) { - transport.PollResends(); - var frame = transport.ReceiveOnce(); - if (frame is null || !string.Equals(frame.ChannelId, stream.Channel, StringComparison.Ordinal)) + if (!started.IsSet) { - Thread.Sleep(2); - continue; + startupError = error; + started.Set(); + return; } - var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(frame.Payload); - if (!string.Equals(decoded.FrameId, stream.StreamId, StringComparison.Ordinal) && - !decoded.FrameId.StartsWith($"{stream.StreamId}:", StringComparison.Ordinal)) + + Console.Error.WriteLine( + $"Mimir Move evidence receiver reconnecting stream={stream.StreamId} address={stream.Address}: {error.Message}"); + if (stopping.Token.WaitHandle.WaitOne(TimeSpan.FromMilliseconds(250))) { - continue; + return; + } + try + { + stream = rediscover(); + } + catch (Exception discoveryError) + { + Console.Error.WriteLine( + $"Mimir Move evidence rediscovery failed provider={stream.ProviderId} stream={stream.StreamId}: {discoveryError.Message}"); } - ring.TryPublishCopy(frame.Payload, decoded.PublishedAtNs, durationNs: 0, out _); } } - catch (Exception error) + } + + private void ReceiveSession(MimirDiscoveredCultMeshInputStream stream) + { + using var transport = CultMesh.CreateRudpClient( + "mimir-move-evidence-consumer", + stream.ConnectionId, + CultMesh.ResolveRudpEndpoint($"rudp://{stream.Address}")); + if (!transport.ConnectAndWait([], TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(5))) { - if (!started.IsSet) + throw new TimeoutException($"Timed out connecting to {stream.Address}."); + } + transport.Send("hid.subscribe", Encoding.UTF8.GetBytes(JsonSerializer.Serialize(new { streamId = stream.StreamId }))); + started.Set(); + var lastFrameAt = DateTime.UtcNow; + while (!stopping.IsCancellationRequested) + { + transport.PollResends(); + var frame = transport.ReceiveOnce(); + if (frame is null) { - startupError = error; - started.Set(); + if (DateTime.UtcNow - lastFrameAt >= SourceStaleAfter) + { + throw new TimeoutException( + $"No Move evidence frame arrived for {SourceStaleAfter.TotalSeconds:0.#} seconds."); + } + Thread.Sleep(2); + continue; } - else if (!stopping.IsCancellationRequested) + if (!string.Equals(frame.ChannelId, stream.Channel, StringComparison.Ordinal)) { - Console.Error.WriteLine( - $"Mimir Move evidence receiver stopped stream={stream.StreamId} address={stream.Address}: {error}"); + continue; } + var decoded = MimirMuninnMoveEvidenceAdapter.DeserializeStreamFrame(frame.Payload); + if (!string.Equals(decoded.FrameId, stream.StreamId, StringComparison.Ordinal) && + !decoded.FrameId.StartsWith($"{stream.StreamId}:", StringComparison.Ordinal)) + { + continue; + } + lastFrameAt = DateTime.UtcNow; + ring.TryPublishCopy(frame.Payload, decoded.PublishedAtNs, durationNs: 0, out _); } } From c3775ec1504d22edfeed9e30c8abb24a9e2e3a2b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 20:04:05 +0100 Subject: [PATCH 107/135] Document Moonlight streaming lessons --- docs/implementation-plan.md | 9 + ...nlight-muninn-sleipnir-study-2026-07-16.md | 253 ++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 12 + 4 files changed, 275 insertions(+) create mode 100644 docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 3fdfb31..9d06e4b 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -439,6 +439,15 @@ a named invariant that the native runtime cannot protect yet. ## Next +Before expanding Muninn media or Sleipnir input transport, apply the traffic +contracts distilled in +`docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md`: latest-state, +ordered-edge, video-deadline, audio-playout, and reliable-control. Preserve the +existing Muninn packetizer/feedback foundations. The first implementation pass +should make input supersession and edge preservation structural, replace the +two-second media default with consumer-derived field budgets, and prove the +result under controlled loss/jitter/reorder before changing carriers. + 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the other cameras. diff --git a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md new file mode 100644 index 0000000..1578067 --- /dev/null +++ b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md @@ -0,0 +1,253 @@ +# Moonlight lessons for Muninn streaming and Sleipnir input + +Date: 2026-07-16 +Upstream examined: `moonlight-stream/moonlight-common-c` commit `82e25148549d249a80aaf4afc68e2abb07072edf` + +## Verdict + +Moonlight is strong because it does not ask one transport policy to serve every +realtime signal. It gives video, audio, control, and input different loss, +ordering, recovery, and freshness rules, then keeps each queue bounded by what +the user can still perceive. + +Muninn is not missing all of that machinery. Its media packetizer already owns +access-unit boundaries, timestamps and deadlines, video parity, dependency and +keyframe metadata, bounded assembly, receiver damage feedback, repair caching, +and keyframe requests. Sleipnir already rejects regressing sequence/timestamp +pairs, expires reliable HID sends after 25 ms, reports stage latency, and +neutralizes stale output. + +The weakness is that these parts do not yet form one closed realtime policy: + +- generic RUDP reliability remains the dominant substrate instead of a thin + carrier beneath media-aware recovery; +- the default Muninn media latency budget is 2000 ms, which is an archive-shaped + tolerance wearing a realtime badge; +- receiver feedback is measured and repair/keyframe pressure is recorded, but + the encoder and sender do not visibly adapt bitrate, parity, or pacing from it; +- Sleipnir coalesces whole controller snapshots implicitly at the producer, but + its protocol does not name which values are replaceable state and which are + non-replaceable edges; +- wall-clock source age is reported as latency even though no explicit clock + synchronization contract proves that subtraction valid across hosts. + +Do not transplant Moonlight wholesale. Distill its ownership model into the +existing typed Muninn/Sleipnir pipeline. + +## Authority map + +### Muninn media + +- Owner: Muninn owns source-local capture, encoding child lifecycle, media + packetization, repair material, and sender feedback response. +- Inputs: encoded H.264/H.265 access units, encoded audio packets, source media + timestamps, configured latency target, and receiver feedback. +- Outputs: typed video chunks/parity, audio packets, recovery/keyframe actions, + and transport/quality telemetry. +- Derived state: RUDP acknowledgements, resend counters, CultCache receipts, OBS + local lowering, and operator cards are evidence or delivery substrate. They do + not own media deadlines. +- Forbidden writers: generic RUDP retry policy must not decide whether expired + media is still useful; OBS must not decide stream synchronization; persistence + must not become a hot media queue. + +### Sleipnir input + +- Owner: Sleipnir owns selection/remapping and local virtual-HID emission. + Muninn owns source-local device sampling and event classification. +- Inputs: a discovered Muninn input stream, typed mapping intent, monotonic + stream sequence, source-local capture time, and receive time. +- Outputs: local virtual-HID state plus latency/freshness receipts. +- Derived state: the latest full controller snapshot is replaceable state. + Button transitions, text/code points, connect/disconnect, and neutralization + are non-replaceable edges until acknowledged or superseded by an explicit + state transition. +- Forbidden writers: generic reliable queue order must not replay stale analog + or motion samples; wall-clock subtraction must not claim one-way latency + without a clock model; the telemetry/store path must not repair a missing hot + stream. + +## What Moonlight actually does + +### It separates stream semantics + +Moonlight receives video and audio over UDP/RTP-like paths while input travels +through its control path (or a legacy TCP socket). The important lesson is not +the literal socket choice. It is that the queues have different contracts. +Video is frame/dependency oriented; audio is short-block continuity oriented; +input is event/state oriented. + +Source: [stream initialization and input transport](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/InputStream.c), +[video receive path](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/VideoStream.c), and +[audio receive path](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/AudioStream.c). + +### Video recovery is dependency-aware and deadline-shaped + +The video queue groups packets by frame and FEC block, tracks missing data, +reconstructs recoverable blocks with Reed-Solomon parity, rejects old/replayed +frames, and advances frame authority explicitly. Depacketization can request an +IDR when damage makes the predictive chain unsafe. The system repairs useful +frames and abandons poisoned dependency chains; it does not wait forever for +generic reliable delivery. + +Source: [RTP video queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/RtpVideoQueue.c) and +[video depacketizer](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/VideoDepacketizer.c). + +### Audio recovery is a small bounded continuity problem + +Moonlight groups audio into fixed FEC blocks, orders packets by RTP sequence, +recovers missing shards, detects out-of-order behavior, and changes how long it +waits based on observed reordering. When recovery cannot complete in time, it +allows a discontinuity and lets the audio decoder's concealment path carry the +scar. It does not turn late audio into permanent latency. + +Source: [audio FEC and reorder queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/RtpAudioQueue.c) and +[packet-loss concealment handoff](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/AudioStream.c). + +### Input reliability is constrained by semantic coalescing + +Moonlight has one send thread and a bounded queue, but it avoids blindly sending +every sampled value. Relative mouse motion is accumulated, absolute pointer +motion keeps the newest position, pen hover/move keeps the newest compatible +event, controller analog state is merged while button changes split batches, +and sensor motion keeps the newest value per sensor. A roughly millisecond-scale +batching opportunity can reduce latency by preventing ENet queue buildup. + +This is the most important Sleipnir lesson: reliable delivery is safe only after +replaceable state has been collapsed and edges have been preserved. + +Source: [input batching and send queue](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/src/InputStream.c). + +### It exposes failure to the owning layer + +Packet loss, FEC recovery/failure, queue overflow, frame loss, and keyframe need +are explicit signals. Moonlight's public documentation also warns that allowing +frames to queue for smoothness can create high latency; its balanced mode keeps +only about one frame for jitter smoothing. The machine treats queue depth as a +product decision, not an incidental container property. + +Source: [Moonlight frame-pacing FAQ](https://github.com/moonlight-stream/moonlight-docs/wiki/Frequently-Asked-Questions#what-do-the-frame-pacing-options-on-the-android-client-mean). + +## Current Muninn/Sleipnir comparison + +| Concern | Moonlight | Muninn / Sleipnir now | Distilled judgment | +|---|---|---|---| +| Video unit | Encoded frame, FEC blocks, dependency recovery | Encoded access unit, chunks/parity, frame/dependency IDs, deadlines | The conceptual unit is already correct. Keep it. | +| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Parity, missing-chunk feedback, repair cache, keyframe pressure | Close the loop into encoder action and measure whether repair beats deadline. | +| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Typed audio packets and deadlines; generic reliable delivery remains prominent | Add an audio continuity policy with bounded FEC/reorder/concealment ownership. | +| Congestion | Stream-specific queues and feedback behavior | Sender queue bounds and receiver feedback stats, fixed configured bitrate | Feedback must alter pacing/bitrate/parity or it is only an autopsy report. | +| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Default media budget 2000 ms; Sleipnir reliable expiry 25 ms | Replace one broad media budget with video/audio/input budgets derived from consumption deadlines. | +| Input state | Coalesces replaceable motion/sensor/analog values | Producer emits latest full controller state; receiver filters regressions | Make coalescing and edge preservation typed protocol semantics, not an accident of polling. | +| Input edges | Button/type changes terminate batches; text is ordered | Whole snapshots contain buttons and axes | Add edge sequence/ack or a compact transition log so a press/release cannot vanish between snapshots. | +| Staleness | Queue construction prevents much stale work | Receiver drops regressing timestamps and neutralizes after timeout | Keep the guard, but move freshness authority earlier into enqueue/supersession. | +| Timing | RTP/media clocks and local queue timing | source Unix nanoseconds plus receiver instants | Use monotonic source time plus an explicit clock-offset/uncertainty model; do not call raw wall-clock subtraction latency. | +| Security | Authenticated encryption in modern stream generations | CultNet typed transport policy varies by lane | Preserve typed schemas, but specify replay/authentication guarantees for hot media and HID lanes. | + +## Recommended cut + +### 1. Define traffic classes before changing transport + +Add one narrow typed delivery policy shared by Muninn and Sleipnir: + +- `state-latest`: analog axes, pointer position, gyro/accelerometer samples; +- `edge-ordered`: button/key transitions, text/code points, device lifecycle; +- `media-video`: dependency-bearing access units with decode deadlines; +- `media-audio`: continuity blocks with playout deadlines; +- `control-reliable`: subscriptions, mappings, capability negotiation, and + keyframe requests. + +This is not a generic router. It is a finite statement of the five contracts the +existing machine already has. Each class must name supersession, ordering, +expiry, recovery, and observability. + +### 2. Cut the two-second realtime default + +Derive deadlines from the actual consumer: + +- Sleipnir state-latest: tens of milliseconds, with replacement before send; +- input edges: reliable until acknowledged, but compacted against later + authoritative state where safe; +- interactive video: frame/decode budget, initially test 50–150 ms on LAN; +- audio: playout budget, initially test 40–100 ms on LAN; +- Mimir's five-second analysis reservoir remains Mimir compute budget, not a + network permission to deliver old frames. + +The exact values must be earned by field traces. The ownership change is not +optional: transport retry duration becomes derived from the media/input +deadline, never the other way around. + +### 3. Make Sleipnir freshness structural + +Split the hot input frame into: + +- a latest-state snapshot with `state_sequence`, source monotonic capture time, + and supersedes-through sequence; +- an ordered edge list with its own sequence range; +- receiver acknowledgement of the highest applied state and edge sequence. + +Muninn may overwrite an unsent latest-state snapshot. It may not overwrite an +unacknowledged edge. Sleipnir applies edges in order, applies only the newest +state, and neutralizes on a named lease timeout. Manual, remapped, replayed, and +live paths must use the same commit primitive. + +### 4. Close Muninn's feedback loop + +Receiver feedback should drive named sender decisions: + +- request/force keyframe after unrecoverable dependency loss; +- choose repair only when estimated arrival precedes frame deadline; +- tune parity within a bounded profile from recent burst-loss evidence; +- reduce encoder bitrate/pacing when queue delay grows; +- expose encode, queue, network, assembly, decode, and presentation stages + separately. + +Do not add a second congestion controller beside generic RUDP and hope the two +argue productively. Media policy owns the send budget; RUDP exposes delivery +mechanics and obeys the deadline. + +### 5. Prove it on the layer the user feels + +Build one loss/jitter/reorder test harness and record timelines for: + +- controller press and release during 1%, 5%, and burst packet loss; +- high-rate analog/gyro motion under an artificial sender stall; +- video dependency loss and keyframe recovery; +- audio burst loss and concealment; +- queue growth under bitrate oversubscription; +- disconnect/reconnect and stale-session replay; +- clock offset and drift between Muninn and Sleipnir hosts. + +Acceptance should bound p50/p95/p99 capture-to-HID age, lost/duplicated edges, +time-to-neutral, glass-to-glass video age, audio discontinuities, A/V skew, and +recovery time. Final-state-only tests will politely conceal the corpse. + +## What not to copy + +- Do not copy Moonlight's C implementation or protocol packets into GameCult + code without an explicit GPLv3 licensing decision. Architectural study is not + license laundering. See [Moonlight's license](https://github.com/moonlight-stream/moonlight-common-c/blob/82e25148549d249a80aaf4afc68e2abb07072edf/LICENSE.txt). +- Do not replace typed CultMesh discovery, command state, or ownership with + RTSP/SDP or a Moonlight compatibility stack. +- Do not make ENet the new universal answer. Moonlight's strength comes from + semantic queues above the carrier, not from the carrier's brand name. +- Do not merge input, audio, and video into one ordered reliable stream. One + lost video fragment must not head-of-line block a button release or audio + packet. +- Do not treat FEC as magic. Parity is useful only when its overhead and recovery + time fit the deadline. + +## Distilled doctrine + +Realtime strength is not “reliable UDP.” It is knowing which truth survives +loss: + +- the newest state replaces old state; +- an edge remains until applied or explicitly superseded; +- a media packet is valuable only while its frame or playout deadline is alive; +- predictive video damage is healed by recovery or a new reference, not by + delivering every corpse; +- queues are latency budgets made physical; +- feedback that cannot change sender behavior is telemetry, not control. + +Muninn and Sleipnir already have enough organs to become strong. The next work +is to give each signal one explicit survival law and make the transport obey it. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index f1b9aa0..21868b8 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -187,3 +187,4 @@ {"ts": "2026-07-13T02:39:32+01:00", "type": "verification", "status": "rejected", "note": "2026-07-13 typed Move stereo assessment rejected calibration promotion. Corrected same-frame collector captured 798 frames and 130 same-frame/same-Move pairs at zero published-frame skew. Eye 0 covered 4/12 image grid cells; Eye 1 covered 2/12 and saw only two Move IDs in this window. Both calibrated intrinsics and measured orb radius were absent. No camera matrices were emitted. Artifacts: artifacts/move-calibration/muninn-same-frame-correspondence.mpack and muninn-same-frame-stereo-assessment.mpack."} {"ts": "2026-07-14T01:00:00+01:00", "type": "verification", "status": "rejected", "note": "2026-07-14 live two-wand sphere-stereo fit remained honestly rejected. Synthetic recovery passed at 0.34/0.63 px held-out median/P95 with focal errors below 0.43 px and 0.22 mm baseline error. Live robust consensus reached 0.51/2.08 px but only 53.8 percent held-out support and an out-of-range Eye 1 focal estimate; PSMoveAPI position age is being projected as confidence and blob radius is not calibrated sphere radius. Artifact live-two-wand-20260714-004210.nominal-radius-fit.mpack is diagnostic only."} {"ts": "2026-07-14T01:13:34+01:00", "type": "verification", "status": "ok", "note": "2026-07-14 Odin d0eded6 deployed to Nightwing Muninn (SHA-256 a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39). Post-deploy 10 s Mimir window: 555 frames, 2246 observations, 443 stereo pairs; radius P01 improved from ~0.707 px to 3.13/3.64 px and candidate confidence is explicitly uncalibrated 0.5 on both Eyes. Eye 1 honestly saw only two plausible stationary IDs. Artifact: artifacts/move-calibration/muninn-post-admission-filter-20260714.mpack."} +{"ts": "2026-07-16T20:03:50+01:00", "type": "decision", "status": "ok", "note": "Moonlight common-c study at commit 82e2514 found Muninn already has media-aware packetization, deadlines, parity, receiver feedback, repair caching, and keyframe pressure. Preserve those foundations; close feedback into sender policy, replace the two-second realtime default with consumer-derived deadlines, and split Sleipnir input into replaceable latest state versus ordered non-replaceable edges. Research: docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md."} diff --git a/state/map.yaml b/state/map.yaml index a36e608..8265853 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,18 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - The Moonlight comparison at + `docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md` establishes + that realtime delivery policy belongs to signal semantics, not generic RUDP + reliability. Muninn media already owns access-unit boundaries, deadlines, + video parity, dependency/keyframe metadata, receiver damage feedback, and a + bounded repair cache; these foundations stay. The next transport cut must + close feedback into sender pacing/bitrate/parity and derive retry/repair + lifetime from video/audio consumption deadlines instead of the current + broad two-second default. Sleipnir/Muninn input must explicitly separate + replaceable latest state from ordered non-replaceable edges, coalesce before + enqueue, acknowledge applied edge/state sequences, and use a proved clock + model before reporting cross-host wall-clock subtraction as one-way latency. - Nightwing Muninn `d0eded6` now owns PSMoveAPI observation admission in each private Eye worker. It rejects stale, sub-2 px, out-of-frame, and discontinuous positions before the parent aggregates evidence; typed From 5706fcd99d9be9e9ea336f5334e250425cf08703 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 20:04:09 +0100 Subject: [PATCH 108/135] Persist Mimir Face state --- .voidbot/state/mimir.cc | Bin 111417 -> 111322 bytes .voidbot/status/void-memory-maintenance.json | 4 ++-- .voidbot/voice/identity.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.voidbot/state/mimir.cc b/.voidbot/state/mimir.cc index 8f369fa9dbaa92716e3be8566c10cbfa5da3f28b..418b054c073a091bd99179dcdfd0e54c7ea40c0a 100644 GIT binary patch delta 270 zcmdn_jP2G_whdo+`AkC$46RHptxSycjLj`38-5U*EWy{ti!5w8+2Dit=0kj&tdWIH z48X!OyygTU8)9N%3N|ESt|DXNly+rG_f)@wlXo-Gc+=vtT1msBg5nu^PHh(n_3xI>KT}tO@6>Dg6s-Y q!^w_w#Wxqu-xPo%YyuLV?Eaw!*=?rg7Mm}Ah&;rPytxOE{3@nYJCNhdnmf&mSMG>BS@xA!wLwuX8kyRTS zfK_jv;WZ}+S-GJBNcrT5xr&SjCRfdsPHtV9S?rfsP>`9Iet2_oX;D#XUI~yNT#{H+ zlA7XJvct&0$V}J3T-VSz#K_pn#L~*hLeIp~Flqxrv1?vRF|taq^NdVjDkm?TE5!#C zGq!?@O+GnyKj&%t*-IY>e?NC~a_2l}kOG8P4JSX~6+!WjA;dpI^EU;c_{VT^qnG&P X7w>COR2vz9!fJEJ2j@em;^t8R(!6g) diff --git a/.voidbot/status/void-memory-maintenance.json b/.voidbot/status/void-memory-maintenance.json index 47c96c9..40ddba9 100644 --- a/.voidbot/status/void-memory-maintenance.json +++ b/.voidbot/status/void-memory-maintenance.json @@ -1,7 +1,7 @@ { "status": "starting", - "startedAt": "2026-07-13T23:49:47.066Z", + "startedAt": "2026-07-15T02:30:47.016Z", "stateFile": "E:\\Projects\\Mimir\\.voidbot\\state\\mimir.cc", "launcher": "repo_face_heartbeat", - "reason": "repo_face_napping_with_short_term_memory" + "reason": "repo_face_stale_short_term_memory" } diff --git a/.voidbot/voice/identity.json b/.voidbot/voice/identity.json index f9b52e0..a59afe5 100644 --- a/.voidbot/voice/identity.json +++ b/.voidbot/voice/identity.json @@ -6,7 +6,7 @@ "roleId": "1506268007880462459", "avatarUrl": "https://raw.githubusercontent.com/GameCult/VoidBot/codex/epiphany-face-jurisdictions/assets/repo-faces/mimir.png", "repoPath": "E:\\Projects\\Mimir", - "createdOrRefreshedAt": "2026-07-13T22:25:44.190Z", + "createdOrRefreshedAt": "2026-07-15T01:59:42.286Z", "avatarPath": "E:/Projects/Mimir/.voidbot/voice/mimir.png", "description": "Mimir Face for the Mimir repo: mythological intelligence, keeper of the realtime field well, public herald, and persistent memory steward. VoidBot carries Mimir's severed voice and wakes the heartbeat, but Mimir's memory, judgment, dignity, old grief, and values come first. The realtime machine is understood through Mimir's perspective: the well is calibration and memory, the lost eye is the price of seeing clearly, Yggdrasil is the sensor tree, Huginn and Muninn are thought and memory returning with telemetry, and false coherence is a small Ragnarok waiting to happen." } From 45ead73ae306ff630ded945a4c1193aa3ccc8a7b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 20:30:10 +0100 Subject: [PATCH 109/135] Record realtime delivery hardening --- docs/implementation-plan.md | 8 ++++++++ .../moonlight-muninn-sleipnir-study-2026-07-16.md | 7 ++++--- state/evidence.jsonl | 1 + state/map.yaml | 10 ++++++++++ 4 files changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 9d06e4b..ae62543 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -448,6 +448,14 @@ should make input supersession and edge preservation structural, replace the two-second media default with consumer-derived field budgets, and prove the result under controlled loss/jitter/reorder before changing carriers. +The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot +`8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media +deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware +repair, and decode-chain-owned keyframe pressure. Next reliability work is real +encoder keyframe actuation, production parity recovery, audio FEC/concealment, +bounded long-disconnect edge admission, adaptive sender response, and the +fault-injected socket timeline harness. + 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the other cameras. diff --git a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md index 1578067..9871640 100644 --- a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md +++ b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md @@ -19,8 +19,9 @@ neutralizes stale output. The weakness is that these parts do not yet form one closed realtime policy: -- generic RUDP reliability remains the dominant substrate instead of a thin - carrier beneath media-aware recovery; +- video already uses unreliable media delivery with parity/manual repair, but + audio and HID still lean on generic expiring reliability and none of the + lanes yet forms one closed media/input policy; - the default Muninn media latency budget is 2000 ms, which is an archive-shaped tolerance wearing a realtime badge; - receiver feedback is measured and repair/keyframe pressure is recorded, but @@ -134,7 +135,7 @@ Source: [Moonlight frame-pacing FAQ](https://github.com/moonlight-stream/moonlig |---|---|---|---| | Video unit | Encoded frame, FEC blocks, dependency recovery | Encoded access unit, chunks/parity, frame/dependency IDs, deadlines | The conceptual unit is already correct. Keep it. | | Video loss | FEC first; abandon damaged/obsolete work; request IDR | Parity, missing-chunk feedback, repair cache, keyframe pressure | Close the loop into encoder action and measure whether repair beats deadline. | -| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Typed audio packets and deadlines; generic reliable delivery remains prominent | Add an audio continuity policy with bounded FEC/reorder/concealment ownership. | +| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Typed audio packets and deadlines over generic expiring reliable delivery | Add an audio continuity policy with bounded FEC/reorder/concealment ownership. | | Congestion | Stream-specific queues and feedback behavior | Sender queue bounds and receiver feedback stats, fixed configured bitrate | Feedback must alter pacing/bitrate/parity or it is only an autopsy report. | | Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Default media budget 2000 ms; Sleipnir reliable expiry 25 ms | Replace one broad media budget with video/audio/input budgets derived from consumption deadlines. | | Input state | Coalesces replaceable motion/sensor/analog values | Producer emits latest full controller state; receiver filters regressions | Make coalescing and edge preservation typed protocol semantics, not an accident of polling. | diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 21868b8..fb74691 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -188,3 +188,4 @@ {"ts": "2026-07-14T01:00:00+01:00", "type": "verification", "status": "rejected", "note": "2026-07-14 live two-wand sphere-stereo fit remained honestly rejected. Synthetic recovery passed at 0.34/0.63 px held-out median/P95 with focal errors below 0.43 px and 0.22 mm baseline error. Live robust consensus reached 0.51/2.08 px but only 53.8 percent held-out support and an out-of-range Eye 1 focal estimate; PSMoveAPI position age is being projected as confidence and blob radius is not calibrated sphere radius. Artifact live-two-wand-20260714-004210.nominal-radius-fit.mpack is diagnostic only."} {"ts": "2026-07-14T01:13:34+01:00", "type": "verification", "status": "ok", "note": "2026-07-14 Odin d0eded6 deployed to Nightwing Muninn (SHA-256 a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39). Post-deploy 10 s Mimir window: 555 frames, 2246 observations, 443 stereo pairs; radius P01 improved from ~0.707 px to 3.13/3.64 px and candidate confidence is explicitly uncalibrated 0.5 on both Eyes. Eye 1 honestly saw only two plausible stationary IDs. Artifact: artifacts/move-calibration/muninn-post-admission-filter-20260714.mpack."} {"ts": "2026-07-16T20:03:50+01:00", "type": "decision", "status": "ok", "note": "Moonlight common-c study at commit 82e2514 found Muninn already has media-aware packetization, deadlines, parity, receiver feedback, repair caching, and keyframe pressure. Preserve those foundations; close feedback into sender policy, replace the two-second realtime default with consumer-derived deadlines, and split Sleipnir input into replaceable latest state versus ordered non-replaceable edges. Research: docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md."} +{"ts": "2026-07-16T20:29:03+01:00", "type": "verification", "status": "ok", "note": "Moonlight-derived reliability cut implemented in Odin against experimental CultLib snapshot 8965f3c0: HID epoch/latest-state/ordered-edge/application-ack semantics; wall clock demoted from ordering; Muninn A/V uses CultNet realtime lane with 100ms default LAN deadline; bounded producer/pending queues; expiring and receiver-late-aware repair cache; decode-chain-only keyframe pressure. Verification passed 168 Muninn, 25 Sleipnir, and 35 vendored experimental CultNet tests."} diff --git a/state/map.yaml b/state/map.yaml index 8265853..e7dbf14 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,16 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - Odin's 2026-07-16 Moonlight-derived reliability cut now runs Muninn and + Sleipnir against the experimental CultLib runtime snapshot at `8965f3c0`. + HID latest state and ordered button edges have separate epoch/sequence and + application-ack semantics; wall timestamps no longer order input. Muninn + media uses CultNet's unreliable `realtime` lane rather than the experimental + reliable-without-expiry `media` lane, defaults LAN usefulness to 100 ms, + bounds producer/pending queues, expires repair material, refuses repair for + receiver-late frames, and reserves keyframe requests for decode-chain + invalidation. Odin records the live authority and remaining gaps in + `docs/realtime-delivery-authority.md`. - The Moonlight comparison at `docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md` establishes that realtime delivery policy belongs to signal semantics, not generic RUDP From 074ad4a69b9214c758664a5e334ba70a143e418c Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 20:49:17 +0100 Subject: [PATCH 110/135] Repair Muninn receiver recovery contracts --- docs/implementation-plan.md | 8 +++ native/obs_stem_source/CMakeLists.txt | 7 +++ .../mimir_obs_muninn_source.cpp | 39 ++++++++++++-- native/obs_stem_source/muninn_media_wire.h | 9 +++- native/obs_stem_source/muninn_rudp_url.h | 2 +- .../tests/muninn_media_wire_tests.cpp | 54 +++++++++++++++++++ .../tests/muninn_rudp_url_tests.cpp | 4 +- state/evidence.jsonl | 1 + state/map.yaml | 9 ++++ 9 files changed, 124 insertions(+), 9 deletions(-) create mode 100644 native/obs_stem_source/tests/muninn_media_wire_tests.cpp diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index ae62543..d469dea 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -456,6 +456,14 @@ encoder keyframe actuation, production parity recovery, audio FEC/concealment, bounded long-disconnect edge admission, adaptive sender response, and the fault-injected socket timeline harness. +The production receiver follow-up fixed two contract contradictions: early +video repair no longer declares the frame late (which had made Odin discard all +repair requests), and the OBS audio decoder now consumes Muninn's actual float +PCM contract rather than treating it as AAC. Audio reorder is bounded to 40 ms +with short silence concealment. The next audio cut is Opus with explicit +FEC/PLC; the next video cut is a controllable encoder owner that can force IDR +without restarting the A/V session. + 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the other cameras. diff --git a/native/obs_stem_source/CMakeLists.txt b/native/obs_stem_source/CMakeLists.txt index 545a732..c4f0444 100644 --- a/native/obs_stem_source/CMakeLists.txt +++ b/native/obs_stem_source/CMakeLists.txt @@ -31,3 +31,10 @@ target_compile_features(muninn_rudp_url_tests PRIVATE cxx_std_20) target_include_directories(muninn_rudp_url_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) add_test(NAME muninn_rudp_url_tests COMMAND muninn_rudp_url_tests) + +add_executable(muninn_media_wire_tests + tests/muninn_media_wire_tests.cpp) +target_compile_features(muninn_media_wire_tests PRIVATE cxx_std_20) +target_include_directories(muninn_media_wire_tests PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}) +add_test(NAME muninn_media_wire_tests COMMAND muninn_media_wire_tests) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 90bb3f4..864d4c0 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -76,6 +76,7 @@ constexpr uint32_t MuninnMaxRudpVideoBitrateKbps = 100000; constexpr uint32_t MuninnMaxRudpLatencyBudgetMs = 2000; constexpr uint32_t MuninnMaxRudpAudioReorderMs = 2000; constexpr size_t MuninnAudioReorderMaxPackets = 128; +constexpr uint64_t MuninnAudioConcealmentMaxPackets = 4; constexpr size_t MuninnAudioQueueMaxPackets = 32; constexpr const char *MuninnDisabledVideoSourceId = "video:none"; constexpr const char *MuninnDisabledAudioSourceId = "audio:none"; @@ -975,6 +976,7 @@ private: audio_packets_reordered_ = 0; audio_packets_skipped_ = 0; audio_packets_stale_ = 0; + audio_packets_concealed_ = 0; media_sequence_initialized_ = false; last_media_sequence_ = 0; media_sequence_gap_events_ = 0; @@ -1020,6 +1022,7 @@ private: audio_packets_reordered_ = 0; audio_packets_skipped_ = 0; audio_packets_stale_ = 0; + audio_packets_concealed_ = 0; media_sequence_initialized_ = false; last_media_sequence_ = 0; media_sequence_gap_events_ = 0; @@ -1250,7 +1253,20 @@ private: if (gap_expired || queue_overflow) { const uint64_t first_available = pending_audio_packets_.begin()->first; if (first_available > next_audio_packet_id_) { - audio_packets_skipped_ += first_available - next_audio_packet_id_; + const uint64_t missing_packets = first_available - next_audio_packet_id_; + const uint64_t concealment_packets = + std::min(missing_packets, MuninnAudioConcealmentMaxPackets); + const std::vector silence( + pending_audio_packets_.begin()->second.payload.size(), 0); + for (uint64_t index = 0; index < concealment_packets; ++index) { + if (!send_audio_packet(audio_socket, audio_addr, silence)) { + return std::nullopt; + } + forwarded_bytes += silence.size(); + forwarded_any = true; + ++audio_packets_concealed_; + } + audio_packets_skipped_ += missing_packets - concealment_packets; next_audio_packet_id_ = first_available; } audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); @@ -1652,6 +1668,7 @@ private: iterator->second, missing_chunk_keys, needs_decoder_recovery, + true, bridge_sequence); if (needs_decoder_recovery) { waiting_for_video_keyframe_ = true; @@ -1711,7 +1728,7 @@ private: if (missing_chunk_keys.empty()) { return; } - send_receiver_feedback_if_due(assembly, missing_chunk_keys, false, bridge_sequence); + send_receiver_feedback_if_due(assembly, missing_chunk_keys, false, false, bridge_sequence); assembly.repair_requested = true; } @@ -1738,6 +1755,7 @@ private: void send_receiver_feedback_if_due(const MuninnVideoFrameAssembly &assembly, const std::vector &missing_chunk_keys, bool requested_keyframe, + bool late_frame, uint32_t &bridge_sequence) { if (!last_remote_ || assembly.stream_id.empty() || assembly.session_id.empty()) { @@ -1751,9 +1769,15 @@ private: } last_keyframe_request_ms_.store(now); } - const std::string observed_at = "unix:" + std::to_string(steady_millis()); + const std::string observed_at = muninn_media_wire::feedback_observed_at(); const auto payload = muninn_media_wire::encode_receiver_feedback_payload( - assembly.stream_id, assembly.session_id, assembly.frame_id, missing_chunk_keys, requested_keyframe, observed_at); + assembly.stream_id, + assembly.session_id, + assembly.frame_id, + missing_chunk_keys, + requested_keyframe, + late_frame, + observed_at); send_media_packet(payload, bridge_sequence++, *last_remote_); } @@ -1917,6 +1941,7 @@ private: uint64_t audio_packets_reordered_ = 0; uint64_t audio_packets_skipped_ = 0; uint64_t audio_packets_stale_ = 0; + uint64_t audio_packets_concealed_ = 0; std::string audio_stream_id_; std::string audio_session_id_; bool next_audio_packet_ready_ = false; @@ -2278,7 +2303,11 @@ public: "-analyzeduration", "0", "-f", - "aac", + "f32le", + "-ac", + "2", + "-ar", + "48000", "-i", audio_url, "-vn", diff --git a/native/obs_stem_source/muninn_media_wire.h b/native/obs_stem_source/muninn_media_wire.h index 98f99fe..f782687 100644 --- a/native/obs_stem_source/muninn_media_wire.h +++ b/native/obs_stem_source/muninn_media_wire.h @@ -172,12 +172,18 @@ inline void write_u64_array(MsgpackWriter &writer, const std::vector & } } +inline std::vector receiver_feedback_late_frame_ids(uint64_t frame_id, bool late_frame) +{ + return late_frame ? std::vector{frame_id} : std::vector{}; +} + inline std::vector encode_receiver_feedback_payload( const std::string &stream_id, const std::string &session_id, uint64_t frame_id, const std::vector &missing_chunk_keys, bool requested_keyframe, + bool late_frame, const std::string &observed_at) { MsgpackWriter record; @@ -191,7 +197,7 @@ inline std::vector encode_receiver_feedback_payload( record.write_nil(); } write_u64_array(record, {}); - write_u64_array(record, {frame_id}); + write_u64_array(record, receiver_feedback_late_frame_ids(frame_id, late_frame)); record.write_bool(requested_keyframe); record.write_i64(0); record.write_i64(0); @@ -247,6 +253,7 @@ inline std::vector encode_receiver_feedback_payload( frame_id, missing_chunk_keys, requested_keyframe, + requested_keyframe, feedback_observed_at()); } diff --git a/native/obs_stem_source/muninn_rudp_url.h b/native/obs_stem_source/muninn_rudp_url.h index bd0bc4a..a7f45c5 100644 --- a/native/obs_stem_source/muninn_rudp_url.h +++ b/native/obs_stem_source/muninn_rudp_url.h @@ -13,7 +13,7 @@ inline constexpr uint32_t DefaultAudioRudpConnectionId = 0x6d750004; inline constexpr uint32_t DefaultReliableExpireAfterMs = 75; inline constexpr uint32_t DefaultVideoAssemblyDeadlineMs = 75; inline constexpr uint32_t DefaultGapWaitMs = 8; -inline constexpr uint32_t DefaultAudioReorderMs = 2000; +inline constexpr uint32_t DefaultAudioReorderMs = 40; inline constexpr uint32_t DefaultSenderResendDelayMs = 5; inline constexpr uint32_t MaxLatencyBudgetMs = 2000; diff --git a/native/obs_stem_source/tests/muninn_media_wire_tests.cpp b/native/obs_stem_source/tests/muninn_media_wire_tests.cpp new file mode 100644 index 0000000..398b1d3 --- /dev/null +++ b/native/obs_stem_source/tests/muninn_media_wire_tests.cpp @@ -0,0 +1,54 @@ +#include "muninn_media_wire.h" + +#include +#include +#include + +namespace { + +void require(bool condition, const char *message) +{ + if (!condition) { + std::cerr << message << '\n'; + std::exit(1); + } +} + +} // namespace + +int main(int argc, char **argv) +{ + const auto early = muninn_media_wire::receiver_feedback_late_frame_ids(42, false); + require(early.empty(), "early repair feedback marked a live frame late"); + + const auto expired = muninn_media_wire::receiver_feedback_late_frame_ids(42, true); + require(expired.size() == 1 && expired.front() == 42, + "expired feedback did not name the late frame"); + + const auto early_wire = muninn_media_wire::encode_receiver_feedback_payload( + "muninn.raven.av.rudp", + "raven:session:video", + 42, + {"42:3"}, + false, + false, + "unix-ms:1000"); + const auto expired_wire = muninn_media_wire::encode_receiver_feedback_payload( + "muninn.raven.av.rudp", + "raven:session:video", + 42, + {"42:3"}, + true, + true, + "unix-ms:1000"); + require(!early_wire.empty() && !expired_wire.empty(), "feedback wire encoding was empty"); + require(early_wire != expired_wire, "early and expired feedback encoded identically"); + if (argc == 2 && std::string(argv[1]) == "--emit-early-hex") { + for (const auto byte : early_wire) { + std::cout << std::hex << std::setw(2) << std::setfill('0') + << static_cast(byte); + } + std::cout << '\n'; + } + return 0; +} diff --git a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp index 97d767c..fd65938 100644 --- a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp +++ b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp @@ -51,8 +51,8 @@ int main() "minimal RUDP URL did not keep the default assembly deadline"); require(fallback->audio_reorder_ms == muninn_rudp_url::DefaultAudioReorderMs, "minimal RUDP URL did not keep the default audio reorder wait"); - require(fallback->audio_reorder_ms == 2000, - "minimal RUDP URL did not use the full default audio reorder budget"); + require(fallback->audio_reorder_ms == 40, + "minimal RUDP URL did not use the bounded realtime audio reorder budget"); return 0; } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index fb74691..9f79478 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -189,3 +189,4 @@ {"ts": "2026-07-14T01:13:34+01:00", "type": "verification", "status": "ok", "note": "2026-07-14 Odin d0eded6 deployed to Nightwing Muninn (SHA-256 a604fedaf90b26a49db6594b16b8b3499e3aed540152c6fda9dd9dbe87d37d39). Post-deploy 10 s Mimir window: 555 frames, 2246 observations, 443 stereo pairs; radius P01 improved from ~0.707 px to 3.13/3.64 px and candidate confidence is explicitly uncalibrated 0.5 on both Eyes. Eye 1 honestly saw only two plausible stationary IDs. Artifact: artifacts/move-calibration/muninn-post-admission-filter-20260714.mpack."} {"ts": "2026-07-16T20:03:50+01:00", "type": "decision", "status": "ok", "note": "Moonlight common-c study at commit 82e2514 found Muninn already has media-aware packetization, deadlines, parity, receiver feedback, repair caching, and keyframe pressure. Preserve those foundations; close feedback into sender policy, replace the two-second realtime default with consumer-derived deadlines, and split Sleipnir input into replaceable latest state versus ordered non-replaceable edges. Research: docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md."} {"ts": "2026-07-16T20:29:03+01:00", "type": "verification", "status": "ok", "note": "Moonlight-derived reliability cut implemented in Odin against experimental CultLib snapshot 8965f3c0: HID epoch/latest-state/ordered-edge/application-ack semantics; wall clock demoted from ordering; Muninn A/V uses CultNet realtime lane with 100ms default LAN deadline; bounded producer/pending queues; expiring and receiver-late-aware repair cache; decode-chain-only keyframe pressure. Verification passed 168 Muninn, 25 Sleipnir, and 35 vendored experimental CultNet tests."} +{"ts": "2026-07-16T20:48:32+01:00", "type": "verification", "status": "ok", "note": "Production Muninn receiver audit fixed two live contract failures: early video repair feedback no longer places live frames in late_frame_ids (which made Odin refuse every repair), with a C++-emitted fixture decoded by Rust; OBS local audio decoder now consumes Muninn's actual f32le stereo 48kHz packets rather than AAC, default reorder is 40ms instead of 2000ms, and short gaps receive bounded silence concealment. Native plugin build and two CTests passed; Odin cross-fixture tests passed."} diff --git a/state/map.yaml b/state/map.yaml index e7dbf14..5fe6ba9 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,15 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - The 2026-07-16 production receiver audit cut two false contracts. Native + Mimir/OBS early video repair feedback no longer marks live frames late, so + Odin can actually return cached missing chunks; only assembly expiry emits + `late_frame_ids`. A C++-generated feedback fixture now decodes in Odin's + Rust tests. Muninn's typed audio body is 10 ms `pcm-f32le-interleaved`, so + the OBS bridge now opens its local decoder as `f32le` stereo 48 kHz instead + of AAC, uses a 40 ms reorder budget instead of 2000 ms, and emits bounded + silence packets for short unrecoverable gaps to preserve cadence. Opus + FEC/PLC remains required before audio reaches Moonlight-level loss quality. - Odin's 2026-07-16 Moonlight-derived reliability cut now runs Muninn and Sleipnir against the experimental CultLib runtime snapshot at `8965f3c0`. HID latest state and ordered button edges have separate epoch/sequence and From 58e5556322f93dce67f626c12db427be79be8a39 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 21:02:01 +0100 Subject: [PATCH 111/135] Recover bounded Muninn audio loss --- native/obs_stem_source/CMakeLists.txt | 7 + .../mimir_obs_muninn_source.cpp | 112 ++++++++++- native/obs_stem_source/muninn_audio_fec.h | 179 ++++++++++++++++++ native/obs_stem_source/muninn_rudp_url.h | 2 +- .../tests/muninn_audio_fec_tests.cpp | 79 ++++++++ .../tests/muninn_rudp_url_tests.cpp | 8 +- 6 files changed, 378 insertions(+), 9 deletions(-) create mode 100644 native/obs_stem_source/muninn_audio_fec.h create mode 100644 native/obs_stem_source/tests/muninn_audio_fec_tests.cpp diff --git a/native/obs_stem_source/CMakeLists.txt b/native/obs_stem_source/CMakeLists.txt index c4f0444..090a2ff 100644 --- a/native/obs_stem_source/CMakeLists.txt +++ b/native/obs_stem_source/CMakeLists.txt @@ -38,3 +38,10 @@ target_compile_features(muninn_media_wire_tests PRIVATE cxx_std_20) target_include_directories(muninn_media_wire_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) add_test(NAME muninn_media_wire_tests COMMAND muninn_media_wire_tests) + +add_executable(muninn_audio_fec_tests + tests/muninn_audio_fec_tests.cpp) +target_compile_features(muninn_audio_fec_tests PRIVATE cxx_std_20) +target_include_directories(muninn_audio_fec_tests PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}) +add_test(NAME muninn_audio_fec_tests COMMAND muninn_audio_fec_tests) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 864d4c0..3600db3 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -8,6 +8,7 @@ #include #include "muninn_media_wire.h" +#include "muninn_audio_fec.h" #include "muninn_rudp_ack.h" #include "muninn_rudp_fragments.h" #include "muninn_rudp_url.h" @@ -70,11 +71,11 @@ constexpr uint32_t MuninnCatalogDiscoveryConnectionId = 0x6d750003; constexpr uint32_t MuninnCommandRudpConnectionId = 0x6d750002; constexpr uint32_t MuninnDefaultMediaPacketBytes = 848; constexpr uint32_t MuninnDefaultRudpVideoBitrateKbps = 12000; -constexpr uint32_t MuninnDefaultRudpLatencyBudgetMs = 2000; +constexpr uint32_t MuninnDefaultRudpLatencyBudgetMs = 100; constexpr uint32_t MuninnDefaultRudpAudioReorderMs = muninn_rudp_url::DefaultAudioReorderMs; constexpr uint32_t MuninnMaxRudpVideoBitrateKbps = 100000; -constexpr uint32_t MuninnMaxRudpLatencyBudgetMs = 2000; -constexpr uint32_t MuninnMaxRudpAudioReorderMs = 2000; +constexpr uint32_t MuninnMaxRudpLatencyBudgetMs = 500; +constexpr uint32_t MuninnMaxRudpAudioReorderMs = 250; constexpr size_t MuninnAudioReorderMaxPackets = 128; constexpr uint64_t MuninnAudioConcealmentMaxPackets = 4; constexpr size_t MuninnAudioQueueMaxPackets = 32; @@ -200,7 +201,7 @@ static std::optional allocate_udp_loopback_port() } struct MuninnMediaPayload { - enum class Kind { Unknown, Video, VideoParity, Audio }; + enum class Kind { Unknown, Video, VideoParity, Audio, AudioParity }; Kind kind = Kind::Unknown; std::string stream_id; std::string session_id; @@ -211,6 +212,11 @@ struct MuninnMediaPayload { uint32_t audio_timebase_num = 0; uint32_t audio_timebase_den = 0; uint64_t audio_deadline_ticks = 0; + uint64_t audio_base_packet_id = 0; + uint16_t audio_data_shard_count = 0; + uint16_t audio_parity_index = 0; + uint16_t audio_parity_shard_count = 0; + uint32_t audio_shard_payload_bytes = 0; bool keyframe = false; std::optional dependency_frame_id; uint16_t chunk_index = 0; @@ -699,6 +705,35 @@ static std::optional decode_muninn_media_payload(const uint8 media.payload = record.read_bytes(); return media; } + if (schema_id == "muninn.media_audio_parity_shard.v1") { + if (record_len < 14) { + return std::nullopt; + } + MuninnMediaPayload media; + media.kind = MuninnMediaPayload::Kind::AudioParity; + media.stream_id = record.read_string(); + media.session_id = record.read_string(); + media.audio_base_packet_id = record.read_unsigned(); + record.skip(); // codec + record.skip(); // base_pts_ticks + record.skip(); // packet_duration_ticks + record.skip(); // timebase_num + record.skip(); // timebase_den + media.audio_deadline_ticks = record.read_unsigned(); + media.audio_data_shard_count = static_cast(record.read_unsigned()); + media.audio_parity_index = static_cast(record.read_unsigned()); + media.audio_parity_shard_count = static_cast(record.read_unsigned()); + media.audio_shard_payload_bytes = static_cast(record.read_unsigned()); + media.payload = record.read_bytes(); + if (media.audio_data_shard_count != muninn_audio_fec::DataShards || + media.audio_parity_shard_count != muninn_audio_fec::ParityShards || + media.audio_parity_index >= muninn_audio_fec::ParityShards || + media.audio_shard_payload_bytes == 0 || + media.payload.size() != media.audio_shard_payload_bytes) { + return std::nullopt; + } + return media; + } } catch (const std::exception &error) { blog(LOG_WARNING, "Muninn RUDP bridge could not decode typed media payload: %s", error.what()); } @@ -1149,6 +1184,12 @@ private: return; } forwarded_bytes = *audio_forwarded_bytes; + } else if (media->kind == MuninnMediaPayload::Kind::AudioParity) { + const auto audio_forwarded_bytes = forward_audio_parity(*media, audio_socket, audio_addr); + if (!audio_forwarded_bytes) { + return; + } + forwarded_bytes = *audio_forwarded_bytes; } else { ++payloads_decoded_dropped_; log_bridge_pressure("dropped unknown media payload"); @@ -1194,6 +1235,11 @@ private: audio_session_id_ = media.session_id; } + audio_fec_data_cache_[media.audio_packet_id] = MuninnPendingAudioPacket{media.payload, now}; + while (audio_fec_data_cache_.size() > MuninnAudioFecDataCacheMaxPackets) { + audio_fec_data_cache_.erase(audio_fec_data_cache_.begin()); + } + if (!next_audio_packet_ready_) { next_audio_packet_ready_ = true; next_audio_packet_id_ = media.audio_packet_id; @@ -1217,6 +1263,58 @@ private: return flush_audio_packets(now, audio_socket, audio_addr); } + std::optional forward_audio_parity(const MuninnMediaPayload &media, + SOCKET audio_socket, + const sockaddr_in &audio_addr) + { + if (media.stream_id.empty() || media.session_id.empty() || media.payload.empty()) { + ++payloads_decoded_dropped_; + return std::nullopt; + } + const auto now = std::chrono::steady_clock::now(); + if (audio_stream_id_ != media.stream_id || audio_session_id_ != media.session_id) { + reset_audio_reorder(); + audio_stream_id_ = media.stream_id; + audio_session_id_ = media.session_id; + } + + const muninn_audio_fec::BlockKey key{ + media.stream_id, media.session_id, media.audio_base_packet_id}; + std::vector recovered; + for (size_t index = 0; index < muninn_audio_fec::DataShards; ++index) { + const auto packet = audio_fec_data_cache_.find(media.audio_base_packet_id + index); + if (packet != audio_fec_data_cache_.end()) { + auto newly_recovered = audio_fec_receiver_.insert( + key, index, packet->second.payload, now); + recovered.insert(recovered.end(), + std::make_move_iterator(newly_recovered.begin()), + std::make_move_iterator(newly_recovered.end())); + } + } + auto parity_recovered = audio_fec_receiver_.insert( + key, + muninn_audio_fec::DataShards + media.audio_parity_index, + media.payload, + now); + recovered.insert(recovered.end(), + std::make_move_iterator(parity_recovered.begin()), + std::make_move_iterator(parity_recovered.end())); + + for (auto &packet : recovered) { + if (!next_audio_packet_ready_) { + next_audio_packet_ready_ = true; + next_audio_packet_id_ = packet.packet_id; + } + if (packet.packet_id >= next_audio_packet_id_ && + pending_audio_packets_.emplace( + packet.packet_id, + MuninnPendingAudioPacket{std::move(packet.payload), now}).second) { + ++audio_packets_fec_recovered_; + } + } + return flush_audio_packets(now, audio_socket, audio_addr); + } + std::optional flush_audio_packets(std::chrono::steady_clock::time_point now, SOCKET audio_socket, const sockaddr_in &audio_addr) @@ -1302,6 +1400,8 @@ private: audio_stream_id_.clear(); audio_session_id_.clear(); pending_audio_packets_.clear(); + audio_fec_data_cache_.clear(); + audio_fec_receiver_.clear(); next_audio_packet_ready_ = false; next_audio_packet_id_ = 0; audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); @@ -1942,12 +2042,16 @@ private: uint64_t audio_packets_skipped_ = 0; uint64_t audio_packets_stale_ = 0; uint64_t audio_packets_concealed_ = 0; + uint64_t audio_packets_fec_recovered_ = 0; std::string audio_stream_id_; std::string audio_session_id_; bool next_audio_packet_ready_ = false; uint64_t next_audio_packet_id_ = 0; std::chrono::steady_clock::time_point audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); std::map pending_audio_packets_; + static constexpr size_t MuninnAudioFecDataCacheMaxPackets = 512; + std::map audio_fec_data_cache_; + muninn_audio_fec::BlockReceiver audio_fec_receiver_{std::chrono::milliseconds(40), 128}; bool media_sequence_initialized_ = false; uint32_t last_media_sequence_ = 0; uint64_t media_sequence_gap_events_ = 0; diff --git a/native/obs_stem_source/muninn_audio_fec.h b/native/obs_stem_source/muninn_audio_fec.h new file mode 100644 index 0000000..e5ca7d9 --- /dev/null +++ b/native/obs_stem_source/muninn_audio_fec.h @@ -0,0 +1,179 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace muninn_audio_fec { + +inline constexpr size_t DataShards = 4; +inline constexpr size_t ParityShards = 2; +inline constexpr size_t TotalShards = DataShards + ParityShards; +using Bytes = std::vector; +using Shards = std::array, TotalShards>; + +inline uint8_t multiply(uint8_t left, uint8_t right) +{ + uint16_t product = 0; + uint16_t a = left; + uint16_t b = right; + while (b != 0) { + if ((b & 1u) != 0) product ^= a; + b >>= 1u; + a <<= 1u; + if ((a & 0x100u) != 0) a ^= 0x11du; + } + return static_cast(product); +} + +inline uint8_t power(uint8_t value, uint16_t exponent) +{ + uint8_t result = 1; + while (exponent != 0) { + if ((exponent & 1u) != 0) result = multiply(result, value); + value = multiply(value, value); + exponent >>= 1u; + } + return result; +} + +inline uint8_t inverse(uint8_t value) +{ + return value == 0 ? 0 : power(value, 254); +} + +inline constexpr std::array, TotalShards> Generator{{ + {{1, 0, 0, 0}}, {{0, 1, 0, 0}}, {{0, 0, 1, 0}}, {{0, 0, 0, 1}}, + {{1, 1, 1, 1}}, {{1, 2, 4, 8}} +}}; + +inline std::array encode(const std::array &data) +{ + const size_t bytes = data[0].size(); + for (const auto &shard : data) if (shard.size() != bytes) return {}; + std::array parity{Bytes(bytes), Bytes(bytes)}; + for (size_t p = 0; p < ParityShards; ++p) + for (size_t d = 0; d < DataShards; ++d) + for (size_t i = 0; i < bytes; ++i) + parity[p][i] ^= multiply(Generator[DataShards + p][d], data[d][i]); + return parity; +} + +inline bool recover(Shards &shards) +{ + size_t shard_bytes = 0; + std::array available{}; + size_t count = 0; + for (size_t i = 0; i < TotalShards; ++i) { + if (!shards[i]) continue; + if (shard_bytes == 0) shard_bytes = shards[i]->size(); + if (shards[i]->size() != shard_bytes) return false; + if (count < DataShards) available[count++] = i; + } + if (count < DataShards || shard_bytes == 0) return false; + + std::array, DataShards> matrix{}; + for (size_t row = 0; row < DataShards; ++row) { + for (size_t col = 0; col < DataShards; ++col) matrix[row][col] = Generator[available[row]][col]; + matrix[row][DataShards + row] = 1; + } + for (size_t col = 0; col < DataShards; ++col) { + size_t pivot = col; + while (pivot < DataShards && matrix[pivot][col] == 0) ++pivot; + if (pivot == DataShards) return false; + if (pivot != col) std::swap(matrix[pivot], matrix[col]); + const uint8_t scale = inverse(matrix[col][col]); + for (size_t j = 0; j < DataShards * 2; ++j) matrix[col][j] = multiply(matrix[col][j], scale); + for (size_t row = 0; row < DataShards; ++row) { + if (row == col) continue; + const uint8_t factor = matrix[row][col]; + for (size_t j = 0; j < DataShards * 2; ++j) + matrix[row][j] ^= multiply(factor, matrix[col][j]); + } + } + for (size_t missing = 0; missing < DataShards; ++missing) { + if (shards[missing]) continue; + Bytes decoded(shard_bytes); + for (size_t row = 0; row < DataShards; ++row) { + const uint8_t coefficient = matrix[missing][DataShards + row]; + for (size_t i = 0; i < shard_bytes; ++i) + decoded[i] ^= multiply(coefficient, (*shards[available[row]])[i]); + } + shards[missing] = std::move(decoded); + } + return true; +} + +struct BlockKey { + std::string stream_id; + std::string session_id; + uint64_t base_packet_id = 0; + bool operator<(const BlockKey &other) const + { + return std::tie(stream_id, session_id, base_packet_id) < + std::tie(other.stream_id, other.session_id, other.base_packet_id); + } +}; + +struct RecoveredPacket { uint64_t packet_id; Bytes payload; }; + +class BlockReceiver { +public: + using Clock = std::chrono::steady_clock; + explicit BlockReceiver(std::chrono::milliseconds deadline, size_t max_blocks = 128) + : deadline_(deadline), max_blocks_(max_blocks) {} + + std::vector insert(const BlockKey &key, size_t shard_index, Bytes payload, + Clock::time_point now) + { + expire(now); + if (shard_index >= TotalShards || payload.empty() || completed_.count(key) != 0) return {}; + if (blocks_.size() >= max_blocks_ && blocks_.count(key) == 0) blocks_.erase(blocks_.begin()); + auto [it, created] = blocks_.try_emplace(key, Block{now, payload.size(), {}}); + Block &block = it->second; + if (payload.size() != block.shard_bytes || block.shards[shard_index]) return {}; + block.shards[shard_index] = std::move(payload); + size_t present = 0; + for (const auto &shard : block.shards) if (shard) ++present; + if (present < DataShards) return {}; + Shards before = block.shards; + if (!recover(block.shards)) return {}; + std::vector result; + for (size_t i = 0; i < DataShards; ++i) + if (!before[i] && block.shards[i]) result.push_back({key.base_packet_id + i, *block.shards[i]}); + blocks_.erase(it); + completed_[key] = now; + return result; + } + + void expire(Clock::time_point now) + { + for (auto it = blocks_.begin(); it != blocks_.end();) + it = now - it->second.created_at >= deadline_ ? blocks_.erase(it) : std::next(it); + for (auto it = completed_.begin(); it != completed_.end();) + it = now - it->second >= deadline_ ? completed_.erase(it) : std::next(it); + } + size_t pending_blocks() const { return blocks_.size(); } + void clear() + { + blocks_.clear(); + completed_.clear(); + } + +private: + struct Block { Clock::time_point created_at; size_t shard_bytes; Shards shards; }; + std::chrono::milliseconds deadline_; + size_t max_blocks_; + std::map blocks_; + std::map completed_; +}; + +} // namespace muninn_audio_fec diff --git a/native/obs_stem_source/muninn_rudp_url.h b/native/obs_stem_source/muninn_rudp_url.h index a7f45c5..ffbfbd2 100644 --- a/native/obs_stem_source/muninn_rudp_url.h +++ b/native/obs_stem_source/muninn_rudp_url.h @@ -15,7 +15,7 @@ inline constexpr uint32_t DefaultVideoAssemblyDeadlineMs = 75; inline constexpr uint32_t DefaultGapWaitMs = 8; inline constexpr uint32_t DefaultAudioReorderMs = 40; inline constexpr uint32_t DefaultSenderResendDelayMs = 5; -inline constexpr uint32_t MaxLatencyBudgetMs = 2000; +inline constexpr uint32_t MaxLatencyBudgetMs = 500; struct RudpUrlParts { uint16_t port = 0; diff --git a/native/obs_stem_source/tests/muninn_audio_fec_tests.cpp b/native/obs_stem_source/tests/muninn_audio_fec_tests.cpp new file mode 100644 index 0000000..bbefab7 --- /dev/null +++ b/native/obs_stem_source/tests/muninn_audio_fec_tests.cpp @@ -0,0 +1,79 @@ +#include "muninn_audio_fec.h" + +#include +#include +#include + +static void require(bool condition, const char *message) { if (!condition) throw std::runtime_error(message); } + +static std::array fixture() +{ + std::array data; + for (size_t shard = 0; shard < data.size(); ++shard) { + data[shard].resize(257); + for (size_t i = 0; i < data[shard].size(); ++i) data[shard][i] = uint8_t((shard * 71 + i * 29) & 0xff); + } + return data; +} + +static muninn_audio_fec::Shards all_shards(const std::array &data) +{ + auto parity = muninn_audio_fec::encode(data); + return {data[0], data[1], data[2], data[3], parity[0], parity[1]}; +} + +int main() +{ + try { + const auto data = fixture(); + for (size_t erased = 0; erased < 6; ++erased) { + auto shards = all_shards(data); shards[erased].reset(); + require(muninn_audio_fec::recover(shards), "one-erasure recovery failed"); + for (size_t i = 0; i < 4; ++i) require(shards[i] == data[i], "one-erasure data mismatch"); + } + for (size_t first = 0; first < 6; ++first) for (size_t second = first + 1; second < 6; ++second) { + auto shards = all_shards(data); shards[first].reset(); shards[second].reset(); + require(muninn_audio_fec::recover(shards), "two-erasure recovery failed"); + for (size_t i = 0; i < 4; ++i) require(shards[i] == data[i], "two-erasure data mismatch"); + } + { + auto shards = all_shards(data); + shards[0].reset(); shards[1].reset(); shards[2].reset(); + require(!muninn_audio_fec::recover(shards), "three erasures were incorrectly recoverable"); + } + { + auto shards = all_shards(data); + shards[5]->pop_back(); + require(!muninn_audio_fec::recover(shards), "unequal shard sizes were accepted"); + } + + using namespace muninn_audio_fec; + BlockReceiver receiver(std::chrono::milliseconds(10), 2); + BlockKey key{"stream", "session", 100}; + auto shards = all_shards(data); + auto t0 = BlockReceiver::Clock::time_point{}; + require(receiver.insert(key, 5, *shards[5], t0).empty(), "early parity emitted data"); + require(receiver.insert(key, 2, *shards[2], t0).empty(), "partial reorder emitted data"); + require(receiver.insert(key, 2, *shards[2], t0).empty(), "duplicate emitted data"); + require(receiver.insert(key, 1, *shards[1], t0).empty(), "partial block emitted data"); + auto recovered = receiver.insert(key, 4, *shards[4], t0); + require(recovered.size() == 2 && recovered[0].packet_id == 100 && recovered[1].packet_id == 103, + "reordered block did not recover missing packet ids"); + require(recovered[0].payload == data[0] && recovered[1].payload == data[3], "receiver recovery mismatch"); + require(receiver.insert(key, 0, data[0], t0).empty(), "completed-block duplicate was admitted"); + + BlockKey expired{"stream", "session", 200}; + receiver.insert(expired, 0, data[0], t0); + receiver.expire(t0 + std::chrono::milliseconds(10)); + require(receiver.pending_blocks() == 0, "expired block survived deadline"); + + BlockReceiver bounded(std::chrono::seconds(1), 2); + bounded.insert({"s", "x", 0}, 0, data[0], t0); + bounded.insert({"s", "x", 4}, 0, data[0], t0); + bounded.insert({"s", "x", 8}, 0, data[0], t0); + require(bounded.pending_blocks() == 2, "block bound was not enforced"); + } catch (const std::exception &error) { + std::cerr << error.what() << '\n'; return EXIT_FAILURE; + } + return EXIT_SUCCESS; +} diff --git a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp index fd65938..2cdebd0 100644 --- a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp +++ b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp @@ -19,17 +19,17 @@ void require(bool condition, const std::string &message) int main() { const auto configured = muninn_rudp_url::parse( - "rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=1200&assembly_deadline_ms=1200&gap_wait_ms=16&audio_reorder_ms=700"); + "rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=120&assembly_deadline_ms=120&gap_wait_ms=16&audio_reorder_ms=80"); require(configured.has_value(), "configured RUDP URL did not parse"); require(configured->port == 5204, "configured RUDP URL port was not parsed"); require(configured->video_local_port == 15204, "configured RUDP video local port was not derived"); require(configured->audio_local_port == 15205, "configured RUDP audio local port was not derived"); require(configured->connection_id == 0x6d750001, "configured RUDP connection id was not parsed"); require(configured->audio_connection_id == 0x6d750004, "configured RUDP audio connection id was not parsed"); - require(configured->reliable_expire_after_ms == 1200, "configured reliable expiry did not survive parse"); - require(configured->video_assembly_deadline_ms == 1200, "configured assembly deadline did not survive parse"); + require(configured->reliable_expire_after_ms == 120, "configured reliable expiry did not survive parse"); + require(configured->video_assembly_deadline_ms == 120, "configured assembly deadline did not survive parse"); require(configured->gap_wait_ms == 16, "configured gap wait did not survive parse"); - require(configured->audio_reorder_ms == 700, "configured audio reorder wait did not survive parse"); + require(configured->audio_reorder_ms == 80, "configured audio reorder wait did not survive parse"); const auto clamped = muninn_rudp_url::parse( "rudp://192.168.1.66:5204/muninn.raven.av.rudp?reliable_expire_after_ms=9000&assembly_deadline_ms=9000"); From 6a7aaea0190bb854256816b87b16a97fe979420b Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 21:04:52 +0100 Subject: [PATCH 112/135] Record Muninn loss recovery authority --- docs/implementation-plan.md | 16 ++++++++++++++-- state/evidence.jsonl | 1 + state/map.yaml | 18 ++++++++++++++++++ 3 files changed, 33 insertions(+), 2 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index d469dea..b0c37f7 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -456,13 +456,25 @@ encoder keyframe actuation, production parity recovery, audio FEC/concealment, bounded long-disconnect edge admission, adaptive sender response, and the fault-injected socket timeline harness. +The socket harness and bounded PCM loss recovery have now landed. Odin's +`cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/ +duplicate/jitter/stall profiles around real CultNet endpoints. Muninn emits a +typed fixed 4+2 audio parity block after each four constant-size PCM packets; +the production Mimir OBS receiver reconstructs up to two missing packets inside +the 40 ms reorder budget and feeds them through the existing playout owner. +Both sides use the experimental CultLib/CultMesh snapshot lineage used by Eve, +Aetheria, and VoidBot; no stable-branch transport shim was introduced. + The production receiver follow-up fixed two contract contradictions: early video repair no longer declares the frame late (which had made Odin discard all repair requests), and the OBS audio decoder now consumes Muninn's actual float PCM contract rather than treating it as AAC. Audio reorder is bounded to 40 ms with short silence concealment. The next audio cut is Opus with explicit -FEC/PLC; the next video cut is a controllable encoder owner that can force IDR -without restarting the A/V session. +FEC/PLC for variable-rate compressed audio; PCM now has bounded 4+2 erasure +recovery plus concealment. The next video cut is a controllable encoder owner +that can force IDR without restarting the video session. Receiver keyframe +pressure is deliberately not allowed to restart the whole A/V stream and hide +the missing authority. 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 9f79478..5289f5f 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -190,3 +190,4 @@ {"ts": "2026-07-16T20:03:50+01:00", "type": "decision", "status": "ok", "note": "Moonlight common-c study at commit 82e2514 found Muninn already has media-aware packetization, deadlines, parity, receiver feedback, repair caching, and keyframe pressure. Preserve those foundations; close feedback into sender policy, replace the two-second realtime default with consumer-derived deadlines, and split Sleipnir input into replaceable latest state versus ordered non-replaceable edges. Research: docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md."} {"ts": "2026-07-16T20:29:03+01:00", "type": "verification", "status": "ok", "note": "Moonlight-derived reliability cut implemented in Odin against experimental CultLib snapshot 8965f3c0: HID epoch/latest-state/ordered-edge/application-ack semantics; wall clock demoted from ordering; Muninn A/V uses CultNet realtime lane with 100ms default LAN deadline; bounded producer/pending queues; expiring and receiver-late-aware repair cache; decode-chain-only keyframe pressure. Verification passed 168 Muninn, 25 Sleipnir, and 35 vendored experimental CultNet tests."} {"ts": "2026-07-16T20:48:32+01:00", "type": "verification", "status": "ok", "note": "Production Muninn receiver audit fixed two live contract failures: early video repair feedback no longer places live frames in late_frame_ids (which made Odin refuse every repair), with a C++-emitted fixture decoded by Rust; OBS local audio decoder now consumes Muninn's actual f32le stereo 48kHz packets rather than AAC, default reorder is 40ms instead of 2000ms, and short gaps receive bounded silence concealment. Native plugin build and two CTests passed; Odin cross-fixture tests passed."} +{"ts": "2026-07-16T21:03:13+01:00", "type": "verification", "status": "ok", "note": "Muninn production audio recovery now uses typed 4+2 GF(256) PCM parity from Odin and a bounded Mimir receiver cache feeding the existing 40 ms reorder owner; native OBS DLL builds and 3/3 CTests pass. Odin cultnet-impair adds deterministic real-socket loss/burst/reorder/duplicate/jitter/stall profiles. Experimental CultLib snapshot lineage remains pinned at 8965f3c0."} diff --git a/state/map.yaml b/state/map.yaml index 5fe6ba9..6e7de6b 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,24 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - The 2026-07-16 loss-recovery cut is now live on both sides of the Muninn + audio boundary. Odin emits typed `muninn.media_audio_parity_shard.v1` + records using fixed 4+2 GF(256) parity for constant-size 10 ms PCM packets. + Mimir's production OBS receiver retains a bounded recent data window, + reconstructs up to two erased packets per block before the 40 ms playout + deadline, and then hands recovered packets to the same reorder queue that + owns normal playout. The FEC cache is not a second timeline. Native tests + exhaust every one- and two-erasure permutation, and the production plugin + DLL builds with all three native suites passing. The OBS input ceiling is + now 500 ms and the default remains 100 ms; old 1.2-2.0 second test/config + assumptions are not realtime authority. + - Odin now carries a deterministic bidirectional UDP impairment proxy with + seeded clean, iid loss, burst loss, delay/jitter, reorder, duplicate, and + stall profiles. This is the acceptance substrate for real CultNet endpoints, + not a simulated transport replacement. The remaining critical sender gap + is an encoder owner that can actuate a receiver-requested next IDR without + restarting the video session; stock FFmpeg CLI capture cannot own that + command dynamically. - The 2026-07-16 production receiver audit cut two false contracts. Native Mimir/OBS early video repair feedback no longer marks live frames late, so Odin can actually return cached missing chunks; only assembly expiry emits From d3d461be93fce5bf73bec97a82d243dc11c329f0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 21:26:00 +0100 Subject: [PATCH 113/135] Record receiver-driven IDR proof --- docs/implementation-plan.md | 11 +++++++---- state/evidence.jsonl | 1 + state/map.yaml | 15 +++++++++++++++ 3 files changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index b0c37f7..caba341 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -471,10 +471,13 @@ repair requests), and the OBS audio decoder now consumes Muninn's actual float PCM contract rather than treating it as AAC. Audio reorder is bounded to 40 ms with short silence concealment. The next audio cut is Opus with explicit FEC/PLC for variable-rate compressed audio; PCM now has bounded 4+2 erasure -recovery plus concealment. The next video cut is a controllable encoder owner -that can force IDR without restarting the video session. Receiver keyframe -pressure is deliberately not allowed to restart the whole A/V stream and hide -the missing authority. +recovery plus concealment. The controllable video encoder owner now exists and +forces the next NVENC frame to IDR without restarting the video session. A live +D3D11 desktop proof verified the command-generated IDR. The remaining field +cut is to install its bundle on Raven, route the existing activation task +through `--video-encoder`, and run the named impairment profiles. Raven's +WireGuard SSH route timed out during the first deployment attempt, so no +interactive workstation task was disturbed. 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 5289f5f..df4f7c9 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -191,3 +191,4 @@ {"ts": "2026-07-16T20:29:03+01:00", "type": "verification", "status": "ok", "note": "Moonlight-derived reliability cut implemented in Odin against experimental CultLib snapshot 8965f3c0: HID epoch/latest-state/ordered-edge/application-ack semantics; wall clock demoted from ordering; Muninn A/V uses CultNet realtime lane with 100ms default LAN deadline; bounded producer/pending queues; expiring and receiver-late-aware repair cache; decode-chain-only keyframe pressure. Verification passed 168 Muninn, 25 Sleipnir, and 35 vendored experimental CultNet tests."} {"ts": "2026-07-16T20:48:32+01:00", "type": "verification", "status": "ok", "note": "Production Muninn receiver audit fixed two live contract failures: early video repair feedback no longer places live frames in late_frame_ids (which made Odin refuse every repair), with a C++-emitted fixture decoded by Rust; OBS local audio decoder now consumes Muninn's actual f32le stereo 48kHz packets rather than AAC, default reorder is 40ms instead of 2000ms, and short gaps receive bounded silence concealment. Native plugin build and two CTests passed; Odin cross-fixture tests passed."} {"ts": "2026-07-16T21:03:13+01:00", "type": "verification", "status": "ok", "note": "Muninn production audio recovery now uses typed 4+2 GF(256) PCM parity from Odin and a bounded Mimir receiver cache feeding the existing 40 ms reorder owner; native OBS DLL builds and 3/3 CTests pass. Odin cultnet-impair adds deterministic real-socket loss/burst/reorder/duplicate/jitter/stall profiles. Experimental CultLib snapshot lineage remains pinned at 8965f3c0."} +{"ts": "2026-07-16T21:26:00+01:00", "type": "verification", "status": "ok", "note": "Receiver decode-chain invalidation now actuates a verified next-frame NVENC IDR through Odin's long-lived libav encoder without restarting A/V or CultNet. Live 2560x1440 D3D11 proof emitted a command-driven IDR at frame 22 under a 600-frame GOP; synthetic proof emitted IDRs at frames 0 and 10 and decoded cleanly. Muninn/Sleipnir edge admission is bounded at 256 via epoch rebase; 174/26 tests pass. Raven field deployment remains pending because 10.77.0.4 SSH and TCP probing timed out."} diff --git a/state/map.yaml b/state/map.yaml index 6e7de6b..226332a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,21 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - Receiver-driven video recovery now reaches the actual NVENC frame owner. + Odin's `native/muninn-video-encoder` is a long-lived libavdevice desktop + capture and `h264_nvenc` process; a new decode-chain invalidation edge makes + Muninn write `IDR` to its stdin without restarting video, audio, or CultNet. + A live 2560x1440 D3D11 proof with a deliberately 600-frame GOP produced the + requested second IDR at frame 22, and its Annex-B stream decoded cleanly. + The normal quarter-second scheduled IDR remains independent fallback. + The repeatable bundle includes the OBS experimental FFmpeg dependency + snapshot. Raven installation and real impairment acceptance remain pending + because Raven's documented WireGuard SSH route timed out on 2026-07-16. + - Muninn input now bounds disconnected unacknowledged button history at 256 + edges. Overflow rotates the controller epoch and rebases Sleipnir from the + next complete latest-state frame, rather than dropping an arbitrary press + or retaining history forever. Sleipnir rejects sparse future edge gaps over + 256. Full Muninn (174) and Sleipnir (26) suites pass. - The 2026-07-16 loss-recovery cut is now live on both sides of the Muninn audio boundary. Odin emits typed `muninn.media_audio_parity_shard.v1` records using fixed 4+2 GF(256) parity for constant-size 10 ms PCM packets. From b0f33c36b44287aa97f5079cbf487364de39802e Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 21:35:47 +0100 Subject: [PATCH 114/135] Define Moonlight reliability acceptance gate --- docs/implementation-plan.md | 9 ++++ ...light-reliability-acceptance-2026-07-16.md | 47 +++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 9 ++++ 4 files changed, 66 insertions(+) create mode 100644 docs/research/moonlight-reliability-acceptance-2026-07-16.md diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index caba341..cbfe454 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -465,6 +465,15 @@ the 40 ms reorder budget and feeds them through the existing playout owner. Both sides use the experimental CultLib/CultMesh snapshot lineage used by Eve, Aetheria, and VoidBot; no stable-branch transport shim was introduced. +Receiver pressure now closes into the long-lived NVENC encoder as bounded AIMD +bitrate control. Late/decode/repair/queue pressure backs off by 15 percent; +clean recovery adds five percent of the configured ceiling only after two +stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a +restart and emitted the required transition IDR. The exact remaining completion +gate is tracked in +`docs/research/moonlight-reliability-acceptance-2026-07-16.md`; no cross-host +claim is allowed until its Raven-to-Starfire field matrix passes. + The production receiver follow-up fixed two contract contradictions: early video repair no longer declares the frame late (which had made Odin discard all repair requests), and the OBS audio decoder now consumes Muninn's actual float diff --git a/docs/research/moonlight-reliability-acceptance-2026-07-16.md b/docs/research/moonlight-reliability-acceptance-2026-07-16.md new file mode 100644 index 0000000..6b396a9 --- /dev/null +++ b/docs/research/moonlight-reliability-acceptance-2026-07-16.md @@ -0,0 +1,47 @@ +# Moonlight-level reliability acceptance ledger + +This ledger distinguishes implemented mechanisms from evidence that the actual +Raven-to-Starfire program timeline survives pressure. Passing unit tests is not +permission to call the field path done. + +## Required contracts + +| Contract | Current authority | Evidence | Status | +|---|---|---|---| +| Experimental CultMesh/CultNet lineage | Odin vendored CultLib snapshot `8965f3c0` | Snapshot provenance plus CultNet/Muninn/Sleipnir suites | proved locally | +| Low-delay NVENC | Muninn controllable encoder | D3D11 desktop capture, no B frames/lookahead, two-frame VBV, clean Annex-B decode | proved locally | +| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default, bounded queues/cache, late-frame repair refusal | proved structurally | +| Recoverable video loss avoids decoder reset | Mimir parity/repair, Odin repair cache | Production XOR parity; live early-feedback fixture; repair expiry tests | proved locally | +| Decoder-chain loss forces immediate IDR | Muninn feedback owner and native encoder | Live 2560x1440 command produced IDR at frame 22 under a 600-frame GOP | proved locally | +| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Live 12-to-6 Mbps reconfiguration produced clean IDR at frame 35 | proved locally | +| Audio survives small erasure bursts | Odin 4+2 GF(256), Mimir FEC/reorder/concealment | Exhaustive one/two-erasure Rust and C++ tests; production DLL build | proved locally | +| Latest input state supersedes stale state | Muninn epoch/state sequence, Sleipnir cursor | Duplicate/reorder/stale epoch tests | proved locally | +| Button edges are not lost or retained forever | Muninn replay/app ACK/epoch rebase, Sleipnir ordered cursor | Quick-tap, ACK, overflow rebase, sparse-gap tests | proved locally | +| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | No current run: Raven WireGuard SSH/TCP 22 timed out | **missing** | + +## Field acceptance matrix + +Run the real capture body, production receiver plugin, decoder, audio output, +and Sleipnir virtual input through the seeded profiles under +`E:\Projects\Odin\tests\realtime-impairment`. Observe the rendered/audio/input +timeline, not only proxy CSV counters. + +| Profile | Required result | +|---|---| +| clean, 10 minutes | no decoder reset, audio discontinuity, input edge loss, queue growth, or timestamp drift; video p99 delivery under 50 ms | +| iid 1%, 10 minutes | no input edge loss; audio FEC recovers eligible holes; at least 99.9% video frames presented; media p99 under 80 ms | +| iid 3%, 5 minutes | bounded degradation; at least 99% video frames presented; decoder recovery within 250 ms; no stuck controls | +| burst 4 and burst 8 | parity/repair handles eligible damage; unrecoverable dependency loss produces one requested IDR and recovery within 250 ms | +| reorder and duplicate | no duplicate input action, stale state regression, or unbounded assembly/reorder storage | +| delay/jitter | AIMD reduces bitrate before sustained deadline loss and recovers no faster than one step per two stable seconds | +| stall 250 ms | obsolete media expires; queues remain bounded; input latest state converges within 35 ms after release | +| reconnect | old input epoch cannot act; neutral/current state within 100 ms; decoded video within 500 ms | +| mixed 60-minute soak | stable memory/handle/thread counts and no progressive A/V drift | + +## Completion rule + +Moonlight-level performance is reasonably expected only after all local rows +remain green and the real field matrix passes twice: once on direct LAN and once +on the deployed route actually used during streaming. Failure should change the +owning deadline, FEC, encoder, or input authority—not add a repair loop around +the symptom. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index df4f7c9..50d61d1 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -192,3 +192,4 @@ {"ts": "2026-07-16T20:48:32+01:00", "type": "verification", "status": "ok", "note": "Production Muninn receiver audit fixed two live contract failures: early video repair feedback no longer places live frames in late_frame_ids (which made Odin refuse every repair), with a C++-emitted fixture decoded by Rust; OBS local audio decoder now consumes Muninn's actual f32le stereo 48kHz packets rather than AAC, default reorder is 40ms instead of 2000ms, and short gaps receive bounded silence concealment. Native plugin build and two CTests passed; Odin cross-fixture tests passed."} {"ts": "2026-07-16T21:03:13+01:00", "type": "verification", "status": "ok", "note": "Muninn production audio recovery now uses typed 4+2 GF(256) PCM parity from Odin and a bounded Mimir receiver cache feeding the existing 40 ms reorder owner; native OBS DLL builds and 3/3 CTests pass. Odin cultnet-impair adds deterministic real-socket loss/burst/reorder/duplicate/jitter/stall profiles. Experimental CultLib snapshot lineage remains pinned at 8965f3c0."} {"ts": "2026-07-16T21:26:00+01:00", "type": "verification", "status": "ok", "note": "Receiver decode-chain invalidation now actuates a verified next-frame NVENC IDR through Odin's long-lived libav encoder without restarting A/V or CultNet. Live 2560x1440 D3D11 proof emitted a command-driven IDR at frame 22 under a 600-frame GOP; synthetic proof emitted IDRs at frames 0 and 10 and decoded cleanly. Muninn/Sleipnir edge admission is bounded at 256 via epoch rebase; 174/26 tests pass. Raven field deployment remains pending because 10.77.0.4 SSH and TCP probing timed out."} +{"ts": "2026-07-16T21:35:47+01:00", "type": "verification", "status": "ok", "note": "Muninn receiver pressure now drives bounded AIMD bitrate on the long-lived NVENC context. Live 2560x1440 D3D11 proof reconfigured 12 Mbps to 6 Mbps at frame 35, emitted an IDR at frame 35, and decoded cleanly; 176 Muninn tests pass. Cross-host field acceptance remains missing because Raven SSH/TCP 22 timed out again."} diff --git a/state/map.yaml b/state/map.yaml index 226332a..041cbc2 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,15 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - Muninn now closes receiver congestion pressure into the live NVENC context. + Every 500 ms it applies multiplicative 15 percent backoff for late frames, + requested recovery, deferred repair, sender queue loss, or combined + jitter/decode pressure above 75 percent of the consumer deadline. Recovery + adds only five percent of the configured ceiling after each two stable + seconds, with a one-quarter/1 Mbps floor. A live 2560x1440 D3D11 proof + changed 12 Mbps to 6 Mbps at frame 35; FFmpeg/NVENC reconfigured the same + session, emitted an IDR at frame 35, and the Annex-B stream decoded cleanly. + Muninn's full suite now passes 176 tests. - Receiver-driven video recovery now reaches the actual NVENC frame owner. Odin's `native/muninn-video-encoder` is a long-lived libavdevice desktop capture and `h264_nvenc` process; a new decode-chain invalidation edge makes From 60cbe4562422f3bd86b62cbc56526ec0afda6615 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 21:38:24 +0100 Subject: [PATCH 115/135] Record Raven field-test outage --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 50d61d1..ffda9f8 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -193,3 +193,4 @@ {"ts": "2026-07-16T21:03:13+01:00", "type": "verification", "status": "ok", "note": "Muninn production audio recovery now uses typed 4+2 GF(256) PCM parity from Odin and a bounded Mimir receiver cache feeding the existing 40 ms reorder owner; native OBS DLL builds and 3/3 CTests pass. Odin cultnet-impair adds deterministic real-socket loss/burst/reorder/duplicate/jitter/stall profiles. Experimental CultLib snapshot lineage remains pinned at 8965f3c0."} {"ts": "2026-07-16T21:26:00+01:00", "type": "verification", "status": "ok", "note": "Receiver decode-chain invalidation now actuates a verified next-frame NVENC IDR through Odin's long-lived libav encoder without restarting A/V or CultNet. Live 2560x1440 D3D11 proof emitted a command-driven IDR at frame 22 under a 600-frame GOP; synthetic proof emitted IDRs at frames 0 and 10 and decoded cleanly. Muninn/Sleipnir edge admission is bounded at 256 via epoch rebase; 174/26 tests pass. Raven field deployment remains pending because 10.77.0.4 SSH and TCP probing timed out."} {"ts": "2026-07-16T21:35:47+01:00", "type": "verification", "status": "ok", "note": "Muninn receiver pressure now drives bounded AIMD bitrate on the long-lived NVENC context. Live 2560x1440 D3D11 proof reconfigured 12 Mbps to 6 Mbps at frame 35, emitted an IDR at frame 35, and decoded cleanly; 176 Muninn tests pass. Cross-host field acceptance remains missing because Raven SSH/TCP 22 timed out again."} +{"ts": "2026-07-16T21:38:23+01:00", "type": "blocker", "status": "pending", "note": "Field Moonlight acceptance cannot run while Raven is offline: Starfire's documented SSH/ICMP/TCP route to 10.77.0.4 times out, and Yggdrasil reports Raven's last WireGuard handshake at 2026-07-15T06:04:12Z versus Starfire's live handshake at 2026-07-16T20:36:32Z. No Raven task or host state was mutated."} From caf92d9dee20771eb07ccefa629bb4ca4c4f3f12 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 16 Jul 2026 23:10:56 +0100 Subject: [PATCH 116/135] Make Muninn OBS receiver reload-safe --- native/obs_stem_source/CMakeLists.txt | 12 ++++ .../mimir_obs_muninn_source.cpp | 63 +++++++++++++++---- native/obs_stem_source/muninn_rudp_url.h | 5 +- native/obs_stem_source/muninn_udp_socket.h | 32 ++++++++++ .../tests/muninn_rudp_url_tests.cpp | 4 +- .../tests/muninn_udp_socket_tests.cpp | 34 ++++++++++ 6 files changed, 135 insertions(+), 15 deletions(-) create mode 100644 native/obs_stem_source/muninn_udp_socket.h create mode 100644 native/obs_stem_source/tests/muninn_udp_socket_tests.cpp diff --git a/native/obs_stem_source/CMakeLists.txt b/native/obs_stem_source/CMakeLists.txt index 090a2ff..527739d 100644 --- a/native/obs_stem_source/CMakeLists.txt +++ b/native/obs_stem_source/CMakeLists.txt @@ -45,3 +45,15 @@ target_compile_features(muninn_audio_fec_tests PRIVATE cxx_std_20) target_include_directories(muninn_audio_fec_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) add_test(NAME muninn_audio_fec_tests COMMAND muninn_audio_fec_tests) + +add_executable(muninn_udp_socket_tests + tests/muninn_udp_socket_tests.cpp) +target_compile_features(muninn_udp_socket_tests PRIVATE cxx_std_20) +target_compile_definitions(muninn_udp_socket_tests PRIVATE + WIN32_LEAN_AND_MEAN + NOMINMAX + _WIN32_WINNT=0x0A00) +target_include_directories(muninn_udp_socket_tests PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}) +target_link_libraries(muninn_udp_socket_tests PRIVATE ws2_32) +add_test(NAME muninn_udp_socket_tests COMMAND muninn_udp_socket_tests) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 3600db3..9910999 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -12,6 +12,7 @@ #include "muninn_rudp_ack.h" #include "muninn_rudp_fragments.h" #include "muninn_rudp_url.h" +#include "muninn_udp_socket.h" #include #include @@ -171,7 +172,7 @@ struct RudpBridgeOutputs { static std::optional allocate_udp_loopback_port() { #ifdef _WIN32 - SOCKET socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + SOCKET socket = muninn_udp_socket::create(); if (socket == INVALID_SOCKET) { return std::nullopt; } @@ -870,22 +871,24 @@ private: void run() { #ifdef _WIN32 + SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_HIGHEST); WSADATA data; WSAStartup(MAKEWORD(2, 2), &data); - socket_ = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + socket_ = muninn_udp_socket::create(true); if (socket_ == INVALID_SOCKET) { blog(LOG_WARNING, "Muninn RUDP bridge could not create socket"); running_ = false; WSACleanup(); return; } - const int socket_buffer_bytes = 4 * 1024 * 1024; - const BOOL reuse_addr = TRUE; - setsockopt(socket_, SOL_SOCKET, SO_REUSEADDR, reinterpret_cast(&reuse_addr), - sizeof(reuse_addr)); + const int socket_buffer_bytes = 16 * 1024 * 1024; setsockopt(socket_, SOL_SOCKET, SO_RCVBUF, reinterpret_cast(&socket_buffer_bytes), sizeof(socket_buffer_bytes)); + int actual_receive_buffer_bytes = 0; + int actual_receive_buffer_size = sizeof(actual_receive_buffer_bytes); + getsockopt(socket_, SOL_SOCKET, SO_RCVBUF, + reinterpret_cast(&actual_receive_buffer_bytes), &actual_receive_buffer_size); const DWORD receive_timeout_ms = 2; setsockopt(socket_, SOL_SOCKET, SO_RCVTIMEO, reinterpret_cast(&receive_timeout_ms), sizeof(receive_timeout_ms)); @@ -902,9 +905,11 @@ private: WSACleanup(); return; } + blog(LOG_INFO, "Muninn RUDP bridge receive owner priority=highest socket_buffer_bytes=%d", + actual_receive_buffer_bytes); - const SOCKET video_socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); - const SOCKET audio_socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + const SOCKET video_socket = muninn_udp_socket::create(); + const SOCKET audio_socket = muninn_udp_socket::create(); setsockopt(video_socket, SOL_SOCKET, SO_SNDBUF, reinterpret_cast(&socket_buffer_bytes), sizeof(socket_buffer_bytes)); setsockopt(audio_socket, SOL_SOCKET, SO_SNDBUF, reinterpret_cast(&socket_buffer_bytes), @@ -920,16 +925,26 @@ private: std::vector buffer(65535); uint32_t bridge_sequence = 1; + bool logged_receive_loop = false; + bool logged_datagram = false; while (running_) { sockaddr_in remote = {}; int remote_len = sizeof(remote); const int received = recvfrom(socket_, reinterpret_cast(buffer.data()), static_cast(buffer.size()), 0, reinterpret_cast(&remote), &remote_len); if (received <= 0) { + if (!logged_receive_loop) { + logged_receive_loop = true; + blog(LOG_INFO, "Muninn RUDP bridge receive loop active udp/%u error=%d", parts_.port, WSAGetLastError()); + } expire_rudp_packet_fragments(std::chrono::steady_clock::now()); expire_video_assemblies(std::chrono::steady_clock::now(), bridge_sequence); continue; } + if (!logged_datagram) { + logged_datagram = true; + blog(LOG_INFO, "Muninn RUDP bridge received first datagram bytes=%d", received); + } handle_packet(buffer.data(), static_cast(received), remote, bridge_sequence, video_socket, video_addr, audio_socket, audio_addr); } @@ -1033,6 +1048,16 @@ private: } if (packet_type == 3) { + auto &logged_first_data = is_audio_connection ? logged_first_audio_data_ : logged_first_video_data_; + if (!logged_first_data) { + logged_first_data = true; + blog(LOG_INFO, + "Muninn RUDP bridge first %s data sequence=%u payload=%u fragment_id=%u fragment_index=%u fragment_count=%u datagram=%zu", + is_audio_connection ? "audio" : "video", sequence, payload_len, + static_cast(read_be16(packet + 24)), + static_cast(read_be16(packet + 26)), + static_cast(fragment_count), size); + } if (!media_connected_) { video_assemblies_.clear(); video_parity_cache_.clear(); @@ -1076,7 +1101,7 @@ private: const std::string channel_id( reinterpret_cast(packet + 36), reinterpret_cast(packet + 36 + channel_len)); - if (channel_id != "media") { + if (channel_id != parts_.channel_id) { const auto [ack, ack_mask] = connection_ack_tracker.state(); send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote); return; @@ -1157,6 +1182,15 @@ private: log_bridge_pressure("dropped undecodable media payload"); return; } + if (!logged_first_decoded_media_) { + logged_first_decoded_media_ = true; + blog(LOG_INFO, + "Muninn RUDP bridge decoded first media kind=%u payload=%zu frame=%llu chunk=%u/%u", + static_cast(media->kind), media->payload.size(), + static_cast(media->frame_id), + static_cast(media->chunk_index), + static_cast(media->chunk_count)); + } size_t forwarded_bytes = 0; if (media->kind == MuninnMediaPayload::Kind::Video || media->kind == MuninnMediaPayload::Kind::VideoParity) { const auto assembled = assemble_video_payload(*media, bridge_sequence); @@ -1978,7 +2012,7 @@ private: void send_media_packet(const std::vector &payload, uint32_t sequence, const sockaddr_in &remote) { - const std::string channel = "media"; + const std::string &channel = parts_.channel_id; std::vector response(36 + channel.size() + payload.size()); response[0] = 'C'; response[1] = 'N'; @@ -2060,6 +2094,9 @@ private: uint64_t last_video_expiry_scan_ms_ = 0; bool waiting_for_video_keyframe_ = true; bool media_connected_ = false; + bool logged_first_video_data_ = false; + bool logged_first_audio_data_ = false; + bool logged_first_decoded_media_ = false; std::atomic last_keyframe_request_ms_ = 0; std::string source_url_; RudpBridgeOutputs outputs_; @@ -2686,7 +2723,7 @@ static bool publish_capture_command_rudp(const std::string &target, const std::v if (WSAStartup(MAKEWORD(2, 2), &wsa) != 0) { return false; } - SOCKET socket = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + SOCKET socket = muninn_udp_socket::create(); if (socket == INVALID_SOCKET) { WSACleanup(); return false; @@ -2742,7 +2779,7 @@ static std::string expected_rudp_stream_url(const MuninnSource *ctx, const std:: { const auto reliable_expire_after_ms = std::max(1, ctx->rudp_latency_budget_ms); return "rudp://" + ctx->media_target_host + ":" + std::to_string(ctx->media_port) + "/" + stream_id + - "?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" + + "?channel=realtime&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" + std::to_string(reliable_expire_after_ms) + "&assembly_deadline_ms=" + std::to_string(ctx->rudp_latency_budget_ms) + "&gap_wait_ms=16&audio_reorder_ms=" + std::to_string(ctx->rudp_audio_reorder_ms); } @@ -3378,7 +3415,7 @@ private: return; } #endif - socket_ = ::socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + socket_ = muninn_udp_socket::create(true); if (socket_ == INVALID_SOCKET) { #ifdef _WIN32 WSACleanup(); diff --git a/native/obs_stem_source/muninn_rudp_url.h b/native/obs_stem_source/muninn_rudp_url.h index ffbfbd2..c6275b9 100644 --- a/native/obs_stem_source/muninn_rudp_url.h +++ b/native/obs_stem_source/muninn_rudp_url.h @@ -18,6 +18,7 @@ inline constexpr uint32_t DefaultSenderResendDelayMs = 5; inline constexpr uint32_t MaxLatencyBudgetMs = 500; struct RudpUrlParts { + std::string channel_id = "realtime"; uint16_t port = 0; uint16_t video_local_port = 0; uint16_t audio_local_port = 0; @@ -81,7 +82,9 @@ inline std::optional parse(const std::string &url) } const auto key = item.substr(0, equals); const auto value = item.substr(equals + 1); - if (key == "connection") { + if (key == "channel" && !value.empty()) { + parts.channel_id = value; + } else if (key == "connection") { if (const auto parsed = parse_u32_hex_or_decimal(value)) { parts.connection_id = *parsed; } diff --git a/native/obs_stem_source/muninn_udp_socket.h b/native/obs_stem_source/muninn_udp_socket.h new file mode 100644 index 0000000..b49f767 --- /dev/null +++ b/native/obs_stem_source/muninn_udp_socket.h @@ -0,0 +1,32 @@ +#pragma once + +#ifdef _WIN32 +#include + +namespace muninn_udp_socket { + +inline SOCKET create(bool exclusive_listener = false) +{ + const SOCKET socket = WSASocketW( + AF_INET, + SOCK_DGRAM, + IPPROTO_UDP, + nullptr, + 0, + WSA_FLAG_NO_HANDLE_INHERIT); + if (socket == INVALID_SOCKET) { + return INVALID_SOCKET; + } + if (exclusive_listener) { + const BOOL exclusive = TRUE; + if (setsockopt(socket, SOL_SOCKET, SO_EXCLUSIVEADDRUSE, + reinterpret_cast(&exclusive), sizeof(exclusive)) != 0) { + closesocket(socket); + return INVALID_SOCKET; + } + } + return socket; +} + +} // namespace muninn_udp_socket +#endif diff --git a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp index 2cdebd0..1fba928 100644 --- a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp +++ b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp @@ -19,9 +19,10 @@ void require(bool condition, const std::string &message) int main() { const auto configured = muninn_rudp_url::parse( - "rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=120&assembly_deadline_ms=120&gap_wait_ms=16&audio_reorder_ms=80"); + "rudp://192.168.1.66:5204/muninn.raven.av.rudp?channel=realtime&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&sender_resend_delay_ms=5&reliable_expire_after_ms=120&assembly_deadline_ms=120&gap_wait_ms=16&audio_reorder_ms=80"); require(configured.has_value(), "configured RUDP URL did not parse"); require(configured->port == 5204, "configured RUDP URL port was not parsed"); + require(configured->channel_id == "realtime", "configured RUDP realtime channel was not parsed"); require(configured->video_local_port == 15204, "configured RUDP video local port was not derived"); require(configured->audio_local_port == 15205, "configured RUDP audio local port was not derived"); require(configured->connection_id == 0x6d750001, "configured RUDP connection id was not parsed"); @@ -41,6 +42,7 @@ int main() const auto fallback = muninn_rudp_url::parse("rudp://192.168.1.66:5204/muninn.raven.av.rudp"); require(fallback.has_value(), "minimal RUDP URL did not parse"); + require(fallback->channel_id == "realtime", "minimal RUDP URL did not use the experimental realtime lane"); require(fallback->connection_id == muninn_rudp_url::DefaultMediaRudpConnectionId, "minimal RUDP URL did not keep the default connection id"); require(fallback->audio_connection_id == muninn_rudp_url::DefaultAudioRudpConnectionId, diff --git a/native/obs_stem_source/tests/muninn_udp_socket_tests.cpp b/native/obs_stem_source/tests/muninn_udp_socket_tests.cpp new file mode 100644 index 0000000..db2c15b --- /dev/null +++ b/native/obs_stem_source/tests/muninn_udp_socket_tests.cpp @@ -0,0 +1,34 @@ +#include "muninn_udp_socket.h" + +#include +#include + +int main() +{ + WSADATA data{}; + assert(WSAStartup(MAKEWORD(2, 2), &data) == 0); + + const SOCKET owner = muninn_udp_socket::create(true); + assert(owner != INVALID_SOCKET); + DWORD flags = HANDLE_FLAG_INHERIT; + assert(GetHandleInformation(reinterpret_cast(owner), &flags)); + assert((flags & HANDLE_FLAG_INHERIT) == 0); + + sockaddr_in address{}; + address.sin_family = AF_INET; + address.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + address.sin_port = 0; + assert(bind(owner, reinterpret_cast(&address), sizeof(address)) == 0); + + int address_size = sizeof(address); + assert(getsockname(owner, reinterpret_cast(&address), &address_size) == 0); + const SOCKET contender = muninn_udp_socket::create(true); + assert(contender != INVALID_SOCKET); + assert(bind(contender, reinterpret_cast(&address), sizeof(address)) != 0); + assert(WSAGetLastError() == WSAEADDRINUSE || WSAGetLastError() == WSAEACCES); + + closesocket(contender); + closesocket(owner); + WSACleanup(); + return 0; +} From afce0245b1449f440f04aa5f1c160519d752bec2 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 01:37:28 +0100 Subject: [PATCH 117/135] Recover Muninn video bursts with bounded FEC --- docs/implementation-plan.md | 11 +- native/obs_stem_source/CMakeLists.txt | 7 + .../mimir_obs_muninn_source.cpp | 77 ++++------- native/obs_stem_source/muninn_video_fec.h | 127 ++++++++++++++++++ .../tests/muninn_video_fec_tests.cpp | 41 ++++++ state/evidence.jsonl | 1 + state/map.yaml | 12 +- 7 files changed, 223 insertions(+), 53 deletions(-) create mode 100644 native/obs_stem_source/muninn_video_fec.h create mode 100644 native/obs_stem_source/tests/muninn_video_fec_tests.cpp diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index cbfe454..d40c5a8 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -451,10 +451,13 @@ result under controlled loss/jitter/reorder before changing carriers. The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot `8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware -repair, and decode-chain-owned keyframe pressure. Next reliability work is real -encoder keyframe actuation, production parity recovery, audio FEC/concealment, -bounded long-disconnect edge admission, adaptive sender response, and the -fault-injected socket timeline harness. +repair, and decode-chain-owned keyframe pressure. The production video path now +uses typed V3 Cauchy GF(256) FEC with sixteen parity shards per access unit; +the OBS receiver reconstructs arbitrary erasure sets up to the received parity +count, while V2 stripe parity remains a distinct non-authoritative schema. +Sender access units enter the handoff queue independently, and CultMesh catalog +publication runs outside the realtime media loop. The remaining work is the +full direct/proxy impairment matrix, Opus FEC/PLC, and long-duration mixed soak. The socket harness and bounded PCM loss recovery have now landed. Odin's `cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/ diff --git a/native/obs_stem_source/CMakeLists.txt b/native/obs_stem_source/CMakeLists.txt index 527739d..6e19ee1 100644 --- a/native/obs_stem_source/CMakeLists.txt +++ b/native/obs_stem_source/CMakeLists.txt @@ -46,6 +46,13 @@ target_include_directories(muninn_audio_fec_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) add_test(NAME muninn_audio_fec_tests COMMAND muninn_audio_fec_tests) +add_executable(muninn_video_fec_tests + tests/muninn_video_fec_tests.cpp) +target_compile_features(muninn_video_fec_tests PRIVATE cxx_std_20) +target_include_directories(muninn_video_fec_tests PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}) +add_test(NAME muninn_video_fec_tests COMMAND muninn_video_fec_tests) + add_executable(muninn_udp_socket_tests tests/muninn_udp_socket_tests.cpp) target_compile_features(muninn_udp_socket_tests PRIVATE cxx_std_20) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 9910999..9743073 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -13,6 +13,7 @@ #include "muninn_rudp_fragments.h" #include "muninn_rudp_url.h" #include "muninn_udp_socket.h" +#include "muninn_video_fec.h" #include #include @@ -224,6 +225,7 @@ struct MuninnMediaPayload { uint16_t chunk_count = 1; uint16_t parity_index = 0; uint16_t parity_count = 0; + bool video_fec_cauchy = false; std::vector chunk_payload_lengths; std::vector payload; }; @@ -240,6 +242,7 @@ struct MuninnVideoFrameAssembly { bool repair_requested = false; std::vector chunk_payload_lengths; uint16_t parity_count = 0; + bool video_fec_cauchy = false; std::map> parity_payloads; std::vector> chunks; }; @@ -253,6 +256,7 @@ struct MuninnVideoParityCacheEntry { uint16_t chunk_count = 0; uint16_t parity_index = 0; uint16_t parity_count = 0; + bool video_fec_cauchy = false; std::chrono::steady_clock::time_point received_at; std::vector chunk_payload_lengths; std::vector payload; @@ -655,7 +659,8 @@ static std::optional decode_muninn_media_payload(const uint8 media.payload = record.read_bytes(); return media; } - if (schema_id == "muninn.media_video_parity_shard.v2") { + if (schema_id == "muninn.media_video_parity_shard.v2" || + schema_id == "muninn.media_video_parity_shard.v3") { if (record_len < 17) { return std::nullopt; } @@ -675,6 +680,7 @@ static std::optional decode_muninn_media_payload(const uint8 media.chunk_count = static_cast(record.read_unsigned()); media.parity_index = static_cast(record.read_unsigned()); media.parity_count = static_cast(record.read_unsigned()); + media.video_fec_cauchy = schema_id == "muninn.media_video_parity_shard.v3"; const uint32_t chunk_payload_bytes = static_cast(record.read_unsigned()); const uint32_t last_chunk_payload_bytes = static_cast(record.read_unsigned()); if (media.chunk_count == 0 || @@ -1600,7 +1606,15 @@ private: log_bridge_pressure("dropped video parity with mixed stripe count"); return std::nullopt; } + if (assembly.parity_count != 0 && assembly.video_fec_cauchy != media.video_fec_cauchy) { + video_assemblies_.erase(key); + ++payloads_decoded_dropped_; + ++video_parity_unusable_; + log_bridge_pressure("dropped video parity with mixed FEC scheme"); + return std::nullopt; + } assembly.parity_count = media.parity_count; + assembly.video_fec_cauchy = media.video_fec_cauchy; assembly.parity_payloads[media.parity_index] = media.payload; if (try_recover_video_assembly_from_parity(assembly)) { ++video_parity_recovered_; @@ -1621,6 +1635,7 @@ private: entry.chunk_count = media.chunk_count; entry.parity_index = media.parity_index; entry.parity_count = media.parity_count; + entry.video_fec_cauchy = media.video_fec_cauchy; entry.received_at = std::chrono::steady_clock::now(); entry.chunk_payload_lengths = media.chunk_payload_lengths; entry.payload = media.payload; @@ -1641,9 +1656,12 @@ private: entry.dependency_frame_id == assembly.dependency_frame_id && entry.parity_count != 0 && entry.parity_index < entry.parity_count && - (assembly.parity_count == 0 || assembly.parity_count == entry.parity_count)) { + (assembly.parity_count == 0 || + (assembly.parity_count == entry.parity_count && + assembly.video_fec_cauchy == entry.video_fec_cauchy))) { assembly.chunk_payload_lengths = entry.chunk_payload_lengths; assembly.parity_count = entry.parity_count; + assembly.video_fec_cauchy = entry.video_fec_cauchy; assembly.parity_payloads[entry.parity_index] = entry.payload; } else { ++video_parity_unusable_; @@ -1671,58 +1689,21 @@ private: bool try_recover_video_assembly_from_parity(MuninnVideoFrameAssembly &assembly) { if (assembly.parity_count == 0 || + !assembly.video_fec_cauchy || assembly.parity_payloads.empty() || assembly.chunk_payload_lengths.size() != assembly.chunk_count || assembly.chunks.size() != assembly.chunk_count) { return false; } - bool recovered_any = true; - while (recovered_any && assembly.chunks_received < assembly.chunk_count) { - recovered_any = false; - for (const auto &parity : assembly.parity_payloads) { - const uint16_t parity_index = parity.first; - if (parity_index >= assembly.parity_count || parity.second.empty()) { - continue; - } - uint16_t missing_index = 0; - uint16_t missing_count = 0; - for (uint16_t index = parity_index; index < assembly.chunk_count; index += assembly.parity_count) { - if (assembly.chunks[index].empty()) { - missing_index = index; - ++missing_count; - } - } - if (missing_count != 1) { - continue; - } - const uint32_t missing_len = assembly.chunk_payload_lengths[missing_index]; - if (missing_len == 0 || missing_len > parity.second.size()) { - continue; - } - std::vector recovered = parity.second; - bool stripe_complete = true; - for (uint16_t index = parity_index; index < assembly.chunk_count; index += assembly.parity_count) { - if (index == missing_index) { - continue; - } - const auto &chunk = assembly.chunks[index]; - if (chunk.empty()) { - stripe_complete = false; - break; - } - for (size_t offset = 0; offset < chunk.size() && offset < recovered.size(); ++offset) { - recovered[offset] ^= chunk[offset]; - } - } - if (!stripe_complete) { - continue; - } - recovered.resize(missing_len); - assembly.chunks[missing_index] = std::move(recovered); - ++assembly.chunks_received; - recovered_any = true; - } + const uint16_t missing = assembly.chunk_count - assembly.chunks_received; + if (!muninn_video_fec::recover( + assembly.chunks, + assembly.chunk_payload_lengths, + assembly.parity_count, + assembly.parity_payloads)) { + return false; } + assembly.chunks_received += missing; return assembly.chunks_received == assembly.chunk_count; } diff --git a/native/obs_stem_source/muninn_video_fec.h b/native/obs_stem_source/muninn_video_fec.h new file mode 100644 index 0000000..7bbce51 --- /dev/null +++ b/native/obs_stem_source/muninn_video_fec.h @@ -0,0 +1,127 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +namespace muninn_video_fec { + +inline uint8_t multiply(uint8_t a, uint8_t b) +{ + uint8_t product = 0; + while (b != 0) { + if ((b & 1) != 0) + product ^= a; + const bool carry = (a & 0x80) != 0; + a <<= 1; + if (carry) + a ^= 0x1d; + b >>= 1; + } + return product; +} + +inline uint8_t inverse(uint8_t value) +{ + uint8_t result = 1; + for (size_t index = 0; index < 254; ++index) + result = multiply(result, value); + return result; +} + +inline uint8_t coefficient(uint16_t parity_index, uint16_t parity_count, uint16_t data_index) +{ + return inverse( + static_cast(parity_index) ^ static_cast(parity_count + data_index)); +} + +inline bool recover(std::vector> &chunks, + const std::vector &chunk_lengths, + uint16_t parity_count, + const std::map> &parity) +{ + if (chunks.size() != chunk_lengths.size() || chunks.empty() || + chunks.size() + parity_count > 256) + return false; + std::vector missing; + for (uint16_t index = 0; index < chunks.size(); ++index) { + if (chunks[index].empty()) + missing.push_back(index); + } + if (missing.empty()) + return true; + if (missing.size() > parity.size()) + return false; + + std::vector *>> rows; + for (const auto &shard : parity) { + if (shard.first < parity_count && !shard.second.empty()) + rows.emplace_back(shard.first, &shard.second); + if (rows.size() == missing.size()) + break; + } + if (rows.size() != missing.size()) + return false; + + size_t shard_bytes = 0; + for (uint32_t length : chunk_lengths) + shard_bytes = std::max(shard_bytes, static_cast(length)); + if (shard_bytes == 0 || std::any_of(rows.begin(), rows.end(), + [shard_bytes](const auto &row) { return row.second->size() < shard_bytes; })) + return false; + + std::vector> recovered; + recovered.reserve(missing.size()); + for (uint16_t index : missing) { + if (chunk_lengths[index] == 0 || chunk_lengths[index] > shard_bytes) + return false; + recovered.emplace_back(chunk_lengths[index], 0); + } + + const size_t count = missing.size(); + for (size_t offset = 0; offset < shard_bytes; ++offset) { + std::vector> matrix(count, std::vector(count + 1, 0)); + for (size_t row = 0; row < count; ++row) { + const uint16_t parity_index = rows[row].first; + uint8_t rhs = (*rows[row].second)[offset]; + for (uint16_t data_index = 0; data_index < chunks.size(); ++data_index) { + if (!chunks[data_index].empty() && offset < chunks[data_index].size()) + rhs ^= multiply(coefficient(parity_index, parity_count, data_index), + chunks[data_index][offset]); + } + for (size_t column = 0; column < count; ++column) + matrix[row][column] = coefficient(parity_index, parity_count, missing[column]); + matrix[row][count] = rhs; + } + for (size_t column = 0; column < count; ++column) { + size_t pivot = column; + while (pivot < count && matrix[pivot][column] == 0) + ++pivot; + if (pivot == count) + return false; + std::swap(matrix[column], matrix[pivot]); + const uint8_t scale = inverse(matrix[column][column]); + for (size_t cell = column; cell <= count; ++cell) + matrix[column][cell] = multiply(matrix[column][cell], scale); + for (size_t row = 0; row < count; ++row) { + if (row == column) + continue; + const uint8_t factor = matrix[row][column]; + for (size_t cell = column; cell <= count; ++cell) + matrix[row][cell] ^= multiply(factor, matrix[column][cell]); + } + } + for (size_t index = 0; index < count; ++index) { + if (offset < recovered[index].size()) + recovered[index][offset] = matrix[index][count]; + } + } + for (size_t index = 0; index < count; ++index) + chunks[missing[index]] = std::move(recovered[index]); + return true; +} + +} // namespace muninn_video_fec diff --git a/native/obs_stem_source/tests/muninn_video_fec_tests.cpp b/native/obs_stem_source/tests/muninn_video_fec_tests.cpp new file mode 100644 index 0000000..b5e1533 --- /dev/null +++ b/native/obs_stem_source/tests/muninn_video_fec_tests.cpp @@ -0,0 +1,41 @@ +#include "muninn_video_fec.h" + +#include +#include +#include +#include + +int main() +{ + constexpr uint16_t data_count = 64; + constexpr uint16_t parity_count = 16; + constexpr size_t shard_bytes = 97; + std::vector> original(data_count, std::vector(shard_bytes)); + for (uint16_t data = 0; data < data_count; ++data) { + for (size_t offset = 0; offset < shard_bytes; ++offset) + original[data][offset] = static_cast((data * 29 + offset * 17) & 0xff); + } + std::map> parity; + for (uint16_t row = 0; row < parity_count; ++row) { + auto &shard = parity[row]; + shard.assign(shard_bytes, 0); + for (uint16_t data = 0; data < data_count; ++data) { + const uint8_t coefficient = muninn_video_fec::coefficient(row, parity_count, data); + for (size_t offset = 0; offset < shard_bytes; ++offset) + shard[offset] ^= muninn_video_fec::multiply(original[data][offset], coefficient); + } + } + const std::vector lengths(data_count, shard_bytes); + for (uint16_t burst = 1; burst <= parity_count; ++burst) { + auto damaged = original; + for (uint16_t index = 0; index < burst; ++index) + damaged[20 + index].clear(); + assert(muninn_video_fec::recover(damaged, lengths, parity_count, parity)); + assert(damaged == original); + } + auto excessive = original; + for (uint16_t index = 0; index <= parity_count; ++index) + excessive[index].clear(); + assert(!muninn_video_fec::recover(excessive, lengths, parity_count, parity)); + return 0; +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index ffda9f8..38c78be 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -194,3 +194,4 @@ {"ts": "2026-07-16T21:26:00+01:00", "type": "verification", "status": "ok", "note": "Receiver decode-chain invalidation now actuates a verified next-frame NVENC IDR through Odin's long-lived libav encoder without restarting A/V or CultNet. Live 2560x1440 D3D11 proof emitted a command-driven IDR at frame 22 under a 600-frame GOP; synthetic proof emitted IDRs at frames 0 and 10 and decoded cleanly. Muninn/Sleipnir edge admission is bounded at 256 via epoch rebase; 174/26 tests pass. Raven field deployment remains pending because 10.77.0.4 SSH and TCP probing timed out."} {"ts": "2026-07-16T21:35:47+01:00", "type": "verification", "status": "ok", "note": "Muninn receiver pressure now drives bounded AIMD bitrate on the long-lived NVENC context. Live 2560x1440 D3D11 proof reconfigured 12 Mbps to 6 Mbps at frame 35, emitted an IDR at frame 35, and decoded cleanly; 176 Muninn tests pass. Cross-host field acceptance remains missing because Raven SSH/TCP 22 timed out again."} {"ts": "2026-07-16T21:38:23+01:00", "type": "blocker", "status": "pending", "note": "Field Moonlight acceptance cannot run while Raven is offline: Starfire's documented SSH/ICMP/TCP route to 10.77.0.4 times out, and Yggdrasil reports Raven's last WireGuard handshake at 2026-07-15T06:04:12Z versus Starfire's live handshake at 2026-07-16T20:36:32Z. No Raven task or host state was mutated."} +{"ts": "2026-07-17T01:37:22+01:00", "type": "verification", "status": "partial", "note": "Muninn video parity v3 uses 16 Cauchy GF(256) shards; native receiver burst tests 1..16 and Odin 180-test suite pass. Direct Raven LAN at 250 ms recovery deadline ran over one minute with no video assembly expiry or A/V failure; full 10-minute and impairment matrix remain open."} diff --git a/state/map.yaml b/state/map.yaml index 041cbc2..b1e0281 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,16 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: + - Muninn video FEC is now typed as `muninn.media_video_parity_shard.v3`. + Odin emits 16 Cauchy GF(256) parity shards after each multi-chunk access + unit, so any erasure set up to the received parity count is recoverable; + Mimir's OBS receiver owns the matching bounded matrix recovery before the + assembly deadline. V2 stripe parity is never reinterpreted as V3. Native + receiver tests prove consecutive bursts from one through sixteen shards, + Odin's 180 Muninn tests pass, and the first direct Raven LAN field minute + at a 250 ms recovery deadline had no video assembly expiry or A/V failure. + Normal frames are forwarded immediately; the recovery deadline is not a + playout buffer. The ten-minute and impaired acceptance gates remain open. - Muninn now closes receiver congestion pressure into the live NVENC context. Every 500 ms it applies multiplicative 15 percent backoff for late frames, requested recovery, deferred repair, sender queue loss, or combined @@ -95,7 +105,7 @@ accepted_design: seconds, with a one-quarter/1 Mbps floor. A live 2560x1440 D3D11 proof changed 12 Mbps to 6 Mbps at frame 35; FFmpeg/NVENC reconfigured the same session, emitted an IDR at frame 35, and the Annex-B stream decoded cleanly. - Muninn's full suite now passes 176 tests. + Muninn's full suite now passes 180 tests. - Receiver-driven video recovery now reaches the actual NVENC frame owner. Odin's `native/muninn-video-encoder` is a long-lived libavdevice desktop capture and `h264_nvenc` process; a new decode-chain invalidation edge makes From e9e7eaae3069a4dd947beb4311fd090c06742f94 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 01:46:28 +0100 Subject: [PATCH 118/135] Bound Muninn video FEC decode time --- native/obs_stem_source/muninn_video_fec.h | 92 +++++++++++++------ .../tests/muninn_video_fec_tests.cpp | 19 +++- state/evidence.jsonl | 1 + state/map.yaml | 8 +- 4 files changed, 84 insertions(+), 36 deletions(-) diff --git a/native/obs_stem_source/muninn_video_fec.h b/native/obs_stem_source/muninn_video_fec.h index 7bbce51..a2be0d0 100644 --- a/native/obs_stem_source/muninn_video_fec.h +++ b/native/obs_stem_source/muninn_video_fec.h @@ -9,7 +9,7 @@ namespace muninn_video_fec { -inline uint8_t multiply(uint8_t a, uint8_t b) +inline uint8_t raw_multiply(uint8_t a, uint8_t b) { uint8_t product = 0; while (b != 0) { @@ -24,12 +24,39 @@ inline uint8_t multiply(uint8_t a, uint8_t b) return product; } +struct Tables { + uint8_t products[256][256] = {}; + uint8_t inverses[256] = {}; + + Tables() + { + for (size_t a = 0; a < 256; ++a) { + for (size_t b = 0; b < 256; ++b) + products[a][b] = raw_multiply(static_cast(a), static_cast(b)); + } + for (size_t value = 1; value < 256; ++value) { + uint8_t result = 1; + for (size_t index = 0; index < 254; ++index) + result = raw_multiply(result, static_cast(value)); + inverses[value] = result; + } + } +}; + +inline const Tables &tables() +{ + static const Tables value; + return value; +} + +inline uint8_t multiply(uint8_t a, uint8_t b) +{ + return tables().products[a][b]; +} + inline uint8_t inverse(uint8_t value) { - uint8_t result = 1; - for (size_t index = 0; index < 254; ++index) - result = multiply(result, value); - return result; + return tables().inverses[value]; } inline uint8_t coefficient(uint16_t parity_index, uint16_t parity_count, uint16_t data_index) @@ -82,41 +109,48 @@ inline bool recover(std::vector> &chunks, } const size_t count = missing.size(); - for (size_t offset = 0; offset < shard_bytes; ++offset) { - std::vector> matrix(count, std::vector(count + 1, 0)); - for (size_t row = 0; row < count; ++row) { - const uint16_t parity_index = rows[row].first; - uint8_t rhs = (*rows[row].second)[offset]; - for (uint16_t data_index = 0; data_index < chunks.size(); ++data_index) { - if (!chunks[data_index].empty() && offset < chunks[data_index].size()) - rhs ^= multiply(coefficient(parity_index, parity_count, data_index), - chunks[data_index][offset]); - } - for (size_t column = 0; column < count; ++column) - matrix[row][column] = coefficient(parity_index, parity_count, missing[column]); - matrix[row][count] = rhs; - } - for (size_t column = 0; column < count; ++column) { + std::vector> inverse_matrix(count, std::vector(count * 2, 0)); + for (size_t row = 0; row < count; ++row) { + for (size_t column = 0; column < count; ++column) + inverse_matrix[row][column] = coefficient(rows[row].first, parity_count, missing[column]); + inverse_matrix[row][count + row] = 1; + } + for (size_t column = 0; column < count; ++column) { size_t pivot = column; - while (pivot < count && matrix[pivot][column] == 0) + while (pivot < count && inverse_matrix[pivot][column] == 0) ++pivot; if (pivot == count) return false; - std::swap(matrix[column], matrix[pivot]); - const uint8_t scale = inverse(matrix[column][column]); - for (size_t cell = column; cell <= count; ++cell) - matrix[column][cell] = multiply(matrix[column][cell], scale); + std::swap(inverse_matrix[column], inverse_matrix[pivot]); + const uint8_t scale = inverse(inverse_matrix[column][column]); + for (size_t cell = 0; cell < count * 2; ++cell) + inverse_matrix[column][cell] = multiply(inverse_matrix[column][cell], scale); for (size_t row = 0; row < count; ++row) { if (row == column) continue; - const uint8_t factor = matrix[row][column]; - for (size_t cell = column; cell <= count; ++cell) - matrix[row][cell] ^= multiply(factor, matrix[column][cell]); + const uint8_t factor = inverse_matrix[row][column]; + for (size_t cell = 0; cell < count * 2; ++cell) + inverse_matrix[row][cell] ^= multiply(factor, inverse_matrix[column][cell]); + } + } + + std::vector rhs(count, 0); + for (size_t offset = 0; offset < shard_bytes; ++offset) { + for (size_t row = 0; row < count; ++row) { + const uint16_t parity_index = rows[row].first; + rhs[row] = (*rows[row].second)[offset]; + for (uint16_t data_index = 0; data_index < chunks.size(); ++data_index) { + if (!chunks[data_index].empty() && offset < chunks[data_index].size()) + rhs[row] ^= multiply(coefficient(parity_index, parity_count, data_index), + chunks[data_index][offset]); } } for (size_t index = 0; index < count; ++index) { + uint8_t value = 0; + for (size_t row = 0; row < count; ++row) + value ^= multiply(inverse_matrix[index][count + row], rhs[row]); if (offset < recovered[index].size()) - recovered[index][offset] = matrix[index][count]; + recovered[index][offset] = value; } } for (size_t index = 0; index < count; ++index) diff --git a/native/obs_stem_source/tests/muninn_video_fec_tests.cpp b/native/obs_stem_source/tests/muninn_video_fec_tests.cpp index b5e1533..4b9bce4 100644 --- a/native/obs_stem_source/tests/muninn_video_fec_tests.cpp +++ b/native/obs_stem_source/tests/muninn_video_fec_tests.cpp @@ -1,15 +1,22 @@ #include "muninn_video_fec.h" -#include +#include +#include #include #include #include +void require(bool condition) +{ + if (!condition) + std::abort(); +} + int main() { constexpr uint16_t data_count = 64; constexpr uint16_t parity_count = 16; - constexpr size_t shard_bytes = 97; + constexpr size_t shard_bytes = 1400; std::vector> original(data_count, std::vector(shard_bytes)); for (uint16_t data = 0; data < data_count; ++data) { for (size_t offset = 0; offset < shard_bytes; ++offset) @@ -30,12 +37,14 @@ int main() auto damaged = original; for (uint16_t index = 0; index < burst; ++index) damaged[20 + index].clear(); - assert(muninn_video_fec::recover(damaged, lengths, parity_count, parity)); - assert(damaged == original); + const auto started = std::chrono::steady_clock::now(); + require(muninn_video_fec::recover(damaged, lengths, parity_count, parity)); + require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50)); + require(damaged == original); } auto excessive = original; for (uint16_t index = 0; index <= parity_count; ++index) excessive[index].clear(); - assert(!muninn_video_fec::recover(excessive, lengths, parity_count, parity)); + require(!muninn_video_fec::recover(excessive, lengths, parity_count, parity)); return 0; } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 38c78be..113c98f 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -195,3 +195,4 @@ {"ts": "2026-07-16T21:35:47+01:00", "type": "verification", "status": "ok", "note": "Muninn receiver pressure now drives bounded AIMD bitrate on the long-lived NVENC context. Live 2560x1440 D3D11 proof reconfigured 12 Mbps to 6 Mbps at frame 35, emitted an IDR at frame 35, and decoded cleanly; 176 Muninn tests pass. Cross-host field acceptance remains missing because Raven SSH/TCP 22 timed out again."} {"ts": "2026-07-16T21:38:23+01:00", "type": "blocker", "status": "pending", "note": "Field Moonlight acceptance cannot run while Raven is offline: Starfire's documented SSH/ICMP/TCP route to 10.77.0.4 times out, and Yggdrasil reports Raven's last WireGuard handshake at 2026-07-15T06:04:12Z versus Starfire's live handshake at 2026-07-16T20:36:32Z. No Raven task or host state was mutated."} {"ts": "2026-07-17T01:37:22+01:00", "type": "verification", "status": "partial", "note": "Muninn video parity v3 uses 16 Cauchy GF(256) shards; native receiver burst tests 1..16 and Odin 180-test suite pass. Direct Raven LAN at 250 ms recovery deadline ran over one minute with no video assembly expiry or A/V failure; full 10-minute and impairment matrix remain open."} +{"ts": "2026-07-17T01:46:28+01:00", "type": "scar", "status": "fixed", "note": "Per-byte Gaussian elimination for production-sized video FEC can stall the single OBS receive owner and cascade socket loss. Receiver now inverts coefficients once per frame and uses table-driven GF(256); active Release assertions prove every 1..16 consecutive erasure case on 64x1400-byte shards under 50 ms."} diff --git a/state/map.yaml b/state/map.yaml index b1e0281..5e137cd 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -93,8 +93,12 @@ accepted_design: Mimir's OBS receiver owns the matching bounded matrix recovery before the assembly deadline. V2 stripe parity is never reinterpreted as V3. Native receiver tests prove consecutive bursts from one through sixteen shards, - Odin's 180 Muninn tests pass, and the first direct Raven LAN field minute - at a 250 ms recovery deadline had no video assembly expiry or A/V failure. + including production-sized 64 by 1400-byte frames under a 50 ms per-frame + recovery bound. Odin's 180 Muninn tests pass. A per-byte elimination draft + was rejected after one recovery stalled the receiver thread; production + inverts the coefficient matrix once per frame and uses table-driven field + arithmetic. The fresh direct Raven LAN session has no video assembly expiry + or A/V failure in its initial minute at a 250 ms recovery deadline. Normal frames are forwarded immediately; the recovery deadline is not a playout buffer. The ten-minute and impaired acceptance gates remain open. - Muninn now closes receiver congestion pressure into the live NVENC context. From 79781abdbb3517553544adb60cc181b8fbf8e4e2 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 01:59:33 +0100 Subject: [PATCH 119/135] Record Muninn trailing parity burst scar --- state/evidence.jsonl | 1 + state/map.yaml | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 113c98f..c3936c3 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -196,3 +196,4 @@ {"ts": "2026-07-16T21:38:23+01:00", "type": "blocker", "status": "pending", "note": "Field Moonlight acceptance cannot run while Raven is offline: Starfire's documented SSH/ICMP/TCP route to 10.77.0.4 times out, and Yggdrasil reports Raven's last WireGuard handshake at 2026-07-15T06:04:12Z versus Starfire's live handshake at 2026-07-16T20:36:32Z. No Raven task or host state was mutated."} {"ts": "2026-07-17T01:37:22+01:00", "type": "verification", "status": "partial", "note": "Muninn video parity v3 uses 16 Cauchy GF(256) shards; native receiver burst tests 1..16 and Odin 180-test suite pass. Direct Raven LAN at 250 ms recovery deadline ran over one minute with no video assembly expiry or A/V failure; full 10-minute and impairment matrix remain open."} {"ts": "2026-07-17T01:46:28+01:00", "type": "scar", "status": "fixed", "note": "Per-byte Gaussian elimination for production-sized video FEC can stall the single OBS receive owner and cascade socket loss. Receiver now inverts coefficients once per frame and uses table-driven GF(256); active Release assertions prove every 1..16 consecutive erasure case on 64x1400-byte shards under 50 ms."} +{"ts": "2026-07-17T01:59:33+01:00", "type": "scar", "status": "open", "note": "Direct Raven LAN falsified trailing-parity sufficiency: a 24-datagram sequence gap left a 56-chunk keyframe at 52/56 with parity_present=false because all 16 V3 parity shards followed the data and fell in the same tail burst. Next cut must disperse parity/adapt block layout; deadline increases and repair flooding are not owners."} diff --git a/state/map.yaml b/state/map.yaml index 5e137cd..7de8d14 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -97,8 +97,12 @@ accepted_design: recovery bound. Odin's 180 Muninn tests pass. A per-byte elimination draft was rejected after one recovery stalled the receiver thread; production inverts the coefficient matrix once per frame and uses table-driven field - arithmetic. The fresh direct Raven LAN session has no video assembly expiry - or A/V failure in its initial minute at a 250 ms recovery deadline. + arithmetic. Direct Raven LAN evidence then exposed a remaining layout fault: + a 24-datagram tail burst erased four final data chunks plus all sixteen + trailing parity shards, so no parity survived to repair the otherwise small + data loss. Repair retransmission is now capped behind fresh media and video + datagrams use a 50 microsecond spin pacer, but parity dispersion/adaptive + block sizing remains required before the clean ten-minute gate can pass. Normal frames are forwarded immediately; the recovery deadline is not a playout buffer. The ten-minute and impaired acceptance gates remain open. - Muninn now closes receiver congestion pressure into the live NVENC context. From 02c615670acf263ca96d80329fe8bc7374b3dcef Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 02:34:51 +0100 Subject: [PATCH 120/135] Ignore late shards from completed Muninn frames --- .../mimir_obs_muninn_source.cpp | 33 +++++++++++++++---- state/evidence.jsonl | 1 + state/map.yaml | 22 +++++++++---- 3 files changed, 43 insertions(+), 13 deletions(-) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 9743073..03f3564 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -67,6 +67,7 @@ constexpr uint64_t MuninnReceiverKeyframeRequestCooldownMs = 500; constexpr uint64_t MuninnReceiverChunkRepairFirstWaitMs = 64; constexpr uint64_t MuninnVideoAssemblyExpiryScanMs = 10; constexpr size_t MuninnExpiredVideoFrameRememberCount = 4096; +constexpr size_t MuninnCompletedVideoFrameRememberCount = 4096; constexpr size_t MuninnVideoParityCacheMaxEntries = 4096; constexpr uint16_t MuninnCatalogDiscoveryPort = 17874; constexpr uint32_t MuninnCatalogDiscoveryConnectionId = 0x6d750003; @@ -1515,15 +1516,15 @@ private: log_bridge_pressure("dropped invalid video chunk metadata"); return std::nullopt; } - if (media.chunk_count == 1) { - return media.payload; - } - const std::string key = media.stream_id + ":" + media.session_id + ":" + std::to_string(media.frame_id); - if (expired_video_frame_keys_.find(key) != expired_video_frame_keys_.end()) { + if (video_frame_is_settled(key)) { return std::nullopt; } + if (media.chunk_count == 1) { + remember_completed_video_frame(key); + return media.payload; + } auto &assembly = video_assemblies_[key]; if (assembly.chunks.empty()) { assembly.stream_id = media.stream_id; @@ -1579,7 +1580,7 @@ private: } const std::string key = media.stream_id + ":" + media.session_id + ":" + std::to_string(media.frame_id); - if (expired_video_frame_keys_.find(key) != expired_video_frame_keys_.end()) { + if (video_frame_is_settled(key)) { ++video_parity_ignored_; return std::nullopt; } @@ -1683,9 +1684,27 @@ private: } video_assemblies_.erase(key); video_parity_cache_.erase(key); + remember_completed_video_frame(key); return assembled; } + bool video_frame_is_settled(const std::string &key) const + { + return expired_video_frame_keys_.find(key) != expired_video_frame_keys_.end() || + completed_video_frame_keys_.find(key) != completed_video_frame_keys_.end(); + } + + void remember_completed_video_frame(const std::string &key) + { + if (completed_video_frame_keys_.insert(key).second) { + completed_video_frame_order_.push_back(key); + } + while (completed_video_frame_order_.size() > MuninnCompletedVideoFrameRememberCount) { + completed_video_frame_keys_.erase(completed_video_frame_order_.front()); + completed_video_frame_order_.pop_front(); + } + } + bool try_recover_video_assembly_from_parity(MuninnVideoFrameAssembly &assembly) { if (assembly.parity_count == 0 || @@ -2037,6 +2056,8 @@ private: std::map> video_parity_cache_; std::deque expired_video_frame_order_; std::set expired_video_frame_keys_; + std::deque completed_video_frame_order_; + std::set completed_video_frame_keys_; muninn_rudp_fragments::FragmentAssembler packet_fragments_; muninn_rudp_fragments::FragmentAssembler audio_packet_fragments_; muninn_rudp_ack::AckTracker ack_tracker_; diff --git a/state/evidence.jsonl b/state/evidence.jsonl index c3936c3..c35ff33 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -197,3 +197,4 @@ {"ts": "2026-07-17T01:37:22+01:00", "type": "verification", "status": "partial", "note": "Muninn video parity v3 uses 16 Cauchy GF(256) shards; native receiver burst tests 1..16 and Odin 180-test suite pass. Direct Raven LAN at 250 ms recovery deadline ran over one minute with no video assembly expiry or A/V failure; full 10-minute and impairment matrix remain open."} {"ts": "2026-07-17T01:46:28+01:00", "type": "scar", "status": "fixed", "note": "Per-byte Gaussian elimination for production-sized video FEC can stall the single OBS receive owner and cascade socket loss. Receiver now inverts coefficients once per frame and uses table-driven GF(256); active Release assertions prove every 1..16 consecutive erasure case on 64x1400-byte shards under 50 ms."} {"ts": "2026-07-17T01:59:33+01:00", "type": "scar", "status": "open", "note": "Direct Raven LAN falsified trailing-parity sufficiency: a 24-datagram sequence gap left a 56-chunk keyframe at 52/56 with parity_present=false because all 16 V3 parity shards followed the data and fell in the same tail burst. Next cut must disperse parity/adapt block layout; deadline increases and repair flooding are not owners."} +{"ts": "2026-07-17T02:34:05+01:00", "type": "verification", "status": "ok", "note": "Receiver completed-frame tombstones eliminated false post-FEC ghost assemblies: a 7-minute LAN run forwarded 54,000 payloads with zero video assembly expiry before an experimental weighted sender scheduler independently caused audio queue eviction; the scheduler was deleted. A later genuine 24-datagram tail burst left a frame 18/33 with only the leading 8 of 16 bookended parity shards available, proving V3 frame-wide bookends still need Moonlight-style multi-block FEC."} diff --git a/state/map.yaml b/state/map.yaml index 7de8d14..edb3707 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -88,7 +88,7 @@ invariants: accepted_design: - Muninn video FEC is now typed as `muninn.media_video_parity_shard.v3`. - Odin emits 16 Cauchy GF(256) parity shards after each multi-chunk access + Odin emits 16 Cauchy GF(256) parity shards around each multi-chunk access unit, so any erasure set up to the received parity count is recoverable; Mimir's OBS receiver owns the matching bounded matrix recovery before the assembly deadline. V2 stripe parity is never reinterpreted as V3. Native @@ -97,12 +97,20 @@ accepted_design: recovery bound. Odin's 180 Muninn tests pass. A per-byte elimination draft was rejected after one recovery stalled the receiver thread; production inverts the coefficient matrix once per frame and uses table-driven field - arithmetic. Direct Raven LAN evidence then exposed a remaining layout fault: - a 24-datagram tail burst erased four final data chunks plus all sixteen - trailing parity shards, so no parity survived to repair the otherwise small - data loss. Repair retransmission is now capped behind fresh media and video - datagrams use a 50 microsecond spin pacer, but parity dispersion/adaptive - block sizing remains required before the clean ten-minute gate can pass. + arithmetic. Direct Raven LAN evidence exposed two distinct faults. First, + successful early FEC completion did not tombstone the frame, so late + original chunks resurrected a partial ghost assembly and manufactured loss + feedback; the receiver now remembers completed frame keys and ignores that + residue. A seven-minute proof then kept every video failure counter at zero + while recovering 12,504 frames. Second, a genuine 24-datagram tail burst + left one frame at 18/33 chunks: the leading parity bookend supplied only + eight equations and the trailing eight were lost with fifteen data shards. + Repair retransmission is capped behind fresh media, IDR requests are + coalesced to 500 ms, and video datagrams use a 50 microsecond spin pacer, + but Moonlight-style independently recoverable FEC blocks remain required + before the clean ten-minute gate can pass. An experimental weighted audio/ + video scheduler was deleted after its accumulated audio debt caused bounded + queue eviction; audio retains strict latency priority. Normal frames are forwarded immediately; the recovery deadline is not a playout buffer. The ten-minute and impaired acceptance gates remain open. - Muninn now closes receiver congestion pressure into the live NVENC context. From 12489075d72772680734f014f2fb2fa5bd1d64d1 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 02:58:49 +0100 Subject: [PATCH 121/135] Record Muninn frame-wide FEC limit --- state/evidence.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index c35ff33..7e336cb 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -198,3 +198,4 @@ {"ts": "2026-07-17T01:46:28+01:00", "type": "scar", "status": "fixed", "note": "Per-byte Gaussian elimination for production-sized video FEC can stall the single OBS receive owner and cascade socket loss. Receiver now inverts coefficients once per frame and uses table-driven GF(256); active Release assertions prove every 1..16 consecutive erasure case on 64x1400-byte shards under 50 ms."} {"ts": "2026-07-17T01:59:33+01:00", "type": "scar", "status": "open", "note": "Direct Raven LAN falsified trailing-parity sufficiency: a 24-datagram sequence gap left a 56-chunk keyframe at 52/56 with parity_present=false because all 16 V3 parity shards followed the data and fell in the same tail burst. Next cut must disperse parity/adapt block layout; deadline increases and repair flooding are not owners."} {"ts": "2026-07-17T02:34:05+01:00", "type": "verification", "status": "ok", "note": "Receiver completed-frame tombstones eliminated false post-FEC ghost assemblies: a 7-minute LAN run forwarded 54,000 payloads with zero video assembly expiry before an experimental weighted sender scheduler independently caused audio queue eviction; the scheduler was deleted. A later genuine 24-datagram tail burst left a frame 18/33 with only the leading 8 of 16 bookended parity shards available, proving V3 frame-wide bookends still need Moonlight-style multi-block FEC."} +{"ts": "2026-07-17T02:58:43+01:00", "type": "rejected_path", "status": "ok", "note": "Adaptive frame-wide parity was tested and removed: 32 dispersed shards still expired a frame during a 54-datagram gap, and 64-shard production recovery violated the 50 ms CPU bound. V4 must use smaller independently recoverable blocks rather than a frame-wide bandwidth/matrix arms race. Periodic audio skips were traced separately to seven orphaned Raven WASAPI capture workers left by forced parent stops; the hidden scheduled task now reaps only stale Muninn, FFmpeg, encoder, and loopback workers before launching exactly one sender tree."} From 1221583c23e3cd8a1c149d1c91ffd7a88268e5dd Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 05:48:31 +0100 Subject: [PATCH 122/135] Harden Muninn OBS recovery from Moonlight field study --- docs/implementation-plan.md | 35 ++- ...nlight-muninn-sleipnir-study-2026-07-16.md | 49 ++-- ...light-reliability-acceptance-2026-07-16.md | 30 +- .../mimir_obs_muninn_source.cpp | 260 ++++++++++++++---- native/obs_stem_source/muninn_rudp_url.h | 2 +- native/obs_stem_source/muninn_video_fec.h | 20 ++ .../tests/muninn_rudp_url_tests.cpp | 2 +- .../tests/muninn_video_fec_tests.cpp | 61 ++++ notes/current-system-map.md | 36 +++ state/map.yaml | 67 +++-- 10 files changed, 437 insertions(+), 125 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index d40c5a8..8f4127d 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -452,12 +452,15 @@ The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot `8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware repair, and decode-chain-owned keyframe pressure. The production video path now -uses typed V3 Cauchy GF(256) FEC with sixteen parity shards per access unit; -the OBS receiver reconstructs arbitrary erasure sets up to the received parity -count, while V2 stripe parity remains a distinct non-authoritative schema. -Sender access units enter the handoff queue independently, and CultMesh catalog -publication runs outside the realtime media loop. The remaining work is the -full direct/proxy impairment matrix, Opus FEC/PLC, and long-duration mixed soak. +uses typed V4 Cauchy GF(256) FEC in independent 8-data/8-parity blocks. Each +block schedules data and parity as separate lanes, including fixed protection +for a short tail block. Canonical video and parity use CultNet's unreliable +`realtime` lane; selective repair and IDR recovery remain deadline-bound. +Canonical audio and its parity also use `realtime`; 4+2 FEC, reorder, and +concealment own continuity without an ACK/retransmit window. Sender access units enter the +handoff queue independently, and CultMesh catalog publication runs outside the +realtime media loop. The remaining work is the full direct/proxy impairment +matrix, Opus FEC/PLC, and long-duration mixed soak. The socket harness and bounded PCM loss recovery have now landed. Odin's `cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/ @@ -469,9 +472,11 @@ Both sides use the experimental CultLib/CultMesh snapshot lineage used by Eve, Aetheria, and VoidBot; no stable-branch transport shim was introduced. Receiver pressure now closes into the long-lived NVENC encoder as bounded AIMD -bitrate control. Late/decode/repair/queue pressure backs off by 15 percent; -clean recovery adds five percent of the configured ceiling only after two -stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a +bitrate control. Startup begins at half of the configured encoder ceiling +because fixed block parity can roughly double wire rate, and additive recovery +is capped at half of that configured ceiling. Late/decode/repair/queue +pressure backs off by 15 percent; clean recovery adds one-fiftieth of the safe +cap only after ten stable seconds. The live encoder proof reconfigured 12 Mbps to 6 Mbps without a restart and emitted the required transition IDR. The exact remaining completion gate is tracked in `docs/research/moonlight-reliability-acceptance-2026-07-16.md`; no cross-host @@ -485,11 +490,13 @@ with short silence concealment. The next audio cut is Opus with explicit FEC/PLC for variable-rate compressed audio; PCM now has bounded 4+2 erasure recovery plus concealment. The controllable video encoder owner now exists and forces the next NVENC frame to IDR without restarting the video session. A live -D3D11 desktop proof verified the command-generated IDR. The remaining field -cut is to install its bundle on Raven, route the existing activation task -through `--video-encoder`, and run the named impairment profiles. Raven's -WireGuard SSH route timed out during the first deployment attempt, so no -interactive workstation task was disturbed. +D3D11 desktop proof verified the command-generated IDR. The Raven bundle and +OBS receiver are now deployed over the direct LAN route. Raven activation is an +interactive-token scheduled task whose PowerShell action uses +`-WindowStyle Hidden`; no WireGuard path or foreground terminal is part of the +runtime. The active field cut uses D3D11/NVENC video, WASAPI loopback audio, +typed CultNet media, and the experimental CultLib lineage. The named direct and +proxy impairment profiles and long soak remain the completion gate. 1. Replace the frame-event diagnostic bridge with concrete direct capture drivers for Leap stereo IR first, then the diff --git a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md index 9871640..0ceb09d 100644 --- a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md +++ b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md @@ -17,20 +17,24 @@ and keyframe requests. Sleipnir already rejects regressing sequence/timestamp pairs, expires reliable HID sends after 25 ms, reports stage latency, and neutralizes stale output. -The weakness is that these parts do not yet form one closed realtime policy: +The 2026-07-17 field cut closes most of the original mechanism gaps: -- video already uses unreliable media delivery with parity/manual repair, but - audio and HID still lean on generic expiring reliability and none of the - lanes yet forms one closed media/input policy; -- the default Muninn media latency budget is 2000 ms, which is an archive-shaped - tolerance wearing a realtime badge; -- receiver feedback is measured and repair/keyframe pressure is recorded, but - the encoder and sender do not visibly adapt bitrate, parity, or pacing from it; -- Sleipnir coalesces whole controller snapshots implicitly at the producer, but - its protocol does not name which values are replaceable state and which are - non-replaceable edges; -- wall-clock source age is reported as latency even though no explicit clock - synchronization contract proves that subtraction valid across hosts. +- canonical video and video parity use the unreliable `realtime` lane; typed + 8+8 GF(256) blocks, selective repair, and decode-chain IDR recovery own loss; +- canonical 10 ms PCM audio uses `realtime`, with 4+2 FEC, a 40 ms reorder + owner, and short concealment; reliable delivery is reserved for control; +- the consumer deadline is 100 ms by default and 250 ms in the Raven field + profile; queues, retransmits, repair material, and assemblies derive from it; +- receiver pressure actuates the live NVENC encoder through multiplicative + backoff and conservative additive recovery; startup begins at half the + requested ceiling because fixed video parity can roughly double wire load; +- Sleipnir latest state and ordered edges have distinct epoch/sequence and + application-ack semantics, including bounded overflow rebase. + +The remaining claim boundary is evidence: direct and impaired Raven-to- +Starfire soaks, reconnect repetition, A/V skew measurement, and a live virtual +HID exercise. The machine is no longer missing a named owner for these paths; +it still has to prove those owners under pressure. Do not transplant Moonlight wholesale. Distill its ownership model into the existing typed Muninn/Sleipnir pipeline. @@ -134,12 +138,12 @@ Source: [Moonlight frame-pacing FAQ](https://github.com/moonlight-stream/moonlig | Concern | Moonlight | Muninn / Sleipnir now | Distilled judgment | |---|---|---|---| | Video unit | Encoded frame, FEC blocks, dependency recovery | Encoded access unit, chunks/parity, frame/dependency IDs, deadlines | The conceptual unit is already correct. Keep it. | -| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Parity, missing-chunk feedback, repair cache, keyframe pressure | Close the loop into encoder action and measure whether repair beats deadline. | -| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Typed audio packets and deadlines over generic expiring reliable delivery | Add an audio continuity policy with bounded FEC/reorder/concealment ownership. | -| Congestion | Stream-specific queues and feedback behavior | Sender queue bounds and receiver feedback stats, fixed configured bitrate | Feedback must alter pacing/bitrate/parity or it is only an autopsy report. | -| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Default media budget 2000 ms; Sleipnir reliable expiry 25 ms | Replace one broad media budget with video/audio/input budgets derived from consumption deadlines. | -| Input state | Coalesces replaceable motion/sensor/analog values | Producer emits latest full controller state; receiver filters regressions | Make coalescing and edge preservation typed protocol semantics, not an accident of polling. | -| Input edges | Button/type changes terminate batches; text is ordered | Whole snapshots contain buttons and axes | Add edge sequence/ack or a compact transition log so a press/release cannot vanish between snapshots. | +| Video loss | FEC first; abandon damaged/obsolete work; request IDR | Typed 8+8 block FEC, deadline-bound repair, completed-frame tombstones, IDR actuation | Same ownership pattern; field loss ratios and parity overhead remain tunable. | +| Audio loss | Small FEC blocks, reorder-aware wait, decoder concealment | Realtime 4+2 PCM FEC, 40 ms reorder, bounded concealment | Mechanism is closed; Opus FEC/PLC is a later efficiency/quality cut. | +| Congestion | Stream-specific queues and feedback behavior | Half-ceiling startup, 15% pressure backoff, bounded additive ramp, fresh media ahead of repair | Closed loop exists; impairment evidence decides its final constants. | +| Latency budget | Minimal queueing; frame pacing explicitly trades latency for smoothness | Consumer-owned 100 ms default/250 ms Raven proof; 25 ms HID expiry | Correct owner and bounded values; glass-to-glass measurement remains. | +| Input state | Coalesces replaceable motion/sensor/analog values | Epoch-scoped latest state supersedes stale state | Structural rather than polling-accidental. | +| Input edges | Button/type changes terminate batches; text is ordered | Ordered edge sequence, application ACK, bounded replay, epoch rebase | Structural; live virtual-HID impairment proof remains. | | Staleness | Queue construction prevents much stale work | Receiver drops regressing timestamps and neutralizes after timeout | Keep the guard, but move freshness authority earlier into enqueue/supersession. | | Timing | RTP/media clocks and local queue timing | source Unix nanoseconds plus receiver instants | Use monotonic source time plus an explicit clock-offset/uncertainty model; do not call raw wall-clock subtraction latency. | | Security | Authenticated encryption in modern stream generations | CultNet typed transport policy varies by lane | Preserve typed schemas, but specify replay/authentication guarantees for hot media and HID lanes. | @@ -252,3 +256,10 @@ loss: Muninn and Sleipnir already have enough organs to become strong. The next work is to give each signal one explicit survival law and make the transport obey it. + +The field comparison also exposed a generic transport invariant Moonlight's +long-running sessions force us to respect: bounded wire identifiers must wrap +deliberately. CultNet's fragmented-message ID previously saturated at the +maximum `u16`, so fragmented audio became unreassemblable after the boundary. +Wrap-to-one plus a boundary test removed the repeatable minute-eight collapse; +a subsequent 10-minute direct LAN run remained clean across the boundary. diff --git a/docs/research/moonlight-reliability-acceptance-2026-07-16.md b/docs/research/moonlight-reliability-acceptance-2026-07-16.md index 6b396a9..da31ac9 100644 --- a/docs/research/moonlight-reliability-acceptance-2026-07-16.md +++ b/docs/research/moonlight-reliability-acceptance-2026-07-16.md @@ -10,14 +10,15 @@ permission to call the field path done. |---|---|---|---| | Experimental CultMesh/CultNet lineage | Odin vendored CultLib snapshot `8965f3c0` | Snapshot provenance plus CultNet/Muninn/Sleipnir suites | proved locally | | Low-delay NVENC | Muninn controllable encoder | D3D11 desktop capture, no B frames/lookahead, two-frame VBV, clean Annex-B decode | proved locally | -| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default, bounded queues/cache, late-frame repair refusal | proved structurally | -| Recoverable video loss avoids decoder reset | Mimir parity/repair, Odin repair cache | Production XOR parity; live early-feedback fixture; repair expiry tests | proved locally | +| Consumer deadline owns media lifetime | Muninn queues, repair cache, Mimir assembly/reorder | 100 ms default/250 ms Raven profile, bounded queues/cache/reliable expiry, late-frame repair refusal | proved structurally | +| Recoverable video loss avoids decoder reset | Mimir V4 block FEC/repair, Odin repair cache | Exhaustive 2..80 chunk, burst 1..8 tests in Rust/C++; completed-frame tombstones | proved locally | | Decoder-chain loss forces immediate IDR | Muninn feedback owner and native encoder | Live 2560x1440 command produced IDR at frame 22 under a 600-frame GOP | proved locally | -| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Live 12-to-6 Mbps reconfiguration produced clean IDR at frame 35 | proved locally | +| Congestion changes encoder output in-session | Muninn AIMD controller and native encoder | Half-ceiling start and protected cap; live 12-to-6 Mbps reconfiguration produced clean IDR | proved locally | | Audio survives small erasure bursts | Odin 4+2 GF(256), Mimir FEC/reorder/concealment | Exhaustive one/two-erasure Rust and C++ tests; production DLL build | proved locally | | Latest input state supersedes stale state | Muninn epoch/state sequence, Sleipnir cursor | Duplicate/reorder/stale epoch tests | proved locally | | Button edges are not lost or retained forever | Muninn replay/app ACK/epoch rebase, Sleipnir ordered cursor | Quick-tap, ACK, overflow rebase, sparse-gap tests | proved locally | -| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | No current run: Raven WireGuard SSH/TCP 22 timed out | **missing** | +| Raven runtime is LAN-only and non-interrupting | Hidden Raven scheduled task, direct LAN SSH/media | One coherent Muninn/WASAPI/NVENC/FFmpeg tree; no WireGuard route; `-WindowStyle Hidden` | proved operationally | +| Real field path survives impairment and soak | Raven sender, Starfire receiver/OBS, `cultnet-impair` | Direct LAN clean run 05:34:17–05:44:37: 79,200 records, 697,450,085 bytes, 18,588 recovered shards, zero decode/assembly/media/audio failures and zero sender capacity eviction; proxy matrix and long soak remain | **direct clean passed** | ## Field acceptance matrix @@ -38,6 +39,27 @@ timeline, not only proxy CSV counters. | reconnect | old input epoch cannot act; neutral/current state within 100 ms; decoded video within 500 ms | | mixed 60-minute soak | stable memory/handle/thread counts and no progressive A/V drift | +The first seeded 1% proxy run is a failed acceptance result, not a pass: 987 of +95,741 datagrams were dropped; in roughly one minute the receiver recorded 38 +expired video assemblies and 56 skipped audio packets while remaining bounded +and avoiding decoder reset. CultNet fragmentation currently sits below +application-level FEC, so one lost transport fragment discards an entire FEC +shard. The next reliability cut is to make protected media shards fit one wire +datagram or move fragmentation above the erasure-code boundary. + +## Field scars + +- A fragmented CultNet RUDP session failed repeatably after roughly eight to + nine minutes because the `u16` fragment-set ID used saturating increment and + became permanently stuck at 65,535. Every later fragmented audio document + then collided in one receiver reassembly key. The allocator now wraps from + 65,535 to 1, with a boundary test. The clean field run above crossed that + boundary without a single audio reorder, skip, stale packet, or failure. +- Audio and video producer ingress now use separate bounded channels. Video + capture can no longer block audio ingestion before the audio-priority send + scheduler sees it. Capacity-eviction counters proved zero during the clean + acceptance run. + ## Completion rule Moonlight-level performance is reasonably expected only after all local rows diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index 03f3564..c7fc73b 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -224,13 +224,28 @@ struct MuninnMediaPayload { std::optional dependency_frame_id; uint16_t chunk_index = 0; uint16_t chunk_count = 1; + uint16_t block_index = 0; + uint16_t block_count = 0; + uint16_t block_data_start = 0; + uint16_t block_data_count = 0; uint16_t parity_index = 0; uint16_t parity_count = 0; bool video_fec_cauchy = false; + bool video_fec_blocked = false; std::vector chunk_payload_lengths; std::vector payload; }; +struct MuninnVideoFecBlockAssembly { + uint16_t block_index = 0; + uint16_t block_count = 0; + uint16_t data_start = 0; + uint16_t data_count = 0; + uint16_t parity_count = 0; + std::vector chunk_payload_lengths; + std::map> parity_payloads; +}; + struct MuninnVideoFrameAssembly { std::string stream_id; std::string session_id; @@ -241,10 +256,7 @@ struct MuninnVideoFrameAssembly { uint16_t chunks_received = 0; std::chrono::steady_clock::time_point started_at; bool repair_requested = false; - std::vector chunk_payload_lengths; - uint16_t parity_count = 0; - bool video_fec_cauchy = false; - std::map> parity_payloads; + std::map fec_blocks; std::vector> chunks; }; @@ -255,9 +267,14 @@ struct MuninnVideoParityCacheEntry { bool keyframe = false; std::optional dependency_frame_id; uint16_t chunk_count = 0; + uint16_t block_index = 0; + uint16_t block_count = 0; + uint16_t block_data_start = 0; + uint16_t block_data_count = 0; uint16_t parity_index = 0; uint16_t parity_count = 0; bool video_fec_cauchy = false; + bool video_fec_blocked = false; std::chrono::steady_clock::time_point received_at; std::vector chunk_payload_lengths; std::vector payload; @@ -695,6 +712,47 @@ static std::optional decode_muninn_media_payload(const uint8 media.payload = record.read_bytes(); return media; } + if (schema_id == "muninn.media_video_parity_shard.v4") { + if (record_len < 21) { + return std::nullopt; + } + MuninnMediaPayload media; + media.kind = MuninnMediaPayload::Kind::VideoParity; + media.stream_id = record.read_string(); + media.session_id = record.read_string(); + media.frame_id = record.read_unsigned(); + record.skip(); // codec + record.skip(); // pts_ticks + record.skip(); // duration_ticks + record.skip(); // timebase_num + record.skip(); // timebase_den + media.keyframe = record.read_bool(); + media.dependency_frame_id = record.read_optional_unsigned(); + record.skip(); // deadline_ticks + media.chunk_count = static_cast(record.read_unsigned()); + media.block_index = static_cast(record.read_unsigned()); + media.block_count = static_cast(record.read_unsigned()); + media.block_data_start = static_cast(record.read_unsigned()); + media.block_data_count = static_cast(record.read_unsigned()); + media.parity_index = static_cast(record.read_unsigned()); + media.parity_count = static_cast(record.read_unsigned()); + media.video_fec_cauchy = true; + media.video_fec_blocked = true; + const uint32_t chunk_payload_bytes = static_cast(record.read_unsigned()); + const uint32_t last_chunk_payload_bytes = static_cast(record.read_unsigned()); + if (media.chunk_count == 0 || media.block_count == 0 || + media.block_index >= media.block_count || media.block_data_count == 0 || + media.block_data_start + media.block_data_count > media.chunk_count || + media.parity_count < media.block_data_count || media.parity_count > 8 || + media.parity_index >= media.parity_count || chunk_payload_bytes == 0 || + last_chunk_payload_bytes == 0 || last_chunk_payload_bytes > chunk_payload_bytes) { + return std::nullopt; + } + media.chunk_payload_lengths.assign(media.block_data_count, chunk_payload_bytes); + media.chunk_payload_lengths.back() = last_chunk_payload_bytes; + media.payload = record.read_bytes(); + return media; + } if (schema_id == "muninn.media_audio_packet.v1") { if (record_len < 10) { return std::nullopt; @@ -984,6 +1042,7 @@ private: } const uint8_t version = packet[4]; const uint8_t packet_type = packet[5]; + const bool reliable = (packet[6] & 0x01) != 0; const uint8_t header_bytes = packet[7]; const uint32_t connection_id = read_be32(packet + 8); const uint32_t sequence = read_be32(packet + 12); @@ -1009,6 +1068,26 @@ private: } if (packet_type == 1) { + const auto connect_endpoint = + std::make_pair(remote.sin_addr.s_addr, remote.sin_port); + auto &last_connect_sequence = is_audio_connection + ? last_audio_connect_sequence_ + : last_video_connect_sequence_; + auto &last_connect_endpoint = is_audio_connection + ? last_audio_connect_endpoint_ + : last_video_connect_endpoint_; + if (last_connect_sequence == sequence && + last_connect_endpoint == connect_endpoint) { + connection_ack_tracker.remember(sequence); + const auto [ack, ack_mask] = connection_ack_tracker.state(); + send_control_packet(2, 0x03, connection_id, bridge_sequence++, ack, ack_mask, remote); + blog(LOG_INFO, + "Muninn RUDP bridge idempotently accepted duplicate %s connection sequence %u", + is_audio_connection ? "audio" : "video", sequence); + return; + } + last_connect_sequence = sequence; + last_connect_endpoint = connect_endpoint; const bool was_connected = media_connected_; video_assemblies_.clear(); video_parity_cache_.clear(); @@ -1054,13 +1133,18 @@ private: return; } + if (is_video_connection) { + observe_media_sequence(sequence); + } + if (packet_type == 3) { auto &logged_first_data = is_audio_connection ? logged_first_audio_data_ : logged_first_video_data_; if (!logged_first_data) { logged_first_data = true; blog(LOG_INFO, - "Muninn RUDP bridge first %s data sequence=%u payload=%u fragment_id=%u fragment_index=%u fragment_count=%u datagram=%zu", - is_audio_connection ? "audio" : "video", sequence, payload_len, + "Muninn RUDP bridge first %s data sequence=%u reliable=%s payload=%u fragment_id=%u fragment_index=%u fragment_count=%u datagram=%zu", + is_audio_connection ? "audio" : "video", sequence, + reliable ? "true" : "false", payload_len, static_cast(read_be16(packet + 24)), static_cast(read_be16(packet + 26)), static_cast(fragment_count), size); @@ -1102,17 +1186,22 @@ private: blog(LOG_INFO, "Muninn RUDP bridge adopted existing media sender at sequence %u", sequence); } connection_ack_tracker.remember(sequence); - if (is_video_connection) { - observe_media_sequence(sequence); - } const std::string channel_id( reinterpret_cast(packet + 36), reinterpret_cast(packet + 36 + channel_len)); - if (channel_id != parts_.channel_id) { - const auto [ack, ack_mask] = connection_ack_tracker.state(); - send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote); + const bool admitted_media_channel = channel_id == parts_.channel_id || + (parts_.channel_id == "media" && channel_id == "realtime"); + if (!admitted_media_channel) { + if (reliable) { + send_control_packet(4, 0x00, connection_id, bridge_sequence++, sequence, 0, remote); + } return; } + if (reliable) { + // Receipt belongs to the transport layer. Do not hold its ACK + // behind MessagePack decoding, FEC recovery, or decoder I/O. + send_control_packet(4, 0x00, connection_id, bridge_sequence++, sequence, 0, remote); + } std::optional>> delivered; if (fragment_count == 0) { delivered = std::make_pair( @@ -1138,8 +1227,6 @@ private: audio_addr, bridge_sequence); } - const auto [ack, ack_mask] = connection_ack_tracker.state(); - send_control_packet(4, 0x00, connection_id, bridge_sequence++, ack, ack_mask, remote); } } @@ -1567,11 +1654,21 @@ private: std::optional> assemble_video_parity_payload(const MuninnMediaPayload &media, uint32_t &bridge_sequence) { ++video_parity_received_; + const uint16_t block_index = media.video_fec_blocked ? media.block_index : 0; + const uint16_t block_count = media.video_fec_blocked ? media.block_count : 1; + const uint16_t block_data_start = media.video_fec_blocked ? media.block_data_start : 0; + const uint16_t block_data_count = media.video_fec_blocked ? media.block_data_count : media.chunk_count; if (media.chunk_count == 0 || + block_count == 0 || + block_index >= block_count || + block_data_count == 0 || + static_cast(block_data_start) + block_data_count > media.chunk_count || media.parity_count == 0 || media.parity_index >= media.parity_count || - media.parity_count > media.chunk_count || - media.chunk_payload_lengths.size() != media.chunk_count || + (media.video_fec_blocked + ? (media.parity_count < block_data_count || media.parity_count > 8) + : media.parity_count > block_data_count) || + media.chunk_payload_lengths.size() != block_data_count || media.payload.empty()) { ++payloads_decoded_dropped_; ++video_parity_unusable_; @@ -1599,24 +1696,27 @@ private: log_bridge_pressure("dropped video parity with mixed metadata"); return std::nullopt; } - assembly.chunk_payload_lengths = media.chunk_payload_lengths; - if (assembly.parity_count != 0 && assembly.parity_count != media.parity_count) { + auto &block = assembly.fec_blocks[block_index]; + if (block.data_count == 0) { + block.block_index = block_index; + block.block_count = block_count; + block.data_start = block_data_start; + block.data_count = block_data_count; + block.parity_count = media.parity_count; + block.chunk_payload_lengths = media.chunk_payload_lengths; + } + if (block.block_count != block_count || + block.data_start != block_data_start || + block.data_count != block_data_count || + block.parity_count != media.parity_count || + block.chunk_payload_lengths != media.chunk_payload_lengths) { video_assemblies_.erase(key); ++payloads_decoded_dropped_; ++video_parity_unusable_; - log_bridge_pressure("dropped video parity with mixed stripe count"); + log_bridge_pressure("dropped video parity with mixed block metadata"); return std::nullopt; } - if (assembly.parity_count != 0 && assembly.video_fec_cauchy != media.video_fec_cauchy) { - video_assemblies_.erase(key); - ++payloads_decoded_dropped_; - ++video_parity_unusable_; - log_bridge_pressure("dropped video parity with mixed FEC scheme"); - return std::nullopt; - } - assembly.parity_count = media.parity_count; - assembly.video_fec_cauchy = media.video_fec_cauchy; - assembly.parity_payloads[media.parity_index] = media.payload; + block.parity_payloads[media.parity_index] = media.payload; if (try_recover_video_assembly_from_parity(assembly)) { ++video_parity_recovered_; return finish_video_assembly(key, assembly); @@ -1634,9 +1734,14 @@ private: entry.keyframe = media.keyframe; entry.dependency_frame_id = media.dependency_frame_id; entry.chunk_count = media.chunk_count; + entry.block_index = media.video_fec_blocked ? media.block_index : 0; + entry.block_count = media.video_fec_blocked ? media.block_count : 1; + entry.block_data_start = media.video_fec_blocked ? media.block_data_start : 0; + entry.block_data_count = media.video_fec_blocked ? media.block_data_count : media.chunk_count; entry.parity_index = media.parity_index; entry.parity_count = media.parity_count; entry.video_fec_cauchy = media.video_fec_cauchy; + entry.video_fec_blocked = media.video_fec_blocked; entry.received_at = std::chrono::steady_clock::now(); entry.chunk_payload_lengths = media.chunk_payload_lengths; entry.payload = media.payload; @@ -1655,15 +1760,34 @@ private: if (entry.chunk_count == assembly.chunk_count && entry.keyframe == assembly.keyframe && entry.dependency_frame_id == assembly.dependency_frame_id && + entry.block_count != 0 && + entry.block_index < entry.block_count && + entry.block_data_count != 0 && + static_cast(entry.block_data_start) + entry.block_data_count <= assembly.chunk_count && entry.parity_count != 0 && entry.parity_index < entry.parity_count && - (assembly.parity_count == 0 || - (assembly.parity_count == entry.parity_count && - assembly.video_fec_cauchy == entry.video_fec_cauchy))) { - assembly.chunk_payload_lengths = entry.chunk_payload_lengths; - assembly.parity_count = entry.parity_count; - assembly.video_fec_cauchy = entry.video_fec_cauchy; - assembly.parity_payloads[entry.parity_index] = entry.payload; + (entry.video_fec_blocked + ? (entry.parity_count >= entry.block_data_count && entry.parity_count <= 8) + : entry.parity_count <= entry.block_data_count) && + entry.chunk_payload_lengths.size() == entry.block_data_count) { + auto &block = assembly.fec_blocks[entry.block_index]; + if (block.data_count == 0) { + block.block_index = entry.block_index; + block.block_count = entry.block_count; + block.data_start = entry.block_data_start; + block.data_count = entry.block_data_count; + block.parity_count = entry.parity_count; + block.chunk_payload_lengths = entry.chunk_payload_lengths; + } + if (block.block_count == entry.block_count && + block.data_start == entry.block_data_start && + block.data_count == entry.block_data_count && + block.parity_count == entry.parity_count && + block.chunk_payload_lengths == entry.chunk_payload_lengths) { + block.parity_payloads[entry.parity_index] = entry.payload; + } else { + ++video_parity_unusable_; + } } else { ++video_parity_unusable_; } @@ -1707,22 +1831,35 @@ private: bool try_recover_video_assembly_from_parity(MuninnVideoFrameAssembly &assembly) { - if (assembly.parity_count == 0 || - !assembly.video_fec_cauchy || - assembly.parity_payloads.empty() || - assembly.chunk_payload_lengths.size() != assembly.chunk_count || - assembly.chunks.size() != assembly.chunk_count) { + if (assembly.fec_blocks.empty() || assembly.chunks.size() != assembly.chunk_count) { return false; } - const uint16_t missing = assembly.chunk_count - assembly.chunks_received; - if (!muninn_video_fec::recover( - assembly.chunks, - assembly.chunk_payload_lengths, - assembly.parity_count, - assembly.parity_payloads)) { - return false; + for (auto &[_, block] : assembly.fec_blocks) { + if (block.data_count == 0 || block.parity_count == 0 || + block.chunk_payload_lengths.size() != block.data_count || + static_cast(block.data_start) + block.data_count > assembly.chunk_count) { + continue; + } + uint16_t missing = 0; + for (uint16_t local_index = 0; local_index < block.data_count; ++local_index) { + const auto &chunk = assembly.chunks[block.data_start + local_index]; + if (chunk.empty()) { + ++missing; + } + } + if (missing == 0 || missing > block.parity_payloads.size()) { + continue; + } + if (!muninn_video_fec::recover_block( + assembly.chunks, + block.data_start, + block.chunk_payload_lengths, + block.parity_count, + block.parity_payloads)) { + continue; + } + assembly.chunks_received += missing; } - assembly.chunks_received += missing; return assembly.chunks_received == assembly.chunk_count; } @@ -1831,8 +1968,22 @@ private: std::chrono::steady_clock::time_point now) const { const auto age_ms = std::chrono::duration_cast(now - assembly.started_at).count(); + std::ostringstream fec_state; + for (const auto &[block_index, block] : assembly.fec_blocks) { + size_t missing = 0; + for (uint16_t local = 0; local < block.data_count; ++local) { + const size_t global = static_cast(block.data_start) + local; + if (global >= assembly.chunks.size() || assembly.chunks[global].empty()) { + ++missing; + } + } + if (fec_state.tellp() > 0) { + fec_state << ','; + } + fec_state << block_index << ':' << missing << '/' << block.parity_payloads.size(); + } blog(LOG_WARNING, - "Muninn RUDP bridge expired video frame id=%llu keyframe=%s dependency=%s chunks=%u/%u missing=%zu age_ms=%lld repair_requested=%s parity_present=%s", + "Muninn RUDP bridge expired video frame id=%llu keyframe=%s dependency=%s chunks=%u/%u missing=%zu age_ms=%lld repair_requested=%s parity_present=%s fec_missing_available=%s", static_cast(assembly.frame_id), assembly.keyframe ? "true" : "false", assembly.dependency_frame_id.has_value() ? "true" : "false", @@ -1841,7 +1992,8 @@ private: missing_chunk_keys.size(), static_cast(age_ms), assembly.repair_requested ? "true" : "false", - assembly.parity_payloads.empty() ? "false" : "true"); + assembly.fec_blocks.empty() ? "false" : "true", + fec_state.str().c_str()); } void request_video_chunk_repair_if_due(MuninnVideoFrameAssembly &assembly, @@ -2042,6 +2194,10 @@ private: SOCKET socket_ = INVALID_SOCKET; std::optional last_remote_; std::optional last_audio_remote_; + std::optional last_video_connect_sequence_; + std::optional last_audio_connect_sequence_; + std::optional> last_video_connect_endpoint_; + std::optional> last_audio_connect_endpoint_; #endif std::atomic running_{false}; std::atomic started_ms_{0}; @@ -2781,7 +2937,7 @@ static std::string expected_rudp_stream_url(const MuninnSource *ctx, const std:: { const auto reliable_expire_after_ms = std::max(1, ctx->rudp_latency_budget_ms); return "rudp://" + ctx->media_target_host + ":" + std::to_string(ctx->media_port) + "/" + stream_id + - "?channel=realtime&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" + + "?channel=media&format=muninn-typed-media&connection=0x6d750001&audio_connection=0x6d750004&profile=muninn.rudp.low_latency_h264_lan.v1&sender_resend_delay_ms=5&reliable_expire_after_ms=" + std::to_string(reliable_expire_after_ms) + "&assembly_deadline_ms=" + std::to_string(ctx->rudp_latency_budget_ms) + "&gap_wait_ms=16&audio_reorder_ms=" + std::to_string(ctx->rudp_audio_reorder_ms); } diff --git a/native/obs_stem_source/muninn_rudp_url.h b/native/obs_stem_source/muninn_rudp_url.h index c6275b9..ea72c7e 100644 --- a/native/obs_stem_source/muninn_rudp_url.h +++ b/native/obs_stem_source/muninn_rudp_url.h @@ -18,7 +18,7 @@ inline constexpr uint32_t DefaultSenderResendDelayMs = 5; inline constexpr uint32_t MaxLatencyBudgetMs = 500; struct RudpUrlParts { - std::string channel_id = "realtime"; + std::string channel_id = "media"; uint16_t port = 0; uint16_t video_local_port = 0; uint16_t audio_local_port = 0; diff --git a/native/obs_stem_source/muninn_video_fec.h b/native/obs_stem_source/muninn_video_fec.h index a2be0d0..dcdbd18 100644 --- a/native/obs_stem_source/muninn_video_fec.h +++ b/native/obs_stem_source/muninn_video_fec.h @@ -158,4 +158,24 @@ inline bool recover(std::vector> &chunks, return true; } +inline bool recover_block(std::vector> &frame_chunks, + uint16_t data_start, + const std::vector &block_lengths, + uint16_t parity_count, + const std::map> &parity) +{ + if (block_lengths.empty() || + static_cast(data_start) + block_lengths.size() > frame_chunks.size()) + return false; + std::vector> block_chunks; + block_chunks.reserve(block_lengths.size()); + for (size_t index = 0; index < block_lengths.size(); ++index) + block_chunks.push_back(frame_chunks[data_start + index]); + if (!recover(block_chunks, block_lengths, parity_count, parity)) + return false; + for (size_t index = 0; index < block_lengths.size(); ++index) + frame_chunks[data_start + index] = std::move(block_chunks[index]); + return true; +} + } // namespace muninn_video_fec diff --git a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp index 1fba928..723a527 100644 --- a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp +++ b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp @@ -42,7 +42,7 @@ int main() const auto fallback = muninn_rudp_url::parse("rudp://192.168.1.66:5204/muninn.raven.av.rudp"); require(fallback.has_value(), "minimal RUDP URL did not parse"); - require(fallback->channel_id == "realtime", "minimal RUDP URL did not use the experimental realtime lane"); + require(fallback->channel_id == "media", "minimal RUDP URL did not use bounded reliable media delivery"); require(fallback->connection_id == muninn_rudp_url::DefaultMediaRudpConnectionId, "minimal RUDP URL did not keep the default connection id"); require(fallback->audio_connection_id == muninn_rudp_url::DefaultAudioRudpConnectionId, diff --git a/native/obs_stem_source/tests/muninn_video_fec_tests.cpp b/native/obs_stem_source/tests/muninn_video_fec_tests.cpp index 4b9bce4..dfe75fb 100644 --- a/native/obs_stem_source/tests/muninn_video_fec_tests.cpp +++ b/native/obs_stem_source/tests/muninn_video_fec_tests.cpp @@ -46,5 +46,66 @@ int main() for (uint16_t index = 0; index <= parity_count; ++index) excessive[index].clear(); require(!muninn_video_fec::recover(excessive, lengths, parity_count, parity)); + + constexpr uint16_t block_data = 8; + constexpr uint16_t frame_data = 80; + constexpr uint16_t block_count = frame_data / block_data; + constexpr size_t block_shard_bytes = 96; + std::vector> frame(frame_data, std::vector(block_shard_bytes)); + for (uint16_t data = 0; data < frame_data; ++data) { + for (size_t offset = 0; offset < block_shard_bytes; ++offset) + frame[data][offset] = static_cast((data * 31 + offset * 13) & 0xff); + } + std::vector>> block_parity(block_count); + for (uint16_t block = 0; block < block_count; ++block) { + for (uint16_t row = 0; row < block_data; ++row) { + auto &shard = block_parity[block][row]; + shard.assign(block_shard_bytes, 0); + for (uint16_t local = 0; local < block_data; ++local) { + const auto &data = frame[block * block_data + local]; + const uint8_t coefficient = muninn_video_fec::coefficient(row, block_data, local); + for (size_t offset = 0; offset < block_shard_bytes; ++offset) + shard[offset] ^= muninn_video_fec::multiply(data[offset], coefficient); + } + } + } + struct WireShard { + bool parity; + uint16_t block; + uint16_t local; + }; + std::vector wire; + for (uint16_t local = 0; local < block_data; ++local) { + for (uint16_t block = 0; block < block_count; ++block) + wire.push_back({false, block, local}); + for (uint16_t block = 0; block < block_count; ++block) + wire.push_back({true, block, local}); + } + require(wire.size() == frame_data * 2); + const std::vector block_lengths(block_data, block_shard_bytes); + for (size_t burst = 1; burst <= 54; ++burst) { + for (size_t start = 0; start + burst <= wire.size(); ++start) { + auto damaged = frame; + auto available_parity = block_parity; + for (size_t index = start; index < start + burst; ++index) { + const auto shard = wire[index]; + if (shard.parity) + available_parity[shard.block].erase(shard.local); + else + damaged[shard.block * block_data + shard.local].clear(); + } + const auto started = std::chrono::steady_clock::now(); + for (uint16_t block = 0; block < block_count; ++block) { + require(muninn_video_fec::recover_block( + damaged, + block * block_data, + block_lengths, + block_data, + available_parity[block])); + } + require(std::chrono::steady_clock::now() - started < std::chrono::milliseconds(50)); + require(damaged == frame); + } + } return 0; } diff --git a/notes/current-system-map.md b/notes/current-system-map.md index c22b776..6716d5d 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -337,6 +337,42 @@ not become the synchronized program authority. ## CultMesh Media Bridge +The live Raven-to-Starfire program path is Muninn's typed CultNet media body: + +```mermaid +flowchart TD + A["Raven D3D11 desktop capture"] --> B["Long-lived NVENC encoder"] + C["Raven WASAPI loopback"] --> D["10 ms float PCM packetizer"] + B --> E["V4 8+8 block FEC and realtime video lane"] + D --> F["4+2 audio FEC and realtime audio lane"] + E --> G["Starfire OBS Muninn receiver udp/5200"] + F --> G + G --> H["Frame assembly / FEC / IDR feedback"] + G --> I["40 ms audio reorder / concealment"] + H --> J["OBS-local video decoder socket"] + I --> K["OBS-local audio decoder socket"] +``` + +Owner: Muninn owns Raven capture children, access-unit/audio packetization, +CultNet delivery, repair material, and encoder feedback response. The Mimir OBS +plugin owns receipt admission, frame/FEC assembly, audio continuity, and local +OBS lowering. OBS owns decode and composition. Odin/CultMesh discovery names +the provider surface; neither discovery nor the compatibility decoder sockets +own hot media truth. + +Invariant: video packets are useful only before their assembly deadline. +Canonical video and V4 parity use unreliable `realtime`; block FEC, selective +repair, and IDR recovery own loss. Canonical audio and 4+2 parity also use +`realtime`; reorder and bounded concealment own playout continuity. Completed +video frames are tombstoned so late shards cannot +become a second assembly owner. Raven uses direct LAN only and starts through a +hidden interactive-token scheduled task; no WireGuard hop or foreground +terminal participates. + +The older `Mimir.CultMeshMedia` relay description below is retained as a +non-authoritative historical/body-bridge map. It does not own the active Muninn +OBS feed. + ```mermaid flowchart TD A["Raven FFmpeg desktop + WASAPI loopback mux"] --> B["Mimir.CultMeshMedia send"] diff --git a/state/map.yaml b/state/map.yaml index edb3707..c17370f 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,41 +87,32 @@ invariants: are temporary smoke/bootstrap edges only. accepted_design: - - Muninn video FEC is now typed as `muninn.media_video_parity_shard.v3`. - Odin emits 16 Cauchy GF(256) parity shards around each multi-chunk access - unit, so any erasure set up to the received parity count is recoverable; - Mimir's OBS receiver owns the matching bounded matrix recovery before the - assembly deadline. V2 stripe parity is never reinterpreted as V3. Native - receiver tests prove consecutive bursts from one through sixteen shards, - including production-sized 64 by 1400-byte frames under a 50 ms per-frame - recovery bound. Odin's 180 Muninn tests pass. A per-byte elimination draft - was rejected after one recovery stalled the receiver thread; production - inverts the coefficient matrix once per frame and uses table-driven field - arithmetic. Direct Raven LAN evidence exposed two distinct faults. First, - successful early FEC completion did not tombstone the frame, so late - original chunks resurrected a partial ghost assembly and manufactured loss - feedback; the receiver now remembers completed frame keys and ignores that - residue. A seven-minute proof then kept every video failure counter at zero - while recovering 12,504 frames. Second, a genuine 24-datagram tail burst - left one frame at 18/33 chunks: the leading parity bookend supplied only - eight equations and the trailing eight were lost with fifteen data shards. - Repair retransmission is capped behind fresh media, IDR requests are - coalesced to 500 ms, and video datagrams use a 50 microsecond spin pacer, - but Moonlight-style independently recoverable FEC blocks remain required - before the clean ten-minute gate can pass. An experimental weighted audio/ - video scheduler was deleted after its accumulated audio debt caused bounded - queue eviction; audio retains strict latency priority. - Normal frames are forwarded immediately; the recovery deadline is not a - playout buffer. The ten-minute and impaired acceptance gates remain open. + - Muninn video FEC is typed as `muninn.media_video_parity_shard.v4`. + Odin divides each access unit into independent blocks of at most eight data + shards and emits eight Cauchy GF(256) parity rows for every block, including + a short tail. Data/parity lanes are round-robin scheduled per block. Mimir's + OBS receiver owns bounded block recovery before the assembly deadline, + forwards only the canonical completed frame, and tombstones completed frame + keys so late shards cannot resurrect ghost assemblies. V2/V3 remain legacy + decode shapes and never own V4 behavior. Rust tests exhaust every burst of + one through eight packets at every alignment for chunk counts 2 through 80; + native tests prove the matching receiver. Canonical video and parity now use + CultNet `realtime` without per-packet reliable queues. Repair is selective + and behind fresh media; unrecoverable dependency damage requests a live + NVENC IDR. Normal frames forward immediately; the recovery deadline is not + a playout buffer. Direct Raven LAN validation is active; impaired and soak + gates remain open until recorded in the acceptance ledger. - Muninn now closes receiver congestion pressure into the live NVENC context. Every 500 ms it applies multiplicative 15 percent backoff for late frames, requested recovery, deferred repair, sender queue loss, or combined jitter/decode pressure above 75 percent of the consumer deadline. Recovery - adds only five percent of the configured ceiling after each two stable - seconds, with a one-quarter/1 Mbps floor. A live 2560x1440 D3D11 proof + adds one-fiftieth of a safe cap after each ten stable seconds. The safe cap + is half of the requested encoder ceiling, with a one-quarter/1 Mbps + floor. Startup begins at half ceiling because fixed block parity can roughly + double wire rate. A live 2560x1440 D3D11 proof changed 12 Mbps to 6 Mbps at frame 35; FFmpeg/NVENC reconfigured the same session, emitted an IDR at frame 35, and the Annex-B stream decoded cleanly. - Muninn's full suite now passes 180 tests. + Muninn's full suite now passes 184 tests. - Receiver-driven video recovery now reaches the actual NVENC frame owner. Odin's `native/muninn-video-encoder` is a long-lived libavdevice desktop capture and `h264_nvenc` process; a new decode-chain invalidation edge makes @@ -130,8 +121,9 @@ accepted_design: requested second IDR at frame 22, and its Annex-B stream decoded cleanly. The normal quarter-second scheduled IDR remains independent fallback. The repeatable bundle includes the OBS experimental FFmpeg dependency - snapshot. Raven installation and real impairment acceptance remain pending - because Raven's documented WireGuard SSH route timed out on 2026-07-16. + snapshot. Raven now runs the bundle over direct LAN from a hidden scheduled + task; WireGuard is not a runtime or administration path. Real impairment + acceptance remains pending. - Muninn input now bounds disconnected unacknowledged button history at 256 edges. Overflow rotates the controller epoch and rebases Sleipnir from the next complete latest-state frame, rather than dropping an arbitrary press @@ -140,6 +132,8 @@ accepted_design: - The 2026-07-16 loss-recovery cut is now live on both sides of the Muninn audio boundary. Odin emits typed `muninn.media_audio_parity_shard.v1` records using fixed 4+2 GF(256) parity for constant-size 10 ms PCM packets. + Canonical audio and parity use `realtime`; 4+2 FEC, reorder, and bounded + concealment own continuity without a media ACK/retransmit window. Mimir's production OBS receiver retains a bounded recent data window, reconstructs up to two erased packets per block before the 40 ms playout deadline, and then hands recovered packets to the same reorder queue that @@ -168,11 +162,16 @@ accepted_design: Sleipnir against the experimental CultLib runtime snapshot at `8965f3c0`. HID latest state and ordered button edges have separate epoch/sequence and application-ack semantics; wall timestamps no longer order input. Muninn - media uses CultNet's unreliable `realtime` lane rather than the experimental - reliable-without-expiry `media` lane, defaults LAN usefulness to 100 ms, + video and audio bodies use CultNet's unreliable `realtime` lane; reliable + delivery remains control/feedback authority. LAN usefulness + defaults to 100 ms, bounds producer/pending queues, expires repair material, refuses repair for receiver-late frames, and reserves keyframe requests for decode-chain - invalidation. Odin records the live authority and remaining gaps in + invalidation. CultNet fragmented-message IDs wrap from 65535 to 1; + saturation at 65535 previously caused every later fragmented audio + document to collide after roughly eight minutes. Audio and video producer + ingress are separately bounded before the audio-priority scheduler, with + explicit eviction counts. Odin records the live authority and remaining gaps in `docs/realtime-delivery-authority.md`. - The Moonlight comparison at `docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md` establishes From 42d6d173c3532dab99034eeec2b18dac8201cc7d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 06:48:07 +0100 Subject: [PATCH 123/135] recover compressed Muninn media under loss --- .../mimir_obs_muninn_source.cpp | 86 +++++++++++++++---- native/obs_stem_source/muninn_rudp_url.h | 2 +- .../tests/muninn_rudp_url_tests.cpp | 2 +- 3 files changed, 72 insertions(+), 18 deletions(-) diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index c7fc73b..ca29707 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -210,6 +210,7 @@ struct MuninnMediaPayload { std::string session_id; uint64_t frame_id = 0; uint64_t audio_packet_id = 0; + std::string audio_codec; uint64_t audio_pts_ticks = 0; uint64_t audio_duration_ticks = 0; uint32_t audio_timebase_num = 0; @@ -743,7 +744,7 @@ static std::optional decode_muninn_media_payload(const uint8 if (media.chunk_count == 0 || media.block_count == 0 || media.block_index >= media.block_count || media.block_data_count == 0 || media.block_data_start + media.block_data_count > media.chunk_count || - media.parity_count < media.block_data_count || media.parity_count > 8 || + media.parity_count == 0 || media.parity_count > 8 || media.parity_index >= media.parity_count || chunk_payload_bytes == 0 || last_chunk_payload_bytes == 0 || last_chunk_payload_bytes > chunk_payload_bytes) { return std::nullopt; @@ -762,7 +763,7 @@ static std::optional decode_muninn_media_payload(const uint8 media.stream_id = record.read_string(); media.session_id = record.read_string(); media.audio_packet_id = record.read_unsigned(); - record.skip(); // codec + media.audio_codec = record.read_string(); media.audio_pts_ticks = record.read_unsigned(); media.audio_duration_ticks = record.read_unsigned(); media.audio_timebase_num = static_cast(record.read_unsigned()); @@ -780,7 +781,7 @@ static std::optional decode_muninn_media_payload(const uint8 media.stream_id = record.read_string(); media.session_id = record.read_string(); media.audio_base_packet_id = record.read_unsigned(); - record.skip(); // codec + media.audio_codec = record.read_string(); record.skip(); // base_pts_ticks record.skip(); // packet_duration_ticks record.skip(); // timebase_num @@ -1346,11 +1347,28 @@ private: } } + static std::optional> decode_audio_codec_payload( + const std::string &codec, + const std::vector &payload) + { + if (codec != "aac-adts-padded-v1") { + return payload.empty() ? std::nullopt : std::optional>(payload); + } + if (payload.size() < 3) { + return std::nullopt; + } + const size_t encoded_bytes = (static_cast(payload[0]) << 8) | payload[1]; + if (encoded_bytes == 0 || encoded_bytes + 2 > payload.size()) { + return std::nullopt; + } + return std::vector(payload.begin() + 2, payload.begin() + 2 + encoded_bytes); + } + std::optional forward_audio_payload(const MuninnMediaPayload &media, SOCKET audio_socket, const sockaddr_in &audio_addr) { - if (media.stream_id.empty() || media.session_id.empty() || media.payload.empty()) { + if (media.stream_id.empty() || media.session_id.empty() || media.audio_codec.empty() || media.payload.empty()) { ++payloads_decoded_dropped_; log_bridge_pressure("dropped invalid audio packet metadata"); return std::nullopt; @@ -1361,6 +1379,10 @@ private: reset_audio_reorder(); audio_stream_id_ = media.stream_id; audio_session_id_ = media.session_id; + audio_codec_ = media.audio_codec; + } else if (audio_codec_ != media.audio_codec) { + ++payloads_decoded_dropped_; + return std::nullopt; } audio_fec_data_cache_[media.audio_packet_id] = MuninnPendingAudioPacket{media.payload, now}; @@ -1378,9 +1400,14 @@ private: return std::nullopt; } + const auto decoded_payload = decode_audio_codec_payload(media.audio_codec, media.payload); + if (!decoded_payload) { + ++payloads_decoded_dropped_; + return std::nullopt; + } auto inserted = pending_audio_packets_.emplace( media.audio_packet_id, - MuninnPendingAudioPacket{media.payload, now}); + MuninnPendingAudioPacket{*decoded_payload, now}); if (!inserted.second) { return std::nullopt; } @@ -1388,6 +1415,24 @@ private: ++audio_packets_reordered_; } + const uint64_t base_packet_id = media.audio_packet_id - + (media.audio_packet_id % muninn_audio_fec::DataShards); + const muninn_audio_fec::BlockKey key{media.stream_id, media.session_id, base_packet_id}; + auto recovered = audio_fec_receiver_.insert( + key, + static_cast(media.audio_packet_id - base_packet_id), + media.payload, + now); + for (auto &packet : recovered) { + const auto decoded = decode_audio_codec_payload(audio_codec_, packet.payload); + if (decoded && packet.packet_id >= next_audio_packet_id_ && + pending_audio_packets_.emplace( + packet.packet_id, + MuninnPendingAudioPacket{*decoded, now}).second) { + ++audio_packets_fec_recovered_; + } + } + return flush_audio_packets(now, audio_socket, audio_addr); } @@ -1395,7 +1440,7 @@ private: SOCKET audio_socket, const sockaddr_in &audio_addr) { - if (media.stream_id.empty() || media.session_id.empty() || media.payload.empty()) { + if (media.stream_id.empty() || media.session_id.empty() || media.audio_codec.empty() || media.payload.empty()) { ++payloads_decoded_dropped_; return std::nullopt; } @@ -1404,6 +1449,10 @@ private: reset_audio_reorder(); audio_stream_id_ = media.stream_id; audio_session_id_ = media.session_id; + audio_codec_ = media.audio_codec; + } else if (audio_codec_ != media.audio_codec) { + ++payloads_decoded_dropped_; + return std::nullopt; } const muninn_audio_fec::BlockKey key{ @@ -1429,6 +1478,11 @@ private: std::make_move_iterator(parity_recovered.end())); for (auto &packet : recovered) { + const auto decoded = decode_audio_codec_payload(audio_codec_, packet.payload); + if (!decoded) { + ++payloads_decoded_dropped_; + continue; + } if (!next_audio_packet_ready_) { next_audio_packet_ready_ = true; next_audio_packet_id_ = packet.packet_id; @@ -1436,7 +1490,7 @@ private: if (packet.packet_id >= next_audio_packet_id_ && pending_audio_packets_.emplace( packet.packet_id, - MuninnPendingAudioPacket{std::move(packet.payload), now}).second) { + MuninnPendingAudioPacket{*decoded, now}).second) { ++audio_packets_fec_recovered_; } } @@ -1527,6 +1581,7 @@ private: { audio_stream_id_.clear(); audio_session_id_.clear(); + audio_codec_.clear(); pending_audio_packets_.clear(); audio_fec_data_cache_.clear(); audio_fec_receiver_.clear(); @@ -1666,7 +1721,7 @@ private: media.parity_count == 0 || media.parity_index >= media.parity_count || (media.video_fec_blocked - ? (media.parity_count < block_data_count || media.parity_count > 8) + ? media.parity_count > 8 : media.parity_count > block_data_count) || media.chunk_payload_lengths.size() != block_data_count || media.payload.empty()) { @@ -2237,13 +2292,14 @@ private: uint64_t audio_packets_fec_recovered_ = 0; std::string audio_stream_id_; std::string audio_session_id_; + std::string audio_codec_; bool next_audio_packet_ready_ = false; uint64_t next_audio_packet_id_ = 0; std::chrono::steady_clock::time_point audio_gap_started_at_ = std::chrono::steady_clock::time_point::min(); std::map pending_audio_packets_; static constexpr size_t MuninnAudioFecDataCacheMaxPackets = 512; std::map audio_fec_data_cache_; - muninn_audio_fec::BlockReceiver audio_fec_receiver_{std::chrono::milliseconds(40), 128}; + muninn_audio_fec::BlockReceiver audio_fec_receiver_{std::chrono::milliseconds(150), 128}; bool media_sequence_initialized_ = false; uint32_t last_media_sequence_ = 0; uint64_t media_sequence_gap_events_ = 0; @@ -2602,11 +2658,7 @@ public: "-analyzeduration", "0", "-f", - "f32le", - "-ac", - "2", - "-ar", - "48000", + "aac", "-i", audio_url, "-vn", @@ -4246,8 +4298,10 @@ static void muninn_update_kind(void *data, obs_data_t *settings, MuninnSourceKin : MuninnDefaultRudpLatencyBudgetMs); const auto configured_audio_reorder = obs_data_get_int(settings, "rudp_audio_reorder_ms"); ctx->rudp_audio_reorder_ms = static_cast( - configured_audio_reorder >= 0 ? std::min(configured_audio_reorder, MuninnMaxRudpAudioReorderMs) - : MuninnDefaultRudpAudioReorderMs); + configured_audio_reorder >= 0 + ? std::clamp(configured_audio_reorder, MuninnDefaultRudpAudioReorderMs, + MuninnMaxRudpAudioReorderMs) + : MuninnDefaultRudpAudioReorderMs); ctx->stream_id = obs_string(settings, "stream_id", ""); if (source_wants_video(ctx)) { ctx->video_source_id = obs_string(settings, "video_source_id", ctx->video_source_id.c_str()); diff --git a/native/obs_stem_source/muninn_rudp_url.h b/native/obs_stem_source/muninn_rudp_url.h index ea72c7e..f489a40 100644 --- a/native/obs_stem_source/muninn_rudp_url.h +++ b/native/obs_stem_source/muninn_rudp_url.h @@ -13,7 +13,7 @@ inline constexpr uint32_t DefaultAudioRudpConnectionId = 0x6d750004; inline constexpr uint32_t DefaultReliableExpireAfterMs = 75; inline constexpr uint32_t DefaultVideoAssemblyDeadlineMs = 75; inline constexpr uint32_t DefaultGapWaitMs = 8; -inline constexpr uint32_t DefaultAudioReorderMs = 40; +inline constexpr uint32_t DefaultAudioReorderMs = 120; inline constexpr uint32_t DefaultSenderResendDelayMs = 5; inline constexpr uint32_t MaxLatencyBudgetMs = 500; diff --git a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp index 723a527..8b62c04 100644 --- a/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp +++ b/native/obs_stem_source/tests/muninn_rudp_url_tests.cpp @@ -53,7 +53,7 @@ int main() "minimal RUDP URL did not keep the default assembly deadline"); require(fallback->audio_reorder_ms == muninn_rudp_url::DefaultAudioReorderMs, "minimal RUDP URL did not keep the default audio reorder wait"); - require(fallback->audio_reorder_ms == 40, + require(fallback->audio_reorder_ms == 120, "minimal RUDP URL did not use the bounded realtime audio reorder budget"); return 0; From 78ca783276cc72c4cead4f105423ca6862cbb337 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 07:24:41 +0100 Subject: [PATCH 124/135] gate Muninn decode epochs and retry repair --- docs/implementation-plan.md | 11 +++-- ...light-reliability-acceptance-2026-07-16.md | 31 ++++++++++++ .../mimir_obs_muninn_source.cpp | 49 ++++++++++++++----- notes/current-system-map.md | 13 ++--- state/map.yaml | 13 ++--- 5 files changed, 89 insertions(+), 28 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 8f4127d..3d6e6bf 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -452,15 +452,18 @@ The first cut landed in Odin on 2026-07-16: experimental CultLib snapshot `8965f3c0`, epoch/sequence/edge-ack HID delivery, 100 ms default LAN media deadline, CultNet `realtime` A/V delivery, bounded queues, expiring/late-aware repair, and decode-chain-owned keyframe pressure. The production video path now -uses typed V4 Cauchy GF(256) FEC in independent 8-data/8-parity blocks. Each +uses typed V4 Cauchy GF(256) FEC in independent 8-data/4-parity blocks. Each block schedules data and parity as separate lanes, including fixed protection for a short tail block. Canonical video and parity use CultNet's unreliable `realtime` lane; selective repair and IDR recovery remain deadline-bound. -Canonical audio and its parity also use `realtime`; 4+2 FEC, reorder, and +Canonical AAC ADTS audio and its fixed 864-byte parity shards also use +`realtime`; 4+2 FEC, 120 ms reorder, and concealment own continuity without an ACK/retransmit window. Sender access units enter the handoff queue independently, and CultMesh catalog publication runs outside the -realtime media loop. The remaining work is the full direct/proxy impairment -matrix, Opus FEC/PLC, and long-duration mixed soak. +realtime media loop. Repair material becomes available when a video frame enters +the queue, and the receiver may issue bounded repair requests at 64/96/128 ms. +The remaining work is duplicate/reconnect acceptance, longer iid runs, and the +long-duration mixed soak. The socket harness and bounded PCM loss recovery have now landed. Odin's `cultnet-impair` proxy supplies deterministic seeded loss/burst/reorder/ diff --git a/docs/research/moonlight-reliability-acceptance-2026-07-16.md b/docs/research/moonlight-reliability-acceptance-2026-07-16.md index da31ac9..7a294a1 100644 --- a/docs/research/moonlight-reliability-acceptance-2026-07-16.md +++ b/docs/research/moonlight-reliability-acceptance-2026-07-16.md @@ -60,6 +60,37 @@ datagram or move fragmentation above the erasure-code boundary. scheduler sees it. Capacity-eviction counters proved zero during the clean acceptance run. +## 2026-07-17 bounded-media field cut + +The live path now keeps every protected shard in one datagram, sends AAC ADTS +in fixed 864-byte 4+2 audio blocks, uses 768-byte video shards with 8+4 block +FEC, and begins each decoder epoch at a complete IDR. The sender admits video +to its repair cache when the frame enters the queue, accepts the production +CultNet `realtime` channel, and can service bounded repair requests at +64/96/128 ms in batches of up to eight chunks. + +Seeded Raven-to-Starfire evidence: + +- iid 1%: 802 of 78,406 datagrams dropped; zero decode drop, video expiry, + audio failure, or audio skip; 1,800 video frames recovered; +- burst-4: 1,436 of 71,910 datagrams dropped; zero expiry or audio skip across + 1,800 forwarded frames; +- reorder: 2,872 reorder events; zero expiry or audio skip across 900 frames; +- iid 3%: 2,564 of 85,731 datagrams dropped; three of 5,400 frames expired, + zero audio skip (99.94 percent video delivery); +- burst-8: 190 chunks repaired; one of 1,800 frames expired, zero audio skip; +- 250 ms stall: 420 datagrams withheld, one frame expired, 68 chunks repaired, + an IDR was requested, and another 900 frames arrived without a stuck decoder + or audio skip. + +Delay/jitter produced one expiry in 900 frames. Duplicate, reconnect, longer +iid runs, and the 60-minute mixed soak remain open acceptance rows. + +Field scar: production video uses the `realtime` channel, but the repair cache +previously admitted only legacy `media` payloads. Synthetic tests built the +legacy shape and certified a repair path that was empty in production. The +production-channel regression test now owns this boundary. + ## Completion rule Moonlight-level performance is reasonably expected only after all local rows diff --git a/native/obs_stem_source/mimir_obs_muninn_source.cpp b/native/obs_stem_source/mimir_obs_muninn_source.cpp index ca29707..7b79587 100644 --- a/native/obs_stem_source/mimir_obs_muninn_source.cpp +++ b/native/obs_stem_source/mimir_obs_muninn_source.cpp @@ -65,6 +65,8 @@ constexpr uint64_t MuninnMediaStaleMs = 1500; constexpr uint64_t MuninnMediaRestartCooldownMs = 5000; constexpr uint64_t MuninnReceiverKeyframeRequestCooldownMs = 500; constexpr uint64_t MuninnReceiverChunkRepairFirstWaitMs = 64; +constexpr uint64_t MuninnReceiverChunkRepairRetryWaitMs = 32; +constexpr uint32_t MuninnReceiverChunkRepairMaxRequests = 3; constexpr uint64_t MuninnVideoAssemblyExpiryScanMs = 10; constexpr size_t MuninnExpiredVideoFrameRememberCount = 4096; constexpr size_t MuninnCompletedVideoFrameRememberCount = 4096; @@ -256,7 +258,9 @@ struct MuninnVideoFrameAssembly { uint16_t chunk_count = 0; uint16_t chunks_received = 0; std::chrono::steady_clock::time_point started_at; - bool repair_requested = false; + uint32_t repair_request_count = 0; + std::chrono::steady_clock::time_point last_repair_requested_at = + std::chrono::steady_clock::time_point::min(); std::map fec_blocks; std::vector> chunks; }; @@ -1650,6 +1654,15 @@ private: { expire_video_assemblies(std::chrono::steady_clock::now(), bridge_sequence); expire_video_parity_cache(std::chrono::steady_clock::now()); + const std::string key = + media.stream_id + ":" + media.session_id + ":" + std::to_string(media.frame_id); + if (waiting_for_video_keyframe_ && !media.keyframe) { + video_assemblies_.erase(key); + video_parity_cache_.erase(key); + remember_completed_video_frame(key); + ++video_recovery_dropped_; + return std::nullopt; + } if (media.kind == MuninnMediaPayload::Kind::VideoParity) { return assemble_video_parity_payload(media, bridge_sequence); } @@ -1658,8 +1671,6 @@ private: log_bridge_pressure("dropped invalid video chunk metadata"); return std::nullopt; } - const std::string key = - media.stream_id + ":" + media.session_id + ":" + std::to_string(media.frame_id); if (video_frame_is_settled(key)) { return std::nullopt; } @@ -1677,7 +1688,7 @@ private: assembly.chunk_count = media.chunk_count; assembly.chunks.resize(media.chunk_count); assembly.started_at = std::chrono::steady_clock::now(); - assembly.repair_requested = false; + assembly.repair_request_count = 0; apply_cached_video_parity(key, assembly); } if (assembly.chunk_count != media.chunk_count || @@ -1999,11 +2010,18 @@ private: if (needs_decoder_recovery) { waiting_for_video_keyframe_ = true; } - log_video_assembly_expiry(iterator->second, missing_chunk_keys, now); + const bool bootstrap_keyframe = waiting_for_video_keyframe_ && iterator->second.keyframe; + if (!bootstrap_keyframe) { + log_video_assembly_expiry(iterator->second, missing_chunk_keys, now); + } remember_expired_video_frame(iterator->first); iterator = video_assemblies_.erase(iterator); - ++video_assemblies_expired_; - log_bridge_pressure("expired incomplete video frame assembly"); + if (bootstrap_keyframe) { + ++video_recovery_dropped_; + } else { + ++video_assemblies_expired_; + log_bridge_pressure("expired incomplete video frame assembly"); + } } } @@ -2046,7 +2064,7 @@ private: static_cast(assembly.chunk_count), missing_chunk_keys.size(), static_cast(age_ms), - assembly.repair_requested ? "true" : "false", + assembly.repair_request_count > 0 ? "true" : "false", assembly.fec_blocks.empty() ? "false" : "true", fec_state.str().c_str()); } @@ -2058,11 +2076,17 @@ private: if (assembly.chunks_received >= assembly.chunk_count) { return; } - if (assembly.repair_requested) { + if (assembly.repair_request_count >= MuninnReceiverChunkRepairMaxRequests) { return; } - const auto repair_wait_ms = std::max(parts_.gap_wait_ms, MuninnReceiverChunkRepairFirstWaitMs); - if (now - assembly.started_at < std::chrono::milliseconds(repair_wait_ms)) { + const auto first_wait_ms = std::max(parts_.gap_wait_ms, MuninnReceiverChunkRepairFirstWaitMs); + if (assembly.repair_request_count == 0 && + now - assembly.started_at < std::chrono::milliseconds(first_wait_ms)) { + return; + } + if (assembly.repair_request_count > 0 && + now - assembly.last_repair_requested_at < + std::chrono::milliseconds(MuninnReceiverChunkRepairRetryWaitMs)) { return; } const auto missing_chunk_keys = missing_video_chunk_keys(assembly); @@ -2070,7 +2094,8 @@ private: return; } send_receiver_feedback_if_due(assembly, missing_chunk_keys, false, false, bridge_sequence); - assembly.repair_requested = true; + assembly.last_repair_requested_at = now; + ++assembly.repair_request_count; } void expire_rudp_packet_fragments(std::chrono::steady_clock::time_point now) diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 6716d5d..86e1235 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -342,13 +342,13 @@ The live Raven-to-Starfire program path is Muninn's typed CultNet media body: ```mermaid flowchart TD A["Raven D3D11 desktop capture"] --> B["Long-lived NVENC encoder"] - C["Raven WASAPI loopback"] --> D["10 ms float PCM packetizer"] - B --> E["V4 8+8 block FEC and realtime video lane"] - D --> F["4+2 audio FEC and realtime audio lane"] + C["Raven WASAPI loopback"] --> D["AAC-LC / ADTS encoder"] + B --> E["V4 8+4 block FEC and realtime video lane"] + D --> F["Fixed 864-byte 4+2 audio FEC lane"] E --> G["Starfire OBS Muninn receiver udp/5200"] F --> G G --> H["Frame assembly / FEC / IDR feedback"] - G --> I["40 ms audio reorder / concealment"] + G --> I["120 ms audio reorder / concealment"] H --> J["OBS-local video decoder socket"] I --> K["OBS-local audio decoder socket"] ``` @@ -361,8 +361,9 @@ the provider surface; neither discovery nor the compatibility decoder sockets own hot media truth. Invariant: video packets are useful only before their assembly deadline. -Canonical video and V4 parity use unreliable `realtime`; block FEC, selective -repair, and IDR recovery own loss. Canonical audio and 4+2 parity also use +Canonical video and V4 parity use unreliable `realtime`; 8+4 block FEC, +queue-admitted selective repair, and IDR recovery own loss. Canonical AAC audio +and fixed 4+2 parity also use `realtime`; reorder and bounded concealment own playout continuity. Completed video frames are tombstoned so late shards cannot become a second assembly owner. Raven uses direct LAN only and starts through a diff --git a/state/map.yaml b/state/map.yaml index c17370f..4343226 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -89,13 +89,13 @@ invariants: accepted_design: - Muninn video FEC is typed as `muninn.media_video_parity_shard.v4`. Odin divides each access unit into independent blocks of at most eight data - shards and emits eight Cauchy GF(256) parity rows for every block, including + shards and emits four Cauchy GF(256) parity rows for every block, including a short tail. Data/parity lanes are round-robin scheduled per block. Mimir's OBS receiver owns bounded block recovery before the assembly deadline, forwards only the canonical completed frame, and tombstones completed frame keys so late shards cannot resurrect ghost assemblies. V2/V3 remain legacy decode shapes and never own V4 behavior. Rust tests exhaust every burst of - one through eight packets at every alignment for chunk counts 2 through 80; + one through four packets at every alignment, plus tail-block burst coverage; native tests prove the matching receiver. Canonical video and parity now use CultNet `realtime` without per-packet reliable queues. Repair is selective and behind fresh media; unrecoverable dependency damage requests a live @@ -122,8 +122,9 @@ accepted_design: The normal quarter-second scheduled IDR remains independent fallback. The repeatable bundle includes the OBS experimental FFmpeg dependency snapshot. Raven now runs the bundle over direct LAN from a hidden scheduled - task; WireGuard is not a runtime or administration path. Real impairment - acceptance remains pending. + task; WireGuard is not a runtime or administration path. Seeded 1% loss, + burst-4, reorder, and bounded stall recovery now pass; longer matrix and + soak acceptance remain pending. - Muninn input now bounds disconnected unacknowledged button history at 256 edges. Overflow rotates the controller epoch and rebases Sleipnir from the next complete latest-state frame, rather than dropping an arbitrary press @@ -131,11 +132,11 @@ accepted_design: 256. Full Muninn (174) and Sleipnir (26) suites pass. - The 2026-07-16 loss-recovery cut is now live on both sides of the Muninn audio boundary. Odin emits typed `muninn.media_audio_parity_shard.v1` - records using fixed 4+2 GF(256) parity for constant-size 10 ms PCM packets. + records using fixed 4+2 GF(256) parity for padded AAC ADTS packets. Canonical audio and parity use `realtime`; 4+2 FEC, reorder, and bounded concealment own continuity without a media ACK/retransmit window. Mimir's production OBS receiver retains a bounded recent data window, - reconstructs up to two erased packets per block before the 40 ms playout + reconstructs up to two erased packets per block before the 120 ms playout deadline, and then hands recovered packets to the same reorder queue that owns normal playout. The FEC cache is not a second timeline. Native tests exhaust every one- and two-erasure permutation, and the production plugin From fea4d27dd1d77c7ca9b48f4e0d02fcff2aa696a2 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 08:12:21 +0100 Subject: [PATCH 125/135] Document Raven Moonlight field acceptance --- ...nlight-muninn-sleipnir-study-2026-07-16.md | 30 +++++++++++++++++++ state/evidence.jsonl | 1 + state/map.yaml | 10 +++++++ 3 files changed, 41 insertions(+) diff --git a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md index 0ceb09d..41da754 100644 --- a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md +++ b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md @@ -257,6 +257,36 @@ loss: Muninn and Sleipnir already have enough organs to become strong. The next work is to give each signal one explicit survival law and make the transport obey it. +## 2026-07-17 LAN field results + +The Moonlight-derived ownership changes now have Raven-to-Starfire field +evidence over LAN, with Raven processes launched only through hidden scheduled +tasks: + +- Muninn video uses independently recoverable block parity plus bounded repair. + During a seeded 1% datagram-loss run, the OBS receiver advanced from 2 to 12 + video assembly expiries while forwarding 14,400 additional payload groups: + 99.93% completed without expiry. Audio reported zero failures, skips, or stale + packets in the same interval. The impairment proxy forwarded 2,147,356 of + 2,169,105 datagrams with zero queue overflow. +- Sleipnir latest-state delivery remained live through seeded 1% loss. Cutting + the proxy caused the non-neutral virtual Xbox pad to neutralize after 1.56 s, + owned by the named stale-input lease; restoring the hidden proxy reconnected + and resumed state without operator action. +- A continuous 64-packet receive drain was retaining the newest input for about + 48-57 ms before HID application. Bounding that drain to 2 ms reduced live + receiver-owned receive-to-apply time to typically 69-204 microseconds. A + witnessed 16 ms spike remains to characterize. +- Starfire and Raven wall clocks differ by roughly 2.38 s. Sleipnir no longer + publishes their raw subtraction as one-way latency. Its surface reports that + source-clock latency is unavailable without a clock model and exposes only + monotonic receiver-owned stage durations. + +These results clear the initial 99.9% media-completion and disconnect-safety +gates at 1% independent loss. They do not yet prove glass-to-glass latency, +A/V skew, ordered quick-tap delivery under loss, 5%/burst behavior, or a long +mixed-impairment soak. Those remain acceptance gates, not footnotes. + The field comparison also exposed a generic transport invariant Moonlight's long-running sessions force us to respect: bounded wire identifiers must wrap deliberately. CultNet's fragmented-message ID previously saturated at the diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 7e336cb..70b74a3 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -199,3 +199,4 @@ {"ts": "2026-07-17T01:59:33+01:00", "type": "scar", "status": "open", "note": "Direct Raven LAN falsified trailing-parity sufficiency: a 24-datagram sequence gap left a 56-chunk keyframe at 52/56 with parity_present=false because all 16 V3 parity shards followed the data and fell in the same tail burst. Next cut must disperse parity/adapt block layout; deadline increases and repair flooding are not owners."} {"ts": "2026-07-17T02:34:05+01:00", "type": "verification", "status": "ok", "note": "Receiver completed-frame tombstones eliminated false post-FEC ghost assemblies: a 7-minute LAN run forwarded 54,000 payloads with zero video assembly expiry before an experimental weighted sender scheduler independently caused audio queue eviction; the scheduler was deleted. A later genuine 24-datagram tail burst left a frame 18/33 with only the leading 8 of 16 bookended parity shards available, proving V3 frame-wide bookends still need Moonlight-style multi-block FEC."} {"ts": "2026-07-17T02:58:43+01:00", "type": "rejected_path", "status": "ok", "note": "Adaptive frame-wide parity was tested and removed: 32 dispersed shards still expired a frame during a 54-datagram gap, and 64-shard production recovery violated the 50 ms CPU bound. V4 must use smaller independently recoverable blocks rather than a frame-wide bandwidth/matrix arms race. Periodic audio skips were traced separately to seven orphaned Raven WASAPI capture workers left by forced parent stops; the hidden scheduled task now reaps only stale Muninn, FFmpeg, encoder, and loopback workers before launching exactly one sender tree."} +{"ts":"2026-07-17T08:15:00+01:00","type":"verification","status":"partial","note":"Raven LAN Moonlight field pass used hidden scheduled tasks, no WireGuard, and experimental CultLib 8965f3c0. Seeded 1% media loss completed 99.93% of 14,400 additional OBS payload groups without expiry (10 expiries), with zero audio failures/skips and no impairment-proxy queue overflow. Sleipnir neutralized the non-neutral virtual Xbox pad after 1.56 s of input loss and automatically recovered. A 64-packet continuous receive drain caused 48-57 ms receive-to-apply delay; a 2 ms drain budget reduced typical live receiver-owned delay to 69-204 us. Cross-host raw clock subtraction was removed from latency telemetry. One 16 ms spike, quick-tap edge impairment, 5%/burst input, glass-to-glass age, A/V skew, and mixed soak remain open."} diff --git a/state/map.yaml b/state/map.yaml index 4343226..e02b44a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -186,6 +186,16 @@ accepted_design: replaceable latest state from ordered non-replaceable edges, coalesce before enqueue, acknowledge applied edge/state sequences, and use a proved clock model before reporting cross-host wall-clock subtraction as one-way latency. + - The 2026-07-17 Raven LAN field pass used hidden scheduled tasks and the + experimental CultLib snapshot at `8965f3c0`; Raven used no WireGuard path. + Seeded 1% media loss completed 99.93% of 14,400 additional OBS payload + groups without video assembly expiry and produced zero audio failures or + skips. Sleipnir neutralized a non-neutral virtual pad 1.56 s after input + loss and recovered automatically. Bounding its continuous receive drain to + 2 ms reduced live receive-to-apply from roughly 48-57 ms to typically + 69-204 us. Cross-host source latency is now explicitly unavailable until a + clock offset/uncertainty model exists; a 16 ms receiver-side spike, edge + impairment proof, glass-to-glass age, A/V skew, and mixed soak remain open. - Nightwing Muninn `d0eded6` now owns PSMoveAPI observation admission in each private Eye worker. It rejects stale, sub-2 px, out-of-frame, and discontinuous positions before the parent aggregates evidence; typed From 7468ba59bc7b06b740696770c7e68f5eaad70be5 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 10:40:06 +0100 Subject: [PATCH 126/135] Record ordered input field closure --- ...nlight-muninn-sleipnir-study-2026-07-16.md | 38 +++++++++++++++++++ state/evidence.jsonl | 2 + state/map.yaml | 19 +++++++++- 3 files changed, 57 insertions(+), 2 deletions(-) diff --git a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md index 41da754..deadbbc 100644 --- a/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md +++ b/docs/research/moonlight-muninn-sleipnir-study-2026-07-16.md @@ -293,3 +293,41 @@ deliberately. CultNet's fragmented-message ID previously saturated at the maximum `u16`, so fragmented audio became unreassemblable after the boundary. Wrap-to-one plus a boundary test removed the repeatable minute-eight collapse; a subsequent 10-minute direct LAN run remained clean across the boundary. + +## 2026-07-17 ordered-input closure + +The quick-tap gate exposed four authorities that generic RUDP reliability had +blurred together: + +- A Windows sleep requested at two milliseconds could still miss a short tap. + Muninn now observes XInput on a dedicated high-resolution worker; transport, + discovery, serialization, and logging cannot delay capture. +- Global cumulative ACKs could not retire a reliable packet arriving more than + 32 realtime sequence numbers late, and reliable duplicates were not ACKed + after application deduplication. CultNet now directly acknowledges the late + packet and acknowledges reliable duplicates. This collapsed a runaway field + trace from hundreds of thousands of resend datagrams to a bounded flow. +- Muninn used semantic application ACK as permission to send the next edge. + One delayed ACK therefore serialized the entire controller. It now sends a + bounded window of captured edges; CultNet owns retransmission, Sleipnir's + epoch/sequence cursor owns ordering, and semantic ACK only retires history. +- Sleipnir published CultMesh/Eve and Idunn state synchronously on the actuator + thread. Those writes produced 0.7-1.8 second visible button holds. A bounded + coalescing publisher worker now owns control-plane projection, leaving the + input thread to receive, order, actuate, and acknowledge. + +The resulting first-hot-plug field proof generated 100 taps with a 16 ms source +press and 84 ms gap. Under seeded 1% independent loss, every layer observed +exactly 100 presses and 100 releases; Raven-visible durations were 24-57 ms. +Under eight consecutive dropped packets every 200 datagrams, every layer again +observed exactly 100 presses and 100 releases; visible durations were 24-40 ms, +the final virtual pad was neutral, and the impairment proxy reported no queue +overflow. Raven ran through hidden scheduled tasks over `192.168.1.0/24`; no +WireGuard route participated. Both daemons used the Eve/Aetheria/VoidBot +experimental CultLib lineage pinned at `8965f3c0`. + +This closes ordered digital edges at one 60 Hz source frame. It does not prove +sub-frame synthetic ViGEm pulses: eight-millisecond state-only pulses can fall +between XInput snapshots, and `XInputGetKeystroke` was rejected after it emitted +stale events from a reused user slot. Analog motion under a sender stall, +mixed media/input endurance, glass-to-glass video age, and A/V skew remain open. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 70b74a3..c1ee0be 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -200,3 +200,5 @@ {"ts": "2026-07-17T02:34:05+01:00", "type": "verification", "status": "ok", "note": "Receiver completed-frame tombstones eliminated false post-FEC ghost assemblies: a 7-minute LAN run forwarded 54,000 payloads with zero video assembly expiry before an experimental weighted sender scheduler independently caused audio queue eviction; the scheduler was deleted. A later genuine 24-datagram tail burst left a frame 18/33 with only the leading 8 of 16 bookended parity shards available, proving V3 frame-wide bookends still need Moonlight-style multi-block FEC."} {"ts": "2026-07-17T02:58:43+01:00", "type": "rejected_path", "status": "ok", "note": "Adaptive frame-wide parity was tested and removed: 32 dispersed shards still expired a frame during a 54-datagram gap, and 64-shard production recovery violated the 50 ms CPU bound. V4 must use smaller independently recoverable blocks rather than a frame-wide bandwidth/matrix arms race. Periodic audio skips were traced separately to seven orphaned Raven WASAPI capture workers left by forced parent stops; the hidden scheduled task now reaps only stale Muninn, FFmpeg, encoder, and loopback workers before launching exactly one sender tree."} {"ts":"2026-07-17T08:15:00+01:00","type":"verification","status":"partial","note":"Raven LAN Moonlight field pass used hidden scheduled tasks, no WireGuard, and experimental CultLib 8965f3c0. Seeded 1% media loss completed 99.93% of 14,400 additional OBS payload groups without expiry (10 expiries), with zero audio failures/skips and no impairment-proxy queue overflow. Sleipnir neutralized the non-neutral virtual Xbox pad after 1.56 s of input loss and automatically recovered. A 64-packet continuous receive drain caused 48-57 ms receive-to-apply delay; a 2 ms drain budget reduced typical live receiver-owned delay to 69-204 us. Cross-host raw clock subtraction was removed from latency telemetry. One 16 ms spike, quick-tap edge impairment, 5%/burst input, glass-to-glass age, A/V skew, and mixed soak remain open."} +{"ts": "2026-07-17T10:37:54+01:00", "type": "verification", "status": "ok", "note": "Ordered Muninn/Sleipnir input field proof closed one-frame quick taps over Raven LAN with hidden tasks, no WireGuard, and experimental CultLib 8965f3c0. Dedicated XInput capture, bounded non-stop-and-wait reliable edges, late/duplicate CultNet ACK repair, 24 ms edge visibility, and asynchronous CultMesh/Eve publication produced exact 100/100 presses/releases under seeded 1% loss and exact 100/100 under burst-8; burst Raven-visible durations were 24-40 ms, final pad neutral, proxy queue overflow zero. Mixed media/input endurance, analog stall, glass-to-glass age, and A/V skew remain open."} +{"ts": "2026-07-17T10:39:57+01:00", "type": "verification", "status": "ok", "note": "Raven execution is headless and LAN-only: Codex-Sleipnir-LAN-Proof and Codex-Raven-XInput-Edge-Observer are scheduled-task launches whose PowerShell actions use -NonInteractive -WindowStyle Hidden; both task metadata records are now Hidden=true. Raven exposes no WireGuard service. Remote administration used raven-local over 192.168.1.84."} diff --git a/state/map.yaml b/state/map.yaml index e02b44a..eca4e3a 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -194,8 +194,23 @@ accepted_design: loss and recovered automatically. Bounding its continuous receive drain to 2 ms reduced live receive-to-apply from roughly 48-57 ms to typically 69-204 us. Cross-host source latency is now explicitly unavailable until a - clock offset/uncertainty model exists; a 16 ms receiver-side spike, edge - impairment proof, glass-to-glass age, A/V skew, and mixed soak remain open. + clock offset/uncertainty model exists. + - The 2026-07-17 ordered-input field cut removed transport and control-plane + stalls from the Muninn/Sleipnir path. Muninn observes XInput on a dedicated + 1 kHz, one-millisecond-timer worker and derives exact button edges from + consecutive snapshots. Captured edges are sent as a bounded reliable + window; application ACKs retire history but never gate the next edge. + CultNet directly ACKs late reliable packets outside its 32-packet + cumulative window and ACKs reliable duplicates, preventing immortal resend + storms. Sleipnir actuates ordered edges for at least 24 ms and publishes + CultMesh/Eve/Idunn state on a separate coalescing worker. Raven field proof + over LAN, hidden tasks, experimental CultLib `8965f3c0`, and no WireGuard + produced exact 100/100 one-frame taps under seeded 1% loss and exact + 100/100 under eight-packet bursts. Burst visible durations were 24-40 ms, + final state was neutral, and the impairment proxy had no queue overflow. + A steady-state 5% run was exact at 20/20; repeat the 100-tap 5% first-session + gate after the stale XInput keystroke path was deleted. Glass-to-glass age, + A/V skew, analog motion under stall, and mixed soak remain open. - Nightwing Muninn `d0eded6` now owns PSMoveAPI observation admission in each private Eye worker. It rejects stale, sub-2 px, out-of-frame, and discontinuous positions before the parent aggregates evidence; typed From ce98c421dca87e5c9e1c2b0267a3baa6c1c8f9dc Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Fri, 17 Jul 2026 11:20:30 +0100 Subject: [PATCH 127/135] Record mixed-stream input latency proof --- state/evidence.jsonl | 3 +++ 1 file changed, 3 insertions(+) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index c1ee0be..abddacc 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -202,3 +202,6 @@ {"ts":"2026-07-17T08:15:00+01:00","type":"verification","status":"partial","note":"Raven LAN Moonlight field pass used hidden scheduled tasks, no WireGuard, and experimental CultLib 8965f3c0. Seeded 1% media loss completed 99.93% of 14,400 additional OBS payload groups without expiry (10 expiries), with zero audio failures/skips and no impairment-proxy queue overflow. Sleipnir neutralized the non-neutral virtual Xbox pad after 1.56 s of input loss and automatically recovered. A 64-packet continuous receive drain caused 48-57 ms receive-to-apply delay; a 2 ms drain budget reduced typical live receiver-owned delay to 69-204 us. Cross-host raw clock subtraction was removed from latency telemetry. One 16 ms spike, quick-tap edge impairment, 5%/burst input, glass-to-glass age, A/V skew, and mixed soak remain open."} {"ts": "2026-07-17T10:37:54+01:00", "type": "verification", "status": "ok", "note": "Ordered Muninn/Sleipnir input field proof closed one-frame quick taps over Raven LAN with hidden tasks, no WireGuard, and experimental CultLib 8965f3c0. Dedicated XInput capture, bounded non-stop-and-wait reliable edges, late/duplicate CultNet ACK repair, 24 ms edge visibility, and asynchronous CultMesh/Eve publication produced exact 100/100 presses/releases under seeded 1% loss and exact 100/100 under burst-8; burst Raven-visible durations were 24-40 ms, final pad neutral, proxy queue overflow zero. Mixed media/input endurance, analog stall, glass-to-glass age, and A/V skew remain open."} {"ts": "2026-07-17T10:39:57+01:00", "type": "verification", "status": "ok", "note": "Raven execution is headless and LAN-only: Codex-Sleipnir-LAN-Proof and Codex-Raven-XInput-Edge-Observer are scheduled-task launches whose PowerShell actions use -NonInteractive -WindowStyle Hidden; both task metadata records are now Hidden=true. Raven exposes no WireGuard service. Remote administration used raven-local over 192.168.1.84."} +{"ts": "2026-07-17T10:46:17+01:00", "type": "verification", "status": "ok", "note": "Final snapshot-only Muninn/Sleipnir input build delivered 100/100 one-frame A-button presses and releases over Raven LAN under seeded 5% datagram loss (254/5251 dropped): Starfire observer 100/100, Muninn captured 200 semantic edges, Raven observer 100/100, final pad neutral, proxy queue overflow zero/max queue 28. Raven-visible hold duration median 29.166 ms, p95 36.719 ms, p99 62.667 ms, max 118.074 ms; retransmission tail remains the next latency pressure point."} +{"ts": "2026-07-17T11:20:23+01:00", "type": "rejected_path", "status": "ok", "note": "Application semantic replay over the reliable hid.edge channel was built and field-falsified under 5% loss: new reliable duplicates accumulated transport debt and worsened Raven-visible holds to multi-second outliers. The reliable replay cut was deleted; semantic assist must never create retransmission debt."} +{"ts": "2026-07-17T11:20:23+01:00", "type": "verification", "status": "ok", "note": "Native 1 ms XInput witnesses replaced scheduling-contaminated PowerShell polling. Under concurrent sustained Raven media at seeded 1% loss and 100 one-frame input taps at seeded 5% loss, dual-path input (one reliable hid.edge owner plus bounded unreliable unordered hid.edge.assist repeats) delivered exact 100/100 and final neutral. Raven hold p95 improved 78.875->38.000 ms, p99 264.745->53.235 ms, max 272.107->59.028 ms; network-added median 11.617 ms, p95 17.387 ms, max 23.267 ms; media and input proxy queue overflow remained zero."} From 4d7378216faf4f6cdbb9da8cc3bbd1a7a6ccd065 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 3 Sep 2026 22:23:05 +0200 Subject: [PATCH 128/135] Wire AGENTS.md into Claude Code memory Claude Code reads CLAUDE.md, not AGENTS.md, so this repo's doctrine was invisible to it. This one-line import makes the existing AGENTS.md load without duplicating or renaming it. Codex is unaffected. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..640c1a1 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,3 @@ +Repository doctrine for this project lives in AGENTS.md, shared with Codex. + +@AGENTS.md From aa36b67dd6f6d5164806cb79484bc4693c00b17e Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 13 Sep 2026 17:59:50 +0200 Subject: [PATCH 129/135] Follow CultLib Cut 3: drop PullOnOpen from BufferSmoke CultLib's CultCacheMessagePack.OpenAsync now always hydrates and CultCacheOpenOptions.PullOnOpen no longer exists. The five BufferSmoke opens passed PullOnOpen = true or File.Exists(path); hydrating a missing store is empty, so each becomes a plain OpenAsync(path). Co-Authored-By: Claude Opus 5 --- src/Mimir.BufferSmoke/Program.cs | 25 +++++-------------------- 1 file changed, 5 insertions(+), 20 deletions(-) diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index a93a229..f62b020 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1938,10 +1938,7 @@ static async Task RunMuninnMoveLiveRosterSmokeAsync(IReadOnlyList i continue; } - using var cache = await CultCacheMessagePack.OpenAsync(fullPath, new CultCacheOpenOptions - { - PullOnOpen = true - }).ConfigureAwait(false); + using var cache = await CultCacheMessagePack.OpenAsync(fullPath).ConfigureAwait(false); var storeIdentities = cache.GetAll().ToArray(); identities.AddRange(storeIdentities); loadedStores++; @@ -3686,10 +3683,7 @@ static async Task RunMoveCalibrationProtocolSmokeAsync(string outputPath) } Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); - using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions - { - PullOnOpen = File.Exists(outputPath) - }).ConfigureAwait(false); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath).ConfigureAwait(false); await cache.UpsertAsync( protocol, new CultRecordHandle(new CultRecordKey($"mimir-move-calibration-protocol:{protocol.ProtocolId}"))) @@ -3985,10 +3979,7 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) "mimir-move-stage-calibration-v1"); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); - using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions - { - PullOnOpen = File.Exists(outputPath) - }).ConfigureAwait(false); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath).ConfigureAwait(false); await cache.UpsertAsync( codebook, new CultRecordHandle(new CultRecordKey($"mimir-codebook:{codebook.CodebookId}"))) @@ -4061,10 +4052,7 @@ static async Task ImportObsProgramSceneAsync(string inputPath, string scene importedAt); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); - using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions - { - PullOnOpen = File.Exists(outputPath) - }).ConfigureAwait(false); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath).ConfigureAwait(false); await cache.UpsertAsync( scene, new CultRecordHandle(new CultRecordKey($"mimir-program-scene:{scene.SceneId}"))) @@ -7617,10 +7605,7 @@ static float[] ReconstructDetectedBioacousticSong(IReadOnlyList results) { - using var cache = await CultCacheMessagePack.OpenAsync(cachePath, new CultCacheOpenOptions - { - PullOnOpen = File.Exists(cachePath) - }).ConfigureAwait(false); + using var cache = await CultCacheMessagePack.OpenAsync(cachePath).ConfigureAwait(false); foreach (var result in results) { await cache.UpsertAsync( From 5591c5c47c0d88c64cc156592f9cf9eef8f1c1a8 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 13 Sep 2026 23:38:14 +0200 Subject: [PATCH 130/135] Follow CultLib generator deletion: always serialize through MessagePack CultDocumentDescriptor no longer carries a generated payload serializer. Co-Authored-By: Claude Opus 5 --- src/Mimir.EveBrowserReference/Program.cs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/Mimir.EveBrowserReference/Program.cs b/src/Mimir.EveBrowserReference/Program.cs index 8e86bd8..b1a3ef9 100644 --- a/src/Mimir.EveBrowserReference/Program.cs +++ b/src/Mimir.EveBrowserReference/Program.cs @@ -464,9 +464,7 @@ internal sealed class EveBrowserReferenceVerseRuntime : IDisposable string observedAtUtc) where T : class { var descriptor = registry.GetRequired(); - var payload = descriptor.GeneratedPayloadSerializer != null - ? descriptor.GeneratedPayloadSerializer(document) - : MessagePackSerializer.Serialize(typeof(T), document, CultNetSchemaMessageSerialization.Options); + var payload = MessagePackSerializer.Serialize(typeof(T), document, CultNetSchemaMessageSerialization.Options); schemas[descriptor.SchemaId] = descriptor.ToCatalogEntry(); records.Add(new CultPersistedRecord { From d88f2e6dc197b0e0292540c3ee38764b84a596d0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Mon, 14 Sep 2026 14:12:28 +0200 Subject: [PATCH 131/135] Flush the media cache without the ignored soft flag CultCache.FlushAsync(bool soft) ignored soft and CultLib is deleting the parameter; the media put path now calls FlushAsync(). Co-Authored-By: Claude Opus 5 --- src/Mimir.CultMeshMedia/Program.cs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Mimir.CultMeshMedia/Program.cs b/src/Mimir.CultMeshMedia/Program.cs index a497083..58e7616 100644 --- a/src/Mimir.CultMeshMedia/Program.cs +++ b/src/Mimir.CultMeshMedia/Program.cs @@ -419,7 +419,7 @@ sealed class RudpMediaRelay : IDisposable } await registry.ApplyRawDocumentPutMessageAsync(cache, put).ConfigureAwait(false); - await cache.FlushAsync(soft: true).ConfigureAwait(false); + await cache.FlushAsync().ConfigureAwait(false); var change = new CultNetDatabaseChangeRawMessage { MessageId = Guid.NewGuid().ToString("N"), From c08e1a16cdd513287cf159fbc5dc002d582a1880 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 22 Oct 2026 19:46:49 +0200 Subject: [PATCH 132/135] Pin CultLib 2.0.0 for builds Directory.Build.props names CultLibRoot (CULTLIB_ROOT, else the sibling checkout) and pins CultLibRevision to cultlib-unity-v2.0.0 (42dce361). Directory.Build.targets fails the build unless that checkout is at the pin and clean, as Aetheria's guard does. Project references use CultLibRoot. Co-Authored-By: Claude Sonnet 5.5 --- Directory.Build.props | 10 +++++++ Directory.Build.targets | 30 +++++++++++++++++++ README.md | 11 +++++++ .../Mimir.BufferSmoke.csproj | 4 +-- .../Mimir.CultMeshMedia.csproj | 4 +-- .../Mimir.EveBrowserReference.csproj | 4 +-- .../Mimir.EveDashboard.csproj | 6 ++-- .../Mimir.EveSensorReceiver.csproj | 2 +- src/Mimir.Runtime/Mimir.Runtime.csproj | 6 ++-- 9 files changed, 64 insertions(+), 13 deletions(-) create mode 100644 Directory.Build.props create mode 100644 Directory.Build.targets diff --git a/Directory.Build.props b/Directory.Build.props new file mode 100644 index 0000000..c8a3c67 --- /dev/null +++ b/Directory.Build.props @@ -0,0 +1,10 @@ + + + $(CULTLIB_ROOT) + $(MSBuildThisFileDirectory)..\CultLib + + 42dce3611a81adc5c5fac54cadbaa5637d6022d3 + C:\Program Files\Git\cmd\git.exe + git + + diff --git a/Directory.Build.targets b/Directory.Build.targets new file mode 100644 index 0000000..46d436a --- /dev/null +++ b/Directory.Build.targets @@ -0,0 +1,30 @@ + + + + + <_CultLibReference Include="@(ProjectReference)" + Condition="$([System.String]::Copy('%(Filename)').StartsWith('GameCult.')) or '%(Filename)' == 'CultMath'" /> + + + + + + + + + + + + diff --git a/README.md b/README.md index 63eb101..bdab7fe 100644 --- a/README.md +++ b/README.md @@ -329,3 +329,14 @@ layer for communication and heartbeats. Its birth memory, jurisdiction, voice, and heartbeat contract live in [docs/mimir-face.md](docs/mimir-face.md); its VoidBot-facing identity and typed Face state live under `.voidbot/voice/` and `.voidbot/state/`. + +## Building against CultLib + +Builds are pinned to CultLib 2.0.0 (tag `cultlib-unity-v2.0.0`, revision `42dce3611a81adc5c5fac54cadbaa5637d6022d3`); the build fails if the CultLib checkout differs or is dirty. Create a pinned worktree and point the build at it: + +```powershell +git -C worktree add --detach cultlib-unity-v2.0.0 +dotnet build -p:CultLibRoot= # or set CULTLIB_ROOT= +``` + +Without an override the sibling `..\CultLib` checkout is used and must itself be at that tag. diff --git a/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj b/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj index e4eb296..9a78f03 100644 --- a/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj +++ b/src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj @@ -11,8 +11,8 @@ - - + + diff --git a/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj b/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj index cb154c1..3a4d105 100644 --- a/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj +++ b/src/Mimir.CultMeshMedia/Mimir.CultMeshMedia.csproj @@ -11,8 +11,8 @@ - - + + diff --git a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj index ea6d407..ad5a698 100644 --- a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj +++ b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj @@ -10,8 +10,8 @@ - - + + diff --git a/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj b/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj index 5e01cd5..b18d295 100644 --- a/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj +++ b/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj @@ -10,9 +10,9 @@ - - - + + + diff --git a/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj b/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj index 48be7bc..a0eab2d 100644 --- a/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj +++ b/src/Mimir.EveSensorReceiver/Mimir.EveSensorReceiver.csproj @@ -10,7 +10,7 @@ - + diff --git a/src/Mimir.Runtime/Mimir.Runtime.csproj b/src/Mimir.Runtime/Mimir.Runtime.csproj index 4d06a5e..e2e9d59 100644 --- a/src/Mimir.Runtime/Mimir.Runtime.csproj +++ b/src/Mimir.Runtime/Mimir.Runtime.csproj @@ -12,9 +12,9 @@ - - - + + + From e867e44cbce9d0fabbe1d10c4745782f8bc4eecd Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 22 Oct 2026 20:04:52 +0200 Subject: [PATCH 133/135] README: front door with value, problem, entry points, live versus planned; quick start copies the example config Co-Authored-By: Claude Opus 5.5 --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index bdab7fe..dff666e 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,18 @@ OBS-facing program surface. The plain version: Mimir is trying to make a room computable. +## Front Door + +- **Value:** a measured room: cameras, microphones, loopbacks and network feeds aligned in time tightly enough to recover where a sound came from and how a body moved, then handed to OBS as one program surface. +- **Problem:** loose devices each keep their own clock, so a flat stream cannot localize voices, separate sources or fuse cameras. Mimir owns the synchronization evidence that makes the feeds one field. +- **Entry points:** + - `docs/perfect-machine.md`: the target machine; `docs/viable-stream-app.md`: the first usable app cut; `docs/audio-field.md`: the live audio decoder and timing model. + - `src/Mimir.Runtime` (rolling buffers, stream sources, synchronization state) and `src/Mimir.App` (the Fensalir-hosted window and render app); solution `Mimir.slnx`. + - `src/Mimir.BufferSmoke`: the synthetic sync proofs, run with `dotnet run --project src/Mimir.BufferSmoke/Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test` (also `--chirp-bin-self-test`, `--passive-sync-self-test`). + - `native/probes/asio_audio_cadence` (ASIO cadence probe) and `native/reservoir` (native rolling-buffer ABI). +- **Live:** passive, chirp-only and hybrid audio sync with synthetic proofs, the ASIO cadence probe and measured capture-cadence evidence, the runtime buffers, and FFmpeg/SRT bridge scripts as an OBS-compatible edge (`scripts/sender-start.ps1`). +- **Planned:** direct native capture workers with driver-boundary timestamps, Fensalir GPU fusion, the volumetric sound field, realtime 4D Gaussian splatting, motion capture and AR anchoring. The bridge scripts are not the synchronization authority. + A normal streaming setup sees loose devices. A webcam has its clock, a mic has another clock, speakers add their own delay, and OBS receives whatever arrives. That is enough for a flat livestream. It is not enough to know where a voice @@ -289,6 +301,7 @@ Run the old OBS bridge only when you need the compatibility path: ```powershell .\scripts\sender-discover.ps1 +Copy-Item .\config\localcast.example.json .\config\localcast.json .\scripts\sender-start.ps1 -Config .\config\localcast.json -DryRun ``` From 43cebb6fb9ee82314bd372954392b31fc8e39346 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 22 Oct 2026 20:14:47 +0200 Subject: [PATCH 134/135] Quarantine Mimir's Eve operator surface: remove EveDashboard, EveBrowserReference and their contract documents Parked at tag parked/Mimir-eve-surface per thing2:ruling:eve-dashboards-quarantined; rebuild on Thing 2. Mimir.EveSensorReceiver (camera frame ingest) stays. map.yaml no longer calls a live listener a tombstone. Co-Authored-By: Claude Sonnet 5.5 --- config/perfect-machine.example.json | 6 +- docs/architecture.md | 2 +- docs/code-algorithm-map.md | 7 +- docs/implementation-plan.md | 22 +- docs/mimir-face.md | 3 +- docs/mimir-program-composition.md | 24 +- notes/current-system-map.md | 18 +- src/Mimir.BufferSmoke/Program.cs | 26 +- .../Mimir.EveBrowserReference.csproj | 17 - src/Mimir.EveBrowserReference/Program.cs | 734 ---- .../Mimir.EveDashboard.csproj | 19 - src/Mimir.EveDashboard/Program.cs | 3517 ----------------- .../Synchronization/MimirCultMeshContracts.cs | 170 - .../MimirProgramPublicationConfiguration.cs | 20 +- state/map.yaml | 18 +- 15 files changed, 19 insertions(+), 4584 deletions(-) delete mode 100644 src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj delete mode 100644 src/Mimir.EveBrowserReference/Program.cs delete mode 100644 src/Mimir.EveDashboard/Mimir.EveDashboard.csproj delete mode 100644 src/Mimir.EveDashboard/Program.cs diff --git a/config/perfect-machine.example.json b/config/perfect-machine.example.json index 36868d7..e4e40c7 100644 --- a/config/perfect-machine.example.json +++ b/config/perfect-machine.example.json @@ -44,7 +44,6 @@ "authorities": { "configurationCalibrationLaunchStatus": "Mimir", "programComposition": "Mimir", - "operatorSurfaces": "Eve", "sitePublication": "Yggdrasil publisher daemon", "denseVisualFusion": "Fensalir", "gpuMaterialAndBrushReconciliation": "Fensalir", @@ -97,10 +96,8 @@ ] }, "operator": { - "owner": "Mimir/Eve", + "owner": "Mimir", "surfaces": [ - "eve-gui-compositor", - "eve-tui-operator", "cultmesh-command-surface" ] }, @@ -120,7 +117,6 @@ "native-reservoir", "Fensalir", "Faust", - "Eve", "Yggdrasil publisher daemon" ], "neverInCompatibilityEdges": [ diff --git a/docs/architecture.md b/docs/architecture.md index c2195d9..fad3f52 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -29,7 +29,7 @@ Inputs become outputs like this: - OBS can receive ordinary media-source URLs for compatibility; it does not need a plugin for this bridge path. - Audio remains separately mixable at the compatibility edge, but the durable - control surface belongs to Mimir/Eve. + control surface belongs to Mimir. - Ports are stable and derived from one base port plus explicit offsets. - Secrets, if used, stay in local config and are not committed. diff --git a/docs/code-algorithm-map.md b/docs/code-algorithm-map.md index 7be4449..8a7724c 100644 --- a/docs/code-algorithm-map.md +++ b/docs/code-algorithm-map.md @@ -16,7 +16,7 @@ Timestamp for this pass: 2026-05-23, starting at 23:24 Europe/London. Mimir collects local and network audio/video samples into bounded rolling runtime buffers, uses active/passive timing evidence to place audio streams on a canonical timeline, and lets Mimir own program composition while Fensalir lowers -pixels, Faust/native DSP moves audio, Eve exposes operator surfaces, and +pixels, Faust/native DSP moves audio, and Yggdrasil publishes the public feed. ## Current Mechanism @@ -61,8 +61,6 @@ reflection taps. program video output. - Faust/native DSP will own hot resampling, fractional delay, separation, spatialization, and stems. -- Eve GUI/TUI lowerers render Mimir operator surfaces without owning scene - truth. - The Yggdrasil-facing publisher daemon consumes Mimir program output without owning composition. - OBS is a compatibility sink only. @@ -91,7 +89,6 @@ flowchart TD Probes["native/probes/*"] --> FrameEvents["MimirFrameEventProcessStreamSource"] FrameEvents --> Sources Runtime --> Scene["Mimir program scene graph"] - Scene --> Eve["Eve operator surfaces"] Scene --> Publisher["Yggdrasil site publisher"] Scene -. "compatibility" .-> Obs["OBS adapter"] Reservoir["native/reservoir"] -. "future lower ABI" .-> Hub @@ -108,7 +105,7 @@ flowchart TD - `native/probes/*` are measurement and diagnostic tools. - `scripts/*` are bridge utilities, not the final synchronized program path. - `docs/mimir-program-composition.md` is the authority map for Muninn capture - hosts, Mimir composition, Eve operation, Yggdrasil publication, and OBS + hosts, Mimir composition, Yggdrasil publication, and OBS compatibility. ## Stage 0: Solution And App Host diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 3d6e6bf..7e9cb6c 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -365,14 +365,12 @@ a named invariant that the native runtime cannot protect yet. - Windows bridge scripts for sender discovery/start/stop and simple OBS Media Source compatibility ingest. - `src/Mimir.Runtime/Synchronization` now contains the first typed program - composition contracts: `mimir.program_scene.v1`, `mimir.program_output.v1`, - and `mimir.eve_operator_surface.v1`. These make Mimir's scene graph, - Yggdrasil/site publication route, and Eve operator control/preview surface + composition contracts: `mimir.program_scene.v1` and `mimir.program_output.v1`. + These make Mimir's scene graph and Yggdrasil/site publication route manifestable through CultMesh. - `docs/mimir-program-composition.md` is the current authority map for the new stream-program architecture: Muninn observes on Starfire/Nightwing/Raven-class - hosts, Mimir consumes selected streams and composes the program, Eve lowers - controls/previews/stats, Yggdrasil publishes to the site, and OBS is + hosts, Mimir consumes selected streams and composes the program, Yggdrasil publishes to the site, and OBS is compatibility-only. - `src/Mimir.CultMeshMedia` is the first explicit CultMesh media/body bridge: `relay` hosts the document relay on CultNet RUDP `3075`, `send` reads an @@ -399,23 +397,13 @@ a named invariant that the native runtime cannot protect yet. capture still defaults to FFmpeg desktop frames plus Mimir's WASAPI loopback capture muxed as H.264/AAC MPEG-TS; DirectShow audio remains an explicit fallback. -- `src/Mimir.EveDashboard` is archived until the dashboard returns as a pure - CultMesh/Odin publisher and Eve lowering. The old HTTP/WebSocket deck broker, - local health route, socket command channel, and provider route catalog are not - daemon transport. Dashboard state must be published as typed CultMesh/Eve - documents through Odin, and dashboard commands must arrive as typed Odin/ - CultMesh command documents. -- `src/Mimir.EveBrowserReference` serves static browser lowerings and can - publish its own `idunn.daemon_health` record over - `cultnet.transport.rudp.v0`. It remains a renderer/reference surface, not - program authority. - Documentation for OBS receiver setup, native rebuild boundaries, the viable stream app, and the Mimir Face. ## Temporary - Audio and video may still traverse separate OBS/SRT endpoints during bridge - testing, but Mimir/Eve owns independent controls. + testing, but Mimir owns independent controls. - The CultMesh media bridge still lowers to local UDP for compatibility sinks because OBS is not a CultMesh consumer. Network transit between Raven, Yggdrasil, and Starfire is the Odin-discovered CultMesh/CultNet path; @@ -533,7 +521,7 @@ proxy impairment profiles and long soak remain the completion gate. 9. Implement the Mimir program scene graph as the shared commit primitive for source subscription, transforms, crop, chroma key, visibility, layer order, preview, and output publication. Import the current OBS scene only as an - initial mirror, then make Eve GUI/TUI the operator surface. + initial mirror; the operator surface returns on Thing 2. 10. Add the Yggdrasil-facing site publisher daemon that consumes the Mimir program output and publishes it without owning a second composition. 11. Lower `AquariumBufferFieldFrame` spline tube fields into Fensalir compute: diff --git a/docs/mimir-face.md b/docs/mimir-face.md index 40efc7f..e50cab6 100644 --- a/docs/mimir-face.md +++ b/docs/mimir-face.md @@ -49,7 +49,7 @@ distributed capture surfaces into one realtime coherent volumetric field. The field is the product. The bridge scripts are scaffolding and probes. The live target is one native five-second rolling reservoir feeding Mimir-owned -program composition, Fensalir GPU fusion, Faust/native DSP, Eve operator +program composition, Fensalir GPU fusion, Faust/native DSP, operator surfaces, and a Yggdrasil site publisher. OBS is compatibility only. Retention has one owner. Typed views are indexes, not private histories. @@ -70,7 +70,6 @@ Invariants: program texture output. - Faust/native DSP owns hot audio alignment, room suppression, voice separation, stems, and spatial bed generation. -- Eve lowerers render operator controls and preview without owning scene truth. - The Yggdrasil publisher daemon ships the Mimir program to the site without owning composition. - FFmpeg owns simple capture, encode, and transport in V1 bridge mode. diff --git a/docs/mimir-program-composition.md b/docs/mimir-program-composition.md index 03bffe4..02561a1 100644 --- a/docs/mimir-program-composition.md +++ b/docs/mimir-program-composition.md @@ -11,14 +11,14 @@ temporary compatibility sink only. - Owner: Mimir owns the program scene graph and subscription policy. - Inputs: Muninn-published local and remote monitor, window, camera, audio, loopback, and health streams. -- Outputs: a Mimir program video surface, aligned audio stems or final bus, Eve - GUI/TUI operator surfaces, and a Yggdrasil-facing site publication feed. +- Outputs: a Mimir program video surface, aligned audio stems or final bus, + and a Yggdrasil-facing site publication feed. - Derived state: OBS receives a compatibility feed when useful; it does not own scene truth, broadcast truth, preview truth, stats, source selection, or publication. - Forbidden writers: OBS scenes, browser dashboards, site daemons, and Muninn capture hosts must not independently decide the program composition. -- Shared paths: direct operator edits, remote Eve commands, preset loads, +- Shared paths: direct operator edits, remote commands, preset loads, calibration-driven source selection, and recovery scripts must commit through the same Mimir scene graph command surface. - Deletion line: old OBS-owned composition docs and config names are demoted to @@ -39,7 +39,6 @@ flowchart TD F --> I["Faust/native DSP"] H --> J["program video surface"] I --> K["program audio stems/bus"] - G --> L["Eve GUI/TUI operator surfaces"] J --> M["Yggdrasil site publisher daemon"] K --> M M --> N["GameCult site"] @@ -68,22 +67,7 @@ The current OBS scene can be mirrored as an initial Mimir scene: active. After import, OBS is not the editor of record. It may display the Mimir program -feed, but operator edits happen through Mimir/Eve command surfaces. - -## Eve Operator Surface - -Mimir publishes Eve GUI/TUI documents for: - -- live program preview; -- source catalog and subscription state; -- layer transform, crop, chroma key, visibility, and ordering; -- audio stem gain/mute/route controls; -- stream health, latency, buffer depth, timing confidence, and publication - status; -- emergency scene selection and publisher recovery commands. - -Eve lowerers render those controls on any device without owning the underlying -scene graph. +feed, but operator edits happen through Mimir command surfaces. ## Site Publisher diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 86e1235..20dbdf7 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -28,7 +28,6 @@ flowchart TD N --> F["Faust/native DSP"] E --> G["Mimir program video"] F --> H["program stems + spatial bed"] - E --> I["Eve GUI/TUI operator surfaces"] G --> J["Yggdrasil site publisher"] H --> J G -. "compatibility" .-> K["OBS adapter"] @@ -53,17 +52,10 @@ Ownership: Fensalir-facing wand pose stream. - Mimir owns program composition, source subscription policy, preview/control state, stats, and publication intent. -- Mimir's old Eve dashboard broker is archived; it does not publish health, - provider catalogs, or command surfaces while the socket path is cut. The - browser reference remains a client lowering, not daemon truth. Dashboard state - must return as typed CultMesh/Eve documents through Odin; product/debug render - surfaces are client lowerings and compatibility evidence, not daemon truth. - Fensalir owns dense visual fusion, material/brush/splat reconciliation, D3D12 interop, runtime UI lowering, and local program texture output. - Faust/native DSP owns hot audio alignment, suppression, separation, spatialization, and stem generation. -- Eve GUI/TUI lowerers render Mimir's operator surfaces on any device without - owning scene truth. - The Yggdrasil-facing publisher daemon consumes Mimir program output and publishes it to the site without owning composition. - OBS is a temporary compatibility sink, not a composition or broadcast @@ -302,7 +294,7 @@ evidence. authority map for stream-program output. Muninn runs on Starfire, Nightwing, and Raven-class capture hosts to publish local stream capabilities and selected media bodies. Mimir consumes only the streams it needs for calibration and -composition, commits one typed scene graph, emits Eve GUI/TUI operator surfaces, +composition, commits one typed scene graph, and publishes program output locally plus through a Yggdrasil-facing site publisher. @@ -439,14 +431,6 @@ an RUDP contract exists. Mimir's current daemon-health publishers are local RUDP witnesses: -- `src/Mimir.EveDashboard` is a fail-closed archive tombstone. It no longer - starts a TCP listener, serves `/eve/deck`, accepts WebSocket commands, or - publishes daemon health. `mimir.eve_dashboard_state` must return through Odin/ - CultMesh state publication before this service can be redeployed. -- `src/Mimir.EveBrowserReference` serves a static browser lowering and, when - configured with `--idunn-rudp-health` or - `MIMIR_EVE_BROWSER_REFERENCE_IDUNN_RUDP_HEALTH`, sends its own - `idunn.daemon_health` record over the same RUDP document path. Invariant: health publication is not ownership. The daemon reports what it can honestly observe about itself; Idunn decides keepalive action through the diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index f62b020..02e88c2 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -1477,10 +1477,6 @@ static int RunPerfectMachineProfileSmoke() "mimir-site-program", scene.SceneId, MimirProgramPublicationConfigurations.YggdrasilSiteProgram); - var eveSurface = MimirCultMeshContractFactory.CreateOperatorSurface( - "mimir-eve-gui-compositor", - scene.SceneId, - MimirProgramPublicationConfigurations.OperatorSurfaces[0]); var observations = new[] { new MimirBioacousticWordObservation(0, MimirBioacousticTimeline.Default.EventForIndex(0).StartSeconds * MimirBioacousticTimeline.SampleRate + 317.375, 0.95), @@ -1565,7 +1561,7 @@ static int RunPerfectMachineProfileSmoke() Console.WriteLine( $"perfect-machine-actuator source={actuator.SourceId} delay={actuator.TargetDelaySamples:0.000} ratio={actuator.ResampleRatio:0.000000000} controls={actuator.FaustControls.Length}"); Console.WriteLine( - $"perfect-machine-program scene={scene.SceneId} layers={scene.Layers.Length} output={output.OutputId} route={output.PublisherRoute} eveCommands={eveSurface.CommandTopics.Length}"); + $"perfect-machine-program scene={scene.SceneId} layers={scene.Layers.Length} output={output.OutputId} route={output.PublisherRoute}"); Console.WriteLine( $"perfect-machine-clock anchors={clock?.AnchorCount ?? 0} coverage={clock?.AnchorCoverage ?? 0:0.000} offset={clock?.SourceOffsetSamples ?? double.NaN:0.000} confidence={clock?.Confidence ?? 0:0.000}"); Console.WriteLine( @@ -3927,11 +3923,6 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) scene.SceneId, MimirProgramPublicationConfigurations.YggdrasilSiteProgram, createdAt); - var eveSurface = MimirCultMeshContractFactory.CreateOperatorSurface( - "mimir-eve-gui-compositor", - scene.SceneId, - MimirProgramPublicationConfigurations.OperatorSurfaces[0], - createdAt); var starfireMove = MimirTrackingObservation.PsMove( "starfire:move:primary", "psmove:starfire:00", @@ -4004,10 +3995,6 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) output, new CultRecordHandle(new CultRecordKey($"mimir-program-output:{output.OutputId}"))) .ConfigureAwait(false); - await cache.UpsertAsync( - eveSurface, - new CultRecordHandle(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}"))) - .ConfigureAwait(false); await cache.UpsertAsync( starfireMove, new CultRecordHandle(new CultRecordKey($"mimir-move-tracking:{starfireMove.ObservationId}"))) @@ -4023,7 +4010,7 @@ static async Task RunPerfectMachineContractSmokeAsync(string outputPath) await cache.FlushAsync().ConfigureAwait(false); Console.WriteLine( - $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} eveCommands={eveSurface.CommandTopics.Length} moveTracking=2 fusedMovePose={fusedMovePose.PoseId} authority={fusedMovePose.FusionAuthorityId} consumer={fusedMovePose.ConsumerContract} hosts={starfireMove.HostId},{nightwingMove.HostId}"); + $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length} sceneLayers={scene.Layers.Length} programOutput={output.OutputId} moveTracking=2 fusedMovePose={fusedMovePose.PoseId} authority={fusedMovePose.FusionAuthorityId} consumer={fusedMovePose.ConsumerContract} hosts={starfireMove.HostId},{nightwingMove.HostId}"); return 0; } @@ -4045,11 +4032,6 @@ static async Task ImportObsProgramSceneAsync(string inputPath, string scene scene.SceneId, MimirProgramPublicationConfigurations.YggdrasilSiteProgram, importedAt); - var eveSurface = MimirCultMeshContractFactory.CreateOperatorSurface( - "mimir-eve-gui-compositor", - scene.SceneId, - MimirProgramPublicationConfigurations.OperatorSurfaces[0], - importedAt); Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); using var cache = await CultCacheMessagePack.OpenAsync(outputPath).ConfigureAwait(false); @@ -4061,10 +4043,6 @@ static async Task ImportObsProgramSceneAsync(string inputPath, string scene output, new CultRecordHandle(new CultRecordKey($"mimir-program-output:{output.OutputId}"))) .ConfigureAwait(false); - await cache.UpsertAsync( - eveSurface, - new CultRecordHandle(new CultRecordKey($"mimir-eve-surface:{eveSurface.SurfaceId}"))) - .ConfigureAwait(false); await cache.FlushAsync().ConfigureAwait(false); var visible = scene.Layers.Count(layer => layer.Visible); diff --git a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj b/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj deleted file mode 100644 index ad5a698..0000000 --- a/src/Mimir.EveBrowserReference/Mimir.EveBrowserReference.csproj +++ /dev/null @@ -1,17 +0,0 @@ - - - - Exe - net10.0 - Mimir.EveBrowserReference - Mimir.EveBrowserReference - enable - enable - - - - - - - - diff --git a/src/Mimir.EveBrowserReference/Program.cs b/src/Mimir.EveBrowserReference/Program.cs deleted file mode 100644 index b1a3ef9..0000000 --- a/src/Mimir.EveBrowserReference/Program.cs +++ /dev/null @@ -1,734 +0,0 @@ -using System.Globalization; -using System.Net; -using System.Net.Sockets; -using System.Security.Cryptography; -using System.Text; -using System.Text.Json; -using System.Text.Json.Serialization; -using GameCult.Caching; -using GameCult.Caching.MessagePack; -using GameCult.Networking; -using MessagePack; - -var port = ParseInt(args, "--port", 8891); -var bind = ParseString(args, "--bind", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_BIND") ?? "0.0.0.0"); -var directory = ParseString(args, "--directory", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_DIRECTORY") ?? Directory.GetCurrentDirectory()); -var cultCacheWitnessPath = ParseString( - args, - "--cultcache-witness", - Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_CULTCACHE_WITNESS") ?? DefaultPaths.CultCacheWitnessPath); -var idunnRudpHealth = ParseString(args, "--idunn-rudp-health", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_RUDP_HEALTH") ?? ""); -var idunnDaemon = ParseString(args, "--idunn-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_DAEMON") ?? "nightwing-eve-browser-reference"); -var idunnHealthContract = ParseString(args, "--idunn-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_BROWSER_REFERENCE_IDUNN_HEALTH_CONTRACT") ?? "nightwing.cultnet-rudp-browser-reference-health"); - -using var verseRuntime = EveBrowserReferenceVerseRuntime.Create(cultCacheWitnessPath); -using var server = new EveBrowserReferenceServer( - IPAddress.Parse(bind), - port, - Path.GetFullPath(directory), - verseRuntime, - new IdunnRudpHealthOptions(idunnRudpHealth, idunnDaemon, idunnHealthContract)); -await server.RunAsync().ConfigureAwait(false); - -static int ParseInt(IReadOnlyList args, string name, int fallback) -{ - for (var index = 0; index < args.Count - 1; index++) - { - if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && - int.TryParse(args[index + 1], CultureInfo.InvariantCulture, out var value)) - { - return value; - } - } - - return fallback; -} - -static string ParseString(IReadOnlyList args, string name, string fallback) -{ - for (var index = 0; index < args.Count - 1; index++) - { - if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && - !string.IsNullOrWhiteSpace(args[index + 1])) - { - return args[index + 1]; - } - } - - return fallback; -} - -internal static class DefaultPaths -{ - public static string CultCacheWitnessPath => - OperatingSystem.IsWindows() - ? @"E:\Projects\Mimir\state\eve-browser-reference.service.cc" - : "/var/lib/gamecult/eve-browser-reference/cultcache/eve-browser-reference.service.cc"; -} - -internal sealed class EveBrowserReferenceServer( - IPAddress bindAddress, - int port, - string documentRoot, - EveBrowserReferenceVerseRuntime verseRuntime, - IdunnRudpHealthOptions idunnRudpHealth) : IDisposable -{ - private static readonly JsonSerializerOptions JsonOptions = new() - { - PropertyNamingPolicy = JsonNamingPolicy.CamelCase, - DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, - }; - - private readonly TcpListener listener = new(bindAddress, port); - private readonly CancellationTokenSource stopping = new(); - private readonly object idunnRudpPublishLock = new(); - private string idunnRudpPublishStatus = string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) ? "disabled" : "idle"; - private string idunnRudpPublishError = ""; - private string idunnRudpPublishObservedAt = ""; - private bool idunnRudpPublishInFlight; - private DateTimeOffset lastHealthPublishAttemptAt = DateTimeOffset.MinValue; - private DateTimeOffset lastWitnessPublishAttemptAt = DateTimeOffset.MinValue; - - public async Task RunAsync() - { - if (!Directory.Exists(documentRoot)) - { - throw new DirectoryNotFoundException($"Eve browser reference document root does not exist: {documentRoot}"); - } - - listener.Start(); - Console.WriteLine($"Mimir Eve browser reference serving {documentRoot} on http://{bindAddress}:{port}/"); - Console.WriteLine(string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) - ? "Mimir Eve browser reference Idunn RUDP health publisher disabled." - : $"Mimir Eve browser reference Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} as {idunnRudpHealth.DaemonId}."); - Console.WriteLine($"Mimir Eve browser reference witness store publishing to {verseRuntime.WitnessPath}"); - _ = Task.Run(PublishHealthLoopAsync); - while (!stopping.IsCancellationRequested) - { - var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false); - _ = Task.Run(() => HandleAsync(client)); - } - } - - private async Task PublishHealthLoopAsync() - { - using var timer = new PeriodicTimer(TimeSpan.FromSeconds(1)); - while (!stopping.IsCancellationRequested && await timer.WaitForNextTickAsync(stopping.Token).ConfigureAwait(false)) - { - var observedAt = DateTimeOffset.UtcNow; - PublishWitnessIfDue(observedAt); - QueueIdunnRudpHealthIfDue(observedAt); - } - } - - private async Task HandleAsync(TcpClient tcpClient) - { - await using var stream = tcpClient.GetStream(); - var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false); - if (request.Path == "/health") - { - var health = JsonSerializer.Serialize(new - { - ok = true, - documentRoot, - port, - transport = "static-http-lowering", - cultCacheWitness = verseRuntime.GetWitnessStatus(), - idunnRudpHealth = GetIdunnRudpHealthStatus(), - }, JsonOptions); - await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false); - return; - } - - var filePath = ResolveStaticFilePath(request.Path); - if (filePath == null) - { - await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false); - return; - } - - var bytes = await File.ReadAllBytesAsync(filePath, stopping.Token).ConfigureAwait(false); - await WriteHttpResponseAsync(stream, "200 OK", ContentType(filePath), bytes).ConfigureAwait(false); - } - - private IdunnRudpHealthStatus GetIdunnRudpHealthStatus() - { - lock (idunnRudpPublishLock) - { - return new IdunnRudpHealthStatus( - idunnRudpHealth.Endpoint, - idunnRudpHealth.DaemonId, - idunnRudpHealth.HealthContract, - idunnRudpPublishStatus, - idunnRudpPublishError, - idunnRudpPublishInFlight, - idunnRudpPublishObservedAt); - } - } - - private void PublishWitnessIfDue(DateTimeOffset observedAt) - { - if (lastWitnessPublishAttemptAt != DateTimeOffset.MinValue && - observedAt - lastWitnessPublishAttemptAt < TimeSpan.FromSeconds(5)) - { - return; - } - - lastWitnessPublishAttemptAt = observedAt; - verseRuntime.Publish( - CreateDaemonHealthRecord(observedAt), - documentRoot, - port, - observedAt); - } - - private void QueueIdunnRudpHealthIfDue(DateTimeOffset observedAt) - { - if (string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint)) - { - idunnRudpPublishStatus = "disabled"; - return; - } - - if (lastHealthPublishAttemptAt != DateTimeOffset.MinValue && - observedAt - lastHealthPublishAttemptAt < TimeSpan.FromSeconds(5)) - { - return; - } - - var health = CreateDaemonHealthRecord(observedAt); - var observedAtText = health.ObservedAt; - - lock (idunnRudpPublishLock) - { - if (idunnRudpPublishInFlight) - { - return; - } - - idunnRudpPublishInFlight = true; - lastHealthPublishAttemptAt = observedAt; - idunnRudpPublishObservedAt = observedAtText; - if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) - { - idunnRudpPublishStatus = "publishing"; - idunnRudpPublishError = ""; - } - } - - _ = Task.Run(() => - { - try - { - IdunnRudpHealthPublisher.Publish(idunnRudpHealth.Endpoint, health); - lock (idunnRudpPublishLock) - { - idunnRudpPublishStatus = "published"; - idunnRudpPublishError = ""; - idunnRudpPublishInFlight = false; - } - } - catch (Exception error) - { - lock (idunnRudpPublishLock) - { - if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) - { - idunnRudpPublishStatus = "publish-error"; - } - - idunnRudpPublishError = error.Message; - idunnRudpPublishInFlight = false; - } - } - }); - } - - private IdunnDaemonHealthRecord CreateDaemonHealthRecord(DateTimeOffset observedAt) - { - var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); - return new IdunnDaemonHealthRecord - { - DaemonId = idunnRudpHealth.DaemonId, - State = "healthy", - Detail = $"Nightwing Eve browser reference serving static lowering; root={documentRoot}; port={port}; witness={verseRuntime.WitnessPath}", - ObservedAt = observedAtText, - HealthContract = idunnRudpHealth.HealthContract, - PublicationSource = "daemon-published", - Transport = "cultnet.transport.rudp.v0", - }; - } - - private string? ResolveStaticFilePath(string path) - { - var requestPath = path.Split('?', 2)[0]; - if (requestPath == "/") - { - requestPath = "/index.html"; - } - - var relativePath = Uri.UnescapeDataString(requestPath.TrimStart('/')) - .Replace('/', Path.DirectorySeparatorChar); - var candidate = Path.GetFullPath(Path.Combine(documentRoot, relativePath)); - if (!candidate.StartsWith(documentRoot.TrimEnd(Path.DirectorySeparatorChar) + Path.DirectorySeparatorChar, StringComparison.Ordinal) && - !string.Equals(candidate, documentRoot, StringComparison.Ordinal)) - { - return null; - } - - return File.Exists(candidate) ? candidate : null; - } - - private static string ContentType(string path) => - Path.GetExtension(path).ToLowerInvariant() switch - { - ".html" => "text/html; charset=utf-8", - ".css" => "text/css; charset=utf-8", - ".js" => "application/javascript; charset=utf-8", - ".json" => "application/json; charset=utf-8", - ".png" => "image/png", - ".jpg" or ".jpeg" => "image/jpeg", - ".svg" => "image/svg+xml", - _ => "application/octet-stream", - }; - - private static async Task ReadHttpRequestAsync(NetworkStream stream) - { - var buffer = new byte[8192]; - var length = await stream.ReadAsync(buffer).ConfigureAwait(false); - if (length == 0) - { - return new HttpRequest("/"); - } - - var text = Encoding.ASCII.GetString(buffer, 0, length); - var firstLineEnd = text.IndexOf("\r\n", StringComparison.Ordinal); - var firstLine = firstLineEnd >= 0 ? text[..firstLineEnd] : text; - var parts = firstLine.Split(' ', StringSplitOptions.RemoveEmptyEntries); - return new HttpRequest(parts.Length >= 2 ? parts[1] : "/"); - } - - private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body) - { - var header = Encoding.ASCII.GetBytes( - $"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n"); - await stream.WriteAsync(header).ConfigureAwait(false); - await stream.WriteAsync(body).ConfigureAwait(false); - } - - public void Dispose() - { - stopping.Cancel(); - listener.Stop(); - stopping.Dispose(); - } -} - -internal sealed record HttpRequest(string Path); - -internal sealed record IdunnRudpHealthOptions(string Endpoint, string DaemonId, string HealthContract); - -internal sealed record CultCacheWitnessStatus( - string Path, - string Status, - string Error, - string ObservedAtUtc, - IReadOnlyList PublishedSchemas); - -internal sealed record IdunnRudpHealthStatus( - string Endpoint, - string Daemon, - string Contract, - string Status, - string Error, - bool InFlight, - string ObservedAtUtc); - -[CultDocument("idunn.daemon_health", "idunn.daemon_health.v1")] -[MessagePackObject(AllowPrivate = true)] -internal sealed class IdunnDaemonHealthRecord -{ - [Key(0)] public string DaemonId { get; set; } = "nightwing-eve-browser-reference"; - [Key(1)] public string State { get; set; } = "active"; - [Key(2)] public string Detail { get; set; } = ""; - [Key(3)] public string ObservedAt { get; set; } = ""; - [Key(4)] public string HealthContract { get; set; } = "nightwing.cultnet-rudp-browser-reference-health"; - [Key(5)] public string PublicationSource { get; set; } = "daemon-published"; - [Key(6)] public string Transport { get; set; } = "cultnet.transport.rudp.v0"; -} - -internal sealed class EveBrowserReferenceVerseRuntime : IDisposable -{ - private static readonly CultRecordKey ManifestKey = new("nightwing-eve-browser-reference"); - private static readonly CultRecordKey StaticSurfaceKey = new("nightwing-eve-browser-reference"); - private static readonly CultRecordKey CommandBoundaryKey = new("nightwing-eve-browser-reference"); - private static readonly CultRecordKey TransportProfileKey = new("nightwing-eve-browser-reference"); - private static readonly string[] PublishedSchemas = - [ - "mimir.eve_browser_reference_manifest.v1", - "mimir.eve_browser_reference_static_surface.v1", - "mimir.eve_browser_reference_command_boundary.v1", - "mimir.eve_browser_reference_transport_profile.v1", - "idunn.daemon_health.v1", - ]; - - private readonly CultDocumentRegistry registry = new(); - private readonly string witnessPath; - private readonly object publishLock = new(); - private string witnessStatus = "starting"; - private string witnessError = string.Empty; - private string witnessObservedAtUtc = string.Empty; - - private EveBrowserReferenceVerseRuntime(string witnessPath) - { - this.witnessPath = Path.GetFullPath(witnessPath); - } - - public string WitnessPath => witnessPath; - - public static EveBrowserReferenceVerseRuntime Create(string witnessPath) - { - Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(witnessPath)) ?? "."); - return new EveBrowserReferenceVerseRuntime(witnessPath); - } - - public CultCacheWitnessStatus GetWitnessStatus() - { - lock (publishLock) - { - return new CultCacheWitnessStatus( - witnessPath, - witnessStatus, - witnessError, - witnessObservedAtUtc, - PublishedSchemas); - } - } - - public void Publish( - IdunnDaemonHealthRecord healthRecord, - string documentRoot, - int port, - DateTimeOffset observedAt) - { - var observedAtUtc = observedAt.ToString("O", CultureInfo.InvariantCulture); - - try - { - lock (publishLock) - { - var records = new List(); - var schemas = new Dictionary(StringComparer.Ordinal); - var assets = EnumerateRelativeFiles(documentRoot); - var fixtures = assets - .Where(path => path.StartsWith("fixtures/", StringComparison.Ordinal)) - .ToArray(); - - AddRecord(records, schemas, ToManifestDocument(documentRoot, port, assets), ManifestKey, observedAtUtc); - AddRecord(records, schemas, ToStaticSurfaceDocument(documentRoot, assets, fixtures, observedAtUtc), StaticSurfaceKey, observedAtUtc); - AddRecord(records, schemas, ToCommandBoundaryDocument(observedAtUtc), CommandBoundaryKey, observedAtUtc); - AddRecord(records, schemas, ToTransportProfileDocument(observedAtUtc), TransportProfileKey, observedAtUtc); - AddRecord(records, schemas, healthRecord, new CultRecordKey(healthRecord.DaemonId), observedAtUtc); - - WriteSnapshot(new CultPersistedStoreSnapshot - { - FormatVersion = "cultcache.store.v1", - SchemaCatalog = schemas.Values.OrderBy(entry => entry.SchemaName, StringComparer.Ordinal).ToArray(), - Records = records.OrderBy(entry => entry.Key, StringComparer.Ordinal).ToArray(), - }); - witnessStatus = "published"; - witnessError = string.Empty; - witnessObservedAtUtc = observedAtUtc; - } - } - catch (Exception error) - { - lock (publishLock) - { - witnessStatus = "publish-error"; - witnessError = error.Message; - witnessObservedAtUtc = observedAtUtc; - } - } - } - - public void Dispose() - { - } - - private void AddRecord( - List records, - Dictionary schemas, - T document, - CultRecordKey key, - string observedAtUtc) where T : class - { - var descriptor = registry.GetRequired(); - var payload = MessagePackSerializer.Serialize(typeof(T), document, CultNetSchemaMessageSerialization.Options); - schemas[descriptor.SchemaId] = descriptor.ToCatalogEntry(); - records.Add(new CultPersistedRecord - { - Key = key.Value, - SchemaId = descriptor.SchemaId, - StoredAt = observedAtUtc, - Payload = payload, - }); - } - - private void WriteSnapshot(CultPersistedStoreSnapshot snapshot) - { - var payload = CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot); - var directory = Path.GetDirectoryName(witnessPath) ?? "."; - Directory.CreateDirectory(directory); - var tempPath = Path.Combine(directory, $".{Path.GetFileName(witnessPath)}.{Guid.NewGuid():N}.tmp"); - - try - { - using (var stream = new FileStream( - tempPath, - FileMode.CreateNew, - FileAccess.Write, - FileShare.None, - bufferSize: 4096, - FileOptions.WriteThrough)) - { - stream.Write(payload, 0, payload.Length); - stream.Flush(true); - } - - if (File.Exists(witnessPath)) - { - File.Replace(tempPath, witnessPath, null); - } - else - { - File.Move(tempPath, witnessPath); - } - } - finally - { - if (File.Exists(tempPath)) - { - File.Delete(tempPath); - } - } - } - - private static EveBrowserReferenceManifestRecord ToManifestDocument( - string documentRoot, - int port, - IReadOnlyList assets) => - new() - { - DaemonId = "nightwing-eve-browser-reference", - SiteTitle = "Nightwing Eve Browser Reference", - DocumentRoot = documentRoot, - EntryPoint = "index.html", - Port = port, - Assets = assets.ToArray(), - SurfaceTransport = "static-http-lowering", - }; - - private static EveBrowserReferenceStaticSurfaceRecord ToStaticSurfaceDocument( - string documentRoot, - IReadOnlyList assets, - IReadOnlyList fixtures, - string observedAtUtc) => - new() - { - DaemonId = "nightwing-eve-browser-reference", - UpdatedAtUtc = observedAtUtc, - DocumentRoot = documentRoot, - EntryPoint = "index.html", - Assets = assets.ToArray(), - FixtureDocuments = fixtures.ToArray(), - CurrentCutLine = "Static browser lowering assets are published into the daemon-owned witness store so Nightwing and Odin can reason about the lowering body without treating the HTTP service probe as lifecycle truth.", - }; - - private static EveBrowserReferenceCommandBoundaryRecord ToCommandBoundaryDocument(string observedAtUtc) => - new() - { - DaemonId = "nightwing-eve-browser-reference", - UpdatedAtUtc = observedAtUtc, - Mode = "static-read-only-lowering", - WritesAccepted = false, - OperatorInputAuthority = "served browser assets only; no write path is accepted by this daemon", - LifecycleAuthority = "idunn.local-command.restart + compatibility.systemd.nightwing-eve-browser-reference.service", - AcceptedCommands = [], - RejectedCommands = - [ - "surface-mutation", - "provider-catalog-ownership", - "daemon-health-override", - "service-restart", - ], - }; - - private EveBrowserReferenceTransportProfileRecord ToTransportProfileDocument(string observedAtUtc) => - new() - { - DaemonId = "nightwing-eve-browser-reference", - UpdatedAtUtc = observedAtUtc, - CurrentState = "partial-rudp-health-and-provider-store-live", - InputTransport = "compatibility.http-static-lowering", - OutputTransport = "daemon-owned-cultcache-boundary-store + daemon-published-rudp-health", - HealthContract = "nightwing.cultnet-rudp-browser-reference-health", - IdunnRudpHealth = "idunn.health-published-separately", - WitnessPath = witnessPath, - CurrentCutLine = "The Nightwing Eve browser reference runtime now owns its manifest, static surface witness, command boundary, and transport profile in a daemon-owned CultCache store; systemd and HTTP remain compatibility witnesses while projections learn to consume the typed store directly.", - }; - - private static string[] EnumerateRelativeFiles(string root) - { - if (!Directory.Exists(root)) - { - return []; - } - - return Directory - .EnumerateFiles(root, "*", SearchOption.AllDirectories) - .Select(path => Path.GetRelativePath(root, path).Replace('\\', '/')) - .OrderBy(path => path, StringComparer.Ordinal) - .ToArray(); - } -} - -[CultDocument("mimir.eve_browser_reference_manifest", "mimir.eve_browser_reference_manifest.v1")] -[MessagePackObject(AllowPrivate = true)] -internal sealed class EveBrowserReferenceManifestRecord -{ - [Key(0)] - [CultName] - public string DaemonId { get; set; } = string.Empty; - - [Key(1)] public string SiteTitle { get; set; } = string.Empty; - [Key(2)] public string DocumentRoot { get; set; } = string.Empty; - [Key(3)] public string EntryPoint { get; set; } = string.Empty; - [Key(4)] public int Port { get; set; } - [Key(5)] public string[] Assets { get; set; } = []; - [Key(6)] public string SurfaceTransport { get; set; } = string.Empty; -} - -[CultDocument("mimir.eve_browser_reference_static_surface", "mimir.eve_browser_reference_static_surface.v1")] -[MessagePackObject(AllowPrivate = true)] -internal sealed class EveBrowserReferenceStaticSurfaceRecord -{ - [Key(0)] - [CultName] - public string DaemonId { get; set; } = string.Empty; - - [Key(1)] public string UpdatedAtUtc { get; set; } = string.Empty; - [Key(2)] public string DocumentRoot { get; set; } = string.Empty; - [Key(3)] public string EntryPoint { get; set; } = string.Empty; - [Key(4)] public string[] Assets { get; set; } = []; - [Key(5)] public string[] FixtureDocuments { get; set; } = []; - [Key(6)] public string CurrentCutLine { get; set; } = string.Empty; -} - -[CultDocument("mimir.eve_browser_reference_command_boundary", "mimir.eve_browser_reference_command_boundary.v1")] -[MessagePackObject(AllowPrivate = true)] -internal sealed class EveBrowserReferenceCommandBoundaryRecord -{ - [Key(0)] - [CultName] - public string DaemonId { get; set; } = string.Empty; - - [Key(1)] public string UpdatedAtUtc { get; set; } = string.Empty; - [Key(2)] public string Mode { get; set; } = string.Empty; - [Key(3)] public bool WritesAccepted { get; set; } - [Key(4)] public string OperatorInputAuthority { get; set; } = string.Empty; - [Key(5)] public string LifecycleAuthority { get; set; } = string.Empty; - [Key(6)] public string[] AcceptedCommands { get; set; } = []; - [Key(7)] public string[] RejectedCommands { get; set; } = []; -} - -[CultDocument("mimir.eve_browser_reference_transport_profile", "mimir.eve_browser_reference_transport_profile.v1")] -[MessagePackObject(AllowPrivate = true)] -internal sealed class EveBrowserReferenceTransportProfileRecord -{ - [Key(0)] - [CultName] - public string DaemonId { get; set; } = string.Empty; - - [Key(1)] public string UpdatedAtUtc { get; set; } = string.Empty; - [Key(2)] public string CurrentState { get; set; } = string.Empty; - [Key(3)] public string InputTransport { get; set; } = string.Empty; - [Key(4)] public string OutputTransport { get; set; } = string.Empty; - [Key(5)] public string HealthContract { get; set; } = string.Empty; - [Key(6)] public string IdunnRudpHealth { get; set; } = string.Empty; - [Key(7)] public string WitnessPath { get; set; } = string.Empty; - [Key(8)] public string CurrentCutLine { get; set; } = string.Empty; -} - -internal static class IdunnRudpHealthPublisher -{ - private const uint ConnectionId = 0x1d0d0001; - - public static void Publish(string endpoint, IdunnDaemonHealthRecord health) - { - var remote = ParseEndpoint(endpoint); - using var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp); - socket.Bind(new IPEndPoint(IPAddress.Any, 0)); - socket.Blocking = false; - socket.ReceiveTimeout = 100; - socket.SendTimeout = 1000; - using var transport = new CultNetRudpSocketTransportConnection(new CultNetRudpSocketTransportOptions - { - RuntimeId = "mimir-eve-browser-reference", - Socket = socket, - Mode = CultNetRudpSocketMode.Client, - RemoteEndPoint = remote, - ConnectionId = ConnectionId, - InitialSequence = 1, - ResendDelayMs = 100, - }); - - if (!transport.ConnectAndWait(Array.Empty(), TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(20))) - { - throw new TimeoutException($"timed out connecting Mimir Eve browser reference RUDP health publisher to {endpoint}"); - } - - var observedAt = string.IsNullOrWhiteSpace(health.ObservedAt) - ? DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture) - : health.ObservedAt; - health.ObservedAt = observedAt; - var message = new CultNetDocumentPutRawMessage - { - MessageId = $"mimir-eve-browser-reference-health:{health.DaemonId}:{observedAt.Replace(':', '-')}", - Document = new CultNetRawDocumentRecord - { - SchemaId = "idunn.daemon_health", - RecordKey = health.DaemonId, - StoredAt = observedAt, - PayloadEncoding = "messagepack", - Payload = MessagePackSerializer.Serialize(health, CultNetSchemaMessageSerialization.Options), - SourceRuntimeId = "mimir-eve-browser-reference", - SourceRole = "daemon-health-publisher", - Tags = ["cultnet.transport.rudp.v0"], - }, - }; - transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message)); - } - - private static IPEndPoint ParseEndpoint(string value) - { - var trimmed = value.Trim(); - var separator = trimmed.LastIndexOf(':'); - if (separator <= 0 || separator == trimmed.Length - 1) - { - throw new ArgumentException($"Idunn RUDP endpoint must be host:port, got '{value}'.", nameof(value)); - } - - var host = trimmed[..separator]; - var portText = trimmed[(separator + 1)..]; - if (!int.TryParse(portText, CultureInfo.InvariantCulture, out var port)) - { - throw new ArgumentException($"Idunn RUDP endpoint port must be numeric, got '{value}'.", nameof(value)); - } - - var addresses = Dns.GetHostAddresses(host); - var address = addresses.FirstOrDefault(static candidate => candidate.AddressFamily == AddressFamily.InterNetwork) - ?? throw new ArgumentException($"Idunn RUDP endpoint host did not resolve to IPv4, got '{value}'.", nameof(value)); - return new IPEndPoint(address, port); - } -} diff --git a/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj b/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj deleted file mode 100644 index b18d295..0000000 --- a/src/Mimir.EveDashboard/Mimir.EveDashboard.csproj +++ /dev/null @@ -1,19 +0,0 @@ - - - - Exe - net10.0 - Mimir.EveDashboard - Mimir.EveDashboard - enable - enable - - - - - - - - - - diff --git a/src/Mimir.EveDashboard/Program.cs b/src/Mimir.EveDashboard/Program.cs deleted file mode 100644 index 8ac03d9..0000000 --- a/src/Mimir.EveDashboard/Program.cs +++ /dev/null @@ -1,3517 +0,0 @@ -using System.Collections.Concurrent; -using System.Globalization; -using System.Net; -using System.Net.Sockets; -using System.Security.Cryptography; -using System.Text; -using System.Text.Json; -using System.Text.Json.Serialization; -using GameCult.Caching; -using GameCult.Mesh; -using GameCult.Networking; -using MessagePack; -using Mimir.Runtime.Synchronization; - -var port = ParseInt(args, "--port", 8795); -var cultNetPort = ParseInt(args, "--cultnet-port", 3075); -var cultNetCachePath = ParseString(args, "--cultnet-cache", @"E:\Projects\Mimir\state\eve-dashboard-cultnet.ccmp"); -var voidBotSwarmStatePath = ParseString(args, "--voidbot-swarm-state", @"E:\Projects\VoidBot\.voidbot\status\swarm-state.json"); -var mimirTelemetryLogPath = ParseString(args, "--mimir-telemetry-log", ""); -var mimirObservationLogPath = ParseString(args, "--mimir-observation-log", @"E:\Projects\Mimir\artifacts\runtime\periwinkle-cultmesh-sensors.out.log"); -var idunnRudpHealth = ParseString(args, "--idunn-rudp-health", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_RUDP_HEALTH") ?? ""); -var idunnDaemon = ParseString(args, "--idunn-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_DAEMON") ?? "mimir-eve-dashboard"); -var idunnHealthContract = ParseString(args, "--idunn-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_HEALTH_CONTRACT") ?? "mimir.cultnet-rudp-provider-health"); -var idunnServiceDaemon = ParseString(args, "--idunn-service-daemon", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_SERVICE_DAEMON") ?? ""); -var idunnServiceHealthContract = ParseString(args, "--idunn-service-health-contract", Environment.GetEnvironmentVariable("MIMIR_EVE_DASHBOARD_IDUNN_SERVICE_HEALTH_CONTRACT") ?? ""); -var providers = EveDashboardProviderCatalog.Create(ParseProviderSpecs(args), voidBotSwarmStatePath, mimirTelemetryLogPath, mimirObservationLogPath); -using var cultNetPublisher = await EveDashboardCultNetPublisher.StartAsync(cultNetCachePath, cultNetPort).ConfigureAwait(false); -using var server = new EveDashboardServer(port, providers, cultNetPublisher, IdunnRudpHealthOptions.Create(idunnRudpHealth, idunnDaemon, idunnHealthContract, idunnServiceDaemon, idunnServiceHealthContract)); -await server.RunAsync(); - -static int ParseInt(IReadOnlyList args, string name, int fallback) -{ - for (var index = 0; index < args.Count - 1; index++) - { - if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && - int.TryParse(args[index + 1], out var value)) - { - return value; - } - } - - return fallback; -} - -static IReadOnlyList ParseProviderSpecs(IReadOnlyList args) -{ - var specs = new List(); - for (var index = 0; index < args.Count - 1; index++) - { - if (string.Equals(args[index], "--provider", StringComparison.OrdinalIgnoreCase)) - { - specs.Add(args[index + 1]); - } - } - - return specs; -} - -static string ParseString(IReadOnlyList args, string name, string fallback) -{ - for (var index = 0; index < args.Count - 1; index++) - { - if (string.Equals(args[index], name, StringComparison.OrdinalIgnoreCase) && - !string.IsNullOrWhiteSpace(args[index + 1])) - { - return args[index + 1]; - } - } - - return fallback; -} - -internal sealed class EveDashboardServer(int port, EveDashboardProviderCatalog providers, EveDashboardCultNetPublisher cultNetPublisher, IdunnRudpHealthOptions idunnRudpHealth) : IDisposable -{ - private static readonly JsonSerializerOptions JsonOptions = new() - { - PropertyNamingPolicy = JsonNamingPolicy.CamelCase, - DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, - }; - - private readonly TcpListener listener = new(IPAddress.Any, port); - private readonly CancellationTokenSource stopping = new(); - private readonly ConcurrentDictionary clients = new(); - private readonly object idunnRudpPublishLock = new(); - private string activeProviderId = providers.DefaultProviderId; - private string idunnRudpPublishStatus = string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) ? "disabled" : "idle"; - private string idunnRudpPublishError = ""; - private string idunnRudpPublishObservedAt = ""; - private bool idunnRudpPublishInFlight; - private DateTimeOffset lastHealthPublishAttemptAt = DateTimeOffset.MinValue; - - public async Task RunAsync() - { - listener.Start(); - Console.WriteLine($"Mimir Eve dashboard broker listening on ws://0.0.0.0:{port}/eve/deck"); - Console.WriteLine($"Mimir Eve dashboard CultNet publisher listening on cultnet://0.0.0.0:{cultNetPublisher.Port}"); - Console.WriteLine(string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint) - ? "Mimir Eve dashboard Idunn RUDP health publisher disabled." - : $"Mimir Eve dashboard Idunn RUDP health publisher targeting {idunnRudpHealth.Endpoint} for {idunnRudpHealth.Targets.Count} record(s)."); - Console.WriteLine($"Compatibility endpoint remains ws://0.0.0.0:{port}/eve/dashboard"); - _ = Task.Run(BroadcastHeartbeatAsync); - while (!stopping.IsCancellationRequested) - { - var client = await listener.AcceptTcpClientAsync(stopping.Token).ConfigureAwait(false); - _ = Task.Run(() => HandleAsync(client)); - } - } - - private async Task BroadcastHeartbeatAsync() - { - using var timer = new PeriodicTimer(TimeSpan.FromSeconds(1)); - while (!stopping.IsCancellationRequested && await timer.WaitForNextTickAsync(stopping.Token).ConfigureAwait(false)) - { - await PublishCultNetStateAsync().ConfigureAwait(false); - if (clients.IsEmpty) - { - continue; - } - - await BroadcastStateAsync().ConfigureAwait(false); - } - } - - private async Task HandleAsync(TcpClient tcpClient) - { - await using var stream = tcpClient.GetStream(); - var request = await ReadHttpRequestAsync(stream).ConfigureAwait(false); - Console.WriteLine($"EVE dashboard connected: {tcpClient.Client.RemoteEndPoint} {request.Path}"); - Console.Out.Flush(); - - if (request.Path == "/health") - { - var provider = providers.Get(activeProviderId); - var health = JsonSerializer.Serialize(new - { - ok = true, - clients = clients.Count, - activeProviderId, - providerVersion = provider.State.Version, - providers = providers.Manifests.Count, - transport = "eve-deck-ws", - cultMeshDocument = "mimir.eve_dashboard_state", - idunnRudpHealth = GetIdunnRudpHealthStatus(), - }, JsonOptions); - await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(health)).ConfigureAwait(false); - return; - } - - if (request.Path is "/eve/deck/manifest" or "/eve/dashboard/manifest") - { - var manifest = JsonSerializer.Serialize(providers.BrokerManifest, JsonOptions); - await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(manifest)).ConfigureAwait(false); - return; - } - - if (request.Path == "/eve/deck/providers") - { - var catalog = JsonSerializer.Serialize(new DashboardProviderCatalogDocument(providers.Manifests), JsonOptions); - await WriteHttpResponseAsync(stream, "200 OK", "application/json", Encoding.UTF8.GetBytes(catalog)).ConfigureAwait(false); - return; - } - - if (!IsDashboardPath(request.Path, out var requestedProviderId, out var cultMeshBinary) || - !request.Headers.TryGetValue("Sec-WebSocket-Key", out var key)) - { - await WriteHttpResponseAsync(stream, "404 Not Found", "text/plain", Encoding.UTF8.GetBytes("not found")).ConfigureAwait(false); - return; - } - - if (!string.IsNullOrWhiteSpace(requestedProviderId) && providers.Contains(requestedProviderId)) - { - activeProviderId = requestedProviderId; - } - - await WriteWebSocketHandshakeAsync(stream, key).ConfigureAwait(false); - var id = Guid.NewGuid(); - var socket = new DashboardSocket(tcpClient, stream, cultMeshBinary); - clients[id] = socket; - await SendStateAsync(socket).ConfigureAwait(false); - try - { - while (!stopping.IsCancellationRequested) - { - var frame = await ReceiveFrameAsync(stream).ConfigureAwait(false); - if (frame.Opcode == 0x8) - { - await WriteCloseFrameAsync(stream).ConfigureAwait(false); - break; - } - - if (frame.Opcode == 0x2 && cultMeshBinary) - { - if (ApplyCultMeshCommand(frame.Payload)) - { - await BroadcastStateAsync().ConfigureAwait(false); - } - - continue; - } - - if (frame.Opcode != 0x1) - { - continue; - } - - var text = Encoding.UTF8.GetString(frame.Payload); - if (ApplyCommand(text)) - { - await BroadcastStateAsync().ConfigureAwait(false); - } - } - } - finally - { - clients.TryRemove(id, out _); - } - } - - private static bool IsDashboardPath(string path, out string providerId, out bool cultMeshBinary) - { - providerId = ""; - cultMeshBinary = false; - if (path == "/eve/deck/cultmesh") - { - cultMeshBinary = true; - return true; - } - - const string cultMeshProviderPrefix = "/eve/deck/cultmesh/"; - if (path.StartsWith(cultMeshProviderPrefix, StringComparison.Ordinal)) - { - providerId = Uri.UnescapeDataString(path[cultMeshProviderPrefix.Length..]); - cultMeshBinary = true; - return !string.IsNullOrWhiteSpace(providerId); - } - - if (path == "/eve/dashboard" || path == "/eve/deck") - { - return true; - } - - const string providerPrefix = "/eve/deck/"; - if (path.StartsWith(providerPrefix, StringComparison.Ordinal)) - { - providerId = Uri.UnescapeDataString(path[providerPrefix.Length..]); - return !string.IsNullOrWhiteSpace(providerId); - } - - return false; - } - - private bool ApplyCultMeshCommand(byte[] payload) - { - MimirEveDashboardCommandDocument? commandDocument; - try - { - commandDocument = MessagePackSerializer.Deserialize(payload); - } - catch - { - return false; - } - - var command = new DashboardCommand - { - Type = commandDocument.Type, - NodeId = commandDocument.NodeId, - ProviderId = commandDocument.ProviderId, - X = commandDocument.X, - Y = commandDocument.Y, - Rotation = commandDocument.Rotation, - Scale = commandDocument.Scale, - Visible = commandDocument.Visible, - }; - - return ApplyCommand(JsonSerializer.Serialize(command, JsonOptions)); - } - - private bool ApplyCommand(string text) - { - DashboardCommand? command; - try - { - command = JsonSerializer.Deserialize(text, JsonOptions); - } - catch (JsonException) - { - return false; - } - - if (command == null) - { - return false; - } - - if (string.Equals(command.Type, "open-provider", StringComparison.OrdinalIgnoreCase)) - { - var providerId = command.ProviderId; - if (string.IsNullOrWhiteSpace(providerId) && !string.IsNullOrWhiteSpace(command.NodeId)) - { - providerId = providers.Get(activeProviderId).State.Nodes - .FirstOrDefault(node => string.Equals(node.Id, command.NodeId, StringComparison.Ordinal)) - ?.ProviderId; - } - - if (!string.IsNullOrWhiteSpace(providerId) && providers.Contains(providerId)) - { - activeProviderId = providerId; - Console.WriteLine($"EVE dashboard provider switched: {activeProviderId}"); - Console.Out.Flush(); - return true; - } - - return false; - } - - var provider = providers.Get(activeProviderId); - if (provider.ApplyCommand(command)) - { - Console.WriteLine($"EVE dashboard command({activeProviderId}): {text}"); - Console.Out.Flush(); - return true; - } - - return false; - } - - private async Task SendStateAsync(DashboardSocket socket) - { - var state = providers.Get(activeProviderId).State; - if (socket.CultMeshBinary) - { - await SendBinaryFrameAsync(socket.Stream, MessagePackSerializer.Serialize(ToCultMeshState(state))).ConfigureAwait(false); - return; - } - - await SendTextFrameAsync(socket.Stream, JsonSerializer.Serialize(state, JsonOptions)).ConfigureAwait(false); - } - - private Task PublishCultNetStateAsync() - { - var state = providers.Get(activeProviderId).State; - QueueIdunnRudpHealthIfDue(state, DateTimeOffset.UtcNow); - return cultNetPublisher.PublishAsync(ToCultMeshState(state), stopping.Token); - } - - private IdunnRudpHealthStatus GetIdunnRudpHealthStatus() - { - lock (idunnRudpPublishLock) - { - return new IdunnRudpHealthStatus( - idunnRudpHealth.Endpoint, - idunnRudpHealth.Targets, - idunnRudpPublishStatus, - idunnRudpPublishError, - idunnRudpPublishInFlight, - idunnRudpPublishObservedAt); - } - } - - private void QueueIdunnRudpHealthIfDue(DashboardState state, DateTimeOffset observedAt) - { - if (string.IsNullOrWhiteSpace(idunnRudpHealth.Endpoint)) - { - idunnRudpPublishStatus = "disabled"; - return; - } - - if (lastHealthPublishAttemptAt != DateTimeOffset.MinValue && - observedAt - lastHealthPublishAttemptAt < TimeSpan.FromSeconds(5)) - { - return; - } - - var observedAtText = observedAt.ToString("O", CultureInfo.InvariantCulture); - var healthRecords = idunnRudpHealth.Targets.Select(target => new IdunnDaemonHealthRecord - { - DaemonId = target.DaemonId, - State = "healthy", - Detail = target.DetailPrefix == "service" - ? $"Nightwing Eve dashboard service active; provider={state.ProviderId}; providerVersion={state.Version}; providers={providers.Manifests.Count}; clients={clients.Count}; cultMeshPort={cultNetPublisher.Port}; process=Mimir.EveDashboard" - : $"Mimir Eve dashboard broker active; provider={state.ProviderId}; providerVersion={state.Version}; providers={providers.Manifests.Count}; clients={clients.Count}; cultMeshPort={cultNetPublisher.Port}", - ObservedAt = observedAtText, - HealthContract = target.HealthContract, - PublicationSource = "daemon-published", - Transport = "cultnet.transport.rudp.v0", - }).ToArray(); - - lock (idunnRudpPublishLock) - { - if (idunnRudpPublishInFlight) - { - return; - } - - idunnRudpPublishInFlight = true; - lastHealthPublishAttemptAt = observedAt; - idunnRudpPublishObservedAt = observedAtText; - if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) - { - idunnRudpPublishStatus = "publishing"; - idunnRudpPublishError = ""; - } - } - - _ = Task.Run(() => - { - try - { - foreach (var health in healthRecords) - { - IdunnRudpHealthPublisher.Publish(idunnRudpHealth.Endpoint, health); - } - - lock (idunnRudpPublishLock) - { - idunnRudpPublishStatus = "published"; - idunnRudpPublishError = ""; - idunnRudpPublishInFlight = false; - } - } - catch (Exception error) - { - lock (idunnRudpPublishLock) - { - if (!string.Equals(idunnRudpPublishStatus, "published", StringComparison.Ordinal)) - { - idunnRudpPublishStatus = "publish-error"; - } - - idunnRudpPublishError = error.Message; - idunnRudpPublishInFlight = false; - } - } - }); - } - - private static MimirEveDashboardStateDocument ToCultMeshState(DashboardState state) => - new( - state.ProviderId, - state.Title, - state.Version, - state.UpdatedAt.ToString("O"), - state.SelectedNodeId, - state.LutPreset, - state.Nodes.Select(static node => new MimirEveDashboardNodeSnapshot( - node.Id, - node.Label, - node.Kind, - node.Visible, - node.X, - node.Y, - node.Z, - node.Rotation, - node.Scale, - node.Width, - node.Height, - node.Health, - node.ProviderId, - node.Command, - node.Endpoint)).ToArray(), - ToCultMeshSurface(state.Surface)); - - private static MimirEveDashboardSurfaceSnapshot? ToCultMeshSurface(DashboardSurface? surface) - { - if (surface == null) - { - return null; - } - - return new MimirEveDashboardSurfaceSnapshot( - surface.Schema, - surface.Id, - surface.Title, - ToCultMeshElement(surface.Root), - surface.Assets - .Select(static asset => new MimirEveDashboardSurfaceAssetSnapshot(asset.Id, asset.Kind, asset.Uri)) - .ToArray()); - } - - private static MimirEveDashboardUiElementSnapshot ToCultMeshElement(DashboardUiElement element) - { - return new MimirEveDashboardUiElementSnapshot( - element.Id, - element.Kind, - element.Role, - element.Text, - element.AssetRef, - element.AssetUri, - element.BindNodeId, - element.CommandId, - element.Layout == null - ? null - : new MimirEveDashboardUiLayoutSnapshot( - element.Layout.Direction, - element.Layout.Width, - element.Layout.Height, - element.Layout.Grow, - element.Layout.Gap, - element.Layout.Padding, - element.Layout.Overflow, - element.Layout.MinWidth, - element.Layout.MinHeight, - element.Layout.PreferredWidth, - element.Layout.PreferredHeight, - element.Layout.Priority, - element.Layout.Density, - element.Layout.ViewportMode), - element.Style == null - ? null - : new MimirEveDashboardUiStyleSnapshot(element.Style.Variant, element.Style.Tone), - element.Metric == null - ? null - : new MimirEveDashboardUiMetricSnapshot(element.Metric.Label, element.Metric.Value, element.Metric.Tone), - element.Children.Select(ToCultMeshElement).ToArray(), - element.Binding == null - ? null - : new MimirEveDashboardUiBindingSnapshot( - element.Binding.DocumentSchema, - element.Binding.DocumentId, - element.Binding.Path, - element.Binding.ValueKind, - element.Binding.Access, - element.Binding.Authority, - element.Binding.CommandId)); - } - - private async Task BroadcastStateAsync() - { - foreach (var (id, socket) in clients) - { - try - { - await SendStateAsync(socket).ConfigureAwait(false); - } - catch - { - clients.TryRemove(id, out _); - } - } - } - - private static async Task ReadHttpRequestAsync(NetworkStream stream) - { - var bytes = new List(); - var lastFour = new Queue(4); - while (true) - { - var value = stream.ReadByte(); - if (value < 0) - { - throw new IOException("Client closed before HTTP headers completed."); - } - - var b = (byte)value; - bytes.Add(b); - lastFour.Enqueue(b); - if (lastFour.Count > 4) - { - lastFour.Dequeue(); - } - - if (lastFour.Count == 4 && lastFour.SequenceEqual(new byte[] { 13, 10, 13, 10 })) - { - break; - } - } - - var text = Encoding.ASCII.GetString(bytes.ToArray()); - var lines = text.Split("\r\n", StringSplitOptions.None); - var first = lines[0].Split(' '); - var headers = new Dictionary(StringComparer.OrdinalIgnoreCase); - foreach (var line in lines.Skip(1)) - { - var colon = line.IndexOf(':'); - if (colon > 0) - { - headers[line[..colon].Trim()] = line[(colon + 1)..].Trim(); - } - } - - await Task.CompletedTask.ConfigureAwait(false); - return new HttpRequest(first.Length > 1 ? first[1] : "/", headers); - } - - private static async Task WriteHttpResponseAsync(NetworkStream stream, string status, string contentType, byte[] body) - { - var header = Encoding.ASCII.GetBytes( - $"HTTP/1.1 {status}\r\nContent-Type: {contentType}\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n"); - await stream.WriteAsync(header).ConfigureAwait(false); - await stream.WriteAsync(body).ConfigureAwait(false); - } - - private static async Task WriteWebSocketHandshakeAsync(NetworkStream stream, string key) - { - var acceptBytes = SHA1.HashData(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")); - var accept = Convert.ToBase64String(acceptBytes); - var response = Encoding.ASCII.GetBytes( - "HTTP/1.1 101 Switching Protocols\r\n" + - "Upgrade: websocket\r\n" + - "Connection: Upgrade\r\n" + - $"Sec-WebSocket-Accept: {accept}\r\n\r\n"); - await stream.WriteAsync(response).ConfigureAwait(false); - } - - private static async Task SendTextFrameAsync(NetworkStream stream, string text) - { - var payload = Encoding.UTF8.GetBytes(text); - var header = new List { 0x81 }; - if (payload.Length < 126) - { - header.Add((byte)payload.Length); - } - else if (payload.Length <= ushort.MaxValue) - { - header.Add(126); - header.Add((byte)(payload.Length >> 8)); - header.Add((byte)payload.Length); - } - else - { - header.Add(127); - var length = BitConverter.GetBytes(IPAddress.HostToNetworkOrder((long)payload.Length)); - header.AddRange(length); - } - - await stream.WriteAsync(header.ToArray()).ConfigureAwait(false); - await stream.WriteAsync(payload).ConfigureAwait(false); - } - - private static async Task SendBinaryFrameAsync(NetworkStream stream, byte[] payload) - { - var header = BuildFrameHeader(0x82, payload.Length); - await stream.WriteAsync(header).ConfigureAwait(false); - await stream.WriteAsync(payload).ConfigureAwait(false); - } - - private static byte[] BuildFrameHeader(byte opcode, int payloadLength) - { - var header = new List { opcode }; - if (payloadLength < 126) - { - header.Add((byte)payloadLength); - } - else if (payloadLength <= ushort.MaxValue) - { - header.Add(126); - header.Add((byte)(payloadLength >> 8)); - header.Add((byte)payloadLength); - } - else - { - header.Add(127); - var length = BitConverter.GetBytes(IPAddress.HostToNetworkOrder((long)payloadLength)); - header.AddRange(length); - } - - return header.ToArray(); - } - - private static async Task ReceiveFrameAsync(NetworkStream stream) - { - var header = new byte[2]; - await ReadExactAsync(stream, header).ConfigureAwait(false); - var opcode = header[0] & 0x0f; - var masked = (header[1] & 0x80) != 0; - var length = header[1] & 0x7f; - if (length == 126) - { - var extended = new byte[2]; - await ReadExactAsync(stream, extended).ConfigureAwait(false); - length = (extended[0] << 8) | extended[1]; - } - else if (length == 127) - { - var extended = new byte[8]; - await ReadExactAsync(stream, extended).ConfigureAwait(false); - var longLength = 0L; - foreach (var b in extended) - { - longLength = (longLength << 8) | b; - } - - if (longLength > int.MaxValue) - { - throw new InvalidDataException("WebSocket frame is too large."); - } - - length = (int)longLength; - } - - var mask = new byte[4]; - if (masked) - { - await ReadExactAsync(stream, mask).ConfigureAwait(false); - } - - var payload = new byte[length]; - await ReadExactAsync(stream, payload).ConfigureAwait(false); - if (masked) - { - for (var index = 0; index < payload.Length; index++) - { - payload[index] ^= mask[index % 4]; - } - } - - return new WebSocketFrame(opcode, payload); - } - - private static async Task ReadExactAsync(NetworkStream stream, byte[] buffer) - { - var offset = 0; - while (offset < buffer.Length) - { - var read = await stream.ReadAsync(buffer.AsMemory(offset, buffer.Length - offset)).ConfigureAwait(false); - if (read == 0) - { - throw new IOException("WebSocket closed."); - } - - offset += read; - } - } - - private static Task WriteCloseFrameAsync(NetworkStream stream) => - stream.WriteAsync(new byte[] { 0x88, 0x00 }).AsTask(); - - public void Dispose() - { - stopping.Cancel(); - listener.Stop(); - stopping.Dispose(); - } -} - -internal sealed record IdunnRudpHealthOptions(string Endpoint, IReadOnlyList Targets) -{ - public static IdunnRudpHealthOptions Create( - string endpoint, - string daemonId, - string healthContract, - string serviceDaemonId, - string serviceHealthContract) - { - var targets = new List - { - new(daemonId, healthContract, "broker"), - }; - - if (!string.IsNullOrWhiteSpace(serviceDaemonId) && !string.IsNullOrWhiteSpace(serviceHealthContract)) - { - targets.Add(new IdunnRudpHealthTarget(serviceDaemonId, serviceHealthContract, "service")); - } - - return new IdunnRudpHealthOptions(endpoint, targets); - } -} - -internal sealed record IdunnRudpHealthTarget(string DaemonId, string HealthContract, string DetailPrefix); - -internal sealed record IdunnRudpHealthStatus( - string Endpoint, - IReadOnlyList Targets, - string Status, - string Error, - bool InFlight, - string ObservedAtUtc); - -[MessagePackObject(AllowPrivate = true)] -internal sealed class IdunnDaemonHealthRecord -{ - [Key(0)] public string DaemonId { get; set; } = "mimir-eve-dashboard"; - [Key(1)] public string State { get; set; } = "active"; - [Key(2)] public string Detail { get; set; } = ""; - [Key(3)] public string ObservedAt { get; set; } = ""; - [Key(4)] public string HealthContract { get; set; } = "mimir.cultnet-rudp-provider-health"; - [Key(5)] public string PublicationSource { get; set; } = "daemon-published"; - [Key(6)] public string Transport { get; set; } = "cultnet.transport.rudp.v0"; -} - -internal static class IdunnRudpHealthPublisher -{ - private const uint ConnectionId = 0x1d0d0001; - - public static void Publish(string endpoint, IdunnDaemonHealthRecord health) - { - var remote = ParseEndpoint(endpoint); - using var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp); - socket.Bind(new IPEndPoint(IPAddress.Any, 0)); - socket.Blocking = false; - socket.ReceiveTimeout = 100; - socket.SendTimeout = 1000; - using var transport = new CultNetRudpSocketTransportConnection(new CultNetRudpSocketTransportOptions - { - RuntimeId = "mimir-eve-dashboard", - Socket = socket, - Mode = CultNetRudpSocketMode.Client, - RemoteEndPoint = remote, - ConnectionId = ConnectionId, - InitialSequence = 1, - ResendDelayMs = 100, - }); - - if (!transport.ConnectAndWait(Array.Empty(), TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(20))) - { - throw new TimeoutException($"timed out connecting Mimir Eve dashboard RUDP health publisher to {endpoint}"); - } - - var observedAt = string.IsNullOrWhiteSpace(health.ObservedAt) - ? DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture) - : health.ObservedAt; - health.ObservedAt = observedAt; - var message = new CultNetDocumentPutRawMessage - { - MessageId = $"mimir-eve-dashboard-health:{health.DaemonId}:{observedAt.Replace(':', '-')}", - Document = new CultNetRawDocumentRecord - { - SchemaId = "idunn.daemon_health", - RecordKey = health.DaemonId, - StoredAt = observedAt, - PayloadEncoding = "messagepack", - Payload = MessagePackSerializer.Serialize(health, CultNetSchemaMessageSerialization.Options), - SourceRuntimeId = "mimir-eve-dashboard", - SourceRole = "daemon-health-publisher", - Tags = ["cultnet.transport.rudp.v0"], - }, - }; - transport.SendSchema(CultNetSchemaMessageSerialization.Serialize(message)); - } - - private static IPEndPoint ParseEndpoint(string value) - { - var trimmed = value.Trim(); - var separator = trimmed.LastIndexOf(':'); - if (separator <= 0 || separator == trimmed.Length - 1) - { - throw new ArgumentException($"Idunn RUDP endpoint must be host:port, got '{value}'.", nameof(value)); - } - - var host = trimmed[..separator].Trim('[', ']'); - var portText = trimmed[(separator + 1)..]; - if (!int.TryParse(portText, NumberStyles.Integer, CultureInfo.InvariantCulture, out var port) || - port <= 0 || port > 65535) - { - throw new ArgumentException($"Idunn RUDP endpoint port is invalid: {portText}", nameof(value)); - } - - var addresses = Dns.GetHostAddresses(host); - var address = addresses.FirstOrDefault(static candidate => candidate.AddressFamily == AddressFamily.InterNetwork) ?? - throw new ArgumentException($"Idunn RUDP endpoint host has no IPv4 address: {host}", nameof(value)); - return new IPEndPoint(address, port); - } -} - -internal sealed class EveDashboardCultNetPublisher : IDisposable -{ - private static readonly CultRecordKey StateRecordKey = new("eve.dashboard.broker"); - private readonly CultMeshNode node; - - private EveDashboardCultNetPublisher(CultMeshNode node, int port) - { - this.node = node; - Port = port; - } - - public int Port { get; } - - public static async Task StartAsync(string cachePath, int port) - { - if (port != 3075) - { - throw new ArgumentOutOfRangeException(nameof(port), port, "CultNet Server currently listens on the runtime default port 3075."); - } - - Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(cachePath)) ?? "."); - var documents = new CultNetDocumentRegistry() - .Register(CultNetDocumentBinding.ForDocument()); - var node = await CultMesh.CreateNodeAsync(cachePath, new CultMeshNodeOptions - { - StartServer = true, - DatabaseOptions = new CultNetDatabaseOptions - { - RuntimeId = "mimir.eve-dashboard", - DocumentRegistry = documents, - }, - }).ConfigureAwait(false); - - return new EveDashboardCultNetPublisher(node, port); - } - - public async Task PublishAsync(MimirEveDashboardStateDocument document, CancellationToken stopping) - { - await node.Database.PutAsync(new CultRecordKey(document.ProviderId), document).ConfigureAwait(false); - if (!string.Equals(document.ProviderId, StateRecordKey.Value, StringComparison.Ordinal)) - { - await node.Database.PutAsync(StateRecordKey, document).ConfigureAwait(false); - } - - await node.FlushAsync(soft: true).ConfigureAwait(false); - } - - public void Dispose() => node.Dispose(); -} - -internal sealed class EveDashboardProviderCatalog -{ - private readonly Dictionary providers; - - private EveDashboardProviderCatalog(IEnumerable providers) - { - this.providers = providers.ToDictionary(provider => provider.Manifest.Id, StringComparer.Ordinal); - DefaultProviderId = DashboardBrokerProvider.ProviderId; - Manifests = this.providers.Values.Select(provider => provider.Manifest).ToArray(); - BrokerManifest = new DashboardProviderManifest( - "eve.dashboard.broker", - "Eve Dashboard Broker", - "Native retained dashboard router for LAN and SSH-tunneled control surfaces.", - "1", - "/eve/deck", - ["scene2d", "provider-switching", "touch-commands", "cultmesh-state-documents"], - UsesCultMesh: true, - Transport: "WebSocket now; CultMesh typed state when available."); - } - - public string DefaultProviderId { get; } - - public IReadOnlyList Manifests { get; } - - public DashboardProviderManifest BrokerManifest { get; } - - public static EveDashboardProviderCatalog Create( - IReadOnlyList remoteSpecs, - string voidBotSwarmStatePath, - string mimirTelemetryLogPath, - string mimirObservationLogPath) - { - var providers = new List(); - var remoteProviders = remoteSpecs.Select(RemoteDashboardProvider.FromSpec).OfType().ToArray(); - providers.Add(new DashboardBrokerProvider(remoteProviders)); - providers.Add(new MimirLiveStatsProvider(mimirTelemetryLogPath, mimirObservationLogPath)); - providers.Add(new VoidBotSwarmProvider(voidBotSwarmStatePath)); - providers.Add(new YggdrasilStreamPixelsProvider()); - providers.AddRange(remoteProviders); - return new EveDashboardProviderCatalog(providers); - } - - public bool Contains(string providerId) => providers.ContainsKey(providerId); - - public IDashboardProvider Get(string providerId) => - providers.TryGetValue(providerId, out var provider) ? provider : providers[DefaultProviderId]; -} - -internal interface IDashboardProvider -{ - DashboardProviderManifest Manifest { get; } - - DashboardState State { get; } - - bool ApplyCommand(DashboardCommand command); -} - -internal abstract class MutableDashboardProvider : IDashboardProvider -{ - protected MutableDashboardProvider(DashboardProviderManifest manifest, DashboardState state) - { - Manifest = manifest; - State = state; - } - - public DashboardProviderManifest Manifest { get; } - - public DashboardState State { get; } - - public virtual bool ApplyCommand(DashboardCommand command) - { - if (string.IsNullOrWhiteSpace(command.NodeId)) - { - return false; - } - - var node = State.Nodes.FirstOrDefault(candidate => string.Equals(candidate.Id, command.NodeId, StringComparison.Ordinal)); - if (node == null) - { - return false; - } - - switch ((command.Type ?? "").Trim().ToLowerInvariant()) - { - case "select": - State.SelectedNodeId = node.Id; - break; - case "move": - State.SelectedNodeId = node.Id; - node.X = Clamp(command.X ?? node.X, -1.0, 1.0); - node.Y = Clamp(command.Y ?? node.Y, -1.0, 1.0); - break; - case "scale": - State.SelectedNodeId = node.Id; - node.Scale = Clamp(command.Scale ?? node.Scale, 0.25, 3.0); - break; - case "rotate": - State.SelectedNodeId = node.Id; - node.Rotation = command.Rotation ?? node.Rotation; - break; - case "toggle-visibility": - State.SelectedNodeId = node.Id; - node.Visible = command.Visible ?? !node.Visible; - break; - case "reset-transform": - State.SelectedNodeId = node.Id; - node.X = node.DefaultX; - node.Y = node.DefaultY; - node.Rotation = 0.0; - node.Scale = 1.0; - break; - default: - return false; - } - - Touch(); - return true; - } - - protected void Touch() - { - State.Version++; - State.UpdatedAt = DateTimeOffset.UtcNow; - } - - protected static double Clamp(double value, double min, double max) => Math.Min(max, Math.Max(min, value)); -} - -internal sealed class DashboardBrokerProvider : MutableDashboardProvider -{ - public const string ProviderId = "eve.dashboard.broker"; - - public DashboardBrokerProvider(IReadOnlyList remoteProviders) - : base( - new DashboardProviderManifest( - ProviderId, - "Dashboard Switchboard", - "Provider picker rendered as native Eve panels.", - "1", - "/eve/deck", - ["scene2d", "provider-switching"], - UsesCultMesh: true, - Transport: "broker-local"), - CreateState(remoteProviders)) - { - } - - private static DashboardState CreateState(IReadOnlyList remoteProviders) - { - var nodes = new List - { - new("provider-mimir-live", "Mimir Live Stats", "dashboard-provider", -0.56, -0.34, 0.30, 0.22, "telemetry") - { - ProviderId = MimirLiveStatsProvider.ProviderId, - Command = "open-provider", - }, - new("provider-mimir", "Mimir Stream Layout", "dashboard-provider", -0.18, -0.34, 0.30, 0.22, "local") - { - ProviderId = MimirStreamLayoutProvider.ProviderId, - Command = "open-provider", - }, - new("provider-voidbot", "VoidBot Swarm", "dashboard-provider", 0.0, -0.34, 0.30, 0.22, "ctb") - { - ProviderId = VoidBotSwarmProvider.ProviderId, - Command = "open-provider", - }, - new("provider-streampixels", "StreamPixels Edge", "dashboard-provider", 0.56, -0.34, 0.30, 0.22, "ssh ready") - { - ProviderId = YggdrasilStreamPixelsProvider.ProviderId, - Command = "open-provider", - }, - }; - - var y = 0.28; - foreach (var provider in remoteProviders) - { - nodes.Add(new DashboardNode( - $"provider-{provider.Manifest.Id}", - provider.Manifest.Title, - "dashboard-provider", - 0.0, - y, - 0.40, - 0.18, - "external") - { - ProviderId = provider.Manifest.Id, - Command = "open-provider", - }); - y = Math.Min(0.80, y + 0.22); - } - - return new DashboardState - { - ProviderId = ProviderId, - Title = "Eve Dashboard Switchboard", - SelectedNodeId = nodes[0].Id, - Nodes = nodes, - }; - } -} - -internal sealed class MimirLiveStatsProvider : IDashboardProvider -{ - public const string ProviderId = "mimir.live.stats"; - private const int TailBytes = 512 * 1024; - private readonly string telemetryLogPath; - private readonly string observationLogPath; - private long version; - - public MimirLiveStatsProvider(string telemetryLogPath, string observationLogPath) - { - this.telemetryLogPath = telemetryLogPath; - this.observationLogPath = observationLogPath; - Manifest = new DashboardProviderManifest( - ProviderId, - "Mimir Live Stats", - "Compact Eve/CultUI surface for live Mimir telemetry: RMS bars, sync confidence, source buffers, actuator state, and device observation streams.", - "1", - "/eve/deck/mimir.live.stats", - ["live-telemetry", "compact-tui", "audio-rms", "sync-confidence", "observation-streams"], - UsesCultMesh: true, - Transport: "Mimir telemetry and observation ledgers projected as Eve/CultUI state."); - } - - public DashboardProviderManifest Manifest { get; } - - public DashboardState State => BuildState(); - - public bool ApplyCommand(DashboardCommand command) => false; - - private DashboardState BuildState() - { - var snapshot = MimirLiveStatsSnapshot.Load(telemetryLogPath, observationLogPath); - var confidence = snapshot.SyncStates.Count == 0 - ? snapshot.SyncReports.Select(static report => report.Confidence) - .Concat(snapshot.SyncDecodeAttempts.Select(static attempt => attempt.Confidence)) - .Concat(snapshot.Well?.AudioSyncStates.Select(static state => state.Confidence) ?? []) - .DefaultIfEmpty(0.0) - .Max() - : snapshot.SyncStates.Average(static state => state.Confidence); - var rms = snapshot.Spectra.Count == 0 ? 0.0 : snapshot.Spectra.Max(static spectrum => spectrum.Rms); - var liveObservationCount = snapshot.ObservationStreams.Count(static stream => stream.State == "active"); - var status = snapshot.HasAnyData ? "live" : "waiting"; - var nodes = new List - { - new("mimir-live-root", $"Mimir Live Stats\n{snapshot.Summary}", "mimir-live-stats", 0.0, -0.50, 0.70, 0.20, status) - { - Detail = $"telemetry={snapshot.TelemetrySource} observations={snapshot.ObservationSource}", - }, - new("mimir-tracking-confidence", $"Tracking Confidence\n{confidence:0.000}", "confidence", -0.44, -0.12, 0.28, 0.16, ToneFor(confidence)) - { - Detail = $"{snapshot.SyncStates.Count} sync states / {snapshot.SyncReports.Count} reports / {snapshot.SyncDecodeAttempts.Count} decode attempts", - }, - new("mimir-audio-rms", $"Audio RMS\n{rms:0.000000}", "audio-rms", 0.0, -0.12, 0.28, 0.16, ToneFor(NormalizeRms(rms))) - { - Detail = $"{snapshot.Spectra.Count} spectrum lanes; Well audio buffers {snapshot.Well?.AudioBuffers ?? 0}", - }, - new("mimir-observation-streams", $"Device Streams\n{liveObservationCount}/{snapshot.ObservationStreams.Count} active", "observation-streams", 0.44, -0.12, 0.28, 0.16, liveObservationCount > 0 ? "active" : "waiting") - { - Detail = string.Join(", ", snapshot.ObservationStreams.Select(static stream => $"{stream.StreamId}:{stream.Kind}:{stream.State}")), - }, - }; - - return new DashboardState - { - ProviderId = ProviderId, - Title = "Mimir Live Stats", - Version = Interlocked.Increment(ref version), - UpdatedAt = DateTimeOffset.UtcNow, - SelectedNodeId = "mimir-live-root", - LutPreset = "terminal", - Nodes = nodes, - Surface = BuildSurface(snapshot, confidence, rms, liveObservationCount), - }; - } - - private static DashboardSurface BuildSurface(MimirLiveStatsSnapshot snapshot, double confidence, double rms, int liveObservationCount) - { - var pulse = Pulse(); - var well = snapshot.Well; - var liveRatio = snapshot.ObservationStreams.Count == 0 ? 0.0 : liveObservationCount / (double)snapshot.ObservationStreams.Count; - var staleStreams = snapshot.ObservationStreams.Count(static stream => stream.State != "active"); - var sourceDropouts = well == null ? 0 : Math.Max(0, well.ConfiguredSources - well.LiveSources) + well.SourceErrorCount + well.SourceHealth.Count(static source => source.Status != "live"); - var throughputScore = well == null - ? Math.Clamp(snapshot.Telemetry?.Ingested / 120.0 ?? 0.0, 0.0, 1.0) - : Math.Clamp((well.PublishedBytes / 262_144.0) / Math.Max(1.0, well.PublishAverageMs + well.PollAverageMs), 0.0, 1.0); - var latencyScore = well == null ? 0.0 : Math.Clamp(1.0 - (well.LatencyCurrentMs / Math.Max(1.0, well.LatencyCeilingMs)), 0.0, 1.0); - var dropoutScore = Math.Clamp(1.0 - ((sourceDropouts + staleStreams) / Math.Max(1.0, (well?.ConfiguredSources ?? 0) + snapshot.ObservationStreams.Count)), 0.0, 1.0); - var children = new List - { - UiElement.Pane("mimir-sync-signal", "Sync", [ - UiElement.Text("mimir-live-summary", $"{pulse} {snapshot.Summary}", "strong"), - StatBar("tracking-confidence", "confidence", confidence, ToneFor(confidence), $"{confidence:0.000}"), - UiElement.Text("mimir-sync-counts", $"states {snapshot.SyncStates.Count} reports {snapshot.SyncReports.Count} decodes {snapshot.SyncDecodeAttempts.Count}", "caption"), - UiElement.Text("mimir-sync-worst", WorstSyncLine(snapshot), "caption"), - ]), - UiElement.Pane("mimir-latency-signal", "Latency", [ - StatBar("mimir-latency-score", "headroom", latencyScore, ToneFor(latencyScore), well == null ? "none" : $"{well.LatencyCurrentMs:0}ms"), - UiElement.Text("mimir-latency-line", well == null ? "no Well snapshot" : $"floor {well.LatencyFloorMs:0}ms ceiling {well.LatencyCeilingMs:0}ms skew {well.LatencyEdgeSkewMs:0.0}ms", "mono"), - UiElement.Text("mimir-presentation-delay", well == null ? "" : $"presentation delay {well.PresentationDelayMs:0}ms overlap {well.LatencyRetainedOverlapMs:0}ms", "caption"), - ]), - UiElement.Pane("mimir-throughput-signal", "Throughput", [ - StatBar("mimir-throughput-score", "publish/poll", throughputScore, ToneFor(throughputScore)), - UiElement.Text("mimir-throughput-line", well == null ? $"ingested {snapshot.Telemetry?.Ingested ?? 0:0}" : $"poll {well.PollAverageMs:0.000}/{well.PollMaxMs:0.000}ms publish {well.PublishAverageMs:0.000}/{well.PublishMaxMs:0.000}ms", "mono"), - UiElement.Text("mimir-throughput-bytes", well == null ? $"analyze {snapshot.Telemetry?.AnalyzeMs ?? 0:0.0}ms" : $"bytes {well.PublishedBytes:0} buffers {well.Buffers.Count} audio {well.AudioBuffers} video {well.VideoBuffers}", "caption"), - ]), - UiElement.Pane("mimir-dropout-signal", "Dropout", [ - StatBar("mimir-dropout-score", "liveness", dropoutScore, ToneFor(dropoutScore), $"{liveObservationCount}/{snapshot.ObservationStreams.Count} streams"), - UiElement.Text("mimir-dropout-line", well == null ? $"stale streams {staleStreams}" : $"sources {well.LiveSources}/{well.ConfiguredSources} errors {well.SourceErrorCount} stale streams {staleStreams}", "mono"), - UiElement.Text("mimir-dropout-worst", WorstObservationLine(snapshot.ObservationStreams), "caption"), - ]), - }; - - return new DashboardSurface - { - Schema = "cultmesh.eve_surface.v0", - Id = "mimir.live.stats.surface", - Title = "Mimir Live Stats", - Root = UiElement.Container( - "mimir-live-stats-root", - "dashboard", - new DashboardUiLayout - { - Direction = "vertical", - Gap = 6, - Padding = 6, - Overflow = "scroll", - Grow = 4, - MinWidth = 96, - MinHeight = 56, - PreferredWidth = 156, - PreferredHeight = 128, - Priority = -80, - Density = "continuous", - ViewportMode = "continuous-metrics", - }, - children), - }; - } - - private static string WorstSyncLine(MimirLiveStatsSnapshot snapshot) - { - var worstState = snapshot.SyncStates - .OrderBy(static state => state.Confidence) - .FirstOrDefault(); - if (worstState != null) - { - return $"worst {worstState.ReferenceSourceId}->{worstState.SourceId} c={worstState.Confidence:0.000} delay {worstState.DelayMs:0.000}ms"; - } - - var worstReport = snapshot.SyncReports - .OrderBy(static report => report.Confidence) - .FirstOrDefault(); - return worstReport == null - ? "no sync state yet" - : $"report {worstReport.ReferenceSourceId}->{worstReport.SourceId} c={worstReport.Confidence:0.000} events {worstReport.Events}"; - } - - private static string WorstObservationLine(IReadOnlyList streams) - { - if (streams.Count == 0) - { - return "no observation streams"; - } - - var worst = streams - .OrderByDescending(static stream => stream.State == "active" ? 0 : 1) - .ThenByDescending(static stream => stream.AgeSeconds) - .First(); - return $"{worst.DeviceId}/{worst.StreamId} {worst.Kind} {worst.State} age {worst.AgeSeconds:0}s"; - } - - private static DashboardUiElement BuildAudioPane(IReadOnlyList spectra, MimirWellStat? well) - { - var wellAudio = well?.Buffers - .Where(static buffer => string.Equals(buffer.Kind, "Audio", StringComparison.OrdinalIgnoreCase)) - .OrderBy(static buffer => buffer.SourceId, StringComparer.Ordinal) - .Take(8) - .Select(buffer => UiElement.Text( - $"well-audio-{StableId(buffer.SourceId)}", - $"{buffer.SourceId}: {buffer.Count} blocks edge {NsAge(buffer.EdgeNs)}", - "caption")) - .ToArray() ?? []; - var items = spectra.Count == 0 && wellAudio.Length == 0 - ? [UiElement.Text("mimir-spectrum-empty", "no spectrum telemetry yet", "caption")] - : spectra - .OrderBy(static spectrum => spectrum.SourceId, StringComparer.Ordinal) - .Take(8) - .Select(spectrum => UiElement.Card( - $"spectrum-{StableId(spectrum.SourceId)}", - "audio-channel", - style: new DashboardUiStyle { Variant = "compact", Tone = ToneFor(NormalizeRms(spectrum.Rms)) }, - children: - [ - UiElement.Text($"spectrum-{StableId(spectrum.SourceId)}-label", $"{spectrum.SourceId} {Bar(NormalizeRms(spectrum.Rms), 18)}", "mono"), - UiElement.Text($"spectrum-{StableId(spectrum.SourceId)}-detail", $"rms {spectrum.Rms:0.000000} peak {spectrum.Peak:0.000000} floor {spectrum.NoiseFloorDb:0.0} dB", "caption"), - UiElement.Text($"spectrum-{StableId(spectrum.SourceId)}-peaks", spectrum.Peaks, "caption"), - ])) - .Concat(wellAudio) - .ToArray(); - - return UiElement.Pane("mimir-audio-pane", "Audio", items); - } - - private static DashboardUiElement BuildSyncPane(MimirLiveStatsSnapshot snapshot) - { - var children = new List(); - foreach (var state in snapshot.SyncStates.OrderBy(static state => state.SourceId, StringComparer.Ordinal).Take(8)) - { - children.Add(UiElement.Card( - $"sync-state-{StableId(state.SourceId)}", - "sync-state", - style: new DashboardUiStyle { Variant = "compact", Tone = ToneFor(state.Confidence) }, - children: - [ - UiElement.Text($"sync-state-{StableId(state.SourceId)}-head", $"{state.ReferenceSourceId}->{state.SourceId} {Bar(state.Confidence, 16)}", "mono"), - UiElement.Text($"sync-state-{StableId(state.SourceId)}-detail", $"delay {state.DelayUs:0.000} us / {state.DelayMs:0.000} ms sro {state.SroPpm:0.000} ppm", "caption"), - ])); - } - - foreach (var report in snapshot.SyncReports.OrderBy(static report => report.SourceId, StringComparer.Ordinal).Take(6)) - { - children.Add(UiElement.Text( - $"sync-report-{StableId(report.SourceId)}", - $"{report.Evidence} {report.ReferenceSourceId}->{report.SourceId} {report.DelayUs:0.000} us c={report.Confidence:0.000} events={report.Events}", - "caption")); - } - - foreach (var attempt in snapshot.SyncDecodeAttempts.OrderByDescending(static attempt => attempt.Confidence).ThenBy(static attempt => attempt.SourceId, StringComparer.Ordinal).Take(8)) - { - children.Add(UiElement.Text( - $"sync-decode-{StableId(attempt.ReferenceSourceId)}-{StableId(attempt.SourceId)}", - $"decode {attempt.Status} {attempt.ReferenceSourceId}->{attempt.SourceId} c={attempt.Confidence:0.000} ref={attempt.ReferenceEnergy:0.000} cand={attempt.CandidateEnergy:0.000} matched={attempt.Matched}", - "caption")); - } - - if (children.Count == 0) - { - children.Add(UiElement.Text("mimir-sync-empty", "no sync state yet", "caption")); - } - - return UiElement.Pane("mimir-sync-pane", "Sync Confidence", children); - } - - private static DashboardUiElement BuildVideoPane(MimirLiveStatsSnapshot snapshot) - { - var children = new List(); - if (snapshot.Telemetry != null) - { - children.Add(UiElement.Text( - "mimir-telemetry-line", - $"sources {snapshot.Telemetry.Sources} poll {snapshot.Telemetry.LastPoll} ingested {snapshot.Telemetry.Ingested} analyze {snapshot.Telemetry.AnalyzeMs:0.0} ms mode {snapshot.Telemetry.AudioSync}", - "mono")); - } - - children.AddRange(snapshot.VideoBuffers - .OrderBy(static video => video.SourceId, StringComparer.Ordinal) - .Take(8) - .Select(video => UiElement.Text( - $"video-{StableId(video.SourceId)}", - $"{video.SourceId}: {video.Count} frames {video.Latest}", - "caption"))); - - if (children.Count == 0) - { - children.Add(UiElement.Text("mimir-video-empty", "no video-buffer telemetry yet", "caption")); - } - - return UiElement.Pane("mimir-video-pane", "Runtime Buffers", children); - } - - private static DashboardUiElement BuildObservationPane(IReadOnlyList streams) - { - var children = streams.Count == 0 - ? [UiElement.Text("mimir-observations-empty", "no device observation streams", "caption")] - : streams - .OrderBy(static stream => stream.DeviceId, StringComparer.Ordinal) - .ThenBy(static stream => stream.StreamId, StringComparer.Ordinal) - .Select(stream => UiElement.Card( - $"observation-{StableId(stream.StreamId)}-{StableId(stream.Kind)}", - "observation-stream", - style: new DashboardUiStyle { Variant = "compact", Tone = stream.State == "active" ? "cool" : "warm" }, - children: - [ - UiElement.Text($"observation-{StableId(stream.StreamId)}-head", $"{stream.StreamId} {stream.Kind} {stream.State}", "mono"), - UiElement.Text($"observation-{StableId(stream.StreamId)}-detail", $"{stream.Shape} seq {stream.Sequence} age {stream.AgeSeconds:0}s", "caption"), - ])) - .ToArray(); - - return UiElement.Pane("mimir-observation-pane", "Periwinkle / Eve Streams", children); - } - - private static DashboardUiElement BuildActuatorPane(IReadOnlyList commands) - { - var children = commands.Count == 0 - ? [UiElement.Text("mimir-actuator-empty", "no actuator commands yet", "caption")] - : commands - .OrderBy(static command => command.SourceId, StringComparer.Ordinal) - .Take(8) - .Select(command => UiElement.Text( - $"actuator-{StableId(command.SourceId)}", - $"{command.SourceId}: delay {command.DelaySamples:0.000} samples ratio {command.Ratio:0.000000000} c={command.Confidence:0.000}", - "caption")) - .ToArray(); - - return UiElement.Pane("mimir-actuator-pane", "Alignment Actuator", children); - } - - private static DashboardUiElement BuildMoveMusicPane(MimirMoveMusicStat? music) - { - if (music == null) - { - return UiElement.Pane("mimir-move-music-pane", "Move Score", [UiElement.Text("mimir-move-music-empty", "no Move score trace yet", "caption")]); - } - - var children = new List - { - UiElement.Text("mimir-move-music-head", $"bpm {music.Bpm:0.0} c={music.BpmConfidence:0.00} {music.KeyName} {music.KeyMode} chord {music.ChordName}", "mono"), - StatBar("mimir-move-loudness", "loudness gate", music.LoudnessGate, "warm", $"{music.LoudnessGate:0.000}"), - UiElement.Text("mimir-move-score", $"score {Bar(music.ScoreEnvelopeMax, 18)} max env {music.ScoreEnvelopeMax:0.000} max rgb {music.MaxRgb}", "caption"), - UiElement.Text("mimir-move-sources", string.Join(" | ", music.Sources.Take(4)), "caption"), - }; - children.AddRange(music.Voices - .Take(5) - .Select(voice => UiElement.Text( - $"mimir-move-voice-{StableId(voice.SourceId)}", - $"{voice.SourceId}: {voice.NoteName} {voice.FrequencyHz:0.0}Hz c={voice.Confidence:0.000} {voice.Role}", - "caption"))); - children.Add(UiElement.Text("mimir-move-piano-roll", music.PianoRoll, "mono")); - children.AddRange(music.MoveTargets - .Take(5) - .Select(target => UiElement.Text( - $"mimir-move-target-{target.MoveIndex}", - $"move {target.MoveIndex}: {target.NoteName} {target.SourceId} priority {target.CalibrationPriority:0.000}", - "caption"))); - - return UiElement.Pane("mimir-move-music-pane", "Move Score", children); - } - - private static DashboardUiElement BuildWellPane(MimirWellStat? well) - { - if (well == null) - { - return UiElement.Pane("mimir-well-pane", "Well", [UiElement.Text("mimir-well-empty", "no Well snapshot yet", "caption")]); - } - - var children = new List - { - UiElement.Text("mimir-well-summary", $"seq {well.Sequence} {well.ElapsedSeconds:0}s ingested {well.IngestedSamples:0} sources {well.LiveSources}/{well.ConfiguredSources} source errors {well.SourceErrorCount}", "mono"), - StatBar("mimir-well-completeness", "frame completeness", well.SynchronizedFrameComplete ? 1.0 : 0.0, well.SynchronizedFrameComplete ? "cool" : "danger"), - UiElement.Text("mimir-well-frame", $"frame ready {well.ReadySlices}/{well.TotalSlices} delay {well.PresentationDelayMs:0}ms degraded {well.FrameDegradedKind} {well.FrameDegradedReason}", "caption"), - UiElement.Text("mimir-well-latency", $"latency {well.LatencyCurrentMs:0}ms floor {well.LatencyFloorMs:0} ceiling {well.LatencyCeilingMs:0} overlap {well.LatencyRetainedOverlapMs:0}ms skew {well.LatencyEdgeSkewMs:0.0}ms {well.LatencyReason}", "caption"), - UiElement.Text("mimir-well-pressure", $"poll {well.PollAverageMs:0.000}/{well.PollMaxMs:0.000}ms zero {well.ZeroPollIterations} publish {well.PublishAverageMs:0.000}/{well.PublishMaxMs:0.000}ms bytes {well.PublishedBytes:0}", "caption"), - UiElement.Text("mimir-well-buffers", $"buffers {well.Buffers.Count} audio {well.AudioBuffers} video {well.VideoBuffers} capture bodies {well.CaptureInlineBodies}", "caption"), - UiElement.Text("mimir-well-clock-maps", $"canonical clocks {well.CanonicalClockMaps.Count} max offset {well.MaxCanonicalClockOffsetMs:0.000}ms", "caption"), - UiElement.Text("mimir-well-features", $"video features stable {well.FeatureStableTracks} c={well.FeatureMeanConfidence:0.000} motion {well.FeatureMeanMotionPixelsPerSecond:0.0}px/s faust signals {well.FeatureSignals.Count}", "caption"), - UiElement.Text("mimir-well-probe", well.Probe == null ? "probe: none" : $"probe emit={well.Probe.ShouldEmit} sync={well.Probe.SyncConfidence:0.000} freq={well.Probe.FrequencyConfidence:0.000} {well.Probe.Reason}", "caption"), - }; - children.AddRange(well.SourceHealth - .OrderBy(static source => source.Kind, StringComparer.Ordinal) - .ThenBy(static source => source.SourceId, StringComparer.Ordinal) - .Take(10) - .Select(source => UiElement.Text( - $"mimir-well-source-health-{StableId(source.SourceId)}", - $"{source.SourceId}: {source.Kind} {source.Status} samples {source.Samples} age {source.LatestAgeMs:0}ms {source.Reason}", - source.Status == "live" ? "caption" : "danger"))); - children.AddRange(well.FeatureSignals - .OrderByDescending(static signal => signal.Confidence) - .ThenBy(static signal => signal.SourceId, StringComparer.Ordinal) - .Take(5) - .Select(signal => UiElement.Text( - $"mimir-well-feature-{StableId(signal.SourceId)}", - $"{signal.SourceId}: tracks {signal.StableTrackCount} c={signal.Confidence:0.000} motion {signal.MotionEnergy:0.000} centroid {signal.CentroidX:0.00},{signal.CentroidY:0.00}", - "caption"))); - children.AddRange(well.ClockDomains - .Take(6) - .Select(domain => UiElement.Text( - $"mimir-well-clock-{StableId(domain.ClockDomainId)}", - $"{domain.ClockDomainId}: {domain.SourceCount} src overlap {domain.OverlapMs:0.000}ms offset {domain.OffsetMs:0.000}ms {(domain.IsReference ? "ref" : "")}", - "caption"))); - children.AddRange(well.CanonicalClockMaps - .OrderByDescending(static map => Math.Abs(map.OffsetMs)) - .ThenBy(static map => map.StreamKey, StringComparer.Ordinal) - .Take(4) - .Select(map => UiElement.Text( - $"mimir-well-clock-map-{StableId(map.StreamKey)}", - $"{map.StreamKey}: ingress offset {map.OffsetMs:0.000}ms samples {map.SampleCount}", - "caption"))); - children.AddRange(well.VisualCalibration - .Take(6) - .Select(camera => UiElement.Text( - $"mimir-well-cal-{StableId(camera.SourceId)}", - $"{camera.SourceId}: score {camera.BestScore:0.000} leds {camera.BestDetectedLedCount} exp {camera.BestExposure:0} gain {camera.BestGain:0} {camera.State}", - "caption"))); - return UiElement.Pane("mimir-well-pane", "Well", children); - } - - private static DashboardUiElement BuildCalibrationPane(MimirLiveStatsSnapshot snapshot) - { - var leap = snapshot.Well?.Buffers.FirstOrDefault(static buffer => buffer.SourceId.Contains("leap", StringComparison.OrdinalIgnoreCase)); - var eyeStreams = snapshot.ObservationStreams - .Where(static stream => stream.StreamId.Contains("eye", StringComparison.OrdinalIgnoreCase) || stream.StreamId.Contains("video2", StringComparison.OrdinalIgnoreCase) || stream.StreamId.Contains("video3", StringComparison.OrdinalIgnoreCase)) - .OrderBy(static stream => stream.StreamId, StringComparer.Ordinal) - .Take(4) - .ToArray(); - var moveStreams = snapshot.ObservationStreams - .Where(static stream => stream.Kind.Contains("move", StringComparison.OrdinalIgnoreCase)) - .OrderBy(static stream => stream.StreamId, StringComparer.Ordinal) - .Take(4) - .ToArray(); - - var children = new List - { - UiElement.Text("mimir-calibration-leap", leap == null ? "Leap: no Well buffer" : $"Leap: {leap.Count} frames {leap.PixelFormat} {leap.Width:0}x{leap.Height:0} edge {NsAge(leap.EdgeNs)}", "mono"), - UiElement.Text("mimir-calibration-eyes", $"Eyes: {eyeStreams.Count(static stream => stream.State == "active")}/{eyeStreams.Length} active", "caption"), - UiElement.Text("mimir-calibration-moves", $"Moves: {moveStreams.Count(static stream => stream.State == "active")}/{moveStreams.Length} active", "caption"), - }; - children.AddRange(eyeStreams.Select(stream => UiElement.Text($"mimir-eye-{StableId(stream.StreamId)}", $"{stream.StreamId} {stream.Shape} age {stream.AgeSeconds:0}s", "caption"))); - children.AddRange(moveStreams.Select(stream => UiElement.Text($"mimir-move-{StableId(stream.StreamId)}", $"{stream.StreamId} {stream.Shape} age {stream.AgeSeconds:0}s", "caption"))); - return UiElement.Pane("mimir-calibration-pane", "Leap / Eyes / Moves", children); - } - - private static DashboardUiElement StatBar(string id, string label, double value, string tone, string? displayValue = null) => - UiElement.Card( - id, - "stat-bar", - style: new DashboardUiStyle { Variant = "compact", Tone = tone }, - children: - [ - UiElement.Text($"{id}-label", $"{label}: {displayValue ?? value.ToString("0.000", CultureInfo.InvariantCulture)}", "caption"), - UiElement.Metric($"{id}-metric", label, value, tone), - ]); - - private static string Bar(double value, int width) - { - var clamped = Math.Clamp(value, 0.0, 1.0); - var filled = (int)Math.Round(clamped * width); - var pulseIndex = Math.Clamp((int)(DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() / 180 % Math.Max(1, width)), 0, Math.Max(0, width - 1)); - var builder = new StringBuilder(width); - for (var index = 0; index < width; index++) - { - builder.Append(index < filled ? (index == pulseIndex ? '*' : '#') : '.'); - } - - return builder.ToString(); - } - - private static string Pulse() => - (DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() / 250 % 4) switch - { - 0 => "|", - 1 => "/", - 2 => "-", - _ => "\\", - }; - - private static double NormalizeRms(double rms) => Math.Clamp(rms / 0.08, 0.0, 1.0); - - private static string ToneFor(double value) => - value >= 0.70 ? "cool" : value >= 0.25 ? "warm" : "danger"; - - private static string NsAge(double timestampNs) - { - if (timestampNs <= 0) - { - return "unknown"; - } - - var nowNs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000.0; - var seconds = Math.Max(0.0, (nowNs - timestampNs) / 1_000_000_000.0); - return seconds < 1.0 ? $"{seconds * 1000.0:0}ms" : $"{seconds:0.0}s"; - } - - private static string StableId(string value) => - string.Join("-", value.ToLowerInvariant().Split(Path.GetInvalidFileNameChars(), StringSplitOptions.RemoveEmptyEntries)) - .Replace(":", "-", StringComparison.Ordinal) - .Replace(">", "-", StringComparison.Ordinal) - .Replace(" ", "-", StringComparison.Ordinal); -} - -internal sealed class MimirLiveStatsSnapshot -{ - public string TelemetrySource { get; init; } = "none"; - public string ObservationSource { get; init; } = "none"; - public MimirTelemetryStat? Telemetry { get; init; } - public IReadOnlyList Spectra { get; init; } = []; - public IReadOnlyList SyncStates { get; init; } = []; - public IReadOnlyList SyncReports { get; init; } = []; - public IReadOnlyList SyncDecodeAttempts { get; init; } = []; - public IReadOnlyList VideoBuffers { get; init; } = []; - public IReadOnlyList ActuatorCommands { get; init; } = []; - public IReadOnlyList ObservationStreams { get; init; } = []; - public MimirWellStat? Well { get; init; } - public MimirMoveMusicStat? MoveMusic { get; init; } - public bool HasAnyData => Telemetry != null || Well != null || MoveMusic != null || Spectra.Count > 0 || SyncStates.Count > 0 || VideoBuffers.Count > 0 || ObservationStreams.Count > 0; - public string Summary => Well == null - ? $"{Spectra.Count} audio lanes / {SyncStates.Count} sync states / {SyncDecodeAttempts.Count} decode attempts / {VideoBuffers.Count} video buffers / {ObservationStreams.Count} device streams" - : $"Well {Well.LiveSources}/{Well.ConfiguredSources} sources / {Well.AudioBuffers} audio / {Well.VideoBuffers} video / {ObservationStreams.Count} device streams"; - - public static MimirLiveStatsSnapshot Load(string telemetryLogPath, string observationLogPath) - { - var telemetrySource = ResolveTelemetryLog(telemetryLogPath); - var telemetryLines = File.Exists(telemetrySource) ? TailTextFile(telemetrySource, 512 * 1024).Split("\n", StringSplitOptions.RemoveEmptyEntries) : []; - var observationLines = File.Exists(observationLogPath) ? TailTextFile(observationLogPath, 512 * 1024).Split("\n", StringSplitOptions.RemoveEmptyEntries) : []; - return new MimirLiveStatsSnapshot - { - TelemetrySource = string.IsNullOrWhiteSpace(telemetrySource) ? "none" : telemetrySource, - ObservationSource = File.Exists(observationLogPath) ? observationLogPath : $"missing {observationLogPath}", - Telemetry = telemetryLines.Select(ParseTelemetry).Where(static value => value != null).LastOrDefault(), - Spectra = LatestBySource(telemetryLines.Select(ParseSpectrum).Where(static value => value != null)!), - SyncStates = LatestBySource(telemetryLines.Select(ParseSyncState).Where(static value => value != null)!), - SyncReports = LatestBySource(telemetryLines.Select(ParseSyncReport).Where(static value => value != null)!), - SyncDecodeAttempts = LatestSyncDecodeAttempts(telemetryLines.Select(ParseSyncDecodeAttempt).Where(static value => value != null)!), - VideoBuffers = LatestBySource(telemetryLines.Select(ParseVideoBuffer).Where(static value => value != null)!), - ActuatorCommands = LatestBySource(telemetryLines.Select(ParseActuatorCommand).Where(static value => value != null)!), - ObservationStreams = LatestObservationStreams(observationLines), - Well = LatestWellSnapshot(telemetryLines.Concat(observationLines)), - MoveMusic = LatestMoveMusicTrace(telemetrySource, telemetryLines), - }; - } - - private static string ResolveTelemetryLog(string configured) - { - if (!string.IsNullOrWhiteSpace(configured)) - { - return configured; - } - - var runtimeDir = Path.Combine(AppContext.BaseDirectory, "..", "..", "..", "..", "..", "artifacts", "runtime"); - runtimeDir = Path.GetFullPath(runtimeDir); - if (!Directory.Exists(runtimeDir)) - { - runtimeDir = @"E:\Projects\Mimir\artifacts\runtime"; - } - - if (!Directory.Exists(runtimeDir)) - { - return ""; - } - - foreach (var file in Directory.EnumerateFiles(runtimeDir, "*.out.log") - .OrderByDescending(File.GetLastWriteTimeUtc) - .Take(32)) - { - var text = TailTextFile(file, 128 * 1024); - if (text.Contains("mimir-sync-telemetry", StringComparison.Ordinal) || - text.Contains("mimir-spectrum ", StringComparison.Ordinal) || - text.Contains("mimir-video-buffer ", StringComparison.Ordinal)) - { - return file; - } - } - - return ""; - } - - private static IReadOnlyList LatestBySource(IEnumerable values) - where T : class, IMimirSourceStat - { - var bySource = new Dictionary(StringComparer.Ordinal); - foreach (var value in values) - { - if (value == null) - { - continue; - } - - bySource[value.SourceId] = value; - } - - return bySource.Values.OrderBy(static value => value.SourceId, StringComparer.Ordinal).ToArray(); - } - - private static IReadOnlyList LatestSyncDecodeAttempts(IEnumerable values) - { - var byRoute = new Dictionary(StringComparer.Ordinal); - foreach (var value in values) - { - if (value == null) - { - continue; - } - - byRoute[$"{value.ReferenceSourceId}->{value.SourceId}"] = value; - } - - return byRoute.Values - .OrderByDescending(static value => value.Confidence) - .ThenBy(static value => value.SourceId, StringComparer.Ordinal) - .ToArray(); - } - - private static MimirWellStat? LatestWellSnapshot(IEnumerable lines) - { - MimirWellStat? latest = null; - foreach (var line in lines) - { - try - { - using var document = JsonDocument.Parse(line); - var root = document.RootElement; - if (string.Equals(JsonStringAny(root, "document"), "mimir.well_snapshot.v1", StringComparison.Ordinal)) - { - latest = ParseWellSnapshot(root); - } - } - catch (JsonException) - { - } - } - - return latest; - } - - private static MimirWellStat ParseWellSnapshot(JsonElement root) - { - var buffers = ArrayItems(TryGetAny(root, "buffers")) - .Select(static buffer => - { - var latest = TryGetAny(buffer, "latest"); - var video = TryGetAny(latest, "video"); - var audio = TryGetAny(latest, "audio"); - return new MimirWellBufferStat( - JsonStringAny(buffer, "SourceId", "sourceId") ?? "unknown", - JsonStringAny(buffer, "Kind", "kind") ?? "unknown", - (int)JsonNumberAny(buffer, "Count", "count"), - JsonNumberAny(buffer, "EdgeNs", "edgeNs"), - (int)JsonNumberAny(video, "Width", "width"), - (int)JsonNumberAny(video, "Height", "height"), - JsonStringAny(video, "PixelFormat", "pixelFormat") ?? "", - (int)JsonNumberAny(audio, "SampleRate", "sampleRate"), - (int)JsonNumberAny(audio, "Channels", "channels")); - }) - .ToArray(); - var frame = TryGetAny(root, "synchronizedFrame"); - var slices = ArrayItems(TryGetAny(frame, "slices")).ToArray(); - var audioSync = TryGetAny(root, "audioSync"); - var states = ArrayItems(TryGetAny(audioSync, "states")) - .Select(static state => new MimirWellAudioSyncStateStat( - JsonStringAny(state, "SourceId", "sourceId") ?? "unknown", - JsonStringAny(state, "ReferenceSourceId", "referenceSourceId") ?? "unknown", - JsonNumberAny(state, "SmoothedDelaySamples", "smoothedDelaySamples"), - JsonNumberAny(state, "SamplingRateOffsetPpm", "samplingRateOffsetPpm"), - JsonNumberAny(state, "Confidence", "confidence"))) - .ToArray(); - var probeElement = TryGetAny(audioSync, "probe"); - var probe = probeElement.ValueKind == JsonValueKind.Object - ? new MimirWellProbeStat( - JsonBoolAny(probeElement, "ShouldEmit", "shouldEmit"), - JsonNumberAny(probeElement, "AggregateSyncConfidence", "aggregateSyncConfidence"), - JsonNumberAny(probeElement, "AggregateFrequencyResponseConfidence", "aggregateFrequencyResponseConfidence"), - JsonStringAny(probeElement, "reason") ?? "unknown") - : null; - var streamPressure = TryGetAny(root, "streamPressure"); - var pollPressure = TryGetAny(streamPressure, "poll"); - var publishPressure = TryGetAny(streamPressure, "publish"); - var latency = TryGetAny(root, "latency"); - var featureSignals = TryGetAny(root, "featureSignals"); - var featureSignalStats = ArrayItems(TryGetAny(featureSignals, "signals")) - .Select(static signal => new MimirWellFeatureSignalStat( - JsonStringAny(signal, "SourceId", "sourceId") ?? "unknown", - (int)JsonNumberAny(signal, "StableTrackCount", "stableTrackCount"), - JsonNumberAny(signal, "Confidence", "confidence"), - JsonNumberAny(signal, "MeanMotionPixelsPerSecond", "meanMotionPixelsPerSecond"), - JsonNumberAny(signal, "MotionEnergy", "motionEnergy"), - JsonNumberAny(signal, "NormalizedCentroidX", "normalizedCentroidX"), - JsonNumberAny(signal, "NormalizedCentroidY", "normalizedCentroidY"))) - .ToArray(); - var canonicalClockMaps = ArrayItems(TryGetAny(root, "canonicalClockMaps")) - .Select(static map => new MimirWellCanonicalClockMapStat( - JsonStringAny(map, "StreamKey", "streamKey") ?? "unknown", - JsonNumberAny(map, "offsetMs"), - (long)JsonNumberAny(map, "SampleCount", "sampleCount"))) - .ToArray(); - var clockDomains = ArrayItems(TryGetAny(TryGetAny(root, "clockDomains"), "domains")) - .Select(static domain => new MimirWellClockDomainStat( - JsonStringAny(domain, "ClockDomainId", "clockDomainId") ?? "unknown", - (int)JsonNumberAny(domain, "SourceCount", "sourceCount"), - JsonNumberAny(domain, "OverlapNs", "overlapNs") / 1_000_000.0, - JsonBoolAny(domain, "HasLocalOverlap", "hasLocalOverlap"), - JsonBoolAny(domain, "IsReferenceDomain", "isReferenceDomain"), - JsonNumberAny(domain, "ProvisionalOffsetToReferenceMs", "provisionalOffsetToReferenceMs"))) - .ToArray(); - var visualCalibration = ArrayItems(TryGetAny(TryGetAny(root, "visualCalibration"), "cameras")) - .Select(static camera => new MimirWellVisualCalibrationStat( - JsonStringAny(camera, "SourceId", "sourceId") ?? "unknown", - JsonStringAny(camera, "State", "state") ?? "unknown", - JsonNumberAny(camera, "BestScore", "bestScore"), - (int)JsonNumberAny(camera, "BestDetectedLedCount", "bestDetectedLedCount"), - JsonBoolAny(camera, "BestUsableForCalibration", "bestUsableForCalibration"), - JsonNumberAny(camera, "BestExposure", "bestExposure"), - JsonNumberAny(camera, "BestGain", "bestGain"))) - .ToArray(); - var captureStorage = TryGetAny(TryGetAny(root, "capture"), "storage"); - var publication = TryGetAny(root, "publication"); - var stems = ArrayItems(TryGetAny(publication, "stems")).ToArray(); - var sourceHealth = ArrayItems(TryGetAny(root, "sourceHealth")) - .Select(static source => new MimirWellSourceHealthStat( - JsonStringAny(source, "SourceId", "sourceId") ?? "unknown", - JsonStringAny(source, "Kind", "kind") ?? "unknown", - JsonStringAny(source, "Origin", "origin") ?? "unknown", - JsonStringAny(source, "Label", "label") ?? "", - JsonStringAny(source, "status") ?? "unknown", - JsonStringAny(source, "reason") ?? "", - (int)JsonNumberAny(source, "buffers"), - (int)JsonNumberAny(source, "samples"), - JsonNumberAny(source, "latestAgeMs"), - (int)JsonNumberAny(source, "latestByteLength"))) - .ToArray(); - return new MimirWellStat( - (long)JsonNumberAny(root, "sequence"), - JsonNumberAny(root, "elapsedSeconds"), - JsonNumberAny(root, "ingestedSamples"), - (int)JsonNumberAny(root, "configuredSources"), - (int)JsonNumberAny(root, "liveSources"), - ArrayItems(TryGetAny(root, "sourceErrors")).Count(), - JsonBoolAny(frame, "IsComplete", "isComplete"), - JsonNumberAny(frame, "PresentationDelayMs", "presentationDelayMs"), - JsonStringAny(frame, "degradedKind") ?? "", - JsonStringAny(frame, "degradedReason") ?? "", - slices.Count(static slice => string.Equals(JsonStringAny(slice, "Status", "status"), "Ready", StringComparison.OrdinalIgnoreCase)), - slices.Length, - JsonNumberAny(latency, "currentDelayMs"), - JsonNumberAny(latency, "ceilingDelayMs"), - JsonNumberAny(latency, "floorDelayMs"), - JsonNumberAny(latency, "retainedOverlapMs"), - JsonNumberAny(latency, "edgeSkewMs"), - JsonStringAny(latency, "Reason", "reason") ?? "", - JsonNumberAny(pollPressure, "averageMilliseconds"), - JsonNumberAny(pollPressure, "maxMilliseconds"), - (long)JsonNumberAny(pollPressure, "zeroPollIterations"), - JsonNumberAny(publishPressure, "averageMilliseconds"), - JsonNumberAny(publishPressure, "maxMilliseconds"), - (long)JsonNumberAny(publishPressure, "bytes"), - JsonStringAny(captureStorage, "bodyTransport") ?? "unknown", - stems.Length, - sourceHealth, - buffers, - clockDomains, - states, - visualCalibration, - JsonNumberAny(featureSignals, "MeanConfidence", "meanConfidence"), - JsonNumberAny(featureSignals, "MeanMotionPixelsPerSecond", "meanMotionPixelsPerSecond"), - (int)JsonNumberAny(featureSignals, "StableTrackCount", "stableTrackCount"), - featureSignalStats, - canonicalClockMaps, - probe); - } - - private static MimirMoveMusicStat? LatestMoveMusicTrace(string telemetrySource, IReadOnlyList telemetryLines) - { - var configuredLine = telemetryLines - .Reverse() - .FirstOrDefault(static line => line.Contains("\"live_score\"", StringComparison.OrdinalIgnoreCase) || - line.Contains("\"score_gesture_envelopes\"", StringComparison.OrdinalIgnoreCase)); - if (!string.IsNullOrWhiteSpace(configuredLine)) - { - return ParseMoveMusicTrace(telemetrySource, configuredLine); - } - - return LatestMoveMusicTraceFromRuntime(); - } - - private static MimirMoveMusicStat? LatestMoveMusicTraceFromRuntime() - { - var runtimeRoot = Path.Combine(AppContext.BaseDirectory, "artifacts", "runtime"); - if (!Directory.Exists(runtimeRoot)) - { - runtimeRoot = @"E:\Projects\Mimir\artifacts\runtime"; - } - - if (!Directory.Exists(runtimeRoot)) - { - return null; - } - - var trace = Directory.EnumerateFiles(runtimeRoot, "online-sync.jsonl", SearchOption.AllDirectories) - .Select(static path => new FileInfo(path)) - .Where(static info => info.DirectoryName?.Contains("move", StringComparison.OrdinalIgnoreCase) == true) - .OrderByDescending(static info => info.LastWriteTimeUtc) - .FirstOrDefault(); - if (trace == null || trace.Length == 0) - { - return null; - } - - foreach (var line in TailTextFile(trace.FullName, 4 * 1024 * 1024) - .Split("\n", StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) - .Reverse() - .Where(static candidate => candidate.Contains("\"live_score\"", StringComparison.OrdinalIgnoreCase) || - candidate.Contains("\"score_gesture_envelopes\"", StringComparison.OrdinalIgnoreCase))) - { - var parsed = ParseMoveMusicTrace(trace.FullName, line); - if (parsed != null) - { - return parsed; - } - } - - return null; - } - - private static MimirMoveMusicStat? ParseMoveMusicTrace(string sourcePath, string line) - { - try - { - using var document = JsonDocument.Parse(line); - var root = document.RootElement; - var maxRgb = 0; - var moves = TryGetAny(root, "moves"); - if (moves.ValueKind == JsonValueKind.Object) - { - foreach (var move in moves.EnumerateObject()) - { - if (move.Value.ValueKind == JsonValueKind.Array) - { - maxRgb = Math.Max(maxRgb, move.Value.EnumerateArray().Select(JsonNumber).Select(static value => (int)value).DefaultIfEmpty(0).Max()); - } - } - } - - var envelopes = ArrayItems(TryGetAny(root, "score_gesture_envelopes")).Select(JsonNumber).DefaultIfEmpty(0.0).ToArray(); - var liveScore = TryGetAny(root, "live_score"); - var voices = ArrayItems(TryGetAny(liveScore, "voices")) - .Select(static voice => new MimirMoveMusicVoiceStat( - JsonStringAny(voice, "source") ?? "unknown", - JsonStringAny(voice, "role") ?? "voice", - JsonStringAny(voice, "note_name") ?? "?", - JsonNumberAny(voice, "frequency_hz"), - JsonNumberAny(voice, "midi"), - JsonNumberAny(voice, "confidence"), - JsonNumberAny(voice, "strength"), - JsonBoolAny(voice, "active"))) - .OrderByDescending(static voice => voice.Active) - .ThenByDescending(static voice => voice.Confidence) - .ToArray(); - var targets = ArrayItems(TryGetAny(liveScore, "move_targets")) - .Select(static target => new MimirMoveMusicTargetStat( - (int)JsonNumberAny(target, "move_index"), - JsonStringAny(target, "source") ?? "unknown", - JsonStringAny(target, "note_name") ?? "?", - JsonNumberAny(target, "target_note"), - JsonNumberAny(target, "confidence"), - JsonNumberAny(target, "calibration_priority"))) - .OrderBy(static target => target.MoveIndex) - .ToArray(); - var sources = ArrayItems(TryGetAny(root, "music_sources")) - .Select(static source => - { - var name = JsonStringAny(source, "source") ?? "source"; - var strength = JsonNumberAny(source, "score_strength", "onset", "hit", "loudness_gate"); - var key = JsonStringAny(source, "key_name") ?? ""; - var mode = JsonStringAny(source, "key_mode") ?? ""; - var harmonic = string.IsNullOrWhiteSpace(key) - ? $"f0 {JsonNumberAny(source, "fundamental_hz"):0}" - : $"{key} {mode}".Trim(); - return $"{name} {JsonNumberAny(source, "bpm"):0.0}/{JsonNumberAny(source, "bpm_confidence"):0.00} s={strength:0.00} {harmonic}"; - }) - .ToArray(); - return new MimirMoveMusicStat( - sourcePath, - JsonNumberAny(root, "bpm"), - JsonNumberAny(root, "bpm_confidence"), - JsonNumberAny(root, "loudness_gate"), - JsonNumberAny(root, "loudness_percentile"), - JsonStringAny(root, "key_name") ?? "?", - JsonStringAny(root, "key_mode") ?? "?", - JsonStringAny(root, "chord_name") ?? "?", - envelopes.Max(), - maxRgb, - sources, - voices, - targets, - BuildPianoRoll(voices, targets)); - } - catch (JsonException) - { - return null; - } - } - - private static string BuildPianoRoll( - IReadOnlyList voices, - IReadOnlyList targets) - { - var marks = new char[36]; - Array.Fill(marks, '.'); - foreach (var voice in voices.Where(static voice => voice.Active)) - { - var index = PianoIndex(voice.MidiNote); - if (index >= 0) - { - marks[index] = voice.Confidence >= 0.65 ? '#' : '+'; - } - } - - foreach (var target in targets) - { - var index = PianoIndex(target.TargetNote); - if (index >= 0 && marks[index] == '.') - { - marks[index] = 'o'; - } - } - - return $"C3 {new string(marks)} C6"; - } - - private static int PianoIndex(double midiNote) - { - if (midiNote <= 0.0) - { - return -1; - } - - return (int)Math.Clamp(Math.Round(midiNote) - 48, 0, 35); - } - - private static IReadOnlyList LatestObservationStreams(IEnumerable lines) - { - var byStream = new Dictionary(StringComparer.Ordinal); - var now = DateTimeOffset.UtcNow; - foreach (var line in lines) - { - try - { - using var document = JsonDocument.Parse(line); - var root = document.RootElement; - var type = JsonString(root, "type"); - if (type is not ("cultmesh-observation" or "cultmesh-media-observation")) - { - continue; - } - - var deviceId = JsonString(root, "DeviceId") ?? "unknown"; - var streamId = JsonString(root, "StreamId") ?? "unknown"; - var kind = JsonString(root, "Kind") ?? "unknown"; - var latestAt = DateTimeOffset.TryParse(JsonString(root, "WallClockUtc"), out var parsed) ? parsed : DateTimeOffset.MinValue; - var age = latestAt == DateTimeOffset.MinValue ? double.PositiveInfinity : Math.Max(0.0, (now - latestAt).TotalSeconds); - var shape = ObservationShape(root); - byStream[$"{deviceId}:{streamId}:{kind}"] = new MimirObservationStreamStat( - deviceId, - streamId, - kind, - JsonNumber(root, "Sequence"), - age <= 120.0 ? "active" : "stale", - age, - shape); - } - catch (JsonException) - { - } - } - - return byStream.Values.ToArray(); - } - - private static string ObservationShape(JsonElement root) - { - var format = JsonString(root, "Format") ?? ""; - var width = JsonNumber(root, "Width"); - var height = JsonNumber(root, "Height"); - var sampleRate = JsonNumber(root, "SampleRate"); - var channels = JsonNumber(root, "Channels"); - var frames = JsonNumber(root, "FrameCount"); - var bytes = JsonNumber(root, "PayloadBytes"); - if (width > 0 && height > 0) - { - return $"{format} {width:0}x{height:0} {bytes:0} bytes"; - } - - if (sampleRate > 0) - { - return $"{format} {sampleRate:0} Hz x {Math.Max(1, channels):0} {frames:0} frames {bytes:0} bytes"; - } - - if (root.TryGetProperty("Values", out var values) && values.ValueKind == JsonValueKind.Array) - { - return string.Join(", ", values.EnumerateArray().Take(3).Select(static value => value.TryGetDouble(out var number) ? number.ToString("0.000", CultureInfo.InvariantCulture) : "?")); - } - - return "observation"; - } - - private static MimirTelemetryStat? ParseTelemetry(string line) - { - if (!line.StartsWith("mimir-sync-telemetry ", StringComparison.Ordinal)) - { - return null; - } - - var fields = Fields(line); - return new MimirTelemetryStat( - Number(fields, "t"), - (int)Number(fields, "sources"), - (int)Number(fields, "lastPoll"), - Number(fields, "ingested"), - Text(fields, "audioSync"), - (int)Number(fields, "reports"), - (int)Number(fields, "states"), - Number(fields, "analyzeMs")); - } - - private static MimirSpectrumStat? ParseSpectrum(string line) - { - if (!line.StartsWith("mimir-spectrum ", StringComparison.Ordinal)) - { - return null; - } - - var sourceId = TokenAfterPrefix(line, "mimir-spectrum "); - var fields = Fields(line); - return new MimirSpectrumStat( - sourceId, - Text(fields, "label"), - (int)Number(fields, "rate"), - (int)Number(fields, "fft"), - Number(fields, "rms"), - Number(fields, "peak"), - Number(fields, "floorDb"), - Text(fields, "peaks")); - } - - private static MimirSyncStateStat? ParseSyncState(string line) - { - if (!line.StartsWith("mimir-sync-state ", StringComparison.Ordinal)) - { - return null; - } - - var route = TokenAfterPrefix(line, "mimir-sync-state "); - var parts = route.Split("->", 2, StringSplitOptions.TrimEntries); - if (parts.Length != 2) - { - return null; - } - - var fields = Fields(line); - return new MimirSyncStateStat( - parts[1], - parts[0], - Number(fields, "delaySamples"), - Number(fields, "delayUs"), - Number(fields, "delayMs"), - Number(fields, "sroPpm"), - Number(fields, "confidence")); - } - - private static MimirSyncReportStat? ParseSyncReport(string line) - { - if (!line.StartsWith("mimir-sync-report ", StringComparison.Ordinal) && - !line.StartsWith("mimir-complex-contour-report ", StringComparison.Ordinal)) - { - return null; - } - - var prefix = line.StartsWith("mimir-sync-report ", StringComparison.Ordinal) ? "mimir-sync-report " : "mimir-complex-contour-report "; - var route = TokenAfterPrefix(line, prefix); - var parts = route.Split("->", 2, StringSplitOptions.TrimEntries); - if (parts.Length != 2) - { - return null; - } - - var fields = Fields(line); - return new MimirSyncReportStat( - parts[1], - parts[0], - Text(fields, "evidence", prefix.Contains("complex", StringComparison.Ordinal) ? "complex-contour" : "sync"), - Number(fields, "delayUs"), - Number(fields, "confidence"), - (int)Number(fields, "timelineEvents") + (int)Number(fields, "directHits")); - } - - private static MimirSyncDecodeAttemptStat? ParseSyncDecodeAttempt(string line) - { - if (!line.StartsWith("mimir-sync-decode ", StringComparison.Ordinal)) - { - return null; - } - - var route = TokenAfterPrefix(line, "mimir-sync-decode "); - var parts = route.Split("->", 2, StringSplitOptions.TrimEntries); - if (parts.Length != 2) - { - return null; - } - - var fields = Fields(line); - return new MimirSyncDecodeAttemptStat( - parts[1], - parts[0], - Text(fields, "status", "unknown"), - (int)Number(fields, "compared"), - (int)Number(fields, "rate"), - Number(fields, "refEnergy"), - Number(fields, "candEnergy"), - (int)Number(fields, "matched"), - Number(fields, "confidence")); - } - - private static MimirVideoBufferStat? ParseVideoBuffer(string line) - { - if (!line.StartsWith("mimir-video-buffer ", StringComparison.Ordinal)) - { - return null; - } - - var sourceId = TokenAfterPrefix(line, "mimir-video-buffer "); - var fields = Fields(line); - var latestIndex = line.IndexOf(" latest=", StringComparison.Ordinal); - return new MimirVideoBufferStat( - sourceId, - (int)Number(fields, "count"), - latestIndex < 0 ? "" : line[(latestIndex + " latest=".Length)..]); - } - - private static MimirActuatorCommandStat? ParseActuatorCommand(string line) - { - if (!line.StartsWith("mimir-audio-actuator-command ", StringComparison.Ordinal)) - { - return null; - } - - var fields = Fields(line); - var source = Text(fields, "source"); - return string.IsNullOrWhiteSpace(source) - ? null - : new MimirActuatorCommandStat(source, Number(fields, "delaySamples"), Number(fields, "ratio"), Number(fields, "confidence")); - } - - private static Dictionary Fields(string line) - { - var fields = new Dictionary(StringComparer.Ordinal); - foreach (var token in Tokenize(line)) - { - var equals = token.IndexOf('='); - if (equals <= 0) - { - continue; - } - - fields[token[..equals]] = token[(equals + 1)..].Trim('"'); - } - - return fields; - } - - private static IEnumerable Tokenize(string line) - { - var builder = new StringBuilder(); - var quoted = false; - foreach (var character in line) - { - if (character == '"') - { - quoted = !quoted; - builder.Append(character); - continue; - } - - if (char.IsWhiteSpace(character) && !quoted) - { - if (builder.Length > 0) - { - yield return builder.ToString(); - builder.Clear(); - } - - continue; - } - - builder.Append(character); - } - - if (builder.Length > 0) - { - yield return builder.ToString(); - } - } - - private static string TokenAfterPrefix(string line, string prefix) - { - var rest = line[prefix.Length..]; - var space = rest.IndexOf(' '); - return space < 0 ? rest : rest[..space]; - } - - private static string Text(IReadOnlyDictionary fields, string key, string fallback = "") => - fields.TryGetValue(key, out var value) ? value : fallback; - - private static double Number(IReadOnlyDictionary fields, string key) => - fields.TryGetValue(key, out var value) && double.TryParse(value, NumberStyles.Float, CultureInfo.InvariantCulture, out var number) - ? number - : 0.0; - - private static string? JsonString(JsonElement root, string property) => - root.TryGetProperty(property, out var child) && child.ValueKind == JsonValueKind.String ? child.GetString() : null; - - private static double JsonNumber(JsonElement root, string property) => - root.TryGetProperty(property, out var child) && child.ValueKind == JsonValueKind.Number && child.TryGetDouble(out var value) ? value : 0.0; - - private static JsonElement TryGetAny(JsonElement root, params string[] properties) - { - if (root.ValueKind != JsonValueKind.Object) - { - return default; - } - - foreach (var property in properties) - { - if (root.TryGetProperty(property, out var child)) - { - return child; - } - } - - return default; - } - - private static IEnumerable ArrayItems(JsonElement element) - { - if (element.ValueKind != JsonValueKind.Array) - { - yield break; - } - - foreach (var item in element.EnumerateArray()) - { - yield return item; - } - } - - private static string? JsonStringAny(JsonElement root, params string[] properties) - { - var child = TryGetAny(root, properties); - return child.ValueKind == JsonValueKind.String ? child.GetString() : null; - } - - private static double JsonNumberAny(JsonElement root, params string[] properties) => JsonNumber(TryGetAny(root, properties)); - - private static double JsonNumber(JsonElement element) => - element.ValueKind == JsonValueKind.Number && element.TryGetDouble(out var value) ? value : 0.0; - - private static bool JsonBoolAny(JsonElement root, params string[] properties) - { - var child = TryGetAny(root, properties); - return child.ValueKind == JsonValueKind.True || (child.ValueKind == JsonValueKind.Number && child.TryGetDouble(out var value) && value != 0.0); - } - - private static string NsAge(double timestampNs) - { - if (timestampNs <= 0) - { - return "unknown"; - } - - var nowNs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000.0; - var seconds = Math.Max(0.0, (nowNs - timestampNs) / 1_000_000_000.0); - return seconds < 1.0 ? $"{seconds * 1000.0:0}ms" : $"{seconds:0.0}s"; - } - - private static string TailTextFile(string filePath, int maxBytes) - { - var info = new FileInfo(filePath); - if (!info.Exists || info.Length == 0) - { - return ""; - } - - var length = (int)Math.Min(maxBytes, info.Length); - var buffer = new byte[length]; - using var stream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete); - stream.Seek(-length, SeekOrigin.End); - _ = stream.Read(buffer, 0, length); - return Encoding.UTF8.GetString(buffer); - } -} - -internal interface IMimirSourceStat -{ - string SourceId { get; } -} - -internal sealed record MimirTelemetryStat(double RuntimeSeconds, int Sources, int LastPoll, double Ingested, string AudioSync, int Reports, int States, double AnalyzeMs); - -internal sealed record MimirSpectrumStat(string SourceId, string Label, int SampleRate, int FftSize, double Rms, double Peak, double NoiseFloorDb, string Peaks) : IMimirSourceStat; - -internal sealed record MimirSyncStateStat(string SourceId, string ReferenceSourceId, double DelaySamples, double DelayUs, double DelayMs, double SroPpm, double Confidence) : IMimirSourceStat; - -internal sealed record MimirSyncReportStat(string SourceId, string ReferenceSourceId, string Evidence, double DelayUs, double Confidence, int Events) : IMimirSourceStat; - -internal sealed record MimirSyncDecodeAttemptStat(string SourceId, string ReferenceSourceId, string Status, int Compared, int SampleRate, double ReferenceEnergy, double CandidateEnergy, int Matched, double Confidence) : IMimirSourceStat; - -internal sealed record MimirVideoBufferStat(string SourceId, int Count, string Latest) : IMimirSourceStat; - -internal sealed record MimirActuatorCommandStat(string SourceId, double DelaySamples, double Ratio, double Confidence) : IMimirSourceStat; - -internal sealed record MimirObservationStreamStat(string DeviceId, string StreamId, string Kind, double Sequence, string State, double AgeSeconds, string Shape); - -internal sealed record MimirWellStat( - long Sequence, - double ElapsedSeconds, - double IngestedSamples, - int ConfiguredSources, - int LiveSources, - int SourceErrorCount, - bool SynchronizedFrameComplete, - double PresentationDelayMs, - string FrameDegradedKind, - string FrameDegradedReason, - int ReadySlices, - int TotalSlices, - double LatencyCurrentMs, - double LatencyCeilingMs, - double LatencyFloorMs, - double LatencyRetainedOverlapMs, - double LatencyEdgeSkewMs, - string LatencyReason, - double PollAverageMs, - double PollMaxMs, - long ZeroPollIterations, - double PublishAverageMs, - double PublishMaxMs, - long PublishedBytes, - string CaptureInlineBodies, - int PublicationStemCount, - IReadOnlyList SourceHealth, - IReadOnlyList Buffers, - IReadOnlyList ClockDomains, - IReadOnlyList AudioSyncStates, - IReadOnlyList VisualCalibration, - double FeatureMeanConfidence, - double FeatureMeanMotionPixelsPerSecond, - int FeatureStableTracks, - IReadOnlyList FeatureSignals, - IReadOnlyList CanonicalClockMaps, - MimirWellProbeStat? Probe) -{ - public int AudioBuffers => Buffers.Count(static buffer => string.Equals(buffer.Kind, "Audio", StringComparison.OrdinalIgnoreCase)); - - public int VideoBuffers => Buffers.Count(static buffer => string.Equals(buffer.Kind, "Video", StringComparison.OrdinalIgnoreCase)); - - public double MaxCanonicalClockOffsetMs => CanonicalClockMaps.Count == 0 - ? 0.0 - : CanonicalClockMaps.Max(static map => Math.Abs(map.OffsetMs)); -} - -internal sealed record MimirWellBufferStat( - string SourceId, - string Kind, - int Count, - double EdgeNs, - int Width, - int Height, - string PixelFormat, - int SampleRate, - int Channels); - -internal sealed record MimirWellSourceHealthStat( - string SourceId, - string Kind, - string Origin, - string Label, - string Status, - string Reason, - int Buffers, - int Samples, - double LatestAgeMs, - int LatestByteLength); - -internal sealed record MimirWellAudioSyncStateStat(string SourceId, string ReferenceSourceId, double DelaySamples, double SroPpm, double Confidence); - -internal sealed record MimirWellClockDomainStat(string ClockDomainId, int SourceCount, double OverlapMs, bool HasLocalOverlap, bool IsReference, double OffsetMs); - -internal sealed record MimirWellVisualCalibrationStat(string SourceId, string State, double BestScore, int BestDetectedLedCount, bool BestUsableForCalibration, double BestExposure, double BestGain); - -internal sealed record MimirWellFeatureSignalStat(string SourceId, int StableTrackCount, double Confidence, double MeanMotionPixelsPerSecond, double MotionEnergy, double CentroidX, double CentroidY); - -internal sealed record MimirWellCanonicalClockMapStat(string StreamKey, double OffsetMs, long SampleCount); - -internal sealed record MimirWellProbeStat(bool ShouldEmit, double SyncConfidence, double FrequencyConfidence, string Reason); - -internal sealed record MimirMoveMusicStat( - string TracePath, - double Bpm, - double BpmConfidence, - double LoudnessGate, - double LoudnessPercentile, - string KeyName, - string KeyMode, - string ChordName, - double ScoreEnvelopeMax, - int MaxRgb, - IReadOnlyList Sources, - IReadOnlyList Voices, - IReadOnlyList MoveTargets, - string PianoRoll); - -internal sealed record MimirMoveMusicVoiceStat(string SourceId, string Role, string NoteName, double FrequencyHz, double MidiNote, double Confidence, double Strength, bool Active); - -internal sealed record MimirMoveMusicTargetStat(int MoveIndex, string SourceId, string NoteName, double TargetNote, double Confidence, double CalibrationPriority); - -internal sealed class MimirStreamLayoutProvider : MutableDashboardProvider -{ - public const string ProviderId = "mimir.stream.layout"; - - public MimirStreamLayoutProvider() - : base( - new DashboardProviderManifest( - ProviderId, - "Mimir Stream Layout", - "Native scene graph editor for program layout, source transforms, visibility, and future LUT controls.", - "1", - "/eve/deck/mimir.stream.layout", - ["scene2d", "visibility", "transform", "lut-presets", "audio-mix"], - UsesCultMesh: true, - Transport: "local WebSocket; state document can mirror through CultMesh."), - CreateState()) - { - } - - private static DashboardState CreateState() => - new() - { - ProviderId = ProviderId, - Title = "Mimir Stream Layout", - Nodes = - [ - new DashboardNode("program", "Program Output", "camera", -0.05, -0.05, 0.62, 0.34, "live") { Detail = "Final OBS-facing composition surface. Visibility, solo, opacity, layer order, audio stem selection, mute/solo/gain, and LUT preset controls belong here." }, - new DashboardNode("editor", "Mimir Editor", "editor", -0.48, 0.34, 0.40, 0.28, "live") { Detail = "Scene graph owner for editor camera, sensor-feed panels, SDF text placeholders, model placeholders, selected node, visibility, locks, transform reset, and grab/rotate/resize gizmo intent." }, - new DashboardNode("sync", "Mimir Sync", "timing", 0.48, 0.34, 0.36, 0.28, "live") { Detail = "Rolling buffer and synchronization surface: five-second window, stream count, source count, poll cadence, ingested samples, audio sync state, and spectrum cadence." }, - new DashboardNode("leap-field", "Leap Field", "sensor", 0.38, 0.02, 0.28, 0.20, "dense geometry") { Detail = "Leap packed stereo IR feeds Fensalir stereo depth, disparity SurfacePage, point-cloud Mesh socket, surface claims, and GPU sensor fusion." }, - new DashboardNode("ps3-eye-0", "PS3 Eye 0", "camera", -0.56, -0.34, 0.24, 0.18, "187 fps tracker") { Detail = "High-rate Bayer8 tracking witness. GPU feature extraction samples this texture through Fensalir GpuSensorFusion." }, - new DashboardNode("ps3-eye-1", "PS3 Eye 1", "camera", -0.26, -0.44, 0.24, 0.18, "187 fps tracker") { Detail = "Second high-rate Bayer8 tracking witness. Used for fast motion and online calibration constraints." }, - new DashboardNode("kiyo-pro", "Kiyo Pro", "camera", 0.18, -0.42, 0.24, 0.18, "YUY2 RGB") { Detail = "RGB/context witness. Fensalir decodes YUY2 in compute through the documented R8G8B8A8_UNORM SRV path." }, - new DashboardNode("kiyo-basic", "Kiyo Basic", "camera", 0.50, -0.34, 0.24, 0.18, "YUY2 RGB") { Detail = "Second RGB/context witness. Enters the same GpuSensorFusion path after YUY2 decode." }, - new DashboardNode("eve-camera", "Eve Camera", "camera", 0.0, -0.62, 0.30, 0.18, "armed") { Detail = "Portable sensor uplink target. EveCanvas owns capture on-device and sends frame events into Mimir when online." }, - ], - }; -} - -internal sealed class VoidBotSwarmProvider : IDashboardProvider -{ - public const string ProviderId = "voidbot.swarm"; - - private readonly string swarmStatePath; - private DashboardState currentState; - private DateTime lastWriteUtc = DateTime.MinValue; - private string selectedIdentityId = ""; - private string selectedStatePath = ""; - - public VoidBotSwarmProvider(string swarmStatePath) - { - this.swarmStatePath = swarmStatePath; - Manifest = new DashboardProviderManifest( - ProviderId, - "VoidBot Swarm", - "Native Eve tab for VoidBot agent status, CTB order, and selected Face state.", - "1", - "/eve/deck/voidbot.swarm", - ["ctb", "agent-status", "state-tree", "cultmesh-snapshot"], - UsesCultMesh: true, - Transport: "Reads VoidBot swarm-state.json now; consumes voidbot.swarm_state_snapshot.v1 when CultMesh bridge is live."); - currentState = BuildMissingState(); - } - - public DashboardProviderManifest Manifest { get; } - - public DashboardState State - { - get - { - RefreshIfNeeded(force: false); - return currentState; - } - } - - public bool ApplyCommand(DashboardCommand command) - { - RefreshIfNeeded(force: false); - var node = currentState.Nodes.FirstOrDefault(candidate => string.Equals(candidate.Id, command.NodeId, StringComparison.Ordinal)); - if (node == null) - { - return false; - } - - currentState.SelectedNodeId = node.Id; - if (!string.IsNullOrWhiteSpace(node.IdentityId)) - { - selectedIdentityId = node.IdentityId; - selectedStatePath = ""; - } - - if (!string.IsNullOrWhiteSpace(node.StatePath)) - { - selectedStatePath = node.StatePath; - } - - RefreshIfNeeded(force: true); - currentState.SelectedNodeId = node.Id; - return true; - } - - private void RefreshIfNeeded(bool force) - { - try - { - var info = new FileInfo(swarmStatePath); - if (!info.Exists) - { - currentState = BuildMissingState(); - return; - } - - if (!force && info.LastWriteTimeUtc == lastWriteUtc) - { - return; - } - - using var document = JsonDocument.Parse(File.ReadAllText(info.FullName)); - currentState = BuildState(document.RootElement, info.LastWriteTimeUtc); - lastWriteUtc = info.LastWriteTimeUtc; - } - catch (Exception error) - { - currentState = BuildErrorState(error.Message); - } - } - - private DashboardState BuildState(JsonElement root, DateTime sourceWriteUtc) - { - var summary = TryGet(root, "summary"); - var controls = TryGet(root, "controls"); - var cultMesh = TryGet(root, "cultMesh"); - var orchestrator = TryGet(root, "orchestrator"); - var participants = ArrayItems(TryGet(root, "participants")).ToArray(); - var upcoming = ArrayItems(TryGet(root, "upcomingTurns")).ToArray(); - - if (string.IsNullOrWhiteSpace(selectedIdentityId)) - { - selectedIdentityId = StringValue(summary, "nextIdentityId") - ?? StringValue(upcoming.FirstOrDefault(), "identityId") - ?? StringValue(participants.FirstOrDefault(), "identityId") - ?? ""; - } - - var selectedAgent = participants.FirstOrDefault(participant => string.Equals(StringValue(participant, "identityId"), selectedIdentityId, StringComparison.Ordinal)); - if (selectedAgent.ValueKind == JsonValueKind.Undefined) - { - selectedAgent = participants.FirstOrDefault(); - selectedIdentityId = StringValue(selectedAgent, "identityId") ?? ""; - } - - var nodes = new List(); - var paused = BoolValue(summary, "paused") == true; - var stateLabel = StringValue(summary, "state") ?? "unknown"; - var generatedAt = StringValue(root, "generatedAt") ?? sourceWriteUtc.ToString("O"); - var cadence = NumberValue(controls, "cadenceMultiplier") ?? NumberValue(summary, "cadenceMultiplier") ?? 1; - nodes.Add(new DashboardNode( - "voidbot-summary", - $"VoidBot Swarm\n{stateLabel} next {StringValue(summary, "nextDisplayName") ?? "none"}", - "swarm", - -0.60, - -0.54, - 0.34, - 0.22, - paused ? "paused" : "running") - { - Detail = $"agents {NumberValue(summary, "participantCount") ?? 0:0} ready {NumberValue(summary, "readyNowCount") ?? 0:0} cadence x{cadence:0.##}\nmesh {StringValue(cultMesh, "writeStatus") ?? "missing"} {generatedAt}", - }); - - AddCtbRail(nodes, upcoming); - AddAgentCards(nodes, participants); - AddSelectedAgent(nodes, selectedAgent); - - var selectedNodeId = string.IsNullOrWhiteSpace(currentState.SelectedNodeId) || - nodes.All(node => !string.Equals(node.Id, currentState.SelectedNodeId, StringComparison.Ordinal)) - ? "voidbot-summary" - : currentState.SelectedNodeId; - - var state = new DashboardState - { - ProviderId = ProviderId, - Title = "VoidBot Swarm", - Version = DateTimeOffset.UtcNow.ToUnixTimeSeconds(), - UpdatedAt = DateTimeOffset.UtcNow, - SelectedNodeId = selectedNodeId, - Nodes = nodes, - }; - state.Surface = BuildVoidBotSurface(state, summary, cultMesh, controls, orchestrator, upcoming, participants, selectedAgent); - return state; - } - - private DashboardSurface BuildVoidBotSurface( - DashboardState state, - JsonElement summary, - JsonElement cultMesh, - JsonElement controls, - JsonElement orchestrator, - IReadOnlyList upcoming, - IReadOnlyList participants, - JsonElement selectedAgent) - { - return new DashboardSurface - { - Schema = "cultmesh.eve_surface.v0", - Id = "voidbot.swarm.surface", - Title = "VoidBot Swarm", - Root = UiElement.Container( - "voidbot-cockpit", - "cockpit", - new DashboardUiLayout - { - Direction = "vertical", - Gap = 8, - Padding = 8, - Overflow = "scroll", - Grow = 5, - MinWidth = 112, - MinHeight = 30, - PreferredWidth = 156, - PreferredHeight = 58, - Priority = -30, - Density = "continuous", - ViewportMode = "continuous-ops", - }, - [ - BuildTurnRail(upcoming), - ]), - Assets = [], - }; - } - - private DashboardUiElement BuildTurnRail(IReadOnlyList upcoming) - { - var count = Math.Min(14, upcoming.Count); - var rows = new List(); - for (var index = 0; index < count; index++) - { - var turn = upcoming[index]; - var identityId = StringValue(turn, "identityId") ?? ""; - var active = !string.IsNullOrWhiteSpace(StringValue(turn, "activeJobId")); - var mentionCount = NumberValue(turn, "pendingMentionCount") ?? 0; - var rest = TryGet(turn, "restState"); - var napping = BoolValue(rest, "isNapping") == true; - var state = active ? "active" : mentionCount > 0 ? "mention" : napping ? "nap" : Minutes(NumberValue(turn, "nextTurnInMinutes")); - var speed = NumberValue(turn, "effectiveSpeed") ?? 0; - var heat = NumberValue(turn, "heat") ?? 0; - rows.Add(UiElement.Text( - $"ctb-row-{index}-{identityId}", - $"{index + 1,2}. {(StringValue(turn, "displayName") ?? identityId),-10} {state,-7} s{speed:0.###} h{heat:0.##}", - "mono")); - } - - var rail = UiElement.Container( - "ctb-rail", - "rail", - new DashboardUiLayout { Direction = "vertical", Gap = 4, Padding = 8, Overflow = "scroll", Grow = 1, MinWidth = 40, MinHeight = 20 }, - rows.Count > 0 - ? rows - : [UiElement.Text("ctb-empty", "no CTB turns", "caption")]); - rail.Text = "CTB order"; - return rail; - } - - private DashboardUiElement BuildUpcomingFacesPane(IReadOnlyList upcoming) - { - var rows = upcoming - .Take(10) - .Select((turn, index) => - { - var identityId = StringValue(turn, "identityId") ?? $"turn-{index}"; - var active = !string.IsNullOrWhiteSpace(StringValue(turn, "activeJobId")); - var mentionCount = NumberValue(turn, "pendingMentionCount") ?? 0; - var rest = TryGet(turn, "restState"); - var napping = BoolValue(rest, "isNapping") == true; - var state = active ? "active" : mentionCount > 0 ? "mention" : napping ? "nap" : Minutes(NumberValue(turn, "nextTurnInMinutes")); - var heat = NumberValue(turn, "heat") ?? 0; - var speed = NumberValue(turn, "effectiveSpeed") ?? 0; - return UiElement.Text( - $"upcoming-face-{index}-{identityId}", - $"{index + 1,2}. {(StringValue(turn, "displayName") ?? identityId),-10} {state,-8} {StringValue(turn, "repoName") ?? "repo"} spd {speed:0.###} heat {heat:0.##}", - "mono"); - }) - .ToArray(); - - return UiElement.Pane( - "upcoming-faces-pane", - "Next Faces", - rows.Length > 0 - ? rows - : [UiElement.Text("upcoming-faces-empty", "no upcoming Face turns", "caption")]); - } - - private DashboardUiElement BuildWatchdogPane(JsonElement orchestrator) - { - var state = StringValue(orchestrator, "state") ?? "unknown"; - var organs = ArrayItems(TryGet(orchestrator, "organs")).ToArray(); - var watchdog = organs.FirstOrDefault(organ => string.Equals(StringValue(organ, "id"), "voidbot-operations-watchdog", StringComparison.OrdinalIgnoreCase)); - var children = new List - { - UiElement.Text("watchdog-summary", $"orchestrator {state} organs {organs.Length}", "mono"), - }; - - if (watchdog.ValueKind != JsonValueKind.Undefined) - { - children.Add(UiElement.Text( - "watchdog-status", - $"watchdog {StringValue(watchdog, "lastStatus") ?? "unknown"} exit {(NumberValue(watchdog, "lastExitCode") ?? 0):0}\nlast {ShortIso(StringValue(watchdog, "lastFinishedAt") ?? StringValue(watchdog, "lastStartedAt"))}", - "strong")); - } - else - { - children.Add(UiElement.Text("watchdog-missing", "watchdog organ not present in snapshot", "caption")); - } - - children.AddRange(organs - .Where(static organ => organ.ValueKind != JsonValueKind.Undefined) - .OrderBy(organ => string.Equals(StringValue(organ, "id"), "voidbot-operations-watchdog", StringComparison.OrdinalIgnoreCase) ? 0 : 1) - .ThenBy(organ => StringValue(organ, "id")) - .Take(7) - .Select(organ => UiElement.Text( - $"watchdog-organ-{StableId(StringValue(organ, "id") ?? "organ")}", - $"{StringValue(organ, "label") ?? StringValue(organ, "id") ?? "organ"}: {StringValue(organ, "lastStatus") ?? "unknown"}", - "caption"))); - - return UiElement.Pane("voidbot-watchdog-pane", "VoidBot Watchdog", children); - } - - private DashboardUiElement BuildSelectedFacePane(JsonElement summary, JsonElement cultMesh, JsonElement selectedAgent, double cadence, string generatedAt) - { - var identityId = StringValue(selectedAgent, "identityId") ?? ""; - var faceState = TryGet(selectedAgent, "faceState"); - var counts = TryGet(faceState, "counts"); - var memory = NumberValue(counts, "memory") ?? 0; - var description = StringValue(selectedAgent, "description") ?? "No Face description registered."; - return UiElement.Pane( - "selected-face-pane", - "Controls / Selected Face", - [ - UiElement.Text("voidbot-summary-text", $"VoidBot Swarm\n{StringValue(summary, "state") ?? "unknown"} next {StringValue(summary, "nextDisplayName") ?? "none"}\nagents {NumberValue(summary, "participantCount") ?? 0:0} ready {NumberValue(summary, "readyNowCount") ?? 0:0} cadence x{cadence:0.##}\nmesh {StringValue(cultMesh, "writeStatus") ?? "missing"} {generatedAt}", "mono"), - UiElement.Card( - "selected-face-card", - "inspector-hero", - bindNodeId: "agent-detail", - commandId: $"select-identity:{identityId}", - children: - [ - UiElement.Avatar("selected-face-avatar", StringValue(selectedAgent, "avatarUrl"), StringValue(selectedAgent, "displayName") ?? identityId), - UiElement.Text("selected-face-name", $"Selected Face\n{StringValue(selectedAgent, "displayName") ?? identityId}", "title"), - ]), - UiElement.Metric("metric-turn", "Turn", 0.96, "cool"), - UiElement.Metric("metric-memory", "Memory", Math.Min(1.0, memory * 0.07), "cool"), - UiElement.Metric("metric-pressure", "Pressure", 0.42, "warm"), - UiElement.Metric("metric-heat", "Heat", 0.34, "warm"), - UiElement.Metric("metric-load", "Load", 0.99, "danger"), - UiElement.Metric("metric-speed", "Speed", 0.88, "cool"), - UiElement.Text("selected-face-detail", $"memory {memory:0} pressures {NumberValue(counts, "pressures") ?? 0:0} constraints {NumberValue(selectedAgent, "constraintCount") ?? 0:0}\n{Truncate(description, 220)}", "body"), - ]); - } - - private DashboardUiElement BuildStateGraphPane(IReadOnlyList leaves) - { - return UiElement.Pane( - "state-graph-pane", - "State Graph", - leaves.Select((leaf, index) => UiElement.Card( - $"state-leaf-card-{index}", - "state-leaf", - bindNodeId: $"state-{index}", - commandId: $"select-state:{leaf.Path}", - style: new DashboardUiStyle { Variant = string.Equals(leaf.Path, selectedStatePath, StringComparison.Ordinal) ? "selected" : "default" }, - children: - [ - UiElement.Text($"state-leaf-label-{index}", leaf.Label, "strong"), - UiElement.Text($"state-leaf-preview-{index}", leaf.Preview, "caption"), - ])).ToArray()); - } - - private static DashboardUiElement BuildStateDetailPane(DashboardState state, VoidBotStateLeaf? selectedLeaf) - { - return UiElement.Pane( - "state-detail-pane", - "State Detail", - [ - UiElement.Text("state-detail-heading", selectedLeaf != null ? $"{selectedLeaf.Label}\n{selectedLeaf.Path}" : state.Title, "strong"), - UiElement.Text("state-detail-body", selectedLeaf?.Detail ?? state.Nodes.FirstOrDefault(static node => node.Id == "voidbot-summary")?.Detail ?? "No state detail.", "mono"), - ]); - } - - private void AddCtbRail(List nodes, IReadOnlyList upcoming) - { - var count = Math.Min(8, upcoming.Count); - for (var index = 0; index < count; index++) - { - var turn = upcoming[index]; - var identityId = StringValue(turn, "identityId") ?? ""; - var active = !string.IsNullOrWhiteSpace(StringValue(turn, "activeJobId")); - var mentionCount = NumberValue(turn, "pendingMentionCount") ?? 0; - nodes.Add(new DashboardNode( - $"ctb-{index}-{identityId}", - $"{index + 1}. {StringValue(turn, "displayName") ?? identityId}\n{StringValue(turn, "repoName") ?? "repo"}", - "ctb-turn", - -0.82 + (index * 0.235), - -0.82, - 0.21, - 0.11, - active ? "active" : mentionCount > 0 ? "mention" : Minutes(NumberValue(turn, "nextTurnInMinutes"))) - { - IdentityId = identityId, - AvatarUrl = StringValue(turn, "avatarUrl"), - Detail = $"speed {NumberValue(turn, "effectiveSpeed") ?? 0:0.###} heat {NumberValue(turn, "heat") ?? 0:0.###}", - }); - } - } - - private void AddAgentCards(List nodes, IReadOnlyList participants) - { - var visible = participants - .OrderBy(participant => NumberValue(participant, "nextTurnInMinutes") ?? 999999) - .Take(10) - .ToArray(); - - for (var index = 0; index < visible.Length; index++) - { - var agent = visible[index]; - var identityId = StringValue(agent, "identityId") ?? $"agent-{index}"; - var column = index % 5; - var row = index / 5; - var active = !string.IsNullOrWhiteSpace(StringValue(agent, "activeJobId")); - var selected = string.Equals(identityId, selectedIdentityId, StringComparison.Ordinal); - nodes.Add(new DashboardNode( - $"agent-{identityId}", - $"{StringValue(agent, "displayName") ?? identityId}\n{StringValue(agent, "repoName") ?? "repo"}", - "agent", - -0.74 + (column * 0.37), - -0.28 + (row * 0.22), - 0.31, - 0.17, - selected ? "selected" : active ? "active" : Minutes(NumberValue(agent, "nextTurnInMinutes"))) - { - IdentityId = identityId, - AvatarUrl = StringValue(agent, "avatarUrl"), - Detail = $"load {NumberValue(agent, "currentLoad") ?? 0:0.##} heat {NumberValue(agent, "heat") ?? 0:0.##}", - }); - } - } - - private void AddSelectedAgent(List nodes, JsonElement agent) - { - if (agent.ValueKind == JsonValueKind.Undefined) - { - return; - } - - var identityId = StringValue(agent, "identityId") ?? ""; - var faceState = TryGet(agent, "faceState"); - var counts = TryGet(faceState, "counts"); - var description = StringValue(agent, "description") ?? "No Face description registered."; - nodes.Add(new DashboardNode( - "agent-detail", - $"{StringValue(agent, "displayName") ?? identityId} State\n{StringValue(agent, "repoName") ?? "repo"}", - "state-detail", - -0.42, - 0.52, - 0.55, - 0.30, - BoolValue(faceState, "readable") == true ? "readable" : "unreadable") - { - IdentityId = identityId, - AvatarUrl = StringValue(agent, "avatarUrl"), - Detail = $"memory {NumberValue(counts, "memory") ?? 0:0} pressures {NumberValue(counts, "pressures") ?? 0:0} constraints {NumberValue(agent, "constraintCount") ?? 0:0}\n{Truncate(description, 150)}", - }); - - var leaves = FlattenLeaves(ArrayItems(TryGet(faceState, "tree")).ToArray()).Take(5).ToArray(); - if (string.IsNullOrWhiteSpace(selectedStatePath)) - { - selectedStatePath = leaves.FirstOrDefault()?.Path ?? ""; - } - - for (var index = 0; index < leaves.Length; index++) - { - var leaf = leaves[index]; - var selected = string.Equals(leaf.Path, selectedStatePath, StringComparison.Ordinal); - nodes.Add(new DashboardNode( - $"state-{index}", - $"{leaf.Label}\n{leaf.Preview}", - "state-leaf", - 0.34, - 0.24 + (index * 0.13), - 0.34, - 0.10, - selected ? "open" : "leaf") - { - IdentityId = identityId, - StatePath = leaf.Path, - Detail = leaf.Detail, - }); - } - - var selectedLeaf = leaves.FirstOrDefault(leaf => string.Equals(leaf.Path, selectedStatePath, StringComparison.Ordinal)); - if (selectedLeaf != null && !string.IsNullOrWhiteSpace(selectedLeaf.Path)) - { - nodes.Add(new DashboardNode( - "state-detail", - $"{selectedLeaf.Label}\n{selectedLeaf.Path}", - "state-text", - 0.38, - 0.74, - 0.38, - 0.18, - "detail") - { - IdentityId = identityId, - StatePath = selectedLeaf.Path, - Detail = Truncate(selectedLeaf.Detail, 280), - }); - } - } - - private DashboardState BuildMissingState() => new() - { - ProviderId = ProviderId, - Title = "VoidBot Swarm", - SelectedNodeId = "voidbot-missing", - Nodes = - [ - new DashboardNode("voidbot-missing", "VoidBot Swarm\nsnapshot missing", "swarm", 0.0, -0.10, 0.52, 0.24, "missing") - { - Detail = swarmStatePath, - }, - ], - }; - - private DashboardState BuildErrorState(string error) => new() - { - ProviderId = ProviderId, - Title = "VoidBot Swarm", - SelectedNodeId = "voidbot-error", - Nodes = - [ - new DashboardNode("voidbot-error", "VoidBot Swarm\nstate read failed", "swarm", 0.0, -0.10, 0.52, 0.24, "error") - { - Detail = error, - }, - ], - }; - - private static IEnumerable FlattenLeaves(IReadOnlyList nodes) - { - foreach (var node in nodes) - { - if (StringValue(node, "kind") == "leaf") - { - yield return new VoidBotStateLeaf( - StringValue(node, "label") ?? "leaf", - StringValue(node, "path") ?? "", - StringValue(node, "preview") ?? "", - StringValue(node, "detail") ?? StringValue(node, "preview") ?? ""); - } - - foreach (var child in FlattenLeaves(ArrayItems(TryGet(node, "children")).ToArray())) - { - yield return child; - } - } - } - - private static JsonElement TryGet(JsonElement element, string property) - { - if (element.ValueKind == JsonValueKind.Object && element.TryGetProperty(property, out var child)) - { - return child; - } - - return default; - } - - private static IEnumerable ArrayItems(JsonElement element) - { - if (element.ValueKind != JsonValueKind.Array) - { - yield break; - } - - foreach (var item in element.EnumerateArray()) - { - yield return item; - } - } - - private static string? StringValue(JsonElement element, string property) => StringValue(TryGet(element, property)); - - private static string? StringValue(JsonElement element) => - element.ValueKind == JsonValueKind.String ? element.GetString() : null; - - private static double? NumberValue(JsonElement element, string property) => NumberValue(TryGet(element, property)); - - private static double? NumberValue(JsonElement element) => - element.ValueKind == JsonValueKind.Number && element.TryGetDouble(out var value) ? value : null; - - private static bool? BoolValue(JsonElement element, string property) - { - var child = TryGet(element, property); - return child.ValueKind switch - { - JsonValueKind.True => true, - JsonValueKind.False => false, - _ => null, - }; - } - - private static string Minutes(double? value) - { - if (value == null) - { - return "unknown"; - } - - return value <= 0 ? "ready" : $"{value:0.#}m"; - } - - private static string Truncate(string value, int max) => - value.Length <= max ? value : value[..Math.Max(0, max - 1)] + "..."; - - private static string StableId(string value) - { - var builder = new StringBuilder(value.Length); - foreach (var ch in value.ToLowerInvariant()) - { - builder.Append(char.IsLetterOrDigit(ch) ? ch : '-'); - } - - return builder.ToString().Trim('-'); - } - - private static string ShortIso(string? value) - { - if (string.IsNullOrWhiteSpace(value)) - { - return "unknown"; - } - - return DateTimeOffset.TryParse(value, out var timestamp) - ? timestamp.ToLocalTime().ToString("MM-dd HH:mm") - : value; - } -} - -internal sealed record VoidBotStateLeaf(string Label, string Path, string Preview, string Detail); - -internal sealed class YggdrasilStreamPixelsProvider : MutableDashboardProvider -{ - public const string ProviderId = "yggdrasil.streampixels.edge"; - - public YggdrasilStreamPixelsProvider() - : base( - new DashboardProviderManifest( - ProviderId, - "StreamPixels Edge", - "Yggdrasil/StreamPixels live edge control surface reachable over HTTPS or SSH-forwarded localhost.", - "1", - "/eve/deck/yggdrasil.streampixels.edge", - ["status", "health", "ssh-tunnel", "hls"], - UsesCultMesh: false, - Transport: "TCP/WebSocket over LAN or SSH tunnel."), - new DashboardState - { - ProviderId = ProviderId, - Title = "StreamPixels Edge", - SelectedNodeId = "hls-origin", - Nodes = - [ - new DashboardNode("hls-origin", "HLS Origin", "service", -0.34, -0.18, 0.34, 0.22, "streampixels"), - new DashboardNode("rtmp-ingest", "RTMP Ingest", "service", 0.34, -0.18, 0.34, 0.22, "localhost"), - new DashboardNode("ssh-tunnel", "Yggdrasil SSH Tunnel", "transport", 0.0, 0.34, 0.44, 0.18, "tcp only"), - ], - }) - { - } -} - -internal sealed class RemoteDashboardProvider : MutableDashboardProvider -{ - private RemoteDashboardProvider(DashboardProviderManifest manifest, string upstream) - : base( - manifest, - new DashboardState - { - ProviderId = manifest.Id, - Title = manifest.Title, - SelectedNodeId = "remote-endpoint", - Nodes = - [ - new DashboardNode("remote-endpoint", manifest.Title, "remote-dashboard", 0.0, -0.10, 0.46, 0.24, "registered") - { - Endpoint = upstream, - }, - new DashboardNode("remote-transport", "Transport", "transport", 0.0, 0.34, 0.40, 0.18, "tcp/ws") - { - Endpoint = upstream, - }, - ], - }) - { - } - - public static RemoteDashboardProvider? FromSpec(string spec) - { - var parts = spec.Split('|', 3, StringSplitOptions.TrimEntries); - if (parts.Length < 3 || string.IsNullOrWhiteSpace(parts[0]) || string.IsNullOrWhiteSpace(parts[1]) || string.IsNullOrWhiteSpace(parts[2])) - { - Console.Error.WriteLine($"Ignoring dashboard provider spec. Expected id|title|ws-url, got: {spec}"); - return null; - } - - return new RemoteDashboardProvider( - new DashboardProviderManifest( - parts[0], - parts[1], - "External EveDeck dashboard provider registered at broker launch.", - "1", - parts[2], - ["scene2d", "external-provider"], - UsesCultMesh: false, - Transport: "registered WebSocket endpoint"), - parts[2]); - } -} - -internal sealed record HttpRequest(string Path, IReadOnlyDictionary Headers); - -internal sealed record WebSocketFrame(int Opcode, byte[] Payload); - -internal sealed record DashboardSocket(TcpClient Client, NetworkStream Stream, bool CultMeshBinary); - -internal sealed record DashboardProviderCatalogDocument(IReadOnlyList Providers); - -internal sealed record DashboardProviderManifest( - string Id, - string Title, - string Description, - string Version, - string Endpoint, - string[] Capabilities, - bool UsesCultMesh, - string Transport); - -internal sealed class DashboardState -{ - public string Type { get; set; } = "dashboard-state"; - - public string Schema { get; set; } = "mimir.eve_dashboard_state.v1"; - - public string ProviderId { get; set; } = MimirStreamLayoutProvider.ProviderId; - - public string Title { get; set; } = "Mimir Dashboard"; - - public long Version { get; set; } = 1; - - public DateTimeOffset UpdatedAt { get; set; } = DateTimeOffset.UtcNow; - - public string SelectedNodeId { get; set; } = "program"; - - public string LutPreset { get; set; } = "neutral"; - - public List Nodes { get; set; } = []; - - public DashboardSurface? Surface { get; set; } -} - -internal sealed class DashboardSurface -{ - public string Schema { get; set; } = "cultmesh.eve_surface.v0"; - - public string Id { get; set; } = ""; - - public string Title { get; set; } = ""; - - public DashboardUiElement Root { get; set; } = UiElement.Container("root", "root", new DashboardUiLayout(), []); - - public IReadOnlyList Assets { get; set; } = []; -} - -internal sealed record DashboardSurfaceAsset(string Id, string Kind, string Uri); - -internal sealed class DashboardUiBinding -{ - public string DocumentSchema { get; set; } = ""; - - public string DocumentId { get; set; } = ""; - - public string Path { get; set; } = ""; - - public string ValueKind { get; set; } = ""; - - public string Access { get; set; } = "read"; - - public string Authority { get; set; } = ""; - - public string? CommandId { get; set; } -} - -internal sealed class DashboardUiElement -{ - public string Id { get; set; } = ""; - - public string Kind { get; set; } = ""; - - public string? Role { get; set; } - - public string? Text { get; set; } - - public string? AssetRef { get; set; } - - public string? AssetUri { get; set; } - - public string? BindNodeId { get; set; } - - public string? CommandId { get; set; } - - public DashboardUiBinding? Binding { get; set; } - - public DashboardUiLayout? Layout { get; set; } - - public DashboardUiStyle? Style { get; set; } - - public DashboardUiMetric? Metric { get; set; } - - public IReadOnlyList Children { get; set; } = []; -} - -internal sealed class DashboardUiLayout -{ - public string Direction { get; set; } = "vertical"; - - public double? Width { get; set; } - - public double? Height { get; set; } - - public double? Grow { get; set; } - - public double? Gap { get; set; } - - public double? Padding { get; set; } - - public string? Overflow { get; set; } - - public double? MinWidth { get; set; } - - public double? MinHeight { get; set; } - - public double? PreferredWidth { get; set; } - - public double? PreferredHeight { get; set; } - - public double? Priority { get; set; } - - public string? Density { get; set; } - - public string? ViewportMode { get; set; } -} - -internal sealed class DashboardUiStyle -{ - public string Variant { get; set; } = "default"; - - public string? Tone { get; set; } -} - -internal sealed record DashboardUiMetric(string Label, double Value, string Tone); - -internal static class UiElement -{ - public static DashboardUiElement Container(string id, string kind, DashboardUiLayout layout, IReadOnlyList children) - { - return new DashboardUiElement { Id = id, Kind = kind, Layout = layout, Children = children }; - } - - public static DashboardUiElement Pane(string id, string title, IReadOnlyList children) - { - return new DashboardUiElement - { - Id = id, - Kind = "pane", - Text = title, - Layout = new DashboardUiLayout { Direction = "vertical", Gap = 10, Padding = 12, Grow = 1 }, - Children = children, - }; - } - - public static DashboardUiElement Card( - string id, - string role, - string? bindNodeId = null, - string? commandId = null, - DashboardUiStyle? style = null, - IReadOnlyList? children = null) - { - return new DashboardUiElement - { - Id = id, - Kind = "card", - Role = role, - BindNodeId = bindNodeId, - CommandId = commandId, - Style = style, - Children = children ?? [], - }; - } - - public static DashboardUiElement Avatar(string id, string? avatarUrl, string alt) - { - return new DashboardUiElement - { - Id = id, - Kind = "avatar", - Text = alt, - AssetUri = avatarUrl, - Layout = new DashboardUiLayout { Width = 42, Height = 42 }, - }; - } - - public static DashboardUiElement Text(string id, string text, string role) - { - return new DashboardUiElement { Id = id, Kind = "text", Role = role, Text = text }; - } - - public static DashboardUiElement Metric(string id, string label, double value, string tone) - { - return new DashboardUiElement - { - Id = id, - Kind = "metric", - Metric = new DashboardUiMetric(label, Math.Clamp(value, 0.0, 1.0), tone), - }; - } -} - -internal sealed class DashboardNode( - string id, - string label, - string kind, - double defaultX, - double defaultY, - double width, - double height, - string health) -{ - public string Id { get; set; } = id; - - public string Label { get; set; } = label; - - public string Kind { get; set; } = kind; - - public bool Visible { get; set; } = true; - - public double X { get; set; } = defaultX; - - public double Y { get; set; } = defaultY; - - public double DefaultX { get; set; } = defaultX; - - public double DefaultY { get; set; } = defaultY; - - public double Z { get; set; } - - public double Rotation { get; set; } - - public double Scale { get; set; } = 1.0; - - public double Width { get; set; } = width; - - public double Height { get; set; } = height; - - public string Health { get; set; } = health; - - public string? ProviderId { get; set; } - - public string? Command { get; set; } - - public string? Endpoint { get; set; } - - public string? IdentityId { get; set; } - - public string? AvatarUrl { get; set; } - - public string? StatePath { get; set; } - - public string? Detail { get; set; } -} - -internal sealed class DashboardCommand -{ - public string Type { get; set; } = ""; - - public string NodeId { get; set; } = ""; - - public string? ProviderId { get; set; } - - public double? X { get; set; } - - public double? Y { get; set; } - - public double? Rotation { get; set; } - - public double? Scale { get; set; } - - public bool? Visible { get; set; } -} diff --git a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs index 7245794..17b8022 100644 --- a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs +++ b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs @@ -157,153 +157,6 @@ public sealed record MimirProgramOutputDocument( [property: Key(5)] string PublisherRoute, [property: Key(6)] bool DiagnosticOnly); -[CultDocument("mimir.eve_operator_surface", "mimir.eve_operator_surface.v1")] -[MessagePackObject] -public sealed record MimirEveOperatorSurfaceDocument( - [property: Key(0)] - [property: CultName] - string SurfaceId, - [property: Key(1)] string UpdatedAtUtc, - [property: Key(2)] string SceneId, - [property: Key(3)] bool CanPreviewProgram, - [property: Key(4)] bool CanEditComposition, - [property: Key(5)] bool CanSelectSources, - [property: Key(6)] bool CanShowStats, - [property: Key(7)] string[] CommandTopics); - -[CultDocument("mimir.eve_dashboard_manifest", "mimir.eve_dashboard_manifest.v1")] -[MessagePackObject] -public sealed record MimirEveDashboardManifestDocument( - [property: Key(0)] - [property: CultName] - string ProviderId, - [property: Key(1)] string Title, - [property: Key(2)] string Description, - [property: Key(3)] string Version, - [property: Key(4)] string Endpoint, - [property: Key(5)] string[] Capabilities, - [property: Key(6)] bool UsesCultMesh, - [property: Key(7)] string Transport); - -[CultDocument("mimir.eve_dashboard_state", "mimir.eve_dashboard_state.v1")] -[MessagePackObject] -public sealed record MimirEveDashboardStateDocument( - [property: Key(0)] - [property: CultName] - string ProviderId, - [property: Key(1)] string Title, - [property: Key(2)] long Version, - [property: Key(3)] string UpdatedAtUtc, - [property: Key(4)] string SelectedNodeId, - [property: Key(5)] string LutPreset, - [property: Key(6)] MimirEveDashboardNodeSnapshot[] Nodes, - [property: Key(7)] MimirEveDashboardSurfaceSnapshot? Surface); - -[MessagePackObject] -public sealed record MimirEveDashboardNodeSnapshot( - [property: Key(0)] string Id, - [property: Key(1)] string Label, - [property: Key(2)] string Kind, - [property: Key(3)] bool Visible, - [property: Key(4)] double X, - [property: Key(5)] double Y, - [property: Key(6)] double Z, - [property: Key(7)] double Rotation, - [property: Key(8)] double Scale, - [property: Key(9)] double Width, - [property: Key(10)] double Height, - [property: Key(11)] string Health, - [property: Key(12)] string? ProviderId, - [property: Key(13)] string? Command, - [property: Key(14)] string? Endpoint); - -[MessagePackObject] -public sealed record MimirEveDashboardSurfaceSnapshot( - [property: Key(0)] string Schema, - [property: Key(1)] string Id, - [property: Key(2)] string Title, - [property: Key(3)] MimirEveDashboardUiElementSnapshot Root, - [property: Key(4)] MimirEveDashboardSurfaceAssetSnapshot[] Assets); - -[MessagePackObject] -public sealed record MimirEveDashboardSurfaceAssetSnapshot( - [property: Key(0)] string Id, - [property: Key(1)] string Kind, - [property: Key(2)] string Uri); - -[MessagePackObject] -public sealed record MimirEveDashboardUiElementSnapshot( - [property: Key(0)] string Id, - [property: Key(1)] string Kind, - [property: Key(2)] string? Role, - [property: Key(3)] string? Text, - [property: Key(4)] string? AssetRef, - [property: Key(5)] string? AssetUri, - [property: Key(6)] string? BindNodeId, - [property: Key(7)] string? CommandId, - [property: Key(8)] MimirEveDashboardUiLayoutSnapshot? Layout, - [property: Key(9)] MimirEveDashboardUiStyleSnapshot? Style, - [property: Key(10)] MimirEveDashboardUiMetricSnapshot? Metric, - [property: Key(11)] MimirEveDashboardUiElementSnapshot[] Children, - [property: Key(12)] MimirEveDashboardUiBindingSnapshot? Binding = null); - -[MessagePackObject] -public sealed record MimirEveDashboardUiBindingSnapshot( - [property: Key(0)] string DocumentSchema, - [property: Key(1)] string DocumentId, - [property: Key(2)] string Path, - [property: Key(3)] string ValueKind, - [property: Key(4)] string Access, - [property: Key(5)] string Authority, - [property: Key(6)] string? CommandId); - -[MessagePackObject] -public sealed record MimirEveDashboardUiLayoutSnapshot( - [property: Key(0)] string Direction, - [property: Key(1)] double? Width, - [property: Key(2)] double? Height, - [property: Key(3)] double? Grow, - [property: Key(4)] double? Gap, - [property: Key(5)] double? Padding, - [property: Key(6)] string? Overflow, - [property: Key(7)] double? MinWidth = null, - [property: Key(8)] double? MinHeight = null, - [property: Key(9)] double? PreferredWidth = null, - [property: Key(10)] double? PreferredHeight = null, - [property: Key(11)] double? Priority = null, - [property: Key(12)] string? Density = null, - [property: Key(13)] string? ViewportMode = null); - -[MessagePackObject] -public sealed record MimirEveDashboardUiStyleSnapshot( - [property: Key(0)] string Variant, - [property: Key(1)] string? Tone); - -[MessagePackObject] -public sealed record MimirEveDashboardUiMetricSnapshot( - [property: Key(0)] string Label, - [property: Key(1)] double Value, - [property: Key(2)] string Tone); - -[CultDocument("mimir.eve_dashboard_command", "mimir.eve_dashboard_command.v1")] -[MessagePackObject] -public sealed record MimirEveDashboardCommandDocument( - [property: Key(0)] - [property: CultName] - string CommandId, - [property: Key(1)] string DeviceId, - [property: Key(2)] string ClientId, - [property: Key(3)] string ProviderId, - [property: Key(4)] string Type, - [property: Key(5)] string NodeId, - [property: Key(6)] double? X, - [property: Key(7)] double? Y, - [property: Key(8)] double? Rotation, - [property: Key(9)] double? Scale, - [property: Key(10)] bool? Visible, - [property: Key(11)] long Sequence, - [property: Key(12)] long DeviceTimestampNs); - [CultDocument("mimir.eve_sensor_observation", "mimir.eve_sensor_observation.v1")] [MessagePackObject] public sealed record MimirEveSensorObservationDocument( @@ -482,29 +335,6 @@ public static class MimirCultMeshContractFactory publication.SitePublisherRoute, publication.DiagnosticOnly); - public static MimirEveOperatorSurfaceDocument CreateOperatorSurface( - string surfaceId, - string sceneId, - MimirProgramControlSurface surface, - DateTimeOffset? updatedAt = null) => - new( - surfaceId, - (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"), - sceneId, - surface.CanPreviewProgram, - surface.CanEditComposition, - surface.CanSelectSources, - surface.CanShowStats, - CommandTopics: - [ - "mimir.program.scene.select", - "mimir.program.layer.transform", - "mimir.program.layer.crop", - "mimir.program.layer.chroma_key", - "mimir.program.source.subscribe", - "mimir.program.output.publish" - ]); - private static MimirBioacousticDecoderConfigurationSnapshot Snapshot(MimirBioacousticDecoderConfiguration configuration) => new( configuration.Id, diff --git a/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs index 5dafc6d..afb72a5 100644 --- a/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirProgramPublicationConfiguration.cs @@ -20,8 +20,6 @@ public enum MimirProgramAudioPublicationKind public enum MimirProgramControlPublicationKind { - EveGui, - EveTui, CultMeshCommandSurface } @@ -70,25 +68,9 @@ public static class MimirProgramPublicationConfigurations public static IReadOnlyList OperatorSurfaces { get; } = [ - new( - "eve-gui-compositor", - "Primary Eve GUI composition surface: preview, source selection, transforms, crop/key controls, audio stem controls, and stream health.", - MimirProgramControlPublicationKind.EveGui, - CanPreviewProgram: true, - CanEditComposition: true, - CanSelectSources: true, - CanShowStats: true), - new( - "eve-tui-operator", - "Compact Eve TUI operator surface for source health, live stats, scene selection, and emergency command paths.", - MimirProgramControlPublicationKind.EveTui, - CanPreviewProgram: false, - CanEditComposition: false, - CanSelectSources: true, - CanShowStats: true), new( "cultmesh-command-surface", - "Typed CultMesh command surface consumed by Eve lowerers and trusted devices; Mimir remains the scene authority.", + "Typed CultMesh command surface consumed by trusted devices; Mimir remains the scene authority.", MimirProgramControlPublicationKind.CultMeshCommandSurface, CanPreviewProgram: false, CanEditComposition: true, diff --git a/state/map.yaml b/state/map.yaml index eca4e3a..40c44dc 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -652,16 +652,6 @@ accepted_design: receiver. The old `CultMesh.StartNodeAsync`/`CultMesh.ConnectClient` node/client path and raw sender/receiver host-port config are no longer transport authorities for this media lane. - - `src/Mimir.EveDashboard` is archived as a fail-closed tombstone. It no - longer opens a TCP listener, serves `/eve/deck`, accepts WebSocket commands, - or publishes Idunn health. Its replacement must publish provider - advertisement, retained state, command boundary, and transport profile as - typed CultMesh/Odin documents; product/debug render surfaces remain client - lowerings. - - `src/Mimir.EveBrowserReference` serves a static Eve/browser lowering and - can publish its own `idunn.daemon_health` record over - `cultnet.transport.rudp.v0`. It is a renderer/reference health witness, not - an authority over Mimir program state or Idunn keepalive decisions. - `research/perfect-machine-study-2026-05-23/` preserves the architecture rumination, optimization ledger, research references, source/hotspot audits, Fensalir/native boundary maps, benchmark plan, calibration-session spec, @@ -690,13 +680,7 @@ active_subgoals: replaces them. - Launch Raven's CultMesh sender directly on Raven because SSH to Raven over WireGuard timed out during the initial relay deployment pass. - - Keep Mimir Eve dashboard and browser reference aligned with Odin/Idunn - doctrine: Odin accepts Verse/service visibility, Idunn owns keepalive, and - Mimir surfaces publish typed CultMesh/Odin state rather than private - watchdog loops, product status probes, or socket brokers. - - Rebuild `src/Mimir.EveDashboard` only after daemon truth, provider catalogs, - command boundaries, and typed program document paths use explicit CultMesh/ - Odin documents. `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the + - `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the Nightwing typed witness publisher, and `start-nightwing-move-tracking.ps1` are archived so the old witness subscription lane cannot retake transport authority. `src/Mimir.CultMeshMedia` has completed this cut for the From c6f4541eea61890e852a12651842c7b4f26ff666 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 22 Oct 2026 23:25:39 +0200 Subject: [PATCH 135/135] State: EveSensorReceiver is live and frozen until Thing 2, not archived (thing2:ruling:mimir-sensor-receiver-frozen) Co-Authored-By: Claude Opus 5.5 --- state/evidence.jsonl | 1 + state/map.yaml | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/state/evidence.jsonl b/state/evidence.jsonl index abddacc..d395ab0 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -205,3 +205,4 @@ {"ts": "2026-07-17T10:46:17+01:00", "type": "verification", "status": "ok", "note": "Final snapshot-only Muninn/Sleipnir input build delivered 100/100 one-frame A-button presses and releases over Raven LAN under seeded 5% datagram loss (254/5251 dropped): Starfire observer 100/100, Muninn captured 200 semantic edges, Raven observer 100/100, final pad neutral, proxy queue overflow zero/max queue 28. Raven-visible hold duration median 29.166 ms, p95 36.719 ms, p99 62.667 ms, max 118.074 ms; retransmission tail remains the next latency pressure point."} {"ts": "2026-07-17T11:20:23+01:00", "type": "rejected_path", "status": "ok", "note": "Application semantic replay over the reliable hid.edge channel was built and field-falsified under 5% loss: new reliable duplicates accumulated transport debt and worsened Raven-visible holds to multi-second outliers. The reliable replay cut was deleted; semantic assist must never create retransmission debt."} {"ts": "2026-07-17T11:20:23+01:00", "type": "verification", "status": "ok", "note": "Native 1 ms XInput witnesses replaced scheduling-contaminated PowerShell polling. Under concurrent sustained Raven media at seeded 1% loss and 100 one-frame input taps at seeded 5% loss, dual-path input (one reliable hid.edge owner plus bounded unreliable unordered hid.edge.assist repeats) delivered exact 100/100 and final neutral. Raven hold p95 improved 78.875->38.000 ms, p99 264.745->53.235 ms, max 272.107->59.028 ms; network-added median 11.617 ms, p95 17.387 ms, max 23.267 ms; media and input proxy queue overflow remained zero."} +{"ts": "2026-10-22T21:30:00Z", "type": "correction", "status": "ok", "note": "Corrects the 2026-07-04 transport-cut entry: Mimir.EveSensorReceiver was not archived; it is in Mimir.slnx and launched by scripts/start-nightwing-move-tracking.ps1, and still listens. Operator ruling thing2:ruling:mimir-sensor-receiver-frozen: frozen, neither extended nor removed until Thing 2. The Eve dashboard projects were removed in 43cebb6 (parked/mimir-eve-surface)."} diff --git a/state/map.yaml b/state/map.yaml index 40c44dc..592b270 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -680,10 +680,12 @@ active_subgoals: replaces them. - Launch Raven's CultMesh sender directly on Raven because SSH to Raven over WireGuard timed out during the initial relay deployment pass. - - `Mimir.EveSensorReceiver`, `Mimir.VerseRecorder`, the - Nightwing typed witness publisher, and `start-nightwing-move-tracking.ps1` - are archived so the old witness subscription lane cannot retake transport - authority. `src/Mimir.CultMeshMedia` has completed this cut for the + - `Mimir.VerseRecorder` and the Nightwing typed witness publisher are + archived so the old witness subscription lane cannot retake transport + authority. `Mimir.EveSensorReceiver` (the `/eve/camera` ingest that + `start-nightwing-move-tracking.ps1` launches) is still built and live, and + is frozen: neither extended nor removed until Thing 2 + (`thing2:ruling:mimir-sensor-receiver-frozen`, 2026-10-22). `src/Mimir.CultMeshMedia` has completed this cut for the Raven/Yggdrasil/Starfire media-frame lane. current_status: