diff --git a/config/mimir-runtime.stream-proof.local.json b/config/mimir-runtime.stream-proof.local.json index 9b3dc6d..c1b85fe 100644 --- a/config/mimir-runtime.stream-proof.local.json +++ b/config/mimir-runtime.stream-proof.local.json @@ -85,7 +85,7 @@ "-flags", "low_delay", "-i", - "srt://0.0.0.0:5200?mode=listener&latency=120000&timeout=5000000", + "srt://0.0.0.0:5210?mode=listener&latency=20000&timeout=5000000", "-an", "-f", "rawvideo", @@ -121,7 +121,7 @@ "-ac", "2", "-i", - "srt://0.0.0.0:5202?mode=listener&latency=120000&timeout=5000000", + "srt://0.0.0.0:5212?mode=listener&latency=20000&timeout=5000000", "-f", "f32le", "-" diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 6e4b06a..80ffed3 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -488,12 +488,15 @@ a named invariant that the native runtime cannot protect yet. become one valid FieldEvidence claim/candidate without claiming 6DoF pose. `config/mimir-runtime.stream-proof.local.json` names the current stream-proof profile: Kiyo Pro AR program view, Leap stereo IR depth root, Raven screen - capture on SRT port 5200, Raven Realtek loopback as decoded f32 PCM on SRT - port 5202, and Scarlett ASIO channels labeled as shotgun/cardioid/local + and Raven Realtek loopback from one muxed Raven NVENC/AAC MPEG-TS SRT stream + on port 5200. Starfire demuxes that single Raven transport locally into + `raven-display` on port 5210 and `raven-realtk-loopback` f32 PCM on port + 5212 because Mimir's current source adapters ingest raw video and PCM + separately. Scarlett ASIO channels remain labeled as shotgun/cardioid/local Raven loopback evidence. Nightwing PS3 Eyes no longer appear as Starfire raw video sources in this proof; they publish compact Eye/Move observations. - `MimirFfmpegPcmAudioStreamSource` is the network PCM audio owner for Raven - Realtek loopback. `config/mimir-runtime.move-stream.local.json` remains a + `MimirFfmpegPcmAudioStreamSource` is the network PCM audio owner after local + Raven demux. `config/mimir-runtime.move-stream.local.json` remains a local/raw-Eye show reference, not the current split-host authority. The distributed authority split is now explicit: Nightwing owns the local PS3 Eye USB reads, PS Move Bluetooth/HID link, LED schedules, device diff --git a/docs/move-controller-stream.md b/docs/move-controller-stream.md index c126baf..ad1aa2f 100644 --- a/docs/move-controller-stream.md +++ b/docs/move-controller-stream.md @@ -98,10 +98,11 @@ The profile declares: - `leap-stereo-ir`: LeapUVC stereo IR / depth root. - `kiyo-pro-rgb`: Kiyo Pro AR program view. -- `raven-display`: Raven screen capture over SRT port `5200`, in the - `raven-sync` clock domain. -- `raven-realtk-loopback`: Raven Realtek render loopback as f32 PCM over SRT - port `5202`, also in `raven-sync`. +- `raven-display`: Raven screen capture from the muxed Raven A/V stream, + locally demuxed to SRT port `5210`, in the `raven-sync` clock domain. +- `raven-realtk-loopback`: Raven Realtek render loopback from the same muxed + Raven A/V stream, locally demuxed as f32 PCM to SRT port `5212`, also in + `raven-sync`. - `focusrite-asio`: Starfire hero mics plus local Raven/program loopback lanes on `asio-ch0..asio-ch3`. @@ -158,30 +159,37 @@ dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --secon ## Raven Sender -On Raven, start screen capture from the logged-in desktop session: +On Raven, start the muxed A/V sender from the logged-in desktop session: ```powershell -E:\Projects\Mimir\scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200 +E:\Projects\Mimir\scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200 ``` -Start Realtek render loopback from the same logged-in desktop session: +That sender captures the desktop through `gdigrab`, encodes video with +`h264_nvenc`, captures Realtek/default render loopback through the repo WASAPI +loopback script, encodes audio as AAC, and muxes both into one MPEG-TS SRT +stream. On Starfire, `scripts\start-stream-proof.ps1` starts +`scripts\start-raven-av-demux.ps1`, which listens on port `5200` and splits the +single transport into local Mimir ingest legs: ```powershell -E:\Projects\Mimir\scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202 +raven-display local SRT 5210, copied H.264/MPEG-TS for decode +raven-realtk-loopback local SRT 5212, decoded f32 PCM ``` If Raven program audio is also routed into Starfire Scarlett on -`asio-ch2/asio-ch3`, that remains the strongest local timing witness. The -Realtek loopback is now a real network PCM audio source in Mimir, not generic -stdout bytes. +`asio-ch2/asio-ch3`, that remains another strong local timing witness. The +Realtek loopback is now part of the same Raven transport as the display before +Starfire demuxes it for Mimir's raw ingest APIs. Raven display sync is not allowed to ride on SRT arrival time. Both `raven-display` and `raven-realtk-loopback` are in the `raven-sync` clock domain. When audio sync estimates the Realtek loopback offset against the Scarlett reference, `MimirSynchronizedBufferPlanner` applies that correction to -the whole `raven-sync` domain, including the display. A later single muxed -Raven A/V transport can tighten this further, but the current separate-leg -profile has an explicit audio-derived correction path. +the whole `raven-sync` domain, including the display. The LAN transport is now +one muxed A/V stream; the local split exists only at the Starfire demux edge +because Mimir currently ingests raw video and PCM through separate source +adapters. ## Kiyo AR Trail Receipt diff --git a/scripts/start-raven-av-demux.ps1 b/scripts/start-raven-av-demux.ps1 new file mode 100644 index 0000000..a90394e --- /dev/null +++ b/scripts/start-raven-av-demux.ps1 @@ -0,0 +1,69 @@ +param( + [int]$InputPort = 5200, + [int]$VideoPort = 5210, + [int]$AudioPort = 5212, + [int]$AudioSampleRate = 48000, + [int]$AudioChannels = 2, + [string]$FfmpegPath = "ffmpeg", + [string]$LogRoot = "E:\Projects\Mimir\artifacts\runtime\stream-proof", + [switch]$DryRun +) + +$ErrorActionPreference = "Stop" + +New-Item -ItemType Directory -Force -Path $LogRoot | Out-Null +$timestamp = Get-Date -Format "yyyyMMdd-HHmmss" +$stdoutLog = Join-Path $LogRoot "raven-av-demux-$timestamp.out.log" +$stderrLog = Join-Path $LogRoot "raven-av-demux-$timestamp.err.log" +$inputEndpoint = "srt://0.0.0.0:${InputPort}?mode=listener&latency=120000&timeout=5000000" +$videoEndpoint = "srt://127.0.0.1:${VideoPort}?mode=caller&latency=20000" +$audioEndpoint = "srt://127.0.0.1:${AudioPort}?mode=caller&latency=20000" + +$arguments = @( + "-hide_banner", + "-nostdin", + "-loglevel", "warning", + "-fflags", "nobuffer", + "-flags", "low_delay", + "-i", $inputEndpoint, + "-map", "0:v:0", + "-an", + "-c:v", "copy", + "-f", "mpegts", + $videoEndpoint, + "-map", "0:a:0", + "-vn", + "-acodec", "pcm_f32le", + "-ar", $AudioSampleRate.ToString(), + "-ac", $AudioChannels.ToString(), + "-f", "f32le", + $audioEndpoint +) + +Write-Host "Raven muxed A/V demux:" +Write-Host " input: $inputEndpoint" +Write-Host " video: $videoEndpoint" +Write-Host " audio: $audioEndpoint" +Write-Host " stdout: $stdoutLog" +Write-Host " stderr: $stderrLog" + +if ($DryRun) { + Write-Host ($FfmpegPath + " " + ($arguments -join " ")) + exit 0 +} + +Get-CimInstance Win32_Process | + Where-Object { $_.ProcessName -like "ffmpeg*" -and $_.CommandLine -match [regex]::Escape($inputEndpoint) } | + ForEach-Object { + try { Stop-Process -Id $_.ProcessId -Force -ErrorAction Stop } catch {} + } + +$process = Start-Process ` + -FilePath $FfmpegPath ` + -ArgumentList $arguments ` + -WindowStyle Hidden ` + -RedirectStandardOutput $stdoutLog ` + -RedirectStandardError $stderrLog ` + -PassThru + +Write-Host "Started Raven muxed A/V demux pid=$($process.Id)" diff --git a/scripts/start-raven-av-sender.ps1 b/scripts/start-raven-av-sender.ps1 new file mode 100644 index 0000000..5391278 --- /dev/null +++ b/scripts/start-raven-av-sender.ps1 @@ -0,0 +1,81 @@ +param( + [string]$TargetHost = "192.168.1.66", + [int]$Port = 5200, + [int]$Width = 1920, + [int]$Height = 1080, + [int]$Framerate = 30, + [int]$AudioSampleRate = 48000, + [int]$AudioChannels = 2, + [string]$FfmpegPath = "ffmpeg", + [string]$Source = "desktop", + [string]$VideoBitrate = "12000k", + [string]$AudioBitrate = "192k", + [string]$LogRoot = "C:\Meta\Mimir\logs", + [switch]$DryRun +) + +$ErrorActionPreference = "Stop" + +New-Item -ItemType Directory -Force -Path $LogRoot | Out-Null +$repo = Split-Path -Parent $PSScriptRoot +$timestamp = Get-Date -Format "yyyyMMdd-HHmmss" +$stdoutLog = Join-Path $LogRoot "raven-av-srt-$timestamp.out.log" +$stderrLog = Join-Path $LogRoot "raven-av-srt-$timestamp.err.log" +$endpoint = "srt://${TargetHost}:${Port}?mode=caller&latency=120000" +$videoSize = "${Width}x${Height}" +$gop = [Math]::Max(1, $Framerate * 2) +$captureScript = Join-Path $repo "scripts\wasapi-loopback-capture.ps1" + +$ffmpegArgs = @( + "-hide_banner", + "-loglevel", "warning", + "-thread_queue_size", "1024", + "-f", "gdigrab", + "-framerate", $Framerate.ToString(), + "-video_size", $videoSize, + "-i", $Source, + "-thread_queue_size", "1024", + "-f", "f32le", + "-ar", $AudioSampleRate.ToString(), + "-ac", $AudioChannels.ToString(), + "-i", "pipe:0", + "-map", "0:v:0", + "-map", "1:a:0", + "-c:v", "h264_nvenc", + "-preset", "p4", + "-tune", "ll", + "-b:v", $VideoBitrate, + "-maxrate", $VideoBitrate, + "-bufsize", "24000k", + "-pix_fmt", "yuv420p", + "-g", $gop.ToString(), + "-c:a", "aac", + "-b:a", $AudioBitrate, + "-ar", $AudioSampleRate.ToString(), + "-ac", $AudioChannels.ToString(), + "-f", "mpegts", + $endpoint +) +$command = "& `"$captureScript`" -Output - -SampleRate $AudioSampleRate -Channels $AudioChannels | & `"$FfmpegPath`" $($ffmpegArgs -join ' ')" + +Write-Host "Raven muxed A/V sender:" +Write-Host " ffmpeg: $FfmpegPath" +Write-Host " target: $endpoint" +Write-Host " video: $Source $videoSize@$Framerate via h264_nvenc" +Write-Host " audio: Realtek/default render loopback ${AudioChannels}ch ${AudioSampleRate}Hz -> AAC" +Write-Host " stdout: $stdoutLog" +Write-Host " stderr: $stderrLog" + +if ($DryRun) { + Write-Host $command + exit 0 +} + +Start-Process ` + -FilePath "powershell" ` + -ArgumentList @("-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", $command) ` + -WindowStyle Hidden ` + -RedirectStandardOutput $stdoutLog ` + -RedirectStandardError $stderrLog ` + -PassThru | + ForEach-Object { Write-Host "Started Raven muxed A/V sender pid=$($_.Id)" } diff --git a/scripts/start-stream-proof.ps1 b/scripts/start-stream-proof.ps1 index 05aeb28..2f37c0a 100644 --- a/scripts/start-stream-proof.ps1 +++ b/scripts/start-stream-proof.ps1 @@ -17,9 +17,9 @@ New-Item -ItemType Directory -Force -Path $logDir | Out-Null if ($DryRun) { Write-Host "MIMIR_RUNTIME_CONFIG=$ConfigPath" Write-Host "Would stage Nightwing witness tools to $NightwingHost and start --track-eyes publisher at $WitnessUrl" + Write-Host "Would start local Raven A/V demux: scripts\start-raven-av-demux.ps1" Write-Host "Would launch: dotnet run --project $repo\src\Mimir.App\Mimir.App.csproj" - Write-Host "Raven screen sender: scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200" - Write-Host "Raven Realtek sender: scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202" + Write-Host "Raven muxed A/V sender: scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200" exit 0 } @@ -82,6 +82,8 @@ if ($LASTEXITCODE -ne 0) { throw "Could not start Nightwing typed witness publisher." } +& (Join-Path $repo "scripts\start-raven-av-demux.ps1") + $env:MIMIR_RUNTIME_CONFIG = $ConfigPath $args = @("run", "--project", (Join-Path $repo "src\Mimir.App\Mimir.App.csproj")) if ($Headless) { @@ -99,6 +101,5 @@ if ($Headless) { Write-Host "MIMIR_RUNTIME_CONFIG=$ConfigPath" Write-Host "Starting stream proof: Leap + Kiyo Pro + Raven screen/audio, Nightwing Eye/Move witness claims" -Write-Host "Raven senders expected: scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200" -Write-Host "Raven audio expected: scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202" +Write-Host "Raven sender expected: scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200" dotnet @args diff --git a/state/evidence.jsonl b/state/evidence.jsonl index aebe443..21fccf9 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -301,3 +301,4 @@ {"ts": "2026-06-01T00:12:00+01:00", "type": "decision", "status": "ok", "note": "Corrected the MVP stream split after user pushback: Starfire should not ingest Nightwing PS3 Eye pixels. The MVP profile now keeps Starfire buffers to local Scarlett ASIO, local Leap stereo IR, and local Kiyo Pro RGB; Nightwing owns PS3 Eye reads and emits compact mimir.move_controller_observation_state.v1 typed witness claims through the /eve/periwinkle CultMesh observation receiver. The launcher refuses to start without LeapUVC enumerated, because Leap is the MVP geometry root."} {"ts": "2026-06-01T00:07:07+01:00", "type": "decision", "status": "ok", "note": "Stream-proof demo authority is Starfire local Leap/Kiyo/Scarlett plus Raven screen/Realtek loopback as network sources; Nightwing owns raw Eye/Move reads and publishes only compact typed Move observations. Added ffmpeg PCM audio source so Raven Realtek loopback enters Mimir as AudioBlock samples, plus a provisional Kiyo AR trail/frustum receipt generator from articulated sweep observations."} {"ts": "2026-06-01T00:15:13+01:00", "type": "decision", "status": "ok", "note": "Raven display sync now has an explicit correction path: audio sync states derived from raven-realtk-loopback carry that source's raven-sync clock domain, so MimirSynchronizedBufferPlanner applies the earned audio offset to raven-display and every other buffer in the same clock domain. This keeps separate Raven SRT legs honest until a single muxed A/V transport is worth cutting."} +{"ts": "2026-06-01T00:19:40+01:00", "type": "decision", "status": "ok", "note": "Raven stream proof now uses one muxed A/V LAN transport: start-raven-av-sender.ps1 captures desktop plus Realtek/default render loopback, encodes video with h264_nvenc and audio as AAC, and sends one MPEG-TS SRT stream on port 5200. Starfire start-raven-av-demux.ps1 listens once, then locally splits copied H.264 video to 5210 and decoded f32 PCM audio to 5212 only because Mimir's current ingest adapters are raw-video and PCM-source shaped."} diff --git a/state/map.yaml b/state/map.yaml index 0679ba0..b69c1b1 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -222,13 +222,16 @@ invariants: The stream-proof profile is the live demo cut: `config/mimir-runtime.stream-proof.local.json` keeps Starfire's canonical window on local Scarlett ASIO, local Leap stereo IR, local Kiyo Pro RGB, - Raven display over SRT port 5200, and Raven Realtek loopback as decoded - f32 PCM audio over SRT port 5202. `MimirFfmpegPcmAudioStreamSource` owns - that network PCM edge and emits real `MimirAudioBlockDescriptor` samples - instead of generic process bytes. `scripts/start-stream-proof.ps1` stages - Nightwing's compact Eye/Move publisher and keeps only one `--track-eyes` - worker alive; `scripts/raven-audio.ps1` starts the Raven Realtek loopback - sender from the logged-in Raven desktop session. `tools/solve_stream_proof_frustum.py` + and Raven display/audio from one muxed Raven NVENC/AAC MPEG-TS SRT stream + on port 5200. `scripts/start-raven-av-sender.ps1` captures Raven's desktop + and Realtek/default render loopback together; `scripts/start-raven-av-demux.ps1` + is the Starfire local demux edge, copying video to local SRT port 5210 and + decoding audio to f32 PCM on local SRT port 5212 for Mimir's current raw + ingest adapters. `MimirFfmpegPcmAudioStreamSource` owns the post-demux PCM + edge and emits real `MimirAudioBlockDescriptor` samples instead of generic + process bytes. `scripts/start-stream-proof.ps1` stages Nightwing's compact + Eye/Move publisher and keeps only one `--track-eyes` worker alive. + `tools/solve_stream_proof_frustum.py` turns articulated sweep observations into a provisional Kiyo AR trail receipt: Nightwing matched views define a temporary local witness space, Kiyo observations are fit into it, and the output @@ -238,9 +241,8 @@ invariants: `raven-realtk-loopback` now carry the `raven-sync` clock domain from the source descriptor, and `MimirSynchronizedBufferPlanner` applies that earned correction to every buffer in the same domain, including `raven-display`. - Separate Raven SRT audio/video legs therefore have a measured correction - path; a single muxed Raven A/V transport remains a future tightening, not a - hidden assumption. + The LAN Raven path is now one muxed A/V transport; the local split is only + a Starfire demux boundary for Mimir's raw video/PCM source adapters. `MimirMoveControllerTracker` is the first Mimir-native overlay owner for multiple PS Move controllers: it tracks bright sphere observations from PS3 Eye luma frames, keeps a bounded history per `controllerId`, preserves