Mux Raven display and loopback transport

This commit is contained in:
Meta Vens 2026-06-01 00:22:27 +01:00
commit 2028f0f805
8 changed files with 199 additions and 34 deletions

View file

@ -85,7 +85,7 @@
"-flags",
"low_delay",
"-i",
"srt://0.0.0.0:5200?mode=listener&latency=120000&timeout=5000000",
"srt://0.0.0.0:5210?mode=listener&latency=20000&timeout=5000000",
"-an",
"-f",
"rawvideo",
@ -121,7 +121,7 @@
"-ac",
"2",
"-i",
"srt://0.0.0.0:5202?mode=listener&latency=120000&timeout=5000000",
"srt://0.0.0.0:5212?mode=listener&latency=20000&timeout=5000000",
"-f",
"f32le",
"-"

View file

@ -488,12 +488,15 @@ a named invariant that the native runtime cannot protect yet.
become one valid FieldEvidence claim/candidate without claiming 6DoF pose.
`config/mimir-runtime.stream-proof.local.json` names the current stream-proof
profile: Kiyo Pro AR program view, Leap stereo IR depth root, Raven screen
capture on SRT port 5200, Raven Realtek loopback as decoded f32 PCM on SRT
port 5202, and Scarlett ASIO channels labeled as shotgun/cardioid/local
and Raven Realtek loopback from one muxed Raven NVENC/AAC MPEG-TS SRT stream
on port 5200. Starfire demuxes that single Raven transport locally into
`raven-display` on port 5210 and `raven-realtk-loopback` f32 PCM on port
5212 because Mimir's current source adapters ingest raw video and PCM
separately. Scarlett ASIO channels remain labeled as shotgun/cardioid/local
Raven loopback evidence. Nightwing PS3 Eyes no longer appear as Starfire raw
video sources in this proof; they publish compact Eye/Move observations.
`MimirFfmpegPcmAudioStreamSource` is the network PCM audio owner for Raven
Realtek loopback. `config/mimir-runtime.move-stream.local.json` remains a
`MimirFfmpegPcmAudioStreamSource` is the network PCM audio owner after local
Raven demux. `config/mimir-runtime.move-stream.local.json` remains a
local/raw-Eye show reference, not the current split-host authority.
The distributed authority split is now explicit: Nightwing owns the local
PS3 Eye USB reads, PS Move Bluetooth/HID link, LED schedules, device

View file

@ -98,10 +98,11 @@ The profile declares:
- `leap-stereo-ir`: LeapUVC stereo IR / depth root.
- `kiyo-pro-rgb`: Kiyo Pro AR program view.
- `raven-display`: Raven screen capture over SRT port `5200`, in the
`raven-sync` clock domain.
- `raven-realtk-loopback`: Raven Realtek render loopback as f32 PCM over SRT
port `5202`, also in `raven-sync`.
- `raven-display`: Raven screen capture from the muxed Raven A/V stream,
locally demuxed to SRT port `5210`, in the `raven-sync` clock domain.
- `raven-realtk-loopback`: Raven Realtek render loopback from the same muxed
Raven A/V stream, locally demuxed as f32 PCM to SRT port `5212`, also in
`raven-sync`.
- `focusrite-asio`: Starfire hero mics plus local Raven/program loopback lanes
on `asio-ch0..asio-ch3`.
@ -158,30 +159,37 @@ dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --secon
## Raven Sender
On Raven, start screen capture from the logged-in desktop session:
On Raven, start the muxed A/V sender from the logged-in desktop session:
```powershell
E:\Projects\Mimir\scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200
E:\Projects\Mimir\scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200
```
Start Realtek render loopback from the same logged-in desktop session:
That sender captures the desktop through `gdigrab`, encodes video with
`h264_nvenc`, captures Realtek/default render loopback through the repo WASAPI
loopback script, encodes audio as AAC, and muxes both into one MPEG-TS SRT
stream. On Starfire, `scripts\start-stream-proof.ps1` starts
`scripts\start-raven-av-demux.ps1`, which listens on port `5200` and splits the
single transport into local Mimir ingest legs:
```powershell
E:\Projects\Mimir\scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202
raven-display local SRT 5210, copied H.264/MPEG-TS for decode
raven-realtk-loopback local SRT 5212, decoded f32 PCM
```
If Raven program audio is also routed into Starfire Scarlett on
`asio-ch2/asio-ch3`, that remains the strongest local timing witness. The
Realtek loopback is now a real network PCM audio source in Mimir, not generic
stdout bytes.
`asio-ch2/asio-ch3`, that remains another strong local timing witness. The
Realtek loopback is now part of the same Raven transport as the display before
Starfire demuxes it for Mimir's raw ingest APIs.
Raven display sync is not allowed to ride on SRT arrival time. Both
`raven-display` and `raven-realtk-loopback` are in the `raven-sync` clock
domain. When audio sync estimates the Realtek loopback offset against the
Scarlett reference, `MimirSynchronizedBufferPlanner` applies that correction to
the whole `raven-sync` domain, including the display. A later single muxed
Raven A/V transport can tighten this further, but the current separate-leg
profile has an explicit audio-derived correction path.
the whole `raven-sync` domain, including the display. The LAN transport is now
one muxed A/V stream; the local split exists only at the Starfire demux edge
because Mimir currently ingests raw video and PCM through separate source
adapters.
## Kiyo AR Trail Receipt

View file

@ -0,0 +1,69 @@
param(
[int]$InputPort = 5200,
[int]$VideoPort = 5210,
[int]$AudioPort = 5212,
[int]$AudioSampleRate = 48000,
[int]$AudioChannels = 2,
[string]$FfmpegPath = "ffmpeg",
[string]$LogRoot = "E:\Projects\Mimir\artifacts\runtime\stream-proof",
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
New-Item -ItemType Directory -Force -Path $LogRoot | Out-Null
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$stdoutLog = Join-Path $LogRoot "raven-av-demux-$timestamp.out.log"
$stderrLog = Join-Path $LogRoot "raven-av-demux-$timestamp.err.log"
$inputEndpoint = "srt://0.0.0.0:${InputPort}?mode=listener&latency=120000&timeout=5000000"
$videoEndpoint = "srt://127.0.0.1:${VideoPort}?mode=caller&latency=20000"
$audioEndpoint = "srt://127.0.0.1:${AudioPort}?mode=caller&latency=20000"
$arguments = @(
"-hide_banner",
"-nostdin",
"-loglevel", "warning",
"-fflags", "nobuffer",
"-flags", "low_delay",
"-i", $inputEndpoint,
"-map", "0:v:0",
"-an",
"-c:v", "copy",
"-f", "mpegts",
$videoEndpoint,
"-map", "0:a:0",
"-vn",
"-acodec", "pcm_f32le",
"-ar", $AudioSampleRate.ToString(),
"-ac", $AudioChannels.ToString(),
"-f", "f32le",
$audioEndpoint
)
Write-Host "Raven muxed A/V demux:"
Write-Host " input: $inputEndpoint"
Write-Host " video: $videoEndpoint"
Write-Host " audio: $audioEndpoint"
Write-Host " stdout: $stdoutLog"
Write-Host " stderr: $stderrLog"
if ($DryRun) {
Write-Host ($FfmpegPath + " " + ($arguments -join " "))
exit 0
}
Get-CimInstance Win32_Process |
Where-Object { $_.ProcessName -like "ffmpeg*" -and $_.CommandLine -match [regex]::Escape($inputEndpoint) } |
ForEach-Object {
try { Stop-Process -Id $_.ProcessId -Force -ErrorAction Stop } catch {}
}
$process = Start-Process `
-FilePath $FfmpegPath `
-ArgumentList $arguments `
-WindowStyle Hidden `
-RedirectStandardOutput $stdoutLog `
-RedirectStandardError $stderrLog `
-PassThru
Write-Host "Started Raven muxed A/V demux pid=$($process.Id)"

View file

@ -0,0 +1,81 @@
param(
[string]$TargetHost = "192.168.1.66",
[int]$Port = 5200,
[int]$Width = 1920,
[int]$Height = 1080,
[int]$Framerate = 30,
[int]$AudioSampleRate = 48000,
[int]$AudioChannels = 2,
[string]$FfmpegPath = "ffmpeg",
[string]$Source = "desktop",
[string]$VideoBitrate = "12000k",
[string]$AudioBitrate = "192k",
[string]$LogRoot = "C:\Meta\Mimir\logs",
[switch]$DryRun
)
$ErrorActionPreference = "Stop"
New-Item -ItemType Directory -Force -Path $LogRoot | Out-Null
$repo = Split-Path -Parent $PSScriptRoot
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$stdoutLog = Join-Path $LogRoot "raven-av-srt-$timestamp.out.log"
$stderrLog = Join-Path $LogRoot "raven-av-srt-$timestamp.err.log"
$endpoint = "srt://${TargetHost}:${Port}?mode=caller&latency=120000"
$videoSize = "${Width}x${Height}"
$gop = [Math]::Max(1, $Framerate * 2)
$captureScript = Join-Path $repo "scripts\wasapi-loopback-capture.ps1"
$ffmpegArgs = @(
"-hide_banner",
"-loglevel", "warning",
"-thread_queue_size", "1024",
"-f", "gdigrab",
"-framerate", $Framerate.ToString(),
"-video_size", $videoSize,
"-i", $Source,
"-thread_queue_size", "1024",
"-f", "f32le",
"-ar", $AudioSampleRate.ToString(),
"-ac", $AudioChannels.ToString(),
"-i", "pipe:0",
"-map", "0:v:0",
"-map", "1:a:0",
"-c:v", "h264_nvenc",
"-preset", "p4",
"-tune", "ll",
"-b:v", $VideoBitrate,
"-maxrate", $VideoBitrate,
"-bufsize", "24000k",
"-pix_fmt", "yuv420p",
"-g", $gop.ToString(),
"-c:a", "aac",
"-b:a", $AudioBitrate,
"-ar", $AudioSampleRate.ToString(),
"-ac", $AudioChannels.ToString(),
"-f", "mpegts",
$endpoint
)
$command = "& `"$captureScript`" -Output - -SampleRate $AudioSampleRate -Channels $AudioChannels | & `"$FfmpegPath`" $($ffmpegArgs -join ' ')"
Write-Host "Raven muxed A/V sender:"
Write-Host " ffmpeg: $FfmpegPath"
Write-Host " target: $endpoint"
Write-Host " video: $Source $videoSize@$Framerate via h264_nvenc"
Write-Host " audio: Realtek/default render loopback ${AudioChannels}ch ${AudioSampleRate}Hz -> AAC"
Write-Host " stdout: $stdoutLog"
Write-Host " stderr: $stderrLog"
if ($DryRun) {
Write-Host $command
exit 0
}
Start-Process `
-FilePath "powershell" `
-ArgumentList @("-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", $command) `
-WindowStyle Hidden `
-RedirectStandardOutput $stdoutLog `
-RedirectStandardError $stderrLog `
-PassThru |
ForEach-Object { Write-Host "Started Raven muxed A/V sender pid=$($_.Id)" }

View file

@ -17,9 +17,9 @@ New-Item -ItemType Directory -Force -Path $logDir | Out-Null
if ($DryRun) {
Write-Host "MIMIR_RUNTIME_CONFIG=$ConfigPath"
Write-Host "Would stage Nightwing witness tools to $NightwingHost and start --track-eyes publisher at $WitnessUrl"
Write-Host "Would start local Raven A/V demux: scripts\start-raven-av-demux.ps1"
Write-Host "Would launch: dotnet run --project $repo\src\Mimir.App\Mimir.App.csproj"
Write-Host "Raven screen sender: scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
Write-Host "Raven Realtek sender: scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202"
Write-Host "Raven muxed A/V sender: scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
exit 0
}
@ -82,6 +82,8 @@ if ($LASTEXITCODE -ne 0) {
throw "Could not start Nightwing typed witness publisher."
}
& (Join-Path $repo "scripts\start-raven-av-demux.ps1")
$env:MIMIR_RUNTIME_CONFIG = $ConfigPath
$args = @("run", "--project", (Join-Path $repo "src\Mimir.App\Mimir.App.csproj"))
if ($Headless) {
@ -99,6 +101,5 @@ if ($Headless) {
Write-Host "MIMIR_RUNTIME_CONFIG=$ConfigPath"
Write-Host "Starting stream proof: Leap + Kiyo Pro + Raven screen/audio, Nightwing Eye/Move witness claims"
Write-Host "Raven senders expected: scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
Write-Host "Raven audio expected: scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202"
Write-Host "Raven sender expected: scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
dotnet @args

View file

@ -301,3 +301,4 @@
{"ts": "2026-06-01T00:12:00+01:00", "type": "decision", "status": "ok", "note": "Corrected the MVP stream split after user pushback: Starfire should not ingest Nightwing PS3 Eye pixels. The MVP profile now keeps Starfire buffers to local Scarlett ASIO, local Leap stereo IR, and local Kiyo Pro RGB; Nightwing owns PS3 Eye reads and emits compact mimir.move_controller_observation_state.v1 typed witness claims through the /eve/periwinkle CultMesh observation receiver. The launcher refuses to start without LeapUVC enumerated, because Leap is the MVP geometry root."}
{"ts": "2026-06-01T00:07:07+01:00", "type": "decision", "status": "ok", "note": "Stream-proof demo authority is Starfire local Leap/Kiyo/Scarlett plus Raven screen/Realtek loopback as network sources; Nightwing owns raw Eye/Move reads and publishes only compact typed Move observations. Added ffmpeg PCM audio source so Raven Realtek loopback enters Mimir as AudioBlock samples, plus a provisional Kiyo AR trail/frustum receipt generator from articulated sweep observations."}
{"ts": "2026-06-01T00:15:13+01:00", "type": "decision", "status": "ok", "note": "Raven display sync now has an explicit correction path: audio sync states derived from raven-realtk-loopback carry that source's raven-sync clock domain, so MimirSynchronizedBufferPlanner applies the earned audio offset to raven-display and every other buffer in the same clock domain. This keeps separate Raven SRT legs honest until a single muxed A/V transport is worth cutting."}
{"ts": "2026-06-01T00:19:40+01:00", "type": "decision", "status": "ok", "note": "Raven stream proof now uses one muxed A/V LAN transport: start-raven-av-sender.ps1 captures desktop plus Realtek/default render loopback, encodes video with h264_nvenc and audio as AAC, and sends one MPEG-TS SRT stream on port 5200. Starfire start-raven-av-demux.ps1 listens once, then locally splits copied H.264 video to 5210 and decoded f32 PCM audio to 5212 only because Mimir's current ingest adapters are raw-video and PCM-source shaped."}

View file

@ -222,13 +222,16 @@ invariants:
The stream-proof profile is the live demo cut:
`config/mimir-runtime.stream-proof.local.json` keeps Starfire's canonical
window on local Scarlett ASIO, local Leap stereo IR, local Kiyo Pro RGB,
Raven display over SRT port 5200, and Raven Realtek loopback as decoded
f32 PCM audio over SRT port 5202. `MimirFfmpegPcmAudioStreamSource` owns
that network PCM edge and emits real `MimirAudioBlockDescriptor` samples
instead of generic process bytes. `scripts/start-stream-proof.ps1` stages
Nightwing's compact Eye/Move publisher and keeps only one `--track-eyes`
worker alive; `scripts/raven-audio.ps1` starts the Raven Realtek loopback
sender from the logged-in Raven desktop session. `tools/solve_stream_proof_frustum.py`
and Raven display/audio from one muxed Raven NVENC/AAC MPEG-TS SRT stream
on port 5200. `scripts/start-raven-av-sender.ps1` captures Raven's desktop
and Realtek/default render loopback together; `scripts/start-raven-av-demux.ps1`
is the Starfire local demux edge, copying video to local SRT port 5210 and
decoding audio to f32 PCM on local SRT port 5212 for Mimir's current raw
ingest adapters. `MimirFfmpegPcmAudioStreamSource` owns the post-demux PCM
edge and emits real `MimirAudioBlockDescriptor` samples instead of generic
process bytes. `scripts/start-stream-proof.ps1` stages Nightwing's compact
Eye/Move publisher and keeps only one `--track-eyes` worker alive.
`tools/solve_stream_proof_frustum.py`
turns articulated sweep observations into a provisional Kiyo AR trail
receipt: Nightwing matched views define a temporary local witness space,
Kiyo observations are fit into it, and the output
@ -238,9 +241,8 @@ invariants:
`raven-realtk-loopback` now carry the `raven-sync` clock domain from the
source descriptor, and `MimirSynchronizedBufferPlanner` applies that earned
correction to every buffer in the same domain, including `raven-display`.
Separate Raven SRT audio/video legs therefore have a measured correction
path; a single muxed Raven A/V transport remains a future tightening, not a
hidden assumption.
The LAN Raven path is now one muxed A/V transport; the local split is only
a Starfire demux boundary for Mimir's raw video/PCM source adapters.
`MimirMoveControllerTracker` is the first Mimir-native overlay owner for
multiple PS Move controllers: it tracks bright sphere observations from
PS3 Eye luma frames, keeps a bounded history per `controllerId`, preserves