Mux Raven display and loopback transport
This commit is contained in:
parent
fe101a579a
commit
2028f0f805
8 changed files with 199 additions and 34 deletions
|
|
@ -85,7 +85,7 @@
|
|||
"-flags",
|
||||
"low_delay",
|
||||
"-i",
|
||||
"srt://0.0.0.0:5200?mode=listener&latency=120000&timeout=5000000",
|
||||
"srt://0.0.0.0:5210?mode=listener&latency=20000&timeout=5000000",
|
||||
"-an",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
|
|
@ -121,7 +121,7 @@
|
|||
"-ac",
|
||||
"2",
|
||||
"-i",
|
||||
"srt://0.0.0.0:5202?mode=listener&latency=120000&timeout=5000000",
|
||||
"srt://0.0.0.0:5212?mode=listener&latency=20000&timeout=5000000",
|
||||
"-f",
|
||||
"f32le",
|
||||
"-"
|
||||
|
|
|
|||
|
|
@ -488,12 +488,15 @@ a named invariant that the native runtime cannot protect yet.
|
|||
become one valid FieldEvidence claim/candidate without claiming 6DoF pose.
|
||||
`config/mimir-runtime.stream-proof.local.json` names the current stream-proof
|
||||
profile: Kiyo Pro AR program view, Leap stereo IR depth root, Raven screen
|
||||
capture on SRT port 5200, Raven Realtek loopback as decoded f32 PCM on SRT
|
||||
port 5202, and Scarlett ASIO channels labeled as shotgun/cardioid/local
|
||||
and Raven Realtek loopback from one muxed Raven NVENC/AAC MPEG-TS SRT stream
|
||||
on port 5200. Starfire demuxes that single Raven transport locally into
|
||||
`raven-display` on port 5210 and `raven-realtk-loopback` f32 PCM on port
|
||||
5212 because Mimir's current source adapters ingest raw video and PCM
|
||||
separately. Scarlett ASIO channels remain labeled as shotgun/cardioid/local
|
||||
Raven loopback evidence. Nightwing PS3 Eyes no longer appear as Starfire raw
|
||||
video sources in this proof; they publish compact Eye/Move observations.
|
||||
`MimirFfmpegPcmAudioStreamSource` is the network PCM audio owner for Raven
|
||||
Realtek loopback. `config/mimir-runtime.move-stream.local.json` remains a
|
||||
`MimirFfmpegPcmAudioStreamSource` is the network PCM audio owner after local
|
||||
Raven demux. `config/mimir-runtime.move-stream.local.json` remains a
|
||||
local/raw-Eye show reference, not the current split-host authority.
|
||||
The distributed authority split is now explicit: Nightwing owns the local
|
||||
PS3 Eye USB reads, PS Move Bluetooth/HID link, LED schedules, device
|
||||
|
|
|
|||
|
|
@ -98,10 +98,11 @@ The profile declares:
|
|||
|
||||
- `leap-stereo-ir`: LeapUVC stereo IR / depth root.
|
||||
- `kiyo-pro-rgb`: Kiyo Pro AR program view.
|
||||
- `raven-display`: Raven screen capture over SRT port `5200`, in the
|
||||
`raven-sync` clock domain.
|
||||
- `raven-realtk-loopback`: Raven Realtek render loopback as f32 PCM over SRT
|
||||
port `5202`, also in `raven-sync`.
|
||||
- `raven-display`: Raven screen capture from the muxed Raven A/V stream,
|
||||
locally demuxed to SRT port `5210`, in the `raven-sync` clock domain.
|
||||
- `raven-realtk-loopback`: Raven Realtek render loopback from the same muxed
|
||||
Raven A/V stream, locally demuxed as f32 PCM to SRT port `5212`, also in
|
||||
`raven-sync`.
|
||||
- `focusrite-asio`: Starfire hero mics plus local Raven/program loopback lanes
|
||||
on `asio-ch0..asio-ch3`.
|
||||
|
||||
|
|
@ -158,30 +159,37 @@ dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --secon
|
|||
|
||||
## Raven Sender
|
||||
|
||||
On Raven, start screen capture from the logged-in desktop session:
|
||||
On Raven, start the muxed A/V sender from the logged-in desktop session:
|
||||
|
||||
```powershell
|
||||
E:\Projects\Mimir\scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200
|
||||
E:\Projects\Mimir\scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200
|
||||
```
|
||||
|
||||
Start Realtek render loopback from the same logged-in desktop session:
|
||||
That sender captures the desktop through `gdigrab`, encodes video with
|
||||
`h264_nvenc`, captures Realtek/default render loopback through the repo WASAPI
|
||||
loopback script, encodes audio as AAC, and muxes both into one MPEG-TS SRT
|
||||
stream. On Starfire, `scripts\start-stream-proof.ps1` starts
|
||||
`scripts\start-raven-av-demux.ps1`, which listens on port `5200` and splits the
|
||||
single transport into local Mimir ingest legs:
|
||||
|
||||
```powershell
|
||||
E:\Projects\Mimir\scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202
|
||||
raven-display local SRT 5210, copied H.264/MPEG-TS for decode
|
||||
raven-realtk-loopback local SRT 5212, decoded f32 PCM
|
||||
```
|
||||
|
||||
If Raven program audio is also routed into Starfire Scarlett on
|
||||
`asio-ch2/asio-ch3`, that remains the strongest local timing witness. The
|
||||
Realtek loopback is now a real network PCM audio source in Mimir, not generic
|
||||
stdout bytes.
|
||||
`asio-ch2/asio-ch3`, that remains another strong local timing witness. The
|
||||
Realtek loopback is now part of the same Raven transport as the display before
|
||||
Starfire demuxes it for Mimir's raw ingest APIs.
|
||||
|
||||
Raven display sync is not allowed to ride on SRT arrival time. Both
|
||||
`raven-display` and `raven-realtk-loopback` are in the `raven-sync` clock
|
||||
domain. When audio sync estimates the Realtek loopback offset against the
|
||||
Scarlett reference, `MimirSynchronizedBufferPlanner` applies that correction to
|
||||
the whole `raven-sync` domain, including the display. A later single muxed
|
||||
Raven A/V transport can tighten this further, but the current separate-leg
|
||||
profile has an explicit audio-derived correction path.
|
||||
the whole `raven-sync` domain, including the display. The LAN transport is now
|
||||
one muxed A/V stream; the local split exists only at the Starfire demux edge
|
||||
because Mimir currently ingests raw video and PCM through separate source
|
||||
adapters.
|
||||
|
||||
## Kiyo AR Trail Receipt
|
||||
|
||||
|
|
|
|||
69
scripts/start-raven-av-demux.ps1
Normal file
69
scripts/start-raven-av-demux.ps1
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
param(
|
||||
[int]$InputPort = 5200,
|
||||
[int]$VideoPort = 5210,
|
||||
[int]$AudioPort = 5212,
|
||||
[int]$AudioSampleRate = 48000,
|
||||
[int]$AudioChannels = 2,
|
||||
[string]$FfmpegPath = "ffmpeg",
|
||||
[string]$LogRoot = "E:\Projects\Mimir\artifacts\runtime\stream-proof",
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $LogRoot | Out-Null
|
||||
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
$stdoutLog = Join-Path $LogRoot "raven-av-demux-$timestamp.out.log"
|
||||
$stderrLog = Join-Path $LogRoot "raven-av-demux-$timestamp.err.log"
|
||||
$inputEndpoint = "srt://0.0.0.0:${InputPort}?mode=listener&latency=120000&timeout=5000000"
|
||||
$videoEndpoint = "srt://127.0.0.1:${VideoPort}?mode=caller&latency=20000"
|
||||
$audioEndpoint = "srt://127.0.0.1:${AudioPort}?mode=caller&latency=20000"
|
||||
|
||||
$arguments = @(
|
||||
"-hide_banner",
|
||||
"-nostdin",
|
||||
"-loglevel", "warning",
|
||||
"-fflags", "nobuffer",
|
||||
"-flags", "low_delay",
|
||||
"-i", $inputEndpoint,
|
||||
"-map", "0:v:0",
|
||||
"-an",
|
||||
"-c:v", "copy",
|
||||
"-f", "mpegts",
|
||||
$videoEndpoint,
|
||||
"-map", "0:a:0",
|
||||
"-vn",
|
||||
"-acodec", "pcm_f32le",
|
||||
"-ar", $AudioSampleRate.ToString(),
|
||||
"-ac", $AudioChannels.ToString(),
|
||||
"-f", "f32le",
|
||||
$audioEndpoint
|
||||
)
|
||||
|
||||
Write-Host "Raven muxed A/V demux:"
|
||||
Write-Host " input: $inputEndpoint"
|
||||
Write-Host " video: $videoEndpoint"
|
||||
Write-Host " audio: $audioEndpoint"
|
||||
Write-Host " stdout: $stdoutLog"
|
||||
Write-Host " stderr: $stderrLog"
|
||||
|
||||
if ($DryRun) {
|
||||
Write-Host ($FfmpegPath + " " + ($arguments -join " "))
|
||||
exit 0
|
||||
}
|
||||
|
||||
Get-CimInstance Win32_Process |
|
||||
Where-Object { $_.ProcessName -like "ffmpeg*" -and $_.CommandLine -match [regex]::Escape($inputEndpoint) } |
|
||||
ForEach-Object {
|
||||
try { Stop-Process -Id $_.ProcessId -Force -ErrorAction Stop } catch {}
|
||||
}
|
||||
|
||||
$process = Start-Process `
|
||||
-FilePath $FfmpegPath `
|
||||
-ArgumentList $arguments `
|
||||
-WindowStyle Hidden `
|
||||
-RedirectStandardOutput $stdoutLog `
|
||||
-RedirectStandardError $stderrLog `
|
||||
-PassThru
|
||||
|
||||
Write-Host "Started Raven muxed A/V demux pid=$($process.Id)"
|
||||
81
scripts/start-raven-av-sender.ps1
Normal file
81
scripts/start-raven-av-sender.ps1
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
param(
|
||||
[string]$TargetHost = "192.168.1.66",
|
||||
[int]$Port = 5200,
|
||||
[int]$Width = 1920,
|
||||
[int]$Height = 1080,
|
||||
[int]$Framerate = 30,
|
||||
[int]$AudioSampleRate = 48000,
|
||||
[int]$AudioChannels = 2,
|
||||
[string]$FfmpegPath = "ffmpeg",
|
||||
[string]$Source = "desktop",
|
||||
[string]$VideoBitrate = "12000k",
|
||||
[string]$AudioBitrate = "192k",
|
||||
[string]$LogRoot = "C:\Meta\Mimir\logs",
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $LogRoot | Out-Null
|
||||
$repo = Split-Path -Parent $PSScriptRoot
|
||||
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
$stdoutLog = Join-Path $LogRoot "raven-av-srt-$timestamp.out.log"
|
||||
$stderrLog = Join-Path $LogRoot "raven-av-srt-$timestamp.err.log"
|
||||
$endpoint = "srt://${TargetHost}:${Port}?mode=caller&latency=120000"
|
||||
$videoSize = "${Width}x${Height}"
|
||||
$gop = [Math]::Max(1, $Framerate * 2)
|
||||
$captureScript = Join-Path $repo "scripts\wasapi-loopback-capture.ps1"
|
||||
|
||||
$ffmpegArgs = @(
|
||||
"-hide_banner",
|
||||
"-loglevel", "warning",
|
||||
"-thread_queue_size", "1024",
|
||||
"-f", "gdigrab",
|
||||
"-framerate", $Framerate.ToString(),
|
||||
"-video_size", $videoSize,
|
||||
"-i", $Source,
|
||||
"-thread_queue_size", "1024",
|
||||
"-f", "f32le",
|
||||
"-ar", $AudioSampleRate.ToString(),
|
||||
"-ac", $AudioChannels.ToString(),
|
||||
"-i", "pipe:0",
|
||||
"-map", "0:v:0",
|
||||
"-map", "1:a:0",
|
||||
"-c:v", "h264_nvenc",
|
||||
"-preset", "p4",
|
||||
"-tune", "ll",
|
||||
"-b:v", $VideoBitrate,
|
||||
"-maxrate", $VideoBitrate,
|
||||
"-bufsize", "24000k",
|
||||
"-pix_fmt", "yuv420p",
|
||||
"-g", $gop.ToString(),
|
||||
"-c:a", "aac",
|
||||
"-b:a", $AudioBitrate,
|
||||
"-ar", $AudioSampleRate.ToString(),
|
||||
"-ac", $AudioChannels.ToString(),
|
||||
"-f", "mpegts",
|
||||
$endpoint
|
||||
)
|
||||
$command = "& `"$captureScript`" -Output - -SampleRate $AudioSampleRate -Channels $AudioChannels | & `"$FfmpegPath`" $($ffmpegArgs -join ' ')"
|
||||
|
||||
Write-Host "Raven muxed A/V sender:"
|
||||
Write-Host " ffmpeg: $FfmpegPath"
|
||||
Write-Host " target: $endpoint"
|
||||
Write-Host " video: $Source $videoSize@$Framerate via h264_nvenc"
|
||||
Write-Host " audio: Realtek/default render loopback ${AudioChannels}ch ${AudioSampleRate}Hz -> AAC"
|
||||
Write-Host " stdout: $stdoutLog"
|
||||
Write-Host " stderr: $stderrLog"
|
||||
|
||||
if ($DryRun) {
|
||||
Write-Host $command
|
||||
exit 0
|
||||
}
|
||||
|
||||
Start-Process `
|
||||
-FilePath "powershell" `
|
||||
-ArgumentList @("-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", $command) `
|
||||
-WindowStyle Hidden `
|
||||
-RedirectStandardOutput $stdoutLog `
|
||||
-RedirectStandardError $stderrLog `
|
||||
-PassThru |
|
||||
ForEach-Object { Write-Host "Started Raven muxed A/V sender pid=$($_.Id)" }
|
||||
|
|
@ -17,9 +17,9 @@ New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
|||
if ($DryRun) {
|
||||
Write-Host "MIMIR_RUNTIME_CONFIG=$ConfigPath"
|
||||
Write-Host "Would stage Nightwing witness tools to $NightwingHost and start --track-eyes publisher at $WitnessUrl"
|
||||
Write-Host "Would start local Raven A/V demux: scripts\start-raven-av-demux.ps1"
|
||||
Write-Host "Would launch: dotnet run --project $repo\src\Mimir.App\Mimir.App.csproj"
|
||||
Write-Host "Raven screen sender: scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
|
||||
Write-Host "Raven Realtek sender: scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202"
|
||||
Write-Host "Raven muxed A/V sender: scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
|
||||
exit 0
|
||||
}
|
||||
|
||||
|
|
@ -82,6 +82,8 @@ if ($LASTEXITCODE -ne 0) {
|
|||
throw "Could not start Nightwing typed witness publisher."
|
||||
}
|
||||
|
||||
& (Join-Path $repo "scripts\start-raven-av-demux.ps1")
|
||||
|
||||
$env:MIMIR_RUNTIME_CONFIG = $ConfigPath
|
||||
$args = @("run", "--project", (Join-Path $repo "src\Mimir.App\Mimir.App.csproj"))
|
||||
if ($Headless) {
|
||||
|
|
@ -99,6 +101,5 @@ if ($Headless) {
|
|||
|
||||
Write-Host "MIMIR_RUNTIME_CONFIG=$ConfigPath"
|
||||
Write-Host "Starting stream proof: Leap + Kiyo Pro + Raven screen/audio, Nightwing Eye/Move witness claims"
|
||||
Write-Host "Raven senders expected: scripts\start-raven-screen-capture-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
|
||||
Write-Host "Raven audio expected: scripts\raven-audio.ps1 -TargetHost 192.168.1.66 -Port 5202"
|
||||
Write-Host "Raven sender expected: scripts\start-raven-av-sender.ps1 -TargetHost 192.168.1.66 -Port 5200"
|
||||
dotnet @args
|
||||
|
|
|
|||
|
|
@ -301,3 +301,4 @@
|
|||
{"ts": "2026-06-01T00:12:00+01:00", "type": "decision", "status": "ok", "note": "Corrected the MVP stream split after user pushback: Starfire should not ingest Nightwing PS3 Eye pixels. The MVP profile now keeps Starfire buffers to local Scarlett ASIO, local Leap stereo IR, and local Kiyo Pro RGB; Nightwing owns PS3 Eye reads and emits compact mimir.move_controller_observation_state.v1 typed witness claims through the /eve/periwinkle CultMesh observation receiver. The launcher refuses to start without LeapUVC enumerated, because Leap is the MVP geometry root."}
|
||||
{"ts": "2026-06-01T00:07:07+01:00", "type": "decision", "status": "ok", "note": "Stream-proof demo authority is Starfire local Leap/Kiyo/Scarlett plus Raven screen/Realtek loopback as network sources; Nightwing owns raw Eye/Move reads and publishes only compact typed Move observations. Added ffmpeg PCM audio source so Raven Realtek loopback enters Mimir as AudioBlock samples, plus a provisional Kiyo AR trail/frustum receipt generator from articulated sweep observations."}
|
||||
{"ts": "2026-06-01T00:15:13+01:00", "type": "decision", "status": "ok", "note": "Raven display sync now has an explicit correction path: audio sync states derived from raven-realtk-loopback carry that source's raven-sync clock domain, so MimirSynchronizedBufferPlanner applies the earned audio offset to raven-display and every other buffer in the same clock domain. This keeps separate Raven SRT legs honest until a single muxed A/V transport is worth cutting."}
|
||||
{"ts": "2026-06-01T00:19:40+01:00", "type": "decision", "status": "ok", "note": "Raven stream proof now uses one muxed A/V LAN transport: start-raven-av-sender.ps1 captures desktop plus Realtek/default render loopback, encodes video with h264_nvenc and audio as AAC, and sends one MPEG-TS SRT stream on port 5200. Starfire start-raven-av-demux.ps1 listens once, then locally splits copied H.264 video to 5210 and decoded f32 PCM audio to 5212 only because Mimir's current ingest adapters are raw-video and PCM-source shaped."}
|
||||
|
|
|
|||
|
|
@ -222,13 +222,16 @@ invariants:
|
|||
The stream-proof profile is the live demo cut:
|
||||
`config/mimir-runtime.stream-proof.local.json` keeps Starfire's canonical
|
||||
window on local Scarlett ASIO, local Leap stereo IR, local Kiyo Pro RGB,
|
||||
Raven display over SRT port 5200, and Raven Realtek loopback as decoded
|
||||
f32 PCM audio over SRT port 5202. `MimirFfmpegPcmAudioStreamSource` owns
|
||||
that network PCM edge and emits real `MimirAudioBlockDescriptor` samples
|
||||
instead of generic process bytes. `scripts/start-stream-proof.ps1` stages
|
||||
Nightwing's compact Eye/Move publisher and keeps only one `--track-eyes`
|
||||
worker alive; `scripts/raven-audio.ps1` starts the Raven Realtek loopback
|
||||
sender from the logged-in Raven desktop session. `tools/solve_stream_proof_frustum.py`
|
||||
and Raven display/audio from one muxed Raven NVENC/AAC MPEG-TS SRT stream
|
||||
on port 5200. `scripts/start-raven-av-sender.ps1` captures Raven's desktop
|
||||
and Realtek/default render loopback together; `scripts/start-raven-av-demux.ps1`
|
||||
is the Starfire local demux edge, copying video to local SRT port 5210 and
|
||||
decoding audio to f32 PCM on local SRT port 5212 for Mimir's current raw
|
||||
ingest adapters. `MimirFfmpegPcmAudioStreamSource` owns the post-demux PCM
|
||||
edge and emits real `MimirAudioBlockDescriptor` samples instead of generic
|
||||
process bytes. `scripts/start-stream-proof.ps1` stages Nightwing's compact
|
||||
Eye/Move publisher and keeps only one `--track-eyes` worker alive.
|
||||
`tools/solve_stream_proof_frustum.py`
|
||||
turns articulated sweep observations into a provisional Kiyo AR trail
|
||||
receipt: Nightwing matched views define a temporary local witness space,
|
||||
Kiyo observations are fit into it, and the output
|
||||
|
|
@ -238,9 +241,8 @@ invariants:
|
|||
`raven-realtk-loopback` now carry the `raven-sync` clock domain from the
|
||||
source descriptor, and `MimirSynchronizedBufferPlanner` applies that earned
|
||||
correction to every buffer in the same domain, including `raven-display`.
|
||||
Separate Raven SRT audio/video legs therefore have a measured correction
|
||||
path; a single muxed Raven A/V transport remains a future tightening, not a
|
||||
hidden assumption.
|
||||
The LAN Raven path is now one muxed A/V transport; the local split is only
|
||||
a Starfire demux boundary for Mimir's raw video/PCM source adapters.
|
||||
`MimirMoveControllerTracker` is the first Mimir-native overlay owner for
|
||||
multiple PS Move controllers: it tracks bright sphere observations from
|
||||
PS3 Eye luma frames, keeps a bounded history per `controllerId`, preserves
|
||||
|
|
|
|||
Loading…
Reference in a new issue