Publish aligned mic field for Faust separation

This commit is contained in:
Meta Vens 2026-05-18 22:09:10 +01:00
commit a4c3042628
12 changed files with 452 additions and 2 deletions

View file

@ -11,11 +11,15 @@ flowchart TD
A["aligned microphone field"] --> B["FOA encoder"]
B --> C["AmbiX ACN/SN3D W,Y,Z,X blocks"]
C --> D["localcast.audio.spatial_frame"]
A --> M["localcast.audio.mic_field"]
M --> N["Faust voice-separation graph"]
A --> H["source-event analysis"]
H --> I["localcast.audio.source_events"]
D --> E["audio-state.msgpack / CultNet document_put"]
M --> O["audio-mic-field.msgpack / CultNet document_put"]
I --> J["audio-events.msgpack / CultNet document_put"]
E --> F["Aquarium client"]
O --> F
J --> F
K["visual-state.msgpack"] --> F
F --> G["native Faust spatial DSP + volumetric renderer"]
@ -24,6 +28,7 @@ flowchart TD
Live files:
- `calibration/runs/audio-state.msgpack`: latest typed AmbiX audio block.
- `calibration/runs/audio-mic-field.msgpack`: latest aligned six-mic block for Faust voice separation.
- `calibration/runs/audio-stream-status.msgpack`: audio publisher heartbeat.
- `calibration/runs/audio-events.msgpack`: latest typed dialogue-focus and transient vector field.
- `calibration/runs/av-sync-status.json`: current Aquarium/Spout publisher view of visual frame id, audio frame id, audio delta, and synchronized event overlay count.
@ -44,12 +49,31 @@ Source event document type:
- payload: MessagePack array decoded as `CultAudioSourceEvents`
- contents: dialogue anchor weights plus witness-dominant transient events with sample time, estimated room position, direction, energy, confidence, and kind.
Mic-field document type:
- `localcast.audio.mic_field`
- schema id: `gamecult.localcast.audio.mic_field.v1`
- key: `localcast.audio.mic-field.live`
- payload: MessagePack array decoded as `CultAudioMicFieldFrame`
- audio format: interleaved `float32`, 48 kHz, aligned mic channels
- default channel order: `host-focusrite`, `co-streamer-focusrite`, `kiyo-0`, `kiyo-1`, `ps-eye-0`, `ps-eye-1`
- graph id: `localcast.faust.voice_separation.v1`
- Faust source: `faust/localcast_voice_separation.dsp`
Publisher smoke command:
```powershell
.\.venv\Scripts\python.exe .\scripts\stream_spatial_audio.py --input .\calibration\runs\audio-full-sync-20260518-165751\field-foa-ambix.wav --loop --duration 5 --smoke-readback
```
Faust mic-field publisher:
```powershell
.\scripts\start-live-faust-mic-field.ps1
```
This publishes the aligned six-channel field directly to Aquarium/Faust while preserving the AmbiX path as a compatibility bed. Stop it with `.\scripts\stop-live-faust-mic-field.ps1`.
Source-event analysis command:
```powershell
@ -65,8 +89,9 @@ Synchronized Aquarium/Spout publisher command:
## Invariants
- Aquarium owns rendering and Faust DSP, not microphone capture or clock correction.
- LocalCastBridge publishes declared AmbiX blocks, not undecoded heterogeneous microphone channels.
- LocalCastBridge publishes declared AmbiX blocks for the spatial bed and aligned six-mic blocks for Faust voice separation. Unaligned heterogeneous microphones still do not cross this boundary.
- LocalCastBridge publishes source-event geometry separately from AmbiX audio; render effects do not have to infer transient positions from the sound bed.
- Faust owns the voice-separation graph once the aligned mic field has crossed into Aquarium. Python may publish controls and telemetry, but it must not become the hot separation engine.
- Aquarium/Spout packaging selects audio source events against each visual frame's `audio_alignment_time_ns`; OBS should receive the already synchronized output rather than trying to align independent media sources.
- The audio frame carries `start_sample` and `audio_time_ns` so visual packets can align against the same bounded-latency timeline.
- OBS may receive a rendered monitor output later, but OBS is not the spatial bus authority.

View file

@ -195,6 +195,14 @@ Encode an offline aligned six-channel WAV to FOA AmbiX:
.\.venv\Scripts\python.exe .\scripts\audio_field.py encode-foa --profile .\config\audio-field.json --input .\calibration\runs\<run-folder>\field-aligned.wav --output .\calibration\runs\<run-folder>\field-foa-ambix.wav
```
Publish the same aligned six-channel field directly to Aquarium/Faust for voice separation:
```powershell
.\scripts\start-live-faust-mic-field.ps1
```
This writes `localcast.audio.mic_field` into `calibration/runs/audio-mic-field.msgpack` with channel roles and graph id `localcast.faust.voice_separation.v1`. The Faust graph lives at `faust/localcast_voice_separation.dsp` and has six inputs plus six stem outputs: host voice, co-streamer voice, ambient, transients, local loopback placeholder, and co-streamer loopback placeholder.
Suppress room/transient witness energy before FOA encoding:
```powershell
@ -237,6 +245,7 @@ The `play-record` and `record-field` commands remain for a `shared-input-device`
- Ultrasonic probes are allowed only as a bounded optimization strategy. They must stay below Nyquist, remain level-capped, and prove themselves by raising measured confidence; hardware may distort or discard them.
- Dense probe textures should be harmonically organized so any lower-band aliases land closer to musical relationships than arbitrary squeal. This is damage control, not a license to trust aliasing.
- FOA output is AmbiX: ACN channel order, SN3D normalization, channels `W,Y,Z,X`.
- The Faust voice-separation input is the aligned six-mic field, not the AmbiX collapse. Voice separation needs anchor and witness channels before spatial rendering destroys useful evidence.
- Speaker calibration is a measurement path, not a generic monitor output.
- Source-event geometry is published beside the AmbiX bed; render effects do not have to infer transient positions from mixed audio.
- Phase/chirplet evidence is reduced into alignment, correction, suppression, and probe-control meaning before it crosses the live cache boundary. Raw phase is an estimator detail, not a renderer API.
@ -256,6 +265,7 @@ The audio code is split so the hard parts can be tested without the room, driver
- `audio_field.phase_meaning` owns live extraction of meaning from phase/chirplet evidence. It consumes known reference audio plus aligned mic windows, keeps the smoothed phase/frequency state internal, and emits only actionable state for alignment, suppression, and probe control.
- Room suppression owns stream-side cleanup: dialogue anchors define desired direct energy, spatial/context mics act as witnesses for room and transient clutter, and the cleaned aligned field feeds FOA/Aquarium/Faust.
- Source-event analysis owns non-vocal spatial facts: dialogue focus weights and localized transient vectors are published beside the AmbiX bed, not hidden inside it.
- Aquarium/Faust owns the hot voice-separation graph after `localcast.audio.mic_field`; LocalCastBridge owns alignment, timing, roles, and control publication.
- `scripts/audio_field.py` stays as the operator CLI and hardware probe surface.
This is the portfolio-piece line: each module owns one invariant, and the tests use mocks/fakes instead of asking a driver stack to please be emotionally available.

View file

@ -43,6 +43,10 @@
- `scripts/stream_phase_field.py` publishes the live meaning document from an aligned mic field plus known program/loopback reference without exposing raw phase bands as the renderer API.
- `audio_field.active_probe` wires low phase-field confidence into `ActiveProbeOptimizer`, emits bounded chirplet WAVs plus `active-probes.jsonl`, and can play probes through the default output device.
- `scripts/start-live-audio-phase-field.ps1` starts the live dense harmonic confidence loop with near-ultrasonic probes; `scripts/stop-live-audio-phase-field.ps1` stops the PID recorded in `logs/audio-phase-field.pid`.
- Faust voice-separation boundary:
- `localcast.audio.mic_field` publishes aligned six-mic float32 blocks at `calibration/runs/audio-mic-field.msgpack`.
- `faust/localcast_voice_separation.dsp` is the first Aquarium-hosted graph surface for host voice, co-streamer voice, ambient, transient, and loopback stems.
- `scripts/start-live-faust-mic-field.ps1` starts the mic-field publisher for Aquarium/Faust.
## Temporary
@ -68,4 +72,4 @@
13. Replace synthetic live-fusion observations with PS3 Eye detector observations and calibrated surface-feature tracks.
14. Keep the neighbor loopback leg running through the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present.
15. Let the co-streamer surface delay drive the shared presentation buffer horizon for audio stems, AmbiX, and remote video.
16. Wire `stream_phase_field.py` to the real live aligned-field producer instead of replaying WAV artifacts, then feed emitted active chirps back through loopback/mic capture so confidence maintenance closes against fresh observations instead of dry-run probe manifests.
16. Wire `stream_phase_field.py` and `stream_faust_mic_field.py` to the real live aligned-field producer instead of replaying WAV artifacts, then feed emitted active chirps back through loopback/mic capture so confidence maintenance closes against fresh observations instead of dry-run probe manifests.