Publish aligned mic field for Faust separation
This commit is contained in:
parent
e7a4bff764
commit
a4c3042628
12 changed files with 452 additions and 2 deletions
|
|
@ -11,11 +11,15 @@ flowchart TD
|
|||
A["aligned microphone field"] --> B["FOA encoder"]
|
||||
B --> C["AmbiX ACN/SN3D W,Y,Z,X blocks"]
|
||||
C --> D["localcast.audio.spatial_frame"]
|
||||
A --> M["localcast.audio.mic_field"]
|
||||
M --> N["Faust voice-separation graph"]
|
||||
A --> H["source-event analysis"]
|
||||
H --> I["localcast.audio.source_events"]
|
||||
D --> E["audio-state.msgpack / CultNet document_put"]
|
||||
M --> O["audio-mic-field.msgpack / CultNet document_put"]
|
||||
I --> J["audio-events.msgpack / CultNet document_put"]
|
||||
E --> F["Aquarium client"]
|
||||
O --> F
|
||||
J --> F
|
||||
K["visual-state.msgpack"] --> F
|
||||
F --> G["native Faust spatial DSP + volumetric renderer"]
|
||||
|
|
@ -24,6 +28,7 @@ flowchart TD
|
|||
Live files:
|
||||
|
||||
- `calibration/runs/audio-state.msgpack`: latest typed AmbiX audio block.
|
||||
- `calibration/runs/audio-mic-field.msgpack`: latest aligned six-mic block for Faust voice separation.
|
||||
- `calibration/runs/audio-stream-status.msgpack`: audio publisher heartbeat.
|
||||
- `calibration/runs/audio-events.msgpack`: latest typed dialogue-focus and transient vector field.
|
||||
- `calibration/runs/av-sync-status.json`: current Aquarium/Spout publisher view of visual frame id, audio frame id, audio delta, and synchronized event overlay count.
|
||||
|
|
@ -44,12 +49,31 @@ Source event document type:
|
|||
- payload: MessagePack array decoded as `CultAudioSourceEvents`
|
||||
- contents: dialogue anchor weights plus witness-dominant transient events with sample time, estimated room position, direction, energy, confidence, and kind.
|
||||
|
||||
Mic-field document type:
|
||||
|
||||
- `localcast.audio.mic_field`
|
||||
- schema id: `gamecult.localcast.audio.mic_field.v1`
|
||||
- key: `localcast.audio.mic-field.live`
|
||||
- payload: MessagePack array decoded as `CultAudioMicFieldFrame`
|
||||
- audio format: interleaved `float32`, 48 kHz, aligned mic channels
|
||||
- default channel order: `host-focusrite`, `co-streamer-focusrite`, `kiyo-0`, `kiyo-1`, `ps-eye-0`, `ps-eye-1`
|
||||
- graph id: `localcast.faust.voice_separation.v1`
|
||||
- Faust source: `faust/localcast_voice_separation.dsp`
|
||||
|
||||
Publisher smoke command:
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\python.exe .\scripts\stream_spatial_audio.py --input .\calibration\runs\audio-full-sync-20260518-165751\field-foa-ambix.wav --loop --duration 5 --smoke-readback
|
||||
```
|
||||
|
||||
Faust mic-field publisher:
|
||||
|
||||
```powershell
|
||||
.\scripts\start-live-faust-mic-field.ps1
|
||||
```
|
||||
|
||||
This publishes the aligned six-channel field directly to Aquarium/Faust while preserving the AmbiX path as a compatibility bed. Stop it with `.\scripts\stop-live-faust-mic-field.ps1`.
|
||||
|
||||
Source-event analysis command:
|
||||
|
||||
```powershell
|
||||
|
|
@ -65,8 +89,9 @@ Synchronized Aquarium/Spout publisher command:
|
|||
## Invariants
|
||||
|
||||
- Aquarium owns rendering and Faust DSP, not microphone capture or clock correction.
|
||||
- LocalCastBridge publishes declared AmbiX blocks, not undecoded heterogeneous microphone channels.
|
||||
- LocalCastBridge publishes declared AmbiX blocks for the spatial bed and aligned six-mic blocks for Faust voice separation. Unaligned heterogeneous microphones still do not cross this boundary.
|
||||
- LocalCastBridge publishes source-event geometry separately from AmbiX audio; render effects do not have to infer transient positions from the sound bed.
|
||||
- Faust owns the voice-separation graph once the aligned mic field has crossed into Aquarium. Python may publish controls and telemetry, but it must not become the hot separation engine.
|
||||
- Aquarium/Spout packaging selects audio source events against each visual frame's `audio_alignment_time_ns`; OBS should receive the already synchronized output rather than trying to align independent media sources.
|
||||
- The audio frame carries `start_sample` and `audio_time_ns` so visual packets can align against the same bounded-latency timeline.
|
||||
- OBS may receive a rendered monitor output later, but OBS is not the spatial bus authority.
|
||||
|
|
|
|||
|
|
@ -195,6 +195,14 @@ Encode an offline aligned six-channel WAV to FOA AmbiX:
|
|||
.\.venv\Scripts\python.exe .\scripts\audio_field.py encode-foa --profile .\config\audio-field.json --input .\calibration\runs\<run-folder>\field-aligned.wav --output .\calibration\runs\<run-folder>\field-foa-ambix.wav
|
||||
```
|
||||
|
||||
Publish the same aligned six-channel field directly to Aquarium/Faust for voice separation:
|
||||
|
||||
```powershell
|
||||
.\scripts\start-live-faust-mic-field.ps1
|
||||
```
|
||||
|
||||
This writes `localcast.audio.mic_field` into `calibration/runs/audio-mic-field.msgpack` with channel roles and graph id `localcast.faust.voice_separation.v1`. The Faust graph lives at `faust/localcast_voice_separation.dsp` and has six inputs plus six stem outputs: host voice, co-streamer voice, ambient, transients, local loopback placeholder, and co-streamer loopback placeholder.
|
||||
|
||||
Suppress room/transient witness energy before FOA encoding:
|
||||
|
||||
```powershell
|
||||
|
|
@ -237,6 +245,7 @@ The `play-record` and `record-field` commands remain for a `shared-input-device`
|
|||
- Ultrasonic probes are allowed only as a bounded optimization strategy. They must stay below Nyquist, remain level-capped, and prove themselves by raising measured confidence; hardware may distort or discard them.
|
||||
- Dense probe textures should be harmonically organized so any lower-band aliases land closer to musical relationships than arbitrary squeal. This is damage control, not a license to trust aliasing.
|
||||
- FOA output is AmbiX: ACN channel order, SN3D normalization, channels `W,Y,Z,X`.
|
||||
- The Faust voice-separation input is the aligned six-mic field, not the AmbiX collapse. Voice separation needs anchor and witness channels before spatial rendering destroys useful evidence.
|
||||
- Speaker calibration is a measurement path, not a generic monitor output.
|
||||
- Source-event geometry is published beside the AmbiX bed; render effects do not have to infer transient positions from mixed audio.
|
||||
- Phase/chirplet evidence is reduced into alignment, correction, suppression, and probe-control meaning before it crosses the live cache boundary. Raw phase is an estimator detail, not a renderer API.
|
||||
|
|
@ -256,6 +265,7 @@ The audio code is split so the hard parts can be tested without the room, driver
|
|||
- `audio_field.phase_meaning` owns live extraction of meaning from phase/chirplet evidence. It consumes known reference audio plus aligned mic windows, keeps the smoothed phase/frequency state internal, and emits only actionable state for alignment, suppression, and probe control.
|
||||
- Room suppression owns stream-side cleanup: dialogue anchors define desired direct energy, spatial/context mics act as witnesses for room and transient clutter, and the cleaned aligned field feeds FOA/Aquarium/Faust.
|
||||
- Source-event analysis owns non-vocal spatial facts: dialogue focus weights and localized transient vectors are published beside the AmbiX bed, not hidden inside it.
|
||||
- Aquarium/Faust owns the hot voice-separation graph after `localcast.audio.mic_field`; LocalCastBridge owns alignment, timing, roles, and control publication.
|
||||
- `scripts/audio_field.py` stays as the operator CLI and hardware probe surface.
|
||||
|
||||
This is the portfolio-piece line: each module owns one invariant, and the tests use mocks/fakes instead of asking a driver stack to please be emotionally available.
|
||||
|
|
|
|||
|
|
@ -43,6 +43,10 @@
|
|||
- `scripts/stream_phase_field.py` publishes the live meaning document from an aligned mic field plus known program/loopback reference without exposing raw phase bands as the renderer API.
|
||||
- `audio_field.active_probe` wires low phase-field confidence into `ActiveProbeOptimizer`, emits bounded chirplet WAVs plus `active-probes.jsonl`, and can play probes through the default output device.
|
||||
- `scripts/start-live-audio-phase-field.ps1` starts the live dense harmonic confidence loop with near-ultrasonic probes; `scripts/stop-live-audio-phase-field.ps1` stops the PID recorded in `logs/audio-phase-field.pid`.
|
||||
- Faust voice-separation boundary:
|
||||
- `localcast.audio.mic_field` publishes aligned six-mic float32 blocks at `calibration/runs/audio-mic-field.msgpack`.
|
||||
- `faust/localcast_voice_separation.dsp` is the first Aquarium-hosted graph surface for host voice, co-streamer voice, ambient, transient, and loopback stems.
|
||||
- `scripts/start-live-faust-mic-field.ps1` starts the mic-field publisher for Aquarium/Faust.
|
||||
|
||||
## Temporary
|
||||
|
||||
|
|
@ -68,4 +72,4 @@
|
|||
13. Replace synthetic live-fusion observations with PS3 Eye detector observations and calibrated surface-feature tracks.
|
||||
14. Keep the neighbor loopback leg running through the scheduled interactive WASAPI capture path and use it as the timing witness once actual neighbor app audio is present.
|
||||
15. Let the co-streamer surface delay drive the shared presentation buffer horizon for audio stems, AmbiX, and remote video.
|
||||
16. Wire `stream_phase_field.py` to the real live aligned-field producer instead of replaying WAV artifacts, then feed emitted active chirps back through loopback/mic capture so confidence maintenance closes against fresh observations instead of dry-run probe manifests.
|
||||
16. Wire `stream_phase_field.py` and `stream_faust_mic_field.py` to the real live aligned-field producer instead of replaying WAV artifacts, then feed emitted active chirps back through loopback/mic capture so confidence maintenance closes against fresh observations instead of dry-run probe manifests.
|
||||
|
|
|
|||
Loading…
Reference in a new issue