3.5 KiB
3.5 KiB
Aquarium Spatial Audio
Objective
Publish the live spatial audio field through the same typed state discipline as the splat renderer. Aquarium consumes the AmbiX bus and hands it to native Faust DSP; LocalCastBridge owns capture, sync, calibration, and stream timing.
Current Mechanism
flowchart TD
A["aligned microphone field"] --> B["FOA encoder"]
B --> C["AmbiX ACN/SN3D W,Y,Z,X blocks"]
C --> D["localcast.audio.spatial_frame"]
A --> H["source-event analysis"]
H --> I["localcast.audio.source_events"]
D --> E["audio-state.msgpack / CultNet document_put"]
I --> J["audio-events.msgpack / CultNet document_put"]
E --> F["Aquarium client"]
J --> F
K["visual-state.msgpack"] --> F
F --> G["native Faust spatial DSP + volumetric renderer"]
Live files:
calibration/runs/audio-state.msgpack: latest typed AmbiX audio block.calibration/runs/audio-stream-status.msgpack: audio publisher heartbeat.calibration/runs/audio-events.msgpack: latest typed dialogue-focus and transient vector field.calibration/runs/av-sync-status.json: current Aquarium/Spout publisher view of visual frame id, audio frame id, audio delta, and synchronized event overlay count.
Document type:
localcast.audio.spatial_frame- schema id:
gamecult.localcast.audio.spatial_frame.v1 - key:
localcast.audio.spatial-frame.live - payload: MessagePack array decoded as
CultSpatialAudioFrame - audio format: interleaved
float32, 48 kHz, AmbiX ACN/SN3D channelsW,Y,Z,X
Source event document type:
localcast.audio.source_events- schema id:
gamecult.localcast.audio.source_events.v1 - key:
localcast.audio.source-events.live - payload: MessagePack array decoded as
CultAudioSourceEvents - contents: dialogue anchor weights plus witness-dominant transient events with sample time, estimated room position, direction, energy, confidence, and kind.
Publisher smoke command:
.\.venv\Scripts\python.exe .\scripts\stream_spatial_audio.py --input .\calibration\runs\audio-full-sync-20260518-165751\field-foa-ambix.wav --loop --duration 5 --smoke-readback
Source-event analysis command:
.\.venv\Scripts\python.exe .\scripts\audio_field.py analyze-source-events --profile .\calibration\runs\audio-field-loud-profile.json --input .\calibration\runs\<run-folder>\field-cleaned.wav --cache .\calibration\runs\audio-events.msgpack
Synchronized Aquarium/Spout publisher command:
.\.venv\Scripts\python.exe .\scripts\stream_spout.py --source cultcache --frame-cache .\calibration\runs\visual-state.msgpack --audio-cache .\calibration\runs\audio-state.msgpack --audio-events-cache .\calibration\runs\audio-events.msgpack --sync-status .\calibration\runs\av-sync-status.json
Invariants
- Aquarium owns rendering and Faust DSP, not microphone capture or clock correction.
- LocalCastBridge publishes declared AmbiX blocks, not undecoded heterogeneous microphone channels.
- LocalCastBridge publishes source-event geometry separately from AmbiX audio; render effects do not have to infer transient positions from the sound bed.
- Aquarium/Spout packaging selects audio source events against each visual frame's
audio_alignment_time_ns; OBS should receive the already synchronized output rather than trying to align independent media sources. - The audio frame carries
start_sampleandaudio_time_nsso visual packets can align against the same bounded-latency timeline. - OBS may receive a rendered monitor output later, but OBS is not the spatial bus authority.