diff --git a/.gitignore b/.gitignore index 8766646..40f5fa7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ config/localcast.json +config/audio-field.json logs/ *.log *.tmp diff --git a/README.md b/README.md index e0c816f..3695433 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,12 @@ flowchart LR The video stream is encoded on the sender. Audio sources are separate endpoints so OBS can mix, mute, filter, and monitor them independently. +## Audio Field + +The expanded rig has a separate audio-field spine for six microphones and two calibration speakers. Start with `docs/audio-field.md` and `config/audio-field.example.json`. + +That path is stricter than the OBS endpoint path: microphones feeding the Ambisonic field need one synchronized capture device, explicit geometry, and calibration before they become an AmbiX FOA bus. Separate USB microphones can still be investigated, but they do not get waved through as a coherent field just because the JSON is feeling brave. + ## Current Status This repo is the first coherent scaffold: architecture docs, persistence machinery, and Windows scripts for device discovery, config validation, and FFmpeg command generation. It is ready for a real two-PC smoke test once FFmpeg with SRT/NVENC support is installed on the sender. diff --git a/config/audio-field.example.json b/config/audio-field.example.json new file mode 100644 index 0000000..3c14779 --- /dev/null +++ b/config/audio-field.example.json @@ -0,0 +1,116 @@ +{ + "sampleRate": 48000, + "blockSize": 1024, + "ambisonicBus": { + "order": 1, + "channelOrder": "ACN", + "normalization": "SN3D", + "channels": ["W", "Y", "Z", "X"] + }, + "inputDevice": { + "query": "Scarlett", + "hostApi": "WASAPI", + "channels": 6 + }, + "outputDevice": { + "query": "Scarlett", + "hostApi": "WASAPI", + "channels": 2 + }, + "microphones": [ + { + "id": "mic_front", + "channel": 0, + "label": "front", + "positionMeters": [0.0, 1.0, 1.3], + "orientationDeg": {"azimuth": 180.0, "elevation": 0.0}, + "directivity": "cardioid", + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + }, + { + "id": "mic_front_left", + "channel": 1, + "label": "front-left", + "positionMeters": [-0.87, 0.5, 1.3], + "orientationDeg": {"azimuth": -120.0, "elevation": 0.0}, + "directivity": "cardioid", + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + }, + { + "id": "mic_back_left", + "channel": 2, + "label": "back-left", + "positionMeters": [-0.87, -0.5, 1.3], + "orientationDeg": {"azimuth": -60.0, "elevation": 0.0}, + "directivity": "cardioid", + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + }, + { + "id": "mic_back", + "channel": 3, + "label": "back", + "positionMeters": [0.0, -1.0, 1.3], + "orientationDeg": {"azimuth": 0.0, "elevation": 0.0}, + "directivity": "cardioid", + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + }, + { + "id": "mic_back_right", + "channel": 4, + "label": "back-right", + "positionMeters": [0.87, -0.5, 1.3], + "orientationDeg": {"azimuth": 60.0, "elevation": 0.0}, + "directivity": "cardioid", + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + }, + { + "id": "mic_front_right", + "channel": 5, + "label": "front-right", + "positionMeters": [0.87, 0.5, 1.3], + "orientationDeg": {"azimuth": 120.0, "elevation": 0.0}, + "directivity": "cardioid", + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + } + ], + "speakers": [ + { + "id": "speaker_left", + "channel": 0, + "label": "left", + "positionMeters": [-0.75, 1.2, 1.2], + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + }, + { + "id": "speaker_right", + "channel": 1, + "label": "right", + "positionMeters": [0.75, 1.2, 1.2], + "gainDb": 0.0, + "delaySamples": 0, + "polarity": 1 + } + ], + "calibration": { + "sweepSeconds": 3.0, + "recordTailSeconds": 1.0, + "sweepStartHz": 80.0, + "sweepEndHz": 18000.0, + "levelDbfs": -18.0, + "referenceMicId": "mic_front" + } +} diff --git a/docs/audio-field.md b/docs/audio-field.md new file mode 100644 index 0000000..b5d319f --- /dev/null +++ b/docs/audio-field.md @@ -0,0 +1,102 @@ +# Audio Field Spine + +## Objective + +Build the audio side as one synchronized six-microphone capture path, a declared first-order Ambisonic field, and a two-speaker calibration/feedback path. + +This is not the old OBS endpoint model. OBS can still receive rendered stems later, but microphones that participate in the field belong to the audio-field pipeline first. The six channels need one clock, one profile, and one calibration ledger. Otherwise the system is just six microphones telling six slightly different lies at once. + +## Current Mechanism + +The repo now has a profile and tool: + +- `config/audio-field.example.json` declares the six microphone channels, two speaker channels, geometry, calibration sweep settings, and FOA AmbiX bus format. +- `scripts/audio_field.py` validates the profile, lists PortAudio devices, records six-channel fields, plays speaker calibration sweeps, analyzes speaker-to-mic returns, and encodes a recorded six-channel WAV into first-order AmbiX. + +The profile assumes one input device exposes at least six synchronized input channels and one output device exposes at least two output channels. If the mics are split across separate USB clocks, stop there. Do not pretend that drift is ambience with better branding. + +## Pipeline + +```mermaid +flowchart TD + A["config/audio-field.json"] --> B["audio_field.py validate"] + C["6 mic channels, one clock"] --> D["raw field WAV"] + E["2 speaker outputs"] --> F["calibration sweeps"] + F --> G["speaker-to-mic returns"] + G --> H["delay / gain / polarity analysis"] + H --> A + D --> I["calibration: delay / gain / polarity"] + I --> J["FOA encoder"] + J --> K["AmbiX ACN/SN3D WAV: W,Y,Z,X"] +``` + +## Commands + +Create a local profile first: + +```powershell +Copy-Item .\config\audio-field.example.json .\config\audio-field.json +``` + +List audio devices: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py devices +``` + +Validate the profile and device match: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py validate --profile .\config\audio-field.json --check-devices +``` + +Generate a calibration sweep without touching hardware: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py make-stimulus --profile .\config\audio-field.json +``` + +Play each speaker and record all six microphones: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py play-record --profile .\config\audio-field.json +``` + +Analyze the calibration run: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py analyze-calibration --profile .\config\audio-field.json --run .\calibration\runs\ +``` + +Record the raw field: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py record-field --profile .\config\audio-field.json --seconds 30 +``` + +Encode an offline raw field WAV to FOA AmbiX: + +```powershell +.\.venv\Scripts\python.exe .\scripts\audio_field.py encode-foa --profile .\config\audio-field.json --input .\calibration\runs\\field-raw.wav --output .\calibration\runs\\field-foa-ambix.wav +``` + +## Invariants + +- One synchronized capture device owns the six microphone channels. +- `audio-field.json` owns channel identity, geometry, gain, delay, polarity, and bus format. +- FOA output is AmbiX: ACN channel order, SN3D normalization, channels `W,Y,Z,X`. +- Speaker calibration is a measurement path, not a generic monitor output. +- Speaker-to-mic feedback belongs to calibration/echo-path estimation before it is used for live correction. +- Camera fusion may publish listener or source pose later, but it does not own audio channel timing. + +## First Cut Limits + +The current encoder is a source-style FOA mix from calibrated microphone channels and declared orientations. It is useful for proving the wiring, timing, and interchange format. It is not a full arbitrary-array sound-field reconstruction. + +The next honest steps are: + +1. Confirm one device exposes all six mic channels at 48 kHz. +2. Replace placeholder geometry with measured positions in the shared world coordinate system. +3. Run speaker sweeps and copy measured delay/gain/polarity back into `config/audio-field.json`. +4. Record a six-channel field and encode a first FOA WAV. +5. Only after that, decide whether the field needs a measured arbitrary-array encoder instead of the source-style FOA mix. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 681589e..0f7cb83 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -16,6 +16,10 @@ - Sender script uses `-nostdin` and repo-root logs so desktop launchers stay alive and diagnostics land in `C:\Meta\LocalCastBridge\logs`. - Sender video config is `1920x1080` for the interactive desktop. The `1024x768` SSH session size is not the streaming target. - Desktop `.cmd` launchers delegate to PowerShell wrappers so paths with spaces and apostrophes are handled in one place. +- Audio-field sidecar profile and tool: + - `config/audio-field.example.json` declares six mic channels, two speaker channels, placeholder geometry, calibration sweep settings, and FOA AmbiX ACN/SN3D output. + - `scripts/audio_field.py` lists devices, validates the profile, generates calibration sweeps, records speaker-to-mic calibration returns, analyzes delay/gain/polarity, records raw six-channel fields, and encodes offline WAVs to FOA. + - `docs/audio-field.md` maps the audio pipeline and its invariants. ## Temporary @@ -34,3 +38,5 @@ 6. Tune SRT latency and FFmpeg buffering for the local network. 7. Decide whether a small OBS scene/source generator is worth adding. 8. Reconsider plugin/fork only if standard OBS Media Source cannot preserve the required behavior. +9. Create local `config/audio-field.json`, confirm whether one device exposes six synchronized mic inputs, and replace placeholder mic/speaker geometry with measured world coordinates. +10. Run audio speaker sweeps, copy measured delay/gain/polarity into the profile, then encode a first FOA AmbiX test WAV. diff --git a/notes/current-system-map.md b/notes/current-system-map.md index af4ad82..4fff75d 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -34,3 +34,28 @@ An OBS plugin would be justified if OBS could not ingest stable LAN streams, or - Some FFmpeg builds omit SRT or NVENC. - Separate endpoints can drift; local latency should be tuned before adding a synchronization layer. - OBS SRT reconnection behavior can be fussy. Use stable ports and source reactivation before treating port changes as a fix. + +## Audio Field Sidecar + +The six-microphone Ambisonic path is separate from the OBS endpoint path. + +```mermaid +flowchart TD + A["config/audio-field.json"] --> B["scripts/audio_field.py"] + C["6 mic channels from one clocked input device"] --> D["raw field recording"] + E["2 speaker outputs"] --> F["calibration sweeps"] + F --> G["speaker-to-mic return recordings"] + G --> H["delay/gain/polarity analysis"] + H --> A + D --> I["FOA encoder"] + I --> J["AmbiX ACN/SN3D bus: W,Y,Z,X"] +``` + +Ownership: + +- `config/audio-field.json` owns mic/speaker identity, channel mapping, geometry, gain, delay, polarity, and Ambisonic bus format. +- `scripts/audio_field.py` owns hardware validation, calibration stimulus/return capture, offline calibration analysis, raw field recording, and FOA encoding. +- The camera/sensor-fusion pipeline may publish world poses later; it does not own audio clocks or channel timing. +- OBS may ingest rendered output later; it is not the authority for the Ambisonic field. + +Invariant: the six field microphones must enter through one synchronized capture path before FOA encoding. If the actual rig uses independent USB microphone clocks, prove the adaptive sync path first; do not treat separate endpoints as raw Ambisonics. diff --git a/scripts/audio_field.py b/scripts/audio_field.py new file mode 100644 index 0000000..113871b --- /dev/null +++ b/scripts/audio_field.py @@ -0,0 +1,465 @@ +import argparse +import json +import math +import sys +from datetime import datetime, timezone +from pathlib import Path + +import numpy as np +from scipy.io import wavfile +from scipy import signal + + +ROOT = Path(__file__).resolve().parents[1] +RUNS = ROOT / "calibration" / "runs" + + +def utc_stamp() -> str: + return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ") + + +def run_dir(kind: str) -> Path: + for suffix in ["", *[f"-{i}" for i in range(1, 100)]]: + path = RUNS / f"{utc_stamp()}-{kind}{suffix}" + try: + path.mkdir(parents=True, exist_ok=False) + return path + except FileExistsError: + continue + raise RuntimeError(f"Could not create unique audio run directory for {kind}") + + +def read_json(path: Path): + return json.loads(path.read_text(encoding="utf-8")) + + +def write_json(path: Path, data) -> None: + path.write_text(json.dumps(data, indent=2), encoding="utf-8") + + +def load_profile(path: Path): + profile = read_json(path) + validate_profile_shape(profile) + return profile + + +def validate_profile_shape(profile) -> None: + errors = [] + if int(profile.get("sampleRate", 0)) <= 0: + errors.append("sampleRate must be a positive integer") + bus = profile.get("ambisonicBus", {}) + if bus.get("order") != 1 or bus.get("channelOrder") != "ACN" or bus.get("normalization") != "SN3D": + errors.append("ambisonicBus must declare first-order ACN/SN3D for this tool") + if bus.get("channels") != ["W", "Y", "Z", "X"]: + errors.append("ambisonicBus.channels must be ['W', 'Y', 'Z', 'X']") + + mics = profile.get("microphones", []) + speakers = profile.get("speakers", []) + if len(mics) != 6: + errors.append("profile must declare exactly 6 microphones") + if len(speakers) != 2: + errors.append("profile must declare exactly 2 speakers") + + seen_mic_channels = set() + for mic in mics: + channel = mic.get("channel") + if not isinstance(channel, int) or channel < 0: + errors.append(f"microphone {mic.get('id', '')} has invalid channel") + if channel in seen_mic_channels: + errors.append(f"duplicate microphone channel {channel}") + seen_mic_channels.add(channel) + if len(mic.get("positionMeters", [])) != 3: + errors.append(f"microphone {mic.get('id', '')} needs positionMeters [x,y,z]") + orientation = mic.get("orientationDeg", {}) + if "azimuth" not in orientation or "elevation" not in orientation: + errors.append(f"microphone {mic.get('id', '')} needs orientationDeg azimuth/elevation") + if mic.get("polarity", 1) not in (-1, 1): + errors.append(f"microphone {mic.get('id', '')} polarity must be 1 or -1") + + seen_speaker_channels = set() + for speaker in speakers: + channel = speaker.get("channel") + if not isinstance(channel, int) or channel < 0: + errors.append(f"speaker {speaker.get('id', '')} has invalid channel") + if channel in seen_speaker_channels: + errors.append(f"duplicate speaker channel {channel}") + seen_speaker_channels.add(channel) + if len(speaker.get("positionMeters", [])) != 3: + errors.append(f"speaker {speaker.get('id', '')} needs positionMeters [x,y,z]") + if speaker.get("polarity", 1) not in (-1, 1): + errors.append(f"speaker {speaker.get('id', '')} polarity must be 1 or -1") + + input_channels = int(profile.get("inputDevice", {}).get("channels", 0)) + output_channels = int(profile.get("outputDevice", {}).get("channels", 0)) + if input_channels < 6: + errors.append("inputDevice.channels must be at least 6") + if output_channels < 2: + errors.append("outputDevice.channels must be at least 2") + + if errors: + raise SystemExit("Invalid audio profile:\n- " + "\n- ".join(errors)) + + +def sounddevice_module(): + try: + import sounddevice as sd + except Exception as exc: + raise SystemExit(f"sounddevice is unavailable: {exc!r}") from exc + return sd + + +def match_device(sd, spec, direction: str): + hostapis = sd.query_hostapis() + devices = sd.query_devices() + query = (spec.get("query") or "").lower() + hostapi_name = (spec.get("hostApi") or "").lower() + needed_channels = int(spec.get("channels") or 0) + matches = [] + for device in devices: + hostapi = hostapis[device["hostapi"]]["name"] + channel_key = "max_input_channels" if direction == "input" else "max_output_channels" + if device[channel_key] < needed_channels: + continue + if query and query not in device["name"].lower(): + continue + if hostapi_name and hostapi_name not in hostapi.lower(): + continue + matches.append({**device, "hostapi_name": hostapi}) + if not matches: + raise SystemExit(f"No {direction} device matches {spec!r}") + return matches[0] + + +def cmd_devices(args): + sd = sounddevice_module() + hostapis = sd.query_hostapis() + rows = [] + for dev in sd.query_devices(): + rows.append( + { + "index": dev["index"], + "name": dev["name"], + "hostapi": hostapis[dev["hostapi"]]["name"], + "inputs": dev["max_input_channels"], + "outputs": dev["max_output_channels"], + "default_samplerate": dev["default_samplerate"], + } + ) + print(json.dumps(rows, indent=2)) + + +def cmd_validate(args): + profile = load_profile(args.profile) + result = { + "profile": str(args.profile), + "sampleRate": profile["sampleRate"], + "microphones": [mic["id"] for mic in profile["microphones"]], + "speakers": [speaker["id"] for speaker in profile["speakers"]], + "ambisonicBus": profile["ambisonicBus"], + "deviceCheck": None, + } + if args.check_devices: + sd = sounddevice_module() + input_device = match_device(sd, profile["inputDevice"], "input") + output_device = match_device(sd, profile["outputDevice"], "output") + result["deviceCheck"] = { + "input": summarize_device(input_device), + "output": summarize_device(output_device), + } + print(json.dumps(result, indent=2)) + + +def summarize_device(device): + return { + "index": device["index"], + "name": device["name"], + "hostapi": device["hostapi_name"], + "inputs": device["max_input_channels"], + "outputs": device["max_output_channels"], + "default_samplerate": device["default_samplerate"], + } + + +def db_to_amp(db: float) -> float: + return 10.0 ** (db / 20.0) + + +def make_sweep(profile): + calibration = profile["calibration"] + sample_rate = int(profile["sampleRate"]) + seconds = float(calibration.get("sweepSeconds", 3.0)) + t = np.arange(int(sample_rate * seconds), dtype=np.float64) / sample_rate + sweep = signal.chirp( + t, + f0=float(calibration.get("sweepStartHz", 80.0)), + f1=float(calibration.get("sweepEndHz", 18000.0)), + t1=seconds, + method="logarithmic", + ) + fade_len = max(1, int(sample_rate * 0.02)) + fade = np.sin(np.linspace(0.0, math.pi / 2.0, fade_len)) ** 2 + sweep[:fade_len] *= fade + sweep[-fade_len:] *= fade[::-1] + sweep *= db_to_amp(float(calibration.get("levelDbfs", -18.0))) + return sweep.astype(np.float32) + + +def cmd_make_stimulus(args): + profile = load_profile(args.profile) + out = args.output or run_dir("audio-stimulus") + out.mkdir(parents=True, exist_ok=True) + sweep = make_sweep(profile) + wavfile.write(out / "calibration-sweep.wav", int(profile["sampleRate"]), sweep) + write_json( + out / "manifest.json", + { + "created_utc": datetime.now(timezone.utc).isoformat(), + "profile": str(args.profile), + "sampleRate": int(profile["sampleRate"]), + "sweep": profile["calibration"], + "file": "calibration-sweep.wav", + }, + ) + print(out) + + +def cmd_play_record(args): + profile = load_profile(args.profile) + sd = sounddevice_module() + input_device = match_device(sd, profile["inputDevice"], "input") + output_device = match_device(sd, profile["outputDevice"], "output") + sample_rate = int(profile["sampleRate"]) + input_channels = int(profile["inputDevice"]["channels"]) + output_channels = int(profile["outputDevice"]["channels"]) + sweep = make_sweep(profile) + tail = int(sample_rate * float(profile["calibration"].get("recordTailSeconds", 1.0))) + silence = np.zeros(tail, dtype=np.float32) + + out = args.output or run_dir("audio-calibration") + out.mkdir(parents=True, exist_ok=True) + records = [] + for speaker in profile["speakers"]: + playback = np.zeros((len(sweep) + tail, output_channels), dtype=np.float32) + playback[: len(sweep), int(speaker["channel"])] = sweep + recorded = sd.playrec( + playback, + samplerate=sample_rate, + channels=input_channels, + dtype="float32", + device=(input_device["index"], output_device["index"]), + blocking=True, + ) + stem = f"{speaker['id']}-return.wav" + wavfile.write(out / stem, sample_rate, recorded.astype(np.float32)) + records.append({"speakerId": speaker["id"], "file": stem}) + + wavfile.write(out / "calibration-sweep.wav", sample_rate, sweep) + write_json( + out / "manifest.json", + { + "created_utc": datetime.now(timezone.utc).isoformat(), + "profile": str(args.profile), + "inputDevice": summarize_device(input_device), + "outputDevice": summarize_device(output_device), + "records": records, + "stimulus": "calibration-sweep.wav", + "unusedSilenceSamples": int(len(silence)), + }, + ) + print(out) + + +def cmd_analyze_calibration(args): + profile = load_profile(args.profile) + run = args.run + manifest = read_json(run / "manifest.json") + sample_rate, sweep = wavfile.read(run / manifest["stimulus"]) + sweep = as_float_matrix(sweep).reshape(-1) + output = { + "created_utc": datetime.now(timezone.utc).isoformat(), + "sourceRun": str(run), + "sampleRate": int(sample_rate), + "paths": [], + } + for record in manifest["records"]: + rec_rate, recorded = wavfile.read(run / record["file"]) + if rec_rate != sample_rate: + raise SystemExit(f"Sample-rate mismatch in {record['file']}: {rec_rate} != {sample_rate}") + recorded = as_float_matrix(recorded) + for mic in profile["microphones"]: + channel = int(mic["channel"]) + if channel >= recorded.shape[1]: + continue + delay, peak, polarity = estimate_delay(recorded[:, channel], sweep) + output["paths"].append( + { + "speakerId": record["speakerId"], + "micId": mic["id"], + "delaySamples": int(delay), + "delayMs": 1000.0 * delay / sample_rate, + "peak": float(abs(peak)), + "polarity": int(polarity), + } + ) + write_json(run / "analysis.json", output) + print(json.dumps(output, indent=2)) + + +def as_float_matrix(data): + arr = np.asarray(data) + if arr.ndim == 1: + arr = arr[:, None] + if np.issubdtype(arr.dtype, np.integer): + arr = arr.astype(np.float32) / np.iinfo(arr.dtype).max + else: + arr = arr.astype(np.float32) + return arr + + +def estimate_delay(recorded, sweep): + corr = signal.fftconvolve(recorded.astype(np.float32), sweep[::-1].astype(np.float32), mode="full") + peak_index = int(np.argmax(np.abs(corr))) + delay = peak_index - (len(sweep) - 1) + peak = float(corr[peak_index]) + polarity = 1 if peak >= 0 else -1 + return delay, peak, polarity + + +def cmd_record_field(args): + profile = load_profile(args.profile) + sd = sounddevice_module() + input_device = match_device(sd, profile["inputDevice"], "input") + sample_rate = int(profile["sampleRate"]) + channels = int(profile["inputDevice"]["channels"]) + frames = int(float(args.seconds) * sample_rate) + out = args.output or run_dir("audio-field-record") + out.mkdir(parents=True, exist_ok=True) + rec = sd.rec( + frames, + samplerate=sample_rate, + channels=channels, + dtype="float32", + device=input_device["index"], + blocking=True, + ) + wavfile.write(out / "field-raw.wav", sample_rate, rec.astype(np.float32)) + write_json( + out / "manifest.json", + { + "created_utc": datetime.now(timezone.utc).isoformat(), + "profile": str(args.profile), + "inputDevice": summarize_device(input_device), + "seconds": float(args.seconds), + "file": "field-raw.wav", + }, + ) + print(out) + + +def cmd_encode_foa(args): + profile = load_profile(args.profile) + sample_rate, data = wavfile.read(args.input) + if int(sample_rate) != int(profile["sampleRate"]): + raise SystemExit(f"Input sample-rate {sample_rate} does not match profile {profile['sampleRate']}") + data = as_float_matrix(data) + foa = encode_foa(profile, data) + peak = float(np.max(np.abs(foa))) if foa.size else 0.0 + if peak > 0.98: + foa = foa / peak * 0.98 + wavfile.write(args.output, sample_rate, foa.astype(np.float32)) + print(json.dumps({"output": str(args.output), "sampleRate": int(sample_rate), "channels": ["W", "Y", "Z", "X"], "peakBeforeLimit": peak}, indent=2)) + + +def encode_foa(profile, data): + frames = data.shape[0] + corrected = [] + weights = [] + for mic in sorted(profile["microphones"], key=lambda item: item["channel"]): + channel = int(mic["channel"]) + if channel >= data.shape[1]: + raise SystemExit(f"Input has {data.shape[1]} channels, missing microphone channel {channel}") + x = data[:, channel].copy() + x = apply_integer_delay(x, int(mic.get("delaySamples", 0))) + x *= db_to_amp(float(mic.get("gainDb", 0.0))) + x *= int(mic.get("polarity", 1)) + corrected.append(x) + weights.append(foa_weights(mic)) + mic_matrix = np.stack(corrected, axis=1) + weight_matrix = np.stack(weights, axis=0) + foa = mic_matrix @ weight_matrix + return foa / max(1, len(corrected)) + + +def apply_integer_delay(x, delay): + if delay == 0: + return x + y = np.zeros_like(x) + if delay > 0: + y[delay:] = x[:-delay] + else: + y[:delay] = x[-delay:] + return y + + +def foa_weights(mic): + orientation = mic["orientationDeg"] + az = math.radians(float(orientation["azimuth"])) + el = math.radians(float(orientation["elevation"])) + ce = math.cos(el) + # ACN/SN3D first order: W, Y, Z, X. + return np.array( + [ + 1.0 / math.sqrt(2.0), + math.sin(az) * ce, + math.sin(el), + math.cos(az) * ce, + ], + dtype=np.float32, + ) + + +def main(): + parser = argparse.ArgumentParser(description="Build and calibrate the LocalCastBridge six-mic Ambisonic audio field.") + sub = parser.add_subparsers(required=True) + + p = sub.add_parser("devices", help="List PortAudio devices as JSON.") + p.set_defaults(func=cmd_devices) + + p = sub.add_parser("validate", help="Validate an audio field profile.") + p.add_argument("--profile", type=Path, default=ROOT / "config" / "audio-field.example.json") + p.add_argument("--check-devices", action="store_true") + p.set_defaults(func=cmd_validate) + + p = sub.add_parser("make-stimulus", help="Generate the calibration sweep WAV.") + p.add_argument("--profile", type=Path, default=ROOT / "config" / "audio-field.example.json") + p.add_argument("--output", type=Path) + p.set_defaults(func=cmd_make_stimulus) + + p = sub.add_parser("play-record", help="Play each speaker calibration sweep while recording all microphones.") + p.add_argument("--profile", type=Path, default=ROOT / "config" / "audio-field.example.json") + p.add_argument("--output", type=Path) + p.set_defaults(func=cmd_play_record) + + p = sub.add_parser("analyze-calibration", help="Estimate speaker-to-mic delay/gain/polarity from a calibration run.") + p.add_argument("--profile", type=Path, default=ROOT / "config" / "audio-field.example.json") + p.add_argument("--run", type=Path, required=True) + p.set_defaults(func=cmd_analyze_calibration) + + p = sub.add_parser("record-field", help="Record the raw synchronized six-channel microphone field.") + p.add_argument("--profile", type=Path, default=ROOT / "config" / "audio-field.example.json") + p.add_argument("--seconds", type=float, default=10.0) + p.add_argument("--output", type=Path) + p.set_defaults(func=cmd_record_field) + + p = sub.add_parser("encode-foa", help="Encode a calibrated six-channel WAV into first-order AmbiX B-format.") + p.add_argument("--profile", type=Path, default=ROOT / "config" / "audio-field.example.json") + p.add_argument("--input", type=Path, required=True) + p.add_argument("--output", type=Path, required=True) + p.set_defaults(func=cmd_encode_foa) + + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/state/evidence.jsonl b/state/evidence.jsonl index f7ae2b2..5d8229c 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -24,3 +24,4 @@ {"ts": "2026-05-18T13:56:00+00:00", "type": "verification", "status": "partial", "note": "With one PS3 Eye active after hub replug and switching to the other Eye, opentrack again enumerates high-rate modes but frame reads fail with payload error data[1]=78/bad header. This reduces suspicion of a single bad Eye and points to libusb0/backend or hub streaming."} {"ts": "2026-05-18T14:02:51+00:00", "type": "verification", "status": "partial", "note": "Both PS3 Eyes direct to motherboard and live MI_00 instances bound as libusbK: opentrack mode-test now reports all listed modes GOOD including 320x240@30, but frame-test still fails with payload error data[1]=78/79 and bad headers. Stale Unknown PS3 Eye instances from previous hub path remain in PnP."} {"ts": "2026-05-18T14:05:40+00:00", "type": "verification", "status": "failed", "note": "WinUSB on the two live PS3 Eye MI_00 instances still enumerates high-rate modes but frame streaming fails: one device open hit Access is denied, the other repeated payload error data[1]=78/79 until killed. libusb0, libusbK, and WinUSB have all failed frame reads via opentrack on this machine."} +{"ts": "2026-05-18T14:22:36+00:00", "type": "verification", "status": "partial", "note": "Audio-field sidecar added and offline FOA encode verified; current PortAudio enumeration shows no matching six-channel WASAPI input device, so live Ambisonic capture needs a shared six-input interface/profile or a separately proven adaptive sync path before using independent USB microphones."} diff --git a/state/map.yaml b/state/map.yaml index 1f8bdef..dc9de90 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -44,6 +44,7 @@ active_subgoals: - Smoke-test each audio endpoint in OBS and tune latency. - Decide whether an OBS source generator or plugin investigation is justified. - Evaluate the Ambisonic AquaSynth path as a separate synchronized multichannel audio pipeline; do not treat heterogeneous mic feeds as raw first-order ambisonic A-format without geometry, calibration, and a shared clock. + - Build the six-microphone/two-speaker audio-field sidecar around one clocked capture path, explicit geometry, speaker calibration, and FOA AmbiX output. research: ambisonic_aquasynth: @@ -121,8 +122,14 @@ current_status: first rough probe. PS3 Eye video interfaces now bind through libusb-win32 and opentrack enumerates high-rate modes, but frame reads fail while all cameras sit on one external hub/root path; USB topology is now a live - suspect. + suspect. Audio-field scaffolding now exists as a separate sidecar: + six-mic/two-speaker profile, validation, calibration sweep capture/analysis, + raw field recording, and offline FOA AmbiX encoding. Current PortAudio + enumeration does not expose a matching six-channel WASAPI input, so live + field capture requires either a shared six-input device/profile or a + separately proven adaptive sync path for independent USB microphones. next_action: > - Test one PS3 Eye alone on the current hub and then on a different root port - if needed; once one streams, add the second at a low tracking rate before - capturing ChArUco intrinsics. + In parallel with the camera driver's next cut, create local + config/audio-field.json, confirm whether one device exposes six synchronized + mic inputs, replace placeholder mic/speaker geometry with measured world + coordinates, then run the two-speaker calibration sweep.