GameCult/Mimir
Watch
9
0
Fork
You've already forked Mimir
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Tame online Move music gestures

Browse source
This commit is contained in:
Meta Vens 2026-06-01 21:41:45 +01:00
commit df03e66e45
4 changed files with 341 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

12
scripts/start-online-verse-daemons.ps1
View file

@ -166,10 +166,20 @@ if (-not $SkipMoves) {
"--duration", $(if ($DurationSeconds -gt 0) { "$DurationSeconds" } else { "31536000" }),
"--out-dir", (Join-Path $runDir "move-sync"),
"--device", "Realtek",
"--asio-music-channels", "0,1,2,3",
"--asio-drain-blocks", "1024",
"--fps", "$MoveFps",
"--fft-size", "$MoveFftSize",
"--onset-cooldown-seconds", "0.065",
"--warmup-seconds", "0.25",
"--warmup-seconds", "2.0",
"--loudness-history-seconds", "14.0",
"--loudness-threshold", "0.86",
"--hit-loudness-threshold", "0.84",
"--loudness-floor", "0.006",
"--loudness-exponent", "2.6",
"--quiet-brightness", "0.008",
"--max-brightness", "0.72",
"--brightness-exponent", "0.9",
"--debruijn-polyrhythm",
"--harmonic-base", "1.5",
"--microtonal-cents", "17"

1
state/evidence.jsonl
View file

@ -337,3 +337,4 @@
{"ts": "2026-06-01T17:42:27+01:00", "type": "verification", "status": "ok", "note": "2026-06-01: Release builds passed for Fensalir Aquarium.Engine, Mimir.EveDashboard, and Mimir.App after adding typed Eve/CultUI binding descriptors. Mimir.BufferSmoke --scene-editor-smoke passed with preview/source placement interactions intact."}
{"ts": "2026-06-01T21:06:07+01:00", "type": "implementation", "status": "ok", "note": "Added the first online Verse observation daemon stack: Mimir.EveSensorReceiver now serves a /eve/periwinkle/subscribe WebSocket lane, Mimir.VerseRecorder subscribes and writes session.json plus observations.jsonl under C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures, and scripts/start-online-verse-daemons.ps1 supervises relay, recorder, Nightwing Eyes/Move witness publication, music-synced Move gestures, and a local BufferSmoke runtime capture worker. Live run online-verse-daemons-20260601-210106 had relay, recorder, Move sync, BufferSmoke, and Nightwing witness running, with observations.jsonl past 1000 records and growing. This captures typed witness observations; raw aligned source windows/stems still need a MimirSynchronizationHub Verse publisher."}
{"ts": "2026-06-01T21:27:15+01:00", "type": "implementation", "status": "partial", "note": "Added Mimir.Well as the long-running Starfire Well daemon and broad config/mimir-runtime.well.local.json. Live run online-verse-daemons-20260601-212502 started relay, recorder, Mimir.Well, music-synced de-Bruijn/microtonal Move gestures, and Nightwing witness PID 1116285. Recorder C:\\Users\\Meta\\Videos\\Mimir\\VerseCaptures\\20260601-212509 reached 292 records and included mimir.well_snapshot.v1, Nightwing builtin camera/mic witness records, and ps3-eye-0/1 Move observations. Well snapshots showed live Scarlett ASIO, Leap stereo IR, Kiyo Pro RGB, and Kiyo Basic RGB buffers; Raven and Eve lanes were configured but empty because their producers were not feeding. Sync frame remains degraded because ASIO device timestamps and camera driver timestamps are not yet mapped into one canonical timeline."}
{"ts": "2026-06-01T21:41:07+01:00", "type": "decision", "status": "ok", "note": "2026-06-01 online Move music sync demoted spectral flux from brightness authority: Realtek loopback remains primary, optional Scarlett ASIO channels add auxiliary onset/tempo evidence, chroma estimates key/chord, and recent loudness percentile gates all Move brightness so quiet/steady-state music stays dark."}

9
state/map.yaml
View file

@ -220,6 +220,12 @@ invariants:
rises approach full brightness. The ASIO reader now drains the native
capture queue to the newest loopback blocks before each LED frame, because
stale queued audio was the plausible seconds-late music-pulse cause.
`tools/online_move_music_sync.py` is now the live multi-Move music witness:
Realtek loopback remains the primary program reference, optional Scarlett
ASIO channels contribute auxiliary mic/loopback onset streams with their
own autocorrelation tempo estimates, chroma scoring estimates key and chord
candidates, and a recent loudness-percentile gate owns whether the Moves
are allowed to brighten. Flux alone no longer has authority to shout.
`tools/move_latency_probe.py` is the diagnostic PS Move latency receipt:
PS3 Eyes are the default fast visual witnesses, Scarlett cardioid is the
default chirp timing witness, and the event train uses an order-3 de Bruijn
@ -319,7 +325,8 @@ invariants:
`mimir.well_snapshot.v1` records containing buffer health, composite
controls, publication/stem contract, and sync status. The online supervisor
now starts relay, recorder, Nightwing Eyes/Move/builtin-camera/builtin-mic
witness publication, de-Bruijn/microtonal music-synced Move gestures, and
witness publication, conservative de-Bruijn/microtonal music-synced Move
gestures with Realtek plus opportunistic ASIO music evidence, and
`Mimir.Well`. This owns typed state publication, not raw CultMesh media
payloads yet. The live timestamp-domain scar is explicit: ASIO device time
and camera driver time are not in one canonical timeline, so sync frame

335
tools/online_move_music_sync.py
View file

@ -25,6 +25,15 @@ from pathlib import Path
import numpy as np
try:
from nightwing_psmove_music_pulse import AsioOnsetReader
except Exception: # pragma: no cover - optional live hardware path
AsioOnsetReader = None # type: ignore[assignment]
PITCH_NAMES = ("C", "C#", "D", "Eb", "E", "F", "F#", "G", "Ab", "A", "Bb", "B")
MAJOR_KEY_PROFILE = np.asarray([6.35, 2.23, 3.48, 2.33, 4.38, 4.09, 2.52, 5.19, 2.39, 3.66, 2.29, 2.88], dtype=np.float32)
MINOR_KEY_PROFILE = np.asarray([6.33, 2.68, 3.52, 5.38, 2.60, 3.53, 2.54, 4.75, 3.98, 2.69, 3.34, 3.17], dtype=np.float32)
REMOTE_MULTI_RECEIVER = r"""
import os
@ -116,11 +125,14 @@ class OnlineAnalyzer:
self.minv = np.zeros(bins, dtype=np.float32)
self.maxv = np.ones(bins, dtype=np.float32) * 1e-4
self.flux_history: deque[float] = deque(maxlen=max(8, int(args.onset_history_seconds * fps)))
self.loudness_history: deque[float] = deque(maxlen=max(8, int(args.loudness_history_seconds * fps)))
self.onset_series: deque[float] = deque(maxlen=max(32, int(args.tempo_window_seconds * fps)))
self.pending = np.zeros(0, dtype=np.float32)
self.last_peak = 0.0
self.onset_env = 0.0
self.level_env = 0.0
self.loudness_env = 0.0
self.chroma_smooth = np.ones(12, dtype=np.float32) / 12.0
self.bpm = 140.0
self.bpm_confidence = 0.0
self.beat_phase = 0.0
@ -150,20 +162,37 @@ class OnlineAnalyzer:
body = float(np.sqrt(np.mean(np.square(mono)))) if mono.size else 0.0
self.flux_history.append(flux)
ordered = sorted(self.flux_history)
rank = 0.0
if ordered:
below = sum(1 for value in ordered if value <= flux)
rank = below / len(ordered)
warm = len(self.flux_history) >= min(self.flux_history.maxlen or 1, int(a.warmup_seconds * self.fps))
self.loudness_history.append(body)
rank = percentile_rank(self.flux_history, flux)
loudness_rank = percentile_rank(self.loudness_history, body)
loudness_scale = self._scaled_loudness(body)
loudness_gate = 0.0
if body >= a.loudness_floor and loudness_rank >= a.loudness_threshold:
span = max(1e-6, 1.0 - a.loudness_threshold)
loudness_gate = ((loudness_rank - a.loudness_threshold) / span) ** a.loudness_exponent
loudness_gate *= loudness_scale
self.loudness_env = max(loudness_gate, self.loudness_env * a.loudness_decay)
warm_needed = int(a.warmup_seconds * self.fps)
warm = (
len(self.flux_history) >= min(self.flux_history.maxlen or 1, warm_needed)
and len(self.loudness_history) >= min(self.loudness_history.maxlen or 1, warm_needed)
)
hit = 0.0
if warm and rank >= a.onset_threshold and (now - self.last_peak) >= a.onset_cooldown_seconds:
if (
warm
and rank >= a.onset_threshold
and loudness_rank >= a.hit_loudness_threshold
and loudness_gate > 0.0
and (now - self.last_peak) >= a.onset_cooldown_seconds
):
hit = min(1.0, max(0.0, rank) ** a.onset_exponent)
hit *= max(0.0, min(1.0, loudness_gate))
self.last_peak = now
self.beat_phase = 0.0
self.onset_env = max(hit, self.onset_env * a.onset_decay)
self.level_env = max(self.onset_env, body * a.body_gain, self.level_env * a.level_decay)
musical_level = self.loudness_env * min(1.0, body * a.body_gain)
self.level_env = max(self.onset_env, musical_level, self.level_env * a.level_decay)
self.onset_series.append(flux)
if len(self.onset_series) >= int(self.fps * 4) and int(now * 2) != int(self.last_t * 2):
self._estimate_tempo()
@ -172,19 +201,38 @@ class OnlineAnalyzer:
self.last_t = now
self.beat_phase = (self.beat_phase + dt * self.bpm / 60.0) % 1.0
f0, color = self._estimate_fundamental_and_color(mag)
chroma, key_name, key_mode, key_confidence, chord_name, chord_confidence, chord_root = self._estimate_chroma_key_chord(mag)
return {
"flux": flux,
"percentile": rank,
"hit": hit,
"body": body,
"loudness_percentile": loudness_rank,
"loudness_scale": loudness_scale,
"loudness_gate": loudness_gate,
"level": self.level_env,
"bpm": self.bpm,
"bpm_confidence": self.bpm_confidence,
"beat_phase": self.beat_phase,
"fundamental_hz": f0,
"color_balance": color,
"chroma": tuple(float(value) for value in chroma),
"key_name": key_name,
"key_mode": key_mode,
"key_confidence": key_confidence,
"chord_name": chord_name,
"chord_confidence": chord_confidence,
"chord_root": chord_root,
}
def _scaled_loudness(self, body: float) -> float:
if len(self.loudness_history) < 4:
return 0.0
values = np.asarray(self.loudness_history, dtype=np.float32)
floor = max(self.args.loudness_floor, float(np.percentile(values, 35)))
ceiling = max(floor + 1e-6, float(np.percentile(values, 96)))
return max(0.0, min(1.0, (body - floor) / (ceiling - floor)))
def _estimate_tempo(self) -> None:
x = np.asarray(self.onset_series, dtype=np.float32)
x = x - float(np.mean(x))
@ -238,6 +286,169 @@ class OnlineAnalyzer:
bands /= max(1e-6, float(np.max(bands)))
return best_f0, (float(bands[0]), float(bands[1]), float(bands[2]))
def _estimate_chroma_key_chord(self, mag: np.ndarray) -> tuple[np.ndarray, str, str, float, str, float, int]:
freqs = np.fft.rfftfreq(self.fft_size, 1.0 / self.rate)
chroma = np.zeros(12, dtype=np.float32)
for f, m in zip(freqs[2:], mag[2:]):
if f < self.args.chroma_min_hz or f > self.args.chroma_max_hz:
continue
midi = 69.0 + 12.0 * math.log2(float(f) / 440.0)
pitch_class = int(round(midi)) % 12
chroma[pitch_class] += float(m)
total = float(np.sum(chroma))
if total > 1e-6:
chroma /= total
chroma = np.power(np.maximum(chroma, 0.0), self.args.chroma_contrast)
chroma /= max(1e-6, float(np.sum(chroma)))
self.chroma_smooth = (
self.args.chroma_smoothing * self.chroma_smooth
+ (1.0 - self.args.chroma_smoothing) * chroma
)
self.chroma_smooth /= max(1e-6, float(np.sum(self.chroma_smooth)))
key_root, key_mode, key_confidence = score_key(self.chroma_smooth)
chord_root, chord_quality, chord_confidence = score_chord(self.chroma_smooth)
key_name = PITCH_NAMES[key_root]
chord_name = f"{PITCH_NAMES[chord_root]}{'' if chord_quality == 'maj' else 'm'}"
return self.chroma_smooth.copy(), key_name, key_mode, key_confidence, chord_name, chord_confidence, chord_root
def percentile_rank(values: deque[float], value: float) -> float:
if not values:
return 0.0
below = sum(1 for item in values if item <= value)
return below / len(values)
def centered_profile_score(chroma: np.ndarray, profile: np.ndarray) -> float:
x = chroma - float(np.mean(chroma))
y = profile - float(np.mean(profile))
denom = float(np.linalg.norm(x) * np.linalg.norm(y))
if denom <= 1e-9:
return 0.0
return float(np.dot(x, y) / denom)
def score_key(chroma: np.ndarray) -> tuple[int, str, float]:
scores: list[tuple[float, int, str]] = []
for root in range(12):
scores.append((centered_profile_score(chroma, np.roll(MAJOR_KEY_PROFILE, root)), root, "major"))
scores.append((centered_profile_score(chroma, np.roll(MINOR_KEY_PROFILE, root)), root, "minor"))
scores.sort(reverse=True)
best, root, mode = scores[0]
runner_up = scores[1][0] if len(scores) > 1 else 0.0
confidence = max(0.0, min(1.0, (best - runner_up) * 2.5 + max(0.0, best) * 0.35))
return root, mode, confidence
def score_chord(chroma: np.ndarray) -> tuple[int, str, float]:
scores: list[tuple[float, int, str]] = []
for root in range(12):
major = float(chroma[root] + 0.85 * chroma[(root + 4) % 12] + chroma[(root + 7) % 12])
minor = float(chroma[root] + 0.85 * chroma[(root + 3) % 12] + chroma[(root + 7) % 12])
scores.append((major, root, "maj"))
scores.append((minor, root, "min"))
scores.sort(reverse=True)
best, root, quality = scores[0]
runner_up = scores[1][0] if len(scores) > 1 else 0.0
confidence = max(0.0, min(1.0, (best - runner_up) * 4.0 + best * 0.25))
return root, quality, confidence
class AuxiliaryOnsetStream:
def __init__(self, name: str, fps: float, args: argparse.Namespace) -> None:
self.name = name
self.fps = fps
self.args = args
self.series: deque[float] = deque(maxlen=max(32, int(args.tempo_window_seconds * fps)))
self.bpm = 140.0
self.confidence = 0.0
self.last_estimate_t = 0.0
self.last_state: dict[str, object] = {
"source": name,
"hit": 0.0,
"body": 0.0,
"percentile": 0.0,
"fundamental_hz": 0.0,
"bpm": self.bpm,
"bpm_confidence": self.confidence,
}
def update(
self,
now: float,
hit: float,
body: float,
color_balance: tuple[float, float, float],
fundamental_hz: float,
range_hit: float,
percentile: float,
) -> dict[str, object]:
onset = max(float(hit), (max(0.0, min(1.0, float(percentile))) ** 2) * max(0.0, min(1.5, float(range_hit))))
self.series.append(onset)
if len(self.series) >= int(self.fps * 4) and now - self.last_estimate_t >= 0.5:
self._estimate_tempo()
self.last_estimate_t = now
self.last_state = {
"source": self.name,
"hit": float(hit),
"body": float(body),
"percentile": float(percentile),
"onset": float(onset),
"fundamental_hz": float(fundamental_hz),
"color_balance": tuple(float(value) for value in color_balance),
"bpm": self.bpm,
"bpm_confidence": self.confidence,
}
return self.last_state
def _estimate_tempo(self) -> None:
x = np.asarray(self.series, dtype=np.float32)
x = x - float(np.mean(x))
if float(np.max(np.abs(x))) < 1e-6:
return
best_bpm = self.bpm
best_score = 0.0
for bpm in np.linspace(self.args.tempo_min_bpm, self.args.tempo_max_bpm, 161):
lag = int(round((60.0 / float(bpm)) * self.fps))
if lag <= 1 or lag >= len(x) - 4:
continue
a = x[lag:]
b = x[:-lag]
denom = float(np.linalg.norm(a) * np.linalg.norm(b))
score = 0.0 if denom <= 1e-9 else float(np.dot(a, b) / denom)
if score > best_score:
best_score = score
best_bpm = float(bpm)
self.bpm = 0.88 * self.bpm + 0.12 * best_bpm
self.confidence = max(0.0, min(1.0, best_score))
def fuse_music_sources(primary: dict[str, object], source_states: list[dict[str, object]]) -> dict[str, object]:
if not source_states:
primary["music_sources"] = [{"source": "realtek-loopback", "bpm": primary["bpm"], "bpm_confidence": primary["bpm_confidence"]}]
return primary
weighted_bpm = float(primary["bpm"]) * (0.25 + float(primary["bpm_confidence"]))
weight_sum = 0.25 + float(primary["bpm_confidence"])
fundamentals = [(float(primary["fundamental_hz"]), 0.5 + float(primary["loudness_gate"]))]
for source in source_states:
confidence = float(source.get("bpm_confidence", 0.0))
weight = 0.15 + confidence
weighted_bpm += float(source.get("bpm", primary["bpm"])) * weight
weight_sum += weight
body_weight = max(0.0, min(1.0, float(source.get("body", 0.0)) * 20.0))
fundamentals.append((float(source.get("fundamental_hz", primary["fundamental_hz"])), body_weight))
if weight_sum > 1e-6:
primary["bpm"] = weighted_bpm / weight_sum
primary["bpm_confidence"] = max(float(primary["bpm_confidence"]), max(float(s.get("bpm_confidence", 0.0)) for s in source_states))
fundamental_weight = sum(weight for _, weight in fundamentals)
if fundamental_weight > 1e-6:
primary["fundamental_hz"] = sum(freq * weight for freq, weight in fundamentals) / fundamental_weight
primary["music_sources"] = [
{"source": "realtek-loopback", "bpm": primary["bpm"], "bpm_confidence": primary["bpm_confidence"]},
*source_states,
]
return primary
DEBRUIJN_2_3 = "00010111"
@ -255,6 +466,8 @@ def move_rgb(move: Move, move_index: int, move_count: int, state: dict[str, obje
level = float(state["level"])
beat = float(state["beat_phase"])
hit = float(state["hit"])
loudness_gate = float(state.get("loudness_gate", 0.0))
loudness_percentile = float(state.get("loudness_percentile", 0.0))
base_h, base_s, _ = rgb_to_hsv01(move.base_rgb)
spectral = state["color_balance"]
assert isinstance(spectral, tuple)
@ -263,15 +476,28 @@ def move_rgb(move: Move, move_index: int, move_count: int, state: dict[str, obje
phase = (beat * harmonic * micro + move_index / max(1, move_count)) % 1.0
poly = debruijn_accent(beat, move_index, args)
gesture = poly * (0.42 + 0.58 * (0.5 + 0.5 * math.sin(2.0 * math.pi * phase)) ** 2.1)
accent = min(1.0, level * gesture + hit * (0.48 + 0.18 * poly))
accent = min(1.0, loudness_gate * (level * gesture + hit * (0.34 + 0.12 * poly)))
hue = (base_h + args.hue_bend * math.sin(2.0 * math.pi * phase) + math.log2(harmonic * micro) * 0.0833) % 1.0
sat = min(1.0, max(0.72, base_s) + 0.18 * hit)
val = min(1.0, accent ** 0.55)
chord_confidence = float(state.get("chord_confidence", 0.0))
chord_root = int(state.get("chord_root", 0))
if chord_confidence > args.chord_hue_threshold:
chord_degrees = (0, 4, 7, 10)
pitch_class = (chord_root + chord_degrees[move_index % len(chord_degrees)]) % 12
pitch_hue = pitch_class / 12.0
mix = min(args.chord_hue_mix, chord_confidence * args.chord_hue_mix)
hue = mix_hue(hue, pitch_hue, mix)
sat = min(1.0, max(0.72, base_s) + 0.22 * loudness_gate + 0.08 * hit)
if loudness_gate <= 0.0:
val = 0.0
else:
val = min(args.max_brightness, max(args.quiet_brightness, accent ** args.brightness_exponent))
if loudness_percentile < args.loudness_threshold:
val = min(val, args.quiet_brightness)
rr, gg, bb = colorsys.hsv_to_rgb(hue, sat, val)
r = rr * (0.45 + 0.55 * float(spectral[0]))
g = gg * (0.45 + 0.55 * float(spectral[1]))
b = bb * (0.45 + 0.55 * float(spectral[2]))
white = max(0.0, hit - 0.92) / 0.08
white = loudness_gate * max(0.0, hit - 0.94) / 0.06
return (
int(min(255, 255 * r + 18 * white)),
int(min(255, 255 * g + 18 * white)),
@ -279,6 +505,11 @@ def move_rgb(move: Move, move_index: int, move_count: int, state: dict[str, obje
)
def mix_hue(a: float, b: float, amount: float) -> float:
delta = ((b - a + 0.5) % 1.0) - 0.5
return (a + delta * max(0.0, min(1.0, amount))) % 1.0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--duration", type=float, default=1800.0)
@ -287,12 +518,25 @@ def main() -> int:
parser.add_argument("--device", default="Realtek")
parser.add_argument("--sample-rate", type=int, default=48000)
parser.add_argument("--channels", type=int, default=2)
parser.add_argument("--asio-dll", default=str(Path(__file__).resolve().parents[1] / "native" / "asio_capture" / "build" / "Release" / "mimir_asio_capture.dll"))
parser.add_argument("--asio-clsid", default="{AC4D0455-50D7-4498-B3CD-9A41D130B759}")
parser.add_argument("--asio-music-channels", default="", help="Optional comma-separated Scarlett ASIO channels to fold into music evidence.")
parser.add_argument("--asio-drain-blocks", type=int, default=1024)
parser.add_argument("--fps", type=float, default=60.0)
parser.add_argument("--fft-size", type=int, default=512)
parser.add_argument("--ssh-target", default="nightwing")
parser.add_argument("--move", action="append", required=True)
parser.add_argument("--body-gain", type=float, default=0.22)
parser.add_argument("--level-decay", type=float, default=0.80)
parser.add_argument("--loudness-decay", type=float, default=0.62)
parser.add_argument("--loudness-history-seconds", type=float, default=12.0)
parser.add_argument("--loudness-threshold", type=float, default=0.84)
parser.add_argument("--hit-loudness-threshold", type=float, default=0.82)
parser.add_argument("--loudness-floor", type=float, default=0.006)
parser.add_argument("--loudness-exponent", type=float, default=2.4)
parser.add_argument("--quiet-brightness", type=float, default=0.012)
parser.add_argument("--max-brightness", type=float, default=0.76)
parser.add_argument("--brightness-exponent", type=float, default=0.82)
parser.add_argument("--onset-decay", type=float, default=0.18)
parser.add_argument("--onset-threshold", type=float, default=0.62)
parser.add_argument("--onset-exponent", type=float, default=2.0)
@ -311,6 +555,12 @@ def main() -> int:
parser.add_argument("--tempo-max-bpm", type=float, default=260.0)
parser.add_argument("--fundamental-min", type=float, default=55.0)
parser.add_argument("--fundamental-max", type=float, default=880.0)
parser.add_argument("--chroma-min-hz", type=float, default=55.0)
parser.add_argument("--chroma-max-hz", type=float, default=4200.0)
parser.add_argument("--chroma-contrast", type=float, default=1.7)
parser.add_argument("--chroma-smoothing", type=float, default=0.92)
parser.add_argument("--chord-hue-threshold", type=float, default=0.16)
parser.add_argument("--chord-hue-mix", type=float, default=0.34)
parser.add_argument("--debruijn-polyrhythm", action="store_true")
parser.add_argument("--debruijn-rate", type=int, default=3)
parser.add_argument("--harmonic-base", type=float, default=1.5)
@ -351,6 +601,21 @@ def main() -> int:
)
assert wasapi.stdout is not None and ssh.stdin is not None
analyzer = OnlineAnalyzer(args.sample_rate, args.fps, args.fft_size, args)
asio_reader = None
asio_tracker = None
if args.asio_music_channels.strip():
if AsioOnsetReader is None:
print("asio-music disabled: nightwing_psmove_music_pulse.AsioOnsetReader is unavailable", file=sys.stderr, flush=True)
else:
try:
asio_channels = {int(part.strip()) for part in args.asio_music_channels.split(",") if part.strip()}
asio_reader = AsioOnsetReader(args.asio_dll, args.asio_clsid, args.sample_rate, asio_channels, args.fft_size)
asio_tracker = AuxiliaryOnsetStream("scarlett-asio", args.fps, args)
print(f"asio-music enabled channels={sorted(asio_channels)}", file=sys.stderr, flush=True)
except Exception as ex:
print(f"asio-music disabled: {ex}", file=sys.stderr, flush=True)
asio_reader = None
asio_tracker = None
frame_samples = max(args.fft_size, int(args.sample_rate / args.fps))
frame_bytes = frame_samples * args.channels * 4
trace_path = out_dir / "online-sync.jsonl"
@ -366,6 +631,39 @@ def main() -> int:
mono = np.mean(block, axis=1)
now = time.monotonic()
state = analyzer.analyze(mono, now)
auxiliary_states: list[dict[str, object]] = []
if asio_reader is not None and asio_tracker is not None:
try:
asio_result = asio_reader.read_onset(
floor=args.loudness_floor,
gain=1.0,
deadline=time.monotonic() + (0.35 / args.fps),
whiten_fast=args.whiten_fast,
whiten_slow=args.whiten_slow,
whiten_delta_fast=args.whiten_delta_fast,
whiten_delta_slow=args.whiten_delta_slow,
whiten_decay=args.whiten_decay,
whiten_contrast=args.whiten_contrast,
fundamental_min=args.fundamental_min,
fundamental_max=args.fundamental_max,
onset_percentile_threshold=0.0,
onset_exponent=args.onset_exponent,
onset_history_ms=args.onset_history_seconds * 1000.0,
onset_cooldown_ms=args.onset_cooldown_seconds * 1000.0,
warmup_ms=args.warmup_seconds * 1000.0,
drain_blocks=args.asio_drain_blocks,
)
if asio_result is not None:
auxiliary_states.append(asio_tracker.update(now, *asio_result))
except Exception as ex:
print(f"asio-music disabled during read: {ex}", file=sys.stderr, flush=True)
try:
asio_reader.close()
except Exception:
pass
asio_reader = None
asio_tracker = None
state = fuse_music_sources(state, auxiliary_states)
rgbs = {}
for index, move in enumerate(moves):
rgb = move_rgb(move, index, len(moves), state, args)
@ -380,6 +678,8 @@ def main() -> int:
"frame": frames,
**{key: value for key, value in state.items() if key != "color_balance"},
"color_balance": list(state["color_balance"]), # type: ignore[index]
"chroma": list(state["chroma"]), # type: ignore[index]
"music_sources": state["music_sources"],
"moves": {name: list(rgb) for name, rgb in rgbs.items()},
}
trace.write(json.dumps(record, sort_keys=True) + "\n")
@ -387,9 +687,11 @@ def main() -> int:
trace.flush()
print(
f"t={record['elapsed_seconds']:.1f}s level={record['level']:.3f} "
f"hit={record['hit']:.3f} pct={record['percentile']:.3f} "
f"gate={record['loudness_gate']:.3f} hit={record['hit']:.3f} "
f"pct={record['percentile']:.3f}/{record['loudness_percentile']:.3f} "
f"bpm={record['bpm']:.1f}/{record['bpm_confidence']:.2f} "
f"f0={record['fundamental_hz']:.1f}",
f"key={record['key_name']} {record['key_mode']} "
f"chord={record['chord_name']} f0={record['fundamental_hz']:.1f}",
flush=True,
)
finally:
@ -406,6 +708,11 @@ def main() -> int:
for proc in (wasapi, ssh):
if proc.poll() is None:
proc.kill()
if asio_reader is not None:
try:
asio_reader.close()
except Exception:
pass
return 0