GameCult/AquaSynth
Watch
9
0
Fork
You've already forked AquaSynth
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

aqua-feel sync: audio-visual onset lag vs perceptual windows #32

Merged
aqua merged 4 commits from aqua/feel-sync into main 2026-11-10 23:42:59 +00:00
Conversation 0 Commits 4 Files changed 10 +727 -1
aqua commented 2026-11-10 23:16:32 +00:00
Member
Copy link

Adds node feel.mjs sync (tools/aqua-feel/sync.mjs, SYNC.md): audio onsets, visual onsets (luma step or motion flux), per-event lag in ms (+ = audio late), verdict against detectability windows (default 25 ms lead / 100 ms lag; BT.1359-1, EBU R37, ATSC IS-191 profiles) judged on a truth interval that accounts for frame quantisation. Input: WAV + per-frame CSV, or a video via ffmpeg (no ffprobe needed). Optional event log.

Three Soul passes (separate context) ran against it; fixed on the branch: bed/DC-fooled onset refinement, understated frame-quantisation uncertainty (now an interval), dense-event mispairing (one-to-one + ambiguous-pairing), unpaired onsets passing, fail-open CLI options, lenient parsers, a video start offset counted twice, resampled-video interval too narrow. Documented limits (pulsing bed, noise gates, the 2.5 Hz cost of the ambiguity rule, wrong-sign reads beyond half the hit spacing) are in SYNC.md.

Measured: 46 checks (37 without ffmpeg), including clips built with ffmpeg whose video or audio stream starts late by a known amount at 15/24/60 fps, and one real headless-Chromium MediaRecorder clip (fixtures/browser-capture.webm: 7 hits, constant +40.5 ms audio late, which is a measurement of that chain, not independently verified). No Playground UI or Perlines capture exists to run. node test.mjs 26/26.

🤖 Generated with Claude Code

Adds `node feel.mjs sync` (tools/aqua-feel/sync.mjs, SYNC.md): audio onsets, visual onsets (luma step or motion flux), per-event lag in ms (+ = audio late), verdict against detectability windows (default 25 ms lead / 100 ms lag; BT.1359-1, EBU R37, ATSC IS-191 profiles) judged on a truth interval that accounts for frame quantisation. Input: WAV + per-frame CSV, or a video via ffmpeg (no ffprobe needed). Optional event log. Three Soul passes (separate context) ran against it; fixed on the branch: bed/DC-fooled onset refinement, understated frame-quantisation uncertainty (now an interval), dense-event mispairing (one-to-one + ambiguous-pairing), unpaired onsets passing, fail-open CLI options, lenient parsers, a video start offset counted twice, resampled-video interval too narrow. Documented limits (pulsing bed, noise gates, the 2.5 Hz cost of the ambiguity rule, wrong-sign reads beyond half the hit spacing) are in SYNC.md. Measured: 46 checks (37 without ffmpeg), including clips built with ffmpeg whose video or audio stream starts late by a known amount at 15/24/60 fps, and one real headless-Chromium MediaRecorder clip (fixtures/browser-capture.webm: 7 hits, constant +40.5 ms audio late, which is a measurement of that chain, not independently verified). No Playground UI or Perlines capture exists to run. `node test.mjs` 26/26. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
Zero-dependency check: audio spectral-flux onsets and visual luma/flux onsets, paired per logged event, lag in ms, verdict against
default/BT.1359-1/EBU R37/ATSC IS-191 windows. Synthetic known-offset fixture and 19 checks. SYNC.md says what the metric cannot tell.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- onset refinement is relative to the steady bed and DC; searched from the flux rise start to the flux peak
- lag is judged on [lag - hop/2, lag + frame period + hop/2], not a point: visual onsets are quantised up to the next frame
- events and onsets pair one-to-one; another onset inside the pairing window is ambiguous-pairing and fails
- unpaired onsets fail unless --allow-unpaired; bad numbers/kinds/profiles/options are errors; parsers refuse unreadable lines
- readVideo: works without ffprobe (ffmpeg -i), resamples VFR to CFR by pts; fixtures/browser-capture.webm + recipe

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- readVideo: ffmpeg only (no ffprobe); stream starts read from the first decoded frame; the padded cfr video is no longer offset twice
  (video starting late read 2x its offset). Known-offset clips built with ffmpeg are now a test.
- ambiguity: a pair is ambiguous when a same-kind neighbour is closer than twice the pairing window (default 200 ms), not when anything is inside it
- audio onsets that refine to the same sample are de-duplicated
- parsers: strict numbers (no null/""/true/hex), header on the first data line, semicolons, trailing commas, column-count consistency; zero windows allowed
- SYNC.md: limits for pulsing beds, noise gates, hit-rate cost of the ambiguity rule

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- a video resampled to constant frame rate gets an interval half a period wider each side (found at 15/24 fps against ffmpeg-built ground truth); tests at 15 and 24 fps
- header lines may contain digits; CR-only line endings split
- SYNC.md: wrong-sign reading beyond half the hit spacing, bed block length, test counts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
aqua merged commit 47dfc540f6 into main 2026-11-10 23:42:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
GameCult/AquaSynth!32
No description provided.