GameCult/Perlines
Watch
9
0
Fork
You've already forked Perlines
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Perlines v2: browser audio visualizer (WebAudio + WebGL2) #1

Merged
aqua merged 7 commits from web-v2 into main 2026-11-10 17:08:39 +00:00
Conversation 1 Commits 7 Files changed 20 +1526 -1
20 changed files with 1526 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

7
README.md
View file

@ -1,6 +1,11 @@
# Perlines
Perlines is the 2012 Aethervis trail idea rebuilt as a Fensalir client: audio becomes a rolling field resource that Fensalir's renderer draws as a storm of curl-noise trails, with no private line renderer in the client.
Perlines turns music into a storm of Perlin-noise trails. Two implementations live here:
- **`web/` (v2, browser):** zero-install WebAudio + WebGL2 visualizer. Static folder, no build, no dependencies. File and microphone input, onset detection, key estimate that tints the scene, three scenes (aurora disk, burning water, vector trails). Entry: `node web/tools/serve.mjs`, then `http://localhost:8080/`; analysis in `web/src/dsp.js`; tests `cd web && npm test`; details in `web/README.md`. **Status: no human has watched it run.** It has synthetic-signal tests and headless-Chrome captures only; no real-GPU frame-time measurement exists yet (`web/BENCH.md`).
- **`src/Perlines/` (Fensalir client, C#):** the 2012 Aethervis trail idea as a client of the Fensalir D3D12 renderer. Described below.
The Fensalir client: Perlines is the 2012 Aethervis trail idea rebuilt as a Fensalir client: audio becomes a rolling field resource that Fensalir's renderer draws as a storm of curl-noise trails, with no private line renderer in the client.
**Valuable here:** the signal path. `src/Perlines/PerlinesSignalProcessor.cs` turns audio into FFT energy, adaptively whitens it per band, detects the key, and projects bins onto musical scale-note lanes. `src/Perlines/PerlinesRuntime.cs` writes one Float32 column per frame into a rolling field resource and hands it to Fensalir (`Perlin4D.cs` and `SmoothNoise.cs` supply the noise). It is a worked example of a Fensalir client that owns a signal and leaves visibility, reservoir reuse, bloom and reconstruction to the renderer.

13
web/BENCH.md Normal file
View file

@ -0,0 +1,13 @@
# Frame-time record
**No GPU measurement exists yet.** The only run so far used headless Chrome with the SwiftShader *software* rasterizer on a shared Linux box. Those numbers say nothing about a real GPU; they show that the scenes run without GL errors and what the JavaScript side costs. The intended target machine (Starfire, GTX 1070) was unreachable (sshd reset every connection) when this was written.
Run 2026-11-05, 1280x720 drawing buffer, synthetic feed, `?bench&sync&w=1280&h=720&ms=4000` (a 1-pixel readback each frame, so the number is cost, not vsync-capped). 16.7 ms is the 60 Hz budget.
| Scene | frame p50 | p95 | max | GPU timer p50 (software) |
|---|---|---|---|---|
| Aurora disk | 21.3 ms | 27.1 | 38.6 | 17.9 |
| Burning water | 46.1 ms | 52.6 | 80.7 | 42.5 |
| Vector trails | 12.2 ms | 17.9 | 21.4 | 9.2 |
JavaScript submit cost (analysis, uploads, draw calls) was 0.8 ms p50 and under 2.7 ms p95 per frame in the vsync run. Software rasterization is CPU-bound by pixels and vertices, so the water scene (full-screen 3 x fbm per pixel) is the one to watch on a real GPU, particularly at 4K or high-dpi. Run `node web/tools/serve.mjs` and open `/?bench&sync&w=1920&h=1080` in Chrome, or use `web/tools/headless.mjs`, on the target to replace this table.

51
web/README.md Normal file
View file

@ -0,0 +1,51 @@
# Perlines v2 (browser)
Zero-install audio visualizer: one static folder, WebAudio for analysis, WebGL2 for drawing, no build step, no dependencies, no network calls.
**Status, plainly: no human has watched this run.** It was written and tested by an agent. The DSP is checked by synthetic-signal tests (below). The renderers were exercised only by headless-Chrome captures and a frame-time script; nobody has listened to music through it, looked at it on a real display, or tried the microphone path. Expect rough looks and tuning mistakes.
## Run
```sh
node web/tools/serve.mjs 8080 # then open http://localhost:8080/
```
Any static server works (`python3 -m http.server -d web`). `localhost` counts as a secure context, so the microphone prompt works; opened from `file://` the mic may be refused. Then: drop an audio file on the page, press Microphone, or press Demo (a silent synthetic 120 bpm spectrum, for looking at scenes without sound). Keys: `1` `2` `3` scenes, `F` fullscreen, `H` HUD.
## What is in it
| Piece | File | What it does |
|---|---|---|
| Onset detector | `web/src/dsp.js` `OnsetDetector` | Four-band spectral flux in dB (40-150, 150-600, 600-2500, 2.5-12 kHz) against a per-bin max of the two previous frames (max-filtered across +-1 bin), per-band median + 8 MAD threshold with a per-band minimum rise, local-peak pick, one refractory period over all bands. Level-independent. Causal: confirms an onset one frame late. Synthetic-drum scores: `tools/onset-bench.mjs`. |
| Key estimate | `web/src/dsp.js` `KeyEstimator` | Peak-based chroma (parabolic-interpolated spectral peaks folded to 12 pitch classes, 4 s half-life) correlated against the 24 Krumhansl-Kessler key profiles. The key picks the scene tint: tonic walks the circle of fifths for hue, mode sets warmth. |
| Analysis state | `web/src/analysis.js` | One object all scenes read: 128 log bands, bass/mid/treble, onset counter, chroma, key, tint. Also `DemoFeed`. |
| Perlin noise | `web/src/noise.js` | Classic 3D Perlin and a 2D curl field (divergence-free flow) for the CPU-side paths. |
| Scenes | `web/src/scenes/` | Aurora disk, burning water, vector trails (below). |
| Persistence | `web/src/gl.js` `Feedback` | Two render targets; each frame fades the previous one and the scene adds on top. |
### Scenes
- **Aurora disk** (`aurora.js`): spectrum-to-mesh extrusion. A 112 x 224 polar mesh; angle is frequency (mirrored), radius is age, vertex height is the band level from a 128-row spectrum-history texture, modulated by noise. Onsets pulse and spin it.
- **Burning water** (`water.js`): a ripple plane lit in flame colours, plus an onset-driven spray. Particles are stateless: onsets write `(birth, seed, strength, lane)` into a ring buffer, and the vertex shader computes every position from age.
- **Vector trails** (`trails.js`): 96 heads, one per spectrum lane, ride a Perlin curl field; each leaves a 72-point polyline. An onset kicks the speed and respawns a spray of heads from the centre.
## Tests and measurements
```sh
cd web && npm test # node --test test/dsp.test.mjs test/onset.test.mjs (node >= 20, nothing to install)
cd web && npm run bench:onsets # full synthetic-drum corpus, before/after table, about 30 s
```
Covers (dsp.test): ten synthetic clicks found at two hop sizes (11.6 ms and 16.7 ms) with no false positives; silence, steady noise and a sustained tone give no onsets; a 20 ms doublet merges under the refractory period; the key estimate reads C major, A minor and (transposed) B minor from synthesized chord progressions. Fixtures are deliberately plain: the estimate is known to confuse relative major/minor when a clip leans on the dominant, and one test fixture was rewritten to put its tonic in more chords (the estimator was not tuned to the fixture).
Covers (onset.test, synthetic drum corpus from `tools/onset-bench.mjs`: kick/snare/hat patterns at 90/120/150 bpm, three levels, bass+noise beds, 30 fps and 512-sample hops): hit rate, false positives and timing floors for the shipped detector, a margin over the frozen legacy detector, level independence, determinism. Synthetic only; the corpus was adjusted twice while building it (coincident kick+snare counted as one onset; voice tails faded so truncation clicks are not onsets), recorded in the commit message.
Frame time: the HUD shows rolling p50/p95 of requestAnimationFrame intervals against the 16.7 ms budget and CPU submit time. `?bench` runs the three scenes on the synthetic feed and prints JSON (`?w=1920&h=1080` fixes the drawing-buffer size, `&sync` reads a pixel back after each frame so the measured frame time is real cost, not capped by vsync, `&ms=` sets seconds per scene). `web/tools/headless.mjs` drives headless Chrome over CDP to collect it. Results of the last run are in `web/BENCH.md`.
## Provenance
The ideas come from the 2013 Unity "Aethervis" project (Perlin-noise paths with spray and trails; spectrum-driven mesh extrusion; the Aurora, BurningWater and vectortrails scenes) and from the author's analysis-vocabulary notes (onsets, chroma, key). No source, shader or asset was copied; this is fresh code under the repository's licence. The analysis is written from published techniques (spectral flux, Krumhansl-Kessler profiles) and does not port Vamp/aubio/QM-DSP code, which is GPL-family.
## Not done
Known weakness: on the synthetic demo feed at a simulated 30 fps the detector finds only about half the kicks (60 fps and 11.6 ms hops are fine); it has never been run on real music. Not built: beat/tempo tracking, per-note lanes (the C# client has them), audio-worklet capture (onsets are quantized to the render frame rate), mobile tuning, a quality auto-scaler.

42
web/index.html Normal file
View file

@ -0,0 +1,42 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Perlines</title>
<style>
html, body { margin: 0; height: 100%; background: #000; color: #cfd; font: 13px/1.4 ui-monospace, Menlo, Consolas, monospace; overflow: hidden; }
canvas { position: fixed; inset: 0; width: 100%; height: 100%; display: block; }
#ui { position: fixed; left: 0; right: 0; top: 0; padding: 10px 12px; display: flex; gap: 8px; flex-wrap: wrap; align-items: center;
background: linear-gradient(#000a, #0000); transition: opacity .4s; }
#ui.idle { opacity: 0; pointer-events: none; }
button, label.btn { background: #0008; color: #cfd; border: 1px solid #4a6; padding: 5px 10px; font: inherit; cursor: pointer; border-radius: 3px; }
button:hover, label.btn:hover, button.on { background: #4a64; }
#hud { position: fixed; left: 12px; bottom: 10px; white-space: pre; opacity: .85; text-shadow: 0 0 4px #000; }
#msg { position: fixed; left: 50%; top: 50%; transform: translate(-50%,-50%); text-align: center; opacity: .8; max-width: 36em; pointer-events: none; }
#msg.hide { display: none; }
audio { position: fixed; right: 12px; bottom: 10px; width: 280px; height: 30px; opacity: .6; }
audio:not([src]) { display: none; }
#bench { position: fixed; inset: 0; margin: 0; padding: 12px; background: #000c; overflow: auto; display: none; }
</style>
</head>
<body>
<canvas id="c"></canvas>
<div id="ui">
<label class="btn">Open audio file<input id="file" type="file" accept="audio/*" hidden></label>
<button id="mic">Microphone</button>
<button id="demo">Demo (silent, synthetic)</button>
<span style="flex:1"></span>
<button data-scene="0">1 Aurora disk</button>
<button data-scene="1">2 Burning water</button>
<button data-scene="2">3 Vector trails</button>
<button id="full">Fullscreen (F)</button>
</div>
<div id="msg"><b>Perlines</b><br>Drop an audio file anywhere, or use the microphone.<br>Keys: 1 2 3 scenes, F fullscreen, H HUD.<br>
<span style="opacity:.6">Nothing leaves your browser: audio is analysed locally.</span></div>
<div id="hud"></div>
<audio id="audio" controls></audio>
<pre id="bench"></pre>
<script type="module" src="src/main.js"></script>
</body>
</html>

13
web/package.json Normal file
View file

@ -0,0 +1,13 @@
{
"name": "perlines-web",
"version": "2.0.0",
"description": "Perlines v2: zero-install browser audio visualizer (WebAudio + WebGL2)",
"type": "module",
"license": "MIT",
"private": true,
"scripts": {
"test": "node --test test/dsp.test.mjs test/onset.test.mjs",
"bench:onsets": "node tools/onset-bench.mjs",
"serve": "node tools/serve.mjs"
}
}

88
web/src/analysis.js Normal file
View file

@ -0,0 +1,88 @@
// Turns a frame of dB spectrum into the shared state every scene reads: log-spaced smoothed bands,
// band levels, onset events, chroma, key and the tint colour derived from the key.
import { dbToLinear, OnsetDetector, KeyEstimator } from './dsp.js';
export const NB = 128; // log-spaced display bands
export const HIST = 128; // history rows (spectrum texture height)
function hslToRgb(h, s, l) {
const f = n => { const k = (n + h * 12) % 12, a = s * Math.min(l, 1 - l); return l - a * Math.max(-1, Math.min(k - 3, 9 - k, 1)); };
return [f(0), f(8), f(4)];
}
export class Analysis {
constructor(sampleRate, binCount) {
this.sampleRate = sampleRate; this.binCount = binCount;
this.lin = new Float32Array(binCount);
this.onsets = new OnsetDetector({ binHz: sampleRate / 2 / binCount });
this.keyEst = new KeyEstimator(sampleRate, binCount);
this.bands = new Float32Array(NB);
this.row = new Uint8Array(NB); // newest row, as texture bytes
this.bass = 0; this.mid = 0; this.treble = 0; this.level = 0;
this.onset = { count: 0, strength: 0, time: -1 }; // count increments per onset; scenes diff it
this.flux = 0;
this.key = this.keyEst.key;
this.hue = 0.6; this.tint = hslToRgb(0.6, 0.7, 0.55);
this.time = 0; this.dt = 1 / 60;
// band -> bin edges, log spaced 40 Hz .. 16 kHz
const hz = sampleRate / 2 / binCount, lo = 40, hi = Math.min(16000, sampleRate / 2 - hz);
this.edges = Array.from({ length: NB + 1 }, (_, i) => Math.min(binCount - 1, Math.round(lo * Math.pow(hi / lo, i / NB) / hz)));
}
/** db: Float32Array in decibels (AnalyserNode.getFloatFrequencyData); t: seconds. */
update(db, t) {
const dt = Math.min(0.1, Math.max(1e-3, t - this.time || 1 / 60));
this.time = t; this.dt = dt;
const lin = dbToLinear(db, this.lin);
const o = this.onsets.push(lin, t);
this.flux = this.onsets.flux;
if (o) { this.onset.count++; this.onset.strength = o.strength; this.onset.time = t; }
// bands: peak of dB inside each band, mapped -90..-25 dB to 0..1, fast attack / slow release
let b = 0, m = 0, tr = 0, nb = 0, nm = 0, nt = 0;
for (let i = 0; i < NB; i++) {
const a = this.edges[i], z = Math.max(a + 1, this.edges[i + 1]);
let pk = -160;
for (let k = a; k < z; k++) if (db[k] > pk) pk = db[k];
const v = Math.max(0, Math.min(1, (pk + 90) / 65));
const cur = this.bands[i];
this.bands[i] = v > cur ? cur + (v - cur) * 0.8 : cur + (v - cur) * (1 - Math.pow(0.02, dt));
this.row[i] = Math.round(this.bands[i] * 255);
if (i < NB * 0.2) { b += this.bands[i]; nb++; } else if (i < NB * 0.6) { m += this.bands[i]; nm++; } else { tr += this.bands[i]; nt++; }
}
this.bass = b / nb; this.mid = m / nm; this.treble = tr / nt;
this.level = (this.bass + this.mid + this.treble) / 3;
const key = this.keyEst.push(lin, dt);
this.key = key;
// tint: hue walks the circle of fifths so related keys sit near each other; confidence gates the move
const target = ((key.tonic * 7) % 12) / 12;
let d = target - this.hue; d -= Math.round(d);
this.hue = (this.hue + d * Math.min(1, dt * (0.15 + key.confidence * 0.8)) + 1) % 1;
const major = key.mode === 'major';
this.tint = hslToRgb(this.hue, major ? 0.8 : 0.6, major ? 0.58 : 0.5);
return this;
}
}
/** Synthetic spectrum for demo mode and benchmarks: a 120 bpm kick/hat pattern over an A-minor pad. */
export class DemoFeed {
constructor(sampleRate, binCount) { this.sr = sampleRate; this.n = binCount; this.db = new Float32Array(binCount); }
frame(t) {
const { db, n, sr } = this, hz = sr / 2 / n, beat = 0.5;
const ph = t % beat, kick = Math.exp(-ph * 9), hat = Math.exp(-((t + beat / 2) % beat) * 40) * ((Math.floor(t / (beat / 2)) % 2) ? 1 : 0.4);
const snare = Math.floor(t / beat) % 2 ? Math.exp(-ph * 14) : 0;
const chord = [220, 261.63, 329.63, 440, 523.25, 659.25];
for (let k = 0; k < n; k++) {
const f = k * hz + 1;
let v = -105 + 12 * Math.sin(k * 12.9898 + t * 3) * 0.3;
v += 78 * kick * Math.exp(-Math.pow(Math.log2(f / 55), 2) * 1.5);
v += 62 * snare * Math.exp(-Math.pow(Math.log2(f / 1800), 2) * 0.35);
v += 55 * hat * Math.exp(-Math.pow(Math.log2(f / 9000), 2) * 0.5);
for (const c of chord) v += 55 * Math.exp(-Math.pow((f - c) / (hz * 0.8 + c * 0.01), 2));
db[k] = v;
}
return db;
}
}

50
web/src/audio.js Normal file
View file

@ -0,0 +1,50 @@
// Audio inputs. All go through one AnalyserNode; the analysis code reads dB frames from it.
// File playback uses an <audio> element (streams, no full decode); the mic is never routed to the speakers.
export class AudioInput {
constructor() {
this.ctx = null; this.analyser = null; this.db = null; this.source = null; this.stream = null; this.mode = 'none';
this.el = null;
}
async ensure() {
if (!this.ctx) {
const AC = window.AudioContext || window.webkitAudioContext;
this.ctx = new AC({ latencyHint: 'interactive' });
this.analyser = this.ctx.createAnalyser();
this.analyser.fftSize = 2048;
this.analyser.smoothingTimeConstant = 0; // onset detection needs the raw frame, scenes smooth for themselves
this.analyser.minDecibels = -120; this.analyser.maxDecibels = 0;
this.db = new Float32Array(this.analyser.frequencyBinCount);
}
if (this.ctx.state !== 'running') await this.ctx.resume();
}
disconnect() {
if (this.source) { try { this.source.disconnect(); } catch {} this.source = null; }
if (this.stream) { this.stream.getTracks().forEach(t => t.stop()); this.stream = null; }
if (this.el) { this.el.pause(); }
}
async useFile(file, audioEl) {
await this.ensure(); this.disconnect();
this.el = audioEl;
if (this._url) URL.revokeObjectURL(this._url);
this._url = URL.createObjectURL(file);
audioEl.src = this._url;
if (!this._elSource) this._elSource = this.ctx.createMediaElementSource(audioEl);
this.source = this._elSource;
this.source.connect(this.analyser);
this.analyser.connect(this.ctx.destination);
await audioEl.play().catch(() => {}); // a click already happened; if play is refused the controls are visible
this.mode = 'file';
}
async useMic() {
await this.ensure(); this.disconnect();
try { this.analyser.disconnect(this.ctx.destination); } catch {}
this.stream = await navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false } });
this.source = this.ctx.createMediaStreamSource(this.stream);
this.source.connect(this.analyser);
this.mode = 'mic';
}
stop() { this.disconnect(); this.mode = 'none'; }
get sampleRate() { return this.ctx ? this.ctx.sampleRate : 48000; }
frame() { this.analyser.getFloatFrequencyData(this.db); return this.db; }
}

202
web/src/dsp.js Normal file
View file

@ -0,0 +1,202 @@
// Perlines v2 analysis: FFT, onset detector, chroma + key estimate.
// Pure ES module, no DOM, no dependencies: the browser and the node tests load the same file.
// Written from the published ideas (spectral flux with a max-filtered reference, adaptive
// median threshold, Krumhansl-Schmuckler key profiles), not from any GPL implementation.
/** In-place radix-2 complex FFT. re/im are Float32Array|Float64Array of power-of-two length. */
export function fft(re, im) {
const n = re.length;
for (let i = 1, j = 0; i < n; i++) {
let bit = n >> 1;
for (; j & bit; bit >>= 1) j ^= bit;
j ^= bit;
if (i < j) {
let t = re[i]; re[i] = re[j]; re[j] = t;
t = im[i]; im[i] = im[j]; im[j] = t;
}
}
for (let len = 2; len <= n; len <<= 1) {
const ang = -2 * Math.PI / len, wr = Math.cos(ang), wi = Math.sin(ang);
for (let i = 0; i < n; i += len) {
let cr = 1, ci = 0;
for (let k = 0; k < len >> 1; k++) {
const a = i + k, b = a + (len >> 1);
const xr = re[b] * cr - im[b] * ci, xi = re[b] * ci + im[b] * cr;
re[b] = re[a] - xr; im[b] = im[a] - xi;
re[a] += xr; im[a] += xi;
const t = cr * wr - ci * wi; ci = cr * wi + ci * wr; cr = t;
}
}
}
}
/** Hann-windowed magnitude spectrum (n/2 bins, scaled so a full-scale sine reads ~0.5). */
export function magnitudeSpectrum(samples, offset, n, out) {
const re = new Float64Array(n), im = new Float64Array(n);
for (let i = 0; i < n; i++) {
const w = 0.5 - 0.5 * Math.cos(2 * Math.PI * i / (n - 1));
re[i] = (samples[offset + i] || 0) * w;
}
fft(re, im);
const m = out || new Float32Array(n >> 1);
for (let k = 0; k < n >> 1; k++) m[k] = Math.hypot(re[k], im[k]) * 2 / n;
return m;
}
/** Decibel array (as WebAudio getFloatFrequencyData gives) to linear magnitude, in place into out. */
export function dbToLinear(db, out) {
for (let i = 0; i < db.length; i++) out[i] = Math.pow(10, Math.max(db[i], -160) / 20);
return out;
}
/**
* Onset detector: band-split spectral flux in decibels. Each band has its own flux (mean positive dB rise
* against a max-filtered previous frame, which ignores vibrato wobble) and its own adaptive threshold
* (median + k * MAD over the last second), so a hat is judged against the hats and a kick against the
* kick, and the level of the track does not matter. Causal: a flux peak on frame n-1 is confirmed when
* frame n arrives. Bands are merged behind one refractory period.
*
* push(mags, t) -> null | { time, strength, band } (time = the onset frame's own timestamp, seconds)
*/
export class OnsetDetector {
constructor(opts = {}) {
this.binHz = opts.binHz ?? 48000 / 2048;
this.edgesHz = opts.edgesHz ?? [40, 150, 600, 2500, 12000];
this.maxBin = opts.maxBin ?? 0; // 0 = all bins
this.floorDb = opts.floorDb ?? -110; // magnitudes below this are silence
this.history = opts.history ?? 1.0; // seconds of flux per band for the threshold
this.k = opts.k ?? 8; // threshold = median + k * MAD (MAD scaled to sigma)
this.minRise = opts.minRise ?? [6, 3, 2, 2]; // and at least this many dB mean rise in the band (per band: the few bass bins are noisy)
this.refractory = opts.refractory ?? 0.05; // seconds, all bands
this.edges = null;
this.lag = opts.lag ?? 2; // reference = per-bin max over this many previous frames (1 or 2)
this.prev = null; this.prev2 = null;
this.hist = null; // per band: { t[], f[] }
this.f1 = null; this.f2 = null; this.t1 = 0; // per band flux[n-1], flux[n-2], time[n-1]
this.last = -1e9;
this.flux = 0; // largest band flux this frame, for display
}
push(mags, t) {
const n = Math.min(this.maxBin || mags.length, mags.length);
if (!this.edges) {
this.edges = this.edgesHz.map(h => Math.min(n, Math.max(1, Math.round(h / this.binHz))));
for (let i = 1; i < this.edges.length; i++) this.edges[i] = Math.min(n, Math.max(this.edges[i], this.edges[i - 1] + 1));
const nb = this.edges.length - 1;
this.hist = Array.from({ length: nb }, () => ({ t: [], f: [] }));
this.f1 = new Float64Array(nb); this.f2 = new Float64Array(nb);
}
const nb = this.edges.length - 1, lo = Math.pow(10, this.floorDb / 20);
const cur = new Float32Array(n);
for (let k = 0; k < n; k++) cur[k] = 20 * Math.log10(Math.max(mags[k], lo));
const flux = new Float64Array(nb);
if (this.prev) {
const p = this.prev, q = this.lag > 1 ? this.prev2 : null;
for (let b = 0; b < nb; b++) {
const a = this.edges[b], z = this.edges[b + 1];
let sum = 0;
for (let k = a; k < z; k++) {
let ref = Math.max(p[k - 1] ?? this.floorDb, p[k], p[k + 1] ?? this.floorDb);
if (q) ref = Math.max(ref, q[k - 1] ?? this.floorDb, q[k], q[k + 1] ?? this.floorDb);
const d = cur[k] - ref;
if (d > 0) sum += d;
}
flux[b] = sum / (z - a);
}
}
this.prev2 = this.prev; this.prev = cur;
this.flux = Math.max(...flux);
// per band: the previous frame is a candidate when it is a local flux maximum over the threshold
let best = null;
for (let b = 0; b < nb; b++) {
const h = this.hist[b], cand = this.f1[b];
if (h.f.length >= 8 && cand > this.f2[b] && cand >= flux[b] && cand >= this.minRise[b]) {
const sorted = h.f.slice().sort((x, y) => x - y), med = sorted[sorted.length >> 1];
const dev = h.f.map(v => Math.abs(v - med)).sort((x, y) => x - y), mad = dev[dev.length >> 1] * 1.4826;
if (cand > med + this.k * mad && (!best || cand > best.flux)) best = { flux: cand, band: b };
}
h.t.push(t); h.f.push(flux[b]);
while (h.t.length > 2 && t - h.t[0] > this.history) { h.t.shift(); h.f.shift(); }
}
let out = null;
if (best && this.t1 - this.last >= this.refractory) {
out = { time: this.t1, strength: Math.min(1, best.flux / 15), band: best.band };
this.last = this.t1;
}
this.f2.set(this.f1); this.f1.set(flux); this.t1 = t;
return out;
}
}
const NOTE_NAMES = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B'];
export { NOTE_NAMES };
// Krumhansl-Kessler probe-tone profiles (published psychology data).
const MAJOR = [6.35, 2.23, 3.48, 2.33, 4.38, 4.09, 2.52, 5.19, 2.39, 3.66, 2.29, 2.88];
const MINOR = [6.33, 2.68, 3.52, 5.38, 2.60, 3.53, 2.54, 4.75, 3.98, 2.69, 3.34, 3.17];
function corr(a, b, rot) {
let ma = 0, mb = 0;
for (let i = 0; i < 12; i++) { ma += a[(i + rot) % 12]; mb += b[i]; }
ma /= 12; mb /= 12;
let num = 0, da = 0, db = 0;
for (let i = 0; i < 12; i++) {
const x = a[(i + rot) % 12] - ma, y = b[i] - mb;
num += x * y; da += x * x; db += y * y;
}
return da > 1e-12 && db > 1e-12 ? num / Math.sqrt(da * db) : 0;
}
/**
* Chroma accumulator + key estimate. Each frame folds spectral peaks between lowHz and highHz into
* 12 pitch classes, a slow exponential average (halfLife seconds) smooths them, and the key is the
* best of 24 rotations of the Krumhansl profiles by correlation.
*/
export class KeyEstimator {
constructor(sampleRate, binCount, opts = {}) {
this.halfLife = opts.halfLife ?? 4;
this.lowHz = opts.lowHz ?? 80;
this.highHz = opts.highHz ?? 2000;
this.chroma = new Float32Array(12);
this.key = { tonic: 0, mode: 'major', confidence: 0, name: 'C' };
this.hzPerBin = sampleRate / 2 / binCount;
this.lo = Math.max(2, Math.floor(this.lowHz / this.hzPerBin));
this.hi = Math.min(binCount - 2, Math.ceil(this.highHz / this.hzPerBin));
}
// Peak-based: bins are wider than a semitone below ~250 Hz, so each spectral peak is refined by
// parabolic interpolation and its energy goes to the pitch class of the refined frequency.
push(mags, dt) {
const frame = new Float32Array(12);
let top = 0;
for (let k = this.lo; k <= this.hi; k++) if (mags[k] > top) top = mags[k];
const floor = top * 0.05;
for (let k = this.lo; k <= this.hi; k++) {
const m = mags[k];
if (m < floor || m < 1e-5 || m < mags[k - 1] || m <= mags[k + 1]) continue;
const a = Math.log(mags[k - 1] + 1e-9), b = Math.log(m + 1e-9), c = Math.log(mags[k + 1] + 1e-9);
const d = 0.5 * (a - c) / (a - 2 * b + c);
const midi = 69 + 12 * Math.log2((k + d) * this.hzPerBin / 440);
const near = Math.round(midi);
frame[((near % 12) + 12) % 12] += Math.sqrt(m) * (1 - Math.min(0.5, Math.abs(midi - near))); // sqrt tames loud-bass dominance
}
const a = Math.pow(0.5, dt / this.halfLife);
for (let i = 0; i < 12; i++) this.chroma[i] = this.chroma[i] * a + frame[i] * (1 - a);
this.key = this.estimate();
return this.key;
}
estimate() {
let best = { tonic: 0, mode: 'major', score: -2 }, second = -2;
for (let r = 0; r < 12; r++) {
for (const [mode, prof] of [['major', MAJOR], ['minor', MINOR]]) {
const s = corr(this.chroma, prof, r);
if (s > best.score) { second = best.score; best = { tonic: r, mode, score: s }; }
else if (s > second) second = s;
}
}
const energy = this.chroma.reduce((x, y) => x + y, 0);
const confidence = energy < 1e-4 ? 0 : Math.max(0, Math.min(1, (best.score - second) * 4 + best.score * 0.3));
return { tonic: best.tonic, mode: best.mode, confidence, name: NOTE_NAMES[best.tonic] + (best.mode === 'minor' ? 'm' : '') };
}
}

119
web/src/gl.js Normal file
View file

@ -0,0 +1,119 @@
// Small WebGL2 helpers: programs, a fullscreen triangle, a feedback (persistence) target, mat4.
export function program(gl, vs, fs) {
const mk = (type, src) => {
const s = gl.createShader(type);
gl.shaderSource(s, src); gl.compileShader(s);
if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s) + '\n' + src);
return s;
};
const p = gl.createProgram();
gl.attachShader(p, mk(gl.VERTEX_SHADER, vs)); gl.attachShader(p, mk(gl.FRAGMENT_SHADER, fs));
gl.linkProgram(p);
if (!gl.getProgramParameter(p, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(p));
const u = {}, n = gl.getProgramParameter(p, gl.ACTIVE_UNIFORMS);
for (let i = 0; i < n; i++) { const a = gl.getActiveUniform(p, i), name = a.name.replace(/\[0\]$/, ''); u[name] = gl.getUniformLocation(p, a.name); }
return { p, u };
}
export const GLSL_HEAD = '#version 300 es\nprecision highp float;\n';
// Hash-based gradient-free value noise + fbm, enough for shader-side flow; the CPU paths use real Perlin.
export const GLSL_NOISE = `
float hash31(vec3 p){ p = fract(p * 0.1031); p += dot(p, p.zyx + 31.32); return fract((p.x + p.y) * p.z); }
float vnoise(vec3 x){
vec3 i = floor(x), f = fract(x); f = f*f*(3.0-2.0*f);
return mix(mix(mix(hash31(i), hash31(i+vec3(1,0,0)), f.x), mix(hash31(i+vec3(0,1,0)), hash31(i+vec3(1,1,0)), f.x), f.y),
mix(mix(hash31(i+vec3(0,0,1)), hash31(i+vec3(1,0,1)), f.x), mix(hash31(i+vec3(0,1,1)), hash31(i+vec3(1,1,1)), f.x), f.y), f.z);
}
float fbm(vec3 p){ float a = 0.5, s = 0.0; for (int i = 0; i < 4; i++){ s += a * vnoise(p); p = p * 2.03 + 11.7; a *= 0.5; } return s; }
`;
export function fullscreen(gl) {
const vao = gl.createVertexArray(); gl.bindVertexArray(vao);
const b = gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER, b);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
gl.enableVertexAttribArray(0); gl.vertexAttribPointer(0, 2, gl.FLOAT, false, 0, 0);
gl.bindVertexArray(null);
return vao;
}
/**
* Persistence: scenes draw into one of two textures; each frame the previous texture is copied
* forward multiplied by a decay (trails), the scene adds on top, and the result is blitted to screen.
*/
export class Feedback {
constructor(gl) {
this.gl = gl; this.w = 0; this.h = 0; this.i = 0; this.tex = []; this.fbo = [];
this.tri = fullscreen(gl);
this.copy = program(gl, GLSL_HEAD + 'layout(location=0) in vec2 p; out vec2 uv; void main(){ uv = p*0.5+0.5; gl_Position = vec4(p,0,1); }',
GLSL_HEAD + 'uniform sampler2D t; uniform float decay; uniform vec2 px; in vec2 uv; out vec4 o; void main(){ vec3 c = texture(t, uv).rgb; vec3 g = texture(t, uv + px).rgb * 0.5 + texture(t, uv - px).rgb * 0.5; o = vec4(mix(c, g, 0.35) * decay, 1.0); }');
this.blit = program(gl, GLSL_HEAD + 'layout(location=0) in vec2 p; out vec2 uv; void main(){ uv = p*0.5+0.5; gl_Position = vec4(p,0,1); }',
GLSL_HEAD + 'uniform sampler2D t; in vec2 uv; out vec4 o; void main(){ vec3 c = texture(t, uv).rgb; c = 1.0 - exp(-c * 1.4); o = vec4(pow(c, vec3(0.9)), 1.0); }');
}
resize(w, h) {
const gl = this.gl;
if (w === this.w && h === this.h) return;
this.w = w; this.h = h;
for (let k = 0; k < 2; k++) {
if (this.tex[k]) { gl.deleteTexture(this.tex[k]); gl.deleteFramebuffer(this.fbo[k]); }
const t = gl.createTexture(); gl.bindTexture(gl.TEXTURE_2D, t);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA8, w, h, 0, gl.RGBA, gl.UNSIGNED_BYTE, null);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR); gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE); gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
const f = gl.createFramebuffer(); gl.bindFramebuffer(gl.FRAMEBUFFER, f);
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, t, 0);
gl.clearColor(0, 0, 0, 1); gl.clear(gl.COLOR_BUFFER_BIT);
this.tex[k] = t; this.fbo[k] = f;
}
gl.bindFramebuffer(gl.FRAMEBUFFER, null);
}
/** Fade the previous frame into the next target and bind it; the scene then draws additively. */
begin(decay) {
const gl = this.gl, prev = this.i, next = 1 - this.i;
gl.bindFramebuffer(gl.FRAMEBUFFER, this.fbo[next]);
gl.viewport(0, 0, this.w, this.h);
gl.disable(gl.BLEND); gl.disable(gl.DEPTH_TEST);
gl.useProgram(this.copy.p);
gl.activeTexture(gl.TEXTURE0); gl.bindTexture(gl.TEXTURE_2D, this.tex[prev]);
gl.uniform1i(this.copy.u.t, 0); gl.uniform1f(this.copy.u.decay, decay); gl.uniform2f(this.copy.u.px, 0.6 / this.w, 0.6 / this.h);
gl.bindVertexArray(this.tri); gl.drawArrays(gl.TRIANGLES, 0, 3);
this.i = next;
gl.enable(gl.BLEND); gl.blendFunc(gl.ONE, gl.ONE);
}
present() {
const gl = this.gl;
gl.bindFramebuffer(gl.FRAMEBUFFER, null);
gl.viewport(0, 0, this.w, this.h);
gl.disable(gl.BLEND); gl.disable(gl.DEPTH_TEST);
gl.useProgram(this.blit.p);
gl.activeTexture(gl.TEXTURE0); gl.bindTexture(gl.TEXTURE_2D, this.tex[this.i]);
gl.uniform1i(this.blit.u.t, 0);
gl.bindVertexArray(this.tri); gl.drawArrays(gl.TRIANGLES, 0, 3);
}
clear() {
const gl = this.gl;
for (const f of this.fbo) { gl.bindFramebuffer(gl.FRAMEBUFFER, f); gl.clearColor(0, 0, 0, 1); gl.clear(gl.COLOR_BUFFER_BIT); }
gl.bindFramebuffer(gl.FRAMEBUFFER, null);
}
}
// column-major mat4
export const mat4 = {
perspective(fovy, aspect, n, f) {
const t = 1 / Math.tan(fovy / 2), m = new Float32Array(16);
m[0] = t / aspect; m[5] = t; m[10] = (f + n) / (n - f); m[11] = -1; m[14] = 2 * f * n / (n - f);
return m;
},
mul(a, b) {
const o = new Float32Array(16);
for (let c = 0; c < 4; c++) for (let r = 0; r < 4; r++) {
let s = 0; for (let k = 0; k < 4; k++) s += a[k * 4 + r] * b[c * 4 + k];
o[c * 4 + r] = s;
}
return o;
},
rotX(a) { const c = Math.cos(a), s = Math.sin(a); return new Float32Array([1, 0, 0, 0, 0, c, s, 0, 0, -s, c, 0, 0, 0, 0, 1]); },
rotY(a) { const c = Math.cos(a), s = Math.sin(a); return new Float32Array([c, 0, -s, 0, 0, 1, 0, 0, s, 0, c, 0, 0, 0, 0, 1]); },
translate(x, y, z) { return new Float32Array([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, x, y, z, 1]); },
};

157
web/src/main.js Normal file
View file

@ -0,0 +1,157 @@
import { AudioInput } from './audio.js';
import { Analysis, DemoFeed, NB, HIST } from './analysis.js';
import { Feedback } from './gl.js';
import { AuroraDisk } from './scenes/aurora.js';
import { BurningWater } from './scenes/water.js';
import { VectorTrails } from './scenes/trails.js';
const q = new URLSearchParams(location.search);
const $ = id => document.getElementById(id);
const canvas = $('c'), hudEl = $('hud'), msg = $('msg'), ui = $('ui');
const gl = canvas.getContext('webgl2', { antialias: false, alpha: false, powerPreference: 'high-performance' });
if (!gl) { msg.textContent = 'WebGL2 is not available in this browser.'; throw new Error('no webgl2'); }
const bench = q.has('bench');
const fixed = q.has('w') ? [+q.get('w'), +q.get('h') || 720] : null;
const dpr = Math.min(window.devicePixelRatio || 1, +(q.get('dpr') || 1.5));
// spectrum history: NB x HIST, one byte per texel, newest row written each frame
const hist = { tex: gl.createTexture(), row: 0, newest: 0 };
gl.bindTexture(gl.TEXTURE_2D, hist.tex);
gl.texStorage2D(gl.TEXTURE_2D, 1, gl.R8, NB, HIST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR); gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE); gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.REPEAT);
gl.pixelStorei(gl.UNPACK_ALIGNMENT, 1);
const fb = new Feedback(gl);
const scenes = [new AuroraDisk(gl), new BurningWater(gl), new VectorTrails(gl)];
let sceneIdx = Math.min(2, Math.max(0, +(q.get('scene') || 0)));
const audio = new AudioInput();
let analysis = new Analysis(48000, 1024);
let demo = null;
const BINS = 1024;
function setScene(i) {
sceneIdx = i; fb.clear();
document.querySelectorAll('[data-scene]').forEach(b => b.classList.toggle('on', +b.dataset.scene === i));
}
function startDemo() {
audio.stop(); demo = new DemoFeed(48000, BINS); analysis = new Analysis(48000, BINS); msg.classList.add('hide');
}
async function useFile(f) {
if (!f) return;
await audio.useFile(f, $('audio')); demo = null; analysis = new Analysis(audio.sampleRate, audio.analyser.frequencyBinCount); msg.classList.add('hide');
}
async function useMic() {
try { await audio.useMic(); } catch (e) { msg.classList.remove('hide'); msg.textContent = 'Microphone unavailable: ' + e.message; return; }
demo = null; analysis = new Analysis(audio.sampleRate, audio.analyser.frequencyBinCount); msg.classList.add('hide');
}
$('file').onchange = e => useFile(e.target.files[0]);
$('mic').onclick = useMic;
$('demo').onclick = startDemo;
$('full').onclick = () => document.fullscreenElement ? document.exitFullscreen() : document.documentElement.requestFullscreen();
document.querySelectorAll('[data-scene]').forEach(b => b.onclick = () => setScene(+b.dataset.scene));
addEventListener('keydown', e => {
if (e.key >= '1' && e.key <= '3') setScene(+e.key - 1);
else if (e.key === 'f' || e.key === 'F') $('full').click();
else if (e.key === 'h' || e.key === 'H') hudEl.style.display = hudEl.style.display === 'none' ? '' : 'none';
});
addEventListener('dragover', e => e.preventDefault());
addEventListener('drop', e => { e.preventDefault(); useFile(e.dataTransfer.files[0]); });
let idleT = 0;
addEventListener('pointermove', () => { ui.classList.remove('idle'); idleT = performance.now(); });
function resize() {
const w = fixed ? fixed[0] : Math.max(2, Math.floor(canvas.clientWidth * dpr));
const h = fixed ? fixed[1] : Math.max(2, Math.floor(canvas.clientHeight * dpr));
if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h; }
fb.resize(w, h);
}
// ---- frame-time bookkeeping: rolling window for the HUD, full record for the bench
const ring = new Float32Array(240); let ri = 0, rn = 0, last = 0, cpuMs = 0, gpuMs = 0;
const timer = gl.getExtension('EXT_disjoint_timer_query_webgl2');
let pendingQ = [];
function pct(a, n, p) { const s = Array.from(a.subarray(0, n)).sort((x, y) => x - y); return s[Math.min(n - 1, Math.floor(p * n))] || 0; }
const record = { scenes: [] };
let benchScene = -1, benchStart = 0, benchDt = [], benchCpu = [], benchGpu = [];
const BENCH_MS = +(q.get('ms') || 6000), WARM_MS = 1000;
const sync = q.has('sync'), syncPx = new Uint8Array(4);
function frame(now) {
const t0 = performance.now();
resize();
const t = now / 1000;
let db;
if (demo) db = demo.frame(t);
else if (audio.mode !== 'none') db = audio.frame();
else { db = new Float32Array(BINS).fill(-140); }
const A = analysis.update(db, t);
hist.row = (hist.row + 1) % HIST; hist.newest = (hist.row + 0.5) / HIST;
gl.bindTexture(gl.TEXTURE_2D, hist.tex);
gl.texSubImage2D(gl.TEXTURE_2D, 0, 0, hist.row, NB, 1, gl.RED, gl.UNSIGNED_BYTE, A.row);
const sc = scenes[sceneIdx];
let qy = null;
if (timer && bench) { qy = gl.createQuery(); gl.beginQuery(timer.TIME_ELAPSED_EXT, qy); }
fb.begin(sc.persistence);
sc.draw(gl, A, canvas.width, canvas.height, hist);
fb.present();
if (qy) { gl.endQuery(timer.TIME_ELAPSED_EXT); pendingQ.push(qy); }
if (sync) gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, syncPx); // gl.finish() does not block under ANGLE; reading a pixel does
cpuMs = performance.now() - t0;
const dt = last ? now - last : 16.7; last = now;
if (!document.hidden) { ring[ri] = dt; ri = (ri + 1) % ring.length; rn = Math.min(ring.length, rn + 1); }
// resolve finished GPU timer queries without stalling
while (pendingQ.length) {
const p = pendingQ[0];
if (!gl.getQueryParameter(p, gl.QUERY_RESULT_AVAILABLE)) break;
if (!gl.getParameter(timer.GPU_DISJOINT_EXT)) { gpuMs = gl.getQueryParameter(p, gl.QUERY_RESULT) / 1e6; if (bench && benchScene >= 0) benchGpu.push(gpuMs); }
gl.deleteQuery(p); pendingQ.shift();
}
if (bench) stepBench(now, dt, cpuMs);
if (now - idleT > 3000 && (demo || audio.mode !== 'none')) ui.classList.add('idle');
if ((ri & 7) === 0) {
const k = analysis.key;
hudEl.textContent =
`${sc.name} ${canvas.width}x${canvas.height}\n` +
`frame ${(rn ? pct(ring, rn, 0.5) : 0).toFixed(1)} ms p50 ${pct(ring, rn, 0.95).toFixed(1)} p95 (budget 16.7) cpu ${cpuMs.toFixed(2)} ms` +
(timer && gpuMs ? ` gpu ${gpuMs.toFixed(2)} ms` : '') + '\n' +
`key ${k.name} (${(k.confidence * 100) | 0}%) onsets ${analysis.onset.count} flux ${analysis.flux.toFixed(2)} ` +
`source ${demo ? 'synthetic demo' : audio.mode}`;
}
requestAnimationFrame(frame);
}
function stepBench(now, dt, cpu) {
if (benchScene === 99) return;
if (benchScene < 0) { benchScene = 0; benchStart = now; setScene(0); }
const el = now - benchStart;
if (el > WARM_MS) { benchDt.push(sync ? cpu : dt); benchCpu.push(cpu); }
if (el > BENCH_MS + WARM_MS) {
const stat = a => { const s = a.slice().sort((x, y) => x - y); return a.length ? { n: a.length, mean: a.reduce((x, y) => x + y, 0) / a.length, p50: s[s.length >> 1], p95: s[Math.floor(s.length * 0.95)], p99: s[Math.floor(s.length * 0.99)], max: s[s.length - 1] } : null; };
record.scenes.push({ scene: scenes[sceneIdx].name, frameMs: stat(benchDt), cpuMs: stat(benchCpu), gpuMs: stat(benchGpu), over16_7: benchDt.filter(x => x > 16.7 * 1.2).length });
benchDt = []; benchCpu = []; benchGpu = [];
if (sceneIdx < 2) { setScene(sceneIdx + 1); benchStart = now; }
else {
record.canvas = [canvas.width, canvas.height]; record.sync = sync; record.gpuTimer = !!timer; record.ua = navigator.userAgent;
record.renderer = (() => { const e = gl.getExtension('WEBGL_debug_renderer_info'); return e ? gl.getParameter(e.UNMASKED_RENDERER_WEBGL) : 'unknown'; })();
const out = JSON.stringify(record, null, 1);
$('bench').style.display = 'block'; $('bench').textContent = out;
console.log('PERLINES_BENCH ' + JSON.stringify(record));
benchScene = 99;
}
}
}
setScene(sceneIdx);
if (bench || q.has('demo')) startDemo();
requestAnimationFrame(frame);
window.__perlines = { setScene, startDemo, analysis: () => analysis };

43
web/src/noise.js Normal file
View file

@ -0,0 +1,43 @@
// Perlin gradient noise (3D) and a divergence-free curl field in 2D, for the CPU-side trail paths.
// Permutation table from a seeded LCG shuffle; gradients are the 12 cube-edge vectors.
const G = [[1,1,0],[-1,1,0],[1,-1,0],[-1,-1,0],[1,0,1],[-1,0,1],[1,0,-1],[-1,0,-1],[0,1,1],[0,-1,1],[0,1,-1],[0,-1,-1]];
export function makeNoise(seed = 1) {
const p = new Uint8Array(512), base = Array.from({ length: 256 }, (_, i) => i);
let s = seed >>> 0 || 1;
for (let i = 255; i > 0; i--) {
s = (s * 1664525 + 1013904223) >>> 0;
const j = s % (i + 1);
[base[i], base[j]] = [base[j], base[i]];
}
for (let i = 0; i < 512; i++) p[i] = base[i & 255];
const fade = t => t * t * t * (t * (t * 6 - 15) + 10);
const dot = (g, x, y, z) => g[0] * x + g[1] * y + g[2] * z;
/** Perlin noise in [-1, 1]-ish. */
function noise3(x, y, z) {
const X = Math.floor(x), Y = Math.floor(y), Z = Math.floor(z);
x -= X; y -= Y; z -= Z;
const xi = X & 255, yi = Y & 255, zi = Z & 255;
const u = fade(x), v = fade(y), w = fade(z);
const A = p[xi] + yi, AA = p[A] + zi, AB = p[A + 1] + zi;
const B = p[xi + 1] + yi, BA = p[B] + zi, BB = p[B + 1] + zi;
const l = (a, b, t) => a + t * (b - a);
return l(
l(l(dot(G[p[AA] % 12], x, y, z), dot(G[p[BA] % 12], x - 1, y, z), u),
l(dot(G[p[AB] % 12], x, y - 1, z), dot(G[p[BB] % 12], x - 1, y - 1, z), u), v),
l(l(dot(G[p[AA + 1] % 12], x, y, z - 1), dot(G[p[BA + 1] % 12], x - 1, y, z - 1), u),
l(dot(G[p[AB + 1] % 12], x, y - 1, z - 1), dot(G[p[BB + 1] % 12], x - 1, y - 1, z - 1), u), v), w);
}
/** Curl of a scalar noise potential: a divergence-free 2D velocity, written into out[0..1]. */
function curl2(x, y, t, out) {
const e = 0.01;
const dx = (noise3(x + e, y, t) - noise3(x - e, y, t)) / (2 * e);
const dy = (noise3(x, y + e, t) - noise3(x, y - e, t)) / (2 * e);
out[0] = dy; out[1] = -dx;
return out;
}
return { noise3, curl2 };
}

76
web/src/scenes/aurora.js Normal file
View file

@ -0,0 +1,76 @@
// Aurora disk: the spectrum history extruded into a polar mesh.
// Angle = frequency (mirrored, bass at the seam), radius = age (centre newest), height = band level.
import { program, GLSL_HEAD, GLSL_NOISE, mat4 } from '../gl.js';
const RINGS = 112, SEGS = 224;
const VS = GLSL_HEAD + GLSL_NOISE + `
layout(location=0) in vec2 pr; // x = ring 0..1, y = segment 0..1
uniform sampler2D hist; uniform float newest, time, kick, spin; uniform mat4 mvp;
out float h; out float ring; out float seg;
void main(){
float ang = pr.y * 6.2831853;
float f = abs(pr.y * 2.0 - 1.0); // mirrored frequency coordinate
float age = pr.x * 0.92;
float row = fract(newest - age);
float v = texture(hist, vec2(f, row)).r;
float n = vnoise(vec3(cos(ang) * 1.6, sin(ang) * 1.6, pr.x * 3.0 - time * 0.35));
h = v * (0.55 + 0.9 * n) * (1.0 - 0.6 * pr.x);
float rad = 0.12 + pr.x * 1.45 * (1.0 + 0.05 * kick);
ang += spin;
vec3 p = vec3(cos(ang) * rad, h * 0.9 + 0.1 * sin(pr.x * 18.0 - time * 2.0) * v, sin(ang) * rad);
ring = pr.x; seg = pr.y;
gl_Position = mvp * vec4(p, 1.0);
}`;
const FS = GLSL_HEAD + `
in float h; in float ring; in float seg;
uniform vec3 tint; uniform float kick; uniform float time;
out vec4 o;
void main(){
// curtain gradient: tint at the base, shifting toward cyan/green-white at the crest, like auroral oxygen/nitrogen lines
vec3 base = tint;
vec3 crest = mix(tint.gbr, vec3(0.8, 1.0, 0.9), 0.55);
vec3 c = mix(base, crest, smoothstep(0.1, 0.9, h));
float lines = 0.55 + 0.45 * smoothstep(0.82, 1.0, abs(sin(seg * 256.0 * 3.14159)));
float fade = (1.0 - ring) * (1.0 - ring) * 0.9 + 0.05;
float a = (0.015 + h * h * 0.42) * lines * fade * (1.0 + 1.2 * kick);
o = vec4(c * a, 1.0);
}`;
export class AuroraDisk {
constructor(gl) {
this.name = 'Aurora disk'; this.persistence = 0.82;
this.prog = program(gl, VS, FS);
const pr = new Float32Array(RINGS * (SEGS + 1) * 2);
let k = 0;
for (let r = 0; r < RINGS; r++) for (let s = 0; s <= SEGS; s++) { pr[k++] = r / (RINGS - 1); pr[k++] = s / SEGS; }
const idx = new Uint32Array((RINGS - 1) * SEGS * 6); k = 0;
for (let r = 0; r < RINGS - 1; r++) for (let s = 0; s < SEGS; s++) {
const a = r * (SEGS + 1) + s, b = a + SEGS + 1;
idx.set([a, b, a + 1, a + 1, b, b + 1], k); k += 6;
}
this.count = idx.length;
this.vao = gl.createVertexArray(); gl.bindVertexArray(this.vao);
gl.bindBuffer(gl.ARRAY_BUFFER, gl.createBuffer()); gl.bufferData(gl.ARRAY_BUFFER, pr, gl.STATIC_DRAW);
gl.enableVertexAttribArray(0); gl.vertexAttribPointer(0, 2, gl.FLOAT, false, 0, 0);
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, gl.createBuffer()); gl.bufferData(gl.ELEMENT_ARRAY_BUFFER, idx, gl.STATIC_DRAW);
gl.bindVertexArray(null);
this.kick = 0; this.spin = 0; this.seen = 0;
}
draw(gl, A, w, h, hist) {
if (A.onset.count !== this.seen) { this.seen = A.onset.count; this.kick = Math.max(this.kick, 0.4 + 0.6 * A.onset.strength); }
this.kick *= Math.pow(0.0008, A.dt);
this.spin += A.dt * (0.05 + 0.4 * this.kick + 0.2 * A.bass);
const proj = mat4.perspective(0.9, w / h, 0.1, 20);
const view = mat4.mul(mat4.translate(0, -0.15, -2.6), mat4.mul(mat4.rotX(0.62 + 0.04 * Math.sin(A.time * 0.2)), mat4.rotY(0)));
const P = this.prog;
gl.useProgram(P.p);
gl.uniformMatrix4fv(P.u.mvp, false, mat4.mul(proj, view));
gl.uniform1f(P.u.newest, hist.newest); gl.uniform1f(P.u.time, A.time); gl.uniform1f(P.u.kick, this.kick); gl.uniform1f(P.u.spin, this.spin);
gl.uniform3fv(P.u.tint, A.tint);
gl.activeTexture(gl.TEXTURE1); gl.bindTexture(gl.TEXTURE_2D, hist.tex); gl.uniform1i(P.u.hist, 1);
gl.bindVertexArray(this.vao);
gl.drawElements(gl.TRIANGLES, this.count, gl.UNSIGNED_INT, 0);
}
}

90
web/src/scenes/trails.js Normal file
View file

@ -0,0 +1,90 @@
// Vector trails: heads ride a divergence-free Perlin curl field, one head per spectrum lane;
// onsets kick speed and respawn a spray of heads from the centre. Each head leaves a polyline trail.
import { program, GLSL_HEAD } from '../gl.js';
import { makeNoise } from '../noise.js';
import { NB } from '../analysis.js';
const HEADS = 96, LEN = 72;
const VS = GLSL_HEAD + `
layout(location=0) in vec2 a; // trail point; slot 0 is the head
uniform float aspect; out float al; out float lane;
void main(){
al = 1.0 - float(gl_VertexID % ${LEN}) / ${LEN - 1}.0;
lane = float(gl_VertexID / ${LEN}) / ${HEADS}.0;
gl_Position = vec4(a.x / aspect, a.y, 0.0, 1.0);
}`;
const FS = GLSL_HEAD + `
in float al; in float lane; uniform vec3 tint; uniform float level; out vec4 o;
void main(){
vec3 c = mix(tint, tint.brg * 0.8 + 0.2, lane);
c = mix(c, vec3(1.0), pow(al, 6.0) * 0.6);
o = vec4(c * pow(al, 1.6) * (0.35 + level), 1.0);
}`;
export class VectorTrails {
constructor(gl) {
this.name = 'Vector trails'; this.persistence = 0.9;
this.prog = program(gl, VS, FS);
this.noise = makeNoise(2013);
this.pos = new Float32Array(HEADS * 2); // head positions
this.vel = new Float32Array(HEADS * 2);
this.trail = new Float32Array(HEADS * LEN * 2); // x, y per point
this.energy = new Float32Array(HEADS);
this.life = new Float32Array(HEADS);
this.scratch = new Float32Array(2);
for (let i = 0; i < HEADS; i++) this.respawn(i, 0.2 + Math.random() * 1.0);
const idx = new Uint16Array(HEADS * (LEN - 1) * 2); let k = 0;
for (let h = 0; h < HEADS; h++) for (let s = 0; s < LEN - 1; s++) { idx[k++] = h * LEN + s; idx[k++] = h * LEN + s + 1; }
this.count = idx.length;
this.vao = gl.createVertexArray(); gl.bindVertexArray(this.vao);
this.buf = gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER, this.buf);
gl.bufferData(gl.ARRAY_BUFFER, this.trail, gl.DYNAMIC_DRAW);
gl.enableVertexAttribArray(0); gl.vertexAttribPointer(0, 2, gl.FLOAT, false, 0, 0);
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, gl.createBuffer()); gl.bufferData(gl.ELEMENT_ARRAY_BUFFER, idx, gl.STATIC_DRAW);
gl.bindVertexArray(null);
this.seen = 0; this.kick = 0;
}
respawn(i, radius) {
const a = Math.random() * Math.PI * 2;
this.pos[i * 2] = Math.cos(a) * radius * 0.2; this.pos[i * 2 + 1] = Math.sin(a) * radius * 0.2;
this.vel[i * 2] = Math.cos(a) * radius; this.vel[i * 2 + 1] = Math.sin(a) * radius;
this.life[i] = 4 + Math.random() * 6;
const base = i * LEN * 2;
for (let s = 0; s < LEN; s++) { this.trail[base + s * 2] = this.pos[i * 2]; this.trail[base + s * 2 + 1] = this.pos[i * 2 + 1]; }
}
draw(gl, A, w, h) {
const dt = A.dt, t = A.time, n = this.noise, c = this.scratch;
if (A.onset.count !== this.seen) {
this.seen = A.onset.count; this.kick = Math.min(1.5, this.kick + 0.5 + A.onset.strength);
const spray = Math.round(8 + 24 * A.onset.strength);
for (let j = 0; j < spray; j++) this.respawn((Math.random() * HEADS) | 0, 1.2 + 2.2 * A.onset.strength);
}
this.kick *= Math.pow(0.02, dt);
for (let i = 0; i < HEADS; i++) {
const band = Math.min(NB - 1, Math.floor(Math.pow(i / HEADS, 1.3) * NB * 0.85));
const e = A.bands[band]; this.energy[i] += (e - this.energy[i]) * 0.3;
const x = this.pos[i * 2], y = this.pos[i * 2 + 1];
n.curl2(x * 0.9 + i * 0.13, y * 0.9, t * 0.18, c);
const speed = 0.25 + 1.2 * this.energy[i] + 1.4 * this.kick;
// steer velocity toward the flow; spray velocity decays toward it, so heads start as a burst and then drift
const k = 1 - Math.pow(0.15, dt);
this.vel[i * 2] += (c[0] * speed * 0.55 - this.vel[i * 2]) * k;
this.vel[i * 2 + 1] += (c[1] * speed * 0.55 - this.vel[i * 2 + 1]) * k;
let nx = x + this.vel[i * 2] * dt, ny = y + this.vel[i * 2 + 1] * dt;
this.life[i] -= dt;
if (this.life[i] <= 0 || Math.abs(nx) > 3 || Math.abs(ny) > 1.6) { this.respawn(i, 0.3 + this.energy[i]); continue; }
this.pos[i * 2] = nx; this.pos[i * 2 + 1] = ny;
const base = i * LEN * 2;
this.trail.copyWithin(base + 2, base, base + (LEN - 1) * 2);
this.trail[base] = nx; this.trail[base + 1] = ny;
}
gl.bindBuffer(gl.ARRAY_BUFFER, this.buf);
gl.bufferSubData(gl.ARRAY_BUFFER, 0, this.trail);
const P = this.prog;
gl.useProgram(P.p);
gl.uniform1f(P.u.aspect, w / h); gl.uniform3fv(P.u.tint, A.tint); gl.uniform1f(P.u.level, A.level * 0.8 + this.kick * 0.3);
gl.bindVertexArray(this.vao);
gl.drawElements(gl.LINES, this.count, gl.UNSIGNED_SHORT, 0);
}
}

120
web/src/scenes/water.js Normal file
View file

@ -0,0 +1,120 @@
// Burning water: a flame-lit ripple plane, with an onset-driven spray of embers lofted off the surface.
// Particles are stateless: the CPU only writes (birth, seed, strength, lane) into a ring buffer on onsets;
// the vertex shader computes every position from age, so the per-frame cost is one point draw.
import { program, fullscreen, GLSL_HEAD, GLSL_NOISE } from '../gl.js';
const MAX = 6144;
const SURF_FS = GLSL_HEAD + GLSL_NOISE + `
in vec2 uv; uniform vec2 res; uniform float time, bass, level; uniform vec3 tint;
uniform vec4 ripples[8]; // x, y (surface coords), birth time, strength
out vec4 o;
void main(){
vec2 p = (uv * 2.0 - 1.0) * vec2(res.x / res.y, 1.0);
// horizon at y = 0.1, plane below it
float d = p.y - 0.1;
if (d > -0.01) { o = vec4(0.0, 0.0, 0.0, 1.0); return; }
float depth = 0.55 / -d; // distance along the plane
vec2 s = vec2(p.x * depth, depth); // surface coords
float hgt = fbm(vec3(s * 0.9, time * 0.25)) * 0.7;
vec2 g = vec2(0.0);
for (int i = 0; i < 8; i++) {
float age = time - ripples[i].z;
if (age < 0.0 || age > 3.0) continue;
float r = length(s - ripples[i].xy);
float front = age * 2.4;
float w = exp(-pow((r - front) * 2.2, 2.0)) * exp(-age * 1.1) * ripples[i].w;
hgt += w * sin((r - front) * 9.0) * 0.6;
}
float e = 0.04;
float hx = fbm(vec3((s + vec2(e, 0)) * 0.9, time * 0.25)) * 0.7;
float hy = fbm(vec3((s + vec2(0, e)) * 0.9, time * 0.25)) * 0.7;
vec3 n = normalize(vec3((hgt - hx) / e * 0.5, 1.0, (hgt - hy) / e * 0.5));
float flame = pow(max(0.0, n.x * 0.6 + n.z * 0.8 + 0.3) , 2.0) + hgt * 0.8;
vec3 hot = mix(vec3(1.0, 0.35, 0.05), vec3(1.0, 0.9, 0.5), smoothstep(0.5, 1.4, flame + level));
vec3 cold = tint * 0.6;
vec3 c = mix(cold, hot, smoothstep(0.2, 0.9, flame + bass)) * (0.25 + 0.75 * exp(-depth * 0.12));
c *= mix(0.35, 1.0, tint.b + 0.2) ;
o = vec4(c * (0.15 + 0.5 * level + 0.35 * bass) * 0.45, 1.0);
}`;
const PVS = GLSL_HEAD + `
layout(location=0) in vec4 a; // birth, seed, strength, lane
uniform float time; uniform vec2 res;
out float life; out float heat;
float h1(float x){ return fract(sin(x * 127.1) * 43758.5453); }
void main(){
float age = time - a.x;
if (a.x < 0.0 || age < 0.0 || age > 2.0) { gl_Position = vec4(2.0, 2.0, 2.0, 1.0); gl_PointSize = 0.0; return; }
float s = a.y, str = a.z;
float vx = (h1(s) - 0.5) * 1.4 * (0.5 + str) + (a.w - 0.5) * 0.7;
float vy = (1.2 + 2.6 * h1(s + 3.0) * str + 0.8 * str);
float x = (a.w * 2.0 - 1.0) * 0.95 + vx * age + 0.08 * sin(age * 7.0 + s * 40.0);
float y = -0.1 + vy * age - 1.9 * age * age;
if (y < -0.1 && age > 0.1) { gl_Position = vec4(2.0, 2.0, 2.0, 1.0); gl_PointSize = 0.0; return; }
life = 1.0 - age / 2.0; heat = life * (0.15 + 0.6 * str);
gl_Position = vec4(x / (res.x / res.y), y, 0.0, 1.0);
gl_PointSize = (2.0 + 9.0 * h1(s + 7.0) * str) * (0.4 + life) * res.y / 900.0;
}`;
const PFS = GLSL_HEAD + `
in float life; in float heat; uniform vec3 tint; out vec4 o;
void main(){
vec2 q = gl_PointCoord * 2.0 - 1.0; float r = dot(q, q); if (r > 1.0) discard;
vec3 fire = mix(vec3(0.9, 0.12, 0.02), vec3(1.0, 0.85, 0.4), heat);
fire = mix(fire, tint * 1.3, 0.15 * (1.0 - heat));
o = vec4(fire * (1.0 - r) * life * 0.9, 1.0);
}`;
export class BurningWater {
constructor(gl) {
this.name = 'Burning water'; this.persistence = 0.78;
this.surf = program(gl, GLSL_HEAD + 'layout(location=0) in vec2 p; out vec2 uv; void main(){ uv = p*0.5+0.5; gl_Position = vec4(p,0,1); }', SURF_FS);
this.part = program(gl, PVS, PFS);
this.tri = fullscreen(gl);
this.data = new Float32Array(MAX * 4).fill(-1);
this.vao = gl.createVertexArray(); gl.bindVertexArray(this.vao);
this.buf = gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER, this.buf);
gl.bufferData(gl.ARRAY_BUFFER, this.data, gl.DYNAMIC_DRAW);
gl.enableVertexAttribArray(0); gl.vertexAttribPointer(0, 4, gl.FLOAT, false, 0, 0);
gl.bindVertexArray(null);
this.head = 0; this.seen = 0; this.ripples = new Float32Array(32).fill(0); this.rhead = 0; this.dirty = [0, 0];
for (let i = 0; i < 8; i++) this.ripples[i * 4 + 2] = -10;
this.seed = 1; this.acc = 0;
}
spawn(n, t, strength) {
const d = this.data;
const first = this.head;
for (let i = 0; i < n; i++) {
const j = this.head; this.head = (this.head + 1) % MAX;
this.seed += 1.618;
d[j * 4] = t; d[j * 4 + 1] = this.seed; d[j * 4 + 2] = strength * (0.5 + 0.5 * Math.random()); d[j * 4 + 3] = Math.random();
}
return first;
}
draw(gl, A, w, h) {
const t = A.time;
if (A.onset.count !== this.seen) {
this.seen = A.onset.count;
const s = A.onset.strength;
this.spawn(Math.round(40 + 260 * s), t, 0.35 + 0.65 * s);
const r = this.rhead; this.rhead = (this.rhead + 1) % 8;
this.ripples.set([(Math.random() - 0.5) * 6, 1.5 + Math.random() * 4, t, 0.5 + s], r * 4);
}
// a thin continuous simmer follows the bass so quiet passages are not dead
this.acc += A.dt * A.bass * 90;
if (this.acc >= 1) { this.spawn(Math.floor(this.acc), t, 0.15 + 0.3 * A.level); this.acc %= 1; }
gl.bindBuffer(gl.ARRAY_BUFFER, this.buf);
gl.bufferSubData(gl.ARRAY_BUFFER, 0, this.data); // 96 KB; simplest correct upload, measured in the frame-time HUD
let P = this.surf;
gl.useProgram(P.p);
gl.uniform2f(P.u.res, w, h); gl.uniform1f(P.u.time, t); gl.uniform1f(P.u.bass, A.bass); gl.uniform1f(P.u.level, A.level);
gl.uniform3fv(P.u.tint, A.tint); gl.uniform4fv(P.u.ripples, this.ripples);
gl.bindVertexArray(this.tri); gl.drawArrays(gl.TRIANGLES, 0, 3);
P = this.part;
gl.useProgram(P.p);
gl.uniform1f(P.u.time, t); gl.uniform2f(P.u.res, w, h); gl.uniform3fv(P.u.tint, A.tint);
gl.bindVertexArray(this.vao); gl.drawArrays(gl.POINTS, 0, MAX);
}
}

107
web/test/dsp.test.mjs Normal file
View file

@ -0,0 +1,107 @@
// node --test web/test (node >= 20). Synthetic-signal checks of the onset detector and key estimate.
import test from 'node:test';
import assert from 'node:assert/strict';
import { magnitudeSpectrum, OnsetDetector, KeyEstimator } from '../src/dsp.js';
const SR = 44100, FFT = 2048;
function rng(seed) { let s = seed >>> 0; return () => ((s = (s * 1664525 + 1013904223) >>> 0) / 4294967296) * 2 - 1; }
function clickTrack(times, seconds, noiseAmp = 0.003) {
const x = new Float32Array(Math.ceil(seconds * SR)), r = rng(7);
for (let i = 0; i < x.length; i++) x[i] = r() * noiseAmp;
for (const t of times) {
const s = Math.round(t * SR);
for (let i = 0; i < 900 && s + i < x.length; i++) x[s + i] += r() * 0.8 * Math.exp(-i / 180);
}
return x;
}
function detect(x, hopSec, opts) {
const det = new OnsetDetector(opts), hop = Math.round(hopSec * SR), found = [];
const mags = new Float32Array(FFT >> 1);
for (let off = 0; off + FFT <= x.length; off += hop) {
magnitudeSpectrum(x, off, FFT, mags);
// the analyser window ends at "now", so a frame's time is its window end, like AnalyserNode
const o = det.push(mags, (off + FFT) / SR);
if (o) found.push(o.time);
}
return found;
}
function score(truth, found, tol) {
// an onset in the window ending at time T is visible from T-FFT/SR .. T; allow that window plus tol
const used = new Set(); let hit = 0;
for (const t of truth) {
const i = found.findIndex((f, j) => !used.has(j) && f >= t - tol && f <= t + FFT / SR + tol);
if (i >= 0) { used.add(i); hit++; }
}
return { recall: hit / truth.length, falsePositives: found.length - used.size };
}
const TRUTH = [0.5, 1.0, 1.4, 2.1, 2.6, 3.3, 3.8, 4.6, 5.0, 5.7];
for (const hop of [512 / SR, 1 / 60]) {
test(`clicks found at hop ${(hop * 1000).toFixed(1)} ms`, () => {
const x = clickTrack(TRUTH, 6.5);
const found = detect(x, hop);
const s = score(TRUTH, found, 0.03);
console.log(` hop ${(hop * 1000).toFixed(1)}ms: recall ${s.recall}, false positives ${s.falsePositives}, found ${found.length}/${TRUTH.length}`);
assert.ok(s.recall >= 0.9, `recall ${s.recall}`);
assert.ok(s.falsePositives <= 1, `false positives ${s.falsePositives}`);
});
}
test('silence and steady noise give no onsets', () => {
assert.equal(detect(new Float32Array(SR * 4), 1 / 60).length, 0);
const r = rng(3), x = new Float32Array(SR * 4).map(() => r() * 0.1);
assert.equal(detect(x, 1 / 60).length, 0);
});
test('a sustained tone does not retrigger', () => {
const x = new Float32Array(SR * 4).map((_, i) => 0.3 * Math.sin(2 * Math.PI * 440 * i / SR));
assert.ok(detect(x, 1 / 60).length <= 1); // the very first rise may count once
});
test('refractory period merges a doublet 20 ms apart', () => {
const x = clickTrack([0.5, 1.5, 2.5, 3.0, 3.02], 4.5);
const found = detect(x, 512 / SR).filter(t => t > 2.9 && t < 3.4);
assert.equal(found.length, 1);
});
function chord(midis, seconds) {
const x = new Float32Array(Math.ceil(seconds * SR));
for (const m of midis) {
const f = 440 * 2 ** ((m - 69) / 12);
for (let h = 1; h <= 4; h++) for (let i = 0; i < x.length; i++) x[i] += Math.sin(2 * Math.PI * f * h * i / SR) * 0.1 / h;
}
return x;
}
function keyOf(x) {
const k = new KeyEstimator(SR, FFT >> 1), mags = new Float32Array(FFT >> 1), hop = 735;
for (let off = 0; off + FFT <= x.length; off += hop) { magnitudeSpectrum(x, off, FFT, mags); k.push(mags, hop / SR); }
return k.key;
}
test('key: C major scale-ish progression reads C major', () => {
// C-F-G-C triads, then lingering C major
const parts = [[60, 64, 67], [65, 69, 72], [67, 71, 74], [60, 64, 67], [60, 64, 67]];
const x = new Float32Array(SR * 15);
parts.forEach((p, i) => x.set(chord(p, 3), i * 3 * SR));
const k = keyOf(x);
assert.equal(k.name, 'C');
assert.ok(k.confidence > 0.2, `confidence ${k.confidence}`);
});
test('key: A minor progression reads A minor, and key tracks a transposition', () => {
// i - iv - i - V - i - i: the E major chord's G# is the leading tone that separates A minor from C major
const am = [[57, 60, 64], [62, 65, 69], [57, 60, 64], [64, 68, 71], [57, 60, 64], [57, 60, 64]];
const x = new Float32Array(SR * 18);
am.forEach((p, i) => x.set(chord(p, 3), i * 3 * SR));
assert.equal(keyOf(x).name, 'Am');
const up = am.map(p => p.map(m => m + 2)); // B minor
const y = new Float32Array(SR * 18);
up.forEach((p, i) => y.set(chord(p, 3), i * 3 * SR));
assert.equal(keyOf(y).name, 'Bm');
});

49
web/test/onset.test.mjs Normal file
View file

@ -0,0 +1,49 @@
// node --test test/onset.test.mjs Synthetic drum corpus (tools/onset-bench.mjs): floors for the shipped detector.
// Synthetic only: these numbers say nothing about real music and nothing was listened to.
import test from 'node:test';
import assert from 'node:assert/strict';
import { run, corpus, synthesize, spectra, score, detect, RATES } from '../tools/onset-bench.mjs';
import { OnsetDetector } from '../src/dsp.js';
const subset = corpus().filter((_, i) => i % 3 === 0); // 27 of 81 signals, every pattern/tempo/level/bed combination class
test('drum corpus: hit rate, false positives and timing at 30 fps and full rate', () => {
const res = run(['legacy', 'current'], subset);
for (const v of ['legacy', 'current']) for (const [r, a] of Object.entries(res[v]))
console.log(` ${v.padEnd(8)} ${r.padEnd(5)} hit ${(100 * a.hitRate).toFixed(1)}% fp ${a.fp} (${a.fpPerMin.toFixed(1)}/min) median err ${a.medianMs.toFixed(1)} ms`);
for (const [r, a] of Object.entries(res.current)) {
assert.ok(a.hitRate >= 0.85, `${r} hit rate ${a.hitRate}`);
assert.ok(a.fpPerMin <= 2, `${r} false positives/min ${a.fpPerMin}`);
assert.ok(a.medianMs <= 40, `${r} median timing error ${a.medianMs} ms`);
assert.ok(a.hitRate > res.legacy[r].hitRate + 0.2, `${r} must beat the legacy detector by 20 points`);
}
});
test('hit rate does not depend on the level of the track', () => {
const res = run(['current'], subset);
for (const [r, a] of Object.entries(res.current)) {
const rates = Object.values(a.lv).map(p => p.hit / p.n);
assert.ok(Math.max(...rates) - Math.min(...rates) < 0.05, `${r} spread across 0/-20/-40 dB: ${rates}`);
}
});
test('corpus is deterministic', () => {
const c = corpus()[5], a = synthesize(c), b = synthesize(c);
assert.deepEqual(a.truth, b.truth);
assert.ok(a.x.every((v, i) => v === b.x[i]));
});
test('steady bass drone with a noise bed and no drums gives no onsets', () => {
const { x } = synthesize({ pattern: 'rock', bpm: 120, level: 1, bed: 'loud', seed: 9 });
const quiet = x.slice(0, 48000 * 1); // the lead-in before the first drum hit
for (const hop of Object.values(RATES)) {
const found = detect(() => new OnsetDetector({ binHz: 48000 / 2048 }), spectra(quiet, hop)).filter(t => t > 0.6);
assert.equal(found.length, 0, `hop ${hop}: ${found}`);
}
});
test('a single hit is reported once, within the match window', () => {
const { x, truth } = synthesize({ pattern: 'four', bpm: 60, level: 0.1, bed: 'none', seed: 4 });
const s = score(truth, detect(() => new OnsetDetector({ binHz: 48000 / 2048 }), spectra(x, 512)));
assert.ok(s.fp <= 1, `fp ${s.fp}`);
});

48
web/tools/headless.mjs Normal file
View file

@ -0,0 +1,48 @@
// Drive headless Chrome over CDP (no dependencies; needs node >= 22 for global WebSocket).
// node tools/headless.mjs <chrome.exe> <url> bench -> prints PERLINES_BENCH json, console errors
// node tools/headless.mjs <chrome.exe> <url> shot <out.png> [waitMs]
// The page must already be served (tools/serve.mjs). Exit code 1 on page errors.
import { spawn } from 'node:child_process';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const [chrome, url, mode, out, waitArg] = process.argv.slice(2);
const port = 9300 + Math.floor(Math.random() * 500);
const dir = mkdtempSync(join(tmpdir(), 'perlines-chrome-'));
const proc = spawn(chrome, [`--remote-debugging-port=${port}`, `--user-data-dir=${dir}`, '--headless=new', '--no-first-run',
'--autoplay-policy=no-user-gesture-required', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--window-size=1280,720',
...(process.env.CHROME_FLAGS ? process.env.CHROME_FLAGS.split(' ') : []), 'about:blank'], { stdio: 'ignore' });
const stop = code => { try { proc.kill(); } catch {} process.exit(code); };
process.on('uncaughtException', e => { console.error(e); stop(1); }); // never leave the browser running if we die first
process.on('exit', () => { try { proc.kill(); } catch {} });
const sleep = ms => new Promise(r => setTimeout(r, ms));
let targets;
for (let i = 0; i < 60; i++) { try { targets = await (await fetch(`http://127.0.0.1:${port}/json`)).json(); if (targets.some(t => t.type === 'page')) break; } catch {} await sleep(250); }
const ws = new WebSocket(targets.find(t => t.type === 'page').webSocketDebuggerUrl);
await new Promise(r => ws.addEventListener('open', r));
let id = 0; const pending = new Map(); let errors = 0, benchLine = null;
ws.addEventListener('message', e => {
const m = JSON.parse(e.data);
if (m.id && pending.has(m.id)) { pending.get(m.id)(m); pending.delete(m.id); return; }
if (m.method === 'Runtime.consoleAPICalled') {
const text = m.params.args.map(a => a.value ?? a.description).join(' ');
if (text.startsWith('PERLINES_BENCH ')) benchLine = text.slice(15); else console.error(`[console.${m.params.type}] ${text}`);
if (m.params.type === 'error') errors++;
} else if (m.method === 'Runtime.exceptionThrown') { errors++; console.error('[exception]', JSON.stringify(m.params.exceptionDetails.exception?.description || m.params.exceptionDetails.text)); }
});
const send = (method, params = {}) => new Promise(r => { const i = ++id; pending.set(i, r); ws.send(JSON.stringify({ id: i, method, params })); });
await send('Runtime.enable'); await send('Page.enable');
await send('Page.navigate', { url });
if (mode === 'bench') {
for (let i = 0; i < 400 && !benchLine; i++) await sleep(250);
if (!benchLine) { console.error('no bench result'); stop(1); }
console.log(benchLine); stop(errors ? 1 : 0);
} else {
await sleep(+waitArg || 4000);
const r = await send('Page.captureScreenshot', { format: 'png' });
writeFileSync(out, Buffer.from(r.result.data, 'base64'));
stop(errors ? 1 : 0);
}

169
web/tools/onset-bench.mjs Normal file
View file

@ -0,0 +1,169 @@
// Onset detector bench: synthesises drum patterns with known onset times, runs a detector over them at the
// page's 30 fps feature rate and at full rate (512-sample hop), and scores hit rate, false positives and timing.
// Deterministic (fixed LCG seeds), no network, no dependencies. node web/tools/onset-bench.mjs [variant ...]
// SYNTHETIC ONLY: nothing here has been listened to, and a score here says nothing about real music.
import { magnitudeSpectrum, OnsetDetector } from '../src/dsp.js';
import { LegacyOnsetDetector } from './onset-legacy.mjs';
export const SR = 48000, FFT = 2048;
export const RATES = { fps30: Math.round(SR / 30), full: 512 }; // hop in samples
const LEAD = 1.0, DURATION = 14, SCORE_FROM = 0.6; // seconds
const MATCH_EARLY = 0.02, MATCH_LATE = 0.09; // a frame at time T (window end) matches an onset at t if t-0.02 <= T <= t+0.09
function rng(seed) { let s = seed >>> 0; return () => ((s = (s * 1664525 + 1013904223) >>> 0) / 4294967296) * 2 - 1; }
// 16 steps per bar. kick/snare/hat step lists.
export const PATTERNS = {
rock: { kick: [0, 8, 10], snare: [4, 12], hat: [0, 2, 4, 6, 8, 10, 12, 14] },
four: { kick: [0, 4, 8, 12], snare: [4, 12], hat: [2, 6, 10, 14] },
busy: { kick: [0, 3, 6, 10, 13], snare: [4, 12], hat: Array.from({ length: 16 }, (_, i) => i) },
};
export const BEDS = { none: { bass: 0, noise: 0 }, bed: { bass: 0.2, noise: 0.02 }, loud: { bass: 0.4, noise: 0.08 } };
export const LEVELS = { '0dB': 1, '-20dB': 0.1, '-40dB': 0.01 };
export const TEMPOS = [90, 120, 150];
// every voice ends with a 60 ms linear fade so a truncated tail cannot click and read as an onset
const taper = (i, n) => Math.min(1, (n - i) / (0.06 * SR));
function addKick(x, s, r) {
let ph = 0; const n = Math.round(0.5 * SR);
for (let i = 0; i < n && s + i < x.length; i++) {
const t = i / SR; ph += 2 * Math.PI * (45 + 90 * Math.exp(-t / 0.03)) / SR;
x[s + i] += taper(i, n) * 0.9 * Math.exp(-t / 0.12) * Math.sin(ph);
}
}
function addSnare(x, s, r) {
const n = Math.round(0.4 * SR); let p = 0;
for (let i = 0; i < n && s + i < x.length; i++) {
const t = i / SR, w = r(), hp = w - p; p = w;
x[s + i] += taper(i, n) * (0.3 * Math.exp(-t / 0.06) * Math.sin(2 * Math.PI * 185 * t) + 0.5 * Math.exp(-t / 0.09) * hp * 0.7);
}
}
function addHat(x, s, r, amp) {
const n = Math.round(0.15 * SR); let p1 = 0, p2 = 0;
for (let i = 0; i < n && s + i < x.length; i++) {
const t = i / SR, w = r(), a = w - p1, b = a - p2; p1 = w; p2 = a;
x[s + i] += taper(i, n) * amp * Math.exp(-t / 0.025) * b * 0.5;
}
}
/** -> { x: Float32Array, truth: [{ time, inst }] } */
export function synthesize({ pattern, bpm, level, bed, seed = 1 }) {
const x = new Float32Array(DURATION * SR), r = rng(seed), P = PATTERNS[pattern], B = BEDS[bed];
const step = 60 / bpm / 4, truth = [];
const place = (inst, k, fn, ...a) => { const t = LEAD + k * step; truth.push({ time: t, inst }); fn(x, Math.round(t * SR), r, ...a); };
for (let bar = 0, k0 = 0; LEAD + k0 * step < DURATION - 0.4; bar++, k0 += 16) {
for (const s of P.kick) if (LEAD + (k0 + s) * step < DURATION - 0.4) place('kick', k0 + s, addKick);
for (const s of P.snare) if (LEAD + (k0 + s) * step < DURATION - 0.4) place('snare', k0 + s, addSnare);
for (const s of P.hat) if (LEAD + (k0 + s) * step < DURATION - 0.4) place('hat', k0 + s, addHat, s % 4 === 0 ? 0.7 : 0.4);
}
// beds: a bass drone that glides between notes (no onsets) and pink-ish noise, faded in over 0.3 s
const notes = [55, 55, 65.4, 49], rb = rng(seed + 100);
let ph = 0, f = 55, pink = 0;
for (let i = 0; i < x.length; i++) {
const t = i / SR, target = notes[Math.floor(t / 2) % notes.length];
f += (target - f) * (1 - Math.exp(-1 / (0.3 * SR)));
ph += 2 * Math.PI * f / SR;
const fade = Math.min(1, t / 0.3), trem = 0.85 + 0.15 * Math.sin(2 * Math.PI * 0.25 * t);
pink = 0.98 * pink + 0.1 * rb();
x[i] += fade * (B.bass * trem * (Math.sin(ph) + 0.4 * Math.sin(2 * ph)) + B.noise * pink * 3);
}
const rf = rng(seed + 200);
for (let i = 0; i < x.length; i++) x[i] = x[i] * level + rf() * 1e-5; // dither floor, ~ -100 dBFS
return { x, truth };
}
/** Magnitude frames the way AnalyserNode hands them over: window ends at 'time', values round-trip through dB clamped at -120. */
export function spectra(x, hop) {
const frames = [], mags = new Float32Array(FFT >> 1);
for (let off = 0; off + FFT <= x.length; off += hop) {
magnitudeSpectrum(x, off, FFT, mags);
const m = new Float32Array(mags.length);
for (let k = 0; k < m.length; k++) m[k] = Math.pow(10, Math.max(20 * Math.log10(mags[k] + 1e-12), -120) / 20);
frames.push({ time: (off + FFT) / SR, mags: m });
}
return frames;
}
/** Coincident hits (a kick and a snare on one step) are one onset: a detector emits one event per moment.
* Greedy one-to-one match in time order; each onset takes the earliest unused detection in its window.
* Per-instrument rows count an instrument as hit when the event it sounds in was hit. */
export function score(truth, found) {
const events = [];
for (const t of truth.slice().sort((p, q) => p.time - q.time)) {
const e = events[events.length - 1];
if (e && t.time - e.time < 0.005) e.insts.push(t.inst); else events.push({ time: t.time, insts: [t.inst] });
}
const used = new Set(), errs = [], per = {};
let n = 0;
for (const e of events) {
if (e.time < SCORE_FROM) continue;
n++;
const i = found.findIndex((f, j) => !used.has(j) && f >= e.time - MATCH_EARLY && f <= e.time + MATCH_LATE);
if (i >= 0) { used.add(i); errs.push(found[i] - e.time); }
for (const inst of e.insts) { const p = per[inst] ??= { n: 0, hit: 0 }; p.n++; if (i >= 0) p.hit++; }
}
return { n, hit: errs.length, fp: found.filter((f, j) => !used.has(j) && f >= SCORE_FROM).length, errs, per };
}
export function detect(make, frames) {
const det = make(), out = [];
for (const fr of frames) { const o = det.push(fr.mags, fr.time); if (o) out.push(o.time); }
return out;
}
export function corpus() {
const cases = [];
for (const pattern of Object.keys(PATTERNS)) for (const bpm of TEMPOS) for (const [lv, level] of Object.entries(LEVELS)) for (const bed of Object.keys(BEDS))
cases.push({ pattern, bpm, lv, level, bed, seed: cases.length + 1 });
return cases;
}
// detector as Analysis builds it: bins up to 8 kHz
const maxBin = Math.round(8000 / (SR / 2 / (FFT / 2))); // what the legacy Analysis passed
export const VARIANTS = {
legacy: () => new LegacyOnsetDetector({ maxBin }),
current: () => new OnsetDetector({ binHz: SR / FFT }),
};
const median = a => { if (!a.length) return NaN; const s = a.slice().sort((p, q) => p - q); return s[s.length >> 1]; };
/** run variants over the corpus -> { [variant]: { [rate]: { n, hit, fp, minutes, medianMs, byInst, byBed, byLevel } } } */
export function run(variantNames = Object.keys(VARIANTS), cases = corpus()) {
const acc = {};
for (const v of variantNames) { acc[v] = {}; for (const r of Object.keys(RATES)) acc[v][r] = { n: 0, hit: 0, fp: 0, seconds: 0, errs: [], inst: {}, bed: {}, lv: {} }; }
for (const c of cases) {
const { x, truth } = synthesize(c);
for (const [rate, hop] of Object.entries(RATES)) {
const frames = spectra(x, hop);
for (const v of variantNames) {
const s = score(truth, detect(VARIANTS[v], frames)), a = acc[v][rate];
a.n += s.n; a.hit += s.hit; a.fp += s.fp; a.seconds += DURATION - SCORE_FROM; a.errs.push(...s.errs);
for (const [k, p] of Object.entries(s.per)) { const q = a.inst[k] ??= { n: 0, hit: 0 }; q.n += p.n; q.hit += p.hit; }
for (const key of ['bed', 'lv']) { const q = a[key][c[key]] ??= { n: 0, hit: 0, fp: 0 }; q.n += s.n; q.hit += s.hit; q.fp += s.fp; }
}
}
}
const out = {};
for (const v of variantNames) { out[v] = {}; for (const [r, a] of Object.entries(acc[v])) out[v][r] = { ...a, hitRate: a.hit / a.n, fpPerMin: a.fp / (a.seconds / 60), medianMs: median(a.errs) * 1000 }; }
return out;
}
const pct = (h, n) => (100 * h / n).toFixed(1).padStart(5) + '%';
export function report(res) {
const lines = [];
for (const [v, rates] of Object.entries(res)) for (const [r, a] of Object.entries(rates)) {
const i = Object.entries(a.inst).map(([k, p]) => `${k} ${pct(p.hit, p.n)}`).join(' ');
const b = Object.entries(a.bed).map(([k, p]) => `${k} ${pct(p.hit, p.n)}/${p.fp}fp`).join(' ');
const l = Object.entries(a.lv).map(([k, p]) => `${k} ${pct(p.hit, p.n)}/${p.fp}fp`).join(' ');
lines.push(`${v.padEnd(9)} ${r.padEnd(5)} hit ${pct(a.hit, a.n)} (${a.hit}/${a.n}) fp ${a.fp} (${a.fpPerMin.toFixed(1)}/min) median err ${a.medianMs.toFixed(1)} ms`);
lines.push(` by instrument: ${i}`, ` by bed: ${b}`, ` by level: ${l}`);
}
return lines.join('\n');
}
if (import.meta.url === `file://${process.argv[1]}`) {
const names = process.argv.slice(2).length ? process.argv.slice(2) : Object.keys(VARIANTS);
const cs = corpus();
console.log(`corpus: ${cs.length} signals x ${DURATION} s, ${Object.keys(RATES).map(r => `${r}=hop ${RATES[r]}`).join(', ')}, SR ${SR}, FFT ${FFT}; match window -${MATCH_EARLY * 1000}..+${MATCH_LATE * 1000} ms of window end`);
console.log(report(run(names, cs)));
}

65
web/tools/onset-legacy.mjs Normal file
View file

@ -0,0 +1,65 @@
// Frozen copy of the v2 onset detector as of commit f53a0d1, kept only so tools/onset-bench.mjs can reproduce the 'before' numbers.
// Not imported by the page. Delete when the baseline stops being interesting.
export class LegacyOnsetDetector {
constructor(opts = {}) {
this.compress = opts.compress ?? 200; // log(1 + compress * mag)
this.window = opts.window ?? 43; // median history, frames
this.delta = opts.delta ?? 0.08; // absolute floor above the median, in flux units
this.ratio = opts.ratio ?? 1.6; // and the flux must exceed ratio * median
this.refractory = opts.refractory ?? 0.06; // seconds
this.minBin = opts.minBin ?? 2;
this.maxBin = opts.maxBin ?? 0; // 0 = all bins
this.bands = opts.bands ?? 6; // log-spaced flux bands between minBin and maxBin
this.edges = null;
this.prev = null;
this.hist = []; // recent flux values
this.f1 = 0; this.f2 = 0; this.t1 = 0; // flux[n-1], flux[n-2], time[n-1]
this.last = -1e9;
this.flux = 0; // latest raw flux, for display
}
push(mags, t) {
const n = this.maxBin || mags.length;
if (!this.edges) {
this.edges = Array.from({ length: this.bands + 1 }, (_, i) => Math.round(this.minBin * Math.pow(n / this.minBin, i / this.bands)));
for (let i = 1; i < this.edges.length; i++) this.edges[i] = Math.max(this.edges[i], this.edges[i - 1] + 1);
}
const cur = new Float32Array(n);
for (let k = 0; k < n; k++) cur[k] = Math.log1p(this.compress * mags[k]);
let flux = 0;
if (this.prev) {
const p = this.prev;
// flux per log-spaced band, each divided by sqrt(its width): a kick (a few bass bins) then counts
// as much as a hat (hundreds of treble bins), which one whole-spectrum sum would bury
for (let b = 0; b < this.edges.length - 1; b++) {
let bf = 0;
for (let k = this.edges[b]; k < this.edges[b + 1]; k++) {
const ref = Math.max(p[k - 1] ?? 0, p[k], p[k + 1] ?? 0);
const d = cur[k] - ref;
if (d > 0) bf += d;
}
flux += bf / Math.sqrt(this.edges[b + 1] - this.edges[b]);
}
flux /= this.edges.length - 1;
}
this.prev = cur;
this.flux = flux;
// candidate is the previous frame: f1 must be a local maximum over f2 and the new flux
let out = null;
const cand = this.f1;
if (this.hist.length >= 8 && cand > this.f2 && cand >= flux) {
const sorted = this.hist.slice().sort((a, b) => a - b);
const med = sorted[sorted.length >> 1];
if (cand > med * this.ratio + this.delta && this.t1 - this.last >= this.refractory) {
out = { time: this.t1, strength: Math.min(1, (cand - med) / (cand + 1e-6) * 1.5) };
this.last = this.t1;
}
}
this.hist.push(flux);
if (this.hist.length > this.window) this.hist.shift();
this.f2 = this.f1; this.f1 = flux; this.t1 = t;
return out;
}
}

18
web/tools/serve.mjs Normal file
View file

@ -0,0 +1,18 @@
// Dev server: `node tools/serve.mjs [port]`, serves web/ over http://localhost (mic needs a secure context; localhost qualifies).
import http from 'node:http';
import { readFile } from 'node:fs/promises';
import { extname, join, normalize } from 'node:path';
import { fileURLToPath } from 'node:url';
const root = join(fileURLToPath(new URL('.', import.meta.url)), '..');
const types = { '.html': 'text/html', '.js': 'text/javascript', '.css': 'text/css', '.json': 'application/json' };
const port = +(process.argv[2] || 8080);
http.createServer(async (req, res) => {
const p = normalize(decodeURIComponent(new URL(req.url, 'http://x').pathname)).replace(/^(\.\.[/\\])+/, '');
try {
const f = join(root, p.endsWith('/') ? p + 'index.html' : p);
if (!f.startsWith(root)) throw new Error('outside root');
const body = await readFile(f);
res.writeHead(200, { 'content-type': types[extname(f)] || 'application/octet-stream' }).end(body);
} catch { res.writeHead(404).end('not found'); }
}).listen(port, '127.0.0.1', () => console.log(`Perlines on http://localhost:${port}/`));