GameCult/AquaSynth
Watch
9
0
Fork
You've already forked AquaSynth
 0
Code Issues Pull requests 4 Projects Releases Packages Wiki Activity Actions
AquaSynth/docs/audio-doctrine.md
Meta Vens 94cf7174f7 Use Qdrant for music knowledge injection
2026-06-01 00:11:36 +01:00

23 KiB
Raw Permalink Blame History

AquaSynth Audio Doctrine

Mission

AquaSynth is the authoring and compiler surface for AquaSynth patches. It should let us sketch sound quickly, compile to Faust, inspect the graph, compare results, and hand stable contracts to the C#/Vortice side.

The point is not to rebuild a whole audio engine in C# because we got excited near a compiler. The point is to make patch intent portable, analyzable, and pleasantly dangerous.

Realtime Law

Realtime audio has one primitive moral fact: the callback deadline wins.

Runtime-facing code must avoid operations with unbounded or scheduler-dependent latency:

  • dynamic allocation or deallocation
  • filesystem, network, console, or logging I/O
  • locks, mutexes, blocking waits, sleeps, or context switches
  • host/API calls not explicitly documented as callback-safe
  • graph compilation, parsing, reflection, or cache misses

Authoring tools can do expensive work. Audio callbacks cannot. Build patches, compile Faust, allocate buffers, score references, and serialize contracts outside the audio thread. Feed the realtime side precompiled DSP, fixed buffers, plain parameters, and lock-free or double-buffered control updates.

Dataflow Shape

Patch scripts are a textual dataflow language. Their job is to describe a graph: voices, controls, modulators, envelopes, filters, and output routing.

Pure Data is useful discipline here:

  • A patch is a graph of objects and connections, not a bag of imperative steps.
  • Audio-rate work is block-based, and block size is part of the execution model.
  • Subpatches can change block size, overlap, resampling, and on/off DSP state; those ideas map to explicit compile/runtime boundaries, not hidden globals.
  • Control events and signal streams are different animals. They can meet, but the crossing should be named.

For this repo that means:

  • Keep SynthPatch explicit and serializable.
  • Keep parser sugar as sugar. Defaults, templates, SFXR atoms, and buses should all lower into visible patch graph data.
  • Keep block size, sample rate, channel count, and latency visible in APIs that consume or produce audio buffers.
  • Prefer deterministic lowering. Same script plus same options must produce the same patch and Faust source.

Filters own post-source spectral selection. Low-pass, high-pass, band-pass, and notch are native AquaSynth filter authorities. Formants remain vowel/body color, and PAD spectrum/profile fields remain source-table construction; do not use those surfaces as substitutes for ordinary subtractive filters.

Scene Voices

Reference audio is a scene, not a naked oscillator. A patch may contain several voices whose jobs differ:

  • primary articulatory or instrumental voice;
  • coupled tract/source noise;
  • release, impact, or transient voice;
  • background room tone, bed noise, hum, or environmental texture;
  • recording-condition voice or processing that approximates microphone, room, codec, or capture coloration.

Those voices can use the whole AquaSynth vocabulary: ordinary voice layers, noise, FM, AM/tremolo, envelopes, filters, formants, additive/spectral sources, macros, and modulation. The contract is honesty, not austerity. Helper voices are allowed to improve full parity, but they do not own articulation or instrument-body evidence unless the relevant lower-layer witnesses move too. When a scene voice exists mainly to match background noise or capture color, the report must name that role and score it as context, not as proof that the primary body learned to speak.

Song Snippet Challenges

Song-reference golfing is a scene-audio challenge, not an IPA or anatomy oracle. Local music clips may be used as private parity pressure, but the repo stores only derived challenge metadata and local artifact paths unless a fixture is explicitly licensed for redistribution.

The worker freezes a random clip into a local reference.wav, then exposes features that patch-writing agents can treat as control hints:

  • tempo comes from onset-envelope autocorrelation and is reported as tempo_bpm, beat_seconds, and tempo_confidence;
  • register comes from dominant-frequency autocorrelation plus spectral fallback and is reported as dominant_hz, register_low_hz, register_high_hz, and root_note;
  • scale hints are deterministic authoring pressure, not musicology truth: suggested_scale and scale_frequencies_hz give agents a bounded pitch set so candidates do not spray every octave while trying to match one song clip.

Current Strudel-like sequencing is expressed through existing AquaSynth controls: declare a param, drive it with curve ... interp=hold loop=true, then bind the parameter into voice fields such as freq=@/seq/pitch or gain=@/seq/kick. Native scale, note, and pattern sugar should be added only when the DSL owns the lowering and the emitted Faust remains inspectable.

Early song-snippet experiments show useful pressure for named scene instruments: alien-lead-chatter, dust-hat-grid, codec-bed-hum, and air-wash-hiss. Those names are not DSP owners yet. They are proposed authoring roles that should lower into explicit param, curve, and voice records. The invariant for future sugar is: role syntax may own brevity and defaults, but the lowered graph still owns pitch lanes, gates, filters, FM/AM, noise, and mix behavior.

Faust Boundary

Faust describes pure DSP: inputs to outputs. Architecture files and host code connect that pure processor to drivers, UI, MIDI, sensors, plugins, and the outside world.

That separation is the model for this repo:

  • The DSL owns musical intent and graph construction.
  • The Faust emitter owns pure signal expression.
  • AquaSynth owns the compile product contract: Faust toolchain selection, target-language options, cache keys, generated artifact layout, parameter and bus manifests, diagnostics, and provenance.
  • The engine owns scheduling, buffers, device I/O, threading, and presentation.
  • The engine may request compilation or load a cached artifact, but dynamic patch compilation belongs to AquaSynth and must complete outside the realtime callback before the engine swaps in the finished DSP.
  • Patch parameters are runtime controls, not compile-time constants. A compiled DSP must be able to expose stable parameter paths so AquaSynth can vary a sound without recompiling the patch.
  • Host parameters should be smoothed where they cross into signal-rate behavior.

Generated Faust should be stable, boring, and inspectable. Cleverness belongs in the DSL compiler only when it lowers to a graph a tired engineer can still read after midnight.

Authoring and parity tests may render through Faust-generated C# because that keeps comparison buffers inside the .NET test harness. Shipping/runtime paths should prefer Faust native targets when the platform permits it. Bundled Faust belongs in an explicit AquaSynth toolchain lane, while consumer runtimes should usually ship compiled DSP artifacts plus manifests instead of silently dragging a compiler into the engine package.

Control And Modulation

The Serum-style promise is not "one primitive that puts an oscillator on everything." The promise is that any meaningful parameter can be a modulation target in an ergonomic, visible way.

Rules:

  • Every modulation target must have a stable semantic name.
  • Every exposed parameter must declare a stable path, default, min, max, step, unit or scale when meaningful, and whether it is safe to automate at control rate.
  • Global controls and voice-local modulators should share target names.
  • Modulators need waveform, rate, depth, phase, and bias.
  • Buses are authoring conveniences; compiled patches should expose individual target lanes.
  • UI/control-rate movement should be smoothed before it becomes audible zipper noise.
  • Recompilation is for graph shape changes. Parameter changes must flow through the hosted DSP control API.

Notes And Envelopes

Notes own pitch and gate. Envelopes own level shape. Do not mix those authorities.

  • Note carries note frequency, gate duration for one-shot patches, and source selection for host/MIDI-driven patches.
  • Envelope is ADSR-shaped: attack time, decay time, sustain level, and release time.
  • Legacy SFXR sustain maps to note gate duration, because it is how long the generated sound is held before release.
  • Legacy SFXR punch maps to a lower sustain level plus voice gain compensation when importing old SFXR material: ADSR peak stays implicit at 1, and the sustain level falls below that peak. Keep that as compatibility mapping, not as a general envelope field.
  • Host/MIDI note mode should expose stable note frequency and note gate controls that the engine can wire to MIDI note-on/note-off behavior.
  • Faust-managed polyphony is the preferred MIDI path. AquaSynth should describe a single voice graph and emit Faust's standard freq, gain, and gate controls with [midi:on][nvoices:n] options, leaving MIDI decoding and voice allocation to Faust architectures unless a concrete target proves that boundary insufficient.
  • DX7 rate/level envelopes may be lowered to labeled ADSR approximations for rebuild work, but that is not exact DX7 envelope execution. Keep exactness pressure visible in reference features instead of smuggling it into ADSR.

Phonetic Tract Boundary

IPA is an authoring surface, not a synth model. Weksa or another language layer owns text, phonology, allophones, and phonetic intent. AquaSynth owns the articulatory realization: gesture planning, morphology constraints, tract-area targets, excitation events, and Faust lowering.

Graph Vocal Sources

Graph voices do not let freq= own vocal pitch. For acoustic-graph voices, freq= is only an initializer/hint for defaults such as valve stiffness when a script has not supplied a physical value. The live source authority is the source port and its acoustic load.

Current authority map:

  • Owner: AcousticSourcePort owns local excitation semantics at a graph terminal.
  • Inputs: reservoir/source pressure, downstream pressure, tension, aperture/opening, mass, damping, stiffness, saturation, tissue loss, aperture shaping, flow loss/resistance, flow scale, load coupling, rest opening, balance, source position, and the incident pressure at the graph node.
  • Outputs: source flow injected into the acoustic graph; rendered pitch is an outcome of valve state plus tract/radiation load.
  • Derived state: kind=syrinx and kind=labial are authoring aliases for model=tissue_valve; they are not species modules.
  • Forbidden writers: labial/syrinx lowering must not emit a commanded os.phasor(... freq ...) oscillator as the source of truth.
  • Shared path: larynx, syrinx, reed-like, and alien morphologies should become graphs plus source ports, not bespoke renderer families.

The reusable primitive is model=tissue_valve: a Faust-friendly nonlinear valve state with pressure drive, delayed velocity/displacement state, cubic aperture saturation, and incident-pressure load feedback. law=one_mass, law=two_mass, and law=body_cover are layered approximations over the same source authority. Two-mass/body-cover lowering exposes upper/lower mass, upper/lower stiffness, coupling stiffness, collision stiffness/damping, and vertical phase as physical control surfaces. A syrinx is two such valves feeding an acoustic graph; a larynx is one such valve feeding a tract. Different species should change morphology and gestures, not add hidden source renderers.

For graph tissue valves, pressure owns energy input. drive is not a vocal physics owner; authoring drive= is accepted only as a compatibility alias for flow_scale, which is an output calibration/control-lane scalar. It must not multiply reservoir pressure, lower the phonation threshold, or brighten a direct displacement term. Ease of oscillation and harmonic richness belong to pressure, mass, stiffness/tension, damping, tissue/collision loss, aperture shaping, flow resistance, and acoustic load.

Morphology is now owned by continuous area_curve records plus acoustic graph paths. A path may inline diameters/areas or reference area_curve=...; the compiler resamples that continuous area function into graph segments while runtime-safe shape controls deform the segment area expressions through stable parameters. Topology remains compile-time: adding or removing valves, paths, branches, terminals, or radiation ports requires recompilation. Length, opening, constriction, tongue/body deformation, velum/branch aperture, and lip/beak opening remain runtime control surfaces when represented as parameters.

Loss and radiation are named physical choices, not anonymous gain patches. loss=none|viscous|birkholz_2024|wall|custom chooses the segment attenuation law. radiation=simple_reflection|lip_piston|beak|nostril|wall chooses the opening-to-reference-area, differentiation, and high-pass behavior at acoustic boundaries. Generated Faust must keep named locals for segment loss, radiation model, radiation flow, source injected flow, source load pressure, and junction energy probes so bad sounds have an inspectable path back to the component that made them.

General control curves are a parameter-layer primitive. They target stable param path= values, not random downstream fields, so authored gestures, realtime host control, and learned controllers share one authority. Faust lowering keeps the realtime hslider as the base value and emits a separate curve-depth hslider, allowing the host to ride, reduce, or disable the authored gesture without recompiling. Downstream field bindings still read the same patch_param_N expression.

The current bird-golf tool extracts reference envelope, dominant pitch, onset/offset, active-duty, and flux features, then maps those into tissue-valve candidate parameters plus pressure/opening/beak curves. This is still a first gesture surface, not a full motor planner.

The current speech-learning lane adds a trainable surface above that boundary: structured utterance metadata lowers into an utterance embedding, then into a synth-driver automation vector for tract controls, envelopes, LFOs, filters, and spectral lanes. Those learned models are controllers. They do not get to erase the tract boundary or make morphology optional.

The boundary must stay split into three inspectable records:

  • PhoneticIntent: language-owned phones, features, timing, and prosody.
  • ArticulatoryPlan: AquaSynth-owned gestures and tract/source targets.
  • ArticulatoryConstraintReport: host-visible diagnostics when morphology cannot produce the requested articulation.
  • VocalTractPlanResult: the host reaction surface that either carries an accepted plan or a rejection with diagnostic codes.

Anatomy has veto power. If a beaked morphology cannot form a bilabial closure, the planner must report the missing capability instead of silently substituting a fake sound and calling it expressive.

The first parity reference is eSpeak NG because it is compact, deterministic, and cheap to render. It is timing and intelligibility pressure, not anatomy authority. Use it to populate loss surfaces and train controller weights; do not mistake matching its formant shortcuts for building a vocal tract.

Speech training receipts must carry timing and confidence, not just loss. For each metadata-to-embedding, embedding-to-automation, render, score, and backpropagation step, record the decision time, observed latency, intended latency budget, confidence estimate, and artifact/model identifiers. A correct sound from an unbounded or unobservable path is not a reliable witness.

Metrics

Analysis exists to catch regressions and support search, not to crown winners. AquaSynth Faust output can be rendered through Faust-generated C# in authoring tests, which gives us candidate audio buffers without depending on the old Rust renderer. That is only half of parity: reference targets still need a real external render or a captured, lawful fixture before the comparison means anything.

Useful metrics:

  • envelope distance for shape and timing
  • log-mel spectrogram distance for perceptual-ish spectral shape
  • speech-parity loss over tiny utterance fixtures before broader IPA claims
  • peak/RMS ratios for loudness sanity
  • zero-crossing and centroid ratios for noise/brightness drift
  • script readability and terseness scores for DSL golf work

Song-scene parity targets must publish target evidence before candidate golf: log-mel spectrogram CSV, per-band statistics, first and second derivative summaries, RMS envelope, envelope autocorrelation, and whitened spectral autocorrelation over band-normalized log-mel flux. Those artifacts are not loose side files only; scoring writes them as typed song challenge evidence records in the .cc CultCache/CultMesh store and references them from each trial result. Qdrant/Ollama retrieval may derive vectors from those records, but the .cc records remain truth.

Song-scene agents should iterate like a tiny studio. Render the candidate, extract the same features from the Faust output, compare target and candidate band motion, envelope autocorrelation, whitened spectral autocorrelation, loudness, centroid, and instrument-role metrics, then move sources, filters, gates, patterns, and gains. A one-shot patch is sample data; a five-attempt revision trace is curriculum.

Noise is not a scene by itself. Background air, dust, codec grit, tape, vinyl, and room color need a role owner that shapes spectrum and time. Prefer texture/noise_texture sugar or an equivalently gated, band-shaped explicit voice for scene noise. Full-duration raw wave=noise beds with broad filters are failed evidence: they sound like broken subtractive synthesis because that is exactly what they are.

Song continuity is a scored production invariant. A one-second phrase followed by low-motion air, pink noise, room tone, or pad wash is mode collapse, not arrangement. The evaluator should publish motion coverage, active coverage, first-second energy share, tail energy share, and mode-collapse risk; distillation should treat high collapse risk as negative pressure even when log-mel similarity looks superficially acceptable.

Composition structure is now part of song-scene authorship. Sound roles answer "what is playing"; composition roles answer "why does time keep mattering." Future song agents should declare meter/time signature, chord or tonal progression, note-degree sampling, instrument-lane gates, section events, composition-scale automation, and mix movement before polishing timbre. The current Strudel-ish sugar is intentionally small: meter stores the beat grid; sequence owns lane gates; scale owns sampled note degrees; chords owns progression/voicing lanes; and mix owns gain automation. Each lowers to ordinary param and curve records, so the realtime truth remains inspectable.

Song-scene instrument roles are scored as curriculum evidence. Voice-like leads should prefer syrinx/acoustic source ownership with pressure/opening motion and radiation shaping; percussion should prefer subtractive pitched body plus filtered noise skin; pads, beds, and drones should prefer additive/PAD banks through layer, harmonics, and spectrum. Simple oscillator blips without one of these owners are chip-distress-risk contrast material, not a strong lesson for the music generator.

Song-scene curriculum admission is producer apprenticeship, not syntax attendance. A candidate that names meter, sequence, kick, hat, or texture has not learned music until it leaves a producer brief, listening journal, Aqua gap ledger, and compact studio lesson tying those choices to the reference. Distillation should promote only rendered candidates with audible form, target-specific roles, and low template/noise-percussion risk. Stock four-lane loops and raw noise percussion belong in failure documents even when they keep RMS and motion metrics nonzero.

Music-production retrieval must be an evidence packet, not a lexical slogan list. Fresh agents should receive Qdrant-backed knowledge hits with owner, summary, transfer rules, AquaSynth patterns, failure modes, and the retrieved chunk text. A pass-level curator agent should read musician model output after scoring and write only novel, actionable entries into the next knowledge store; raw musician reports remain evidence, not doctrine.

Do not confuse high metric agreement with musical success. The game context, the reference sound, and the user’s ear still outrank the spreadsheet. Annoying, but civilization has survived worse.

During the generalized vocal-core expansion, Pink Trombone log-mel tests are diagnostic artifact writers, not acceptance gates. They must prove the graph renders bounded/nonzero audio and preserve WAV/report artifacts for inspection. Hard PT thresholds return only after the shared physical primitives have passed structural, passivity, and render-smoke checks; otherwise the test suite rewards parity whack-a-mole over a coherent voice machine.

For graph vocal sources, freq= is not pitch authority. It is only a stiffness initializer when the source has no explicit or parameter-bound stiffness. Once a tissue valve exposes stiffness, authored curves, realtime controls, or learned controllers own that degree of freedom, and the lowering must not reintroduce a fixed-frequency floor underneath it. Valve displacement shapes aperture; it is not a second hidden audio oscillator. Source flow comes from pressure through dynamic aperture, modal tissue motion, and acoustic load.

Birdsong and speech gestures are continuous motor acts. Linear control ramps are valid when the author wants mechanical straight-line motion, but the default mental model for vocal articulation should be smooth pressure, tension, stiffness, aperture, and radiation trajectories. Use smooth gesture interpolation for vocal motifs before blaming morphology or adding spectral compensators; a straight spectrogram trace is often a straight control curve leaking through the physics.

Source Distillations