AquaSynth/docs/reference-synth-roadmap.md

Reference Synth Roadmap

Current Focus

The active AquaSynth work has moved to the utterance parity curriculum: structured utterance metadata lowers through a learned utterance embedding and a learned synth-driver automation layer into vocal-tract controls, envelopes, LFOs, filters, and spectral lanes. The immediate proof is a fast parity harness against a simple speech synthesizer reference, so many controller weights can be tested as rendered synth output versus generated ground truth.

DX7, ZynAddSubFX, Faust examples, Surge, and Cardinal remain reference pressure. They are on the backburner while the speech lane earns IPA parity. Do not start new synth-digestion work unless it directly improves the vocal-tract harness, the Faust render path, or the loss/automation surfaces.

AquaSynth needs external synthesis references for the same reason the SFXR path was useful: a real target prevents the DSL from becoming a pile of locally plausible knobs. The goal is not to clone every synth's internals. The goal is to use known engines and patch formats as pressure tests, then rebuild selected sounds through AquaSynth's own graph until the missing abstractions are obvious.

Objective

Build a reference-driven synth library workflow:

• Parse or describe external synth patches as structured reference models.
• Render references when legally and technically practical.
• Rebuild selected references in AquaSynth patch DSL.
• Compare audio with analysis metrics and listening notes.
• Promote only the abstractions that make repeated reference rebuilds simpler.
• Reject reference targets we cannot turn into parity tests. Parsing a format is inventory; proving AquaSynth can reproduce the behavior is the point.

The DSL should grow because a reference target demanded a missing invariant, not because a field name seemed maybe useful at 2 AM.

Live Invariants

• AquaSynth owns its patch graph. External synths are reference targets, not runtime dependencies.
• Expressive parameters are part of the patch contract. Callers must be able to change exposed values at runtime through the hosted DSP parameter API instead of recompiling the patch.
• Reference importers must preserve source provenance, license status, and the source format hash.
• Tests must compare against rendered fixtures, extracted parameter topology, or known structural expectations that prove AquaSynth can express the same behavior. Weak "it parsed" tests do not prove expressivity, and references without a credible parity test path should not be consumed.
• AquaSynth Faust output can be rendered through Faust-generated C# for candidate audio comparison. Reference audio still needs a real source fixture or renderer; do not compare a rebuild only against itself and call that parity.
• DX7 reference audio uses dexed-py when available, with public-domain SysEx fixtures stored under the test project. These are development references, not package content.
• ZynAddSubFX source is pinned under external/zynaddsubfx as GPL test/development material. It may be compiled or wrapped only for parity tests; AquaSynth must not ship, link, or runtime-depend on it.
• Do not add a generic graph escape hatch until a specific family of references forces it.
• When a reference target exceeds the current voice-centric DSL, document the missing graph feature instead of smuggling it through unrelated parameters.

Target Families

New Lane: IPA Vocal Tract / Weksa

This is the current main lane, not merely another reference synth. Structured utterance metadata lowers into a learned utterance embedding, then into learned synth automation for the vocal tract and associated envelopes, LFOs, filters, and spectral controls. IPA text is the first curriculum surface for phonetic coverage. The roadmap lives in docs/ipa-vocal-tract-roadmap.md, and the research pass lives in docs/utterance-parity-research.md.

Pressure on AquaSynth DSL:

• IPA as an input surface, not an internal patch model.
• Phonology profiles separated from tract morphology.
• Gesture planning and coarticulation before audio rendering.
• Human and alien vocal anatomies driven by the same structured phonetic intent.
• Mockable tokenizer, feature mapper, gesture planner, morphology, emitter, and renderer seams.

First deliverable:

• A parity harness that renders tiny eSpeak NG utterances, extracts log-mel and timing evidence, and trains the chained utterance encoder plus synth driver.
• A compiled Faust speech-loss surface that can vary /speech/output/N controls across many candidates without recompiling topology.
• Stage 0/1 IPA tokenization and vowel feature fixtures tied to the harness.
• Human baseline vowel tract-area planner and audible vowel proof.
• CV/CVC fixtures only after the vowel/control harness produces useful loss evidence. These are intelligibility/timing references, not tract-anatomy truth.

Tier 1: DX7/Dexed

Use this first. DX7 SysEx is compact, structured, and musically meaningful: six operators, thirty-two algorithms, per-operator envelopes, ratios, feedback, LFO modulation, pitch envelopes, key scaling, and output levels.

Dexed is a GPL-3.0 DX7-compatible open-source synth that is closely modeled on the Yamaha DX7 and accepts SysEx cartridge data. Its msfa synth engine remains Apache-2.0, which may matter if we use code as a reference rather than only format notes.

Pressure on AquaSynth DSL:

• Operator graph representation.
• Multi-stage envelopes beyond ADSR.
• Algorithm presets as topology, not per-voice decoration.
• Feedback as a first-class edge.
• Ratio/fixed-frequency operator modes.
• Feature and audio parity tests against rendered references.

First deliverable:

• Add a Dx7SysEx parser for single voice and cartridge payloads.
• Convert one or two public-domain or project-authored DX7-style voices into ReferencePatch records.
• Build an AquaSynth DSL approximation using explicit FM templates.
• Add mel/envelope/spectral comparison against rendered Dexed output from the public-domain fixture. AquaSynth candidate rendering and dexed-py reference rendering exist, and analog1.syx voice 13 has the first thresholded parity rebuild. The next pressure is a harder voice that actually needs audible operator routing rather than a sine-like behavioral match. Staged operator envelopes now exist as readable env=rl rates=... levels=... declarations with per-segment curves. Direct DX7 rate/level lowering on PRC SYNTH1 forced EG timing/gain calibration; the current applied-envelope lowering and stronger summed/cascaded route scale give PRC the same character with much closer stacked-modulation brightness. The hard PRC probe writes ignored listening artifacts under artifacts/parity/dx7-prc-synth1 so metrics and ears can steer the next calibration pass together. Listening showed aggregate score was a weak judge for this target, so the hard PRC probe gates primarily on log-mel distance. The calibration ladder lives in dx7-calibration-plan.md and should be followed before more PRC-specific tuning.

Source: Dexed

Tier 2: ZynAddSubFX

ZynAddSubFX is a GPL-2.0-or-later synth with additive, subtractive, and PAD synthesis engines. It ships a large instrument library and exposes formant filters, state-variable filters, free envelope shapes, layered kits, modulation, and effects routing.

Pressure on AquaSynth DSL:

• Additive harmonic banks.
• Non-ADSR envelope curves.
• Layered instruments and kits.
• Formant and state-variable filter intent.
• PAD-style spectral/wavetable generation as a distinct source type.
• Part effects versus voice-local effects.

First deliverable:

• Parse a small set of Zyn instrument XML files into a neutral reference model. The first parser/classifier slice exists for project-authored .xiz fixtures and supports plain XML plus gzip-compressed XML.
• Classify features used by each instrument before trying to synthesize them. The current classifier identifies active ADD/SUB/PAD engines, enabled kit items, layering, envelopes, free envelopes, LFOs, filters, formants, and effects.
• Rebuild one additive lead, one PAD texture, and one formant/vocal patch. The first rebuild catalog entries now parse/export and record the first Zyn missing abstractions. Named layers, additive harmonic-bank syntax, PAD spectral-cloud syntax, and staged rate/level envelopes for normal voices now exist; exact PADsynth wavetable generation, arbitrary Zyn free-envelope point curves, fuller kit/effect routing, and richer formant/vowel morphing remain pressure.

The upstream ZynAddSubFX engine source is pinned as external/zynaddsubfx for parity tests, currently at 3ab608c432996ba4d582176572c0b0f82328c825. That checkout is GPL reference machinery, not AquaSynth implementation. Do not vendor the upstream Zyn instrument-bank submodule until its preset license/provenance is explicit enough for this repo. Use project-authored fixtures for parser shape and only promote external fixtures with clean source notes.

The current external-bank pressure survey is in zynaddsubfx-pressure-survey.md. Its strongest signal is that named kit/layer routing should probably precede syntax golf for PAD, additive banks, or formant motion.

The first language-golf response is layer: a named layer declaration owns engine tags, optional MIDI key range metadata, default gain, and an effect-send label while lowering to ordinary AquaSynth voices. It is intentionally only an ownership/routing scaffold for now; it does not add PAD synthesis, free envelopes, or effect buses by stealth.

The second response is harmonics: additive layers can declare ratio/gain partial banks such as harmonics layer=body root=220 partials=1:.16,2:.075. The parser preserves the bank as authored structure and lowers it to ordinary voices. This gives additive lead, drawbar, and future partial-bank targets a reusable surface without pretending to implement Zyn ADDsynth phase, bandwidth, or free-envelope details.

The third response is staged voice envelopes. Normal voices and layer defaults can now use the existing readable rate/level surface:

layer name=pad engine=pad env=rl rates=.5,.7,1.0,1.4 levels=1,.85,.7,0 gate=2.2
voice layer=pad freq=220

This promotes the envelope authority already used by operator graphs into the voice path. It is enough to express layered PAD/vocal onset and release shapes, but it is not yet a Zyn free-envelope point editor.

The fourth response is spectrum: PAD layers can declare a source-level partial cloud that emits as a Faust wavetable source before normal voice treatment:

layer name=pad_low engine=pad
spectrum layer=pad_low root=130.8128 spread=.012 partials=1:.07,1.5:.052

The source table is generated by an authoring-time PADsynth-style path: harmonics are spread into a frequency-domain amplitude profile, deterministic random phase is applied, one inverse FFT produces the looped table, and Faust only reads that baked table. This gives PAD texture targets a reusable source authority without inflating the voice graph. It still does not claim full Zyn PAD engine parity across all harmonic profiles, randomness controls, or multi-table pitch-zone behavior. root, spread, and partials are authoring-time table shape, not runtime controls.

The canonical AquaSynth spelling for richer spectral-table shaping is neutral: pad_mode, pad_bandwidth, pad_bwscale, pad_profile, and pad_position. The Zyn importer may populate those fields from Zyn's PAD controls, and the parser still accepts older zyn_* names in generated parity artifacts, but authored or newly generated .aqua should not use reference-synth names for general DSP intent.

Sources:

• ZynAddSubFX
• ZynAddSubFX feature list
• Paul Nasca's PADsynth algorithm

Tier 3: Faust Examples And Libraries

Faust remains the best reference for general DSP graph topology. Its official repo contains categorized examples, and the standard libraries include oscillators, filters, delays, reverbs, physical models, DX7, virtual analog effects, wave-digital models, spatialization, and more.

Pressure on AquaSynth DSL:

• General signal graph expression.
• Multichannel routing.
• Delay networks and feedback delay networks.
• Reverb topologies.
• Physical modeling: waveguides, modal resonators, mass interaction, exciters.
• Wave-digital circuit trees.

First deliverable:

• Add an ExternalFaustReference catalog entry type that stores source path, license/provenance, graph family, and expected compile behavior.
• Select small fixtures from official examples first: comb string, additive drum, stereo reverb, physical model, WDF filter.
• Use Faust references to decide whether AquaSynth needs a lower-level graph DSL beneath the current instrument patch surface.

Sources:

• Faust examples
• Faust library docs
• Faust physical models
• Faust reverbs
• Faust wave-digital models

Tier 4: Surge XT

Surge XT is a modern GPL-3.0 hybrid synth with a large real-world patch corpus. It is a strong long-term target, but it is too broad for the first import pass.

Pressure on AquaSynth DSL:

• Multi-scene patch structure.
• Multiple oscillator algorithms.
• Wavetables.
• Deep modulation routing.
• Filter/fx chains.
• Macro controls and performance mapping.

First deliverable:

• Inspect factory patch format and licensing.
• Build a feature extractor before attempting translation.
• Choose one simple subtractive patch, one wavetable patch, and one modulation patch as future targets.

Source: Surge XT

Tier 5: Cardinal/VCV-Style Modular Patches

Cardinal is a GPL-3.0 self-contained open-source modular synth based on VCV Rack code. It is the right target once AquaSynth needs to prove arbitrary modular graph expression.

Pressure on AquaSynth DSL:

• Module graph representation.
• Cables as first-class edges.
• Control voltage semantics.
• Audio-rate versus control-rate modulation.
• Feedback and clocking.
• Polyphonic lanes.

First deliverable:

• Do not start here.
• When the lower-level graph layer exists, parse a small Cardinal patch and classify modules, ports, cables, and rates.

Source: Cardinal

Data Model Slice

Add reference models before importers grow teeth:

public sealed record ReferencePatch(
    string Id,
    string Family,
    string Name,
    ReferenceSource Source,
    IReadOnlyList<ReferenceFeature> Features,
    IReadOnlyList<PatchParameter> Parameters,
    string? AquaSynthScript);

public sealed record ReferenceSource(
    string Kind,
    string Uri,
    string License,
    string Hash,
    string Notes);

public sealed record PatchParameter(
    string Path,
    string Label,
    float Default,
    float Min,
    float Max,
    float Step,
    string Unit,
    string AutomationRate,
    string Notes);

This keeps source authority and runtime control authority visible. A reference can be "known target, not yet translated" without pretending it is an AquaSynth patch, and an AquaSynth patch can say which values callers may vary without changing graph shape.

Runtime Parameter Surface

Faust supports this directly through UI controls such as hslider, vslider, nentry, button, and checkbox. Hosted DSP code can expose a parameter map and update control zones with setParamValue/getParamValue style APIs, so AquaSynth should compile once for a graph shape and then push parameter values at runtime.

AquaSynth DSL needs an explicit parameter declaration surface, likely separate from modulation buses:

param name=brightness path=/macro/brightness default=0.45 min=0 max=1 step=0.001
param name=strike path=/exciter/strike default=0.2 min=0 max=1 step=0.001

Lowering rules:

• Parameter paths must be stable across recompiles unless the script changes the declared path.
• Parameters lower to Faust controls with the declared default, range, and step.
• Audio-sensitive parameters should be smoothed at the signal crossing.
• Modulators and exposed parameters may target the same semantic value, but the combined authority must be explicit.
• Recompile only when the graph, routing, oscillator type, operator topology, or other structural shape changes.

Work Order

1. Create reference model contracts, parameter contracts, and tests.
2. Add explicit patch parameter declarations and Faust parameter emission.
3. Add DX7 SysEx parser and feature extraction.
4. Add two DX7-derived AquaSynth DSL rebuilds with exposed macro controls.
5. Add audio fixture workflow for rendered references.
6. Add Zyn XML feature extraction.
7. Add Faust reference catalog entries and compile checks.
8. Reassess DSL shape: keep voice DSL, split lower-level graph DSL, or add explicit operator/routing sublanguage.
9. Only then touch Surge or Cardinal.

Rejected Shortcuts

• Do not vendor giant preset banks before license and provenance are explicit.
• Do not translate everything into the current voice command by adding dozens of fields.
• Do not compare only parse success. That proves the parser is lenient, not that the DSL can express the sound.
• Do not use Faust as a hidden escape hatch for AquaSynth patches. If a reference requires raw Faust, the DSL is missing an abstraction or the reference belongs in a lower-level graph layer.

Success Criteria

• A new agent can pick one target family and know where to start.
• Every reference fixture says what feature pressure it applies.
• The DSL roadmap is driven by reference failures, not taste fumes.
• AquaSynth can keep using readable reference patches while the synth library grows toward serious DSP graph expression.