# Utterance Parity Research Notes
Research pass: 2026-05-20.
## Current Machine
AquaSynth is growing a speech curriculum, not a theatrical pile of IPA
decorations. The active pipeline is:
```text
structured utterance metadata
-> learned utterance embedding
-> learned synth automation
-> vocal tract, envelopes, LFOs, filters, and spectral controls
-> rendered audio
-> loss against a speech reference
```
Each arrow is a machine-learning artifact. The first useful job is not
universal speech. It is a tight parity harness where small utterances can be
rendered against a simple reference synthesizer, scored quickly, and used to
probe hundreds of controller weights. The loss landscape comes first. Poetry
can wait outside and smoke.
## Prior Work Signals
### Physical And Articulatory Synthesis
Kelly-Lochbaum-style tube models remain the practical starting point for
realtime tract acoustics. Pink Trombone and related ports prove that a simplified
tract, glottis, noise source, and tract geometry can be interactive enough to
use as a control target. Recent Pink Trombone optimization work treats that
model as a realtime Kelly-Lochbaum articulatory synthesizer and fits physical
parameters, which is close to AquaSynth's black-box rendered-loss pressure.
VocalTractLab is the heavyweight reference for articulatory synthesis: a 3D
tract model, glottal source, aeroacoustic simulation, and articulatory control.
Its papers are a warning label: speech quality depends on control trajectories
and coarticulation, not only on static phoneme shapes.
Story's area-function model is especially useful for AquaSynth because it gives
a middle representation between symbolic phonetic intent and tract acoustics.
Vowels are shaped by tract modes; consonants can be expressed as constriction
location, area, and range imposed on that substrate. That matches the local
doctrine: area curves are the synthesis authority, formants are diagnostics.
### Gesture Planning
Articulatory phonology and task dynamics are the right control metaphor. Phones
should lower into overlapping gestures, not a row of isolated presets. Haskins
TADA is important because it treats an utterance as a gestural score feeding a
task-dynamic model. AquaSynth's utterance schema should eventually resemble a
score of targets, timings, coupling, emphasis, and speaking style; it should not
be a giant enum of phonemes with cute flags.
### Classical Formant References
Klatt-style and eSpeak-style formant synthesis are useful early references
because they are compact, deterministic, intelligible, and easy to render in
bulk. They are not anatomy truth. If AquaSynth learns only to imitate eSpeak's
formant shortcuts, it will build a loyal little mimic and call it a throat.
The correct use is narrow: use eSpeak NG for IPA/text coverage, timing pressure,
log-mel targets, and fast generated ground truth while the physical tract earns
basic vowels and consonants. Then move the curriculum toward tract-shaped
targets and listening fixtures.
### Differentiable DSP And Neural Controllers
DDSP is the strongest precedent for this shape of work: combine neural networks
with known signal models so the model learns control rather than reinventing
acoustics from raw samples. Magenta's DDSP work shows the general bargain:
smaller models, stronger inductive bias, and trainable synthesis controls.
More recent DDSP articulatory vocoder work synthesizes speech from articulatory
measurements, F0, and loudness. That is not AquaSynth's exact setup, because
AquaSynth starts from metadata and generated speech references rather than EMA
capture. The useful lesson is architectural: split source controls from tract or
filter controls, keep features interpretable, and train against rendered audio.
Neural source-filter models are also relevant. They show that neural pieces can
drive physically meaningful source/filter structure instead of replacing the
whole backend with a black box. AquaSynth should use that as permission to learn
automation and embeddings, not permission to throw away tract ownership.
### IPA And Feature References
PanPhon maps IPA segments to articulatory feature vectors and PHOIBLE provides
language inventories plus distinctive features. Both are reference material for
metadata and sanity checks. Neither should become the permanent internal truth.
The reason is simple: Weksa and alien morphologies need "human IPA-like intent"
without letting human phonetics own nonhuman anatomy. IPA features can seed
intent; morphology owns whether the requested articulation is possible.
## Implications For AquaSynth
Keep the curriculum brutally small:
1. Render tiny eSpeak fixtures and extract normalized log-mel evidence.
2. Train the utterance encoder and synth-driver chain from that rendered loss.
3. Add a compiled Faust candidate path that can vary controller outputs without
recompiling topology.
4. Use vowels first, then CV/CVC contrasts, then broader IPA.
5. Only move to intonation, prosody, emotional context, and personality after
the harness can show phoneme-level loss improvements without handwaving.
Keep training receipts as first-class evidence:
- when each stage decided;
- how much latency it spent;
- what latency budget it was trying to fit;
- how confident the mapping was;
- which request, result, checkpoint, artifact, and model version produced the
evidence.
Time is not bookkeeping garnish here. If the mapping is correct only after an
unbounded wait, cache accident, or hidden retry, the system has learned a
beautiful falsehood.
Keep scoring layered. The patch under test may use all of AquaSynth: FM, AM,
modulators, envelopes, filters, auxiliary animated voices, breath/noise/room
emulation, and post-processing around both the vocal output and its gesture
inputs. That is necessary for matching real IPA references, which carry speaker
and recording conditions. But the reports must not let a noise layer impersonate
articulation:
- `gesture_score` measures descriptor-to-surface coverage, motion direction,
spline contour/timing, primitive timeline consequences, and optional external
articulation landmarks.
- `clean_vocal_score` measures broad phoneme identity from the vocal primitive
path before heavy dressing.
- `full_parity_score` measures the whole patch against the reference clip.
Only the third score is allowed to reward recording-condition helpers. The first
score is where IPA labels and anatomical descriptors prove that the right organs
were asked to move.
The first scalable IPA training loop starts with frozen gesture rounds, not a
free-running optimizer. `IpaGestureExperiment.WriteRound` produces candidate
scripts, primitive timelines, gesture metrics, a manifest, and evidence JSONL
for a whole batch. `IpaGestureExperiment.AnalyzeRound` then writes metric
summaries, score-band clusters, and a science brief. A science worker can
inspect that immutable bundle for clusters, outliers, loss surfaces, and
promising descriptor/variant changes while the foreground process writes the
next batch. This keeps hypothesis generation, primitive evidence, and
statistical analysis separate enough to argue with each other. Good. Machines
get less stupid when their witnesses are allowed to disagree.
Do not start serious gradient descent from toy embedding packets. The first
training contract coordinated with Weksa is
`weksa.utterance_embedding_handoff.v0.1`: 1024 speech-text floats from
`bge-m3:latest`, variable-length PanPhon sequence evidence, a 256-float
AquaSynth-owned phonetic realization embedding from
`aquasynth.panphon_sequence_encoder.v0.1`, 32 deterministic prosody/emphasis
floats, 64 projected character-state floats, and a 64-float AquaSynth-owned
utterance embedding output. Anything smaller is a plumbing fixture, not a
training corpus.
Keep semantic text and phonetic realization separate. English text embeddings
may help preserve meaning and phrasing, but an IPA string is not English prose
with a spicy alphabet. Alien phones belong in PanPhon sequence evidence and the
AquaSynth-trained phonetic sequence encoder.
The universal utterance schema should stay boring at first:
- segment inventory and source spans;
- duration, stress, tone, and boundary hints;
- speaker/morphology identity;
- pitch, loudness, speaking-rate, and breath controls;
- emphasis and emotional/context vectors as named side inputs;
- diagnostics when a field is not yet consumed.
Do not encode every future expressive dimension before the parity harness can
make `/a pa ta ka sa ma/` less embarrassing. Grand schemas are where bugs go to
earn tenure.
## Source Ledger
- Pink Trombone interactive vocal tract:
- Pink Trombone optimization / Kelly-Lochbaum parameter fitting:
- Sndkit tract notes, Pink Trombone/Kelly-Lochbaum lineage:
- VocalTractLab project:
- VocalTractLab features:
- VocalTractLab coarticulation model paper:
- Story 2005 area-function model:
- Stop consonants and Story area-function model discussion:
- Haskins TADA:
- eSpeak NG repository and formant-synthesis reference:
- PanPhon package:
- PanPhon paper:
- PHOIBLE:
- PHOIBLE FAQ:
- Magenta DDSP:
- DDSP paper:
- DDSP review for music and speech synthesis:
- Introduction to differentiable synthesizer programming:
- DDSP articulatory vocoder:
- Neural source-filter waveform model:
- LF/voice source model comparison:
- LF model fitting for statistical parametric speech synthesis: