diff --git a/docs/research/articulatory-vocal-tract-sources.md b/docs/research/articulatory-vocal-tract-sources.md index 32a0f93..d1f350e 100644 --- a/docs/research/articulatory-vocal-tract-sources.md +++ b/docs/research/articulatory-vocal-tract-sources.md @@ -143,7 +143,17 @@ Re-measure 2026-11-12. The stress, nasal and lax-vowel figures were taken on mai The cause (the shorter oral side branch, 7.1 cm against 9.6 cm from the velum junction) is a candidate, not shown. Open. - Lax vowels: /ɪ/ and /ʊ/ are in the DSL table (`StoryVowelAreas`) and render in `WordRender` (/bɪn/, /bʊn/, isolated). Burg F1/F2: /ɪ/ 375/2024 against the tube chain 346/1987 (+8.5/+1.8 %), Hillenbrand 429/2034 (-12.5/-0.5 %), Peterson & Barney 392/1993 (-4.3/+1.5 %); /ʊ/ 486/894 against 478/850 (+1.7/+5.2 %), 469/1123 (+3.7/-20.4 %), 437/1023 (+11.3/-12.6 %). No defect found. -- Phones the tract cannot make: `WordRender` refuses /t/, /d/, /k/, /l/ with exit 4 (`UNSUPPORTED PHONES (the tract has no gesture for them; nothing rendered): t`); the table holds p b m n s z ʃ ʒ ʍ w and the Story vowels. A direct `phoneme_gesture` on an unknown IPA symbol still falls back to an alveolar approximant (`GestureDescriptor`); not changed here. +- Burst place, 2026-11-12 (`burst_spectrum` in `tools/formant-check/weak_spots.py`: render minus `--no-burst` render, 20 ms Hann window from the release at 0.65 of the consonant, spectrum smoothed over 400 Hz, peak searched from 1 kHz up, a DESIGN floor; /aCa/, 140 Hz, 48 kHz, explicit plosive descriptors). Before: the burst joined the path input at the glottal end, so place reached it only through the constriction; + peak/centroid /t/ 4172/8729 Hz, /k/ 6984/10218, /p/ 7087/10423 (burst 0.019): /t/ below /k/, the order Chodroff and Wilson 2014 have reversed (t 5216, k 2868, p 2236 Hz). The decision gate said inject. Now a contact whose position has a control surface (made by the first non-labial plosive of a word) injects its burst there: `LadderInjection`, spread over the two cells whose centres bracket the place, + scaled by cell area over first-cell area (the ladder carries flow waves). /t/ 4008/6681 Hz, /d/ 4008/6693, /k/ 3747/9312 (peak 3747 without the floor too); /t/ is above /k/ by 261 Hz, short of the 2348 Hz of the corpus. Burst window peak /t/ 0.160, /k/ 0.148, /p/ 0.019; closure residue under 0.004. Unscaled, the injection made the closure 30 times louder (outside peak 0.058). + /p/ and /b/ keep the glottal end and their Faust source byte for byte (`PlaceTableKeepsLabialAndAlveolarPositions`); their burst peak above 1 kHz is 7087 Hz, above /t/, and below 1 kHz it is a 434 Hz low resonance, not the place cue: the labial site is not moved. Below 1 kHz /t/ and /d/ peak at 480 Hz, the same low resonance. Unheard. +- /t/ closure leak, 2026-11-12 (explicit descriptor, not in the table; /ata/ against /apa/ without burst, closure = 0.2 to 0.65 of the consonant, dB re the mean power of the flanking vowels): the alveolar closure was held at the nasal's 0.03 cm aperture and opened at 0.5, and read -14.3 dB against /p/ -22.1 (the lips seal /p/ whatever the tongue does); the same window with the aperture 0 for the whole gesture, -24.3. A non-labial plosive now holds aperture 0 to 0.65: -24.9 dB (test run) against /p/ -20.4. +- /k/, 2026-11-12 (the same /aka/ frame, closure 0.2 to 0.65 of the consonant, dB re the flanking vowels, burst removed): closure -14.9 dB against /p/ -20.4 and /t/ -24.9; the same closure with the glottal source's flow scale held at 0, -22.7 dB, so the tract seals and the leak is the voicing (150 Hz) the voiceless source still carries, reaching the output through a closure that leaves a short rear cavity. At the /t/ place (cell 36.7) the same /k/ gesture reads -24.8 dB, so it is the place and not the gesture. Silencing voicing in voiceless stops would change /p/ and /t/ (pinned); reported as an unmet criterion, not worked around. Velar place 9.81 cm behind the lips (0.44 of the tract; between /u/ narrowest dorsal section 26, 6.97 cm, and /a/ pharyngeal section 13, 12.14 cm). Burst: peak 0.0506 against the following vowel's 0.1129 (0.45), spectral peak 3100 Hz against Chodroff and Wilson 2014 /k/ 2868 Hz (/t/ 3950 Hz by the same measure); centroid 4626 Hz against /t/ 3629 Hz by the same measure (not judged: no literature centroid retrieved). + Its injected burst peaked at 0.76 of the following vowel's peak at the shared gain 0.06 (/b/ 0.19): `TongueStopBurstGain` 0.03 gives 0.39 (0.0438 against 0.1129). Burst spectrum unchanged by the gain: peak 4011 Hz, centroid 5796 Hz, inside 3.5-7 kHz. Not met: /t/ above /p/ in burst peak (/p/ 7087 Hz above 1 kHz, 434 Hz below): the labial burst is still injected at the glottal end, which this branch pins. /k/ closure at the old 0.44 fraction is not sealed (core -15.7 dB). Unheard. +- /d/, 2026-11-13 (table entry `voiced_alveolar_plosive`, no new mechanism: the `voiced` token selects the voice bar as it does for /b/): its consonant splines are identical to /t/'s (7 of 7: closure, contact, burst site and gain, source level aside), so closure position, duration and burst are /t/'s. Voice bar (word-initial, closure 0.03-0.058 s against the following vowel, deviation): /d/ -28.9 dB, /b/ -29.2, /t/ -38.7, /p/ -40.1; /d/ over /t/ 9.8 dB. Burst spectral peak 3950 Hz for both /d/ and /t/ (Chodroff and Wilson 2014: /d/ 4878, /t/ 5216, so the same place gives the same peak here, 0 Hz apart against their 338). Burst 0.0603 peak at its own release, 0.00002 before and 0.00000 after. The alveolar closure's voice bar does not differ from the labial one at this resolution (0.3 dB), so no number needed tuning. Unheard. +- /l/, 2026-11-13 (gate then build; /ɑlɑ/ at 140 Hz, scaffold durations; no source for English /l/ formants or level is in the mind (`huginn known`), so every target is a DESIGN value from the cut spec: F1 250-400 Hz, F2 1000-1300 Hz, a dip at 2-3 kHz, level -4 to -10 dB re the flanking vowels, no burst). Gate: the ladder holds a lateral's antiresonance but not its parallel channel. A branch port joins the oral path at any cell (here the ridge, 2.555 cm from the lips) and a closed stub path (4 cells, cross-section 1 cm^2, end reflection +1, aperture 0) beside a sustained /ɑ/ notches the harmonics: stub 2.5 cm -12.3 dB at 2940 Hz, 3.0 cm (the built length) -13.4 dB at 2800 Hz in /ɑlɑ/, 3.6 cm -6.9 dB at 2940 Hz, 4.5 cm -7.9 dB at 2380 Hz, all against the same render with the port shut, which is the render without the stub to 0.0 dB. What the ladder cannot hold is a second oral channel that rejoins the oral path past the contact (a branch has one junction and ends in its own radiation), so the midline is narrowed rather than closed with the flow going round it. Built: the `lateral` token opens the `side_stub` branch for the gesture and narrows the midline to 0.06 cm (held to 0.8), the stub declared in the scaffold only for a word with a lateral (/apa aba ana ata/ keep their script and Faust source byte for byte). + Measured (Burg order 12, 20 ms frames, 0.205-0.255 s; level by RMS of the middle 60 percent re the flanking vowels): level -5.0 dB (band met), F1 263 Hz (met), F2 725 Hz (NOT met: 1000-1300; the narrowed tongue-tip constriction on the /ɑ/ body has no dorsal control to raise it), dip 13.4 dB under the stub-shut render at 2800 Hz (met), burst none (the render with and without release_flow is the same render). The midline diameter is a steep and non-monotonic lever on level and on formants together, so level and F1/F2 could not be set independently: midline 0.028 cm reads -7.2 dB but the spectrum is the sealed-tract leak (energy under 1 kHz, no F2); 0.04 -2.6; 0.05 -5.8 (F1 202); 0.06 -5.0 (F1 263, F2 725); 0.065 -2.5; 0.07 +0.4; 0.08 -0.5 (F1 287, F2 713); 0.15 -0.9 (F1 463, F2 737). Unheard. +- Phones the tract cannot make: `WordRender` refuses a phone outside the table with exit 4 (`UNSUPPORTED PHONES (the tract has no gesture for them; nothing rendered): `); the table holds p b t d k m n l s z ʃ ʒ ʍ w and the Story vowels. A direct `phoneme_gesture` on an unknown IPA symbol still falls back to an alveolar approximant (`GestureDescriptor`); not changed here. Burst-window mutants (hand-applied in a scratch worktree on af0e44f plus the cut, never committed, 2026-11-12; `BurstGainAroundGesture`). The rendered test `PlosiveBurstExistsOnlyAroundItsOwnRelease` did not kill any of them, so `PlosiveBurstGainWindowIsOneRampEitherSide` reads the window off the spline (opens one ramp, 0.15 of the gesture, before the plosive; closed one ramp after it): diff --git a/src/AquaSynth.Core/FaustExport.VocalLadder.cs b/src/AquaSynth.Core/FaustExport.VocalLadder.cs index c3a9ca0..a7a4e66 100644 --- a/src/AquaSynth.Core/FaustExport.VocalLadder.cs +++ b/src/AquaSynth.Core/FaustExport.VocalLadder.cs @@ -30,6 +30,13 @@ public static partial class FaustEmitter public string Refl(int i) => $"{Prefix}_r{i}"; } + // A flow that enters the path at a cell rather than at the glottal end: Position is a Faust expression for the normalized place (0 glottis, + // 1 lips) and Flow the signal as it would enter the first cell. It is spread over the two cells whose centres bracket the place (a hat of + // width one cell), so the place can move between gestures and a place at or behind the first cell centre is the glottal end. The ladder + // carries flow waves, so the same noise pressure in a cell of area A is a flow of Flow * A / A(first cell): a burst at the closure, + // where the area is nearly zero, is small until the constriction opens (measured: unscaled it was 30 times louder in the closure). + private sealed record LadderInjection(string Position, string Flow); + private sealed record LadderJunction(LadderPath Oral, LadderPath Branch, int Interface, string Velum, string Left, string Right, string Nose); private static int LadderCells(WaveguidePath path, TractAreaFunction shape, FaustExportOptions options) => @@ -47,6 +54,7 @@ public static partial class FaustEmitter string name, IReadOnlyList members, IReadOnlyDictionary inputs, + IReadOnlyDictionary> injections, IReadOnlyDictionary admittances, ParameterMap parameters, List warnings) @@ -88,6 +96,15 @@ public static partial class FaustEmitter foreach (var member in members) { var targetJunction = targets.GetValueOrDefault(member); + var sites = new List<(string Site, string Flow)>(); + foreach (var injection in injections.GetValueOrDefault(member.Path.Name) ?? []) + { + var site = $"{member.Prefix}_injection{sites.Count}_site"; + source.AppendLine($" {site} = min({F(member.Cells - 0.5f)}, max(0.5, ({injection.Position}) * {F(member.Cells)}));"); + sites.Add((site, injection.Flow)); + } + + string Injected(int cell) => string.Concat(sites.Select(entry => $" + max(0.0, 1.0 - abs({F(cell + 0.5f)} - {entry.Site})) * ({entry.Flow}) * {member.A(cell)} / {member.A(0)}")); for (var i = 0; i < member.Cells; i++) { if (i == 0 && targetJunction is not null) @@ -113,12 +130,12 @@ public static partial class FaustEmitter equations.AppendLine(targetJunction is { } entering ? $" '{member.R(0)} = ({entering.Oral.L(entering.Interface)} + {entering.Oral.R(entering.Interface - 1)}) * (1.0 + {entering.Nose}) * {loss} + {entering.Nose} * {member.L(0)} * {loss};" - : $" '{member.R(0)} = ({member.L(0)} * {F(LadderGlottalReflection)} + {(inputs.TryGetValue(member.Path.Name, out var input) ? input : "0.0")}) * {loss};"); + : $" '{member.R(0)} = ({member.L(0)} * {F(LadderGlottalReflection)} + {(inputs.TryGetValue(member.Path.Name, out var input) ? input : "0.0")}{Injected(0)}) * {loss};"); for (var i = 1; i < member.Cells; i++) { equations.AppendLine(joinedAt.TryGetValue((member, i), out var join) ? $" '{member.R(i)} = ({join.Right} * {member.L(i)} + (1.0 + {join.Right}) * ({member.R(i - 1)} + {join.Branch.L(0)})) * {loss};" - : $" '{member.R(i)} = ({member.R(i - 1)} - {member.Refl(i)} * ({member.R(i - 1)} + {member.L(i)})) * {loss};"); + : $" '{member.R(i)} = ({member.R(i - 1)} - {member.Refl(i)} * ({member.R(i - 1)} + {member.L(i)}){Injected(i)}) * {loss};"); } for (var i = 0; i < last; i++) diff --git a/src/AquaSynth.Core/FaustExport.cs b/src/AquaSynth.Core/FaustExport.cs index ca70c36..e006102 100644 --- a/src/AquaSynth.Core/FaustExport.cs +++ b/src/AquaSynth.Core/FaustExport.cs @@ -608,6 +608,7 @@ public static partial class FaustEmitter // are both in this network is a ladder junction instead and exchanges no flow term. var memberNames = members.Select(member => member.Path.Name).ToHashSet(StringComparer.OrdinalIgnoreCase); var inputs = new Dictionary(StringComparer.OrdinalIgnoreCase); + var injections = new Dictionary>(StringComparer.OrdinalIgnoreCase); foreach (var member in members) { var path = member.Path; @@ -618,13 +619,24 @@ public static partial class FaustEmitter .Where(term => term is not null) .Cast() .ToList(); - var contactTerms = network.Contacts - .Select(contactName => contacts.TryGetValue(contactName, out var contact) && contact.Path.Equals(path.Name, StringComparison.OrdinalIgnoreCase) - ? PrimitiveContactExpression(source, patch, name, contact, parameters, options, keepAlive) - : null) - .Where(term => term is not null) - .Cast() - .ToList(); + var contactTerms = new List(); + var pathInjections = new List(); + foreach (var contactName in network.Contacts) + { + if (!contacts.TryGetValue(contactName, out var contact) || !contact.Path.Equals(path.Name, StringComparison.OrdinalIgnoreCase)) continue; + var (released, burst) = PrimitiveContactExpression(source, patch, name, contact, parameters, options, keepAlive); + // A contact that has a position control surface lets its burst go in at that place; one whose place is a fixed number + // keeps the burst at the glottal end with the rest of the path input. + var position = OwnerField($"/vocal/contacts/{ConstrictionContactIndex(patch, contact)}", "position"); + if (patch.ControlSurfaces.Any(surface => surface.FieldPath.Equals(position, StringComparison.OrdinalIgnoreCase))) + { + contactTerms.Add(released); + pathInjections.Add(new LadderInjection(parameters.Expression(position, contact.Position), burst)); + } + else contactTerms.Add($"({released} + {burst})"); + } + + injections[path.Name] = pathInjections; var branchTerms = network.Branches .Select(branchName => branches.TryGetValue(branchName, out var branch) && branch.FromPath.Equals(path.Name, StringComparison.OrdinalIgnoreCase) && !memberNames.Contains(branch.ToPath) ? PrimitiveBranchExpression(source, patch, name, branch, parameters, options, keepAlive) @@ -650,7 +662,7 @@ public static partial class FaustEmitter var (lips, glottalPressures) = members.Count == 0 ? (new Dictionary(), new Dictionary()) - : EmitVocalLadder(source, patch, network, name, members, inputs, admittances, parameters, warnings); + : EmitVocalLadder(source, patch, network, name, members, inputs, injections, admittances, parameters, warnings); foreach (var member in members) { @@ -743,7 +755,7 @@ public static partial class FaustEmitter return flow; } - private static string PrimitiveContactExpression( + private static (string Released, string Burst) PrimitiveContactExpression( StringBuilder source, SynthPatch patch, string voiceName, @@ -770,7 +782,7 @@ public static partial class FaustEmitter keepAlive.Add(reservoir); keepAlive.Add(burst); keepAlive.Add(released); - return $"({released} + {burst})"; + return (released, burst); } // Open fraction of a branch port: the ladder reads it as the velum area when the branch joins two paths of the network, diff --git a/src/AquaSynth.Core/IpaGestureExperiment.cs b/src/AquaSynth.Core/IpaGestureExperiment.cs index b343df3..84a2c69 100644 --- a/src/AquaSynth.Core/IpaGestureExperiment.cs +++ b/src/AquaSynth.Core/IpaGestureExperiment.cs @@ -222,7 +222,33 @@ public static class IpaGestureExperiment start += duration; } - return Scaffold(lines, Math.Clamp(start + tailSeconds, 0.15f, 4f), frequency); + return Scaffold(lines, Math.Clamp(start + tailSeconds, 0.15f, 4f), frequency, phones.Any(phone => NeedsSideStub(phone.Target))); + } + + // A lateral is the one gesture that opens the side stub (PatchScript.SideStubBranch); a word without one is scaffolded without it, byte for byte. + private static bool NeedsSideStub(IpaGestureExperimentTarget target) => + (target.Descriptor.Length > 0 ? target.Descriptor : PatchScript.TryIpaDescriptor(target.Ipa, out var known) ? known : "") + .Contains("lateral", StringComparison.OrdinalIgnoreCase); + + // The sublingual cavity beside a lateral's midline contact: a closed side branch of the oral tract at the alveolar ridge. Its quarter-wave + // antiresonance (c / 4L, lowered by the junction and the orifice) is the lateral's dip; the length is a DESIGN VALUE placing it between 2 and 3 kHz + // (no primary dimension retrieved). Cross-section 1 cm^2, four cells of the stub's own morphology; the first cell is the orifice, the branch + // opening times the port maximum (NasalTractAreas). The end is closed (reflection +1, aperture 0: it radiates nothing). + private const string SideStubLengthCentimeters = "3"; + private const string SideStubAreas = "1,1,1,1"; + + private static string SideStubDeclarations(bool sideStub) + { + if (!sideStub) return ""; + PatchScript.TryPlaceDistanceCentimeters("alveolar", out var ridge); + var at = 1f - ridge / StoryVowelAreas.MeanLengthCentimeters; + return $""" + morphology name=side_stub length_cm={SideStubLengthCentimeters} areas={SideStubAreas} + waveguide_path name=side_stub_path morphology=side_stub speed={TractSpeedMetersPerSecond} strategy=thiran order=1 max_delay=4096 loss=.98 + branch_port name={PatchScript.SideStubBranch} from=oral_path from_position={Format(at)} to=side_stub_path opening=0 coupling=1 + radiation_load name=side_stub_end path=side_stub_path kind=lip position=1 aperture=0 reflection=1 impedance=.28 + + """; } private static string GestureLine(IpaGestureExperimentTarget target, float start, float duration, float intensity) => @@ -237,7 +263,7 @@ public static class IpaGestureExperiment // murmur read -4.2 dB (/m/) and -2.2 dB (/n/) re the flanking vowels; the target is 8 to 16 dB under (no primary level or bandwidth retrieved). private const string NasalPathLoss = ".975"; - private static string Scaffold(IReadOnlyList gestureLines, float sustain, float frequency) => $$""" + private static string Scaffold(IReadOnlyList gestureLines, float sustain, float frequency, bool sideStub = false) => $$""" patch gain=.45 morphology name=oral length_cm={{Format(StoryVowelAreas.MeanLengthCentimeters)}} diameters=.55,.75,1.05,1.45,1.25,.8 tongue_index=3 tongue_diameter=1.4 constriction_index=4 constriction_diameter=1 constriction_width=.03 lip_opening=4 vowel_set area_function=oral source={{StoryVowelAreas.Source}} @@ -250,7 +276,7 @@ public static class IpaGestureExperiment radiation_load name=mouth path=oral_path kind=lip position=1 aperture=.8 reflection=-1 impedance=.28 radiation_load name=nostril path=nasal_path kind=lip position=1 aperture=.8 reflection=-1 impedance=.28 probe_timeline name=flow network=voice blocks=12 block_size=64 - vocal_network name=voice paths=oral_path,nasal_path sources=folds contacts=contact branches=velopharynx radiation=mouth,nostril probe=flow + {{SideStubDeclarations(sideStub)}}vocal_network name=voice paths=oral_path,nasal_path{{(sideStub ? ",side_stub_path" : "")}} sources=folds contacts=contact branches=velopharynx{{(sideStub ? "," + PatchScript.SideStubBranch : "")}} radiation=mouth,nostril{{(sideStub ? ",side_stub_end" : "")}} probe=flow {{string.Join("\n", gestureLines)}} vocal network=voice freq={{Format(frequency)}} gain=.7 sustain={{Format(sustain)}} """; diff --git a/src/AquaSynth.Core/PatchScript.cs b/src/AquaSynth.Core/PatchScript.cs index 3bac4fb..38a619b 100644 --- a/src/AquaSynth.Core/PatchScript.cs +++ b/src/AquaSynth.Core/PatchScript.cs @@ -13,6 +13,10 @@ public static class PatchScript /// that names another phone falls back to an alveolar approximant; a caller that must not be approximated asks here first. /// Vowels come from , the one table that also says which measured tract shape each selects. /// + /// The branch port a lateral gesture opens: a closed side stub beside the oral path at the alveolar ridge (the sublingual cavity). + /// A script that gestures a lateral without declaring it is refused, not approximated. + public const string SideStubBranch = "side_stub"; + public static bool TryIpaDescriptor(string ipa, out string descriptor) { if (StoryVowelAreas.TryVowelForIpa(ipa, out var vowelDescriptor, out _)) @@ -25,8 +29,12 @@ public static class PatchScript { "p" => "voiceless_bilabial_plosive", "b" => "voiced_bilabial_plosive", + "t" => "voiceless_alveolar_plosive", + "d" => "voiced_alveolar_plosive", + "k" => "voiceless_velar_plosive", "m" => "voiced_bilabial_nasal", "n" => "voiced_alveolar_nasal", + "l" => "voiced_alveolar_lateral_approximant", "s" => "voiceless_alveolar_fricative", "z" => "voiced_alveolar_fricative", "ʃ" => "voiceless_postalveolar_fricative", @@ -39,6 +47,11 @@ public static class PatchScript return known is not null; } + /// The distance in cm from the lip plane at which a descriptor's place closes, or false when the place has no distance + /// (fraction only, or no place named): a caller must not invent one. + public static bool TryPlaceDistanceCentimeters(string descriptor, out float centimeters) => + Compiler.TryPlaceDistanceCentimeters(descriptor, out centimeters); + public static SynthPatch Parse(string script) { var compiler = new Compiler(); @@ -1513,7 +1526,6 @@ public static class PatchScript voiced = false; } - var place = PlaceTarget(tokens); var constriction = ConstrictionTarget(tokens); var pressure = Math.Clamp(0.45f + 0.30f * intensity, 0, 1); AddAuthoredSpline($"{name}_source_pressure", "/vocal/sources/0/pressure", start, duration, (0, 0.4f), (0.2f, pressure), (1, pressure * 0.85f)); @@ -1531,13 +1543,14 @@ public static class PatchScript return; } + var length = _areaFunctions.Count > 0 ? _areaFunctions[0].Shape.LengthCentimeters : StoryVowelAreas.MeanLengthCentimeters; + var place = PlaceFraction(tokens, length); if (_areaFunctions.Count > 0 && _areaFunctions[0].Vowels is not null && _areaFunctions[0].Deformation is { } control) { // Under vowel_set the constriction is placed by distance along the tract: the surface is a section index, so a normalized // place fraction would land at fraction * sections / (sections - 1) of the tract, past the lips for a labial. var sections = Math.Max(2, _areaFunctions[0].Shape.Sections); - var position = ConsonantPositionFromGlottis(tokens, place, _areaFunctions[0].Shape.LengthCentimeters); - HoldOnSurface($"{name}_constriction_index", "/vocal/areas/0/area/constriction_index", start, duration, position * (sections - 1) / control.IndexScale); + HoldOnSurface($"{name}_constriction_index", "/vocal/areas/0/area/constriction_index", start, duration, place * (sections - 1) / control.IndexScale); } else { @@ -1547,8 +1560,13 @@ public static class PatchScript // A nasal stop holds its oral closure for as long as the velum and the contact stay put (to 0.8 of the gesture): the murmur is the // closed-mouth sound, and an alveolar closure that began to open at 0.5 let the oral tract radiate again over the last third (/n/ read // -3.8 dB under its vowels against /m/ -13.2 dB, whose lips hold to 0.7). - var closureHold = tokens.Contains("nasal") ? 0.8f : 0.5f; - AddAuthoredSpline($"{name}_constriction_diameter", "/vocal/areas/0/area/constriction_diameter", start, duration, (0, constriction), (closureHold, constriction), (1, Math.Min(0.65f, constriction + 0.2f))); + var stopsBehindTheLips = (tokens.Contains("plosive") || tokens.Contains("stop")) && !(tokens.Contains("bilabial") || tokens.Contains("labiodental") || tokens.Contains("labial")); + // A stop whose seal is the tongue (not the lips) is closed completely and until the contact opens (0.65): at the nasal's 0.03 cm + // aperture a /t/ closure measured -10.4 dB re its vowels against /p/ -19.7 (the lips seal /p/ whatever the tongue does), and + // rendered -21.5 dB with the aperture at 0 (area floor 1e-6 cm^2). + var closureHold = tokens.Contains("nasal") || tokens.Contains("lateral") ? 0.8f : stopsBehindTheLips ? 0.65f : 0.5f; + var closed = stopsBehindTheLips ? 0f : constriction; + AddAuthoredSpline($"{name}_constriction_diameter", "/vocal/areas/0/area/constriction_diameter", start, duration, (0, closed), (closureHold, closed), (1, Math.Min(0.65f, constriction + 0.2f))); if (tokens.Contains("bilabial") || tokens.Contains("labiodental") || tokens.Contains("labial") || tokens.Contains("rounded")) { @@ -1570,7 +1588,8 @@ public static class PatchScript // it by the stored pressure and the opening (see PrimitiveContactExpression). The gain exists only around this plosive: // a surface holds its last value after its last point (and its first before its first), so a gain left at its level // made noise at every later contact state, the final nasal's closure and the tail included. - BurstGainAroundGesture($"{name}_release_flow", "/vocal/contacts/0/release_flow", start, duration, PlosiveBurstGain); + GainAroundGesture($"{name}_release_flow", "/vocal/contacts/0/release_flow", start, duration, stopsBehindTheLips ? TongueStopBurstGain : PlosiveBurstGain); + PlaceBurst(name, tokens, place, start, duration); AddAuthoredSpline($"{name}_velum", "/vocal/branches/0/opening", start, duration, (0, VelumClosed), (1, VelumClosed)); } else if (tokens.Contains("fricative")) @@ -1583,6 +1602,13 @@ public static class PatchScript { AddAuthoredSpline($"{name}_contact", "/vocal/contacts/0/opening", start, duration, (0, 0.55f), (1, 0.65f)); } + + // A lateral opens the side stub for its gesture and only then (the stub is closed before and after, as the burst gain is). + if (tokens.Contains("lateral") && _branchPortsByName.ContainsKey(SideStubBranch)) + { + var branch = _branchPorts.FindIndex(candidate => candidate.Name.Equals(SideStubBranch, StringComparison.OrdinalIgnoreCase)); + GainAroundGesture($"{name}_side_stub", $"/vocal/branches/{branch}/opening", start, duration, 1f); + } } // A vowel gesture raises the weight of the descriptor's Story vowel to 1 and holds every other vowel weight at 0 (the schwa and @@ -1634,9 +1660,9 @@ public static class PatchScript AddAuthoredSpline(name, surfacePath, start, duration, (0, normalized), (1, normalized)); } - // Zero before and after the gesture, the gain through it; the ramps are 0.15 of the gesture long and run into the neighbouring + // A surface that is zero before and after the gesture and the given level through it (the plosive burst gain, the lateral's side stub opening); the ramps are 0.15 of the gesture long and run into the neighbouring // phones (the leading one is cut where the word begins). - private void BurstGainAroundGesture(string name, string surfacePath, float start, float duration, float gain) + private void GainAroundGesture(string name, string surfacePath, float start, float duration, float gain) { if (!_controlSurfacesByPath.TryGetValue(surfacePath, out var surface) || surface.MaxValue <= surface.MinValue) { @@ -1657,6 +1683,40 @@ public static class PatchScript _controlSplinesByName[spline.Name] = spline; } + // The place a plosive's burst enters the tract: the contact's position. A labial plosive keeps its burst at the glottal end (position 0) + // and writes nothing while no other plosive has driven the position, so a word of labials is unchanged (the position surface does not + // exist, and the exporter sends a burst in at the glottal end unless its contact has that surface); once any other plosive + // drives it, every labial plosive of the word writes position 0 too, or it would hold that plosive's place. + private bool _burstSiteDriven; + private readonly List<(string Name, float Start, float Duration)> _labialPlosives = []; + + private void PlaceBurst(string name, HashSet tokens, float place, float start, float duration) + { + var labial = tokens.Contains("bilabial") || tokens.Contains("labiodental") || tokens.Contains("labial"); + if (labial && !_burstSiteDriven) + { + _labialPlosives.Add((name, start, duration)); + return; + } + + if (!_burstSiteDriven) + { + _burstSiteDriven = true; + if (_constrictionContacts.Count > 0) + { + var contact = _constrictionContacts[0]; + RegisterNormalizedPrimitiveSurface(OwnerField("/vocal/contacts/0", "position"), contact.Name, "position", contact.Position); + } + + foreach (var (labialName, labialStart, labialDuration) in _labialPlosives) + { + HoldOnSurface($"{labialName}_burst_site", "/vocal/contacts/0/position", labialStart, labialDuration, 0); + } + } + + HoldOnSurface($"{name}_burst_site", "/vocal/contacts/0/position", start, duration, labial ? 0 : place); + } + private static string GestureDescriptor(string ipa, IReadOnlyDictionary fields) { var explicitDescriptor = GetAny(fields, ["descriptor", "desc", "features"], ""); @@ -1691,40 +1751,79 @@ public static class PatchScript // following vowel; no published burst level was retrieved. private const float PlosiveBurstGain = 0.06f; + // The same for a stop sealed by the tongue, whose burst goes in at the closure (LadderInjection) and reaches the lips almost + // unfiltered: at 0.06 the isolated /t/ burst peaked at 0.76 of the following vowel's peak (/b/ 0.19; the rule is under half), at + // 0.03 it is about half that. A DESIGN VALUE set by that measurement. + private const float TongueStopBurstGain = 0.03f; + // Distance from the lip plane to the alveolar ridge in cm. A DESIGN VALUE; no source for the distance was retrieved. 2.555 cm puts the // closure on the centre of ladder cell 20 of 24 of the scaffold tract (17.52 cm at 48 kHz and 350 m/s: (20.5 / 24) * 17.52 = 14.965 cm // from the glottis), the cell grid being what a narrow constriction (scaffold constriction_width .03, e-fold 0.63 cm) can close; it is // the ladder's counterpart of the tube check's closed end after section 37 of 44 (14.73 cm). private const float AlveolarRidgeCentimetersFromLips = 2.555f; - // Normalized position (0 glottis, 1 lips) of a consonant's constriction in the vowel_set path: labials at the lip plane, alveolars - // AlveolarRidgeCentimetersFromLips behind it, every other place at its PlaceTarget fraction of the tract length. - private static float ConsonantPositionFromGlottis(HashSet tokens, float placeFraction, float lengthCentimeters) + // The velar closure, cm behind the lip plane: between the narrowest dorsal section of /u/ (section 26, 6.97 cm) and the narrowest + // pharyngeal section of /a/ (section 13, 12.14 cm) of the Story 2008 area functions. 9.81 cm is the 0.44 fraction the table held before + // it had distances; it lies within 0.05 cm of the centre of oral ladder cell 10 of 24 (0.4375 of the tract), and the closure's Gaussian is narrower + // than a cell; the mean of the two sections, 9.555 cm, would fall 0.3 cm from the nearest cell centre (not tried). + private const float VelarCentimetersFromLips = 9.81f; + + // The lateral's midline: the tongue tip meets the ridge and the flow leaves by the sides, here the side stub's neighbour, the oral path + // through a midline held nearly shut for the gesture (to 0.8, as the nasal). A DESIGN VALUE set so the lateral reads between the vowels and the nasals. + private const float LateralMidlineDiameter = 0.06f; + + // Where a consonant closes. One entry per place, first token match wins. A place with a distance is placed by it, in cm behind the + // lip plane, and its normalized position (0 glottis, 1 lips) is 1 - distance / tract length, for the vowel_set section index and the + // hand-written-script fraction alike. A place with no distance is "fraction only": its fraction of the tract length from the glottis, + // a hand-set DESIGN VALUE, and no later cut may read a distance from it (TryPlaceDistanceCentimeters says absent). + private readonly record struct PlaceEntry(string[] Tokens, float? CentimetersFromLips, float Fraction); + + private static readonly PlaceEntry[] Places = + [ + new(["bilabial", "labiodental", "labial"], 0f, 0f), + new(["dental"], null, 0.88f), + new(["alveolar"], AlveolarRidgeCentimetersFromLips, 0f), + new(["postalveolar", "palatoalveolar"], null, 0.70f), + new(["retroflex"], null, 0.64f), + new(["palatal"], null, 0.56f), + new(["velar"], VelarCentimetersFromLips, 0f), + new(["uvular"], null, 0.34f), + new(["pharyngeal", "epiglottal"], null, 0.20f), + new(["glottal"], null, 0.05f), + ]; + + private const float PlaceFractionWhenUnnamed = 0.70f; + + private static PlaceEntry? FindPlace(HashSet tokens) { - if (tokens.Contains("bilabial") || tokens.Contains("labiodental") || tokens.Contains("labial")) return 1; - if (tokens.Contains("alveolar")) return 1 - AlveolarRidgeCentimetersFromLips / lengthCentimeters; - return placeFraction; + foreach (var place in Places) + { + if (place.Tokens.Any(tokens.Contains)) return place; + } + + return null; } - private static float PlaceTarget(HashSet tokens) + internal static bool TryPlaceDistanceCentimeters(string descriptor, out float centimeters) { - if (tokens.Contains("bilabial") || tokens.Contains("labiodental") || tokens.Contains("labial")) return 0.96f; - if (tokens.Contains("dental")) return 0.88f; - if (tokens.Contains("alveolar")) return 0.78f; - if (tokens.Contains("postalveolar") || tokens.Contains("palatoalveolar")) return 0.70f; - if (tokens.Contains("retroflex")) return 0.64f; - if (tokens.Contains("palatal")) return 0.56f; - if (tokens.Contains("velar")) return 0.44f; - if (tokens.Contains("uvular")) return 0.34f; - if (tokens.Contains("pharyngeal") || tokens.Contains("epiglottal")) return 0.20f; - if (tokens.Contains("glottal")) return 0.05f; - return 0.70f; + centimeters = FindPlace(DescriptorTokens(descriptor))?.CentimetersFromLips ?? float.NaN; + return !float.IsNaN(centimeters); } + // Normalized position of a place along a tract of the given length: 0 glottis, 1 lips. + private static float PlaceFraction(HashSet tokens, float lengthCentimeters) => + FindPlace(tokens) switch + { + { CentimetersFromLips: { } cm } => 1 - cm / lengthCentimeters, + { } place => place.Fraction, + _ => PlaceFractionWhenUnnamed + }; + private static float ConstrictionTarget(HashSet tokens) { if (tokens.Contains("plosive") || tokens.Contains("stop") || tokens.Contains("nasal")) return 0.03f; if (tokens.Contains("fricative")) return 0.16f; + if (tokens.Contains("lateral")) return LateralMidlineDiameter; if (tokens.Contains("approximant")) return 0.48f; if (tokens.Contains("trill") || tokens.Contains("tap") || tokens.Contains("flap")) return 0.24f; return 0.55f; @@ -2714,6 +2813,11 @@ public static class PatchScript var intensity = Math.Clamp(GetFloat(fields, line, 1, "intensity", "gain", "strength"), 0, 2); var pitchRatio = Math.Clamp(GetFloat(fields, line, 1, "pitch_ratio", "f0_ratio"), 0.5f, 2f); var descriptor = GestureDescriptor(ipa, fields); + if (DescriptorTokens(descriptor).Contains("lateral") && !_branchPortsByName.ContainsKey(SideStubBranch)) + { + throw new PatchScriptException(line, $"a lateral gesture needs the `{SideStubBranch}` branch port (the sublingual stub); it is not approximated by a plain approximant"); + } + var name = TryGetAny(fields, ["name", "n"], out var nameText) ? nameText : $"ipa_{SafeName(ipa)}_{_phonemeGestures.Count}"; diff --git a/tests/AquaSynth.Dsl.Tests/NasalCouplingTests.cs b/tests/AquaSynth.Dsl.Tests/NasalCouplingTests.cs index cac20de..638bf3f 100644 --- a/tests/AquaSynth.Dsl.Tests/NasalCouplingTests.cs +++ b/tests/AquaSynth.Dsl.Tests/NasalCouplingTests.cs @@ -91,7 +91,8 @@ public sealed class NasalCouplingTests [Fact] public void WithoutVowelSetTheConstrictionStaysOnTheNormalizedPlaceFraction() { - // The earlier surface semantics are kept for hand-written scripts: the place fraction is the normalized spline value itself. + // The earlier surface semantics are kept for hand-written scripts: the place fraction is the normalized spline value itself. A place + // with a distance in the place table is that distance from the lips over the morphology's length (1 - 2.555 / 17 for /n/, 1 for /m/). var patch = PatchScript.Parse(""" morphology name=oral length_cm=17 diameters=.6,.8,1.2,1.6,1.3,.9 tongue_index=3 tongue_diameter=1.4 constriction_index=4 constriction_diameter=1 lip_opening=1.5 morphology name=nasal length_cm=12 diameters=.05,.35,.6,.8 @@ -107,7 +108,7 @@ public sealed class NasalCouplingTests vocal network=voice freq=150 gain=.2 sustain=.6 """); - Assert.Equal(.78f, Spline(patch, "en_constriction_index", "/vocal/areas/0/area/constriction_index")); - Assert.Equal(.96f, Spline(patch, "em_constriction_index", "/vocal/areas/0/area/constriction_index")); + Assert.Equal(1 - 2.555f / 17, Spline(patch, "en_constriction_index", "/vocal/areas/0/area/constriction_index"), 5); + Assert.Equal(1f, Spline(patch, "em_constriction_index", "/vocal/areas/0/area/constriction_index"), 5); } } diff --git a/tests/AquaSynth.Dsl.Tests/PlaceTableTests.cs b/tests/AquaSynth.Dsl.Tests/PlaceTableTests.cs new file mode 100644 index 0000000..6813300 --- /dev/null +++ b/tests/AquaSynth.Dsl.Tests/PlaceTableTests.cs @@ -0,0 +1,139 @@ +using System.Security.Cryptography; +using System.Text; +using AquaSynth.Dsl; +using AquaSynth.Faust; +using Xunit.Abstractions; + +namespace AquaSynth.Dsl.Tests; + +// The place table in PatchScript owns where a consonant closes (cm behind the lip plane). Text-level tests: no Faust compile. +public sealed class PlaceTableTests(ITestOutputHelper output) +{ + private static string Word(string ipa, string descriptor = "") => IpaGestureExperiment.BuildWordScript( + [(new IpaGestureExperimentTarget("w_0_a", "a", "", 0.18f), 1f), (new IpaGestureExperimentTarget("w_1_c", ipa, descriptor, 0.095f), 1f), (new IpaGestureExperimentTarget("w_2_a", "a", "", 0.18f), 1f)], 140); + + private static string Sha(string text) => Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text)))[..16]; + + private static float ConstrictionIndex(string script, string name) => + PatchScript.Parse(script).ControlSplines + .Single(s => s.Name.StartsWith($"{name}_constriction_index", StringComparison.Ordinal) && s.SurfacePath == "/vocal/areas/0/area/constriction_index").Points[0].Value; + + // Pre-cut values on af0e44f: the vowel_set constriction_index spline value of /b/ and /n/ (and the hash of the whole script and of the + // Faust source for /apa/ and /aba/, which this cut must not change by a byte; the ids are a SHA-256 prefix). + private const float BIndex = 0.8333333f, NIndex = 0.7118056f; + + [Fact] + public void PlaceTableKeepsLabialAndAlveolarPositions() + { + var b = ConstrictionIndex(Word("b"), "w_1_c"); + var n = ConstrictionIndex(Word("n"), "w_1_c"); + output.WriteLine($"b {b:R} n {n:R}"); + foreach (var (ipa, scriptSha, faustSha) in new[] { ("p", "61A96E0AE8C68F52", "28022E56A77DB5B1"), ("b", "5D5A6B653DC5D37D", "2FB84E792B878BFE") }) + { + var script = Word(ipa); + var faust = FaustEmitter.Emit(PatchScript.Parse(script), new FaustExportOptions("pin", SampleRate: 48000)).Source; + Assert.Equal(scriptSha, Sha(script)); + Assert.Equal(faustSha, Sha(faust)); + } + + Assert.Equal(BIndex, b); + Assert.Equal(NIndex, n); + } + + // The side stub is declared only for a word with a lateral: the script and the Faust source of /apa/ /aba/ /ana/ /ata/ are the bytes the scaffold + // gave before the lateral cut (SHA-256 prefixes taken on the head of /k/, 8d50819 plus /d/). + [Fact] + public void ExistingWordsSourcesUnchanged() + { + foreach (var (ipa, scriptSha, faustSha) in new[] + { + ("p", "61A96E0AE8C68F52", "28022E56A77DB5B1"), ("b", "5D5A6B653DC5D37D", "2FB84E792B878BFE"), + ("n", "BDF24A66659D7BD6", "ABEB5D03DC9637DB"), ("t", "9DFBCDB5131F9776", "30E144D54F7E026F"), + }) + { + var script = Word(ipa); + var faust = FaustEmitter.Emit(PatchScript.Parse(script), new FaustExportOptions("pin", SampleRate: 48000)).Source; + Assert.Equal(scriptSha, Sha(script)); + Assert.Equal(faustSha, Sha(faust)); + } + + Assert.Contains(PatchScript.SideStubBranch, Word("l")); + } + + [Fact] + public void UnknownPlaceHasNoDistance() + { + Assert.True(PatchScript.TryPlaceDistanceCentimeters("voiceless_bilabial_plosive", out var lips)); + Assert.Equal(0f, lips); + Assert.True(PatchScript.TryPlaceDistanceCentimeters("voiced_alveolar_nasal", out var ridge)); + Assert.Equal(2.555f, ridge); + // velar: between the narrowest dorsal section of /u/ (section 26, 6.97 cm from the lips) and the narrowest pharyngeal section of /a/ + // (section 13, 12.14 cm); 9.81 cm is the 0.44 fraction of the 17.52 cm tract, which sits within 0.05 cm of the centre of a ladder cell + // (their mean, 9.555 cm, is 0.3 cm from one) + Assert.True(PatchScript.TryPlaceDistanceCentimeters("voiceless_velar_plosive", out var velar)); + Assert.InRange(velar, 6.97f, 12.14f); + Assert.Equal(9.81f, velar, 3); + foreach (var descriptor in new[] { "voiced_uvular_fricative", "voiced_palatal_approximant", "voiced_plosive", "" }) + { + Assert.False(PatchScript.TryPlaceDistanceCentimeters(descriptor, out var cm), descriptor); + Assert.True(float.IsNaN(cm), $"{descriptor} claims {cm}"); + } + } + + private static bool HasPositionSurface(string script) => + PatchScript.Parse(script).ControlSurfaces.Any(surface => surface.Path == "/vocal/contacts/0/position"); + + private static float[] SitePoints(string script, string name) => + PatchScript.Parse(script).ControlSplines.Single(s => s.Name.StartsWith($"{name}_burst_site", StringComparison.Ordinal)).Points.Select(p => p.Value).ToArray(); + + private static string Faust(string script) => FaustEmitter.Emit(PatchScript.Parse(script), new FaustExportOptions("site", SampleRate: 48000)).Source; + + [Fact] + public void AlveolarBurstGoesInAtTheRidgeAndLabialBurstStaysAtTheGlottalEnd() + { + var t = Word("t", "voiceless_alveolar_plosive"); + Assert.True(HasPositionSurface(t)); + Assert.All(SitePoints(t, "w_1_c"), v => Assert.Equal(1 - 2.555f / StoryVowelAreas.MeanLengthCentimeters, v, 4)); + Assert.Contains("_injection0_site", Faust(t)); + + // a word of labials never makes the surface: no spline, and the burst joins the glottal path input as before + var p = Word("p"); + Assert.False(HasPositionSurface(p)); + Assert.DoesNotContain("_injection", Faust(p)); + } + + [Fact] + public void ALabialAfterAnotherPlosiveInTheSameWordIsPlacedAtTheGlottalEnd() + { + IpaGestureExperimentTarget Phone(string label, string ipa, float s) => new($"w_{label}", ipa, "", s); + var script = IpaGestureExperiment.BuildWordScript( + [(Phone("0_b", "b", 0.095f), 1f), (Phone("1_a", "a", 0.18f), 1f), (new IpaGestureExperimentTarget("w_2_t", "t", "voiceless_alveolar_plosive", 0.095f), 1f), (Phone("3_a", "a", 0.18f), 1f), (Phone("4_p", "p", 0.095f), 1f)], 140); + Assert.All(SitePoints(script, "w_0_b"), v => Assert.Equal(0f, v)); + Assert.All(SitePoints(script, "w_4_p"), v => Assert.Equal(0f, v)); + Assert.All(SitePoints(script, "w_2_t"), v => Assert.True(v > 0.8f)); + } + + [Fact] + public void ATongueSealedStopIsClosedCompletelyUntilItsContactOpensAndALabialKeepsItsOldSpline() + { + float[] Diameter(string script) => PatchScript.Parse(script).ControlSplines + .Single(s => s.Name.StartsWith("w_1_c_constriction_diameter", StringComparison.Ordinal)).Points.Select(p => p.Value).ToArray(); + var t = PatchScript.Parse(Word("t", "voiceless_alveolar_plosive")).ControlSplines + .Single(s => s.Name.StartsWith("w_1_c_constriction_diameter", StringComparison.Ordinal)).Points; + Assert.Equal(0f, t[0].Value); + Assert.Equal(0f, t[1].Value); + Assert.Equal(0.65f, (t[1].TimeSeconds - 0.18f) / 0.095f, 3); + var b = Diameter(Word("b")); + Assert.True(b[0] > 0 && Math.Abs(b[0] - b[1]) < 1e-6f, "the lip-sealed stop keeps the 0.03 aperture spline (its source text is pinned)"); + } + + [Fact] + public void VelarBurstGoesInAtTheVelarPlaceAndFurtherBackThanTheRidge() + { + var k = Word("k", "voiceless_velar_plosive"); + Assert.True(HasPositionSurface(k)); + Assert.All(SitePoints(k, "w_1_c"), v => Assert.Equal(1 - 9.81f / StoryVowelAreas.MeanLengthCentimeters, v, 4)); + Assert.Contains("_injection0_site", Faust(k)); + Assert.Equal(1 - 9.81f / StoryVowelAreas.MeanLengthCentimeters, ConstrictionIndex(k, "w_1_c") / BIndex, 4); + } +} diff --git a/tests/AquaSynth.Dsl.Tests/StressAndReleaseRenderTests.cs b/tests/AquaSynth.Dsl.Tests/StressAndReleaseRenderTests.cs index b25670a..1873a27 100644 --- a/tests/AquaSynth.Dsl.Tests/StressAndReleaseRenderTests.cs +++ b/tests/AquaSynth.Dsl.Tests/StressAndReleaseRenderTests.cs @@ -144,11 +144,11 @@ public sealed class StressAndReleaseRenderTests(ITestOutputHelper output) } // /ɑ C ɑ/ at the scaffold's own durations (vowel 0.18 s, consonant 0.095 s). Returns the render and the consonant's [start, end). - private (float[]? Samples, double From, double To) VowelConsonantVowel(string ipa, float frequency = 140) + private (float[]? Samples, double From, double To) VowelConsonantVowel(string ipa, float frequency = 140, string descriptor = "", Func? edit = null) { var script = IpaGestureExperiment.BuildWordScript( - [(new IpaGestureExperimentTarget("w_0_a", "a", "", 0.18f), 1f), (new IpaGestureExperimentTarget("w_1_c", ipa, "", 0.095f), 1f), (new IpaGestureExperimentTarget("w_2_a", "a", "", 0.18f), 1f)], frequency); - return (Render(script, 0.5f), 0.18, 0.275); + [(new IpaGestureExperimentTarget("w_0_a", "a", "", 0.18f), 1f), (new IpaGestureExperimentTarget("w_1_c", ipa, descriptor, 0.095f), 1f), (new IpaGestureExperimentTarget("w_2_a", "a", "", 0.18f), 1f)], frequency); + return (Render(script, 0.5f, edit), 0.18, 0.275); } // Level of the consonant's middle 60 percent against the mean power of the steady parts of the two flanking vowels (the 70 ms before the @@ -220,14 +220,326 @@ public sealed class StressAndReleaseRenderTests(ITestOutputHelper output) [FaustTheory] [InlineData("b", -34, -26)] [InlineData("p", -60, -38)] + [InlineData("d", -34, -26)] public void PlosiveClosureCarriesTheVoiceBarOnlyWhenVoiced(string ipa, double lowDb, double highDb) + { + var db = VoiceBarDb(ipa); + if (double.IsNaN(db)) return; + output.WriteLine($"/{ipa}/ closure {db:F1} dB re the following vowel"); + Assert.InRange(db, lowDb, highDb); + } + + // The closure's deviation (the slow offset of a shutting tract is not sound) against the following vowel's, for a word-initial consonant. + private double VoiceBarDb(string ipa) { var script = IpaGestureExperiment.BuildWordScript( [(new IpaGestureExperimentTarget("w_0_c", ipa, "", 0.095f), 1f), (new IpaGestureExperimentTarget("w_1_a", "a", "", 0.18f), 1f)], 140); var x = Render(script, 0.4f); - var db = 20 * Math.Log10(Math.Max(1e-9, Deviation(x, 0.03, 0.058)) / Deviation(x, 0.15, 0.25)); - output.WriteLine($"/{ipa}/ closure {db:F1} dB re the following vowel"); - Assert.InRange(db, lowDb, highDb); + return x is null ? double.NaN : 20 * Math.Log10(Math.Max(1e-9, Deviation(x, 0.03, 0.058)) / Deviation(x, 0.15, 0.25)); + } + + private static SynthPatch WithoutBurst(SynthPatch p) => + p with { ControlSplines = p.ControlSplines.Where(s => !s.SurfacePath.EndsWith("/release_flow", StringComparison.Ordinal)).ToArray() }; + + // The closure of a stop is the part before its contact opens (0.65 of the gesture), 0.2 of the gesture in from its start, without the burst. + // A tongue closure at the nasal's 0.03 cm aperture leaked: /t/ read -14.3 dB re the flanking vowels against /p/ -22.1 (the lips seal /p/); + // completely closed it reads -24.3. + [Fact] + public void ATongueSealedStopClosureIsAsQuietAsTheLipSealedOne() + { + double Closure(string ipa, string descriptor) + { + var (x, from, to) = VowelConsonantVowel(ipa, descriptor: descriptor, edit: WithoutBurst); + if (x is null) return double.NaN; + var vowels = Math.Sqrt((Math.Pow(Rms(x, from - 0.08, from - 0.01), 2) + Math.Pow(Rms(x, to + 0.03, to + 0.12), 2)) / 2); + return 20 * Math.Log10(Rms(x, from + 0.2 * (to - from), from + 0.65 * (to - from)) / vowels); + } + + var p = Closure("p", "voiceless_bilabial_plosive"); + var t = Closure("t", "voiceless_alveolar_plosive"); + output.WriteLine($"closure re vowels: /p/ {p:F1} dB, /t/ {t:F1} dB"); + if (double.IsNaN(p)) return; + Assert.True(t <= p + 1, $"the /t/ closure ({t:F1} dB) is as quiet as /p/ ({p:F1} dB)"); + } + + [Fact] + public void ATongueSealedStopBurstStaysUnderHalfTheFollowingVowel() + { + var (x, from, to) = VowelConsonantVowel("t", descriptor: "voiceless_alveolar_plosive"); + var (nb, _, _) = VowelConsonantVowel("t", descriptor: "voiceless_alveolar_plosive", edit: WithoutBurst); + if (x is null || nb is null) return; + var burst = x.Zip(nb, (a, b) => a - b).ToArray(); + var peak = Peak(burst, from + 0.6 * (to - from), to + 0.01); + var vowel = Peak(x, to + 0.06, to + 0.17); + output.WriteLine($"/t/ burst peak {peak:F4}, vowel peak {vowel:F4}, ratio {peak / vowel:F2}"); + Assert.True(peak > 0.004, "audible"); + Assert.True(peak < vowel / 2, "under half the following vowel's peak, as /b/"); + } + + // The /b/ test over the alveolar stop, by its table entry (before the entry the phone was unsupported, and a hand script approximated it). + [Fact] + public void AlveolarStopBurstExistsOnlyAroundItsOwnRelease_t() + { + IpaGestureExperimentTarget Phone(string label, string ipa, float seconds) => new($"w_{label}", ipa, "", seconds); + var script = IpaGestureExperiment.BuildWordScript( + [(Phone("0_n", "n", 0.095f), 1f), (Phone("1_a", "a", 0.18f), 1f), (Phone("2_t", "t", 0.095f), 1f), (Phone("3_a", "a", 0.18f), 1f), (Phone("4_n", "n", 0.095f), 1f)], 140); + var withBurst = Render(script, 0.77f); + var withoutBurst = Render(script, 0.77f, WithoutBurst); + if (withBurst is null || withoutBurst is null) return; + var burst = withBurst.Zip(withoutBurst, (x, y) => x - y).ToArray(); + const double tFrom = 0.275, tTo = 0.37; + var atRelease = Peak(burst, tFrom + 0.095 * 0.6, tTo + 0.02); + var leading = Peak(burst, 0, tFrom); + var trailing = Peak(burst, tTo + 0.12, 0.77); + output.WriteLine($"/t/ burst peak {atRelease:F4} at the release, {leading:F5} before the plosive, {trailing:F5} after it"); + Assert.True(atRelease > 0.004, "the burst is a transient of audible size"); + Assert.True(leading < atRelease / 20, "no burst before the plosive's own gesture"); + Assert.True(trailing < atRelease / 10, "no burst after the plosive"); + } + + // Chodroff and Wilson 2014: /t/ burst peak 5216 Hz, /k/ 2868. The band 3.5 to 7 kHz is a DESIGN tolerance around the /t/ figure. + // Peak of the isolated burst's spectrum from 1 kHz up (the weak_spots.burst_spectrum definition: 20 ms Hann window from the release, 400 Hz smoothing). + [Fact] + public void StopTBurstSpectrumIsInTheLiteratureBand() + { + var (x, from, to) = VowelConsonantVowel("t"); + var (nb, _, _) = VowelConsonantVowel("t", edit: WithoutBurst); + if (x is null || nb is null) return; + var start = (int)((from + 0.65 * (to - from)) * Rate); + const int n = 960; + var seg = Enumerable.Range(0, n).Select(i => (x[start + i] - nb[start + i]) * (0.5 - 0.5 * Math.Cos(2 * Math.PI * i / (n - 1)))).ToArray(); + double Power(double hz) + { + double re = 0, im = 0; + for (var i = 0; i < n; i++) { var a = 2 * Math.PI * hz * i / Rate; re += seg[i] * Math.Cos(a); im -= seg[i] * Math.Sin(a); } + return re * re + im * im; + } + + var grid = Enumerable.Range(0, 180).Select(k => 1000.0 + 50 * k).ToArray(); + var power = grid.Select(Power).ToArray(); + var smooth = power.Select((_, k) => Enumerable.Range(Math.Max(0, k - 4), Math.Min(9, power.Length - Math.Max(0, k - 4))).Average(j => power[j])).ToArray(); + var peak = grid[Array.IndexOf(smooth, smooth.Max())]; + output.WriteLine($"/t/ burst spectral peak {peak:F0} Hz"); + Assert.InRange(peak, 3500, 7000); + } + + // /k/ over the /t/ tests: its own place (the velar distance in the place table), closure sealed by the tongue, burst under half the vowel. + // The spec asks /k/'s closure to be at or below /p/'s. It is not: with the voiceless source as /p/ and /t/ have it the closure reads + // -14.9 dB re the vowels (/p/ -20.4, /t/ -24.9). The tract itself seals: with the glottal source's flow scale held at 0 over the gesture the + // same closure reads -22.7 dB, under /p/; with the source on, the 150 Hz voicing is what reaches the output through a closure that leaves + // only a short rear cavity (the glottal pressure that the tract wall radiates is higher the smaller that cavity is). Silencing voicing in + // voiceless stops would change /p/ and /t/ (pinned byte for byte), so the criterion is reported as unmet (fork), not met by a /k/ special case. + // The second assertion is a regression bound on the measured leak, not the criterion. + [Fact] + public void AVelarStopClosureSealsAndItsLeakIsTheGlottalSource() + { + static SynthPatch WithoutSource(SynthPatch p) => + WithoutBurst(p) with + { + ControlSplines = WithoutBurst(p).ControlSplines + .Select(s => s.Name.StartsWith("w_1_c_source_level", StringComparison.Ordinal) + ? s with { Points = s.Points.Select(q => q with { Value = 0, OutValue = 0, InValue = 0 }).ToArray() } + : s).ToArray() + }; + + double Closure(string ipa, string descriptor, Func edit) + { + var (x, from, to) = VowelConsonantVowel(ipa, descriptor: descriptor, edit: edit); + if (x is null) return double.NaN; + var vowels = Math.Sqrt((Math.Pow(Rms(x, from - 0.08, from - 0.01), 2) + Math.Pow(Rms(x, to + 0.03, to + 0.12), 2)) / 2); + return 20 * Math.Log10(Rms(x, from + 0.2 * (to - from), from + 0.65 * (to - from)) / vowels); + } + + var p = Closure("p", "voiceless_bilabial_plosive", WithoutBurst); + var k = Closure("k", "voiceless_velar_plosive", WithoutBurst); + var kSilent = Closure("k", "voiceless_velar_plosive", WithoutSource); + output.WriteLine($"closure re vowels: /p/ {p:F1} dB, /k/ {k:F1} dB, /k/ with the glottal source off {kSilent:F1} dB"); + if (double.IsNaN(p)) return; + Assert.True(kSilent <= p, $"the /k/ closure seals: without the glottal source it is {kSilent:F1} dB, at or under /p/ ({p:F1} dB)"); + Assert.True(k <= -12, $"the /k/ closure with the source is {k:F1} dB (regression bound; the criterion, at or under /p/ {p:F1} dB, is not met)"); + } + + [Fact] + public void AVelarStopBurstStaysUnderHalfTheFollowingVowel() + { + var (x, from, to) = VowelConsonantVowel("k", descriptor: "voiceless_velar_plosive"); + var (nb, _, _) = VowelConsonantVowel("k", descriptor: "voiceless_velar_plosive", edit: WithoutBurst); + if (x is null || nb is null) return; + var burst = x.Zip(nb, (a, b) => a - b).ToArray(); + var peak = Peak(burst, from + 0.6 * (to - from), to + 0.01); + var vowel = Peak(x, to + 0.06, to + 0.17); + output.WriteLine($"/k/ burst peak {peak:F4}, vowel peak {vowel:F4}, ratio {peak / vowel:F2}"); + Assert.True(peak > 0.004, "audible"); + Assert.True(peak < vowel / 2, "under half the following vowel's peak, as /t/ and /b/"); + } + + [Fact] + public void VelarStopBurstExistsOnlyAroundItsOwnRelease_k() + { + IpaGestureExperimentTarget Phone(string label, string ipa, float seconds) => new($"w_{label}", ipa, "", seconds); + var script = IpaGestureExperiment.BuildWordScript( + [(Phone("0_n", "n", 0.095f), 1f), (Phone("1_a", "a", 0.18f), 1f), (Phone("2_k", "k", 0.095f), 1f), (Phone("3_a", "a", 0.18f), 1f), (Phone("4_n", "n", 0.095f), 1f)], 140); + var withBurst = Render(script, 0.77f); + var withoutBurst = Render(script, 0.77f, WithoutBurst); + if (withBurst is null || withoutBurst is null) return; + var burst = withBurst.Zip(withoutBurst, (x, y) => x - y).ToArray(); + const double kFrom = 0.275, kTo = 0.37; + var atRelease = Peak(burst, kFrom + 0.095 * 0.6, kTo + 0.02); + var leading = Peak(burst, 0, kFrom); + var trailing = Peak(burst, kTo + 0.12, 0.77); + output.WriteLine($"/k/ burst peak {atRelease:F4} at the release, {leading:F5} before the plosive, {trailing:F5} after it"); + Assert.True(atRelease > 0.004, "the burst is a transient of audible size"); + Assert.True(leading < atRelease / 20, "no burst before the plosive's own gesture"); + Assert.True(trailing < atRelease / 10, "no burst after the plosive"); + } + + // Chodroff and Wilson 2014: /k/ burst peak 2868 Hz. The band 2 to 4 kHz is a DESIGN tolerance around it, fixed before the first measurement. + // The peak is judged against the literature and against /t/ measured the same way (a velar burst enters further back, so its peak is lower). + // The centroid is reported, not asserted: over this 1 to 10 kHz grid /k/ reads 4626 Hz against /t/ 3629 Hz, higher though its peak is lower, + // because the /k/ burst has broadband energy above its peak, and no literature centroid for /k/ was retrieved to judge it by. + private (double Peak, double Centroid) BurstSpectrum(string ipa, string descriptor) + { + var (x, from, to) = VowelConsonantVowel(ipa, descriptor: descriptor); + var (nb, _, _) = VowelConsonantVowel(ipa, descriptor: descriptor, edit: WithoutBurst); + if (x is null || nb is null) return (double.NaN, double.NaN); + var start = (int)((from + 0.65 * (to - from)) * Rate); + const int n = 960; + var seg = Enumerable.Range(0, n).Select(i => (x[start + i] - nb[start + i]) * (0.5 - 0.5 * Math.Cos(2 * Math.PI * i / (n - 1)))).ToArray(); + double Power(double hz) + { + double re = 0, im = 0; + for (var i = 0; i < n; i++) { var a = 2 * Math.PI * hz * i / Rate; re += seg[i] * Math.Cos(a); im -= seg[i] * Math.Sin(a); } + return re * re + im * im; + } + + var grid = Enumerable.Range(0, 180).Select(k => 1000.0 + 50 * k).ToArray(); + var power = grid.Select(Power).ToArray(); + var smooth = power.Select((_, k) => Enumerable.Range(Math.Max(0, k - 4), Math.Min(9, power.Length - Math.Max(0, k - 4))).Average(j => power[j])).ToArray(); + return (grid[Array.IndexOf(smooth, smooth.Max())], grid.Zip(power, (f, w) => f * w).Sum() / power.Sum()); + } + + [Fact] + public void StopKBurstPeakIsInTheLiteratureBandAndLowerThanT() + { + var (peak, centroid) = BurstSpectrum("k", "voiceless_velar_plosive"); + if (double.IsNaN(peak)) return; + var (tPeak, tCentroid) = BurstSpectrum("t", "voiceless_alveolar_plosive"); + output.WriteLine($"/k/ burst spectral peak {peak:F0} Hz, centroid {centroid:F0} Hz; /t/ peak {tPeak:F0} Hz, centroid {tCentroid:F0} Hz"); + Assert.InRange(peak, 2000, 4000); + Assert.True(peak < tPeak, $"/k/ peak {peak:F0} Hz is under /t/ {tPeak:F0} Hz"); + } + + // /d/ is /t/ with voicing, as /b/ is /p/ with voicing: the same gesture, so the splines that shape the tract are identical and the two differ + // in the voice bar through the closure and in nothing else the burst carries. Chodroff and Wilson 2014: /d/ burst peak 4878 Hz, /t/ 5216. + [Fact] + public void StopDDiffersFromStopTOnlyByVoicing() + { + static SynthPatch Word(string ipa) => PatchScript.Parse(IpaGestureExperiment.BuildWordScript( + [(new IpaGestureExperimentTarget("w_0_a", "a", "", 0.18f), 1f), (new IpaGestureExperimentTarget("w_1_c", ipa, "", 0.095f), 1f), (new IpaGestureExperimentTarget("w_2_a", "a", "", 0.18f), 1f)], 140)); + static string[] Shape(SynthPatch p) => p.ControlSplines + .Where(s => s.Name.StartsWith("w_1_c", StringComparison.Ordinal) && !s.Name.StartsWith("w_1_c_source", StringComparison.Ordinal)) + .Select(s => s.SurfacePath + "|" + s.Name[5..] + "|" + string.Join(";", s.Points.Select(q => $"{q.TimeSeconds:R}:{q.Value:R}"))) + .Order().ToArray(); + var t = Shape(Word("t")); + var d = Shape(Word("d")); + output.WriteLine($"/t/ {t.Length} consonant splines, /d/ {d.Length}"); + Assert.NotEmpty(d); + Assert.Equal(t, d); + + var tDb = VoiceBarDb("t"); + var dDb = VoiceBarDb("d"); + output.WriteLine($"closure re the following vowel: /t/ {tDb:F1} dB, /d/ {dDb:F1} dB"); + if (double.IsNaN(tDb)) return; + Assert.True(dDb >= tDb + 8, $"the voice bar lifts the /d/ closure ({dDb:F1} dB) at least 8 dB over /t/ ({tDb:F1} dB)"); + var (dPeak, _) = BurstSpectrum("d", ""); + var (tPeak, _) = BurstSpectrum("t", ""); + output.WriteLine($"burst spectral peak: /t/ {tPeak:F0} Hz, /d/ {dPeak:F0} Hz"); + Assert.True(Math.Abs(dPeak - tPeak) <= 1000, $"/d/ burst peak {dPeak:F0} Hz within 1 kHz of /t/ {tPeak:F0} Hz"); + } + + [Fact] + public void StopDBurstExistsOnlyAroundItsOwnRelease() + { + IpaGestureExperimentTarget Phone(string label, string ipa, float seconds) => new($"w_{label}", ipa, "", seconds); + var script = IpaGestureExperiment.BuildWordScript( + [(Phone("0_n", "n", 0.095f), 1f), (Phone("1_a", "a", 0.18f), 1f), (Phone("2_d", "d", 0.095f), 1f), (Phone("3_a", "a", 0.18f), 1f), (Phone("4_n", "n", 0.095f), 1f)], 140); + var withBurst = Render(script, 0.77f); + var withoutBurst = Render(script, 0.77f, WithoutBurst); + if (withBurst is null || withoutBurst is null) return; + var burst = withBurst.Zip(withoutBurst, (x, y) => x - y).ToArray(); + const double dFrom = 0.275, dTo = 0.37; + var atRelease = Peak(burst, dFrom + 0.095 * 0.6, dTo + 0.02); + var leading = Peak(burst, 0, dFrom); + var trailing = Peak(burst, dTo + 0.12, 0.77); + output.WriteLine($"/d/ burst peak {atRelease:F4} at the release, {leading:F5} before the plosive, {trailing:F5} after it"); + Assert.True(atRelease > 0.004, "the burst is a transient of audible size"); + Assert.True(leading < atRelease / 20, "no burst before the plosive's own gesture"); + Assert.True(trailing < atRelease / 10, "no burst after the plosive"); + } + + // /l/ over /ɑ _ ɑ/ at the scaffold's durations. Envelope of the consonant's middle 60 percent at the harmonics of f0 (the largest power within a + // quarter of f0 of k * f0), in dB: the spectrum of a periodic source is a line spectrum, so extrema are read from the harmonic envelope. + private (double[] Hz, double[] Db) HarmonicEnvelope(float[] x, double from, double to, double f0 = 140, double maxHz = 5000) + { + var a = (int)((from + 0.2 * (to - from)) * Rate); + var n = (int)(0.6 * (to - from) * Rate); + var seg = Enumerable.Range(0, n).Select(i => x[a + i] * (0.5 - 0.5 * Math.Cos(2 * Math.PI * i / (n - 1)))).ToArray(); + double Power(double hz) + { + double re = 0, im = 0; + for (var i = 0; i < n; i++) { var w = 2 * Math.PI * hz * i / Rate; re += seg[i] * Math.Cos(w); im -= seg[i] * Math.Sin(w); } + return re * re + im * im; + } + + var hz = new List(); + var db = new List(); + for (var k = 1; k * f0 < maxHz; k++) + { + hz.Add(k * f0); + db.Add(10 * Math.Log10(Math.Max(1e-30, Enumerable.Range(-3, 7).Max(j => Power(k * f0 + j * f0 / 12))))); + } + + return (hz.ToArray(), db.ToArray()); + } + + // The lateral's own dip: the envelope with the side stub open less the same render with it shut, at its lowest between 2 and 3 kHz. The + // difference isolates the stub's antiresonance from the valleys between the vowel's formants, which the shut render has as deep. + private static (double DipHz, double Depth) LateralDip(double[] hz, double[] db, double[] shutDb) + { + var i = Enumerable.Range(0, hz.Length).Where(j => hz[j] >= 2000 && hz[j] <= 3000).MinBy(j => db[j] - shutDb[j]); + return (hz[i], shutDb[i] - db[i]); + } + + // The lateral's antiresonance: the closed side stub beside the oral path (sublingual cavity). With the stub open the 2 to 3 kHz harmonics fall at + // least 6 dB under the same render with it shut (a DESIGN criterion; no /l/ spectrum was retrieved). The lateral is continuous: the render with + // and without release_flow is the same render. + [Fact] + public void LateralHasAnAntiresonanceAndNoBurst() + { + var (x, from, to) = VowelConsonantVowel("l"); + if (x is null) return; + var (hz, db) = HarmonicEnvelope(x, from, to); + var (shut, _, _) = VowelConsonantVowel("l", edit: WithShutSideStub); + var (_, shutDb) = HarmonicEnvelope(shut!, from, to); + var (dipHz, depth) = LateralDip(hz, db, shutDb); + var (nb, _, _) = VowelConsonantVowel("l", edit: WithoutBurst); + output.WriteLine($"/l/ stub open {depth:F1} dB under stub shut at {dipHz:F0} Hz (2-3 kHz); max |with - without release_flow| {x.Zip(nb!, (a, b) => Math.Abs(a - b)).Max():G3}"); + Assert.True(depth >= 6, $"the 2-3 kHz dip is {depth:F1} dB under the shut-stub render"); + Assert.Equal(x, nb); + } + + private static SynthPatch WithShutSideStub(SynthPatch p) => + p with { ControlSplines = p.ControlSplines.Where(s => s.SurfacePath != "/vocal/branches/1/opening").ToArray() }; + + // Between the vowels and the nasals in level: -4 to -10 dB re the flanking vowels (a DESIGN band; /n/ reads -9.6, /m/ -11.8 in the same measure). + [Fact] + public void LateralLevelIsBetweenTheVowelsAndTheNasals() + { + var (x, from, to) = VowelConsonantVowel("l"); + if (x is null) return; + var db = ConsonantDbReVowels(x, from, to); + output.WriteLine($"/l/ {db:F1} dB re the flanking vowels"); + Assert.InRange(db, -10, -4); } } diff --git a/tests/AquaSynth.Dsl.Tests/WeksaPhonesTests.cs b/tests/AquaSynth.Dsl.Tests/WeksaPhonesTests.cs index 34349af..3f0994f 100644 --- a/tests/AquaSynth.Dsl.Tests/WeksaPhonesTests.cs +++ b/tests/AquaSynth.Dsl.Tests/WeksaPhonesTests.cs @@ -68,6 +68,77 @@ public sealed class WeksaPhonesTests Assert.Empty(plan.Phones); } + [Fact] + public void WeksaWordPlanAcceptsT() + { + var word = WeksaPhonesReader.Parse(Line($"[{Ph("ɑ", "none")},{Ph("t", "none")},{Ph("ɑ", "none")}]")); + var plan = WeksaWordPlan.From(word, "ata"); + Assert.True(plan.Renderable); + Assert.Empty(plan.Unsupported); + // t names a plosive, not the approximant GestureDescriptor falls back to for a phone the table lacks + Assert.True(PatchScript.TryIpaDescriptor("t", out var descriptor)); + Assert.Equal("voiceless_alveolar_plosive", descriptor); + var patch = PatchScript.Parse(IpaGestureExperiment.BuildWordScript(plan.Phones.ToArray(), 140)); + Assert.Contains(patch.ControlSplines, s => s.SurfacePath == "/vocal/contacts/0/release_flow" && s.Name.StartsWith("ata_1_t", StringComparison.Ordinal)); + } + + [Fact] + public void WeksaWordPlanAcceptsD() + { + var word = WeksaPhonesReader.Parse(Line($"[{Ph("ɑ", "none")},{Ph("d", "none")},{Ph("ɑ", "none")}]")); + var plan = WeksaWordPlan.From(word, "ada"); + Assert.True(plan.Renderable, string.Join(",", plan.Unsupported.Select(p => p.Ipa))); + Assert.Empty(plan.Unsupported); + // d names a plosive, not the approximant GestureDescriptor falls back to for a phone the table lacks + Assert.True(PatchScript.TryIpaDescriptor("d", out var descriptor)); + Assert.Equal("voiced_alveolar_plosive", descriptor); + var patch = PatchScript.Parse(IpaGestureExperiment.BuildWordScript(plan.Phones.ToArray(), 140)); + Assert.Contains(patch.ControlSplines, s => s.SurfacePath == "/vocal/contacts/0/release_flow" && s.Name.StartsWith("ada_1_d", StringComparison.Ordinal)); + } + + // The lateral gate passed (a closed side stub on the oral path gives a dip: LateralHasAnAntiresonanceAndNoBurst), so /l/ is in the table, as a + // lateral and never as the approximant GestureDescriptor falls back to. + [Fact] + public void LateralGateOutcomeIsRecorded() + { + var word = WeksaPhonesReader.Parse(Line($"[{Ph("ɑ", "none")},{Ph("l", "none")},{Ph("ɑ", "none")}]")); + var plan = WeksaWordPlan.From(word, "ala"); + Assert.True(plan.Renderable, string.Join(",", plan.Unsupported.Select(p => p.Ipa))); + Assert.Empty(plan.Unsupported); + Assert.True(PatchScript.TryIpaDescriptor("l", out var descriptor)); + Assert.Equal("voiced_alveolar_lateral_approximant", descriptor); + Assert.NotEqual("voiced_alveolar_approximant", descriptor); + var script = IpaGestureExperiment.BuildWordScript(plan.Phones.ToArray(), 140); + var patch = PatchScript.Parse(script); + Assert.Contains(patch.BranchPorts, b => b.Name == PatchScript.SideStubBranch); + Assert.Contains(patch.ControlSplines, s => s.SurfacePath == "/vocal/branches/1/opening" && s.Name.StartsWith("ala_1_l", StringComparison.Ordinal)); + // continuous, no burst: the lateral has no release_flow spline at all + Assert.DoesNotContain(patch.ControlSplines, s => s.SurfacePath == "/vocal/contacts/0/release_flow" && s.Name.StartsWith("ala_1_l", StringComparison.Ordinal)); + } + + // A lateral gesture on a scaffold without the side stub is refused, not rendered as an approximant. + [Fact] + public void ALateralWithoutTheSideStubIsRefused() + { + var script = IpaGestureExperiment.BuildWordScript([(new IpaGestureExperimentTarget("w_0_a", "a", "", 0.18f), 1f)], 140) + + "\nphoneme_gesture name=w_hand ipa=l descriptor=voiced_alveolar_lateral_approximant start=.2 dur=.1 intensity=1"; + var error = Assert.Throws(() => PatchScript.Parse(script)); + Assert.Contains(PatchScript.SideStubBranch, error.Message); + } + + [Fact] + public void WeksaWordPlanAcceptsK() + { + var word = WeksaPhonesReader.Parse(Line($"[{Ph("ɑ", "none")},{Ph("k", "none")},{Ph("ɑ", "none")}]")); + var plan = WeksaWordPlan.From(word, "aka"); + Assert.True(plan.Renderable, string.Join(",", plan.Unsupported.Select(p => p.Ipa))); + Assert.Empty(plan.Unsupported); + Assert.True(PatchScript.TryIpaDescriptor("k", out var descriptor)); + Assert.Equal("voiceless_velar_plosive", descriptor); + var patch = PatchScript.Parse(IpaGestureExperiment.BuildWordScript(plan.Phones.ToArray(), 140)); + Assert.Contains(patch.ControlSplines, s => s.SurfacePath == "/vocal/contacts/0/release_flow" && s.Name.StartsWith("aka_1_k", StringComparison.Ordinal)); + } + [Fact] public void NearCloseVowelsRenderAsVowelsWithTheirOwnStoryShape() { @@ -135,7 +206,7 @@ public sealed class WeksaPhonesTests { foreach (var ipa in new[] { "n", "i", "b", "u", "a", "ə", "ɪ", "ʊ", "e", "ɛ", "æ", "ʌ", "ɑ", "ɔ", "o" }) Assert.True(PatchScript.TryIpaDescriptor(ipa, out var descriptor) && descriptor.EndsWith("_vowel", StringComparison.Ordinal) == "iuaəɪʊeɛæʌɑɔo".Contains(ipa)); - foreach (var ipa in new[] { "ŋ", "θ", "ɹ", "l" }) + foreach (var ipa in new[] { "ŋ", "θ", "ɹ" }) Assert.False(PatchScript.TryIpaDescriptor(ipa, out _)); } } diff --git a/tools/aqua-listen/README.md b/tools/aqua-listen/README.md index 855b828..44b9f5a 100644 --- a/tools/aqua-listen/README.md +++ b/tools/aqua-listen/README.md @@ -24,7 +24,7 @@ Privacy: the page has no network access at all (the browser is told to refuse it ``` node build.mjs --manifest kit.json --clips ./clips --out ./dist # writes dist/listen.html -node test.mjs # 16 checks, no dependencies, no browser +node test.mjs # 17 checks, no dependencies, no browser node smoke.mjs dist/listen.html # optional browser smoke, see below ``` @@ -54,6 +54,17 @@ The blank verdict files this was meant to match were not found under `/opt/commo `PLAYWRIGHT_CORE=/path/to/playwright-core CHROME=/path/to/chrome node smoke.mjs dist/listen.html` ticks through consent, answers a clip, reloads, exports, and fails on any network request or page error. It is not in `node test.mjs` because the repo carries no browser dependency. +### Kits in the repo + +`kits/tract-weak-spots` and `kits/tract-stops` hold a `kit.json`, the `phones/*.json` that WordRender reads, and a `render.sh` that renders the clips (`render.sh `; needs CULTLIB_ROOT and AQUASYNTH_FAUST_HOME, see the script header). Build a page from the rendered clips: +`node tools/aqua-listen/build.mjs --manifest tools/aqua-listen/kits/tract-stops/kit.json --clips /clips --out /dist`. The titles in those kit.json files carry measured figures, a prediction and a falsifier; the page hides them from the listener. + +### Pictures and burst notes (for a reader who cannot listen) + +`clip_images.py [--kit kit.json --renders ]` writes `.spec.png` (spectrogram, Hann 512 / hop 128, 0-8 kHz, -90..0 dB re the clip peak) and `.wave.png` (fixed +-0.5) for every wav, byte-identical between runs, and with `--kit --renders` also `burst-notes.md` and `burst-notes.json` next to the images: per plosive closure start, release (start of the with-minus-without `--no-burst` difference above 1/20 of its peak), burst end and the window to look in. Both kits' `render.sh` end with it. Requirement: `python3` with numpy and matplotlib (Agg), set `PYTHON=` to one that has them; no scipy, nothing for the dotnet build. Test: `python3 -I test_clip_images.py`. + +`build.mjs --images ` embeds a clip's two pictures in a collapsed "Pictures of this sound" block (matched by the wav's stem) and lets the page's policy admit `data:` images, only then. The burst notes are never read by the build and never enter `listen.html` or `kit.json`: give the page to the listener and the notes to whoever reads the pictures. + ## Files -`logic.cjs` validation, ordering, export shape (shared by build, page, tests) · `build.mjs` assembles the page · `web/` page template, glue and styles · `test.mjs` · `smoke.mjs` +`clip_images.py` + `test_clip_images.py` pictures and burst notes · `kits/` · `logic.cjs` validation, ordering, export shape (shared by build, page, tests) · `build.mjs` assembles the page · `web/` page template, glue and styles · `test.mjs` · `smoke.mjs` diff --git a/tools/aqua-listen/build.mjs b/tools/aqua-listen/build.mjs index 16526ab..938f5bb 100644 --- a/tools/aqua-listen/build.mjs +++ b/tools/aqua-listen/build.mjs @@ -1,10 +1,11 @@ #!/usr/bin/env node // Builds one self-contained listening page. Usage: -// node build.mjs --manifest kit.json --clips DIR --out DIR (writes DIR/listen.html; --clips defaults to the manifest's folder) +// node build.mjs --manifest kit.json --clips DIR --out DIR [--images DIR] (writes DIR/listen.html; --clips defaults to the manifest's folder) +// --images: clip_images.py's output; a clip whose wav stem has .spec.png and .wave.png there gets both in a collapsed block on its card. import { readFileSync, writeFileSync, mkdirSync, existsSync, statSync } from "node:fs"; import { createHash } from "node:crypto"; import { createRequire } from "node:module"; -import { dirname, extname, join, resolve } from "node:path"; +import { basename, dirname, extname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; const here = dirname(fileURLToPath(import.meta.url)); @@ -12,7 +13,7 @@ const L = createRequire(import.meta.url)("./logic.cjs"); const MIME = { ".wav": "audio/wav", ".mp3": "audio/mpeg", ".ogg": "audio/ogg", ".flac": "audio/flac" }; // Throws one Error listing every manifest problem and every missing file. Returns {html, hash, bytes}. -export function buildKit(manifest, clipsDir) { +export function buildKit(manifest, clipsDir, imagesDir) { const errs = L.validateManifest(manifest); if (errs.length === 0) for (const c of manifest.clips) { const f = resolve(clipsDir, c.file); @@ -27,13 +28,20 @@ export function buildKit(manifest, clipsDir) { h.update(buf); audio[c.id] = { mime: MIME[extname(c.file).toLowerCase()], b64: buf.toString("base64") }; } + const images = {}; + if (imagesDir) for (const c of manifest.clips) { + const stem = basename(c.file, extname(c.file)), pics = [["spectrogram", "spec"], ["waveform", "wave"]].map(([label, k]) => [label, resolve(imagesDir, `${stem}.${k}.png`)]); + if (!pics.every(([, f]) => existsSync(f))) continue; + images[c.id] = pics.map(([label, f]) => { const buf = readFileSync(f); h.update(buf); return { label, b64: buf.toString("base64") }; }); + } const hash = h.digest("hex").slice(0, 8); - const data = JSON.stringify({ manifest, hash, audio }).replace(/{const module={exports:{}};${readFileSync(join(here, "logic.cjs"), "utf8")}\nreturn module.exports})();`; const part = (n) => readFileSync(join(here, "web", n), "utf8"); const esc = (s) => s.replace(/&/g, "&").replace(/ Object.keys(images).length ? "media-src blob:; img-src data:" : "media-src blob:") .replace("/*TITLE*/", () => esc(manifest.kit.title)).replace("/*STYLES*/", () => part("styles.css")) .replace("/*DATA*/", () => data).replace("/*LOGIC*/", () => logic.replace(/<\/script/gi, "<\\/script")).replace("/*PAGE*/", () => part("page.js").replace(/<\/script/gi, "<\\/script")); return { html, hash, bytes: Buffer.byteLength(html) }; @@ -42,10 +50,10 @@ export function buildKit(manifest, clipsDir) { if (process.argv[1] === fileURLToPath(import.meta.url)) { const a = process.argv.slice(2), opt = (n) => { const i = a.indexOf(n); return i < 0 ? undefined : a[i + 1]; }; const mf = opt("--manifest"), out = opt("--out"); - if (!mf || !out) { console.error("usage: node build.mjs --manifest kit.json --clips DIR --out DIR"); process.exit(2); } + if (!mf || !out) { console.error("usage: node build.mjs --manifest kit.json --clips DIR --out DIR [--images DIR]"); process.exit(2); } try { const manifest = JSON.parse(readFileSync(mf, "utf8")); - const r = buildKit(manifest, resolve(opt("--clips") ?? dirname(resolve(mf)))); + const r = buildKit(manifest, resolve(opt("--clips") ?? dirname(resolve(mf))), opt("--images") && resolve(opt("--images"))); mkdirSync(out, { recursive: true }); writeFileSync(join(out, "listen.html"), r.html); console.log(`built ${join(out, "listen.html")}\nclips ${manifest.clips.length} bytes ${r.bytes} hash ${r.hash}`); diff --git a/tools/aqua-listen/clip_images.py b/tools/aqua-listen/clip_images.py new file mode 100644 index 0000000..bf5c9ca --- /dev/null +++ b/tools/aqua-listen/clip_images.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""A spectrogram and a waveform PNG for every clip of a kit, and (with --kit and --renders) the burst notes. + + python3 -I clip_images.py [--kit kit.json --renders ] + +Needs numpy and matplotlib (Agg); no scipy. Writes /.spec.png and .wave.png for each *.wav in (sorted). +The images are deterministic: fixed figure size, dpi and STFT (Hann 512, hop 128, 0-8 kHz, -90..0 dB re the clip's loudest bin), no PNG +metadata, so two runs give identical bytes. Nothing in the pictures marks a time; the notes do, and they are written next to +(burst-notes.md and burst-notes.json) and are not read by build.mjs. + +--renders is the out-root of a kit's render.sh (out//script.aqua and out//.wav, outnb//.wav). For each clip of --kit the +note gives, for each plosive gesture of the word's script: closure start (the gesture's start), release = start of the with-minus-without +(--no-burst) waveform difference above 1/20 of its peak in the gesture's window (the definition PlosiveBurstExistsOnlyAroundItsOwnRelease +uses), burst end = last such sample of the same burst, and the window to look at. A --no-burst clip says 'no burst expected'. +""" +import argparse, json, re, sys, wave +from pathlib import Path +import numpy as np +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt + +WIN, HOP, FMAX = 512, 128, 8000.0 +DB_LO = -90.0 + + +def read_wav(path): + with wave.open(str(path), "rb") as w: + assert w.getsampwidth() == 2 and w.getnchannels() == 1, f"{path}: expected 16-bit mono" + return np.frombuffer(w.readframes(w.getnframes()), dtype=" and outnb/, per plosive.""" + script = root / "out" / stem / "script.aqua" + if not script.exists(): + return {"burst": "not measured: no render of this word under out/"} + ps = plosives(script) + if not ps: + return {"burst": "none: the word has no plosive"} + twin = root / "outnb" / stem / f"{stem}.wav" + if not twin.exists(): + return {"burst": "not measured: no --no-burst twin render of this word"} + a, sr = read_wav(root / "out" / stem / f"{stem}.wav") + b, _ = read_wav(twin) + n = min(len(a), len(b)); d = np.abs(a[:n] - b[:n]) + rows = [] + for start, dur in ps: + lo, hi = int(start * sr), min(n, int((start + dur + 0.12) * sr)) + peak = d[lo:hi].max() if hi > lo else 0.0 + hot = np.flatnonzero(d[lo:hi] > peak / 20) + lo if peak > 0 else np.array([], dtype=int) + if not len(hot): + rows.append({"closure_start_s": start, "release_s": None, "burst_end_s": None, "look_s": None, "note": "no difference between the renders"}) + continue + end = hot[0] + for i in hot[1:]: # the first burst: stop at a gap over 5 ms + if i - end > 0.005 * sr: break + end = i + rel, fin = hot[0] / sr, end / sr + rows.append({"closure_start_s": round(start, 4), "release_s": round(rel, 4), "burst_end_s": round(fin, 4), + "difference_peak": round(float(peak), 4), "look_s": [round(max(0.0, rel - 0.03), 4), round(fin + 0.05, 4)]}) + return {"burst": rows} + + +def render_of(root, clip): + """The out/ render a clip file was copied from (byte-identical wav), or None: two clip ids can be one render.""" + data = clip.read_bytes() if clip.exists() else None + for d in sorted((root / "out").glob("*/")): + w = d / f"{d.name}.wav" + if data is not None and w.exists() and w.read_bytes() == data: return d.name + return None + + +def notes(kit, root, clips): + out = [] + for c in kit["clips"]: + stem = Path(c["file"]).stem + if stem.endswith("_noburst"): + out.append({"id": c["id"], "file": c["file"], "word": stem[:-8], "burst": "no burst expected"}); continue + word = render_of(root, clips / c["file"]) or stem + out.append({"id": c["id"], "file": c["file"], "word": word, **burst_note(root, word)}) + return out + + +def notes_md(rows): + lines = ["# Burst notes", "", "Times in seconds from the clip start. Release = start of the with-minus-without difference above 1/20 of its peak. Look in the spectrogram at 2-8 kHz inside the window.", ""] + lines += ["| clip id | file | closure start | release | burst end | look in |", "|---|---|---|---|---|---|"] + for r in rows: + if isinstance(r["burst"], str): + lines.append(f"| {r['id']} | {r['file']} | {r['burst']} | | | |"); continue + for p in r["burst"]: + look = f"{p['look_s'][0]} to {p['look_s'][1]}" if p.get("look_s") else p.get("note", "") + lines.append(f"| {r['id']} | {r['file']} | {p['closure_start_s']} | {p['release_s']} | {p['burst_end_s']} | {look} |") + return "\n".join(lines) + "\n" + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("clips"); ap.add_argument("images"); ap.add_argument("--kit"); ap.add_argument("--renders") + a = ap.parse_args() + clips, out = Path(a.clips), Path(a.images) + if bool(a.kit) != bool(a.renders): ap.error("--kit and --renders go together") + wavs = sorted(clips.glob("*.wav")) + if not wavs: sys.exit(f"no .wav files in {clips}") + out.mkdir(parents=True, exist_ok=True) + for w in wavs: images(w, out) + print(f"{len(wavs)} clips, {2 * len(wavs)} images in {out}") + if a.kit: + rows = notes(json.loads(Path(a.kit).read_text(encoding="utf-8")), Path(a.renders), clips) + (out.parent / "burst-notes.json").write_text(json.dumps(rows, indent=1) + "\n", encoding="utf-8") + (out.parent / "burst-notes.md").write_text(notes_md(rows), encoding="utf-8") + print(f"burst notes for {len(rows)} clips in {out.parent}") + + +if __name__ == "__main__": + main() diff --git a/tools/aqua-listen/kits/tract-stops/kit.json b/tools/aqua-listen/kits/tract-stops/kit.json new file mode 100644 index 0000000..cc7d44b --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/kit.json @@ -0,0 +1,78 @@ +{ + "kit": { + "id": "tract-stops-1", + "title": "Articulatory vocal tract: stops and the lateral", + "note": "Short synthetic words from the tract model. Tell us what you hear; there are no wrong answers. Some are pairs: rate both, then say which you prefer." + }, + "clips": [ + { + "id": "stop-t", + "file": "ata.wav", + "group": "stop", + "title": "/ɑtɑ/ at 140 Hz. Measured: closure -24.9 dB re the flanking vowels (/p/ -20.4), burst peak 0.39 of the vowel peak, burst spectral peak 3950 Hz (inside 3.5-7 kHz; CW14 5216), burst only around its own release. PREDICTION: heard as 'ah-tah', a crisp tongue-tip tap-and-release with a sharp click on the release; the twin without the burst loses that click and sounds softer, like a stop with no edge. FALSIFIED if heard as 'ah-kah' or 'ah-pah', or if the burst is not audible against the no-burst twin.", + "feel": "'ah-tah': the tongue tip taps behind the teeth between two open vowels", + "pair": "stop-t-noburst" + }, + { + "id": "stop-t-noburst", + "file": "ata_noburst.wav", + "group": "stop", + "title": "/ɑtɑ/ rendered with --no-burst (plosive release_flow splines dropped), otherwise the same word. Measured: no burst energy by construction (the release_flow splines are dropped); closure and transitions as in the burst render. PREDICTION: still a stop by closure and formant transitions, but soft, with no click at the release. FALSIFIED if it sounds the same as the burst twin.", + "feel": "the same word with a gentler, softer release", + "pair": "stop-t" + }, + { + "id": "stop-d", + "file": "ada.wav", + "group": "stop", + "title": "/ɑdɑ/ at 140 Hz. Measured: voice bar -28.9 dB re the following vowel (/t/ -38.7, /b/ -29.2, /p/ -40.1), burst spectral peak 3950 Hz, same as /t/ (CW14 d 4878), burst peak 0.0603 at the release and none before or after; the 7 consonant splines are identical to /t/'s. PREDICTION: 'ah-dah': a tongue-tip stop with voicing through the closure, a softer release than /t/. FALSIFIED if heard as /t/ or /n/ or /l/, or if no voicing is audible in the closure.", + "feel": "'ah-dah': the tongue tip taps behind the teeth, with the voice kept humming", + "pair": "stop-d-noburst" + }, + { + "id": "stop-d-noburst", + "file": "ada_noburst.wav", + "group": "stop", + "title": "/ɑdɑ/ rendered with --no-burst, otherwise the same word. Measured: voice bar unchanged at the closure; no burst energy. PREDICTION: the voice bar still marks a stop, but the release is smooth. FALSIFIED if it sounds the same as the burst twin.", + "feel": "the same word with a gentler, softer release", + "pair": "stop-d" + }, + { + "id": "stop-k", + "file": "aka.wav", + "group": "stop", + "title": "/ɑkɑ/ at 140 Hz. Measured: burst peak 0.45 of the vowel peak, burst spectral peak 3100 Hz (Chodroff and Wilson 2014: 2868 Hz; /t/ 3950 Hz by the same measure), burst only at its own release; closure -14.9 dB re vowels against /p/ -20.4, so the closure is NOT at or under /p/ (the voicing the voiceless source still carries leaks through it). PREDICTION: 'ah-kah', a back-of-the-tongue stop with a duller, lower burst than /t/; the closure is audibly less empty than in /apa/. FALSIFIED if heard as /t/ or /p/, or if the closure carries a clear hum that makes it sound like /g/.", + "feel": "'ah-kah': the back of the tongue lifts to the roof of the mouth between two open vowels", + "pair": "stop-k-noburst" + }, + { + "id": "stop-k-noburst", + "file": "aka_noburst.wav", + "group": "stop", + "title": "/ɑkɑ/ rendered with --no-burst, otherwise the same word. Measured: no burst energy; closure unchanged from the burst render. PREDICTION: a softer, less distinct stop whose place is carried by the transitions alone. FALSIFIED if it sounds the same as the burst twin, or if the place is lost altogether.", + "feel": "the same word with a gentler, softer release", + "pair": "stop-k" + }, + { + "id": "stop-l", + "file": "ala.wav", + "group": "lateral", + "title": "/ɑlɑ/ at 140 Hz, a lateral with a closed side stub (midline narrowed to 0.06 cm, not closed: the ladder cannot hold parallel side channels). Measured: level -5.0 dB re the flanking vowels, F1 263 Hz (design 250-400: met), F2 725 Hz (design 1000-1300: NOT MET, F2 is about 300 Hz too low), side stub notch 13.4 dB at 2.8 kHz against the same render with the stub shut, no burst. PREDICTION: heard as a liquid between the vowels, possibly dark ('ahl-ah' as in a dark /l/) because F2 is low; may be heard as a nasal or /w/ rather than /l/. FALSIFIED if heard as a plain vowel glide with no consonant, or as a stop.", + "feel": "'ah-lah': the tongue tip touches behind the teeth while the sound flows past the sides" + }, + { + "id": "stop-p", + "file": "apa.wav", + "group": "stop", + "title": "/ɑpɑ/ at 140 Hz, the labial reference. Measured: closure -20.4 dB re the flanking vowels (-19.9 in the burst-site probe), burst enters at the glottal end (pinned labial burst), /p/ voice bar -40.1 dB. PREDICTION: 'ah-pah', a lip stop; the reference against which /t/ and /k/ are judged. FALSIFIED if heard as /b/ (voiced), or if the lip closure sounds leaky.", + "feel": "'ah-pah': the lips close and pop open between two open vowels" + }, + { + "id": "stop-b", + "file": "aba.wav", + "group": "stop", + "title": "/ɑbɑ/ at 140 Hz, the voiced labial reference. Measured: voice bar -29.2 dB re the following vowel (/p/ -40.1). PREDICTION: 'ah-bah', lips close with voicing kept through the closure; softer than /p/. FALSIFIED if heard as /p/ or as /m/.", + "feel": "'ah-bah': the lips close and open with the voice humming" + } + ] +} diff --git a/tools/aqua-listen/kits/tract-stops/phones/aba.json b/tools/aqua-listen/kits/tract-stops/phones/aba.json new file mode 100644 index 0000000..ecc650e --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/phones/aba.json @@ -0,0 +1 @@ +{"language": "en-US", "source": "ɑbɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "b", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]} \ No newline at end of file diff --git a/tools/aqua-listen/kits/tract-stops/phones/ada.json b/tools/aqua-listen/kits/tract-stops/phones/ada.json new file mode 100644 index 0000000..00e45a2 --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/phones/ada.json @@ -0,0 +1 @@ +{"language": "en-US", "source": "ɑdɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "d", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]} \ No newline at end of file diff --git a/tools/aqua-listen/kits/tract-stops/phones/aka.json b/tools/aqua-listen/kits/tract-stops/phones/aka.json new file mode 100644 index 0000000..4a5ba05 --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/phones/aka.json @@ -0,0 +1 @@ +{"language": "en-US", "source": "ɑkɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "k", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]} \ No newline at end of file diff --git a/tools/aqua-listen/kits/tract-stops/phones/ala.json b/tools/aqua-listen/kits/tract-stops/phones/ala.json new file mode 100644 index 0000000..f117fe8 --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/phones/ala.json @@ -0,0 +1 @@ +{"language": "en-US", "source": "ɑlɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "l", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]} \ No newline at end of file diff --git a/tools/aqua-listen/kits/tract-stops/phones/apa.json b/tools/aqua-listen/kits/tract-stops/phones/apa.json new file mode 100644 index 0000000..355a8b3 --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/phones/apa.json @@ -0,0 +1 @@ +{"language": "en-US", "source": "ɑpɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "p", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]} \ No newline at end of file diff --git a/tools/aqua-listen/kits/tract-stops/phones/ata.json b/tools/aqua-listen/kits/tract-stops/phones/ata.json new file mode 100644 index 0000000..b201cb1 --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/phones/ata.json @@ -0,0 +1 @@ +{"language": "en-US", "source": "ɑtɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "t", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]} \ No newline at end of file diff --git a/tools/aqua-listen/kits/tract-stops/render.sh b/tools/aqua-listen/kits/tract-stops/render.sh new file mode 100755 index 0000000..4d0cc87 --- /dev/null +++ b/tools/aqua-listen/kits/tract-stops/render.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# Renders the tract-stops kit (WordRender, 140 Hz, 48 kHz mono 16-bit): /apa aba ata ada aka ala/ (the vowel is /ɑ/) and the +# --no-burst twins of ata ada aka, whose release_flow splines are dropped so a pair isolates the burst. +# render.sh [jobs] +# Needs CULTLIB_ROOT at the admitted CultLib revision and AQUASYNTH_FAUST_HOME. Under load each render is minutes (a Faust compile). +# /out//.wav /outnb//.wav (--no-burst) /clips/*.wav the files kit.json names +# /images, burst-notes.* clip_images.py: PNGs per clip and the burst times (needs numpy + matplotlib: PYTHON=) +# Build the page: node tools/aqua-listen/build.mjs --manifest tools/aqua-listen/kits/tract-stops/kit.json --clips /clips --out /dist [--images /images] +set -eu +REPO=$(cd "$1" && pwd); OUT=$2; JOBS=${3:-4}; HERE=$REPO/tools/aqua-listen/kits/tract-stops +dotnet build "$REPO/tools/WordRender" -c Release -v q -o "$OUT/wr" | grep -E "rror\(s\)" +mkdir -p "$OUT/out" "$OUT/outnb" "$OUT/clips" +job() { n=$1; o=$2; f=$3; shift 3; dotnet "$OUT/wr/WordRender.dll" "$OUT/$o/$n" "$n" "$f" "$HERE/phones/$n.json" "$@" > "$OUT/$o/$n.log" 2>&1 && echo "$n $o ok" || echo "$n $o FAILED (see $OUT/$o/$n.log)"; } +export -f job; export OUT HERE +{ + for n in ata ada aka ala apa aba; do echo "$n out 140"; done + for n in ata ada aka apa aba; do echo "$n outnb 140 --no-burst"; done # apa aba twins feed the burst notes only, no clip +} | xargs -P "$JOBS" -L1 bash -c 'job "$@"' _ +C=$OUT/clips +for n in ata ada aka ala apa aba; do cp "$OUT/out/$n/$n.wav" "$C/$n.wav"; done +for n in ata ada aka; do cp "$OUT/outnb/$n/$n.wav" "$C/${n}_noburst.wav"; done +${PYTHON:-python3} -I "$REPO/tools/aqua-listen/clip_images.py" "$C" "$OUT/images" --kit "$HERE/kit.json" --renders "$OUT" diff --git a/tools/aqua-listen/kits/tract-weak-spots/render.sh b/tools/aqua-listen/kits/tract-weak-spots/render.sh index 3d63080..ecdd7f4 100755 --- a/tools/aqua-listen/kits/tract-weak-spots/render.sh +++ b/tools/aqua-listen/kits/tract-weak-spots/render.sh @@ -6,6 +6,7 @@ # /out//.wav 140 Hz renders; v__s is the vowel with primary stress, nibu has it on /i/ # /out148/v_ the unstressed vowel at 148.4 Hz (140 x 1.06: the f0 rise of primary stress alone) # /outnb/ the same word with --no-burst (plosive release_flow splines dropped) +# /images, burst-notes.* clip_images.py: PNGs per clip and the burst times (needs numpy + matplotlib: PYTHON=) # /clips/*.wav the files kit.json names; weak_spots.py measures the rest set -eu REPO=$(cd "$1" && pwd); OUT=$2; PRE=${3:-}; JOBS=${4:-4}; HERE=$REPO/tools/aqua-listen/kits/tract-weak-spots @@ -15,7 +16,7 @@ job() { n=$1; o=$2; f=$3; shift 3; dotnet "$OUT/wr/WordRender.dll" "$OUT/$o/$n" export -f job; export OUT HERE { for n in nibu nibu_flat v_i v_i_s bin bun ba ama ana v_ih v_oo_short v_ah v_ah_s v_e v_e_s v_o v_o_s v_u v_u_s; do echo "$n out 140"; done - for n in bin ba bun; do echo "$n outnb 140 --no-burst"; done + for n in bin ba bun nibu nibu_flat; do echo "$n outnb 140 --no-burst"; done for v in i e ah o u ih oo_short; do echo "v_$v out148 148.4"; done } | xargs -P "$JOBS" -L1 bash -c 'job "$@"' _ if [ -n "$PRE" ]; then @@ -27,3 +28,4 @@ C=$OUT/clips cp "$OUT/out/nibu_flat/nibu_flat.wav" "$C/nibu_flat.wav"; cp "$OUT/out/nibu_flat/nibu_flat.wav" "$C/nibu_flat_after.wav" # one render, two ids: the before/after pair is one-to-one for n in nibu v_i v_i_s v_ih v_oo_short ama ana ba bun; do cp "$OUT/out/$n/$n.wav" "$C/$n.wav"; done for n in ba bun; do cp "$OUT/outnb/$n/$n.wav" "$C/${n}_noburst.wav"; done +${PYTHON:-python3} -I "$REPO/tools/aqua-listen/clip_images.py" "$C" "$OUT/images" --kit "$HERE/kit.json" --renders "$OUT" diff --git a/tools/aqua-listen/test.mjs b/tools/aqua-listen/test.mjs index 702e122..b385f7e 100644 --- a/tools/aqua-listen/test.mjs +++ b/tools/aqua-listen/test.mjs @@ -101,6 +101,17 @@ test("build refuses a missing file and a manifest escape, naming them", () => { assert.throws(() => buildKit(mf([clip("a", { file: "../x.wav" })]), dir), /relative path/); assert.throws(() => buildKit(mf([clip("a", { title: "" })]), dir), /title is required/); }); +test("pictures ride in a collapsed block only for clips that have both, and only then does the policy admit data: images", () => { + const pics = mkdtempSync(join(tmpdir(), "aqua-listen-img-")), png = Buffer.from("89504e470d0a1a0a", "hex"); + writeFileSync(join(pics, "a.spec.png"), png); writeFileSync(join(pics, "a.wave.png"), png); writeFileSync(join(pics, "b.spec.png"), png); + const withPics = buildKit(manifest, dir, pics), data = JSON.parse(/" , feel: ""})]); const h = buildKit(m, dir).html; diff --git a/tools/aqua-listen/test_clip_images.py b/tools/aqua-listen/test_clip_images.py new file mode 100644 index 0000000..9d03d8d --- /dev/null +++ b/tools/aqua-listen/test_clip_images.py @@ -0,0 +1,67 @@ +"""python3 -I test_clip_images.py (numpy + matplotlib; synthetic clips, no render needed)""" +import hashlib, json, sys, tempfile, unittest, wave +from pathlib import Path +import numpy as np +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import clip_images as ci + +SR, BURST_AT = 48000, 0.30 + + +def write(path, x): + with wave.open(str(path), "wb") as w: + w.setnchannels(1); w.setsampwidth(2); w.setframerate(SR) + w.writeframes((np.clip(x, -1, 1) * 32767).astype(" 0.05) # a vowel with a gap round the release + if with_burst: + i = int(BURST_AT * SR); n = int(0.012 * SR) + x[i:i + n] += np.random.default_rng(1).normal(0, 0.1, n) * np.hanning(n) + return x + + +class Fixture(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()); self.clips = self.tmp / "clips"; self.clips.mkdir() + write(self.clips / "ba.wav", word(True)); write(self.clips / "ba_noburst.wav", word(False)) + self.root = self.tmp / "r" + for sub, wb in (("out", True), ("outnb", False)): + (self.root / sub / "ba").mkdir(parents=True); write(self.root / sub / "ba" / "ba.wav", word(wb)) + (self.root / "out" / "ba" / "script.aqua").write_text("phoneme_gesture name=ba_0_ɑ ipa=ɑ start=0 dur=0.2 intensity=1\nphoneme_gesture name=ba_1_b ipa=b start=0.2 dur=0.095 intensity=1\n") + self.kit = self.tmp / "kit.json" + self.kit.write_text(json.dumps({"clips": [{"id": "x-ba", "file": "ba.wav"}, {"id": "x-ba-nb", "file": "ba_noburst.wav"}]})) + + def run_tool(self, out, *extra): + sys.argv = ["clip_images", str(self.clips), str(out), *extra]; ci.main() + + def test_images_are_deterministic_and_complete(self): + a, b = self.tmp / "a" / "images", self.tmp / "b" / "images" + self.run_tool(a); self.run_tool(b) + names = sorted(p.name for p in a.iterdir()) + self.assertEqual(names, ["ba.spec.png", "ba.wave.png", "ba_noburst.spec.png", "ba_noburst.wave.png"]) + for n in names: + self.assertEqual(hashlib.sha256((a / n).read_bytes()).hexdigest(), hashlib.sha256((b / n).read_bytes()).hexdigest(), n) + + def test_notes_find_the_release_and_the_twin_says_none(self): + self.run_tool(self.tmp / "o" / "images", "--kit", str(self.kit), "--renders", str(self.root)) + rows = json.loads((self.tmp / "o" / "burst-notes.json").read_text()) + p = rows[0]["burst"][0] + self.assertAlmostEqual(p["release_s"], BURST_AT, delta=0.003) + self.assertGreater(p["burst_end_s"], BURST_AT + 0.004); self.assertLess(p["burst_end_s"], BURST_AT + 0.02) + self.assertEqual(rows[1]["burst"], "no burst expected") + self.assertIn("no burst expected", (self.tmp / "o" / "burst-notes.md").read_text()) + + def test_burst_shows_in_the_spectrogram_at_the_noted_time_only(self): + def hi_db(x, at): + db, f, t = ci.stft_db(x, SR); col = int(np.argmin(np.abs(t - at))) + lin = 10 ** (db[f > 2000, col] / 20); return 20 * np.log10(np.sqrt((lin ** 2).sum()) + 1e-12) + w, nb = word(True), word(False) + self.assertGreaterEqual(hi_db(w, BURST_AT + 0.006) - hi_db(nb, BURST_AT + 0.006), 10) + self.assertLess(hi_db(w, BURST_AT + 0.106) - hi_db(nb, BURST_AT + 0.106), 10) # a note 100 ms late would find nothing + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/aqua-listen/web/page.js b/tools/aqua-listen/web/page.js index 48eafe3..241df90 100644 --- a/tools/aqua-listen/web/page.js +++ b/tools/aqua-listen/web/page.js @@ -1,4 +1,4 @@ -// Page glue. The kit (manifest, content hash, base64 audio) is in #kit-data; logic is AquaListen (logic.cjs, inlined). +// Page glue. The kit (manifest, content hash, base64 audio, optional base64 PNGs per clip id) is in #kit-data; logic is AquaListen (logic.cjs, inlined). (() => { const L = globalThis.AquaListen, kit = JSON.parse(document.getElementById("kit-data").textContent); const app = document.getElementById("app"), KEY = `aqua-listen:${kit.manifest.kit.id}:${kit.hash}`; @@ -62,6 +62,7 @@ h("h2", {}, o.slot ? `Sound ${o.slot}` : `Sound ${n}`), h("p", { class: "feel" }, `It should feel: ${c.feel}`), audio, + (kit.images[id] || []).length ? h("details", {}, h("summary", {}, "Pictures of this sound (optional)"), kit.images[id].map((p) => h("img", { src: "data:image/png;base64," + p.b64, alt: p.label, class: "pic" }))) : [], h("div", { class: "row" }, h("span", {}, "Keep, fix or drop?"), radios("v-" + id, L.VERDICTS, a.verdict, (v) => touch(id, { verdict: v }))), h("div", { class: "row" }, h("span", {}, "Did it land? 1 = not at all, 5 = completely"), radios("l-" + id, [1, 2, 3, 4, 5], a.landed, (v) => touch(id, { landed: Number(v) }))), reason); diff --git a/tools/aqua-listen/web/styles.css b/tools/aqua-listen/web/styles.css index 929821e..7cc3313 100644 --- a/tools/aqua-listen/web/styles.css +++ b/tools/aqua-listen/web/styles.css @@ -16,4 +16,5 @@ button:disabled { opacity: 0.4; cursor: default; } .bar { position: sticky; top: 0; background: var(--bg); padding: 0.5rem 0; border-bottom: 1px solid var(--line); z-index: 1; } .pair { border: 2px dashed var(--line); border-radius: 10px; padding: 0 0.75rem; margin: 1rem 0; } small, .hint { opacity: 0.75; } +img.pic { max-width: 100%; height: auto; display: block; margin: 0.5rem 0; } textarea[readonly] { font: 0.8rem/1.3 ui-monospace, monospace; } diff --git a/tools/formant-check/test_burst_spectrum.py b/tools/formant-check/test_burst_spectrum.py new file mode 100644 index 0000000..470a180 --- /dev/null +++ b/tools/formant-check/test_burst_spectrum.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +"""burst_spectrum isolates the burst: the difference of a render and its --no-burst twin is zero outside the release, and the peak and +centroid it reports are those of the noise that was added there. python3 tools/formant-check/test_burst_spectrum.py""" +import os, tempfile, unittest, wave +import numpy as np +import weak_spots as w + +SR = 48000 + +def write(root, sub, name, x): + d = os.path.join(root, sub, name); os.makedirs(d, exist_ok=True) + with wave.open(os.path.join(d, name + ".wav"), "wb") as f: + f.setnchannels(1); f.setsampwidth(2); f.setframerate(SR); f.writeframes((np.clip(x, -1, 1) * 32767).astype(" 1e-9)), "file_end_rms_last_100ms": round(rms(x, sr, len(x) / sr - .1, len(x) / sr), 5), "vowel_rms": round(rms(x, sr, c1 + .06, c1 + .17), 4)} +BURST_BANDS = ((0, 1000), (1000, 2000), (2000, 3500), (3500, 7000), (7000, 24000)) + +def burst_spectrum(root, word, release_s, window_ms=20, sub="out", smooth_hz=400, min_hz=1000): + """The isolated burst's spectrum: the difference of out/ and outnb/ (renders with and without release_flow), window_ms from + release_s, Hann windowed. peak_hz is the maximum of the power spectrum smoothed over smooth_hz and searched from min_hz up (a noise + burst has no single line; below 1 kHz the release carries the tract's own low resonance, 430-480 Hz in /ata/ and /apa/, which is not + the place cue: DESIGN choice of the floor), peak_hz_all the same without the floor, centroid_hz the power-weighted mean frequency from + min_hz up, band_db the power in BURST_BANDS (dB, absolute), window_peak the largest sample in the window, outside_peak the largest + sample outside release_s - 5 ms .. release_s + 120 ms (the closure's residue: 0 for a burst that exists only in the window).""" + x, sr = load(root, sub, word); nb, _ = load(root, "outnb", word) + n = min(len(x), len(nb)); d = x[:n] - nb[:n] + a = int(release_s * sr); m = int(window_ms / 1000 * sr); s = d[a:a + m] * np.hanning(m) + nfft = 1 << 14 + P = np.abs(np.fft.rfft(s, nfft)) ** 2; f = np.fft.rfftfreq(nfft, 1 / sr) + k = max(1, int(smooth_hz / (sr / nfft))); Ps = np.convolve(P, np.ones(k) / k, mode="same") + up = f >= min_hz + out_mask = np.ones(n, bool); out_mask[max(0, int((release_s - .005) * sr)):int((release_s + .12) * sr)] = False + return {"peak_hz": round(float(f[up][np.argmax(Ps[up])]), 0), "peak_hz_all": round(float(f[int(np.argmax(Ps))]), 0), + "centroid_hz": round(float((f[up] * P[up]).sum() / max(P[up].sum(), 1e-30)), 0), + "band_db": {f"{lo}_{hi}": round(float(10 * np.log10(P[(f >= lo) & (f < hi)].sum() + 1e-30)), 1) for lo, hi in BURST_BANDS}, + "window_peak": round(float(np.abs(d[a:a + m]).max()), 5), "outside_peak": float(np.abs(d[out_mask]).max()) if out_mask.any() else 0.0} + def bands(x, sr, a, b, edges=((0, 400), (400, 800), (800, 1600), (1600, 3200))): s = x[int(a * sr):int(b * sr)]; s = s * np.hanning(len(s)) P = np.abs(np.fft.rfft(s, 1 << 15)) ** 2; f = np.fft.rfftfreq(1 << 15, 1 / sr)