Tract weak spots re-measured (unheard): burst gain no longer leaks past its plosive; 14-clip listening kit #47
26 changed files with 357 additions and 8 deletions
|
|
@ -127,9 +127,23 @@ resolution), release 60-95 ms peak 0.0695, vowel 0.129. The wall radiation reads
|
|||
level of -30 dB; the primary level was not retrieved, and the 2.4 manual's skin radiation, Birkholz & Zhang 2025, is the reference to read). In a vowel the term moves the f0 line by under 1 dB; the 200 and 300 Hz harmonics move up to +1.5 dB on one open vowel (wall-only measurement, Soul pass). The band is two second-order sections, not a brick wall: with the mouth and the velum shut the bar sits about 25 dB down above 300 Hz. The burst
|
||||
itself (difference of renders with and without `release_flow`) peaks at about 0.006-0.008, -24 dB re the vowel peak; the release peak in the table above is the voicing breaking through as the contact opens. Burst gain 0.06 is a DESIGN VALUE.
|
||||
|
||||
Weak spots found in the renders: (1) the vowel level is not steady, it rises for about 100 ms after onset (ɑ: rms 0.028 at 80 ms to 0.079 at 160 ms) so any window early in a vowel
|
||||
reads low; the cause is not isolated (surface smoothing and the pressure ramp are the suspects). (2) /n/ murmur 2 dB above /m/ (above). (3) The voice bar's level is a design value and its band
|
||||
Weak spots found in the renders, 2026-11-08 (not re-measured on 2026-11-12 unless named in the next paragraph): (1) the vowel level is not steady, it rises for about 100 ms after onset (ɑ: rms 0.028 at 80 ms to 0.079 at 160 ms) so any window early in a vowel
|
||||
reads low; the cause is not isolated (surface smoothing and the pressure ramp are the suspects). (3) The voice bar's level is a design value and its band
|
||||
(90-300 Hz) assumes f0 of 90 Hz and up. (4) Only f0 and level of the first vowel are checked per cue; the cues' perceptual strength is unknown. Everything here is a measurement, nothing was heard.
|
||||
|
||||
Re-measure 2026-11-12 on main c0e6b98 (`tools/formant-check/weak_spots.py` over renders from `tools/aqua-listen/kits/tract-weak-spots/render.sh`; the renders reproduce the table above: /ɪ/ 375/2024 Hz, /ʊ/ 486/894 Hz, murmurs -11.81 and -9.64 dB, /i/ in /nibu/ +0.21 dB). Unheard.
|
||||
- Stress level (100-170 ms of each vowel, stressed against unstressed, f0 rise split from the rest by a render of the unstressed vowel at 148.4 Hz): total i +1.22, ɑ +1.67, u +2.07, e +2.99, o +1.69 dB (mean 1.93, sd 0.60);
|
||||
the f0 rise alone -0.80, +0.03, +0.17, +1.23, -0.07 dB; the rest (the amplitude factor 1.30 = 2.28 dB nominal, with the longer vowel) +2.02, +1.64, +1.90, +1.76, +1.76 dB (sd 0.14). The scatter between vowels is the f0 rise; the amplitude cue itself is even.
|
||||
No level the f0 rise cannot move was found cheap, so the five-vowel-mean basis stands (`aqua-body:question:stress-amplitude-calibration-basis`); the rest sits 0.46 dB under the nominal 2.28 dB, cause not isolated.
|
||||
- Plosive release: a defect was found and fixed. `release_flow` was held at the burst gain from the plosive's gesture on (a surface holds its first value before its first point and its last after its last), so every later contact state made noise:
|
||||
the difference of the renders with and without `release_flow` peaked at 0.0302 in the final /n/ and tail of /bɪn/ (above its 0.0226 at the release, 81 ms), 0.0126 in /bʊn/; in /nibu/ (flat) the leading /n/ carried it backward: peak 0.0688, rms 0.0067 against the /n/'s own 0.0058.
|
||||
After the fix (`BurstGainAroundGesture`: zero before and after the plosive, the gain through it) the same difference is 0.0 from 120 ms after the release and the burst itself is unchanged (0.0226 at 81 ms in /bɪn/). Burst alone, after the fix: /bɪn/ 0.0226 (-15.1 dB re its vowel peak 0.129), /ba/ 0.0142 (-24.0 dB re 0.225), /bʊn/ 0.0130 (-24.3 dB re 0.213),
|
||||
each falling to a tenth in 15-17 ms; the earlier "0.006-0.008, -24 dB" figure above does not hold for /bɪn/. The release peak of the full render (voicing breaking through) is -5.4, -6.3, -5.9 dB re the vowel peak. Test `PlosiveBurstExistsOnlyAroundItsOwnRelease`.
|
||||
- Nasals: /m/ -11.81 dB, /n/ -9.64 dB re the flanking vowels, Burg N1 295 and 332 Hz. The 2 dB is not harmonic sampling: /n/ minus /m/ is 2.01, 2.17, 2.05, 1.61 dB at f0 110, 140, 170, 200 Hz. Band levels of the murmur (0-400, 400-800, 800-1600, 1600-3200 Hz): /m/ 36.3, 23.6, 21.3, 2.7; /n/ 38.4, 28.2, 16.8, 0.9 dB.
|
||||
The cause (the shorter oral side branch, 7.1 cm against 9.6 cm from the velum junction) is a candidate, not shown. Open.
|
||||
- Lax vowels: /ɪ/ and /ʊ/ are in the DSL table (`StoryVowelAreas`) and render in `WordRender` (/bɪn/, /bʊn/, isolated). Burg F1/F2: /ɪ/ 375/2024 against the tube chain 346/1987 (+8.5/+1.8 %), Hillenbrand 429/2034 (-12.5/-0.5 %), Peterson & Barney 392/1993 (-4.3/+1.5 %);
|
||||
/ʊ/ 486/894 against 478/850 (+1.7/+5.2 %), 469/1123 (+3.7/-20.4 %), 437/1023 (+11.3/-12.6 %). No defect found.
|
||||
- Phones the tract cannot make: `WordRender` refuses /t/, /d/, /k/, /l/ with exit 4 (`UNSUPPORTED PHONES (the tract has no gesture for them; nothing rendered): t`); the table holds p b m n s z ʃ ʒ ʍ w and the Story vowels. A direct `phoneme_gesture` on an unknown IPA symbol still falls back to an alveolar approximant (`GestureDescriptor`); not changed here.
|
||||
|
||||
Mutations (hand-applied, restored, each run against the named tests): pitch ratio ignored, burst removed, burst not shaped by opening, burst gain x20 (0.06 to 1.2), stress duration ignored,
|
||||
stress pitch ignored, stress amplitude ignored, secondary equal to primary, probe timeline sample rate re-hardcoded to 48000: nine of nine killed. Script: `work/mut-b3.sh` in the aqua workspace.
|
||||
|
|
|
|||
|
|
@ -1567,8 +1567,10 @@ public static class PatchScript
|
|||
AddAuthoredSpline($"{name}_contact", "/vocal/contacts/0/opening", start, duration, (0, 0.02f), (0.65f, 0.02f), (1, 0.9f));
|
||||
AddAuthoredSpline($"{name}_stored_pressure", "/vocal/contacts/0/stored_pressure", start, duration, (0, 0.4f), (0.65f, 0.75f), (1, 0.05f));
|
||||
// The release burst is turbulence at the opening constriction: its gain is held for the plosive and the exporter shapes
|
||||
// it by the stored pressure and the opening (see PrimitiveContactExpression).
|
||||
HoldOnSurface($"{name}_release_flow", "/vocal/contacts/0/release_flow", start, duration, PlosiveBurstGain);
|
||||
// it by the stored pressure and the opening (see PrimitiveContactExpression). The gain exists only around this plosive:
|
||||
// a surface holds its last value after its last point (and its first before its first), so a gain left at its level
|
||||
// made noise at every later contact state, the final nasal's closure and the tail included.
|
||||
BurstGainAroundGesture($"{name}_release_flow", "/vocal/contacts/0/release_flow", start, duration, PlosiveBurstGain);
|
||||
AddAuthoredSpline($"{name}_velum", "/vocal/branches/0/opening", start, duration, (0, VelumClosed), (1, VelumClosed));
|
||||
}
|
||||
else if (tokens.Contains("fricative"))
|
||||
|
|
@ -1627,6 +1629,29 @@ public static class PatchScript
|
|||
AddAuthoredSpline(name, surfacePath, start, duration, (0, normalized), (1, normalized));
|
||||
}
|
||||
|
||||
// Zero before and after the gesture, the gain through it; the ramps are 0.15 of the gesture long and run into the neighbouring
|
||||
// phones (the leading one is cut where the word begins).
|
||||
private void BurstGainAroundGesture(string name, string surfacePath, float start, float duration, float gain)
|
||||
{
|
||||
if (!_controlSurfacesByPath.TryGetValue(surfacePath, out var surface) || surface.MaxValue <= surface.MinValue)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var level = Math.Clamp((gain - surface.MinValue) / (surface.MaxValue - surface.MinValue), 0, 1);
|
||||
var handle = duration * 0.25f;
|
||||
ControlSplinePoint At(float time, float value) => new(time, value, handle, value, handle, value);
|
||||
var ramp = duration * 0.15f;
|
||||
var points = new List<ControlSplinePoint>();
|
||||
if (start > 0) points.Add(At(Math.Max(0, start - ramp), 0));
|
||||
points.Add(At(start, level));
|
||||
points.Add(At(start + duration, level));
|
||||
points.Add(At(start + duration + ramp, 0));
|
||||
var spline = new ControlSpline(UniqueSplineName(name), surfacePath, points.ToArray(), ControlSplineInterpolation.Bezier);
|
||||
_controlSplines.Add(spline);
|
||||
_controlSplinesByName[spline.Name] = spline;
|
||||
}
|
||||
|
||||
private static string GestureDescriptor(string ipa, IReadOnlyDictionary<string, string> fields)
|
||||
{
|
||||
var explicitDescriptor = GetAny(fields, ["descriptor", "desc", "features"], "");
|
||||
|
|
|
|||
|
|
@ -114,6 +114,30 @@ public sealed class StressAndReleaseRenderTests(ITestOutputHelper output)
|
|||
Assert.True(burstPeak < vowelPeak / 2, "the burst stays well under the following vowel's peak (the b level fix)");
|
||||
}
|
||||
|
||||
// The burst is noise whose amplitude is release_flow times the stored pressure through the opening, and release_flow was held at its
|
||||
// gain from the plosive's gesture on, so it never ended: every later contact state made noise too. In /bin/ the "burst" alone (the
|
||||
// difference of the renders with and without release_flow) peaked at 0.0302 in the final /n/ and the tail, above its 0.0226 at the
|
||||
// release itself, and in /nibu/ the leading /n/ and /i/ carried it backward (a spline holds its first value before its first point).
|
||||
[Fact]
|
||||
public void PlosiveBurstExistsOnlyAroundItsOwnRelease()
|
||||
{
|
||||
IpaGestureExperimentTarget Phone(string label, string ipa, float seconds) => new($"w_{label}", ipa, "", seconds);
|
||||
var script = IpaGestureExperiment.BuildWordScript(
|
||||
[(Phone("0_n", "n", 0.095f), 1f), (Phone("1_a", "a", 0.18f), 1f), (Phone("2_b", "b", 0.095f), 1f), (Phone("3_a", "a", 0.18f), 1f), (Phone("4_n", "n", 0.095f), 1f)], 140);
|
||||
var withBurst = Render(script, 0.77f);
|
||||
var withoutBurst = Render(script, 0.77f, p => p with { ControlSplines = p.ControlSplines.Where(s => !s.SurfacePath.EndsWith("/release_flow", StringComparison.Ordinal)).ToArray() });
|
||||
if (withBurst is null || withoutBurst is null) return;
|
||||
var burst = withBurst.Zip(withoutBurst, (x, y) => x - y).ToArray();
|
||||
const double bFrom = 0.275, bTo = 0.37;
|
||||
var atRelease = Peak(burst, bFrom + 0.095 * 0.6, bTo + 0.02);
|
||||
var leading = Peak(burst, 0, bFrom);
|
||||
var trailing = Peak(burst, bTo + 0.12, 0.77);
|
||||
output.WriteLine($"burst peak {atRelease:F4} at the release, {leading:F4} before the plosive (n, a), {trailing:F4} after it (a, n, tail)");
|
||||
Assert.True(atRelease > 0.004, "the burst is still there at the release");
|
||||
Assert.True(leading < atRelease / 20, "no burst noise before the plosive's own gesture");
|
||||
Assert.True(trailing < atRelease / 10, "no burst noise in the vowel, the nasal and the tail after the plosive");
|
||||
}
|
||||
|
||||
// /ɑ C ɑ/ at the scaffold's own durations (vowel 0.18 s, consonant 0.095 s). Returns the render and the consonant's [start, end).
|
||||
private (float[]? Samples, double From, double To) VowelConsonantVowel(string ipa, float frequency = 140)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -8,10 +8,12 @@ using AquaSynth.Faust;
|
|||
// Offline probe: one word, segmented by Weksa (the phones line of `weksa lower --phones`), as a run of IPA gestures on the
|
||||
// IpaGestureExperiment scaffold, rendered through the native libfaust path to a mono 16-bit WAV plus script.aqua, source.dsp
|
||||
// and manifest.json. It measures peak, rms and NaN; it cannot say how the word sounds.
|
||||
// usage: WordRender <outDir> <label> <freq-hz> <phones.json>
|
||||
// usage: WordRender <outDir> <label> <freq-hz> <phones.json> [--no-burst]
|
||||
// --no-burst drops the plosive release_flow splines (the same edit the burst test makes), so a render pair isolates the burst.
|
||||
// exit: 0 rendered, 1 compile or non-finite output, 2 usage, 3 phones file refused, 4 phones the tract cannot make (listed, nothing rendered)
|
||||
const int Rate = 48000, Frames = 512;
|
||||
if (args.Length != 4) { Console.Error.WriteLine("usage: WordRender <outDir> <label> <freq-hz> <phones.json>"); return 2; }
|
||||
var noBurst = args.Length == 5 && args[4] == "--no-burst";
|
||||
if (args.Length != 4 && !noBurst) { Console.Error.WriteLine("usage: WordRender <outDir> <label> <freq-hz> <phones.json> [--no-burst]"); return 2; }
|
||||
var outDir = Path.GetFullPath(args[0]);
|
||||
var label = args[1];
|
||||
var freq = float.Parse(args[2], System.Globalization.CultureInfo.InvariantCulture);
|
||||
|
|
@ -41,7 +43,9 @@ Console.WriteLine($"faust home: {home ?? "(not found)"}");
|
|||
using var compiler = new AquaSynthPatchCompiler(new AquaSynthNativeOptions(SampleRate: Rate));
|
||||
// The exporter emits one merged spline per surface (the ControlSplineTimeline curve), so the DSP carries the word's motion
|
||||
// itself and no slider is driven from outside.
|
||||
var source = FaustEmitter.Emit(PatchScript.Parse(script), new FaustExportOptions(label)).Source;
|
||||
var parsed = PatchScript.Parse(script);
|
||||
if (noBurst) parsed = parsed with { ControlSplines = parsed.ControlSplines.Where(s => !s.SurfacePath.EndsWith("/release_flow", StringComparison.Ordinal)).ToArray() };
|
||||
var source = FaustEmitter.Emit(parsed, new FaustExportOptions(label)).Source;
|
||||
File.WriteAllText(Path.Combine(outDir, "source.dsp"), source);
|
||||
if (!compiler.TryCompileSource(new AquaSynthCompileIdentity(label, label, ""), source, seconds, out var patch0, out var cerr))
|
||||
{
|
||||
|
|
@ -83,7 +87,7 @@ using (var w = new BinaryWriter(File.Create(wav)))
|
|||
var rms = Math.Sqrt(sum / pcm.Length);
|
||||
File.WriteAllText(Path.Combine(outDir, "manifest.json"), JsonSerializer.Serialize(new
|
||||
{
|
||||
label, freq, weksa = new { word.Language, word.Source, Phones = word.Phones.Select(p => new { p.Ipa, Stress = p.Stress.ToString() }) }, seconds, sampleRate = Rate, frames = pcm.Length, peak, rms, nonFinite = nan,
|
||||
label, freq, noBurst, weksa = new { word.Language, word.Source, Phones = word.Phones.Select(p => new { p.Ipa, Stress = p.Stress.ToString() }) }, seconds, sampleRate = Rate, frames = pcm.Length, peak, rms, nonFinite = nan,
|
||||
wavSha256 = Convert.ToHexString(SHA256.HashData(File.ReadAllBytes(wav))).ToLowerInvariant(),
|
||||
scriptSha256 = Convert.ToHexString(SHA256.HashData(File.ReadAllBytes(Path.Combine(outDir, "script.aqua")))).ToLowerInvariant(),
|
||||
os = RuntimeInformation.OSDescription, faustHome = home
|
||||
|
|
|
|||
117
tools/aqua-listen/kits/tract-weak-spots/kit.json
Normal file
117
tools/aqua-listen/kits/tract-weak-spots/kit.json
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
{
|
||||
"kit": {
|
||||
"id": "tract-weak-spots-1",
|
||||
"title": "Articulatory vocal tract: stress, release, nasals, lax vowels",
|
||||
"note": "Short synthetic words from the tract model. Tell us what you hear; there are no wrong answers. Some are pairs: rate both, then say which you prefer."
|
||||
},
|
||||
"clips": [
|
||||
{
|
||||
"id": "stress-nibu-flat",
|
||||
"file": "nibu_flat.wav",
|
||||
"group": "stress",
|
||||
"pair": "stress-nibu",
|
||||
"title": "nibu, no stress marks. Measured: /i/ 150-250 ms rms vs the stressed render +0.21 dB; unstressed vowel 180 ms, f0 139.9 Hz. PREDICTION: reads as an even two-syllable word with no strong first syllable. FALSIFIED if listeners hear it as stressed on a syllable or cannot tell it from the stressed twin.",
|
||||
"feel": "a made-up two-syllable word, 'nee-boo', both syllables even"
|
||||
},
|
||||
{
|
||||
"id": "stress-nibu",
|
||||
"file": "nibu.wav",
|
||||
"group": "stress",
|
||||
"pair": "stress-nibu-flat",
|
||||
"title": "nibu with primary stress on /i/ (NI-bu). Measured: /i/ 247 ms vs 180 ms (+75 ms; design +68), f0 148.1 vs 139.9 Hz (x1.059), amplitude factor 1.30; level of /i/ in the 150-250 ms window only +0.21 dB (five-vowel mean +1.93 dB, per vowel +1.2 to +3.0). PREDICTION: first syllable heard as the prominent one, mostly through length and pitch, little through loudness; B should be picked as 'stressed on the first syllable'. FALSIFIED if it sounds no more prominent than the flat twin, if it sounds like a question or a pitch accent rather than stress, or if the level is what is heard (then the f0-scatter question is wrong).",
|
||||
"feel": "the same made-up word, with the first syllable stronger: 'NEE-boo'"
|
||||
},
|
||||
{
|
||||
"id": "stress-i-flat",
|
||||
"file": "v_i.wav",
|
||||
"group": "stress",
|
||||
"pair": "stress-i",
|
||||
"title": "isolated /i/ 180 ms at 140 Hz, no stress. Measured: level 100-170 ms is the reference. PREDICTION: plain short vowel. FALSIFIED if it already sounds emphasised.",
|
||||
"feel": "a single short 'ee' vowel, plain"
|
||||
},
|
||||
{
|
||||
"id": "stress-i",
|
||||
"file": "v_i_s.wav",
|
||||
"group": "stress",
|
||||
"pair": "stress-i-flat",
|
||||
"title": "isolated /i/ with primary stress: 247 ms, f0 x1.06, amplitude 1.30. Measured against the flat /i/: +1.22 dB total (f0 rise alone -0.80 dB, the amplitude and the rest +2.02 dB). PREDICTION: heard as longer and a little higher; the 1.2 dB louder is under what a listener can judge, so 'longer and higher' is the answer, 'louder' is not. FALSIFIED if it sounds the same length and pitch, or clearly louder.",
|
||||
"feel": "the same 'ee' vowel, held longer and said with emphasis"
|
||||
},
|
||||
{
|
||||
"id": "leak-nibu-flat-before",
|
||||
"file": "nibu_flat_before.wav",
|
||||
"group": "release-leak",
|
||||
"pair": "leak-nibu-flat-after",
|
||||
"title": "nibu (flat) rendered at main c0e6b98, before the release_flow fix. Measured: burst-gain leak vs the fixed render, peak 0.0688 and rms 0.0067 in the leading /n/ (0-95 ms; the /n/ itself is rms 0.0058, so the hiss is louder than the murmur), peak 0.0504 over /i/ onset 95-150 ms, 0.0019 after; whole-render peak 0.487 vs 0.399. PREDICTION: an audible hiss or click as the /n/ opens into the vowel, the first sound of the word being a noisy 'sn'-like or 'tsh' onset instead of a clean nasal hum. FALSIFIED if the two are indistinguishable (then the leak is inaudible and the fix is cosmetic).",
|
||||
"feel": "a made-up word that should start with a clean, smooth hum into the vowel: 'nee-boo'"
|
||||
},
|
||||
{
|
||||
"id": "leak-nibu-flat-after",
|
||||
"file": "nibu_flat_after.wav",
|
||||
"group": "release-leak",
|
||||
"pair": "leak-nibu-flat-before",
|
||||
"title": "nibu (flat) after the fix: release_flow is zero outside its own plosive. Same audio as stress-nibu-flat, kept as its own clip so the before/after pair is one-to-one. Measured: leak 0.0 from 275 ms on, burst of the b itself unchanged (bin 0.0226 at 81 ms before and after). PREDICTION: clean hum into the vowel, no noisy onset; preferred over 'before' if the leak is audible. FALSIFIED if 'before' is preferred or no difference is heard.",
|
||||
"feel": "a made-up word that should start with a clean, smooth hum into the vowel: 'nee-boo'"
|
||||
},
|
||||
{
|
||||
"id": "burst-ba-off",
|
||||
"file": "ba_noburst.wav",
|
||||
"group": "burst",
|
||||
"pair": "burst-ba-on",
|
||||
"title": "ba with the release_flow splines dropped (WordRender --no-burst). Measured: closure peak 0.0078 (voice bar), release (voicing breaking through) peak 0.1087 = -6.3 dB re vowel peak 0.2247, no burst noise. PREDICTION: a soft, voiced 'b' with the vowel starting smoothly; may sound slightly 'w'-like or lack the pop. FALSIFIED if listeners cannot hear any difference from the burst twin (then burst gain 0.06 is inaudible).",
|
||||
"feel": "the syllable 'bah' with a soft start"
|
||||
},
|
||||
{
|
||||
"id": "burst-ba-on",
|
||||
"file": "ba.wav",
|
||||
"group": "burst",
|
||||
"pair": "burst-ba-off",
|
||||
"title": "ba with the plosive burst (gain 0.06, a design value). Measured: burst alone peaks 0.0142 at 85 ms = -24.0 dB re vowel peak, falls to a tenth in 17 ms, burst rms in the 30 ms after its peak -31.0 dB re vowel rms. PREDICTION: a slightly crisper, more 'b'-like onset than B/off; the burst is quiet and short so the difference is small, 'a little pop' at most. FALSIFIED if it sounds like a hiss, a click, or a separate noise after the vowel started, or if no difference is heard.",
|
||||
"feel": "the syllable 'bah' with a clean, crisp 'b' at the start"
|
||||
},
|
||||
{
|
||||
"id": "burst-bun-off",
|
||||
"file": "bun_noburst.wav",
|
||||
"group": "burst",
|
||||
"pair": "burst-bun-on",
|
||||
"title": "bun, /bʊn/, no burst. Measured: closure peak 0.0043, release peak 0.1074 = -5.9 dB re vowel peak 0.2126. PREDICTION: soft 'b' then a short back vowel then a nasal; the final /n/ is a clean hum (no leak since the fix). FALSIFIED as for burst-ba-off.",
|
||||
"feel": "a short word like 'bunn' or 'boon' said quickly, soft start"
|
||||
},
|
||||
{
|
||||
"id": "burst-bun-on",
|
||||
"file": "bun.wav",
|
||||
"group": "burst",
|
||||
"pair": "burst-bun-off",
|
||||
"title": "bun, /bʊn/, with burst. Measured: burst peak 0.0130 at 81 ms = -24.3 dB re vowel peak, tenth in 15 ms; tail leak after the fix 0.0 (was 0.0126 at the end before). PREDICTION: slightly crisper onset than off; final /n/ clean. FALSIFIED as for burst-ba-on.",
|
||||
"feel": "a short word like 'bunn' or 'boon' said quickly, crisp 'b' start"
|
||||
},
|
||||
{
|
||||
"id": "nasal-m",
|
||||
"file": "ama.wav",
|
||||
"group": "nasal",
|
||||
"title": "/ɑmɑ/. Measured: murmur -11.81 dB re the flanking vowels (design band -16..-8), Burg N1 295 Hz, band levels 0-400/400-800/800-1600/1600-3200 Hz of the murmur 36.3/23.6/21.3/2.7 dB. PREDICTION: heard as 'ah-mah', a lip-closed hum between the vowels, clearly softer than the vowels. FALSIFIED if heard as 'abah' or 'awah', if the hum is as loud as the vowels, or if there is a gap or click.",
|
||||
"feel": "'ah-mah': lips close for a soft hum between two open vowels"
|
||||
},
|
||||
{
|
||||
"id": "nasal-n",
|
||||
"file": "ana.wav",
|
||||
"group": "nasal",
|
||||
"title": "/ɑnɑ/. Measured: murmur -9.64 dB re the flanking vowels (2.17 dB above /m/; the gap is 2.01/2.17/2.05/1.61 dB at f0 110/140/170/200 Hz, so not harmonic sampling), Burg N1 332 Hz, bands 38.4/28.2/16.8/0.9 dB. PREDICTION: heard as 'ah-nah', a tongue-tip hum, a shade stronger than the /m/ one; the 2 dB is not expected to be noticed. FALSIFIED if heard as 'ama' or 'anga', if it sounds much louder than the /m/ clip, or if there is hiss.",
|
||||
"feel": "'ah-nah': the tongue touches behind the teeth for a soft hum between two open vowels"
|
||||
},
|
||||
{
|
||||
"id": "lax-ih",
|
||||
"file": "v_ih.wav",
|
||||
"group": "lax",
|
||||
"title": "isolated /ɪ/ 180 ms, 140 Hz. Measured: Burg F1/F2 375/2024 Hz; tube chain 346/1987 (+8.5/+1.8 %); Hillenbrand 1995 male 429/2034 (-12.5/-0.5 %); Peterson & Barney 392/1993 (-4.3/+1.5 %). Present in the DSL table and renderable. PREDICTION: heard as the vowel of 'bit' (or a slightly high, 'ee'-leaning version of it). FALSIFIED if heard as 'ee' of 'beet', as 'eh' of 'bet', or as a different phone.",
|
||||
"feel": "the short vowel in 'bit'"
|
||||
},
|
||||
{
|
||||
"id": "lax-oo-short",
|
||||
"file": "v_oo_short.wav",
|
||||
"group": "lax",
|
||||
"title": "isolated /ʊ/ 180 ms, 140 Hz. Measured: Burg F1/F2 486/894 Hz; tube chain 478/850 (+1.7/+5.2 %); Hillenbrand 469/1123 (+3.7/-20.4 %); Peterson & Barney 437/1023 (+11.3/-12.6 %). Present in the DSL table and renderable. PREDICTION: heard as the vowel of 'book', probably darker or rounder than a natural one (F2 20 % under the population mean, the back-vowel gap). FALSIFIED if heard as 'oo' of 'boot', 'o' of 'boat' or 'uh' of 'but'.",
|
||||
"feel": "the short vowel in 'book'"
|
||||
}
|
||||
]
|
||||
}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/ama.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/ama.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ɑmɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "m", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/ana.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/ana.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ɑnɑ", "phones": [{"ipa": "ɑ", "stress": "none"}, {"ipa": "n", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/ba.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/ba.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ba", "phones": [{"ipa": "b", "stress": "none"}, {"ipa": "ɑ", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/bin.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/bin.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "bɪn", "phones": [{"ipa": "b", "stress": "none"}, {"ipa": "ɪ", "stress": "primary"}, {"ipa": "n", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/bun.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/bun.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "bʊn", "phones": [{"ipa": "b", "stress": "none"}, {"ipa": "ʊ", "stress": "primary"}, {"ipa": "n", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/nibu.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/nibu.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈnibu", "phones": [{"ipa": "n", "stress": "none"}, {"ipa": "i", "stress": "primary"}, {"ipa": "b", "stress": "none"}, {"ipa": "u", "stress": "none"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈnibu", "phones": [{"ipa": "n", "stress": "none"}, {"ipa": "i", "stress": "none"}, {"ipa": "b", "stress": "none"}, {"ipa": "u", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_ah.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_ah.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ɑ", "phones": [{"ipa": "ɑ", "stress": "none"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈɑ", "phones": [{"ipa": "ɑ", "stress": "primary"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_e.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_e.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "e", "phones": [{"ipa": "e", "stress": "none"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈe", "phones": [{"ipa": "e", "stress": "primary"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_i.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_i.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "i", "phones": [{"ipa": "i", "stress": "none"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈi", "phones": [{"ipa": "i", "stress": "primary"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_ih.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_ih.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ɪ", "phones": [{"ipa": "ɪ", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_o.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_o.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "o", "phones": [{"ipa": "o", "stress": "none"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈo", "phones": [{"ipa": "o", "stress": "primary"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ʊ", "phones": [{"ipa": "ʊ", "stress": "none"}]}
|
||||
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_u.json
Normal file
1
tools/aqua-listen/kits/tract-weak-spots/phones/v_u.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "u", "phones": [{"ipa": "u", "stress": "none"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"language": "en-US", "source": "ˈu", "phones": [{"ipa": "u", "stress": "primary"}]}
|
||||
29
tools/aqua-listen/kits/tract-weak-spots/render.sh
Executable file
29
tools/aqua-listen/kits/tract-weak-spots/render.sh
Executable file
|
|
@ -0,0 +1,29 @@
|
|||
#!/bin/bash
|
||||
# Renders the tract-weak-spots kit and the renders tools/formant-check/weak_spots.py measures (WordRender, 140 Hz, 48 kHz mono 16-bit).
|
||||
# render.sh <repo-root> <out-root> [<pre-fix-repo-root>] [jobs]
|
||||
# <pre-fix-repo-root> is a checkout of main c0e6b98 (before the release_flow fix); without it the leak-nibu-flat-before clip is not made.
|
||||
# Needs CULTLIB_ROOT at the admitted CultLib revision and AQUASYNTH_FAUST_HOME. Under load each render is minutes (a Faust compile).
|
||||
# <out-root>/out/<w>/<w>.wav 140 Hz renders; v_<vowel>_s is the vowel with primary stress, nibu has it on /i/
|
||||
# <out-root>/out148/v_<vowel> the unstressed vowel at 148.4 Hz (140 x 1.06: the f0 rise of primary stress alone)
|
||||
# <out-root>/outnb/<w> the same word with --no-burst (plosive release_flow splines dropped)
|
||||
# <out-root>/clips/*.wav the files kit.json names; weak_spots.py <out-root> measures the rest
|
||||
set -eu
|
||||
REPO=$(cd "$1" && pwd); OUT=$2; PRE=${3:-}; JOBS=${4:-4}; HERE=$REPO/tools/aqua-listen/kits/tract-weak-spots
|
||||
dotnet build "$REPO/tools/WordRender" -c Release -v q -o "$OUT/wr" | grep -E "rror\(s\)"
|
||||
mkdir -p "$OUT/out" "$OUT/out148" "$OUT/outnb" "$OUT/clips" "$OUT/pre"
|
||||
job() { n=$1; o=$2; f=$3; shift 3; dotnet "$OUT/wr/WordRender.dll" "$OUT/$o/$n" "$n" "$f" "$HERE/phones/$n.json" "$@" > "$OUT/$o/$n.log" 2>&1 && echo "$n $o ok" || echo "$n $o FAILED (see $OUT/$o/$n.log)"; }
|
||||
export -f job; export OUT HERE
|
||||
{
|
||||
for n in nibu nibu_flat v_i v_i_s bin bun ba ama ana v_ih v_oo_short v_ah v_ah_s v_e v_e_s v_o v_o_s v_u v_u_s; do echo "$n out 140"; done
|
||||
for n in bin ba bun; do echo "$n outnb 140 --no-burst"; done
|
||||
for v in i e ah o u ih oo_short; do echo "v_$v out148 148.4"; done
|
||||
} | xargs -P "$JOBS" -L1 bash -c 'job "$@"' _
|
||||
if [ -n "$PRE" ]; then
|
||||
dotnet build "$PRE/tools/WordRender" -c Release -v q -o "$OUT/wr-pre" | grep -E "rror\(s\)"
|
||||
dotnet "$OUT/wr-pre/WordRender.dll" "$OUT/pre/nibu_flat" nibu_flat 140 "$HERE/phones/nibu_flat.json" > "$OUT/pre/nibu_flat.log" 2>&1
|
||||
cp "$OUT/pre/nibu_flat/nibu_flat.wav" "$OUT/clips/nibu_flat_before.wav"
|
||||
fi
|
||||
C=$OUT/clips
|
||||
cp "$OUT/out/nibu_flat/nibu_flat.wav" "$C/nibu_flat.wav"; cp "$OUT/out/nibu_flat/nibu_flat.wav" "$C/nibu_flat_after.wav" # one render, two ids: the before/after pair is one-to-one
|
||||
for n in nibu v_i v_i_s v_ih v_oo_short ama ana ba bun; do cp "$OUT/out/$n/$n.wav" "$C/$n.wav"; done
|
||||
for n in ba bun; do cp "$OUT/outnb/$n/$n.wav" "$C/${n}_noburst.wav"; done
|
||||
117
tools/formant-check/weak_spots.py
Normal file
117
tools/formant-check/weak_spots.py
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Re-measures the tract voice's four listening weak spots from WordRender outputs. Measures; it does not listen.
|
||||
weak_spots.py ROOT [--json OUT.json]
|
||||
ROOT holds: out/<word>/<word>.wav (140 Hz renders: v_<vowel>, v_<vowel>_s with primary stress, ama, ana, bin, bun, ba, nibu, nibu_flat),
|
||||
out148/v_<vowel> (the five stress vowels unstressed at 148.4 Hz: the f0 rise alone), outnb/<word> (the same word with --no-burst); --burst-sub names the directory of the with-burst bin/ba/bun renders.
|
||||
A stress: level of the steady part (100-170 ms) of stressed against unstressed vowels, split into the f0 rise alone and the rest.
|
||||
B /b/ release: burst alone (render minus --no-burst render), its peak and decay against the vowel peak, the closure, the tail.
|
||||
C nasals: murmur level against the flanking vowels (the steady-part method of verify_renders.py), Burg N1, band levels of m against n.
|
||||
D lax vowels: Burg F1/F2 of /ɪ/ /ʊ/ against the tube chain and Hillenbrand 1995 / Peterson & Barney 1952 male means.
|
||||
"""
|
||||
import sys, os, json, wave, argparse
|
||||
import numpy as np
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
import burg_formants as bf
|
||||
|
||||
def wav(path):
|
||||
with wave.open(path) as w:
|
||||
return np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(float) / 32768.0, w.getframerate()
|
||||
|
||||
def rms(x, sr, a, b):
|
||||
s = x[int(a * sr):int(b * sr)]
|
||||
return float(np.sqrt((s ** 2).mean())) if len(s) else 0.0
|
||||
|
||||
def peak(x, sr, a, b):
|
||||
s = x[int(a * sr):int(b * sr)]
|
||||
return float(np.abs(s).max()) if len(s) else 0.0
|
||||
|
||||
def db(a, b): return round(20 * np.log10(max(a, 1e-12) / max(b, 1e-12)), 2)
|
||||
|
||||
def load(root, sub, name):
|
||||
return wav(os.path.join(root, sub, name, name + ".wav"))
|
||||
|
||||
def burg(root, name, t0, t1):
|
||||
x, sr = load(root, "out", name)
|
||||
y = bf.resample(x, sr, 10000); y = np.append(y[0], y[1:] - 0.97 * y[:-1]); fs = 10000
|
||||
n = int(0.03 * fs); hop = int(0.01 * fs); rows = []
|
||||
for s in range(int(t0 * fs), int(t1 * fs) - n, hop):
|
||||
fr = y[s:s + n]
|
||||
if fr.std() < 1e-6: continue
|
||||
rows.append(bf.formants(fr, fs, 12)[:4])
|
||||
return [round(float(np.median([r[i] for r in rows if len(r) > i])), 1) if sum(len(r) > i for r in rows) >= max(3, len(rows) // 2) else None for i in range(3)]
|
||||
|
||||
FIVE = ["i", "ah", "u", "e", "o"] # the vowels the test PrimaryStressRendersAboutTwoDecibelsLouderOnAverageOverVowels holds the mean over
|
||||
HILL = {"ih": (429, 2034), "oo_short": (469, 1123)}
|
||||
PB = {"ih": (392, 1993), "oo_short": (437, 1023)}
|
||||
TUBE = {"ih": (346, 1987), "oo_short": (478, 850)} # tube_formants.py at 17.52 cm
|
||||
|
||||
def stress(root):
|
||||
out = {}
|
||||
for v in FIVE:
|
||||
x0, sr = load(root, "out", f"v_{v}"); xf, _ = load(root, "out148", f"v_{v}"); xs, _ = load(root, "out", f"v_{v}_s")
|
||||
l0, lf, ls = (rms(q, sr, .10, .17) for q in (x0, xf, xs))
|
||||
out[v] = {"total_db": db(ls, l0), "f0_rise_only_db": db(lf, l0), "rest_db": round(db(ls, l0) - db(lf, l0), 2)}
|
||||
for tag, sel in (("mean_five", FIVE),):
|
||||
out[tag] = {k: round(float(np.mean([out[v][k] for v in sel])), 2) for k in ("total_db", "f0_rise_only_db", "rest_db")}
|
||||
out[tag]["total_sd"] = round(float(np.std([out[v]["total_db"] for v in sel])), 2)
|
||||
xa, sr = load(root, "out", "nibu_flat"); xb, _ = load(root, "out", "nibu")
|
||||
out["nibu_i_150_250ms_db"] = db(rms(xb, sr, .15, .25), rms(xa, sr, .15, .25))
|
||||
return out
|
||||
|
||||
def burst(root, name, c0, c1, sub="out"):
|
||||
"""c0..c1: the b gesture's start and end in seconds; v1: the vowel's steady part after it."""
|
||||
x, sr = load(root, sub, name); nb, _ = load(root, "outnb", name)
|
||||
n = min(len(x), len(nb)); d = x[:n] - nb[:n]
|
||||
vp = peak(x, sr, c1 + .06, c1 + .17)
|
||||
i = int(np.argmax(np.abs(d))); tpk = i / sr
|
||||
env = np.array([np.abs(d[k:k + int(.002 * sr)]).max() for k in range(0, n - int(.002 * sr), int(.002 * sr))])
|
||||
pk = float(np.abs(d).max())
|
||||
below = [k for k in range(int(tpk * 500), len(env)) if env[k] < pk * .1]
|
||||
decay_ms = round((below[0] / 500 - tpk) * 1000, 0) if below else None
|
||||
last = x[int((c1 + .06) * sr):]
|
||||
return {"closure_peak": round(peak(x, sr, c0, c0 + .6 * (c1 - c0)), 4), "release_peak_full": round(peak(x, sr, c0 + .6 * (c1 - c0), c1 + .005), 4),
|
||||
"burst_peak": round(pk, 4), "burst_peak_ms": round(tpk * 1000), "burst_to_tenth_ms": decay_ms, "vowel_peak": round(vp, 4),
|
||||
"burst_re_vowel_db": db(pk, vp), "release_peak_re_vowel_db": db(peak(x, sr, c0 + .6 * (c1 - c0), c1 + .005), vp),
|
||||
"burst_rms_in_30ms_after_peak_re_vowel_rms_db": db(rms(d, sr, tpk, tpk + .03), rms(x, sr, c1 + .06, c1 + .17)),
|
||||
"burst_peak_from_120ms_after_release_to_end": round(peak(d, sr, c1 + .12, n / sr), 4), "burst_diff_samples_nonzero_before_closure_end": int(np.count_nonzero(np.abs(d[:int((c0 + .6 * (c1 - c0)) * sr)]) > 1e-9)),
|
||||
"file_end_rms_last_100ms": round(rms(x, sr, len(x) / sr - .1, len(x) / sr), 5), "vowel_rms": round(rms(x, sr, c1 + .06, c1 + .17), 4)}
|
||||
|
||||
def bands(x, sr, a, b, edges=((0, 400), (400, 800), (800, 1600), (1600, 3200))):
|
||||
s = x[int(a * sr):int(b * sr)]; s = s * np.hanning(len(s))
|
||||
P = np.abs(np.fft.rfft(s, 1 << 15)) ** 2; f = np.fft.rfftfreq(1 << 15, 1 / sr)
|
||||
return [float(10 * np.log10(P[(f >= lo) & (f < hi)].sum() + 1e-30)) for lo, hi in edges]
|
||||
|
||||
def nasals(root):
|
||||
out = {}
|
||||
for w in ("ama", "ana"):
|
||||
x, sr = load(root, "out", w); c0, c1 = .18, .275
|
||||
v1, v2 = rms(x, sr, c0 - .08, c0 - .01), rms(x, sr, c1 + .03, c1 + .12)
|
||||
m = rms(x, sr, c0 + .2 * (c1 - c0), c1 - .2 * (c1 - c0))
|
||||
out[w] = {"murmur_db": db(m, np.sqrt((v1 ** 2 + v2 ** 2) / 2)), "N1_N2_N3": burg(root, w, .20, .26),
|
||||
"band_db_0_400_400_800_800_1600_1600_3200": [round(q, 1) for q in bands(x, sr, c0 + .2 * (c1 - c0), c1 - .2 * (c1 - c0))],
|
||||
"vowel_band_db": [round(q, 1) for q in bands(x, sr, c0 - .08, c0 - .01)]}
|
||||
out["n_minus_m_band_db"] = [round(a - b, 1) for a, b in zip(out["ana"]["band_db_0_400_400_800_800_1600_1600_3200"], out["ama"]["band_db_0_400_400_800_800_1600_1600_3200"])]
|
||||
out["n_minus_m_vowel_band_db"] = [round(a - b, 1) for a, b in zip(out["ana"]["vowel_band_db"], out["ama"]["vowel_band_db"])]
|
||||
return out
|
||||
|
||||
def lax(root):
|
||||
out = {}
|
||||
for v in ("ih", "oo_short"):
|
||||
f = burg(root, f"v_{v}", .07, .15)
|
||||
pct = lambda a, r: round(100 * (a - r) / r, 1)
|
||||
out[v] = {"burg_F1_F2": f[:2], "tube": TUBE[v], "hillenbrand": HILL[v], "pb": PB[v],
|
||||
"vs_tube_pct": [pct(f[0], TUBE[v][0]), pct(f[1], TUBE[v][1])], "vs_hillenbrand_pct": [pct(f[0], HILL[v][0]), pct(f[1], HILL[v][1])],
|
||||
"vs_pb_pct": [pct(f[0], PB[v][0]), pct(f[1], PB[v][1])]}
|
||||
return out
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("root"); ap.add_argument("--json"); ap.add_argument("--burst-sub", default="out", help="subdirectory of ROOT holding the with-burst renders of bin, ba, bun (outpost after the release_flow fix)"); a = ap.parse_args()
|
||||
res = {"A_stress": stress(a.root), "B_release": {"bin": burst(a.root, "bin", 0, .095, a.burst_sub), "ba": burst(a.root, "ba", 0, .095, a.burst_sub),
|
||||
"bun": burst(a.root, "bun", 0, .095, a.burst_sub)},
|
||||
"C_nasals": nasals(a.root), "D_lax": lax(a.root)}
|
||||
s = json.dumps(res, indent=1, default=float, ensure_ascii=False)
|
||||
print(s)
|
||||
if a.json: open(a.json, "w").write(s + "\n")
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
Loading…
Reference in a new issue