From e1f3226219b1e01ba34d5d618f5541a7dfefa704 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 19 May 2026 16:35:41 +0100 Subject: [PATCH] Define speech lowering handoff --- docs/english-pronunciation-lowering.md | 130 +++++++++++++ docs/flavored-english-lowering.md | 9 + docs/line-expression-vector.md | 124 ++++++++++++ docs/roadmap.md | 4 + .../nibu-proof-001/expression-vector.yaml | 66 +++++++ .../nibu-proof-001/pronunciation-plan.yaml | 182 ++++++++++++++++++ schemas/README.md | 5 + schemas/english-pronunciation-plan.md | 37 ++++ schemas/line-expression-vector.md | 32 +++ state/map.yaml | 14 ++ 10 files changed, 603 insertions(+) create mode 100644 docs/english-pronunciation-lowering.md create mode 100644 docs/line-expression-vector.md create mode 100644 examples/english-lowering/nibu-proof-001/expression-vector.yaml create mode 100644 examples/english-lowering/nibu-proof-001/pronunciation-plan.yaml create mode 100644 schemas/english-pronunciation-plan.md create mode 100644 schemas/line-expression-vector.md diff --git a/docs/english-pronunciation-lowering.md b/docs/english-pronunciation-lowering.md new file mode 100644 index 0000000..702e38b --- /dev/null +++ b/docs/english-pronunciation-lowering.md @@ -0,0 +1,130 @@ +# English Pronunciation Lowering + +Version: `0.1-draft` + +Weksa owns the step from flavored English text to a pronunciation plan. AquaSynth +owns the later step from pronunciation intent to sound. + +```text +interlingua packet + -> projected agent-local context + -> flavored English dialogue + -> English pronunciation plan + -> downstream speech system +``` + +This is still Weksa work because the pronunciation plan belongs to the meaning +and dialogue layer: phrase grouping, emphasis, reduced forms, and word choice are +part of the utterance Weksa just created. + +The long-term primary handoff is the learned +[Line Expression Vector](./line-expression-vector.md). This pronunciation plan is +the inspectable audit/debug lane: useful for fixtures, phoneme strings, and +review, but not the only representation of delivery. + +## Objective + +Produce a deterministic, inspectable pronunciation plan for English output. + +The plan should say: + +- what text is being pronounced +- how it was normalized +- which tokens are words, punctuation, or phrase boundaries +- which pronunciation source was used for each word +- what broad IPA sequence should be sent downstream +- where phrase breaks and emphasis fall +- what remains uncertain or hand-authored + +## Authority + +Weksa owns: + +- text normalization +- tokenization +- lexicon lookup +- OOV pronunciation choice +- ARPABET-to-IPA conversion when using CMUdict-style entries +- phrase grouping +- lexical stress preservation +- dialogue-driven emphasis +- pronunciation trace + +AquaSynth owns: + +- IPA parsing +- phonetic feature interpretation +- language/voice phonology profile +- timing curves +- articulatory gestures +- tract morphology +- rendered audio + +## Reference Spine + +Recommended first spine: + +1. Use CMUdict-style ARPABET entries for known English words. +2. Preserve CMUdict stress digits on vowels. +3. Convert ARPABET to broad IPA with a documented table. +4. Add phrase breaks and emphasis from the English lowering output. +5. Mark hand-authored or uncertain pronunciations explicitly. + +Useful reference tools: + +- CMU Pronouncing Dictionary for English ARPABET pronunciations. +- eSpeak NG `--ipa` or phoneme output as an external checker. +- Phonetisaurus or another G2P model later for OOV words. + +The first version may be hand-authored fixture data. The contract matters before +automation. Yes, this is paperwork with teeth. Annoying, but it bites the right +thing. + +## Pronunciation Plan Shape + +```yaml +schema_version: weksa.english_pronunciation_plan.v0 +source_output_ref: +dialect: +orthographic_text: +normalization: + text: + notes: +tokens: + - id: + text: + kind: + pronunciation_source: + arpabet: + ipa: + stress: + notes: +phrasing: + groups: + - id: + token_ids: + boundary_after: + prominence: +ipa: + broad: + with_boundaries: +trace: + decisions: + uncertainties: +``` + +## Minimum Pass Criteria + +- Every pronounced word has a token. +- Every word token has IPA. +- Unknown or hand-authored pronunciations are labeled. +- Phrase boundaries do not contradict punctuation or intended delivery. +- Emphasis follows dialogue intent, not arbitrary prettiness. +- The plan can be consumed without reading the original prose document. + +## References + +- [CMU Pronouncing Dictionary](https://www.speech.cs.cmu.edu/cgi-bin/cmudict) +- [eSpeak command options](https://espeak.sourceforge.net/commands.html) +- [eSpeak NG documentation index](https://chromium.googlesource.com/chromiumos/third_party/espeak-ng/+/HEAD/docs/index.md) +- [Phonetisaurus overview](https://cmusphinx.github.io/2012/05/phonetisaurus-a-wfst-driven-phoneticizer-e2-80-93-framework-review/) diff --git a/docs/flavored-english-lowering.md b/docs/flavored-english-lowering.md index 0ceb779..63e9bad 100644 --- a/docs/flavored-english-lowering.md +++ b/docs/flavored-english-lowering.md @@ -130,6 +130,15 @@ Weksa's first deliverable is the English line and traceable projection context. When speech work needs it, AquaSynth can consume `spoken_text` and decide how to lower it into its own phonetic and articulatory contracts. +Weksa still owns the optional pronunciation plan for English output: tokenizing +the line, choosing word pronunciations, preserving stress, and stringing broad +IPA together. See [English Pronunciation Lowering](./english-pronunciation-lowering.md). + +Weksa should also emit a learned line-expression vector when training data and +encoder support exist. That vector is the intended compact handoff for prosody, +emphasis, and character delivery pressure. See +[Line Expression Vector](./line-expression-vector.md). + ## First Target: Nibu Nibu is the first practical target because her state is live, opinionated, and diff --git a/docs/line-expression-vector.md b/docs/line-expression-vector.md new file mode 100644 index 0000000..978c96f --- /dev/null +++ b/docs/line-expression-vector.md @@ -0,0 +1,124 @@ +# Line Expression Vector + +Version: `0.1-draft` + +The line expression vector is Weksa's learned handoff from flavored English +lowering to downstream voice control. + +```text +interlingua packet + -> projected agent-local context + -> flavored English line + -> Weksa line-expression encoder + -> line expression vector + -> AquaSynth synth-control decoder +``` + +The goal is two small learned artifacts in a tight loop: + +1. Weksa learns to encode a character's line as a compact expression vector. +2. AquaSynth learns to decode that vector into synth-control intent. + +No giant oracle gets to own the seam. No hidden theatrical mush. Just a line, +its meaning, its speaker context, and a learned vector with enough audit surface +to tell whether it is lying. + +## Authority + +Weksa owns: + +- interlingua packet +- projected agent-local context +- flavored English line +- line-expression vector generation +- audit projections for meaning, emphasis, and character pressure + +AquaSynth owns: + +- mapping expression vectors to synthesis controls +- consuming pronunciation or IPA plans when Weksa provides them +- phonetic feature interpretation +- timing, articulation, tract, and DSP +- audio evaluation + +## Vector Contract + +The vector is dense and model-owned, but the packet around it must be explicit. + +```yaml +schema_version: weksa.line_expression_vector.v0 +source_output_ref: +speaker_agent_id: +encoder: + model_id: + version: + training_set: +text: + spoken_text: + normalized_text: +vector: + dimensionality: + values_ref: +audit_projection: + semantic_preservation: + emphasis: + prosody: + character_pressure: + delivery_shape: +training_refs: + positive_examples: + negative_examples: +trace: + notes: + uncertainties: +``` + +`values_ref` may point to an external binary or JSON artifact once the vector is +large. Early fixtures may inline tiny toy vectors. + +## Audit Projection + +The audit projection is not the vector. It is a human-readable probe of what the +vector is supposed to contain. + +Useful audit families: + +- `semantic_preservation`: what source meaning must remain audible. +- `emphasis`: which words or spans carry prominence. +- `prosody`: clipped, rising, flat, hesitant, compressed, etc. +- `character_pressure`: active state pressures shaping delivery. +- `delivery_shape`: pace, pause pattern, intensity, warmth, contempt, tenderness, + threat, or play. + +These are labels for review and training. The learned vector is allowed to find +better internal dimensions, but it does not get to skip accountability. + +## Training Loop + +First loop: + +1. Author or generate an interlingua packet. +2. Project agent-local context. +3. Lower to English. +4. Human-review whether the line preserves meaning and voice. +5. Encode the reviewed line into a line-expression vector. +6. AquaSynth decodes the vector into synth-control intent. +7. Compare rendered speech against human notes and reference takes when present. +8. Feed corrections back into the Weksa encoder and AquaSynth decoder. + +The same fixture can carry: + +- accepted line +- rejected line +- why the rejected line failed +- audit projection +- vector artifact +- downstream AquaSynth evaluation pointer + +## Non-Goals + +- Do not make Weksa emit raw vocal-tract controls. +- Do not make AquaSynth infer character psychology from prose. +- Do not treat named audit axes as the true latent space. +- Do not make realtime neural voice output the authority. +- Do not hide bad semantic preservation behind pleasant delivery. diff --git a/docs/roadmap.md b/docs/roadmap.md index d26773c..e7bcd7f 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -16,6 +16,10 @@ - Validate one minimal interlingua packet. - Lower one packet through Nibu's projected agent state into flavored English. +- Add one English pronunciation plan that strings the lowered line into broad + IPA. +- Add one toy line-expression vector fixture, then replace hand-authored values + with a learned encoder. - Preserve the first lowering proof as fixture data under `examples/english-lowering/`. - Parse or load one language project definition. diff --git a/examples/english-lowering/nibu-proof-001/expression-vector.yaml b/examples/english-lowering/nibu-proof-001/expression-vector.yaml new file mode 100644 index 0000000..406928d --- /dev/null +++ b/examples/english-lowering/nibu-proof-001/expression-vector.yaml @@ -0,0 +1,66 @@ +schema_version: weksa.line_expression_vector.v0 +source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml +speaker_agent_id: nibu +encoder: + model_id: weksa-line-expression-toy + version: 0.0-hand-authored-fixture + training_set: none +text: + spoken_text: Take your hand off that control unless you want your nervous system to become my least useful circuit diagram. + normalized_text: take your hand off that control unless you want your nervous system to become my least useful circuit diagram +vector: + dimensionality: 12 + values: + - 0.92 + - 0.78 + - 0.84 + - 0.18 + - 0.71 + - 0.66 + - 0.22 + - 0.88 + - 0.74 + - 0.31 + - 0.69 + - 0.57 + notes: + - Toy vector is hand-authored only to exercise the contract. + - Real vectors should come from a learned Weksa encoder. +audit_projection: + semantic_preservation: + must_preserve: + - player must stop touching the control + - control is dangerous + - Nibu is the warning source + status: preserved + emphasis: + prominent_spans: + - take your hand off + - that control + - least useful circuit diagram + prosody: + pace: clipped_fast + phrase_breaks: + - after control + contour: sharp_command_then_dry_threat + character_pressure: + precise_abrasive_warning: high + irritated_protection: high + generic_cruelty: low + customer_service_politeness: absent + delivery_shape: + warmth: low + contempt: medium_high + threat: medium + technical_specificity: high +training_refs: + positive_examples: + - examples/english-lowering/nibu-proof-001/lowered-output.yaml + negative_examples: [] +trace: + notes: + - First fixture for learned line-expression vector contract. + - Values are placeholders, not calibrated latent dimensions. + uncertainties: + - Need real encoder dimensionality. + - Need paired AquaSynth decoder target format. diff --git a/examples/english-lowering/nibu-proof-001/pronunciation-plan.yaml b/examples/english-lowering/nibu-proof-001/pronunciation-plan.yaml new file mode 100644 index 0000000..b0b4838 --- /dev/null +++ b/examples/english-lowering/nibu-proof-001/pronunciation-plan.yaml @@ -0,0 +1,182 @@ +schema_version: weksa.english_pronunciation_plan.v0 +source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml +dialect: en-US-general +orthographic_text: Take your hand off that control unless you want your nervous system to become my least useful circuit diagram. +normalization: + text: take your hand off that control unless you want your nervous system to become my least useful circuit diagram + notes: + - Lowercased for lookup. + - Period removed from pronunciation token stream. + - First pass is hand-authored against CMUdict-style ARPABET expectations. +tokens: + - id: t01 + text: take + kind: word + pronunciation_source: cmudict_expected + arpabet: [T, EY1, K] + ipa: teɪk + stress: primary + - id: t02 + text: your + kind: word + pronunciation_source: cmudict_expected + arpabet: [Y, UH1, R] + ipa: jʊɹ + stress: primary + notes: Chosen unreduced form for clipped warning clarity. + - id: t03 + text: hand + kind: word + pronunciation_source: cmudict_expected + arpabet: [HH, AE1, N, D] + ipa: hænd + stress: primary + - id: t04 + text: off + kind: word + pronunciation_source: cmudict_expected + arpabet: [AO1, F] + ipa: ɔf + stress: primary + - id: t05 + text: that + kind: word + pronunciation_source: cmudict_expected + arpabet: [DH, AE1, T] + ipa: ðæt + stress: primary + - id: t06 + text: control + kind: word + pronunciation_source: cmudict_expected + arpabet: [K, AH0, N, T, R, OW1, L] + ipa: kənˈtɹoʊl + stress: final_primary + - id: b01 + text: "," + kind: boundary + pronunciation_source: authored_phrase_boundary + ipa: "|" + notes: Inserted after imperative phrase. + - id: t07 + text: unless + kind: word + pronunciation_source: cmudict_expected + arpabet: [AH0, N, L, EH1, S] + ipa: ənˈlɛs + stress: final_primary + - id: t08 + text: you + kind: word + pronunciation_source: cmudict_expected + arpabet: [Y, UW1] + ipa: ju + stress: primary + - id: t09 + text: want + kind: word + pronunciation_source: cmudict_expected + arpabet: [W, AA1, N, T] + ipa: wɑnt + stress: primary + - id: t10 + text: your + kind: word + pronunciation_source: cmudict_expected + arpabet: [Y, UH1, R] + ipa: jʊɹ + stress: primary + - id: t11 + text: nervous + kind: word + pronunciation_source: cmudict_expected + arpabet: [N, ER1, V, AH0, S] + ipa: ˈnɝvəs + stress: initial_primary + - id: t12 + text: system + kind: word + pronunciation_source: cmudict_expected + arpabet: [S, IH1, S, T, AH0, M] + ipa: ˈsɪstəm + stress: initial_primary + - id: t13 + text: to + kind: word + pronunciation_source: reduced_function_word + arpabet: [T, AH0] + ipa: tə + stress: unstressed + notes: Reduced form fits fluent warning. + - id: t14 + text: become + kind: word + pronunciation_source: cmudict_expected + arpabet: [B, IH0, K, AH1, M] + ipa: bɪˈkʌm + stress: final_primary + - id: t15 + text: my + kind: word + pronunciation_source: cmudict_expected + arpabet: [M, AY1] + ipa: maɪ + stress: primary + - id: t16 + text: least + kind: word + pronunciation_source: cmudict_expected + arpabet: [L, IY1, S, T] + ipa: list + stress: primary + - id: t17 + text: useful + kind: word + pronunciation_source: cmudict_expected + arpabet: [Y, UW1, S, F, AH0, L] + ipa: ˈjusfəl + stress: initial_primary + - id: t18 + text: circuit + kind: word + pronunciation_source: cmudict_expected + arpabet: [S, ER1, K, AH0, T] + ipa: ˈsɝkət + stress: initial_primary + - id: t19 + text: diagram + kind: word + pronunciation_source: cmudict_expected + arpabet: [D, AY1, AH0, G, R, AE2, M] + ipa: ˈdaɪəˌɡɹæm + stress: primary_secondary +phrasing: + groups: + - id: imperative + token_ids: [t01, t02, t03, t04, t05, t06] + boundary_after: short_pause + prominence: + - t01 + - t03 + - t06 + - id: threat_consequence + token_ids: [t07, t08, t09, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19] + boundary_after: final + prominence: + - t09 + - t16 + - t18 + - t19 +ipa: + broad: teɪk jʊɹ hænd ɔf ðæt kənˈtɹoʊl ənˈlɛs ju wɑnt jʊɹ ˈnɝvəs ˈsɪstəm tə bɪˈkʌm maɪ list ˈjusfəl ˈsɝkət ˈdaɪəˌɡɹæm + with_boundaries: teɪk jʊɹ hænd ɔf ðæt kənˈtɹoʊl | ənˈlɛs ju wɑnt jʊɹ ˈnɝvəs ˈsɪstəm tə bɪˈkʌm maɪ list ˈjusfəl ˈsɝkət ˈdaɪəˌɡɹæm +trace: + decisions: + - Used broad General American IPA. + - Kept first "your" unreduced for command clarity. + - Reduced "to" to schwa in the consequence clause. + - Inserted phrase boundary after "control" to match warning delivery. + uncertainties: + - "off" may be /ɑf/ for some General American speakers. + - "your" may reduce toward /jɚ/ depending on voice profile. + - "diagram" stress varies; this pass keeps CMUdict-style first primary stress with later secondary stress. diff --git a/schemas/README.md b/schemas/README.md index d292ecc..60e7cfa 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -18,6 +18,11 @@ steer the machine, describe its authority here. contract between clients, authoring tools, and language projects. - [english-lowering-packet.md](./english-lowering-packet.md): packet contract for projecting interlingua through agent state into flavored English. +- [english-pronunciation-plan.md](./english-pronunciation-plan.md): plan for + stringing flavored English into IPA/phoneme sequences before downstream speech. +- [line-expression-vector.md](./line-expression-vector.md): learned vector + contract for handing prosody, emphasis, and character pressure to downstream + synth control. ## Source Of Truth diff --git a/schemas/english-pronunciation-plan.md b/schemas/english-pronunciation-plan.md new file mode 100644 index 0000000..e9efad7 --- /dev/null +++ b/schemas/english-pronunciation-plan.md @@ -0,0 +1,37 @@ +# English Pronunciation Plan Contract + +This is the human-facing schema receipt for English pronunciation plans. + +The plan sits after flavored-English lowering and before downstream speech +systems such as AquaSynth. + +## Required Fields + +- `schema_version`: currently `weksa.english_pronunciation_plan.v0`. +- `source_output_ref`: file or packet reference for the lowered English output. +- `dialect`: pronunciation target such as `en-US-general`. +- `orthographic_text`: final English text being pronounced. +- `tokens`: token-level pronunciation records. +- `ipa`: broad IPA output. +- `trace`: decisions and uncertainties. + +## Token Records + +Each token should include: + +- `id` +- `text` +- `kind`: `word`, `punctuation`, `boundary`, or `symbol` +- `pronunciation_source` +- `arpabet` +- `ipa` +- `stress` +- `notes` + +`arpabet` may be omitted for punctuation and non-CMUdict sources. + +## Boundary + +This contract does not define phonetic features, articulatory gestures, tract +morphology, or audio. It only defines the pronunciation string and the trace that +explains where it came from. diff --git a/schemas/line-expression-vector.md b/schemas/line-expression-vector.md new file mode 100644 index 0000000..bbf7eaf --- /dev/null +++ b/schemas/line-expression-vector.md @@ -0,0 +1,32 @@ +# Line Expression Vector Contract + +This is the human-facing schema receipt for Weksa line-expression vectors. + +The vector sits after flavored-English lowering and before AquaSynth synth +control. It is the learned representation of a character's line: meaning, +emphasis, prosody, and delivery pressure in one compact artifact. + +## Required Fields + +- `schema_version`: currently `weksa.line_expression_vector.v0`. +- `source_output_ref`: lowered English output reference. +- `speaker_agent_id`: speaker identity. +- `encoder`: model ID and version metadata. +- `text`: spoken and normalized text. +- `vector`: dimensionality and either inline values or `values_ref`. +- `audit_projection`: human-readable probe of what the vector is intended to + preserve. +- `trace`: notes and uncertainties. + +## Boundary + +The vector is not canonical character state. It is a derived runtime/training +artifact. + +The audit projection is not the true latent space. It is a review surface for +humans, tests, and training data. + +## Downstream + +AquaSynth may consume the vector through its own decoder. Weksa does not define +the synth-control schema here. diff --git a/state/map.yaml b/state/map.yaml index 9b943c2..d605cec 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -39,6 +39,10 @@ constraints: internals must not leak into generated dialogue. - AquaSynth owns IPA, phoneme, reference-synth, vocal-tract, and speech parity work. Weksa owns interlingua-to-English projection. + - Weksa still owns English pronunciation planning: tokenization, word + pronunciations, phrase grouping, stress, and broad IPA stringing. + - Long-term speech handoff should use learned line-expression vectors from + Weksa and learned synth-control decoding in AquaSynth. invariants: - Ontology owns meaning. @@ -63,6 +67,9 @@ data_flow: - accept or decompose source into a Weksa interlingua packet - optionally project the packet through an agent-local state context - lower projected meaning into flavored English dialogue + - optionally produce an English pronunciation plan for the lowered dialogue + - optionally encode the line as a learned expression vector for downstream + synth control - apply alien ontology and cultural salience - select grammatical commitments - derive morphology and lexical families @@ -194,6 +201,11 @@ open_questions: agent state for English lowering? - What exact pass/fail rubric proves a Nibu-flavored English line preserves interlingua meaning instead of merely sounding sharp? + - What English G2P path should become executable first: CMUdict lookup plus + ARPABET-to-IPA conversion, eSpeak NG oracle comparison, or a trained OOV + model? + - What dimensionality and training corpus should the first Weksa + line-expression encoder use? landed_baseline: - Language-neutral state spine. @@ -206,4 +218,6 @@ landed_baseline: - Draft Weksa interlingua standard and packet schema receipt. - Draft flavored-English lowering contract. - Nibu lowering target note. + - First English pronunciation plan fixture for Nibu proof output. + - First toy line-expression vector fixture for Nibu proof output. - First Nibu interlingua-to-English lowering fixture.