Define speech lowering handoff

2026-05-19 16:35:41 +01:00 · 2026-05-19 16:35:41 +01:00 · e1f3226219
commit e1f3226219
parent 559c8e4cee
10 changed files with 603 additions and 0 deletions
--- a/docs/english-pronunciation-lowering.md
+++ b/docs/english-pronunciation-lowering.md
@ -0,0 +1,130 @@
+# English Pronunciation Lowering
+
+Version: `0.1-draft`
+
+Weksa owns the step from flavored English text to a pronunciation plan. AquaSynth
+owns the later step from pronunciation intent to sound.
+
+```text
+interlingua packet
+  -> projected agent-local context
+  -> flavored English dialogue
+  -> English pronunciation plan
+  -> downstream speech system
+```
+
+This is still Weksa work because the pronunciation plan belongs to the meaning
+and dialogue layer: phrase grouping, emphasis, reduced forms, and word choice are
+part of the utterance Weksa just created.
+
+The long-term primary handoff is the learned
+[Line Expression Vector](./line-expression-vector.md). This pronunciation plan is
+the inspectable audit/debug lane: useful for fixtures, phoneme strings, and
+review, but not the only representation of delivery.
+
+## Objective
+
+Produce a deterministic, inspectable pronunciation plan for English output.
+
+The plan should say:
+
+- what text is being pronounced
+- how it was normalized
+- which tokens are words, punctuation, or phrase boundaries
+- which pronunciation source was used for each word
+- what broad IPA sequence should be sent downstream
+- where phrase breaks and emphasis fall
+- what remains uncertain or hand-authored
+
+## Authority
+
+Weksa owns:
+
+- text normalization
+- tokenization
+- lexicon lookup
+- OOV pronunciation choice
+- ARPABET-to-IPA conversion when using CMUdict-style entries
+- phrase grouping
+- lexical stress preservation
+- dialogue-driven emphasis
+- pronunciation trace
+
+AquaSynth owns:
+
+- IPA parsing
+- phonetic feature interpretation
+- language/voice phonology profile
+- timing curves
+- articulatory gestures
+- tract morphology
+- rendered audio
+
+## Reference Spine
+
+Recommended first spine:
+
+1. Use CMUdict-style ARPABET entries for known English words.
+2. Preserve CMUdict stress digits on vowels.
+3. Convert ARPABET to broad IPA with a documented table.
+4. Add phrase breaks and emphasis from the English lowering output.
+5. Mark hand-authored or uncertain pronunciations explicitly.
+
+Useful reference tools:
+
+- CMU Pronouncing Dictionary for English ARPABET pronunciations.
+- eSpeak NG `--ipa` or phoneme output as an external checker.
+- Phonetisaurus or another G2P model later for OOV words.
+
+The first version may be hand-authored fixture data. The contract matters before
+automation. Yes, this is paperwork with teeth. Annoying, but it bites the right
+thing.
+
+## Pronunciation Plan Shape
+
+```yaml
+schema_version: weksa.english_pronunciation_plan.v0
+source_output_ref:
+dialect:
+orthographic_text:
+normalization:
+  text:
+  notes:
+tokens:
+  - id:
+    text:
+    kind:
+    pronunciation_source:
+    arpabet:
+    ipa:
+    stress:
+    notes:
+phrasing:
+  groups:
+    - id:
+      token_ids:
+      boundary_after:
+      prominence:
+ipa:
+  broad:
+  with_boundaries:
+trace:
+  decisions:
+  uncertainties:
+```
+
+## Minimum Pass Criteria
+
+- Every pronounced word has a token.
+- Every word token has IPA.
+- Unknown or hand-authored pronunciations are labeled.
+- Phrase boundaries do not contradict punctuation or intended delivery.
+- Emphasis follows dialogue intent, not arbitrary prettiness.
+- The plan can be consumed without reading the original prose document.
+
+## References
+
+- [CMU Pronouncing Dictionary](https://www.speech.cs.cmu.edu/cgi-bin/cmudict)
+- [eSpeak command options](https://espeak.sourceforge.net/commands.html)
+- [eSpeak NG documentation index](https://chromium.googlesource.com/chromiumos/third_party/espeak-ng/+/HEAD/docs/index.md)
+- [Phonetisaurus overview](https://cmusphinx.github.io/2012/05/phonetisaurus-a-wfst-driven-phoneticizer-e2-80-93-framework-review/)
--- a/docs/flavored-english-lowering.md
+++ b/docs/flavored-english-lowering.md
@ -130,6 +130,15 @@ Weksa's first deliverable is the English line and traceable projection context.
 When speech work needs it, AquaSynth can consume `spoken_text` and decide how to
 lower it into its own phonetic and articulatory contracts.

+Weksa still owns the optional pronunciation plan for English output: tokenizing
+the line, choosing word pronunciations, preserving stress, and stringing broad
+IPA together. See [English Pronunciation Lowering](./english-pronunciation-lowering.md).
+
+Weksa should also emit a learned line-expression vector when training data and
+encoder support exist. That vector is the intended compact handoff for prosody,
+emphasis, and character delivery pressure. See
+[Line Expression Vector](./line-expression-vector.md).
+
 ## First Target: Nibu

 Nibu is the first practical target because her state is live, opinionated, and
--- a/docs/line-expression-vector.md
+++ b/docs/line-expression-vector.md
@ -0,0 +1,124 @@
+# Line Expression Vector
+
+Version: `0.1-draft`
+
+The line expression vector is Weksa's learned handoff from flavored English
+lowering to downstream voice control.
+
+```text
+interlingua packet
+  -> projected agent-local context
+  -> flavored English line
+  -> Weksa line-expression encoder
+  -> line expression vector
+  -> AquaSynth synth-control decoder
+```
+
+The goal is two small learned artifacts in a tight loop:
+
+1. Weksa learns to encode a character's line as a compact expression vector.
+2. AquaSynth learns to decode that vector into synth-control intent.
+
+No giant oracle gets to own the seam. No hidden theatrical mush. Just a line,
+its meaning, its speaker context, and a learned vector with enough audit surface
+to tell whether it is lying.
+
+## Authority
+
+Weksa owns:
+
+- interlingua packet
+- projected agent-local context
+- flavored English line
+- line-expression vector generation
+- audit projections for meaning, emphasis, and character pressure
+
+AquaSynth owns:
+
+- mapping expression vectors to synthesis controls
+- consuming pronunciation or IPA plans when Weksa provides them
+- phonetic feature interpretation
+- timing, articulation, tract, and DSP
+- audio evaluation
+
+## Vector Contract
+
+The vector is dense and model-owned, but the packet around it must be explicit.
+
+```yaml
+schema_version: weksa.line_expression_vector.v0
+source_output_ref:
+speaker_agent_id:
+encoder:
+  model_id:
+  version:
+  training_set:
+text:
+  spoken_text:
+  normalized_text:
+vector:
+  dimensionality:
+  values_ref:
+audit_projection:
+  semantic_preservation:
+  emphasis:
+  prosody:
+  character_pressure:
+  delivery_shape:
+training_refs:
+  positive_examples:
+  negative_examples:
+trace:
+  notes:
+  uncertainties:
+```
+
+`values_ref` may point to an external binary or JSON artifact once the vector is
+large. Early fixtures may inline tiny toy vectors.
+
+## Audit Projection
+
+The audit projection is not the vector. It is a human-readable probe of what the
+vector is supposed to contain.
+
+Useful audit families:
+
+- `semantic_preservation`: what source meaning must remain audible.
+- `emphasis`: which words or spans carry prominence.
+- `prosody`: clipped, rising, flat, hesitant, compressed, etc.
+- `character_pressure`: active state pressures shaping delivery.
+- `delivery_shape`: pace, pause pattern, intensity, warmth, contempt, tenderness,
+  threat, or play.
+
+These are labels for review and training. The learned vector is allowed to find
+better internal dimensions, but it does not get to skip accountability.
+
+## Training Loop
+
+First loop:
+
+1. Author or generate an interlingua packet.
+2. Project agent-local context.
+3. Lower to English.
+4. Human-review whether the line preserves meaning and voice.
+5. Encode the reviewed line into a line-expression vector.
+6. AquaSynth decodes the vector into synth-control intent.
+7. Compare rendered speech against human notes and reference takes when present.
+8. Feed corrections back into the Weksa encoder and AquaSynth decoder.
+
+The same fixture can carry:
+
+- accepted line
+- rejected line
+- why the rejected line failed
+- audit projection
+- vector artifact
+- downstream AquaSynth evaluation pointer
+
+## Non-Goals
+
+- Do not make Weksa emit raw vocal-tract controls.
+- Do not make AquaSynth infer character psychology from prose.
+- Do not treat named audit axes as the true latent space.
+- Do not make realtime neural voice output the authority.
+- Do not hide bad semantic preservation behind pleasant delivery.
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@ -16,6 +16,10 @@

 - Validate one minimal interlingua packet.
 - Lower one packet through Nibu's projected agent state into flavored English.
+- Add one English pronunciation plan that strings the lowered line into broad
+  IPA.
+- Add one toy line-expression vector fixture, then replace hand-authored values
+  with a learned encoder.
 - Preserve the first lowering proof as fixture data under
  `examples/english-lowering/`.
 - Parse or load one language project definition.
--- a/examples/english-lowering/nibu-proof-001/expression-vector.yaml
+++ b/examples/english-lowering/nibu-proof-001/expression-vector.yaml
@ -0,0 +1,66 @@
+schema_version: weksa.line_expression_vector.v0
+source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml
+speaker_agent_id: nibu
+encoder:
+  model_id: weksa-line-expression-toy
+  version: 0.0-hand-authored-fixture
+  training_set: none
+text:
+  spoken_text: Take your hand off that control unless you want your nervous system to become my least useful circuit diagram.
+  normalized_text: take your hand off that control unless you want your nervous system to become my least useful circuit diagram
+vector:
+  dimensionality: 12
+  values:
+    - 0.92
+    - 0.78
+    - 0.84
+    - 0.18
+    - 0.71
+    - 0.66
+    - 0.22
+    - 0.88
+    - 0.74
+    - 0.31
+    - 0.69
+    - 0.57
+  notes:
+    - Toy vector is hand-authored only to exercise the contract.
+    - Real vectors should come from a learned Weksa encoder.
+audit_projection:
+  semantic_preservation:
+    must_preserve:
+      - player must stop touching the control
+      - control is dangerous
+      - Nibu is the warning source
+    status: preserved
+  emphasis:
+    prominent_spans:
+      - take your hand off
+      - that control
+      - least useful circuit diagram
+  prosody:
+    pace: clipped_fast
+    phrase_breaks:
+      - after control
+    contour: sharp_command_then_dry_threat
+  character_pressure:
+    precise_abrasive_warning: high
+    irritated_protection: high
+    generic_cruelty: low
+    customer_service_politeness: absent
+  delivery_shape:
+    warmth: low
+    contempt: medium_high
+    threat: medium
+    technical_specificity: high
+training_refs:
+  positive_examples:
+    - examples/english-lowering/nibu-proof-001/lowered-output.yaml
+  negative_examples: []
+trace:
+  notes:
+    - First fixture for learned line-expression vector contract.
+    - Values are placeholders, not calibrated latent dimensions.
+  uncertainties:
+    - Need real encoder dimensionality.
+    - Need paired AquaSynth decoder target format.
--- a/examples/english-lowering/nibu-proof-001/pronunciation-plan.yaml
+++ b/examples/english-lowering/nibu-proof-001/pronunciation-plan.yaml
@ -0,0 +1,182 @@
+schema_version: weksa.english_pronunciation_plan.v0
+source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml
+dialect: en-US-general
+orthographic_text: Take your hand off that control unless you want your nervous system to become my least useful circuit diagram.
+normalization:
+  text: take your hand off that control unless you want your nervous system to become my least useful circuit diagram
+  notes:
+    - Lowercased for lookup.
+    - Period removed from pronunciation token stream.
+    - First pass is hand-authored against CMUdict-style ARPABET expectations.
+tokens:
+  - id: t01
+    text: take
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [T, EY1, K]
+    ipa: teɪk
+    stress: primary
+  - id: t02
+    text: your
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [Y, UH1, R]
+    ipa: jʊɹ
+    stress: primary
+    notes: Chosen unreduced form for clipped warning clarity.
+  - id: t03
+    text: hand
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [HH, AE1, N, D]
+    ipa: hænd
+    stress: primary
+  - id: t04
+    text: off
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [AO1, F]
+    ipa: ɔf
+    stress: primary
+  - id: t05
+    text: that
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [DH, AE1, T]
+    ipa: ðæt
+    stress: primary
+  - id: t06
+    text: control
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [K, AH0, N, T, R, OW1, L]
+    ipa: kənˈtɹoʊl
+    stress: final_primary
+  - id: b01
+    text: ","
+    kind: boundary
+    pronunciation_source: authored_phrase_boundary
+    ipa: "|"
+    notes: Inserted after imperative phrase.
+  - id: t07
+    text: unless
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [AH0, N, L, EH1, S]
+    ipa: ənˈlɛs
+    stress: final_primary
+  - id: t08
+    text: you
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [Y, UW1]
+    ipa: ju
+    stress: primary
+  - id: t09
+    text: want
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [W, AA1, N, T]
+    ipa: wɑnt
+    stress: primary
+  - id: t10
+    text: your
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [Y, UH1, R]
+    ipa: jʊɹ
+    stress: primary
+  - id: t11
+    text: nervous
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [N, ER1, V, AH0, S]
+    ipa: ˈnɝvəs
+    stress: initial_primary
+  - id: t12
+    text: system
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [S, IH1, S, T, AH0, M]
+    ipa: ˈsɪstəm
+    stress: initial_primary
+  - id: t13
+    text: to
+    kind: word
+    pronunciation_source: reduced_function_word
+    arpabet: [T, AH0]
+    ipa: tə
+    stress: unstressed
+    notes: Reduced form fits fluent warning.
+  - id: t14
+    text: become
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [B, IH0, K, AH1, M]
+    ipa: bɪˈkʌm
+    stress: final_primary
+  - id: t15
+    text: my
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [M, AY1]
+    ipa: maɪ
+    stress: primary
+  - id: t16
+    text: least
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [L, IY1, S, T]
+    ipa: list
+    stress: primary
+  - id: t17
+    text: useful
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [Y, UW1, S, F, AH0, L]
+    ipa: ˈjusfəl
+    stress: initial_primary
+  - id: t18
+    text: circuit
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [S, ER1, K, AH0, T]
+    ipa: ˈsɝkət
+    stress: initial_primary
+  - id: t19
+    text: diagram
+    kind: word
+    pronunciation_source: cmudict_expected
+    arpabet: [D, AY1, AH0, G, R, AE2, M]
+    ipa: ˈdaɪəˌɡɹæm
+    stress: primary_secondary
+phrasing:
+  groups:
+    - id: imperative
+      token_ids: [t01, t02, t03, t04, t05, t06]
+      boundary_after: short_pause
+      prominence:
+        - t01
+        - t03
+        - t06
+    - id: threat_consequence
+      token_ids: [t07, t08, t09, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19]
+      boundary_after: final
+      prominence:
+        - t09
+        - t16
+        - t18
+        - t19
+ipa:
+  broad: teɪk jʊɹ hænd ɔf ðæt kənˈtɹoʊl ənˈlɛs ju wɑnt jʊɹ ˈnɝvəs ˈsɪstəm tə bɪˈkʌm maɪ list ˈjusfəl ˈsɝkət ˈdaɪəˌɡɹæm
+  with_boundaries: teɪk jʊɹ hænd ɔf ðæt kənˈtɹoʊl | ənˈlɛs ju wɑnt jʊɹ ˈnɝvəs ˈsɪstəm tə bɪˈkʌm maɪ list ˈjusfəl ˈsɝkət ˈdaɪəˌɡɹæm
+trace:
+  decisions:
+    - Used broad General American IPA.
+    - Kept first "your" unreduced for command clarity.
+    - Reduced "to" to schwa in the consequence clause.
+    - Inserted phrase boundary after "control" to match warning delivery.
+  uncertainties:
+    - "off" may be /ɑf/ for some General American speakers.
+    - "your" may reduce toward /jɚ/ depending on voice profile.
+    - "diagram" stress varies; this pass keeps CMUdict-style first primary stress with later secondary stress.
--- a/schemas/README.md
+++ b/schemas/README.md
@ -18,6 +18,11 @@ steer the machine, describe its authority here.
  contract between clients, authoring tools, and language projects.
 - [english-lowering-packet.md](./english-lowering-packet.md): packet contract
  for projecting interlingua through agent state into flavored English.
+- [english-pronunciation-plan.md](./english-pronunciation-plan.md): plan for
+  stringing flavored English into IPA/phoneme sequences before downstream speech.
+- [line-expression-vector.md](./line-expression-vector.md): learned vector
+  contract for handing prosody, emphasis, and character pressure to downstream
+  synth control.

 ## Source Of Truth

--- a/schemas/english-pronunciation-plan.md
+++ b/schemas/english-pronunciation-plan.md
@ -0,0 +1,37 @@
+# English Pronunciation Plan Contract
+
+This is the human-facing schema receipt for English pronunciation plans.
+
+The plan sits after flavored-English lowering and before downstream speech
+systems such as AquaSynth.
+
+## Required Fields
+
+- `schema_version`: currently `weksa.english_pronunciation_plan.v0`.
+- `source_output_ref`: file or packet reference for the lowered English output.
+- `dialect`: pronunciation target such as `en-US-general`.
+- `orthographic_text`: final English text being pronounced.
+- `tokens`: token-level pronunciation records.
+- `ipa`: broad IPA output.
+- `trace`: decisions and uncertainties.
+
+## Token Records
+
+Each token should include:
+
+- `id`
+- `text`
+- `kind`: `word`, `punctuation`, `boundary`, or `symbol`
+- `pronunciation_source`
+- `arpabet`
+- `ipa`
+- `stress`
+- `notes`
+
+`arpabet` may be omitted for punctuation and non-CMUdict sources.
+
+## Boundary
+
+This contract does not define phonetic features, articulatory gestures, tract
+morphology, or audio. It only defines the pronunciation string and the trace that
+explains where it came from.
--- a/schemas/line-expression-vector.md
+++ b/schemas/line-expression-vector.md
@ -0,0 +1,32 @@
+# Line Expression Vector Contract
+
+This is the human-facing schema receipt for Weksa line-expression vectors.
+
+The vector sits after flavored-English lowering and before AquaSynth synth
+control. It is the learned representation of a character's line: meaning,
+emphasis, prosody, and delivery pressure in one compact artifact.
+
+## Required Fields
+
+- `schema_version`: currently `weksa.line_expression_vector.v0`.
+- `source_output_ref`: lowered English output reference.
+- `speaker_agent_id`: speaker identity.
+- `encoder`: model ID and version metadata.
+- `text`: spoken and normalized text.
+- `vector`: dimensionality and either inline values or `values_ref`.
+- `audit_projection`: human-readable probe of what the vector is intended to
+  preserve.
+- `trace`: notes and uncertainties.
+
+## Boundary
+
+The vector is not canonical character state. It is a derived runtime/training
+artifact.
+
+The audit projection is not the true latent space. It is a review surface for
+humans, tests, and training data.
+
+## Downstream
+
+AquaSynth may consume the vector through its own decoder. Weksa does not define
+the synth-control schema here.
--- a/state/map.yaml
+++ b/state/map.yaml
@ -39,6 +39,10 @@ constraints:
    internals must not leak into generated dialogue.
  - AquaSynth owns IPA, phoneme, reference-synth, vocal-tract, and speech parity
    work. Weksa owns interlingua-to-English projection.
+  - Weksa still owns English pronunciation planning: tokenization, word
+    pronunciations, phrase grouping, stress, and broad IPA stringing.
+  - Long-term speech handoff should use learned line-expression vectors from
+    Weksa and learned synth-control decoding in AquaSynth.

 invariants:
  - Ontology owns meaning.
@ -63,6 +67,9 @@ data_flow:
    - accept or decompose source into a Weksa interlingua packet
    - optionally project the packet through an agent-local state context
    - lower projected meaning into flavored English dialogue
+    - optionally produce an English pronunciation plan for the lowered dialogue
+    - optionally encode the line as a learned expression vector for downstream
+      synth control
    - apply alien ontology and cultural salience
    - select grammatical commitments
    - derive morphology and lexical families
@ -194,6 +201,11 @@ open_questions:
    agent state for English lowering?
  - What exact pass/fail rubric proves a Nibu-flavored English line preserves
    interlingua meaning instead of merely sounding sharp?
+  - What English G2P path should become executable first: CMUdict lookup plus
+    ARPABET-to-IPA conversion, eSpeak NG oracle comparison, or a trained OOV
+    model?
+  - What dimensionality and training corpus should the first Weksa
+    line-expression encoder use?

 landed_baseline:
  - Language-neutral state spine.
@ -206,4 +218,6 @@ landed_baseline:
  - Draft Weksa interlingua standard and packet schema receipt.
  - Draft flavored-English lowering contract.
  - Nibu lowering target note.
+  - First English pronunciation plan fixture for Nibu proof output.
+  - First toy line-expression vector fixture for Nibu proof output.
  - First Nibu interlingua-to-English lowering fixture.