Weksa/docs/utterance-embedding.md

Utterance Embedding Handoff

Version: 0.1

The utterance embedding is the compact learned handoff from Weksa-authored line intent to AquaSynth voice control. It is one speech target projection among others, not the only way Weksa-authored utterances can become sound.

interlingua packet
  -> projected Ghostlight/Epiphany-shaped character context
  -> flavored English line
  -> speech text embedding
  -> PanPhon feature sequence
  -> AquaSynth phonetic sequence encoder
  -> phonetic realization embedding
  -> prosody and emphasis hints
  -> character state vector
  -> AquaSynth utterance embedding encoder
  -> utterance embedding
  -> AquaSynth synth driver

Weksa does not own the optimizer. Weksa owns the inspected ingredients that make training possible:

• source interlingua and lowering trace;
• spoken and normalized text;
• speech text embedding reference when the utterance has meaningful text;
• PanPhon feature sequence for IPA/phones, pronunciation, and alien speech surfaces;
• prosody and emphasis hints;
• projected character-state vector;
• audit projection for semantic preservation and delivery pressure;
• training references and human review notes.

AquaSynth owns:

• the tiny learned utterance-embedding encoder;
• the shared packed-buffer gradient-descent/Adam primitive;
• the synth-driver model that consumes the utterance embedding;
• phonetic feature interpretation, timing, articulation, tract, and DSP;
• audio evaluation.

The point is one learned speech handoff, not two little neural courthouses arguing about who gets to wear the robe.

External providers such as MiMo may consume a different Weksa target projection: accepted target-language spoken_text plus provider-neutral delivery intent lowered into provider-specific style controls. That path does not replace this AquaSynth contract and must not mutate AquaSynth-owned embedding fields.

Packet Contract

This is the first fixed training contract. Weksa may evolve later versions, but weksa.utterance_embedding_handoff.v0.1 must remain stable enough for AquaSynth to train against it without discovering that the input floor moved under its feet.

Vector sizes:

• speech_text_embedding: 1024 floats, bge-m3:latest, normalized to [-1, 1] when inlined or stored. This is semantic/textual evidence, not a pronunciation or alien-phonetics embedding.
• panphon_sequence: variable-length IPA/phone sequence with PanPhon feature vectors plus stress, length, tone, boundary, and timing markers. This is the speech-surface evidence channel for English pronunciations, IPA strings, and alien phonetic material.
• phonetic_realization_embedding: 256 floats, AquaSynth-owned learned output from the PanPhon sequence encoder.
• prosody_emphasis_hints: 32 floats, deterministic Weksa projection.
• character_state_vector: 64 floats, deterministic Ghostlight/Epiphany-shaped speaker-state projection.
• utterance_embedding: 64 floats, AquaSynth-owned learned output.

Weksa owns the semantic text vector, PanPhon sequence evidence, prosody vector, character-state vector, and audit projection. AquaSynth owns the 256-float phonetic realization embedding, the 64-float learned utterance embedding, and all training that maps those surfaces into speech automation. If a vector field is not known yet or does not apply, Weksa emits zeroes and records the uncertainty; it does not resize the vector. Vector resizing is a schema-version change.

Do not feed IPA, romanized alien speech, or phoneme strings into speech_text_embedding and pretend the result means anything. English text semantics and phonetic realization are separate witnesses. For pure IPA or alien phonetic fixtures, the text embedding may be zero or point to the source meaning packet while panphon_sequence carries the sound-shape evidence.

The first emitted training artifact is examples/speech-training/tiny-panphon-v0.1/batch.json. It contains six pure IPA seed packets (a, pa, ta, ka, sa, and ma) with zeroed semantic text embeddings, inline PanPhon 22-feature frames, deterministic prosody/emphasis hints, and the Epiphany-compatible neutral 64-slot character state vector. The AquaSynth-owned learned fields are present but marked pending, so training can begin from a real Weksa artifact without pretending Weksa owns the optimizer.

tools/validate_tiny_panphon_handoffs.py is the executable smoke for this fixture. It validates the checked-in batch against the fixed v0.1 dimensions and channel split without creating a general schema framework.

schema_version: weksa.utterance_embedding_handoff.v0.1
source_output_ref:
speaker_agent_id:
text:
  spoken_text:
  normalized_text:
inputs:
  speech_text_embedding:
    model_id: bge-m3:latest
    dimensionality: 1024
    values_ref:
  panphon_sequence:
    model_id: panphon
    source:
    ipa:
    frames_ref:
  prosody_emphasis_hints:
    dimensionality: 32
    values:
  character_state_vector:
    source:
    dimensionality: 64
    values_ref:
phonetic_realization_embedding:
  owner: AquaSynth
  model_id: aquasynth.panphon_sequence_encoder.v0.1
  dimensionality: 256
  values_ref:
utterance_embedding:
  owner: AquaSynth
  model_id:
  version:
  training_set:
  dimensionality: 64
  values_ref:
audit_projection:
  semantic_preservation:
  emphasis:
  prosody:
  character_pressure:
  delivery_shape:
training_refs:
  positive_examples:
  negative_examples:
trace:
  notes:
  uncertainties:

Early fixtures may use toy schema versions for smoke tests, but training fixtures for AquaSynth must use the fixed v0.1 sizes. Large vectors should use values_ref; tiny examples may show truncated previews only if they are marked as non-training fixtures.

Prosody And Character Axes

The v0.1 vector names are review labels, not the true learned latent space. They stabilize Weksa output and help humans inspect what the training packet is trying to preserve.

panphon_sequence is deterministic speech-surface evidence. Its v0.1 implementation uses PanPhon's 22 ternary articulatory features as the per-phone spine, then adds stress, length, tone, boundary, and timing markers. It is not a semantic embedding. AquaSynth trains aquasynth.panphon_sequence_encoder.v0.1 to compress that variable-length sequence into the 256-float phonetic_realization_embedding.

prosody_emphasis_hints uses 32 slots:

1. mean pitch pressure
2. pitch rise
3. pitch fall
4. pitch range
5. loudness pressure
6. loudness attack
7. loudness release
8. speaking rate
9. articulation precision
10. pause pressure
11. phrase-final lengthening
12. stress contrast
13. clippedness
14. breathiness
15. creak pressure
16. smoothness
17. warmth
18. threat
19. urgency
20. hesitation
21. irony/dryness
22. tenderness
23. contempt
24. command force
25. question contour
26. exclamation pressure
27. secrecy/low projection
28. ritual/formality pressure
29. playful pressure
30. fatigue
31. emotional containment
32. reserved

character_state_vector uses 64 slots. The first 32 are current-state pressure, the second 32 are slower speaker traits or situational context. Unknown slots are zero and explained in trace.uncertainties.

Audit Projection

The audit projection is not the embedding. It is a human-readable probe of what the embedding should preserve.

Useful audit families:

• semantic_preservation: what source meaning must remain audible.
• emphasis: which words or spans carry prominence.
• prosody: clipped, rising, flat, hesitant, compressed, etc.
• character_pressure: active state pressures shaping delivery.
• delivery_shape: pace, pause pattern, intensity, warmth, contempt, tenderness, threat, or play.

These labels guide review and training. The learned embedding may find better internal dimensions, but it does not get to skip accountability.

Training Loop

First loop:

1. Author or generate an interlingua packet.
2. Project Ghostlight/Epiphany-shaped character context.
3. Lower to English.
4. Generate or attach a speech text embedding when the line has text semantics.
5. Emit PanPhon sequence evidence from pronunciation/IPA material.
6. Emit prosody/emphasis hints and a projected character state vector.
7. AquaSynth trains or applies the phonetic sequence encoder to produce the 256-float phonetic realization embedding.
8. Human-review whether the line preserves meaning, pronunciation, and voice.
9. AquaSynth trains the utterance embedding encoder.
10. AquaSynth feeds the utterance embedding to the synth driver.
11. Compare rendered speech against human notes and reference takes when present.

The same fixture can carry accepted lines, rejected lines, audit projections, input vectors, downstream AquaSynth evaluation pointers, and notes about why a candidate failed.

Non-Goals

• Do not make Weksa emit raw vocal-tract controls.
• Do not make Weksa train its own speech embedding optimizer.
• Do not make AquaSynth infer character psychology from prose alone.
• Do not treat named audit axes as the true latent space.
• Do not make realtime neural voice output the authority.
• Do not hide bad semantic preservation behind pleasant delivery.