diff --git a/docs/utterance-embedding.md b/docs/utterance-embedding.md index 6528b9d..08f6cf1 100644 --- a/docs/utterance-embedding.md +++ b/docs/utterance-embedding.md @@ -1,6 +1,6 @@ # Utterance Embedding Handoff -Version: `0.1-draft` +Version: `0.1` The utterance embedding is the compact learned handoff from Weksa-authored line intent to AquaSynth voice control. @@ -10,6 +10,9 @@ interlingua packet -> projected Ghostlight/Epiphany-shaped character context -> flavored English line -> speech text embedding + -> PanPhon feature sequence + -> AquaSynth phonetic sequence encoder + -> phonetic realization embedding -> prosody and emphasis hints -> character state vector -> AquaSynth utterance embedding encoder @@ -22,7 +25,9 @@ training possible: - source interlingua and lowering trace; - spoken and normalized text; -- speech text embedding reference; +- speech text embedding reference when the utterance has meaningful text; +- PanPhon feature sequence for IPA/phones, pronunciation, and alien speech + surfaces; - prosody and emphasis hints; - projected character-state vector; - audit projection for semantic preservation and delivery pressure; @@ -41,8 +46,43 @@ arguing about who gets to wear the robe. ## Packet Contract +This is the first fixed training contract. Weksa may evolve later versions, but +`weksa.utterance_embedding_handoff.v0.1` must remain stable enough for +AquaSynth to train against it without discovering that the input floor moved +under its feet. + +Vector sizes: + +- `speech_text_embedding`: 1024 floats, `bge-m3:latest`, normalized to `[-1, 1]` + when inlined or stored. This is semantic/textual evidence, not a pronunciation + or alien-phonetics embedding. +- `panphon_sequence`: variable-length IPA/phone sequence with PanPhon feature + vectors plus stress, length, tone, boundary, and timing markers. This is the + speech-surface evidence channel for English pronunciations, IPA strings, and + alien phonetic material. +- `phonetic_realization_embedding`: 256 floats, AquaSynth-owned learned output + from the PanPhon sequence encoder. +- `prosody_emphasis_hints`: 32 floats, deterministic Weksa projection. +- `character_state_vector`: 64 floats, deterministic Ghostlight/Epiphany-shaped + speaker-state projection. +- `utterance_embedding`: 64 floats, AquaSynth-owned learned output. + +Weksa owns the semantic text vector, PanPhon sequence evidence, prosody vector, +character-state vector, and audit projection. AquaSynth owns the 256-float +phonetic realization embedding, the 64-float learned utterance embedding, and +all training that maps those surfaces into speech automation. If a vector field +is not known yet or does not apply, Weksa emits zeroes and records the +uncertainty; it does not resize the vector. Vector resizing is a schema-version +change. + +Do not feed IPA, romanized alien speech, or phoneme strings into +`speech_text_embedding` and pretend the result means anything. English text +semantics and phonetic realization are separate witnesses. For pure IPA or alien +phonetic fixtures, the text embedding may be zero or point to the source +meaning packet while `panphon_sequence` carries the sound-shape evidence. + ```yaml -schema_version: weksa.utterance_embedding_handoff.v0 +schema_version: weksa.utterance_embedding_handoff.v0.1 source_output_ref: speaker_agent_id: text: @@ -50,22 +90,32 @@ text: normalized_text: inputs: speech_text_embedding: - model_id: - dimensionality: + model_id: bge-m3:latest + dimensionality: 1024 values_ref: + panphon_sequence: + model_id: panphon + source: + ipa: + frames_ref: +phonetic_realization_embedding: + owner: AquaSynth + model_id: aquasynth.panphon_sequence_encoder.v0.1 + dimensionality: 256 + values_ref: prosody_emphasis_hints: - dimensionality: + dimensionality: 32 values: character_state_vector: source: - dimensionality: + dimensionality: 64 values_ref: utterance_embedding: owner: AquaSynth model_id: version: training_set: - dimensionality: + dimensionality: 64 values_ref: audit_projection: semantic_preservation: @@ -81,8 +131,62 @@ trace: uncertainties: ``` -Early fixtures may inline tiny toy vectors. Real embeddings should use -`values_ref` once the dimensions stop being decorative furniture. +Early fixtures may use toy schema versions for smoke tests, but training +fixtures for AquaSynth must use the fixed v0.1 sizes. Large vectors should use +`values_ref`; tiny examples may show truncated previews only if they are marked +as non-training fixtures. + +## Prosody And Character Axes + +The v0.1 vector names are review labels, not the true learned latent space. +They stabilize Weksa output and help humans inspect what the training packet is +trying to preserve. + +`panphon_sequence` is deterministic speech-surface evidence. Its first v0.1 +implementation should use PanPhon-style ternary articulatory features as the +per-phone spine, then add stress, length, tone, boundary, and timing markers. +It is not a semantic embedding. AquaSynth trains +`aquasynth.panphon_sequence_encoder.v0.1` to compress that variable-length +sequence into the 256-float `phonetic_realization_embedding`. + +`prosody_emphasis_hints` uses 32 slots: + +1. mean pitch pressure +2. pitch rise +3. pitch fall +4. pitch range +5. loudness pressure +6. loudness attack +7. loudness release +8. speaking rate +9. articulation precision +10. pause pressure +11. phrase-final lengthening +12. stress contrast +13. clippedness +14. breathiness +15. creak pressure +16. smoothness +17. warmth +18. threat +19. urgency +20. hesitation +21. irony/dryness +22. tenderness +23. contempt +24. command force +25. question contour +26. exclamation pressure +27. secrecy/low projection +28. ritual/formality pressure +29. playful pressure +30. fatigue +31. emotional containment +32. reserved + +`character_state_vector` uses 64 slots. The first 32 are current-state pressure, +the second 32 are slower speaker traits or situational context. Unknown slots +are zero and explained in `trace.uncertainties`. ## Audit Projection @@ -108,12 +212,13 @@ First loop: 1. Author or generate an interlingua packet. 2. Project Ghostlight/Epiphany-shaped character context. 3. Lower to English. -4. Generate or attach a speech text embedding. -5. Emit prosody/emphasis hints and a projected character state vector. -6. Human-review whether the line preserves meaning and voice. -7. AquaSynth trains the utterance embedding encoder. -8. AquaSynth feeds the utterance embedding to the synth driver. -9. Compare rendered speech against human notes and reference takes when present. +4. Generate or attach a speech text embedding when the line has text semantics. +5. Emit or attach a phonetic realization vector from pronunciation/IPA evidence. +6. Emit prosody/emphasis hints and a projected character state vector. +7. Human-review whether the line preserves meaning, pronunciation, and voice. +8. AquaSynth trains the utterance embedding encoder. +9. AquaSynth feeds the utterance embedding to the synth driver. +10. Compare rendered speech against human notes and reference takes when present. The same fixture can carry accepted lines, rejected lines, audit projections, input vectors, downstream AquaSynth evaluation pointers, and notes about why a diff --git a/examples/english-lowering/nibu-proof-001/utterance-embedding.yaml b/examples/english-lowering/nibu-proof-001/utterance-embedding.yaml index a133573..4059317 100644 --- a/examples/english-lowering/nibu-proof-001/utterance-embedding.yaml +++ b/examples/english-lowering/nibu-proof-001/utterance-embedding.yaml @@ -1,4 +1,4 @@ -schema_version: weksa.utterance_embedding_handoff.v0 +schema_version: weksa.utterance_embedding_handoff.v0-toy source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml speaker_agent_id: nibu text: @@ -83,8 +83,9 @@ training_refs: negative_examples: [] trace: notes: - - First fixture for learned utterance embedding handoff contract. + - First toy fixture for learned utterance embedding handoff contract. - Values are placeholders, not calibrated latent dimensions. + - This fixture predates the v0.1 fixed-size training contract and must not be used as an AquaSynth training packet. uncertainties: - Need real encoder dimensionality. - Need paired AquaSynth decoder target format. diff --git a/schemas/utterance-embedding-handoff.md b/schemas/utterance-embedding-handoff.md index b6b22c2..98f2700 100644 --- a/schemas/utterance-embedding-handoff.md +++ b/schemas/utterance-embedding-handoff.md @@ -5,22 +5,29 @@ embedding handoff packets. The packet sits after flavored-English lowering and before AquaSynth speech control. It carries the inspected inputs for a learned utterance embedding: -speech text embedding, prosody/emphasis hints, and a projected character-state -vector from the speaker's Ghostlight/Epiphany-shaped profile. +speech text embedding, PanPhon sequence evidence, prosody/emphasis hints, a +projected character-state vector from the speaker's Ghostlight/Epiphany-shaped +profile, and AquaSynth's phonetic realization embedding when available. ## Required Fields -- `schema_version`: currently `weksa.utterance_embedding_handoff.v0`. +- `schema_version`: currently `weksa.utterance_embedding_handoff.v0.1`. - `source_output_ref`: lowered English output reference. - `speaker_agent_id`: speaker identity. - `text`: spoken and normalized text. - `inputs.speech_text_embedding`: model metadata, dimensionality, and values or - `values_ref`. + `values_ref`; v0.1 uses `bge-m3:latest` with 1024 floats. This channel is for + text semantics, not IPA or alien phonetic strings. +- `inputs.panphon_sequence`: pronunciation/IPA/phone metadata plus PanPhon + feature frames and stress, length, tone, boundary, and timing markers. - `inputs.prosody_emphasis_hints`: compact deterministic hints for emphasis, - contour, pace, intensity, and pause pressure. -- `inputs.character_state_vector`: projected speaker-state vector and source. + contour, pace, intensity, and pause pressure; v0.1 uses 32 floats. +- `inputs.character_state_vector`: projected speaker-state vector and source; + v0.1 uses 64 floats. +- `phonetic_realization_embedding`: AquaSynth-owned 256-float embedding produced + by `aquasynth.panphon_sequence_encoder.v0.1` when available. - `utterance_embedding`: AquaSynth-owned model metadata and produced vector - reference when available. + reference when available; v0.1 uses 64 floats. - `audit_projection`: human-readable probe of what the embedding should preserve. - `trace`: notes and uncertainties. @@ -35,3 +42,25 @@ runtime/training artifact. The audit projection is not the true latent space. It is a review surface for humans, tests, and training data. + +## Stability Rule + +Training packets must not change vector dimensionality inside a schema version. +If Weksa does not know a value yet, it emits `0` in the reserved slot and records +the uncertainty in `trace.uncertainties`. Adding, removing, or reordering vector +slots requires a new schema version and an AquaSynth migration note. + +## Channel Split + +The text embedding and PanPhon sequence must stay separate. A rich +English text embedding can carry meaning, register, and local phrasing, but it +does not become meaningful just because the input string is IPA. IPA and alien +phonetic material belong in `inputs.panphon_sequence`. Pure +phonetic fixtures may zero `inputs.speech_text_embedding` or point it to a +source meaning packet, but they must not smuggle phone strings through the +semantic-text channel. + +The v0.1 phonetic channel uses PanPhon-style articulatory features as its +per-phone base representation. Weksa supplies that sequence evidence. AquaSynth +owns the learned compression from variable-length PanPhon sequence into the +256-float `phonetic_realization_embedding`. diff --git a/state/map.yaml b/state/map.yaml index f781674..ca6a36e 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -210,8 +210,9 @@ open_questions: - What English G2P path should become executable first: CMUdict lookup plus ARPABET-to-IPA conversion, eSpeak NG oracle comparison, or a trained OOV model? - - What dimensionality and training corpus should the first AquaSynth-owned - utterance embedding encoder use? + - What training corpus should the first AquaSynth-owned utterance embedding + encoder use now that the v0.1 handoff fixes Weksa inputs at 1024 + 32 + 64 + floats and AquaSynth output at 64 floats? landed_baseline: - Language-neutral state spine. @@ -226,6 +227,17 @@ landed_baseline: - Nibu lowering target note. - First English pronunciation plan fixture for Nibu proof output. - First toy utterance embedding handoff fixture for Nibu proof output. + - Weksa-to-AquaSynth utterance embedding handoff v0.1 fixes the training + vector contract: 1024-float `bge-m3:latest` speech text embedding, + variable-length PanPhon sequence evidence, a 256-float AquaSynth-owned + phonetic realization embedding, 32-float prosody/emphasis hints, 64-float + projected character state, and 64-float AquaSynth-owned utterance embedding + output. Toy fixtures are marked non-training if they do not use those + dimensions. + - Speech text embedding and phonetic realization are separate channels. + English text semantics may use `bge-m3`; IPA strings and alien phonetic + material belong in PanPhon sequence evidence that AquaSynth compresses into + the phonetic realization embedding. - First Nibu interlingua-to-English lowering fixture. - `bge-m3:latest` installed on neighboring Ollama box `192.168.1.84` and verified through a 1024-dimensional utterance embedding.