From cbfe9088f9ceb8dca5e218cc2137f023e395428c Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Tue, 19 May 2026 17:47:31 +0100 Subject: [PATCH] Document utterance embedding handoff --- docs/english-pronunciation-lowering.md | 2 +- docs/flavored-english-lowering.md | 9 +- docs/line-expression-vector.md | 124 ----------------- docs/roadmap.md | 4 +- docs/utterance-embedding.md | 129 ++++++++++++++++++ ...n-vector.yaml => utterance-embedding.yaml} | 40 ++++-- schemas/README.md | 2 +- schemas/line-expression-vector.md | 32 ----- schemas/utterance-embedding-handoff.md | 37 +++++ state/map.yaml | 17 ++- 10 files changed, 217 insertions(+), 179 deletions(-) delete mode 100644 docs/line-expression-vector.md create mode 100644 docs/utterance-embedding.md rename examples/english-lowering/nibu-proof-001/{expression-vector.yaml => utterance-embedding.yaml} (70%) delete mode 100644 schemas/line-expression-vector.md create mode 100644 schemas/utterance-embedding-handoff.md diff --git a/docs/english-pronunciation-lowering.md b/docs/english-pronunciation-lowering.md index 702e38b..aadfc0a 100644 --- a/docs/english-pronunciation-lowering.md +++ b/docs/english-pronunciation-lowering.md @@ -18,7 +18,7 @@ and dialogue layer: phrase grouping, emphasis, reduced forms, and word choice ar part of the utterance Weksa just created. The long-term primary handoff is the learned -[Line Expression Vector](./line-expression-vector.md). This pronunciation plan is +[Utterance Embedding Handoff](./utterance-embedding.md). This pronunciation plan is the inspectable audit/debug lane: useful for fixtures, phoneme strings, and review, but not the only representation of delivery. diff --git a/docs/flavored-english-lowering.md b/docs/flavored-english-lowering.md index 63e9bad..d395f9e 100644 --- a/docs/flavored-english-lowering.md +++ b/docs/flavored-english-lowering.md @@ -134,10 +134,11 @@ Weksa still owns the optional pronunciation plan for English output: tokenizing the line, choosing word pronunciations, preserving stress, and stringing broad IPA together. See [English Pronunciation Lowering](./english-pronunciation-lowering.md). -Weksa should also emit a learned line-expression vector when training data and -encoder support exist. That vector is the intended compact handoff for prosody, -emphasis, and character delivery pressure. See -[Line Expression Vector](./line-expression-vector.md). +Weksa should also emit the ingredients for an AquaSynth-owned utterance +embedding: speech text embedding, prosody/emphasis hints, and a projected +Ghostlight/Epiphany-shaped character-state vector. AquaSynth trains the tiny +encoder and consumes the resulting embedding in its synth driver. See +[Utterance Embedding Handoff](./utterance-embedding.md). ## First Target: Nibu diff --git a/docs/line-expression-vector.md b/docs/line-expression-vector.md deleted file mode 100644 index 978c96f..0000000 --- a/docs/line-expression-vector.md +++ /dev/null @@ -1,124 +0,0 @@ -# Line Expression Vector - -Version: `0.1-draft` - -The line expression vector is Weksa's learned handoff from flavored English -lowering to downstream voice control. - -```text -interlingua packet - -> projected agent-local context - -> flavored English line - -> Weksa line-expression encoder - -> line expression vector - -> AquaSynth synth-control decoder -``` - -The goal is two small learned artifacts in a tight loop: - -1. Weksa learns to encode a character's line as a compact expression vector. -2. AquaSynth learns to decode that vector into synth-control intent. - -No giant oracle gets to own the seam. No hidden theatrical mush. Just a line, -its meaning, its speaker context, and a learned vector with enough audit surface -to tell whether it is lying. - -## Authority - -Weksa owns: - -- interlingua packet -- projected agent-local context -- flavored English line -- line-expression vector generation -- audit projections for meaning, emphasis, and character pressure - -AquaSynth owns: - -- mapping expression vectors to synthesis controls -- consuming pronunciation or IPA plans when Weksa provides them -- phonetic feature interpretation -- timing, articulation, tract, and DSP -- audio evaluation - -## Vector Contract - -The vector is dense and model-owned, but the packet around it must be explicit. - -```yaml -schema_version: weksa.line_expression_vector.v0 -source_output_ref: -speaker_agent_id: -encoder: - model_id: - version: - training_set: -text: - spoken_text: - normalized_text: -vector: - dimensionality: - values_ref: -audit_projection: - semantic_preservation: - emphasis: - prosody: - character_pressure: - delivery_shape: -training_refs: - positive_examples: - negative_examples: -trace: - notes: - uncertainties: -``` - -`values_ref` may point to an external binary or JSON artifact once the vector is -large. Early fixtures may inline tiny toy vectors. - -## Audit Projection - -The audit projection is not the vector. It is a human-readable probe of what the -vector is supposed to contain. - -Useful audit families: - -- `semantic_preservation`: what source meaning must remain audible. -- `emphasis`: which words or spans carry prominence. -- `prosody`: clipped, rising, flat, hesitant, compressed, etc. -- `character_pressure`: active state pressures shaping delivery. -- `delivery_shape`: pace, pause pattern, intensity, warmth, contempt, tenderness, - threat, or play. - -These are labels for review and training. The learned vector is allowed to find -better internal dimensions, but it does not get to skip accountability. - -## Training Loop - -First loop: - -1. Author or generate an interlingua packet. -2. Project agent-local context. -3. Lower to English. -4. Human-review whether the line preserves meaning and voice. -5. Encode the reviewed line into a line-expression vector. -6. AquaSynth decodes the vector into synth-control intent. -7. Compare rendered speech against human notes and reference takes when present. -8. Feed corrections back into the Weksa encoder and AquaSynth decoder. - -The same fixture can carry: - -- accepted line -- rejected line -- why the rejected line failed -- audit projection -- vector artifact -- downstream AquaSynth evaluation pointer - -## Non-Goals - -- Do not make Weksa emit raw vocal-tract controls. -- Do not make AquaSynth infer character psychology from prose. -- Do not treat named audit axes as the true latent space. -- Do not make realtime neural voice output the authority. -- Do not hide bad semantic preservation behind pleasant delivery. diff --git a/docs/roadmap.md b/docs/roadmap.md index e7bcd7f..fb338ce 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -18,8 +18,8 @@ - Lower one packet through Nibu's projected agent state into flavored English. - Add one English pronunciation plan that strings the lowered line into broad IPA. -- Add one toy line-expression vector fixture, then replace hand-authored values - with a learned encoder. +- Add one toy utterance embedding handoff fixture, then replace hand-authored + values with the AquaSynth-owned learned encoder. - Preserve the first lowering proof as fixture data under `examples/english-lowering/`. - Parse or load one language project definition. diff --git a/docs/utterance-embedding.md b/docs/utterance-embedding.md new file mode 100644 index 0000000..6528b9d --- /dev/null +++ b/docs/utterance-embedding.md @@ -0,0 +1,129 @@ +# Utterance Embedding Handoff + +Version: `0.1-draft` + +The utterance embedding is the compact learned handoff from Weksa-authored line +intent to AquaSynth voice control. + +```text +interlingua packet + -> projected Ghostlight/Epiphany-shaped character context + -> flavored English line + -> speech text embedding + -> prosody and emphasis hints + -> character state vector + -> AquaSynth utterance embedding encoder + -> utterance embedding + -> AquaSynth synth driver +``` + +Weksa does not own the optimizer. Weksa owns the inspected ingredients that make +training possible: + +- source interlingua and lowering trace; +- spoken and normalized text; +- speech text embedding reference; +- prosody and emphasis hints; +- projected character-state vector; +- audit projection for semantic preservation and delivery pressure; +- training references and human review notes. + +AquaSynth owns: + +- the tiny learned utterance-embedding encoder; +- the shared packed-buffer gradient-descent/Adam primitive; +- the synth-driver model that consumes the utterance embedding; +- phonetic feature interpretation, timing, articulation, tract, and DSP; +- audio evaluation. + +The point is one learned speech handoff, not two little neural courthouses +arguing about who gets to wear the robe. + +## Packet Contract + +```yaml +schema_version: weksa.utterance_embedding_handoff.v0 +source_output_ref: +speaker_agent_id: +text: + spoken_text: + normalized_text: +inputs: + speech_text_embedding: + model_id: + dimensionality: + values_ref: + prosody_emphasis_hints: + dimensionality: + values: + character_state_vector: + source: + dimensionality: + values_ref: +utterance_embedding: + owner: AquaSynth + model_id: + version: + training_set: + dimensionality: + values_ref: +audit_projection: + semantic_preservation: + emphasis: + prosody: + character_pressure: + delivery_shape: +training_refs: + positive_examples: + negative_examples: +trace: + notes: + uncertainties: +``` + +Early fixtures may inline tiny toy vectors. Real embeddings should use +`values_ref` once the dimensions stop being decorative furniture. + +## Audit Projection + +The audit projection is not the embedding. It is a human-readable probe of what +the embedding should preserve. + +Useful audit families: + +- `semantic_preservation`: what source meaning must remain audible. +- `emphasis`: which words or spans carry prominence. +- `prosody`: clipped, rising, flat, hesitant, compressed, etc. +- `character_pressure`: active state pressures shaping delivery. +- `delivery_shape`: pace, pause pattern, intensity, warmth, contempt, tenderness, + threat, or play. + +These labels guide review and training. The learned embedding may find better +internal dimensions, but it does not get to skip accountability. + +## Training Loop + +First loop: + +1. Author or generate an interlingua packet. +2. Project Ghostlight/Epiphany-shaped character context. +3. Lower to English. +4. Generate or attach a speech text embedding. +5. Emit prosody/emphasis hints and a projected character state vector. +6. Human-review whether the line preserves meaning and voice. +7. AquaSynth trains the utterance embedding encoder. +8. AquaSynth feeds the utterance embedding to the synth driver. +9. Compare rendered speech against human notes and reference takes when present. + +The same fixture can carry accepted lines, rejected lines, audit projections, +input vectors, downstream AquaSynth evaluation pointers, and notes about why a +candidate failed. + +## Non-Goals + +- Do not make Weksa emit raw vocal-tract controls. +- Do not make Weksa train its own speech embedding optimizer. +- Do not make AquaSynth infer character psychology from prose alone. +- Do not treat named audit axes as the true latent space. +- Do not make realtime neural voice output the authority. +- Do not hide bad semantic preservation behind pleasant delivery. diff --git a/examples/english-lowering/nibu-proof-001/expression-vector.yaml b/examples/english-lowering/nibu-proof-001/utterance-embedding.yaml similarity index 70% rename from examples/english-lowering/nibu-proof-001/expression-vector.yaml rename to examples/english-lowering/nibu-proof-001/utterance-embedding.yaml index 406928d..a133573 100644 --- a/examples/english-lowering/nibu-proof-001/expression-vector.yaml +++ b/examples/english-lowering/nibu-proof-001/utterance-embedding.yaml @@ -1,14 +1,38 @@ -schema_version: weksa.line_expression_vector.v0 +schema_version: weksa.utterance_embedding_handoff.v0 source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml speaker_agent_id: nibu -encoder: - model_id: weksa-line-expression-toy - version: 0.0-hand-authored-fixture - training_set: none text: spoken_text: Take your hand off that control unless you want your nervous system to become my least useful circuit diagram. normalized_text: take your hand off that control unless you want your nervous system to become my least useful circuit diagram -vector: +inputs: + speech_text_embedding: + model_id: toy-text-embedding + dimensionality: 4 + values: + - 0.91 + - 0.22 + - 0.68 + - 0.44 + prosody_emphasis_hints: + dimensionality: 4 + values: + - 0.95 + - 0.82 + - 0.18 + - 0.74 + character_state_vector: + source: projected-nibu-proof-context + dimensionality: 4 + values: + - 0.88 + - 0.71 + - 0.24 + - 0.66 +utterance_embedding: + owner: AquaSynth + model_id: aquasynth-utterance-embedding-toy + version: 0.0-hand-authored-fixture + training_set: none dimensionality: 12 values: - 0.92 @@ -25,7 +49,7 @@ vector: - 0.57 notes: - Toy vector is hand-authored only to exercise the contract. - - Real vectors should come from a learned Weksa encoder. + - Real utterance embeddings should come from the learned AquaSynth encoder. audit_projection: semantic_preservation: must_preserve: @@ -59,7 +83,7 @@ training_refs: negative_examples: [] trace: notes: - - First fixture for learned line-expression vector contract. + - First fixture for learned utterance embedding handoff contract. - Values are placeholders, not calibrated latent dimensions. uncertainties: - Need real encoder dimensionality. diff --git a/schemas/README.md b/schemas/README.md index 60e7cfa..a6108f4 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -20,7 +20,7 @@ steer the machine, describe its authority here. for projecting interlingua through agent state into flavored English. - [english-pronunciation-plan.md](./english-pronunciation-plan.md): plan for stringing flavored English into IPA/phoneme sequences before downstream speech. -- [line-expression-vector.md](./line-expression-vector.md): learned vector +- [utterance-embedding-handoff.md](./utterance-embedding-handoff.md): learned utterance embedding handoff contract for handing prosody, emphasis, and character pressure to downstream synth control. diff --git a/schemas/line-expression-vector.md b/schemas/line-expression-vector.md deleted file mode 100644 index bbf7eaf..0000000 --- a/schemas/line-expression-vector.md +++ /dev/null @@ -1,32 +0,0 @@ -# Line Expression Vector Contract - -This is the human-facing schema receipt for Weksa line-expression vectors. - -The vector sits after flavored-English lowering and before AquaSynth synth -control. It is the learned representation of a character's line: meaning, -emphasis, prosody, and delivery pressure in one compact artifact. - -## Required Fields - -- `schema_version`: currently `weksa.line_expression_vector.v0`. -- `source_output_ref`: lowered English output reference. -- `speaker_agent_id`: speaker identity. -- `encoder`: model ID and version metadata. -- `text`: spoken and normalized text. -- `vector`: dimensionality and either inline values or `values_ref`. -- `audit_projection`: human-readable probe of what the vector is intended to - preserve. -- `trace`: notes and uncertainties. - -## Boundary - -The vector is not canonical character state. It is a derived runtime/training -artifact. - -The audit projection is not the true latent space. It is a review surface for -humans, tests, and training data. - -## Downstream - -AquaSynth may consume the vector through its own decoder. Weksa does not define -the synth-control schema here. diff --git a/schemas/utterance-embedding-handoff.md b/schemas/utterance-embedding-handoff.md new file mode 100644 index 0000000..b6b22c2 --- /dev/null +++ b/schemas/utterance-embedding-handoff.md @@ -0,0 +1,37 @@ +# Utterance Embedding Handoff Contract + +This is the human-facing schema receipt for Weksa-to-AquaSynth utterance +embedding handoff packets. + +The packet sits after flavored-English lowering and before AquaSynth speech +control. It carries the inspected inputs for a learned utterance embedding: +speech text embedding, prosody/emphasis hints, and a projected character-state +vector from the speaker's Ghostlight/Epiphany-shaped profile. + +## Required Fields + +- `schema_version`: currently `weksa.utterance_embedding_handoff.v0`. +- `source_output_ref`: lowered English output reference. +- `speaker_agent_id`: speaker identity. +- `text`: spoken and normalized text. +- `inputs.speech_text_embedding`: model metadata, dimensionality, and values or + `values_ref`. +- `inputs.prosody_emphasis_hints`: compact deterministic hints for emphasis, + contour, pace, intensity, and pause pressure. +- `inputs.character_state_vector`: projected speaker-state vector and source. +- `utterance_embedding`: AquaSynth-owned model metadata and produced vector + reference when available. +- `audit_projection`: human-readable probe of what the embedding should + preserve. +- `trace`: notes and uncertainties. + +## Boundary + +Weksa owns the packet ingredients and audit surface. AquaSynth owns the learned +encoder, gradient descent, synth-driver model, and audio evaluation. + +The utterance embedding is not canonical character state. It is a derived +runtime/training artifact. + +The audit projection is not the true latent space. It is a review surface for +humans, tests, and training data. diff --git a/state/map.yaml b/state/map.yaml index 9403148..f781674 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -41,8 +41,10 @@ constraints: work. Weksa owns interlingua-to-English projection. - Weksa still owns English pronunciation planning: tokenization, word pronunciations, phrase grouping, stress, and broad IPA stringing. - - Long-term speech handoff should use learned line-expression vectors from - Weksa and learned synth-control decoding in AquaSynth. + - Long-term speech handoff should use Weksa-provided speech text embeddings, + prosody/emphasis hints, and Ghostlight/Epiphany-shaped character-state + vectors; AquaSynth owns the learned utterance embedding encoder, gradient + descent, and synth-control decoding. - Large offline text-embedding jobs should use `bge-m3` on the neighboring Ollama inference box documented in `E:\Projects\gamecult-ops` rather than running on the local Weksa workspace by default. @@ -186,8 +188,9 @@ open_questions: morphology, and diachrony definitions? - What compiled runtime bundle format keeps game-time rendering fast and inspectable? - - How should Weksa persist or cache utterance embeddings generated by - `bge-m3` on the neighboring Ollama box? + - How should Weksa persist or cache speech text embeddings generated by + `bge-m3` on the neighboring Ollama box before AquaSynth turns them into + utterance embeddings? - How much deterministic validation belongs in v0? - What is the minimal Rust to Dust language-project slice that proves the engine without hard-coding the setting into it? @@ -207,8 +210,8 @@ open_questions: - What English G2P path should become executable first: CMUdict lookup plus ARPABET-to-IPA conversion, eSpeak NG oracle comparison, or a trained OOV model? - - What dimensionality and training corpus should the first Weksa - line-expression encoder use? + - What dimensionality and training corpus should the first AquaSynth-owned + utterance embedding encoder use? landed_baseline: - Language-neutral state spine. @@ -222,7 +225,7 @@ landed_baseline: - Draft flavored-English lowering contract. - Nibu lowering target note. - First English pronunciation plan fixture for Nibu proof output. - - First toy line-expression vector fixture for Nibu proof output. + - First toy utterance embedding handoff fixture for Nibu proof output. - First Nibu interlingua-to-English lowering fixture. - `bge-m3:latest` installed on neighboring Ollama box `192.168.1.84` and verified through a 1024-dimensional utterance embedding.