GameCult/Weksa
Watch
9
0
Fork
You've already forked Weksa
 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

Document utterance embedding handoff

Browse source
This commit is contained in:
Meta Vens 2026-05-19 17:47:31 +01:00
commit cbfe9088f9
10 changed files with 217 additions and 179 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/english-pronunciation-lowering.md
View file

@ -18,7 +18,7 @@ and dialogue layer: phrase grouping, emphasis, reduced forms, and word choice ar
part of the utterance Weksa just created.
The long-term primary handoff is the learned
[Line Expression Vector](./line-expression-vector.md). This pronunciation plan is
[Utterance Embedding Handoff](./utterance-embedding.md). This pronunciation plan is
the inspectable audit/debug lane: useful for fixtures, phoneme strings, and
review, but not the only representation of delivery.

9
docs/flavored-english-lowering.md
View file

@ -134,10 +134,11 @@ Weksa still owns the optional pronunciation plan for English output: tokenizing
the line, choosing word pronunciations, preserving stress, and stringing broad
IPA together. See [English Pronunciation Lowering](./english-pronunciation-lowering.md).
Weksa should also emit a learned line-expression vector when training data and
encoder support exist. That vector is the intended compact handoff for prosody,
emphasis, and character delivery pressure. See
[Line Expression Vector](./line-expression-vector.md).
Weksa should also emit the ingredients for an AquaSynth-owned utterance
embedding: speech text embedding, prosody/emphasis hints, and a projected
Ghostlight/Epiphany-shaped character-state vector. AquaSynth trains the tiny
encoder and consumes the resulting embedding in its synth driver. See
[Utterance Embedding Handoff](./utterance-embedding.md).
## First Target: Nibu

124
docs/line-expression-vector.md
View file

@ -1,124 +0,0 @@
# Line Expression Vector
Version: `0.1-draft`
The line expression vector is Weksa's learned handoff from flavored English
lowering to downstream voice control.
```text
interlingua packet
-> projected agent-local context
-> flavored English line
-> Weksa line-expression encoder
-> line expression vector
-> AquaSynth synth-control decoder
```
The goal is two small learned artifacts in a tight loop:
1. Weksa learns to encode a character's line as a compact expression vector.
2. AquaSynth learns to decode that vector into synth-control intent.
No giant oracle gets to own the seam. No hidden theatrical mush. Just a line,
its meaning, its speaker context, and a learned vector with enough audit surface
to tell whether it is lying.
## Authority
Weksa owns:
- interlingua packet
- projected agent-local context
- flavored English line
- line-expression vector generation
- audit projections for meaning, emphasis, and character pressure
AquaSynth owns:
- mapping expression vectors to synthesis controls
- consuming pronunciation or IPA plans when Weksa provides them
- phonetic feature interpretation
- timing, articulation, tract, and DSP
- audio evaluation
## Vector Contract
The vector is dense and model-owned, but the packet around it must be explicit.
```yaml
schema_version: weksa.line_expression_vector.v0
source_output_ref:
speaker_agent_id:
encoder:
model_id:
version:
training_set:
text:
spoken_text:
normalized_text:
vector:
dimensionality:
values_ref:
audit_projection:
semantic_preservation:
emphasis:
prosody:
character_pressure:
delivery_shape:
training_refs:
positive_examples:
negative_examples:
trace:
notes:
uncertainties:
```
`values_ref` may point to an external binary or JSON artifact once the vector is
large. Early fixtures may inline tiny toy vectors.
## Audit Projection
The audit projection is not the vector. It is a human-readable probe of what the
vector is supposed to contain.
Useful audit families:
- `semantic_preservation`: what source meaning must remain audible.
- `emphasis`: which words or spans carry prominence.
- `prosody`: clipped, rising, flat, hesitant, compressed, etc.
- `character_pressure`: active state pressures shaping delivery.
- `delivery_shape`: pace, pause pattern, intensity, warmth, contempt, tenderness,
threat, or play.
These are labels for review and training. The learned vector is allowed to find
better internal dimensions, but it does not get to skip accountability.
## Training Loop
First loop:
1. Author or generate an interlingua packet.
2. Project agent-local context.
3. Lower to English.
4. Human-review whether the line preserves meaning and voice.
5. Encode the reviewed line into a line-expression vector.
6. AquaSynth decodes the vector into synth-control intent.
7. Compare rendered speech against human notes and reference takes when present.
8. Feed corrections back into the Weksa encoder and AquaSynth decoder.
The same fixture can carry:
- accepted line
- rejected line
- why the rejected line failed
- audit projection
- vector artifact
- downstream AquaSynth evaluation pointer
## Non-Goals
- Do not make Weksa emit raw vocal-tract controls.
- Do not make AquaSynth infer character psychology from prose.
- Do not treat named audit axes as the true latent space.
- Do not make realtime neural voice output the authority.
- Do not hide bad semantic preservation behind pleasant delivery.

4
docs/roadmap.md
View file

@ -18,8 +18,8 @@
- Lower one packet through Nibu's projected agent state into flavored English.
- Add one English pronunciation plan that strings the lowered line into broad
IPA.
- Add one toy line-expression vector fixture, then replace hand-authored values
with a learned encoder.
- Add one toy utterance embedding handoff fixture, then replace hand-authored
values with the AquaSynth-owned learned encoder.
- Preserve the first lowering proof as fixture data under
`examples/english-lowering/`.
- Parse or load one language project definition.

129
docs/utterance-embedding.md Normal file
View file

@ -0,0 +1,129 @@
# Utterance Embedding Handoff
Version: `0.1-draft`
The utterance embedding is the compact learned handoff from Weksa-authored line
intent to AquaSynth voice control.
```text
interlingua packet
-> projected Ghostlight/Epiphany-shaped character context
-> flavored English line
-> speech text embedding
-> prosody and emphasis hints
-> character state vector
-> AquaSynth utterance embedding encoder
-> utterance embedding
-> AquaSynth synth driver
```
Weksa does not own the optimizer. Weksa owns the inspected ingredients that make
training possible:
- source interlingua and lowering trace;
- spoken and normalized text;
- speech text embedding reference;
- prosody and emphasis hints;
- projected character-state vector;
- audit projection for semantic preservation and delivery pressure;
- training references and human review notes.
AquaSynth owns:
- the tiny learned utterance-embedding encoder;
- the shared packed-buffer gradient-descent/Adam primitive;
- the synth-driver model that consumes the utterance embedding;
- phonetic feature interpretation, timing, articulation, tract, and DSP;
- audio evaluation.
The point is one learned speech handoff, not two little neural courthouses
arguing about who gets to wear the robe.
## Packet Contract
```yaml
schema_version: weksa.utterance_embedding_handoff.v0
source_output_ref:
speaker_agent_id:
text:
spoken_text:
normalized_text:
inputs:
speech_text_embedding:
model_id:
dimensionality:
values_ref:
prosody_emphasis_hints:
dimensionality:
values:
character_state_vector:
source:
dimensionality:
values_ref:
utterance_embedding:
owner: AquaSynth
model_id:
version:
training_set:
dimensionality:
values_ref:
audit_projection:
semantic_preservation:
emphasis:
prosody:
character_pressure:
delivery_shape:
training_refs:
positive_examples:
negative_examples:
trace:
notes:
uncertainties:
```
Early fixtures may inline tiny toy vectors. Real embeddings should use
`values_ref` once the dimensions stop being decorative furniture.
## Audit Projection
The audit projection is not the embedding. It is a human-readable probe of what
the embedding should preserve.
Useful audit families:
- `semantic_preservation`: what source meaning must remain audible.
- `emphasis`: which words or spans carry prominence.
- `prosody`: clipped, rising, flat, hesitant, compressed, etc.
- `character_pressure`: active state pressures shaping delivery.
- `delivery_shape`: pace, pause pattern, intensity, warmth, contempt, tenderness,
threat, or play.
These labels guide review and training. The learned embedding may find better
internal dimensions, but it does not get to skip accountability.
## Training Loop
First loop:
1. Author or generate an interlingua packet.
2. Project Ghostlight/Epiphany-shaped character context.
3. Lower to English.
4. Generate or attach a speech text embedding.
5. Emit prosody/emphasis hints and a projected character state vector.
6. Human-review whether the line preserves meaning and voice.
7. AquaSynth trains the utterance embedding encoder.
8. AquaSynth feeds the utterance embedding to the synth driver.
9. Compare rendered speech against human notes and reference takes when present.
The same fixture can carry accepted lines, rejected lines, audit projections,
input vectors, downstream AquaSynth evaluation pointers, and notes about why a
candidate failed.
## Non-Goals
- Do not make Weksa emit raw vocal-tract controls.
- Do not make Weksa train its own speech embedding optimizer.
- Do not make AquaSynth infer character psychology from prose alone.
- Do not treat named audit axes as the true latent space.
- Do not make realtime neural voice output the authority.
- Do not hide bad semantic preservation behind pleasant delivery.

40
examples/english-lowering/nibu-proof-001/expression-vector.yaml → examples/english-lowering/nibu-proof-001/utterance-embedding.yaml
View file

@ -1,14 +1,38 @@
schema_version: weksa.line_expression_vector.v0
schema_version: weksa.utterance_embedding_handoff.v0
source_output_ref: examples/english-lowering/nibu-proof-001/lowered-output.yaml
speaker_agent_id: nibu
encoder:
model_id: weksa-line-expression-toy
version: 0.0-hand-authored-fixture
training_set: none
text:
spoken_text: Take your hand off that control unless you want your nervous system to become my least useful circuit diagram.
normalized_text: take your hand off that control unless you want your nervous system to become my least useful circuit diagram
vector:
inputs:
speech_text_embedding:
model_id: toy-text-embedding
dimensionality: 4
values:
- 0.91
- 0.22
- 0.68
- 0.44
prosody_emphasis_hints:
dimensionality: 4
values:
- 0.95
- 0.82
- 0.18
- 0.74
character_state_vector:
source: projected-nibu-proof-context
dimensionality: 4
values:
- 0.88
- 0.71
- 0.24
- 0.66
utterance_embedding:
owner: AquaSynth
model_id: aquasynth-utterance-embedding-toy
version: 0.0-hand-authored-fixture
training_set: none
dimensionality: 12
values:
- 0.92
@ -25,7 +49,7 @@ vector:
- 0.57
notes:
- Toy vector is hand-authored only to exercise the contract.
- Real vectors should come from a learned Weksa encoder.
- Real utterance embeddings should come from the learned AquaSynth encoder.
audit_projection:
semantic_preservation:
must_preserve:
@ -59,7 +83,7 @@ training_refs:
negative_examples: []
trace:
notes:
- First fixture for learned line-expression vector contract.
- First fixture for learned utterance embedding handoff contract.
- Values are placeholders, not calibrated latent dimensions.
uncertainties:
- Need real encoder dimensionality.

2
schemas/README.md
View file

@ -20,7 +20,7 @@ steer the machine, describe its authority here.
for projecting interlingua through agent state into flavored English.
- [english-pronunciation-plan.md](./english-pronunciation-plan.md): plan for
stringing flavored English into IPA/phoneme sequences before downstream speech.
- [line-expression-vector.md](./line-expression-vector.md): learned vector
- [utterance-embedding-handoff.md](./utterance-embedding-handoff.md): learned utterance embedding handoff
contract for handing prosody, emphasis, and character pressure to downstream
synth control.

32
schemas/line-expression-vector.md
View file

@ -1,32 +0,0 @@
# Line Expression Vector Contract
This is the human-facing schema receipt for Weksa line-expression vectors.
The vector sits after flavored-English lowering and before AquaSynth synth
control. It is the learned representation of a character's line: meaning,
emphasis, prosody, and delivery pressure in one compact artifact.
## Required Fields
- `schema_version`: currently `weksa.line_expression_vector.v0`.
- `source_output_ref`: lowered English output reference.
- `speaker_agent_id`: speaker identity.
- `encoder`: model ID and version metadata.
- `text`: spoken and normalized text.
- `vector`: dimensionality and either inline values or `values_ref`.
- `audit_projection`: human-readable probe of what the vector is intended to
preserve.
- `trace`: notes and uncertainties.
## Boundary
The vector is not canonical character state. It is a derived runtime/training
artifact.
The audit projection is not the true latent space. It is a review surface for
humans, tests, and training data.
## Downstream
AquaSynth may consume the vector through its own decoder. Weksa does not define
the synth-control schema here.

37
schemas/utterance-embedding-handoff.md Normal file
View file

@ -0,0 +1,37 @@
# Utterance Embedding Handoff Contract
This is the human-facing schema receipt for Weksa-to-AquaSynth utterance
embedding handoff packets.
The packet sits after flavored-English lowering and before AquaSynth speech
control. It carries the inspected inputs for a learned utterance embedding:
speech text embedding, prosody/emphasis hints, and a projected character-state
vector from the speaker's Ghostlight/Epiphany-shaped profile.
## Required Fields
- `schema_version`: currently `weksa.utterance_embedding_handoff.v0`.
- `source_output_ref`: lowered English output reference.
- `speaker_agent_id`: speaker identity.
- `text`: spoken and normalized text.
- `inputs.speech_text_embedding`: model metadata, dimensionality, and values or
`values_ref`.
- `inputs.prosody_emphasis_hints`: compact deterministic hints for emphasis,
contour, pace, intensity, and pause pressure.
- `inputs.character_state_vector`: projected speaker-state vector and source.
- `utterance_embedding`: AquaSynth-owned model metadata and produced vector
reference when available.
- `audit_projection`: human-readable probe of what the embedding should
preserve.
- `trace`: notes and uncertainties.
## Boundary
Weksa owns the packet ingredients and audit surface. AquaSynth owns the learned
encoder, gradient descent, synth-driver model, and audio evaluation.
The utterance embedding is not canonical character state. It is a derived
runtime/training artifact.
The audit projection is not the true latent space. It is a review surface for
humans, tests, and training data.

17
state/map.yaml
View file

@ -41,8 +41,10 @@ constraints:
work. Weksa owns interlingua-to-English projection.
- Weksa still owns English pronunciation planning: tokenization, word
pronunciations, phrase grouping, stress, and broad IPA stringing.
- Long-term speech handoff should use learned line-expression vectors from
Weksa and learned synth-control decoding in AquaSynth.
- Long-term speech handoff should use Weksa-provided speech text embeddings,
prosody/emphasis hints, and Ghostlight/Epiphany-shaped character-state
vectors; AquaSynth owns the learned utterance embedding encoder, gradient
descent, and synth-control decoding.
- Large offline text-embedding jobs should use `bge-m3` on the neighboring
Ollama inference box documented in `E:\Projects\gamecult-ops` rather than
running on the local Weksa workspace by default.
@ -186,8 +188,9 @@ open_questions:
morphology, and diachrony definitions?
- What compiled runtime bundle format keeps game-time rendering fast and
inspectable?
- How should Weksa persist or cache utterance embeddings generated by
`bge-m3` on the neighboring Ollama box?
- How should Weksa persist or cache speech text embeddings generated by
`bge-m3` on the neighboring Ollama box before AquaSynth turns them into
utterance embeddings?
- How much deterministic validation belongs in v0?
- What is the minimal Rust to Dust language-project slice that proves the
engine without hard-coding the setting into it?
@ -207,8 +210,8 @@ open_questions:
- What English G2P path should become executable first: CMUdict lookup plus
ARPABET-to-IPA conversion, eSpeak NG oracle comparison, or a trained OOV
model?
- What dimensionality and training corpus should the first Weksa
line-expression encoder use?
- What dimensionality and training corpus should the first AquaSynth-owned
utterance embedding encoder use?
landed_baseline:
- Language-neutral state spine.
@ -222,7 +225,7 @@ landed_baseline:
- Draft flavored-English lowering contract.
- Nibu lowering target note.
- First English pronunciation plan fixture for Nibu proof output.
- First toy line-expression vector fixture for Nibu proof output.
- First toy utterance embedding handoff fixture for Nibu proof output.
- First Nibu interlingua-to-English lowering fixture.
- `bge-m3:latest` installed on neighboring Ollama box `192.168.1.84` and
verified through a 1024-dimensional utterance embedding.