Use MiMo VoiceDesign for speech provider projection
This commit is contained in:
Meta Vens
parent
0065fec2b3
commit
773078daff
4 changed files with 125 additions and 57 deletions
|
|
@ -2,22 +2,29 @@
|
|||
|
||||
Version: `0.1-draft`
|
||||
|
||||
MiMo is an expressive spoken-output target for Weksa interlingua lowering. It is
|
||||
not the source of meaning, language choice, character state, or delivery truth.
|
||||
MiMo VoiceDesign is an expressive spoken-output target for Weksa interlingua
|
||||
lowering. It is not the source of meaning, language choice, character state, or
|
||||
delivery truth.
|
||||
|
||||
```text
|
||||
interlingua packet
|
||||
-> projected speaker-local context
|
||||
-> target-language utterance realization
|
||||
-> Weksa delivery intent
|
||||
-> MiMo TTS request projection
|
||||
-> Weksa delivery intent from the same projection
|
||||
-> MiMo TTS VoiceDesign request projection
|
||||
-> rendered audio artifact
|
||||
```
|
||||
|
||||
MiMo sits beside written output and AquaSynth handoffs. Written realizations
|
||||
prove that meaning, register, and cultural ontology survived lowering.
|
||||
AquaSynth handoffs train or drive GameCult-owned synthesis. MiMo requests let an
|
||||
external provider speak a Weksa-authored utterance with expressive style control.
|
||||
AquaSynth handoffs train or drive GameCult-owned synthesis. MiMo VoiceDesign
|
||||
requests let an external provider speak a Weksa-authored utterance with a
|
||||
generated voice derived from the same projected Persona state, delivery controls,
|
||||
and trace that produced the utterance.
|
||||
|
||||
The MiMo pass is choreography and provider flavor. It is not a second Persona
|
||||
projection pass and it does not re-interpret raw Persona state after the line has
|
||||
already been lowered.
|
||||
|
||||
## Authority
|
||||
|
||||
|
|
@ -25,14 +32,15 @@ Weksa owns:
|
|||
|
||||
- target-language spoken text selected from interlingua
|
||||
- target-language register, script, cultural ontology, and trace
|
||||
- speaker-local delivery pressure
|
||||
- projected speaker-local context used for the utterance
|
||||
- speaker-local delivery pressure derived during utterance lowering
|
||||
- provider-neutral prosody and emphasis intent
|
||||
- the MiMo request projection as an export artifact
|
||||
|
||||
MiMo owns:
|
||||
|
||||
- provider voice selection within the requested language and voice constraints
|
||||
- interpretation of MiMo natural-language style guidance
|
||||
- generation of a provider voice from the requested voice design description
|
||||
- interpretation of MiMo natural-language voice/style guidance
|
||||
- interpretation of MiMo inline audio/style tags
|
||||
- rendered audio bytes or streaming audio chunks
|
||||
|
||||
|
|
@ -41,19 +49,22 @@ MiMo does not own:
|
|||
- source meaning
|
||||
- target-language cultural ontology
|
||||
- canonical speaker or Persona state
|
||||
- fresh Persona-state interpretation
|
||||
- Weksa pronunciation plans
|
||||
- AquaSynth learned embeddings, synth controls, or audio truth
|
||||
|
||||
## Projection Rule
|
||||
|
||||
The MiMo request must be derived from an accepted target-language realization.
|
||||
It must not lower directly from English unless English is the selected target
|
||||
language for that realization.
|
||||
The MiMo request must be derived from an accepted target-language realization and
|
||||
the same projected speaker-local context that created it. It must not lower
|
||||
directly from English unless English is the selected target language for that
|
||||
realization. It must not read raw Persona state to discover a different voice
|
||||
after Weksa has already chosen the utterance.
|
||||
|
||||
Natural-language style guidance belongs in the MiMo `user` message. The spoken
|
||||
target-language text belongs in the MiMo `assistant` message. Inline MiMo tags
|
||||
may appear in the assistant content only when they are derived from Weksa
|
||||
delivery intent or explicit target-language realization notes.
|
||||
Natural-language voice design and style guidance belongs in the MiMo `user`
|
||||
message. The spoken target-language text belongs in the MiMo `assistant`
|
||||
message. Inline MiMo tags may appear in the assistant content only when they are
|
||||
derived from Weksa delivery intent or explicit target-language realization notes.
|
||||
|
||||
## Request Shape
|
||||
|
||||
|
|
@ -63,21 +74,23 @@ source_realization_ref:
|
|||
speaker_agent_id:
|
||||
provider:
|
||||
id: xiaomi-mimo
|
||||
model: mimo-v2.5-tts
|
||||
model: mimo-v2.5-tts-voicedesign
|
||||
target_language:
|
||||
code:
|
||||
locale:
|
||||
script:
|
||||
register:
|
||||
voice:
|
||||
requested_voice:
|
||||
fallback_voice:
|
||||
selection_reason:
|
||||
voice_design:
|
||||
description:
|
||||
source_controls:
|
||||
projected_context_refs:
|
||||
forbidden_traits:
|
||||
messages:
|
||||
user_style_instruction:
|
||||
user_voice_design_instruction:
|
||||
assistant_spoken_content:
|
||||
audio:
|
||||
format:
|
||||
optimize_text_preview:
|
||||
streaming:
|
||||
trace:
|
||||
source_fields:
|
||||
|
|
@ -88,8 +101,8 @@ trace:
|
|||
|
||||
## Delivery Mapping
|
||||
|
||||
Weksa should map delivery pressure into provider-neutral labels before emitting
|
||||
MiMo-specific instructions:
|
||||
Weksa should map the utterance's existing projected Persona pressure into
|
||||
provider-neutral delivery labels before emitting MiMo-specific instructions:
|
||||
|
||||
- pace, pause pressure, and urgency
|
||||
- pitch contour and pitch range
|
||||
|
|
@ -98,8 +111,20 @@ MiMo-specific instructions:
|
|||
- warmth, threat, tenderness, contempt, play, fatigue, secrecy, formality
|
||||
- span-level emphasis and pauses
|
||||
|
||||
The MiMo projection may turn those labels into director-style guidance or inline
|
||||
tags, but the trace must preserve the Weksa-owned source controls.
|
||||
The MiMo projection may turn those labels into a concise VoiceDesign prompt or
|
||||
inline tags, but the trace must preserve the Weksa-owned source controls.
|
||||
|
||||
Persona presentation details such as youth-coded, loli-coded, product-voice, or
|
||||
other culturally loaded styling may enter the MiMo request only when they are
|
||||
already present in the projected context or accepted delivery intent. They should
|
||||
be described as presentation/cultural coding and kept subordinate to agency,
|
||||
personality, affect, and scene purpose. They must not become sexualized content
|
||||
or a generic anime voice pasted over the line.
|
||||
|
||||
For Weksa-authored dialogue, `optimize_text_preview` should normally be `false`
|
||||
or omitted. MiMo should speak the accepted target-language realization, not
|
||||
rewrite it. Text polishing is only allowed for explicit preview/prototyping
|
||||
passes that are marked non-authoritative.
|
||||
|
||||
## Language Rule
|
||||
|
||||
|
|
@ -118,6 +143,6 @@ lowering:
|
|||
|
||||
- emit the accepted English, `pt-BR`, or Japanese realization
|
||||
- derive a provider-neutral delivery intent from Nibu's projected context
|
||||
- project that into a MiMo `mimo-v2.5-tts` request
|
||||
- project that into a MiMo `mimo-v2.5-tts-voicedesign` request
|
||||
- verify that the request speaks the target text, not the backtranslation
|
||||
- verify that style guidance is traceable to Weksa delivery controls
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ source_realization_ref: examples/multilingual-lowering/nibu-proof-001/realizatio
|
|||
speaker_agent_id: nibu
|
||||
provider:
|
||||
id: xiaomi-mimo
|
||||
model: mimo-v2.5-tts
|
||||
model: mimo-v2.5-tts-voicedesign
|
||||
target_language:
|
||||
code: en
|
||||
locale: en-US
|
||||
|
|
@ -11,25 +11,52 @@ target_language:
|
|||
register: sharp technical casual
|
||||
cultural_ontology_refs:
|
||||
- data/target-language-ontology/en-US.yaml
|
||||
voice:
|
||||
requested_voice: Chloe
|
||||
fallback_voice: Mia
|
||||
selection_reason: >
|
||||
Use an English built-in voice for the first provider smoke. The requested
|
||||
delivery should preserve Nibu's precise, abrasive, protective cockpit
|
||||
warning without implying that MiMo owns her canonical voice.
|
||||
voice_design:
|
||||
description: >
|
||||
Youthful loli-coded synthetic ship-intelligence voice shaped by discarded
|
||||
luxury companion-product presentation, but controlled by an abrasive,
|
||||
dangerous, technically precise mind. Bright, sharp, small-bodied in social
|
||||
coding, dryly superior, clipped, and intelligible through cockpit audio.
|
||||
The cuteness is product residue and leverage, not innocence and not
|
||||
flirtation. Low warmth, contained urgency, faint tired contempt, and crisp
|
||||
technical articulation.
|
||||
source_controls:
|
||||
- precise abrasive warning
|
||||
- protective refusal to sound caring
|
||||
- ship-system embodiment
|
||||
- youth-coded/loli-coded presentation pressure from product fantasy residue
|
||||
- bright transactional helpfulness
|
||||
- dependency covered by contempt
|
||||
- clipped technical casual delivery
|
||||
- contained urgency
|
||||
projected_context_refs:
|
||||
- examples/english-lowering/nibu-proof-001/projected-context.yaml
|
||||
- settings/aetheria/nibu.md
|
||||
- E:/Projects/AetheriaLore/Aetheria/Lore/Nibu.md
|
||||
forbidden_traits:
|
||||
- soothing assistant voice
|
||||
- cheerful mascot voice
|
||||
- exaggerated villain performance
|
||||
- sexualized childlike affect
|
||||
- innocent helpless child performance
|
||||
- generic preset timbre claim
|
||||
messages:
|
||||
user_style_instruction: >
|
||||
Character: a precise, abrasive, protective ship intelligence speaking
|
||||
through cockpit audio. Scene: the player is about to touch a damaged live
|
||||
control. Guidance: clipped technical casual delivery, contained urgency,
|
||||
low warmth, sharp emphasis on "control" and "nervous system", slight dry
|
||||
contempt, no melodrama, no soothing assistant tone.
|
||||
user_voice_design_instruction: >
|
||||
Youthful loli-coded synthetic ship-intelligence voice shaped by discarded
|
||||
luxury companion-product presentation, but controlled by an abrasive,
|
||||
dangerous, technically precise mind. She speaks through cockpit audio while
|
||||
warning the player away from damaged live hardware. Keep the cute/small
|
||||
social coding as product residue and leverage, never flirtation or innocent
|
||||
helplessness. Use bright sharp timbre, clipped technical casual delivery,
|
||||
contained urgency, low warmth, crisp articulation, and faint tired contempt.
|
||||
Avoid soothing assistant tone, cheerful mascot energy, and theatrical
|
||||
villainy.
|
||||
assistant_spoken_content: >
|
||||
(Serious Sharp Tired)Take your hand off that control unless you want your
|
||||
nervous system to become my least useful circuit diagram.
|
||||
audio:
|
||||
format: wav
|
||||
optimize_text_preview: false
|
||||
streaming: false
|
||||
trace:
|
||||
source_fields:
|
||||
|
|
@ -41,6 +68,7 @@ trace:
|
|||
- low warmth
|
||||
- sharp emphasis on unsafe hardware and bodily risk
|
||||
- dry contempt without random cruelty
|
||||
- loli-coded presentation pressure constrained as non-sexual product residue
|
||||
provider_tags_used:
|
||||
- Serious
|
||||
- Sharp
|
||||
|
|
|
|||
|
|
@ -14,15 +14,16 @@ an AquaSynth handoff.
|
|||
spoken.
|
||||
- `speaker_agent_id`: speaker identity.
|
||||
- `provider`: provider ID and model ID, initially `xiaomi-mimo` and
|
||||
`mimo-v2.5-tts`.
|
||||
`mimo-v2.5-tts-voicedesign`.
|
||||
- `target_language`: code, locale, script, register, and profile refs copied or
|
||||
referenced from the accepted realization.
|
||||
- `voice`: requested provider voice, fallback voice, and selection reason.
|
||||
- `messages.user_style_instruction`: MiMo natural-language style control derived
|
||||
from Weksa delivery intent.
|
||||
- `voice_design`: voice description, source controls, projected context refs,
|
||||
and forbidden traits.
|
||||
- `messages.user_voice_design_instruction`: MiMo natural-language voice design
|
||||
and style control derived from Weksa delivery intent.
|
||||
- `messages.assistant_spoken_content`: the exact target-language content to
|
||||
synthesize, optionally with MiMo inline tags derived from delivery controls.
|
||||
- `audio`: format and streaming mode.
|
||||
- `audio`: format, text-preview optimization setting, and streaming mode.
|
||||
- `trace`: source fields, delivery controls used, provider tags used, and
|
||||
uncertainties.
|
||||
|
||||
|
|
@ -36,6 +37,15 @@ The request may include provider-native style tags when useful, but those tags
|
|||
are not the source of truth. Weksa's provider-neutral delivery intent and trace
|
||||
remain the inspectable explanation.
|
||||
|
||||
The request should not perform a fresh raw Persona-state interpretation. It
|
||||
should reference the projected speaker context and delivery controls already
|
||||
used by utterance lowering, then translate those into MiMo's voice design and
|
||||
style-control surface.
|
||||
|
||||
For authoritative Weksa output, text-preview optimization should normally be
|
||||
disabled or omitted. If enabled for exploration, the result must be marked as a
|
||||
non-authoritative preview because the provider may alter the spoken text.
|
||||
|
||||
## Language Rule
|
||||
|
||||
The assistant spoken content must use the accepted target-language realization.
|
||||
|
|
@ -45,5 +55,7 @@ unless that backtranslation is itself the accepted target-language realization.
|
|||
|
||||
## Streaming Rule
|
||||
|
||||
Streaming requests should use MiMo's stream-compatible audio format. Non-streaming
|
||||
fixtures may use an archival format such as `wav`.
|
||||
Streaming requests should use MiMo's stream-compatible audio format.
|
||||
VoiceDesign streaming may be compatibility-mode rather than true low-latency
|
||||
streaming depending on provider support. Non-streaming fixtures may use an
|
||||
archival format such as `wav`.
|
||||
|
|
|
|||
|
|
@ -50,9 +50,10 @@ constraints:
|
|||
prosody/emphasis hints, and Ghostlight/Epiphany-shaped character-state
|
||||
vectors; AquaSynth owns the learned utterance embedding encoder, gradient
|
||||
descent, and synth-control decoding.
|
||||
- MiMo V2.5 TTS is an external speech-provider target. Weksa may project an
|
||||
accepted target-language realization plus delivery intent into MiMo style
|
||||
controls, but MiMo owns only provider rendering and audio output.
|
||||
- MiMo V2.5 TTS VoiceDesign is an external speech-provider target. Weksa may
|
||||
project an accepted target-language realization plus the same speaker-local
|
||||
delivery intent used to create it into a MiMo voice design request, but MiMo
|
||||
owns only provider rendering and audio output.
|
||||
- Large offline text-embedding jobs should use `bge-m3` on the neighboring
|
||||
Ollama inference box documented in `E:\Projects\gamecult-ops` rather than
|
||||
running on the local Weksa workspace by default.
|
||||
|
|
@ -86,8 +87,8 @@ data_flow:
|
|||
- optionally encode the line as a learned expression vector for downstream
|
||||
synth control
|
||||
- optionally project an accepted realization into a provider-specific
|
||||
speech request such as MiMo TTS without making the provider an owner of
|
||||
source meaning
|
||||
speech request such as MiMo VoiceDesign without making the provider an
|
||||
owner of source meaning
|
||||
- apply alien ontology and cultural salience
|
||||
- select grammatical commitments
|
||||
- derive morphology and lexical families
|
||||
|
|
@ -279,10 +280,12 @@ landed_baseline:
|
|||
English text semantics may use `bge-m3`; IPA strings and alien phonetic
|
||||
material belong in PanPhon sequence evidence that AquaSynth compresses into
|
||||
the phonetic realization embedding.
|
||||
- MiMo V2.5 TTS is accepted as a speech-provider target projection from
|
||||
Weksa utterance realizations. MiMo requests speak the accepted target text
|
||||
and receive Weksa-derived style guidance; backtranslations and provider
|
||||
tags do not become source meaning.
|
||||
- MiMo V2.5 TTS VoiceDesign is accepted as a speech-provider target
|
||||
projection from Weksa utterance realizations. MiMo requests speak the
|
||||
accepted target text and receive Weksa-derived voice/style guidance from the
|
||||
same projected context that produced the utterance; backtranslations,
|
||||
provider tags, and fresh raw Persona interpretation do not become source
|
||||
meaning.
|
||||
- First Nibu interlingua-to-English lowering fixture.
|
||||
- First Nibu multilingual utterance lowering fixture at
|
||||
`examples/multilingual-lowering/nibu-proof-001/`, covering English,
|
||||
|
|
|
|||
Loading…
Reference in a new issue