diff --git a/docs/ipa-vocal-tract-roadmap.md b/docs/ipa-vocal-tract-roadmap.md index f49ef98..4795901 100644 --- a/docs/ipa-vocal-tract-roadmap.md +++ b/docs/ipa-vocal-tract-roadmap.md @@ -526,6 +526,33 @@ spent, what latency budget it was meant to respect, and how confident the mapping was. A tiny audible witness that sounds right but arrives from a clockless swamp is already teaching the model to lie with a lovely voice. +## CultMesh Render/Scoring Work + +CultMesh is the transport, admission, and worker-orchestration layer for speech +render/scoring work. It is not a magical distributed trainer where every +compatible device mutates model weights directly. The authority trainer owns +the active `SpeechTrainingCheckpoint`; workers may produce gradients and witness +receipts, but they do not apply optimizer steps or commit new checkpoints. + +The clean path is: + +1. AquaSynth emits typed `SpeechRenderRequest` documents. +2. CultMesh transfers the required worker payloads: compiled Faust renderer, + scoring code, small model or encoder assembly, batch/artifact references. +3. Compatible runtimes admit the payload locally under a worker lease. +4. Workers render and score examples, then return `SpeechRenderResult` + documents with loss, gradients, artifacts, and timing receipts. +5. The authority trainer aggregates/applies gradients through + `SpeechBackpropagationPipeline`. +6. New checkpoints are committed through normal CultCache/CultMesh shard + authority. + +The missing layer is therefore payload transport, runtime admission, worker +leases, and result collection. The first target is to distribute compiled Faust +render/scoring jobs, collect gradients/results, and apply them centrally or by +one shard-primary trainer. Federated averaging can be explored later only after +the simple authority path is boring. + ## Optional eSpeak NG Workout The first IPA-adjacent parity lane uses eSpeak NG as a development reference. diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 26571af..192960e 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -125,3 +125,4 @@ {"date":"2026-05-20","topic":"weksa-utterance-embedding-contract","lesson":"Do not start serious AquaSynth speech gradient descent from toy Weksa vectors, and do not treat semantic text embeddings as IPA embeddings. The coordinated Weksa handoff contract is `weksa.utterance_embedding_handoff.v0.1`: 1024 `bge-m3:latest` semantic speech-text floats, variable-length PanPhon sequence evidence, 256 AquaSynth-owned phonetic-realization floats produced by `aquasynth.panphon_sequence_encoder.v0.1`, 32 prosody/emphasis floats, 64 projected character-state floats, and a 64-float AquaSynth-owned utterance embedding output.","evidence":"Updated E:/Projects/weksa docs/schema/state to split speech text embedding from PanPhon sequence evidence and marked the existing Nibu toy fixture as non-training; mirrored the training block in AquaSynth IPA roadmap, utterance parity research notes, spine, and memory."} {"date":"2026-05-20","topic":"panphon-sequence-encoder-organ","lesson":"The 256D phonetic realization surface is now an AquaSynth-owned learned organ, not fixed phone slots. Weksa supplies variable-length PanPhon feature sequences; AquaSynth encodes them through a trainable phonetic sequence encoder before the utterance embedding encoder consumes the 256D result.","evidence":"Added PhoneticSequenceEncoding.cs with PanPhon feature frames, deterministic sequence sketching, PhoneticSequenceNeuralEncoder, training records, and tests proving the encoder learns compact PanPhon realization embeddings."} {"date":"2026-05-20","topic":"weksa-v01-training-artifact","lesson":"The Weksa blocker has moved from schema definition to artifact consumption. Weksa now emits a machine-readable `weksa.utterance_embedding_handoff.v0.1` tiny PanPhon seed batch for `a`, `pa`, `ta`, `ka`, `sa`, and `ma`; AquaSynth's next serious training pass should load that artifact rather than local toy vectors.","evidence":"Added Weksa emitter `E:/Projects/weksa/tools/emit_tiny_panphon_handoffs.py` and generated `E:/Projects/weksa/examples/speech-training/tiny-panphon-v0.1/batch.json` with fixed 1024/32/64 input widths, inline PanPhon 22-feature frames, and pending AquaSynth-owned 256/64 learned outputs."} +{"date":"2026-05-24","topic":"cultmesh-speech-worker-authority","lesson":"CultMesh should distribute speech render/scoring work, not optimizer authority. AquaSynth emits typed SpeechRenderRequest documents; CultMesh delivers compiled Faust renderers, scoring code, small model/encoder assemblies, and batch/artifact refs to compatible leased workers; workers return SpeechRenderResult documents with loss, gradients, artifacts, and timing receipts. The authority trainer or shard-primary applies gradients through SpeechBackpropagationPipeline and commits SpeechTrainingCheckpoint documents through normal CultCache/CultMesh shard authority.","evidence":"Recorded the authority map in docs/ipa-vocal-tract-roadmap.md, state/spine.yaml, state/map.yaml, state/memory.json, and state/scratch.md. The next implementation layer is transport, admission, worker leases, payload delivery, and result collection; federated averaging waits until the simple authority path is boring."} diff --git a/state/map.yaml b/state/map.yaml index ec6bea5..9197de5 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -87,6 +87,7 @@ boundaries: - chained utterance-embedding-to-synth-driver backpropagation that returns synth input gradients and updates the utterance encoder from downstream speech loss - external-gradient speech backpropagation entry point for rendered loss surfaces - distributed speech render request/result/checkpoint CultCache contracts for multi-machine Faust rendering and local backpropagation + - CultMesh speech render/scoring worker orchestration as transport/admission/lease/result collection; workers return gradients and receipts, while one authority trainer owns optimizer updates and checkpoint commits - optional eSpeak NG generated ground-truth fixture for supervised utterance-embedding and synth-driver gradient descent - Faust-rendered speech loss surface in AquaSynth.Faust, using rendered audio analysis and finite-difference controller-output gradients - compiled native Faust speech-loss surface with exported /speech/output/N controls for batch candidate rendering without recompiling DSP diff --git a/state/memory.json b/state/memory.json index fd29018..ad309e8 100644 --- a/state/memory.json +++ b/state/memory.json @@ -57,6 +57,7 @@ "analysis/comparison metrics", "Faust-generated C# candidate rendering", "speech timing/confidence receipts for render requests, results, and checkpoints", + "CultMesh speech render/scoring orchestration as payload transport, runtime admission, worker leasing, and result collection while the authority trainer owns optimizer steps and checkpoint commits", "script scoring", "preset catalogs", "Faust emission and compiler wrapper" @@ -90,9 +91,9 @@ "rule": "For speech, add schema fields and learned-control lanes only when the parity harness has a loss surface that can exercise them. For reference synths, add DSL abstractions only when rebuilds prove the missing invariant. Parsing alone is inventory, not evidence of expressivity." }, "ipa_vocal_tract_path": { - "status": "This is now the active project center. Roadmap and research notes align the curriculum around structured utterance metadata -> learned utterance embedding -> learned synth automation -> vocal tract/envelope/LFO/filter/spectral controls -> rendered audio loss. The first Core contracts still defend the boundary: PhoneticIntent is the Weksa/language-owned handoff, ArticulatoryPlan is the AquaSynth realization layer, VocalTractMorphology declares anatomy capabilities, ArticulatoryConstraintReport is the host-visible veto surface, and VocalTractPlanResult is the tiny end-to-end proof artifact for host branching. IPA is an authoring input only: tokenization and feature mapping lower into language phonology profiles, timed articulatory gestures, morphology-specific tract/excitation curves, and Faust DSP. The staged support plan starts with tokenization and vowels, then core pulmonic consonants, expanded human IPA, airstream/phonation, and Weksa/alien morphologies. Optional eSpeak NG tests provide the first generated speech workout when ESPEAK_NG or espeak-ng is installed; eSpeak is intelligibility/timing/log-mel pressure, not anatomy truth. AquaSynth owns the learned utterance embedding encoder and synth driver: speech text embeddings plus prosody/emphasis hints plus Ghostlight/Epiphany-shaped character state vectors compress into utterance embeddings, and phonetic features plus that embedding feed a packed-buffer native C# backprop/Adam mapper that predicts tract, AM/FM, LFO, filter, and mel spectral-envelope controls. SpeechBackpropagationPipeline chains the two models: synth-driver target loss returns gradients with respect to its input, slices the utterance-embedding gradient, and updates the utterance encoder from downstream speech loss. The generated ground-truth training fixture renders eSpeak syllables, extracts log-mel evidence, and trains that chained pipeline toward eSpeak-seeded mel/control targets.", + "status": "This is now the active project center. Roadmap and research notes align the curriculum around structured utterance metadata -> learned utterance embedding -> learned synth automation -> vocal tract/envelope/LFO/filter/spectral controls -> rendered audio loss. The first Core contracts still defend the boundary: PhoneticIntent is the Weksa/language-owned handoff, ArticulatoryPlan is the AquaSynth realization layer, VocalTractMorphology declares anatomy capabilities, ArticulatoryConstraintReport is the host-visible veto surface, and VocalTractPlanResult is the tiny end-to-end proof artifact for host branching. IPA is an authoring input only: tokenization and feature mapping lower into language phonology profiles, timed articulatory gestures, morphology-specific tract/excitation curves, and Faust DSP. The staged support plan starts with tokenization and vowels, then core pulmonic consonants, expanded human IPA, airstream/phonation, and Weksa/alien morphologies. Optional eSpeak NG tests provide the first generated speech workout when ESPEAK_NG or espeak-ng is installed; eSpeak is intelligibility/timing/log-mel pressure, not anatomy truth. AquaSynth owns the learned utterance embedding encoder and synth driver: speech text embeddings plus prosody/emphasis hints plus Ghostlight/Epiphany-shaped character state vectors compress into utterance embeddings, and phonetic features plus that embedding feed a packed-buffer native C# backprop/Adam mapper that predicts tract, AM/FM, LFO, filter, and mel spectral-envelope controls. SpeechBackpropagationPipeline chains the two models: synth-driver target loss returns gradients with respect to its input, slices the utterance-embedding gradient, and updates the utterance encoder from downstream speech loss. The generated ground-truth training fixture renders eSpeak syllables, extracts log-mel evidence, and trains that chained pipeline toward eSpeak-seeded mel/control targets. CultMesh should distribute compiled Faust render/scoring payloads and collect worker gradients/results, but workers do not own model updates; the authority trainer or shard-primary applies gradients and commits SpeechTrainingCheckpoint documents through normal shard authority.", "lesson": "Do not build a complete IPA parser before the tract can say basic syllables. The durable machine is tokenizer -> feature mapper -> phonology profile -> gesture planner -> morphology -> tract plan -> Faust renderer, with dependency-injected seams and mocks at each boundary. Weksa should use IPA-like phonetic input without letting human IPA semantics own alien anatomy. Anatomy has veto power: if a morphology lacks bilabial closure or another required capability, AquaSynth reports that failure instead of smearing over it.", - "next": "Start the next speech gradient descent pass from Weksa's emitted v0.1 seed batch at `E:\\Projects\\weksa\\examples\\speech-training\\tiny-panphon-v0.1\\batch.json`: 1024 bge-m3 semantic speech-text floats, variable-length PanPhon sequence evidence, 256 AquaSynth-owned phonetic-realization embedding floats from `aquasynth.panphon_sequence_encoder.v0.1`, 32 prosody/emphasis floats, 64 projected character-state floats, and 64 AquaSynth utterance-embedding output floats. Keep semantic text embeddings and phonetic realization separate. Next tighten the parity harness: load that batch into AquaSynth training records, render tiny utterance fixtures, sweep controller outputs through compiled Faust controls, score log-mel/timing/envelope loss, record per-stage decision time/latency/budget/confidence receipts, backpropagate into the utterance encoder plus synth driver, and connect Stage 0/1 IPA token spans, vowel feature bundles, and human baseline tract-area targets to that loop." + "next": "Start the next speech gradient descent pass from Weksa's emitted v0.1 seed batch at `E:\\Projects\\weksa\\examples\\speech-training\\tiny-panphon-v0.1\\batch.json`: 1024 bge-m3 semantic speech-text floats, variable-length PanPhon sequence evidence, 256 AquaSynth-owned phonetic-realization embedding floats from `aquasynth.panphon_sequence_encoder.v0.1`, 32 prosody/emphasis floats, 64 projected character-state floats, and 64 AquaSynth utterance-embedding output floats. Keep semantic text embeddings and phonetic realization separate. Next tighten the parity harness: load that batch into AquaSynth training records, render tiny utterance fixtures, sweep controller outputs through compiled Faust controls, score log-mel/timing/envelope loss, record per-stage decision time/latency/budget/confidence receipts, backpropagate into the utterance encoder plus synth driver, and connect Stage 0/1 IPA token spans, vowel feature bundles, and human baseline tract-area targets to that loop. For distributed work, make the first CultMesh target boring: payload delivery, runtime admission, worker leases, result collection, and central or shard-primary gradient application before any federated averaging experiment." }, "dx7_reference_path": { "status": "DX7 SysEx parser extracts voice edit buffers, packed voices, packed 32-voice banks, six operators, envelopes, algorithm, feedback, oscillator sync, LFO, transpose, voice name, checksum validation, ReferencePatch features, algorithm topology from the reverse-engineered OPS algorithm ROM table, envelope-to-ADSR approximations, measured carrier output amplitude, isolated two-op modulation route scale, summed/cascaded route scale, calibrated feedback amount table, ratio detune lowering, voice transpose lowering, fixed-frequency operator lowering, ROM COM output compensation cut, operator key/rate scaling at the effective played note, topology-aware route lowering, staged operator envelopes with per-segment curves, raw and block-interpolated DX7 envelope tracing, applied-envelope curved lowering with optional sustain floor, two AquaSynth DSL rebuild attempts, first-class OperatorGraph support with graph-level vibrato, public-domain SysEx fixture coverage, dexed-py reference rendering, project-authored Dexed patch rendering, AquaSynth candidate rendering through Faust-generated C#, one thresholded rendered parity test for analog1.syx voice 13, a hard PRC SYNTH1 parity probe with listening WAV artifacts and log-mel <= 0.145, project-authored algorithm-8 summed-stack/cascade/summed-pair parity, rendered community parity for Mooger, Piano Bass, RES SYNTH1, and ANLGSYN 1 with a sustained high-band gate, a MELLOWSOLO pressure artifact test, and renderable delayed operator feedback.", diff --git a/state/scratch.md b/state/scratch.md index 8589309..97f1d6e 100644 --- a/state/scratch.md +++ b/state/scratch.md @@ -41,6 +41,13 @@ Current slice: - `SpeechDistributedTrainingCultCacheStore` reads and writes request, result, and checkpoint `.cc` stores through CultLib. No JSON queue, no bespoke serializer, no trainer pretending to own the renderer. +- New durable CultMesh training constraint: + CultMesh should distribute compiled Faust render/scoring payloads and collect + worker `SpeechRenderResult` gradients, artifacts, timing receipts, and losses. + Workers may witness and compute; they must not mutate model checkpoints. The + authority trainer or one shard-primary trainer applies gradients through + `SpeechBackpropagationPipeline` and commits `SpeechTrainingCheckpoint` + documents through normal CultCache/CultMesh shard authority. - Verification: `dotnet test tests\AquaSynth.Dsl.Tests\AquaSynth.Dsl.Tests.csproj --filter "SpeechDistributedTrainingTests" -v minimal`: 2 passed. diff --git a/state/spine.yaml b/state/spine.yaml index 501238c..ddac57b 100644 --- a/state/spine.yaml +++ b/state/spine.yaml @@ -112,6 +112,7 @@ live_invariants: - eSpeak NG is an early parity reference for timing, IPA coverage, and log-mel pressure, not an anatomy oracle or final target. - Do not expand into prosody, emotional context, personality, or broad Weksa expressivity until the tiny IPA utterance harness can show useful loss movement. - Speech training receipts must keep time as a witness: every request, render result, and checkpoint should record stage decision time, latency spent, latency budget, confidence, and artifact/model identifiers. + - CultMesh is the speech render/scoring transport and worker-orchestration layer, not distributed checkpoint authority. Workers may render, score, return gradients, artifacts, and timing receipts under a lease; the authority trainer or shard-primary applies gradients through `SpeechBackpropagationPipeline` and commits checkpoints through normal CultCache/CultMesh shard authority. - Serious speech gradient descent may now start from Weksa's first emitted v0.1 seed batch at `E:\Projects\weksa\examples\speech-training\tiny-panphon-v0.1\batch.json`: 1024 semantic speech-text floats, variable-length PanPhon sequence evidence, 256 AquaSynth-owned phonetic realization embedding floats, 32 prosody/emphasis floats, 64 projected character-state floats, and a 64-float AquaSynth utterance embedding output. - Do not use semantic text embeddings as a proxy for IPA or alien phonetic strings. Phonetic/IPA realization starts as PanPhon sequence evidence and is compressed by AquaSynth's phonetic sequence encoder. @@ -122,12 +123,13 @@ reference_targets: sources: - "docs/ipa-vocal-tract-roadmap.md" - "docs/utterance-parity-research.md" - status: "Boundary contracts, packed neural networks, utterance encoder, phonetic sequence encoder, synth-driver mapper, chained backprop, optional eSpeak rendered fixtures, Faust finite-difference loss surface, compiled Faust speech-loss controls, distributed render request/result/checkpoint contracts, timing/confidence receipts, sound-claim cards, and a Weksa-emitted v0.1 tiny PanPhon batch exist. The active need is a loader/training pass that consumes Weksa's batch artifact and tests many controller weights as rendered speech output against simple ground truth." + status: "Boundary contracts, packed neural networks, utterance encoder, phonetic sequence encoder, synth-driver mapper, chained backprop, optional eSpeak rendered fixtures, Faust finite-difference loss surface, compiled Faust speech-loss controls, distributed render request/result/checkpoint contracts, timing/confidence receipts, sound-claim cards, and a Weksa-emitted v0.1 tiny PanPhon batch exist. CultMesh should now be used for transport, runtime admission, worker leases, payload delivery, and result collection, while the authority trainer owns optimizer application and checkpoint commits. The active need is a loader/training pass that consumes Weksa's batch artifact and tests many controller weights as rendered speech output against simple ground truth." first_slice: - "Keep the utterance schema small and testable: segments, spans, duration/stress/boundary hints, speaker/morphology identity, pitch/loudness/rate/breath controls, and named context vectors." - "Load Weksa's tiny PanPhon v0.1 batch artifact into AquaSynth training records before the next serious gradient descent run." - "Use eSpeak NG fixtures for generated ground truth while remembering that formant parity is not anatomy parity." - "Connect Stage 0/1 IPA and vowel tract-area fixtures to the compiled Faust loss surface before expanding consonants." + - "Use CultMesh first for compiled Faust render/scoring jobs: ship payloads, admit compatible workers under leases, collect `SpeechRenderResult` gradients/receipts, and apply centrally or by one shard-primary trainer." dx7_dexed: priority: 1 purpose: "Structured FM/operator graph pressure test."