Research: fine-tuned lowerers and embedding-mapped lexicons

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/research/2026-10-29-lowering-strategy-prior-art.md

@@ -0,0 +1,71 @@
+# Prior art for Weksa lowering strategy: fine-tuning and embedding mapping, 2026-10-29
+
+Eyes pass, campaign `weksa-lowering`. Facts and pointers only; no recommendation. Tags: [V] read this pass (fetched page/PDF, or a search-result summary, marked "(ss)" when only a search summary was seen); [B] background, not re-verified; [GAP] looked for, not found. Covered already and not redone: docs/research/2026-10-29-interlingua-lowering-prior-art.md, 2026-10-29-golf-loop-prior-art.md.
+
+## 0. Project's own prior thinking (F:\Projects\weksa\docs)
+
+- targets/interlingua-lowering.md: lowerer shape is an open fork (deterministic, model, or both with one contract). Invariants include packet-owns-meaning, lowering-traced, profile-owns-register, no-surface-leak, corpus-clean (no Discord). [V]
+- persona-finetuning-dataset-exploration.md: already plans fine-tuning two separate adapters, "Persona intent extraction" and "Weksa lowering"; says do not fine-tune for direct context->utterance impersonation; row schemas `weksa.finetune.lowering_row.v0` and `weksa.finetune.persona_interlingua_row.v0`; interlingua labels would be derived from target utterances by a model pass ("marked as derived from answer-key surface evidence"); source corpus is the Discord archive (18,049 pre-2025 authored messages; 16,499 casual candidates), which the target doc now forbids as training data (`gamecult-chat:ruling:discord-taints-training`). Splits by year and channel holdouts proposed. [V]
+- utterance-embedding.md: embeddings in Weksa today serve the AquaSynth speech handoff only (bge-m3, 1024 floats, `speech_text_embedding`); text embedding explicitly "not a pronunciation or alien-phonetics embedding". No doc uses embeddings for concept lexicon or translation. [V] language-model.md lists "concept clustering" as an allowed model use; best-modern-approach.md line 33: "Embeddings and LLMs can cluster English concepts, but the ontology decides what..." [V]
+- No existing fine-tune run, adapter, or training-job record found in docs. [GAP: checked docs only, not scripts/.weksa-runs]
+
+## 1. Fine-tuned lowerers (structure/data to text)
+
+- AMR-to-text with pretrained seq2seq: T5-large about 45.8 BLEU plain, about 49.7 after task-adaptive training on silver data (Ribeiro et al., "Investigating Pretrained Language Models for Graph-to-Text Generation", arXiv 2007.08426); same paper varied training data 1%-100%: BART and T5 already good at 40% of data, T5 about 5.6 BLEU ahead of BART at 1% on WebNLG. [V (ss)]
+- 2025 WiNLP (GPT4AMR) says top AMR-to-text models (AMRBART, BiBL) reach roughly 50 BLEU with a remaining quality gap; LLM paraphrase after AMR-to-text gave no automatic-metric gain but was preferred in human evaluation. [V (ss)] https://preview.aclanthology.org/setup/2025.winlp-main.2
+- Few-shot/low-resource D2T: T5-base fine-tuned on 100 triple sets used as low-resource baseline (Cycle training, ACL 2023, arXiv 2305.14793); another summary reports neural D2T "high fidelity with about 300 annotated examples". [V (ss), second claim weakly sourced]
+- AMR-to-text fine-tuned LLM vs prompted frontier LLM head-to-head on AMR 3.0 with BLEU or faithfulness: [GAP] (also noted as gap in the earlier prior-art file).
+- Open LLMs zero-shot on D2T: Kasner and Dusek, ACL 2024 (arXiv 2401.10186): Llama 2, Mistral, Zephyr on QuintD (fresh API-derived records); human annotators and GPT-4 metric both flag over 80% of outputs as containing a semantic error (v1 said 80% human, 91% GPT-4). [V (ss)] No frontier-API or fine-tuned comparison in that paper as seen. [GAP]
+- Faithfulness with model size: Mahapatra and Garain 2024 (arXiv 2407.14088), 12 models in T5/BART/OPT/BLOOM/Llama 2 families fine-tuned on five D2T sets incl. E2E, DART, WebNLG: larger fine-tuned models improve readability/informativeness, faithfulness (BARTScore) often decreases with size. [V (ss)]
+- Error type: SPOR (arXiv 2405.10650) reports stronger fine-tuned models (Mistral-7B, Llama-2-13B) err mostly by omission on E2E; a Cambridge MLMI dissertation reports T5 outputs showed rare hallucination, many omissions. Cycle training reduced human-counted hallucination errors from 14.84 to 2.57 and factual errors from 8.05 to 0.49 (units as in paper; low-resource fine-tune baseline). [V (ss)]
+- SCOPE (arXiv 2502.13674): faithfulness hallucination described as common in standard fine-tuned LLMs; self-supervised fix; notes WebNLG and E2E are of limited complexity. [V (ss)]
+- Persona/style fine-tunes: no study found comparing LoRA/QLoRA persona-style tunes with strong prompting on voice fidelity. Nearest: (a) 2026 arXiv 2609.21857: two small open models fine-tuned on personality-labelled data were no better at role-playing personalities than baselines (authors note low inter-rater agreement); (b) PersonaEval (2508.10014): role-knowledge fine-tuning did not help role identification and may degrade it; (c) NeurIPS 2025 multi-turn RL with consistency rewards cut persona inconsistency by over 55% (simulated users). [V (ss)] [GAP: LoRA-specific voice-holding evidence]
+- Style imitation: "Catch Me If You Can? Not Yet" (arXiv 2509.14543): few-shot beats zero-shot, 2 to 10 shots adds little, informal/everyday-author style imitation remains weak; no fine-tuning baseline run. "How Well Do LLMs Imitate Human Writing Style?": exemplar-based prompting and text completion are key; imitation remains lower-perplexity than human text. [V (ss)] [GAP: fine-tuned vs prompted on same authors]
+- LoRA vs full fine-tuning: Biderman et al. 2024 (TMLR, arXiv 2405.09673): LoRA trails full FT in code/math in standard settings, forgets less outside target domain, sensitive to learning rate/target modules/scale more than rank. Follow-up 2410.21228: LoRA forgets less but "intruder dimensions" matter. [V (ss)]
+
+## 2. Training data from the loop itself
+
+- Back-translation, Sennrich/Haddow/Birch ACL 2016 (arXiv 1511.06709): synthetic source from target-side monolingual text mixed with real parallel data; +2.8-3.7 BLEU WMT15 en->de, +2.1-3.4 BLEU IWSLT14 tr->en. [V (ss)]
+- Edunov et al. EMNLP 2018 (arXiv 1808.09381): sampled or noised-beam synthetic sources beat beam outputs by 1.7 BLEU average; beam still fine in very low-resource settings. Iterative back-translation (Hoang et al. 2018; Cotterell and Kreutzer 2018) cited second-hand only. [V (ss); Hoang itself GAP]
+- D2T cycle training (ACL 2023, arXiv 2305.14793): trains text->data and data->text models in a cycle on unpaired data; fewer hallucination/factual errors than a 100-pair fine-tune (see §1). [V (ss)]
+- Collapse: Shumailov et al., Nature 631:755 (2024): models trained recursively on generated data lose tails of the distribution and produce repetitive output, in LLMs, VAEs, GMMs. Whether mixing/accumulating real data prevents it: not confirmed from the sources seen. [V (ss)] [GAP: accumulate-vs-replace results]
+- Round-trip leakage/inflation (same model both directions, consistency not accuracy): from prior pass, §2 of interlingua-lowering-prior-art.md. [V earlier pass]
+- Specific to this loop: encoder output (packet) of an item and its English lowering form a synthetic pair where the English is the original; the weksa dataset note already marks such interlingua labels as derived from answer-key surface evidence. [V, weksa docs] [GAP: published work on training a generator on packets produced by an LLM encoder from the same targets; closest is back-translation/cycle training above]
+
+## 3. Feasibility, hosts, and Anthropic fine-tuning
+
+- Hosts (gamecult-ops/inventory.md): Yggdrasil = Netcup KVM, 16 vCPUs, 62 GiB RAM, ~2 TB, Debian 13; no GPU listed; carries live public services; CPU model/AVX flags and memory bandwidth not recorded. [V] [GAP: CPU model]
+- GPUs found in inventory: Raven RTX 4060 Ti 16380 MiB (household gaming machine and Vili's Kimodo jobs; Bonsai 27B takes about 10.4 GiB on demand and is "never resident" by ruling 2026-09-19; WSL ceiling 7.7 GiB RAM; operator workstation, no reboot without approval); Nightwing GTX 860M running qwen3-embedding:0.6b via Ollama, laptop, "not a build host"; Starfire GPU not recorded in inventory. [V] [GAP: Starfire GPU]
+- Starfire (memory workstation-load-budget.md): heavy builds/tests go to Yggdrasil; Starfire does only Windows-only work, one heavy job at a time, froze 2026-09-22, hard-reset 2026-10-18 (cause unproven), 2026-10-28 reboot was a tripped cable; Yggdrasil verify queue has 5 slots. [V]
+- QLoRA cost: tutorial/vendor sources say 8B QLoRA needs about 16-24 GB VRAM; one estimate 50k examples x 3 epochs on Llama 3 8B about 6 h on an RTX 4090 (2.5 h with Unsloth), about $2-5 rented; another 10k samples x 3 epochs 2 h on RTX 5090. Unverified vendor numbers. [V (ss), low confidence]
+- CPU LoRA training: blogs claim 0.5-3B models in 15-60 min on 16-32 GB CPUs, unbenchmarked; llama.cpp's own training README calls finetune "very much WIP" and mentions LLaMA 3.2 1B working with 24 GB. HF PEFT on CPU PyTorch is the documented path. [V (ss), low confidence] [GAP: measured CPU LoRA run on a 16-vCPU server]
+- CPU inference: Llama 3.1 8B Q4_K_M 14.2 tok/s at 12 threads, 12.5 at 8, 7.1 at 4 on an EPYC 9354 (markaicode blog); Qwen3 8B Q4_K_M 26.4 tok/s generation, 210 tok/s prompt on Ryzen 9 7950X with 16 threads; 3.2 tok/s on a 4-core Xeon. Generation is memory-bandwidth-bound. [V (ss), blogs] [GAP: numbers for Yggdrasil's CPU; a KVM guest's bandwidth unknown]
+- Anthropic fine-tuning, current: platform.claude.com model overview (fetched 2026-10-29) lists Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5 on the Claude API, Bedrock, Google Cloud, Foundry, Claude Platform on AWS, with no mention of fine-tuning. [V] The old docs path /build-with-claude/fine-tuning returns 404. [V] AWS Bedrock supported-for-fine-tuning table (docs.aws.amazon.com/bedrock/latest/userguide/custom-model-fine-tuning.html, fetched): Anthropic row is only Claude 3 Haiku, us-west-2; others Nova, Titan, Llama 3.1/3.2/3.3. [V] AWS blog on Claude 3 Haiku fine-tuning carries notice "Claude 3 Haiku reached end of life on September 10th, 2026. Use Claude 3.5 Haiku instead." and says nothing on whether fine-tuning survives. [V] Bedrock also offers reinforcement fine-tuning and distillation (teacher-student) as methods; which models support them was not read. [V] [GAP: Claude fine-tuning on Claude API directly; any Claude model newer than 3 Haiku fine-tunable]
+- Bedrock customization is billed per training tokens x epochs plus monthly model storage. [V]
+
+## 4. Word-embedding mapping idea
+
+- Mikolov, Le, Sutskever 2013 (arXiv 1309.4168) [V, PDF read]: linear translation matrix learned from a small dictionary (5K most frequent words) on monolingual word2vec spaces. Table 2 (WMT11): En->Sp P@1 33%, P@5 51%; Sp->En 35%/52%; En->Cz 27%/47%; Cz->En 23%/42% (readings from table layout; row/direction attribution partly ambiguous in the extracted text). The abstract's "almost 90% precision@5" applies to a confidence-filtered subset (about half of words in frequency ranks 5K-6K, distance threshold), not all words. Small monolingual training sets cut P@5 to about 25%. The paper says the method "was less useful for more [distant pairs]" (English/Czech, Vietnamese tested).
+- MUSE, Conneau et al. ICLR 2018 (arXiv 1710.04087) [V, PDF read]: Table 1 P@1, unsupervised Adv-Refine-CSLS: en-es 81.7, es-en 83.3, en-fr 82.3, fr-en 82.1, en-de 74.0, de-en 72.2, en-ru 44.0, ru-en 59.1, en-zh 32.5, zh-en 31.4, en-eo 28.2, eo-en 25.6 (fastText on Wikipedia, 1,500 queries vs 200k targets). Word-by-word translation en<->eo BLEU 11.1 and 14.3; authors say meaning mostly conveyed with "simple errors" and fixable by an LM.
+- Søgaard, Ruder, Vulic ACL 2018 (P18-1072): word-embedding spaces "far from isomorphic" even for close languages; unsupervised BDI degrades on morphologically rich non-dependent-marking languages, different domains/embedding algorithms; graph-similarity metric correlates near-perfectly with induction success; identical-word weak supervision helps. [V (ss)]
+- VecMap (Artetxe et al.) as named; not read this pass. [B]
+- Conlang with no corpus: [GAP] found no paper that places invented words at concept vectors or assigns an invented lexicon by clustering an embedding space. Nearest: Toki Pona word2vec study (Huang and Wu, arXiv 2606.17299, about 130 words: ANIMALS/COLORS form exclusive clusters, SHAPES/BODY/DIRECTION overlap) [V (ss)]; Cree lexicon classified semi-automatically with English word2vec vectors (AmericasNLP 2021) [V (ss)]; Spanish embedding-cluster "semantic domains" paper [V (ss)].
+- Conlang LLM work: ConlangCrafter (arXiv 2508.06094) multi-stage LLM pipeline generating phonology, morphology, syntax, lexicon, translation, with self-refinement; evaluated on consistency and typological diversity. ConlangBench (arXiv 2608.03505): 21 conlangs, over 21M parallel pairs (Esperanto-dominated; 430K across the other 20), 321K vocab entries; models translate better into a posteriori conlangs (vocabulary from natural languages); models could learn all eight conlangs with sufficient parallel data. [V]
+- MTOB (Tanzer et al., ICLR 2024, arXiv 2309.16575): Kalamang from one grammar book (~217K tokens) plus word list and sentences; best model about 44.7/45.8 chrF (kgv->en / en->kgv) vs human learner 51.6/57.0; follow-up 2508.06665 (GPT-4.1) finds parallel sentences alone beat full-grammar prompting and grammar alone fails on complex sentences. [V (ss)] Relevant as work on LLM + grammar documents for a language with no web corpus.
+- Embedding inversion: vec2text (Morris et al., EMNLP 2023, arXiv 2310.06816): recovers 92% of 32-token inputs exactly from a GTR-base style embedding; on MSMARCO with ada-002: 83.4 BLEU/60.9% exact at 32 tokens, 55.0 BLEU/8.0% exact at 128 tokens; a 2025 reproduction notes length sensitivity. Inversion targets the same encoder's own English text; no conlang use found. [V (ss)] [GAP: inversion into non-English or invented surface]
+- Sentence-embedding interlingua: SONAR (Meta, 200 languages text) used as concept space with a decoder in Large Concept Models (arXiv 2412.08821); SONAR reported better than LASER3 and LaBSE on xsim/xsim++. [V (ss)] LaBSE/LASER coverage details [B]. Weksa's current embedding: bge-m3 [V].
+- "Nearest English cluster" as gloss vs fluent text: only indirect evidence: MUSE word-by-word en<->eo (above), Toki Pona wiki note that word-level glossing is the simplest approach and neural translators often err [V (ss)]. [GAP: measurement of nearest-cluster gloss vs fluent generation].
+
+## 5. Hybrids
+
+- Overgenerate-and-rank: Velldal (LOGON/Oslo) discriminative realization ranker trained on a few thousand treebank items beat a surface LM; Varges instance-based ranker over rule candidates; critique that overgeneration is expensive. [V (ss)]
+- Neural ranking for word ordering in surface realization (pointwise/pairwise/listwise), five languages (2021). [V (ss)]
+- Grammar-constrained decoding: Geng et al. EMNLP 2023 (arXiv 2305.13971): any CFG plugged into constrained beam search; constrained LMs beat unconstrained and sometimes finetuned models on IE/entity disambiguation with LLaMA. Outlines (Willard and Louf 2023, arXiv 2307.09702) regex/grammar guided generation; SynCode (2403.01632); Grammar-Aligned Decoding (2405.21047) argues grammar constraints distort the LM distribution. [V (ss)] Prior pass: Shin et al. 2021 (constrained decoding into a canonical sublanguage then deterministic map), Tam et al. 2024 (format constraints cost reasoning; two-step NL-then-convert keeps quality). [V earlier pass]
+- GF + neural: Chalmers 2024 thesis builds a transformer parser to GF trees (prior pass). [GAP] no paper found on neural lexeme/register choice feeding a GF or SimpleNLG-style realiser, nor on LLM output constrained to a GF concrete syntax.
+- Abstract Wikipedia/Ninai/Udiron (prior pass): symbolic realisers with lexeme lookup; no learned component. [V earlier pass]
+
+## Evidence pointers (URLs)
+https://docs.aws.amazon.com/bedrock/latest/userguide/custom-model-fine-tuning.html ; https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html ; https://platform.claude.com/docs/en/about-claude/models/overview ; https://aws.amazon.com/blogs/machine-learning/fine-tune-anthropics-claude-3-haiku-in-amazon-bedrock-to-boost-model-accuracy-and-quality/ ; arXiv 1309.4168, 1710.04087 (PDFs read, text at scratchpad mik.txt, muse.txt), aclanthology P18-1072, 1511.06709, 1808.09381, 2305.14793, 2007.08426, 2401.10186, 2407.14088, 2405.10650, 2502.13674, 2405.09673, 2410.21228, 2509.14543, 2310.06816, 2412.08821, 2305.13971, 2309.16575, 2508.06665, 2508.06094, 2608.03505, 2606.17299, 2609.21857, 2508.10014, 2511.00222; Nature 631:755 (2024). Local: F:\Projects\gamecult-ops\inventory.md (lines ~51-110 Raven, ~350 Nightwing, ~571-615 Yggdrasil); C:\Users\Meta\.claude\projects\F--Projects-CultLib\memory\workstation-load-budget.md; F:\Projects\weksa\docs\persona-finetuning-dataset-exploration.md.
+
+## Caveats
+Most web claims were seen as search-result summaries, not full papers; the two PDF reads (Mikolov, MUSE) are first-hand. Training-cost and CPU-speed numbers come from blogs/vendors and are low confidence. Mikolov table-row attribution is partly ambiguous in text extraction.