Research: golf-loop metric, leakage and corpus prior art

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/research/2026-10-29-golf-loop-prior-art.md

@@ -0,0 +1,88 @@
+# Prior art for the Weksa golf loop: metric, leakage, corpus, 2026-10-29
+
+Eyes pass, campaign `weksa-lowering`. Facts and pointers only; no recommendation. Tags: [V] read this pass (page or search result returned it; "search-only" = seen only in a search summary, not the paper); [B] background, not re-verified; [GAP] looked for, not found. Several arXiv PDFs came back as unreadable binary; where so, the HTML/abs version was used or the claim is downgraded. Not re-covered: UNL, KANT, GF, AMR, Abstract Wikipedia, LLM AMR parsing (see docs/research/2026-10-29-interlingua-lowering-prior-art.md).
+
+## 1. Metrics for meaning AND nuance
+
+### 1.1 Metric vs human correlation, measured
+- Briakou, Agrawal, Tetreault, Carpuat, EMNLP 2021, multilingual formality transfer (en, it, br-pt, fr). [V via ar5iv] https://ar5iv.labs.arxiv.org/html/2110.10668 ; https://aclanthology.org/2021.emnlp-main.100
+  - Meaning preservation, Spearman rho vs human, en/it/pt/fr: chrF 0.59/0.67/0.70/0.75; BERTScore 0.59/0.62/0.65/0.68; METEOR 0.54/0.58/0.64/0.60; s-BLEU (vs input) 0.48/0.59/0.67/0.66; r-BLEU (vs reference) 0.31/0.00/0.05/0.12; zero-shot XLM-R STS ~0.65-0.68 (it/pt/fr). Authors recommend dropping reference-BLEU.
+  - Fluency: XLM-R pseudo-log-likelihood best (0.50/0.46/0.55/0.61), "moderate"; KenLM 0.27-0.45.
+  - Formality (style strength): XLM-R regression, zero-shot cross-lingual, rho 67/72/59/61 (en/pt/fr/it); mBERT lower; binary classifiers over-predict "formal" outside English, authors recommend regression, not classification.
+  - System-level: human and automatic metrics do not always agree on best system; r-BLEU got 30/40 pairwise system comparisons right vs chrF 25/40 (authors: suggestive only).
+  - Not evaluated there: BLEURT, COMET, MAUVE. Only one style axis (formality), a single sentence-level task; no hedging, humour, affect, stance.
+- Mir, Felbo, Obradovich, Rahwan, NAACL 2019 "Evaluating Style Transfer for Text" (Yelp sentiment). Three axes: style-transfer intensity, content preservation, naturalness. Proposed: Word Mover's Distance on style-masked text for content; direction-corrected EMD for intensity; adversarial classifier for naturalness; models show trade-offs, so compare at points on a trade-off curve. [V search-only] https://aclanthology.org/N19-1049/ . Correlation numbers [GAP: PDF not read].
+- Briakou et al. GEM 2021 "A Review of Human Evaluation for Style Transfer": surveys 97 papers' human-evaluation practice. [V search-only] https://aclanthology.org/2021.gem-1.6
+- BERTScore (Zhang et al. 2020) claims better human correlation than n-gram metrics and robustness on PAWS adversarial paraphrase [V search-only] https://arxiv.org/abs/1904.09675 . BLEURT is a learned metric trained on human ratings; the paper notes learned metrics inherit training-data bias [V search-only] https://arxiv.org/pdf/2004.04696 .
+- On Evaluation Metrics for Paraphrase Generation (2022): most common metrics align poorly with human annotation; embedding metrics give very high scores to a candidate identical to the input; reference-free variants align better than reference-based. [V search-only] https://arxiv.org/pdf/2202.08479
+- MAUVE (Pillutla et al. 2021): distribution-level divergence between model text and human text in quantised embedding space; authors claim human-judgment correlation on three open-ended tasks. It scores a set of texts, not a pair. [V abstract] https://arxiv.org/abs/2102.01454 . No figures read.
+- COMET: no source read this pass. [GAP]
+- Metric-vs-human numbers for hedging, humour, affect, stance, politeness-as-nuance, persona voice: [GAP] none found. Nearest: formality (Briakou above). Also "Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer" (2022), title only [V] https://ar5iv.labs.arxiv.org/html/2204.07549 .
+
+### 1.2 Style / authorship embeddings
+- STEL (Wegmann & Nguyen, EMNLP 2021): similarity-based style evaluation over four dimensions (formal/informal, simple/complex, contraction, number substitution) with content-controlled sentence pairs; BERT-based methods beat simple style measures. [V search-only] https://arxiv.org/abs/2109.04817 . Human-baseline and accuracy numbers [GAP: PDF unreadable].
+- Wegmann, Schraagen, Nguyen 2022 "Same Author or Just Same Topic?": authorship-verification-trained representations may encode content; controlling for conversation labels gives more content-independent style representations than domain control or none. [V abstract] https://arxiv.org/abs/2204.04907 . STEL-or-Content benchmark introduced there.
+- StyleDistance (2024): says LUAR is sometimes used for automatic style-transfer evaluation, does not explicitly train for content independence; claims lead among content-independent style embeddings on STEL and STEL-or-Content. [V search-only] https://arxiv.org/pdf/2410.12757 . LUAR paper itself (Rivera-Soto et al., EMNLP 2021) [B], not fetched. Per-model numbers [GAP].
+- STEB: Style Text Embedding Benchmark (2026) notes LUAR is evaluated only on authorship retrieval. [V search-only] https://huggingface.co/papers/2606.31741.md
+- Style embeddings tested against persona-voice/hedging/humour: [GAP].
+
+### 1.3 LLM-as-judge and its biases
+- Zheng et al. 2023 (MT-Bench/Chatbot Arena) [V via arxiv html] https://arxiv.org/abs/2306.05685 : position-bias consistency (default prompt) Claude-v1 23.8%, GPT-3.5 46.2%, GPT-4 65.0%; repetitive-list verbosity attack failure Claude-v1 91.3%, GPT-3.5 91.3%, GPT-4 8.7%; self-enhancement: GPT-4 about +10 points win rate for own answers, Claude-v1 about +25, authors say data too limited to confirm; GPT-4 vs human agreement 85% (ties excluded) vs human-human 81-82%; 66% counting ties.
+- Wataoka et al. 2024 "Self-Preference Bias in LLM-as-a-Judge" [V] https://arxiv.org/html/2410.21819v1 : eight models, ~33,000 Chatbot Arena dialogues; GPT-4 largest bias (equal-opportunity metric 0.520; demographic-parity 0.749); judges favour lower-perplexity text regardless of authorship; authors hypothesise familiarity drives self-preference.
+- Li et al. 2025 "Preference Leakage": contamination when the data-generating and judging LLM are the same, inherited, or same family; judges favour related students; harder to detect than earlier biases. [V abstract] https://arxiv.org/abs/2502.01534
+- TwinVoice (persona simulation benchmark): judge-vs-human Spearman 0.591 for scoring vs 0.605 human-human; Cohen kappa 0.646 vs 0.673 for ranking. [V search-only] https://arxiv.org/pdf/2510.25536
+- "Can LLM be a Personalized Judge?": low agreement with human ground truth on PRISM/OpinionQA/empathetic dialogue; ~80% on a high-confidence subset. [V search-only] https://arxiv.org/pdf/2406.11657
+- Secondary-source claims (blogs): 20-40% of verdicts flip on swap for closely matched pairs; up to 40% swing across prompt templates. [V search-only, practitioner blogs, uncontrolled] e.g. https://tianpan.co/blog/2026-04-27-llm-judge-bias-audit-length-position-format . Treat as unverified.
+- Pairwise vs pointwise stability, per a glossary page: absolute scores fluctuate more across judge models than pairwise. [V search-only, low authority] https://howaiworks.ai/glossary/llm-as-a-judge
+- Judge agreement specifically on register/hedging/humour/affect with numbers: [GAP].
+
+## 2. Goodhart and leakage in round-trip / autoencoder setups
+
+- Round-trip fidelity caveats already catalogued in the prior-art note section 2 (consistency not accuracy; same model both ways inflates). Not repeated.
+- Self-bias in model-built benchmarks: Xu et al. 2025 (MT as testbed): generated test data and the evaluating LLM each contribute self-bias, combining amplifies it; "each model ranks itself first"; stylistic homogeneity persists despite diversity controls; source-text diversity (their diversity metric) partly mitigates; also appears in open-ended generation (Chatbot Arena). [V abstract] https://arxiv.org/abs/2509.26600 (EMNLP 2026 version https://aclanthology.org/2026.emnlp-main.629.pdf). Silencer (Yuan et al. 2025) weights generated benchmarks using heterogeneity across multiple generators. [V search-only] https://www.arxiv.org/pdf/2505.20738
+- LLM bottleneck/autoencoder leakage of surface form: no paper found on an encoder smuggling target surface form through a natural-language/structured bottleneck, nor on paraphrase-invariance tests for it. [GAP]. Adjacent: a paper proposing separate "Exact Reconstructive Effectiveness" and "Semantic Reconstruction Effectiveness" for LLM compression (separates verbatim recovery from preserved intent) [V search-only, paper not identified; see https://cdnjs.deepai.org/publication/semantic-compression-with-large-language-models]; ICAE/gist-token compression papers [V search-only] https://arxiv.org/abs/2505.17117 .
+- Emergent communication [mostly V search-only]:
+  - Keresztury & Bruni 2020: deep-learning solutions often lack abstract reasoning; compositionality linked to out-of-distribution generalisation. https://arxiv.org/abs/2001.08618 [V abstract]
+  - Li & Bowling NeurIPS 2019: resetting listeners (ease-of-teaching) pushes emergent languages toward more structure; reviewers doubted transfer to complex settings. https://arxiv.org/abs/1906.02403 [V]
+  - ACL 2026 long paper proposes a bijectivity-based metric claimed to track whether agents can speak/hear an emergent language better than topographic similarity (Brighton & Kirby 2006) or bag-of-symbols disentanglement (Chaabouni et al. 2020). https://aclanthology.org/2026.acl-long.1389.pdf [V search-only]
+  - CELEBI (NeurIPS 2025): discrete bottleneck + iterated learning, reports gains in reconstruction and topographic similarity on Shapes3D/MPI3D. https://neurips.cc/virtual/2025/poster/115077 [V search-only]
+  - Chaabouni et al. 2020 "Compositionality and generalization in emergent languages" (ACL) [B]; not fetched. No source found tying emergent-communication leakage directly to LLM agents using natural-language packets. [GAP]
+- Defences named in sources found: judge and generator from different model families, check item overlap with generator training data (search summary of the self-bias literature); multi-generator weighting (Silencer). Vocabulary restriction and paraphrase-invariance as leakage defences: [GAP: no source].
+
+## 3. Iterating a schema / spec against a metric
+
+- OPRO (Yang et al., Large Language Models as Optimizers): the base setup uses no validation set; training accuracy often 5-20 points above test; suggests more training data and early stopping. [V via search summary] https://arxiv.org/pdf/2309.03409 . A follow-up (Learning from Contrastive Prompts) reports it holds out a validation set and shows less overfitting than OPRO. https://arxiv.org/pdf/2409.15199 [V search-only]
+- TextGrad (Yuksekgonul et al., arXiv 2406.07496): per a 2026 evaluation paper, each step is followed by a validation step and the proposed instruction is kept only if it scores higher on validation; MIPRO (DSPy) run with capped 200 train / 300 validation samples. https://arxiv.org/pdf/2604.27637 "Optimization before Evaluation" [V search-only]. TextGrad's own text on this: [GAP: paper not read].
+- DSPy docs: separates train / validation / held-out test; says for prompt optimisers to start with 20% train and 80% validation (opposite of neural-net practice). [V search-only; my direct fetch of dspy.ai returned only a redirect link] https://dspy.ai/learn/optimization/overview
+- LLM-AutoDiff compares against MIPROv2 with best-validation checkpoint chosen before test. https://arxiv.org/pdf/2501.16673 [V search-only]
+- Growing a meaning-representation inventory from failures: [GAP] nothing on UMR/AMR guideline revision driven by generation failures. Found only: FrameNet lexical-unit induction (Pennacchiotti et al. EMNLP 2008, distributional + WordNet; release then 795 frames, ~10,000 LUs) https://aclanthology.org/D08-1048 ; NutFrame LREC-COLING 2024, LLM frame/FE/relation induction from FrameNet, "remains difficult" https://aclanthology.org/2024.lrec-main.1079 ; Annotating FrameNet via structure-conditioned LM generation (2024) https://arxiv.org/html/2406.04834v2 ; 2025 ACL-SRW frame induction on C4 web text. All [V search-only].
+- Weksa's own promotion rule (new fields as namespaced extensions, promoted after use in 2 real packets) is in interlingua-standard.md [V, prior note]. Overfitting defences specific to spec-growth loops: [GAP].
+
+## 4. Corpora and licences
+
+Verified licence facts are few. Discord data excluded by policy; not read.
+- DailyDialog: CC BY-NC-SA 4.0; 13,118 dialogues (11,118/1,000/1,000). [V HF page] https://huggingface.co/datasets/li2017dailydialog/daily_dialog . Non-commercial plus share-alike; whether that bars model training in this campaign is a reading not made here.
+- PersonaChat (Zhang et al. 2018): ParlAI project page, archived 2020-08-01, states no licence, only "You may continue to use the data without issue". [V] https://parl.ai/projects/personachat/ ; paper https://arxiv.org/abs/1801.07243 says data released open source via ParlAI; licence text [GAP]. Crowdworker-written, persona-statement-conditioned chit-chat [B].
+- LIGHT: ParlAI page states no licence; contents: 663 locations, 3,462 objects, 1,755 character types, 11,000 episodes; WILD 41,131 training episodes from game deployment (human players, so a different provenance than crowdwork). [V] https://parl.ai/projects/light/ . Licence [GAP: check ParlAI repo].
+- Cornell Movie-Dialogs Corpus: licence/terms page fetch failed (certificate rejected) https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html . [GAP] Underlying scripts are copyrighted film text [B]; dataset licence unverified.
+- Project Gutenberg: "vast majority" of eBooks public domain in the US, so no one can grant or withhold permission for uses; PG cannot sublicense copyrighted items; subject to its Terms of Use, which the permission page links and does not summarise; no statement on ML/bulk use on that page. [V] https://www.gutenberg.org/policy/permission.html . Terms-of-Use text on bulk/automated access and the trademark licence: [GAP, not read]. Items public domain in the US may not be elsewhere (same page).
+- Google Synthetic-Persona-Chat (model-generated, persona-chat-style): CC BY-NC? listed as cc-by-4.0 on HF. [V search-only] https://huggingface.co/datasets/google/Synthetic-Persona-Chat
+- Operator-authored or model-generated corpus: no licence question beyond provider terms [B, unread this pass]. Problems with evaluating on model-generated text: see 2 (self-bias, preference leakage, stylistic homogeneity: Xu et al. 2025; Li et al. 2025). The same Xu et al. finding says it extends to open-ended generation.
+- Expressive-register-annotated English corpora (hedging, humour, affect, politeness labelled), with permissive terms: [GAP] not searched beyond the list above (e.g. Stanford Politeness Corpus, GYAFC formality, Enron etc. are [B] candidates, unverified, GYAFC is known to carry Yahoo Answers terms [B]).
+
+## 5. Conlang projection and evaluation without a corpus
+
+- GF/Abstract Wikipedia: see prior note. Related: "Multilingual Text Generation for Abstract Wikipedia in Grammatical Framework: Prospects and Challenges" (Chalmers) [V title/abstract page] https://research.chalmers.se/en/publication/549205 . GF conlang grammars (Esperanto, Lojban, Toki Pona): [GAP] none found in search. (GF RGL status table lists natural languages only per prior note.)
+- Toki Pona: arXiv 1712.09359 describes the language (14 letters, ~124 lemmas, ~10 syntax rules), vocabulary analysis, a text-synthesis tool, WordNet synsets; tooling, not a meaning-representation realiser. [V search-only] https://arxiv.org/pdf/1712.09359
+- Lojban: a student semantic analyser mapping a Lojban subset to logical form / Prolog (parsing direction) and a 2005 WPI thesis using Lojban as an interface to a first-order-logic reasoner with speech-synthesised answers [V search-only] https://lojban.org/files/papers/lojban_parser_paper ; https://digital.wpi.edu/downloads/g732db469 . A community English-to-Lojban fine-tune of an OPUS-MT model on Tatoeba pairs, 1,000 dev, 39 gold held-out pairs. [V search-only] https://huggingface.co/linguardia/opus-mt-en-jbo . A realiser from logical/semantic form into Lojban or Esperanto in the literature: [GAP].
+- Esperanto: first systematic MT benchmark (de Gibert & de Gibert, SIGUL 2026) over rule-based, encoder-decoder and LLM systems, 600M-9B parameters; NLLB family best in all pairs, followed by compact trained models and a fine-tuned LLM. [V search-only] https://aclanthology.org/2026.sigul-1.31/ ; https://arxiv.org/abs/2603.29345 . Esperanto has >20M English pair sentences per ConlangBench's framing.
+- ConlangBench (arXiv 2608.03505): 21 existing conlangs, 21M+ conlang-English pairs (430K+ from the 20 non-Esperanto ones, including Toki Pona classified as engelang); translation performance associated with natural-language similarity and conlang characteristics; smaller conlangs to be read cautiously. [V search-only] https://arxiv.org/pdf/2608.03505
+- Evaluating a lowering into a language with no corpus: no source addressing constructed-language generation evaluation directly. [GAP]. Adjacent: reference-free MT evaluation for low-resource languages: Sindhujan et al. LoResLM 2025 found prompt-based LLM evaluation outperformed by fine-tuned encoder QE models, errors from tokenisation, transliteration, named entities https://aclanthology.org/2025.loreslm-1.33 ; Rahman et al. 2025 Sylheti-English reference-less framework, top Spearman gain +0.1083 https://arxiv.org/abs/2505.12273 . Both [V search-only]; both for natural low-resource languages, not conlangs. Back-translation/backgloss as the check is Weksa's own design (multilingual-utterance-lowering.md, prior note) and the round-trip caveats in the earlier note apply.
+
+## Gaps worth knowing exist
+- No per-metric human-correlation numbers for hedging, humour, affect, stance, persona voice; nearest is formality (Briakou 2021). COMET, BLEURT, MAUVE correlations not obtained.
+- No source on surface-form smuggling through an LLM bottleneck or its defences (vocab restriction, paraphrase invariance).
+- No source on schema/inventory growth from generation failures, or its overfitting controls.
+- Cornell, LIGHT, PersonaChat licence texts not obtained; Gutenberg ToU on bulk use not read.
+- No realiser or evaluation protocol for constructed languages from a meaning representation.
+- Several arXiv PDFs (Briakou, Zheng, STEL, LUAR/StyleDistance) fetched as binary; numbers above come from the ar5iv/arXiv-HTML versions where marked, otherwise search summaries.