Scorer coverage: words break at any non-word character; unspaced scripts read by run; near-miss chrF pairs per script #17
2026-11-09
Scorer coverage: words break at any non-word character, unspaced scripts read by run; near-miss chrF pairs per script
Soul scorer s3 Lows. words.rs no longer lists separators (_ and .) or defers to UAX #29, which splits Han, kana and Thai per character and made honest single-word glosses read as an n>=2 leak. A word is a run of letters, digits, marks and ZWNJ/ZWJ; a run of Han, kana, Thai, Lao, Khmer or Burmese is one word. Tests pin separators, marks, joiners, NFC, every range edge, and five sacrebleu 2.4.3 near-miss pairs (Ko, Tr, Fi, Th vowel-strip, Ja). 40 hand mutants killed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>