Scorer coverage: words break at any non-word character; unspaced scripts read by run; near-miss chrF pairs per script #17

Merged

weksa merged 1 commit from weksa-lowering/scorer-coverage-scripts into main

2026-11-09 09:11:05 +00:00

weksa commented

2026-11-09 07:47:25 +00:00

Member

Soul scorer s3 Lows (weksa-lowering:follow_up:scorer-coverage-scripts).

words.rs: a word is a run of letters, digits, combining marks and ZWNJ/ZWJ. Any other character ends it (: · apostrophes ＿ / | ...), plus lower-to-upper transitions. Apostrophes now split (Brokkr'ın reads brokkr ın), symmetric for glosses and corpus.
Han, kana, Thai, Lao, Khmer and Burmese write no word boundaries and the scorer holds no dictionary. A run of them is one word, so honest single-word glosses no longer share per-character bigrams with a held-out sentence. Cost: a leak in those scripts shows at n=1 and in chrF, not at n>=2. Respects form-bound-bytes-only (no word count anywhere) and equal treatment: one rule, no per-language table beyond the script ranges. A dictionary segmenter is a dependency choice for the ja-JP panel and needs a ruling.
Five non-identical pairs (Ko, Tr, Fi, Th vowel-strip, Ja) pinned against sacrebleu 2.4.3 sentence_chrf.

Tests written first (3 failed before the change). 40 hand mutants on words.rs and chrf.rs, all killed. cargo test --locked -p weksa green.

Not run: cargo-mutants (not installed), ygg-verify, full cargo clippy --all-targets: main.rs EntityCommand trips clippy::large_enum_variant on this toolchain, already on main and untouched here; --lib --tests clippy of the touched code is clean apart from that bin target.

🤖 Generated with Claude Code

Soul scorer s3 Lows (`weksa-lowering:follow_up:scorer-coverage-scripts`). 1. `words.rs`: a word is a run of letters, digits, combining marks and ZWNJ/ZWJ. Any other character ends it (`:` `·` apostrophes `＿` `/` `|` ...), plus lower-to-upper transitions. Apostrophes now split (`Brokkr'ın` reads `brokkr ın`), symmetric for glosses and corpus. 2. Han, kana, Thai, Lao, Khmer and Burmese write no word boundaries and the scorer holds no dictionary. A run of them is one word, so honest single-word glosses no longer share per-character bigrams with a held-out sentence. Cost: a leak in those scripts shows at n=1 and in chrF, not at n>=2. Respects `form-bound-bytes-only` (no word count anywhere) and equal treatment: one rule, no per-language table beyond the script ranges. A dictionary segmenter is a dependency choice for the ja-JP panel and needs a ruling. 3. Five non-identical pairs (Ko, Tr, Fi, Th vowel-strip, Ja) pinned against sacrebleu 2.4.3 sentence_chrf. Tests written first (3 failed before the change). 40 hand mutants on words.rs and chrf.rs, all killed. `cargo test --locked -p weksa` green. Not run: cargo-mutants (not installed), ygg-verify, full `cargo clippy --all-targets`: main.rs `EntityCommand` trips clippy::large_enum_variant on this toolchain, already on main and untouched here; `--lib --tests` clippy of the touched code is clean apart from that bin target. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

weksa added 1 commit

2026-11-09 07:47:25 +00:00

Scorer coverage: words break at any non-word character, unspaced scripts read by run; near-miss chrF pairs per script 5e2a4a0ebb

Soul scorer s3 Lows. words.rs no longer lists separators (_ and .) or defers to UAX #29, which splits Han, kana and Thai per character and made honest single-word glosses read as an n>=2 leak. A word is a run of letters, digits, marks and ZWNJ/ZWJ; a run of Han, kana, Thai, Lao, Khmer or Burmese is one word. Tests pin separators, marks, joiners, NFC, every range edge, and five sacrebleu 2.4.3 near-miss pairs (Ko, Tr, Fi, Th vowel-strip, Ja). 40 hand mutants killed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

weksa commented

2026-11-09 09:11:04 +00:00

Author

Member

Rerun on head 5e2a4a0ebb (merge-base = main 73ce2de): cargo test --locked -p weksa all suites ok, 0 failed. Touches scorer words and docs only, no canon. Merging.

Rerun on head 5e2a4a0ebb (merge-base = main 73ce2de): `cargo test --locked -p weksa` all suites ok, 0 failed. Touches scorer words and docs only, no canon. Merging.

weksa merged commit 96a6630fac into main

2026-11-09 09:11:05 +00:00

weksa referenced this pull request from a commit

2026-11-09 09:11:07 +00:00

Merge pull request 'Scorer coverage: words break at any non-word character; unspaced scripts read by run; near-miss chrF pairs per script' (#17) from weksa-lowering/scorer-coverage-scripts into main

weksa referenced this pull request

2026-11-09 09:11:53 +00:00

Packet carries an addressee-consent fact (schema v5, V14, V15) #14