The hand-written is_alphanumeric neighbour rule in SourceSpan::locate got
several boundary cases wrong: it accepted "I can" inside "I can't go" (an
apostrophe is not alphanumeric, so it read as a boundary even though it
binds the contraction into one word), accepted "the cafe" ending right
before a combining acute mark, accepted "run" inside "run_fast", and
falsely refused every CJK quote because every ideograph counts as
alphanumeric and CJK prose carries no spaces.
Replace the rule with real UAX #29 word segmentation from the
unicode-segmentation crate: collect the boundary byte offsets
split_word_bound_indices() reports for the source, plus source.len(), and
accept an occurrence only when both its start and end land in that set.
Delete starts_on_word_boundary/ends_on_word_boundary; the crate is the
authority now, not a hand rule.
unicode-segmentation is pinned to 1.x in the workspace
[workspace.dependencies] table (Cargo.lock gains only that one package)
and added to ghostlight-persona-projection's own Cargo.toml.
Verified against UAX #29 with new tests: contractions (straight and curly
apostrophe) are refused, a quote ending before a combining mark is
refused, an underscore-joined word is refused, a leading-CJK two-character
quote is accepted, a quote with its own leading space is accepted, a
hyphen-bounded quote is accepted, and "Stop." in "Stop.Now" is refused
(the crate glues a MidNumLet full stop between two letters into one word,
so this is the crate's called shot, not a special case carved into the
rule).
Callers (Interpreter capture, Dungeon's span_is_exact) are unchanged; the
lib and Dungeon bin suites both still pass in full (618 lib tests with the
one known PA.f77 flake, 63 Dungeon bin tests).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4fea8fdd36