Aetheria/docs/research/tmp-fe0f-ligatures.md

TMP strips FE0F before ligature matching: why 13 emoji sequences stayed three pieces

Raised by question aetheria-release❓sequences-fe0f-zwj-ligature (cut emoji-sequences); answered by ruling aetheria-release:ruling:fe0f-less-ligatures-in-font-build. Diagnosis by Eyes (eyes-aeth-tmp-fe0f), 2026-10-12, read-only, committed verbatim below.

TMP = P = C:\aeth-wt\emoji-sequences\Library\PackageCache\com.unity.ugui@b996e7548785\Runtime\TMP (com.unity.ugui 2.0.0, source "builtin" per Packages/packages-lock.json:239-242). P\TextMeshProUGUI.cs is byte-identical (diff 0 lines) to the editor install copy C:\Program Files\Unity\Hub\Editor\6000.3.24f1\Editor\Data\Resources\PackageManager\BuiltInPackages\com.unity.ugui, so this is stock, unpatched.

1. Lookup: FE0F is stripped from the processing array BEFORE ligature matching

TextMeshProUGUI.cs (TextMeshPro.cs has the same code at 1718-1731 / 1737-1798):

• 1929: nextCharacter = textProcessingArray[i+1].unicode.
• 1932: emoji fallback route (IsEmoji(base) && next==FE0F, or presentation form).
• 2030-2047 VARIATION SELECTOR region: if nextCharacter is FE00-FE0F or E0100-E01EF: GetGlyphVariantIndex(base, vs); then textProcessingArray[i + 1].unicode = 0x1A; i += 1; (2044-2045). FE0F is overwritten with 0x1A (consumed marker; skipped later at TMP_Text.cs:2749) and i advances onto it.
• 2049-2120 LIGATURES region, runs after, only if ligature (2050, liga in m_ActiveFontFeatures, 1824). Lookup by first glyph: m_LigatureSubstitutionRecordLookup.TryGetValue(character.glyphIndex) (2050). Match loop 2063-2093: idx = i+1; for each remaining component, codepoint = array[idx].unicode; glyphIndex = GetGlyphIndex(codepoint); if glyph != expected AND codepoint is ignorable (TMP_TextParsingUtilities.IsIgnorableForLigature, TMP_TextParsingUtilities.cs:172-178: ZWJ, FE00-FE0F, E0100-E01EF) it is skipped (2076-2080); exact glyph match advances (2083-2088); else mismatch (2091). Trailing ignorable consumed at 2099-2100.
• Consequence: for base FE0F ZWJ second, the array seen by the matcher is [base(i), 0x1A(FE0F), ZWJ, second]. i already points at the 0x1A slot, so idx starts at ZWJ. A record whose component[1] is FE0F: ZWJ is not the expected glyph but is ignorable -> skipped; then second vs expected FE0F glyph -> mismatch -> no substitution. FE0F can never match because it was replaced before the match and skipped over by i+=1. Hands' hypothesis (FE0F dropped before matching) is confirmed by this source. A skip rule exists for extra ignorables but nothing inserts a missing expected component.
• ZWJ: no special handling beyond being ignorable-skippable (2076); 0x200D appears in TMP_Text.cs:2397,3070 only as the <zwj>-style tag replacement. FontAsset's synthesized-ZWJ line is commented out (TMP_FontAsset.cs:1102-1103).

2. Twemoji.ttf GSUB (fontTools 4.66.1; script gsub.py beside this file)

• Tables: COLR CPAL GSUB cmap glyf ...; GSUB has one feature, ccmp; one lookup (LigatureSubst) with 2582 records; component-count histogram {2:941, 3:81, 4:450, 5:606, 6:12, 7:269, 8:128, 10:95}. Glyphs uniFE0F gid 12, uni200D gid 13, uni20E3 present in cmap.
• Records are the sequence's glyphs as keyed in Twemoji, which differ per family: keycap = [numbersign, uni20E3] (NO FE0F); eye in speech bubble = [u1F441, uni200D, u1F5E8] (NO FE0F); but 15 records carry FE0F as component[1]: 26D3 FE0F 200D 1F4A5; 26F9 FE0F 200D 2640/2642 FE0F; 2764 FE0F 200D 1F525 / 1FA79; 1F3CB, 1F3CC, 1F574, 1F575 each FE0F 200D 2640/2642 FE0F; 1F3F3 FE0F 200D 26A7 FE0F; 1F3F3 FE0F 200D 1F308.
• Against tools\emoji\emoji-sequences.txt (2760 lines): 26 corpus lines have FE0F at index 1; exactly 13 are an exact record (these are the 13 failing: heart-on-fire 2764 FE0F 200D 1F525, mending heart 2764 FE0F 200D 1FA79, 1F575 FE0F 200D 2642/2640 FE0F, 1F3CC FE0F 200D 2642/2640 FE0F, 26F9 FE0F 200D 2642/2640 FE0F, 1F3CB FE0F 200D 2642/2640 FE0F, 26D3 FE0F 200D 1F4A5, 1F3F3 FE0F 200D 1F308, 1F3F3 FE0F 200D 26A7 FE0F); the other 13 are the 12 keycaps (stripped form 23 20E3 etc. IS a record, so they collapse) plus 1F441 FE0F 200D 1F5E8 FE0F (record has no FE0F at index 1; passes). The 13 failing in check4.log (grep "^sequence", 13 lines) equal this set exactly. 1F574 FE0F 200D 2640/2642 FE0F records exist but are not in the corpus.
• Correction to the question text: keycap records are [base, 20E3] with no FE0F, not [FE0F, 20E3]; keycaps collapse because TMP's FE0F strip happens to match Twemoji's FE0F-less keying. The cause is specific to records that keep FE0F as a middle component, not specific to ZWJ.
• No FE0F-less variant of the 13 exists in the font: 2764 has only 2 ligature records (both FE0F-ful); the first-glyph record lists for the 13 bases contain only FE0F-ful (and skin-tone) forms.

3. Emoji.asset

• Assets\Fonts\Emoji\Emoji.asset (5791 bytes): m_AtlasPopulationMode 1 (dynamic), m_GlyphTable [], m_CharacterTable [], m_FontFeatureTable.m_LigatureSubstitutionRecords [] (Emoji.asset lines ~131-136), m_GetFontFeatures 1, m_ShouldReimportFontFeatures 0, m_ClearDynamicDataOnBuild 1; EmojiCheck ends with ClearFontAssetData(true) (EmojiCheck.cs). So there are NO serialized records to inspect; records are fetched at runtime from the GSUB by FontEngine.GetLigatureSubstitutionRecords for newly added glyphs (TMP_FontAsset.cs:3124-3133, 3113-3116, AddLigatureSubstitutionRecords 3136-3166, lookup keyed by first component glyph 3150-3163). Whether the dynamic asset holds these 13 at runtime was not probed; the 2540 other sequences that exactly equal a record collapse, so records are being imported. No evidence of a component-count cutoff: 8-component family records (e.g. 1F468 200D 2764 FE0F 200D 1F48B 200D 1F468) are not in the failing set. (Native FontEngine not read; "no truncation" is inferred from pass/fail pattern.)

4. What would make them match (options as facts; Self decides)

Matcher requires, for component FE0F, a FE0F in the array after the base; the VS block (2030-2047) removes it first whenever next is FE00-FE0F. So a match needs either a. records without FE0F at index 1: [base, ZWJ, second(, FE0F)] -> same ligature glyph, added to the font's GSUB (fetch_emoji_font.py / Linux container, option font-add-fe0f-less-records) or added to Emoji.asset's m_LigatureSubstitutionRecords by hand (dynamic asset: ClearFontAssetData clears the table, 3537-3539, and refetch is from the TTF, so manual records would not persist unless the asset or TTF carries them); the trailing FE0F of 1F575/1F3CC/26F9/1F3CB variants is optional because 2099-2100 consume one trailing ignorable and a trailing FE0F component (index 4) would still match as a literal; or b. text preprocessing that gives TMP a form its matcher can match (strip the FE0F at index 1 of those 13 before the text reaches TMP; or ITextPreProcessor, P\ITextPreProcessor.cs exists); or c. patching the VS block so it does not consume FE0F when a ligature record expects it (needs a package fork; ugui is "builtin" and stock-identical to the editor install); or d. accept the 13. There is no TMP setting that changes this path (emojiFallbackSupport only governs the font route at 1932). Note a pure text-side strip (b) leaves base+ZWJ+second, which matches only if FE0F-less records exist (a) — Twemoji has none for these 13. So (b) alone does not work with the current TTF; (a) or (c) is required, (b)+(a) or (c) alone fixes it. Mind reads: fetched cut_report cut-emoji-sequences.h2 and question sequences-fe0f-zwj-ligature (view only, kinds CutReport/Question; first calls failed on arg shape).