Skip to content

Make GenerateConfusables generate NFKD instead of NFKC #1205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 8, 2025
Merged

Make GenerateConfusables generate NFKD instead of NFKC #1205

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
000c8bd
Make GenerateConfusables generate NFKD instead of NFKC
roozbehp Sep 12, 2025
cfdd481
Fix a bunch of variable names and comments
roozbehp Sep 12, 2025
c865013
Use singleton pattern
roozbehp Oct 7, 2025
7ce2677
Fix Java style
roozbehp Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions unicodetools/data/security/dev/IdentifierStatus.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# IdentifierStatus.txt
# Date: 2025-08-01, 18:11:48 GMT
# Date: 2025-09-12, 03:25:00 GMT
# © 2025 Unicode®, Inc.
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
# For terms of use and license, see https://www.unicode.org/terms_of_use.html
#
# Unicode Security Mechanisms for UTS #39
# Version: 17.0.0
# Version: 18.0.0
#
# For documentation and usage, see https://www.unicode.org/reports/tr39
#
Expand Down
4 changes: 2 additions & 2 deletions unicodetools/data/security/dev/IdentifierType.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# IdentifierType.txt
# Date: 2025-08-04, 21:58:43 GMT
# Date: 2025-09-12, 03:24:49 GMT
# © 2025 Unicode®, Inc.
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
# For terms of use and license, see https://www.unicode.org/terms_of_use.html
#
# Unicode Security Mechanisms for UTS #39
# Version: 17.0.0
# Version: 18.0.0
#
# For documentation and usage, see https://www.unicode.org/reports/tr39
#
Expand Down
190 changes: 94 additions & 96 deletions unicodetools/data/security/dev/confusables.txt
View file
Open in desktop

Large diffs are not rendered by default.

586 changes: 292 additions & 294 deletions unicodetools/data/security/dev/confusablesSummary.txt
View file
Open in desktop

Large diffs are not rendered by default.

74 changes: 40 additions & 34 deletions unicodetools/data/security/dev/data/confusablesSummaryIdentifier.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# confusablesSummaryIdentifier.txt
# Date: 2025-07-22, 05:49:37 GMT
# Date: 2025-09-12, 03:24:49 GMT
# © 2025 Unicode®, Inc.
# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries.
# For terms of use and license, see https://www.unicode.org/terms_of_use.html
#
# Unicode Security Mechanisms for UTS #39
# Version: 17.0.0
# Version: 18.0.0
#
# For documentation and usage, see https://www.unicode.org/reports/tr39
#
Expand Down Expand Up @@ -69,21 +69,25 @@
← (‎ І ‎) 0406 CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I # →I→
← (‎ Ӏ ‎) 04C0 CYRILLIC LETTER PALOCHKA # →I→

# lٕ 1ٕ إ
# lٴ lٔ ٴl 1ٔ 1ٴ أ اٴ ٴ1 ٴا ٲ أ
(‎ 1ٔ ‎) 0031 0654 DIGIT ONE, ARABIC HAMZA ABOVE
← (‎ lٴ ‎) 006C 0674 LATIN SMALL LETTER L, ARABIC LETTER HIGH HAMZA # →‎اٴ‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ lٔ ‎) 006C 0654 LATIN SMALL LETTER L, ARABIC HAMZA ABOVE # →‎أ‎→
← (‎ ٴl ‎) 0674 006C ARABIC LETTER HIGH HAMZA, LATIN SMALL LETTER L # →‎ٴا‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ 1ٴ ‎) 0031 0674 DIGIT ONE, ARABIC LETTER HIGH HAMZA # →‎اٴ‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ أ ‎) 0627 0654 ARABIC LETTER ALEF, ARABIC HAMZA ABOVE
← (‎ اٴ ‎) 0627 0674 ARABIC LETTER ALEF, ARABIC LETTER HIGH HAMZA # →‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ ٴ1 ‎) 0674 0031 ARABIC LETTER HIGH HAMZA, DIGIT ONE # →‎ٴا‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ ٴا ‎) 0674 0627 ARABIC LETTER HIGH HAMZA, ARABIC LETTER ALEF # →‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ ٲ ‎) 0672 ARABIC LETTER ALEF WITH WAVY HAMZA ABOVE # →‎أ‎→→‎ٵ‎→→‎ﺃ‎→→‎أ‎→
← (‎ أ ‎) 0623 ARABIC LETTER ALEF WITH HAMZA ABOVE # →‎ٵ‎→→‎ﺃ‎→→‎أ‎→

# lٕ 1ٕ إ إ
(‎ 1ٕ ‎) 0031 0655 DIGIT ONE, ARABIC HAMZA BELOW
← (‎ lٕ ‎) 006C 0655 LATIN SMALL LETTER L, ARABIC HAMZA BELOW # →‎اٟ‎→
← (‎ إ ‎) 0627 0655 ARABIC LETTER ALEF, ARABIC HAMZA BELOW # →lٕ→→‎اٟ‎→
← (‎ إ ‎) 0625 ARABIC LETTER ALEF WITH HAMZA BELOW # →‎ٳ‎→→‎اٟ‎→

# lٴ ٴl 1ٴ اٴ ٴ1 ٴا ٲ أ
(‎ 1ٴ ‎) 0031 0674 DIGIT ONE, ARABIC LETTER HIGH HAMZA
← (‎ lٴ ‎) 006C 0674 LATIN SMALL LETTER L, ARABIC LETTER HIGH HAMZA # →‎اٴ‎→
← (‎ ٴl ‎) 0674 006C ARABIC LETTER HIGH HAMZA, LATIN SMALL LETTER L # →‎ٴا‎→→‎ٵ‎→→‎اٴ‎→
← (‎ اٴ ‎) 0627 0674 ARABIC LETTER ALEF, ARABIC LETTER HIGH HAMZA
← (‎ ٴ1 ‎) 0674 0031 ARABIC LETTER HIGH HAMZA, DIGIT ONE # →‎ٴا‎→→‎ٵ‎→→‎اٴ‎→
← (‎ ٴا ‎) 0674 0627 ARABIC LETTER HIGH HAMZA, ARABIC LETTER ALEF # →‎ٵ‎→→‎اٴ‎→
← (‎ ٲ ‎) 0672 ARABIC LETTER ALEF WITH WAVY HAMZA ABOVE # →‎أ‎→→‎ٵ‎→→‎اٴ‎→
← (‎ أ ‎) 0623 ARABIC LETTER ALEF WITH HAMZA ABOVE # →‎ٵ‎→→‎اٴ‎→

# 3 ३ ૩ З Ӡ
(‎ 3 ‎) 0033 DIGIT THREE
← (‎ ३ ‎) 0969 DEVANAGARI DIGIT THREE # →૩→
Expand Down Expand Up @@ -361,6 +365,12 @@
(‎ ô ‎) 006F 0302 LATIN SMALL LETTER O, COMBINING CIRCUMFLEX ACCENT
← (‎ ۿ ‎) 06FF ARABIC LETTER HEH WITH INVERTED V # →‎ھٛ‎→

# oٔ هٔ ۀ ۂ
(‎ oٔ ‎) 006F 0654 LATIN SMALL LETTER O, ARABIC HAMZA ABOVE
← (‎ هٔ ‎) 0647 0654 ARABIC LETTER HEH, ARABIC HAMZA ABOVE # →‎ۀ‎→
← (‎ ۀ ‎) 06D5 0654 ARABIC LETTER AE, ARABIC HAMZA ABOVE
← (‎ ۂ ‎) 06C2 ARABIC LETTER HEH GOAL WITH HAMZA ABOVE # →‎ﮤ‎→→‎ۀ‎→

# oာ ဝာ တ თ
(‎ oာ ‎) 006F 102C LATIN SMALL LETTER O, MYANMAR VOWEL SIGN AA
← (‎ ဝာ ‎) 101D 102C MYANMAR LETTER WA, MYANMAR VOWEL SIGN AA
Expand Down Expand Up @@ -610,16 +620,21 @@
(‎ ձ ‎) 0571 ARMENIAN SMALL LETTER JA
← (‎ ኔ ‎) 1294 ETHIOPIC SYLLABLE NEE

# وٴ ٴو ؤ
# وٴ ؤ ٴو ؤ
(‎ ؤ ‎) 0624 ARABIC LETTER WAW WITH HAMZA ABOVE
← (‎ وٴ ‎) 0648 0674 ARABIC LETTER WAW, ARABIC LETTER HIGH HAMZA # →‎ٶ‎→
← (‎ ؤ ‎) 0648 0654 ARABIC LETTER WAW, ARABIC HAMZA ABOVE # →‎ﺅ‎→→‎ٶ‎→
← (‎ ٴو ‎) 0674 0648 ARABIC LETTER HIGH HAMZA, ARABIC LETTER WAW # →‎ٶ‎→

# ىٴ يٴ ٴى ئ
# ىٴ ىٔ ئ يٴ ٴى ۓ ࢨ ئ
(‎ ئ ‎) 0626 ARABIC LETTER YEH WITH HAMZA ABOVE
← (‎ ىٴ ‎) 0649 0674 ARABIC LETTER ALEF MAKSURA, ARABIC LETTER HIGH HAMZA # →‎يٴ‎→→‎ٸ‎→
← (‎ ىٔ ‎) 0649 0654 ARABIC LETTER ALEF MAKSURA, ARABIC HAMZA ABOVE # →‎ئ‎→→‎ﺉ‎→→‎ٸ‎→
← (‎ ئ ‎) 064A 0654 ARABIC LETTER YEH, ARABIC HAMZA ABOVE # →‎ﺉ‎→→‎ٸ‎→
← (‎ يٴ ‎) 064A 0674 ARABIC LETTER YEH, ARABIC LETTER HIGH HAMZA # →‎ٸ‎→
← (‎ ٴى ‎) 0674 0649 ARABIC LETTER HIGH HAMZA, ARABIC LETTER ALEF MAKSURA # →‎ٸ‎→
← (‎ ۓ ‎) 06D2 0654 ARABIC LETTER YEH BARREE, ARABIC HAMZA ABOVE # →‎ىٔ‎→→‎ئ‎→→‎ﺉ‎→→‎ٸ‎→
← (‎ ࢨ ‎) 08A8 ARABIC LETTER YEH WITH TWO DOTS BELOW AND HAMZA ABOVE # →‎ئ‎→→‎ﺉ‎→→‎ٸ‎→

# ت ٺ
(‎ ت ‎) 062A ARABIC LETTER TEH
Expand Down Expand Up @@ -715,11 +730,6 @@
(‎ ٹ ‎) 0679 ARABIC LETTER TTEH
← (‎ ڻ ‎) 06BB ARABIC LETTER RNOON # →‎ﮢ‎→→‎ﭨ‎→

# ىٔ ئ ࢨ
(‎ ىٔ ‎) 0649 0654 ARABIC LETTER ALEF MAKSURA, ARABIC HAMZA ABOVE
← (‎ ئ ‎) 064A 0654 ARABIC LETTER YEH, ARABIC HAMZA ABOVE
← (‎ ࢨ ‎) 08A8 ARABIC LETTER YEH WITH TWO DOTS BELOW AND HAMZA ABOVE # →‎ئ‎→

# پ̆ ࢾ
(‎ پ̆ ‎) 067E 0306 ARABIC LETTER PEH, COMBINING BREVE
← (‎ ࢾ ‎) 08BE ARABIC LETTER PEH WITH SMALL V # →‎پٚ‎→
Expand Down Expand Up @@ -761,10 +771,6 @@
(‎ چ̆ ‎) 0686 0306 ARABIC LETTER TCHEH, COMBINING BREVE
← (‎ ࣁ ‎) 08C1 ARABIC LETTER TCHEH WITH SMALL V # →‎چٚ‎→

# ۀ ۂ
(‎ ۀ ‎) 06C0 ARABIC LETTER HEH WITH YEH ABOVE
← (‎ ۂ ‎) 06C2 ARABIC LETTER HEH GOAL WITH HAMZA ABOVE # →‎ﮤ‎→

# ݧ ࢩ
(‎ ݧ ‎) 0767 ARABIC LETTER NOON WITH TWO DOTS BELOW
← (‎ ࢩ ‎) 08A9 ARABIC LETTER YEH WITH TWO DOTS BELOW AND DOT ABOVE # →‎ݔ‎→
Expand Down Expand Up @@ -823,10 +829,10 @@
(‎ उ ‎) 0909 DEVANAGARI LETTER U
← (‎ ਤ ‎) 0A24 GURMUKHI LETTER TA

# एॅ ए̆ ऍ
# ए̆ एॅ ऍ
(‎ ऍ ‎) 090D DEVANAGARI LETTER CANDRA E
← (‎ एॅ ‎) 090F 0945 DEVANAGARI LETTER E, DEVANAGARI VOWEL SIGN CANDRA E
← (‎ ए̆ ‎) 090F 0306 DEVANAGARI LETTER E, COMBINING BREVE # →एॅ→
← (‎ एॅ ‎) 090F 0945 DEVANAGARI LETTER E, DEVANAGARI VOWEL SIGN CANDRA E

# एॆ ऎ
(‎ ऎ ‎) 090E DEVANAGARI LETTER SHORT E
Expand Down Expand Up @@ -956,7 +962,7 @@
(‎ র ‎) 09B0 BENGALI LETTER RA
← (‎ ৰ ‎) 09F0 BENGALI LETTER RA WITH MIDDLE DIAGONAL

# ਅੈ ਅै ਐ
# ਅै ਅੈ ਐ
(‎ ਅै ‎) 0A05 0948 GURMUKHI LETTER A, DEVANAGARI VOWEL SIGN AI
← (‎ ਅੈ ‎) 0A05 0A48 GURMUKHI LETTER A, GURMUKHI VOWEL SIGN AI
← (‎ ਐ ‎) 0A10 GURMUKHI LETTER AI # →ਅੈ→
Expand Down Expand Up @@ -1086,15 +1092,15 @@
(‎ ெள ‎) 0BC6 0BB3 TAMIL VOWEL SIGN E, TAMIL LETTER LLA
← (‎ ௌ ‎) 0BCC TAMIL VOWEL SIGN AU

# െെ ெெ ൈ
# ெெ െെ ൈ
(‎ ெெ ‎) 0BC6 0BC6 TAMIL VOWEL SIGN E, TAMIL VOWEL SIGN E
← (‎ െെ ‎) 0D46 0D46 MALAYALAM VOWEL SIGN E, MALAYALAM VOWEL SIGN E
← (‎ ൈ ‎) 0D48 MALAYALAM VOWEL SIGN AI # →െെ→

# എെ ெഎ എெ െഎ ഐ
# ெഎ എெ എെ െഎ ഐ
(‎ ெഎ ‎) 0BC6 0D0E TAMIL VOWEL SIGN E, MALAYALAM LETTER E
← (‎ എെ ‎) 0D0E 0D46 MALAYALAM LETTER E, MALAYALAM VOWEL SIGN E # →ഐ→→െഎ→
← (‎ എெ ‎) 0D0E 0BC6 MALAYALAM LETTER E, TAMIL VOWEL SIGN E # →എെ→→ഐ→→െഎ→
← (‎ എെ ‎) 0D0E 0D46 MALAYALAM LETTER E, MALAYALAM VOWEL SIGN E # →ഐ→→െഎ→
← (‎ െഎ ‎) 0D46 0D0E MALAYALAM VOWEL SIGN E, MALAYALAM LETTER E
← (‎ ഐ ‎) 0D10 MALAYALAM LETTER AI # →െഎ→

Expand Down Expand Up @@ -1260,7 +1266,7 @@
(‎ ര് ‎) 0D30 0D4D MALAYALAM LETTER RA, MALAYALAM SIGN VIRAMA
← (‎ ർ ‎) 0D7C MALAYALAM LETTER CHILLU RR # →൪→

# ဂာ രာ က
# രာ ဂာ က
(‎ രာ ‎) 0D30 102C MALAYALAM LETTER RA, MYANMAR VOWEL SIGN AA
← (‎ ဂာ ‎) 1002 102C MYANMAR LETTER GA, MYANMAR VOWEL SIGN AA
← (‎ က ‎) 1000 MYANMAR LETTER KA # →ဂာ→
Expand All @@ -1271,7 +1277,7 @@
← (‎ က္က ‎) 1000 1039 1000 MYANMAR LETTER KA, MYANMAR SIGN VIRAMA, MYANMAR LETTER KA
← (‎ ဣ ‎) 1023 MYANMAR LETTER I # →က္က→

# ဂှ രှ ႁ
# രှ ဂှ ႁ
(‎ രှ ‎) 0D30 103E MALAYALAM LETTER RA, MYANMAR CONSONANT SIGN MEDIAL HA
← (‎ ဂှ ‎) 1002 103E MYANMAR LETTER GA, MYANMAR CONSONANT SIGN MEDIAL HA
← (‎ ႁ ‎) 1081 MYANMAR LETTER SHAN HA # →ဂှ→
Expand Down Expand Up @@ -1446,7 +1452,7 @@
(‎ သြ ‎) 101E 103C MYANMAR LETTER SA, MYANMAR CONSONANT SIGN MEDIAL RA
← (‎ ဩ ‎) 1029 MYANMAR LETTER O

# သြော် သြେာ် ဩော် ဪ
# သြେာ် သြော် ဩော် ဪ
(‎ သြେာ် ‎) 101E 103C 0B47 102C 103A MYANMAR LETTER SA, MYANMAR CONSONANT SIGN MEDIAL RA, ORIYA VOWEL SIGN E, MYANMAR VOWEL SIGN AA, MYANMAR SIGN ASAT
← (‎ သြော် ‎) 101E 103C 1031 102C 103A MYANMAR LETTER SA, MYANMAR CONSONANT SIGN MEDIAL RA, MYANMAR VOWEL SIGN E, MYANMAR VOWEL SIGN AA, MYANMAR SIGN ASAT # →ဩော်→
← (‎ ဩော် ‎) 1029 1031 102C 103A MYANMAR LETTER O, MYANMAR VOWEL SIGN E, MYANMAR VOWEL SIGN AA, MYANMAR SIGN ASAT
Expand All @@ -1464,5 +1470,5 @@
(‎ へ ‎) 3078 HIRAGANA LETTER HE
← (‎ ヘ ‎) 30D8 KATAKANA LETTER HE

# total : 518
# total : 527

Loading
Loading