Mismatch between glibc and X11 locale.alias

[serhiy-storchaka]

BPO	20087
Nosy	@malemburg, @loewis, @benjaminp, @serhiy-storchaka, @Licht-T
PRs	update locale aliases for glibc 2.24 #422 bpo-20087: Revert "make the glibc alias table take precedence over thee X11 one (#422)" #713 bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. #6708 [3.7] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (ПР-6708) #6713 [3.6] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (ПР-6708) #6714 [2.7] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708). #6717

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-17.12:12:03.181>
created_at = <Date 2013-12-28.09:29:49.864>
labels = ['3.7', 'type-bug', 'library']
title = 'Mismatch between glibc and X11 locale.alias'
updated_at = <Date 2021-02-17.12:12:03.180>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-02-17.12:12:03.180>
actor = 'lemburg'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-17.12:12:03.181>
closer = 'lemburg'
components = ['Library (Lib)']
creation = <Date 2013-12-28.09:29:49.864>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 20087
keywords = ['patch']
message_count = 33.0
messages = ['207025', '288871', '289174', '289176', '289179', '289205', '289210', '289222', '289223', '289231', '289232', '289242', '289277', '289282', '289283', '289284', '289286', '289290', '289340', '289377', '289386', '289439', '289787', '290131', '290273', '316214', '316216', '316224', '316226', '316227', '316228', '316234', '387148']
nosy_count = 6.0
nosy_names = ['lemburg', 'loewis', 'benjamin.peterson', 'Arfrever', 'serhiy.storchaka', 'licht-t']
pr_nums = ['422', '713', '6708', '6713', '6714', '6717']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20087'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[serhiy-storchaka]

The locale module uses locale alias table derived from X11 locale.alias file for mapping bare locale names without encodings to locale names with encodings. However sometimes glibc default encoding for a locale differs from that used in X11 locale.alias.

Here is full differences table:

             GLibc                 X11 locale.alias

az_az az_AZ.UTF-8 az_AZ.ISO8859-9E
ca_ad ca_AD.ISO8859-15 ca_AD.ISO8859-1
ca_fr ca_FR.ISO8859-15 ca_FR.ISO8859-1
ca_it ca_IT.ISO8859-15 ca_IT.ISO8859-1
cy_gb cy_GB.ISO8859-14 cy_GB.ISO8859-1
en_in en_IN.UTF-8 en_IN.ISO8859-1
et_ee et_EE.ISO8859-1 et_EE.ISO8859-15
fi_fi fi_FI.ISO8859-1 fi_FI.ISO8859-15
gd_gb gd_GB.ISO8859-15 gd_GB.ISO8859-1
hi_in hi_IN.UTF-8 hi_IN.ISCII-DEV
iu_ca iu_CA.UTF-8 iu_CA.NUNACOM-8
iw_il iw_IL.ISO8859-8 he_IL.ISO8859-8
ka_ge ka_GE.GEORGIAN_PS ka_GE.GEORGIAN-ACADEMY
lo_la lo_LA.UTF-8 lo_LA.MULELAO-1
mi_nz mi_NZ.ISO8859-13 mi_NZ.ISO8859-1
nr_za nr_ZA.UTF-8 nr_ZA.ISO8859-1
nso_za nso_ZA.UTF-8 nso_ZA.ISO8859-15
ru_ru ru_RU.ISO8859-5 ru_RU.UTF-8
rw_rw rw_RW.UTF-8 rw_RW.ISO8859-1
sq_al sq_AL.ISO8859-1 sq_AL.ISO8859-2
ss_za ss_ZA.UTF-8 ss_ZA.ISO8859-1
ta_in ta_IN.UTF-8 ta_IN.TSCII-0
tg_tj tg_TJ.KOI8_T tg_TJ.KOI8-C
th_th th_TH.TIS_620 th_TH.ISO8859-11
tn_za tn_ZA.UTF-8 tn_ZA.ISO8859-15
ts_za ts_ZA.UTF-8 ts_ZA.ISO8859-1
tt_ru tt_RU.UTF-8 tt_RU.TATAR-CYR
ur_pk ur_PK.UTF-8 ur_PK.CP1256
uz_uz uz_UZ.ISO8859-1 uz_UZ.UTF-8
uz_uz@cyrillic uz_UZ.UTF-8@cyrillic uz_UZ.UTF-8
vi_vn vi_VN.UTF-8 vi_VN.TCVN
zh_cn zh_CN.GB2312 zh_CN.gb2312
zh_tw zh_TW.BIG5 zh_TW.big5
zh_tw.euctw zh_TW.EUC_TW zh_TW.eucTW

For example with the en_IN encoding:

>>> import locale, _locale
>>> _locale.setlocale(locale.LC_CTYPE)
'en_IN'
>>> locale.getlocale()
('en_IN', 'ISO8859-1')
>>> locale.nl_langinfo(locale.CODESET)
'UTF-8'
>>> locale.setlocale(locale.LC_CTYPE, locale.getlocale())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/locale.py", line 592, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

[serhiy-storchaka]

Needed a test for few common locales (en_IN, ru_RU) and maybe for unusual locales (uz_uz, uz_uz@cyrillic).

I would prefer to have a separate issue that updates the aliases table to glibc 2.24.

[malemburg]

I agree that it's reasonable to have glibc's aliases override
the X.org ones, but this patch makes some pretty significant changes to Python's default assumptions with respect to default encodings for several locales.

While some changes obviously make sense (e.g. 'ca_AD.ISO8859-1' to 'ca_AD.ISO8859-15'), others are less clear (e.g. 'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14' or 'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T' or several of the moves from ISO encodings to UTF-8). Is there some reference for why glibc chose different values than X.org for these ?

I also don't understand why some "xx.utf-8" locale mappings were removed - I don't think we should remove those, unless they are no lot needed due to some other logic implying these mappings.

Since these are major changes, we need an appropriate warning in the NEWS file (and the "What's New" document), an update of the top comment (under "### Database") to mention that the glibc database takes precedence and where to find it,

[serhiy-storchaka]

'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14'

Looks as just fixing an error. The default West-European ISO8859-1 is changed to Celtic cy_GB.ISO8859-14. This looks better option for Welsh.

'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T'

KOI8-C is not supported by Python, but KOI8-T is supported. I don't know what KOI8-C means, there are several rarely used incompatible encodings with this name.

I also don't understand why some "xx.utf-8" locale mappings were removed - I don't think we should remove those, unless they are no lot needed due to some other logic implying these mappings.

The aliases table is a table of exceptions. Removed entries no longer are exceptional.

[malemburg]

On 07.03.2017 18:23, Serhiy Storchaka wrote:

Serhiy Storchaka added the comment:

> 'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14'

Looks as just fixing an error. The default West-European ISO8859-1 is changed to Celtic cy_GB.ISO8859-14. This looks better option for Welsh.

> 'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T'

KOI8-C is not supported by Python, but KOI8-T is supported. I don't know what KOI8-C means, there are several rarely used incompatible encodings with this name.

While all this may make sense, I'm missing some more reasoning
behind the differences between X.org and glibc.

This change also looks strange:

-    'ka_ge':                                'ka_GE.GEORGIAN-ACADEMY',
+    'ka_ge':                                'ka_GE.GEORGIAN_PS',
     'ka_ge.georgianacademy':                'ka_GE.GEORGIAN-ACADEMY',
     'ka_ge.georgianps':                     'ka_GE.GEORGIAN-PS',
     'ka_ge.georgianrs':                     'ka_GE.GEORGIAN-ACADEMY',

Why is GEORGIAN_PS written with an underscore whereas the other
mappings use dashes ?

Or this one:

• 'fi_fi': 'fi_FI.ISO8859-15',
  + 'fi_fi': 'fi_FI.ISO8859-1',

Why would a locale switch away from an encoding having
the Euro sign to one without it ?

Or why is this latin variant removed:

• 'nan_tw@latin': 'nan_TW.UTF-8@latin',

Why should Russians switch back to ISO ?

• 'ru_ru': 'ru_RU.UTF-8',
  + 'ru_ru': 'ru_RU.ISO8859-5',

or from ISO to KOI ?

• 'russian': 'ru_RU.ISO8859-5',
  + 'russian': 'ru_RU.KOI8-R',

The more I look at these changes, the more I believe we
should not simply take everything we find in the files
for granted. They obviously both have bugs.

> I also don't understand why some "xx.utf-8" locale mappings were removed - I don't think we should remove those, unless they are no longer needed due to some other logic implying these mappings.

The aliases table is a table of exceptions. Removed entries no longer are exceptional.

It's not a table of exceptions, it's a table mapping commonly
used locale settings to ones which the lib C understands :-)

But regardless, I checked the code and it is already
smart enough to convert lib C incompatible spellings such
as "utf8" to "UTF-8", so these entries can indeed be
removed, but only if the locale is otherwise listed.

In some cases, it's probably better to drop the ".utf8"
to have more generic mappings, e.g.

+ 'bhb_in.utf8': 'bhb_IN.UTF-8',

or

 'de_li.utf8':                           'de_LI.UTF-8',

though I'd expect that mapping to be:

 'de_li':                           'de_LI.ISO8859-1',

as for all other "de" entries.

[benjaminp]

Why is the X11 locale alias map used at all? It seems like it can only create confusion with libc.

[serhiy-storchaka]

Not all platforms use glibc 2.24 as libc.

Ideally most of entries should even not exist. We should ask libc for the default encoding if it is not included in the locale name. The aliases table should be used only for mapping commonly used but unsupported by libc locales to supported by libc locales.

[malemburg]

On 08.03.2017 08:20, Serhiy Storchaka wrote:

Serhiy Storchaka added the comment:

Not all platforms use glibc 2.24 as libc.

True. Many don't even use glibc.

Ideally most of entries should even not exist. We should ask libc for the default encoding if it is not included in the locale name. The aliases table should be used only for mapping commonly used but unsupported by libc locales to supported by libc locales.

I think you have a wrong understanding of what this alias table
is used for: we need it to determine the lib C compatible locale
name without using lib C APIs such as setlocale(), since these are
not thread safe and have side-effects for the whole process.

The alias table is there to avoid having to go to the lib C
to ask it indirectly for more details. Unfortunately, there are
no cross-platform lib C APIs which would allow querying these
details without also changing the local settings of the process.

I know that Python still plays the usual "save current locale,
run setlocale(), revert to previous locale" trick in a couple
of places and this works if Python is the only thread running,
but it doesn't when embedded into other applications.

Regarding the patch: we cannot simply use the output from the
script to set new values. The changes have to be manually
reviewed as well.

E.g. this entry in the table is clearly a typo:

'en_zw.utf8':                           'en_ZS.UTF-8',

(it should read en_ZW.UTF-8)

This entry appears wrong as well:

'eo':                                   'eo_XX.ISO8859-3',

(XX is not a valid country ISO code)

How should we go about this ? Mark all the problems in the PR ?

[serhiy-storchaka]

The problem is that that table can get incorrect result for non-Linux platforms (or for Linux with old glibc).

[malemburg]

On 08.03.2017 07:27, Benjamin Peterson wrote:

Why is the X11 locale alias map used at all? It seems like it can only create confusion with libc.

Because it was the only such maintained mapping available at the
time. It's also used for the X.org system, which has a rather strong
focus on user interfaces where locale matter a lot, unlike
the lib C :-)

[malemburg]

On 08.03.2017 10:37, Serhiy Storchaka wrote:

The problem is that that table can get incorrect result for non-Linux platforms (or for Linux with old glibc).

Sure, it's a best effort approach.

Also note that on today's systems you often don't have the full set of
locales available anymore - instead these have to either be installed
separately or generated on the target system.

Our locale database works on all these system, regardless of
what's installed or not.

[malemburg]

Why was the PR merged while we were still discussing it ?