Skip to content

Make test_localization tests execute on windows targets #46597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
smarie opened this issue Apr 1, 2022 · 1 comment
Open

Make test_localization tests execute on windows targets #46597

smarie opened this issue Apr 1, 2022 · 1 comment
Labels
Localization Internationalization of data Testing pandas testing functions or related to the test suite Windows Windows OS

Comments

@smarie
Copy link
Contributor

smarie commented Apr 1, 2022 •
edited
Loading

EDIT: See next post, this original issue about the get_locale returning an empty list on windows, has been extended to a broader one.

Reproducible Example

On a windows platform

from pandas._config.localization import get_locales

print(type(get_locales()))  # NoneType

Issue Description

On windows machines locale -a does not exist and therefore get_locales() returns None

This may be an issue for locale-related test coverage on windows CI runners, because @_skip_if_only_one_locale is always True.

I discovered this when I tried to execute test_set_locale on my machine and found out that it was not executing at all (all skipped!)

Expected Behavior

A list of locale available should be returned. The solution might not be easy to find however :(
See https://stackoverflow.com/questions/19709026/how-can-i-list-all-available-windows-locales-in-python-console

Workaround: I think that we can avoid fixing this, if we change @_skip_if_only_one_locale so that on windows runners it is less strict.

Installed Versions

Replace this line with the output of pd.show_versions()
@smarie smarie added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 1, 2022
@smarie
Copy link
Contributor Author

smarie commented Apr 1, 2022

Oh I see that even when commenting out the decorator, the whole test_localization module is marked as skipped still.

pytestmark = pytest.mark.skipif(
    is_platform_windows() or not _all_locales, reason="Need non-Windows and locales"
)

So nevermind, the issue is broader and could be renamed "Make test_localization tests execute on windows targets"

@smarie smarie changed the title BUG: get_locales returns an empty list on windows platforms, so @_skip_if_only_one_locale is always True Make test_localization tests execute on windows targets Apr 1, 2022
smarie pushed a commit to smarie/pandas that referenced this issue Apr 1, 2022
Fixed test_localization so that pandas-dev#46595 can be demonstrated …
a1f3773 
…on windows targets (even if today these do not run on windows targets, see pandas-dev#46597)
@simonjayhawkins simonjayhawkins added the Windows Windows OS label Apr 3, 2022
@mroeschke mroeschke added Localization Internationalization of data Testing pandas testing functions or related to the test suite and removed Needs Triage Issue that has not been reviewed by a pandas team member Bug labels Jul 6, 2022
mroeschke pushed a commit that referenced this issue Sep 8, 2022
@smarie
BUG: Fixed Unicode decoding error in Period.strftime when a locale-…
ae6dc97 
…specific directive is used (#46405)

* Added test representative of #46319. Should fail on CI

* Added a gha worker with non utf 8 zh_CN encoding

* Attempt to fix the encoding so that locale works

* Added the fix, but not using it for now, until CI is able to reproduce the issue.

* Crazy idea: maybe simply removing the .utf8 modifier will use the right encoding !

* Hopefully fixing the locale not available error

* Now simply generating the locale, not updating the ubuntu one

* Trying to install the locale without enabling it

* Stupid mistake

* Testing the optional locale generator condition

* Put back all runners

* Added whatsnew

* Now using the fix

* As per code review: moved locale-switching fixture `overridden_locale` to conftest

* Flake8

* Added comments on the runner

* Added a non-utf8 locale in the `it_IT` runner. Added the zh_CN.utf8 locale in the tests

* Improved readability of fixture `overridden_locale` as per code review

* Added two comments on default encoding

* Fixed #46319 by adding a new `char_to_string_locale` function in the `tslibs.util` module, able to decode char* using the current locale.

* As per code review: modified the test to contain non-utf8 chars. Fixed the resulting issue.

* Split the test in two for clarity

* Fixed test and flake8 error.

* Updated whatsnew to ref #46468 . Updated test name

* Removing wrong whatsnew bullet

* Nitpick on whatsnew as per code review

* Fixed build error rst directive

* Names incorrectly reverted in last merge commit

* Fixed test_localization so that #46595 can be demonstrated on windows targets (even if today these do not run on windows targets, see #46597)

* Fixed `tm.set_locale` context manager, it could error and leak when category LC_ALL was used. Fixed #46595

* Removed the fixture as per code review, and added corresponding parametrization in tests.

* Dummy mod to trigger CI again

* reverted dummy mod

* Attempt to fix the remaining error on the numpy worker

* Fixed issue in `_from_ordinal`

* Added asserts to try to understand

* Reverted debugging asserts and applied fix for numpy repeat from #47670.

* Fixed the last issue on numpy dev: a TypeError message had changed

* Code review: Removed `EXTRA_LOC`

* Code review: removed commented line

* Code review: reverted out of scope change

* Code review: reverted out of scope change

* Fixed unused import

* Fixed revert mistake

* Moved whatsnew to 1.6.0

* Update pandas/tests/io/parser/test_quoting.py

Co-authored-by: Sylvain MARIE <[email protected]>
noatamir pushed a commit to noatamir/pandas that referenced this issue Nov 9, 2022
@smarie @noatamir
BUG: Fixed Unicode decoding error in Period.strftime when a locale-…
250d886 
…specific directive is used (pandas-dev#46405)

* Added test representative of pandas-dev#46319. Should fail on CI

* Added a gha worker with non utf 8 zh_CN encoding

* Attempt to fix the encoding so that locale works

* Added the fix, but not using it for now, until CI is able to reproduce the issue.

* Crazy idea: maybe simply removing the .utf8 modifier will use the right encoding !

* Hopefully fixing the locale not available error

* Now simply generating the locale, not updating the ubuntu one

* Trying to install the locale without enabling it

* Stupid mistake

* Testing the optional locale generator condition

* Put back all runners

* Added whatsnew

* Now using the fix

* As per code review: moved locale-switching fixture `overridden_locale` to conftest

* Flake8

* Added comments on the runner

* Added a non-utf8 locale in the `it_IT` runner. Added the zh_CN.utf8 locale in the tests

* Improved readability of fixture `overridden_locale` as per code review

* Added two comments on default encoding

* Fixed pandas-dev#46319 by adding a new `char_to_string_locale` function in the `tslibs.util` module, able to decode char* using the current locale.

* As per code review: modified the test to contain non-utf8 chars. Fixed the resulting issue.

* Split the test in two for clarity

* Fixed test and flake8 error.

* Updated whatsnew to ref pandas-dev#46468 . Updated test name

* Removing wrong whatsnew bullet

* Nitpick on whatsnew as per code review

* Fixed build error rst directive

* Names incorrectly reverted in last merge commit

* Fixed test_localization so that pandas-dev#46595 can be demonstrated on windows targets (even if today these do not run on windows targets, see pandas-dev#46597)

* Fixed `tm.set_locale` context manager, it could error and leak when category LC_ALL was used. Fixed pandas-dev#46595

* Removed the fixture as per code review, and added corresponding parametrization in tests.

* Dummy mod to trigger CI again

* reverted dummy mod

* Attempt to fix the remaining error on the numpy worker

* Fixed issue in `_from_ordinal`

* Added asserts to try to understand

* Reverted debugging asserts and applied fix for numpy repeat from pandas-dev#47670.

* Fixed the last issue on numpy dev: a TypeError message had changed

* Code review: Removed `EXTRA_LOC`

* Code review: removed commented line

* Code review: reverted out of scope change

* Code review: reverted out of scope change

* Fixed unused import

* Fixed revert mistake

* Moved whatsnew to 1.6.0

* Update pandas/tests/io/parser/test_quoting.py

Co-authored-by: Sylvain MARIE <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Localization Internationalization of data Testing pandas testing functions or related to the test suite Windows Windows OS
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@smarie @mroeschke @simonjayhawkins