BUG: tm.set_locale does not correctly set back the initial locale when locale.LC_ALL is used. tm.can_set_locale is impacted by the leak

[smarie]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import locale
import pandas._testing as tm

current_locale = locale.setlocale(locale.LC_ALL)

print(current_locale)
# On my machine (FR):
# >>> 'LC_COLLATE=C;LC_CTYPE=French_France.1252;LC_MONETARY=C;LC_NUMERIC=C;LC_TIME=C'

print(tm.can_set_locale("it_IT.utf8", locale.LC_ALL))  # False

new_current_locale = locale.setlocale(locale.LC_ALL)
assert new_current_locale == current_locale  # AssertionError: the contextmanager leaked
print(new_current_locale)  # 'it_IT.UTF-8' !!!!

Issue Description

The call to tm.can_set_locale actually contains a silent error when the inner tm.set_locale tries to set back the original locale. Because this error happens, the original locale is not set back !

The issue is actually due to a mistake in the code for set_locale: indeed the way it captures the initial locale (to set it back at the end) is wrong.

current_locale = locale.getlocale() is used, however this returns only the locale for the default category LC_CTYPE (see python doc)

What should be used instead is current_locale = locale.setlocale(lc_var). That way it is also working if lc_var is LC_ALL

Note that the example in the cpython documentation is misleading, I'll open an issue there as well.

Expected Behavior

No assertion error at the end of the example.

Installed Versions

AssertionError: C:\Miniconda3\envs\pandas-dev\lib\distutils\core.py

[smarie]

I'll fix this in #46405

[smarie]

Cpython doc issue opened: https://bugs.python.org/issue47187

[andjc]

Although,

if you use

import locale
import pandas._testing as tm

current_locale = locale.setlocale(locale.LC_ALL, '')

print(current_locale)
print(tm.can_set_locale("it_IT.utf8", locale.LC_ALL)) 
new_current_locale = locale.setlocale(locale.LC_ALL, '')
assert new_current_locale == current_locale  
print(new_current_locale)  

Then it works as expected. The python documentation does suggest using the form locale.setlocale(locale.LC_ALL, '')

locale.setlocale(locale.LC_ALL) would be the same as locale.setlocale(locale.LC_ALL, None), i.e. set the locale to the current value. Which is different to saying locale.setlocale(locale.LC_ALL, ''), i.e. set locale to the default locale.

[smarie]

Thanks for the feedback @andjc !
Indeed you are right, this usage works.

Still, the reported bug exists so I'm convinced that it should be fixed to avoid anyone falling into that nasty trap.

[andjc]

@smarie

The problem is, your code was saying:

1. set it to the current locale (which at the beginning is your default locale
2. set it to Italian
3. set it to the current locale (which should be Italian)

[smarie]

@andjc , I think that you got my initial example wrong.

I use locale.setlocale(locale.LC_ALL) to GET the current locale (not to set it). According to the python docs,

If locale is omitted or None, the current setting for category is returned.

so this is the way to get the current locale for LC_ALL. Indeed getLocale can not be used to get the current locale for LC_ALL (this is explicitly stated in https://docs.python.org/3/library/locale.html#locale.getlocale)

category may be one of the LC_* values except LC_ALL.

Anyway, this example is more straightforward if you prefer:

import locale
import pandas._testing as tm

print(locale.getlocale())
print(tm.can_set_locale("it_IT.utf8", locale.LC_ALL))  # Should be True but is False
print(locale.getlocale())

yields

('fr_FR', 'cp1252')
False
('it_IT', 'UTF-8')

So tm.can_set_locale returns False (first issue) AND changes the locale (second issue). It should just return True without changing anything.

The root of this issue can be found by looking at the code from set_locale:
https://github.com/pandas-dev/pandas/blob/main/pandas/_config/localization.py#L42-L52

The "finally" block does not work and raises locale.Error: unsupported locale setting in the example above. The reason is that current_locale is obtained with current_locale = locale.getlocale(), while in the finally block there is an attempt to set it using locale.setlocale, which does not work when lc_var is LC_ALL with the value returned by locale.getLocale. The fix can be seen here: https://github.com/pandas-dev/pandas/pull/46405/files#diff-52be4ac24f9b9731e939169af395829a61cd2cb2f254b14c5c0e83bdd2296ddcR42

Feel free to provide an example that would show better the issue, if you like !