BUG: UnicodeError when using Period.strftime with non-utf8 locale

[smarie]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import locale

locale.setlocale(locale.LC_ALL, "zh_CN")
p = pd.Period("2018-03-11 13:00", freq="H")
print(p.strftime("%p"))  # UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf ...

Issue Description

Period.strftime("%p") prints the locale-specific version of AM or PM. When the locale uses a non-utf8 compliant encoding, it crashes.

This bug does not happen with others, for example Timestamp.strftime.

Expected Behavior

No error, printing the actual string representing AM or PM

Installed Versions

This is on the main branch head

[smarie]

I'll try to fix it in #46116

[smarie]

It actually seems that util.char_to_string is used after c_strftime . This relies on PyUnicode_FromString.

Maybe we should rather use PyUnicode_DecodeLocale https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_DecodeLocale

[smarie]

Note that this issue may also happen when plotting timeseries with matplotlib. Indeed the x-axis labels formatter seems to be using Period.strftime. For example this is a stracktrace I got from sphinx-gallery with figures using matplotlib :

WARNING: C:\(...)\doc\examples\1_demo.py failed to execute correctly: Traceback (most recent call last):
  File "C:\(...)\.nox\doc\lib\site-packages\sphinx_gallery\scrapers.py", line 378, in save_figures
    rst = scraper(block, block_vars, gallery_conf)
  File "C:\(...)\.nox\doc\lib\site-packages\sphinx_gallery\scrapers.py", line 171, in matplotlib_scraper
    fig.savefig(image_path, **these_kwargs)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\figure.py", line 3019, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\backend_bases.py", line 2319, in print_figure
    result = print_method(
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\backend_bases.py", line 1648, in wrapper
    return func(*args, **kwargs)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\_api\deprecation.py", line 412, in wrapper
    return func(*inner_args, **inner_kwargs)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\backends\backend_agg.py", line 540, in print_png
    FigureCanvasAgg.draw(self)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\backends\backend_agg.py", line 436, in draw
    self.figure.draw(self.renderer)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\artist.py", line 73, in draw_wrapper
    result = draw(artist, renderer, *args, **kwargs)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\artist.py", line 50, in draw_wrapper
    return draw(artist, renderer)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\figure.py", line 2810, in draw
    mimage._draw_list_compositing_images(
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\image.py", line 132, in _draw_list_compositing_images
    a.draw(renderer)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\artist.py", line 50, in draw_wrapper
    return draw(artist, renderer)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\axes\_base.py", line 3082, in draw
    mimage._draw_list_compositing_images(
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\image.py", line 132, in _draw_list_compositing_images
    a.draw(renderer)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\artist.py", line 50, in draw_wrapper
    return draw(artist, renderer)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\axis.py", line 1158, in draw
    ticks_to_draw = self._update_ticks()
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\axis.py", line 1046, in _update_ticks
    major_labels = self.major.formatter.format_ticks(major_locs)
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\ticker.py", line 224, in format_ticks
    return [self(value, i) for i, value in enumerate(values)]
  File "C:\(...)\.nox\doc\lib\site-packages\matplotlib\ticker.py", line 224, in <listcomp>
    return [self(value, i) for i, value in enumerate(values)]
  File "C:\(...)\.nox\doc\lib\site-packages\pandas\plotting\_matplotlib\converter.py", line 1074, in __call__
    return Period(ordinal=int(x), freq=self.freq).strftime(fmt)
  File "pandas\_libs\tslibs\period.pyx", line 2458, in pandas._libs.tslibs.period._Period.strftime
  File "pandas\_libs\tslibs\period.pyx", line 1225, in pandas._libs.tslibs.period.period_format
  File "pandas\_libs\tslibs\period.pyx", line 1258, in pandas._libs.tslibs.period._period_strftime
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

[smarie]

Finally note that the issue also happens when there is a non-utf8 character in the formatting string:

import pandas as pd
import locale

locale.setlocale(locale.LC_ALL, "fr_FR")
per = pd.Period("2018-03-11 13:00", freq="H")
assert per.strftime("é") == "é"  # AssertionError

In that case no error is raised but the output string does not correspond to the expected one.

Thanks @jreback for suggesting that this might fail !

[smarie]

I moved the above into a dedicated issue for clarity, as the issues have similar causes but are not related