BUG: series.astype(str) vs Index.astype(str) inconsistency

[jbrockmendel]

ser = pd.Series([b'foo'], dtype=object)
idx = pd.Index(ser)

In [3]: ser.astype(str)[0]
Out[3]: "b'foo'"

In [4]: idx.astype(str)[0]
Out[4]: 'foo'

Series goes through astype_nansafe and from there through ensure_string_array. Index just calls calls the ndarray's .astype method. The Index behavior is specifically tested in test_astype_str_from_bytes xref #38607.

Index.astype can raise on non-ascii bytestrs:

val =  'あ'
bval = val.encode("UTF-8")

ser2 = pd.Series([bval], dtype=object)
idx2 = pd.Index(ser2)

In [21]: ser2.astype(str)[0]
Out[21]: "b'\\xe3\\x81\\x82'"

In [22]: idx2.astype(str)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
[...]
TypeError: Cannot cast Index to dtype <U0

These behaviors should match. I don't really have a preference between them.

[jbrockmendel]

@jorisvandenbossche thoughts on preferred API?

[jorisvandenbossche]

Index.astype can raise on non-ascii bytestrs:

In case we would keep the decoding-like behaviour, I think that example should also work? We currently rely on numpy, which seems to only accept ascii in astype (although you are astyping to their unicode dtype). It seems they have a specific decoding function that can do more than just ascii:

>>> idx2._values
array([b'\xe3\x81\x82'], dtype=object)
>>> idx2._values.astype(str)
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)

>>> idx2._values.astype(np.bytes_)
array([b'\xe3\x81\x82'], dtype='|S3')
>>> idx2._values.astype(np.bytes_).astype(str)  # same error
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)

>>> np.char.decode(idx2._values.astype(np.bytes_), "UTF-8")
array(['あ'], dtype='<U1')