BUG: DataFrame.to_parquet does not round-trip index dtype

[randolf-scholz]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

df = pd.DataFrame(
    np.random.randint(0, 9, size=(5, 3)),
    columns=["a", "b", "c"],
    dtype="int32[pyarrow]",
).set_index("a")

df.to_parquet("demo.parquet")
df2 = pd.read_parquet("demo.parquet")
pd.testing.assert_frame_equal(df, df2)

Issue Description

The code snippet yields

AssertionError: DataFrame.index are different

Attribute "dtype" are different
[left]:  int32[pyarrow]
[right]: int32

Expected Behavior

The data types should be round tripped for both columns and index.

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.10.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.0-46-generic
Version : #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.3
numpy : 1.25.0
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.8.0
pip : 23.1.2
Cython : None
pytest : 7.3.2
hypothesis : None
sphinx : 7.0.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : 1.0.3
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.14.0
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 12.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : 2.0.16
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[randolf-scholz]

Explicitly settting the option dtype_backend="pyarrow" makes the index use pyarrow types.

[randolf-scholz]

However, this messes things up when string dtypes are involved:

import pandas as pd

df = (
    pd.DataFrame({"a": [1, 2, 3], "b": [1.0, 2.0, None], "c": [None, "x", "y"]})
    .astype({"a": "int64[pyarrow]", "b": "float64[pyarrow]", "c": "string[pyarrow]"})
    .set_index("a")
)
df.to_parquet("demo.parquet")
df2 = pd.read_parquet("demo.parquet", dtype_backend="pyarrow")
pd.testing.assert_frame_equal(df, df2)

results in

Attribute "dtype" are different
[left]:  string[pyarrow]
[right]: string[pyarrow]

[lithomas1]

I think the OP is intentional.
When we write to pyarrow via parquet, we would convert to an Arrow dtype before writing, so on top of my head there is no way to tell if the original DF is backed by numpy/Arrow.

Long term, I think #51846 is the right solution (e.g. opting into the arrow engines will use arrow dtypes).

The second example is definitely a bug, though (lol).

I think it stems from

>>> type(df['c'].dtype)
<class 'pandas.core.arrays.string_.StringDtype'>
>>> type(df2['c'].dtype)
<class 'pandas.core.dtypes.dtypes.ArrowDtype'>

cc @phofl if I missed anything here.

[randolf-scholz]

@lithomas1 for the second example I had opened a separate issue: #54190

So maybe this one can be closed? Although there is a solution to

so on top of my head there is no way to tell if the original DF is backed by numpy/Arrow.

pandas could write the schema information into metadata when serializing as parquet, and then autocast when deserializing.