BUG: convert_dtypes(dtype_backend="pyarrow") converts "timestamp[ms][pyarrow]" to "timestamp[ns][pyarrow]"

[randolf-scholz]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
s = pd.Series(pd.date_range("2020-01-01", "2020-01-02", freq="1min"))
s = s.astype("timestamp[ms][pyarrow]")
t = s.convert_dtypes(dtype_backend="pyarrow")
pd.testing.assert_series_equal(s, t)

raises

Attribute "dtype" are different
[left]:  timestamp[ms][pyarrow]
[right]: timestamp[ns][pyarrow]

Issue Description

"timestamp[ms][pyarrow]" is already a valid nullable pyarrow-type, therefore convert_dtypes(dtype_backend="pyarrow") should leave it unaffected.

Expected Behavior

convert_dtypes(dtype_backend="pyarrow") should probably be NOOP for series that are already pyarrow data types.

Installed Versions

INSTALLED VERSIONS
------------------
commit           : 0f437949513225922d851e9581723d82120684a6
python           : 3.11.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.19.0-46-generic
Version          : #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 2.0.3
numpy            : 1.24.3
pytz             : 2023.3
dateutil         : 2.8.2
setuptools       : 68.0.0
pip              : 23.2
Cython           : 0.29.36
pytest           : 7.4.0
hypothesis       : None
sphinx           : 7.0.1
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.9.3
html5lib         : None
pymysql          : 1.4.6
psycopg2         : None
jinja2           : 3.1.2
IPython          : 8.14.0
pandas_datareader: None
bs4              : 4.12.2
bottleneck       : None
brotli           : None
fastparquet      : 2023.7.0
fsspec           : 2023.6.0
gcsfs            : None
matplotlib       : 3.7.2
numba            : 0.57.1
numexpr          : 2.8.4
odfpy            : None
openpyxl         : 3.1.2
pandas_gbq       : None
pyarrow          : 12.0.1
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.11.1
snappy           : None
sqlalchemy       : 1.4.49
tables           : 3.8.0
tabulate         : 0.9.0
xarray           : 2023.7.0
xlrd             : None
zstandard        : None
tzdata           : 2023.3
qtpy             : None
pyqt5            : None

[gpransshu]

I am volunteering to fix the reported issue with convert_dtypes in Pandas. I'll create a bug report, propose a fix, and engage with the community to contribute to the solution. Seeking guidance throughout the process.

[Julian048]

take

[Julian048]

I can confirm the issue being found in pandas version 2.0.3, but in the current dev version (2.1.0) I could not reproduce the issue. Will be investigating how it gets fixed.

[luke396]

I can confirm the issue being found in pandas version 2.0.3, but in the current dev version (2.1.0) I could not reproduce the issue. Will be investigating how it gets fixed.

You can try to convert the version to 2.0.3 (or other version which bug exists), maybe using git to checkout specific tag.

Then add a test to make sure the issue doesn't exist anymore.

[Julian048]

So the issue in version 2.0.3 was a result of pyarrow interacting with to_pandas_dtype() incorrectly and always outputting ns for the timestamp datatype regardless of the associated metadata.
(As per this issue from the pyarrow repository )

In the current main branch this error no longer exists as if conditionals were added within the inference aspect of the convert_dtypes() function to work around the pyarrow issue. Code in question linked [here](

pandas/pandas/core/dtypes/dtypes.py

Lines 2129 to 2142 in 4f55280

	def numpy_dtype(self) -> np.dtype:
	"""Return an instance of the related numpy dtype"""
	if pa.types.is_timestamp(self.pyarrow_dtype):
	# pa.timestamp(unit).to_pandas_dtype() returns ns units
	# regardless of the pyarrow timestamp units.
	# This can be removed if/when pyarrow addresses it:
	# https://github.com/apache/arrow/issues/34462
	return np.dtype(f"datetime64[{self.pyarrow_dtype.unit}]")
	if pa.types.is_duration(self.pyarrow_dtype):
	# pa.duration(unit).to_pandas_dtype() returns ns units
	# regardless of the pyarrow duration units
	# This can be removed if/when pyarrow addresses it:
	# https://github.com/apache/arrow/issues/34462
	return np.dtype(f"timedelta64[{self.pyarrow_dtype.unit}]")

Pyarrow has since fixed and closed the issue in their current main branch (dev version 13.0) and I have tested it to make sure that using version dev 13 those if conditions could be removed without affecting program execution outcome. They could, and so when pyarrow version 13 is released it should be good to remove those if conditions. I also suggest this issue be tagged with "Arrow".

Will be writing the pytest tests for this.