BUG: differing syntax for datetime column dtype specification causes failure with assert_frame_equal()

[EsmeMaxwell]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
from zoneinfo import ZoneInfo

# Create two identical dataframes
dates = pd.date_range("1/1/2000", periods=5, name="Date")
df = pd.DataFrame(np.ones((5, 3)), index=dates, columns=["A", "B", "C"])
df2 = pd.DataFrame(np.ones((5, 3)), index=dates, columns=["A", "B", "C"])

# Add a datetime column to each with identical data but the (same?) dtype specified using different syntax.
df["F_pd_DatetimeTZDtype"] = pd.array(
    ["2010/01/31 10:23:01", "1990", "2025/01/01 00:00+1", 1, "2024-12-31"],
    dtype=pd.DatetimeTZDtype(tz=ZoneInfo("Europe/Paris")),
)
df2["F_pd_DatetimeTZDtype"] = pd.array(
    ["2010/01/31 10:23:01", "1990", "2025/01/01 00:00+1", 1, "2024-12-31"],
    dtype="datetime64[ns, Europe/Paris]",
)

# Check if the dataframes are equal (this fails)
pd.testing.assert_frame_equal(df, df2)

Issue Description

As far as I'm aware, the assert_frame_equal() in the above example shouldn't fail but it thinks the dtypes are different:

AssertionError                            Traceback (most recent call last)
Cell In[9], [line 1](vscode-notebook-cell:?execution_count=9&line=1)
----> [1](vscode-notebook-cell:?execution_count=9&line=1) pd.testing.assert_frame_equal(df, df2)

    [... skipping hidden 3 frame]

File [~/miniconda3/envs/exio-datamanager/lib/python3.11/site-packages/pandas/_testing/asserters.py:614](https://file+.vscode-resource.vscode-cdn.net/home/peterma/exio_dev/repo/datamanager/scratch/~/miniconda3/envs/exio-datamanager/lib/python3.11/site-packages/pandas/_testing/asserters.py:614), in raise_assert_detail(obj, message, left, right, diff, first_diff, index_values)
    [611](https://file+.vscode-resource.vscode-cdn.net/home/peterma/exio_dev/repo/datamanager/scratch/~/miniconda3/envs/exio-datamanager/lib/python3.11/site-packages/pandas/_testing/asserters.py:611) if first_diff is not None:
    [612](https://file+.vscode-resource.vscode-cdn.net/home/peterma/exio_dev/repo/datamanager/scratch/~/miniconda3/envs/exio-datamanager/lib/python3.11/site-packages/pandas/_testing/asserters.py:612)     msg += f"\n{first_diff}"
--> [614](https://file+.vscode-resource.vscode-cdn.net/home/peterma/exio_dev/repo/datamanager/scratch/~/miniconda3/envs/exio-datamanager/lib/python3.11/site-packages/pandas/_testing/asserters.py:614) raise AssertionError(msg)

AssertionError: Attributes of DataFrame.iloc[:, 3] (column name="F_pd_DatetimeTZDtype") are different

Attribute "dtype" are different
[left]:  datetime64[ns, Europe/Paris]
[right]: datetime64[ns, Europe/Paris]

Both an eyeball check and using df.compare() suggests the data itself is the same. Checking the dtype, and type of the dtype, looks the same too, e.g.

print(df["F_pd_DatetimeTZDtype"].dtype)
print(type(df["F_pd_DatetimeTZDtype"].dtype))
print(df2["F_pd_DatetimeTZDtype"].dtype)
print(type(df2["F_pd_DatetimeTZDtype"].dtype))

produces

datetime64[ns, Europe/Paris]
<class 'pandas.core.dtypes.dtypes.DatetimeTZDtype'>
datetime64[ns, Europe/Paris]
<class 'pandas.core.dtypes.dtypes.DatetimeTZDtype'>

However it fails with assert_frame_equal().

Expected Behavior

That the dtype specified by the different syntax(es) dtype=pd.DatetimeTZDtype(tz=ZoneInfo("Europe/Paris") vs dtype="datetime64[ns, Europe/Paris]" shouldn't cause assert_frame_equal() to fail for the associated column.

Unless I've badly misunderstood expected behaviour, which is quite possible.

Installed Versions

INSTALLED VERSIONS

commit : f538741
python : 3.11.8.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-97-generic
Version : #107~20.04.1-Ubuntu SMP Fri Feb 9 14:20:11 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.2.0
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 69.1.0
pip : 24.0
Cython : None
pytest : 8.0.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.21.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : 2023.10.1
fsspec : 2024.2.0
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.9.0
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 15.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

This is why it's raising.

print(type(df["F_pd_DatetimeTZDtype"].dtype.tz))
# <class 'zoneinfo.ZoneInfo'>

print(type(df2["F_pd_DatetimeTZDtype"].dtype.tz))
# <class 'pytz.tzfile.Europe/Paris'>

Not sure if this is expected, cc @jbrockmendel

[jbrockmendel]

i think its expected, but agreed it can be annoying. These dtypes are mostly equivalent, but there are some corner cases where things can be different, so i think until @mroeschke's work converting the default to zoneinfo is done we need to just live with this

[rhshadrach]

@EsmeMaxwell - as mentioned above, this will be resolved by switching the default to zoneinfo. Closing.