BUG: Merging on two non-monotonic np.intc arrays fails

[tehunter]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "join_key": pd.Series([0, 2, 1], dtype=np.intc)
})
df_details = pd.DataFrame({
    "join_key": pd.Series([0, 1, 2], dtype=np.intc),
    "value": ["a", "b", "c"]
})

merged = pd.merge(df, df_details, on="join_key", how="left")

Issue Description

When merging on two np.intc arrays (where at least one is non-monotonic), pandas throws a "Buffer dtype mismatch" error. It only seems to happen when both arrays are np.intc type. I encountered this issue because that is the dtype returned from pd.DatetimeIndex.weekday.

Traceback (most recent call last):
  File "merge_bug_test.py", line 26, in <module>
    merged_1 = pd.merge(df, df_details, on="join_key", how="left")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 184, in merge
    return op.get_result(copy=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 886, in get_result
    join_index, left_indexer, right_indexer = self._get_join_info()
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1151, in _get_join_info
    (left_indexer, right_indexer) = self._get_join_indexers()
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1125, in _get_join_indexers
    return get_join_indexers(
           ^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1759, in get_join_indexers
    lidx, ridx = get_join_indexers_non_unique(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 1793, in get_join_indexers_non_unique
    lkey, rkey, count = _factorize_keys(left, right, sort=sort)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\env\Lib\site-packages\pandas\core\reshape\merge.py", line 2561, in _factorize_keys
    llab = rizer.factorize(lk)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^
  File "pandas\\_libs\\hashtable_class_helper.pxi", line 3045, in pandas._libs.hashtable.Int64Factorizer.factorize
ValueError: Buffer dtype mismatch, expected 'const int64_t' but got 'int'

It seems this issue was raised in #52451, the solution that was implemented involves checking if np.intc is not np.int32. This check returns True, meaning that np.intc is mapped to Int64Factorizer. It appears that this check no longer works as intended, or it is not robust enough.

A similar issue is raised in #58713 for uintc. In the proposed solution (#58727), it was suggested to use .itemsize as a more robust check. I think that suggestion might get to the root cause, as my np.intc(0).itemsize is 4, so np.intc should be mapped to Int32Factorizer

Expected Behavior

Merging should succeed if dtypes for both keys are the same.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.11.5
python-bits : 64
OS : Windows
OS-release : 10
Version : ...
machine : AMD64
processor : Intel64 Family 6 ...
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.2.3
numpy : 2.0.2
pytz : 2024.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : 1.4.0
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.9.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : None
numba : 0.60.0
numexpr : 2.10.1
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : None
pyarrow : 17.0.0
pyreadstat : 1.2.7
pytest : None
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2024.9.0
scipy : None
sqlalchemy : 2.0.35
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : None
tzdata : 2024.2
qtpy : 2.4.1
pyqt5 : None

[rhshadrach]

Thanks for the report! How is this not a duplicate of #58713?

[tehunter]

Thanks for the report! How is this not a duplicate of #58713?

They are basically the same, but #58713 implies the issue is specifically with np.uintc since a fix for np.intc was already implemented in #52451. The latter either didn't fully solve the issue or has since had a regression. Happy to move over to #58713 if you'd rather it be a single issue!

[rhshadrach]

Sounds good - we can leave both open. The solution for each is very similar and can be part of the same PR to fix. PRs are welcome!

[rhshadrach]

On second thought, I think they should be combined. I moved the details over to the other issue, closing here.