BUG: ExtensionArray no longer can be merged with Pandas 2.2

[hoxbro]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from datashader.datatypes import RaggedArray

key = RaggedArray([[0, 1], [1, 2, 3, 4]], dtype='float64')
df = pd.DataFrame({"key": key, "val": [1, 2]})

pd.merge(df, df, on='key')

Issue Description

We have an extension array, and after upgrading to Pandas 2.2. Some of our tests are beginning to fail (test_merge_on_extension_array). This is related to changes made in #56523. More specifically, these lines:

pandas/pandas/core/reshape/merge.py

Lines 1751 to 1755 in 1aeaf6b

	if (
	left.is_monotonic_increasing
	and right.is_monotonic_increasing
	and (left.is_unique or right.is_unique)
	):

This will raise the error below, with the example above

Traceback

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 7
      4 key = RaggedArray([[0, 1], [1, 2, 3, 4]], dtype='float64')
      5 df = pd.DataFrame({"key": key, "val": [1, 2]})
----> 7 pd.merge(df, df, on='key')

File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:184, in merge(left, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator, validate)
    169 else:
    170     op = _MergeOperation(
    171         left_df,
    172         right_df,
   (...)
    182         validate=validate,
    183     )
--> 184     return op.get_result(copy=copy)

File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:886, in _MergeOperation.get_result(self, copy)
    883 if self.indicator:
    884     self.left, self.right = self._indicator_pre_merge(self.left, self.right)
--> 886 join_index, left_indexer, right_indexer = self._get_join_info()
    888 result = self._reindex_and_concat(
    889     join_index, left_indexer, right_indexer, copy=copy
    890 )
    891 result = result.__finalize__(self, method=self._merge_type)

File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:1151, in _MergeOperation._get_join_info(self)
   1147     join_index, right_indexer, left_indexer = _left_join_on_index(
   1148         right_ax, left_ax, self.right_join_keys, sort=self.sort
   1149     )
   1150 else:
-> 1151     (left_indexer, right_indexer) = self._get_join_indexers()
   1153     if self.right_index:
   1154         if len(self.left) > 0:

File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:1125, in _MergeOperation._get_join_indexers(self)
   1123 # make mypy happy
   1124 assert self.how != "asof"
-> 1125 return get_join_indexers(
   1126     self.left_join_keys, self.right_join_keys, sort=self.sort, how=self.how
   1127 )

File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:1753, in get_join_indexers(left_keys, right_keys, sort, how)
   1749 left = Index(lkey)
   1750 right = Index(rkey)
   1752 if (
-> 1753     left.is_monotonic_increasing
   1754     and right.is_monotonic_increasing
   1755     and (left.is_unique or right.is_unique)
   1756 ):
   1757     _, lidx, ridx = left.join(right, how=how, return_indexers=True, sort=sort)
   1758 else:

File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/indexes/base.py:2251, in Index.is_monotonic_increasing(self)
   2229 @property
   2230 def is_monotonic_increasing(self) -> bool:
   2231     """
   2232     Return a boolean if the values are equal or increasing.
   2233 
   (...)
   2249     False
   2250     """
-> 2251     return self._engine.is_monotonic_increasing

File index.pyx:262, in pandas._libs.index.IndexEngine.is_monotonic_increasing.__get__()

File index.pyx:283, in pandas._libs.index.IndexEngine._do_monotonic_check()

File index.pyx:297, in pandas._libs.index.IndexEngine._call_monotonic()

File algos.pyx:853, in pandas._libs.algos.is_monotonic()

ValueError: operands could not be broadcast together with shapes (4,) (2,) 

I can get things to run if I guard the check in a try/except like this:

try:
    check = (
        left.is_monotonic_increasing 
        and right.is_monotonic_increasing 
        and (left.is_unique or right.is_unique) 
     )
except Exception:
     check = False

if check:
     ...

Expected Behavior

It will work like Pandas 2.1 and merge the two dataframes.

Installed Versions

INSTALLED VERSIONS
------------------
commit                : f538741432edf55c6b9fb5d0d496d2dd1d7c2457
python                : 3.12.1.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 6.6.10-76060610-generic
Version               : #202401051437~1704728131~22.04~24d69e2 SMP PREEMPT_DYNAMIC Mon J
machine               : x86_64
processor             : x86_64
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 2.2.0
numpy                 : 1.26.4
pytz                  : 2024.1
dateutil              : 2.8.2
setuptools            : 69.0.3
pip                   : 24.0
Cython                : None
pytest                : 7.4.4
hypothesis            : None
sphinx                : 5.3.0
blosc                 : None
feather               : None
xlsxwriter            : None
lxml.etree            : 5.1.0
html5lib              : None
pymysql               : None
psycopg2              : None
jinja2                : 3.1.3
IPython               : 8.21.0
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : 2023.10.1
fsspec                : 2024.2.0
gcsfs                 : None
matplotlib            : 3.8.2
numba                 : 0.59.0
numexpr               : 2.9.0
odfpy                 : None
openpyxl              : 3.1.2
pandas_gbq            : None
pyarrow               : 15.0.0
pyreadstat            : None
python-calamine       : None
pyxlsb                : None
s3fs                  : 2024.2.0
scipy                 : 1.12.0
sqlalchemy            : 2.0.25
tables                : None
tabulate              : 0.9.0
xarray                : 2024.1.1
xlrd                  : None
zstandard             : None
tzdata                : 2023.4
qtpy                  : None
pyqt5                 : None

[phofl]

cc @lukemanley

[lukemanley]

Thanks for the report. The underlying issue here is that Index.is_monotonic_increasing is raising in this case:

import pandas as pd
from datashader.datatypes import RaggedArray

key = RaggedArray([[0, 1], [1, 2, 3, 4]], dtype='float64')

pd.Index(key).is_monotonic_increasing   # <- ValueError

I put a fix in #57333 to catch ValueError in addition to the other exception types already being caught.