Skip to content

REGR: merge with 3rd party EA's can sometimes raise ValueError #57333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 10, 2024
Merged

REGR: merge with 3rd party EA's can sometimes raise ValueError #57333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
79936d7
catch ValueError
lukemanley Feb 10, 2024
8302b97
whatsnew
lukemanley Feb 10, 2024
ea07bfd
add test
lukemanley Feb 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.2.1.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Fixed regressions
- Fixed regression in :meth:`.DataFrameGroupBy.idxmin`, :meth:`.DataFrameGroupBy.idxmax`, :meth:`.SeriesGroupBy.idxmin`, :meth:`.SeriesGroupBy.idxmax` ignoring the ``skipna`` argument (:issue:`57040`)
- Fixed regression in :meth:`.DataFrameGroupBy.idxmin`, :meth:`.DataFrameGroupBy.idxmax`, :meth:`.SeriesGroupBy.idxmin`, :meth:`.SeriesGroupBy.idxmax` where values containing the minimum or maximum value for the dtype could produce incorrect results (:issue:`57040`)
- Fixed regression in :meth:`DataFrame.loc` raising ``IndexError`` for non-unique, masked dtype indexes where result has more than 10,000 rows (:issue:`57027`)
- Fixed regression in :meth:`DataFrame.merge` raising ``ValueError`` for certain types of 3rd-party extension arrays (:issue:`57316`)
- Fixed regression in :meth:`DataFrame.sort_index` not producing a stable sort for a index with duplicates (:issue:`57151`)
- Fixed regression in :meth:`DataFrame.to_dict` with ``orient='list'`` and datetime or timedelta types returning integers (:issue:`54824`)
- Fixed regression in :meth:`DataFrame.to_json` converting nullable integers to floats (:issue:`57224`)
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/index.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ cdef class IndexEngine:
values = self.values
self.monotonic_inc, self.monotonic_dec, is_strict_monotonic = \
self._call_monotonic(values)
except (TypeError, InvalidOperation):
except (TypeError, InvalidOperation, ValueError):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we test this somehow? We have "test_downstream.py" for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would you be ok with just testing against a pyarrow list type? e.g. this raises a ValueError:

import pyarrow as pa
import pandas as pd

arr = pd.array([[1], [2, 3]], dtype=pd.ArrowDtype(pa.list_(pa.int64())))

pd.Index(arr).is_monotonic_increasing  # ValueError

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep that’s fine

Copy link
Member

@phofl phofl Feb 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want to avoid that someone removes the ValueError and the tests don’t fail

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, test added

self.monotonic_inc = 0
self.monotonic_dec = 0
is_strict_monotonic = 0
Expand Down
10 changes: 10 additions & 0 deletions pandas/tests/indexes/test_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1686,3 +1686,13 @@ def test_nan_comparison_same_object(op):

result = op(idx, idx.copy())
tm.assert_numpy_array_equal(result, expected)


@td.skip_if_no("pyarrow")
def test_is_monotonic_pyarrow_list_type():
# GH 57333
import pyarrow as pa

idx = Index([[1], [2, 3]], dtype=pd.ArrowDtype(pa.list_(pa.int64())))
assert not idx.is_monotonic_increasing
assert not idx.is_monotonic_decreasing