Skip to content

Commit 0541833

Browse files
authored
Backport PR pandas-dev#53672 on branch 2.0.x (BUG/CoW: is_range_indexer can't handle very large arrays) (pandas-dev#53691)
BUG/CoW: is_range_indexer can't handle very large arrays (pandas-dev#53672) * BUG: is_range_indexer can't handle very large arrays * fix test on 32-bit (cherry picked from commit 905fe6b)
1 parent c504ed6 commit 0541833

File tree

3 files changed

+15
-2
lines changed

3 files changed

+15
-2
lines changed

doc/source/whatsnew/v2.0.3.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ Bug fixes
2424
- Bug in :func:`RangeIndex.union` when using ``sort=True`` with another :class:`RangeIndex` (:issue:`53490`)
2525
- Bug in :func:`read_csv` when defining ``dtype`` with ``bool[pyarrow]`` for the ``"c"`` and ``"python"`` engines (:issue:`53390`)
2626
- Bug in :meth:`Series.str.split` and :meth:`Series.str.rsplit` with ``expand=True`` for :class:`ArrowDtype` with ``pyarrow.string`` (:issue:`53532`)
27-
-
27+
- Bug in indexing methods (e.g. :meth:`DataFrame.__getitem__`) where taking the entire :class:`DataFrame`/:class:`Series` would raise an ``OverflowError`` when Copy on Write was enabled and the length of the array was over the maximum size a 32-bit integer can hold (:issue:`53616`)
2828

2929
.. ---------------------------------------------------------------------------
3030
.. _whatsnew_203.other:

pandas/_libs/lib.pyx

+1-1
Original file line numberDiff line numberDiff line change
@@ -651,7 +651,7 @@ ctypedef fused int6432_t:
651651

652652
@cython.wraparound(False)
653653
@cython.boundscheck(False)
654-
def is_range_indexer(ndarray[int6432_t, ndim=1] left, int n) -> bool:
654+
def is_range_indexer(ndarray[int6432_t, ndim=1] left, Py_ssize_t n) -> bool:
655655
"""
656656
Perform an element by element comparison on 1-d integer arrays, meant for indexer
657657
comparisons

pandas/tests/libs/test_lib.py

+13
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
lib,
77
writers as libwriters,
88
)
9+
from pandas.compat import IS64
910

1011
from pandas import Index
1112
import pandas._testing as tm
@@ -248,6 +249,18 @@ def test_is_range_indexer(self, dtype):
248249
left = np.arange(0, 100, dtype=dtype)
249250
assert lib.is_range_indexer(left, 100)
250251

252+
@pytest.mark.skipif(
253+
not IS64,
254+
reason="2**31 is too big for Py_ssize_t on 32-bit. "
255+
"It doesn't matter though since you cannot create an array that long on 32-bit",
256+
)
257+
@pytest.mark.parametrize("dtype", ["int64", "int32"])
258+
def test_is_range_indexer_big_n(self, dtype):
259+
# GH53616
260+
left = np.arange(0, 100, dtype=dtype)
261+
262+
assert not lib.is_range_indexer(left, 2**31)
263+
251264
@pytest.mark.parametrize("dtype", ["int64", "int32"])
252265
def test_is_range_indexer_not_equal(self, dtype):
253266
# GH#50592

0 commit comments

Comments
 (0)