Skip to content

PERF: Avoid np.divmod in maybe_sequence_to_range #57812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Mar 21, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pandas/_libs/lib.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -231,3 +231,7 @@ def is_range_indexer(
left: np.ndarray,
n: int, # np.ndarray[np.int64, ndim=1]
) -> bool: ...
def is_range(
sequence: np.ndarray,
diff: int, # np.ndarray[np.int64, ndim=1]
) -> bool: ...
20 changes: 20 additions & 0 deletions pandas/_libs/lib.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -678,6 +678,26 @@ def is_range_indexer(ndarray[int6432_t, ndim=1] left, Py_ssize_t n) -> bool:
return True


@cython.wraparound(False)
@cython.boundscheck(False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don’t we only need int64?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes we'll be calling self._shallow_copy(Index[int]._values) so I suppose the int type could be non-int64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe should just use np.intp ? That matches what range would use internally

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like np.intp didn't work for the 32 bit and Windows build 0b484bd

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you seeing build failures just doing what @jbrockmendel suggests with int64 then? The error message ValueError: Buffer dtype mismatch, expected 'intp_t' but got 'long long' on the 32 bit build would seemingly indicate the 32 bit part of the fused type is unused, since long long is by definition at least 64 bits

https://github.com/pandas-dev/pandas/actions/runs/8268872377/job/22622796389#step:4:43677

Copy link
Member Author

@mroeschke mroeschke Mar 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With just int64_t, 32 bit and Windows builds are failing with ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'long' https://github.com/pandas-dev/pandas/actions/runs/8284194543/job/22669177062?pr=57812

def is_range(ndarray[int6432_t, ndim=1] sequence, int64_t diff) -> bool:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this have a more verbose-but-descriptive name? from this name alone id expect this to be a somehow-optimized isinstance(sequence, range)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing. Renamed to is_sequence_range

"""
Check if sequence is equivalent to a range with the specified diff.
"""
cdef:
Py_ssize_t i, n = len(sequence)

if diff == 0:
return False

for i in range(n):

if sequence[i] != sequence[0] + i * diff:
return False

return True


ctypedef fused ndarr_object:
ndarray[object, ndim=1]
ndarray[object, ndim=2]
Expand Down
11 changes: 3 additions & 8 deletions pandas/core/indexes/range.py
Original file line number Diff line number Diff line change
Expand Up @@ -476,14 +476,9 @@ def _shallow_copy(self, values, name: Hashable = no_default):
# GH 46675 & 43885: If values is equally spaced, return a
# more memory-compact RangeIndex instead of Index with 64-bit dtype
diff = values[1] - values[0]
if not missing.isna(diff) and diff != 0:
maybe_range_indexer, remainder = np.divmod(values - values[0], diff)
if (
lib.is_range_indexer(maybe_range_indexer, len(maybe_range_indexer))
and not remainder.any()
):
new_range = range(values[0], values[-1] + diff, diff)
return type(self)._simple_new(new_range, name=name)
if not missing.isna(diff) and lib.is_range(values, diff):
new_range = range(values[0], values[-1] + diff, diff)
return type(self)._simple_new(new_range, name=name)
return self._constructor._simple_new(values, name=name)

def _view(self) -> Self:
Expand Down