Skip to content

BUG: Index.getitem returning wrong result with negative step for arrow #55832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Nov 22, 2023
+35 −3
Merged

BUG: Index.getitem returning wrong result with negative step for arrow #55832

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
ff6e271
BUG: Index.getitem returning wrong result with negative step for arrow
phofl Nov 5, 2023
7518250
Merge branch 'main' into indexing_negative_steps
phofl Nov 5, 2023
7684911
Update
phofl Nov 5, 2023
7474cb2
Update
phofl Nov 5, 2023
b27b0f8
Fix
phofl Nov 5, 2023
659577e
Update array.py
phofl Nov 5, 2023
30097eb
Merge branch 'main' into indexing_negative_steps
phofl Nov 15, 2023
5c4bc50
Merge remote-tracking branch 'upstream/main' into indexing_negative_s…
phofl Nov 16, 2023
86fe4f1
Fix
phofl Nov 16, 2023
2425529
Add gh ref
phofl Nov 17, 2023
ff4bd23
Merge remote-tracking branch 'upstream/main' into indexing_negative_s…
phofl Nov 21, 2023
06b4f89
Update
phofl Nov 21, 2023
102209e
Merge branch 'main' into indexing_negative_steps
phofl Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.1.3.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Fixed regressions
Bug fixes
~~~~~~~~~
- Bug in :meth:`DatetimeIndex.diff` raising ``TypeError`` (:issue:`55080`)
- Bug in :meth:`Index.__getitem__` returning wrong result for Arrow dtypes and negative stepsize (:issue:`55832`)
- Bug in :meth:`Index.isin` raising for Arrow backed string and ``None`` value (:issue:`55821`)

.. ---------------------------------------------------------------------------
Expand Down
7 changes: 7 additions & 0 deletions pandas/core/arrays/arrow/array.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -553,6 +553,13 @@ def __getitem__(self, item: PositionalIndexer):
)
# We are not an array indexer, so maybe e.g. a slice or integer
# indexer. We dispatch to pyarrow.
if isinstance(item, slice):
if item.start == item.stop:
pass
elif item.start == -len(self) - 1:
Copy link
Member

@lukemanley lukemanley Nov 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be item.start < -len(self)?

Not sure if you want to mimic numpy behavior here but numpy allows slicing beyond the len of the array:

In [1]: import numpy as np

In [2]: np.array(["a", "b", "c"])[slice(-10, 10)]
Out[2]: array(['a', 'b', 'c'], dtype='<U1')

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that makes sense, updated

item = slice(None, item.stop, item.step)
elif item.stop == -len(self) - 1:
item = slice(item.start, None, item.step)
value = self._pa_array[item]
if isinstance(value, pa.ChunkedArray):
return type(self)(value)
Expand Down
14 changes: 11 additions & 3 deletions pandas/tests/indexes/object/test_indexing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import pytest

from pandas._libs.missing import is_matching_na
import pandas.util._test_decorators as td

import pandas as pd
from pandas import Index
Expand Down Expand Up @@ -144,6 +145,13 @@ def test_get_indexer_non_unique_np_nats(self, np_nat_fixture, np_nat_fixture2):


class TestSliceLocs:
@pytest.mark.parametrize(
"dtype",
[
"object",
pytest.param("string[pyarrow_numpy]", marks=td.skip_if_no("pyarrow")),
],
)
@pytest.mark.parametrize(
"in_slice,expected",
[
Expand All @@ -167,12 +175,12 @@ class TestSliceLocs:
(pd.IndexSlice["m":"m":-1], ""), # type: ignore[misc]
],
)
def test_slice_locs_negative_step(self, in_slice, expected):
index = Index(list("bcdxy"))
def test_slice_locs_negative_step(self, in_slice, expected, dtype):
index = Index(list("bcdxy"), dtype=dtype)

s_start, s_stop = index.slice_locs(in_slice.start, in_slice.stop, in_slice.step)
result = index[s_start : s_stop : in_slice.step]
expected = Index(list(expected))
expected = Index(list(expected), dtype=dtype)
tm.assert_index_equal(result, expected)

def test_slice_locs_dup(self):
Expand Down