Skip to content

BUG: Bug in loc raised ValueError when setting value via boolean list #37761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
May 25, 2021
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e38dc1c
Fix listlike loc bug
phofl Nov 11, 2020
e606f33
Add test
phofl Nov 11, 2020
8e02de7
Run black
phofl Nov 11, 2020
c8bacf5
Fix imports
phofl Nov 11, 2020
64c3dfc
Change if condition and move test
phofl Nov 12, 2020
17c0cda
Modify test
phofl Nov 12, 2020
272f53b
Add parametrization
phofl Nov 13, 2020
e37d37c
Convert boolean list for iloc
phofl Nov 13, 2020
9adcd05
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl Nov 13, 2020
0a45a73
Parametrize test
phofl Nov 13, 2020
29ec9b5
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl Nov 13, 2020
83745b1
Remove if condition part
phofl Nov 13, 2020
7db17ce
Change whatsnew
phofl Nov 13, 2020
a53ae1b
Move test
phofl Nov 13, 2020
b5dd453
Change order of condition
phofl Nov 13, 2020
834c816
Adress review
phofl Nov 19, 2020
da88226
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl Nov 19, 2020
1293109
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl Dec 21, 2020
99a9330
Move whatsnew
phofl Dec 21, 2020
0f9f515
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl Jan 21, 2021
08b0362
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl Jan 28, 2021
b971750
Move validation
phofl Jan 28, 2021
15ad761
Cast to array
phofl Jan 28, 2021
c716b31
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl May 14, 2021
c5c15c9
Merge branch 'master' of https://github.com/pandas-dev/pandas into 20438
phofl May 24, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,7 @@ Indexing
- Bug in :meth:`Series.loc` and :meth:`DataFrame.loc` raises when numeric label was given for object :class:`Index` although label was in :class:`Index` (:issue:`26491`)
- Bug in :meth:`DataFrame.loc` returned requested key plus missing values when ``loc`` was applied to single level from :class:`MultiIndex` (:issue:`27104`)
- Bug in indexing on a :class:`Series` or :class:`DataFrame` with a :class:`CategoricalIndex` using a listlike indexer containing NA values (:issue:`37722`)
- Bug in :meth:`Series.loc` raised an Error when input was filtered with a boolean list and values to set were a list with lower dimension (:issue:`20438`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a specific exception we can say instead of Error?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueError, added it


Missing
^^^^^^^
Expand Down
4 changes: 3 additions & 1 deletion pandas/core/indexing.py
Original file line number Diff line number Diff line change
Expand Up @@ -602,7 +602,7 @@ def _get_setitem_indexer(self, key):
"""
Convert a potentially-label-based key into a positional indexer.
"""
if self.name == "loc":
if self.name == "loc" or com.is_bool_indexer(key):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i haven't fully formed an opinion, but i think another alternative would be to properly handle boolean masks in Block.setitem.

Copy link
Member Author

@phofl phofl Nov 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, loc is not failing, this means we do not need this or. Removed it. Must have mixed something up before

Copy link
Member Author

@phofl phofl Nov 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lists can be handled in Block.setitem already, but arrays from dtype object don't work. Lists are killed by the validation mentioned above in pandas.core.indexers. If we let them through, setitem seems to be able to handle them in the iloc testcases

self._ensure_listlike_indexer(key)

if self.axis is not None:
Expand Down Expand Up @@ -1521,6 +1521,8 @@ def _convert_to_indexer(self, key, axis: int, is_setter: bool = False):

def _get_setitem_indexer(self, key):
# GH#32257 Fall through to let numpy do validation
if com.is_bool_indexer(key) and isinstance(key, list):
return np.array(key)
return key

# -------------------------------------------------------------------
Expand Down
14 changes: 14 additions & 0 deletions pandas/tests/indexing/test_loc.py
Original file line number Diff line number Diff line change
Expand Up @@ -1645,6 +1645,20 @@ def test_loc_setitem_mask_td64_series_value(self):
assert expected == result
tm.assert_frame_equal(df, df_copy)

@pytest.mark.parametrize("func", ["loc", "iloc"])
@pytest.mark.parametrize("rhs_func", [list, np.array])
def test_loc_setitem_boolean_list(self, func, rhs_func):
# GH#20438 testing specifically list key, not arraylike
ser = Series([0, 1, 2])
getattr(ser, func)[[True, False, True]] = rhs_func([5, 10])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also parameterize on the indexing type as well (e.g. list / array / pd.Index)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Index is not allowed for iloc, added the remaining ones

expected = Series([5, 1, 10])
tm.assert_series_equal(ser, expected)

df = DataFrame({"a": [0, 1, 2]})
getattr(df, func)[[True, False, True]] = rhs_func([[5], [10]])
expected = DataFrame({"a": [5, 1, 10]})
tm.assert_frame_equal(df, expected)


class TestLocListlike:
@pytest.mark.parametrize("box", [lambda x: x, np.asarray, list])
Expand Down