Skip to content

BUG: IndexError when slicing a series, if a level of MultiIndex contains only NaN values #42055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
sziem opened this issue Jun 16, 2021 · 2 comments · Fixed by #42335
Closed
2 of 3 tasks
Labels
Bug Indexing Related to indexing on series/frames, not to indexes themselves MultiIndex
Milestone

Comments

@sziem
Copy link

sziem commented Jun 16, 2021

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample, a copy-pastable example

import pandas as pd

s = pd.Series([0, 1], index=[[pd.NA, pd.NA], ["bar", "baz"]])
s.loc[pd.IndexSlice[:, "bar"]]
Traceback
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-1-c6419722729f> in <module>
      2 
      3 s = pd.Series([0, 1], index=[[pd.NA, pd.NA], ["bar", "baz"]])
----> 4 s.loc[pd.IndexSlice[:, "bar"]]

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexing.py in __getitem__(self, key)
    887                     # AttributeError for IntervalTree get_value
    888                     return self.obj._get_value(*key, takeable=self._takeable)
--> 889             return self._getitem_tuple(key)
    890         else:
    891             # we by definition only have the 0th axis

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_tuple(self, tup)
   1058     def _getitem_tuple(self, tup: Tuple):
   1059         with suppress(IndexingError):
-> 1060             return self._getitem_lowerdim(tup)
   1061 
   1062         # no multi-index, so validate all of the indexers

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_lowerdim(self, tup)
    789         # we may have a nested tuples indexer here
    790         if self._is_nested_tuple_indexer(tup):
--> 791             return self._getitem_nested_tuple(tup)
    792 
    793         # we maybe be using a tuple to represent multiple dimensions here

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_nested_tuple(self, tup)
    845                 raise ValueError("Too many indices")
    846             with suppress(IndexingError):
--> 847                 return self._handle_lowerdim_multi_index_axis0(tup)
    848 
    849             # this is a series with a multi-index specified a tuple of

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexing.py in _handle_lowerdim_multi_index_axis0(self, tup)
   1078         try:
   1079             # fast path for series or for tup devoid of slices
-> 1080             return self._get_label(tup, axis=axis)
   1081         except (TypeError, InvalidIndexError):
   1082             # slices are unhashable

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexing.py in _get_label(self, label, axis)
   1071     def _get_label(self, label, axis: int):
   1072         # GH#5667 this will fail if the label is not present in the axis.
-> 1073         return self.obj.xs(label, axis=axis)
   1074 
   1075     def _handle_lowerdim_multi_index_axis0(self, tup: Tuple):

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/generic.py in xs(self, key, axis, level, drop_level)
   3731         if isinstance(index, MultiIndex):
   3732             try:
-> 3733                 loc, new_index = index._get_loc_level(
   3734                     key, level=0, drop_level=drop_level
   3735                 )

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexes/multi.py in _get_loc_level(self, key, level, drop_level)
   3064                     indexer = slice(None, None)
   3065                 ilevels = [i for i in range(len(key)) if key[i] != slice(None, None)]
-> 3066                 return indexer, maybe_mi_droplevels(indexer, ilevels, drop_level)
   3067         else:
   3068             indexer = self._get_level_indexer(key, level=level)

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexes/multi.py in maybe_mi_droplevels(indexer, levels, drop_level)
   2981             for i in sorted(levels, reverse=True):
   2982                 try:
-> 2983                     new_index = new_index._drop_level_numbers([i])
   2984                 except ValueError:
   2985 

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexes/base.py in _drop_level_numbers(self, levnums)
   1639             # set nan if needed
   1640             mask = new_codes[0] == -1
-> 1641             result = new_levels[0].take(new_codes[0])
   1642             if mask.any():
   1643                 result = result.putmask(mask, np.nan)

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/indexes/base.py in take(self, indices, axis, allow_fill, fill_value, **kwargs)
    749         # Note: we discard fill_value and use self._na_value, only relevant
    750         #  in the case where allow_fill is True and fill_value is not None
--> 751         taken = algos.take(
    752             self._values, indices, allow_fill=allow_fill, fill_value=self._na_value
    753         )

~/.cache/pypoetry/virtualenvs/env-d6X_GUCN-py3.9/lib/python3.9/site-packages/pandas/core/algorithms.py in take(arr, indices, axis, allow_fill, fill_value)
   1655     else:
   1656         # NumPy style
-> 1657         result = arr.take(indices, axis=axis)
   1658     return result
   1659 

IndexError: cannot do a non-empty take from an empty axes.

Problem description

The example runs as expected, when the index contains a non-NaN value such as "foo"

import pandas as pd

s3 = pd.Series([0, 1], index=[[pd.NA, "foo"], ["bar", "baz"]])
s3[pd.IndexSlice[:, "bar"]]

Expected Output

Output identical to
pd.Series(0, index=[pd.NA])

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.41-1-MANJARO
Version : #1 SMP PREEMPT Fri May 28 19:10:32 UTC 2021
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.utf8
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 54.1.2
Cython : None
pytest : 6.2.4
hypothesis : 6.14.0
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.24.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : None

@sziem sziem added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 16, 2021
@jreback
Copy link
Contributor

jreback commented Jun 16, 2021

try on 1.3.0rc1

@sziem
Copy link
Author

sziem commented Jun 16, 2021

I get the same error on 1.3.0rc1

@phofl phofl added Indexing Related to indexing on series/frames, not to indexes themselves and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 16, 2021
@jreback jreback added this to the 1.4 milestone Jul 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Indexing Related to indexing on series/frames, not to indexes themselves MultiIndex
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants