BUG: `KeyError: '[nan] not in index'` when using `nan` to index #52234

ssche · 2023-03-27T06:28:02Z

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas. (tested on 2.0.0rc1, assuming that's close enough to main)

Reproducible Example

>>> import numpy as np
>>> import pandas as pd
>>>
>>> df = pd.DataFrame(index=pd.Index([np.nan, 100.0, 200.0, 300, 100.0]))
>>> ix = pd.Index([100, np.nan])
>>> df.loc[ix]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6133, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: '[nan] not in index'

Issue Description

I'm using a row index which can contain nan values, but I'm unable to use it to index the rows in the dataframe.

However when I try to convert the index to a mask, it seems to be working:

>>> m = df.index.isin(ix)
>>> m
array([ True,  True, False, False,  True])
>>> df[m]
Empty DataFrame
Columns: []
Index: [nan, 100.0]

Expected Behavior

>>> df.loc[ix]

returns

Empty DataFrame
Columns: []
Index: [nan, 100.0]

just like df[m] does.

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.8.16.final.0
python-bits : 64
OS : Linux
OS-release : 6.1.18-200.fc37.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:09:14 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 1.5.3
numpy : 1.23.2
pytz : 2020.4
dateutil : 2.8.1
setuptools : 59.6.0
pip : 22.2.2
Cython : 0.29.32
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 0.9.6
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6
jinja2 : 2.11.2
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : 1.3.5
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 1.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
snappy : None
sqlalchemy : 1.3.23
tables : 3.7.0
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
zstandard : None
tzdata : None

The text was updated successfully, but these errors were encountered:

ssche · 2023-03-27T22:23:22Z

Interestingly, indexing with nan works, if it's the only element.

See this example for which I added an extra column 'a' compared to the initial post (to make it easier to follow):

>>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5]}, index=pd.Index([np.nan, 100.0, 200.0, 300, 100.0]))
>>>
>>> df
       a
NaN    1
100.0  2
200.0  3
300.0  4
100.0  5

Indexing with np.nan works as expected...

>>> df.loc[np.nan, 'a']
1

... just like like indexing with a number...

>>> df.loc[200.0, 'a']
3

... or multiple index elements...

>>> df.loc[[100, 200], 'a']
100.0    2
100.0    5
200.0    3
Name: a, dtype: int64

... but as soon as np.nan is part of to the index, a KeyError is raised:

>>> df.loc[[100, 200, np.nan], 'a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1067, in __getitem__
    return self._getitem_tuple(key)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1247, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 991, in _getitem_lowerdim
    return getattr(section, self.name)[new_key]
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "env/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "env/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6133, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: '[nan] not in index'

As soon as np.nan is used in a list (or similar), the KeyError occurs (Note: df.loc[np.nan, 'a'] works just fine).

>>> df.loc[[np.nan], 'a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1067, in __getitem__
    return self._getitem_tuple(key)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1247, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 991, in _getitem_lowerdim
    return getattr(section, self.name)[new_key]
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Float64Index([nan], dtype='float64')] are in the [index]"

ssche · 2023-03-27T22:31:46Z

This is also an issue in pandas 2.0.0rc1

>>> df.loc[[np.nan], 'a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1097, in __getitem__
    return self._getitem_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1280, in _getitem_tuple
    return self._getitem_lowerdim(tup)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1024, in _getitem_lowerdim
    return getattr(section, self.name)[new_key]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1103, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1332, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1272, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1462, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexes/base.py", line 5873, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexes/base.py", line 5932, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index([nan], dtype='object')] are in the [index]"

INSTALLED VERSIONS
------------------
commit           : c2a7f1ae753737e589617ebaaff673070036d653
python           : 3.11.2.final.0
python-bits      : 64
OS               : Linux
OS-release       : 6.1.18-200.fc37.x86_64
Version          : #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:09:14 UTC 2023
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_AU.UTF-8
LOCALE           : en_AU.UTF-8

pandas           : 2.0.0rc1
numpy            : 1.24.2
pytz             : 2023.2
dateutil         : 2.8.2
setuptools       : 62.6.0
pip              : 22.2.2
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
matplotlib       : None
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : None
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
zstandard        : None
tzdata           : None
qtpy             : None
pyqt5            : None

ltauro · 2023-03-28T01:53:39Z

take

ssche · 2023-03-28T23:22:34Z

It seems that "all" that's missing to resolve this is to remove the check

    self._raise_if_missing(keyarr, indexer, axis_name)

in pandas/core/indexes/base.py, line 6070, in _get_indexer_strict

>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5]}, index=pd.Index([np.nan, 100.0, 200.0, 300, 100.0]))
>>> df.loc[[np.nan], 'a']
100.0   NaN
Name: a, dtype: float64
>>> 
>>> 
>>> df.loc[[np.nan, np.nan], 'a']
100.0   NaN
100.0   NaN
Name: a, dtype: float64

Can this be done in pandas 1.5.x?

ssche · 2023-03-28T23:24:29Z

In the interim, would someone be able to propose a workaround that does the same thing (Index and MultiIndex)?

My current workaround is to monkeypatch Index, but that may hide required exceptions (although in base.py I couldn't find another place where _raise_if_missing is called):

from pandas.core.indexes.base import Index

def _ignore_if_missing(self, *args, **kwargs):
    pass

setattr(Index, '_raise_if_missing', _ignore_if_missing)

ltauro · 2023-04-12T18:23:07Z

@ssche I was able to reproduce this error. I am going to work on fixing this and submit a pull request as soon as possible.

…v#52234

vanxLord90 · 2023-06-07T15:30:29Z

i had the same issue, i downloaded pandas in May from pip, what exactly is the solution to this issue?

* Copied fix from https://github.com/ltauro/pandas/tree/nan-not-in-index and applied review comments

* Fixed #52234 * Copied fix from https://github.com/ltauro/pandas/tree/nan-not-in-index and applied review comments * Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <[email protected]> --------- Co-authored-by: Matthew Roeschke <[email protected]>

* Fixed pandas-dev#52234 * Copied fix from https://github.com/ltauro/pandas/tree/nan-not-in-index and applied review comments * Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <[email protected]> --------- Co-authored-by: Matthew Roeschke <[email protected]>

ssche added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 27, 2023

ssche added Indexing Related to indexing on series/frames, not to indexes themselves Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Mar 27, 2023

github-actions bot assigned ltauro Mar 28, 2023

ltauro added a commit to ltauro/pandas that referenced this issue Apr 13, 2023

Modify get_indexer_non_unique to account for nan keys - Bug pandas-de…

bd8ee57

…v#52234

ltauro mentioned this issue Apr 13, 2023

BUG: Modify get_indexer_non_unique to account for nan keys #52655

Closed

5 tasks

topper-123 removed the Needs Triage Issue that has not been reviewed by a pandas team member label May 15, 2023

ssche added a commit to ssche/pandas that referenced this issue Jun 16, 2023

Fixed pandas-dev#52234

31844d7

* Copied fix from https://github.com/ltauro/pandas/tree/nan-not-in-index and applied review comments

ssche mentioned this issue Jun 16, 2023

Fixed #52234 #53698

Merged

5 tasks

ssche added a commit to ssche/pandas that referenced this issue Jun 23, 2023

Fixed pandas-dev#52234

b0f1373

* Copied fix from https://github.com/ltauro/pandas/tree/nan-not-in-index and applied review comments

mroeschke closed this as completed in #53698 Jun 23, 2023

ssche added this to the 2.1 milestone Aug 7, 2023

ssche self-assigned this Aug 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: `KeyError: '[nan] not in index'` when using `nan` to index #52234

BUG: `KeyError: '[nan] not in index'` when using `nan` to index #52234

ssche commented Mar 27, 2023 •

edited

Loading

INSTALLED VERSIONS

ssche commented Mar 27, 2023 •

edited

Loading

ssche commented Mar 27, 2023

ltauro commented Mar 28, 2023

ssche commented Mar 28, 2023

ssche commented Mar 28, 2023 •

edited

Loading

ltauro commented Apr 12, 2023

vanxLord90 commented Jun 7, 2023

BUG: KeyError: '[nan] not in index' when using nan to index #52234

BUG: KeyError: '[nan] not in index' when using nan to index #52234

Comments

ssche commented Mar 27, 2023 • edited Loading

Pandas version checks

Reproducible Example

Issue Description

Expected Behavior

Installed Versions

INSTALLED VERSIONS

ssche commented Mar 27, 2023 • edited Loading

ssche commented Mar 27, 2023

ltauro commented Mar 28, 2023

ssche commented Mar 28, 2023

ssche commented Mar 28, 2023 • edited Loading

ltauro commented Apr 12, 2023

vanxLord90 commented Jun 7, 2023

BUG: `KeyError: '[nan] not in index'` when using `nan` to index #52234

BUG: `KeyError: '[nan] not in index'` when using `nan` to index #52234

ssche commented Mar 27, 2023 •

edited

Loading

ssche commented Mar 27, 2023 •

edited

Loading

ssche commented Mar 28, 2023 •

edited

Loading