Inconsistent .loc slicing behaviour with NaNs in MultiIndex dataframe

[pepicello]

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np

a = pd.DataFrame([[1, 2, 3], [4, 5, 6], [7, 8, 9]]).set_index([0, 1])
b = pd.DataFrame([[1, np.nan, 3], [4, np.nan, 6], [7, np.nan, 9]]).set_index([0, 1])
a_idx = a.index[1]
b_idx = b.index[1]

>>> a.loc[a_idx:]
     2
0 1
4 5  6
7 8  9
>>> a.loc[:a_idx]
     2
0 1
1 2  3
4 5  6
>>> b.loc[b_idx:]
       2
0 1
7 NaN  9
>>> b.loc[:b_idx]
       2
0 1
1 NaN  3
4 NaN  6

Problem description

As above, slicing on MultiIndex has different behaviours with nans: it excludes the current index of selection from the slice. This is possibly because - rightly so - there is no match on NaNs when comparing the indices, but if that is the case, this should also not work:

>>> b.loc[b_idx]
2    6
Name: (4, nan), dtype: int64

Note: might be a duplicate of #25154, but OP does not slice directly on the MultiIndex with NaNs, so I am not sure if fixing that would also fix this issue.

Expected Output

>>> b.loc[b_idx:]
       2
0 1
4 NaN  6
7 NaN  9

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 0.25.1
numpy : 1.16.4
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.2
setuptools : 41.0.1
Cython : 0.29.13
pytest : 5.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.1.8
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.8.0
pandas_datareader: None
bs4 : None
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : 2.6.2
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : 1.3.7
tables : None
xarray : 0.12.3
xlrd : 1.2.0
xlwt : None
xlsxwriter : 1.1.8

[phofl]

Works now.

     2
0 1   
4 5  6
7 8  9
     2
0 1   
1 2  3
4 5  6
       2
0 1     
4 NaN  6
7 NaN  9
       2
0 1     
1 NaN  3
4 NaN  6

May need some tests

[ankushduacodes]

Hi @phofl, I would love to contribute to this issue, Please let me know if I may :)

[phofl]

Yes for sure. As long as it is unassigned, everyone is free to work on

[kasim95]

@ankushduacodes seems like you are not working on this issue as it is unassigned

[kasim95]

take

[ankushduacodes]

@kasim95 hi... I didn't know contributors could assign themselves as i am new to the repo. But fair enough, I will work on some other issue.

[kasim95]

Oh, i assumed that the issue is stale since there was no comment/update in the past 21 days. Are you still working on this issue? Let me know if you want to work on it so that I'll unassign myself.

[ankushduacodes]

I was just busy working on some other stuff. I have not started working on it. So its all you. Thank your for confirmation.😀