BUG: slicing a MultiIndex does not preserve the sequence of the index since pandas 1.2.0rc0

[johannes-mueller]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Between 1.1.5 and 1.2.0rc2 the behavior of slicing MultiIndexes has changed. Up to 1.1.5 slicing a MultiIndex like df.loc[(slice(None), somel_items), :] preserved the sequence of the sliced DataFrame. From 1.2.0rc0 the sequence is changed.

Code Sample

import pandas as pd

print("pandas version %s" % pd.__version__)

index = pd.MultiIndex.from_tuples([(1, 1), (1, 2), (1, 7), (1, 6),
                                   (2, 2), (2, 3), (2, 8), (2, 7)])

all_items = index.get_level_values(1) # all items from level 1

df = pd.DataFrame({'x': range(8)}, index=index)


df_sliced = df.loc[(slice(None), all_items), :]
# df_sliced, should be identical with df, as all_items contains all items from level 1

print(df_sliced)

pd.testing.assert_frame_equal(df, df_sliced)
# works if and only if pd.__version__ < 1.2.0rc0

print("Success")

Problem description

Running the sample code in 1.1.5 gives the following output:

pandas version 1.1.5
     x
1 1  0
  2  1
  7  2
  6  3
2 2  4
  3  5
  8  6
  7  7
Success

whereas in 1.2.4 it gives

pandas version 1.2.4
     x
1 1  0
  6  3
  2  1
2 2  4
  3  5
  8  6
1 7  2
2 7  7
Traceback (most recent call last):
  File "tmp/pandas_demo.py", line 19, in <module>
    pd.testing.assert_frame_equal(df, df_sliced)
  File "/home/jmu3si/Devel/pylife/.venv/lib/python3.8/site-packages/pandas/_testing.py", line 1657, in assert_frame_equal
    assert_index_equal(
  File "/home/jmu3si/Devel/pylife/.venv/lib/python3.8/site-packages/pandas/_testing.py", line 805, in assert_index_equal
    assert_index_equal(
  File "/home/jmu3si/Devel/pylife/.venv/lib/python3.8/site-packages/pandas/_testing.py", line 825, in assert_index_equal
    _testing.assert_almost_equal(
  File "pandas/_libs/testing.pyx", line 46, in pandas._libs.testing.assert_almost_equal
  File "pandas/_libs/testing.pyx", line 161, in pandas._libs.testing.assert_almost_equal
  File "/home/jmu3si/Devel/pylife/.venv/lib/python3.8/site-packages/pandas/_testing.py", line 1073, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: MultiIndex level [0] are different

MultiIndex level [0] values are different (25.0 %)
[left]:  Int64Index([1, 1, 1, 1, 2, 2, 2, 2], dtype='int64')
[right]: Int64Index([1, 1, 1, 2, 2, 2, 1, 2], dtype='int64')

I am not sure if this is necessarily a problem, I stumbled across it because a test suite that used pd.testing.assert_frame_equal() failed due to this. So either the actual sequence should be preserved when slicing a DataFrame or pd.testing.assert_frame_equal() should not fail if the sequence is shuffled (but the index is correct).

Expected Output

As discussed above.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-71-lowlatency
Version : #79-Ubuntu SMP PREEMPT Wed Mar 24 12:38:51 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : de_DE.UTF-8
LOCALE : de_DE.UTF-8

pandas : 1.2.4 (resp. 1.1.5)
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 50.3.0.post20201006
Cython : 0.29.23
pytest : 6.2.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : None
tables : None
tabulate : None
xarray : 0.17.0
xlrd : None
xlwt : None
numba : None

[phofl]

The reordering is expected since you've got duplicates in the second level. But the exact result is a bit unexpected. I would have expected the same as in

df.loc[(slice(None), all_items.unique()), :]

1 1  0
  2  1
2 2  4
1 7  2
2 7  7
1 6  3
2 3  5
  8  6

see #31330

[johannes-mueller]

Ok, I get the logic behind it, that the resulting DataFrame is ordered in the same way as the index by which I sliced. Kinda makes sense, so I see its a "works as intended".

Yet, it is a little bit unsatisfying because it leaves me with the question, how to efficiently select items of a pandas object while preserving the original sequence. What I found fo that is something like

s = pd.Series([1,2,3,4,5], index=[0,9,8,7,6])
s.iloc[np.sort(s.index.get_indexer([0,8,9]))]

which results in

0    1
9    2
8    3
dtype: int64

as desired. However, it has the np.sort() call, which is actually only to preserve a sequence that is already there.

Any better ideas for that?

[phofl]

Intersection with sort=False is first thing which comes to mind.

Loc should in theory select in the order your indexer is, so your incoming sequence has to be the same order somehow.

[johannes-mueller]

Intersection with sort=False is first thing which comes to mind.

Cool. That seems to do it.

Thanks a lot, closing issue

[phofl]

The ordering is wrong for duplicates like in your example, will fix this shortly but lets leave open till then

[johannes-mueller]

Just for completeness: in order to slice a MultiIndex while preserving the order like I tried in my demo above

df_sliced = df.loc[df.index.isin(all_items, level=1), :]

does the trick.