Skip to content

BUG: changed behavior of df.loc on multiindex since 1.3.0 #43599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
auderson opened this issue Sep 16, 2021 · 6 comments
Closed
2 of 3 tasks

BUG: changed behavior of df.loc on multiindex since 1.3.0 #43599

auderson opened this issue Sep 16, 2021 · 6 comments
Assignees
Labels
good first issue Indexing Related to indexing on series/frames, not to indexes themselves MultiIndex Needs Tests Unit test(s) needed to prevent regressions
Milestone

Comments

@auderson
Copy link
Contributor

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

df = pd.DataFrame(index=pd.MultiIndex.from_product([list('ab'), list('cd'), list('e')]), columns=['Val'])

df.loc[np.s_[:,'c',:]]

Issue Description

for versions later than 1.3, this selection will keep the second level in the index. But for versions earlier than 1.3, the 2nd level will be dropped:

image

image

image

Expected Behavior

I've read the release notes of 1.3 but can't find a documentation about this. Is this a changed behavior?

Installed Versions

INSTALLED VERSIONS

commit : 7c48ff4
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-156-generic
Version : #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.5
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 58.0.4
Cython : 0.29.23
pytest : 6.2.4
hypothesis : None
sphinx : 4.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : 0.9.3
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 3.0.1
IPython : 7.27.0
pandas_datareader: 0.9.0
bs4 : 4.9.3
bottleneck : None
fsspec : 0.8.5
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : 4.0.1
pyxlsb : None
s3fs : None
scipy : 1.7.1
sqlalchemy : 1.4.17
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
numba : 0.54.0

INSTALLED VERSIONS

commit : 73c6825
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-156-generic
Version : #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.3
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 58.0.4
Cython : 0.29.23
pytest : 6.2.4
hypothesis : None
sphinx : 4.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : 0.9.3
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 3.0.1
IPython : 7.27.0
pandas_datareader: 0.9.0
bs4 : 4.9.3
bottleneck : None
fsspec : 0.8.5
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : 4.0.1
pyxlsb : None
s3fs : None
scipy : 1.7.1
sqlalchemy : 1.4.17
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
numba : 0.54.0

@auderson auderson added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 16, 2021
@auderson
Copy link
Contributor Author

auderson commented Sep 16, 2021

image

df = pd.Series(index=pd.MultiIndex.from_product([list('ab'), list('cd'), list('e')]))

on both versions, if df is a series, the 2nd level will dropped

@auderson auderson changed the title BUG: changed behavior of .loc since 1.3.0 BUG: changed behavior of .loc on multiindex since 1.3.0 Sep 16, 2021
@auderson auderson changed the title BUG: changed behavior of .loc on multiindex since 1.3.0 BUG: changed behavior of df.loc on multiindex since 1.3.0 Sep 16, 2021
@attack68 attack68 added Indexing Related to indexing on series/frames, not to indexes themselves MultiIndex labels Sep 19, 2021
@attack68
Copy link
Contributor

On master this works as previous
Screen Shot 2021-09-19 at 21 53 07
:

@attack68 attack68 added Closing Candidate May be closeable, needs more eyeballs Needs Tests Unit test(s) needed to prevent regressions and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 19, 2021
@mroeschke mroeschke added good first issue and removed Bug Closing Candidate May be closeable, needs more eyeballs labels Oct 1, 2021
gamenerd457 added a commit to gamenerd457/pandas that referenced this issue Oct 4, 2021
@Ale-Cas
Copy link

Ale-Cas commented Feb 1, 2022

Hello, I am a first-time contributor. Is the issue still open?
I would like to work on it.

@ChrisAlbertsen
Copy link
Contributor

@Ale-Cas

From the looks of it the two prior PRs were closed due to inactivity, and no one else has been assigned the issue, so I think you can go ahead and work on it.
Looks like the latest of the two PRs got close to a working test, so you could take a look at it.

If you want the issue assigned to you try writing a single comment in here with the text "take" - this will make a bot assign the issue to you, letting other people know, that it is being worked on.

If you haven't read it from one end to another I'd suggest reading the "Contributing to pandas"-section. As a newbie I think its supportive :-)
https://pandas.pydata.org/docs/dev/development/contributing.html (you can also find a link in the repos landing page)

@rogimus
Copy link
Contributor

rogimus commented Mar 1, 2022

take

@jreback jreback added this to the 1.5 milestone Mar 1, 2022
@mroeschke
Copy link
Member

Closed by #46190

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Indexing Related to indexing on series/frames, not to indexes themselves MultiIndex Needs Tests Unit test(s) needed to prevent regressions
Projects
None yet
Development

No branches or pull requests

7 participants