Skip to content

BUG: DataFrame.pivot drops column level names when both rows and columns are multiindexed #36655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Oct 31, 2020

Conversation

arw2019
Copy link
Member

@arw2019 arw2019 commented Sep 26, 2020

Fixed by checking for multi-indexed columns in DataFrameGroupBy._wrap_aggregated_output

@arw2019 arw2019 force-pushed the GH36360 branch 4 times, most recently from 6004334 to fa879b7 Compare September 30, 2020 03:37

agg_axis = self._obj_with_exclusions._get_axis(1 - self.axis)
if isinstance(agg_axis, MultiIndex):
columns = Index([key.label for key in output], names=agg_axis.names)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this line alone work? I thought we fixed construction for this. alt does ensure_index work here? I really do not like special casing like this anywhere except in the index module itself (and even then its not great).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does! I've removed the if-else clause



def test_pivot_multiindexed_rows_and_cols():
# GH 36360
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have appropraite tests if index OR columns are a MI and the other is not?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do (in this file and in test_pivot.py)

What existing tests don't cover is the case when pivot_table produces a result with multiple MutliIndexed columns - added here

@jreback jreback added Bug MultiIndex Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Oct 7, 2020
@arw2019
Copy link
Member Author

arw2019 commented Oct 8, 2020

@jreback not sure you'll accept 390e9a9 (but atm it's needed to pass CI)

in dfc609f we're passing Index.names as the name kwarg

This means that for a simple index we end up with name=[['index_name']] in the Index constructor so I check for that and set name=['index_name']

@arw2019 arw2019 force-pushed the GH36360 branch 2 times, most recently from 390e9a9 to 8dd1161 Compare October 9, 2020 15:34
@@ -404,6 +404,11 @@ def __new__(
)
# other iterable of some kind
subarr = com.asarray_tuplesafe(data, dtype=object)

if isinstance(name, list) and len(name) == 1:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do not want to add this here, this is a very complicated routine; what are you trying to accomplish?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking for the functionality of Index._get_names() / MultiIndex._get_names()

Reverted this + using _get_names now

@arw2019 arw2019 force-pushed the GH36360 branch 3 times, most recently from 861baa9 to ffb4540 Compare October 11, 2020 20:47
@arw2019
Copy link
Member Author

arw2019 commented Oct 16, 2020

@jreback this one is ready for re-review. I now handle simple index and multiindex together with _set_names(aggregation_axis.names)

@jreback jreback added this to the 1.2 milestone Oct 16, 2020
@jreback jreback merged commit 02a7420 into pandas-dev:master Oct 31, 2020
@jreback
Copy link
Contributor

jreback commented Oct 31, 2020

thanks @arw2019

kesmit13 pushed a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020
ukarroum pushed a commit to ukarroum/pandas that referenced this pull request Nov 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug MultiIndex Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: MutliIndex names lost during pivot
2 participants