Adding Multiindex support to dataframe pivot function(Fixes #21425) #21467

NikhilKumarM · 2018-06-13T17:09:24Z

closes BUG: DataFrame.pivot fails on multiple columns to set as index #21425
tests added/passed:
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew : Now dataframe.pivot function can take multiple columns as index
Added extra case to handle indexing on multiple colums in dataframe.pivot function

codecov · 2018-06-13T20:52:58Z

Codecov Report

Merging #21467 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #21467      +/-   ##
==========================================
+ Coverage    91.9%    91.9%   +<.01%     
==========================================
  Files         154      154              
  Lines       49657    49667      +10     
==========================================
+ Hits        45638    45648      +10     
  Misses       4019     4019

Flag	Coverage Δ
#multiple	`90.28% <100%> (ø)`	⬆️
#single	`41.89% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/reshape/reshape.py	`99.79% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc45fba...faab956. Read the comment docs.

gfyoung · 2018-06-13T23:03:32Z

@NikhilKumarM : Off to a good start here! Going to need a whatsnew entry + test (can use the example that @jorisvandenbossche gave).

NikhilKumarM · 2018-06-14T00:24:43Z

Thanks @gfyoung . I have added test cases and whatsnew entry

gfyoung · 2018-06-14T00:58:53Z

@NikhilKumarM : You actually need to add the whatsnew entry (search for the v0.23.2.txt file and test cases to your PR changes 😄 .

jorisvandenbossche · 2018-06-14T14:14:57Z

@NikhilKumarM Did you push your new changes? Because they do not show up on github here. Normally you only need to add new commits to the same branch, push that to github, and the PR here should get updated.

gfyoung · 2018-06-14T17:40:46Z

@jorisvandenbossche : If you look at the PR description, you can see that @NikhilKumarM accidentally added them there. If this PR goes unresponsive, I can bring it to the finish line.

…umns as index

pep8speaks · 2018-06-15T00:36:49Z

Hello @NikhilKumarM! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on July 01, 2018 at 13:52 Hours UTC

gfyoung

Nice! LGTM!

cc @jorisvandenbossche

jorisvandenbossche

Can you also add the ability to pass multiple values for columns ?
Can you also update the docstring of DataFrame.pivot ?

jorisvandenbossche · 2018-06-15T08:37:50Z

doc/source/whatsnew/v0.23.2.txt

@@ -66,7 +66,7 @@ Bug Fixes

 **Reshaping**

-
+- Bug in: DataFrame.pivot() function where error was raised when multiple columns are set as index


Can you move this to the 0.24.0.txt file? And you can mention it in the Enhancements section

(as this is an new feature, not really a bug fix, I think, as it was just not yet implemented)

jorisvandenbossche · 2018-06-15T08:40:08Z

pandas/core/reshape/reshape.py

@@ -392,12 +392,22 @@ def pivot(self, index=None, columns=None, values=None):
        cols = [columns] if index is None else [index, columns]
        append = index is None
        indexed = self.set_index(cols, append=append)
+    # adding the support for multi-index in pivot function


this comment is not really needed I think

jorisvandenbossche · 2018-06-15T08:40:43Z

pandas/tests/reshape/test_pivot.py

@@ -283,6 +283,23 @@ def test_pivot_multi_functions(self):
        expected = concat([means, stds], keys=['mean', 'std'], axis=1)
        tm.assert_frame_equal(result, expected)

+    # adding the test case for multiple columns as index (#21425)


please move this comment to the first line of the test function

jorisvandenbossche · 2018-06-15T08:45:24Z

pandas/tests/reshape/test_pivot.py

+        expected = DataFrame([[0, 1], [2, 3], [4, 5], [6, 7]],
+                             exp_index,
+                             exp_columns)
+        tm.assert_frame_equal(result, expected)


Can you add an example where the values argument is not mentioned? eg df.pivot(index=['lev1', 'lev2'], columns='lev3')
(I think this will not yet work, but should work)

NikhilKumarM · 2018-06-15T16:37:10Z

Sure. I will try to add ability to pass multiple values for columns.
I have not worked on docstrings before. Is there any good resource to start doing that? @jorisvandenbossche

WillAyd · 2018-06-15T16:51:15Z

@NikhilKumarM there is a section in the contributing guide focused on docstrings:

https://pandas.pydata.org/pandas-docs/stable/contributing_docstring.html

jreback · 2018-06-15T17:23:39Z

pandas/core/reshape/reshape.py

+            index = MultiIndex.from_arrays([index, self[columns]])
+        # added this case to handle multi-index
+        elif isinstance(index, list):
+            indexes = []


can you make this a list-comprehension

jreback · 2018-06-15T17:23:58Z

pandas/core/reshape/reshape.py

    else:
        if index is None:
            index = self.index
+            index = MultiIndex.from_arrays([index, self[columns]])
+        # added this case to handle multi-index


can you make a comment that reflects what is here (not added)

jreback · 2018-06-15T17:24:48Z

doc/source/whatsnew/v0.24.0.txt

@@ -16,6 +16,8 @@ Other Enhancements
 - :func:`Series.mode` and :func:`DataFrame.mode` now support the ``dropna`` parameter which can be used to specify whether NaN/NaT values should be considered (:issue:`17534`)
 - :func:`to_csv` now supports ``compression`` keyword when a file handle is passed. (:issue:`21227`)
 - :meth:`Index.droplevel` is now implemented also for flat indexes, for compatibility with MultiIndex (:issue:`21115`)
+- :func:'Dataframe.pivot' now supports multiple columns as index. (:issue:'21425')


'columns as an index'

Sure. I will make those changes.

NikhilKumarM · 2018-06-21T15:49:33Z

Why is continuous-integration/travis-ci failing?

gfyoung · 2018-06-21T16:51:10Z

Why is continuous-integration/travis-ci failing?

@NikhilKumarM : Restarted Travis for you. Failure unrelated to PR.

jreback

small change otherwise lgtm. does the doc string need updating?

jreback · 2018-06-22T23:33:37Z

pandas/core/reshape/reshape.py

    else:
        if index is None:
            index = self.index
+            index = MultiIndex.from_arrays([index, self[columns]])
+        elif isinstance(index, list):


use is_list_like instead here

Sure. I will do it.

NikhilKumarM · 2018-06-30T17:47:07Z

@jreback @gfyoung @jorisvandenbossche
Could you please check my last commit and suggest changes. I have modified the code to handle when there is no 'values' argument for 'pivot' function and also added a test case for that. But I do not know if have done it in an optimal way.

gfyoung · 2018-07-01T05:47:16Z

@NikhilKumarM : Got a linting error on Travis:

https://travis-ci.org/pandas-dev/pandas/jobs/398632775#L2904

NikhilKumarM · 2018-07-01T15:17:38Z

@gfyoung Fixed the linting error.

NikhilKumarM · 2018-08-09T12:03:08Z

@gfyoung @jreback Could you please give me your feedback on my last commit?

jreback

pls add a whatsnew note for 0.24. I guess this is a bug fix, though was this documented before? if so its an enhancement. Can you also add an exampl ein reshape.rst with this. And in the pivot doc-string as well.

jreback · 2018-08-09T14:47:10Z

pandas/core/reshape/reshape.py

@@ -395,15 +395,29 @@ def pivot(self, index=None, columns=None, values=None):
    See DataFrame.pivot
    """
    if values is None:
-        cols = [columns] if index is None else [index, columns]
+        if index is None:
+            cols = [columns]


do we have tests for each one of these paths here? e.g. branching a lot more now

Yes, this is an enhancement. I have added the test cases and need to update the docs. Thanks for your time.

jreback · 2018-08-09T14:48:38Z

also needs to rebase

jreback · 2018-11-04T16:14:18Z

can you merge master

NikhilKumarM · 2018-11-05T15:01:43Z

Sure!

jreback

can you merge master & add a whatsnew note.

jreback · 2018-11-11T23:55:38Z

pandas/core/reshape/reshape.py

-        cols = [columns] if index is None else [index, columns]
+        if index is None:
+            cols = [columns]
+        else:


can you add a comment explaining what is going on here, ideally we would have a comment for each branch of the if/else

jreback · 2018-11-18T22:06:48Z

can you update

jreback · 2018-12-09T20:11:55Z

can you merge master

NikhilKumarM · 2018-12-11T14:40:22Z

I think pivot function has been removed from reshape.py and put in pivot.py. I will rebase with the master and make necessary changes by this weekend. Thank you !!

jreback · 2019-01-05T15:37:27Z

can you merge master and update

WillAyd · 2019-02-06T03:30:55Z

Closing as stale - ping if you'd like to reopen and continue working this

Adding Multiindex support to dataframe pivot function(Fixes #21425)

79dacb2

gfyoung added Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Jun 13, 2018

gfyoung requested a review from jorisvandenbossche June 13, 2018 23:03

gfyoung modified the milestones: 0.24.0, 0.23.2 Jun 14, 2018

jorisvandenbossche modified the milestones: 0.23.2, 0.24.0 Jun 14, 2018

NikhilKumarM added 2 commits June 14, 2018 19:12

added whatsnew entry in v0.23.2.txt file

f95f854

Added test case for testing dataframe pivot function with multile col…

100a4bc

…umns as index

gfyoung approved these changes Jun 15, 2018

View reviewed changes

NikhilKumarM added 2 commits June 14, 2018 21:00

Made changes as per PEP8 requirements

ecc19d2

changed the type of exp_columns to Index

7f6c09c

jorisvandenbossche reviewed Jun 15, 2018

View reviewed changes

NikhilKumarM added 2 commits June 15, 2018 12:11

Moved whatsnew entry from v0.23.2.txt to v0.24.0.txt

34fb002

Removed the comments that are not required

787b289

jreback requested changes Jun 15, 2018

View reviewed changes

NikhilKumarM added 2 commits June 15, 2018 16:16

corrected whatsnew entry

358e0ea

Changed indexes to list comprehension

811ab86

jreback requested changes Jun 22, 2018

View reviewed changes

NikhilKumarM and others added 4 commits June 29, 2018 11:08

used is_list_like to check if input is list

7b6b5a4

resolving merge conflict in whatsnew

3bcfc16

trying to solve merge conflict

0234b38

Merge branch 'master' into master

a8f80ed

NikhilKumarM added 2 commits June 30, 2018 17:38

Handling the case when there is no values argument in pivot func

7a050d6

added one more test case for pivot function with multi column index

beb768b

added whitespace to remove linting error

faab956

jreback reviewed Aug 9, 2018

View reviewed changes

jreback requested changes Nov 11, 2018

View reviewed changes

jreback removed this from the 0.24.0 milestone Nov 18, 2018

WillAyd closed this Feb 6, 2019

Adding Multiindex support to dataframe pivot function(Fixes #21425) #21467

Adding Multiindex support to dataframe pivot function(Fixes #21425) #21467

Conversation

NikhilKumarM commented Jun 13, 2018 • edited Loading

codecov bot commented Jun 13, 2018 • edited Loading

Codecov Report

gfyoung commented Jun 13, 2018

NikhilKumarM commented Jun 14, 2018 • edited Loading

gfyoung commented Jun 14, 2018

jorisvandenbossche commented Jun 14, 2018

gfyoung commented Jun 14, 2018 • edited Loading

pep8speaks commented Jun 15, 2018 • edited Loading

Comment last updated on July 01, 2018 at 13:52 Hours UTC

gfyoung left a comment

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikhilKumarM commented Jun 15, 2018 • edited Loading

WillAyd commented Jun 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikhilKumarM commented Jun 21, 2018 • edited Loading

gfyoung commented Jun 21, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikhilKumarM commented Jun 30, 2018

gfyoung commented Jul 1, 2018

NikhilKumarM commented Jul 1, 2018 • edited Loading

NikhilKumarM commented Aug 9, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 9, 2018

jreback commented Nov 4, 2018

NikhilKumarM commented Nov 5, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 18, 2018

jreback commented Dec 9, 2018

NikhilKumarM commented Dec 11, 2018

jreback commented Jan 5, 2019

WillAyd commented Feb 6, 2019

NikhilKumarM commented Jun 13, 2018 •

edited

Loading

codecov bot commented Jun 13, 2018 •

edited

Loading

NikhilKumarM commented Jun 14, 2018 •

edited

Loading

gfyoung commented Jun 14, 2018 •

edited

Loading

pep8speaks commented Jun 15, 2018 •

edited

Loading

NikhilKumarM commented Jun 15, 2018 •

edited

Loading

NikhilKumarM commented Jun 21, 2018 •

edited

Loading

NikhilKumarM commented Jul 1, 2018 •

edited

Loading