BUG: DataFrame.stack(..., dropna=False) with partial MultiIndex.

[seth-p]

Closes #8844

Fixes DataFrame.stack(..., dropna=False) when the columns consist of a "partial" MultiIndex, i.e. one in which the labels don't reference all the levels.

[jreback]

maybe add a test which compares against a known result (e.g. that you are not using df.stack to create, e.g. construct the resultant/frame series. don't need for all, maybe just say [1,0].)

[seth-p]

Yep, still a work in progress...

[jreback]

this looks fine

cc @rockg

@seth-p want to do a quick speed test (maybe even add a vbench?)

[seth-p]

Afraid I'm not set up to run vbench (on windows). Can do simple speed tests Monday or Tuesday.

[seth-p]

Any comments on my question at #8844 (comment)?

[jreback]

You are stacking in the SAME order as presented yes (so if it happens to be sorted as is most of the time), then the output columns will be in that order, otherwise they will be in the indicated order, yes?

I think that is correct and fine. Granted you almost always have a sorted MultiIndex, but it shouldn't prevent it from working. I would guarantee that input order -> output order, and not required sortedness.

[seth-p]

Yes, I am keeping them in the same order they appear in the MultiIndex.levels -- which is not necessarily the order in which they appear in the MultiIndex.labels. This reproduces the existing behavior when all levels are used.

[seth-p]

The proposed change has no effect for a super-long DataFrame (as suggested in #8844 (comment)):
Existing code:

In [12]: df1 = pd.DataFrame(np.arange(500000*10).reshape(500000,10),
                            columns=pd.MultiIndex.from_tuples([(100*x, -y) for x in range(5)
                                                                           for y in range(2)],
                                                              names=['X', 'Y']))

In [13]: %timeit df1.stack()
10 loops, best of 3: 156 ms per loop

In [14]: %timeit df1.stack(level=0)
1 loops, best of 3: 254 ms per loop

In [15]: %timeit df1.stack(level=1)
10 loops, best of 3: 156 ms per loop

In [17]: %timeit df1.stack(level=[0,1])
1 loops, best of 3: 382 ms per loop

With this PR:

In [12]: %timeit df1.stack()
10 loops, best of 3: 156 ms per loop

In [13]: %timeit df1.stack(level=0)
1 loops, best of 3: 252 ms per loop

In [14]: %timeit df1.stack(level=1)
10 loops, best of 3: 156 ms per loop

In [15]: %timeit df1.stack(level=[0,1])
1 loops, best of 3: 381 ms per loop

For a super-wide DataFrame, the new code is a tad slower, but I think tolerably so:
Existing code:

In [3]: df = pd.DataFrame(np.arange(5*500000).reshape(5,500000), index=list('abcde'),
                          columns=pd.MultiIndex.from_tuples([(100*x, -y) for x in range(500)
                                                                         for y in range(1000)],
                                                            names=['X', 'Y']))

In [4]: %timeit df.stack()
1 loops, best of 3: 782 ms per loop

In [6]: %timeit df.stack(level=0)
1 loops, best of 3: 1.43 s per loop

In [8]: %timeit df.stack(level=1)
1 loops, best of 3: 758 ms per loop

In [18]: %timeit df.stack(level=[0,1])
1 loops, best of 3: 1.51 s per loop

With this PR:

In [4]: %timeit df.stack()
1 loops, best of 3: 862 ms per loop

In [6]: %timeit df.stack(level=0)
1 loops, best of 3: 1.53 s per loop

In [8]: %timeit df.stack(level=1)
1 loops, best of 3: 867 ms per loop

In [16]: %timeit df.stack(level=[0,1])
1 loops, best of 3: 1.62 s per loop

[jreback]

thanks!