Quick Frame Shift Implementation

[gouthambs]

Fix for issue #5609

[jreback]

ok....when you run the tests...lots of failure....pls investigate....

[gouthambs]

I see that the indexer in pandas/core/internal is unused. Removing this argument will mean we can take out _shift_indexer. Should I remove the indexer argument in pandas/core/internal ?

[jreback]

yep indexer can go

[gouthambs]

Taking indexer out clashes with the shift in the SparseBlock. Should the same roll logic work here ? Any idea what are the other shift calls that may need to be changed here?

[jreback]

So I would take the indexer out of the Block.shift method as you don't need it their. Take the argument out of the SparseBlock.shfit as well, and compute the indexer their (move the _shift_indexer code into that function).

[gouthambs]

The indexer is obtained as:
indexer = com._shift_indexer(len(self._get_axis(axis)), periods)

The first argument to _shift_indexer requires the length of the axis using _get_axis method defined in NDFrame. Is there an equivalent call inside Block?

Is it the same as saying len(self.values) or len(self.values.T)?

[jreback]

just take the guts of _shift_indexer (and remove it from common.py). The length it is looking for is len(self.values) i believe.

[jreback]

can you post the results of the vbench?

If you can rebase/squash would be great

[gouthambs]

Are there any docs on how to setup and run vbench? I couldn't get it running! I tested the old and new using timeit.

[jreback]

go to vb_suite; run ./test_perf.sh -b master -t HEAD

[TomAugspurger]

I put some notes I had written up for statsmodels at https://github.com/pydata/pandas/wiki/Performance-Testing

@jreback hopefully everything is accurate. I'll add more later.

[jreback]

@TomAugspurger awesome just what was needed!

[gouthambs]

@jreback I couldn't run the test like you said. Perhaps it has to do with the fact that I use a Windows system. Can some one run this? I do have the squashed commit here.

[jreback]

ok just post the results them of the benchmark before and after the change

[gouthambs]

Test code

import pandas as pd
import numpy as np
import timeit

xdim = 10000
ydim = 500
repeats = 10
df = pd.DataFrame(np.random.rand(xdim,ydim))

# axis 0 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=0)
e = timeit.default_timer()
print "Axis 0 shift",e-s

# axis 1 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=1)
e = timeit.default_timer()
print "Axis 1 shift",e-s

Results

V0.13.1

Axis 0 shift 0.958936203491
Axis 1 shift: IndexError: index 500 is out of bounds for size 500

----------------------------------------------------------------------------------
V0.13.1-330-gb562eb2

Axis 0 shift 0.1622
Axis 1 shift 0.3466 

[gouthambs]

@jreback I am not sure if the squashed set is supposed to have all these commits. Also I notice that after squashing, the shift has the axis in reverse. Not sure what happened here!!!!!

[jreback]

try

git  rebase upstream/master
git push yourforkname yourbranchname -f 

[gouthambs]

@jreback The rebase was a little confusing. I am getting a lot of conflicts. Not sure why! Sorry I am a little new to git. My squashed commits are in b562eb2 + ff18ac9 shown above. Any chance we can work with this. As of the last commit, I checked that all the tests were passing.

[jreback]

ok...here's what you need to do:

git checkout -b new_branch --track origin/master
git cherry-pick <commit1> thisbranch
git cherry-pick <commit2> thisbranch
git push myfork thisbranch

where and 2 are your commits from above

this takes the commits out and puts them in a new branch

push this branch up and do a new PR, closing this one.

[gouthambs]

Please refer to #6672 for the new pull request.