ENH: Allow parameters method and min_periods in DataFrame.corrwith()

[anthonyho]

Added new keyword parameters for DataFrame.corrwith(), which allows methods other than Pearson to be used. See #9490.

• closes Allow method keyword for DataFrame.corrwith() #9490
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

[jreback]

add versionsddes tag (and small comment here)

[jreback]

this is going to be very slow

we need to rework nancorr to do this instead

[anthonyho]

I think the new implementation (which calls nancorr which in turns calls numpy/scipy correlation functions) is actually significantly faster than the current implementation (manually computing Pearson correlation using DataFrame.mean(), DataFrame.sum(), and DataFrame.std())

For example:

Current implementation:

>>> import pandas as pd; import timeit
>>> pd.__version__
u'0.19.2'
>>> iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')
>>> timeit.timeit(lambda: iris.corrwith(iris), number=10000)
50.891642808914185
>>> timeit.timeit(lambda: iris.T.corrwith(iris.T), number=10000)
42.0677649974823

New implementation:

>>> import pandas as pd; import timeit
>>> pd.__version__
'0.19.0+539.g0b77680.dirty'
>>> iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')
>>> timeit.timeit(lambda: iris.corrwith(iris, method='pearson'), number=10000)
28.622286081314087
>>> timeit.timeit(lambda: iris.T.corrwith(iris.T, method='pearson'), number=10000)
21.898916959762573

I'm pretty new to this, so please let me know if I'm missing anything here.

[jreback]

look thru the benchmarks and pls add some asv as appropriate

include wide and talk data

on wide data this will be slower

[jreback]

can you update

[jreback]

can you rebase, add some benchmarks to asv and show them.

[jreback]

can you rebase and update?

[jreback]

closing as stale