PERF: Speed up DatetimeConverter by using Matplotlib's epoch2num when possible...

[agijsberts]

closes #6636

This fixes the performance bottleneck described in #6636 by using the vectorized epoch2num for DataIndex arrays. The included test ensures that the DatetimeConverter produces the same results as the Matplotlib's date2num for individual Timestamp objects as well as DatetimeIndex array objects.

[agijsberts]

Numpy's assert_almost_equal is necessary, since pandas' variant does not seem to allow to specify the required significance.

[jreback]

maybe let's modify pandas version to pass thru the significance chrck

[agijsberts]

The cython function decimal_almost_equal (https://github.com/pydata/pandas/blob/master/pandas/src/testing.pyx#L37) seems to do what we need, but I don't know how/where it is exposed.

Edit: Sorry, decimal_almost_equal implements just the scalar variant. The problem with pandas' assert_almost_equal is not only that it hardcodes the number of decimal, but also that the meaning of decimal is different. Besides numpy.testing.assert_almost_equal, also numpy.allclose could be an option here.

[agijsberts]

Will do. A quick question though: I just noticed there are already tests in pandas/tseries/test/test_converter.py. Would that be the correct place to add them instead, rather than pandas/tests/test_tseries.py?

[jreback]

ok to put then in the converter test suite

[jorisvandenbossche]

Maybe also add a vbench speed benchmark?

[agijsberts]

I created a simple vbench benchmark for datetimeindex conversion in vb_suite/timeseries.py. Before I commit though, 2 small questions:

1. What should I use as start_date? I would imagine the date that DatetimeConverter was added to the project, any way to find out when that was?
2. DatetimeConverter depends on matplotlib. Should I deal with possible import failures?

[jreback]

you can use 1/1/13 is fine
vbench will catch import failures so just import

[jreback]

you need to define rng in the setup. pls post a run of this vbench as well

[agijsberts]

Just to be sure, rng is already defined in common_setup; it is best practice to redefine it?

[jreback]

sorry....you are right..ok then

[agijsberts]

vbench run:

-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
datetimeindex_converter                      |   0.8059 | 411.0680 |   0.0020 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [dc95c4c] : Added vbench performance benchmark for speed up of DatetimeConverter.
Base   [4216178] : Merge pull request #5505 from jsexauer/fix5505

[jreback]

@TomAugspurger ok with this?

[TomAugspurger]

Yep this looks good. Should I merge it?

[jreback]

@agijsberts just need a release note

[agijsberts]

Latest commit implements comments and adds release notes. Ok like this?

[agijsberts]

Sorry, some tests fail in the last commit. If I replace from pandas import Series with from pandas.core.series import Series all is fine. Anybody knows why this is?

[jreback]

the first causes a circular import

[agijsberts]

Good to know, thanks!

[jreback]

@TomAugspurger you have the honors!

[TomAugspurger]

Thanks @agijsberts.