Updates for asv suite

[qwhelan]

@jorisvandenbossche As you requested in #10849, here's the current state of affairs. The changes are:

• First patch:
  • Fix the translation script so that functions defined in the vbench setup string get defined as classmethods and called appropriately. This fixes most failing tests.
  • Some backwards-compatibility fixes around imports. Raising NotImplementedError inside setup() is equivalent to SkipTest, so I've tried to do that in a couple necessary cases (such as things using test_parallel)
  • Speaking of test_parallel, this required a bit of hackery to support not-existing. I implemented a no-op decorator by that name so that the actual definition of test cases wouldn't fail. We simply raise NotImplementedError rather than test without it, however.
• Second patch:
  • Support Python 3, the changes are essentially what I mentioned elsewhere.
  • from pandas_vb_common import * -> from .pandas_vb_common import *
  • Replace xrange with range and add from pandas.compat import range in pandas_vb_common
  • Replace string.uppercase with string.ascii_uppercase
• Third patch
  • asv instructions in contributing.rst, largely lifted from the vbench description. The current examples are minimalist, but most comparable to what vbench outputs.
• Fourth patch:
  • Re-run the translation script. I think some other additions may be included in the first patch due to having run it a few times while making changes.

cc @jreback @TomAugspurger

[jreback]

I would put all of these in a base class, maybe Base and inherit all tests from it (rather than object). You might want several levels of this for various ones (e.g. IOBase) or whatever. Just to remove some boilerplate.

You could e.g. in IOBase have teardown always do a .remove

[qwhelan]

My expectation is that any of the benchmark modules will have a handful of classes post-cleanup. asv doesn't require different classes per benchmark, so we can put time_foo() and time_bar() on the same class. Also, these changes are all auto-generated.

[jorisvandenbossche]

@jreback regarding your comments, I think that mainly are the things that will have to be handled manually (@qwhelan is that correct?) That is the clean-up mentioned in #10849

[jreback]

sure that is fine
yes iirc that is a further cleanup

ok so merge when ready then

[qwhelan]

@jorisvandenbossche Yes, that is correct. The only manual changes were in vbench_to_asv.py and inside vb_suite (which were propagated to asv_bench by rerunning the former). I also manually ensured some new asv tests added by others didn't get overwritten.

Just pushed some small changes (StringIO change and added a new line in contributing). I think it's good to merge.

[jorisvandenbossche]

@qwhelan Thanks a lot for this!

[qwhelan]

Of course.

I also have a bunch of asv run data I should throw up on a github.io page
so the EuroScipy folks can look into some of the regressions (won't get to
this for probably 6+ hours).
On Aug 29, 2015 3:42 PM, "Joris Van den Bossche" [email protected]
wrote:

@qwhelan https://github.com/qwhelan Thanks a lot for this!

—
Reply to this email directly or view it on GitHub
#10928 (comment).

[qwhelan]

@jorisvandenbossche My run data should be accessible here: http://qwhelan.github.io/pandas_asv/ .

An explicit list of regressions can be seen here: http://qwhelan.github.io/pandas_asv/#/regressions

[RafalSkolasinski]

Why did you change pytables to tables? It doesn't work for me know with conda. It ask if I meant pytables.

[qwhelan]

@RafalSkolasinski Looks like Anaconda changed the package name - it's on PyPI as tables: https://pypi.python.org/pypi/tables . I didn't want to force people to use conda or virtualenv, but I don't see an easy way to support both due to this name change.

[RafalSkolasinski]

@qwhelan I see... That's weird that package is named tables in pypi. even original webpage reads http://www.pytables.org/

[jorisvandenbossche]

but the 'import name' is tables I think.

In any case, we are probably not going to be able to specify a fully correct asv.conf.json file (eg also for windows an extra entry has to be added). So I would propose to choose one case (I would say: conda on linux) and make it work for that, and then clearly explain in the docs or readme what has to be changed to use it in the other cases.

[behzadnouri]

from pandas_vb_common import * -> from .pandas_vb_common import *

has broken test_perf.sh. running benchmarks fails, with ImportError: No module named pandas_vb_common. (on python 2)

[jorisvandenbossche]

Now we did the conversion (and it is the idea we shouldn't have to do this again, as we start changing the asv benchmarks direclty), we can change this (from pandas_vb_common import *) back to how it was in the vb_suite benchmarks.