ENH: evaluate datetime ops in python with eval

[cpcloud]

closes #4897

Also adds DatetimeIndex comparisons in query expressions and a corresponding vbench for a simple Timestamp vs either Series or DatetimeIndex.

[jreback]

maybe add some tests where use is comparing a date column with a non-date
? so should raise

as an aside in future min/max/diff/shift on dates have to be done in python land

[cpcloud]

that should be rhs

[cpcloud]

@jreback @jtratner comments?

[cpcloud]

this should eventually handle timedelta as well

[jreback]

under 1.7! they r like ints so I bet ne can handle them but prob better to handle in python because of NaT

[cpcloud]

i ithink i'll do a new pr for that

[cpcloud]

💥

In [5]: td = timedelta64(1, 'D')

In [6]: ne.evaluate('td + 1')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-f2c950d373ae> in <module>()
----> 1 ne.evaluate('td + 1')

/home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/numexpr/necompiler.pyc in evaluate(ex, local_dict, global_dict, out, order, casting, **kwargs)
    728
    729     # Create a signature
--> 730     signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)]
    731
    732     # Look up numexpr if possible.

/home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/numexpr/necompiler.pyc in getType(a)
    627     if kind == 'S':
    628         return bytes
--> 629     raise ValueError("unkown type %s" % a.dtype.name)
    630
    631

ValueError: unkown type timedelta64[D]

[jreback]

are you setting the kind somewhere? I woldn't do this, instead Timestamp(self._value).value if you really need the i8 value. why would these need to be i8 anyhow? are they evaluated in python space? (for which you are calling functions)?

[cpcloud]

I'm not actually getting the i8 value. That's what I was doing previously, to evaluate datetime ops using numexpr. kind is just a string repr of either the class name or the dtype, it's a computed property

[jreback]

ok....I just wouldn't allow a datetime time at all (instead just wrap Timestamp around it)....makes consistent

[cpcloud]

@jreback comments?

[jreback]

in the future maybe this function should be in core/generic?

[jreback]

looks fine otherwise

[cpcloud]

moved to generic and added a smoke test for failing panel/panel4d with multiiindex

[cpcloud]

slight hit when using DatetimeIndex vs a column of Series, need to clear that up b4 merging

[cpcloud]

turns out it's because a datetime64 Series with an Int64Index compares faster than a datetime64 Series with a DatetimeIndex. This is the case before 8a9a4f2 (the just merged timestamp compare fix) so that didn't introduce any performance regressions (I also ran the full vbench suite on it and no changes). I ran a vbench on this as well and nothing changed.

[cpcloud]

bombs away

[jreback]

@cpcloud do you recall if we fixed this for timedelta ops as well (which should be evaluated in python space and not by numexpr)

[cpcloud]

Don't think there's a test for them in top level query/eval, but there are some tests in test_pytables.py.