use numexpr for Series comparisons

[jbrockmendel]

subsumes #31984.

[jorisvandenbossche]

Does this give a better performance?

[simonjayhawkins]

can we test explicitly with one message and then the other instead? or even better can we make the error messages consistent?

[jbrockmendel]

yep, turns out we only need the new one, updated.

[jbrockmendel]

Does this give a better performance?

After running some timeings, I found a surprising and unacceptable 2x slowdown in homogeneous dataframe ops, so I reverted the edit in dispatch_to_series (which I thought, apparently incorrectly, was not doing anything useful).

These timings are after that reversion (I'll push shortly). We pick up some good ground for Series ops, its a wash for homogeneous DataFrame ops, and we lose a small (and confusing) amount of ground for heterogeneous DataFrame ops.

In [5]: ser = pd.Series(range(10**7))                                                                                                                           
In [6]: %timeit ser == ser                                                                                                                                      
6.86 ms ± 295 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   # <-- master
4.68 ms ± 129 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   # <-- PR

In [7]: df = pd.DataFrame({n: range(10**6) for n in range(10)})                                                                                                 
In [8]: %timeit df == df                                                                                                                                        
6.53 ms ± 61.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
6.47 ms ± 124 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR

In [8]: df[1] = df[1].astype(float)                                                                                                                             
In [9]: %timeit df == df                                                                                                                                        
12 ms ± 33.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
12.8 ms ± 338 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR

[jreback]

so the reason we orignally wrote eval was to provide a quite significant speedup using multiple chained ops / comparisions, e.g.

https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#expression-evaluation-via-eval

(this is slightly orthogonal to this issue)

however,

https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#accelerated-operations are very relevant. a) we should have asv's for these, and be sure we are actually using the accelerated options (and likley update this link)

[jbrockmendel]

yah, I think it was @TomAugspurger who recently discussed being more careful about what we pass to numexpr. I think in particular the tests can be improved to be not just "was numexpr called" but "was it called without falling back to python"

This is a blocker for the branch that implements frame-with-frame ops blockwise, so id like to minimize side-tracking.

[jreback]

what hits this AND is a is_cmp? can you add a comment

[jbrockmendel]

so we have 7 cases that get here. 4 of these are something like ndarray[float] > datetime, and would pass if we removed this is_cmp check. Of the remaining three:

• 2 involves passing a DatetimeArray to masked_arith_op, which expects ndarray. this should probably be prevented.
• one involves comparions of complex numbers, which numpy does differently than python (which is its own PITA). falling through to the masked op would break test_bool_flex_frame_complex_dtype.

[jreback]

why is thi not handled in na_arithmetic_op? seems odd to handle here

[jbrockmendel]

sure

[jreback]

thanks.