PERF: reduce overhead in groupby _cython_operation

[jorisvandenbossche]

The BaseGrouper._cython_operation has lots of checking of the dtype etc, which gives quite some overhead compared to the actual aggregation function (for small data). This is a first set of small changes to cut down that overhead.

Using the GroupManyLabels benchmark we have

ncols = 1000
N = 1000
data = np.random.randn(N, ncols)
labels = np.random.randint(0, 100, size=N)
df = DataFrame(data)
df_am = df._as_manager('array')

gives

In [2]: %timeit df_am.groupby(labels).sum()
48.9 ms ± 1.92 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- master
39.1 ms ± 695 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- PR

[jorisvandenbossche]

maybe_fill is only used in _cython_operation, and thus always used with an ndarray and with np.nan as fill value, so therefore simplified this function a bit, based on those assumptions.

[jorisvandenbossche]

The ensure_float was needed before to ensure that a nullable float EA was converted to the float ndarray. But now EAs already take a different code path above.

[jbrockmendel]

can you also report how the non-AM perf is affected?

[jorisvandenbossche]

can you also report how the non-AM perf is affected?

For the benchmark, it's not really affected. The overhead I am optimizing here is only around 1% for the BlockManager case. It's only when doing those checks 1000x instead of 1x that this becomes a noticeable bottleneck (the benchmarks has 1000 columns / single block).

I assume that for a small Series, the overhead might also be noticeable for non-AM. But trying this, and I don't really see much difference, and that's because for a single column, the dominant time is the actual factorization (which for 1000 columns is also only done once). Using the code from above:

In [2]: %timeit df[0].groupby(labels).sum()
384 µs ± 4.76 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  <-- master
374 µs ± 4.18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  <-- PR

[jorisvandenbossche]

(some failing tests related to (unsupported) operations on categorical column that I still need to address)

[jorisvandenbossche]

This is all passing now

[jbrockmendel]

-> np.ndarray? (or just remove the return statement i guess)

[jbrockmendel]

if we're micro-optimizing, these can be replaced with isinstance(dtype, FooDtype) checks

[jorisvandenbossche]

I was planning to do a follow-up PR with several other dtype check optimizations, so will include it there

[jbrockmendel]

LGTM ex one requested annotation, cc @jreback

[jreback]

lgtm, surprised this has any speedup...

once you can fix the annotation go ahead and merge