PERF: lazify blknos and blklocs

[jbrockmendel]

The benchmark I'm using for this is the same as for #32224, based on the asv that is most affected by removing fast_apply (see #32086)

import numpy as np
from pandas import *
%load_ext line_profiler


def get_df():
    N = 10 ** 4
    labels = np.random.randint(0, 2000, size=N)
    labels2 = np.random.randint(0, 3, size=N)
    df = DataFrame(
        {
            "key": labels,
            "key2": labels2,
            "value1": np.random.randn(N),
            "value2": ["foo", "bar", "baz", "qux"] * (N // 4),
        }
    )
    return df

df = get_df()

gb = df.groupby("key")

%prun -s cumulative gb.apply(lambda x: 1)

If we disable fast_apply on master, this gives:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.207    0.207 groupby.py:701(apply)
        1    0.000    0.000    0.207    0.207 groupby.py:750(_python_apply_general)
        1    0.008    0.008    0.204    0.204 ops.py:151(apply)
     1994    0.003    0.000    0.191    0.000 ops.py:858(__iter__)
     1993    0.003    0.000    0.187    0.000 ops.py:889(_chop)
     1993    0.003    0.000    0.184    0.000 indexing.py:814(__getitem__)
     1993    0.001    0.000    0.180    0.000 indexing.py:1462(_getitem_axis)
     1993    0.003    0.000    0.179    0.000 indexing.py:1488(_get_slice_axis)
     1993    0.007    0.000    0.167    0.000 generic.py:3474(_slice)
     1993    0.007    0.000    0.140    0.000 managers.py:713(get_slice)
     1994    0.004    0.000    0.068    0.000 managers.py:125(__init__)
     1994    0.027    0.000    0.059    0.000 managers.py:215(_rebuild_blknos_and_blklocs)
     1993    0.002    0.000    0.050    0.000 managers.py:723(<listcomp>)
     5979    0.010    0.000    0.048    0.000 blocks.py:310(getitem_block)
     5983    0.003    0.000    0.033    0.000 blocks.py:275(make_block_same_class)

If we disable fast_apply on this PR:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.136    0.136 groupby.py:701(apply)
        1    0.000    0.000    0.136    0.136 groupby.py:750(_python_apply_general)
        1    0.007    0.007    0.134    0.134 ops.py:151(apply)
     1984    0.002    0.000    0.120    0.000 ops.py:903(__iter__)
     1983    0.002    0.000    0.117    0.000 ops.py:934(_chop)
     1983    0.003    0.000    0.114    0.000 indexing.py:814(__getitem__)
     1983    0.001    0.000    0.110    0.000 indexing.py:1462(_getitem_axis)
     1983    0.003    0.000    0.109    0.000 indexing.py:1488(_get_slice_axis)
     1983    0.006    0.000    0.098    0.000 generic.py:3474(_slice)
     1983    0.006    0.000    0.076    0.000 managers.py:742(get_slice)
     1983    0.002    0.000    0.048    0.000 managers.py:752(<listcomp>)
     5949    0.010    0.000    0.046    0.000 blocks.py:310(getitem_block)
     5957    0.003    0.000    0.032    0.000 blocks.py:275(make_block_same_class)
     5960    0.006    0.000    0.029    0.000 blocks.py:3023(make_block)

[jbrockmendel]

Some asv results

$ asv continuous -E virtualenv -f 1.1 master HEAD -b reshape -b reindex -b replace -b join_merge -b frame_ctor
[...]
       before           after         ratio
     <master>         <consolidate_less>
-         357±2μs          323±2μs     0.90  reindex.Reindex.time_reindex_dates
-         284±2μs          250±4μs     0.88  join_merge.Concat.time_concat_empty_left(0)
-         285±2μs          251±4μs     0.88  join_merge.Concat.time_concat_empty_right(0)
-       120±0.7μs       86.5±0.5μs     0.72  frame_ctor.FromSeries.time_mi_series
-        99.5±2μs       71.1±0.3μs     0.71  frame_ctor.FromNDArray.time_frame_from_ndarray
-        89.5±3μs       61.1±0.3μs     0.68  replace.ReplaceList.time_replace_list(True)

[jbrockmendel]

Near-identical asv results from a second run

[jreback]

to make this more transparent, I would change the return signature of _rebuild_blknos_and_blklocs to return a tuple of (blknos, blklocs) and set them here. It is much more transparent.

[jbrockmendel]

that turns out to get messy because of the way we update these in-place in some other methods. so id like to skip this for now, and revisit it in the upcoming even-lazier pass

[jreback]

ok.