PERF: specialized 1D take version

[jorisvandenbossche]

A part that I took out of #39692, doing it separately here.

This gives a bit of code duplication (but all large parts of the function were already factored out into shared helper functions before, like _take_preprocess_indexer_and_fill_value), but having a 1D version gives less overhead compared to take_nd.

Using the same benchmark case (unstack ASV case) as in #39692, this gives:

arrays = [np.arange(100).repeat(100), np.roll(np.tile(np.arange(100), 100), 25)]
index = pd.MultiIndex.from_arrays(arrays)
df = pd.DataFrame(np.random.randn(10000, 4), index=index)
df_am = df._as_manager("array")

In [3]:  %timeit df_am.unstack()
3.31 ms ± 60.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- master
2.95 ms ± 57.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR

It's a small difference (around 10% improvement), but I repeated the timings above multiple times it was quite consistently a bit faster.

[jreback]

This gives a bit of code duplication (but all large parts of the function were already factored out into shared helper functions before, like _take_preprocess_indexer_and_fill_value), but having a 1D version gives less overhead compared to take_nd.

can you factor even more, this ultimately should be used in the other functions.

[jorisvandenbossche]

I don't think more is easily possible. In the existing take_nd, there are several places with additional checks related to 2D, and I don't want to add that back / share that in a common helper, because it's exactly the purpose of this specialized 1D version to not have to care about (and have overhead due) 2D handling.

[jorisvandenbossche]

this ultimately should be used in the other functions.

It is used in other functions, eg in unstack (added here, and now the other PR is merged, can also use it in reindex)

[jbrockmendel]

-> ArrayLike?

[jreback]

pandas/core/internals/array_manager.py:997: error: Name 'take_nd' is not defined [name-defined]

[jorisvandenbossche]

Remaining failure also happens on master.

[jbrockmendel]

@jorisvandenbossche pls resist the temptation to self-merge. i'll sleep better.

(no complaints about the PRs you've merged recently, just as you've commented elsewhere about premature merging, better to have good habits and stay on the safe side)

can this new function be used anywhere else, e.g. in Series/Index?

[jorisvandenbossche]

To be clear, I only do that for PRs that are trivial and/or have been reviewed and are approved or have no active discussion (and thus not for eg PRs that impact the API). Of course that's always a bit subjective, so I have no problem in being called out for in case you think I made a wrong judgement call. And we are using git, so it's always easy to revert something if needed.

can this new function be used anywhere else, e.g. in Series/Index?

Maybe? That would need to be investigated for each individual case whether the constraints are met (eg the input already validated etc). And I am currently not planning to do such investigation (in general, unless it's within a performance bottleneck, I would personally mostly limit the use of this function to the internals.

[jbrockmendel]

in case you think I made a wrong judgement call

All the case so far i think have been benign.

have been reviewed and are approved

In practice I doubt it will make a difference, but I will be moderately more comfortable if you ping me/jeff. On the other hand if we're not reasonably responsive (timezones notwithstanding), do let me know.

Really the failure mode that I'm most worried about is something slipping through the cracks of my inbox.