changed groupby_op to _groupby_op and added a docstring making explicit that this API is experimental.

Open to formatting suggestions for the kwargs part of the docstring.

Thinking about the namespace question whether to have a single dispatching method (_groupby_op) vs a bunch (_grouped_sum, _grouped_std, ...). With the single-method (this PR), we have **kwargs in the signature, and it is clumsy to document what kwargs to expect for each "how". Also should (but dont yet) document the expected shape of the returned array, which varies depending on which method we're talking about (fortunately only a few cases here).

Maybe a middle ground with e.g. _grouped_reduce, _grouped_transform, _grouped_quantile, _grouped_rank so that each of them can have their kwargs made explicit? cc @phofl @mroeschke @jorisvandenbossche

Goal is to allow EAs (including our own pyarrow-backed) to implement performant GroupBy reductions/transforms without necessarily having to convert to numpy.

Is there actually a demand from ExtensionArray authors to be able to implement the groupby reductions? (I mean, I can certainly see that it could be useful for certain kind of extension dtypes, but I would still be wary to implement it if it's not really requested)
Or is this more as a way to clean up the dispatching in the groupby code for our own EAs? Because then we should discuss if we actually want to make this public of external EAs, or only do it for our internal ones for now (also the pyarrow-backed arrays could be done internally just like we currently special case the masked arrays)

Is there actually a demand from ExtensionArray authors to be able to implement the groupby reductions? (I mean, I can certainly see that it could be useful for certain kind of extension dtypes, but I would still be wary to implement it if it's not really requested)

No, but I think there is a chicken/egg issue in that some libraries (i have in mind cudf, dask, modin) haven't implemented EAs at all and this is a necessary step to making EAs a viable option for them.

Or is this more as a way to clean up the dispatching in the groupby code for our own EAs? Because then we should discuss if we actually want to make this public of external EAs, or only do it for our internal ones for now (also the pyarrow-backed arrays could be done internally just like we currently special case the masked arrays)

That is definitely a motivator. This is much nicer abstraction-wise. We agreed long ago that ideally our internal EAs would be treated symmetrically with 3rd party EAs.

@MarcoGorelli this is the one i mentioned

@jorisvandenbossche feedback in #51471 seems mostly positive. any thoughts?

@MarcoGorelli gentle ping

+1 here. to the extent this cleans up generic pandas code this is quite valuable (not to mention that EA authors can use this).

@mroeschke gentle ping, this is the way forward for e.g. #51996

I get this solves #51996 in the short term, but my only reservation here is the dependence on the existing grouping machinery (the determination of ids) w.r.t. pyarrow. It would be nice if we could utilize pyarrow.Table's groupby machinery which then leads to using their groupby aggregation machinery too. If we had some notion of "object that manages EA" (#51471), that could be a pyarrow Table for ArrowExtensionArrays where we can dispatch these groupby ops to.

This also applies probably to other dataframe libraries like cuDF/Dask/etc that we hope could utilize the EA interface, but groupby is a "frame-wise" operation where this concern also applies.

Aside from the 3rd-party concerns, our own EA-specific logic fits much better here than it does in groupby.ops. I've changed this to use a private name to somewhat safeguard against future changes.

Okay I see the benefit for our EAs currently. Just hoping that you are open to a future API where 3rd party EAs (including pyarrow) could also dictate the grouping + op steps together.

Just hoping that you are open to a future API where 3rd party EAs (including pyarrow) could also dictate the grouping + op steps together

Absolutely open to it. I'd love to rip out a bunch of our libgroupby code one day.

@MarcoGorelli gentle ping

Apologies, was off last week (though still did some minor things). I've made it a priority to look at this next week

I had https://github.com/pandas-dev/pandas/pull/51166/files#r1139482980 but otherwise looked good

thx @jbrockmendel