ENH: DataFrame.describe allows UDFs and/or selectable metrics

[attack68]

DataFrame.describe() generates descriptive statistics for columns in a DataFrame. The "descriptive statistics" have been specifically chosen and hard-coded, and are also somewhat dtype dependent.

I find it quite odd that the function has a lot of customisation for which columns to include or to exclude based on dtype, or which percentiles to sample, but doesn't offer the ability to chain a set of predefined functions that the user might want to see (or not see: such as percentiles).

A rough idea is to propose a new argument, e.g. metrics, which overwrites and defines the metrics to a specifc set of functions, defined as str or callable:

def udf_name(s):
    return s.sum()

df = DataFrame([[1, 2], [3, 4]], columns=["A", "B"])
df.describe(metrics=["sum", Series.mean, lambda s: s.count(), lambda s: s.dtype, udf_name])
              A        B
sum           4        6       
mean          2        3
<lambda>      2        2
<lambda>  int64    int64
udf_name      4        6

Note this came up in the context of trying to add sub-total or additional rows to a Styler, based on the underlying data (#43894), and following issues:

• ENH: Add option to display dtypes below column headers #43875
• Skew and kurtosis in describe method #44751

[Delengowski]

What about an appends=False kwarg that if passed True will append metrics to the default list? If metrics is passed and appends=False (default value) then the default metrics is replaced with new metrics. If metrics is passed and appends=True then metrics is appended to the default metrics.

[rhshadrach]

What's the difference between this and agg?

df.agg(["sum", Series.mean, lambda s: s.count(), lambda s: s.dtype, udf_name])

              A      B
sum           4      6
mean        2.0    3.0
<lambda>      2      2
<lambda>  int64  int64
udf_name      4      6

[attack68]

What's the difference between this and agg?

That is an excellent point. agg does exactly this, somewhat ashamed I didn't think this could apply here, and it works just as I would expect.

I'm going to blame the advertising, :). I would recommend a link on the docs for describe and possibly even an example showing the comparison!

[rhshadrach]

Agreed on the docs. My use case for describe was always some quick statistics on a data set and if I needed to do more I turn to agg. In other words, to me at least, the purpose of describe is to be quick, not flexible.

[jreback]

this was actually the original usecase of .agg; i think i even described (pun intended) this somewhere......

[Delengowski]

this was actually the original usecase of .agg; i think i even described (pun intended) this somewhere......

Really it's a good example of just how large the api is. I can understand the core teams hesitation with adding anything more