BUG: Series/DataFrame.rank returns empty object on failure

[rhshadrach]

This was noticed when working on #40288

Example:

df = DataFrame({"A": 3 * [object]})
print(df.rank())
print(df.A.rank())

The last two lines are an empty DataFrame and empty Series respectively.

The docstring for numeric_only says:

For DataFrame objects, rank only numeric columns if set to True.

The current behavior with numeric_only=None (the default value) is:

Try with all columns. If a TypeError is raised, try with numeric_only=True.

When numeric_only is True and the Series/DataFrame contain no numeric columns, rank then operates on an empty object returning an empty result.

This is causing issues when rank is used in transform lists and dictionaries. Namely, we'd like to have partial-failure for TypeErrors, but rank is returning an empty result instead of raising a TypeError. This could be special-cased, but it seems to me that returning an empty object (at least when numeric_only is not True) is undesirable itself.

Some options I see are:

• Remove None as being an option and replace the default numeric_only=None with numeric_only=False.
• If numeric_only is None and the fallback of selecting only numeric columns results in an empty object, raise a TypeError.
• If selecting only numeric columns results in an empty object (even when numeric_only=True), raise a TypeError.

[iamrajhans]

@rhshadrach would like to take this one

[rhshadrach]

Sounds great @iamrajhans. It's not clear to me what the desired behavior here is. I would recommend looking into the behavior of numeric_only for other pandas' methods and make this consistent.

[iamrajhans]

@rhshadrach so I checked numeric_only behaviour, in the case of numerice_only=True it is returning empty Dataframe/Series

>>> import pandas as pd
>>> df = pd.DataFrame({"A": 3 * [object]})
>>> df.sum(numeric_only=True)
Series([], dtype: float64)

but in the case of numeric_only=False, it is giving TypeError: unsupported operand type(s) for +: 'type' and 'type

[rhshadrach]

Thanks @iamrajhans, it seems to me there are two separate APIs for reducer vs transformer:

• Reducers have a numeric_only argument, will return an empty object when numeric_only=True if there are no numeric columns.
• Transformers don't have a numeric_only argument, will raise if attempting to operate on non-numeric data if it can't be handled.

The only exception I see to this (but I may be missing others) is rank, which is a transformer whose API fits in the reducer camp.

I don't see any advantage to have two different APIs for reducer vs transformer, it'd be nice if these were more common. One step in this direction that I think would be non-controversial would be to add a numeric_only argument to the rest of the transformers.

Another question I have is whether the returning of an empty DataFrame (for both transformers and reducers) is desired behavior. I personally would expect/like a TypeError to be raised instead, but I wonder if there are other thoughts or aspects of pandas that depend on returning an empty object.

[rhshadrach]

No longer relevant now that numeric_only=None has been removed.