BUG/API (string dtype): return float dtype for series[str].rank()

[jorisvandenbossche]

Noticed in #59758 (comment)

This is partially fixing a bug, because currently for cases where we actually need a float result but try to convert it back to an int, we get an error:

In [58]: pd.Series([2, 1, 1]).rank()
Out[58]: 
0    3.0
1    1.5
2    1.5
dtype: float64

In [59]: pd.Series(["2", "1", "1"], dtype="string[pyarrow]").rank()
...
File ~/scipy/repos/pandas/pandas/core/arrays/string_arrow.py:445, in _convert_int_result(self, result)

File ~/scipy/repos/pandas/pandas/core/arrays/numeric.py:93, in NumericDtype.__from_arrow__(self, array)
---> 93     array = array.cast(pyarrow_type)
 ...
ArrowInvalid: Float value 1.5 was truncated converting to int64

But in general we should also decide what to do return here. For our default dtypes, it seems we decided in the past to simply always return float64, even for rank methods that could return ints.
For the ArrowDtype, then, it was decided to keep the dtype returned by pyarrow: #50264

For now, this PR updates StringDtype to simply always return float64, to be consistent between the pyarrow vs python storage. But we could also consider, for those newer dtypes, to actually keep the distinction between float/int results (just not always int like it is done now).

[mroeschke]

Is it worth an entry in the 2.3.0 whatsnew?

[WillAyd]

Why does this change the string[python] case to float64 from Float64? I guess yet another discussion for PDEP-13, but shouldn't the if string_dtype_no_object.na_value invariant still remain?

[jorisvandenbossche]

To be clear this PR doesn't "change" anything regarding the flavor of dtypes (just changing int to float) and is only testing what we have right now, but you are certainly right this is inconsistent.

The technical reason is (AFAIK) that we just never implemented (yet) rank for our nullable dtypes in general (so also the nullable int returns numpy floats instead of nullable float). This can be considered as a missing part of the implementation.
And at some point we added a custom rank implementation on ArrowExtensionArray using pyarrow compute, and because of the subclass structure, the ArrowStringArray (i.e. "string[pyarrow]") inherits that but converts the pyarrow result to a nullable dtype (and "string[python]" array class does not inherit from this, so uses the base class rank implementation with always returns numpy types).

[WillAyd]

Ah OK...very confusing. Thanks for clarifying

[jorisvandenbossche]

Is it worth an entry in the 2.3.0 whatsnew?

Yes, good idea, added one.

[mroeschke]

Thanks @jorisvandenbossche