BUG: make mean() raise an exception for strings

[kinshukdua]

• closes BUG: sum() and mean() return wrong values when applied to integers as strings #44008
• closes BUG: String upgraded to complex128 (and cascading to other columns) in df.mean() and df.agg('mean') #36703
• closes BUG: Series / DataFrame reductions convert string data to numeric #34671
• closes BUG: df.sum() of string columns depends on whether or not they can be coerced to numbers #22642
• closes Dataframe.mean with numeric_only=False results in error with strings #26927
• closes sum in pandas can concatenate strings #13916
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

This PR makes mean() raise an exception when sum() returns a string. Sum performs as expected since it concatenates. However _ensure_numeric causes implicit conversion of str to a numeric datatype

pandas/pandas/core/nanops.py

Line 694 in f3f90c3

the_sum = _ensure_numeric(values.sum(axis, dtype=dtype_sum))

Instead of this, the return value of sum() will be checked and if it is a string, an exception will be raised.
This is a breaking change.

[jbrockmendel]

There are a bunch of issues about reductions with strings. can you see if this closes any of the others?

[kinshukdua]

Added all the issues I could find that this PR would close, I think we also need to decide whether we want to keep the behavior of sum() concatenating strings, add a note about the same or change it.
Regarding the failing tests, I'll try to see why they are failing and try to fix that.

[phofl]

Your change broke tests. Please add tests for every related issue

[jorisvandenbossche]

Regarding the failing tests (reporting from gitter here): I also can't reproduce those locally when checking out this branch. Not fully sure what would cause this discrepancy (in any case it doesn't seem to be related to the python or numpy version, since CI is failing for several versions of those).

[kinshukdua]

@phofl I reran the tests after installing the environments from ci using conda and the tests pass locally even on the environments (variants of actions-38.yaml) for which the tests are failing on ci. Any idea why this is happening?

[jreback]

ok if you are going to close an issue, pls explicitly make sure its listed in the test its (e.g. a reference) or have an explict test.

[jreback]

dont' worry about .sum it is fine.

[phofl]

Not really, no

[jbrockmendel]

@kinshukdua can you add tests specific to each of the issues this is closing

[kinshukdua]

@jbrockmendel yes, I'll add them, sorry for the delay.

[jreback]

can you replicate these examples in tests:
#22642
#36703
#44008

also pls list the issues in the whatsnew note

[jreback]

this warrents a sub-section in the whatsnew e.g. showing what previously worked and the new

[kinshukdua]

can you replicate these examples in tests: #22642 #36703 #44008

also pls list the issues in the whatsnew note

The column "A" tests the string to integer implicit conversion in #22642 and #44008, since the column won't be dropped if they are integers.

The column "D" in the test dataframe takes care of the cascading to complex-128 issue in #36703. If J is converted to complex, we would see that in the output series.

Should I create a different test for each issue?

[jreback]

can you merge master and will have a look

[kinshukdua]

@jreback, sorry for the delay, I rebased to fix the merge conflict.

[jreback]

@kinshukdua this is failing tests

[jreback]

this is worthile but needs to be passing, and pls merge master

[kinshukdua]

@jreback as I've explained before, for some reason the tests are all passing on my local environment, I've tried to reproduce the CI as close as possible by using the yaml files to recreate my testing environment. However, the tests still seem to pass. @jorisvandenbossche confirmed it by doing the same . I'm not sure what I should do.

[jreback]

@jreback as I've explained before, for some reason the tests are all passing on my local environment, I've tried to reproduce the CI as close as possible by using the yaml files to recreate my testing environment. However, the tests still seem to pass. @jorisvandenbossche confirmed it by doing the same . I'm not sure what I should do.

well sure, we are running multiple environments in CI which may have slightly different behavior.

you need to install one of the failing environments and investigate.

we cannot merge until CI is green.

[jbrockmendel]

FWIW i wasn't able to reproduce the errors on either my "normal" build or a py38+np1.18.5 build. @kinshukdua is right that something weird is going on here.

Doesn't change the fact that the CI needs to get to green before this can be merged. Just saying it's not an easy problem.

[jreback]

@kinshukdua if you can merge master and get this passing

[mroeschke]

Thanks for the effort on this pull request, but it appears to have gone stale. If interested in continuing, please merge in main and we can reopen.