pd.to_numeric(series, downcast='integer') does not prpoerly handle floats over 10,000

[gryBox]

Hi - I came across this issue in stackoverflow while testing pd.to_numeric()

If all floats in column are over 10000 it loses precision and converts them to integers.

tst_df = pd.DataFrame({'colA':['a','b','c','a','z', 'q'],
                      'colB': pd.date_range(end=datetime.datetime.now() , periods=6),
                      'colC' : ['a1','b2','c3','a4','z5', 'q6'],
                      'colD': [10000.0, 20000, 3000, 40000.36, 50000, 50000.00]})

pd.to_numeric(tst_df['colD'],  downcast='integer')

This doesn't seem like the desired behavior.

[jreback]

so this is correct, it is effectively doing an .astype(int). though I suppose this is too aggressive and the downcast from float -> int should only succeed if they are actually equal.

@gfyoung thoughts

[gryBox]

Just so I am clear, this is the behavior expected? Not this?

tst_df = pd.DataFrame({'colA':['a','b','c','a','z', 'q'],
                      'colB': pd.date_range(end=datetime.datetime.now() , periods=6),
                      'colC' : ['a1','b2','c3','a4','z5', 'q6'],
                      'colD': [1000.0, 2000, 3000, 4000.36, 5000, 50000.00]})

pd.to_numeric(tst_df['colD'],  downcast='integer')

[jreback]

hmm, this looks buggy. if you want to have a look see inside and see what's going on would be great.

[gryBox]

@jreback I will try - I am sort of new to this. Is this the module I should be looking at?
pandas/pandas/tools/util.py

[jreback]

yes

[gfyoung]

@gryBox : First of all, thanks for pointing this out! If you follow the code from where you started, the bug traces here. You can see here yourself:

>>> import numpy as np
>>>
>>> arr = [1000000]
>>> arr2 = [1000000.5]
>>>
>>> np.allclose(arr, arr2)  # This is what we do now
True
>>> np.allclose(arr, arr2, rtol=0)  # This is what we probably should do
False

I think if passing rtol=0, you should be able to patch this behavior. It also explains why it breaks with large numbers (the numbers are so large rtol isn't small enough to indicate that they're not close).

[gryBox]

@gfyoung : The credit goes to tworec at stack.

[gfyoung]

@gryBox : I'm not sure there is such a method. Also, numpy dtype checking is not 100% compatible with all of our pandas objects (deliberate), hence we prefer to stay away from such methods in numpy.