Fix to numeric on decimal fields

[RogerThomas]

• closes Pandas.to_numeric raises ValueError when applied to python Decimal object #14827
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

[RogerThomas]

@jreback Does this look ok to you?

[jorisvandenbossche]

@RogerThomas You don't need to open a new PR to update. You can just push again the same branch, and the PR on github will automatically be updated.

[RogerThomas]

@jorisvandenbossche, thanks yeah, I thought the same, but the other PR doesn't seem to have been updated, maybe it only updates if its status is open. Anyway, I'll leave this one here and leave the other one closed if it's ok with you.

[jorisvandenbossche]

maybe it only updates if its status is open.

indeed

[jreback]

this should not be special cased

instead remove the use of np.isscalar and use is_scalar, which I think should work with decimal

[RogerThomas]

pandas.lib.isscalar doesn't return true for decimal

[jreback]

try changing that and see if anything breaks

[RogerThomas]

@jreback I did, I tried

if lib.isscalar(arg):
    ...

But I don't see how lib.isscalar could return True for decimal

this is the isscalar code

def isscalar(object val):
    """
    Return True if given value is scalar.

    This includes:
    - numpy array scalar (e.g. np.int64)
    - Python builtin numerics
    - Python builtin byte arrays and strings
    - None
    - instances of datetime.datetime
    - instances of datetime.timedelta
    - Period

    """

    return (np.PyArray_IsAnyScalar(val)
            # As of numpy-1.9, PyArray_IsAnyScalar misses bytearrays on Py3.
            or PyBytes_Check(val)
            # We differ from numpy (as of 1.10), which claims that None is
            # not scalar in np.isscalar().
            or val is None
            or PyDate_Check(val)
            or PyDelta_Check(val)
            or PyTime_Check(val)
            or util.is_period_object(val))

[RogerThomas]

Hey @jreback any more thoughts on this?

[RogerThomas]

@jreback I understand where you're coming from, but I don't see how to get around it in this case.
If you're happy with it, this PR should be ready to go

[jreback]

were you able to move addtl logic to is_scalar as well (IOW, did that break things irrespective of this change)?

[RogerThomas]

@jreback I can easily move the isinstance(value, Decimal) check to the isscalar function, but I don't see the point, I'd still have to check again after this in order to call float(value)

[jreback]

ok this is what I'd like to see; we are trying to treat decimal.Decimal in a more thoughtful way

In [7]: d = decimal.Decimal('1.0')

In [8]: pandas.types.common.is_scalar(d)
Out[8]: False

In [9]: pandas.types.common.is_number(d)
Out[9]: True

I'd like [7] to be True.
further I'd like to add a is_decimal in pandas.types.inference

then the last part can be something like:

    elif isinstance(arg, (list, tuple)):
        values = np.array(arg, dtype='O')
    elif is_scalar(arg): # previous np.isscalar(arg):
        if is_decimal(arg):
            # coerce
            arg = float(arg)
        if is_number(arg):
            return arg
        is_scalar = True
        values = np.array([arg], dtype='O')

[jreback]

also need to add some tests where decimals are in a list/array (they will be then subseuqnetly converted via maybe_convert_numeric which is also fine.

[codecov-io]

Current coverage is 84.64% (diff: 100%)

Merging #14842 into master will decrease coverage by <.01%

@@             master     #14842   diff @@
==========================================
  Files           144        144          
  Lines         51016      51019     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43185      43187     +2   
- Misses         7831       7832     +1   
  Partials          0          0          

Powered by Codecov. Last update 8e630b6...91d989b

[jreback]

import these from pandas.types.common (like everything else). you prob have to fix is_decimal to import also (it should be imported into pandas.types.inference, which pandas.types.common subseuqently imports). pandas.types.common is the internal API.

[jreback]

almost perfect. left some comments.

[RogerThomas]

Thanks @jreback, I actually had it like that originally but decided to just directly import it from lib. Changed back now

[jreback]

ok lgtm. ping on green.

[RogerThomas]

@jreback all ready to go

[jreback]

thanks!