PR for Pandas issue #14872 / fillna() TZ datetime64 rounded

[RodolfoRFR]

Hi Jeff,

The change adds a "datetime64[ns, UTC]" object to the _DATELIKE_DTYPES set in pandas/types/common.py.

No such object existed in this set and that is the reason the wrong method was executed on the inputs.

Object is created with "DatetimeTZDtype.new"

Thanks,
Rodolfo

• closes fillna() containing timezone aware datetime64 rounded #14872
• [ X] tests added / passed
• [ X] passes git diff upstream/master | flake8 --diff (returns the same number of skips, errors and failures before changes)
• whatsnew entry

[jreback]

this goes way overboard

see the pull request which closed the original issue

[RodolfoRFR]

The changes in request #6587 added "dtype in _DATELIKE_DTYPES" to the method selection in the backfill_2d function (now in missing.py),

-        _method = getattr(algos, 'backfill_2d_inplace_%s' % dtype, None)
-    elif is_datetime64_dtype(values):
+        _method = getattr(algos, 'backfill_2d_inplace_%s' % dtype.name, None)
+    elif dtype in _DATELIKE_DTYPES or is_datetime64_dtype(values):

I initially tried adding an extra is_datetime64tz_dtype() to this line with no success. After tracing the process with pdb I found that the values are already coerced to integers in pandas/core/internals.py,

    def _interpolate_with_fill(self, method='pad', axis=0, inplace=False,
                               limit=None, fill_value=None, coerce=False,
                               downcast=None, mgr=None):
        """ fillna but using the interpolate machinery """
        .
        .
        .
        values, _, fill_value, _ = self._try_coerce_args(values, fill_value)

before being passed to /pandas/core/missing.interpolate_2d and then to backfill_2d. The dtype information is passed as dtype=datetime64[ns, UTC]. When it gets to these lines in the backfill_2d function,

elif dtype in _DATELIKE_DTYPES or is_datetime64_dtype(values):
      method = _backfill_2d_datetime

the return is always false because datetime64[ns, UTC] is not in _DATELIKE_DTYPES and the 'values' are integers.

Another way would be to either replace datetime64[ns, UTC] with datetime64[ns] before we get to this line or skip the coercion somehow. Unless I am missing something.

[jreback]

so this should simply be changed to pass the actual value (and not coerced ones) when this is called and drop al of the _DATELIKE_DTYPES that is really old code

use the is_datetime64_dtype and is_datetime64tz_dtype

[jreback]

actually you just need to check the dtype instead (of the values)

e.g.

(Pdb) p is_datetime64tz_dtype(dtype)
True

[codecov-io]

Current coverage is 84.57% (diff: 75.00%)

Merging #14905 into master will increase coverage by 0.01%

@@             master     #14905   diff @@
==========================================
  Files           144        144          
  Lines         51043      51058    +15   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43160      43181    +21   
+ Misses         7883       7877     -6   
  Partials          0          0          

Powered by Codecov. Last update 73bc6cf...18802b4

[RodolfoRFR]

Done! I was indeed missing something. Thank you for the feedback!

[jreback]

pls add a whatsnew entry (0.19.2) ok if u can make these changes in next day or so

[jreback]

this test needs to go in test_missing (search and out it with other similar tests)

[jreback]

needs to take self

[jreback]

thanks!