BUG: fillna() for tz-aware timestamps causes an exception

[chrisaycock]

Suppose I have a DataFrame of timestamps:

df = pd.DataFrame({'A': [pd.Timestamp('2012-11-11 00:00:00'),
                         pd.Timestamp('2012-11-11 00:00:00')]})

I can attempt to fill any nulls:

In [23]: df.fillna(method='pad')
Out[23]:
           A
0 2012-11-11
1 2012-11-11

Now suppose my timestsamps are tz-aware:

df = pd.DataFrame({'A': [pd.Timestamp('2012-11-11 00:00:00+00:00'),
                         pd.Timestamp('2012-11-11 00:00:00+00:00')]})

This causes an error:

In [25]: df.fillna(method='pad')
...
ValueError: total size of new array must be unchanged

This does not appear to be fixed by #14960.

[jreback]

hmm, does look buggy. These should be coerced to/from i8, filled, then re-localized.

[chrisaycock]

Also, this works as expected:

pd.DataFrame({'test': [pd.Timestamp('2012-01-01 13:00:00'),
                       pd.Timestamp('2012-01-01 13:00:00')]}) == -1

However, this raises an exception:

pd.DataFrame({'test': [pd.Timestamp('2012-01-01 13:00:00+00:00'),
                       pd.Timestamp('2012-01-01 13:00:00+00:00')]}) == -1

---

Furthermore, this works as expected:

pd.DataFrame({'time': [pd.Timestamp('2012-01-01 13:00:00+00:00')],
              'A': [3]}).groupby('A', as_index=False).head(1)

However, this loses the timezone:

pd.DataFrame({'time': [pd.Timestamp('2012-01-01 13:00:00+00:00')],
              'A': [3]}).groupby('A', as_index=False).first()

[jreback]

pd.DataFrame({'test': [pd.Timestamp('2012-01-01 13:00:00+00:00'),
                       pd.Timestamp('2012-01-01 13:00:00+00:00')]}) == -1

is the same exact bug (the issue is it can't reshape properly), it should have a try/except around that .reshape. same issue as in #15869

[jreback]

can you make a new issue for:

pd.DataFrame({'time': [pd.Timestamp('2012-01-01 13:00:00+00:00')],
              'A': [3]}).groupby('A', as_index=False).first()

I thought we had an issue for this. But can't seem to find it.

[chrisaycock]

I have split-out that issue as #15884.

[chrisaycock]

This is something that fixes the fillna() issue:

diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index 8db801f..5736188 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -2475,7 +2475,7 @@ class DatetimeTZBlock(NonConsolidatableMixIn, DatetimeBlock):
         if isinstance(result, np.ndarray):
             # allow passing of > 1dim if its trivial
             if result.ndim > 1:
-                result = result.reshape(len(result))
+                result = result.reshape(np.prod(result.shape))
             result = self.values._shallow_copy(result)
 
         return result

I.e., use the total size of the result instead of just the length of one dimension. Here's an example of it at work:

In [12]: df = pd.DataFrame({'A': [pd.Timestamp('2012-11-11 00:00:00+01:00'),
    ...:                          pd.NaT]})
    ...: df.fillna(method='pad')
    ...: 
Out[12]: 
                          A
0 2012-11-11 00:00:00+01:00
1 2012-11-11 00:00:00+01:00

Does this make sense as a fix, or is there more to it than that?

[jreback]

in other cases like this, we have a try/except around reshape (in internals). but this is ok too. does it pass everything?

[chrisaycock]

It passes

$ find . -name test_missing.py | xargs nosetest

I can package this as a proper PR and see what happens with the CI.

[chrisaycock]

I've split-out the other issue I reported as its own item: #15966