Skip to content

BUG: Series.ffill() of Series with mixed dtypes containing tz-aware datetimes, fails. #14956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RobertasA opened this issue Dec 22, 2016 · 3 comments
Labels
Dtype Conversions Unexpected or buggy dtype conversions Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timezones Timezone data dtype
Milestone

Comments

@RobertasA
Copy link

Problem description

Doing a ffill/bfill on a mixed dtype series fails if there are tz-aware datetime.datetime instances.
It does work with timestamps, but as it's mixed type column anyway, it probably should just work with non-native types, leaving them as it is. Furthermore, it used to work in 0.18.1.

Code Sample, a copy-pastable example if possible

>>> s = pd.Series([datetime.datetime(2015, 1, 1, tzinfo=pytz.utc), 1])
>>> s.ffill()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/generic.py", line 3302, in ffill
    limit=limit, downcast=downcast)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/series.py", line 2368, in fillna
    **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/generic.py", line 3250, in fillna
    downcast=downcast)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 3177, in interpolate
    return self.apply('interpolate', **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 3056, in apply
    applied = getattr(b, f)(**kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 918, in interpolate
    downcast=downcast, mgr=mgr)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 960, in _interpolate_with_fill
    return self._maybe_downcast(blocks, downcast)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 1917, in _maybe_downcast
    for b in blocks])
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 1876, in convert
    self.values.ravel(), **fn_kwargs).reshape(self.values.shape)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/types/cast.py", line 641, in _soft_convert_objects
    values = lib.maybe_convert_objects(values, convert_datetime=datetime)
  File "pandas/src/inference.pyx", line 842, in pandas.lib.maybe_convert_objects (pandas/lib.c:56711)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/tseries/index.py", line 319, in __new__
    yearfirst=yearfirst)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/tseries/tools.py", line 428, in to_datetime
    return _convert_listlike(arg, box, format)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/tseries/tools.py", line 413, in _convert_listlike
    raise e
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

Expected Output

(Output returned in 0.18.1)

>>> s = pd.Series([datetime.datetime(2015, 1, 1, tzinfo=pytz.utc), 1])
>>> s.ffill()
0    2015-01-01 00:00:00+00:00
1                            1
dtype: object

Output of pd.show_versions()

Paste the output here pd.show_versions() here

INSTALLED VERSIONS ------------------ commit: None python: 2.7.10.final.0 python-bits: 64 OS: Darwin OS-release: 15.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: None.None
pandas: 0.19.1
nose: 1.3.7
pip: 1.5.6
setuptools: 25.1.1
Cython: 0.23.2
numpy: 1.11.1
scipy: 0.14.0
statsmodels: 0.6.1
xarray: None
IPython: 2.3.1
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.9
apiclient: 1.1
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.7.3
boto: 2.26.0
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Dec 22, 2016

you are trying to fill an object series which is not idiomatic, nor do you actually have nan values.

but I suppose this should not raise.

This is generally what one does

In [7]: s = pd.Series([datetime.datetime(2015, 1, 1, tzinfo=pytz.utc), np.nan])

In [8]: s
Out[8]:
0   2015-01-01 00:00:00+00:00
1                         NaT
dtype: datetime64[ns, UTC]

In [9]: s.ffill()
Out[9]:
0   2015-01-01 00:00:00+00:00
1   2015-01-01 00:00:00+00:00
dtype: datetime64[ns, UTC]

@jreback jreback changed the title Series.ffill() of Series with mixed dtypes containing tz-aware datetimes, fails. BUG: Series.ffill() of Series with mixed dtypes containing tz-aware datetimes, fails. Dec 22, 2016
@jreback
Copy link
Contributor

jreback commented Dec 22, 2016

a pull request to fix would be appreciated

@jreback jreback added Difficulty Intermediate Dtype Conversions Unexpected or buggy dtype conversions Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timezones Timezone data dtype labels Dec 22, 2016
@jreback jreback added this to the Next Major Release milestone Dec 22, 2016
@grutts
Copy link

grutts commented Dec 22, 2016

I've got a fix which seems to work locally for me and passes a subset of the test suite. I'll put it together in a PR if the full suite passes. Presumably you'd like a unittest to cover this case as well?

@jreback jreback modified the milestones: 0.20.0, Next Major Release Dec 23, 2016
ShaharBental pushed a commit to ShaharBental/pandas that referenced this issue Dec 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dtype Conversions Unexpected or buggy dtype conversions Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timezones Timezone data dtype
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants