Skip to content

fillna is broken for Datetime columns [regression] #27419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vnlitvinov opened this issue Jul 16, 2019 · 2 comments · Fixed by #27425
Closed

fillna is broken for Datetime columns [regression] #27419

vnlitvinov opened this issue Jul 16, 2019 · 2 comments · Fixed by #27425
Labels
Blocker Blocking issue or pull request for an upcoming release Datetime Datetime data dtype Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Milestone

Comments

@vnlitvinov
Copy link
Contributor

In 823af76 a call to block._try_coerce_args() was incorrectly replaced with call to block._can_hold_element() (I think under assumption that all block types have identical implementations of _try_coerce_args which is not true).

For Datetime blocks this leads to blowing up with RecursionError: maximum recursion depth exceeded in comparison.

Prior to the commit I indicated it worked fine, so I consider this a regression.

Small reproducer

import pandas as pd
import numpy as np
from io import StringIO as sio

d1 = pd.read_csv(sio('2010,\n,\n2000\n'),header=None,parse_dates=[0])
f1 = np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]')
d1[0].fillna(f1)

Problem description

When the reproducer is launched, it's output is a long stacktrace caused by RecursionError.

Expected Output

The sample should not raise.

Output of pd.show_versions()

commit : None
python : 3.6.8.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-47-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.0rc0+67.g2ed1f28cf
numpy : 1.16.2
pytz : 2018.4
dateutil : 2.6.0
pip : 10.0.1
setuptools : 39.0.1.post20180504
Cython : 0.29.6
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : None
pandas_datareader: None
bs4 : 4.7.1
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.9.0
pytables : None
s3fs : None
scipy : 1.2.1
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

Proposed solution

Revert the changes to fillna() function here: 823af76?diff=unified#diff-8f51ce8697bdff680b3f71b196c582eeL388

@TomAugspurger
Copy link
Contributor

cc @jbrockmendel.

Right now, the value in ExtensionArray.fillna(value) is supposed to match the ExtensionArray.dtype.type. We don't allow coercion of the array here.

We'll probably want to special-case this for backwards compatibility. Not sure about whether we should deprecate non-ns precisions value.

@TomAugspurger TomAugspurger added this to the 0.25.0 milestone Jul 16, 2019
@TomAugspurger TomAugspurger added Blocker Blocking issue or pull request for an upcoming release Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Datetime Datetime data dtype labels Jul 16, 2019
@jbrockmendel
Copy link
Member

I'll take a look at this. Ultimately we should be dispatching the try_coerce/can_hold stuff to DTA/TDA/PA/EA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocker Blocking issue or pull request for an upcoming release Datetime Datetime data dtype Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants