TypeError: Cannot interpolate with all NaNs.

[david-r-wasserman]

Problem description

I got an error: "TypeError: Cannot interpolate with all NaNs." This error was raised on line 6011 of pandas\core\generic.py. Sorry, I'm not able to provide a simple code sample to reproduce the problem.

I can't see why this error should ever be raised. If the input is all NaNs, interpolate() should output all NaNs. Users of interpolate() don't necessarily know what data they will get. Why make them handle this error?

Expected Output

All NaNs

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.0
pytest: 3.5.1
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

Let us know if you can provide a minimal example: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[Liam3851]

@david-r-wasserman I think the problem is you have dtype=object:

In [26]: pd.DataFrame({'a':pd.Series([2, 3, 1])}).interpolate()
Out[26]:
   a
0  2
1  3
2  1

In [27]: pd.DataFrame({'a':pd.Series([2, 3, 1], dtype=object)}).interpolate()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-87d427d43aa7> in <module>()
----> 1 pd.DataFrame({'a':pd.Series([2, 3, 1], dtype=object)}).interpolate()

C:\Miniconda2\lib\site-packages\pandas\core\generic.pyc in interpolate(self, method, axis, limit, inplace, limit_direction, downcast, **kwargs)
   3668         if _maybe_transposed_self._data.get_dtype_counts().get(
   3669                 'object') == len(_maybe_transposed_self.T):
-> 3670             raise TypeError("Cannot interpolate with all NaNs.")
   3671
   3672         # create/use the index

TypeError: Cannot interpolate with all NaNs.

@TomAugspurger I think the TypeError here is confusing. The code check is that all columns of the DataFrame have dtype object, but the message says there's a problem with NaNs, even if the dataset has no NaNs.

[TomAugspurger]

Agreed, that's a confusing error message... It's also not really clear to me why we're doing that in the first place (and I think I wrote that code 😄). Any investigation here would be welcome.

[Liam3851]

Yup, it's your commit (from 2013!), revision aff7346 added

        if self._data.get_dtype_counts().get('object') == len(self.T):
            raise TypeError("Cannot interpolate with all NaNs.")

That one commit added basically all the type checking currently in the function.. Only thought is that right below in the same commit is another check:

        if pd.isnull(index).any():
            raise NotImplementedError("Interpolation with NaNs in the index "
                                      "has not been implemented. Try filling "
                                      "those NaNs before interpolating.")

At this point I assume you meant to write TypeError("Cannot interpolate with all objects"), had the second check on the brain and typoed?

[pambot]

I decided to just fix the error message because it's just making sure that all columns aren't object. Confirmed by making sure this doesn't trigger it:

pd.concat((pd.DataFrame({'a':pd.Series([2, 3, 1], dtype=object)}), pd.DataFrame({'b':pd.Series([2, 3, 1])}))).interpolate()

But this still does:

pd.concat((pd.DataFrame({'a':pd.Series([2, 3, 1], dtype=object)}), pd.DataFrame({'b':pd.Series([2, 3, 1], dtype=object)}))).interpolate()