pandas give a warning when working with np.NaN values #9950

domenb · 2015-04-20T09:57:19Z

When I create a pandas.Series or pandas.DataFrame having np.NaN values, I get the following warning.

Example:

>>> import pandas as pd, numpy as np
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1   NaN
2   NaN
3   NaN
dtype: float64
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:1995: RuntimeWarning: invalid value encountered in absolute
abs_vals = np.abs(self.values)
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:1998: RuntimeWarning: invalid value encountered in greater
has_large_values = (abs_vals > 1e8).any()
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:1999: RuntimeWarning: invalid value encountered in less
has_small_values = ((abs_vals < 10 ** (-self.digits)) &
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:2000: RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()

I am using Anaconda-2.2.0-Windows-x86_64, with Python 2.7.9. and pandas 0.16.0.

I would ask you to advise.

The text was updated successfully, but these errors were encountered:

jreback · 2015-04-20T10:56:44Z

this was fixed by #9764 and will be in 0.16.1.

j1729 · 2015-05-19T21:02:51Z

I'm experiencing the exact same problem, but I have 0.16.1. Did the fix not make it in, or am I doing something wrong? Thanks in advance for your help.

>>> import pandas as pd, numpy as np
>>> pd.__version__
'0.16.1'
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1   NaN
2   NaN
3   NaN
dtype: float64
C:\Anaconda\lib\site-packages\pandas\core\format.py:2009: RuntimeWarning: invalid value encountered in absolute
  abs_vals = np.abs(self.values)
C:\Anaconda\lib\site-packages\pandas\core\format.py:2012: RuntimeWarning: invalid value encountered in greater
  has_large_values = (abs_vals > 1e8).any()
C:\Anaconda\lib\site-packages\pandas\core\format.py:2013: RuntimeWarning: invalid value encountered in less
  has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
C:\Anaconda\lib\site-packages\pandas\core\format.py:2014: RuntimeWarning: invalid value encountered in greater
  (abs_vals > 0)).any()
>>>

jreback · 2015-05-20T10:57:50Z

@domenb you should check whether you have a 'partially' updated installation. There was an older bug with conda that would if you were running python processes not update fully. Do this.

make sure all python processes are killed
conda remove pandas
conda install pandas

j1729 · 2015-05-20T13:23:51Z

I did the following:

check that there were no python processes in the task manager
conda remove pandas
conda remove numpy (just in case)
confirmed that the pandas and numpy folders were gone (in C:\Anaconda\Lib\site-packages)
confirmed that pandas and numpy would not import
check for no python processes
conda install pandas

C:\>conda install pandas
Fetching package metadata: Could not connect to https://repo.continuum.io/pkgs/f
ree/noarch/
.Could not connect to https://repo.continuum.io/pkgs/free/win-32/
Could not connect to https://repo.continuum.io/pkgs/pro/noarch/
.Could not connect to https://repo.continuum.io/pkgs/pro/win-32/
..
Solving package specifications: .
Package plan for installation in environment C:\Anaconda:

The following NEW packages will be INSTALLED:

    numpy:  1.9.2-py27_0
    pandas: 0.16.1-np19py27_0

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%

I then repeated the above test, and the behavior was identical. I have conda 3.11.0. I just deleted and reinstalled Anaconda last week. It is Anaconda-2.2.0-Windows-x86. I have numpy 1.9.2. I don't know if @domenb got this working, but please let me know if you think this is unrelated, or if I should create a separate issue.

Thanks

jorisvandenbossche · 2015-06-03T11:25:57Z

@jreback reopened, so we certainly look at it. The warnings also appeared in #10256 (also 0.16.1)

jreback · 2015-06-03T11:36:51Z

This is a user installation problem. I would guess that they have maybe anaconda AND miniconda installed and picking up a different version. So as I said above. need to look at the environment.

In [7]: pd.__version__
Out[7]: '0.16.1'

In [8]: pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
Out[8]: 
1   NaN
2   NaN
3   NaN
dtype: float64

cousin333 · 2015-06-03T14:33:00Z

I am the one who have created #10256, and - just as jorisvandenbossche wrote above - I've also experienced the problem, while writing to the console. That is, I wrote it into a "normal" console (included in the Spyder IDE), not an IPython one as you did above. In the IPython Console (or Notebook) it also works just fine. Besides I am using neither conda nor miniconda, but the latest WinPython, that includes 0.16.1 "by default".

>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.16.1'
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1   NaN
2   NaN
3   NaN
dtype: float64
R:\WinPython\python-3.4.3.amd64\lib\site-packages\pandas\core\format.py:2012: RuntimeWarning: invalid value encountered in greater
  has_large_values = (abs_vals > 1e8).any()
R:\WinPython\python-3.4.3.amd64\lib\site-packages\pandas\core\format.py:2013: RuntimeWarning: invalid value encountered in less
  has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
R:\WinPython\python-3.4.3.amd64\lib\site-packages\pandas\core\format.py:2014: RuntimeWarning: invalid value encountered in greater
  (abs_vals > 0)).any()
>>>

I have already included my environment parameters in the above mentioned #10256.

jreback · 2015-06-03T18:26:15Z

Well, this is 2.7.9, numpy 1.9.2, pandas 0.16.1 on windows. And its python, not ipython.
Still no error. So you'll have to step thru the debugger and see if you can figure out what the issue.

Python 2.7.9 |Continuum Analytics, Inc.| (default, Dec 18 2014, 16:57:52) [MSC v.1500 64 bit (AMD64)] on win32           
Type "help", "copyright", "credits" or "license" for more information.                                                   
Anaconda is brought to you by Continuum Analytics.                                                                       
Please check out: http://continuum.io/thanks and https://binstar.org                                                     
>>> import pandas as pd, numpy as np                                                                                     
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])                                                                      
1   NaN                                                                                                                  
2   NaN                                                                                                                  
3   NaN                                                                                                                  
dtype: float64                                                                                                           
>>> pd.__version__                                                                                                       
'0.16.1'                                                                                                                 
>>> np.__version__                                                                                                       
'1.9.2'                                                                                                                  
>>> pd.show_versions()                                                                                                   

INSTALLED VERSIONS                                                                                                       
------------------                                                                                                       
commit: None                                                                                                             
python: 2.7.9.final.0                                                                                                    
python-bits: 64                                                                                                          
OS: Windows                                                                                                              
OS-release: 7                                                                                                            
machine: AMD64                                                                                                           
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel                                                            
byteorder: little                                                                                                        
LC_ALL: None                                                                                                             
LANG: None                                                                                                               

pandas: 0.16.1                                                                                                           
nose: None                                                                                                               
Cython: None                                                                                                             
numpy: 1.9.2                                                                                                             
scipy: 0.15.1                                                                                                            
statsmodels: None                                                                                                        
IPython: 2.4.1                                                                                                           
sphinx: None                                                                                                             
patsy: None                                                                                                              
dateutil: 2.4.2                                                                                                          
pytz: 2015.2                                                                                                             
bottleneck: None                                                                                                         
tables: 3.1.1                                                                                                            
numexpr: 2.3.1                                                                                                           
matplotlib: 1.4.2                                                                                                        
openpyxl: None                                                                                                           
xlrd: 0.9.3                                                                                                              
xlwt: None                                                                                                               
xlsxwriter: 0.7.2                                                                                                        
lxml: None                                                                                                               
bs4: None                                                                                                                
html5lib: None                                                                                                           
httplib2: None                                                                                                           
apiclient: None                                                                                                          
sqlalchemy: 1.0.4                                                                                                        
pymysql: None                                                                                                            
psycopg2: None

quantstud · 2015-06-09T18:03:26Z

Hi I am encountering the same issue. I have the latest versions of pandas and numpy. Details are below.

h
/anaconda3/lib/python3.4/site-packages/pandas/core/format.py:2012: RuntimeWarning: invalid value encountered in greater
  has_large_values = (abs_vals > 1e8).any()
/anaconda3/lib/python3.4/site-packages/pandas/core/format.py:2013: RuntimeWarning: invalid value encountered in less
  has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
/anaconda3/lib/python3.4/site-packages/pandas/core/format.py:2014: RuntimeWarning: invalid value encountered in greater
  (abs_vals > 0)).any()
Out[135]: 
                               X0      X1      X2
DATE                                                   
2013-01-01 00:00:00+00:00 -0.220539  0.166106       NaN
2013-01-02 00:00:00+00:00 -0.229826  0.304425 -0.023634

pd.__version__
Out[136]: '0.16.1'

np.__version__
Out[137]: '1.9.2'

iyer · 2015-07-23T21:32:09Z

I come across the same issue too. I did a fresh conda install (anaconda-3.2.3) on my linux machine and ran into issues when dataframes started containing nans

Below are the pandas and numpy versions

pd.version
Out[16]: '0.16.2'

np.version
Out[17]: '1.9.2'

This relates to formatting floats (which is what an np.nan is represented as). You can work around it by defining your own formatter function and setting it as an option.

pd.set_option('display.float_format', lambda x:'%f'%x)

jdmcbr · 2016-01-19T21:27:23Z

conda install -f pandas solved this for me, whereas conda remove pandas followed by conda install pandas had not.

nickeubank · 2016-03-09T06:23:35Z

@quantstud @jdmcbr @cousin333 Has this problem stayed fixed for you? I'm still having issues with this -- conda remove pandas / conda install pandas not fixing.

dacoex · 2016-03-09T08:59:29Z

The issue as mentioned above under spyder (RuntimeWarning: with pandas.dataframes spyder-ide/spyder#2991) aslo occurs when using WinPython instead of conda.

nickeubank · 2016-03-09T20:06:50Z

@dacoex have we concluded this is a spyder problem and not a pandas problem?

jdmcbr · 2016-03-09T20:43:04Z

@nickeubank It was not actually fixed, as it turned out. I looked through the issue @dacoex mentions, and that matches my experiences perfectly. After trying the conda install -f pandas suggested here, I happened to have good (or bad, I suppose) luck to not run into the error in multiple tests, so incorrectly thought it was solved. I forgot to come back and say so next time I ran into the error. In any case, it looks like an issue with spyder, not pandas.

nickeubank · 2016-03-09T20:50:05Z

K - @jreback , sounds like this is still an issue for many people. Could we re-open?

jdmcbr · 2016-03-09T20:56:25Z

@nickeubank I just tested, and only saw this issue in a python console within Spyder. It did not occur in a python console outside of Spyder. So I think it is appropriately closed here.

jreback · 2016-03-09T21:15:34Z

again I suspect its a non-complete update for some folks. esp on windows, you have to make sure that nothing is touching the env when you updated things. This has been fixed in newer versions of conda, but if you are doing it some other way then you may have issues.

ccordoba12 · 2016-03-20T19:05:19Z

(Spyder maintainer here). My findings are:

This problem only occurs inside Spyder consoles, and it doesn't have to do with wrong conda installations. I still don't understand why it occurs only in Spyder.
I can reproduce this problem with Pandas 0.17 but not with Pandas 0.18. So to those affected by this bug I suggest you to update :-)
For those who are unable to update, I'll provide a fix for it in Spyder 2.3.9 (to be released this week :-)

Conclusion: Please stop bothering @jreback and instead bother me in spyder-ide/spyder#2991 ;-)

ccordoba12 · 2016-03-20T19:48:00Z

@jreback, is it ok for us to simply filter this warning? I still don't understand where it's coming from (we import Pandas in several places, but those imports don't seem to have an effect on suppressing the warning).

jreback · 2016-03-20T20:20:25Z

@ccordoba12 yeah all of this code was changed for 0.18.0 (unreleated to this, much more to make it DRY & to make sure FloatIndexes were printing correctly).

I relooked at this issue, its on 32-bit windows. See the warning (coming from numpy) below. I would say this is really a numpy bug. I'll create an issue. As far as filtering, sure I don't think it actually hurts anything, but I don't think you need to on 0.18.0.

[pandas_test] C:\Miniconda3-32\Scripts>python
Python 2.7.11 |Continuum Analytics, Inc.| (default, Jan 29 2016, 15:36:56) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy as np
>>> np.__version__
'1.9.2'
>>> np.abs(np.array([0,np.nan],dtype='float64'))
__main__:1: RuntimeWarning: invalid value encountered in absolute
array([  0.,  nan])
>>>

jreback · 2016-03-20T20:21:48Z

xref numpy/numpy#7440

mhabets · 2016-04-07T16:37:47Z

I get the same warning message on linux 64bits in Spyder 2.3.8 as soon as I type the last parenthesis of df[df['Email Domain'].isin(
but not with the mentioned command np.abs(np.array([0,np.nan],dtype='float64'))

Python 3.5.1 |Anaconda 4.0.0 (64-bit)| (default, Dec 7 2015, 11:16:01)
IPython 4.1.2 -- An enhanced Interactive Python.
Numpy 1.10.4
Pandas 0.18.0

I can't reproduce the warning message with the standard ipython console.

ccordoba12 · 2016-04-07T18:07:17Z

@mhabets, as I said, this is going to be fixed in Spyder 2.3.9.

linshilogin · 2016-07-22T01:19:24Z

@ccordoba12 the problem is seems not fixed in Spyder 2.3.9

khairullahhamsafar · 2025-02-19T20:07:24Z

I am getting this type warnings again now in 2025. This problem is raised again. Right now, I am facing on Kaggle:

df1.head()

/usr/local/lib/python3.10/dist-packages/pandas/io/formats/format.py:1458: RuntimeWarning: invalid value encountered in greater
  has_large_values = (abs_vals > 1e6).any()
/usr/local/lib/python3.10/dist-packages/pandas/io/formats/format.py:1459: RuntimeWarning: invalid value encountered in less
  has_small_values = ((abs_vals < 10 ** (-self.digits)) & (abs_vals > 0)).any()
/usr/local/lib/python3.10/dist-packages/pandas/io/formats/format.py:1459: RuntimeWarning: invalid value encountered in greater
  has_small_values = ((abs_vals < 10 ** (-self.digits)) & (abs_vals > 0)).any()
/usr/local/lib/python3.10/dist-packages/pandas/io/formats/format.py:1458: RuntimeWarning: invalid value encountered in greater
  has_large_values = (abs_vals > 1e6).any()
/usr/local/lib/python3.10/dist-packages/pandas/io/formats/format.py:1459: RuntimeWarning: invalid value encountered in less
  has_small_values = ((abs_vals < 10 ** (-self.digits)) & (abs_vals > 0)).any()
/usr/local/lib/python3.10/dist-packages/pandas/io/formats/format.py:1459: RuntimeWarning: invalid value encountered in greater
  has_small_values = ((abs_vals < 10 ** (-self.digits)) & (abs_vals > 0)).any()

How I can resolve this issue?

jreback closed this as completed Apr 20, 2015

jreback added Output-Formatting __repr__ of pandas objects, to_string Numeric Operations Arithmetic, Comparison, and Logical operations labels Apr 20, 2015

jorisvandenbossche added this to the 0.16.2 milestone Jun 3, 2015

jorisvandenbossche reopened this Jun 3, 2015

jreback closed this as completed Jun 3, 2015

dacoex mentioned this issue Feb 17, 2016

RuntimeWarning with pandas.dataframes that contain np.nan values spyder-ide/spyder#2991

Closed

joshmorel mentioned this issue Feb 26, 2016

pandas with numpy.nan in dictionary causes runtime error spyder-ide/spyder#3008

Closed

wholmgren mentioned this issue May 13, 2016

Overflow in clearsky.haurwitz when apparent_zenith is almost 90 pvlib/pvlib-python#167

Closed

jorisvandenbossche modified the milestones: No action, 0.16.2 Jul 22, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pandas give a warning when working with np.NaN values #9950

pandas give a warning when working with np.NaN values #9950

domenb commented Apr 20, 2015

jreback commented Apr 20, 2015

j1729 commented May 19, 2015

jreback commented May 20, 2015

j1729 commented May 20, 2015

jorisvandenbossche commented Jun 3, 2015

jreback commented Jun 3, 2015

cousin333 commented Jun 3, 2015

jreback commented Jun 3, 2015

quantstud commented Jun 9, 2015

iyer commented Jul 23, 2015

jdmcbr commented Jan 19, 2016

nickeubank commented Mar 9, 2016

dacoex commented Mar 9, 2016

nickeubank commented Mar 9, 2016

jdmcbr commented Mar 9, 2016

nickeubank commented Mar 9, 2016

jdmcbr commented Mar 9, 2016

jreback commented Mar 9, 2016

ccordoba12 commented Mar 20, 2016

ccordoba12 commented Mar 20, 2016

jreback commented Mar 20, 2016

jreback commented Mar 20, 2016

mhabets commented Apr 7, 2016

ccordoba12 commented Apr 7, 2016

linshilogin commented Jul 22, 2016

khairullahhamsafar commented Feb 19, 2025

How I can resolve this issue?

pandas give a warning when working with np.NaN values #9950

pandas give a warning when working with np.NaN values #9950

Comments

domenb commented Apr 20, 2015

jreback commented Apr 20, 2015

j1729 commented May 19, 2015

jreback commented May 20, 2015

j1729 commented May 20, 2015

jorisvandenbossche commented Jun 3, 2015

jreback commented Jun 3, 2015

cousin333 commented Jun 3, 2015

jreback commented Jun 3, 2015

quantstud commented Jun 9, 2015

iyer commented Jul 23, 2015

jdmcbr commented Jan 19, 2016

nickeubank commented Mar 9, 2016

dacoex commented Mar 9, 2016

nickeubank commented Mar 9, 2016

jdmcbr commented Mar 9, 2016

nickeubank commented Mar 9, 2016

jdmcbr commented Mar 9, 2016

jreback commented Mar 9, 2016

ccordoba12 commented Mar 20, 2016

ccordoba12 commented Mar 20, 2016

jreback commented Mar 20, 2016

jreback commented Mar 20, 2016

mhabets commented Apr 7, 2016

ccordoba12 commented Apr 7, 2016

linshilogin commented Jul 22, 2016

khairullahhamsafar commented Feb 19, 2025

I am getting this type warnings again now in 2025. This problem is raised again. Right now, I am facing on Kaggle:

How I can resolve this issue?