OverflowError when trying to use date/times

[adamwasi]

Hello, I have an issue that I posted to SO that wasn't able to be resolved there, seems like it should be straightforward.

https://stackoverflow.com/questions/47227268/pandas-dataset-overflowerror-when-trying-to-use-datetime-data

I have since hardcoded in some sample data to ease error reproduction:

import datetime
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import matplotlib.cbook as cbook
import pandas
import StringIO

##f = open(r'clean data.csv')
##
###Make a string buffer and read in the CSV file while stripping \x00's
##output = StringIO.StringIO()
##for x in f.readlines():
##    output.write(x.replace('\x00',''))
##
###Close out input file
##f.close()
##
###Set position back to start for pandas to read
##output.seek(0)

datain = '''Time,101 <Stuff1> (C),102 <Stuff2> (C),103 <Stuff3> (C),104 <Stuff4> (C),107 <Stuff7> (C),108 <Air> (C)
8/25/2017 14:54,20.727,20.897,20.955,21.025,21.064,21.216
8/25/2017 14:55,20.851,20.953,21.025,21.055,21.1,21.264
8/25/2017 14:56,20.808,20.956,21.018,21.037,21.101,21.282
8/25/2017 14:57,20.799,20.944,20.99,21.029,21.095,21.189
8/25/2017 14:58,-9.824,21.006,21.026,21.064,21.093,21.218'''

output = StringIO.StringIO()
for x in datain.split('\n'):
    output.write(x+'\n')
output.seek(0)

#, skiprows=38
df = pandas.read_csv(output, parse_dates=['Time'], index_col="Time")

fig, ax = plt.subplots()
ax.xaxis.set_major_locator(mdates.DayLocator())
ax.format_xdata = mdates.DateFormatter('%Y-%m-%d')
ax.plot(df.index,df['108 <Air> (C)'])
fig.autofmt_xdate()
plt.show()

Result:

Traceback (most recent call last):
  File "C:\Users\awasilen\Desktop\graph.py", line 41, in <module>
    fig.autofmt_xdate()
  File "C:\Python27\lib\site-packages\matplotlib\figure.py", line 491, in autofmt_xdate
    for label in self.axes[0].get_xticklabels(which=which):
  File "C:\Python27\lib\site-packages\matplotlib\axes\_base.py", line 3054, in get_xticklabels
    which=which))
  File "C:\Python27\lib\site-packages\matplotlib\axis.py", line 1234, in get_ticklabels
    return self.get_majorticklabels()
  File "C:\Python27\lib\site-packages\matplotlib\axis.py", line 1188, in get_majorticklabels
    ticks = self.get_major_ticks()
  File "C:\Python27\lib\site-packages\matplotlib\axis.py", line 1339, in get_major_ticks
    numticks = len(self.get_major_locator()())
  File "C:\Python27\lib\site-packages\matplotlib\dates.py", line 871, in __call__
    dmin, dmax = self.viewlim_to_dt()
  File "C:\Python27\lib\site-packages\matplotlib\dates.py", line 832, in viewlim_to_dt
    return num2date(vmin, self.tz), num2date(vmax, self.tz)
  File "C:\Python27\lib\site-packages\matplotlib\dates.py", line 441, in num2date
    return _from_ordinalf(x, tz)
  File "C:\Python27\lib\site-packages\matplotlib\dates.py", line 256, in _from_ordinalf
    dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC)
OverflowError: Python int too large to convert to C long

[TomAugspurger]

You're trying to use matplotlib's DateFormatter on pandas' datetimes, which doesn't currently work (though I suspect it will in the next version of matplotlib)

See #18344 and https://github.com/matplotlib/matplotlib/pull/9794/files

[jorisvandenbossche]

@TomAugspurger any idea why we now get at once two such bug reports? It seems coincidental with the matplotlib registering stuff, but I don't see how it can be influenced by that.

[TomAugspurger]

I think it is just a strange coincidence :)

@adamwasi, any reason you just recently tried to use matplotlib's formatters on pandas objects?

[adamwasi]

@TomAugspurger @jorisvandenbossche If you follow the SO thread, the question before the one I posted I asked how to get the data into matplotlib, someone recommended pandas was good and getting data in, so I used it. I thought they would have known better than me that I wouldn't work for the example I was trying to work off of. Looks like I'm back to square 1 trying to get data in matplotlib again.

[jorisvandenbossche]

@adamwasi If you are using the latest version of pandas (0.21.0), this should work for you.

The reason your example is failing because pandas registers some plotting converters that can deal with datetime64 data, but as a result, the matplotlib formatters (eg the DateFormatter) don't know what to do with such data.
But with the newest version of pandas, this automatic registration is not done anymore. But note that you will need to convert the datetime64 data to python datetime.datetime objects, otherwise matplotlib will fail trying to plot them.

[brandon9vt]

I'm having the same problem. I'm not sure I understand what I should do to fix it. I have pandas 0.21.0 and matplotlib 2.1.0. Do I need to convert them to datetime objects?