beta 0.8.0b2: creating timeseries with index in year 1400 yields wrong timestamp

[lesteve]

There seems to be some kind of wrap-around phenomenon if your index is too far away from now. For example if I create a timeseries with an index in 1400, the timeseries I get back has an index which is some time in 1984:

In [1]: import datetime, pandas

In [2]: pandas.version.version
Out[2]: '0.8.0b2'

In [3]: ts = pandas.TimeSeries(index=[ datetime.datetime(1400,1,2) ], data = [1])

In [4]: ts
Out[4]: 1984-07-22 23:34:33.709551    1

It doesn't seem to be a display issue because I can do:

In [7]: ts.index[0].to_pydatetime()
Warning: discarding nonzero nanoseconds
Out[7]: datetime.datetime(1984, 7, 22, 23, 34, 33, 709551)

I tested I could get the same behaviour both using 32-bit and 64-bit python on windows. It seems to work fine for 0.7.3:

In [1]: import datetime, pandas

In [2]: pandas.version.version
Out[2]: '0.7.3'

In [3]: ts = pandas.TimeSeries(index=[datetime.datetime(1400,1,2)], data = [1])

In [4]: ts.index[0]
Out[4]: datetime.datetime(1400, 1, 2, 0, 0)

[lesteve]

Actually it seems to work fine when constructing the index from strings instead of datetimes:

In [1]: import pandas

In [2]: ts = pandas.TimeSeries(index=['1400-02-01'], data = [1])

In [3]: ts
Out[3]: 1400-02-01    1

and the numpy.datetime64 construction from datetime.datetime seems fine as well:

In [5]: np.datetime64(datetime.datetime(1400,1,2))
Out[5]: 1400-01-02 00:00:00

so the blame seems to be on pandas for this one.

[wesm]

DatetimeIndex, which uses a nanosecond unit, doesn't support dates that far back in time. This was a deliberate design tradeoff in work revamping the time series processing infrastructure. You can, however, represent dates that far back using PeriodIndex. Could you tell me a bit more about your use case?

[lesteve]

The use case is the following: we are using the Boost.Date_time library in our C++ application. Timeseries created in our C++ application can be queried using python and we are looking into using pandas for the timeseries objects returned inside python instead of some bespoke python timeseries classes we are currently using.

In the Boost.Date_time library, 1400-Jan-01 00:00:00 is the minimum timestamp and 9999-Dec-31 23:59:59.999999 is the maximum timestamp. As such our timeseries can have any timestamps in between in principle, although our main use case of crazy (where crazy means very far away from our current times) timestamps is due to constant timeseries which have a single timestamp (1400-Jan-01). In most cases with crazy indices, timeseries have a single timestamp on 1400-Jan-01 and then a big gap until more reasonable indices come along.

Out of interest, is the limitation in DatetimeIndex documented somewhere? In particular what are the minimum and maximum timestamp I can use with DatetimeIndex?

[wesm]

I've just pushed some checks for out of bounds timestamps and documented the limitation here (with suggested workaround for representing pre-year 1677 and post-year 2262 dates):

http://pandas.pydata.org/pandas-docs/dev/gotchas.html#timestamp-limitations

[wesm]

Actually going to leave this open. Need to do more rigorous checking when converting a list or array of datetime.datetime to DatetimeIndex (your code example is still an issue)

[lesteve]

Thanks for that I do get an error now. I still doesn't get any error if I use PeriodIndex.to_datetime:

In [1]: import pandas

In [2]: pandas.version.version
Out[2]: '0.8.0.dev-87d6da1'

In [3]: ts = pandas.TimeSeries(index=pandas.PeriodIndex([pandas.Period('1400-01-01')]), data=[1])

In [4]: ts.index.to_datetime()
Out[4]:
<class 'pandas.tseries.index.DatetimeIndex'>
[1969-12-31 23:59:59.999791812, ..., 1969-12-31 23:59:59.999791812]
Length: 1, Freq: None, Timezone: None

Is that expected?

Also out of interest, is using PeriodIndex vs DatetimeIndex having impact on performance, for example when aligning indexes?

[wesm]

Obviously not? =) There are many more places where overflow checks need to be inserted.

Why not just replace the minimum Boost timestamp with a crazy, but in range pandas timestamp (like 1700-01-01)}

Performance in alignment should be roughly equivalent (int64 data under the hood).

[lesteve]

The timestamp replacement from a crazy to another crazy but in range timestamp was something I thought about but we might have some edge use cases, although very few of them I would guess, where you need dates earlier than 1677.

Maybe going with PeriodIndex is the best solution for now.