AmbiguousTimeError when retrieving Series from HDFStore in table format with DST change

[JackKelly]

In Pandas 0.17, if I put a Series into an HDFStore in table format, and if this Series contains a day light saving transition, then Pandas crashes with an AmbiguousTimeError.

Here's the minimal code example to reproduce the bug:

import pandas as pd
index = pd.date_range("2013-10-26 23:00", "2013-10-27 01:00",
                      tz="Europe/London", freq="H")
series = pd.Series(0, index=index)
with pd.HDFStore("test.hdf", mode="w") as store:
    store.put("series", series, format='table')
    store["series"]

And here's the traceback:

In [4]: ---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
<ipython-input-4-5deb3c83fafd> in <module>()
----> 1 __pyfile = open('''/tmp/py7201vZi''');exec(compile(__pyfile.read(), '''/home/jack/temp/pandas_dst_bug.py''', 'exec'));__pyfile.close(); import os; os.remove('''/tmp/py7201vZi''')

/home/jack/temp/pandas_dst_bug.py in <module>()
      2 index = pd.date_range("2013-10-26 23:00", "2013-10-27 01:00",
      3                       tz="Europe/London", freq="H")
----> 4 series = pd.Series(0, index=index)
      5 with pd.HDFStore("test.hdf", mode="w") as store:
      6     store.put("series", series, format='table')

/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.pyc in date_range(start, end, periods, freq, tz, normalize, name, closed)
   1912     return DatetimeIndex(start=start, end=end, periods=periods,
   1913                          freq=freq, tz=tz, normalize=normalize, name=name,
-> 1914                          closed=closed)
   1915 
   1916 

/usr/local/lib/python2.7/dist-packages/pandas/util/decorators.pyc in wrapper(*args, **kwargs)
     87                 else:
     88                     kwargs[new_arg_name] = new_arg_value
---> 89             return func(*args, **kwargs)
     90         return wrapper
     91     return _deprecate_kwarg

/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.pyc in __new__(cls, data, freq, start, end, periods, copy, name, tz, verify_integrity, normalize, closed, ambiguous, dtype, **kwargs)
    234             return cls._generate(start, end, periods, name, freq,
    235                                  tz=tz, normalize=normalize, closed=closed,
--> 236                                  ambiguous=ambiguous)
    237 
    238         if not isinstance(data, (np.ndarray, Index, ABCSeries)):

/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.pyc in _generate(cls, start, end, periods, name, offset, tz, normalize, ambiguous, closed)
    450 
    451                 if end is not None and end.tz is None:
--> 452                     end = end.tz_localize(tz)
    453 
    454             if start and end:

pandas/tslib.pyx in pandas.tslib.Timestamp.tz_localize (pandas/tslib.c:11965)()

pandas/tslib.pyx in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:64516)()

AmbiguousTimeError: Cannot infer dst time from Timestamp('2013-10-27 01:00:00'), try using the 'ambiguous' argument

Everything works fine if:

• we use format=fixed instead of format=table
• we used format=table but the Series does not include a DST change

I'm not certain but I expect this bug is related to #11619

I'm 99.999% certain that this bug did not exist in Pandas 0.16.2

In [30]: show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Linux
OS-release: 3.19.0-33-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

pandas: 0.17.0
nose: 1.3.7
pip: 1.5.6
setuptools: 15.2
Cython: 0.23.1
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 4.0.0
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.6
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.2
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: 0.9
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.5.3 (dt dec pq3 ext)

Finally, just to say another huge thank you to everyone who supports Pandas. It must be a huge amount of work and I am hugely grateful. Pandas is an awesome tool. Thank you.

[jreback]

you understand that this has nothing to do with HDF5 at all. This is failing at the index construction.

index = pd.date_range("2013-10-26 23:00", "2013-10-27 01:00",
                      tz="Europe/London", freq="H")
series = pd.Series(0, index=index)

[JackKelly]

Ah, yes, sorry - I'm really not sure why I got confused about that... I'll close this issue and open a new one...

[jreback]

no its ok, just about to push a fix anyhow

[JackKelly]

oops, sorry, I was over-eager :)

I'll stop fiddling now so I'll leave it up to you which of these two issues to keep open / kill :)

Thanks again,
Jack