Skip to content

BUG: HDFStore failures on timezone-aware data #20595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

BUG: HDFStore failures on timezone-aware data #20595

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
05cd38f
HDFStore timezone handling for Series and empty DataFrames
adshieh Apr 3, 2018
fc7753e
Whatsnew entry
adshieh Apr 3, 2018
9394dcf
Parametrize test on dtype
adshieh Apr 3, 2018
b0c8f82
Use accessor, add tests, and fix whatsnew entry
adshieh Apr 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.23.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1099,6 +1099,7 @@ I/O
- Bug in :meth:`pandas.io.json.json_normalize` where subrecords are not properly normalized if any subrecords values are NoneType (:issue:`20030`)
- Bug in ``usecols`` parameter in :func:`pandas.io.read_csv` and :func:`pandas.io.read_table` where error is not raised correctly when passing a string. (:issue:`20529`)
- Bug in :func:`HDFStore.keys` when reading a file with a softlink causes exception (:issue:`20523`)
- Bug in :class:`HDFStore` for Series and empty DataFrames with timezone-aware data (:issue:`20594`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not very clear, is this for Series generally and empty DataFrames? this is only for a fixed store.


Plotting
^^^^^^^^
Expand Down
44 changes: 21 additions & 23 deletions pandas/io/pytables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2695,30 +2695,24 @@ def write_array(self, key, value, items=None):
_tables().ObjectAtom())
vlarr.append(value)
else:
if empty_array:
if is_datetime64_dtype(value.dtype):
self._handle.create_array(self.group, key, value.view('i8'))
getattr(self.group, key)._v_attrs.value_type = 'datetime64'
elif is_datetime64tz_dtype(value.dtype):
# store as UTC
# with a zone
self._handle.create_array(self.group, key, value.asi8)

node = getattr(self.group, key)
node._v_attrs.tz = _get_tz(value.tz)
node._v_attrs.value_type = 'datetime64'
elif is_timedelta64_dtype(value.dtype):
self._handle.create_array(self.group, key, value.view('i8'))
getattr(self.group, key)._v_attrs.value_type = 'timedelta64'
elif empty_array:
self.write_array_empty(key, value)
else:
if is_datetime64_dtype(value.dtype):
self._handle.create_array(
self.group, key, value.view('i8'))
getattr(
self.group, key)._v_attrs.value_type = 'datetime64'
elif is_datetime64tz_dtype(value.dtype):
# store as UTC
# with a zone
self._handle.create_array(self.group, key,
value.asi8)

node = getattr(self.group, key)
node._v_attrs.tz = _get_tz(value.tz)
node._v_attrs.value_type = 'datetime64'
elif is_timedelta64_dtype(value.dtype):
self._handle.create_array(
self.group, key, value.view('i8'))
getattr(
self.group, key)._v_attrs.value_type = 'timedelta64'
else:
self._handle.create_array(self.group, key, value)
self._handle.create_array(self.group, key, value)

getattr(self.group, key)._v_attrs.transposed = transposed

Expand Down Expand Up @@ -2771,7 +2765,11 @@ def read(self, **kwargs):
def write(self, obj, **kwargs):
super(SeriesFixed, self).write(obj, **kwargs)
self.write_index('index', obj.index)
self.write_array('values', obj.values)
if is_datetime64tz_dtype(obj.dtype):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is reaching into the internal implementation way too much.

you can use obj.get_values() here

Copy link
Author

@adshieh adshieh Apr 5, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with obj.get_values() is that it returns an array with dtype datetime64[ns], but we need a DatetimeIndex to preserve the timezone information. How about obj.dt._get_values()?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so write_array already handles the objects corrctly, try just passing obj into that (you may need to add a case in write_array, but see where this gets you

values = obj._data.blocks[0].values
else:
values = obj.values
self.write_array('values', values)
self.attrs.name = obj.name


Expand Down
20 changes: 16 additions & 4 deletions pandas/tests/io/test_pytables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2792,10 +2792,22 @@ def test_empty_series_frame(self):
self._check_roundtrip(df1, tm.assert_frame_equal)
self._check_roundtrip(df2, tm.assert_frame_equal)

def test_empty_series(self):
for dtype in [np.int64, np.float64, np.object, 'm8[ns]', 'M8[ns]']:
s = Series(dtype=dtype)
self._check_roundtrip(s, tm.assert_series_equal)
@pytest.mark.parametrize('dtype', [
np.int64, np.float64, np.object, 'm8[ns]', 'M8[ns]',
'datetime64[ns, UTC]'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add US/Eastern as well

])
def test_empty_series(self, dtype):
s = Series(dtype=dtype)
self._check_roundtrip(s, tm.assert_series_equal)

def test_series_timezone(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameterize these both with US/Eastern as well

s = Series([0], dtype='datetime64[ns, UTC]')
self._check_roundtrip(s, tm.assert_series_equal)

def test_empty_frame_timezone(self):
s = Series(dtype='datetime64[ns, UTC]')
df = DataFrame({'A': s})
self._check_roundtrip(df, tm.assert_frame_equal)

def test_can_serialize_dates(self):

Expand Down