BUG: preserve timezone info when writing empty tz-aware series to HDF5 #37072
Conversation
arw2019
force-pushed
the
GH20594
branch
2 times, most recently
from
11f00bb to
869bf8c
Compare
pandas/io/pytables.py
Outdated
| @@ -2960,11 +2960,17 @@ def write_array_empty(self, key: str, value: ArrayLike): | |||
| node._v_attrs.value_type = str(value.dtype) | |||
| node._v_attrs.shape = value.shape | |||
|
|
|||
| def write_array(self, key: str, value: ArrayLike, items: Optional[Index] = None): | |||
| # TODO: we only have one test that gets here, the only EA | |||
| def write_array(self, key: str, obj, items: Optional[Index] = None): | |||
There was a problem hiding this comment.
why is the typing changed? can you be explicit here
There was a problem hiding this comment.
Added a type hint.
AFAIU for series to_numpy() drops timezone info so we need to pass the original object in SeriesFixed.read(...)and special case here on timezone-aware data
xref (from stale PR that worked on this) https://github.com/pandas-dev/pandas/pull/20595/files#r179816447
pandas/io/pytables.py
Outdated
| elif is_datetime64tz_dtype(obj): | ||
| value = obj.dt._get_values() | ||
| else: | ||
| value = obj.values |
Ok will do! For this one I'll wait until the other is merge (since this is a dependant) |
arw2019
force-pushed
the
GH20594
branch
2 times, most recently
from
d30f5bf to
2ea93bb
Compare
|
@jreback this is ready for re-review
|
pandas/io/pytables.py
Outdated
| # that gets passed is DatetimeArray, and we never have | ||
| # both self._filters and EA | ||
| assert isinstance(value, (np.ndarray, ABCExtensionArray)), type(value) | ||
|
|
||
| if isinstance(obj, (np.ndarray, ABCExtensionArray)): |
There was a problem hiding this comment.
is the annotation of obj as a Series accurate? if so, this branch is never hit
There was a problem hiding this comment.
No it's not. Should be SeriesOrFrame (will fix)
pandas/io/pytables.py
Outdated
| if isinstance(obj, (np.ndarray, ABCExtensionArray)): | ||
| value = obj | ||
| elif is_datetime64tz_dtype(obj): | ||
| value = obj.dt._get_values() |
There was a problem hiding this comment.
i dont know off the top of my head what _get_values returns, is it a DatetimeArray? The standard pattern would be to use extract_array(obj, extract_numpy=True) at the beginning of this method
There was a problem hiding this comment.
_get_values returns a DatetimeIndex
But actually the extract_array pattern does the job of this entire if/elif/else block here - thanks for suggesting! (I'll remember for future)
|
CI Green + ready for re-review |
|
thanks @arw2019 |
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffxref #20595 (stale PR that looked into this)