BUG: to_hdf and HDFStore for subclasses #38262
Conversation
ivanovmg
force-pushed
the
to_hdf_33748
branch
from
3690de4 to
a98e2c2
Compare
ivanovmg
force-pushed
the
to_hdf_33748
branch
from
a98e2c2 to
e38fae8
Compare
ivanovmg
force-pushed
the
to_hdf_33748
branch
from
c831cab to
6633145
Compare
| with HDFStore(path) as store: | ||
| store.put("ser", sser) | ||
| result = read_hdf(path, "ser").values | ||
| tm.assert_numpy_array_equal(result, expected) |
There was a problem hiding this comment.
is it worth leaving TODOs here about switching to assert_series_equal later
There was a problem hiding this comment.
I figured out that the problem with the assertion was that actually read_hdf would return DataFrame, not SubDataFrame. As a result assertion failed at the stage of comparing types.
Probably need to figure out how one can extract the same subclass of the DataFrame by using read_hdf...
But I guess, this is slightly beyond the scope of the present PR.
So, now I expect that read_hdf would return DataFrame with the given values and datatypes and use it for the assertion via assert_frame_equal.
| with ensure_clean_path("temp.h5") as path: | ||
| with HDFStore(path) as store: | ||
| store.put("ser", sser) | ||
| result = read_hdf(path, "ser").values |
There was a problem hiding this comment.
i remember something about preferring to_numpy over values but obv both work
|
|
||
| expected = np.array([[1, 3], [2, 4]], dtype=np.intp) | ||
|
|
||
| with ensure_clean_path("temp.h5") as path: |
| @@ -4888,6 +4888,50 @@ def test_unsuppored_hdf_file_error(self, datapath): | |||
| with pytest.raises(ValueError, match=message): | |||
| pd.read_hdf(data_path) | |||
|
|
|||
| def test_supported_for_subclasses_dataframe(self): | |||
There was a problem hiding this comment.
put in a new file test_subclass.py
There was a problem hiding this comment.
Moved to a separate file.
Right, read_hdf does not recreate the subclass. It seems that it would return the DataFrame with the same content.
ivanovmg
force-pushed
the
to_hdf_33748
branch
from
47bc2f5 to
6088f4c
Compare
| from pandas.io.pytables import HDFStore, read_hdf | ||
|
|
||
|
|
||
| class TestHDFStoreSubclass: |
There was a problem hiding this comment.
in pandas._testing we have SubclassedSeries and SubclassedDataFrame. Should we be using those here?
There was a problem hiding this comment.
Thank you for pointing this out!
Changed that.
| @@ -1646,8 +1646,10 @@ def error(t): | |||
| "nor a value are passed" | |||
| ) | |||
| else: | |||
| _TYPE_MAP = {Series: "series", DataFrame: "frame"} | |||
| pt = _TYPE_MAP[type(value)] | |||
| if isinstance(value, Series): | |||
There was a problem hiding this comment.
total nit but black permitting we could do
pt = "series" if isinstance(value, Series) else "frame"
|
thanks @ivanovmg |
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffFix the bug that the subclasses of a dataframe and a series cannot be saved
to_hdf.The tests were added.
When testing I found that
assert_frame_equaldoes not work on subclasses as well - probably need to fix it in a separate PR.For now in the tests I compare values only.