Skip to content

BUG: to_hdf and HDFStore for subclasses #38262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Dec 19, 2020
6 changes: 4 additions & 2 deletions pandas/io/pytables.py
Original file line number Diff line number Diff line change
Expand Up @@ -1646,8 +1646,10 @@ def error(t):
"nor a value are passed"
)
else:
_TYPE_MAP = {Series: "series", DataFrame: "frame"}
pt = _TYPE_MAP[type(value)]
if isinstance(value, Series):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total nit but black permitting we could do

pt = "series" if isinstance(value, Series) else "frame"

pt = "series"
else:
pt = "frame"

# we are actually a table
if format == "table":
Expand Down
43 changes: 43 additions & 0 deletions pandas/tests/io/pytables/test_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -4888,6 +4888,49 @@ def test_unsuppored_hdf_file_error(self, datapath):
with pytest.raises(ValueError, match=message):
pd.read_hdf(data_path)

def test_supported_for_subclasses_dataframe(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put in a new file test_subclass.py

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to a separate file.

Right, read_hdf does not recreate the subclass. It seems that it would return the DataFrame with the same content.

class SubDataFrame(DataFrame):
@property
def _constructor(self):
return SubDataFrame

data = {"a": [1, 2], "b": [3, 4]}
sdf = SubDataFrame(data, dtype=np.intp)

expected = np.array([[1, 3], [2, 4]], dtype=np.intp)

with ensure_clean_path("temp.h5") as path:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as for series

sdf.to_hdf(path, "df")
result = read_hdf(path, "df").values
tm.assert_numpy_array_equal(result, expected)

with ensure_clean_path("temp.h5") as path:
with HDFStore(path) as store:
store.put("df", sdf)
result = read_hdf(path, "df").values
tm.assert_numpy_array_equal(result, expected)

def test_supported_for_subclasses_series(self):
class SubSeries(Series):
@property
def _constructor(self):
return SubSeries

sser = SubSeries([1, 2, 3], dtype=np.intp)

expected = np.array([1, 2, 3], dtype=np.intp)

with ensure_clean_path("temp.h5") as path:
sser.to_hdf(path, "ser")
result = read_hdf(path, "ser").values
tm.assert_numpy_array_equal(result, expected)

with ensure_clean_path("temp.h5") as path:
with HDFStore(path) as store:
store.put("ser", sser)
result = read_hdf(path, "ser").values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i remember something about preferring to_numpy over values but obv both work

tm.assert_numpy_array_equal(result, expected)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it worth leaving TODOs here about switching to assert_series_equal later

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured out that the problem with the assertion was that actually read_hdf would return DataFrame, not SubDataFrame. As a result assertion failed at the stage of comparing types.
Probably need to figure out how one can extract the same subclass of the DataFrame by using read_hdf...
But I guess, this is slightly beyond the scope of the present PR.

So, now I expect that read_hdf would return DataFrame with the given values and datatypes and use it for the assertion via assert_frame_equal.



@pytest.mark.parametrize("bad_version", [(1, 2), (1,), [], "12", "123"])
def test_maybe_adjust_name_bad_version_raises(bad_version):
Expand Down