Skip to content

BUG: to_hdf append string column to incompatible column #41897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 17, 2021
Merged

BUG: to_hdf append string column to incompatible column #41897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
197782c
BUG: to_hdf append string column to incompatible column
lrepiton Jun 9, 2021
ed5143a
Wrapping long lines for PEP 8
lrepiton Jun 9, 2021
0641bb1
Adding PR number in comments
lrepiton Jun 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1064,6 +1064,7 @@ I/O
- Bug in the conversion from PyArrow to pandas (e.g. for reading Parquet) with nullable dtypes and a PyArrow array whose data buffer size is not a multiple of the dtype size (:issue:`40896`)
- Bug in :func:`read_excel` would raise an error when pandas could not determine the file type, even when user specified the ``engine`` argument (:issue:`41225`)
- Bug in :func:`read_clipboard` copying from an excel file shifts values into the wrong column if there are null values in first column (:issue:`41108`)
- Bug in :meth:`DataFrame.to_hdf` and :meth:`Series.to_hdf` raising a ``TypeError`` when trying to append a string column to an incompatible column (:issue:`41897`)

Period
^^^^^^
Expand Down
2 changes: 1 addition & 1 deletion pandas/io/pytables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4988,7 +4988,7 @@ def _maybe_convert_for_string_atom(
# check for column in the values conflicts
if existing_col is not None:
eci = existing_col.validate_col(itemsize)
if eci > itemsize:
if eci is not None and eci > itemsize:
itemsize = eci

data_converted = data_converted.astype(f"|S{itemsize}", copy=False)
Expand Down
16 changes: 16 additions & 0 deletions pandas/tests/io/pytables/test_append.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -774,6 +774,22 @@ def test_append_raise(setup_path):
with pytest.raises(ValueError, match=msg):
store.append("df", df)

# incompatible type (GH 41897)
_maybe_remove(store, "df")
df["foo"] = Timestamp("20130101")
store.append("df", df)
df["foo"] = "bar"
msg = re.escape(
"invalid combination of [values_axes] on appending data "
"[name->values_block_1,cname->values_block_1,"
"dtype->bytes24,kind->string,shape->(1, 30)] "
"vs current table "
"[name->values_block_1,cname->values_block_1,"
"dtype->datetime64,kind->datetime64,shape->None]"
)
with pytest.raises(ValueError, match=msg):
store.append("df", df)


def test_append_with_timedelta(setup_path):
# GH 3577
Expand Down