Skip to content

BUG: Fixing DataFrame.Update crashes when NaT present #49395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 15, 2022
Merged

BUG: Fixing DataFrame.Update crashes when NaT present #49395

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2d126dd
Fixes DataFrame.update crashes when NaT present. GH16713
wxing11 Oct 30, 2022
b4c792a
Formatting
wxing11 Oct 30, 2022
c63d1cc
No longer reindexing columns, skipping non-matching columns
wxing11 Nov 3, 2022
7bea232
switching to using set intersection to find shared columns
wxing11 Nov 3, 2022
9ee4329
switching to using index intersection
wxing11 Nov 3, 2022
f191629
Removing unnecessary variable creation
wxing11 Nov 8, 2022
35fa0a8
Formatting and removing test
wxing11 Nov 8, 2022
ead244a
Adding to whatsnew
wxing11 Nov 8, 2022
bd07c5a
updating whatsnew message
wxing11 Nov 12, 2022
8a1e0f7
Merge branch 'main' into fixing-dataframe-update-crashes
MarcoGorelli Nov 14, 2022
d8167ea
Update doc/source/whatsnew/v2.0.0.rst
MarcoGorelli Nov 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pandas/core/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8195,6 +8195,10 @@ def update(
for col in self.columns:
this = self[col]._values
that = other[col]._values

if all(isna(that)):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the original issue, it suggests that this may be an issue in the reindex_like operations being incompatible with something, so I don't think a patched-over check like this is appropriate

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thanks for reviewing!

This was my assessment of it in the original issue: #16713 (comment)

Based on that, I think the issue in reindex_like is just that when it creates columns that aren't in the pre-reindex dataframe that they aren't datetime/datetime compatible by default. Is that the part I should be looking to change instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that the part I should be looking to change instead?

Ideally yes, or possibly when the masks are compared.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will look into this more, thanks for the guidance!

continue

if filter_func is not None:
with np.errstate(all="ignore"):
mask = ~filter_func(this) | isna(that)
Expand Down
34 changes: 34 additions & 0 deletions pandas/tests/frame/methods/test_update.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,37 @@ def test_update_modify_view(self, using_copy_on_write):
tm.assert_frame_equal(result_view, df2_orig)
else:
tm.assert_frame_equal(result_view, expected)

def test_update_dt_column_with_NaT_create_column(self):
df = DataFrame(
{
"A": [1, None],
"B": [
pd.NaT,
pd.to_datetime("2016-01-01"),
],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we keep this on a single line?

}
)
df2 = DataFrame({"A": [2, 3]})

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's remove all these newlines in the tests

df.update(df2, overwrite=False)

expected = DataFrame(
{"A": [1.0, 3.0], "B": [pd.NaT, pd.to_datetime("2016-01-01")]}
)

tm.assert_frame_equal(df, expected)

def test_update_dt_column_with_NaT_create_row(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really sure what this test adds, I'd suggest to either:

  • parametrize over the first test
  • just remove this one


df = DataFrame({"A": [1, None], "B": [pd.to_datetime("2017-1-1"), pd.NaT]})

df2 = DataFrame({"A": [2], "B": [pd.to_datetime("2016-01-01")]})

df.update(df2, overwrite=False)

expected = DataFrame(
{"A": [1, None], "B": [pd.to_datetime("2017-1-1"), pd.NaT]}
)

tm.assert_frame_equal(df, expected)