Skip to content

BUG: timedelta64(NaT) incorrectly treated as datetime in some dataframe ops #28049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Sep 20, 2019

Conversation

jbrockmendel
Copy link
Member

  • closes #xxxx
  • tests added / passed
  • passes black pandas
  • passes git diff upstream/master -u -- "*.py" | flake8 --diff
  • whatsnew entry

@jbrockmendel
Copy link
Member Author

@TomAugspurger thoughts? lots of ops stuff in the works, getting really close to fixing the perf problem

@TomAugspurger TomAugspurger added this to the 1.0 milestone Sep 3, 2019
Copy link
Contributor

@TomAugspurger TomAugspurger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Could a user hit this from normal code? (i.e. do we need a release note?)

@jbrockmendel
Copy link
Member Author

Could a user hit this from normal code? (i.e. do we need a release note?)

Yes. e.g. the test this implements incorrectly raises TypeError on master. Will update with release note.

@jbrockmendel
Copy link
Member Author

@jreback gentle ping

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, just move the note to 1.0

@@ -18,7 +18,7 @@ Categorical

Datetimelike
^^^^^^^^^^^^

- Bug in :class:`DataFrame` arithmetic operations when operating with a :class:`Series` with dtype `'timedelta64[ns]'` (:issue:`28049`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can move to 1.0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated+green

Copy link
Member

@WillAyd WillAyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor clarification point. Was about to merge but want to double check before.

Note I fixed up unneeded changes to the 0.25.2 whatsnew so would need to pull locally if a change required here (or edit directly in GH)

right = np.asarray(right)

def column_op(a, b):
return {i: func(a.iloc[:, i], b[i]) for i in range(len(a.columns))}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to double check - is the second argument to func supposed to be accessed via .iloc here as well?

Side note - an alternate approach for by column iteration is to call df.columns; not sure if there is a perf difference but have seen both in code base

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the second argument to func supposed to be accessed via .iloc here as well?

No, at this point b is an ndarray.

Side note - an alternate approach for by column iteration is to call df.columns; not sure if there is a perf difference but have seen both in code base

That runs in to difficulties if there are duplicate columns.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if there is a perf difference

Might be worth looking at using iat instead of iloc for perf

@WillAyd
Copy link
Member

WillAyd commented Sep 18, 2019 via email

@jreback
Copy link
Contributor

jreback commented Sep 20, 2019

lgtm merge on green

@jbrockmendel
Copy link
Member Author

@WillAyd gentle ping

@WillAyd WillAyd merged commit f08a1e6 into pandas-dev:master Sep 20, 2019
@WillAyd
Copy link
Member

WillAyd commented Sep 20, 2019

Thanks @jbrockmendel

@jbrockmendel jbrockmendel deleted the tdfix branch September 20, 2019 23:45
proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019
proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants