Skip to content

CLN: avoid values_from_object in reshape.merge #32537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 12, 2020

Conversation

jbrockmendel
Copy link
Member

No description provided.

Copy link
Member

@WillAyd WillAyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check perf?

and is_extension_array_dtype(rk.dtype)
and lk.dtype == rk.dtype
):
elif is_extension_array_dtype(lk.dtype) and lk.dtype == rk.dtype:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this guaranteed to be equivalent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure. How could you have this condition but not is_extension_array_dtype(rk.dtype)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should change to lk.is_dtype_equal(rk) (technically the lk.dtype == rk.dtype is unsafe and we should never use this)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will update

@WillAyd WillAyd added Clean Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Mar 9, 2020
lk = getattr(lk, "_values", lk)._data
rk = getattr(rk, "_values", rk)._data
# Extract the ndarray (UTC-localized) values
# Note: we dont need the dtypes to match, as these can still be compared
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That does happen somewhere else?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, in the EA-dtype check below we require matching dtypes

@jreback jreback added this to the 1.1 milestone Mar 11, 2020
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am thinking this actually enables some cases? though we do have pretty comprehensive testing here.

comment

and is_extension_array_dtype(rk.dtype)
and lk.dtype == rk.dtype
):
elif is_extension_array_dtype(lk.dtype) and lk.dtype == rk.dtype:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should change to lk.is_dtype_equal(rk) (technically the lk.dtype == rk.dtype is unsafe and we should never use this)

@jbrockmendel
Copy link
Member Author

i am thinking this actually enables some cases?

not sure i follow. can you give an example?

@jreback
Copy link
Contributor

jreback commented Mar 11, 2020

i am thinking this actually enables some cases?

not sure i follow. can you give an example?

are there any merge cases that work that didn't previously?

@jbrockmendel
Copy link
Member Author

are there any merge cases that work that didn't previously?

there are some dtype combinations that arent tested that looked sketchy. there's a TODO comment about adding tests for them

@jreback
Copy link
Contributor

jreback commented Mar 11, 2020

are there any merge cases that work that didn't previously?

there are some dtype combinations that arent tested that looked sketchy. there's a TODO comment about adding tests for them

kk

@jreback
Copy link
Contributor

jreback commented Mar 11, 2020

lgtm. merge o ngreen.

@jreback
Copy link
Contributor

jreback commented Mar 11, 2020

hmm, something broke

@jbrockmendel
Copy link
Member Author

updated+green

@WillAyd WillAyd merged commit da67e1e into pandas-dev:master Mar 12, 2020
@WillAyd
Copy link
Member

WillAyd commented Mar 12, 2020

Thanks @jbrockmendel

@jbrockmendel jbrockmendel deleted the no-values_from_object-merge branch March 12, 2020 15:23
SeeminSyed pushed a commit to CSCD01-team01/pandas that referenced this pull request Mar 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Clean Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants