-
-
Notifications
You must be signed in to change notification settings - Fork 18.4k
REF: dont use numexpr in places where it doesnt help. #31984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -95,7 +95,10 @@ def run_binary(self, df, other): | |||
expr.get_test_result() | |||
result = op(df, other) | |||
used_numexpr = expr.get_test_result() | |||
assert used_numexpr, "Did not use numexpr as expected." | |||
|
|||
# We don't currently use numexpr for comparisons in Series, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm, the original and entire purpose IS to use numexpr for comparisions. I agree consistency is important, but this is turned off for DataFrames? (for regular dtypes)????
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yah, the right place to do that is in array_ops (which will cause is to do it for Series too)
I've got a branch that implements dataframe-with-dataframe ops block-wise, but the diff for that is pretty big, so im splitting pieces off. the numexpr-for-series-comparisons part will be next
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok great
yeah comparison ops should. r a pretty big speedup as it’s parallle
computation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, im in the process of splitting off the relevant parts of that branch. That may end up affecting the tests here, but the core.ops edits here will be correct regardless.
I think we need to improve the used_numexpr check in the tests to not just check whether numexpr was called, but whether it actually was used effectively
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also xref #31297 will smooth this out
closing in favor of #32047. |
DataFrame ops will still use it when dispatching to series/block ops.
Perf neutral: