Skip to content

REF: dont use numexpr in places where it doesnt help. #31984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

REF: dont use numexpr in places where it doesnt help. #31984

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ac37a18
dont use numexpr where it wont help
jbrockmendel Feb 12, 2020
3487b4e
Merge branch 'master' of https://github.com/pandas-dev/pandas into le…
jbrockmendel Feb 12, 2020
ba9e2dd
Merge branch 'master' of https://github.com/pandas-dev/pandas into le…
jbrockmendel Feb 12, 2020
c023fb1
Merge branch 'master' of https://github.com/pandas-dev/pandas into le…
jbrockmendel Feb 13, 2020
c428d6a
Merge branch 'master' of https://github.com/pandas-dev/pandas into le…
jbrockmendel Feb 14, 2020
30f3e53
Merge branch 'master' of https://github.com/pandas-dev/pandas into le…
jbrockmendel Feb 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions pandas/core/ops/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,6 @@ def dispatch_to_series(left, right, func, str_rep=None, axis=None):
"""
# Note: we use iloc to access columns for compat with cases
# with non-unique columns.
import pandas.core.computation.expressions as expressions

right = lib.item_from_zerodim(right)
if lib.is_scalar(right) or np.ndim(right) == 0:
Expand Down Expand Up @@ -419,7 +418,7 @@ def column_op(a, b):
# Remaining cases have less-obvious dispatch rules
raise NotImplementedError(right)

new_data = expressions.evaluate(column_op, str_rep, left, right)
new_data = column_op(left, right)
return new_data


Expand Down
5 changes: 4 additions & 1 deletion pandas/tests/test_expressions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,10 @@ def run_binary(self, df, other):
expr.get_test_result()
result = op(df, other)
used_numexpr = expr.get_test_result()
assert used_numexpr, "Did not use numexpr as expected."

# We don't currently use numexpr for comparisons in Series,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, the original and entire purpose IS to use numexpr for comparisions. I agree consistency is important, but this is turned off for DataFrames? (for regular dtypes)????

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yah, the right place to do that is in array_ops (which will cause is to do it for Series too)

I've got a branch that implements dataframe-with-dataframe ops block-wise, but the diff for that is pretty big, so im splitting pieces off. the numexpr-for-series-comparisons part will be next

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok great

yeah comparison ops should. r a pretty big speedup as it’s parallle
computation

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, im in the process of splitting off the relevant parts of that branch. That may end up affecting the tests here, but the core.ops edits here will be correct regardless.

I think we need to improve the used_numexpr check in the tests to not just check whether numexpr was called, but whether it actually was used effectively

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also xref #31297 will smooth this out

# so dont for DataFrame either.
assert not used_numexpr, "unexpectedly numexpr as expected."
tm.assert_equal(expected, result)

def run_frame(self, df, other, run_binary=True):
Expand Down