Skip to content

PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e6 #40502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Mar 20, 2021
Merged

PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e6 #40502

merged 4 commits into from
Mar 20, 2021

Conversation

jorisvandenbossche
Copy link
Member

See #40500 for the analysis.

@jorisvandenbossche jorisvandenbossche added Performance Memory or execution speed performance Numeric Operations Arithmetic, Comparison, and Logical operations labels Mar 18, 2021
@jorisvandenbossche jorisvandenbossche changed the title PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e5 PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e6 Mar 19, 2021
@jorisvandenbossche jorisvandenbossche mentioned this pull request Mar 19, 2021
Closed
@jreback
Copy link
Contributor

jreback commented Mar 19, 2021

prob need to update the docs a bit here https://pandas.pydata.org/docs/user_guide/enhancingperf.html

@jorisvandenbossche
Copy link
Member Author

prob need to update the docs a bit here https://pandas.pydata.org/docs/user_guide/enhancingperf.html

That page is about using numexpr in eval/query, so that is independent from this change. On the basic page is a section about accelerated ops: https://pandas.pydata.org/docs/user_guide/basics.html?highlight=numexpr#accelerated-operations
But it doesn't mention the minimum number of items, so that doesn't need to be updated (we should update the table id at some point switching to the ArrayManager, as that will be influenced, but so that's still for the future)

jbrockmendel
jbrockmendel approved these changes Mar 19, 2021
View reviewed changes
Copy link
Member

@jbrockmendel jbrockmendel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending green

@jreback jreback added this to the 1.3 milestone Mar 20, 2021
@jreback jreback merged commit bd8c79f into pandas-dev:master Mar 20, 2021
@jbrockmendel
Copy link
Member

Looks like this wasnt green when merged, could be responsible for broken CI?

jbrockmendel added a commit that referenced this pull request Mar 20, 2021
@jbrockmendel
Revert "PERF: increase the minimum number of elements to use numexpr …
7269f5c 
…for ops from 1e4 to 1e6 (#40502)"

This reverts commit bd8c79f.
@jbrockmendel jbrockmendel mentioned this pull request Mar 20, 2021
Merged
rhshadrach pushed a commit that referenced this pull request Mar 20, 2021
@jbrockmendel
Revert "PERF: increase the minimum number of elements to use numexpr …
9c4c821 
…for ops from 1e4 to 1e6 (#40502)" (#40541)
@jorisvandenbossche jorisvandenbossche deleted the ops-perf-numexpr-min-elements branch March 24, 2021 08:28
@jorisvandenbossche jorisvandenbossche mentioned this pull request Mar 24, 2021
Merged
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@jorisvandenbossche @JulianWgs
PERF: increase the minimum number of elements to use numexpr for ops …
31283ff 
…from 1e4 to 1e6 (pandas-dev#40502)
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@jbrockmendel @JulianWgs
Revert "PERF: increase the minimum number of elements to use numexpr …
8c719c5 
…for ops from 1e4 to 1e6 (pandas-dev#40502)" (pandas-dev#40541)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonjayhawkins simonjayhawkins simonjayhawkins left review comments

@jbrockmendel jbrockmendel jbrockmendel approved these changes

Assignees
No one assigned
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance
Projects
None yet
Milestone
1.3
Development

Successfully merging this pull request may close these issues.
4 participants
@jorisvandenbossche @jreback @jbrockmendel @simonjayhawkins