Skip to content

PERF: do DataFrame.op(series, axis=0) blockwise #31296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Mar 8, 2020
Merged

PERF: do DataFrame.op(series, axis=0) blockwise #31296

merged 16 commits into from
Mar 8, 2020

Conversation

jbrockmendel
Copy link
Member

Also fixes the same bug as #31271 (with the same test ported), so if this is accepted that will be closeable.

~2000x speedups for very-wide mixed-dtype cases.

arr = np.arange(10**6).reshape(100, -1)    
df = pd.DataFrame(arr)
df["C"] = 1.0
ser = df[0]

In [11]: %timeit df.eq(ser, axis=0)
1.58 s ± 20 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- master
595 µs ± 79.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  <-- PR

In [12]: %timeit df.add(ser, axis=0)
2.06 s ± 52.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- master                                                                                                                   
1.04 ms ± 5.07 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR

@gfyoung gfyoung added Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Bug Performance Memory or execution speed performance labels Jan 25, 2020
gfyoung
gfyoung reviewed Jan 25, 2020
View reviewed changes
pandas/core/ops/array_ops.py Outdated
@@ -200,6 +213,27 @@ def arithmetic_op(
return res_values


def _broadcast_comparison_op(lvalues, rvalues, op):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Brief) docstring?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add this (and types if you can)

jreback
jreback requested changes Jan 25, 2020
View reviewed changes
pandas/core/ops/array_ops.py
@@ -200,6 +213,27 @@ def arithmetic_op(
return res_values


def _broadcast_comparison_op(lvalues, rvalues, op):
if isinstance(rvalues, np.ndarray):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like the case on L81 in array_ops.py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you follow up with consolidate if this is the case

jreback
jreback reviewed Feb 9, 2020
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, pls rebase. i need to go over again.

@jbrockmendel jbrockmendel added Numeric Operations Arithmetic, Comparison, and Logical operations and removed Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Bug labels Feb 19, 2020
@jbrockmendel jbrockmendel changed the title PERF: do DataFrame.op(series, axis=0) blockwsie PERF: do DataFrame.op(series, axis=0) blockwise Feb 19, 2020
jreback
jreback requested changes Feb 22, 2020
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks fine. rebase. I think this needs the other branch that pushes down str_rep (may have merged already), if not (or already merged), then lgtm. ping on green.

pandas/core/ops/array_ops.py Outdated
@@ -200,6 +213,27 @@ def arithmetic_op(
return res_values


def _broadcast_comparison_op(lvalues, rvalues, op):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add this (and types if you can)

pandas/core/ops/array_ops.py


def _can_broadcast(lvalues, rvalues) -> bool:
# We assume that lengths dont match
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

@jreback jreback added this to the 1.1 milestone Feb 22, 2020
@jreback
Copy link
Contributor

jreback commented Feb 22, 2020

add a whatsnew in perf section

@jbrockmendel
Copy link
Member Author

add a whatsnew in perf section

done

@jbrockmendel
Copy link
Member Author

rebased+green

@jreback
Copy link
Contributor

jreback commented Mar 6, 2020

is the github author thing fixed?

@jbrockmendel
Copy link
Member Author

is the github author thing fixed?

not that im aware of

@jbrockmendel
Copy link
Member Author

GH thing is fixed now i think

jreback
jreback approved these changes Mar 8, 2020
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, followon comment

pandas/core/ops/array_ops.py
@@ -200,6 +213,27 @@ def arithmetic_op(
return res_values


def _broadcast_comparison_op(lvalues, rvalues, op):
if isinstance(rvalues, np.ndarray):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you follow up with consolidate if this is the case

@jreback jreback merged commit 19ae087 into pandas-dev:master Mar 8, 2020
@jreback
Copy link
Contributor

jreback commented Mar 8, 2020

thanks

@jbrockmendel jbrockmendel deleted the perf-arith-series-col branch March 8, 2020 16:28
SeeminSyed pushed a commit to CSCD01-team01/pandas that referenced this pull request Mar 22, 2020
@jbrockmendel @SeeminSyed
PERF: do DataFrame.op(series, axis=0) blockwise (pandas-dev#31296)
403ed44
@jorisvandenbossche jorisvandenbossche mentioned this pull request May 22, 2020
Closed
@simonjayhawkins simonjayhawkins mentioned this pull request Aug 5, 2020
Open
@AlexKirko AlexKirko mentioned this pull request Sep 29, 2020
Closed
3 tasks
@rebecca-palmer rebecca-palmer mentioned this pull request Aug 15, 2023
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gfyoung gfyoung gfyoung left review comments

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance
Projects
None yet
Milestone
1.1
Development

Successfully merging this pull request may close these issues.
3 participants
@jbrockmendel @jreback @gfyoung