Skip to content

PERF: apply perf enhancements #6024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 21, 2014
Merged

PERF: apply perf enhancements #6024

merged 3 commits into from
Jan 21, 2014

Conversation

jreback
Copy link
Contributor

@jreback jreback commented Jan 21, 2014

added vbenches and make most apply operations a fair bit faster

closes #6013

I can't put this one in the vbench as its cython based, but previously was about 100ms (on my machine)
0.12 clocks in at 15ms (the remaining difference in time is due to the sub-indexing, e.g. x['a'] which in theory could also be speeded but a bit tricky and not a lot more to win)

In [6]: %timeit df.apply(lambda x: integrate_f_typed(x['a'], x['b'], x['N']), axis=1)
10 loops, best of 3: 26.2 ms per loop

Here's a related benchmark from #5654 (a version of which is in the vbenches):

In [16]: s = pd.Series(np.arange(4096.))

In [17]: df = pd.DataFrame({i:s for i in range(4096)})

0.12

In [18]: %timeit df.apply((lambda x: np.corrcoef(x, s)[0, 1]), axis=1)
1 loops, best of 3: 1.11 s per loop

with this PR

In [3]: In [18]: %timeit df.apply((lambda x: np.corrcoef(x, s)[0, 1]), axis=1)

1 loops, best of 3: 1.24 s per loop
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_apply_ref_by_name                      |  15.4320 |  32.2250 |   0.4789 | like the cython example above
frame_apply_user_func                        | 166.7013 | 232.7064 |   0.7164 | bench for example right above
frame_apply_axis_1                           | 108.5210 | 109.1857 |   0.9939 |
frame_apply_pass_thru                        |   6.5626 |   6.4860 |   1.0118 |
frame_apply_np_mean                          |   3.9033 |   0.9290 |   4.2018 | this suffers from some construction overhead, not much I can do
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

PERF: allow apply to use the fast-path in mixed type frames except where datelike
      are present
jreback added a commit that referenced this pull request Jan 21, 2014
PERF: apply perf enhancements
@jreback jreback merged commit 2258efd into pandas-dev:master Jan 21, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Perf regression in 0.13+ for df.apply
1 participant