PERF: avoid creating many Series in apply_standard #34909
Conversation
| mgr = ser._mgr | ||
| blk = mgr.blocks[0] | ||
|
|
||
| for (arr, name) in zip(values, self.index): |
There was a problem hiding this comment.
can you push this to an internals method instead?
There was a problem hiding this comment.
im looking at that now. the other place where this pattern could be really useful is in groupby.ops, but its tougher there
There was a problem hiding this comment.
sure also exposing an api for this would be ok as well (eg another internals method)
There was a problem hiding this comment.
im still troubleshooting the groupby.ops usage, would like to punt on making this an internals method for the time being
…rf-apply_standard
jreback
added
Apply
|
does this fix #34506 ? |
|
The groupby.ops analogue of this is continuing to be a PITA, so im exploring other avenues there (xred #34982). This should be good to go though, and as a follow-up we can rip out libreduction.Reducer. |
|
thanks, yeah incremental refactors are good. |
|
This commit caused some regressions, eg https://pandas.pydata.org/speed/pandas/#frame_methods.Apply.time_apply_np_mean, Was that expected? (above you mention "achieves parity on that asv.") |
|
Ah, I suppose this is just fixed by #35166 |
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffThis avoids going through perennial-problem-causing libreduction code (xref #34014, #34080) and instead does the same trick in python-space by re-assigning block.values instead of creating new Series objects.
If we avoid libreduction but dont do this optimization, the most-affected asv is
time_apply_ref_by_namethat clocks in at 6.92x slower. This achieves parity on that asv.ATM I'm still getting 4 test failures locally, need to troubleshoot.Update: passing