Parallel build for CI #30452
Parallel build for CI #30452
Conversation
|
lgtm pending green |
|
Nice! - I was just about to ask on #30356 why we don't do this in CI |
|
@alimcmaster1 tried previously in #30214 but had to revert as the Py36 builds would intermittently fail thereafter. Certainly open to any input you have on this |
|
can you rebase again so we have a fresh comparison vs master. can you show the build or CI times (just see a representative sample) |
| @@ -121,7 +121,7 @@ conda list pandas | |||
| # Make sure any error below is reported as such | |||
|
|
|||
| echo "[Build extensions]" | |||
| python setup.py build_ext -q -i | |||
| python setup.py build_ext -q -i -j2 | |||
There was a problem hiding this comment.
Any reason for using 2 processes? I'd set -j0 and use as many processes are cores available.
There was a problem hiding this comment.
Is this documented somewhere? I tried locally and -j0 doesn't seem to parallelize anything
There was a problem hiding this comment.
You're right, not sure why I was sure this could be automatically parallelized to the number of cores. I think it was intended to have this implemented, see https://github.com/python/cpython/blob/master/Lib/distutils/command/build_ext.py#L453 but in practice there is no way to make parallel equal to True.
Given that, I'd use:
-j `nproc`
There was a problem hiding this comment.
does the parallel extensions flag not work? is -j the new thing?
There was a problem hiding this comment.
Parallel and j are the same thing for distutils:
And have the same requirements for specifying an integer (the assignment to self.parallel as shown in Marc's link I think can be misleading).
Here is the GNU make documentation on that:
https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel
There was a problem hiding this comment.
@datapythonista not sure that works for macOS unless you know of an alternate syntax?
There was a problem hiding this comment.
I don't know much about mac, but -j $(nproc) is equivalent, and works when the shell is not bash I think
There was a problem hiding this comment.
Hmm so I don't think nproc is available by default in macOS, so reverting to 2 for now (though definitely a nice idea)
There was a problem hiding this comment.
it looks like the OSX equivalent is sysctl -n hw.ncpu, but that's one of the namespaces that apple is liable to change every release
|
Overall build time tends to fluctuate but this looks to chop off 2-5 minutes on average from non-Windows |
|
lgtm. thanks @WillAyd |
Letting this sit in draft state for a while to see if issues pop up
Note that this doesn't affect Windows (not sure yet if it can). Also previous attempts had
-j4but I believe we only get 2 virtual cores for the CI machines so that was probably overkill.