Skip to content

Parallel build for CI #30452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Dec 26, 2019
Merged

Parallel build for CI #30452

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8a227c5
parallel build for CI
WillAyd Dec 24, 2019
4c4d2c1
Merge remote-tracking branch 'upstream/master' into parallel-build
WillAyd Dec 24, 2019
f313e4a
Merge remote-tracking branch 'upstream/master' into parallel-build
WillAyd Dec 26, 2019
fc7cd09
Merge remote-tracking branch 'upstream/master' into parallel-build
WillAyd Dec 26, 2019
9b012d1
nproc
WillAyd Dec 26, 2019
86344a5
alternate syntax
WillAyd Dec 26, 2019
47a8bc2
back to 2
WillAyd Dec 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ci/setup_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ conda list pandas
# Make sure any error below is reported as such

echo "[Build extensions]"
python setup.py build_ext -q -i
python setup.py build_ext -q -i -j2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for using 2 processes? I'd set -j0 and use as many processes are cores available.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this documented somewhere? I tried locally and -j0 doesn't seem to parallelize anything

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, not sure why I was sure this could be automatically parallelized to the number of cores. I think it was intended to have this implemented, see https://github.com/python/cpython/blob/master/Lib/distutils/command/build_ext.py#L453 but in practice there is no way to make parallel equal to True.

Given that, I'd use:

-j `nproc`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the parallel extensions flag not work? is -j the new thing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallel and j are the same thing for distutils:

https://github.com/python/cpython/blob/03c8e5d46673e4704c8dc955d52735966ffdc2a4/Lib/distutils/command/build_ext.py#L87

And have the same requirements for specifying an integer (the assignment to self.parallel as shown in Marc's link I think can be misleading).

Here is the GNU make documentation on that:

https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datapythonista not sure that works for macOS unless you know of an alternate syntax?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know much about mac, but -j $(nproc) is equivalent, and works when the shell is not bash I think

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm so I don't think nproc is available by default in macOS, so reverting to 2 for now (though definitely a nice idea)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like the OSX equivalent is sysctl -n hw.ncpu, but that's one of the namespaces that apple is liable to change every release


# XXX: Some of our environments end up with old versions of pip (10.x)
# Adding a new enough version of pip to the requirements explodes the
Expand Down