DOC: enable parallel building of the docs

[jorisvandenbossche]

Sphinx supports building the docs in parallel using multiple processes (with eg -j 4), see http://www.sphinx-doc.org/en/stable/invocation.html#cmdoption-sphinx-build-j

I quickly tested this, and that seems to reduce on my laptop (with 4 processes) build time from ca 17 min to 8 min.
This also takes a lot more memory, so not sure if we want to enable it by default, but it would be nice to at least give the possibility to specify this with python make.py html.

@jreback do you know if we can use multiple processes in parallel on travis?

What this needs:

• update the sphinxext (ipython directive and numpydoc) to allow parallel read (incorporate some upstream changes into our vendored versions)
• update make.py to be able to pass the -j argument

[jreback]

we do this from for testing now with pytest-xdist
but only -n 2

as travis machines have 2 cores (and more doesn't seem to help)

[jorisvandenbossche]

OK, so the option might be useful for building the docs on travis as well.

[jreback]

I would also pass a single -v (which is only slightly more verbose)

[jorisvandenbossche]

@jreback reopening this, as this not yet fully solved (the vendored sphinx extension need updates (otherwise only the writing is parallel), and optionally provide this as an option to make.py)

[jreback]

the vendored sphinx extension need updates (otherwise only the writing is parallel

what does this mean? (e.g. 1.5.1 i not working)?

optionally provide this as an option to make.py

why I don't think this is a big deal anyhow

[jorisvandenbossche]

Our vendored versions are not 'parallel_read_safe' to sphinx, and therefore you get the message like

WARNING: the ipython_sphinxext.ipython_directive extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit
WARNING: doing serial read

and it is actually not using the parallel feature (for reading)

[jreback]

interesting, I don't see that locally....

[datapythonista]

What's the decision regarding this thread? Does it make sense to add a --jobs option to make.py that is given to sphinx-build -j argument? I guess that could be useful for contributors with many cores, even if the CI has only 2.