Build directories are messed up with SHARE_SPHINX_DOCTREE

[mgeier]

Later builds seem to happen in the "doctree" directory of the previous build, then in the "doctree" directory of the "doctree" directory and so on ...

I guess this is happening because of the SHARE_SPHINX_DOCTREE flag.

First build, everything looks OK:

python /home/docs/checkouts/readthedocs.org/user_builds/nbsphinx/envs/latest/bin/sphinx-build -T -j auto -E -b readthedocs -d _build/doctrees -D language=en . _build/html
[...]    
building [readthedocs]: targets for 33 source files that are out of date
updating environment: [new config] 33 added, 0 changed, 0 removed
reading sources... [  3%] a-normal-rst-file
[...]

Second build, source files are suddenly in _build/doctrees/nbsphinx/:

python /home/docs/checkouts/readthedocs.org/user_builds/nbsphinx/envs/latest/bin/sphinx-build -T -j auto -b readthedocssinglehtmllocalmedia -d _build/doctrees -D language=en . _build/localmedia
[...]
updating environment: 58 added, 2 changed, 0 removed
reading sources... [  1%] _build/doctrees/nbsphinx/allow-errors
[...]

Third build, now we are in _build/doctrees/nbsphinx/_build/doctrees/nbsphinx/:

python /home/docs/checkouts/readthedocs.org/user_builds/nbsphinx/envs/latest/bin/sphinx-build -b latex -j auto -D language=en -d _build/doctrees . _build/latex
[...]
updating environment: 58 added, 8 changed, 0 removed
reading sources... [  1%] _build/doctrees/nbsphinx/_build/doctrees/nbsphinx/allow-errors
[...]

... and there are some warning messages, because files are not found, which isn't surprising given the strange build/source directories.

Details

• Read the Docs project URL: https://nbsphinx.readthedocs.io/
• Build URL (if applicable): https://readthedocs.org/projects/nbsphinx/builds/11266830/

[stsewd]

Actually this is happening in all builds, and outside rtd too. The reason is that the "_build" directory gets included into the doctree directory on each run.

I can replicate this even with the defaults from sphinx, so a solution from our side would be to have the build directory outside the docs dir.

Another solution (from your side) is to add exclude_patterns = ['_build'] to your conf.py file (this one is the default from sphinx-quickstart).

I'm not sure if this is causing any problems though, (aside from doing an extra copy operation).

Anyway, I don't think we are going to change our build directories, since there is a lot of code that depends on that structure... But we can add exclude_patterns = ['_build'] to all conf.py files if isn't defined.

[mgeier]

Thanks for looking into this!

Indeed, adding exclude_patterns = ['_build'] solves the problem.

Here's an example: https://readthedocs.org/projects/nbsphinx/builds/11497225/

The problem in my example is that my extension nbsphinx stores the executed Jupyter notebooks in the doctrees directory and copies them to the HTML output dir later.

But since those notebooks are themselves valid source files, they are unnecessarily parsed in the next step.

I'm not sure if this is causing any problems though, (aside from doing an extra copy operation).

Yes, it cancels the effect that SHARE_SPHINX_DOCTREE should have in the first place.

The idea is that in subsequent runs, no more sources have to be parsed, but since there are new sources in the _build directory now, they get parsed again.

It shouldn't have an effect on the result, but it does have an effect on the runtime and it produces spurious warnings.

so a solution from our side would be to have the build directory outside the docs dir.

Yes, that would make very much sense.

I don't think we are going to change our build directories, since there is a lot of code that depends on that structure...

OK, that's understandable.

But we can add exclude_patterns = ['_build'] to all conf.py files if isn't defined.

Yes, please!

Since you (= RTD) are choosing the _build directory, I think it is also your responsibility to add it to exclude_patterns.

[mgeier]

Thanks @stsewd for the quick solution, that's great!