Hitting memory limit for figure generation

[r-barnes]

Details

• Project URL: https://richdem.readthedocs.io/en/latest/
• Build URL (if applicable): https://readthedocs.org/projects/richdem/builds/6481868/
• Read the Docs username (if applicable): rbarnes

Expected Result

My project manipulates geospatial raster data. I've included a 16MB test file in the git repo. The documentation includes many beautiful pictures of the test dataset that are generated as the documentation is built using matplotlib's plot directive.

Actual Result

Unfortunately, the build runs out of memory.

Ideas

1. I could switch to a smaller dataset and hope this uses less memory. This kind of guess-and-check strategy is frustrating and may not be sustainable if the number of images increases.

2. I could commit the compiled documentation to a separate repo and upload that to RTD. This prevents the repo from growing every time an image changes. I'm not sure how to tell RTD about this documentation, though.

3. Maybe there's some way of reducing the amount of memory Sphinx is using?

Perhaps you have other thougths? I'm sure I'm not the only one who has had this issue.

[r-barnes]

I decide to try Option (2).

I gamely rewrote matplotlib's plot directive so that it could load cached imagery and committed that imagery to a submodule. The build process should now require no computation from my code, so I'd expect the memory usage to be low.

Unfortunately, I'm still hitting the memory consumption limit. But now I really don't know why.

The build is at: https://readthedocs.org/projects/richdem/builds/6486214/

[r-barnes]

Okay. Everything is working now. I've stashed the big images in a submodule. I'll write a full report tomorrow.

[r-barnes]

I ended up going with Option 2: I still use RTD to generate textual documentation, but generate image documentation on my own computer.

To do so, I modified the matplotlib's Sphinx plot directive in the following ways.

1. I added an option so that the user can specify the output names of images. This eliminates the ambiguity as to which image is associated with which code chunk.

2. I added a configuration option which will place copies of the named output images in a separate directory where they can be version controlled. Images in this directory are copied into the build output prior to running the user's figure generation code; this pre-empts the need to run the code, reducing computation time.

I then modified my Sphinx conf.py file to load and use this new plotting module.

Finally, I saved the resulting imagery in a submodule.

In order to update documentation, I now use the following workflow:

1. Run make html locally.

2. Commit changes to the imagery submodule and push it.

3. Commit changes on the primary repo and push it. This triggers RTD to rebuild.

4. RTD automagically loads the submodule, therefore acquiring the computationally expensive imagery and runs make html on their build server. However, with the imagery present, no intensive computation is done.

Modified conf.py

#This line tells Sphinx to look for modules in the directory
#containing `conf.py`. This way it finds `plot_directive.py`
sys.path.append(os.path.abspath('.'))

#This must come before plot_directive is loaded by Sphinx
plot_preserve_dir = 'imagery-submodule-directory'

extensions = [
  #...
  'plot_directive',
  #...
]

My modified version of plot_directive.py is available here.

[r-barnes]

I'll leave this open in hopes that one of the RTD folks can weigh in about whether there might be a better way of handling this that I haven't thought of.

If there isn't, I'm happy to have this issue closed.

[stsewd]

@r-barnes thanks for document all the process here, I don't this there is a better way of handling this. Perhaps you can automate the build of the images in travis instead of doing it locally.