Cannot autogenerate API documentation for a project that uses numpy and matplotlib

[breedlun]

I originally posted this on stack overflow, but I have not gotten any replies there, so I thought I would post it here.

I am able to use Sphinx to auto generate my project's API documentation locally, but I run into trouble when I try to use ReadTheDocs. When ReadTheDocs builds the documentation, the API pages are all blank (see this, for example). After I read this blog entry, I realized I should try to mock things like numpy and matplotlib. So I added the following line to my conf.py

autodoc_mock_imports = ['numpy', 'matplotlib', 'matplotlib.pyplot', 'matplotlib.patches', 'matplotlib.path', 'matplotlib.lines', 'matplotlib.text', 'matplotlib.transforms', 'matplotlib.artist', 'cpickle']

With that addition, I cannot build the API documentation locally or on ReadTheDocs, because I set a bunch of matplotlib.rcParam (a dictionary) values, and I check the version of matplotlib using matplotlib.__version__.split("."). For more details see my stack overflow post and my code on github. These issues are likely possible to fix, but they look like a pain from reading the mock documentation.

Perhaps there is a better way. From reading issue #1671 it looks like numpy and matplotlib are now installed on the readthedoc servers, but I have not had any luck building my API documentation when I turn off mocking in conf.py and delete the reference to my "pip requirements file" under my project's Advanced Settings.

Can anyone give me a hand?

P.S. I think I would be able to troubleshoot things more easily if ReadTheDocs let me see the full sphinx output when it is trying to build my documentation. Otherwise, I am left trying to reproduce an issue locally with an unknown cause.

[shoyer]

You want to check the box next to " Give the virtual environment access to the global site-packages dir." in the Advanced Settings for your project on ReadTheDocs. This will give sphinx access to slightly outdated versions of numpy and matplotlib -- you won't need to mock them anymore.

[breedlun]

Thanks for the suggestion, but I already had that box checked. I don't know if others can see my advanced settings, but here is the path.

I also downgraded my local version of matplotlib to 1.3.1 and verified that I can still build the API documentation on my local machine. I then toggled the box next to "Install your project inside a virtualenv using setup.py install" on and off, but it did not help.

Still stuck...

[shoyer]

Okay, so your build is using numpy and matplotlib on read the docs? One other thing to check is the Sphinx version. Try pinning sphinx==1.2.3. The latest released version of Sphinx has a bug that breaks some uses of autodoc.

On Wed, Nov 4, 2015 at 9:55 AM, Benjamin Reedlunn
[email protected] wrote:

Thanks for the suggestion, but I already had that box checked. I don't know if others can see my advanced settings, but [here is the path][1].
I also downgraded my local version of matplotlib to 1.3.1 and verified that I can still build the API documentation on my local machine. I then toggled the box next to "Install your project inside a virtualenv using setup.py install" on and off, but it did not help.
Still stuck...

[1]: https://readthedocs.org/dashboard/clearplot/advanced/

Reply to this email directly or view it on GitHub:
#1803 (comment)

[agjohnson]

I doubt this is an issue with Sphinx 1.3.x. The issue is mocking isn't complete, looking at your SO post. You'll need to fully mock out the dictionary, or supply a dummy object that can mock the version return.

[breedlun]

I actually do not know how to check whether my build is using numpy and matplotlib on the read the docs server. I would like it to, since that would simplify things, but I cannot figure out how to make that work.

I just tried pinning sphinx==1.2.3 and it did use version 1.2.3, but the API documentation did not auto-generate.

Yes, I realize I have not completed the mocking of the dictionary or the version return. At first glance, it just looked like a lot of work, and I was hoping for another way. If there isn't, that's fine. I can figure out how to mock those things, plus any others that turn up.

[shoyer]

To check if you have numpy and matplotlib installed, add some lines to your sphinx conf.py file. Here's what I did for xray: https://github.com/xray/xray/blob/52d38a7796eb1003fd3cc18660063c9c7a4e5938/doc/conf.py#L18-L24

[breedlun]

I just added your suggestions to my conf.py file. As expected, it prints my numpy and matplotlib versions to screen when I build my documentation on my local machine. How do I view the output from the sphinx build process when I build on the ReadTheDocs servers? Wait a second...

Oh man! I just realized that if I click on the build commands it opens up and shows me the output from the command. I wish I had figured that out before. Now I should be able to properly troubleshoot this.

[breedlun]

It turns out my PYTHONPATH was set correctly on my .bashrc file on my local machine, but it was one directory off in conf.py. That's why it built fine on my local machine, but failed on ReadTheDocs.org. It was a silly mistake, but I was not able to troubleshoot it until I realized that I can see the output of the build process by clicking on the proper build command. I really think you guys ought to put a note like, "Click on a build command to see its terminal output" above the list of build commands. Otherwise, you'll have guys like me coming and asking annoying questions.

Anyways, I've gotten my API documentation to build, but I had to do something strange. If I make sure to import matplotlib.pyplot in my conf.py file, everything works. If I do not, I get the following error:

/home/docs/checkouts/readthedocs.org/user_builds/clearplot/checkouts/latest/doc/source/api/clearplot.rst:19: WARNING: autodoc: failed to import module u'clearplot'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/clearplot/envs/latest/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 385, in import_object
    __import__(self.modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/clearplot/checkouts/latest/doc/source/../../clearplot/__init__.py", line 51, in <module>
    import figure
  File   "/home/docs/checkouts/readthedocs.org/user_builds/clearplot/checkouts/latest/doc/source/../../clearplot/figure.py", line 6, in <module>
    import matplotlib.pyplot as _plt
  File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 98, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/__init__.py", line 28, in pylab_setup
    globals(),locals(),[backend_name],0)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_qt4agg.py", line 13, in <module> 
    from backend_qt4 import QtCore, QtGui, FigureManagerQT, FigureCanvasQT,\
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_qt4.py", line 25, in <module>
    from qt4_compat import QtCore, QtGui, _getSaveFileName, __version__
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/qt4_compat.py", line 36, in <module>
    import sip
ImportError: No module named sip

Does anyone know what is going on here? I don't mind having import matplotlib.pyplot in my conf.py file, but it would be nice to understand why it is necessary.

[ericholscher]

Might be worth trying conda, as it makes the scientific python stack much easier to install.

[agjohnson]

This might be a case of mocking out those sub-dependencies as well, or of course, as @ericholscher said, use conda.

Closing this due to inactivity, but feel free to reopen.