Build failing after installing dependencies via conda

[molpopgen]

Details

• Project URL: http://github.com/molpopgen/fwdpy
• Build URL (if applicable): https://readthedocs.org/projects/fwdpy/builds/5121049/
• Read the Docs username (if applicable): molpopgen

Expected Result

The project is a Cython extension module that depends on C++11 libraries. The relevant dependent libraries are installed via conda (docs/environment.yml) in the repo. In order for Sphinx to work, the module must be built or installed, at which point the docs should be built.

Actual Result

The header file from one of the dependencies installed via conda cannot be found (see build log). I've not taken the time to replicate this behavior via the docker image, but it seems like I'm doing something wrong. The gcc for the docker image is is g++5, which is more than good enough.

The problem seems to be that after installing the conda packages, attempting to install my project is done in an environment where the conda installs do not exist. I've not been able to find explicit docs on how to use dependent C/C++ libraries in a build beyond including them in the environment.yml file.

[humitos]

@molpopgen I'm not 100% sure, but it seems that after creating the whole conda env RTD is trying to install your own project into a virtualenv by doing pip install . at the root of your project. I think this is because the option python.pip_install which you have in True, so as your project depends on some C or C++ libraries to be installed it fails.

I need to download the docker image in my computer (will take some time) to test this better.

I think that is the reason, in fact this commit (molpopgen/fwdpy@468af4d) that has that setting in false was built successfully: https://readthedocs.org/projects/fwdpy/builds/5117047/

Please, check that and confirm.

[molpopgen]

Turning pip_install to false only results in some of the documentation being built. The manual is generated from the .rst files but not from any of the "autodoc" stuff, which requires loading the module, which in turn requires compiling it.

In other words, the build you link to is also a failure, even though it passes all of RTD's checks.

If I were doing this on my machine (not using Docker), I would do the following:

#Build the package in-place 
python setup.py build_ext -i
#Build the html manual
cd docs && make html

My understanding is that I need to install the package in order to be able to load it. However, I cannot do this on the Docker container--there are many issues re: conda environs, permissions, etc., that I cannot sort out.

[molpopgen]

Any thoughts on this? I'm struggling to find an example of a project hosted on RTD that:

1. installs C or C++ dependencies through conda
2. is itself a Python in C or C++ extension package requiring compilation

I am aware of one packages satisfying number 2. In speaking with the dev, he gave up on having RTD build the package and instead provides mock documentation.

Is there a test package that you have used to illustrate that this is indeed possible?

[astrojuanlu]

@molpopgen I noticed that you do not have continuous integration for your project. I have encountered similar problems to this one in the past and my advice is that you try to setup up Travis CI, so you can decouple installation problems from documentation problems.

For instance, I tried to install your library locally and failed:

(test36) juanlu@nebulae /tmp/fwdpy-master :( $ conda install -c bioconda libsequence -yq
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/juanlu/.miniconda36/envs/test36:

The following NEW packages will be INSTALLED:

    curl:        7.52.1-0               
    htslib:      1.3.2-0        bioconda
    libgcc:      5.2.0-0                
    libsequence: 1.9.0-0        bioconda
    tbb:         4.4_20150728-0 bioconda

Proceed ([y]/n)? 
(test36) juanlu@nebulae /tmp/fwdpy-master $ pip install .
Processing /tmp/fwdpy-master
    Complete output from command python setup.py egg_info:
    check_deps/test_libseq.cc:1:32: fatal error: Sequence/SimData.hpp: No such file or directory
     #include <Sequence/SimData.hpp>
                                    ^
    compilation terminated.
    make: *** [<builtin>: check_deps/test_libseq.o] Error 1
    libsequence version 1.9.0  found.
    GSL version  2.3  found.
    Attempting to compile and link a test program using fwdpy's dependencies
    2
    Could not compile and link test programs!
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-k20ckd2g-build/

This suggests there is some problem with the libsequence package. This is not the place to discuss it, but just a general advice and a signal that this is not a RTD problem :)

[molpopgen]

@Juanlu001

Not sure where you're getting this, but Travis CI is indeed set up for fwdpy, and works. See the .travis file: https://github.com/molpopgen/fwdpy/blob/master/.travis.yml

The installations are known to work, both from source and from bioconda.

There is a dev branch with an even more streamlined build setup via conda, too, that gets rid of the need for some setup.py flags.

[astrojuanlu]

Not sure where you're getting this, but Travis CI is indeed set up for fwdpy, and works. See the .travis file: https://github.com/molpopgen/fwdpy/blob/master/.travis.yml

My fault, I assume everybody is using these colourful shields :)

The installations are known to work, both from source and from bioconda.

Let's go on the conversation elsewhere: molpopgen/fwdpy#6

[jakirkham]

What if you drop the Python version specification from your readthedocs.yml file, @molpopgen?

[molpopgen]

Is there a reason why that would that matter? The problem is with conda. You install a dependency, try to compile an extension module using it, and it's headers can't be found. This cannot be mocked out, as no module exists for making docs until the build succeeds. It seems like this should be independent of the python version. I've requested a link to a working example, but none has been provided.

On Mon, Apr 3, 2017, 4:29 PM jakirkham ***@***.***> wrote: What if you drop the Python version specification from your readthedocs.yml file, @molpopgen <https://github.com/molpopgen>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2707 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGHnH7z6dudK1grrDnSNPBZZFkKgXcSEks5rsYAfgaJpZM4MXDZ9> .

-- Kevin Thornton Associate Professor Ecology and Evolutionary Biology UC Irvine http://www.molpopgen.org http://github.com/ThorntonLab http://guthub.com/molpopgen

[molpopgen]

I tried it on another similar project, and again I get failures:

https://readthedocs.org/projects/fwdpy11/builds/5243308/

The main code is here:

https://github.com/molpopgen/fwdpy11/tree/rtd

The failure is not being able to find a GSL header installed by conda.

I am reasonably certain that the conda support is broken as far as things that need to be compiled against external dependencies are concerned. It does not do what conda is meant to, which is get rid of the need for dealing with paths, etc.

Again, another request for one of the devs to provide a working example...

[jakirkham]

Is there a reason why that would that matter?

Perhaps not. Just thought it was something that could be easily handled by the environment.yml file and didn't know if this was confounding the problem.

...it's headers can't be found.

conda structures things in a pretty simply. In Python, sys.prefix for a stock python package in conda will point to the environment prefix. From there one can search for includes in sys.prefix + "/include" and libraries in sys.prefix + "/lib". Obviously one would use os.path.join to combine the paths, just trying to keep things simple. These can then be added in with whatever other search paths one checks anyways. I suppose RTD could handle adding these to the search path environment variables for us, but it is not too difficult to do ourselves.

[molpopgen]

conda structures things in a pretty simply. In Python, sys.prefix for a stock python package in conda will point to the environment prefix. From there one can search for includes in sys.prefix + "/include" and libraries in sys.prefix + "/lib". Obviously one would use os.path.join to combine the paths, just trying to keep things simple. These can then be added in with whatever other search paths one checks anyways. I suppose RTD could handle adding these to the search path environment variables for us, but it is not too difficult to do ourselves.

Is there an example you can point to for any project hosted on RTD that does this? Your example refers to a Python package, but what about my case, which is a C library (GSL in this case)?

Again, to be clear: in order for docs to be build, the extension module must be compiled. Compiling it requires external C/C++ libraries in addition to Python packages. We are not talking about using headers installed by other Python packages.

The usual use case here would be to call gsl-config --cflags to get the include directories, but it is not clear to me that the conda bin directory is even available to the user.

[molpopgen]

I suppose RTD could handle adding these to the search path environment variables for us, but it is not too difficult to do ourselves.

Isn't this precisely why conda is used in the first place? The fact that this is not working would seem to be strong evidence that the way conda is used by RTD is somehow flawed--perhaps installed somewhere funny and then not documented?

[jakirkham]

Is there an example you can point to for any project hosted on RTD that does this?

Yes, sorry, should have included this in that comment. This isn't merged yet, but PR ( pyFFTW/pyFFTW#161 ) works with Cython and a C library (FFTW). Just recently composed it, so should be able to answer questions. We handled the paths in PR ( pyFFTW/pyFFTW#162 ).

Isn't this precisely why conda is used in the first place?

The activate script in conda doesn't actually change the include or library paths. So I don't think this is a bug with RTD given conda is not doing it either. In fact, I would not expect conda to muck with these environment variables normally. It would open the door to some unintended consequences that could cause more pain for those unaware of them.

[molpopgen]

OK, so that seems to fix the compilation part.

I still need to deal with the next failure, which has to do with running doxygen to build a manual for the C++ API.

I'm getting the following error:

doxygen: error while loading shared libraries: libiconv.so.2: cannot open shared object file: No such file or directory

Presumably, the runtime path is not set correctly. I can try skipping this, for now.

[molpopgen]

@jakirkham

OK, so it seems like many problems have been resolved. However, after installing the module, the actual building of the manual fails to find it.

There are several modules, and an exception is raised for each (see below).

Does this mean I somehow need to get PYTHONPATH adjusted?

/home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/checkouts/latest/doc/index.rst:35: WARNING: autodoc: failed to import module 'fwdpy11.fwdpy11_types'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/conda/latest/lib/python3.5/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/checkouts/latest/fwdpy11/__init__.py", line 20, in <module>
    from .fwdpp_types import *
ImportError: No module named 'fwdpy11.fwdpp_types'

[jakirkham]

We ran into something similar. Not sure if it is the same thing you are encountering or not.

It turns out when RTD installs the package it doesn't do a development mode install. Instead it just installs the package in site-packages. So it can get confused and try to import things from the source directory, but maybe they only exist in site-packages. The error we get back ends up being a bit cryptic. I think it may be worthwhile for RTD to change to development mode installs to avoid this sort of confusion.

To fix it, we had to comment one line in the Sphinx conf.py file that was messing with the path so it would import the package from site-packages instead. Though I suppose one could add a guard and check the READTHEDOCS environment variable. Depends on one's preferences really.

[molpopgen]

@jakirkham

I made the following change to my conf.py, and am still getting import errors:

if (os.environ.get('READTHEDOCS')==True) is False:
    sys.path.insert(0, os.path.abspath('..'))
else:
    import site
    p=site.getsitepackages()[0]
    sys.path.insert(0,p)

I even tried doing nothing if READTHEDOCS is True, and get the same problem.

The specific error comes when importing the main module. The init.py tries to import some of the compiled modules and cannot find them.

[jakirkham]

All environment variables are strings. So instead of True it should be "True".

[pllim]

Got the error "site has no attribute getsitepackages" when using virtualenv on RTD, possible workaround at kivy/python-for-android#610 (comment)

[humitos]

@molpopgen some things regarding the build process has changed since this issue was reported and I see the last build was like a year ago. Can you try again to re-build your docs and let us know if the problem is still there?

[molpopgen]

@humitos, I no longer update this project, but I do work on a similar one replacing it. RTD worked fine for that project for a while, but is suddenly plagued by problems again. These issues came on w/o warning, and then suddenly seem to have gone away.

My current project is this. I won't know if the build successes are luck or not until I make further changes.

[humitos]

@molpopgen thanks for your reply.

I think that no one else is having this issue and you are not working on this project anymore so I will close this issue here --but feel free to reopen it if anyone else consider approapiate.

Also, please keep us updated regarding your current project and the builds by openning new issues. I'd appreciate that :)

[molpopgen]

Also, please keep us updated regarding your current project and the builds by openning new issues. I'd appreciate that :)

Here is an example of a failed build: https://readthedocs.org/projects/fwdpy11/builds/6928074/

Nothing has changed about the RTD setup, and the only difference from the successful build of the same branch 3 weeks ago is a minor PR which passes on Travis (as well as on several of my own systems): https://travis-ci.org/molpopgen/fwdpy11/builds/356684112

No idea why pthread is failing.

[stsewd]

I won't know if the build successes are luck or not until I make further changes.

Maybe some memory limit?

I will test this project on my local instance tomorrow.

[molpopgen]

Maybe some memory limit?

If so, then there's no indication of that in the latest build failure: https://readthedocs.org/projects/fwdpy11/builds/6928074/.

If you do test the project, the dev branch is the relevant one.

[stsewd]

If so, then there's no indication of that in the latest build failure

Yeah, there are several projects with the same problem, the current implementation can't detect all the memory limit errors #3613. I see you mention doxygen on some previous comment, maybe #3580 could help?

[molpopgen]

Yeah, there are several projects with the same problem, the current implementation can't detect all the memory limit errors #3613. I see you mention doxygen on some previous comment, maybe #3580 could help?

I no longer use doxygen. The frustration here is that the failures are random, suggesting that limits are hit on some build nodes but not others maybe?

For this latest build, there's even less code to compile than the last successful build, and the point of failure seems to have been on one of the lighter files, too.

Is it possible to actually see detailed build logs?

[humitos]

If so, then there's no indication of that in the latest build failure: readthedocs.org/projects/fwdpy11/builds/6928074.

I see this error in the log,

    gcc -pthread -B /home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/conda/dev/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/conda/dev/include/python3.5m -c /tmp/tmpnpq5stvc.cpp -o tmp/tmpnpq5stvc.o -std=c++14
    gcc: error: unrecognized command line option ‘-std=c++14’

Could it be that the reason of the failure? So, maybe the version of gcc that RTD has on its Docker image is old? (we are using Ubuntu 16.04 for the builds).

https://stackoverflow.com/questions/31965413/compile-c14-code-with-g/31965806#31965806

docs@9c125177efbd:~$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[stsewd]

@humitos of what I understand the builds sometimes success and others not. And I was trying to build the docs on my local instance and got a memory limit error.

[molpopgen]

I see this error in the log,

No. That is an intentional part of the build system. There are tests performed to determine if C++14 is available. The test is based on a method provided by pybind11, upon which this project is based. If this error were the show-stopper, the build log would have ended there.

This repo is known to build correctly in conda systems, both on Travis and in the real word. The C++14 detection also works on non-Travis systems that have up-to-date compilers. Any build failures are specific to RTD, and seemingly a bit random...

[molpopgen]

And I was trying to build the docs on my local instance and got a memory limit error.

@stsewd Did it die at the same point as my last RTD build?

[molpopgen]

The failures seen in my latest build attempt of the dev branch end with this as the error:

    /home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/conda/dev/compiler_compat/ld: cannot find -lm
    /home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/conda/dev/compiler_compat/ld: cannot find -lpthread
    /home/docs/checkouts/readthedocs.org/user_builds/fwdpy11/conda/dev/compiler_compat/ld: cannot find -lc

Is that compatible with a memory limit error? It really seems like an environment issue, and one that I cannot control. I'm installing the dependencies via conda as I always do, using methods which work for Travis, bioconda, etc.

[stsewd]

@molpopgen I tested your dev branch with memory limit increase, it fails on the same step now :/

    collect2: error: ld returned 1 exit status
    error: command 'g++' failed with exit status 1

[molpopgen]

@stsewd Thanks. I strongly believe that there's something going amiss with setting up the RTD environments. I just manually triggered a new build of the dev branch,which has had no new commits to it. The build passed: https://readthedocs.org/projects/fwdpy11/builds/6932137/

[molpopgen]

@stsewd -- my mistake! I triggered a build of "latest", not "dev" yesterday.

[molpopgen]

@stsewd I managed to get the dev branch working. The trick was to remove gcc from the list of conda dependencies. This removal results in using whatever built-in compiler happens to be there.

Not using the conda gcc may or may not be ok in general. I'm unable to figure out what packages the RTD build environment is using, but if it is GCC >= 5, then there's the risk of link-failure when compiling a project on the build system and linking it against a C++ dependency installed via conda.

I'm going to attempt to install both gcc and libgcc via conda and see what happens...

[molpopgen]

@stsewd Installing gcc + libgcc from conda also fails.

It should probably be documented NOT to use the conda gcc.

[humitos]

I'm unable to figure out what packages the RTD build environment is using, but if it is GCC >= 5

It's 5.4.0. I pasted in my previous comment: #2707 (comment)

If you want to check any version of any package used to build the docs, you can pull the docker image used to build the docs:

docker pull readthedocs/build:2.0

Then run bash on this image:

docker run --rm -it readthedocs/build:2.0 bash

and use apt or whatever you want check the package/library versions.

It should probably be documented NOT to use the conda gcc.

Agreed. I'd say that we need more information (other projects with the same problem, understand exactly what's happening --I'm a little lost, etc), and mention these things possible problems in that note.

Let me know if that helps.

[molpopgen]

@humitos -- Thanks.

So, having GCC 5.4 in the env is a feature or a bug, depending on the goals. The feature is that you can use all of c++11 and most/all of C++14. That's great.

The bug side is that RTD intends to support building docs for packages needing compilation, and that dependencies can be pulled in via conda. The current conda compiler is GCC 4.8.5. That gives you C++11, but not C++14. (That works for me, at least for now, but it is unlikely to work for everyone.) The reason why this matters is that you cannot link C++ code compiled with GCC5 to a C++ library compiled with GCC4, due to an ABI incompatibility introduced to accommodate the new C++11 standard. (Note that it is possible to have a dual-ABI GCC5, though.)

[molpopgen]

@humitos -- Related to this docker stuff, is there a specific set of commands one should use to attempt to build a package using the exact method that RTD uses?

A different project of mine is failing to build, and I cannot reproduce the error on any of my machines, nor can I find a "for dummies recipe" for building via Docker. I see there's a tasks.py referred to here, but I'm not seeing any clues on how to use it?

[molpopgen]

@humitos I managed to build the dev branch of my other project in the Docker image. Even though I probably didn't use the exact same commands, I reproduced the build failure there and pushed a fix for it that seems to work ok in general.

However, after the fix, there are still missing symbols errors when attempting to build the manual. These are in a C++ dependency installed via conda, which happens to be another of my projects. This dependency gets installed via the bioconda channel, where it is build with GCC 4.8.5.

This error is exactly what I am referring to above, and are classic symptoms of ABI issues:

1. RTD's docker builds pylibseq with GCC 5.4
2. pylibseq pulls in a dependency compiled with GCC 4.8.5
3. The ABI incompatibility is not seen at link-time, but rather at run time, which here is during "make html" of my documentation.
4. All of this works on Travis setups where everything is done with the conda GCC.

apt-cache search shows that GCC 4.8 and 4.9 are available to this Docker image. It would be really valuable to make one or both of those available, and perhaps have DOCKERGCC/DOCKERCXX environment variables be settable to the desired C and C++ compilers, respectively.

[humitos]

@molpopgen nice that you found the problem and thanks for explained it to us.

If what you are saying here is the solution:

apt-cache search shows that GCC 4.8 and 4.9 are available to this Docker image. It would be really valuable to make one or both of those available, and perhaps have DOCKERGCC/DOCKERCXX environment variables be settable to the desired C and C++ compilers, respectively.

I think it would be great to open a specific issue to track this solution and I also invite you to propose a PR in the https://github.com/rtfd/readthedocs-docker-images repository, but also, we have to think "how the user will change that value from his documentation code" and make a PR for this repository with that feature.

[molpopgen]

@humitos I'll see what I can do. I have zero experience with Docker/containers in general, though, so it may be rough going.

[humitos]

I'm not sure how is the correct way to solve this, but having a PR that "more or less" draws the idea is a good starting point. We will probably need to make a design decision with the rest of the core team also.

I think the most complicated point is how the user decide what version to use and how RTD source code communicates that decision to the docker container.

[molpopgen]

Do you mean an issue ticket rather than a PR? I literally wouldn't know where to start a PR, given that I know zero about the RTD code repo, Docker, etc. I can, however, raise this as an issue.