Git submodules intermittently failing to clone resulting in failed builds

[tannerjfco]

Details

• Project URL: https://github.com/drud/ddev
• Build URL (if applicable): Failed Build: https://readthedocs.org/projects/ddev/builds/5535145/
• Read the Docs username (if applicable): ddev

Expected Result

The project consistently clones and builds successfully.

Actual Result

The project intermittently fails to build. The error on these failed builds is the following:

Failed to import project: Failed to get code from 'https://github.com/drud/ddev' (git clone): 1 git clone error output: Cloning into 'containers/docker.nginx-php-fpm'... Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Clone of '[email protected]:drud/docker.nginx-php-fpm.git' into submodule path 'containers/docker.nginx-php-fpm' failed

The submodule repositories for this project are all public, so there is no reason why a permission denied error should occur. If i manually rebuild the project in the UI, it will usually end up succeeding after 1-2 attempts. There does not seem to be any discernible pattern to the failures.

[jeras]

I have a similar issue but in my case the submodule is private and no attempt should be made to clone it.
In most cases the build succeeds, the failure rate is low.
What I am concerned with is future updates. If the behavior is made consistent, builds might fail every time.
So I would like to be able to specify which submodules should be cloned.

[tannerjfco]

This is still occurring for us on this project, pretty much all the time. Every merge we make into master is almost guaranteed to result in a failed readthedocs build that we go in and manually rebuild in hopes of eventually succeeding. It would be helpful to get a response on this issue. Thanks

[barseghyanartur]

@tannerjfco:

Same for me. If you happen to solve the issue, please, let me know how.

[stsewd]

I think this issue is already solved, I was able to build your docs successfully. And your repository url must be https://github.com/drud/ddev.git, note the .git.

[barseghyanartur]

@stsewd:

Yep. Thanks. I gave it another try yesterday and it worked.

[RichardLitt]

Great!

[RichardLitt]

Trigger happy. @tannerjfco - can you confirm this works?

[barseghyanartur]

@RichardLitt:

I had to set the Programming Language (in Settings) to Only Words (was Python) and then it worked. Without the change it keeps failing. Other repositories that I have (that do not have sub-modules) do work fine with Programming Language set to Python. It looks like the combination of the two causes problems.

[tannerjfco]

@RichardLitt unfortunately this does not appear to be resolved. While the error message has changed, it looks to be the same problem, with the same behavior. I can attempt rebuilds and eventually get a success, but the issue will turn up again later.

Updated error message:
"There was a problem connecting to your repository, ensure that your repository URL is correct and your repository is public. Private repositories are not supported."

I have tested the repo URL both with '.git' at the end and without, and can't discern any difference between the two. The repo being cloned it public.

[sylvainbeo]

Does not seems to be resolved for me neither. I've tried @barseghyanartur tip (changing Programming Language from python to Only words), bu with no success. I alos got a public repository.

Error is :

There was a problem connecting to your repository, ensure that your repository URL is correct and your repository is public. Private repositories are not supported.

When i try to build manually, it works about 1 time out of 3 or 4.

[barseghyanartur]

@sylvainbeo:

How I finally solved it (what works now) - I have a mirror repository on BitBucket (without git sub-module). The docs are built from there with no problems.

For BitBucket I use Mercurial. Thus, I commit twice (to both GitHub and BitBucket). I'm quite please with the way it works now.

[sylvainbeo]

@barseghyanartur Thanks for the feedback, we will try this.

[humitos]

The original problem seems to be solved. I'm closing this one in favor of #4043 that mention a possible error with the submodule URLs and relates to other issues on how to select the submodules to check out.

We have a temporal flag now that we can use to skip all the submodule checkout. In case anyone want to enable that, just ping me here and I will do it.

I'm closing this one.

The future general solution is readthedocs/readthedocs-build#47