Builds fail without explanation

[vfaronov]

Details

• Project URL: https://readthedocs.org/projects/httpolice/ (and others)
• Build URL (if applicable): https://readthedocs.org/projects/httpolice/builds/5704942/ (and others)
• Read the Docs username (if applicable): vfaronov

Expected Result

Builds should succeed, or at least produce an explanation of why they failed.

Actual Result

In the past few days* most of my builds have been failing without explanation. I start a build, it shows up as “Triggered” or “Cloning”, then I refresh the page and it’s already “Failed”. On the build page, all I see is:

This is happening with various versions, various projects. Sometimes I can unbreak it by wiping the version, or by disabling it in Admin → Versions and enabling again. Other times I can’t. Right now the httpolice project is stuck. I have not been able to discern a pattern.

* “In the past few days” may not be relevant information because I hadn’t triggered a build for some time prior to that.

[towicode]

This is happening to me too.

Project URL: https://readthedocs.org/projects/portalmehpc/
Build URL: https://readthedocs.org/projects/portalmehpc/builds/5713163/
Username: Toddwickizer

Here is an image:
http://i.imgur.com/bBHyznB.png

Zero information at all. it builds locally just fine with make html. I am very new to RTD so any help would be great!

[EricWilbanks]

Identical issue here. My last successful build was triggered July 8, 2017 1:15am and my next-most recent changes (today) result in the same Unexpected Error.

Project URL: https://readthedocs.org/projects/fasealign/
Build URL: https://readthedocs.org/projects/fasealign/builds/5713532/ (most recent)
RTD Username: EricWilbanks

EDIT: My builds have since started passing sometime in the interim; may or may not be the same issues as other folks here.

[dhermes]

Same here.

• Project URL: https://readthedocs.org/projects/bezier/
• Build URLs
  • https://readthedocs.org/projects/bezier/builds/5713807/
  • https://readthedocs.org/projects/bezier/builds/5713690/
  • https://readthedocs.org/projects/bezier/builds/5713589/
• RTD Username: dhermes

[willingc]

Hi @agjohnson @ericholscher, I tried to troubleshoot these builds without much success. It looks like @towicode's build logs that it was passing 1 day ago but failing today.

Interestingly, it looks like @vfaronov's project build is passing now.

I wonder if the 24 hour max age in #3005 introduced a weird caching issue for builds.

[vfaronov]

@willingc

Interestingly, it looks like @vfaronov's project build is passing now.

As I mentioned, this issue seems to be intermittent or with non-obvious conditions. I just tried triggering stable and got the same failure: https://readthedocs.org/projects/httpolice/builds/5715813/

I’m not sure my issue is the same as that of others here, as in my builds there’s not even that one line of output about creating a virtualenv.

I wonder if the 24 hour max age in #3005 introduced a weird caching issue for builds.

I first observed this issue 6 days ago, earlier than #3005.

[vfaronov]

Something else I just noticed: some of my failed builds have the (correct) commit hash listed on their page:

while others don’t even have that:

[dhermes]

My two projects that were failing have started succeeding again (e.g. 1, e.g. 2)

[willingc]

Thanks @dhermes. It looks like everyone's docs but @vfaronov's are building now.

@vfaronov Have you tried wiping and cleaning the builds that are failing?

[vfaronov]

@willingc

Have you tried wiping and cleaning the builds that are failing?

Yes, I have.

[willingc]

Recent PRs #3009 and #2979 appear to have resolved this issue.

@vfaronov, your doc build is now reporting passing https://readthedocs.org/projects/httpolice/builds/5732769/ Do you wish to go ahead and close this issue out?

[vfaronov]

@willingc Thank you! My affected projects and versions seem to be working right now. But, if you don’t mind, I’ll wait until tomorrow before closing this issue, just to be sure.

[vfaronov]

@willingc I’m afraid the problem persists: https://readthedocs.org/projects/httpolice/builds/5740724/

[willingc]

@vfaronov This recent failure may be unrelated as I am having some build failures currently. Will you check again in a few hours? Thanks.

[vfaronov]

@willingc Still failing: https://readthedocs.org/projects/httpolice/builds/5742426/

[willingc]

@vfaronov I spent some time looking at your source code. Your docs build environment and process are both a bit different than common sphinx builds on RTD. I spent about 30 minutes trying to work on getting your docs to build myself. I suspect that the reason your docs are not building is due to a regression introduced in either your code or RTD's code. I would recommend getting the docs to build correctly using a conda environment or requirements.txt file (one that is not in tools subdirectory) as well as a readthedocs.yml file in your root directory.

Good luck. The jupyterhub and nbgrader docs along with others from Project Jupyter are good examples.

[vfaronov]

@willingc

Your docs build environment and process are both a bit different than common sphinx builds on RTD.

I did recently add something unusual to the HTTPolice project docs. And I started observing this issue about the same time.

But this issue also affects previous versions of this project — such as 0.5.2 (build #5750046) — which had a straightforward Sphinx configuration that had not been touched in a while, and which did not experience this issue a few months ago. I even disabled “Install your project inside a virtualenv” and it’s still the same.

Do you think deleting the project from Read the Docs and creating it anew might help?

[vfaronov]

Actually, the first manifestation of the issue was before I started changing my Sphinx conf, so these recent changes can’t be the culprit.

[willingc]

I wouldn't delete the project since it is serving your cached docs. I spent about an hour trying to get it to work yesterday.

@ericholscher @agjohnson Is there a way to completely reset a build other than using Wipe in the version settings?

[leotsarev]

Same for me http://readthedocs.org/projects/joinrpg/builds/5756878/

[willingc]

@leotsarev Your issue is different. You should add a requirements.txt file.

[dair]

@willingc I've added requirements.txt; no effect. I can download the project onto my Linux host, pip install -r requirements.txt and after that make html works fine. But still not on RTD unfortunately.

(Same project as @leotsarev's, https://readthedocs.org/projects/joinrpg/builds/5757539/)

[vfaronov]

I was mistaken when I originally said that I had multiple affected projects. Of my projects, only one seems to be affected by this issue. Other two were suffering from an unrelated issue, which had confused me.

[pierre-24]

Our project seems affected since a month, without any explanation.

[willingc]

@vfaronov It looks like your docs are passing now.

@pierre-24 In the RTD admin page for the project, are you building in a virtualenv using setup.py?

[vfaronov]

@willingc

It looks like your docs are passing now.

No, they are failing intermittently, like before.

[pierre-24]

@willingc Yes we are. When I build a branch which was part of a recent pull request (from the end of July), it builds without any issue.

[pierre-24]

So, now that I can check, this project and that one have the same configuration, and are based on the same github repo. The second one is just a fork of the first one. But the build of the first one systematically fails (no matter the version, the branch or the tag), while there is no problem with the second. And I really don't understand why.

[pierre-24]

any news ?

[willingc]

@pierre-24 I'm traveling for the next few weeks. Pinging @ericholscher @agjohnson re: any ideas on the first build???

[leotsarev]

Sorry for bugging, but any news?

[leotsarev]

Guys, what's the timeframe? Should we consider migrating out of RTD? What we can do to help investigating?

[agjohnson]

I've addressed this in #3073. There is a hotfix out, folks should test their builds again

[vfaronov]

Thank you @agjohnson, my project seems to be OK now. @leotsarev @pierre-24 what about yours?

[leotsarev]

Last build passed. I'll play around and report if it is stable.
Anyways, thanks a lot, that's great.