Recent build failures for Godot Engine docs ("godot" project)

[akien-mga]

Details

• Project URL: https://readthedocs.org/projects/godot
• Build URL (if applicable):
  • Timeout: https://readthedocs.org/projects/godot/builds/6631323/
  • ImportError: no module named debug: https://readthedocs.org/projects/godot/builds/6631922/
  • IOError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/godot/envs/latest/local/lib/python2.7/site-packages/readthedocs_ext/_templates/readthedocs-insert.html.tmpl': https://readthedocs.org/projects/godot/builds/6631962/
• Read the Docs username (if applicable): Akien

Expected Result

Godot documentation should build successfully.

Actual Result

Godot has a pretty heavy documentation (GitHub repo) due to a very high number of pages (600-700 pages I think, including ~500 pages of API reference), many of those including images and hyperlinks to each other (especially in the API reference).

This seems to cause various issues, which are a bit hard to reproduce (at least each build seems to give me different errors today..):

• We already had random build issues due to flirting too much with the set timeout of 1800 s (sphinx-build takes ages), as shown in this build log. Lately it happens a lot more.
• Yesterday, I ran into several out of memory issues making the build abort. Strangely enough, today those error messages are no longer visible in the logs of those builds, which only report a build timeout...
• Today, I'm getting two new issues (so far):
  • ImportError: no module named debug: https://readthedocs.org/projects/godot/builds/6631922/
  • IOError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/godot/envs/latest/local/lib/python2.7/site-packages/readthedocs_ext/_templates/readthedocs-insert.html.tmpl': https://readthedocs.org/projects/godot/builds/6631962/

So I'm a bit at a loss here, any help welcome :)
I'd gladly split the various issues mentioned above in separate tickets if relevant; as I had a hard time reproducing them I went for a bulk issue to start going further in debugging with your help.

[humitos]

@akien-mga I think all this problems are because OOM (which produces strange things) and/or build timeout.

I just imported your project in my local instance and I ran the build with 1 hour as building time and 2gb of memory. stable and latest built without any problem but it took like 30 minutes :/

Besides, this build passed on RTD platform: https://readthedocs.org/projects/godot/builds/6634320/ but it took 25 minutes...

[akien-mga]

Besides, this build passed on RTD platform: https://readthedocs.org/projects/godot/builds/6634320/ but it took 25 minutes...

Yeah we have a lot of documentation and lots of hyperlinks to the class reference, so builds take a while. I'm not a Sphinx expert so I'd welcome feedback on how to improve build times as ~30 min is a lot.

If possible, I would also request increase the build timeout for the "godot" project, since as mentioned above, our build times are around 25-35 minutes, 30 min being the current timeout (1800 s), so they can randomly fail: https://readthedocs.org/projects/godot/builds/

[akien-mga]

If possible, I would also request increase the build timeout for the "godot" project, since as mentioned above, our build times are around 25-35 minutes, 30 min being the current timeout (1800 s), so they can randomly fail: https://readthedocs.org/projects/godot/builds/

Additionally (as just mentioned on IRC), would it possible to increase the timeout of those new l10n projects to the same value? "godot-fr", "godot-es" and "godot-zh-cn". More will be coming but I'm starting with those three to define and test the translation workflow.

[davidfischer]

I increased your build timeout to 4800s (1.5hrs) so hopefully that won't happen again.

I'm syncing the docs now to try to figure out why they take so long and while I don't think I need to do anything on the translations, I'll double check.

Edit: I'm bad at math... I'm changing it to 5400

[akien-mga]

Thanks a lot!

[akien-mga]

while I don't think I need to do anything on the translations, I'll double check.

Note that the translations currently build fast because I removed 98% of the documentation while testing. I'll soon be adding back all the normal Godot docs to the translation repo, and then builds will take as long as in the main project.

[davidfischer]

Ahhh, that makes sense. I can go ahead and increase their time limit as well.

[davidfischer]

In terms of the build, a local build took 20 full minutes on my local CPU (core i7 2.3ghz). CPU was maxed 99%+ on a single core for almost all of the time (Yay, GIL). Memory usage was less than 200MB. All but a small percentage of the time was spent on the writing output... step.

This tells me a few things:

• You have a lot of docs - over 700 .rst files
• Memory does not appear to be a limiting factor, CPU does

I think the immediate problem is done here so I'm going to close this. I created a separate issue #3870 to track improvements to build performance.

Feel free to re-open this if you see any other issues.

[akien-mga]

@davidfischer Could I ask you to increase the timeout to 5400 for the following new localization projects?

godot-de
godot-fr     (already done in the past but I since deleted and reimported it to fix GitHub hooks)
godot-pt-br
godot-uk

Thanks in advance.

I might have another batch in the future, but for now I only created projects for locales with a good level of completion: https://hosted.weblate.org/projects/godot-engine/godot-docs/

[davidfischer]

@akien-mga These are done!

[akien-mga]

Thanks!

[akien-mga]

@davidfischer Could I bother you again with another batch of Godot projects in need of a timeout bump to 5400?

I recently created:

godot-ko
godot-pl
godot-rus

Thanks in advance.

[davidfischer]

No problem. It's done.

[akien-mga]

@davidfischer Could you bump timeout again to 5400 for godot-ru?

I recreated it today (deletion + new project) as the GitHub webhook was invalid after the godot-rus->godot-ru renaming in #4202, and I couldn't find an easy way to edit it - but I forgot that it would mean the timeout needs to be increased anew.

[davidfischer]

Could you bump timeout again to 5400 for godot-ru?

Done!