Provide option to turn off single page build (readthedocssinglehtmllocalmedia)

[edmorley]

Hi!

I noticed that our build includes a readthedocssinglehtmllocalmedia step, which is something we don't need. We've already turned off the PDF/epub builds, but there doesn't appear to be an option to disable this one?

On a related note, I don't know that we really need the json build either? (Unless it's used to power search that is)

Many thanks,

Ed

[stsewd]

I think the json build is to power the search. The singlehtmlocalmedia is to downloading the docs for reading offline, so I think this feature shouldn't be removed.

[edmorley]

The download docs feature is not something we need, so would be fine with disabling that if disabling this build type on a project.

[stsewd]

But probably your users need it, or your project is very specific to work only online? Or this is for internal usage, perhaps?

[edmorley]

To me, the download docs feature is very niche, and not something I have ever used once on any read the docs project (both my own and those of packages I use). In particular, for certain projects it doesn't make sense. eg to use our project's "get a development environment set up" or "how to run data ingestion locally" guides, you need an internet connection for Virtualbox/Vagrant download, VM provision, pip install, and then actually using the app itself.

[humitos]

How does this affect you? I mean, is there any particular reason for what you want to remove/disable this feature?

I think we need to understand a bit more the situation, otherwise it will probably be marked as wontfix.

[edmorley]

Hmm trying to remember back I think it was because a few months ago we were seeing quite a few RTD automated build failures, some of which were due to concurrent builds racing with each other (eg #2901), so it seemed sensible to try and cut out as many unneeded parts of the build as possible, to reduce chance of overlap. (Plus as an added bonus, to consume fewer RTD resources)

However I understand if you'd prefer not to add another option and would rather wontfix :-)

[humitos]

Thanks for your update.

I'd say that is something that it's "a service" from Read the Docs and we shouldn't provide an option to disable it (if we think about the usage of resources).

On the other hand, if we found that this could be a problem as you mention (linked to another issue) we should take a look and see how this affects that.

[ericholscher]

I'm +1 on making this optional. It should be allowed to be set in the formats setting?

[agjohnson]

This seems like something we could make optional, I don't see a problem with this.

The JSON build has to stick around for now though, as we aren't going to special case our search usage. Eventually this will be a single build pass, but requires us implementing an HTML + JSON single step Sphinx builder.

[giovannipizzi]

I would add my +1 for having the readthedocssinglehtmllocalmedia be optional: for us this step is an issue, because a few days ago we updated a number of dependencies, and now while the HTML and JSON steps complete successfully, the readthedocssinglehtmllocalmedia fails (typically with "Command killed due to excessive memory consumption", but sometime - randomly - with a timeout).
See e.g. http://readthedocs.org/projects/aiida-core/builds/7152586/

We're still not sure why this is happening, which were the packages that increased significantly the memory footprint and why there is so much memory usage, but for now we would like to just remove that step if it is not needed, as now our documentation is not compiling anymore... Thanks!

[humitos]

@stsewd I think this should be added through a YAML setting. Eric mentioned the formats also, which I think it's the right place. What do you think?

[stsewd]

@humitos yeah, I was thinking the same here :)

[giovannipizzi]

Dear administrators, while this issue is being resolved, could it be possible to increase the memory limit for our project (aiidateam/aiida_core, https://aiida-core.readthedocs.io/en/latest/ on RTD)?

We did quite a bit of investigation on the memory usage of our code, see e.g. aiidateam/aiida-core#1472 and we also believe that the autoapi might be the reason, due to the large number of dependencies we have, as discussed also e.g. here:
#2715 (comment)

But at this point it's not obvious to us which one of the dependencies is really the problem...

If you could increase the memory for now, it would be great - as soon as it is possible to remove the singlehtml step, we will disable it and the problem should go away. Thanks in advance!

[humitos]

@giovannipizzi Hi! We are working to make this option available through the YAML file configuration. The code is almost ready to support it but it's still not completed. I suppose we will have news about this in the following weeks. I'm sorry for the time that this is taking but I'm happy we are in the right direction now.

[stsewd]

@giovannipizzi and everyone, good news! Turns out that this feature is already implemented. You can add this to your readthedocs.yml file

formats: []

[stsewd]

Now we have this documented https://docs.readthedocs.io/en/latest/guides/build-using-too-many-resources.html

[giovannipizzi]

@stsewd This is awesome, thanks a lot!
I already added this to our code - it might have worked, even if it's hard at this stage to say if it really did, as there were three builds triggered at the same time and

• 1 failed due to timeout
• 1 passed (but build the single html)
• 1 passed (but did not build the single html as we wanted)

I'll report back if the problem is not solved, otherwise thanks for fixing this!

[stsewd]

You can try wiping the environments to have a clean build (Versions section, edit button)