Builds use a smaller builder when moving from pip to conda

[chrisburr]

Details

• Read the Docs project URL:
  • New: https://readthedocs.org/projects/cburrdirac/
  • Upstream: https://readthedocs.org/projects/dirac/
• Build URL (if applicable):
  • New: https://readthedocs.org/projects/cburrdirac/builds/13209839/
  • Upstream: https://readthedocs.org/projects/dirac/builds/13209708/
• Read the Docs username (if applicable): chrisburr

We decided to try enabling the conda for the build environment for our docs so we could avoid mocking dependencies and remove the duplication between the tests and docs setup. I developed this by making a new project on readthedocs to build documentation using my fork and then opened a PR to the main repository. Surprisingly the builds now always fail with memory errors or timeouts.

It seems like the small build nodes are being used instead of the larger conda compatible ones. I've done some digging and it seems like #7912/#7933 added a feature to "lock" external builds to the queue used for the latest build:

readthedocs.org/readthedocs/builds/tasks.py

Lines 85 to 104 in 7450d6d

	# Use last queue used by the default version for external versions
	# We always want the same queue as the previous default version,
	# so that users will have the same outcome for PR's as normal builds.
	if version.type == EXTERNAL:
	last_build_for_default_version = (
	project.builds
	.filter(version__slug=project.get_default_version(), builder__isnull=False)
	.order_by('-date')
	.first()
	)
	if last_build_for_default_version:
	if 'default' in last_build_for_default_version.builder:
	routing_queue = self.BUILD_DEFAULT_QUEUE
	else:
	routing_queue = self.BUILD_LARGE_QUEUE
	log.info(
	'Routing task because is a external version. project=%s queue=%s',
	project.slug, routing_queue,
	)
	return routing_queue

I'm not sure how easy this is to fix. If it's difficult perhaps the easiest solution is to document this quirk in the conda troubleshooting section. I'm happy to make a PR for the docs if that's the way you want to go.

Expected Result

Builds in a copied repository work the same as in the original one.

Actual Result

Moving to the upstream repository caused builds to fail.

[stsewd]

Hi, looks like both builds are using the same build (big builders). Looks like this is due to an error on the build rather than a memory issue.

grep: _build/html/CodeDocumentation: No such file or directory

[chrisburr]

@stsewd Are you looking at https://readthedocs.org/projects/dirac/builds/13216630/? If so that was intentionally a test to bypass the logic I linked above and shorten the the amount of time the build would take.

I'm fairly sure the build I linked in the original post (https://readthedocs.org/projects/dirac/builds/13209708/) must have ran on a small builder as it failed with "Command killed due to excessive memory consumption".

[stsewd]

Hmm, yeah, that one run on a smaller builder. Not sure how I ended up in the other build.

If it's difficult perhaps the easiest solution is to document this quirk in the conda troubleshooting section. I'm happy to make a PR for the docs if that's the way you want to go.

the docs already mention to contact us to increase resources in case of problems.

So, the problem is with our task router, yeah. We could move the conda check before doing the external version check. /cc @humitos.

[humitos]

This is an edge case and I'm not sure how to solve it. We use historical builds data to know which would be the best queue to build the current build. At this point we don't have the build.config attribute, so we rely on previous build to decide which one to use. At that point, previous builds were using build:default queue but the current build is using conda so it should use build:large, but I don't know how we would know that.

I think we should mark this project to use build:large queue and close this issue.

NOTE that after a couple of failed builds for a regular Version it will start using build:large again for regular Versions but also for Pull Request since we will have some data for build.config:

readthedocs.org/readthedocs/builds/tasks.py

Line 111 in 7450d6d

if build.config.get('conda', None):