Build time out when hitting the api to get active versions

[ChoiByungWook]

Details:

• Project URL: https://readthedocs.org/projects/sagemaker/
• Build URL(if applicable): https://readthedocs.org/projects/sagemaker/builds/9817944/
• Read the Docs username(if applicable): AnonymousUser

Expected Result

Build succeed

Actual Result

Based on the build history, the builds have been failing and passing inconsistently.

The following things are inconsistent when looking at our build history:

1. Build time (succeeds and failures ranging from 70s - 220s)
2. Build log output (log outputs for failed builds are inconsistent)
3. Error indication (A lot of the failed builds don't display any obvious error, as listed below.)

Can we get some insight into why the builds failed, specifically for these:

• https://readthedocs.org/api/v2/build/9813741.txt
• https://readthedocs.org/api/v2/build/9813710.txt
• https://readthedocs.org/api/v2/build/9813681.txt
• https://readthedocs.org/api/v2/build/9813550.txt
• https://readthedocs.org/api/v2/build/9813542.txt
• https://readthedocs.org/api/v2/build/9813533.txt
• https://readthedocs.org/api/v2/build/9807045.txt

The peculiar thing is that this next build succeeded but it was a one line change in the code, with not much changes in the RST or documentation.

• https://readthedocs.org/projects/sagemaker/builds/9817945/

Corresponding changes for the build above:

• aws/sagemaker-python-sdk@9c07f06
• https://github.com/aws/sagemaker-python-sdk/pull/1087/files

[laurenyu]

any update? we're continuing to see a lot of transient failures, especially when multiple builds in the same project are run simultaneously

[stsewd]

It may be related to time outs, I'll increase the build time of the projects on monday

[laurenyu]

thanks for your help!

[laurenyu]

@stsewd were you able to increase the build time yesterday? the failures have become more consistent now, but the error message simply says "There was a problem with Read the Docs while building your documentation. Please try again later. However, if this problem persists, please report this to us with your build id"

[stsewd]

Sorry, I wasn't able to get to this yesterday, I'm checking now.

[stsewd]

Looking at this doesn't look like it's from timeout. Not sure what's causing it. I'm investigating a little more

[stsewd]

I wasn't able to reproduce this locally, but I found this in the logs

readthedocs.doc_builder.environments:619[24217]: ERROR (Build) [sagemaker:stable] HTTPSConnectionPool(host='10.32.16.11', port=10443): Max retries exceeded with url: /api/v2/project/164476/active_versions/ (Caused by ResponseError('too many 502 error responses',))

[laurenyu]

hmm was there anything around that line that might suggest what could be causing the 502 response?

[stsewd]

I'm investigating that. It's probably a slow query.

[stsewd]

Looks like the slowness is because the amount of active versions your project has (149). I'm seeing the same on projects that have more than 100 active versions. This number of project with more than 100 active versions is really low, though. There are only 7 projects affected by this.

The explanation, we are hitting azure cloud storage to check if the epub, zip, pdf files exists

readthedocs.org/readthedocs/projects/models.py

Lines 779 to 791 in d298ff5

	def has_media(self, type_, version_slug=LATEST, version_type=None):
	path = self.get_production_media_path(
	type_=type_, version_slug=version_slug
	)
	if os.path.exists(path):
	return True
	storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
	storage_path = self.get_storage_path(
	type_=type_, version_slug=version_slug,
	version_type=version_type
	)
	return storage.exists(storage_path)

The API returns that here

readthedocs.org/readthedocs/api/v2/serializers.py

Line 79 in d298ff5

downloads = serializers.DictField(source='get_downloads', read_only=True)

(but we actually don't use that value for building)

Anyway, we are in the progress of a solution to avoid querying azure that much #6278

In the mean time, if you don't really need to keep all 149 versions active, you can disable them https://readthedocs.org/projects/sagemaker/versions/. Having less than 100 active versions should make the builds happy again.

[laurenyu]

Great, thanks for digging into this! I see #6278 has been merged - is there an ETA on when that will be released?

[stsewd]

@laurenyu the above PR is just one part of what needs to be done to reduce the number of queries.

[stsewd]

We now don't query azure anymore

readthedocs.org/readthedocs/builds/models.py

Lines 375 to 399 in ae6aeb5

	def get_downloads(self, pretty=False):
	project = self.project
	data = {}
	def prettify(k):
	return k if pretty else k.lower()
	if self.has_pdf:
	data[prettify('PDF')] = project.get_production_media_url(
	'pdf',
	self.slug,
	)
	if self.has_htmlzip:
	data[prettify('HTML')] = project.get_production_media_url(
	'htmlzip',
	self.slug,
	)
	if self.has_epub:
	data[prettify('Epub')] = project.get_production_media_url(
	'epub',
	self.slug,
	)
	return data