Search: stop relying on the DB when indexing

[stsewd]

What's the problem this feature will solve?

Currently, we are keeping track of all HTML files (this is one of our largest table). We do this mainly for re-indexing (we will be relying on these models for handling 404s, but we don't need to keep track of all files for that).

Describe the solution you'd like

Instead of relying on the DB, walk the storage. And we can get the search ignore/ranking patterns from the config of the build object attached to the version.

readthedocs.org/readthedocs/builds/models.py

Lines 310 to 327 in 84f889a

	def config(self):
	"""
	Proxy to the configuration of the build.
	:returns: The configuration used in the last successful build.
	:rtype: dict
	"""
	last_build = (
	self.builds(manager=INTERNAL).filter(
	state=BUILD_STATE_FINISHED,
	success=True,
	).order_by('-date')
	.only('_config')
	.first()
	)
	if last_build:
	return last_build.config
	return None

We would still need to create HTMLFile/ImportedFile models, but only the ones that are needed for our 404 handler (**/index.html, 404.html, and robots.txt)

Alternative solutions

None

Additional context

ref #10512

[humitos]

This is good to me 👍🏼

I'd like to see some data about:

• how many AWS S3 requests we would need to do for a normal-size project?
• what would happen for projects with a bunch of files and directories/sub-directories?

[stsewd]

@humitos times would be similar to the fileify task https://one.newrelic.com/nr1-core/apm-features/transactions/NjM3Njd8QVBNfEFQUExJQ0FUSU9OfDIzNzkwOTY?account=63767&duration=7776000000&filters=%28domain%20IN%20%28%27APM%27%2C%20%27EXT%27%29%20AND%20type%20IN%20%28%27APPLICATION%27%2C%20%27SERVICE%27%29%29&state=67722edc-1dda-77a2-6e2d-45498ce1eca2, to optimize the process, we can trigger the search index when searching for index.html/404.html files.

[humitos]

That is 9.11s avg to index a project completely? From that graph, I see that 2.5s from those are because of boto3 calls to the storage.