Search: Only use generic parsers

[benjaoming]

Since #10128 removed specific Sphinx domain logic in the SphinxParser, we aren't doing much more search indexing that's specific for Sphinx or MkDocs.

• Make SphinxParser redundant (move anything essential into the GenericParser)
• Make MkDocsParser redundant (move anything essential into the GenericParser)
• Remove entire readthedocs.sphinx_domains application?
• Backup and delete database table (ops)

CC: @stsewd does this list look good? Perhaps we can break into separate issues/PRs if any change becomes too complex.

[stsewd]

Looks good. Not sure if we need to back up the tables, since we aren't using them anymore, but not strong opinion about not doing it.

We should also check if we can remove the generation of the readthedocs-sphinx-domain-names.json file from our sphinx extension and the fjson files too.

readthedocs.org/readthedocs/projects/tasks/search.py

Line 145 in 40d21dc

type_file = build_media_storage.join(json_storage_path, 'readthedocs-sphinx-domain-names.json')

https://github.com/readthedocs/readthedocs-sphinx-ext/blob/a46f5ca05c7c8ec7ec30b44cc724ac334ce91a38/readthedocs_ext/readthedocs.py#L198

https://github.com/readthedocs/readthedocs-sphinx-ext/blob/f1145b1819458643f442b21083b3397aeebc8e72/readthedocs_ext/readthedocs.py#L280-L280

[humitos]

We should also check if we can remove the generation of the readthedocs-sphinx-domain-names.json file from our sphinx extension and the fjson files too.

💯 , this will speed up the build time a little bit. I'm 👍🏼 on removing it if possible.

[humitos]

It would be awesome to return to this work and finish this project (cc @ericholscher). The work on this issue will allow us to use always the same parser without matter the doctool the user is using (which will give them the exact same search experience). We are pretty close now! I arrived here while doing the cleanup of the feature flags at #9779

Remember to remove the INDEX_FROM_HTML_FILES as well when doing this work. It won't be required anymore, since everybody will be indexing from HTML files.

[ericholscher]

Agreed, this is a useful next step on this work. I thought we'd already finished making everything use the HTML-only parsing.

[humitos]

While reviewing #10042 I found the work from this issue is, in part, required to removed the Version.documentation_type attribute.

[ericholscher]

We're slowly rolling this out via feature flag 👍