Dump sphinx domains docstrings

[dojutsu-user]

This is working correctly on local.
Tested on projects:

• docs
• kuma (no file was generated because there were no sphinx domain objects except std:label and std:doc)
• sphinx docs
• notfound
• requests docs

In this json file, we will be having id and content of the sphinx domains.
We can then load the file in our tasks.py and get the content of the sphinx domains from its anchor property (which comes from objects.inv fle).

Ref docs: https://www.sphinx-doc.org/en/master/extdev/nodes.html#nodes-for-domain-specific-object-descriptions

Related PR: readthedocs/readthedocs.org#5979

[dojutsu-user]

docutils released a new version 0.15, and it is giving import error in python 2.7.

[ericholscher]

Hrm, were you not able to get the text content out of the domain objects in Sphinx themselves? If we're just parsing the HTML directly, we could likely be doing that just from the HTML output and the pyquery approach we are already taking in search indexing.

Currently, this will only let us parse content for docs that are rebuilt. If we are doing HTML parsing, we can do it in the existing search code with existing HTML, which will work for all built docs.

[dojutsu-user]

@ericholscher
No -- unfortunately, I couldn't find the data in domain objects in Sphinx.

[dojutsu-user]

@ericholscher

Currently, this will only let us parse content for docs that are rebuilt. If we are doing HTML parsing, we can do it in the existing search code with existing HTML, which will work for all built docs.

But we need to parse the content before creating sphinx objects, I am thinking to store the docstrings with this step -- https://github.com/readthedocs/readthedocs.org/blob/70494250385978e72f788ea7e62225e0aaaa5186/readthedocs/projects/tasks.py#L1449

we may have trouble figuring out which docstrings belongs to which domain if we parse the content with parse_json.py logic.

Also -- a project needs to be rebuild to index domain objects properly. So, I think we can go in this direction.

[ericholscher]

Is there an approach where we can dump the data we need here, mapping the names to their domain and anything else we need. Then we can parse it in parse_json.py, and add the additional data if we have it? That is the approach we're taking w/ other data, and seems like the best outcome, since we can support all built docs, but then make rebuilds better 👍

[dojutsu-user]

@ericholscher
I think, this can be achieved by readthedocs/readthedocs.org#5979

[dojutsu-user]

I am thinking that this PR might not be needed here.
Will update the status.

[dojutsu-user]

I think this is not needed anymore.
Closing on readthedocs/readthedocs.org#5979.