Intregration with Dash

[Kapeli]

I'll list here all the issues I have found while trying to integrate ReadTheDocs with Dash. It would be great if this list could be worked through so that Dash could someday integrate with RTD.

Searching issues:

• I mainly want to add RTD integration to cover Python packages, and the issue is that there doesn't seem to be a correlation between Python packages listed at pypi.python.org and RTD project names. For example, beautifulsoup.readthedocs.org has docs for beautifulsoup in Chinese, while beautiful-soup-4.readthedocs.org contains the actual docs for beautifulsoup. A Dash user will most likely search for "beautifulsoup" and install the Chinese docs.
  • Possible workaround: wiki.python.org/moin/CheeseShopDev has a TODO entry for "A link from package to RTFD". When they add that I should be able to use it.
• Searching for something like "zend" doesn't give you the "zf2" project. Ideally, a search for both "zf" and "zend" should return the "zf2" project.
  • Possible workaround: one idea would be for RTD to search the title of the main page of the generated docs.

Organization issues:

• As far as I can tell, anyone can claim any project name and have docs generated for it without any kind of verification/review. That's kind of bad, as you get issues like the one I mentioned regarding beautifulsoup. I've also seen some other Chinese docs that are listed as English.

Versioning issues:

• Versions are not easy to navigate using the API. All the downloads listed in the JSON response are for "latest", regardless of the version. It's also unclear which versions are public and which are not (e.g. compare versions listed at readthedocs.org/projects/pip to readthedocs.org/api/v1/version/pip).

[Kapeli]

Another one in Chinese: redis.readthedocs.org.

[ericholscher]

Yea. We have no reliable way of mapping these things. Since RTD is not just
for Python projects, I don't want to introduce more Python-specific
functionality.

Any reason you only want to use RTD for Python projects?

On Wed, Feb 12, 2014 at 6:20 AM, Bogdan Popescu [email protected]:

Another one in Chinese: redis.readthedocs.org.

Reply to this email directly or view it on GitHubhttps://github.com//issues/662#issuecomment-34815442
.

Eric Holscher
Maker of the internet residing in Portland, Or
http://ericholscher.com

[Kapeli]

The plan is to use RTD for everything, but with a slight focus on Python.

However, Python users would most likely search for Python packages, using the same names from PyPI and they'd get the Chinese docs or not find anything at all.

It sucks even for non-Python searches. For example, searching for redis you get redis.readthedocs.org.

For a search of mongodb the API returns this, where the first result (mongodb) is a 404 and the second result (mongodb-documentation) is in Chinese.

I guess the main problem is that RTD has no kind of moderation and/or organisation going on...

[ericholscher]

Sure. The only canonical resource that we have for projects is their source code URL. Pypi doesn't index that data, so there is no good way to link them together. I don't know what kind of moderation you want/expect us to do?

[Kapeli]

I don't know what kind of moderation you want/expect us to do?

In terms of moderation, I'd want that there would be no Chinese (or anything but English) docs on the English side of ReadTheDocs, or have the other language docs clearly marked as such.

Going to http://redis.readthedocs.org/en/latest/ gives you Chinese docs for Redis, despite of the /en/ in the URL. As far as I understand, RTD supports language separation, but it's not enforced in any way.

[zxaos]

@Kapeli Doesn't PyPi recommend official documentation at pythonhosted.org anyway instead of readthedocs? Would searching that be a workable alternative?

[Kapeli]

@zxaos it would be an alternative, but I see no way of searching it and I also couldn't find any packages that have docs there. Do you know any?

[zxaos]

@Kapeli I don't think there's a search per-se. It looks like each library has its own subdirectory, based on the library name.

Many, but not all of them, seem to use Sphinx docs if it makes parsing easier.

Some examples:

• http://pythonhosted.org/setuptools/
• http://pythonhosted.org/six/
• http://pythonhosted.org/pythran/
• http://pythonhosted.org/blinker/
• http://pythonhosted.org/Pyro4/

[Kapeli]

Very few of the packages seem to have docs there. Also, there doesn't seem to be a way to download the docs, so I can't use it, sorry.

[odeckmyn]

Why not let the Dash2 user enter the RTFD url he wants as a 1st step. This solves every issues seen in this thread while Python community solves the “catalog of docs” issue ?

User would enter any Sphinx generated URL to integrate into (as far as I understand correctly your work)

Let me give http://beautiful-soup-4.readthedocs.org URL to Dash2 in order to integrate BeautifulSoup documentation. Or http://scikit-learn.org/stable/documentation.html for SciKit documentation (on his own URL, neither RTFD nor pythonhosted platform, but still Sphinx doc).

What do you think ?

[Kapeli]

This is what I don't like about that:

1. I have no way to download the docs for offline use, RTD provides that, other websites might as well, but I would have no way of guessing the URL. I'd have to write an offline website cloner, which is not feasible because a clone might take a very long time and I can't show any kind of progress for it.
2. Versioning. How would Dash check for updates and know when to regenerate the docs?

These 2 issues would be fixed if I'd only support RTD (i.e. only allow RTD URLs), but I'm not that ok with that since I'm basically telling users to go find the docs on RTD and come back, which would be a source of complaints and would boil down to the same initial problem: you can't really find projects on RTD.

[odeckmyn]

I understand all that. Just for your information, I never search for a python documentation from a documentation site (neither RTD nor python-hosted), but 99% of times from the github file (=README most of times). So, from my point of view, asking user to enter the URL to documentation is not an issue. But I’m not the one in charge of user support :)

BTW, thanx for your work.

[chris-erickson]

I also usually just find the docs on the github page as opposed to searching for them. I would be completely happy if I had to manually link in the packages I want docs for. I don't typically go through that many, so this would scale fine for me - even if I had to make sure I have the most up-to-date versions. I do understand the bigger problems this might create, however.

[pavarnos]

+1 for downloadable docs. I travel a lot, and its very handy to have downloadable versions of package docs so I can work on a plane or in an airport where there is no free wifi. For example, I found this issue after hunting for downloadable docs for the phptools/phpword and phptools/phpexcel toolset before boarding a 12 hour flight. These are large libraries with non-trivial multi page manuals that are so much easier to use of you can look up the manuals.

[GaretJax]

Another possible (ad interim) solution is to build Dash docsets (*.docset) during the compilation process on RTD and then offering a download link at the same place where the download links for PDF etc. are found.

[ghuntley]

Project @reactiveui would love to see this functionality added. Ref: reactiveui/ReactiveUI#771

We are a .NET / C# / Windows / Mono / Xamarin project.

[jacobsvante]

I agree with @chris-erickson that being able to manually add the docs that you want would be perfectly good. Together with @GaretJax's idea on having RTD generate .docset files I think we're on to something.

What do you think @Kapeli?

[Kapeli]

I'm very reluctant to add half-baked features to Dash, because they cause a lot of confusion and complaints from users. I might do it sometime in the future though, but I'd much rather work on features that I can add properly.

[jacobsvante]

@Kapeli I understand your point! Sucks to be left out though because you're a Python junkie 😢

[ericholscher]

Closing this, as I think it doesn't make sense to support Dash as things are currently architected. If the Python world ever ends up with a specific API doc tool, it would make sense to integrate that with Dash, but since Dash isn't used for Prose documentation, Read the Docs will likely never support it fully.

[Kapeli]

Dash works fine with Prose docs. It's a lot harder to support than API docs, yes, but there are a lot of docsets that are not API-related or API-only. That's my job though and you're not supposed to do anything about it. I'm not expecting RTD to generate docsets.

My issues with supporting RTD have nothing to do with the docs themselves. I don't think I mentioned the docs anywhere in this issue, I might be wrong though. I admit I have lots of issues with Sphinx, but I can work around them.

My issues are with the services RTD provides (searching, versioning and organisation). If RTD would fix those issues, or at least some of them, I'd be able to integrate it.

If this is something you're not willing to do, that's fine and I understand and respect your decision. Don't blame it on Dash and the Prose docs though. The main issue is RTD's API and organisation. Everything else I can work around.

[ericholscher]

Good points. The main issue on our side is that most package repositories (PyPI specifically) don't allow any kind of oauth claiming, and we don't really have a sane way to namespace and claim packages. Since we aren't a language specific tool, there are a large number of things that could be named "sphinx" for example.

I guess the proper solution would be to allow a project to specify a link to an existing package in a repository somewhere (eg. 'beautifulsoup' on RTD -> bs4 on PyPI), but again without platform support, it's almost impossible for us to verify a claim of ownership.

[Kapeli]

Project naming is an issue, but I think I can work around it a bit on my end by warning users about the issue, sorting the results and possibly allowing users to search using the full RTD URL of a project they want.

Things have changed with Dash and RTD since this issue was opened, so maybe it's best if we go through it step by step.

How do I search for projects using the API? The only thing I could find was https://read-the-docs.readthedocs.org/en/latest/api.html#file-search, but it doesn't seem to work (see http://readthedocs.org/api/v1/file/search/?format=json&q=virtualenvwrapper). File search is not really what I want though, just project search. For my needs, I think I'd want an API that searches projects using the RTD name (i.e. pip), the RTD URL (i.e. http://pip.readthedocs.org/) and the project URL (i.e. https://pip.pypa.io/) and nothing else. I can work with any kind of project search API you have though.

[stantond]

@Kapeli If this is still a thing, that search problem could sit in a new issue to be solved in isolation

(I'd love to see RTD integration in Dash!)

[Kapeli]

@stantond Good idea. I've done that. One thing I've noticed is that there's an API to get the complete list of projects on RTD. I can definitely use that to just grab the list of all projects and implement my own search over it.

I'll start working on it and experimenting.

I'm closing this issue as it's become too old/outdated.

[jonashaag]

@Kapeli what's the state of Dash and RTD integration? I'd be interested in helping with this.

[Kapeli]

I've given up on it for now. Even if I come up with a way to get what I need from RTD (search, versioning and so on), there's still the issue with Sphinx, which doesn't have a strict/common format between docs and is hard to index right. Using the intersphinx index is the best approach I have so far, but it often misses things and is incomplete, even for the main Python docs.

It's one of those features that I'm afraid to add because then I'll be stuck maintaining it forever and constantly add fixes/workarounds for. Docs for Node.js packages are in the same boat.

[asfaltboy]

I admit I haven't read all the thread above, but would really love to see more rtfd docsets in dash. Maybe we should undertake this feature iteratively, for example first creating an rtfd scraper which handles a few common sphinx themes, and try to make it modular enough to allow people to add their own?

If one was to write this sort of parser, where would y'all suggest to start?

[rgov]

There's a newer RTD API.

1. GET https://readthedocs.org/api/v2/project/?slug=requests
2. Pick a result and get its ID, e.g., 668
3. GET https://readthedocs.org/api/v2/project/668/active_versions/

The download links appear to be correct for all versions. The canonical URL field may not be correct.

[humitos]

@Kapeli Hi! This thread caught my attention because I've been working on a new APIv3 (see https://docs.readthedocs.io/en/latest/api/v3.html and #6169). It's currently on Beta and we are planning to make a stable release soon. Although, I'd love to know if you are still interested on having an integration between Dash and Read the Docs and what are exactly the things that are missing from our side for this to happen.

From what I understand from here, only a search endpoint where you can find projects by name, url or project url. Am I correct?

Regarding the consistency around the names/slugs of the projects, I think there isn't too much we can do there.

[Kapeli]

@humitos Yes, the search endpoint is all that's needed now. If possible, the results should also include the language the project is in, so I can clearly show it in the interface.

[dufferzafar]

@Kapeli What tool do you use to convert a read the docs site to a dash docset?

[Kapeli]

@Kapeli What tool do you use to convert a read the docs site to a dash docset?

I made a custom tool for almost every Python lib I support. Try doc2dash for something general purpose.