point doc archive to numpy.org

[mattip]

As per the new spec, point to numpy.org/doc for archived documents and numpy.org/neps for neps

This still leaves the numpy org with 4 html repos (3 once we fold neps into devdocs) dedicated to github pages.

Content repo	Trigger	Destination Repo	Web link	Notes
numpy.org	Merge to master	numpy.github.com	https://numpy.org/
numpy	Merge to master	devdocs	https://numpy.org/devdocs
numpy	Merge to master	neps	https://numpy.org/neps	Can be folded into devdocs
numpy	Manual release step	doc	https://numpy.org/doc	Now the canonical home of docu- mentation archive

[mattip]

https://numpy.org/doc is "live", the doc repo is a 77MB download with the 1.14, 1.15, 1.16 documentation. Should I add more? I guess we need 1.13 since that is the version SciPy builds against.

Once numpy/numpy#13990 goes in (fixes make dist) I will work on adding another make target to the numpy/numpy documentation Makefile to automate adding a numpy version's documentation to the numpy/doc repo

[rgommers]

https://numpy.org/doc is "live", the doc repo is a 77MB download with the 1.14, 1.15, 1.16 documentation. Should I add more? I guess we need 1.13 since that is the version SciPy builds against.

What do you propose to do with the older ones? Just link to http://docs.scipy.org/doc/ as the archive for older versions?

Yes, 1.13 definitely, not sure if anything else is needed, perhaps not. Also, can you please use the docs for the latest released versions on each branch, not just .dev0+<some-commit>?

It would also be good to know how expensive (in terms of size) it is to add a new release. Content is often similar, but not clear to me that that helps. So if you add 1.16.4 docs and remove 1.16.5.dev0+ without rewriting git history (as an experiment, rewriting is probably best), how many MB does that add?

[mattip]

What do you propose to do with the older ones?

We could either link to them or drop them. I think I would prefer to drop them. If there is opposition we could link to scipy.org

.> .. can you please use the docs for the latest released versions on each branch, not just .dev0+?

Ahh, I think I checked out maintenance/1.aa.x instead of v1.aa.b

So if you add 1.16.4 docs and remove 1.16.5.dev0+ without rewriting git history (as an experiment, rewriting is probably best), how many MB does that add?

29MB. Probably less if I do not git rm -r but instead rm -rf, but just the html.zip and pdf files are 17MB, We may not want to update the documentation for new point releases.

git rm -r 1.16
# copy in the built docs
git add 1.16
git commit -m"replace 1.16.x with 1.16.4"

[rgommers]

What do you propose to do with the older ones?

We could either link to them or drop them. I think I would prefer to drop them. If there is opposition we could link to scipy.org

I'd suggest a link, it doesn't hurt. Older releases have some value, e.g. if a user wants to check if a function is present in that release.

[rgommers]

We may not want to update the documentation for new point releases.

Yeah, just 1.X.0 then, and perhaps an update only in case of important changes (release notes are always most up-to-date in devdocs).

[mattip]

added 1.13 docs to numpy/doc (visible as https://numpy.org/doc/1.13) and unzipped the zip archive for 1.16, 1.15, 1.14 docs without git rm. Added link to sicpy.org/doc/index.html for older versions.

du -sh .git now reports 119MB

[dashohoxha]

du -sh .git now reports 119MB

Maybe I am wrong, but I have a feeling that if html pages can be auto-generated, probably they should not be kept in git.

[mattip]

The four repos I mentioned above all contain generated documentation that is served via github pages, which is a convenient way to serve static pages. This requires checking the rendered documentation into a repo. Another option is to host the pages on a hosting service where we would copy the static pages. @dashohoxha do you know of such hosting that would be reliable, free, stable, and would require no maintenance from us?

[dashohoxha]

Lately I have started using GitLab, and it allows you to run your own build script, which generates the pages that you want to publish. It looks something like this:
https://gitlab.com/dashohoxha/101-problema-programimi/blob/master/.gitlab-ci.yml

For GitHub I was not sure whether this could be done, however I found this doc which explains how to do it:
https://circleci.com/blog/deploying-documentation-to-github-pages-with-continuous-integration/

It seems much more complicated than GitLab, but at least it is possible.
However, being such a complicated thing, I am not sure whether it is worth the trouble. Maybe generating html pages locally and pushing them to GitHub is OK.

[dashohoxha]

The right blog is this one: https://circleci.com/blog/deploying-documentation-to-github-pages-with-continuous-integration/

[mattip]

I would love to avoid committing the archived documentation into a repo, but the other choices seem less optimal. My departure point is that we do not want to maintain our own document server.

The blog post you linked to states

Conversely, the least infrastructure-dependent workflow for building documentation is for developers to build docs locally and check the results into the project repository.

What workflow would support generating on-the-fly different versions of the documentation from the numpy/numpy repo without actually storing the artifacts in some kind of repository? I think it would have to

for v in ['1.13', '1.14', '1.15', '1.16', 1.17']:

• set up a venv for the version of python best suited to generating that version of the documentation
• build and install numpy so the docstrings are correct
• generate the documentation and copy it to a common deploy directory
• trigger some action to tell the hosting service "use this new deploy directory"

Each time we wish to update the archived documentation, we would have to build all the versions, which sounds brittle. Perhaps at some point, when this repo grows to be unwieldy, we will be forced to find another solution, but I think this proposal will keep us going for another decade or so.

[dashohoxha]

What workflow would support generating on-the-fly different versions of the documentation from the numpy/numpy repo without actually storing the artifacts in some kind of repository?

I think that we would have to use a build script that does all these steps, for example build-docs.sh (or build-docs.py, I am more familiar with bash scripts). We would also have to build a docker image and a docker container, to ensure that the build script runs without any surprises. This container should include all the tools that are needed to generate the docs (python, sphinx, etc.). Most probably there are already such docker images available, so that we don't have to build and maintain our own.

[mattip]

@dashohoxha Given that 5 years of documentation fits into 120MB, I think archiving versions is preferable to containerization + tooling + CI integration, especially since we are discussing a run-only-at-release process that should not get a lot of traffic. We can revisit this decision if the release manager finds the git clone step is slowing them down.

[dashohoxha]

@mattip I do not insist, I just made a suggestion. Keeping it simple is better.

[mattip]

@rgommers ping

[rgommers]

thanks for the ping @mattip, merged.

I did note that there's no way to get back from numpy.org/doc to numpy.org, that would be nice to fix in a follow-up

[mattip]

get back from numpy.org/doc to numpy.org

That has to do with templating in the numpy/numpy repo. I will see if I can change it.

[stefanv]

@mattip I could not quite distill from this entire discussion the current build workflow. Does the documentation auto-update for 1.17.x releases, and those all end up at the same URL? That's what we do for skimage, e.g.:

https://scikit-image.org/docs/0.15.x/

We do not try to keep the archive small for scikit-image, but at some point we'll probably need to do that by removing a bunch of history. I get the feeling that NumPy would take a while longer to run into that problem. And I thought (although I might be mistaken), that you can now clone --depth=1 and then push back to the originating repo, avoiding the need to download the entire archive each time you build docs.

Are steps planned to highlight the current version of the documentation displayed, and to provide a mechanism to navigate to newer/older versions?

[mattip]

@stefanv the documentation release workflow is now (in the numpy repo, not here)

• tag a release
• make merge-doc will clone and update numpy/doc with the latest documentation archive, using the tag to put the archive into a y.x directory, so turning 1.17.x -> 1.17
• hand editing the index.html of the numpy/doc clone to reference a new directory (if needed for a new version)
• commit the changes to numpy/doc locally
• push to numpy/doc, which will go live on github pages at https://numpy.org/doc

If there are clever ways to improve the clone/add/commit/push git workflow, they can be added to the doc/source/Makefile for make merge-doc.

Adding a version widget was PR numpy/numpy#12097 which was closed not merged.