Deployment

[rgommers]

We need to complete several steps:

• Integrate surge.sh to update a sample site per git push for review.
• Add deployment to Github Pages using Github Actions. We should use a different Github Pages than what's currently hosting numpy.org (we can use mine if needed, I think each profile only get one?). This should deploy automatically when PR's are merged to master.

@Shekharrajak, am I missing anything?

[rgommers]

@joelachance converting these to issues, and adding the correct handle for Shekhar

[Shekharrajak]

For 2nd point

• Add deployment to Github Pages using Github Actions. We should use a different Github Pages than what's currently hosting numpy.org (we can use mine if needed, I think each profile only get one?). This should deploy automatically when PR's are merged to master.

As far as I understand there are repositories: https://github.com/numpy/neps, https://github.com/numpy/devdocs, https://github.com/numpy/doc, https://github.com/numpy/numpy.github.com which have commits - the auto generated static HTML files using https://github.com/numpy/numpy, https://github.com/numpy/numpy.org .

So now if we use Hugo for all the static pages like NEPs and pages like: about, community, conduct etc. then we don't need of separate repos: neps, numpy.github.io .

We can have https://github.com/numpy/doc which will contain the generated HTML for the docstrings from the sphinx tool. So it will be deployed on https://numpy.org/doc (since https://numpy.github.io is pointing to https://numpy.org).

What do you people think? Let me know if I misunderstood anything. I also want to understand more about how numpy-circleci-bot do commit automatically. I can see only single commit here : https://github.com/numpy/devdocs

[mattip]

Copying the table from this comment, here are the github repos under the numpy organization, the trigger that currently causes an update, the destination dummy repo used for github pages, and the link where these github pages are published.

Content repo	Trigger	Destination Repo	Web link	Notes
numpy.org	Merge to master then make upload	numpy.github.com	https://numpy.org/
numpy	Merge to master via circleCI integration	devdocs	https://numpy.org/devdocs
numpy	Merge to master via cirecleCI integration	neps	https://numpy.org/neps	Can be folded into devdocs?
numpy	Manual release step via make merge-doc	doc	https://numpy.org/doc	Now the canonical home of docu- mentation archive

[Shekharrajak]

1st point :

Integrate surge.sh to update a sample site per git push for review.

For this we need to have some environment variables like SURGE_LOGIN and SURGE_TOKEN which can be set in settings->secrets . I think currently I don't have access for adding secret keys (I am not able to see the settings tab in this repository).

Once I get access I will start working on surge integration .

[mattip]

Note AFAICT github actions is another name for azure pipelines. I hope github actions is still a yaml-style file that is added to the repo and not some obscure gui-based, github-only thing that it will be difficult to manage.

[mattip]

The circleCI integration is in these lines, as part of the CI (not an additional bot).

[Shekharrajak]

Yeah thank! I checked the yml file: https://github.com/numpy/numpy/blob/master/.circleci/config.yml

[Shekharrajak]

Few points I have in my mind, just want to share:

1. https://numpy.org/devdocs/ static contents (like: Reporting bugs, About NumPy, NumPy Enhancement Proposals, etc whichever is independent of the docstring ) should be moved to numpy.org Hugo app.

2. Later on we can move the numpy/doc/source/dev .rst files to Hugo app (since the content is mostly about the development workflow and not related to the numpy docstrings).

3. There are numpy/doc/source/docs .rst files to Hugo (the content is mostly about how to Build the NumPy API and reference docs).

4. Mostly numpy/doc/source/reference and numpy/doc/source/user is the contents which refers to the generated reference files (from the docstring). So those will remain same (some folder structure can be improved later) and should be deployed on https://numpy.org/doc using new sphinx theme.

[mattip]

Where would the documents in (1-3) end up in the numpy.org site layout? Right now they are part of the /devdoc and /doc hierarchy (except for the NEPS), I understand you are proposing they go somewhere else? I am missing the big picture design

[Shekharrajak]

Where would the documents in (1-3) end up in the numpy.org site layout? Right now they are part of the /devdoc and /doc hierarchy (except for the NEPS), I understand you are proposing they go somewhere else? I am missing the big picture design

Since they are static .rst files I believe it should be in this repository (numpy.org) Hugo version(newsite-branch).
So that we don't need to maintain separate repositories.

Once we have the newsite ready then we can push/commit the build (the generated HTML file from the Hugo) into the numpy.github.io repository.

WDYT? Is there any specific reason to put contents in different repositories (neps, numpy.github.io, devdocs) ?

[Shekharrajak]

Summary of my proposal:

Hugo application - We will try to move all the static files which are not referring to docstrings (numpy codebase), can be written simply in .md file.

So no this will make the maintains of the docs easy and contributors can add/modify the static pages they don't have to setup sphinx for it and Hugo makes the documentation and development easy.

Docstring part - Using Sphinx we will generate the static html files for the docstring and quickstart, user guide files. Now the build can be pushed to numpy/doc repository (using github actions CI/CD tool whenever new merge happens - similar to circle-ci deploy job ).

I am trying to make the development of sphinx part easy. Please have a look at this repo: https://github.com/Shekharrajak/scipy-sphinx-theme-v2

This repo uses the grunt task runner and livereload functionality makes the development easy. Because when user changes the docs file (any .rst file) then it rebuild and serve the new content dynamically in localhost:2121 .

WDYT?

[rgommers]

Note AFAICT github actions is another name for azure pipelines.

It's not, it's GitHub's CI/CD that's still in beta: https://github.com/features/actions. Is quite promising.

1. https://numpy.org/devdocs/ static contents  (like: Reporting bugs, About NumPy, NumPy Enhancement Proposals, etc whichever is independent of the docstring ) should be moved to numpy.org Hugo app.

About, Reporting Bugs and License yes, I agree. NumPy Enhancement Proposals should stay in Sphinx.

2. Later on we can move the [numpy/doc/source/dev](https://github.com/numpy/numpy/tree/master/doc/source/dev) .rst files to Hugo app (since the content is mostly about the development workflow and not related to the numpy docstrings).

That decision I would postpone. Could make sense, but low-prio.

3. There are [numpy/doc/source/docs](https://github.com/numpy/numpy/tree/master/doc/source/docs) .rst files to Hugo (the content is mostly about how to Build the NumPy API and reference docs).

This I'm also not sure about, let's leave them where they are for now.

4. Mostly [numpy/doc/source/reference](https://github.com/numpy/numpy/tree/master/doc/source/reference) and [numpy/doc/source/user ](https://github.com/numpy/numpy/tree/master/doc/source/user)is the contents which refers to the generated reference files (from the docstring). So those will remain same (some folder structure can be improved later) and should be deployed on https://numpy.org/doc using new sphinx theme.

Indeed.

WDYT? Is there any specific reason to put contents in different repositories (neps, numpy.github.io, devdocs) ?

`Those are all deployment repos, so they can go if we have a better deployment method. Whatever works I'd say - since those repos don't affect where and how people work on content, it's not too critical.

Hugo application - We will try to move all the static files which are not referring to docstrings (numpy codebase), can be written simply in .md file.

I'd start with only the parts that clearly belong on the website, like About and Reporting Bugs. The larger set of content is a bit painful to migrate from ReST to Markdown and it's not obvious that it will help. Also, it may interfere with the translation plan.

Docstring part - Using Sphinx ....

Yep that all looks great to me!

[Shekharrajak]

Surge integration: #46, #58

[rgommers]

Deployment tasks are done, things work well with preview and with using GH Actions for deploying via the numpy.github.com repo.