WEB: Deployment of development website with the docs (pandas-dev#28497)

Author: datapythonista

README.md

@@ -225,7 +225,7 @@ Most development discussion is taking place on github in this repo. Further, the
 
 All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.
 
-A detailed overview on how to contribute can be found in the **[contributing guide](https://dev.pandas.io/contributing.html)**. There is also an [overview](.github/CONTRIBUTING.md) on GitHub.
+A detailed overview on how to contribute can be found in the **[contributing guide](https://dev.pandas.io/docs/contributing.html)**. There is also an [overview](.github/CONTRIBUTING.md) on GitHub.
 
 If you are simply looking to start working with the pandas codebase, navigate to the [GitHub "issues" tab](https://github.com/pandas-dev/pandas/issues) and start looking through interesting issues. There are a number of issues listed under [Docs](https://github.com/pandas-dev/pandas/issues?labels=Docs&sort=updated&state=open) and [good first issue](https://github.com/pandas-dev/pandas/issues?labels=good+first+issue&sort=updated&state=open) where you could start out.
 

azure-pipelines.yml

@@ -104,7 +104,7 @@ jobs:
     displayName: 'Running benchmarks'
     condition: true
 
-- job: 'Docs'
+- job: 'Web_and_Docs'
   pool:
     vmImage: ubuntu-16.04
   timeoutInMinutes: 90
@@ -119,6 +119,11 @@ jobs:
       ci/setup_env.sh
     displayName: 'Setup environment and build pandas'
 
+  - script: |
+      source activate pandas-dev
+      python web/pandas_web.py web/pandas
+    displayName: 'Build website'
+
   - script: |
       source activate pandas-dev
       # Next we should simply have `doc/make.py --warnings-are-errors`, everything else is required because the ipython directive doesn't fail the build on errors (https://github.com/ipython/ipython/issues/11547)
@@ -128,15 +133,21 @@ jobs:
     displayName: 'Build documentation'
 
   - script: |
-      cd doc/build/html
+      mkdir -p to_deploy/docs
+      cp -r web/build/* to_deploy/
+      cp -r doc/build/html/* to_deploy/docs/
+    displayName: 'Merge website and docs'
+
+  - script: |
+      cd to_deploy
       git init
       touch .nojekyll
       echo "dev.pandas.io" > CNAME
       printf "User-agent: *\nDisallow: /" > robots.txt
       git add --all .
       git config user.email "[email protected]"
-      git config user.name "pandas-docs-bot"
-      git commit -m "pandas documentation in master"
+      git config user.name "pandas-bot"
+      git commit -m "pandas web and documentation in master"
     displayName: 'Create git repo for docs build'
     condition : |
       and(not(eq(variables['Build.Reason'], 'PullRequest')),
@@ -163,7 +174,7 @@ jobs:
       cd doc/build/html
       git remote add origin [email protected]:pandas-dev/pandas-dev.github.io.git
       git push -f origin master
-    displayName: 'Publish docs to GitHub pages'
+    displayName: 'Publish web and docs to GitHub pages'
     condition : |
       and(not(eq(variables['Build.Reason'], 'PullRequest')),
           eq(variables['Build.SourceBranch'], 'refs/heads/master'))

doc/source/whatsnew/v0.25.0.rst

@@ -828,7 +828,7 @@ If installed, we now require:
 | pytest (dev)    | 4.0.2           |          |
 +-----------------+-----------------+----------+
 
-For `optional libraries <https://dev.pandas.io/install.html#dependencies>`_ the general recommendation is to use the latest version.
+For `optional libraries <https://dev.pandas.io/docs/install.html#dependencies>`_ the general recommendation is to use the latest version.
 The following table lists the lowest version per library that is currently being tested throughout the development of pandas.
 Optional libraries below the lowest tested version may still work, but are not considered supported.
 

web/pandas/config.yml

@@ -26,15 +26,15 @@ navbar:
   - name: "Documentation"
     target:
     - name: "Getting started"
-      target: https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html
+      target: /docs/getting_started/index.html
     - name: "User guide"
-      target: https://pandas.pydata.org/pandas-docs/stable/user_guide/index.html
+      target: /docs/user_guide/index.html
     - name: "API reference"
-      target: https://pandas.pydata.org/pandas-docs/stable/reference/index.html
+      target: /docs/reference/index.html
     - name: "Contributing to pandas"
-      target: https://pandas.pydata.org/pandas-docs/stable/development/index.html
+      target: /docs/development/index.html
     - name: "Release notes"
-      target: https://pandas.pydata.org/pandas-docs/stable/whatsnew/index.html
+      target: /docs/whatsnew/index.html
   - name: "Community"
     target:
     - name: "About pandas"

web/pandas/donate.md

@@ -7,7 +7,7 @@ You can find more information about current developers and supporters in the [te
 Financial contributions will mainly be used to advance in the [pandas roadmap](community/roadmap.html).
 
 - If your **company or organization** is interested in helping make pandas better, please contact us at [[email protected]](mailto:[email protected])
-- If you want to contribute to _pandas_ with your **time**, please visit the [contributing page](https://pandas.pydata.org/pandas-docs/stable/development/index.html)
+- If you want to contribute to _pandas_ with your **time**, please visit the [contributing page]({{ base_url }}/docs/development/index.html)
 - If you want to support _pandas_ with a **donation**, please use the form below:
 
 

web/pandas/index.html

@@ -20,16 +20,16 @@ <h5>Getting started</h5>
                         <ul>
                             <!-- <li><a href="{{ base_url }}/try.html">Try pandas online</a></li> -->
                             <li><a href="{{ base_url }}/install.html">Install pandas</a></li>
-                            <li><a href="https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html">Getting started</a></li>
+                            <li><a href="{{ base_url }}/docs/getting_started/index.html">Getting started</a></li>
                         </ul>
                     </div>
                     <div class="col-md-4">
                         <h5>Documentation</h5>
                         <ul>
-                            <li><a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/index.html">User guide</a></li>
-                            <li><a href="https://pandas.pydata.org/pandas-docs/stable/reference/index.html">API reference</a></li>
-                            <li><a href="https://pandas.pydata.org/pandas-docs/stable/development/index.html">Contributing to pandas</a></li>
-                            <li><a href="https://pandas.pydata.org/pandas-docs/stable/whatsnew/index.html">Release notes</a></li>
+                            <li><a href="{{ base_url }}/docs/user_guide/index.html">User guide</a></li>
+                            <li><a href="{{ base_url }}/docs/reference/index.html">API reference</a></li>
+                            <li><a href="{{ base_url }}/docs/development/index.html">Contributing to pandas</a></li>
+                            <li><a href="{{ base_url }}/docs/whatsnew/index.html">Release notes</a></li>
                         </ul>
                     </div>
                     <div class="col-md-4">
@@ -58,10 +58,10 @@ <h5>With the support of:</h5>
                 {% if releases %}
                     <h4>Latest version: {{ releases[0].name }}</h4>
                     <ul>
-                        <li><a href="https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.25.0.html">What's new in {{ releases[0].name }}</a></li>
+                        <li><a href="docs/whatsnew/v0.25.0.html">What's new in {{ releases[0].name }}</a></li>
                         <li>Release date:<br/>{{ releases[0].published.strftime("%b %d, %Y") }}</li>
-                        <li><a href="https://pandas.pydata.org/pandas-docs/stable/">Documentation (web)</a></li>
-                        <li><a href="https://pandas.pydata.org/pandas-docs/stable/pandas.pdf">Documentation (pdf)</a></li>
+                        <li><a href="{{ base_url}}/docs/">Documentation (web)</a></li>
+                        <li><a href="{{ base_url }}/docs/pandas.pdf">Documentation (pdf)</a></li>
                         <li><a href="{{ releases[0].url }}">Download source code</a></li>
                     </ul>
                 {% endif %}

web/pandas/install.md

@@ -2,7 +2,7 @@
 
 The next steps provides the easiest and recommended way to set up your
 environment to use pandas. Other installation options can be found in
-the [advanced installation page](https://pandas.pydata.org/pandas-docs/stable/install.html).
+the [advanced installation page]({{ base_url}}/docs/install.html).
 
 1. Download [Anaconda](https://www.anaconda.com/distribution/) for your operating system and
    the latest Python version, run the installer, and follow the steps. Detailed instructions