Merge branch 'master' into support-mkdocs-html-pages

Author: stsewd

CHANGELOG.rst

@@ -1,3 +1,25 @@
+Version 4.1.3
+-------------
+
+:Date: April 07, 2020
+
+* `@stsewd <https://github.com/stsewd>`__: Don't do unnecessary queries when listing subprojects (`#6869 <https://github.com/readthedocs/readthedocs.org/pull/6869>`__)
+* `@stsewd <https://github.com/stsewd>`__: Don't do extra query if the project is a translation (`#6865 <https://github.com/readthedocs/readthedocs.org/pull/6865>`__)
+* `@stsewd <https://github.com/stsewd>`__: Remove private argument from resolver (`#6864 <https://github.com/readthedocs/readthedocs.org/pull/6864>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Make development docs a bit easier to find (`#6861 <https://github.com/readthedocs/readthedocs.org/pull/6861>`__)
+* `@davidfischer <https://github.com/davidfischer>`__: Add an advertising API timeout (`#6856 <https://github.com/readthedocs/readthedocs.org/pull/6856>`__)
+* `@humitos <https://github.com/humitos>`__: Add more exceptions as WARNING log level (`#6851 <https://github.com/readthedocs/readthedocs.org/pull/6851>`__)
+* `@humitos <https://github.com/humitos>`__: Limit concurrent builds (`#6847 <https://github.com/readthedocs/readthedocs.org/pull/6847>`__)
+* `@humitos <https://github.com/humitos>`__: Release 4.1.2 (`#6840 <https://github.com/readthedocs/readthedocs.org/pull/6840>`__)
+* `@humitos <https://github.com/humitos>`__: Report build status in a smarter way (`#6839 <https://github.com/readthedocs/readthedocs.org/pull/6839>`__)
+* `@stsewd <https://github.com/stsewd>`__: Update messages-extends to latest version (`#6838 <https://github.com/readthedocs/readthedocs.org/pull/6838>`__)
+* `@humitos <https://github.com/humitos>`__: Do not save pip cache when using CACHED_ENVIRONMENT (`#6820 <https://github.com/readthedocs/readthedocs.org/pull/6820>`__)
+* `@stsewd <https://github.com/stsewd>`__: Force to reinstall package (`#6817 <https://github.com/readthedocs/readthedocs.org/pull/6817>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Denormalize from_url_without_rest onto the redirects model (`#6780 <https://github.com/readthedocs/readthedocs.org/pull/6780>`__)
+* `@davidfischer <https://github.com/davidfischer>`__: Developer docs emphasize the Docker setup (`#6682 <https://github.com/readthedocs/readthedocs.org/pull/6682>`__)
+* `@davidfischer <https://github.com/davidfischer>`__: Document setting up connected accounts in dev (`#6681 <https://github.com/readthedocs/readthedocs.org/pull/6681>`__)
+* `@humitos <https://github.com/humitos>`__: Return full path URL (including `.html`) on `/api/v2/docurl/` endpoint (`#6082 <https://github.com/readthedocs/readthedocs.org/pull/6082>`__)
+
 Version 4.1.2
 -------------
 

docs/_static/images/development/bitbucket-oauth-setup.png

@@ -25,7 +25,7 @@ Contributing to development
 
 If you want to deep dive and help out with development on Read the Docs, then
 first get the project installed locally according to the
-:doc:`Installation Guide <development/install>`. After that is done we
+:doc:`Installation Guide </development/install>`. After that is done we
 suggest you have a look at tickets in our issue tracker that are labelled `Good
 First Issue`_. These are meant to be a great way to get a smooth start and
 won't put you in front of the most complex parts of the system.
@@ -40,46 +40,7 @@ They are simply things that are explained.
 If you still didn't find something to work on, search for the `Sprintable`_ label.
 Those tickets are meant to be standalone and can be worked on ad-hoc.
 
-When contributing code, then please follow the standard Contribution
-Guidelines set forth at `contribution-guide.org`_.
-
-We have a strict code style that is easy to follow since you just have to
-install `pre-commit`_ and it will automatically run different linting tools
-(`autoflake`_, `autopep8`_, `docformatter`_, `isort`_, `prospector`_, `unify`_
-and `yapf`_) to check your changes before you commit them. `pre-commit` will let
-you know if there were any problems that it wasn't able to fix automatically.
-
-To run the `pre-commit` command and check your changes:
-
-.. prompt:: bash $
-
-    pip install -U pre-commit
-    git add <your-modified-files>
-    pre-commit run
-
-or to run against a specific file:
-
-.. prompt:: bash $
-
-    pre-commit run --files <file.py>
-
-`pre-commit` can also be run as a git pre-commit hook. You can set this up
-with:
-
-.. prompt:: bash $
-
-    pre-commit install
-
-After this installation, the next time you run `git commit` the `pre-commit run`
-command will be run immediately and will inform you of the changes and errors.
-
-.. note::
-
-    Our code base is still maturing and the core team doesn't yet recommend
-    running this as a pre-commit hook due to the number of changes this will
-    cause while constructing a pull request. Independent pull requests with
-    linting changes would be a great help to making this possible.
-
+You can read all of our :doc:`/development/index` to understand more the development of Read the Docs. When contributing code, then please follow the standard Contribution Guidelines set forth at `contribution-guide.org`_.
 
 .. _Feature: https://github.com/readthedocs/readthedocs.org/issues?direction=desc&labels=Feature&page=1&sort=updated&state=open
 .. _Improvement: https://github.com/readthedocs/readthedocs.org/issues?q=is%3Aopen+is%3Aissue+label%3AImprovement
@@ -88,14 +49,6 @@ command will be run immediately and will inform you of the changes and errors.
 .. _Sprintable: https://github.com/readthedocs/readthedocs.org/issues?q=is%3Aopen+is%3Aissue+label%3ASprintable
 .. _contribution-guide.org: http://www.contribution-guide.org/#submitting-bugs
 
-.. _pre-commit: https://github.com/pre-commit/pre-commit
-.. _autoflake: https://github.com/myint/autoflake
-.. _autopep8: https://github.com/hhatto/autopep8
-.. _docformatter: https://github.com/myint/docformatter
-.. _isort: https://github.com/timothycrosley/isort
-.. _prospector: https://prospector.landscape.io/en/master
-.. _unify: https://github.com/myint/unify
-.. _yapf: https://github.com/google/yapf
 
 Contributing to documentation
 -----------------------------
@@ -106,34 +59,6 @@ There are guidelines around writing and formatting documentation for the project
 For full details, including how to build it, see :doc:`/development/docs`.
 
 
-Developer documentation
------------------------
-
-These are guides and helpful documentation to running your own local version of Read the Docs
-for development or taking the open source Read the Docs codebase
-for your own :doc:`custom installation <custom_installs/index>`.
-
-.. toctree::
-   :maxdepth: 1
-
-   development/install
-   development/standards
-   development/search
-   development/architecture
-   development/tests
-   development/docs
-   development/front-end
-   development/design/index
-   development/buildenvironments
-   development/symlinks
-   development/settings
-   development/i18n
-   development/issue-labels
-   development/design
-   RTD Theme <https://sphinx-rtd-theme.readthedocs.io/en/latest/>
-
-
-
 Triaging tickets
 ----------------
 

docs/contribute.rst

@@ -19,4 +19,5 @@ It has those features and more!
     :maxdepth: 2
     :glob:
 
-    *
+    local_rtd_vm
+    customization

docs/custom_installs/index.rst

@@ -1,5 +1,17 @@
-Local VM Install
-================
+Local Install
+=============
+
+Before proceeding to setup your own local Read the Docs instance,
+it may be worth familiarizing yourself with the Read the Docs
+:doc:`development setup and standards </development/standards>`,
+which details the local development setup with Docker,
+and the :doc:`installation guide </development/install>`.
+
+Again, if your primary reasons for considering a private installation
+are to connect to private repositories or to have fine grained access control over your documentation,
+please consider :doc:`Read the Docs for Business </commercial/index>`.
+It has those features and many more and helps support Read the Docs!
+
 
 Assumptions and Prerequisites
 -----------------------------

docs/custom_installs/local_rtd_vm.rst

@@ -0,0 +1,26 @@
+Developer documentation
+-----------------------
+
+These are guides and helpful documentation to running your own local version of Read the Docs
+for development or taking the open source Read the Docs codebase
+for your own :doc:`custom installation </custom_installs/index>`.
+
+.. toctree::
+   :maxdepth: 1
+
+   standards
+   search
+   architecture
+   tests
+   docs
+   front-end
+   design/index
+   buildenvironments
+   symlinks
+   settings
+   install
+   i18n
+   issue-labels
+   design
+   RTD Theme <https://sphinx-rtd-theme.readthedocs.io/en/latest/>
+

docs/development/index.rst

@@ -8,6 +8,14 @@ Installation
 Here is a step by step guide on how to install Read the Docs.
 It will get you to a point of having a local running instance.
 
+.. note::
+
+    It may be worth familiarizing yourself with the Read the Docs
+    :doc:`development setup and standards </development/standards>`.
+    That document details the local setup with Docker which may be useful
+    for setting up your own instance.
+
+
 Requirements
 ------------
 

docs/development/install.rst

@@ -4,42 +4,54 @@ Search
 Read The Docs uses Elasticsearch_ instead of the built in Sphinx search for providing better search
 results. Documents are indexed in the Elasticsearch index and the search is made through the API.
 All the Search Code is open source and lives in the `GitHub Repository`_.
-Currently we are using the `Elasticsearch 6.3`_ version.
+Currently we are using `Elasticsearch 6.3`_.
 
 Local Development Configuration
 -------------------------------
 
-Installing and running Elasticsearch
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Elasticsearch is installed and run as part of the :doc:`development setup </development/standards>`.
 
-You need to install and run Elasticsearch_ version 6.3 on your local development machine.
-You can get the installation instructions
-`here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_.
-Otherwise, you can also start an Elasticsearch Docker container by running the following command::
-
-    docker run -p 9200:9200 -p 9300:9300 \
-           -e "discovery.type=single-node" \
-           docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-
-You need to override the ``ES_HOSTS`` and ``ELASTICSEARCH_DSL`` settings to point to ``127.0.0.1:9200``.
 
 Indexing into Elasticsearch
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 For using search, you need to index data to the Elasticsearch Index. Run ``reindex_elasticsearch``
-management command::
+management command:
+
+.. prompt:: bash
 
-    ./manage.py reindex_elasticsearch
+    inv docker.manage reindex_elasticsearch
 
 For performance optimization, we implemented our own version of management command rather than
 the built in management command provided by the `django-elasticsearch-dsl`_ package.
 
+
 Auto Indexing
 ^^^^^^^^^^^^^
 By default, Auto Indexing is turned off in development mode. To turn it on, change the
 ``ELASTICSEARCH_DSL_AUTOSYNC`` settings to `True` in the `readthedocs/settings/dev.py` file.
 After that, whenever a documentation successfully builds, or project gets added,
 the search index will update automatically.
 
+
+Manual Elasticsearch installation and setup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Usually you can just rely on the Docker Compose
+:doc:`development setup </development/standards>` which includes Elasticsearch.
+However, if you're developing or testing Read the Docs' search integration, you may need this.
+
+You need to install and run Elasticsearch_ version 6.3 on your local development machine.
+You can get the installation instructions
+`here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_.
+Otherwise, you can also start an Elasticsearch Docker container by running the following command:
+
+.. prompt:: bash
+
+    docker run -p 9200:9200 -p 9300:9300 \
+           -e "discovery.type=single-node" \
+           docker.elastic.co/elasticsearch/elasticsearch:6.3.2
+
+
 Architecture
 ------------
 The search architecture is divided into 2 parts.

docs/development/search.rst

@@ -1,16 +1,9 @@
-Development Standards
-=====================
+Development Setup and Standards
+===============================
 
-These are build standards that are adhered to by the core development team while
-developing Read the Docs and related services. If you are a new contributor to
-Read the Docs, it might a be a good idea to follow these guidelines as well. The
-:doc:`standard installation instructions <install>` do cover what it takes to
-get started with a local installation of Read the Docs and can be used for local
-development by non-core team.
-
-.. warning::
-
-   Take into account that Core team does not offer support on this setup currently.
+These are development setup and standards that are adhered to by the core development team while
+developing Read the Docs and related services. If you are a contributor to Read the Docs,
+it might a be a good idea to follow these guidelines as well.
 
 
 Core team standards
@@ -69,7 +62,7 @@ Serve documentation via El Proxito
 
 Search enabled by default
     Elasticsearch is properly configured and enabled by default.
-    Besides, all the documentation indexes are updated after a build is finished.
+    All the documentation indexes are updated after a build is finished.
 
 
 Set up your environment
@@ -216,3 +209,33 @@ to connect to the debug process port:
 
 The ``rdb`` debugger is similar to ``pdb``, there is no ``ipdb`` for remote
 debugging currently.
+
+
+Configuring connected accounts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These are optional steps to setup the :doc:`connected accounts </connected-accounts>`
+(GitHub, GitLab, and BitBucket) in your development environment.
+This will allow you to login to your local development instance
+using your GitHub, Bitbucket, or GitLab credentials
+and this makes the process of importing repositories easier.
+
+However, because these services will not be able to connect back to your local development instance,
+:doc:`webhooks </webhooks>` will not function correctly.
+For some services, the webhooks will fail to be added when the repository is imported.
+For others, the webhook will simply fail to connect when there are new commits to the repository.
+
+.. figure:: ../_static/images/development/bitbucket-oauth-setup.png
+    :align: center
+    :figwidth: 80%
+    :target: ../_static/images/development/bitbucket-oauth-setup.png
+
+    Configuring an OAuth consumer for local development on Bitbucket
+
+* Configure the applications on GitHub, Bitbucket, and GitLab.
+  For each of these, the callback URI is ``http://community.dev.readthedocs.io/accounts/<provider>/login/callback/``
+  where ``<provider>`` is one of ``github``, ``gitlab``, or ``bitbucket_oauth2``.
+  When setup, you will be given a "Client ID" (also called an "Application ID" or just "Key") and a "Secret".
+* Take the "Client ID" and "Secret" for each service and enter it in your local Django admin at:
+  ``http://community.dev.readthedocs.io/admin/socialaccount/socialapp/``.
+  Make sure to apply it to the "Site".

docs/development/standards.rst

@@ -7,36 +7,46 @@ and your code style passes our code linting suite.
 Read the Docs uses `Tox`_ to execute testing and linting procedures. Tox is the
 only dependency you need to run linting or our test suite, the remainder of our
 requirements will be installed by Tox into environment specific virtualenv
-paths. Before testing, make sure you have Tox installed::
+paths. Before testing, make sure you have Tox installed:
+
+.. prompt:: bash
 
     pip install tox
 
 To run the full test and lint suite against your changes, simply run Tox. Tox
 should return without any errors. You can run Tox against all of our
-environments by running::
+environments by running:
+
+.. prompt:: bash
 
     tox
 
 By default, tox won't run tests from search,
 in order to run all test including the search tests,
 you need to override tox's posargs.
 If you don't have any additional arguments to pass,
-you can also set the ``TOX_POSARGS`` environment variable to an empty string::
+you can also set the ``TOX_POSARGS`` environment variable to an empty string:
+
+.. prompt:: bash
 
     TOX_POSARGS='' tox
 
 .. note::
 
    If you need to override tox's posargs, but you still don't want to run the search tests,
-   you need to include ``-m 'not search'`` to your command::
+   you need to include ``-m 'not search'`` to your command:
+
+.. prompt:: bash
 
        tox -- -m 'not search' -x
 
 .. warning::
 
-   Running tests for search needs an Elasticsearch :ref:`instance running locally <development/search:Installing and running Elasticsearch>`.
+   Running tests for search needs an Elasticsearch :ref:`instance running locally <development/search:Manual Elasticsearch installation and setup>`.
+
+To target a specific environment:
 
-To target a specific environment::
+.. prompt:: bash
 
     tox -e py36