Merge branch 'rel' of github.com:readthedocs/readthedocs.org into rel

Author: humitos

CHANGELOG.rst

@@ -1,3 +1,42 @@
+Version 4.1.0
+-------------
+
+:Date: March 17, 2020
+
+* `@ericholscher <https://github.com/ericholscher>`__: Properly proxy the Proxito headers via nginx/sendfile (`#6782 <https://github.com/readthedocs/readthedocs.org/pull/6782>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Revert "Add feature flag to just completely skip sync and symlink operations (#6689)" (`#6781 <https://github.com/readthedocs/readthedocs.org/pull/6781>`__)
+* `@humitos <https://github.com/humitos>`__: Upgrade django-storages to support URLs with more http methods (`#6771 <https://github.com/readthedocs/readthedocs.org/pull/6771>`__)
+* `@davidfischer <https://github.com/davidfischer>`__: Use the hotfixed version of django-messages-extends (`#6767 <https://github.com/readthedocs/readthedocs.org/pull/6767>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Release 4.0.3 (`#6766 <https://github.com/readthedocs/readthedocs.org/pull/6766>`__)
+* `@stsewd <https://github.com/stsewd>`__: Document usage or pytest marks (`#6764 <https://github.com/readthedocs/readthedocs.org/pull/6764>`__)
+* `@humitos <https://github.com/humitos>`__: Pull/Push cached environment using storage (`#6763 <https://github.com/readthedocs/readthedocs.org/pull/6763>`__)
+* `@stsewd <https://github.com/stsewd>`__: Refactor search view to make use of permission_classes (`#6761 <https://github.com/readthedocs/readthedocs.org/pull/6761>`__)
+* `@stsewd <https://github.com/stsewd>`__: Run proxito tests with proxito (`#6714 <https://github.com/readthedocs/readthedocs.org/pull/6714>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxy footer api on docs' domains (`#6630 <https://github.com/readthedocs/readthedocs.org/pull/6630>`__)
+
+Version 4.0.3
+-------------
+
+:Date: March 10, 2020
+
+* `@stsewd <https://github.com/stsewd>`__: Document usage or pytest marks (`#6764 <https://github.com/readthedocs/readthedocs.org/pull/6764>`__)
+* `@stsewd <https://github.com/stsewd>`__: Update some dependencies (`#6762 <https://github.com/readthedocs/readthedocs.org/pull/6762>`__)
+* `@stsewd <https://github.com/stsewd>`__: Refactor search view to make use of permission_classes (`#6761 <https://github.com/readthedocs/readthedocs.org/pull/6761>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Revert "Merge pull request #6739 from readthedocs/agj/docs-tos-pdf" (`#6760 <https://github.com/readthedocs/readthedocs.org/pull/6760>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Expand the logic in our proxito mixin. (`#6759 <https://github.com/readthedocs/readthedocs.org/pull/6759>`__)
+* `@comradekingu <https://github.com/comradekingu>`__: Spelling: "Set up your environment" (`#6752 <https://github.com/readthedocs/readthedocs.org/pull/6752>`__)
+* `@humitos <https://github.com/humitos>`__: Use `storage.exists` on HEAD method (`#6751 <https://github.com/readthedocs/readthedocs.org/pull/6751>`__)
+* `@humitos <https://github.com/humitos>`__: Pull only latest image for development (`#6750 <https://github.com/readthedocs/readthedocs.org/pull/6750>`__)
+* `@humitos <https://github.com/humitos>`__: Update common submodule (`#6749 <https://github.com/readthedocs/readthedocs.org/pull/6749>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Release 4.0.2 (`#6741 <https://github.com/readthedocs/readthedocs.org/pull/6741>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Add TOS PDF output (`#6739 <https://github.com/readthedocs/readthedocs.org/pull/6739>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Don't call virtualenv with `--no-site-packages` (`#6738 <https://github.com/readthedocs/readthedocs.org/pull/6738>`__)
+* `@GallowayJ <https://github.com/GallowayJ>`__: Drop mock dependency (`#6723 <https://github.com/readthedocs/readthedocs.org/pull/6723>`__)
+* `@stsewd <https://github.com/stsewd>`__: Run proxito tests with proxito (`#6714 <https://github.com/readthedocs/readthedocs.org/pull/6714>`__)
+* `@humitos <https://github.com/humitos>`__: New block on footer template to override from corporate (`#6702 <https://github.com/readthedocs/readthedocs.org/pull/6702>`__)
+* `@humitos <https://github.com/humitos>`__: Point users to support email instead asking to open an issue (`#6650 <https://github.com/readthedocs/readthedocs.org/pull/6650>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxy footer api on docs' domains (`#6630 <https://github.com/readthedocs/readthedocs.org/pull/6630>`__)
+
 Version 4.0.2
 -------------
 

common

@@ -20,8 +20,12 @@ server {
     location / {
         proxy_pass http://proxito:8000;
         proxy_set_header Host $host;
-        proxy_intercept_errors on;
-        error_page 404 = @notfoundfallback;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Host $host;
+
+        proxy_intercept_errors on;  
+        error_page 404 = @proxito404;
 
         add_header X-Served Django-Proxito always;
     }
@@ -31,17 +35,29 @@ server {
         internal;
         # Nginx will strip the `/proxito/` and pass just the `$storage/$type/$proj/$ver/$filename`
         proxy_pass http://storage:10000/;
+
         proxy_set_header Host storage:10000;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Host $host;
 
-
         proxy_intercept_errors on;
         error_page 404 = @notfoundfallback;
 
-        add_header X-Served Nginx-Proxito-Sendfile always;
-        add_header X-RTD-Project $upstream_http_x_rtd_project always;
+        # Nginx is silly, so we have to *set* these then use them.
+        # Using them directly gets the values from the storage upstream.
+        # I think the `set` must get read before doing the proxy_pass
+        # but the `add_header` is read after
+        set $rtd_project $upstream_http_x_rtd_project;
+        add_header X-RTD-Project $rtd_project always;
+        set $rtd_version $upstream_http_x_rtd_version;
+        add_header X-RTD-Version $rtd_version always;
+        set $rtd_path $upstream_http_x_rtd_path;
+        add_header X-RTD-Path $rtd_path always;
+        set $rtd_domain $upstream_http_x_rtd_domain;
+        add_header X-RTD-Domain $rtd_domain always;
+        set $rtd_method $upstream_http_x_rtd_version_method;
+        add_header X-RTD-Version-Method $rtd_method always;
     }
 
     # Serve 404 pages here

dockerfiles/nginx/proxito.conf

@@ -72,8 +72,8 @@ Search enabled by default
     Besides, all the documentation indexes are updated after a build is finished.
 
 
-Setup your environment
-----------------------
+Set up your environment
+-----------------------
 
 After cloning ``readthedocs.org`` repository, you need to
 
@@ -105,7 +105,7 @@ After cloning ``readthedocs.org`` repository, you need to
 
    .. prompt:: bash
 
-      inv docker.pull
+      inv docker.pull --only-latest
 
 #. start all the containers:
 
@@ -172,6 +172,7 @@ save some work while typing docker compose commands. This section explains these
 ``inv docker.pull``
     Downloads and tags all the Docker images required for builders.
 
+    * ``--only-latest`` does not pull ``stable`` and ``testing`` images.
 
 Adding a new Python dependency
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/development/standards.rst

@@ -59,6 +59,33 @@ docs
 .. _`pyflakes`: https://github.com/pyflakes/pyflakes
 .. _`pep8`: https://pep8.readthedocs.io/en/latest/index.html
 
+
+Pytest marks
+------------
+
+The Read the Docs code base is deployed as three instances:
+
+- Main: where you can see the dashboard.
+- Build: where the builds happen.
+- Serve/proxito: It is in charge of serving the documentation pages.
+
+Each instance has its own settings.
+To make sure we test each part as close as possible to its real settings,
+we use `pytest marks <https://docs.pytest.org/en/latest/mark.html>`__.
+This allow us to run each set of tests with different settings files,
+or skip some (like search tests)::
+
+
+  DJANGO_SETTINGS_MODULE=custom.settings.file pytest -m mark
+  DJANGO_SETTINGS_MODULE=another.settings.file pytest -m "not mark"
+
+Current marks are:
+
+- search (tests that require Elastic Search)
+- proxito (tests from the serve/proxito instance)
+
+Tests without mark are from the main instance.
+
 Continuous Integration
 ----------------------
 

docs/development/tests.rst

@@ -3,6 +3,7 @@ addopts = --reuse-db --strict-markers
 markers =
     search
     serve
+    proxito
 python_files = tests.py test_*.py *_tests.py
 filterwarnings =
     # Ignore external dependencies warning deprecations

pytest.ini

@@ -91,7 +91,7 @@ class IsAuthorizedToViewVersion(permissions.BasePermission):
     """
     Checks if the user from the request has permissions to see the version.
 
-    This permission class used in the FooterHTML view.
+    This permission class used in the FooterHTML and PageSearchAPIView views.
 
     .. note::
 

readthedocs/api/v2/permissions.py

@@ -0,0 +1,18 @@
+"""
+Proxied API URLs.
+
+Served from the same domain docs are served,
+so they can make use of features that require to have access to their cookies.
+"""
+
+from django.conf.urls import include, url
+
+from .views.proxied import ProxiedFooterHTML
+from readthedocs.search.proxied_api import ProxiedPageSearchAPIView
+
+api_footer_urls = [
+    url(r'footer_html/', ProxiedFooterHTML.as_view(), name='footer_html'),
+    url(r'docsearch/$', ProxiedPageSearchAPIView.as_view(), name='doc_search'),
+]
+
+urlpatterns = api_footer_urls

readthedocs/api/v2/proxied_urls.py

@@ -0,0 +1,15 @@
+from readthedocs.core.utils.extend import SettingsOverrideObject
+
+from .footer_views import BaseFooterHTML
+
+
+class BaseProxiedFooterHTML(BaseFooterHTML):
+
+    # DRF has BasicAuthentication and SessionAuthentication as default classes.
+    # We don't support neither in the community site.
+    authentication_classes = []
+
+
+class ProxiedFooterHTML(SettingsOverrideObject):
+
+    _default_class = BaseProxiedFooterHTML

readthedocs/api/v2/views/proxied.py

@@ -8,6 +8,7 @@
 
 import regex
 from django.conf import settings
+from django.core.files.storage import get_storage_class
 from django.db import models
 from django.db.models import F
 from django.urls import reverse
@@ -318,37 +319,33 @@ def save(self, *args, **kwargs):  # pylint: disable=arguments-differ
         """Add permissions to the Version for all owners on save."""
         from readthedocs.projects import tasks
         obj = super().save(*args, **kwargs)
-        if not self.project.has_feature(feature_id='skip_sync'):
-            broadcast(
-                type='app',
-                task=tasks.symlink_project,
-                args=[self.project.pk],
-            )
+        broadcast(
+            type='app',
+            task=tasks.symlink_project,
+            args=[self.project.pk],
+        )
         return obj
 
     def delete(self, *args, **kwargs):  # pylint: disable=arguments-differ
         from readthedocs.projects import tasks
         log.info('Removing files for version %s', self.slug)
-        has_skip_sync = self.project.has_feature(feature_id='skip_sync')
-        if not has_skip_sync:
-            broadcast(
-                type='app',
-                task=tasks.remove_dirs,
-                args=[self.get_artifact_paths()],
-            )
+        broadcast(
+            type='app',
+            task=tasks.remove_dirs,
+            args=[self.get_artifact_paths()],
+        )
 
         # Remove resources if the version is not external
         if self.type != EXTERNAL:
             tasks.clean_project_resources(self.project, self)
 
         project_pk = self.project.pk
         super().delete(*args, **kwargs)
-        if not has_skip_sync:
-            broadcast(
-                type='app',
-                task=tasks.symlink_project,
-                args=[project_pk],
-            )
+        broadcast(
+            type='app',
+            task=tasks.symlink_project,
+            args=[project_pk],
+        )
 
     @property
     def identifier_friendly(self):
@@ -451,6 +448,11 @@ def get_storage_paths(self):
 
         return paths
 
+    def get_storage_environment_cache_path(self):
+        """Return the path of the cached environment tar file."""
+        storage = get_storage_class(settings.RTD_BUILD_ENVIRONMENT_STORAGE)()
+        return storage.join(self.project.slug, f'{self.slug}.tar')
+
     def clean_build_path(self):
         """
         Clean build path for project version.

readthedocs/builds/models.py

@@ -186,3 +186,16 @@ def listdir(self, path):
         if not self.exists(path):
             return [], []
         return super().listdir(path)
+
+    def url(self, name, *args, **kwargs):  # noqa
+        """
+        Override to accept extra arguments and ignore them all.
+
+        This method helps us to bring compatibility between Azure Blob Storage
+        (which does not use the HTTP method) and Amazon S3 (who requires HTTP
+        method to build the signed URL).
+
+        ``FileSystemStorage`` does not support any other argument than ``name``.
+        https://docs.djangoproject.com/en/2.2/ref/files/storage/#django.core.files.storage.Storage.url
+        """
+        return super().url(name)

readthedocs/builds/storage.py

@@ -2,6 +2,8 @@
 
 import os
 
+from django.conf import settings
+from django.core.files.storage import get_storage_class
 from django.shortcuts import get_object_or_404
 
 from readthedocs.core.utils import broadcast
@@ -24,3 +26,7 @@ def wipe_version_via_slugs(version_slug, project_slug):
     ]
     for del_dir in del_dirs:
         broadcast(type='build', task=remove_dirs, args=[(del_dir,)])
+
+    # Delete the cache environment from storage
+    storage = get_storage_class(settings.RTD_BUILD_ENVIRONMENT_STORAGE)()
+    storage.delete(version.get_storage_environment_cache_path())

readthedocs/core/utils/general.py

@@ -129,12 +129,15 @@ def _pip_cache_cmd_argument(self):
 
         The decision is made considering if the directories are going to be
         cleaned after the build (``RTD_CLEAN_AFTER_BUILD=True`` or project has
-        the ``CLEAN_AFTER_BUILD`` feature enabled). In this case, there is no
-        need to cache anything.
+        the ``CLEAN_AFTER_BUILD`` feature enabled) and project has not the
+        feature ``CACHED_ENVIRONMENT``. In this case, there is no need to cache
+        anything.
         """
         if (
-            settings.RTD_CLEAN_AFTER_BUILD or
-            self.project.has_feature(Feature.CLEAN_AFTER_BUILD)
+            (
+                settings.RTD_CLEAN_AFTER_BUILD or
+                self.project.has_feature(Feature.CLEAN_AFTER_BUILD)
+            ) and not self.project.has_feature(Feature.CACHED_ENVIRONMENT)
         ):
             return [
                 '--no-cache-dir',

readthedocs/doc_builder/python_environments.py

@@ -1514,7 +1514,7 @@ def add_features(sender, **kwargs):
     ALL_VERSIONS_IN_HTML_CONTEXT = 'all_versions_in_html_context'
     SKIP_SYNC_TAGS = 'skip_sync_tags'
     SKIP_SYNC_BRANCHES = 'skip_sync_branches'
-    SKIP_SYNC = 'skip_sync'
+    CACHED_ENVIRONMENT = 'cached_environment'
 
     FEATURES = (
         (USE_SPHINX_LATEST, _('Use latest version of Sphinx')),
@@ -1573,17 +1573,17 @@ def add_features(sender, **kwargs):
                 'when building with Sphinx'
             ),
         ),
-        (
-            SKIP_SYNC_TAGS,
-            _('Skip syncing tags'),
-        ),
         (
             SKIP_SYNC_BRANCHES,
             _('Skip syncing branches'),
         ),
         (
-            SKIP_SYNC,
-            _('Skip symlinking and file syncing to webs'),
+            SKIP_SYNC_TAGS,
+            _('Skip syncing tags'),
+        ),
+        (
+            CACHED_ENVIRONMENT,
+            _('Cache the environment (virtualenv, conda, pip cache, repository) in storage'),
         ),
     )