Merge branch 'master' into unify-feature-check

Author: stsewd

.circleci/config.yml

@@ -45,6 +45,7 @@ jobs:
             echo 'export PATH=~/.nvm/versions/node/v${NODE_VERSION}/bin:$PATH' >> $BASH_ENV
             source $BASH_ENV
       - run: tox -e migrations
+      - run: tox -e pre-commit
       - run: tox -e lint
       - run: tox -e docs-lint
       - run: tox -e docs

.github/CODEOWNERS

@@ -1,3 +1,3 @@
 *       @readthedocs/backend
 
-docs/*  @readthedocs/advocacy
+/docs/  @readthedocs/advocacy

.style.yapf

@@ -1,3 +1,72 @@
+Version 7.4.1
+-------------
+
+:Date: March 07, 2022
+
+* `@humitos <https://github.com/humitos>`__: Upgrade common submodule (`#9001 <https://github.com/readthedocs/readthedocs.org/pull/9001>`__)
+* `@humitos <https://github.com/humitos>`__: Build: `RepositoryError` message (`#8999 <https://github.com/readthedocs/readthedocs.org/pull/8999>`__)
+* `@humitos <https://github.com/humitos>`__: Requirements: remove `django-permissions-policy` (`#8987 <https://github.com/readthedocs/readthedocs.org/pull/8987>`__)
+* `@stsewd <https://github.com/stsewd>`__: Archive builds: avoid filtering by commands__isnull (`#8986 <https://github.com/readthedocs/readthedocs.org/pull/8986>`__)
+* `@humitos <https://github.com/humitos>`__: Build: cancel error message (`#8984 <https://github.com/readthedocs/readthedocs.org/pull/8984>`__)
+* `@humitos <https://github.com/humitos>`__: API: validate `RemoteRepository` when creating a `Project` (`#8983 <https://github.com/readthedocs/readthedocs.org/pull/8983>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: trigger `archive_builds` frequently with a lower limit (`#8981 <https://github.com/readthedocs/readthedocs.org/pull/8981>`__)
+* `@pyup-bot <https://github.com/pyup-bot>`__: pyup:  Scheduled weekly dependency update for week 09 (`#8977 <https://github.com/readthedocs/readthedocs.org/pull/8977>`__)
+* `@stsewd <https://github.com/stsewd>`__: MkDocs: allow None on extra_css/extra_javascript (`#8976 <https://github.com/readthedocs/readthedocs.org/pull/8976>`__)
+* `@stsewd <https://github.com/stsewd>`__: CDN: avoid cache tags collision (`#8969 <https://github.com/readthedocs/readthedocs.org/pull/8969>`__)
+* `@stsewd <https://github.com/stsewd>`__: Docs: warn about custom domains on subprojects (`#8945 <https://github.com/readthedocs/readthedocs.org/pull/8945>`__)
+* `@humitos <https://github.com/humitos>`__: Code style: format the code using `darker` (`#8875 <https://github.com/readthedocs/readthedocs.org/pull/8875>`__)
+* `@dogukanteber <https://github.com/dogukanteber>`__: Use django-storages' manifest files class instead of the overriden class (`#8781 <https://github.com/readthedocs/readthedocs.org/pull/8781>`__)
+* `@nienn <https://github.com/nienn>`__: Docs: Add links to documentation on creating custom classes (`#8466 <https://github.com/readthedocs/readthedocs.org/pull/8466>`__)
+* `@stsewd <https://github.com/stsewd>`__: Integrations: allow to pass more data about external versions (`#7692 <https://github.com/readthedocs/readthedocs.org/pull/7692>`__)
+
+Version 7.4.0
+-------------
+
+:Date: March 01, 2022
+
+* `@stsewd <https://github.com/stsewd>`__: Fix typo on exception (`#8975 <https://github.com/readthedocs/readthedocs.org/pull/8975>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: increase timeout limit for `sync_remote_repositories` task (`#8974 <https://github.com/readthedocs/readthedocs.org/pull/8974>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Fix a couple integration admin bugs (`#8964 <https://github.com/readthedocs/readthedocs.org/pull/8964>`__)
+* `@humitos <https://github.com/humitos>`__: Build: allow NULL when updating the config (`#8962 <https://github.com/readthedocs/readthedocs.org/pull/8962>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Release 7.3.0 (`#8957 <https://github.com/readthedocs/readthedocs.org/pull/8957>`__)
+* `@pyup-bot <https://github.com/pyup-bot>`__: pyup:  Scheduled weekly dependency update for week 08 (`#8954 <https://github.com/readthedocs/readthedocs.org/pull/8954>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: upgrade to latest version (`#8952 <https://github.com/readthedocs/readthedocs.org/pull/8952>`__)
+* `@humitos <https://github.com/humitos>`__: Requirements: upgrade gitpython because of security issue (`#8950 <https://github.com/readthedocs/readthedocs.org/pull/8950>`__)
+* `@humitos <https://github.com/humitos>`__: Test: remove verbose warnings (`#8949 <https://github.com/readthedocs/readthedocs.org/pull/8949>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Pin storages with boto3 (`#8947 <https://github.com/readthedocs/readthedocs.org/pull/8947>`__)
+* `@stsewd <https://github.com/stsewd>`__: Optimize archive_builds task (`#8944 <https://github.com/readthedocs/readthedocs.org/pull/8944>`__)
+* `@humitos <https://github.com/humitos>`__: Build: reset build error before start building (`#8943 <https://github.com/readthedocs/readthedocs.org/pull/8943>`__)
+* `@humitos <https://github.com/humitos>`__: Build: comment `record=False` usage (`#8939 <https://github.com/readthedocs/readthedocs.org/pull/8939>`__)
+* `@humitos <https://github.com/humitos>`__: Django3: use new JSON fields instead of old TextFields (`#8934 <https://github.com/readthedocs/readthedocs.org/pull/8934>`__)
+* `@humitos <https://github.com/humitos>`__: Build: ability to cancel a running build from dashboard (`#8850 <https://github.com/readthedocs/readthedocs.org/pull/8850>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: remove queue priority (`#8848 <https://github.com/readthedocs/readthedocs.org/pull/8848>`__)
+
+Version 7.3.0
+-------------
+
+:Date: February 21, 2022
+
+* `@humitos <https://github.com/humitos>`__: Requirements: upgrade gitpython because of security issue (`#8950 <https://github.com/readthedocs/readthedocs.org/pull/8950>`__)
+* `@humitos <https://github.com/humitos>`__: Test: remove verbose warnings (`#8949 <https://github.com/readthedocs/readthedocs.org/pull/8949>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Pin storages with boto3 (`#8947 <https://github.com/readthedocs/readthedocs.org/pull/8947>`__)
+* `@stsewd <https://github.com/stsewd>`__: Fix codeowners (`#8946 <https://github.com/readthedocs/readthedocs.org/pull/8946>`__)
+* `@stsewd <https://github.com/stsewd>`__: Optimize archive_builds task (`#8944 <https://github.com/readthedocs/readthedocs.org/pull/8944>`__)
+* `@humitos <https://github.com/humitos>`__: Build: reset build error before start building (`#8943 <https://github.com/readthedocs/readthedocs.org/pull/8943>`__)
+* `@humitos <https://github.com/humitos>`__: Build: comment `record=False` usage (`#8939 <https://github.com/readthedocs/readthedocs.org/pull/8939>`__)
+* `@humitos <https://github.com/humitos>`__: Django3: use new JSON fields instead of old TextFields (`#8934 <https://github.com/readthedocs/readthedocs.org/pull/8934>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Tune build config migration (`#8931 <https://github.com/readthedocs/readthedocs.org/pull/8931>`__)
+* `@humitos <https://github.com/humitos>`__: Build: use `ubuntu-20.04` image for setup VCS step (`#8930 <https://github.com/readthedocs/readthedocs.org/pull/8930>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: remove duplication of task names (`#8929 <https://github.com/readthedocs/readthedocs.org/pull/8929>`__)
+* `@humitos <https://github.com/humitos>`__: Sentry and Celery: do not log `RepositoryError` in Sentry (`#8928 <https://github.com/readthedocs/readthedocs.org/pull/8928>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Add x-hoverxref-version to CORS (`#8927 <https://github.com/readthedocs/readthedocs.org/pull/8927>`__)
+* `@humitos <https://github.com/humitos>`__: Deploy: avoid locking the table when adding new JSON field (`#8926 <https://github.com/readthedocs/readthedocs.org/pull/8926>`__)
+* `@humitos <https://github.com/humitos>`__: Build: access valid arguments (`#8925 <https://github.com/readthedocs/readthedocs.org/pull/8925>`__)
+* `@humitos <https://github.com/humitos>`__: Comment: add comment from PR review (`#8921 <https://github.com/readthedocs/readthedocs.org/pull/8921>`__)
+* `@stsewd <https://github.com/stsewd>`__: Move subscription tasks (`#8916 <https://github.com/readthedocs/readthedocs.org/pull/8916>`__)
+* `@pyup-bot <https://github.com/pyup-bot>`__: pyup:  Scheduled weekly dependency update for week 07 (`#8915 <https://github.com/readthedocs/readthedocs.org/pull/8915>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxito: add top-level docs (`#8913 <https://github.com/readthedocs/readthedocs.org/pull/8913>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: remove queue priority (`#8848 <https://github.com/readthedocs/readthedocs.org/pull/8848>`__)
+
 Version 7.2.1
 -------------
 

CHANGELOG.rst

@@ -77,3 +77,8 @@ you can include the theme's normal CSS using the CSS
     body {
         /* ... */
     }
+
+
+.. seealso::
+
+   You can also add custom classes to your html elements. See `Docutils Class <https://docutils.sourceforge.io/docs/ref/rst/directives.html#class>`_ and this related `Sphinx footnote... <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#id2>`_ for more information.

common

@@ -15,7 +15,7 @@ The documentation for ``Bar`` will be available under this same path:
 
 https://foo.readthedocs.io/projects/bar/en/latest/
 
-Adding a Subproject
+Adding a subproject
 -------------------
 
 In the admin dashboard for your project, select "Subprojects" from the menu.
@@ -24,9 +24,9 @@ From this page you can add a subproject by typing in the project slug.
 Subproject aliases
 ~~~~~~~~~~~~~~~~~~
 
-You can use an alias for the subproject when it is created. This allows you to override the URL that is used to access it, giving more configurability to how you want to structure your projects. 
+You can use an alias for the subproject when it is created. This allows you to override the URL that is used to access it, giving more configurability to how you want to structure your projects.
 
-Sharing a Custom Domain
+Sharing a custom domain
 -----------------------
 
 Projects and subprojects can also be used to share a custom domain with a number
@@ -39,6 +39,18 @@ If the example project ``Foo`` was set up with a custom domain,
 respectively be at: https://docs.example.com/en/latest/ and
 https://docs.example.com/projects/bar/en/latest/
 
+Custom domain on subprojects
+----------------------------
+
+Adding a custom domain to a subproject is allowed,
+but your documentation will always be served from
+the domain of the parent project.
+
+For example, if the domain of a parent project is ``https://docs.example.com``,
+and you add the ``https://subproject.example.com/`` domain to one of its subprojects,
+it will always redirect to the domain of the parent project
+``https://docs.example.com/projects/subproject/``.
+
 Search
 ------
 

docs/user/guides/adding-custom-css.rst

@@ -1,6 +1,6 @@
 {
   "name": "readthedocs",
-  "version": "7.1.2",
+  "version": "7.4.1",
   "description": "Read the Docs build dependencies",
   "author": "Anthony Johnson <[email protected]>",
   "scripts": {

docs/user/subprojects.rst

@@ -17,5 +17,11 @@ filterwarnings =
     ignore:Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0.*:DeprecationWarning
     # kombu
     ignore:SelectableGroups dict interface is deprecated.*:DeprecationWarning
-    ignore:RemovedInDjango40Warning
-    ignore:RemovedInDjango41Warning
+
+    # builtins
+    ignore:DateTimeField .* received a naive datetime .* while time zone support is active:RuntimeWarning
+    ignore:.*:DeprecationWarning
+
+    ignore:.*:django.utils.deprecation.RemovedInDjango40Warning
+    ignore:.*:django.utils.deprecation.RemovedInDjango41Warning
+    ignore:.*:elasticsearch.exceptions.ElasticsearchWarning

package.json

@@ -1,5 +1,7 @@
 import structlog
 
+from readthedocs.core.utils import get_cache_tag
+
 log = structlog.get_logger(__name__)
 
 
@@ -37,10 +39,10 @@ def _get_cache_tags(self):
             version = self._get_version()
             tags = [
                 project.slug,
-                f'{project.slug}-{version.slug}',
+                get_cache_tag(project.slug, version.slug),
             ]
             if self.project_cache_tag:
-                tags.append(f'{project.slug}-{self.project_cache_tag}')
+                tags.append(get_cache_tag(project.slug, self.project_cache_tag))
             return tags
         except Exception as e:
             log.debug('Error while retrieving project and version for this view.')

pytest.ini

@@ -145,7 +145,7 @@ class BuildSerializer(serializers.ModelSerializer):
     commit_url = serializers.ReadOnlyField(source='get_commit_url')
     # Jsonfield needs an explicit serializer
     # https://github.com/dmkoch/django-jsonfield/issues/188#issuecomment-300439829
-    config = serializers.JSONField(required=False)
+    config = serializers.JSONField(required=False, allow_null=True)
 
     class Meta:
         model = Build

readthedocs/api/v2/mixins.py

@@ -3,9 +3,10 @@
 import hashlib
 import hmac
 import json
-import structlog
 import re
+from functools import namedtuple
 
+import structlog
 from django.shortcuts import get_object_or_404
 from rest_framework import permissions, status
 from rest_framework.exceptions import NotFound, ParseError
@@ -14,11 +15,7 @@
 from rest_framework.status import HTTP_400_BAD_REQUEST
 from rest_framework.views import APIView
 
-from readthedocs.core.signals import (
-    webhook_bitbucket,
-    webhook_github,
-    webhook_gitlab,
-)
+from readthedocs.core.signals import webhook_bitbucket, webhook_github, webhook_gitlab
 from readthedocs.core.views.hooks import (
     build_branches,
     build_external_version,
@@ -27,7 +24,7 @@
     trigger_sync_versions,
 )
 from readthedocs.integrations.models import HttpExchange, Integration
-from readthedocs.projects.models import Feature, Project
+from readthedocs.projects.models import Project
 
 log = structlog.get_logger(__name__)
 
@@ -55,6 +52,12 @@
 BITBUCKET_PUSH = 'repo:push'
 
 
+ExternalVersionData = namedtuple(
+    "ExternalVersionData",
+    ["id", "source_branch", "base_branch", "commit"],
+)
+
+
 class WebhookMixin:
 
     """Base class for Webhook mixins."""
@@ -228,20 +231,22 @@ def get_external_version_response(self, project):
         :param project: Project instance
         :type project: readthedocs.projects.models.Project
         """
-        identifier, verbose_name = self.get_external_version_data()
+        version_data = self.get_external_version_data()
         # create or get external version object using `verbose_name`.
         external_version = get_or_create_external_version(
-            project, identifier, verbose_name
+            project=project,
+            version_data=version_data,
         )
         # returns external version verbose_name (pull/merge request number)
         to_build = build_external_version(
-            project=project, version=external_version, commit=identifier
+            project=project,
+            version=external_version,
         )
 
         return {
-            'build_triggered': True,
-            'project': project.slug,
-            'versions': [to_build],
+            "build_triggered": bool(to_build),
+            "project": project.slug,
+            "versions": [to_build] if to_build else [],
         }
 
     def get_deactivated_external_version_response(self, project):
@@ -259,9 +264,10 @@ def get_deactivated_external_version_response(self, project):
         :param project: Project instance
         :type project: Project
         """
-        identifier, verbose_name = self.get_external_version_data()
+        version_data = self.get_external_version_data()
         deactivated_version = deactivate_external_version(
-            project, identifier, verbose_name
+            project=project,
+            version_data=version_data,
         )
         return {
             'version_deactivated': bool(deactivated_version),
@@ -320,13 +326,16 @@ def get_data(self):
     def get_external_version_data(self):
         """Get Commit Sha and pull request number from payload."""
         try:
-            identifier = self.data['pull_request']['head']['sha']
-            verbose_name = str(self.data['number'])
-
-            return identifier, verbose_name
-
-        except KeyError:
-            raise ParseError('Parameters "sha" and "number" are required')
+            data = ExternalVersionData(
+                id=str(self.data["number"]),
+                commit=self.data["pull_request"]["head"]["sha"],
+                source_branch=self.data["pull_request"]["head"]["ref"],
+                base_branch=self.data["pull_request"]["base"]["ref"],
+            )
+            return data
+        except KeyError as e:
+            key = e.args[0]
+            raise ParseError(f"Invalid payload. {key} is required.")
 
     def is_payload_valid(self):
         """
@@ -431,7 +440,7 @@ def handle_webhook(self):
                 (created or deleted),
         ]):
             integration = self.get_integration()
-            events = integration.provider_data.get('events', [])
+            events = integration.provider_data.get('events', []) if integration.provider_data else []  # noqa
             if any([
                     GITHUB_CREATE in events,
                     GITHUB_DELETE in events,
@@ -523,13 +532,16 @@ def is_payload_valid(self):
     def get_external_version_data(self):
         """Get commit SHA and merge request number from payload."""
         try:
-            identifier = self.data['object_attributes']['last_commit']['id']
-            verbose_name = str(self.data['object_attributes']['iid'])
-
-            return identifier, verbose_name
-
-        except KeyError:
-            raise ParseError('Parameters "id" and "iid" are required')
+            data = ExternalVersionData(
+                id=str(self.data["object_attributes"]["iid"]),
+                commit=self.data["object_attributes"]["last_commit"]["id"],
+                source_branch=self.data["object_attributes"]["source_branch"],
+                base_branch=self.data["object_attributes"]["target_branch"],
+            )
+            return data
+        except KeyError as e:
+            key = e.args[0]
+            raise ParseError(f"Invalid payload. {key} is required.")
 
     def handle_webhook(self):
         """