Merge tag '7.2.1' into rel

Author: humitos

.readthedocs.yml

@@ -19,7 +19,7 @@ search:
   ranking:
     # Deprecated content
     api/v1.html: -1
-    config-file/v1.html: -1
+    config-file/v1.html: -5
 
     # Useful content, but not something we want most users finding
     changelog.html: -6

CHANGELOG.rst

@@ -1,3 +1,29 @@
+Version 7.2.1
+-------------
+
+:Date: February 15, 2022
+
+* `@humitos <https://github.com/humitos>`__: Sentry: ignore logging known exceptions (`#8919 <https://github.com/readthedocs/readthedocs.org/pull/8919>`__)
+* `@humitos <https://github.com/humitos>`__: Build: do not send notifications on known failed builds (`#8918 <https://github.com/readthedocs/readthedocs.org/pull/8918>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: use `on_retry` to handle `BuildMaxConcurrencyError` (`#8917 <https://github.com/readthedocs/readthedocs.org/pull/8917>`__)
+* `@lkeegan <https://github.com/lkeegan>`__: typo fix (`#8911 <https://github.com/readthedocs/readthedocs.org/pull/8911>`__)
+* `@stsewd <https://github.com/stsewd>`__: Fix tests (`#8908 <https://github.com/readthedocs/readthedocs.org/pull/8908>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Throw an exception from Celery retry() (`#8905 <https://github.com/readthedocs/readthedocs.org/pull/8905>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Reduce verbose logging on generic command failure (`#8904 <https://github.com/readthedocs/readthedocs.org/pull/8904>`__)
+* `@humitos <https://github.com/humitos>`__: Build: define missing variable (`#8903 <https://github.com/readthedocs/readthedocs.org/pull/8903>`__)
+* `@humitos <https://github.com/humitos>`__: Search: call apply_async to fix the issue (`#8900 <https://github.com/readthedocs/readthedocs.org/pull/8900>`__)
+* `@humitos <https://github.com/humitos>`__: Build: allow to not record commands on sync_repository_task (`#8899 <https://github.com/readthedocs/readthedocs.org/pull/8899>`__)
+* `@humitos <https://github.com/humitos>`__: Release 7.2.0 (`#8898 <https://github.com/readthedocs/readthedocs.org/pull/8898>`__)
+* `@stsewd <https://github.com/stsewd>`__: Fix linter (`#8897 <https://github.com/readthedocs/readthedocs.org/pull/8897>`__)
+* `@stsewd <https://github.com/stsewd>`__: Support for CDN when privacy levels are enabled (`#8896 <https://github.com/readthedocs/readthedocs.org/pull/8896>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Don't be so excited always in our emails :) (`#8888 <https://github.com/readthedocs/readthedocs.org/pull/8888>`__)
+* `@humitos <https://github.com/humitos>`__: Docs: reduce visibility of Config File V1 (`#8887 <https://github.com/readthedocs/readthedocs.org/pull/8887>`__)
+* `@humitos <https://github.com/humitos>`__: Builder: unpin pip (`#8886 <https://github.com/readthedocs/readthedocs.org/pull/8886>`__)
+* `@stsewd <https://github.com/stsewd>`__: Dependencies: remove unused packages (`#8881 <https://github.com/readthedocs/readthedocs.org/pull/8881>`__)
+* `@stsewd <https://github.com/stsewd>`__: Subscriptions: move some querysets (`#8876 <https://github.com/readthedocs/readthedocs.org/pull/8876>`__)
+* `@humitos <https://github.com/humitos>`__: Django3: delete old JSONField and use the new ones (`#8869 <https://github.com/readthedocs/readthedocs.org/pull/8869>`__)
+* `@humitos <https://github.com/humitos>`__: Django3: add new `django.db.models.JSONField` (`#8868 <https://github.com/readthedocs/readthedocs.org/pull/8868>`__)
+
 Version 7.2.0
 -------------
 

docs/user/config-file/index.rst

@@ -25,4 +25,3 @@ The main advantages of using a configuration file over the web interface are:
     :maxdepth: 3
 
     Version 2 <v2>
-    Version 1 <v1>

docs/user/config-file/v1.rst

@@ -1,3 +1,5 @@
+:orphan:
+
 Configuration File V1 (Deprecated)
 ==================================
 
@@ -6,7 +8,7 @@ Read the Docs has support for configuring builds with a YAML file.
 
 .. warning::
 
-   Version 1 shouldn't be used.
+   Version 1 is deprecated and shouldn't be used.
    The version 2 of the configuration file is now available.
    See the :ref:`new features <config-file/v2:New settings>`
    and :ref:`how to migrate from v1 <config-file/v2:Migrating from v1>`.

docs/user/config-file/v2.rst

@@ -747,8 +747,8 @@ the ``apt_packages`` key described above.
 
 .. warning::
 
-   When using the new speficiation,
-   the ``build.os`` and ``python.version`` options cannot be used.
+   When using the new specification,
+   the ``build.image`` and ``python.version`` options cannot be used.
    Doing so will error the build.
 
 build (legacy)

readthedocs/builds/migrations/0038_add_new_jsonfields.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.2.11 on 2022-01-31 11:47
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('builds', '0037_alter_build_cold_storage'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='build',
+            name='_config_json',
+            field=models.JSONField(default=dict, verbose_name='Configuration used in the build'),
+        ),
+    ]

readthedocs/builds/migrations/0039_migrate_config_data.py

@@ -0,0 +1,24 @@
+from django.db import migrations, models
+from django.db.models import F
+from django.db.models.functions import Cast
+
+
+def forwards_func(apps, schema_editor):
+    Build = apps.get_model('builds', 'Build')
+    (
+        Build.objects
+        .annotate(_config_in_json=Cast('_config', output_field=models.JSONField()))
+        # Copy `_config` JSONField (text) into `_config_json` (jsonb)
+        .update(_config_json=F('_config_in_json'))
+    )
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('builds', '0038_add_new_jsonfields'),
+    ]
+
+    operations = [
+        migrations.RunPython(forwards_func)
+    ]

readthedocs/builds/migrations/0040_remove_old_jsonfields.py

@@ -0,0 +1,22 @@
+# Generated by Django 3.2.11 on 2022-01-31 12:12
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('builds', '0039_migrate_config_data'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='build',
+            name='_config',
+        ),
+        migrations.RenameField(
+            model_name='build',
+            old_name='_config_json',
+            new_name='_config',
+        ),
+    ]

readthedocs/builds/models.py

@@ -1,13 +1,12 @@
 """Models for the builds app."""
 
 import datetime
-import structlog
 import os.path
 import re
 from functools import partial
-from shutil import rmtree
 
 import regex
+import structlog
 from django.conf import settings
 from django.db import models
 from django.db.models import F
@@ -20,7 +19,6 @@
     ModificationDateTimeField,
 )
 from django_extensions.db.models import TimeStampedModel
-from jsonfield import JSONField
 from polymorphic.models import PolymorphicModel
 
 import readthedocs.builds.automation_actions as actions
@@ -80,6 +78,7 @@
     GITLAB_URL,
     MEDIA_TYPES,
     PRIVACY_CHOICES,
+    PRIVATE,
     SPHINX,
     SPHINX_HTMLDIR,
     SPHINX_SINGLEHTML,
@@ -196,6 +195,18 @@ def __str__(self):
             ),
         )
 
+    @property
+    def is_private(self):
+        """
+        Check if the version is private (taking external versions into consideration).
+
+        The privacy level of an external version is given by
+        the value of ``project.external_builds_privacy_level``.
+        """
+        if self.is_external:
+            return self.project.external_builds_privacy_level == PRIVATE
+        return self.privacy_level == PRIVATE
+
     @property
     def is_external(self):
         return self.type == EXTERNAL
@@ -650,7 +661,7 @@ class Build(models.Model):
         null=True,
         blank=True,
     )
-    _config = JSONField(_('Configuration used in the build'), default=dict)
+    _config = models.JSONField(_('Configuration used in the build'), default=dict)
 
     length = models.IntegerField(_('Build Length'), null=True, blank=True)
 
@@ -719,6 +730,10 @@ def config(self):
         Build object (it could be stored in this object or one of the previous
         ones).
         """
+        # TODO: now that we are using a proper JSONField here, we could
+        # probably change this field to be a ForeignKey to avoid repeating the
+        # config file over and over again and re-use them to save db data as
+        # well
         if self.CONFIG_KEY in self._config:
             return (
                 Build.objects
@@ -762,6 +777,11 @@ def save(self, *args, **kwargs):  # noqa
             self.version_name = self.version.verbose_name
             self.version_slug = self.version.slug
             self.version_type = self.version.type
+
+        # TODO: delete copying config after deploy
+        # Copy `_config` into the new `_config_json` JSONField
+        self._config_json = self._config
+
         super().save(*args, **kwargs)
         self._config_changed = False
 

readthedocs/core/mixins.py

@@ -1,8 +1,12 @@
 """Common mixin classes for views."""
+from functools import lru_cache
 
+from django.conf import settings
 from django.contrib.auth.mixins import LoginRequiredMixin
 from vanilla import ListView
 
+from readthedocs.projects.models import Feature
+
 
 class ListViewWithForm(ListView):
 
@@ -24,3 +28,42 @@ class ProxiedAPIMixin:
     # DRF has BasicAuthentication and SessionAuthentication as default classes.
     # We don't support neither in the community site.
     authentication_classes = []
+
+
+class CachedView:
+
+    """
+    Allow to cache views at the CDN level when privacy levels are enabled.
+
+    The cache control header is only used when privacy levels
+    are enabled (otherwise everything is public by default).
+
+    Views that can be cached should always return the same response for all
+    users (anonymous and authenticated users), like when the version attached
+    to the request is public.
+
+    To cache a view you can either set the `cache_request` attribute to `True`,
+    or override the `can_be_cached` method.
+
+    We use ``CDN-Cache-Control``, to control caching at the CDN level only.
+    This doesn't affect caching at the browser level (``Cache-Control``).
+
+    See https://developers.cloudflare.com/cache/about/cdn-cache-control.
+    """
+
+    cache_request = False
+
+    def dispatch(self, request, *args, **kwargs):
+        response = super().dispatch(request, *args, **kwargs)
+        if settings.ALLOW_PRIVATE_REPOS and self.can_be_cached(request):
+            response.headers['CDN-Cache-Control'] = 'public'
+        return response
+
+    def can_be_cached(self, request):
+        return self.cache_request
+
+    @lru_cache(maxsize=1)
+    def _is_cache_enabled(self, project):
+        """Helper function to check if CND is enabled for a project."""
+        # TODO: check for the organization's plan.
+        return settings.ALLOW_PRIVATE_REPOS and project.has_feature(Feature.CDN_ENABLED)

readthedocs/core/utils/__init__.py

@@ -186,8 +186,8 @@ def prepare_build(
                 project_slug=project.slug,
                 version_slug=version.slug,
             )
-            options['countdown'] = 5 * 60
-            options['max_retries'] = 25
+            options['countdown'] = settings.RTD_BUILDS_RETRY_DELAY
+            options['max_retries'] = settings.RTD_BUILDS_MAX_RETRIES
             build.error = BuildMaxConcurrencyError.message.format(
                 limit=max_concurrent_builds,
             )

readthedocs/doc_builder/environments.py

@@ -444,24 +444,21 @@ def run_command_class(
             self.commands.append(build_cmd)
 
         if build_cmd.failed:
-            msg = 'Command {cmd} failed'.format(cmd=build_cmd.get_command())
-
-            # TODO: improve this error report. This is showing _the full_
-            # stdout to the user exposing it at the top of the Build Detail's
-            # page in red. It would be good to reduce the noise here and just
-            # point the user to take a look at its output from the command's
-            # output itself.
-            if build_cmd.output:
-                msg += ':\n{out}'.format(out=build_cmd.output)
-
             if warn_only:
+                msg = 'Command {cmd} failed'.format(cmd=build_cmd.get_command())
+                if build_cmd.output:
+                    msg += ':\n{out}'.format(out=build_cmd.output)
                 log.warning(
                     msg,
                     project_slug=self.project.slug if self.project else '',
                     version_slug=self.version.slug if self.version else '',
                 )
             else:
-                raise BuildUserError(msg)
+                # TODO: for now, this still outputs a generic error message
+                # that is the same across all commands. We could improve this
+                # with more granular error messages that vary by the command
+                # being run.
+                raise BuildUserError()
         return build_cmd
 
 

readthedocs/doc_builder/python_environments.py

@@ -297,13 +297,7 @@ def install_core_requirements(self):
             Feature.DONT_INSTALL_LATEST_PIP,
             # 20.3 uses the new resolver by default.
             positive='pip<20.3',
-
-            # We are pinning pip to 21.3.1 because builds are failing when
-            # using a newer version. This is a temporal workaround to avoid
-            # builds failing at this step, but we should come back to this and
-            # unpin pip for this case.
-            # https://github.com/readthedocs/readthedocs.org/issues/8864#issuecomment-1025499598
-            negative='pip<=21.3.1',
+            negative='pip',
         )
         cmd = pip_install_cmd + [pip_version, 'setuptools<58.3.0']
         self.build_env.run(

readthedocs/integrations/admin.py

@@ -93,6 +93,7 @@ class IntegrationAdmin(admin.ModelAdmin):
     search_fields = ('project__slug', 'project__name')
     readonly_fields = ['exchanges']
 
+    # TODO: review this now that we are using official Django's JSONField
     def exchanges(self, obj):
         """
         Manually make an inline-ish block.

readthedocs/integrations/migrations/0008_add_new_jsonfields.py

@@ -0,0 +1,28 @@
+# Generated by Django 3.2.11 on 2022-01-31 11:47
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('integrations', '0007_update-provider-data'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='httpexchange',
+            name='request_headers_json',
+            field=models.JSONField(default=None, null=True, verbose_name='Request headers'),
+        ),
+        migrations.AddField(
+            model_name='httpexchange',
+            name='response_headers_json',
+            field=models.JSONField(default=None, null=True, verbose_name='Request headers'),
+        ),
+        migrations.AddField(
+            model_name='integration',
+            name='provider_data_json',
+            field=models.JSONField(default=dict, verbose_name='Provider data'),
+        ),
+    ]