Merge tag '9.8.0' into rel

Author: humitos

CHANGELOG.rst

@@ -1,6 +1,40 @@
+Version 9.8.0
+-------------
+
+:Date: March 07, 2023
+
+* `@humitos <https://github.com/humitos>`__: Downloadable artifacts: make PDF and ePub opt-in by default (`#10115 <https://github.com/readthedocs/readthedocs.org/pull/10115>`__)
+* `@humitos <https://github.com/humitos>`__: Proxito: cacheops right model (`#10111 <https://github.com/readthedocs/readthedocs.org/pull/10111>`__)
+* `@stsewd <https://github.com/stsewd>`__: Docs: fix term reference (`#10110 <https://github.com/readthedocs/readthedocs.org/pull/10110>`__)
+* `@humitos <https://github.com/humitos>`__: Development: allow to define the logging level via an env variable (`#10109 <https://github.com/readthedocs/readthedocs.org/pull/10109>`__)
+* `@humitos <https://github.com/humitos>`__: Celery: cheat `job_status` view to return `finished` after 5 polls (`#10107 <https://github.com/readthedocs/readthedocs.org/pull/10107>`__)
+* `@humitos <https://github.com/humitos>`__: Proxito: use cacheops for 2 more models (`#10106 <https://github.com/readthedocs/readthedocs.org/pull/10106>`__)
+* `@github-actions[bot] <https://github.com/github-actions[bot]>`__: Dependencies: all packages updated via pip-tools (`#10104 <https://github.com/readthedocs/readthedocs.org/pull/10104>`__)
+* `@stsewd <https://github.com/stsewd>`__: Remove unused tests (`#10099 <https://github.com/readthedocs/readthedocs.org/pull/10099>`__)
+* `@stsewd <https://github.com/stsewd>`__: Canonical redirects: check if the project supports custom domains (`#10098 <https://github.com/readthedocs/readthedocs.org/pull/10098>`__)
+* `@stsewd <https://github.com/stsewd>`__: Fix .com tests (`#10097 <https://github.com/readthedocs/readthedocs.org/pull/10097>`__)
+* `@stsewd <https://github.com/stsewd>`__: Fix migration (`#10096 <https://github.com/readthedocs/readthedocs.org/pull/10096>`__)
+* `@benjaoming <https://github.com/benjaoming>`__: Docs: Move a reference and remove an empty paranthesis (`#10093 <https://github.com/readthedocs/readthedocs.org/pull/10093>`__)
+* `@benjaoming <https://github.com/benjaoming>`__: Docs: Update documentation for search.ignore (`#10091 <https://github.com/readthedocs/readthedocs.org/pull/10091>`__)
+* `@benjaoming <https://github.com/benjaoming>`__: Fix intersphinx references to myst-parser (updated in myst-parser 0.19) (`#10090 <https://github.com/readthedocs/readthedocs.org/pull/10090>`__)
+* `@stsewd <https://github.com/stsewd>`__: Update changelog with security fixes (`#10088 <https://github.com/readthedocs/readthedocs.org/pull/10088>`__)
+* `@humitos <https://github.com/humitos>`__: Analytics: add Plausible to our dashboard (`#10087 <https://github.com/readthedocs/readthedocs.org/pull/10087>`__)
+* `@humitos <https://github.com/humitos>`__: Docs: add Plausible (`#10086 <https://github.com/readthedocs/readthedocs.org/pull/10086>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxito: refactor canonical redirects (`#10069 <https://github.com/readthedocs/readthedocs.org/pull/10069>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxito: simplify caching logic (`#10067 <https://github.com/readthedocs/readthedocs.org/pull/10067>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Add X-Content-Type-Options as a custom domain header (`#10062 <https://github.com/readthedocs/readthedocs.org/pull/10062>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxito V2 (`#10044 <https://github.com/readthedocs/readthedocs.org/pull/10044>`__)
+* `@stsewd <https://github.com/stsewd>`__: Proxito: adapt unresolver to make it usable for proxito (`#10037 <https://github.com/readthedocs/readthedocs.org/pull/10037>`__)
+* `@agjohnson <https://github.com/agjohnson>`__: Add beta version of doc diff library for testing (`#9546 <https://github.com/readthedocs/readthedocs.org/pull/9546>`__)
+* `@davidfischer <https://github.com/davidfischer>`__: Support the new Google analytics gtag.js (`#7691 <https://github.com/readthedocs/readthedocs.org/pull/7691>`__)
+
 Version 9.7.0
 -------------
 
+**This release contains one security fix. For more information, see:**
+
+- `GHSA-h4cf-8gv8-4chf <https://github.com/readthedocs/readthedocs.org/security/advisories/GHSA-h4cf-8gv8-4chf>`__
+
 :Date: February 28, 2023
 
 * `@humitos <https://github.com/humitos>`__: Celery: delete Telemetry data that's at most 3 months older (`#10079 <https://github.com/readthedocs/readthedocs.org/pull/10079>`__)
@@ -49,6 +83,10 @@ Version 9.6.0
 Version 9.5.0
 -------------
 
+**This release contains one security fix. For more information, see:**
+
+- `GHSA-mp38-vprc-7hf5 <https://github.com/readthedocs/readthedocs.org/security/advisories/GHSA-mp38-vprc-7hf5>`__
+
 :Date: February 13, 2023
 
 * `@agjohnson <https://github.com/agjohnson>`__: Bump to latest common (`#10019 <https://github.com/readthedocs/readthedocs.org/pull/10019>`__)
@@ -81,6 +119,10 @@ Version 9.5.0
 Version 9.4.0
 -------------
 
+**This release contains one security fix. For more information, see:**
+
+- `GHSA-5w8m-r7jm-mhp9 <https://github.com/readthedocs/readthedocs.org/security/advisories/GHSA-5w8m-r7jm-mhp9>`__
+
 :Date: February 07, 2023
 
 * `@agjohnson <https://github.com/agjohnson>`__: Fix ordering of filter for most recently built project (`#9992 <https://github.com/readthedocs/readthedocs.org/pull/9992>`__)

dockerfiles/nginx/proxito.conf.template

@@ -82,6 +82,8 @@ server {
         add_header Access-Control-Allow-Headers $access_control_allow_headers always;
         set $x_frame_options $upstream_http_x_frame_options;
         add_header X-Frame-Options $x_frame_options always;
+        set $x_content_type_options $upstream_http_x_content_type_options;
+        add_header X-Content-Type-Options $x_content_type_options always;
         # Minio sets this header on the response, and we don't want to copy it to the response
         proxy_hide_header Content-Security-Policy;
         set $content_security_policy $upstream_http_content_security_policy;

docs/_static/js/readthedocs-doc-diff.js

@@ -1,5 +1,12 @@
 {% extends "!layout.html" %}
 
+{% block extrahead %}
+  {{ super() }}
+
+  {% if READTHEDOCS %}
+  <script defer data-domain="{{ plausible_domain }}" src="https://plausible.io/js/script.js"></script>
+  {% endif %}
+{% endblock extrahead %}
 
 {% block document %}
   {{ super() }}

docs/_templates/layout.html

@@ -67,7 +67,7 @@
 
 master_doc = "index"
 copyright = "Read the Docs, Inc & contributors"
-version = "9.7.0"
+version = "9.8.0"
 release = version
 exclude_patterns = ["_build", "shared", "_includes"]
 default_role = "obj"
@@ -150,6 +150,9 @@
 html_css_files = ["css/custom.css", "css/sphinx_prompt_css.css"]
 html_js_files = ["js/expand_tabs.js"]
 
+if os.environ.get("READTHEDOCS_VERSION_TYPE") == "external":
+    html_js_files.append("js/readthedocs-doc-diff.js")
+
 html_logo = "img/logo.svg"
 html_theme_options = {
     "logo_only": True,
@@ -160,6 +163,8 @@
     # TODO: remove once we support different rtd config
     # files per project.
     "conf_py_path": f"/docs/{docset}/",
+    # Use to generate the Plausible "data-domain" attribute from the template
+    "plausible_domain": f"{os.environ.get('READTHEDOCS_PROJECT')}.readthedocs.io",
 }
 
 hoverxref_auto_ref = True

docs/conf.py

@@ -61,7 +61,7 @@ Good practice ✅
   keep original file names rather than going for low-impact URL renaming.
   Renaming an article's title is great for the reader and great for SEO,
   but this does not have to involve the URL.
-* Establish your understanding of the *latest* and *:term:`default version`* of your documentation at the beginning. Changing their meaning is very disruptive to incoming links.
+* Establish your understanding of the *latest* and :term:`default version` of your documentation at the beginning. Changing their meaning is very disruptive to incoming links.
 * Keep development versions :ref:`hidden <versions:Hidden>` so people do not find them on search engines by mistake.
   This is the best way to ensure that nobody links to URLs that are intended for development purposes.
 * Use a :ref:`version warning <versions:Version warning>` to ensure the reader is aware in case they are reading an old (archived) version.

docs/user/automatic-redirects.rst

@@ -751,13 +751,13 @@ check :ref:`config-file/v2:search.ignore`.
 search.ignore
 `````````````
 
-Don't index files matching a pattern.
-This is, you won't see search results from these files.
+List of paths to ignore and exclude from the search index.
+Paths matched will not be included in search results.
 
 :Type: ``list`` of patterns
 :Default: ``['search.html', 'search/index.html', '404.html', '404/index.html']``
 
-Patterns are matched against the final html pages produced by the build
+Patterns are matched against the relative path of html files produced by the build
 (you should try to match `index.html`, not `docs/index.rst`).
 Patterns can include some special characters:
 
@@ -771,13 +771,13 @@ Patterns can include some special characters:
 
    search:
       ignore:
-        # Ignore a single file
+        # Ignore a single file in the root of the output directory
         - 404.html
 
         # Ignore all files under the search/ directory
         - search/*
 
-        # Ignore all files that end with ref.html
+        # Ignore all files named ref.html nested inside one or more sub-folders
         - '*/ref.html'
 
 .. code-block:: yaml

docs/user/config-file/v2.rst

@@ -29,7 +29,7 @@ with two markup options: reStructuredText and MyST (Markdown).
 - If you are not familiar with reStructuredText,
   check :doc:`sphinx:usage/restructuredtext/basics` for a quick introduction.
 - If you want to learn more about the MyST Markdown dialect,
-  check out :doc:`myst-parser:syntax/syntax`.
+  check out :doc:`myst-parser:syntax/reference`.
 
 .. contents:: Table of contents
    :local:

docs/user/guides/cross-referencing-with-sphinx.rst

@@ -331,7 +331,7 @@ However, there are some differences between them:
   whereas MyST-NB uses `MyST-Parser`_
   to directly convert the Markdown text to docutils AST.
   Therefore, nbsphinx assumes `pandoc flavored Markdown <https://pandoc.org/MANUAL.html#pandocs-markdown>`_,
-  whereas MyST-NB uses :doc:`MyST flavored Markdown <myst-parser:syntax/syntax>`.
+  whereas MyST-NB uses :doc:`MyST flavored Markdown <myst-parser:index>`.
   Both Markdown flavors are mostly equal,
   but they have some differences.
 - nbsphinx executes each notebook during the parsing phase,

docs/user/guides/jupyter.rst

@@ -27,8 +27,8 @@ and some others in reStructuredText, for whatever reason.
 Luckily, Sphinx supports reading both types of markup at the same time without problems.
 
 To start using MyST in your existing Sphinx project,
-first {ref}``install the `myst-parser` Python package <myst-parser:intro.md#installation>``
-and then {ref}`enable it on your configuration <myst-parser:intro.md#enable-myst-in-sphinx>`:
+first [install the `myst-parser` Python package](https://myst-parser.readthedocs.io/en/stable/intro.html#installation)
+and then [enable it on your configuration](https://myst-parser.readthedocs.io/en/stable/intro.html#enable-myst-in-sphinx):
 
 ```{code-block} py
 :caption: conf.py

docs/user/guides/migrate-rest-myst.md

@@ -91,8 +91,8 @@ You can find out more about our all the :doc:`features </reference/features>` in
 First steps
 -----------
 
-Are you new to software documentation
-or are you looking to use your existing docs with Read the Docs?
+Are you new to software documentation?
+Are you looking to use your existing docs with Read the Docs?
 Learn about documentation authoring tools such as Sphinx and MkDocs
 to help you create fantastic documentation for your project.
 

docs/user/index.rst

@@ -1,6 +1,3 @@
-.. old label
-.. _User-defined Redirects:
-
 Custom and built-in redirects on Read the Docs
 ==============================================
 
@@ -16,7 +13,7 @@ the bad user experience of a 404 page is usually best to avoid.
     Allows for simple and long-term sharing of external references to your documentation.
 
 `User-defined redirects`_ ⬇️
-    Makes it easier to move contents around (see: )
+    Makes it easier to move contents around
 
 .. seealso::
 
@@ -95,6 +92,8 @@ They are intended to be easy and short for people to type.
 
 You can reach these docs at https://docs.rtfd.io.
 
+.. old label
+.. _User-defined Redirects:
 
 User-defined redirects
 ----------------------

docs/user/user-defined-redirects.rst

@@ -1,6 +1,6 @@
 {
   "name": "readthedocs",
-  "version": "9.7.0",
+  "version": "9.8.0",
   "description": "Read the Docs build dependencies",
   "author": "Read the Docs, Inc <[email protected]>",
   "scripts": {

package.json

@@ -1,4 +1,4 @@
 """Read the Docs."""
 
 
-__version__ = "9.7.0"
+__version__ = "9.8.0"

readthedocs/__init__.py

@@ -8,12 +8,13 @@
 
 from readthedocs.analytics.models import PageView
 from readthedocs.api.v2.permissions import IsAuthorizedToViewVersion
+from readthedocs.core.mixins import CDNCacheControlMixin
 from readthedocs.core.unresolver import UnresolverError, unresolve
 from readthedocs.core.utils.extend import SettingsOverrideObject
 from readthedocs.projects.models import Project
 
 
-class BaseAnalyticsView(APIView):
+class BaseAnalyticsView(CDNCacheControlMixin, APIView):
 
     """
     Track page views.
@@ -25,6 +26,9 @@ class BaseAnalyticsView(APIView):
     - absolute_uri: Full path with domain.
     """
 
+    # We always want to hit our analytics endpoint,
+    # so we capture all views/interactions.
+    cache_response = False
     http_method_names = ['get']
     permission_classes = [IsAuthorizedToViewVersion]
 

readthedocs/analytics/proxied_api.py

@@ -128,6 +128,11 @@ def test_invalid_uri(self):
         self.client.get(url, HTTP_HOST=self.host)
         assert PageView.objects.all().count() == 0
 
+    def test_cache_headers(self):
+        resp = self.client.get(self.url, HTTP_HOST=self.host)
+        self.assertEqual(resp.status_code, 200)
+        self.assertEqual(resp["CDN-Cache-Control"], "private")
+
     def test_increase_page_view_count(self):
         assert (
             PageView.objects.all().count() == 0

readthedocs/analytics/tests.py

@@ -1,61 +1,45 @@
 """Endpoints relating to task/job status, etc."""
 
 import structlog
-
+from django.core.cache import cache
 from django.urls import reverse
-from redis import ConnectionError
 from rest_framework import decorators, permissions
 from rest_framework.renderers import JSONRenderer
 from rest_framework.response import Response
 
-from readthedocs.core.utils.tasks import TaskNoPermission, get_public_task_data
 from readthedocs.oauth import tasks
 
-
 log = structlog.get_logger(__name__)
 
-SUCCESS_STATES = ('SUCCESS',)
-FAILURE_STATES = (
-    'FAILURE',
-    'REVOKED',
-)
-FINISHED_STATES = SUCCESS_STATES + FAILURE_STATES
-STARTED_STATES = ('RECEIVED', 'STARTED', 'RETRY') + FINISHED_STATES
-
-
-def get_status_data(task_name, state, data, error=None):
-    data = {
-        'name': task_name,
-        'data': data,
-        'started': state in STARTED_STATES,
-        'finished': state in FINISHED_STATES,
-        # When an exception is raised inside the task, we keep this as SUCCESS
-        # and add the exception message into the 'error' key
-        'success': state in SUCCESS_STATES and error is None,
-    }
-    if error is not None:
-        data['error'] = error
-    return data
-
 
 @decorators.api_view(['GET'])
 @decorators.permission_classes((permissions.AllowAny,))
 @decorators.renderer_classes((JSONRenderer,))
 def job_status(request, task_id):
-    try:
-        task_name, state, public_data, error = get_public_task_data(
-            request,
-            task_id,
-        )
-    except (TaskNoPermission, ConnectionError):
-        return Response(get_status_data('unknown', 'PENDING', {}),)
-    return Response(get_status_data(task_name, state, public_data, error),)
+    """Retrieve Celery task function state from frontend."""
+    # HACK: always poll up to N times and after that return the sync has
+    # finished. This is a way to avoid re-enabling Celery result backend for now.
+    # TODO remove this API and RemoteRepo sync UI when we have better auto syncing
+    poll_n = cache.get(task_id, 0)
+    poll_n += 1
+    cache.set(task_id, poll_n, 5 * 60)
+    finished = poll_n == 5
+
+    data = {
+        "name": "sync_remote_repositories",
+        "data": {},
+        "started": True,
+        "finished": finished,
+        "success": finished,
+    }
+    return Response(data)
 
 
 @decorators.api_view(['POST'])
 @decorators.permission_classes((permissions.IsAuthenticated,))
 @decorators.renderer_classes((JSONRenderer,))
 def sync_remote_repositories(request):
+    """Trigger a re-sync of remote repositories for the user."""
     result = tasks.sync_remote_repositories.delay(user_id=request.user.id,)
     task_id = result.task_id
     return Response({

readthedocs/api/v2/views/task_views.py

@@ -217,6 +217,16 @@ def is_private(self):
             return self.project.external_builds_privacy_level == PRIVATE
         return self.privacy_level == PRIVATE
 
+    @property
+    def is_public(self):
+        """
+        Check if the version is public (taking external versions into consideration).
+
+        This is basically ``is_private`` negated,
+        ``is_private`` understands both normal and external versions
+        """
+        return not self.is_private
+
     @property
     def is_external(self):
         return self.type == EXTERNAL