Merge branch 'main' into refactor-search-serializer-context

Author: stsewd

.github/workflows/pip-tools.yaml

@@ -11,6 +11,9 @@ on:
     # Run weekly on day 0 at 00:00 UTC
     - cron: "0 0 * * 0"
 
+permissions:
+  contents: read
+
 jobs:
   update-dependencies:
     permissions:

CHANGELOG.rst

@@ -1,3 +1,24 @@
+Version 8.6.0
+-------------
+
+:Date: September 28, 2022
+
+* `@github-actions[bot] <https://github.com/github-actions[bot]>`__: Dependencies: all packages updated via pip-tools (`#9621 <https://github.com/readthedocs/readthedocs.org/pull/9621>`__)
+* `@evildmp <https://github.com/evildmp>`__: Made some small changes to the MyST migration how-to (`#9620 <https://github.com/readthedocs/readthedocs.org/pull/9620>`__)
+* `@boahc077 <https://github.com/boahc077>`__: ci: add minimum GitHub at the workflow level for pip-tools.yaml (`#9617 <https://github.com/readthedocs/readthedocs.org/pull/9617>`__)
+* `@stsewd <https://github.com/stsewd>`__: Search: refactor API view (`#9613 <https://github.com/readthedocs/readthedocs.org/pull/9613>`__)
+* `@sashashura <https://github.com/sashashura>`__: GitHub Workflows security hardening (`#9609 <https://github.com/readthedocs/readthedocs.org/pull/9609>`__)
+* `@stsewd <https://github.com/stsewd>`__: Redirects: test with/without organizations (`#9605 <https://github.com/readthedocs/readthedocs.org/pull/9605>`__)
+* `@humitos <https://github.com/humitos>`__: Builds: concurrency small optimization (`#9602 <https://github.com/readthedocs/readthedocs.org/pull/9602>`__)
+* `@uvidyadharan <https://github.com/uvidyadharan>`__: Update intersphinx.rst (`#9601 <https://github.com/readthedocs/readthedocs.org/pull/9601>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Release 8.5.0 (`#9600 <https://github.com/readthedocs/readthedocs.org/pull/9600>`__)
+* `@github-actions[bot] <https://github.com/github-actions[bot]>`__: Dependencies: all packages updated via pip-tools (`#9596 <https://github.com/readthedocs/readthedocs.org/pull/9596>`__)
+* `@stsewd <https://github.com/stsewd>`__: OAuth: save refresh token (`#9594 <https://github.com/readthedocs/readthedocs.org/pull/9594>`__)
+* `@stsewd <https://github.com/stsewd>`__: Redirects: allow update (`#9593 <https://github.com/readthedocs/readthedocs.org/pull/9593>`__)
+* `@stsewd <https://github.com/stsewd>`__: Unresolver: strict validation for external versions and other fixes (`#9534 <https://github.com/readthedocs/readthedocs.org/pull/9534>`__)
+* `@stsewd <https://github.com/stsewd>`__: New unresolver implementation (`#9500 <https://github.com/readthedocs/readthedocs.org/pull/9500>`__)
+* `@stsewd <https://github.com/stsewd>`__: API v3: fix organizations permissions (`#8771 <https://github.com/readthedocs/readthedocs.org/pull/8771>`__)
+
 Version 8.5.0
 -------------
 

docs/conf.py

@@ -69,7 +69,7 @@
 
 master_doc = "index"
 copyright = "2010, Read the Docs, Inc & contributors"
-version = "8.5.0"
+version = "8.6.0"
 release = version
 exclude_patterns = ["_build"]
 default_role = "obj"

docs/user/guides/authors.rst

@@ -16,4 +16,4 @@ and :doc:`/intro/getting-started-with-mkdocs`.
    cross-referencing-with-sphinx
    intersphinx
    jupyter
-   migrate-rest-myst
+   Migrate from rST to MyST <migrate-rest-myst>

docs/user/guides/intersphinx.rst

@@ -83,8 +83,8 @@ Result:
    provided by Intersphinx:
 
    .. prompt:: bash $
-      
-      python -msphinx.ext.intersphinx https://www.sphinx-doc.org/en/master/objects.inv
+
+      python -m sphinx.ext.intersphinx https://www.sphinx-doc.org/en/master/objects.inv
 
 Intersphinx in Read the Docs
 ----------------------------
@@ -167,7 +167,7 @@ You can use it like this:
    The inventory file is by default located at ``objects.inv``, for example ``https://readthedocs-docs.readthedocs-hosted.com/en/latest/objects.inv``.
 
    .. code:: python
-      
+
       # conf.py file
 
       intersphinx_mapping = {

docs/user/guides/migrate-rest-myst.md

@@ -1,4 +1,8 @@
-# Migrating from reStructuredText to MyST Markdown
+# How to migrate from reStructuredText to MyST Markdown
+
+In this guide, you will find
+how you can start writing Markdown in your existing reStructuredText project,
+or migrate it completely.
 
 Sphinx is usually associated with reStructuredText, the markup language
 {pep}`designed for the CPython project in the early '00s <287>`.
@@ -9,15 +13,11 @@ The most powerful of such extensions is {doc}`MyST-Parser <myst-parser:index>`,
 which implements a CommonMark-compliant, extensible Markdown dialect
 with support for the Sphinx roles and directives that make it so useful.
 
-In this guide, you will find
-how you can start writing Markdown in your existing reStructuredText project,
-or migrate it completely.
-
 If, **instead of migrating**, you are starting a new project from scratch,
 have a look at {doc}`myst-parser:intro`.
 If you are starting a **project for Jupyter**, you can start with Jupyter Book, which uses ``MyST-Parser``, see the official Jupyter Book tutorial: {doc}`jupyterbook:start/your-first-book`
 
-## Writing your content both in reStructuredText and MyST
+## How to write  your content both in reStructuredText and MyST
 
 It is useful to ask whether a migration is necessary in the first place.
 Doing bulk migrations of large projects with lots of work in progress
@@ -54,7 +54,7 @@ If you want to use a different suffix, you can do so by changing your
 `source_suffix` configuration value in `conf.py`.
 ```
 
-## Converting existing reStructuredText documentation to MyST
+## How to convert existing reStructuredText documentation to MyST
 
 To convert existing reST documents to MyST, you can use
 the `rst2myst` CLI script shipped by {doc}`rst-to-myst:index`.
@@ -71,7 +71,7 @@ $ rst2myst convert docs/**/*.rst  # Convert every .rst file under the docs direc
 
 This will create a `.md` MyST file for every `.rst` source file converted.
 
-### Advanced usage of `rst2myst`
+### How to modify the behaviour of `rst2myst`
 
 The `rst2myst` accepts several flags to modify its behavior.
 All of them have sensible defaults, so you don't have to specify them
@@ -90,7 +90,7 @@ These are a few options you might find useful:
 You can read the full list of options in
 {doc}``the `rst2myst` documentation <rst-to-myst:cli>``.
 
-## Enabling optional syntax
+## How to enable optional syntax
 
 Some reStructuredText syntax will require you to enable certain MyST plugins.
 For example, to write [reST definition lists], you need to add a
@@ -108,7 +108,7 @@ in their documentation.
 
 [reST definition lists]: https://docutils.sourceforge.io/docs/user/rst/quickref.html#definition-lists
 
-## Writing reStructuredText syntax within MyST
+## How to write reStructuredText syntax within MyST
 
 There is a small chance that `rst2myst` does not properly understand a piece of reST syntax,
 either because there is a bug in the tool

package.json

@@ -1,6 +1,6 @@
 {
   "name": "readthedocs",
-  "version": "8.5.0",
+  "version": "8.6.0",
   "description": "Read the Docs build dependencies",
   "author": "Read the Docs, Inc <[email protected]>",
   "scripts": {

readthedocs/__init__.py

@@ -1,4 +1,4 @@
 """Read the Docs."""
 
 
-__version__ = "8.5.0"
+__version__ = "8.6.0"

readthedocs/core/unresolver.py

@@ -1,6 +1,6 @@
 import re
 from dataclasses import dataclass
-from urllib.parse import urlparse
+from urllib.parse import ParseResult, urlparse
 
 import structlog
 from django.conf import settings
@@ -27,8 +27,7 @@ class UnresolvedURL:
 
     version: Version = None
     filename: str = None
-    query: str = None
-    fragment: str = None
+    parsed_url: ParseResult = None
     domain: Domain = None
     external: bool = False
 
@@ -41,59 +40,92 @@ class Unresolver:
     # - /en/latest/
     # - /en/latest/file/name/
     multiversion_pattern = re.compile(
-        r"^/(?P<language>{lang_slug})(/((?P<version>{version_slug})(/(?P<file>{filename_slug}))?)?)?$".format(  # noqa
+        r"""
+        ^/(?P<language>{lang_slug})  # Must have the language slug.
+        (/((?P<version>{version_slug})(/(?P<file>{filename_slug}))?)?)?$  # Optionally a version followed by a file.  # noqa
+        """.format(
             **pattern_opts
-        )
+        ),
+        re.VERBOSE,
     )
 
     # This pattern matches:
     # - /projects/subproject
     # - /projects/subproject/
     # - /projects/subproject/file/name/
     subproject_pattern = re.compile(
-        r"^/projects/(?P<project>{project_slug}+)(/(?P<file>{filename_slug}))?$".format(
+        r"""
+        ^/projects/  # Must have the `projects` prefix.
+        (?P<project>{project_slug}+)  # Followed by the subproject alias.
+        (/(?P<file>{filename_slug}))?$  # Optionally a filename, which will be recursively resolved.
+        """.format(
             **pattern_opts
-        )
+        ),
+        re.VERBOSE,
     )
 
-    def unresolve(self, url, add_index=True):
+    def unresolve(self, url, append_indexhtml=True):
         """
         Turn a URL into the component parts that our views would use to process them.
 
         This is useful for lots of places,
         like where we want to figure out exactly what file a URL maps to.
 
         :param url: Full URL to unresolve (including the protocol and domain part).
-        :param add_index: If `True` the filename will be normalized
+        :param append_indexhtml: If `True` directories will be normalized
          to end with ``/index.html``.
         """
         parsed = urlparse(url)
         domain = self.get_domain_from_host(parsed.netloc)
-        project_slug, domain_object, external = self.unresolve_domain(domain)
-        if not project_slug:
+        (
+            parent_project_slug,
+            domain_object,
+            external_version_slug,
+        ) = self.unresolve_domain(domain)
+        if not parent_project_slug:
             return None
 
-        parent_project = Project.objects.filter(slug=project_slug).first()
+        parent_project = Project.objects.filter(slug=parent_project_slug).first()
         if not parent_project:
             return None
 
-        project, version, filename = self._unresolve_path(
+        current_project, version, filename = self._unresolve_path(
             parent_project=parent_project,
             path=parsed.path,
         )
 
-        if add_index and filename and filename.endswith("/"):
+        # Make sure we are serving the external version from the subdomain.
+        if external_version_slug and version:
+            if external_version_slug != version.slug:
+                log.warning(
+                    "Invalid version for external domain.",
+                    domain=domain,
+                    version_slug=version.slug,
+                )
+                version = None
+                filename = None
+            elif not version.is_external:
+                log.warning(
+                    "Attempt of serving a non-external version from RTD_EXTERNAL_VERSION_DOMAIN.",
+                    domain=domain,
+                    version_slug=version.slug,
+                    version_type=version.type,
+                    url=url,
+                )
+                version = None
+                filename = None
+
+        if append_indexhtml and filename and filename.endswith("/"):
             filename += "index.html"
 
         return UnresolvedURL(
             parent_project=parent_project,
-            project=project or parent_project,
+            project=current_project or parent_project,
             version=version,
             filename=filename,
-            query=parsed.query,
-            fragment=parsed.fragment,
+            parsed_url=parsed,
             domain=domain_object,
-            external=external,
+            external=bool(external_version_slug),
         )
 
     @staticmethod
@@ -109,7 +141,11 @@ def _match_multiversion_project(self, parent_project, path):
         Try to match a multiversion project.
 
         If the translation exists, we return a result even if the version doesn't,
-        so the translation is taken as the canonical project (useful for 404 pages).
+        so the translation is taken as the current project (useful for 404 pages).
+
+        :returns: None or a tuple with the current project, version and file.
+         A tuple with only the project means we weren't able to find a version,
+         but the translation was correct.
         """
         match = self.multiversion_pattern.match(path)
         if not match:
@@ -138,24 +174,40 @@ def _match_subproject(self, parent_project, path):
 
         If the subproject exists, we try to resolve the rest of the path
         with the subproject as the canonical project.
+
+        If the subproject exists, we return a result even if version doesn't,
+        so the subproject is taken as the current project (useful for 404 pages).
+
+        :returns: None or a tuple with the current project, version and file.
+         A tuple with only the project means we were able to find the subproject,
+         but we weren't able to resolve the rest of the path.
         """
         match = self.subproject_pattern.match(path)
         if not match:
             return None
 
-        project_slug = match.group("project")
+        subproject_alias = match.group("project")
         file = self._normalize_filename(match.group("file"))
         project_relationship = (
-            parent_project.subprojects.filter(alias=project_slug)
+            parent_project.subprojects.filter(alias=subproject_alias)
             .prefetch_related("child")
             .first()
         )
         if project_relationship:
-            return self._unresolve_path(
-                parent_project=project_relationship.child,
+            # We use the subproject as the new parent project
+            # to resolve the rest of the path relative to it.
+            subproject = project_relationship.child
+            response = self._unresolve_path(
+                parent_project=subproject,
                 path=file,
                 check_subprojects=False,
             )
+            # If we got a valid response, return that,
+            # otherwise return the current subproject
+            # as the current project without a valid version or path.
+            if response:
+                return response
+            return subproject, None, None
         return None
 
     def _match_single_version_project(self, parent_project, path):
@@ -182,10 +234,19 @@ def _unresolve_path(self, parent_project, path, check_subprojects=True):
         If the returned version is `None`, then we weren't able to
         unresolve the path into a valid version of the project.
 
+        The checks are done in the following order:
+
+        - Check for multiple versions if the parent project
+          isn't a single version project.
+        - Check for subprojects.
+        - Check for single versions if the parent project isn’t
+          a multi version project.
+
         :param parent_project: The project that owns the path.
         :param path: The path to unresolve.
         :param check_subprojects: If we should check for subprojects,
-         this is used to call this function recursively.
+         this is used to call this function recursively when
+         resolving the path from a subproject (we don't support subprojects of subprojects).
 
         :returns: A tuple with: project, version, and file name.
         """
@@ -216,7 +277,7 @@ def _unresolve_path(self, parent_project, path, check_subprojects=True):
             if response:
                 return response
 
-        return None, None, None
+        return parent_project, None, None
 
     @staticmethod
     def get_domain_from_host(host):
@@ -234,8 +295,8 @@ def unresolve_domain(self, domain):
         Unresolve domain by extracting relevant information from it.
 
         :param str domain: Domain to extract the information from.
-        :returns: A tuple with: the project slug, domain object, and if the domain
-         is from an external version.
+        :returns: A tuple with: the project slug, domain object, and the
+         external version slug if the domain is from an external version.
         """
         public_domain = self.get_domain_from_host(settings.PUBLIC_DOMAIN)
         external_domain = self.get_domain_from_host(
@@ -250,22 +311,22 @@ def unresolve_domain(self, domain):
             if public_domain == root_domain:
                 project_slug = subdomain
                 log.debug("Public domain.", domain=domain)
-                return project_slug, None, False
+                return project_slug, None, None
 
             # TODO: This can catch some possibly valid domains (docs.readthedocs.io.com)
             # for example, but these might be phishing, so let's ignore them for now.
             log.warning("Weird variation of our domain.", domain=domain)
-            return None, None, False
+            return None, None, None
 
         # Serve PR builds on external_domain host.
         if external_domain == root_domain:
             try:
+                project_slug, version_slug = subdomain.rsplit("--", maxsplit=1)
                 log.debug("External versions domain.", domain=domain)
-                project_slug, _ = subdomain.rsplit("--", maxsplit=1)
-                return project_slug, None, True
+                return project_slug, None, version_slug
             except ValueError:
                 log.info("Invalid format of external versions domain.", domain=domain)
-                return None, None, False
+                return None, None, None
 
         # Custom domain.
         domain_object = (