Build: install all the latest Python "core requirements" (#10508)

* Build: install all the latest Python "core requirements"

I created a new feature flag called `INSTALL_LATEST_CORE_REQUIREMENTS` that does
not depend on any other feature flag. Its goal is to always install the latest
Python packages.

We are trying to move away from pinning the dependencies on our side and telling
users to do that on their side if they want to have reproducible builds over
time.

I think this is the best outcome for new projects since they will immediately
use the latest versions of everything instead of old (and maybe broken)
dependencies.

I propose to set a particular date for this feature flag and make new projects
to start building with all the latest dependencies. This will, at least, stop
making the problem bigger. Later, we can decide how to communicate to old
projects that we are going to install the latest requirements and if they don't
want that, they should pin their dependencies. We can follow our new and shiny
deprecation path we have built and tested in the last month.

Related readthedocs/meta#8
Related #9562
Related #10402
Related #9081

* Build: remove deprecated dependencies

* Build: install latest core requirements when using Conda

* Docs: update default versions documentation to match the changes

* Feedback from review

Do not install `sphinx-rtd-theme` and clarify the docstring.

* Update documentation

Author: humitos

docs/user/build-default-versions.rst

@@ -46,29 +46,36 @@ Mkdocs:
 sphinx-rtd-theme:
   Projects created before October 20, 2020 (January 21, 2021 for :doc:`/commercial/index`) use ``0.4.3``.
   New projects use the latest version.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 pip:
   Latest version by default.
 
 setuptools:
   Projects using ``setup.py install`` as installation method use ``58.2.0`` or older.
   All other projects use the latest version.
+  Projects created after August 7, 2023 will always use the latest version.
 
 mock:
-  ``1.0.1`` (could be removed in the future).
+  ``1.0.1``.
+  Projects created after August 7, 2023 won't install this dependency by default.
+
 
 pillow:
-  ``5.4.1`` when using Python 2.7, 3.4, 3.5, 3.6, 3.7. Otherwise, its latest version
-  (could be removed in the future).
+  ``5.4.1`` when using Python 2.7, 3.4, 3.5, 3.6, 3.7. Otherwise, its latest version.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 alabaster:
-  ``0.7.x`` (could be removed in the future).
+  ``0.7.x``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 commonmark:
-  ``0.8.1`` (could be removed in the future).
+  ``0.8.1``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 recommonmark:
-  ``0.5.0`` (could be removed in the future).
+  ``0.5.0``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 Conda
 ~~~~~
@@ -87,15 +94,19 @@ Sphinx:
 
 sphinx-rtd-theme:
   Latest version by default installed via ``conda``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 mock:
-  Latest version by default installed via ``pip`` (could be removed in the future).
+  Latest version by default installed via ``pip``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 pillow:
-  Latest version by default installed via ``pip`` (could be removed in the future).
+  Latest version by default installed via ``pip``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 recommonmark:
-  Latest version by default installed via ``conda`` (could be removed in the future).
+  Latest version by default installed via ``conda``.
+  Projects created after August 7, 2023 won't install this dependency by default.
 
 Internal dependencies
 ---------------------

readthedocs/doc_builder/python_environments.py

@@ -163,6 +163,60 @@ def install_core_requirements(self):
             '--no-cache-dir',
         ]
 
+        if self.project.has_feature(Feature.INSTALL_LATEST_CORE_REQUIREMENTS):
+            self._install_latest_requirements(pip_install_cmd)
+        else:
+            self._install_old_requirements(pip_install_cmd)
+
+    def _install_latest_requirements(self, pip_install_cmd):
+        """
+        Install all the latest core requirements.
+
+        By enabling the feature flag ``INSTALL_LATEST_CORE_REQUIREMENTS``
+        projects will automatically get installed all the latest core
+        requirements: pip, setuptools, sphinx, readthedocs-sphinx-ext and mkdocs.
+
+        This is the new behavior and where we are moving towards.
+        """
+        # First, upgrade pip and setuptools to their latest versions
+        cmd = pip_install_cmd + ["pip", "setuptools"]
+        self.build_env.run(
+            *cmd,
+            bin_path=self.venv_bin(),
+            cwd=self.checkout_path,
+        )
+
+        # Second, install all the latest core requirements
+        requirements = []
+
+        if self.config.doctype == "mkdocs":
+            requirements.append("mkdocs")
+        else:
+            requirements.extend(
+                [
+                    "sphinx",
+                    "readthedocs-sphinx-ext",
+                ]
+            )
+
+        cmd = copy.copy(pip_install_cmd)
+        cmd.extend(requirements)
+        self.build_env.run(
+            *cmd,
+            bin_path=self.venv_bin(),
+            cwd=self.checkout_path,
+        )
+
+    def _install_old_requirements(self, pip_install_cmd):
+        """
+        Install old core requirements.
+
+        There are bunch of feature flags that will be taken in consideration to
+        decide whether or not upgrade some of the core dependencies to their
+        latest versions.
+
+        This is the old behavior and the one we want to get rid off.
+        """
         # Install latest pip and setuptools first,
         # so it is used when installing the other requirements.
         pip_version = self.project.get_feature_value(
@@ -416,7 +470,10 @@ def _append_core_requirements(self):
         else:
             # Append conda dependencies directly to ``dependencies`` and pip
             # dependencies to ``dependencies.pip``
-            pip_requirements, conda_requirements = self._get_core_requirements()
+            if self.project.has_feature(Feature.INSTALL_LATEST_CORE_REQUIREMENTS):
+                pip_requirements, conda_requirements = self._get_new_core_requirements()
+            else:
+                pip_requirements, conda_requirements = self._get_old_core_requirements()
             dependencies = environment.get('dependencies', [])
             pip_dependencies = {'pip': pip_requirements}
 
@@ -454,7 +511,22 @@ def _append_core_requirements(self):
                     'environment file.',
                 )
 
-    def _get_core_requirements(self):
+    def _get_new_core_requirements(self):
+        # Use conda for requirements it packages
+        conda_requirements = []
+
+        # Install pip-only things.
+        pip_requirements = []
+
+        if self.config.doctype == "mkdocs":
+            pip_requirements.append("mkdocs")
+        else:
+            pip_requirements.append("readthedocs-sphinx-ext")
+            conda_requirements.extend(["sphinx"])
+
+        return pip_requirements, conda_requirements
+
+    def _get_old_core_requirements(self):
         # Use conda for requirements it packages
         conda_requirements = [
             'mock',
@@ -483,7 +555,7 @@ def install_core_requirements(self):
             # create`` step.
             return
 
-        pip_requirements, conda_requirements = self._get_core_requirements()
+        pip_requirements, conda_requirements = self._get_old_core_requirements()
         # Install requirements via ``conda install`` command if they were
         # not appended to the ``environment.yml`` file.
         cmd = [

readthedocs/projects/models.py

@@ -1938,6 +1938,7 @@ def add_features(sender, **kwargs):
     USE_SPHINX_LATEST = 'use_sphinx_latest'
     DEFAULT_TO_MKDOCS_0_17_3 = 'default_to_mkdocs_0_17_3'
     USE_SPHINX_RTD_EXT_LATEST = 'rtd_sphinx_ext_latest'
+    INSTALL_LATEST_CORE_REQUIREMENTS = "install_latest_core_requirements"
 
     # Search related features
     DISABLE_SERVER_SIDE_SEARCH = 'disable_server_side_search'
@@ -2049,6 +2050,12 @@ def add_features(sender, **kwargs):
             USE_SPHINX_RTD_EXT_LATEST,
             _("Sphinx: Use latest version of the Read the Docs Sphinx extension"),
         ),
+        (
+            INSTALL_LATEST_CORE_REQUIREMENTS,
+            _(
+                "Build: Install all the latest versions of Read the Docs core requirements"
+            ),
+        ),
 
         # Search related features.
         (