implement multiple .readthedocs.yml files per repo (#10001)

* initial work at multiple .readthedocs.yml files per repo

* darker

* Updates: Rename new field to build_config_file, add on Build and Version data, relax user input validation, introduce very basic reproducibility of builds

* Fix for test failures

* Fix up tests to expect no build_config_file when default is used

* Trimming down the scope by not storing a Version.build_config_path

* DDD = Documentation-driven-development: Adding the first draft for a how-to before moving on :)

* How-to: Rephrase bad intro

* Basic test that a custom path is applied and loaded, default is then ignored

* Test that a project-defined build config is read by build tasks

* Validate properly in clean_conf_py_file

* Validate input, check that we can also handle non-yaml extensions, add test case for Advanced Project page

* Handle unrelated linting errors

* Update migration

* New validation for build config file

* Don't fix clean_conf_py_file

* Update help_text and test case name

* Apply suggestions from @ericholscher's code review

Co-authored-by: Eric Holscher <[email protected]>

* Disable fetching per-build build_config_file + clarify logic about reproduciblity

* New validator, configurable with allowed file names and no regex @humitos :)

* Update readthedocs/doc_builder/director.py

Co-authored-by: Manuel Kaufmann <[email protected]>

* Use new validator on model fields and add database assertion on view test

* Swap around if condition blocks

* Apply suggestions from @humitos code review

Co-authored-by: Manuel Kaufmann <[email protected]>

* Update docs/user/guides/setup/monorepo.rst

* Improve "next steps"

* Unrelated: Fix breakage from similar issue as executablebooks/MyST-Parser#731

* I have no idea about why pre-commit in Circle CI wants this and my local conf does. The line is not > 100 chars!

* Update migrations + refactor validator, it wasn't serializable for the migration introspect

* Add a seealso for subprojects

* Refactor build_config_file=>readthedocs_yaml_path, config_file=>readthedocs_yaml_path

* Remove view test from projecs/tests/test_views, this module seems to be mis-located, mosts tests are in rtd_tests

* Anohter renaming fix config_file=>readthedocs_yaml_path

* Improve help text

* Fix a strange formulation in docs

* Mark validators as safe strings

* Clarify paths that are tested wrt. validation

* Update readthedocs/doc_builder/director.py

Co-authored-by: Manuel Kaufmann <[email protected]>

* Removing pattern from a more generic validator

* Add link to Wikipedia definition

* Mention ability to use different documentation tools. Clarify introduction text, don't say "nested".

* Arrested Software Development: Hello darker my old friend, you've come to talk with me again

---------

Co-authored-by: Benjamin Bach <[email protected]>
Co-authored-by: Benjamin Bach <[email protected]>
Co-authored-by: Benjamin Balder Bach <[email protected]>
Co-authored-by: Eric Holscher <[email protected]>
Co-authored-by: Manuel Kaufmann <[email protected]>

Author: 4 people

docs/_static/js/readthedocs-doc-diff.js

@@ -24,6 +24,10 @@ The following how-to guides help you solve common tasks and challenges in the se
     Need several projects under the same umbrella?
     Start using subprojects, which is a way to host multiple projects under a "main project".
 
+⏩️ :doc:`Using a .readthedocs.yaml file in a sub-folder </guides/setup/monorepo>`
+    This guide shows how to configure a Read the Docs project to use a custom path for the ``.readthedocs.yaml`` build configuration.
+    *Monorepos* that have multiple documentation projects in the same Git repository can benefit from this feature.
+
 ⏩️ :doc:`Hiding a version </guides/hiding-a-version>`
     Is your version (flyout) menu overwhelmed and hard to navigate?
     Here's how to make it shorter.
@@ -44,4 +48,5 @@ The following how-to guides help you solve common tasks and challenges in the se
    Managing custom domains </guides/custom-domains>
    Managing subprojects </guides/subprojects>
    Hiding a version </guides/hiding-a-version>
+   Using a .readthedocs.yaml file in a sub-folder </guides/setup/monorepo>
    Using custom URL redirects in documentation projects </guides/redirects>

docs/user/guides/setup/index.rst

@@ -0,0 +1,97 @@
+.. Next steps: Show an example pattern for a monorepo layout or link to an example project
+
+How to use a .readthedocs.yaml file in a sub-folder
+===================================================
+
+This guide shows how to configure a Read the Docs project to use a custom path for the ``.readthedocs.yaml`` build configuration.
+`Monorepos <https://en.wikipedia.org/wiki/Monorepo>`__ that have multiple documentation projects in the same Git repository can benefit from this feature.
+
+By default,
+Read the Docs will use the ``.readthedocs.yaml`` at the top level of your Git repository.
+But if a Git repository contains multiple documentation projects that need different build configurations,
+you will need to have a ``.readthedocs.yaml`` file in multiple sub-folders.
+
+.. seealso::
+
+   `sphinx-multiproject <https://sphinx-multiproject.readthedocs.io/en/latest/>`__
+       If you are only using Sphinx projects and want to share the same build configuration,
+       you can also use the ``sphinx-multiproject`` extension.
+
+   :doc:`/guides/environment-variables`
+       You might also be able to reuse the same configuration file across multiple projects,
+       using only environment variables.
+       This is possible if the configuration pattern is very similar and the documentation tool is the same.
+
+Implementation considerations
+-----------------------------
+
+This feature is currently *project-wide*.
+A custom build configuration file path is applied to all versions of your documentation.
+
+.. warning::
+
+   Changing the configuration path will apply to all versions.
+   Different versions of the project may not be able to build again if this path is changed.
+
+Adding an additional project from the same repository
+-----------------------------------------------------
+
+Once you have added the first project from the :ref:`Import Wizard <intro/import-guide:Automatically import your docs>`,
+it will show as if it has already been imported and cannot be imported again.
+In order to add another project with the same repository,
+you will need to use the :ref:`Manual Import <intro/import-guide:Manually import your docs>`.
+
+Setting the custom build configuration file
+-------------------------------------------
+
+Once you have added a Git repository to a project that needs a custom configuration file path,
+navigate to :menuselection:`Admin --> Advanced Settings` and add the path to the :guilabel:`Build configuration file` field.
+
+.. image:: /img/screenshot-howto-build-configuration-file.png
+   :alt: Screenshot of where to find the :guilabel:`Build configuration file` setting.
+
+After pressing :guilabel:`Save`,
+you need to ensure that relevant versions of your documentation are built again.
+
+.. tip::
+
+   Having multiple different build configuration files can be complex.
+   We recommend setting up 1-2 projects in your Monorepo and getting them to build and publish successfully before adding additional projects to the equation.
+
+Next steps
+----------
+
+Once you have your monorepo pattern implemented and tested and it's ready to roll out to all your projects,
+you should also consider the Read the Docs project setup for these individual projects.
+
+Having individual projects gives you the full flexibility of the Read the Docs platform to make individual setups for each project.
+
+For each project, it's now possible to configure:
+
+* Sets of maintainers (or :doc:`organizations </commercial/organizations>` on |com_brand|)
+* :doc:`Custom redirect rules </guides/custom-domains>`
+* :doc:`Custom domains </guides/custom-domains>`
+* :doc:`Automation rules </automation-rules>`
+* :doc:`Traffic and search analytics </reference/analytics>`
+* Additional documentation tools with individual :doc:`build processes </build-customization>`:
+  One project might use :doc:`Sphinx <sphinx:index>`,
+  while another project setup might use `Asciidoctor <https://asciidoctor.org/>`__.
+
+...and much more. *All* settings for a Read the Docs project is available for each individual project.
+
+.. seealso::
+
+   :doc:`/guides/subprojects`
+      More information on nesting one project inside another project.
+      In this setup, it is still possible to use the same monorepo for each subproject.
+
+Other tips
+----------
+
+For a monorepo,
+it's not desirable to have changes in unrelated sub-folders trigger new builds.
+
+Therefore,
+you should consider setting up :ref:`conditional build cancellation rules <build-customization:Cancel build based on a condition>`.
+The configuration is added in each ``.readthedocs.yaml``,
+making it possible to write one conditional build rules per documentation project in the Monorepo 💯️

docs/user/guides/setup/monorepo.rst

@@ -73,26 +73,27 @@ def get_skip(self, obj):
 
     class Meta(ProjectSerializer.Meta):
         fields = ProjectSerializer.Meta.fields + (
-            'enable_epub_build',
-            'enable_pdf_build',
-            'conf_py_file',
-            'analytics_code',
-            'analytics_disabled',
-            'cdn_enabled',
-            'container_image',
-            'container_mem_limit',
-            'container_time_limit',
-            'install_project',
-            'use_system_packages',
-            'skip',
-            'requirements_file',
-            'python_interpreter',
-            'features',
-            'has_valid_clone',
-            'has_valid_webhook',
-            'show_advertising',
-            'environment_variables',
-            'max_concurrent_builds',
+            "enable_epub_build",
+            "enable_pdf_build",
+            "conf_py_file",
+            "analytics_code",
+            "analytics_disabled",
+            "cdn_enabled",
+            "container_image",
+            "container_mem_limit",
+            "container_time_limit",
+            "install_project",
+            "use_system_packages",
+            "skip",
+            "requirements_file",
+            "python_interpreter",
+            "features",
+            "has_valid_clone",
+            "has_valid_webhook",
+            "show_advertising",
+            "environment_variables",
+            "max_concurrent_builds",
+            "readthedocs_yaml_path",
         )
 
 

docs/user/img/screenshot-howto-build-configuration-file.png

@@ -33,6 +33,7 @@ class BuildAdmin(admin.ModelAdmin):
         "date",
         "builder",
         "length",
+        "readthedocs_yaml_path",
         "pretty_config",
     )
     readonly_fields = (

readthedocs/api/v2/serializers.py

@@ -0,0 +1,27 @@
+# Generated by Django 3.2.18 on 2023-04-04 13:03
+
+from django.db import migrations, models
+
+import readthedocs.projects.validators
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("builds", "0049_automation_rule_copy"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="build",
+            name="readthedocs_yaml_path",
+            field=models.CharField(
+                blank=True,
+                default=None,
+                max_length=1024,
+                null=True,
+                validators=[readthedocs.projects.validators.validate_build_config_file],
+                verbose_name="Custom build configuration file path used in this build",
+            ),
+        ),
+    ]

readthedocs/builds/admin.py

@@ -81,6 +81,7 @@
     SPHINX_SINGLEHTML,
 )
 from readthedocs.projects.models import APIProject, Project
+from readthedocs.projects.validators import validate_build_config_file
 from readthedocs.projects.version_handling import determine_stable_version
 
 log = structlog.get_logger(__name__)
@@ -707,6 +708,14 @@ class Build(models.Model):
         null=True,
         blank=True,
     )
+    readthedocs_yaml_path = models.CharField(
+        _("Custom build configuration file path used in this build"),
+        max_length=1024,
+        default=None,
+        blank=True,
+        null=True,
+        validators=[validate_build_config_file],
+    )
 
     length = models.IntegerField(_('Build Length'), null=True, blank=True)
 

readthedocs/builds/migrations/0050_build_readthedocs_yaml_path.py

@@ -43,17 +43,18 @@
 )
 
 __all__ = (
-    'ALL',
-    'load',
-    'BuildConfigV1',
-    'BuildConfigV2',
-    'ConfigError',
-    'ConfigOptionNotSupportedError',
-    'ConfigFileNotFound',
-    'InvalidConfig',
-    'PIP',
-    'SETUPTOOLS',
-    'LATEST_CONFIGURATION_VERSION',
+    "ALL",
+    "load",
+    "BuildConfigV1",
+    "BuildConfigV2",
+    "ConfigError",
+    "ConfigOptionNotSupportedError",
+    "ConfigFileNotFound",
+    "DefaultConfigFileNotFound",
+    "InvalidConfig",
+    "PIP",
+    "SETUPTOOLS",
+    "LATEST_CONFIGURATION_VERSION",
 )
 
 ALL = 'all'
@@ -96,6 +97,17 @@ def __init__(self, directory):
         )
 
 
+class DefaultConfigFileNotFound(ConfigError):
+
+    """Error when we can't find a configuration file."""
+
+    def __init__(self, directory):
+        super().__init__(
+            f"No default configuration file in: {directory}",
+            CONFIG_FILE_REQUIRED,
+        )
+
+
 class ConfigOptionNotSupportedError(ConfigError):
 
     """Error for unsupported configuration options in a version."""
@@ -1369,18 +1381,28 @@ def search(self):
         return Search(**self._config['search'])
 
 
-def load(path, env_config):
+def load(path, env_config, readthedocs_yaml_path=None):
     """
     Load a project configuration and the top-most build config for a given path.
 
     That is usually the root of the project, but will look deeper. According to
     the version of the configuration a build object would be load and validated,
     ``BuildConfigV1`` is the default.
     """
-    filename = find_one(path, CONFIG_FILENAME_REGEX)
-
-    if not filename:
-        raise ConfigFileNotFound(path)
+    # Custom non-default config file location
+    if readthedocs_yaml_path:
+        filename = os.path.join(path, readthedocs_yaml_path)
+        # When a config file is specified and not found, we raise ConfigError
+        # because ConfigFileNotFound
+        if not os.path.exists(filename):
+            raise ConfigFileNotFound(os.path.relpath(filename, path))
+    # Default behavior
+    else:
+        filename = find_one(path, CONFIG_FILENAME_REGEX)
+        if not filename:
+            # This exception is current caught higher up and will result in an attempt
+            # to load the v1 config schema.
+            raise DefaultConfigFileNotFound(path)
 
     # Allow symlinks, but only the ones that resolve inside the base directory.
     with safe_open(

readthedocs/builds/models.py

@@ -16,8 +16,8 @@
     BuildConfigV1,
     BuildConfigV2,
     ConfigError,
-    ConfigFileNotFound,
     ConfigOptionNotSupportedError,
+    DefaultConfigFileNotFound,
     InvalidConfig,
     load,
 )
@@ -80,7 +80,7 @@ def get_build_config(config, env_config=None, source_file='readthedocs.yml'):
 def test_load_no_config_file(tmpdir, files):
     apply_fs(tmpdir, files)
     base = str(tmpdir)
-    with raises(ConfigFileNotFound) as e:
+    with raises(DefaultConfigFileNotFound) as e:
         with override_settings(DOCROOT=tmpdir):
             load(base, {})
     assert e.value.code == CONFIG_FILE_REQUIRED
@@ -196,6 +196,67 @@ def test_build_config_has_source_file(tmpdir):
     assert build.source_file == os.path.join(base, 'readthedocs.yml')
 
 
+def test_load_non_default_filename(tmpdir):
+    """
+    Load a config file name with a non-default name.
+
+    Verifies that we can load a custom config path and that an existing default config file is
+    correctly ignored.
+
+    Note: Our CharField validator for readthedocs_yaml_path currently ONLY allows a file to be
+    called .readthedocs.yaml.
+    This test just verifies that the loader doesn't care since we support different file names
+    in the backend.
+    """
+    non_default_filename = "myconfig.yaml"
+    apply_fs(
+        tmpdir,
+        {
+            non_default_filename: textwrap.dedent(
+                """
+                version: 2
+                """
+            ),
+            ".readthedocs.yaml": "illegal syntax but should not load",
+        },
+    )
+    base = str(tmpdir)
+    with override_settings(DOCROOT=tmpdir):
+        build = load(base, {}, readthedocs_yaml_path="myconfig.yaml")
+    assert isinstance(build, BuildConfigV2)
+    assert build.source_file == os.path.join(base, non_default_filename)
+
+
+def test_load_non_yaml_extension(tmpdir):
+    """
+    Load a config file name from non-default path.
+
+    In this version, we verify that we can handle non-yaml extensions
+    because we allow the user to do that.
+
+    See docstring of test_load_non_default_filename.
+    """
+    non_default_filename = ".readthedocs.skrammel"
+    apply_fs(
+        tmpdir,
+        {
+            "subdir": {
+                non_default_filename: textwrap.dedent(
+                    """
+                    version: 2
+                    """
+                ),
+            },
+            ".readthedocs.yaml": "illegal syntax but should not load",
+        },
+    )
+    base = str(tmpdir)
+    with override_settings(DOCROOT=tmpdir):
+        build = load(base, {}, readthedocs_yaml_path="subdir/.readthedocs.skrammel")
+    assert isinstance(build, BuildConfigV2)
+    assert build.source_file == os.path.join(base, "subdir/.readthedocs.skrammel")
+
+
 def test_build_config_has_list_with_single_empty_value(tmpdir):
     base = str(apply_fs(
         tmpdir, {

readthedocs/config/config.py

@@ -45,6 +45,8 @@ class BaseMkdocs(BaseBuilder):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
+
+        # This is the *MkDocs* yaml file
         self.yaml_file = self.get_yaml_config()
 
         # README: historically, the default theme was ``readthedocs`` but in