Skip to content

Latest commit

 

History

History
310 lines (214 loc) · 12.5 KB

builder-maintainability.rst

File metadata and controls

310 lines (214 loc) · 12.5 KB

Builder Maintainability

This document extends the proposal done by Manuel at Future Builder to replace the last 3 goals:

  • Allow us to add a feature with a defined contract without worry about breaking old builds
  • Introduce build.builder: 2 config (does not install pre-defined packages) for these new features
  • Motivate users to migrate to v2 to finally deprecate this magic by educating users

It proposes a way to version the current "magical" builder so we can freeze its current status and extend it if needed by doing a new release of it. Besides, it removes the needing of using build.builder: 2 since the new builder will extend the "magical" builder behavior and just override what's required.

Goals

  • Keep the current "magical" builder working as-is (freeze this version)
  • Make "magical" builder maintainable over time
  • Allow users to specify a version of the "magical" builder
  • Define a code API to split in a modularized way the builder's code
  • Use the API to extend builder's code to support the "new builder"
  • Split the builders' code in: BuilderBase, SphinxMagicalBuilder and SphinxDummyBuilder
  • Fail the build when mixing config for commands (e.g. sphinx.fail_on_warning, submodules, etc) with the new builder (e.g. build.jobs and build.commands)
  • Fail the build of new projects if users doesn't pin the version of the builder
  • Fail the build if new builder is used without pinning the builder version
  • Set current "magical" builder version as default for current projects
  • Move the build process out from Read the Docs core on their own packages

"Magic" currently executed by our builder

This section describes all the "magic" that Read the Docs does on behalf of the user in the build process. The magic is split in two different scenarios: environment and tool.

Note

This section is strongly related with a similar research done by Santos at Explicit Builders

Environment: virtualenv

Magic happening at virtualenv setup process:

  • Creates virtualenv
  • Tries possible pip's requirement files if not defined
  • Install package: repository itself
  • Updates pip and installs setuptools (uses feature flag: DONT_INSTALL_LATEST_PIP)
  • Installs core dependencies (uses feature flag: DEFAULT_TO_MKDOCS_0_17_3, USE_MKDOCS_LATEST, USE_SPHINX_LATEST, USE_SPHINX_RTD_EXT_LATEST)
  • Install requirements file (uses feature flag: PIP_ALWAYS_UPGRADE)

Note

All this magic lives in the readthedocs/doc_builder/python_environments.py file.

Environment: conda

Magic happening at conda environment setup process:

  • Creates conda environment
  • Install package: repository itself
  • Updates conda (uses feature flag: CONDA_USES_MAMBA)
  • Installs mamba (uses feature flag: CONDA_USES_MAMBA)
  • Installs core dependencies
  • Append conda core requirements to environment file (uses feature flag: CONDA_APPEND_CORE_REQUIREMENTS)

Note

All this magic lives in the readthedocs/doc_builder/python_environments.py file.

Tool: Sphinx

Magic happening at Sphinx build process:

  • Creates a index.rst if it doesn't exist

  • Searches for different conf.py paths if not provided

  • Creates conf.py based on a template if it does not exist

  • Appends default content to conf.py (uses feature flags: ALL_VERSIONS_IN_HTML_CONTEXT, ALL_VERSIONS_IN_HTML_CONTEXT, DONT_OVERWRITE_SPHINX_CONTEXT, DISABLE_SERVER_SIDE_SEARCH)

  • Call build command with specific arguments (uses feature flags: SPHINX_PARALLEL, USE_SPHINX_BUILDERS)

  • Move the output (and change the filename) to a specific path

  • Emulates make all-pdf based on the backend engine: pdflatex or latexmk: - pdflatex:

    • calls pdflatex over all tex files (two passes!)
    • calls makeindex over all tex files
    • latexmk: - calls extractbb for each image file - finds the correct latexmkrc file based on language - execute latexmk with specific options

Note

All this magic lives in the readthedocs/doc_builder/backends/sphinx.py file.

Tool: MkDocs

Magic happening at MkDocs building process:

  • Creates a index.md if it doesn't exist
  • Default to mkdocs.yaml file config at the root level if not defined
  • Creates a default mkdocs.yaml file if it doesn't exist
  • Requires parsing (outside Docker container) mkdocs.yaml to modify some configs
  • Default the docs_dir to docs
  • Generates readthedocs-data.js with project's data (uses feature flag: ENABLE_MKDOCS_SERVER_SIDE_SEARCH, DISABLE_SERVER_SIDE_SEARCH)
  • Add extra_css and extra_javascript with our custom files (e.g. readhedocs-data.js, readthedocs-doc-embed.js, readthedocs-analytics.js, badge_only.css, readthedocs-doc-embed.css)
  • Remove MkDocs google_analytics config
  • Override MkDocs theme config (uses feature flag: MKDOCS_THEME_RTD)
  • Call build command with specific arguments
  • Move the HTML output to a specific path

Note

All this magic lives in the readthedocs/doc_builder/backends/mkdocs.py file.

Make "magical" builder maintainable

This document understands the "magical" builder as a core and key product of Read the Docs. It has been used for +10 years now and it's the only way users have to use the platform. However, maintaining it over this time has produced builds to be broken after an update and also it made impossible to add a new feature because it's not backward compatible; producing the adoption of multiple feature flags that become unmaintaible at this point. On the other hand, the time spent thinking about how to make everything keep working the same way while introducing a new feature that works for all the use cases has been exponentially increased over time.

Some of these problems can be solved by:

  • Freezing the "magical" builder as-is (and stop adding new features)
  • Versioning the "magical" builder and allow users to pin to a version

Either way, the initial step would be to freeze/version the current state and breaking it out from Read the Docs into its own repository, converted into a Python package. On the build process, the builder will be treat as a regular dependency: readthedocs-sphinx-magical-builder==1.0.0 or readthedocs-mkdocs-magical-builder==1.0.0.

The current "magical" builder state (e.g. version 1.0.0) will be installed by default if the user does not specify it, allowing us to add breaking changes to the builder without worrying about breaking old projects/builds while giving new features to users whom will be able to opt-in to the new version of the "magical" builder (e.g. 2.0.0).

Builder's code API and breakout into packages

readthedocs-base-builder

class BuilderBase:

    def __init__(self, environment, doctool):
        # Virtualenv or Conda for now.
        # In the future this could be Rust/Nodejs/Ruby/etc environment.
        self.environment = environment

        # SphinxBuilder or MkDocs class for now.
        # In the future this could be HugoBuilder, PelicanBuilder, etc.
        self.doctool = doctool

    # .. other pre/post and jobs methods

    def pre_create_envirnoment(self):
        self.environment.pre_create_environment()

    def create_environment(self):
        self.environment.create_environment()

    def post_create_envirnoment(self):
        self.environment.post_create_environment()
        self.doctool.post_create_environment()

    # .. other pre/post and jobs methods

    def run(self):
        # Executes all the steps in order to perform the build
        pass


class Virtualenv(PythonEnvironment):

    def create_environment(self):
        return 'python3.9 -m virtualenv --system-site-packages env'

    def post_create_environment(self):
        # Updates pip and install setuptools
        return 'pip install -U pip setuptools'


class Conda(PythonEnvironment):

    def pre_create_environment(self):
        # Append Read the Docs core dependencies to conda's environment file
        pass

    def create_environment(self):
        return 'conda env create -f environment.yaml`

readthedocs-sphinx-magical-builder

class SphinxMagicalBuilder(BuilderBase):

    # .. other pre/post and job methods

    def post_create_environment(self):
        # Install Read the Docs core dependencies
        return 'pip install mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 '
               'sphinx sphinx-rtd-theme readthedocs-sphinx-ext'

    def pre_build(self):
        # Create ``index.rst`` if not found
        # Generate ``conf.py`` if not found
        # Append settings to ``conf.py``
        # etc
        pass

    def build(self):
        return 'sphinx-build -T -j auto -E -b html -d _build/doctrees -D language=en . _build/html'

    # .. other pre/post and job methods

readthedocs-mkdocs-magical-builder

class MkDocsMagicalBuilder(BuilderBase):

    # .. other pre/post and job methods

    def post_create_environment(self):
        # Install Read the Docs core dependencies
        return 'pip install mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 '
               'mkdocs==0.17.3'

    def pre_build(self):
        # Creates ``mkdocs.yaml`` file if not found
        # Generates ``readthedocs-data.js`` with project's data
        # Updates some configs in ``mkdocs.yaml``
        pass

    def build(self):
        return 'mkdocs build --clean --site-dir docs/ --config-file mkdocs.yaml'

    # .. other pre/post and job methods

The new non-"magical" builder

Users that don't want to use the "magical" builder will be able to install a different builder: readthedocs-sphinx-builder==1.0.0. This builder won't execute any pre/post jobs on behalf of the user and won't share code with the "magical" builder package. It will only contains the normal steps to build a Sphinx project and will fail if the project doesn't follow the standard structure. Any non-standard requirement can be supported by overriding the proper build.job. step.

Un-answered questions

  • Do we really need build.version: 2?
  • How do we allow people to remove all our magic without build.version: 2?
  • Should build.commands do not execute any readthedocs_ methods?
  • If build.jobs.install is overwritten with conda env create -f environment.yaml, should we execute our readthedocs_pre_install that appends our core requirements?
  • Overwritting the job itself should remove automatic pre/post hooks?
  • If we decide to use build.version: 2 to remove all the magic, how we communicate to users what's the magic removed?
  • How users will decide what builder to use? Should this be a config like build.builder: readthedocs-sphinx-magical-builder==1.0.0? Maybe a Python class path readthedocs.builder.sphinx.SphinxMagicalBuilder that we can import?
  • How do we audit code for new builders (e.g. PelicanBuilder done by the community)? The builder is currently executed _outside_ the Docker container.
  • Does it make sense to isolate the build process from our Python code and ran it inside the container instead? (e.g. $ readthedocs build --builder sphinx --source docs/) This code won't have access to the db nor our app code.