Design document for new Docker images structure

- Explains a little what's the current situations
- Mention problems we already had
- Propose a new and different naming for images
- Ideas to support custom Docker images

Author: humitos

docs/development/design/build-images.rst

@@ -0,0 +1,216 @@
+Build Images
+============
+
+This document describes how Read the Docs uses the Docker Build Images and how they are named.
+Besides, it proposes a new way to create and name them to allow
+sharing as many image layers as possible to support more customization while keeping the stability.
+
+
+Introduction
+------------
+
+We use Docker images to build user's documentation.
+Each time a build is triggered, one of our VMs picks the task
+and go through different steps:
+
+#. run some application code to spin up a Docker image into a container
+#. execute git inside the container to clone the repository
+#. analyze and parse files from the repository *outside* the container
+#. create the environment and install docs' dependencies inside the container
+#. execute build commnands inside the container
+#. push the output generated by builds commands to the storage
+
+
+*All* those steps depends on specific commands versions: ``git``, ``python``, ``virtualenv``, ``conda``, etc.
+Currently, we are pinning only a few of them in our Docker images and that have caused issues
+when re-deploying these images with bugfixes: **the images are not reproducible in time**.
+
+.. note::
+
+   The repoducibility of the images will be fixed once
+   https://github.com/readthedocs/readthedocs-docker-images/pull/145 and
+   https://github.com/readthedocs/readthedocs-docker-images/pull/146
+   get merged.
+
+To allow users to pin the image we ended up exposing three images: ``stable``, ``latest`` and ``testing``.
+With that naming, we were able to bugfix issues and add more features
+on each image without asking the users to change the image selected in their config file.
+
+Then, when a completely different image appeared and after testing ``testing`` image enough,
+we discarded ``stable``, old ``latest`` became the new ``stable`` and old ``testing`` became the new ``latest``.
+This produced issues to people pinning their images to any of these names because after this change,
+*we changed all the images for all the users* and many build issues arrised!
+
+
+Goals
+-----
+
+* release a completely new Docker image without forcing users to change their pinned image
+* allow users to stick with an image "forever" (~years)
+* use a ``base`` image with the dependencies that don't change frequently (OS and base requirements)
+* reduce size on builder VM disks by sharing Docker image layers
+* deprecate ``stable``, ``latest`` and ``testing``
+* allow use custom images for particular users/customers by sharing most layers
+* create a small ``nopdf`` image version without LaTeX dependencies for local development
+
+
+New build image structure
+-------------------------
+
+.. Taken from https://github.com/readthedocs/readthedocs-docker-images/blob/master/Dockerfile
+
+* ``ubuntu20-base``
+  * labels
+  * environment variables
+  * system dependencies
+  * install requirements
+  * user requirements
+    * plantuml, imagemagick, rsgv-convert, swig
+    * sphinx-js dependencies
+    * rust
+  * UID and GID
+
+* ``ubuntu20-pdf`` (from ``ubuntu20-base``)
+  * PDF/LaTeX dependencies
+
+* ``ubuntu20`` (from ``ubuntu20-pdf``)
+  * all Python versions (2, 3.6, 3.7, 3.8, 3.9)
+  * conda
+  * future extra user requirements
+  * labels
+
+We will also build a ``nopdf`` version to allow quick testing in local development:
+
+* ``ubuntu20-nopdf`` (from ``ubuntu20-base``)
+  * same as ``ubuntu20`` but based on ``ubuntu20-base`` instead
+
+.. note::
+
+   I don't think it's useful to have ``ubuntu20-py37`` exposed to users,
+   since the Python version is selected by using the config file's ``python.version`` keyword,
+   we only update patch versions and we don't remove them (unless together with OS changes).
+
+.. Build all these images with Docker
+   docker build -t readthedocs/build:ubuntu20-base -f Dockerfile.base .
+   docker build -t readthedocs/build:ubuntu20-nopdf -f Dockerfile.nopdf .
+   docker build -t readthedocs/build:ubuntu20-pdf -f Dockerfile.pdf .
+   docker build -t readthedocs/build:ubuntu20 -f Dockerfile .
+
+   Check the shared space between images
+   docker system df --verbose | grep -E 'SHARED SIZE|readthedocs'
+
+
+Custom images
+-------------
+
+There are some dependencies that are not easy to update and keep compatibility with all the users at the same time.
+Upgrading ``nodejs`` may make lot of old projects expecting the older version to start failing all their builds.
+On the other hand, sticking with an old version avoid users requiring a newer version to build their documentation.
+To handle this case and others, we have been thinking on supporting custom Docker images.
+
+It's not clear to me how it would be the implementation of this, but I see different paths to discuss and explore:
+
+#. Allow a ``build.dockerfile`` config pointing to a ``Dockerfile``
+  * ``FROM readthedocs/build:ubuntu20`` is required to be a valid image (to share layers)
+  * the image is build each time a build is triggered consuming build time
+#. Create a branch per custom image in ``readthedocs-docker-images`` repository
+  * use ``ubuntu20`` as base image and add the custom extra requirements
+  * build the image using our current process (Docker Hub)
+  * add the custom image to our ``-ops`` repository
+  * re-build builders to pull down the new custom image
+  * set the project to use this custom image, eg. ``readthedocs/build:<project-slug>``
+
+
+Updating versions over time
+---------------------------
+
+How do we add/upgrade a Python version?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Python patch versions can be upgraded and backported to all the images without problems.
+There is only needed to rebuild ``ubuntu20`` and most of the layers will remain shared with ``-base`` and ``-pdf``.
+
+In case we need to *add* a new Python version, the situation is similar.
+We can add the new version by using ``pyenv`` and rebuilding the ``ubuntu20`` image.
+
+
+How do we upgrade system versions?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We usually don't upgrade these dependencies unless we upgrade the Ubuntu version.
+So, they will be only upgraded when we go from Ubuntu 18.04 LTS to Ubuntu 20.04 LTS for example.
+
+Examples of these versions are:
+
+* doxygen
+* git
+* subversion
+* pandoc
+* nodejs / npm
+* swig
+* rust
+
+
+How do we add an extra requirement?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If a user asks for a new requirement (eg. azure CLI, ``az`` command) it should go into the
+"user requirements" section in the ``ubuntu20-base`` image.
+However, that will force us to rebuild all the images.
+
+We could use the section named as "future user extra requirements" for this,
+and it will force us to only rebuild the ``ubuntu20`` image.
+
+Both approaches will require to rebuild all the custom docker images from our users/customers
+that are based on the ``ubuntu20`` image.
+
+
+How do we remove an old Python version?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+At some point an old version of Python will be deprecated (eg. 3.4) and will be removed from our Docker images.
+These versions should only be removed when the OS in the ``base`` is upgraded (eg. from ``ubuntu20`` to ``ubuntu22``).
+
+
+Deprecation plan
+----------------
+
+It seems we have ~50Gb free on builders disks.
+Considering that the new images will be sized approximately (built locally as test):
+
+* ``base``: ~2.5Gb
+* ``nopdf``: ~5.5Gb
+* ``pdf``: ~1.5Gb
+
+which is about ~10Gb in total, we will still have space to support multiple custom images.
+
+We could keep ``stable``, ``latest`` and ``testing`` for some time without worry too much.
+New projects shouldn't be able to select these images and they will be forced to use ``ubuntu20``
+or any other custom image.
+
+We may want to keep the three latest Ubuntu LTS releases available in production.
+At the moment of writing this they are:
+
+* Ubuntu 16.04 LTS (we are not using it anymore)
+* Ubuntu 18.04 LTS (our ``stable``, ``latest`` and ``testing`` images)
+* Ubuntu 20.04 LTS (our new ``ubuntu20``)
+
+Once Ubuntu 22.04 LTS is released, we should deprecate Ubuntu 16.04 LTS,
+and give users 6 months to migrate to a newer image.
+User with custom images based on Ubuntu 16.04 LTS will be forced to migrate as well.
+
+
+Conclusion
+----------
+
+I don't think we need to differentiate the images by its state (stable, latest, testing)
+but by its main base difference: OS. The version of the OS will change many library versions,
+LaTeX dependencies, basic required commands like git and more,
+that doesn't seem to be useful to have the same OS version with different states.
+
+Also, splitting images by Python version sounds complicated to maintain.
+Each time we need to make a small change into one of the base layers, we will end up rebuilding many images.
+Besides, the key ``python.version`` won't make sense anymore and bring confusions.
+
+Custom images is something that needs more exploration still,
+but both proposals seem doable in weeks as an initial proof of concept.