Skip to content

Commit e9b0590

Browse files
committed
Docker build images: update design doc
Update the design doc with the latest conversation we had about this topic. Hopefully, we can agree on it and start making some tests in production.
1 parent d8e99b7 commit e9b0590

File tree

1 file changed

+163
-90
lines changed

1 file changed

+163
-90
lines changed

docs/development/design/build-images.rst

+163-90
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ Build Images
33

44
This document describes how Read the Docs uses the `Docker Images`_ and how they are named.
55
Besides, it proposes a path forward about a new way to create and name our Docker build images to allow sharing as many image layers as possible
6-
and support installation of OS level packages as well as extra requirements.
6+
and support installation of other languages (e.g. nodejs, rust, go) as extra requirements.
77

88
.. _Docker Images: https://github.com/readthedocs/readthedocs-docker-images
99

@@ -25,15 +25,12 @@ and go through different steps:
2525

2626
*All* those steps depends on specific commands versions: ``git``, ``python``, ``virtualenv``, ``conda``, etc.
2727
Currently, we are pinning only a few of them in our Docker images and that have caused issues
28-
when re-deploying these images with bugfixes: **the images are not reproducible in time**.
28+
when re-deploying these images with bugfixes: **the images are not reproducible over time**.
2929

3030
.. note::
3131

32-
The reproducibility of the images will be better once these PRs are merged,
33-
but OS packages still won't be 100% the exact same versions.
34-
35-
* https://github.com/readthedocs/readthedocs-docker-images/pull/145
36-
* https://github.com/readthedocs/readthedocs-docker-images/pull/146
32+
We have been improving the reproducibility of our images by adding some tests cases.
33+
These are run inside the Docker image after it's built and check that it contains the versions we expect.
3734

3835
To allow users to pin the image we ended up exposing three images: ``stable``, ``latest`` and ``testing``.
3936
With that naming, we were able to bugfix issues and add more features
@@ -48,59 +45,72 @@ This produced issues to people pinning their images to any of these names becaus
4845
Goals
4946
-----
5047

51-
* release completely new Docker images without forcing users to change their pinned image
52-
* allow users to stick with an image "forever" (~years)
48+
* release completely new Docker images without forcing users to change their pinned image (``stable``, ``latest``, ``testing``)
49+
* allow users to select language requirements instead of an image name
5350
* use a ``base`` image with the dependencies that don't change frequently (OS and base requirements)
5451
* ``base`` image naming is tied to the OS version (e.g. Ubuntu LTS)
5552
* allow us to add/update a Python version without affecting the ``base`` image
5653
* reduce size on builder VM disks by sharing Docker image layers
57-
* allow users to specify extra dependencies (apt packages, node, rust, etc)
58-
* automatically build & push *all* images on commit
59-
* deprecate ``stable``, ``latest`` and ``testing``
54+
* allow users to specify extra languages (e.g. nodejs, rust, go)
55+
* de-motivate the usage of ``stable``, ``latest`` and ``testing``; and promote declaring language requirements instead
6056
* new images won't contain old/deprecated OS (eg. Ubuntu 18) and Python versions (eg. 3.5, miniconda2)
57+
* install language requirements *at built time* using ``asdf`` and its plugins
58+
* pre-build images with most common languages combination used by users to speed up built times
59+
* deleting a pre-built image won't make builds to fail; only make them slower
60+
* support only the latest Ubuntu LTS version and give 1 year to users to migrate from the previous one
6161

6262

6363
Non goals
6464
---------
6565

6666
* allow creation/usage of custom Docker images
6767
* allow to execute arbitrary commands via hooks (eg. ``pre_build``)
68+
* automatically build & push *all* images on commit
6869

6970

70-
New build image structure
71-
-------------------------
71+
Pre-built build image structure
72+
-------------------------------
7273

7374
.. Taken from https://github.com/readthedocs/readthedocs-docker-images/blob/master/Dockerfile
7475
76+
The new pre-built images will use Ubuntu OS + Python (Python, Conda, Mamba) version.
77+
They will contain all the requirements to add extra languages support at built time via ``asdf`` command.
78+
7579
* ``ubuntu20-base``
7680

7781
* labels
7882
* environment variables
7983
* system dependencies
8084
* install requirements
8185
* LaTeX dependencies (for PDF generation)
82-
* other languages version managers (``pyenv``, ``nodenv``, etc)
86+
* languages version manager (``asdf``) and its plugins for each language
8387
* UID and GID
8488

8589
The following images all are based on ``ubuntu20-base``:
8690

8791
* ``ubuntu20-py*``
8892

89-
* Python version installed via ``pyenv``
93+
* Python version installed via ``asdf``
9094
* default Python packages (pinned versions)
95+
9196
* pip
9297
* setuptools
9398
* virtualenv
99+
94100
* labels
95101

96102
* ``ubuntu20-conda*``
97103

98-
* same as ``-py*`` versions
99-
* Conda version installed via ``pyenv``
100-
* ``mamba`` executable (installed via ``conda``)
104+
* Conda (``miniconda3-*``) version installed via ``asdf``
105+
* labels
106+
107+
* ``ubuntu20-mamba*``
108+
109+
* Mamba (``mambaforge*``) version installed via ``asdf``
110+
* labels
101111

102-
Note that all these images only need to run ``pyenv install ${PYTHON_VERSION}``
103-
to install a specific Python/Conda version.
112+
Note that all these images only need to run ``asdf install python ${PYTHON_VERSION}``
113+
to install a specific Python/Conda/Mamba version.
104114

105115
.. Build all these images with Docker
106116
@@ -115,60 +125,98 @@ to install a specific Python/Conda version.
115125
https://github.com/readthedocs/readthedocs-docker-images/pull/166
116126
117127
118-
Specifying extra user's dependencies
119-
------------------------------------
128+
Specifying extra languages requirements
129+
---------------------------------------
120130

121-
Different users may have different requirements. We were already requested to install
122-
``swig``, ``imagemagick``, ``libmysqlclient-dev``, ``lmod``, ``rust``, ``poppler-utils``, etc.
123-
124-
People with specific dependencies will be able to install them as APT packages or as extras
125-
using ``.readthedocs.yaml`` config file. Example:
131+
Different users may have different requirements.
132+
People with specific language dependencies will be able to install them by using ``.readthedocs.yaml`` config file.
133+
Example:
126134

127135
.. code:: yaml
128136
137+
version: 3
129138
build:
130-
image: ubuntu20
131-
python: 3.9
132-
system_packages:
133-
- swig
134-
- imagemagick
135-
extras:
136-
- node==14
137-
- rust==1.46
139+
os: ubuntu20
140+
languages:
141+
python: "3.9" # supports "pypy3", "miniconda3" and "mambaforge"
142+
nodejs: "14"
143+
rust: "1.54.0"
144+
golang: "1.17"
138145
139146
Important highlights:
140147

141-
* users won't be able to use custom Ubuntu PPAs to install packages
142-
* all APT packages installed will be from official Ubuntu repositories
143-
* not specifying ``build.image`` will pick the latest OS image available
144-
* not specifying ``build.python`` will pick the latest Python version available
145-
* Ubuntu 18 will still be available via ``stable`` and ``latest`` images
146-
* all ``node`` (major) pre-compiled versions on ``nodenv`` are available to select
147-
* all ``rust`` (minor) pre-compiled versions on ``rustup`` are available to select
148-
* knowing exactly what packages users are installing,
149-
could allow us to prebuild extra images: ``ubuntu20-py37+node14``
148+
* do not treat Python language different from the others (will help us to support other non-Python doctools in the future)
149+
* specifying ``build.languages.python: "3"`` will use Python version ``3.x.y``, and may different between builds
150+
* specifying ``build.languages.python: "3.9"`` will use Python version ``3.9.y``, and may different between builds
151+
* specifying ``build.languages.nodejs: "14"`` will use nodejs version ``14.x.y``, and may different between builds
152+
* if no full version is declared, it will use the latest available in pre-built images first; if there is no pre-built images for that version; the latest available on ``asdf`` will be installed
153+
* not specifying ``build.os`` will make the config file parser to fail
154+
* not specifying ``build.languages`` will make the config file parsing to fail (at least one is required)
155+
* specifying only ``build.languages.nodejs`` and using Sphinx to build the docs, will make the build to fail
156+
* ``build.image`` is incompatible with ``build.os`` or ``build.languages``
157+
* Ubuntu 18 will still be available via ``stable`` and ``latest`` images, but not in new ones
158+
* a subset (not defined yet) of ``python``, ``nodejs``, ``rust`` and ``go`` versions on ``asdf`` are available to select
159+
160+
.. note::
161+
162+
We are moving away from users specifying a particular Docker image.
163+
With the new approach, users will specify the languages requirements they need,
164+
and Read the Docs will decide if it will use a pre-built image or will spin up the base one and install these languages on the fly.
165+
166+
However, ``build.image`` will be still available for backward compatibility with ``stable``, ``latest`` and ``testing`` but won't support the new ``build.languages`` config.
167+
168+
Note that knowing exactly what packages users are installing,
169+
could allow us to pre-build the most common combinations used images: ``ubuntu20-py39+node14``.
170+
171+
172+
Time required to install languages at build time
173+
------------------------------------------------
174+
175+
In my testings using ``time`` command in ASG instances,
176+
installing extra languages took these "real" times:
150177

151-
.. admonition:: Implementation
178+
* ``build-default``
152179

153-
We talked about using a ``Dockerfile.custom`` and build it on every build.
154-
However, at this point it requires extra work to change our build pipeline.
155-
We decided to install OS packages from the application itself for now using
156-
Docker API to call ``docker exec`` as ``root`` user.
180+
* python 3.9.6: 2m21.331s
181+
* mambaforge 4.10.1: 0m26.291s
182+
* miniconda3 4.7.12: 0m9.955s
183+
* nodejs 14.17.5: 0m5.603s
184+
* rust 1.54.0: 0m13.587s
185+
* golang 1.17: 1m30.428s
157186

158-
This reduces the amount of work required but also allows us to add this feature
159-
to our current existing images (they require a rebuild to add ``nodenv`` and ``rustup``)
187+
* ``build-large``
188+
189+
* python 3.9.6: 2m33.688s
190+
* mambaforge 4.10.1: 0m28.781s
191+
* miniconda3 4.7.12: 0m10.551s
192+
* nodejs 14.17.5: 0m6.136s
193+
* rust 1.54.0: 0m14.716s
194+
* golang 1.17: 1m36.470s
195+
196+
Note that the only one that required compilation was Python.
197+
All the others, spend 100% of its time downloading the binary.
198+
These download times are *way better from EU* with my home internet connection.
199+
200+
In the worst scenario: "none of the specified language version has a pre-built image",
201+
the build will require ~5 minutes to install all the language requirements.
202+
By providing *only* pre-built images with the Python version (that's the most time consuming),
203+
build times will only require ~2 minutes to install the others.
204+
However, requiring one version of each language is not a common case.
160205

161206

162207
Updating versions over time
163208
---------------------------
164209

165-
How do we add/upgrade a Python version?
166-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
210+
How do we upgrade a Python version?
211+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
167212

168213
Python patch versions can be upgraded on the affected image.
169214
As the ``base`` image won't change for this case, it will only modify the layers after it.
170215
All the OS package versions will remain the same.
171216

217+
How do we add a Python version?
218+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
219+
172220
In case we need to *add* a new Python version, we just need to build a new image based on ``base``:
173221
``ubuntu20-py310`` that will contain Python 3.10 and none of the other images are affected.
174222
This also allow us to test new Python (eg. 3.11rc1) versions without breaking people's builds.
@@ -191,34 +239,51 @@ Examples of these versions are:
191239

192240
This case will introduce a new ``base`` image. Example, ``ubuntu22-base`` in 2022.
193241
Note that these images will be completely isolated from the rest and don't require them to rebuild.
194-
This also allow us to test new Ubuntu versions without breaking people's builds.
242+
This also allow us to start testing a newer Ubuntu version (e.g. 22.04 LTS) without breaking people's builds,
243+
even before it's officially released.
244+
245+
We can start just with the ``base`` image and install all the languages dependencies at built time for the tests,
246+
without building many images based on the new OS version and having to store them on disk without people using them.
247+
195248

196249
How do we add an extra requirement?
197250
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
198251

199252
In case we need to add an extra requirement to the ``base`` image,
200253
we will need to rebuild all of them.
201254
The new image *may have different package versions* since there may be updates on the Ubuntu repositories.
202-
This conveys some small risk here, but in general we shouldn't require to add packages to the base images.
255+
This conveys some risk here, but in general we shouldn't require to add packages to the base images.
256+
257+
In case we need an extra requirement for *all our images*,
258+
I'd recommend to add it when creating a new base image.
259+
260+
If it's strongly needed and we can't wait for a new base image,
261+
we could install it at build time in a similar way as we do with ``build.apt_packages`` as a temporal workaround.
203262

204-
Users with specific requirements could use ``build.system_packages`` and/or ``build.extras`` in the config file.
205263

206264
How do we remove an old Python version?
207265
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
208266

209-
At some point an old version of Python will be deprecated (eg. 3.4) and will be removed.
210-
To achieve this, we can just remove the Docker image affected: ``ubuntu20-py34``,
211-
once there are no users depending on it anymore.
267+
At some point, an old version of Python will be deprecated (eg. 3.4) and will be removed.
268+
To achieve this, we can just remove the the pre-built Docker image affected: ``ubuntu20-py34``,
269+
once there are no users depending on it anymore (``build.os: ubuntu20`` and ``build.languages.python: 3.4``).
212270

213-
We will know which projects are using these images because they are pinning it in the config file.
271+
We will know which projects are using these images because they are pinning these specific versions in the config file.
214272
We could show a message in the build output page and also send them an email with the EOL date for this image.
215273

274+
However, removing an image that it's being currently used by some users won't make their builds to fail.
275+
Instead, that Python version will be installed at build time from the ``base`` image;
276+
adding a "penalization" time to those projects and motivating them to move forward to a newer version.
277+
278+
216279
Deprecation plan
217280
----------------
218281

219282
It seems we have ~50Gb free on builders disks.
220283
Considering that the new images will be sized approximately (built locally as test):
221284

285+
.. TODO: re-do this testing by updating the PR linked above
286+
222287
* ``ubuntu20-base``: ~5Gb
223288
* ``ubuntu20-py27``: ~150Mb
224289
* ``ubuntu20-py36``: ~210Mb
@@ -227,46 +292,54 @@ Considering that the new images will be sized approximately (built locally as te
227292

228293
which is about ~6Gb in total, we still have plenty of space.
229294

230-
We could keep ``stable``, ``latest`` and ``testing`` for some time without worry too much.
231-
New projects shouldn't be able to select these images and they will be forced to use ``ubuntu20``
232-
if they don't specify one.
295+
We could keep ``stable``, ``latest`` and ``testing`` for some good amount of time without worry too much.
296+
However, new projects shouldn't be able to select these images and they will be forced to use ``build.os`` and ``build.languages``.
297+
298+
We may want to keep only the latest Ubuntu LTS releases available in production,
299+
with a special consideration for our current Ubuntu 18.04 LTS on ``stable``, ``latest`` and ``testing`` because 100% of the projects depend on them currently.
300+
Once Ubuntu 22.04 LTS is released, we should communicate that Ubuntu 20.04 LTS is deprecated,
301+
and give users 1 year to migrate to a newer image.
302+
303+
304+
Work required and rollout plan
305+
------------------------------
306+
307+
The following steps are required to support the full proposal of this document.
308+
233309

234-
We may want to keep the two latest Ubuntu LTS releases available in production.
235-
At the moment of writing this they are:
236310

237-
* Ubuntu 18.04 LTS (our ``stable``, ``latest`` and ``testing`` images)
238-
* Ubuntu 20.04 LTS (our new ``ubuntu20``)
311+
#. allow users to install extras languages requirements via config file
239312

240-
Once Ubuntu 22.04 LTS is released, we should deprecate Ubuntu 18.04 LTS,
241-
and give users 6 months to migrate to a newer image.
313+
* update config file to support ``build.os`` and ``build.languages`` config
314+
* modify builder code to run ``asdf install`` for all supported languages
242315

316+
#. build new Docker images with new structure (``ubuntu20-base``)
243317

244-
Work required
245-
-------------
318+
* build new images with Ubuntu 20.04 LTS and pre-installed ``asdf`` with all its plugins
319+
* do not install any language version on pre-built images
320+
* deploy builders with new base image
246321

247-
There are a lot of work to do here.
248-
However, we want to prioritize it based on users' impact.
322+
#. update builders to install ``build.languages`` selected by the user
249323

250-
#. allow users to install packages with APT
251324

252-
* update config file to support ``build.system_packages`` config
253-
* modify builder code to run ``apt-get install`` as ``root`` user
325+
At this point, we will have a full working setup.
326+
It will be opt-in by using the new Config File V3.
327+
However, *all languages* will be installed at build time;
328+
which will "penalize" all projects because all of them will have to install Python.
254329

255-
#. allow users to install extras via config file
330+
After testing this for some time, we can continue with the following steps that provides pre-built images:
256331

257-
* update config file to support ``build.extras`` config
258-
* modify builder code to run ``nodenv install`` / ``rustup install``
259-
* re-build our current images with pre-installed nodenv and rustup
260-
* make sure that all the versions are the same we have in production
261-
* deploy builders with newer images
332+
#. pre-build latest 3 Python versions and Python 2.7, latest conda and latest mamba without extra languages
262333

263-
#. pre-build commands (not covered in this document)
334+
* ``ubuntu20-py27``
335+
* ``ubuntu20-py37``, ``ubuntu20-py38``, ``ubuntu20-py39``
336+
* ``ubuntu20-miniconda47``
337+
* ``ubuntu20-mambaforge410``
338+
* deploy builders with new pre-built images
264339

265-
#. new structure
340+
#. add feature flag to force new projects to use Config File V3 (``build.os`` and ``build.language``)
266341

267-
* update config file to support new image names for ``build.image``
268-
* automate Docker image building
269-
* deploy builders with newer images
342+
#. collect some data on most used languages combinations and pre-built Docker images for them
270343

271344

272345
Conclusion
@@ -278,7 +351,7 @@ The version of the OS will change many library versions,
278351
LaTeX dependencies, basic required commands like git and more,
279352
that doesn't seem to be useful to have the same OS version with different states.
280353

281-
Allowing users to install system dependencies and extras will cover most of the support requests we have had in the past.
354+
Allowing users extra languages by using the Config File will cover most of the support requests we have had in the past.
282355
It also will allow us to know more about how our users are using the platform to make future decisions based on this data.
283356
Exposing users how we want them to use our platform will allow us to be able to maintain it longer,
284-
than giving them totally freedom on the Docker image.
357+
than giving the option to select a specific Docker image by name that it's not freezed.

0 commit comments

Comments
 (0)