You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Different users may have different requirements. We were already requested to install
122
-
``swig``, ``imagemagick``, ``libmysqlclient-dev``, ``lmod``, ``rust``, ``poppler-utils``, etc.
128
+
Specifying extra language requirements
129
+
--------------------------------------
123
130
124
-
People with specific dependencies will be able to install them as APT packages or as extras
125
-
using ``.readthedocs.yaml`` config file. Example:
131
+
Different users may have different requirements.
132
+
People with specific language dependencies will be able to install them by using ``.readthedocs.yaml`` config file.
133
+
Example:
126
134
127
135
.. code:: yaml
128
136
137
+
version: 3
129
138
build:
130
-
image: ubuntu20
131
-
python: 3.9
132
-
system_packages:
133
-
- swig
134
-
- imagemagick
135
-
extras:
136
-
- node==14
137
-
- rust==1.46
139
+
os: ubuntu20
140
+
languages:
141
+
python: "3.9"# supports "pypy3", "miniconda3" and "mambaforge"
142
+
nodejs: "14"
143
+
rust: "1.54.0"
144
+
golang: "1.17"
138
145
139
146
Important highlights:
140
147
141
-
* users won't be able to use custom Ubuntu PPAs to install packages
142
-
* all APT packages installed will be from official Ubuntu repositories
143
-
* not specifying ``build.image`` will pick the latest OS image available
144
-
* not specifying ``build.python`` will pick the latest Python version available
145
-
* Ubuntu 18 will still be available via ``stable`` and ``latest`` images
146
-
* all ``node`` (major) pre-compiled versions on ``nodenv`` are available to select
147
-
* all ``rust`` (minor) pre-compiled versions on ``rustup`` are available to select
148
-
* knowing exactly what packages users are installing,
149
-
could allow us to prebuild extra images: ``ubuntu20-py37+node14``
148
+
* do not treat Python language different from the others (will help us to support other non-Python doctools in the future)
149
+
* not specifying ``build.os`` will pick the latest OS image available
150
+
* not specifying ``build.languages`` will make the config file parsing to fail
151
+
* ``build.image`` is incompatible with ``build.os`` or ``build.languages``
152
+
* Ubuntu 18 will still be available via ``stable`` and ``latest`` images, but not in new ones
153
+
* a subset (not defined yet) of ``python``, ``nodejs``, ``rust`` and ``go`` versions on ``asdf`` are available to select
154
+
155
+
.. note::
156
+
157
+
We are moving away from users specifying a particular Docker image.
158
+
With the new approach, users will specify the languages requirements they need,
159
+
and Read the Docs will decide if it will use a pre-built image or will spin up the base one and install these languages on the fly.
160
+
161
+
However, ``build.image`` will be still available for backward compatibility with ``stable``, ``latest`` and ``testing`` but won't support the new ``build.languages`` config.
162
+
163
+
Note that knowing exactly what packages users are installing,
164
+
could allow us to pre-build the most common used images: ``ubuntu20-py39+node14``.
165
+
150
166
151
-
.. admonition:: Implementation
167
+
Time required to install languages at build time
168
+
------------------------------------------------
152
169
153
-
We talked about using a ``Dockerfile.custom`` and build it on every build.
154
-
However, at this point it requires extra work to change our build pipeline.
155
-
We decided to install OS packages from the application itself for now using
156
-
Docker API to call ``docker exec`` as ``root`` user.
170
+
In my testings using ``time`` command in ASG instances,
171
+
installing extra languages took these "real" times:
157
172
158
-
This reduces the amount of work required but also allows us to add this feature
159
-
to our current existing images (they require a rebuild to add ``nodenv`` and ``rustup``)
173
+
* ``build-default``
174
+
175
+
* python 3.9.6: 2m21.331s
176
+
* mambaforge 4.10.1: 0m26.291s
177
+
* miniconda3 4.7.12: 0m9.955s
178
+
* nodejs 14.17.5: 0m5.603s
179
+
* rust 1.54.0: 0m13.587s
180
+
* golang 1.17: 1m30.428s
181
+
182
+
* ``build-large``
183
+
184
+
* python 3.9.6: 2m33.688s
185
+
* mambaforge 4.10.1: 0m28.781s
186
+
* miniconda3 4.7.12: 0m10.551s
187
+
* nodejs 14.17.5: 0m6.136s
188
+
* rust 1.54.0: 0m14.716s
189
+
* golang 1.17: 1m36.470s
190
+
191
+
Note that the only one that required compilation was Python.
192
+
All the others, spend 100% of its time downloading the binary.
193
+
These download times are *way better from EU* with my home internet connection.
194
+
195
+
In the worst scenario: "none of the specified language version has a pre-built iamge",
196
+
the build will require ~5 minutes to install all the language requirements.
197
+
By providing *only* pre-built images with the Python version (that's the most time consuming),
198
+
build times will only require ~2 minutes to install the others.
160
199
161
200
162
201
Updating versions over time
@@ -193,6 +232,9 @@ This case will introduce a new ``base`` image. Example, ``ubuntu22-base`` in 202
193
232
Note that these images will be completely isolated from the rest and don't require them to rebuild.
194
233
This also allow us to test new Ubuntu versions without breaking people's builds.
195
234
235
+
Note that this allows us to start testing Ubuntu 22.04 LTS before its official release.
236
+
237
+
196
238
How do we add an extra requirement?
197
239
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
198
240
@@ -201,24 +243,36 @@ we will need to rebuild all of them.
201
243
The new image *may have different package versions* since there may be updates on the Ubuntu repositories.
202
244
This conveys some small risk here, but in general we shouldn't require to add packages to the base images.
203
245
204
-
Users with specific requirements could use ``build.system_packages`` and/or ``build.extras`` in the config file.
246
+
In case we need an extra requirement for *all our images*,
247
+
I'd recommend to add it when creating a new base image.
248
+
249
+
If it's strongly needed and we can't wait for a new base image,
250
+
we could install it at build time in a similar way as we do with ``build.apt_packages``.
251
+
205
252
206
253
How do we remove an old Python version?
207
254
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
208
255
209
-
At some point an old version of Python will be deprecated (eg. 3.4) and will be removed.
210
-
To achieve this, we can just remove the Docker image affected: ``ubuntu20-py34``,
211
-
once there are no users depending on it anymore.
256
+
At some point, an old version of Python will be deprecated (eg. 3.4) and will be removed.
257
+
To achieve this, we can just remove the the pre-built Docker image affected: ``ubuntu20-py34``,
258
+
once there are no users depending on it anymore (``build.os: ubuntu20`` and ``build.languages.python: 3.4``).
212
259
213
-
We will know which projects are using these images because they are pinning it in the config file.
260
+
We will know which projects are using these images because they are pinning these specific versions in the config file.
214
261
We could show a message in the build output page and also send them an email with the EOL date for this image.
215
262
263
+
However, removing an image that it's being currently used by some users won't make their builds to fail.
264
+
Instead, that Python version will be installed at build time from the ``base`` image;
265
+
adding a "penalization" time to those projects.
266
+
267
+
216
268
Deprecation plan
217
269
----------------
218
270
219
271
It seems we have ~50Gb free on builders disks.
220
272
Considering that the new images will be sized approximately (built locally as test):
221
273
274
+
.. TODO: re-do this testing by updating the PR linked above
275
+
222
276
* ``ubuntu20-base``: ~5Gb
223
277
* ``ubuntu20-py27``: ~150Mb
224
278
* ``ubuntu20-py36``: ~210Mb
@@ -227,46 +281,54 @@ Considering that the new images will be sized approximately (built locally as te
227
281
228
282
which is about ~6Gb in total, we still have plenty of space.
229
283
230
-
We could keep ``stable``, ``latest`` and ``testing`` for some time without worry too much.
231
-
New projects shouldn't be able to select these images and they will be forced to use ``ubuntu20``
232
-
if they don't specify one.
284
+
We could keep ``stable``, ``latest`` and ``testing`` for some good amount of time without worry too much.
285
+
However, new projects shouldn't be able to select these images and they will be forced to use ``build.os`` and ``build.languages``.
286
+
287
+
We may want to keep only the latest Ubuntu LTS releases available in production,
288
+
with a special consideration for our current Ubuntu 18.04 LTS on ``stable``, ``latest`` and ``testing`` because 100% of the projects depend on them currently.
289
+
Once Ubuntu 22.04 LTS is released, we should communicate that Ubuntu 20.04 LTS is deprecated,
290
+
and give users 1 year to migrate to a newer image.
291
+
292
+
293
+
Work required and rollout plan
294
+
------------------------------
295
+
296
+
The following steps are required to support the full proposal of this document.
297
+
233
298
234
-
We may want to keep the two latest Ubuntu LTS releases available in production.
235
-
At the moment of writing this they are:
236
299
237
-
* Ubuntu 18.04 LTS (our ``stable``, ``latest`` and ``testing`` images)
238
-
* Ubuntu 20.04 LTS (our new ``ubuntu20``)
300
+
#. allow users to install extras languages requirements via config file
239
301
240
-
Once Ubuntu 22.04 LTS is released, we should deprecate Ubuntu 18.04 LTS,
241
-
and give users 6 months to migrate to a newer image.
302
+
* update config file to support ``build.os`` and ``build.languages`` config
303
+
* modify builder code to run ``asdf install`` for all supported languages
242
304
305
+
#. build new Docker images with new structure (``ubuntu20-base``)
243
306
244
-
Work required
245
-
-------------
307
+
* build new images with Ubuntu 20.04 LTS and pre-installed ``asdf`` with all its plugins
308
+
* do not install any language version on pre-built images
309
+
* deploy builders with new base image
246
310
247
-
There are a lot of work to do here.
248
-
However, we want to prioritize it based on users' impact.
311
+
#. update builders to install ``build.languages`` selected by the user
249
312
250
-
#. allow users to install packages with APT
251
313
252
-
* update config file to support ``build.system_packages`` config
253
-
* modify builder code to run ``apt-get install`` as ``root`` user
314
+
At this point, we will have a full working setup.
315
+
It will be opt-in by using the new Config File V3.
316
+
However, *all languages* will be installed at build time;
317
+
which will "penalize" all projects because all of them will have to install Python.
254
318
255
-
#. allow users to install extras via config file
319
+
After testing this for some time, we can continue with the following steps that provides pre-built images:
256
320
257
-
* update config file to support ``build.extras`` config
258
-
* modify builder code to run ``nodenv install`` / ``rustup install``
259
-
* re-build our current images with pre-installed nodenv and rustup
260
-
* make sure that all the versions are the same we have in production
261
-
* deploy builders with newer images
321
+
#. pre-build latest 3 Python versions and Python 2.7, latest conda and latest mamba without extra languages
262
322
263
-
#. pre-build commands (not covered in this document)
0 commit comments