Invalid certificates (Let's Encrypt related)

[tfardet]

Probably due to the fact that the Let's Encrypt root certificate expired on 30 September 2021, trying to pull from many non-Github repositories leads to server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none.
Would it be possible to update the certificates to resolve this?
This is for instance an issue when using submodules that are not hosted on GitHub.

[humitos]

Would it be possible to update the certificates to resolve this?

What are the required steps that Read the Docs should follow to do this?

[humitos]

trying to pull from many non-Github repositories leads to server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none.

Do you have an example of a build URL failing with this error?

[tfardet]

Do you have an example of a build URL failing with this error?

https://readthedocs.org/api/v2/build/14898416.txt

What are the required steps that Read the Docs should follow to do this?

I don't know exactly how you manage your VMs but you need to patch the system certificate of your Linux (Ubuntu LTS?) to use the new Let'sEncrypt certificate (most probably Ubuntu release an update for OpenSSL or something like that, I don't use this distro so I don't know exactly)

[humitos]

Thanks for your reply.

I found what's required: apt-get update and apt-get install ca-certificates. After doing that, I'm able to clone the repository and its submodules from inside the VMs.

Now the problems 😞 . To update ca-certificates in our build images, we have to rebuild them all and that will bring known issues that we can't cleanly solve (basically, re-building an image doesn't make it exactly the same with the same software versions and people's builds start failing because of that). So, I'm not sure we will be able to re-build all the Docker images (stable, latest, testing).

However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

[humitos]

Another workaround would be to execute apt-get install ca-certificates as part of the build process from inside Read the Docs application itself.

[agjohnson]

I think we should be able to rebuild all the images, not just our beta image. Ubuntu releases are stable enough that I would trust things to work. Package versions do not drift heavily once in an LTS

[tfardet]

However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

That's fine with me 👍 can you ping me once that's done?

I don't think a workaround is a good idea, the first option would probably save you some work when others start having that same issue

[rjw57]

However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

Wouldn't that introduce a chicken and egg problem whereby the repo needs to be able to be cloned so that the configuration required to clone the repo can be read?

[ExtremOPS]

I am experiencing the same problem as op described. My code repository is hosted on a GitLab server.

If there is something with which I (a noob) could help, please let me know.

[humitos]

@rjw57

Wouldn't that introduce a chicken and egg problem whereby the repo needs to be able to be cloned so that the configuration required to clone the repo can be read?

Yes. I may express myself wrong here.

We use a default Docker image to run git commands inside it (currently we use readthedocs/build:latest) and I think we could use the new image readthedocs/build:ubuntu-20.04 with a newer version of ca-certificates to do the git commands.

However, even if the git commands succeed, there may be other commands that may fail in the build process due to the old CA certificates. In that case, users could define readthedocs/build:ubuntu-20.04 as the build image to fix that problem.

[astrojuanlu]

xref https://stackoverflow.com/q/69439345/554319

[tfardet]

@humitos can I approve the PR or does it have to be someone from RtD?

[stsewd]

@humitos can I approve the PR or does it have to be someone from RtD?

that's a core team decision, but if you have any feedback there, feel free to comment!

[tfardet]

that's a core team decision, but if you have any feedback there, feel free to comment!

thought so, so no, nothing to change on my side; I think this is the right server-side fix for now

[humitos]

We deployed a quick and temporal fix for now. Please, let us know if you still have issues with the certificates. Thanks!

[guigomcha]

I can confirm that the hotfix fixed our project build. Thanks a lot everyone!