Feature Request: ESP32 SSL/TLS Certificate Store

[yknivag]

Is it possible to handle https connections on the ESP32 without knowing in advance the root CA for the service one is trying to connect to?

For example, on the ESP8266 this is possible using CertStoreBearSSL (see https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WiFi).

This is particularly important for https services which redirect from one domain to another where one would need to know all the certificates in advance on the ESP32 whereas on the ESP8266 one needs only to add certs.ar to SPIFFS and it is possible to connect to almost any https service.

I may have missed a way that this is already possible but I haven't been able to find one. How complex would it be to port the CertStoreBearSSL functionality to the ESP32?

[lbernstone]

Porting BearSSL would be a very significant task, particularly to use the hardware acceleration. However, if you attempt to create an https connection in WiFiClientSecure with no CA/cert information, it will make permissive requests, and should accept any certificate offered. Perhaps posting your actual issue (with code and logs) instead of your proposed fix can resolve the root problem.

[yknivag]

make permissive requests, and should accept any certificate offered.

That really isn't a solution though is it? That's just leaving a key under the doormat because you keep losing your keys - you may as well just leave the door open.

What I want to be able to do is connect to any arbitrary https web service from within a sketch without having to load an individual root CA for each domain and keep track of which is which.

I can do that on the ESP8266 using CertStoreBearSSL as the certs.ar file allows any certificate to be authenticated against it's root CA without knowing which domains are required at the time of writing the code.

Essentially I would like to recreate this ESP8266 sketch on the ESP32 and have it function with the same level of security.

When I replace #include <ESP8266WiFi.h> with #include <WiFi.h> I get the following error in the Arduino IDE:

Arduino: 1.8.10 (Windows 7), Board: "ESP32 Dev Module, Disabled, Default 4MB with spiffs (1.2MB APP/1.5MB SPIFFS), 240MHz (WiFi/BT), QIO, 80MHz, 4MB (32Mb), 921600, None"

Test:26:30: error: CertStoreBearSSL.h: No such file or directory

[lbernstone]

Ok, so you want a certificate store, not BearSSL. I'd suggest you change the title if you want anybody to look at it- a cert store is significantly easier than an encryption library. The feature set of esp32 is completely different than esp8266, so won't be an exact analogue, but it should be possible to do it with less resources.

[yknivag]

Thank you for your guidance, I have changed the title as you suggested.

won't be an exact analogue

It would make sense for it to be implemented with the same "API" as the CertStoreBearSSL for the ESP8266 so that the interaction is the same. In the same way that the WiFi client offers a similar interface for ESP8266, ESP32, Arduino and others. That way it would be possible to pass a reference to a certStore object into a function and have it work the same way no matter the underlying hardware in the same way as one may now pass a reference to a WiFi client.

[pranavb25]

Porting BearSSL would be a very significant task, particularly to use the hardware acceleration. However, if you attempt to create an https connection in WiFiClientSecure with no CA/cert information, it will make permissive requests, and should accept any certificate offered. Perhaps posting your actual issue (with code and logs) instead of your proposed fix can resolve the root problem.

@lbernstone I just observed this behaviour. Could you please shed some light into why this is allowed and why this happens? If no certificate/a corrupted certificate is provided, shouldn't the library throw an error? It currently says "Certificate verified" which is incorrect.

I am not an expert on SSL, just wanting to understand the implications of this behaviour and the need of even providing a cert in light of this (and associated security concerns). I personally noticed this while trying to corrupt my ca cert and noticing what happens.

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[yknivag]

Would be better if this could be flagged as a feature request rather than a stale issue.

[stale]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[kinafu]

Did you already take a look at https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_crt_bundle.html?

[yknivag]

@kinafu - that's a promising start. Unfortunately only available in the ESP-IDF though. If that functionality (particularly the ability to store the cert bundle in SPIFFS/LittleFS) were to be ported here to Arduino-ESP32 then that would, I think, close this feature request.

[meltdown03]

@kinafu - that's a promising start. Unfortunately only available in the ESP-IDF though. If that functionality (particularly the ability to store the cert bundle in SPIFFS/LittleFS) were to be ported here to Arduino-ESP32 then that would, I think, close this feature request.

With a few modifications to the ssl_client.cpp/h files and adding the esp_crt_bundle.c/h component, I was able to use the esp_crt_bundle_attach() function to use specific certificates for the Google IOT MQTT Smart Outlet Example.

I had to manually create the x509_crt_bundle binary with the the esp_crt_bundle/gen_crt_bundle.py script and embedded into my sketch. I use platformio, so I'm going to make an extra_script to automate that process. In IDF, the CMakeLists.txt file does this for you.

[atanisoft]

I had to manually create the x509_crt_bundle binary with the the esp_crt_bundle/gen_crt_bundle.py script and embedded into my sketch. I use platformio, so I'm going to make an extra_script to automate that process. In IDF, the CMakeLists.txt file does this for you.

PlatformIO offers similar functionality for embedding files in the binary: http://docs.platformio.org/en/latest/platforms/espressif32.html#embedding-binary-data

[meltdown03]

Yeah, that's what I used to embed the binary that the script created.

…

On Tue, Jun 23, 2020, 7:36 AM Mike Dunston ***@***.***> wrote: I had to manually create the x509_crt_bundle binary with the the esp_crt_bundle/gen_crt_bundle.py script and embedded into my sketch. I use platformio, so I'm going to make an extra_script to automate that process. In IDF, the CMakeLists.txt file does this for you. PlatformIO offers similar functionality for embedding files in the binary: http://docs.platformio.org/en/latest/platforms/espressif32.html#embedding-binary-data — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3646 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPBQCCWAH7V5H5OUBG6DGDRYCOWHANCNFSM4KIZ2FZA> .

[kinafu]

That sounds promising!

Would you mind sharing your modified ssl_client implementation?
I guess the solution would be

• to add a method which does

esp_crt_bundle_set(myx509_bundle)
esp_crt_bundle_attach(&ssl_client->ssl_conf)

• and call that method in the ssl_init() or start_ssl_client() (?)

[meltdown03]

I added the modified esp_cert_bundle.c file to libraries/WiFiClientSecure/src. (Modified so logging would work). The diff is for the entire arduino-esp32 directory. Just drop the txt extensions.
esp_cert_bundle.diff.txt
esp_crt_bundle.c.txt

[meltdown03]

TL;DR - None of this is necessary...

build_flags =
     -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_DEBUG
     -DCONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_NONE=1
     -DCONFIG_MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE=1
     -DCONFIG_MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE_PATH=""
     -UCONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL

EDIT: undef added to build_flags
You could also edit the sdkconfig.h file that Arduino uses (tools/sdk/esp32/include/config/sdkconfig.h)

[meltdown03]

It seems like changing the #defines is not necessary. I wrote a simple extra_script for PlatformIO to automatically generate the x509_crt_bundle file:

Import("env")

env.Execute("python3 gen_crt_bundle.py --input ./certs")

It assumes the certificates are in a ./certs subdirectory of the project dir, and the gen_crt_bundle.py is in the project directory. The resulting x509_crt_bundle ends up in the project directory also. I'm not that familiar with the Arduino IDE, so maybe someone else can figure out how to implement this and the embedding of binary files like PlatformIO does.

[meltdown03]

That sounds promising!

Would you mind sharing your modified ssl_client implementation?
I guess the solution would be

• to add a method which does

esp_crt_bundle_set(myx509_bundle)
esp_crt_bundle_attach(&ssl_client->ssl_conf)

• and call that method in the ssl_init() or start_ssl_client() (?)

It's my understanding that the esp_crt_bundle_attach() method calls esp_crt_bundle_set(embedded_certs), so it is not required to call on it's own.

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.

[kinafu]

I modified the arduino-esp32 according to the discussed / proposed changes (thx @meltdown03).

To test the modifications I created an example sketch (as platformio project).
It's based on the BasicHttpsClient example.

I looked into the problem for some hours now and did not get it to work unfortunately.
At runtime on the device I always get the message:
[E][ssl_client.cpp:36] _handle_error(): [start_ssl_client():216]: (-12288) X509 - A fatal error occured, eg the chain is too long or the vrfy callback failed

You can try my changes quickly by checking out my test repository in your editor with platformio installed.
The test sketch will compile using my changes (set in platformio.ini).
Just do not forget to enter your own SSID and PASSWORD.

The example sketch should be fine. The failure probably lies in my arduino-esp32 modifications.

[kinafu]

hey stale bot, reopen the issue 🥺

[Onno-Dirkzwager]

@meltdown03 please do! I'm interested to see what you come up with. I've failed miserably in solving this ;)

Keep in mind that we would like to make a PR that should work for Arduino users out of the box and not everyone will use the bundle.

For our personal cases this already workes. So thanks for your original post pointing us in this direction!

[Onno-Dirkzwager]

poking the issue a bit so stale bot doesn't get angry.

[lbernstone]

Just an fyi, and it isn't there yet in arduino, but esp-idf now includes a certificate bundle as cert binaries. It is not a comprehensive set, but will get you into many cloud services, and will be 64kb of code space.

[Onno-Dirkzwager]

@lbernstone not sure but I believe we are already using what you are referring to:

• For PIO @Duckle29 used @meltdown03 's modified esp_cert_bundle.c to enable using the embeded bundle
  esp_crt_bundle_attach
• To prevent compile error's in Arduino I added a build flag.
• And to enable loading the bundle from FS I modified Duckl29's work here

If this was not what you meant or we are on the wrong track please let me know!?

Otherwise, I'll try and make a PR this weekend as @Duckle29 seems too busy and does not have the time...

[lbernstone]

I mean it will be included in the compiled libraries once the weird naming from esp-idf is resolved. You will not need any external library, just an include.

[Onno-Dirkzwager]

@lbernstone thank you for clarifying!
I think a perfect / easier alternative for @Duckle29

Would you still be interested in my PR regarding client.loadCertBundle() which lets you basically load the same bundle but then from a FS?
This would answer the original issue by @yknivag
And I would not mind this feature myself as this lets the user update the bundle(SPIFFS) separately OTA or with an SD card.

[lbernstone]

Anything goes on external libraries. If you make an example using one of those root CAs, then it should be usable as an example here too.

[Onno-Dirkzwager]

@lbernstone thank you, Ill make a PR this weekend.

[kubo6472]

Bumping the issue :)

[witnessmenow]

Also interested in this feature, @yknivag has a nice approach for doing OTA based on GitHub releases that relies on this and I think it would be a really worthy addition to the esp32

[kubo6472]

Bumping the issue :)

Pinging it again so Stalebot doesn’t make a move first.

[zekageri]

Is there any news on this?

[kubo6472]

Is there any news on this?

Just that it still isn't implemented

[esp32wrangler]

I was also trying to solve this issue, and this ticket was very helpful.
However, I think the proposed solution of loading the certificates from SPIFFS is very wasteful of RAM, 64k is a large chunk of the usable heap. As far as I can tell, if the the data is embedded with the embed_txtfiles approach (https://docs.platformio.org/en/latest/platforms/espressif32.html#embedding-binary-data), then it is read directly from flash through the cache mechanism (https://www.exploreembedded.com/wiki/File:Address_map.JPG) and the only RAM needed is the index that esp_crt_bundle_init builds.
Perhaps it would be best to just use the approach the original Espressif authors intended (see the EMBEDDED_BUNDLE == 1 section of esp_crt_bundle_attach).

Edit: Here is the modified example to use the certificates directly from flash: https://github.com/esp32wrangler/esp32-certBundle

But modifying on top of all the existing modifications is unnecessarily messy. It is enough to

1. add the esp_crt_bundle.h and cpp to the arduino package
2. change "x509_crt_imported_bundle_bin_start" in esp_crt_bundle.cpp to "_binary_data_cert_x509_crt_imported_bundle_bin_start"
3. modify start_ssl_client in ssl_client.cpp to automatically call esp_crt_bundle_attach if (rootCABuff == NULL && insecure == false)
4. instruct the user to include the certificates under the file name "data/cert/x509_crt_imported_bundle.bin" in their platformio project
5. instruct the user to add board_build.embed_txtfiles = data/cert/x509_crt_imported_bundle.bin in their config

[esp32wrangler]

I cleaned up the changes and made a change set that may be acceptable upstream. I have zero experience with this project, so I would appreciate if someone could check if the changes follow the project norms.

esp32wrangler@974a7d9

[Twilight-Logic]

I am also interested in this capability. I am developing an IoT project in the Arduino IDE which communicates with Google App Script and am unfortunately getting this error whenever I perform a GET request:

[E][ssl_client.cpp:36] _handle_error(): [data_to_read():287]: (-29184) SSL - An invalid SSL record was received

I am using the HTTPSRedirect library, which just wraps the WiFiClientSecure GET method without any modification.

An initial client.connect() works provided that .setInsecure() is used first, however a subsequent GET request just hangs the ESP32 although I still get the above message in the Serial Monitor when 'debug' log level is selected. Whether that's down to Google increasing security, changing their cert or the library not working as it should be, I don't know although I am assuming at this point that the problem is that the GET request can't be performed without having a valid certificate chain in place?
I have no problem accessing the app web page in a standard browser on the PC. Google does re-direct from script.google.com to script.googleusercontent.com hence the use of the HTTPSredirect library.

I have tested ESP32 Library versions 1.0.5 and 1.0.6 with the same result.

[Parsaabasi]

Hello,

Due to the overwhelming volume of issues currently being addressed, we have decided to close the previously received tickets. If you still require assistance or if the issue persists, please don't hesitate to reopen the ticket.

Thanks.