WifiClientSecure (arduino-esp32 v. 3.x) connect() fail on some server (e.g. "speechgen.io") - same code working well with 2.x

[kaloprojects]

Board

ESP32 DevKit

Version

v3.0.3

IDE Name

Arduino IDE (1.8.19)

Operating System

Windows 7

Flash frequency

80MHz

PSRAM enabled

yes

Upload speed

115200

Description

Hope someone can review this potential issue:
WifiClientSecure client.connect("some_server_xy", 443) fail

Since upgrade to 3.x (currently using 3.0.3) my earlier code is no longer working. I spent day in searching for root cause (contacted e.g. speechgen.io support team), but found root cause yesterday (more by accident): seems to be a library 2.x to 3.x issue. I downgraded for verification to arduino-esp32 2.x (2.0.16) and same code is working well (server connect successful ..i can call their TTS service API).

So can someone maybe check why this code snippet fails with 3.x: (sketch see below) ?

Hope some folks can analyze or fix a potential bug (as my complete project is on hold now due this issue, because downgrading to 2.x is no longer an option (because lot of other 3.x features needed e.g. new /i2s_std.h).

I also tried '<NetworkClientSecure.h>', HTTPClient etc. ..instead of WifiClientSecure .. all with same result. I also used tried with actual CA certificate (instead client.setInsecure()) .. same result: Successful client.connect() with Arduino-esp32 v. 2.x , but failing with 3.x on server "speechgen.io". Some dedicated server (e.g."api.openai.com") connect succesfully with 3.x and 2.x .. but my urgent needed "speechgen.io" server fails since 3.x

Do you @me-no-dev or any other folks have an idea what might have been changed on .connect() in 3.x .. and how to solve ?

• Is this a well known issue (some server failing on 3.x .. and some not) ?
• is there any workaround to connect e.g. "speechgen.io" successfully also with 3.x ? (all idea so much welcome!)

Thx !

Sketch

// Https Connect to Speechgen server "speechgen.io" port 443 
// -> always fails with Arduino-esp32 v. 3.x library (but same code result success with earlier Arduino-esp32 v. 2.x )
// -> other server (e.g. "api.openai.com" are working (also with 3.x)

#include <WiFi.h>           
#include <WiFiClientSecure.h> 
/* .. */
WiFiClientSecure client;  
client.setInsecure();                          
/* .. */  
if ( client.connect("speechgen.io", 443) )   // <- it FAILS here always (with ESP 3.x, only working with 2.x)
{  Serial.println("> Succesfully connected to speeschgen.io port 443"); 
}
else
{ Serial.println("ERROR - WifiClientSecure.connect to SpeechGen.io failed!");    // <-  so we always end here with ESP 3.x
}

Debug Message

ERROR - WifiClientSecure.connect to SpeechGen.io failed!
(means: client.connect("speechgen.io", 443) fail)

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[lbernstone]

Try adding client.setPlainStart(); before you connect.

[kaloprojects]

cool .. give me a second ;) .. i'll check

[kaloprojects]

i tested .. in first moment i was HAPPY ! .. because now it gets connected ! :)
.. but been too early pleased .. after sending the header (via POST request) the client.available() gets true (forever) .. but never receiving a valid response from server, the code can't leave this loop.:

while (client.available())    
 { char c = client.read();
   jsonResponse += String(c);
 }

I tested same with "api.openai.com" (which works with 3.x still), just for verification .. when adding a 'client.setPlainStart()' before .connect it connects (same as it does without) but fails too (available always true).

What is 'client.setPlainStart()' doing ? any idea what i could test next ?
Thx so much btw (much appreciated)!

[kaloprojects]

ohh, hold on .. what i wrote is wrong, sorry. There is a response, let me analyze the server string in detail. I will follow up.

[kaloprojects]

this happens:

• it connects to server (which is great ;)
• now reading the server response with this code snippet:

    WiFiClientSecure client_tcp; 
    client_tcp.setInsecure(); 

    // == creating Request body (complete payload):
    /* String Prompt = "{"; ....... removed here to keep the snippet small */

    client_tcp.setPlainStart();   // ####################
    if (client_tcp.connect("speechgen.io", 443))  
    { 
      // sending header:
      client_tcp.println("POST https://speechgen.io/index.php?r=api/text HTTP/1.1"); 
      client_tcp.println("Connection: close");                                              
      client_tcp.println("Host: speechgen.io"); 
      client_tcp.println("Content-Type: application/json; charset=utf-8"); 
      client_tcp.println("Content-Length: " + String(Prompt.length())); 
      client_tcp.println(); 
      
      //sending body:
      client_tcp.println(Prompt); 
      
      // Receiving SpeechGen.IO response 
      /* header is sent .. now we wait response  ...*/
      String json_Response = "";
      while ( json_Response == "" )    
      { while (client_tcp.available())    
        { char c = client_tcp.read();
          Serial.print( c );     // ## for debug print
          json_Response += String(c);               
        }
      }       
    } 

and this is the (Serial.print( c )) printed response: from "speechgen.io" server now:

HTTP/1.1 400 Bad Request
Server: nginx/1.14.1
Date: Wed, 24 Jul 2024 19:58:55 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close

10f
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.14.1</center>
</body>
</html>0

⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮

and client.available() is true forever, that's why it stays in endless loop and printing endless ⸮

(with arduino-esp32 2.x (e.g. 2.0.16) i get a correct json response, with a OK header first then a valid payload, just an example (payload sends me an url to the server generated speech mp3): '{"id":"23384565", "status":1,"file": "https://speechgen.io/texttomp3/... 878333_508.mp3", "file_cors": ...} etc)

hope you have an idea more .. at least i got connection, seems like 'client.setPlainStart()' raises an http vs. https issue ?

[kaloprojects]

just 2 screenshots .. to compare 2.0.16 and 3.0.3 (might help maybe)

Speechgen with 2.0.16 - working well:

Speechgen with 3.0.3 - Not working:

[kaloprojects]

Does someone have an idea why it fails on 3.x ? .. in case of a well know issue:
Is there any workaround to get it running ? Thx !

[lbernstone]

The server is asking for a cipher renegotiation, and that is failing. It works fine if TLS 1.3 is enabled in IDF 5.1, but I have no idea why it succeeds in IDF 4.4. This issue comes from upstream, so you can open an issue at https://github.com/espressif/esp-idf/issues and tell them you have a problem with the https_request example when you change to the URL/server you want.

[kaloprojects]

I just posted the issue here: espressif/esp-idf#14298
Hope we get some feedback/ideas.

[kaloprojects]

@lbernstone : we got feedback from the Espressif team :) .. see here:

" .. Thanks for the issue. This is due to the fact that the MBEDTLS_HARDWARE_AES is turned on and the required cipher is not supported by the hardware.
This bug has already been fixed with espressif/esp-idf@2c85399

This has been kept disabled by default on older branches to avoid silently increasing the flash footprint for users. Please enable the fix using Component Config -> mbedTLS -> Enable support for non-AES ciphers in GCM operation. After that the handshake should succeed."

[kaloprojects]

I assume this are good news, right ? Does this info help to include it in the next arduino-esp32 version ... or is there anything i have to add in my code ? Sorry in case that's a dummy question (i am not experienced in this ;)

[me-no-dev]

this is enough info. @P-R-O-C-H-Y and @lucasssvaz please one of you to make the PR to the lib-builder

[lucasssvaz]

@me-no-dev I will

[kaloprojects]

thank you folks for all your help ! :)
@me-no-dev: Just for my understanding: This means it will work with next arduino-esp32 (3.1.0) in few weeks, did i get it correctly ?