Skip to content

error in updating Firmware OTA on esp32 using LTE module #6644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 task done
mudgalp opened this issue Apr 27, 2022 · 34 comments · Fixed by #7525
Closed
1 task done

error in updating Firmware OTA on esp32 using LTE module #6644

mudgalp opened this issue Apr 27, 2022 · 34 comments · Fixed by #7525
Assignees
@SuGlider
Labels
Area: BT&Wifi BT & Wifi related issues Resolution: Awaiting response Waiting for response of author
Milestone
2.0.6

Comments

@mudgalp
Copy link

mudgalp commented Apr 27, 2022

Board

ESP32 DEV MODULE

Device Description

The device is a RS485 Based Modbus to 4G_LTE Dataloger device.
Consist of esp32-wroom module 4MB Flash Size.
MAX485CSA IC for Modbus Communication >> connected on serial1 at 9600 baudrate
Quectel 4G-LTE module EC200S for Operating in remote area(There is no wifi network) >> connected on serial2 at 115200 baudrate.
and other 12-24v to different power regulations.

all the debugging is done from serial0.

Hardware Configuration

#define ESP32_U1_TX 25 // MAX485_RX -->
#define ESP32_U1_RX 27 // MAX485_TX -->
#define ESP32_U2_TX 17 // GSM_RX -->
#define ESP32_U2_RX 16 // GSM_TX -->
#define MAX485_EN 26
#define LED 2
#define BUZZ 33
#define CONFIG 35
#define ADC 34
#define GSMPWR 32
#define GSMDTR 4

Version

v1.0.6

IDE Name

Arduino IDE

Operating System

Windows 11

Flash frequency

80Mhz

PSRAM enabled

no

Upload speed

921600

Description

hello everyone,
I'm working on a device in which I'm trying to do Firmware Over the air update of esp32-wroom module,
but i'm using another 4G-LTE module from Quictel (EC200S) to download the bin file from server.

everything looks fine but in the last it gives me error to update as >>Error Occurred. Error #: 6

here is the ota Function i'm using.
please help

Sketch

bool FirmwareVersionCheck()
{
  bool value;
  int httpCode ;
  String fwurl = "";
  fwurl = URL_fw_Version;
  Serial.println(fwurl);
  Serial.println();

  if (WiFi.status() != WL_CONNECTED)
  {
    Network_Check_GSM();
    if (GSM_Network_STS == true)
    {
      Serial2.write("AT+QIACT?\r\n");
      delay(100);
      input = Serial2.readString();
      if (input.indexOf("OK") >= 0)
      {
        Serial.println("GPRS CONNECT SUCCESSFULLY");
        Serial.println();
        input = Serial2.readString();
        input.remove(0);
        Serial2.print("AT+QHTTPURL=83,30\r\n");
        delay(100);
        input = Serial2.readString();
        if (input.indexOf("CONNECT") >= 0)
        {
          Serial2.print(fwurl);
          delay(100);
          input = Serial2.readString();
          if (input.indexOf("OK") >= 0)
          {
            Serial.println("URL CONNECT SUCCESSFULLY");
            Serial.println();
            input = Serial2.readString();
            input.remove(0);
            int count = 0;
            Serial2.write("AT+QHTTPGET=30\r\n");
            input = Serial2.readString();
            while (input.indexOf("+QHTTPGET") < 0)
            {
              Serial.println("WAITING FOR GET COMMAND RESPONSE");
              Serial.println();
              if (count >= 5)
              {
                value = false;
                break;
              }
              count++;
              input = Serial2.readString();
            }

            String GetResponse = input.substring(input.indexOf("+QHTTPGET") + 13, input.indexOf("+QHTTPGET") + 16);
            Serial.print("GET RESPONSE IS:");
            Serial.println(GetResponse);
            Serial.println();
            if (GetResponse == "200")
            {
              Serial.println("GET RESPONSE IS OK");
              Serial.println();
              input = Serial2.readString();
              input.remove(0);
              Serial2.write("AT+QHTTPREAD=30\r\n");
              while (!Serial2.available())
              {
              }

              while (Serial2.available())
              {
                String line = Serial2.readStringUntil('\n');
                Serial.print("line is:" + line); Serial.println();
                line.trim();// remove white/empty space from the line.
                if (!line.length())// if the the line is empty,this is end of headers,//break the while and feed the remaining `Serial2` to the Update.writeStream()
                {
                  Serial.println("This Line Was empty");//headers ended
                  httpResponseLineCount ++;
                  Serial.println("httpResponseLineCount is :" + String(httpResponseLineCount));
                  if (httpResponseLineCount == 2)
                  {
                    Serial.println("This Line Was empty Two Times");//headers ended
                    Serial.println("Breaking While() Now");
                    httpResponseLineCount = 0;
                    break;
                  }

                }

                if (line.startsWith("HTTP/1.1"))// Check if the HTTP Response is 200, else break and Exit Update.
                {
                  if (line.indexOf("200") < 0)
                  {
                    Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
                    break;
                  }
                }

                if (line.startsWith("Content-Length: "))// extract headers here, Start with content length
                {
                  contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
                  Serial.println("Got " + String(contentLength) + " bytes from server");
                }

                if (line.startsWith("Content-Type: "))// Next, the content type
                {
                  String contentType = getHeaderValue(line, "Content-Type: ");
                  Serial.println("Got " + contentType + " payload.");
                  if (contentType == "text/plain")
                  {
                    isValidContentType = true;
                  }
                }

              }//-------------------------------------while

              Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

              if (contentLength && isValidContentType)// check contentLength and content type
              {
                String GetVersion = Serial2.readStringUntil('#');

                GetVersion = GetVersion.substring(GetVersion.indexOf("$") + 1, GetVersion.indexOf("#"));
                Serial.println("Found Firmware Version is:" + GetVersion);

                if (GetVersion == FirmwareVer)
                {
                  Serial.println("Device is already on the latest Firmware Version:" + GetVersion );
                  value = false;
                }

                if (GetVersion != FirmwareVer)
                {
                  Serial.println("This is a New Firmware Version" + GetVersion );
                  value = true;
                }
              }

              else
              {
                Serial.println("There was no content in the response");
                Serial2.flush();
                value = false;
              }

            }

            input = Serial2.readString();
            input.remove(0);
            Serial2.write("AT+QHTTPSTOP\r\n");
            delay(100);
            input = Serial2.readString();
            input.remove(0);
          }

          else
          {
            Serial.println("HTTP RESPONSE ERROR");
            Serial.println();
            input.remove(0);
            value = false;
          }

        }

        else
        {
          Serial.println("URL CONNECT FAILED");
          Serial.println();
          value = false;
        }

      }

      else
      {
        Serial.println("GPRS CONNECT FAILED");
        Serial.println();
        value = false;
      }

    }

    else if (GSM_Network_STS == false)
    {
      value = false;
    }


  }

  if (WiFi.status() == WL_CONNECTED)
  {
    WiFiClient * client = new WiFiClient;
    if (client)
    {
      HTTPClient http;
      if (http.begin( * client, fwurl))
      {
        httpCode = http.GET();
        Serial.print("HTTP_Code Recieved : ");
        Serial.print(httpCode);
        Serial.println();
        delay(1000);
        if (httpCode == HTTP_CODE_OK) // if version received
        {
          payload = http.getString();
          payload = payload.substring(payload.indexOf("$") + 1, payload.indexOf("#"));

          Serial.println("Found Firmware Version is: " + payload );
          Serial.println();

          if (payload == FirmwareVer)
          {
            Serial.printf("\nDevice is already on the latest Firmware Version:%s\n", payload);
            Serial.println();
            value = false;
          }

          else if (payload != FirmwareVer)
          {
            Serial.println("This is a New Firmware Version: " + payload);
            Serial.println();
            value = true;
          }

        }

        else
        {
          Serial.print("Error in Downloading version file:");
          Serial.println(httpCode);
          Serial.println();
          value = false;
        }

        http.end();
      }
      delete client;
    }

  }

  return value;
}

//----------------------------------------------------------------------------------

bool DoFirmwareUpdate()
{
  Serial.println("<<<  ATTENTION !!! DO NOT TURN OFF THE DEVICE...  >>>"); Serial.println();
  Serial.println("Updating the Device Firmware Now!"); Serial.println();
  Serial.println("Please Wait untill the Device Restarts"); Serial.println();
  bool value;
  String fwurl = "";
  fwurl = URL_fw_Bin;
  Serial.println(fwurl); Serial.println();

  if (WiFi.status() != WL_CONNECTED)
  {
    Network_Check_GSM();
    if (GSM_Network_STS == true)
    {
      Serial2.write("AT+QIACT?\r\n");
      delay(100);
      input = Serial2.readString();
      if (input.indexOf("OK") >= 0)
      {
        Serial.println("GPRS CONNECT SUCCESSFULLY");
        Serial.println();
        input = Serial2.readString();
        input.remove(0);
        Serial2.print("AT+QHTTPURL=79,30\r\n");
        delay(100);
        input = Serial2.readString();
        if (input.indexOf("CONNECT") >= 0)
        {
          Serial2.print(fwurl);
          delay(100);
          input = Serial2.readString();
          if (input.indexOf("OK") >= 0)
          {
            Serial.println("URL CONNECT SUCCESSFULLY");
            Serial.println();
            input = Serial2.readString();
            input.remove(0);
            int count = 0;
            Serial2.write("AT+QHTTPGET=30\r\n");
            input = Serial2.readString();
            while (input.indexOf("+QHTTPGET") < 0)
            {
              Serial.println("WAITING FOR GET COMMAND RESPONSE");
              Serial.println();
              if (count >= 5)
              {
                break;
              }
              count++;
              input = Serial2.readString();
            }

            String GetResponse = input.substring(input.indexOf("+QHTTPGET") + 13, input.indexOf("+QHTTPGET") + 16);
            Serial.print("GET RESPONSE IS:");
            Serial.println(GetResponse);
            Serial.println();
            if (GetResponse == "200")
            {
              Serial.println("GET RESPONSE IS OK");
              Serial.println();
              input = Serial2.readString();
              input.remove(0);
              httpResponseLineCount = 0;
              Serial2.write("AT+QHTTPREAD=30\r\n");
              while (!Serial2.available())
              {
              }
              while (Serial2.available())
              {
                // read line till /n
                String line = Serial2.readStringUntil('\n');
                // remove space, to check if the line is end of headers
                Serial.print("line:");
                Serial.println(line);
                line.trim();

                // if the the line is empty,
                // this is end of headers
                // break the while and feed the
                // remaining `client` to the
                // Update.writeStream();
                if (!line.length())
                {
                  Serial.println("This Line Was empty one times");//headers ended
                  Serial.println("httpResponseLineCount is :" + String(httpResponseLineCount));
                  if (httpResponseLineCount == 1)
                  {
                    Serial.println("This Line Was empty two times");//headers ended
                    Serial.println("Breaking While() Now");
                    httpResponseLineCount = 0;
                    break; // and get the OTA started
                  }
                  httpResponseLineCount ++;
                  //break; // and get the OTA started
                }

                // Check if the HTTP Response is 200
                // else break and Exit Update
                if (line.startsWith("HTTP/1.1"))
                {
                  if (line.indexOf("200") < 0)
                  {
                    Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
                    break;
                  }
                }

                // extract headers here
                // Start with content length
                if (line.startsWith("Content-Length: "))
                {
                  contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
                  Serial.println("Got " + String(contentLength) + " bytes from server");
                }

                // Next, the content type
                if (line.startsWith("Content-Type: "))
                {
                  String contentType = getHeaderValue(line, "Content-Type: ");
                  Serial.println("Got " + contentType + " payload.");
                  if (contentType == "application/octet-stream")
                  {
                    isValidContentType = true;
                  }
                }

              }//-------------------------------------while

              Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

              // check contentLength and content type
              if (contentLength && isValidContentType)
              {
                // Check if there is enough to OTA Update
                bool canBegin = Update.begin(contentLength, 0, 2, 1, NULL);

                // If yes, begin
                if (canBegin)
                {
                  Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
                  // No activity would appear on the Serial monitor
                  // So be patient. This may take 2 - 5mins to complete


                  size_t written = Update.writeStream(Serial2);

                  if (written == contentLength)
                  {
                    Serial.println("Written : " + String(written) + " successfully");
                  }
                  else
                  {
                    Serial.println("Written only : " + String(written) + "/" + String(contentLength) + ". Retry?" );
                    // retry??
                    // execOTA();
                  }

                  if (Update.end())
                  {
                    Serial.println("OTA done!");
                    if (Update.isFinished())
                    {
                      Serial.println("Update successfully completed. Rebooting.");
                      ESP.restart();
                    }
                    else
                    {
                      Serial.println("Update not finished? Something went wrong!");
                    }
                  }
                  else
                  {
                    Serial.println("Error Occurred. Error #: " + String(Update.getError()));
                  }
                }
                else
                {
                  // not enough space to begin OTA
                  // Understand the partitions and
                  // space availability
                  Serial.println("Not enough space to begin OTA");
                  Serial2.flush();
                }
              }
              else
              {
                Serial.println("There was no content in the response");
                Serial2.flush();
              }

            }

            input = Serial2.readString();
            input.remove(0);
            Serial2.write("AT+QHTTPSTOP\r\n");
            delay(100);
            input = Serial2.readString();
            input.remove(0);
          }

          else
          {
            Serial.println("HTTP RESPONSE ERROR");
            Serial.println();
            input.remove(0);
            value = false;
          }

        }

        else
        {
          Serial.println("URL CONNECT FAILED");
          Serial.println();
          value = false;
        }

      }

      else
      {
        Serial.println("GPRS CONNECT FAILED");
        Serial.println();
        value = false;
      }

      input = Serial2.readString();
      input.remove(0);
      Serial2.write("AT+QIDEACT=1\r\n");
      delay(100);
      input = Serial2.readString();
      input.remove(0);

    }

    else if (GSM_Network_STS == false)
    {
      value = false;
    }


  }

  if (WiFi.status() == WL_CONNECTED)
  {
    WiFiClient client;
    t_httpUpdate_return ret = httpUpdate.update(client, URL_fw_Bin);

    switch (ret)
    {
      case HTTP_UPDATE_FAILED:
        Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", httpUpdate.getLastError(), httpUpdate.getLastErrorString().c_str());
        break;

      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("HTTP_UPDATE_NO_UPDATES");
        break;

      case HTTP_UPDATE_OK:
        Serial.println("HTTP_UPDATE_OK");
        break;
    }

  }

}

//----------------------------------------------------------------------------------

String getHeaderValue(String header, String headerName)
{
  return header.substring(strlen(headerName.c_str()));
}

//----------------------------------------------------------------------------------

Debug Message

[4-25 14:46:22.2]FoxTerm Info: Logging started.
[4-25 14:46:23.4]ets Jun  8 2016 00:22:57
[4-25 14:46:23.4]
[4-25 14:46:23.4]rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)C¡¬ËË¥�Í¥Áé 0, SPIWP:0xee
[4-25 14:46:23.4]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00C¡ë��éDIO, clock div:1
[4-25 14:46:23.4]load:0x3fff0018,len:4
[4-25 14:46:23.4]load:0x3fff001c,len:1216
[4-25 14:46:23.4]ho 0 tail 12 room 4
[4-25 14:46:23.4]load'ÂÑ0078000,len:10944
[4-25 14:46:23.4]load:0x40080400,len:6388
[4-25 14:46:23.4]entry 0x400806b4
[4-25 14:46:23.7]��¨µUÍÁÍ2-hal-cpu.c:189] setCpuFrequencyMhz(): PLL: 320 / 4 = 80 Mhz, APB: 80000000 Hz


[4-25 14:46:25.2]<<--Developed By SILICIS ROAD INDUSTRIES LLP copyright2022-->>
[4-25 14:46:25.2]
[4-25 14:46:25.2]<----------BOOTING UP THE DEVICE---------->
[4-25 14:46:25.2]
[4-25 14:46:25.3]Flash Memory Begin Successfully
[4-25 14:46:25.3]
[4-25 14:46:25.3]Listing directory: /
[4-25 14:46:30.7]Listing directory: /
[4-25 14:46:30.7]EEPROM Begin Successfully
[4-25 14:46:30.7]
[4-25 14:46:30.7]Last Data Upload Interval = 1
[4-25 14:46:30.7]
[4-25 14:46:30.7]Files Stored In Flash = 0
[4-25 14:46:30.7]
[4-25 14:46:30.7]SSID :
[4-25 14:46:30.7]
[4-25 14:46:30.7]PASS :
[4-25 14:46:30.7]
[4-25 14:46:30.9][D][WiFiGeneric.cpp:374] _eventCallback(): Event: 0 - WIFI_READY
[4-25 14:46:30.9][D][WiFiGeneric.cpp:374] _eventCallback(): Event: 2 - STA_START
[4-25 14:46:32.4][E][WiFiMulti.cpp:55] addAP(): [WIFI][APlistAdd] no ssid or ssid too long
[4-25 14:46:32.4][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: silicis
[4-25 14:46:32.4][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: PM12
[4-25 14:46:32.4][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: PM13
[4-25 14:46:32.4]
[4-25 14:46:34.7][D][WiFiGeneric.cpp:374] _eventCallback(): Event: 1 - SCAN_DONE
[4-25 14:46:34.7][I][WiFiMulti.cpp:114] run(): [WIFI] scan done
[4-25 14:46:34.7][I][WiFiMulti.cpp:119] run(): [WIFI] 3 networks found
[4-25 14:46:34.7][D][WiFiMulti.cpp:151] run():        0: [5][52:9B:68:86:29:7B] PM (-48) *
[4-25 14:46:34.7][D][WiFiMulti.cpp:151] run():        1: [4][24:0B:88:F7:B1:49] Tanmay wifi-2.4g (-79) *
[4-25 14:46:34.7][D][WiFiMulti.cpp:151] run():        2: [13][50:2B:73:3D:EE:60] Tenda_3DEE60 (-92) *
[4-25 14:46:34.7][E][WiFiMulti.cpp:191] run(): [WIFI] no matching wifi found!
[4-25 14:46:35.7]UNABLE TO CONNECT ANY WIFI
[4-25 14:46:35.7]
[4-25 14:46:35.8]CHECKING GSM MODULE IN 10Sec.
[4-25 14:46:35.8]
[4-25 14:46:50.0]MODULE OK
[4-25 14:46:50.0]
[4-25 14:46:50.0]SETTING OTHER PARAMETERS TO GSM
[4-25 14:46:50.0]
[4-25 14:46:52.3]IMEI : 861190059737848
[4-25 14:46:52.3]
[4-25 14:46:53.9]CHECKING FOR SIM CARD
[4-25 14:46:53.9]
[4-25 14:46:55.0]SIM CARD FOUND OK
[4-25 14:46:55.0]
[4-25 14:46:56.1]GSM INITIALIZATION COMPLETE
[4-25 14:46:56.1]
[4-25 14:46:57.2]GSM IS REGISTERED ON NETWORK
[4-25 14:46:57.2]
[4-25 14:46:58.4]NETWORK TIME SYNC SUCCESSFUL
[4-25 14:46:58.4]
[4-25 14:46:59.5]SETTING SMS COMMANDS
[4-25 14:46:59.5]
[4-25 14:47:00.1]SETTING HTTP COMMANDS
[4-25 14:47:00.1]
[4-25 14:47:03.5]<--APN SET FOR JIO-->
[4-25 14:47:03.5]
[4-25 14:47:06.0]STARTING GPRS/PDP CONTEXT CONNECTION
[4-25 14:47:08.0]CONTEXT PROFILE IS OK
[4-25 14:47:08.0]
[4-25 14:47:09.1]GPRS/PDP CONTEXT PROFILE IS ACTIVATED
[4-25 14:47:09.1]
[4-25 14:47:10.1]
[4-25 14:47:10.1]+QIACT: 1,1,1,"26.85.155.38"
[4-25 14:47:10.1]
[4-25 14:47:10.1]OK
[4-25 14:47:10.1]
[4-25 14:47:10.1]Device Firmware Version is : 1.0
[4-25 14:47:10.1]
[4-25 14:47:10.1]Checking Firmware update...
[4-25 14:47:10.1]
[4-25 14:47:10.1]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/fw_version.txt
[4-25 14:47:10.1]
[4-25 14:47:11.1]GSM REGISTERED ON NETWORK
[4-25 14:47:11.1]
[4-25 14:47:12.1]Signal Strength : 31
[4-25 14:47:12.1]
[4-25 14:47:13.2]GPRS CONNECT SUCCESSFULLY
[4-25 14:47:13.2]
[4-25 14:47:16.4]URL CONNECT SUCCESSFULLY
[4-25 14:47:16.4]
[4-25 14:47:18.4]WAITING FOR GET COMMAND RESPONSE
[4-25 14:47:18.4]
[4-25 14:47:19.6]GET RESPONSE IS:200
[4-25 14:47:19.6]
[4-25 14:47:19.6]GET RESPONSE IS OK
[4-25 14:47:19.6]
[4-25 14:47:20.6]line is:
[4-25 14:47:20.6]This Line Was empty
[4-25 14:47:20.6]httpResponseLineCount is :1
[4-25 14:47:20.6]line is:CONNECT
[4-25 14:47:20.6]line is:HTTP/1.1 200 OK
[4-25 14:47:20.6]line is:Server: nginx/1.18.0 (Ubuntu)
[4-25 14:47:20.6]line is:Date: Mon, 25 Apr 2022 09:17:30 GMT
[4-25 14:47:20.6]line is:Content-Type: text/plain
[4-25 14:47:20.6]Got text/plain payload.
[4-25 14:47:20.6]line is:Content-Length: 5
[4-25 14:47:20.6]Got 5 bytes from server
[4-25 14:47:20.6]line is:Last-Modified: Tue, 19 Apr 2022 03:56:14 GMT
[4-25 14:47:20.8]line is:Connection: keep-alive
[4-25 14:47:20.8]line is:ETag: "625e32de-5"
[4-25 14:47:20.8]line is:Accept-Ranges: bytes
[4-25 14:47:20.8]line is:
[4-25 14:47:20.8]This Line Was empty
[4-25 14:47:20.8]httpResponseLineCount is :2
[4-25 14:47:20.8]This Line Was empty Two Times
[4-25 14:47:20.8]Breaking While() Now
[4-25 14:47:20.8]contentLength : 5, isValidContentType : 1
[4-25 14:47:20.8]Found Firmware Version is:1.1
[4-25 14:47:20.8]This is a New Firmware Version1.1
[4-25 14:47:22.8]Starting FOTA to update New Firmware
[4-25 14:47:22.8]
[4-25 14:47:22.8]<<<  ATTENTION !!! DO NOT TURN OFF THE DEVICE...  >>>
[4-25 14:47:22.8]
[4-25 14:47:22.8]Updating the Device Firmware Now!
[4-25 14:47:22.8]
[4-25 14:47:22.8]Please Wait untill the Device Restarts
[4-25 14:47:22.8]
[4-25 14:47:22.8]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/update.bin
[4-25 14:47:22.8]
[4-25 14:47:23.8]GSM REGISTERED ON NETWORK
[4-25 14:47:23.8]
[4-25 14:47:24.8]Signal Strength : 31
[4-25 14:47:24.8]
[4-25 14:47:25.9]GPRS CONNECT SUCCESSFULLY
[4-25 14:47:25.9]
[4-25 14:47:29.1]URL CONNECT SUCCESSFULLY
[4-25 14:47:29.1]
[4-25 14:47:31.3]GET RESPONSE IS:200
[4-25 14:47:31.3]
[4-25 14:47:31.3]GET RESPONSE IS OK
[4-25 14:47:31.3]
[4-25 14:47:32.3]line:
[4-25 14:47:32.3]This Line Was empty one times
[4-25 14:47:32.3]httpResponseLineCount is :0
[4-25 14:47:32.3]line:CONNECT
[4-25 14:47:32.3]line:HTTP/1.1 200 OK
[4-25 14:47:32.3]line:Server: nginx/1.18.0 (Ubuntu)
[4-25 14:47:32.3]line:Date: Mon, 25 Apr 2022 09:17:41 GMT
[4-25 14:47:32.3]line:Content-Type: application/octet-stream
[4-25 14:47:32.3]Got application/octet-stream payload.
[4-25 14:47:32.3]line:Content-Length: 848560
[4-25 14:47:32.3]Got 848560 bytes from server
[4-25 14:47:32.3]line:Last-Modified: Sun, 24 Apr 2022 15:25:09 GMT
[4-25 14:47:32.3]line:Connection: keep-alive
[4-25 14:47:32.3]line:ETag: "62656bd5-cf2b0"
[4-25 14:47:32.3]line:Accept-Ranges: bytes
[4-25 14:47:32.3]line:
[4-25 14:47:32.3]This Line Was empty one times
[4-25 14:47:32.3]httpResponseLineCount is :1
[4-25 14:47:32.3]This Line Was empty two times
[4-25 14:47:32.3]Breaking While() Now
[4-25 14:47:32.3]contentLength : 848560, isValidContentType : 1
[4-25 14:47:32.3][D][Updater.cpp:132] begin(): OTA Partition: app1
[4-25 14:47:32.3]Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
[4-25 14:54:16.9]Written only : 784906/848560. Retry?
[4-25 14:54:16.9]Error Occurred. Error #: 6
[4-25 14:54:23.1]<----------BOOT PROCESS COMPLETE---------->
[4-25 14:54:23.1]

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

  • I confirm I have checked existing issues, online documentation and Troubleshooting guide.
@mudgalp mudgalp added the Status: Awaiting triage Issue is waiting for triage label Apr 27, 2022
@SuGlider
Copy link
Collaborator

@mudgalp
We are not supporting Arduino Core 1.0.6 anymore.
Would it be possible for you to test it using the latest Arduino Core Version (2.0.2 or 2.0.3-RC1)?

Thanks!

@SuGlider SuGlider self-assigned this Apr 27, 2022
@SuGlider SuGlider added Resolution: Awaiting response Waiting for response of author and removed Status: Awaiting triage Issue is waiting for triage labels Apr 27, 2022
@mudgalp
Copy link
Author

mudgalp commented Apr 27, 2022

tried updating the Esp32 Arduino core to 2.0.2
also updated the arduino ide to 1.8.19

still getting same response. please help

here is the new log recorded out of the device.

Thank you

[4-28 01:42:27.3]Starting FOTA to update New Firmware
[4-28 01:42:27.3]
[4-28 01:42:27.3]<<< ATTENTION !!! DO NOT TURN OFF THE DEVICE... >>>
[4-28 01:42:27.3]
[4-28 01:42:27.3]Updating the Device Firmware Now!
[4-28 01:42:27.3]
[4-28 01:42:27.3]Please Wait untill the Device Restarts
[4-28 01:42:27.3]
[4-28 01:42:27.3]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/update.bin
[4-28 01:42:27.3]
[4-28 01:42:28.3]GSM REGISTERED ON NETWORK
[4-28 01:42:28.3]
[4-28 01:42:29.4]Signal Strength : 20
[4-28 01:42:29.4]
[4-28 01:42:30.5]GPRS CONNECT SUCCESSFULLY
[4-28 01:42:30.5]
[4-28 01:42:35.1]URL CONNECT SUCCESSFULLY
[4-28 01:42:35.1]
[4-28 01:42:37.8]WAITING FOR GET COMMAND RESPONSE
[4-28 01:42:37.8]
[4-28 01:42:38.8]WAITING FOR GET COMMAND RESPONSE
[4-28 01:42:38.8]
[4-28 01:42:40.4]GET RESPONSE IS:200
[4-28 01:42:40.4]
[4-28 01:42:40.4]GET RESPONSE IS OK
[4-28 01:42:40.4]
[4-28 01:42:41.4]line:
[4-28 01:42:41.4]This Line Was empty one times
[4-28 01:42:41.4]httpResponseLineCount is :0
[4-28 01:42:41.4]line:CONNECT
[4-28 01:42:41.4]line:HTTP/1.1 200 OK
[4-28 01:42:41.4]line:Server: nginx/1.18.0 (Ubuntu
[4-28 01:42:41.4]line:Date: Wed, 27Apr 2022 20:12:3� GMT
[4-28 01:42:41.4]line:Content-Type: application/octet-stream
[4-28 01:42:41.4]Got application/octet-stream payload.
[4-28 01:42:41.4]line:Content-Length: 848560
[4-28 01:42:41.4]Got 848560 bytes from server
[4-28 01:42:41.4]line:Last-Modified: Sun, 24 Apr 2022 15:25:09 GMT
[4-28 01:42:41.4]line:Connection: kee`-alive
[4-28 01:42:41.4]line:ETag: "62656bd5-cf2b0"
[4-28 01:42:41.4]line:Accept-Ranges: bytes
[4-28 01:42:41.4]line:
[4-28 01:42:41.4]This Line Was empty one times
[4-28 01:42:41.4]httpResponseLineCount is :1
[4-28 01:42:41.4]This Line Was empty two times
[4-28 01:42:41.4]Breaking While() Now
[4-28 01:42:41.4]contentLength : 848560, isValidContentType : 1
[4-28 01:42:41.4][ 75057][D][Updater.cpp:133] begin(): OTA Partition: app1
[4-28 01:42:41.4]Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
[4-28 01:49:25.8]Written only : 743006/848560. Retry?
[4-28 01:49:25.8]Error Occurred. Error #: 6

@VojtechBartoska VojtechBartoska moved this to Under investigation in Arduino ESP32 Core Project Roadmap Apr 28, 2022
@mudgalp
Copy link
Author

mudgalp commented May 4, 2022

waiting for response...

@VojtechBartoska VojtechBartoska added Status: Needs investigation We need to do some research before taking next steps on this issue Area: BT&Wifi BT & Wifi related issues and removed Resolution: Awaiting response Waiting for response of author labels May 4, 2022
@mudgalp
Copy link
Author

mudgalp commented May 10, 2022 •
edited
Loading

Tried printing the error using Update.printError(Serial); here are the logs.

its says Error Occurred. Error #: 6

Stream Read Timeout

'
//-----------------------------------------------------------------------------------------------------------------

[5-10 09:42:22.3]<----------BOOTING UP THE DEVICE---------->
[5-10 09:42:22.3]
[5-10 09:42:22.3]EEPROM Begin Successfully
[5-10 09:42:22.3]
[5-10 09:42:22.3]Last Data Upload Interval = 1
[5-10 09:42:22.3]
[5-10 09:42:22.3]Files Stored In Flash = 0
[5-10 09:42:22.3]
[5-10 09:42:22.3]SSID :
[5-10 09:42:22.3]
[5-10 09:42:22.3]PASS :
[5-10 09:42:22.3]
[5-10 09:42:22.3][ 2064][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 0 - WIFI_READY
[5-10 09:42:22.5][ 2166][V][WiFiGeneric.cpp:283] _arduino_event_cb(): STA Started
[5-10 09:42:22.5][ 2167][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 2 - STA_START
[5-10 09:42:24.0][ 3667][E][WiFiMulti.cpp:55] addAP(): [WIFI][APlistAdd] no ssid or ssid too long
[5-10 09:42:24.0][ 3667][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: silicis
[5-10 09:42:24.0][ 3669][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: PM12
[5-10 09:42:24.0][ 3676][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: PM13
[5-10 09:42:24.0]CONNECTING TO WIFI...
[5-10 09:42:24.0]
[5-10 09:42:26.2][ 5918][V][WiFiGeneric.cpp:317] _arduino_event_cb(): SCAN Done: ID: 128, Status: 0, Results: 2
[5-10 09:42:26.2][ 5919][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 1 - SCAN_DONE
[5-10 09:42:26.2][ 5923][I][WiFiMulti.cpp:114] run(): [WIFI] scan done
[5-10 09:42:26.2][ 5928][I][WiFiMulti.cpp:119] run(): [WIFI] 2 networks found
[5-10 09:42:26.2][ 5933][D][WiFiMulti.cpp:151] run(): 0: [11][52:9B:68:86:29:7B] PM (-45) *
[5-10 09:42:26.2][ 5941][D][WiFiMulti.cpp:151] run(): 1: [6][40:95:BD:0C:05:0C] JioFi_10C050C (-88) *
[5-10 09:42:26.3][ 5949][E][WiFiMulti.cpp:191] run(): [WIFI] no matching wifi found!
[5-10 09:42:27.3]UNABLE TO CONNECT ANY WIFI
[5-10 09:42:27.3]
[5-10 09:42:27.4]CHECKING GSM MODULE IN 10Sec.
[5-10 09:42:27.4]
[5-10 09:42:39.9]MODULE OK
[5-10 09:42:39.9]
[5-10 09:42:39.9]SETTING OTHER PARAMETERS TO GSM
[5-10 09:42:39.9]
[5-10 09:42:42.1]IMEI : 861190059737848
[5-10 09:42:42.1]
[5-10 09:42:43.7]CHECKING FOR SIM CARD
[5-10 09:42:43.7]
[5-10 09:42:44.8]SIM CARD FOUND OK
[5-10 09:42:44.8]
[5-10 09:42:45.9]GSM INITIALIZATION COMPLETE
[5-10 09:42:45.9]
[5-10 09:42:47.0]GSM IS REGISTERED ON NETWORK
[5-10 09:42:47.0]
[5-10 09:42:48.2]NETWORK TIME SYNC SUCCESSFUL
[5-10 09:42:48.2]
[5-10 09:42:49.3]SETTING SMS COMMANDS
[5-10 09:42:49.3]
[5-10 09:42:49.9]SETTING HTTP COMMANDS
[5-10 09:42:49.9]
[5-10 09:42:53.2]<--APN SET FOR JIO-->
[5-10 09:42:53.2]
[5-10 09:42:55.8]STARTING GPRS/PDP CONTEXT CONNECTION
[5-10 09:42:57.8]CONTEXT PROFILE IS OK
[5-10 09:42:57.8]
[5-10 09:42:58.8]GPRS/PDP CONTEXT PROFILE IS ACTIVATED
[5-10 09:42:58.8]
[5-10 09:42:59.8]
[5-10 09:42:59.8]+QIACT: 1,1,1,"25.117.218.4"
[5-10 09:42:59.8]
[5-10 09:42:59.8]OK
[5-10 09:42:59.8]
[5-10 09:42:59.8]Device Firmware Version is : 1.0
[5-10 09:42:59.8]
[5-10 09:42:59.8]Checking Firmware update...
[5-10 09:42:59.8]
[5-10 09:42:59.8]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/fw_version.txt
[5-10 09:42:59.8]
[5-10 09:43:00.8]GSM REGISTERED ON NETWORK
[5-10 09:43:00.8]
[5-10 09:43:01.8]Signal Strength : 31
[5-10 09:43:01.8]
[5-10 09:43:02.9]GPRS CONNECT SUCCESSFULLY
[5-10 09:43:02.9]
[5-10 09:43:06.1]URL CONNECT SUCCESSFULLY
[5-10 09:43:06.1]
[5-10 09:43:08.1]WAITING FOR GET COMMAND RESPONSE
[5-10 09:43:08.1]
[5-10 09:43:09.5]GET RESPONSE IS:200
[5-10 09:43:09.5]
[5-10 09:43:09.5]GET RESPONSE IS OK
[5-10 09:43:09.5]
[5-10 09:43:10.5]line is:
[5-10 09:43:10.5]This Line Was empty
[5-10 09:43:10.5]httpResponseLineCount is :1
[5-10 09:43:10.5]line is:CONNECT
[5-10 09:43:10.5]line is:HTTP/1.1 200 OK
[5-10 09:43:10.5]line is:Server: nginx/1.18.0 (Ubuntu)
[5-10 09:43:10.5]line is:Date: Tue, 10 May 2022 04:13:08 GMT
[5-10 09:43:10.5]line is:Content-Type: text/plain
[5-10 09:43:10.5]Got text/plain payload.
[5-10 09:43:10.5]line is:Content-Length: 5
[5-10 09:43:10.5]Got 5 bytes from server
[5-10 09:43:10.5]line is:Last-Modified: Tue, 19 Apr 2022 03:56:14 GMT
[5-10 09:43:10.5]line is:Connection: keep-alive
[5-10 09:43:10.5]line is:ETag: "625e32de-5"
[5-10 09:43:10.5]line is:Accept-Ranges: bytes
[5-10 09:43:10.5]line is:
[5-10 09:43:10.5]This Line Was empty
[5-10 09:43:10.5]httpResponseLineCount is :2
[5-10 09:43:10.5]This Line Was empty Two Times
[5-10 09:43:10.5]Breaking While() Now
[5-10 09:43:10.5]contentLength : 5, isValidContentType : 1
[5-10 09:43:10.5]Found Firmware Version is:1.1
[5-10 09:43:10.5]This is a New Firmware Version1.1
[5-10 09:43:12.6]Starting FOTA to update New Firmware
[5-10 09:43:12.6]
[5-10 09:43:12.6]<<< ATTENTION !!! DO NOT TURN OFF THE DEVICE... >>>
[5-10 09:43:12.6]
[5-10 09:43:12.6]Updating the Device Firmware Now!
[5-10 09:43:12.6]
[5-10 09:43:12.6]Please Wait untill the Device Restarts
[5-10 09:43:12.6]
[5-10 09:43:12.6]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/update.bin
[5-10 09:43:12.6]
[5-10 09:43:13.7]GSM REGISTERED ON NETWORK
[5-10 09:43:13.7]
[5-10 09:43:14.7]Signal Strength : 31
[5-10 09:43:14.7]
[5-10 09:43:15.8]GPRS CONNECT SUCCESSFULLY
[5-10 09:43:15.8]
[5-10 09:43:19.0]URL CONNECT SUCCESSFULLY
[5-10 09:43:19.0]
[5-10 09:43:21.1]GET RESPONSE IS:200
[5-10 09:43:21.1]
[5-10 09:43:21.1]GET RESPONSE IS OK
[5-10 09:43:21.1]
[5-10 09:43:22.2]line:
[5-10 09:43:22.2]This Line Was empty one times
[5-10 09:43:22.2]httpResponseLineCount is :0
[5-10 09:43:22.2]line:CONNECT
[5-10 09:43:22.2]line:HTTP/1.1 200 OK
[5-10 09:43:22.2]line:Server: nginx/1.18.0 (Ubuntu)
[5-10 09:43:22.2]line:Date: Tue, 10 May 2022 04:13:20 GMT
[5-10 09:43:22.2]line:Content-Type: application/octet-stream
[5-10 09:43:22.2]Got application/octet-stream payload.
[5-10 09:43:22.2]line:Content-Length: 774096
[5-10 09:43:22.2]Got 774096 bytes from server
[5-10 09:43:22.2]line:Last-Modified: Mon, 09 May 2022 03:38:48 GMT
[5-10 09:43:22.2]line:Connection: keep-alive
[5-10 09:43:22.2]line:ETag: "62788cc8-bcfd0"
[5-10 09:43:22.2]line:Accept-Ranges: bytes
[5-10 09:43:22.2]line:
[5-10 09:43:22.2]This Line Was empty one times
[5-10 09:43:22.2]httpResponseLineCount is :1
[5-10 09:43:22.2]This Line Was empty two times
[5-10 09:43:22.2]Breaking While() Now
[5-10 09:43:22.2]contentLength : 774096, isValidContentType : 1
[5-10 09:43:22.2][ 61899][D][Updater.cpp:133] begin(): OTA Partition: app1
[5-10 09:43:22.2]Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
[5-10 09:50:00.0]Written only : 686073/774096. Retry?
[5-10 09:50:00.0]Stream Read Timeout
[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout
[5-10 09:50:06.2]<----------BOOT PROCESS COMPLETE---------->
[5-10 09:50:06.2]

//-----------------------------------------------------------------------------------------------------------------
'

@SuGlider
Copy link
Collaborator

I have no LTE modem/module in order to test and reproduce the issue with your sketch.
Could you simplify it to the smallest possible sketch that reproduces the error, not using the LTE module?

@SuGlider
Copy link
Collaborator

SuGlider commented May 10, 2022 •
edited
Loading

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.

You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/

Do it right after Serial2.begin()

@mudgalp
Copy link
Author

mudgalp commented May 11, 2022

ok I'll try and get back to you.
thanks

@mudgalp
Copy link
Author

mudgalp commented May 11, 2022 •
edited
Loading

I have no LTE modem/module in order to test and reproduce the issue with your sketch.
Could you simplify it to the smallest possible sketch that reproduces the error, not using the LTE module?

there is no error without the GSM/LTE module, it only occurs when the .bin file is being downloaded using GSM/LTE module and provided to the Update.WriteStream() function.

i'm performing OTA using WIFi and httpUpdate, its works fine everytime.

@mudgalp
Copy link
Author

mudgalp commented May 11, 2022

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.

You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/

Do it right after Serial2.begin()

i'll try this and get back to you.

@mudgalp
Copy link
Author

mudgalp commented May 11, 2022 •
edited
Loading

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.

You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/

Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000).
slowed every serial2 write and read event.
and didn't solve the issue.

@SuGlider
Copy link
Collaborator

I see.
What do you use as OTA server of the firmware?

@SuGlider SuGlider mentioned this issue May 11, 2022
Closed
1 task
@SuGlider
Copy link
Collaborator

From #6727 there is a hint...
Can you set a big RxBuffer for the Serial2?

  Serial2.setRxBufferSize(4096);
  Serial2.setTimeout(0);
  Serial2.begin(115200);

@primus192
Copy link

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.
You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/
Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000). slowed every serial2 write and read event. and didn't solve the issue.

While on arduino 1.0.6 Try setting setRxBufferSize(1024); and flush(); directly before executing Update.writeStream(Serial2); It fixed the problem for me in arduino 1.0.6. However this method did not work for me in new framework even if i set buffer size to 4096 or more before begin of the serial. GSM modules tend to hang once in a while and then throwing large chunks of data at once and the buffer overflows.

Try this way (arduino 1.0.6)

Serial2.setRxBufferSize(1024);
Serial2.flush();
if (Update.begin(_contentLength)) {
size_t written = Update.writeStream(Serial2);
if (written == _contentLength) {
log_i("Written: %s successfully.", String(written));
}

@mudgalp
Copy link
Author

mudgalp commented May 12, 2022

I see. What do you use as OTA server of the firmware?

im using linux based http server made by my team.

@mudgalp
Copy link
Author

mudgalp commented May 12, 2022

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.
You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/
Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000). slowed every serial2 write and read event. and didn't solve the issue.

While on arduino 1.0.6 Try setting setRxBufferSize(1024); and flush(); directly before executing Update.writeStream(Serial2); It fixed the problem for me in arduino 1.0.6. However this method did not work for me in new framework even if i set buffer size to 4096 or more before begin of the serial. GSM modules tend to hang once in a while and then throwing large chunks of data at once and the buffer overflows.

Try this way (arduino 1.0.6)

Serial2.setRxBufferSize(1024); Serial2.flush(); if (Update.begin(_contentLength)) { size_t written = Update.writeStream(Serial2); if (written == _contentLength) { log_i("Written: %s successfully.", String(written)); }

will try this and let you know.

Thank you

@mudgalp
Copy link
Author

mudgalp commented May 12, 2022

From #6727 there is a hint... Can you set a big RxBuffer for the Serial2?

  Serial2.setRxBufferSize(4096);
  Serial2.setTimeout(0);
  Serial2.begin(115200);

tried this, no response.

@mudgalp
Copy link
Author

mudgalp commented May 12, 2022 •
edited
Loading

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.
You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/
Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000). slowed every serial2 write and read event. and didn't solve the issue.

While on arduino 1.0.6 Try setting setRxBufferSize(1024); and flush(); directly before executing Update.writeStream(Serial2); It fixed the problem for me in arduino 1.0.6. However this method did not work for me in new framework even if i set buffer size to 4096 or more before begin of the serial. GSM modules tend to hang once in a while and then throwing large chunks of data at once and the buffer overflows.

Try this way (arduino 1.0.6)

Serial2.setRxBufferSize(1024); Serial2.flush(); if (Update.begin(_contentLength)) { size_t written = Update.writeStream(Serial2); if (written == _contentLength) { log_i("Written: %s successfully.", String(written)); }

I think this is the problem in your new framework not working issue.

### code

Serial2.setRxBufferSize(1024);
Serial2.flush();// Check if there is enough to OTA Update
bool canBegin = Update.begin(contentLength, 0, 2, 1, NULL);

debug output>>

[ 77722][E][HardwareSerial.cpp:494] setRxBufferSize(): RX Buffer can't be resized when Serial is already running.

@primus192
Copy link

primus192 commented May 12, 2022 •
edited
Loading

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.
You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/
Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000). slowed every serial2 write and read event. and didn't solve the issue.

While on arduino 1.0.6 Try setting setRxBufferSize(1024); and flush(); directly before executing Update.writeStream(Serial2); It fixed the problem for me in arduino 1.0.6. However this method did not work for me in new framework even if i set buffer size to 4096 or more before begin of the serial. GSM modules tend to hang once in a while and then throwing large chunks of data at once and the buffer overflows.
Try this way (arduino 1.0.6)
Serial2.setRxBufferSize(1024); Serial2.flush(); if (Update.begin(_contentLength)) { size_t written = Update.writeStream(Serial2); if (written == _contentLength) { log_i("Written: %s successfully.", String(written)); }

I think this is the problem in your new framework not working issue.

### code

Serial2.setRxBufferSize(1024); Serial2.flush();// Check if there is enough to OTA Update bool canBegin = Update.begin(contentLength, 0, 2, 1, NULL);

debug output>>

[ 77722][E][HardwareSerial.cpp:494] setRxBufferSize(): RX Buffer can't be resized when Serial is already running.

I know that, i moved this before serial begin and it still did not work in new framework. This solution only works for me in 1.0.6. Tried doing this in 1.0.6?

@SuGlider
Copy link
Collaborator

SuGlider commented May 12, 2022 •
edited
Loading

Possible Arduino Stream Class issue related to #6733 (comment)

Investigating it...

@mudgalp
Copy link
Author

mudgalp commented May 13, 2022

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.
You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/
Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000). slowed every serial2 write and read event. and didn't solve the issue.

While on arduino 1.0.6 Try setting setRxBufferSize(1024); and flush(); directly before executing Update.writeStream(Serial2); It fixed the problem for me in arduino 1.0.6. However this method did not work for me in new framework even if i set buffer size to 4096 or more before begin of the serial. GSM modules tend to hang once in a while and then throwing large chunks of data at once and the buffer overflows.
Try this way (arduino 1.0.6)
Serial2.setRxBufferSize(1024); Serial2.flush(); if (Update.begin(_contentLength)) { size_t written = Update.writeStream(Serial2); if (written == _contentLength) { log_i("Written: %s successfully.", String(written)); }

I think this is the problem in your new framework not working issue.
### code
Serial2.setRxBufferSize(1024); Serial2.flush();// Check if there is enough to OTA Update bool canBegin = Update.begin(contentLength, 0, 2, 1, NULL);
debug output>>
[ 77722][E][HardwareSerial.cpp:494] setRxBufferSize(): RX Buffer can't be resized when Serial is already running.

I know that, i moved this before serial begin and it still did not work in new framework. This solution only works for me in 1.0.6. Tried doing this in 1.0.6?

i'll try that now.

@mudgalp
Copy link
Author

mudgalp commented May 15, 2022

[5-10 09:50:00.0]Error Occurred. Error #: 6
[5-10 09:50:00.0]Stream Read Timeout

Apparently there is an issue with the communication to the OTA server over LTE, which is timing out.
You can try to increase the timeout, by using Serial2.setTimeout(milliseconds) -- default of Stream Class is 1000 (1 second) time out. https://www.arduino.cc/reference/en/language/functions/communication/stream/streamsettimeout/
Do it right after Serial2.begin()

tried this , Serial2.setTimeout(5000). slowed every serial2 write and read event. and didn't solve the issue.

While on arduino 1.0.6 Try setting setRxBufferSize(1024); and flush(); directly before executing Update.writeStream(Serial2); It fixed the problem for me in arduino 1.0.6. However this method did not work for me in new framework even if i set buffer size to 4096 or more before begin of the serial. GSM modules tend to hang once in a while and then throwing large chunks of data at once and the buffer overflows.
Try this way (arduino 1.0.6)
Serial2.setRxBufferSize(1024); Serial2.flush(); if (Update.begin(_contentLength)) { size_t written = Update.writeStream(Serial2); if (written == _contentLength) { log_i("Written: %s successfully.", String(written)); }

I think this is the problem in your new framework not working issue.
### code
Serial2.setRxBufferSize(1024); Serial2.flush();// Check if there is enough to OTA Update bool canBegin = Update.begin(contentLength, 0, 2, 1, NULL);
debug output>>
[ 77722][E][HardwareSerial.cpp:494] setRxBufferSize(): RX Buffer can't be resized when Serial is already running.

I know that, i moved this before serial begin and it still did not work in new framework. This solution only works for me in 1.0.6. Tried doing this in 1.0.6?

tried this... no success.

@SuGlider
Copy link
Collaborator

Try this:

Xinyuan-LilyGO/LilyGo-T-Call-SIM800#132 (comment)

@mudgalp
Copy link
Author

mudgalp commented May 20, 2022

I'm not using GSM Library and if i want to that this library does not support the LTE Module I'm using in my project.

@mudgalp
Copy link
Author

mudgalp commented May 20, 2022 •
edited
Loading

but i tried this method using my AT commands and getting issue in this loop.

here the problem is that this while loop break before all data bytes downloads.

that means the data available on the server will not be downloaded completely and this while loop will break.

 while (Serial2.available())
        {
                // read file data to spiffs
            if (!file.print(char(client.read())))
            {
                Serial.println("Appending file");
            }
            //Serial.print((char)c);       // Uncomment this to show data
            //crc.update(c);
            readLength++;

            if (readLength % (contentLength / 13) == 0)
            {
                printPercent(readLength, contentLength);
            }
            timeout = millis();
        }

@SuGlider
Copy link
Collaborator

SuGlider commented Jun 6, 2022

There is a possible work around that may fix this issue.

Pease try this code/example when starting Serial2 (used to communicate to the Modem and then receive the OTA firmware):

#include "driver/uart.h"  // necessary for the function uart_set_always_rx_timeout()
void setup(){
        Serial2.setRxBufferSize(4096);
        Serial2.begin(115200);
        Serial2.setRxTimeout(1);
        uart_set_always_rx_timeout(2, true); // 2 is for Serial1, in this case
        Serial2.flush();
}

Please let me know if this solves the issue.

@mudgalp
Copy link
Author

mudgalp commented Jul 3, 2022

Good Day all.
There are few updates with the problem.

1-----------------------------------------

#include "driver/uart.h"
void setup(){
Serial1.setRxBufferSize(1024);
Serial1.begin(115200,SERIAL_8N1,26,27);
Serial1.setRxTimeout(1);
uart_set_always_rx_timeout(1, true); // 1 is for Serial1, in this case
Serial1.flush();
}

dear @SuGlider , I tried this but there is no difference in the problem.

2-------------------------------------------------------------------------------------------
Other thing is this that i discussed this problem with the Quectel's Technical support team and they asked me to try using slower baud rate like for example 9600 and i tried so, and it worked for some part.

now the esp32 takes all the binary file and writes it to the ota partition successfully but it does not gets restart on the newer firmware version instead it gives me a different error as follows.

the function i used
`

bool DoFirmwareUpdate()
{
Serial.println("<<< ATTENTION !!! DO NOT TURN OFF THE DEVICE... >>>"); Serial.println();
Serial.println("Updating the Device Firmware Now!"); Serial.println();
Serial.println("Please Wait untill the Device Restarts"); Serial.println();
bool value;
String fwurl = "";
fwurl = URL_fw_Bin;
Serial.println(fwurl); Serial.println();
if (WiFi.status() != WL_CONNECTED)
{
Network_Check_GSM();
if (GSM_Network_STS == true)
{
Serial2.write("AT+QIACT?\r\n");
delay(100);
input = Serial2.readString();
if (input.indexOf("OK") >= 0)
{
Serial.println("GPRS CONNECT SUCCESSFULLY");
Serial.println();
input = Serial2.readString();
input.remove(0);
Serial2.print("AT+QHTTPURL=79,30\r\n");
delay(100);
input = Serial2.readString();
if (input.indexOf("CONNECT") >= 0)
{
Serial2.print(fwurl);
delay(100);
input = Serial2.readString();
if (input.indexOf("OK") >= 0)
{
Serial.println("URL CONNECT SUCCESSFULLY");
Serial.println();
input = Serial2.readString();
input.remove(0);
int count = 0;
Serial2.write("AT+QHTTPGET=30\r\n");
input = Serial2.readString();
while (input.indexOf("+QHTTPGET") < 0)
{
Serial.println("WAITING FOR GET COMMAND RESPONSE");
Serial.println();
if (count >= 5)
{
break;
}
count++;
input = Serial2.readString();
}
String GetResponse = input.substring(input.indexOf("+QHTTPGET") + 13, input.indexOf("+QHTTPGET") + 16);
Serial.print("GET RESPONSE IS:");
Serial.println(GetResponse);
Serial.println();
if (GetResponse == "200")
{
Serial.println("GET RESPONSE IS OK");
Serial.println();
input = Serial2.readString();
input.remove(0);
httpResponseLineCount = 0;
Serial2.write("AT+QHTTPREAD=30\r\n");
while (!Serial2.available())
{
}
while (Serial2.available())
{
// read line till /n
String line = Serial2.readStringUntil('\n');
// remove space, to check if the line is end of headers
Serial.print("line:");
Serial.println(line);
line.trim();
// if the the line is empty,
// this is end of headers
// break the while and feed the
// remaining client to the
// Update.writeStream();
if (!line.length())
{
Serial.println("This Line Was empty one times");//headers ended
Serial.println("httpResponseLineCount is :" + String(httpResponseLineCount));
if (httpResponseLineCount == 1)
{
Serial.println("This Line Was empty two times");//headers ended
Serial.println("Breaking While() Now");
httpResponseLineCount = 0;
break; // and get the OTA started
}
httpResponseLineCount ++;
//break; // and get the OTA started
}
// Check if the HTTP Response is 200
// else break and Exit Update
if (line.startsWith("HTTP/1.1"))
{
if (line.indexOf("200") < 0)
{
Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
break;
}
}
// extract headers here
// Start with content length
if (line.startsWith("Content-Length: "))
{
contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
Serial.println("Got " + String(contentLength) + " bytes from server");
}
// Next, the content type
if (line.startsWith("Content-Type: "))
{
String contentType = getHeaderValue(line, "Content-Type: ");
Serial.println("Got " + contentType + " payload.");
if (contentType == "application/octet-stream")
{
isValidContentType = true;
}
}
}//-------------------------------------while
Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));
// check contentLength and content type
if (contentLength && isValidContentType)
{
bool canBegin = Update.begin(contentLength, 0, 2, 1, NULL);// Check if there is enough to OTA Update
if (canBegin)// If yes, begin
{
Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
// No activity would appear on the Serial monitor
// So be patient. This may take 2 - 5mins to complete
size_t written = Update.writeStream(Serial2);
if (written == contentLength)
{
Serial.println("Written : " + String(written) + " successfully");
}
else
{
Serial.println("Written only : " + String(written) + "/" + String(contentLength) + ". Retry?" );
Update.printError(Serial);
}
if (Update.end())
{
Serial.println("OTA done!");
if (Update.isFinished())
{
Serial.println("Update successfully completed. Rebooting.");
ESP.restart();
}
else
{
Serial.println("Update not finished? Something went wrong!");
}
}
else
{
Serial.println("Error Occurred. Error #: " + String(Update.getError()));
Update.printError(Serial);
}
}
else
{
// not enough space to begin OTA
// Understand the partitions and
// space availability
Serial.println("Not enough space to begin OTA");
Serial2.flush();
}
}
else
{
Serial.println("There was no content in the response");
Serial2.flush();
}
}
input = Serial2.readString();
input.remove(0);
Serial2.write("AT+QHTTPSTOP\r\n");
delay(100);
input = Serial2.readString();
input.remove(0);
}
else
{
Serial.println("HTTP RESPONSE ERROR");
Serial.println();
input.remove(0);
value = false;
}
}
else
{
Serial.println("URL CONNECT FAILED");
Serial.println();
value = false;
}
}
else
{
Serial.println("GPRS CONNECT FAILED");
Serial.println();
value = false;
}
input = Serial2.readString();
input.remove(0);
Serial2.write("AT+QIDEACT=1\r\n");
delay(100);
input = Serial2.readString();
input.remove(0);
}
else if (GSM_Network_STS == false)
{
value = false;
}
}
if (WiFi.status() == WL_CONNECTED)
{
WiFiClient client;
httpUpdate.setLedPin(LED);
t_httpUpdate_return ret = httpUpdate.update(client, URL_fw_Bin);
switch (ret)
{
case HTTP_UPDATE_FAILED:
Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", httpUpdate.getLastError(), httpUpdate.getLastErrorString().c_str());
break;
case HTTP_UPDATE_NO_UPDATES:
Serial.println("HTTP_UPDATE_NO_UPDATES");
break;
case HTTP_UPDATE_OK:
Serial.println("HTTP_UPDATE_OK");
break;
}
}
return value;
}`

the logs i get...

[6-28 19:35:44.9]<----------BOOTING UP THE DEVICE---------->
[6-28 19:35:44.9]
[6-28 19:35:44.9][ 2051][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 0 - WIFI_READY
[6-28 19:35:45.0][ 2147][V][WiFiGeneric.cpp:283] _arduino_event_cb(): STA Started
[6-28 19:35:45.0][ 2148][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 2 - STA_START
[6-28 19:35:46.5][ 3648][E][WiFiMulti.cpp:55] addAP(): [WIFI][APlistAdd] no ssid or ssid too long
[6-28 19:35:46.5][ 3648][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: silicis
[6-28 19:35:46.5][ 3650][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: PM12
[6-28 19:35:46.5][ 3657][I][WiFiMulti.cpp:84] addAP(): [WIFI][APlistAdd] add SSID: PM13
[6-28 19:35:46.5]CONNECTING TO WIFI...
[6-28 19:35:46.5]
[6-28 19:35:48.6][ 5699][V][WiFiGeneric.cpp:317] _arduino_event_cb(): SCAN Done: ID: 128, Status: 0, Results: 1
[6-28 19:35:48.6][ 5700][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 1 - SCAN_DONE
[6-28 19:35:48.6][ 5704][I][WiFiMulti.cpp:114] run(): [WIFI] scan done
[6-28 19:35:48.6][ 5709][I][WiFiMulti.cpp:119] run(): [WIFI] 1 networks found
[6-28 19:35:48.6][ 5714][D][WiFiMulti.cpp:151] run(): 0: [5][52:9B:68:86:29:7B] PM (-52) *
[6-28 19:35:48.6][ 5721][E][WiFiMulti.cpp:191] run(): [WIFI] no matching wifi found!
[6-28 19:35:49.6]UNABLE TO CONNECT ANY WIFI
[6-28 19:35:49.6]
[6-28 19:35:49.6]CHECKING GSM MODULE IN 10Sec.
[6-28 19:35:49.6]
[6-28 19:36:18.3]MODULE OK
[6-28 19:36:18.3]
[6-28 19:36:18.3]SETTING OTHER PARAMETERS TO GSM
[6-28 19:36:18.3]
[6-28 19:36:20.5]IMEI : 861190059739927
[6-28 19:36:20.5]
[6-28 19:36:22.1]CHECKING FOR SIM CARD
[6-28 19:36:22.1]
[6-28 19:36:23.2]SIM CARD FOUND OK
[6-28 19:36:23.2]
[6-28 19:36:24.3]GSM INITIALIZATION COMPLETE
[6-28 19:36:24.3]
[6-28 19:36:25.4]GSM IS REGISTERED ON NETWORK
[6-28 19:36:25.4]
[6-28 19:36:26.6]NETWORK TIME SYNC SUCCESSFUL
[6-28 19:36:26.6]
[6-28 19:36:27.7]SETTING SMS COMMANDS
[6-28 19:36:27.7]
[6-28 19:36:28.3]SETTING HTTP COMMANDS
[6-28 19:36:28.3]
[6-28 19:36:31.7]<--APN SET FOR JIO-->
[6-28 19:36:31.7]
[6-28 19:36:34.3]STARTING GPRS/PDP CONTEXT CONNECTION
[6-28 19:36:36.3]CONTEXT PROFILE IS OK
[6-28 19:36:36.3]
[6-28 19:36:37.3]GPRS/PDP CONTEXT PROFILE IS ACTIVATED
[6-28 19:36:37.3]
[6-28 19:36:38.4]
[6-28 19:36:38.4]+QIACT: 1,1,1,"25.37.35.187"
[6-28 19:36:38.4]
[6-28 19:36:38.4]OK
[6-28 19:36:38.4]
[6-28 19:36:38.4]Device Firmware Version is : 1.0
[6-28 19:36:38.4]
[6-28 19:36:38.4]Checking Firmware update...
[6-28 19:36:38.4]
[6-28 19:36:38.4]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/fw_version.txt
[6-28 19:36:38.4]
[6-28 19:36:39.5]GSM REGISTERED ON NETWORK
[6-28 19:36:39.5]
[6-28 19:36:40.5]Signal Strength : 31
[6-28 19:36:40.5]
[6-28 19:36:41.6]GPRS CONNECT SUCCESSFULLY
[6-28 19:36:41.6]
[6-28 19:36:44.8]URL CONNECT SUCCESSFULLY
[6-28 19:36:44.8]
[6-28 19:36:46.8]WAITING FOR GET COMMAND RESPONSE
[6-28 19:36:46.8]
[6-28 19:36:48.2]GET RESPONSE IS:200
[6-28 19:36:48.2]
[6-28 19:36:48.2]GET RESPONSE IS OK
[6-28 19:36:48.2]
[6-28 19:36:49.3]line is:
[6-28 19:36:49.3]This Line Was empty
[6-28 19:36:49.3]httpResponseLineCount is :1
[6-28 19:36:49.3]line is:CONNECT
[6-28 19:36:49.3]line is:HTTP/1.1 200 OK
[6-28 19:36:49.3]line is:Server: nginx/1.18.0 (Ubuntu)
[6-28 19:36:49.3]line is:Date: Tue, 28 Jun 2022 14:06:48 GMT
[6-28 19:36:49.5]line is:Content-Type: text/plain
[6-28 19:36:49.5]Got text/plain payload.
[6-28 19:36:49.5]line is:Content-Length: 5
[6-28 19:36:49.5]Got 5 bytes from server
[6-28 19:36:49.5]line is:Last-Modified: Tue, 19 Apr 2022 03:56:14 GMT
[6-28 19:36:49.5]line is:Connection: keep-alive
[6-28 19:36:49.5]line is:ETag: "625e32de-5"
[6-28 19:36:49.5]line is:Accept-Ranges: bytes
[6-28 19:36:49.5]line is:
[6-28 19:36:49.5]This Line Was empty
[6-28 19:36:49.5]httpResponseLineCount is :2
[6-28 19:36:49.5]This Line Was empty Two Times
[6-28 19:36:49.5]Breaking While() Now
[6-28 19:36:49.5]contentLength : 5, isValidContentType : 1
[6-28 19:36:49.5]Found Firmware Version is:1.1
[6-28 19:36:49.5]This is a New Firmware Version1.1
[6-28 19:36:51.6]Starting FOTA to update New Firmware
[6-28 19:36:51.6]
[6-28 19:36:51.6]<<< ATTENTION !!! DO NOT TURN OFF THE DEVICE... >>>
[6-28 19:36:51.6]
[6-28 19:36:51.6]Updating the Device Firmware Now!
[6-28 19:36:51.6]
[6-28 19:36:51.6]Please Wait untill the Device Restarts
[6-28 19:36:51.6]
[6-28 19:36:51.6]http://data.silicisroadindustries.com/silicis_fota/buslog4g_fota_lpa/update.bin
[6-28 19:36:51.6]
[6-28 19:36:52.7]GSM REGISTERED ON NETWORK
[6-28 19:36:52.7]
[6-28 19:36:53.7]Signal Strength : 20
[6-28 19:36:53.7]
[6-28 19:36:54.8]GPRS CONNECT SUCCESSFULLY
[6-28 19:36:54.8]
[6-28 19:36:58.0]URL CONNECT SUCCESSFULLY
[6-28 19:36:58.0]
[6-28 19:37:00.3]GET RESPONSE IS:200
[6-28 19:37:00.3]
[6-28 19:37:00.3]GET RESPONSE IS OK
[6-28 19:37:00.3]
[6-28 19:37:01.5]line:
[6-28 19:37:01.5]This Line Was empty one times
[6-28 19:37:01.5]httpResponseLineCount is :0
[6-28 19:37:01.5]line:CONNECT
[6-28 19:37:01.5]line:HTTP/1.1 200 OK
[6-28 19:37:01.5]line:Server: nginx/1.18.0 (Ubuntu)
[6-28 19:37:01.5]line:Date: Tue, 28 Jun 2022 14:07:00 GMT
[6-28 19:37:01.6]line:Content-Type: application/octet-stream
[6-28 19:37:01.6]Got application/octet-stream payload.
[6-28 19:37:01.6]line:Content-Length: 213296
[6-28 19:37:01.6]Got 213296 bytes from server
[6-28 19:37:01.6]line:Last-Modified: Fri, 20 May 2022 08:03:51 GMT
[6-28 19:37:01.6]line:Connection: keep-alive
[6-28 19:37:01.7]line:ETag: "62874b67-34130"
[6-28 19:37:01.7]line:Accept-Ranges: bytes
[6-28 19:37:01.7]line:
[6-28 19:37:01.7]This Line Was empty one times
[6-28 19:37:01.7]httpResponseLineCount is :1
[6-28 19:37:01.7]This Line Was empty two times
[6-28 19:37:01.7]Breaking While() Now
[6-28 19:37:01.7]contentLength : 213296, isValidContentType : 1
[6-28 19:37:01.7][ 78868][D][Updater.cpp:133] begin(): OTA Partition: app1
[6-28 19:37:01.7]Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
[6-28 19:40:43.9]Written : 213296 successfully
[6-28 19:40:43.9]E (301042) esp_image: invalid segment length 0xf4400827
[6-28 19:40:43.9]Error Occurred. Error #: 9
[6-28 19:40:43.9]Could Not Activate The Firmware
[6-28 19:40:50.1]<----------BOOT PROCESS COMPLETE---------->
[6-28 19:40:50.1]

dear @SuGlider please check the progress and help me get the solution.

Thank you

@SuGlider
Copy link
Collaborator

We have done a few updates to UART in the new Arduino Core 2.0.5 version.
There are new functions that may help your OTA application:

  • void onReceive(OnReceiveCb function, bool onlyOnTimeout = false);
    onReceive() will setup a callback that will be called whenever an UART interruption occurs (UART_INTR_RXFIFO_FULL or
    UART_INTR_RXFIFO_TOUT). This means that it is possible to set up a callback that will be executed as soon as data is
    received.
    UART_INTR_RXFIFO_FULL interrupt triggers at UART_FULL_THRESH_DEFAULT bytes received (defined as 120 bytes by
    default in IDF). This default value of 120 can be changed using setRxFIFOFull() with a value from 1 to 127.
    UART_INTR_RXFIFO_TOUT interrupt triggers at UART_TOUT_THRESH_DEFAULT symbols passed without any reception
    (defined as 2 symbols by default). This can also be changed with setRxTimeout()

    onlyOnTimeout parameter will define how onReceive will behave:
    true -- The callback will only be called when RX Timeout happens.
    Whole stream of bytes will be ready for being read on the callback function at once.
    This option may lead to Rx Overflow depending on the Rx Buffer Size and number of bytes received
    in the streaming
    Default: false -- The callback will be called when FIFO reaches RXFIFO_FULL bytes and also on RX Timeout.
    The stream of incommig bytes will be "split" into blocks of minimum RXFIFO_FULL bytes on each callback.
    This option avoids any sort of Rx Overflow, but leaves the UART packet reassembling work
    to the Application.

  • void onReceiveError(OnReceiveErrorCb function);
    onReceiveError() will be called on error events (see hardwareSerial_error_t) and it will also call the onReceive() callback in
    case some was defined. By this, it is possible to read data using onReceive() callback when a BREAK is sent to UART, for
    instance.

  • void setRxTimeout(uint8_t symbols_timeout);
    setRxTimeout() sets the timeout after which onReceive callback will be called (after receiving data, it waits for this time of UART rx inactivity to call the callback fnc)
    param symbols_timeout defines a timeout threshold in uart symbol periods.
    Setting 0 (zero) symbol timeout disables the callback call by timeout.
    Maximum timeout setting is calculacted automatically by IDF. If set above the maximum, it is ignored and an error is printed on Serial0 (check console).
    For example symbols_timeout=1 defines a timeout equal to transmission time of one symbol (~11 bit) on current baudrate.
    For a baudrate of 9600, SERIAL_8N1 (10 bit symbol) and symbols_timeout = 3, the timeout would be 3 / (9600 / 10) = 3.125 ms

  • void setRxFIFOFull(uint8_t fifoBytes);
    setRxFIFOFull() will set the number of bytes that will trigger UART_INTR_RXFIFO_FULL interrupt and fill up RxRingBuffer
    This affects some functions such as Serial::available() and Serial.read() because, in a UART flow of receiving data,
    Serial internal RxRingBuffer will be filled only after these number of bytes arrive or a RX Timeout happens.
    This parameter can be set to a low value, such as 1 in order to receive byte by byte, but it will also consume more
    CPU time as the ISR will be activates often.

@SuGlider
Copy link
Collaborator

Using Arduino Core 2.0.5, please try to setup the Serial used for OTA/LTE like this:

void setup(){
        Serial.setRxBufferSize(1024);
        Serial.begin(115200,SERIAL_8N1,26,27);
        Serial.setRxTimeout(1);
        Serial.setRxFIFOFull(1);
        Serial.flush();
}

This will force that whenever a single byte arrives to the Serial Port, it will be made immediately available for Serial.read().

@VojtechBartoska VojtechBartoska added this to the 2.0.6 milestone Sep 21, 2022
@SuGlider
Copy link
Collaborator

After some testing with Arduino Core 2.0.5, the best configuration shall be:

change is on Serial.setRxFIFOFull(5); - it will produce a better result with UART.

void setup(){
        Serial.setRxBufferSize(1024);
        Serial.begin(115200,SERIAL_8N1,26,27);
        Serial.setRxTimeout(1);
        Serial.setRxFIFOFull(5);
        Serial.flush();
}

@SuGlider SuGlider moved this from Under investigation to In Progress in Arduino ESP32 Core Project Roadmap Nov 20, 2022
@primus192
Copy link

primus192 commented Dec 1, 2022 •
edited
Loading

I tested the code above, it did not help. Maybe i can record a video showing difference in updating on arduino 1.0.6 vs 2.0.5 so we can investigate and understand the problem better.

@SuGlider
Copy link
Collaborator

SuGlider commented Dec 1, 2022

After verifying the differences from UART v1.0.6 to 2.0.x, I see that because 2.0.x uses IDF UART driver, it is slower.
It takes almost the double of the time that 1.0.6 takes to read a single byte.

Unfortunatelly, we can't move away from IDF.
In order to mitigate part of this issue, I have submitted a PR to make reading more than 1 byte from UART faster, using HardwareSerial::read(buffer, size).
But this PR doesn't help on reading a single byte, and as result, it also doesn't help Stream::readBytes() as well.

The issue with Update Library and Arduino Core 2.0.x is that Stream API is slower now...
That may lead to timeout issues such as in the initial example of this post.

/*
      Writes the remaining bytes from the Stream to the flash
      Uses readBytes() and sets UPDATE_ERROR_STREAM on timeout
      Returns the bytes written
      Should be equal to the remaining bytes when called
      Usable for slow streams like Serial
*/
    size_t writeStream(Stream &data);

readBytes(buffer, size) shall be replaced by HardwareSerial::read(buffer, size) in an alternative version of Update.cpp.
https://github.com/espressif/arduino-esp32/blob/master/libraries/Update/src/Updater.cpp#L371

I don't know how possible would be for you guys to make such code change and test it...

@primus192
Copy link

I am goin

After verifying the differences from UART v1.0.6 to 2.0.x, I see that because 2.0.x uses IDF UART driver, it is slower. It takes almost the double of the time that 1.0.6 takes to read a single byte.

Unfortunatelly, we can't move away from IDF. In order to mitigate part of this issue, I have submitted a PR to make reading more than 1 byte from UART faster, using HardwareSerial::read(buffer, size). But this PR doesn't help on reading a single byte, and as result, it also doesn't help Stream::readBytes() as well.

The issue with Update Library and Arduino Core 2.0.x is that Stream API is slower now... That may lead to timeout issues such as in the initial example of this post.

/*
      Writes the remaining bytes from the Stream to the flash
      Uses readBytes() and sets UPDATE_ERROR_STREAM on timeout
      Returns the bytes written
      Should be equal to the remaining bytes when called
      Usable for slow streams like Serial
*/
    size_t writeStream(Stream &data);

readBytes(buffer, size) shall be replaced by HardwareSerial::read(buffer, size) in an alternative version of Update.cpp. https://github.com/espressif/arduino-esp32/blob/master/libraries/Update/src/Updater.cpp#L371

I don't know how possible would be for you guys to make such code change and test it...

That is a pity. I am going to test that

@SuGlider
Copy link
Collaborator

SuGlider commented Dec 4, 2022 •
edited
Loading

@mudgalp @primus192

Please try using lastest master branch from Arduino ESP32 Github to build you application.
There is a new PR that may fix the issue: #7525
It will be necessary to apply this PR over the master brach while it is not merged.

This PR fixes Stream::readBytes() to make it faster using IDF in the new Core 2.0.x

I think that it may solve the timeout issue.

Please use this setup:

void setup(){
        Serial.setRxBufferSize(2048);
        Serial.begin(115200,SERIAL_8N1,26,27);  // RxPin = 26 & TxPin = 27
        Serial.setRxTimeout(1);
        Serial.setRxFIFOFull(120);
        Serial.flush();
}

@SuGlider SuGlider mentioned this issue Dec 4, 2022
Merged
@VojtechBartoska VojtechBartoska added Resolution: Awaiting response Waiting for response of author and removed Status: Needs investigation We need to do some research before taking next steps on this issue labels Dec 5, 2022
Repository owner moved this from In Progress to Done in Arduino ESP32 Core Project Roadmap Dec 13, 2022
@Sanjeet-Dhankhar
Copy link

Dear @mudgalp,

Please provide me the files of the project. I don't need GSM library which you are using for this project rest library kindly share.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SuGlider SuGlider

Labels
Area: BT&Wifi BT & Wifi related issues Resolution: Awaiting response Waiting for response of author
Projects
Arduino ESP32 Core Project Roadmap
Status: Done
Milestone
2.0.6
Development

Successfully merging a pull request may close this issue.

Improves UART reading performance SuGlider/arduino-esp32
5 participants
@mudgalp @SuGlider @VojtechBartoska @primus192 @Sanjeet-Dhankhar