Server ota updates

[PeWos]

It is possible to update my esp32 via ota from a server like the esp8266. I have read this page http://esp8266.github.io/Arduino/versions/2.0.0/doc/ota_updates/ota_updates.html#http-server but the function can't be compiled with esp32.

Thanks in advance

[gellis12]

You can use the included BasicOTA example, but you need to add #include <Update.h> to the top of the file. Once you add that, it should work fine.

Edit: My answer seems to be irrelevant after PeWos edited the question.

[PeWos]

Hi @gellis12 I edited the question before you answered me, i had write bad a word. I have tried adding the update library but it only works in local with the arduino ide. What function do I have to add that is like the ESP8266 ESPhttpUpdate function?

[copercini]

I am porting it to ESP32, but actually have 2 working versions: the only http one (using WifiClient) and the only https one(using WifiClientSecure). So I am trying to unify both (to keep closer to ESP8266) and make a PR.

[vvs551]

Could you publish http version of OTA update?

[PhilColbert]

Any chance of publishing a working version ? :) thanks

[copercini]

Sorry for say that, but it's still missing some important functions for full port, like:

• SPIFFS
• Verify magic bytes (WiFiClient doesn't peek bytes)
• getSketchMD5, getFreeSketchSpace and getSketchSize functions

I think it's too early to release but HTTPClient port is one step closer...

[PhilColbert]

Ok thanks, am just looking for an easy way to do OTA updates - if it works at all, that would be great !

[inforaudio]

So we are looking for OTA Updates for ESP32 in Arduino IDE. Something can help me?

[skateone]

there is a nice example i used called AWS_S3_OTA_Update.ino
its part of the arduino esp32 bundle and works the same as the ESP-IDF updater.

hope this helps.

[peterzcruz]

does AWS_S3_OTA_Update.ino really work?

here is the serial reply on my test:

[code]
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0008,len:8
load:0x3fff0010,len:160
load:0x40078000,len:10632
load:0x40080000,len:252
entry 0x40080034
Connecting to 1900acOpenWRT
...
Connected to 1900acOpenWRT
Connecting to: esp32-testbucket.s3.ap-south-1.amazonaws.com
Fetching Bin: /StartCounter.ino.bin
contentLength : 0, isValidContentType : 0
There was no content in the response
[/code]

1. my S3 bucket is on https. is there a way to convert it to non secure http?
   there is a note on the sample:
   "Non https. For HTTPS 443. As of today, HTTPS doesn't work."

2. something is off with my test. there is no indication that i got connected to s3.
   im expecting this line on my serial terminal:

   • Serial.println("Connection to " + String(host) + " failed. Please check your setup");

   but i did not see that, nor the indication that i got connected, starting from this line:

   • Serial.println("Fetching Bin: " + String(bin));

ota sample from esp32-idf is working fine in my test.

any suggestion?
thanks

[lbernstone]

Peter-
S3 is pretty slow. The delay is only 100 ms in the example. I'd bump that up to 500ms or else loop a few times until client.available() is true.

[copercini]

For NonSSL as @lbernstone say, 100ms is too low for almost servers response.

For SSL mbedtls still doesn't peek bytes so WiFiClientSecure can't get the magic header from new firmware and updater fail at this part

[peterzcruz]

Thank you very much for the replies.

I changed the delay to this:

unsigned long timeout = millis();
while (client.available() == 0) {
    if (millis() - timeout > 5000) {
        Serial.println(">>> Client Timeout !");
        client.stop();
        return;
    }
}

s3 bucket is http accessible even if it gave me https link to the bin file after upload.

here is the reply I got in my terminal, added:
Serial.println(line);
after:
line.trim();

HTTP/1.1 200 OK
x-amz-id-2: 3prqlD2UPukbLhG4uY4KUqTOgF+Df9Cgxhwv3YAftxGIzGIa9dl0YpXX8pUj6OQaCcfusgnyJ24=
x-amz-request-id: 41588F5E690964D3
Date: Tue, 18 Jul 2017 10:27:21 GMT
Last-Modified: Tue, 18 Jul 2017 00:36:24 GMT
ETag: "ba994d72b01c874c7f5191d314d3009f"
Accept-Ranges: bytes
Content-Type: application/macbinary
Content-Length: 500976
Server: AmazonS3
Connection: close

End of line

Got 500976 bytes from server
Got application/macbinary payload.

contentLength : 500976, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written only : 0/500976. Retry?
Error Occurred. Error #: 8

here is the code used to get headers:

while(client.available()) {
    String line = client.readStringUntil('\r');
    line.trim();
    Serial.println(line);
    if (!line.length()) {
        Serial.println("End of line");
        breal;
    }
    if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
            Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
            break;
        }
    }
    if (line.startsWith("Content-Type: ")) {
        contentType = getHeaderValue(line, "Content-Type: ");
        //if (contentType == "application/octet-stream") {
        if (contentType == "application/macbinary") {
            isValidContentType = true;
        }
    }
    
    if (line.startsWith("Content-Length: ")) {
        contentLength = atoi((getHeaderValue(line, "Content-Length: ")).c_str());
    }
}
Serial.println("");
Serial.println("Got " + String(contentLength) + " bytes from server");
if (isValidContentType) {
    Serial.println("Got " + contentType + " payload.");
}
Serial.println("");
Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

from the original code, I added:

   Serial.println("End of line");

to indicate im getting end of line

rearranged so I clearly see the header reply:

Serial.println("");
Serial.println("Got " + String(contentLength) + " bytes from server");
if (isValidContentType) {
    Serial.println("Got " + contentType + " payload.");
}
Serial.println("");
Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

content type from my S3 header is "application/macbinary", so I changed that also, just to set the isValidContentType

if (line.startsWith("Content-Type: ")) {
    contentType = getHeaderValue(line, "Content-Type: ");
    //if (contentType == "application/octet-stream") {
    if (contentType == "application/macbinary") {
        isValidContentType = true;
    }
}

the remaining codes is unchanged, but I get this always:

contentLength : 500976, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written only : 0/500976. Retry?
Error Occurred. Error #: 8

[peterzcruz]

tried also with my EC2 as the file server
(EC2 content type is application/octet-stream)

here is the reply in my terminal:

HTTP/1.1 200 OK
Date: Tue, 18 Jul 2017 21:57:49 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) PHP/5.4.16
Last-Modified: Tue, 18 Jul 2017 00:12:56 GMT
ETag: "7a4f0-5548c6093b200"
Accept-Ranges: bytes
Content-Length: 500976
Connection: close
Content-Type: application/octet-stream

End of line

Got 500976 bytes from server
Got application/octet-stream payload.

contentLength : 500976, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written only : 0/500976. Retry?
Error Occurred. Error #: 8

am always getting Error Occurred. Error #: 8

Im using EC2 server on esp32-idf ota sample without problem

any suggestions?
thanks

[copercini]

Error Occurred. Error #: 8 Means UPDATE_ERROR_MAGIC_BYTE
it's the first byte of .bin file and should be 0xE9

Opening on a Hex editor:

on esp32-idf OTA it's not checked but on Arduino implementation it is.

[lbernstone]

Update::writeStream does a stream peek, which as @copercini mentioned, doesn't work yet. I removed lines 305-308 (the magic byte check) from Update.h and it works.

[peterzcruz]

thank you very much for your help. let me answer my previous question:
" does AWS_S3_OTA_Update.ino really work? "

answer:
yes. AWS_S3_OTA_Update.ino works.

and for those who has slow internet like mine, just need to change the delay after GET request.

here is the code I used:

/**
   AWS S3 OTA Update
   Date: 14th June 2017
   Author: Arvind Ravulavaru <https://github.com/arvindr21>
   Purpose: Perform an OTA update from a bin located in Amazon S3 (HTTP Only)
   Upload:
   Step 1 : Download the sample bin file from the examples folder
   Step 2 : Upload it to your Amazon S3 account, in a bucket of your choice
   Step 3 : Once uploaded, inside S3, select the bin file >> More (button on top of the file list) >> Make Public
   Step 4 : You S3 URL => http://bucket-name.s3.ap-south-1.amazonaws.com/sketch-name.ino.bin
   Step 5 : Build the above URL and fire it either in your browser or curl it `curl -I -v http://bucket-name.ap-south-1.amazonaws.com/sketch-name.ino.bin` to validate the same
   Step 6:  Plug in your SSID, Password, S3 Host and Bin file below
   Build & upload
   Step 1 : Menu > Sketch > Export Compiled Library. The bin file will be saved in the sketch folder (Menu > Sketch > Show Sketch folder)
   Step 2 : Upload bin to S3 and continue the above process
   // Check the bottom of this sketch for sample serial monitor log, during and after successful OTA Update
*/

#include <WiFi.h>
#include <Update.h>

WiFiClient client;

// Variables to validate
// response from S3
int contentLength = 0;
bool isValidContentType = false;

// Your SSID and PSWD that the chip needs
// to connect to
char* SSID = "YOUR-SSID";
char* PSWD = "YOUR-SSID-PSWD";

// S3 Bucket Config
String host = "bucket-name.s3.ap-south-1.amazonaws.com"; // Host => bucket-name.s3.region.amazonaws.com
int port = 80; // Non https. For HTTPS 443. As of today, HTTPS doesn't work.
String bin = "/StartCounter.ino.bin"; // bin file name with a slash in front.
String contentType = "";
// Utility to extract header value from headers
String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

// OTA Logic 
void execOTA() {
  Serial.println("Connecting to: " + String(host));
  // Connect to S3
  if (client.connect(host.c_str(), port)) {
    // Connection Succeed.
    // Fecthing the bin
    Serial.println("Fetching Bin: " + String(bin));
    Serial.println("");

    // Get the contents of the bin file
    client.print(String("GET ") + bin + " HTTP/1.1\r\n" +
                 "Host: " + host + "\r\n" +
                 "Cache-Control: no-cache\r\n" +
                 "Connection: close\r\n\r\n");

    // Check what is being sent
    //    Serial.print(String("GET ") + bin + " HTTP/1.1\r\n" +
    //                 "Host: " + host + "\r\n" +
    //                 "Cache-Control: no-cache\r\n" +
    //                 "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (client.available() == 0) {
        if (millis() - timeout > 5000) {
            Serial.println(">>> Client Timeout !");
            client.stop();
            return;
        }
    }
    
    // Once the response is available,
    // check stuff

    /*
       Response Structure
        HTTP/1.1 200 OK
        x-amz-id-2: NVKxnU1aIQMmpGKhSwpCBh8y2JPbak18QLIfE+OiUDOos+7UftZKjtCFqrwsGOZRN5Zee0jpTd0=
        x-amz-request-id: 2D56B47560B764EC
        Date: Wed, 14 Jun 2017 03:33:59 GMT
        Last-Modified: Fri, 02 Jun 2017 14:50:11 GMT
        ETag: "d2afebbaaebc38cd669ce36727152af9"
        Accept-Ranges: bytes
        Content-Type: application/octet-stream
        Content-Length: 357280
        Server: AmazonS3
                                   
        {{BIN FILE CONTENTS}}
    */
    while (client.available()) {
      // read line till /n
      String line = client.readStringUntil('\n');
      // remove space, to check if the line is end of headers
      line.trim();
      Serial.println(line);

      // if the the line is empty,
      // this is end of headers
      // break the while and feed the
      // remaining `client` to the
      // Update.writeStream();
      if (!line.length()) {
        //headers ended
        break; // and get the OTA started
      }

      // Check if the HTTP Response is 200
      // else break and Exit Update
      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      // extract headers here
      // Start with content length
      if (line.startsWith("Content-Length: ")) {
        contentLength = atoi((getHeaderValue(line, "Content-Length: ")).c_str());
        //Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        contentType = getHeaderValue(line, "Content-Type: ");
        //Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream" || contentType == "application/macbinary") {
          isValidContentType = true;
        }
      }
    }
  } else {
    // Connect to S3 failed
    // May be try?
    // Probably a choppy network?
    Serial.println("Connection to " + String(host) + " failed. Please check your setup");
    // retry??
    // execOTA();
  }

  Serial.println("Got " + contentType + " payload.");
  Serial.println("Got " + String(contentLength) + " bytes from server");
  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));
  Serial.println("");

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
      // No activity would appear on the Serial monitor
      // So be patient. This may take 2 - 5mins to complete
      size_t written = Update.writeStream(client);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength) + ". Retry?" );
        // retry??
        // execOTA();
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting.");
          ESP.restart();
        } else {
          Serial.println("Update not finished? Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred. Error #: " + String(Update.getError()));
      }
    } else {
      // not enough space to begin OTA
      // Understand the partitions and
      // space availability
      Serial.println("Not enough space to begin OTA");
      client.flush();
    }
  } else {
    Serial.println("There was no content in the response");
    client.flush();
  }
}

void setup() {
  //Begin Serial
  Serial.begin(115200);
  delay(10);

  Serial.println("Connecting to " + String(SSID));

  // Connect to provided SSID and PSWD
  WiFi.begin(SSID, PSWD);

  // Wait for connection to establish
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print("."); // Keep the serial monitor lit!
    delay(500);
  }

  // Connection Succeed
  Serial.println("");
  Serial.println("Connected to " + String(SSID));

  // Execute OTA Update
  execOTA();
}

void loop() {
  // chill
}

/*
 * Serial Monitor log for this sketch
 * 
 * If the OTA succeeded, it would load the preference sketch, with a small modification. i.e.
 * Print `OTA Update succeeded!! This is an example sketch : Preferences > StartCounter`
 * And then keeps on restarting every 10 seconds, updating the preferences
 * 
 * 
      rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
      configsip: 0, SPIWP:0x00
      clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
      mode:DIO, clock div:1
      load:0x3fff0008,len:8
      load:0x3fff0010,len:160
      load:0x40078000,len:10632
      load:0x40080000,len:252
      entry 0x40080034
      Connecting to SSID
      ......
      Connected to SSID
      Connecting to: bucket-name.s3.ap-south-1.amazonaws.com
      Fetching Bin: /StartCounter.ino.bin
      Got application/octet-stream payload.
      Got 357280 bytes from server
      contentLength : 357280, isValidContentType : 1
      Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
      Written : 357280 successfully
      OTA done!
      Update successfully completed. Rebooting.
      ets Jun  8 2016 00:22:57
      
      rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
      configsip: 0, SPIWP:0x00
      clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
      mode:DIO, clock div:1
      load:0x3fff0008,len:8
      load:0x3fff0010,len:160
      load:0x40078000,len:10632
      load:0x40080000,len:252
      entry 0x40080034
      
      OTA Update succeeded!! This is an example sketch : Preferences > StartCounter
      Current counter value: 1
      Restarting in 10 seconds...
      E (102534) wifi: esp_wifi_stop 802 wifi is not init
      ets Jun  8 2016 00:22:57
      
      rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
      configsip: 0, SPIWP:0x00
      clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
      mode:DIO, clock div:1
      load:0x3fff0008,len:8
      load:0x3fff0010,len:160
      load:0x40078000,len:10632
      load:0x40080000,len:252
      entry 0x40080034
      
      OTA Update succeeded!! This is an example sketch : Preferences > StartCounter
      Current counter value: 2
      Restarting in 10 seconds...
      ....
 * 
 */

here is the response on my serial terminal

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0008,len:8
load:0x3fff0010,len:160
load:0x40078000,len:10632
load:0x40080000,len:252
entry 0x40080034
Connecting to 1900acOpenWRT
...
Connected to 1900acOpenWRT
Connecting to: <YOUR S3 BUCKET HERE>
Fetching Bin: /StartCounter.ino.bin

HTTP/1.1 200 OK
x-amz-id-2: DbCR8mV1TwWIEFEMzHKvh7MI1c4MBI1IzPeoDncO7Atbqg8PyF4/HICy7xgngleR1CHu/bnElRE=
x-amz-request-id: 53184D5BFB566BA6
Date: Wed, 19 Jul 2017 23:09:19 GMT
Last-Modified: Tue, 18 Jul 2017 01:06:41 GMT
ETag: "19c47f5a6952adc7704c87609625f3a6"
Accept-Ranges: bytes
Content-Type: application/macbinary
Content-Length: 357280
Server: AmazonS3
Connection: close

Got application/macbinary payload.
Got 357280 bytes from server
contentLength : 357280, isValidContentType : 1

Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written : 357280 successfully
OTA done!
Update successfully completed. Rebooting.
ets Jun  8 2016 00:22:57

[copercini]

@peterzcruz it works with magic byte check or excluding it?

[peterzcruz]

I did not exclude the magic byte. no changes made on Updater.cpp and Update.h.
I don't know why on my first tries, I'm getting Error #: 8.
I just restart my home router, then it worked fine.
It already worked ok yesterday, after your reply regarding UPDATE_ERROR_MAGIC_BYTE.
Just didn't have much time to update here.

thank you @copercini and @lbernstone

[copercini]

Just some updates:

• the delay of AWS_S3_OTA_Update was increased by default 5c1b10f
• @me-no-dev give me some ideas to implement peekbytes in mbedtls, will try it soon

[huyhl248]

Dear @lbernstone ,
Here is my result after removing the magic byte check!

Connected to ACRONICS
Connecting to: 192.168.1.69
Fetching Bin: /firmware.bin
Got 481008 bytes from server
Got application/octet-stream payload.
contentLength : 481008, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written only : 481009/481008. Retry?
Error Occurred. Error #: 6

[pjunni]

Has anyone been able to do something like this to update ESP32 firmware through a URL: ESPhttpUpdate.update(firmwareURL); ?

[ipsecguy]

Hi Petter,

I have trouble to find the proper includes/library for the ESPhttpUpdate thing. I am on "v3.0-dev-1024-g92748142". I want to auto-update ESPs in the Field that phone home every once in a while (like every day or week or at reboot).

Andreas

[kherNI]

@huyhl248 I experienced the same as you. The AWS_S3_OTA_Update.ino example works fine but if the new .bin file is relatively large (in my case ~900 - 975KBKB and in your example ~481KB) it returns an Error #6 code (UPDATE_ERROR_STREAM). Also the bytes written always exceed the new .bin file size.

Content-Length : 974384, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might take a while...please be patient.
Written only : 974449/974384. Retry?
Error Occurred. Error #: 6

Has anyone figured out a workaround?

[copercini]

Working with SSL now! =)

After add WiFiClientSecure::peek(); febcda0

[krish2487]

Hello,
I just pitched in to report that the issue still persists. :-(
We are using a adapted version of the AWS S3 OTA in our application and the patch for the peek function still returns an error 8. Upon closer inspection, the magic byte read and the actual magic byte are not equal. I just used a couple of serial.print at the verifyheader function in update.cpp library to print out the two values.
The defined magic byte is 233 ( 0xE9 ) but we end up reading 255 ( FF ).
Can you verify this on your end @copercini .

The only working fix still seems to be disabling the actual verifyheader check in the writestream function. The OTA sketch works, but only when we disable the verifyheader / peek function.

[marcelstoer]

I can confirm @krish2487's observations. OTA upgrades over TLS only work if the Stream.peak call at https://github.com/espressif/arduino-esp32/blob/master/libraries/Update/src/Updater.cpp#L311-L314 is deactivated.

[malbrook]

using the same code as @peterzcruz but with my own server (dedicated server on UKFast in UK) I can successfully upload a modified version of the AsyncBrowser that then allows access to files previously uploaded to the SPIFFS area on the ESP32. The .bin file is stored in a subfolder (/img) on the server and reports as 756kbytes in size but is actually 773376 bytes in size which is what the uploader reports.

Connected to myrouter
Connecting to: mywebserver
Fetching Bin: /img/ESP32_AsyncBrowser.ino.bin
Got 773376 bytes from server
Got application/octet-stream payload.
contentLength : 773376, isValidContentType : 1
[D][Updater.cpp:129] begin(): OTA Partition: app1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written : 773376 successfully
OTA done!
Update successfully completed. Rebooting.
ets Jun  8 2016 00:22:57

I am working on a package to include a web server running on 1 core of the ESP32, an application running on the other core, with the ability to update the package from our own servers.

[tmsio]

Hi, same errore here:
Error #6 using @peterzcruz code.
Testing the code using my phone Hot Spot instead my router, and the ota update it s done.
I just modify [your path]\Arduino\hardware\espressif\esp32\cores\esp32\Stream.h
in line 55 :
_timeout = 1000;
to
_timeout = 3000;
And now no problem.
Bye

[g01d10x]

The delay listed by @tmsio on August 24 was the ticket for me .. works now with AWS and also my own web server :) THANKS to all !!!

[Donderda]

Hi everyone,
I also have this problem. Editing the streams.h didn't work out for me :(. I've got a large binary, might this be the problem? Has anyone an idea how to fix it? Sometimes the update process works fine, sometimes it just stops the writing:

Connecting to: 10.2.7.101
Fetching Bin: /firmware/repeater.bin
Got application/octet-stream payload.
Got 1269632 bytes from server
contentLength : 1269632, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Task watchdog got triggered. The following tasks did not reset the watchdog in time:
 - IDLE (CPU 1)
Tasks currently running:
CPU 0: ipc0
CPU 1: IDLE
Written only : 785197/1269632. Retry?

And sometime the kernel just panics:

Connecting to: 10.2.7.101
Fetching Bin: /firmware/repeater.bin
Got application/octet-stream payload.
Got 1269632 bytes from server
contentLength : 1269632, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Guru Meditation Error: Core  1 panic'ed (LoadStoreError). Exception was unhandled.
Core 1 register dump:
PC      : 0x40149ee8  PS      : 0x00060c30  A0      : 0x80149f88  A1      : 0x3ffd86e0  
0x40149ee8: WiFiClientRxBuffer::fillBuffer() at /Users/user/esp/project/components/arduino/libraries/WiFi/src/WiFiClient.cpp:507

A2      : 0x3ffea48c  A3      : 0x3ffd8720  A4      : 0x00000001  A5      : 0x40000000  
A6      : 0x000152f4  A7      : 0x00000000  A8      : 0x00000000  A9      : 0xffffffff  
A10     : 0x00000000  A11     : 0x3ffead0c  A12     : 0x3ffc4340  A13     : 0x3ffd85f0  
A14     : 0x00000003  A15     : 0x00060023  SAR     : 0x0000000a  EXCCAUSE: 0x00000003  
EXCVADDR: 0x40000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  

Backtrace: 0x40149ee8:0x3ffd86e0 0x40149f85:0x3ffd8700 0x401a784e:0x3ffd8720 0x401483e2:0x3ffd8750 0x40148411:0x3ffd8770 0x40149bd2:0x3ffd8790 0x40125a9f:0x3ffd87b0 0x40125199:0x3ffd8a10 0x400e277f:0x3ffd8a90
0x40149ee8: WiFiClientRxBuffer::fillBuffer() at /Users/user/esp/project/components/arduino/libraries/WiFi/src/WiFiClient.cpp:507

0x40149f85: WiFiClientRxBuffer::read(unsigned char*, unsigned int) at /Users/user/esp/project/components/arduino/libraries/WiFi/src/WiFiClient.cpp:507

0x401a784e: std::__cxx11::collate<wchar_t>::transform(wchar_t const*, wchar_t const*) const at /Volumes/build/idf/crosstool-NG/.build/xtensa-esp32-elf/build/build-cc-gcc-final/xtensa-esp32-elf/libstdc++-v3/include/bits/locale_classes.h:710

0x401483e2: HardwareSerial::begin(unsigned long, unsigned int, signed char, signed char, bool) at /Users/user/esp/project/components/arduino/cores/esp32/HardwareSerial.cpp:112

0x40148411: HardwareSerial::begin(unsigned long, unsigned int, signed char, signed char, bool) at /Users/user/esp/project/components/arduino/cores/esp32/HardwareSerial.cpp:112

0x40149bd2: UpdateClass::end(bool) at /Users/user/esp/project/components/arduino/libraries/Update/src/Updater.cpp:343

Here's my current partitions.csv:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x180000,
app1,     app,  ota_1,   0x190000,0x180000,
eeprom,   data, 0x99,    0x310000,0x1000,

[tmsio]

Hi @Donderda , try to create a hotspot on your phone and try again.
Tell me me if work.
Also try to set _timeout to 5000

Regards.
Emiliano

[huyhl248]

Setting _timeout to 5000 didn't work for me! :(

[tmsio]

Hi @huyhl248, can you try another HotSpot?
Can you give us your log?

Thank You
Regards

[huyhl248]

Hi @tmsio ,
I tried to use a local server with same website code and it worked , but no luck with the real server.
Here is the log:

############################### UPDATE FIRMWARE ################################
/firmware.php?mac=240AC4105ED8
Connecting to: www.example.com
Fetching Bin: /firmware.php?mac=240AC4105ED8
 
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Got application/octet-stream payload.
Content-Length: 1252704
Got 1252704 bytes from server
Connection: close
Date: Thu, 17 Jan 2019 02:49:54 GMT
Server: Apache
X-Powered-By: PHP/5.6.39
Content-Disposition: attachment; filename=firmware_00.00.01.bin
Accept-Ranges: bytes
Content-Transfer-Encoding: binary
Expires: 0
Cache-Control: must-revalidate
Pragma: public

Got application/octet-stream payload.
Got 1252704 bytes from server
content_length : 1252704, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while ... Patience!
Task watchdog got triggered. The following tasks did not feed the watchdog in time:
Tasks currently running:
CPU 0: getDataLooptask
CPU 1: IDLE
Task watchdog got triggered. The following tasks did not feed the watchdog in time:
Tasks currently running:
CPU 0: getDataLooptask
CPU 1: displayLoopTask
Task watchdog got triggered. The following tasks did not feed the watchdog in time:
Tasks currently running:
CPU 0: getDataLooptask
CPU 1: displayLoopTask
Written only : 255244/1252704. Retry?

Please help me!
Thanks a lot!

[LorenzoCerda]

Hi, same errore here:
Error #6 using @peterzcruz code.
Testing the code using my phone Hot Spot instead my router, and the ota update it s done.
I just modify [your path]\Arduino\hardware\espressif\esp32\cores\esp32\Stream.h
in line 55 :
_timeout = 1000;
to
_timeout = 3000;
And now no problem.
Bye

Hello @tmsio and everybody,

Your trick worked perfectly for me, but I understand that modifying a library is not a good practice (if the library gets updated, you loose the modification, for example). But the good news is that the Stream library has a function that allows you to modify its timeout parameter: .setTimeout(msegs). So in my case I used
client.setTimeout(3000);
just at the beginning of the function execOTA() and it worked ok (and I returned to the original version of the Stream.h file).

That's it. I hope it helps.

All the best

[TonyTrupp]

My issue with this was that while reading the headers, I was reading to the \r new line character, and there was another trailing \n new line character, which was showing up as 10 as the first byte instead of e9.

To fix I'm doing this before Update.writeStream(*client), to advance by one more character before it checks the first byte:
if(client->peek()==10) client->read();

[IoTThinks]

Hello,
I just pitched in to report that the issue still persists. :-(
We are using a adapted version of the AWS S3 OTA in our application and the patch for the peek function still returns an error 8. Upon closer inspection, the magic byte read and the actual magic byte are not equal. I just used a couple of serial.print at the verifyheader function in update.cpp library to print out the two values.
The defined magic byte is 233 ( 0xE9 ) but we end up reading 255 ( FF ).
Can you verify this on your end @copercini .

The only working fix still seems to be disabling the actual verifyheader check in the writestream function. The OTA sketch works, but only when we disable the verifyheader / peek function.

I use TinyGSM to dowload via Update. Also hit FFFF... at the beginning.
At the end, I need to disable verify Header.
I don't know how FFFF jumps into the frame, via AT command?

[IoTThinks]

Hello,
I just pitched in to report that the issue still persists. :-(
We are using a adapted version of the AWS S3 OTA in our application and the patch for the peek function still returns an error 8. Upon closer inspection, the magic byte read and the actual magic byte are not equal. I just used a couple of serial.print at the verifyheader function in update.cpp library to print out the two values.
The defined magic byte is 233 ( 0xE9 ) but we end up reading 255 ( FF ).
Can you verify this on your end @copercini .
The only working fix still seems to be disabling the actual verifyheader check in the writestream function. The OTA sketch works, but only when we disable the verifyheader / peek function.

I use TinyGSM to dowload via Update. Also hit FFFF... at the beginning.
At the end, I need to disable verify Header.
I don't know how FFFF jumps into the frame, via AT command?

It is due to unimplemented peek() in TinyGSM. The peek() always return -1 aka 255 or FF.
vshymanskyy/TinyGSM#396 (comment)

[JosephShepin]

i'm running an esp32 with the OTA sketch and I am getting

Got 357280 bytes from server
contentLength : 357280, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
Written only : 0/357280. Retry?
Error Occurred. Error #: 8

Is there still no other solution other than modifying the library? This was even using the StartCounter.ino.bin that was in the example folder - not even my own sketch. I was able to get it working by removing the tester for that, but I do want to understand why it's not working the way it should.