feature/HTTPUpdate-headers: add response headers with sketch and flash sizes, and a SHA256 #2116

Jeroen88 · 2018-11-27T19:31:15Z

This PR is a follow up of some of the left overs still to do in PR #1979.

The request headers x-ESP32-free-space, x-ESP32-sketch-size and x-ESP32-chip-size are added. Because a SHA256 is already supported by the core functions while a MD5 isn't, a x-ESP32-sketch-sha256 is added instead of a x-ESP32-sketch-md5 header.

Logging for sketch size and free sketch space is added.
For a sketch the length is checked to fit in the free space before flashing starts. For SPIFFS this feature is still “to do”. This is not essential, the size is checked later by Updater::begin().

Because peekBytes() is not implemented in WiFiClient, only the magic header byte 0xE9 is checked. After this magic byte follow bytes that indicate the length of the sketch to be uploaded. This length could be checked with the available length. This is also not essential, the library checks if it fits in another step (at Updater::begin()).

WiFiUDP::stopAll()and WiFiClient::stopAllExcept(tcp) as called in the ESP8266 version of the library are not called in this library, because these functions are not implemented (yet). As I recall correctly these functions were introduced due to issues with the ESP8266 WiFi chip. I did not encounter any issues on my ESP32 not calling these functions.

me-no-dev · 2018-11-27T20:27:47Z

Nice :)

me-no-dev · 2018-11-27T20:28:46Z

BTW those F("strings") are pointless :) just use the "string" as is.

Jeroen88 · 2018-11-27T21:15:52Z

@me-no-dev thnx for the compliment and for the SHA256 idea. I created a new PR for removing the F() macro's.
I am still a git beginner. Just to be sure I created a new PR. Could I also reuse the branche I created and push again?

TLS1000 · 2018-12-04T19:23:08Z

I'm trying to use the httpupdate demo with AWS S3 bucket.
code below works with the demo sketch from the 'update' library

the demo sketch with from the 'update' library works on S3 with following arguments:
// S3 Bucket Config
String host = "espfirmware.s3-eu-west-1.amazonaws.com"; // Host => bucket-name.s3.region.amazonaws.com
int port = 80; // Non https. For HTTPS 443. As of today, HTTPS doesn't work.
String bin = "/AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin"; // bin file name with a slash in front.

I'm trying different things but can't get it right:

t_httpUpdate_return ret = httpUpdate.update(client, "espfirmware.s3-eu-west-1.amazonaws.com/AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin");
    // Or:
    //t_httpUpdate_return ret = httpUpdate.update(client, "http://espfirmware.s3-eu-west-1.amazonaws.com", 80, "AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin");

connection refused, connection closed, core panic,.... but nothing that works. Any suggestions?

Jeroen88 · 2018-12-05T06:57:33Z

Can you post your full sketch? I am not familiar with AWS S3, does it check the HTTP request headers that are sent?

Jeroen88 · 2018-12-05T07:07:11Z

Maybe the error is caused because you miss http:// in front of the url?

TLS1000 · 2018-12-05T07:48:50Z

Hi Jeroen, I tried all sorts of combinations, I think the problem is something else.
The BIN file on AWS is public, it's just the example sketch from the 'update' library. Feel free to try to try to upload it, the file is 'public':
[
https://s3-eu-west-1.amazonaws.com/espfirmware/AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin ]

the non secured http download from the 'upload' library uses [(http://espfirmware.s3-eu-west-1.amazonaws.com/AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin)
]

this is the code from the 'update' library that works, it's the identical example sketch just modified wifi name and SSID, and AWS URL:
`/**
AWS S3 OTA Update
Date: 14th June 2017
Author: Arvind Ravulavaru https://github.com/arvindr21
Purpose: Perform an OTA update from a bin located in Amazon S3 (HTTP Only)

Upload:
Step 1 : Download the sample bin file from the examples folder
Step 2 : Upload it to your Amazon S3 account, in a bucket of your choice
Step 3 : Once uploaded, inside S3, select the bin file >> More (button on top of the file list) >> Make Public
Step 4 : You S3 URL => http://bucket-name.s3.ap-south-1.amazonaws.com/sketch-name.ino.bin
Step 5 : Build the above URL and fire it either in your browser or curl it curl -I -v http://bucket-name.ap-south-1.amazonaws.com/sketch-name.ino.bin to validate the same
Step 6: Plug in your SSID, Password, S3 Host and Bin file below

Build & upload
Step 1 : Menu > Sketch > Export Compiled Library. The bin file will be saved in the sketch folder (Menu > Sketch > Show Sketch folder)
Step 2 : Upload bin to S3 and continue the above process

// Check the bottom of this sketch for sample serial monitor log, during and after successful OTA Update
*/

#include <WiFi.h>
#include <Update.h>

WiFiClient client;

// Variables to validate
// response from S3
int contentLength = 0;
bool isValidContentType = false;

// Your SSID and PSWD that the chip needs
// to connect to
const char* SSID = "xxxxx";
const char* PSWD = "xxxxx";

// S3 Bucket Config
String host = "espfirmware.s3-eu-west-1.amazonaws.com"; // Host => bucket-name.s3.region.amazonaws.com
int port = 80; // Non https. For HTTPS 443. As of today, HTTPS doesn't work.
String bin = "/AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin"; // bin file name with a slash in front.

void execOTA() {
Serial.println("Connecting to: " + String(host));
// Connect to S3
if (client.connect(host.c_str(), port)) {
// Connection Succeed.
// Fecthing the bin
Serial.println("Fetching Bin: " + String(bin));

// Get the contents of the bin file
client.print(String("GET ") + bin + " HTTP/1.1\r\n" +
             "Host: " + host + "\r\n" +
             "Cache-Control: no-cache\r\n" +
             "Connection: close\r\n\r\n");

// Check what is being sent

// Serial.print(String("GET ") + bin + " HTTP/1.1\r\n" +
// "Host: " + host + "\r\n" +
// "Cache-Control: no-cache\r\n" +
// "Connection: close\r\n\r\n");

unsigned long timeout = millis();
while (client.available() == 0) {
  if (millis() - timeout > 5000) {
    Serial.println("Client Timeout !");
    client.stop();
    return;
  }
}
// Once the response is available,
// check stuff

/*
   Response Structure
    HTTP/1.1 200 OK
    x-amz-id-2: NVKxnU1aIQMmpGKhSwpCBh8y2JPbak18QLIfE+OiUDOos+7UftZKjtCFqrwsGOZRN5Zee0jpTd0=
    x-amz-request-id: 2D56B47560B764EC
    Date: Wed, 14 Jun 2017 03:33:59 GMT
    Last-Modified: Fri, 02 Jun 2017 14:50:11 GMT
    ETag: "d2afebbaaebc38cd669ce36727152af9"
    Accept-Ranges: bytes
    Content-Type: application/octet-stream
    Content-Length: 357280
    Server: AmazonS3
                               
    {{BIN FILE CONTENTS}}

*/
while (client.available()) {
  // read line till /n
  String line = client.readStringUntil('\n');
  // remove space, to check if the line is end of headers
  line.trim();

  // if the the line is empty,
  // this is end of headers
  // break the while and feed the
  // remaining `client` to the
  // Update.writeStream();
  if (!line.length()) {
    //headers ended
    break; // and get the OTA started
  }

  // Check if the HTTP Response is 200
  // else break and Exit Update
  if (line.startsWith("HTTP/1.1")) {
    if (line.indexOf("200") < 0) {
      Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
      break;
    }
  }

  // extract headers here
  // Start with content length
  if (line.startsWith("Content-Length: ")) {
    contentLength = atoi((getHeaderValue(line, "Content-Length: ")).c_str());
    Serial.println("Got " + String(contentLength) + " bytes from server");
    Serial.println(line);
  }
  Serial.println(line.substring(0,100));
  // Next, the content type
  if (line.startsWith("Content-Type: ")) {
    String contentType = getHeaderValue(line, "Content-Type: ");
    Serial.println("Got " + contentType + " payload.");
    if (contentType == "binary/octet-stream") {
      isValidContentType = true;
    }
  }
}

} else {
// Connect to S3 failed
// May be try?
// Probably a choppy network?
Serial.println("Connection to " + String(host) + " failed. Please check your setup");
// retry??
// execOTA();
}

// OTA Logic

// Check what is the contentLength and if content type is application/octet-stream
//Serial.println(line.substring(0,100));
Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

// check contentLength and content type
if (contentLength && isValidContentType) {
// Check if there is enough to OTA Update
bool canBegin = Update.begin(contentLength);

// If yes, begin
if (canBegin) {
  Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
  // No activity would appear on the Serial monitor
  // So be patient. This may take 2 - 5mins to complete
  size_t written = Update.writeStream(client);

  if (written == contentLength) {
    Serial.println("Written : " + String(written) + " successfully");
  } else {
    Serial.println("Written only : " + String(written) + "/" + String(contentLength) + ". Retry?" );
    // retry??
    // execOTA();
  }

  if (Update.end()) {
    Serial.println("OTA done!");
    if (Update.isFinished()) {
      Serial.println("Update successfully completed. Rebooting.");
      ESP.restart();
    } else {
      Serial.println("Update not finished? Something went wrong!");
    }
  } else {
    Serial.println("Error Occurred. Error #: " + String(Update.getError()));
  }
} else {
  // not enough space to begin OTA
  // Understand the partitions and
  // space availability
  Serial.println("Not enough space to begin OTA");
  client.flush();
}

} else {
Serial.println("There was no content in the response");
client.flush();
}
}

// Utility to extract header value from headers
String getHeaderValue(String header, String headerName) {
return header.substring(strlen(headerName.c_str()));
}

void setup() {
//Begin Serial
Serial.begin(115200);
delay(10);

Serial.println("Connecting to " + String(SSID));

// Connect to provided SSID and PSWD
WiFi.begin(SSID, PSWD);

// Wait for connection to establish
while (WiFi.status() != WL_CONNECTED) {
Serial.print("."); // Keep the serial monitor lit!
delay(500);
}

// Connection Succeed
Serial.println("");
Serial.println("Connected to " + String(SSID));

// Execute OTA Update
execOTA();
}

void loop() {
// chill
}`

Jeroen88 · 2018-12-05T13:22:20Z

The error had nothing to do with the HTTPUpdate library. It was caused by PR #2148 in the function WiFiClient::available()
I just pushed PR #2155 to fix it.

Testing on your URL your firmware is downloaded. I get an error though:
assertion "false && "item should have been present in cache"" failed: file "/Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/nvs_flash/src/nvs_item_hash_list.cpp", line 85, function: void nvs::HashList::erase(size_t)

I suspect this has something to do with this:
https://www.esp32.com/viewtopic.php?t=6763
and this:
espressif/esp-idf@c455d6f

I do not understand that issue completely, but could you compile the test sketch that you want to serve from AWS and the sketch you are running both with the latest core?

I will do another PR to do a few additional checks in HTTPUpdate(), but this does not cause your trouble.

TLS1000 · 2018-12-05T13:30:52Z

the sketch in AWS was probably not compiled with the latest core, I'll update the bin file and let you know.
Would be great if S3update with version control would work, in my opinion the 'holy grale' of OTA-updates. setting up S3 storage is super easy and takes just minutes even for non-ITspecialists like me.

Jeroen88 · 2018-12-05T14:06:56Z

The server could check what version you are requesting on several ways:

The url includes the file name you are requesting. You could code versions in the file name
If you add a third parameter, a String with a current version, this string is passed to the server in a HTTP request header. This header is named x-ESP32-version
Now that the SHA256 hash is passed in the HTTP request header x-ESP32-sketch-sha256 you could at least determine on your server that the sketch running differs from the one you are serving and let that be the criteria for update

TLS1000 · 2018-12-05T18:18:38Z

hi Jeroen, now the binary file in AWS is compiled with latest firmware 1,0,1-rc. I don't see any update possibility to use your latest changes to httpupdate library. What is the best way?

Anyway, when I run the httpupdate sketch, it returns: HTTP_UPDATE_FAILD Error (-100): Not Enough space

Jeroen88 · 2018-12-05T20:18:53Z

@TLS1000 Is it true that your sketch checks the chip id and will fail if it isn't a predefined value? In that case I succeeded in downloading your sketch. I get:

chip id and program id don't match!

The Not enough space issue is because recently a function was added to check the free space. This checks the free space of the memory partition where the sketch is running in. However the ESP32 has a second partition, and that size should be used. I am going to ask the developers to add a function to get the size of the other partition and will make a PR afterwards.

If you want to try it now you can make the changes yourself. Just edit two files. You can find these files in a sub directory where you installed the ESP32 Arduino core. The path is hardware/espressif/esp32/libraries/

Edit WiFi/src/WiFiClient.cpp

find

int WiFiClient::available()

Replace this whole function with:

int WiFiClient::available()
{
    if(!_rxBuffer)
    {
        return 0;
    }
    int res = _rxBuffer->available();
    if(_rxBuffer->failed()) {
        log_e("%d", errno);
        stop();
    }
    return res;
}

Next edit HTTPUpdate/src/HTTPUpdate.cpp

Find the line

if(len > (int) ESP.getFreeSketchSpace()) {

Replace with

if(false && len > (int) ESP.getFreeSketchSpace()) {

That should do the trick!

Jeroen88 · 2018-12-05T22:18:32Z

@TLS1000 I created PR #2161 that fixes the size check and the error in WiFiClient.

Instead of editing the files, you could wait for the PR to be merged, which will be soon I hope!

TLS1000 · 2018-12-06T13:19:06Z

Yes, I create bin files that only work on one board, since the files are public on AWS...

If you see that message it means the OTA was successful.

One thing I found out is that AWS S3 puts an 'ETag' in the response header, from what I understood some sort of CRC/MD5 of the file. Via the ETag in the response header the ESP can also check if there is different version available. I store the ETag in EEPROM so it's still available after OTA.

Jeroen88 · 2018-12-06T13:45:23Z

If you see that message it means the OTA was successful.

That's nice :)

Jeroen88 · 2018-12-06T16:40:57Z

I think I finally understand what you want. The AWS is just a file server, so you can not control the response headers sent. If you store the ETag in EEPROM you could compare it with the ETag presently served at AWS using your URL. If they differ you know the sketch has to be updated.
The HTTP HEAD request asks the server just to send the response headers, not the file. The HEAD command is not supported by HTTPClient, but it can be easily done:

/**
  * HTTPClient_HEAD.ino
  *
  * Created on: 06.12.2018
  *
  * By J. Döll
*/

#include <Arduino.h>

#include <WiFi.h>
#include <WiFiMulti.h>

#include <HTTPClient.h>

#include <WiFiClient.h>

WiFiMulti WiFiMulti;

void setup() {

  Serial.begin(115200);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("SSID", "PASSWORD");
}

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    std::unique_ptr<WiFiClient>client(new WiFiClient);

    HTTPClient http;

    Serial.println("[HTTP] begin..");
    if (http.begin(*client, "http://espfirmware.s3-eu-west-1.amazonaws.com/AWS_S3_OTA_Update_WKE.ino.doitESP32devkitV1.bin")) {  // HTTP
      // Set the header to collect
      const char* headerNames[] = { "ETag" };
      http.collectHeaders(headerNames, sizeof(headerNames)/sizeof(headerNames[0]));

      Serial.println("[HTTP] HEAD...");
      // start connection and send HTTP headers
      int httpCode = http.sendRequest("HEAD");

      // httpCode will be negative on error
      if (httpCode > 0) {
        // HTTP header has been sent and Server response header has been handled
        Serial.printf("[HTTP] HEAD... code: %d\n", httpCode);

        // file found at server
        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          // Check if the response header was returned
          if (http.hasHeader("ETag")) {
            String ETag = http.header("ETag");
            Serial.printf("[HTTP] ETag header is %s\n", ETag.c_str());
          } else {
            Serial.println("[HTTP] Response header was not sent");
          }
        }
      } else {
        Serial.printf("[HTTP] HEAD... failed, error: %s\n", http.errorToString(httpCode).c_str());
      }

      http.end();
    } else {
      Serial.println("[HTTP] Unable to connect");
    }
  }

  Serial.println("Wait 10s before next round...");
  delay(10000);
}

This gives the following output:

[SETUP] WAIT 4...
[SETUP] WAIT 3...
[SETUP] WAIT 2...
[SETUP] WAIT 1...
[HTTP] begin..
[HTTP] HEAD...
[HTTP] HEAD... code: 200
[HTTP] ETag header is "ce7358251022d9143d324084f9f5bcb5"
Wait 10s before next round...

Don't forget to set your WiFi password!

TLS1000 · 2018-12-06T17:02:09Z

haha, and I thought you understood me the whole way :-)
yes, that's what I want to do.
It's working now, great stuff, thanks Jeroen!

EEPROM library is also doing great stuff for me with functions like the EEPROM.writeString() and readString() to easily write and read the ETag value.

Now cracking the hornbill AWS IOT library and I can do both remote process monitoring and firmware update.

Jeroen88 · 2018-12-06T17:10:19Z

:-D, you're welcome, good luck with the IoT library!

Jeroen88 · 2018-12-07T17:13:00Z

@TLS1000 I just made PR #2164 and #2165, maybe these are useful for you :)

Add response headers with sketch and flash sizes, and a SHA256

b503375

Jeroen88 mentioned this pull request Nov 27, 2018

HTTPUpdate not working. Update.h:44:10: note: candidate expects 2 arguments, 4 provided #2094

Closed

me-no-dev merged commit 7e9afe8 into espressif:master Nov 27, 2018

Jeroen88 mentioned this pull request Nov 27, 2018

Remove F() macro's from HTTPUpdate.cpp #2121

Merged

Jeroen88 mentioned this pull request Dec 3, 2018

compare OTA1 with OTA2 partition, then decide to update or not? #2143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature/HTTPUpdate-headers: add response headers with sketch and flash sizes, and a SHA256 #2116

feature/HTTPUpdate-headers: add response headers with sketch and flash sizes, and a SHA256 #2116

Jeroen88 commented Nov 27, 2018

me-no-dev commented Nov 27, 2018

me-no-dev commented Nov 27, 2018

Jeroen88 commented Nov 27, 2018

TLS1000 commented Dec 4, 2018 •

edited

Loading

Jeroen88 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018

TLS1000 commented Dec 5, 2018 •

edited

Loading

Jeroen88 commented Dec 5, 2018

TLS1000 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018

TLS1000 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018 •

edited

Loading

TLS1000 commented Dec 6, 2018

Jeroen88 commented Dec 6, 2018

Jeroen88 commented Dec 6, 2018 •

edited

Loading

TLS1000 commented Dec 6, 2018

Jeroen88 commented Dec 6, 2018

Jeroen88 commented Dec 7, 2018

feature/HTTPUpdate-headers: add response headers with sketch and flash sizes, and a SHA256 #2116

feature/HTTPUpdate-headers: add response headers with sketch and flash sizes, and a SHA256 #2116

Conversation

Jeroen88 commented Nov 27, 2018

me-no-dev commented Nov 27, 2018

me-no-dev commented Nov 27, 2018

Jeroen88 commented Nov 27, 2018

TLS1000 commented Dec 4, 2018 • edited Loading

Jeroen88 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018

TLS1000 commented Dec 5, 2018 • edited Loading

Jeroen88 commented Dec 5, 2018

TLS1000 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018

TLS1000 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018

Jeroen88 commented Dec 5, 2018 • edited Loading

TLS1000 commented Dec 6, 2018

Jeroen88 commented Dec 6, 2018

Jeroen88 commented Dec 6, 2018 • edited Loading

TLS1000 commented Dec 6, 2018

Jeroen88 commented Dec 6, 2018

Jeroen88 commented Dec 7, 2018

TLS1000 commented Dec 4, 2018 •

edited

Loading

TLS1000 commented Dec 5, 2018 •

edited

Loading

Jeroen88 commented Dec 5, 2018 •

edited

Loading

Jeroen88 commented Dec 6, 2018 •

edited

Loading