Skip to content

Feature/http update #1979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Nov 19, 2018
Merged

Feature/http update #1979

merged 8 commits into from
Nov 19, 2018

Conversation

Jeroen88
Copy link
Contributor

@Jeroen88 Jeroen88 commented Oct 18, 2018

Added HTTPUpdate class for downloading sketches from a server. It is somewhat similar to AWS_S3_OTA_update.ino from the libraries/Update/examples, however the adapted HTTPClient from PR #1973 is used in stead of printing HTTP headers directly to the client.

An example usage httpUpdateSecure.ino is given. This updates a firmware over a secure connection using the WiFiClientSecure class and a CA certificate. Of course a non secure WiFiClient may also be used.

The good thing is: it works. A bad thing might be that a few features that are present in the similar library for the ESP8266 are not present in this library. The reason for this is that these features rely on functions not (yet) implemented in the ESP32 libraries. However, these features are not essential to the functionality of the HTTPUpdate library. The missing features are:

  • The HTTP Request Headers for free sketch space, sketch size, sketch MD5 and flash chip real size are not set due to missing functions in the ESP class. These headers are not essential for serving an update, the server hosting the binaries just should not expect these headers
  • Log information showing sizes is also missing. This is also unessential
  • The HTTPUpdate library does not check beforehand if the binaries fit into either flash or spiffs, also because the needed ESP functions are missing. However the UpdateClass::begin() function is called at the start of the update and this function does check if the binary fits.
  • Because peekBytes() is not implemented in WiFiClient, only the magic header byte 0xE9 is checked. After this magic byte follow bytes that indicate the length of the sketch to be uploaded. This length could be checked with the available length, but the available length could also not be determined (see bullit 2). This is also not essential, the library checks if it fits in another step (at Updater::begin())
  • WiFiUDP::stopAll()and WiFiClient::stopAllExcept(tcp) are not called, also because these functions are not implemented (yet). As I recall correctly these functions were introduced due to issues with the ESP8266 WiFi chip. I did not encounter any issues on my ESP32 not calling these functions
  • The code for all above missing features is still present in the library but commented out with // To do

The Update class is also changed:

  • A small bug is solved: the UpdateClass::writeStream() function always tries to read a full buffer at the end of the stream. But at the end the last buffer is almost never full, and readBytes() exits on a timeout. With the fix the correct number of bytes is read, and the unneeded waiting is removed
  • Two optional parameters ledPin and ledOn are added to UpdateClass::begin(). Settng the ledPin to LED_BUILTIN and ledOn to HIGH causes the built in LED to flash. The LED is on while reading a buffer from the internet and it is off during writing that buffer to flash. In this way visual feedback of the update process may be given. Passing -1 for LedPin causes this feature to be disabled. The included example makes use of this feature.

The HTTPClient from PR #1973 is included in this PR, because HTTPUpdate depends on it

@Jeroen88
Copy link
Contributor Author

Jeroen88 commented Oct 22, 2018

I observed, among others, quite some timeouts on doing a secure firmware update using HTTPClient (as stated before, actually I do not write directly to a HTTPClient for an update but I use HTTPUpdate of this PR, and HTTPUpdate uses HTTPClient).

I think the root cause of this lies in the fact that, in contrast to the ESP8266 WiFiClient, the ESP32 WiFiClient has a method WiFiClient::setTimeout(uint32_t seconds). This means that when HTTPClient calls _client -> setTimeout() it calls WiFiClient::setTimeout(uint32_t seconds) and not Stream::setTimeout(unsigned long timeout). Also note that the WiFiClient::setTimout() parameter is in seconds while the Stream::setTimeout() parameter is in milliseconds
Also, because of PR #788, the HTTClient::setTimeout() tests for the underlying socket not to be secure (if(connected() && !_secure) …) before calling _client -> setTimeout(). This causing _client -> setTimeout() not to be called on secure connections while actually it should.

The consequences of these errors are:

  • On a non secure connection, _client -> setTimeout() is called. This is WiFiClient::setTimeout(). It is called with a milliseconds parameter while the function expects seconds. So a the timeout is a factor 1,000 to high, resulting in effectively no timeout at all
  • On a secure connection, _client -> setTimeout() is not called at all

To make things even worse, WiFiClient::setTimeout() does not call Stream::setTimeout(). This means that the timeout for the Stream is the default timeout of 1,000 ms. This again means that Stream::readBytes(), Stream::readBytesUntil(), Stream::readString() and Stream::readStringUntil() will not get the same timeout as HTTPClient(), which is normally much longer (5000 ms, and even longer). This is true for both insecure and secure connections!

The next thing I observe is that PR #934 introduces a test on !_secure in HTTPClient::getStream(). It is perfectly legal to return a secure stream and reading from it, so this check is invalid.

To correct all this, I made the following changes:

  • In HTTPClient::setTimeout() and HTTPClient::getStream() remove the test for !_secure
  • In HTTPClient::setTimeout() divide and round the millisecond timeout to the nearest second in calling WiFiClient::setTimeout()
  • In WiFiClient::setTimeout() call Stream::setTimeout() and multiply the seconds by 1,000 again, because Stream::setTimeout expects milliseconds.

In this way the library functions behavior change is kept to the bare minimum necessary for correcting the errors. Still the consequence of this is that an insecure connection now gets a reasonable timeout while before it had a timeout that was 1,000 times too long (so virtually no timeout at all). Also now a secure connection gets a timeout. Finally, since reasonable timeouts on a HTTPClient are longer that 1,000 ms (the default for the Stream timeout), the Stream::readBytes(), Stream::readBytesUntil(), Stream::readString() and Stream::readStringUntil() now get more time than 1s to complete.

I expect this PR will solve issue #1443.

I expect that the change in this PR will lead to less timeouts and more completed downloads!

@Jeroen88 Jeroen88 mentioned this pull request Oct 22, 2018
@Jeroen88
Copy link
Contributor Author

@lbernstone @me-no-dev Any chance that these commits will be merged soon?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants