Add HTTP header with MD5 sum to ESP8266httpUpdate OTA update

[JeroenVogelpoel]

Considering that we now have access to ESP.getSketchSize() and ESP.getSketchMD5(), I would like to ask that an HTTP header with the key 'x-ESP8266-sketch-md5' and the value of the 'String(ESP.getSketchMD5()))' call be added to Arduino/libraries/ESP8266httpUpdate/src/ESP8266httpUpdate.cpp

The rationale behind this is that if the MD5 sums match; the binary to be loaded is the same as the one already on device and we can skip the upload with HTTP response code 304 (Not Modified) meaning no binary will be transfered and no flashing will be done.

It's a bit more solid imho than a freeform version string that needs to be persisted somewhere.

Thoughts/opinions?

[DeanCording]

Have you actually gotten this to work? I had the thought too, that if the device sent the MD5 hash of its current firmware then the server could compare it to the MD5 hash of the current firmware bin file to determine if an update was available. This would allow you to just copy new firmware to the server and it would automatically sort out versions. Unfortunately the MD5 hashes never match between the sketch and the bin file.

[JeroenVogelpoel]

That's generally the idea behind it and how it's supposed to work, yes.

Could you please post some code (ESP & web server serving updates) to have a look at and which version you're currently using?

EDIT:
Odd, your findings seem to contradict what I have seen. At home I have a setup going where the ESP reports it's sketch MD5 via headers as a request for OTA. At that point, the webserver compares the inbound MD5 with the most recent bin file MD5.

If there's a match; the ESP contains the most recent bin and OTA is halted by returning HTTP code 304, "Not Modified". If there is no match, the most recent bin will be pushed to the ESP as part of an OTA update.

[DeanCording]

On the ESP it is just the standard ESP8266httpUpdate example code using the latest version of the library with the MD5 header added.

I am developing a server using Node Red which is an event/flow environment built on node.js.

On my server I have a flow for uploading firmware that receives the file, calculates the MD5 hash of it, and then saves both the file and the hash. There is another flow that handles httpUpdate requests from ESPs and compares the sent MD5 hash with the saved MD5 hash.

It's ok, I have it working now - there was an issue with the MD5 calculation of binary files on my server.

[SensorsIot]

I am also interested in this topic. I have the normal httpUpdate with the versions working, but think, your proposal is much easier. Which version of the boards manager files did you use? My version (2.3.0-rc2 does not send the md5.
I use php to calculate the md5 (md5_file($fileN)) of the latest file. Is this correct?

[DeanCording]

You need to use the 2.4.0rc version of ESP8266httpUpdate,cpp

or you can just apply this patch

Then it is just a matter of comparing the md5 sent in the request with the md5 of the bin file.

[mfalkvidd]

@SensorsIot yes, md5_file is the correct command in php. Example code is available at https://github.com/esp8266/Arduino/blob/edba2d2829db3b9788e7fc96a9dfeb40bf067326/doc/ota_updates/readme.md#advanced-updater-1

[JeroenVogelpoel]

Please notice that you wrapped the correct md5_file call in another md5
call. You're now getting the MD5 hash of $fileN's MD5 hash.

Apparently the changes haven't been released yet; you can patch the changes
in yourself with the aforementioned link.

On Tue, Sep 20, 2016 at 4:07 PM, SensorsIOT [email protected]
wrote:

I am also interested in this topic. I have the normal httpUpdate with the
versions working, but think, your proposal is much easier. Which version of
the boards manager files did you use? My version (2.3.0-rc2 does not send
the md5.
I use php to calculate the md5 (md5_file($fileN)) of the latest file. Is
this correct?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#2228 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKgbe2o4YU_dLXyYTh_kGRPBcjkyv-Nsks5qr-i9gaJpZM4JEqCO
.

[SensorsIot]

Thanks for the comments and the links. I looked at the php file and it is exactly what I need. Unfortunately, I do not know where to get the 2.4rc version of ESP8266httpUpdate,cpp. Can anybody help?
BTW, I did a YouTube video using the 2.3 version: https://youtu.be/UiAc3yYBsNU (enhanced with the separation of credentials from code)

[DeanCording]

You can just add the patch that I linked to directly to the esp8266httpUpdate.cpp file in your
installation. It is only one line and the library will be automatically recompiled when you next
compile your program.

On Tuesday, 20 September 2016 10:37:37 PM AEST SensorsIOT wrote:

Thanks for the comments and the links. I looked at the php file and it is exactly what I need.
Unfortunately, I do not know where to get the 2.4rc version of ESP8266httpUpdate,cpp. Can
anybody help?BTW, I did a YouTube video using the 2.3 version: https://youtu.be/
UiAc3yYBsNU[1](enhanced with the separation of credentials from code)
—You are receiving this because you commented.Reply to this email directly, view it on
GitHub[2], or mute the thread[3].[4]

---

[1] https://youtu.be/UiAc3yYBsNU
[2] #2228 (comment)
[3] https://github.com/notifications/unsubscribe-auth/
AAPVIhwtPnUvrMGEQxwg-24O4P1v9Qu-ks5qsMKhgaJpZM4JEqCO
[4] https://github.com/notifications/beacon/
AAPVIgbv3yeHQ9FwxM9eZnYEJTDUBA8hks5qsMKhgaJpZM4JEqCO.gif

[SensorsIot]

Thanks for the fast reply. Now I get the error: error: 'class EspClass' has no member named 'getSketchMD5'. I am using 2.3.0-rc2. Do I need to include a special library?

[DeanCording]

Ahh, sorry my bad.

You will also need this patch

Might be simpler just to copy and replace the entire Arduino/cores/esp8266/Esp.cpp file.

[SensorsIot]

Thanks for your help. Now it works. I just had to comment the last line of the PHP script (header($_SERVER["SERVER_PROTOCOL"].' 500 no version for ESP MAC', true, 500);) because it created a warning and did not send the file. I have to admit, I do not know a lot about PHP ;-)
Now it is much easier to do OTA. Just compile, transfer the file to the webserver, and reboot. Great!

[DeanCording]

Even easier - set the ESP8266 to deep sleep for a short period and it will automatically reboot and update. Great for in-situ development.

[neilyoung]

In case someone is interested: Here is a simple NodeJS version of the update server.

The version check can surely be improved. It requires you to provide a version number string on ESP while calling ESPhttpUpdate.update()

http = require("http");
url = require("url");
path = require("path");
fs = require("fs");
md5 = require('md5-file');

function check_header(request, header, value) {
	if (!request.headers[header.toLowerCase()])
		return false;
	
	if (value && request.headers[header.toLowerCase()] !== value)
		return false;
	return true;
}

var available_version = "4712";

http.createServer(function(request,response){
	
	if (!check_header(request, 'USER-AGENT', 'ESP8266-http-Update') ||
		!check_header(request, 'X-ESP8266-STA-MAC') ||
		!check_header(request, 'X-ESP8266-AP-MAC') ||
    	!check_header(request, 'X-ESP8266-FREE-SPACE') ||
    	!check_header(request, 'X-ESP8266-SKETCH-SIZE') ||
		!check_header(request, 'X-ESP8266-SKETCH-MD5') ||
    	!check_header(request, 'X-ESP8266-CHIP-SIZE') ||
    	!check_header(request, 'X-ESP8266-SDK-VERSION' ||
		!check_header(request, "X-ESP8266-VERSION"))) {
			response.writeHeader(403, {"Content-Type": "text/plain"});
			response.write("403 Forbidden");
		
	}
	else {
		if (check_header(request, "X-ESP8266-VERSION", available_version)) {
			response.writeHeader(304, {"Content-Type": "text/plain"});  
			response.write("304 Not Modified\n");  
			response.end();
		}
		else {
			var my_path = url.parse(request.url).pathname;
			var full_path = path.join(process.cwd(),my_path);
			
			fs.exists(full_path,function(exists){
				if (!exists) {
					response.writeHeader(404, {"Content-Type": "text/plain"});  
					response.write("404 Not Found\n");  
					response.end();
				}
				else {
					fs.readFile(full_path, "binary", function(err, file) {  
						if (err) {  
							response.writeHeader(500, {"Content-Type": "text/plain"});  
							response.write(err + "\n");  
							response.end();  
						}  
						else{
							response.writeHeader(200, 
								{"Content-Type": "application/octet-stream", 
								"Content-Disposition": "attachment;filename="+path.basename(full_path),
								"Content-Length": ""+fs.statSync(full_path)["size"],
								"x-MD5":  md5.sync(full_path)});  
							response.write(file, "binary");  
							response.end();
						}
				
					});				
				}	
			});
		}


	}



}).listen(8080);
console.log("Server running");

[SensorsIot]

We implemented the MD5 mechanism in our IOTappstory.com infrastructure and it works great.
Now, we also implemented the download of the SPIFFS OTA from the same site. Unfortunately, MD5 is not supported by the SPIFFS downloader. Would it be possible to write a patch for that, too?

[earlephilhower]

Fixed w/ #2236 some time back.