Add simple RLE compression for ArduinoOTA #6609
Conversation
When uploading large, sparse filesystems, there are often many repeated, empty sectors. These take a long time to transmit, even over WiFi. Implement a basic RLE compression in espota.py (and only use it if it actually saves upload size), and a simple decompressor in ArduinoOTA. The actual flash update image is decompressed as received to the staging location, so the bootloader/etc. do not need to be aware of it (i.e. a 1MB filesystem still takes 1MB of flash in the update area, even if it only took 100KB to upload it). Fixes esp8266#4277
|
Forgot to note in the commit, this change transparently supports uploading to older ArduinoOTA servers by adding the "I can compress" flag at the end of the invite, in a section the older OTA simply ignores. |
There was a problem hiding this comment.
Several objections to the approach:
- The RLE algorithm is implemented entangled with Stream, and the class even has a WiFiClient inside. It is therefore usable only in this specific scenario, and I think it would be very difficult to extend for usage in other scenarios. The compression algorithm is completely unrelated to Stream, WiFiClient, or other things, so should be implemented in a generic way (something similar to base64 comes to mind). If such an implementation were done, then this class should tie that implementation with the other things. Then, the underlying algorithm would be available for other use cases. Also, it's just good design practice not to entangle functionality.
- The implementation is specific to ArduinoOTA-espota.py. It doesn't allow use with e.g.: OTA over webserver or via httpclient or other means. In addition, the compression code is contained within espota.py, and therefore can't be used from outside of that particular script on the PC side.
If point 1 above is addressed, it would make it easier to address this point, or at least leave the door open to address it in the future. - If it is left for the future to implement support for the other OTA cases, a doc entry is needed somewhere to explain that (transparent) RLE compression is supported only for OTA via ArduinoOTA, and not for the other OTA cases.
- I had outlined the requirements for bin compression in this comment, where point 2 references RLE. This implementation doesn't address any of the cases there.
- The discussion surrounding bin compression centered around 2 parts: impact for transmission speed, and impact for written image size, where the latter has the advantage of easing the bin size constraints for boards with smaller flash sizes (sketch size near the size/2 limit). Given points 1 and 2 above, this implementation allows only one very limited use case of the first and precludes the second.
Let's discuss further before moving forward.
It's a The decompression can also be pushed down into Updater (maybe a special header to ID it), but then all the MD5 calculations would need to be adjusted to the RLE compressed bin. If minigz also gets ported, it too should have a
That's 20KB + the window size (8KB or whatever). So it would need to be done while in eboot (and therefore minigz code image needs to fit in the ~2kb of free eboot space). Having 28KB+ free while doing OTA is possible (not with HTTPS), but I doubt it's common...
Fair enough. We don't really import local modules into the existing Python code, but the actual compression bit is a 20 line function that can migrate to its own file simply enough.
Re the 1st point there, no compression guarantees it'll actually be able to shrink data (your point about uploading larger images than free flash), so best-effort is all you can hope for there. RLE does speed uploads, a lot, for sparse FSes, so the 2nd point is taken care of. It also doesn't slow down anything since it checks compressed size < raw before sending compressed. Bin size increase for RLE is about as small as you're likely to get, 3rd point. As Meatloaf said, "Two out of three ain't bad." :)
That's a good idea, and I can almost guarantee a simple RLE decompress will fit in the eboot sector. I hear your concern, though, so let's consider this a strawman thrown out there and see where it goes. |
| delete[] _block; | ||
| } | ||
|
|
||
| virtual int read() { |
| return ret; | ||
| } | ||
|
|
||
| virtual int peek() { |
|
|
||
| size_t read(uint8_t*a, size_t&b) { return readBytes((char*)a, b); } | ||
|
|
||
| virtual size_t readBytes(char *buffer, size_t length) { |
|
|
||
| virtual size_t write(uint8_t b) { return _client.write(b); } | ||
|
|
||
| virtual int available() { |
|
What's the code size overhead ? I like the feature, but ArduinoOTA is one of the classes that many will use, and the footprint increases in the last few releases didn't make it easy to upgrade. |
|
@dirkmueller it adds 976 bytes to the ArduinoOTA code and ~150 bytes of heap when used. RLE was used to be a simple and low-mem way of compressing FS updates. This doesn't really shrink apps, so they'd just use uncompressed uploads w/the logic in espota. |
|
Thanks for the quick reply. Imho the 976 bytes addition is quite a lot. Can we make it a compile time option? The current way it is implemented means I think that it will always be included. |
|
Closing this for now. I should be able to fit this into eboot with room to spare (so 0 impact on program size), and push a new PR that way. Avoid the whole issue of other modes not being able to utilize it, too. |
When uploading large, sparse filesystems, there are often many repeated,
empty sectors. These take a long time to transmit, even over WiFi.
Implement a basic RLE compression in espota.py (and only use it if it
actually saves upload size), and a simple decompressor in ArduinoOTA.
The actual flash update image is decompressed as received to the staging
location, so the bootloader/etc. do not need to be aware of it (i.e. a
1MB filesystem still takes 1MB of flash in the update area, even if it
only took 100KB to upload it).
Fixes #4277