Skip to content

Add run-length encoding for compressing OTA SPIFFS? #4277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
sheffieldnikki opened this issue Feb 2, 2018 · 4 comments
Closed

Add run-length encoding for compressing OTA SPIFFS? #4277

sheffieldnikki opened this issue Feb 2, 2018 · 4 comments
Assignees
@earlephilhower
Labels
component: OTA type: enhancement
Milestone
2.7.0

Comments

@sheffieldnikki
Copy link

Basic Infos

Hardware

Hardware: ESP-12F
Core Version: 2.4.0

Description

Would it be possible to add a very simple run-length encoding scheme to compress SPIFFS file as they are transferred to the ESP8266 ? Because the SPIFFS are generally limited to a few options (e.g. 1MB or 3MB) users are very likely to have unused space in their SPIFFS image, which is easily compressible runs of 0xFF.

Alternatively, transfer pages of SPIFFS data that have data in them (uncompressed), but embed a code to indicate to the ESP8266 that "all following pages are to be written as empty" ?

In my tests it takes 51 secs to transfer a 1MB SPIFFS image OTA to an ESP8266 over a strong WiFi signal, or about 20KB/sec.

(My workaround for now is defining a small SPIFFS area (e.g. 64KB) in my boards.txt that will upload in just a few seconds.)
@earlephilhower earlephilhower self-assigned this Jan 24, 2019
@earlephilhower
Copy link
Collaborator

I mentioned #5635 fixing this before, but I didn't realize you were talking OTA. #5635 speeds up only serial uploads (and does it a LOT in my experience on SPIFFS), but has nothing to do w/OTA. The stub compression would need to be added to the Arduino OTA infrastructure, including 8266 driver code and host-based espota.py.

@davisonja
Copy link

If there's that much free space how about a transfer that only sent used bytes and where to put them?

@earlephilhower
Copy link
Collaborator

Simple RLE, only looking at # of consecutive 0xff or 0x00. I like it!

earlephilhower added a commit to earlephilhower/Arduino that referenced this issue Oct 6, 2019
@earlephilhower
Add simple RLE compression for ArduinoOTA
75dad16 
When uploading large, sparse filesystems, there are often many repeated,
empty sectors.  These take a long time to transmit, even over WiFi.

Implement a basic RLE compression in espota.py (and only use it if it
actually saves upload size), and a simple decompressor in ArduinoOTA.

The actual flash update image is decompressed as received to the staging
location, so the bootloader/etc. do not need to be aware of it (i.e. a
1MB filesystem still takes 1MB of flash in the update area, even if it
only took 100KB to upload it).

Fixes esp8266#4277
@earlephilhower earlephilhower mentioned this issue Oct 6, 2019
Closed
@devyte devyte added this to the 2.7.0 milestone Nov 9, 2019
@earlephilhower
Copy link
Collaborator

#6820 solved this with gzunzip support in eboot, I believe. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@earlephilhower earlephilhower

Labels
component: OTA type: enhancement
Projects
None yet
Milestone
2.7.0
Development

Successfully merging a pull request may close this issue.

Add simple RLE compression for ArduinoOTA earlephilhower/Arduino
4 participants
@earlephilhower @davisonja @sheffieldnikki @devyte