HTTP OTA with sketch size more than 512K

[sukretniy]

Platform

• Hardware: [ESP-WROOM-02](it is based on esp8266 with 2MB flash size)
• Core Version: [latest git hash or date]
• Development Env: [Arduino IDE]
• Operating System: [Windows]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [dio]
• Flash Size: [2MB]
• lwip Variant: [v2 Lower Memory]
• Reset Method: [ck]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

I receive HTTP_UPDATE_FAILED Error (-100): Not Enough space
while updating firmware with ESP8266httpUpdate (HTTP OTA)

My sketch has 549965 bytes (more, then 512K), but ESP.getFlashChipRealSize() gives me 2097152

In ESP8266httpUpdate I found this

if(len > (int) ESP.getFreeSketchSpace()) {
                    DEBUG_HTTP_UPDATE("[httpUpdate] FreeSketchSpace to low (%d) needed: %d\n", ESP.getFreeSketchSpace(), len);
                    startUpdate = false;
                }

Maximum sketch size can be 1044464.
Am i rigth that if i have sketch more, then half of 1044464 i can not update it with ESP8266httpUpdate, despite that flash size is 2097152?

What can be done, to overcome this limitation?

[Marc-Vreeburg]

Maybe your flash got corrupted. Try wiping your flash.

[Pablo2048]

You need to set more than your "Maximum skecth size 1044464" because of http://esp8266.github.io/Arduino/versions/2.0.0/doc/ota_updates/ota_updates.html especially:
"Basic Requirements
Flash chip size is 2x the size of the sketch."

[devyte]

@sukretniy I don't understand your issue. First you say that you tried to update with a sketch that is 549965 in size and it failed, than you ask whether the 1M limitation can be overcome. That doesn't make sense, please clarify.

The maximum sketch size is a bit less than half of your flash size. If you flash size is 2MB, then your max sketch size is slightly less than 1MB.
In addition, I understand that the Espressif SDK has a limitation that allows sketches of only up to 1MB (I'm not 100% certain of this).
In both cases, the only option is to migrate to a board with more flash and use SPIFFS (see next comment).
Off the top of my head, I find your bin size to be huge. My own app is just over 400KB in size, and it passed 50K lines of code a few months ago. Are you using embedded html/js/css? If so, I suggest moving that to a small SPIFFS to decrease the size of the bin. The SPIFFS size doesn't have the bin size limitation.

[sukretniy]

I'll try to clarify

1. I flush ESP-WROOM-02 (which has 2M) with firmware, that has 549965 bytes, using com port (option Flash Size in Arduino IDE Is set as "2M (1M SPIFFS)", but to clarify, in any place in my project i do not use SPIFFS.begin())
   At this point ESP.getFreeSketchSpace() returns 491520 (may be because Espressif SDK has a limitation that allows sketches of only up to 1MB, the question is if it is possible to overcome this limitation)
   And ESP.getFlashChipRealSize() returns 2097152

2. I have update server, on which there is new firmware with size 549965 and existing firmware checks update server for the new firmware existence and tries to update. At this point i receive HTTP_UPDATE_FAILED Error (-100): Not Enough space,
   because ESP8266httpUpdate.cpp has this code

if(len > (int) ESP.getFreeSketchSpace()) {
                    DEBUG_HTTP_UPDATE("[httpUpdate] FreeSketchSpace to low (%d) needed: %d\n", ESP.getFreeSketchSpace(), len);
                    startUpdate = false;
                }

Len here is size of new firmware = 549965 and ESP.getFreeSketchSpace() = 491520.

I have to mention, that if new firmware size less, than ESP.getFreeSketchSpace() everything goes fine.

My sketch size dramatically increased after i start using this library
https://github.com/Azure/azure-iot-arduino

[sukretniy]

Ok, I got it #4252

[devyte]

@sukretniy a quick look into that azure lib tells me that
-the code is rather intertwined
-because of the previous, the current lack of link time optimization is likely playing against you
-the code may work on the ESP8266, but it does not look optimized for it (lots of strings that aren't declared on flash)

Not much can be done there, but I thought I'd mention it.

About SPIFFS, the fact that you don't call SPIFFS.begin() makes no difference. The space for the filesystem is set aside in the flash mapping depending on whatever SPIFFS is selected in the IDE menu. In your case, you have a 1MB SPIFFS on a 2MB flash, so that leaves only about 1MB for both sketch and OTA, hence about 512KB max sketch size. What you would need is a smaller SPIFFS, or maybe a non-SPIFFS setup.
Given #4252 , I think @d-a-v would be the person to discuss that with.
My thought is that a small/non SPIFFS setup should be a general option available for all boards.

[d-a-v]

2M flash configuration was the only one without unused flash size,
this is fixed by #4988 .
Also, Let's not forget about #905 that would allow to OTA with 1M sketches and at least 2M chips.

[d-a-v]

Nothing can further be done for this issue. Two-steps OTA may help, or #905.
Suggested 2M0 configuration is merged.