Updater class corrupting FS uploads by setting flash mode byte in 1st sector

[mike-s123]

Basic Info

• This issue complies with the [issue POLICY doc]
• I have read the documentation at... and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: WEMOS D1 Mini Pro (clone)
• Core Version: Core:2.6.2-4-g074a8016=20602004
• Development Env: Arduino IDE 1.8.10
• Operating System: Win7-64

Settings in IDE

• Module: Lolin D1 Mini Pro
• Flash Size: 16MB (FS:14MB...)
• lwip Variant: v2 Higher Bandwidth
• CPU Frequency: 160MHz
• Upload Using: OTA

Problem Description

Uploading a large (~14MB) LittleFS image via ESP8266HTTPUpdateServer doesn't work (i.e. LittleFS.open() fails and the sketch can not open/read files).

Uploading the same image using USB and esptool.py (either manually or with the IDE uploader) works fine. Then, uploading the same image using OTA causes failure.

With an identically sized SPIFFs image, both USB and OTA uploads work fine. OTA firmware uploads work fine.

This has been 100% repeatable.

using esptool.py (this works, OTA doesn't)

$ esptool.py --before default_reset --after hard_reset -p COM10 --baud 921600 write_flash 0x200000 FS.mklittlefs.bin
esptool.py v2.6
Serial port COM10
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
MAC: xxxx
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 16MB
Compressed 14655488 bytes to 1038072...
Wrote 14655488 bytes (1038072 compressed) at 0x00200000 in 42.1 seconds (effective 2784.4 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

[earlephilhower]

Can you try with spiffs, too? Should be the same, as far as the uploader knows it's just bits.

[mike-s123]

As stated, SPIFFs (and firmware) uploads work just fine.

[mike-s123]

@earlephilhower
Update. I created a simple program to dump the first 16 bytes of the flash starting at 0x200000, where the filesystem starts. The upload using esptool.py exactly matches the original file. The upload using ESP8266HTTPUpdateServer had corrupted the first few bytes (specifically, it set the third byte to 2 when it was originally 0):

Good (esptool.py):
Successfully mounted LittleFS
spi_flash_read result :0
read0:1
read1:f7ff0ff0
read2:7474696c
read3:7366656c

Bad (OTA):
An Error has occurred while mounting LittleFS
spi_flash_read result :0
read0:20001   
read1:f7ff0ff0
read2:7474696c
read3:7366656c

File (note that due to "endedness", byte order is reversed compared to what's shown above):
Generated by HexEdit
00000000  01 00 00 00 f0 0f ff f7 6c 69 74 74 6c 65 66 73  ....ð.ÿ÷littlefs

[earlephilhower]

Very strange indeed. Have you uploaded a smaller FS (say, 1M)? I'm thinking if it's data corruption like that, then the FS size itself would not impact things. Might also be the Updater patching in something that it shouldn't (like it does when patching in ROM size in app images).

[earlephilhower]

Arduino/cores/esp8266/Updater.cpp

Lines 326 to 347 in a738884

	// If the flash settings don't match what we already have, modify them.
	// But restore them after the modification, so the hash isn't affected.
	// This is analogous to what esptool.py does when it receives a --flash_mode argument.
	bool modifyFlashMode = false;
	FlashMode_t flashMode = FM_QIO;
	FlashMode_t bufferFlashMode = FM_QIO;
	if (_currentAddress == _startAddress + FLASH_MODE_PAGE) {
	flashMode = ESP.getFlashChipMode();
	#ifdef DEBUG_UPDATER
	DEBUG_UPDATER.printf_P(PSTR("Header: 0x%1X %1X %1X %1X\n"), _buffer[0], _buffer[1], _buffer[2], _buffer[3]);
	#endif
	bufferFlashMode = ESP.magicFlashChipMode(_buffer[FLASH_MODE_OFFSET]);
	if (bufferFlashMode != flashMode) {
	#ifdef DEBUG_UPDATER
	DEBUG_UPDATER.printf_P(PSTR("Set flash mode from 0x%1X to 0x%1X\n"), bufferFlashMode, flashMode);
	#endif
	_buffer[FLASH_MODE_OFFSET] = flashMode;
	modifyFlashMode = true;
	}
	}

Looks like the Updater class is always potentially corrupting the first sector of a FS upload. SPIFFS doesn't check its structures very much (if at all) when mounting, so it might just ignore this.

The flashmode change update needs to be predicated on !U_FS.

[mike-s123]

@earlephilhower

Configured as a Lolin D1 (4MB flash), with FS:2MB, which I'd expect to be a common config for a more common board, since it should allow OTA updates for both code and FS. Similar results:

Upload via esptool.py:
Successfully mounted LittleFS
spi_flash_read result :0
read0:1
read1:f7ff0ff0
read2:7474696c
read3:7366656c

Upload via OTA:
An Error has occurred while mounting LittleFS
spi_flash_read result :0
read0:20001
read1:f7ff0ff0
read2:7474696c
read3:7366656c

File:
Generated by HexEdit
00000000  01 00 00 00 f0 0f ff f7 6c 69 74 74 6c 65 66 73  ....ð.ÿ÷littlefs

[earlephilhower]

Changed title to underlying problem, @mike-s123, so that the changelist will be more meaningful. Thanks for the hex dump, it pinpointed this very quickly!

[earlephilhower]

@mike-s123 please try out the PR in #6891 and report back. Now the 1 byte change should only happen to applications, not filesystems.

[mike-s123]

Works for me. Thanks, Earle.