Uploading large files to SPIFFS introduces wrong bytes in the files

[me-no-dev]

I have a verry interesting problem uploading larger files to the SPIFFS
if the file is larger than 226KB, I get at very particular points wrong bytes
those are exactly the next byte after 226KB, 364KB, 728KB and I imagine every 364KB after that
offsets correctpond to 0x38801, 0x5B001, 0xB6001, ... so on
and the wrong byte is usually only missing the first bit (if it's set) but sometimes the value is all around wrong

 $ cmp -l map.png ~/Downloads/map.png 
 231425 145 144
 372737 265 264
 745473 135 134

[luc-github]

Yes can reproduce with PR SPIFFS 0.3.3 also
C:\Users\user>cmp -l C:\Users\user\Desktop\moizhi.png C:\Users\user\Desktop\moizhi2.png
231425 325 324

[mytrain]

@me-no-dev
I've got the same issue when uploading large file on the SPIFFS (see issue #792 that is still open)
The only work-arround was to reduce file size (by zipping them outside of the ESP) and forcing the content header in the ESP web server. [sendHeader("Content-Encoding", "gzip");].

I thing issue #792 can be stapled with the present one.
Regards

[me-no-dev]

back a few months ago when we had the first SPIFFS implementation there were no such issues. I think something in the driver is causing it and since @iggr wrote it this time, I hope he can help figuring it out

[me-no-dev]

@luc-github what is "PR SPIFFS 0.3.3"?

[igrr]

Thanks for spotting this! Please try the fix in 4a3d20c.

[me-no-dev]

same thing :(

[luc-github]

@me-no-dev Sorry I should use link : #899

[igrr]

I was able to reproduce the issue using FSBrowser sample and 1 mb file filled with 0xff, before the fix. Not able to reproduce now. Are you sure the fix was applied? Can you try to re-upload clean FS image?

[luc-github]

@igrr what you mean by clean FS Image, use mkspiffs ? I did delete file and uploaded then downloaded without patch I have :
C:\Users\user\Desktop>cmp -l moizhi.png moizhi2.png
231425 325 324

I have deleted the file and applied the patch then uploaded then downloaded and I have :
C:\Users\user\Desktop>cmp -l moizhi.png moizhitest2.png
231423 25 377
231424 260 25
231425 325 220
372736 304 377
372738 25 4

then I deleted, removed patch , upload then download and got :
C:\Users\user\Desktop>cmp -l moizhi.png moizhinopatch.png
231425 325 324

I am using #899 but not the FBBrowser sample but my simplified upload js

[me-no-dev]

@igrr I introduced an error in the source which showed during compilation, so patch is applied. No go on the fix though.
Tried format of the FS, upload using the form, using curl and still the same :)

[me-no-dev]

maybe try using real data like larger image (what I use)

[iggr]

Hi guys, could you please be more accurate when referencing someone in your conversations,
"unfortunately" my nick is @iggr which is very similar to the @igrr and every time someone mention me by mistake I got email notification. Thanks~

[me-no-dev]

Sorry mate, seems like I made the typo :)

[luc-github]

Actually with patch it is worse for me, not for you @me-no-dev ?

[igrr]

@luc-github could you please share the complete sample to reproduce (sketch + data + png)?

[luc-github]

I use FM I did on my project (https://github.com/luc-github/ESP8266)

the data is my dog's picture no issue to share 😄

• I can do a sample sketch - and html page with my simplified uploader, it would be more easy if you want

[me-no-dev]

@luc-github I did not check with larger image, but saw the error at the same exact spot which meant it did not work :) maybe should have checked if it had gotten worst

[igrr]

Okay, got it to reproduce with a different file. Looking into it.

[igrr]

That was a pretty lame bug. I fixed the number of bytes written for short (1-2 bytes) unaligned writes, but messed up the offset at which they were written in the process. Please try 74c16ec.

[luc-github]

New patch fix issue for me, thank you 😄

Also here a sample sketch/data as requested with my simplied uploader/ FM - sorry I am slow
https://github.com/luc-github/ESP8266/tree/master/testfiles/upload

[me-no-dev]

I can also confirm the fix works