WIP - Add time to SPIFFS for 3.0.0

[earlephilhower]

Support the ESP32 File::getLastWrite() call and setting the time on
all filesystems automatically (assuming the system clock has
been set properly and time(NULL) returns the proper time!).

Adds Dir::fileTime() to get the time of a file being listed, similar to
Dir::fileName() and Dir::fileSize().

Adds ::setTimeCallback(time_t (*cb)()) to File, Dir, and FS, allowing
users to override the default timestamp on a per-file, directory, or
filesystem basis. By default, a simple callback returning time(nullptr)
is implemented.

LittleFS uses the 't' attribute and should be backwards compatible.

SD/SDFS work and include wrappers for obsolete SdFat timestamp callbacks
using the MSDOS time.

SPIFFS has been updated to support either the existing on-flash format
without any timestamps, or a new on-flash format with a flag area and a
timestamp metadata element added.

SPIFFS will try and mount any existing filesystem as non-timestamp, and
continue to write them in the old format.

New filesystems may be formatted to include timestamps (default) or in
the old format, configurable via the SPIFFS.setConfig call.

The SPIFFS work took as starting point all of work of @luc-github in #3730
and extended it into a backwards compatible mode by doing much surgery
on the SPIFFS library itself and the on-flash format.

Includes an updated SD/listfiles and SPIFFS_time example.

[earlephilhower]

mkspiffs and mklittlefs need to be updated to match the new format. Will look at that after we get some approval on the changes here. Using the old mkspiffs will run fine, but it won't include timestamps on the generated filesystem.

[earlephilhower]

@luc-github, can you give this a look-see? I've got a way to make your SPIFFS change #3730 compatible with the existing flash layout, so that we don't have to wait until 3.0 or make people wipe their existing FSes.

As you found out, SPIFFS unfortunately doesn't have anything like a FSInfo block describing the filesystem config options and it will happily mount a metadata=4 filesystem with a metadata=0 application (resulting in silent data corruption on reads and writes). The way I handled it was to add a poison page to the beginning of the FS flash space. At runtime I now check and see if the poison is there, and if so we know it's a new md=4 FS and mount(flashstart+4K). If it's not, then it's mounted as a md=0 FS. The poison also makes sure that old code can't mount and read/write corrupted data to a md=4 FS.

[luc-github]

Ho looks good - the trick is clever - thank you ^_^
Will do my best to try ASAP

[earlephilhower]

Functionality is there, but moving to WIP because I want to include a data protection test to either host tests or device tests that will verify the correct behavior (i.e. guarantee no corruption of a non-time, older FS, and that it preserves existing older FS w/o metadata when run with the new code; and guarantee that the new FS can not be mounted by the old code (which would also cause corruption since the old code cannot handle metadata).

[luc-github]

@earlephilhower postpone affect LittleFS also ?

[earlephilhower]

@luc-github ,For LittleFS there is 0 risk in adding this (attributes are part and parcel of the FS and don't affect any layout considerations), but unfortunately with this PR I've upgraded everything. So if SPIFFS doesn't go in, neither does SD or LittleFS.

I will try to get a host test this weekend with a pre-built old-format and new-format FS. If that doesn't happen I can probably split this PR into one for LittleFS+SD and one for SPIFFS. The LittleFS can go in the next rev, no question, and the SPIFFS will be a matter of if it gets enough testing.

I've worked at data storage companies for a long time, and the two worst words I can hear are "data corruption," so I want to make sure SPIFFS is solid.

[luc-github]

Thank you for your great contribution. I have implemented LittleFS support in my project. I was just checking if I can enable time support, and saw the postpone comment.
I will wait the pr, no problem.
Thank you again

[d-a-v]

Replaced by #6544

[earlephilhower]

Reopening just because, once #6544 is done, this PR will end up just being the changes to SPIFFS to make it support the metadata size as a variable.

It's lots of little things, and will be used if/when we go to a partition setup with embedded FS info blocks to avoid the poison page added.

So, not for 2.x, but I don't want to lose the SPIFFS changes which will still be needed (I think).

[NicoZuid]

According to the documentation of the file system
{url} https://arduino-esp8266.readthedocs.io/en/latest/filesystem.html {url]
there is a (date) time of the last Write available per file.
After reading the above discussion I wonder if the documentation of ESP SDK version 2.6.3 is correct.
It says: "The SD, SDFS, and LittleFS filesystems support a file timestamp, updated when the file is opened for writing." Do I not have to wait for version 3.0.0?
(I have been trying to make a PIFFS and SD version of my program for several hours :))

[earlephilhower]

@NicoZuid if you use LittleFS and SD, you'll be all set with a file write time. I updated SPIFFS to support it, too, but because it is a massive change and relies on a hack to determine which FS version is in use, you'll have to wait for V3.0 for SPIFFs.

[NicoZuid]

@earlephilhower, Thanks for your answer.
I am trying to create an SDFS version of my program.
Unfortunately, the SD card suddenly became completely unusable.
The card is no longer even recognized as an SD card on a PC.
Cause? Hardware card reader, SD card or software.

[earlephilhower]

That really sounds like a hardware issue. You need to be very careful with the voltages applied and pinout on the card/adapter. A SW issue could only scramble the data, at worst, and the card itself would still be recognized (but your OS would probably ask you to reformat).

This really isn't the right spot for this, though. You'll be able to get more help on https://gitter.im/esp8266/Arduino .

[NicoZuid]

As mentioned above, the SD card was suddenly no longer usable on both the esp hardware and on a Windows PC. I have just been able to view the SD card with a PC with Ubuntu.
The volume header and / or the directory header is probably currupt.
Both give: Free space: 2.0 GB from 2.0 GB (0% used).
I am sure the information was previously correct ALSO through my program on the ESP8266 hardware. In reality there were around 120 files with a total of around 5.5 MB.
If I can reproduce the problem in due course I will return to it in a separate issue.

[earlephilhower]

Closing due to SPIFFS deprecation. I think the code's pretty solid, but I don't think it's appropriate to do such a massive change to a dead-end codebase.