Add LittleFS as an optional filesystem, API compatible w/SPIFFS (but not on-flash-format compatible) (#5511)

* Add LittleFS as internal flash filesystem

Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.

LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.

LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling.  This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.

Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.

**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ .  The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.

The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.

Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next().  On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).

On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs.  So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements.  The returned filenames also only have the
filename returned, not full paths.  So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS.  This is consistent with
POSIX ideas about reading directories and more natural for a FS.

Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.

Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.

Subdirectories are made silently when they do not exist when you
try and create a file in a subdir.  They are silently removed when
the last file in them is deleted.  This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior.  Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.

SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.

* Fix up merge blank line issue

* Merge in the FSConfig changs from SDFS PR

Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.

* Fix merge errors

* Update to use v2-alpha branch

The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).

* V2 of littlefs is now in upstream/master

* Update test to support non-creation-ordered files

In a directory, the order in which "readNextFile()" will return a name
is undefined.  SPIFFS may return it in order, but LittleFS does not as
of V2.  Update the test to look for files by name when doing
readNextFile() testing.

* Fix LittleFS.truncate implementation

* Fix SDFS tests

SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.

* Update to point to mklittlefs v2

Upgrade mklittlefs to V2 format support

* Remove extra FS::write(const char *s) method

This was removed in #5861 and erroneously re-introduced here.

* Minimize spurious differences from master

* Dramatically reduce memory usage

Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.

* Add @d-a-v's host emulation for LittleFS

* Fix SW Serial library version

* Fix free space reporting

Thanks to @TD-er for discovering the issue

* Update littlefs to latest upstream

* Remove sdfat version included by accident

* Update SDFAT to include MOCK changes required

* Update to include SD.h test of file append

Author: earlephilhower

.gitmodules

@@ -7,6 +7,9 @@
 [submodule "libraries/SoftwareSerial"]
 	path = libraries/SoftwareSerial
 	url = https://github.com/plerup/espsoftwareserial.git
+[submodule "libraries/LittleFS/lib/littlefs"]
+	path = libraries/LittleFS/lib/littlefs
+	url = https://github.com/ARMmbed/littlefs.git
 [submodule "libraries/ESP8266SdFat"]
 	path = libraries/ESP8266SdFat
 	url = https://github.com/earlephilhower/ESP8266SdFat.git

README.md

@@ -143,3 +143,5 @@ ESP8266 core files are licensed under LGPL.
 [axTLS](http://axtls.sourceforge.net/) library written by Cameron Rich, built from https://github.com/igrr/axtls-8266, is used in this project. It is distributed under [BSD license](https://github.com/igrr/axtls-8266/blob/master/LICENSE).
 
 [BearSSL](https://bearssl.org) library written by Thomas Pornin, built from https://github.com/earlephilhower/bearssl-esp8266, is used in this project.  It is distributed under the [MIT License](https://bearssl.org/#legal-details).
+
+[LittleFS](https://github.com/ARMmbed/littlefs) library written by ARM Limited and released under the [BSD 3-clause license](https://github.com/ARMmbed/littlefs/blob/master/LICENSE.md).

cores/esp8266/Esp.cpp

@@ -519,14 +519,14 @@ uint32_t EspClass::getSketchSize() {
     return result;
 }
 
-extern "C" uint32_t _SPIFFS_start;
+extern "C" uint32_t _FS_start;
 
 uint32_t EspClass::getFreeSketchSpace() {
 
     uint32_t usedSize = getSketchSize();
     // round one sector up
     uint32_t freeSpaceStart = (usedSize + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
-    uint32_t freeSpaceEnd = (uint32_t)&_SPIFFS_start - 0x40200000;
+    uint32_t freeSpaceEnd = (uint32_t)&_FS_start - 0x40200000;
 
 #ifdef DEBUG_SERIAL
     DEBUG_SERIAL.printf("usedSize=%u freeSpaceStart=%u freeSpaceEnd=%u\r\n", usedSize, freeSpaceStart, freeSpaceEnd);

cores/esp8266/FS.h

@@ -62,7 +62,7 @@ class File : public Stream
     int read() override;
     int peek() override;
     void flush() override;
-    size_t readBytes(char *buffer, size_t length)  override {
+    size_t readBytes(char *buffer, size_t length) override {
         return read((uint8_t*)buffer, length);
     }
     size_t read(uint8_t* buf, size_t size);

cores/esp8266/Updater.cpp

@@ -24,7 +24,7 @@ extern "C" {
     #include "user_interface.h"
 }
 
-extern "C" uint32_t _SPIFFS_start;
+extern "C" uint32_t _FS_start;
 
 UpdaterClass::UpdaterClass()
 : _async(false)
@@ -87,8 +87,8 @@ bool UpdaterClass::begin(size_t size, int command, int ledPin, uint8_t ledOn) {
   }
 
 #ifdef DEBUG_UPDATER
-  if (command == U_SPIFFS) {
-    DEBUG_UPDATER.println(F("[begin] Update SPIFFS."));
+  if (command == U_FS) {
+    DEBUG_UPDATER.println(F("[begin] Update Filesystem."));
   }
 #endif
 
@@ -112,7 +112,7 @@ bool UpdaterClass::begin(size_t size, int command, int ledPin, uint8_t ledOn) {
     //size of current sketch rounded to a sector
     size_t currentSketchSize = (ESP.getSketchSize() + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
     //address of the end of the space available for sketch and update
-    uintptr_t updateEndAddress = (uintptr_t)&_SPIFFS_start - 0x40200000;
+    uintptr_t updateEndAddress = (uintptr_t)&_FS_start - 0x40200000;
     //size of the update rounded to a sector
     size_t roundedSize = (size + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
     //address where we will start writing the update
@@ -130,8 +130,8 @@ bool UpdaterClass::begin(size_t size, int command, int ledPin, uint8_t ledOn) {
       return false;
     }
   }
-  else if (command == U_SPIFFS) {
-     updateStartAddress = (uintptr_t)&_SPIFFS_start - 0x40200000;
+  else if (command == U_FS) {
+     updateStartAddress = (uintptr_t)&_FS_start - 0x40200000;
   }
   else {
     // unknown command
@@ -279,8 +279,8 @@ bool UpdaterClass::end(bool evenIfRemaining){
 #ifdef DEBUG_UPDATER
     DEBUG_UPDATER.printf_P(PSTR("Staged: address:0x%08X, size:0x%08zX\n"), _startAddress, _size);
   }
-  else if (_command == U_SPIFFS) {
-    DEBUG_UPDATER.printf_P(PSTR("SPIFFS: address:0x%08X, size:0x%08zX\n"), _startAddress, _size);
+  else if (_command == U_FS) {
+    DEBUG_UPDATER.printf_P(PSTR("Filesystem: address:0x%08X, size:0x%08zX\n"), _startAddress, _size);
 #endif
   }
 
@@ -392,7 +392,7 @@ bool UpdaterClass::_verifyHeader(uint8_t data) {
             return false;
         }
         return true;
-    } else if(_command == U_SPIFFS) {
+    } else if(_command == U_FS) {
         // no check of SPIFFS possible with first byte.
         return true;
     }
@@ -426,7 +426,7 @@ bool UpdaterClass::_verifyEnd() {
         }
 
         return true;
-    } else if(_command == U_SPIFFS) {
+    } else if(_command == U_FS) {
         // SPIFFS is already over written checks make no sense any more.
         return true;
     }

cores/esp8266/Updater.h

@@ -21,7 +21,7 @@
 #define UPDATE_ERROR_SIGN               (12)
 
 #define U_FLASH   0
-#define U_SPIFFS  100
+#define U_FS      100
 #define U_AUTH    200
 
 #ifdef DEBUG_ESP_UPDATER

cores/esp8266/spiffs_hal.cpp renamed to cores/esp8266/flash_hal.cpp

@@ -21,8 +21,8 @@
 #include <Arduino.h>
 #include <stdlib.h>
 #include <algorithm>
-#include "spiffs/spiffs.h"
 #include "debug.h"
+#include "flash_hal.h"
 
 extern "C" {
 #include "c_types.h"
@@ -42,10 +42,10 @@ alignedBegin:            ^
 alignedEnd:                      ^
 */
 
-int32_t spiffs_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
+int32_t flash_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
     optimistic_yield(10000);
 
-    uint32_t result = SPIFFS_OK;
+    uint32_t result = FLASH_HAL_OK;
     uint32_t alignedBegin = (addr + 3) & (~3);
     uint32_t alignedEnd = (addr + size) & (~3);
     if (alignedEnd < alignedBegin) {
@@ -58,7 +58,7 @@ int32_t spiffs_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
         if (!ESP.flashRead(alignedBegin - 4, &tmp, 4)) {
             DEBUGV("_spif_read(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                 __LINE__, addr, size, alignedBegin, alignedEnd);
-            return SPIFFS_ERR_INTERNAL;
+            return FLASH_HAL_READ_ERROR;
         }
         memcpy(dst, ((uint8_t*) &tmp) + 4 - nb, nb);
     }
@@ -68,7 +68,7 @@ int32_t spiffs_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
                 alignedEnd - alignedBegin)) {
             DEBUGV("_spif_read(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                 __LINE__, addr, size, alignedBegin, alignedEnd);
-            return SPIFFS_ERR_INTERNAL;
+            return FLASH_HAL_READ_ERROR;
         }
     }
 
@@ -78,7 +78,7 @@ int32_t spiffs_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
         if (!ESP.flashRead(alignedEnd, &tmp, 4)) {
             DEBUGV("_spif_read(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                 __LINE__, addr, size, alignedBegin, alignedEnd);
-            return SPIFFS_ERR_INTERNAL;
+            return FLASH_HAL_READ_ERROR;
         }
 
         memcpy(dst + size - nb, &tmp, nb);
@@ -99,7 +99,7 @@ int32_t spiffs_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
 
 static const int UNALIGNED_WRITE_BUFFER_SIZE = 512;
 
-int32_t spiffs_hal_write(uint32_t addr, uint32_t size, uint8_t *src) {
+int32_t flash_hal_write(uint32_t addr, uint32_t size, const uint8_t *src) {
     optimistic_yield(10000);
 
     uint32_t alignedBegin = (addr + 3) & (~3);
@@ -116,7 +116,7 @@ int32_t spiffs_hal_write(uint32_t addr, uint32_t size, uint8_t *src) {
         if (!ESP.flashWrite(alignedBegin - 4, (uint32_t*) tmp, 4)) {
             DEBUGV("_spif_write(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                 __LINE__, addr, size, alignedBegin, alignedEnd);
-            return SPIFFS_ERR_INTERNAL;
+            return FLASH_HAL_WRITE_ERROR;
         }
     }
 
@@ -128,7 +128,7 @@ int32_t spiffs_hal_write(uint32_t addr, uint32_t size, uint8_t *src) {
                     alignedEnd - alignedBegin)) {
                 DEBUGV("_spif_write(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                     __LINE__, addr, size, alignedBegin, alignedEnd);
-                return SPIFFS_ERR_INTERNAL;
+                return FLASH_HAL_WRITE_ERROR;
             }
         }
         else {
@@ -140,7 +140,7 @@ int32_t spiffs_hal_write(uint32_t addr, uint32_t size, uint8_t *src) {
                 if (!ESP.flashWrite(alignedBegin, (uint32_t*) buf, willCopy)) {
                     DEBUGV("_spif_write(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                         __LINE__, addr, size, alignedBegin, alignedEnd);
-                    return SPIFFS_ERR_INTERNAL;
+                    return FLASH_HAL_WRITE_ERROR;
                 }
 
                 sizeLeft -= willCopy;
@@ -158,14 +158,14 @@ int32_t spiffs_hal_write(uint32_t addr, uint32_t size, uint8_t *src) {
         if (!ESP.flashWrite(alignedEnd, &tmp, 4)) {
             DEBUGV("_spif_write(%d) addr=%x size=%x ab=%x ae=%x\r\n",
                 __LINE__, addr, size, alignedBegin, alignedEnd);
-            return SPIFFS_ERR_INTERNAL;
+            return FLASH_HAL_WRITE_ERROR;
         }
     }
 
-    return SPIFFS_OK;
+    return FLASH_HAL_OK;
 }
 
-int32_t spiffs_hal_erase(uint32_t addr, uint32_t size) {
+int32_t flash_hal_erase(uint32_t addr, uint32_t size) {
     if ((size & (SPI_FLASH_SEC_SIZE - 1)) != 0 ||
         (addr & (SPI_FLASH_SEC_SIZE - 1)) != 0) {
         DEBUGV("_spif_erase called with addr=%x, size=%d\r\n", addr, size);
@@ -177,8 +177,8 @@ int32_t spiffs_hal_erase(uint32_t addr, uint32_t size) {
         optimistic_yield(10000);
         if (!ESP.flashEraseSector(sector + i)) {
             DEBUGV("_spif_erase addr=%x size=%d i=%d\r\n", addr, size, i);
-            return SPIFFS_ERR_INTERNAL;
+            return FLASH_HAL_ERASE_ERROR;
         }
     }
-    return SPIFFS_OK;
+    return FLASH_HAL_OK;
 }

cores/esp8266/flash_hal.h

@@ -0,0 +1,49 @@
+#ifndef flash_hal_h
+#define flash_hal_h
+
+/*
+ flash_hal.h - API for accessing raw flash for filesystems
+ Copyright (c) 2015 Ivan Grokhotkov. All rights reserved.
+
+ This code was influenced by NodeMCU and Sming libraries, and first version of
+ Arduino wrapper written by Hristo Gochkov.
+
+ This file is part of the esp8266 core for Arduino environment.
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+#ifdef ARDUINO
+extern "C" uint32_t _FS_start;
+extern "C" uint32_t _FS_end;
+extern "C" uint32_t _FS_page;
+extern "C" uint32_t _FS_block;
+
+#define FS_PHYS_ADDR ((uint32_t) (&_FS_start) - 0x40200000)
+#define FS_PHYS_SIZE ((uint32_t) (&_FS_end) - (uint32_t) (&_FS_start))
+#define FS_PHYS_PAGE ((uint32_t) &_FS_page)
+#define FS_PHYS_BLOCK ((uint32_t) &_FS_block)
+#endif
+
+// Return values of the following functions
+#define FLASH_HAL_OK          (0)
+#define FLASH_HAL_READ_ERROR  (-1)
+#define FLASH_HAL_WRITE_ERROR (-2)
+#define FLASH_HAL_ERASE_ERROR (-3)
+
+extern int32_t flash_hal_write(uint32_t addr, uint32_t size, const uint8_t *src);
+extern int32_t flash_hal_erase(uint32_t addr, uint32_t size);
+extern int32_t flash_hal_read(uint32_t addr, uint32_t size, uint8_t *dst);
+
+#endif // !defined(flash_hal_h)

cores/esp8266/spiffs/spiffs.h

@@ -84,7 +84,7 @@ struct spiffs_t;
 /* spi read call function type */
 typedef s32_t (*spiffs_read)(struct spiffs_t *fs, u32_t addr, u32_t size, u8_t *dst);
 /* spi write call function type */
-typedef s32_t (*spiffs_write)(struct spiffs_t *fs, u32_t addr, u32_t size, u8_t *src);
+typedef s32_t (*spiffs_write)(struct spiffs_t *fs, u32_t addr, u32_t size, const u8_t *src);
 /* spi erase call function type */
 typedef s32_t (*spiffs_erase)(struct spiffs_t *fs, u32_t addr, u32_t size);
 
@@ -93,7 +93,7 @@ typedef s32_t (*spiffs_erase)(struct spiffs_t *fs, u32_t addr, u32_t size);
 /* spi read call function type */
 typedef s32_t (*spiffs_read)(u32_t addr, u32_t size, u8_t *dst);
 /* spi write call function type */
-typedef s32_t (*spiffs_write)(u32_t addr, u32_t size, u8_t *src);
+typedef s32_t (*spiffs_write)(u32_t addr, u32_t size, const u8_t *src);
 /* spi erase call function type */
 typedef s32_t (*spiffs_erase)(u32_t addr, u32_t size);
 #endif // SPIFFS_HAL_CALLBACK_EXTRA

cores/esp8266/spiffs_api.cpp

@@ -25,6 +25,8 @@
 
 using namespace fs;
 
+namespace spiffs_impl {
+
 FileImplPtr SPIFFSImpl::open(const char* path, OpenMode openMode, AccessMode accessMode)
 {
     if (!isSpiffsFilenameValid(path)) {
@@ -108,6 +110,8 @@ bool isSpiffsFilenameValid(const char* name)
     return len > 0 && len < SPIFFS_OBJ_NAME_LEN;
 }
 
+}; // namespace
+
 // these symbols should be defined in the linker script for each flash layout
 #ifndef CORE_MOCK
 #ifdef ARDUINO
@@ -116,11 +120,11 @@ bool isSpiffsFilenameValid(const char* name)
 #endif
 
 #if !defined(NO_GLOBAL_INSTANCES) && !defined(NO_GLOBAL_SPIFFS)
-FS SPIFFS = FS(FSImplPtr(new SPIFFSImpl(
-                             SPIFFS_PHYS_ADDR,
-                             SPIFFS_PHYS_SIZE,
-                             SPIFFS_PHYS_PAGE,
-                             SPIFFS_PHYS_BLOCK,
+FS SPIFFS = FS(FSImplPtr(new spiffs_impl::SPIFFSImpl(
+                             FS_PHYS_ADDR,
+                             FS_PHYS_SIZE,
+                             FS_PHYS_PAGE,
+                             FS_PHYS_BLOCK,
                              SPIFFS_MAX_OPEN_FILES)));
 #endif // ARDUINO
 #endif // !CORE_MOCK